├── .coveragerc ├── .travis.yml ├── CHANGELOG.rst ├── LICENSE.txt ├── MANIFEST.in ├── Makefile ├── README.rst ├── apt_mirror_updater ├── __init__.py ├── backends │ ├── __init__.py │ ├── debian.py │ ├── elementary.py │ └── ubuntu.py ├── cli.py ├── http.py ├── releases.py └── tests.py ├── docs ├── api.rst ├── changelog.rst ├── conf.py ├── index.rst └── readme.rst ├── requirements-checks.txt ├── requirements-tests.txt ├── requirements-travis.txt ├── requirements.txt ├── scripts ├── collect-full-coverage.sh └── install-on-travis.sh ├── setup.cfg ├── setup.py └── tox.ini /.coveragerc: -------------------------------------------------------------------------------- 1 | [run] 2 | branch = true 3 | concurrency = multiprocessing 4 | omit = apt_mirror_updater/tests.py 5 | parallel = True 6 | source = apt_mirror_updater 7 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | sudo: true 2 | language: python 3 | python: 4 | - "2.7" 5 | - "3.5" 6 | - "3.6" 7 | - "3.7" 8 | - "3.8" 9 | - "pypy" 10 | install: 11 | - scripts/install-on-travis.sh 12 | script: 13 | - make check 14 | - make full-coverage 15 | after_success: 16 | - coveralls 17 | branches: 18 | except: 19 | - /^[0-9]/ 20 | -------------------------------------------------------------------------------- /CHANGELOG.rst: -------------------------------------------------------------------------------- 1 | Changelog 2 | ========= 3 | 4 | The purpose of this document is to list all of the notable changes to this 5 | project. The format was inspired by `Keep a Changelog`_. This project adheres 6 | to `semantic versioning`_. 7 | 8 | .. contents:: 9 | :local: 10 | 11 | .. _Keep a Changelog: http://keepachangelog.com/ 12 | .. _semantic versioning: http://semver.org/ 13 | 14 | `Release 7.3`_ (2021-09-15) 15 | --------------------------- 16 | 17 | **Significant changes:** 18 | 19 | - Updated the releases bundled in the :mod:`~apt_mirror_updater.releases` 20 | module to include the following: 21 | 22 | - Ubuntu 20.10 (Groovy Gorilla) 23 | - Ubuntu 21.04 (Hirsute Hippo) 24 | - Ubuntu 21.10 (Impish Indri) 25 | 26 | - Relaxed the :pypi:`beautifulsoup4` requirement to facilitate Python 2 27 | compatibility (release 4.9.3 was the last to support Python 2 whereas 28 | our requirements insisted on >= 4.4.1). 29 | 30 | **Miscellaenous changes:** 31 | 32 | - Updated the :func:`~apt_mirror_updater.releases.parse_version()` documentation 33 | to refer to :class:`decimal.Decimal` instead of "a floating point number" (to 34 | avoid any confusion). 35 | 36 | .. _Release 7.3: https://github.com/xolox/python-apt-mirror-updater/compare/7.2...7.3 37 | 38 | `Release 7.2`_ (2020-04-18) 39 | --------------------------- 40 | 41 | Add support for `Elementary OS`_ (suggested in `issue #10`_). 42 | 43 | Because Elementary OS is based on Ubuntu the new backend is nothing more than a 44 | thin wrapper for the Ubuntu backend. Its most significant content is a data 45 | structure with known Elementary OS releases and how they map to Ubuntu. 46 | 47 | .. _Release 7.2: https://github.com/xolox/python-apt-mirror-updater/compare/7.1...7.2 48 | .. _Elementary OS: https://en.wikipedia.org/wiki/Elementary_OS 49 | .. _issue #10: https://github.com/xolox/python-apt-mirror-updater/issues/10 50 | 51 | `Release 7.1`_ (2020-04-17) 52 | --------------------------- 53 | 54 | After Ubuntu 14.04 went EOL it became clear that when a release goes EOL and 55 | when it is archived (moved from the package mirrors to the old-releases 56 | environment) can differ by a year... 57 | 58 | This release of apt-mirror-updater acknowledges this distinction in the code 59 | and properly handles the situation where a release has already gone EOL but has 60 | not yet been archived. 61 | 62 | This was also reported in `issue #9`_ by a colleague and friend of mine. 63 | 64 | .. _Release 7.1: https://github.com/xolox/python-apt-mirror-updater/compare/7.0...7.1 65 | .. _issue #9: https://github.com/xolox/python-apt-mirror-updater/issues/9 66 | 67 | `Release 7.0`_ (2020-04-16) 68 | --------------------------- 69 | 70 | This is a maintenance release that (primarily) updates Python compatibility. 71 | 72 | **Backwards incompatible changes:** 73 | 74 | - Python 3.7 and 3.8 are now officially supported. 75 | 76 | - Python 2.6 and 3.4 are no longer supported. 77 | 78 | - Added ``python_requires`` to ``setup.py`` to aid :pypi:`pip` in version 79 | selection given these compatibility changes. 80 | 81 | **Significant changes:** 82 | 83 | - Updated the releases bundled in the :mod:`apt_mirror_updater.releases` module 84 | to include the following: 85 | 86 | - Ubuntu 19.04 (Disco Dingo) 87 | - Ubuntu 19.10 (Eoan Ermine) 88 | - Ubuntu 20.04 (Focal Fossa). 89 | 90 | - Bug fix for "accidental tuple" in :func:`apt_mirror_updater.releases.parse_csv_file()`. 91 | 92 | **Miscellaenous changes:** 93 | 94 | - Spent some time stabilizing the test suite on Travis CI (tests were passing 95 | for me locally but not on Travis CI because the mirror selection differed). 96 | As a result the test suite got a bit slower, but it's not too bad. 97 | 98 | - Move caching decorator to :pypi:`humanfriendly`. 99 | 100 | - Fixed deprecation warnings emitted by recent :pypi:`humanfriendly` releases 101 | and bumped requirements I authored that went through similar changes. 102 | 103 | - Made :mod:`multiprocessing` usage compatible with coverage collection. Note 104 | that I don't expect this to increase coverage, I just wanted to get rid of 105 | the warnings 😇 (because warnings about harmless things are just as 106 | distracting as more pertinent warnings). 107 | 108 | - Default to Python 3 for local development (required by :pypi:`Sphinx` among 109 | other things). 110 | 111 | - Fixed existing :pypi:`Sphinx` reference warnings in the documentation and 112 | changed the :man:`sphinx-build` invocation to promote warnings to errors (to 113 | aid me in the discipline of not introducing broken references from now on). 114 | 115 | .. _Release 7.0: https://github.com/xolox/python-apt-mirror-updater/compare/6.1...7.0 116 | 117 | `Release 6.1`_ (2018-10-19) 118 | --------------------------- 119 | 120 | - Bug fix for Ubuntu keyring selection that prevented 121 | ``ubuntu-archive-removed-keys.gpg`` from being used. 122 | - Bug fix for :func:`~apt_mirror_updater.releases.coerce_release()` 123 | when given a release number. 124 | - Moved pathnames of Debian and Ubuntu keyring files to constants. 125 | - Added logging to enable debugging of keyring selection process. 126 | - Added proper tests for keyring selection and release coercion. 127 | 128 | .. _Release 6.1: https://github.com/xolox/python-apt-mirror-updater/compare/6.0...6.1 129 | 130 | `Release 6.0`_ (2018-10-14) 131 | --------------------------- 132 | 133 | Enable the creation of Ubuntu <= 12.04 chroots on Ubuntu >= 17.04 hosts by 134 | working around (what I am convinced is) a bug in :man:`debootstrap` which picks 135 | the wrong keyring when setting up chroots of old releases. For more information 136 | refer to issue `#8`_. 137 | 138 | I've bumped the major version number for this release because the highly 139 | specific ``apt_mirror_updater.eol`` module changed into the much more generic 140 | :mod:`apt_mirror_updater.releases` module. Also the ``release_label`` property was 141 | removed. 142 | 143 | .. _Release 6.0: https://github.com/xolox/python-apt-mirror-updater/compare/5.2...6.0 144 | .. _#8: https://github.com/xolox/python-apt-mirror-updater/issues/8 145 | 146 | `Release 5.2`_ (2018-10-08) 147 | --------------------------- 148 | 149 | Use `mirrors.ubuntu.com/mirrors.txt`_ without placing our full trust in it like 150 | older versions of :pypi:`apt-mirror-updater` did 😇. 151 | 152 | Feedback in issue `#6`_ suggested that `mirrors.ubuntu.com/mirrors.txt`_ is 153 | working properly (again) and should be preferred over scraping Launchpad. 154 | However I prefer for :pypi:`apt-mirror-updater` to be a reliable "do what I 155 | mean" program and `mirrors.ubuntu.com/mirrors.txt`_ has proven to be unreliable 156 | in the past (see the discussion in `#6`_). As a compromise I've changed the 157 | Ubuntu mirror discovery as follows: 158 | 159 | 1. Discover Ubuntu mirrors on Launchpad. 160 | 161 | 2. Try to discover mirrors using `mirrors.ubuntu.com/mirrors.txt`_ and iff 162 | successful, narrow down the list produced in step 1 based on the URLs 163 | reported in step 2. 164 | 165 | 3. Rank the discovered / narrowed down mirrors and pick the best one. 166 | 167 | The reason why I've decided to add this additional complexity is because it has 168 | bothered me in the past that Ubuntu mirror discovery was slow and this does 169 | help a lot. Also, why not use a service provided by Ubuntu to speed things up? 170 | 171 | Unrelated to the use of `mirrors.ubuntu.com/mirrors.txt`_ I've also bumped the 172 | :pypi:`executor` requirement (twice) in order to pull in upstream improvements 173 | discussed in `executor issue #10`_ and `executor issue #15`_. 174 | 175 | .. _Release 5.2: https://github.com/xolox/python-apt-mirror-updater/compare/5.1...5.2 176 | .. _mirrors.ubuntu.com/mirrors.txt: http://mirrors.ubuntu.com/mirrors.txt 177 | .. _#6: https://github.com/xolox/python-apt-mirror-updater/issues/6 178 | .. _executor issue #10: https://github.com/xolox/python-executor/issues/10 179 | .. _executor issue #15: https://github.com/xolox/python-executor/issues/15 180 | 181 | `Release 5.1`_ (2018-06-22) 182 | --------------------------- 183 | 184 | Work on release 5.1 started with the intention of publishing a 5.0.2 bug fix 185 | release for the EOL detection of Debian LTS releases reported in `#5`_, however 186 | unrelated changes were required to stabilize the test suite. This explains how 187 | 5.0.2 became 5.1 😇. 188 | 189 | When I started working on resolving the issue reported in `#5`_ it had been 190 | quite a while since the previous release (233 days) and so some technical debt 191 | had accumulated in the project, causing the test suite to break. Most 192 | significantly, Travis CI switched their workers from Ubuntu 12.04 to 14.04. 193 | 194 | Here's a detailed overview of changes: 195 | 196 | - Bug fix for EOL detection of Debian LTS releases (reported in `#5`_). 197 | - Bug fix for trivial string matching issue in test suite (caused by a naively 198 | written test). 199 | - Bug fix for recursive ``repr()`` calls potentially causing infinite 200 | recursion, depending on logging level (see e.g. build 395421319_). 201 | - Updated bundled EOL dates based on distro-info-data available in Ubuntu 18.04. 202 | - Added this changelog to the documentation, including a link in the readme. 203 | - Make sure the ``test_gather_eol_dates`` test method runs on Travis CI (by 204 | installing the distro-info-data_ package). This exposed a Python 3 205 | incompatibility (in build 395410569_) that has since been resolved. 206 | - Include documentation in source distributions (``MANIFEST.in``). 207 | - Silence flake8 complaining about bogus D402 issues. 208 | - Add license='MIT' key to ``setup.py`` script. 209 | - Bumped copyright to 2018. 210 | 211 | .. _Release 5.1: https://github.com/xolox/python-apt-mirror-updater/compare/5.0.1...5.1 212 | .. _#5: https://github.com/xolox/python-apt-mirror-updater/issues/5 213 | .. _395421319: https://travis-ci.org/xolox/python-apt-mirror-updater/jobs/395421319 214 | .. _distro-info-data: https://packages.ubuntu.com/distro-info-data 215 | .. _395410569: https://travis-ci.org/xolox/python-apt-mirror-updater/jobs/395410569 216 | 217 | `Release 5.0.1`_ (2017-11-01) 218 | ----------------------------- 219 | 220 | Bug fix release for invalid enumeration value (oops). 221 | 222 | .. _Release 5.0.1: https://github.com/xolox/python-apt-mirror-updater/compare/5.0...5.0.1 223 | 224 | `Release 5.0`_ (2017-11-01) 225 | --------------------------- 226 | 227 | .. |smart_update| replace:: :func:`~apt_mirror_updater.AptMirrorUpdater.smart_update()` 228 | .. |validate_mirror| replace:: :func:`~apt_mirror_updater.AptMirrorUpdater.validate_mirror()` 229 | 230 | Reliable end of life (EOL) detection. 231 | 232 | Recently I ran into the issue that the logic to check whether a release is EOL 233 | (that works by checking if the security mirror serves a ``Release.gpg`` file 234 | for the release) failed on me. More specifically the following URL existed at 235 | the time of writing (2017-11-01) even though Ubuntu 12.04 went EOL back in 236 | April: 237 | 238 | http://security.ubuntu.com/ubuntu/dists/precise/Release.gpg 239 | 240 | At the same time issue `#1`_ and pull request `#2`_ were also indications that 241 | the EOL detection was fragile and error prone. This potential fragility had 242 | bugged me ever since publishing :pypi:`apt-mirror-updater` and this week I 243 | finally finished a more robust and deterministic EOL detection scheme. 244 | 245 | This release includes pull requests `#2`_ and `#4`_, fixing issues `#1`_ and 246 | `#3`_. Here's a detailed overview of changes: 247 | 248 | - Addition: Allow optional arguments to ``apt-get update`` (`#3`_, `#4`_). 249 | 250 | - I simplified and improved the feature requested in issue `#3`_ and 251 | implemented in pull request `#4`_ by switching from an optional list 252 | argument to 'star-args' and applying the same calling convention to 253 | |smart_update| as well. 254 | 255 | - This is backwards incompatible with the implementation in pull request 256 | `#4`_ (which I merged into the ``dev`` branch but never published to PyPI) 257 | and it's also technically backwards incompatible in the sense that keyword 258 | arguments could previously be given to |smart_update| as positional 259 | arguments. This explains why I'm bumping the major version number. 260 | 261 | - Bug fix for incorrect marking of EOL when HTTP connections fail (`#2`_). 262 | - Refactoring: Apply timeout handling to HTTP response bodies. 263 | - Refactoring: Distinguish 404 from other HTTP errors: 264 | 265 | - This change enhances |validate_mirror| by making a distinction between 266 | a confirmed HTTP 404 response versus other error conditions which may be of 267 | a more transient nature. 268 | - The goal of this change is to preserve the semantics requested in issue 269 | `#1`_ and implemented in pull request `#2`_ without needing the additional 270 | HTTP request performed by ``can_connect_to_mirror()``. 271 | - Because |validate_mirror| previously returned a boolean but now returns 272 | an enumeration member this change is technically backwards incompatible, 273 | then again |validate_mirror| isn't specifically intended for callers 274 | because it concerns internal logic of apt-mirror-updater. I'm nevertheless 275 | bumping the major version number. 276 | 277 | - Refactoring: Improve HTTP request exception handling: 278 | 279 | - 404 responses and timeouts are no longer subject to retrying. 280 | - The exception :exc:`apt_mirror_updater.http.NotFoundError` is now raised on 281 | HTTP 404 responses. Other unexpected HTTP response codes raise 282 | :exc:`apt_mirror_updater.http.InvalidResponseError`. 283 | - The specific distinction between 404 and !200 was made because the 404 284 | response has become significant in checking for EOL status. 285 | 286 | .. _Release 5.0: https://github.com/xolox/python-apt-mirror-updater/compare/4.0...5.0 287 | .. _#1: https://github.com/xolox/python-apt-mirror-updater/issues/1 288 | .. _#2: https://github.com/xolox/python-apt-mirror-updater/pull/2 289 | .. _#3: https://github.com/xolox/python-apt-mirror-updater/issues/3 290 | .. _#4: https://github.com/xolox/python-apt-mirror-updater/pull/4 291 | 292 | `Release 4.0`_ (2017-06-14) 293 | --------------------------- 294 | 295 | Robust validation of available mirrors (backwards incompatible). 296 | 297 | .. _Release 4.0: https://github.com/xolox/python-apt-mirror-updater/compare/3.1...4.0 298 | 299 | `Release 3.1`_ (2017-06-13) 300 | --------------------------- 301 | 302 | Made mirror comparison more robust. 303 | 304 | .. _Release 3.1: https://github.com/xolox/python-apt-mirror-updater/compare/3.0...3.1 305 | 306 | `Release 3.0`_ (2017-06-13) 307 | --------------------------- 308 | 309 | Added Debian archive support (with old releases): 310 | 311 | - Addition: Added Debian archive support (old releases). 312 | - Improvement: Don't bother validating archive / old-releases mirror. 313 | - Refactoring: Moved URLs to backend specific modules. 314 | 315 | .. _Release 3.0: https://github.com/xolox/python-apt-mirror-updater/compare/2.1...3.0 316 | 317 | `Release 2.1`_ (2017-06-12) 318 | --------------------------- 319 | 320 | Restored Python 3 compatibility, improved robustness: 321 | 322 | - Improvement: Make the ``is_available`` and ``is_updating`` properties of the 323 | ``CandidateMirror`` class more robust. 324 | - Bug fix: I suck at Unicode in Python (most people do :-p). 325 | - Cleanup: Remove unused import from test suite. 326 | 327 | .. _Release 2.1: https://github.com/xolox/python-apt-mirror-updater/compare/2.0...2.1 328 | 329 | `Release 2.0`_ (2017-06-11) 330 | --------------------------- 331 | 332 | Generation of ``sources.list`` files and chroot creation. 333 | 334 | Detailed overview of changes: 335 | 336 | - Addition: Added a simple :man:`debootstrap` wrapper. 337 | - Addition: Programmatic ``/etc/apt/sources.list`` generation 338 | - Bug fix for ``check_suite_available()``. 339 | - Bug fix: Never apply Ubuntu's old release handling to Debian. 340 | - Bug fix: Never remove ``/var/lib/apt/lists/lock`` file. 341 | - Improvement: Enable stable mirror selection 342 | - Improvement: Make it possible to override distributor ID and codename 343 | - Improvement: Render interactive spinner during mirror ranking. 344 | - Refactoring: Generalize AptMirrorUpdater initializer (backwards incompatible!) 345 | - Refactoring: Generalize backend module loading 346 | - Refactoring: Modularize ``/etc/apt/sources.list`` writing. 347 | 348 | .. _Release 2.0: https://github.com/xolox/python-apt-mirror-updater/compare/1.0...2.0 349 | 350 | `Release 1.0`_ (2017-06-08) 351 | --------------------------- 352 | 353 | Improved Ubuntu mirror discovery, added an automated test suite, and more. 354 | 355 | The bump to version 1.0 isn't so much intended to communicate that this 356 | is now mature software, it's just that I made several backwards 357 | incompatible changes in order to improve the modularity of the code 358 | base, make it easier to develop automated tests, maintain platform 359 | support, etc :-). 360 | 361 | A more detailed overview of (significant) changes: 362 | 363 | - Improved Ubuntu mirror discovery (by scraping Launchpad instead). 364 | - Extracted mirror discovery to separate (backend specific) modules. 365 | - Extracted HTTP handling to a separate module. 366 | - Enable Control-C to interrupt concurrent connection tests. 367 | - Expose limit in Python API and command line interface and make limit optional by passing 0. 368 | - Bug fix for Python 3 incompatibility: Stop using :data:`sys.maxint` :-). 369 | 370 | .. _Release 1.0: https://github.com/xolox/python-apt-mirror-updater/compare/0.3.1...1.0 371 | 372 | `Release 0.3.1`_ (2016-06-29) 373 | ----------------------------- 374 | 375 | Avoid 'nested' smart updates (the old code worked fine but gave confusing 376 | output and performed more work than necessary, which bothered me :-). 377 | 378 | .. _Release 0.3.1: https://github.com/xolox/python-apt-mirror-updater/compare/0.3...0.3.1 379 | 380 | `Release 0.3`_ (2016-06-29) 381 | --------------------------- 382 | 383 | Make smart update understand EOL suites. 384 | 385 | .. _Release 0.3: https://github.com/xolox/python-apt-mirror-updater/compare/0.2...0.3 386 | 387 | `Release 0.2`_ (2016-06-29) 388 | --------------------------- 389 | 390 | Bug fix: Replace ``security.ubuntu.com`` as well. 391 | 392 | .. _Release 0.2: https://github.com/xolox/python-apt-mirror-updater/compare/0.1.2...0.2 393 | 394 | `Release 0.1.2`_ (2016-06-29) 395 | ----------------------------- 396 | 397 | Bug fix: Explicitly terminate multiprocessing pool. 398 | 399 | .. _Release 0.1.2: https://github.com/xolox/python-apt-mirror-updater/compare/0.1.1...0.1.2 400 | 401 | `Release 0.1.1`_ (2016-03-10) 402 | ----------------------------- 403 | 404 | Initial release (added ``MANIFEST.in``). 405 | 406 | .. _Release 0.1.1: https://github.com/xolox/python-apt-mirror-updater/compare/0.1...0.1.1 407 | 408 | `Release 0.1`_ (2016-03-10) 409 | --------------------------- 410 | 411 | Initial commit. 412 | 413 | .. _Release 0.1: https://github.com/xolox/python-apt-mirror-updater/tree/0.1 414 | -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | Copyright (c) 2020 Peter Odding 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining 4 | a copy of this software and associated documentation files (the 5 | "Software"), to deal in the Software without restriction, including 6 | without limitation the rights to use, copy, modify, merge, publish, 7 | distribute, sublicense, and/or sell copies of the Software, and to 8 | permit persons to whom the Software is furnished to do so, subject to 9 | the following conditions: 10 | 11 | The above copyright notice and this permission notice shall be 12 | included in all copies or substantial portions of the Software. 13 | 14 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 15 | EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 16 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 17 | NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE 18 | LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 19 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION 20 | WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 21 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include *.rst 2 | include *.txt 3 | graft docs 4 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | # Makefile for the 'apt-mirror-updater' package. 2 | # 3 | # Author: Peter Odding 4 | # Last Change: April 15, 2020 5 | # URL: https://apt-mirror-updater.readthedocs.io 6 | 7 | PACKAGE_NAME = apt-mirror-updater 8 | WORKON_HOME ?= $(HOME)/.virtualenvs 9 | VIRTUAL_ENV ?= $(WORKON_HOME)/$(PACKAGE_NAME) 10 | PYTHON ?= python3 11 | PATH := $(VIRTUAL_ENV)/bin:$(PATH) 12 | MAKE := $(MAKE) --no-print-directory 13 | SHELL = bash 14 | 15 | default: 16 | @echo "Makefile for $(PACKAGE_NAME)" 17 | @echo 18 | @echo 'Usage:' 19 | @echo 20 | @echo ' make install install the package in a virtual environment' 21 | @echo ' make reset recreate the virtual environment' 22 | @echo ' make check check coding style (PEP-8, PEP-257)' 23 | @echo ' make test run the test suite, report coverage' 24 | @echo ' make tox run the tests on all Python versions' 25 | @echo ' make releases update apt_mirror_updater.releases module' 26 | @echo ' make readme update usage in readme' 27 | @echo ' make docs update documentation using Sphinx' 28 | @echo ' make publish publish changes to GitHub/PyPI' 29 | @echo ' make clean cleanup all temporary files' 30 | @echo 31 | 32 | install: 33 | @test -d "$(VIRTUAL_ENV)" || mkdir -p "$(VIRTUAL_ENV)" 34 | @test -x "$(VIRTUAL_ENV)/bin/python" || virtualenv --python=$(PYTHON) --quiet "$(VIRTUAL_ENV)" 35 | @pip install --quiet --requirement=requirements.txt 36 | @pip uninstall --yes $(PACKAGE_NAME) &>/dev/null || true 37 | @pip install --quiet --no-deps --ignore-installed . 38 | 39 | reset: 40 | @$(MAKE) clean 41 | @rm -Rf "$(VIRTUAL_ENV)" 42 | @$(MAKE) install 43 | 44 | check: install 45 | @pip install --upgrade --quiet --requirement=requirements-checks.txt 46 | @flake8 47 | 48 | test: install 49 | @pip install --quiet --requirement=requirements-tests.txt 50 | @py.test --cov 51 | @coverage html 52 | 53 | tox: install 54 | @pip install --quiet tox 55 | @tox 56 | 57 | # The following makefile target isn't documented on purpose, I don't want 58 | # people to execute this without them knowing very well what they're doing. 59 | 60 | full-coverage: install 61 | @pip install --quiet --requirement=requirements-tests.txt 62 | @scripts/collect-full-coverage.sh 63 | @coverage html 64 | 65 | cog: install 66 | @echo Installing cog ... 67 | @pip install --quiet cogapp 68 | 69 | releases: cog 70 | @cog.py -r apt_mirror_updater/releases.py 71 | 72 | readme: cog 73 | @cog.py -r README.rst 74 | 75 | docs: releases readme 76 | @pip install --quiet sphinx 77 | @cd docs && sphinx-build -nWb html -d build/doctrees . build/html 78 | 79 | publish: install 80 | @git push origin && git push --tags origin 81 | @$(MAKE) clean 82 | @pip install --quiet twine wheel 83 | @python setup.py sdist bdist_wheel 84 | @twine upload dist/* 85 | @$(MAKE) clean 86 | 87 | clean: 88 | @rm -Rf *.egg .cache .coverage .tox build dist docs/build htmlcov 89 | @find -depth -type d -name __pycache__ -exec rm -Rf {} \; 90 | @find -type f -name '*.pyc' -delete 91 | 92 | .PHONY: default install reset check test tox full-coverage cog releases readme docs publish clean 93 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | apt-mirror-updater: Automated Debian/Ubuntu mirror selection 2 | ============================================================ 3 | 4 | .. image:: https://travis-ci.org/xolox/python-apt-mirror-updater.svg?branch=master 5 | :target: https://travis-ci.org/xolox/python-apt-mirror-updater 6 | 7 | .. image:: https://coveralls.io/repos/xolox/python-apt-mirror-updater/badge.svg?branch=master 8 | :target: https://coveralls.io/r/xolox/python-apt-mirror-updater?branch=master 9 | 10 | The `apt-mirror-updater` package automates robust apt-get_ mirror selection for 11 | Debian_ and Ubuntu_ by enabling discovery of available mirrors, ranking of 12 | available mirrors, automatic switching between mirrors and robust package list 13 | updating (see features_). It's currently tested on Python 2.7, 3.5+ and PyPy 14 | (although test coverage is still rather low, see status_). 15 | 16 | .. contents:: 17 | :local: 18 | 19 | .. _features: 20 | 21 | Features 22 | -------- 23 | 24 | **Discovery of available mirrors** 25 | Debian_ and Ubuntu_ mirrors are discovered automatically by querying the 26 | `Debian mirror list `_ or the `Ubuntu 27 | mirror list `_ (the applicable 28 | mirror list is automatically selected based on the current platform). 29 | 30 | **Ranking of available mirrors** 31 | Discovered mirrors are ranked by bandwidth (to pick the fastest mirror) and 32 | excluded if they're being updated (see `issues with mirror updates`_). 33 | 34 | **Automatic switching between mirrors** 35 | The main mirror configured in ``/etc/apt/sources.list`` can be changed with a 36 | single command. The new (to be configured) mirror can be selected 37 | automatically or configured explicitly by the user. 38 | 39 | **Robust package list updating** 40 | Several apt-get_ subcommands can fail if the current mirror is being updated 41 | (see `issues with mirror updates`_) and `apt-mirror-updater` tries to work 42 | around this by wrapping ``apt-get update`` to retry on failures and 43 | automatically switch to a different mirror when it looks like the current 44 | mirror is being updated (because I've seen such updates take more than 15 45 | minutes and it's not always acceptable to wait for so long, especially in 46 | automated solutions). 47 | 48 | .. _status: 49 | 50 | Status 51 | ------ 52 | 53 | On the one hand the `apt-mirror-updater` package was developed based on quite a 54 | few years of experience in using apt-get_ on Debian_ and Ubuntu_ systems and 55 | large scale automation of apt-get (working on 150+ remote systems). On the 56 | other hand the Python package itself is relatively new: it was developed and 57 | published in March 2016. As such: 58 | 59 | .. warning:: Until `apt-mirror-updater` has been rigorously tested I consider 60 | it a proof of concept (beta software) so if it corrupts your 61 | system you can't complain that you weren't warned! I've already 62 | tested it on a variety of Ubuntu systems but haven't found the 63 | time to set up a Debian virtual machine for testing. Most of the 64 | logic is exactly the same though. The worst that can happen 65 | (assuming you trust my judgement ;-) is that 66 | ``/etc/apt/sources.list`` is corrupted however a backup copy is 67 | made before any changes are applied, so I don't see how this can 68 | result in irreversible corruption. 69 | 70 | I'm working on an automated test suite but at the moment I'm still a bit fuzzy 71 | on how to create representative tests for the error handling code paths (also, 72 | writing a decent test suite requires a significant chunk of time :-). 73 | 74 | Installation 75 | ------------ 76 | 77 | The `apt-mirror-updater` package is available on PyPI_ which means installation 78 | should be as simple as: 79 | 80 | .. code-block:: sh 81 | 82 | $ pip install apt-mirror-updater 83 | 84 | There's actually a multitude of ways to install Python packages (e.g. the `per 85 | user site-packages directory`_, `virtual environments`_ or just installing 86 | system wide) and I have no intention of getting into that discussion here, so 87 | if this intimidates you then read up on your options before returning to these 88 | instructions ;-). 89 | 90 | Usage 91 | ----- 92 | 93 | There are two ways to use the `apt-mirror-updater` package: As the command line 94 | program ``apt-mirror-updater`` and as a Python API. For details about the 95 | Python API please refer to the API documentation available on `Read the Docs`_. 96 | The command line interface is described below. 97 | 98 | .. contents:: 99 | :local: 100 | 101 | .. A DRY solution to avoid duplication of the `apt-mirror-updater --help' text: 102 | .. 103 | .. [[[cog 104 | .. from humanfriendly.usage import inject_usage 105 | .. inject_usage('apt_mirror_updater.cli') 106 | .. ]]] 107 | 108 | **Usage:** `apt-mirror-updater [OPTIONS]` 109 | 110 | The apt-mirror-updater program automates robust apt-get mirror selection for 111 | Debian and Ubuntu by enabling discovery of available mirrors, ranking of 112 | available mirrors, automatic switching between mirrors and robust package list 113 | updating. 114 | 115 | **Supported options:** 116 | 117 | .. csv-table:: 118 | :header: Option, Description 119 | :widths: 30, 70 120 | 121 | 122 | "``-r``, ``--remote-host=SSH_ALIAS``","Operate on a remote system instead of the local system. The ``SSH_ALIAS`` 123 | argument gives the SSH alias of the remote host. It is assumed that the 124 | remote account has root privileges or password-less sudo access." 125 | "``-f``, ``--find-current-mirror``","Determine the main mirror that is currently configured in 126 | /etc/apt/sources.list and report its URL on standard output." 127 | "``-b``, ``--find-best-mirror``","Discover available mirrors, rank them, select the best one and report its 128 | URL on standard output." 129 | "``-l``, ``--list-mirrors``",List available (ranked) mirrors on the terminal in a human readable format. 130 | "``-c``, ``--change-mirror=MIRROR_URL``",Update /etc/apt/sources.list to use the given ``MIRROR_URL``. 131 | "``-a``, ``--auto-change-mirror``","Discover available mirrors, rank the mirrors by connection speed and update 132 | status and update /etc/apt/sources.list to use the best available mirror." 133 | "``-u``, ``--update``, ``--update-package-lists``","Update the package lists using ""apt-get update"", retrying on failure and 134 | automatically switch to a different mirror when it looks like the current 135 | mirror is being updated." 136 | "``-x``, ``--exclude=PATTERN``","Add a pattern to the mirror selection blacklist. ``PATTERN`` is expected to be 137 | a shell pattern (containing wild cards like ""?"" and ""\*"") that is matched 138 | against the full URL of each mirror." 139 | "``-m``, ``--max=COUNT``","Don't query more than ``COUNT`` mirrors for their connection status 140 | (defaults to 50). If you give the number 0 no limit will be applied. 141 | 142 | Because Ubuntu mirror discovery can report more than 300 mirrors it's 143 | useful to limit the number of mirrors that are queried, otherwise the 144 | ranking of mirrors will take a long time (because over 300 connections 145 | need to be established)." 146 | "``-v``, ``--verbose``",Increase logging verbosity (can be repeated). 147 | "``-q``, ``--quiet``",Decrease logging verbosity (can be repeated). 148 | "``-h``, ``--help``",Show this message and exit. 149 | 150 | .. [[[end]]] 151 | 152 | .. _issues with mirror updates: 153 | 154 | Issues with mirror updates 155 | -------------------------- 156 | 157 | Over the past five years my team (`at work`_) and I have been managing a 158 | cluster of 150+ Ubuntu servers, initially using manual system administration 159 | but over time automating ``apt-get`` for a variety of use cases (provisioning, 160 | security updates, deployments, etc.). As we increased our automation we started 161 | running into various transient failure modes of ``apt-get``, primarily with 162 | ``apt-get update`` but incidentally also with other subcommands. 163 | 164 | The most frequent failure that we run into is ``apt-get update`` crapping out 165 | with 'hash sum mismatch' errors (see also `Debian bug #624122`_). When this 166 | happens a file called ``Archive-Update-in-Progress-*`` can sometimes be found 167 | on the index page of the mirror that is being used (see also `Debian bug 168 | #110837`_). I've seen these situations last for more than 15 minutes. 169 | 170 | My working theory about these 'hash sum mismatch' errors is that they are 171 | caused by the fact that mirror updates aren't atomic, apparently causing 172 | ``apt-get update`` to download a package list whose datafiles aren't consistent 173 | with each other. If this assumption proves to be correct (and also assuming 174 | that different mirrors are updated at different times :-) then the command 175 | ``apt-mirror-updater --update-package-lists`` should work around this annoying 176 | failure mode (by automatically switching to a different mirror when 'hash sum 177 | mismatch' errors are encountered). 178 | 179 | Publishing `apt-mirror-updater` to the world is my attempt to contribute to 180 | this situation instead of complaining in bug trackers (see above) where no 181 | robust and automated solution is emerging (at the time of writing). Who knows, 182 | maybe some day these issues will be resolved by moving logic similar to what 183 | I've implemented here into ``apt-get`` itself. Of course it would also help if 184 | mirror updates were atomic... 185 | 186 | Contact 187 | ------- 188 | 189 | The latest version of `apt-mirror-updater` is available on PyPI_ and GitHub_. 190 | The documentation is hosted on `Read the Docs`_ and includes a changelog_. For 191 | bug reports please create an issue on GitHub_. If you have questions, 192 | suggestions, etc. feel free to send me an e-mail at `peter@peterodding.com`_. 193 | 194 | License 195 | ------- 196 | 197 | This software is licensed under the `MIT license`_. 198 | 199 | © 2020 Peter Odding. 200 | 201 | 202 | .. External references: 203 | .. _apt-get: https://en.wikipedia.org/wiki/Advanced_Packaging_Tool 204 | .. _at work: http://www.paylogic.com/ 205 | .. _changelog: https://apt-mirror-updater.readthedocs.io/changelog.html 206 | .. _Debian bug #110837: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=110837 207 | .. _Debian bug #624122: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=624122 208 | .. _Debian: https://en.wikipedia.org/wiki/Debian 209 | .. _documentation: https://apt-mirror-updater.readthedocs.io 210 | .. _GitHub: https://github.com/xolox/python-apt-mirror-updater 211 | .. _MIT license: http://en.wikipedia.org/wiki/MIT_License 212 | .. _per user site-packages directory: https://www.python.org/dev/peps/pep-0370/ 213 | .. _peter@peterodding.com: peter@peterodding.com 214 | .. _PyPI: https://pypi.python.org/pypi/apt-mirror-updater 215 | .. _Read the Docs: https://apt-mirror-updater.readthedocs.io 216 | .. _Ubuntu: https://en.wikipedia.org/wiki/Ubuntu_(operating_system) 217 | .. _virtual environments: http://docs.python-guide.org/en/latest/dev/virtualenvs/ 218 | -------------------------------------------------------------------------------- /apt_mirror_updater/__init__.py: -------------------------------------------------------------------------------- 1 | # Automated, robust apt-get mirror selection for Debian and Ubuntu. 2 | # 3 | # Author: Peter Odding 4 | # Last Change: September 15, 2021 5 | # URL: https://apt-mirror-updater.readthedocs.io 6 | 7 | """ 8 | Automated, robust ``apt-get`` mirror selection for Debian and Ubuntu. 9 | 10 | The main entry point for this module is the :class:`AptMirrorUpdater` class, so 11 | if you don't know where to start that would be a good place :-). You can also 12 | take a look at the source code of the :mod:`apt_mirror_updater.cli` module for 13 | an example that uses the :class:`AptMirrorUpdater` class. 14 | """ 15 | 16 | # Standard library modules. 17 | import fnmatch 18 | import logging 19 | import os 20 | import sys 21 | import time 22 | 23 | # Python 2.x / 3.x compatibility. 24 | try: 25 | from enum import Enum 26 | except ImportError: 27 | from flufl.enum import Enum 28 | 29 | # External dependencies. 30 | from capturer import CaptureOutput 31 | from executor.contexts import ChangeRootContext, LocalContext 32 | from humanfriendly import Timer, format_timespan 33 | from humanfriendly.text import compact, pluralize 34 | from humanfriendly.terminal.spinners import AutomaticSpinner 35 | from property_manager import ( 36 | PropertyManager, 37 | cached_property, 38 | key_property, 39 | lazy_property, 40 | mutable_property, 41 | set_property, 42 | ) 43 | from six import text_type 44 | from six.moves.urllib.parse import urlparse 45 | 46 | # Modules included in our package. 47 | from apt_mirror_updater.http import NotFoundError, fetch_concurrent, fetch_url, get_default_concurrency 48 | from apt_mirror_updater.releases import coerce_release 49 | 50 | # Semi-standard module versioning. 51 | __version__ = '7.3' 52 | 53 | MAIN_SOURCES_LIST = '/etc/apt/sources.list' 54 | """The absolute pathname of the list of configured APT data sources (a string).""" 55 | 56 | SOURCES_LIST_ENCODING = 'UTF-8' 57 | """The text encoding of :data:`MAIN_SOURCES_LIST` (a string).""" 58 | 59 | MAX_MIRRORS = 50 60 | """A sane default value for :attr:`AptMirrorUpdater.max_mirrors`.""" 61 | 62 | LAST_UPDATED_DEFAULT = 60 * 60 * 24 * 7 * 4 63 | """A default, pessimistic :attr:`~CandidateMirror.last_updated` value (a number).""" 64 | 65 | # Initialize a logger for this module. 66 | logger = logging.getLogger(__name__) 67 | 68 | 69 | class AptMirrorUpdater(PropertyManager): 70 | 71 | """Python API for the `apt-mirror-updater` program.""" 72 | 73 | repr_properties = ( 74 | 'architecture', 75 | 'backend', 76 | 'blacklist', 77 | 'concurrency', 78 | 'context', 79 | 'distribution_codename', 80 | 'distributor_id', 81 | 'max_mirrors', 82 | 'old_releases_url', 83 | 'security_url', 84 | ) 85 | """ 86 | Override the list of properties included in :func:`repr()` output (a tuple of strings). 87 | 88 | The :class:`~property_manager.PropertyManager` superclass defines a 89 | :meth:`~property_manager.PropertyManager.__repr__()` method that includes 90 | the values of computed properties in its output. 91 | 92 | In the case of `apt-mirror-updater` this behavior would trigger external 93 | command execution and (lots of) HTTP calls, sometimes with unintended side 94 | effects, namely `infinite recursion`_. 95 | 96 | By setting :attr:`repr_properties` to a list of "safe" properties this 97 | problematic behavior can be avoided. 98 | 99 | .. _infinite recursion: https://travis-ci.org/xolox/python-apt-mirror-updater/jobs/395421319 100 | """ 101 | 102 | @mutable_property 103 | def architecture(self): 104 | """ 105 | The name of the Debian package architecture (a string like 'i386' or 'amd64'). 106 | 107 | The package architecture is used to detect whether `Debian LTS`_ status 108 | applies to the given system (the Debian LTS team supports a specific 109 | subset of package architectures). 110 | 111 | .. _Debian LTS: https://wiki.debian.org/LTS 112 | """ 113 | value = self.context.capture('dpkg', '--print-architecture') 114 | set_property(self, 'architecture', value) 115 | return value 116 | 117 | @cached_property 118 | def available_mirrors(self): 119 | """ 120 | A list of :class:`CandidateMirror` objects (ordered from best to worst). 121 | 122 | On Ubuntu the mirrors will be ordered by the time since they were most 123 | recently updated. On Debian this information isn't available and the 124 | ordering of the list should be considered arbitrary. 125 | """ 126 | mirrors = set() 127 | if self.release_is_archived: 128 | logger.debug("Skipping mirror discovery because %s is EOL and has been archived.", self.release) 129 | else: 130 | for candidate in self.backend.discover_mirrors(): 131 | if any(fnmatch.fnmatch(candidate.mirror_url, pattern) for pattern in self.blacklist): 132 | logger.warning("Ignoring blacklisted mirror %s.", candidate.mirror_url) 133 | else: 134 | candidate.updater = self 135 | mirrors.add(candidate) 136 | # We make an attempt to incorporate the system's current mirror in 137 | # the candidates but we don't propagate failures while doing so. 138 | try: 139 | # Gotcha: We should never include the system's current mirror in 140 | # the candidates when we're bootstrapping a chroot for a different 141 | # platform. 142 | if self.distributor_id == self.context.distributor_id: 143 | mirrors.add(CandidateMirror(mirror_url=self.current_mirror, updater=self)) 144 | except Exception as e: 145 | logger.warning("Failed to add current mirror to set of available mirrors! (%s)", e) 146 | # Sort the mirrors based on the currently available information. 147 | return sorted(mirrors, key=lambda c: c.sort_key, reverse=True) 148 | 149 | @cached_property 150 | def backend(self): 151 | """ 152 | The backend module whose name matches :attr:`distributor_id`. 153 | 154 | :raises: :exc:`~exceptions.EnvironmentError` when no matching backend 155 | module is available. 156 | """ 157 | module_path = "%s.backends.%s" % (__name__, self.distributor_id) 158 | logger.debug("Checking if '%s' module can be imported ..", module_path) 159 | try: 160 | __import__(module_path) 161 | except ImportError: 162 | msg = "%s platform is unsupported! (only Debian and Ubuntu are supported)" 163 | raise EnvironmentError(msg % self.distributor_id.capitalize()) 164 | else: 165 | return sys.modules[module_path] 166 | 167 | @cached_property 168 | def best_mirror(self): 169 | """ 170 | The URL of the first mirror in :attr:`ranked_mirrors` (a string). 171 | 172 | This is a shortcut for using :attr:`ranked_mirrors` to select the 173 | best mirror from :attr:`available_mirrors`, falling back to the 174 | old releases URL when :attr:`release_is_archived` is :data:`True`. 175 | """ 176 | logger.debug("Selecting best %s mirror ..", self.distributor_id.capitalize()) 177 | if self.release_is_archived: 178 | logger.info("%s is EOL and has been archived, using %s.", self.release, self.old_releases_url) 179 | return self.old_releases_url 180 | else: 181 | return self.ranked_mirrors[0].mirror_url 182 | 183 | @cached_property 184 | def blacklist(self): 185 | """ 186 | A set of strings with :mod:`fnmatch` patterns (defaults to an empty set). 187 | 188 | When :attr:`available_mirrors` encounters a mirror whose URL matches 189 | one of the patterns in :attr:`blacklist` the mirror will be ignored. 190 | """ 191 | return set() 192 | 193 | @mutable_property 194 | def concurrency(self): 195 | """ 196 | The number of concurrent HTTP connections allowed while ranking mirrors (a number). 197 | 198 | The value of this property defaults to the value computed by 199 | :func:`.get_default_concurrency()`. 200 | """ 201 | return get_default_concurrency() 202 | 203 | @mutable_property(cached=True) 204 | def context(self): 205 | """ 206 | An execution context created using :mod:`executor.contexts`. 207 | 208 | The value of this property defaults to a 209 | :class:`~executor.contexts.LocalContext` object. 210 | """ 211 | return LocalContext() 212 | 213 | @cached_property 214 | def current_mirror(self): 215 | """ 216 | The URL of the main mirror in use in :data:`MAIN_SOURCES_LIST` (a string). 217 | 218 | The :attr:`current_mirror` property's value is computed using 219 | :func:`find_current_mirror()`. 220 | """ 221 | logger.debug("Parsing %s to find current mirror of %s ..", MAIN_SOURCES_LIST, self.context) 222 | return find_current_mirror(self.get_sources_list()) 223 | 224 | @mutable_property 225 | def distribution_codename(self): 226 | """ 227 | The distribution codename (a lowercase string like 'trusty' or 'xenial'). 228 | 229 | The value of this property defaults to the value of the 230 | :attr:`executor.contexts.AbstractContext.distribution_codename` 231 | property which is the right choice 99% of the time. 232 | """ 233 | return self.context.distribution_codename 234 | 235 | @mutable_property 236 | def distributor_id(self): 237 | """ 238 | The distributor ID (a lowercase string like 'debian' or 'ubuntu'). 239 | 240 | The default value of this property is based on the 241 | :attr:`~apt_mirror_updater.releases.Release.distributor_id` property of 242 | :attr:`release` (which in turn is based on :attr:`distribution_codename`). 243 | 244 | Because Debian and Ubuntu code names are unambiguous this means that in 245 | practice you can provide a value for :attr:`distribution_codename` and 246 | omit :attr:`distributor_id` and everything should be fine. 247 | """ 248 | return self.release.distributor_id 249 | 250 | @mutable_property 251 | def max_mirrors(self): 252 | """Limits the number of mirrors to rank (a number, defaults to :data:`MAX_MIRRORS`).""" 253 | return MAX_MIRRORS 254 | 255 | @mutable_property 256 | def old_releases_url(self): 257 | """The URL of the mirror that serves old releases for this :attr:`backend` (a string).""" 258 | return self.backend.OLD_RELEASES_URL 259 | 260 | @cached_property 261 | def ranked_mirrors(self): 262 | """ 263 | A list of :class:`CandidateMirror` objects (ordered from best to worst). 264 | 265 | The value of this property is computed by concurrently testing the 266 | mirrors in :attr:`available_mirrors` for the following details: 267 | 268 | - availability (:attr:`~CandidateMirror.is_available`) 269 | - connection speed (:attr:`~CandidateMirror.bandwidth`) 270 | - update status (:attr:`~CandidateMirror.is_updating`) 271 | 272 | The number of mirrors to test is limited to :attr:`max_mirrors` and you 273 | can change the number of simultaneous HTTP connections allowed by 274 | setting :attr:`concurrency`. 275 | """ 276 | timer = Timer() 277 | # Sort the candidates based on the currently available information 278 | # (and transform the input argument into a list in the process). 279 | mirrors = sorted(self.available_mirrors, key=lambda c: c.sort_key, reverse=True) 280 | # Limit the number of candidates to a reasonable number? 281 | if self.max_mirrors and len(mirrors) > self.max_mirrors: 282 | mirrors = mirrors[:self.max_mirrors] 283 | # Prepare the Release.gpg URLs to fetch. 284 | mapping = dict((c.release_gpg_url, c) for c in mirrors) 285 | num_mirrors = pluralize(len(mapping), "mirror") 286 | logger.info("Checking %s for availability and performance ..", num_mirrors) 287 | # Concurrently fetch the Release.gpg files. 288 | with AutomaticSpinner(label="Checking mirrors"): 289 | for url, data, elapsed_time in fetch_concurrent(mapping.keys(), concurrency=self.concurrency): 290 | candidate = mapping[url] 291 | candidate.release_gpg_contents = data 292 | candidate.release_gpg_latency = elapsed_time 293 | # Concurrently check for Archive-Update-in-Progress markers. 294 | update_mapping = dict((c.archive_update_in_progress_url, c) for c in mirrors if c.is_available) 295 | logger.info("Checking %s for Archive-Update-in-Progress marker ..", 296 | pluralize(len(update_mapping), "mirror")) 297 | with AutomaticSpinner(label="Checking mirrors"): 298 | for url, data, elapsed_time in fetch_concurrent(update_mapping.keys(), concurrency=self.concurrency): 299 | update_mapping[url].is_updating = data is not None 300 | # Sanity check our results. 301 | mirrors = list(mapping.values()) 302 | logger.info("Finished checking %s (took %s).", num_mirrors, timer) 303 | if not any(c.is_available for c in mirrors): 304 | raise Exception("It looks like all %s are unavailable!" % num_mirrors) 305 | if all(c.is_updating for c in mirrors): 306 | logger.warning("It looks like all %s are being updated?!", num_mirrors) 307 | return sorted(mirrors, key=lambda c: c.sort_key, reverse=True) 308 | 309 | @cached_property 310 | def release(self): 311 | """A :class:`.Release` object corresponding to :attr:`distributor_id` and :attr:`distribution_codename`.""" 312 | return coerce_release(self.distribution_codename) 313 | 314 | @cached_property 315 | def release_is_eol(self): 316 | """ 317 | :data:`True` if the release is EOL (end of life), :data:`False` otherwise. 318 | 319 | There are three ways in which the value of this property can be computed: 320 | 321 | - When available, the first of the following EOL dates will be compared 322 | against the current date to determine whether the release is EOL: 323 | 324 | - If the :attr:`backend` module contains a ``get_eol_date()`` 325 | function (only the :mod:`~apt_mirror_updater.backends.debian` 326 | module does at the time of writing) then it is called and if it 327 | returns a number, this number is the EOL date for the release. 328 | 329 | This function was added to enable apt-mirror-updater backend 330 | modules to override the default EOL dates, more specifically to 331 | respect the `Debian LTS`_ release schedule (see also `issue #5`_). 332 | 333 | - Otherwise the :attr:`~apt_mirror_updater.releases.Release.eol_date` 334 | of :attr:`release` is used. 335 | 336 | - As a fall back :func:`validate_mirror()` is used to check whether 337 | :attr:`security_url` results in :data:`MirrorStatus.MAYBE_EOL`. 338 | 339 | .. seealso:: The :attr:`release_is_archived` property 340 | 341 | .. _Debian LTS: https://wiki.debian.org/LTS 342 | .. _issue #5: https://github.com/xolox/python-apt-mirror-updater/issues/5 343 | """ 344 | release_is_eol = None 345 | logger.debug("Checking whether %s is EOL ..", self.release) 346 | # Check if the backend provides custom EOL dates. 347 | if hasattr(self.backend, 'get_eol_date'): 348 | eol_date = self.backend.get_eol_date(self) 349 | if eol_date: 350 | release_is_eol = (time.time() >= eol_date) 351 | source = "custom EOL dates" 352 | # Check if the bundled data contains an applicable EOL date. 353 | if release_is_eol is None and self.release.eol_date: 354 | release_is_eol = self.release.is_eol 355 | source = "known EOL dates" 356 | # Validate the security mirror as a fall back. 357 | if release_is_eol is None: 358 | release_is_eol = (self.validate_mirror(self.security_url) == MirrorStatus.MAYBE_EOL) 359 | source = "security mirror" 360 | logger.debug( 361 | "%s is %s (based on %s).", self.release, 362 | "EOL" if release_is_eol else "supported", source, 363 | ) 364 | return release_is_eol 365 | 366 | @cached_property 367 | def release_is_archived(self): 368 | """ 369 | :data:`True` if the release has been archived, :data:`False` otherwise. 370 | 371 | Archived releases are no longer available on regular package mirrors, 372 | instead they're served from a dedicated old-releases environment. 373 | 374 | This property was added to acknowledge the discrepancy between when a 375 | release hits its EOL date and when it's actually removed from mirrors: 376 | 377 | - The EOL date of Ubuntu 14.04 (Trusty Tahr) is 2019-04-25. 378 | 379 | - At the time of writing it is 2020-04-17 and this release still hasn't 380 | been archived! (a year later) 381 | 382 | For more information please refer to `issue #9`_. 383 | 384 | .. seealso:: The :attr:`release_is_eol` property 385 | 386 | .. _issue #9: https://github.com/xolox/python-apt-mirror-updater/issues/9 387 | """ 388 | if self.release_is_eol: 389 | logger.info("Release %s is EOL, checking if it has been archived ..", self.release) 390 | if self.validate_mirror(self.old_releases_url) == MirrorStatus.AVAILABLE: 391 | logger.info("Confirmed that release %s has been archived.", self.release) 392 | return True 393 | else: 394 | logger.info("No release %s hasn't been archived yet.", self.release) 395 | return False 396 | 397 | @mutable_property 398 | def security_url(self): 399 | """The URL of the mirror that serves security updates for this :attr:`backend` (a string).""" 400 | return self.backend.SECURITY_URL 401 | 402 | @cached_property 403 | def stable_mirror(self): 404 | """ 405 | A mirror URL that is stable for the given execution context (a string). 406 | 407 | The value of this property defaults to the value of 408 | :attr:`current_mirror`, however if the current mirror can't be 409 | determined or is deemed inappropriate by :func:`validate_mirror()` 410 | then :attr:`best_mirror` will be used instead. 411 | 412 | This provides a stable mirror selection algorithm which is useful 413 | because switching mirrors causes ``apt-get update`` to unconditionally 414 | download all package lists and this takes a lot of time so should it be 415 | avoided when unnecessary. 416 | """ 417 | if self.release_is_archived: 418 | logger.debug("%s is EOL and has been archived, falling back to %s.", self.release, self.old_releases_url) 419 | return self.old_releases_url 420 | else: 421 | try: 422 | logger.debug("Trying to select current mirror as stable mirror ..") 423 | return self.current_mirror 424 | except Exception: 425 | logger.debug("Failed to determine current mirror, selecting best mirror instead ..") 426 | return self.best_mirror 427 | 428 | @cached_property 429 | def validated_mirrors(self): 430 | """Dictionary of validated mirrors (used by :func:`validate_mirror()`).""" 431 | return {} 432 | 433 | def change_mirror(self, new_mirror=None, update=True): 434 | """ 435 | Change the main mirror in use in :data:`MAIN_SOURCES_LIST`. 436 | 437 | :param new_mirror: The URL of the new mirror (a string, defaults to 438 | :attr:`best_mirror`). 439 | :param update: Whether an ``apt-get update`` should be run after 440 | changing the mirror (a boolean, defaults to 441 | :data:`True`). 442 | """ 443 | timer = Timer() 444 | # Default to the best available mirror. 445 | if new_mirror: 446 | logger.info("Changing mirror of %s to %s ..", self.context, new_mirror) 447 | else: 448 | logger.info("Changing mirror of %s to best available mirror ..", self.context) 449 | new_mirror = self.best_mirror 450 | logger.info("Selected mirror: %s", new_mirror) 451 | # Parse /etc/apt/sources.list to replace the old mirror with the new one. 452 | sources_list = self.get_sources_list() 453 | mirrors_to_replace = [normalize_mirror_url(find_current_mirror(sources_list))] 454 | if self.release_is_archived: 455 | # When a release is archived the security updates mirrors stop 456 | # serving that release as well, so we need to remove them. 457 | logger.debug("Replacing %s URLs as well ..", self.security_url) 458 | mirrors_to_replace.append(normalize_mirror_url(self.security_url)) 459 | else: 460 | logger.debug("Not replacing %s URLs.", self.security_url) 461 | lines = sources_list.splitlines() 462 | for i, line in enumerate(lines): 463 | # The first token should be `deb' or `deb-src', the second token is 464 | # the mirror's URL, the third token is the `distribution' and any 465 | # further tokens are `components'. 466 | tokens = line.split() 467 | if (len(tokens) >= 4 and 468 | tokens[0] in ('deb', 'deb-src') and 469 | normalize_mirror_url(tokens[1]) in mirrors_to_replace): 470 | tokens[1] = new_mirror 471 | lines[i] = u' '.join(tokens) 472 | # Install the modified package resource list. 473 | self.install_sources_list(u'\n'.join(lines)) 474 | # Clear (relevant) cached properties. 475 | del self.current_mirror 476 | # Make sure previous package lists are removed. 477 | self.clear_package_lists() 478 | # Make sure the package lists are up to date. 479 | if update: 480 | self.smart_update(switch_mirrors=False) 481 | logger.info("Finished changing mirror of %s in %s.", self.context, timer) 482 | 483 | def clear_package_lists(self): 484 | """Clear the package list cache by removing the files under ``/var/lib/apt/lists``.""" 485 | timer = Timer() 486 | logger.info("Clearing package list cache of %s ..", self.context) 487 | self.context.execute( 488 | # We use an ugly but necessary find | xargs pipeline here because 489 | # find's -delete option implies -depth which negates -prune. Sigh. 490 | 'find /var/lib/apt/lists -type f -name lock -prune -o -type f -print0 | xargs -0 rm -f', 491 | sudo=True, 492 | ) 493 | logger.info("Successfully cleared package list cache of %s in %s.", self.context, timer) 494 | 495 | def create_chroot(self, directory, arch=None): 496 | """ 497 | Bootstrap a basic Debian or Ubuntu system using debootstrap_. 498 | 499 | :param directory: The pathname of the target directory (a string). 500 | :param arch: The target architecture (a string or :data:`None`). 501 | :returns: A :class:`~executor.contexts.ChangeRootContext` object. 502 | 503 | If `directory` already exists and isn't empty then it is assumed that 504 | the chroot has already been created and debootstrap_ won't be run. 505 | Before this method returns it changes :attr:`context` to the chroot. 506 | 507 | .. _debootstrap: https://manpages.debian.org/debootstrap 508 | """ 509 | logger.debug("Checking if chroot already exists (%s) ..", directory) 510 | if self.context.exists(directory) and self.context.list_entries(directory): 511 | logger.debug("The chroot already exists, skipping initialization.") 512 | first_run = False 513 | else: 514 | # Ensure the `debootstrap' program is installed. 515 | if not self.context.find_program('debootstrap'): 516 | logger.info("Installing `debootstrap' program ..") 517 | self.context.execute('apt-get', 'install', '--yes', 'debootstrap', sudo=True) 518 | # Use the `debootstrap' program to create the chroot. 519 | timer = Timer() 520 | logger.info("Creating %s chroot in %s ..", self.release, directory) 521 | debootstrap_command = ['debootstrap'] 522 | if arch: 523 | debootstrap_command.append('--arch=%s' % arch) 524 | debootstrap_command.append('--keyring=%s' % self.release.keyring_file) 525 | debootstrap_command.append(self.release.upstream_series) 526 | debootstrap_command.append(directory) 527 | debootstrap_command.append(self.best_mirror) 528 | self.context.execute(*debootstrap_command, sudo=True) 529 | logger.info("Took %s to create %s chroot.", timer, self.release) 530 | first_run = True 531 | # Switch the execution context to the chroot and reset the locale (to 532 | # avoid locale warnings emitted by post-installation scripts run by 533 | # `apt-get install'). 534 | self.context = ChangeRootContext( 535 | chroot=directory, 536 | environment=dict(LC_ALL='C'), 537 | ) 538 | # Clear the values of cached properties that can be 539 | # invalidated by switching the execution context. 540 | del self.current_mirror 541 | del self.stable_mirror 542 | # The following initialization only needs to happen on the first 543 | # run, but it requires the context to be set to the chroot. 544 | if first_run: 545 | # Make sure the `lsb_release' program is available. It is 546 | # my experience that this package cannot be installed using 547 | # `debootstrap --include=lsb-release', it specifically 548 | # needs to be installed afterwards. 549 | self.context.execute('apt-get', 'install', '--yes', 'lsb-release', sudo=True) 550 | # Cleanup downloaded *.deb archives. 551 | self.context.execute('apt-get', 'clean', sudo=True) 552 | # Install a suitable /etc/apt/sources.list file. The logic behind 553 | # generate_sources_list() depends on the `lsb_release' program. 554 | self.install_sources_list(self.generate_sources_list()) 555 | # Make sure the package lists are up to date. 556 | self.smart_update() 557 | return self.context 558 | 559 | def dumb_update(self, *args): 560 | """ 561 | Update the system's package lists (by running ``apt-get update``). 562 | 563 | :param args: Command line arguments to ``apt-get update`` (zero or more strings). 564 | 565 | The :func:`dumb_update()` method doesn't do any error handling or 566 | retrying, if that's what you're looking for then you need 567 | :func:`smart_update()` instead. 568 | """ 569 | timer = Timer() 570 | logger.info("Updating package lists of %s ..", self.context) 571 | self.context.execute('apt-get', 'update', *args, sudo=True) 572 | logger.info("Finished updating package lists of %s in %s.", self.context, timer) 573 | 574 | def generate_sources_list(self, **options): 575 | """ 576 | Generate the contents of ``/etc/apt/sources.list``. 577 | 578 | If no `mirror_url` keyword argument is given then :attr:`stable_mirror` 579 | is used as a default. 580 | 581 | Please refer to the documentation of the Debian 582 | (:func:`apt_mirror_updater.backends.debian.generate_sources_list()`) 583 | and Ubuntu (:func:`apt_mirror_updater.backends.ubuntu.generate_sources_list()`) 584 | backend implementations of this method for details on argument handling 585 | and the return value. 586 | """ 587 | if options.get('mirror_url') is None: 588 | options['mirror_url'] = self.stable_mirror 589 | options.setdefault('codename', self.distribution_codename) 590 | return self.backend.generate_sources_list(**options) 591 | 592 | def get_sources_list(self): 593 | """ 594 | Get the contents of :data:`MAIN_SOURCES_LIST`. 595 | 596 | :returns: A Unicode string. 597 | 598 | This code currently assumes that the ``sources.list`` file is encoded 599 | using :data:`SOURCES_LIST_ENCODING`. I'm not actually sure if this is 600 | correct because I haven't been able to find a formal specification! 601 | Feedback is welcome :-). 602 | """ 603 | contents = self.context.read_file(MAIN_SOURCES_LIST) 604 | return contents.decode(SOURCES_LIST_ENCODING) 605 | 606 | def ignore_mirror(self, pattern): 607 | """ 608 | Add a pattern to the mirror discovery :attr:`blacklist`. 609 | 610 | :param pattern: A shell pattern (containing wild cards like ``?`` and 611 | ``*``) that is matched against the full URL of each 612 | mirror. 613 | 614 | When a pattern is added to the blacklist any previously cached values 615 | of :attr:`available_mirrors`, :attr:`best_mirror`, :attr:`ranked_mirrors` 616 | and :attr:`stable_mirror` are cleared. This makes sure that mirrors 617 | blacklisted after mirror discovery has already run are ignored. 618 | """ 619 | # Update the blacklist. 620 | logger.info("Adding pattern to mirror discovery blacklist: %s", pattern) 621 | self.blacklist.add(pattern) 622 | # Clear (relevant) cached properties. 623 | del self.available_mirrors 624 | del self.best_mirror 625 | del self.ranked_mirrors 626 | del self.stable_mirror 627 | 628 | def install_sources_list(self, contents): 629 | """ 630 | Install a new ``/etc/apt/sources.list`` file. 631 | 632 | :param contents: The new contents of the sources list (a Unicode 633 | string). You can generate a suitable value using 634 | the :func:`generate_sources_list()` method. 635 | """ 636 | if isinstance(contents, text_type): 637 | contents = contents.encode(SOURCES_LIST_ENCODING) 638 | logger.info("Installing new %s ..", MAIN_SOURCES_LIST) 639 | with self.context: 640 | # Write the sources.list contents to a temporary file. We make sure 641 | # the file always ends in a newline to adhere to UNIX conventions. 642 | temporary_file = '/tmp/apt-mirror-updater-sources-list-%i.txt' % os.getpid() 643 | self.context.write_file(temporary_file, b'%s\n' % contents.rstrip()) 644 | # Make sure the temporary file is cleaned up when we're done with it. 645 | self.context.cleanup('rm', '--force', temporary_file) 646 | # Make a backup copy of /etc/apt/sources.list in case shit hits the fan? 647 | if self.context.exists(MAIN_SOURCES_LIST): 648 | backup_copy = '%s.save.%i' % (MAIN_SOURCES_LIST, time.time()) 649 | logger.info("Backing up contents of %s to %s ..", MAIN_SOURCES_LIST, backup_copy) 650 | self.context.execute('cp', MAIN_SOURCES_LIST, backup_copy, sudo=True) 651 | # Move the temporary file into place without changing ownership and permissions. 652 | self.context.execute( 653 | 'cp', '--no-preserve=mode,ownership', 654 | temporary_file, MAIN_SOURCES_LIST, 655 | sudo=True, 656 | ) 657 | 658 | def smart_update(self, *args, **kw): 659 | """ 660 | Update the system's package lists (switching mirrors if necessary). 661 | 662 | :param args: Command line arguments to ``apt-get update`` (zero or more strings). 663 | :param max_attempts: The maximum number of attempts at successfully 664 | updating the system's package lists (an integer, 665 | defaults to 10). 666 | :param switch_mirrors: :data:`True` if we're allowed to switch mirrors 667 | on 'hash sum mismatch' errors, :data:`False` 668 | otherwise. 669 | :raises: If updating of the package lists fails 10 consecutive times 670 | (`max_attempts`) an exception is raised. 671 | 672 | While :func:`dumb_update()` simply runs ``apt-get update`` the 673 | :func:`smart_update()` function works quite differently: 674 | 675 | - First the system's package lists are updated using 676 | :func:`dumb_update()`. If this is successful we're done. 677 | - If the update fails we check the command's output for the phrase 678 | 'hash sum mismatch'. If we find this phrase we assume that the 679 | current mirror is faulty and switch to another one. 680 | - Failing ``apt-get update`` runs are retried up to `max_attempts`. 681 | """ 682 | backoff_time = 10 683 | max_attempts = kw.get('max_attempts', 10) 684 | switch_mirrors = kw.get('switch_mirrors', True) 685 | for i in range(1, max_attempts + 1): 686 | with CaptureOutput() as session: 687 | try: 688 | self.dumb_update(*args) 689 | return 690 | except Exception: 691 | if i < max_attempts: 692 | output = session.get_text() 693 | # Check for EOL releases. This somewhat peculiar way of 694 | # checking is meant to ignore 404 responses from 695 | # `secondary package mirrors' like PPAs. If the output 696 | # of `apt-get update' implies that the release is EOL 697 | # we need to verify our assumption. 698 | if any(self.current_mirror in line and u'404' in line.split() for line in output.splitlines()): 699 | logger.warning("%s may be EOL, checking ..", self.release) 700 | if self.release_is_archived: 701 | if switch_mirrors: 702 | logger.warning( 703 | "Switching to old releases mirror because %s is EOL and has been archived ..", 704 | self.release, 705 | ) 706 | self.change_mirror(self.old_releases_url, update=False) 707 | continue 708 | else: 709 | raise Exception(compact(""" 710 | Failed to update package lists because it looks like 711 | the current release (%s) is end of life but I'm not 712 | allowed to switch mirrors! (there's no point in 713 | retrying so I'm not going to) 714 | """, self.distribution_codename)) 715 | # Check for `hash sum mismatch' errors. 716 | if switch_mirrors and u'hash sum mismatch' in output.lower(): 717 | logger.warning("Detected 'hash sum mismatch' failure, switching to other mirror ..") 718 | self.ignore_mirror(self.current_mirror) 719 | self.change_mirror(update=False) 720 | else: 721 | logger.warning("Retrying after `apt-get update' failed (%i/%i) ..", i, max_attempts) 722 | # Deal with unidentified (but hopefully transient) failures by retrying but backing off 723 | # to give the environment (network connection, mirror state, etc.) time to stabilize. 724 | logger.info("Sleeping for %s before retrying update ..", format_timespan(backoff_time)) 725 | time.sleep(backoff_time) 726 | if backoff_time <= 120: 727 | backoff_time *= 2 728 | else: 729 | backoff_time += backoff_time / 3 730 | raise Exception("Failed to update package lists %i consecutive times?!" % max_attempts) 731 | 732 | def validate_mirror(self, mirror_url): 733 | """ 734 | Make sure a mirror serves :attr:`distribution_codename`. 735 | 736 | :param mirror_url: The base URL of the mirror (a string). 737 | :returns: One of the values in the :class:`MirrorStatus` enumeration. 738 | 739 | This method assumes that :attr:`old_releases_url` is always valid. 740 | """ 741 | mirror_url = normalize_mirror_url(mirror_url) 742 | key = (mirror_url, self.distribution_codename) 743 | value = self.validated_mirrors.get(key) 744 | if value is None: 745 | logger.info("Checking if %s is available on %s ..", self.release, mirror_url) 746 | # Try to download the Release.gpg file, in the assumption that 747 | # this file should always exist and is more or less guaranteed 748 | # to be relatively small. 749 | try: 750 | mirror = CandidateMirror(mirror_url=mirror_url, updater=self) 751 | mirror.release_gpg_contents = fetch_url(mirror.release_gpg_url, retry=False) 752 | value = (MirrorStatus.AVAILABLE if mirror.is_available else MirrorStatus.UNAVAILABLE) 753 | except NotFoundError: 754 | # When the mirror is serving 404 responses it can be an 755 | # indication that the release has gone end of life. In any 756 | # case the mirror is unavailable. 757 | value = MirrorStatus.MAYBE_EOL 758 | except Exception: 759 | # When we get an unspecified error that is not a 404 760 | # response we conclude that the mirror is unavailable. 761 | value = MirrorStatus.UNAVAILABLE 762 | # Cache the mirror status that we just determined. 763 | self.validated_mirrors[key] = value 764 | return value 765 | 766 | 767 | class CandidateMirror(PropertyManager): 768 | 769 | """A candidate mirror groups a mirror URL with its availability and performance metrics.""" 770 | 771 | @mutable_property 772 | def bandwidth(self): 773 | """ 774 | The bytes per second achieved while fetching :attr:`release_gpg_url` (a number or :data:`None`). 775 | 776 | The value of this property is computed based on the values of 777 | :attr:`release_gpg_contents` and :attr:`release_gpg_latency`. 778 | """ 779 | if self.release_gpg_contents and self.release_gpg_latency: 780 | return len(self.release_gpg_contents) / self.release_gpg_latency 781 | 782 | @lazy_property 783 | def archive_update_in_progress_url(self): 784 | """ 785 | The URL of the file whose existence indicates that the mirror is being updated (a string). 786 | 787 | The value of this property is computed based on the value of 788 | :attr:`mirror_url`. 789 | """ 790 | return '%s/Archive-Update-in-Progress-%s' % ( 791 | self.mirror_url, urlparse(self.mirror_url).netloc, 792 | ) 793 | 794 | @key_property 795 | def mirror_url(self): 796 | """The base URL of the mirror (a string).""" 797 | 798 | @mirror_url.setter 799 | def mirror_url(self, value): 800 | """Normalize the mirror URL when set.""" 801 | set_property(self, 'mirror_url', normalize_mirror_url(value)) 802 | 803 | @mutable_property 804 | def is_available(self): 805 | """ 806 | :data:`True` if :attr:`release_gpg_contents` contains the expected header, :data:`False` otherwise. 807 | 808 | The value of this property is computed by checking whether 809 | :attr:`release_gpg_contents` contains the expected ``BEGIN PGP 810 | SIGNATURE`` header. This may seem like a rather obscure way of 811 | validating a mirror, but it was specifically chosen to detect 812 | all sorts of ways in which mirrors can be broken: 813 | 814 | - Webservers with a broken configuration that return an error page for 815 | all URLs. 816 | 817 | - Mirrors whose domain name registration has expired, where the domain 818 | is now being squatted and returns HTTP 200 OK responses for all URLs 819 | (whether they "exist" or not). 820 | """ 821 | value = False 822 | if self.release_gpg_contents: 823 | value = b'BEGIN PGP SIGNATURE' in self.release_gpg_contents 824 | if not value: 825 | logger.debug("Missing GPG header, considering mirror unavailable (%s).", self.release_gpg_url) 826 | set_property(self, 'is_available', value) 827 | return value 828 | 829 | @mutable_property 830 | def is_updating(self): 831 | """:data:`True` if the mirror is being updated, :data:`False` otherwise.""" 832 | 833 | @mutable_property 834 | def last_updated(self): 835 | """The time in seconds since the most recent mirror update (a number or :data:`None`).""" 836 | 837 | @mutable_property 838 | def release_gpg_contents(self): 839 | """ 840 | The contents downloaded from :attr:`release_gpg_url` (a string or :data:`None`). 841 | 842 | By downloading the file available at :attr:`release_gpg_url` and 843 | setting :attr:`release_gpg_contents` and :attr:`release_gpg_latency` 844 | you enable the :attr:`bandwidth` and :attr:`is_available` properties to 845 | be computed. 846 | """ 847 | 848 | @mutable_property 849 | def release_gpg_latency(self): 850 | """ 851 | The time it took to download :attr:`release_gpg_url` (a number or :data:`None`). 852 | 853 | By downloading the file available at :attr:`release_gpg_url` and 854 | setting :attr:`release_gpg_contents` and :attr:`release_gpg_latency` 855 | you enable the :attr:`bandwidth` and :attr:`is_available` properties to 856 | be computed. 857 | """ 858 | 859 | @mutable_property 860 | def release_gpg_url(self): 861 | """ 862 | The URL of the ``Release.gpg`` file that will be used to test the mirror (a string or :data:`None`). 863 | 864 | The value of this property is based on :attr:`mirror_url` and the 865 | :attr:`~AptMirrorUpdater.distribution_codename` property of the 866 | :attr:`updater` object. 867 | """ 868 | if self.updater and self.updater.release.upstream_series: 869 | return '%s/dists/%s/Release.gpg' % ( 870 | self.mirror_url, self.updater.release.upstream_series, 871 | ) 872 | 873 | @mutable_property 874 | def sort_key(self): 875 | """ 876 | A tuple that can be used to sort the mirror by its availability/performance metrics. 877 | 878 | The tuple created by this property contains four numbers in the following order: 879 | 880 | 1. The number 1 when :attr:`is_available` is :data:`True` or 881 | the number 0 when :attr:`is_available` is :data:`False` 882 | (because most importantly a mirror must be available). 883 | 2. The number 0 when :attr:`is_updating` is :data:`True` or 884 | the number 1 when :attr:`is_updating` is :data:`False` 885 | (because being updated at this very moment is *bad*). 886 | 3. The negated value of :attr:`last_updated` (because the 887 | lower :attr:`last_updated` is, the better). If :attr:`last_updated` 888 | is :data:`None` then :data:`LAST_UPDATED_DEFAULT` is used instead. 889 | 4. The value of :attr:`bandwidth` (because the higher 890 | :attr:`bandwidth` is, the better). 891 | 892 | By sorting :class:`CandidateMirror` objects on these tuples in 893 | ascending order, the last mirror in the sorted results will be the 894 | "most suitable mirror" (given the available information). 895 | """ 896 | return (int(self.is_available), 897 | int(not self.is_updating), 898 | -(self.last_updated if self.last_updated is not None else LAST_UPDATED_DEFAULT), 899 | self.bandwidth or 0) 900 | 901 | @mutable_property(repr=False) 902 | def updater(self): 903 | """A reference to the :class:`AptMirrorUpdater` object that created the candidate.""" 904 | 905 | 906 | class MirrorStatus(Enum): 907 | 908 | """Enumeration for mirror statuses determined by :func:`AptMirrorUpdater.validate_mirror()`.""" 909 | 910 | AVAILABLE = 1 911 | """The mirror is accepting connections and serving the expected content.""" 912 | 913 | MAYBE_EOL = 2 914 | """The mirror is serving HTTP 404 "Not Found" responses instead of the expected content.""" 915 | 916 | UNAVAILABLE = 3 917 | """The mirror is not accepting connections or not serving the expected content.""" 918 | 919 | 920 | def find_current_mirror(sources_list): 921 | """ 922 | Find the URL of the main mirror that is currently in use by ``apt-get``. 923 | 924 | :param sources_list: The contents of apt's package resource list, e.g. the 925 | contents of :data:`MAIN_SOURCES_LIST` (a string). 926 | :returns: The URL of the main mirror in use (a string). 927 | :raises: If the main mirror can't be determined 928 | :exc:`~exceptions.EnvironmentError` is raised. 929 | 930 | The main mirror is determined by looking for the first ``deb`` or 931 | ``deb-src`` directive in apt's package resource list whose URL uses the 932 | HTTP or FTP scheme and whose components contain ``main``. 933 | """ 934 | for line in sources_list.splitlines(): 935 | # The first token should be `deb' or `deb-src', the second token is 936 | # the mirror's URL, the third token is the `distribution' and any 937 | # further tokens are `components'. 938 | tokens = line.split() 939 | if (len(tokens) >= 4 and 940 | tokens[0] in ('deb', 'deb-src') and 941 | tokens[1].startswith(('http://', 'ftp://')) and 942 | 'main' in tokens[3:]): 943 | return tokens[1] 944 | raise EnvironmentError("Failed to determine current mirror in apt's package resource list!") 945 | 946 | 947 | def mirrors_are_equal(a, b): 948 | """ 949 | Check whether two mirror URLS are equal. 950 | 951 | :param a: The first mirror URL (a string). 952 | :param b: The second mirror URL (a string). 953 | :returns: :data:`True` if the mirror URLs are equal, 954 | :data:`False` otherwise. 955 | """ 956 | return normalize_mirror_url(a) == normalize_mirror_url(b) 957 | 958 | 959 | def normalize_mirror_url(url): 960 | """ 961 | Normalize a mirror URL so it can be compared using string equality comparison. 962 | 963 | :param url: The mirror URL to normalize (a string). 964 | :returns: The normalized mirror URL (a string). 965 | """ 966 | return url.rstrip('/') 967 | -------------------------------------------------------------------------------- /apt_mirror_updater/backends/__init__.py: -------------------------------------------------------------------------------- 1 | """Namespace for platform specific mirror discovery in :mod:`apt_mirror_updater`.""" 2 | -------------------------------------------------------------------------------- /apt_mirror_updater/backends/debian.py: -------------------------------------------------------------------------------- 1 | # Automated, robust apt-get mirror selection for Debian and Ubuntu. 2 | # 3 | # Author: Peter Odding 4 | # Last Change: April 15, 2020 5 | # URL: https://apt-mirror-updater.readthedocs.io 6 | 7 | """ 8 | Discovery of Debian package archive mirrors. 9 | 10 | Here are references to some of the material that I've needed to consult while 11 | working on this module: 12 | 13 | - `Notes about sources.list on the Debian wiki `_ 14 | - `The Debian backports webpages `_ 15 | - `Documentation about the "proposed-updates" mechanism `_ 16 | """ 17 | 18 | # Standard library modules. 19 | import logging 20 | 21 | # External dependencies. 22 | from bs4 import BeautifulSoup 23 | from humanfriendly import Timer 24 | from humanfriendly.text import format, pluralize 25 | 26 | # Modules included in our package. 27 | from apt_mirror_updater import CandidateMirror, mirrors_are_equal 28 | from apt_mirror_updater.http import fetch_url 29 | 30 | LTS_ARCHITECTURES = ('i386', 'amd64', 'armel', 'armhf') 31 | """The names of the architectures supported by the Debian LTS team (a tuple of strings).""" 32 | 33 | LTS_RELEASES = { 34 | 'jessie': 1593468000, # 2020-06-30 35 | 'stretch': 1656540000, # 2022-06-30 36 | } 37 | """ 38 | A dictionary with `Debian LTS`_ releases and their EOL dates. 39 | 40 | This is needed because distro-info-data_ doesn't contain information 41 | about Debian LTS releases but nevertheless ``archive.debian.org`` 42 | doesn't adopt a release until the LTS status expires (this was 43 | originally reported in `issue #5`_). 44 | 45 | .. _Debian LTS: https://wiki.debian.org/LTS 46 | .. _issue #5: https://github.com/xolox/python-apt-mirror-updater/issues/5 47 | """ 48 | 49 | MIRRORS_URL = 'https://www.debian.org/mirror/list' 50 | """The URL of the HTML page listing all primary Debian mirrors (a string).""" 51 | 52 | SECURITY_URL = 'http://security.debian.org/' 53 | """The base URL of the Debian mirror with security updates (a string).""" 54 | 55 | OLD_RELEASES_URL = 'http://archive.debian.org/debian-archive/debian/' 56 | """The URL where EOL (end of life) Debian releases are hosted (a string).""" 57 | 58 | DEFAULT_SUITES = 'release', 'security', 'updates' 59 | """A tuple of strings with the Debian suites that are enabled by default.""" 60 | 61 | VALID_COMPONENTS = 'main', 'contrib', 'non-free' 62 | """A tuple of strings with the names of the components available in the Debian package repositories.""" 63 | 64 | VALID_SUITES = 'release', 'security', 'updates', 'backports', 'proposed-updates' 65 | """A tuple of strings with the names of the suites available in the Debian package repositories.""" 66 | 67 | # Initialize a logger for this module. 68 | logger = logging.getLogger(__name__) 69 | 70 | 71 | def discover_mirrors(): 72 | """ 73 | Discover available Debian mirrors by querying :data:`MIRRORS_URL`. 74 | 75 | :returns: A set of :class:`.CandidateMirror` objects that have their 76 | :attr:`~.CandidateMirror.mirror_url` property set. 77 | :raises: If no mirrors are discovered an exception is raised. 78 | 79 | An example run: 80 | 81 | >>> from apt_mirror_updater.backends.debian import discover_mirrors 82 | >>> from pprint import pprint 83 | >>> pprint(discover_mirrors()) 84 | set([CandidateMirror(mirror_url='http://ftp.at.debian.org/debian/'), 85 | CandidateMirror(mirror_url='http://ftp.au.debian.org/debian/'), 86 | CandidateMirror(mirror_url='http://ftp.be.debian.org/debian/'), 87 | CandidateMirror(mirror_url='http://ftp.bg.debian.org/debian/'), 88 | CandidateMirror(mirror_url='http://ftp.br.debian.org/debian/'), 89 | CandidateMirror(mirror_url='http://ftp.by.debian.org/debian/'), 90 | CandidateMirror(mirror_url='http://ftp.ca.debian.org/debian/'), 91 | CandidateMirror(mirror_url='http://ftp.ch.debian.org/debian/'), 92 | CandidateMirror(mirror_url='http://ftp.cn.debian.org/debian/'), 93 | CandidateMirror(mirror_url='http://ftp.cz.debian.org/debian/'), 94 | ...]) 95 | """ 96 | timer = Timer() 97 | logger.info("Discovering Debian mirrors at %s ..", MIRRORS_URL) 98 | data = fetch_url(MIRRORS_URL, retry=True) 99 | soup = BeautifulSoup(data, 'html.parser') 100 | tables = soup.findAll('table') 101 | if not tables: 102 | raise Exception("Failed to locate element in Debian mirror page! (%s)" % MIRRORS_URL) 103 | mirrors = set(CandidateMirror(mirror_url=a['href']) for a in tables[0].findAll('a', href=True)) 104 | if not mirrors: 105 | raise Exception("Failed to discover any Debian mirrors! (using %s)" % MIRRORS_URL) 106 | logger.info("Discovered %s in %s.", pluralize(len(mirrors), "Debian mirror"), timer) 107 | return mirrors 108 | 109 | 110 | def generate_sources_list(mirror_url, codename, 111 | suites=DEFAULT_SUITES, 112 | components=VALID_COMPONENTS, 113 | enable_sources=False): 114 | """ 115 | Generate the contents of ``/etc/apt/sources.list`` for a Debian system. 116 | 117 | :param mirror_url: The base URL of the mirror (a string). 118 | :param codename: The codename of a Debian release (a string like 'wheezy' 119 | or 'jessie') or a Debian release class (a string like 120 | 'stable', 'testing', etc). 121 | :param suites: An iterable of strings (defaults to 122 | :data:`DEFAULT_SUITES`, refer to 123 | :data:`VALID_SUITES` for details). 124 | :param components: An iterable of strings (refer to 125 | :data:`VALID_COMPONENTS` for details). 126 | :param enable_sources: :data:`True` to include ``deb-src`` entries, 127 | :data:`False` to omit them. 128 | :returns: The suggested contents of ``/etc/apt/sources.list`` (a string). 129 | """ 130 | # Validate the suites. 131 | invalid_suites = [s for s in suites if s not in VALID_SUITES] 132 | if invalid_suites: 133 | msg = "Invalid Debian suite(s) given! (%s)" 134 | raise ValueError(msg % invalid_suites) 135 | # Validate the components. 136 | invalid_components = [c for c in components if c not in VALID_COMPONENTS] 137 | if invalid_components: 138 | msg = "Invalid Debian component(s) given! (%s)" 139 | raise ValueError(msg % invalid_components) 140 | # Generate the /etc/apt/sources.list file contents. 141 | lines = [] 142 | directives = ('deb', 'deb-src') if enable_sources else ('deb',) 143 | for suite in suites: 144 | for directive in directives: 145 | lines.append(format( 146 | '{directive} {mirror} {suite} {components}', directive=directive, 147 | mirror=(OLD_RELEASES_URL if mirrors_are_equal(mirror_url, OLD_RELEASES_URL) 148 | else (SECURITY_URL if suite == 'security' else mirror_url)), 149 | suite=(codename if suite == 'release' else ( 150 | ('%s/updates' % codename if suite == 'security' 151 | else codename + '-' + suite))), 152 | components=' '.join(components), 153 | )) 154 | return '\n'.join(lines) 155 | 156 | 157 | def get_eol_date(updater): 158 | """ 159 | Override the EOL date for `Debian LTS`_ releases. 160 | 161 | :param updater: The :class:`~apt_mirror_updater.AptMirrorUpdater` object. 162 | :returns: The overridden EOL date (a number) or :data:`None`. 163 | """ 164 | if updater.architecture in LTS_ARCHITECTURES: 165 | return LTS_RELEASES.get(updater.distribution_codename) 166 | -------------------------------------------------------------------------------- /apt_mirror_updater/backends/elementary.py: -------------------------------------------------------------------------------- 1 | # Automated, robust apt-get mirror selection for Debian and Ubuntu. 2 | # 3 | # Author: Peter Odding 4 | # Last Change: April 16, 2020 5 | # URL: https://apt-mirror-updater.readthedocs.io 6 | 7 | """ 8 | Support for `Elementary OS`_ package archive mirror selection. 9 | 10 | Elementary OS is based on Ubuntu LTS releases and as such this module is a very 11 | thin wrapper for the :mod:`apt_mirror_updater.backends.ubuntu` module. 12 | 13 | .. _Elementary OS: https://en.wikipedia.org/wiki/Elementary_OS 14 | """ 15 | 16 | # Standard library modules. 17 | import datetime 18 | import decimal 19 | 20 | # Modules included in our package. 21 | from apt_mirror_updater.backends import ubuntu 22 | from apt_mirror_updater.releases import Release 23 | 24 | # Public identifiers that require documentation. 25 | __all__ = ( 26 | 'KNOWN_RELEASES', 27 | 'OLD_RELEASES_URL', 28 | 'SECURITY_URL', 29 | 'discover_mirrors', 30 | 'generate_sources_list', 31 | ) 32 | 33 | OLD_RELEASES_URL = ubuntu.OLD_RELEASES_URL 34 | """Alias for :attr:`apt_mirror_updater.backends.ubuntu.OLD_RELEASES_URL`.""" 35 | 36 | SECURITY_URL = ubuntu.SECURITY_URL 37 | """Alias for :attr:`apt_mirror_updater.backends.ubuntu.SECURITY_URL`.""" 38 | 39 | discover_mirrors = ubuntu.discover_mirrors 40 | """Alias for :func:`apt_mirror_updater.backends.ubuntu.discover_mirrors`.""" 41 | 42 | generate_sources_list = ubuntu.generate_sources_list 43 | """Alias for :func:`apt_mirror_updater.backends.ubuntu.generate_sources_list`.""" 44 | 45 | KNOWN_RELEASES = [ 46 | Release( 47 | codename='Jupiter', 48 | created_date=datetime.date(2011, 3, 31), 49 | distributor_id='elementary', 50 | upstream_distributor_id='ubuntu', 51 | upstream_series='maverick', 52 | upstream_version=decimal.Decimal('10.10'), 53 | is_lts=False, 54 | series='jupiter', 55 | version=decimal.Decimal('0.1'), 56 | ), 57 | Release( 58 | codename='Luna', 59 | created_date=datetime.date(2013, 8, 10), 60 | distributor_id='elementary', 61 | upstream_distributor_id='ubuntu', 62 | upstream_series='precise', 63 | upstream_version=decimal.Decimal('12.04'), 64 | is_lts=False, 65 | series='luna', 66 | version=decimal.Decimal('0.2'), 67 | ), 68 | Release( 69 | codename='Freya', 70 | created_date=datetime.date(2015, 4, 11), 71 | distributor_id='elementary', 72 | upstream_distributor_id='ubuntu', 73 | upstream_series='trusty', 74 | upstream_version=decimal.Decimal('14.04'), 75 | is_lts=False, 76 | series='freya', 77 | version=decimal.Decimal('0.3'), 78 | ), 79 | Release( 80 | codename='Loki', 81 | created_date=datetime.date(2016, 9, 9), 82 | distributor_id='elementary', 83 | upstream_distributor_id='ubuntu', 84 | upstream_series='xenial', 85 | upstream_version=decimal.Decimal('16.04'), 86 | is_lts=False, 87 | series='loki', 88 | version=decimal.Decimal('0.4'), 89 | ), 90 | Release( 91 | codename='Juno', 92 | created_date=datetime.date(2018, 10, 16), 93 | distributor_id='elementary', 94 | upstream_distributor_id='ubuntu', 95 | upstream_series='bionic', 96 | upstream_version=decimal.Decimal('18.04'), 97 | is_lts=False, 98 | series='juno', 99 | version=decimal.Decimal('5.0'), 100 | ), 101 | Release( 102 | codename='Hera', 103 | created_date=datetime.date(2019, 12, 3), 104 | distributor_id='elementary', 105 | upstream_distributor_id='ubuntu', 106 | upstream_series='bionic', 107 | upstream_version=decimal.Decimal('18.04'), 108 | is_lts=False, 109 | series='hera', 110 | version=decimal.Decimal('5.1'), 111 | ), 112 | ] 113 | """ 114 | List of :class:`.Release` objects corresponding to known elementary OS 115 | releases based on the summary table on the following web page: 116 | https://en.wikipedia.org/wiki/Elementary_OS#Summary_table 117 | """ 118 | -------------------------------------------------------------------------------- /apt_mirror_updater/backends/ubuntu.py: -------------------------------------------------------------------------------- 1 | # Automated, robust apt-get mirror selection for Debian and Ubuntu. 2 | # 3 | # Author: Peter Odding 4 | # Last Change: April 15, 2020 5 | # URL: https://apt-mirror-updater.readthedocs.io 6 | 7 | """Discovery of Ubuntu package archive mirrors.""" 8 | 9 | # Standard library modules. 10 | import logging 11 | 12 | # External dependencies. 13 | from bs4 import BeautifulSoup, UnicodeDammit 14 | from humanfriendly import Timer 15 | from humanfriendly.text import format, pluralize 16 | 17 | # Modules included in our package. 18 | from apt_mirror_updater import CandidateMirror, mirrors_are_equal 19 | from apt_mirror_updater.http import fetch_url 20 | 21 | MIRRORS_URL = 'https://launchpad.net/ubuntu/+archivemirrors' 22 | """The URL of the HTML page listing official Ubuntu mirrors (a string).""" 23 | 24 | MIRROR_SELECTION_URL = 'http://mirrors.ubuntu.com/mirrors.txt' 25 | """The URL of a plain text listing of "geographically suitable" mirror URLs (a string).""" 26 | 27 | OLD_RELEASES_URL = 'http://old-releases.ubuntu.com/ubuntu/' 28 | """The URL where EOL (end of life) Ubuntu releases are hosted (a string).""" 29 | 30 | SECURITY_URL = 'http://security.ubuntu.com/ubuntu' 31 | """The URL where Ubuntu security updates are hosted (a string).""" 32 | 33 | DEFAULT_SUITES = 'release', 'updates', 'backports', 'security' 34 | """A tuple of strings with the Ubuntu suites that are enabled by default.""" 35 | 36 | VALID_COMPONENTS = 'main', 'restricted', 'universe', 'multiverse' 37 | """A tuple of strings with the names of the components available in the Ubuntu package repositories.""" 38 | 39 | VALID_SUITES = 'release', 'security', 'updates', 'backports', 'proposed' 40 | """ 41 | A tuple of strings with the names of the suites available in the Ubuntu package 42 | repositories. 43 | 44 | The actual name of the 'release' suite is the codename of the relevant Ubuntu 45 | release, while the names of the other suites are formed by concatenating the 46 | codename with the suite name (separated by a dash). 47 | 48 | As an example to make things more concrete, Ubuntu 16.04 has the following five 49 | suites available: ``xenial`` (this is the release suite), ``xenial-security``, 50 | ``xenial-updates``, ``xenial-backports`` and ``xenial-proposed``. 51 | """ 52 | 53 | MIRROR_STATUSES = ( 54 | ('Up to date', 0), 55 | ('One hour behind', 60 * 60), 56 | ('Two hours behind', 60 * 60 * 2), 57 | ('Four hours behind', 60 * 60 * 4), 58 | ('Six hours behind', 60 * 60 * 6), 59 | ('One day behind', 60 * 60 * 24), 60 | ('Two days behind', 60 * 60 * 24 * 2), 61 | ('One week behind', 60 * 60 * 24 * 7), 62 | ('Unknown', None), 63 | ) 64 | r""" 65 | A tuple of tuples with Launchpad mirror statuses. Each tuple consists of two values: 66 | 67 | 1. The human readable mirror latency (a string) as used on :data:`MIRRORS_URL`. 68 | 2. The mirror latency expressed in seconds (a number). 69 | 70 | The 'known statuses' used by Launchpad were checked as follows: 71 | 72 | .. code-block:: sh 73 | 74 | $ curl -s https://launchpad.net/+icing/rev18391/combo.css | tr '{},.' '\n' | grep distromirrorstatus 75 | distromirrorstatusUP 76 | distromirrorstatusONEHOURBEHIND 77 | distromirrorstatusTWOHOURSBEHIND 78 | distromirrorstatusFOURHOURSBEHIND 79 | distromirrorstatusSIXHOURSBEHIND 80 | distromirrorstatusONEDAYBEHIND 81 | distromirrorstatusTWODAYSBEHIND 82 | distromirrorstatusONEWEEKBEHIND 83 | distromirrorstatusUNKNOWN 84 | """ 85 | 86 | # Initialize a logger for this module. 87 | logger = logging.getLogger(__name__) 88 | 89 | 90 | def discover_mirrors(): 91 | """ 92 | Discover available Ubuntu mirrors. 93 | 94 | :returns: A set of :class:`.CandidateMirror` objects that have their 95 | :attr:`~.CandidateMirror.mirror_url` property set and may have 96 | the :attr:`~.CandidateMirror.last_updated` property set. 97 | :raises: If no mirrors are discovered an exception is raised. 98 | 99 | This queries :data:`MIRRORS_URL` and :data:`MIRROR_SELECTION_URL` to 100 | discover available Ubuntu mirrors. Here's an example run: 101 | 102 | >>> from apt_mirror_updater.backends.ubuntu import discover_mirrors 103 | >>> from pprint import pprint 104 | >>> pprint(discover_mirrors()) 105 | set([CandidateMirror(mirror_url='http://archive.ubuntu.com/ubuntu/'), 106 | CandidateMirror(mirror_url='http://ftp.nluug.nl/os/Linux/distr/ubuntu/'), 107 | CandidateMirror(mirror_url='http://ftp.snt.utwente.nl/pub/os/linux/ubuntu/'), 108 | CandidateMirror(mirror_url='http://ftp.tudelft.nl/archive.ubuntu.com/'), 109 | CandidateMirror(mirror_url='http://mirror.1000mbps.com/ubuntu/'), 110 | CandidateMirror(mirror_url='http://mirror.amsiohosting.net/archive.ubuntu.com/'), 111 | CandidateMirror(mirror_url='http://mirror.i3d.net/pub/ubuntu/'), 112 | CandidateMirror(mirror_url='http://mirror.nforce.com/pub/linux/ubuntu/'), 113 | CandidateMirror(mirror_url='http://mirror.nl.leaseweb.net/ubuntu/'), 114 | CandidateMirror(mirror_url='http://mirror.transip.net/ubuntu/ubuntu/'), 115 | ...]) 116 | """ 117 | timer = Timer() 118 | mirrors = set() 119 | logger.info("Discovering Ubuntu mirrors at %s ..", MIRRORS_URL) 120 | data = fetch_url(MIRRORS_URL, retry=True) 121 | soup = BeautifulSoup(data, 'html.parser') 122 | for table in soup.findAll('table'): 123 | for tr in table.findAll('tr'): 124 | for a in tr.findAll('a', href=True): 125 | # Check if the link looks like a mirror URL. 126 | if (a['href'].startswith(('http://', 'https://')) and 127 | a['href'].endswith('/ubuntu/')): 128 | # Try to figure out the mirror's reported latency. 129 | last_updated = None 130 | text = u''.join(tr.findAll(text=True)) 131 | for status_label, num_seconds in MIRROR_STATUSES: 132 | if status_label in text: 133 | last_updated = num_seconds 134 | break 135 | # Add the mirror to our overview. 136 | mirrors.add(CandidateMirror( 137 | mirror_url=a['href'], 138 | last_updated=last_updated, 139 | )) 140 | # Skip to the next row. 141 | break 142 | if not mirrors: 143 | raise Exception("Failed to discover any Ubuntu mirrors! (using %s)" % MIRRORS_URL) 144 | # Discover fast (geographically suitable) mirrors to speed up ranking. 145 | # See also https://github.com/xolox/python-apt-mirror-updater/issues/6. 146 | selected_mirrors = discover_mirror_selection() 147 | slow_mirrors = mirrors ^ selected_mirrors 148 | fast_mirrors = mirrors ^ slow_mirrors 149 | if len(fast_mirrors) > 10: 150 | # Narrow down the list of candidate mirrors to fast mirrors. 151 | logger.info("Discovered %s in %s (narrowed down from %s).", 152 | pluralize(len(fast_mirrors), "Ubuntu mirror"), 153 | timer, pluralize(len(mirrors), "mirror")) 154 | mirrors = fast_mirrors 155 | else: 156 | logger.info("Discovered %s in %s.", pluralize(len(mirrors), "Ubuntu mirror"), timer) 157 | return mirrors 158 | 159 | 160 | def discover_mirror_selection(): 161 | """Discover "geographically suitable" Ubuntu mirrors.""" 162 | timer = Timer() 163 | logger.info("Identifying fast Ubuntu mirrors using %s ..", MIRROR_SELECTION_URL) 164 | data = fetch_url(MIRROR_SELECTION_URL, retry=False) 165 | dammit = UnicodeDammit(data) 166 | mirrors = set( 167 | CandidateMirror(mirror_url=mirror_url.strip()) 168 | for mirror_url in dammit.unicode_markup.splitlines() 169 | if mirror_url and not mirror_url.isspace() 170 | ) 171 | logger.debug("Found %s in %s.", pluralize(len(mirrors), "fast Ubuntu mirror"), timer) 172 | return mirrors 173 | 174 | 175 | def generate_sources_list(mirror_url, codename, 176 | suites=DEFAULT_SUITES, 177 | components=VALID_COMPONENTS, 178 | enable_sources=False): 179 | """ 180 | Generate the contents of ``/etc/apt/sources.list`` for an Ubuntu system. 181 | 182 | :param mirror_url: The base URL of the mirror (a string). 183 | :param codename: The codename of the Ubuntu release (a string like 'trusty' or 'xenial'). 184 | :param suites: An iterable of strings (defaults to :data:`DEFAULT_SUITES`, 185 | refer to :data:`VALID_SUITES` for details). 186 | :param components: An iterable of strings (refer to 187 | :data:`VALID_COMPONENTS` for details). 188 | :param enable_sources: :data:`True` to include ``deb-src`` entries, 189 | :data:`False` to omit them. 190 | :returns: The suggested contents of ``/etc/apt/sources.list`` (a string). 191 | """ 192 | # Validate the suites. 193 | invalid_suites = [s for s in suites if s not in VALID_SUITES] 194 | if invalid_suites: 195 | msg = "Invalid Ubuntu suite(s) given! (%s)" 196 | raise ValueError(msg % invalid_suites) 197 | # Validate the components. 198 | invalid_components = [c for c in components if c not in VALID_COMPONENTS] 199 | if invalid_components: 200 | msg = "Invalid Ubuntu component(s) given! (%s)" 201 | raise ValueError(msg % invalid_components) 202 | # Generate the /etc/apt/sources.list file contents. 203 | lines = [] 204 | directives = ('deb', 'deb-src') if enable_sources else ('deb',) 205 | for suite in suites: 206 | for directive in directives: 207 | lines.append(format( 208 | '{directive} {mirror} {suite} {components}', directive=directive, 209 | mirror=(OLD_RELEASES_URL if mirrors_are_equal(mirror_url, OLD_RELEASES_URL) 210 | else (SECURITY_URL if suite == 'security' else mirror_url)), 211 | suite=(codename if suite == 'release' else codename + '-' + suite), 212 | components=' '.join(components), 213 | )) 214 | return '\n'.join(lines) 215 | -------------------------------------------------------------------------------- /apt_mirror_updater/cli.py: -------------------------------------------------------------------------------- 1 | # Automated, robust apt-get mirror selection for Debian and Ubuntu. 2 | # 3 | # Author: Peter Odding 4 | # Last Change: April 15, 2020 5 | # URL: https://apt-mirror-updater.readthedocs.io 6 | 7 | """ 8 | Usage: apt-mirror-updater [OPTIONS] 9 | 10 | The apt-mirror-updater program automates robust apt-get mirror selection for 11 | Debian and Ubuntu by enabling discovery of available mirrors, ranking of 12 | available mirrors, automatic switching between mirrors and robust package list 13 | updating. 14 | 15 | Supported options: 16 | 17 | -r, --remote-host=SSH_ALIAS 18 | 19 | Operate on a remote system instead of the local system. The SSH_ALIAS 20 | argument gives the SSH alias of the remote host. It is assumed that the 21 | remote account has root privileges or password-less sudo access. 22 | 23 | -f, --find-current-mirror 24 | 25 | Determine the main mirror that is currently configured in 26 | /etc/apt/sources.list and report its URL on standard output. 27 | 28 | -b, --find-best-mirror 29 | 30 | Discover available mirrors, rank them, select the best one and report its 31 | URL on standard output. 32 | 33 | -l, --list-mirrors 34 | 35 | List available (ranked) mirrors on the terminal in a human readable format. 36 | 37 | -c, --change-mirror=MIRROR_URL 38 | 39 | Update /etc/apt/sources.list to use the given MIRROR_URL. 40 | 41 | -a, --auto-change-mirror 42 | 43 | Discover available mirrors, rank the mirrors by connection speed and update 44 | status and update /etc/apt/sources.list to use the best available mirror. 45 | 46 | -u, --update, --update-package-lists 47 | 48 | Update the package lists using `apt-get update', retrying on failure and 49 | automatically switch to a different mirror when it looks like the current 50 | mirror is being updated. 51 | 52 | -x, --exclude=PATTERN 53 | 54 | Add a pattern to the mirror selection blacklist. PATTERN is expected to be 55 | a shell pattern (containing wild cards like `?' and `*') that is matched 56 | against the full URL of each mirror. 57 | 58 | -m, --max=COUNT 59 | 60 | Don't query more than COUNT mirrors for their connection status 61 | (defaults to 50). If you give the number 0 no limit will be applied. 62 | 63 | Because Ubuntu mirror discovery can report more than 300 mirrors it's 64 | useful to limit the number of mirrors that are queried, otherwise the 65 | ranking of mirrors will take a long time (because over 300 connections 66 | need to be established). 67 | 68 | -v, --verbose 69 | 70 | Increase logging verbosity (can be repeated). 71 | 72 | -q, --quiet 73 | 74 | Decrease logging verbosity (can be repeated). 75 | 76 | -h, --help 77 | 78 | Show this message and exit. 79 | """ 80 | 81 | # Standard library modules. 82 | import functools 83 | import getopt 84 | import logging 85 | import sys 86 | 87 | # External dependencies. 88 | import coloredlogs 89 | from executor.contexts import LocalContext, RemoteContext 90 | from humanfriendly import format_size, format_timespan 91 | from humanfriendly.tables import format_smart_table 92 | from humanfriendly.terminal import connected_to_terminal, output, usage, warning 93 | 94 | # Modules included in our package. 95 | from apt_mirror_updater import MAX_MIRRORS, AptMirrorUpdater 96 | 97 | # Initialize a logger for this module. 98 | logger = logging.getLogger(__name__) 99 | 100 | 101 | def main(): 102 | """Command line interface for the ``apt-mirror-updater`` program.""" 103 | # Initialize logging to the terminal and system log. 104 | coloredlogs.install(syslog=True) 105 | # Command line option defaults. 106 | context = LocalContext() 107 | updater = AptMirrorUpdater(context=context) 108 | limit = MAX_MIRRORS 109 | actions = [] 110 | # Parse the command line arguments. 111 | try: 112 | options, arguments = getopt.getopt(sys.argv[1:], 'r:fblc:aux:m:vqh', [ 113 | 'remote-host=', 'find-current-mirror', 'find-best-mirror', 114 | 'list-mirrors', 'change-mirror', 'auto-change-mirror', 'update', 115 | 'update-package-lists', 'exclude=', 'max=', 'verbose', 'quiet', 116 | 'help', 117 | ]) 118 | for option, value in options: 119 | if option in ('-r', '--remote-host'): 120 | if actions: 121 | msg = "The %s option should be the first option given on the command line!" 122 | raise Exception(msg % option) 123 | context = RemoteContext(value) 124 | updater = AptMirrorUpdater(context=context) 125 | elif option in ('-f', '--find-current-mirror'): 126 | actions.append(functools.partial(report_current_mirror, updater)) 127 | elif option in ('-b', '--find-best-mirror'): 128 | actions.append(functools.partial(report_best_mirror, updater)) 129 | elif option in ('-l', '--list-mirrors'): 130 | actions.append(functools.partial(report_available_mirrors, updater)) 131 | elif option in ('-c', '--change-mirror'): 132 | actions.append(functools.partial(updater.change_mirror, value)) 133 | elif option in ('-a', '--auto-change-mirror'): 134 | actions.append(updater.change_mirror) 135 | elif option in ('-u', '--update', '--update-package-lists'): 136 | actions.append(updater.smart_update) 137 | elif option in ('-x', '--exclude'): 138 | actions.insert(0, functools.partial(updater.ignore_mirror, value)) 139 | elif option in ('-m', '--max'): 140 | limit = int(value) 141 | elif option in ('-v', '--verbose'): 142 | coloredlogs.increase_verbosity() 143 | elif option in ('-q', '--quiet'): 144 | coloredlogs.decrease_verbosity() 145 | elif option in ('-h', '--help'): 146 | usage(__doc__) 147 | return 148 | else: 149 | assert False, "Unhandled option!" 150 | if not actions: 151 | usage(__doc__) 152 | return 153 | # Propagate options to the Python API. 154 | updater.max_mirrors = limit 155 | except Exception as e: 156 | warning("Error: Failed to parse command line arguments! (%s)" % e) 157 | sys.exit(1) 158 | # Perform the requested action(s). 159 | try: 160 | for callback in actions: 161 | callback() 162 | except Exception: 163 | logger.exception("Encountered unexpected exception! Aborting ..") 164 | sys.exit(1) 165 | 166 | 167 | def report_current_mirror(updater): 168 | """Print the URL of the currently configured ``apt-get`` mirror.""" 169 | output(updater.current_mirror) 170 | 171 | 172 | def report_best_mirror(updater): 173 | """Print the URL of the "best" mirror.""" 174 | output(updater.best_mirror) 175 | 176 | 177 | def report_available_mirrors(updater): 178 | """Print the available mirrors to the terminal (in a human friendly format).""" 179 | if connected_to_terminal(): 180 | have_bandwidth = any(c.bandwidth for c in updater.ranked_mirrors) 181 | have_last_updated = any(c.last_updated is not None for c in updater.ranked_mirrors) 182 | column_names = ["Rank", "Mirror URL", "Available?", "Updating?"] 183 | if have_last_updated: 184 | column_names.append("Last updated") 185 | if have_bandwidth: 186 | column_names.append("Bandwidth") 187 | data = [] 188 | for i, candidate in enumerate(updater.ranked_mirrors, start=1): 189 | row = [i, candidate.mirror_url, 190 | "Yes" if candidate.is_available else "No", 191 | "Yes" if candidate.is_updating else "No"] 192 | if have_last_updated: 193 | row.append("Up to date" if candidate.last_updated == 0 else ( 194 | "%s behind" % format_timespan(candidate.last_updated) 195 | if candidate.last_updated else "Unknown" 196 | )) 197 | if have_bandwidth: 198 | row.append("%s/s" % format_size(round(candidate.bandwidth, 2)) 199 | if candidate.bandwidth else "Unknown") 200 | data.append(row) 201 | output(format_smart_table(data, column_names=column_names)) 202 | else: 203 | output(u"\n".join( 204 | candidate.mirror_url for candidate in updater.ranked_mirrors 205 | if candidate.is_available and not candidate.is_updating 206 | )) 207 | -------------------------------------------------------------------------------- /apt_mirror_updater/http.py: -------------------------------------------------------------------------------- 1 | # Automated, robust apt-get mirror selection for Debian and Ubuntu. 2 | # 3 | # Author: Peter Odding 4 | # Last Change: April 15, 2020 5 | # URL: https://apt-mirror-updater.readthedocs.io 6 | 7 | """Simple, robust and concurrent HTTP requests (designed for one very narrow use case).""" 8 | 9 | # Standard library modules. 10 | import logging 11 | import multiprocessing 12 | import signal 13 | 14 | # External dependencies. 15 | from humanfriendly import Timer, format_size 16 | from six.moves.urllib.request import urlopen 17 | from stopit import SignalTimeout, TimeoutException 18 | 19 | # Initialize a logger for this module. 20 | logger = logging.getLogger(__name__) 21 | 22 | # Stop the `stopit' logger from logging tracebacks. 23 | logging.getLogger('stopit').setLevel(logging.ERROR) 24 | 25 | 26 | def fetch_url(url, timeout=10, retry=False, max_attempts=3): 27 | """ 28 | Fetch a URL, optionally retrying on failure. 29 | 30 | :param url: The URL to fetch (a string). 31 | :param timeout: The maximum time in seconds that's allowed to pass before 32 | the request is aborted (a number, defaults to 10 seconds). 33 | :param retry: Whether to retry on failure (defaults to :data:`False`). 34 | :param max_attempts: The maximum number of attempts when retrying is 35 | enabled (an integer, defaults to three). 36 | :returns: The response body (a byte string). 37 | :raises: Any of the following exceptions can be raised: 38 | 39 | - :exc:`NotFoundError` when the URL returns a 404 status code. 40 | - :exc:`InvalidResponseError` when the URL returns a status code 41 | that isn't 200. 42 | - `stopit.TimeoutException`_ when the request takes longer 43 | than `timeout` seconds (refer to the linked documentation for 44 | details). 45 | - Any exception raised by Python's standard library in the last 46 | attempt (assuming all attempts raise an exception). 47 | 48 | .. _stopit.TimeoutException: https://pypi.org/project/stopit/#exception 49 | """ 50 | timer = Timer() 51 | logger.debug("Fetching %s ..", url) 52 | for i in range(1, max_attempts + 1): 53 | try: 54 | with SignalTimeout(timeout, swallow_exc=False): 55 | response = urlopen(url) 56 | status_code = response.getcode() 57 | if status_code != 200: 58 | exc_type = (NotFoundError if status_code == 404 else InvalidResponseError) 59 | raise exc_type("URL returned unexpected status code %s! (%s)" % (status_code, url)) 60 | response_body = response.read() 61 | logger.debug("Took %s to fetch %s.", timer, url) 62 | return response_body 63 | except (NotFoundError, TimeoutException): 64 | # We never retry 404 responses and timeouts. 65 | raise 66 | except Exception as e: 67 | if retry and i < max_attempts: 68 | logger.warning("Failed to fetch %s, retrying (%i/%i, error was: %s)", url, i, max_attempts, e) 69 | else: 70 | raise 71 | 72 | 73 | def fetch_concurrent(urls, concurrency=None): 74 | """ 75 | Fetch the given URLs concurrently using :mod:`multiprocessing`. 76 | 77 | :param urls: An iterable of URLs (strings). 78 | :param concurrency: Override the concurrency (an integer, defaults to the 79 | value computed by :func:`get_default_concurrency()`). 80 | :returns: A list of tuples like those returned by :func:`fetch_worker()`. 81 | """ 82 | if concurrency is None: 83 | concurrency = get_default_concurrency() 84 | pool = multiprocessing.Pool(concurrency) 85 | try: 86 | results = pool.map(fetch_worker, urls, chunksize=1) 87 | pool.close() 88 | pool.join() 89 | return results 90 | except Exception: 91 | pool.terminate() 92 | pool.join() 93 | raise 94 | 95 | 96 | def get_default_concurrency(): 97 | """ 98 | Get the default concurrency for :func:`fetch_concurrent()`. 99 | 100 | :returns: A positive integer number. 101 | """ 102 | return max(4, multiprocessing.cpu_count() * 2) 103 | 104 | 105 | def fetch_worker(url): 106 | """ 107 | Fetch the given URL for :func:`fetch_concurrent()`. 108 | 109 | :param url: The URL to fetch (a string). 110 | :returns: A tuple of three values: 111 | 112 | 1. The URL that was fetched (a string). 113 | 2. The data that was fetched (a string or :data:`None`). 114 | 3. The number of seconds it took to fetch the URL (a number). 115 | """ 116 | # Ignore Control-C instead of raising KeyboardInterrupt because (due to a 117 | # quirk in multiprocessing) this can cause the parent and child processes 118 | # to get into a deadlock kind of state where only Control-Z will get you 119 | # your precious terminal back; super annoying IMHO. 120 | signal.signal(signal.SIGINT, signal.SIG_IGN) 121 | timer = Timer() 122 | try: 123 | data = fetch_url(url, retry=False) 124 | except Exception as e: 125 | logger.debug("Failed to fetch %s! (%s)", url, e) 126 | data = None 127 | else: 128 | kbps = format_size(round(len(data) / timer.elapsed_time, 2)) 129 | logger.debug("Downloaded %s at %s per second.", url, kbps) 130 | return url, data, timer.elapsed_time 131 | 132 | 133 | class InvalidResponseError(Exception): 134 | 135 | """Raised by :func:`fetch_url()` when a URL returns a status code that isn't 200.""" 136 | 137 | 138 | class NotFoundError(InvalidResponseError): 139 | 140 | """Raised by :func:`fetch_url()` when a URL returns a 404 status code.""" 141 | -------------------------------------------------------------------------------- /apt_mirror_updater/releases.py: -------------------------------------------------------------------------------- 1 | # Easy to use metadata on Debian and Ubuntu releases. 2 | # 3 | # Author: Peter Odding 4 | # Last Change: September 15, 2021 5 | # URL: https://apt-mirror-updater.readthedocs.io 6 | 7 | """ 8 | Easy to use metadata on Debian and Ubuntu releases. 9 | 10 | This module started out with the purpose of reliable `end of life`_ (EOL) 11 | detection for Debian and Ubuntu releases based on data provided by the 12 | distro-info-data_ package. Since then the need arose to access more of the 13 | available metadata and so the ``eol`` module became the ``releases`` module. 14 | 15 | Debian and Ubuntu releases have an EOL date that marks the end of support for 16 | each release. At that date the release stops receiving further (security) 17 | updates and some time after package mirrors stop serving the release. 18 | 19 | The distro-info-data_ package contains CSV files with metadata about Debian and 20 | Ubuntu releases. This module parses those CSV files to make this metadata 21 | available in Python. This enables `apt-mirror-updater` to make an informed 22 | decision about the following questions: 23 | 24 | 1. Is a given Debian or Ubuntu release expected to be available on mirrors or 25 | will it only be available in the archive of old releases? 26 | 27 | 2. Is the signing key of a given Ubuntu release expected to be included in the 28 | main keyring (:data:`UBUNTU_KEYRING_CURRENT`) or should the keyring with 29 | removed keys (:data:`UBUNTU_KEYRING_REMOVED`) be used? 30 | 31 | To make it possible to run `apt-mirror-updater` without direct access to the 32 | CSV files, a copy of the relevant information has been embedded in the source 33 | code. 34 | 35 | .. _end of life: https://en.wikipedia.org/wiki/End-of-life_(product) 36 | .. _distro-info-data: https://packages.debian.org/distro-info-data 37 | """ 38 | 39 | # Standard library modules. 40 | import csv 41 | import datetime 42 | import decimal 43 | import glob 44 | import logging 45 | import numbers 46 | import os 47 | 48 | # External dependencies. 49 | from executor import execute 50 | from humanfriendly.decorators import cached 51 | from property_manager import PropertyManager, key_property, lazy_property, required_property, writable_property 52 | from six import string_types 53 | 54 | DISTRO_INFO_DIRECTORY = '/usr/share/distro-info' 55 | """The pathname of the directory with CSV files containing release metadata (a string).""" 56 | 57 | DEBIAN_KEYRING_CURRENT = '/usr/share/keyrings/debian-keyring.gpg' 58 | """The pathname of the main Debian keyring file (a string).""" 59 | 60 | UBUNTU_KEYRING_CURRENT = '/usr/share/keyrings/ubuntu-archive-keyring.gpg' 61 | """The pathname of the main Ubuntu keyring file (a string).""" 62 | 63 | UBUNTU_KEYRING_REMOVED = '/usr/share/keyrings/ubuntu-archive-removed-keys.gpg' 64 | """The pathname of the Ubuntu keyring file with removed keys (a string).""" 65 | 66 | # Public identifiers that require documentation. 67 | __all__ = ( 68 | 'DISTRO_INFO_DIRECTORY', 69 | 'DEBIAN_KEYRING_CURRENT', 70 | 'UBUNTU_KEYRING_CURRENT', 71 | 'UBUNTU_KEYRING_REMOVED', 72 | 'Release', 73 | 'coerce_release', 74 | 'discover_releases', 75 | 'is_version_string', 76 | 'logger', 77 | 'parse_csv_file', 78 | 'parse_date', 79 | 'parse_version', 80 | 'ubuntu_keyring_updated', 81 | ) 82 | 83 | # Initialize a logger. 84 | logger = logging.getLogger(__name__) 85 | 86 | 87 | def coerce_release(value): 88 | """ 89 | Try to coerce the given value to a Debian or Ubuntu release. 90 | 91 | :param value: The value to coerce (a number, a string or a :class:`Release` object). 92 | :returns: A :class:`Release` object. 93 | :raises: :exc:`~exceptions.ValueError` when the given value cannot be coerced to a known release. 94 | 95 | The following values can be coerced: 96 | 97 | - Numbers and numbers formatted as strings match :attr:`Release.version`. 98 | - Strings match :attr:`Release.codename` (case insensitive). 99 | 100 | .. warning:: Don't use floating point numbers like 10.04 because their 101 | actual value will be something like 10.039999999999999147 102 | which won't match the intended release. 103 | """ 104 | # Release objects pass through untouched. 105 | if isinstance(value, Release): 106 | return value 107 | # Numbers and version strings are matched against release versions. 108 | if isinstance(value, numbers.Number) or is_version_string(value): 109 | typed_value = decimal.Decimal(value) 110 | matches = [release for release in discover_releases() if release.version == typed_value] 111 | if len(matches) != 1: 112 | msg = "The number %s doesn't match a known Debian or Ubuntu release!" 113 | raise ValueError(msg % value) 114 | return matches[0] 115 | # Other strings are matched against release code names. 116 | matches = [release for release in discover_releases() if value.lower() in release.codename.lower()] 117 | if len(matches) != 1: 118 | msg = "The string %r doesn't match a known Debian or Ubuntu release!" 119 | raise ValueError(msg % value) 120 | return matches[0] 121 | 122 | 123 | @cached 124 | def discover_releases(): 125 | """ 126 | Discover known Debian, Elementary OS and Ubuntu releases. 127 | 128 | :returns: A list of discovered :class:`Release` objects sorted by 129 | :attr:`~Release.distributor_id` and :attr:`~Release.version`. 130 | 131 | The first time this function is called it will try to parse the CSV files 132 | in ``/usr/share/distro-info`` using :func:`parse_csv_file()` and merge any 133 | releases it finds with the releases embedded into the source code of this 134 | module and the releases defined by 135 | :data:`apt_mirror_updater.backends.elementary.KNOWN_RELEASES`. The result 136 | is cached and returned each time the function is called. It's not a problem 137 | if the ``/usr/share/distro-info`` directory doesn't exist or doesn't 138 | contain any ``*.csv`` files (it won't cause a warning or error). Of course 139 | in this case only the embedded releases will be returned. 140 | """ 141 | # Discover the known releases on the first call to discover_releases(). 142 | # First we check the CSV files on the system where apt-mirror-updater 143 | # is running, because those files may be more up-to-date than the 144 | # bundled information is. 145 | result = set() 146 | for filename in glob.glob(os.path.join(DISTRO_INFO_DIRECTORY, '*.csv')): 147 | for release in parse_csv_file(filename): 148 | result.add(release) 149 | # Add the Debian and Ubuntu releases bundled with apt-mirror-updater to the 150 | # result without causing duplicate entries (due to the use of a set and key 151 | # properties). 152 | result.update(BUNDLED_RELEASES) 153 | # Add the Elementary OS releases bundled with apt-mirror-updater. 154 | # We import the known releases here to avoid circular imports. 155 | from apt_mirror_updater.backends import elementary 156 | result.update(elementary.KNOWN_RELEASES) 157 | # Sort the releases by distributor ID and version / series. 158 | return sorted(result, key=lambda r: (r.distributor_id, r.version or 0, r.series)) 159 | 160 | 161 | def is_version_string(value): 162 | """Check whether the given value is a string containing a positive number.""" 163 | try: 164 | return isinstance(value, string_types) and float(value) > 0 165 | except Exception: 166 | return False 167 | 168 | 169 | def parse_csv_file(filename): 170 | """ 171 | Parse a CSV file in the format of the ``/usr/share/distro-info/*.csv`` files. 172 | 173 | :param filename: The pathname of the CSV file (a string). 174 | :returns: A generator of :class:`Release` objects. 175 | """ 176 | # We import this here to avoid a circular import. 177 | from apt_mirror_updater.backends.debian import LTS_RELEASES 178 | basename, extension = os.path.splitext(os.path.basename(filename)) 179 | distributor_id = basename.lower() 180 | with open(filename) as handle: 181 | for entry in csv.DictReader(handle): 182 | yield Release( 183 | codename=entry['codename'], 184 | is_lts=( 185 | entry['series'] in LTS_RELEASES if distributor_id == 'debian' else ( 186 | 'LTS' in entry['version'] if distributor_id == 'ubuntu' else ( 187 | # Neither Debian nor Ubuntu, let's not assume anything... 188 | False 189 | ) 190 | ) 191 | ), 192 | created_date=parse_date(entry['created']), 193 | distributor_id=distributor_id, 194 | eol_date=parse_date(entry['eol']), 195 | extended_eol_date=( 196 | # Special handling for Debian LTS releases. 197 | datetime.datetime.fromtimestamp(LTS_RELEASES[entry['series']]).date() 198 | if distributor_id == 'debian' and entry['series'] in LTS_RELEASES 199 | # Ubuntu LTS releases are defined by the CSV file. 200 | else parse_date(entry.get('eol-server')) 201 | ), 202 | release_date=parse_date(entry['release']), 203 | series=entry['series'], 204 | version=parse_version(entry['version']) if entry['version'] else None, 205 | ) 206 | 207 | 208 | def parse_date(value): 209 | """Convert a ``YYYY-MM-DD`` string to a :class:`datetime.date` object.""" 210 | return datetime.datetime.strptime(value, '%Y-%m-%d').date() if value else None 211 | 212 | 213 | def parse_version(value): 214 | """Convert a version string to a :class:`~decimal.Decimal` number.""" 215 | for token in value.split(): 216 | try: 217 | return decimal.Decimal(token) 218 | except ValueError: 219 | pass 220 | msg = "Failed to convert version string to number! (%r)" 221 | raise ValueError(msg % value) 222 | 223 | 224 | @cached 225 | def ubuntu_keyring_updated(): 226 | """ 227 | Detect update `#1363482`_ to the ``ubuntu-keyring`` package. 228 | 229 | :returns: :data:`True` when version ``2016.10.27`` or newer is installed, 230 | :data:`False` when an older version is installed. 231 | 232 | This function checks if the changes discussed in Launchpad bug `#1363482`_ 233 | apply to the current system using the ``dpkg-query --show`` and ``dpkg 234 | --compare-versions`` commands. For more details refer to `issue #8`_. 235 | 236 | .. _#1363482: https://bugs.launchpad.net/ubuntu/+source/ubuntu-keyring/+bug/1363482 237 | .. _issue #8: https://github.com/xolox/python-apt-mirror-updater/issues/8 238 | """ 239 | # Use external commands to check the installed version of the package. 240 | version = execute('dpkg-query', '--show', '--showformat=${Version}', 'ubuntu-keyring', capture=True) 241 | logger.debug("Detected ubuntu-keyring package version: %s", version) 242 | updated = execute('dpkg', '--compare-versions', version, '>=', '2016.10.27', check=False, silent=True) 243 | logger.debug("Does Launchpad bug #1363482 apply? %s", updated) 244 | return updated 245 | 246 | 247 | class Release(PropertyManager): 248 | 249 | """Data class for metadata on Debian, Elementary OS and Ubuntu releases.""" 250 | 251 | @key_property 252 | def codename(self): 253 | """The long version of :attr:`series` (a string).""" 254 | 255 | @required_property 256 | def created_date(self): 257 | """The date on which the release was created (a :class:`~datetime.date` object).""" 258 | 259 | @key_property 260 | def distributor_id(self): 261 | """The name of the distributor (one of the strings ``debian``, ``elementary`` or ``ubuntu``).""" 262 | 263 | @writable_property 264 | def eol_date(self): 265 | """The date on which the desktop release stops being supported (a :class:`~datetime.date` object).""" 266 | 267 | @writable_property 268 | def extended_eol_date(self): 269 | """The date on which the server release stops being supported (a :class:`~datetime.date` object).""" 270 | 271 | @lazy_property 272 | def is_eol(self): 273 | """Whether the release has reached its end-of-life date (a boolean or :data:`None`).""" 274 | eol_date = self.extended_eol_date or self.eol_date 275 | if eol_date: 276 | return datetime.date.today() >= eol_date 277 | else: 278 | return False 279 | 280 | @writable_property 281 | def is_lts(self): 282 | """Whether a release is a long term support release (a boolean).""" 283 | 284 | @writable_property 285 | def release_date(self): 286 | """The date on which the release was published (a :class:`~datetime.date` object).""" 287 | 288 | @key_property 289 | def series(self): 290 | """The short version of :attr:`codename` (a string).""" 291 | 292 | @writable_property 293 | def upstream_distributor_id(self): 294 | """The upstream distributor ID (a string, defaults to :attr:`distributor_id`).""" 295 | return self.distributor_id 296 | 297 | @writable_property 298 | def upstream_series(self): 299 | """The upstream series (a string, defaults to :attr:`series`).""" 300 | return self.series 301 | 302 | @writable_property 303 | def upstream_version(self): 304 | """The upstream version (a string, defaults to :attr:`version`).""" 305 | return self.version 306 | 307 | @writable_property 308 | def version(self): 309 | """ 310 | The version number of the release (a :class:`~decimal.Decimal` number). 311 | 312 | This property has a :class:`~decimal.Decimal` value to enable proper 313 | sorting based on numeric comparison. 314 | """ 315 | 316 | @lazy_property 317 | def keyring_file(self): 318 | """ 319 | The pathname of the keyring with signing keys for this release (a string). 320 | 321 | This property exists to work around a bug in ``debootstrap`` which may 322 | use the wrong keyring to create Ubuntu chroots, for more details refer 323 | to :func:`ubuntu_keyring_updated()`. 324 | """ 325 | filename = None 326 | reason = None 327 | logger.debug("Selecting keyring file for %s ..", self) 328 | if self.upstream_distributor_id == 'debian': 329 | filename = DEBIAN_KEYRING_CURRENT 330 | reason = "only known keyring" 331 | elif self.upstream_distributor_id == 'ubuntu': 332 | if ubuntu_keyring_updated(): 333 | if self.upstream_version > decimal.Decimal('12.04'): 334 | filename = UBUNTU_KEYRING_CURRENT 335 | reason = "new keyring package / new release" 336 | else: 337 | filename = UBUNTU_KEYRING_REMOVED 338 | reason = "new keyring package / old release" 339 | else: 340 | filename = UBUNTU_KEYRING_CURRENT 341 | reason = "old keyring package" 342 | else: 343 | msg = "Unsupported distributor ID! (%s)" 344 | raise EnvironmentError(msg % self.distributor_id) 345 | logger.debug("Using %s (reason: %s).", filename, reason) 346 | return filename 347 | 348 | def __str__(self): 349 | """ 350 | Render a human friendly representation of a :class:`Release` object. 351 | 352 | The result will be something like this: 353 | 354 | - Debian 9 (stretch) 355 | - Ubuntu 18.04 (bionic) 356 | """ 357 | label = [self.distributor_id.capitalize()] 358 | if self.version: 359 | label.append(str(self.version)) 360 | label.append("(%s)" % self.series) 361 | if self.upstream_distributor_id and self.upstream_version: 362 | label.extend(("based on", self.upstream_distributor_id.title(), str(self.upstream_version))) 363 | return " ".join(label) 364 | 365 | 366 | # [[[cog 367 | # 368 | # import cog 369 | # import decimal 370 | # from apt_mirror_updater.releases import discover_releases 371 | # 372 | # indent = " " * 4 373 | # cog.out("\nBUNDLED_RELEASES = [\n") 374 | # for release in discover_releases(): 375 | # if release.distributor_id == 'elementary': 376 | # # Don't duplicate the Elementary OS releases. 377 | # continue 378 | # cog.out(indent + "Release(\n") 379 | # for name in release.find_properties(cached=False): 380 | # value = getattr(release, name) 381 | # if ((name == 'upstream_distributor_id' and value == release.distributor_id) or 382 | # (name == 'upstream_series' and value == release.series) or 383 | # (name == 'upstream_version' and value == release.version)): 384 | # # Skip redundant values. 385 | # continue 386 | # if value is not None: 387 | # if isinstance(value, decimal.Decimal): 388 | # # It seems weirdly inconsistency to me that this is needed 389 | # # for decimal.Decimal() but not for datetime.date() but I 390 | # # guess the simple explanation is that repr() output simply 391 | # # isn't guaranteed to be accepted by eval(). 392 | # value = "decimal." + repr(value) 393 | # else: 394 | # value = repr(value) 395 | # cog.out(indent * 2 + name + "=" + value + ",\n") 396 | # cog.out(indent + "),\n") 397 | # cog.out("]\n\n") 398 | # 399 | # ]]] 400 | 401 | BUNDLED_RELEASES = [ 402 | Release( 403 | codename='Experimental', 404 | created_date=datetime.date(1993, 8, 16), 405 | distributor_id='debian', 406 | is_lts=False, 407 | series='experimental', 408 | ), 409 | Release( 410 | codename='Sid', 411 | created_date=datetime.date(1993, 8, 16), 412 | distributor_id='debian', 413 | is_lts=False, 414 | series='sid', 415 | ), 416 | Release( 417 | codename='Buzz', 418 | created_date=datetime.date(1993, 8, 16), 419 | distributor_id='debian', 420 | eol_date=datetime.date(1997, 6, 5), 421 | is_lts=False, 422 | release_date=datetime.date(1996, 6, 17), 423 | series='buzz', 424 | version=decimal.Decimal('1.1'), 425 | ), 426 | Release( 427 | codename='Rex', 428 | created_date=datetime.date(1996, 6, 17), 429 | distributor_id='debian', 430 | eol_date=datetime.date(1998, 6, 5), 431 | is_lts=False, 432 | release_date=datetime.date(1996, 12, 12), 433 | series='rex', 434 | version=decimal.Decimal('1.2'), 435 | ), 436 | Release( 437 | codename='Bo', 438 | created_date=datetime.date(1996, 12, 12), 439 | distributor_id='debian', 440 | eol_date=datetime.date(1999, 3, 9), 441 | is_lts=False, 442 | release_date=datetime.date(1997, 6, 5), 443 | series='bo', 444 | version=decimal.Decimal('1.3'), 445 | ), 446 | Release( 447 | codename='Hamm', 448 | created_date=datetime.date(1997, 6, 5), 449 | distributor_id='debian', 450 | eol_date=datetime.date(2000, 3, 9), 451 | is_lts=False, 452 | release_date=datetime.date(1998, 7, 24), 453 | series='hamm', 454 | version=decimal.Decimal('2.0'), 455 | ), 456 | Release( 457 | codename='Slink', 458 | created_date=datetime.date(1998, 7, 24), 459 | distributor_id='debian', 460 | eol_date=datetime.date(2000, 10, 30), 461 | is_lts=False, 462 | release_date=datetime.date(1999, 3, 9), 463 | series='slink', 464 | version=decimal.Decimal('2.1'), 465 | ), 466 | Release( 467 | codename='Potato', 468 | created_date=datetime.date(1999, 3, 9), 469 | distributor_id='debian', 470 | eol_date=datetime.date(2003, 7, 30), 471 | is_lts=False, 472 | release_date=datetime.date(2000, 8, 15), 473 | series='potato', 474 | version=decimal.Decimal('2.2'), 475 | ), 476 | Release( 477 | codename='Woody', 478 | created_date=datetime.date(2000, 8, 15), 479 | distributor_id='debian', 480 | eol_date=datetime.date(2006, 6, 30), 481 | is_lts=False, 482 | release_date=datetime.date(2002, 7, 19), 483 | series='woody', 484 | version=decimal.Decimal('3.0'), 485 | ), 486 | Release( 487 | codename='Sarge', 488 | created_date=datetime.date(2002, 7, 19), 489 | distributor_id='debian', 490 | eol_date=datetime.date(2008, 3, 30), 491 | is_lts=False, 492 | release_date=datetime.date(2005, 6, 6), 493 | series='sarge', 494 | version=decimal.Decimal('3.1'), 495 | ), 496 | Release( 497 | codename='Etch', 498 | created_date=datetime.date(2005, 6, 6), 499 | distributor_id='debian', 500 | eol_date=datetime.date(2010, 2, 15), 501 | is_lts=False, 502 | release_date=datetime.date(2007, 4, 8), 503 | series='etch', 504 | version=decimal.Decimal('4.0'), 505 | ), 506 | Release( 507 | codename='Lenny', 508 | created_date=datetime.date(2007, 4, 8), 509 | distributor_id='debian', 510 | eol_date=datetime.date(2012, 2, 6), 511 | is_lts=False, 512 | release_date=datetime.date(2009, 2, 14), 513 | series='lenny', 514 | version=decimal.Decimal('5.0'), 515 | ), 516 | Release( 517 | codename='Squeeze', 518 | created_date=datetime.date(2009, 2, 14), 519 | distributor_id='debian', 520 | eol_date=datetime.date(2014, 5, 31), 521 | is_lts=False, 522 | release_date=datetime.date(2011, 2, 6), 523 | series='squeeze', 524 | version=decimal.Decimal('6.0'), 525 | ), 526 | Release( 527 | codename='Wheezy', 528 | created_date=datetime.date(2011, 2, 6), 529 | distributor_id='debian', 530 | eol_date=datetime.date(2016, 4, 26), 531 | is_lts=False, 532 | release_date=datetime.date(2013, 5, 4), 533 | series='wheezy', 534 | version=decimal.Decimal('7'), 535 | ), 536 | Release( 537 | codename='Jessie', 538 | created_date=datetime.date(2013, 5, 4), 539 | distributor_id='debian', 540 | eol_date=datetime.date(2018, 6, 6), 541 | extended_eol_date=datetime.date(2020, 6, 30), 542 | is_lts=True, 543 | release_date=datetime.date(2015, 4, 25), 544 | series='jessie', 545 | version=decimal.Decimal('8'), 546 | ), 547 | Release( 548 | codename='Stretch', 549 | created_date=datetime.date(2015, 4, 25), 550 | distributor_id='debian', 551 | eol_date=datetime.date(2020, 7, 6), 552 | extended_eol_date=datetime.date(2022, 6, 30), 553 | is_lts=True, 554 | release_date=datetime.date(2017, 6, 17), 555 | series='stretch', 556 | version=decimal.Decimal('9'), 557 | ), 558 | Release( 559 | codename='Buster', 560 | created_date=datetime.date(2017, 6, 17), 561 | distributor_id='debian', 562 | is_lts=False, 563 | release_date=datetime.date(2019, 7, 6), 564 | series='buster', 565 | version=decimal.Decimal('10'), 566 | ), 567 | Release( 568 | codename='Bullseye', 569 | created_date=datetime.date(2019, 7, 6), 570 | distributor_id='debian', 571 | is_lts=False, 572 | series='bullseye', 573 | version=decimal.Decimal('11'), 574 | ), 575 | Release( 576 | codename='Bookworm', 577 | created_date=datetime.date(2021, 8, 1), 578 | distributor_id='debian', 579 | is_lts=False, 580 | series='bookworm', 581 | version=decimal.Decimal('12'), 582 | ), 583 | Release( 584 | codename='Warty Warthog', 585 | created_date=datetime.date(2004, 3, 5), 586 | distributor_id='ubuntu', 587 | eol_date=datetime.date(2006, 4, 30), 588 | is_lts=False, 589 | release_date=datetime.date(2004, 10, 20), 590 | series='warty', 591 | version=decimal.Decimal('4.10'), 592 | ), 593 | Release( 594 | codename='Hoary Hedgehog', 595 | created_date=datetime.date(2004, 10, 20), 596 | distributor_id='ubuntu', 597 | eol_date=datetime.date(2006, 10, 31), 598 | is_lts=False, 599 | release_date=datetime.date(2005, 4, 8), 600 | series='hoary', 601 | version=decimal.Decimal('5.04'), 602 | ), 603 | Release( 604 | codename='Breezy Badger', 605 | created_date=datetime.date(2005, 4, 8), 606 | distributor_id='ubuntu', 607 | eol_date=datetime.date(2007, 4, 13), 608 | is_lts=False, 609 | release_date=datetime.date(2005, 10, 12), 610 | series='breezy', 611 | version=decimal.Decimal('5.10'), 612 | ), 613 | Release( 614 | codename='Dapper Drake', 615 | created_date=datetime.date(2005, 10, 12), 616 | distributor_id='ubuntu', 617 | eol_date=datetime.date(2009, 7, 14), 618 | extended_eol_date=datetime.date(2011, 6, 1), 619 | is_lts=True, 620 | release_date=datetime.date(2006, 6, 1), 621 | series='dapper', 622 | version=decimal.Decimal('6.06'), 623 | ), 624 | Release( 625 | codename='Edgy Eft', 626 | created_date=datetime.date(2006, 6, 1), 627 | distributor_id='ubuntu', 628 | eol_date=datetime.date(2008, 4, 25), 629 | is_lts=False, 630 | release_date=datetime.date(2006, 10, 26), 631 | series='edgy', 632 | version=decimal.Decimal('6.10'), 633 | ), 634 | Release( 635 | codename='Feisty Fawn', 636 | created_date=datetime.date(2006, 10, 26), 637 | distributor_id='ubuntu', 638 | eol_date=datetime.date(2008, 10, 19), 639 | is_lts=False, 640 | release_date=datetime.date(2007, 4, 19), 641 | series='feisty', 642 | version=decimal.Decimal('7.04'), 643 | ), 644 | Release( 645 | codename='Gutsy Gibbon', 646 | created_date=datetime.date(2007, 4, 19), 647 | distributor_id='ubuntu', 648 | eol_date=datetime.date(2009, 4, 18), 649 | is_lts=False, 650 | release_date=datetime.date(2007, 10, 18), 651 | series='gutsy', 652 | version=decimal.Decimal('7.10'), 653 | ), 654 | Release( 655 | codename='Hardy Heron', 656 | created_date=datetime.date(2007, 10, 18), 657 | distributor_id='ubuntu', 658 | eol_date=datetime.date(2011, 5, 12), 659 | extended_eol_date=datetime.date(2013, 5, 9), 660 | is_lts=True, 661 | release_date=datetime.date(2008, 4, 24), 662 | series='hardy', 663 | version=decimal.Decimal('8.04'), 664 | ), 665 | Release( 666 | codename='Intrepid Ibex', 667 | created_date=datetime.date(2008, 4, 24), 668 | distributor_id='ubuntu', 669 | eol_date=datetime.date(2010, 4, 30), 670 | is_lts=False, 671 | release_date=datetime.date(2008, 10, 30), 672 | series='intrepid', 673 | version=decimal.Decimal('8.10'), 674 | ), 675 | Release( 676 | codename='Jaunty Jackalope', 677 | created_date=datetime.date(2008, 10, 30), 678 | distributor_id='ubuntu', 679 | eol_date=datetime.date(2010, 10, 23), 680 | is_lts=False, 681 | release_date=datetime.date(2009, 4, 23), 682 | series='jaunty', 683 | version=decimal.Decimal('9.04'), 684 | ), 685 | Release( 686 | codename='Karmic Koala', 687 | created_date=datetime.date(2009, 4, 23), 688 | distributor_id='ubuntu', 689 | eol_date=datetime.date(2011, 4, 29), 690 | is_lts=False, 691 | release_date=datetime.date(2009, 10, 29), 692 | series='karmic', 693 | version=decimal.Decimal('9.10'), 694 | ), 695 | Release( 696 | codename='Lucid Lynx', 697 | created_date=datetime.date(2009, 10, 29), 698 | distributor_id='ubuntu', 699 | eol_date=datetime.date(2013, 5, 9), 700 | extended_eol_date=datetime.date(2015, 4, 29), 701 | is_lts=True, 702 | release_date=datetime.date(2010, 4, 29), 703 | series='lucid', 704 | version=decimal.Decimal('10.04'), 705 | ), 706 | Release( 707 | codename='Maverick Meerkat', 708 | created_date=datetime.date(2010, 4, 29), 709 | distributor_id='ubuntu', 710 | eol_date=datetime.date(2012, 4, 10), 711 | is_lts=False, 712 | release_date=datetime.date(2010, 10, 10), 713 | series='maverick', 714 | version=decimal.Decimal('10.10'), 715 | ), 716 | Release( 717 | codename='Natty Narwhal', 718 | created_date=datetime.date(2010, 10, 10), 719 | distributor_id='ubuntu', 720 | eol_date=datetime.date(2012, 10, 28), 721 | is_lts=False, 722 | release_date=datetime.date(2011, 4, 28), 723 | series='natty', 724 | version=decimal.Decimal('11.04'), 725 | ), 726 | Release( 727 | codename='Oneiric Ocelot', 728 | created_date=datetime.date(2011, 4, 28), 729 | distributor_id='ubuntu', 730 | eol_date=datetime.date(2013, 5, 9), 731 | is_lts=False, 732 | release_date=datetime.date(2011, 10, 13), 733 | series='oneiric', 734 | version=decimal.Decimal('11.10'), 735 | ), 736 | Release( 737 | codename='Precise Pangolin', 738 | created_date=datetime.date(2011, 10, 13), 739 | distributor_id='ubuntu', 740 | eol_date=datetime.date(2017, 4, 26), 741 | extended_eol_date=datetime.date(2017, 4, 26), 742 | is_lts=True, 743 | release_date=datetime.date(2012, 4, 26), 744 | series='precise', 745 | version=decimal.Decimal('12.04'), 746 | ), 747 | Release( 748 | codename='Quantal Quetzal', 749 | created_date=datetime.date(2012, 4, 26), 750 | distributor_id='ubuntu', 751 | eol_date=datetime.date(2014, 5, 16), 752 | is_lts=False, 753 | release_date=datetime.date(2012, 10, 18), 754 | series='quantal', 755 | version=decimal.Decimal('12.10'), 756 | ), 757 | Release( 758 | codename='Raring Ringtail', 759 | created_date=datetime.date(2012, 10, 18), 760 | distributor_id='ubuntu', 761 | eol_date=datetime.date(2014, 1, 27), 762 | is_lts=False, 763 | release_date=datetime.date(2013, 4, 25), 764 | series='raring', 765 | version=decimal.Decimal('13.04'), 766 | ), 767 | Release( 768 | codename='Saucy Salamander', 769 | created_date=datetime.date(2013, 4, 25), 770 | distributor_id='ubuntu', 771 | eol_date=datetime.date(2014, 7, 17), 772 | is_lts=False, 773 | release_date=datetime.date(2013, 10, 17), 774 | series='saucy', 775 | version=decimal.Decimal('13.10'), 776 | ), 777 | Release( 778 | codename='Trusty Tahr', 779 | created_date=datetime.date(2013, 10, 17), 780 | distributor_id='ubuntu', 781 | eol_date=datetime.date(2019, 4, 25), 782 | extended_eol_date=datetime.date(2019, 4, 25), 783 | is_lts=True, 784 | release_date=datetime.date(2014, 4, 17), 785 | series='trusty', 786 | version=decimal.Decimal('14.04'), 787 | ), 788 | Release( 789 | codename='Utopic Unicorn', 790 | created_date=datetime.date(2014, 4, 17), 791 | distributor_id='ubuntu', 792 | eol_date=datetime.date(2015, 7, 23), 793 | is_lts=False, 794 | release_date=datetime.date(2014, 10, 23), 795 | series='utopic', 796 | version=decimal.Decimal('14.10'), 797 | ), 798 | Release( 799 | codename='Vivid Vervet', 800 | created_date=datetime.date(2014, 10, 23), 801 | distributor_id='ubuntu', 802 | eol_date=datetime.date(2016, 1, 23), 803 | is_lts=False, 804 | release_date=datetime.date(2015, 4, 23), 805 | series='vivid', 806 | version=decimal.Decimal('15.04'), 807 | ), 808 | Release( 809 | codename='Wily Werewolf', 810 | created_date=datetime.date(2015, 4, 23), 811 | distributor_id='ubuntu', 812 | eol_date=datetime.date(2016, 7, 22), 813 | is_lts=False, 814 | release_date=datetime.date(2015, 10, 22), 815 | series='wily', 816 | version=decimal.Decimal('15.10'), 817 | ), 818 | Release( 819 | codename='Xenial Xerus', 820 | created_date=datetime.date(2015, 10, 22), 821 | distributor_id='ubuntu', 822 | eol_date=datetime.date(2021, 4, 21), 823 | extended_eol_date=datetime.date(2021, 4, 21), 824 | is_lts=True, 825 | release_date=datetime.date(2016, 4, 21), 826 | series='xenial', 827 | version=decimal.Decimal('16.04'), 828 | ), 829 | Release( 830 | codename='Yakkety Yak', 831 | created_date=datetime.date(2016, 4, 21), 832 | distributor_id='ubuntu', 833 | eol_date=datetime.date(2017, 7, 20), 834 | is_lts=False, 835 | release_date=datetime.date(2016, 10, 13), 836 | series='yakkety', 837 | version=decimal.Decimal('16.10'), 838 | ), 839 | Release( 840 | codename='Zesty Zapus', 841 | created_date=datetime.date(2016, 10, 13), 842 | distributor_id='ubuntu', 843 | eol_date=datetime.date(2018, 1, 13), 844 | is_lts=False, 845 | release_date=datetime.date(2017, 4, 13), 846 | series='zesty', 847 | version=decimal.Decimal('17.04'), 848 | ), 849 | Release( 850 | codename='Artful Aardvark', 851 | created_date=datetime.date(2017, 4, 13), 852 | distributor_id='ubuntu', 853 | eol_date=datetime.date(2018, 7, 19), 854 | is_lts=False, 855 | release_date=datetime.date(2017, 10, 19), 856 | series='artful', 857 | version=decimal.Decimal('17.10'), 858 | ), 859 | Release( 860 | codename='Bionic Beaver', 861 | created_date=datetime.date(2017, 10, 19), 862 | distributor_id='ubuntu', 863 | eol_date=datetime.date(2023, 4, 26), 864 | extended_eol_date=datetime.date(2023, 4, 26), 865 | is_lts=True, 866 | release_date=datetime.date(2018, 4, 26), 867 | series='bionic', 868 | version=decimal.Decimal('18.04'), 869 | ), 870 | Release( 871 | codename='Cosmic Cuttlefish', 872 | created_date=datetime.date(2018, 4, 26), 873 | distributor_id='ubuntu', 874 | eol_date=datetime.date(2019, 7, 18), 875 | is_lts=False, 876 | release_date=datetime.date(2018, 10, 18), 877 | series='cosmic', 878 | version=decimal.Decimal('18.10'), 879 | ), 880 | Release( 881 | codename='Disco Dingo', 882 | created_date=datetime.date(2018, 10, 18), 883 | distributor_id='ubuntu', 884 | eol_date=datetime.date(2020, 1, 18), 885 | is_lts=False, 886 | release_date=datetime.date(2019, 4, 18), 887 | series='disco', 888 | version=decimal.Decimal('19.04'), 889 | ), 890 | Release( 891 | codename='Eoan Ermine', 892 | created_date=datetime.date(2019, 4, 18), 893 | distributor_id='ubuntu', 894 | eol_date=datetime.date(2020, 7, 17), 895 | is_lts=False, 896 | release_date=datetime.date(2019, 10, 17), 897 | series='eoan', 898 | version=decimal.Decimal('19.10'), 899 | ), 900 | Release( 901 | codename='Focal Fossa', 902 | created_date=datetime.date(2019, 10, 17), 903 | distributor_id='ubuntu', 904 | eol_date=datetime.date(2025, 4, 23), 905 | extended_eol_date=datetime.date(2025, 4, 23), 906 | is_lts=True, 907 | release_date=datetime.date(2020, 4, 23), 908 | series='focal', 909 | version=decimal.Decimal('20.04'), 910 | ), 911 | Release( 912 | codename='Groovy Gorilla', 913 | created_date=datetime.date(2020, 4, 23), 914 | distributor_id='ubuntu', 915 | eol_date=datetime.date(2021, 7, 22), 916 | is_lts=False, 917 | release_date=datetime.date(2020, 10, 22), 918 | series='groovy', 919 | version=decimal.Decimal('20.10'), 920 | ), 921 | Release( 922 | codename='Hirsute Hippo', 923 | created_date=datetime.date(2020, 10, 22), 924 | distributor_id='ubuntu', 925 | eol_date=datetime.date(2022, 1, 20), 926 | is_lts=False, 927 | release_date=datetime.date(2021, 4, 22), 928 | series='hirsute', 929 | version=decimal.Decimal('21.04'), 930 | ), 931 | Release( 932 | codename='Impish Indri', 933 | created_date=datetime.date(2021, 4, 22), 934 | distributor_id='ubuntu', 935 | eol_date=datetime.date(2022, 7, 14), 936 | is_lts=False, 937 | release_date=datetime.date(2021, 10, 14), 938 | series='impish', 939 | version=decimal.Decimal('21.10'), 940 | ), 941 | ] 942 | 943 | # [[[end]]] 944 | -------------------------------------------------------------------------------- /apt_mirror_updater/tests.py: -------------------------------------------------------------------------------- 1 | # Automated, robust apt-get mirror selection for Debian and Ubuntu. 2 | # 3 | # Author: Peter Odding 4 | # Last Change: April 15, 2020 5 | # URL: https://apt-mirror-updater.readthedocs.io 6 | 7 | """Test suite for the ``apt-mirror-updater`` package.""" 8 | 9 | # Standard library modules. 10 | import decimal 11 | import logging 12 | import os 13 | import time 14 | 15 | # External dependencies. 16 | from executor import execute 17 | from executor.contexts import LocalContext 18 | from humanfriendly.testing import TestCase, run_cli 19 | from stopit import TimeoutException 20 | 21 | # Modules included in our package. 22 | from apt_mirror_updater import AptMirrorUpdater, normalize_mirror_url 23 | from apt_mirror_updater.cli import main 24 | from apt_mirror_updater.http import fetch_url 25 | from apt_mirror_updater.releases import ( 26 | DEBIAN_KEYRING_CURRENT, 27 | UBUNTU_KEYRING_CURRENT, 28 | UBUNTU_KEYRING_REMOVED, 29 | coerce_release, 30 | discover_releases, 31 | ubuntu_keyring_updated, 32 | ) 33 | 34 | # Initialize a logger for this module. 35 | logger = logging.getLogger(__name__) 36 | 37 | 38 | class AptMirrorUpdaterTestCase(TestCase): 39 | 40 | """:mod:`unittest` compatible container for the :mod:`apt_mirror_updater` test suite.""" 41 | 42 | def check_debian_mirror(self, url): 43 | """Ensure the given URL looks like a Debian mirror URL.""" 44 | if not self.is_debian_mirror(url): 45 | msg = "Invalid Debian mirror URL! (%r)" 46 | raise AssertionError(msg % url) 47 | 48 | def check_mirror_url(self, url): 49 | """Check whether the given URL looks like a mirror URL for the system running the test suite.""" 50 | if not hasattr(self, 'context'): 51 | self.context = LocalContext() 52 | if self.context.distributor_id == 'debian': 53 | self.check_debian_mirror(url) 54 | elif self.context.distributor_id == 'ubuntu': 55 | self.check_ubuntu_mirror(url) 56 | else: 57 | raise Exception("Unsupported platform!") 58 | 59 | def check_ubuntu_mirror(self, url): 60 | """Ensure the given URL looks like a Ubuntu mirror URL.""" 61 | if not self.is_ubuntu_mirror(url): 62 | msg = "Invalid Ubuntu mirror URL! (%r)" 63 | raise AssertionError(msg % url) 64 | 65 | def is_debian_mirror(self, url): 66 | """Check whether the given URL looks like a Debian mirror URL.""" 67 | return self.is_mirror_url(url, '/dists/stable/Release.gpg', b'-----BEGIN PGP SIGNATURE-----') 68 | 69 | def is_mirror_url(self, base_url, stable_resource, expected_content): 70 | """Validate a given mirror URL based on a stable resource URL and its expected response.""" 71 | base_url = normalize_mirror_url(base_url) 72 | if base_url.startswith(('http://', 'https://')): 73 | if not hasattr(self, 'mirror_cache'): 74 | self.mirror_cache = {} 75 | cache_key = (base_url, stable_resource, expected_content) 76 | if cache_key not in self.mirror_cache: 77 | try: 78 | # Look for a file with a stable filename (assumed to always be available). 79 | resource_url = base_url + stable_resource 80 | response = fetch_url(resource_url) 81 | # Check the contents of the response. 82 | if expected_content in response: 83 | logger.info("URL %s served expected content.", resource_url) 84 | self.mirror_cache[cache_key] = True 85 | else: 86 | logger.warning("URL %s didn't serve expected content!", resource_url) 87 | self.mirror_cache[cache_key] = False 88 | except TimeoutException: 89 | logger.warning("URL %s reported timeout, not failing test suite on this ..") 90 | self.mirror_cache[cache_key] = True 91 | except Exception: 92 | logger.warning("URL %s triggered exception!", resource_url, exc_info=True) 93 | self.mirror_cache[cache_key] = False 94 | return self.mirror_cache[cache_key] 95 | return False 96 | 97 | def is_ubuntu_mirror(self, url): 98 | """ 99 | Check whether the given URL looks like a Ubuntu mirror URL. 100 | 101 | This is a bit convoluted because different mirrors forbid access to 102 | different resources (resulting in HTTP 403 responses) apparently based 103 | on individual webmaster's perceptions of what expected clients 104 | (apt-get) should and shouldn't be accessing :-). 105 | """ 106 | if url == 'http://ubuntu.cs.utah.edu/ubuntu': 107 | # This mirror intermittently serves 404 errors on arbitrary URLs. 108 | # Apart from that it does look to contain the expected directory 109 | # layout. Seems like they're load balancing between good and bad 110 | # servers (where the bad servers have a broken configuration). 111 | return True 112 | # At the time of writing the following test seems to work on all 113 | # mirrors apart from the exceptions noted in this method. 114 | if self.is_mirror_url(url, '/project/ubuntu-archive-keyring.gpg', b'ftpmaster@ubuntu.com'): 115 | return True 116 | # The mirror http://mirrors.codec-cluster.org/ubuntu fails the above 117 | # test because of a 403 response so we have to compensate. Because 118 | # other mirrors may behave similarly in the future this is implemented 119 | # as a generic test (not based on the mirror URL). 120 | return self.is_mirror_url(url, '/dists/devel/Release.gpg', b'-----BEGIN PGP SIGNATURE-----') 121 | 122 | def test_debian_mirror_discovery(self): 123 | """Test the discovery of Debian mirror URLs.""" 124 | from apt_mirror_updater.backends.debian import discover_mirrors 125 | mirrors = discover_mirrors() 126 | assert len(mirrors) > 10 127 | for candidate in mirrors: 128 | self.check_debian_mirror(candidate.mirror_url) 129 | 130 | def test_ubuntu_mirror_discovery(self): 131 | """Test the discovery of Ubuntu mirror URLs.""" 132 | from apt_mirror_updater.backends.ubuntu import discover_mirrors 133 | mirrors = discover_mirrors() 134 | assert len(mirrors) > 10 135 | for candidate in mirrors: 136 | self.check_ubuntu_mirror(candidate.mirror_url) 137 | 138 | def test_adaptive_mirror_discovery(self): 139 | """Test the discovery of mirrors for the current type of system.""" 140 | updater = AptMirrorUpdater() 141 | assert len(updater.available_mirrors) > 10 142 | for candidate in updater.available_mirrors: 143 | self.check_mirror_url(candidate.mirror_url) 144 | 145 | def test_mirror_ranking(self): 146 | """Test the ranking of discovered mirrors.""" 147 | updater = AptMirrorUpdater() 148 | # Make sure that multiple discovered mirrors are available. 149 | assert sum(m.is_available for m in updater.ranked_mirrors) > 10 150 | 151 | def test_best_mirror_selection(self): 152 | """Test the selection of a "best" mirror.""" 153 | updater = AptMirrorUpdater() 154 | self.check_mirror_url(updater.best_mirror) 155 | 156 | def test_current_mirror_discovery(self): 157 | """Test that the current mirror can be extracted from ``/etc/apt/sources.list``.""" 158 | exit_code, output = run_cli(main, '--find-current-mirror') 159 | assert exit_code == 0 160 | self.check_mirror_url(output.strip()) 161 | 162 | def test_dumb_update(self): 163 | """Test that our dumb ``apt-get update`` wrapper works.""" 164 | if os.getuid() != 0: 165 | return self.skipTest("root privileges required to opt in") 166 | updater = AptMirrorUpdater() 167 | # Remove all existing package lists. 168 | updater.clear_package_lists() 169 | # Verify that package lists aren't available. 170 | assert not have_package_lists() 171 | # Run `apt-get update' to download the package lists. 172 | updater.dumb_update() 173 | # Verify that package lists are again available. 174 | assert have_package_lists() 175 | 176 | def test_smart_update(self): 177 | """ 178 | Test that our smart ``apt-get update`` wrapper works. 179 | 180 | Currently this test simply ensures coverage of the happy path. 181 | Ideally it will evolve to test the handled edge cases as well. 182 | """ 183 | if os.getuid() != 0: 184 | return self.skipTest("root privileges required to opt in") 185 | updater = AptMirrorUpdater() 186 | # Remove all existing package lists. 187 | updater.clear_package_lists() 188 | # Verify that package lists aren't available. 189 | assert not have_package_lists() 190 | # Run `apt-get update' to download the package lists. 191 | updater.smart_update() 192 | # Verify that package lists are again available. 193 | assert have_package_lists() 194 | 195 | def test_discover_releases(self): 196 | """Test that release discovery works properly.""" 197 | releases = discover_releases() 198 | # Check that a reasonable number of Debian and Ubuntu releases was discovered. 199 | assert len([r for r in releases if r.distributor_id == 'debian']) > 10 200 | assert len([r for r in releases if r.distributor_id == 'ubuntu']) > 10 201 | # Check that LTS releases of Debian as well as Ubuntu were discovered. 202 | assert any(r.distributor_id == 'debian' and r.is_lts for r in releases) 203 | assert any(r.distributor_id == 'ubuntu' and r.is_lts for r in releases) 204 | # Sanity check against duplicate releases. 205 | assert sum(r.series == 'bionic' for r in releases) == 1 206 | assert sum(r.series == 'jessie' for r in releases) == 1 207 | # Sanity check some known LTS releases. 208 | assert any(r.series == 'bionic' and r.is_lts for r in releases) 209 | assert any(r.series == 'stretch' and r.is_lts for r in releases) 210 | 211 | def test_coerce_release(self): 212 | """Test the coercion of release objects.""" 213 | # Test coercion of short code names. 214 | assert coerce_release('lucid').version == decimal.Decimal('10.04') 215 | assert coerce_release('woody').distributor_id == 'debian' 216 | # Test coercion of version numbers. 217 | assert coerce_release('10.04').series == 'lucid' 218 | 219 | def test_keyring_selection(self): 220 | """Make sure keyring selection works as intended.""" 221 | # Check Debian keyring selection. 222 | lenny = coerce_release('lenny') 223 | assert lenny.keyring_file == DEBIAN_KEYRING_CURRENT 224 | # Check Ubuntu <= 12.04 keyring selection. 225 | precise = coerce_release('precise') 226 | if ubuntu_keyring_updated(): 227 | assert precise.keyring_file == UBUNTU_KEYRING_REMOVED 228 | else: 229 | assert precise.keyring_file == UBUNTU_KEYRING_CURRENT 230 | # Check Ubuntu > 12.04 keyring selection. 231 | bionic = coerce_release('bionic') 232 | assert bionic.keyring_file == UBUNTU_KEYRING_CURRENT 233 | 234 | def test_debian_lts_eol_date(self): 235 | """ 236 | Regression test for `issue #5`_. 237 | 238 | .. _issue #5: https://github.com/xolox/python-apt-mirror-updater/issues/5 239 | """ 240 | updater = AptMirrorUpdater( 241 | distributor_id='debian', 242 | distribution_codename='jessie', 243 | architecture='amd64', 244 | ) 245 | eol_expected = (time.time() >= 1593468000) 246 | assert updater.release_is_eol == eol_expected 247 | 248 | 249 | def have_package_lists(): 250 | """ 251 | Check if apt's package lists are available. 252 | 253 | :returns: :data:`True` when package lists are available, 254 | :data:`False` otherwise. 255 | 256 | This function checks that the output of ``apt-cache show python`` contains 257 | a ``Filename: ...`` key/value pair which indicates that apt knows where to 258 | download the package archive that installs the ``python`` package. 259 | """ 260 | return 'Filename:' in execute('apt-cache', 'show', 'python', check=False, capture=True) 261 | -------------------------------------------------------------------------------- /docs/api.rst: -------------------------------------------------------------------------------- 1 | API documentation 2 | ================= 3 | 4 | The following documentation is based on the source code of version |release| of 5 | the `apt-mirror-updater` package. The following modules are available: 6 | 7 | .. contents:: 8 | :local: 9 | 10 | :mod:`apt_mirror_updater` 11 | ------------------------- 12 | 13 | .. automodule:: apt_mirror_updater 14 | :members: 15 | 16 | :mod:`apt_mirror_updater.backends.debian` 17 | ----------------------------------------- 18 | 19 | .. automodule:: apt_mirror_updater.backends.debian 20 | :members: 21 | 22 | :mod:`apt_mirror_updater.backends.elementary` 23 | --------------------------------------------- 24 | 25 | .. automodule:: apt_mirror_updater.backends.elementary 26 | :members: 27 | 28 | :mod:`apt_mirror_updater.backends.ubuntu` 29 | ----------------------------------------- 30 | 31 | .. automodule:: apt_mirror_updater.backends.ubuntu 32 | :members: 33 | 34 | :mod:`apt_mirror_updater.cli` 35 | ----------------------------- 36 | 37 | .. automodule:: apt_mirror_updater.cli 38 | :members: 39 | 40 | :mod:`apt_mirror_updater.http` 41 | ------------------------------ 42 | 43 | .. automodule:: apt_mirror_updater.http 44 | :members: 45 | 46 | :mod:`apt_mirror_updater.releases` 47 | ---------------------------------- 48 | 49 | .. automodule:: apt_mirror_updater.releases 50 | :members: 51 | -------------------------------------------------------------------------------- /docs/changelog.rst: -------------------------------------------------------------------------------- 1 | .. include:: ../CHANGELOG.rst 2 | -------------------------------------------------------------------------------- /docs/conf.py: -------------------------------------------------------------------------------- 1 | # Automated, robust apt-get mirror selection for Debian and Ubuntu. 2 | # 3 | # Author: Peter Odding 4 | # Last Change: April 15, 2020 5 | # URL: https://apt-mirror-updater.readthedocs.io 6 | 7 | """Sphinx documentation configuration for the `apt-mirror-updater` package.""" 8 | 9 | import os 10 | import sys 11 | 12 | # Add the 'apt-mirror-updater' source distribution's root directory to the module path. 13 | sys.path.insert(0, os.path.abspath(os.pardir)) 14 | 15 | # -- General configuration ----------------------------------------------------- 16 | 17 | # Sphinx extension module names. 18 | extensions = [ 19 | 'sphinx.ext.autodoc', 20 | 'sphinx.ext.intersphinx', 21 | 'sphinx.ext.viewcode', 22 | 'humanfriendly.sphinx', 23 | ] 24 | 25 | # Sort members by the source order instead of alphabetically. 26 | autodoc_member_order = 'bysource' 27 | 28 | # Paths that contain templates, relative to this directory. 29 | templates_path = ['templates'] 30 | 31 | # The suffix of source filenames. 32 | source_suffix = '.rst' 33 | 34 | # The master toctree document. 35 | master_doc = 'index' 36 | 37 | # General information about the project. 38 | project = 'apt-mirror-updater' 39 | copyright = '2020, Peter Odding' 40 | 41 | # The version info for the project you're documenting, acts as replacement for 42 | # |version| and |release|, also used in various other places throughout the 43 | # built documents. 44 | 45 | # Find the package version and make it the release. 46 | from apt_mirror_updater import __version__ as updater_version # noqa 47 | 48 | # The short X.Y version. 49 | version = '.'.join(updater_version.split('.')[:2]) 50 | 51 | # The full version, including alpha/beta/rc tags. 52 | release = updater_version 53 | 54 | # The language for content autogenerated by Sphinx. Refer to documentation 55 | # for a list of supported languages. 56 | language = 'en' 57 | 58 | # List of patterns, relative to source directory, that match files and 59 | # directories to ignore when looking for source files. 60 | exclude_patterns = ['build'] 61 | 62 | # If true, '()' will be appended to :func: etc. cross-reference text. 63 | add_function_parentheses = True 64 | 65 | # The name of the Pygments (syntax highlighting) style to use. 66 | pygments_style = 'sphinx' 67 | 68 | # Refer to the Python standard library. 69 | # From: http://twistedmatrix.com/trac/ticket/4582. 70 | intersphinx_mapping = dict( 71 | python2=('https://docs.python.org/2', None), 72 | python3=('https://docs.python.org/3', None), 73 | executor=('https://executor.readthedocs.io/en/latest/', None), 74 | propertymanager=('https://property-manager.readthedocs.io/en/latest/', None), 75 | ) 76 | 77 | # -- Options for HTML output --------------------------------------------------- 78 | 79 | # The theme to use for HTML and HTML Help pages. See the documentation for 80 | # a list of builtin themes. 81 | html_theme = 'nature' 82 | 83 | # Output file base name for HTML help builder. 84 | htmlhelp_basename = 'aptmirrorupdaterdoc' 85 | -------------------------------------------------------------------------------- /docs/index.rst: -------------------------------------------------------------------------------- 1 | apt-mirror-updater: Automated Debian/Ubuntu mirror selection 2 | ============================================================ 3 | 4 | Welcome to the documentation of `apt-mirror-updater` version |release|! 5 | The following sections are available: 6 | 7 | .. contents:: 8 | :local: 9 | 10 | User documentation 11 | ------------------ 12 | 13 | The readme is the best place to start reading, it's targeted at all users and 14 | documents the command line interface: 15 | 16 | .. toctree:: 17 | readme.rst 18 | 19 | API documentation 20 | ----------------- 21 | 22 | The following API documentation is automatically generated from the source code: 23 | 24 | .. toctree:: 25 | api.rst 26 | 27 | Change log 28 | ---------- 29 | 30 | The change log lists notable changes to the project: 31 | 32 | .. toctree:: 33 | changelog.rst 34 | -------------------------------------------------------------------------------- /docs/readme.rst: -------------------------------------------------------------------------------- 1 | .. include:: ../README.rst 2 | -------------------------------------------------------------------------------- /requirements-checks.txt: -------------------------------------------------------------------------------- 1 | # Python packages required to run `make check'. 2 | flake8 >= 2.6.0 3 | flake8-docstrings >= 0.2.8 4 | pyflakes >= 1.2.3 5 | -------------------------------------------------------------------------------- /requirements-tests.txt: -------------------------------------------------------------------------------- 1 | pytest >= 2.6.1 2 | pytest-cov >= 2.2.1 3 | -------------------------------------------------------------------------------- /requirements-travis.txt: -------------------------------------------------------------------------------- 1 | --requirement=requirements-checks.txt 2 | --requirement=requirements-tests.txt 3 | --requirement=requirements.txt 4 | coveralls 5 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | # Installation requirements for apt-mirror-updater. 2 | # 3 | # NB: Release 4.9.3 of beautifulsoup4 is the last version to support Python 2. 4 | 5 | beautifulsoup4 >= 4.9.3 6 | capturer >= 3.0 7 | coloredlogs >= 10.0 8 | executor >= 21.3 9 | flufl.enum >= 4.0.1 10 | humanfriendly >= 8.1 11 | property-manager >= 3.0 12 | six >= 1.10.0 13 | stopit >= 1.1.1 14 | -------------------------------------------------------------------------------- /scripts/collect-full-coverage.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash -e 2 | 3 | # This shell script is used by the makefile and Travis CI to run the 4 | # apt-mirror-updater test suite as root, allowing it to make changes 5 | # to the system that's running the test suite (one of my laptops 6 | # or a Travis CI worker). 7 | 8 | # Run the test suite with root privileges. 9 | sudo $(which py.test) --cov 10 | 11 | # Restore the ownership of the coverage data. 12 | sudo chown --reference="$PWD" --recursive $PWD 13 | 14 | # Update the HTML coverage overview. 15 | coverage html 16 | -------------------------------------------------------------------------------- /scripts/install-on-travis.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash -e 2 | 3 | # Install the required Python packages. 4 | pip install --requirement=requirements-travis.txt 5 | 6 | # Install the project itself, making sure that potential character encoding 7 | # and/or decoding errors in the setup script are caught as soon as possible. 8 | LC_ALL=C pip install . 9 | 10 | # Let apt-get, dpkg and related tools know that we want the following 11 | # commands to be 100% automated (no interactive prompts). 12 | export DEBIAN_FRONTEND=noninteractive 13 | 14 | # Update apt-get's package lists. 15 | sudo -E apt-get update -qq 16 | 17 | # Make sure the /usr/share/distro-info/*.csv files are available, 18 | # this enables the test_gather_eol_dates() test. 19 | sudo -E apt-get install --yes distro-info-data 20 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | # Enable universal wheels because `apt-mirror-updater' 2 | # is pure Python and works on Python 2 and 3 alike. 3 | 4 | [wheel] 5 | universal=1 6 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | # Setup script for the `apt-mirror-updater' package. 4 | # 5 | # Author: Peter Odding 6 | # Last Change: April 15, 2020 7 | # URL: https://apt-mirror-updater.readthedocs.io 8 | 9 | """ 10 | Setup script for the `apt-mirror-updater` package. 11 | 12 | **python setup.py install** 13 | Install from the working directory into the current Python environment. 14 | 15 | **python setup.py sdist** 16 | Build a source distribution archive. 17 | 18 | **python setup.py bdist_wheel** 19 | Build a wheel distribution archive. 20 | """ 21 | 22 | # Standard library modules. 23 | import codecs 24 | import os 25 | import re 26 | 27 | # De-facto standard solution for Python packaging. 28 | from setuptools import find_packages, setup 29 | 30 | 31 | def get_contents(*args): 32 | """Get the contents of a file relative to the source distribution directory.""" 33 | with codecs.open(get_absolute_path(*args), 'r', 'UTF-8') as handle: 34 | return handle.read() 35 | 36 | 37 | def get_version(*args): 38 | """Extract the version number from a Python module.""" 39 | contents = get_contents(*args) 40 | metadata = dict(re.findall('__([a-z]+)__ = [\'"]([^\'"]+)', contents)) 41 | return metadata['version'] 42 | 43 | 44 | def get_requirements(*args): 45 | """Get requirements from pip requirement files.""" 46 | requirements = set() 47 | with open(get_absolute_path(*args)) as handle: 48 | for line in handle: 49 | # Strip comments. 50 | line = re.sub(r'^#.*|\s#.*', '', line) 51 | # Ignore empty lines 52 | if line and not line.isspace(): 53 | requirements.add(re.sub(r'\s+', '', line)) 54 | return sorted(requirements) 55 | 56 | 57 | def get_absolute_path(*args): 58 | """Transform relative pathnames into absolute pathnames.""" 59 | return os.path.join(os.path.dirname(os.path.abspath(__file__)), *args) 60 | 61 | 62 | setup( 63 | name='apt-mirror-updater', 64 | version=get_version('apt_mirror_updater', '__init__.py'), 65 | description="Automated, robust apt-get mirror selection for Debian and Ubuntu", 66 | long_description=get_contents('README.rst'), 67 | url='https://apt-mirror-updater.readthedocs.io', 68 | author='Peter Odding', 69 | author_email='peter@peterodding.com', 70 | license='MIT', 71 | packages=find_packages(), 72 | install_requires=get_requirements('requirements.txt'), 73 | entry_points=dict(console_scripts=[ 74 | 'apt-mirror-updater = apt_mirror_updater.cli:main', 75 | ]), 76 | python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*', 77 | classifiers=[ 78 | 'Development Status :: 4 - Beta', 79 | 'Environment :: Console', 80 | 'Intended Audience :: Developers', 81 | 'Intended Audience :: Information Technology', 82 | 'Intended Audience :: System Administrators', 83 | 'License :: OSI Approved :: MIT License', 84 | 'Natural Language :: English', 85 | 'Operating System :: POSIX :: Linux', 86 | 'Programming Language :: Python', 87 | 'Programming Language :: Python :: 2', 88 | 'Programming Language :: Python :: 2.7', 89 | 'Programming Language :: Python :: 3', 90 | 'Programming Language :: Python :: 3.5', 91 | 'Programming Language :: Python :: 3.6', 92 | 'Programming Language :: Python :: 3.7', 93 | 'Programming Language :: Python :: 3.8', 94 | 'Programming Language :: Python :: Implementation :: CPython', 95 | 'Programming Language :: Python :: Implementation :: PyPy', 96 | 'Topic :: Software Development', 97 | 'Topic :: Software Development :: Libraries :: Python Modules', 98 | 'Topic :: System :: Shells', 99 | 'Topic :: System :: System Shells', 100 | 'Topic :: System :: Systems Administration', 101 | 'Topic :: Terminals', 102 | 'Topic :: Utilities', 103 | ]) 104 | -------------------------------------------------------------------------------- /tox.ini: -------------------------------------------------------------------------------- 1 | [tox] 2 | envlist = py27, py35, py36, py37, py38, pypy 3 | 4 | [testenv] 5 | deps = -rrequirements-tests.txt 6 | commands = py.test {posargs} 7 | 8 | [pytest] 9 | addopts = --verbose 10 | python_files = apt_mirror_updater/tests.py 11 | 12 | [flake8] 13 | exclude = .tox 14 | extend-ignore = D211,D400,D401,D402 15 | max-line-length = 120 16 | --------------------------------------------------------------------------------