├── .gitignore ├── LICENSE.rst ├── README.md ├── appveyor ├── README.md ├── install-miniconda.ps1 ├── pinned └── windows_sdk.cmd ├── conda_environment.yml ├── pip_pinnings.txt ├── test_env.py ├── tox.ini ├── travis ├── hack_version_numbers.py ├── setup_conda.sh ├── setup_conda_linux.sh ├── setup_conda_osx.sh ├── setup_conda_windows.sh ├── setup_dependencies_common.sh └── setup_python.sh └── utils └── import_submodules.py /.gitignore: -------------------------------------------------------------------------------- 1 | # Ignore py.test generated dirs 2 | .cache/ 3 | __pycache__ -------------------------------------------------------------------------------- /LICENSE.rst: -------------------------------------------------------------------------------- 1 | Copyright (c) 2015-2017, Brigitta M. Sipocz & Thomas P. Robitaille 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without modification, 5 | are permitted provided that the following conditions are met: 6 | 7 | * Redistributions of source code must retain the above copyright notice, this 8 | list of conditions and the following disclaimer. 9 | * Redistributions in binary form must reproduce the above copyright notice, this 10 | list of conditions and the following disclaimer in the documentation and/or 11 | other materials provided with the distribution. 12 | * Neither the name of the Astropy Team nor the names of its contributors may be 13 | used to endorse or promote products derived from this software without 14 | specific prior written permission. 15 | 16 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 17 | ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 18 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 19 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR 20 | ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 21 | (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 22 | LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON 23 | ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 24 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 25 | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 26 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## Important notices 2 | 3 | This package is no longer actively developed and has been archived. Please read on below for service-specific 4 | information. 5 | 6 | ### Travis CI 7 | 8 | Astropy Project has decided to move away from Travis CI after they dropped support 9 | for OSS by removing their free-tier plan. Please see 10 | [this announcement](https://groups.google.com/g/astropy-dev/c/45frnTEAX-U) 11 | for more information. We recommend switching to GitHub Actions instead; 12 | you can see an example of it on [astropy](https://github.com/astropy/astropy) 13 | and on [package template](https://github.com/astropy/package-template). 14 | 15 | ### Appveyor 16 | 17 | Scripts for ``appveyor.yml`` for the [AppVeyor](https://www.appveyor.com/) 18 | service are no longer supported. Please use the 19 | [Windows build on Travis](https://docs.travis-ci.com/user/reference/windows/) 20 | instead. 21 | 22 | For the usage of the deprecated scripts see [Appveyor scripts README](https://github.com/astropy/ci-helpers/blob/main/appveyor/README.md). 23 | 24 | ## About 25 | 26 | This repository contains a set of scripts that are used by the 27 | ``.travis.yml`` file of Astropy packages for the 28 | [Travis](https://travis-ci.com) service. 29 | 30 | The idea is to clone these at the last minute when the continuous 31 | integration is about to be run. This is better than including this 32 | repository as a Git sub-module, because this allows updates to this repository 33 | to take effect immediately, and not have to update the Git sub-module every time 34 | a change is made. 35 | 36 | ## How to use 37 | 38 | ### Travis (with conda) 39 | 40 | *Note that you can also set up Python without conda using ci-helpers - see [here](#setting-up-python-without-conda-on-travis) for more details* 41 | 42 | Include the following lines at the start of the ``install`` section in 43 | ``.travis.yml``: 44 | 45 | ```yaml 46 | install: 47 | - git clone --depth 1 git://github.com/astropy/ci-helpers.git 48 | - source ci-helpers/travis/setup_conda.sh 49 | ``` 50 | 51 | This does the following: 52 | 53 | - Set up Miniconda. 54 | - Set up the PATH appropriately. 55 | - Set up a conda environment named 'test' and switch to it. 56 | - Set the ``always_yes`` config option for conda to ``true`` so that you don't 57 | need to include ``--yes``. 58 | - Register the specified channels. 59 | - ``export PYTHONIOENCODING=UTF8`` 60 | - Supports custom skip tags included in the commit message that are not yet 61 | natively provided by Travis. 62 | To run only the docs build: ``[build docs]`` or 63 | ``[docs only]``. The latter requires ``SETUP_CMD`` (see below) to be set to 64 | ``build_docs`` or ``build_sphinx``. 65 | 66 | Following this, various dependencies are installed depending on the following 67 | environment variables 68 | 69 | * ``MAIN_CMD``: if this starts with ``pycodestyle``, ``flake``, or 70 | ``pylint`` then the only package that gets installed is the 71 | ``pycodestyle``, ``flake``, or ``pylint`` package. Please note that the 72 | former name of the ``pycodestyle`` package is ``pep8``, and ci-helpers 73 | still accepts it, too. 74 | 75 | * ``SETUP_CMD``: this can be set to various values: 76 | 77 | * ``egg_info``: no dependencies are installed once the conda environment 78 | has been created and any other environment variables are ignored. 79 | 80 | * ``build_docs`` or ``build_sphinx``: the Sphinx and matplotlib packages 81 | are installed in addition to other packages that might be requested 82 | via other environment variables. 83 | 84 | * ``test``: runs the test suite after the dependencies are installed. 85 | 86 | In addition, if ``SETUP_CMD`` contains the following flags, extra dependencies 87 | are installed: 88 | 89 | * ``--coverage``: the coverage, coveralls, and codecov packages are installed 90 | * ``-cov``: the pytest-cov, coveralls, and codecov packages are installed 91 | * ``--parallel`` or ``--numprocesses``: the pytest-xdist package is 92 | installed 93 | * ``--open-files``: the psutil package is installed 94 | 95 | * ``MAMBA``: if set to ``True``, conda packages 96 | will be installed with `mamba `_, which is 97 | both faster than conda and gives more readable errors in cases where there are 98 | conflicts. 99 | 100 | * ``NUMPY_VERSION``: if set to ``dev`` or ``development``, the latest 101 | developer version of Numpy is installed along with Cython. If set to a 102 | version number, that version is installed. If set to ``stable``, install 103 | the latest stable version of Numpy. If set to ``prerelease``, the 104 | pre-release version of Numpy gets installed if there is any, otherwise the 105 | build exits and passes on Travis without running the tests. We try to 106 | avoid downloading and installing mkl, so unless ``mkl`` is specified as a 107 | dependency in ``CONDA_DEPENDENCIES``, ``nomkl`` is used. On Windows the 108 | is only MKL, so while the ``nomkl`` package exists it does nothing, 109 | ``mkl`` is always needed to be installed. 110 | 111 | * ``ASTROPY_VERSION``: if set to ``dev`` or ``development``, the latest 112 | developer version of Astropy is installed, along with Cython and jinja2, 113 | which are compile-time dependencies. If set to a version number, that 114 | version is installed. If set to ``stable``, install the latest stable 115 | version of Astropy. If set to ``prerelease``, the pre-release version of 116 | Astropy gets installed if there is any, otherwise the build exits and 117 | passes on Travis without running the tests. If set to ``lts`` the latest 118 | long term support (LTS) version is installed (more info about LTS can be 119 | found 120 | [here](https://github.com/astropy/astropy-APEs/blob/main/APE2.rst#version-numbering). 121 | 122 | * ``SUNPY_VERSION``: if set to ``dev`` or ``development``, the latest 123 | developer version of Sunpy is installed. If set to a 124 | version number, that version is installed. If set to ``stable``, install 125 | the latest stable version of Sunpy. If set to ``prerelease``, the 126 | pre-release version of Sunpy gets installed if there is any, otherwise the 127 | build exits and passes on Travis without running the tests. 128 | 129 | * ``MINICONDA_VERSION``: This sets the version of Miniconda that will be 130 | installed. Use this to override a pinned version if necessary. 131 | 132 | * ``CONDA_DEPENDENCIES``: this should be a space-separated string of 133 | package names that will be installed with conda. Version numbers of these 134 | dependencies can be overridden/specified with the ``PACKAGENAME_VERSION`` 135 | environment variables. 136 | 137 | * ``PIP_DEPENDENCIES``: this should be a space-separated string of package 138 | names that will be installed with pip. 139 | 140 | * ``CONDA_DEPENDENCIES_FLAGS``: additional flags to pass to conda when 141 | installing ``CONDA_DEPENDENCIES`` 142 | 143 | * ``PIP_DEPENDENCIES_FLAGS``: additional flags to pass to pip when 144 | installing ``PIP_DEPENDENCIES`` 145 | 146 | * ``CONDA_CHANNELS``: this should be a space-separated string of conda 147 | channel names. We don't add any channel by default. 148 | 149 | * ``CONDA_ENVIRONMENT``: this is a path to a file that should be used with 150 | ``conda env create -f $CONDA_ENVIRONMENT``. This is applied to set up the 151 | test environment *before* the conda and pip dependencies (which otherwise 152 | act additively with this option). 153 | 154 | * ``DEBUG``: if `True` this turns on the shell debug mode in the install 155 | scripts, and provides information on the current conda install and 156 | switches off the ``-q`` conda flag for verbose output. 157 | 158 | * ``SETUP_XVFB``: if True this makes sure e.g., interactive matplotlib 159 | backends work by starting up a X virtual framebuffer. 160 | 161 | * ``MPLBACKEND``: If not specified it is set to ``Agg`` as the default backend. 162 | 163 | * ``PACKAGENAME_VERSION``: ``PACKAGENAME`` is the name of the package to 164 | specify the version for (e.g. ``MATPLOTLIB_VERSION``). Due to shell 165 | limitations, all hyphens in the conda package name should be changed to 166 | underscores in ``PACKAGENAME_VERSION`` (e.g. for scikit-image it should 167 | be ``SCIKIT_IMAGE_VERSION``). If specified it will override any version 168 | number limitations listed in ``CONDA_DEPENDENCIES``. 169 | 170 | * ``CONDA_CHANNEL_PRIORITY``: can be set to ``strict``, ``flexible`` or ``disabled``, and 171 | affects the ``channel_priority`` conda setting (as discussed 172 | [here](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-channels.html). The default is 173 | ``disabled``. 174 | 175 | * ``EVENT_TYPE``: this should be a space-separated string of event 176 | types. If given, the build will run only if the ``TRAVIS_EVENT_TYPE`` 177 | matches with any of the listed ones. Otherwise the build exits and passes 178 | on Travis without running the tests. This is a way to control builds to 179 | run only on pushes to main, or for Travis cron jobs. Valid event types 180 | are: ``push``, ``pull_request``, ``api`` or ``cron``. 181 | 182 | * ``PIP_FALLBACK``: the default behaviour is to fall back to try to pip 183 | install a package if installing it with conda fails for any reason. Set 184 | this variable to ``false`` to opt out of this. 185 | 186 | * ``RETRY_ERRORS``: a space-separated string of error names. If not set, this 187 | will default to ``RETRY_ERRORS="CondaHTTPError"``. When package installation 188 | via conda fails, the respective command's output (stdout and stderr) is 189 | searched for the strings in ``RETRY_ERRORS``. If any of these is found, the 190 | installation will be automatically retried. 191 | 192 | * ``RETRY_MAX``: an integer specifying the maximum number of automatic retries. 193 | If not set, this will default to ``RETRY_MAX=3``. Setting ``RETRY_MAX`` to 194 | zero will disable automatic retries. 195 | 196 | * ``RETRY_DELAY``: a positive integer specifying the number of seconds to wait 197 | before retrying. If not set, this will default to ``RETRY_DELAY=2``. 198 | 199 | * ``MACOSX_DEPLOYMENT_TARGET`` (OSX only): If left blank, the minimum OSX target 200 | version for LLVM/Clang builds will be set to ``10.9``. 201 | If set to ``"clang_default"``, determining the minimum OSX target version is 202 | left to LLVM/Clang. If set to any different value, that value will be used. 203 | 204 | The idea behind the ``MAIN_CMD`` and ``SETUP_CMD`` environment variables is 205 | that the ``script`` section of the ``.travis.yml`` file can be set to: 206 | 207 | ``` 208 | script: 209 | - $MAIN_CMD $SETUP_CMD 210 | ``` 211 | 212 | The typical usage will then be to set ``MAIN_CMD`` to default to ``python 213 | setup.py`` and then set ``SETUP_CMD='test'``, and this then allows special 214 | builds that have ``MAIN_CMD='pycodestyle'`` and ``SETUP_CMD=''``. 215 | 216 | Packages can also choose to not use the ``MAIN_CMD`` variable and instead 217 | to set the ``script`` section to: 218 | 219 | ``` 220 | script: 221 | - python setup.py $SETUP_CMD 222 | ``` 223 | 224 | and simply adjust ``SETUP_CMD`` as needed. 225 | 226 | Following the set-up, if additional packages need to be installed, the 227 | ``CONDA_INSTALL`` environment variable should be used to make sure that the 228 | Python and Numpy versions stay fixed to those requested, e.g. 229 | 230 | ``` 231 | - $CONDA_INSTALL another_package 232 | ``` 233 | 234 | ### Setting up Python without conda on Travis 235 | 236 | We also provide a script to set up Python on MacOS X and Windows without making 237 | use of conda. To use this include the following lines at the start of the 238 | ``install`` section in ``.travis.yml``: 239 | 240 | ```yaml 241 | install: 242 | - git clone --depth 1 git://github.com/astropy/ci-helpers.git 243 | - source ci-helpers/travis/setup_python.sh 244 | ``` 245 | 246 | You will need to set the ``PYTHON_VERSION`` environment variable to the 247 | major.minor version of Python that you want to have installed (e.g. 3.8) 248 | 249 | The script does nothing on Linux, so it is safe to call as above without special 250 | casing the operating system. On Linux, you should instead use ``language: 251 | python`` provide the Python version with ``python: ...``. 252 | 253 | The script also sets up a virtual environment using 254 | [venv](https://docs.python.org/3/library/venv.html) and upgrades pip to the 255 | latest version, but does not install any other packages. This is deliberate as we want to keep this script as minimal as possible. 256 | 257 | ### pip pinnings 258 | 259 | We also provide a file called 260 | [pip_pinnings.txt](https://github.com/astropy/ci-helpers/blob/main/pip_pinnings.rst) 261 | which contains any version pins we currently recommend. This file is in the 262 | [pip requirements](https://pip.pypa.io/en/stable/user_guide/#requirements-files) format. 263 | Often this file will be empty if no pinnings are recommended. This file is suitable for 264 | use with any tools that understand pip requirements files, including for example 265 | [tox-pypi-filter](https://pypi.org/project/tox-pypi-filter/). 266 | 267 | ### Utils 268 | 269 | A directory to collect all kinds of useful scripts to be used during various 270 | CI runs. 271 | 272 | * ``import_submodules.py`` - utility script to make it possible to test 273 | importing submodules when optional dependencies, including pytest, are 274 | missing. 275 | 276 | ## Details 277 | 278 | The scripts include: 279 | 280 | * ``travis/setup_dependencies_common.sh`` - set up conda packages on Linux and 281 | MacOS X 282 | * ``travis/setup_conda.sh`` - set up conda on MacOS X or Linux, users should use 283 | this directly rather than the OS specific ones below 284 | * ``travis/setup_conda_linux.sh`` - set up conda on Linux 285 | * ``travis/setup_conda_osx.sh`` - set up conda on MacOS X 286 | * ``travis/setup_python.sh`` - set up Python on MacOS X and Windows without conda 287 | 288 | This repository can be cloned directly from the ``.travis.yml`` 289 | file when about to run tests and does not need to be included 290 | as a Git sub-module in repositories. 291 | -------------------------------------------------------------------------------- /appveyor/README.md: -------------------------------------------------------------------------------- 1 | About 2 | ----- 3 | 4 | Appveyor is no longer supported. The scripts are provided as-is. 5 | They will be removed at some point in the future. 6 | 7 | ### AppVeyor 8 | 9 | Include the following lines at the start of the ``install`` section in 10 | ``appveyor.yml``: 11 | 12 | ```yaml 13 | install: 14 | - "git clone --depth 1 git://github.com/astropy/ci-helpers.git" 15 | - "powershell ci-helpers/appveyor/install-miniconda.ps1" 16 | - "conda activate test" 17 | ``` 18 | 19 | This does the following: 20 | 21 | - Set up Miniconda. 22 | - Set up the PATH appropriately. 23 | - Set up a conda environment named 'test' and switch to it. 24 | - Set the ``always_yes`` config option for conda to ``true`` so that you don't 25 | need to include ``--yes``. 26 | 27 | Following this, various dependencies are installed depending on the following 28 | environment variables: 29 | 30 | * ``NUMPY_VERSION``: if set to ``dev`` or ``development``, the latest 31 | developer version of Numpy is installed along with Cython. If set to a 32 | version number, that version is installed. If set to ``stable``, install 33 | the latest stable version of Numpy. 34 | 35 | * ``ASTROPY_VERSION``: if set to ``dev`` or ``development``, the latest 36 | developer version of Astropy is installed, along with Cython and jinja2, 37 | which are compile-time dependencies. If set to a version number, that 38 | version is installed. If set to ``stable``, install the latest stable 39 | version of Astropy. If set to ``lts`` the latest long term support (LTS) 40 | version is installed (more info about LTS can be found 41 | [here](https://github.com/astropy/astropy-APEs/blob/main/APE2.rst#version-numbering)). 42 | 43 | * ``SUNPY_VERSION``: if set to ``dev`` or ``development``, the latest 44 | developer version of Sunpy is installed. If set to a 45 | version number, that version is installed. If set to ``stable``, install 46 | the latest stable version of Sunpy. 47 | 48 | * ``MINICONDA_VERSION``: this sets the version of Miniconda that will be 49 | installed. Use this to override a pinned version if necessary. 50 | 51 | * ``CONDA_DEPENDENCIES``: this should be a space-separated string of package 52 | names that will be installed with conda. 53 | 54 | * ``CONDA_CHANNELS``: this should be a space-separated string of conda 55 | channel names. We don't add any channel by default. 56 | 57 | * ``DEBUG``: if `True` this turns on the shell debug mode in the install 58 | scripts, and provides information on the current conda install and 59 | switches off the ``-q`` conda flag for verbose output. 60 | 61 | * ``PIP_FALLBACK``: the default behaviour is to fall back to try to pip 62 | install a package if installing it with conda fails for any reason. Set 63 | this variable to ``false`` to opt out of this. 64 | 65 | * ``RETRY_ERRORS``: a comma-separated string array containing error names. 66 | If not set, this will default to ``$RETRY_ERRORS="CondaHTTPError"``. When 67 | package installation via conda fails, the respective command's error output 68 | (stderr) is searched for the strings in ``RETRY_ERRORS``. If any of these is 69 | found, the installation will be automatically retried. Setting 70 | ``RETRY_ERRORS`` in ``appveyor.yml`` will *overwrite* the default. 71 | 72 | * ``RETRY_MAX``: an integer specifying the maximum number of automatic retries. 73 | If not set, this will default to ``$RETRY_MAX=3``. Setting ``RETRY_MAX`` to 74 | zero will disable automatic retries. 75 | 76 | * ``RETRY_DELAY``: a positive integer specifying the number of seconds to wait 77 | before retrying. If not set, this will default to ``$RETRY_DELAY=2``. 78 | 79 | Details 80 | ------- 81 | 82 | The scripts include: 83 | 84 | * ``appveyor/install-miniconda.ps1`` - set up conda on Windows 85 | * ``appveyor/windows_sdk.cmd`` - set up the compiler environment on Windows 86 | -------------------------------------------------------------------------------- /appveyor/install-miniconda.ps1: -------------------------------------------------------------------------------- 1 | # Script to set up Miniconda with a test environment 2 | 3 | # This script has been heavily adapted from a script by Olivier Grisel and Kyle 4 | # Kastner licensed under a BSD 3-clause license, and subsequently modified by 5 | # Stuart Mumford before being adapted to its current form in ci-helper. 6 | 7 | # We use the following function to exit the script after any failing command 8 | function checkLastExitCode { 9 | if ($lastExitCode) { 10 | Write-Host "ERROR: the last command returned the following exit code: $lastExitCode" 11 | Exit $lastExitCode 12 | } 13 | } 14 | 15 | $QUIET = "-q" 16 | 17 | if ($env:DEBUG) { 18 | if($env:DEBUG -match "True") { 19 | 20 | # Show all commands 21 | Set-PSDebug -Trace 1 22 | 23 | # Print out environment variables 24 | Get-ChildItem Env: 25 | 26 | # Disable Quiet mode 27 | $QUIET = "" 28 | 29 | } 30 | } 31 | 32 | # If not set from outside, initialize parameters for the retry_on_known_error() 33 | # function: 34 | 35 | # If a command wrapped by the 'retry_on_known_error' function fails, its output 36 | # (stdout and stderr) is parsed for the strings in $env:RETRY_ERRORS and 37 | # scheduled for retry if any of the strings is found. 38 | # CAUTION: $env:RETRY_ERRORS is assumed to be an ARRAY of strings, i.e., a 39 | # comma-separated list of individual strings (NOT a whitespace-separated 40 | # string)! 41 | # Correct usage example: $env:RETRY_ERRORS = "CondaHTTPError", "SomeOtherError" 42 | if (! (Test-Path $env:RETRY_ERRORS)) { 43 | $env:RETRY_ERRORS = "CondaHTTPError" 44 | } 45 | 46 | # Maximum number of retries (integer): 47 | if (! (Test-Path $env:RETRY_MAX)) { 48 | $env:RETRY_MAX = 3 49 | } 50 | 51 | # Delay before retrying in seconds (non-negative integer): 52 | if (! (Test-Path $env:RETRY_DELAY)) { 53 | $env:RETRY_DELAY = 2 54 | } 55 | 56 | # A wrapper for commands that should be repeated if their output contains any of 57 | # the strings in $env:RETRY_ERRORS. 58 | function retry_on_known_error { 59 | if ($args.Count -eq 0) { 60 | Throw (New-Object -TypeName System.ArgumentException -ArgumentList "Function retry_on_known_error() called without arguments.") 61 | } 62 | # We need to flatten the command arguments since arrays are not automatically expanded: 63 | $flat_args = @() 64 | foreach ($arg in $args) { 65 | if ($arg.GetType().IsArray) { 66 | foreach ($subarg in $arg) { 67 | $flat_args += $subarg 68 | } 69 | } 70 | else { 71 | $flat_args += $arg 72 | } 73 | } 74 | $_n_retries = 0 75 | $_retry = $true 76 | $_exitcode = 0 77 | $_process_info = New-Object System.Diagnostics.ProcessStartInfo 78 | $_process_info.FileName = $flat_args[0] 79 | if ($flat_args.Count -gt 1) { 80 | $_process_info.Arguments = $flat_args[1..($flat_args.length - 1)] 81 | } 82 | $_process_info.RedirectStandardError = $true 83 | $_process_info.RedirectStandardOutput = $true 84 | $_process_info.RedirectStandardInput = $true 85 | $_process_info.UseShellExecute = $false 86 | $_process = New-Object System.Diagnostics.Process 87 | $_process.StartInfo = $_process_info 88 | $_stdout_builder = New-Object -TypeName System.Text.StringBuilder 89 | $_stderr_builder = New-Object -TypeName System.Text.StringBuilder 90 | $_output_event_handler = { 91 | if (! [String]::IsNullOrEmpty($EventArgs.Data)) { 92 | $Event.MessageData.AppendLine($EventArgs.Data) 93 | } 94 | } 95 | if ($env:DEBUG -match "True") { 96 | Write-Host DEBUG: Running ""$flat_args"" 97 | } 98 | while ($_retry) { 99 | $_retry = $false 100 | $_stdout_builder.Clear() | Out-Null # System.Text.StringBuilder.Clear() requires .Net >= 4.0 101 | $_stderr_builder.Clear() | Out-Null # System.Text.StringBuilder.Clear() requires .Net >= 4.0 102 | $_stdout_event = Register-ObjectEvent -InputObject $_process ` 103 | -Action $_output_event_handler ` 104 | -EventName 'OutputDataReceived' ` 105 | -MessageData $_stdout_builder 106 | $_stderr_event = Register-ObjectEvent -InputObject $_process ` 107 | -Action $_output_event_handler ` 108 | -EventName 'ErrorDataReceived' ` 109 | -MessageData $_stderr_builder 110 | $_process.Start() 111 | $_process.BeginOutputReadLine() 112 | $_process.BeginErrorReadLine() 113 | $_process.WaitForExit() 114 | Unregister-Event -SourceIdentifier $_stdout_event.Name 115 | Unregister-Event -SourceIdentifier $_stderr_event.Name 116 | $_stdout = $_stdout_builder.ToString() 117 | $_stderr = $_stderr_builder.ToString() 118 | $_process.CancelOutputRead() 119 | $_process.CancelErrorRead() 120 | $global:lastexitcode = $_process.ExitCode 121 | # If the command was successful, bail out: 122 | if ($lastexitcode -eq 0) { 123 | break 124 | } 125 | # The command errored, so let's check its stderr output for the specified error 126 | # strings: 127 | if ($_n_retries -lt $env:RETRY_MAX) { 128 | foreach ($err in $env:RETRY_ERRORS) { 129 | # If a known error string was found, throw a warning and wait a 130 | # certain number of seconds before invoking the command again: 131 | if ($_stderr | Select-String $err) { 132 | Write-Warning "The command ""$args"" failed due to a $err, retrying after $env:RETRY_DELAY seconds." 133 | $_n_retries++ 134 | $_retry = $true 135 | Start-Sleep $env:RETRY_DELAY 136 | break 137 | } 138 | } 139 | } 140 | } 141 | return $_stdout, $_stderr 142 | } 143 | 144 | $MINICONDA_URL = "https://repo.continuum.io/miniconda/" 145 | 146 | # We will use the 2.0.x releases as "stable" for Python 2.7 and 3.4 147 | if ((python -c "from distutils.version import LooseVersion; import os; print(LooseVersion(os.environ['PYTHON_VERSION']) < str(3.6))") -match "False") { 148 | $env:LATEST_ASTROPY_STABLE = "4.0" 149 | } 150 | else { 151 | $env:LATEST_ASTROPY_STABLE = "2.0.16" 152 | $env:NO_PYTEST_ASTROPY = "True" 153 | } 154 | 155 | $env:ASTROPY_LTS_VERSION = "4.0" 156 | $env:LATEST_NUMPY_STABLE = "1.17" 157 | $env:LATEST_SUNPY_STABLE = "1.0.6" 158 | 159 | # We pin the version for conda as it's not the most stable package from 160 | # release to release. Add note here if version is pinned due to a bug upstream. 161 | if (! $env:CONDA_VERSION) { 162 | $env:CONDA_VERSION = "4.7" 163 | } 164 | 165 | if (! $env:PIP_FALLBACK) { 166 | $env:PIP_FALLBACK = "True" 167 | } 168 | 169 | function DownloadMiniconda ($version, $platform_suffix) { 170 | $webclient = New-Object System.Net.WebClient 171 | $filename = "Miniconda3-" + $version + "-Windows-" + $platform_suffix + ".exe" 172 | 173 | $url = $MINICONDA_URL + $filename 174 | 175 | $basedir = $pwd.Path + "\" 176 | $filepath = $basedir + $filename 177 | if (Test-Path $filename) { 178 | Write-Host "Reusing" $filepath 179 | return $filepath 180 | } 181 | 182 | # Download and retry up to 3 times in case of network transient errors. 183 | Write-Host "Downloading" $filename "from" $url 184 | $retry_attempts = 2 185 | for($i=0; $i -lt $retry_attempts; $i++){ 186 | try { 187 | $webclient.DownloadFile($url, $filepath) 188 | break 189 | } 190 | Catch [Exception]{ 191 | Start-Sleep 1 192 | } 193 | } 194 | if (Test-Path $filepath) { 195 | Write-Host "File saved at" $filepath 196 | } else { 197 | # Retry once to get the error message if any at the last try 198 | $webclient.DownloadFile($url, $filepath) 199 | } 200 | return $filepath 201 | } 202 | 203 | function InstallMiniconda ($miniconda_version, $architecture, $python_home) { 204 | Write-Host "Installing miniconda" $miniconda_version "for" $architecture "bit architecture to" $python_home 205 | if (Test-Path $python_home) { 206 | Write-Host $python_home "already exists, skipping." 207 | return $false 208 | } 209 | if ($architecture -eq "x86") { 210 | $platform_suffix = "x86" 211 | } else { 212 | $platform_suffix = "x86_64" 213 | } 214 | $filepath = DownloadMiniconda $miniconda_version $platform_suffix 215 | Write-Host "Installing" $filepath "to" $python_home 216 | $args = "/InstallationType=AllUsers /S /AddToPath=1 /RegisterPython=1 /D=" + $python_home 217 | Write-Host $filepath $args 218 | Start-Process -FilePath $filepath -ArgumentList $args -Wait -Passthru 219 | #Start-Sleep -s 15 220 | if (Test-Path $python_home) { 221 | Write-Host "Miniconda $miniconda_version ($architecture) installation complete" 222 | } else { 223 | Write-Host "Failed to install Python in $python_home" 224 | Exit 1 225 | } 226 | } 227 | 228 | # Install miniconda, if no version is given use the latest 229 | if (! $env:MINICONDA_VERSION) { 230 | # Note that we pin the Miniconda version to avoid issues when new versions are released. 231 | # This can be updated from time to time. 232 | $env:MINICONDA_VERSION="4.7.10" 233 | } 234 | 235 | InstallMiniconda $env:MINICONDA_VERSION $env:PLATFORM $env:PYTHON 236 | checkLastExitCode 237 | 238 | # Add conda to path 239 | & "${env:PYTHON}\Scripts\activate.bat" 240 | # Equivalent to conda init 241 | $env:PATH = "${env:PYTHON}\condabin;" + $env:PATH 242 | conda init cmd.exe 243 | 244 | # Conda config 245 | 246 | conda config --set always_yes true 247 | checkLastExitCode 248 | 249 | conda config --add channels defaults 250 | checkLastExitCode 251 | 252 | if ($env:CONDA_CHANNELS) { 253 | $CONDA_CHANNELS=$env:CONDA_CHANNELS.split(" ") 254 | foreach ($CONDA_CHANNEL in $CONDA_CHANNELS) { 255 | conda config --add channels $CONDA_CHANNEL 256 | checkLastExitCode 257 | } 258 | # This shouldn't be passed to conda. 259 | Remove-Variable CONDA_CHANNELS 260 | rm env:CONDA_CHANNELS 261 | } 262 | 263 | # Install the build and runtime dependencies of the project. 264 | # Pulled out retry_on_knwon_error for now 265 | conda install $QUIET conda=$env:CONDA_VERSION 266 | checkLastExitCode 267 | 268 | if (! $env:CONDA_CHANNEL_PRIORITY) { 269 | $CONDA_CHANNEL_PRIORITY="disabled" 270 | } else { 271 | $CONDA_CHANNEL_PRIORITY=$env:CONDA_CHANNEL_PRIORITY.ToLower() 272 | } 273 | 274 | # We need to add this after the update, otherwise the ``channel_priority`` 275 | # key may not yet exists 276 | conda config --set channel_priority $CONDA_CHANNEL_PRIORITY 277 | checkLastExitCode 278 | 279 | # Create a conda environment using the astropy bonus packages 280 | if (! $env:CONDA_ENVIRONMENT ) { 281 | # This was preceded by retry_on_known_error but that 282 | # appears to be broken. 283 | conda create $QUIET -n test python=$env:PYTHON_VERSION 284 | } else { 285 | conda env create $QUIET -n test -f $env:CONDA_ENVIRONMENT 286 | } 287 | checkLastExitCode 288 | 289 | conda activate test 290 | checkLastExitCode 291 | 292 | # Set environment variables for environment (activate test doesn't seem to do the trick) 293 | $env:PATH = "${env:PYTHON}\envs\test;${env:PYTHON}\envs\test\Scripts;${env:PYTHON}\envs\test\Library\bin;" + $env:PATH 294 | 295 | if (! $env:MPLBACKEND) { 296 | $env:MPLBACKEND = "Agg" 297 | } 298 | 299 | # Check that we have the expected version of Python 300 | python --version 301 | checkLastExitCode 302 | 303 | # CORE DEPENDENCIES 304 | # any pinned version should be set in `pinned`. We also need to pin the 305 | # python version, as conda sometimes tries to upgrade from e.g. 3.5 to 3.6 306 | # and it's totally unaccapteble for CI. 307 | Add-Content ci-helpers\appveyor\pinned "`npython $env:PYTHON_VERSION*" 308 | 309 | if ($env:PYTEST_VERSION) { 310 | Add-Content ci-helpers\appveyor\pinned "`npytest $env:PYTEST_VERSION*" 311 | } 312 | 313 | Copy-Item ci-helpers\appveyor\pinned ${env:PYTHON}\envs\test\conda-meta\pinned 314 | 315 | if ($env:DEBUG) { 316 | Get-Content ${env:PYTHON}\envs\test\conda-meta\pinned 317 | } 318 | 319 | # Pulled out retry_on_known_error 320 | conda install $QUIET -n test pytest"$env:PYTEST_VERSION" pip 321 | checkLastExitCode 322 | 323 | # In case of older python versions there isn't an up-to-date version of pip 324 | # which may lead to ignore install dependencies of the package we test. 325 | # This update should not interfere with the rest of the functionalities 326 | # here. 327 | if ($env:PIP_NO_UPGRADE -notmatch "True") { 328 | pip install --upgrade pip 329 | } 330 | # Check whether a specific version of Numpy is required 331 | if ($env:NUMPY_VERSION) { 332 | if($env:NUMPY_VERSION -match "stable") { 333 | $NUMPY_OPTION = "numpy=" + $env:LATEST_NUMPY_STABLE 334 | } elseif($env:NUMPY_VERSION -match "dev") { 335 | $NUMPY_OPTION = "Cython pip".Split(" ") 336 | } else { 337 | $NUMPY_OPTION = "numpy=" + $env:NUMPY_VERSION 338 | } 339 | # Pulled out retry_on_known_error 340 | conda install -n test $QUIET $NUMPY_OPTION 341 | checkLastExitCode 342 | } else { 343 | $NUMPY_OPTION = "" 344 | } 345 | 346 | # Check whether a specific version of Astropy is required 347 | if ($env:ASTROPY_VERSION) { 348 | if($env:ASTROPY_VERSION -match "stable") { 349 | if($env:NO_PYTEST_ASTROPY -match "True") { 350 | $ASTROPY_OPTION = "astropy=" + $env:LATEST_ASTROPY_STABLE 351 | } else { 352 | $ASTROPY_OPTION = "astropy=" + $env:LATEST_ASTROPY_STABLE + " pytest-astropy" 353 | } 354 | } elseif($env:ASTROPY_VERSION -match "dev") { 355 | $ASTROPY_OPTION = "Cython pip jinja2 pytest-astropy".Split(" ") 356 | } elseif($env:ASTROPY_VERSION -match "lts") { 357 | $ASTROPY_OPTION = "astropy=" + $env:ASTROPY_LTS_VERSION 358 | } else { 359 | $ASTROPY_OPTION = "astropy=" + $env:ASTROPY_VERSION 360 | } 361 | if ($env:PIP_FALLBACK -match "True") { 362 | # Pulled out retry_on_known_error 363 | $output = conda install -n test $QUIET $NUMPY_OPTION $ASTROPY_OPTION.Split(" ") 364 | Write-Host $output 365 | if ($output | select-string UnsatisfiableError) { 366 | Write-Warning "Installing astropy with conda was unsuccessful, using pip instead" 367 | $ASTROPY_OPTION = $ASTROPY_OPTION -replace '=','==' 368 | pip install $ASTROPY_OPTION.Split(" ") 369 | checkLastExitCode 370 | } else { 371 | checkLastExitCode 372 | } 373 | } else { 374 | # Pulled out retry_on_known_error 375 | conda install -n test $QUIET $NUMPY_OPTION $ASTROPY_OPTION.Split(" ") 376 | checkLastExitCode 377 | } 378 | 379 | } else { 380 | $ASTROPY_OPTION = "" 381 | } 382 | 383 | # Check whether a specific version of Sunpy is required 384 | if ($env:SUNPY_VERSION) { 385 | if($env:SUNPY_VERSION -match "stable") { 386 | $SUNPY_OPTION = "sunpy" 387 | } elseif($env:SUNPY_VERSION -match "dev") { 388 | $SUNPY_OPTION = "" 389 | } else { 390 | $SUNPY_OPTION = "sunpy=" + $env:SUNPY_VERSION 391 | } 392 | if ($env:PIP_FALLBACK -match "True") { 393 | # Pulled out retry_on_known_error 394 | $output = conda install -n test $QUIET $NUMPY_OPTION $SUNPY_OPTION 395 | Write-Host $output 396 | if ($output | select-string UnsatisfiableError) { 397 | Write-Warning "Installing sunpy with conda was unsuccessful, using pip instead" 398 | $SUNPY_OPTION = $SUNPY_OPTION -replace '=','==' 399 | pip install $SUNPY_OPTION 400 | checkLastExitCode 401 | } else { 402 | checkLastExitCode 403 | } 404 | } else { 405 | # Pulled out retry_on_known_error 406 | conda install -n test $QUIET $NUMPY_OPTION $SUNPY_OPTION 407 | checkLastExitCode 408 | } 409 | } else { 410 | $SUNPY_OPTION = "" 411 | } 412 | 413 | # Install the specified versions of numpy and other dependencies 414 | if ($env:CONDA_DEPENDENCIES) { 415 | $CONDA_DEPENDENCIES = $env:CONDA_DEPENDENCIES.split(" ") 416 | } else { 417 | $CONDA_DEPENDENCIES = "" 418 | } 419 | 420 | # If NUMPY_OPTION and CONDA_DEPENDENCIES are both empty, we skip this step 421 | if ($NUMPY_OPTION -or $CONDA_DEPENDENCIES) { 422 | 423 | if ($env:PIP_FALLBACK -match "True") { 424 | # Pulled out retry_on_known_error 425 | $output = conda install -n test $QUIET $NUMPY_OPTION $CONDA_DEPENDENCIES 426 | Write-Host $output 427 | if ($output | select-string UnsatisfiableError, PackageNotFoundError, PackagesNotFoundError) { 428 | Write-Warning "Installing dependencies with conda was unsuccessful, using pip instead" 429 | pip install $CONDA_DEPENDENCIES 430 | checkLastExitCode 431 | } else { 432 | checkLastExitCode 433 | } 434 | } else { 435 | # Pulled out retry_on_known_error 436 | conda install -n test $QUIET $NUMPY_OPTION $CONDA_DEPENDENCIES 437 | checkLastExitCode 438 | } 439 | 440 | } 441 | 442 | # Check whether the developer version of Numpy is required and if yes install it 443 | if ($env:NUMPY_VERSION -match "dev") { 444 | Invoke-Expression "${env:CMD_IN_ENV} pip install git+https://github.com/numpy/numpy.git#egg=numpy --upgrade --no-deps" 445 | checkLastExitCode 446 | } 447 | 448 | # TODO: Use conda to install pyerfa when it is available. 449 | # Check whether the developer version of Astropy is required and if yes install 450 | # it. We need to include --no-deps to make sure that Numpy doesn't get upgraded. 451 | if ($env:ASTROPY_VERSION -match "dev") { 452 | Invoke-Expression "${env:CMD_IN_ENV} pip install pyerfa --upgrade --no-deps" 453 | Invoke-Expression "${env:CMD_IN_ENV} pip install git+https://github.com/astropy/astropy.git#egg=astropy --upgrade --no-deps" 454 | checkLastExitCode 455 | } 456 | 457 | # Check whether the developer version of Sunpy is required and if yes install 458 | # it. We need to include --no-deps to make sure that Numpy doesn't get upgraded. 459 | if ($env:SUNPY_VERSION -match "dev") { 460 | Invoke-Expression "${env:CMD_IN_ENV} pip install git+https://github.com/sunpy/sunpy.git#egg=sunpy --upgrade --no-deps" 461 | checkLastExitCode 462 | } 463 | 464 | # We finally install the dependencies listed in PIP_DEPENDENCIES. We do this 465 | # after installing the Numpy versions of Numpy or Astropy. If we didn't do this, 466 | # then calling pip earlier could result in the stable version of astropy getting 467 | # installed, and then overritten later by the dev version (which would waste 468 | # build time) 469 | 470 | if ($env:PIP_FLAGS) { 471 | $PIP_FLAGS = $env:PIP_FLAGS.split(" ") 472 | } else { 473 | $PIP_FLAGS = "" 474 | } 475 | 476 | if ($env:PIP_DEPENDENCIES) { 477 | $PIP_DEPENDENCIES = $env:PIP_DEPENDENCIES.split(" ") 478 | } else { 479 | $PIP_DEPENDENCIES = "" 480 | } 481 | 482 | if ($env:PIP_DEPENDENCIES) { 483 | pip install $PIP_DEPENDENCIES $PIP_FLAGS 484 | checkLastExitCode 485 | } 486 | -------------------------------------------------------------------------------- /appveyor/pinned: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/astropy/ci-helpers/78f1095e71a45b187385fd735b786cb0b98b9d35/appveyor/pinned -------------------------------------------------------------------------------- /appveyor/windows_sdk.cmd: -------------------------------------------------------------------------------- 1 | :: To build extensions for 64 bit Python 3.5 or later no special environment needs 2 | :: to be configured. 3 | :: 4 | :: To build extensions for 64 bit Python 3.4 or earlier, we need to configure environment 5 | :: variables to use the MSVC 2010 C++ compilers from GRMSDKX_EN_DVD.iso of: 6 | :: MS Windows SDK for Windows 7 and .NET Framework 4 (SDK v7.1) 7 | :: 8 | :: To build extensions for 64 bit Python 2, we need to configure environment 9 | :: variables to use the MSVC 2008 C++ compilers from GRMSDKX_EN_DVD.iso of: 10 | :: MS Windows SDK for Windows 7 and .NET Framework 3.5 (SDK v7.0) 11 | :: 12 | :: 32 bit builds do not require specific environment configurations. 13 | :: 14 | :: Note: this script needs to be run with the /E:ON and /V:ON flags for the 15 | :: cmd interpreter, at least for (SDK v7.0) 16 | :: 17 | :: More details at: 18 | :: https://github.com/cython/cython/wiki/64BitCythonExtensionsOnWindows 19 | :: https://stackoverflow.com/a/13751649/163740 20 | :: 21 | :: Original Author: Olivier Grisel 22 | :: License: CC0 1.0 Universal: https://creativecommons.org/publicdomain/zero/1.0/ 23 | :: This version based on updates for python 3.5 by Phil Elson at: 24 | :: https://github.com/pelson/Obvious-CI/tree/master/scripts 25 | 26 | @ECHO OFF 27 | 28 | SET COMMAND_TO_RUN=%* 29 | SET WIN_SDK_ROOT=C:\Program Files\Microsoft SDKs\Windows 30 | 31 | SET MAJOR_PYTHON_VERSION="%PYTHON_VERSION:~0,1%" 32 | SET MINOR_PYTHON_VERSION=%PYTHON_VERSION:~2,1% 33 | IF %MAJOR_PYTHON_VERSION% == "2" ( 34 | SET WINDOWS_SDK_VERSION="v7.0" 35 | SET SET_SDK_64=Y 36 | ) ELSE IF %MAJOR_PYTHON_VERSION% == "3" ( 37 | SET WINDOWS_SDK_VERSION="v7.1" 38 | IF %MINOR_PYTHON_VERSION% LEQ 4 ( 39 | SET SET_SDK_64=Y 40 | ) ELSE ( 41 | SET SET_SDK_64=N 42 | ) 43 | ) ELSE ( 44 | ECHO Unsupported Python version: "%MAJOR_PYTHON_VERSION%" 45 | EXIT 1 46 | ) 47 | 48 | IF "%PYTHON_ARCH%"=="64" ( 49 | IF %SET_SDK_64% == Y ( 50 | ECHO Configuring Windows SDK %WINDOWS_SDK_VERSION% for Python %MAJOR_PYTHON_VERSION% on a 64 bit architecture 51 | SET DISTUTILS_USE_SDK=1 52 | SET MSSdk=1 53 | "%WIN_SDK_ROOT%\%WINDOWS_SDK_VERSION%\Setup\WindowsSdkVer.exe" -q -version:%WINDOWS_SDK_VERSION% 54 | "%WIN_SDK_ROOT%\%WINDOWS_SDK_VERSION%\Bin\SetEnv.cmd" /x64 /release 55 | ECHO Executing: %COMMAND_TO_RUN% 56 | call %COMMAND_TO_RUN% || EXIT 1 57 | ) ELSE ( 58 | ECHO Using default MSVC build environment for 64 bit architecture 59 | ECHO Executing: %COMMAND_TO_RUN% 60 | call %COMMAND_TO_RUN% || EXIT 1 61 | ) 62 | ) ELSE ( 63 | ECHO Using default MSVC build environment for 32 bit architecture 64 | ECHO Executing: %COMMAND_TO_RUN% 65 | call %COMMAND_TO_RUN% || EXIT 1 66 | ) 67 | -------------------------------------------------------------------------------- /conda_environment.yml: -------------------------------------------------------------------------------- 1 | name: ci_helpers 2 | channels: 3 | - defaults 4 | dependencies: 5 | - python=3.5 6 | - numpy 7 | - matplotlib 8 | -------------------------------------------------------------------------------- /pip_pinnings.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/astropy/ci-helpers/78f1095e71a45b187385fd735b786cb0b98b9d35/pip_pinnings.txt -------------------------------------------------------------------------------- /test_env.py: -------------------------------------------------------------------------------- 1 | import os 2 | import re 3 | import sys 4 | from distutils.version import LooseVersion 5 | 6 | import pytest 7 | 8 | PYTEST_LT_3 = LooseVersion(pytest.__version__) < LooseVersion('3') 9 | 10 | PYTEST_LT_37 = LooseVersion(pytest.__version__) < LooseVersion('3.7') 11 | 12 | 13 | # If we are on Travis or AppVeyor, we should check if we are running the tests 14 | # in the ci-helpers repository, or whether for example someone else is running 15 | # 'py.test' without arguments after having cloned ci-helpers. 16 | 17 | if 'APPVEYOR_PROJECT_SLUG' in os.environ: 18 | if os.environ['APPVEYOR_PROJECT_SLUG'] != 'ci-helpers': 19 | if PYTEST_LT_3: 20 | pytest.skip() 21 | else: 22 | pytestmark = pytest.mark.skip() 23 | 24 | if 'TRAVIS_REPO_SLUG' in os.environ: 25 | if os.environ['TRAVIS_REPO_SLUG'].split('/')[1] != 'ci-helpers': 26 | if PYTEST_LT_3: 27 | pytest.skip() 28 | else: 29 | pytestmark = pytest.mark.skip() 30 | 31 | # The test scripts accept 'stable' for ASTROPY_VERSION to test that it's 32 | # properly parsed hard-wire the latest stable branch version here 33 | 34 | 35 | if not LooseVersion(sys.version) < '3.6': 36 | LATEST_ASTROPY_STABLE = '4.0' 37 | # This is not for windows but appveyor only 38 | LATEST_ASTROPY_STABLE_WIN = '4.0' 39 | LATEST_NUMPY_STABLE = '1.18' 40 | else: 41 | LATEST_ASTROPY_STABLE = '2.0.16' 42 | # This is not for windows but appveyor only 43 | LATEST_ASTROPY_STABLE_WIN = '2.0.16' 44 | LATEST_NUMPY_STABLE = '1.16' 45 | 46 | LATEST_ASTROPY_LTS = '4.0' 47 | # This is not for windows but appveyor only 48 | LATEST_ASTROPY_LTS_WIN = '4.0' 49 | LATEST_NUMPY_STABLE_WIN = '1.17' 50 | LATEST_SUNPY_STABLE = '1.0.6' 51 | 52 | if os.environ.get('PIP_DEPENDENCIES', None) is not None: 53 | PIP_DEPENDENCIES = os.environ['PIP_DEPENDENCIES'].split(' ') 54 | else: 55 | PIP_DEPENDENCIES = [] 56 | 57 | if os.environ.get('CONDA_DEPENDENCIES', None) is not None: 58 | CONDA_DEPENDENCIES = os.environ['CONDA_DEPENDENCIES'].split(' ') 59 | else: 60 | CONDA_DEPENDENCIES = [] 61 | 62 | 63 | # In this dependency list we should only store the package names, 64 | # not the required versions 65 | dependency_list = ([re.split('=|<|>', PIP_DEPENDENCIES[i])[0] 66 | for i in range(len(PIP_DEPENDENCIES))] + 67 | [re.split('=|<|>', CONDA_DEPENDENCIES[i])[0] 68 | for i in range(len(CONDA_DEPENDENCIES))]) 69 | 70 | 71 | def test_python_version(): 72 | if 'PYTHON_VERSION' in os.environ: 73 | assert sys.version.startswith(os.environ['PYTHON_VERSION']) 74 | elif 'TRAVIS_PYTHON_VERSION' in os.environ: 75 | assert sys.version.startswith(os.environ['TRAVIS_PYTHON_VERSION']) 76 | 77 | 78 | def test_exported_variables(): 79 | if 'TRAVIS' in os.environ and os.environ.get('SETUP_MODE', '') == 'conda': 80 | assert os.environ.get('ASTROPY_LTS_VERSION', '') == LATEST_ASTROPY_LTS 81 | assert os.environ.get('LATEST_NUMPY_STABLE', '') == LATEST_NUMPY_STABLE 82 | assert os.environ.get('LATEST_SUNPY_STABLE', '') == LATEST_SUNPY_STABLE 83 | 84 | def test_numpy(): 85 | if 'NUMPY_VERSION' in os.environ: 86 | import numpy 87 | np_version = numpy.__version__ 88 | os_numpy_version = os.environ['NUMPY_VERSION'].lower() 89 | if 'dev' in os_numpy_version: 90 | assert 'dev' in np_version 91 | elif 'pre' in os_numpy_version: 92 | assert re.match("[0-9.]*[0-9](a[0-9]|b[0-9]|rc[0-9])", np_version) 93 | else: 94 | if 'stable' in os_numpy_version: 95 | if 'APPVEYOR' in os.environ: 96 | assert np_version.startswith(LATEST_NUMPY_STABLE_WIN) 97 | else: 98 | assert np_version.startswith(LATEST_NUMPY_STABLE) 99 | else: 100 | assert np_version.startswith(os_numpy_version) 101 | assert re.match("^[0-9]+\.[0-9]+\.[0-9]", np_version) 102 | 103 | 104 | def test_pytest(): 105 | if 'PYTEST_VERSION' in os.environ: 106 | if 'dev' in os.environ['PYTEST_VERSION'].lower(): 107 | assert 'dev' in pytest.__version__ 108 | 109 | 110 | def test_astropy(): 111 | if 'ASTROPY_VERSION' in os.environ: 112 | import astropy 113 | os_astropy_version = os.environ['ASTROPY_VERSION'].lower() 114 | if 'dev' in os_astropy_version: 115 | assert 'dev' in astropy.__version__ 116 | else: 117 | if 'pre' in os_astropy_version: 118 | assert re.match("[0-9.]*[0-9](rc[0-9])", astropy.__version__) 119 | elif 'stable' in os_astropy_version: 120 | if 'APPVEYOR' in os.environ: 121 | assert astropy.__version__.startswith(LATEST_ASTROPY_STABLE_WIN) 122 | else: 123 | assert astropy.__version__.startswith(LATEST_ASTROPY_STABLE) 124 | elif 'lts' in os_astropy_version: 125 | if 'APPVEYOR' in os.environ: 126 | assert astropy.__version__.startswith(LATEST_ASTROPY_LTS_WIN) 127 | else: 128 | assert astropy.__version__.startswith(LATEST_ASTROPY_LTS) 129 | else: 130 | assert astropy.__version__.startswith(os_astropy_version) 131 | assert 'dev' not in astropy.__version__ 132 | 133 | # We should not install pytest-astropy for the 2.0.x series, since 134 | # pytest-astropy is a metapackage, check one of its components 135 | if LATEST_ASTROPY_STABLE.startswith('2'): 136 | with pytest.raises(ImportError): 137 | import pytest_doctestplus 138 | 139 | 140 | def test_mpl(): 141 | if 'MATPLOTLIB_VERSION' in os.environ: 142 | import matplotlib 143 | os_mpl_version = os.environ['MATPLOTLIB_VERSION'].lower() 144 | 145 | # Revise when figured out the exact rule behind mpl dev versioning 146 | if 'dev' in os_mpl_version: 147 | assert '+' in matplotlib.__version__ 148 | else: 149 | assert matplotlib.__version__.startswith(os_mpl_version) 150 | assert '+' not in matplotlib.__version__ 151 | 152 | 153 | def test_sunpy(): 154 | if 'SUNPY_VERSION' in os.environ: 155 | import sunpy 156 | os_sunpy_version = os.environ['SUNPY_VERSION'].lower() 157 | if 'dev' in os_sunpy_version: 158 | assert 'dev' in sunpy.__version__ 159 | else: 160 | if 'pre' in os_sunpy_version: 161 | assert re.match("[0-9.]*[0-9](rc[0-9])", sunpy.__version__) 162 | elif 'stable' in os_sunpy_version: 163 | assert sunpy.__version__.startswith(LATEST_SUNPY_STABLE) 164 | else: 165 | assert sunpy.__version__.startswith(os_sunpy_version) 166 | assert 'dev' not in sunpy.__version__ 167 | 168 | 169 | # Check whether everything is installed and importable 170 | def test_dependency_imports(): 171 | 172 | # We have to ignore the special case where we are running with --no-deps 173 | # because we don't expect that import to work. 174 | if os.environ.get('CONDA_DEPENDENCIES_FLAGS', '') == '--no-deps': 175 | return 176 | 177 | for package in dependency_list: 178 | if package == 'pyqt5': 179 | __import__('PyQt5') 180 | elif package == 'scikit-image': 181 | __import__('skimage') 182 | elif package == 'scikit-learn': 183 | __import__('sklearn') 184 | elif package == 'openjpeg': 185 | continue 186 | elif package == 'pytest-cov': 187 | __import__('pytest_cov') 188 | elif package == 'nomkl': 189 | import subprocess 190 | assert 'nomkl' in str(subprocess.check_output(["conda", "list"])) 191 | elif package == 'mkl': 192 | import subprocess 193 | assert 'nomkl' not in str(subprocess.check_output(["conda", "list"])) 194 | assert 'mkl' in str(subprocess.check_output(["conda", "list"])) 195 | elif package == 'pillow': 196 | __import__('PIL') 197 | elif package == 'attrs': 198 | continue 199 | elif package == 'libtiff': 200 | continue 201 | elif package == '': 202 | continue 203 | else: 204 | __import__(package) 205 | 206 | 207 | def test_sphinx(): 208 | if 'SETUP_CMD' in os.environ: 209 | if ('build_sphinx' in os.environ['SETUP_CMD'] or 210 | 'build_docs' in os.environ['SETUP_CMD']): 211 | import sphinx 212 | 213 | 214 | def test_open_files(): 215 | if 'open-files' in os.environ.get('SETUP_CMD', ''): 216 | import psutil 217 | 218 | 219 | def test_conda_flags(): 220 | if (os.environ.get('CONDA_DEPENDENCIES_FLAGS', '') == '--no-deps' and 221 | os.environ.get('CONDA_DEPENDENCIES', '') == 'matplotlib'): 222 | try: 223 | import numpy 224 | except: 225 | pass 226 | else: 227 | raise Exception("Numpy should not be installed") 228 | else: 229 | pass 230 | 231 | 232 | def test_pip_flags(): 233 | pip_flags = os.environ.get('PIP_DEPENDENCIES_FLAGS', '') 234 | if pip_flags.startswith('--log'): 235 | assert os.path.exists(pip_flags.split()[1]) 236 | else: 237 | pass 238 | 239 | 240 | def test_regression_mkl(): 241 | 242 | # Regression test to make sure that if the developer version of Numpy is 243 | # used, scipy still works correctly. At some point, the conda packages for 244 | # Numpy and Scipy were compiled with the MKL, and this then led to issues 245 | # if installing Numpy dev with pip without making sure it was also using 246 | # the MKL. The solution is to simply make sure that we install the 247 | # ``nomkl`` conda pacakge when installing the developer version of Numpy. 248 | 249 | if os.environ.get('NUMPY_VERSION', '') == 'dev': 250 | 251 | try: 252 | import scipy 253 | except ImportError: 254 | return 255 | 256 | import numpy as np 257 | from scipy.linalg import inv 258 | 259 | x = np.random.random((3, 3)) 260 | inv(x) 261 | 262 | 263 | def test_conda_channel_priority(): 264 | 265 | if os.environ.get('SETUP_MODE', '') == 'conda': 266 | 267 | channel_priority = os.environ.get('CONDA_CHANNEL_PRIORITY', 'disabled') 268 | 269 | with open(os.path.expanduser('~/.condarc'), 'r') as f: 270 | content = f.read() 271 | 272 | assert 'channel_priority: {0}'.format(channel_priority.lower()) in content 273 | -------------------------------------------------------------------------------- /tox.ini: -------------------------------------------------------------------------------- 1 | [tox] 2 | envlist = test 3 | 4 | [testenv] 5 | skip_install = true 6 | whitelist_externals = 7 | touch 8 | commands = 9 | touch tox_worked.log 10 | -------------------------------------------------------------------------------- /travis/hack_version_numbers.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import re 3 | 4 | pkg_pat1 = re.compile(r'.+pinned spec ([^><=\[]+)=(.+) conflicts.+') 5 | pkg_pat2 = re.compile(r'.+pinned spec ([^><=[]+)(\[.+?\]) conflicts.+') 6 | 7 | spec_conflicts = sys.argv[1] 8 | 9 | if len(sys.argv) == 3: 10 | # We got the conda command in one shot 11 | original_args = sys.argv[2].split(" ") 12 | else: 13 | # The arguments were split by the shell 14 | original_args = sys.argv[2:] 15 | 16 | # Just crash if the command wasn't actually conda install 17 | assert original_args[0] == 'conda' 18 | assert original_args[1] == 'install' 19 | 20 | with open(spec_conflicts) as f: 21 | lines = f.readlines() 22 | 23 | versions = dict() 24 | 25 | for line in lines: 26 | # Packages with overriden specs will match either pattern 1 or 2 27 | match = pkg_pat1.match(line) 28 | if not match: 29 | match = pkg_pat2.match(line) 30 | 31 | if match: 32 | package = match.group(1) 33 | versions[package] = "=" + match.group(2) 34 | 35 | # We never want to add a spec to the "conda install" part of the command 36 | new_args = original_args[:2] 37 | 38 | for arg in original_args[2:]: 39 | if arg in versions: 40 | # Add a version spec 41 | pkg_version = versions[arg] 42 | new_args.append(arg + pkg_version) 43 | else: 44 | new_args.append(arg) 45 | 46 | # Our "return" value is a new command with pinned specs 47 | print(" ".join(new_args)) 48 | -------------------------------------------------------------------------------- /travis/setup_conda.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Note to the future: keep the conda scripts separate for each OS because many 4 | # packages call ci-helpers with: 5 | # 6 | # source ci-helpers/travis/setup_conda_$TRAVIS_OS_NAME.sh 7 | # 8 | # The present script was added later. 9 | 10 | if [[ $DEBUG == True ]]; then 11 | set -x 12 | fi 13 | 14 | # First check: if the build should be run at all based on the event type 15 | 16 | if [[ ! -z $EVENT_TYPE ]]; then 17 | for event in $EVENT_TYPE; do 18 | if [[ $TRAVIS_EVENT_TYPE = $event ]]; then 19 | allow_to_build=True 20 | fi 21 | done 22 | if [[ $allow_to_build != True ]]; then 23 | travis_terminate 0 24 | fi 25 | fi 26 | 27 | # Second check: if any of the custom tags are used to skip the build 28 | 29 | DOCS_ONLY="\[docs only|build docs\]" 30 | 31 | # Travis doesn't provide the commit message of the top of the branch for 32 | # PRs, only the commit message of the merge. Thus this ugly workaround is 33 | # needed for now. 34 | 35 | if [[ $TRAVIS_PULL_REQUEST == false ]]; then 36 | COMMIT_MESSAGE=${TRAVIS_COMMIT_MESSAGE} 37 | else 38 | COMMIT_MESSAGE=$(git show -s $TRAVIS_COMMIT_RANGE | awk 'BEGIN{count=0}{if ($1=="Author:") count++; if (count==1) print $0}') 39 | fi 40 | 41 | if [[ ! -z $(echo ${COMMIT_MESSAGE} | grep -E "${DOCS_ONLY}") ]]; then 42 | if [[ ! $SETUP_CMD =~ build_docs|build_sphinx|pycodestyle|pylint|flake8|pep8 ]] && [[ ! $MAIN_CMD =~ pycodestyle|pylint|flake8|pep8 ]]; then 43 | # we also allow the style checkers to run here 44 | echo "Only docs build was requested by the commit message, exiting." 45 | travis_terminate 0 46 | fi 47 | fi 48 | 49 | echo "==================== Starting executing ci-helpers scripts =====================" 50 | 51 | source ./ci-helpers/travis/setup_conda_$TRAVIS_OS_NAME.sh; 52 | 53 | echo "================= Returning executing local .travis.yml script =================" 54 | 55 | set +x 56 | -------------------------------------------------------------------------------- /travis/setup_conda_linux.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Install conda (http://conda.pydata.org/docs/travis.html#the-travis-yml-file) 4 | # Note that we pin the Miniconda version to avoid issues when new versions are released. 5 | # This can be updated from time to time. 6 | if [[ -z "${MINICONDA_VERSION}" ]]; then 7 | MINICONDA_VERSION=4.7.10 8 | fi 9 | wget https://repo.continuum.io/miniconda/Miniconda3-${MINICONDA_VERSION}-Linux-x86_64.sh -O miniconda.sh --progress=dot:mega 10 | # Create .conda directory before install to workaround conda bug 11 | # See https://github.com/ContinuumIO/anaconda-issues/issues/11148 12 | mkdir $HOME/.conda 13 | bash miniconda.sh -b -p $HOME/miniconda 14 | $HOME/miniconda/bin/conda init bash 15 | source ~/.bash_profile 16 | conda activate base 17 | 18 | # Install common Python dependencies 19 | source "$( dirname "${BASH_SOURCE[0]}" )"/setup_dependencies_common.sh 20 | 21 | if [[ $SETUP_XVFB == True ]]; then 22 | export DISPLAY=:99.0 23 | /sbin/start-stop-daemon --start --quiet --pidfile /tmp/custom_xvfb_99.pid --make-pidfile --background --exec /usr/bin/Xvfb -- :99 -screen 0 1920x1200x24 -ac +extension GLX +render -noreset 24 | fi 25 | -------------------------------------------------------------------------------- /travis/setup_conda_osx.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Workaround for https://github.com/travis-ci/travis-ci/issues/6307, which 4 | # caused the following error on MacOS X workers: 5 | # 6 | # Warning, RVM 1.26.0 introduces signed releases and automated check of signatures when GPG software found. 7 | # /Users/travis/build.sh: line 109: shell_session_update: command not found 8 | # 9 | command curl -sSL https://rvm.io/mpapis.asc | gpg --import -; 10 | rvm get stable 11 | 12 | # Install conda (http://conda.pydata.org/docs/travis.html#the-travis-yml-file) 13 | # Note that we pin the Miniconda version to avoid issues when new versions are released. 14 | # This can be updated from time to time. 15 | if [[ -z "${MINICONDA_VERSION}" ]]; then 16 | MINICONDA_VERSION=4.7.10 17 | fi 18 | 19 | # Set default OSX deployment target version to 10.9, since this is required for 20 | # compiling C++ code with llvm/clang when -stdlib=libc++ is specified. 21 | # Note that, for linking, version 10.7 should suffice. 22 | if [[ -z "${MACOSX_DEPLOYMENT_TARGET}" ]]; then 23 | export MACOSX_DEPLOYMENT_TARGET=10.9 24 | elif [[ "${MACOSX_DEPLOYMENT_TARGET}" == "clang_default" ]]; then 25 | export MACOSX_DEPLOYMENT_TARGET="" 26 | fi 27 | 28 | wget https://repo.continuum.io/miniconda/Miniconda3-${MINICONDA_VERSION}-MacOSX-x86_64.sh -O miniconda.sh 29 | # Create .conda directory before install to workaround conda bug 30 | # See https://github.com/ContinuumIO/anaconda-issues/issues/11148 31 | mkdir $HOME/.conda 32 | bash miniconda.sh -b -p $HOME/miniconda 33 | $HOME/miniconda/bin/conda init bash 34 | source ~/.bash_profile 35 | conda activate base 36 | 37 | # Install common Python dependencies 38 | source "$( dirname "${BASH_SOURCE[0]}" )"/setup_dependencies_common.sh 39 | -------------------------------------------------------------------------------- /travis/setup_conda_windows.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # This script assumes we are running under git-bash (MinGW) on Windows 3 | 4 | if [[ $DEBUG == True ]]; then 5 | set -x 6 | fi 7 | 8 | if [[ -z "${MINICONDA_VERSION}" ]]; then 9 | MINICONDA_VERSION=4.6.14 10 | fi 11 | 12 | echo "installing miniconda3" 13 | choco install miniconda3 --params="'/AddToPath:1'" --version="$MINICONDA_VERSION"; 14 | /c/tools/miniconda3/scripts/conda init bash 15 | source "/c/Users/travis/.bash_profile" 16 | conda activate base 17 | 18 | PIN_FILE_CONDA="/c/tools/miniconda3/conda-meta/pinned" 19 | PIN_FILE="/c/tools/miniconda3/envs/test/conda-meta/pinned" 20 | 21 | # Install common Python dependencies 22 | echo "setting up common dependencies" 23 | source "$( dirname "${BASH_SOURCE[0]}" )"/setup_dependencies_common.sh 24 | -------------------------------------------------------------------------------- /travis/setup_dependencies_common.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash -x 2 | 3 | hash -r 4 | 5 | set -e 6 | 7 | 8 | # If not set from outside, initialize parameters for the retry_on_known_error() 9 | # function: 10 | 11 | # If a command wrapped by the 'retry_on_known_error' function fails, its output 12 | # (stdout and stderr) is parsed for the strings in RETRY_ERRORS and scheduled 13 | # for retry if any of the strings is found. 14 | if [ -z "$RETRY_ERRORS" ]; then 15 | RETRY_ERRORS="CondaHTTPError" # add more errors if needed (space-separated) 16 | fi 17 | 18 | # Maximum number of retries: 19 | if [ -z "$RETRY_MAX" ]; then 20 | RETRY_MAX=3 21 | fi 22 | 23 | # Delay before retrying in seconds: 24 | if [ -z "$RETRY_DELAY" ]; then 25 | RETRY_DELAY=2 26 | fi 27 | 28 | # A wrapper for calls that should be repeated if their output contains any of 29 | # the strings in RETRY_ERRORS. 30 | ############################################################################## 31 | # CAUTION: This function will *unify* stdout and stderr of the wrapped call: # 32 | # In case of success, the call's entire output will go to stdout. # 33 | # In case of failure, the call's entire output will go to stderr. # 34 | ############################################################################## 35 | function retry_on_known_error() { 36 | if [ -z "$*" ]; then 37 | echo "ERROR: Function retry_on_known_error() called without arguments." 1>&2 38 | return 1 39 | fi 40 | _tmp_output_file="tmp.txt" 41 | _n_retries=0 42 | _exitval=0 43 | _retry=true 44 | while $_retry; do 45 | _retry=false 46 | # Execute the wrapped command and get its unified output. 47 | # This command needs to run in the current shell/environment in case 48 | # it sets environment variables (like 'conda install' does) 49 | # 50 | # tee will both echo output to stdout and save it in a file. The file 51 | # is needed for some error checks later. Output to stdout is needed in 52 | # the event a conda solve takes a really long time (>10 min). If 53 | # there is no output on travis for that long, the job is cancelled. 54 | set +e 55 | $@ > $_tmp_output_file 2>&1 56 | _exitval="$?" 57 | # Keep the cat here...otherwise _exitval is always 0 58 | # even if the conda install failed. 59 | cat $_tmp_output_file 60 | set -e 61 | 62 | # The hack below is to work around a bug in conda 4.7 in which a spec 63 | # pinned in a pin file is not respected if that package is listed 64 | # explicitly on the command line even if there is no version spec on 65 | # the command line. See: 66 | # 67 | # https://github.com/conda/conda/issues/9052 68 | # 69 | # The hacky workaround is to identify overridden specs and add the 70 | # spec from the pin file back to the command line. 71 | if [[ -n $(grep "conflicts with explicit specs" $_tmp_output_file) ]]; then 72 | # Roll back the command than generated the conflict message. 73 | # To do this, we get the most recent environment revision number, 74 | # then roll back to the one before that. 75 | # To ensure we don't need to activate, direct output of conda to 76 | # a file instead of piping 77 | _revision_file="revisions.txt" 78 | conda list --revision > _revision_file 79 | _current_revision=$(cat _revision_file | grep \(rev | tail -1 | cut -d' ' -f5 | cut -d')' -f1) 80 | conda install --revision=$(( $_current_revision - 1 )) 81 | _tmp_spec_conflicts=bad_spec.txt 82 | # Isolate the problematic specs 83 | grep "conflicts with explicit specs" $_tmp_output_file > $_tmp_spec_conflicts 84 | 85 | # Do NOT turn the three lines below into one by putting the python in a 86 | # $()...we need to make sure we stay in the shell in which conda is activated, 87 | # not a subshell. 88 | _tmp_updated_conda_command=new_command.txt 89 | python ci-helpers/travis/hack_version_numbers.py $_tmp_spec_conflicts "$@" > $_tmp_updated_conda_command 90 | revised_command=$(cat $_tmp_updated_conda_command) 91 | echo $revised_command 92 | # Try it; if it still has conflicts then just give up 93 | $revised_command > $_tmp_output_file 2>&1 94 | _exitval="$?" 95 | # Keep the cat here...otherwise _exitval is always 0 96 | # even if the conda install failed. 97 | cat $_tmp_output_file 98 | if [[ -n $(grep "conflicts with explicit specs" $_tmp_output_file) ]]; then 99 | echo "STOPPING conda attempts because unable to resolve conda pinning issues" 100 | rm -f $_tmp_output_file 101 | return 1 102 | fi 103 | fi 104 | 105 | # If the command was sucessful, abort the retry loop: 106 | if [ "$_exitval" == "0" ]; then 107 | break 108 | fi 109 | 110 | # The command errored, so let's check its output for the specified error 111 | # strings: 112 | if [[ $_n_retries -lt $RETRY_MAX ]]; then 113 | # If a known error string was found, throw a warning and wait a 114 | # certain number of seconds before invoking the command again: 115 | for _error in $RETRY_ERRORS; do 116 | if [ -n "$(grep "$_error" "$_tmp_output_file")" ]; then 117 | echo "WARNING: The comand \"$@\" failed due to a $_error, retrying." 1>&2 118 | _n_retries=$(($_n_retries + 1)) 119 | _retry=true 120 | sleep $RETRY_DELAY 121 | break 122 | fi 123 | done 124 | fi 125 | done 126 | # remove the temporary output file 127 | rm -f "$_tmp_output_file" 128 | # Finally, return the command's exit code: 129 | return $_exitval 130 | } 131 | 132 | # We need to do this before updating conda, as $CONDA_CHANNELS may be a 133 | # conda environment variable for some Miniconda versions, too that needs to 134 | # be space separated. 135 | if [[ ! -z $CONDA_CHANNELS ]]; then 136 | for channel in $CONDA_CHANNELS; do 137 | conda config --add channels $channel 138 | done 139 | fi 140 | 141 | # This used to be in the conditional above, but even if empty it shouldn't 142 | # be passed to conda. 143 | unset CONDA_CHANNELS 144 | 145 | conda config --set always_yes yes --set changeps1 no 146 | 147 | shopt -s nocasematch 148 | 149 | if [[ -z $PYTHON_VERSION ]]; then 150 | export PYTHON_VERSION=$TRAVIS_PYTHON_VERSION 151 | fi 152 | 153 | # We will use the 2.0.x releases as "stable" for Python 2.7 and 3.4 154 | if [[ $(python -c "from distutils.version import LooseVersion; import os;\ 155 | print(LooseVersion(os.environ['PYTHON_VERSION']) < '3.6')") == False ]]; then 156 | export LATEST_ASTROPY_STABLE=4.0 157 | export LATEST_NUMPY_STABLE=1.18 158 | else 159 | export LATEST_ASTROPY_STABLE=2.0.16 160 | export NO_PYTEST_ASTROPY=True 161 | export LATEST_NUMPY_STABLE=1.16 162 | fi 163 | export ASTROPY_LTS_VERSION=4.0 164 | export LATEST_SUNPY_STABLE=1.0.6 165 | 166 | 167 | is_number='[0-9]' 168 | is_eq_number='=[0-9]' 169 | is_eq_float="=[0-9]+\.[0-9]+" 170 | 171 | if [[ -z $PIP_FALLBACK ]]; then 172 | PIP_FALLBACK=true 173 | fi 174 | 175 | if [[ $DEBUG == True ]]; then 176 | QUIET='' 177 | else 178 | QUIET='-q' 179 | fi 180 | 181 | if [[ -z $CONDA_DEPENDENCIES_FLAGS ]]; then 182 | CONDA_DEPENDENCIES_FLAGS='' 183 | fi 184 | 185 | if [[ -z $PIP_DEPENDENCIES_FLAGS ]]; then 186 | PIP_DEPENDENCIES_FLAGS='' 187 | fi 188 | 189 | # We pin the version for conda as it's not the most stable package from 190 | # release to release. Add note here if version is pinned due to a bug upstream. 191 | if [[ -z $CONDA_VERSION ]]; then 192 | if [[ $MAMBA == True ]]; then 193 | CONDA_VERSION=">=4.8" 194 | else 195 | CONDA_VERSION=4.7.11 196 | fi 197 | fi 198 | 199 | if [[ -z $PIN_FILE_CONDA ]]; then 200 | PIN_FILE_CONDA=$HOME/miniconda/conda-meta/pinned 201 | fi 202 | 203 | echo "conda ${CONDA_VERSION}" > $PIN_FILE_CONDA 204 | 205 | retry_on_known_error conda install $QUIET conda 206 | 207 | if [[ -z $CONDA_CHANNEL_PRIORITY ]]; then 208 | CONDA_CHANNEL_PRIORITY=disabled 209 | else 210 | # Make lowercase 211 | CONDA_CHANNEL_PRIORITY=$(echo $CONDA_CHANNEL_PRIORITY | awk '{print tolower($0)}') 212 | fi 213 | 214 | # We need to add this after the update, otherwise the ``channel_priority`` 215 | # key may not yet exists 216 | conda config --set channel_priority $CONDA_CHANNEL_PRIORITY 217 | 218 | # Use utf8 encoding. Should be default, but this is insurance against 219 | # future changes 220 | export PYTHONIOENCODING=UTF8 221 | 222 | # Making sure we don't upgrade python accidentally 223 | if [[ ! -z $PYTHON_VERSION ]]; then 224 | PYTHON_OPTION="python=$PYTHON_VERSION" 225 | else 226 | PYTHON_OPTION="" 227 | fi 228 | 229 | # Setting the MPL backend to a default to avoid occational segfaults with the qt backend 230 | if [[ -z $MPLBACKEND ]]; then 231 | export MPLBACKEND=Agg 232 | fi 233 | 234 | 235 | # Python 3.4 is only available on conda's "free" channel, which was removed in 236 | # conda 4.7. 237 | if [[ $PYTHON_VERSION == 3.4* ]]; then 238 | conda config --set restore_free_channel true 239 | fi 240 | 241 | # CONDA 242 | if [[ -z $CONDA_ENVIRONMENT ]]; then 243 | retry_on_known_error conda create $QUIET -n test $PYTHON_OPTION 244 | else 245 | retry_on_known_error conda env create $QUIET -n test -f $CONDA_ENVIRONMENT 246 | fi 247 | conda activate test 248 | 249 | # PIN FILE 250 | if [[ -z $PIN_FILE ]]; then 251 | PIN_FILE=$HOME/miniconda/envs/test/conda-meta/pinned 252 | fi 253 | 254 | # ensure the PIN_FILE exists 255 | touch $PIN_FILE 256 | 257 | if [[ $DEBUG == True ]]; then 258 | conda config --show 259 | fi 260 | 261 | # EGG_INFO 262 | if [[ $SETUP_CMD == egg_info ]]; then 263 | return # no more dependencies needed 264 | fi 265 | 266 | # install mamba and use it from now on 267 | if [[ $MAMBA == True ]]; then 268 | CONDA_INSTALL_COMMAND='mamba install' 269 | conda install -c conda-forge mamba 270 | else 271 | CONDA_INSTALL_COMMAND='conda install' 272 | fi 273 | 274 | # CORE DEPENDENCIES 275 | 276 | if [[ ! -z $PYTEST_VERSION && $PYTEST_VERSION != dev* ]]; then 277 | echo "pytest ${PYTEST_VERSION}.*" >> $PIN_FILE 278 | fi 279 | 280 | if [[ ! -z $PIP_VERSION ]]; then 281 | echo "pip ${PIP_VERSION}.*" >> $PIN_FILE 282 | fi 283 | 284 | export PIP_INSTALL='python -m pip install' 285 | 286 | retry_on_known_error $CONDA_INSTALL_COMMAND --no-channel-priority $QUIET $PYTHON_OPTION pytest pip || { \ 287 | $PIP_FALLBACK && { \ 288 | if [[ ! -z $PYTEST_VERSION ]]; then 289 | echo "Installing pytest with conda was unsuccessful, using pip instead" 290 | retry_on_known_error $CONDA_INSTALL_COMMAND $QUIET $PYTHON_OPTION pip 291 | if [[ $(echo $PYTEST_VERSION | cut -c 1) =~ $is_number ]]; then 292 | PIP_PYTEST_VERSION='=='${PYTEST_VERSION}.* 293 | elif [[ $(echo $PYTEST_VERSION | cut -c 1-2) =~ $is_eq_number ]]; then 294 | PIP_PYTEST_VERSION='='${PYTEST_VERSION} 295 | else 296 | PIP_PYTEST_VERSION=${PYTEST_VERSION} 297 | fi 298 | $PIP_INSTALL pytest${PIP_PYTEST_VERSION} 299 | awk '{if ($1 != "pytest") print $0}' $PIN_FILE > /tmp/pin_file_temp 300 | mv /tmp/pin_file_temp $PIN_FILE 301 | fi;} 302 | } 303 | 304 | # In case of older python versions there isn't an up-to-date version of pip 305 | # which may lead to ignore install dependencies of the package we test. 306 | # This update should not interfere with the rest of the functionalities 307 | # here. 308 | # 309 | # This *may* be leading to inconsistent conda environments, definitely means 310 | # that conda is not aware of pip installs, and is often overridden by 311 | # subsequent conda installs because conda is configured to install pip by 312 | # default now. 313 | # 314 | # For really old pythons it may be necessary, though, so check pip version and 315 | # install this way if the major version is less than 19. 316 | if [[ -z $PIP_VERSION ]]; then 317 | old_pip=$(python -c "from distutils.version import LooseVersion;\ 318 | import os; import pip;\ 319 | print(LooseVersion(pip.__version__) <\ 320 | LooseVersion('19.0.0'))") 321 | if [[ $old_pip == True ]]; then 322 | $PIP_INSTALL --upgrade pip 323 | fi 324 | fi 325 | 326 | # PEP8 327 | # PEP8 has been renamed to pycodestyle, keep both here for now 328 | if [[ $MAIN_CMD == pep8* ]]; then 329 | $PIP_INSTALL pep8 330 | return # no more dependencies needed 331 | fi 332 | 333 | if [[ $MAIN_CMD == pycodestyle* ]]; then 334 | $PIP_INSTALL pycodestyle 335 | return # no more dependencies needed 336 | fi 337 | 338 | if [[ $MAIN_CMD == flake8* ]]; then 339 | $PIP_INSTALL flake8 340 | return # no more dependencies needed 341 | fi 342 | 343 | if [[ $MAIN_CMD == pylint* ]]; then 344 | $PIP_INSTALL pylint 345 | 346 | # Installing backports when using python 2.7. Add required backports to 347 | # the list 348 | if [[ $PYTHON_VERSION == 2.7 ]]; then 349 | $PIP_INSTALL backports.functools_lru_cache 350 | fi 351 | 352 | return # no more dependencies needed 353 | fi 354 | 355 | # Pin required versions for dependencies, howto is in FAQ of conda 356 | # https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-pkgs.html#preventing-packages-from-updating-pinning 357 | if [[ ! -z $CONDA_DEPENDENCIES ]]; then 358 | 359 | if [[ -z $(echo $CONDA_DEPENDENCIES | grep '\bmkl\b') && 360 | $TRAVIS_OS_NAME != windows && ! -z $NUMPY_VERSION ]]; then 361 | CONDA_DEPENDENCIES=${CONDA_DEPENDENCIES}" nomkl" 362 | fi 363 | 364 | # The astropy testrunner is not compatible with coverage 5.0+, thus we limit the version 365 | if [[ ! -z $(echo $CONDA_DEPENDENCIES | grep -i pytest-cov) ]]; then 366 | if [[ -z $(echo $CONDA_DEPENDENCIES | grep -i coverage) ]]; then 367 | CONDA_DEPENDENCIES=${CONDA_DEPENDENCIES}" coverage<5" 368 | fi 369 | fi 370 | 371 | echo $CONDA_DEPENDENCIES | awk '{print tolower($0)}' | tr " " "\n" | \ 372 | sed -E -e 's|([a-z0-9]+)([=>> $PIN_FILE 373 | 374 | if [[ $DEBUG == True ]]; then 375 | cat $PIN_FILE 376 | fi 377 | 378 | # Let env variable version number override this pinned version 379 | for package in $(awk '{print $1}' $PIN_FILE); do 380 | version=$(eval echo -e \$$(echo $package | tr "-" "_" | \ 381 | awk '{print toupper($0)"_VERSION"}')) 382 | if [[ ! -z $version && ($version != dev* && $version != pre*) ]]; then 383 | awk -v package=$package -v version=$version \ 384 | '{if ($1 == package) print package" " version".*"; 385 | else print $0}' \ 386 | $PIN_FILE > /tmp/pin_file_temp 387 | mv /tmp/pin_file_temp $PIN_FILE 388 | fi 389 | done 390 | 391 | # Do in the pin file what conda silently does on the command line, to 392 | # extend the underspecified version numbers with * 393 | awk -F == '{if (NF==1) print $0; else print $1, $2".*"}' \ 394 | $PIN_FILE > /tmp/pin_file_temp 395 | mv /tmp/pin_file_temp $PIN_FILE 396 | 397 | # We should remove the version numbers from CONDA_DEPENDENCIES to avoid 398 | # the conflict with the *_VERSION env variables 399 | CONDA_DEPENDENCIES=$(awk '{printf tolower($1)" "}' $PIN_FILE) 400 | # Cutting off the trailing space 401 | CONDA_DEPENDENCIES=${CONDA_DEPENDENCIES%?} 402 | 403 | if [[ $DEBUG == True ]]; then 404 | cat $PIN_FILE 405 | echo $CONDA_DEPENDENCIES 406 | fi 407 | fi 408 | 409 | if [[ ! -z $CONDA_DEPENDENCIES ]]; then 410 | # Do a dry run of the conda install here to make sure that pins are 411 | # ACTUALLY being respected. This will become unnecessary when 412 | # https://github.com/conda/conda/issues/9052 413 | # is fixed 414 | 415 | # NOTE: it is important that the expression below remain in an if context 416 | # because of the 'set -e' above, which causes the shell to immediately 417 | # exit if the last command in a pipeline has a non-zero exit status, 418 | # UNLESS the pipeline is in a few specific contexts. From 419 | # https://www.gnu.org/software/bash/manual/html_node/The-Set-Builtin.html: 420 | # 421 | # The shell does not exit if the command that fails is ...part of the 422 | # test in an if statement... 423 | # 424 | # Use tee to print output to console and to file to avoid travis timing out 425 | _tmp_output_file="tmp.txt" 426 | # do not exit on failure of the dry run because pip fallback may succeed 427 | set +e 428 | $CONDA_INSTALL_COMMAND --dry-run $CONDA_DEPENDENCIES > $_tmp_output_file 2>&1 429 | cat $_tmp_output_file 430 | set -e 431 | # 'grep' returns non-zero exit status if no lines match. 432 | if [[ ! -z $(grep "conflicts with explicit specs" $_tmp_output_file) ]]; then 433 | echo "restoring free channel" 434 | # Restoring the free channel only helps if the channel priority 435 | # is not strict, so check that first. If it is strict, fail instead 436 | # of changing the solve logic. 437 | 438 | # ...but only if the free channel is not ruled out by strict 439 | # channel priority 440 | if [[ $CONDA_CHANNEL_PRIORITY == strict ]]; then 441 | # If the channel priority is strict we should fail instead of silently 442 | # changing how the solve is done. 443 | echo "WARNING: May not be able to solve this environment with pinnings and strict channel priority" 444 | # Keep going, because retry_on_known_errors now checks for pinning 445 | # problems and will trigger a pip fallback if they continue. 446 | fi 447 | # Add the free channel, which might fix this... 448 | conda config --set restore_free_channel true 449 | 450 | # Try the dry run again, fail if pinnings are still ignored 451 | echo "Re-running with free channel restored" 452 | 453 | # do not exit on failure of the dry run because pip fallback may succeed 454 | set +e 455 | $CONDA_INSTALL_COMMAND --dry-run $CONDA_DEPENDENCIES > >(tee $_tmp_output_file) 2>&1 456 | set -e 457 | if [[ ! -z $(grep "conflicts with explicit specs" $_tmp_output_file) ]]; then 458 | # No clue how to fix this, so just give up 459 | echo "WARNING: conda is ignoring pinnings" 460 | # Actually, just continue. retry_on_known_errors now checks for 461 | # pinning problems and will trigger a pip fallback if they continue. 462 | fi 463 | fi 464 | 465 | # Clean up 466 | rm -f $_tmp_output_file 467 | fi 468 | 469 | # NUMPY 470 | 471 | # Older versions of numpy are only available on the "free" channel, which 472 | # has been removed as of conda 4.7 from the list of default channels. 473 | # This adds it back if needed. 474 | 475 | if [[ ! -z $NUMPY_VERSION ]]; then 476 | # We only want to do a check for old versions of numpy, not for dev or stable 477 | if [[ $NUMPY_VERSION =~ [0-9]+(\.[0-9]){1,2} ]]; then 478 | old_numpy=$(python -c "from distutils.version import LooseVersion;\ 479 | import os;\ 480 | print(LooseVersion(os.environ['NUMPY_VERSION']) <\ 481 | LooseVersion('1.11.0'))") 482 | if [[ $old_numpy == True ]]; then 483 | conda config --set restore_free_channel true 484 | fi 485 | fi 486 | fi 487 | 488 | # We use --no-pin to avoid installing other dependencies just yet. 489 | 490 | 491 | MKL='nomkl' 492 | if [[ ! -z $(echo $CONDA_DEPENDENCIES | grep '\bmkl\b') || 493 | $TRAVIS_OS_NAME == windows || -z $NUMPY_VERSION ]]; then 494 | MKL='' 495 | fi 496 | 497 | # determine how to install numpy: 498 | NUMPY_INSTALL='' 499 | if [[ $NUMPY_VERSION == dev* ]]; then 500 | # We use C99 to build Numpy. 501 | # If CFLAGS already defined by calling pkg, it's up to them to set this. 502 | if [[ -z $CFLAGS ]]; then 503 | export CFLAGS="-std=c99" 504 | fi 505 | # We install nomkl here to make sure that Numpy and Scipy versions 506 | # installed subsequently don't depend on the MKL. If we don't do this, then 507 | # we run into issues when we install the developer version of Numpy 508 | # because it is then not compiled against the MKL, and one runs into issues 509 | # if Scipy *is* still compiled against the MKL. 510 | retry_on_known_error $CONDA_INSTALL_COMMAND $QUIET --no-pin $PYTHON_OPTION $MKL 511 | # We then install Numpy itself at the bottom of this script 512 | export CONDA_INSTALL="$CONDA_INSTALL_COMMAND $QUIET $PYTHON_OPTION $MKL" 513 | elif [[ $NUMPY_VERSION == stable ]]; then 514 | export NUMPY_OPTION="numpy=$LATEST_NUMPY_STABLE" 515 | export CONDA_INSTALL="$CONDA_INSTALL_COMMAND $QUIET $PYTHON_OPTION $NUMPY_OPTION $MKL" 516 | NUMPY_INSTALL="$CONDA_INSTALL_COMMAND $QUIET --no-pin $PYTHON_OPTION $NUMPY_OPTION $MKL" 517 | elif [[ $NUMPY_VERSION == pre* ]]; then 518 | export NUMPY_OPTION="" 519 | export CONDA_INSTALL="$CONDA_INSTALL_COMMAND $QUIET $PYTHON_OPTION $MKL" 520 | NUMPY_INSTALL="$CONDA_INSTALL_COMMAND $QUIET --no-pin $PYTHON_OPTION $MKL numpy" 521 | if [[ -z $(pip list -o --pre | grep numpy | \ 522 | grep -E "[0-9]rc[0-9]|[0-9][ab][0-9]") ]]; then 523 | # We want to stop the script if there isn't a pre-release available, 524 | # as in that case it would be just another build using the stable 525 | # version. 526 | echo "Prerelease for numpy is not available, stopping test" 527 | travis_terminate 0 528 | fi 529 | elif [[ ! -z $NUMPY_VERSION ]]; then 530 | export NUMPY_OPTION="numpy=$NUMPY_VERSION" 531 | export CONDA_INSTALL="$CONDA_INSTALL_COMMAND $QUIET $PYTHON_OPTION $NUMPY_OPTION $MKL" 532 | NUMPY_INSTALL="$CONDA_INSTALL_COMMAND $QUIET --no-pin $PYTHON_OPTION $NUMPY_OPTION $MKL" 533 | 534 | else 535 | export NUMPY_OPTION="" 536 | export CONDA_INSTALL="$CONDA_INSTALL_COMMAND $QUIET $PYTHON_OPTION $MKL" 537 | fi 538 | 539 | # try to install numpy: 540 | if [[ ! -z $NUMPY_INSTALL ]]; then 541 | retry_on_known_error $NUMPY_INSTALL || { \ 542 | if [[ -z $NUMPY_OPTION ]]; then 543 | PIP_NUMPY_OPTION="numpy" 544 | else 545 | # add wildcard for float-like version specs: 546 | if [[ $NUMPY_OPTION =~ ^numpy$is_eq_float$ ]]; then 547 | PIP_NUMPY_OPTION="numpy==${NUMPY_OPTION#*=}.*" 548 | # use exact version definitions as is: 549 | elif [[ $NUMPY_OPTION =~ ^numpy=.* ]]; then 550 | PIP_NUMPY_OPTION="numpy==${NUMPY_OPTION#numpy=}" 551 | # Should version specs with 'numpy>=X' etc. ever be used, 552 | # use these as is: 553 | else 554 | PIP_NUMPY_OPTION="$NUMPY_OPTION" 555 | fi 556 | fi 557 | echo -e "\nInstalling $NUMPY_OPTION with conda was unsuccessful," \ 558 | "removing from conda install and adding $PIP_NUMPY_OPTION" \ 559 | " to PIP_DEPENDENCIES instead.\n" 560 | export PIP_DEPENDENCIES="$PIP_DEPENDENCIES $PIP_NUMPY_OPTION" 561 | # now that numpy will be installed via pip later, 562 | # remove it from CONDA_INSTALL: 563 | export CONDA_INSTALL=$(echo $CONDA_INSTALL | sed -e 's/numpy[a-zA-Z0-9.=]*\( \|$\)//g') 564 | } 565 | fi 566 | 567 | # ASTROPY 568 | if [[ ! -z $ASTROPY_VERSION ]]; then 569 | if [[ $ASTROPY_VERSION == dev* ]]; then 570 | : # Install at the bottom of this script 571 | elif [[ $ASTROPY_VERSION == pre* ]]; then 572 | # We use --no-pin to avoid installing other dependencies just yet 573 | retry_on_known_error $CONDA_INSTALL_COMMAND --no-pin $PYTHON_OPTION astropy 574 | if [[ -z $(pip list -o --pre | grep astropy | \ 575 | grep -E "[0-9]rc[0-9]|[0-9][ab][0-9]") ]]; then 576 | # We want to stop the script if there isn't a pre-release available, 577 | # as in that case it would be just another build using the stable 578 | # version. 579 | echo "Prerelease for astropy is not available, stopping test" 580 | travis_terminate 0 581 | fi 582 | elif [[ $ASTROPY_VERSION == stable ]]; then 583 | # We add astropy to the pin file to make sure it won't get downgraded 584 | echo "astropy ${LATEST_ASTROPY_STABLE}.*" >> $PIN_FILE 585 | 586 | if [[ $NO_PYTEST_ASTROPY == True ]]; then 587 | ASTROPY_OPTION="$LATEST_ASTROPY_STABLE" 588 | else 589 | ASTROPY_OPTION="$LATEST_ASTROPY_STABLE pytest-astropy" 590 | fi 591 | 592 | elif [[ $ASTROPY_VERSION == lts ]]; then 593 | # We add astropy to the pin file to make sure it won't get updated 594 | echo "astropy ${ASTROPY_LTS_VERSION}.*" >> $PIN_FILE 595 | ASTROPY_OPTION=$ASTROPY_LTS_VERSION 596 | else 597 | # We add astropy to the pin file to make sure it won't get updated 598 | echo "astropy ${ASTROPY_VERSION}.*" >> $PIN_FILE 599 | if [[ $(echo ${ASTROPY_VERSION} | cut -b 1) -ge 3 ]]; then 600 | ASTROPY_OPTION="$ASTROPY_VERSION pytest-astropy" 601 | else 602 | ASTROPY_OPTION=$ASTROPY_VERSION 603 | fi 604 | fi 605 | if [[ ! -z $ASTROPY_OPTION ]]; then 606 | retry_on_known_error $CONDA_INSTALL_COMMAND --no-pin $QUIET $PYTHON_OPTION $NUMPY_OPTION astropy=$ASTROPY_OPTION || { \ 607 | $PIP_FALLBACK && { \ 608 | echo "Installing astropy with conda was unsuccessful, using pip instead" 609 | $PIP_INSTALL astropy==$ASTROPY_OPTION 610 | if [[ -f $PIN_FILE ]]; then 611 | awk '{if ($1 != "astropy") print $0}' $PIN_FILE > /tmp/pin_file_temp 612 | mv /tmp/pin_file_temp $PIN_FILE 613 | fi;};} 614 | fi 615 | 616 | fi 617 | 618 | # SUNPY 619 | if [[ ! -z $SUNPY_VERSION ]]; then 620 | if [[ $SUNPY_VERSION == dev* ]]; then 621 | : # Install at the bottom of the script 622 | elif [[ $SUNPY_VERSION == pre* ]]; then 623 | # We use --no-pin to avoid installing other 624 | retry_on_known_error $CONDA_INSTALL_COMMAND --no-pin $PYTHON_OPTION sunpy 625 | if [[ -z $(pip list -o --pre | grep sunpy | \ 626 | grep -E "[0-9]rc[0-9]|[0-9][ab][0-9]") ]]; then 627 | # We want to stop the script if there isn't a pre-release available, 628 | # as in that case it would be just another build using the stable 629 | # version. 630 | echo "Prerelease for sunpy is not available, stopping test" 631 | travis_terminate 0 632 | fi 633 | elif [[ $SUNPY_VERSION == stable ]]; then 634 | SUNPY_OPTION=$LATEST_SUNPY_STABLE 635 | else 636 | # We add sunpy to the pin file to make sure it won't get updated 637 | echo "sunpy ${SUNPY_VERSION}.*" >> $PIN_FILE 638 | SUNPY_OPTION=$SUNPY_VERSION 639 | fi 640 | if [[ ! -z $SUNPY_OPTION ]]; then 641 | retry_on_known_error $CONDA_INSTALL_COMMAND --no-pin $QUIET $PYTHON_OPTION $NUMPY_OPTION sunpy=$SUNPY_OPTION || { \ 642 | $PIP_FALLBACK && { \ 643 | echo "Installing sunpy with conda was unsuccessful, using pip instead" 644 | $PIP_INSTALL sunpy==$SUNPY_OPTION 645 | if [[ -f $PIN_FILE ]]; then 646 | awk '{if ($1 != "sunpy") print $0}' $PIN_FILE > /tmp/pin_file_temp 647 | mv /tmp/pin_file_temp $PIN_FILE 648 | fi;};} 649 | fi 650 | 651 | fi 652 | 653 | 654 | # DOCUMENTATION DEPENDENCIES 655 | # build_sphinx needs sphinx and matplotlib (for plot_directive). 656 | if [[ $SETUP_CMD == *build_sphinx* ]] || [[ $SETUP_CMD == *build_docs* ]]; then 657 | # Check whether there are any version setting env variables, pin them if 658 | # there are (only need to deal with the case when they aren't listed in 659 | # CONDA_DEPENDENCIES, otherwise this was already dealt with) 660 | 661 | if [[ ! -z $MATPLOTLIB_VERSION ]]; then 662 | if [[ -z $(grep matplotlib $PIN_FILE) ]]; then 663 | echo "matplotlib ${MATPLOTLIB_VERSION}.*" >> $PIN_FILE 664 | fi 665 | fi 666 | 667 | 668 | # Temporary version limitation due to mpl segfaulting for the docs build 669 | # (issue tbd). sip needed to be added to the list of packages below to 670 | # be manually installed so this version pinning actually being taken 671 | # account 672 | if [[ -z $SIP_VERSION ]]; then 673 | echo "sip <4.19" >> $PIN_FILE 674 | fi 675 | 676 | if [[ ! -z $SPHINX_VERSION ]]; then 677 | if [[ -z $(grep sphinx $PIN_FILE) ]]; then 678 | echo "sphinx ${SPHINX_VERSION}.*" >> $PIN_FILE 679 | fi 680 | fi 681 | 682 | # We don't want to install everything listed in the PIN_FILE in this 683 | # section, but respect the pinned version of packages that are already 684 | # installed 685 | 686 | # Adding sip temporarily here, too to take into account the version 687 | # pinning added above 688 | conda list > /tmp/installed 689 | for package in sip matplotlib sphinx; do 690 | mv $PIN_FILE /tmp/pin_file_copy 691 | 692 | awk -v package=$package '{if ($1 == package) print $0}' /tmp/pin_file_copy > $PIN_FILE 693 | awk 'FNR==NR{a[$1]=$1;next} $1 in a{print $0}' /tmp/installed /tmp/pin_file_copy >> $PIN_FILE 694 | 695 | retry_on_known_error $CONDA_INSTALL $package && mv /tmp/pin_file_copy $PIN_FILE || { \ 696 | $PIP_FALLBACK && { \ 697 | echo "Installing $package with conda was unsuccessful, using pip instead." 698 | PIP_PACKAGE_VERSION=$(grep $package $PIN_FILE | awk '{print $2}') 699 | # Debugging.... 700 | echo "WHAT IS GOING ON HERE (TAKE 2)" 701 | conda info -a 702 | conda config --show 703 | conda list 704 | cat $PIN_FILE 705 | if [[ $(echo $PIP_PACKAGE_VERSION | cut -c 1) =~ $is_number ]]; then 706 | PIP_PACKAGE_VERSION='=='${PIP_PACKAGE_VERSION} 707 | elif [[ $(echo $PIP_PACKAGE_VERSION | cut -c 1-2) =~ $is_eq_number ]]; then 708 | PIP_PACKAGE_VERSION='='${PIP_PACKAGE_VERSION} 709 | fi 710 | $PIP_INSTALL ${package}${PIP_PACKAGE_VERSION} 711 | awk -v package=$package '{if ($1 != package) print $0}' /tmp/pin_file_copy > $PIN_FILE 712 | };} 713 | done 714 | 715 | if [[ $DEBUG == True ]]; then 716 | cat $PIN_FILE 717 | fi 718 | 719 | fi 720 | 721 | # ADDITIONAL DEPENDENCIES (can include optionals, too) 722 | if [[ ! -z $CONDA_DEPENDENCIES ]]; then 723 | 724 | retry_on_known_error $CONDA_INSTALL $CONDA_DEPENDENCIES $CONDA_DEPENDENCIES_FLAGS || { \ 725 | $PIP_FALLBACK && { \ 726 | # If there is a problem with conda install, try pip install one-by-one 727 | cp $PIN_FILE /tmp/pin_copy 728 | for package in $(echo $CONDA_DEPENDENCIES); do 729 | # We need to avoid other dependencies picked up from the pin file 730 | awk -v package=$package '{if ($1 == package) print $0}' /tmp/pin_copy > $PIN_FILE 731 | if [[ $DEBUG == True ]]; then 732 | cat $PIN_FILE 733 | fi 734 | retry_on_known_error $CONDA_INSTALL $package $CONDA_DEPENDENCIES_FLAGS || { \ 735 | echo "Installing the dependency $package with conda was unsuccessful, using pip instead." 736 | # We need to remove the problematic package from the pin 737 | # file, otherwise further conda install commands may fail, 738 | # too. Also we may need to limit the version installed by pip. 739 | PIP_PACKAGE_VERSION=$(awk '{print $2}' $PIN_FILE) 740 | 741 | # Deal with as for specific version, otherwise the limitation can be passed on as is 742 | if [[ $(echo $PIP_PACKAGE_VERSION | cut -c 1) =~ $is_number ]]; then 743 | PIP_PACKAGE_VERSION='=='${PIP_PACKAGE_VERSION} 744 | elif [[ $(echo $PIP_PACKAGE_VERSION | cut -c 1-2) =~ $is_eq_number ]]; then 745 | PIP_PACKAGE_VERSION='='${PIP_PACKAGE_VERSION} 746 | fi 747 | awk -v package=$package '{if ($1 != package) print $0}' /tmp/pin_copy > /tmp/pin_copy_temp 748 | mv /tmp/pin_copy_temp /tmp/pin_copy 749 | $PIP_INSTALL $package${PIP_PACKAGE_VERSION};}; 750 | done 751 | mv /tmp/pin_copy $PIN_FILE;};} 752 | fi 753 | 754 | # PARALLEL BUILDS 755 | if [[ $SETUP_CMD == *parallel* || $SETUP_CMD == *numprocesses* ]]; then 756 | $PIP_INSTALL pytest-xdist 757 | fi 758 | 759 | # OPEN FILES 760 | if [[ $SETUP_CMD == *open-files* ]]; then 761 | retry_on_known_error $CONDA_INSTALL psutil 762 | fi 763 | 764 | # NUMPY DEV and PRE 765 | 766 | # We now install Numpy dev - this has to be done last, otherwise conda might 767 | # install a stable version of Numpy as a dependency to another package, which 768 | # would override Numpy dev or pre. 769 | 770 | if [[ $NUMPY_VERSION == dev* ]]; then 771 | retry_on_known_error $CONDA_INSTALL_COMMAND $QUIET Cython 772 | $PIP_INSTALL git+https://github.com/numpy/numpy.git#egg=numpy --upgrade --no-deps 773 | fi 774 | 775 | if [[ $NUMPY_VERSION == pre* ]]; then 776 | $PIP_INSTALL --pre --upgrade numpy 777 | fi 778 | 779 | # MATPLOTLIB DEV 780 | 781 | # We now install Matplotlib dev - this has to be done last, otherwise conda might 782 | # install a stable version of matplotlib as a dependency to another package, which 783 | # would override matplotlib dev. 784 | 785 | if [[ $MATPLOTLIB_VERSION == dev* ]]; then 786 | $PIP_INSTALL git+https://github.com/matplotlib/matplotlib.git#egg=matplotlib --upgrade --no-deps 787 | fi 788 | 789 | if [[ $MATPLOTLIB_VERSION == pre* ]]; then 790 | $PIP_INSTALL --pre --upgrade --no-deps matplotlib 791 | fi 792 | 793 | 794 | # SCIPY_DEV 795 | 796 | # We now install Scipy dev - this has to be done last, otherwise conda might 797 | # install a stable version of matplotlib as a dependency to another package, which 798 | # would override matplotlib dev. 799 | 800 | if [[ $SCIPY_VERSION == dev* ]]; then 801 | retry_on_known_error $CONDA_INSTALL Cython 802 | 803 | $PIP_INSTALL git+https://github.com/scipy/scipy.git#egg=scipy --upgrade --no-deps 804 | fi 805 | 806 | if [[ $SCIPY_VERSION == pre* ]]; then 807 | $PIP_INSTALL --pre --upgrade --no-deps scipy 808 | fi 809 | 810 | 811 | # SCIKIT_LEARN DEV 812 | 813 | # We now install scikit-learn dev - this has to be done last, otherwise conda might 814 | # install a stable version of matplotlib as a dependency to another package, which 815 | # would override matplotlib dev. 816 | 817 | if [[ $SCIKIT_LEARN_VERSION == dev* ]]; then 818 | retry_on_known_error $CONDA_INSTALL Cython 819 | 820 | $PIP_INSTALL git+https://github.com/scikit-learn/scikit-learn.git#egg=sklearn --upgrade --no-deps 821 | fi 822 | 823 | if [[ $SCIKIT_LEARN_VERSION == pre* ]]; then 824 | $PIP_INSTALL --pre --upgrade --no-deps scikit-learn 825 | fi 826 | 827 | # PYTEST DEV 828 | 829 | if [[ $PYTEST_VERSION == dev* ]]; then 830 | $PIP_INSTALL iniconfig 831 | $PIP_INSTALL git+https://github.com/pytest-dev/pytest.git#egg=pytest --upgrade --no-deps 832 | fi 833 | 834 | # ASTROPY DEV and PRE 835 | 836 | # We now install Astropy dev - this has to be done last, otherwise conda might 837 | # install a stable version of Astropy as a dependency to another package, which 838 | # would override Astropy dev. Also, if we are installing Numpy dev, we need to 839 | # compile Astropy dev against Numpy dev. We need to include --no-deps to make 840 | # sure that Numpy doesn't get upgraded. 841 | 842 | if [[ $ASTROPY_VERSION == dev* ]]; then 843 | $PIP_INSTALL Cython jinja2 pytest-astropy pyerfa 844 | $PIP_INSTALL git+https://github.com/astropy/astropy.git#egg=astropy --upgrade --no-deps 845 | fi 846 | 847 | if [[ $ASTROPY_VERSION == pre* ]]; then 848 | $PIP_INSTALL --pre --upgrade --no-deps astropy 849 | fi 850 | 851 | # SUNPY DEV and PRE 852 | 853 | # We now install sunpy dev - this has to be done last, otherwise conda might 854 | # install a stable version of sunpy as a dependency to another package, which 855 | # would override sunpy dev. Also, if we are installing Numpy dev, we need to 856 | # compile sunpy dev against Numpy dev. We need to include --no-deps to make 857 | # sure that Numpy doesn't get upgraded. 858 | 859 | if [[ $SUNPY_VERSION == dev* ]]; then 860 | $PIP_INSTALL git+https://github.com/sunpy/sunpy.git#egg=sunpy --upgrade --no-deps 861 | fi 862 | 863 | if [[ $SUNPY_VERSION == pre* ]]; then 864 | $PIP_INSTALL --pre --upgrade --no-deps sunpy 865 | fi 866 | 867 | 868 | 869 | # ASTROPY STABLE 870 | 871 | # Due to recent instability in conda, this workaround ensures that we use the 872 | # latest stable version of astropy. 873 | 874 | if [[ $ASTROPY_VERSION == stable ]]; then 875 | old_astropy=$(python -c "from distutils.version import LooseVersion;\ 876 | import astropy; import os;\ 877 | print(LooseVersion(astropy.__version__) <\ 878 | LooseVersion(os.environ['LATEST_ASTROPY_STABLE']))") 879 | 880 | if [[ $old_astropy == True ]]; then 881 | # First remove astropy from conda to make sure the version installed 882 | # by pip will be used. We use --force to make sure things that depend 883 | # on astropy don't cause issues or get uninstalled. 884 | conda remove astropy --force 885 | $PIP_INSTALL --upgrade --no-deps --ignore-installed astropy==$LATEST_ASTROPY_STABLE pytest-astropy 886 | fi 887 | fi 888 | 889 | # PIP DEPENDENCIES 890 | 891 | # We finally install the dependencies listed in PIP_DEPENDENCIES. We do this 892 | # after installing the Numpy versions of Numpy or Astropy. If we didn't do this, 893 | # then calling pip earlier could result in the stable version of astropy getting 894 | # installed, and then overritten later by the dev version (which would waste 895 | # build time) 896 | 897 | if [[ ! -z $PIP_DEPENDENCIES ]]; then 898 | # The astropy testrunner is not compatible with coverage 5.0+, thus we limit the version 899 | if [[ ! -z $(echo $PIP_DEPENDENCIES | grep -i pytest-cov) ]]; then 900 | if [[ -z $(echo $PIP_DEPENDENCIES | grep -i coverage) ]]; then 901 | PIP_DEPENDENCIES=${PIP_DEPENDENCIES}" coverage<5" 902 | fi 903 | fi 904 | 905 | $PIP_INSTALL $PIP_DEPENDENCIES $PIP_DEPENDENCIES_FLAGS 906 | fi 907 | 908 | 909 | # COVERAGE DEPENDENCIES 910 | 911 | # Both cpp-coveralls and coveralls install a 'coveralls' command, but we want 912 | # the one from the coveralls package to always take precedence, so we have to 913 | # install this now in case the user installs cpp-coveralls via PIP_DEPENDENCIES. 914 | 915 | if [[ $SETUP_CMD == *coverage* ]]; then 916 | # We install requests since it's required by coveralls. 917 | # Limit the version number as the astropy testrunner is not compatible with v5 918 | $PIP_INSTALL coveralls codecov 'coverage<5' requests 919 | fi 920 | 921 | if [[ $SETUP_CMD == *-cov* ]]; then 922 | $PIP_INSTALL coveralls codecov pytest-cov 'coverage<5' 923 | fi 924 | 925 | 926 | # SPHINX BUILD MPL FONT CACHING WORKAROUND 927 | 928 | # This is required to avoid Sphinx build failures due to a warning that 929 | # comes from the mpl FontManager(). The workaround is to initialize the 930 | # cache before starting the tests/docs build. See details in 931 | # https://github.com/matplotlib/matplotlib/issues/5836 932 | 933 | if [[ $SETUP_CMD == *build_sphinx* ]] || [[ $SETUP_CMD == *build_docs* ]]; then 934 | python -c "import matplotlib.pyplot" 935 | fi 936 | 937 | # DEBUG INFO 938 | 939 | if [[ $DEBUG == True ]]; then 940 | # include debug information about the current conda install 941 | # There was once an install of a _license package here, which does not 942 | # exist for python >=3.7 943 | conda info -a 944 | fi 945 | 946 | if [[ ! -z $ASTROPY_VERSION ]]; then 947 | # Force uninstall hypothesis if it's silently installed as an upstream 948 | # dependency as the astropy <2.0.3 machinery is incompatible with 949 | # it. But if it's an explicit dependency in PIP_DEPENDENCIES or 950 | # CONDA_DEPENDENCIES then we only issue a warning. 951 | # https://github.com/astropy/astropy/issues/6919 952 | 953 | old_astropy=$(python -c "from distutils.version import LooseVersion;\ 954 | import astropy; \ 955 | print(LooseVersion(astropy.__version__) <\ 956 | LooseVersion('2.0.3'))") 957 | if [[ $(echo $CONDA_DEPENDENCIES $PIP_DEPENDENCIES | grep hypothesis) ]]; then 958 | no_explicit_dependency=false 959 | echo "WARNING: the package 'hypothesis' is incompatible with the Astropy testing mechanism prior version v2.0.3, expect issues during doctesting." 960 | fi 961 | 962 | if [[ $old_astropy == True ]] && $no_explicit_dependency; then 963 | conda remove --force hypothesis || true 964 | fi 965 | fi 966 | 967 | set +ex 968 | -------------------------------------------------------------------------------- /travis/setup_python.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash -ex 2 | 3 | # Script to set up Python using native platform tools rather than conda 4 | 5 | echo "==================== Starting executing ci-helpers scripts =====================" 6 | 7 | if [[ -z $PYTHON_VERSION ]]; then 8 | echo "PYTHON_VERSION needs to be set"; 9 | exit 1; 10 | fi 11 | 12 | if [[ $PYTHON_VERSION == 3.6 ]]; then 13 | FULL_PYTHON_VERSION=3.6.8; 14 | elif [[ $PYTHON_VERSION == 3.7 ]]; then 15 | FULL_PYTHON_VERSION=3.7.9; 16 | elif [[ $PYTHON_VERSION == 3.8 ]]; then 17 | FULL_PYTHON_VERSION=3.8.6; 18 | elif [[ $PYTHON_VERSION == 3.9 ]]; then 19 | FULL_PYTHON_VERSION=3.9.0; 20 | fi 21 | 22 | if [[ $TRAVIS_OS_NAME == windows ]]; then 23 | CONDENSED_PYTHON_VERSION="${PYTHON_VERSION//.}" 24 | choco install --no-progress python --version $FULL_PYTHON_VERSION; 25 | export PATH="/c/Python$CONDENSED_PYTHON_VERSION:/c/Python$CONDENSED_PYTHON_VERSION/Scripts:$PATH" 26 | python -m venv ~/python; 27 | source ~/python/Scripts/activate; 28 | python -m pip install --upgrade pip; 29 | fi 30 | 31 | if [[ $TRAVIS_OS_NAME == osx ]]; then 32 | 33 | wget https://www.python.org/ftp/python/$FULL_PYTHON_VERSION/python-$FULL_PYTHON_VERSION-macosx10.9.pkg 34 | sudo installer -pkg python-$FULL_PYTHON_VERSION-macosx10.9.pkg -target / 35 | /Applications/Python\ $PYTHON_VERSION/Install\ Certificates.command 36 | python$PYTHON_VERSION -m venv ~/python; 37 | source ~/python/bin/activate; 38 | python -m pip install --upgrade pip; 39 | fi 40 | 41 | echo "Checking default python version:" 42 | echo `python --version` 43 | 44 | echo "================= Returning executing local .travis.yml script =================" 45 | -------------------------------------------------------------------------------- /utils/import_submodules.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import pkgutil 3 | import importlib 4 | 5 | 6 | def import_submodules(package, skip_modules='', recursive=True): 7 | """ 8 | Import all submodules of a module, recursively, including subpackages 9 | 10 | Original form is from StackOverflow: 11 | https://stackoverflow.com/a/25562415 12 | 13 | Parameters 14 | ---------- 15 | 16 | package : str or module name 17 | name of the package or the imported package to import the 18 | submodules from. 19 | skip_modules : comma separated str 20 | Comma separated string of module names to be skipped. 21 | recursive : bool 22 | Import only the top level if recursive is False. 23 | """ 24 | 25 | skip_modules_list = skip_modules.split(',') 26 | 27 | if isinstance(package, str): 28 | package = importlib.import_module(package) 29 | results = {} 30 | errors = [] 31 | 32 | for loader, name, is_pkg in pkgutil.walk_packages(package.__path__): 33 | full_name = package.__name__ + '.' + name 34 | 35 | # Sometimes ``walk_packages`` pickes up fuller namespaces thus we may 36 | # miss skipping the modules we intended to skip 37 | all_names = full_name.split('.') 38 | 39 | name_skip = set(all_names).intersection(skip_modules_list) 40 | is_non_public = any((n.startswith('_') for n in (all_names[-1], name))) 41 | full_name_skip = any([full_name.startswith(m) for m in skip_modules_list]) 42 | 43 | if is_non_public or full_name_skip or name_skip: 44 | continue 45 | try: 46 | results[full_name] = importlib.import_module(full_name) 47 | if recursive and is_pkg: 48 | result, error = import_submodules(full_name, skip_modules) 49 | results.update(result) 50 | errors.append(error) 51 | except ImportError as errs: 52 | print("Cannot import {}".format(full_name)) 53 | errors.append(str(errs)) 54 | 55 | return results, errors 56 | 57 | 58 | if __name__ == "__main__": 59 | if len(sys.argv) < 2: 60 | raise IndexError("specify a packagename to import.") 61 | package_name = sys.argv[1] 62 | if len(sys.argv) >= 3: 63 | skip_modules = sys.argv[2] 64 | else: 65 | skip_modules = '' 66 | 67 | results, errors = import_submodules(package_name, skip_modules) 68 | 69 | if len(errors) > 0: 70 | raise ImportError("Cannot import from {} module(s).".format(len(errors))) 71 | --------------------------------------------------------------------------------