├── LICENSE ├── README.md ├── bin ├── compile ├── detect ├── release ├── steps │ ├── conda_compile │ └── hooks │ │ ├── post_compile │ │ └── pre_compile └── utils └── test ├── conda-requirements.txt └── requirements.txt /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License: 2 | 3 | Copyright (C) 2013 Kenneth Reitz 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 6 | 7 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 8 | 9 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 10 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | Conda Buildpack 2 | =============== 3 | 4 | This is a [Heroku Buildpack](https://devcenter.heroku.com/articles/buildpacks) for [Conda](http://conda.pydata.org/), the Python distribution for scientific computing by Continuum Analytics. 5 | 6 | ----------------------- 7 | 8 | **Note:** Heroku now recommends using their beta Docker support to achieve [similar results](https://github.com/heroku-examples/python-miniconda). 9 | 10 | ----------------------- 11 | 12 | This buildpack enables the installation of binary packages through the 13 | open source [conda](http://conda.pydata.org/) application. Conda is 14 | recognized as being core to Continuum's Anaconda Scientific Python distro 15 | but it's also at the heart of the lighter weight 16 | [Miniconda](http://conda.pydata.org/miniconda.html) distro which we use 17 | here to install _only_ the binary packages we need for our apps deployed 18 | on Heroku. 19 | 20 | To control what binary packages are installed by conda, supply a 21 | `conda-requirements.txt` file (which can be created by capturing the output 22 | of `conda list -e` for your working conda environment). 23 | Like when using the standard buildpack for python from Heroku, you can also 24 | still supply a `requirements.txt` file for [pip](https://github.com/pypa/pip) 25 | to process. In this way, you can install binary packages via conda for 26 | everything you can and still use pip for anything you can't. 27 | Usage 28 | ----- 29 | 30 | Example usage: 31 | 32 | ```console 33 | $ ls 34 | Procfile conda-requirements.txt numbercrunch.py 35 | 36 | $ heroku create --buildpack https://github.com/kennethreitz/conda-buildpack.git 37 | 38 | $ git push heroku master 39 | ... 40 | -----> Fetching custom git buildpack... done 41 | -----> Python/Miniconda app detected 42 | -----> Preparing Python/Miniconda Environment (3.5.2) 43 | installing: python-2.7.6-2 ... 44 | -----> Installing dependencies using Conda 45 | Fetching packages ... 46 | bitarray-0.8.1 100% |###############################| Time: 0:00:00 17.53 MB/s00 B/s 47 | dateutil-2.1-p 100% |###############################| Time: 0:00:00 2.29 MB/s00 B/s 48 | h5py-2.3.0-np1 100% |###############################| Time: 0:00:00 13.49 MB/s00 B/s 49 | hdf5-1.8.9-1.t 100% |###############################| Time: 0:00:00 12.20 MB/s00 B/s 50 | libpng-1.5.13- 100% |###############################| Time: 0:00:00 8.05 MB/s00 B/s 51 | llvm-3.3-0.tar 100% |###############################| Time: 0:00:03 10.65 MB/s00 B/s 52 | llvmpy-0.12.6- 100% |###############################| Time: 0:00:00 9.65 MB/s00 B/s 53 | nltk-2.0.4-np1 100% |###############################| Time: 0:00:00 6.26 MB/s00 B/s 54 | numba-0.13.2-n 100% |###############################| Time: 0:00:00 11.54 MB/s00 B/s 55 | numexpr-2.3.1- 100% |###############################| Time: 0:00:00 6.80 MB/s00 B/s 56 | numpy-1.8.1-py 100% |###############################| Time: 0:00:00 8.82 MB/s00 B/s 57 | pandas-0.14.0- 100% |###############################| Time: 0:00:00 9.90 MB/s00 B/s 58 | pyside-1.2.1-p 100% |###############################| Time: 0:00:00 6.00 MB/s00 B/s 59 | pytables-3.1.1 100% |###############################| Time: 0:00:00 9.24 MB/s00 B/s 60 | pytz-2014.3-py 100% |###############################| Time: 0:00:00 1.54 MB/s00 B/s 61 | qt-4.8.5-0.tar 100% |###############################| Time: 0:00:01 17.36 MB/s00 B/s 62 | reportlab-3.1. 100% |###############################| Time: 0:00:00 4.87 MB/s00 B/s 63 | scikit-image-0 100% |###############################| Time: 0:00:01 12.81 MB/s00 B/s 64 | scikit-learn-0 100% |###############################| Time: 0:00:00 8.70 MB/s00 B/s 65 | scipy-0.14.0-n 100% |###############################| Time: 0:00:02 15.16 MB/s00 B/s 66 | shiboken-1.2.1 100% |###############################| Time: 0:00:00 3.33 MB/s00 B/s 67 | six-1.6.1-py27 100% |###############################| Time: 0:00:00 12.54 MB/s00 B/s 68 | ``` 69 | 70 | You can also add it to upcoming builds of an existing application: 71 | 72 | ```console 73 | $ heroku config:add BUILDPACK_URL=https://github.com/kennethreitz/conda-buildpack.git 74 | ``` 75 | 76 | 77 | Simple test: 78 | 79 | ```python 80 | >>> bitarray.test() 81 | bitarray is installed in: /app/.heroku/anaconda/lib/python2.7/site-packages/bitarray 82 | bitarray version: 0.8.0 83 | 2.7.3 |Continuum Analytics, Inc.| (default, Feb 25 2013, 18:46:31) 84 | [GCC 4.1.2 20080704 (Red Hat 4.1.2-52)] 85 | ................................................................................................................................. 86 | ---------------------------------------------------------------------- 87 | Ran 129 tests in 1.375s 88 | 89 | OK 90 | 91 | ``` 92 | 93 | ## Fair Warning 94 | 95 | Heroku limits the final application footprint (slug) size to 300MB. Start small. In case the slug size limit is exceeded, deleting the build cache through the [heroku-repo plugin](https://github.com/heroku/heroku-repo#purge_cache) might help. 96 | 97 | ॐ 98 | -------------------------------------------------------------------------------- /bin/compile: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # Usage: 4 | # 5 | # $ bin/compile 6 | 7 | 8 | # Fail fast and fail hard. 9 | set -eo pipefail 10 | 11 | # Prepend proper path for virtualenv hackery. This will be deprecated soon. 12 | export PATH=:/usr/local/bin:$PATH 13 | 14 | # Paths. 15 | BIN_DIR=$(cd $(dirname $0); pwd) # absolute path 16 | ROOT_DIR=$(dirname $BIN_DIR) 17 | BUILD_DIR=$1 18 | CACHE_DIR=$2 19 | 20 | CACHED_DIRS=".heroku" 21 | 22 | # Static configurations for virtualenv caches. 23 | PROFILE_PATH="$BUILD_DIR/.profile.d/miniconda.sh" 24 | 25 | # Sanitizing environment variables. 26 | unset GIT_DIR PYTHONHOME PYTHONPATH LD_LIBRARY_PATH LIBRARY_PATH 27 | 28 | # We'll need to send these statics to other scripts we `source`. 29 | export BUILD_DIR CACHE_DIR BIN_DIR PROFILE_PATH 30 | 31 | # Syntax sugar. 32 | source $BIN_DIR/utils 33 | 34 | # Directory Hacks for path consistiency. 35 | APP_DIR='/app' 36 | TMP_APP_DIR=$CACHE_DIR/tmp_app_dir 37 | 38 | # Copy Anvil app dir to temporary storage... 39 | mkdir -p $TMP_APP_DIR 40 | deep-mv $APP_DIR $TMP_APP_DIR 41 | 42 | # Copy Application code in. 43 | deep-mv $BUILD_DIR $APP_DIR 44 | 45 | # Set new context. 46 | ORIG_BUILD_DIR=$BUILD_DIR 47 | BUILD_DIR=$APP_DIR 48 | 49 | # Prepend proper path buildpack use. 50 | export PATH=$BUILD_DIR/.heroku/miniconda/bin:$PATH 51 | export PYTHONUNBUFFERED=1 52 | export LANG=en_US.UTF-8 53 | 54 | # Switch to the repo's context. 55 | cd $BUILD_DIR 56 | 57 | # Experimental pre_compile hook. 58 | source $BIN_DIR/steps/hooks/pre_compile 59 | 60 | # ### The Cache 61 | mkdir -p $CACHE_DIR 62 | 63 | # Restore old artifacts from the cache. 64 | for dir in $CACHED_DIRS; do 65 | cp -R $CACHE_DIR/$dir . &> /dev/null || true 66 | done 67 | 68 | set +e 69 | # Create set-aside `.heroku` folder. 70 | mkdir .heroku &> /dev/null 71 | set -e 72 | 73 | # Make profile.d directory. 74 | mkdir -p $(dirname $PROFILE_PATH) 75 | 76 | # Actuall do the conda steps. 77 | source $BIN_DIR/steps/conda_compile 78 | 79 | # ### Finalize 80 | # 81 | 82 | # Store new artifacts in cache. 83 | for dir in $CACHED_DIRS; do 84 | rm -rf $CACHE_DIR/$dir 85 | cp -R $dir $CACHE_DIR/ 86 | done 87 | 88 | # Set context environment variables. 89 | set-env PATH '$HOME/.heroku/miniconda/bin:$PATH' 90 | set-env PYTHONUNBUFFERED true 91 | set-default-env LANG en_US.UTF-8 92 | set-default-env PYTHONHASHSEED random 93 | set-default-env PYTHONPATH /app/ 94 | 95 | # Experimental post_compile hook. 96 | source $BIN_DIR/steps/hooks/post_compile 97 | 98 | # ### Fin. 99 | 100 | deep-mv $BUILD_DIR $ORIG_BUILD_DIR 101 | deep-mv $TMP_APP_DIR $APP_DIR 102 | -------------------------------------------------------------------------------- /bin/detect: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # This script serves as the 4 | # [**Python Buildpack**](https://github.com/heroku/heroku-buildpack-python) 5 | # detector. 6 | # 7 | # A [buildpack](http://devcenter.heroku.com/articles/buildpacks) is an 8 | # adapter between a Python application and Heroku's runtime. 9 | 10 | # ## Usage 11 | # Compiling an app into a slug is simple: 12 | # 13 | # $ bin/detect 14 | 15 | BUILD_DIR=$1 16 | 17 | # Exit early if app is clearly not Python. 18 | if [ ! -f $BUILD_DIR/requirements.txt ] && [ ! -f $BUILD_DIR/conda-requirements.txt ] && [ ! -f $BUILD_DIR/setup.py ]; then 19 | exit 1 20 | fi 21 | 22 | echo Python/Conda 23 | -------------------------------------------------------------------------------- /bin/release: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | # bin/release 3 | 4 | cat < $HOME/.heroku/miniconda/conda-meta/pinned 11 | echo "added pinned file in $HOME/.heroku/miniconda/conda-meta/pinned" 12 | conda install nomkl 13 | 14 | 15 | if [ -f conda-requirements.txt ]; then 16 | puts-step "Installing dependencies using Conda" 17 | conda install --file conda-requirements.txt --yes | indent 18 | fi 19 | 20 | if [ -f requirements.txt ]; then 21 | puts-step "Installing dependencies using Pip" 22 | pip install -r requirements.txt --exists-action=w --allow-all-external | indent 23 | fi 24 | 25 | # Clean up the installation environment . 26 | conda clean -pt --yes > /dev/null 27 | -------------------------------------------------------------------------------- /bin/steps/hooks/post_compile: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | if [ -f bin/post_compile ]; then 4 | echo "-----> Running post-compile hook" 5 | chmod +x bin/post_compile 6 | bin/post_compile 7 | fi -------------------------------------------------------------------------------- /bin/steps/hooks/pre_compile: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | if [ -f bin/pre_compile ]; then 4 | echo "-----> Running pre-compile hook" 5 | chmod +x bin/pre_compile 6 | bin/pre_compile 7 | fi -------------------------------------------------------------------------------- /bin/utils: -------------------------------------------------------------------------------- 1 | shopt -s extglob 2 | 3 | [ $(uname) == "Darwin" ] && SED_FLAG='-l' || SED_FLAG='-u' 4 | 5 | # Syntax sugar. 6 | indent() { 7 | RE="s/^/ /" 8 | sed $SED_FLAG "$RE" 9 | } 10 | 11 | # Clean up pip output 12 | cleanup() { 13 | sed $SED_FLAG -e 's/\.\.\.\+/.../g' | sed $SED_FLAG '/already satisfied/Id' | sed $SED_FLAG -e '/Overwriting/Id' | sed $SED_FLAG -e '/python executable/Id' | sed $SED_FLAG -e '/no previously-included files/Id' 14 | } 15 | 16 | # Buildpack Steps. 17 | function puts-step (){ 18 | echo "-----> $@" 19 | } 20 | 21 | # Buildpack Warnings. 22 | function puts-warn (){ 23 | echo " ! $@" 24 | } 25 | 26 | # Usage: $ set-env key value 27 | function set-env (){ 28 | echo "export $1=$2" >> $PROFILE_PATH 29 | } 30 | 31 | # Usage: $ set-default-env key value 32 | function set-default-env (){ 33 | echo "export $1=\${$1:-$2}" >> $PROFILE_PATH 34 | } 35 | 36 | # Usage: $ set-default-env key value 37 | function un-set-env (){ 38 | echo "unset $1" >> $PROFILE_PATH 39 | } 40 | 41 | # Does some serious copying. 42 | function deep-cp (){ 43 | find -H $1 -maxdepth 1 -name '.*' -a \( -type d -o -type f -o -type l \) -exec cp -a '{}' $2 \; 44 | cp -r $1/!(tmp) $2 45 | # echo copying $1 to $2 46 | } 47 | 48 | # Does some serious moving. 49 | function deep-mv (){ 50 | deep-cp $1 $2 51 | 52 | rm -fr $1/!(tmp) 53 | find -H $1 -maxdepth 1 -name '.*' -a \( -type d -o -type f -o -type l \) -exec rm -fr '{}' \; 54 | } 55 | -------------------------------------------------------------------------------- /test/conda-requirements.txt: -------------------------------------------------------------------------------- 1 | scipy 2 | numpy 3 | bitarray 4 | reportlab 5 | pandas 6 | -------------------------------------------------------------------------------- /test/requirements.txt: -------------------------------------------------------------------------------- 1 | requests 2 | httpbin 3 | gunicorn --------------------------------------------------------------------------------