├── .gitignore ├── mkcopip ├── copipoff.sh ├── bash_condarc ├── cactivate ├── copipon.sh ├── COPYING.txt ├── copip.py └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | *~ 2 | .#* 3 | [#]*# 4 | .DS_Store 5 | __pycache__ 6 | .cache 7 | .coverage 8 | .ipynb_checkpoints 9 | -------------------------------------------------------------------------------- /mkcopip: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | if __name__ == '__main__': 4 | import sys 5 | import copip 6 | sys.exit(copip.main(sys.argv[1:])) 7 | -------------------------------------------------------------------------------- /copipoff.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # Deactivate a developer environment overlay. 4 | 5 | export PATH=$_PATH_COPIP_OLD 6 | export PIP_USER=$_PIP_USER_COPIP_OLD 7 | export PREFIX=$_PREFIX_COPIP_OLD 8 | 9 | unset PYTHONUSERBASE 10 | -------------------------------------------------------------------------------- /bash_condarc: -------------------------------------------------------------------------------- 1 | # RC file for conda environment subshells 2 | 3 | source ~/.bashrc 4 | 5 | # >>> conda initialize >>> 6 | # !! Contents within this block are managed by 'conda init' !! 7 | __conda_setup="$('/Users/fperez/local/conda/bin/conda' 'shell.bash' 'hook' 2> /dev/null)" 8 | if [ $? -eq 0 ]; then 9 | eval "$__conda_setup" 10 | else 11 | if [ -f "/Users/fperez/local/conda/etc/profile.d/conda.sh" ]; then 12 | . "/Users/fperez/local/conda/etc/profile.d/conda.sh" 13 | else 14 | export PATH="/Users/fperez/local/conda/bin:$PATH" 15 | fi 16 | fi 17 | unset __conda_setup 18 | # <<< conda initialize <<< 19 | 20 | 21 | #source activate $_CONDA_ENV_NAME 22 | conda activate $_CONDA_ENV_NAME 23 | 24 | if [ $? -ne 0 ]; then 25 | echo "*** ERROR activating environment, exiting subshell." 26 | exit $? 27 | fi 28 | -------------------------------------------------------------------------------- /cactivate: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | # 3 | # Replacement for conda's "activate" file, that instead starts a subshell 4 | # for the new environment. 5 | 6 | # Location of this hardcoded for now. If this became standard conda practice, 7 | # it could be replaced with `conda info --root`/bash_condarc 8 | BASH_CONDARC="$HOME/dev/copip/bash_condarc" 9 | 10 | _CONDA_ENV_NAME=${1-root} 11 | 12 | # We are going to transfer control to a new subshell, so that getting back to 13 | # the previous envirionment is possible just by exiting the subshell. The 14 | # subshell will then call "source activate" within it. 15 | # 16 | # We set the name of the new conda env so we can call "source activate $ENV" 17 | # correctly, preserving compatibility with how conda currently works. 18 | 19 | export _CONDA_ENV_NAME 20 | echo "*** Starting sub-shell for conda env: $_CONDA_ENV_NAME" 21 | bash --rcfile $BASH_CONDARC 22 | unset _CONDA_ENV_NAME 23 | -------------------------------------------------------------------------------- /copipon.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # Activate a conda/pip developer environment overlay. 4 | 5 | PREFIX=$CONDA_PREFIX/copip 6 | 7 | # Before further changes, save some variables we need to restore on deactivation. 8 | export _PATH_COPIP_OLD=$PATH 9 | export _PIP_USER_COPIP_OLD=$PIP_USER 10 | export _PREFIX_COPIP_OLD=$PREFIX 11 | 12 | # Make pip *always* install as if --user had been typed, and then configure 13 | # various variables so packages installed this way are found for execution, use 14 | # by Jupyter, etc. 15 | export PIP_USER=True 16 | export PREFIX 17 | export PYTHONUSERBASE=$PREFIX 18 | export PATH=$PREFIX/bin:$PATH 19 | # Note - ideally JUPYTER_PATH wouldn't need to be set separately of other Python 20 | # variables, as e.g. JupyterLab extensions can be pip-installed. But the process 21 | # of finding their non-python pieces is complex and how to do it without extra 22 | # info isn't settled, so for now we need an extra explicit variable. 23 | # See https://github.com/jupyter/jupyter_core/pull/209 for lots of details... 24 | export JUPYTER_PATH=$PREFIX/share/jupyter:$JUPYTER_PATH 25 | 26 | echo "*** Environment developer overlay active at PREFIX=$PREFIX" # dbg 27 | 28 | # TODO: This will require some extra utilities if we want to also manage multiple 29 | # other environment variables. For later. 30 | #export_paths "$PREFIX" 31 | -------------------------------------------------------------------------------- /COPYING.txt: -------------------------------------------------------------------------------- 1 | Copyright 2017, Fernando Perez 2 | 3 | Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 4 | 5 | 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 6 | 7 | 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 8 | 9 | 3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. 10 | 11 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 12 | -------------------------------------------------------------------------------- /copip.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | """ 3 | Create a conda-pip 'copip' development overlay. 4 | 5 | Usage: 6 | mkcopip env_name 7 | """ 8 | 9 | # Stdlib imports 10 | 11 | import os 12 | import sys 13 | import typing as T 14 | 15 | from pathlib import Path 16 | from subprocess import check_output as sh 17 | 18 | # Config global constants 19 | CONDA_BASE = Path(sh(['conda', 'info', '--base']).decode().strip())/'envs' 20 | COPIP_DIR = Path(__file__).parent 21 | COPIP_ON = Path('copipon.sh') 22 | COPIP_OFF = Path('copipoff.sh') 23 | 24 | 25 | # Function definitions 26 | def main(args: T.Optional[list]=None) -> int: 27 | if args is None: 28 | args = sys.argv[1:] 29 | 30 | try: 31 | ename = args[0] 32 | except IndexError: 33 | print(__doc__, file=sys.stderr) 34 | return 64 35 | 36 | # Create directories for holding installed files and env. config 37 | copip_dir = CONDA_BASE/ename/'copip' 38 | acti_dir = CONDA_BASE/ename/'etc/conda/activate.d' 39 | deac_dir = CONDA_BASE/ename/'etc/conda/deactivate.d' 40 | 41 | if not (CONDA_BASE/ename).is_dir(): 42 | print(f"Environment {ename} doesn't exist, exiting.", file=sys.stderr) 43 | return 64 44 | 45 | for d in [copip_dir, acti_dir, deac_dir]: 46 | d.mkdir(parents=True, exist_ok=True) 47 | 48 | # Symlink env. config scripts inside conda activ/deact directories 49 | for script, cdir in [(COPIP_ON, acti_dir), (COPIP_OFF, deac_dir)]: 50 | dest = cdir/script 51 | if not dest.is_file(): 52 | os.link(COPIP_DIR/script, dest) 53 | 54 | print(f"Environment dev overlay `{ename}` ready at `{copip_dir}`") 55 | 56 | return 0 57 | 58 | 59 | # Unit tests 60 | def test_no_args(): 61 | assert main([]) == 64 62 | 63 | 64 | def test_noenv(): 65 | assert main(['__BADENV_NAME_zyxw__']) == 64 66 | 67 | 68 | def test_normal(): 69 | import functools 70 | import subprocess 71 | 72 | sh = functools.partial(subprocess.run, shell=True, check=True) 73 | 74 | ename = '__tmp_copip_env__' 75 | copip = CONDA_BASE/ename 76 | sh(f"conda create -n {ename} --yes") 77 | try: 78 | assert main([ename]) == 0 79 | assert copip.is_dir() 80 | 81 | for script, cdir in [(COPIP_ON, 'activate.d'), 82 | (COPIP_OFF, 'deactivate.d')]: 83 | src = CONDA_BASE/ename/'etc/conda'/cdir/script 84 | assert src.is_file() 85 | assert src.samefile(COPIP_DIR/script) 86 | finally: 87 | sh(f"conda remove -n {ename} --all --yes") 88 | 89 | 90 | # Main entry point 91 | if __name__ == '__main__': 92 | sys.exit(main(sys.argv[1:])) 93 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # copip: conda-pip environment development overlays 2 | 3 | A tool to better manage PyPI and source packages (installed using `pip`, with or without `-e .`) in a conda-based workflow. 4 | 5 | The approach taken is to create a separate filesystem area where *all* non-conda installations go. Then, we "hijack" the `--user` flag of Python's installation process by *always* using `--user` during any pip-based installation. When using the conda root environment such area already exists: `~/.local/...` (on *nix, there's a Windows-specific location too). When using custom conda environments, we set the `$PYTHONUSERBASE` variable to point to a location named the same as the conda environment but separate in the filesystem, and set `$PATH` accordingly. 6 | 7 | This avoids the problem of conflicts arising after packages have been added to a conda environment and a conda update potentially overwrites them. 8 | 9 | I have used this a fair amount for a few months, and so far it hasn't failed me. As usual, caveat emptor. 10 | 11 | 12 | ## Setup 13 | 14 | There's no packaging/installation yet. For now, you need to: 15 | 16 | - symlink/copy the `mkcopip` command somewhere in your `$PATH`. 17 | 18 | 19 | ## Usage 20 | 21 | Assuming your paths are properly set up as above, you can use this to manage an overlay on your conda env `foo` by running 22 | 23 | ```bash 24 | mkcopip foo 25 | ``` 26 | 27 | Since this tool is designed to direct all pip installs to the user overlay, it sets the environment variable `PIP_USER=True` unconditionally on environment activation. 28 | 29 | If you want to have the same behavior in your root conda env, you should: 30 | 31 | - Also set `PIP_USER=True` in your regular shell config file ( `~/.bashrc` or equivalent). 32 | 33 | - Ensure that `~/.local/bin` is in your `PATH`, so that script entry points installed by new packages are also found first. 34 | 35 | 36 | ## Todo 37 | 38 | Besides packaging/configurability, the key thing to test next is usage with complex C extensions built this way. That requires setting lots more environment variables related to compilers, linkers, etc. I have old code for that which can be reused if there's interest. 39 | 40 | 41 | ## Requirements 42 | 43 | The 'driver' script is Python 3.6-only. This would be easy to avoid, but I wanted to use it as an opportunity to play with some Python 3.6-specific features, like f-strings and standard library support for pathlib. The resulting code is indeed nicer, so I'm keeping it that way. 44 | 45 | 46 | ## Advanced: `cactivate` with a sub-shell 47 | 48 | I personally prefer to run my environments in a brand-new subshell. By having a subshell, I'm guaranteed to get back to my parent environment 100% unmodified once I'm done with the environment I activated, since exiting a sub-shell destroys any env. variables or other context. This is much more robust than trusting that `source deactivate` will do the right thing in all cases. 49 | 50 | For this, I use the `cactivate` shell script included here helps, but I haven't made it portable yet. If you want to test this approach, you'll need to: 51 | 52 | - modify the `cactivate` shell script to include your path to the `bash_condarc` path on your system. 53 | 54 | - symlink the `cactivate` script somewhere in your `$PATH`. 55 | 56 | Then, when you want to use environment `foo`, instead of `source activate foo`, you should run `source cactivate foo` (note the 'c'). This will activate your environment in a subshell, which you can terminate to exit the environment (no need to run `source deactivate`, you simply exit the subshell). 57 | 58 | 59 | ## License 60 | 61 | Released under the terms of the 3-clause ("new") BSD license. 62 | --------------------------------------------------------------------------------