├── .gitignore ├── LICENSE.md ├── Makefile ├── README.md ├── all-spark-notebook ├── Dockerfile ├── README.md └── kernel.json ├── cuda ├── Dockerfile └── README.md ├── datascience-notebook ├── Dockerfile └── README.md ├── examples └── make-deploy │ ├── Dockerfile │ ├── Makefile │ ├── README.md │ ├── letsencrypt.makefile │ ├── self-signed.makefile │ ├── softlayer.makefile │ └── virtualbox.makefile ├── minimal-kernel ├── Dockerfile └── README.md ├── minimal-notebook ├── Dockerfile ├── README.md ├── jupyter_notebook_config.py └── start-notebook.sh ├── pyspark-notebook ├── Dockerfile └── README.md ├── r-notebook ├── Dockerfile └── README.md ├── scipy-notebook ├── Dockerfile └── README.md ├── singleuser ├── Dockerfile ├── README.md └── singleuser.sh └── systemuser ├── Dockerfile ├── README.md └── systemuser.sh /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | 3 | *~ 4 | 5 | __pycache__/ 6 | *.py[cod] 7 | 8 | # C extensions 9 | *.so 10 | 11 | # Distribution / packaging 12 | .Python 13 | env/ 14 | build/ 15 | develop-eggs/ 16 | dist/ 17 | downloads/ 18 | eggs/ 19 | .eggs/ 20 | lib/ 21 | lib64/ 22 | parts/ 23 | sdist/ 24 | var/ 25 | *.egg-info/ 26 | .installed.cfg 27 | *.egg 28 | 29 | # PyInstaller 30 | # Usually these files are written by a python script from a template 31 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 32 | *.manifest 33 | *.spec 34 | 35 | # Installer logs 36 | pip-log.txt 37 | pip-delete-this-directory.txt 38 | 39 | # Unit test / coverage reports 40 | htmlcov/ 41 | .tox/ 42 | .coverage 43 | .coverage.* 44 | .cache 45 | nosetests.xml 46 | coverage.xml 47 | *,cover 48 | 49 | # Translations 50 | *.mo 51 | *.pot 52 | 53 | # Django stuff: 54 | *.log 55 | 56 | # Sphinx documentation 57 | docs/_build/ 58 | 59 | # PyBuilder 60 | target/ 61 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | # Licensing terms 2 | 3 | This project is licensed under the terms of the Modified BSD License 4 | (also known as New or Revised or 3-Clause BSD), as follows: 5 | 6 | - Copyright (c) 2001-2015, IPython Development Team 7 | - Copyright (c) 2015-, Jupyter Development Team 8 | 9 | All rights reserved. 10 | 11 | Redistribution and use in source and binary forms, with or without 12 | modification, are permitted provided that the following conditions are met: 13 | 14 | Redistributions of source code must retain the above copyright notice, this 15 | list of conditions and the following disclaimer. 16 | 17 | Redistributions in binary form must reproduce the above copyright notice, this 18 | list of conditions and the following disclaimer in the documentation and/or 19 | other materials provided with the distribution. 20 | 21 | Neither the name of the Jupyter Development Team nor the names of its 22 | contributors may be used to endorse or promote products derived from this 23 | software without specific prior written permission. 24 | 25 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 26 | ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 27 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 28 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE 29 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 30 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 31 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 32 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 33 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 34 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 35 | 36 | ## About the Jupyter Development Team 37 | 38 | The Jupyter Development Team is the set of all contributors to the Jupyter project. 39 | This includes all of the Jupyter subprojects. 40 | 41 | The core team that coordinates development on GitHub can be found here: 42 | https://github.com/jupyter/. 43 | 44 | ## Our Copyright Policy 45 | 46 | Jupyter uses a shared copyright model. Each contributor maintains copyright 47 | over their contributions to Jupyter. But, it is important to note that these 48 | contributions are typically only changes to the repositories. Thus, the Jupyter 49 | source code, in its entirety is not the copyright of any single person or 50 | institution. Instead, it is the collective copyright of the entire Jupyter 51 | Development Team. If individual contributors want to maintain a record of what 52 | changes/contributions they have specific copyright on, they should indicate 53 | their copyright in the commit message of the change, when they commit the 54 | change to one of the Jupyter repositories. 55 | 56 | With this in mind, the following banner should be used in any source code file 57 | to indicate the copyright and license terms: 58 | 59 | # Copyright (c) Jupyter Development Team. 60 | # Distributed under the terms of the Modified BSD License. 61 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | # Copyright (c) Jupyter Development Team. 2 | # Distributed under the terms of the Modified BSD License. 3 | 4 | .PHONY: build dev help server 5 | 6 | OWNER:=jupyter 7 | # need to list these manually because there's a dependency tree 8 | ALL_STACKS:=minimal-kernel \ 9 | minimal-notebook \ 10 | r-notebook \ 11 | scipy-notebook \ 12 | datascience-notebook \ 13 | pyspark-notebook \ 14 | all-spark-notebook 15 | GIT_MASTER_HEAD_SHA:=$(shell git rev-parse --short=12 --verify HEAD) 16 | 17 | help: 18 | @echo 19 | @echo ' build/ - builds the latest image for the stack' 20 | @echo ' dev/ - runs a foreground container for the stack' 21 | @echo ' push/ - pushes the latest and HEAD git SHA tags for the stack to Docker Hub' 22 | @echo ' refresh/ - runs a foreground container for the stack' 23 | @echo ' release-all - refresh, build, tag, and push all stacks' 24 | @echo ' tag/ - tags the latest stack image with the HEAD git SHA' 25 | 26 | build/%: DARGS?= 27 | build/%: 28 | docker build $(DARGS) --rm --force-rm -t $(OWNER)/$(notdir $@):latest ./$(notdir $@) 29 | 30 | dev/%: ARGS?= 31 | dev/%: DARGS?= 32 | dev/%: PORT?=8888 33 | dev/%: 34 | docker run -it --rm -p $(PORT):8888 $(DARGS) $(OWNER)/$(notdir $@) $(ARGS) 35 | 36 | environment-check: 37 | test -e ~/.docker-stacks-builder 38 | 39 | push/%: 40 | docker push $(OWNER)/$(notdir $@):latest 41 | docker push $(OWNER)/$(notdir $@):$(GIT_MASTER_HEAD_SHA) 42 | 43 | refresh/%: 44 | # skip if error: a stack might not be on dockerhub yet 45 | -docker pull $(OWNER)/$(notdir $@):latest 46 | 47 | release-all: environment-check \ 48 | $(patsubst %,refresh/%, $(ALL_STACKS)) \ 49 | $(patsubst %,build/%, $(ALL_STACKS)) \ 50 | $(patsubst %,tag/%, $(ALL_STACKS)) \ 51 | $(patsubst %,push/%, $(ALL_STACKS)) 52 | 53 | tag/%: 54 | # always tag the latest build with the git sha 55 | docker tag -f $(OWNER)/$(notdir $@):latest $(OWNER)/$(notdir $@):$(GIT_MASTER_HEAD_SHA) 56 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # docker-stacks 2 | 3 | [![Join the chat at https://gitter.im/jupyter/jupyter](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/jupyter/jupyter?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) 4 | 5 | Opinionated stacks of ready-to-run Jupyter applications in Docker. 6 | 7 | ## Quick Start 8 | 9 | If you're familiar with Docker, have it configured, and know exactly what you'd like to run, this one-liner should work in most cases: 10 | 11 | ``` 12 | docker run -d -P jupyter/ 13 | ``` 14 | 15 | ## Getting Started 16 | 17 | If this is your first time using Docker or any of the Jupyter projects, do the following to get started. 18 | 19 | 1. [Install Docker](https://docs.docker.com/installation/) on your host of choice. 20 | 2. Open the README in one of the folders in this git repository. 21 | 3. Follow the README for that stack. 22 | 23 | ## Tips and Gotchas 24 | 25 | * `tini -- start-notebook.sh` is the default Docker entrypoint-plus-command in every notebook stack. If you plan to modify it any way, be sure to check the *Notebook Options* section of your stack's README to understand the consequences. 26 | * Check the [Docker recipes wiki page](https://github.com/jupyter/docker-stacks/wiki/Docker-Recipes) attached to this project for information about extending and deploying the Docker images defined here. Add to the wiki if you have relevant information. 27 | 28 | ## Stacks, Tags, Versioning, and Progress 29 | 30 | Starting with [git commit SHA 9bd33dcc8688](https://github.com/jupyter/docker-stacks/tree/9bd33dcc8688): 31 | 32 | * Every folder here on GitHub has an equivalent `jupyter/` on Docker Hub. 33 | * The `latest` tag in each Docker Hub repository tracks the `master` branch `HEAD` reference on GitHub. 34 | * Any 12-character image tag on Docker Hub refers to a git commit SHA here on GitHub. 35 | * Stack contents (e.g., new library versions) will be updated upon request via PRs against this project. 36 | * Users looking to remain on older builds should refer to specific git SHA tagged images in their work, not `latest`. 37 | * Users who want to know the contents of a specific tagged image on Docker Hub can take its tag and use it as a git SHA to inspect the state of this repo at the time of that image build. 38 | * For legacy reasons, there are two additional tags named `3.2` and `4.0` on Docker Hub which point to images prior to our versioning scheme switch. 39 | 40 | ## Maintainer Workflow 41 | 42 | For PRs that impact the definition of one or more stacks: 43 | 44 | 1. Pull a PR branch locally. 45 | 2. Try building the affected stack(s). 46 | 3. If everything builds OK locally, merge it. 47 | 4. `ssh -i ~/.ssh/your-github-key build@docker-stacks.cloudet.xyz` 48 | 5. Run these commands on that VM. 49 | 50 | ``` 51 | cd docker-stacks 52 | # make sure we're always on clean master from github 53 | git fetch origin 54 | git reset --hard origin/master 55 | # if this fails, just run it again and again (idempotent) 56 | make release-all 57 | ``` 58 | 59 | When there's a security fix in the Debian base image, do the following in place of the last command: 60 | 61 | ``` 62 | docker pull debian:jessie 63 | make release-all DARGS=--no-cache 64 | ``` 65 | 66 | This will take time as the entire set of stacks will rebuild. 67 | 68 | When there's a new stack, do the following **before** trying to `make release-all`: 69 | 70 | 1. Create a new repo in the `jupyter` org on Docker Hub named after the stack folder in the git repo. 71 | 2. Grant the `stacks` team permission to write to the repo. 72 | 3. Copy/paste the short and long descriptions from one of the other docker-stacks repos on Docker Hub. Modify the appropriate values. -------------------------------------------------------------------------------- /all-spark-notebook/Dockerfile: -------------------------------------------------------------------------------- 1 | # Copyright (c) Jupyter Development Team. 2 | # Distributed under the terms of the Modified BSD License. 3 | FROM jupyter/minimal-notebook 4 | 5 | MAINTAINER Jupyter Project 6 | 7 | USER root 8 | 9 | # Spark dependencies 10 | ENV APACHE_SPARK_VERSION 1.5.1 11 | RUN apt-get -y update && \ 12 | apt-get install -y --no-install-recommends openjdk-7-jre-headless && \ 13 | apt-get clean 14 | RUN wget -qO - http://d3kbcqa49mib13.cloudfront.net/spark-${APACHE_SPARK_VERSION}-bin-hadoop2.6.tgz | tar -xz -C /usr/local/ 15 | RUN cd /usr/local && ln -s spark-${APACHE_SPARK_VERSION}-bin-hadoop2.6 spark 16 | 17 | # Mesos dependencies 18 | RUN apt-key adv --keyserver keyserver.ubuntu.com --recv E56151BF && \ 19 | DISTRO=debian && \ 20 | CODENAME=wheezy && \ 21 | echo "deb http://repos.mesosphere.io/${DISTRO} ${CODENAME} main" > /etc/apt/sources.list.d/mesosphere.list && \ 22 | apt-get -y update && \ 23 | apt-get --no-install-recommends -y --force-yes install mesos=0.22.1-1.0.debian78 && \ 24 | apt-get clean 25 | 26 | # Scala Spark kernel (build and cleanup) 27 | RUN cd /tmp && \ 28 | echo deb http://dl.bintray.com/sbt/debian / > /etc/apt/sources.list.d/sbt.list && \ 29 | apt-get update && \ 30 | git clone https://github.com/ibm-et/spark-kernel.git && \ 31 | apt-get install -yq --force-yes --no-install-recommends sbt && \ 32 | cd spark-kernel && \ 33 | git checkout 3905e47815 && \ 34 | make dist SHELL=/bin/bash && \ 35 | mv dist/spark-kernel /opt/spark-kernel && \ 36 | chmod +x /opt/spark-kernel && \ 37 | rm -rf ~/.ivy2 && \ 38 | rm -rf ~/.sbt && \ 39 | rm -rf /tmp/spark-kernel && \ 40 | apt-get remove -y sbt && \ 41 | apt-get clean 42 | 43 | # Spark and Mesos pointers 44 | ENV SPARK_HOME /usr/local/spark 45 | ENV R_LIBS_USER $SPARK_HOME/R/lib 46 | ENV PYTHONPATH $SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.8.2.1-src.zip 47 | ENV MESOS_NATIVE_LIBRARY /usr/local/lib/libmesos.so 48 | ENV SPARK_OPTS --driver-java-options=-Xms1024M --driver-java-options=-Xmx4096M --driver-java-options=-Dlog4j.logLevel=info 49 | 50 | # R pre-requisites 51 | RUN apt-get update && \ 52 | apt-get install -y --no-install-recommends \ 53 | fonts-dejavu \ 54 | gfortran \ 55 | gcc && apt-get clean 56 | 57 | USER jovyan 58 | 59 | # Install Python 3 packages 60 | RUN conda install --yes \ 61 | 'ipywidgets=4.0*' \ 62 | 'pandas=0.17*' \ 63 | 'matplotlib=1.4*' \ 64 | 'scipy=0.16*' \ 65 | 'seaborn=0.6*' \ 66 | 'scikit-learn=0.16*' \ 67 | && conda clean -yt 68 | 69 | # Install Python 2 packages 70 | RUN conda create -p $CONDA_DIR/envs/python2 python=2.7 \ 71 | 'ipython=4.0*' \ 72 | 'ipywidgets=4.0*' \ 73 | 'pandas=0.17*' \ 74 | 'matplotlib=1.4*' \ 75 | 'scipy=0.16*' \ 76 | 'seaborn=0.6*' \ 77 | 'scikit-learn=0.16*' \ 78 | pyzmq \ 79 | && conda clean -yt 80 | 81 | # R packages 82 | RUN conda config --add channels r 83 | RUN conda install --yes \ 84 | 'r-base=3.2*' \ 85 | 'r-irkernel=0.5*' \ 86 | 'r-ggplot2=1.0*' \ 87 | 'r-rcurl=1.95*' && conda clean -yt 88 | 89 | # Scala Spark kernel spec 90 | RUN mkdir -p /opt/conda/share/jupyter/kernels/scala 91 | COPY kernel.json /opt/conda/share/jupyter/kernels/scala/ 92 | 93 | USER root 94 | 95 | # Install Python 2 kernel spec globally to avoid permission problems when NB_UID 96 | # switching at runtime. 97 | RUN $CONDA_DIR/envs/python2/bin/python \ 98 | $CONDA_DIR/envs/python2/bin/ipython \ 99 | kernelspec install-self 100 | 101 | USER jovyan 102 | -------------------------------------------------------------------------------- /all-spark-notebook/README.md: -------------------------------------------------------------------------------- 1 | # Jupyter Notebook Python, Scala, R, Spark, Mesos Stack 2 | 3 | ## What it Gives You 4 | 5 | * Jupyter Notebook 4.0.x 6 | * Conda Python 3.x and Python 2.7.x environments 7 | * Conda R 3.2.x environment 8 | * Scala 2.10.x 9 | * pyspark, pandas, matplotlib, scipy, seaborn, scikit-learn pre-installed for Python 10 | * ggplot2, rcurl preinstalled for R 11 | * Spark 1.5.1 for use in local mode or to connect to a cluster of Spark workers 12 | * Mesos client 0.22 binary that can communicate with a Mesos master 13 | * Unprivileged user `jovyan` (uid=1000, configurable, see options) in group `users` (gid=100) with ownership over `/home/jovyan` and `/opt/conda` 14 | * [tini](https://github.com/krallin/tini) as the container entrypoint and [start-notebook.sh](../minimal-notebook/start-notebook.sh) as the default command 15 | * Options for HTTPS, password auth, and passwordless `sudo` 16 | 17 | ## Basic Use 18 | 19 | The following command starts a container with the Notebook server listening for HTTP connections on port 8888 without authentication configured. 20 | 21 | ``` 22 | docker run -d -p 8888:8888 jupyter/all-spark-notebook 23 | ``` 24 | 25 | ## Using Spark Local Mode 26 | 27 | This configuration is nice for using Spark on small, local data. 28 | 29 | ### In a Python Notebook 30 | 31 | 0. Run the container as shown above. 32 | 1. Open a Python 2 or 3 notebook. 33 | 2. Create a `SparkContext` configured for local mode. 34 | 35 | For example, the first few cells in a Python 3 notebook might read: 36 | 37 | ```python 38 | import pyspark 39 | sc = pyspark.SparkContext('local[*]') 40 | 41 | # do something to prove it works 42 | rdd = sc.parallelize(range(1000)) 43 | rdd.takeSample(False, 5) 44 | ``` 45 | 46 | In a Python 2 notebook, prefix the above with the following code to ensure the local workers use Python 2 as well. 47 | 48 | ```python 49 | import os 50 | os.environ['PYSPARK_PYTHON'] = 'python2' 51 | 52 | # include pyspark cells from above here ... 53 | ``` 54 | 55 | ### In a R Notebook 56 | 57 | 0. Run the container as shown above. 58 | 1. Open a R notebook. 59 | 2. Initialize `sparkR` for local mode. 60 | 3. Initialize `sparkRSQL`. 61 | 62 | For example, the first few cells in a R notebook might read: 63 | 64 | ``` 65 | library(SparkR) 66 | 67 | sc <- sparkR.init("local[*]") 68 | sqlContext <- sparkRSQL.init(sc) 69 | 70 | # do something to prove it works 71 | data(iris) 72 | df <- createDataFrame(sqlContext, iris) 73 | head(filter(df, df$Petal_Width > 0.2)) 74 | ``` 75 | 76 | ### In a Scala Notebook 77 | 78 | 0. Run the container as shown above. 79 | 1. Open a Scala notebook. 80 | 2. Use the pre-configured `SparkContext` in variable `sc`. 81 | 82 | For example: 83 | 84 | ``` 85 | val rdd = sc.parallelize(0 to 999) 86 | rdd.takeSample(false, 5) 87 | ``` 88 | 89 | ## Connecting to a Spark Cluster on Mesos 90 | 91 | This configuration allows your compute cluster to scale with your data. 92 | 93 | 0. [Deploy Spark on Mesos](http://spark.apache.org/docs/latest/running-on-mesos.html). 94 | 1. Configure each slave with [the `--no-switch_user` flag](https://open.mesosphere.com/reference/mesos-slave/) or create the `jovyan` user on every slave node. 95 | 2. Run the Docker container with `--net=host` in a location that is network addressable by all of your Spark workers. (This is a [Spark networking requirement](http://spark.apache.org/docs/latest/cluster-overview.html#components).) 96 | * NOTE: When using `--net=host`, you must also use the flags `--pid=host -e TINI_SUBREAPER=true`. See https://github.com/jupyter/docker-stacks/issues/64 for details. 97 | 3. Follow the language specific instructions below. 98 | 99 | ### In a Python Notebook 100 | 101 | 0. Open a Python 2 or 3 notebook. 102 | 1. Create a `SparkConf` instance in a new notebook pointing to your Mesos master node (or Zookeeper instance) and Spark binary package location. 103 | 2. Create a `SparkContext` using this configuration. 104 | 105 | For example, the first few cells in a Python 3 notebook might read: 106 | 107 | ```python 108 | import os 109 | # make sure pyspark tells workers to use python3 not 2 if both are installed 110 | os.environ['PYSPARK_PYTHON'] = '/usr/bin/python3' 111 | 112 | import pyspark 113 | conf = pyspark.SparkConf() 114 | 115 | # point to mesos master or zookeeper entry (e.g., zk://10.10.10.10:2181/mesos) 116 | conf.setMaster("mesos://10.10.10.10:5050") 117 | # point to spark binary package in HDFS or on local filesystem on all slave 118 | # nodes (e.g., file:///opt/spark/spark-1.5.1-bin-hadoop2.6.tgz) 119 | conf.set("spark.executor.uri", "hdfs://10.10.10.10/spark/spark-1.5.1-bin-hadoop2.6.tgz") 120 | # set other options as desired 121 | conf.set("spark.executor.memory", "8g") 122 | conf.set("spark.core.connection.ack.wait.timeout", "1200") 123 | 124 | # create the context 125 | sc = pyspark.SparkContext(conf=conf) 126 | 127 | # do something to prove it works 128 | rdd = sc.parallelize(range(100000000)) 129 | rdd.sumApprox(3) 130 | ``` 131 | 132 | To use Python 2 in the notebook and on the workers, change the `PYSPARK_PYTHON` environment variable to point to the location of the Python 2.x interpreter binary. If you leave this environment variable unset, it defaults to `python`. 133 | 134 | Of course, all of this can be hidden in an [IPython kernel startup script](http://ipython.org/ipython-doc/stable/development/config.html?highlight=startup#startup-files), but "explicit is better than implicit." :) 135 | 136 | ### In a R Notebook 137 | 138 | 0. Run the container as shown above. 139 | 1. Open a R notebook. 140 | 2. Initialize `sparkR` Mesos master node (or Zookeeper instance) and Spark binary package location. 141 | 3. Initialize `sparkRSQL`. 142 | 143 | For example, the first few cells in a R notebook might read: 144 | 145 | ``` 146 | library(SparkR) 147 | 148 | # point to mesos master or zookeeper entry (e.g., zk://10.10.10.10:2181/mesos)\ 149 | # as the first argument 150 | # point to spark binary package in HDFS or on local filesystem on all slave 151 | # nodes (e.g., file:///opt/spark/spark-1.5.1-bin-hadoop2.6.tgz) in sparkEnvir 152 | # set other options in sparkEnvir 153 | sc <- sparkR.init("mesos://10.10.10.10:5050", sparkEnvir=list( 154 | spark.executor.uri="hdfs://10.10.10.10/spark/spark-1.5.1-bin-hadoop2.6.tgz", 155 | spark.executor.memory="8g" 156 | ) 157 | ) 158 | sqlContext <- sparkRSQL.init(sc) 159 | 160 | # do something to prove it works 161 | data(iris) 162 | df <- createDataFrame(sqlContext, iris) 163 | head(filter(df, df$Petal_Width > 0.2)) 164 | ``` 165 | 166 | ### In a Scala Notebook 167 | 168 | 0. Open a terminal via *New -> Terminal* in the notebook interface. 169 | 1. Add information about your cluster to the Scala kernel spec file in `~/.ipython/kernels/scala/kernel.json`. (See below.) 170 | 2. Open a Scala notebook. 171 | 3. Use the pre-configured `SparkContext` in variable `sc`. 172 | 173 | The Scala kernel automatically creates a `SparkContext` when it starts based on configuration information from its command line arguments and environments. Therefore, you must add it to the Scala kernel spec file. You cannot, at present, configure it yourself within a notebook. 174 | 175 | For instance, a kernel spec file with information about a Mesos master, Spark binary location in HDFS, and an executor option appears here: 176 | 177 | ``` 178 | { 179 | "display_name": "Scala 2.10.4", 180 | "language": "scala", 181 | "argv": [ 182 | "/opt/sparkkernel/bin/sparkkernel", 183 | "--profile", 184 | "{connection_file}", 185 | "--master=mesos://10.10.10.10:5050" 186 | ], 187 | "env": { 188 | "SPARK_CONFIGURATION": "spark.executor.memory=8g,spark.executor.uri=hdfs://10.10.10.10/spark/spark-1.5.1-bin-hadoop2.6.tgz" 189 | } 190 | } 191 | ``` 192 | 193 | Note that this is the same information expressed in a notebook in the Python case above. Once the kernel spec has your cluster information, you can test your cluster in a Scala notebook like so: 194 | 195 | ``` 196 | // should print the value of --master in the kernel spec 197 | println(sc.master) 198 | 199 | // do something to prove it works 200 | val rdd = sc.parallelize(0 to 99999999) 201 | rdd.sum() 202 | ``` 203 | 204 | ## Notebook Options 205 | 206 | You can pass [Jupyter command line options](http://jupyter.readthedocs.org/en/latest/config.html#command-line-arguments) through the [`start-notebook.sh` command](https://github.com/jupyter/docker-stacks/blob/master/minimal-notebook/start-notebook.sh#L15) when launching the container. For example, to set the base URL of the notebook server you might do the following: 207 | 208 | ``` 209 | docker run -d -p 8888:8888 jupyter/all-spark-notebook start-notebook.sh --NotebookApp.base_url=/some/path 210 | ``` 211 | 212 | You can sidestep the `start-notebook.sh` script entirely by specifying a command other than `start-notebook.sh`. If you do, the `NB_USER` and `GRANT_SUDO` features documented below will not work. See the Docker Options section for details. 213 | 214 | ## Docker Options 215 | 216 | You may customize the execution of the Docker container and the Notebook server it contains with the following optional arguments. 217 | 218 | * `-e PASSWORD="YOURPASS"` - Configures Jupyter Notebook to require the given password. Should be conbined with `USE_HTTPS` on untrusted networks. 219 | * `-e USE_HTTPS=yes` - Configures Jupyter Notebook to accept encrypted HTTPS connections. If a `pem` file containing a SSL certificate and key is not found in `/home/jovyan/.ipython/profile_default/security/notebook.pem`, the container will generate a self-signed certificate for you. 220 | * `-e NB_UID=1000` - Specify the uid of the `jovyan` user. Useful to mount host volumes with specific file ownership. For this option to take effect, you must run the container with `--user root`. (The `start-notebook.sh` script will `su jovyan` after adjusting the user id.) 221 | * `-e GRANT_SUDO=yes` - Gives the `jovyan` user passwordless `sudo` capability. Useful for installing OS packages. For this option to take effect, you must run the container with `--user root`. (The `start-notebook.sh` script will `su jovyan` after adding `jovyan` to sudoers.) **You should only enable `sudo` if you trust the user or if the container is running on an isolated host.** 222 | * `-v /some/host/folder/for/work:/home/jovyan/work` - Host mounts the default working directory on the host to preserve work even when the container is destroyed and recreated (e.g., during an upgrade). 223 | * `-v /some/host/folder/for/server.pem:/home/jovyan/.local/share/jupyter/notebook.pem` - Mounts a SSL certificate plus key for `USE_HTTPS`. Useful if you have a real certificate for the domain under which you are running the Notebook server. 224 | * `-p 4040:4040` - Opens the port for the [Spark Monitoring and Instrumentation UI](http://spark.apache.org/docs/latest/monitoring.html). Note every new spark context that is created is put onto an incrementing port (ie. 4040, 4041, 4042, etc.), and it might be necessary to open multiple ports. `docker run -d -p 8888:8888 -p 4040:4040 -p 4041:4041 jupyter/all-spark-notebook` 225 | 226 | ## Conda Environments 227 | 228 | The default Python 3.x [Conda environment](http://conda.pydata.org/docs/using/envs.html) resides in `/opt/conda`. A second Python 2.x Conda environment exists in `/opt/conda/envs/python2`. You can [switch to the python2 environment](http://conda.pydata.org/docs/using/envs.html#change-environments-activate-deactivate) in a shell by entering the following: 229 | 230 | ``` 231 | source activate python2 232 | ``` 233 | 234 | You can return to the default environment with this command: 235 | 236 | ``` 237 | source deactivate 238 | ``` 239 | 240 | The commands `ipython`, `python`, `pip`, `easy_install`, and `conda` (among others) are available in both environments. 241 | -------------------------------------------------------------------------------- /all-spark-notebook/kernel.json: -------------------------------------------------------------------------------- 1 | { 2 | "display_name": "Scala 2.10.4", 3 | "language": "scala", 4 | "argv": [ 5 | "/opt/spark-kernel/bin/spark-kernel", 6 | "--profile", 7 | "{connection_file}" 8 | ] 9 | } -------------------------------------------------------------------------------- /cuda/Dockerfile: -------------------------------------------------------------------------------- 1 | # From https://github.com/Kaixhin/dockerfiles/tree/master/cuda/cuda_v7.5/Dockerfile 2 | # And https://github.com/Kaixhin/dockerfiles/blob/master/cudnn/cudnn_v7.5/Dockerfile 3 | # Adapted for CUDA 8.0 4 | FROM ubuntu:16.04 5 | MAINTAINER Haoyang Zeng 6 | 7 | # Install wget and build-essential 8 | RUN apt-get update && apt-get install -y \ 9 | build-essential \ 10 | wget \ 11 | module-init-tools \ 12 | curl && \ 13 | apt-get clean && \ 14 | rm -rf /var/lib/apt/lists/* 15 | 16 | # Change to the /tmp directory 17 | RUN cd /tmp && \ 18 | #wget http://us.download.nvidia.com/tesla/384.111/NVIDIA-Linux-x86_64-384.111.run -o /dev/null && \ 19 | #bash NVIDIA-Linux-x86_64-*.run -s --no-kernel-module && \ 20 | # Download run file 21 | wget https://developer.nvidia.com/compute/cuda/9.0/Prod/local_installers/cuda_9.0.176_384.81_linux-run -o /dev/null && \ 22 | # Make the run file executable and extract 23 | chmod +x cuda_*_linux-run && ./cuda_*_linux-run -extract=`pwd` && \ 24 | ls && \ 25 | # Install CUDA drivers (silent, no kernel) 26 | ./NVIDIA-Linux-x86_64-*.run -s --no-kernel-module && \ 27 | # Install toolkit (silent) 28 | ./cuda-linux*.run -noprompt | cat > /dev/null && \ 29 | # Clean up 30 | rm -rf * 31 | 32 | # Add to path 33 | ENV PATH=/usr/local/cuda/bin:$PATH \ 34 | LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH 35 | RUN ldconfig /usr/local/cuda/lib64 36 | 37 | # Install cuDNN v7 38 | RUN CUDNN_DOWNLOAD_SUM=1a3e076447d5b9860c73d9bebe7087ffcb7b0c8814fd1e506096435a2ad9ab0e && \ 39 | curl -fsSL http://developer.download.nvidia.com/compute/redist/cudnn/v7.0.5/cudnn-9.0-linux-x64-v7.tgz -O && \ 40 | echo "$CUDNN_DOWNLOAD_SUM cudnn-9.0-linux-x64-v7.tgz" | sha256sum -c - && \ 41 | tar --no-same-owner -xzf cudnn-9.0-linux-x64-v7.tgz -C /usr/local --wildcards 'cuda/lib64/libcudnn.so.*' && \ 42 | rm cudnn-9.0-linux-x64-v7.tgz && \ 43 | ldconfig 44 | 45 | ## Install cuDNN v5 46 | #RUN CUDNN_DOWNLOAD_SUM=c10719b36f2dd6e9ddc63e3189affaa1a94d7d027e63b71c3f64d449ab0645ce && \ 47 | # curl -fsSL http://developer.download.nvidia.com/compute/redist/cudnn/v5.1/cudnn-8.0-linux-x64-v5.1.tgz -O && \ 48 | # echo "$CUDNN_DOWNLOAD_SUM cudnn-8.0-linux-x64-v5.1.tgz" | sha256sum -c --strict - && \ 49 | # tar -xzf cudnn-8.0-linux-x64-v5.1.tgz -C /usr/local && \ 50 | # rm cudnn-8.0-linux-x64-v5.1.tgz && \ 51 | # ldconfig 52 | 53 | # Install cuDNN v4 54 | #ENV ML_REPO_PKG=nvidia-machine-learning-repo-ubuntu1404_4.0-2_amd64.deb 55 | #RUN wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1404/x86_64/$ML_REPO_PKG && \ 56 | # dpkg -i $ML_REPO_PKG && \ 57 | # rm $ML_REPO_PKG && \ 58 | # apt-get update && apt-get install -y libcudnn4 libcudnn4-dev 59 | -------------------------------------------------------------------------------- /cuda/README.md: -------------------------------------------------------------------------------- 1 | [![Docker Pulls](https://img.shields.io/docker/pulls/kaixhin/cuda.svg)](https://hub.docker.com/r/kaixhin/cuda/) 2 | [![Docker Stars](https://img.shields.io/docker/stars/kaixhin/cuda.svg)](https://hub.docker.com/r/kaixhin/cuda/) 3 | 4 | cuda 5 | ==== 6 | **DEPRECATED:** Please use [NVIDIA Docker](https://github.com/NVIDIA/nvidia-docker). 7 | 8 | Ubuntu Core 14.04 + [CUDA 7.5.18](http://www.nvidia.com/object/cuda_home_new.html). 9 | 10 | Requirements 11 | ------------ 12 | 13 | - Host with corresponding CUDA drivers (v. 352.39) installed for the kernel module. 14 | 15 | Usage 16 | ----- 17 | The container must have all NVIDIA devices attached to it for CUDA to work properly. 18 | Therefore the command will be as such: `docker run -it --device /dev/nvidiactl --device /dev/nvidia-uvm --device /dev/nvidia0 kaixhin/cuda`. 19 | With 4 GPUs this would also have to include `--device /dev/nvidia1 --device /dev/nvidia2 --device /dev/nvidia3`. 20 | 21 | For more information on CUDA on Docker, see the [repo readme](https://github.com/Kaixhin/dockerfiles#cuda). 22 | 23 | Samples 24 | ------- 25 | 26 | As the image is intended to be lightweight, the CUDA samples were not installed. If you wish to experiment with the samples you will need to install them yourself. The steps are as below: 27 | 28 | ``` 29 | wget http://developer.download.nvidia.com/compute/cuda/7.5/Prod/local_installers/cuda_7.5.18_linux.run 30 | chmod +x cuda_*_linux.run 31 | ./cuda_*_linux.run -extract=`pwd` 32 | ./cuda-samples-linux-*.run -noprompt 33 | ``` 34 | 35 | Please note that you may need to install other packages in order to compile some of the CUDA samples. 36 | 37 | Citation 38 | -------- 39 | If you find this useful in research please consider [citing this work](https://github.com/Kaixhin/dockerfiles/blob/master/CITATION.md). 40 | -------------------------------------------------------------------------------- /datascience-notebook/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM giffordlab/minimal-notebook 2 | 3 | MAINTAINER Matt Edwards 4 | 5 | USER root 6 | 7 | # Hacky access control for the image 8 | RUN echo "matted ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers 9 | RUN echo "thashim ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers 10 | RUN echo "zeng ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers 11 | 12 | RUN apt-get update -qq && apt-get install -qqy software-properties-common && \ 13 | add-apt-repository ppa:marutter/rrutter && \ 14 | add-apt-repository ppa:staticfloat/juliareleases && \ 15 | add-apt-repository ppa:staticfloat/julia-deps && \ 16 | apt-get update && \ 17 | apt-get install -y --no-install-recommends \ 18 | emacs24-nox \ 19 | fonts-dejavu \ 20 | gcc \ 21 | gfortran \ 22 | graphviz \ 23 | julia \ 24 | nano \ 25 | wget \ 26 | zsh \ 27 | && apt-get clean \ 28 | && rm -rf /var/lib/apt/lists/* 29 | 30 | # Install Python 3 packages 31 | RUN conda install --yes \ 32 | 'ipywidgets=5.2*' \ 33 | 'pandas=0.22*' \ 34 | 'numexpr=2.6*' \ 35 | 'matplotlib=2.1*' \ 36 | 'scipy=1.0*' \ 37 | 'seaborn=0.7*' \ 38 | 'scikit-learn=0.19*' \ 39 | 'pytables=3.4.2*' \ 40 | 'sympy=1.0*' \ 41 | 'cython=0.27*' \ 42 | 'patsy=0.4*' \ 43 | 'statsmodels=0.8*' \ 44 | 'cloudpickle=0.5*' \ 45 | 'dill=0.2*' \ 46 | 'numba=0.36*' \ 47 | 'bokeh=0.12*' \ 48 | 'h5py=2.7*' \ 49 | 'theano=1.0.1' \ 50 | biopython \ 51 | && conda clean -yat 52 | 53 | 54 | RUN pip --no-cache-dir install tensorflow-gpu 55 | 56 | 57 | # Install Python 2 packages 58 | RUN conda create -p $CONDA_DIR/envs/python2 python=2.7 \ 59 | 'ipython=5.4*' \ 60 | 'ipywidgets=5.2*' \ 61 | 'pandas=0.22*' \ 62 | 'numexpr=2.6*' \ 63 | 'matplotlib=2.1*' \ 64 | 'scipy=1.0*' \ 65 | 'seaborn=0.7*' \ 66 | 'scikit-learn=0.19*' \ 67 | 'pytables=3.4.2*' \ 68 | 'sympy=1.0*' \ 69 | 'cython=0.27*' \ 70 | 'patsy=0.4*' \ 71 | 'statsmodels=0.8*' \ 72 | 'cloudpickle=0.5*' \ 73 | 'dill=0.2*' \ 74 | 'numba=0.36*' \ 75 | 'bokeh=0.12*' \ 76 | 'h5py=2.7*' \ 77 | 'theano=0.8.2' \ 78 | pyzmq \ 79 | biopython \ 80 | pydot \ 81 | pillow \ 82 | && conda clean -yat 83 | 84 | RUN $CONDA_DIR/envs/python2/bin/pip --no-cache-dir install tensorflow-gpu 85 | 86 | # R packages including IRKernel which gets installed globally. 87 | RUN conda config --add channels r 88 | RUN conda install --yes \ 89 | 'rpy2=2.8*' \ 90 | 'r-base=3.3.2' \ 91 | 'r-irkernel=0.7*' \ 92 | 'r-plyr=1.8*' \ 93 | 'r-devtools=1.12*' \ 94 | 'r-dplyr=0.5*' \ 95 | 'r-ggplot2=2.2*' \ 96 | 'r-tidyr=0.6*' \ 97 | 'r-shiny=0.14*' \ 98 | 'r-rmarkdown=1.2*' \ 99 | 'r-forecast=7.3*' \ 100 | 'r-stringr=1.1*' \ 101 | 'r-rsqlite=1.1*' \ 102 | 'r-reshape2=1.4*' \ 103 | 'r-caret=6.0*' \ 104 | 'r-rcurl=1.95*' \ 105 | 'r-randomforest=4.6*' && conda clean -yat 106 | 107 | RUN julia -e 'Pkg.add("IJulia")' && \ 108 | julia -e 'Pkg.add("Gadfly")' && \ 109 | julia -F -e 'Pkg.add("HDF5")' && \ 110 | chmod -R go+rx $CONDA_DIR/share/jupyter 111 | 112 | # Install deep learning packages with pip, for both Python versions. 113 | RUN $CONDA_DIR/envs/python2/bin/pip install --upgrade --no-deps git+https://github.com/Lasagne/lasagne && \ 114 | $CONDA_DIR/envs/python2/bin/pip install --upgrade --no-deps git+https://github.com/dnouri/nolearn@0.5 && \ 115 | $CONDA_DIR/envs/python2/bin/pip install --upgrade --no-deps git+https://github.com/fchollet/keras@1.2.1 && \ 116 | $CONDA_DIR/envs/python2/bin/pip install --upgrade --no-deps git+https://github.com/maxpumperla/hyperas@0.3 117 | 118 | RUN conda install -p $CONDA_DIR/envs/python2 pytorch torchvision cuda90 -c pytorch 119 | 120 | RUN pip install --upgrade --no-deps git+https://github.com/Lasagne/lasagne && \ 121 | pip install --upgrade --no-deps git+https://github.com/dnouri/nolearn@0.5 && \ 122 | pip install --upgrade --no-deps git+https://github.com/fchollet/keras@1.2.1 && \ 123 | pip install --upgrade --no-deps git+https://github.com/maxpumperla/hyperas@0.3 124 | 125 | RUN conda install -p $CONDA_DIR pytorch torchvision cuda90 -c pytorch 126 | 127 | ENV THEANO_FLAGS='cuda.root=/usr/local/cuda,device=gpu,floatX=float32,lib.cnmem=0' 128 | ENV CUDA_HOME=/usr/local/cuda 129 | ENV LD_LIBRARY_PATH=/usr/local/cuda/lib64 130 | 131 | RUN pip install bash_kernel && \ 132 | python -m bash_kernel.install && \ 133 | $CONDA_DIR/envs/python2/bin/python \ 134 | $CONDA_DIR/envs/python2/bin/ipython \ 135 | kernelspec install-self 136 | 137 | USER jovyan 138 | -------------------------------------------------------------------------------- /datascience-notebook/README.md: -------------------------------------------------------------------------------- 1 | # Jupyter Notebook Data Science Stack 2 | 3 | ## What it Gives You 4 | 5 | * Jupyter Notebook 4.0.x 6 | * Conda Python 3.x and Python 2.7.x environments 7 | * pandas, matplotlib, scipy, seaborn, scikit-learn, scikit-image, sympy, cython, patsy, statsmodel, cloudpickle, dill, numba, bokeh pre-installed 8 | * Conda R v3.2.x and channel 9 | * plyr, devtools, dplyr, ggplot2, tidyr, shiny, rmarkdown, forecast, stringr, rsqlite, reshape2, nycflights13, caret, rcurl, and randomforest pre-installed 10 | * Julia v0.3.x with Gadfly and RDatasets pre-installed 11 | * Unprivileged user `jovyan` (uid=1000, configurable, see options) in group `users` (gid=100) with ownership over `/home/jovyan` and `/opt/conda` 12 | * [tini](https://github.com/krallin/tini) as the container entrypoint and [start-notebook.sh](../minimal-notebook/start-notebook.sh) as the default command 13 | * Options for HTTPS, password auth, and passwordless `sudo` 14 | 15 | ## Basic Use 16 | 17 | The following command starts a container with the Notebook server listening for HTTP connections on port 8888 without authentication configured. 18 | 19 | ``` 20 | docker run -d -p 8888:8888 jupyter/datascience-notebook 21 | ``` 22 | 23 | ## Notebook Options 24 | 25 | You can pass [Jupyter command line options](http://jupyter.readthedocs.org/en/latest/config.html#command-line-arguments) through the [`start-notebook.sh` command](https://github.com/jupyter/docker-stacks/blob/master/minimal-notebook/start-notebook.sh#L15) when launching the container. For example, to set the base URL of the notebook server you might do the following: 26 | 27 | ``` 28 | docker run -d -p 8888:8888 jupyter/datascience-notebook start-notebook.sh --NotebookApp.base_url=/some/path 29 | ``` 30 | 31 | You can sidestep the `start-notebook.sh` script entirely by specifying a command other than `start-notebook.sh`. If you do, the `NB_USER` and `GRANT_SUDO` features documented below will not work. See the Docker Options section for details. 32 | 33 | ## Docker Options 34 | 35 | You may customize the execution of the Docker container and the Notebook server it contains with the following optional arguments. 36 | 37 | * `-e PASSWORD="YOURPASS"` - Configures Jupyter Notebook to require the given password. Should be conbined with `USE_HTTPS` on untrusted networks. 38 | * `-e USE_HTTPS=yes` - Configures Jupyter Notebook to accept encrypted HTTPS connections. If a `pem` file containing a SSL certificate and key is not found in `/home/jovyan/.ipython/profile_default/security/notebook.pem`, the container will generate a self-signed certificate for you. 39 | * `-e NB_UID=1000` - Specify the uid of the `jovyan` user. Useful to mount host volumes with specific file ownership. For this option to take effect, you must run the container with `--user root`. (The `start-notebook.sh` script will `su jovyan` after adjusting the user id.) 40 | * `-e GRANT_SUDO=yes` - Gives the `jovyan` user passwordless `sudo` capability. Useful for installing OS packages. For this option to take effect, you must run the container with `--user root`. (The `start-notebook.sh` script will `su jovyan` after adding `jovyan` to sudoers.) **You should only enable `sudo` if you trust the user or if the container is running on an isolated host.** 41 | * `-v /some/host/folder/for/work:/home/jovyan/work` - Host mounts the default working directory on the host to preserve work even when the container is destroyed and recreated (e.g., during an upgrade). 42 | * `-v /some/host/folder/for/server.pem:/home/jovyan/.local/share/jupyter/notebook.pem` - Mounts a SSL certificate plus key for `USE_HTTPS`. Useful if you have a real certificate for the domain under which you are running the Notebook server. 43 | 44 | ## Conda Environments 45 | 46 | The default Python 3.x [Conda environment](http://conda.pydata.org/docs/using/envs.html) resides in `/opt/conda`. A second Python 2.x Conda environment exists in `/opt/conda/envs/python2`. You can [switch to the python2 environment](http://conda.pydata.org/docs/using/envs.html#change-environments-activate-deactivate) in a shell by entering the following: 47 | 48 | ``` 49 | source activate python2 50 | ``` 51 | 52 | You can return to the default environment with this command: 53 | 54 | ``` 55 | source deactivate 56 | ``` 57 | 58 | The commands `ipython`, `python`, `pip`, `easy_install`, and `conda` (among others) are available in both environments. 59 | -------------------------------------------------------------------------------- /examples/make-deploy/Dockerfile: -------------------------------------------------------------------------------- 1 | # Copyright (c) Jupyter Development Team. 2 | # Distributed under the terms of the Modified BSD License. 3 | 4 | # Pick your favorite docker-stacks image 5 | FROM jupyter/minimal-notebook:2d125a7161b5 6 | 7 | USER jovyan 8 | 9 | # Add permanent pip/conda installs, data files, other user libs here 10 | # e.g., RUN pip install jupyter_dashboards 11 | 12 | USER root 13 | 14 | # Add permanent apt-get installs and other root commands here 15 | # e.g., RUN apt-get install npm nodejs 16 | -------------------------------------------------------------------------------- /examples/make-deploy/Makefile: -------------------------------------------------------------------------------- 1 | # Copyright (c) Jupyter Development Team. 2 | # Distributed under the terms of the Modified BSD License. 3 | 4 | .PHONY: help check image notebook 5 | 6 | IMAGE:=my-notebook 7 | 8 | # Common, extensible docker run command 9 | define RUN_NOTEBOOK 10 | @docker volume create --name $(WORK_VOLUME) > /dev/null 11 | -@docker rm -f $(NAME) 2> /dev/null 12 | @docker run -d -p $(PORT):8888 \ 13 | --name $(NAME) \ 14 | -v $(WORK_VOLUME):/home/jovyan/work \ 15 | $(DOCKER_ARGS) \ 16 | $(IMAGE) bash -c "$(PRE_CMD) chown jovyan /home/jovyan/work && start-notebook.sh $(ARGS)" > /dev/null 17 | @echo "DONE: Notebook '$(NAME)' listening on $$(docker-machine ip $$(docker-machine active)):$(PORT)" 18 | endef 19 | 20 | help: 21 | @cat README.md 22 | 23 | check: 24 | @which docker-machine > /dev/null || (echo "ERROR: docker-machine not found (brew install docker-machine)"; exit 1) 25 | @which docker > /dev/null || (echo "ERROR: docker not found (brew install docker)"; exit 1) 26 | @docker | grep volume > /dev/null || (echo "ERROR: docker 1.9.0+ required"; exit 1) 27 | 28 | image: DOCKER_ARGS?= 29 | image: 30 | @docker build --rm $(DOCKER_ARGS) -t $(IMAGE) . 31 | 32 | notebook: PORT?=80 33 | notebook: NAME?=notebook 34 | notebook: WORK_VOLUME?=$(NAME)-data 35 | notebook: check 36 | $(RUN_NOTEBOOK) 37 | 38 | # docker-machine drivers 39 | include virtualbox.makefile 40 | include softlayer.makefile 41 | 42 | # Preset notebook configurations 43 | include self-signed.makefile 44 | include letsencrypt.makefile -------------------------------------------------------------------------------- /examples/make-deploy/README.md: -------------------------------------------------------------------------------- 1 | This folder contains a Makefile and a set of supporting files demonstrating how to run a docker-stack notebook container on a docker-machine controlled host. 2 | 3 | ## Prerequisites 4 | 5 | * make 3.81+ 6 | * Ubuntu users: Be aware of [make 3.81 defect 483086](https://bugs.launchpad.net/ubuntu/+source/make-dfsg/+bug/483086) which exists in 14.04 LTS but is fixed in 15.04+ 7 | * docker-machine 0.5.0+ 8 | * docker 1.9.0+ 9 | 10 | ## Quickstart 11 | 12 | To show what's possible, here's how to run the `jupyter/minimal-notebook` on a brand new local virtualbox. 13 | 14 | ``` 15 | # create a new VM 16 | make virtualbox-vm NAME=dev 17 | # make the new VM the active docker machine 18 | eval $(docker-machine env dev) 19 | # pull a docker stack and build a local image from it 20 | make image 21 | # start a notebook server in a container 22 | make notebook 23 | ``` 24 | 25 | The last command will log the IP address and port to visit in your browser. 26 | 27 | ## FAQ 28 | 29 | ### Can I run multiple notebook containers on the same VM? 30 | 31 | Yes. Specify a unique name and port on the `make notebook` command. 32 | 33 | ``` 34 | make notebook NAME=my-notebook PORT=9000 35 | make notebook NAME=your-notebook PORT=9001 36 | ``` 37 | 38 | ### Can multiple notebook containers share their notebook directory? 39 | 40 | Yes. 41 | 42 | ``` 43 | make notebook NAME=my-notebook PORT=9000 WORK_VOLUME=our-work 44 | make notebook NAME=your-notebook PORT=9001 WORK_VOLUME=our-work 45 | ``` 46 | 47 | ### How do I run over HTTPS? 48 | 49 | Instead of `make notebook`, run `make self-signed-notebook PASSWORD=your_desired_password`. This target gives you a notebook wtih a self-signed certificate. 50 | 51 | ### That self-signed certificate is a pain. Let's Encrypt? 52 | 53 | Yes. Please. 54 | 55 | ``` 56 | make letsencrypt FQDN=host.mydomain.com EMAIL=myemail@somewhere.com 57 | make letsencrypt-notebook 58 | ``` 59 | 60 | The first command creates a Docker volume named after the notebook container with a `-secrets` suffix. It then runs the `letsencrypt` client with a slew of options (one of which has you automatically agreeing to the Let's Encrypt Terms of Service, see the Makefile). The second command mounts the secrets volume and configures Jupyter to use the full-chain certificate and private key. 61 | 62 | Be aware: Let's Encrypt has a pretty [low rate limit per domain](https://community.letsencrypt.org/t/public-beta-rate-limits/4772/3) at the moment. Don't exhaust your requests playing around! 63 | 64 | Also, keep in mind Let's Encrypt certificates are short lived: 90 days at the moment. You'll need to manually setup a cron job to run the renewal steps at the moment. (You can reuse the first command above.) 65 | 66 | ### My pip/conda/apt-get installs disappear every time I restart the container. Can I make them permanent? 67 | 68 | ``` 69 | # add your pip, conda, apt-get, etc. permanent features to the Dockerfile where 70 | # indicated by the comments in the Dockerfile 71 | vi Dockerfile 72 | make image 73 | make notebook 74 | ``` 75 | 76 | ### How do I upgrade my Docker container? 77 | 78 | ``` 79 | make image DOCKER_ARGS=--pull 80 | make notebook 81 | ``` 82 | 83 | The first line pulls the latest version of the Docker image used in the local Dockerfile. Then it rebuilds the local Docker image containing any customizations you may have added to it. The second line kills your currently running notebook container, and starts a fresh one using the new image. 84 | 85 | ### Can I run on another VM provider other than VirtualBox? 86 | 87 | Yes. As an example, there's a `softlayer.makefile` included in this repo as an example. You would use it like so: 88 | 89 | ``` 90 | make softlayer-vm NAME=myhost \ 91 | SOFTLAYER_DOMAIN=your_desired_domain \ 92 | SOFTLAYER_USER=your_user_id \ 93 | SOFTLAYER_API_KEY=your_api_key 94 | eval $(docker-machine env myhost) 95 | # optional, creates a real DNS entry for the VM using the machine name as the hostname 96 | make softlayer-dns SOFTLAYER_DOMAIN=your_desired_domain 97 | make image 98 | make notebook 99 | ``` 100 | 101 | If you'd like to add support for another docker-machine driver, use the `softlayer.makefile` as a template. 102 | 103 | ### Where are my notebooks stored? 104 | 105 | `make notebook` creates a Docker volume named after the notebook container with a `-data` suffix. 106 | 107 | ### Uh ... make? 108 | 109 | Yes, sorry Windows users. It got the job done for a simple example. We can certainly accept other deployment mechanism examples in the parent folder or in other repos. 110 | 111 | ### Are there any other options? 112 | 113 | Yes indeed. `cat` the Makefiles and look at the target parameters. 114 | -------------------------------------------------------------------------------- /examples/make-deploy/letsencrypt.makefile: -------------------------------------------------------------------------------- 1 | # Copyright (c) Jupyter Development Team. 2 | # Distributed under the terms of the Modified BSD License. 3 | 4 | letsencrypt: NAME?=notebook 5 | letsencrypt: SECRETS_VOLUME?=$(NAME)-secrets 6 | letsencrypt: 7 | @test -n "$(FQDN)" || \ 8 | (echo "ERROR: FQDN not defined or blank"; exit 1) 9 | @test -n "$(EMAIL)" || \ 10 | (echo "ERROR: EMAIL not defined or blank"; exit 1) 11 | @docker volume create --name $(SECRETS_VOLUME) > /dev/null 12 | # Specifying an alternative cert path doesn't work with the --duplicate 13 | # setting which we want to use for renewal. 14 | @docker run -it --rm -p 80:80 \ 15 | -v $(SECRETS_VOLUME):/etc/letsencrypt \ 16 | quay.io/letsencrypt/letsencrypt:latest \ 17 | certonly \ 18 | --standalone \ 19 | --standalone-supported-challenges http-01 \ 20 | --agree-tos \ 21 | --duplicate \ 22 | --domain '$(FQDN)' \ 23 | --email '$(EMAIL)' 24 | # The lets encrypt image has an entrypoint so we use the notebook image 25 | # instead which we know uses tini as the entry and can run arbitrary commands. 26 | # Here we need to set the permissions so nobody in the proxy container can read 27 | # the cert and key. Plus we want to symlink the certs into the root of the 28 | # /etc/letsencrypt directory so that the FQDN doesn't have to be known later. 29 | @docker run -it --rm \ 30 | -v $(SECRETS_VOLUME):/etc/letsencrypt \ 31 | $(NOTEBOOK_IMAGE) \ 32 | bash -c "ln -s /etc/letsencrypt/live/$(FQDN)/* /etc/letsencrypt/ && \ 33 | find /etc/letsencrypt -type d -exec chmod 755 {} +" 34 | 35 | letsencrypt-notebook: PORT?=443 36 | letsencrypt-notebook: NAME?=notebook 37 | letsencrypt-notebook: WORK_VOLUME?=$(NAME)-data 38 | letsencrypt-notebook: SECRETS_VOLUME?=$(NAME)-secrets 39 | letsencrypt-notebook: DOCKER_ARGS:=-e USE_HTTPS=yes \ 40 | -e PASSWORD=$(PASSWORD) \ 41 | -v $(SECRETS_VOLUME):/etc/letsencrypt 42 | letsencrypt-notebook: ARGS:=\ 43 | --NotebookApp.certfile=/etc/letsencrypt/fullchain.pem \ 44 | --NotebookApp.keyfile=/etc/letsencrypt/privkey.pem 45 | letsencrypt-notebook: check 46 | @test -n "$(PASSWORD)" || \ 47 | (echo "ERROR: PASSWORD not defined or blank"; exit 1) 48 | $(RUN_NOTEBOOK) 49 | -------------------------------------------------------------------------------- /examples/make-deploy/self-signed.makefile: -------------------------------------------------------------------------------- 1 | # Copyright (c) Jupyter Development Team. 2 | # Distributed under the terms of the Modified BSD License. 3 | 4 | self-signed-notebook: PORT?=443 5 | self-signed-notebook: NAME?=notebook 6 | self-signed-notebook: WORK_VOLUME?=$(NAME)-data 7 | self-signed-notebook: DOCKER_ARGS:=-e USE_HTTPS=yes \ 8 | -e PASSWORD=$(PASSWORD) 9 | self-signed-notebook: check 10 | @test -n "$(PASSWORD)" || \ 11 | (echo "ERROR: PASSWORD not defined or blank"; exit 1) 12 | $(RUN_NOTEBOOK) 13 | -------------------------------------------------------------------------------- /examples/make-deploy/softlayer.makefile: -------------------------------------------------------------------------------- 1 | # Copyright (c) Jupyter Development Team. 2 | # Distributed under the terms of the Modified BSD License. 3 | 4 | softlayer-vm: export SOFTLAYER_CPU?=4 5 | softlayer-vm: export SOFTLAYER_DISK_SIZE?=100 6 | softlayer-vm: export SOFTLAYER_MEMORY?=4096 7 | softlayer-vm: export SOFTLAYER_REGION?=wdc01 8 | softlayer-vm: check 9 | @test -n "$(NAME)" || \ 10 | (echo "ERROR: NAME not defined (make help)"; exit 1) 11 | @test -n "$(SOFTLAYER_API_KEY)" || \ 12 | (echo "ERROR: SOFTLAYER_API_KEY not defined (make help)"; exit 1) 13 | @test -n "$(SOFTLAYER_USER)" || \ 14 | (echo "ERROR: SOFTLAYER_USER not defined (make help)"; exit 1) 15 | @test -n "$(SOFTLAYER_DOMAIN)" || \ 16 | (echo "ERROR: SOFTLAYER_DOMAIN not defined (make help)"; exit 1) 17 | @docker-machine create -d softlayer $(NAME) 18 | @echo "DONE: Docker host '$(NAME)' up at $$(docker-machine ip $(NAME))" 19 | 20 | softlayer-dns: HOST_NAME:=$$(docker-machine active) 21 | softlayer-dns: IP:=$$(docker-machine ip $(HOST_NAME)) 22 | softlayer-dns: check 23 | @which slcli > /dev/null || (echo "softlayer cli not found (pip install softlayer)"; exit 1) 24 | @test -n "$(SOFTLAYER_DOMAIN)" || \ 25 | (echo "ERROR: SOFTLAYER_DOMAIN not defined (make help)"; exit 1) 26 | @slcli dns record-add $(SOFTLAYER_DOMAIN) $(HOST_NAME) A $(IP) 27 | -------------------------------------------------------------------------------- /examples/make-deploy/virtualbox.makefile: -------------------------------------------------------------------------------- 1 | # Copyright (c) Jupyter Development Team. 2 | # Distributed under the terms of the Modified BSD License. 3 | 4 | virtualbox-vm: export VIRTUALBOX_CPU_COUNT?=4 5 | virtualbox-vm: export VIRTUALBOX_DISK_SIZE?=100000 6 | virtualbox-vm: export VIRTUALBOX_MEMORY_SIZE?=4096 7 | virtualbox-vm: check 8 | @test -n "$(NAME)" || \ 9 | (echo "ERROR: NAME not defined (make help)"; exit 1) 10 | @docker-machine create -d virtualbox $(NAME) 11 | -------------------------------------------------------------------------------- /minimal-kernel/Dockerfile: -------------------------------------------------------------------------------- 1 | # Copyright (c) Jupyter Development Team. 2 | # Distributed under the terms of the Modified BSD License. 3 | FROM kaixhin/cuda-lasagne:8.0 4 | 5 | MAINTAINER Matt Edwards 6 | 7 | USER root 8 | 9 | # Install all OS dependencies for fully functional notebook server 10 | ENV DEBIAN_FRONTEND noninteractive 11 | RUN apt-get update && apt-get install -yq --no-install-recommends \ 12 | python3-setuptools \ 13 | python3-zmq \ 14 | && apt-get clean 15 | 16 | # Install Tini 17 | RUN python3 -c 'from urllib.request import urlretrieve; \ 18 | urlretrieve("https://github.com/krallin/tini/releases/download/v0.6.0/tini", "tini")' && \ 19 | echo "d5ed732199c36a1189320e6c4859f0169e950692f451c03e7854243b95f4234b *tini" | sha256sum -c - && \ 20 | mv tini /usr/local/bin/tini && \ 21 | chmod +x /usr/local/bin/tini 22 | 23 | # Configure environment 24 | ENV KG_USER jovyan 25 | ENV KG_UID 1000 26 | 27 | # Create jovyan user with UID=1000 28 | RUN useradd -m -s /bin/bash -N -u $KG_UID $KG_USER 29 | 30 | # Install modern pip then kernel gateway 31 | RUN apt-get update && \ 32 | apt-get install -yq --no-install-recommends \ 33 | build-essential \ 34 | python3-dev && \ 35 | easy_install3 pip && \ 36 | pip install jupyter_kernel_gateway==0.2.0 && \ 37 | apt-get remove --purge -y \ 38 | build-essential \ 39 | python3-dev && \ 40 | apt-get autoremove -y && \ 41 | apt-get clean 42 | 43 | # Configure container startup 44 | EXPOSE 8888 45 | WORKDIR /tmp 46 | ENTRYPOINT ["tini", "--", "jupyter", "kernelgateway"] 47 | CMD ["--KernelGatewayApp.ip=0.0.0.0"] 48 | 49 | # Run container as jovyan 50 | USER jovyan 51 | -------------------------------------------------------------------------------- /minimal-kernel/README.md: -------------------------------------------------------------------------------- 1 | # Kernel Gateway Stack 2 | 3 | ## What it Gives You 4 | 5 | * [Jupyter Kernel Gateway](https://github.com/jupyter-incubator/kernel_gateway) that enables programmatic access to kernels 6 | * Python 3 kernel 7 | * No preinstalled scientific computing packages 8 | * [tini](https://github.com/krallin/tini) as the container entrypoint 9 | 10 | ## Basic Use 11 | 12 | The following command starts a container with the Kernel Gateway server listening for HTTP connections on port 8888. 13 | 14 | ``` 15 | docker run -d -p 8888:8888 jupyter/minimal-kernel 16 | ``` 17 | 18 | ## Docker Options and More Information 19 | * For more information on the Kernel Gateway and its configuration options see the 20 | [Kernel Gateway Repository](https://github.com/jupyter-incubator/kernel_gateway#what-it-gives-you). 21 | -------------------------------------------------------------------------------- /minimal-notebook/Dockerfile: -------------------------------------------------------------------------------- 1 | # Copyright (c) Jupyter Development Team. 2 | # Distributed under the terms of the Modified BSD License. 3 | FROM giffordlab/cuda 4 | 5 | MAINTAINER Matt Edwards 6 | 7 | USER root 8 | 9 | # Install all OS dependencies for fully functional notebook server 10 | ENV DEBIAN_FRONTEND noninteractive 11 | RUN apt-get update && apt-get install -yq --no-install-recommends \ 12 | git \ 13 | vim \ 14 | wget \ 15 | build-essential \ 16 | python-dev \ 17 | ca-certificates \ 18 | bzip2 \ 19 | unzip \ 20 | libsm6 \ 21 | pandoc \ 22 | sudo \ 23 | locales \ 24 | libxrender1 \ 25 | && apt-get clean \ 26 | && rm -rf /var/lib/apt/lists/* 27 | RUN echo "en_US.UTF-8 UTF-8" > /etc/locale.gen && \ 28 | locale-gen 29 | 30 | # Install Tini 31 | RUN wget --quiet https://github.com/krallin/tini/releases/download/v0.6.0/tini && \ 32 | echo "d5ed732199c36a1189320e6c4859f0169e950692f451c03e7854243b95f4234b *tini" | sha256sum -c - && \ 33 | mv tini /usr/local/bin/tini && \ 34 | chmod +x /usr/local/bin/tini 35 | 36 | # Configure environment 37 | ENV CONDA_DIR /opt/conda 38 | ENV PATH $CONDA_DIR/bin:$PATH 39 | ENV SHELL /bin/bash 40 | ENV NB_USER jovyan 41 | ENV NB_UID 1000 42 | # ENV LC_ALL en_US.UTF-8 43 | # ENV LANG en_US.UTF-8 44 | # ENV LANGUAGE en_US.UTF-8 45 | 46 | # Create jovyan user with UID=1000 and in the 'users' group 47 | RUN useradd -m -s /bin/bash -N -u $NB_UID $NB_USER && \ 48 | mkdir -p /opt/conda && \ 49 | chown jovyan /opt/conda 50 | 51 | USER jovyan 52 | 53 | # Setup jovyan home directory 54 | RUN mkdir /home/$NB_USER/work && \ 55 | mkdir /home/$NB_USER/.jupyter && \ 56 | mkdir /home/$NB_USER/.local 57 | 58 | # Install conda as jovyan 59 | RUN cd /tmp && \ 60 | mkdir -p $CONDA_DIR && \ 61 | wget --quiet --no-check-certificate https://repo.continuum.io/miniconda/Miniconda3-4.3.31-Linux-x86_64.sh && \ 62 | echo "5551f01f436b6409d467412c33e12ecc4f43b5e029290870f8fdeca403c274e6 *Miniconda3-4.3.31-Linux-x86_64.sh" | sha256sum -c - && \ 63 | /bin/bash Miniconda3-4.3.31-Linux-x86_64.sh -f -b -p $CONDA_DIR && \ 64 | rm Miniconda3-4.3.31-Linux-x86_64.sh 65 | 66 | # Install Jupyter notebook as jovyan 67 | RUN conda install --yes \ 68 | 'notebook=5.4*' \ 69 | terminado \ 70 | && conda clean -yat 71 | 72 | USER root 73 | 74 | # Configure container startup as root 75 | EXPOSE 8888 76 | WORKDIR /home/$NB_USER/work 77 | ENTRYPOINT ["tini", "--"] 78 | CMD ["start-notebook.sh"] 79 | 80 | # Add local files as late as possible to avoid cache busting 81 | COPY start-notebook.sh /usr/local/bin/ 82 | COPY jupyter_notebook_config.py /home/$NB_USER/.jupyter/ 83 | RUN chown -R $NB_USER:users /home/$NB_USER/.jupyter 84 | 85 | # Switch back to jovyan to avoid accidental container runs as root 86 | USER jovyan 87 | -------------------------------------------------------------------------------- /minimal-notebook/README.md: -------------------------------------------------------------------------------- 1 | # Minimal Jupyter Notebook Stack 2 | 3 | ## What it Gives You 4 | 5 | * Jupyter Notebook 4.0.x 6 | * Conda Python 3.x 7 | * No preinstalled scientific computing packages 8 | * Unprivileged user `jovyan` (uid=1000, configurable, see options) in group `users` (gid=100) with ownership over `/home/jovyan` and `/opt/conda` 9 | * [tini](https://github.com/krallin/tini) as the container entrypoint and [start-notebook.sh](./start-notebook.sh) as the default command 10 | * Options for HTTPS, password auth, and passwordless `sudo` 11 | 12 | ## Basic Use 13 | 14 | The following command starts a container with the Notebook server listening for HTTP connections on port 8888 without authentication configured. 15 | 16 | ``` 17 | docker run -d -p 8888:8888 jupyter/minimal-notebook 18 | ``` 19 | 20 | ## Notebook Options 21 | 22 | You can pass [Jupyter command line options](http://jupyter.readthedocs.org/en/latest/config.html#command-line-arguments) through the [`start-notebook.sh` command](https://github.com/jupyter/docker-stacks/blob/master/minimal-notebook/start-notebook.sh#L15) when launching the container. For example, to set the base URL of the notebook server you might do the following: 23 | 24 | ``` 25 | docker run -d -p 8888:8888 jupyter/minimal-notebook start-notebook.sh --NotebookApp.base_url=/some/path 26 | ``` 27 | 28 | You can sidestep the `start-notebook.sh` script entirely by specifying a command other than `start-notebook.sh`. If you do, the `NB_USER` and `GRANT_SUDO` features documented below will not work. See the Docker Options section for details. 29 | 30 | ## Docker Options 31 | 32 | You may customize the execution of the Docker container and the Notebook server it contains with the following optional arguments. 33 | 34 | * `-e PASSWORD="YOURPASS"` - Configures Jupyter Notebook to require the given password. Should be conbined with `USE_HTTPS` on untrusted networks. 35 | * `-e USE_HTTPS=yes` - Configures Jupyter Notebook to accept encrypted HTTPS connections. If a `pem` file containing a SSL certificate and key is not provided (see below), the container will generate a self-signed certificate for you. 36 | * `-e NB_UID=1000` - Specify the uid of the `jovyan` user. Useful to mount host volumes with specific file ownership. For this option to take effect, you must run the container with `--user root`. (The `start-notebook.sh` script will `su jovyan` after adjusting the user id.) 37 | * `-e GRANT_SUDO=yes` - Gives the `jovyan` user passwordless `sudo` capability. Useful for installing OS packages. For this option to take effect, you must run the container with `--user root`. (The `start-notebook.sh` script will `su jovyan` after adding `jovyan` to sudoers.) **You should only enable `sudo` if you trust the user or if the container is running on an isolated host.** 38 | * `-v /some/host/folder/for/work:/home/jovyan/work` - Host mounts the default working directory on the host to preserve work even when the container is destroyed and recreated (e.g., during an upgrade). 39 | * `-v /some/host/folder/for/server.pem:/home/jovyan/.local/share/jupyter/notebook.pem` - Mounts a SSL certificate plus key for `USE_HTTPS`. Useful if you have a real certificate for the domain under which you are running the Notebook server. 40 | 41 | ## Conda Environment 42 | 43 | The default Python 3.x [Conda environment](http://conda.pydata.org/docs/using/envs.html) resides in `/opt/conda`. The commands `ipython`, `python`, `pip`, `easy_install`, and `conda` (among others) are available in this environment. 44 | -------------------------------------------------------------------------------- /minimal-notebook/jupyter_notebook_config.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Jupyter Development Team. 2 | from jupyter_core.paths import jupyter_data_dir 3 | import subprocess 4 | import os 5 | import errno 6 | import stat 7 | 8 | PEM_FILE = os.path.join(jupyter_data_dir(), 'notebook.pem') 9 | 10 | c = get_config() 11 | c.NotebookApp.ip = '*' 12 | c.NotebookApp.port = 8888 13 | c.NotebookApp.open_browser = False 14 | 15 | # Set a certificate if USE_HTTPS is set to any value 16 | if 'USE_HTTPS' in os.environ: 17 | if not os.path.isfile(PEM_FILE): 18 | # Ensure PEM_FILE directory exists 19 | dir_name = os.path.dirname(PEM_FILE) 20 | try: 21 | os.makedirs(dir_name) 22 | except OSError as exc: # Python >2.5 23 | if exc.errno == errno.EEXIST and os.path.isdir(dir_name): 24 | pass 25 | else: raise 26 | # Generate a certificate if one doesn't exist on disk 27 | subprocess.check_call(['openssl', 'req', '-new', 28 | '-newkey', 'rsa:2048', '-days', '365', '-nodes', '-x509', 29 | '-subj', '/C=XX/ST=XX/L=XX/O=generated/CN=generated', 30 | '-keyout', PEM_FILE, '-out', PEM_FILE]) 31 | # Restrict access to PEM_FILE 32 | os.chmod(PEM_FILE, stat.S_IRUSR | stat.S_IWUSR) 33 | c.NotebookApp.certfile = PEM_FILE 34 | 35 | # Set a password if PASSWORD is set 36 | if 'PASSWORD' in os.environ: 37 | from IPython.lib import passwd 38 | c.NotebookApp.password = passwd(os.environ['PASSWORD']) 39 | del os.environ['PASSWORD'] -------------------------------------------------------------------------------- /minimal-notebook/start-notebook.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Handle special flags if we're root 4 | if [ $UID == 0 ] ; then 5 | # Change UID of NB_USER to NB_UID if it does not match 6 | if [ "$NB_UID" != $(id -u $NB_USER) ] ; then 7 | usermod -u $NB_UID $NB_USER 8 | chown -R $NB_UID $CONDA_DIR 9 | fi 10 | 11 | # Enable sudo if requested 12 | if [ ! -z "$GRANT_SUDO" ]; then 13 | echo "$NB_USER ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/notebook 14 | fi 15 | 16 | # Start the notebook server 17 | exec su $NB_USER -c "env PATH=$PATH jupyter notebook $*" 18 | else 19 | # Otherwise just exec the notebook 20 | exec jupyter notebook $* 21 | fi 22 | -------------------------------------------------------------------------------- /pyspark-notebook/Dockerfile: -------------------------------------------------------------------------------- 1 | # Copyright (c) Jupyter Development Team. 2 | # Distributed under the terms of the Modified BSD License. 3 | FROM jupyter/minimal-notebook 4 | 5 | MAINTAINER Jupyter Project 6 | 7 | USER root 8 | 9 | # Spark dependencies 10 | ENV APACHE_SPARK_VERSION 1.5.1 11 | RUN apt-get -y update && \ 12 | apt-get install -y --no-install-recommends openjdk-7-jre-headless && \ 13 | apt-get clean 14 | RUN wget -qO - http://d3kbcqa49mib13.cloudfront.net/spark-${APACHE_SPARK_VERSION}-bin-hadoop2.6.tgz | tar -xz -C /usr/local/ 15 | RUN cd /usr/local && ln -s spark-${APACHE_SPARK_VERSION}-bin-hadoop2.6 spark 16 | 17 | # Mesos dependencies 18 | RUN apt-key adv --keyserver keyserver.ubuntu.com --recv E56151BF && \ 19 | DISTRO=debian && \ 20 | CODENAME=wheezy && \ 21 | echo "deb http://repos.mesosphere.io/${DISTRO} ${CODENAME} main" > /etc/apt/sources.list.d/mesosphere.list && \ 22 | apt-get -y update && \ 23 | apt-get --no-install-recommends -y --force-yes install mesos=0.22.1-1.0.debian78 && \ 24 | apt-get clean 25 | 26 | # Spark and Mesos pointers 27 | ENV SPARK_HOME /usr/local/spark 28 | ENV PYTHONPATH $SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.8.2.1-src.zip 29 | ENV MESOS_NATIVE_LIBRARY /usr/local/lib/libmesos.so 30 | 31 | USER jovyan 32 | 33 | # Install Python 3 packages 34 | RUN conda install --yes \ 35 | 'ipywidgets=4.0*' \ 36 | 'pandas=0.17*' \ 37 | 'matplotlib=1.4*' \ 38 | 'scipy=0.16*' \ 39 | 'seaborn=0.6*' \ 40 | 'scikit-learn=0.16*' \ 41 | && conda clean -yt 42 | 43 | # Install Python 2 packages and kernel spec 44 | RUN conda create -p $CONDA_DIR/envs/python2 python=2.7 \ 45 | 'ipython=4.0*' \ 46 | 'ipywidgets=4.0*' \ 47 | 'pandas=0.17*' \ 48 | 'matplotlib=1.4*' \ 49 | 'scipy=0.16*' \ 50 | 'seaborn=0.6*' \ 51 | 'scikit-learn=0.16*' \ 52 | pyzmq \ 53 | && conda clean -yt 54 | 55 | USER root 56 | 57 | # Install Python 2 kernel spec globally to avoid permission problems when NB_UID 58 | # switching at runtime. 59 | RUN $CONDA_DIR/envs/python2/bin/python \ 60 | $CONDA_DIR/envs/python2/bin/ipython \ 61 | kernelspec install-self 62 | 63 | USER jovyan 64 | 65 | -------------------------------------------------------------------------------- /pyspark-notebook/README.md: -------------------------------------------------------------------------------- 1 | # Jupyter Notebook Python, Spark, Mesos Stack 2 | 3 | ## What it Gives You 4 | 5 | * Jupyter Notebook 4.0.x 6 | * Conda Python 3.x and Python 2.7.x environments 7 | * pyspark, pandas, matplotlib, scipy, seaborn, scikit-learn pre-installed 8 | * Spark 1.5.1 for use in local mode or to connect to a cluster of Spark workers 9 | * Mesos client 0.22 binary that can communicate with a Mesos master 10 | * Unprivileged user `jovyan` (uid=1000, configurable, see options) in group `users` (gid=100) with ownership over `/home/jovyan` and `/opt/conda` 11 | * [tini](https://github.com/krallin/tini) as the container entrypoint and [start-notebook.sh](../minimal-notebook/start-notebook.sh) as the default command 12 | * Options for HTTPS, password auth, and passwordless `sudo` 13 | 14 | ## Basic Use 15 | 16 | The following command starts a container with the Notebook server listening for HTTP connections on port 8888 without authentication configured. 17 | 18 | ``` 19 | docker run -d -p 8888:8888 jupyter/pyspark-notebook 20 | ``` 21 | 22 | ## Using Spark Local Mode 23 | 24 | This configuration is nice for using Spark on small, local data. 25 | 26 | 0. Run the container as shown above. 27 | 2. Open a Python 2 or 3 notebook. 28 | 3. Create a `SparkContext` configured for local mode. 29 | 30 | For example, the first few cells in a Python 3 notebook might read: 31 | 32 | ```python 33 | import pyspark 34 | sc = pyspark.SparkContext('local[*]') 35 | 36 | # do something to prove it works 37 | rdd = sc.parallelize(range(1000)) 38 | rdd.takeSample(False, 5) 39 | ``` 40 | 41 | In a Python 2 notebook, prefix the above with the following code to ensure the local workers use Python 2 as well. 42 | 43 | ```python 44 | import os 45 | os.environ['PYSPARK_PYTHON'] = 'python2' 46 | 47 | # include pyspark cells from above here ... 48 | ``` 49 | 50 | ## Connecting to a Spark Cluster on Mesos 51 | 52 | This configuration allows your compute cluster to scale with your data. 53 | 54 | 0. [Deploy Spark on Mesos](http://spark.apache.org/docs/latest/running-on-mesos.html). 55 | 1. Configure each slave with [the `--no-switch_user` flag](https://open.mesosphere.com/reference/mesos-slave/) or create the `jovyan` user on every slave node. 56 | 2. Ensure Python 2.x and/or 3.x and any Python libraries you wish to use in your Spark lambda functions are installed on your Spark workers. 57 | 3. Run the Docker container with `--net=host` in a location that is network addressable by all of your Spark workers. (This is a [Spark networking requirement](http://spark.apache.org/docs/latest/cluster-overview.html#components).) 58 | * NOTE: When using `--net=host`, you must also use the flags `--pid=host -e TINI_SUBREAPER=true`. See https://github.com/jupyter/docker-stacks/issues/64 for details. 59 | 4. Open a Python 2 or 3 notebook. 60 | 5. Create a `SparkConf` instance in a new notebook pointing to your Mesos master node (or Zookeeper instance) and Spark binary package location. 61 | 6. Create a `SparkContext` using this configuration. 62 | 63 | For example, the first few cells in a Python 3 notebook might read: 64 | 65 | ```python 66 | import os 67 | # make sure pyspark tells workers to use python3 not 2 if both are installed 68 | os.environ['PYSPARK_PYTHON'] = '/usr/bin/python3' 69 | 70 | import pyspark 71 | conf = pyspark.SparkConf() 72 | 73 | # point to mesos master or zookeeper entry (e.g., zk://10.10.10.10:2181/mesos) 74 | conf.setMaster("mesos://10.10.10.10:5050") 75 | # point to spark binary package in HDFS or on local filesystem on all slave 76 | # nodes (e.g., file:///opt/spark/spark-1.5.1-bin-hadoop2.6.tgz) 77 | conf.set("spark.executor.uri", "hdfs://10.122.193.209/spark/spark-1.5.1-bin-hadoop2.6.tgz") 78 | # set other options as desired 79 | conf.set("spark.executor.memory", "8g") 80 | conf.set("spark.core.connection.ack.wait.timeout", "1200") 81 | 82 | # create the context 83 | sc = pyspark.SparkContext(conf=conf) 84 | 85 | # do something to prove it works 86 | rdd = sc.parallelize(range(100000000)) 87 | rdd.sumApprox(3) 88 | ``` 89 | 90 | To use Python 2 in the notebook and on the workers, change the `PYSPARK_PYTHON` environment variable to point to the location of the Python 2.x interpreter binary. If you leave this environment variable unset, it defaults to `python`. 91 | 92 | Of course, all of this can be hidden in an [IPython kernel startup script](http://ipython.org/ipython-doc/stable/development/config.html?highlight=startup#startup-files), but "explicit is better than implicit." :) 93 | 94 | ## Notebook Options 95 | 96 | You can pass [Jupyter command line options](http://jupyter.readthedocs.org/en/latest/config.html#command-line-arguments) through the [`start-notebook.sh` command](https://github.com/jupyter/docker-stacks/blob/master/minimal-notebook/start-notebook.sh#L15) when launching the container. For example, to set the base URL of the notebook server you might do the following: 97 | 98 | ``` 99 | docker run -d -p 8888:8888 jupyter/pyspark-notebook start-notebook.sh --NotebookApp.base_url=/some/path 100 | ``` 101 | 102 | You can sidestep the `start-notebook.sh` script entirely by specifying a command other than `start-notebook.sh`. If you do, the `NB_USER` and `GRANT_SUDO` features documented below will not work. See the Docker Options section for details. 103 | 104 | ## Docker Options 105 | 106 | You may customize the execution of the Docker container and the Notebook server it contains with the following optional arguments. 107 | 108 | * `-e PASSWORD="YOURPASS"` - Configures Jupyter Notebook to require the given password. Should be conbined with `USE_HTTPS` on untrusted networks. 109 | * `-e USE_HTTPS=yes` - Configures Jupyter Notebook to accept encrypted HTTPS connections. If a `pem` file containing a SSL certificate and key is not found in `/home/jovyan/.ipython/profile_default/security/notebook.pem`, the container will generate a self-signed certificate for you. 110 | * `-e NB_UID=1000` - Specify the uid of the `jovyan` user. Useful to mount host volumes with specific file ownership. For this option to take effect, you must run the container with `--user root`. (The `start-notebook.sh` script will `su jovyan` after adjusting the user id.) 111 | * `-e GRANT_SUDO=yes` - Gives the `jovyan` user passwordless `sudo` capability. Useful for installing OS packages. For this option to take effect, you must run the container with `--user root`. (The `start-notebook.sh` script will `su jovyan` after adding `jovyan` to sudoers.) **You should only enable `sudo` if you trust the user or if the container is running on an isolated host.** 112 | * `-v /some/host/folder/for/work:/home/jovyan/work` - Host mounts the default working directory on the host to preserve work even when the container is destroyed and recreated (e.g., during an upgrade). 113 | * `-v /some/host/folder/for/server.pem:/home/jovyan/.local/share/jupyter/notebook.pem` - Mounts a SSL certificate plus key for `USE_HTTPS`. Useful if you have a real certificate for the domain under which you are running the Notebook server. 114 | * `-p 4040:4040` - Opens the port for the [Spark Monitoring and Instrumentation UI](http://spark.apache.org/docs/latest/monitoring.html). Note every new spark context that is created is put onto an incrementing port (ie. 4040, 4041, 4042, etc.), and it might be necessary to open multiple ports. `docker run -d -p 8888:8888 -p 4040:4040 -p 4041:4041 jupyter/pyspark-notebook` 115 | 116 | 117 | ## Conda Environments 118 | 119 | The default Python 3.x [Conda environment](http://conda.pydata.org/docs/using/envs.html) resides in `/opt/conda`. A second Python 2.x Conda environment exists in `/opt/conda/envs/python2`. You can [switch to the python2 environment](http://conda.pydata.org/docs/using/envs.html#change-environments-activate-deactivate) in a shell by entering the following: 120 | 121 | ``` 122 | source activate python2 123 | ``` 124 | 125 | You can return to the default environment with this command: 126 | 127 | ``` 128 | source deactivate 129 | ``` 130 | 131 | The commands `ipython`, `python`, `pip`, `easy_install`, and `conda` (among others) are available in both environments. 132 | -------------------------------------------------------------------------------- /r-notebook/Dockerfile: -------------------------------------------------------------------------------- 1 | # Copyright (c) Jupyter Development Team. 2 | # Distributed under the terms of the Modified BSD License. 3 | FROM jupyter/minimal-notebook 4 | 5 | MAINTAINER Jupyter Project 6 | 7 | USER root 8 | 9 | # R pre-requisites 10 | RUN apt-get update && \ 11 | apt-get install -y --no-install-recommends \ 12 | fonts-dejavu \ 13 | gfortran \ 14 | gcc && apt-get clean 15 | 16 | USER jovyan 17 | 18 | # R packages 19 | RUN conda config --add channels r 20 | RUN conda install --yes \ 21 | 'r-base=3.2*' \ 22 | 'r-irkernel=0.5*' \ 23 | 'r-plyr=1.8*' \ 24 | 'r-devtools=1.9*' \ 25 | 'r-dplyr=0.4*' \ 26 | 'r-ggplot2=1.0*' \ 27 | 'r-tidyr=0.3*' \ 28 | 'r-shiny=0.12*' \ 29 | 'r-rmarkdown=0.8*' \ 30 | 'r-forecast=5.8*' \ 31 | 'r-stringr=0.6*' \ 32 | 'r-rsqlite=1.0*' \ 33 | 'r-reshape2=1.4*' \ 34 | 'r-nycflights13=0.1*' \ 35 | 'r-caret=6.0*' \ 36 | 'r-rcurl=1.95*' \ 37 | 'r-randomforest=4.6*' && conda clean -yt 38 | -------------------------------------------------------------------------------- /r-notebook/README.md: -------------------------------------------------------------------------------- 1 | # Jupyter Notebook R Stack 2 | 3 | ## What it Gives You 4 | 5 | * Jupyter Notebook 4.0.x 6 | * Conda R v3.2.x and channel 7 | * plyr, devtools, dplyr, ggplot2, tidyr, shiny, rmarkdown, forecast, stringr, rsqlite, reshape2, nycflights13, caret, rcurl, and randomforest pre-installed 8 | * Unprivileged user `jovyan` (uid=1000, configurable, see options) in group `users` (gid=100) with ownership over `/home/jovyan` and `/opt/conda` 9 | * [tini](https://github.com/krallin/tini) as the container entrypoint and [start-notebook.sh](../minimal-notebook/start-notebook.sh) as the default command 10 | * Options for HTTPS, password auth, and passwordless `sudo` 11 | 12 | ## Basic Use 13 | 14 | The following command starts a container with the Notebook server listening for HTTP connections on port 8888 without authentication configured. 15 | 16 | ``` 17 | docker run -d -p 8888:8888 jupyter/r-notebook 18 | ``` 19 | 20 | ## Notebook Options 21 | 22 | You can pass [Jupyter command line options](http://jupyter.readthedocs.org/en/latest/config.html#command-line-arguments) through the [`start-notebook.sh` command](https://github.com/jupyter/docker-stacks/blob/master/minimal-notebook/start-notebook.sh#L15) when launching the container. For example, to set the base URL of the notebook server you might do the following: 23 | 24 | ``` 25 | docker run -d -p 8888:8888 jupyter/r-notebook start-notebook.sh --NotebookApp.base_url=/some/path 26 | ``` 27 | 28 | You can sidestep the `start-notebook.sh` script entirely by specifying a command other than `start-notebook.sh`. If you do, the `NB_USER` and `GRANT_SUDO` features documented below will not work. See the Docker Options section for details. 29 | 30 | ## Docker Options 31 | 32 | You may customize the execution of the Docker container and the Notebook server it contains with the following optional arguments. 33 | 34 | * `-e PASSWORD="YOURPASS"` - Configures Jupyter Notebook to require the given password. Should be conbined with `USE_HTTPS` on untrusted networks. 35 | * `-e USE_HTTPS=yes` - Configures Jupyter Notebook to accept encrypted HTTPS connections. If a `pem` file containing a SSL certificate and key is not found in `/home/jovyan/.ipython/profile_default/security/notebook.pem`, the container will generate a self-signed certificate for you. 36 | * `-e NB_UID=1000` - Specify the uid of the `jovyan` user. Useful to mount host volumes with specific file ownership. For this option to take effect, you must run the container with `--user root`. (The `start-notebook.sh` script will `su jovyan` after adjusting the user id.) 37 | * `-e GRANT_SUDO=yes` - Gives the `jovyan` user passwordless `sudo` capability. Useful for installing OS packages. For this option to take effect, you must run the container with `--user root`. (The `start-notebook.sh` script will `su jovyan` after adding `jovyan` to sudoers.) **You should only enable `sudo` if you trust the user or if the container is running on an isolated host.** 38 | * `-v /some/host/folder/for/work:/home/jovyan/work` - Host mounts the default working directory on the host to preserve work even when the container is destroyed and recreated (e.g., during an upgrade). 39 | * `-v /some/host/folder/for/server.pem:/home/jovyan/.local/share/jupyter/notebook.pem` - Mounts a SSL certificate plus key for `USE_HTTPS`. Useful if you have a real certificate for the domain under which you are running the Notebook server. 40 | -------------------------------------------------------------------------------- /scipy-notebook/Dockerfile: -------------------------------------------------------------------------------- 1 | # Copyright (c) Jupyter Development Team. 2 | # Distributed under the terms of the Modified BSD License. 3 | FROM giffordlab/minimal-notebook 4 | 5 | MAINTAINER Matt Edwards 6 | 7 | USER jovyan 8 | 9 | # Install Python 3 packages 10 | RUN conda install --yes \ 11 | 'ipywidgets=4.0*' \ 12 | 'pandas=0.17*' \ 13 | 'matplotlib=1.4*' \ 14 | 'scipy=0.16*' \ 15 | 'seaborn=0.6*' \ 16 | 'scikit-learn=0.16*' \ 17 | 'scikit-image=0.11*' \ 18 | 'sympy=0.7*' \ 19 | 'cython=0.22*' \ 20 | 'patsy=0.4*' \ 21 | 'statsmodels=0.6*' \ 22 | 'cloudpickle=0.1*' \ 23 | 'dill=0.2*' \ 24 | 'numba=0.22*' \ 25 | 'bokeh=0.10*' \ 26 | && conda clean -yt 27 | 28 | # Install Python 2 packages 29 | RUN conda create -p $CONDA_DIR/envs/python2 python=2.7 \ 30 | 'ipython=4.0*' \ 31 | 'ipywidgets=4.0*' \ 32 | 'pandas=0.17*' \ 33 | 'matplotlib=1.4*' \ 34 | 'scipy=0.16*' \ 35 | 'seaborn=0.6*' \ 36 | 'scikit-learn=0.16*' \ 37 | 'scikit-image=0.11*' \ 38 | 'sympy=0.7*' \ 39 | 'cython=0.22*' \ 40 | 'patsy=0.4*' \ 41 | 'statsmodels=0.6*' \ 42 | 'cloudpickle=0.1*' \ 43 | 'dill=0.2*' \ 44 | 'numba=0.22*' \ 45 | 'bokeh=0.10*' \ 46 | pyzmq \ 47 | && conda clean -yt 48 | 49 | USER root 50 | 51 | # Install Python 2 kernel spec globally to avoid permission problems when NB_UID 52 | # switching at runtime. 53 | RUN $CONDA_DIR/envs/python2/bin/python \ 54 | $CONDA_DIR/envs/python2/bin/ipython \ 55 | kernelspec install-self 56 | 57 | USER jovyan 58 | -------------------------------------------------------------------------------- /scipy-notebook/README.md: -------------------------------------------------------------------------------- 1 | # Jupyter Notebook Scientific Python Stack 2 | 3 | ## What it Gives You 4 | 5 | * Jupyter Notebook 4.0.x 6 | * Conda Python 3.x and Python 2.7.x environments 7 | * pandas, matplotlib, scipy, seaborn, scikit-learn, scikit-image, sympy, cython, patsy, statsmodel, cloudpickle, dill, numba, bokeh pre-installed 8 | * Unprivileged user `jovyan` (uid=1000, configurable, see options) in group `users` (gid=100) with ownership over `/home/jovyan` and `/opt/conda` 9 | * [tini](https://github.com/krallin/tini) as the container entrypoint and [start-notebook.sh](../minimal-notebook/start-notebook.sh) as the default command 10 | * Options for HTTPS, password auth, and passwordless `sudo` 11 | 12 | ## Basic Use 13 | 14 | The following command starts a container with the Notebook server listening for HTTP connections on port 8888 without authentication configured. 15 | 16 | ``` 17 | docker run -d -p 8888:8888 jupyter/scipy-notebook 18 | ``` 19 | 20 | ## Notebook Options 21 | 22 | You can pass [Jupyter command line options](http://jupyter.readthedocs.org/en/latest/config.html#command-line-arguments) through the [`start-notebook.sh` command](https://github.com/jupyter/docker-stacks/blob/master/minimal-notebook/start-notebook.sh#L15) when launching the container. For example, to set the base URL of the notebook server you might do the following: 23 | 24 | ``` 25 | docker run -d -p 8888:8888 jupyter/scipy-notebook start-notebook.sh --NotebookApp.base_url=/some/path 26 | ``` 27 | 28 | You can sidestep the `start-notebook.sh` script entirely by specifying a command other than `start-notebook.sh`. If you do, the `NB_USER` and `GRANT_SUDO` features documented below will not work. See the Docker Options section for details. 29 | 30 | ## Docker Options 31 | 32 | You may customize the execution of the Docker container and the Notebook server it contains with the following optional arguments. 33 | 34 | * `-e PASSWORD="YOURPASS"` - Configures Jupyter Notebook to require the given password. Should be conbined with `USE_HTTPS` on untrusted networks. 35 | * `-e USE_HTTPS=yes` - Configures Jupyter Notebook to accept encrypted HTTPS connections. If a `pem` file containing a SSL certificate and key is not found in `/home/jovyan/.ipython/profile_default/security/notebook.pem`, the container will generate a self-signed certificate for you. 36 | * `-e NB_UID=1000` - Specify the uid of the `jovyan` user. Useful to mount host volumes with specific file ownership. For this option to take effect, you must run the container with `--user root`. (The `start-notebook.sh` script will `su jovyan` after adjusting the user id.) 37 | * `-e GRANT_SUDO=yes` - Gives the `jovyan` user passwordless `sudo` capability. Useful for installing OS packages. For this option to take effect, you must run the container with `--user root`. (The `start-notebook.sh` script will `su jovyan` after adding `jovyan` to sudoers.) **You should only enable `sudo` if you trust the user or if the container is running on an isolated host.** 38 | * `-v /some/host/folder/for/work:/home/jovyan/work` - Host mounts the default working directory on the host to preserve work even when the container is destroyed and recreated (e.g., during an upgrade). 39 | * `-v /some/host/folder/for/server.pem:/home/jovyan/.local/share/jupyter/notebook.pem` - Mounts a SSL certificate plus key for `USE_HTTPS`. Useful if you have a real certificate for the domain under which you are running the Notebook server. 40 | 41 | ## Conda Environments 42 | 43 | The default Python 3.x [Conda environment](http://conda.pydata.org/docs/using/envs.html) resides in `/opt/conda`. A second Python 2.x Conda environment exists in `/opt/conda/envs/python2`. You can [switch to the python2 environment](http://conda.pydata.org/docs/using/envs.html#change-environments-activate-deactivate) in a shell by entering the following: 44 | 45 | ``` 46 | source activate python2 47 | ``` 48 | 49 | You can return to the default environment with this command: 50 | 51 | ``` 52 | source deactivate 53 | ``` 54 | 55 | The commands `ipython`, `python`, `pip`, `easy_install`, and `conda` (among others) are available in both environments. 56 | -------------------------------------------------------------------------------- /singleuser/Dockerfile: -------------------------------------------------------------------------------- 1 | # Build as giffordlab/jupyter-singleuser 2 | # Run with the DockerSpawner in JupyterHub 3 | 4 | FROM giffordlab/datascience-notebook 5 | 6 | MAINTAINER Matt Edwards 7 | 8 | USER root 9 | 10 | # Get the script at the commit for Jupyterhub release 0.7.0.. 11 | RUN wget -q https://raw.githubusercontent.com/jupyterhub/jupyterhub/c3faef8e2ae011f6650730804a2fe88ec5b6f572/scripts/jupyterhub-singleuser \ 12 | -O /usr/local/bin/jupyterhub-singleuser && \ 13 | chmod 755 /usr/local/bin/jupyterhub-singleuser 14 | 15 | ADD install_jupyterhub /tmp/install_jupyterhub 16 | ARG JUPYTERHUB_VERSION=master 17 | # install pinned jupyterhub and ensure notebook is installed 18 | RUN python3 /tmp/install_jupyterhub && \ 19 | python3 -m pip install notebook 20 | 21 | ADD singleuser.sh /srv/singleuser/singleuser.sh 22 | USER jovyan 23 | # smoke test that it's importable at least 24 | # RUN sh /srv/singleuser/singleuser.sh -h 25 | CMD ["sh", "/srv/singleuser/singleuser.sh"] 26 | -------------------------------------------------------------------------------- /singleuser/README.md: -------------------------------------------------------------------------------- 1 | # jupyter/singleuser 2 | 3 | Built from the `jupyter/scipy-notebook` base image. 4 | 5 | This image contains a single user notebook server for use with 6 | [JupyterHub](https://github.com/jupyter/jupyterhub). In particular, it is meant 7 | to be used with the 8 | [DockerSpawner](https://github.com/jupyter/dockerspawner/blob/master/dockerspawner/dockerspawner.py) 9 | class to launch user notebook servers within docker containers. 10 | 11 | This particular server runs (within the container) as the `jupyter` user, with 12 | home directory at `/home/jupyter`, and the IPython example notebooks at 13 | `/home/jupyter/examples`. This home directory is *not* persistent by default; 14 | thus, this image should be used with temporary or demonstration JupyterHub 15 | servers. 16 | -------------------------------------------------------------------------------- /singleuser/singleuser.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | exec jupyterhub-singleuser \ 4 | --port=8888 \ 5 | --ip=0.0.0.0 \ 6 | --user=$JPY_USER \ 7 | --cookie-name=$JPY_COOKIE_NAME \ 8 | --base-url=$JPY_BASE_URL \ 9 | --hub-prefix=$JPY_HUB_PREFIX \ 10 | --hub-api-url=$JPY_HUB_API_URL \ 11 | --notebook-dir='~/notebooks' 12 | -------------------------------------------------------------------------------- /systemuser/Dockerfile: -------------------------------------------------------------------------------- 1 | # Build as giffordlab/jupyter-systemuser 2 | # Run with the DockerSpawner in JupyterHub 3 | 4 | FROM giffordlab/jupyter-singleuser 5 | 6 | MAINTAINER Matt Edwards 7 | 8 | USER root 9 | ENV SHELL /bin/bash 10 | ADD systemuser.sh /srv/singleuser/systemuser.sh 11 | 12 | CMD ["sh", "/srv/singleuser/systemuser.sh"] 13 | -------------------------------------------------------------------------------- /systemuser/README.md: -------------------------------------------------------------------------------- 1 | # jupyter/systemuser 2 | 3 | Built from the `jupyter/scipy-notebook` base image. 4 | 5 | This image contains a single user notebook server for use with 6 | [JupyterHub](https://github.com/jupyter/jupyterhub). In particular, it is meant 7 | to be used with the 8 | [SystemUserSpawner](https://github.com/jupyter/dockerspawner/blob/master/dockerspawner/systemuserspawner.py) 9 | class to launch user notebook servers within docker containers. 10 | 11 | This particular server initially runs (within the container) as the `root` user. 12 | When the container is run, it expects to have access to environment variables 13 | for `$USER`, `$USER_ID`, and `$HOME`. It will create a user inside the container 14 | with the specified username and id, and then run the notebook server as that 15 | user (using `sudo`). It also expects the user's home directory (specified by 16 | `$HOME`) to exist -- for example, by being mounted as a volume in the container 17 | when it is run. 18 | -------------------------------------------------------------------------------- /systemuser/systemuser.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | set -e 3 | sudo -E PATH="${CONDA_DIR}/bin:$PATH" -u $USER jupyterhub-singleuser \ 4 | --port=8888 \ 5 | --ip=0.0.0.0 \ 6 | --user=$JPY_USER \ 7 | --cookie-name=$JPY_COOKIE_NAME \ 8 | --base-url=$JPY_BASE_URL \ 9 | --hub-prefix=$JPY_HUB_PREFIX \ 10 | --hub-api-url=$JPY_HUB_API_URL \ 11 | --notebook-dir='~/notebooks' \ 12 | $@ 13 | --------------------------------------------------------------------------------