├── .gitignore ├── LICENSE ├── Makefile ├── README.md ├── build_images.sh ├── cuda-dl-lab └── Dockerfile ├── docker-compose.yml └── jupyterhub ├── Dockerfile └── jupyterhub_config.py /.gitignore: -------------------------------------------------------------------------------- 1 | # Environment variables (configuration secrets) 2 | .env 3 | # Certificates 4 | jupyterhub/cert/ 5 | # Firewall 6 | fw/ 7 | # Drivers 8 | nvidia-drivers.run 9 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | BSD 3-Clause License 2 | 3 | Copyright (c) 2021, Benjamin D. Evans 4 | All rights reserved. 5 | 6 | Redistribution and use in source and binary forms, with or without 7 | modification, are permitted provided that the following conditions are met: 8 | 9 | 1. Redistributions of source code must retain the above copyright notice, this 10 | list of conditions and the following disclaimer. 11 | 12 | 2. Redistributions in binary form must reproduce the above copyright notice, 13 | this list of conditions and the following disclaimer in the documentation 14 | and/or other materials provided with the distribution. 15 | 16 | 3. Neither the name of the copyright holder nor the names of its 17 | contributors may be used to endorse or promote products derived from 18 | this software without specific prior written permission. 19 | 20 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 21 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 22 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 23 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 24 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 25 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 26 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 27 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 28 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | help: 2 | @cat Makefile 3 | 4 | builder: 5 | docker buildx use default 6 | 7 | build: builder 8 | docker compose down 9 | #./build_images.sh 10 | docker compose build 11 | 12 | clean: builder 13 | docker compose down 14 | docker system prune -f 15 | docker buildx prune 16 | #./build_images.sh -n 17 | docker compose build --no-cache 18 | 19 | login: 20 | docker login 21 | 22 | push: login 23 | docker tag cuda-dl-lab biocomplab/cuda-dl-lab 24 | docker image push biocomplab/cuda-dl-lab 25 | 26 | stop: 27 | docker compose down 28 | 29 | hub: 30 | docker compose up -d jupyterhub 31 | 32 | test: 33 | docker compose up cuda-test 34 | 35 | .PHONY: build builder clean login push stop hub test 36 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # dl-hub 2 | **A Dockerised JupyterHub environment for Deep Learning with GPUs** 3 | 4 | JupyterHub is a customisable, flexible, scalable, portable system for bringing Jupyter notebooks (labs) to groups of users. It gives users access to computing resources (including GPUs!) through a browser without them needing to install, configure or maintain the computing environment. 5 | 6 | | [JupyterHub Schematic](https://jupyterhub.readthedocs.io/en/stable/) | 7 | | :--: | 8 | | *JupyterHub schematic from the [official documentation](https://jupyterhub.readthedocs.io/en/stable/).* | 9 | 10 | This repository builds a hub which spawns isolated, dockerised [JupyterLab](https://jupyterlab.readthedocs.io/en/latest/) environments with mounted GPUs for deep learning acceleration. The containers are spawned from images based on the [Jupyter Docker Stacks](https://github.com/jupyter/docker-stacks) but built using an [NVIDIA CUDA base image](https://hub.docker.com/r/nvidia/cuda). Note that GPUs are currently shared between all spawned JupyterLab environments although it may be possible to allocate them in a round-robin system (PRs accepted!). 11 | 12 | ## Setup 13 | These instructions assume you are using the latest Ubuntu LTS on your server. To install and setup the required packages, execute these commands: 14 | 15 | ### Install NVIDIA drivers 16 | First update the system and blacklist the `nouveau` drivers. 17 | ```bash 18 | # Update the installed packages 19 | sudo apt-get update && sudo apt-get upgrade && sudo apt-get install curl 20 | 21 | # Blacklist noveau 22 | sudo bash -c "echo blacklist nouveau > /etc/modprobe.d/blacklist-nvidia-nouveau.conf" 23 | sudo bash -c "echo options nouveau modeset=0 >> /etc/modprobe.d/blacklist-nvidia-nouveau.conf" 24 | 25 | # If the `nouveau` driver was already in use, it is necessary to rebuild the kernel and reboot 26 | # Regenerate initramfs 27 | sudo update-initramfs -u 28 | 29 | # Reboot 30 | sudo reboot 31 | ``` 32 | 33 | Check the installed Graphics card with: `sudo lshw -C display`. 34 | 35 | ```bash 36 | # Versions default to the last (tested working) versions 37 | # Search here: https://www.nvidia.com/Download/index.aspx?lang=en-uk 38 | export NVIDIA_DRIVER_VERSION=545.29.06 # ${1:-460.39} 39 | 40 | # [Optional] Stop X-server if a GUI is installed 41 | # sudo service lightdm stop # Assuming a lightdm desktop. Alternative: gdm | kdm 42 | # sudo init 3 # This may also be necessary 43 | 44 | # Install NVIDIA drivers 45 | sudo apt-get install build-essential gcc-multilib dkms 46 | curl -o nvidia-drivers-$NVIDIA_DRIVER_VERSION.run https://uk.download.nvidia.com/XFree86/Linux-x86_64/$NVIDIA_DRIVER_VERSION/NVIDIA-Linux-x86_64-$NVIDIA_DRIVER_VERSION.run 47 | chmod +x nvidia-drivers-$NVIDIA_DRIVER_VERSION.run 48 | sudo ./nvidia-drivers-$NVIDIA_DRIVER_VERSION.run --dkms --no-opengl-files 49 | # run nvidia-xconfig: Y 50 | 51 | # Verify installation 52 | nvidia-smi 53 | # read -p "Press any key to reboot..." -n1 -s 54 | sudo reboot # Alternative: sudo service lightdm start 55 | ``` 56 | 57 | ### Install Docker Engine, Docker Compose and NVIDIA Container Toolkit 58 | ```bash 59 | # Install Docker Engine and Docker Compose plugin 60 | # https://docs.docker.com/engine/install/ubuntu/ 61 | # https://docs.docker.com/compose/install/linux/ 62 | sudo apt-get install \ 63 | ca-certificates \ 64 | curl \ 65 | gnupg 66 | # apt-transport-https \ 67 | # gnupg-agent \ 68 | # software-properties-common 69 | 70 | sudo install -m 0755 -d /etc/apt/keyrings 71 | curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg 72 | sudo chmod a+r /etc/apt/keyrings/docker.gpg 73 | 74 | echo \ 75 | "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \ 76 | "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \ 77 | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null 78 | 79 | sudo apt-get update && sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin 80 | 81 | # Verify installation 82 | sudo docker run hello-world 83 | 84 | # Install NVIDIA Container Toolkit 85 | # https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#docker 86 | distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \ 87 | && curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \ 88 | && curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \ 89 | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \ 90 | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list 91 | 92 | sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit 93 | 94 | sudo nvidia-ctk runtime configure --runtime=docker 95 | # Alternatively use containerd or crio as the runtime 96 | 97 | sudo systemctl restart docker 98 | ``` 99 |
Alternatively install the deprecated `nvidia-docker2` package for Kubernetes [click to expand]

100 | 101 | ```bash 102 | # NOTE: nvidia-docker2 is still required for Kubernetes but otherwise only nvidia-container-toolkit 103 | # https://docs.nvidia.com/datacenter/cloud-native/kubernetes/install-k8s.html 104 | distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \ 105 | && curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add - \ 106 | && curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list 107 | 108 | # NOTE: I had to manually edit /etc/apt/sources.list.d/nvidia-docker.list to change 18.04 to 20.04 109 | # Install nvidia-docker2 to provide the legacy runtime=nvidia for use with docker-compose (see: https://github.com/NVIDIA/nvidia-docker/issues/1268#issuecomment-632692949) 110 | sudo apt-get update && sudo apt-get install -y nvidia-docker2 111 | # sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit 112 | sudo systemctl restart docker 113 | ``` 114 |

115 | 116 | ```bash 117 | # Verify installation 118 | docker run --rm --gpus all nvidia/cuda:12.2.2-cudnn8-devel-ubuntu22.04 nvidia-smi 119 | ``` 120 | 121 | ### Post Installation Steps 122 | ```bash 123 | # Add users to the `docker` group to let them use docker on the server without `sudo` 124 | # https://docs.docker.com/engine/install/linux-postinstall/ 125 | sudo groupadd docker 126 | sudo usermod -aG docker $USER 127 | 128 | # Activate changes 129 | newgrp docker 130 | 131 | # Verify 132 | docker run hello-world 133 | 134 | # Configure Docker to start on boot with systemd 135 | sudo systemctl enable docker.service 136 | sudo systemctl enable containerd.service 137 | ``` 138 | 139 | ### Create `.env` file for sensitive configuration details 140 | In addition to these files, create a `.env` file with the necessary secrets set as variables e.g.: 141 | 142 | ```python 143 | COMPOSE_PROJECT_NAME=dl_hub 144 | AUTH_SERVER_ADDRESS=authenticator.uni.ac.uk 145 | ADMIN_USERS='user1 user2 user3' # A string of user names separated by spaces 146 | # DOCKER_NETWORK_NAME=${COMPOSE_PROJECT_NAME}_default 147 | ``` 148 | See [here](https://docs.docker.com/compose/environment-variables/) for documentation on setting and passing environment variables to `docker compose`. 149 | 150 | ### Authentication 151 | 152 | Depending on your environment, you will probably want to configure a more sophisticated authenticator e.g. the [`PAMAuthenticator` or `ldapauthenticator`](https://github.com/jupyterhub/jupyterhub#configuration). You will need configuration details from the university system administrators for this in order to use the existing user authentication systems. These details should be configured in [`jupyterhub/jupyterhub_config.py`](https://github.com/bdevans/dl-hub/blob/main/jupyterhub/jupyterhub_config.py) (with secrets in `.env` as necessary). 153 | 154 | Your organisation may also be able to issue and sign SSL certificates for the server. This repository currently assumes they are in `jupyterhub/cert/`. Appropriate configuration settings then need to be set in [`jupyterhub/jupyterhub_config.py`](https://github.com/bdevans/dl-hub/blob/main/jupyterhub/jupyterhub_config.py) e.g.: 155 | 156 | ```python 157 | # Configure SSL 158 | c.JupyterHub.ssl_key = '/srv/jupyterhub/hub.key' 159 | c.JupyterHub.ssl_cert = '/srv/jupyterhub/chain.crt' 160 | c.JupyterHub.port = 443 161 | 162 | # Configure configurable-http-proxy to redirect http to https 163 | c.ConfigurableHTTPProxy.command = ['configurable-http-proxy', '--redirect-port', '80'] 164 | ``` 165 | 166 | The corresponding lines where the certificates are installed in [`jupyterhub/Dockerfile`](https://github.com/bdevans/dl-hub/blob/main/jupyterhub/Dockerfile) will also need to be edited. 167 | 168 | ### Optional additional steps 169 | 170 | * Customise JupyterHub 171 | - Edit `jupyterhub_config.py` 172 | * Automatically add new users to the `docker` group to let them use docker on the server without `sudo` 173 | - `sudo nano /etc/adduser.conf` then add the following lines 174 | * `EXTRA_GROUPS="docker"` # Separate groups with spaces e.g. `"docker users"` 175 | * `ADD_EXTRA_GROUPS=1` 176 | * Mount additional partitions 177 | * Move Docker disk to separate partition 178 | - `sudo systemctl stop docker` 179 | - Copy or move the data e.g.: `sudo rsync -aP /var/lib/docker/ /path/to/your/docker_data` 180 | - Edit `/etc/docker/daemon.json` to add `"data-root": "/path/to/your/docker_data"` 181 | - `sudo systemctl start docker` 182 | * Set up build target of `jupyter/docker-stacks with --build-arg` 183 | * Install extras, e.g.: 184 | - `screen` 185 | - `tmux` 186 | - `htop` 187 | - `nvtop` 188 | * Create a list or dictionary of allowed images which will be presented as a dropdown list of options for users at logon e.g.: 189 | - `c.DockerSpawner.allowed_images = {"Latest": "cuda-dl-lab:11.4.2-cudnn8", "Previous": "cuda-dl-lab:11.2.2-cudnn8"}` 190 | - `c.DockerSpawner.allowed_images = ["cuda-dl-lab:11.4.2-cudnn8", "cuda-dl-lab:11.2.2-cudnn8"]` 191 | * Schedule a backup! 192 | 193 | ## Updating 194 | 195 | ### [NVIDIA drivers](https://www.nvidia.co.uk/Download/index.aspx?lang=en-uk) 196 | * Find the latest Linux 64-bit drivers for your graphics cards: https://www.nvidia.co.uk/Download/index.aspx?lang=en-uk 197 | 198 | ```bash 199 | # sudo service lightdm stop # or gdm or kdm depending on your display manager 200 | curl -o nvidia-drivers.run https://uk.download.nvidia.com/XFree86/Linux-x86_64/$NVIDIA_DRIVER_VERSION/NVIDIA-Linux-x86_64-$NVIDIA_DRIVER_VERSION.run 201 | chmod +x nvidia-drivers-$NVIDIA_DRIVER_VERSION.run 202 | sudo ./nvidia-drivers-$NVIDIA_DRIVER_VERSION.run --dkms --no-opengl-files 203 | nvidia-smi 204 | sudo reboot 205 | ``` 206 | * Confirm the drivers work: `docker run --rm --gpus all nvidia/cuda:12.2.2-cudnn8-devel-ubuntu22.04 nvidia-smi` 207 | 208 | ### Docker, Docker Compose and `nvidia-container-toolkit` 209 | * `sudo apt update && sudo apt upgrade` 210 | 211 | ### [Docker CUDA images](https://hub.docker.com/r/nvidia/cuda/tags?page=1&name=devel-ubuntu) 212 | * Edit `docker-compose.yml` (or `build_images.sh` if you prefer to use the script) to update: 213 | * `CUDA_VERSION` 214 | * `CUDNN_VERSION` 215 | * Eventually `ubuntu-22.04` 216 | * Alternatively, you can set the variables in `.env` or pass arguments e.g. `docker compose build --build-arg CUDA_VERSION=x.y.z` 217 | 218 | * Edit `cuda-dl-lab/Dockerfile` to update with new versions: 219 | * [`'tensorflow-gpu==2.12.0'`](https://github.com/tensorflow/tensorflow/releases): [Docs](https://www.tensorflow.org/install/gpu); [Code](https://github.com/tensorflow/tensorflow) 220 | * [`TF_MODELS_VERSION=v2.12.0`](https://github.com/tensorflow/models/releases): [Code](https://github.com/tensorflow/models) 221 | * [`'torch==2.0.0'`](https://github.com/pytorch/pytorch/releases): [Docs](https://pytorch.org/get-started/locally/); [Code](https://github.com/pytorch/pytorch) 222 | * [`magma-cuda118`](https://anaconda.org/search?q=magma): https://anaconda.org/search?q=magma 223 | 224 | * `make build` or `make clean` to rebuild the `cuda-dl-lab` images from scratch 225 | 226 | ### [JupyterHub](https://github.com/jupyterhub/jupyterhub/tags) 227 | * Update `JUPYTERHUB_VERSION=4.0.2` in: 228 | - `docker-compose.yml` 229 | - `jupyterhub/Dockerfile` (optional) 230 | 231 | * Edit `jupyterhub/jupyterhub_config.py` for any additional volumes 232 | 233 | ### Restart the Hub 234 | * `make stop` (in case the hub is running) 235 | * `make hub` 236 | -------------------------------------------------------------------------------- /build_images.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | usage() { echo "Usage: $0 [-c ] [-d ] [-n]" 1>&2; exit 1; } 4 | 5 | #CUDA_VERSION=12.1.1 6 | CUDA_VERSION=11.8.0 7 | CUDNN_VERSION=8 8 | OS_BASE=ubuntu22.04 9 | NO_CACHE="" 10 | 11 | while getopts ":c:d:b:n" opt; do 12 | case "${opt}" in 13 | c) 14 | CUDA_VERSION="${OPTARG}" 15 | ;; 16 | d) 17 | CUDNN_VERSION="${OPTARG}" 18 | ;; 19 | b) 20 | OS_BASE="${OPTARG}" 21 | ;; 22 | n) 23 | NO_CACHE=--no-cache 24 | ;; 25 | *) 26 | echo "Unknown option" 27 | usage 28 | ;; 29 | esac 30 | done 31 | shift $((OPTIND-1)) 32 | 33 | CUDA_BASE=nvidia/cuda:${CUDA_VERSION}-cudnn${CUDNN_VERSION}-devel-${OS_BASE} 34 | 35 | # docker rmi cuda-dl-lab:previous 36 | # docker tag cuda-dl-lab:latest cuda-dl-lab:previous 37 | LAST_ID=$(docker images cuda-dl-lab:latest --format "{{.ID}}") 38 | docker tag $LAST_ID cuda-dl-lab:previous 39 | docker build -t cuda-stacks-foundation ${NO_CACHE} --build-arg ROOT_CONTAINER=${CUDA_BASE} https://github.com/jupyter/docker-stacks.git#main:images/docker-stacks-foundation/ 40 | docker build -t cuda-base-notebook ${NO_CACHE} --build-arg BASE_CONTAINER=cuda-stacks-foundation https://github.com/jupyter/docker-stacks.git#main:images/base-notebook/ 41 | docker build -t cuda-minimal-notebook --build-arg BASE_CONTAINER=cuda-base-notebook https://github.com/jupyter/docker-stacks.git#main:images/minimal-notebook/ 42 | docker build -t cuda-scipy-notebook --build-arg BASE_CONTAINER=cuda-minimal-notebook https://github.com/jupyter/docker-stacks.git#main:images/scipy-notebook/ 43 | docker build -t cuda-dl-lab -t cuda-dl-lab:${CUDA_VERSION}-cudnn${CUDNN_VERSION} --build-arg BASE_CONTAINER=cuda-scipy-notebook --build-arg CUDA_VERSION=${CUDA_VERSION} --build-arg CUDNN_VERSION=${CUDNN_VERSION} ./cuda-dl-lab/ 44 | -------------------------------------------------------------------------------- /cuda-dl-lab/Dockerfile: -------------------------------------------------------------------------------- 1 | ARG BASE_CONTAINER=cuda-scipy-notebook 2 | FROM ${BASE_CONTAINER} 3 | 4 | # Install extra packages 5 | USER root 6 | RUN apt-get update && \ 7 | apt-get install -y --no-install-recommends \ 8 | texlive-latex-extra \ 9 | texlive-fonts-recommended \ 10 | dvipng \ 11 | cm-super \ 12 | rsync && \ 13 | apt-get clean && rm -rf /var/lib/apt/lists/* 14 | USER ${NB_UID} 15 | WORKDIR "${HOME}" 16 | 17 | # Pass --build-arg values to set library versions 18 | ARG CUDA_VERSION 19 | ENV CUDA_VERSION=${CUDA_VERSION} 20 | ARG CUDNN_VERSION 21 | ENV CUDNN_VERSION=${CUDNN_VERSION} 22 | 23 | # Install Tensorflow 24 | RUN pip install --quiet \ 25 | 'pyyaml>=6.0.1' \ 26 | 'tensorflow==2.15.*' \ 27 | 'tf-models-official' \ 28 | tf-explain && \ 29 | fix-permissions "${CONDA_DIR}" && \ 30 | fix-permissions "${HOME}" 31 | 32 | RUN pip install --quiet \ 33 | 'torch==2.1.*' \ 34 | torchvision \ 35 | torchaudio \ 36 | pytorch-ignite && \ 37 | fix-permissions "${CONDA_DIR}" && \ 38 | fix-permissions "/home/${NB_USER}" 39 | # --index-url https://download.pytorch.org/whl/cu118 40 | 41 | # Install extenstions 42 | RUN conda install --quiet --yes \ 43 | opencv \ 44 | watermark \ 45 | papermill && \ 46 | #jupyterlab-git && \ 47 | conda clean --all -f -y && \ 48 | # Use --pre flag to get beta 49 | pip install --quiet --pre \ 50 | jupyterlab-git && \ 51 | pip install \ 52 | jupyterlab-drawio && \ 53 | jupyter labextension install @jupyter-widgets/jupyterlab-manager --no-build && \ 54 | jupyter lab build && \ 55 | #npm cache clean --force && \ 56 | #jupyter lab clean && \ 57 | #rm -rf /home/$NB_USER/.cache/yarn && \ 58 | fix-permissions "${CONDA_DIR}" && \ 59 | fix-permissions "/home/${NB_USER}" 60 | 61 | #RUN conda install --quiet --yes -c pytorch \ 62 | # pytorch \ 63 | # torchvision \ 64 | # torchaudio \ 65 | # ignite \ 66 | # magma-cuda110 && \ 67 | # conda clean --all -f -y && \ 68 | # npm cache clean --force && \ 69 | # jupyter notebook --generate-config && \ 70 | # jupyter lab clean && \ 71 | # rm -rf /home/$NB_USER/.cache/yarn && \ 72 | # fix-permissions $CONDA_DIR && \ 73 | # fix-permissions /home/$NB_USER 74 | 75 | LABEL maintainer="ben.d.evans@gmail.com" \ 76 | name="Deep learning image" \ 77 | description="A Python based GPU-accelerated Jupyter image for deep learning." \ 78 | org.opencontainers.image.authors="Ben Evans" \ 79 | org.opencontainers.image.url="https://github.com/biocomplab/dl-hub" 80 | -------------------------------------------------------------------------------- /docker-compose.yml: -------------------------------------------------------------------------------- 1 | 2 | version: "3.9" 3 | 4 | services: 5 | jupyterhub: 6 | build: 7 | context: ./jupyterhub 8 | args: 9 | - JUPYTERHUB_VERSION=${JUPYTERHUB_VERSION:-4.0.2} 10 | image: jupyterhub 11 | ports: 12 | #- "80:8000" 13 | - "8000:8000" 14 | - "80:80" 15 | # This may be unnecessary as the proxy may handle port mapping for SSL 16 | - "443:443" 17 | container_name: jupyterhub-container 18 | deploy: 19 | resources: 20 | reservations: 21 | devices: 22 | - capabilities: 23 | - gpu 24 | volumes: 25 | - /var/run/docker.sock:/var/run/docker.sock 26 | #- /shared:/shared 27 | #- jupyterhub_data:/srv/jupyterhub 28 | # Volumes for PAM authentication. Require restart to recognise new users 29 | #- /etc/passwd:/etc/passwd 30 | #- /etc/shadow:/etc/shadow 31 | #- /etc/group:/etc/group 32 | environment: 33 | NVIDIA_VISIBLE_DEVICES: all 34 | NVIDIA_DRIVER_CAPABILITIES: all 35 | DOCKER_JUPYTER_IMAGE: cuda-dl-lab 36 | #DOCKER_JUPYTER_IMAGE: jupyterhub/singleuser 37 | DOCKER_NETWORK_NAME: ${COMPOSE_PROJECT_NAME}_default 38 | HUB_IP: jupyterhub 39 | #HUB_IP: "0.0.0.0" 40 | #AUTH_SERVER_ADDRESS: ${AUTH_SERVER_ADDRESS} 41 | ADMIN_USERS: ${ADMIN_USERS} 42 | # WIP: These entries specify how to build each image with `docker-compose build` or 43 | # `docker-compose build --no-cache`. Variables can be set in the `.env` file or with `--build-arg`. 44 | # They also created containers which are made to exit with the `echo` command. 45 | # TODO: Prevent containers being spawned at all (except jupyterhub-container). 46 | # This can be achieved manually by specifying the service: `docker-compose up -d jupyterhub` 47 | cuda-stacks-foundation: 48 | build: 49 | context: https://github.com/jupyter/docker-stacks.git#main:images/docker-stacks-foundation 50 | dockerfile: Dockerfile 51 | args: 52 | - ROOT_CONTAINER=nvidia/cuda:${CUDA_VERSION:-11.8.0}-cudnn${CUDNN_VERSION:-8}-devel-${OS_BASE:-ubuntu22.04} 53 | image: cuda-stacks-foundation 54 | command: echo 55 | cuda-base-notebook: 56 | build: 57 | context: https://github.com/jupyter/docker-stacks.git#main:images/base-notebook 58 | dockerfile: Dockerfile 59 | args: 60 | - BASE_CONTAINER=cuda-stacks-foundation 61 | depends_on: 62 | - cuda-stacks-foundation 63 | image: cuda-base-notebook 64 | command: echo 65 | cuda-minimal-notebook: 66 | build: 67 | context: https://github.com/jupyter/docker-stacks.git#main:images/minimal-notebook 68 | dockerfile: Dockerfile 69 | args: 70 | - BASE_CONTAINER=cuda-base-notebook 71 | depends_on: 72 | - cuda-base-notebook 73 | image: cuda-minimal-notebook 74 | command: echo 75 | cuda-scipy-notebook: 76 | build: 77 | context: https://github.com/jupyter/docker-stacks.git#main:images/scipy-notebook 78 | dockerfile: Dockerfile 79 | args: 80 | - BASE_CONTAINER=cuda-minimal-notebook 81 | depends_on: 82 | - cuda-minimal-notebook 83 | image: cuda-scipy-notebook 84 | command: echo 85 | cuda-dl-lab: 86 | build: 87 | context: ./cuda-dl-lab 88 | args: 89 | - BASE_CONTAINER=cuda-scipy-notebook 90 | - CUDA_VERSION=${CUDA_VERSION:-11.8.0} 91 | - CUDNN_VERSION=${CUDNN_VERSION:-8} 92 | - OS_BASE=${OS_BASE:-ubuntu22.04} 93 | depends_on: 94 | - cuda-scipy-notebook 95 | image: cuda-dl-lab 96 | command: echo 97 | 98 | # This service exists to test that the containers can mount GPUs 99 | # Run with: `docker-compose up cuda-test` 100 | cuda-test: 101 | image: nvidia/cuda:${CUDA_VERSION:-11.8.0}-cudnn${CUDNN_VERSION:-8}-devel-${OS_BASE:-ubuntu22.04} 102 | command: nvidia-smi 103 | deploy: 104 | resources: 105 | reservations: 106 | devices: 107 | - capabilities: 108 | - gpu 109 | 110 | volumes: 111 | jupyterhub_data: 112 | -------------------------------------------------------------------------------- /jupyterhub/Dockerfile: -------------------------------------------------------------------------------- 1 | ARG JUPYTERHUB_VERSION=4.0.2 2 | FROM jupyterhub/jupyterhub:${JUPYTERHUB_VERSION} 3 | RUN pip install --upgrade pip 4 | RUN pip install \ 5 | dockerspawner \ 6 | #https://github.com/andreas-h/sshauthenticator/archive/v0.1.zip 7 | #oauthenticator \ 8 | jupyterhub-dummyauthenticator \ 9 | #jupyterhub-ldapauthenticator \ 10 | #jupyterhub-kerberosauthenticator \ 11 | nbgrader 12 | 13 | WORKDIR /srv/jupyterhub/ 14 | #RUN jupyterhub --generate-config 15 | COPY jupyterhub_config.py . 16 | #COPY cert/chain.crt . 17 | #COPY cert/key/titan.key . 18 | #ENV DOCKER_JUPYTER_CONTAINER=gpu-lab-container \ 19 | # DOCKER_JUPYTER_IMAGE='jupyterhub/singleuser:latest' \ 20 | # DOCKER_NETWORK_NAME=hub_default \ 21 | # HUB_IP='' 22 | CMD ["jupyterhub", "-f", "jupyterhub_config.py"] 23 | -------------------------------------------------------------------------------- /jupyterhub/jupyterhub_config.py: -------------------------------------------------------------------------------- 1 | import os 2 | 3 | 4 | c.JupyterHub.hub_ip = os.environ["HUB_IP"] 5 | c.JupyterHub.spawner_class = "dockerspawner.DockerSpawner" 6 | 7 | # Configure Docker Spawner 8 | c.DockerSpawner.image = os.environ["DOCKER_JUPYTER_IMAGE"] # or 'jupyterhub/singleuser:latest' 9 | c.DockerSpawner.network_name = os.environ["DOCKER_NETWORK_NAME"] 10 | # https://jupyterhub-dockerspawner.readthedocs.io/en/latest/spawner-types.html#using-docker-swarm-not-swarm-mode 11 | # c.DockerSpawner.host_ip = "0.0.0.0" 12 | c.DockerSpawner.host_ip = os.environ["DOCKER_NETWORK_NAME"] 13 | # NOTE: This may need nvidia-docker2 14 | c.DockerSpawner.extra_host_config = {'runtime': 'nvidia'} 15 | # c.DockerSpawner.extra_host_config = {'gpus': 'all'} 16 | c.DockerSpawner.remove_containers = True 17 | 18 | # Set up volumes and mounts within spawned containers 19 | notebook_dir = os.environ.get('DOCKER_NOTEBOOK_DIR') or '/home/jovyan/work' 20 | c.DockerSpawner.notebook_dir = notebook_dir 21 | # Mount the real user's Docker volume on the host to the notebook user's 22 | # notebook directory in the container 23 | c.DockerSpawner.volumes = { 24 | 'jupyterhub-user-{username}': notebook_dir, 25 | #'jupyterhub-shared': '/home/jovyan/work/shared', 26 | #'/shared': '/home/jovyan/work/shared', 27 | '/shared': '/shared', 28 | #'jupyterhub-data': '/home/jovyan/work/data' 29 | '/home/{username}': '/home/jovyan/work/data' 30 | } 31 | 32 | # This is the simplest authenticator for setup and testing 33 | c.JupyterHub.authenticator_class = 'dummyauthenticator.DummyAuthenticator' 34 | # Set the password by adding PASSWORD="" in the .env file 35 | c.DummyAuthenticator.password = os.environ.get("PASSWORD") 36 | 37 | # SSH Authenticator 38 | #c.JupyterHub.authenticator_class = 'sshauthenticator.SSHAuthenticator' 39 | #c.SSHAuthenticator.server_address = os.environ["AUTH_SERVER_ADDRESS"] 40 | #c.SSHAuthenticator.server_port = 22 41 | 42 | c.Authenticator.admin_users = set(os.environ.get("ADMIN_USERS").split(" ")) 43 | # c.JupyterHub.admin_access = True # Admins can log in as other users 44 | 45 | # TODO: Configure SSL 46 | #c.JupyterHub.ssl_key = '/srv/jupyterhub/titan.key' 47 | #c.JupyterHub.ssl_cert = '/srv/jupyterhub/chain.crt' 48 | #c.JupyterHub.port = 443 49 | c.JupyterHub.port = 80 50 | # Configure configurable-http-proxy to redirect http to https 51 | #c.ConfigurableHTTPProxy.command = ['configurable-http-proxy', '--redirect-port', '80'] 52 | 53 | c.Spawner.environment = {'GRANT_SUDO': 'yes'} 54 | # c.Spawner.default_url = '/lab' 55 | c.Spawner.cmd=["jupyter-labhub"] 56 | --------------------------------------------------------------------------------