├── .gitignore ├── README.md ├── older-versions ├── all-pt112-tf29-jax0314-py39 │ ├── Dockerfile │ └── README.md ├── all-pt200-tf212-jax048-py311 │ ├── Dockerfile │ └── README.md ├── all-tf24-py36 │ └── Dockerfile ├── all-tf29-pt112-py39 │ ├── README.md │ └── dockerfile ├── lean-tf115-py36 │ └── Dockerfile ├── lean-tf24-py36 │ └── Dockerfile ├── pt112-tf29-jax0314-py39-updated │ └── Dockerfile ├── pt112-tf29-jax0314-py39 │ ├── Dockerfile │ └── README.md └── ultralean-tf24-py38 │ ├── Dockerfile │ └── README.md └── pt211-tf215-cudatk120-py311 ├── Dockerfile └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Gradient base Docker image 2 | 3 | This repo contains the Dockerfiles used to build the Gradient base Machine Learning Docker image which includes PyTorch, TensorFlow, Transformers, and other popular ML/AI libraries. The Docker image is hosted on [Dockerhub](https://hub.docker.com/r/paperspace/gradient-base). 4 | -------------------------------------------------------------------------------- /older-versions/all-pt112-tf29-jax0314-py39/Dockerfile: -------------------------------------------------------------------------------- 1 | # Paperspace Dockerfile for Gradient base image 2 | # Paperspace image is located in Dockerhub registry: paperspace/gradient_base 3 | 4 | # ================================================================== 5 | # module list 6 | # ------------------------------------------------------------------ 7 | # python 3.9.13 (apt) 8 | # jupyter latest (pip) 9 | # pytorch latest (pip) 10 | # tensorflow latest (pip) 11 | # jupyterlab latest (pip) 12 | # keras latest (pip) # Comes installed with Tensorflow 13 | # opencv 4.5.1 (git) 14 | # numpy latest (pip) 15 | # scipy latest (pip) 16 | # pandas latest (pip) 17 | # cloudpickle latest (pip) 18 | # scikit-image latest (pip) 19 | # scikit-learn latest (pip) 20 | # matplotlib latest (pip) 21 | # ipython latest (pip) 22 | # ipykernel latest (pip) 23 | # ipywidgets latest (pip) 24 | # gradient latest (pip) 25 | # Cython latest (pip) 26 | # tqdm latest (pip) 27 | # gdown latest (pip) 28 | # xgboost latest (pip) 29 | # pillow latest (pip) 30 | # seaborn latest (pip) 31 | # SQLAlchemy latest (pip) 32 | # spacy latest (pip) 33 | # nltk latest (pip) 34 | # jsonify latest (pip) 35 | # boto3 latest (pip) 36 | # transformers latest (pip) 37 | # sentence-transformers latest (pip) 38 | # opencv-python latest (pip) 39 | # JAX latest (pip) 40 | # JAXlib latest (pip) 41 | # ================================================================== 42 | 43 | # Ubuntu 20.04, CUDA Toolkit 11.2, CUDNN 8 44 | ## Check with Josh about the driver interactions here <- 45 | 46 | FROM nvcr.io/nvidia/cuda:11.2.1-cudnn8-devel-ubuntu20.04 47 | ENV LANG C.UTF-8 48 | 49 | # Setting shell to bash 50 | ENV SHELL=/bin/bash 51 | SHELL ["/bin/bash", "-c"] 52 | 53 | RUN APT_INSTALL="apt-get install -y --no-install-recommends" && \ 54 | PIP_INSTALL="python3 -m pip --no-cache-dir install --upgrade" && \ 55 | GIT_CLONE="git clone --depth 10" && \ 56 | 57 | rm -rf /var/lib/apt/lists/* \ 58 | /etc/apt/sources.list.d/cuda.list \ 59 | /etc/apt/sources.list.d/nvidia-ml.list && \ 60 | 61 | apt-get update && \ 62 | 63 | # ================================================================== 64 | # tools 65 | # ------------------------------------------------------------------ 66 | 67 | sed -e '\|/usr/share/man|s|^#*|#|g' -i /etc/dpkg/dpkg.cfg.d/excludes && \ 68 | 69 | DEBIAN_FRONTEND=noninteractive $APT_INSTALL \ 70 | build-essential \ 71 | apt-utils \ 72 | ca-certificates \ 73 | wget \ 74 | rsync \ 75 | git \ 76 | vim \ 77 | libssl-dev \ 78 | curl \ 79 | openssh-client \ 80 | unzip \ 81 | unrar \ 82 | zip \ 83 | awscli \ 84 | csvkit \ 85 | emacs \ 86 | joe \ 87 | jq \ 88 | dialog \ 89 | man-db \ 90 | manpages \ 91 | manpages-dev \ 92 | manpages-posix \ 93 | manpages-posix-dev \ 94 | nano \ 95 | iputils-ping \ 96 | sudo \ 97 | ffmpeg \ 98 | libsm6 \ 99 | libxext6 \ 100 | && \ 101 | 102 | rm -f /usr/bin/man && \ 103 | dpkg-divert --quiet --remove --rename /usr/bin/man && \ 104 | rm -f /usr/share/man/man1/sh.1.gz && \ 105 | dpkg-divert --quiet --remove --rename /usr/share/man/man1/sh.1.gz && \ 106 | 107 | $GIT_CLONE https://github.com/Kitware/CMake ~/cmake && \ 108 | cd ~/cmake && \ 109 | ./bootstrap && \ 110 | make -j"$(nproc)" install && \ 111 | 112 | 113 | # ================================================================== 114 | # Python 115 | # ------------------------------------------------------------------ 116 | 117 | # Installing python3.9 118 | DEBIAN_FRONTEND=noninteractive \ 119 | $APT_INSTALL software-properties-common && \ 120 | add-apt-repository ppa:deadsnakes/ppa -y && \ 121 | 122 | DEBIAN_FRONTEND=noninteractive $APT_INSTALL \ 123 | python3.9 \ 124 | python3.9-dev \ 125 | python3-distutils-extra \ 126 | && \ 127 | 128 | # Installing pip 129 | wget -O ~/get-pip.py \ 130 | https://bootstrap.pypa.io/get-pip.py && \ 131 | python3.9 ~/get-pip.py && \ 132 | 133 | # Add symlink so python and python3 commands use same 134 | # python3.9 executable 135 | ln -s /usr/bin/python3.9 /usr/local/bin/python3 && \ 136 | ln -s /usr/bin/python3.9 /usr/local/bin/python && \ 137 | 138 | # Intalling Python packages 139 | $PIP_INSTALL \ 140 | numpy \ 141 | scipy \ 142 | pandas \ 143 | cloudpickle \ 144 | scikit-image \ 145 | scikit-learn \ 146 | matplotlib \ 147 | ipython \ 148 | ipykernel \ 149 | ipywidgets \ 150 | gradient \ 151 | Cython \ 152 | tqdm \ 153 | gdown \ 154 | xgboost \ 155 | pillow \ 156 | seaborn \ 157 | SQLAlchemy \ 158 | spacy \ 159 | nltk \ 160 | jsonify \ 161 | boto3 \ 162 | transformers \ 163 | sentence-transformers \ 164 | datasets \ 165 | opencv-python \ 166 | && \ 167 | 168 | 169 | # ================================================================== 170 | # boost 171 | # ------------------------------------------------------------------ 172 | 173 | DEBIAN_FRONTEND=noninteractive $APT_INSTALL \ 174 | libboost-all-dev \ 175 | && \ 176 | 177 | 178 | # ================================================================== 179 | # jupyter 180 | # ------------------------------------------------------------------ 181 | 182 | $PIP_INSTALL \ 183 | jupyter \ 184 | && \ 185 | 186 | 187 | # ================================================================== 188 | # PyTorch 189 | # ------------------------------------------------------------------ 190 | 191 | $PIP_INSTALL torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu116 && \ 192 | 193 | 194 | # ================================================================== 195 | # TensorFlow 196 | # ------------------------------------------------------------------ 197 | 198 | # Based on https://www.tensorflow.org/install and 199 | # https://www.tensorflow.org/install/pip, so is now not -gpu 200 | 201 | $PIP_INSTALL \ 202 | tensorflow \ 203 | && \ 204 | 205 | # ================================================================== 206 | # JAX 207 | # ------------------------------------------------------------------ 208 | 209 | 210 | $PIP_INSTALL \ 211 | "jax[cuda111]" -f https://storage.googleapis.com/jax-releases/jax_releases.html \ 212 | https://storage.googleapis.com/jax-releases/cuda11/jaxlib-0.3.8+cuda11.cudnn82-cp39-none-manylinux2014_x86_64.whl \ 213 | && \ 214 | 215 | # ================================================================== 216 | # JupyterLab 217 | # ------------------------------------------------------------------ 218 | 219 | $PIP_INSTALL \ 220 | jupyterlab \ 221 | && \ 222 | 223 | 224 | # ================================================================== 225 | # Node.js and Jupyter Notebook Extensions 226 | # ------------------------------------------------------------------ 227 | 228 | curl -sL https://deb.nodesource.com/setup_16.x | bash && \ 229 | $APT_INSTALL nodejs && \ 230 | $PIP_INSTALL jupyter_contrib_nbextensions jupyterlab-git && \ 231 | jupyter contrib nbextension install --sys-prefix && \ 232 | 233 | 234 | # ================================================================== 235 | # Conda 236 | # ------------------------------------------------------------------ 237 | 238 | # wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh && \ 239 | # /bin/bash ~/miniconda.sh -b -p /opt/conda && \ 240 | 241 | 242 | # ================================================================== 243 | # Config & Cleanup 244 | # ------------------------------------------------------------------ 245 | 246 | ldconfig && \ 247 | apt-get clean && \ 248 | apt-get autoremove && \ 249 | rm -rf /var/lib/apt/lists/* /tmp/* ~/* 250 | 251 | 252 | EXPOSE 8888 6006 -------------------------------------------------------------------------------- /older-versions/all-pt112-tf29-jax0314-py39/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gradient-ai/base-container/b1cffa23de83edece0d64762569abe55131cdafd/older-versions/all-pt112-tf29-jax0314-py39/README.md -------------------------------------------------------------------------------- /older-versions/all-pt200-tf212-jax048-py311/Dockerfile: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Paperspace Dockerfile for Gradient base image 4 | # Paperspace image is located in Dockerhub registry: paperspace/gradient_base 5 | 6 | # ================================================================== 7 | # Module list 8 | # ------------------------------------------------------------------ 9 | # python 3.11.0 (apt) 10 | # torch 2.0 (pip) 11 | # torchvision 0.13.1 (pip) 12 | # torchaudio 0.12.1 (pip) 13 | # tensorflow 2.9.2 (pip) 14 | # jax 0.3.23 (pip) 15 | # transformers 4.21.3 (pip) 16 | # datasets 2.4.0 (pip) 17 | # jupyterlab 3.4.6 (pip) 18 | # numpy 1.23.4 (pip) 19 | # scipy 1.9.2 (pip) 20 | # pandas 1.5.0 (pip) 21 | # cloudpickle 2.2.0 (pip) 22 | # scikit-image 0.19.3 (pip) 23 | # scikit-learn 1.1.2 (pip) 24 | # matplotlib 3.6.1 (pip) 25 | # ipython 8.5.0 (pip) 26 | # ipykernel 6.16.0 (pip) 27 | # ipywidgets 8.0.2 (pip) 28 | # cython 0.29.32 (pip) 29 | # tqdm 4.64.1 (pip) 30 | # gdown 4.5.1 (pip) 31 | # xgboost 1.6.2 (pip) 32 | # pillow 9.2.0 (pip) 33 | # seaborn 0.12.0 (pip) 34 | # sqlalchemy 1.4.41 (pip) 35 | # spacy 3.4.1 (pip) 36 | # nltk 3.7 (pip) 37 | # boto3 1.24.90 (pip) 38 | # tabulate 0.9.0 (pip) 39 | # future 0.18.2 (pip) 40 | # gradient 2.0.6 (pip) 41 | # jsonify 0.5 (pip) 42 | # opencv-python 4.6.0.66 (pip) 43 | # sentence-transformers 2.2.2 (pip) 44 | # wandb 0.13.4 (pip) 45 | # nodejs 16.x latest (apt) 46 | # default-jre latest (apt) 47 | # default-jdk latest (apt) 48 | 49 | 50 | # ================================================================== 51 | # Initial setup 52 | # ------------------------------------------------------------------ 53 | 54 | # Ubuntu 20.04 as base image 55 | FROM ubuntu:20.04 56 | RUN yes| unminimize 57 | 58 | # Set ENV variables 59 | ENV LANG C.UTF-8 60 | ENV SHELL=/bin/bash 61 | ENV DEBIAN_FRONTEND=noninteractive 62 | 63 | ENV APT_INSTALL="apt-get install -y --no-install-recommends" 64 | ENV PIP_INSTALL="python3 -m pip --no-cache-dir install --upgrade" 65 | ENV GIT_CLONE="git clone --depth 10" 66 | 67 | 68 | # ================================================================== 69 | # Tools 70 | # ------------------------------------------------------------------ 71 | 72 | RUN apt-get update && \ 73 | $APT_INSTALL \ 74 | apt-utils \ 75 | gcc \ 76 | make \ 77 | pkg-config \ 78 | apt-transport-https \ 79 | build-essential \ 80 | ca-certificates \ 81 | wget \ 82 | rsync \ 83 | git \ 84 | vim \ 85 | mlocate \ 86 | libssl-dev \ 87 | curl \ 88 | openssh-client \ 89 | unzip \ 90 | unrar \ 91 | zip \ 92 | csvkit \ 93 | emacs \ 94 | joe \ 95 | jq \ 96 | dialog \ 97 | man-db \ 98 | manpages \ 99 | manpages-dev \ 100 | manpages-posix \ 101 | manpages-posix-dev \ 102 | nano \ 103 | iputils-ping \ 104 | sudo \ 105 | ffmpeg \ 106 | libsm6 \ 107 | libxext6 \ 108 | libboost-all-dev \ 109 | cifs-utils \ 110 | software-properties-common 111 | 112 | 113 | # ================================================================== 114 | # Python 115 | # ------------------------------------------------------------------ 116 | 117 | #Based on https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa 118 | 119 | # Adding repository for python3.11 120 | RUN add-apt-repository ppa:deadsnakes/ppa -y && \ 121 | 122 | # Installing python3.11 123 | $APT_INSTALL \ 124 | python3.11 \ 125 | python3.11-dev \ 126 | python3.11-venv \ 127 | python3-distutils-extra 128 | 129 | # Add symlink so python and python3 commands use same python3.9 executable 130 | RUN ln -s /usr/bin/python3.11 /usr/local/bin/python3 && \ 131 | ln -s /usr/bin/python3.11 /usr/local/bin/python 132 | 133 | # Installing pip 134 | RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.11 135 | ENV PATH=$PATH:/root/.local/bin 136 | 137 | 138 | # ================================================================== 139 | # Installing CUDA packages (CUDA Toolkit 11.6.2 & CUDNN 8.4.1) 140 | # ------------------------------------------------------------------ 141 | 142 | # Based on https://developer.nvidia.com/cuda-toolkit-archive 143 | # Based on https://developer.nvidia.com/rdp/cudnn-archive 144 | # Based on https://docs.nvidia.com/deeplearning/cudnn/install-guide/index.html#package-manager-ubuntu-install 145 | 146 | # Installing CUDA Toolkit 147 | 148 | RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin && \ 149 | mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600 && \ 150 | wget https://developer.download.nvidia.com/compute/cuda/12.1.0/local_installers/cuda-repo-ubuntu2004-12-1-local_12.1.0-530.30.02-1_amd64.deb && \ 151 | dpkg -i cuda-repo-ubuntu2004-12-1-local_12.1.0-530.30.02-1_amd64.deb && \ 152 | cp /var/cuda-repo-ubuntu2004-12-1-local/cuda-*-keyring.gpg /usr/share/keyrings/ && \ 153 | apt-get update && \ 154 | apt-get -y install cuda 155 | 156 | ENV PATH=$PATH:/usr/local/cuda-12.1/bin 157 | ENV LD_LIBRARY_PATH=/usr/local/cuda-12.1/lib64 158 | 159 | # Installing CUDNN 160 | RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin && \ 161 | mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600 && \ 162 | apt-get install dirmngr -y && \ 163 | apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub && \ 164 | add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /" && \ 165 | apt-get update && \ 166 | apt-get install libcudnn8=8.9.0.*-1+cuda12.1 -y && \ 167 | apt-get install libcudnn8-dev=8.9.0.*-1+cuda12.1 -y && \ 168 | rm /etc/apt/preferences.d/cuda-repository-pin-600 169 | 170 | 171 | # ================================================================== 172 | # PyTorch 173 | # ------------------------------------------------------------------ 174 | 175 | # Based on https://pytorch.org/get-started/locally/ 176 | 177 | RUN $PIP_INSTALL torch torchvision torchaudio torchtext && \ 178 | 179 | 180 | # ================================================================== 181 | # JAX 182 | # ------------------------------------------------------------------ 183 | 184 | # Based on https://github.com/google/jax#pip-installation-gpu-cuda 185 | 186 | $PIP_INSTALL "jax[cuda12_cudnn88]" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html && \ 187 | $PIP_INSTALL flax==0.6.3 && \ 188 | 189 | 190 | # ================================================================== 191 | # TensorFlow 192 | # ------------------------------------------------------------------ 193 | 194 | # Based on https://www.tensorflow.org/install/pip 195 | 196 | $PIP_INSTALL tensorflow && \ 197 | 198 | 199 | # ================================================================== 200 | # Hugging Face 201 | # ------------------------------------------------------------------ 202 | 203 | # Based on https://huggingface.co/docs/transformers/installation 204 | # Based on https://huggingface.co/docs/datasets/installation 205 | 206 | $PIP_INSTALL datasets && \ 207 | 208 | 209 | # ================================================================== 210 | # JupyterLab 211 | # ------------------------------------------------------------------ 212 | 213 | # Based on https://jupyterlab.readthedocs.io/en/stable/getting_started/installation.html#pip 214 | 215 | $PIP_INSTALL jupyterlab==3.4.6 && \ 216 | 217 | 218 | # ================================================================== 219 | # Additional Python Packages 220 | # ------------------------------------------------------------------ 221 | 222 | $PIP_INSTALL \ 223 | numpy==1.23.4 \ 224 | scipy==1.9.2 \ 225 | pandas==1.5.0 \ 226 | cloudpickle==2.2.0 \ 227 | scikit-image \ 228 | matplotlib==3.6.1 \ 229 | ipython==8.5.0 \ 230 | ipykernel==6.16.0 \ 231 | ipywidgets==8.0.2 \ 232 | cython==0.29.32 \ 233 | tqdm==4.64.1 \ 234 | gdown \ 235 | xgboost==1.6.2 \ 236 | pillow==9.2.0 \ 237 | seaborn==0.12.0 \ 238 | sqlalchemy==1.4.41 \ 239 | spacy==3.4.1 \ 240 | nltk==3.7 \ 241 | boto3==1.24.90 \ 242 | tabulate==0.9.0 \ 243 | future==0.18.2 \ 244 | gradient==2.0.6 \ 245 | jsonify==0.5 \ 246 | opencv-python==4.6.0.66 \ 247 | sentence-transformers==2.2.2 \ 248 | wandb==0.13.4 \ 249 | awscli==1.25.91 \ 250 | jupyterlab-snippets==0.4.1 \ 251 | tornado==6.1 252 | # ================================================================== 253 | # Installing transformers & scikit image (fixed) 254 | # ------------------------------------------------------------------ 255 | RUN pip install -U git+https://github.com/huggingface/transformers 256 | 257 | RUN pip install --pre scikit-learn 258 | # ================================================================== 259 | # Installing JRE and JDK 260 | # ------------------------------------------------------------------ 261 | 262 | RUN $APT_INSTALL \ 263 | default-jre \ 264 | default-jdk 265 | 266 | 267 | # ================================================================== 268 | # CMake 269 | # ------------------------------------------------------------------ 270 | 271 | RUN $GIT_CLONE https://github.com/Kitware/CMake ~/cmake && \ 272 | cd ~/cmake && \ 273 | ./bootstrap && \ 274 | make -j"$(nproc)" install 275 | 276 | 277 | # ================================================================== 278 | # Node.js and Jupyter Notebook Extensions 279 | # ------------------------------------------------------------------ 280 | 281 | RUN curl -sL https://deb.nodesource.com/setup_16.x | bash && \ 282 | $APT_INSTALL nodejs && \ 283 | $PIP_INSTALL jupyter_contrib_nbextensions jupyterlab-git && \ 284 | jupyter contrib nbextension install --user 285 | 286 | 287 | # ================================================================== 288 | # Startup 289 | # ------------------------------------------------------------------ 290 | 291 | EXPOSE 8888 6006 292 | 293 | CMD jupyter lab --allow-root --ip=0.0.0.0 --no-browser --ServerApp.trust_xheaders=True --ServerApp.disable_check_xsrf=False --ServerApp.allow_remote_access=True --ServerApp.allow_origin='*' --ServerApp.allow_credentials=True 294 | -------------------------------------------------------------------------------- /older-versions/all-pt200-tf212-jax048-py311/README.md: -------------------------------------------------------------------------------- 1 | # Gradient Base Image 2 | 3 | This repo houses a Dockerfile used to create an image used for Gradient runtimes. This image should be used for all Gradient runtimes, either solely or as the base layer for other images. 4 | 5 | The goal of this image is to provide common packages for an advanced data science user that allows them to utilize the GPU hardware on Gradient. This image targets popular Machine Learning frameworks and includes packages commonly used for Computer Vision and Natural Language Processing tasks as well as general Data Science work. 6 | 7 | 8 | ## Software included 9 | 10 | | Category | Software | Version | Install Method | Why / Notes | 11 | | ------------- | ------------- | ------------- | ------------- | ------------- | 12 | | GPU | NVidia Driver | 510.73.05 | pre-installed | Enable Nvidia GPUs | 13 | | | CUDA | 11.6.2 | Apt | Nvidia A100 GPUs require CUDA 11+ to work, so 10.x is not suitable | 14 | | | CUDA toolkit | 11.6.2 | Apt | Needed for `nvcc` command for cuDNN | 15 | | | cuDNN | 8.4.1.*-1+cuda11.6 | Apt | Nvidia GPU deep learning library | 16 | | Python | Python | 3.9.15 | Apt | Most widely used programming language for data science | 17 | | | pip3 | 22.2.2 | Apt | Enable easy installation of 1000s of other data science, etc., packages. | 18 | | | NumPy | 1.23.4 | pip3 | Handle arrays, matrices, etc., in Python | 19 | | | SciPy | 1.9.2 | pip3 | Fundamental algorithms for scientific computing in Python | 20 | | | Pandas | 1.5.0 | pip3 | De facto standard for data science data exploration/preparation in Python | 21 | | | Cloudpickle | 2.2.0 | pip3 | Makes it possible to serialize Python constructs not supported by the default pickle module | 22 | | | Matplotlib | 3.6.1 | pip3 | Widely used plotting library in Python for data science, e.g., scikit-learn plotting requires it | 23 | | | Ipython | 8.5.0 | pip3 | Provides a rich architecture for interactive computing | 24 | | | IPykernel | 6.16.0 | pip3 | Provides the IPython kernel for Jupyter. | 25 | | | IPywidgets | 8.0.2 | pip3 | Interactive HTML widgets for Jupyter notebooks and the IPython kernel | 26 | | | Cython | 0.29.32 | pip3 | Enables writing C extensions for Python | 27 | | | tqdm | 4.64.1 | pip3 | Fast, extensible progress meter | 28 | | | gdown | 4.5.1 | pip3 | Google drive direct download of big files | 29 | | | Pillow | 9.2.0 | pip3 | Python imaging library | 30 | | | seaborn | 0.12.0 | pip3 | Python visualization library based on matplotlib | 31 | | | SQLAlchemy | 1.4.41 | pip3 | Python SQL toolkit and Object Relational Mapper that gives application developers the full power and flexibility of SQL | 32 | | | spaCy | 3.4.1 | pip3 | library for advanced Natural Language Processing in Python and Cython | 33 | | | nltk | 3.7 | pip3 | Natural Language Toolkit (NLTK) is a Python package for natural language processing | 34 | | | boto3 | 1.24.90 | pip3 | Amazon Web Services (AWS) Software Development Kit (SDK) for Python | 35 | | | tabulate | 0.9.0 | pip3 | Pretty-print tabular data in Python | 36 | | | future | 0.18.2 | pip3 | The missing compatibility layer between Python 2 and Python 3 | 37 | | | gradient | 2.0.6 | pip3 | CLI and Python SDK for Paperspace Core and Gradient | 38 | | | jsonify | 0.5 | pip3 | Provides the ability to take a .csv file as input and outputs a file with the same data in .json format | 39 | | | opencv-python | 4.6.0.66 | pip3 | Includes several hundreds of computer vision algorithms | 40 | | | JupyterLab | 3.4.6 | pip3 | De facto standard for data science using Jupyter notebooks | 41 | | | wandb | 0.13.4 | pip3 | CLI and library to interact with the Weights & Biases API (model tracking) | 42 | | Machine Learning | Scikit-learn | 1.1.2 | pip3 | Widely used ML library for data science, generally for smaller data or models | 43 | | | Scikit-image | 0.19.3 | pip3 | Collection of algorithms for image processing | 44 | | | TensorFlow | 2.9.2 | pip3 | Most widely used deep learning library, alongside PyTorch | 45 | | | torch | 1.12.1 | pip3 | Most widely used deep learning library, alongside TensorFlow | 46 | | | torchvision | 0.13.1 | pip3 | Most widely used deep learning library, alongside TensorFlow | 47 | | | torchaudio | 0.12.1 | pip3 | Most widely used deep learning library, alongside TensorFlow | 48 | | | Jax | 0.3.23 | pip3 | Popular deep learning library brought to you by Google | 49 | | | Transformers | 4.21.3 | pip3 | Popular deep learning library for NLP brought to you by HuggingFace | 50 | | | Datasets | 2.4.0 | pip3 | A supporting library for NLP use cases and the Transformers library brought to you by HuggingFace | 51 | | | XGBoost | 1.6.2 | pip3 | An optimized distributed gradient boosting library | 52 | | | Sentence Transformers | 2.2.2 | pip3 | A ML framework for sentence, paragraph and image embeddings | 53 | 54 | ### Licenses 55 | 56 | | Software | License | Source | 57 | | --------------- | ------------- | ------------- | 58 | | CUDA | NVidia EULA | https://docs.nvidia.com/cuda/eula/index.html | 59 | | cuDNN | NVidia EULA | https://docs.nvidia.com/deeplearning/cudnn/sla/index.html | 60 | | JupyterLab | New BSD | https://github.com/jupyterlab/jupyterlab/blob/master/LICENSE | 61 | | Matplotlib | PSF-based | https://matplotlib.org/stable/users/license.html | 62 | | Numpy | New BSD | https://numpy.org/doc/stable/license.html | 63 | | NVidia Docker | Apache 2.0 | https://github.com/NVIDIA/nvidia-docker/blob/master/LICENSE | 64 | | NVidia Driver | NVidia EULA | https://www.nvidia.com/en-us/drivers/nvidia-license/ | 65 | | Pandas | New BSD | https://github.com/pandas-dev/pandas/blob/master/LICENSE | 66 | | Pip3 | MIT | https://github.com/pypa/pip/blob/main/LICENSE.txt | 67 | | Python | PSF | https://en.wikipedia.org/wiki/Python_(programming_language) | 68 | | Scikit-learn | New BSD | https://github.com/scikit-learn/scikit-learn/blob/main/COPYING | 69 | | Scikit-image | New BSD | https://github.com/scikit-image/scikit-image/blob/main/LICENSE.txt | 70 | | TensorFlow | Apache 2.0 | https://github.com/tensorflow/tensorflow/blob/master/LICENSE | 71 | | PyTorch | New BSD | https://github.com/pytorch/pytorch/blob/master/LICENSE | 72 | | Jax | Apache 2.0 | https://github.com/google/jax/blob/main/LICENSE | 73 | | Transformers | Apache 2.0 | https://github.com/huggingface/transformers/blob/main/LICENSE | 74 | | Datasets | Apache 2.0 | https://github.com/huggingface/datasets/blob/main/LICENSE | 75 | | XGBoost | Apache 2.0 | https://github.com/dmlc/xgboost/blob/master/LICENSE | 76 | | Sentence Transformers | Apache 2.0 | https://github.com/UKPLab/sentence-transformers/blob/master/LICENSE | 77 | | SciPy | New BSD | https://github.com/scipy/scipy/blob/main/LICENSE.txt | 78 | | Cloudpickle | New BSD | https://github.com/cloudpipe/cloudpickle/blob/master/LICENSE | 79 | | Ipython | New BSD | https://github.com/ipython/ipython/blob/main/LICENSE | 80 | | IPykernel | New BSD | https://github.com/ipython/ipykernel/blob/main/COPYING.md | 81 | | IPywidgets | New BSD | https://github.com/jupyter-widgets/ipywidgets/blob/master/LICENSE | 82 | | Cython | Apache 2.0 | https://github.com/cython/cython/blob/master/LICENSE.txt | 83 | | tqdm | MIT | https://github.com/tqdm/tqdm/blob/master/LICENCE | 84 | | gdown | MIT | https://github.com/wkentaro/gdown/blob/main/LICENSE | 85 | | Pillow | HPND | https://github.com/python-pillow/Pillow/blob/main/LICENSE | 86 | | seaborn | New BSD | https://github.com/mwaskom/seaborn/blob/master/LICENSE.md | 87 | | SQLAlchemy | MIT | https://github.com/sqlalchemy/sqlalchemy/blob/main/LICENSE | 88 | | spaCy | MIT | https://github.com/explosion/spaCy/blob/master/LICENSE | 89 | | nltk | Apache 2.0 | https://github.com/nltk/nltk/blob/develop/LICENSE.txt | 90 | | boto3 | Apache 2.0 | https://github.com/boto/boto3/blob/develop/LICENSE | 91 | | tabulate | MIT | https://github.com/astanin/python-tabulate/blob/master/LICENSE | 92 | | future | MIT | https://github.com/PythonCharmers/python-future/blob/master/LICENSE.txt | 93 | | gradient | ISC | https://github.com/Paperspace/gradient-cli/blob/master/LICENSE.txt | 94 | | jsonify | MIT | https://pypi.org/project/jsonify/0.5/#data | 95 | | opencv-python | MIT | https://github.com/opencv/opencv-python/blob/4.x/LICENSE.txt | 96 | | wandb | MIT | https://github.com/wandb/wandb/blob/master/LICENSE | 97 | 98 | 99 | Information about license types: 100 | 101 | Apache 2.0: https://opensource.org/licenses/Apache-2.0 102 | MIT: https://opensource.org/licenses/MIT 103 | New BSD: https://opensource.org/licenses/BSD-3-Clause 104 | PSF = Python Software Foundation: https://en.wikipedia.org/wiki/Python_Software_Foundation_License 105 | HPND = Historical Permission Notice and Disclaimer: https://opensource.org/licenses/HPND 106 | ISC: https://opensource.org/licenses/ISC 107 | 108 | Open source software can be used for commercial purposes: https://opensource.org/docs/osd#fields-of-endeavor. 109 | 110 | 111 | ## Software not included 112 | 113 | Other software considered but not included. 114 | 115 | The potential data science stack is far larger than any one person will use so we don't attempt to cover everything here. 116 | 117 | Some generic categories of software not included: 118 | 119 | - Non-data-science software 120 | - Commercial software 121 | - Software not licensed to be used on an available VM template 122 | - Software only used in particular specialized data science subfields (although we assume our users probably want a GPU) 123 | 124 | | Category | Software | Why Not | 125 | | ------------- | ------------- | ------------- | 126 | | Apache | Kafka, Parquet | | 127 | | Classifiers | libsvm | H2O contains SVM and GBM, save on installs | 128 | | Collections | ELKI, GNU Octave, Weka, Mahout | | 129 | | Connectors | Academic Torrents | | 130 | | Dashboarding | panel, dash, voila, streamlit | | 131 | | Databases | MySQL, Hive, PostgreSQL, Prometheus, Neo4j, MongoDB, Cassandra, Redis | No particular infra to connect to databases | 132 | | Deep Learning | Caffe, Caffe2, Theano, PaddlePaddle, Chainer, MXNet | PyTorch and TensorFlow are dominant, rest niche | 133 | | Deployment | Dash, TFServing, R Shiny, Flask | Use Gradient Deployments | 134 | | Distributed | Horovod, OpenMPI | Use Gradient distributed | 135 | | Feature store | Feast | | 136 | | Interpretability | LIME/SHAP, Fairlearn, AI Fairness 360, InterpretML | | 137 | | Languages | R, SQL, Julia, C++, JavaScript, Python2, Scala | Python is dominant for data science | 138 | | Monitoring | Grafana | | 139 | | NLP | GenSim | | 140 | | Notebooks | Jupyter, Zeppelin | JupyterLab includes Jupyter notebook | 141 | | Orchestrators | Kubernetes | Use Gradient cluster| 142 | | Partners | fast.ai | Currently support a specific fast.ai runtime | 143 | | Pipelines | AirFlow, MLFlow, Intake, Kubeflow | | 144 | | Python libraries | statsmodels, pymc3, geopandas, Geopy, LIBSVM | Too many to attempt to cover | 145 | | PyTorch extensions | Lightning | | 146 | | R packages | ggplot, tidyverse | Could add R if customer demand | 147 | | Recommenders | TFRS, scikit-surprise | | 148 | | Scalable | Dask, Numba, Spark 1 or 2, Koalas, Hadoop | | 149 | | TensorFlow | TF 1.15, Recommenders, TensorBoard, TensorRT | Could add TensorFlow 1.x if customer demand. Requires separate tensorflow-gpu for GPU support. | 150 | | Viz | Bokeh, Plotly, Holoviz (Datashader), Google FACETS, Excalidraw, GraphViz, ggplot2, d3.js | | -------------------------------------------------------------------------------- /older-versions/all-tf24-py36/Dockerfile: -------------------------------------------------------------------------------- 1 | # ================================================================== 2 | # module list 3 | # ------------------------------------------------------------------ 4 | # python 3.6 (apt) 5 | # jupyter latest (pip) 6 | # pytorch latest (pip) 7 | # tensorflow latest (pip) 8 | # jupyterlab latest (pip) 9 | # keras latest (pip) 10 | # opencv 4.5.1 (git) 11 | # caffe latest (git) 12 | # ================================================================== 13 | 14 | FROM nvidia/cuda:11.0-cudnn8-devel-ubuntu18.04 15 | ENV LANG C.UTF-8 16 | RUN APT_INSTALL="apt-get install -y --no-install-recommends" && \ 17 | PIP_INSTALL="python -m pip --no-cache-dir install --upgrade" && \ 18 | GIT_CLONE="git clone --depth 10" && \ 19 | 20 | rm -rf /var/lib/apt/lists/* \ 21 | /etc/apt/sources.list.d/cuda.list \ 22 | /etc/apt/sources.list.d/nvidia-ml.list && \ 23 | 24 | apt-get update && \ 25 | 26 | # ================================================================== 27 | # tools 28 | # ------------------------------------------------------------------ 29 | 30 | DEBIAN_FRONTEND=noninteractive $APT_INSTALL \ 31 | build-essential \ 32 | apt-utils \ 33 | ca-certificates \ 34 | wget \ 35 | rsync \ 36 | git \ 37 | vim \ 38 | libssl-dev \ 39 | curl \ 40 | openssh-client \ 41 | unzip \ 42 | unrar \ 43 | zip \ 44 | && \ 45 | 46 | $GIT_CLONE https://github.com/Kitware/CMake ~/cmake && \ 47 | cd ~/cmake && \ 48 | ./bootstrap && \ 49 | make -j"$(nproc)" install && \ 50 | 51 | # ================================================================== 52 | # python 53 | # ------------------------------------------------------------------ 54 | 55 | DEBIAN_FRONTEND=noninteractive $APT_INSTALL \ 56 | software-properties-common \ 57 | && \ 58 | add-apt-repository ppa:deadsnakes/ppa && \ 59 | apt-get update && \ 60 | DEBIAN_FRONTEND=noninteractive $APT_INSTALL \ 61 | python3.6 \ 62 | python3.6-dev \ 63 | python3-distutils-extra \ 64 | && \ 65 | wget -O ~/get-pip.py \ 66 | https://bootstrap.pypa.io/get-pip.py && \ 67 | python3.6 ~/get-pip.py && \ 68 | ln -s /usr/bin/python3.6 /usr/local/bin/python3 && \ 69 | ln -s /usr/bin/python3.6 /usr/local/bin/python && \ 70 | $PIP_INSTALL \ 71 | setuptools \ 72 | && \ 73 | $PIP_INSTALL \ 74 | numpy \ 75 | scipy \ 76 | pandas \ 77 | cloudpickle \ 78 | scikit-image>=0.14.2 \ 79 | scikit-learn \ 80 | matplotlib \ 81 | ipython \ 82 | ipykernel \ 83 | gradient-sdk \ 84 | Cython \ 85 | tqdm \ 86 | && \ 87 | 88 | # ================================================================== 89 | # boost 90 | # ------------------------------------------------------------------ 91 | 92 | DEBIAN_FRONTEND=noninteractive $APT_INSTALL \ 93 | libboost-all-dev \ 94 | && \ 95 | 96 | # ================================================================== 97 | # jupyter 98 | # ------------------------------------------------------------------ 99 | 100 | $PIP_INSTALL \ 101 | jupyter \ 102 | && \ 103 | 104 | # ================================================================== 105 | # pytorch 106 | # ------------------------------------------------------------------ 107 | 108 | $PIP_INSTALL \ 109 | future \ 110 | numpy \ 111 | protobuf \ 112 | enum34 \ 113 | pyyaml \ 114 | typing \ 115 | && \ 116 | $PIP_INSTALL \ 117 | --pre torch torchvision -f \ 118 | https://download.pytorch.org/whl/nightly/cu110/torch_nightly.html \ 119 | && \ 120 | 121 | # ================================================================== 122 | # tensorflow 123 | # ------------------------------------------------------------------ 124 | 125 | $PIP_INSTALL \ 126 | tensorflow-gpu \ 127 | && \ 128 | 129 | # ================================================================== 130 | # jupyterlab 131 | # ------------------------------------------------------------------ 132 | 133 | $PIP_INSTALL \ 134 | jupyterlab \ 135 | && \ 136 | 137 | # ================================================================== 138 | # keras 139 | # ------------------------------------------------------------------ 140 | 141 | $PIP_INSTALL \ 142 | h5py \ 143 | keras \ 144 | && \ 145 | 146 | # ================================================================== 147 | # opencv 148 | # ------------------------------------------------------------------ 149 | 150 | DEBIAN_FRONTEND=noninteractive $APT_INSTALL \ 151 | libatlas-base-dev \ 152 | libgflags-dev \ 153 | libgoogle-glog-dev \ 154 | libhdf5-serial-dev \ 155 | libleveldb-dev \ 156 | liblmdb-dev \ 157 | libprotobuf-dev \ 158 | libsnappy-dev \ 159 | protobuf-compiler \ 160 | && \ 161 | 162 | $GIT_CLONE --branch 4.5.1 https://github.com/opencv/opencv ~/opencv && \ 163 | mkdir -p ~/opencv/build && cd ~/opencv/build && \ 164 | cmake -D CMAKE_BUILD_TYPE=RELEASE \ 165 | -D CMAKE_INSTALL_PREFIX=/usr/local \ 166 | -D WITH_IPP=OFF \ 167 | -D WITH_CUDA=OFF \ 168 | -D WITH_OPENCL=OFF \ 169 | -D BUILD_TESTS=OFF \ 170 | -D BUILD_PERF_TESTS=OFF \ 171 | -D BUILD_DOCS=OFF \ 172 | -D BUILD_EXAMPLES=OFF \ 173 | .. && \ 174 | make -j"$(nproc)" install && \ 175 | ln -s /usr/local/include/opencv4/opencv2 /usr/local/include/opencv2 && \ 176 | 177 | # ================================================================== 178 | # caffe 179 | # ------------------------------------------------------------------ 180 | 181 | apt-get update && \ 182 | DEBIAN_FRONTEND=noninteractive $APT_INSTALL \ 183 | caffe-cuda \ 184 | && \ 185 | # ================================================================== 186 | # config & cleanup 187 | # ------------------------------------------------------------------ 188 | 189 | ldconfig && \ 190 | apt-get clean && \ 191 | apt-get autoremove && \ 192 | rm -rf /var/lib/apt/lists/* /tmp/* ~/* 193 | 194 | EXPOSE 8888 6006 195 | -------------------------------------------------------------------------------- /older-versions/all-tf29-pt112-py39/README.md: -------------------------------------------------------------------------------- 1 | Dockerfile for new base image, updated from Tom G's https://github.com/gradient-ai/base-container/tree/main/all-tf24-py36 by Nick and Josh. 2 | 3 | See https://www.notion.so/paperspace/Container-Runtime-Content-01740f6142c9405b99e980006f7604a7 for more info, in particular, https://www.notion.so/paperspace/New-Dockerfile-for-blank-container-5baa2abb726a44889763274f776e3b81 . 4 | -------------------------------------------------------------------------------- /older-versions/all-tf29-pt112-py39/dockerfile: -------------------------------------------------------------------------------- 1 | # Paperspace Dockerfile for Gradient base image 2 | # Paperspace image is located in Dockerhub registry: paperspace/gradient_base 3 | 4 | # ================================================================== 5 | # module list 6 | # ------------------------------------------------------------------ 7 | # python 3.9.13 (apt) 8 | # jupyter latest (pip) 9 | # pytorch latest (pip) 10 | # tensorflow latest (pip) 11 | # jupyterlab latest (pip) 12 | # keras latest (pip) # Comes installed with Tensorflow 13 | # opencv 4.5.1 (git) 14 | # numpy latest (pip) 15 | # scipy latest (pip) 16 | # pandas latest (pip) 17 | # cloudpickle latest (pip) 18 | # scikit-image latest (pip) 19 | # scikit-learn latest (pip) 20 | # matplotlib latest (pip) 21 | # ipython latest (pip) 22 | # ipykernel latest (pip) 23 | # ipywidgets latest (pip) 24 | # gradient latest (pip) 25 | # Cython latest (pip) 26 | # tqdm latest (pip) 27 | # gdown latest (pip) 28 | # xgboost latest (pip) 29 | # pillow latest (pip) 30 | # seaborn latest (pip) 31 | # SQLAlchemy latest (pip) 32 | # spacy latest (pip) 33 | # nltk latest (pip) 34 | # jsonify latest (pip) 35 | # boto3 latest (pip) 36 | # transformers latest (pip) 37 | # sentence-transformers latest (pip) 38 | # opencv-python latest (pip) 39 | # ================================================================== 40 | 41 | # Ubuntu 20.04, CUDA Toolkit 11.2, CUDNN 8 42 | FROM nvidia/cuda:11.2.1-cudnn8-runtime-ubuntu20.04 43 | ENV LANG C.UTF-8 44 | 45 | # Setting shell to bash 46 | ENV SHELL=/bin/bash 47 | SHELL ["/bin/bash", "-c"] 48 | 49 | RUN APT_INSTALL="apt-get install -y --no-install-recommends" && \ 50 | PIP_INSTALL="python3 -m pip --no-cache-dir install --upgrade" && \ 51 | GIT_CLONE="git clone --depth 10" && \ 52 | 53 | rm -rf /var/lib/apt/lists/* \ 54 | /etc/apt/sources.list.d/cuda.list \ 55 | /etc/apt/sources.list.d/nvidia-ml.list && \ 56 | 57 | apt-get update && \ 58 | 59 | # ================================================================== 60 | # tools 61 | # ------------------------------------------------------------------ 62 | 63 | sed -e '\|/usr/share/man|s|^#*|#|g' -i /etc/dpkg/dpkg.cfg.d/excludes && \ 64 | 65 | DEBIAN_FRONTEND=noninteractive $APT_INSTALL \ 66 | build-essential \ 67 | apt-utils \ 68 | ca-certificates \ 69 | wget \ 70 | rsync \ 71 | git \ 72 | vim \ 73 | libssl-dev \ 74 | curl \ 75 | openssh-client \ 76 | unzip \ 77 | unrar \ 78 | zip \ 79 | awscli \ 80 | csvkit \ 81 | emacs \ 82 | joe \ 83 | jq \ 84 | dialog \ 85 | man-db \ 86 | manpages \ 87 | manpages-dev \ 88 | manpages-posix \ 89 | manpages-posix-dev \ 90 | nano \ 91 | iputils-ping \ 92 | sudo \ 93 | ffmpeg \ 94 | libsm6 \ 95 | libxext6 \ 96 | && \ 97 | 98 | rm -f /usr/bin/man && \ 99 | dpkg-divert --quiet --remove --rename /usr/bin/man && \ 100 | rm -f /usr/share/man/man1/sh.1.gz && \ 101 | dpkg-divert --quiet --remove --rename /usr/share/man/man1/sh.1.gz && \ 102 | 103 | $GIT_CLONE https://github.com/Kitware/CMake ~/cmake && \ 104 | cd ~/cmake && \ 105 | ./bootstrap && \ 106 | make -j"$(nproc)" install && \ 107 | 108 | 109 | # ================================================================== 110 | # Python 111 | # ------------------------------------------------------------------ 112 | 113 | # Installing python3.9 114 | DEBIAN_FRONTEND=noninteractive \ 115 | $APT_INSTALL software-properties-common && \ 116 | add-apt-repository ppa:deadsnakes/ppa -y && \ 117 | 118 | DEBIAN_FRONTEND=noninteractive $APT_INSTALL \ 119 | python3.9 \ 120 | python3.9-dev \ 121 | python3-distutils-extra \ 122 | && \ 123 | 124 | # Installing pip 125 | wget -O ~/get-pip.py \ 126 | https://bootstrap.pypa.io/get-pip.py && \ 127 | python3.9 ~/get-pip.py && \ 128 | 129 | # Add symlink so python and python3 commands use same 130 | # python3.9 executable 131 | ln -s /usr/bin/python3.9 /usr/local/bin/python3 && \ 132 | ln -s /usr/bin/python3.9 /usr/local/bin/python && \ 133 | 134 | # Intalling Python packages 135 | $PIP_INSTALL \ 136 | numpy \ 137 | scipy \ 138 | pandas \ 139 | cloudpickle \ 140 | scikit-image \ 141 | scikit-learn \ 142 | matplotlib \ 143 | ipython \ 144 | ipykernel \ 145 | ipywidgets \ 146 | gradient \ 147 | Cython \ 148 | tqdm \ 149 | gdown \ 150 | xgboost \ 151 | pillow \ 152 | seaborn \ 153 | SQLAlchemy \ 154 | spacy \ 155 | nltk \ 156 | jsonify \ 157 | boto3 \ 158 | transformers \ 159 | sentence-transformers \ 160 | datasets \ 161 | opencv-python \ 162 | && \ 163 | 164 | 165 | # ================================================================== 166 | # boost 167 | # ------------------------------------------------------------------ 168 | 169 | DEBIAN_FRONTEND=noninteractive $APT_INSTALL \ 170 | libboost-all-dev \ 171 | && \ 172 | 173 | 174 | # ================================================================== 175 | # jupyter 176 | # ------------------------------------------------------------------ 177 | 178 | $PIP_INSTALL \ 179 | jupyter \ 180 | && \ 181 | 182 | 183 | # ================================================================== 184 | # PyTorch 185 | # ------------------------------------------------------------------ 186 | 187 | $PIP_INSTALL torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu116 && \ 188 | 189 | 190 | # ================================================================== 191 | # TensorFlow 192 | # ------------------------------------------------------------------ 193 | 194 | # Based on https://www.tensorflow.org/install and 195 | # https://www.tensorflow.org/install/pip, so is now not -gpu 196 | 197 | $PIP_INSTALL \ 198 | tensorflow \ 199 | && \ 200 | 201 | 202 | # ================================================================== 203 | # JupyterLab 204 | # ------------------------------------------------------------------ 205 | 206 | $PIP_INSTALL \ 207 | jupyterlab \ 208 | && \ 209 | 210 | 211 | # ================================================================== 212 | # Node.js and Jupyter Notebook Extensions 213 | # ------------------------------------------------------------------ 214 | 215 | curl -sL https://deb.nodesource.com/setup_16.x | bash && \ 216 | $APT_INSTALL nodejs && \ 217 | $PIP_INSTALL jupyter_contrib_nbextensions jupyterlab-git && \ 218 | jupyter contrib nbextension install --sys-prefix && \ 219 | 220 | 221 | # ================================================================== 222 | # Conda 223 | # ------------------------------------------------------------------ 224 | 225 | # wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh && \ 226 | # /bin/bash ~/miniconda.sh -b -p /opt/conda && \ 227 | 228 | 229 | # ================================================================== 230 | # Config & Cleanup 231 | # ------------------------------------------------------------------ 232 | 233 | ldconfig && \ 234 | apt-get clean && \ 235 | apt-get autoremove && \ 236 | rm -rf /var/lib/apt/lists/* /tmp/* ~/* 237 | 238 | 239 | EXPOSE 8888 6006 240 | -------------------------------------------------------------------------------- /older-versions/lean-tf115-py36/Dockerfile: -------------------------------------------------------------------------------- 1 | # ================================================================== 2 | # module list 3 | # ------------------------------------------------------------------ 4 | # python 3.6 (apt) 5 | # jupyter latest (pip) 6 | # pytorch latest (pip) 7 | # tensorflow 1.15 (pip) 8 | # jupyterlab latest (pip) 9 | # keras 2.3.1 (pip) 10 | # ================================================================== 11 | 12 | FROM nvidia/cuda:10.0-cudnn7-runtime-ubuntu18.04 13 | ENV LANG C.UTF-8 14 | RUN APT_INSTALL="apt-get install -y --no-install-recommends" && \ 15 | PIP_INSTALL="python -m pip --no-cache-dir install --upgrade" && \ 16 | GIT_CLONE="git clone --depth 10" && \ 17 | 18 | rm -rf /var/lib/apt/lists/* \ 19 | /etc/apt/sources.list.d/cuda.list \ 20 | /etc/apt/sources.list.d/nvidia-ml.list && \ 21 | 22 | apt-get update && \ 23 | 24 | # ================================================================== 25 | # tools 26 | # ------------------------------------------------------------------ 27 | 28 | DEBIAN_FRONTEND=noninteractive $APT_INSTALL \ 29 | build-essential \ 30 | apt-utils \ 31 | ca-certificates \ 32 | wget \ 33 | rsync \ 34 | git \ 35 | vim \ 36 | libssl-dev \ 37 | curl \ 38 | openssh-client \ 39 | unzip \ 40 | unrar \ 41 | zip \ 42 | && \ 43 | 44 | $GIT_CLONE https://github.com/Kitware/CMake ~/cmake && \ 45 | cd ~/cmake && \ 46 | ./bootstrap && \ 47 | make -j"$(nproc)" install && \ 48 | 49 | # ================================================================== 50 | # python 51 | # ------------------------------------------------------------------ 52 | 53 | DEBIAN_FRONTEND=noninteractive $APT_INSTALL \ 54 | software-properties-common \ 55 | && \ 56 | add-apt-repository ppa:deadsnakes/ppa && \ 57 | apt-get update && \ 58 | DEBIAN_FRONTEND=noninteractive $APT_INSTALL \ 59 | python3.6 \ 60 | python3.6-dev \ 61 | python3-distutils-extra \ 62 | && \ 63 | wget -O ~/get-pip.py \ 64 | https://bootstrap.pypa.io/get-pip.py && \ 65 | python3.6 ~/get-pip.py && \ 66 | ln -s /usr/bin/python3.6 /usr/local/bin/python3 && \ 67 | ln -s /usr/bin/python3.6 /usr/local/bin/python && \ 68 | $PIP_INSTALL \ 69 | setuptools \ 70 | && \ 71 | $PIP_INSTALL \ 72 | numpy \ 73 | scipy \ 74 | pandas \ 75 | cloudpickle \ 76 | scikit-image>=0.14.2 \ 77 | scikit-learn \ 78 | matplotlib \ 79 | ipython \ 80 | ipykernel \ 81 | gradient-sdk \ 82 | Cython \ 83 | tqdm \ 84 | && \ 85 | 86 | # ================================================================== 87 | # jupyter 88 | # ------------------------------------------------------------------ 89 | 90 | $PIP_INSTALL \ 91 | jupyter \ 92 | && \ 93 | 94 | # ================================================================== 95 | # pytorch 96 | # ------------------------------------------------------------------ 97 | 98 | $PIP_INSTALL \ 99 | future \ 100 | numpy \ 101 | protobuf \ 102 | enum34 \ 103 | pyyaml \ 104 | typing \ 105 | && \ 106 | $PIP_INSTALL \ 107 | --pre torch torchvision -f \ 108 | https://download.pytorch.org/whl/nightly/cu100/torch_nightly.html \ 109 | && \ 110 | 111 | # ================================================================== 112 | # tensorflow 113 | # ------------------------------------------------------------------ 114 | 115 | $PIP_INSTALL \ 116 | tensorflow-gpu==1.15 \ 117 | && \ 118 | 119 | # ================================================================== 120 | # jupyterlab 121 | # ------------------------------------------------------------------ 122 | 123 | $PIP_INSTALL \ 124 | jupyterlab \ 125 | && \ 126 | 127 | # ================================================================== 128 | # keras 129 | # ------------------------------------------------------------------ 130 | 131 | $PIP_INSTALL \ 132 | h5py==2.10.0 \ 133 | keras==2.3.1 \ 134 | && \ 135 | 136 | # ================================================================== 137 | # config & cleanup 138 | # ------------------------------------------------------------------ 139 | 140 | ldconfig && \ 141 | apt-get clean && \ 142 | apt-get autoremove && \ 143 | rm -rf /var/lib/apt/lists/* /tmp/* ~/* 144 | 145 | EXPOSE 8888 6006 146 | -------------------------------------------------------------------------------- /older-versions/lean-tf24-py36/Dockerfile: -------------------------------------------------------------------------------- 1 | # ================================================================== 2 | # module list 3 | # ------------------------------------------------------------------ 4 | # python 3.6 (apt) 5 | # jupyter latest (pip) 6 | # pytorch latest (pip) 7 | # tensorflow latest (pip) 8 | # jupyterlab latest (pip) 9 | # keras latest (pip) 10 | # ================================================================== 11 | 12 | FROM nvidia/cuda:11.0-cudnn8-runtime-ubuntu18.04 13 | ENV LANG C.UTF-8 14 | RUN APT_INSTALL="apt-get install -y --no-install-recommends" && \ 15 | PIP_INSTALL="python -m pip --no-cache-dir install --upgrade" && \ 16 | GIT_CLONE="git clone --depth 10" && \ 17 | 18 | rm -rf /var/lib/apt/lists/* \ 19 | /etc/apt/sources.list.d/cuda.list \ 20 | /etc/apt/sources.list.d/nvidia-ml.list && \ 21 | 22 | apt-get update && \ 23 | 24 | # ================================================================== 25 | # tools 26 | # ------------------------------------------------------------------ 27 | 28 | DEBIAN_FRONTEND=noninteractive $APT_INSTALL \ 29 | build-essential \ 30 | apt-utils \ 31 | ca-certificates \ 32 | wget \ 33 | rsync \ 34 | git \ 35 | vim \ 36 | libssl-dev \ 37 | curl \ 38 | openssh-client \ 39 | unzip \ 40 | unrar \ 41 | zip \ 42 | && \ 43 | 44 | $GIT_CLONE https://github.com/Kitware/CMake ~/cmake && \ 45 | cd ~/cmake && \ 46 | ./bootstrap && \ 47 | make -j"$(nproc)" install && \ 48 | 49 | # ================================================================== 50 | # python 51 | # ------------------------------------------------------------------ 52 | 53 | DEBIAN_FRONTEND=noninteractive $APT_INSTALL \ 54 | software-properties-common \ 55 | && \ 56 | add-apt-repository ppa:deadsnakes/ppa && \ 57 | apt-get update && \ 58 | DEBIAN_FRONTEND=noninteractive $APT_INSTALL \ 59 | python3.6 \ 60 | python3.6-dev \ 61 | python3-distutils-extra \ 62 | && \ 63 | wget -O ~/get-pip.py \ 64 | https://bootstrap.pypa.io/get-pip.py && \ 65 | python3.6 ~/get-pip.py && \ 66 | ln -s /usr/bin/python3.6 /usr/local/bin/python3 && \ 67 | ln -s /usr/bin/python3.6 /usr/local/bin/python && \ 68 | $PIP_INSTALL \ 69 | setuptools \ 70 | && \ 71 | $PIP_INSTALL \ 72 | numpy \ 73 | scipy \ 74 | pandas \ 75 | cloudpickle \ 76 | scikit-image>=0.14.2 \ 77 | scikit-learn \ 78 | matplotlib \ 79 | ipython \ 80 | ipykernel \ 81 | gradient-sdk \ 82 | Cython \ 83 | tqdm \ 84 | && \ 85 | 86 | # ================================================================== 87 | # jupyter 88 | # ------------------------------------------------------------------ 89 | 90 | $PIP_INSTALL \ 91 | jupyter \ 92 | && \ 93 | 94 | # ================================================================== 95 | # pytorch 96 | # ------------------------------------------------------------------ 97 | 98 | $PIP_INSTALL \ 99 | future \ 100 | numpy \ 101 | protobuf \ 102 | enum34 \ 103 | pyyaml \ 104 | typing \ 105 | && \ 106 | $PIP_INSTALL \ 107 | --pre torch torchvision -f \ 108 | https://download.pytorch.org/whl/nightly/cu110/torch_nightly.html \ 109 | && \ 110 | 111 | # ================================================================== 112 | # tensorflow 113 | # ------------------------------------------------------------------ 114 | 115 | $PIP_INSTALL \ 116 | tensorflow-gpu \ 117 | && \ 118 | 119 | # ================================================================== 120 | # jupyterlab 121 | # ------------------------------------------------------------------ 122 | 123 | $PIP_INSTALL \ 124 | jupyterlab \ 125 | && \ 126 | 127 | # ================================================================== 128 | # keras 129 | # ------------------------------------------------------------------ 130 | 131 | $PIP_INSTALL \ 132 | h5py \ 133 | keras \ 134 | && \ 135 | 136 | # ================================================================== 137 | # config & cleanup 138 | # ------------------------------------------------------------------ 139 | 140 | ldconfig && \ 141 | apt-get clean && \ 142 | apt-get autoremove && \ 143 | rm -rf /var/lib/apt/lists/* /tmp/* ~/* 144 | 145 | EXPOSE 8888 6006 146 | -------------------------------------------------------------------------------- /older-versions/pt112-tf29-jax0314-py39-updated/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM paperspace/gradient-base:pt112-tf29-jax0314-py39 2 | 3 | RUN pip install --upgrade pip \ 4 | pip install pyyaml==5.4.1 --ignore-installed -------------------------------------------------------------------------------- /older-versions/pt112-tf29-jax0314-py39/Dockerfile: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Paperspace Dockerfile for Gradient base image 4 | # Paperspace image is located in Dockerhub registry: paperspace/gradient_base 5 | 6 | # ================================================================== 7 | # Module list 8 | # ------------------------------------------------------------------ 9 | # python 3.9.15 (apt) 10 | # torch 1.12.1 (pip) 11 | # torchvision 0.13.1 (pip) 12 | # torchaudio 0.12.1 (pip) 13 | # tensorflow 2.9.2 (pip) 14 | # jax 0.3.23 (pip) 15 | # transformers 4.21.3 (pip) 16 | # datasets 2.4.0 (pip) 17 | # jupyterlab 3.4.6 (pip) 18 | # numpy 1.23.4 (pip) 19 | # scipy 1.9.2 (pip) 20 | # pandas 1.5.0 (pip) 21 | # cloudpickle 2.2.0 (pip) 22 | # scikit-image 0.19.3 (pip) 23 | # scikit-learn 1.1.2 (pip) 24 | # matplotlib 3.6.1 (pip) 25 | # ipython 8.5.0 (pip) 26 | # ipykernel 6.16.0 (pip) 27 | # ipywidgets 8.0.2 (pip) 28 | # cython 0.29.32 (pip) 29 | # tqdm 4.64.1 (pip) 30 | # gdown 4.5.1 (pip) 31 | # xgboost 1.6.2 (pip) 32 | # pillow 9.2.0 (pip) 33 | # seaborn 0.12.0 (pip) 34 | # sqlalchemy 1.4.41 (pip) 35 | # spacy 3.4.1 (pip) 36 | # nltk 3.7 (pip) 37 | # boto3 1.24.90 (pip) 38 | # tabulate 0.9.0 (pip) 39 | # future 0.18.2 (pip) 40 | # gradient 2.0.6 (pip) 41 | # jsonify 0.5 (pip) 42 | # opencv-python 4.6.0.66 (pip) 43 | # sentence-transformers 2.2.2 (pip) 44 | # wandb 0.13.4 (pip) 45 | # nodejs 16.x latest (apt) 46 | # default-jre latest (apt) 47 | # default-jdk latest (apt) 48 | 49 | 50 | # ================================================================== 51 | # Initial setup 52 | # ------------------------------------------------------------------ 53 | 54 | # Ubuntu 20.04 as base image 55 | FROM ubuntu:20.04 56 | RUN yes| unminimize 57 | 58 | # Set ENV variables 59 | ENV LANG C.UTF-8 60 | ENV SHELL=/bin/bash 61 | ENV DEBIAN_FRONTEND=noninteractive 62 | 63 | ENV APT_INSTALL="apt-get install -y --no-install-recommends" 64 | ENV PIP_INSTALL="python3 -m pip --no-cache-dir install --upgrade" 65 | ENV GIT_CLONE="git clone --depth 10" 66 | 67 | 68 | # ================================================================== 69 | # Tools 70 | # ------------------------------------------------------------------ 71 | 72 | RUN apt-get update && \ 73 | $APT_INSTALL \ 74 | apt-utils \ 75 | gcc \ 76 | make \ 77 | pkg-config \ 78 | apt-transport-https \ 79 | build-essential \ 80 | ca-certificates \ 81 | wget \ 82 | rsync \ 83 | git \ 84 | vim \ 85 | mlocate \ 86 | libssl-dev \ 87 | curl \ 88 | openssh-client \ 89 | unzip \ 90 | unrar \ 91 | zip \ 92 | csvkit \ 93 | emacs \ 94 | joe \ 95 | jq \ 96 | dialog \ 97 | man-db \ 98 | manpages \ 99 | manpages-dev \ 100 | manpages-posix \ 101 | manpages-posix-dev \ 102 | nano \ 103 | iputils-ping \ 104 | sudo \ 105 | ffmpeg \ 106 | libsm6 \ 107 | libxext6 \ 108 | libboost-all-dev \ 109 | cifs-utils \ 110 | software-properties-common 111 | 112 | 113 | # ================================================================== 114 | # Python 115 | # ------------------------------------------------------------------ 116 | 117 | #Based on https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa 118 | 119 | # Adding repository for python3.9 120 | RUN add-apt-repository ppa:deadsnakes/ppa -y && \ 121 | 122 | # Installing python3.9 123 | $APT_INSTALL \ 124 | python3.9 \ 125 | python3.9-dev \ 126 | python3.9-venv \ 127 | python3-distutils-extra 128 | 129 | # Add symlink so python and python3 commands use same python3.9 executable 130 | RUN ln -s /usr/bin/python3.9 /usr/local/bin/python3 && \ 131 | ln -s /usr/bin/python3.9 /usr/local/bin/python 132 | 133 | # Installing pip 134 | RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.9 135 | ENV PATH=$PATH:/root/.local/bin 136 | 137 | 138 | # ================================================================== 139 | # Installing CUDA packages (CUDA Toolkit 11.6.2 & CUDNN 8.4.1) 140 | # ------------------------------------------------------------------ 141 | 142 | # Based on https://developer.nvidia.com/cuda-toolkit-archive 143 | # Based on https://developer.nvidia.com/rdp/cudnn-archive 144 | # Based on https://docs.nvidia.com/deeplearning/cudnn/install-guide/index.html#package-manager-ubuntu-install 145 | 146 | # Installing CUDA Toolkit 147 | RUN wget https://developer.download.nvidia.com/compute/cuda/11.6.2/local_installers/cuda_11.6.2_510.47.03_linux.run && \ 148 | bash cuda_11.6.2_510.47.03_linux.run --silent --toolkit && \ 149 | rm cuda_11.6.2_510.47.03_linux.run 150 | ENV PATH=$PATH:/usr/local/cuda-11.6/bin 151 | ENV LD_LIBRARY_PATH=/usr/local/cuda-11.6/lib64 152 | 153 | # Installing CUDNN 154 | RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin && \ 155 | mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600 && \ 156 | apt-get install dirmngr -y && \ 157 | apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub && \ 158 | add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /" && \ 159 | apt-get update && \ 160 | apt-get install libcudnn8=8.4.1.*-1+cuda11.6 -y && \ 161 | apt-get install libcudnn8-dev=8.4.1.*-1+cuda11.6 -y && \ 162 | rm /etc/apt/preferences.d/cuda-repository-pin-600 163 | 164 | 165 | # ================================================================== 166 | # PyTorch 167 | # ------------------------------------------------------------------ 168 | 169 | # Based on https://pytorch.org/get-started/locally/ 170 | 171 | RUN $PIP_INSTALL torch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu116 && \ 172 | 173 | 174 | # ================================================================== 175 | # JAX 176 | # ------------------------------------------------------------------ 177 | 178 | # Based on https://github.com/google/jax#pip-installation-gpu-cuda 179 | 180 | $PIP_INSTALL "jax[cuda11_cudnn82]" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html && \ 181 | $PIP_INSTALL flax==0.6.3 && \ 182 | 183 | 184 | # ================================================================== 185 | # TensorFlow 186 | # ------------------------------------------------------------------ 187 | 188 | # Based on https://www.tensorflow.org/install/pip 189 | 190 | $PIP_INSTALL tensorflow==2.9.2 && \ 191 | 192 | 193 | # ================================================================== 194 | # Hugging Face 195 | # ------------------------------------------------------------------ 196 | 197 | # Based on https://huggingface.co/docs/transformers/installation 198 | # Based on https://huggingface.co/docs/datasets/installation 199 | 200 | $PIP_INSTALL transformers==4.21.3 datasets==2.4.0 && \ 201 | 202 | 203 | # ================================================================== 204 | # JupyterLab 205 | # ------------------------------------------------------------------ 206 | 207 | # Based on https://jupyterlab.readthedocs.io/en/stable/getting_started/installation.html#pip 208 | 209 | $PIP_INSTALL jupyterlab==3.4.6 && \ 210 | 211 | 212 | # ================================================================== 213 | # Additional Python Packages 214 | # ------------------------------------------------------------------ 215 | 216 | $PIP_INSTALL \ 217 | numpy==1.23.4 \ 218 | scipy==1.9.2 \ 219 | pandas==1.5.0 \ 220 | cloudpickle==2.2.0 \ 221 | scikit-image==0.19.3 \ 222 | scikit-learn==1.1.2 \ 223 | matplotlib==3.6.1 \ 224 | ipython==8.5.0 \ 225 | ipykernel==6.16.0 \ 226 | ipywidgets==8.0.2 \ 227 | cython==0.29.32 \ 228 | tqdm==4.64.1 \ 229 | gdown==4.5.1 \ 230 | xgboost==1.6.2 \ 231 | pillow==9.2.0 \ 232 | seaborn==0.12.0 \ 233 | sqlalchemy==1.4.41 \ 234 | spacy==3.4.1 \ 235 | nltk==3.7 \ 236 | boto3==1.24.90 \ 237 | tabulate==0.9.0 \ 238 | future==0.18.2 \ 239 | gradient==2.0.6 \ 240 | jsonify==0.5 \ 241 | opencv-python==4.6.0.66 \ 242 | sentence-transformers==2.2.2 \ 243 | wandb==0.13.4 \ 244 | awscli==1.25.91 \ 245 | jupyterlab-snippets==0.4.1 \ 246 | tornado==6.1 247 | 248 | 249 | # ================================================================== 250 | # Installing JRE and JDK 251 | # ------------------------------------------------------------------ 252 | 253 | RUN $APT_INSTALL \ 254 | default-jre \ 255 | default-jdk 256 | 257 | 258 | # ================================================================== 259 | # CMake 260 | # ------------------------------------------------------------------ 261 | 262 | RUN $GIT_CLONE https://github.com/Kitware/CMake ~/cmake && \ 263 | cd ~/cmake && \ 264 | ./bootstrap && \ 265 | make -j"$(nproc)" install 266 | 267 | 268 | # ================================================================== 269 | # Node.js and Jupyter Notebook Extensions 270 | # ------------------------------------------------------------------ 271 | 272 | RUN curl -sL https://deb.nodesource.com/setup_16.x | bash && \ 273 | $APT_INSTALL nodejs && \ 274 | $PIP_INSTALL jupyter_contrib_nbextensions jupyterlab-git && \ 275 | jupyter contrib nbextension install --user 276 | 277 | 278 | # ================================================================== 279 | # Startup 280 | # ------------------------------------------------------------------ 281 | 282 | EXPOSE 8888 6006 283 | 284 | CMD jupyter lab --allow-root --ip=0.0.0.0 --no-browser --ServerApp.trust_xheaders=True --ServerApp.disable_check_xsrf=False --ServerApp.allow_remote_access=True --ServerApp.allow_origin='*' --ServerApp.allow_credentials=True 285 | -------------------------------------------------------------------------------- /older-versions/pt112-tf29-jax0314-py39/README.md: -------------------------------------------------------------------------------- 1 | # Gradient Base Image 2 | 3 | This repo houses a Dockerfile used to create an image used for Gradient runtimes. This image should be used for all Gradient runtimes, either solely or as the base layer for other images. 4 | 5 | The goal of this image is to provide common packages for an advanced data science user that allows them to utilize the GPU hardware on Gradient. This image targets popular Machine Learning frameworks and includes packages commonly used for Computer Vision and Natural Language Processing tasks as well as general Data Science work. 6 | 7 | 8 | ## Software included 9 | 10 | | Category | Software | Version | Install Method | Why / Notes | 11 | | ------------- | ------------- | ------------- | ------------- | ------------- | 12 | | GPU | NVidia Driver | 510.73.05 | pre-installed | Enable Nvidia GPUs | 13 | | | CUDA | 11.6.2 | Apt | Nvidia A100 GPUs require CUDA 11+ to work, so 10.x is not suitable | 14 | | | CUDA toolkit | 11.6.2 | Apt | Needed for `nvcc` command for cuDNN | 15 | | | cuDNN | 8.4.1.*-1+cuda11.6 | Apt | Nvidia GPU deep learning library | 16 | | Python | Python | 3.9.15 | Apt | Most widely used programming language for data science | 17 | | | pip3 | 22.2.2 | Apt | Enable easy installation of 1000s of other data science, etc., packages. | 18 | | | NumPy | 1.23.4 | pip3 | Handle arrays, matrices, etc., in Python | 19 | | | SciPy | 1.9.2 | pip3 | Fundamental algorithms for scientific computing in Python | 20 | | | Pandas | 1.5.0 | pip3 | De facto standard for data science data exploration/preparation in Python | 21 | | | Cloudpickle | 2.2.0 | pip3 | Makes it possible to serialize Python constructs not supported by the default pickle module | 22 | | | Matplotlib | 3.6.1 | pip3 | Widely used plotting library in Python for data science, e.g., scikit-learn plotting requires it | 23 | | | Ipython | 8.5.0 | pip3 | Provides a rich architecture for interactive computing | 24 | | | IPykernel | 6.16.0 | pip3 | Provides the IPython kernel for Jupyter. | 25 | | | IPywidgets | 8.0.2 | pip3 | Interactive HTML widgets for Jupyter notebooks and the IPython kernel | 26 | | | Cython | 0.29.32 | pip3 | Enables writing C extensions for Python | 27 | | | tqdm | 4.64.1 | pip3 | Fast, extensible progress meter | 28 | | | gdown | 4.5.1 | pip3 | Google drive direct download of big files | 29 | | | Pillow | 9.2.0 | pip3 | Python imaging library | 30 | | | seaborn | 0.12.0 | pip3 | Python visualization library based on matplotlib | 31 | | | SQLAlchemy | 1.4.41 | pip3 | Python SQL toolkit and Object Relational Mapper that gives application developers the full power and flexibility of SQL | 32 | | | spaCy | 3.4.1 | pip3 | library for advanced Natural Language Processing in Python and Cython | 33 | | | nltk | 3.7 | pip3 | Natural Language Toolkit (NLTK) is a Python package for natural language processing | 34 | | | boto3 | 1.24.90 | pip3 | Amazon Web Services (AWS) Software Development Kit (SDK) for Python | 35 | | | tabulate | 0.9.0 | pip3 | Pretty-print tabular data in Python | 36 | | | future | 0.18.2 | pip3 | The missing compatibility layer between Python 2 and Python 3 | 37 | | | gradient | 2.0.6 | pip3 | CLI and Python SDK for Paperspace Core and Gradient | 38 | | | jsonify | 0.5 | pip3 | Provides the ability to take a .csv file as input and outputs a file with the same data in .json format | 39 | | | opencv-python | 4.6.0.66 | pip3 | Includes several hundreds of computer vision algorithms | 40 | | | JupyterLab | 3.4.6 | pip3 | De facto standard for data science using Jupyter notebooks | 41 | | | wandb | 0.13.4 | pip3 | CLI and library to interact with the Weights & Biases API (model tracking) | 42 | | Machine Learning | Scikit-learn | 1.1.2 | pip3 | Widely used ML library for data science, generally for smaller data or models | 43 | | | Scikit-image | 0.19.3 | pip3 | Collection of algorithms for image processing | 44 | | | TensorFlow | 2.9.2 | pip3 | Most widely used deep learning library, alongside PyTorch | 45 | | | torch | 1.12.1 | pip3 | Most widely used deep learning library, alongside TensorFlow | 46 | | | torchvision | 0.13.1 | pip3 | Most widely used deep learning library, alongside TensorFlow | 47 | | | torchaudio | 0.12.1 | pip3 | Most widely used deep learning library, alongside TensorFlow | 48 | | | Jax | 0.3.23 | pip3 | Popular deep learning library brought to you by Google | 49 | | | Transformers | 4.21.3 | pip3 | Popular deep learning library for NLP brought to you by HuggingFace | 50 | | | Datasets | 2.4.0 | pip3 | A supporting library for NLP use cases and the Transformers library brought to you by HuggingFace | 51 | | | XGBoost | 1.6.2 | pip3 | An optimized distributed gradient boosting library | 52 | | | Sentence Transformers | 2.2.2 | pip3 | A ML framework for sentence, paragraph and image embeddings | 53 | 54 | ### Licenses 55 | 56 | | Software | License | Source | 57 | | --------------- | ------------- | ------------- | 58 | | CUDA | NVidia EULA | https://docs.nvidia.com/cuda/eula/index.html | 59 | | cuDNN | NVidia EULA | https://docs.nvidia.com/deeplearning/cudnn/sla/index.html | 60 | | JupyterLab | New BSD | https://github.com/jupyterlab/jupyterlab/blob/master/LICENSE | 61 | | Matplotlib | PSF-based | https://matplotlib.org/stable/users/license.html | 62 | | Numpy | New BSD | https://numpy.org/doc/stable/license.html | 63 | | NVidia Docker | Apache 2.0 | https://github.com/NVIDIA/nvidia-docker/blob/master/LICENSE | 64 | | NVidia Driver | NVidia EULA | https://www.nvidia.com/en-us/drivers/nvidia-license/ | 65 | | Pandas | New BSD | https://github.com/pandas-dev/pandas/blob/master/LICENSE | 66 | | Pip3 | MIT | https://github.com/pypa/pip/blob/main/LICENSE.txt | 67 | | Python | PSF | https://en.wikipedia.org/wiki/Python_(programming_language) | 68 | | Scikit-learn | New BSD | https://github.com/scikit-learn/scikit-learn/blob/main/COPYING | 69 | | Scikit-image | New BSD | https://github.com/scikit-image/scikit-image/blob/main/LICENSE.txt | 70 | | TensorFlow | Apache 2.0 | https://github.com/tensorflow/tensorflow/blob/master/LICENSE | 71 | | PyTorch | New BSD | https://github.com/pytorch/pytorch/blob/master/LICENSE | 72 | | Jax | Apache 2.0 | https://github.com/google/jax/blob/main/LICENSE | 73 | | Transformers | Apache 2.0 | https://github.com/huggingface/transformers/blob/main/LICENSE | 74 | | Datasets | Apache 2.0 | https://github.com/huggingface/datasets/blob/main/LICENSE | 75 | | XGBoost | Apache 2.0 | https://github.com/dmlc/xgboost/blob/master/LICENSE | 76 | | Sentence Transformers | Apache 2.0 | https://github.com/UKPLab/sentence-transformers/blob/master/LICENSE | 77 | | SciPy | New BSD | https://github.com/scipy/scipy/blob/main/LICENSE.txt | 78 | | Cloudpickle | New BSD | https://github.com/cloudpipe/cloudpickle/blob/master/LICENSE | 79 | | Ipython | New BSD | https://github.com/ipython/ipython/blob/main/LICENSE | 80 | | IPykernel | New BSD | https://github.com/ipython/ipykernel/blob/main/COPYING.md | 81 | | IPywidgets | New BSD | https://github.com/jupyter-widgets/ipywidgets/blob/master/LICENSE | 82 | | Cython | Apache 2.0 | https://github.com/cython/cython/blob/master/LICENSE.txt | 83 | | tqdm | MIT | https://github.com/tqdm/tqdm/blob/master/LICENCE | 84 | | gdown | MIT | https://github.com/wkentaro/gdown/blob/main/LICENSE | 85 | | Pillow | HPND | https://github.com/python-pillow/Pillow/blob/main/LICENSE | 86 | | seaborn | New BSD | https://github.com/mwaskom/seaborn/blob/master/LICENSE.md | 87 | | SQLAlchemy | MIT | https://github.com/sqlalchemy/sqlalchemy/blob/main/LICENSE | 88 | | spaCy | MIT | https://github.com/explosion/spaCy/blob/master/LICENSE | 89 | | nltk | Apache 2.0 | https://github.com/nltk/nltk/blob/develop/LICENSE.txt | 90 | | boto3 | Apache 2.0 | https://github.com/boto/boto3/blob/develop/LICENSE | 91 | | tabulate | MIT | https://github.com/astanin/python-tabulate/blob/master/LICENSE | 92 | | future | MIT | https://github.com/PythonCharmers/python-future/blob/master/LICENSE.txt | 93 | | gradient | ISC | https://github.com/Paperspace/gradient-cli/blob/master/LICENSE.txt | 94 | | jsonify | MIT | https://pypi.org/project/jsonify/0.5/#data | 95 | | opencv-python | MIT | https://github.com/opencv/opencv-python/blob/4.x/LICENSE.txt | 96 | | wandb | MIT | https://github.com/wandb/wandb/blob/master/LICENSE | 97 | 98 | 99 | Information about license types: 100 | 101 | Apache 2.0: https://opensource.org/licenses/Apache-2.0 102 | MIT: https://opensource.org/licenses/MIT 103 | New BSD: https://opensource.org/licenses/BSD-3-Clause 104 | PSF = Python Software Foundation: https://en.wikipedia.org/wiki/Python_Software_Foundation_License 105 | HPND = Historical Permission Notice and Disclaimer: https://opensource.org/licenses/HPND 106 | ISC: https://opensource.org/licenses/ISC 107 | 108 | Open source software can be used for commercial purposes: https://opensource.org/docs/osd#fields-of-endeavor. 109 | 110 | 111 | ## Software not included 112 | 113 | Other software considered but not included. 114 | 115 | The potential data science stack is far larger than any one person will use so we don't attempt to cover everything here. 116 | 117 | Some generic categories of software not included: 118 | 119 | - Non-data-science software 120 | - Commercial software 121 | - Software not licensed to be used on an available VM template 122 | - Software only used in particular specialized data science subfields (although we assume our users probably want a GPU) 123 | 124 | | Category | Software | Why Not | 125 | | ------------- | ------------- | ------------- | 126 | | Apache | Kafka, Parquet | | 127 | | Classifiers | libsvm | H2O contains SVM and GBM, save on installs | 128 | | Collections | ELKI, GNU Octave, Weka, Mahout | | 129 | | Connectors | Academic Torrents | | 130 | | Dashboarding | panel, dash, voila, streamlit | | 131 | | Databases | MySQL, Hive, PostgreSQL, Prometheus, Neo4j, MongoDB, Cassandra, Redis | No particular infra to connect to databases | 132 | | Deep Learning | Caffe, Caffe2, Theano, PaddlePaddle, Chainer, MXNet | PyTorch and TensorFlow are dominant, rest niche | 133 | | Deployment | Dash, TFServing, R Shiny, Flask | Use Gradient Deployments | 134 | | Distributed | Horovod, OpenMPI | Use Gradient distributed | 135 | | Feature store | Feast | | 136 | | Interpretability | LIME/SHAP, Fairlearn, AI Fairness 360, InterpretML | | 137 | | Languages | R, SQL, Julia, C++, JavaScript, Python2, Scala | Python is dominant for data science | 138 | | Monitoring | Grafana | | 139 | | NLP | GenSim | | 140 | | Notebooks | Jupyter, Zeppelin | JupyterLab includes Jupyter notebook | 141 | | Orchestrators | Kubernetes | Use Gradient cluster| 142 | | Partners | fast.ai | Currently support a specific fast.ai runtime | 143 | | Pipelines | AirFlow, MLFlow, Intake, Kubeflow | | 144 | | Python libraries | statsmodels, pymc3, geopandas, Geopy, LIBSVM | Too many to attempt to cover | 145 | | PyTorch extensions | Lightning | | 146 | | R packages | ggplot, tidyverse | Could add R if customer demand | 147 | | Recommenders | TFRS, scikit-surprise | | 148 | | Scalable | Dask, Numba, Spark 1 or 2, Koalas, Hadoop | | 149 | | TensorFlow | TF 1.15, Recommenders, TensorBoard, TensorRT | Could add TensorFlow 1.x if customer demand. Requires separate tensorflow-gpu for GPU support. | 150 | | Viz | Bokeh, Plotly, Holoviz (Datashader), Google FACETS, Excalidraw, GraphViz, ggplot2, d3.js | | -------------------------------------------------------------------------------- /older-versions/ultralean-tf24-py38/Dockerfile: -------------------------------------------------------------------------------- 1 | # ================================================================== 2 | # module list 3 | # ------------------------------------------------------------------ 4 | # python 3.8 (apt) 5 | # jupyter latest (pip) 6 | # pytorch latest (pip) 7 | # tensorflow latest (pip) 8 | # jupyterlab latest (pip) 9 | # keras latest (pip) 10 | # ================================================================== 11 | 12 | FROM ubuntu:20.10 13 | ENV LANG C.UTF-8 14 | RUN APT_INSTALL="apt-get install -y --no-install-recommends" && \ 15 | DPKG_INSTALL="dpkg -i --force-all" && \ 16 | PIP_INSTALL="python -m pip --no-cache-dir install --upgrade" && \ 17 | GIT_CLONE="git clone --depth 10" && \ 18 | 19 | rm -rf /var/lib/apt/lists/* \ 20 | /etc/apt/sources.list.d/cuda.list \ 21 | /etc/apt/sources.list.d/nvidia-ml.list && \ 22 | 23 | apt-get update && \ 24 | 25 | # ================================================================== 26 | # tools 27 | # ------------------------------------------------------------------ 28 | 29 | DEBIAN_FRONTEND=noninteractive $APT_INSTALL \ 30 | build-essential \ 31 | apt-utils \ 32 | ca-certificates \ 33 | wget \ 34 | rsync \ 35 | git \ 36 | gpg-agent \ 37 | vim \ 38 | dkms \ 39 | libssl-dev \ 40 | libatlas-base-dev \ 41 | libcublas11 \ 42 | libcudart11.0 \ 43 | libcufft10 \ 44 | libcufftw10 \ 45 | libcurand10 \ 46 | libcusolver10 \ 47 | libcusparse11 \ 48 | libgl1 \ 49 | libegl1 \ 50 | libopengl0 \ 51 | libgles2 \ 52 | libx11-6 \ 53 | libxext6 \ 54 | xserver-xorg-core \ 55 | curl \ 56 | openssh-client \ 57 | unzip \ 58 | unrar \ 59 | zip \ 60 | && \ 61 | 62 | wget -nv -O /tmp/nvidia_450.36_cuda11.0_cudnn8.tgz \ 63 | http://softupdate.paperspace.io/base-container/nvidia_450.36_cuda11.0_cudnn8.tgz && \ 64 | tar xzf /tmp/nvidia_450.36_cuda11.0_cudnn8.tgz -C /tmp && \ 65 | 66 | DEBIAN_FRONTEND=noninteractive $DPKG_INSTALL \ 67 | /tmp/nvidia/*.deb \ 68 | && \ 69 | 70 | $GIT_CLONE https://github.com/Kitware/CMake ~/cmake && \ 71 | cd ~/cmake && \ 72 | ./bootstrap && \ 73 | make -j"$(nproc)" install && \ 74 | 75 | # ================================================================== 76 | # python 77 | # ------------------------------------------------------------------ 78 | 79 | DEBIAN_FRONTEND=noninteractive $APT_INSTALL \ 80 | software-properties-common \ 81 | && \ 82 | apt-get update && \ 83 | DEBIAN_FRONTEND=noninteractive $APT_INSTALL \ 84 | python3.8 \ 85 | python3.8-dev \ 86 | python3-distutils \ 87 | && \ 88 | wget -O ~/get-pip.py \ 89 | https://bootstrap.pypa.io/get-pip.py && \ 90 | python3.8 ~/get-pip.py && \ 91 | ln -s /usr/bin/python3.8 /usr/local/bin/python3 && \ 92 | ln -s /usr/bin/python3.8 /usr/local/bin/python && \ 93 | $PIP_INSTALL \ 94 | setuptools \ 95 | && \ 96 | $PIP_INSTALL \ 97 | numpy \ 98 | scipy \ 99 | pandas \ 100 | cloudpickle \ 101 | scikit-image>=0.14.2 \ 102 | scikit-learn \ 103 | matplotlib \ 104 | ipython \ 105 | ipykernel \ 106 | gradient-sdk \ 107 | Cython \ 108 | tqdm \ 109 | && \ 110 | 111 | # ================================================================== 112 | # jupyter 113 | # ------------------------------------------------------------------ 114 | 115 | $PIP_INSTALL \ 116 | jupyter \ 117 | && \ 118 | 119 | # ================================================================== 120 | # pytorch 121 | # ------------------------------------------------------------------ 122 | 123 | $PIP_INSTALL \ 124 | future \ 125 | numpy \ 126 | protobuf \ 127 | enum34 \ 128 | pyyaml \ 129 | typing \ 130 | && \ 131 | $PIP_INSTALL \ 132 | --pre torch torchvision -f \ 133 | https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html \ 134 | && \ 135 | 136 | # ================================================================== 137 | # tensorflow 138 | # ------------------------------------------------------------------ 139 | 140 | $PIP_INSTALL \ 141 | tensorflow \ 142 | && \ 143 | 144 | # ================================================================== 145 | # jupyterlab 146 | # ------------------------------------------------------------------ 147 | 148 | $PIP_INSTALL \ 149 | jupyterlab \ 150 | && \ 151 | 152 | # ================================================================== 153 | # keras 154 | # ------------------------------------------------------------------ 155 | 156 | $PIP_INSTALL \ 157 | h5py==2.10.0 \ 158 | keras \ 159 | && \ 160 | 161 | # ================================================================== 162 | # config & cleanup 163 | # ------------------------------------------------------------------ 164 | 165 | ldconfig && \ 166 | apt-get clean && \ 167 | apt-get autoremove && \ 168 | rm -rf /root/.cache/* && \ 169 | rm -rf /var/lib/apt/lists/* /tmp/* ~/* 170 | 171 | EXPOSE 8888 6006 172 | -------------------------------------------------------------------------------- /older-versions/ultralean-tf24-py38/README.md: -------------------------------------------------------------------------------- 1 | In an attempt to make the leanest functional container we start with a bare Ubuntu 20.10 and load only the minimal requirements for ML processing with latest tensorflow running on CUDA 11.0. 2 | 3 | Being bare bone, launching this container requires passing the `NVIDIA_VISIBLE_DEVICES=all` and the `NVIDIA_DRIVER_CAPABILITIES=compute,utility` env variables to the container on start, in addition to nvidia runtime. 4 | 5 | For more info see: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/user-guide.html#driver-capabilities 6 | -------------------------------------------------------------------------------- /pt211-tf215-cudatk120-py311/Dockerfile: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Paperspace Dockerfile for Gradient base image 4 | # Paperspace image is located in Dockerhub registry: paperspace/gradient_base 5 | 6 | # ================================================================== 7 | # Module list 8 | # ------------------------------------------------------------------ 9 | # python 3.11.6 (apt) 10 | # pip3 23.3.1 (apt) 11 | # cuda toolkit 12.0.0 (apt) 12 | # cudnn 8.9.7 (apt) 13 | # torch 2.1.1 (pip) 14 | # torchvision 0.16.1 (pip) 15 | # torchaudio 2.1.1 (pip) 16 | # tensorflow 2.15.0 (pip) 17 | # transformers 4.35.2 (pip) 18 | # datasets 2.14.5 (pip) 19 | # peft 0.6.2 (pip) 20 | # tokenizers 0.13.3 (pip) 21 | # accelerate 0.24.1 (pip) 22 | # diffusers 0.21.4 (pip) 23 | # safetensors 0.4.0 (pip) 24 | # jupyterlab 3.6.5 (pip) 25 | # bitsandbytes 0.41.2 (pip) 26 | # cloudpickle 2.2.1 (pip) 27 | # scikit-image 0.21.0 (pip) 28 | # scikit-learn 1.3.0 (pip) 29 | # matplotlib 3.7.3 (pip) 30 | # ipywidgets 8.1.1 (pip) 31 | # cython 3.0.2 (pip) 32 | # tqdm 4.66.1 (pip) 33 | # gdown 4.7.1 (pip) 34 | # xgboost 1.7.6 (pip) 35 | # pillow 9.5.0 (pip) 36 | # seaborn 0.12.2 (pip) 37 | # sqlalchemy 2.0.21 (pip) 38 | # spacy 3.6.1 (pip) 39 | # nltk 3.8.1 (pip) 40 | # boto3 1.28.51 (pip) 41 | # tabulate 0.9.0 (pip) 42 | # future 0.18.3 (pip) 43 | # jsonify 0.5 (pip) 44 | # opencv-python 4.8.0.76 (pip) 45 | # pyyaml 5.4.1 (pip) 46 | # sentence-transformers 2.2.2 (pip) 47 | # wandb 0.15.10 (pip) 48 | # deepspeed 0.10.3 (pip) 49 | # cupy-cuda12x 12.2.0 (pip) 50 | # timm 0.9.7 (pip) 51 | # omegaconf 2.3.0 (pip) 52 | # scipy 1.11.2 (pip) 53 | # gradient 2.0.6 (pip) 54 | # attrs 23.1.0 (pip) 55 | # default-jre latest (apt) 56 | # default-jdk latest (apt) 57 | # nodejs 20.x latest (apt) 58 | # jupyter_contrib_nbextensions 0.7.0 (pip) 59 | # jupyterlab-git 0.43.0 (pip) 60 | 61 | 62 | # ================================================================== 63 | # Initial setup 64 | # ------------------------------------------------------------------ 65 | 66 | # Ubuntu 22.04 as base image 67 | FROM ubuntu:22.04 68 | RUN yes| unminimize 69 | 70 | # Set ENV variables 71 | ENV LANG C.UTF-8 72 | ENV SHELL=/bin/bash 73 | ENV DEBIAN_FRONTEND=noninteractive 74 | 75 | ENV APT_INSTALL="apt-get install -y --no-install-recommends" 76 | ENV PIP_INSTALL="python3 -m pip --no-cache-dir install --upgrade" 77 | ENV GIT_CLONE="git clone --depth 10" 78 | 79 | 80 | # ================================================================== 81 | # Tools 82 | # ------------------------------------------------------------------ 83 | 84 | RUN apt-get update && \ 85 | $APT_INSTALL \ 86 | gcc \ 87 | make \ 88 | pkg-config \ 89 | apt-transport-https \ 90 | build-essential \ 91 | apt-utils \ 92 | ca-certificates \ 93 | wget \ 94 | rsync \ 95 | git \ 96 | vim \ 97 | mlocate \ 98 | libssl-dev \ 99 | curl \ 100 | openssh-client \ 101 | unzip \ 102 | unrar \ 103 | zip \ 104 | awscli \ 105 | csvkit \ 106 | emacs \ 107 | joe \ 108 | jq \ 109 | dialog \ 110 | man-db \ 111 | manpages \ 112 | manpages-dev \ 113 | manpages-posix \ 114 | manpages-posix-dev \ 115 | nano \ 116 | iputils-ping \ 117 | sudo \ 118 | ffmpeg \ 119 | libsm6 \ 120 | libxext6 \ 121 | libboost-all-dev \ 122 | gnupg \ 123 | cifs-utils \ 124 | zlib1g \ 125 | software-properties-common 126 | 127 | # ================================================================== 128 | # Git-lfs 129 | # ------------------------------------------------------------------ 130 | 131 | RUN curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | sudo bash && \ 132 | $APT_INSTALL git-lfs 133 | 134 | 135 | # ================================================================== 136 | # Python 137 | # ------------------------------------------------------------------ 138 | 139 | #Based on https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa 140 | 141 | # Adding repository for python3.11 142 | RUN add-apt-repository ppa:deadsnakes/ppa -y && \ 143 | $APT_INSTALL \ 144 | python3.11 \ 145 | python3.11-dev \ 146 | python3.11-venv \ 147 | python3-distutils-extra 148 | 149 | # Add symlink so python and python3 commands use same python3.9 executable 150 | RUN ln -s /usr/bin/python3.11 /usr/local/bin/python3 && \ 151 | ln -s /usr/bin/python3.11 /usr/local/bin/python 152 | 153 | # Installing pip 154 | RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3 155 | ENV PATH=$PATH:/root/.local/bin 156 | 157 | # ================================================================== 158 | # Installing CUDA packages (CUDA Toolkit 12.0 and CUDNN 8.9.7) 159 | # ------------------------------------------------------------------ 160 | RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-ubuntu2204.pin && \ 161 | mv cuda-ubuntu2204.pin /etc/apt/preferences.d/cuda-repository-pin-600 && \ 162 | wget https://developer.download.nvidia.com/compute/cuda/12.0.0/local_installers/cuda-repo-ubuntu2204-12-0-local_12.0.0-525.60.13-1_amd64.deb && \ 163 | dpkg -i cuda-repo-ubuntu2204-12-0-local_12.0.0-525.60.13-1_amd64.deb && \ 164 | cp /var/cuda-repo-ubuntu2204-12-0-local/cuda-*-keyring.gpg /usr/share/keyrings/ && \ 165 | apt-get update && \ 166 | $APT_INSTALL cuda && \ 167 | rm cuda-repo-ubuntu2204-12-0-local_12.0.0-525.60.13-1_amd64.deb 168 | 169 | # Installing CUDNN 170 | RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/3bf863cc.pub && \ 171 | add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/ /" && \ 172 | apt-get update && \ 173 | $APT_INSTALL libcudnn8=8.9.7.29-1+cuda12.2 \ 174 | libcudnn8-dev=8.9.7.29-1+cuda12.2 175 | 176 | 177 | ENV PATH=$PATH:/usr/local/cuda/bin 178 | ENV LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH 179 | 180 | 181 | # ================================================================== 182 | # PyTorch 183 | # ------------------------------------------------------------------ 184 | 185 | # Based on https://pytorch.org/get-started/locally/ 186 | 187 | RUN $PIP_INSTALL torch==2.1.1 torchvision==0.16.1 torchaudio==2.1.1 --extra-index-url https://download.pytorch.org/whl/cu121 188 | 189 | # ================================================================== 190 | # TensorFlow 191 | # ------------------------------------------------------------------ 192 | 193 | # Based on https://www.tensorflow.org/install/pip 194 | 195 | RUN $PIP_INSTALL tensorflow==2.15.0 && \ 196 | 197 | 198 | # ================================================================== 199 | # Hugging Face 200 | # ------------------------------------------------------------------ 201 | 202 | # Based on https://huggingface.co/docs/transformers/installation 203 | # Based on https://huggingface.co/docs/datasets/installation 204 | 205 | $PIP_INSTALL transformers==4.35.2 \ 206 | datasets==2.14.5 \ 207 | peft==0.6.2 \ 208 | tokenizers \ 209 | accelerate==0.24.1 \ 210 | diffusers==0.21.4 \ 211 | safetensors==0.4.0 && \ 212 | 213 | 214 | # ================================================================== 215 | # JupyterLab 216 | # ------------------------------------------------------------------ 217 | 218 | # Based on https://jupyterlab.readthedocs.io/en/stable/getting_started/installation.html#pip 219 | 220 | $PIP_INSTALL jupyterlab==3.6.5 221 | 222 | 223 | # ================================================================== 224 | # Additional Python Packages 225 | # ------------------------------------------------------------------ 226 | 227 | RUN $PIP_INSTALL \ 228 | bitsandbytes==0.41.2 \ 229 | cloudpickle==2.2.1 \ 230 | scikit-image==0.21.0 \ 231 | scikit-learn==1.3.0 \ 232 | matplotlib==3.7.3 \ 233 | ipywidgets==8.1.1 \ 234 | cython==3.0.2 \ 235 | tqdm==4.66.1 \ 236 | gdown==4.7.1 \ 237 | xgboost==1.7.6 \ 238 | pillow==9.5.0 \ 239 | seaborn==0.12.2 \ 240 | sqlalchemy==2.0.21 \ 241 | spacy==3.6.1 \ 242 | nltk==3.8.1 \ 243 | boto3==1.28.51 \ 244 | tabulate==0.9.0 \ 245 | future==0.18.3 \ 246 | jsonify==0.5 \ 247 | opencv-python==4.8.0.76 \ 248 | openpyxl==3.1.2 \ 249 | pyyaml==5.4.1 \ 250 | sentence-transformers==2.2.2 \ 251 | wandb==0.15.10 \ 252 | deepspeed==0.10.3 \ 253 | cupy-cuda12x==12.2.0 \ 254 | timm==0.9.7 \ 255 | omegaconf==2.3.0 \ 256 | scipy==1.11.2 \ 257 | gradient==2.0.6 258 | RUN $PIP_INSTALL attrs==23.1.0 259 | 260 | 261 | # ================================================================== 262 | # Installing JRE and JDK 263 | # ------------------------------------------------------------------ 264 | 265 | RUN $APT_INSTALL \ 266 | default-jre \ 267 | default-jdk 268 | 269 | 270 | # ================================================================== 271 | # CMake 272 | # ------------------------------------------------------------------ 273 | 274 | RUN $GIT_CLONE https://github.com/Kitware/CMake ~/cmake && \ 275 | cd ~/cmake && \ 276 | ./bootstrap && \ 277 | make -j"$(nproc)" install 278 | 279 | 280 | # ================================================================== 281 | # Node.js and Jupyter Notebook Extensions 282 | # ------------------------------------------------------------------ 283 | 284 | RUN curl -sL https://deb.nodesource.com/setup_20.x | bash && \ 285 | $APT_INSTALL nodejs && \ 286 | $PIP_INSTALL jupyter_contrib_nbextensions==0.7.0 jupyterlab-git==0.43.0 && \ 287 | jupyter contrib nbextension install --user 288 | 289 | 290 | # ================================================================== 291 | # Startup 292 | # ------------------------------------------------------------------ 293 | 294 | EXPOSE 8888 6006 295 | 296 | CMD jupyter lab --allow-root --ip=0.0.0.0 --no-browser --ServerApp.trust_xheaders=True --ServerApp.disable_check_xsrf=False --ServerApp.allow_remote_access=True --ServerApp.allow_origin='*' --ServerApp.allow_credentials=True -------------------------------------------------------------------------------- /pt211-tf215-cudatk120-py311/README.md: -------------------------------------------------------------------------------- 1 | # Gradient Base Image 2 | 3 | This repo houses a Dockerfile used to create an image used for Gradient runtimes. This image should be used for all Gradient runtimes, either solely or as the base layer for other images. 4 | 5 | The goal of this image is to provide common packages for an advanced data science user that allows them to utilize the GPU hardware on Gradient. This image targets popular Machine Learning frameworks and includes packages commonly used for Computer Vision and Natural Language Processing tasks as well as general Data Science work. 6 | 7 | 8 | ## Software included 9 | 10 | | Category | Software | Version | Install Method | Why / Notes | 11 | | ------------- | ------------- | ------------- | ------------- | ------------- | 12 | | GPU | NVidia Driver | 525.116.04 | pre-installed | Enable Nvidia GPUs | 13 | | | CUDA | 12.0.0 | Apt | Nvidia A100 GPUs require CUDA 11+ to work, so 10.x is not suitable | 14 | | | CUDA toolkit | 12.0.0 | Apt | Needed for `nvcc` command for cuDNN | 15 | | | cuDNN | 8.9.7.29-1+cuda12.2 | Apt | Nvidia GPU deep learning library | 16 | | Python | Python | 3.11.6 | Apt | Most widely used programming language for data science | 17 | | | pip3 | 23.3.1 | Apt | Enable easy installation of 1000s of other data science, etc., packages. | 18 | | | NumPy | 1.26.3 | pip3 | Handle arrays, matrices, etc., in Python | 19 | | | SciPy | 1.11.2 | pip3 | Fundamental algorithms for scientific computing in Python | 20 | | | Pandas | 2.1.4 | pip3 | De facto standard for data science data exploration/preparation in Python | 21 | | | Cloudpickle | 2.2.1 | pip3 | Makes it possible to serialize Python constructs not supported by the default pickle module | 22 | | | Matplotlib | 3.7.3 | pip3 | Widely used plotting library in Python for data science, e.g., scikit-learn plotting requires it | 23 | | | Ipython | 8.19.0 | pip3 | Provides a rich architecture for interactive computing | 24 | | | IPykernel | 6.28.0 | pip3 | Provides the IPython kernel for Jupyter. | 25 | | | IPywidgets | 8.1.1 | pip3 | Interactive HTML widgets for Jupyter notebooks and the IPython kernel | 26 | | | Cython | 3.0.2 | pip3 | Enables writing C extensions for Python | 27 | | | tqdm | 4.66.1 | pip3 | Fast, extensible progress meter | 28 | | | gdown | 4.7.1 | pip3 | Google drive direct download of big files | 29 | | | Pillow | 9.5.0 | pip3 | Python imaging library | 30 | | | seaborn | 0.12.2 | pip3 | Python visualization library based on matplotlib | 31 | | | SQLAlchemy | 2.0.21 | pip3 | Python SQL toolkit and Object Relational Mapper that gives application developers the full power and flexibility of SQL | 32 | | | spaCy | 3.6.1 | pip3 | library for advanced Natural Language Processing in Python and Cython | 33 | | | nltk | 3.8.1 | pip3 | Natural Language Toolkit (NLTK) is a Python package for natural language processing | 34 | | | boto3 | 1.28.51 | pip3 | Amazon Web Services (AWS) Software Development Kit (SDK) for Python | 35 | | | tabulate | 0.9.0 | pip3 | Pretty-print tabular data in Python | 36 | | | future | 0.18.3 | pip3 | The missing compatibility layer between Python 2 and Python 3 | 37 | | | gradient | 2.0.6 | pip3 | CLI and Python SDK for Paperspace Core and Gradient | 38 | | | jsonify | 0.5 | pip3 | Provides the ability to take a .csv file as input and outputs a file with the same data in .json format | 39 | | | opencv-python | 4.6.0.76 | pip3 | Includes several hundreds of computer vision algorithms | 40 | | | JupyterLab | 3.6.5 | pip3 | De facto standard for data science using Jupyter notebooks | 41 | | | wandb | 0.15.10 | pip3 | CLI and library to interact with the Weights & Biases API (model tracking) | 42 | | Machine Learning | Scikit-learn | 1.3.0 | pip3 | Widely used ML library for data science, generally for smaller data or models | 43 | | | Scikit-image | 0.21.0 | pip3 | Collection of algorithms for image processing | 44 | | | TensorFlow | 2.15.0 | pip3 | Most widely used deep learning library, alongside PyTorch | 45 | | | torch | 2.1.1 | pip3 | Most widely used deep learning library, alongside TensorFlow | 46 | | | torchvision | 0.16.1 | pip3 | Most widely used deep learning library, alongside TensorFlow | 47 | | | torchaudio | 2.1.1 | pip3 | Most widely used deep learning library, alongside TensorFlow | 48 | | | Transformers | 4.35.2 | pip3 | Popular deep learning library for NLP brought to you by HuggingFace | 49 | | | Datasets | 2.14.5 | pip3 | A supporting library for NLP use cases and the Transformers library brought to you by HuggingFace | 50 | | | XGBoost | 1.7.6 | pip3 | An optimized distributed gradient boosting library | 51 | | | Sentence Transformers | 2.2.2 | pip3 | A ML framework for sentence, paragraph and image embeddings | 52 | 53 | ### Licenses 54 | 55 | | Software | License | Source | 56 | | --------------- | ------------- | ------------- | 57 | | CUDA | NVidia EULA | https://docs.nvidia.com/cuda/eula/index.html | 58 | | cuDNN | NVidia EULA | https://docs.nvidia.com/deeplearning/cudnn/sla/index.html | 59 | | JupyterLab | New BSD | https://github.com/jupyterlab/jupyterlab/blob/master/LICENSE | 60 | | Matplotlib | PSF-based | https://matplotlib.org/stable/users/license.html | 61 | | Numpy | New BSD | https://numpy.org/doc/stable/license.html | 62 | | NVidia Docker | Apache 2.0 | https://github.com/NVIDIA/nvidia-docker/blob/master/LICENSE | 63 | | NVidia Driver | NVidia EULA | https://www.nvidia.com/en-us/drivers/nvidia-license/ | 64 | | Pandas | New BSD | https://github.com/pandas-dev/pandas/blob/master/LICENSE | 65 | | Pip3 | MIT | https://github.com/pypa/pip/blob/main/LICENSE.txt | 66 | | Python | PSF | https://en.wikipedia.org/wiki/Python_(programming_language) | 67 | | Scikit-learn | New BSD | https://github.com/scikit-learn/scikit-learn/blob/main/COPYING | 68 | | Scikit-image | New BSD | https://github.com/scikit-image/scikit-image/blob/main/LICENSE.txt | 69 | | TensorFlow | Apache 2.0 | https://github.com/tensorflow/tensorflow/blob/master/LICENSE | 70 | | PyTorch | New BSD | https://github.com/pytorch/pytorch/blob/master/LICENSE | 71 | | Jax | Apache 2.0 | https://github.com/google/jax/blob/main/LICENSE | 72 | | Transformers | Apache 2.0 | https://github.com/huggingface/transformers/blob/main/LICENSE | 73 | | Datasets | Apache 2.0 | https://github.com/huggingface/datasets/blob/main/LICENSE | 74 | | XGBoost | Apache 2.0 | https://github.com/dmlc/xgboost/blob/master/LICENSE | 75 | | Sentence Transformers | Apache 2.0 | https://github.com/UKPLab/sentence-transformers/blob/master/LICENSE | 76 | | SciPy | New BSD | https://github.com/scipy/scipy/blob/main/LICENSE.txt | 77 | | Cloudpickle | New BSD | https://github.com/cloudpipe/cloudpickle/blob/master/LICENSE | 78 | | Ipython | New BSD | https://github.com/ipython/ipython/blob/main/LICENSE | 79 | | IPykernel | New BSD | https://github.com/ipython/ipykernel/blob/main/COPYING.md | 80 | | IPywidgets | New BSD | https://github.com/jupyter-widgets/ipywidgets/blob/master/LICENSE | 81 | | Cython | Apache 2.0 | https://github.com/cython/cython/blob/master/LICENSE.txt | 82 | | tqdm | MIT | https://github.com/tqdm/tqdm/blob/master/LICENCE | 83 | | gdown | MIT | https://github.com/wkentaro/gdown/blob/main/LICENSE | 84 | | Pillow | HPND | https://github.com/python-pillow/Pillow/blob/main/LICENSE | 85 | | seaborn | New BSD | https://github.com/mwaskom/seaborn/blob/master/LICENSE.md | 86 | | SQLAlchemy | MIT | https://github.com/sqlalchemy/sqlalchemy/blob/main/LICENSE | 87 | | spaCy | MIT | https://github.com/explosion/spaCy/blob/master/LICENSE | 88 | | nltk | Apache 2.0 | https://github.com/nltk/nltk/blob/develop/LICENSE.txt | 89 | | boto3 | Apache 2.0 | https://github.com/boto/boto3/blob/develop/LICENSE | 90 | | tabulate | MIT | https://github.com/astanin/python-tabulate/blob/master/LICENSE | 91 | | future | MIT | https://github.com/PythonCharmers/python-future/blob/master/LICENSE.txt | 92 | | gradient | ISC | https://github.com/Paperspace/gradient-cli/blob/master/LICENSE.txt | 93 | | jsonify | MIT | https://pypi.org/project/jsonify/0.5/#data | 94 | | opencv-python | MIT | https://github.com/opencv/opencv-python/blob/4.x/LICENSE.txt | 95 | | wandb | MIT | https://github.com/wandb/wandb/blob/master/LICENSE | 96 | 97 | 98 | Information about license types: 99 | 100 | Apache 2.0: https://opensource.org/licenses/Apache-2.0 101 | MIT: https://opensource.org/licenses/MIT 102 | New BSD: https://opensource.org/licenses/BSD-3-Clause 103 | PSF = Python Software Foundation: https://en.wikipedia.org/wiki/Python_Software_Foundation_License 104 | HPND = Historical Permission Notice and Disclaimer: https://opensource.org/licenses/HPND 105 | ISC: https://opensource.org/licenses/ISC 106 | 107 | Open source software can be used for commercial purposes: https://opensource.org/docs/osd#fields-of-endeavor. 108 | 109 | 110 | ## Software not included 111 | 112 | Other software considered but not included. 113 | 114 | The potential data science stack is far larger than any one person will use so we don't attempt to cover everything here. 115 | 116 | Some generic categories of software not included: 117 | 118 | - Non-data-science software 119 | - Commercial software 120 | - Software not licensed to be used on an available VM template 121 | - Software only used in particular specialized data science subfields (although we assume our users probably want a GPU) 122 | 123 | | Category | Software | Why Not | 124 | | ------------- | ------------- | ------------- | 125 | | Apache | Kafka, Parquet | | 126 | | Classifiers | libsvm | H2O contains SVM and GBM, save on installs | 127 | | Collections | ELKI, GNU Octave, Weka, Mahout | | 128 | | Connectors | Academic Torrents | | 129 | | Dashboarding | panel, dash, voila, streamlit | | 130 | | Databases | MySQL, Hive, PostgreSQL, Prometheus, Neo4j, MongoDB, Cassandra, Redis | No particular infra to connect to databases | 131 | | Deep Learning | Caffe, Caffe2, Theano, PaddlePaddle, Chainer, MXNet | PyTorch and TensorFlow are dominant, rest niche | 132 | | Deployment | Dash, TFServing, R Shiny, Flask | Use Gradient Deployments | 133 | | Distributed | Horovod, OpenMPI | Use Gradient distributed | 134 | | Feature store | Feast | | 135 | | Interpretability | LIME/SHAP, Fairlearn, AI Fairness 360, InterpretML | | 136 | | Languages | R, SQL, Julia, C++, JavaScript, Python2, Scala | Python is dominant for data science | 137 | | Monitoring | Grafana | | 138 | | NLP | GenSim | | 139 | | Notebooks | Jupyter, Zeppelin | JupyterLab includes Jupyter notebook | 140 | | Orchestrators | Kubernetes | Use Gradient cluster| 141 | | Partners | fast.ai | Currently support a specific fast.ai runtime | 142 | | Pipelines | AirFlow, MLFlow, Intake, Kubeflow | | 143 | | Python libraries | statsmodels, pymc3, geopandas, Geopy, LIBSVM | Too many to attempt to cover | 144 | | PyTorch extensions | Lightning | | 145 | | R packages | ggplot, tidyverse | Could add R if customer demand | 146 | | Recommenders | TFRS, scikit-surprise | | 147 | | Scalable | Dask, Numba, Spark 1 or 2, Koalas, Hadoop | | 148 | | TensorFlow | TF 1.15, Recommenders, TensorBoard, TensorRT | Could add TensorFlow 1.x if customer demand. Requires separate tensorflow-gpu for GPU support. | 149 | | Viz | Bokeh, Plotly, Holoviz (Datashader), Google FACETS, Excalidraw, GraphViz, ggplot2, d3.js | | 150 | --------------------------------------------------------------------------------