├── .gitignore ├── LICENSE ├── README.md ├── cmd.bat ├── docker ├── Dockerfile ├── build.sh ├── prune.sh ├── run.sh ├── ssh-into-llm.sh └── trunc-llm-containers.sh ├── install.bat ├── install └── install.ps1 ├── start.bat ├── stop.bat └── uninstall.bat /.gitignore: -------------------------------------------------------------------------------- 1 | /install/ext4.vhdx -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2022 Radosław Gryta 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # LLM on Docker (WSL2) 2 | 3 | ## About 4 | 5 | This repository is meant to allow for easy installation of LLM on Windows. One click to install. Second click to start. 6 | 7 | It's a direct copy of what https://github.com/rgryta/Stable-Diffusion-WSL2-Docker is for AUTOMATIC1111 webui, but for LLM and using oobabooga webui for LLMs. 8 | 9 | ## Prerequisites 10 | 11 | Before following through with these instructions. Please verify below. 12 | 13 | 1. You have virtualization support - easiest way is to check if you can see "Virtualization" section in Windows Task Manager -> Performance -> CPU (it's located under "More details" if you don't see the Performance tab). 14 | 1. You have virtualization enabled - you have to enable it in your BIOS if you don't. 15 | 1. You have Windows 11 Pro - you can also use Windows 11 Home (also Windows 10 above certain version), but I cannot guarantee that provided scripts will work their magic. 16 | 1. You have Nvidia GPU - this is mandatory for current configuration. Support for AMD is presumably possible, but won't be added until such request shows up. Make sure you also have the newest drivers! Whole repository is based on CUDA 12 - you will be limited to GTX 900-series or higher. 17 | 1. You need admin access. These scripts use a PowerShell library that I've prepared, called [WSLTools](https://github.com/rgryta/PowerShell-WSLTools) (handles automatic and interactive installation of WSL distributions from source), you need to have admin privileges to install this module. 18 | 19 | ## How to use 20 | 21 | After installation simply execute start.bat file to start the LLM app. You can open it under [http://localhost:7860/?__theme=dark](http://localhost:7860/?__theme=dark). 22 | 23 | If you want to close the app - simply launch stop.bat, it will terminate the application and close the terminals. 24 | 25 | Note! Keep in mind that stop.bat will terminate and remove all containers based on LLM webui image. If you have downloaded additional models while the application was running - they will have to be redownloaded again. 26 | 27 | ## Installation 28 | 29 | ### Automatic 30 | 31 | Run install.bat in order to install the LLM. This will take a while - as long as you don't see red errors - everything's fine. 32 | 33 | ### Manual 34 | 35 | 1. Install Windows 11 36 | 1. Install WSL from MS Store (https://www.microsoft.com/store/productId/9P9TQF7MRM4R) 37 | 1. Search for "Turn Windows features on or off" and enable "Hyper-V" 38 | 1. Set WSL to use v2: `wsl --set-default-version 2` 39 | 1. Install Linux distro of your choice (Ubuntu given as example): `wsl --install Ubuntu` 40 | 1. Set up your username and password 41 | 1. (In distro command line) `sudo sh -c 'echo "[boot]\nsystemd=true" > /etc/wsl.conf'` 42 | 1. Check your distro name using `wsl --list` 43 | 1. Shutdown all distros `wsl --shutdown` and restart the one we're using `wsl --distribution Ubuntu` 44 | 1. Make sure you have nvidia drivers installed on Windows 45 | 1. Now open WSL. From now on, everything is executed from there. 46 | 1. Execute following scripts (installs cuda drivers): 47 | ```bash 48 | sudo apt-key del 7fa2af80 49 | wget https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-wsl-ubuntu.pin 50 | sudo mv cuda-wsl-ubuntu.pin /etc/apt/preferences.d/cuda-repository-pin-600 51 | sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/3bf863cc.pub 52 | sudo add-apt-repository 'deb https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/ /' 53 | sudo apt-get update 54 | sudo apt-get -y install cuda 55 | ``` 56 | 1. Check if you're able to see your GPU in WSL: nvidia-smi 57 | 1. Install docker: 58 | ```bash 59 | curl https://get.docker.com | sh \ 60 | && sudo systemctl --now enable docker 61 | ``` 62 | 1. Prepare gpg keys to install nvidia-docker: 63 | ```bash 64 | distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \ 65 | && curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \ 66 | && curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \ 67 | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \ 68 | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list``` 69 | 1. Now we can install it: `sudo apt-get install -y nvidia-docker2` 70 | 1. Restart docker service: `sudo systemctl restart docker` 71 | 1. Check if docker container also sees your GPU: `sudo docker run --rm --gpus all nvidia/cuda:12.0.1-base-ubuntu22.04 nvidia-smi` 72 | 1. Run `./build.sh` from repo directory to build the container. You can uncomment depth, upscaler, inpainting and gfpgan from Dockerfile (first generated image) but it will take much more space - default installation is ~65GB total. 73 | 1. Run `./run.sh` to start container. Open http://localhost:7860/?__theme=dark to access the webui - you can do so from Windows of course. 74 | 75 | ## Sources 76 | 77 | 1. [WizardLM HuggingFace](https://huggingface.co/ehartford) 78 | 1. [oobabooga webui](https://github.com/oobabooga/text-generation-webui) 79 | 1. [Nvidia Container Runtime](https://nvidia.github.io/nvidia-container-runtime/) 80 | 1. [Ubuntu GPU acceleration on WSL2](https://ubuntu.com/tutorials/enabling-gpu-acceleration-on-ubuntu-on-wsl2-with-the-nvidia-cuda-platform#3-install-nvidia-cuda-on-ubuntu) 81 | 1. [MS WSL systemd](https://devblogs.microsoft.com/commandline/systemd-support-is-now-available-in-wsl/) 82 | 1. [Nvidia WSL](https://docs.nvidia.com/cuda/wsl-user-guide/index.html) 83 | -------------------------------------------------------------------------------- /cmd.bat: -------------------------------------------------------------------------------- 1 | wsl -d ubuntu-llm -------------------------------------------------------------------------------- /docker/Dockerfile: -------------------------------------------------------------------------------- 1 | ################ Download models ################ 2 | FROM alpine:latest AS dw_models 3 | RUN apk add wget curl jq 4 | 5 | RUN mkdir -p /llm/models 6 | 7 | ### MODELS: 8 | 9 | ## WizardLM 10 | RUN mkdir /llm/models/WizardLM 11 | WORKDIR /llm/models/WizardLM 12 | 13 | RUN for file in `curl https://huggingface.co/api/models/TheBloke/WizardLM-13B-V1.0-Uncensored-GPTQ | jq -r .siblings[].rfilename`; \ 14 | do curl -L -C - https://huggingface.co/TheBloke/WizardLM-13B-V1.0-Uncensored-GPTQ/resolve/main/$file --create-dirs -o $file; done 15 | 16 | # RUN for file in `curl https://huggingface.co/api/models/ehartford/WizardLM-7B-V1.0-Uncensored | jq -r .siblings[].rfilename`; \ 17 | # do curl -L -C - https://huggingface.co/ehartford/WizardLM-7B-V1.0-Uncensored/resolve/main/$file --create-dirs -o $file; done 18 | # 19 | # Replace with this to use 13B model (or a different one) 20 | # RUN for file in `curl https://huggingface.co/api/models/ehartford/WizardLM-13B-V1.0-Uncensored | jq -r .siblings[].rfilename`; \ 21 | # do curl -L -C - https://huggingface.co/ehartford/WizardLM-13B-V1.0-Uncensored/resolve/main/$file --create-dirs -o $file; done 22 | 23 | ## OpenLlama 24 | # RUN mkdir /llm/models/OpenLlama 25 | # WORKDIR /llm/models/OpenLlama 26 | # 27 | # RUN for file in `curl https://huggingface.co/api/models/openlm-research/open_llama_13b | jq -r .siblings[].rfilename`; \ 28 | # do curl -L -C - https://huggingface.co/openlm-research/open_llama_13b/resolve/main/$file --create-dirs -o $file; done 29 | 30 | 31 | 32 | ################ Main container ################ 33 | FROM nvidia/cuda:12.1.1-devel-ubuntu22.04 as llm_webui 34 | 35 | ENV LLM_HOME=/home/llm 36 | ENV LLM_PROJECT=$LLM_HOME/text-generation-webui 37 | 38 | # Install necessary packages 39 | RUN apt-get update && \ 40 | apt-get install --no-install-recommends --no-install-suggests -y \ 41 | sudo git g++ python3 python3-pip python3-venv python3-dev ffmpeg libsm6 libxext6 curl wget vim 42 | 43 | # Create LLM user 44 | RUN useradd -m -G sudo -p $(openssl passwd -1 llm) llm 45 | USER llm 46 | 47 | # Python setup - PYTHONUNBUFFERED set to 1 makes logs pipe into the container output 48 | RUN pip install --upgrade pip 49 | # Torch support list based on https://github.com/pytorch/builder/blob/main/conda/pytorch-nightly/build.sh 50 | # and https://arnon.dk/matching-sm-architectures-arch-and-gencode-for-various-nvidia-cards/ 51 | ENV PYTHONUNBUFFERED=1 \ 52 | PATH="$PATH:$LLM_HOME/.local/bin" \ 53 | TORCH_CUDA_ARCH_LIST=All \ 54 | FORCE_CUDA=1 55 | 56 | RUN pip install torch torchvision torchaudio xformers 57 | 58 | # Clone text generation webui repo 59 | RUN cd $LLM_HOME && git clone https://github.com/oobabooga/text-generation-webui.git 60 | 61 | WORKDIR $LLM_PROJECT 62 | RUN pip install -r requirements.txt 63 | 64 | 65 | # Copy models 66 | COPY --from=dw_models --chown=llm /llm/ $LLM_PROJECT/ 67 | 68 | EXPOSE 7860/tcp 69 | 70 | # If using non-GPTQ models remove --loader exllama_hf 71 | CMD ["/bin/bash", "-c", "python3 server.py --chat --auto-devices --xformers --listen --load-in-4bit --loader exllama_hf"] -------------------------------------------------------------------------------- /docker/build.sh: -------------------------------------------------------------------------------- 1 | docker buildx build -t llm:latest . -------------------------------------------------------------------------------- /docker/prune.sh: -------------------------------------------------------------------------------- 1 | docker system prune -f -------------------------------------------------------------------------------- /docker/run.sh: -------------------------------------------------------------------------------- 1 | echo "Removing old containers" 2 | docker container stop `docker ps -a | grep llm | awk '{print $1}' | awk 'BEGIN { ORS = " " } { print }'` 3 | docker container rm `docker ps -a | grep llm | awk '{print $1}' | awk 'BEGIN { ORS = " " } { print }'` 4 | 5 | echo "Starting new container" 6 | docker run -p 127.0.0.1:7860:7860 --gpus all llm -------------------------------------------------------------------------------- /docker/ssh-into-llm.sh: -------------------------------------------------------------------------------- 1 | docker exec -it `docker ps -a | grep llm | head -1 | awk '{print $1}'` /bin/bash -------------------------------------------------------------------------------- /docker/trunc-llm-containers.sh: -------------------------------------------------------------------------------- 1 | docker container stop `docker ps -a | grep llm | awk '{print $1}' | awk 'BEGIN { ORS = " " } { print }'` 2 | docker container rm `docker ps -a | grep llm | awk '{print $1}' | awk 'BEGIN { ORS = " " } { print }'` -------------------------------------------------------------------------------- /install.bat: -------------------------------------------------------------------------------- 1 | powershell.exe "Start-Process powershell -Verb RunAs -ArgumentList '-NoExit -ExecutionPolicy Bypass -file %~dp0\install\install.ps1'" 2 | -------------------------------------------------------------------------------- /install/install.ps1: -------------------------------------------------------------------------------- 1 | # Install dependencies for installing Linux 2 | if (-not (Get-Module -ListAvailable -Name WSLTools)) { 3 | Install-Module -Name WSLTools -Force 4 | } 5 | 6 | Import-Module WSLTools -WarningAction SilentlyContinue 7 | if (-not (Ensure-WSL)) { 8 | $question = "Yes","No" 9 | $selected = Get-Select -Prompt "[OPER] Would you like to install HyperV and WSL now?" -Options $question 10 | if ($selected -eq "Yes") { 11 | iex ((New-Object System.Net.WebClient).DownloadString('https://raw.githubusercontent.com/rgryta/PowerShell-WSLTools/main/install-wsl.ps1')) 12 | Write-Host "Reboot your system now and then restart the script" 13 | } 14 | if ($selected -eq "No") { 15 | Write-Host "Please set up HyperV and WSL manually and then relaunch the script" 16 | } 17 | return $false 18 | } 19 | 20 | # Install Ubuntu on WSL2 21 | $scriptPath = Split-Path -Parent $MyInvocation.MyCommand.Definition 22 | 23 | $distro = 'ubuntu-llm' 24 | $ignr = wsl --unregister $distro 25 | 26 | WSL-Ubuntu-Install -DistroAlias $distro -InstallPath $scriptPath -Version jammy # lunar has no docker, and kinetic has no nvidia-container-toolkit 27 | $ignr = wsl -d $distro -u root -e sh -c "apt-get install -y apt-utils sudo curl systemd jq libssl-dev build-essential checkinstall zlib1g-dev git wget" 28 | 29 | # Creating new user 30 | wsl -d $distro -u root -e sh -c 'useradd -m -G sudo -p $(echo `openssl passwd -1 llm`) llm' 31 | 32 | # Enabling Systemd (with fix: https://github.com/microsoft/WSL/issues/9602#issuecomment-1421897547) 33 | $ignr = wsl -d $distro -u root -e sh -c 'echo "[boot]\\nsystemd=true" > /etc/wsl.conf' 34 | $ignr = wsl -d $distro -u root -e sh -c 'echo "[user]\\ndefault=llm" >> /etc/wsl.conf' 35 | $ignr = wsl -d $distro -u root -e sh -c 'ln -s /usr/lib/systemd/systemd /sbin/init' 36 | 37 | Write-Host "Waiting for Ubuntu setup to finish..." 38 | while ($true) 39 | { 40 | $username = wsl -d $distro -e sh -c "grep -v '/usr/sbin/nologin' /etc/passwd | grep 'llm:' | awk -F: '{print `$1}'" 41 | if ($username -ne $null) { 42 | Write-Host "Created default WSL user: $username" 43 | break 44 | } 45 | } 46 | 47 | # Waiting for processes to finish and restarting 48 | while ($true) 49 | { 50 | $setupstatus = wsl -d $distro -u root -e sh -c "ps -Ao comm --no-headers | grep 'adduser\|passwd' | wc -l" 51 | if ($setupstatus -eq '0') { 52 | break 53 | } 54 | } 55 | $ignr = wsl -t $distro 56 | 57 | # Setting up CUDA drivers 58 | $ignr = wsl -d $distro -u root -e sh -c "apt-key del 7fa2af80 > /dev/null 2>&1 `&`& wget https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-wsl-ubuntu.pin > /dev/null 2>&1 `&`& mv cuda-wsl-ubuntu.pin /etc/apt/preferences.d/cuda-repository-pin-600 `&`& apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/3bf863cc.pub > /dev/null 2>&1 `&`& add-apt-repository -y 'deb https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/ /' > /dev/null 2>&1 `&`& apt-get update > /dev/null 2>&1 `&`& apt-get -y install cuda > /dev/null 2>&1" 59 | 60 | 61 | Write-Host 'If you can see your GPU listed below, everything went smoothly so far:' 62 | Start-Sleep -Seconds 5 63 | wsl -d $distro -e sh -c 'nvidia-smi' 64 | 65 | # Installing Docker 66 | $ignr = wsl -d $distro -u root -e sh -c "curl https://get.docker.com | sh > /dev/null 2>&1 `&`& systemctl --now enable docker > /dev/null 2>&1 " 67 | 68 | $ignr = wsl -d $distro -u root -e sh -c "distribution=`$(. /etc/os-release;echo `$ID`$VERSION_ID) `&`& curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg > /dev/null 2>&1 `&`& curl -s -L https://nvidia.github.io/libnvidia-container/`$distribution/libnvidia-container.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | tee /etc/apt/sources.list.d/nvidia-container-toolkit.list" 69 | 70 | # Installing Nvidia Docker 71 | Write-Host 'Installing nvidia-docker' 72 | 73 | $ignr = wsl -d $distro -u root -e sh -c "apt-get update > /dev/null 2>&1 `&`& apt-get install -y nvidia-docker2 > /dev/null 2>&1" 74 | $ignr = wsl -d $distro -u root -e sh -c 'systemctl restart docker' 75 | $ignr = wsl -d $distro -u root -e sh -c "apt-get upgrade -y > /dev/null 2>&1 `&`& apt-get autoremove -y `&`& apt-get autoclean -y" 76 | 77 | # Testing Nvidia Docker 78 | Clear-Host 79 | Write-Host 'Pulling base NVIDIA CUDA container' 80 | $ignr = wsl -d $distro -u root -e sh -c "usermod -aG docker $username" 81 | $ignr = wsl -d $distro -e sh -c "docker pull nvidia/cuda:12.0.1-base-ubuntu22.04" 82 | 83 | Write-Host 'Verify if you are able to see your GPU below - this time within docker container:' 84 | wsl -d $distro -e sh -c "docker run --rm --gpus all nvidia/cuda:12.0.1-base-ubuntu22.04 nvidia-smi" 85 | 86 | Write-Host 'Now local container will be built. This can take about an hour depending on your CPU and internet speed.' 87 | wsl -d $distro -e sh -c "cd ``wslpath -a '$scriptPath'``/../docker && ./build.sh" 88 | 89 | Clear-Host 90 | Write-Host 'You can now use `run.bat` to launch Stable Diffusion. Closing this window in 5 seconds...' 91 | Start-Sleep -Seconds 5 -------------------------------------------------------------------------------- /start.bat: -------------------------------------------------------------------------------- 1 | wsl -d ubuntu-llm -e sh -c "cd `wslpath -a '%~dp0'`/docker && ./run.sh" 2 | -------------------------------------------------------------------------------- /stop.bat: -------------------------------------------------------------------------------- 1 | wsl -d ubuntu-llm -e sh -c "cd `wslpath -a '%~dp0'`/docker && ./trunc-llm-containers.sh" 2 | -------------------------------------------------------------------------------- /uninstall.bat: -------------------------------------------------------------------------------- 1 | wsl --unregister ubuntu-llm 2 | --------------------------------------------------------------------------------