├── requirements.txt ├── .gitignore ├── images ├── raspberrypi.jpeg ├── screenshot1.png ├── screenshot2.png └── screenshot3.png ├── README.md └── run-llm-on-raspberry-pi.md /requirements.txt: -------------------------------------------------------------------------------- 1 | openai==1.37.1 2 | pandas==2.2.2 -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .venv*/* 2 | *.pyc 3 | *.pyo 4 | *.pyd 5 | __pycache__/ 6 | .DS_Store 7 | .env -------------------------------------------------------------------------------- /images/raspberrypi.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/adamcohenhillel/LLMs-Cheatsheet/HEAD/images/raspberrypi.jpeg -------------------------------------------------------------------------------- /images/screenshot1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/adamcohenhillel/LLMs-Cheatsheet/HEAD/images/screenshot1.png -------------------------------------------------------------------------------- /images/screenshot2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/adamcohenhillel/LLMs-Cheatsheet/HEAD/images/screenshot2.png -------------------------------------------------------------------------------- /images/screenshot3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/adamcohenhillel/LLMs-Cheatsheet/HEAD/images/screenshot3.png -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # LLMs-Cheatsheet 2 | 3 | This repo aims to be a comprehensive resource for those interested in Large Language Models (LLMs), with a specific focus on running and fine-tuning small models on personal hardware. Our goal is to empower users to work with powerful, locally-run LLMs that are tailored to their needs, ensuring data privacy and maintaining digital sovereignty. By avoiding reliance on third-party providers, you retain full control over your models and information, free from the risk of sudden access restrictions or policy changes. 4 | 5 | ## Resources: 6 | 7 | - [Running LLMs on a Raspberry Pi](./run-llm-on-raspberry-pi.md) 8 | - [Fine-tune small LLama3 8B model with output from Big LLM Llama 3 405B](./fine-tune-llama3-8b-with-llama3-405b-data.ipynb) 9 | 10 | ## Contributing 11 | 12 | We welcome contributions! If you have resources, guides, or insights to share, please do! 13 | 14 | --- 15 | 16 | This repository is a work in progress. Stay tuned for updates and feel free to star or watch the project for notifications on new content! 17 | -------------------------------------------------------------------------------- /run-llm-on-raspberry-pi.md: -------------------------------------------------------------------------------- 1 |
2 | 3 | # How to run LLM (mistral 7b) on Raspberry Pi 5 4 | 5 | How to run LLM (mistral 7b) on Raspberry Pi 5 6 | 7 | Step-by-Step Guide on how to run Large Language Model on a Raspberry Pi 5 (might work on 4 too, haven't tested it yet) 8 | 9 |
10 | 11 | ### Table of Content: 12 | 13 | 1. [Prerequisite](#prerequisite) 14 | 2. [Setup Raspberry Pi](#setup-raspberry-pi-headless-setup) 15 | 3. [Option 1: Run LLMs using Ollama](#option-1-run-llms-using-ollama) 16 | 4. [Option 2: Run LLMs using Llama.cpp](#option-2-run-llms-using-llamacpp) 17 | 5. [Extra Resoucres](#extra-resoucres) 18 | 19 | --- 20 | 21 | ### Prerequisite 22 | 23 | - [Raspberry Pi 5, 8GB RAM](https://www.raspberrypi.com/products/raspberry-pi-5/) 24 | - 32GB SD Card 25 | 26 | ### Setup Raspberry Pi (Headless-setup) 27 | 28 | You can also follow along this [YouTube video](https://www.youtube.com/watch?v=9fEnvDgxwbI) instead. 29 | 30 | 1. Connect the SD card to your laptop 31 | 2. Download Raspberry Pi OS (bootloader): [https://www.raspberrypi.com/software/](https://www.raspberrypi.com/software/) 32 | 3. Run it, and you should see: 33 | ![screenshot1.png](./images/screenshot1.png) 34 | - "Choose Device" - choose Raspberry Pi 5 35 | - OS, choose the latest (64bit is the recommended) 36 | - "Choose Storage" - choose the inserted SD card 37 | 4. Now click next, and it will ask you if you want to edit the settings, click "Edit settings" 38 | ![screenshot2.png](./images/screenshot2.png) 39 | 5. Configure 40 | ![screenshot3.png](./images/screenshot3.png) 41 | - enable hostname and set it to `raspberrypi`.local 42 | - Set username and password you will remember, we will use them shortly 43 | - Enable "Configure Wireless LAN" and add your wifi name and password 44 | - Click save, and contiue. it will take a few minutes to write everything to the SD 45 | 6. Insert the SD card to your raspberry pi, and connect it to the electricity 46 | 7. SSH into the Raspberry PI: 47 | 48 | ```bash 49 | ssh ssh @raspberrypi.local 50 | ``` 51 | 52 | ### Option 1: Run LLMs using Ollama 53 | 54 | 1. Install Ollama: 55 | 56 | ```bash 57 | curl -fsSL https://ollama.com/install.sh | sh 58 | ``` 59 | 60 | 2. Download & Run Mistral model: 61 | 62 | ```bash 63 | ollama run mistral 64 | ``` 65 | 66 | That is it! 67 | 68 | ### Option 2: Run LLMs using llama.cpp: 69 | 70 | 1. Install: 71 | 72 | ```bash 73 | sudo apt update && sudo apt install git g++ wget build-essential 74 | ``` 75 | 76 | 2. Download llama.cpp repo: 77 | 78 | ```bash 79 | git clone https://github.com/ggerganov/llama.cpp 80 | cd llama.cpp 81 | ``` 82 | 83 | 3. Compile: 84 | 85 | ```bash 86 | make -j 87 | ``` 88 | 89 | 4. Download Mistral model: 90 | 91 | ```bash 92 | cd models 93 | wget https://huggingface.co/TheBloke/Mistral-7B-v0.1-GGUF/resolve/main/mistral-7b-v0.1.Q4_K_S.gguf) 94 | ``` 95 | 96 | 5. Go back to repo root folder, and run: 97 | 98 | ```bash 99 | cd .. 100 | ./main -m models/mistral-7b-v0.1.Q4_K_S.gguf -p "Whatsup?" -n 400 -e 101 | ``` 102 | 103 | That is it! 104 | 105 | ### Extra Resoucres: 106 | 107 | - [Youtube video on how to run Ollama on Raspberry Pi](https://www.youtube.com/watch?v=ewXANEIC8pY) 108 | - [Llama.cpp own example](https://github.com/garyexplains/examples/blob/master/how-to-run-llama-cpp-on-raspberry-pi.md) 109 | - [Youtube video on how to setup Raspberry PI headlessly](https://www.youtube.com/watch?v=9fEnvDgxwbI) 110 | - [Adam on Twitter](https://twitter.com/adamcohenhillel) 111 | --------------------------------------------------------------------------------