├── .DS_Store ├── .gitignore ├── LICENSE ├── README.md ├── app ├── app.py ├── config.json ├── dockerfile ├── ollama.py ├── requirements.txt ├── static │ ├── css │ │ └── style.css │ ├── favicon.ico │ └── js │ │ └── main.js └── templates │ └── index.html └── assets ├── config_demo.gif ├── demo.mp4 ├── demo.png └── run_server.gif /.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dublit-Development/ollama-api/f6b8bfcb8e1ce17ee9321c923876b94ed230716e/.DS_Store -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | /pycache/ 2 | .DS_store -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2023 Dublit 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Ollama API: A UI and Backend Server to interact with Ollama and Stable Diffusion 2 | Ollama is a fantastic software that allows you to get up and running open-source LLM models quickly alongside with Stable Diffusion this repository is the quickest way to chat with multiple LLMs, generate images and perform VLM analysis. The complied code will deploy a Flask Server on your choice of hardware. 3 | 4 | It would be great if you can support the project and give it a ⭐️. 5 | 6 | ![demo](/assets/demo.png) 7 | 8 | ## Roadmap 9 | - **Apache Support**: We plan to support a production service API using WSGI 10 | - **Restful Support** Creating a quick RESTful deployment to query your favorite models with ease 11 | - **Docker** Ensure deployment is seamless and simple using docker 12 | - **API Route Documentation** Documentation to create your own interfaces or interactions with the backend service 13 | 14 | ## How to Run 15 | 1. Complete all the prequisite steps 16 | 2. Run the program `python3 app.py` 17 | 18 | ![run](./assets/run_server.gif) 19 | 20 | 21 | #### Hardware Specs 22 | Ensure that you have a machine with the following Hardware Specifications: 23 | 1. Ubuntu Linux or Macintosh (Windows is not supported) 24 | 2. 32 GB of RAM 25 | 3. 6 CPUs/vCPUs 26 | 4. 50 GB of Storage 27 | 5. NVIDIA GPU 28 | 29 | #### Prerequisites 30 | 1. In order to run Ollama including Stable Diffusion models you must create a read-only HuggingFace API key. [Creation of API Key](https://huggingface.co/docs/hub/security-tokens) 31 | 2. Upon completion of generating an API Key you need to edit the config.json located in the `./app/config.json` 32 | 33 | ![config](./assets/config_demo.gif) 34 | 3. Install neccessary dependencies and requirements: 35 | 36 | ```sh 37 | # Update your machine (Linux Only) 38 | sudo apt-get update 39 | # Install pip 40 | sudo apt-get install python3-pip 41 | # Navigate to the directory containing requirements.txt 42 | ./app 43 | # Run the pip install command 44 | pip install -r requirements.txt 45 | 46 | # Enable port 5000 (ufw) 47 | sudo ufw allow 5000 48 | sudo ufw status 49 | 50 | # CUDA update drivers 51 | sudo apt-get install -y cuda-drivers 52 | ``` 53 | 54 | ### Front End Features 55 | 56 | - **Dynamic Model Selection**: Users can select from a range of installed language models to interact with. 57 | - **Installation Management**: Users can install or uninstall models by dragging them between lists. 58 | - **Chat Interface**: Interactive chat area for users to communicate with the chosen language model. 59 | - **Support for Text-to-Image Generation**: It includes a feature to send requests to a Stable Diffusion endpoint for text-to-image creation. 60 | - **Image Uploads for LLaVA**: Allows image uploads when interacting with the LLaVA model. 61 | 62 | ### Frontend 63 | 64 | - **HTML**: `templates/index.html` provides the structure of the chat interface and model management area. 65 | - **JavaScript**: `static/js/script.js` contains all the interactive logic, including event listeners, fetch requests, and functions for managing models. 66 | - **CSS**: `static/css/style.css` presents the styling for the web interface. 67 | 68 | ### Proxy-Backend 69 | 70 | - **Python with Flask**: `main.py` acts as the server, handling the various API endpoints, requests to the VALDI endpoint, and serving the frontend files. While python, this is more of a frontend file than backend; similar to cloud functions on firebase. It functions as a serverless backend endpoint, but is a proxy to your real backend 71 | 72 | ### API Endpoints 73 | This directly interacts with the Backend Server hosted on VALDI. 74 | 75 | - `/`: Serves the main chat interface. 76 | - `/api/chat`: Handles chat messages sent to different language models. 77 | - `/api/llava`: Specialized chat handler for the LLaVA model that includes image data. 78 | - `/txt2img`: Endpoint for handling text-to-image generation requests. 79 | - `/list-models`: Returns the list of available models installed on the server. 80 | - `/install-model`: Installs a given model. 81 | - `/uninstall-model`: Uninstalls a given model. 82 | - `/install`: Endpoint used for initial setup, installing necessary components. 83 | 84 | ## Credits ✨ 85 | This project would not be possible without continous contributions from the Open Source Community. 86 | ### Ollama 87 | [Ollama Github](https://github.com/jmorganca/ollama) 88 | 89 | [Ollama Website](https://ollama.ai/) 90 | 91 | ### @cantrell 92 | [Cantrell Github](https://github.com/cantrell) 93 | 94 | [Stable Diffusion API Server](https://github.com/cantrell/stable-diffusion-api-server) 95 | 96 | ### Valdi 97 | Our preferred HPC partner 🖥️ 98 | 99 | [Valdi](https://valdi.ai/) 100 | 101 | [Support us](https://valdi.ai/signup?ref=YZl7RDQZ) 102 | 103 | ### Replit 104 | Our preferred IDE and deployment platform 🚀 105 | 106 | [Replit](https://replit.com/) 107 | 108 | ---- 109 | Created by [Dublit](https://dublit.org/) - Delivering Ollama to the masses 110 | 111 | ## Troubleshooting 112 | Our prefered HPC provider is Valdi. We access machine's securly by generating a privte and public key ssh file. You will need to ensure the permissions are correct before accessing any machine. 113 | ```sh 114 | chmod 600 Ojama.pem 115 | ``` 116 | ### Python versions 117 | We support python versions 3.8 and above, however code can run more efficiently on most stable versions of python such as 3.10 or 3.11. Here is a helpful guide as to how your python version can be updated. 118 | 119 | https://cloudbytes.dev/snippets/upgrade-python-to-latest-version-on-ubuntu-linux -------------------------------------------------------------------------------- /app/app.py: -------------------------------------------------------------------------------- 1 | from flask import Flask, request, jsonify, render_template 2 | import requests, random 3 | import subprocess, sys 4 | import json 5 | import re 6 | import sys 7 | import base64 8 | from PIL import Image 9 | from io import BytesIO 10 | import torch 11 | import diffusers 12 | import os 13 | 14 | 15 | app = Flask(__name__) 16 | 17 | VALDI_ENDPOINT = 'http://localhost:5000' 18 | 19 | ''' 20 | made with <3 by 21 | .o8 .o8 oooo o8o . 22 | "888 "888 `888 `"' .o8 23 | .oooo888 oooo oooo 888oooo. 888 oooo .o888oo 24 | d88' `888 `888 `888 d88' `88b 888 `888 888 25 | 888 888 888 888 888 888 888 888 888 26 | 888 888 888 888 888 888 888 888 888 . 27 | `Y8bod88P" `V88V"V8P' `Y8bod8P' o888o o888o "888" 28 | 29 | below is a mix of Flask API functionality and Stable Diffusion model installations 30 | 31 | The Stable Diffusion installations will check for what isinstalled on your machine, and install the 32 | requirements that are missing. After the first boot, the installer will only connect and load 33 | the engines into memory and not install anything new. Roughly 6GB of disk space needed 34 | to download all of the necessary packages for the models 35 | 36 | 37 | Massive thank you to @cantrell on GitHub for their open source contributions related 38 | to Stable Diffusion 39 | ''' 40 | OLLAMA_INSTALL = False 41 | 42 | # ------------> BACKEND FUNCTIONS <------- 43 | @app.route('/') 44 | def index(): 45 | # Render the index.html template 46 | return render_template('index.html') 47 | 48 | @app.route('/api/question', methods=['POST']) 49 | def process_question(): 50 | # Get the question from the request 51 | data = request.get_json() 52 | question = data.get('question', '') 53 | model = data.get('model', '') 54 | 55 | # Run a command and capture the output 56 | result = run_model_question(question, model) 57 | print(result) 58 | 59 | # Return the result as JSON 60 | return jsonify({"message":result}) 61 | 62 | @app.route('/api/vlm', methods=['POST']) 63 | def vlm_model(): 64 | data = request.get_json() 65 | model = data.get('model', '') 66 | prompt = data.get('prompt', '') 67 | image = data.get('image', '') 68 | 69 | result = run_vlm_question(model, prompt, image) 70 | 71 | # Return the result as a JSON 72 | return jsonify({"message":result}) 73 | 74 | @app.route('/api/pull', methods=['POST']) 75 | def pull_model(): 76 | data = request.get_json() 77 | model = data.get('model', '') 78 | 79 | result = run_pull_model(model) 80 | print(result) 81 | 82 | return jsonify({"message":result}) 83 | 84 | # change to post if it doesnt work. some intermediaries block json body when attatched to a DELETE 85 | @app.route('/api/delete', methods=['DELETE']) 86 | def delete_model(): 87 | data = request.get_json() 88 | model = data.get('model', '') 89 | 90 | result = run_delete_model(model) 91 | print(result) 92 | 93 | return jsonify({"message":result}) 94 | 95 | @app.route('/api/install', methods=['GET']) 96 | def install(): 97 | global OLLAMA_INSTALL 98 | 99 | if not OLLAMA_INSTALL: 100 | response = install_ollama() 101 | OLLAMA_INSTALL = True 102 | return jsonify({'message': response}) 103 | else: 104 | return jsonify({'message': 'OLLAMA_INSTALL is already set to True'}) 105 | 106 | @app.route('/api/list-models', methods=['GET']) 107 | def listModels(): 108 | res = listInstalledModels() 109 | return jsonify({'models':res}) 110 | 111 | ###### HELPER FUNCTIONS ###### 112 | def listInstalledModels(): 113 | curl_command = f'curl http://localhost:11434/api/tags' 114 | 115 | output = subprocess.check_output(curl_command, shell=True, encoding='utf-8') 116 | res = json.loads(output) 117 | 118 | return res 119 | 120 | def run_delete_model(model): 121 | # Define the curl command 122 | curl_command = f'curl -X DELETE http://localhost:11434/api/delete -d \'{{"name": "{model}"}}\'' 123 | 124 | output = subprocess.check_output(curl_command, shell=True, encoding='utf-8') 125 | 126 | response = json.loads(output) 127 | return response 128 | 129 | def run_pull_model(model): 130 | # Define the curl command 131 | curl_command = f'curl http://localhost:11434/api/pull -d \'{{"name": "{model}", "stream": false}}\'' 132 | 133 | # Run the command and capture the output 134 | output = subprocess.check_output(curl_command, shell=True, encoding='utf-8') 135 | 136 | response = json.loads(output) 137 | return response 138 | 139 | def run_vlm_question(model, prompt, image): 140 | # Define the curl command 141 | curl_command = f'curl http://localhost:11434/api/generate -d \'{{"model": "{model}", "prompt": "{prompt}", "stream": false, "images": ["{image[0]}"]}}\'' 142 | 143 | # Run the command and capture the output 144 | output = subprocess.check_output(curl_command, shell=True, encoding='utf-8') 145 | print(output) 146 | # Parse the JSON string in the output variable 147 | output_json = json.loads(output) 148 | print(output_json) 149 | 150 | # Extract the "response" value 151 | responses = output_json.get("response", None) 152 | 153 | # Create a JSON containing only "response" values 154 | response_json = {'responses': responses} 155 | 156 | return response_json 157 | 158 | def run_model_question(question, model): 159 | # Define the curl command 160 | curl_command = f'curl http://localhost:11434/api/generate -d \'{{"model": "{model}", "prompt": "{question}"}}\'' 161 | 162 | # Run the command and capture the output 163 | output = subprocess.check_output(curl_command, shell=True, encoding='utf-8') 164 | 165 | # Process the output as JSON and extract "response" values 166 | responses = [json.loads(response)["response"] for response in output.strip().split('\n')] 167 | 168 | # Create a JSON containing only "response" values 169 | response_json = {'responses': responses} 170 | 171 | return response_json 172 | 173 | def working_directory(): 174 | 175 | # Get the current working directory 176 | current_directory = os.getcwd() 177 | 178 | # Define the file name you are looking for 179 | file_to_find = "config.json" 180 | 181 | # Contruct the full path to the file 182 | file_path = os.path.join(current_directory, file_to_find) 183 | 184 | return file_path 185 | 186 | def install_ollama(): 187 | try: 188 | curl_command = 'curl https://ollama.ai/install.sh | sh' 189 | # Use curl to install latest Ollama package 190 | subprocess.check_call(curl_command, shell=True, encoding='utf-8') 191 | 192 | return "Success" 193 | 194 | except subprocess.CalledProcessError as e: 195 | # print(f"Error downloading Ollama: {e}") 196 | return e 197 | 198 | ################################################## 199 | # Utils 200 | 201 | def retrieve_param(key, data, cast, default): 202 | if key in data: 203 | value = request.form[ key ] 204 | value = cast( value ) 205 | return value 206 | return default 207 | 208 | def pil_to_b64(input): 209 | buffer = BytesIO() 210 | input.save( buffer, 'PNG' ) 211 | output = base64.b64encode( buffer.getvalue() ).decode( 'utf-8' ).replace( '\n', '' ) 212 | buffer.close() 213 | return output 214 | 215 | def b64_to_pil(input): 216 | output = Image.open( BytesIO( base64.b64decode( input ) ) ) 217 | return output 218 | 219 | def get_compute_platform(context): 220 | try: 221 | import torch 222 | if torch.cuda.is_available(): 223 | return 'cuda' 224 | elif torch.backends.mps.is_available() and context == 'engine': 225 | return 'mps' 226 | else: 227 | return 'cpu' 228 | except ImportError: 229 | return 'cpu' 230 | 231 | ################################################## 232 | # Engines 233 | 234 | class Engine(object): 235 | def __init__(self): 236 | pass 237 | 238 | def process(self, kwargs): 239 | return [] 240 | 241 | class EngineStableDiffusion(Engine): 242 | def __init__(self, pipe, sibling=None, custom_model_path=None, requires_safety_checker=True): 243 | super().__init__() 244 | if sibling == None: 245 | self.engine = pipe.from_pretrained( 'runwayml/stable-diffusion-v1-5', use_auth_token=hf_token.strip() ) 246 | elif custom_model_path: 247 | if requires_safety_checker: 248 | self.engine = diffusers.StableDiffusionPipeline.from_pretrained(custom_model_path, 249 | safety_checker=sibling.engine.safety_checker, 250 | feature_extractor=sibling.engine.feature_extractor) 251 | else: 252 | self.engine = diffusers.StableDiffusionPipeline.from_pretrained(custom_model_path, 253 | feature_extractor=sibling.engine.feature_extractor) 254 | else: 255 | self.engine = pipe( 256 | vae=sibling.engine.vae, 257 | text_encoder=sibling.engine.text_encoder, 258 | tokenizer=sibling.engine.tokenizer, 259 | unet=sibling.engine.unet, 260 | scheduler=sibling.engine.scheduler, 261 | safety_checker=sibling.engine.safety_checker, 262 | feature_extractor=sibling.engine.feature_extractor 263 | ) 264 | self.engine.to( get_compute_platform('engine') ) 265 | 266 | def process(self, kwargs): 267 | output = self.engine( **kwargs ) 268 | return {'image': output.images[0], 'nsfw':output.nsfw_content_detected[0]} 269 | 270 | class EngineManager(object): 271 | def __init__(self): 272 | self.engines = {} 273 | 274 | def has_engine(self, name): 275 | return ( name in self.engines ) 276 | 277 | def add_engine(self, name, engine): 278 | if self.has_engine( name ): 279 | return False 280 | self.engines[ name ] = engine 281 | return True 282 | 283 | def get_engine(self, name): 284 | if not self.has_engine( name ): 285 | return None 286 | engine = self.engines[ name ] 287 | return engine 288 | 289 | ################################################## 290 | # App 291 | 292 | # Load and parse the config file: 293 | try: 294 | config_file = open(working_directory(), 'r') 295 | except: 296 | sys.exit('config.json not found.') 297 | 298 | config = json.loads(config_file.read()) 299 | 300 | hf_token = config['hf_token'] 301 | 302 | if (hf_token == None): 303 | sys.exit('No Hugging Face token found in config.json.') 304 | 305 | custom_models = config['custom_models'] if 'custom_models' in config else [] 306 | 307 | # Initialize engine manager: 308 | manager = EngineManager() 309 | 310 | # Add supported engines to manager: 311 | manager.add_engine( 'txt2img', EngineStableDiffusion( diffusers.StableDiffusionPipeline, sibling=None ) ) 312 | manager.add_engine( 'img2img', EngineStableDiffusion( diffusers.StableDiffusionImg2ImgPipeline, sibling=manager.get_engine( 'txt2img' ) ) ) 313 | manager.add_engine( 'masking', EngineStableDiffusion( diffusers.StableDiffusionInpaintPipeline, sibling=manager.get_engine( 'txt2img' ) ) ) 314 | for custom_model in custom_models: 315 | manager.add_engine( custom_model['url_path'], 316 | EngineStableDiffusion( diffusers.StableDiffusionPipeline, sibling=manager.get_engine( 'txt2img' ), 317 | custom_model_path=custom_model['model_path'], 318 | requires_safety_checker=custom_model['requires_safety_checker'] ) ) 319 | 320 | # Define routes: 321 | @app.route('/ping', methods=['GET']) 322 | def stable_ping(): 323 | return jsonify( {'status':'success'} ) 324 | 325 | @app.route('/custom_models', methods=['GET']) 326 | def stable_custom_models(): 327 | if custom_models == None: 328 | return jsonify( [] ) 329 | else: 330 | return custom_models 331 | 332 | @app.route('/txt2img', methods=['POST','GET']) 333 | def stable_txt2img(): 334 | return _generate('txt2img') 335 | 336 | @app.route('/img2img', methods=['POST']) 337 | def stable_img2img(): 338 | return _generate('img2img') 339 | 340 | @app.route('/masking', methods=['POST']) 341 | def stable_masking(): 342 | return _generate('masking') 343 | 344 | @app.route('/custom/', methods=['POST']) 345 | def stable_custom(model): 346 | return _generate('txt2img', model) 347 | 348 | def _generate(task, engine=None): 349 | # Retrieve engine: 350 | if engine == None: 351 | engine = task 352 | 353 | engine = manager.get_engine( engine ) 354 | 355 | # Prepare output container: 356 | output_data = {} 357 | 358 | # Handle request: 359 | try: 360 | seed = retrieve_param( 'seed', request.form, int, 0 ) 361 | count = retrieve_param( 'num_outputs', request.form, int, 1 ) 362 | total_results = [] 363 | for i in range( count ): 364 | if (seed == 0): 365 | generator = torch.Generator( device=get_compute_platform('generator') ) 366 | else: 367 | generator = torch.Generator( device=get_compute_platform('generator') ).manual_seed( seed ) 368 | new_seed = generator.seed() 369 | prompt = request.get_json(force=True).get('prompt') 370 | args_dict = { 371 | 'prompt' : [ prompt ], 372 | 'num_inference_steps' : retrieve_param( 'num_inference_steps', request.form, int, 100 ), 373 | 'guidance_scale' : retrieve_param( 'guidance_scale', request.form, float, 7.5 ), 374 | 'eta' : retrieve_param( 'eta', request.form, float, 0.0 ), 375 | 'generator' : generator 376 | } 377 | if (task == 'txt2img'): 378 | args_dict[ 'width' ] = retrieve_param( 'width', request.form, int, 512 ) 379 | args_dict[ 'height' ] = retrieve_param( 'height', request.form, int, 512 ) 380 | if (task == 'img2img' or task == 'masking'): 381 | init_img_b64 = request.form[ 'init_image' ] 382 | init_img_b64 = re.sub( '^data:image/png;base64,', '', init_img_b64 ) 383 | init_img_pil = b64_to_pil( init_img_b64 ) 384 | args_dict[ 'init_image' ] = init_img_pil 385 | args_dict[ 'strength' ] = retrieve_param( 'strength', request.form, float, 0.7 ) 386 | if (task == 'masking'): 387 | mask_img_b64 = request.form[ 'mask_image' ] 388 | mask_img_b64 = re.sub( '^data:image/png;base64,', '', mask_img_b64 ) 389 | mask_img_pil = b64_to_pil( mask_img_b64 ) 390 | args_dict[ 'mask_image' ] = mask_img_pil 391 | # Perform inference: 392 | pipeline_output = engine.process( args_dict ) 393 | pipeline_output[ 'seed' ] = new_seed 394 | total_results.append( pipeline_output ) 395 | # Prepare response 396 | output_data[ 'status' ] = 'success' 397 | images = [] 398 | for result in total_results: 399 | images.append({ 400 | 'base64' : pil_to_b64( result['image'].convert( 'RGB' ) ), 401 | 'seed' : result['seed'], 402 | 'mime_type': 'image/png', 403 | 'nsfw': result['nsfw'] 404 | }) 405 | output_data[ 'images' ] = images 406 | except RuntimeError as e: 407 | output_data[ 'status' ] = 'failure' 408 | output_data[ 'message' ] = 'A RuntimeError occurred. You probably ran out of GPU memory. Check the server logs for more details.' 409 | print(str(e)) 410 | return jsonify( output_data ) 411 | 412 | 413 | # ------------> FORONTEND INTERACTIONS <------------ 414 | # HANDLE A BASIC CHAT REQUEST 415 | @app.route('/api/chat', methods=['POST']) 416 | def chat(): 417 | # Error handling for not receiving JSON 418 | if not request.is_json: 419 | return jsonify({"error": "Missing JSON in request"}), 400 420 | 421 | data = request.get_json() 422 | 423 | # Error handling for missing model or message in the JSON 424 | if 'model' not in data or 'message' not in data: 425 | return jsonify({"error": "Missing 'model' or 'message' in JSON request"}), 400 426 | 427 | model = data['model'] 428 | message = data['message'] 429 | 430 | response = process_model_request(model=model, message=message) 431 | 432 | if 'error' in response: 433 | # Error responses will be a tuple with (response, status_code) 434 | return jsonify(response[0]), response[1] 435 | 436 | return response, 200 437 | 438 | # HANDLE A BASIC CHAT REQUEST 439 | @app.route('/api/llava', methods=['POST']) 440 | def llavaChat(): 441 | # Error handling for not receiving JSON 442 | if not request.is_json: 443 | return jsonify({"error": "Missing JSON in request"}), 400 444 | 445 | data = request.get_json() 446 | 447 | # Error handling for missing model or message in the JSON 448 | if 'model' not in data or 'message' not in data: 449 | return jsonify({"error": "Missing 'model' or 'message' in JSON request"}), 400 450 | 451 | model = data['model'] 452 | message = data['message'] 453 | image = data['image'] 454 | 455 | try: 456 | response = requests.post( 457 | url=f"{VALDI_ENDPOINT}/api/vlm", 458 | headers={'Content-Type': 'application/json'}, 459 | data=json.dumps({'prompt':message, 'model':model, 'image': [image]}) 460 | ) 461 | response.raise_for_status() 462 | return response.json() 463 | except requests.RequestException as e: 464 | return {"error": str(e)}, 500 465 | 466 | return response, 200 467 | 468 | # PROCESS THAT ACTS AS A PROXY TO CHAT REQUESTS 469 | def process_model_request(model, message): 470 | def post_to_valdi(model, message): 471 | try: 472 | response = requests.post( 473 | url=f"{VALDI_ENDPOINT}/api/question", 474 | headers={'Content-Type': 'application/json'}, 475 | data=json.dumps({'question': message, 'model': model}) 476 | ) 477 | response.raise_for_status() 478 | return response.json() 479 | except requests.RequestException as e: 480 | return {"error": str(e)}, 500 481 | 482 | if model == 'llama2': 483 | return post_to_valdi('llama2', message) 484 | elif model == 'mistral': 485 | return post_to_valdi('mistral', message) 486 | elif model == 'vlm': 487 | return post_to_valdi('vlm', message) 488 | else: 489 | try: 490 | response = requests.post( 491 | url=f"{VALDI_ENDPOINT}/api/question", 492 | headers={'Content-Type': 'application/json'}, 493 | data=json.dumps({'question': message, 'model': model}) 494 | ) 495 | response.raise_for_status() 496 | return response.json() 497 | except Exception as e: 498 | return {"error": f"Model '{model}' is unsupported, {e}"}, 404 499 | 500 | 501 | # HANDLE A REQUEST TO THE STABLE DIFFUSION ENDPOINT 502 | @app.route('/txt2img-trigger', methods=['POST']) 503 | def txt2img_route(): 504 | try: 505 | request_data = request.get_json() 506 | 507 | prompt = request_data.get('prompt', '') 508 | seed = random.randint(1,1000000) 509 | outputs = request_data.get('num_outputs', 1) 510 | width = request_data.get('width', 512) 511 | height = request_data.get('height', 512) 512 | steps = request_data.get('num_inference_steps', 10) 513 | guidance_scale = request_data.get('guidance_scale', 0.5) 514 | 515 | url = VALDI_ENDPOINT + '/txt2img' 516 | 517 | request_body = { 518 | "prompt": prompt, 519 | "seed": seed, 520 | "num_outputs": outputs, 521 | "width": width, 522 | "height": height, 523 | "num_inference_steps": steps, 524 | "guidance_scale": guidance_scale 525 | } 526 | 527 | print(request_body) 528 | 529 | headers = { 530 | 'Content-Type': 'application/json', 531 | # Add any other headers if needed 532 | } 533 | 534 | 535 | response = requests.post(url, headers=headers, data=json.dumps(request_body)) 536 | response_data = response.json() 537 | 538 | if response_data['status'] == 'success': 539 | image_data = response_data['images'][0] 540 | image_bytes = BytesIO(base64.b64decode(image_data['base64'])) 541 | img = Image.open(image_bytes) 542 | img_bytes = BytesIO() 543 | img.save(img_bytes, format='PNG') 544 | img_data = img_bytes.getvalue() 545 | 546 | return jsonify({ 547 | 'status': 'success', 548 | 'message': 'Image data retrieved successfully', 549 | 'image_data': base64.b64encode(img_data).decode('utf-8') 550 | }) 551 | else: 552 | return jsonify({'error': f"Request failed: {response_data['message']}"}), 500 553 | except Exception as error: 554 | return jsonify({'error': f"Error: {error}"}), 500 555 | 556 | # LIST ALL OF THE OLLAMA MODELS ON THE MACHINE 557 | @app.route('/list-models', methods=['GET']) 558 | def list_models_route(): 559 | try: 560 | url = VALDI_ENDPOINT + '/api/list-models' 561 | response = requests.get(url) 562 | response_data = response.json() 563 | 564 | return jsonify({ 565 | 'status': 'success', 566 | 'message': 'Models retrieved successfully', 567 | 'models': response_data['models'] 568 | }) 569 | except Exception as error: 570 | return jsonify({'error': f"Error: {error}"}), 500 571 | 572 | 573 | # Flask route to install a model 574 | @app.route('/install-model', methods=['POST']) 575 | def install_model(): 576 | try: 577 | data = request.get_json() 578 | model_name = data.get('model_name') 579 | if not model_name: 580 | return jsonify({'error': 'Model name is required'}), 400 581 | install_url = f"{VALDI_ENDPOINT}/api/pull" 582 | response = requests.post(install_url, json={'model': model_name}) 583 | result = response.json().get('message') 584 | return result 585 | except Exception as error: 586 | return jsonify({'error': str(error)}), 500 587 | 588 | # Flask route to uninstall a model 589 | @app.route('/uninstall-model', methods=['POST']) 590 | def uninstall_model(): 591 | try: 592 | data = request.get_json() 593 | model_name = data.get('model_name') 594 | if not model_name: 595 | return jsonify({'error': 'Model name is required'}), 400 596 | uninstall_url = f"{VALDI_ENDPOINT}/api/delete" 597 | response = requests.delete(uninstall_url, json={'model': model_name}) 598 | result = response.json().get('message') 599 | if(result == None): 600 | result = "Model successfully uninstalled" 601 | return jsonify({"message":result}) 602 | except Exception as error: 603 | return jsonify({'error': str(error)}), 500 604 | 605 | # install ollama 606 | @app.route('/install', methods=['GET']) 607 | def install_get(): 608 | try: 609 | install_url = f"{VALDI_ENDPOINT}/api/install" 610 | response = requests.get(install_url) 611 | result = response.json().get('message') 612 | return result 613 | except Exception as error: 614 | return jsonify({'error': str(error)}), 500 615 | 616 | 617 | # DENY GET REQUESTS TO THE CHAT ENDPOINT 618 | @app.route('/api/chat', methods=['GET']) 619 | def get_chat(): 620 | return jsonify({"message": "GET method is not supported for /api/chat"}), 405 621 | 622 | 623 | 624 | 625 | def run_api(): 626 | app.run(host='0.0.0.0', port=5000, debug=True, use_reloader=False) 627 | 628 | run_api() -------------------------------------------------------------------------------- /app/config.json: -------------------------------------------------------------------------------- 1 | { 2 | "hf_token":"HF-TOKEN" 3 | } -------------------------------------------------------------------------------- /app/dockerfile: -------------------------------------------------------------------------------- 1 | # Use an official Python runtime as a parent image 2 | FROM python:3.8 3 | 4 | # Set the working directory to /app 5 | WORKDIR /app 6 | 7 | # Copy the current directory contents into the container at /app 8 | COPY . /app 9 | 10 | # Set default environment variable values during build 11 | ARG MY_ENV_VAR_DEFAULT=value_from_dockerfile 12 | ENV MY_ENV_VAR=$MY_ENV_VAR_DEFAULT 13 | 14 | # Install Python dependencies 15 | RUN pip install --no-cache-dir -r requirements.txt 16 | 17 | # Run a script during build to update config.json 18 | RUN echo '{"hf_token": "'"$MY_ENV_VAR"'"}' > config.json 19 | 20 | # Run app.py when the container launches 21 | CMD ["python", "app.py"] 22 | -------------------------------------------------------------------------------- /app/ollama.py: -------------------------------------------------------------------------------- 1 | import subprocess, shlex 2 | import json 3 | 4 | ### This code is the automation and foundation for the support of multi model responses. The code procures a question with content (chat history) and runs it through multiple models installed. 5 | def listInstalledModels(): 6 | curl_command = f'curl http://localhost:11434/api/tags' 7 | 8 | output = subprocess.check_output(curl_command, shell=True, encoding='utf-8') 9 | res = json.loads(output) 10 | 11 | # Extract only the 'name' attribute and remove ':latest' 12 | model_names = [model.get('name', '').replace(':latest', '') for model in res.get('models', [])] 13 | 14 | return model_names 15 | 16 | def listModels(): 17 | model_names = listInstalledModels() 18 | return {'model_names': model_names} 19 | 20 | # Now you can print the result or do whatever you want with it 21 | result = listModels() 22 | print(result) 23 | 24 | 25 | def run_model_generate(question, content): 26 | # Get the list of installed models (replace listInstalledModels with the correct function) 27 | model_names = listInstalledModels() 28 | 29 | # Initialize a dictionary to store responses for each model 30 | all_responses = {} 31 | 32 | for model in model_names: 33 | # Use shlex.quote for question and context to handle special characters 34 | quoted_question = shlex.quote(question) 35 | quoted_content = shlex.quote(content) 36 | 37 | # Define the data payload as a dictionary 38 | data_payload = { 39 | "model": model, 40 | "prompt": quoted_question, 41 | "content": quoted_content 42 | } 43 | 44 | # Convert the data payload to a JSON string 45 | json_data = json.dumps(data_payload) 46 | 47 | # Run the command and capture the output 48 | process = subprocess.Popen(['curl', 'http://localhost:11434/api/chat', '-d', json_data], 49 | stdout=subprocess.PIPE, stderr=subprocess.PIPE) 50 | output, error = process.communicate() 51 | 52 | # Decode the output from bytes to UTF-8 string 53 | output_str = output.decode('utf-8') 54 | 55 | # Print the output for debugging 56 | print("Raw Output:", output_str) 57 | 58 | # Check for errors 59 | if process.returncode != 0: 60 | print(f"Error running command. Error message: {error.decode('utf-8')}") 61 | return # or exit the function, depending on your requirements 62 | 63 | # Process the output as JSON and extract "response" values 64 | try: 65 | responses = [json.loads(response)["response"] for response in output_str.strip().split('\n')] 66 | except json.JSONDecodeError as e: 67 | print(f"Error decoding JSON response. Error message: {e}") 68 | return # or exit the function, depending on your requirements 69 | 70 | # Add the responses to the dictionary for all models 71 | all_responses[model] = responses 72 | 73 | return all_responses 74 | 75 | def run_model_chat(question, content): 76 | # Replace listInstalledModels with the correct function to get the model names 77 | model_names = listInstalledModels() 78 | 79 | # Initialize a dictionary to store responses for each model 80 | all_responses = {} 81 | 82 | for model in model_names: 83 | # Use shlex.quote for question and content to handle special characters 84 | quoted_question = shlex.quote(question) 85 | quoted_content = shlex.quote(content) 86 | 87 | # Define the data payload as a dictionary 88 | data_payload = { 89 | "model": model, 90 | "messages": [ 91 | {"role": "user", "content": quoted_question}, 92 | {"role": "assistant", "content": quoted_content} 93 | ], 94 | "stream": False 95 | } 96 | 97 | # Convert the data payload to a JSON string 98 | json_data = json.dumps(data_payload) 99 | 100 | # Run the command and capture the output 101 | process = subprocess.Popen(['curl', 'http://localhost:11434/api/chat', '-d', json_data], 102 | stdout=subprocess.PIPE, stderr=subprocess.PIPE) 103 | output, error = process.communicate() 104 | 105 | # Decode the output from bytes to UTF-8 string 106 | output_str = output.decode('utf-8') 107 | 108 | # Print the output for debugging 109 | # print("Raw Output:", output_str) 110 | 111 | # Check for errors 112 | if process.returncode != 0: 113 | print(f"Error running command. Error message: {error.decode('utf-8')}") 114 | return # or exit the function, depending on your requirements 115 | 116 | # Process the output as JSON 117 | try: 118 | response_json = json.loads(output_str) 119 | assistant_response = response_json.get('message', {}).get('content', '') 120 | all_responses[model] = assistant_response 121 | except json.JSONDecodeError as e: 122 | print(f"Error decoding JSON response. Error message: {e}") 123 | return # or exit the function, depending on your requirements 124 | 125 | return all_responses 126 | 127 | # Run the question for all installed models 128 | results = run_model_chat("Question", "Content") 129 | 130 | print(results) -------------------------------------------------------------------------------- /app/requirements.txt: -------------------------------------------------------------------------------- 1 | flask 2 | requests 3 | pillow 4 | torch 5 | diffusers 6 | transformers 7 | accelerate -------------------------------------------------------------------------------- /app/static/css/style.css: -------------------------------------------------------------------------------- 1 | @import url('https://fonts.googleapis.com/css2?family=Cherry+Bomb+One&family=Foldit:wght@600&family=IBM+Plex+Mono&family=Inter&family=Monoton&family=Nunito&family=Open+Sans&family=Plaster&family=Raleway:wght@300;500&family=Roboto&family=Roboto+Mono&display=swap'); 2 | 3 | body { 4 | font-family:system-ui, -apple-system,'Raleway', BlinkMacSystemFont, 'Segoe UI', Roboto, Oxy; 5 | margin:0 auto; 6 | padding:0; 7 | background-color:#1a1c47; 8 | color:#fffdfa; 9 | } 10 | 11 | /* default scrollbar */ 12 | ::-webkit-scrollbar { 13 | width: 10px; 14 | } 15 | 16 | ::-webkit-scrollbar-track { 17 | background: #00000000; 18 | border-radius: 8px; 19 | } 20 | 21 | ::-webkit-scrollbar-thumb { 22 | background: #888; 23 | border-radius: 8px; 24 | } 25 | 26 | ::-webkit-scrollbar-thumb:hover { 27 | background: #555; 28 | } 29 | /* scrollbar for divs */ 30 | div::-webkit-scrollbar { 31 | width: 10px; 32 | } 33 | 34 | div::-webkit-scrollbar-track { 35 | background: #00000000; 36 | border-radius: 8px; 37 | } 38 | 39 | div::-webkit-scrollbar-thumb { 40 | background: #888; 41 | border-radius: 8px; 42 | } 43 | 44 | div::-webkit-scrollbar-thumb:hover { 45 | background: #555; 46 | } 47 | 48 | h1, h3 { 49 | text-align:center; 50 | font-family: 'Raleway'; 51 | } 52 | 53 | #chat-area { 54 | height:80%; 55 | max-height:80%; 56 | width:99%; 57 | overflow-y:scroll; 58 | padding:8px; 59 | border-radius:8px; 60 | position:relative; 61 | display:flex; 62 | flex-direction:column; 63 | gap:8px; 64 | } 65 | 66 | #container { 67 | margin:0 auto; 68 | width:90%; 69 | height:70vh; 70 | background:linear-gradient(145deg, #4f4cdc50, #1d1b8850); 71 | padding:12px; 72 | border-radius:8px 24px 8px 8px; 73 | } 74 | 75 | #chat-inputs { 76 | width:100%; 77 | border-top:#cccccc90 1px solid; 78 | display:flex; 79 | flex-direction:row; 80 | gap:0px; 81 | height:10%; 82 | } 83 | #chat-inputs input { 84 | margin-top:12px; 85 | width:80%; 86 | padding:0px; 87 | border-radius:6px 0px 0px 6px; 88 | outline:none; 89 | border:1px solid #00000000; 90 | color:#fffdfa; 91 | background-color:#fffdfa20; 92 | transition: all ease-out 0.2s; 93 | font-size:1rem; 94 | } 95 | #chat-inputs input:hover { 96 | box-shadow:0px 0px 24px #22222280; 97 | } 98 | #chat-inputs input:focus { 99 | box-shadow:0px 0px 24px #22222280; 100 | } 101 | #chat-inputs button { 102 | margin-top:12px; 103 | width:20%; 104 | padding:0px; 105 | border-radius:0px 6px 6px 0px; 106 | outline:none; 107 | border:1px solid #00000000; 108 | cursor:pointer; 109 | background-color:#fffdfa50; 110 | font-size:1rem; 111 | } 112 | #chat-inputs button:hover { 113 | box-shadow:0px 0px 24px #22222280; 114 | } 115 | #chat-inputs button:focus { 116 | box-shadow:0px 0px 24px #22222280; 117 | } 118 | 119 | #model-select { 120 | width:200px; 121 | height:48px; 122 | border-radius:8px; 123 | position:relative; 124 | cursor:pointer; 125 | background-color:#00000000; 126 | border:1px solid #fffdfa; 127 | color:#fffdfa; 128 | } 129 | #model-select option { 130 | background-color:#1a1c47; 131 | color:#fffdfa; 132 | } 133 | 134 | .userMessage { 135 | position:relative; 136 | background-color: #007AFF; 137 | color: #fff; 138 | position:relative; 139 | border-radius: 18px 18px 4px 18px; 140 | max-width: 60%; 141 | padding: 10px 15px; 142 | margin: 10px; 143 | align-self: flex-end; 144 | word-wrap: break-word; 145 | left:10px; 146 | } 147 | 148 | .botMessage { 149 | position:relative; 150 | background-color: #00000000; 151 | border-radius: 18px 18px 18px 4px; 152 | max-width: 60%; 153 | padding: 10px 15px; 154 | margin: 10px; 155 | align-self: flex-start; 156 | word-wrap: break-word; 157 | right:10px; 158 | border:1px solid #cccccc; 159 | } 160 | 161 | .botMessage img { 162 | border-radius:6px; 163 | max-width:96%; 164 | } 165 | 166 | #controls { 167 | display:flex; 168 | flex-direction:row; 169 | width:90%; 170 | justify-content:space-between; 171 | border-radius:16px; 172 | background-color:#fffdfa30; 173 | backdrop-filter:blur(12px); 174 | -webkit-backdrop-filter:blur(12px); 175 | margin: 0 auto; 176 | padding:8px; 177 | max-height:250px; 178 | height:128px; 179 | transition: all ease-out 0.2s; 180 | overflow-y:auto; 181 | overflow-x:hidden; 182 | margin-bottom:8px; 183 | } 184 | #model-lists { 185 | display:flex; 186 | flex-direction:row; 187 | gap:12px; 188 | transition:all ease-out 0.2s; 189 | text-align:center; 190 | } 191 | .model-list { 192 | background-color: #fffdfa20; 193 | padding:4px; 194 | border-radius:3px 6px 12px 6px; 195 | font-size:12px; 196 | transition:all ease-out 0.2s; 197 | min-height:48px; 198 | min-width:82px; 199 | display:flex; 200 | flex-direction:column; 201 | gap:6px; 202 | } 203 | .list-wrapper h4 { 204 | padding:0px; 205 | margin:0px; 206 | transition:all ease-out 0.2s; 207 | } 208 | 209 | #menu_toggle { 210 | position:absolute; 211 | margin:0 auto; 212 | width:48px; 213 | left:48%; 214 | transform:rotate(180deg); 215 | fill:#fffdfa; 216 | margin-top:-32px; 217 | cursor:pointer; 218 | transition: all ease-in-out 0.2s; 219 | } 220 | #menu_toggle svg { 221 | width:100%; 222 | transition: all ease-in-out 0.2s; 223 | } 224 | #menu_toggle:hover { 225 | transform:translate(0px,-2px) rotate(180deg); 226 | } 227 | .model-installer { 228 | border:1px solid #fffdfa; 229 | border-radius:64px; 230 | text-align:center; 231 | padding:2px; 232 | cursor:pointer; 233 | } 234 | .model-installer:active { 235 | cursor:grab; 236 | } 237 | 238 | 239 | @keyframes pulse { 240 | 0% { 241 | opacity: 0.5; 242 | } 243 | 50% { 244 | opacity: 1; 245 | } 246 | 100% { 247 | opacity: 0.5; 248 | } 249 | } 250 | @keyframes rotate { 251 | 0% { 252 | transform: rotate(0deg) scale(0.95); 253 | } 254 | 50% { 255 | transform: (1.05); 256 | } 257 | 100% { 258 | transform: rotate(360deg) scale(0.95); 259 | } 260 | } 261 | .loaderAnimation { 262 | animation: rotate 2s infinite; 263 | } 264 | #svgPath { 265 | animation: pulse 2s infinite; 266 | } 267 | 268 | #loader svg{ 269 | height:100%; 270 | fill:#fffdfa; 271 | } 272 | #loader { 273 | display:none; 274 | height:64px; 275 | width:64px; 276 | } 277 | 278 | #llava-file-upload { 279 | display:none; 280 | } -------------------------------------------------------------------------------- /app/static/favicon.ico: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dublit-Development/ollama-api/f6b8bfcb8e1ce17ee9321c923876b94ed230716e/app/static/favicon.ico -------------------------------------------------------------------------------- /app/static/js/main.js: -------------------------------------------------------------------------------- 1 | 2 | 3 | /* 4 | .o8 .o8 oooo o8o . 5 | "888 "888 `888 `"' .o8 6 | .oooo888 oooo oooo 888oooo. 888 oooo .o888oo 7 | d88' `888 `888 `888 d88' `88b 888 `888 888 8 | 888 888 888 888 888 888 888 888 888 9 | 888 888 888 888 888 888 888 888 888 . 10 | `Y8bod88P" `V88V"V8P' `Y8bod8P' o888o o888o "888" 11 | */ 12 | 13 | 14 | 15 | document.addEventListener('DOMContentLoaded', async function() { 16 | var currb64Img = '' 17 | 18 | // install ollama on boot 19 | await fetch('/install', { 20 | method: 'GET', 21 | }).then(async(repsonse) => { 22 | if (repsonse.status == 200) { 23 | console.log('Stable Diffusion and Ollama are installed on your VALDI machine 🥳🎉') 24 | } 25 | 26 | }); 27 | 28 | let isProcessing = false; // flag to control dragging during install/uninstall 29 | 30 | // message sending 31 | var chatInput = document.getElementById('user-input'); 32 | var chatButton = document.getElementById('send-button'); 33 | var chatArea = document.getElementById('chat-area'); 34 | var modelSelect = document.getElementById('model-select'); 35 | 36 | // helper function to keep the chat area scrolled to the bottom 37 | function scrollToBottom() { 38 | chatArea.scroll({ 39 | top: chatArea.scrollHeight, 40 | behavior: 'smooth' // this will make the scroll behavior smooth 41 | }); 42 | } 43 | 44 | 45 | 46 | chatButton.addEventListener('click', async function() { 47 | if (chatInput.value.trim() != '') { 48 | var message = chatInput.value; 49 | var model = modelSelect.value; 50 | chatInput.value = ''; 51 | chatArea.innerHTML += `

${message}

`; 52 | scrollToBottom(); 53 | 54 | 55 | 56 | 57 | 58 | // SEND REQUEST 59 | 60 | 61 | switch (model) { 62 | // text to image endpoint 63 | case "stablediff": 64 | await txt2imgRequest(message,'',1,512,512,10,0.5); 65 | break; 66 | case "llava": 67 | await llavaRequest(currb64Img,message,model) 68 | break; 69 | // case for all text2text chats 70 | default: 71 | try { 72 | const response = await fetch('/api/chat', { 73 | method: 'POST', 74 | headers: { 75 | 'Content-Type': 'application/json' 76 | }, 77 | body: JSON.stringify({ 78 | message: message, 79 | model: model 80 | }) 81 | }); 82 | 83 | const data = await response.json(); 84 | var resArr = data.message.responses 85 | var resStr = resArr.join(' ') 86 | chatArea.innerHTML += `

${resStr}

`; 87 | scrollToBottom(); 88 | } catch (error) { 89 | console.log(error); 90 | chatArea.innerHTML += `

${error}

`; 91 | scrollToBottom(); 92 | } 93 | } 94 | 95 | } 96 | }); 97 | 98 | // send on enter 99 | chatInput.addEventListener('keydown', function(event) { 100 | if (event.key === 'Enter' & chatInput.value.trim() != '') { 101 | chatButton.click(); 102 | } 103 | }); 104 | 105 | // Function to make the POST request 106 | async function txt2imgRequest(prompt,seed,outputs,width,height,steps,guidance_scale) { 107 | const url = '/txt2img-trigger'; // CHANGE TO YOUR ENDPOINT IN SECRETS 108 | 109 | const requestBody = { 110 | "prompt": prompt, 111 | "seed": seed, // Numeric seed 112 | "num_outputs": outputs, // Number of images 113 | "width": width, // Width of results 114 | "height": height, // Height of results 115 | "num_inference_steps": steps, // Number of inference steps 116 | "guidance_scale": guidance_scale // Prompt strength 117 | }; 118 | 119 | try { 120 | const response = await fetch(url, { 121 | method: 'POST', 122 | headers: { 123 | 'Content-Type': 'application/json', 124 | // Add any other headers if needed 125 | }, 126 | body: JSON.stringify(requestBody), 127 | }); 128 | 129 | const responseData = await response.json(); 130 | 131 | if (responseData.status === 'success') { 132 | try { 133 | const image = responseData.image_data; 134 | // Handle the image, e.g., display it in an HTML img tag 135 | const imgElement = document.createElement('img'); 136 | // Generate a random ID for the

element 137 | const randomId = `image_${Math.floor(Math.random() * 100000)}`; 138 | 139 | imgElement.src = `data:image/png;base64,${image}`; 140 | chatArea.innerHTML += `

`; 141 | document.getElementById(randomId).append(imgElement) 142 | scrollToBottom(); 143 | 144 | //chatArea.innerHTML += `

${responseData.message}

`; 145 | } 146 | catch { 147 | chatArea.innerHTML += `

caught error

`; 148 | scrollToBottom(); 149 | } 150 | 151 | } 152 | else { 153 | console.error(`Request failed: ${responseData.message}`); 154 | chatArea.innerHTML += `

${error}

`; 155 | scrollToBottom(); 156 | } 157 | } 158 | catch (error) { 159 | console.error('Error:', error); 160 | chatArea.innerHTML += `

${error}

`; 161 | scrollToBottom(); 162 | } 163 | } 164 | 165 | 166 | // function to make a request to llava 167 | async function llavaRequest(baseEncodedImage,message,model) { 168 | bImg = baseEncodedImage.split('data:image/png;base64,')[1] 169 | try { 170 | const response = await fetch('/api/llava', { 171 | method: 'POST', 172 | headers: { 173 | 'Content-Type': 'application/json' 174 | }, 175 | body: JSON.stringify({ 176 | 'message': message, 177 | 'model': model, 178 | 'image': bImg 179 | }) 180 | }); 181 | 182 | const data = await response.json(); 183 | var resArr = data.message.responses 184 | var resStr = resArr 185 | chatArea.innerHTML += `

${resStr}

`; 186 | scrollToBottom(); 187 | } catch (error) { 188 | console.log(error); 189 | chatArea.innerHTML += `

${error}

`; 190 | scrollToBottom(); 191 | } 192 | } 193 | 194 | 195 | var installedModelsList = document.getElementById('installed'); 196 | var uninstalledModelsList = document.getElementById('uninstalled'); 197 | 198 | // Assuming we've got lists of models somewhere, we'd make them draggable like this: 199 | 200 | 201 | installedModelsList.innerHTML += '
Stable Diffusion
'; 202 | 203 | // Making sure the model lists allow dropping 204 | installedModelsList.ondragover = allowDrop; 205 | uninstalledModelsList.ondragover = allowDrop; 206 | installedModelsList.ondrop = drop; 207 | uninstalledModelsList.ondrop = drop; 208 | 209 | 210 | const menuToggler = document.getElementById('menu_toggle'); 211 | const controls = document.getElementById('controls'); 212 | const modelInstaller = document.getElementById('model-lists') 213 | var open = true; 214 | menuToggler.addEventListener('click', async(e) => { 215 | switch(open) { 216 | case true: 217 | modelSelect.style.display = 'none'; 218 | modelInstaller.style.display = 'none'; 219 | controls.style.maxHeight = "12px"; 220 | menuToggler.style.marginTop = '-24px'; 221 | menuToggler.style.transform = 'rotate(0deg)'; 222 | open = false; 223 | break; 224 | case false: 225 | modelSelect.style.display = 'flex'; 226 | modelInstaller.style.display = 'flex'; 227 | controls.style.maxHeight = "250px"; 228 | controls.style.height = "128px"; 229 | menuToggler.style.marginTop = '-32px'; 230 | menuToggler.style.transform = 'rotate(180deg)'; 231 | open = true; 232 | break; 233 | case _: 234 | // do nothing 235 | } 236 | 237 | }); 238 | 239 | 240 | 241 | /* DETECT MODELS INSTALLED */ 242 | var availableModels = [ 243 | 'neural-chat', 244 | 'starling-lm', 245 | 'mistral', 246 | 'llama2', 247 | 'codellama', 248 | 'llama2-uncensored', 249 | 'llama2:13b', 250 | 'llama2:70b', 251 | 'orca-mini', 252 | 'vicuna', 253 | 'llava', 254 | 'medllama2' 255 | ] 256 | 257 | async function getInstalledModels() { 258 | // call on the list models request 259 | var response = await fetch('/list-models'); 260 | var data = await response.json(); 261 | var models = data.models; 262 | 263 | return models 264 | } 265 | 266 | // get the installed models on boot 267 | var installedModels = await getInstalledModels(); 268 | var namesArray = []; 269 | for (var i = 0; i < installedModels.models.length; i++) { 270 | var modelName = installedModels.models[i].name; 271 | namesArray.push(modelName); 272 | } 273 | var verifiedModels = []; 274 | var unverifiedModels = []; 275 | var isEmpty = true; 276 | for (var i = 0; i < availableModels.length; i++) { 277 | isEmpty = true; // Reset isEmpty flag for each available model 278 | for (var j = 0; j < namesArray.length; j++) { 279 | if (namesArray[j].includes(availableModels[i])) { 280 | verifiedModels.push(availableModels[i]); 281 | isEmpty = false; // Set isEmpty to false if a match is found 282 | break; // Break out of the inner loop once a match is found 283 | } 284 | } 285 | if (isEmpty) { 286 | unverifiedModels.push(availableModels[i]); 287 | } 288 | } 289 | // inject the verified models into the installed bin 290 | verifiedModels.forEach(model => { 291 | var presentedName = '' 292 | switch(model) { 293 | case 'neural-chat': 294 | presentedName = 'Neural Chat' 295 | break; 296 | case 'starling-lm': 297 | presentedName = 'Starling LM' 298 | break; 299 | case 'mistral': 300 | presentedName = 'Mistral' 301 | break; 302 | case 'llama2': 303 | presentedName = 'Llama 2' 304 | break; 305 | case 'codellama': 306 | presentedName = 'CodeLlama' 307 | break; 308 | case 'llama2-uncensored': 309 | presentedName = 'Uncensored Llama 2' 310 | break; 311 | case 'llama2:13b': 312 | presentedName = 'Llama 2 (13B)' 313 | break; 314 | case 'llama2:70b': 315 | presentedName = 'Llama 2 (70B)' 316 | break; 317 | case 'orca-mini': 318 | presentedName = 'Orca Mini' 319 | break; 320 | case 'vicuna': 321 | presentedName = 'Vicuna' 322 | break; 323 | default: 324 | presentedName = model 325 | } 326 | 327 | var installedModelsList = document.getElementById('installed'); 328 | 329 | installedModelsList.innerHTML += `
${presentedName}
`; 330 | 331 | }); 332 | 333 | // inject the unverified models into the uninstalled bin 334 | unverifiedModels.forEach(model => { 335 | var presentedName = '' 336 | switch(model) { 337 | case 'neural-chat': 338 | presentedName = 'Neural Chat' 339 | break; 340 | case 'starling-lm': 341 | presentedName = 'Starling LM' 342 | break; 343 | case 'mistral': 344 | presentedName = 'Mistral' 345 | break; 346 | case 'llama2': 347 | presentedName = 'Llama 2' 348 | break; 349 | case 'codellama': 350 | presentedName = 'CodeLlama' 351 | break; 352 | case 'llama2-uncensored': 353 | presentedName = 'Uncensored Llama 2' 354 | break; 355 | case 'llama2:13b': 356 | presentedName = 'Llama 2 (13B)' 357 | break; 358 | case 'llama2:70b': 359 | presentedName = 'Llama 2 (70B)' 360 | break; 361 | case 'orca-mini': 362 | presentedName = 'Orca Mini' 363 | break; 364 | case 'vicuna': 365 | presentedName = 'Vicuna' 366 | break; 367 | default: 368 | presentedName = model 369 | } 370 | var uninstalledModelsList = document.getElementById('uninstalled'); 371 | uninstalledModelsList.innerHTML += `
${presentedName}
`; 372 | 373 | }); 374 | function populateModelsInSelect(){ 375 | // add all installed models to the select dropdown 376 | var installedModels = document.getElementById('installed').getElementsByClassName('model-installer') 377 | // clear out the options 378 | modelSelect.innerHTML = ` 379 | 380 | ` 381 | for(var i = 0; i < installedModels.length; i++){ 382 | modelSelect.innerHTML += ` 383 | 384 | ` 385 | } 386 | } 387 | populateModelsInSelect(); 388 | 389 | 390 | // install/uninstall model on drop 391 | // Function to install a model 392 | async function installModel(modelName) { 393 | if(modelName == 'stablediff'){ 394 | return 'stablediffusion is not able to be uninstalled' 395 | } 396 | else { 397 | try { 398 | const response = await fetch('/install-model', { 399 | method: 'POST', 400 | headers: { 401 | 'Content-Type': 'application/json', 402 | }, 403 | body: JSON.stringify({ model_name: modelName }), 404 | }); 405 | 406 | const data = await response.json(); 407 | if (response.ok) { 408 | return data.message 409 | } else { 410 | console.error('Installation error:', data.error); 411 | return data 412 | } 413 | } 414 | catch (error) { 415 | console.error('Fetch error:', error); 416 | return error 417 | } 418 | } 419 | 420 | } 421 | 422 | // Function to uninstall a model 423 | async function uninstallModel(modelName) { 424 | if(modelName == 'stablediff'){ 425 | return 'stablediff is not able to be uninstalled' 426 | } 427 | else { 428 | try { 429 | const response = await fetch('/uninstall-model', { 430 | method: 'POST', 431 | headers: { 432 | 'Content-Type': 'application/json', 433 | }, 434 | body: JSON.stringify({ 'model_name': modelName }), 435 | }); 436 | const data = await response.json(); 437 | if (response.ok) { 438 | return data.message 439 | } else { 440 | console.error('Uninstallation error:', data.error); 441 | return data 442 | } 443 | } catch (error) { 444 | console.error('Fetch error:', error); 445 | return error 446 | } 447 | } 448 | } 449 | 450 | 451 | // WINDOW DRAG AND DROP MUST GO AT BOTTOM OF LOADED FUNCS 452 | window.allowDrop = function(event) { 453 | if(!isProcessing){ 454 | event.preventDefault(); 455 | } 456 | }; 457 | 458 | window.drag = function(event) { 459 | if(!isProcessing){ 460 | event.dataTransfer.setData("text", event.target.id); 461 | } 462 | }; 463 | 464 | 465 | // Modified drop function 466 | window.drop = async function(event) { 467 | if (!isProcessing) { // Only allow dropping if not processing 468 | event.preventDefault(); 469 | var data = event.dataTransfer.getData("text"); 470 | var draggableElement = document.getElementById(data); 471 | 472 | // Find the closest parent that has the 'model-list' class 473 | var dropTarget = event.target.closest('.model-list'); 474 | 475 | // Only allow drop if the target is '.model-list' 476 | if (dropTarget) { 477 | dropTarget.appendChild(draggableElement); // Append to '.model-list' 478 | 479 | isProcessing = true; 480 | var boxes = document.getElementById('model-lists') 481 | var loader = document.getElementById('loader'); 482 | boxes.style.display = 'none'; 483 | loader.style.display = 'flex'; 484 | 485 | // Check if the model was dropped into 'installed' or 'uninstalled' 486 | if (dropTarget.id === 'installed') { 487 | // Call the install model function 488 | await installModel(draggableElement.id); 489 | } else if (dropTarget.id === 'uninstalled') { 490 | // Call the uninstall model function 491 | await uninstallModel(draggableElement.id); 492 | } 493 | 494 | isProcessing = false; 495 | 496 | loader.style.display = 'none'; 497 | boxes.style.display = 'flex'; 498 | 499 | 500 | } 501 | } 502 | } 503 | 504 | // Drag and Drop functionality for model installation 505 | function allowDrop(event) { 506 | event.preventDefault(); 507 | } 508 | 509 | function drag(event) { 510 | event.dataTransfer.setData("text", event.target.id); 511 | } 512 | 513 | async function drop(event) { 514 | if (!isProcessing) { // Only allow dropping if not processing 515 | event.preventDefault(); 516 | var data = event.dataTransfer.getData("text"); 517 | var draggableElement = document.getElementById(data); 518 | 519 | // Find the closest parent that has the 'model-list' class 520 | var dropTarget = event.target.closest('.model-list'); 521 | 522 | // Only allow drop if the target is '.model-list' 523 | if (dropTarget) { 524 | dropTarget.appendChild(draggableElement); // Append to '.model-list' 525 | 526 | isProcessing = true; 527 | var boxes = document.getElementById('model-lists') 528 | var loader = document.getElementById('loader'); 529 | boxes.style.display = 'none'; 530 | loader.style.display = 'flex'; 531 | 532 | // Check if the model was dropped into 'installed' or 'uninstalled' 533 | if (dropTarget.id === 'installed') { 534 | // Call the install model function 535 | await installModel(draggableElement.id); 536 | } else if (dropTarget.id === 'uninstalled') { 537 | // Call the uninstall model function 538 | await uninstallModel(draggableElement.id); 539 | } 540 | 541 | isProcessing = false; 542 | 543 | loader.style.display = 'none'; 544 | boxes.style.display = 'flex'; 545 | } 546 | 547 | 548 | // add all installed models to the select dropdown 549 | var installedModels = document.getElementById('installed').getElementsByClassName('model-installer') 550 | // clear out the options 551 | modelSelect.innerHTML = ` 552 | 553 | ` 554 | for(var i = 0; i < installedModels.length; i++){ 555 | modelSelect.innerHTML += ` 556 | 557 | ` 558 | } 559 | } 560 | } 561 | 562 | 563 | // HANDLE CONDITIONALS 564 | 565 | 566 | // Handle file selection and processing for llava 567 | function handleFileSelect(event) { 568 | // Retrieve the first file from the FileList object 569 | let file = event.target.files[0]; 570 | if (file) { 571 | const reader = new FileReader(); 572 | 573 | reader.onload = function(loadEvent) { 574 | const base64String = loadEvent.target.result; 575 | // Now you have the base64 string, you can use it where you need it 576 | currb64Img = base64String 577 | // Here you could also set up a request to send the base64 string 578 | // to your server, or handle it in some other way according to your needs. 579 | }; 580 | reader.readAsDataURL(file); 581 | } 582 | } 583 | 584 | 585 | 586 | document.getElementById('llava-upload-input').addEventListener('change', handleFileSelect, false); 587 | 588 | modelSelect.addEventListener('change', function() { 589 | if(modelSelect.value == "llava"){ 590 | document.getElementById('llava-file-upload').style.display = 'flex'; 591 | } 592 | else { 593 | document.getElementById('llava-file-upload').style.display = 'none'; 594 | } 595 | // Hide file upload area when the model other than llava is selected 596 | let uploadArea = document.getElementById('llava-file-upload'); 597 | 598 | }); 599 | 600 | 601 | 602 | 603 | 604 | }); 605 | 606 | 607 | 608 | 609 | 610 | var consoleArt =` 611 | .o8 .o8 oooo o8o . 612 | "888 "888 888 "' .o8 613 | .oooo888 oooo oooo 888oooo. 888 oooo .o888oo 614 | d88' 888 888 888 d88' 88b 888 888 888 615 | 888 888 888 888 888 888 888 888 888 616 | 888 888 888 888 888 888 888 888 888 . 617 | Y8bod88P" V88V"V8P' 'Y8bod8P' o888o o888o "888" 618 | ` 619 | 620 | console.log(consoleArt); 621 | console.log(`interested in projects like this? check out https://dublit.org/`); 622 | 623 | // Call the function to make the request 624 | //txt2imgRequest(); -------------------------------------------------------------------------------- /app/templates/index.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | Ollama API 9 | 12 | 13 | 14 |

VALDI Modular LLM

15 | 16 |
17 | 20 | 21 |
22 |
23 |

Installed

24 |
25 | 26 |
27 |
28 |
29 |

Uninstalled

30 |
31 | 32 |
33 |
34 |
35 | 36 |
37 | 38 | 39 |
40 | 41 |
42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 |
53 |
54 | 55 |
56 |
57 | 58 | 59 |
60 |
61 | 62 | 63 |
64 |
65 | 66 | 67 | 68 | 69 | -------------------------------------------------------------------------------- /assets/config_demo.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dublit-Development/ollama-api/f6b8bfcb8e1ce17ee9321c923876b94ed230716e/assets/config_demo.gif -------------------------------------------------------------------------------- /assets/demo.mp4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dublit-Development/ollama-api/f6b8bfcb8e1ce17ee9321c923876b94ed230716e/assets/demo.mp4 -------------------------------------------------------------------------------- /assets/demo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dublit-Development/ollama-api/f6b8bfcb8e1ce17ee9321c923876b94ed230716e/assets/demo.png -------------------------------------------------------------------------------- /assets/run_server.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Dublit-Development/ollama-api/f6b8bfcb8e1ce17ee9321c923876b94ed230716e/assets/run_server.gif --------------------------------------------------------------------------------