├── .gitignore ├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── requirements.txt ├── resources └── screenshot.gif └── source └── rich-chat.py /.gitignore: -------------------------------------------------------------------------------- 1 | *.backup 2 | *.history 3 | *.log 4 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing to Rich Chat 2 | 3 | Thank you for considering contributing to Rich Chat! We welcome contributions from the community. Before submitting a pull request, please read and follow these guidelines. 4 | 5 | ## Reporting Bugs 6 | If you find a bug in Rich Chat, please report it by opening an issue on this repository. Make sure to include: 7 | - A clear description of the issue 8 | - Steps to reproduce the issue 9 | - Any relevant logs or error messages 10 | 11 | ## Contributing Code 12 | We welcome contributions to Rich Chat! Before submitting a pull request, please ensure your changes meet these requirements: 13 | 14 | 1. Fork this repository and create a new branch for your changes. 15 | 2. Make sure your changes pass all tests (unit, integration, etc.). 16 | 3. Write clear, concise commit messages that describe the changes you made. 17 | 4. Include any necessary documentation updates. 18 | 5. Ensure that your contribution adheres to our code style guidelines. 19 | 20 | ## Attribution 21 | By contributing to Rich Chat, you agree that your contributions will be licensed under the MIT License. We kindly ask that you include the following attribution notice in your project if you use any part of Rich Chat: 22 | 23 | "Rich Chat console application is based on Akarshan Biswas's original work. For more information, please visit https://github.com/akarshanbiswas/rich-chat." -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2024 Akarshan Biswas 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Rich Chat: A Console App for Interactive Chatting with LLMs using Rich Text 2 | 3 | Rich Chat is a Python console application designed to provide an engaging and visually appealing chat experience on Unix-like consoles or Terminals. This app utilizes the **rich** text library to render attractive text, creating a chat interface reminiscent of instant messaging applications. 4 | Rich Chat offers an interactive console experience with a visually appealing chat interface using the rich text library. 5 | 6 | ![](https://github.com/akarshanbiswas/rich-chat/blob/main/resources/screenshot.gif) 7 | 8 | 9 | ## Installation 10 | 11 | To use Rich Chat, first install the required dependencies: 12 | 13 | ```bash 14 | pip install -r requirements.txt 15 | ``` 16 | 17 | 18 | ## Usage 19 | 20 | Run the program with the following command line options: 21 | 22 | ```bash 23 | python source/rich-chat.py [options] 24 | ``` 25 | 26 | ### Commandline Options 27 | * --help or -h: Show this help message and exit. 28 | * --server SERVER: Set the OpenAI compatible server chat endpoint, e.g., chat.example.com. 29 | * --model-frame-color MODEL_FRAME_COLOR: Frame color of Large language Model (default: blue). 30 | * --topk TOPK: Set the top_k value to sample the top N number of tokens, where N is an integer. 31 | * --topp TOPP: Set the top_p value. 32 | * --temperature TEMPERATURE: Controls the randomness of text generation (default: 0.5). 33 | * --n-predict N_PREDICT: Define how many tokens to predict by the model (default: infinity until [stop] token). 34 | 35 | These options are currently inexhaustible. More will be added later. 36 | 37 | ## Roadmap 38 | 39 | - Expand the options 40 | 41 | - Proper RAG support(Both internet and documents) 42 | 43 | - Multimodal 44 | 45 | Please stay tuned for updates. 46 | Contributions are surely welcome!! 47 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | certifi==2024.7.4 2 | charset-normalizer==3.3.2 3 | idna==3.7 4 | markdown-it-py==3.0.0 5 | mdurl==0.1.2 6 | Pygments==2.17.2 7 | requests==2.32.2 8 | rich==13.7.0 9 | urllib3==2.2.2 10 | prompt-toolkit==3.0.43 11 | -------------------------------------------------------------------------------- /resources/screenshot.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/qnixsynapse/rich-chat/84fe1f1edbfb7514e62ffa0395e49424e2e951e6/resources/screenshot.gif -------------------------------------------------------------------------------- /source/rich-chat.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import json 3 | import os 4 | 5 | import requests 6 | from prompt_toolkit import PromptSession 7 | from prompt_toolkit.history import FileHistory 8 | from rich.console import Console 9 | from rich.live import Live 10 | from rich.markdown import Markdown 11 | 12 | 13 | def remove_lines_console(num_lines): 14 | for _ in range(num_lines): 15 | print("\x1b[A", end="\r", flush=True) 16 | 17 | 18 | def estimate_lines(text): 19 | columns, _ = os.get_terminal_size() 20 | line_count = 1 21 | text_lines = text.split("\n") 22 | for text_line in text_lines: 23 | lines_needed = (len(text_line) // columns) + 1 24 | 25 | line_count += lines_needed 26 | 27 | return line_count 28 | 29 | 30 | def handle_console_input(session: PromptSession) -> str: 31 | return session.prompt("(Prompt: ⌥ + ⏎) | (Exit: ⌘ + c): ", multiline=True).strip() 32 | 33 | 34 | class conchat: 35 | def __init__( 36 | self, 37 | server_addr, 38 | min_p: float, 39 | repeat_penalty: float, 40 | seed: int, 41 | top_k=10, 42 | top_p=0.95, 43 | temperature=0.12, 44 | n_predict=-1, 45 | stream: bool = True, 46 | cache_prompt: bool = True, 47 | model_frame_color: str = "red", 48 | ) -> None: 49 | self.model_frame_color = model_frame_color 50 | self.serveraddr = server_addr 51 | self.topk = top_k 52 | self.top_p = top_p 53 | self.seed = seed 54 | self.min_p = min_p 55 | self.repeat_penalty = repeat_penalty 56 | self.temperature = temperature 57 | self.n_predict = n_predict 58 | self.stream = stream 59 | self.cache_prompt = cache_prompt 60 | self.headers = {"Content-Type": "application/json"} 61 | self.chat_history = [] 62 | self.model_name = "" 63 | 64 | self.console = Console() 65 | 66 | # TODO: Gracefully handle user input history file. 67 | self.session = PromptSession(history=FileHistory(".rich-chat.history")) 68 | 69 | def chat_generator(self, prompt): 70 | endpoint = self.serveraddr + "/v1/chat/completions" 71 | self.chat_history.append({"role": "user", "content": prompt}) 72 | 73 | payload = { 74 | "messages": self.chat_history, 75 | "temperature": self.temperature, 76 | "top_k": self.topk, 77 | "top_p": self.top_p, 78 | "n_predict": self.n_predict, 79 | "stream": self.stream, 80 | "cache_prompt": self.cache_prompt, 81 | "seed": self.seed, 82 | "repeat_penalty": self.repeat_penalty, 83 | "min_p": self.min_p, 84 | } 85 | try: 86 | response = requests.post( 87 | url=endpoint, 88 | data=json.dumps(payload), 89 | headers=self.headers, 90 | stream=self.stream, 91 | ) 92 | assert ( 93 | response.status_code == 200 94 | ), "Failed to establish proper connection to the server! Please check server health!" 95 | for chunk in response.iter_lines(): 96 | if chunk: 97 | chunk = chunk.decode("utf-8") 98 | if chunk.startswith("data: "): 99 | chunk = chunk.replace("data: ", "") 100 | chunk = chunk.strip() 101 | # print(chunk) 102 | chunk = json.loads(chunk) 103 | # if "content" in chunk: 104 | yield chunk 105 | except Exception as e: 106 | print(f"GeneratorError: {e}") 107 | 108 | def health_checker(self): 109 | try: 110 | endpoint = self.serveraddr + "/health" 111 | response = requests.get(url=endpoint, headers=self.headers) 112 | assert ( 113 | response.status_code == 200 114 | ), "Unable to reach server! Please check if server is running or your Internet connection is working or not." 115 | status = json.loads(response.content.decode("utf-8"))["status"] 116 | return status 117 | except Exception as e: 118 | print(f"HealthError: {e}") 119 | 120 | def get_model_name(self): 121 | try: 122 | endpoint = self.serveraddr + "/slots" 123 | response = requests.get(url=endpoint) 124 | assert response.status_code == 200, "Server not reachable!" 125 | data = json.loads(response.content.decode("utf-8"))[0]["model"] 126 | return data 127 | except Exception as e: 128 | print(f"SlotsError: {e}") 129 | 130 | def handle_streaming(self, prompt): 131 | self.console.print(Markdown("**>**"), end=" ") 132 | text = "" 133 | block = "█ " 134 | with Live( 135 | console=self.console, 136 | ) as live: 137 | for token in self.chat_generator(prompt=prompt): 138 | if "content" in token["choices"][0]["delta"]: 139 | text = text + token["choices"][0]["delta"]["content"] 140 | if token["choices"][0]["finish_reason"] is not None: 141 | block = "" 142 | markdown = Markdown(text + block) 143 | live.update( 144 | markdown, 145 | refresh=True, 146 | ) 147 | self.chat_history.append({"role": "assistant", "content": text}) 148 | 149 | def chat(self): 150 | status = self.health_checker() 151 | assert status == "ok", "Server not ready or error!" 152 | self.model_name = self.get_model_name() 153 | while True: 154 | try: 155 | user_m = handle_console_input(self.session) 156 | self.handle_streaming(prompt=user_m) 157 | 158 | # NOTE: Ctrl + c (keyboard) or Ctrl + d (eof) to exit 159 | # Adding EOFError prevents an exception and gracefully exits. 160 | except (KeyboardInterrupt, EOFError): 161 | exit() 162 | 163 | 164 | def main(): 165 | parser = argparse.ArgumentParser( 166 | description="Console Inference of LLM models. Works with any OpenAI compatible server." 167 | ) 168 | parser.add_argument( 169 | "--server", 170 | type=str, 171 | help="Any OpenAI compatible server chat endpoint. Like chat.example.com, excluding 'v1/chat' etc.", 172 | ) 173 | parser.add_argument( 174 | "--model-frame-color", 175 | type=str, 176 | default="white", 177 | help="Frame color of Large language Model", 178 | ) 179 | parser.add_argument( 180 | "--topk", 181 | type=int, 182 | help="top_k value to sample the top n number of tokens where n is an integer.", 183 | ) 184 | parser.add_argument("--topp", type=float, help="top_p value") 185 | parser.add_argument( 186 | "--temperature", 187 | type=float, 188 | help="Controls the randomness of the text generation. Default: 0.5", 189 | ) 190 | parser.add_argument( 191 | "--n-predict", 192 | type=int, 193 | help="The number defines how many tokens to be predict by the model. Default: infinity until [stop] token.", 194 | ) 195 | parser.add_argument( 196 | "--minp", 197 | type=float, 198 | default=0.5, 199 | help="The minimum probability for a token to be considered, relative to the probability of the most likely token (default: 0.05).", 200 | ) 201 | parser.add_argument( 202 | "--repeat-penalty", 203 | type=float, 204 | default=1.1, 205 | help="Control the repetition of token sequences in the generated text (default: 1.1).", 206 | ) 207 | parser.add_argument( 208 | "--seed", 209 | type=int, 210 | default=-1, 211 | help="Set the random number generator (RNG) seed (default: -1, -1 = random seed).", 212 | ) 213 | 214 | args = parser.parse_args() 215 | chat = conchat( 216 | server_addr=args.server, 217 | top_k=args.topk, 218 | top_p=args.topp, 219 | temperature=args.temperature, 220 | model_frame_color=args.model_frame_color, 221 | min_p=args.minp, 222 | seed=args.seed, 223 | repeat_penalty=args.repeat_penalty, 224 | ) 225 | chat.chat() 226 | 227 | 228 | if __name__ == "__main__": 229 | main() 230 | --------------------------------------------------------------------------------