├── example.env ├── .github ├── version.json ├── groq-logo.png ├── ISSUE_TEMPLATE │ ├── feature_request.md │ └── bug_report.md ├── PULL_REQUEST_TEMPLATE │ └── pull_request.md ├── SECURITY.md ├── CHANGELOG.md ├── CONTRIBUTING.md └── CODE_OF_CONDUCT.md ├── examples ├── example_chat.py └── example_text.py ├── requirements.txt ├── LICENSE ├── loading.py ├── config.py ├── client.py ├── .gitignore ├── cli.py ├── groq.py └── README.md /example.env: -------------------------------------------------------------------------------- 1 | GROQ_API_KEY=your_api_key_here -------------------------------------------------------------------------------- /.github/version.json: -------------------------------------------------------------------------------- 1 | { 2 | "version": "v1.0.1" 3 | } 4 | -------------------------------------------------------------------------------- /examples/example_chat.py: -------------------------------------------------------------------------------- 1 | from groq import Chat 2 | 3 | Chat().run() -------------------------------------------------------------------------------- /.github/groq-logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RMNCLDYO/groq-ai-toolkit/HEAD/.github/groq-logo.png -------------------------------------------------------------------------------- /examples/example_text.py: -------------------------------------------------------------------------------- 1 | from groq import Text 2 | 3 | Text().run(prompt="Explain the importance of low latency LLMs.") -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | # Core dependencies 2 | requests 3 | 4 | # Optional dependencies 5 | python-dotenv 6 | 7 | # Additional dependencies may be added as needed 8 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/feature_request.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Feature request 3 | about: Suggest an idea for this project 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Is your feature request related to a problem? Please describe.** 11 | A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] 12 | 13 | **Describe the solution you'd like** 14 | A clear and concise description of what you want to happen. 15 | 16 | **Describe alternatives you've considered** 17 | A clear and concise description of any alternative solutions or features you've considered. 18 | 19 | **Additional context** 20 | Add any other context or screenshots about the feature request here. 21 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/bug_report.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Bug report 3 | about: Create a report to help us improve 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Describe the bug** 11 | A clear and concise description of what the bug is. 12 | 13 | **To Reproduce** 14 | Steps to reproduce the behavior: 15 | 1. Go to '...' 16 | 2. Click on '....' 17 | 3. Scroll down to '....' 18 | 4. See error 19 | 20 | **Expected behavior** 21 | A clear and concise description of what you expected to happen. 22 | 23 | **Screenshots** 24 | If applicable, add screenshots to help explain your problem. 25 | 26 | **Desktop (please complete the following information):** 27 | - OS: [e.g. iOS] 28 | - Browser [e.g. chrome, safari] 29 | - Version [e.g. 22] 30 | 31 | **Smartphone (please complete the following information):** 32 | - Device: [e.g. iPhone6] 33 | - OS: [e.g. iOS8.1] 34 | - Browser [e.g. stock browser, safari] 35 | - Version [e.g. 22] 36 | 37 | **Additional context** 38 | Add any other context about the problem here. 39 | -------------------------------------------------------------------------------- /.github/PULL_REQUEST_TEMPLATE/pull_request.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Pull Request 3 | about: Propose changes to the codebase 4 | title: '[PR] ' 5 | labels: enhancement 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Description of the Changes** 11 | A clear and concise description of what the pull request does. 12 | 13 | **Related Issue** 14 | Link to the issue that this pull request addresses. 15 | 16 | **Motivation and Context** 17 | Why is this change required? What problem does it solve? 18 | 19 | **How Has This Been Tested?** 20 | Please describe in detail how you tested your changes. 21 | 22 | **Screenshots (if appropriate):** 23 | 24 | **Types of Changes** 25 | - [ ] Bug fix (non-breaking change which fixes an issue) 26 | - [ ] New feature (non-breaking change which adds functionality) 27 | - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) 28 | 29 | **Checklist:** 30 | - [ ] My code follows the project's code style 31 | - [ ] I have read the CONTRIBUTING document 32 | - [ ] I have added tests to cover my changes 33 | - [ ] All new and existing tests passed 34 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2024 RMNCLDYO 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /.github/SECURITY.md: -------------------------------------------------------------------------------- 1 | # Security Policy 2 | 3 | ## Supported Versions 4 | 5 | | Version | Supported | 6 | | ------- | ------------------ | 7 | | 1.0.x | :white_check_mark: | 8 | | < 1.0 | :x: | 9 | 10 | ## Reporting a Vulnerability 11 | 12 | We take the security of our software seriously. If you have discovered a security vulnerability in the project, please follow these steps to report it responsibly: 13 | 14 | 1. **Do Not Publish the Vulnerability**: Avoid sharing the details of the vulnerability in public forums, issues, or other public channels. 15 | 16 | 2. **Email the Maintainers**: Send an email to the maintainers of the project. Provide a clear description of the vulnerability, including steps to reproduce it. 17 | 18 | 3. **Wait for Response**: Allow a reasonable amount of time for the maintainers to respond to your report and address the vulnerability. 19 | 20 | 4. **Disclosure**: After the issue has been resolved and announced, you may consider disclosing the issue to the public in a responsible manner. 21 | 22 | We appreciate your efforts to responsibly disclose your findings and will make every effort to acknowledge your contributions. 23 | 24 | ## Contact Information 25 | 26 | For any security concerns, please contact us. 27 | -------------------------------------------------------------------------------- /loading.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import threading 3 | import time 4 | 5 | class Loading: 6 | def __init__(self): 7 | self.spinner = '|/-\\' 8 | self.spinner_index = 0 9 | self.running = False 10 | self.thread = None 11 | 12 | def hide_cursor(self): 13 | sys.stdout.write('\033[?25l') 14 | sys.stdout.flush() 15 | 16 | def show_cursor(self): 17 | sys.stdout.write('\033[?25h') 18 | sys.stdout.flush() 19 | 20 | def clear_line(self): 21 | sys.stdout.write('\r\033[K') 22 | sys.stdout.flush() 23 | 24 | def update(self): 25 | while self.running: 26 | sys.stdout.write('\rWaiting for assistant response... ' + self.spinner[self.spinner_index]) 27 | sys.stdout.flush() 28 | self.spinner_index = (self.spinner_index + 1) % len(self.spinner) 29 | time.sleep(0.1) 30 | self.clear_line() 31 | self.show_cursor() 32 | 33 | def start(self): 34 | if not self.running: 35 | self.running = True 36 | self.hide_cursor() 37 | self.thread = threading.Thread(target=self.update) 38 | self.thread.start() 39 | 40 | def stop(self): 41 | if self.running: 42 | self.running = False 43 | self.thread.join() -------------------------------------------------------------------------------- /config.py: -------------------------------------------------------------------------------- 1 | import os 2 | 3 | def load_required_env_variables(var_name: str): 4 | value = os.getenv(var_name) 5 | if value is None: 6 | try: 7 | from dotenv import load_dotenv 8 | load_dotenv() 9 | value = os.getenv(var_name) 10 | if value is None or value.strip() == "": 11 | print(f"Error: {var_name} environment variable is not defined. Please define it in a .env file or directly in your environment. You can also pass it as an argument to the function.") 12 | exit(1) 13 | except ImportError: 14 | print("Error: dotenv package is not installed. Please install it with 'pip install python-dotenv' or define the environment variables directly.") 15 | exit(1) 16 | except Exception as e: 17 | print(f"Error loading environment variables: {e}") 18 | exit(1) 19 | return value 20 | 21 | def load_config(api_key=None): 22 | if not api_key: 23 | api_key = load_required_env_variables('GROQ_API_KEY') 24 | 25 | return { 26 | 'api_key': api_key, 27 | 'model': os.getenv('GROQ_MODEL', 'llama3-8b-8192'), 28 | 'base_url': os.getenv('GROQ_BASE_URL', 'https://api.groq.com'), 29 | 'completions_endpoint': os.getenv('GROQ_COMPLETIONS_ENDPOINT', 'chat/completions'), 30 | 'timeout': int(os.getenv('GROQ_TIMEOUT', 20)), 31 | 'version': os.getenv('GROQ_VERSION', 'openai/v1') 32 | } -------------------------------------------------------------------------------- /.github/CHANGELOG.md: -------------------------------------------------------------------------------- 1 | CHANGELOG.md# Changelog 2 | 3 | All notable changes to the project will be documented in this file. 4 | 5 | The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). 6 | 7 | ## [1.0.1] - 04/23/2024 8 | This update enhances the Groq AI Toolkit with additional features, improved error handling, and expanded functionality, particularly introducing support for 'LLama 3', JSON mode, and the ability to add a seed for sampling. 9 | 10 | ### Added 11 | - JSON Output Format: Users can now receive responses in JSON format, making it easier to integrate into applications. 12 | - Seed for Sampling: Adds the ability to specify a seed for deterministic output in sampling, aiding in reproducibility. 13 | - Model Additions: Introduced support for new `LLama3` models, expanding the toolkit's capabilities and model options. 14 | 15 | ### Fixed 16 | - Improved error handling in the `post` and `stream_post` methods in `client.py` to manage JSON parsing errors and HTTP exceptions more effectively. 17 | 18 | ### Changed 19 | - Updated model names in the configuration to include `LLama3` versions, reflecting the latest available models. 20 | 21 | ### Improved 22 | - Enhanced the speed and efficiency of response handling in streamed and post response modes to ensure faster and more reliable outputs. 23 | - Updated documentation and README to reflect new features, improving clarity and user guidance on new options and configurations. 24 | 25 | ## [1.1.0] - 03/05/2024 26 | 27 | ### Added 28 | - Initial release. 29 | -------------------------------------------------------------------------------- /.github/CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing to the project! 2 | 3 | We welcome contributions! This document provides guidelines and instructions for contributing to this project. 4 | 5 | ## Getting Started 6 | 7 | 1. **Fork the Repository**: Begin by forking the repository to your GitHub account. 8 | 2. **Clone the Repository**: Clone your forked repository to your local machine. 9 | 3. **Create a Branch**: Create a new branch for your contribution. 10 | 11 | ## Contribution Guidelines 12 | 13 | - **Code Style**: Follow the established code style in the project. 14 | - **Commit Messages**: Write meaningful commit messages that clearly explain the changes. 15 | - **Pull Requests**: Submit pull requests to the `main` branch. Ensure your code is well-tested and documented. 16 | 17 | ## Submitting Pull Requests 18 | 19 | 1. **Update Your Fork**: Regularly sync your fork with the main repository to keep it up-to-date. 20 | 2. **Make Your Changes**: Implement your feature or fix. 21 | 3. **Test Your Changes**: Ensure your changes do not break existing functionality. 22 | 4. **Document Your Changes**: Update the README or documentation if necessary. 23 | 5. **Submit a Pull Request**: Push your changes to your fork and open a pull request against the main repository. 24 | 25 | ## Reporting Issues 26 | 27 | - Use the GitHub issue tracker to report bugs or suggest enhancements. 28 | - Provide as much information as possible, including steps to reproduce the issue. 29 | - Check if the issue has already been reported to avoid duplicates. 30 | 31 | ## Code of Conduct 32 | 33 | By contributing to this project, you agree to abide by its [Code of Conduct](CODE_OF_CONDUCT.md). 34 | 35 | ## Questions or Suggestions 36 | 37 | Feel free to open an issue or contact the maintainers if you have any questions or suggestions. 38 | 39 | Thank you for contributing! 40 | -------------------------------------------------------------------------------- /.github/CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Code of Conduct 2 | 3 | ## Our Pledge 4 | 5 | In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation. 6 | 7 | ## Our Standards 8 | 9 | Examples of behavior that contributes to creating a positive environment include: 10 | 11 | - Using welcoming and inclusive language 12 | - Being respectful of differing viewpoints and experiences 13 | - Gracefully accepting constructive criticism 14 | - Focusing on what is best for the community 15 | - Showing empathy towards other community members 16 | 17 | Examples of unacceptable behavior by participants include: 18 | 19 | - The use of sexualized language or imagery and unwelcome sexual attention or advances 20 | - Trolling, insulting/derogatory comments, and personal or political attacks 21 | - Public or private harassment 22 | - Publishing others' private information, such as a physical or electronic address, without explicit permission 23 | - Other conduct which could reasonably be considered inappropriate in a professional setting 24 | 25 | ## Our Responsibilities 26 | 27 | Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior. 28 | 29 | Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful. 30 | 31 | ## Scope 32 | 33 | This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers. 34 | 35 | ## Enforcement 36 | 37 | Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team. All complaints will be reviewed and investigated and will result in a response that is deemed necessary and appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately. 38 | 39 | Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership. 40 | 41 | ## Attribution 42 | 43 | This Code of Conduct is adapted from the [Contributor Covenant](https://www.contributor-covenant.org/), version 1.4, available at [https://www.contributor-covenant.org/version/1/4/code-of-conduct.html](https://www.contributor-covenant.org/version/1/4/code-of-conduct.html). 44 | -------------------------------------------------------------------------------- /client.py: -------------------------------------------------------------------------------- 1 | import json 2 | import requests 3 | from config import load_config 4 | from loading import Loading 5 | 6 | print("------------------------------------------------------------------\n") 7 | print(" Groq AI Toolkit \n") 8 | print(" API Wrapper & Command-line Interface \n") 9 | print(" [v1.0.1] by @rmncldyo \n") 10 | print("------------------------------------------------------------------\n") 11 | 12 | class Client: 13 | def __init__(self, api_key=None): 14 | self.config = load_config(api_key=api_key) 15 | self.api_key = api_key if api_key else self.config.get('api_key') 16 | self.base_url = self.config.get('base_url') 17 | self.version = self.config.get('version') 18 | self.headers = { 19 | "Authorization": f"Bearer {self.api_key}", 20 | "Content-Type": "application/json" 21 | } 22 | 23 | def post(self, endpoint, data): 24 | loading = Loading() 25 | url = f"{self.base_url}/{self.version}/{endpoint}" 26 | try: 27 | loading.start() 28 | response = requests.post(url, headers=self.headers, json=data) 29 | response = response.json() 30 | try: 31 | response["error"]["message"] 32 | if "Failed to generate JSON" in response["error"]["message"]: 33 | response = "Failed to generate JSON" 34 | return response 35 | else: 36 | response = response["error"]["message"] 37 | return response 38 | except: 39 | pass 40 | try: 41 | response = response["choices"][0]["message"]["content"] 42 | return response 43 | except Exception as e: 44 | loading.stop() 45 | raise Exception(f"Error: {response}") 46 | except Exception as e: 47 | print(f"HTTP Error: {e}") 48 | raise 49 | finally: 50 | loading.stop() 51 | 52 | def stream_post(self, endpoint, data): 53 | loading = Loading() 54 | url = f"{self.base_url}/{self.version}/{endpoint}" 55 | full_response = [] 56 | try: 57 | loading.start() 58 | response = requests.post(url, headers=self.headers, json=data, stream=True) 59 | response.raise_for_status() 60 | loading.stop() 61 | response_content = "" 62 | print("Assistant: ", end="", flush=True) 63 | for chunk in response.iter_lines(): 64 | if chunk: 65 | json_data = chunk.decode('utf-8').split('data: ')[1] 66 | try: 67 | data_dict = json.loads(json_data) 68 | if data_dict['choices'][0]['delta'] != {}: 69 | print(data_dict['choices'][0]['delta']['content'], end="", flush=True) 70 | response_content += data_dict['choices'][0]['delta']['content'] 71 | else: 72 | if data_dict['choices'][0]['finish_reason'] == "stop": 73 | break 74 | except Exception as e: 75 | if json_data == "[DONE]": 76 | break 77 | else: 78 | print(f"An error occurred: {json_data}") 79 | full_response.append(response_content) 80 | print() 81 | return full_response[0] 82 | except Exception as e: 83 | print(f"Stream HTTP Error: {e}") 84 | raise 85 | finally: 86 | loading.stop() 87 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | build/ 12 | develop-eggs/ 13 | dist/ 14 | downloads/ 15 | eggs/ 16 | .eggs/ 17 | lib/ 18 | lib64/ 19 | parts/ 20 | sdist/ 21 | var/ 22 | wheels/ 23 | share/python-wheels/ 24 | *.egg-info/ 25 | .installed.cfg 26 | *.egg 27 | MANIFEST 28 | 29 | # PyInstaller 30 | # Usually these files are written by a python script from a template 31 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 32 | *.manifest 33 | *.spec 34 | 35 | # Installer logs 36 | pip-log.txt 37 | pip-delete-this-directory.txt 38 | 39 | # Unit test / coverage reports 40 | htmlcov/ 41 | .tox/ 42 | .nox/ 43 | .coverage 44 | .coverage.* 45 | .cache 46 | nosetests.xml 47 | coverage.xml 48 | *.cover 49 | *.py,cover 50 | .hypothesis/ 51 | .pytest_cache/ 52 | cover/ 53 | 54 | # Translations 55 | *.mo 56 | *.pot 57 | 58 | # Django stuff: 59 | *.log 60 | local_settings.py 61 | db.sqlite3 62 | db.sqlite3-journal 63 | 64 | # Flask stuff: 65 | instance/ 66 | .webassets-cache 67 | 68 | # Scrapy stuff: 69 | .scrapy 70 | 71 | # Sphinx documentation 72 | docs/_build/ 73 | 74 | # PyBuilder 75 | .pybuilder/ 76 | target/ 77 | 78 | # Jupyter Notebook 79 | .ipynb_checkpoints 80 | 81 | # IPython 82 | profile_default/ 83 | ipython_config.py 84 | 85 | # pyenv 86 | # For a library or package, you might want to ignore these files since the code is 87 | # intended to run in multiple environments; otherwise, check them in: 88 | # .python-version 89 | 90 | # pipenv 91 | # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. 92 | # However, in case of collaboration, if having platform-specific dependencies or dependencies 93 | # having no cross-platform support, pipenv may install dependencies that don't work, or not 94 | # install all needed dependencies. 95 | #Pipfile.lock 96 | 97 | # poetry 98 | # Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control. 99 | # This is especially recommended for binary packages to ensure reproducibility, and is more 100 | # commonly ignored for libraries. 101 | # https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control 102 | #poetry.lock 103 | 104 | # pdm 105 | # Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control. 106 | #pdm.lock 107 | # pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it 108 | # in version control. 109 | # https://pdm.fming.dev/#use-with-ide 110 | .pdm.toml 111 | 112 | # PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm 113 | __pypackages__/ 114 | 115 | # Celery stuff 116 | celerybeat-schedule 117 | celerybeat.pid 118 | 119 | # SageMath parsed files 120 | *.sage.py 121 | 122 | # Environments 123 | *.env 124 | .env 125 | .venv 126 | env/ 127 | venv/ 128 | ENV/ 129 | env.bak/ 130 | venv.bak/ 131 | 132 | # Spyder project settings 133 | .spyderproject 134 | .spyproject 135 | 136 | # Rope project settings 137 | .ropeproject 138 | 139 | # mkdocs documentation 140 | /site 141 | 142 | # mypy 143 | .mypy_cache/ 144 | .dmypy.json 145 | dmypy.json 146 | 147 | # Pyre type checker 148 | .pyre/ 149 | 150 | # pytype static type analyzer 151 | .pytype/ 152 | 153 | # Cython debug symbols 154 | cython_debug/ 155 | 156 | # PyCharm 157 | # JetBrains specific template is maintained in a separate JetBrains.gitignore that can 158 | # be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore 159 | # and can be added to the global gitignore or merged into this file. For a more nuclear 160 | # option (not recommended) you can uncomment the following to ignore the entire idea folder. 161 | #.idea/ 162 | 163 | # macOS 164 | # .DS_Store files are created by macOS Finder, the default file manager on macOS systems. 165 | # These files are hidden and store custom attributes of a folder, such as the position of 166 | # icons, background images, and other metadata related to the folder's display settings. 167 | # If you are using macOS, you should uncomment this option. 168 | .DS_Store 169 | -------------------------------------------------------------------------------- /cli.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | from groq import Chat, Text 3 | 4 | def main(): 5 | class CustomFormatter(argparse.ArgumentDefaultsHelpFormatter, 6 | argparse.RawDescriptionHelpFormatter): 7 | pass 8 | parser = argparse.ArgumentParser( 9 | description=""" 10 | ------------------------------------------------------------------ 11 | Groq AI Toolkit 12 | API Wrapper & Command-line Interface 13 | [v1.0.1] by @rmncldyo 14 | ------------------------------------------------------------------ 15 | 16 | An API wrapper & CLI for Groq AI's breakthrough LPU Inference Engine, allowing seamless interactions with the latest LLMs. 17 | 18 | | Option(s) | Description | Example Usage | 19 | |------------------------|--------------------------------------|--------------------------------------------------------| 20 | | -c, --chat | Enable chat mode | --chat | 21 | | -t, --text | Enable text mode | --text | 22 | | -p, --prompt | User prompt | --prompt "Explain the importance of low latency LLMs." | 23 | | -a, --api_key | API key for authentication | --api_key "api_key_goes_here" | 24 | | -m, --model | The model you would like to use | --model "llama3-8b-8192" | 25 | | -sp, --system_prompt | System prompt (instructions) | --system_prompt "You are a helpful assistant." | 26 | | -st, --stream | Enable streaming mode for responses | --stream | 27 | | -js, --json | Output response in JSON format | --json | 28 | | -tm, --temperature | Sampling temperature | --temperature 0.7 | 29 | | -mt, --max_tokens | Maximum number of tokens to generate | --max_tokens 1024 | 30 | | -tp, --top_p | Nucleus sampling threshold | --top_p 0.9 | 31 | | -sd, --seed | Seed for sampling | --seed 123456789 | 32 | | -ss, --stop | Stop sequences for completion | --stop ", 6" | 33 | """, 34 | formatter_class=CustomFormatter, 35 | epilog="For detailed usage information, visit our ReadMe here: github.com/RMNCLDYO/groq-ai-toolkit" 36 | ) 37 | parser.add_argument('-c', '--chat', action='store_true', help='Enable chat mode') 38 | parser.add_argument('-t', '--text', action='store_true', help='Enable text mode') 39 | parser.add_argument('-p', '--prompt', type=str, help='User prompt', metavar='') 40 | parser.add_argument('-a', '--api_key', type=str, help='API key for authentication', metavar='') 41 | parser.add_argument('-m', '--model', type=str, default='llama3-8b-8192', help='The model you would like to use', metavar='') 42 | parser.add_argument('-sp', '--system_prompt', type=str, help='System prompt (instructions)', metavar='') 43 | parser.add_argument('-st', '--stream', action='store_true', help='Enable streaming mode for responses') 44 | parser.add_argument('-js', '--json', action='store_true', help='Output response in JSON format') 45 | parser.add_argument('-tm', '--temperature', type=float, help='Sampling temperature', metavar='') 46 | parser.add_argument('-mt', '--max_tokens', type=int, help='Maximum number of tokens to generate', metavar='') 47 | parser.add_argument('-tp', '--top_p', type=float, help='Nucleus sampling threshold', metavar='') 48 | parser.add_argument('-sd', '--seed', type=int, help='Seed for sampling', metavar='') 49 | parser.add_argument('-ss', '--stop', type=str, nargs='+', help='Stop sequences for completion', metavar='') 50 | 51 | args = parser.parse_args() 52 | 53 | if args.chat: 54 | Chat().run(args.api_key, args.model, args.prompt, args.system_prompt, args.stream, args.json, args.temperature, args.max_tokens, args.top_p, args.seed, args.stop) 55 | elif args.text: 56 | Text().run(args.api_key, args.model, args.prompt, args.system_prompt, args.stream, args.json, args.temperature, args.max_tokens, args.top_p, args.seed, args.stop) 57 | else: 58 | print("Error: Please specify a mode to use. Use --help for more information.") 59 | exit() 60 | 61 | if __name__ == "__main__": 62 | main() 63 | -------------------------------------------------------------------------------- /groq.py: -------------------------------------------------------------------------------- 1 | from client import Client 2 | 3 | class Chat: 4 | def __init__(self): 5 | self.client = None 6 | 7 | def run(self, api_key=None, model=None, prompt=None, system_prompt=None, stream=None, json=None, temperature=None, max_tokens=None, top_p=None, seed=None, stop=None): 8 | 9 | self.client = Client(api_key=api_key) 10 | self.model = model if model else self.client.config.get('model') 11 | 12 | conversation_history = [] 13 | 14 | if system_prompt: 15 | conversation_history.append({"role": "system", "content": system_prompt}) 16 | 17 | print("Type 'exit' or 'quit' at any time to end the conversation.\n") 18 | 19 | print("Assistant: Hello! How can I assist you today?") 20 | while True: 21 | if prompt: 22 | user_input = prompt.strip() 23 | print(f"User: {user_input}") 24 | prompt = None 25 | else: 26 | user_input = input("User: ").strip() 27 | if user_input.lower() in ['exit', 'quit']: 28 | print("\nThank you for using the Groq AI toolkit. Have a great day!") 29 | break 30 | 31 | if not user_input: 32 | print("Invalid input detected. Please enter a valid message.") 33 | continue 34 | 35 | if json: 36 | if stream: 37 | print("Error: JSON mode does not support streaming.") 38 | exit(1) 39 | if stop: 40 | print("Error: JSON mode does not support stop sequences.") 41 | exit(1) 42 | if "json" not in user_input: 43 | user_input = user_input + " | Respond in JSON. The JSON schema should include at minimum: {'response': 'string', 'status': 'string'}" 44 | response_format = {"type": "json_object"} 45 | else: 46 | response_format = None 47 | 48 | conversation_history.append({"role": "user", "content": user_input}) 49 | 50 | data = { 51 | "messages": conversation_history, 52 | "response_format": response_format, 53 | "model": self.model, 54 | "temperature": temperature, 55 | "max_tokens": max_tokens, 56 | "top_p": top_p, 57 | "stream": stream, 58 | "seed": seed, 59 | "stop": stop 60 | } 61 | data = {k: v for k, v in data.items() if v is not None} 62 | 63 | endpoint = self.client.config.get('completions_endpoint') 64 | 65 | if stream: 66 | response = self.client.stream_post(endpoint, data) 67 | assistant_response = response 68 | else: 69 | response = self.client.post(endpoint, data) 70 | assistant_response = response 71 | print(f"Assistant: {assistant_response}") 72 | conversation_history.append({"role": "assistant", "content": assistant_response}) 73 | 74 | class Text: 75 | def __init__(self): 76 | self.client = None 77 | 78 | def run(self, api_key=None, model=None, prompt=None, system_prompt=None, stream=None, json=None, temperature=None, max_tokens=None, top_p=None, seed=None, stop=None): 79 | 80 | self.client = Client(api_key=api_key) 81 | self.model = model if model else self.client.config.get('model') 82 | self.max_tokens = max_tokens if max_tokens else 1024 83 | 84 | if not prompt: 85 | print("Error: { Invalid input detected }. Please enter a valid message.") 86 | exit(1) 87 | 88 | if json: 89 | if stream: 90 | print("Error: JSON mode does not support streaming.") 91 | exit(1) 92 | if stop: 93 | print("Error: JSON mode does not support stop sequences.") 94 | exit(1) 95 | if "json" not in prompt: 96 | prompt = prompt + " | Respond in JSON. The JSON schema should include at minimum: {'response': 'string', 'status': 'string'}" 97 | response_format = {"type": "json_object"} 98 | else: 99 | response_format = None 100 | 101 | if system_prompt: 102 | message = [ 103 | {"role": "system", "content": system_prompt}, 104 | {"role": "user", "content": prompt} 105 | ] 106 | else: 107 | message = [{"role": "user", "content": prompt}] 108 | 109 | data = { 110 | "messages": message, 111 | "response_format": response_format, 112 | "model": self.model, 113 | "temperature": temperature, 114 | "max_tokens": max_tokens, 115 | "top_p": top_p, 116 | "stream": stream, 117 | "seed": seed, 118 | "stop": stop 119 | } 120 | data = {k: v for k, v in data.items() if v is not None} 121 | 122 | endpoint = self.client.config.get('completions_endpoint') 123 | 124 | if stream: 125 | response = self.client.stream_post(endpoint, data) 126 | assistant_response = response 127 | else: 128 | response = self.client.post(endpoint, data) 129 | assistant_response = response 130 | print(f"Assistant: {assistant_response}") 131 | 132 | print("\nThank you for using the Groq AI toolkit. Have a great day!") -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 |

2 | 3 | Groq AI 4 | 5 |

6 | 7 |

8 | 9 | Version 10 | 11 |

12 | 13 |

14 | maintained - yes 15 | contributions - welcome 16 |

17 | 18 |

19 | 20 | 21 | 22 | 23 | Groq AI 24 | 25 | 26 |

27 | 28 | ## Overview 29 | The Groq AI Toolkit makes it easy to use Groq's breakthrough LPU Inference Engine for creating chatbots and generating text with near-real-time responses (800 tokens/sec). It's designed for everyone, from beginners to experienced developers, allowing quick addition of AI features to projects with simple commands. While it offers simplicity and lightweight integration, it doesn't compromise on power; experienced developers can access the full suite of advanced options available via the API, ensuring robust customization and control. This toolkit is perfect for those looking to efficiently tap into advanced AI without getting bogged down in technical details, yet it still provides the depth needed for complex project requirements. 30 | 31 | ## Key Features 32 | - **Conversational AI**: Create interactive, real-time chat experiences (chatbots) or AI assistants. 33 | - **Text Generation**: Produce coherent and contextually relevant text and answers from simple prompts. 34 | - **Ultra-Fast Performance**: Achieve near-real-time responses with an impressive speed of 800 tokens per second. 35 | - **Highly Customizable**: Tailor settings like streaming, JSON outputs, system prompts and more to suit your specific requirements. 36 | - **Lightweight Integration**: Efficiently designed with minimal dependencies, requiring only the `requests` package for core functionality. 37 | 38 | ## Prerequisites 39 | - `Python 3.x` 40 | - An API key from Groq AI 41 | 42 | ## Dependencies 43 | The following Python packages are required: 44 | - `requests`: For making HTTP requests to Groq's API. 45 | 46 | The following Python packages are optional: 47 | - `python-dotenv`: For managing API keys and other environment variables. 48 | 49 | ## Installation 50 | To use the Groq AI Toolkit, clone the repository to your local machine and install the required Python packages. 51 | 52 | Clone the repository: 53 | ```bash 54 | git clone https://github.com/RMNCLDYO/groq-ai-toolkit.git 55 | ``` 56 | 57 | Navigate to the repositories folder: 58 | ```bash 59 | cd groq-ai-toolkit 60 | ``` 61 | 62 | Install the required dependencies: 63 | ```bash 64 | pip install -r requirements.txt 65 | ``` 66 | 67 | ## Configuration 68 | 1. Obtain an API key from [Groq AI](https://console.groq.com/). 69 | 2. You have three options for managing your API key: 70 |
71 | Click here to view the API key configuration options 72 | 73 | - **Setting it as an environment variable on your device (recommended for everyday use)** 74 | - Navigate to your terminal. 75 | - Add your API key like so: 76 | ```shell 77 | export GROQ_API_KEY=your_api_key 78 | ``` 79 | This method allows the API key to be loaded automatically when using the wrapper or CLI. 80 | 81 | - **Using an .env file (recommended for development):** 82 | - Install python-dotenv if you haven't already: `pip install python-dotenv`. 83 | - Create a .env file in the project's root directory. 84 | - Add your API key to the .env file like so: 85 | ```makefile 86 | GROQ_API_KEY=your_api_key 87 | ``` 88 | This method allows the API key to be loaded automatically when using the wrapper or CLI, assuming you have python-dotenv installed and set up correctly. 89 | 90 | - **Direct Input:** 91 | - If you prefer not to use a `.env` file, you can directly pass your API key as an argument to the CLI or the wrapper functions. 92 | 93 | ***CLI*** 94 | ```shell 95 | --api_key "your_api_key" 96 | ``` 97 | ***Wrapper*** 98 | ```shell 99 | api_key="your_api_key" 100 | ``` 101 | This method requires manually inputting your API key each time you initiate an API call, ensuring flexibility for different deployment environments. 102 |
103 | 104 | ## Usage 105 | The Groq AI Toolkit can be used in two different modes: `Chat` and `Text`. Each mode is designed for specific types of interactions with Groq's open models. 106 | 107 | ## Chat Mode 108 | Chat mode is intended for chatting with an AI model (similar to a chatbot) or building conversational applications. 109 | 110 | #### Example Usage 111 | 112 | ***CLI*** 113 | ```bash 114 | python cli.py --chat 115 | ``` 116 | 117 | ***Wrapper*** 118 | ```python 119 | from groq import Chat 120 | 121 | Chat().run() 122 | ``` 123 | 124 | > An executable version of this example can be found [here](./examples/example_chat.py). (*You must move this file to the root folder before running the program.*) 125 | 126 | ## Text Mode 127 | Text mode is suitable for generating text content based on a provided prompt. 128 | 129 | #### Example Usage 130 | 131 | ***CLI*** 132 | ```bash 133 | python cli.py --text --prompt "Explain the importance of low latency LLMs." 134 | ``` 135 | 136 | ***Wrapper*** 137 | ```python 138 | from groq import Text 139 | 140 | Text().run(prompt="Explain the importance of low latency LLMs.") 141 | ``` 142 | 143 | > An executable version of this example can be found [here](./examples/example_text.py). (*You must move this file to the root folder before running the program.*) 144 | 145 | ## Advanced Configuration 146 | 147 | ### CLI and Wrapper Options 148 | | **Description** | **CLI Flags** | **CLI Usage** | **Wrapper Usage** | 149 | |--------------------------------------|--------------------------|--------------------------------------------------------|------------------------------------------------------| 150 | | Enable chat mode | `-c`, `--chat` | --chat | *See mode usage above.* | 151 | | Enable text mode | `-t`, `--text` | --text | *See mode usage above.* | 152 | | User prompt | `-p`, `--prompt` | --prompt "Explain the importance of low latency LLMs." | prompt="Explain the importance of low latency LLMs." | 153 | | API key for authentication | `-a`, `--api_key` | --api_key "your_api_key" | api_key="your_api_key" | 154 | | Model name | `-m`, `--model` | --model "llama-3.1-8b-instant" | model="llama-3.1-8b-instant" | 155 | | System prompt (instructions) | `-sp`, `--system_prompt` | --system_prompt "You are a helpful assistant." | system_prompt="You are a helpful assistant." | 156 | | Enable streaming mode | `-st`, `--stream` | --stream | stream=True | 157 | | Enable json mode | `-js`, `--json` | --json | json=True | 158 | | Sampling temperature | `-tm`, `--temperature` | --temperature 0.7 | temperature=0.7 | 159 | | Maximum number of tokens to generate | `-mt`, `--max_tokens` | --max_tokens 1024 | max_tokens=1024 | 160 | | Nucleus sampling threshold | `-tp`, `--top_p` | --top_p 0.9 | top_p=0.9 | 161 | | Seed used for sampling | `-sd`, `-seed` | --seed 123456789 | seed=123456789 | 162 | | Stop sequence for completion | `-ss`, `--stop` | --stop "\n" | stop="\n" | 163 | 164 | > *To exit the program at any time, you can type **`exit`** or **`quit`**. This command works similarly whether you're interacting with the program via the CLI or through the Python wrapper ensuring that you can easily and safely conclude your work with the Groq AI Toolkit without having to resort to interrupt signals or forcibly closing the terminal or command prompt.* 165 | 166 | ## Available Models 167 | 168 | | **Model** | **Max Tokens** | 169 | |-------------------------------------|----------------| 170 | | `llama-3.1-70b-versatile` (Preview) | 131072 | 171 | | `llama-3.1-8b-instant` (Preview) | 131072 | 172 | | `llava-v1.5-7b-4096-preview` | 4096 | 173 | | `llama-guard-3-8b` | 8192 | 174 | | `llama3-70b-8192` | 8192 | 175 | | `llama3-8b-8192` | 8192 | 176 | | `llama2-70b-4096` | 4096 | 177 | | `mixtral-8x7b-32768` | 32768 | 178 | | `gemma-7b-it` | 8192 | 179 | | `gemma2-9b-it` | 8192 | 180 | 181 | > [!NOTE] 182 | > *While in Preview, Groq is limiting 3.1 models to max tokens of 8k.* 183 | 184 | ## Contributing 185 | Contributions are welcome! 186 | 187 | Please refer to [CONTRIBUTING.md](.github/CONTRIBUTING.md) for detailed guidelines on how to contribute to this project. 188 | 189 | ## Reporting Issues 190 | Encountered a bug? We'd love to hear about it. Please follow these steps to report any issues: 191 | 192 | 1. Check if the issue has already been reported. 193 | 2. Use the [Bug Report](.github/ISSUE_TEMPLATE/bug_report.md) template to create a detailed report. 194 | 3. Submit the report [here](https://github.com/RMNCLDYO/groq-ai-toolkit/issues). 195 | 196 | Your report will help us make the project better for everyone. 197 | 198 | ## Feature Requests 199 | Got an idea for a new feature? Feel free to suggest it. Here's how: 200 | 201 | 1. Check if the feature has already been suggested or implemented. 202 | 2. Use the [Feature Request](.github/ISSUE_TEMPLATE/feature_request.md) template to create a detailed request. 203 | 3. Submit the request [here](https://github.com/RMNCLDYO/groq-ai-toolkit/issues). 204 | 205 | Your suggestions for improvements are always welcome. 206 | 207 | ## Versioning and Changelog 208 | Stay up-to-date with the latest changes and improvements in each version: 209 | 210 | - [CHANGELOG.md](.github/CHANGELOG.md) provides detailed descriptions of each release. 211 | 212 | ## Security 213 | Your security is important to us. If you discover a security vulnerability, please follow our responsible disclosure guidelines found in [SECURITY.md](.github/SECURITY.md). Please refrain from disclosing any vulnerabilities publicly until said vulnerability has been reported and addressed. 214 | 215 | ## License 216 | Licensed under the MIT License. See [LICENSE](LICENSE) for details. 217 | --------------------------------------------------------------------------------