├── LICENSE
├── README.md
└── training.py
/LICENSE:
--------------------------------------------------------------------------------
1 | MIT License
2 |
3 | Copyright (c) 2023 Reed Bender
4 |
5 | Permission is hereby granted, free of charge, to any person obtaining a copy
6 | of this software and associated documentation files (the "Software"), to deal
7 | in the Software without restriction, including without limitation the rights
8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9 | copies of the Software, and to permit persons to whom the Software is
10 | furnished to do so, subject to the following conditions:
11 |
12 | The above copyright notice and this permission notice shall be included in all
13 | copies or substantial portions of the Software.
14 |
15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21 | SOFTWARE.
22 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | OpenAgent
2 |
3 |
4 | 
5 |
6 |
7 | 
8 |
9 |
10 |
11 | 
12 |
13 |
14 | 
15 |
16 |
17 |
18 |
19 | 
20 |
21 |
22 | Join the community on Discord!
23 |
24 |
25 | 
26 |
27 |
28 |
29 | ---
30 |
31 |
32 | An open-source initiative dedicated to enhancing the capabilities of Large Language Models to act as autonomous agents.
33 |
34 | ## Mission
35 |
36 | OpenAgent is committed to spearheading the use of Large Language Models (LLMs) as efficient, autonomous agents. We strive to enhance the performance, usability, and adaptability of these models through a collaborative, open-source initiative. We aim to establish a decentralized, open-source, and community-driven agent ecosystem that is independent of proprietary models like OpenAI's GPT-4.
37 |
38 | Our vision extends to creating tools that can be widely customized and economically deployed, sparking a new generation of agents capable of addressing a broad spectrum of problems. However, our mission is not only to develop and fine-tune agent-specific LLMs, but also to foster a community of researchers, developers, and enthusiasts who share our vision for the future of autonomous agents. At OpenAgent, we value innovation, continuous improvement, and inclusivity.
39 |
40 | ## Roadmap
41 |
42 | OpenAgent is currently in its early stages. While the initial groundwork has been laid, our community is actively working on fleshing out a comprehensive strategy for the project's future. Our ambitions are high and the journey has just begun, and we're excited to shape the trajectory of this initiative.
43 |
44 | A key facet of our early development is the creation of an agent-specific markup language, AgentML. This language is being designed with a special emphasis on maximizing the efficiency and effectiveness of Large Language Models in the realm of autonomous agents.
45 |
46 | Upon the completion and stabilization of AgentML, our next focus will be on the collection and synthesis of training data. A robust dataset will enable us to fine-tune LLMs for optimal performance in various agent-based applications.
47 |
48 | In terms of the specific models we plan to work with, both MPT-7B and Falcon-40B are being strongly considered. These models are noted for their impressive abilities and potential, making them fitting subjects for our fine-tuning efforts.
49 |
50 | As we navigate through the early stages of our journey, we encourage participation and contribution in various forms – from coding to idea sharing and constructive feedback. We envision a future where LLMs play a crucial role in shaping the way we interact with technology and the world. Join us as we collectively push the boundaries of what's possible with LLMs and autonomous agents.
51 |
52 | ## Resources
53 | * [🦜🔗 LangChain Structured Chat Agent](https://python.langchain.com/en/latest/modules/agents/agents/examples/structured_chat.html)
54 | * [🤗 MPT-7B on Hugging Face](https://huggingface.co/mosaicml/mpt-7b)
55 | * [🤗 Falcon-40B on Hugging Face](https://huggingface.co/tiiuae/falcon-40b)
56 | * [Fine-Tuning MPT-7B](https://www.youtube.com/watch?v=KSlWkrByc0o&t=17s)
57 | * [LoRa for Fine-Tuning](https://bdtechtalks.com/2023/05/22/what-is-lora/)
58 | * [Sophia, a Second-order Optimizer](https://arxiv.org/abs/2305.14342)
59 | * [🦜🔗 LangChain-Datasets](https://huggingface.co/LangChainDatasets)
60 |
--------------------------------------------------------------------------------
/training.py:
--------------------------------------------------------------------------------
1 | import sagemaker
2 | from sagemaker.huggingface import HuggingFace
3 | import boto3
4 |
5 | try:
6 | role = sagemaker.get_execution_role()
7 | except ValueError:
8 | iam = boto3.client('iam')
9 | role = iam.get_role(RoleName='sagemaker_execution_role')['Role']['Arn']
10 |
11 | hyperparameters = {
12 | 'model_name_or_path': 'mosaicml/mpt-7b',
13 | 'output_dir': '/opt/ml/model'
14 | # add your remaining hyperparameters
15 | # more info here https://github.com/huggingface/transformers/tree/v4.26.0/examples/pytorch/language-modeling
16 | }
17 |
18 | # git configuration to download our fine-tuning script
19 | git_config = {
20 | 'repo': 'https://github.com/huggingface/transformers.git', 'branch': 'v4.26.0'}
21 |
22 | # creates Hugging Face estimator
23 | huggingface_estimator = HuggingFace(
24 | entry_point='run_clm.py',
25 | source_dir='./examples/pytorch/language-modeling',
26 | instance_type='ml.p3.2xlarge',
27 | instance_count=1,
28 | role=role,
29 | git_config=git_config,
30 | transformers_version='4.26.0',
31 | pytorch_version='1.13.1',
32 | py_version='py39',
33 | hyperparameters=hyperparameters
34 | )
35 |
36 | # starting the train job
37 | huggingface_estimator.fit()
38 |
--------------------------------------------------------------------------------