├── LICENSE ├── README.md ├── ReadMeCrewAI.ipynb ├── asset ├── FINAL_README.md ├── code_content.txt ├── model.jpg └── style_found_output.txt └── readmecrewai.py /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2024 Mahan 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # 🚀 Automated README Generator using Multi-Agent CrewAI 🤖 2 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1LQAnuuqdHNWhfrBc6KH03WDKyTlc4AeR?usp=sharing) 3 | ![Python](https://img.shields.io/badge/python-3.7%20%7C%203.8%20%7C%203.9-blue) 4 | ![Status](https://img.shields.io/badge/status-active-green) 5 | 6 | Creating a perfect README has always been a crucial part of any project. It's the first impression developers and collaborators get from your repository. However, crafting a well-structured, detailed, and visually appealing README takes time and effort. That's why I decided to automate this process using **CrewAI** and **LLaMA 3 - 70B**. 7 | 8 | The core of this system revolves around **three collaborative crews**: 9 | 1. **Style Crew**: This crew analyzes a sample README that reflects my preferred style, such as the one from [RL Practices - DQN](https://github.com/MahanVeisi8/RL_practices/tree/main/Cartpole/1%20-%20DQN). 10 | 2. **Code Crew**: This crew digs deep into the project code, like in [Readahead Optimization using ML Models](https://github.com/MahanVeisi8/Readahead-Optimization-Using-ML-Models), extracting key information to ensure the README is accurate and thorough. 11 | 3. **README Generator Crew**: Combining insights from the Style and Code Crews, this team writes the final README, complete with structure, details, and creative touches. 12 | 13 | The project overview below demonstrates how the three crews work together, processing inputs like sample READMEs and source code, powered by **CrewAI** and the powerful **LLaMA 3 - 70B** model. 14 | 15 | ![CrewAI Project Overview](asset/model.jpg) 16 | 17 | With this setup, I’ve automated the task of producing top-quality, personalized READMEs that align perfectly with my style preferences—making the process faster, consistent, and fun! 18 | 19 | ## 🧠 **Easy-to-Use LLM Setup** 20 | 21 | This project leverages the powerful **LLaMA 3 - 70B** language model to analyze and generate natural language content. To make the project accessible for all users, we’ve implemented everything in **Google Colab**. You can run the entire pipeline from any device, regardless of its computational power, and see the results in real-time. 22 | 23 | Simply click the Colab badge to start: 24 | 25 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1LQAnuuqdHNWhfrBc6KH03WDKyTlc4AeR?usp=sharing) 26 | 27 | --- 28 | 29 | ### **Why LLaMA 3 - 70B?** 30 | LLaMA 3 - 70B offers a great balance between computational efficiency and power, making it ideal for natural language tasks without the high cost of stronger models like GPT-4. While more powerful models like GPT-4, Claude, and PaLM 2 are available, **LLaMA 3 - 70B** performs impressively in the free tier, allowing for effective automation without the need for expensive paid APIs. 31 | 32 | ### **Setting Up the LLM** 33 | 34 | To get started with the LLM, we integrate it with **Langchain** and **CREW AI**, which allow us to break down complex README generation tasks into smaller, manageable steps. Follow these simple steps to set up the environment: 35 | 36 | 1. **Install the required libraries**: 37 | ```bash 38 | pip install --upgrade langchain langchain_core crewai langchain_groq 39 | ``` 40 | 41 | 2. **Set up your GROQ API key** by adding it to your environment variables: 42 | ```python 43 | import os 44 | os.environ["GROQ_API_KEY"] = "your-key-here" 45 | ``` 46 | 47 | 3. **Initialize the LLM** with the **LLaMA 3 - 70B** model: 48 | ```python 49 | from langchain_groq import ChatGroq 50 | 51 | # Setting up the LLM (GROQ_LLM) 52 | GROQ_LLM = ChatGroq( 53 | api_key=os.getenv("GROQ_API_KEY"), 54 | model="llama3-70b-8192" 55 | ) 56 | ``` 57 | 58 | This setup allows you to fully automate the README generation process without worrying about infrastructure or heavy computation. 59 | 60 | ## 🤖 **CREW AI: Agents and Their Roles** 61 | 62 | To automate the creation of the README, the project divides tasks among three specialized agents—each responsible for a distinct aspect of the documentation process. Here’s a breakdown: 63 | 64 | ### 1. **Style Crew** 📝 65 | The **Style Crew** ensures that the generated README matches the preferred style by analyzing a sample README provided by the user. In this case, we used the README from [RL Practices - DQN](https://github.com/MahanVeisi8/RL_practices/tree/main/Cartpole/1%20-%20DQN). 66 | 67 | - **Directory Analyzer**: Scans the directories to locate the README.md files for analysis. 68 | - **Content Analyzer**: Reads and extracts the style, structure, and key elements from the sample README. 69 | - **Template Creator**: Based on the analysis, this agent generates a reusable README template, which serves as a skeleton for the final output. 70 | 71 | ### Sample Output from the Style Crew: 72 | This is an excerpt from the **first part of the final analyzed README** that the Style Crew generated based on the preferred style: 73 | 74 | ```md 75 | # Introduction 76 | 77 | This repository provides a comprehensive implementation of [Project Name], a [brief description of the project]. The project aims to [briefly describe the project's objective]. 78 | 79 | ## Setup 80 | 81 | ### Prerequisites 82 | 83 | * List of prerequisites, e.g., Python version, libraries, etc. 84 | * Installation instructions, e.g., pip install, etc. 85 | 86 | ### Environment Setup 87 | 88 | * Instructions for setting up the environment, e.g., creating a virtual environment, etc. 89 | 90 | ## Implementing Key Components 91 | 92 | ### [Component 1] 93 | 94 | * Brief description of the component 95 | * Code snippet or example 96 | * Explanation of the component's functionality 97 | 98 | ### [Component 2] 99 | 100 | * Brief description of the component 101 | * Code snippet or example 102 | * Explanation of the component's functionality 103 | 104 | ## Results and Performance Analysis 105 | 106 | ### [Result 1] 107 | 108 | * Description of the result, including any visual aids, e.g., images, tables, etc. 109 | * Analysis of the result, including any relevant metrics or statistics 110 | 111 | ### [Result 2] 112 | 113 | * Description of the result, including any visual aids, e.g., images, tables, etc. 114 | * Analysis of the result, including any relevant metrics or statistics 115 | 116 | ## Summary 117 | 118 | This project provides a comprehensive implementation of [Project Name], demonstrating [key findings or achievements]. The results show [briefly describe the results], highlighting the effectiveness of [key components or techniques]. 119 | ``` 120 | 121 | *For the **full style template found** by the Style Crew, you can check the `style_found_output.txt` in the [asset directory](asset/style_found_output.txt).* 122 | 123 | 124 | ### 2. **Code Crew** 💻 125 | The Code Crew dives into the project's codebase to extract technical details that need to be documented in the README. This crew ensures that important components like algorithms, functions, and models are clearly described. 126 | 127 | - **File Finder**: Identifies relevant code files and passes them to the next agent for analysis. 128 | - **ML Analyzer**: Reads and analyzes machine learning methods and their implementation in the code. 129 | - **Report Writer**: Generates a structured report based on the analysis, which is then included in the README. 130 | 131 | For instance, in [Readahead Optimization using ML Models](https://github.com/MahanVeisi8/Readahead-Optimization-Using-ML-Models), the Code Crew extracted critical details about the training process, models used, and performance metrics. 132 | 133 | --- 134 | 135 | ### 3. **README Generator Crew** 📄 136 | 137 | The **README Generator Crew** is responsible for compiling the insights from both the **Style Crew** and **Code Crew** into a polished, professional README. 138 | 139 | - **File Reader**: Reads the output from the Style and Code Crews to merge content. 140 | - **Content Merger**: Combines the style template with the technical content to produce a coherent and structured README draft. 141 | - **README Editor**: Finalizes the draft by adding creative touches like emojis, formatting enhancements, and technical precision. 142 | 143 | --- 144 | 145 | ## 📄 Sample README Output 146 | 147 | Here’s an example of the README generated by CrewAI: 148 | 149 | ### Final README Content: 150 | 151 | ```md 152 | Final README Content: Here is the rewritten README file: 153 | 154 | # Optimizing Readahead Feature of Linux Page Cache using Machine Learning 📊💻 155 | 156 | This repository provides a comprehensive implementation of optimizing the Readahead feature of the Linux Page Cache under varying workloads using machine learning techniques. 157 | 158 | ## Setup 💻 159 | 160 | ### Prerequisites 161 | 162 | * Python version: 3.x 163 | * Libraries: scikit-learn, numpy, pandas, etc. 164 | * Installation instructions: `pip install -r requirements.txt` 165 | 166 | ### Environment Setup 167 | 168 | * Create a virtual environment: `python -m venv env` 169 | * Activate the virtual environment: `source env/bin/activate` 170 | 171 | ## Implementing Machine Learning Components 🤖 172 | 173 | ### Feature Importance Analysis 174 | 175 | * Brief description: Random Forest Classifier was used to analyze feature importance, and non-important features were removed. 176 | * Code snippet or example: [Insert code snippet] 177 | * Explanation of the component's functionality: This component is used to identify the most important features that affect the Readahead size. 178 | 179 | ### Dimensionality Reduction 180 | 181 | * Brief description: T-SNE was used to visualize the data in 2D. 182 | * Code snippet or example: [Insert code snippet] 183 | * Explanation of the component's functionality: This component is used to reduce the dimensionality of the data and visualize it in 2D. 184 | 185 | ### Model Training 🚀 186 | 187 | * **Neural Network** 188 | + Brief description: MLPClassifier was used with hidden layers of 64 and 32 neurons. 189 | + Code snippet or example: [Insert code snippet] 190 | + Explanation of the component's functionality: This component is used to train a neural network model to classify workload types and suggest optimal Readahead sizes. 191 | * **Decision Tree** 192 | + Brief description: DecisionTreeClassifier was used. 193 | + Code snippet or example: [Insert code snippet] 194 | + Explanation of the component's functionality: This component is used to train a decision tree model to classify workload types and suggest optimal Readahead sizes. 195 | * **Random Forest** 196 | + Brief description: RandomForestClassifier was used with 100 estimators. 197 | + Code snippet or example: [Insert code snippet] 198 | + Explanation of the component's functionality: This component is used to train a random forest model to classify workload types and suggest optimal Readahead sizes. 199 | 200 | ## Results and Performance Analysis 📊 201 | 202 | ### Model Comparison 203 | 204 | | Model | Accuracy | Notes | 205 | |------------------|-----------|---------------------------------------------| 206 | | Decision Tree | 100.00% | Simple, interpretable, perfect accuracy | 207 | | Neural Network | 99.85% | High accuracy, complex model with slight variability in precision | 208 | | Random Forest | 100.00% | Combines multiple trees for perfect accuracy and generalization | 209 | 210 | ### Performance Comparison 211 | 212 | * The results show that both the Decision Tree and Random Forest models achieved perfect accuracy, while the Neural Network model had a slightly lower accuracy. 213 | 214 | ## Summary 📚 215 | 216 | This project provides a comprehensive implementation of optimizing the Readahead feature of the Linux Page Cache under varying workloads using machine learning techniques, demonstrating the effectiveness of machine learning techniques in optimizing the Readahead feature under varying workloads. The results show that the Random Forest model stands out for its combination of accuracy and interpretability, making it a strong candidate for real-time systems that require dynamic adjustment of Readahead sizes based on current workloads. 217 | Final README saved to: FINAL_README.md 218 | ``` 219 | 220 | *For the complete output, check out the file [FINAL_README.md](asset/FINAL_README.md)* 221 | 222 | --- 223 | 224 | ## 🔚 Conclusion 225 | 226 | **CrewAI** effectively automates the creation of high-quality, structured documentation with minimal manual effort. Using **LLaMA 3 - 70B**, the project balances efficiency and content generation, but more powerful LLMs like **GPT-4** or **PaLM 2** could enhance analysis for larger, more complex projects. 227 | 228 | For technical challenges like [Readahead Optimization](https://github.com/MahanVeisi8/Readahead-Optimization-Using-ML-Models), **CrewAI** proves that well-maintained, comprehensive documentation can be generated efficiently without sacrificing quality. 229 | 230 | -------------------------------------------------------------------------------- /asset/FINAL_README.md: -------------------------------------------------------------------------------- 1 | Here is the rewritten README file: 2 | 3 | # Optimizing Readahead Feature of Linux Page Cache using Machine Learning 📊💻 4 | 5 | This repository provides a comprehensive implementation of optimizing the Readahead feature of the Linux Page Cache under varying workloads using machine learning techniques. 6 | 7 | ## Setup 💻 8 | 9 | ### Prerequisites 10 | 11 | * Python version: 3.x 12 | * Libraries: scikit-learn, numpy, pandas, etc. 13 | * Installation instructions: `pip install -r requirements.txt` 14 | 15 | ### Environment Setup 16 | 17 | * Create a virtual environment: `python -m venv env` 18 | * Activate the virtual environment: `source env/bin/activate` 19 | 20 | ## Implementing Machine Learning Components 🤖 21 | 22 | ### Feature Importance Analysis 23 | 24 | * Brief description: Random Forest Classifier was used to analyze feature importance, and non-important features were removed. 25 | * Code snippet or example: [Insert code snippet] 26 | * Explanation of the component's functionality: This component is used to identify the most important features that affect the Readahead size. 27 | 28 | ### Dimensionality Reduction 29 | 30 | * Brief description: T-SNE was used to visualize the data in 2D. 31 | * Code snippet or example: [Insert code snippet] 32 | * Explanation of the component's functionality: This component is used to reduce the dimensionality of the data and visualize it in 2D. 33 | 34 | ### Model Training 🚀 35 | 36 | * **Neural Network** 37 | + Brief description: MLPClassifier was used with hidden layers of 64 and 32 neurons. 38 | + Code snippet or example: [Insert code snippet] 39 | + Explanation of the component's functionality: This component is used to train a neural network model to classify workload types and suggest optimal Readahead sizes. 40 | * **Decision Tree** 41 | + Brief description: DecisionTreeClassifier was used. 42 | + Code snippet or example: [Insert code snippet] 43 | + Explanation of the component's functionality: This component is used to train a decision tree model to classify workload types and suggest optimal Readahead sizes. 44 | * **Random Forest** 45 | + Brief description: RandomForestClassifier was used with 100 estimators. 46 | + Code snippet or example: [Insert code snippet] 47 | + Explanation of the component's functionality: This component is used to train a random forest model to classify workload types and suggest optimal Readahead sizes. 48 | 49 | ## Results and Performance Analysis 📊 50 | 51 | ### Model Comparison 52 | 53 | | Model | Accuracy | Notes | 54 | |------------------|-----------|---------------------------------------------| 55 | | Decision Tree | 100.00% | Simple, interpretable, perfect accuracy | 56 | | Neural Network | 99.85% | High accuracy, complex model with slight variability in precision | 57 | | Random Forest | 100.00% | Combines multiple trees for perfect accuracy and generalization | 58 | 59 | ### Performance Comparison 60 | 61 | * The results show that both the Decision Tree and Random Forest models achieved perfect accuracy, while the Neural Network model had a slightly lower accuracy. 62 | 63 | ## Summary 📚 64 | 65 | This project provides a comprehensive implementation of optimizing the Readahead feature of the Linux Page Cache under varying workloads using machine learning techniques, demonstrating the effectiveness of machine learning techniques in optimizing the Readahead feature under varying workloads. The results show that the Random Forest model stands out for its combination of accuracy and interpretability, making it a strong candidate for real-time systems that require dynamic adjustment of Readahead sizes based on current workloads. -------------------------------------------------------------------------------- /asset/code_content.txt: -------------------------------------------------------------------------------- 1 | ``` 2 | **Machine Learning Report** 3 | 4 | **Problem Definition:** 5 | The project aims to optimize the Readahead feature of the Linux Page Cache under varying workloads using machine learning techniques. The problem is to develop a model that dynamically adjusts the Readahead size based on workload characteristics. 6 | 7 | **Objectives:** 8 | 9 | 1. Data Collection: Gather data on various I/O operations using RocksDB benchmarks and Linux's LTTng tracing framework. 10 | 2. Feature Engineering: Process the collected data to extract relevant features. 11 | 3. Model Training: Implement and train different models (Decision Tree, Neural Network, Random Forest) to classify workload types and suggest optimal Readahead sizes. 12 | 4. Performance Evaluation: Compare the performance of the models and determine the best approach. 13 | 14 | **Methods:** 15 | 16 | 1. **Feature Importance Analysis:** Random Forest Classifier was used to analyze feature importance, and non-important features were removed. 17 | 2. **Dimensionality Reduction:** T-SNE was used to visualize the data in 2D. 18 | 3. **Model Training:** 19 | * **Neural Network:** MLPClassifier was used with hidden layers of 64 and 32 neurons. The model was trained using Stratified K-Fold Cross-Validation, and the average accuracy was 99.85%. 20 | * **Decision Tree:** DecisionTreeClassifier was used, and the accuracy was 100.00%. 21 | * **Random Forest:** RandomForestClassifier was used with 100 estimators, and the accuracy was 100.00%. 22 | 23 | **Results:** 24 | The results show that both the Decision Tree and Random Forest models achieved perfect accuracy, while the Neural Network model had a slightly lower accuracy. 25 | 26 | **Performance Comparison:** 27 | 28 | | Model | Accuracy | Notes | 29 | |------------------|-----------|---------------------------------------------| 30 | | Decision Tree | 100.00% | Simple, interpretable, perfect accuracy | 31 | | Neural Network | 99.85% | High accuracy, complex model with slight variability in precision | 32 | | Random Forest | 100.00% | Combines multiple trees for perfect accuracy and generalization | 33 | 34 | **Conclusion:** 35 | The project developed and compared three models to optimize the Readahead feature under varying workloads. The Random Forest model stands out for its combination of accuracy and interpretability, making it a strong candidate for real-time systems that require dynamic adjustment of Readahead sizes based on current workloads. 36 | ``` 37 | The report is generated and written to a file named "ML_Report". -------------------------------------------------------------------------------- /asset/model.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MahanVeisi8/Automated-readme-generator-with-Multi-Agent-CrewAI/ea463fea581b1b0a961d8a44f57c617349706a81/asset/model.jpg -------------------------------------------------------------------------------- /asset/style_found_output.txt: -------------------------------------------------------------------------------- 1 | ``` 2 | # Introduction 3 | 4 | This repository provides a comprehensive implementation of [Project Name], a [brief description of the project]. The project aims to [briefly describe the project's objective]. 5 | 6 | ## Setup 7 | 8 | ### Prerequisites 9 | 10 | * List of prerequisites, e.g., Python version, libraries, etc. 11 | * Installation instructions, e.g., pip install, etc. 12 | 13 | ### Environment Setup 14 | 15 | * Instructions for setting up the environment, e.g., creating a virtual environment, etc. 16 | 17 | ## Implementing DQN Components 18 | 19 | ### [Component 1] 20 | 21 | * Brief description of the component 22 | * Code snippet or example 23 | * Explanation of the component's functionality 24 | 25 | ### [Component 2] 26 | 27 | * Brief description of the component 28 | * Code snippet or example 29 | * Explanation of the component's functionality 30 | 31 | ## Results and Performance Analysis 32 | 33 | ### [Result 1] 34 | 35 | * Description of the result, including any visual aids, e.g., images, tables, etc. 36 | * Analysis of the result, including any relevant metrics or statistics 37 | 38 | ### [Result 2] 39 | 40 | * Description of the result, including any visual aids, e.g., images, tables, etc. 41 | * Analysis of the result, including any relevant metrics or statistics 42 | 43 | ## Summary 44 | 45 | This project provides a comprehensive implementation of [Project Name], demonstrating [key findings or achievements]. The results show [briefly describe the results], highlighting the effectiveness of [key components or techniques]. 46 | ``` 47 | This is the final answer, a standardized README template based on the identified patterns and style. -------------------------------------------------------------------------------- /readmecrewai.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ReadMeCrewAI.ipynb 3 | 4 | Automatically generated by Colab. 5 | 6 | Original file is located at 7 | https://colab.research.google.com/drive/1LQAnuuqdHNWhfrBc6KH03WDKyTlc4AeR 8 | 9 | # **🚀 CREW AI Project: Automated README Generator using AI** 10 | 11 | ## **Introduction 📖** 12 | 13 | This notebook demonstrates how to automate the generation of a professional and well-structured `README.md` file using **CREW AI** in combination with **Langchain** and **GROQ LLM**. The project utilizes multiple agents, each with specific tasks such as scanning directories, reading content, analyzing information, and producing the final `README.md`. By breaking down the task of creating documentation into smaller, manageable steps, this approach significantly reduces the manual effort required to maintain high-quality project documentation. 14 | 15 | ##**🛠 Installation and Setup** 16 | 17 | To get started, we need to install the necessary libraries and set up the environment. This includes **Langchain**, **CREW AI**, and **Google Generative AI** tools. 18 | """ 19 | 20 | pip install --upgrade langchain langchain_core crewai langchain_groq duckduckgo-search crewai[tools] 21 | 22 | """## **🔑 Setting Up Environment Variables** 23 | 24 | We need to set up our **GROQ API Key** and install any remaining required packages. Ensure you have access to the necessary APIs before proceeding. 25 | """ 26 | 27 | import os 28 | 29 | # Set your API key as an environment variable 30 | os.environ["GROQ_API_KEY"] = pass # removed the KEY after using it for privacy 31 | 32 | """## **🤖 Setting Up a Crew of Agents to Read and Analyze README Files** 33 | 34 | ### **📂 Cloning the GitHub Repository** 35 | 36 | In this section, we will clone the repository containing various projects for which we need to analyze the style of `README.md` files. This will help us create a standardized template. 37 | """ 38 | 39 | # Commented out IPython magic to ensure Python compatibility. 40 | # Clone the GitHub repository 41 | !git clone https://github.com/MahanVeisi8/RL_practices.git 42 | 43 | # Navigate to the repository directory 44 | # %cd RL_practices 45 | 46 | # Verify the contents of the directory 47 | !ls -R 48 | 49 | """### **Step 1: Setting Up the Large Language Model (LLM)** 50 | 51 | We will set up **GROQ LLM**, a large language model, to handle natural language processing tasks such as reading and analyzing the `README.md` file. 52 | """ 53 | 54 | from crewai import Agent 55 | from crewai_tools import DirectoryReadTool, FileReadTool, FileWriterTool # Correct import of tools 56 | from langchain_groq import ChatGroq 57 | import time 58 | from crewai import Agent, Task 59 | 60 | # Setting up the LLM (GROQ_LLM) 61 | GROQ_LLM = ChatGroq( 62 | api_key=os.getenv("GROQ_API_KEY"), 63 | model="llama3-70b-8192" 64 | ) 65 | 66 | # Test the LLM with a simple prompt using invoke 67 | response = GROQ_LLM.invoke("What is the capital of France?") 68 | print("LLM Response:", response) 69 | 70 | """### **Step 2: Initializing Tools for Directory and File Operations** 71 | 72 | We will initialize the tools required to read directories and files from the project. These tools will be used by various agents throughout the process. 73 | 74 | ### **🕵️‍♂️ Step 3: Defining the Agents for the README Style Analysis Crew** 75 | 76 | This section defines the **three agents** that will perform the key tasks of reading the `README.md` files, analyzing their content, and creating a standardized template based on the identified style. 77 | """ 78 | 79 | # Initialize the tools correctly 80 | directory_read_tool = DirectoryReadTool() # Initialize the DirectoryReadTool 81 | file_read_tool = FileReadTool() # Initialize the FileReadTool 82 | file_write_tool = FileWriterTool() 83 | 84 | # Define the variables for dynamic input 85 | target_directory = "Cartpole" 86 | target_subdirectory = "1 - DQN" 87 | 88 | """#### **Agent 1: Directory Analyzer** 89 | 90 | The **Directory Analyzer** is responsible for scanning the specified directory (`Cartpole`) and identifying the `README.md` files that need to be analyzed. 91 | """ 92 | 93 | # Directory Analyzer Agent 94 | directory_analyzer = Agent( 95 | role='Directory Analyzer', 96 | goal=f'Identify all README.md files in the {target_directory} directory and its subdirectories. Pass the path of a README.md file that is in a directory named something similar to {target_subdirectory}.', 97 | backstory="""You are a methodical analyst, excellent at traversing and cataloging directory contents.""", 98 | verbose=True, 99 | llm=GROQ_LLM, 100 | tools=[directory_read_tool], # Use the initialized DirectoryReadTool 101 | max_iterations=100, # Increase iterations 102 | time_limit=300 # Increase time limit in seconds 103 | ) 104 | 105 | # Directory Scanning Task 106 | directory_scan_task = Task( 107 | description=f"Identify and list all README.md files within the {target_directory} directory and its subdirectories. " 108 | f"Find a README.md file that is located in a directory named something similar to {target_subdir}. " 109 | "Pass the file path to the next agent.", 110 | agent=directory_analyzer, 111 | expected_output="A full path to the README.md file located in the target directory." 112 | ) 113 | 114 | """#### **Agent 2: Content Reader and Analyzer** 115 | 116 | The **Content Reader and Analyzer** agent reads the identified `README.md` file and provides a detailed analysis of its style, structure, and key elements. This analysis will be passed to the next agent for template creation. 117 | """ 118 | 119 | # Content Reader Agent 120 | reader_agent = Agent( 121 | role='Content Reader and Analyzer', 122 | goal=f'Read the content of the README.md file that its path is identified by the Directory Analyzer and Analyze the content of it identify its styles, structure, and key elements with details. Pass the analysis to the Template Creator.', 123 | backstory="""You are an avid reader with a sharp memory for details. You are an AI expert with a deep understanding of algorithms and a knack for recognizing writing styles in README files.""", 124 | verbose=True, 125 | llm=GROQ_LLM, 126 | tools=[file_read_tool], # Use the initialized FileReadTool 127 | ) 128 | 129 | # Content Reading Task 130 | content_reading_and_analysis_task = Task( 131 | description="Read the content of the README.md file that its path is identified by the Directory Analyzer and Analyze the content it. Build up an understanding of the style, structure, and key elements used in the README. Provide a comprehensive summary of the style to the Template Creator agent." 132 | "Once the content is read, pass it to the Content Analyzer agent for further processing.", 133 | agent=reader_agent, 134 | expected_output="An explanation of how the repository creator prefers to write README files. including the style, structure, and key elements." 135 | ) 136 | 137 | """#### **Agent 3: Template Creator** 138 | 139 | The **Template Creator** synthesizes the analysis provided by the Content Reader and creates a standardized, reusable `README.md` template based on the patterns identified in the original file. 140 | """ 141 | 142 | # Template Creator Agent 143 | template_creator_agent = Agent( 144 | role='Template Creator', 145 | goal=f'Create a standardized README template based on the analysis provided by the Content Reader and Analyzer agent. Use the identified patterns to create a coherent and reusable template.', 146 | backstory="""You are a skilled technical writer, capable of synthesizing insights into a coherent and reusable template.""", 147 | verbose=True, 148 | llm=GROQ_LLM, 149 | tools=[file_write_tool], # Use the initialized DirectoryReadTool 150 | max_iterations=100, # Increase iterations 151 | time_limit=300 # Increase time limit in seconds 152 | ) 153 | 154 | # Template Creation Task 155 | template_creation_task = Task( 156 | description="Create a standardized README template based on the common patterns identified by the Content Analyzer. " 157 | "Ensure that the template reflects the style, structure, and key elements derived from the analysis.", 158 | agent=template_creator_agent, 159 | expected_output="A new README template file that follows the identified style and structure." 160 | ) 161 | 162 | """### **🚀 Running the Style Analysis Crew** 163 | 164 | Now that we have defined the agents and their tasks, we can launch the **Style Analysis Crew** to perform the task of reading and analyzing the style of the `README.md` files. 165 | """ 166 | 167 | from crewai import Crew, Process 168 | 169 | # Define the crew 170 | crew = Crew( 171 | agents=[directory_analyzer, reader_agent, template_creator_agent], 172 | tasks=[directory_scan_task, content_reading_and_analysis_task, template_creation_task], 173 | process=Process.sequential, # Ensures tasks are executed one after the other 174 | verbose=True 175 | ) 176 | 177 | # Kickoff the crew - start the analysis 178 | style_found = crew.kickoff(inputs={"Cartpole_directory": "Cartpole"}) 179 | 180 | # Print the result to confirm output 181 | print("Style Found: ", style_found) 182 | 183 | # Extract the final task output (from the Template Creator Agent) 184 | style_template = style_found.tasks_output[-1].raw # Access the last task's raw output 185 | 186 | # Save the result to a file 187 | output_file_path = "style_found_output.txt" 188 | with open(output_file_path, "w") as output_file: 189 | output_file.write(style_template) 190 | 191 | print(f"Style analysis and template saved to: {output_file_path}") 192 | 193 | style_template 194 | 195 | """### **Summary of the Style Analysis Crew**: 196 | - **Directory Analyzer** scans and locates the relevant `README.md` files. 197 | - **Content Reader** reads and analyzes the style, structure, and key elements of the `README.md`. 198 | - **Template Creator** uses the analysis to generate a reusable `README.md` template. 199 | 200 | ## **🤖 Setting Up the ML Code Analysis Crew** 201 | 202 | In this section, we will use agents to locate a specific file (`os_labfinalpy`), read and analyze its content (focused on ML methods), and generate a detailed report. This report will include the file’s methods, problems addressed, and the results. 203 | 204 | ### **📂 Cloning the GitHub Repository for ML Code** 205 | 206 | We will clone a different repository containing machine learning models for readahead optimization and navigate into the appropriate directory. 207 | """ 208 | 209 | cd .. 210 | 211 | # Commented out IPython magic to ensure Python compatibility. 212 | # Clone the GitHub repository 213 | !git clone https://github.com/MahanVeisi8/Readahead-Optimization-Using-ML-Models.git 214 | 215 | # Navigate to the repository directory 216 | # %cd Readahead-Optimization-Using-ML-Models 217 | 218 | # Verify the contents of the directory 219 | !ls -R 220 | 221 | """### **Initializing Agents for the ML Code Analysis Crew** 222 | 223 | In this step, we define the three agents responsible for locating, reading, and analyzing the target file (`os_labfinalpy`), followed by generating a structured report. 224 | 225 | #### **Agent 1: File Finder** 226 | 227 | This agent is tasked with locating the target file (`os_labfinalpy`) within the project directory or subdirectories. 228 | """ 229 | 230 | # Define the variables for dynamic input 231 | target_file = "os_labfinalpy" 232 | 233 | # File Finder Agent 234 | file_finder_agent = Agent( 235 | role='File Finder', 236 | goal=f'Identify the {target_file} file in the current directory. Pass the file path to the next agent.', 237 | backstory=f"You are a methodical analyst, skilled at locating files in complex directory structures. Your task is to locate the {target_file} file.", 238 | verbose=True, 239 | llm=GROQ_LLM, 240 | tools=[directory_read_tool], # Use the initialized DirectoryReadTool 241 | max_iterations=100, 242 | time_limit=300 243 | ) 244 | 245 | # File Finding Task 246 | file_finding_task = Task( 247 | description=f"Locate the {target_file} file in the current directory or its subdirectories. Pass the path of the file to the File Reader agent.", 248 | agent=file_finder_agent, 249 | expected_output=f"Path to the {target_file} file." 250 | ) 251 | 252 | """#### **Agent 2: File Reader and ML Analyzer** 253 | 254 | This agent reads the content of the identified file, focusing on analyzing ML methods, the problem it addresses, and the results. The analysis is then passed to the Report Writer Agent. 255 | """ 256 | 257 | # File Reader Agent 258 | file_reader_agent = Agent( 259 | role='ML Expert Reader and Analyzer', 260 | goal=f'Read the content of the {target_file} file identified by the File Finder agent, analyze the ML methods, the problem it addresses, the results obtained, and other important elements. Provide a comprehensive analysis and review for the Report Generator agent.', 261 | backstory=f'You are an avid reader capable of handling large files and an expert in ML and algorithms. Your task is to retrieve and read the file content, handle large files if necessary, and analyze the code in the {target_file} file, identifying its key methods, problem scope, and results. Provide an overview with details of everything.', 262 | verbose=True, 263 | llm=GROQ_LLM, # Using the powerful flash_llm for content reading and analysis 264 | tools=[file_read_tool], # Use the initialized FileReadTool 265 | max_iterations=1000, 266 | time_limit=900 267 | ) 268 | 269 | # File Reading Task 270 | file_reading_task = Task( 271 | description=f"Read the content of the {target_file} file identified by the File Finder and analyze the code. Identify key ML methods, the problem addressed, and the results. Provide a comprehensive analysis.", 272 | agent=file_reader_agent, 273 | expected_output=f"A comprehensive analysis of the ML methods, problem, and results, to be passed to the Report Writer agent." 274 | ) 275 | 276 | """#### **Agent 3: Report Writer** 277 | 278 | This agent organizes the analysis provided by the File Reader Agent into a detailed, structured report, including an introduction, methods, results, and conclusion. 279 | """ 280 | 281 | # Report Writer Agent 282 | report_writer_agent = Agent( 283 | role='Report Writer', 284 | goal=f'Generate a structured and detailed report based on the analysis provided by the ML Expert Reader and Analyzer agent. Organize the analysis into a clear format with sections for introduction, methodology, results, and conclusion.', 285 | backstory=f'You are a skilled technical writer capable of synthesizing complex technical information into a well-organized report. Use the analysis provided by the ML Expert to generate a formal report.', 286 | verbose=True, 287 | llm=GROQ_LLM, # You can use a standard LLM here, no need for the more powerful model 288 | tools=[file_write_tool], # Tool to write the report to a file 289 | max_iterations=100, 290 | time_limit=600 291 | ) 292 | 293 | # Report Writing Task 294 | report_writing_task = Task( 295 | description="Take the analysis from the ML Expert Reader and Analyzer agent and generate a detailed, structured report. Organize the content into a professional format with an introduction, methods, results, and conclusion.", 296 | agent=report_writer_agent, 297 | expected_output="A detailed written report based on the analysis of the ML practice code." 298 | ) 299 | 300 | """### **🚀 Running the ML Code Analysis Crew** 301 | 302 | Now that we have defined the agents and their tasks, we can launch the **ML Code Analysis Crew** to locate, analyze, and generate a report on the target file. 303 | """ 304 | 305 | # Define the crew with the new Report Writer Agent 306 | crew_ML = Crew( 307 | agents=[file_finder_agent, file_reader_agent, report_writer_agent], 308 | tasks=[file_finding_task, file_reading_task, report_writing_task], 309 | process=Process.sequential, # Ensures tasks are executed one after the other 310 | verbose=True 311 | ) 312 | 313 | # Kickoff the crew - start the process 314 | ML_report = crew_ML.kickoff() 315 | 316 | """#### **Saving the Final Report to a File** 317 | 318 | Once the analysis is completed and the report is generated, we will save the final report to a file named `code_content.txt`. 319 | """ 320 | 321 | # Extract the final report (from the Report Writer Agent) 322 | ML_analysis = ML_report.tasks_output[-1].raw # Access the last task's raw output 323 | 324 | # Print the result to confirm output 325 | print("ML Analysis Report: ", ML_analysis) 326 | 327 | # Save the result to a file 328 | output_file_path = "code_content.txt" 329 | with open(output_file_path, "w") as output_file: 330 | output_file.write(ML_analysis) 331 | 332 | print(f"code content and template saved to: {output_file_path}") 333 | 334 | """## **📝 Setting Up the Final README Generation Crew** 335 | 336 | In this section, we will set up a crew of agents to read the two reports generated from the previous analysis (style and code), merge their content, and create a professional `README.md` file. This final README will be polished, structured, and enhanced with creative elements. 337 | """ 338 | 339 | cd .. 340 | 341 | """### **🤖 Defining the Agents for the Final README Generation** 342 | 343 | We define three agents: one to read the reports, one to merge the content, and one to generate and refine the final README. 344 | """ 345 | 346 | code_report = "code_content.txt" 347 | style_report = "style_found_output.txt" 348 | 349 | """#### **Agent 1: File Reader** 350 | 351 | This agent will read the content of both the `style_report` and `code_report` files and pass the content to the next agent. 352 | """ 353 | 354 | # File Reader Agent: Reads content from the two text files 355 | file_reader_agent = Agent( 356 | role='File Reader', 357 | goal=f'Read the content of both {style_report} and {code_report} and pass them to the Content Creator (content_merger_agent) Agent.', 358 | backstory="You are a methodical reader with a focus on retrieving and passing content for analysis.", 359 | verbose=True, 360 | llm=GROQ_LLM, # Use the LLM for handling complex content reading tasks 361 | tools=[file_read_tool], # Use the initialized FileReadTool 362 | max_iterations=100, 363 | time_limit=300 364 | ) 365 | 366 | 367 | # File Reading Task: Read content from the two text files 368 | file_reading_task = Task( 369 | description=f"Read the content of both {style_report} and {code_report}. Make sure to clearly distinguish between the two. When passing the content to the Content Creator (content_merger_agent) Agent, explain that the first section is for {style_report} (which contains the preferred style and structure) and the second section is for {code_report} (which contains the code analysis and details). Provide a detailed breakdown of both sections to help the Content Creator Agent form a clear understanding.", 370 | agent=file_reader_agent, 371 | expected_output="Content of both {style_report} and {code_report} in one message, clearly divided into two sections." 372 | ) 373 | 374 | """#### **Agent 2: Content Merger** 375 | 376 | This agent merges the content from the `style_report` (style and structure) and `code_report` (code analysis) into a coherent README draft. 377 | """ 378 | 379 | # Content Merger Agent: Merges the content from the two files 380 | content_merger_agent = Agent( 381 | role='Content Creator', 382 | goal=f'Write a Readme file based on the style and structure of {style_report} section and with the info of the current code report {code_report} section that all has been provided by the File Reader already. So when the File Reader agent gives you the message, do not ask for more info from it. use analyse the info given and create a good readme. Pass the created README to the README editor (readme_generator_agent) and try to interact with it whenever it wants info from you.', 383 | backstory="You are a skilled synthesizer, capable of combining information from multiple sources into a unified format and create a good readme based on the code report info and style wanted given.", 384 | verbose=True, 385 | llm=GROQ_LLM, 386 | tools=[], # No specific tools needed, just the LLM 387 | max_iterations=100, 388 | time_limit=300 389 | ) 390 | 391 | # Content Merging Task: Merge the content from the two files into a structured README draft 392 | content_merging_task = Task( 393 | description=f"Using the content provided by the File Reader (file_reader_agent) Agent, combine the two sections into a coherent and well-organized README draft. Make sure that the style from {style_report} is applied to the project analysis from {code_report}. Structure the README in a way that fits well with the identified style, while including the detailed information from the code analysis. Once the draft is created, pass it to the README editor (readme_generator_agent) Agent for final review and improvements. Be sure to interact with the README Editor Agent if it requires further details or clarification.", 394 | agent=content_merger_agent, 395 | expected_output="A structured README text draft with the merged content, applying the preferred style to the code analysis.to be passed to the README editor agent." 396 | ) 397 | 398 | """#### **Agent 3: README Generator** 399 | 400 | This agent refines the draft README, adding creative elements such as emojis and formatting enhancements, and writes the final `README.md` file. 401 | """ 402 | 403 | # README Generator Agent: Writes the final README file 404 | readme_generator_agent = Agent( 405 | role='README editor', 406 | goal='Have a review based on the readme file given by the Content Creator and rewrite a final README and edit it well to produce a really cool and detiled Readme file. Add lots of fun things too, for exmaple adding emojies to create a great readme. Try to rewrite it and add emojies or anything other needed for make it perfect. Ensure it is formatted professionally and coherently and write the final README file.', 407 | backstory="You are an expert technical writer and reviewr for README files, proficient at organizing information into professional documentation formats. You have a great knowledge in the features of the cool README files and try to rewrite readme files to make them great.", 408 | verbose=True, 409 | llm=GROQ_LLM, 410 | tools=[file_write_tool], # Use the FileWriterTool to save the README 411 | max_iterations=100, 412 | time_limit=300 413 | ) 414 | 415 | 416 | 417 | # README Writing Task: Finalize and enhance the README file 418 | readme_writing_task = Task( 419 | description="Review the draft README provided by the Content Creator (content_merger_agent) Agent. Refine the content for clarity and coherence, making sure that it adheres to professional standards. Add enhancements such as emojis, icons, and other creative elements to make the README engaging and visually appealing. Ensure the README follows best practices for formatting and layout. Once finalized, give the final README content.", 420 | agent=readme_generator_agent, 421 | expected_output="A polished and professionally formatted README content (maybe as string or txt) with added creative touches such as emojis." 422 | ) 423 | 424 | """### **🚀 Running the Final README Generation Crew** 425 | 426 | Now that the agents and tasks are defined, we can launch the final crew to generate the `README.md` file. 427 | """ 428 | 429 | # Define the new crew 430 | crew_readme = Crew( 431 | agents=[file_reader_agent, content_merger_agent, readme_generator_agent], 432 | tasks=[file_reading_task, content_merging_task, readme_writing_task], 433 | process=Process.sequential, # Tasks are executed one after the other 434 | verbose=True 435 | ) 436 | 437 | # Kickoff the crew - start the process 438 | final_readme_creation = crew_readme.kickoff() 439 | 440 | """### **💾 Saving the Final README to a File** 441 | 442 | Once the `README.md` is generated, we save the result to a file. 443 | """ 444 | 445 | # Extract the final README (from the README Generator Agent) 446 | final_readme_content = final_readme_creation.tasks_output[-1].raw # Access the last task's raw output 447 | 448 | # Print the result to confirm output 449 | print("Final README Content: ", final_readme_content) 450 | 451 | # Save the result to a README file 452 | output_readme_file_path = "FINAL_README.md" 453 | with open(output_readme_file_path, "w") as output_file: 454 | output_file.write(final_readme_content) 455 | 456 | print(f"Final README saved to: {output_readme_file_path}") 457 | 458 | """### **Summary of the Final README Generation Crew** 459 | 460 | - **File Reader** reads both the style and code reports and passes the content to the Content Merger. 461 | - **Content Merger** combines the two reports into a structured README draft. 462 | - **README Generator** enhances the README by adding creative elements and saves the final result. 463 | 464 | ## **🔚 Conclusion** 465 | 466 | In this project, we successfully automated the generation of a professional `README.md` file by utilizing **CREW AI**, **Langchain**, and **GROQ LLM**. We developed a multi-step process involving agents that scanned directories, read and analyzed project files, and combined this information into a well-structured and visually engaging README. 467 | 468 | By breaking down the workflow into manageable tasks—analyzing the style, reviewing the code, and merging the content—we created a reusable solution that can be applied to a variety of repositories. This project showcases the power of AI in automating documentation tasks, making it easier to maintain high-quality project documentation with minimal manual effort. 469 | """ --------------------------------------------------------------------------------