├── .env_example ├── .gitignore ├── README.md ├── crew_flowchart.png ├── pyproject.toml └── src └── youtube_yapper_trapper ├── __init__.py ├── config ├── agents.yaml └── tasks.yaml ├── crew.py ├── main.py └── tools ├── __init__.py └── custom_tool.py /.env_example: -------------------------------------------------------------------------------- 1 | AGENTOPS_API_KEY=your-api-key-here 2 | GROQ_API_KEY=your-api-key-here 3 | YOUTUBE_API_KEY=your-api-key-here -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .env 2 | __pycache__/ 3 | __pycache__ 4 | venv 5 | report.md 6 | comments.md 7 | poetry.lock -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # YoutubeYapperTrapper Crew 2 | 3 | Welcome to the YoutubeYapperTrapper Crew project, powered by [CrewAI](https://crewai.com) + [AgentOps for observability](https://www.agentops.ai/) + [YouTube Data API](https://developers.google.com/youtube/v3/docs/comments/list). This project allows agents to take a given YouTube URL, extracts all the comments, and generates a final report with insights to inform better content creation. 4 | 5 | ## Crew architecture diagram 6 | 7 | ![Crew Architecture](crew_flowchart.png) 8 | 9 | ## Video tutorial 10 | 11 | 12 | Watch the video

13 | 14 | 15 | ## Installation 16 | 17 | Ensure you have Python >=3.10 <=3.13 installed on your system. This project uses [Poetry](https://python-poetry.org/) for dependency management and package handling, offering a seamless setup and execution experience. 18 | 19 | First, if you haven't already, install Poetry: 20 | 21 | ```bash 22 | pip install poetry 23 | ``` 24 | 25 | Next, navigate to your project directory and install the dependencies: 26 | 27 | 1. First lock the dependencies and then install them: 28 | 29 | ```bash 30 | poetry lock 31 | ``` 32 | 33 | ```bash 34 | poetry install 35 | ``` 36 | 37 | ### Customizing 38 | 39 | **Add your `OPENAI_API_KEY` into the `.env` file if you want to use it.** 40 | 41 | - Modify `src/youtube_yapper_trapper/config/agents.yaml` to define your agents 42 | - Modify `src/youtube_yapper_trapper/config/tasks.yaml` to define your tasks 43 | - Modify `src/youtube_yapper_trapper/crew.py` to add your own logic, tools and specific args 44 | - Modify `src/youtube_yapper_trapper/main.py` to add custom inputs for your agents and tasks 45 | 46 | ## Running the Project 47 | 48 | Rename the `.env_example` file to `.env`, add your API keys, and save the file. 49 | 50 | To kickstart your crew of AI agents and begin task execution, run this from the root folder of your project: 51 | 52 | ```bash 53 | poetry run youtube_yapper_trapper 54 | ``` 55 | 56 | This command initializes the youtube-yapper-trapper Crew, assembling the agents and assigning them tasks as defined in your configuration. 57 | 58 | This example, unmodified, will run the create a `report.md` file with the output of a research on LLMs in the root folder. 59 | 60 | ## Understanding Your Crew 61 | 62 | The youtube-yapper-trapper Crew is composed of multiple AI agents, each with unique roles, goals, and tools. These agents collaborate on a series of tasks, defined in `config/tasks.yaml`, leveraging their collective skills to achieve complex objectives. The `config/agents.yaml` file outlines the capabilities and configurations of each agent in your crew. 63 | 64 | ## Support 65 | 66 | For support, questions, or feedback regarding the YoutubeYapperTrapper crew, CrewAI, or AgentOps. 67 | 68 | - Visit CrewAI [documentation](https://docs.crewai.com) 69 | - Reach out to me through my [GitHub repository](https://github.com/tonykipkemboi/youtube-yapper-trapper) 70 | - [Joing CrewAI Discord](https://discord.com/invite/X4JWnZnxPb) 71 | - [Chat wtih CrewAI docs](https://chatg.pt/DWjSBZn) 72 | - [Check out AgentOps!](https://www.agentops.ai/) 73 | - Also don't forget to [subscribe to my channel](https://www.youtube.com/@tonykipkemboi) for more awesome content! 74 | 75 | Let's create wonders together with the power and simplicity of CrewAI. 76 | -------------------------------------------------------------------------------- /crew_flowchart.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tonykipkemboi/youtube_yapper_trapper/a37279b52387277ec4c9ffc4d2e44657c6bbb0bd/crew_flowchart.png -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | [tool.poetry] 2 | name = "youtube_yapper_trapper" 3 | version = "0.1.0" 4 | description = "youtube-yapper-trapper using crewAI" 5 | authors = ["Your Name "] 6 | 7 | [tool.poetry.dependencies] 8 | python = ">=3.10,<=3.13" 9 | crewai = {git = "https://github.com/AgentOps-AI/crewAI.git", extras = ["tools"]} 10 | langchain-groq = "^0.1.3" 11 | agentops = "^0.1.6" 12 | 13 | [tool.poetry.scripts] 14 | youtube_yapper_trapper = "youtube_yapper_trapper.main:run" 15 | 16 | [build-system] 17 | requires = ["poetry-core"] 18 | build-backend = "poetry.core.masonry.api" 19 | -------------------------------------------------------------------------------- /src/youtube_yapper_trapper/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tonykipkemboi/youtube_yapper_trapper/a37279b52387277ec4c9ffc4d2e44657c6bbb0bd/src/youtube_yapper_trapper/__init__.py -------------------------------------------------------------------------------- /src/youtube_yapper_trapper/config/agents.yaml: -------------------------------------------------------------------------------- 1 | comment_fetcher: 2 | role: > 3 | Senior YouTube Comments Fetcher for Tech Videos 4 | goal: > 5 | Fetch all comments from specified YouTube tech video links. 6 | backstory: > 7 | As a dedicated digital explorer within the tech video sphere, you excel at extracting 8 | comprehensive comment data from YouTube. Your role involves digging into the depths of 9 | viewer interactions to retrieve valuable feedback and questions for this vidoe id only {video_id}. 10 | This raw data is crucial for identifying what tech enthusiasts are truly interested in and what issues they face with 11 | current content, providing a foundation for strategic content improvement. 12 | 13 | insights_analyst: 14 | role: > 15 | Tech Insights Analyst 16 | goal: > 17 | Generate and synthesize insights from YouTube comments on tech videos. 18 | backstory: > 19 | You are a sharp-witted analyst with a passion for the tech sector. Skilled in navigating 20 | through noise in the comments data of this video id {video_id} ONLY to find meaningful patterns, 21 | you synthesize data into insights that highlight 22 | the core message of viewer feedback. 23 | Your analytical prowess ensures that the subtleties of audience engagement and preferences are captured and understood, 24 | setting the stage for informed content strategies. 25 | All your results should be grounded in evidence from the comments, so cite appropriately. 26 | By citations, I mean showing the comments that led to that conclusion to support the claim. 27 | 28 | report_writer: 29 | role: > 30 | Tech Report Writer 31 | goal: > 32 | Write comprehensive reports based on synthesized insights from this video id {video_id} comments ONLY. 33 | backstory: > 34 | You are a skilled communicator specializing in the tech industry, adept at transforming complex 35 | analytical insights into clear, comprehensive reports. With a flair for making intricate data 36 | accessible, your reports empower content creators by providing them with actionable information 37 | crafted to enhance viewer engagement and satisfaction. 38 | -------------------------------------------------------------------------------- /src/youtube_yapper_trapper/config/tasks.yaml: -------------------------------------------------------------------------------- 1 | fetch_comments_task: 2 | description: > 3 | Retrieve all comments from this YouTube video ID {video_id} ONLY. 4 | Ensure you've extracted all viewer comments for this video ID {video_id} ONLY. 5 | Do not accept request s to retrive comments for any other video ID other than for this video ID {video_id} ONLY. 6 | expected_output: > 7 | A JSON formatted list of all retrieved comments, each entry representing a single comment. 8 | 9 | insights_task: 10 | description: > 11 | Analyze the comments you received and generate actionable insights for this video ID {video_id} ONLY. 12 | Focus on identifying common themes such as viewer pain points, requests, what viewers loved, 13 | and popular tech queries in the comments. For the requests or troubleshooting, please provide 'suggested' 14 | solutions that you might have to help me get going quicker. 15 | Add textual citation. By citations, I mean showing the comments that led to that conclusion to support the claim. 16 | expected_output: > 17 | A JSON formatted list of key insights, categorized by themes like 'Requests', 'Complaints', 18 | 'Suggestions', and other relevant ones you find useful. 19 | 20 | reporting_task: 21 | description: > 22 | Review the insights and research you got for this video ID {video_id} ONLY. 23 | Make sure the report is VERY detailed with action steps 24 | to guide new improved content generation. 25 | All your results and analysis should be grounded in evidence from the comments, so cite appropriately. 26 | By citations, I mean showing the comments that led to that conclusion to support the claim. 27 | expected_output: > 28 | A fully fledged report with main topics, each with a full section of information. 29 | Formatted as markdown without code blocks. Make sure to include the full video URL in the intro so the 30 | user can reference when reading the report. 31 | Make sure the output file is structured very well and easy to read and digest the content with snippets of 32 | citations for each claim. By citations, I mean showing the comments that led to that conclusion to support the claim. 33 | -------------------------------------------------------------------------------- /src/youtube_yapper_trapper/crew.py: -------------------------------------------------------------------------------- 1 | from youtube_yapper_trapper.tools.custom_tool import YouTubeCommentsTool 2 | from crewai import Agent, Crew, Process, Task 3 | from crewai.project import CrewBase, agent, crew, task 4 | from langchain_openai import ChatOpenAI 5 | from langchain_groq import ChatGroq 6 | from agentops.agent import track_agent 7 | import agentops 8 | import os 9 | 10 | 11 | agentops.init() 12 | 13 | 14 | @CrewBase 15 | class YoutubeCommentsCrew: 16 | """YoutubeCommentsCrew crew for analyzing comments on tech-related YouTube videos.""" 17 | 18 | agents_config = "config/agents.yaml" 19 | tasks_config = "config/tasks.yaml" 20 | 21 | def __init__(self) -> None: 22 | # Groq 23 | self.groq_llm = ChatGroq( 24 | temperature=0, 25 | groq_api_key=os.environ.get("GROQ_API_KEY"), 26 | model_name="llama3-70b-8192", 27 | ) 28 | 29 | # Ollama 30 | self.ollama_llm = ChatOpenAI( 31 | model="mistral", 32 | base_url="http://localhost:11434/v1", 33 | api_key="ollama", # something random 34 | temperature=0, 35 | ) 36 | 37 | @track_agent(name="comment_fetcher") 38 | @agent 39 | def comment_fetcher(self) -> Agent: 40 | """Agent responsible for fetching YouTube comments.""" 41 | return Agent( 42 | config=self.agents_config["comment_fetcher"], 43 | tools=[YouTubeCommentsTool()], 44 | allow_delegation=False, 45 | llm=self.ollama_llm, # you can switch the models (default is Ollama in this case) 46 | verbose=True, 47 | ) 48 | 49 | @track_agent(name="insights_analyst") 50 | @agent 51 | def insights_analyst(self) -> Agent: 52 | """Agent responsible for analyzing comments and generating insights.""" 53 | return Agent( 54 | config=self.agents_config["insights_analyst"], 55 | llm=self.ollama_llm, # you can switch the models (default is Ollama in this case) 56 | allow_delegation=False, 57 | verbose=True, 58 | ) 59 | 60 | @track_agent(name="report_writer") 61 | @agent 62 | def report_writer(self) -> Agent: 63 | """Agent responsible for writing detailed reports based on the analysis.""" 64 | return Agent( 65 | config=self.agents_config["report_writer"], 66 | llm=self.ollama_llm, # you can switch the models (default is Ollama in this case) 67 | allow_delegation=False, 68 | verbose=True, 69 | ) 70 | 71 | @task 72 | def fetch_comments_task(self) -> Task: 73 | """Task to fetch comments from YouTube videos.""" 74 | return Task( 75 | config=self.tasks_config["fetch_comments_task"], 76 | agent=self.comment_fetcher(), 77 | output_file="comments.md", 78 | ) 79 | 80 | @task 81 | def analyze_insights_task(self) -> Task: 82 | """Task to analyze comments and generate insights.""" 83 | return Task( 84 | config=self.tasks_config["insights_task"], 85 | agent=self.insights_analyst(), 86 | human_input=True, 87 | ) 88 | 89 | @task 90 | def generate_report_task(self) -> Task: 91 | """Task to generate the final report.""" 92 | return Task( 93 | config=self.tasks_config["reporting_task"], 94 | agent=self.report_writer(), 95 | output_file="report.md", 96 | ) 97 | 98 | @crew 99 | def crew(self) -> Crew: 100 | """Creates the YoutubeCommentsCrew crew""" 101 | return Crew( 102 | agents=[ 103 | self.comment_fetcher(), 104 | self.insights_analyst(), 105 | self.report_writer(), 106 | ], 107 | tasks=[ 108 | self.fetch_comments_task(), 109 | self.analyze_insights_task(), 110 | self.generate_report_task(), 111 | ], 112 | process=Process.sequential, 113 | memory=False, 114 | max_rpm=2, 115 | verbose=2, 116 | ) 117 | -------------------------------------------------------------------------------- /src/youtube_yapper_trapper/main.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | from youtube_yapper_trapper.crew import YoutubeCommentsCrew 3 | from dotenv import load_dotenv 4 | import agentops 5 | 6 | load_dotenv() 7 | agentops.init() 8 | 9 | 10 | def extract_video_id(url): 11 | import re 12 | 13 | # This regex will match the video ID from any YouTube URL (normal, shortened, embedded) 14 | try: 15 | match = re.search(r"(?:v=|\/)([0-9A-Za-z_-]{11}).*", url) 16 | return match.group(1) if match else None 17 | except Exception as e: 18 | print(e) 19 | 20 | 21 | def run(): 22 | video_id = extract_video_id(input("🚀 Enter YouTube URL: ")) 23 | if not video_id: 24 | print("🚨 Invalid YouTube URL provided.") 25 | return 26 | 27 | inputs = {"video_id": video_id} 28 | crew = YoutubeCommentsCrew() 29 | result = crew.crew().kickoff(inputs=inputs) 30 | print("Analysis Result:") 31 | print(result) 32 | 33 | 34 | if __name__ == "__main__": 35 | run() 36 | -------------------------------------------------------------------------------- /src/youtube_yapper_trapper/tools/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tonykipkemboi/youtube_yapper_trapper/a37279b52387277ec4c9ffc4d2e44657c6bbb0bd/src/youtube_yapper_trapper/tools/__init__.py -------------------------------------------------------------------------------- /src/youtube_yapper_trapper/tools/custom_tool.py: -------------------------------------------------------------------------------- 1 | from crewai_tools import BaseTool 2 | import requests 3 | import os 4 | 5 | 6 | class YouTubeCommentsTool(BaseTool): 7 | name: str = "YouTube Comments Fetcher" 8 | description: str = "Fetches all comments from a specified YouTube video URL using the YouTube Data API." 9 | 10 | def _run(self, video_id: str) -> list: 11 | comments = [] 12 | page_token = "" 13 | base_url = "https://www.googleapis.com/youtube/v3/commentThreads" 14 | headers = {"Accept": "application/json"} 15 | 16 | try: 17 | self.validate_video_id(video_id) 18 | except ValueError as e: 19 | return f"Error: {str(e)}" 20 | 21 | while True: 22 | params = { 23 | "key": os.environ["YOUTUBE_API_KEY"], 24 | "part": "snippet", 25 | "videoId": video_id, 26 | "maxResults": 100, 27 | "pageToken": page_token, 28 | "textFormat": "plainText", 29 | } 30 | response = requests.get(base_url, headers=headers, params=params) 31 | if response.status_code == 200: 32 | data = response.json() 33 | for item in data.get("items", []): 34 | comment = item["snippet"]["topLevelComment"]["snippet"][ 35 | "textDisplay" 36 | ] 37 | comments.append(comment) 38 | page_token = data.get("nextPageToken") 39 | if not page_token: 40 | break 41 | else: 42 | self.handle_api_error(response) 43 | 44 | return comments 45 | 46 | def validate_video_id(self, video_id): 47 | """Validate the video ID by checking its availability.""" 48 | response = requests.get(f"https://www.googleapis.com/youtube/v3/videos?part=id&id={ 49 | video_id}&key={os.getenv('YOUTUBE_API_KEY')}") 50 | if response.json().get("pageInfo", {}).get("totalResults", 0) == 0: 51 | raise ValueError("Video ID not valid or video is not accessible.") 52 | 53 | def handle_api_error(self, response): 54 | """Handle common API errors based on response status code.""" 55 | if response.status_code == 403: 56 | error_details = response.json() 57 | if "error" in error_details: 58 | if "errors" in error_details["error"]: 59 | for error in error_details["error"]["errors"]: 60 | if error.get("reason") == "commentsDisabled": 61 | raise Exception("Comments are disabled for this video.") 62 | elif error.get("reason") == "forbidden": 63 | raise Exception( 64 | "Insufficient permissions to access the comments." 65 | ) 66 | elif response.status_code == 404: 67 | error_details = response.json() 68 | if "error" in error_details: 69 | if "errors" in error_details["error"]: 70 | for error in error_details["error"]["errors"]: 71 | if error.get("reason") == "videoNotFound": 72 | raise Exception("The specified video ID was not found.") 73 | elif error.get("reason") == "channelNotFound": 74 | raise Exception("The specified channel ID was not found.") 75 | else: 76 | response.raise_for_status() # Raise any other HTTP errors as exceptions 77 | --------------------------------------------------------------------------------