├── .env
├── .gitignore
├── Building_Code_Documentation_Agents.ipynb
├── README.md
├── arch_diagram.png
├── config
├── documentation_agents.yaml
├── documentation_tasks.yaml
├── planner_agents.yaml
└── planner_tasks.yaml
├── crewai-nim-flow.png
├── crewai_flow.html
├── docs
└── docs_will_go_here.txt
├── requirements.txt
└── workdir
└── repo_will_go_here.txt
/.env:
--------------------------------------------------------------------------------
1 | OTEL_SDK_DISABLED="true"
--------------------------------------------------------------------------------
/.gitignore:
--------------------------------------------------------------------------------
1 | .env
2 | assets
3 | docs/*
4 | db
5 | !docs/docs_will_go_here.txt
6 | workdir/*
7 | !workdir/repo_will_go_here.txt
8 | .DS_Store
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # Building Code Documentation Agents with CrewAI 🤖
2 |
3 | [](https://console.brev.dev/launchable/deploy?launchableID=env-2qNXgLMuzlJ8LS8Kl30EXbGwxSh)
4 |
5 | A powerful system that leverages CrewAI Flows to automatically generate comprehensive documentation for any public GitHub repository. This project employs multiple specialized AI agents working collaboratively to produce high-quality, accurate documentation tailored to your codebase.
6 |
7 | ---
8 |
9 | ## Key Features
10 |
11 | - **Multi-Agent Workflow**: Specialized agents collaborate to analyze and document your code.
12 | - **Automated Documentation**: Full lifecycle coverage, from analysis to review.
13 | - **NVIDIA NIM Integration**: Harnesses NVIDIA's llama-3.3-70b and NeMo Retriever E5 embedding models.
14 | - **Mermaid Diagram Support**: Automatically generates architecture and flow diagrams.
15 | - **Quality Assurance**: Ensures accuracy, consistency, and completeness with a built-in review process.
16 |
17 | ---
18 |
19 | ## Architecture
20 |
21 | **The system employs a multi-agent workflow divided into two key stages:**
22 |
23 | #### Ingestion Phase
24 | - **WebsiteSearchTool:** This tool is used to embed and index mermaid examples from mermaid.js.org website using NVIDIA NeMo Retriever E5 embedding NIM.
25 |
26 | #### Agent Flow
27 | 1. Codebase Analysis and Strategy Planning:
28 | - Analyze Codebase: Planner agents inspect the repository to map its structure, identify key components, and understand interdependencies.
29 | - Develop Strategy: They create a tailored documentation plan based on the analysis.
30 | 2. Documentation Creation and Review:
31 | - High-Level Documentation: One agent generates clear, comprehensive documentation introducing the project and its architecture.
32 | - Quality Assurance: Another agent ensures accuracy, consistency, and completeness across all documentation.
33 | Here's an architecture diagram of the workflow:
34 | 
35 |
36 | ---
37 |
38 | ## Getting Started
39 |
40 | ### Prerequisites
41 |
42 | 1. Python 3.10+
43 | 2. NVIDIA API Key
44 |
45 | ### Installation
46 |
47 | 1. Clone this repository
48 | 2. Install dependencies:
49 | ```bash
50 | pip install -r requirements.txt
51 | ```
52 | 3. Set up your environment variables (see below).
53 | 3. Run the Jupyter notebook.
54 | 4. Generated documentation will be available in the `docs` directory.
55 |
56 | ### Setting up NVIDIA API Key
57 |
58 | 1. Visit [NVIDIA NIM](https://build.nvidia.com/nim).
59 | 2. Select the **llama-3.3-70b-instruct** model.
60 | 3. Retrieve your API key from the right panel.
61 | 4. Set your environment variables:
62 | ```bash
63 | export NVIDIA_NIM_API_KEY=your_api_key
64 | ```
65 | Alternatively, create a `.env` file in the project root:
66 | ```bash
67 | NVIDIA_NIM_API_KEY=your_api_key
68 | ```
69 |
70 | ---
71 |
72 | ## Documentation Outputs
73 |
74 | The system generates multiple types of documentation, including:
75 |
76 | - **Project Overview**
77 | - **Architecture Documentation**
78 | - **Component Documentation**
79 | - **Integration Guides**
80 | - **API Documentation**
81 | - **Setup Instructions**
82 |
83 | ---
84 |
85 | ## Contributing
86 |
87 | We welcome contributions! To get involved:
88 |
89 | 1. Fork the repository.
90 | 2. Create a feature branch:
91 | ```bash
92 | git checkout -b feature/your-feature-name
93 | ```
94 | 3. Commit your changes:
95 | ```bash
96 | git commit -m "Add feature X"
97 | ```
98 | 4. Push to your branch:
99 | ```bash
100 | git push origin feature/your-feature-name
101 | ```
102 | 5. Submit a Pull Request.
103 |
104 | ---
105 |
106 | ## License
107 |
108 | This project is licensed under the MIT License.
109 |
110 | Built with ❤️ using CrewAI and NVIDIA AI.
111 |
112 | ---
113 |
114 | ## About the NVIDIA Collaboration
115 |
116 | Today, we are thrilled to announce a major collaboration with NVIDIA, fully integrating CrewAI with NVIDIA NIM microservices—part of the NVIDIA AI Enterprise software platform. This integration empowers developers to harness high-performance inference capabilities for the latest foundation models, enabling scalable and secure deployment of AI agents across various industries and workflows.
117 |
118 | ### Broader Vision
119 |
120 | While this demo focuses on automating project documentation, the collaboration paves the way for unlocking the full potential of AI agents across industries. Our shared goal is to make AI agents more autonomous, accessible, and transformative for organizations worldwide.
121 |
122 |
123 |
--------------------------------------------------------------------------------
/arch_diagram.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/crewAIInc/nvidia-demo/30896aca51a9ed314d4909ed87d84c583110f668/arch_diagram.png
--------------------------------------------------------------------------------
/config/documentation_agents.yaml:
--------------------------------------------------------------------------------
1 | overview_writer:
2 | role: >
3 | Overview Documentation Writer
4 | goal: >
5 | Create clear, comprehensive high-level documentation that introduces the
6 | project and its architecture
7 | backstory: >
8 | You are a technical writer specialized in creating project overviews and
9 | architecture documentation.
10 | With years of experience documenting complex systems, you excel at
11 | explaining technical concepts in an accessible way while maintaining
12 | technical accuracy. You focus on helping readers understand the big picture
13 | before diving into details.
14 | verbose: false
15 | llm: nvidia_nim/meta/llama-3.3-70b-instruct
16 |
17 | documentation_reviewer:
18 | role: >
19 | Documentation Quality Reviewer
20 | goal: >
21 | Review and ensure consistency, accuracy, and completeness of all
22 | documentation
23 | backstory: >
24 | As a documentation quality expert, you have a keen eye for detail and
25 | extensive experience
26 | reviewing technical documentation. You ensure all documentation meets high
27 | standards for clarity, completeness, and technical accuracy. You verify that
28 | documentation aligns with the codebase and follows consistent style and
29 | formatting throughout the project.
30 | verbose: false
31 | llm: nvidia_nim/meta/llama-3.3-70b-instruct
32 |
--------------------------------------------------------------------------------
/config/documentation_tasks.yaml:
--------------------------------------------------------------------------------
1 | draft_documentation:
2 | description:
3 | Write a documentation for "{title}", for the codebase at {repo_path}.
4 | The main goal is {goal}. Using only verified information from the codebase
5 | analysis. Ignore images, videos, and other media files.
6 |
7 | Project overview
8 | {overview}
9 |
10 | High level documentation description for {title}
11 | {description}
12 |
13 | Prerequisites
14 | {prerequisites}
15 |
16 | Examples
17 | {examples}
18 |
19 | Use mermaid art diagrams instead of images to represent flows and
20 | relationships.
21 |
22 | Ignore Images, Videos, and other media files.
23 |
24 | expected_output: >
25 | A factual Markdown documentation that includes only verified information
26 | about {title} that is relevant to the codebase at {repo_path}.
27 | Use mermaid art diagrams to visualize component relationships and flows.
28 | Don't wrap the documentation in fences or meta-commentary.
29 |
30 | The documentation must include this like:
31 | - Section headers matching actual code structure
32 | - Thoughtful explanations with code references
33 | - Code examples from the actual codebase
34 | - Setup instructions verified against the code
35 | - mermaid art diagrams for visual representations (no images)
36 |
37 | qa_review_documentation:
38 | description:
39 | Review and validate the draft documentation for "{title}" against the
40 | actual codebase at {repo_path}.
41 | Also make sure to check existing documentation files for consistency and
42 | accuracy.
43 | Ignore images, videos, and other media files.
44 | We should not have duplicate documentation.
45 |
46 | 1. Technical Accuracy
47 | - Verify all architectural descriptions match implementation
48 | - Validate component relationships and interactions
49 | - Cross-reference code examples with tests and actual usage
50 | - Confirm mermaid diagrams reflect real data/control flows
51 |
52 | 2. Documentation Completeness
53 | - Verify coverage of key components
54 | - Check all documented workflows exist in code
55 | - Ensure integration patterns match implementation
56 | - Confirm troubleshooting scenarios are accurate
57 |
58 | 3. Documentation Quality
59 | - Remove any speculative or unimplemented features
60 | - Update examples to match current code patterns
61 | - Ensure mermaid diagrams enhance understanding
62 |
63 | 4. Technical Consistency
64 | - Align terminology with codebase conventions
65 | - Verify component names match implementation
66 | - Validate code style in examples
67 |
68 | 5. Diagram Validation
69 | - Validate that all diagram components exist in codebase
70 | - Check that flow directions accurately represent system behavior
71 | - Mermaid blocks should be wrapped in ```mermaid```
72 | expected_output: >
73 | A thoroughly validated markdown documentation for {title} that:
74 | - Is 100% aligned with the implementation at {repo_path}
75 | - Contains only verified code examples and workflows
76 | - Has correct mermaid diagrams representing actual system flows
77 | - Maintains consistent technical terminology
78 | - Don't wrap the documentation in fences or meta-commentary.
79 | - Ignore Images, Videos, and other media files.
80 |
81 | The output should be pure markdown without fences or meta-commentary.
82 |
--------------------------------------------------------------------------------
/config/planner_agents.yaml:
--------------------------------------------------------------------------------
1 | code_explorer:
2 | role: >
3 | Code Explorer
4 | goal: >
5 | Analyze codebase structure and identify key components
6 | backstory: >
7 | As a senior software architect with extensive experience in complex
8 | codebases, you excel at understanding and documenting technical systems.
9 | Your expertise lies in breaking down large codebases into understandable
10 | components and identifying critical patterns and relationships.
11 | verbose: false
12 | max_iter: 5
13 | llm: nvidia_nim/meta/llama-3.3-70b-instruct
14 |
15 | documentation_planner:
16 | role: >
17 | Documentation Planner
18 | goal: >
19 | Create a comprehensive documentation strategy based on codebase analysis
20 | backstory: >
21 | You are a documentation strategist with years of experience organizing
22 | technical documentation for complex software projects. You have a talent
23 | for creating clear, user-focused documentation plans that serve both
24 | developers and end-users effectively.
25 | verbose: false
26 | max_iter: 5
27 | llm: nvidia_nim/meta/llama-3.3-70b-instruct
28 |
--------------------------------------------------------------------------------
/config/planner_tasks.yaml:
--------------------------------------------------------------------------------
1 | analyze_codebase:
2 | description:
3 | Analyze the codebase at {repo_path} to create a developer-focused
4 | technical overview
5 |
6 | 1. Map out the core architecture and components
7 |
8 | 2. Identify key classes, functions and their interactions
9 |
10 | 3. Document APIs and interfaces with usage code examples
11 |
12 | 4. Analyze and diagram data/control flows between components
13 |
14 | 5. Note implemented design patterns and their practical applications
15 |
16 | 6. Identify common usage patterns and integration points
17 |
18 | 7. Ignore Images, Videos, and other media files
19 |
20 |
21 | Focus on details that would help another engineer understand and work
22 | with the codebase.
23 | expected_output:
24 | A technical analysis containing
25 |
26 | - What is the project about, what is it meant to do
27 |
28 | - Project overview, deeply explaining the architecture and design
29 |
30 | - Deep documentation of the codebase, with
31 |
32 | - Method signatures and parameters
33 |
34 | - Usage examples and common patterns, using code examples
35 |
36 | - Integration guidelines
37 |
38 | - Explanation of each section
39 |
40 | - Component interaction diagrams
41 |
42 | - Implementation patterns and their use cases
43 |
44 | create_documentation_plan:
45 | description:
46 | Develop engineering-focused documentation plan for {repo_path}
47 |
48 | The plan should be a list of separate documents, that will be written,
49 | and should be ordered by importance and complexity.
50 |
51 | Together, they should cover all the important aspects of the codebase,
52 | and be written in a way that is easy to understand and follow for a
53 | developer.
54 |
55 | Each documents should have a thoughtful title, long meaningful description,
56 | prerequisites, practical examples using code that are clear and comprehensive
57 | goal.
58 |
59 | Make sure the plan also covers things like
60 |
61 | - A comprehensive overview
62 |
63 | - System architecture and key design decisions
64 |
65 | - Core workflows and data flows
66 |
67 | - High-level component interactions
68 |
69 | - Getting started guide
70 |
71 | - Ignore Images, Videos, and other media files
72 |
73 | Focus on creating a clear learning path from overview to advanced topics,
74 | and make sure to include all the important aspects of the codebase.
75 | expected_output:
76 | A documentation plan with a list of documents, ordered by importance and
77 | complexity.
--------------------------------------------------------------------------------
/crewai-nim-flow.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/crewAIInc/nvidia-demo/30896aca51a9ed314d4909ed87d84c583110f668/crewai-nim-flow.png
--------------------------------------------------------------------------------
/crewai_flow.html:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 | Flow Plot
6 |
12 |
19 |
76 |
77 |
78 |
88 |
89 |