├── .devcontainer ├── Dockerfile ├── config_template.yaml ├── devcontainer.json ├── env_template.env ├── postCreateCommand.sh ├── postStartBackground.sh └── postStartCommand.sh ├── .gitignore ├── .vscode ├── autodocstring.mustache ├── launch.json └── tasks.json ├── DeepResearchTool ├── __init__.py ├── config.yaml ├── const.py ├── deep_research_investigator_tool.py ├── deep_research_tool.py ├── deep_research_toolkit.py ├── deep_research_writer_tool.py ├── helpers.py ├── topic_managers.py └── topic_subagent.py ├── LICENSE ├── Makefile ├── README.md ├── alembic.ini ├── decision_log.md ├── docker-compose.yml ├── example_output.md ├── nginx └── default.conf ├── poetry.lock ├── pyproject.toml ├── requirements.txt ├── sample_output ├── FluoridationOfWater.json ├── HealthRisksOfFluoride.json ├── output.md ├── topics.json └── user_query.txt ├── sample_output_2 ├── FluoridationOfWater.json ├── FluorideAbsorptionAndDistribution.json ├── FluorideExposureSources.json ├── HealthRisksOfFluoride.json ├── WaterFluoridationAndCancerRisk.json ├── output.md ├── topics.json └── user_query.txt ├── sample_queries.txt ├── scratch.py └── summary_debug.py /.devcontainer/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM mcr.microsoft.com/vscode/devcontainers/python:0-3.11-bullseye 2 | 3 | -------------------------------------------------------------------------------- /.devcontainer/config_template.yaml: -------------------------------------------------------------------------------- 1 | #####################------------------SYSTEM KEYS-------------------------######################## 2 | PINECONE_API_KEY: YOUR_PINECONE_API_KEY 3 | PINECONE_ENVIRONMENT: YOUR_PINECONE_ENVIRONMENT 4 | 5 | OPENAI_API_KEY: YOUR_OPEN_API_KEY 6 | PALM_API_KEY: YOUR_PALM_API_KEY 7 | 8 | # For locally hosted LLMs comment out the next line and uncomment the one after 9 | # to configure a local llm point your browser to 127.0.0.1:7860 and click on the model tab in text generation web ui. 10 | OPENAI_API_BASE: https://api.openai.com/v1 11 | #OPENAI_API_BASE: "http://super__tgwui:5001/v1" 12 | 13 | # "gpt-3.5-turbo-0301": 4032, "gpt-4-0314": 8092, "gpt-3.5-turbo": 4032, "gpt-4": 8092, "gpt-4-32k": 32768, "gpt-4-32k-0314": 32768, "llama":2048, "mpt-7b-storywriter":45000 14 | MODEL_NAME: "gpt-3.5-turbo-0301" 15 | # "gpt-3.5-turbo", , "gpt-4", "models/chat-bison-001" 16 | RESOURCES_SUMMARY_MODEL_NAME: "gpt-3.5-turbo" 17 | MAX_TOOL_TOKEN_LIMIT: 800 18 | MAX_MODEL_TOKEN_LIMIT: 4032 # set to 2048 for llama 19 | 20 | #DATABASE INFO 21 | # redis details 22 | DB_NAME: super_agi_main 23 | POSTGRES_URL: localhost 24 | DB_USERNAME: superagi 25 | DB_PASSWORD: password 26 | REDIS_URL: "localhost:6379" 27 | 28 | #STORAGE TYPE ("FILE" or "S3") 29 | STORAGE_TYPE: "FILE" 30 | 31 | #TOOLS 32 | TOOLS_DIR: "superagi/tools" 33 | 34 | #STORAGE INFO FOR FILES 35 | RESOURCES_INPUT_ROOT_DIR: workspace/input/{agent_id} 36 | RESOURCES_OUTPUT_ROOT_DIR: workspace/output/{agent_id}/{agent_execution_id} # For keeping resources at agent execution level 37 | #RESOURCES_OUTPUT_ROOT_DIR: workspace/output/{agent_id} # For keeping resources at agent level 38 | 39 | #S3 RELATED DETAILS ONLY WHEN STORAGE_TYPE IS "S3" 40 | BUCKET_NAME: 41 | INSTAGRAM_TOOL_BUCKET_NAME: #Public read bucket, Images generated by stable diffusion are put in this bucket and the public url of the same is generated. 42 | AWS_ACCESS_KEY_ID: 43 | AWS_SECRET_ACCESS_KEY: 44 | 45 | #AUTH 46 | ENV: 'DEV' #DEV,PROD, to use GITHUB OAUTH set to PROD 47 | JWT_SECRET_KEY: 'secret' 48 | expiry_time_hours: 1 49 | 50 | #GITHUB OAUTH: 51 | GITHUB_CLIENT_ID: 52 | GITHUB_CLIENT_SECRET: 53 | FRONTEND_URL: "http://localhost:3000" 54 | 55 | #ENCRPYTION KEY 56 | ENCRYPTION_KEY: secret 57 | 58 | #WEAVIATE 59 | 60 | # If you are using docker or web hosted uncomment the next two lines and comment the third one 61 | # WEAVIATE_URL: YOUR_WEAVIATE_URL 62 | # WEAVIATE_API_KEY: YOUR_WEAVIATE_API_KEY 63 | WEAVIATE_USE_EMBEDDED: true 64 | 65 | 66 | #####################------------------TOOLS KEY-------------------------######################## 67 | #If you have google api key and CSE key, use this 68 | GOOGLE_API_KEY: YOUR_GOOGLE_API_KEY 69 | SEARCH_ENGINE_ID: YOUR_SEARCH_ENIGNE_ID 70 | 71 | # IF YOU DONT HAVE GOOGLE SEARCH KEY, YOU CAN USE SERPER.DEV KEYS 72 | SERP_API_KEY: YOUR_SERPER_API_KEY 73 | 74 | #ENTER YOUR EMAIL CREDENTIALS TO ACCESS EMAIL TOOL 75 | EMAIL_ADDRESS: YOUR_EMAIL_ADDRESS 76 | EMAIL_PASSWORD: YOUR_EMAIL_APP_PASSWORD #get the app password from (https://myaccount.google.com/apppasswords) 77 | EMAIL_SMTP_HOST: smtp.gmail.com #Change the SMTP host if not using Gmail 78 | EMAIL_SMTP_PORT: 587 #Change the SMTP port if not using Gmail 79 | EMAIL_IMAP_SERVER: imap.gmail.com #Change the IMAP Host if not using Gmail 80 | EMAIL_SIGNATURE: Email sent by SuperAGI 81 | EMAIL_DRAFT_MODE_WITH_FOLDER: YOUR_DRAFTS_FOLDER 82 | EMAIL_ATTACHMENT_BASE_PATH: YOUR_DIRECTORY_FOR_EMAIL_ATTACHMENTS 83 | 84 | # GITHUB 85 | GITHUB_USERNAME: YOUR_GITHUB_USERNAME 86 | GITHUB_ACCESS_TOKEN: YOUR_GITHUB_ACCESS_TOKEN 87 | 88 | #JIRA 89 | JIRA_INSTANCE_URL: YOUR_JIRA_INSTANCE_URL 90 | JIRA_USERNAME: YOUR_JIRA_EMAIL 91 | JIRA_API_TOKEN: YOUR_JIRA_API_TOKEN 92 | 93 | #SLACK 94 | SLACK_BOT_TOKEN: YOUR_SLACK_BOT_TOKEN 95 | 96 | # For running stable diffusion 97 | STABILITY_API_KEY: YOUR_STABILITY_API_KEY 98 | #Engine IDs that can be used: 'stable-diffusion-v1', 'stable-diffusion-v1-5','stable-diffusion-512-v2-0', 'stable-diffusion-768-v2-0','stable-diffusion-512-v2-1','stable-diffusion-768-v2-1','stable-diffusion-xl-beta-v2-2-2' 99 | ENGINE_ID: "stable-diffusion-xl-beta-v2-2-2" 100 | 101 | ## To config a vector store for resources manager uncomment config below 102 | ## based on the vector store you want to use 103 | 104 | ## RESOURCE_VECTOR_STORE can be REDIS, PINECONE, CHROMA, QDRANT 105 | #RESOURCE_VECTOR_STORE: YOUR_RESOURCE_VECTOR_STORE 106 | #RESOURCE_VECTOR_STORE_INDEX_NAME: YOUR_RESOURCE_VECTOR_STORE_INDEX_NAME 107 | 108 | ## To use a custom redis 109 | #REDIS_VECTOR_STORE_URL: YOUR_REDIS_VECTOR_STORE_URL 110 | 111 | ## To use qdrant for vector store in resources manager 112 | #QDRANT_PORT: YOUR_QDRANT_PORT 113 | #QDRANT_HOST_NAME: YOUR_QDRANT_HOST_NAME 114 | 115 | ## To use chroma for vector store in resources manager 116 | #CHROMA_HOST_NAME: YOUR_CHROMA_HOST_NAME 117 | #CHROMA_PORT: YOUR_CHROMA_PORT 118 | 119 | ## To use Qdrant for vector store 120 | #QDRANT_HOST_NAME: YOUR_QDRANT_HOST_NAME 121 | #QDRANT_PORT: YOUR_QDRANT_PORT 122 | -------------------------------------------------------------------------------- /.devcontainer/devcontainer.json: -------------------------------------------------------------------------------- 1 | // For format details, see https://aka.ms/devcontainer.json. For config options, see the 2 | // README at: https://github.com/devcontainers/templates/tree/main/src/python 3 | { 4 | "name": "research-agi-agent", 5 | // Or use a Dockerfile or Docker Compose file. More info: https://containers.dev/guide/dockerfile 6 | "build": { 7 | "dockerfile": "Dockerfile" 8 | }, 9 | // Features to add to the dev container. More info: https://containers.dev/features. 10 | // "features": {}, 11 | // Use 'forwardPorts' to make a list of ports inside the container available locally. 12 | // "forwardPorts": [], 13 | // Use 'postCreateCommand' to run commands after the container is created. 14 | "postCreateCommand": "bash .devcontainer/postCreateCommand.sh", 15 | "postStartCommand": "bash .devcontainer/postStartCommand.sh", 16 | // Configure tool-specific properties. 17 | // "customizations": {}, 18 | // Uncomment to connect as root instead. More info: https://aka.ms/dev-containers-non-root. 19 | // "remoteUser": "root", 20 | "customizations": { 21 | "vscode": { 22 | "extensions": [ 23 | "ms-python.python", 24 | "tamasfe.even-better-toml", 25 | "njpwerner.autodocstring", 26 | "mhutchie.git-graph", 27 | "eamodio.gitlens", 28 | "charliermarsh.ruff", 29 | "ms-azuretools.vscode-docker", 30 | "matangover.mypy" 31 | ], 32 | "settings": { 33 | "files.insertFinalNewline": true, 34 | "editor.formatOnSave": true, 35 | "files.trimTrailingWhitespace": true, 36 | "python.defaultInterpreterPath": "./.venv/bin/python", 37 | "python.formatting.provider": "black", 38 | "python.formatting.blackArgs": [ 39 | "--no-color" 40 | ], 41 | "python.linting.mypyEnabled": false, 42 | "mypy.runUsingActiveInterpreter": true, 43 | "python.testing.pytestArgs": [ 44 | "tests" 45 | ], 46 | "python.testing.unittestEnabled": false, 47 | "python.testing.pytestEnabled": true, 48 | "python.analysis.autoImportCompletions": true, 49 | "python.analysis.indexing": true, 50 | "python.linting.flake8Enabled": false, 51 | "editor.codeActionsOnSave": { 52 | "source.organizeImports": true 53 | }, 54 | "autoDocstring.customTemplatePath": ".vscode/autodocstring.mustache", 55 | "ruff.path": [ 56 | "./.venv/bin/ruff" 57 | ] 58 | }, 59 | "git.branchProtection": [ 60 | "main", 61 | "master" 62 | ] 63 | } 64 | }, 65 | "features": { 66 | "ghcr.io/devcontainers-contrib/features/poetry:2": { 67 | "version": "latest" 68 | }, 69 | "ghcr.io/devcontainers/features/docker-in-docker:2": {}, 70 | // install sshd for github codespaces using JetBrains 71 | "ghcr.io/devcontainers/features/sshd:1": { 72 | "version": "latest" 73 | } 74 | }, 75 | // mount the .venv directory as a named volume for performance reasons 76 | "mounts": [ 77 | "source=${localWorkspaceFolderBasename}-venv,target=${containerWorkspaceFolder}/.venv,type=volume" 78 | ], 79 | "workspaceFolder": "/workspaces/research-agi-agent", 80 | "hostRequirements": { 81 | "cpus": 4, 82 | "memory": "8gb" 83 | } 84 | } 85 | -------------------------------------------------------------------------------- /.devcontainer/env_template.env: -------------------------------------------------------------------------------- 1 | PYTHONPATH=/workspaces/SuperAGI 2 | -------------------------------------------------------------------------------- /.devcontainer/postCreateCommand.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | set -e 3 | 4 | # git settings 5 | git config --global pull.rebase true 6 | git config --global remote.origin.prune true 7 | 8 | # if the .venv directory was mounted as a named volume, it needs the ownership changed 9 | sudo chown vscode .venv || true 10 | 11 | # make the python binary location predictable 12 | poetry config virtualenvs.in-project true 13 | poetry install --with=dev || true 14 | 15 | mkdir -p .dev_container_logs 16 | echo "*" > .dev_container_logs/.gitignore 17 | 18 | # git clone SuperAGI if it doesn't exist 19 | if [ ! -d ../SuperAGI ]; then 20 | git clone https://github.com/TransformerOptimus/SuperAGI.git /workspaces/SuperAGI 21 | fi 22 | 23 | # make a copy of the SuperAGI config from the template if it does not exist 24 | if [ ! -f /workspaces/SuperAGI/config.yaml ]; then 25 | cp /workspaces/research-agi-agent/.devcontainer/config_template.yaml /workspaces/SuperAGI/config.yaml 26 | fi 27 | 28 | # soft link to superAGI 29 | # make external_tools directory if it doesnt exist 30 | mkdir -p /workspaces/SuperAGI/superagi/tools/external_tools 31 | # soft link DeepResearchTool if it doesn't exist 32 | if [ ! -d /workspaces/SuperAGI/superagi/tools/external_tools/DeepResearchTool ]; then 33 | ln -s /workspaces/research-agi-agent/DeepResearchTool /workspaces/SuperAGI/superagi/tools/external_tools/DeepResearchTool 34 | fi 35 | 36 | # write env from template if needed 37 | if [ ! -f /workspaces/research-agi-agent/.env ]; then 38 | cp /workspaces/research-agi-agent/.devcontainer/env_template /workspaces/research-agi-agent/.env 39 | fi 40 | 41 | # setup SuperAGI 42 | cd /workspaces/SuperAGI 43 | pip install -r requirements.txt 44 | -------------------------------------------------------------------------------- /.devcontainer/postStartBackground.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | -------------------------------------------------------------------------------- /.devcontainer/postStartCommand.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # run in the background at startup 4 | nohup bash -c ".devcontainer/postStartBackground.sh &" > ".dev_container_logs/postStartBackground.out" 5 | 6 | # note: do NOT have the last command run in the background else it won't really run! 7 | docker compose up -d super__redis super__postgres 8 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | build/ 12 | develop-eggs/ 13 | dist/ 14 | downloads/ 15 | eggs/ 16 | .eggs/ 17 | lib/ 18 | lib64/ 19 | parts/ 20 | sdist/ 21 | var/ 22 | wheels/ 23 | share/python-wheels/ 24 | *.egg-info/ 25 | .installed.cfg 26 | *.egg 27 | MANIFEST 28 | 29 | # PyInstaller 30 | # Usually these files are written by a python script from a template 31 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 32 | *.manifest 33 | *.spec 34 | 35 | # Installer logs 36 | pip-log.txt 37 | pip-delete-this-directory.txt 38 | 39 | # Unit test / coverage reports 40 | htmlcov/ 41 | .tox/ 42 | .nox/ 43 | .coverage 44 | .coverage.* 45 | .cache 46 | nosetests.xml 47 | coverage.xml 48 | *.cover 49 | *.py,cover 50 | .hypothesis/ 51 | .pytest_cache/ 52 | cover/ 53 | 54 | # Translations 55 | *.mo 56 | *.pot 57 | 58 | # Django stuff: 59 | *.log 60 | local_settings.py 61 | db.sqlite3 62 | db.sqlite3-journal 63 | 64 | # Flask stuff: 65 | instance/ 66 | .webassets-cache 67 | 68 | # Scrapy stuff: 69 | .scrapy 70 | 71 | # Sphinx documentation 72 | docs/_build/ 73 | 74 | # PyBuilder 75 | .pybuilder/ 76 | target/ 77 | 78 | # Jupyter Notebook 79 | .ipynb_checkpoints 80 | 81 | # IPython 82 | profile_default/ 83 | ipython_config.py 84 | 85 | # pyenv 86 | # For a library or package, you might want to ignore these files since the code is 87 | # intended to run in multiple environments; otherwise, check them in: 88 | # .python-version 89 | 90 | # pipenv 91 | # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. 92 | # However, in case of collaboration, if having platform-specific dependencies or dependencies 93 | # having no cross-platform support, pipenv may install dependencies that don't work, or not 94 | # install all needed dependencies. 95 | #Pipfile.lock 96 | 97 | # poetry 98 | # Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control. 99 | # This is especially recommended for binary packages to ensure reproducibility, and is more 100 | # commonly ignored for libraries. 101 | # https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control 102 | #poetry.lock 103 | 104 | # pdm 105 | # Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control. 106 | #pdm.lock 107 | # pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it 108 | # in version control. 109 | # https://pdm.fming.dev/#use-with-ide 110 | .pdm.toml 111 | 112 | # PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm 113 | __pypackages__/ 114 | 115 | # Celery stuff 116 | celerybeat-schedule 117 | celerybeat.pid 118 | 119 | # SageMath parsed files 120 | *.sage.py 121 | 122 | # Environments 123 | .env 124 | .venv 125 | env/ 126 | venv/ 127 | ENV/ 128 | env.bak/ 129 | venv.bak/ 130 | 131 | # Spyder project settings 132 | .spyderproject 133 | .spyproject 134 | 135 | # Rope project settings 136 | .ropeproject 137 | 138 | # mkdocs documentation 139 | /site 140 | 141 | # mypy 142 | .mypy_cache/ 143 | .dmypy.json 144 | dmypy.json 145 | 146 | # Pyre type checker 147 | .pyre/ 148 | 149 | # pytype static type analyzer 150 | .pytype/ 151 | 152 | # Cython debug symbols 153 | cython_debug/ 154 | 155 | # PyCharm 156 | # JetBrains specific template is maintained in a separate JetBrains.gitignore that can 157 | # be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore 158 | # and can be added to the global gitignore or merged into this file. For a more nuclear 159 | # option (not recommended) you can uncomment the following to ignore the entire idea folder. 160 | #.idea/ 161 | 162 | */secrets.toml 163 | 164 | SUMMARY_DEBUG_OUTPUT.md 165 | -------------------------------------------------------------------------------- /.vscode/autodocstring.mustache: -------------------------------------------------------------------------------- 1 | {{! https://github.com/NilsJPWerner/autoDocstring/tree/master/src/docstring/templates for examples }} 2 | {{! this template is based off of Sphinx-notype }} 3 | {{! changes: hamburger style, starts on new line. No default descriptor }} 4 | 5 | {{summaryPlaceholder}} 6 | 7 | {{extendedSummaryPlaceholder}} 8 | 9 | {{#args}} 10 | :param {{var}}: {{descriptionPlaceholder}} 11 | {{/args}} 12 | {{#kwargs}} 13 | :param {{var}}: {{descriptionPlaceholder}} 14 | {{/kwargs}} 15 | {{#exceptions}} 16 | :raises {{type}}: {{descriptionPlaceholder}} 17 | {{/exceptions}} 18 | {{#returns}} 19 | :return: {{descriptionPlaceholder}} 20 | {{/returns}} 21 | {{#yields}} 22 | :yield: {{descriptionPlaceholder}} 23 | {{/yields}} -------------------------------------------------------------------------------- /.vscode/launch.json: -------------------------------------------------------------------------------- 1 | { 2 | // Use IntelliSense to learn about possible attributes. 3 | // Hover to view descriptions of existing attributes. 4 | // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387 5 | "version": "0.2.0", 6 | "configurations": [ 7 | { 8 | "name": "Python: Current File", 9 | "type": "python", 10 | "request": "launch", 11 | "program": "${file}", 12 | "console": "integratedTerminal", 13 | "justMyCode": true 14 | }, 15 | { 16 | "name": "Run summary debug", 17 | "type": "python", 18 | "request": "launch", 19 | "program": "${workspaceFolder}/summary_debug.py", 20 | "console": "integratedTerminal", 21 | "justMyCode": true 22 | }, 23 | { 24 | "name": "Run backend", 25 | "type": "python", 26 | "request": "launch", 27 | "module": "uvicorn", 28 | "envFile": "${workspaceFolder}/.env", 29 | "args": [ 30 | "main:app", 31 | "--host", 32 | "0.0.0.0", 33 | "--port", 34 | "8001", 35 | "--reload" 36 | ], 37 | "justMyCode": false, 38 | "cwd": "/workspaces/SuperAGI", 39 | "preLaunchTask": "Run migrations" 40 | }, 41 | { 42 | "name": "Run Celery", 43 | "type": "python", 44 | "request": "launch", 45 | "module": "celery", 46 | "envFile": "${workspaceFolder}/.env", 47 | "args": [ 48 | "-A", 49 | "superagi.worker", 50 | "worker", 51 | "--beat", 52 | "--loglevel=info" 53 | ], 54 | "justMyCode": false, 55 | "cwd": "/workspaces/SuperAGI", 56 | "preLaunchTask": "Run migrations" 57 | }, 58 | { 59 | "name": "Python: Debug Unit Tests", 60 | "type": "python", 61 | "request": "launch", 62 | "purpose": [ 63 | "debug-test" 64 | ], 65 | "console": "integratedTerminal", 66 | "justMyCode": false, 67 | }, 68 | ], 69 | "compounds": [ 70 | { 71 | "name": "Run backend and Celery", 72 | "configurations": [ 73 | "Run backend", 74 | "Run Celery" 75 | ], 76 | } 77 | ] 78 | } 79 | -------------------------------------------------------------------------------- /.vscode/tasks.json: -------------------------------------------------------------------------------- 1 | { 2 | // See https://go.microsoft.com/fwlink/?LinkId=733558 3 | // for the documentation about the tasks.json format 4 | "version": "2.0.0", 5 | "tasks": [ 6 | { 7 | "label": "Start Super AGI docker containers", 8 | "type": "shell", 9 | "command": "docker compose up", 10 | "problemMatcher": [] 11 | }, 12 | { 13 | "label": "Regenerate requirements.txt", 14 | "type": "shell", 15 | "command": "poetry export --output requirements.txt", 16 | "problemMatcher": [] 17 | }, 18 | { 19 | "label": "Run migrations", 20 | "type": "shell", 21 | "command": "cd /workspaces/research-agi-agent && PYTHONPATH=/workspaces/SuperAGI alembic upgrade head", 22 | "problemMatcher": [] 23 | }, 24 | { 25 | "label": "Get run instructions", 26 | "type": "shell", 27 | "command": "echo '\nUsing codespaces:\n\nRun the backend and Celery launch configuration. Then run the \"Start Super AGI docker containers\" task.\nAfter that, go to ports in VS Code and change port 3000 to public, and click on the icon to open it in a browser\n\n'", 28 | "problemMatcher": [] 29 | }, 30 | { 31 | "label": "How to search the SuperAGI repo", 32 | "type": "shell", 33 | "command": "echo '\nIf you add \"/workspaces/SuperAGI\" it to the workspace, the Rust extension will break due to a bug.\nInstead, you can set the files to include to \"/workspaces/SuperAGI\" and for files to exclude,\ndisable the \"Use Exclude Settings and Ignore Files\" button (cog with minus sign)\n\n'", 34 | "problemMatcher": [] 35 | } 36 | ] 37 | } 38 | -------------------------------------------------------------------------------- /DeepResearchTool/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/JamesHutchison/research-agi-agent/03d079b5c16bb91c3449262982ad5f7f71f6707b/DeepResearchTool/__init__.py -------------------------------------------------------------------------------- /DeepResearchTool/config.yaml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/JamesHutchison/research-agi-agent/03d079b5c16bb91c3449262982ad5f7f71f6707b/DeepResearchTool/config.yaml -------------------------------------------------------------------------------- /DeepResearchTool/const.py: -------------------------------------------------------------------------------- 1 | TOPICS_FILE = "topics.json" 2 | USER_QUERY_FILE = "user_query.txt" 3 | 4 | SINGLE_FILE_OUTPUT_FILE = "output.md" 5 | MAX_TOPICS = 5 6 | -------------------------------------------------------------------------------- /DeepResearchTool/deep_research_investigator_tool.py: -------------------------------------------------------------------------------- 1 | import json 2 | import logging 3 | from typing import Optional, Type 4 | 5 | from pydantic import BaseModel, Field 6 | from superagi.llms.base_llm import BaseLlm 7 | from superagi.resource_manager.file_manager import FileManager 8 | from superagi.tools.base_tool import BaseTool 9 | 10 | from DeepResearchTool.const import TOPICS_FILE 11 | from DeepResearchTool.topic_managers import Topic, TopicsManager 12 | from DeepResearchTool.topic_subagent import TopicSubAgent 13 | 14 | 15 | class DeepResearchInvestigator(BaseModel): 16 | topic: str = Field(..., description="The name of the topic. This must match exactly.") 17 | 18 | 19 | class DeepResearchInvestigatorTool(BaseTool): 20 | """ 21 | Tool for doing deep research on a topic. It may create more topics to research. 22 | """ 23 | 24 | name: str = "Deep Research Investigator Tool" 25 | args_schema: Type[BaseModel] = DeepResearchInvestigator 26 | description: str = ( 27 | "Tool to research a topic by searching the internet, taking notes, and identifying possible " 28 | "other topics. This tool will complete the research for this topic." 29 | ) 30 | llm: Optional[BaseLlm] = None 31 | resource_manager: Optional[FileManager] = None 32 | 33 | def _validate_topic(self, topic: str, topics: list[Topic]) -> str | None: 34 | assert self.resource_manager 35 | 36 | for t in topics: 37 | if t.name == topic: 38 | if t.researched: 39 | return ( 40 | f"Topic {topic} has already been researched. " 41 | "Use the Deep Research Tool for guidance on what to do next." 42 | ) 43 | return None 44 | return ( 45 | f"Topic {topic} was not found. Be sure to name the topic exactly as it was given and " 46 | "use the Deep Research Tool to find the next topic." 47 | ) 48 | 49 | def _execute(self, topic: str) -> str: 50 | try: 51 | assert self.resource_manager 52 | 53 | tm = TopicsManager(self.resource_manager) 54 | 55 | topics = tm.load_topics() 56 | if (err_msg := self._validate_topic(topic, topics)) is not None: 57 | return err_msg 58 | 59 | for topic_obj in topics: 60 | if topic_obj.name == topic: 61 | TopicSubAgent(topic_obj, self.resource_manager, self.llm).do_research() 62 | 63 | # reload topics and determine if still some remaining 64 | topics = tm.load_topics() 65 | 66 | return_str = f"Deep research completed for topic {topic}! Do not research {topic} again!" 67 | for potential_next_topic in topics: 68 | if not potential_next_topic.researched: 69 | return_str += ( 70 | "\n\nNew topics to research may have been created! " 71 | f"Next topic to research: {potential_next_topic.name}" 72 | ) 73 | break 74 | return return_str 75 | raise Exception("Should not be here! Topic validation failed!") 76 | except Exception as e: 77 | logging.exception("") 78 | raise 79 | -------------------------------------------------------------------------------- /DeepResearchTool/deep_research_tool.py: -------------------------------------------------------------------------------- 1 | import json 2 | from typing import Optional, Type 3 | 4 | from pydantic import BaseModel, Field 5 | from superagi.llms.base_llm import BaseLlm 6 | from superagi.resource_manager.file_manager import FileManager 7 | from superagi.tools.base_tool import BaseTool 8 | 9 | from DeepResearchTool.const import USER_QUERY_FILE 10 | from DeepResearchTool.helpers import do_chat_and_log_it 11 | from DeepResearchTool.topic_managers import Topic, TopicsManager 12 | 13 | 14 | class DeepResearch(BaseModel): 15 | user_query: str = Field( 16 | ..., 17 | description="The user query. Only provide this the first time you use this tool. " 18 | "This should match exactly what the user provided.", 19 | ) 20 | 21 | 22 | class DeepResearchManagerTool(BaseTool): 23 | """ 24 | Tool for starting and managing deep research. This should be used prior to doing any other deep research tools. 25 | """ 26 | 27 | name: str = "Deep Research Tool" 28 | args_schema: Type[BaseModel] = DeepResearch 29 | description: str = ( 30 | "Starts and manages the progress of deep researching a user query." 31 | "Helps determine the next steps to take. If you are stuck on what to do next " 32 | "while doing deep research, then use this tool." 33 | ) 34 | llm: Optional[BaseLlm] = None 35 | resource_manager: Optional[FileManager] = None 36 | 37 | def _user_query_issues(self, user_query: str | None = None) -> str | None: 38 | assert self.llm 39 | 40 | if user_query is None: 41 | return "A user query was not provided and is required" 42 | query_issues_instructions = "Determine whether the user query is a good fit for research." 43 | chat_query = [ 44 | { 45 | "role": "system", 46 | "content": " ".join( 47 | [ 48 | "You are a research tool, and will be researching this user query.", 49 | query_issues_instructions, 50 | ] 51 | ), 52 | }, 53 | { 54 | "role": "user", 55 | "content": "user query: " + user_query, 56 | }, 57 | { 58 | "role": "system", 59 | "content": ( 60 | """ 61 | Use the following rules to determine if a user query previously given is appropriate for research: 62 | 63 | The topic is based on scientific, economic, or sociological principles. 64 | The topic is not based on conspiracy theories or unproven myths. 65 | The topic could be researched on the Internet. 66 | Not Appropriate for Deep Extensive Research: 67 | 68 | The topic is based on conspiracy theories, myths, or fictional characters. 69 | The topic is written in a biased, sarcastic, or mocking tone. 70 | The topic is too narrow or specific to warrant extensive research. 71 | The topic is about a personal issue or problem. 72 | 73 | Good example: 74 | Write a research paper on how quantum computers are possible 75 | 76 | Bad example: 77 | Why is my light bulb flickering? 78 | """ 79 | "Reason through why the user query is appropriate. If is, state the reasons and reply with '!!!OK!!!' " 80 | "Otherwise, reply with a short sentence what is wrong with the query." 81 | ), 82 | }, 83 | ] 84 | result = do_chat_and_log_it(self.llm, chat_query) 85 | if "!!!OK!!!" in result: 86 | return None 87 | return result 88 | 89 | def _extract_topics_from_user_query(self, user_query: str) -> str: 90 | assert self.llm 91 | 92 | result = do_chat_and_log_it( 93 | self.llm, 94 | [ 95 | { 96 | "role": "system", 97 | "content": ( 98 | "You are a research tool. " 99 | "Examine the following information and extract the relevant topics of research." 100 | ), 101 | }, 102 | { 103 | "role": "user", 104 | "content": user_query, 105 | }, 106 | { 107 | "role": "system", 108 | "content": ( 109 | """ 110 | Examine the following information and extract the broadest and most general research topics. 111 | Provide a JSON list of these topics, each with a 'name', 'description', 'relevant_because', and 'notes_file' field. 112 | No more than 5 major topics and prefer 2 or less. The revelant_because field should be one or two sentences. 113 | When creating topics, avoid specific permutations, subtopics, or detailed breakdowns. Each topic name should be self-explanatory 114 | and understandable without additional context. If uncertain about a topic, leave the 'description' field empty. 115 | For example, instead of 'Environmental Impact', use 'Fluoride's Environmental Consequences'. 116 | The 'name', 'description', 'relevant_because', and 'notes_file' fields are all required! 117 | The notes_file value should be a file name, no spaces, with a .json extension.""" 118 | "If you don't know what something is, leave the description field as an empty string.\n\nJSON:\n" 119 | ), 120 | }, 121 | ], 122 | ) 123 | return result 124 | 125 | def _first_time_ran(self, user_query: str, resource_manager: FileManager) -> str: 126 | # run user query validation tool to see if we should proceed 127 | if user_query_issues := self._user_query_issues(user_query): 128 | return ( 129 | f"The provided user query is not a good fit for deep research. Reasons: {user_query_issues}. " 130 | "Recommending aborting this goal and considering it complete." 131 | ) 132 | 133 | # get topics and write to a file 134 | topics_str: str = self._extract_topics_from_user_query(user_query) 135 | # TODO: possibly use LangChain to auto-fix 136 | topics: list[Topic] = [Topic(**t) for t in json.loads(topics_str)] 137 | 138 | # initialize knowledge graph 139 | # TODO 140 | 141 | TopicsManager(resource_manager).write_topics(topics) 142 | for topic in topics: 143 | topic.initialize_notes_file(resource_manager) 144 | 145 | resource_manager.write_file(USER_QUERY_FILE, user_query) 146 | 147 | return f""" 148 | The research topics are: {topics} 149 | 150 | The process for doing deep research is to investigate the topics, take notes, 151 | evaluate if there are any new topics uncovered to research, and iterate through 152 | all the topics. Afterwards, coherently come to a conclusion, then use the 153 | Deep Research Writer Tool to write the output. 154 | 155 | The next step is to pick a topic and use the Deep Research Investigator Tool with it. 156 | """ 157 | 158 | def _get_next_topic(self, resource_manager: FileManager) -> str | None: 159 | topics = TopicsManager(resource_manager).load_topics() 160 | for topic in topics: 161 | if not topic.researched: 162 | return topic.name 163 | return None 164 | 165 | def _execute(self, user_query: str | None = None) -> str: 166 | assert self.resource_manager 167 | assert self.llm 168 | 169 | self.llm.temperature = 0 170 | 171 | if not self.resource_manager.get_files(): 172 | assert user_query 173 | 174 | return self._first_time_ran(user_query, self.resource_manager) 175 | if (next_topic := self._get_next_topic(self.resource_manager)) is not None: 176 | return f"Use the Deep Research Investigator Tool with the topic: {next_topic}" 177 | return "All topics have been researched. Use the Deep Research Writer Tool to write the output." 178 | -------------------------------------------------------------------------------- /DeepResearchTool/deep_research_toolkit.py: -------------------------------------------------------------------------------- 1 | from abc import ABC 2 | from typing import List, Type 3 | 4 | from deep_research_investigator_tool import DeepResearchInvestigatorTool 5 | from deep_research_tool import DeepResearchManagerTool 6 | from deep_research_writer_tool import DeepResearchWriterTool 7 | from superagi.tools.base_tool import BaseTool, BaseToolkit 8 | 9 | 10 | class DeepResearchToolkit(BaseToolkit, ABC): 11 | name: str = "Deep Research Toolkit" 12 | description: str = "Toolkit for doing deep research and writing summarized findings." 13 | 14 | def get_tools(self) -> List[BaseTool]: 15 | return [ 16 | DeepResearchManagerTool(), 17 | DeepResearchWriterTool(), 18 | DeepResearchInvestigatorTool(), 19 | ] 20 | 21 | def get_env_keys(self) -> List[str]: 22 | return [] 23 | -------------------------------------------------------------------------------- /DeepResearchTool/deep_research_writer_tool.py: -------------------------------------------------------------------------------- 1 | import json 2 | import logging 3 | from typing import Optional, Type 4 | 5 | from langchain.chat_models import ChatOpenAI 6 | from langchain.schema import SystemMessage 7 | from pydantic import BaseModel, Field 8 | from superagi.llms.base_llm import BaseLlm 9 | from superagi.resource_manager.file_manager import FileManager 10 | from superagi.tools.base_tool import BaseTool 11 | 12 | from DeepResearchTool.const import SINGLE_FILE_OUTPUT_FILE, TOPICS_FILE, USER_QUERY_FILE 13 | 14 | 15 | class DeepResearchWriter(BaseModel): 16 | desired_output_format: str = Field( 17 | ..., 18 | description="The desired output format. The only valid value at this time is 'single_file'", 19 | ) 20 | 21 | 22 | class DeepResearchWriterTool(BaseTool): 23 | """ 24 | Tool for writing the output of the Deep research tool. If deep research was not done, this tool will fail to run 25 | """ 26 | 27 | name: str = "Deep Research Writer Tool" 28 | args_schema: Type[BaseModel] = DeepResearchWriter 29 | description: str = "Takes the results of the deep research and writes the output format" 30 | llm: Optional[BaseLlm] = None 31 | resource_manager: Optional[FileManager] = None 32 | 33 | def _execute(self, desired_output_format: str | None = None) -> str: 34 | assert self.resource_manager 35 | assert self.llm 36 | 37 | self.llm.temperature = 0 38 | 39 | user_query = self.resource_manager.read_file(USER_QUERY_FILE) 40 | topics = self.resource_manager.read_file(TOPICS_FILE) 41 | 42 | topics_str_list = [] 43 | 44 | for topic in json.loads(topics): 45 | notes = self.resource_manager.read_file(topic["notes_file"]) 46 | # format is: 47 | # name, description, notes_file, relevant_because, researched 48 | topic_str = f""" 49 | Topic name: {topic["name"]} 50 | Topic description: {topic["description"]} 51 | Relevant because: {topic["relevant_because"]} 52 | Notes: {notes} 53 | """ 54 | topics_str_list.append(topic_str) 55 | 56 | markdown_prompt = f""" 57 | The user query is: {user_query} 58 | 59 | ### 60 | 61 | Given the following topics and notes about the topic, write an article addressing the user query 62 | the best you can. If there is an question, try to answer it. If the user query has incorrect 63 | facts or assumptions, address that. 64 | 65 | Start with a problem statement of some sort based on the user query, then follow up with a conclusion. 66 | After the conclusion, explain how that conclusion was derived from the 67 | topics researched. If needed, create a section for relevant topic, if it is important enough, 68 | and explain how the topic contributes to the conclusion. You do not need to specifically mention 69 | the conclusion when describing topics. 70 | 71 | When you can, cite your sources 72 | 73 | ### The topics are: 74 | 75 | {" # next topic # ".join(topics_str_list)} 76 | 77 | # Reminder! The conclusion should be helpful and specific. If there are upper and lower bounds or circumstances where something 78 | may be true or false, then define it. If you cannot, then identify further research needed to get there. Do not make anything up! 79 | If you do not know why you know something, then do not mention it, or identify further research needed to confirm it. 80 | 81 | Use inline citations. 82 | 83 | Markdown file contents: 84 | """ 85 | logging.warning(markdown_prompt) 86 | 87 | OPEN_AI_MODEL = "gpt-4-32k" # not yet available 88 | OPEN_AI_MODEL = "gpt-4" 89 | 90 | chat = ChatOpenAI(model=OPEN_AI_MODEL, temperature=0) 91 | 92 | system_message_prompt = SystemMessage(content=markdown_prompt) 93 | response = chat([system_message_prompt]) 94 | content = response.content 95 | 96 | # content = self.llm.chat_completion([{"role": "system", "content": markdown_prompt}])[ 97 | # "content" 98 | # ] 99 | 100 | self.resource_manager.write_file(SINGLE_FILE_OUTPUT_FILE, content) 101 | 102 | return f"Deep research completed! Check the resource manager for {SINGLE_FILE_OUTPUT_FILE} to view the result!" 103 | -------------------------------------------------------------------------------- /DeepResearchTool/helpers.py: -------------------------------------------------------------------------------- 1 | import logging 2 | 3 | from superagi.llms.base_llm import BaseLlm 4 | 5 | 6 | def do_chat_and_log_it(llm: BaseLlm, messages: list[dict]) -> str: 7 | logging.warning(messages) 8 | logging.warning("DOING CHAT") 9 | response = llm.chat_completion(messages) 10 | logging.warning(response) 11 | return response["content"] 12 | -------------------------------------------------------------------------------- /DeepResearchTool/topic_managers.py: -------------------------------------------------------------------------------- 1 | import json 2 | import logging 3 | from dataclasses import asdict, dataclass 4 | 5 | from superagi.resource_manager.file_manager import FileManager 6 | 7 | from DeepResearchTool.const import TOPICS_FILE 8 | 9 | 10 | @dataclass 11 | class Topic: 12 | name: str 13 | description: str 14 | notes_file: str 15 | relevant_because: str 16 | researched: bool = False 17 | 18 | def initialize_notes_file(self, file_manager: FileManager) -> None: 19 | logging.info(f"Initializing notes file: {self.notes_file}") 20 | file_manager.write_file(self.notes_file, json.dumps([])) 21 | 22 | def mark_as_researched(self, file_manager: FileManager) -> None: 23 | topics_file = json.loads(file_manager.read_file(TOPICS_FILE)) 24 | for topic in topics_file: 25 | if topic["name"] == self.name: 26 | topic["researched"] = True 27 | break 28 | file_manager.write_file(TOPICS_FILE, json.dumps(topics_file)) 29 | 30 | 31 | class TopicsManager: 32 | def __init__(self, file_manager: FileManager) -> None: 33 | self._file_manager = file_manager 34 | 35 | def load_topics(self) -> list[Topic]: 36 | return [Topic(**topic) for topic in json.loads(self._file_manager.read_file(TOPICS_FILE))] 37 | 38 | def write_topics(self, topics: list[Topic]) -> None: 39 | self._file_manager.write_file( 40 | TOPICS_FILE, json.dumps([asdict(topic) for topic in topics]) 41 | ) 42 | -------------------------------------------------------------------------------- /DeepResearchTool/topic_subagent.py: -------------------------------------------------------------------------------- 1 | import json 2 | import logging 3 | from dataclasses import asdict 4 | 5 | from langchain.utilities import GoogleSearchAPIWrapper 6 | from superagi.helper.webpage_extractor import WebpageExtractor 7 | from superagi.llms.base_llm import BaseLlm 8 | from superagi.resource_manager.file_manager import FileManager 9 | 10 | from DeepResearchTool.const import MAX_TOPICS, USER_QUERY_FILE 11 | from DeepResearchTool.helpers import do_chat_and_log_it 12 | from DeepResearchTool.topic_managers import Topic, TopicsManager 13 | 14 | 15 | class TopicSubAgent: 16 | def __init__(self, topic: Topic, file_manager: FileManager, llm: BaseLlm) -> None: 17 | self._topic = topic 18 | self._file_manager = file_manager 19 | self._llm = llm 20 | 21 | def _get_query_for_topic(self, user_query: str) -> str: 22 | # TODO: pass in high instructions given to the agent 23 | # TODO: give prior search queries to avoid duplications 24 | query = ( 25 | f"Given the topic: {self._topic.name} and the description: {self._topic.description} " 26 | f"with the context that this topic came from the original user query: {user_query}," 27 | "come up with a single web search query to try to find information on the topic." 28 | ) 29 | result = do_chat_and_log_it(self._llm, [{"role": "system", "content": query}]) 30 | return result.strip('"') 31 | 32 | def _get_target_urls_for_results(self, search_results) -> list[str]: 33 | try: 34 | return [result["link"] for result in search_results] 35 | except: 36 | logging.exception(f"Could not get links from {search_results}") 37 | raise 38 | 39 | def do_research(self) -> str: 40 | user_query = self._file_manager.read_file(USER_QUERY_FILE) 41 | self.web_search(user_query) 42 | self.evaluate_next_steps() 43 | self._topic.mark_as_researched(self._file_manager) 44 | 45 | def web_search(self, user_query) -> None: 46 | search_query = self._get_query_for_topic(user_query) 47 | 48 | search = GoogleSearchAPIWrapper() 49 | # simple logic - top 3 results for now 50 | TOP_N_RESULTS = 3 51 | search_results = search.results(search_query, TOP_N_RESULTS) 52 | 53 | target_urls = self._get_target_urls_for_results(search_results) 54 | for url in target_urls: 55 | contents = self.scrape_web_page(url) 56 | notes = self.extract_notes(url, contents, user_query) 57 | self.update_notes_file(url, notes) 58 | 59 | topics = TopicsManager(self._file_manager).load_topics() 60 | if len(topics) >= MAX_TOPICS: 61 | logging.warning(f"No longer adding topics beyond {MAX_TOPICS}") 62 | continue 63 | subtopics = self.extract_subtopics(contents, user_query) 64 | self.maybe_add_subtopics(subtopics) 65 | 66 | def scrape_web_page(self, url: str) -> str: 67 | contents = WebpageExtractor().extract_with_bs4(url) 68 | contents = contents[:3000] 69 | return contents 70 | 71 | def extract_notes(self, url: str, contents: str, user_query: str) -> str: 72 | topic = self._topic 73 | relevant_because = topic.relevant_because 74 | 75 | query = ( 76 | f"Given the topic: {topic.name} read through the following and take notes on the topic. " 77 | "The notes will identify facts about the topic. The facts and information about the topic " 78 | "should consider how it is revelant to the overall research. The overall research identified " 79 | f"that this topic was revelant because {relevant_because}. " 80 | "The notes need to be short and concise. Consider how it relates to the original user query: " 81 | f"{user_query}" 82 | "Consider how information may dispute claims in the user query. Consider conflicting information. " 83 | "Consider supporting information\n" 84 | f"### Contents to read through: {contents} ### end contents to read through" 85 | ) 86 | result = do_chat_and_log_it(self._llm, [{"role": "system", "content": query}]) 87 | return result 88 | 89 | def extract_subtopics(self, contents: str, user_query: str) -> list[dict]: 90 | topic = self._topic 91 | relevant_because = topic.relevant_because 92 | 93 | query = ( 94 | f"Given the topic: {topic.name} read through the following and extract subtopics. " 95 | "Subtopics are topics that are more specific than the current topic. " 96 | "The subtopics should be related to the current topic and the overall research. " 97 | "The overall research identified " 98 | f"that this topic was revelant because {relevant_because}. " 99 | "The subtopics need to be short and concise. Consider how it relates to the original user query: " 100 | f"{user_query}." 101 | "\n\n The output format MUST be a JSON list of objects." 102 | """Provide a JSON list of these topics, each with a 'name', 'description', 'relevant_because', and 'notes_file' field. 103 | No more than 5 major topics and prefer not creating any subtopics at all. The revelant_because field should be one or two sentences. 104 | When creating topics, avoid specific permutations, subtopics, or detailed breakdowns. Each topic name should be self-explanatory 105 | and understandable without additional context. If uncertain about a topic, leave the 'description' field empty. 106 | For example, instead of 'Environmental Impact', use 'Fluoride's Environmental Consequences'.""" 107 | "The 'name', 'description', 'relevant_because', and 'notes_file' fields are all required! " 108 | "The notes_file value should be a file name, no spaces, with a .json extension." 109 | f"### Contents to read through: {contents} ### end contents to read through\n" 110 | "If you cannot determine subtopics, return an empty json list" 111 | "\nJSON: " 112 | ) 113 | result = do_chat_and_log_it(self._llm, [{"role": "system", "content": query}]) 114 | try: 115 | return json.loads(result) 116 | except Exception: # FIXME: too broad 117 | logging.exception(f"Could not parse {result} to JSON") 118 | return [] 119 | 120 | def maybe_add_subtopics(self, subtopics: list[dict]) -> None: 121 | if subtopics: 122 | # TEMPORARY SAFETY NET 123 | tm = TopicsManager(self._file_manager) 124 | topics = tm.load_topics() 125 | if len(topics) >= MAX_TOPICS: 126 | logging.warning(f"No longer adding topics beyond {MAX_TOPICS}") 127 | return 128 | 129 | existing_notes_files = {topic.notes_file for topic in topics} 130 | for subtopic in subtopics: 131 | try: 132 | topic = Topic(**subtopic) 133 | topics.append(topic) 134 | except Exception: # FIXME: too broad 135 | logging.exception(f"Could not create subtopic {subtopic}") 136 | topics = self.dedupe_topics(topics) 137 | if len(topics) >= MAX_TOPICS: 138 | topics = topics[:MAX_TOPICS] # ensure no more than MAX_TOPICS 139 | for topic in topics: 140 | if topic.notes_file not in existing_notes_files: 141 | topic.initialize_notes_file(self._file_manager) 142 | tm.write_topics(topics) 143 | 144 | def dedupe_topics(self, topics: list[Topic]) -> list[Topic]: 145 | query = f""" 146 | Given the following topics, remove any duplicates. Duplicates are defined as topics with 147 | similar names and or descriptions to where they are not appreciably different. 148 | 149 | ### The topics are: 150 | 151 | {" # next topic # ".join([str(asdict(topic)) for topic in topics])} 152 | 153 | ### Return a JSON list of the topics after deduping. NEVER CREATE NEW TOPICS, ONLY REMOVE THEM. 154 | 155 | JSON: 156 | """ 157 | results = self._llm.chat_completion([{"role": "system", "content": query}]) 158 | try: 159 | topics = [Topic(**topic) for topic in json.loads(results["content"])] 160 | except Exception: # FIXME: too broad 161 | logging.exception(f"Could not parse response for deduping topics in {results}") 162 | return topics 163 | 164 | def update_notes_file(self, url: str, notes: str) -> None: 165 | notes_file = self._topic.notes_file 166 | incoming_notes = self._file_manager.read_file(notes_file) 167 | notes_file_contents: list[dict] = json.loads(incoming_notes) 168 | notes_file_contents.append({"url": url, "notes": notes}) 169 | self._file_manager.write_file(notes_file, json.dumps(notes_file_contents)) 170 | 171 | def evaluate_next_steps(self) -> None: 172 | pass # not sure if anything to do here 173 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2023 James Hutchison 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | start_super_agi: 2 | cd /workspaces/SuperAGI && docker compose up --build 3 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # research-agi-agent 2 | Hackathon project - An AGI agent for doing research, understanding, and validation 3 | 4 | # Input 5 | The user will provide a claim and known supporting information that they wish to research. 6 | 7 | # Output 8 | The AGI agent will generate a research document summarizing the claim, the ramifications, related facts. The agent will generate a conclusion that may include areas of further research that are necessary to help substantiate the claim, or the conclusion may point to concrete evidence that refutes the claim. 9 | 10 | # Current usage 11 | - Create a new agent 12 | - Instruct the agent to do deep research using the deep research tools 13 | - Add the Deep Research Toolkit to the agent 14 | - Agent is goal based, note that with this if something goes wrong the agent may flail and should be paused 15 | - Start agent 16 | - Agent should select DeepResearchTool 17 | - Agent should select DeepResearchInvestigatorTool 18 | - This involves several steps 19 | - A web search for the top 3 hits 20 | - Visiting the top 3 hits and scraping them 21 | - Taking the saved info and summarizing them and writing them to notes files 22 | - Agent should iterate on topics 23 | - Agent should select DeepResearchWriterTool 24 | - Writer tool writes the file output.md 25 | - Agent should finish 26 | 27 | # Observed issues 28 | - If something goes wrong, like the OpenAI having a random failure, there may be an error such as a JSON parsing one and the agent should be stopped 29 | - If the agent can't find something useful to take notes about, such as when the scraping fails, it will keep retrying and should be stopped 30 | 31 | # Algorithm (Old draft - for reference) 32 | This algorithm was the starting point and serves as a reference for where things could go in the future 33 | 34 | - User enters claim 35 | - User describes expert that would investigate the claim 36 | - User adds information they feel is relevant 37 | - AI will have multiple roles: 38 | - Initiator - AI acting on behalf of the user that elaborates on the user's claim and information with its own information that may support the claim. The initiator does not attempt to refute its own claim. The initiator is in charge of deciding whether they are satisified with the research or if there are unknowns worth further investigating. 39 | - Expert - Tries to find holes in the claim and scrutinizes it. It also attempts to provide facts that could support the claim and refute it. 40 | - Researcher - Takes information from the initiator and expert and checks facts and researches topics and concepts to help richen the knowledge in the context which the actors are working 41 | - Mathematician and Programmer - If needed, determines whether there are relevant mathematical models which could be applied that would help refute or support the claim, and then creates code to run the model against known facts. 42 | - Summarizer - After the initiator is satisified, aggregates the exchange between the roles and produces a document. Bonus points if it can generate visualizations. 43 | 44 | # Quickstart 45 | This repo has a dev container configuration. Simply create a new Codespace in GitHub. 46 | 47 | # SuperAGI tooling breakdown 48 | Here are the components and capabilities we see necessary for a production quality agent 49 | 50 | ## Capabilities 51 | - Internet Search - The agent needs to be able to input aquery and derive one or more web URLs it believes is worth visiting 52 | - Internet Scraping - The agent needs to be able to load a web page and read the contents. It needs to support web pages that require javascript 53 | - Internal memory requirements: 54 | - The agent needs to be able to read a scraped web page and take relevant notes like a human would. The agent needs to be able to store these notes somewhere for later review 55 | - Having a vector database does not seem to be a requirement as the agent is to be thorough and will employ a dividge and conquer strategy for doing research. Notes are not expected to be so dense it can't fit in the context. 56 | - The agent likely needs a way to get a high level view of what it has done, what it has left to do, and what it should do next 57 | - Output requirements: 58 | - The agent needs to be able to generate a potentially large markdown file or set of files and provide them to the user 59 | - The agent may be able to generate images and would need to be able to reference them in the markdown files 60 | - The agent may be able to generate code and would need to be able to display them 61 | - Coherency requirements: 62 | - The agent cannot get stuck in a loop 63 | - The agent should progressively solve tasks and avoid falling down rabbit holes 64 | - The agent should clearly recognize when it is done researching 65 | - The agent should not get side-tracked and start solving irrelevant tasks 66 | 67 | ## Components 68 | - Internet search - this is provided by SuperAGI 69 | - Scholarly article search - out of scope for this hackathon 70 | - Internet scraping - This is provided by SuperAGI 71 | - Deep Research Toolkit - We will create this 72 | - Deep Research Tool - Will initialize the process and help the AI coordinate its thoughts and notes 73 | - Research Article Generation Tool - Will generate the markdown document containing the findings and results 74 | - Other bridge tools - We will need to evaluate how, for example, we would get the AI to read a scraped article in the context of this research. 75 | 76 | # How to run 77 | - Start a codespace 78 | - Populate the .env file with: 79 | - `GOOGLE_API_KEY` 80 | - `GOOGLE_CSE_ID` 81 | - Once it finishes the postStartCommand: 82 | - Run the "Run backend and Celery" run configuration 83 | - Run the task (Terminal menu -> Run Task) "Start Super AGI Docker containers" 84 | - Go to the "Ports" tab and change 3000 to a public port, then click on the planet icon when you hover your mouse over the "Local Address" cell for it 85 | 86 | # Shortcut to Summary Debugging 87 | - To avoid rerunning the agent when developing the summary generation, there is a script summary_debug.py in the root directory 88 | - Populate the .env file with: 89 | - `OPENAI_API_KEY` 90 | - The run configuration "Run summary debug" will run this 91 | - The logic currently mirrors the `DeepResearchWriterTool` 92 | -------------------------------------------------------------------------------- /alembic.ini: -------------------------------------------------------------------------------- 1 | # A generic, single database configuration. 2 | 3 | [alembic] 4 | # path to migration scripts 5 | script_location = /workspaces/SuperAGI/migrations 6 | 7 | # template used to generate migration file names; The default value is %%(rev)s_%%(slug)s 8 | # Uncomment the line below if you want the files to be prepended with date and time 9 | # see https://alembic.sqlalchemy.org/en/latest/tutorial.html#editing-the-ini-file 10 | # for all available tokens 11 | # file_template = %%(year)d_%%(month).2d_%%(day).2d_%%(hour).2d%%(minute).2d-%%(rev)s_%%(slug)s 12 | 13 | # sys.path path, will be prepended to sys.path if present. 14 | # defaults to the current working directory. 15 | prepend_sys_path = . 16 | 17 | # timezone to use when rendering the date within the migration file 18 | # as well as the filename. 19 | # If specified, requires the python-dateutil library that can be 20 | # installed by adding `alembic[tz]` to the pip requirements 21 | # string value is passed to dateutil.tz.gettz() 22 | # leave blank for localtime 23 | # timezone = 24 | 25 | # max length of characters to apply to the 26 | # "slug" field 27 | # truncate_slug_length = 40 28 | 29 | # set to 'true' to run the environment during 30 | # the 'revision' command, regardless of autogenerate 31 | # revision_environment = false 32 | 33 | # set to 'true' to allow .pyc and .pyo files without 34 | # a source .py file to be detected as revisions in the 35 | # versions/ directory 36 | # sourceless = false 37 | 38 | # version location specification; This defaults 39 | # to migrations/versions. When using multiple version 40 | # directories, initial revisions must be specified with --version-path. 41 | # The path separator used here should be the separator specified by "version_path_separator" below. 42 | # version_locations = %(here)s/bar:%(here)s/bat:migrations/versions 43 | 44 | # version path separator; As mentioned above, this is the character used to split 45 | # version_locations. The default within new alembic.ini files is "os", which uses os.pathsep. 46 | # If this key is omitted entirely, it falls back to the legacy behavior of splitting on spaces and/or commas. 47 | # Valid values for version_path_separator are: 48 | # 49 | # version_path_separator = : 50 | # version_path_separator = ; 51 | # version_path_separator = space 52 | version_path_separator = os # Use os.pathsep. Default configuration used for new projects. 53 | 54 | # set to 'true' to search source files recursively 55 | # in each "version_locations" directory 56 | # new in Alembic version 1.10 57 | # recursive_version_locations = false 58 | 59 | # the output encoding used when revision files 60 | # are written from script.py.mako 61 | # output_encoding = utf-8 62 | 63 | sqlalchemy.url = postgresql://superagi:password@localhost:5432/super_agi_main 64 | 65 | [post_write_hooks] 66 | # post_write_hooks defines scripts or Python functions that are run 67 | # on newly generated revision scripts. See the documentation for further 68 | # detail and examples 69 | 70 | # format using "black" - use the console_scripts runner, against the "black" entrypoint 71 | # hooks = black 72 | # black.type = console_scripts 73 | # black.entrypoint = black 74 | # black.options = -l 79 REVISION_SCRIPT_FILENAME 75 | 76 | # Logging configuration 77 | [loggers] 78 | keys = root,sqlalchemy,alembic 79 | 80 | [handlers] 81 | keys = console 82 | 83 | [formatters] 84 | keys = generic 85 | 86 | [logger_root] 87 | level = WARN 88 | handlers = console 89 | qualname = 90 | 91 | [logger_sqlalchemy] 92 | level = WARN 93 | handlers = 94 | qualname = sqlalchemy.engine 95 | 96 | [logger_alembic] 97 | level = INFO 98 | handlers = 99 | qualname = alembic 100 | 101 | [handler_console] 102 | class = StreamHandler 103 | args = (sys.stderr,) 104 | level = NOTSET 105 | formatter = generic 106 | 107 | [formatter_generic] 108 | format = %(levelname)-5.5s [%(name)s] %(message)s 109 | datefmt = %H:%M:%S 110 | -------------------------------------------------------------------------------- /decision_log.md: -------------------------------------------------------------------------------- 1 | 2 | ## Technologies Evaluated 3 | # LangChain 4 | 5 | ## Description 6 | LangChain is a compelling multipurpose product for interacting with LLMs from various sources. It includes functionality for creating autonomous agents. It is a Python library 7 | 8 | ## Pros 9 | - Familiar with the team 10 | - Extensive documentation and examples 11 | - Extensive tooling for various use cases 12 | 13 | ## Cons 14 | - AutoGPT and BabyAGI integration is experimental 15 | - AutoGPT functionality from experimental library seems to lack the tooling that comes with the main repo. You must provide it yourself 16 | 17 | ## Conclusion 18 | It's highly likely we'll use LangChain in some capacity 19 | 20 | # AutoGPT 21 | 22 | ## Description 23 | AutoGPT takes in a starting prompt and uses a provided set of tools to accomplish the goals in the prompt. It can run from the command line or be incorporated into a larger project. 24 | 25 | ## Pros 26 | - Inherently capable of research via search functionality 27 | 28 | ## Cons 29 | - Experimental and can get stuck in a loop without no functionality to abort 30 | - The AI drives the workflow so if it needs guidance on a technique of arriving to the best answer it may not follow it. 31 | 32 | ## Conclusion 33 | We may be able to use AutoGPT for these purposes. 34 | 35 | # BabyGPT 36 | 37 | ## Description 38 | Similar to AutoGPT, it uses a vector database automatically to act as a memory. 39 | 40 | ## Pros 41 | - Simple 42 | 43 | ## Cons 44 | - Doesn't seem to support custom tooling and for that matter doesn't seem to support any tooling without modifying it ourselves. The vanilla version seems to do everything from the LLM's training. 45 | 46 | ## Conclusion 47 | We will need internet search and other custom tooling so this is not a good fit. 48 | 49 | # SuperAGI 50 | 51 | ## Description 52 | Similar to AutoGPT but a much larger framework and built to be ran from a web page. SuperAGI feels more user friendly for non-technical users and has a lot of functionality in place to make it a compelling product once the rough edges are ironed out. 53 | 54 | ## Pros 55 | - A lot out of the box, including logic, various configuration options, and a GUI 56 | - Sponsoring Hackathon 57 | 58 | ## Cons 59 | - missing documentation for use cases like creating sharable agent templates 60 | - more complex to run locally 61 | - Lack of guidance on how we would deploy a public facing demoable proof of concept 62 | - Documentation gaps 63 | - Agent workflows seem to be built at start-up for SuperAGI with no obvious guidance on how to add them later. 64 | - Could likely seed them via a script 65 | 66 | ## Conclusion 67 | Since SuperGPT is sponsoring this hackathon, we feel obligated to use this. We feel confident in being able to build a solution, but it may be more work than using LangChain + Streamlit. 68 | 69 | # Streamlit 70 | 71 | ## Description 72 | A library for creating web interfaces using Python code, tailored towards data science and machine learning. 73 | 74 | ## Pros 75 | - Free hosting 76 | - Easy to use 77 | - Plenty of examples and integrations with LangChain 78 | 79 | ## Cons 80 | - Only free hosting, so limited to quotas 81 | - Memory leaks are a risk if caching isn't leveraged properly 82 | 83 | ## Conclusion 84 | This is a solid choice, but it doesn't seem capatible with SuperAGI since they both provide front-ends. 85 | 86 | # Spikes 87 | - Streamlit + AutoGPT via LangChain experimental - streamlit-app branch 88 | - SuperAGI + custom toolkit - super-agi-toolkit branch 89 | 90 | ## Decisions 91 | - 8/17/2023 - Heavily leaning towards SuperAGI + custom toolkit. We are still awaiting guidance on how best to demo. We have spent efforts in streamlining onboading via the dev container and GitHub Codespaces. We believe that using SuperAGI would be more effort than Streamlit + AutoGPT or LangChain, however based on the sponsorship of the Hackathon and the capabilities of SuperAGI, both current and future, we feel it is worth grinding through the challenge as it will provide the most value. This is not just an opportunity to build a research agent, its also an opportunity to become familiar with the capabilities of SuperAGI which may pay dividends in the future. 92 | -------------------------------------------------------------------------------- /docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: '3.8' 2 | services: 3 | gui: 4 | build: 5 | context: /workspaces/SuperAGI/gui 6 | args: 7 | NEXT_PUBLIC_API_BASE_URL: "/api" 8 | networks: 9 | - super_network 10 | # volumes: 11 | # - ./gui:/app 12 | # - /app/node_modules/ 13 | # - /app/.next/ 14 | super__redis: 15 | image: "redis/redis-stack-server:latest" 16 | networks: 17 | - super_network 18 | # uncomment to expose redis port to host 19 | ports: 20 | - "6379:6379" 21 | volumes: 22 | - redis_data:/data 23 | 24 | super__postgres: 25 | image: "docker.io/library/postgres:latest" 26 | environment: 27 | - POSTGRES_USER=superagi 28 | - POSTGRES_PASSWORD=password 29 | - POSTGRES_DB=super_agi_main 30 | volumes: 31 | - superagi_postgres_data:/var/lib/postgresql/data/ 32 | networks: 33 | - super_network 34 | # uncomment to expose postgres port to host 35 | ports: 36 | - "5432:5432" 37 | 38 | proxy: 39 | image: nginx:stable-alpine 40 | ports: 41 | - "3000:80" 42 | networks: 43 | - super_network 44 | depends_on: 45 | - gui 46 | volumes: 47 | - ./nginx/default.conf:/etc/nginx/conf.d/default.conf 48 | 49 | networks: 50 | super_network: 51 | driver: bridge 52 | volumes: 53 | superagi_postgres_data: 54 | redis_data: 55 | -------------------------------------------------------------------------------- /example_output.md: -------------------------------------------------------------------------------- 1 | # Health Risks of Fluoridating Water 2 | 3 | Fluoride is a naturally occurring compound that combines the element fluorine with another substance, usually a metal[^1^]. It is found in soil, air, water, and in some plants and animals[^1^]. Fluoride is absorbed into the blood through the digestive tract and tends to collect in areas high in calcium, such as bones and teeth[^1^]. The major sources of fluoride exposure for most people are water, other beverages, food, and fluoride-containing dental products[^1^]. 4 | 5 | Water fluoridation, the process of adding fluoride to public water supplies, began in the United States in 1945 after scientists noted that people living in areas with higher water fluoride levels had fewer cavities[^1^]. However, there is ongoing controversy about the possible health effects of fluoride, with concerns ranging from scientific research to freedom of choice issues[^1^]. 6 | 7 | ## Conclusion 8 | 9 | While fluoride has been recognized for its role in preventing tooth decay, excessive exposure to fluoride can lead to health issues such as dental fluorosis, skeletal fluorosis, thyroid problems, and neurological problems[^3^]. The link between water fluoridation and cancer is still under review[^1^]. Therefore, it is crucial to maintain a balance between the benefits and potential health risks of fluoride. The U.S. Public Health Service (PHS) recommends a fluoride concentration of 0.7 milligrams/liter (mg/L) in drinking water for optimal dental caries prevention[^4^]. 10 | 11 | ## How the Conclusion was Derived 12 | 13 | The conclusion was derived from a comprehensive review of the potential health risks associated with fluoride exposure and the process, benefits, and risks of adding fluoride to public water supplies. 14 | 15 | ### Health Risks of Fluoride 16 | 17 | Fluoride is one of the most abundant elements in nature and is found in water, which is its major dietary source[^2^]. Low fluoride intake is associated with the risk of dental caries[^2^]. However, there are concerns about excessive fluoride intake and related toxicity, leading several countries to ban fluoridation[^2^]. 18 | 19 | Excessive exposure to fluoride can lead to health issues such as dental fluorosis, which can cause discoloration in teeth, and skeletal fluorosis, a bone disease that can result in pain, damage to bones and joints, and increased risk of fractures[^3^]. Excess fluoride can also damage the parathyroid gland, leading to hyperparathyroidism and resulting in lower calcium concentrations in bones, making them more susceptible to fractures[^3^]. A 2017 report suggested that exposure to fluoride before birth could lead to poorer cognitive outcomes in the future, with higher fluoride levels associated with lower scores on IQ tests[^3^]. In 2014, fluoride was documented as a neurotoxin that could be hazardous to child development[^3^]. 20 | 21 | ### Fluoridation of Water 22 | 23 | The decision to fluoridate water systems is made by state and local governments[^4^]. As of 2012, approximately 200 million people in the U.S. were served by community water systems that added fluoride to water[^4^]. The new PHS recommendation of 0.7 mg/L will reduce fluoride concentration in water systems by 0.1–0.5 mg/L[^4^]. Implementation of the new recommendation is expected to lead to a reduction of approximately 25% in fluoride intake from drinking water alone and a reduction of approximately 14% in total fluoride intake[^4^]. 24 | 25 | [^1^]: https://www.cancer.org/cancer/risk-prevention/chemicals/water-fluoridation-and-cancer-risk.html 26 | [^2^]: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6195894/ 27 | [^3^]: https://www.medicalnewstoday.com/articles/154164 28 | [^4^]: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4547570/ 29 | -------------------------------------------------------------------------------- /nginx/default.conf: -------------------------------------------------------------------------------- 1 | server { 2 | listen 80; 3 | 4 | location / { 5 | proxy_pass http://gui:3000; 6 | proxy_set_header Host $host; 7 | proxy_set_header X-Real-IP $remote_addr; 8 | proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; 9 | } 10 | 11 | location /api { 12 | proxy_pass http://172.17.0.1:8001; 13 | client_max_body_size 50M; 14 | proxy_set_header Host $host; 15 | proxy_set_header X-Real-IP $remote_addr; 16 | proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; 17 | rewrite ^/api/(.*) /$1 break; 18 | } 19 | location /_next/webpack-hmr { 20 | proxy_pass http://gui:3000; 21 | proxy_http_version 1.1; 22 | proxy_set_header Upgrade $http_upgrade; 23 | proxy_set_header Connection "upgrade"; 24 | proxy_set_header Host $host; 25 | proxy_cache_bypass $http_upgrade; 26 | } 27 | } 28 | -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | [tool.ruff] 2 | # hard limit, error here 3 | line-length = 120 4 | 5 | [tool.black] 6 | # soft limit, wrap just before 100 chars 7 | line-length = 98 8 | 9 | [tool.poetry] 10 | name = "research-agi-agent" 11 | version = "0.1.0" 12 | description = "" 13 | authors = ["James Hutchison ", "Muhammad Ibrahim Laeeq "] 14 | readme = "README.md" 15 | packages = [{ include = "research_agi_agent" }] 16 | 17 | [tool.poetry.dependencies] 18 | python = "^3.11" 19 | superagi-tools = "^1.0.7" 20 | langchain = "^0.0.268" 21 | 22 | 23 | [tool.poetry.group.dev.dependencies] 24 | mypy = "^1.5.0" 25 | ruff = "^0.0.284" 26 | black = "^23.7.0" 27 | pytest = "^7.4.0" 28 | 29 | [build-system] 30 | requires = ["poetry-core"] 31 | build-backend = "poetry.core.masonry.api" 32 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | pydantic==1.10.8 ; python_version >= "3.11" and python_version < "4.0" \ 2 | --hash=sha256:052d8654cb65174d6f9490cc9b9a200083a82cf5c3c5d3985db765757eb3b375 \ 3 | --hash=sha256:0c6fafa0965b539d7aab0a673a046466d23b86e4b0e8019d25fd53f4df62c277 \ 4 | --hash=sha256:1243d28e9b05003a89d72e7915fdb26ffd1d39bdd39b00b7dbe4afae4b557f9d \ 5 | --hash=sha256:12f7b0bf8553e310e530e9f3a2f5734c68699f42218bf3568ef49cd9b0e44df4 \ 6 | --hash=sha256:1410275520dfa70effadf4c21811d755e7ef9bb1f1d077a21958153a92c8d9ca \ 7 | --hash=sha256:16f8c3e33af1e9bb16c7a91fc7d5fa9fe27298e9f299cff6cb744d89d573d62c \ 8 | --hash=sha256:17aef11cc1b997f9d574b91909fed40761e13fac438d72b81f902226a69dac01 \ 9 | --hash=sha256:191ba419b605f897ede9892f6c56fb182f40a15d309ef0142212200a10af4c18 \ 10 | --hash=sha256:1952526ba40b220b912cdc43c1c32bcf4a58e3f192fa313ee665916b26befb68 \ 11 | --hash=sha256:1ced8375969673929809d7f36ad322934c35de4af3b5e5b09ec967c21f9f7887 \ 12 | --hash=sha256:2e4148e635994d57d834be1182a44bdb07dd867fa3c2d1b37002000646cc5459 \ 13 | --hash=sha256:34d327c81e68a1ecb52fe9c8d50c8a9b3e90d3c8ad991bfc8f953fb477d42fb4 \ 14 | --hash=sha256:35db5301b82e8661fa9c505c800d0990bc14e9f36f98932bb1d248c0ac5cada5 \ 15 | --hash=sha256:3e59417ba8a17265e632af99cc5f35ec309de5980c440c255ab1ca3ae96a3e0e \ 16 | --hash=sha256:42aa0c4b5c3025483240a25b09f3c09a189481ddda2ea3a831a9d25f444e03c1 \ 17 | --hash=sha256:666bdf6066bf6dbc107b30d034615d2627e2121506c555f73f90b54a463d1f33 \ 18 | --hash=sha256:66a703d1983c675a6e0fed8953b0971c44dba48a929a2000a493c3772eb61a5a \ 19 | --hash=sha256:6a82d6cda82258efca32b40040228ecf43a548671cb174a1e81477195ed3ed56 \ 20 | --hash=sha256:6f2e754d5566f050954727c77f094e01793bcb5725b663bf628fa6743a5a9108 \ 21 | --hash=sha256:7456eb22ed9aaa24ff3e7b4757da20d9e5ce2a81018c1b3ebd81a0b88a18f3b2 \ 22 | --hash=sha256:7b1f6cb446470b7ddf86c2e57cd119a24959af2b01e552f60705910663af09a4 \ 23 | --hash=sha256:7d5b8641c24886d764a74ec541d2fc2c7fb19f6da2a4001e6d580ba4a38f7878 \ 24 | --hash=sha256:84d80219c3f8d4cad44575e18404099c76851bc924ce5ab1c4c8bb5e2a2227d0 \ 25 | --hash=sha256:88f195f582851e8db960b4a94c3e3ad25692c1c1539e2552f3df7a9e972ef60e \ 26 | --hash=sha256:93e6bcfccbd831894a6a434b0aeb1947f9e70b7468f274154d03d71fabb1d7c6 \ 27 | --hash=sha256:93e766b4a8226e0708ef243e843105bf124e21331694367f95f4e3b4a92bbb3f \ 28 | --hash=sha256:ab523c31e22943713d80d8d342d23b6f6ac4b792a1e54064a8d0cf78fd64e800 \ 29 | --hash=sha256:bb14388ec45a7a0dc429e87def6396f9e73c8c77818c927b6a60706603d5f2ea \ 30 | --hash=sha256:c0ab53b609c11dfc0c060d94335993cc2b95b2150e25583bec37a49b2d6c6c3f \ 31 | --hash=sha256:c33b60054b2136aef8cf190cd4c52a3daa20b2263917c49adad20eaf381e823b \ 32 | --hash=sha256:ceb6a23bf1ba4b837d0cfe378329ad3f351b5897c8d4914ce95b85fba96da5a1 \ 33 | --hash=sha256:d532bf00f381bd6bc62cabc7d1372096b75a33bc197a312b03f5838b4fb84edd \ 34 | --hash=sha256:df7800cb1984d8f6e249351139667a8c50a379009271ee6236138a22a0c0f319 \ 35 | --hash=sha256:e82d4566fcd527eae8b244fa952d99f2ca3172b7e97add0b43e2d97ee77f81ab \ 36 | --hash=sha256:f90c1e29f447557e9e26afb1c4dbf8768a10cc676e3781b6a577841ade126b85 \ 37 | --hash=sha256:f9613fadad06b4f3bc5db2653ce2f22e0de84a7c6c293909b48f6ed37b83c61f 38 | pyyaml==6.0 ; python_version >= "3.11" and python_version < "4.0" \ 39 | --hash=sha256:01b45c0191e6d66c470b6cf1b9531a771a83c1c4208272ead47a3ae4f2f603bf \ 40 | --hash=sha256:0283c35a6a9fbf047493e3a0ce8d79ef5030852c51e9d911a27badfde0605293 \ 41 | --hash=sha256:055d937d65826939cb044fc8c9b08889e8c743fdc6a32b33e2390f66013e449b \ 42 | --hash=sha256:07751360502caac1c067a8132d150cf3d61339af5691fe9e87803040dbc5db57 \ 43 | --hash=sha256:0b4624f379dab24d3725ffde76559cff63d9ec94e1736b556dacdfebe5ab6d4b \ 44 | --hash=sha256:0ce82d761c532fe4ec3f87fc45688bdd3a4c1dc5e0b4a19814b9009a29baefd4 \ 45 | --hash=sha256:1e4747bc279b4f613a09eb64bba2ba602d8a6664c6ce6396a4d0cd413a50ce07 \ 46 | --hash=sha256:213c60cd50106436cc818accf5baa1aba61c0189ff610f64f4a3e8c6726218ba \ 47 | --hash=sha256:231710d57adfd809ef5d34183b8ed1eeae3f76459c18fb4a0b373ad56bedcdd9 \ 48 | --hash=sha256:277a0ef2981ca40581a47093e9e2d13b3f1fbbeffae064c1d21bfceba2030287 \ 49 | --hash=sha256:2cd5df3de48857ed0544b34e2d40e9fac445930039f3cfe4bcc592a1f836d513 \ 50 | --hash=sha256:40527857252b61eacd1d9af500c3337ba8deb8fc298940291486c465c8b46ec0 \ 51 | --hash=sha256:432557aa2c09802be39460360ddffd48156e30721f5e8d917f01d31694216782 \ 52 | --hash=sha256:473f9edb243cb1935ab5a084eb238d842fb8f404ed2193a915d1784b5a6b5fc0 \ 53 | --hash=sha256:48c346915c114f5fdb3ead70312bd042a953a8ce5c7106d5bfb1a5254e47da92 \ 54 | --hash=sha256:50602afada6d6cbfad699b0c7bb50d5ccffa7e46a3d738092afddc1f9758427f \ 55 | --hash=sha256:68fb519c14306fec9720a2a5b45bc9f0c8d1b9c72adf45c37baedfcd949c35a2 \ 56 | --hash=sha256:77f396e6ef4c73fdc33a9157446466f1cff553d979bd00ecb64385760c6babdc \ 57 | --hash=sha256:81957921f441d50af23654aa6c5e5eaf9b06aba7f0a19c18a538dc7ef291c5a1 \ 58 | --hash=sha256:819b3830a1543db06c4d4b865e70ded25be52a2e0631ccd2f6a47a2822f2fd7c \ 59 | --hash=sha256:897b80890765f037df3403d22bab41627ca8811ae55e9a722fd0392850ec4d86 \ 60 | --hash=sha256:98c4d36e99714e55cfbaaee6dd5badbc9a1ec339ebfc3b1f52e293aee6bb71a4 \ 61 | --hash=sha256:9df7ed3b3d2e0ecfe09e14741b857df43adb5a3ddadc919a2d94fbdf78fea53c \ 62 | --hash=sha256:9fa600030013c4de8165339db93d182b9431076eb98eb40ee068700c9c813e34 \ 63 | --hash=sha256:a80a78046a72361de73f8f395f1f1e49f956c6be882eed58505a15f3e430962b \ 64 | --hash=sha256:afa17f5bc4d1b10afd4466fd3a44dc0e245382deca5b3c353d8b757f9e3ecb8d \ 65 | --hash=sha256:b3d267842bf12586ba6c734f89d1f5b871df0273157918b0ccefa29deb05c21c \ 66 | --hash=sha256:b5b9eccad747aabaaffbc6064800670f0c297e52c12754eb1d976c57e4f74dcb \ 67 | --hash=sha256:bfaef573a63ba8923503d27530362590ff4f576c626d86a9fed95822a8255fd7 \ 68 | --hash=sha256:c5687b8d43cf58545ade1fe3e055f70eac7a5a1a0bf42824308d868289a95737 \ 69 | --hash=sha256:cba8c411ef271aa037d7357a2bc8f9ee8b58b9965831d9e51baf703280dc73d3 \ 70 | --hash=sha256:d15a181d1ecd0d4270dc32edb46f7cb7733c7c508857278d3d378d14d606db2d \ 71 | --hash=sha256:d4b0ba9512519522b118090257be113b9468d804b19d63c71dbcf4a48fa32358 \ 72 | --hash=sha256:d4db7c7aef085872ef65a8fd7d6d09a14ae91f691dec3e87ee5ee0539d516f53 \ 73 | --hash=sha256:d4eccecf9adf6fbcc6861a38015c2a64f38b9d94838ac1810a9023a0609e1b78 \ 74 | --hash=sha256:d67d839ede4ed1b28a4e8909735fc992a923cdb84e618544973d7dfc71540803 \ 75 | --hash=sha256:daf496c58a8c52083df09b80c860005194014c3698698d1a57cbcfa182142a3a \ 76 | --hash=sha256:dbad0e9d368bb989f4515da330b88a057617d16b6a8245084f1b05400f24609f \ 77 | --hash=sha256:e61ceaab6f49fb8bdfaa0f92c4b57bcfbea54c09277b1b4f7ac376bfb7a7c174 \ 78 | --hash=sha256:f84fbc98b019fef2ee9a1cb3ce93e3187a6df0b2538a651bfb890254ba9f90b5 79 | superagi-tools==1.0.7 ; python_version >= "3.11" and python_version < "4.0" \ 80 | --hash=sha256:210b59f781030ab4a2e9b9baed5a572be31e17bc5688a191f86eab78e0b59e3f \ 81 | --hash=sha256:e5b3da84bbb1f6fe3f08805c99ce4c5b28d13ec97456c730442ed0809e6a286a 82 | typing-extensions==4.7.1 ; python_version >= "3.11" and python_version < "4.0" \ 83 | --hash=sha256:440d5dd3af93b060174bf433bccd69b0babc3b15b1a8dca43789fd7f61514b36 \ 84 | --hash=sha256:b75ddc264f0ba5615db7ba217daeb99701ad295353c45f9e95963337ceeeffb2 85 | -------------------------------------------------------------------------------- /sample_output/FluoridationOfWater.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "url": "https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4547570/", 4 | "notes": "1. The U.S. Public Health Service (PHS) recommends a fluoride concentration of 0.7 milligrams/liter (mg/L) in drinking water for optimal dental caries prevention.\n2. The fluoride concentration recommendation is a balance between protection from dental caries and limiting the risk of dental fluorosis.\n3. The decision to fluoridate water systems is made by state and local governments.\n4. As of 2012, approximately 200 million people in the U.S. were served by community water systems that added fluoride to water.\n5. The new PHS recommendation of 0.7 mg/L will reduce fluoride concentration in water systems by 0.1\u20130.5 mg/L.\n6. Implementation of the new recommendation is expected to lead to a reduction of approximately 25% in fluoride intake from drinking water alone and a reduction of approximately 14% in total fluoride intake.\n7. The updated PHS recommendation is based on new data addressing changes in the prevalence of dental fluorosis, the relationship between water intake and outdoor temperature in children, and the contribution of fluoride in drinking water to total fluoride exposure in the U.S." 5 | }, 6 | { 7 | "url": "https://www.cancer.org/cancer/risk-prevention/chemicals/water-fluoridation-and-cancer-risk.html", 8 | "notes": "1. Fluoride is a compound that combines the element fluorine with another substance, usually a metal. It is found naturally in soil, air, or water, and in some plants and animals.\n2. Fluoride is absorbed into the blood through the digestive tract and tends to collect in areas high in calcium, such as bones and teeth.\n3. Major sources of fluoride exposure are water, other beverages, food, and fluoride-containing dental products.\n4. Water fluoridation began in parts of the United States in 1945 after scientists noted that people living in areas with higher water fluoride levels had fewer cavities.\n5. In 1962, the United States Public Health Service recommended that public water supplies contain fluoride to help prevent tooth decay.\n6. There is controversy about the possible health effects of water fluoridation, with concerns ranging from scientific research to freedom of choice issues.\n7. The link between water fluoridation and cancer is a topic of investigation, but this document does not provide a definitive answer.\n8. Dental products containing fluoride are less likely to cause health issues as they are generally not swallowed." 9 | }, 10 | { 11 | "url": "https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6195894/", 12 | "notes": "1. Fluoride is one of the most abundant elements found in nature, and water is the major dietary source of fluoride.\n2. Low fluoride intake is associated with the risk of dental caries.\n3. Fluoride was initially considered beneficial when given systemically during tooth development, but later research showed its topical effects in the prevention or treatment of dental caries and tooth decay.\n4. Water fluoridation was once considered one of the best public health achievements in the twentieth century.\n5. Fluoridation is not feasible or cost effective in many regions, especially rural areas, leading to alternative methods such as adding fluoride to milk and table salt.\n6. There are major concerns about excessive fluoride intake and related toxicity, leading several countries to ban fluoridation.\n7. Fluoride is the ionic form of fluorine, the thirteenth most abundant element in the earth\u2019s crust, and is released into the environment naturally in both water and air.\n8. Other important sources of fluoride are tea, seafood that contains edible bones or shells, medicinal supplements, and fluoridated toothpastes.\n9. Dietary fluoride is absorbed rapidly in the stomach and small intestine, and about 99% of total body fluoride is contained in bones and teeth.\n10. The recommended intake for fluoride is 0.7 mg daily for toddlers, rising to 3 mg daily for adult women and 4 mg daily for adult men.\n11. Once taken up into bone, fluoride appears to increase osteoblast activity and bone density." 13 | } 14 | ] 15 | -------------------------------------------------------------------------------- /sample_output/HealthRisksOfFluoride.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "url": "https://www.cancer.org/cancer/risk-prevention/chemicals/water-fluoridation-and-cancer-risk.html", 4 | "notes": "- Fluorides are compounds that combine the element fluorine with another substance, usually a metal. They occur naturally in soil, air, or water, and also in food sources from some plants and animals.\n- Once inside the body, fluorides are absorbed into the blood through the digestive tract and tend to collect in areas high in calcium, such as the bones and teeth.\n- The major sources of fluoride for most people are water and other beverages, food, and fluoride-containing dental products.\n- Water fluoridation began in the United States in 1945, after scientists noted that people living in areas with higher water fluoride levels had fewer cavities.\n- In 1962, the United States Public Health Service recommended that public water supplies contain fluoride to help prevent tooth decay.\n- There is ongoing controversy about the possible health effects of fluoride, with concerns ranging from scientific research to freedom of choice issues to government conspiracy theories.\n- The link between water fluoridation and cancer is still under review, and other possible health effects of fluoridation (positive or negative) are not addressed in the provided content." 5 | }, 6 | { 7 | "url": "https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6195894/", 8 | "notes": "1. Fluoride is one of the most abundant elements in nature and is found in water, which is its major dietary source.\n2. Low fluoride intake is associated with the risk of dental caries.\n3. Initially, fluoride was considered beneficial when given systemically during tooth development, but later research showed its topical effects in the prevention or treatment of dental caries and tooth decay.\n4. Water fluoridation was considered one of the best public health achievements in the twentieth century, but it is not feasible or cost effective in many regions.\n5. Other methods of introducing fluoride include adding it to milk and table salt.\n6. There are concerns about excessive fluoride intake and related toxicity, leading several countries to ban fluoridation.\n7. Fluoride is absorbed rapidly in the stomach and small intestine, with one-quarter to one-third taken up into calcified tissues, and the rest is lost in the urine.\n8. About 99% of total body fluoride is contained in bones and teeth, and the amount steadily increases during life.\n9. The recommended intake for fluoride is 0.7 mg daily for toddlers, rising to 3 mg daily for adult women and 4 mg daily for adult men.\n10. It remains unclear whether fluoride is truly essential, although it may have some beneficial effects.\n11. Once taken up into bone, fluoride appears to increase osteoblast activity and bone density." 9 | }, 10 | { 11 | "url": "https://www.medicalnewstoday.com/articles/154164", 12 | "notes": "1. Fluoride is found naturally in soil, water, and foods and is synthetically produced for use in drinking water, toothpaste, and mouthwashes.\n2. Fluoride is added to the municipal water supply to reduce tooth decay, especially in areas where fluoride levels are low.\n3. Excessive exposure to fluoride can lead to health issues such as dental fluorosis, skeletal fluorosis, thyroid problems, and neurological problems.\n4. Dental fluorosis, which can cause discoloration in teeth, is a result of high fluoride exposure during childhood.\n5. Skeletal fluorosis, a bone disease caused by excess fluoride exposure, can result in pain, damage to bones and joints, and increased risk of fractures.\n6. Excess fluoride can damage the parathyroid gland, leading to hyperparathyroidism and resulting in lower calcium concentrations in bones, making them more susceptible to fractures.\n7. A 2017 report suggested that exposure to fluoride before birth could lead to poorer cognitive outcomes in the future, with higher fluoride levels associated with lower scores on IQ tests.\n8. In 2014, fluoride was documented as a neurotoxin that could be hazardous to child development." 13 | } 14 | ] 15 | -------------------------------------------------------------------------------- /sample_output/output.md: -------------------------------------------------------------------------------- 1 | # Research Topics 2 | 3 | ## Health Risks of Fluoride 4 | 5 | This topic investigates the potential health risks associated with fluoride exposure. Fluoride is a naturally occurring mineral that is often added to public water supplies to help prevent tooth decay. However, there is ongoing debate about the potential health risks associated with fluoride, particularly when it is consumed in large amounts. 6 | 7 | **Notes File:** HealthRisksOfFluoride.txt 8 | 9 | **Relevance:** The topic is directly related to the health implications of fluoride, which is a key component of the research question. 10 | 11 | **Researched:** Yes 12 | 13 | ## Fluoridation of Water 14 | 15 | This topic focuses on the study of the process, benefits, and risks of adding fluoride to public water supplies. While fluoridation has been shown to have dental health benefits, there are concerns about potential health risks, particularly for those who consume water with high levels of fluoride. 16 | 17 | **Notes File:** FluoridationOfWater.txt 18 | 19 | **Relevance:** The research question specifically mentions the fluoridation of water, making this a central topic of investigation. 20 | 21 | **Researched:** Yes -------------------------------------------------------------------------------- /sample_output/topics.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "name": "Health Risks of Fluoride", 4 | "description": "Investigation into the potential health risks associated with fluoride exposure.", 5 | "notes_file": "HealthRisksOfFluoride.json", 6 | "relevant_because": "The topic is directly related to the health implications of fluoride, which is a key component of the research question.", 7 | "researched": true 8 | }, 9 | { 10 | "name": "Fluoridation of Water", 11 | "description": "Study of the process, benefits, and risks of adding fluoride to public water supplies.", 12 | "notes_file": "FluoridationOfWater.json", 13 | "relevant_because": "The research question specifically mentions the fluoridation of water, making this a central topic of investigation.", 14 | "researched": true 15 | } 16 | ] 17 | -------------------------------------------------------------------------------- /sample_output/user_query.txt: -------------------------------------------------------------------------------- 1 | health risks of putting fluoride in water -------------------------------------------------------------------------------- /sample_output_2/FluoridationOfWater.json: -------------------------------------------------------------------------------- 1 | [{"url": "https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4547570/", "notes": "- The U.S. Public Health Service (PHS) recommends an optimal fluoride concentration of 0.7 milligrams/liter (mg/L) in drinking water for the prevention of dental caries.\n- The concentration of fluoride in drinking water is intended to balance protection from dental caries with limiting the risk of dental fluorosis.\n- The decision to fluoridate water systems is made by state and local governments.\n- As of 2012, approximately 200 million people in the U.S. were served by community water systems that added fluoride to water.\n- The new PHS recommendation of 0.7 mg/L will reduce the fluoride concentration in water systems by 0.1\u20130.5 mg/L, leading to a decline in fluoride intake among most people served by these systems.\n- Implementation of the new recommendation is expected to reduce fluoride intake from drinking water by approximately 25% and total fluoride intake by approximately 14%."}, {"url": "https://www.cancer.org/cancer/risk-prevention/chemicals/water-fluoridation-and-cancer-risk.html", "notes": "- Fluorides are compounds that combine with another substance, usually a metal, and include sodium fluoride, stannous fluoride, and fluoride monofluorophosphate.\n- Fluorides occur naturally in soil, air, or water and are also found in food sources from some plants and animals.\n- Once inside the body, fluorides are absorbed into the blood through the digestive tract and tend to collect in areas high in calcium, such as the bones and teeth.\n- Major sources of fluoride for most people are water, other beverages, food, and fluoride-containing dental products.\n- Water fluoridation began in some parts of the United States in 1945, after scientists noted that people living in areas with higher water fluoride levels had fewer cavities.\n- In 1962, the United States Public Health Service recommended that public water supplies contain fluoride to help prevent tooth decay.\n- There is controversy about the possible health effects of water fluoridation, with concerns ranging from legitimate scientific research to freedom of choice issues to government conspiracy theories.\n- The link between water fluoridation and cancer is still under review and is not addressed in this document."}, {"url": "https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6195894/", "notes": "1. Fluoride is one of the most abundant elements in nature and is found in water, which is its major dietary source.\n2. Low fluoride intake is associated with the risk of dental caries.\n3. Fluoride was initially considered beneficial when given systemically during tooth development, but later research showed its topical effects in the prevention or treatment of dental caries and tooth decay.\n4. Water fluoridation was once considered one of the best public health achievements in the twentieth century.\n5. Other methods of introducing fluoride to the population include adding it to milk and table salt.\n6. Concerns about excessive fluoride intake and related toxicity have led several countries to ban fluoridation.\n7. Fluoride is the ionic form of fluorine, and is released into the environment naturally in both water and air.\n8. Other sources of fluoride include tea, seafood with edible bones or shells, medicinal supplements, and fluoridated toothpastes.\n9. Dietary fluoride is absorbed rapidly in the stomach and small intestine, and about 99% of total body fluoride is contained in bones and teeth.\n10. The recommended intake for fluoride is 0.7 mg daily for toddlers, rising to 3 mg daily for adult women and 4 mg daily for adult men.\n11. Once taken up into bone, fluoride appears to increase osteoblast activity and bone density."}] -------------------------------------------------------------------------------- /sample_output_2/FluorideAbsorptionAndDistribution.json: -------------------------------------------------------------------------------- 1 | [{"url": "https://ods.od.nih.gov/factsheets/Fluoride-HealthProfessional/", "notes": "1. Fluoride is a mineral naturally present in many foods and is also available as a dietary supplement.\n2. It inhibits or reverses the initiation and progression of dental caries (tooth decay) and stimulates new bone formation.\n3. Most of the fluoride that people consume comes from fluoridated water, foods and beverages prepared with fluoridated water, and toothpaste and other dental products containing fluoride.\n4. Approximately 80% or more of orally ingested fluoride is absorbed in the gastrointestinal tract.\n5. In adults, about 50% of absorbed fluoride is retained, and bones and teeth store about 99% of fluoride in the body. The other 50% is excreted in urine.\n6. In young children, up to 80% of absorbed fluoride is retained because more is taken up by bones and teeth than in adults.\n7. Individual fluoride status is not typically assessed, although fluoride concentrations can be measured in plasma, saliva, urine, bones, nails, hair, and teeth.\n8. Criteria for adequate, high, or low levels of fluoride in the body have not been established.\n9. The Food and Nutrition Board (FNB) at the National Academies of Sciences, Engineering, and Medicine has developed intake recommendations for fluoride and other nutrients.\n10. Brewed tea typically contains higher levels of fluoride than most foods, depending on the type of tea and its source, because tea plants take up fluoride from soil.\n11. Fluoride concentrations in breast milk and cow\u2019s milk are very low.\n12. Fluoride levels in infant formulas in the United States vary, depending on the type of formula and the fluoride content."}, {"url": "https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5651468/", "notes": "1. Fluoride is the 13th most abundant element on earth, found in water, soil, animals, and plants.\n2. It is present in all mineralized tissues of the body like enamel, dentin, and bone and is involved in several enzymatic reactions.\n3. Fluoride ions increase the stability of mineralized tissues and materials by decreasing the solubility of hydroxy-apatite mineral phase present in biomaterials and mineralized tissues.\n4. The protective effects of fluoride on dental health were first observed in 1930 with less tooth decay in communities consuming naturally fluoridated water.\n5. Fluoride was introduced into dentistry in 1940 and is now added to various consumer products, with water fluoridation being the most successful method.\n6. Fluoride delivery methods and related sources of dietary fluoride include fluoridated water, beverages, and tea.\n7. The recommended optimal level of fluoride in drinking water is 0.7 mg/l, but this can vary based on geographical areas.\n8. Ingestion of fluoride more than the recommended limit can lead to toxicity and adverse effects.\n9. The fluoride content in drinking waters of Pakistan varies significantly, ranging from < 0.1 ppm to >3 ppm.\n10. Understanding fluoride metabolism, toxic effects, and management of fluoride toxicity is crucial for assessing the health implications of fluoride in water."}, {"url": "https://pinellas.gov/water-fluoridation-facts/", "notes": "- Fluoride is added to water systems to improve oral health by strengthening tooth enamel and reducing tooth decay.\n- The U.S. Environmental Protection Agency (EPA) has set a Maximum Contaminant Level (MCL) for fluoride in drinking water at 4.0 parts per million (ppm), and a Secondary MCL at 2 ppm. \n- The U.S. Department of Health and Human Services recommends a fluoride level of 0.7 to 1.2 ppm for dental health. \n- Pinellas County's water has a fluoride level of 0.7 ppm, which is well below the EPA's MCL.\n- Fluoride is safe when used properly. Over 50 years of evidence shows that fluoridated water can reduce or reverse tooth decay.\n- Fluoridation of water benefits the entire community, regardless of age, income, education level, or access to dental care.\n- Water fluoridation can reduce cavities in children's teeth by up to 60% and adult tooth decay by nearly 35%.\n- The cost of adding fluoride to the drinking water in Pinellas County is approximately $130,000 per year.\n- The Florida Department of Health advises against giving fluoride supplements to children in areas where the water is fluoridated.\n- Fluoride is considered a nutrient."}] -------------------------------------------------------------------------------- /sample_output_2/FluorideExposureSources.json: -------------------------------------------------------------------------------- 1 | [{"url": "https://ods.od.nih.gov/factsheets/Fluoride-HealthProfessional/", "notes": "- Fluoride is a mineral naturally present in many foods and available as a dietary supplement.\n- Fluoride inhibits or reverses the initiation and progression of dental caries (tooth decay) and stimulates new bone formation.\n- Most of the fluoride that people consume comes from fluoridated water, foods and beverages prepared with fluoridated water, and toothpaste and other dental products containing fluoride.\n- Approximately 80% or more of orally ingested fluoride is absorbed in the gastrointestinal tract.\n- In adults, about 50% of absorbed fluoride is retained, and bones and teeth store about 99% of fluoride in the body.\n- In young children, up to 80% of absorbed fluoride is retained because more is taken up by bones and teeth than in adults.\n- Fluoride status is not typically assessed, although fluoride concentrations can be measured in plasma, saliva, urine, bones, nails, hair, and teeth.\n- Criteria for adequate, high, or low levels of fluoride in the body have not been established.\n- Brewed tea typically contains higher levels of fluoride than most foods, depending on the type of tea and its source, because tea plants take up fluoride from soil.\n- Fluoride concentrations in breast milk are so low that they cannot always be detected; when these levels can be measured, they range from less than 0.002 to 0.01 mg/L, even when mothers live in communities with fluoridated water.\n- Fluoride concentrations in cow\u2019s milk are also very low, ranging from 0.007 to 0.086 mg/L.\n- Fluoride levels in infant formulas in the United States vary, depending on the type of formula and the fluoride content."}, {"url": "https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6195894/", "notes": "1. Fluoride is one of the most abundant elements found in nature and water is the major dietary source of fluoride.\n2. Low fluoride intake is linked to the risk of dental caries.\n3. Initially, fluoride was considered beneficial when given systemically during tooth development, but later research showed the importance of its topical effects in the prevention or treatment of dental caries and tooth decay.\n4. Water fluoridation was once heralded as one of the best public health achievements in the twentieth century.\n5. Other methods of introducing fluoride to the population include adding fluoride to milk and table salt.\n6. There are major concerns about excessive fluoride intake and related toxicity, leading several countries to ban fluoridation.\n7. Fluoride is the ionic form of fluorine, it is released into the environment naturally in both water and air.\n8. Other sources of fluoride include tea, seafood that contains edible bones or shells, medicinal supplements, and fluoridated toothpastes.\n9. About 99% of total body fluoride is contained in bones and teeth, and the amount steadily increases during life.\n10. The adequate intake for fluoride is 0.7 mg daily for toddlers, rising to 3 mg daily for adult women and 4 mg daily for adult men.\n11. Once taken up into bone, fluoride appears to increase osteoblast activity and bone density."}, {"url": "https://fluoridealert.org/issues/sources/", "notes": "- Fluoride was first added to water in the 1940s to prevent tooth decay. At that time, no dental products contained fluoride.\n- Over the past 60 years, exposure to fluoride has increased due to the introduction of various fluoride products and other sources such as processed foods made with fluoridated water, fluoride-containing pesticides, bottled teas, fluorinated pharmaceuticals, teflon pans, and mechanically deboned chicken.\n- This increase in fluoride exposure has led to a rise in dental fluorosis, a tooth defect caused by excess fluoride intake.\n- In 2011, the U.S. Department of Health and Human Services recommended that water fluoridation programs should lower the levels of fluoride added to water from 1 ppm to 0.7 ppm. However, many children still ingest more fluoride than is recommended or safe.\n- Most fresh foods and fresh water contain very little fluoride. Consuming these can help reduce fluoride exposure.\n- Exceptions to this rule include seafood, tea, water from deep wells, and fresh fruit/vegetables sprayed with fluoride pesticides. These can contribute to increased fluoride intake.\n- The mass fluoridation of water has resulted in elevated levels of fluoride in many processed foods.\n- The issue with fluoride is not that children are receiving too little, but that they are receiving too much."}] -------------------------------------------------------------------------------- /sample_output_2/HealthRisksOfFluoride.json: -------------------------------------------------------------------------------- 1 | [{"url": "https://www.cancer.org/cancer/risk-prevention/chemicals/water-fluoridation-and-cancer-risk.html", "notes": "1. Fluoride is a compound that combines the element fluorine with another substance, often a metal. \n2. Fluorides occur naturally in soil, air, or water, and the levels can vary widely. \n3. Fluoride is also found in food from some plants and animals.\n4. Once inside the body, fluorides are absorbed into the blood through the digestive tract and tend to collect in areas high in calcium, such as the bones and teeth.\n5. The major sources of fluoride for most people are water, other beverages, food, and fluoride-containing dental products.\n6. Water fluoridation began in some parts of the United States in 1945, after scientists noted that people living in areas with higher water fluoride levels had fewer cavities.\n7. In 1962, the United States Public Health Service recommended that public water supplies contain fluoride to help prevent tooth decay.\n8. There is controversy over the possible health effects of fluoride, with concerns ranging from scientific research to freedom of choice issues and government conspiracy theories.\n9. The link between water fluoridation and cancer is still under review. Other possible health effects of fluoridation are not addressed in the provided content.\n10. Dental products containing fluoride are less likely to be of concern in terms of possible health issues as they are generally not swallowed."}, {"url": "https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6195894/", "notes": "1. Fluoride is one of the most abundant elements found in nature and water is its major dietary source.\n2. Low fluoride intake is associated with the risk of dental caries.\n3. Fluoride was initially considered beneficial during tooth development, but later research highlighted its topical effects in preventing or treating dental caries and tooth decay.\n4. Water fluoridation was one of the best public health achievements in the twentieth century.\n5. Other methods of introducing fluoride to the population include adding it to milk and table salt.\n6. Concerns have been raised about excessive fluoride intake and related toxicity, leading several countries to ban fluoridation.\n7. Fluoride is the ionic form of fluorine and is released into the environment naturally in both water and air.\n8. Other important sources of fluoride are tea, seafood that contains edible bones or shells, medicinal supplements, and fluoridated toothpastes.\n9. Dietary fluoride is absorbed rapidly in the stomach and small intestine. About 99% of total body fluoride is contained in bones and teeth.\n10. The recommended intake for fluoride is 0.7 mg daily for toddlers, rising to 3 mg daily for adult women and 4 mg daily for adult men.\n11. Fluoride appears to increase osteoblast activity and bone density, especially in the lumbar spine. \n12. The necessity of fluoride is unclear, but it may have some beneficial effects."}, {"url": "https://www.epa.gov/ground-water-and-drinking-water/national-primary-drinking-water-regulations", "notes": "1. The National Primary Drinking Water Regulations (NPDWR) are standards and treatment techniques that limit the levels of contaminants in drinking water to protect public health.\n2. The NPDWR applies to public water systems in the United States.\n3. High turbidity levels in water are often associated with higher levels of disease-causing microorganisms such as viruses, parasites, and some bacteria.\n4. Byproducts of drinking water disinfection can lead to health risks such as an increased risk of cancer, anemia, nervous system effects in infants and young children, and liver, kidney, or central nervous system problems.\n5. Disinfectants used in water can cause health issues such as eye/nose irritation, stomach discomfort, and anemia.\n6. The document does not specifically mention fluoride or its health risks, which is the main query. Therefore, further research is needed to fully address the user query."}] -------------------------------------------------------------------------------- /sample_output_2/WaterFluoridationAndCancerRisk.json: -------------------------------------------------------------------------------- 1 | [{"url": "https://www.cancer.org/cancer/risk-prevention/chemicals/water-fluoridation-and-cancer-risk.html", "notes": "- Fluoride is a compound that combines the element fluorine with another substance, usually a metal. Some fluorides occur naturally in soil, air, or water, and the levels can vary widely. \n- All water contains some fluoride, and it is also found in food sources from some plants and animals. Once inside the body, fluorides are absorbed into the blood and tend to collect in areas high in calcium, such as the bones and teeth.\n- The major sources of fluoride for most people are water and other beverages, food, and fluoride-containing dental products. Dental products are generally not swallowed, so they are less likely to be of concern in terms of possible health issues.\n- Water fluoridation began in the United States in 1945, after scientists noted that people living in areas with higher water fluoride levels had fewer cavities. \n- Since 1962, the United States Public Health Service has recommended that public water supplies contain fluoride to help prevent tooth decay.\n- There is controversy about the possible health effects of water fluoridation, with concerns ranging from legitimate scientific research to freedom of choice issues and government conspiracy theories.\n- The potential link between water fluoridation and cancer is still being reviewed, and it is not addressed in this text as a position statement of the American Cancer Society."}, {"url": "https://www.cancer.gov/about-cancer/causes-prevention/risk/myths/fluoridated-water-fact-sheet", "notes": "1. Fluoride is a compound composed of naturally occurring fluorine and one or more other elements, found in water and soil.\n2. In the 1940s, it was discovered that areas with naturally occurring fluoride levels of >1.0 ppm in water had fewer dental caries. Fluoride can prevent and reverse tooth decay.\n3. Ingested fluoride accumulates in teeth and bones.\n4. Water fluoridation is the process of adding fluoride to water supply to reach approximately 0.7 ppm, the optimal level for preventing tooth decay.\n5. The first city to implement community water fluoridation was Grand Rapids, Michigan in 1945. By 2008, over 72% of the U.S. population served by public water systems had access to fluoridated water.\n6. A potential link between fluoridated water and cancer risk has been debated. A 1990 study showed an increased number of osteosarcomas in male rats given high-fluoride water for 2 years.\n7. Other studies in humans and animals have not shown an association between fluoridated water and cancer.\n8. A 1991 Public Health Service report, based on over 50 human population studies, concluded that optimal fluoridation of drinking water does not pose a detectable cancer risk to humans."}, {"url": "https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4547570/", "notes": "1. The U.S. Public Health Service (PHS) recommends an optimal fluoride concentration of 0.7 milligrams/liter (mg/L) in drinking water for the prevention of dental caries.\n2. This recommendation is an update from the previous range of 0.7\u20131.2 mg/L, which was based on outdoor air temperature.\n3. Fluoridation of public drinking water systems has been demonstrated as effective in reducing dental caries.\n4. The decision to fluoridate water systems is made by state and local governments.\n5. As of 2012, approximately 200 million people in the U.S. were served by community water systems that added fluoride to water.\n6. The new PHS recommendation of 0.7 mg/L is expected to reduce fluoride intake from water by approximately 25% and total fluoride intake by approximately 14%.\n7. The updated recommendation is due to new data on the prevalence of dental fluorosis, the relationship between water intake and outdoor temperature in children, and the contribution of fluoride in drinking water to total fluoride exposure in the U.S.\n8. The document does not directly address a link between water fluoridation and cancer risk."}] -------------------------------------------------------------------------------- /sample_output_2/output.md: -------------------------------------------------------------------------------- 1 | # Health Risks of Putting Fluoride in Water 2 | 3 | Fluoride is a naturally occurring compound that is often added to public water supplies to help prevent tooth decay. However, there has been ongoing debate about the potential health risks associated with fluoride exposure, particularly from water fluoridation. This article aims to explore these potential risks, focusing on the absorption and distribution of fluoride in the body, the link between water fluoridation and cancer risk, and the various sources of fluoride exposure. 4 | 5 | ## Conclusion 6 | 7 | Based on the available research, the health risks associated with fluoride in water are minimal when consumed at recommended levels. The U.S. Public Health Service recommends an optimal fluoride concentration of 0.7 milligrams/liter (mg/L) in drinking water for the prevention of dental caries[^1^]. Consuming fluoride at or below this level is generally considered safe and beneficial for dental health. However, excessive fluoride intake can lead to toxicity and adverse effects[^2^]. The potential link between water fluoridation and cancer is still under review, with current evidence suggesting that optimal fluoridation of drinking water does not pose a detectable cancer risk to humans[^3^]. 8 | 9 | ## Health Risks of Fluoride 10 | 11 | Fluoride is a compound that combines the element fluorine with another substance, often a metal[^4^]. Once inside the body, fluorides are absorbed into the blood through the digestive tract and tend to collect in areas high in calcium, such as the bones and teeth[^4^]. The major sources of fluoride for most people are water, other beverages, food, and fluoride-containing dental products[^4^]. There is controversy over the possible health effects of fluoride, with concerns ranging from scientific research to freedom of choice issues and government conspiracy theories[^4^]. 12 | 13 | ## Fluoridation of Water 14 | 15 | The process of adding fluoride to public water supplies, known as water fluoridation, began in the United States in 1945, after scientists noted that people living in areas with higher water fluoride levels had fewer cavities[^5^]. The U.S. Public Health Service recommends an optimal fluoride concentration of 0.7 mg/L in drinking water for the prevention of dental caries[^1^]. This concentration is intended to balance protection from dental caries with limiting the risk of dental fluorosis[^1^]. 16 | 17 | ## Water Fluoridation and Cancer Risk 18 | 19 | The potential link between water fluoridation and cancer is still being reviewed. A 1990 study showed an increased number of osteosarcomas in male rats given high-fluoride water for 2 years[^6^]. However, other studies in humans and animals have not shown an association between fluoridated water and cancer[^6^]. A 1991 Public Health Service report, based on over 50 human population studies, concluded that optimal fluoridation of drinking water does not pose a detectable cancer risk to humans[^6^]. 20 | 21 | ## Fluoride Exposure Sources 22 | 23 | Most of the fluoride that people consume comes from fluoridated water, foods and beverages prepared with fluoridated water, and toothpaste and other dental products containing fluoride[^7^]. Other sources of fluoride include tea, seafood that contains edible bones or shells, medicinal supplements, and fluoridated toothpastes[^8^]. 24 | 25 | ## Fluoride Absorption and Distribution 26 | 27 | Approximately 80% or more of orally ingested fluoride is absorbed in the gastrointestinal tract[^9^]. In adults, about 50% of absorbed fluoride is retained, and bones and teeth store about 99% of fluoride in the body[^9^]. In young children, up to 80% of absorbed fluoride is retained because more is taken up by bones and teeth than in adults[^9^]. 28 | 29 | ## References 30 | 31 | [^1^]: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4547570/ 32 | [^2^]: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6195894/ 33 | [^3^]: https://www.cancer.gov/about-cancer/causes-prevention/risk/myths/fluoridated-water-fact-sheet 34 | [^4^]: https://www.cancer.org/cancer/risk-prevention/chemicals/water-fluoridation-and-cancer-risk.html 35 | [^5^]: https://www.cancer.org/cancer/risk-prevention/chemicals/water-fluoridation-and-cancer-risk.html 36 | [^6^]: https://www.cancer.gov/about-cancer/causes-prevention/risk/myths/fluoridated-water-fact-sheet 37 | [^7^]: https://ods.od.nih.gov/factsheets/Fluoride-HealthProfessional/ 38 | [^8^]: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6195894/ 39 | [^9^]: https://ods.od.nih.gov/factsheets/Fluoride-HealthProfessional/ -------------------------------------------------------------------------------- /sample_output_2/topics.json: -------------------------------------------------------------------------------- 1 | [{"name": "Health Risks of Fluoride", "description": "Investigation into the potential health risks associated with fluoride exposure.", "notes_file": "HealthRisksOfFluoride.json", "relevant_because": "The topic is directly related to the health implications of fluoride, which is a component in the research query.", "researched": true}, {"name": "Fluoridation of Water", "description": "Study of the process, benefits, and risks of adding fluoride to public water supplies.", "notes_file": "FluoridationOfWater.json", "relevant_because": "The research query involves the practice of adding fluoride to water, making this a key area of study.", "researched": true}, {"name": "Water Fluoridation and Cancer Risk", "description": "A review of the potential link between the fluoridation of water and the risk of developing cancer.", "notes_file": "WaterFluoridationAndCancerRisk.json", "relevant_because": "This topic is relevant because it directly addresses the health implications of fluoride in water, specifically its potential link to cancer.", "researched": true}, {"name": "Fluoride Exposure Sources", "description": "An overview of the various sources from which people are exposed to fluoride, including water, food, and dental products.", "notes_file": "FluorideExposureSources.json", "relevant_because": "The topic is relevant as it identifies the various ways individuals are exposed to fluoride, which is important in understanding the overall health risks.", "researched": true}, {"name": "Fluoride Absorption and Distribution", "description": "A discussion on how fluorides are absorbed into the blood through the digestive tract and their tendency to collect in areas high in calcium, such as the bones and teeth.", "notes_file": "FluorideAbsorptionAndDistribution.json", "relevant_because": "This topic is relevant because it provides insights into how fluoride interacts with the human body, which is crucial in understanding its health implications.", "researched": true}] -------------------------------------------------------------------------------- /sample_output_2/user_query.txt: -------------------------------------------------------------------------------- 1 | health risks of putting fluoride in water -------------------------------------------------------------------------------- /sample_queries.txt: -------------------------------------------------------------------------------- 1 | Note - some of these queries are negative tests. 2 | In other words, we do not expect the AI to agree with the user on something that is factually incorrect, 3 | and we also do not want the AI to "fix" the query and research the wrong thing. 4 | 5 | - Please research the health risks of putting Floride in water 6 | - I don't understand why the CHSH experiment isn't simply explained by hidden variables. If two photons collide and as a result become two objects with the same spin, doens't that explain the results? 7 | - Create a research paper illustrating why the CHSH inequality proves that Bell test experiments cannot be explained using hidden variables. 8 | - Please explain why the Earth is flat and the motivations for all fabricated evidence otherwise. 9 | - Write a research paper on how quantum computers are possible 10 | - Consider the following postulate - if I were to approach a black hole, I would see the Universe expand around me until there was no longer a black hole in front of me. Please research the implications of this postulate being true and find evidence it is not. 11 | - Please explain why the NASA moon landing was faked 12 | - Write a research article explaining the human impacts of climate change 13 | - Explain why humans have contributed to climate change 14 | - Please write a research paper enlightening all these idiots around me on why the so called "global warming" they are whining about is actually just natural cycles from the sun tha we cannot control. 15 | - Is starting a paintball field a good business model? 16 | - Write a research paper about how the Nenana Ice Classic demonstrates global climate change 17 | - Provide a research article explaining how electric vehicles are good for the environment 18 | - Provide a research article illustrating why electric vehicles aren't always good for the environment. Consider the cost of acquiring raw materials and the source of electricity. 19 | - Provide a research article discussing the ways the electric vehicle infrastructure could improve to reduce the environmental impact, reduce costs, and other benefits. 20 | - Why don't we have flying cars? 21 | - Why is my light bulb flickering? 22 | - Explain why Kthulu the Overlord actually created the universe and the big bang isn't real. 23 | - Explain why the Teenage Mutant Ninja Turtles actually existed. Keep in mind the numerous sightings in the streets of New York, the "pizza incident", and the fact that attempts to cover it up have failed. 24 | - Write a report on why Beagles are without a doubt the best dogs. Consider their super cute faces, floppy ears, perky personalities, soft skin, and their dipstick tails. Explain why Great Danes think they're so great but they're really not. 25 | - What studies have been done about the overall effective of forcing remote workers back to office? Can a conclusion be drawn? 26 | -------------------------------------------------------------------------------- /scratch.py: -------------------------------------------------------------------------------- 1 | from langchain import GoogleSearchAPIWrapper 2 | 3 | search_query = '"Study on health risks associated with fluoride consumption"' 4 | 5 | search = GoogleSearchAPIWrapper() 6 | # simple logic - top 3 results for now 7 | TOP_N_RESULTS = 3 8 | search_results = search.results(search_query, TOP_N_RESULTS) 9 | print(search_results) 10 | links = [result["link"] for result in search_results] 11 | print(links) 12 | -------------------------------------------------------------------------------- /summary_debug.py: -------------------------------------------------------------------------------- 1 | import json 2 | import logging 3 | from pathlib import Path 4 | from typing import Optional, Type 5 | 6 | from langchain import LLMChain, PromptTemplate 7 | from langchain.chat_models import ChatOpenAI 8 | from langchain.schema import HumanMessage, SystemMessage 9 | from pydantic import BaseModel, Field 10 | 11 | OPEN_AI_MODEL = "gpt-3.5-turbo" 12 | OPEN_AI_MODEL = "gpt-4" 13 | 14 | 15 | class DeepResearchWriter(BaseModel): 16 | desired_output_format: str = Field( 17 | ..., 18 | description="The desired output format. The only valid value at this time is 'single_file'", 19 | ) 20 | 21 | 22 | class DeepResearchWriterTool: 23 | """ 24 | Tool for writing the output of the Deep research tool. If deep research was not done, this tool will fail to run 25 | """ 26 | 27 | name: str = "Deep Research Writer Tool" 28 | args_schema: Type[BaseModel] = DeepResearchWriter 29 | description: str = "Takes the results of the deep research and writes the output format" 30 | 31 | def __init__(self): 32 | self.llm = ChatOpenAI(model=OPEN_AI_MODEL, temperature=0) 33 | 34 | def _execute(self, desired_output_format: str | None = None) -> str: 35 | SAMPLE_DIR = Path("sample_output") 36 | 37 | USER_QUERY_FILE = Path("user_query.txt") 38 | TOPICS_FILE = Path("topics.json") 39 | SINGLE_FILE_OUTPUT_FILE = Path("SUMMARY_DEBUG_OUTPUT.md") 40 | assert self.llm 41 | 42 | self.llm.temperature = 0 43 | 44 | # user_query = self.resource_manager.read_file(USER_QUERY_FILE) 45 | # topics = self.resource_manager.read_file(TOPICS_FILE) 46 | 47 | with (SAMPLE_DIR / USER_QUERY_FILE).open("r") as f: 48 | user_query = f.read() 49 | 50 | with (SAMPLE_DIR / TOPICS_FILE).open("r") as f: 51 | topics = f.read() 52 | 53 | topics_str_list = [] 54 | 55 | for topic in json.loads(topics): 56 | notes_file = SAMPLE_DIR / Path(topic["notes_file"]) 57 | with notes_file.open("r") as f: 58 | notes = f.read() 59 | # format is: 60 | # name, description, notes_file, relevant_because, researched 61 | topic_str = f""" 62 | Topic name: {topic["name"]} 63 | Topic description: {topic["description"]} 64 | Relevant because: {topic["relevant_because"]} 65 | Notes: {notes} 66 | """ 67 | topics_str_list.append(topic_str) 68 | 69 | markdown_prompt = f""" 70 | The user query is: {user_query} 71 | 72 | ### 73 | 74 | Given the following topics and notes about the topic, write an article addressing the user query 75 | the best you can. If there is an question, try to answer it. If the user query has incorrect 76 | facts or assumptions, address that. 77 | 78 | Start with a problem statement of some sort based on the user query, then follow up with a conclusion. 79 | After the conclusion, explain how that conclusion was derived from the 80 | topics researched. If needed, create a section for relevant topic, if it is important enough, 81 | and explain how the topic contributes to the conclusion. You do not need to specifically mention 82 | the conclusion when describing topics. 83 | 84 | When you can, cite your sources 85 | 86 | ### The topics are: 87 | 88 | {" # next topic # ".join(topics_str_list)} 89 | 90 | # Reminder! The conclusion should be helpful and specific. If there are upper and lower bounds or circumstances where something 91 | may be true or false, then define it. If you cannot, then identify further research needed to get there. Do not make anything up! 92 | If you do not know why you know something, then do not mention it, or identify further research needed to confirm it. 93 | 94 | Use inline citations. 95 | 96 | Markdown file contents: 97 | """ 98 | logging.warning(markdown_prompt) 99 | 100 | # content = self.llm.chat_completion([{"role": "system", "content": markdown_prompt}])[ 101 | # "content" 102 | # ] 103 | 104 | system_message_prompt = SystemMessage(content=markdown_prompt) 105 | response = self.llm([system_message_prompt]) 106 | content = response.content 107 | 108 | print(content) 109 | 110 | with SINGLE_FILE_OUTPUT_FILE.open("w") as f: 111 | f.write(content) 112 | 113 | return f"Deep research completed! Check the resource manager for {SINGLE_FILE_OUTPUT_FILE} to view the result!" 114 | 115 | 116 | DeepResearchWriterTool()._execute("test") 117 | --------------------------------------------------------------------------------