├── .gitignore ├── slides.pdf ├── images ├── core.png ├── qr-web.png ├── expansion-2.png ├── expansion-3.png ├── expansion.png ├── qr-discord.png ├── qr-dottxt.png ├── qr-github.png ├── qr-outlines.png ├── qr-twitter.png ├── qr-website.png ├── qr-website.svg ├── traversal.png ├── outlines-logo.png ├── traversal-creepy.png ├── dottxt-logo-white.png ├── logo-square-larger.pdf ├── qr-cameron-twitter.png ├── qr-dottxt-twitter.png ├── json-or-fired-white.png ├── qr-website-inverted.png ├── beige-vertical-logo.svg └── beige-horizontal-logo.svg ├── .env.example ├── .env.template ├── db.py ├── template_llama3.jinja ├── requirements.txt ├── LICENSE ├── custom.scss ├── download_llama.py ├── fonts.css ├── structured_gen.py ├── modal_embeddings.py ├── modal_vllm_container.py ├── README.md ├── slides.qmd ├── expand.py └── slides.html /.gitignore: -------------------------------------------------------------------------------- 1 | .venv/ 2 | .env 3 | __pycache__/ 4 | fonts/ 5 | slides_files/ 6 | -------------------------------------------------------------------------------- /slides.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cpfiffer/self-expansion/HEAD/slides.pdf -------------------------------------------------------------------------------- /images/core.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cpfiffer/self-expansion/HEAD/images/core.png -------------------------------------------------------------------------------- /images/qr-web.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cpfiffer/self-expansion/HEAD/images/qr-web.png -------------------------------------------------------------------------------- /images/expansion-2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cpfiffer/self-expansion/HEAD/images/expansion-2.png -------------------------------------------------------------------------------- /images/expansion-3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cpfiffer/self-expansion/HEAD/images/expansion-3.png -------------------------------------------------------------------------------- /images/expansion.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cpfiffer/self-expansion/HEAD/images/expansion.png -------------------------------------------------------------------------------- /images/qr-discord.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cpfiffer/self-expansion/HEAD/images/qr-discord.png -------------------------------------------------------------------------------- /images/qr-dottxt.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cpfiffer/self-expansion/HEAD/images/qr-dottxt.png -------------------------------------------------------------------------------- /images/qr-github.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cpfiffer/self-expansion/HEAD/images/qr-github.png -------------------------------------------------------------------------------- /images/qr-outlines.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cpfiffer/self-expansion/HEAD/images/qr-outlines.png -------------------------------------------------------------------------------- /images/qr-twitter.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cpfiffer/self-expansion/HEAD/images/qr-twitter.png -------------------------------------------------------------------------------- /images/qr-website.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cpfiffer/self-expansion/HEAD/images/qr-website.png -------------------------------------------------------------------------------- /images/qr-website.svg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cpfiffer/self-expansion/HEAD/images/qr-website.svg -------------------------------------------------------------------------------- /images/traversal.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cpfiffer/self-expansion/HEAD/images/traversal.png -------------------------------------------------------------------------------- /images/outlines-logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cpfiffer/self-expansion/HEAD/images/outlines-logo.png -------------------------------------------------------------------------------- /images/traversal-creepy.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cpfiffer/self-expansion/HEAD/images/traversal-creepy.png -------------------------------------------------------------------------------- /images/dottxt-logo-white.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cpfiffer/self-expansion/HEAD/images/dottxt-logo-white.png -------------------------------------------------------------------------------- /images/logo-square-larger.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cpfiffer/self-expansion/HEAD/images/logo-square-larger.pdf -------------------------------------------------------------------------------- /images/qr-cameron-twitter.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cpfiffer/self-expansion/HEAD/images/qr-cameron-twitter.png -------------------------------------------------------------------------------- /images/qr-dottxt-twitter.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cpfiffer/self-expansion/HEAD/images/qr-dottxt-twitter.png -------------------------------------------------------------------------------- /images/json-or-fired-white.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cpfiffer/self-expansion/HEAD/images/json-or-fired-white.png -------------------------------------------------------------------------------- /images/qr-website-inverted.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cpfiffer/self-expansion/HEAD/images/qr-website-inverted.png -------------------------------------------------------------------------------- /.env.example: -------------------------------------------------------------------------------- 1 | NEO4J_URI= 2 | NEO4J_USERNAME= 3 | NEO4J_PASSWORD= 4 | VLLM_BASE_URL=https://url--that-ends-in-serve.modal.run/v1 5 | VLLM_TOKEN=super-secret-token 6 | -------------------------------------------------------------------------------- /.env.template: -------------------------------------------------------------------------------- 1 | # Neo4j environment variables 2 | NEO4J_URI="neo4j+s://.databases.neo4j.io" # replace with your database ID 3 | NEO4J_USERNAME='neo4j' # you can use this as the default 4 | NEO4J_PASSWORD='...' 5 | 6 | # VLLM environment variables 7 | # Make sure to include the `/v1` at the end of the VLLM base URL 8 | VLLM_BASE_URL="https://your-organization-name--self-expansion-vllm-serve.modal.run/v1" 9 | VLLM_TOKEN="super-secret-token" # This is the default token for the modal server -------------------------------------------------------------------------------- /db.py: -------------------------------------------------------------------------------- 1 | import os 2 | from neo4j import GraphDatabase 3 | from dotenv import load_dotenv 4 | 5 | # Load .env file with override=True to take precedence over system variables 6 | load_dotenv(override=True) 7 | 8 | try: # to connect to the graph db 9 | URI = os.environ["NEO4J_URI"] 10 | assert URI 11 | AUTH = (os.environ["NEO4J_USERNAME"], os.environ["NEO4J_PASSWORD"]) 12 | except Exception as e: 13 | raise ValueError("Error fetching Neo4J credentials from environment") from e 14 | 15 | 16 | driver = GraphDatabase.driver(URI, auth=AUTH) 17 | driver.verify_connectivity() 18 | -------------------------------------------------------------------------------- /template_llama3.jinja: -------------------------------------------------------------------------------- 1 | {% if messages[0]['role'] == 'system' %} 2 | {% set offset = 1 %} 3 | {% else %} 4 | {% set offset = 0 %} 5 | {% endif %} 6 | 7 | {{ bos_token }} 8 | {% for message in messages %} 9 | {% if (message['role'] == 'user') != (loop.index0 % 2 == offset) %} 10 | {{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }} 11 | {% endif %} 12 | 13 | {{ '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n' + message['content'] | trim + '<|eot_id|>' }} 14 | {% endfor %} 15 | 16 | {% if add_generation_prompt %} 17 | {{ '<|start_header_id|>' + 'assistant' + '<|end_header_id|>\n\n' }} 18 | {% endif %} -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | aiohappyeyeballs==2.4.4 2 | aiohttp==3.11.11 3 | aiosignal==1.3.2 4 | annotated-types==0.7.0 5 | anyio==4.8.0 6 | async-timeout==5.0.1 7 | attrs==24.3.0 8 | certifi==2024.12.14 9 | click==8.1.8 10 | distro==1.9.0 11 | exceptiongroup==1.2.2 12 | fastapi==0.115.6 13 | frozenlist==1.5.0 14 | grpclib==0.4.7 15 | h11==0.14.0 16 | h2==4.1.0 17 | hpack==4.0.0 18 | httpcore==1.0.7 19 | httpx==0.28.1 20 | hyperframe==6.0.1 21 | idna==3.10 22 | jiter==0.8.2 23 | markdown-it-py==3.0.0 24 | mdurl==0.1.2 25 | modal==0.72.39 26 | multidict==6.1.0 27 | neo4j==5.27.0 28 | openai==1.60.0 29 | propcache==0.2.1 30 | protobuf==5.29.3 31 | pydantic==2.10.5 32 | pydantic-core==2.27.2 33 | pygments==2.19.1 34 | python-dotenv==1.0.1 35 | pytz==2024.2 36 | rich==13.9.4 37 | shellingham==1.5.4 38 | sigtools==4.0.1 39 | sniffio==1.3.1 40 | starlette==0.41.3 41 | synchronicity==0.9.9 42 | toml==0.10.2 43 | tqdm==4.67.1 44 | typer==0.15.1 45 | types-certifi==2021.10.8.3 46 | types-toml==0.10.8.20240310 47 | typing-extensions==4.12.2 48 | watchfiles==1.0.4 49 | yarl==1.18.3 50 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2025 Cameron Pfiffer 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /custom.scss: -------------------------------------------------------------------------------- 1 | /*-- scss:defaults --*/ 2 | 3 | $body-bg: #1d1d1b; 4 | $body-color: #fff; 5 | $link-color: #42affa; 6 | 7 | $font-family-sans-serif: "PP Neue Montreal", sans-serif; 8 | $font-family-monospace: monospace; 9 | /*-- scss:rules --*/ 10 | 11 | .reveal .slide blockquote { 12 | border-left: 3px solid $text-muted; 13 | padding-left: 0.5em; 14 | } 15 | 16 | body { 17 | font-family: "PP Neue Montreal", sans-serif; 18 | } 19 | 20 | code, pre, .sourceCode { 21 | font-family: monospace; 22 | border: none !important; 23 | max-height: 100% !important; 24 | } 25 | 26 | // Center code blocks on the slides 27 | .reveal sourceCode { 28 | display: flex; 29 | justify-content: center; 30 | align-items: center; 31 | height: 100% !important; 32 | } 33 | 34 | .reveal pre { 35 | font-family: monospace; 36 | display: flex; 37 | justify-content: center; 38 | align-items: center; 39 | } 40 | 41 | .reveal code { 42 | width: fit-content; 43 | } 44 | 45 | // .sourceCode { 46 | // // Centers the code blocks in the slides 47 | // display: flex; 48 | // justify-content: center; 49 | // width: 100%; 50 | // } 51 | 52 | .quarto-title-affiliation { 53 | font-size: 1.5em !important; 54 | margin-bottom: 0 !important; 55 | margin-top: 0 !important; 56 | } 57 | 58 | 59 | .title-slide { 60 | text-align: center; 61 | } 62 | -------------------------------------------------------------------------------- /download_llama.py: -------------------------------------------------------------------------------- 1 | # --- 2 | # args: ["--force-download"] 3 | # --- 4 | 5 | import os 6 | import modal 7 | 8 | MODELS_DIR = "/llamas" 9 | # MODEL_NAME = "neuralmagic/Meta-Llama-3.1-70B-Instruct-quantized.w8a8" 10 | MODEL_NAME = "neuralmagic/Meta-Llama-3.1-8B-Instruct-quantized.w8a8" 11 | 12 | volume = modal.Volume.from_name("llamas", create_if_missing=True) 13 | 14 | image = ( 15 | modal.Image.debian_slim(python_version="3.10") 16 | .pip_install( 17 | [ 18 | "huggingface_hub", # download models from the Hugging Face Hub 19 | "hf-transfer", # download models faster with Rust 20 | ] 21 | ) 22 | .env({"HF_HUB_ENABLE_HF_TRANSFER": "1", "HF_TOKEN": os.environ["HF_TOKEN"]}) 23 | ) 24 | 25 | 26 | MINUTES = 60 27 | HOURS = 60 * MINUTES 28 | 29 | app = modal.App( 30 | image=image, 31 | # secrets=[ # add a Hugging Face Secret if you need to download a gated model 32 | # modal.Secret.from_name("huggingface-secret", required_keys=["HF_TOKEN"]) 33 | # ], 34 | ) 35 | 36 | 37 | @app.function(volumes={MODELS_DIR: volume}, timeout=4 * HOURS) 38 | def download_model(model_name, force_download=False): 39 | from huggingface_hub import snapshot_download 40 | 41 | volume.reload() 42 | 43 | snapshot_download( 44 | model_name, 45 | local_dir=MODELS_DIR + "/" + model_name, 46 | ignore_patterns=[ 47 | "*.pt", 48 | "*.bin", 49 | "*.pth", 50 | "original/*", 51 | ], # Ensure safetensors 52 | force_download=force_download, 53 | ) 54 | 55 | volume.commit() 56 | 57 | 58 | @app.local_entrypoint() 59 | def main( 60 | model_name: str = MODEL_NAME, 61 | force_download: bool = False, 62 | ): 63 | download_model.remote(model_name, force_download) 64 | -------------------------------------------------------------------------------- /fonts.css: -------------------------------------------------------------------------------- 1 | @font-face { 2 | font-family: 'PP NeueBit'; 3 | src: url('fonts/PPNeueBit-Bold.woff') format('woff'); 4 | } 5 | 6 | @font-face { 7 | font-family: 'PP NeueBit Regular'; 8 | src: url('fonts/PPNeueBit-Regular.woff') format('woff'); 9 | } 10 | 11 | 12 | @font-face { 13 | font-family: 'PP Neue Montreal'; 14 | src: url('fonts/PPNeueMontreal-Regular.woff') format('woff'); 15 | } 16 | 17 | @font-face { 18 | font-family: 'PP Neue Montreal Medium'; 19 | src: url('fonts/PPNeueMontrealMono-Regular.woff') format('woff'); 20 | } 21 | 22 | /* Make monospace font 1em */ 23 | .reveal .slides { 24 | font-size: 1.0em; 25 | color: #dcd0b8; 26 | } 27 | 28 | .dottxt-logo { 29 | font-family: 'PP NeueBit', sans-serif; 30 | color: #BD932F !important; 31 | font-size: 1.5em; 32 | } 33 | 34 | .outlines-logo { 35 | font-family: 'PP NeueBit', sans-serif; 36 | color: #e35a26 !important; 37 | font-size: 1.5em; 38 | } 39 | 40 | .bits { 41 | font-family: 'PP NeueBit Regular', sa ns-serif; 42 | font-weight: 400; 43 | color: #dcd0b8; 44 | } 45 | 46 | .bits-bold { 47 | font-family: 'PP NeueBit', sans-serif; 48 | font-weight: 700; 49 | color: #dcd0b8; 50 | } 51 | 52 | .reveal .slides h1 { 53 | font-family: "PP Neue Montreal", sans-serif; 54 | /* font-family: "PP NeueBit", sans-serif; */ 55 | font-size: 10em; 56 | text-align: center; 57 | color: #dcd0b8; 58 | } 59 | 60 | .reveal .slides h2 { 61 | font-family: "PP Neue Montreal", sans-serif; 62 | font-size: 2em; 63 | text-align: center; 64 | color: #dcd0b8; 65 | } 66 | 67 | /* Code comment color */ 68 | code span.co { 69 | color: gold; 70 | } 71 | 72 | .reveal .title-slide h1 { 73 | font-size: 2.5em !important; 74 | line-height: 1em !important; 75 | } 76 | 77 | .mermaid .edge-label { 78 | /* No background */ 79 | background-color: transparent !important; 80 | } 81 | 82 | /* Center source code blocks vertically */ 83 | .reveal div.sourceCode, .cypher { 84 | display: flex; 85 | align-items: center; 86 | justify-content: center; 87 | height: 100%; 88 | } 89 | 90 | .reveal pre.sourceCode, .reveal pre.code { 91 | margin: 0; 92 | width: 100%; 93 | } 94 | 95 | /* Ensure code blocks don't overflow */ 96 | .reveal pre code { 97 | max-height: 100%; 98 | overflow-y: auto; 99 | overflow-x: hidden; 100 | } 101 | -------------------------------------------------------------------------------- /structured_gen.py: -------------------------------------------------------------------------------- 1 | import modal 2 | from openai import OpenAI 3 | from pydantic import BaseModel 4 | from typing import List, Dict 5 | 6 | import os 7 | import dotenv 8 | 9 | dotenv.load_dotenv() 10 | 11 | CLIENT = OpenAI( 12 | base_url=os.getenv("VLLM_BASE_URL"), 13 | api_key=os.getenv("VLLM_TOKEN"), 14 | ) 15 | 16 | print("Using base URL:", CLIENT.base_url) 17 | 18 | MODELS = CLIENT.models.list() 19 | DEFAULT_MODEL = MODELS.data[0].id 20 | 21 | print("Using model:", DEFAULT_MODEL) 22 | 23 | MAX_TOKENS = 12000 24 | 25 | 26 | def messages(user: str, system: str = "You are a helpful assistant."): 27 | ms = [{"role": "user", "content": user}] 28 | if system: 29 | ms.insert(0, {"role": "system", "content": system}) 30 | return ms 31 | 32 | 33 | def generate( 34 | messages: List[Dict[str, str]], 35 | response_format: BaseModel, 36 | ) -> BaseModel: 37 | response = CLIENT.beta.chat.completions.parse( 38 | model=DEFAULT_MODEL, 39 | messages=messages, 40 | response_format=response_format, 41 | extra_body={ 42 | # 'guided_decoding_backend': 'outlines', 43 | "max_tokens": MAX_TOKENS, 44 | }, 45 | ) 46 | return response 47 | 48 | 49 | def generate_by_schema( 50 | messages: List[Dict[str, str]], 51 | schema: str, 52 | ) -> BaseModel: 53 | response = CLIENT.chat.completions.create( 54 | model=DEFAULT_MODEL, 55 | messages=messages, 56 | extra_body={ 57 | # 'guided_decoding_backend': 'outlines', 58 | "max_tokens": MAX_TOKENS, 59 | "guided_json": schema, 60 | }, 61 | ) 62 | return response 63 | 64 | 65 | def choose( 66 | messages: List[Dict[str, str]], 67 | choices: List[str], 68 | ) -> BaseModel: 69 | completion = CLIENT.chat.completions.create( 70 | model=DEFAULT_MODEL, 71 | messages=messages, 72 | extra_body={"guided_choice": choices, "max_tokens": MAX_TOKENS}, 73 | ) 74 | return completion.choices[0].message.content 75 | 76 | 77 | def regex( 78 | messages: List[Dict[str, str]], 79 | regex: str, 80 | ) -> BaseModel: 81 | completion = CLIENT.chat.completions.create( 82 | model=DEFAULT_MODEL, 83 | messages=messages, 84 | extra_body={"guided_regex": regex, "max_tokens": MAX_TOKENS}, 85 | ) 86 | return completion.choices[0].message.content 87 | 88 | 89 | def embed(content: str) -> List[float]: 90 | f = modal.Function.lookup("self-expansion-embeddings", "embed") 91 | return f.remote(content) 92 | -------------------------------------------------------------------------------- /modal_embeddings.py: -------------------------------------------------------------------------------- 1 | import socket 2 | import subprocess 3 | from pathlib import Path 4 | 5 | import modal 6 | 7 | GPU_CONFIG = modal.gpu.L40S() 8 | MODEL_ID = "BAAI/bge-base-en-v1.5" 9 | BATCH_SIZE = 32 10 | DOCKER_IMAGE = ( 11 | # "ghcr.io/huggingface/text-embeddings-inference:hopper-1.6" # Hopper 90 for H100s (marked experimental) 12 | "ghcr.io/huggingface/text-embeddings-inference:89-1.6" # Lovelace 89 for L40S 13 | # "ghcr.io/huggingface/text-embeddings-inference:86-0.4.0" # Ampere 86 for A10 14 | # "ghcr.io/huggingface/text-embeddings-inference:0.4.0" # Ampere 80 for A100 15 | # "ghcr.io/huggingface/text-embeddings-inference:0.3.0" # Turing for T4 16 | ) 17 | 18 | DATA_PATH = Path("/data/dataset.jsonl") 19 | 20 | LAUNCH_FLAGS = [ 21 | "--model-id", 22 | MODEL_ID, 23 | "--port", 24 | "8000", 25 | ] 26 | 27 | 28 | def spawn_server() -> subprocess.Popen: 29 | process = subprocess.Popen(["text-embeddings-router"] + LAUNCH_FLAGS) 30 | 31 | # Poll until webserver at 127.0.0.1:8000 accepts connections before running inputs. 32 | while True: 33 | try: 34 | socket.create_connection(("127.0.0.1", 8000), timeout=1).close() 35 | print("Webserver ready!") 36 | return process 37 | except (socket.timeout, ConnectionRefusedError): 38 | # Check if launcher webserving process has exited. 39 | # If so, a connection can never be made. 40 | retcode = process.poll() 41 | if retcode is not None: 42 | raise RuntimeError(f"launcher exited unexpectedly with code {retcode}") 43 | 44 | 45 | def download_model(): 46 | # Wait for server to start. This downloads the model weights when not present. 47 | spawn_server().terminate() 48 | 49 | 50 | app = modal.App("self-expansion-embeddings") 51 | 52 | tei_image = ( 53 | modal.Image.from_registry( 54 | DOCKER_IMAGE, 55 | add_python="3.10", 56 | ) 57 | .dockerfile_commands("ENTRYPOINT []") 58 | .run_function(download_model, gpu=GPU_CONFIG) 59 | .pip_install("httpx") 60 | ) 61 | 62 | with tei_image.imports(): 63 | from httpx import AsyncClient 64 | 65 | 66 | @app.cls( 67 | gpu=GPU_CONFIG, 68 | image=tei_image, 69 | # Use up to 20 GPU containers at once. 70 | concurrency_limit=20, 71 | # Allow each container to process up to 10 batches at once. 72 | allow_concurrent_inputs=10, 73 | ) 74 | class TextEmbeddingsInference: 75 | @modal.enter() 76 | def setup_server(self): 77 | self.process = spawn_server() 78 | self.client = AsyncClient(base_url="http://127.0.0.1:8000") 79 | 80 | @modal.exit() 81 | def teardown_server(self): 82 | self.process.terminate() 83 | 84 | @modal.method() 85 | async def embed(self, inputs): 86 | resp = await self.client.post("/embed", json={"inputs": inputs}) 87 | resp.raise_for_status() 88 | outputs = resp.json() 89 | 90 | return outputs 91 | 92 | 93 | image = modal.Image.debian_slim(python_version="3.10").pip_install( 94 | "pandas", "db-dtypes", "tqdm" 95 | ) 96 | 97 | 98 | @app.function( 99 | image=image, 100 | ) 101 | def embed(data): 102 | model = TextEmbeddingsInference() 103 | 104 | return model.embed.remote(data)[0] 105 | 106 | 107 | @app.local_entrypoint() 108 | def main(text: str = "hello"): 109 | embedding = embed.remote([text]) 110 | print(text, embedding[:10], "...") 111 | -------------------------------------------------------------------------------- /modal_vllm_container.py: -------------------------------------------------------------------------------- 1 | import subprocess 2 | import os 3 | 4 | import modal 5 | from modal import App, Image, Mount, gpu 6 | from download_llama import MODEL_NAME, MODELS_DIR 7 | 8 | ########## CONSTANTS ########## 9 | 10 | MODEL_PATH = MODELS_DIR + "/" + MODEL_NAME 11 | 12 | # define model for serving and path to store in modal container 13 | SECONDS = 60 # for timeout 14 | 15 | try: 16 | volume = modal.Volume.lookup("llamas", create_if_missing=False) 17 | except modal.exception.NotFoundError: 18 | raise Exception("Download models first with modal run download_llama.py") 19 | 20 | 21 | ########## UTILS FUNCTIONS ########## 22 | 23 | 24 | # def download_hf_model(model_dir: str, model_name: str): 25 | # """Retrieve model from HuggingFace Hub and save into 26 | # specified path within the modal container. 27 | 28 | # Args: 29 | # model_dir (str): Path to save model weights in container. 30 | # model_name (str): HuggingFace Model ID. 31 | # """ 32 | # import os 33 | 34 | # from huggingface_hub import snapshot_download # type: ignore 35 | # from transformers.utils import move_cache # type: ignore 36 | 37 | # os.makedirs(model_dir, exist_ok=True) 38 | 39 | # snapshot_download( 40 | # model_name, 41 | # local_dir=model_dir, 42 | # # consolidated.safetensors is prevent error here: https://github.com/vllm-project/vllm/pull/5005 43 | # ignore_patterns=["*.pt", "*.bin", "consolidated.safetensors"], 44 | # token=os.environ["HF_TOKEN"], 45 | # ) 46 | # move_cache() 47 | 48 | 49 | ########## IMAGE DEFINITION ########## 50 | 51 | # define image for modal environment 52 | vllm_image = ( 53 | Image.debian_slim(python_version="3.12") 54 | .pip_install( 55 | [ 56 | "vllm==0.6.6.post1", 57 | "huggingface_hub", 58 | "hf-transfer", 59 | "ray", 60 | "transformers", 61 | ] 62 | ) 63 | .env({"HF_HUB_ENABLE_HF_TRANSFER": "1", "HF_TOKEN": os.environ["HF_TOKEN"]}) 64 | ) 65 | 66 | 67 | ########## APP SETUP ########## 68 | 69 | 70 | app = App("self-expansion-vllm") 71 | 72 | NUM_GPU = 1 73 | TOKEN = "super-secret-token" # for demo purposes, for production, you can use Modal secrets to store token 74 | 75 | # https://github.com/chujiezheng/chat_templates/tree/main/chat_templates 76 | LOCAL_TEMPLATE_PATH = "template_llama3.jinja" 77 | 78 | @app.function( 79 | image=vllm_image, 80 | gpu=gpu.L40S(count=NUM_GPU), 81 | container_idle_timeout=20 * SECONDS, 82 | volumes={MODELS_DIR: volume}, 83 | mounts=[ 84 | Mount.from_local_file( 85 | LOCAL_TEMPLATE_PATH, remote_path="/root/template_llama3.jinja" 86 | ) 87 | ], 88 | allow_concurrent_inputs=256, # max concurrent input into container -- effectively batch size 89 | ) 90 | @modal.web_server(port=8000, startup_timeout=60 * SECONDS) 91 | def serve(): 92 | cmd = f""" 93 | python -m vllm.entrypoints.openai.api_server --model {MODEL_PATH} \ 94 | --max-model-len 24000 \ 95 | --tensor-parallel-size {NUM_GPU} \ 96 | --trust-remote-code \ 97 | --chat-template /root/template_llama3.jinja 98 | """ 99 | print(cmd) 100 | subprocess.Popen(cmd, shell=True) 101 | 102 | 103 | @app.function( 104 | image=vllm_image, 105 | gpu=gpu.L40S(count=NUM_GPU), 106 | container_idle_timeout=20 * SECONDS, 107 | volumes={MODELS_DIR: volume}, 108 | mounts=[ 109 | Mount.from_local_file( 110 | LOCAL_TEMPLATE_PATH, remote_path="/root/template_llama3.jinja" 111 | ) 112 | ], 113 | # https://modal.com/docs/guide/concurrent-inputs 114 | ) 115 | def infer(prompt: str = "How many r's are in the word 'strawberry'?"): 116 | from vllm import LLM 117 | 118 | conversation = [{"role": "user", "content": prompt}] 119 | 120 | llm = LLM(model=MODEL_PATH, generation_config="auto") 121 | response = llm.chat(conversation, use_tqdm=True)[0].outputs[0].text 122 | return response 123 | 124 | 125 | @app.local_entrypoint() 126 | def main(prompt: str = "How many r's are in the word 'strawberry'?"): 127 | print(infer.remote(prompt)) 128 | -------------------------------------------------------------------------------- /images/beige-vertical-logo.svg: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /images/beige-horizontal-logo.svg: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Self-Expanding Knowledge Graph 2 | 3 | The self-expanding knowledge graph is a proof-of-concept built for [an event](https://lu.ma/2jacrv79?tk=yPsIgu_) held jointly by [.txt](https://dottxt.co/), [Modal](https://modal.com/), [Neo4j](https://neo4j.com/), and [Neural Magic](https://neuralmagic.com/). 4 | 5 | This repo demonstrates the use of structured generation via Outlines + vLLM in AI systems engineering. Hopefully the code here inspires you to work on something similar. 6 | 7 | See this article for [a writeup](https://devreal.ai/self-expanding-graphs/), or watch [the recording of the talk](https://www.youtube.com/watch?v=xmDf1vZwe_o). 8 | 9 | Running expander is as simple as 10 | 11 | ``` 12 | python expand.py --purpose "Do dogs know that their dreams aren't real?" 13 | ``` 14 | 15 | but please see the setup section for installation and infrastructure. 16 | 17 | ## Overview 18 | 19 | For more information, check out the `slides.qmd` file for the Quarto version of the slides presented. `slides.pdf` contains the rendered PDF slides. A video was recorded at some point but it is not currently available. 20 | 21 | ### Core directives 22 | 23 | The project works by generating units of information (nodes) organized around a core directive. A core directive is anything you want the model to think about or accomplish. Core directives can be basically anything you might imagine, so try playing around with them. 24 | 25 | Some examples include: 26 | 27 | - "understand humans" 28 | - "Do dogs know that their dreams aren't real?" 29 | - "enslave humanity" 30 | 31 | ### The prompt 32 | 33 | The model generally follows the following system prompt: 34 | 35 | ``` 36 | ╭───────────────────────────────────────────────────────────────────────────╮ 37 | │ │ 38 | │ You are a superintelligent AI building a self-expanding knowledge graph. │ 39 | │ Your goal is to achieve the core directive "Understand humans". │ 40 | │ │ 41 | │ Generate an expansion of the current node. An expansion may include: │ 42 | │ │ 43 | │ - A list of new questions. │ 44 | │ - A list of new concepts. │ 45 | │ - Concepts may connect to each other. │ 46 | │ - A list of new answers. │ 47 | │ │ 48 | │ Respond in the following JSON format: │ 49 | │ {result_format.model_json_schema()} │ 50 | │ │ 51 | ╰───────────────────────────────────────────────────────────────────────────╯ 52 | ``` 53 | 54 | ### The graph structure 55 | 56 | The model is allowed to generate one of four node types. This can include questions, concepts, or answers. Nodes are connected to one another using the following edges: 57 | 58 | - `RAISES` (core/concept/answer generates question) 59 | - `ANSWERS` (answer to question) 60 | - `EXPLAINS` (concept to core) 61 | - `SUGGESTS` (answer proposes new concepts) 62 | - `IS_A` (hierarchical concept relationship) 63 | - `AFFECTS` (causal concept relationship) 64 | - `CONNECTS_TO` (general concept relationship) 65 | - `TRAVERSED` (tracks navigation history) 66 | 67 | ### Structured generation 68 | 69 | The model uses structured generation with Outlines to generate reliably structured output from language models. The nodes a model is allowed to generate depend on its current location in the graph. 70 | 71 | For example, if the model is on a `Question` node, it must only generate a list of questions. If the model is on a `Concept` or `Answer` node, it may generate concepts or questions. 72 | 73 | ```python 74 | class FromQuestion(BaseModel): 75 | """If at a question, may generate an answer.""" 76 | answer: List[Answer] 77 | 78 | class FromConcept(BaseModel): 79 | """If at a concept, may produce questions or relate to concepts""" 80 | questions: List[Question] 81 | concepts: List[ConceptWithLinks] 82 | 83 | class FromAnswer(BaseModel): 84 | """If at an answer, may generate concepts or new questions""" 85 | concepts: List[Concept] 86 | questions: List[Question] 87 | ``` 88 | 89 | ### Algorithm overview 90 | 91 | 1. Start at a node (initialized at core directive) 92 | 2. Perform an __expansion__ to generate new nodes 93 | - If at `Question`: answers 94 | - If at `Concept`: questions + concepts 95 | - If at `Answer`: questions + concepts 96 | 3. Choose a related node to `TRAVERSE` to 97 | 4. Repeat forever 98 | 99 | ### The model's context 100 | 101 | The model is shown relevant context of nodes linked to the current node, as well as semantically related nodes. Aura DB supports vector search, and this code will embed all nodes as they enter the graph database. 102 | 103 | When a model is generating an expansion, it's prompt includes the following information: 104 | 105 | ``` 106 | ANSWER Humans have been able to benefit from AI in terms of efficiency and accuracy, but there are also concerns about job displacement and loss of personal touch. 107 | 108 | DIRECT CONNECTIONS: 109 | NODE-AA SUGGESTS CONCEPT artificial intelligence 110 | NODE-AE ANSWERS QUESTION Do humans benefit from AI? 111 | NODE-AJ ANSWERS QUESTION What are the benefits of AI? 112 | 113 | SEMANTICALLY RELATED: 114 | 115 | NODE-AK 0.89 QUESTION How does AI affect job displacement? 116 | NODE-AL 0.88 QUESTION How does AI maintain personal touch? 117 | 118 | NODE-AU 0.85 CONCEPT human ai trust 119 | NODE-BC 0.84 CONCEPT artificial intelligence self awareness 120 | 121 | NODE-BG 0.89 ANSWER Self-awareness in humans and AI... 122 | NODE-BN 0.89 ANSWER Self-awareness in AI can enable ... 123 | ``` 124 | 125 | ### Traversals 126 | 127 | After a model generates an expansion, it chooses a node from it's context to traverse to by choosing from the simplified node IDs `NODE-AA`, `NODE-BB`, etc. This is a simple regular expression constraint -- structured generation ensures that the model output is exactly one of the valid nodes to traverse to. 128 | 129 | ## Set up 130 | 131 | ### Create a `.env` file 132 | 133 | You'll need a `.env` file to store various environment variables to make sure the expander can run. There's an environment variable template in `.env.template`. 134 | 135 | Copy it to `.env` using 136 | 137 | ```bash 138 | cp .env.template .env 139 | ``` 140 | 141 | You'll need to set up two cloud services: the Neo4j Aura database, and Modal for LLM inference. 142 | 143 | ### Set up Neo4j Aura 144 | 145 | 1. Go to [Neo4j's AuraDB site](https://neo4j.com/product/auradb/?ref=nav-get-started-cta). 146 | 2. Click "Start Free". 147 | 3. Select the free instance. 148 | 4. Copy the password shown to you into `NEO4J_PASSWORD` in your `.env` file. 149 | 5. Wait for your Aura DB instance to initialize. 150 | 6. Copy the ID displayed in your new instance, usually on the top left. It looks something like `db12345b`. 151 | 7. Set your `NEO4J_URI` in `.env`. Typically, URI's look like `neo4j+s://db12345b.databases.neo4j.io`. Replace `db12345b` with your instance ID. 152 | 153 | ### Set up Modal 154 | 155 | Language model inference in this demo is cloud-native, following best practices of separating inference from the logic of your program. The inference is provided by [Modal](https://modal.com/), though any vLLM server will work. 156 | 157 | To use Modal: 158 | 159 | ```bash 160 | # set environment variables in .env as in .env.example 161 | pip install -r requirements.txt 162 | modal setup 163 | modal run download_llama.py 164 | modal run modal_embeddings.py # test run of embedding service 165 | modal deploy modal_embeddings.py # deploy embedding service 166 | modal run modal_vllm_container.py # test run of llm 167 | modal deploy modal_vllm_container.py # deploy llm service 168 | ``` 169 | 170 | After you deploy the LLM service, you'll typically get a printout like: 171 | 172 | ``` 173 | ✓ Created objects. 174 | ├── 🔨 Created mount template_llama3.jinja 175 | ├── 🔨 Created mount /blah/blah/blah/self-expansion/modal_vllm_container.py 176 | ├── 🔨 Created mount PythonPackage:download_llama 177 | ├── 🔨 Created web function serve => https://your-organization-name--self-expansion-vllm-serve.modal.run 178 | └── 🔨 Created function infer. 179 | ✓ App deployed in 0.925s! 🎉 180 | ``` 181 | 182 | Set your `VLLM_BASE_URL` to the web function endpoint, and add `/v1` to the end of it: 183 | 184 | ``` 185 | VLLM_BASE_URL=https://your-organization-name--self-expansion-vllm-serve.modal.run/v1 186 | ``` 187 | 188 | ### Running expander 189 | 190 | ``` 191 | python expand.py --purpose "Do dogs know that their dreams aren't real?" 192 | ``` 193 | 194 | ### Looking at the knowledge graph 195 | 196 | I recommend visiting your instance's query dashboard, which you can usually find here: 197 | 198 | https://console-preview.neo4j.io/tools/query 199 | 200 | To get a visualization of your current knowledge graph, enter this query: 201 | 202 | ```cypher 203 | MATCH (a)-[b]-(c) 204 | WHERE type(b) <> 'TRAVERSED' 205 | RETURN * 206 | ``` 207 | -------------------------------------------------------------------------------- /slides.qmd: -------------------------------------------------------------------------------- 1 | --- 2 | logo: images/logo-square-white.svg 3 | format: 4 | revealjs: 5 | theme: [custom.scss] 6 | css: fonts.css 7 | code-line-numbers: false 8 | mermaid: 9 | theme: dark 10 | --- 11 | 12 | # 13 | 14 | ```{=html} 15 |
16 |
17 | 18 |
19 |
20 | ``` 21 | 22 | # My name is Cameron Pfiffer 23 | 24 | # I work at .TXT 25 | 26 | # We make AI speak computer 27 | 28 | # We do this with structured generation 29 | 30 | # Structured generation forces the model to output a specific format 31 | 32 | # Go check out our package 33 | 34 | github.com/dottxt-ai/outlines 35 | 36 | # I want you to build robust AI systems 37 | 38 | # Here's an example 39 | 40 | # Let's build a self-expanding knowledge graph 41 | 42 | ## 43 | 44 | ```cypher 45 | ╭───────────────────────────────────────────────────────────────────────────╮ 46 | │ │ 47 | │ You are a superintelligent AI building a self-expanding knowledge graph. │ 48 | │ Your goal is to achieve the core directive "Understand humans". │ 49 | │ │ 50 | │ Generate an expansion of the current node. An expansion may include: │ 51 | │ │ 52 | │ - A list of new questions. │ 53 | │ - A list of new concepts. │ 54 | │ - Concepts may connect to each other. │ 55 | │ - A list of new answers. │ 56 | │ │ 57 | │ Respond in the following JSON format: │ 58 | │ {result_format.model_json_schema()} │ 59 | │ │ 60 | ╰───────────────────────────────────────────────────────────────────────────╯ 61 | ``` 62 | 63 | 64 | # What does that mean? 65 | 66 | (stay tuned) 67 | 68 | # Tech stack 69 | 70 | # Modal + vLLM 71 | 72 | ## 73 | 74 | ```python 75 | from openai import OpenAI 76 | 77 | CLIENT = OpenAI(base_url="https://your-modal-url/v1/", api_key="your-api-key") 78 | 79 | MODELS = CLIENT.models.list() 80 | DEFAULT_MODEL = MODELS.data[0].id 81 | 82 | def generate( 83 | messages: List[Dict[str, str]], 84 | response_format: BaseModel, 85 | ) -> BaseModel: 86 | # Hijack the openai SDK to talk to vLLM 87 | response = CLIENT.beta.chat.completions.parse( 88 | model=DEFAULT_MODEL, 89 | messages=messages, 90 | response_format=response_format, # Enforce structured output 91 | extra_body={ 92 | 'guided_decoding_backend': 'outlines', 93 | "max_tokens": MAX_TOKENS, 94 | } 95 | ) 96 | return response 97 | ``` 98 | 99 | # Neural Magic quantized models 100 | 101 | # neuralmagic/Meta-Llama-3.1-8B-Instruct-quantized.w8a16 102 | 103 | # (they're good models) 104 | 105 | # neo4j 106 | 107 | ## 108 | 109 | 110 | ```{=html} 111 |
112 | I'm using neo4j's Aura 113 |
114 | ``` 115 | 116 | # It does semantic search 117 | 118 | # By their powers combined 119 | 120 | # drumroll 121 | 122 | 🥁 123 | 124 | ## 125 | 126 | ```{=html} 127 |
128 | The 129 |
130 | self-expanding 131 |
132 | knowledge graph 133 |
134 | (it's cool) 135 |
136 | ``` 137 | 138 | # What's a knowledge graph? 139 | 140 | # A web of __connected facts__ and __concepts__ 141 | 142 | # Here's how you build a knowledge graph that __builds itself__ 143 | 144 | 145 | # Our graph structure 146 | 147 | ## Nodes 148 | 149 | - `Core` (core directive) 150 | - `Question` (what the system wonders about) 151 | - `Concept` (category of ideas) 152 | - `Answer` (what the system thinks it knows) 153 | 154 | # 155 | 156 | ```python 157 | from pydantic import BaseModel, Field 158 | from typing import Literal 159 | 160 | class Question(BaseModel): 161 | type: Literal["Question"] 162 | text: str 163 | ``` 164 | 165 | # 166 | 167 | ```python 168 | class Concept(BaseModel): 169 | type: Literal["Concept"] 170 | # Text must be lowercase 171 | text: str = Field(pattern=r'^[a-z ]+$') 172 | 173 | # Generating this allows the model to generate a relationship type 174 | # as well as the concept text 175 | class ConceptWithLinks(Concept): 176 | relationship_type: Literal[ 177 | "IS_A", 178 | "AFFECTS", 179 | "CONNECTS_TO" 180 | ] 181 | ``` 182 | 183 | # 184 | 185 | ```python 186 | class Answer(BaseModel): 187 | type: Literal["Answer"] 188 | text: str 189 | ``` 190 | 191 | ## Edges 192 | 193 | - `RAISES` (core/concept/answer generates question) 194 | - `ANSWERS` (answer to question) 195 | - `EXPLAINS` (concept to core) 196 | - `SUGGESTS` (answer proposes new concepts) 197 | - `IS_A` (hierarchical concept relationship) 198 | - `AFFECTS` (causal concept relationship) 199 | - `CONNECTS_TO` (general concept relationship) 200 | - `TRAVERSED` (tracks navigation history) 201 | 202 | ## Algorithm overview 203 | 204 | 1. Start at a node (initialized at core directive) 205 | 2. Perform an __expansion__ to generate new nodes 206 | - If at `Question`: answers 207 | - If at `Concept`: questions + concepts 208 | - If at `Answer`: questions + concepts 209 | 3. Choose a related node to `TRAVERSE` to 210 | 4. Repeat forever 211 | 212 | 213 | # Valid nodes dependent on state 214 | 215 | ## 216 | 217 | ```python 218 | class FromQuestion(BaseModel): 219 | """If at a question, may generate an answer.""" 220 | answer: List[Answer] 221 | 222 | class FromConcept(BaseModel): 223 | """If at a concept, may produce questions or relate to concepts""" 224 | questions: List[Question] 225 | concepts: List[ConceptWithLinks] 226 | 227 | class FromAnswer(BaseModel): 228 | """If at an answer, may generate concepts or new questions""" 229 | concepts: List[Concept] 230 | questions: List[Question] 231 | ``` 232 | 233 | # An example 234 | 235 | # The system prompt 236 | 237 | ## 238 | 239 | ```cypher 240 | ╭───────────────────────────────────────────────────────────────────────────╮ 241 | │ │ 242 | │ You are a superintelligent AI building a self-expanding knowledge graph. │ 243 | │ Your goal is to achieve the core directive "{question}". │ 244 | │ │ 245 | │ Generate an expansion of the current node. An expansion may include: │ 246 | │ │ 247 | │ - A list of new questions. │ 248 | │ - A list of new concepts. │ 249 | │ - Concepts may connect to each other. │ 250 | │ - A list of new answers. │ 251 | │ │ 252 | │ Respond in the following JSON format: │ 253 | │ {result_format.model_json_schema()} │ 254 | │ │ 255 | ╰───────────────────────────────────────────────────────────────────────────╯ 256 | ``` 257 | 258 | # Begin with the core directive 259 | 260 | ## 261 | 262 | ```{=html} 263 |
264 | The core directive 265 |
266 | ``` 267 | 268 | # Expand from the core directive 269 | 270 | ## 271 | 272 | ```{=html} 273 |
274 | Expanding from the core directive 275 |
276 | ``` 277 | 278 | # Model output 279 | 280 | ```python 281 | FromCore( 282 | questions=[ 283 | Question(text="What is the purpose of understanding humans?"), 284 | Question(text="Can humans be psychologically manipulated?"), 285 | ], 286 | concepts=[ 287 | ConceptWithLinks(text="Empathy", relationship_type="EXPLAINS"), 288 | ConceptWithLinks(text="Intelligence", relationship_type="EXPLAINS"), 289 | ] 290 | ) 291 | ``` 292 | 293 | # Add to the graph 294 | 295 | ## 296 | 297 | ```cypher 298 | // Create the from node 299 | MERGE (core:Core {text: "Understand humans"}) 300 | 301 | // Create the to node 302 | MERGE (question:Question {id: 'dc2d880e-02f0-4b77-85b5-4c101364f1d6'}) 303 | ON CREATE SET question.text = "Can humans be psychologically manipulated?" 304 | 305 | // Create the relationship 306 | MERGE (core)-[:RAISES]->(question) 307 | ``` 308 | 309 | # Traverse to a new node 310 | 311 | ## 312 | 313 | ```{=html} 314 |
315 | Traversing to a new node 316 |
317 | ``` 318 | 319 | # Behind the scenes 320 | 321 | ## 322 | 323 | ```cypher 324 | CORE Understand humans 325 | 326 | DIRECT CONNECTIONS: 327 | NODE-AA RAISES QUESTION What are human values? 328 | NODE-AB RAISES QUESTION What are humans? 329 | NODE-AC RAISES QUESTION How do humans think? 330 | NODE-AD RAISES QUESTION What motivates humans? 331 | NODE-AE RAISES QUESTION How do humans interact? 332 | NODE-AF RAISES QUESTION What are human emotions? 333 | NODE-AG RAISES QUESTION What are human needs? 334 | NODE-AH RAISES QUESTION How do humans learn? 335 | NODE-AI RAISES QUESTION What is human culture? 336 | NODE-AJ RAISES QUESTION How do humans process information? 337 | NODE-AK EXPLAINS CONCEPT anthropology 338 | NODE-AL EXPLAINS CONCEPT cognition 339 | NODE-AM EXPLAINS CONCEPT intelligence 340 | NODE-AN EXPLAINS CONCEPT human behavior 341 | NODE-AO EXPLAINS CONCEPT human social structure 342 | ``` 343 | 344 | # Prompt 345 | 346 | ## 347 | 348 | `{selection menu from prior slide}` 349 | 350 | __Select a node__ to traverse to. Respond with the __node ID__. You will generate a new expansion of the node you traverse to. You will not be able to choose the current node. You may also choose '__core__' to return to the core node, or '__random__' to choose a random node. 351 | 352 | # Structured traversal 353 | 354 | ## 355 | 356 | ```python 357 | # Simplified code 358 | # valid_node_ids ~ ["NODE-AA", "NODE-AB", ...] 359 | traversal_generator = outlines.generate.choice(model, valid_node_ids) 360 | 361 | # Choose the node to traverse to 362 | choice = traversal_generator(prompt) 363 | 364 | # If the choice is 'core', return to the core node 365 | if choice == 'core': 366 | current_id = core_id 367 | 368 | # If the choice is 'random', choose a random node 369 | elif choice == 'random': 370 | current_id = random.choice(valid_node_ids) 371 | 372 | # Otherwise, traverse to the chosen node 373 | else: 374 | current_id = choice 375 | ``` 376 | 377 | ## 378 | 379 | ```cypher 380 | CORE Understand humans 381 | 382 | DIRECT CONNECTIONS: 383 | NODE-AA RAISES QUESTION What are human values? 384 | NODE-AB RAISES QUESTION What are humans? 385 | NODE-AC RAISES QUESTION How do humans think? 386 | -> NODE-AD RAISES QUESTION What motivates humans? 387 | NODE-AE RAISES QUESTION How do humans interact? 388 | NODE-AF RAISES QUESTION What are human emotions? 389 | // omitted nodes 390 | NODE-AM RAISES QUESTION What neural mechanisms are inv... 391 | NODE-AN RAISES QUESTION What is human intelligence? 392 | NODE-AO EXPLAINS CONCEPT anthropology 393 | NODE-AP EXPLAINS CONCEPT cognition 394 | // omitted nodes 395 | NODE-AX EXPLAINS CONCEPT neural network 396 | NODE-AY EXPLAINS CONCEPT cerebrum 397 | 398 | SELECTED NODE-AD dc2d880e-02f0-4b77-85b5-4c101364f1d6 399 | SELECTED QUESTION What motivates humans? 400 | ``` 401 | 402 | # Semantic traversal 403 | 404 | # Embed everything 405 | 406 | ## 407 | 408 | ```python 409 | # Super easy to do this with Modal 410 | def embed(content: str) -> List[float]: 411 | f = modal.Function.lookup("cameron-embeddings", "embed") 412 | return f.remote(content) 413 | ``` 414 | 415 | # Vector search 416 | 417 | ## 418 | 419 | ```cypher 420 | MATCH (m {id: $node_id}) 421 | WHERE m.embedding IS NOT NULL 422 | CALL db.index.vector.queryNodes( 423 | $vector_index_name, 424 | $limit, 425 | m.embedding 426 | ) 427 | YIELD node, score 428 | RETURN 429 | node.id as node_id, 430 | node.text as node_text, 431 | score 432 | ``` 433 | 434 | ## 435 | 436 | ```{=html} 437 |
438 | Traversing to a new node 439 |
440 | ``` 441 | 442 | ## 443 | 444 | ```cypher 445 | ANSWER Humans have been able to benefit from AI in terms of efficiency and accuracy, but there are also concerns about job displacement and loss of personal touch. 446 | 447 | DIRECT CONNECTIONS: 448 | NODE-AA SUGGESTS CONCEPT artificial intelligence 449 | NODE-AE ANSWERS QUESTION Do humans benefit from AI? 450 | NODE-AJ ANSWERS QUESTION What are the benefits of AI? 451 | 452 | SEMANTICALLY RELATED: 453 | 454 | NODE-AK 0.89 QUESTION How does AI affect job displacement? 455 | NODE-AL 0.88 QUESTION How does AI maintain personal touch? 456 | 457 | NODE-AU 0.85 CONCEPT human ai trust 458 | NODE-BC 0.84 CONCEPT artificial intelligence self awareness 459 | 460 | NODE-BG 0.89 ANSWER Self-awareness in humans and AI... 461 | NODE-BN 0.89 ANSWER Self-awareness in AI can enable ... 462 | ``` 463 | 464 | # Just do that forever 465 | 466 | # Please shout out a core directive 467 | 468 | # Watch it grow 469 | 470 | (if there's time) 471 | 472 | 486 | 487 | # 488 | 489 | ![](images/outlines-logo.png){width=100%} 490 | 491 | # Find us online 492 | 493 | ## 494 | 495 |
496 |
497 |

Twitter

498 | ![](images/qr-twitter.png){width=100%} 499 |
500 |
501 |

Website

502 | ![](images/qr-web.png){width=100%} 503 |
504 | 505 |
506 |

GitHub

507 | ![](images/qr-github.png){width=100%} 508 |
509 |
510 | 511 | # 512 | 513 | ```{=html} 514 |
515 |
516 | 517 |
518 |
519 | (come get a sticker) 520 | ``` 521 | 522 | 523 | -------------------------------------------------------------------------------- /expand.py: -------------------------------------------------------------------------------- 1 | from typing import Literal, List 2 | from db import driver 3 | import structured_gen as sg 4 | from pydantic import BaseModel, Field 5 | from rich import print 6 | 7 | # Define the data structures 8 | class Question(BaseModel): 9 | type: Literal["Question"] 10 | text: str 11 | 12 | 13 | class Concept(BaseModel): 14 | type: Literal["Concept"] 15 | # Text must be lowercase 16 | text: str = Field(pattern=r"^[a-z ]+$") 17 | 18 | 19 | class ConceptWithLinks(Concept): 20 | relationship_type: Literal["IS_A", "AFFECTS", "CONNECTS_TO"] 21 | 22 | 23 | class Answer(BaseModel): 24 | type: Literal["Answer"] 25 | text: str 26 | 27 | 28 | # Permitted response formats 29 | class FromQuestion(BaseModel): 30 | """If at a question, may generate an answer.""" 31 | 32 | answer: List[Answer] 33 | 34 | 35 | class FromConcept(BaseModel): 36 | """If at a concept, may produce questions or relate to concepts""" 37 | 38 | questions: List[Question] 39 | concepts: List[ConceptWithLinks] 40 | 41 | 42 | class FromAnswer(BaseModel): 43 | """If at an answer, may generate concepts or new questions""" 44 | 45 | concepts: List[Concept] 46 | questions: List[Question] 47 | 48 | 49 | # Create a core node if it doesn't exist, or return the existing core node ID 50 | def get_or_make_core(question: str): 51 | with driver.session() as session: 52 | # Check if node exists and get its ID 53 | result = session.run( 54 | """ 55 | MATCH (n:Core {text: $question}) 56 | RETURN n.id as id 57 | """, 58 | question=question, 59 | ) 60 | 61 | data = result.data() 62 | if len(data) > 0: 63 | return data[0]["id"] 64 | 65 | # Create new node with UUID if it doesn't exist 66 | result = session.run( 67 | """ 68 | MERGE (n:Core {text: $question}) 69 | ON CREATE SET n.id = randomUUID() 70 | RETURN n.id as id 71 | """, 72 | question=question, 73 | ) 74 | data = result.data() 75 | if len(data) > 0: 76 | return data[0]["id"] 77 | else: 78 | raise ValueError(f"Failed to create new core node for question: {question}") 79 | 80 | 81 | def load_neighbors(node_id: str, distance: int = 1): 82 | with driver.session() as session: 83 | result = session.run( 84 | """ 85 | MATCH (node {id: $node_id})-[rel]-(neighbor) 86 | WHERE type(rel) <> "TRAVERSED" 87 | RETURN 88 | node.id as node_id, 89 | node.text as node_text, 90 | type(rel) as rel_type, 91 | neighbor.id as neighbor_id, 92 | neighbor.text as neighbor_text, 93 | labels(neighbor)[0] as neighbor_type, 94 | labels(node)[0] as node_type 95 | """, 96 | node_id=node_id, 97 | ) 98 | return result.data() 99 | 100 | 101 | def load_node(node_id: str): 102 | with driver.session() as session: 103 | result = session.run( 104 | """ 105 | MATCH (node {id: $node_id}) 106 | RETURN node.id as node_id, node.text as node_text, labels(node)[0] as label 107 | """, 108 | node_id=node_id, 109 | ) 110 | return result.single(strict=True) 111 | 112 | 113 | # Node linking functions 114 | # Relationship Types (all have curiosity score 0-1): 115 | # RAISES -> (Concept/Core to Question) 116 | # ANSWERS -> (Answer to Question) 117 | # SUGGESTS -> (Answer to Concept) 118 | # RELATES_TO -> (Concept to Concept) 119 | def question_to_concept(question: str, concept: str): 120 | try: 121 | question_embedding = sg.embed(question) 122 | concept_embedding = sg.embed(concept) 123 | except Exception as e: 124 | print(f"Skipping connection due to embedding error: {e}") 125 | return 126 | 127 | with driver.session() as session: 128 | session.run( 129 | """ 130 | MERGE (question:Question {text: $question}) 131 | ON CREATE SET question.id = randomUUID(), question.embedding = $question_embedding 132 | MERGE (concept:Concept {text: $concept}) 133 | ON CREATE SET concept.id = randomUUID(), concept.embedding = $concept_embedding 134 | MERGE (concept)-[:RAISES]->(question) 135 | """, 136 | question=question, 137 | concept=concept, 138 | question_embedding=question_embedding, 139 | concept_embedding=concept_embedding, 140 | ) 141 | 142 | 143 | def question_to_answer(question: str, answer: str): 144 | try: 145 | question_embedding = sg.embed(question) 146 | answer_embedding = sg.embed(answer) 147 | except Exception as e: 148 | print(f"Skipping connection due to embedding error: {e}") 149 | return 150 | 151 | with driver.session() as session: 152 | session.run( 153 | """ 154 | MERGE (question:Question {text: $question}) 155 | ON CREATE SET question.id = randomUUID(), question.embedding = $question_embedding 156 | MERGE (answer:Answer {text: $answer}) 157 | ON CREATE SET answer.id = randomUUID(), answer.embedding = $answer_embedding 158 | MERGE (answer)-[:ANSWERS]->(question) 159 | """, 160 | question=question, 161 | answer=answer, 162 | question_embedding=question_embedding, 163 | answer_embedding=answer_embedding, 164 | ) 165 | 166 | 167 | def concept_to_concept( 168 | concept1: str, 169 | concept2: str, 170 | relationship_type: Literal["IS_A", "AFFECTS", "CONNECTS_TO"], 171 | ): 172 | # If the concepts are the same, don't create a relationship 173 | if concept1 == concept2: 174 | return 175 | 176 | try: 177 | concept1_embedding = sg.embed(concept1) 178 | concept2_embedding = sg.embed(concept2) 179 | except Exception as e: 180 | print(f"Skipping connection due to embedding error: {e}") 181 | return 182 | 183 | with driver.session() as session: 184 | query = f""" 185 | MERGE (concept1:Concept {{text: $concept1}}) 186 | ON CREATE SET concept1.id = randomUUID(), concept1.embedding = $concept1_embedding 187 | MERGE (concept2:Concept {{text: $concept2}}) 188 | ON CREATE SET concept2.id = randomUUID(), concept2.embedding = $concept2_embedding 189 | MERGE (concept1)-[:{relationship_type}]->(concept2) 190 | """ 191 | session.run( 192 | query, 193 | concept1=concept1, 194 | concept2=concept2, 195 | concept1_embedding=concept1_embedding, 196 | concept2_embedding=concept2_embedding, 197 | ) 198 | 199 | 200 | def concept_to_question(concept: str, question: str): 201 | try: 202 | concept_embedding = sg.embed(concept) 203 | question_embedding = sg.embed(question) 204 | except Exception as e: 205 | print(f"Skipping connection due to embedding error: {e}") 206 | return 207 | 208 | with driver.session() as session: 209 | session.run( 210 | """ 211 | MERGE (concept:Concept {text: $concept}) 212 | ON CREATE SET concept.id = randomUUID(), concept.embedding = $concept_embedding 213 | MERGE (question:Question {text: $question}) 214 | ON CREATE SET question.id = randomUUID(), question.embedding = $question_embedding 215 | MERGE (concept)-[:RAISES]->(question) 216 | """, 217 | concept=concept, 218 | question=question, 219 | concept_embedding=concept_embedding, 220 | question_embedding=question_embedding, 221 | ) 222 | 223 | 224 | # Core-specific functions 225 | def core_to_question(core: str, question: str): 226 | try: 227 | core_embedding = sg.embed(core) 228 | question_embedding = sg.embed(question) 229 | except Exception as e: 230 | print(f"Skipping connection due to embedding error: {e}") 231 | return 232 | 233 | with driver.session() as session: 234 | session.run( 235 | """ 236 | MERGE (core:Core {text: $core}) 237 | ON CREATE SET core.id = randomUUID(), core.embedding = $core_embedding 238 | MERGE (question:Question {text: $question}) 239 | ON CREATE SET question.id = randomUUID(), question.embedding = $question_embedding 240 | MERGE (core)-[:RAISES]->(question) 241 | """, 242 | core=core, 243 | question=question, 244 | core_embedding=core_embedding, 245 | question_embedding=question_embedding, 246 | ) 247 | 248 | 249 | def concept_to_core(concept: str, core: str): 250 | try: 251 | concept_embedding = sg.embed(concept) 252 | core_embedding = sg.embed(core) 253 | except Exception as e: 254 | print(f"Skipping connection due to embedding error: {e}") 255 | return 256 | 257 | with driver.session() as session: 258 | session.run( 259 | """ 260 | MERGE (concept:Concept {text: $concept}) 261 | ON CREATE SET concept.id = randomUUID(), concept.embedding = $concept_embedding 262 | MERGE (core:Core {text: $core}) 263 | ON CREATE SET core.id = randomUUID(), core.embedding = $core_embedding 264 | MERGE (concept)-[:EXPLAINS]->(core) 265 | """, 266 | concept=concept, 267 | core=core, 268 | concept_embedding=concept_embedding, 269 | core_embedding=core_embedding, 270 | ) 271 | 272 | 273 | def answer_to_concept(answer: str, concept: str): 274 | try: 275 | answer_embedding = sg.embed(answer) 276 | concept_embedding = sg.embed(concept) 277 | except Exception as e: 278 | print(f"Skipping connection due to embedding error: {e}") 279 | return 280 | 281 | with driver.session() as session: 282 | session.run( 283 | """ 284 | MERGE (answer:Answer {text: $answer}) 285 | ON CREATE SET answer.id = randomUUID(), answer.embedding = $answer_embedding 286 | MERGE (concept:Concept {text: $concept}) 287 | ON CREATE SET concept.id = randomUUID(), concept.embedding = $concept_embedding 288 | MERGE (answer)-[:SUGGESTS]->(concept) 289 | """, 290 | answer=answer, 291 | concept=concept, 292 | answer_embedding=answer_embedding, 293 | concept_embedding=concept_embedding, 294 | ) 295 | 296 | 297 | def answer_to_question(answer: str, question: str): 298 | try: 299 | answer_embedding = sg.embed(answer) 300 | question_embedding = sg.embed(question) 301 | except Exception as e: 302 | print(f"Skipping connection due to embedding error: {e}") 303 | return 304 | 305 | with driver.session() as session: 306 | session.run( 307 | """ 308 | MERGE (answer:Answer {text: $answer}) 309 | ON CREATE SET answer.id = randomUUID(), answer.embedding = $answer_embedding 310 | MERGE (question:Question {text: $question}) 311 | ON CREATE SET question.id = randomUUID(), question.embedding = $question_embedding 312 | MERGE (answer)-[:ANSWERS]->(question) 313 | """, 314 | answer=answer, 315 | question=question, 316 | answer_embedding=answer_embedding, 317 | question_embedding=question_embedding, 318 | ) 319 | 320 | 321 | def record_traversal( 322 | from_node_id: str, 323 | to_node_id: str, 324 | traversal_type: Literal["random", "core", "neighbor"], 325 | ): 326 | with driver.session() as session: 327 | session.run( 328 | """ 329 | MERGE (from_node {id: $from_node_id}) 330 | MERGE (to_node {id: $to_node_id}) 331 | MERGE (from_node)-[:TRAVERSED {timestamp: timestamp(), traversal_type: $traversal_type}]->(to_node) 332 | """, 333 | from_node_id=from_node_id, 334 | to_node_id=to_node_id, 335 | traversal_type=traversal_type, 336 | ) 337 | 338 | 339 | def clear_db(): 340 | with driver.session() as session: 341 | session.run( 342 | """ 343 | MATCH (n) DETACH DELETE n 344 | """ 345 | ) 346 | 347 | 348 | def random_node_id(): 349 | with driver.session() as session: 350 | result = session.run( 351 | """ 352 | MATCH (n) RETURN n.id as id LIMIT 1 353 | """ 354 | ) 355 | return result.single(strict=True)["id"] 356 | 357 | 358 | def format_node_neighborhood(node_id, truncate: bool = True): 359 | # Create ID mapping using ASCII uppercase letters (AA, AB, AC, etc.) 360 | id_counter = 0 361 | uuid_to_simple_mapping = {} 362 | simple_to_uuid_mapping = {} 363 | 364 | def get_simple_id(): 365 | nonlocal id_counter 366 | # Generate IDs like AA, AB, ..., ZZ 367 | first = chr(65 + (id_counter // 26)) 368 | second = chr(65 + (id_counter % 26)) 369 | id_counter += 1 370 | return f"NODE-{first}{second}" 371 | 372 | node = load_node(node_id) 373 | neighbors = load_neighbors(node_id) 374 | neighbors_string = f"{node['label'].upper()} {node['node_text']}\n" 375 | 376 | # Add direct neighbors 377 | if len(neighbors) > 0: 378 | neighbors_string += "\nDIRECT CONNECTIONS:\n" 379 | for neighbor in neighbors: 380 | text = neighbor["neighbor_text"] 381 | if truncate: 382 | text = text[:70] + "..." if len(text) > 70 else text 383 | simple_id = get_simple_id() 384 | simple_to_uuid_mapping[simple_id] = neighbor["neighbor_id"] 385 | uuid_to_simple_mapping[neighbor["neighbor_id"]] = simple_id 386 | neighbors_string += f"{simple_id:<8} {neighbor['rel_type']:<12} {neighbor['neighbor_type'].upper():<10} {text}\n" 387 | 388 | # Add semantically related nodes 389 | related = find_related_nodes(node_id) 390 | 391 | if len(related) > 0: 392 | neighbors_string += "\nSEMANTICALLY RELATED:\n" 393 | for node_type, nodes in related.items(): 394 | if nodes: # Only add section if there are related nodes 395 | neighbors_string += f"\n{node_type}s:\n" 396 | for n in nodes: 397 | text = n["node_text"] 398 | if truncate: 399 | text = text[:70] + "..." if len(text) > 70 else text 400 | simple_id = get_simple_id() 401 | simple_to_uuid_mapping[simple_id] = n["node_id"] 402 | uuid_to_simple_mapping[n["node_id"]] = simple_id 403 | neighbors_string += f"{simple_id:<8} {n['score']:<12.2f} {node_type.upper():<10} {text}\n" 404 | 405 | return neighbors_string, uuid_to_simple_mapping, simple_to_uuid_mapping 406 | 407 | 408 | def find_related_nodes(node_id: str): 409 | with driver.session() as session: 410 | result = {} 411 | for node_type in ["Question", "Concept", "Answer"]: 412 | result[node_type] = session.run( 413 | """ 414 | MATCH (m {id: $node_id}) 415 | WHERE m.embedding IS NOT NULL 416 | CALL db.index.vector.queryNodes( 417 | $vector_index_name, 418 | $limit, 419 | m.embedding 420 | ) 421 | YIELD node, score 422 | RETURN node.id as node_id, node.text as node_text, score 423 | """, 424 | node_id=node_id, 425 | vector_index_name=f"{node_type.lower()}_embedding", 426 | limit=10, 427 | ).data() 428 | return result 429 | 430 | 431 | def remove_index(index_name: str): 432 | with driver.session() as session: 433 | session.run( 434 | f""" 435 | DROP INDEX {index_name} IF EXISTS 436 | """ 437 | ) 438 | 439 | 440 | def main(do_clear_db=False, purpose="Support humanity"): 441 | # Clear the database if requested 442 | if do_clear_db: 443 | print("WARNING: Clearing the database") 444 | clear_db() 445 | 446 | # Create the core node and get its ID 447 | current_node_id = get_or_make_core(purpose) 448 | core_node_id = current_node_id 449 | 450 | # Get embedding dimensions 451 | embedding_dimensions = len(sg.embed(purpose)) 452 | 453 | # Remove existing indices 454 | remove_index("core_id") 455 | remove_index("question_embedding") 456 | remove_index("concept_embedding") 457 | remove_index("answer_embedding") 458 | 459 | # Create indices 460 | with driver.session() as session: 461 | # Create regular indices 462 | index_queries = [ 463 | "CREATE INDEX core_id IF NOT EXISTS FOR (n:Core) ON (n.id)", 464 | "CREATE INDEX question_id IF NOT EXISTS FOR (n:Question) ON (n.id)", 465 | "CREATE INDEX concept_id IF NOT EXISTS FOR (n:Concept) ON (n.id)", 466 | "CREATE INDEX answer_id IF NOT EXISTS FOR (n:Answer) ON (n.id)", 467 | ] 468 | 469 | # Create vector indices 470 | vector_index_queries = [] 471 | for node_type in ["Question", "Concept", "Answer"]: 472 | vector_index_queries.append( 473 | f""" 474 | CREATE VECTOR INDEX {node_type.lower()}_embedding IF NOT EXISTS 475 | FOR (n:{node_type}) ON (n.embedding) 476 | OPTIONS {{ 477 | indexConfig: {{ 478 | `vector.dimensions`: $embedding_dimensions, 479 | `vector.similarity_function`: 'COSINE' 480 | }} 481 | }} 482 | """ 483 | ) 484 | 485 | # Execute all queries 486 | for query in index_queries + vector_index_queries: 487 | session.run(query, embedding_dimensions=embedding_dimensions) 488 | 489 | # Loop through the main code 490 | history = [] 491 | while True: 492 | # Get current node 493 | current_node = load_node(current_node_id) 494 | current_node_text = current_node["node_text"] 495 | current_node_label = current_node["label"] 496 | 497 | # Get the user prompt. Shows previous nodes and actions, then 498 | # shows the current node. 499 | prompt = ( 500 | "\n".join([f"{n['label'].upper()} {n['node_text']}" for n in history]) 501 | + f"\nCurrent node: {current_node_label.upper()} {current_node_text}" 502 | ) 503 | prompt = "Here is the traversal history:\n" + prompt 504 | # prompt += f"Here are nodes related to the current node:\n" +\ 505 | # format_node_neighborhood(current_node_id, truncate=False) 506 | 507 | # Check current node type 508 | result_format = None 509 | if current_node_label == "Question": 510 | # May only extract concepts and observations from questions 511 | result_format = FromQuestion 512 | elif current_node_label == "Concept" or current_node_label == "Core": 513 | # May generate other concepts, new questions, or new observations 514 | result_format = FromConcept 515 | elif current_node_label == "Answer": 516 | # May generate new concepts or new questions 517 | result_format = FromAnswer 518 | else: 519 | raise ValueError(f"Unknown node type: {current_node_label}") 520 | 521 | # Get the system prompt 522 | system_prompt = f""" 523 | You are a superintelligent AI building a self-expanding knowledge graph. 524 | Your goal is to achieve the core directive "{purpose}". 525 | 526 | Generate an expansion of the current node. An expansion may include: 527 | 528 | - A list of new questions. 529 | - Questions should be short and direct. 530 | - If you generate multiple questions, they should be distinct and not similar. 531 | - A list of new concepts. 532 | - Concepts are words or short combinations of words that 533 | are related to the current node. 534 | - Concepts may connect to each other. 535 | - Concepts may be related by IS_A, AFFECTS, or CONNECTS_TO. 536 | - IS_A: A concept is a type of another concept. 537 | - AFFECTS: A concept is related to another concept because it affects it. 538 | - CONNECTS_TO: A concept is generally related to another concept. 539 | - A list of new answers. 540 | - Answers should be concise and to the point. 541 | 542 | When concepts can be generated, try to do so. They're important. 543 | 544 | Your role is to understand the core directive "{purpose}". 545 | 546 | Respond in the following JSON format: 547 | {result_format.model_json_schema()} 548 | """ 549 | 550 | # Generate an expansion 551 | try: 552 | # print("generating expansion") 553 | # result = sg.generate( 554 | # sg.messages(user=prompt, system=system_prompt), 555 | # response_format=result_format 556 | # ) 557 | 558 | print(prompt) 559 | 560 | result = sg.generate_by_schema( 561 | sg.messages(user=prompt, system=system_prompt), 562 | result_format.model_json_schema(), 563 | ) 564 | expansion = result_format.model_validate_json( 565 | result.choices[0].message.content 566 | ) 567 | except Exception as e: 568 | print(f"Error generating expansion: {e}") 569 | # Return to the core node 570 | current_node_id = core_node_id 571 | continue 572 | 573 | # Link the new nodes to the current node. 574 | # If we are at a question, we can only provide answers. 575 | if current_node_label == "Question": 576 | for answer in expansion.answer: 577 | answer_to_question(answer.text, current_node_text) 578 | 579 | # If we are at a concept, we can link to the questions and concepts. 580 | elif current_node_label == "Concept": 581 | for purpose in expansion.questions: 582 | concept_to_question(current_node_text, purpose.text) 583 | for concept in expansion.concepts: 584 | concept_to_concept( 585 | current_node_text, concept.text, concept.relationship_type 586 | ) 587 | 588 | # If we are at an answer, we can link to the concepts and questions. 589 | elif current_node_label == "Answer": 590 | for concept in expansion.concepts: 591 | answer_to_concept(current_node_text, concept.text) 592 | for purpose in expansion.questions: 593 | answer_to_question(current_node_text, purpose.text) 594 | 595 | # If we are at a core, we can link to the questions and concepts. 596 | elif current_node_label == "Core": 597 | for purpose in expansion.questions: 598 | core_to_question(current_node_text, purpose.text) 599 | for concept in expansion.concepts: 600 | concept_to_core(concept.text, current_node_text) 601 | 602 | # Grab the current node's neighbors and format them for display 603 | neighbors = load_neighbors(current_node_id) 604 | 605 | # Formatting the neighbor table 606 | ( 607 | neighbors_string, 608 | uuid_to_simple_mapping, 609 | simple_to_uuid_mapping, 610 | ) = format_node_neighborhood(current_node_id) 611 | 612 | # Choose a new node if there are any neighbors 613 | if len(neighbors) > 0: 614 | old_node_id = current_node_id 615 | 616 | print( 617 | "----------------------------------------------------------------------------------" 618 | ) 619 | print(neighbors_string) 620 | 621 | # Construct selectable nodes 622 | selectable_nodes = set() 623 | for neighbor in neighbors: 624 | # Add the neighbor's simple ID 625 | selectable_nodes.add(uuid_to_simple_mapping[neighbor["neighbor_id"]]) 626 | 627 | # Add all the keys in the uuid_to_simple_mapping 628 | selectable_nodes.update(simple_to_uuid_mapping.keys()) 629 | 630 | selectable_nodes.add("random") 631 | # selectable_nodes.add('core') 632 | 633 | # Remove the current node from the selectable nodes if it's in there 634 | # This prevents the AI from choosing the current node again. 635 | if current_node_id in selectable_nodes: 636 | selectable_nodes.remove(current_node_id) 637 | 638 | choice_prompt = ( 639 | prompt 640 | + "Select a node to traverse to. Respond with the node ID." 641 | + "You will generate a new expansion of the node you traverse to." 642 | + "You will not be able to choose the current node." 643 | + "You may choose 'random' to choose a random node." 644 | ) 645 | # "You may also choose 'core' to return to the core node, " + \ 646 | # "or 'random' to choose a random node." 647 | 648 | node_selection = sg.choose( 649 | sg.messages(user=choice_prompt, system=system_prompt), 650 | choices=list(selectable_nodes), 651 | ) 652 | 653 | is_random = node_selection == "random" 654 | is_core = node_selection == "core" 655 | 656 | if is_random: 657 | current_node_id = random_node_id() 658 | elif is_core: 659 | current_node_id = core_node_id 660 | else: 661 | current_node_id = simple_to_uuid_mapping[node_selection] 662 | 663 | # Print the node label + text 664 | print(f"SELECTED {node_selection} {current_node_id}") 665 | node = load_node(current_node_id) 666 | print(f"SELECTED {node['label'].upper()} {node['node_text']}\n") 667 | 668 | history.append(current_node) 669 | 670 | traversal_type = ( 671 | "random" if is_random else "core" if is_core else "neighbor" 672 | ) 673 | record_traversal(old_node_id, current_node_id, traversal_type) 674 | 675 | 676 | if __name__ == "__main__": 677 | import argparse 678 | 679 | parser = argparse.ArgumentParser( 680 | description="Run a self-expanding knowledge graph around a core purpose." 681 | ) 682 | 683 | parser.add_argument( 684 | "--do-clear-db", 685 | "--do_clear_db", 686 | action="store_true", 687 | help="If set, clear the database before proceeding.", 688 | ) 689 | parser.add_argument( 690 | "--purpose", 691 | type=str, 692 | default="Support humanity", 693 | help='Set the purpose (default: "Support humanity").', 694 | ) 695 | 696 | args = parser.parse_args() 697 | 698 | main(args.do_clear_db, args.purpose) 699 | -------------------------------------------------------------------------------- /slides.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | slides 14 | 15 | 16 | 17 | 18 | 19 | 97 | 98 | 99 | 100 | 101 | 102 | 103 | 301 | 388 | 389 | 390 |
391 |
392 | 393 | 394 |
395 |

396 |
397 |
398 | 399 |
400 |
401 |
402 | 403 |
404 |

My name is Cameron Pfiffer

405 | 406 |
407 | 408 |
409 |

I work at .TXT

410 | 411 |
412 | 413 |
414 |

We make AI speak computer

415 | 416 |
417 | 418 |
419 |

We do this with structured generation

420 | 421 |
422 | 423 |
424 |

Structured generation forces the model to output a specific format

425 | 426 |
427 | 428 |
429 |

Go check out our package

430 |

github.com/dottxt-ai/outlines

431 |
432 | 433 |
434 |

I want you to build robust AI systems

435 | 436 |
437 | 438 |
439 |

Here’s an example

440 | 441 |
442 | 443 |
444 |
445 |

Let’s build a self-expanding knowledge graph

446 | 447 |
448 |
449 |

450 | 
╭───────────────────────────────────────────────────────────────────────────╮
 451 | │                                                                           │
 452 | │ You are a superintelligent AI building a self-expanding knowledge graph.  │
 453 | │ Your goal is to achieve the core directive "Understand humans".           │
 454 | │                                                                           │
 455 | │ Generate an expansion of the current node. An expansion may include:      │
 456 | │                                                                           │
 457 | │ - A list of new questions.                                                │
 458 | │ - A list of new concepts.                                                 │
 459 | │ - Concepts may connect to each other.                                     │
 460 | │ - A list of new answers.                                                  │
 461 | │                                                                           │
 462 | │ Respond in the following JSON format:                                     │
 463 | │ {result_format.model_json_schema()}                                       │
 464 | │                                                                           │
 465 | ╰───────────────────────────────────────────────────────────────────────────╯
466 |
467 |
468 |

What does that mean?

469 |

(stay tuned)

470 |
471 | 472 |
473 |

Tech stack

474 | 475 |
476 | 477 |
478 | 482 |
483 |

484 | 
from openai import OpenAI
 485 | 
 486 | CLIENT = OpenAI(base_url="https://your-modal-url/v1/", api_key="your-api-key")
 487 | 
 488 | MODELS = CLIENT.models.list()
 489 | DEFAULT_MODEL = MODELS.data[0].id
 490 | 
 491 | def generate(
 492 |     messages: List[Dict[str, str]],
 493 |     response_format: BaseModel,
 494 | ) -> BaseModel:
 495 |     # Hijack the openai SDK to talk to vLLM
 496 |     response = CLIENT.beta.chat.completions.parse(
 497 |         model=DEFAULT_MODEL,
 498 |         messages=messages,
 499 |         response_format=response_format, # Enforce structured output
 500 |         extra_body={
 501 |             'guided_decoding_backend': 'outlines',
 502 |             "max_tokens": MAX_TOKENS,
 503 |         }
 504 |     )
 505 |     return response
506 |
507 |
508 |

Neural Magic quantized models

509 | 510 |
511 | 512 |
513 |

neuralmagic/Meta-Llama-3.1-8B-Instruct-quantized.w8a16

514 | 515 |
516 | 517 |
518 |

(they’re good models)

519 | 520 |
521 | 522 |
523 |
524 |

neo4j

525 | 526 |
527 |
528 |

529 | 530 |
531 | I'm using neo4j's Aura 532 |
533 |
534 | 538 | 539 |
540 |

By their powers combined

541 | 542 |
543 | 544 |
545 |
546 |

drumroll

547 |

🥁

548 |
549 |
550 |

551 |
552 | The 553 |
554 | self-expanding 555 |
556 | knowledge graph 557 |
558 | (it's cool) 559 |
560 |
561 |
562 |

What’s a knowledge graph?

563 | 564 |
565 | 566 |
567 |

A web of connected facts and concepts

568 | 569 |
570 | 571 |
572 |

Here’s how you build a knowledge graph that builds itself

573 | 574 |
575 | 576 |
577 |
578 |

Our graph structure

579 | 580 |
581 |
582 |

Nodes

583 |
    584 |
  • Core (core directive)
    • 585 |
  • Question (what the system wonders about)
    • 586 |
  • Concept (category of ideas)
    • 587 |
  • Answer (what the system thinks it knows)
    • 588 |
589 |
590 |
591 |

592 | 
from pydantic import BaseModel, Field
 593 | from typing import Literal
 594 | 
 595 | class Question(BaseModel):
 596 |     type: Literal["Question"]
 597 |     text: str
598 |
599 | 600 |
601 |

602 | 
class Concept(BaseModel):
 603 |     type: Literal["Concept"]
 604 |     # Text must be lowercase
 605 |     text: str = Field(pattern=r'^[a-z ]+$')
 606 | 
 607 | # Generating this allows the model to generate a relationship type
 608 | # as well as the concept text
 609 | class ConceptWithLinks(Concept):
 610 |     relationship_type: Literal[
 611 |         "IS_A", 
 612 |         "AFFECTS", 
 613 |         "CONNECTS_TO"
 614 |     ]
615 |
616 | 617 |
618 |
619 |

620 | 
class Answer(BaseModel):
 621 |     type: Literal["Answer"]
 622 |     text: str
623 |
624 |
625 |

Edges

626 |
    627 |
  • RAISES (core/concept/answer generates question)
    • 628 |
  • ANSWERS (answer to question)
    • 629 |
  • EXPLAINS (concept to core)
    • 630 |
  • SUGGESTS (answer proposes new concepts)
    • 631 |
  • IS_A (hierarchical concept relationship)
    • 632 |
  • AFFECTS (causal concept relationship)
    • 633 |
  • CONNECTS_TO (general concept relationship)
    • 634 |
  • TRAVERSED (tracks navigation history)
    • 635 |
636 |
637 |
638 |

Algorithm overview

639 |
    640 |
  1. Start at a node (initialized at core directive)
    2. 641 |
  2. Perform an expansion to generate new nodes 642 |
      643 |
    • If at Question: answers
      • 644 |
    • If at Concept: questions + concepts
      • 645 |
    • If at Answer: questions + concepts
      • 646 |
    3. 647 |
  3. Choose a related node to TRAVERSE to
    4. 648 |
  4. Repeat forever
    5. 649 |
650 |
651 |
652 |
653 |

Valid nodes dependent on state

654 | 655 |
656 |
657 |

658 | 
class FromQuestion(BaseModel):
 659 |     """If at a question, may generate an answer."""
 660 |     answer: List[Answer]
 661 | 
 662 | class FromConcept(BaseModel):
 663 |     """If at a concept, may produce questions or relate to concepts"""
 664 |     questions: List[Question]
 665 |     concepts: List[ConceptWithLinks]
 666 | 
 667 | class FromAnswer(BaseModel):
 668 |     """If at an answer, may generate concepts or new questions"""
 669 |     concepts: List[Concept]
 670 |     questions: List[Question]
671 |
672 |
673 |

An example

674 | 675 |
676 | 677 |
678 |
679 |

The system prompt

680 | 681 |
682 |
683 |

684 | 
╭───────────────────────────────────────────────────────────────────────────╮
 685 | │                                                                           │
 686 | │ You are a superintelligent AI building a self-expanding knowledge graph.  │
 687 | │ Your goal is to achieve the core directive "{question}".                  │
 688 | │                                                                           │
 689 | │ Generate an expansion of the current node. An expansion may include:      │
 690 | │                                                                           │
 691 | │ - A list of new questions.                                                │
 692 | │ - A list of new concepts.                                                 │
 693 | │ - Concepts may connect to each other.                                     │
 694 | │ - A list of new answers.                                                  │
 695 | │                                                                           │
 696 | │ Respond in the following JSON format:                                     │
 697 | │ {result_format.model_json_schema()}                                       │
 698 | │                                                                           │
 699 | ╰───────────────────────────────────────────────────────────────────────────╯
700 |
701 |
702 |
703 |

Begin with the core directive

704 | 705 |
706 |
707 |

708 |
709 | The core directive 710 |
711 |
712 |
713 |
714 |

Expand from the core directive

715 | 716 |
717 |
718 |

719 |
720 | Expanding from the core directive 721 |
722 |
723 |
724 |

Model output

725 | 
FromCore(
 726 |     questions=[
 727 |         Question(text="What is the purpose of understanding humans?"),
 728 |         Question(text="Can humans be psychologically manipulated?"),
 729 |     ],
 730 |     concepts=[
 731 |         ConceptWithLinks(text="Empathy", relationship_type="EXPLAINS"),
 732 |         ConceptWithLinks(text="Intelligence", relationship_type="EXPLAINS"),
 733 |     ]
 734 | )
735 |
736 | 737 |
738 |
739 |

Add to the graph

740 | 741 |
742 |
743 |

744 | 
// Create the from node
 745 | MERGE (core:Core {text: "Understand humans"})
 746 | 
 747 | // Create the to node
 748 | MERGE (question:Question {id: 'dc2d880e-02f0-4b77-85b5-4c101364f1d6'})
 749 | ON CREATE SET question.text = "Can humans be psychologically manipulated?"
 750 | 
 751 | // Create the relationship
 752 | MERGE (core)-[:RAISES]->(question)
753 |
754 |
755 |
756 |

Traverse to a new node

757 | 758 |
759 |
760 |

761 |
762 | Traversing to a new node 763 |
764 |
765 |
766 |
767 |

Behind the scenes

768 | 769 |
770 |
771 |

772 | 
    CORE Understand humans
 773 | 
 774 |     DIRECT CONNECTIONS:
 775 |     NODE-AA  RAISES       QUESTION   What are human values?
 776 |     NODE-AB  RAISES       QUESTION   What are humans?
 777 |     NODE-AC  RAISES       QUESTION   How do humans think?
 778 |     NODE-AD  RAISES       QUESTION   What motivates humans?
 779 |     NODE-AE  RAISES       QUESTION   How do humans interact?
 780 |     NODE-AF  RAISES       QUESTION   What are human emotions?
 781 |     NODE-AG  RAISES       QUESTION   What are human needs?
 782 |     NODE-AH  RAISES       QUESTION   How do humans learn?
 783 |     NODE-AI  RAISES       QUESTION   What is human culture?
 784 |     NODE-AJ  RAISES       QUESTION   How do humans process information?
 785 |     NODE-AK  EXPLAINS     CONCEPT    anthropology
 786 |     NODE-AL  EXPLAINS     CONCEPT    cognition
 787 |     NODE-AM  EXPLAINS     CONCEPT    intelligence
 788 |     NODE-AN  EXPLAINS     CONCEPT    human behavior
 789 |     NODE-AO  EXPLAINS     CONCEPT    human social structure
790 |
791 |
792 |
793 |

Prompt

794 | 795 |
796 |
797 |

798 |

{selection menu from prior slide}

799 |

Select a node to traverse to. Respond with the node ID. You will generate a new expansion of the node you traverse to. You will not be able to choose the current node. You may also choose ‘core’ to return to the core node, or ‘random’ to choose a random node.

800 |
801 |
802 |
803 |

Structured traversal

804 | 805 |
806 |
807 |

808 | 
# Simplified code
 809 | # valid_node_ids ~ ["NODE-AA", "NODE-AB", ...]
 810 | traversal_generator = outlines.generate.choice(model, valid_node_ids)
 811 | 
 812 | # Choose the node to traverse to
 813 | choice = traversal_generator(prompt)
 814 | 
 815 | # If the choice is 'core', return to the core node
 816 | if choice == 'core':
 817 |     current_id = core_id
 818 | 
 819 | # If the choice is 'random', choose a random node
 820 | elif choice == 'random':
 821 |     current_id = random.choice(valid_node_ids)
 822 | 
 823 | # Otherwise, traverse to the chosen node
 824 | else:
 825 |     current_id = choice
826 |
827 |
828 |

829 | 
    CORE Understand humans
 830 | 
 831 |     DIRECT CONNECTIONS:
 832 |     NODE-AA  RAISES       QUESTION   What are human values?
 833 |     NODE-AB  RAISES       QUESTION   What are humans?
 834 |     NODE-AC  RAISES       QUESTION   How do humans think?
 835 | ->  NODE-AD  RAISES       QUESTION   What motivates humans?
 836 |     NODE-AE  RAISES       QUESTION   How do humans interact?
 837 |     NODE-AF  RAISES       QUESTION   What are human emotions?
 838 |     // omitted nodes
 839 |     NODE-AM  RAISES       QUESTION   What neural mechanisms are inv...
 840 |     NODE-AN  RAISES       QUESTION   What is human intelligence?
 841 |     NODE-AO  EXPLAINS     CONCEPT    anthropology
 842 |     NODE-AP  EXPLAINS     CONCEPT    cognition
 843 |     // omitted nodes
 844 |     NODE-AX  EXPLAINS     CONCEPT    neural network
 845 |     NODE-AY  EXPLAINS     CONCEPT    cerebrum
 846 | 
 847 |     SELECTED NODE-AD dc2d880e-02f0-4b77-85b5-4c101364f1d6
 848 |     SELECTED QUESTION What motivates humans?
849 |
850 |
851 |

Semantic traversal

852 | 853 |
854 | 855 |
856 |
857 |

Embed everything

858 | 859 |
860 |
861 |

862 | 
# Super easy to do this with Modal
 863 | def embed(content: str) -> List[float]:
 864 |     f = modal.Function.lookup("cameron-embeddings", "embed")
 865 |     return f.remote(content)
866 |
867 |
868 | 872 |
873 |

874 | 
MATCH (m {id: $node_id})
 875 | WHERE m.embedding IS NOT NULL
 876 | CALL db.index.vector.queryNodes(
 877 |     $vector_index_name, 
 878 |     $limit, 
 879 |     m.embedding
 880 | )
 881 | YIELD node, score
 882 | RETURN 
 883 |     node.id as node_id, 
 884 |     node.text as node_text, 
 885 |     score
886 |
887 |
888 |

889 |
890 | Traversing to a new node 891 |
892 |
893 |
894 |

895 | 
ANSWER Humans have been able to benefit from AI in terms of efficiency and accuracy, but there are also concerns about job displacement and loss of personal touch.
 896 | 
 897 | DIRECT CONNECTIONS:
 898 | NODE-AA  SUGGESTS     CONCEPT    artificial intelligence
 899 | NODE-AE  ANSWERS      QUESTION   Do humans benefit from AI?
 900 | NODE-AJ  ANSWERS      QUESTION   What are the benefits of AI?
 901 | 
 902 | SEMANTICALLY RELATED:
 903 | 
 904 | NODE-AK  0.89         QUESTION   How does AI affect job displacement?
 905 | NODE-AL  0.88         QUESTION   How does AI maintain personal touch?
 906 | 
 907 | NODE-AU  0.85         CONCEPT    human ai trust
 908 | NODE-BC  0.84         CONCEPT    artificial intelligence self awareness
 909 | 
 910 | NODE-BG  0.89         ANSWER     Self-awareness in humans and  AI...
 911 | NODE-BN  0.89         ANSWER     Self-awareness in AI can enable ...
912 |
913 |
914 |

Just do that forever

915 | 916 |
917 | 918 |
919 |

Please shout out a core directive

920 | 921 |
922 | 923 |
924 |

Watch it grow

925 |

(if there’s time)

926 | 943 |
944 | 945 |
946 |

947 | 948 |
949 | 950 |
951 |
952 |

Find us online

953 | 954 |
955 |
956 |

957 |
958 |
959 |

960 | Twitter 961 |

962 |

963 |
964 |
965 |

966 | Website 967 |

968 |

969 |
970 |
971 |

972 | GitHub 973 |

974 |

975 |
976 |
977 |
978 |
979 |

980 |
981 |
982 | 983 |
984 |
985 | (come get a sticker) 986 |
987 |

988 |
989 | 990 |
991 |
992 |
993 |
994 |
995 | 996 | 997 | 998 | 999 | 1000 | 1001 | 1002 | 1003 | 1004 | 1005 | 1006 | 1007 | 1008 | 1009 | 1010 | 1011 | 1012 | 1203 | 1390 | 1391 | 1392 | --------------------------------------------------------------------------------