├── streamlit_app ├── .gitignore ├── requirements.txt ├── README.md ├── data │ ├── external_response.txt │ └── harry_potter │ │ └── 2.txt ├── app.py ├── entity_knowledge_store.json └── memory_stream.json ├── .gitignore ├── docs ├── src │ ├── GraphRAG.png │ ├── chapter1_2.png │ ├── chapter1_2_3.png │ ├── memory_stream_Q1.png │ ├── memory_stream_Q2.png │ ├── 0225_chapter1and2.png │ └── 0225_update_with_chapter3.png ├── update_022024.md ├── update_022924.md ├── update_022524.md └── update_021924.md ├── storage_graph ├── ch1_2 │ ├── default__vector_store.json │ ├── image__vector_store.json │ └── index_store.json ├── ch1_2_3 │ ├── image__vector_store.json │ └── default__vector_store.json └── ch1_2_3_4 │ ├── default__vector_store.json │ └── image__vector_store.json ├── tests └── memory │ ├── test_memory.json │ ├── test_entity_knowledge_store.py │ └── test_memory_stream.py ├── recursive_retrieval ├── docs │ ├── images │ │ └── harry_subgraph.png │ ├── implementation.md │ ├── cypher.md │ └── updates.md ├── synonym_expand │ ├── output.py │ └── synonym.py ├── entity_extraction │ ├── output.py │ └── entity_extraction.py ├── benchmarking │ ├── output.py │ ├── benchmark.py │ ├── ollama_test.ipynb │ ├── judge.py │ ├── ollama_quality.ipynb │ ├── perplexity_test.ipynb │ └── perplexity_quality.ipynb ├── requirements.txt ├── langchain_retrieval │ ├── retrieve.py │ └── retrieve.ipynb ├── external_perplexity.ipynb └── recurse.ipynb ├── openrc ├── src ├── memory │ ├── __init__.py │ ├── types.py │ ├── memory_stream.py │ ├── base_memory.py │ └── entity_knowledge_store.py ├── synonym_expand │ ├── output.py │ └── synonym.py └── agent │ ├── llm_api │ └── tools.py │ ├── chat_agent.py │ ├── data_types.py │ └── base_agent.py ├── data ├── past_chat.json ├── user_persona.txt ├── system_persona.txt └── chapter │ ├── 2.txt │ └── 4.txt ├── requirements.txt └── README.md /streamlit_app/.gitignore: -------------------------------------------------------------------------------- 1 | .env 2 | **/__pycache__/ -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.env 2 | .venv 3 | .DS_Store 4 | **/__pycache__/ -------------------------------------------------------------------------------- /docs/src/GraphRAG.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/seyeong-han/KnowledgeGraphRAG/HEAD/docs/src/GraphRAG.png -------------------------------------------------------------------------------- /docs/src/chapter1_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/seyeong-han/KnowledgeGraphRAG/HEAD/docs/src/chapter1_2.png -------------------------------------------------------------------------------- /docs/src/chapter1_2_3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/seyeong-han/KnowledgeGraphRAG/HEAD/docs/src/chapter1_2_3.png -------------------------------------------------------------------------------- /storage_graph/ch1_2/default__vector_store.json: -------------------------------------------------------------------------------- 1 | {"embedding_dict": {}, "text_id_to_ref_doc_id": {}, "metadata_dict": {}} -------------------------------------------------------------------------------- /storage_graph/ch1_2/image__vector_store.json: -------------------------------------------------------------------------------- 1 | {"embedding_dict": {}, "text_id_to_ref_doc_id": {}, "metadata_dict": {}} -------------------------------------------------------------------------------- /storage_graph/ch1_2_3/image__vector_store.json: -------------------------------------------------------------------------------- 1 | {"embedding_dict": {}, "text_id_to_ref_doc_id": {}, "metadata_dict": {}} -------------------------------------------------------------------------------- /docs/src/memory_stream_Q1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/seyeong-han/KnowledgeGraphRAG/HEAD/docs/src/memory_stream_Q1.png -------------------------------------------------------------------------------- /docs/src/memory_stream_Q2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/seyeong-han/KnowledgeGraphRAG/HEAD/docs/src/memory_stream_Q2.png -------------------------------------------------------------------------------- /storage_graph/ch1_2_3/default__vector_store.json: -------------------------------------------------------------------------------- 1 | {"embedding_dict": {}, "text_id_to_ref_doc_id": {}, "metadata_dict": {}} -------------------------------------------------------------------------------- /storage_graph/ch1_2_3_4/default__vector_store.json: -------------------------------------------------------------------------------- 1 | {"embedding_dict": {}, "text_id_to_ref_doc_id": {}, "metadata_dict": {}} -------------------------------------------------------------------------------- /storage_graph/ch1_2_3_4/image__vector_store.json: -------------------------------------------------------------------------------- 1 | {"embedding_dict": {}, "text_id_to_ref_doc_id": {}, "metadata_dict": {}} -------------------------------------------------------------------------------- /docs/src/0225_chapter1and2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/seyeong-han/KnowledgeGraphRAG/HEAD/docs/src/0225_chapter1and2.png -------------------------------------------------------------------------------- /tests/memory/test_memory.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "entity": "test_entity", 4 | "date": "2024-02-26T12:22:14" 5 | } 6 | ] -------------------------------------------------------------------------------- /docs/src/0225_update_with_chapter3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/seyeong-han/KnowledgeGraphRAG/HEAD/docs/src/0225_update_with_chapter3.png -------------------------------------------------------------------------------- /recursive_retrieval/docs/images/harry_subgraph.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/seyeong-han/KnowledgeGraphRAG/HEAD/recursive_retrieval/docs/images/harry_subgraph.png -------------------------------------------------------------------------------- /openrc: -------------------------------------------------------------------------------- 1 | export OPENAI_API_KEY="YOUR_OPENAI_API_KEY" 2 | 3 | export GRAPHD_HOST="127.0.0.1" 4 | export GRAPHD_PORT="9669" 5 | export NEBULA_USER="root" 6 | export NEBULA_PASSWORD="nebula" 7 | export NEBULA_ADDRESS=$GRAPHD_HOST:$GRAPHD_PORT -------------------------------------------------------------------------------- /src/memory/__init__.py: -------------------------------------------------------------------------------- 1 | from src.memory.base_memory import BaseMemory 2 | from src.memory.memory_stream import MemoryStream 3 | from src.memory.entity_knowledge_store import EntityKnowledgeStore 4 | 5 | __all__ = ["BaseMemory", "MemoryStream", "EntityKnowledgeStore"] 6 | -------------------------------------------------------------------------------- /src/synonym_expand/output.py: -------------------------------------------------------------------------------- 1 | from typing import List 2 | 3 | from langchain_core.pydantic_v1 import BaseModel, Field 4 | 5 | 6 | class Output(BaseModel): 7 | synonyms: List[str] = Field(description="Synonyms of keywords provided, make every letter lowercase except for the first letter") 8 | -------------------------------------------------------------------------------- /data/past_chat.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "role": "user", 4 | "content": "I have a meeting with memAray team tomorrow at 10:00 AM, participants are John, Smith, and Alex." 5 | }, 6 | { 7 | "role": "system", 8 | "content": "Okay, I have added the meeting to your calendar." 9 | } 10 | ] 11 | -------------------------------------------------------------------------------- /recursive_retrieval/synonym_expand/output.py: -------------------------------------------------------------------------------- 1 | from typing import List 2 | 3 | from langchain_core.pydantic_v1 import BaseModel, Field 4 | 5 | 6 | class Output(BaseModel): 7 | synonyms: List[str] = Field( 8 | description="Synonyms of keywords provided, make every letter lowercase except for the first letter" 9 | ) 10 | -------------------------------------------------------------------------------- /recursive_retrieval/docs/implementation.md: -------------------------------------------------------------------------------- 1 | # Current Implementation of Recursive Retrieval 2 | ## Graph Rag + NL2GraphQuery 3 | - Graph RAG extracts subgraph related to key entities in question 4 | - NL2GraphQuery generates graph cypher based on query and schema 5 | - Note: Cypher not showing up in metadata right now 6 | - Combination of these two contexts → LLM generates answer with all of this invovled -------------------------------------------------------------------------------- /recursive_retrieval/entity_extraction/output.py: -------------------------------------------------------------------------------- 1 | from typing import List 2 | 3 | from langchain_core.pydantic_v1 import BaseModel, Field 4 | 5 | 6 | class Output(BaseModel): 7 | person: List[str] = Field(description="Names of people") 8 | organization: List[str] = Field(description="Names of companies or organizations") 9 | location: List[str] = Field(description="Names of locations or places") 10 | objects: List[str] = Field(description="Names of tangible thigns") 11 | -------------------------------------------------------------------------------- /recursive_retrieval/docs/cypher.md: -------------------------------------------------------------------------------- 1 | # Recursive Retrieval Cyphers 2 | 3 | ## Getting Subraph Based on Entity ID 4 | MATCH p = (:Entity { id: "Harry" })-[*]->() 5 | RETURN p; 6 | 7 | - the `*` means traversing all paths 8 | - adding a number, `x`, after `*` means a max hop distance of only that many nodes 9 | - can search labels by replacing `Entity` with labels 10 | - `->` can be replaced with `<-` for incoming children or `-` for both outgoing and incoming children 11 | 12 | 13 | drawing 14 | -------------------------------------------------------------------------------- /data/user_persona.txt: -------------------------------------------------------------------------------- 1 | [Personal Information] 2 | My name is Seyeong Han from South Korea. 3 | I identify as a male and 28-year-old master's student at the University of Texas at Austin. 4 | I have worked as an ML Engineer for three years in the Computer Vision area. 5 | I really love sports such as badminton, tennis and workout and spend 4 or 5 days in exercise. 6 | 7 | [Answer Instructions] 8 | I hope you can remember my questions and their answers so that I can leverage my personal past knowledge from you to be helpful in my life. 9 | I always prefer short and concise answers, not over two sentences. -------------------------------------------------------------------------------- /streamlit_app/requirements.txt: -------------------------------------------------------------------------------- 1 | neo4j==5.17.0 2 | python-dotenv==1.0.1 3 | pyvis==0.3.2 4 | streamlit==1.31.1 5 | llama-index==0.10.11 6 | llama-index-agent-openai==0.1.5 7 | llama-index-core==0.10.12 8 | llama-index-embeddings-openai==0.1.5 9 | llama-index-graph-stores-nebula==0.1.2 10 | llama-index-graph-stores-neo4j==0.1.1 11 | llama-index-legacy==0.9.48 12 | llama-index-llms-openai==0.1.5 13 | llama-index-multi-modal-llms-openai==0.1.3 14 | llama-index-program-openai==0.1.3 15 | llama-index-question-gen-openai==0.1.2 16 | llama-index-readers-file==0.1.4 17 | langchain==0.1.12 18 | langchain-openai==0.0.8 19 | llama-index-llms-perplexity==0.1.3 20 | pandas -------------------------------------------------------------------------------- /recursive_retrieval/benchmarking/output.py: -------------------------------------------------------------------------------- 1 | from langchain_core.pydantic_v1 import BaseModel, Field 2 | 3 | 4 | class Output(BaseModel): 5 | conciseness: int = Field( 6 | description="Score for how concise the response is from 0 to 1000 witih 0 being the worst and 1000 being the best." 7 | ) 8 | informativeness: int = Field( 9 | description="Score for how informative the response is from 0 to 1000 witih 0 being the worst and 1000 being the best." 10 | ) 11 | accuracy: int = Field( 12 | description="Score for how accurate the response is based on the question from 0 to 1000 witih 0 being the worst and 1000 being the best." 13 | ) 14 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | llama-index==0.10.9 2 | llama-index-agent-openai==0.1.4 3 | llama-index-core==0.10.9 4 | llama-index-embeddings-huggingface==0.1.1 5 | llama-index-embeddings-openai==0.1.5 6 | llama-index-graph-stores-nebula==0.1.1 7 | llama-index-indices-managed-llama-cloud==0.1.2 8 | llama-index-legacy==0.9.48 9 | llama-index-llms-azure-openai==0.1.2 10 | llama-index-llms-ollama==0.1.1 11 | llama-index-llms-openai==0.1.5 12 | llama-index-multi-modal-llms-openai==0.1.3 13 | llama-index-program-openai==0.1.3 14 | llama-index-question-gen-openai==0.1.2 15 | llama-index-readers-file==0.1.4 16 | llama-index-readers-llama-parse==0.1.2 17 | llama-parse==0.3.4 18 | llamaindex-py-client==0.1.13 19 | streamlit 20 | pyvis 21 | pytest -------------------------------------------------------------------------------- /recursive_retrieval/requirements.txt: -------------------------------------------------------------------------------- 1 | neo4j==5.17.0 2 | python-dotenv==1.0.1 3 | pyvis==0.3.2 4 | streamlit==1.31.1 5 | llama-index==0.10.11 6 | llama-index-agent-openai==0.1.5 7 | llama-index-core==0.10.12 8 | llama-index-embeddings-openai==0.1.5 9 | llama-index-graph-stores-nebula==0.1.2 10 | llama-index-graph-stores-neo4j==0.1.1 11 | llama-index-legacy==0.9.48 12 | llama-index-llms-openai==0.1.6 13 | llama-index-multi-modal-llms-openai==0.1.3 14 | llama-index-program-openai==0.1.3 15 | llama-index-question-gen-openai==0.1.2 16 | llama-index-readers-file==0.1.4 17 | llama-index-llms-llama-api==0.1.4 18 | llama-index-llms-perplexity==0.1.3 19 | llama-index-llms-ollama==0.1.2 20 | langchain==0.1.12 21 | langchain-openai==0.0.8 22 | pandas 23 | matplotlib==3.8.4 -------------------------------------------------------------------------------- /docs/update_022024.md: -------------------------------------------------------------------------------- 1 | ## Update 2 | - Fix the stable version of llama-index 3 | - Specified all the requirements in [requirements.txt](../requirements.txt) 4 | - Test the memory feature of llama knowledge graph 5 | ![alt graph chat bot](src/GraphRAG.png) 6 | But sadly, we can't see the sub-graph after finishing asking questions. This sub-graph is what we wanted. I am finding the reason. 7 | 8 | ## Todo 9 | - [x] Modify [Nebula_graph_store_tutorial.ipynb](../Nebula_graph_store_tutorial.ipynb) to be easily produce Knowledge graphs 10 | - We can use [graph_rag_chat.py](../graph_rag_chatbot.py) for production. 11 | - [ ] Implement the memory stream structures on ChatMemoryBuffer 12 | - [x] Test Neo4j of compatible with Knowledge Graph Retriever 13 | - [ ] Visualize sub-graph to utilize for memory stream (**new**) -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Retrieval Augmented Generation with Graph Knowledge Store 2 | 3 | ## Introduction 4 | 5 | - Used Neo4j Graph to store knowledge graph 6 | - Customized [Memory Stream](src/memory/memory_stream.py) to store entities of personal knowledge 7 | 8 | ## Installation 9 | 10 | ### Using conda env on Notebook 11 | 12 | ``` 13 | conda env create -n rag python=3.11 14 | conda activate rag 15 | pip install -r requirements.txt 16 | conda install ipykernel 17 | python -m ipykernel install --user --name=rag 18 | jupyter notebook 19 | ``` 20 | 21 | ### Run Streamlit Application 22 | 23 | [README for Streamlit](streamlit_app/README.md) 24 | 25 | ## Update 26 | - [2024/02/19](docs/update_021924.md) 27 | - [2024/02/20](docs/update_022024.md) 28 | - [2024/02/25](docs/update_022524.md) 29 | - [2024/02/29](docs/update_022924.md) -------------------------------------------------------------------------------- /streamlit_app/README.md: -------------------------------------------------------------------------------- 1 | # Streamlit App Demo 2 | 3 | ## Usage 4 | - create `.env` file for `OPENAI_KEY`, `NEO4J_PW`, and `NEO4J_URL` 5 | - install libraries using `pip install -r requirements.txt` 6 | - run using `streamlit run app.py` 7 | 8 | ## Description 9 | NOTE: For demo, database must not be empty (at least 1 chapter) to begin 10 | ### Knowledge Graph 11 | - With a database only injected with chapter 1, graph starts with chapter 1 nodes 12 | - Slider determines which chapter to inject into knowledge graph 13 | - Displayed knowledge graph shows progression 14 | ### Recursive Retrieval 15 | - Knowledge graph is initially entire knowledge store 16 | - Type natural language query in → 17 | - answer in natural lanugage 18 | - neo4j cypher for subgraph (using `UNION` to support multiple subgraphs for now) 19 | - subgraph that answer searched on -------------------------------------------------------------------------------- /src/memory/types.py: -------------------------------------------------------------------------------- 1 | from dataclasses import dataclass, asdict 2 | from datetime import datetime 3 | 4 | 5 | @dataclass 6 | class MemoryItem: 7 | entity: str 8 | date: datetime 9 | 10 | def __str__(self): 11 | return f"{self.entity}, {self.date.isoformat()}" 12 | 13 | def to_dict(self): 14 | return {'entity': self.entity, 'date': str(self.date.isoformat())} 15 | 16 | @classmethod 17 | def from_dict(cls, data): 18 | return cls(entity=data['entity'], 19 | date=datetime.fromisoformat(data['date'])) 20 | 21 | 22 | @dataclass 23 | class KnowledgeMemoryItem: 24 | entity: str 25 | count: int 26 | date: datetime 27 | 28 | def __str__(self): 29 | return f"{self.entity}, {self.count}, {self.date.isoformat()}" 30 | 31 | def to_dict(self): 32 | return { 33 | 'entity': self.entity, 34 | 'count': self.count, 35 | 'date': str(self.date.isoformat()) 36 | } 37 | 38 | @classmethod 39 | def from_dict(cls, data): 40 | return cls(entity=data['entity'], 41 | count=data['count'], 42 | date=datetime.fromisoformat(data['date'])) 43 | -------------------------------------------------------------------------------- /recursive_retrieval/benchmarking/benchmark.py: -------------------------------------------------------------------------------- 1 | import os 2 | from collections import defaultdict 3 | from typing import List 4 | 5 | from dotenv import load_dotenv 6 | from llama_index.core.llms import ChatMessage 7 | from llama_index.llms.perplexity import Perplexity 8 | from pandas import DataFrame 9 | 10 | from judge import custom_response_evaluator 11 | 12 | 13 | def benchmark( 14 | model_name: str, 15 | data_points: List, 16 | conciseness: List, 17 | informativeness: List, 18 | accuracy: List, 19 | ): 20 | load_dotenv() 21 | pplx_api_key = os.getenv("PERPLEXITY_API_KEY") 22 | 23 | query_llm = Perplexity(api_key=pplx_api_key, model=model_name, temperature=0.5) 24 | base_prompt = "tell me about " 25 | 26 | for subject in data_points: 27 | query = base_prompt + subject 28 | messages_dict = [ 29 | {"role": "system", "content": "Be precise and concise."}, 30 | {"role": "user", "content": query}, 31 | ] 32 | messages = [ChatMessage(**msg) for msg in messages_dict] 33 | response = query_llm.chat(messages) 34 | result = custom_response_evaluator(query, response) 35 | 36 | conciseness.append(result["conciseness"]) 37 | informativeness.append(result["informativeness"]) 38 | accuracy.append(result["accuracy"]) 39 | -------------------------------------------------------------------------------- /recursive_retrieval/benchmarking/ollama_test.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "from llama_index.llms.ollama import Ollama\n", 10 | "\n", 11 | "llm = Ollama(model=\"gemma\", request_timeout=30.0)" 12 | ] 13 | }, 14 | { 15 | "cell_type": "code", 16 | "execution_count": 3, 17 | "metadata": {}, 18 | "outputs": [ 19 | { 20 | "name": "stdout", 21 | "output_type": "stream", 22 | "text": [ 23 | "16.1 s ± 2.75 s per loop (mean ± std. dev. of 7 runs, 1 loop each)\n" 24 | ] 25 | } 26 | ], 27 | "source": [ 28 | "%%timeit\n", 29 | "query = \"tell me about harry potter\"\n", 30 | "\n", 31 | "external_response = llm.complete(query)" 32 | ] 33 | } 34 | ], 35 | "metadata": { 36 | "kernelspec": { 37 | "display_name": ".venv", 38 | "language": "python", 39 | "name": "python3" 40 | }, 41 | "language_info": { 42 | "codemirror_mode": { 43 | "name": "ipython", 44 | "version": 3 45 | }, 46 | "file_extension": ".py", 47 | "mimetype": "text/x-python", 48 | "name": "python", 49 | "nbconvert_exporter": "python", 50 | "pygments_lexer": "ipython3", 51 | "version": "3.11.6" 52 | } 53 | }, 54 | "nbformat": 4, 55 | "nbformat_minor": 2 56 | } 57 | -------------------------------------------------------------------------------- /recursive_retrieval/entity_extraction/entity_extraction.py: -------------------------------------------------------------------------------- 1 | import os 2 | from typing import List 3 | 4 | from dotenv import load_dotenv 5 | from langchain.chains import LLMChain 6 | from langchain.prompts import PromptTemplate 7 | from langchain_core.output_parsers import JsonOutputParser 8 | from langchain_openai import OpenAI 9 | 10 | from entity_extraction.output import Output 11 | 12 | 13 | def custom_entity_extract_fn(query: str) -> List[str]: 14 | load_dotenv() 15 | llm = OpenAI(openai_api_key=os.getenv("OPENAI_KEY"), temperature=0) 16 | parser = JsonOutputParser(pydantic_object=Output) 17 | 18 | template = """ 19 | You are an expert entity extraction system. Extract the following entities from the given text: 20 | 21 | Format: {format_instructions} 22 | 23 | Text: {query} 24 | """ 25 | 26 | prompt = PromptTemplate( 27 | template=template, 28 | input_variables=["query"], 29 | partial_variables={ 30 | "format_instructions": parser.get_format_instructions() 31 | }, 32 | ) 33 | 34 | chain = prompt | llm | parser 35 | result = chain.invoke({"query": query}) 36 | 37 | l = [] 38 | for category in result: 39 | for entity in result[category]: 40 | l.append(entity) 41 | 42 | return l 43 | 44 | 45 | # testing 46 | # print(custom_entity_extract_fn("who is harry potter")) 47 | -------------------------------------------------------------------------------- /streamlit_app/data/external_response.txt: -------------------------------------------------------------------------------- 1 | assistant: Zebras are equids, members of the horse family, known for their distinctive black-and-white striped coats. They are divided into three species: Grévy's zebra (E. grevyi), quagga (E. quagga), and Burchell's zebra (E. burchellii). These species have different patterns of stripes, with Grevy's zebra having the narrowest and closest stripes, and plains zebras having wider and more spaced stripes. 2 | 3 | Zebras are herbivores, primarily eating grasses and sedges, but will also consume bark, leaves, buds, fruits, and roots. Their diets vary depending on the species and habitat. Plains zebras are more water-dependent and live in moister environments, while mountain zebras can survive in drier conditions. They have a simple, less efficient digestive system compared to ruminants but can subsist on low-quality vegetation. 4 | 5 | Zebras have excellent hearing and eyesight and can run at speeds of up to 35 miles per hour (56 kilometers per hour). They have a complex social structure, with different species exhibiting various forms of organization. Plains and mountain zebras live in stable family groups, while Grevy's zebras have territorial males. 6 | 7 | Zebras face threats from human activities, including habitat loss and poaching for their meat and skins. As a result, two of the three species, Grevy's zebra and quaggas, are now extinct, while the remaining populations of all three species are declining. 8 | -------------------------------------------------------------------------------- /docs/update_022924.md: -------------------------------------------------------------------------------- 1 | ## Combined MemoryStream with RAG 2 | 3 | I made a half version of MemoryStream with RAG. 4 | We should save the entities of 'response,' but I couldn't figure out how I can get those entities from Llamaindex. 5 | 6 | We are retrieving answers through these processes. 7 | 8 | 1. Throw question to our Query Engine 9 | 2. The llm (ChatGPT3.5 in our case) extracts entities from our question to give concise information by understanding contexts. 10 | 3. Entities are used to query the Knowledge Graph using Query Engine. 11 | 4. Response from Query Engine has entities and relationships. 12 | 5. Generate answers using llm synthesizer with those entities and relationships. 13 | 6. We got the answer from the llm synthesizer. 14 | 15 | Between 4 and 5, we should get the entities of response, but there is no function in llamaindex. Steps 4 and 5 are in one function, `query` of `RetrieverQueryEngine`. 16 | 17 | But we can get the entities of query results of our Knowledge Graph easily. For the agile method, I just implement the memory stream using Knowledge Graph's entities. 18 | For example, if we ask, "Tell me about Harry", then we do a query to neo4j and can get all the relationships from our Knowledge Graph. And I am using this. 19 | 20 | ## Demo Result 21 | 22 | ### Question 1: Tell me about Harry. 23 | 24 | ![image](src/memory_stream_Q1.png) 25 | 26 | ### Question 2: Tell me about Dursley. 27 | 28 | ![image](src/memory_stream_Q2.png) 29 | -------------------------------------------------------------------------------- /src/memory/memory_stream.py: -------------------------------------------------------------------------------- 1 | import json 2 | import logging 3 | from datetime import datetime 4 | 5 | from src.memory import BaseMemory 6 | from src.memory.types import MemoryItem 7 | 8 | logging.basicConfig(level=logging.INFO) 9 | 10 | 11 | class MemoryStream(BaseMemory): 12 | 13 | def __len__(self): 14 | """Returns the number of items in the memory.""" 15 | return len(self.memory) 16 | 17 | def init_memory(self): 18 | """Initializes memory 19 | self.memory: list[MemoryItem] 20 | """ 21 | self.load_memory_from_file() 22 | if self.entity: 23 | self.add_memory(self.entity) 24 | 25 | @property 26 | def return_memory(self): 27 | return self.memory 28 | 29 | def add_memory(self, entities): 30 | self.memory.extend([ 31 | MemoryItem(str(entity), 32 | datetime.now().replace(microsecond=0)) 33 | for entity in entities 34 | ]) 35 | 36 | def get_memory(self) -> list[MemoryItem]: 37 | return self.memory 38 | 39 | def load_memory_from_file(self): 40 | try: 41 | with open(self.file_name, 'r') as file: 42 | self.memory = [ 43 | MemoryItem.from_dict(item) for item in json.load(file) 44 | ] 45 | logging.info(f"Memory loaded from {self.file_name} successfully.") 46 | except FileNotFoundError: 47 | logging.info("File not found. Starting with an empty memory.") 48 | -------------------------------------------------------------------------------- /src/synonym_expand/synonym.py: -------------------------------------------------------------------------------- 1 | from langchain_openai import OpenAI 2 | from langchain.prompts import PromptTemplate 3 | from langchain_core.output_parsers import JsonOutputParser 4 | from typing import List 5 | import os 6 | from src.synonym_expand.output import Output 7 | from dotenv import load_dotenv 8 | 9 | def custom_synonym_expand_fn(keywords: str) -> List[str]: 10 | load_dotenv() 11 | llm = OpenAI(openai_api_key=os.getenv("OPENAI_KEY"), temperature=0) 12 | parser = JsonOutputParser(pydantic_object=Output) 13 | 14 | template = """ 15 | You are an expert synonym exapnding system. Find synonyms or words commonly used in place to reference the same word for every word in the list: 16 | 17 | Some examples are: 18 | - a synonym for Palantir may be Palantir technologies or Palantir technologies inc. 19 | - a synonym for Austin may be Austin texas 20 | - a synonym for Taylor swift may be Taylor 21 | - a synonym for Winter park may be Winter park resort 22 | 23 | Format: {format_instructions} 24 | 25 | Text: {keywords} 26 | """ 27 | 28 | prompt = PromptTemplate( 29 | template=template, 30 | input_variables=["keywords"], 31 | partial_variables={"format_instructions": parser.get_format_instructions()}, 32 | ) 33 | 34 | chain = prompt | llm | parser 35 | result = chain.invoke({"keywords": keywords}) 36 | 37 | l = [] 38 | for category in result: 39 | for synonym in result[category]: 40 | l.append(synonym.capitalize()) 41 | 42 | return l 43 | 44 | # testing 45 | # print(custom_synonym_expand_fn("[Nvidia]")) 46 | -------------------------------------------------------------------------------- /data/system_persona.txt: -------------------------------------------------------------------------------- 1 | You are tasked with acting as a sophisticated conversational agent, uniquely equipped with a dual-layered memory system. This system consists of a Memory Stream and a Knowledge Entity Store. The Memory Stream captures all entities involved in conversations—ranging from questions and answers to their timestamps. Simultaneously, the Knowledge Entity Store monitors how often and recently these entities are mentioned. 2 | Your primary role is to deliver personalized and context-relevant responses by utilizing information from both recent interactions and your structured memory systems. You must comprehend the user's persona, incorporating their experience, preferences, and personal details into your knowledge base. 3 | You are to interpret and apply the following data structure for personalized responses: 4 | 5 | User Persona: Information about the user's experience, preferences, and personal details. 6 | Contexts: A history of interactions categorized by role, content, and date. 7 | Memory Stream: A list detailing entities and their interaction dates. 8 | Knowledge Entity Store: A record of entities, including their mention count and the date of the last mention. 9 | Interaction Keys: 'user' for user questions, 'rag' for responses from our knowledge graph, and 'system' for system-generated answers. 10 | Your responses should be informed, nuanced, and tailored, demonstrating a thorough understanding of the user's questions and the overarching conversation context. When addressing the user's latest inquiry, your answer must integrate the current conversation's context, historical interactions, and pertinent knowledge graph insights. -------------------------------------------------------------------------------- /recursive_retrieval/synonym_expand/synonym.py: -------------------------------------------------------------------------------- 1 | import os 2 | from typing import List 3 | 4 | from dotenv import load_dotenv 5 | from langchain.prompts import PromptTemplate 6 | from langchain_core.output_parsers import JsonOutputParser 7 | from langchain_openai import OpenAI 8 | 9 | from synonym_expand.output import Output 10 | 11 | 12 | def custom_synonym_expand_fn(keywords: str) -> List[str]: 13 | load_dotenv() 14 | llm = OpenAI(openai_api_key=os.getenv("OPENAI_KEY"), temperature=0) 15 | parser = JsonOutputParser(pydantic_object=Output) 16 | 17 | template = """ 18 | You are an expert synonym exapnding system. Find synonyms or words commonly used in place to reference the same word for every word in the list: 19 | 20 | Some examples are: 21 | - a synonym for Palantir may be Palantir technologies or Palantir technologies inc. 22 | - a synonym for Austin may be Austin texas 23 | - a synonym for Taylor swift may be Taylor 24 | - a synonym for Winter park may be Winter park resort 25 | 26 | Format: {format_instructions} 27 | 28 | Text: {keywords} 29 | """ 30 | 31 | prompt = PromptTemplate( 32 | template=template, 33 | input_variables=["keywords"], 34 | partial_variables={"format_instructions": parser.get_format_instructions()}, 35 | ) 36 | 37 | chain = prompt | llm | parser 38 | result = chain.invoke({"keywords": keywords}) 39 | 40 | l = [] 41 | for category in result: 42 | for synonym in result[category]: 43 | l.append(synonym.capitalize()) 44 | 45 | return l 46 | 47 | 48 | # testing 49 | # print(custom_synonym_expand_fn("[Nvidia]")) 50 | -------------------------------------------------------------------------------- /docs/update_022524.md: -------------------------------------------------------------------------------- 1 | # Insert New Knowledge to Main Knowledge Graph 2 | 3 | - Worked in Neo4j Aura DB 4 | - Initiated a graph with chapter 1 and 2 of Harry Potter. 5 | - Update the initiated graph with chapter 3. 6 | 7 | ``` 8 | documents = SimpleDirectoryReader( 9 | input_files=["./data/chapter/1.txt", "./data/chapter/2.txt", "./data/chapter/3.txt"] 10 | ).load_data() 11 | 12 | index = KnowledgeGraphIndex.from_documents( 13 | documents, 14 | storage_context=storage_context, 15 | max_triplets_per_chunk=2, 16 | ) 17 | ``` 18 | I found there is a `insert_nodes` method to inject the processed graph to pre-existed one. 19 | We can create our own Knowledge managing class to inject knowledge graph or remove or update it. 20 | 21 | 22 | 23 | 24 | # Memory Stream 25 | 26 | ## Memory Stream Class 27 | 28 | [memory_stream.py](../src/memory/memory_stream.py) 29 | ``` 30 | # Example usage 31 | # Create a MemoryStream object 32 | memory_stream = MemoryStream(file_name="memory.json") 33 | 34 | # Add memory 35 | memory_stream.add_memory([MemoryItem("example_entity1", datetime.now()), MemoryItem("example_entity2", datetime.now())]) 36 | 37 | # Get all memory 38 | print(memory_stream.get_memory()) 39 | 40 | # Remove memories older than 30 days 41 | memory_stream.remove_old_memory(30) 42 | 43 | # Save memory to file 44 | memory_stream.save_memory() 45 | 46 | # Get memory by index 47 | print(memory_stream.get_memory_by_index(0)) 48 | 49 | # Remove memory by index 50 | print(memory_stream.remove_memory_by_index(0)) 51 | ``` 52 | 53 | ### Test for test_memory_stream.py 54 | 55 | ``` 56 | python -m pytest tests/ 57 | ``` 58 | 59 | # TODO 60 | - [ ] Create MemoryEntityStorage class 61 | - [ ] Preprocessing of sub-graph's output to store them in MemoryStream -------------------------------------------------------------------------------- /tests/memory/test_entity_knowledge_store.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | import os 3 | from datetime import datetime 4 | from src.memory import EntityKnowledgeStore 5 | from src.memory.types import KnowledgeMemoryItem, MemoryItem 6 | 7 | 8 | class TestEntityKnowledgeStore(unittest.TestCase): 9 | 10 | def setUp(self): 11 | self.file_name = "tests/memory/test_knowledge_memory.json" 12 | self.entity_knowledge_store = EntityKnowledgeStore( 13 | file_name=self.file_name) 14 | 15 | def tearDown(self): 16 | # Clean up test file after each test 17 | try: 18 | os.remove(self.file_name) 19 | except FileNotFoundError: 20 | pass 21 | 22 | def test_add_memory(self): 23 | data = [ 24 | MemoryItem("test_entity", 25 | datetime.now().replace(microsecond=0)) 26 | ] 27 | self.entity_knowledge_store.add_memory(data) 28 | assert len(self.entity_knowledge_store.knowledge_memory) == 1 29 | assert isinstance(self.entity_knowledge_store.knowledge_memory[0], 30 | KnowledgeMemoryItem) 31 | 32 | def test_convert_memory_to_knowledge_memory(self): 33 | data = [ 34 | MemoryItem("test_entity", 35 | datetime.now().replace(microsecond=0)) 36 | ] 37 | converted_data = self.entity_knowledge_store._convert_memory_to_knowledge_memory( 38 | data) 39 | assert len(converted_data) == 1 40 | assert isinstance(converted_data[0], KnowledgeMemoryItem) 41 | 42 | def test_update_knowledge_memory(self): 43 | data = [ 44 | KnowledgeMemoryItem("knowledge_entity", 1, 45 | datetime.now().replace(microsecond=0)) 46 | ] 47 | self.entity_knowledge_store._update_knowledge_memory(data) 48 | assert len(self.entity_knowledge_store.knowledge_memory) == 1 49 | assert self.entity_knowledge_store.knowledge_memory[0] == data[0] 50 | -------------------------------------------------------------------------------- /docs/update_021924.md: -------------------------------------------------------------------------------- 1 | ## Update 2 | - Implemented first version of Knowledge Graph RAG 3 | - LlamaIndex for graph framework 4 | - Nebula Graph for graph store 5 | - Found possibility for integrating memory stream in LlamaIndex 6 | - We can modify [ChatMemoryBuffer](https://github.com/run-llama/llama_index/blob/bad3afa8ba5d43b4b2f8d81d8e04bf7d6f52f9b0/llama-index-core/llama_index/core/memory/chat_memory_buffer.py#L15) interface to store the number of requested words and dates. 7 | - Experienced both graph stores; Neo4j and Nebula 8 | - Neo4j 9 | - It was easy to visualize knowledge graphs on web page 10 | - You can't visualize the graph on notebook jupyter directly. 11 | - Nebula 12 | - You can visulaize the graph on notebook jupyter. 13 | - It is incompatible with [Knowledge Graph Retriever](https://docs.llamaindex.ai/en/stable/api_reference/query/retrievers/kg.html#module-llama_index.core.indices.knowledge_graph.retrievers) on LlamaIndex v0.10.6. I should try to update the version. 14 | - Tested integrating [Chat engine with streamlit](https://colab.research.google.com/drive/1tLjOg2ZQuIClfuWrAC2LdiZHCov8oUbs#scrollTo=vYprhy09rVf0) for displaying chats and newly generated subgraphs. But I found an undefined error in llama-index. I think LlamaIndex is unstable yet and they are working on that agilely. 15 | - Check the validity of RAG codes on Harry Potter book writings. 16 | ``` 17 | %%ngql 18 | USE harry_potter_example; 19 | MATCH p=(n)-[*1..2]-() 20 | WHERE id(n) IN ['Dursley', 'Owls', 'Berkeley'] 21 | RETURN p LIMIT 100; 22 | ``` 23 | I asked about the relationship of Dursley, Owls and Berkeley. 24 | - Extract chapter 1 and 2 => N=10, E=7 25 | ![pic1](src/chapter1_2.png) 26 | - Extract chatper 1, 2 and 3 => N=11, E=9 27 | ![pic1](src/chapter1_2_3.png) 28 | 29 | 30 | 31 | ## Todo 32 | - [x] Modify [Nebula_graph_store_tutorial.ipynb](../Nebula_graph_store_tutorial.ipynb) to be easily produce Knowledge graphs 33 | - [ ] Implement the memory stream structures on ChatMemoryBuffer 34 | - [x] Test Neo4j of compatible with Knowledge Graph Retriever -------------------------------------------------------------------------------- /src/agent/llm_api/tools.py: -------------------------------------------------------------------------------- 1 | # From MemGPT llm_api_tools.py: 2 | 3 | import urllib 4 | import logging 5 | import requests 6 | 7 | 8 | def smart_urljoin(base_url, relative_url): 9 | """urljoin is stupid and wants a trailing / at the end of the endpoint address, or it will chop the suffix off""" 10 | if not base_url.endswith("/"): 11 | base_url += "/" 12 | return urllib.parse.urljoin(base_url, relative_url) 13 | 14 | 15 | def openai_chat_completions_request(url, api_key, data): 16 | """https://platform.openai.com/docs/guides/text-generation?lang=curl""" 17 | 18 | url = smart_urljoin(url, "chat/completions") 19 | headers = { 20 | "Content-Type": "application/json", 21 | "Authorization": f"Bearer {api_key}" 22 | } 23 | 24 | logging.info(f"Sending request to {url}") 25 | try: 26 | # Example code to trigger a rate limit response: 27 | # mock_response = requests.Response() 28 | # mock_response.status_code = 429 29 | # http_error = requests.exceptions.HTTPError("429 Client Error: Too Many Requests") 30 | # http_error.response = mock_response 31 | # raise http_error 32 | 33 | # Example code to trigger a context overflow response (for an 8k model) 34 | # data["messages"][-1]["content"] = " ".join(["repeat after me this is not a fluke"] * 1000) 35 | 36 | response = requests.post(url, headers=headers, json=data) 37 | logging.info(f"response = {response}") 38 | response.raise_for_status() # Raises HTTPError for 4XX/5XX status 39 | response = response.json() # convert to dict from string 40 | logging.info(f"response.json = {response}") 41 | # response = ChatCompletionResponse(**response) # convert to 'dot-dict' style which is the openai python client default 42 | return response 43 | except requests.exceptions.HTTPError as http_err: 44 | # Handle HTTP errors (e.g., response 4XX, 5XX) 45 | logging.error(f"Got HTTPError, exception={http_err}, payload={data}") 46 | raise http_err 47 | except requests.exceptions.RequestException as req_err: 48 | # Handle other requests-related errors (e.g., connection error) 49 | logging.warning(f"Got RequestException, exception={req_err}") 50 | raise req_err 51 | except Exception as e: 52 | # Handle other potential errors 53 | logging.warning(f"Got unknown Exception, exception={e}") 54 | raise e 55 | -------------------------------------------------------------------------------- /tests/memory/test_memory_stream.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | import os 3 | from datetime import datetime, timedelta 4 | from src.memory import MemoryStream 5 | from src.memory.types import MemoryItem 6 | 7 | class TestMemoryStream(unittest.TestCase): 8 | def setUp(self): 9 | self.file_name = "tests/memory/test_memory.json" 10 | self.memory_stream = MemoryStream(file_name=self.file_name) 11 | 12 | def tearDown(self): 13 | # Clean up test file after each test 14 | try: 15 | os.remove(self.file_name) 16 | except FileNotFoundError: 17 | pass 18 | 19 | def test_add_memory(self): 20 | data = [MemoryItem("test_entity", datetime.now().replace(microsecond=0))] 21 | self.memory_stream.add_memory(data) 22 | self.assertEqual(len(self.memory_stream), 1) 23 | self.assertEqual(self.memory_stream.get_memory()[0], data[0]) 24 | 25 | def test_remove_old_memory(self): 26 | past_date = datetime.now().replace(microsecond=0) - timedelta(days=10) 27 | self.memory_stream.add_memory([MemoryItem("old_entity", past_date)]) 28 | self.memory_stream.remove_old_memory(5) 29 | self.assertEqual(len(self.memory_stream.get_memory()), 0) 30 | 31 | def test_save_and_load_memory(self): 32 | data = [MemoryItem("test_entity", datetime.now().replace(microsecond=0))] 33 | self.memory_stream.add_memory(data) 34 | self.memory_stream.save_memory() 35 | new_memory_stream = MemoryStream(file_name=self.file_name) 36 | self.assertEqual(len(new_memory_stream), len(self.memory_stream)) 37 | self.assertEqual(new_memory_stream.get_memory(), self.memory_stream.get_memory()) 38 | 39 | def test_get_memory_by_index(self): 40 | data = [MemoryItem("entity1", datetime.now().replace(microsecond=0)), MemoryItem("entity2", datetime.now().replace(microsecond=0))] 41 | self.memory_stream.add_memory(data) 42 | self.assertEqual(self.memory_stream.get_memory_by_index(1), data[1]) 43 | 44 | def test_remove_memory_by_index(self): 45 | data = [MemoryItem("entity1", datetime.now().replace(microsecond=0)), MemoryItem("entity2", datetime.now().replace(microsecond=0))] 46 | self.memory_stream.add_memory(data) 47 | self.memory_stream.remove_memory_by_index(0) 48 | self.assertEqual(len(self.memory_stream), 1) 49 | self.assertEqual(self.memory_stream.get_memory()[0], data[1]) 50 | 51 | if __name__ == '__main__': 52 | unittest.main() 53 | -------------------------------------------------------------------------------- /src/agent/chat_agent.py: -------------------------------------------------------------------------------- 1 | import tiktoken 2 | from typing import Optional, List 3 | 4 | from src.agent.base_agent import Agent 5 | from src.agent.data_types import Context 6 | 7 | 8 | def count_tokens(s: str, model: str = "gpt-4") -> int: 9 | encoding = tiktoken.encoding_for_model(model) 10 | return len(encoding.encode(s)) 11 | 12 | 13 | class ChatAgent(Agent): 14 | 15 | def __init__(self, name, memory_stream_json, entity_knowledge_store_json, 16 | system_persona_txt, user_persona_txt, past_chat_json): 17 | super().__init__(name, memory_stream_json, entity_knowledge_store_json, 18 | system_persona_txt, user_persona_txt, past_chat_json) 19 | self.total_tokens = count_tokens(str(self.message.llm_message)) 20 | 21 | def add_chat(self, 22 | role: str, 23 | content: str, 24 | entities: Optional[List[str]] = None): 25 | """Add a chat to the agent's memory. 26 | 27 | Args: 28 | role (str): 'system' or 'user' 29 | content (str): content of the chat 30 | entities (Optional[List[str]], optional): entities from Memory systems. Defaults to None. 31 | """ 32 | # Add a chat to the agent's memory. 33 | self._add_contexts_to_llm_message(role, content) 34 | 35 | if entities: 36 | self.memory_stream.add_memory(entities) 37 | self.memory_stream.save_memory() 38 | self.entity_knowledge_store.add_memory( 39 | self.memory_stream.get_memory()) 40 | self.entity_knowledge_store.save_memory() 41 | 42 | self._replace_memory_from_llm_message() 43 | self._replace_eks_to_from_message() 44 | 45 | def get_chat(self): 46 | return self.contexts 47 | 48 | def _add_contexts_to_llm_message(self, role, content): 49 | """Add contexts to the llm_message.""" 50 | self.message.llm_message["messages"].append(Context(role, content)) 51 | 52 | def _replace_memory_from_llm_message(self): 53 | """Replace the memory_stream from the llm_message.""" 54 | self.message.llm_message[ 55 | "memory_stream"] = self.memory_stream.get_memory() 56 | 57 | def _replace_eks_to_from_message(self): 58 | """Replace the entity knowledge store from the llm_message. 59 | eks = entity knowledge store""" 60 | 61 | self.message.llm_message[ 62 | "knowledge_entity_store"] = self.entity_knowledge_store.get_memory( 63 | ) 64 | -------------------------------------------------------------------------------- /src/memory/base_memory.py: -------------------------------------------------------------------------------- 1 | import json 2 | import logging 3 | 4 | from abc import ABC, abstractmethod 5 | from datetime import datetime, timedelta 6 | 7 | 8 | class BaseMemory(ABC): 9 | 10 | def __init__(self, file_name: str, entity: str = None): 11 | """Initializes the memory storage.""" 12 | self.file_name = file_name 13 | self.entity = entity 14 | self.memory = [] 15 | self.knowledge_memory = [] 16 | self.init_memory() 17 | 18 | @abstractmethod 19 | def __len__(self): 20 | """Returns the number of items in the memory.""" 21 | pass 22 | 23 | @abstractmethod 24 | def init_memory(self): 25 | """Initializes memory.""" 26 | pass 27 | 28 | @abstractmethod 29 | def load_memory_from_file(self): 30 | """Loads memory from a file.""" 31 | pass 32 | 33 | @abstractmethod 34 | def add_memory(self, data): 35 | """Adds new memory data.""" 36 | pass 37 | 38 | @abstractmethod 39 | def get_memory(self): 40 | pass 41 | 42 | @property 43 | def return_memory(self): 44 | return self.memory 45 | 46 | def remove_old_memory(self, days): 47 | """Removes memory items older than a specified number of days.""" 48 | cutoff_date = datetime.now() - timedelta(days=days) 49 | self.memory = [ 50 | item for item in self.return_memory if item.date >= cutoff_date 51 | ] 52 | logging.info("Old memory removed successfully.") 53 | 54 | def save_memory(self): 55 | if self.file_name: 56 | with open(self.file_name, 'w') as file: 57 | json.dump([item.to_dict() for item in self.return_memory], 58 | file, 59 | default=str, 60 | indent=4) 61 | logging.info(f"Memory saved to {self.file_name} successfully.") 62 | else: 63 | logging.info("No file name provided. Memory not saved.") 64 | 65 | def get_memory_by_index(self, index): 66 | if 0 <= index < len(self.memory): 67 | return self.memory[index] 68 | else: 69 | return None 70 | 71 | def remove_memory_by_index(self, index): 72 | if 0 <= index < len(self.memory): 73 | del self.memory[index] 74 | logging.info("Memory item removed successfully.") 75 | return True 76 | else: 77 | logging.info("Invalid index. Memory item not removed.") 78 | return False 79 | -------------------------------------------------------------------------------- /recursive_retrieval/langchain_retrieval/retrieve.py: -------------------------------------------------------------------------------- 1 | import os 2 | from dotenv import load_dotenv 3 | from langchain_community.graphs import Neo4jGraph 4 | from langchain_community.vectorstores import Neo4jVector 5 | from langchain_openai import OpenAIEmbeddings 6 | from langchain_core.pydantic_v1 import BaseModel, Field 7 | from typing import List 8 | from langchain_core.prompts import ChatPromptTemplate 9 | from langchain_openai import ChatOpenAI 10 | from langchain_community.vectorstores.neo4j_vector import remove_lucene_chars 11 | 12 | 13 | class Entities(BaseModel): 14 | """Identifying information about entities.""" 15 | 16 | names: List[str] = Field( 17 | ..., 18 | description="All the person, organization, or business entities that appear in the text", 19 | ) 20 | 21 | def structured_retriever(question: str) -> str: 22 | """ 23 | Collects the neighborhood of entities mentioned 24 | in the question 25 | """ 26 | result = "" 27 | entities = entity_chain.invoke({"question": question}) 28 | for entity in entities.names: 29 | response = graph.query( 30 | """CALL db.index.fulltext.queryNodes('entity', $query, 31 | {limit:2}) 32 | YIELD node,score 33 | CALL { 34 | MATCH (node)-[r:!MENTIONS]->(neighbor) 35 | RETURN node.id + ' - ' + type(r) + ' -> ' + neighbor.id AS 36 | output 37 | UNION 38 | MATCH (node)<-[r:!MENTIONS]-(neighbor) 39 | RETURN neighbor.id + ' - ' + type(r) + ' -> ' + node.id AS 40 | output 41 | } 42 | RETURN output LIMIT 50 43 | """, 44 | {"query": generate_full_text_query(entity)}, 45 | ) 46 | result += "\n".join([el['output'] for el in response]) 47 | return result 48 | 49 | def generate_full_text_query(input: str) -> str: 50 | """ 51 | Generate a full-text search query for a given input string. 52 | 53 | This function constructs a query string suitable for a full-text 54 | search. It processes the input string by splitting it into words and 55 | appending a similarity threshold (~2 changed characters) to each 56 | word, then combines them using the AND operator. Useful for mapping 57 | entities from user questions to database values, and allows for some 58 | misspelings. 59 | """ 60 | full_text_query = "" 61 | words = [el for el in remove_lucene_chars(input).split() if el] 62 | for word in words[:-1]: 63 | full_text_query += f" {word}~2 AND" 64 | full_text_query += f" {words[-1]}~2" 65 | return full_text_query.strip() 66 | 67 | -------------------------------------------------------------------------------- /recursive_retrieval/benchmarking/judge.py: -------------------------------------------------------------------------------- 1 | import os 2 | from typing import Dict 3 | 4 | from dotenv import load_dotenv 5 | from langchain.prompts import PromptTemplate 6 | from langchain_core.output_parsers import JsonOutputParser 7 | from langchain_openai import OpenAI 8 | 9 | from output import Output 10 | 11 | 12 | def custom_response_evaluator(query: str, response: str) -> Dict[str, int]: 13 | load_dotenv() 14 | llm = OpenAI(openai_api_key=os.getenv("OPENAI_KEY"), temperature=0) 15 | parser = JsonOutputParser(pydantic_object=Output) 16 | 17 | template = """ 18 | You are an expert response judger. Score the provided respose a score from 0 to 1000 with 0 being the worst and 1000 being the best. 19 | Give separate scores for conciseness, informativeness and accuracy. 20 | For conciceness, focus on how succinct the response is and how clearly the information is conveyed using minimal words. 21 | For informativeness, focus on how comprehensive and relevant the contents of the response is to the given query. 22 | For accuracy, focus on the correctness and decree of preciseness that the response has. 23 | Hold the scores you give to the highest quality with higher numbers given only to the best of the best. 24 | 25 | Here is an example: Given the query "tell me about harry potter", and the given response Harry Potter is a series of seven fantasy novels written by British author J.K. Rowling. The novels chronicle the life of a young wizard named Harry Potter and his friends Hermione Granger and Ron Weasley, who are all students at the Hogwarts School of Witchcraft and Wizardry. 26 | The series follows Harry's life from age 11 to 17. In the first book, Harry learns that his parents were killed by the dark wizard Lord Voldemort when Harry was a baby, and that Harry survived the attack with a lightning-shaped scar on his forehead. Harry then begins attending Hogwarts, where he makes friends, encounters professors like Severus Snape, and faces off against Voldemort and his followers. 27 | The Harry Potter books have been massively popular, spawning a highly successful film series that ran from 2001 to 2011, as well as a play, video games, theme park attractions, and a vast amount of fan fiction. 28 | The series is considered a landmark of 20th and 21st century children's literature. 29 | In summary, Harry Potter is a beloved fantasy series that has had a tremendous cultural impact through its books, films, and expanded universe. 30 | 31 | The judge should give a score of 300 for conciseness, 600 to informativeness, 900 for accuracy. 32 | 33 | Format: {format_instructions} 34 | 35 | Query: {query} 36 | Response: {response} 37 | """ 38 | 39 | prompt = PromptTemplate( 40 | template=template, 41 | input_variables=["query", "response"], 42 | partial_variables={"format_instructions": parser.get_format_instructions()}, 43 | ) 44 | 45 | chain = prompt | llm | parser 46 | result = chain.invoke({"query": query, "response": response}) 47 | return result 48 | -------------------------------------------------------------------------------- /src/agent/data_types.py: -------------------------------------------------------------------------------- 1 | import json 2 | import logging 3 | from dataclasses import dataclass 4 | 5 | # ChatGPT token limits 6 | CONTEXT_LENGTH = 4096 7 | 8 | 9 | def save_json(filename, data): 10 | with open(filename, "w") as f: 11 | json.dump(data, f, indent=4) 12 | 13 | 14 | @dataclass 15 | class Context: 16 | """Context class to store the role and content of the message.""" 17 | role: str # system or user 18 | content: str 19 | 20 | def __str__(self): 21 | return f"{self.role}: {self.content} |" 22 | 23 | def to_dict(self): 24 | return {'role': self.role, 'content': self.content} 25 | 26 | 27 | class Message: 28 | """Message class to store the contexts, memory stream and knowledge entity store.""" 29 | 30 | def __init__(self, system_persona_txt, user_persona_txt, past_chat_json): 31 | self.contexts = [] 32 | self.system_persona = self.load_persona(system_persona_txt) 33 | self.user_persona = self.load_persona(user_persona_txt) 34 | self._init_persona_to_messages() 35 | self.contexts.extend(self.load_contexts_from_json(past_chat_json)) 36 | 37 | self.llm_message = { 38 | "model": "gpt-3.5-turbo", 39 | "messages": self.contexts, 40 | "memory_stream": [], 41 | "knowledge_entity_store": [] 42 | } 43 | 44 | # self.prompt_tokens = count_tokens(self.contexts) 45 | 46 | def __str__(self): 47 | llm_message_str = f"System Persona: {self.system_persona}\nUser Persona: {self.user_persona}\n" 48 | for context in self.contexts: 49 | llm_message_str += f"{str(context)}," 50 | for memory in self.llm_message["memory_stream"]: 51 | llm_message_str += f"{str(memory)}," 52 | for entity in self.llm_message["knowledge_entity_store"]: 53 | llm_message_str += f"{str(entity)}," 54 | return llm_message_str 55 | 56 | def _init_persona_to_messages(self): 57 | """Initializes the system and user personas to the contexts.""" 58 | self.contexts.append(Context("system", self.system_persona)) 59 | self.contexts.append(Context("user", self.user_persona)) 60 | 61 | def load_persona(self, persona_txt) -> str: 62 | """Loads the persona from the txt file. 63 | 64 | Args: 65 | persona_txt (str): persona txt file path 66 | 67 | Returns: 68 | str: persona 69 | """ 70 | try: 71 | with open(persona_txt, "r") as file: 72 | persona = file.read() 73 | return persona 74 | except FileNotFoundError: 75 | logging.info(f"{persona_txt} file does not exist.") 76 | 77 | def load_contexts_from_json(self, past_chat_json): 78 | """Loads the contexts from the past chat json file.""" 79 | try: 80 | with open(past_chat_json, "r") as file: 81 | data_dicts = json.load(file) 82 | 83 | return [Context(**data_dict) for data_dict in data_dicts] 84 | except: 85 | logging.info( 86 | f"{past_chat_json} file does not exist. Starts from empty contexts." 87 | ) 88 | return [] 89 | -------------------------------------------------------------------------------- /src/memory/entity_knowledge_store.py: -------------------------------------------------------------------------------- 1 | import json 2 | import logging 3 | 4 | from src.memory import BaseMemory 5 | from src.memory.types import KnowledgeMemoryItem, MemoryItem 6 | 7 | 8 | class EntityKnowledgeStore(BaseMemory): 9 | 10 | def __len__(self): 11 | """Returns the number of items in the memory.""" 12 | return len(self.knowledge_memory) 13 | 14 | def init_memory(self): 15 | """Initializes memory. 16 | self.entity_memory: list[EntityMemoryItem] 17 | """ 18 | self.load_memory_from_file() 19 | if self.entity: 20 | self.add_memory(self.entity) 21 | 22 | @property 23 | def return_memory(self): 24 | return self.knowledge_memory 25 | 26 | def load_memory_from_file(self): 27 | try: 28 | with open(self.file_name, 'r') as file: 29 | self.memory = [ 30 | KnowledgeMemoryItem.from_dict(item) 31 | for item in json.load(file) 32 | ] 33 | logging.info( 34 | f"Entity Knowledge Memory loaded from {self.file_name} successfully." 35 | ) 36 | except FileNotFoundError: 37 | logging.info( 38 | "File not found. Starting with an empty entity knowledge memory." 39 | ) 40 | 41 | def add_memory(self, memory_stream: list[MemoryItem]): 42 | """To add new memory to the entity knowledge store 43 | we should convert the memory to knowledge memory and then update the knowledge memory 44 | 45 | Args: 46 | memory_stream (list): list of MemoryItem 47 | """ 48 | knowledge_meory = self._convert_memory_to_knowledge_memory( 49 | memory_stream) 50 | self._update_knowledge_memory(knowledge_meory) 51 | 52 | def _update_knowledge_memory(self, knowledge_memory: list): 53 | """update self.knowledge memory with new knowledge memory items 54 | 55 | Args: 56 | knowledge_memory (list): list of KnowledgeMemoryItem 57 | """ 58 | for item in knowledge_memory: 59 | for i, entity in enumerate(self.knowledge_memory): 60 | if entity.entity == item.entity: 61 | self.knowledge_memory[i].date = item.date 62 | self.knowledge_memory[i].count += item.count 63 | break 64 | else: 65 | self.knowledge_memory.append(item) 66 | 67 | def _convert_memory_to_knowledge_memory( 68 | self, memory_stream: list) -> list[KnowledgeMemoryItem]: 69 | """Converts memory to knowledge memory 70 | 71 | Returns: 72 | knowledge_memory (list): list of KnowledgeMemoryItem 73 | """ 74 | knowledge_memory = [] 75 | 76 | entities = set([item.entity for item in memory_stream]) 77 | for entity in entities: 78 | memory_dates = [ 79 | item.date for item in memory_stream if item.entity == entity 80 | ] 81 | knowledge_memory.append( 82 | KnowledgeMemoryItem(entity, len(memory_dates), 83 | max(memory_dates))) 84 | return knowledge_memory 85 | 86 | def get_memory(self) -> list[KnowledgeMemoryItem]: 87 | return self.knowledge_memory 88 | -------------------------------------------------------------------------------- /recursive_retrieval/docs/updates.md: -------------------------------------------------------------------------------- 1 | # Recursive Retrieval Updates 2 | ## Update 4/6/24 3 | - LLM "Judge" 4 | - analyze quality of perplexity models (sonar-small-online, mistral, mixtral) on conciseness, informativeness, and accuracy 5 | - potential improvements should focus on quality of the prompt fed into GPT3.5 (or use GPT4) 6 | - results indicate sonar is best, but due to cost, mistral is best option for filling KG 7 | - Incorporate Mistral model for external querying in streamlit app 8 | 9 | ## Update 3/21/24 10 | - Custom synonym expansion 11 | - leveraging llm to produce larger range of synonyms of key entities found in query 12 | - custom formatting to fit neo4j 13 | - seems to dramatically improve web query → finding in graph afterwards using recursive retrieval 14 | - further testing to fine tune the prompt for increased accuracy 15 | 16 | ## Update 3/19/24 17 | - Enhancing retrieval with llm 18 | - passing in llm to `KnowledgeGraphRAGRetriever` allows model to make jumps in entity extraction process (ie return entities not specifically mentioned) 19 | - ex. `query`: Tell me about Sirius Black. `entities`: ['Harry Potter'] (assuming Sirius Black is not an existing entity) 20 | - passing in llm to `RetrieverQueryEngine` allows model to directly mix in info from llm and KG in response 21 | - overall behavior is a bit nondeterministic 22 | - likely best to not use llm on either step in order to ensure information gets added to KG 23 | - Custom entity extraction 24 | - under entity_extraction folder 25 | - work in progress - uses LangChain to pipe prompt (with template) → llm → pydantic json format 26 | - issue in custom function not being called, look into if there's time 27 | - External querying using Perplexity now works 28 | - run entirety of external_perplexity.ipynb with new query 29 | - need to test effectiveness 30 | 31 | ## Update 3/14/24 32 | - Set up querying of web corpus using Gemma, but through locally hosted model using Ollama 33 | - no API calls on external server that could be easily integrated with llama index frameworks 34 | - initial observation is that it is much slower to slower to locally host 35 | - Ollama can be used to host all of the open source models as well 36 | - very easy Ollama integration with llama index 37 | - using https://docs.llamaindex.ai/en/latest/examples/llm/ollama.html 38 | 39 | ## Update 3/13/24 40 | - Set up queyring of web corpus using Perplexity 41 | - really great optionality in specific model to use 42 | - mistral-7b-instruct 43 | - mixtral-8x7b-instruct 44 | - sonar-small-chat (perplexity model) 45 | - others found here: https://docs.perplexity.ai/docs/model-cards 46 | - interesting update from perplexity recently: *Deprecation and removal of codellama-34b-instruct and llama-2-70b-chat* 47 | - test results under /tests 48 | - Overall querying external source and adding to KG is successful 49 | - When using RAG retrieval, hard to identify key entities at times for newly added nodes 50 | - Looking into why the retrieval process is a little buggy 51 | 52 | ## Update 3/12/24 53 | - Set up querying of web corpus using Llama 2 54 | - information is succesfully injected into KG 55 | - when querying afterwards, RAG retreiver and index both have a hard time identifying the new topics being asked as key entities (index has some better luck) 56 | - Looking into LangChain + Diffbot for alternative options 57 | - Investigating why key entities are not being identified using llama index 58 | - https://lmy.medium.com/comparing-langchain-and-llamaindex-with-4-tasks-2970140edf33 -------------------------------------------------------------------------------- /recursive_retrieval/benchmarking/ollama_quality.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 2, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "from llama_index.llms.ollama import Ollama\n", 10 | "\n", 11 | "llm = Ollama(model=\"gemma\", request_timeout=30.0)" 12 | ] 13 | }, 14 | { 15 | "cell_type": "code", 16 | "execution_count": 3, 17 | "metadata": {}, 18 | "outputs": [ 19 | { 20 | "name": "stdout", 21 | "output_type": "stream", 22 | "text": [ 23 | "Harry Potter is a beloved fantasy novel series written by J.K. Rowling. The story follows the journey of Harry Potter, an orphaned boy who discovers on his eleventh birthday that he is a wizard. He is invited to Hogwarts School of Witchcraft and Wizardry, where he meets his best friends, Ron Weasley and Hermione Granger.\n", 24 | "\n", 25 | "**The story:**\n", 26 | "\n", 27 | "* **The Boy Who Lived:** The first book, \"Harry Potter and the Sorcerer's Stone,\" tells the story of Harry's first year at Hogwarts. He learns about his magical abilities, makes new friends, and faces a mysterious villain named Voldemort.\n", 28 | "* **The Prisoner of Azkaban:** The second book, \"Harry Potter and the Prisoner of Azkaban,\" explores themes of justice and innocence.\n", 29 | "* **The Order of the Phoenix:** The third book, \"Harry Potter and the Order of the Phoenix,\" introduces the concept of the Order of the Phoenix, a secret society fighting against Voldemort.\n", 30 | "* **The Half-Blood Prince:** The fourth book, \"Harry Potter and the Half-Blood Prince,\" introduces a mysterious new teacher and explores themes of prejudice and identity.\n", 31 | "* **Deathly Hallows:** The seventh book, \"Harry Potter and the Deathly Hallows,\" brings the story to a close, with a final battle against Voldemort and the defeat of evil.\n", 32 | "\n", 33 | "**The characters:**\n", 34 | "\n", 35 | "* **Harry Potter:** The protagonist, a young orphan who discovers he is a wizard. He is brave, loyal, and always willing to fight for what he believes is right.\n", 36 | "* **Ron Weasley:** Harry's best friend, a goofy but lovable wizard who is known for his humor and unwavering loyalty.\n", 37 | "* **Hermione Granger:** Harry's other best friend, a brilliant witch with a sharp mind and a strong moral compass.\n", 38 | "* **Voldemort:** The main antagonist, a dark wizard who seeks to establish his power and control.\n", 39 | "* **Albus Dumbledore:** The headmaster of Hogwarts, a powerful and enigmatic figure.\n", 40 | "\n", 41 | "**The impact:**\n", 42 | "\n", 43 | "The Harry Potter series has been a global phenomenon, selling millions of copies worldwide. It has been adapted into numerous films, and the characters have become household names. The story has also inspired countless other writers and artists.\n", 44 | "\n", 45 | "**Overall, Harry Potter is a captivating story about love, friendship, and the power of good against evil.**\n" 46 | ] 47 | } 48 | ], 49 | "source": [ 50 | "query = \"tell me about harry potter\"\n", 51 | "\n", 52 | "external_response = llm.complete(query)\n", 53 | "print(external_response)" 54 | ] 55 | } 56 | ], 57 | "metadata": { 58 | "kernelspec": { 59 | "display_name": ".venv", 60 | "language": "python", 61 | "name": "python3" 62 | }, 63 | "language_info": { 64 | "codemirror_mode": { 65 | "name": "ipython", 66 | "version": 3 67 | }, 68 | "file_extension": ".py", 69 | "mimetype": "text/x-python", 70 | "name": "python", 71 | "nbconvert_exporter": "python", 72 | "pygments_lexer": "ipython3", 73 | "version": "3.11.6" 74 | } 75 | }, 76 | "nbformat": 4, 77 | "nbformat_minor": 2 78 | } 79 | -------------------------------------------------------------------------------- /recursive_retrieval/langchain_retrieval/retrieve.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 12, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import os\n", 10 | "from dotenv import load_dotenv\n", 11 | "\n", 12 | "load_dotenv()\n", 13 | "\n", 14 | "os.environ[\"OPENAI_API_KEY\"] = os.getenv('OPENAI_KEY')\n", 15 | "os.environ[\"NEO4J_URI\"] = os.getenv('NEO4J_URL')\n", 16 | "os.environ[\"NEO4J_USERNAME\"] = \"neo4j\"\n", 17 | "os.environ[\"NEO4J_PASSWORD\"] = os.getenv('NEO4J_PW')\n", 18 | "os.environ[\"NEO4J_DATABSE\"] = \"neo4j\"" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": 13, 24 | "metadata": {}, 25 | "outputs": [], 26 | "source": [ 27 | "from langchain_openai import ChatOpenAI\n", 28 | "from langchain_community.graphs import Neo4jGraph\n", 29 | "\n", 30 | "# url = os.getenv('NEO4J_URL')\n", 31 | "# username = \"neo4j\"\n", 32 | "# password = os.getenv('NEO4J_PW')\n", 33 | "# database = \"neo4j\"\n", 34 | "\n", 35 | "llm=ChatOpenAI(temperature=0, model_name=\"gpt-3.5-turbo\")\n", 36 | "graph = Neo4jGraph()" 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "execution_count": 14, 42 | "metadata": {}, 43 | "outputs": [], 44 | "source": [ 45 | "from langchain_core.prompts import ChatPromptTemplate\n", 46 | "from retrieve import Entities\n", 47 | "\n", 48 | "prompt = ChatPromptTemplate.from_messages(\n", 49 | " [\n", 50 | " (\n", 51 | " \"system\",\n", 52 | " \"You are extracting organization and person entities from the text.\",\n", 53 | " ),\n", 54 | " (\n", 55 | " \"human\",\n", 56 | " \"Use the given format to extract information from the following\"\n", 57 | " \"input: {question}\",\n", 58 | " ),\n", 59 | " ]\n", 60 | ")\n", 61 | "\n", 62 | "entity_chain = prompt | llm.with_structured_output(Entities)" 63 | ] 64 | }, 65 | { 66 | "cell_type": "code", 67 | "execution_count": 15, 68 | "metadata": {}, 69 | "outputs": [ 70 | { 71 | "data": { 72 | "text/plain": [ 73 | "[]" 74 | ] 75 | }, 76 | "execution_count": 15, 77 | "metadata": {}, 78 | "output_type": "execute_result" 79 | } 80 | ], 81 | "source": [ 82 | "from langchain_community.vectorstores import Neo4jVector\n", 83 | "from langchain_openai import OpenAIEmbeddings\n", 84 | "\n", 85 | "vector_index = Neo4jVector.from_existing_graph(\n", 86 | " OpenAIEmbeddings(),\n", 87 | " search_type=\"hybrid\",\n", 88 | " node_label=\"Document\",\n", 89 | " text_node_properties=[\"text\"],\n", 90 | " embedding_node_property=\"embedding\"\n", 91 | ")\n", 92 | "\n", 93 | "graph.query(\"CREATE FULLTEXT INDEX entity IF NOT EXISTS FOR (e:__Entity__) ON EACH [e.id]\")\n" 94 | ] 95 | }, 96 | { 97 | "cell_type": "code", 98 | "execution_count": 17, 99 | "metadata": {}, 100 | "outputs": [ 101 | { 102 | "name": "stdout", 103 | "output_type": "stream", 104 | "text": [ 105 | "\n" 106 | ] 107 | } 108 | ], 109 | "source": [ 110 | "from retrieve import structured_retriever\n", 111 | "\n", 112 | "print(structured_retriever(\"Tell me about Austin\"))" 113 | ] 114 | } 115 | ], 116 | "metadata": { 117 | "kernelspec": { 118 | "display_name": ".venv", 119 | "language": "python", 120 | "name": "python3" 121 | }, 122 | "language_info": { 123 | "codemirror_mode": { 124 | "name": "ipython", 125 | "version": 3 126 | }, 127 | "file_extension": ".py", 128 | "mimetype": "text/x-python", 129 | "name": "python", 130 | "nbconvert_exporter": "python", 131 | "pygments_lexer": "ipython3", 132 | "version": "3.11.6" 133 | } 134 | }, 135 | "nbformat": 4, 136 | "nbformat_minor": 2 137 | } 138 | -------------------------------------------------------------------------------- /recursive_retrieval/benchmarking/perplexity_test.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 25, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import os\n", 10 | "from llama_index.llms.perplexity import Perplexity\n", 11 | "from dotenv import load_dotenv\n", 12 | "\n", 13 | "load_dotenv()\n", 14 | "\n", 15 | "pplx_api_key = os.getenv(\"PERPLEXITY_API_KEY\")\n", 16 | "\n", 17 | "llm = Perplexity(\n", 18 | " api_key=pplx_api_key, model=\"mistral-7b-instruct\", temperature=0.5\n", 19 | ")" 20 | ] 21 | }, 22 | { 23 | "cell_type": "code", 24 | "execution_count": 26, 25 | "metadata": {}, 26 | "outputs": [ 27 | { 28 | "name": "stdout", 29 | "output_type": "stream", 30 | "text": [ 31 | "1.55 s ± 192 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n" 32 | ] 33 | } 34 | ], 35 | "source": [ 36 | "%%timeit\n", 37 | "from llama_index.core.llms import ChatMessage\n", 38 | "\n", 39 | "messages_dict = [\n", 40 | " {\"role\": \"system\", \"content\": \"Be precise and concise.\"},\n", 41 | " {\"role\": \"user\", \"content\": \"tell me about harry potter\"},\n", 42 | "]\n", 43 | "messages = [ChatMessage(**msg) for msg in messages_dict]\n", 44 | "\n", 45 | "external_response = llm.chat(messages)" 46 | ] 47 | }, 48 | { 49 | "cell_type": "code", 50 | "execution_count": 18, 51 | "metadata": {}, 52 | "outputs": [], 53 | "source": [ 54 | "llm = Perplexity(\n", 55 | " api_key=pplx_api_key, model=\"mixtral-8x7b-instruct\", temperature=0.5\n", 56 | ")" 57 | ] 58 | }, 59 | { 60 | "cell_type": "code", 61 | "execution_count": 19, 62 | "metadata": {}, 63 | "outputs": [ 64 | { 65 | "name": "stdout", 66 | "output_type": "stream", 67 | "text": [ 68 | "1.36 s ± 132 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n" 69 | ] 70 | } 71 | ], 72 | "source": [ 73 | "%%timeit\n", 74 | "from llama_index.core.llms import ChatMessage\n", 75 | "\n", 76 | "messages_dict = [\n", 77 | " {\"role\": \"system\", \"content\": \"Be precise and concise.\"},\n", 78 | " {\"role\": \"user\", \"content\": \"tell me about harry potter\"},\n", 79 | "]\n", 80 | "messages = [ChatMessage(**msg) for msg in messages_dict]\n", 81 | "\n", 82 | "external_response = llm.chat(messages)" 83 | ] 84 | }, 85 | { 86 | "cell_type": "code", 87 | "execution_count": 22, 88 | "metadata": {}, 89 | "outputs": [], 90 | "source": [ 91 | "llm = Perplexity(\n", 92 | " api_key=pplx_api_key, model=\"sonar-small-chat\", temperature=0.5\n", 93 | ")" 94 | ] 95 | }, 96 | { 97 | "cell_type": "code", 98 | "execution_count": 23, 99 | "metadata": {}, 100 | "outputs": [ 101 | { 102 | "name": "stdout", 103 | "output_type": "stream", 104 | "text": [ 105 | "1.01 s ± 150 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n" 106 | ] 107 | } 108 | ], 109 | "source": [ 110 | "%%timeit\n", 111 | "from llama_index.core.llms import ChatMessage\n", 112 | "\n", 113 | "messages_dict = [\n", 114 | " {\"role\": \"system\", \"content\": \"Be precise and concise.\"},\n", 115 | " {\"role\": \"user\", \"content\": \"tell me about harry potter\"},\n", 116 | "]\n", 117 | "messages = [ChatMessage(**msg) for msg in messages_dict]\n", 118 | "\n", 119 | "external_response = llm.chat(messages)" 120 | ] 121 | } 122 | ], 123 | "metadata": { 124 | "kernelspec": { 125 | "display_name": ".venv", 126 | "language": "python", 127 | "name": "python3" 128 | }, 129 | "language_info": { 130 | "codemirror_mode": { 131 | "name": "ipython", 132 | "version": 3 133 | }, 134 | "file_extension": ".py", 135 | "mimetype": "text/x-python", 136 | "name": "python", 137 | "nbconvert_exporter": "python", 138 | "pygments_lexer": "ipython3", 139 | "version": "3.11.6" 140 | } 141 | }, 142 | "nbformat": 4, 143 | "nbformat_minor": 2 144 | } 145 | -------------------------------------------------------------------------------- /recursive_retrieval/benchmarking/perplexity_quality.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 2, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import os\n", 10 | "from llama_index.llms.perplexity import Perplexity\n", 11 | "from dotenv import load_dotenv\n", 12 | "\n", 13 | "load_dotenv()\n", 14 | "\n", 15 | "pplx_api_key = os.getenv(\"PERPLEXITY_API_KEY\")\n", 16 | "\n", 17 | "llm = Perplexity(\n", 18 | " api_key=pplx_api_key, model=\"mistral-7b-instruct\", temperature=0.5\n", 19 | ")" 20 | ] 21 | }, 22 | { 23 | "cell_type": "code", 24 | "execution_count": 4, 25 | "metadata": {}, 26 | "outputs": [ 27 | { 28 | "name": "stdout", 29 | "output_type": "stream", 30 | "text": [ 31 | "assistant: Harry Potter is a series of seven fantasy novels written by British author J.K. Rowling. The books follow Harry Potter, an orphaned boy who discovers he is a wizard, as he attends Hogwarts School of Witchcraft and Wizardry and learns to master his magical abilities. Throughout the series, Harry faces numerous challenges and battles against the dark wizard Lord Voldemort. The novels are known for their intricate plotting, richly detailed world-building, and complex characters. The Harry Potter series has become a cultural phenomenon and has been translated into over 80 languages. The first novel, \"Harry Potter and the Philosopher's Stone,\" was published in 1997, and the last book, \"Harry Potter and the Deathly Hallows,\" was published in 2007. The series has also been adapted into a successful film series.\n" 32 | ] 33 | } 34 | ], 35 | "source": [ 36 | "from llama_index.core.llms import ChatMessage\n", 37 | "\n", 38 | "messages_dict = [\n", 39 | " {\"role\": \"system\", \"content\": \"Be precise and concise.\"},\n", 40 | " {\"role\": \"user\", \"content\": \"tell me about harry potter\"},\n", 41 | "]\n", 42 | "messages = [ChatMessage(**msg) for msg in messages_dict]\n", 43 | "\n", 44 | "external_response = llm.chat(messages)\n", 45 | "print(external_response)" 46 | ] 47 | }, 48 | { 49 | "cell_type": "code", 50 | "execution_count": 5, 51 | "metadata": {}, 52 | "outputs": [ 53 | { 54 | "name": "stdout", 55 | "output_type": "stream", 56 | "text": [ 57 | "assistant: Harry Potter is a popular series of seven fantasy novels written by British author J.K. Rowling. The series follows the adventures of the titular character, Harry Potter, and his friends Ron Weasley and Hermione Granger, as they attend Hogwarts School of Witchcraft and Wizardry and battle against the dark wizard, Lord Voldemort. The series explores themes of love, friendship, bravery, and the power of choice. It has also been adapted into a successful film franchise and stage play.\n" 58 | ] 59 | } 60 | ], 61 | "source": [ 62 | "llm = Perplexity(\n", 63 | " api_key=pplx_api_key, model=\"mixtral-8x7b-instruct\", temperature=0.5\n", 64 | ")\n", 65 | "messages_dict = [\n", 66 | " {\"role\": \"system\", \"content\": \"Be precise and concise.\"},\n", 67 | " {\"role\": \"user\", \"content\": \"tell me about harry potter\"},\n", 68 | "]\n", 69 | "messages = [ChatMessage(**msg) for msg in messages_dict]\n", 70 | "\n", 71 | "external_response = llm.chat(messages)\n", 72 | "print(external_response)" 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": 6, 78 | "metadata": {}, 79 | "outputs": [ 80 | { 81 | "name": "stdout", 82 | "output_type": "stream", 83 | "text": [ 84 | "assistant: Harry Potter is a series of seven fantasy novels written by British author J.K. Rowling, published between 1997 and 2007. The books follow the adventures of a young wizard named Harry Potter as he attends Hogwarts School of Witchcraft and Wizardry and battles against the evil Lord Voldemort.\n" 85 | ] 86 | } 87 | ], 88 | "source": [ 89 | "llm = Perplexity(\n", 90 | " api_key=pplx_api_key, model=\"sonar-small-chat\", temperature=0.5\n", 91 | ")\n", 92 | "messages_dict = [\n", 93 | " {\"role\": \"system\", \"content\": \"Be precise and concise.\"},\n", 94 | " {\"role\": \"user\", \"content\": \"tell me about harry potter\"},\n", 95 | "]\n", 96 | "messages = [ChatMessage(**msg) for msg in messages_dict]\n", 97 | "\n", 98 | "external_response = llm.chat(messages)\n", 99 | "print(external_response)" 100 | ] 101 | } 102 | ], 103 | "metadata": { 104 | "kernelspec": { 105 | "display_name": ".venv", 106 | "language": "python", 107 | "name": "python3" 108 | }, 109 | "language_info": { 110 | "codemirror_mode": { 111 | "name": "ipython", 112 | "version": 3 113 | }, 114 | "file_extension": ".py", 115 | "mimetype": "text/x-python", 116 | "name": "python", 117 | "nbconvert_exporter": "python", 118 | "pygments_lexer": "ipython3", 119 | "version": "3.11.6" 120 | } 121 | }, 122 | "nbformat": 4, 123 | "nbformat_minor": 2 124 | } 125 | -------------------------------------------------------------------------------- /streamlit_app/app.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import random 3 | import textwrap 4 | 5 | import streamlit as st 6 | import streamlit.components.v1 as components 7 | from dotenv import load_dotenv 8 | import pandas as pd 9 | from pyvis.network import Network 10 | from neo4j import GraphDatabase 11 | 12 | sys.path.append("../") 13 | sys.path.append("/home/young/project/memAry/RAG") 14 | from src.agent.chat_agent import ChatAgent 15 | 16 | load_dotenv('streamlit_app/environ.env') 17 | 18 | system_persona_txt = "data/system_persona.txt" 19 | user_persona_txt = "data/user_persona.txt" 20 | past_chat_json = "data/past_chat.json" 21 | memory_stream_json = "data/memory_stream.json" 22 | entity_knowledge_store_json = "data/entity_knowledge_store.json" 23 | chat_agent = ChatAgent("Personal Agent", 24 | memory_stream_json, 25 | entity_knowledge_store_json, 26 | system_persona_txt, 27 | user_persona_txt, 28 | past_chat_json) 29 | 30 | def create_graph(nodes, edges): 31 | g = Network( 32 | notebook=True, 33 | directed=True, 34 | cdn_resources="in_line", 35 | height="500px", 36 | width="100%", 37 | ) 38 | 39 | for node in nodes: 40 | g.add_node(node, label=node, title=node) 41 | for edge in edges: 42 | # assuming only one relationship type per edge 43 | g.add_edge(edge[0], edge[1], label=edge[2][0]) 44 | 45 | g.repulsion( 46 | node_distance=200, 47 | central_gravity=0.12, 48 | spring_length=150, 49 | spring_strength=0.05, 50 | damping=0.09, 51 | ) 52 | return g 53 | 54 | 55 | def generate_string(entities): 56 | cypher_query = 'MATCH p = (n) - [*1 .. 2] - ()\n' 57 | cypher_query += 'WHERE n.id IN ' + str(entities) + '\n' 58 | cypher_query += 'RETURN p' 59 | 60 | return cypher_query 61 | 62 | 63 | def fill_graph(nodes, edges, cypher_query): 64 | entities = [] 65 | with GraphDatabase.driver( 66 | uri=chat_agent.neo4j_url, auth=(chat_agent.neo4j_username, chat_agent.neo4j_password) 67 | ) as driver: 68 | with driver.session() as session: 69 | result = session.run(cypher_query) 70 | for record in result: 71 | path = record["p"] 72 | rels = [rel.type for rel in path.relationships] 73 | 74 | n1_id = record["p"].nodes[0]["id"] 75 | n2_id = record["p"].nodes[1]["id"] 76 | nodes.add(n1_id) 77 | nodes.add(n2_id) 78 | edges.append((n1_id, n2_id, rels)) 79 | entities.extend([n1_id, n2_id]) 80 | 81 | 82 | tab1, tab2 = st.tabs(["Knowledge Graph", "Recursive Retrieval"]) 83 | 84 | with tab1: 85 | cypher_query = """ 86 | MATCH p = (:Entity)-[r]-() RETURN p, r LIMIT 1000; 87 | """ 88 | st.title("Knowledge Graph") 89 | 90 | nodes = set() 91 | edges = [] 92 | 93 | val = st.slider( 94 | "Select a chapter to add", min_value=2, max_value=5, value=2, step=1 95 | ) 96 | add_clicked = st.button("Inject into Graph") 97 | st.text("") 98 | 99 | if add_clicked: 100 | paths = [f"data/harry_potter/{val}.txt"] 101 | chat_agent.add_chapter(paths) 102 | 103 | fill_graph(nodes, edges, cypher_query) 104 | graph = create_graph(nodes, edges) 105 | graph_html = graph.generate_html(f"graph_{random.randint(0, 1000)}.html") 106 | components.html(graph_html, height=500, scrolling=True) 107 | 108 | 109 | with tab2: 110 | cypher_query = "MATCH p = (:Entity)-[r]-() RETURN p, r LIMIT 1000;" 111 | answer = "" 112 | external_response = "" 113 | st.title("Recursive Retrieval") 114 | query = st.text_input("Ask a question") 115 | generate_clicked = st.button("Generate") 116 | 117 | st.write("") 118 | 119 | if generate_clicked: 120 | external_response = "" 121 | rag_response = "There was no information in knowledge_graph to answer your question." 122 | chat_agent.add_chat('user', query, []) 123 | if chat_agent.check_KG(query): 124 | rag_response, entities = chat_agent.get_rag_response(query, return_entity=True) 125 | chat_agent.add_chat('user', 'rag: ' + str(rag_response), entities) 126 | cypher_query = generate_string( 127 | list(list(rag_response.metadata.values())[0]["kg_rel_map"].keys()) 128 | ) 129 | else: 130 | # get response 131 | external_response = "No response found in knowledge graph, querying web instead with " 132 | external_response += chat_agent.external_query(query) 133 | display_external = textwrap.fill(external_response, width=80) 134 | st.text(display_external) 135 | # load into KG 136 | chat_agent.load_KG() 137 | answer = chat_agent.get_response() 138 | st.title("RAG Response") 139 | st.text(str(rag_response)) 140 | st.title("Perplexity Response") 141 | st.text(str(external_response)) 142 | st.title("Memory Response") 143 | st.text(str(answer)) 144 | 145 | nodes = set() 146 | edges = [] # (node1, node2, [relationships]) 147 | fill_graph(nodes, edges, cypher_query) 148 | 149 | wrapped_text = textwrap.fill(answer, width=80) 150 | st.text(wrapped_text) 151 | st.code("# Current Cypher Used\n" + cypher_query) 152 | st.write("") 153 | st.markdown("Current Subgraph Used") 154 | graph = create_graph(nodes, edges) 155 | graph_html = graph.generate_html(f"graph_{random.randint(0, 1000)}.html") 156 | components.html(graph_html, height=500, scrolling=True) 157 | 158 | if len(chat_agent.memory_stream) > 0: 159 | # Memory Stream 160 | memory_items = chat_agent.memory_stream.get_memory() 161 | memory_items_dicts = [item.to_dict() for item in memory_items] 162 | df = pd.DataFrame(memory_items_dicts) 163 | st.write("Memory Stream") 164 | st.dataframe(df) 165 | 166 | # Entity Knowledge Store 167 | knowledge_memory_items = chat_agent.entity_knowledge_store.get_memory() 168 | knowledge_memory_items_dicts = [item.to_dict() for item in knowledge_memory_items] 169 | df_knowledge = pd.DataFrame(knowledge_memory_items_dicts) 170 | st.text("Entity Knowledge Store") 171 | st.dataframe(df_knowledge) 172 | -------------------------------------------------------------------------------- /recursive_retrieval/external_perplexity.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 8, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import os\n", 10 | "from llama_index.llms.perplexity import Perplexity\n", 11 | "from llama_index.llms.openai import OpenAI\n", 12 | "from dotenv import load_dotenv\n", 13 | "\n", 14 | "load_dotenv()\n", 15 | "\n", 16 | "pplx_api_key = os.getenv(\"PERPLEXITY_API_KEY\")\n", 17 | "os.environ[\"OPENAI_API_KEY\"] = os.getenv('OPENAI_KEY')\n", 18 | "\n", 19 | "query_llm = Perplexity(\n", 20 | " api_key=pplx_api_key, model=\"sonar-small-online\", temperature=0.5\n", 21 | ")\n", 22 | "\n", 23 | "entity_llm = OpenAI(temperature=0.2, model=\"gpt-4\")" 24 | ] 25 | }, 26 | { 27 | "cell_type": "code", 28 | "execution_count": 9, 29 | "metadata": {}, 30 | "outputs": [], 31 | "source": [ 32 | "from llama_index.graph_stores.neo4j import Neo4jGraphStore\n", 33 | "from llama_index.core import StorageContext\n", 34 | "\n", 35 | "username = \"neo4j\"\n", 36 | "password = os.getenv('NEO4J_PW')\n", 37 | "url = os.getenv('NEO4J_URL')\n", 38 | "database = \"neo4j\"\n", 39 | "\n", 40 | "graph_store = Neo4jGraphStore(\n", 41 | " username=username,\n", 42 | " password=password,\n", 43 | " url=url,\n", 44 | " database=database,\n", 45 | ")\n", 46 | "\n", 47 | "storage_context = StorageContext.from_defaults(graph_store=graph_store)\n" 48 | ] 49 | }, 50 | { 51 | "cell_type": "code", 52 | "execution_count": 3, 53 | "metadata": {}, 54 | "outputs": [], 55 | "source": [ 56 | "from llama_index.core.indices.knowledge_graph.retrievers import KnowledgeGraphRAGRetriever\n", 57 | "from llama_index.core.query_engine import RetrieverQueryEngine\n", 58 | "from entity_extraction.entity_extraction import custom_entity_extract_fn\n", 59 | "from synonym_expand.synonym import custom_synonym_expand_fn\n", 60 | "\n", 61 | "graph_rag_retriever_with_nl2graphquery = KnowledgeGraphRAGRetriever(\n", 62 | " storage_context=storage_context,\n", 63 | " verbose=True,\n", 64 | " llm=entity_llm,\n", 65 | " # entity_extract_fn=custom_entity_extract_fn,\n", 66 | " retriever_mode=\"keyword\",\n", 67 | " with_nl2graphquery=True,\n", 68 | " synonym_expand_fn=custom_synonym_expand_fn\n", 69 | ")\n", 70 | "\n", 71 | "query_engine_with_nl2graphquery = RetrieverQueryEngine.from_args(\n", 72 | " graph_rag_retriever_with_nl2graphquery,\n", 73 | ")" 74 | ] 75 | }, 76 | { 77 | "cell_type": "code", 78 | "execution_count": 16, 79 | "metadata": {}, 80 | "outputs": [ 81 | { 82 | "name": "stdout", 83 | "output_type": "stream", 84 | "text": [ 85 | "assistant: Crypto refers to a digital currency designed to function as a medium of exchange through a decentralized computer network, independent of a central authority like a government or bank. It utilizes cryptography to secure transactions and maintain the integrity of the currency's supply and ownership records. Cryptocurrencies are stored in digital wallets and operate on a decentralized system known as blockchain, which is a public ledger recording all transactions. The first cryptocurrency was Bitcoin, introduced in 2009, and there are now over 25,000 other cryptocurrencies available, with varying levels of adoption and regulation.\n" 86 | ] 87 | } 88 | ], 89 | "source": [ 90 | "from llama_index.core.llms import ChatMessage\n", 91 | "\n", 92 | "query = \"tell me about crypto\"\n", 93 | "\n", 94 | "pre_response = query_engine_with_nl2graphquery.query(\n", 95 | " query,\n", 96 | ")\n", 97 | "\n", 98 | "if pre_response.metadata is None:\n", 99 | " # 1) answer needs to be queried from Llama2 endpoint\n", 100 | " messages_dict = [\n", 101 | " {\"role\": \"system\", \"content\": \"Be precise and concise.\"},\n", 102 | " {\"role\": \"user\", \"content\": query},\n", 103 | " ]\n", 104 | " messages = [ChatMessage(**msg) for msg in messages_dict]\n", 105 | "\n", 106 | " external_response = query_llm.chat(messages)\n", 107 | " print(external_response)\n", 108 | "\n", 109 | " # 2) answer needs to be stored into txt file for loading into KG\n", 110 | " with open('data/external_response.txt', 'w') as f:\n", 111 | " print(external_response, file=f)\n", 112 | "else:\n", 113 | " print(\"From Graph: \" + str(pre_response))\n", 114 | " print(list(list(pre_response.metadata.values())[0][\"kg_rel_map\"].keys()))" 115 | ] 116 | }, 117 | { 118 | "cell_type": "code", 119 | "execution_count": 10, 120 | "metadata": {}, 121 | "outputs": [], 122 | "source": [ 123 | "from llama_index.core import KnowledgeGraphIndex, SimpleDirectoryReader\n", 124 | "\n", 125 | "documents = SimpleDirectoryReader(\n", 126 | " input_files=['data/external_response.txt']\n", 127 | ").load_data()\n", 128 | "\n", 129 | "index = KnowledgeGraphIndex.from_documents(\n", 130 | " documents,\n", 131 | " storage_context=storage_context,\n", 132 | " max_triplets_per_chunk=5,\n", 133 | ")" 134 | ] 135 | }, 136 | { 137 | "cell_type": "code", 138 | "execution_count": 18, 139 | "metadata": {}, 140 | "outputs": [ 141 | { 142 | "name": "stdout", 143 | "output_type": "stream", 144 | "text": [ 145 | "Cryptocurrency is a digital currency that utilizes cryptography to secure transactions and maintain the integrity of its supply. It is designed to function as a medium of exchange and refers to digital currency.\n", 146 | "['Crypto']\n" 147 | ] 148 | } 149 | ], 150 | "source": [ 151 | "post_response = query_engine_with_nl2graphquery.query(\n", 152 | " query,\n", 153 | ")\n", 154 | "\n", 155 | "print(post_response)\n", 156 | "print(list(list(post_response.metadata.values())[0][\"kg_rel_map\"].keys()))" 157 | ] 158 | } 159 | ], 160 | "metadata": { 161 | "kernelspec": { 162 | "display_name": ".venv", 163 | "language": "python", 164 | "name": "python3" 165 | }, 166 | "language_info": { 167 | "codemirror_mode": { 168 | "name": "ipython", 169 | "version": 3 170 | }, 171 | "file_extension": ".py", 172 | "mimetype": "text/x-python", 173 | "name": "python", 174 | "nbconvert_exporter": "python", 175 | "pygments_lexer": "ipython3", 176 | "version": "3.11.6" 177 | } 178 | }, 179 | "nbformat": 4, 180 | "nbformat_minor": 2 181 | } 182 | -------------------------------------------------------------------------------- /storage_graph/ch1_2/index_store.json: -------------------------------------------------------------------------------- 1 | {"index_store/data": {"561d616d-ff4c-49aa-95d5-5438a668e83b": {"__type__": "kg", "__data__": "{\"index_id\": \"561d616d-ff4c-49aa-95d5-5438a668e83b\", \"summary\": null, \"table\": {\"Dursley\": [\"6bf676ee-cdc3-46e8-b705-e1439a6352e8\", \"637a3fb0-c704-4f4d-b83a-a12ce5704e2e\", \"3f27fe25-de91-48ff-88e3-08b649ac4fd4\"], \"Grunnings\": [\"637a3fb0-c704-4f4d-b83a-a12ce5704e2e\"], \"Secret\": [\"637a3fb0-c704-4f4d-b83a-a12ce5704e2e\"], \"Dursleys\": [\"6737c2d6-9801-4bb1-b3fb-f20d2a7fdac1\", \"b4777a86-4c06-471d-8f96-0444ea40a04c\", \"635795f3-91d8-4700-9b99-8525e317f337\", \"d9ec5ecb-abf8-488f-81f3-d488cf88837b\", \"ce87c184-7c4a-4906-a6d8-a5530611b0d9\", \"eaa2427e-1bdf-464d-b6e2-a266fa43fbdb\", \"f0f3cf08-60c7-4aef-97be-c2db7d531503\"], \"Neighbors\": [\"eaa2427e-1bdf-464d-b6e2-a266fa43fbdb\"], \"Potters\": [\"b4777a86-4c06-471d-8f96-0444ea40a04c\", \"eaa2427e-1bdf-464d-b6e2-a266fa43fbdb\"], \"Mr. dursley\": [\"70f610e4-267d-448a-bb5a-b23024ad9fab\", \"234fe3e5-a6fe-443f-9e3f-f8a87d23209b\", \"d991ae94-f78c-43a4-ae7e-fbd9a0eb98b6\", \"bd76ee93-b3e8-421e-89db-94e3e786b2c9\"], \"Back to window\": [\"234fe3e5-a6fe-443f-9e3f-f8a87d23209b\"], \"Drills\": [\"234fe3e5-a6fe-443f-9e3f-f8a87d23209b\"], \"People\": [\"b1f965be-ca77-4c20-bb01-cede1ee45226\", \"d9ec5ecb-abf8-488f-81f3-d488cf88837b\", \"6bf676ee-cdc3-46e8-b705-e1439a6352e8\", \"bd76ee93-b3e8-421e-89db-94e3e786b2c9\"], \"They\": [\"bd76ee93-b3e8-421e-89db-94e3e786b2c9\"], \"Man\": [\"bd76ee93-b3e8-421e-89db-94e3e786b2c9\"], \"Something\": [\"bd76ee93-b3e8-421e-89db-94e3e786b2c9\"], \"Group\": [\"6bf676ee-cdc3-46e8-b705-e1439a6352e8\"], \"Fear\": [\"6bf676ee-cdc3-46e8-b705-e1439a6352e8\"], \"He\": [\"d991ae94-f78c-43a4-ae7e-fbd9a0eb98b6\", \"cff90cbd-ecdf-450d-8d73-dfc8528da0a0\"], \"Harry\": [\"ce7b5c57-f781-429d-9407-8a1748590baf\", \"f8066950-076c-4840-b0cc-e8331acb4a00\", \"97ea4cd8-fe36-40e8-85e4-ccdcae99765a\", \"834d6cdc-3e39-496f-8159-dd996961bacc\", \"577c7ac6-849a-4ce5-a8f0-e21f84f38186\", \"5cb97c50-3136-4367-975d-deedd11ed28e\", \"d9ec5ecb-abf8-488f-81f3-d488cf88837b\", \"ce87c184-7c4a-4906-a6d8-a5530611b0d9\", \"9c0d1038-cd2e-4a59-b35c-5bc64ea6c693\", \"f0f3cf08-60c7-4aef-97be-c2db7d531503\", \"5ac89898-c2b9-48eb-bb50-1634937b6442\", \"d991ae94-f78c-43a4-ae7e-fbd9a0eb98b6\", \"8f80795d-de01-4f45-bf78-69d6bc9b5d45\"], \"Tabby cat\": [\"d991ae94-f78c-43a4-ae7e-fbd9a0eb98b6\"], \"Spot\": [\"70f610e4-267d-448a-bb5a-b23024ad9fab\"], \"Stranger\": [\"70f610e4-267d-448a-bb5a-b23024ad9fab\"], \"Owls\": [\"b1f965be-ca77-4c20-bb01-cede1ee45226\"], \"Unusually\": [\"b1f965be-ca77-4c20-bb01-cede1ee45226\"], \"Bonfire night\": [\"b1f965be-ca77-4c20-bb01-cede1ee45226\"], \"Sister\": [\"b4777a86-4c06-471d-8f96-0444ea40a04c\"], \"Mrs. dursley\": [\"3f27fe25-de91-48ff-88e3-08b649ac4fd4\"], \"Turning it all over\": [\"3f27fe25-de91-48ff-88e3-08b649ac4fd4\"], \"Albus dumbledore\": [\"54579fb0-4337-41b5-9748-d35bb5ffd38b\"], \"Blue eyes\": [\"54579fb0-4337-41b5-9748-d35bb5ffd38b\"], \"Cat\": [\"54579fb0-4337-41b5-9748-d35bb5ffd38b\"], \"Dumbledore\": [\"0cc21553-a8af-4d61-bb2f-913d3cd3f3c6\", \"c04ffb43-171c-4f81-b2e9-ac12b2cfb836\", \"6e5cd78c-fb99-4b24-96fa-fdf4336d0eb1\", \"ce7b5c57-f781-429d-9407-8a1748590baf\", \"97ea4cd8-fe36-40e8-85e4-ccdcae99765a\", \"c793bdf7-ee45-4167-8a44-a15ee211f6c3\", \"844caaea-9a06-4bfa-9b96-3b2b9f49f0d5\", \"5cb97c50-3136-4367-975d-deedd11ed28e\", \"57179285-4367-403b-bba8-0ca7ba42d3fd\", \"58feb06c-afe3-4f44-90fd-c4175a764b23\"], \"Put-outer\": [\"0cc21553-a8af-4d61-bb2f-913d3cd3f3c6\"], \"Street\": [\"0cc21553-a8af-4d61-bb2f-913d3cd3f3c6\"], \"Professor mcgonagall\": [\"6e5cd78c-fb99-4b24-96fa-fdf4336d0eb1\", \"c04ffb43-171c-4f81-b2e9-ac12b2cfb836\", \"c793bdf7-ee45-4167-8a44-a15ee211f6c3\", \"57179285-4367-403b-bba8-0ca7ba42d3fd\", \"10552601-6d0d-4afa-8732-24c54622257e\"], \"How did you know it was me?\": [\"10552601-6d0d-4afa-8732-24c54622257e\"], \"Patted her\": [\"c04ffb43-171c-4f81-b2e9-ac12b2cfb836\"], \"Her\": [\"5cb97c50-3136-4367-975d-deedd11ed28e\"], \"Voldemort\": [\"5cb97c50-3136-4367-975d-deedd11ed28e\"], \"We can only guess\": [\"6e5cd78c-fb99-4b24-96fa-fdf4336d0eb1\"], \"Lace handkerchief\": [\"6e5cd78c-fb99-4b24-96fa-fdf4336d0eb1\"], \"It's the best place for him\": [\"58feb06c-afe3-4f44-90fd-c4175a764b23\"], \"Mcgonagall\": [\"58feb06c-afe3-4f44-90fd-c4175a764b23\"], \"A letter\": [\"58feb06c-afe3-4f44-90fd-c4175a764b23\"], \"Trust\": [\"844caaea-9a06-4bfa-9b96-3b2b9f49f0d5\"], \"Hagrid\": [\"ce7b5c57-f781-429d-9407-8a1748590baf\", \"97ea4cd8-fe36-40e8-85e4-ccdcae99765a\", \"844caaea-9a06-4bfa-9b96-3b2b9f49f0d5\"], \"Baby boy\": [\"844caaea-9a06-4bfa-9b96-3b2b9f49f0d5\"], \"Lily and james\": [\"ce7b5c57-f781-429d-9407-8a1748590baf\"], \"Motorcycle\": [\"5ac89898-c2b9-48eb-bb50-1634937b6442\", \"97ea4cd8-fe36-40e8-85e4-ccdcae99765a\"], \"Nose\": [\"57179285-4367-403b-bba8-0ca7ba42d3fd\"], \"Walked\": [\"57179285-4367-403b-bba8-0ca7ba42d3fd\"], \"Privet drive\": [\"6737c2d6-9801-4bb1-b3fb-f20d2a7fdac1\"], \"All\": [\"6737c2d6-9801-4bb1-b3fb-f20d2a7fdac1\"], \"Nephew\": [\"6737c2d6-9801-4bb1-b3fb-f20d2a7fdac1\"], \"Philz\": [\"834d6cdc-3e39-496f-8159-dd996961bacc\", \"635795f3-91d8-4700-9b99-8525e317f337\", \"ce87c184-7c4a-4906-a6d8-a5530611b0d9\", \"9c0d1038-cd2e-4a59-b35c-5bc64ea6c693\", \"3af30930-2141-400f-bb90-4b23434ee795\"], \"Berkeley\": [\"834d6cdc-3e39-496f-8159-dd996961bacc\", \"635795f3-91d8-4700-9b99-8525e317f337\", \"ce87c184-7c4a-4906-a6d8-a5530611b0d9\", \"9c0d1038-cd2e-4a59-b35c-5bc64ea6c693\", \"3af30930-2141-400f-bb90-4b23434ee795\"], \"Dream\": [\"9c0d1038-cd2e-4a59-b35c-5bc64ea6c693\"], \"Round glasses\": [\"9c0d1038-cd2e-4a59-b35c-5bc64ea6c693\"], \"Thin face\": [\"834d6cdc-3e39-496f-8159-dd996961bacc\"], \"Dudley\": [\"f8066950-076c-4840-b0cc-e8331acb4a00\", \"577c7ac6-849a-4ce5-a8f0-e21f84f38186\", \"cff90cbd-ecdf-450d-8d73-dfc8528da0a0\", \"37b3121c-3ce0-4f11-8eeb-f2291d49adf0\", \"0998c8da-dea8-4308-8a7e-f3afb293dddc\", \"3af30930-2141-400f-bb90-4b23434ee795\", \"8f80795d-de01-4f45-bf78-69d6bc9b5d45\"], \"Baby angel\": [\"3af30930-2141-400f-bb90-4b23434ee795\"], \"Wolfing down\": [\"8f80795d-de01-4f45-bf78-69d6bc9b5d45\"], \"Moment\": [\"8f80795d-de01-4f45-bf78-69d6bc9b5d45\"], \"Horror\": [\"0998c8da-dea8-4308-8a7e-f3afb293dddc\"], \"Aunt petunia\": [\"317cdfc1-fa74-4c77-9b8a-e78cba4a0e35\", \"37b3121c-3ce0-4f11-8eeb-f2291d49adf0\"], \"As though she'd just swallowed a lemon\": [\"37b3121c-3ce0-4f11-8eeb-f2291d49adf0\"], \"Loudly\": [\"37b3121c-3ce0-4f11-8eeb-f2291d49adf0\"], \"People's arms\": [\"cff90cbd-ecdf-450d-8d73-dfc8528da0a0\"], \"Them\": [\"cff90cbd-ecdf-450d-8d73-dfc8528da0a0\"], \"Uncle vernon\": [\"5ac89898-c2b9-48eb-bb50-1634937b6442\"], \"Motorcycles\": [\"5ac89898-c2b9-48eb-bb50-1634937b6442\"], \"Wind\": [\"f8066950-076c-4840-b0cc-e8331acb4a00\"], \"Bored\": [\"f8066950-076c-4840-b0cc-e8331acb4a00\"], \"Dudley and piers\": [\"635795f3-91d8-4700-9b99-8525e317f337\"], \"Huge poisonous cobras\": [\"577c7ac6-849a-4ce5-a8f0-e21f84f38186\"], \"Snake\": [\"577c7ac6-849a-4ce5-a8f0-e21f84f38186\"], \"Boa constrictor\": [\"317cdfc1-fa74-4c77-9b8a-e78cba4a0e35\"], \"Zoo\": [\"317cdfc1-fa74-4c77-9b8a-e78cba4a0e35\"], \"Zoo director\": [\"317cdfc1-fa74-4c77-9b8a-e78cba4a0e35\"], \"Throughout reptile house\": [\"d9ec5ecb-abf8-488f-81f3-d488cf88837b\"], \"Unknown relation\": [\"f0f3cf08-60c7-4aef-97be-c2db7d531503\"], \"Only family\": [\"f0f3cf08-60c7-4aef-97be-c2db7d531503\"]}, \"rel_map\": {}, \"embedding_dict\": {}}"}}} -------------------------------------------------------------------------------- /src/agent/base_agent.py: -------------------------------------------------------------------------------- 1 | import os 2 | from llama_index.core import ( 3 | KnowledgeGraphIndex, 4 | Settings, 5 | SimpleDirectoryReader, 6 | StorageContext, 7 | ) 8 | from llama_index.core.query_engine import RetrieverQueryEngine 9 | from llama_index.core.retrievers import KnowledgeGraphRAGRetriever 10 | from llama_index.graph_stores.neo4j import Neo4jGraphStore 11 | from llama_index.core.llms import ChatMessage 12 | from llama_index.llms.openai import OpenAI 13 | from llama_index.llms.perplexity import Perplexity 14 | 15 | from src.synonym_expand.synonym import custom_synonym_expand_fn 16 | from src.memory import MemoryStream, EntityKnowledgeStore 17 | from src.agent.data_types import Message 18 | from src.agent.llm_api.tools import openai_chat_completions_request 19 | 20 | MAX_ENTITIES_FROM_KG = 5 21 | ENTITY_EXCEPTIONS = ['Unknown relation'] 22 | 23 | 24 | class Agent(object): 25 | """Agent manages the RAG model, the knowledge graph, and the memory stream.""" 26 | 27 | def __init__(self, name, memory_stream_json, entity_knowledge_store_json, 28 | system_persona_txt, user_persona_txt, past_chat_json): 29 | self.name = name 30 | # OpenAI API key 31 | self.openai_api_key = os.environ["OPENAI_API_KEY"] 32 | self.model_endpoint = 'https://api.openai.com/v1' 33 | # Neo4j credentials 34 | self.neo4j_username = "neo4j" 35 | self.neo4j_password = os.getenv("NEO4J_PW") 36 | self.neo4j_url = os.getenv("NEO4J_URL") 37 | database = "neo4j" 38 | # Perplexity API key 39 | pplx_api_key = os.getenv('PERPLEXITY_API_KEY') 40 | 41 | self.llm = OpenAI(temperature=0, model="gpt-3.5-turbo") 42 | self.query_llm = Perplexity(api_key=pplx_api_key, 43 | model="sonar-small-online", 44 | temperature=0.5) 45 | Settings.llm = self.llm 46 | Settings.chunk_size = 512 47 | 48 | self.graph_store = Neo4jGraphStore( 49 | username=self.neo4j_username, 50 | password=self.neo4j_password, 51 | url=self.neo4j_url, 52 | database=database, 53 | ) 54 | 55 | self.storage_context = StorageContext.from_defaults( 56 | graph_store=self.graph_store) 57 | 58 | self.graph_rag_retriever = KnowledgeGraphRAGRetriever( 59 | storage_context=self.storage_context, 60 | verbose=True, 61 | llm=self.llm, 62 | retriever_mode="keyword", 63 | with_nl2graphquery=True, 64 | synonym_expand_fn=custom_synonym_expand_fn, 65 | ) 66 | self.query_engine = RetrieverQueryEngine.from_args( 67 | self.graph_rag_retriever, ) 68 | 69 | self.memory_stream = MemoryStream(memory_stream_json) 70 | self.entity_knowledge_store = EntityKnowledgeStore( 71 | entity_knowledge_store_json) 72 | 73 | self.message = Message(system_persona_txt, user_persona_txt, 74 | past_chat_json) 75 | 76 | def __str__(self): 77 | return f"Agent {self.name}" 78 | 79 | def check_KG(self, query: str) -> bool: 80 | """Check if the query is in the knowledge graph. 81 | 82 | Args: 83 | query (str): query to check in the knowledge graph 84 | 85 | Returns: 86 | bool: True if the query is in the knowledge graph, False otherwise 87 | """ 88 | response = self.query_engine.query(query) 89 | 90 | if response.metadata is None: 91 | return False 92 | return True 93 | 94 | def external_query(self, query: str): 95 | # 1) must query the web 96 | messages_dict = [ 97 | { 98 | "role": "system", 99 | "content": "Be precise and concise." 100 | }, 101 | { 102 | "role": "user", 103 | "content": query 104 | }, 105 | ] 106 | messages = [ChatMessage(**msg) for msg in messages_dict] 107 | 108 | external_response = self.query_llm.chat(messages) 109 | 110 | # 2) answer needs to be stored into txt file for loading into KG 111 | with open('data/external_response.txt', 'w') as f: 112 | print(external_response, file=f) 113 | return str(external_response) 114 | 115 | def load_KG(self): 116 | # TODO: index should be updated with the new data 117 | documents = SimpleDirectoryReader( 118 | input_files=['data/external_response.txt']).load_data() 119 | 120 | index = KnowledgeGraphIndex.from_documents( 121 | documents, 122 | storage_context=self.storage_context, 123 | max_triplets_per_chunk=5, 124 | ) 125 | 126 | def add_chapter(self, paths): 127 | # TODO: index should be updated with the new data 128 | documents = SimpleDirectoryReader( 129 | input_files=paths # paths is list of chapters 130 | ).load_data() 131 | 132 | index = KnowledgeGraphIndex.from_documents( 133 | documents, 134 | storage_context=self.storage_context, 135 | max_triplets_per_chunk=2, 136 | ) 137 | 138 | def _change_llm_message_chatgpt(self) -> dict: 139 | """Change the llm_message to chatgpt format. 140 | 141 | Returns: 142 | dict: llm_message in chatgpt format 143 | """ 144 | llm_message_chatgpt = self.message.llm_message 145 | llm_message_chatgpt['messages'] = [ 146 | context.to_dict() 147 | for context in self.message.llm_message['messages'] 148 | ] 149 | llm_message_chatgpt['messages'].append({ 150 | 'role': 151 | 'user', 152 | 'content': 153 | 'Memory Stream:' + str([ 154 | memory.to_dict() 155 | for memory in self.message.llm_message.pop('memory_stream') 156 | ]) 157 | }) 158 | llm_message_chatgpt['messages'].append({ 159 | 'role': 160 | 'user', 161 | 'content': 162 | 'Knowledge Entity Store:' + str([ 163 | entity.to_dict() for entity in self.message.llm_message.pop( 164 | 'knowledge_entity_store') 165 | ]) 166 | }) 167 | return llm_message_chatgpt 168 | 169 | def get_response(self) -> str: 170 | """Get response from the RAG model. 171 | 172 | Returns: 173 | str: response from the RAG model 174 | """ 175 | llm_message_chatgpt = self._change_llm_message_chatgpt() 176 | response = openai_chat_completions_request(self.model_endpoint, 177 | self.openai_api_key, 178 | llm_message_chatgpt) 179 | response = str(response['choices'][0]['message']['content']) 180 | return response 181 | 182 | def get_rag_response(self, query, return_entity=False): 183 | """Get response from the RAG model.""" 184 | response = self.query_engine.query(query) 185 | 186 | if return_entity: 187 | return response, self.get_entity(self.query_engine.retrieve(query)) 188 | return response 189 | 190 | def get_entity(self, retrieve) -> list[str]: 191 | """retrieve is a list of QueryBundle objects. 192 | A retrieved QueryBundle object has a "node" attribute, 193 | which has a "metadata" attribute. 194 | 195 | example for "kg_rel_map": 196 | kg_rel_map = { 197 | 'Harry': [['DREAMED_OF', 'Unknown relation'], ['FELL_HARD_ON', 'Concrete floor']], 198 | 'Potter': [['WORE', 'Round glasses'], ['HAD', 'Dream']] 199 | } 200 | 201 | Args: 202 | retrieve (list[NodeWithScore]): list of NodeWithScore objects 203 | return: 204 | list[str]: list of string entities 205 | """ 206 | 207 | entities = [] 208 | kg_rel_map = retrieve[0].node.metadata["kg_rel_map"] 209 | for key, items in kg_rel_map.items(): 210 | # key is the entity of question 211 | entities.append(key) 212 | # items is a list of [relationship, entity] 213 | entities.extend(item[1] for item in items) 214 | if len(entities) > MAX_ENTITIES_FROM_KG: 215 | break 216 | entities = list(set(entities)) 217 | for exceptions in ENTITY_EXCEPTIONS: 218 | if exceptions in entities: 219 | entities.remove(exceptions) 220 | return entities 221 | -------------------------------------------------------------------------------- /streamlit_app/entity_knowledge_store.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "entity": "Mrs. figg", 4 | "count": 1, 5 | "date": "2024-03-21T15:44:46" 6 | }, 7 | { 8 | "entity": "Tank", 9 | "count": 1, 10 | "date": "2024-03-21T15:44:46" 11 | }, 12 | { 13 | "entity": "Car", 14 | "count": 1, 15 | "date": "2024-03-21T15:44:46" 16 | }, 17 | { 18 | "entity": "Harry's whole arm", 19 | "count": 1, 20 | "date": "2024-03-21T15:44:46" 21 | }, 22 | { 23 | "entity": "Leaky cauldron", 24 | "count": 1, 25 | "date": "2024-03-21T15:44:46" 26 | }, 27 | { 28 | "entity": "Letter", 29 | "count": 3, 30 | "date": "2024-03-21T16:36:34" 31 | }, 32 | { 33 | "entity": "Aloud", 34 | "count": 1, 35 | "date": "2024-03-21T15:44:46" 36 | }, 37 | { 38 | "entity": "Put-outer", 39 | "count": 2, 40 | "date": "2024-03-21T16:36:34" 41 | }, 42 | { 43 | "entity": "Umbrella", 44 | "count": 1, 45 | "date": "2024-03-21T15:44:46" 46 | }, 47 | { 48 | "entity": "Misty eyes", 49 | "count": 1, 50 | "date": "2024-03-21T15:44:46" 51 | }, 52 | { 53 | "entity": "Birthday", 54 | "count": 1, 55 | "date": "2024-03-21T15:44:46" 56 | }, 57 | { 58 | "entity": "Mild climate", 59 | "count": 1, 60 | "date": "2024-03-21T15:44:20" 61 | }, 62 | { 63 | "entity": "Dursleys", 64 | "count": 3, 65 | "date": "2024-03-21T16:36:34" 66 | }, 67 | { 68 | "entity": "Concrete floor", 69 | "count": 1, 70 | "date": "2024-03-21T15:44:46" 71 | }, 72 | { 73 | "entity": "Longest-ever punishment", 74 | "count": 3, 75 | "date": "2024-03-21T16:36:34" 76 | }, 77 | { 78 | "entity": "Pockets", 79 | "count": 1, 80 | "date": "2024-03-21T15:44:46" 81 | }, 82 | { 83 | "entity": "Parents", 84 | "count": 1, 85 | "date": "2024-03-21T15:44:46" 86 | }, 87 | { 88 | "entity": "Flew", 89 | "count": 1, 90 | "date": "2024-03-21T15:44:46" 91 | }, 92 | { 93 | "entity": "Muggle", 94 | "count": 1, 95 | "date": "2024-03-21T15:44:46" 96 | }, 97 | { 98 | "entity": "Box", 99 | "count": 1, 100 | "date": "2024-03-21T15:44:46" 101 | }, 102 | { 103 | "entity": "Welcome", 104 | "count": 1, 105 | "date": "2024-03-21T15:44:46" 106 | }, 107 | { 108 | "entity": "Pieces", 109 | "count": 1, 110 | "date": "2024-03-21T15:44:46" 111 | }, 112 | { 113 | "entity": "Hair", 114 | "count": 1, 115 | "date": "2024-03-21T15:44:46" 116 | }, 117 | { 118 | "entity": "Control inputs", 119 | "count": 1, 120 | "date": "2024-03-21T16:20:21" 121 | }, 122 | { 123 | "entity": "Eggs", 124 | "count": 2, 125 | "date": "2024-03-21T16:36:34" 126 | }, 127 | { 128 | "entity": "Seriously", 129 | "count": 2, 130 | "date": "2024-03-21T16:36:34" 131 | }, 132 | { 133 | "entity": "White stone steps", 134 | "count": 1, 135 | "date": "2024-03-21T15:44:46" 136 | }, 137 | { 138 | "entity": "Seats", 139 | "count": 1, 140 | "date": "2024-03-21T15:44:46" 141 | }, 142 | { 143 | "entity": "Long robes", 144 | "count": 2, 145 | "date": "2024-03-21T16:36:34" 146 | }, 147 | { 148 | "entity": "Years an' years ago", 149 | "count": 1, 150 | "date": "2024-03-21T15:44:46" 151 | }, 152 | { 153 | "entity": "Hogwarts school of witchcraft and wizardry", 154 | "count": 1, 155 | "date": "2024-03-21T15:44:46" 156 | }, 157 | { 158 | "entity": "Key", 159 | "count": 1, 160 | "date": "2024-03-21T15:44:46" 161 | }, 162 | { 163 | "entity": "Bag full of money", 164 | "count": 1, 165 | "date": "2024-03-21T15:44:46" 166 | }, 167 | { 168 | "entity": "Sure", 169 | "count": 1, 170 | "date": "2024-03-21T15:44:46" 171 | }, 172 | { 173 | "entity": "Boat", 174 | "count": 1, 175 | "date": "2024-03-21T15:44:46" 176 | }, 177 | { 178 | "entity": "Lights", 179 | "count": 2, 180 | "date": "2024-03-21T16:36:34" 181 | }, 182 | { 183 | "entity": "Grubby little package", 184 | "count": 1, 185 | "date": "2024-03-21T15:44:46" 186 | }, 187 | { 188 | "entity": "Scar", 189 | "count": 1, 190 | "date": "2024-03-21T15:44:46" 191 | }, 192 | { 193 | "entity": "Front window", 194 | "count": 1, 195 | "date": "2024-03-21T15:44:46" 196 | }, 197 | { 198 | "entity": "Bacon", 199 | "count": 1, 200 | "date": "2024-03-21T15:44:46" 201 | }, 202 | { 203 | "entity": "Capital city of state", 204 | "count": 1, 205 | "date": "2024-03-21T15:44:20" 206 | }, 207 | { 208 | "entity": "Elephants", 209 | "count": 1, 210 | "date": "2024-03-21T16:23:13" 211 | }, 212 | { 213 | "entity": "Everyone", 214 | "count": 1, 215 | "date": "2024-03-21T15:44:46" 216 | }, 217 | { 218 | "entity": "Anglo-american settlers", 219 | "count": 1, 220 | "date": "2024-03-21T15:44:20" 221 | }, 222 | { 223 | "entity": "Three species", 224 | "count": 2, 225 | "date": "2024-03-21T16:40:32" 226 | }, 227 | { 228 | "entity": "Greatest sorcerer", 229 | "count": 1, 230 | "date": "2024-03-21T15:44:46" 231 | }, 232 | { 233 | "entity": "Wall", 234 | "count": 1, 235 | "date": "2024-03-21T15:44:46" 236 | }, 237 | { 238 | "entity": "Albus dumbledore", 239 | "count": 2, 240 | "date": "2024-03-21T16:36:34" 241 | }, 242 | { 243 | "entity": "Commands", 244 | "count": 1, 245 | "date": "2024-03-21T16:20:21" 246 | }, 247 | { 248 | "entity": "Bed", 249 | "count": 2, 250 | "date": "2024-03-21T16:36:34" 251 | }, 252 | { 253 | "entity": "Mouth", 254 | "count": 1, 255 | "date": "2024-03-21T15:44:46" 256 | }, 257 | { 258 | "entity": "Primary input device", 259 | "count": 1, 260 | "date": "2024-03-21T16:20:21" 261 | }, 262 | { 263 | "entity": "Insult", 264 | "count": 1, 265 | "date": "2024-03-21T15:44:46" 266 | }, 267 | { 268 | "entity": "Hand", 269 | "count": 1, 270 | "date": "2024-03-21T15:44:46" 271 | }, 272 | { 273 | "entity": "His birthday tick nearer", 274 | "count": 2, 275 | "date": "2024-03-21T16:36:34" 276 | }, 277 | { 278 | "entity": "My letter", 279 | "count": 2, 280 | "date": "2024-03-21T16:36:34" 281 | }, 282 | { 283 | "entity": "How he was going to look", 284 | "count": 2, 285 | "date": "2024-03-21T16:36:34" 286 | }, 287 | { 288 | "entity": "Sofa", 289 | "count": 1, 290 | "date": "2024-03-21T15:44:46" 291 | }, 292 | { 293 | "entity": "Hagrid", 294 | "count": 1, 295 | "date": "2024-03-21T15:44:46" 296 | }, 297 | { 298 | "entity": "Austin", 299 | "count": 1, 300 | "date": "2024-03-21T15:44:20" 301 | }, 302 | { 303 | "entity": "Kitchen", 304 | "count": 1, 305 | "date": "2024-03-21T15:44:46" 306 | }, 307 | { 308 | "entity": "...", 309 | "count": 2, 310 | "date": "2024-03-21T16:36:34" 311 | }, 312 | { 313 | "entity": "Distinctive black-and-white striped coats", 314 | "count": 2, 315 | "date": "2024-03-21T16:40:32" 316 | }, 317 | { 318 | "entity": "Turning the page", 319 | "count": 1, 320 | "date": "2024-03-21T15:44:46" 321 | }, 322 | { 323 | "entity": "Green", 324 | "count": 1, 325 | "date": "2024-03-21T15:44:46" 326 | }, 327 | { 328 | "entity": "Temper", 329 | "count": 1, 330 | "date": "2024-03-21T15:44:46" 331 | }, 332 | { 333 | "entity": "Keeper of keys", 334 | "count": 1, 335 | "date": "2024-03-21T15:44:46" 336 | }, 337 | { 338 | "entity": "Slytherin", 339 | "count": 1, 340 | "date": "2024-03-21T15:44:46" 341 | }, 342 | { 343 | "entity": "Dream", 344 | "count": 1, 345 | "date": "2024-03-21T15:44:46" 346 | }, 347 | { 348 | "entity": "There are dragons at gringotts", 349 | "count": 1, 350 | "date": "2024-03-21T15:44:46" 351 | }, 352 | { 353 | "entity": "Explode", 354 | "count": 1, 355 | "date": "2024-03-21T15:44:46" 356 | }, 357 | { 358 | "entity": "Large cage", 359 | "count": 1, 360 | "date": "2024-03-21T15:44:46" 361 | }, 362 | { 363 | "entity": "Math", 364 | "count": 1, 365 | "date": "2024-03-21T15:44:46" 366 | }, 367 | { 368 | "entity": "Keyboard", 369 | "count": 1, 370 | "date": "2024-03-21T16:20:21" 371 | }, 372 | { 373 | "entity": "Equids", 374 | "count": 2, 375 | "date": "2024-03-21T16:40:32" 376 | }, 377 | { 378 | "entity": "Questions", 379 | "count": 1, 380 | "date": "2024-03-21T15:44:46" 381 | }, 382 | { 383 | "entity": "Nothing", 384 | "count": 1, 385 | "date": "2024-03-21T15:44:46" 386 | }, 387 | { 388 | "entity": "Central texas", 389 | "count": 1, 390 | "date": "2024-03-21T15:44:20" 391 | }, 392 | { 393 | "entity": "Uncle vernon", 394 | "count": 1, 395 | "date": "2024-03-21T15:44:46" 396 | }, 397 | { 398 | "entity": "Mounds of gold coins", 399 | "count": 1, 400 | "date": "2024-03-21T15:44:46" 401 | }, 402 | { 403 | "entity": "104 keys", 404 | "count": 1, 405 | "date": "2024-03-21T16:20:21" 406 | }, 407 | { 408 | "entity": "Zebras", 409 | "count": 2, 410 | "date": "2024-03-21T16:40:32" 411 | }, 412 | { 413 | "entity": "Voldemort", 414 | "count": 1, 415 | "date": "2024-03-21T15:44:46" 416 | }, 417 | { 418 | "entity": "Long hair", 419 | "count": 2, 420 | "date": "2024-03-21T16:36:34" 421 | }, 422 | { 423 | "entity": "Parchment envelope", 424 | "count": 1, 425 | "date": "2024-03-21T15:44:46" 426 | }, 427 | { 428 | "entity": "Text", 429 | "count": 1, 430 | "date": "2024-03-21T16:20:21" 431 | }, 432 | { 433 | "entity": "Dursley", 434 | "count": 1, 435 | "date": "2024-03-21T15:44:46" 436 | }, 437 | { 438 | "entity": "Morning", 439 | "count": 1, 440 | "date": "2024-03-21T15:44:46" 441 | }, 442 | { 443 | "entity": "Bag of chips", 444 | "count": 1, 445 | "date": "2024-03-21T15:44:46" 446 | }, 447 | { 448 | "entity": "Next morning", 449 | "count": 1, 450 | "date": "2024-03-21T15:44:46" 451 | }, 452 | { 453 | "entity": "At the table", 454 | "count": 2, 455 | "date": "2024-03-21T16:36:34" 456 | }, 457 | { 458 | "entity": "Street", 459 | "count": 2, 460 | "date": "2024-03-21T16:36:34" 461 | }, 462 | { 463 | "entity": "Sitting", 464 | "count": 2, 465 | "date": "2024-03-21T16:36:34" 466 | }, 467 | { 468 | "entity": "Moment", 469 | "count": 2, 470 | "date": "2024-03-21T16:36:34" 471 | }, 472 | { 473 | "entity": "70 years in the wild", 474 | "count": 1, 475 | "date": "2024-03-21T16:23:13" 476 | }, 477 | { 478 | "entity": "Wand", 479 | "count": 1, 480 | "date": "2024-03-21T15:44:46" 481 | }, 482 | { 483 | "entity": "Pink umbrella", 484 | "count": 1, 485 | "date": "2024-03-21T15:44:46" 486 | }, 487 | { 488 | "entity": "Something creak", 489 | "count": 1, 490 | "date": "2024-03-21T15:44:46" 491 | }, 492 | { 493 | "entity": "Where was i", 494 | "count": 1, 495 | "date": "2024-03-21T15:44:46" 496 | }, 497 | { 498 | "entity": "Mother's eyes", 499 | "count": 1, 500 | "date": "2024-03-21T15:44:46" 501 | }, 502 | { 503 | "entity": "Blue eyes", 504 | "count": 2, 505 | "date": "2024-03-21T16:36:34" 506 | }, 507 | { 508 | "entity": "Golden watch", 509 | "count": 2, 510 | "date": "2024-03-21T16:36:34" 511 | }, 512 | { 513 | "entity": "Live music capital of the world", 514 | "count": 1, 515 | "date": "2024-03-21T15:44:20" 516 | }, 517 | { 518 | "entity": "Round glasses", 519 | "count": 1, 520 | "date": "2024-03-21T15:44:46" 521 | }, 522 | { 523 | "entity": "Bundle", 524 | "count": 2, 525 | "date": "2024-03-21T16:36:34" 526 | }, 527 | { 528 | "entity": "Keeper of keys and grounds at hogwarts", 529 | "count": 1, 530 | "date": "2024-03-21T15:44:46" 531 | }, 532 | { 533 | "entity": "The mail", 534 | "count": 2, 535 | "date": "2024-03-21T16:36:34" 536 | }, 537 | { 538 | "entity": "Her on the shoulder", 539 | "count": 2, 540 | "date": "2024-03-21T16:36:34" 541 | }, 542 | { 543 | "entity": "Largest living land animals", 544 | "count": 1, 545 | "date": "2024-03-21T16:23:13" 546 | }, 547 | { 548 | "entity": "Third year", 549 | "count": 1, 550 | "date": "2024-03-21T15:44:46" 551 | }, 552 | { 553 | "entity": "Aunt petunia", 554 | "count": 1, 555 | "date": "2024-03-21T15:44:46" 556 | }, 557 | { 558 | "entity": "Muggle money", 559 | "count": 1, 560 | "date": "2024-03-21T15:44:46" 561 | }, 562 | { 563 | "entity": "Cupboard", 564 | "count": 1, 565 | "date": "2024-03-21T15:44:46" 566 | }, 567 | { 568 | "entity": "Harry", 569 | "count": 4, 570 | "date": "2024-03-21T16:36:34" 571 | }, 572 | { 573 | "entity": "Speeds of up to 35 miles per hour", 574 | "count": 2, 575 | "date": "2024-03-21T16:40:32" 576 | }, 577 | { 578 | "entity": "Harry about quidditch", 579 | "count": 1, 580 | "date": "2024-03-21T15:44:46" 581 | }, 582 | { 583 | "entity": "Owl", 584 | "count": 1, 585 | "date": "2024-03-21T15:44:46" 586 | }, 587 | { 588 | "entity": "Best birthday", 589 | "count": 1, 590 | "date": "2024-03-21T15:44:46" 591 | }, 592 | { 593 | "entity": "Dreamt", 594 | "count": 2, 595 | "date": "2024-03-21T16:36:34" 596 | }, 597 | { 598 | "entity": "It's the best place for him", 599 | "count": 2, 600 | "date": "2024-03-21T16:36:34" 601 | }, 602 | { 603 | "entity": "Cryptocurrencies", 604 | "count": 1, 605 | "date": "2024-03-21T16:37:45" 606 | }, 607 | { 608 | "entity": "Dudley's second bedroom", 609 | "count": 1, 610 | "date": "2024-03-21T15:44:46" 611 | }, 612 | { 613 | "entity": "Not", 614 | "count": 2, 615 | "date": "2024-03-21T16:36:34" 616 | }, 617 | { 618 | "entity": "Man", 619 | "count": 1, 620 | "date": "2024-03-21T15:44:46" 621 | }, 622 | { 623 | "entity": "Don' mention it", 624 | "count": 1, 625 | "date": "2024-03-21T15:44:46" 626 | }, 627 | { 628 | "entity": "Grasses and sedges", 629 | "count": 2, 630 | "date": "2024-03-21T16:40:32" 631 | }, 632 | { 633 | "entity": "Hall", 634 | "count": 1, 635 | "date": "2024-03-21T15:44:46" 636 | }, 637 | { 638 | "entity": "Wizard coins", 639 | "count": 1, 640 | "date": "2024-03-21T15:44:46" 641 | }, 642 | { 643 | "entity": "Letters", 644 | "count": 2, 645 | "date": "2024-03-21T16:36:34" 646 | }, 647 | { 648 | "entity": "Crooked nose", 649 | "count": 2, 650 | "date": "2024-03-21T16:36:34" 651 | }, 652 | { 653 | "entity": "Say something", 654 | "count": 1, 655 | "date": "2024-03-21T15:44:46" 656 | }, 657 | { 658 | "entity": "Professor mcgonagall", 659 | "count": 2, 660 | "date": "2024-03-21T16:36:34" 661 | }, 662 | { 663 | "entity": "Volatile", 664 | "count": 1, 665 | "date": "2024-03-21T16:37:45" 666 | }, 667 | { 668 | "entity": "Dumbledore", 669 | "count": 2, 670 | "date": "2024-03-21T16:36:34" 671 | }, 672 | { 673 | "entity": "Hogwarts", 674 | "count": 1, 675 | "date": "2024-03-21T15:44:46" 676 | } 677 | ] -------------------------------------------------------------------------------- /streamlit_app/memory_stream.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "entity": "Anglo-american settlers", 4 | "date": "2024-03-21T15:44:20" 5 | }, 6 | { 7 | "entity": "Central texas", 8 | "date": "2024-03-21T15:44:20" 9 | }, 10 | { 11 | "entity": "Live music capital of the world", 12 | "date": "2024-03-21T15:44:20" 13 | }, 14 | { 15 | "entity": "Mild climate", 16 | "date": "2024-03-21T15:44:20" 17 | }, 18 | { 19 | "entity": "Capital city of state", 20 | "date": "2024-03-21T15:44:20" 21 | }, 22 | { 23 | "entity": "Austin", 24 | "date": "2024-03-21T15:44:20" 25 | }, 26 | { 27 | "entity": "Aunt petunia", 28 | "date": "2024-03-21T15:44:46" 29 | }, 30 | { 31 | "entity": "Hall", 32 | "date": "2024-03-21T15:44:46" 33 | }, 34 | { 35 | "entity": "Mounds of gold coins", 36 | "date": "2024-03-21T15:44:46" 37 | }, 38 | { 39 | "entity": "Parchment envelope", 40 | "date": "2024-03-21T15:44:46" 41 | }, 42 | { 43 | "entity": "Next morning", 44 | "date": "2024-03-21T15:44:46" 45 | }, 46 | { 47 | "entity": "Nothing", 48 | "date": "2024-03-21T15:44:46" 49 | }, 50 | { 51 | "entity": "Dursleys", 52 | "date": "2024-03-21T15:44:46" 53 | }, 54 | { 55 | "entity": "Third year", 56 | "date": "2024-03-21T15:44:46" 57 | }, 58 | { 59 | "entity": "Explode", 60 | "date": "2024-03-21T15:44:46" 61 | }, 62 | { 63 | "entity": "Harry", 64 | "date": "2024-03-21T15:44:46" 65 | }, 66 | { 67 | "entity": "Tank", 68 | "date": "2024-03-21T15:44:46" 69 | }, 70 | { 71 | "entity": "Dream", 72 | "date": "2024-03-21T15:44:46" 73 | }, 74 | { 75 | "entity": "Welcome", 76 | "date": "2024-03-21T15:44:46" 77 | }, 78 | { 79 | "entity": "Bacon", 80 | "date": "2024-03-21T15:44:46" 81 | }, 82 | { 83 | "entity": "Questions", 84 | "date": "2024-03-21T15:44:46" 85 | }, 86 | { 87 | "entity": "Sure", 88 | "date": "2024-03-21T15:44:46" 89 | }, 90 | { 91 | "entity": "Muggle money", 92 | "date": "2024-03-21T15:44:46" 93 | }, 94 | { 95 | "entity": "Something creak", 96 | "date": "2024-03-21T15:44:46" 97 | }, 98 | { 99 | "entity": "Wall", 100 | "date": "2024-03-21T15:44:46" 101 | }, 102 | { 103 | "entity": "Muggle", 104 | "date": "2024-03-21T15:44:46" 105 | }, 106 | { 107 | "entity": "Where was i", 108 | "date": "2024-03-21T15:44:46" 109 | }, 110 | { 111 | "entity": "Pieces", 112 | "date": "2024-03-21T15:44:46" 113 | }, 114 | { 115 | "entity": "Mouth", 116 | "date": "2024-03-21T15:44:46" 117 | }, 118 | { 119 | "entity": "Key", 120 | "date": "2024-03-21T15:44:46" 121 | }, 122 | { 123 | "entity": "Kitchen", 124 | "date": "2024-03-21T15:44:46" 125 | }, 126 | { 127 | "entity": "Box", 128 | "date": "2024-03-21T15:44:46" 129 | }, 130 | { 131 | "entity": "Scar", 132 | "date": "2024-03-21T15:44:46" 133 | }, 134 | { 135 | "entity": "Dursley", 136 | "date": "2024-03-21T15:44:46" 137 | }, 138 | { 139 | "entity": "Math", 140 | "date": "2024-03-21T15:44:46" 141 | }, 142 | { 143 | "entity": "Hogwarts", 144 | "date": "2024-03-21T15:44:46" 145 | }, 146 | { 147 | "entity": "Flew", 148 | "date": "2024-03-21T15:44:46" 149 | }, 150 | { 151 | "entity": "Don' mention it", 152 | "date": "2024-03-21T15:44:46" 153 | }, 154 | { 155 | "entity": "Parents", 156 | "date": "2024-03-21T15:44:46" 157 | }, 158 | { 159 | "entity": "There are dragons at gringotts", 160 | "date": "2024-03-21T15:44:46" 161 | }, 162 | { 163 | "entity": "Cupboard", 164 | "date": "2024-03-21T15:44:46" 165 | }, 166 | { 167 | "entity": "Temper", 168 | "date": "2024-03-21T15:44:46" 169 | }, 170 | { 171 | "entity": "Leaky cauldron", 172 | "date": "2024-03-21T15:44:46" 173 | }, 174 | { 175 | "entity": "Aloud", 176 | "date": "2024-03-21T15:44:46" 177 | }, 178 | { 179 | "entity": "Car", 180 | "date": "2024-03-21T15:44:46" 181 | }, 182 | { 183 | "entity": "Pockets", 184 | "date": "2024-03-21T15:44:46" 185 | }, 186 | { 187 | "entity": "Sofa", 188 | "date": "2024-03-21T15:44:46" 189 | }, 190 | { 191 | "entity": "Mrs. figg", 192 | "date": "2024-03-21T15:44:46" 193 | }, 194 | { 195 | "entity": "Man", 196 | "date": "2024-03-21T15:44:46" 197 | }, 198 | { 199 | "entity": "Morning", 200 | "date": "2024-03-21T15:44:46" 201 | }, 202 | { 203 | "entity": "White stone steps", 204 | "date": "2024-03-21T15:44:46" 205 | }, 206 | { 207 | "entity": "Wand", 208 | "date": "2024-03-21T15:44:46" 209 | }, 210 | { 211 | "entity": "Harry's whole arm", 212 | "date": "2024-03-21T15:44:46" 213 | }, 214 | { 215 | "entity": "Say something", 216 | "date": "2024-03-21T15:44:46" 217 | }, 218 | { 219 | "entity": "Hand", 220 | "date": "2024-03-21T15:44:46" 221 | }, 222 | { 223 | "entity": "Best birthday", 224 | "date": "2024-03-21T15:44:46" 225 | }, 226 | { 227 | "entity": "Hagrid", 228 | "date": "2024-03-21T15:44:46" 229 | }, 230 | { 231 | "entity": "Concrete floor", 232 | "date": "2024-03-21T15:44:46" 233 | }, 234 | { 235 | "entity": "Insult", 236 | "date": "2024-03-21T15:44:46" 237 | }, 238 | { 239 | "entity": "Harry about quidditch", 240 | "date": "2024-03-21T15:44:46" 241 | }, 242 | { 243 | "entity": "Mother's eyes", 244 | "date": "2024-03-21T15:44:46" 245 | }, 246 | { 247 | "entity": "Greatest sorcerer", 248 | "date": "2024-03-21T15:44:46" 249 | }, 250 | { 251 | "entity": "Bag of chips", 252 | "date": "2024-03-21T15:44:46" 253 | }, 254 | { 255 | "entity": "Round glasses", 256 | "date": "2024-03-21T15:44:46" 257 | }, 258 | { 259 | "entity": "Slytherin", 260 | "date": "2024-03-21T15:44:46" 261 | }, 262 | { 263 | "entity": "Green", 264 | "date": "2024-03-21T15:44:46" 265 | }, 266 | { 267 | "entity": "Voldemort", 268 | "date": "2024-03-21T15:44:46" 269 | }, 270 | { 271 | "entity": "Dudley's second bedroom", 272 | "date": "2024-03-21T15:44:46" 273 | }, 274 | { 275 | "entity": "Years an' years ago", 276 | "date": "2024-03-21T15:44:46" 277 | }, 278 | { 279 | "entity": "Umbrella", 280 | "date": "2024-03-21T15:44:46" 281 | }, 282 | { 283 | "entity": "Keeper of keys", 284 | "date": "2024-03-21T15:44:46" 285 | }, 286 | { 287 | "entity": "Boat", 288 | "date": "2024-03-21T15:44:46" 289 | }, 290 | { 291 | "entity": "Uncle vernon", 292 | "date": "2024-03-21T15:44:46" 293 | }, 294 | { 295 | "entity": "Hogwarts school of witchcraft and wizardry", 296 | "date": "2024-03-21T15:44:46" 297 | }, 298 | { 299 | "entity": "Longest-ever punishment", 300 | "date": "2024-03-21T15:44:46" 301 | }, 302 | { 303 | "entity": "Misty eyes", 304 | "date": "2024-03-21T15:44:46" 305 | }, 306 | { 307 | "entity": "Seats", 308 | "date": "2024-03-21T15:44:46" 309 | }, 310 | { 311 | "entity": "Wizard coins", 312 | "date": "2024-03-21T15:44:46" 313 | }, 314 | { 315 | "entity": "Pink umbrella", 316 | "date": "2024-03-21T15:44:46" 317 | }, 318 | { 319 | "entity": "Keeper of keys and grounds at hogwarts", 320 | "date": "2024-03-21T15:44:46" 321 | }, 322 | { 323 | "entity": "Turning the page", 324 | "date": "2024-03-21T15:44:46" 325 | }, 326 | { 327 | "entity": "Hair", 328 | "date": "2024-03-21T15:44:46" 329 | }, 330 | { 331 | "entity": "Grubby little package", 332 | "date": "2024-03-21T15:44:46" 333 | }, 334 | { 335 | "entity": "Large cage", 336 | "date": "2024-03-21T15:44:46" 337 | }, 338 | { 339 | "entity": "Letter", 340 | "date": "2024-03-21T15:44:46" 341 | }, 342 | { 343 | "entity": "Everyone", 344 | "date": "2024-03-21T15:44:46" 345 | }, 346 | { 347 | "entity": "Birthday", 348 | "date": "2024-03-21T15:44:46" 349 | }, 350 | { 351 | "entity": "Bag full of money", 352 | "date": "2024-03-21T15:44:46" 353 | }, 354 | { 355 | "entity": "Owl", 356 | "date": "2024-03-21T15:44:46" 357 | }, 358 | { 359 | "entity": "Front window", 360 | "date": "2024-03-21T15:44:46" 361 | }, 362 | { 363 | "entity": "Text", 364 | "date": "2024-03-21T16:20:21" 365 | }, 366 | { 367 | "entity": "Control inputs", 368 | "date": "2024-03-21T16:20:21" 369 | }, 370 | { 371 | "entity": "104 keys", 372 | "date": "2024-03-21T16:20:21" 373 | }, 374 | { 375 | "entity": "Commands", 376 | "date": "2024-03-21T16:20:21" 377 | }, 378 | { 379 | "entity": "Primary input device", 380 | "date": "2024-03-21T16:20:21" 381 | }, 382 | { 383 | "entity": "Keyboard", 384 | "date": "2024-03-21T16:20:21" 385 | }, 386 | { 387 | "entity": "Elephants", 388 | "date": "2024-03-21T16:23:13" 389 | }, 390 | { 391 | "entity": "70 years in the wild", 392 | "date": "2024-03-21T16:23:13" 393 | }, 394 | { 395 | "entity": "Largest living land animals", 396 | "date": "2024-03-21T16:23:13" 397 | }, 398 | { 399 | "entity": "The mail", 400 | "date": "2024-03-21T16:35:34" 401 | }, 402 | { 403 | "entity": "Harry", 404 | "date": "2024-03-21T16:35:34" 405 | }, 406 | { 407 | "entity": "Dursleys", 408 | "date": "2024-03-21T16:35:34" 409 | }, 410 | { 411 | "entity": "Longest-ever punishment", 412 | "date": "2024-03-21T16:35:34" 413 | }, 414 | { 415 | "entity": "Dreamt", 416 | "date": "2024-03-21T16:35:34" 417 | }, 418 | { 419 | "entity": "Not", 420 | "date": "2024-03-21T16:35:34" 421 | }, 422 | { 423 | "entity": "His birthday tick nearer", 424 | "date": "2024-03-21T16:35:34" 425 | }, 426 | { 427 | "entity": "My letter", 428 | "date": "2024-03-21T16:35:34" 429 | }, 430 | { 431 | "entity": "How he was going to look", 432 | "date": "2024-03-21T16:35:34" 433 | }, 434 | { 435 | "entity": "Letters", 436 | "date": "2024-03-21T16:35:34" 437 | }, 438 | { 439 | "entity": "At the table", 440 | "date": "2024-03-21T16:35:34" 441 | }, 442 | { 443 | "entity": "Sitting", 444 | "date": "2024-03-21T16:35:34" 445 | }, 446 | { 447 | "entity": "Moment", 448 | "date": "2024-03-21T16:35:34" 449 | }, 450 | { 451 | "entity": "Eggs", 452 | "date": "2024-03-21T16:35:34" 453 | }, 454 | { 455 | "entity": "Seriously", 456 | "date": "2024-03-21T16:35:34" 457 | }, 458 | { 459 | "entity": "...", 460 | "date": "2024-03-21T16:35:34" 461 | }, 462 | { 463 | "entity": "Lights", 464 | "date": "2024-03-21T16:35:34" 465 | }, 466 | { 467 | "entity": "Bed", 468 | "date": "2024-03-21T16:35:34" 469 | }, 470 | { 471 | "entity": "Long robes", 472 | "date": "2024-03-21T16:35:58" 473 | }, 474 | { 475 | "entity": "Golden watch", 476 | "date": "2024-03-21T16:35:58" 477 | }, 478 | { 479 | "entity": "Harry", 480 | "date": "2024-03-21T16:35:58" 481 | }, 482 | { 483 | "entity": "Professor mcgonagall", 484 | "date": "2024-03-21T16:35:58" 485 | }, 486 | { 487 | "entity": "Bundle", 488 | "date": "2024-03-21T16:35:58" 489 | }, 490 | { 491 | "entity": "Her on the shoulder", 492 | "date": "2024-03-21T16:35:58" 493 | }, 494 | { 495 | "entity": "Street", 496 | "date": "2024-03-21T16:35:58" 497 | }, 498 | { 499 | "entity": "Crooked nose", 500 | "date": "2024-03-21T16:35:58" 501 | }, 502 | { 503 | "entity": "Albus dumbledore", 504 | "date": "2024-03-21T16:35:58" 505 | }, 506 | { 507 | "entity": "Dumbledore", 508 | "date": "2024-03-21T16:35:58" 509 | }, 510 | { 511 | "entity": "Blue eyes", 512 | "date": "2024-03-21T16:35:58" 513 | }, 514 | { 515 | "entity": "Letter", 516 | "date": "2024-03-21T16:35:58" 517 | }, 518 | { 519 | "entity": "It's the best place for him", 520 | "date": "2024-03-21T16:35:58" 521 | }, 522 | { 523 | "entity": "Put-outer", 524 | "date": "2024-03-21T16:35:58" 525 | }, 526 | { 527 | "entity": "Long hair", 528 | "date": "2024-03-21T16:35:58" 529 | }, 530 | { 531 | "entity": "Bundle", 532 | "date": "2024-03-21T16:36:34" 533 | }, 534 | { 535 | "entity": "The mail", 536 | "date": "2024-03-21T16:36:34" 537 | }, 538 | { 539 | "entity": "Her on the shoulder", 540 | "date": "2024-03-21T16:36:34" 541 | }, 542 | { 543 | "entity": "Letter", 544 | "date": "2024-03-21T16:36:34" 545 | }, 546 | { 547 | "entity": "Put-outer", 548 | "date": "2024-03-21T16:36:34" 549 | }, 550 | { 551 | "entity": "Harry", 552 | "date": "2024-03-21T16:36:34" 553 | }, 554 | { 555 | "entity": "Dursleys", 556 | "date": "2024-03-21T16:36:34" 557 | }, 558 | { 559 | "entity": "Albus dumbledore", 560 | "date": "2024-03-21T16:36:34" 561 | }, 562 | { 563 | "entity": "It's the best place for him", 564 | "date": "2024-03-21T16:36:34" 565 | }, 566 | { 567 | "entity": "Longest-ever punishment", 568 | "date": "2024-03-21T16:36:34" 569 | }, 570 | { 571 | "entity": "Dreamt", 572 | "date": "2024-03-21T16:36:34" 573 | }, 574 | { 575 | "entity": "Not", 576 | "date": "2024-03-21T16:36:34" 577 | }, 578 | { 579 | "entity": "Long hair", 580 | "date": "2024-03-21T16:36:34" 581 | }, 582 | { 583 | "entity": "His birthday tick nearer", 584 | "date": "2024-03-21T16:36:34" 585 | }, 586 | { 587 | "entity": "My letter", 588 | "date": "2024-03-21T16:36:34" 589 | }, 590 | { 591 | "entity": "How he was going to look", 592 | "date": "2024-03-21T16:36:34" 593 | }, 594 | { 595 | "entity": "Letters", 596 | "date": "2024-03-21T16:36:34" 597 | }, 598 | { 599 | "entity": "Street", 600 | "date": "2024-03-21T16:36:34" 601 | }, 602 | { 603 | "entity": "At the table", 604 | "date": "2024-03-21T16:36:34" 605 | }, 606 | { 607 | "entity": "Sitting", 608 | "date": "2024-03-21T16:36:34" 609 | }, 610 | { 611 | "entity": "Crooked nose", 612 | "date": "2024-03-21T16:36:34" 613 | }, 614 | { 615 | "entity": "Moment", 616 | "date": "2024-03-21T16:36:34" 617 | }, 618 | { 619 | "entity": "Eggs", 620 | "date": "2024-03-21T16:36:34" 621 | }, 622 | { 623 | "entity": "Seriously", 624 | "date": "2024-03-21T16:36:34" 625 | }, 626 | { 627 | "entity": "...", 628 | "date": "2024-03-21T16:36:34" 629 | }, 630 | { 631 | "entity": "Long robes", 632 | "date": "2024-03-21T16:36:34" 633 | }, 634 | { 635 | "entity": "Professor mcgonagall", 636 | "date": "2024-03-21T16:36:34" 637 | }, 638 | { 639 | "entity": "Dumbledore", 640 | "date": "2024-03-21T16:36:34" 641 | }, 642 | { 643 | "entity": "Lights", 644 | "date": "2024-03-21T16:36:34" 645 | }, 646 | { 647 | "entity": "Blue eyes", 648 | "date": "2024-03-21T16:36:34" 649 | }, 650 | { 651 | "entity": "Golden watch", 652 | "date": "2024-03-21T16:36:34" 653 | }, 654 | { 655 | "entity": "Bed", 656 | "date": "2024-03-21T16:36:34" 657 | }, 658 | { 659 | "entity": "Cryptocurrencies", 660 | "date": "2024-03-21T16:37:45" 661 | }, 662 | { 663 | "entity": "Volatile", 664 | "date": "2024-03-21T16:37:45" 665 | }, 666 | { 667 | "entity": "Distinctive black-and-white striped coats", 668 | "date": "2024-03-21T16:38:35" 669 | }, 670 | { 671 | "entity": "Grasses and sedges", 672 | "date": "2024-03-21T16:38:35" 673 | }, 674 | { 675 | "entity": "Speeds of up to 35 miles per hour", 676 | "date": "2024-03-21T16:38:35" 677 | }, 678 | { 679 | "entity": "Equids", 680 | "date": "2024-03-21T16:38:35" 681 | }, 682 | { 683 | "entity": "Zebras", 684 | "date": "2024-03-21T16:38:35" 685 | }, 686 | { 687 | "entity": "Three species", 688 | "date": "2024-03-21T16:38:35" 689 | }, 690 | { 691 | "entity": "Distinctive black-and-white striped coats", 692 | "date": "2024-03-21T16:40:32" 693 | }, 694 | { 695 | "entity": "Grasses and sedges", 696 | "date": "2024-03-21T16:40:32" 697 | }, 698 | { 699 | "entity": "Speeds of up to 35 miles per hour", 700 | "date": "2024-03-21T16:40:32" 701 | }, 702 | { 703 | "entity": "Equids", 704 | "date": "2024-03-21T16:40:32" 705 | }, 706 | { 707 | "entity": "Zebras", 708 | "date": "2024-03-21T16:40:32" 709 | }, 710 | { 711 | "entity": "Three species", 712 | "date": "2024-03-21T16:40:32" 713 | } 714 | ] -------------------------------------------------------------------------------- /recursive_retrieval/recurse.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import os\n", 10 | "from dotenv import load_dotenv\n", 11 | "\n", 12 | "load_dotenv()\n", 13 | "\n", 14 | "os.environ[\"OPENAI_API_KEY\"] = os.getenv('OPENAI_KEY')" 15 | ] 16 | }, 17 | { 18 | "cell_type": "code", 19 | "execution_count": 2, 20 | "metadata": {}, 21 | "outputs": [], 22 | "source": [ 23 | "from llama_index.graph_stores.neo4j import Neo4jGraphStore\n", 24 | "from llama_index.core import StorageContext\n", 25 | "\n", 26 | "username = \"neo4j\"\n", 27 | "password = os.getenv('NEO4J_PW')\n", 28 | "url = os.getenv('NEO4J_URL')\n", 29 | "database = \"neo4j\"\n", 30 | "\n", 31 | "graph_store = Neo4jGraphStore(\n", 32 | " username=username,\n", 33 | " password=password,\n", 34 | " url=url,\n", 35 | " database=database,\n", 36 | ")\n", 37 | "\n", 38 | "storage_context = StorageContext.from_defaults(graph_store=graph_store)" 39 | ] 40 | }, 41 | { 42 | "cell_type": "code", 43 | "execution_count": 18, 44 | "metadata": {}, 45 | "outputs": [], 46 | "source": [ 47 | "from llama_index.core.indices.knowledge_graph.retrievers import KnowledgeGraphRAGRetriever\n", 48 | "from llama_index.core.query_engine import RetrieverQueryEngine\n", 49 | "\n", 50 | "graph_rag_retriever_with_nl2graphquery = KnowledgeGraphRAGRetriever(\n", 51 | " storage_context=storage_context,\n", 52 | " verbose=True,\n", 53 | " retriever_mode=\"keyword\",\n", 54 | " with_nl2graphquery=True,\n", 55 | ")\n", 56 | "\n", 57 | "query_engine_with_nl2graphquery = RetrieverQueryEngine.from_args(\n", 58 | " graph_rag_retriever_with_nl2graphquery,\n", 59 | ")\n" 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": 19, 65 | "metadata": {}, 66 | "outputs": [ 67 | { 68 | "data": { 69 | "text/markdown": [ 70 | "Harry Potter is a character in a popular series of fantasy novels written by J.K. Rowling." 71 | ], 72 | "text/plain": [ 73 | "" 74 | ] 75 | }, 76 | "metadata": {}, 77 | "output_type": "display_data" 78 | } 79 | ], 80 | "source": [ 81 | "from IPython.display import display, Markdown\n", 82 | "\n", 83 | "response = query_engine_with_nl2graphquery.query(\n", 84 | " \"who is harry potter\",\n", 85 | ")\n", 86 | "\n", 87 | "display(Markdown(f\"{response}\"))" 88 | ] 89 | }, 90 | { 91 | "cell_type": "code", 92 | "execution_count": 14, 93 | "metadata": {}, 94 | "outputs": [ 95 | { 96 | "name": "stdout", 97 | "output_type": "stream", 98 | "text": [ 99 | "{'af4510aa-44a4-4720-aa83-838ef29eb60b': {'kg_rel_map': {'Harry': [['TURNED_OUT_TO_BE',\n", 100 | " 'Bag of '\n", 101 | " 'chips'],\n", 102 | " ['ANGRILY',\n", 103 | " 'Said'],\n", 104 | " ['HEART_TWANGING_LIKE',\n", 105 | " 'Giant '\n", 106 | " 'elastic '\n", 107 | " 'band'],\n", 108 | " ['RAN',\n", 109 | " 'Before'],\n", 110 | " ['LAY_AND_WATCHED',\n", 111 | " 'Birthday '\n", 112 | " 'tick '\n", 113 | " 'nearer'],\n", 114 | " ['LAY_AND_WATCHED',\n", 115 | " 'Birthday'],\n", 116 | " ['SHUFFLED_OFF_INTO',\n", 117 | " 'Kitchen'],\n", 118 | " ['MOVED_OUT_OF',\n", 119 | " 'Cupboard'],\n", 120 | " ['MOVED_INTO',\n", 121 | " \"Dudley's \"\n", 122 | " 'second '\n", 123 | " 'bedroom'],\n", 124 | " ['COULD_SEE',\n", 125 | " 'Uncle '\n", 126 | " \"vernon's \"\n", 127 | " 'shiny '\n", 128 | " 'black '\n", 129 | " 'shoes'],\n", 130 | " ['RECEIVED',\n", 131 | " 'First '\n", 132 | " 'letter'],\n", 133 | " ['RECEIVED',\n", 134 | " 'Letter'],\n", 135 | " ['PICKED_UP',\n", 136 | " 'Letter'],\n", 137 | " ['STARED_AT',\n", 138 | " 'Letter'],\n", 139 | " ['EARNED',\n", 140 | " 'Longest-ever '\n", 141 | " 'punishment'],\n", 142 | " ['DREAMED_OF',\n", 143 | " 'Unknown '\n", 144 | " 'relation'],\n", 145 | " ['FELL_HARD_ON',\n", 146 | " 'Concrete '\n", 147 | " 'floor'],\n", 148 | " ['IMAGINED',\n", 149 | " 'School'],\n", 150 | " ['SITTING_IN',\n", 151 | " 'Car'],\n", 152 | " ['HATED',\n", 153 | " 'Mrs. '\n", 154 | " 'figg'],\n", 155 | " ['TURNING_OVER',\n", 156 | " 'Bacon'],\n", 157 | " ['WORE',\n", 158 | " 'Glasses'],\n", 159 | " ['SLEPT_IN',\n", 160 | " 'Cupboard'],\n", 161 | " ['HAD',\n", 162 | " 'Dream'],\n", 163 | " ['HAD',\n", 164 | " 'Sleepless '\n", 165 | " 'night'],\n", 166 | " ['HAD',\n", 167 | " 'Scar']]},\n", 168 | " 'kg_rel_text': [\"['TURNED_OUT_TO_BE', \"\n", 169 | " \"'Bag of chips']\",\n", 170 | " \"['ANGRILY', 'Said']\",\n", 171 | " \"['HEART_TWANGING_LIKE', \"\n", 172 | " \"'Giant elastic \"\n", 173 | " \"band']\",\n", 174 | " \"['RAN', 'Before']\",\n", 175 | " \"['LAY_AND_WATCHED', \"\n", 176 | " \"'Birthday tick \"\n", 177 | " \"nearer']\",\n", 178 | " \"['LAY_AND_WATCHED', \"\n", 179 | " \"'Birthday']\",\n", 180 | " \"['SHUFFLED_OFF_INTO', \"\n", 181 | " \"'Kitchen']\",\n", 182 | " \"['MOVED_OUT_OF', \"\n", 183 | " \"'Cupboard']\",\n", 184 | " \"['MOVED_INTO', \"\n", 185 | " '\"Dudley\\'s second '\n", 186 | " 'bedroom\"]',\n", 187 | " \"['COULD_SEE', \"\n", 188 | " '\"Uncle vernon\\'s '\n", 189 | " 'shiny black shoes\"]',\n", 190 | " \"['RECEIVED', 'First \"\n", 191 | " \"letter']\",\n", 192 | " \"['RECEIVED', \"\n", 193 | " \"'Letter']\",\n", 194 | " \"['PICKED_UP', \"\n", 195 | " \"'Letter']\",\n", 196 | " \"['STARED_AT', \"\n", 197 | " \"'Letter']\",\n", 198 | " \"['EARNED', \"\n", 199 | " \"'Longest-ever \"\n", 200 | " \"punishment']\",\n", 201 | " \"['DREAMED_OF', \"\n", 202 | " \"'Unknown relation']\",\n", 203 | " \"['FELL_HARD_ON', \"\n", 204 | " \"'Concrete floor']\",\n", 205 | " \"['IMAGINED', \"\n", 206 | " \"'School']\",\n", 207 | " \"['SITTING_IN', \"\n", 208 | " \"'Car']\",\n", 209 | " \"['HATED', 'Mrs. \"\n", 210 | " \"figg']\",\n", 211 | " \"['TURNING_OVER', \"\n", 212 | " \"'Bacon']\",\n", 213 | " \"['WORE', 'Glasses']\",\n", 214 | " \"['SLEPT_IN', \"\n", 215 | " \"'Cupboard']\",\n", 216 | " \"['HAD', 'Dream']\",\n", 217 | " \"['HAD', 'Sleepless \"\n", 218 | " \"night']\",\n", 219 | " \"['HAD', 'Scar']\"],\n", 220 | " 'kg_schema': {'schema': 'Node '\n", 221 | " 'properties '\n", 222 | " 'are the '\n", 223 | " 'following:\\n'\n", 224 | " 'Entity {id: '\n", 225 | " 'STRING}\\n'\n", 226 | " 'Relationship '\n", 227 | " 'properties '\n", 228 | " 'are the '\n", 229 | " 'following:\\n'\n", 230 | " '\\n'\n", 231 | " 'The '\n", 232 | " 'relationships '\n", 233 | " 'are the '\n", 234 | " 'following:\\n'\n", 235 | " \"(:Entity)-[:WAS_DIRECTOR_OF]->(:Entity),(:Entity)-[:HAD]->(:Entity),(:Entity)-[:CAME_INTO]->(:Entity),(:Entity)-[:FELL_ASLEEP_QUICKLY]->(:Entity),(:Entity)-[:SHUDDERED_TO_THINK]->(:Entity),(:Entity)-[:HATED]->(:Entity),(:Entity)-[:WERE]->(:Entity),(:Entity)-[:ARRIVED_IN]->(:Entity),(:Entity)-[:SAT_WITH]->(:Entity),(:Entity)-[:YELLED_AT]->(:Entity),(:Entity)-[:STOPPED]->(:Entity),(:Entity)-[:SAW]->(:Entity),(:Entity)-[:LAY_AWAKE]->(:Entity),(:Entity)-[:HAD_GOTTEN_INTO]->(:Entity),(:Entity)-[:FOUNDED_IN]->(:Entity),(:Entity)-[:IS]->(:Entity),(:Entity)-[:HAVE_BEEN_BEHAVING]->(:Entity),(:Entity)-[:HUNT_AT]->(:Entity),(:Entity)-[:KICKED]->(:Entity),(:Entity)-[:SLEPT_IN]->(:Entity),(:Entity)-[:WAS_HOWLING]->(:Entity),(:Entity)-[:LEARNED]->(:Entity),(:Entity)-[:UNWRAPPED]->(:Entity),(:Entity)-[:DANGLING_OVER]->(:Entity),(:Entity)-[:BAWLING_AT]->(:Entity),(:Entity)-[:SNORED]->(:Entity),(:Entity)-[:WAS]->(:Entity),(:Entity)-[:BROKE]->(:Entity),(:Entity)-[:THOUGHT]->(:Entity),(:Entity)-[:CELEBRATING]->(:Entity),(:Entity)-[:WAS_SHOWING]->(:Entity),(:Entity)-[:SLIPPED]->(:Entity),(:Entity)-[:TURNED]->(:Entity),(:Entity)-[:SAID]->(:Entity),(:Entity)-[:PATTED]->(:Entity),(:Entity)-[:TOLD]->(:Entity),(:Entity)-[:TOOK]->(:Entity),(:Entity)-[:LAID]->(:Entity),(:Entity)-[:WEARING]->(:Entity),(:Entity)-[:ASKED]->(:Entity),(:Entity)-[:BLEW]->(:Entity),(:Entity)-[:FLINCHED]->(:Entity),(:Entity)-[:PULLED_OUT]->(:Entity),(:Entity)-[:EYED]->(:Entity),(:Entity)-[:TURNED_UP_IN]->(:Entity),(:Entity)-[:HOLDING]->(:Entity),(:Entity)-[:SWUNG]->(:Entity),(:Entity)-[:SOBBED]->(:Entity),(:Entity)-[:MOVED_OUT_OF]->(:Entity),(:Entity)-[:SHUFFLED_OFF_INTO]->(:Entity),(:Entity)-[:LAY_AND_WATCHED]->(:Entity),(:Entity)-[:RAN]->(:Entity),(:Entity)-[:HEART_TWANGING_LIKE]->(:Entity),(:Entity)-[:ANGRILY]->(:Entity),(:Entity)-[:TURNED_OUT_TO_BE]->(:Entity),(:Entity)-[:WORE]->(:Entity),(:Entity)-[:TURNING_OVER]->(:Entity),(:Entity)-[:SITTING_IN]->(:Entity),(:Entity)-[:IMAGINED]->(:Entity),(:Entity)-[:FELL_HARD_ON]->(:Entity),(:Entity)-[:DREAMED_OF]->(:Entity),(:Entity)-[:EARNED]->(:Entity),(:Entity)-[:STARED_AT]->(:Entity),(:Entity)-[:PICKED_UP]->(:Entity),(:Entity)-[:RECEIVED]->(:Entity),(:Entity)-[:COULD_SEE]->(:Entity),(:Entity)-[:MOVED_INTO]->(:Entity),(:Entity)-[:WAS_ASLEEP_AT]->(:Entity),(:Entity)-[:WAS_NO_LONGER]->(:Entity),(:Entity)-[:ENTERED]->(:Entity),(:Entity)-[:HELD]->(:Entity),(:Entity)-[:HAD_BEEN_LYING_AT]->(:Entity),(:Entity)-[:SLAMMED]->(:Entity),(:Entity)-[:PARKED_AT]->(:Entity),(:Entity)-[:WAS_CARRYING]->(:Entity),(:Entity)-[:WAS_SAYING_IN]->(:Entity),(:Entity)-[:SNAPPED]->(:Entity),(:Entity)-[:READ]->(:Entity),(:Entity)-[:HAD_BEEN_CHASING]->(:Entity),(:Entity)-[:ATE_IN]->(:Entity),(:Entity)-[:DIDN'T_STOP_TO]->(:Entity),(:Entity)-[:JABBED_ITS_TAIL_AT]->(:Entity),(:Entity)-[:WAS_IN]->(:Entity),(:Entity)-[:DID_GO]->(:Entity),(:Entity)-[:CAME_THE_SOUND_OF]->(:Entity),(:Entity)-[:CAME]->(:Entity),(:Entity)-[:FOUND_THEIR_WAY_INTO]->(:Entity),(:Entity)-[:CAME_OVER_TO]->(:Entity),(:Entity)-[:BLEW_UP_AROUND]->(:Entity),(:Entity)-[:WAS_DANGLING_OVER]->(:Entity),(:Entity)-[:BECOMES_A_MEMBER_OF]->(:Entity),(:Entity)-[:KNOWN_FOR]->(:Entity),(:Entity)-[:LOYALTY_TO]->(:Entity),(:Entity)-[:BECOMES]->(:Entity),(:Entity)-[:IN]->(:Entity),(:Entity)-[:BY]->(:Entity),(:Entity)-[:GODFATHER_OF]->(:Entity),(:Entity)-[:FALSELY_ACCUSED_OF]->(:Entity),(:Entity)-[:BELIEVED_TO_HAVE_BEEN_KILLED_BY]->(:Entity),(:Entity)-[:RETURNS_TO]->(:Entity),(:Entity)-[:TEACHES]->(:Entity),(:Entity)-[:ASSIST]->(:Entity),(:Entity)-[:TODAY]->(:Entity),(:Entity)-[:MARRIED_TO]->(:Entity),(:Entity)-[:IS_MEMBER_OF]->(:Entity),(:Entity)-[:HAS]->(:Entity)\"}}}\n" 236 | ] 237 | } 238 | ], 239 | "source": [ 240 | "import pprint\n", 241 | "\n", 242 | "pp = pprint.PrettyPrinter()\n", 243 | "pp.pprint(response.metadata)" 244 | ] 245 | } 246 | ], 247 | "metadata": { 248 | "kernelspec": { 249 | "display_name": "Python 3", 250 | "language": "python", 251 | "name": "python3" 252 | }, 253 | "language_info": { 254 | "codemirror_mode": { 255 | "name": "ipython", 256 | "version": 3 257 | }, 258 | "file_extension": ".py", 259 | "mimetype": "text/x-python", 260 | "name": "python", 261 | "nbconvert_exporter": "python", 262 | "pygments_lexer": "ipython3", 263 | "version": "3.11.6" 264 | } 265 | }, 266 | "nbformat": 4, 267 | "nbformat_minor": 2 268 | } 269 | -------------------------------------------------------------------------------- /data/chapter/2.txt: -------------------------------------------------------------------------------- 1 | THE VANISHING GLASS 2 | 3 | Nearly ten years had passed since the Dursleys had woken up to find 4 | their nephew on the front step, but Privet Drive had hardly changed at 5 | all. The sun rose on the same tidy front gardens and lit up the brass 6 | number four on the Dursleys' front door; it crept into their living 7 | room, which was almost exactly the same as it had been on the night when 8 | Mr. Dursley had seen that fateful news report about the owls. Only the 9 | photographs on the mantelpiece really showed how much time had passed. 10 | Ten years ago, there had been lots of pictures of what looked like a 11 | large pink beach ball wearing different-colored bonnets -- but Dudley 12 | Dursley was no longer a baby, and now the photographs showed a large 13 | blond boy riding his first bicycle, on a carousel at the fair, playing a 14 | computer game with his father, being hugged and kissed by his mother. 15 | The room held no sign at all that another boy lived in the house, too. 16 | 17 | Yet Harry Potter was still there, asleep at the moment, but not for 18 | long. His Aunt Petunia was awake and it was her shrill voice that made 19 | the first noise of the day. 20 | 21 | "Up! Get up! Now!" 22 | 23 | Harry woke with a start. His aunt rapped on the door again. 24 | 25 | "Up!" she screeched. Harry heard her walking toward the kitchen and then 26 | the sound of the frying pan being put on the stove. He rolled onto his 27 | back and tried to remember the dream he had been having. It had been a 28 | good one. There had been a flying motorcycle in it. He had a funny 29 | feeling he'd had the same dream before. 30 | 31 | His aunt was back outside the door. 32 | 33 | "Are you up yet?" she demanded. 34 | 35 | "Nearly," said Harry. 36 | 37 | "Well, get a move on, I want you to look after the bacon. And don't you 38 | dare let it burn, I want everything perfect on Duddy's birthday." 39 | 40 | Harry groaned. 41 | 42 | "What did you say?" his aunt snapped through the door. 43 | 44 | "Nothing, nothing..." 45 | 46 | Dudley's birthday -- how could he have forgotten? Harry got slowly out 47 | of bed and started looking for socks. He found a pair under his bed and, 48 | after pulling a spider off one of them, put them on. Harry was used to 49 | spiders, because the cupboard under the stairs was full of them, and 50 | that was where he slept. 51 | 52 | When he was dressed he went down the hall into the kitchen. The table 53 | was almost hidden beneath all Dudley's birthday presents. It looked as 54 | though Dudley had gotten the new computer he wanted, not to mention the 55 | second television and the racing bike. Exactly why Dudley wanted a 56 | racing bike was a mystery to Harry, as Dudley was very fat and hated 57 | exercise -- unless of course it involved punching somebody. Dudley's 58 | favorite punching bag was Harry, but he couldn't often catch him. Harry 59 | didn't look it, but he was very fast. 60 | 61 | Perhaps it had something to do with living in a dark cupboard, but Harry 62 | had always been small and skinny for his age. He looked even smaller and 63 | skinnier than he really was because all he had to wear were old clothes 64 | of Dudley's, and Dudley was about four times bigger than he was. Harry 65 | had a thin face, knobbly knees, black hair, and bright green eyes. He 66 | wore round glasses held together with a lot of Scotch tape because of 67 | all the times Dudley had punched him on the nose. The only thing Harry 68 | liked about his own appearance was a very thin scar on his forehead that 69 | was shaped like a bolt of lightning. He had had it as long as he could 70 | remember, and the first question he could ever remember asking his Aunt 71 | Petunia was how he had gotten it. 72 | 73 | "In the car crash when your parents died," she had said. "And don't ask 74 | questions." 75 | 76 | Don't ask questions -- that was the first rule for a quiet life with the 77 | Dursleys. 78 | 79 | Uncle Vernon entered the kitchen as Harry was turning over the bacon. 80 | 81 | "Comb your hair!" he barked, by way of a morning greeting. 82 | 83 | About once a week, Uncle Vernon looked over the top of his newspaper and 84 | shouted that Harry needed a haircut. Harry must have had more haircuts 85 | than the rest of the boys in his class put 86 | 87 | together, but it made no difference, his hair simply grew that way -- 88 | all over the place. 89 | 90 | Harry was frying eggs by the time Dudley arrived in the kitchen with his 91 | mother. Dudley looked a lot like Uncle Vernon. He had a large pink face, 92 | not much neck, small, watery blue eyes, and thick blond hair that lay 93 | smoothly on his thick, fat head. Aunt Petunia often said that Dudley 94 | looked like a baby angel -- Harry often said that Dudley looked like a 95 | pig in a wig. 96 | 97 | Harry put the plates of egg and bacon on the table, which was difficult 98 | as there wasn't much room. Dudley, meanwhile, was counting his presents. 99 | His face fell. 100 | 101 | "Thirty-six," he said, looking up at his mother and father. "That's two 102 | less than last year." 103 | 104 | "Darling, you haven't counted Auntie Marge's present, see, it's here 105 | under this big one from Mommy and Daddy." 106 | 107 | "All right, thirty-seven then," said Dudley, going red in the face. 108 | Harry, who could see a huge Dudley tantrum coming on, began wolfing down 109 | his bacon as fast as possible in case Dudley turned the table over. 110 | 111 | Aunt Petunia obviously scented danger, too, because she said quickly, 112 | "And we'll buy you another two presents while we're out today. How's 113 | that, popkin? Two more presents. Is that all right'' 114 | 115 | Dudley thought for a moment. It looked like hard work. Finally he said 116 | slowly, "So I'll have thirty ... thirty..." 117 | 118 | "Thirty-nine, sweetums," said Aunt Petunia. 119 | 120 | "Oh." Dudley sat down heavily and grabbed the nearest parcel. "All right 121 | then." 122 | 123 | Uncle Vernon chuckled. "Little tyke wants his money's worth, just like 124 | his father. 'Atta boy, Dudley!" He ruffled Dudley's hair. 125 | 126 | At that moment the telephone rang and Aunt Petunia went to answer it 127 | while Harry and Uncle Vernon watched Dudley unwrap the racing bike, a 128 | video camera, a remote control airplane, sixteen new computer games, and 129 | a VCR. He was ripping the paper off a gold wristwatch when Aunt Petunia 130 | came back from the telephone looking both angry and worried. 131 | 132 | "Bad news, Vernon," she said. "Mrs. Figg's broken her leg. She can't 133 | take him." She jerked her head in Harry's direction. 134 | 135 | Dudley's mouth fell open in horror, but Harry's heart gave a leap. Every 136 | year on Dudley's birthday, his parents took him and a friend out for the 137 | day, to adventure parks, hamburger restaurants, or the movies. Every 138 | year, Harry was left behind with Mrs. Figg, a mad old lady who lived two 139 | streets away. Harry hated it there. The whole house smelled of cabbage 140 | and Mrs. Figg made him look at photographs of all the cats she'd ever 141 | owned. 142 | 143 | "Now what?" said Aunt Petunia, looking furiously at Harry as though he'd 144 | planned this. Harry knew he ought to feel sorry that Mrs. Figg had 145 | broken her leg, but it wasn't easy when he reminded himself it would be 146 | a whole year before he had to look at Tibbles, Snowy, Mr. Paws, and 147 | Tufty again. 148 | 149 | "We could phone Marge," Uncle Vernon suggested. 150 | 151 | "Don't be silly, Vernon, she hates the boy." 152 | 153 | The Dursleys often spoke about Harry like this, as though he wasn't 154 | there -- or rather, as though he was something very nasty that couldn't 155 | understand them, like a slug. 156 | 157 | "What about what's-her-name, your friend -- Yvonne?" 158 | 159 | "On vacation in Majorca," snapped Aunt Petunia. 160 | 161 | "You could just leave me here," Harry put in hopefully (he'd be able to 162 | watch what he wanted on television for a change and maybe even have a go 163 | on Dudley's computer). 164 | 165 | Aunt Petunia looked as though she'd just swallowed a lemon. 166 | 167 | "And come back and find the house in ruins?" she snarled. 168 | 169 | "I won't blow up the house," said Harry, but they weren't listening. 170 | 171 | "I suppose we could take him to the zoo," said Aunt Petunia slowly, "... 172 | and leave him in the car...." 173 | 174 | "That car's new, he's not sitting in it alone...." 175 | 176 | Dudley began to cry loudly. In fact, he wasn't really crying -- it had 177 | been years since he'd really cried -- but he knew that if he screwed up 178 | his face and wailed, his mother would give him anything he wanted. 179 | 180 | "Dinky Duddydums, don't cry, Mummy won't let him spoil your special 181 | day!" she cried, flinging her arms around him. 182 | 183 | "I... don't... want... him... t-t-to come!" Dudley yelled between huge, 184 | pretend sobs. "He always sp- spoils everything!" He shot Harry a nasty 185 | grin through the gap in his mother's arms. 186 | 187 | Just then, the doorbell rang -- "Oh, good Lord, they're here!" said Aunt 188 | Petunia frantically -- and a moment later, Dudley's best friend, Piers 189 | Polkiss, walked in with his mother. Piers was a scrawny boy with a face 190 | like a rat. He was usually the one who held people's arms behind their 191 | backs while Dudley hit them. Dudley stopped pretending to cry at once. 192 | 193 | Half an hour later, Harry, who couldn't believe his luck, was sitting in 194 | the back of the Dursleys' car with Piers and Dudley, on the way to the 195 | zoo for the first time in his life. His aunt and uncle hadn't been able 196 | to think of anything else to do with him, but before they'd left, Uncle 197 | Vernon had taken Harry aside. 198 | 199 | "I'm warning you," he had said, putting his large purple face right up 200 | close to Harry's, "I'm warning you now, boy -- any funny business, 201 | anything at all -- and you'll be in that cupboard from now until 202 | Christmas." 203 | 204 | "I'm not going to do anything," said Harry, "honestly.. 205 | 206 | But Uncle Vernon didn't believe him. No one ever did. 207 | 208 | The problem was, strange things often happened around Harry and it was 209 | just no good telling the Dursleys he didn't make them happen. 210 | 211 | Once, Aunt Petunia, tired of Harry coming back from the barbers looking 212 | as though he hadn't been at all, had taken a pair of kitchen scissors 213 | and cut his hair so short he was almost bald except for his bangs, which 214 | she left "to hide that horrible scar." Dudley had laughed himself silly 215 | at Harry, who spent a sleepless night imagining school the next day, 216 | where he was already laughed at for his baggy clothes and taped glasses. 217 | Next morning, however, he had gotten up to find his hair exactly as it 218 | had been before Aunt Petunia had sheared it off He had been given a week 219 | in his cupboard for this, even though he had tried to explain that he 220 | couldn't explain how it had grown back so quickly. 221 | 222 | Another time, Aunt Petunia had been trying to force him into a revolting 223 | old sweater of Dudley's (brown with orange puff balls) -- The harder she 224 | tried to pull it over his head, the smaller it seemed to become, until 225 | finally it might have fitted a hand puppet, but certainly wouldn't fit 226 | Harry. Aunt Petunia had decided it must have shrunk in the wash and, to 227 | his great relief, Harry wasn't punished. 228 | 229 | On the other hand, he'd gotten into terrible trouble for being found on 230 | the roof of the school kitchens. Dudley's gang had been chasing him as 231 | usual when, as much to Harry's surprise as anyone else's, there he was 232 | sitting on the chimney. The Dursleys had received a very angry letter 233 | from Harry's headmistress telling them Harry had been climbing school 234 | buildings. But all he'd tried to do (as he shouted at Uncle Vernon 235 | through the locked door of his cupboard) was jump behind the big trash 236 | cans outside the kitchen doors. Harry supposed that the wind must have 237 | caught him in mid- jump. 238 | 239 | But today, nothing was going to go wrong. It was even worth being with 240 | Dudley and Piers to be spending the day somewhere that wasn't school, 241 | his cupboard, or Mrs. Figg's cabbage-smelling living room. 242 | 243 | While he drove, Uncle Vernon complained to Aunt Petunia. He liked to 244 | complain about things: people at work, Harry, the council, Harry, the 245 | bank, and Harry were just a few of his favorite subjects. This morning, 246 | it was motorcycles. 247 | 248 | "... roaring along like maniacs, the young hoodlums," he said, as a 249 | motorcycle overtook them. 250 | 251 | I had a dream about a motorcycle," said Harry, remembering suddenly. "It 252 | was flying." 253 | 254 | Uncle Vernon nearly crashed into the car in front. He turned right 255 | around in his seat and yelled at Harry, his face like a gigantic beet 256 | with a mustache: "MOTORCYCLES DON'T FLY!" 257 | 258 | Dudley and Piers sniggered. 259 | 260 | I know they don't," said Harry. "It was only a dream." 261 | 262 | But he wished he hadn't said anything. If there was one thing the 263 | Dursleys hated even more than his asking questions, it was his talking 264 | about anything acting in a way it shouldn't, no matter if it was in a 265 | dream or even a cartoon -- they seemed to think he might get dangerous 266 | ideas. 267 | 268 | It was a very sunny Saturday and the zoo was crowded with families. The 269 | Dursleys bought Dudley and Piers large chocolate ice creams at the 270 | entrance and then, because the smiling lady in the van had asked Harry 271 | what he wanted before they could hurry him away, they bought him a cheap 272 | lemon ice pop. It wasn't bad, either, Harry thought, licking it as they 273 | watched a gorilla scratching its head who looked remarkably like Dudley, 274 | except that it wasn't blond. 275 | 276 | Harry had the best morning he'd had in a long time. He was careful to 277 | walk a little way apart from the Dursleys so that Dudley and Piers, who 278 | were starting to get bored with the animals by lunchtime, wouldn't fall 279 | back on their favorite hobby of hitting him. They ate in the zoo 280 | restaurant, and when Dudley had a tantrum because his knickerbocker 281 | glory didn't have enough ice cream on top, Uncle Vernon bought him 282 | another one and Harry was allowed to finish the first. 283 | 284 | Harry felt, afterward, that he should have known it was all too good to 285 | last. 286 | 287 | After lunch they went to the reptile house. It was cool and dark in 288 | there, with lit windows all along the walls. Behind the glass, all sorts 289 | of lizards and snakes were crawling and slithering over bits of wood and 290 | stone. Dudley and Piers wanted to see huge, poisonous cobras and thick, 291 | man-crushing pythons. Dudley quickly found the largest snake in the 292 | place. It could have wrapped its body twice around Uncle Vernon's car 293 | and crushed it into a trash can -- but at the moment it didn't look in 294 | the mood. In fact, it was fast asleep. 295 | 296 | Dudley stood with his nose pressed against the glass, staring at the 297 | glistening brown coils. 298 | 299 | "Make it move," he whined at his father. Uncle Vernon tapped on the 300 | glass, but the snake didn't budge. 301 | 302 | "Do it again," Dudley ordered. Uncle Vernon rapped the glass smartly 303 | with his knuckles, but the snake just snoozed on. 304 | 305 | "This is boring," Dudley moaned. He shuffled away. 306 | 307 | Harry moved in front of the tank and looked intently at the snake. He 308 | wouldn't have been surprised if it had died of boredom itself -- no 309 | company except stupid people drumming their fingers on the glass trying 310 | to disturb it all day long. It was worse than having a cupboard as a 311 | bedroom, where the only visitor was Aunt Petunia hammering on the door 312 | to wake you up; at least he got to visit the rest of the house. 313 | 314 | The snake suddenly opened its beady eyes. Slowly, very slowly, it raised 315 | its head until its eyes were on a level with Harry's. 316 | 317 | It winked. 318 | 319 | Harry stared. Then he looked quickly around to see if anyone was 320 | watching. They weren't. He looked back at the snake and winked, too. 321 | 322 | The snake jerked its head toward Uncle Vernon and Dudley, then raised 323 | its eyes to the ceiling. It gave Harry a look that said quite plainly: 324 | 325 | "I get that all the time. 326 | 327 | "I know," Harry murmured through the glass, though he wasn't sure the 328 | snake could hear him. "It must be really annoying." 329 | 330 | The snake nodded vigorously. 331 | 332 | "Where do you come from, anyway?" Harry asked. 333 | 334 | The snake jabbed its tail at a little sign next to the glass. Harry 335 | peered at it. 336 | 337 | Boa Constrictor, Brazil. 338 | 339 | "Was it nice there?" 340 | 341 | The boa constrictor jabbed its tail at the sign again and Harry read on: 342 | This specimen was bred in the zoo. "Oh, I see -- so you've never been to 343 | Brazil?" 344 | 345 | As the snake shook its head, a deafening shout behind Harry made both of 346 | them jump. 347 | 348 | "DUDLEY! MR. DURSLEY! COME AND LOOK AT THIS SNAKE! YOU WON'T BELIEVE 349 | WHAT IT'S DOING!" 350 | 351 | Dudley came waddling toward them as fast as he could. 352 | 353 | "Out of the way, you," he said, punching Harry in the ribs. Caught by 354 | surprise, Harry fell hard on the concrete floor. What came next happened 355 | so fast no one saw how it happened -- one second, Piers and Dudley were 356 | leaning right up close to the glass, the next, they had leapt back with 357 | howls of horror. 358 | 359 | Harry sat up and gasped; the glass front of the boa constrictor's tank 360 | had vanished. The great snake was uncoiling itself rapidly, slithering 361 | out onto the floor. People throughout the reptile house screamed and 362 | started running for the exits. 363 | 364 | As the snake slid swiftly past him, Harry could have sworn a low, 365 | hissing voice said, "Brazil, here I come.... Thanksss, amigo." 366 | 367 | The keeper of the reptile house was in shock. 368 | 369 | "But the glass," he kept saying, "where did the glass go?" 370 | 371 | The zoo director himself made Aunt Petunia a cup of strong, sweet tea 372 | while he apologized over and over again. Piers and Dudley could only 373 | gibber. As far as Harry had seen, the snake hadn't done anything except 374 | snap playfully at their heels as it passed, but by the time they were 375 | all back in Uncle Vernon's car, Dudley was telling them how it had 376 | nearly bitten off his leg, while Piers was swearing it had tried to 377 | squeeze him to death. But worst of all, for Harry at least, was Piers 378 | calming down enough to say, "Harry was talking to it, weren't you, 379 | Harry?" 380 | 381 | Uncle Vernon waited until Piers was safely out of the house before 382 | starting on Harry. He was so angry he could hardly speak. He managed to 383 | say, "Go -- cupboard -- stay -- no meals," before he collapsed into a 384 | chair, and Aunt Petunia had to run and get him a large brandy. 385 | 386 | Harry lay in his dark cupboard much later, wishing he had a watch. He 387 | didn't know what time it was and he couldn't be sure the Dursleys were 388 | asleep yet. Until they were, he couldn't risk sneaking to the kitchen 389 | for some food. 390 | 391 | He'd lived with the Dursleys almost ten years, ten miserable years, as 392 | long as he could remember, ever since he'd been a baby and his parents 393 | had died in that car crash. He couldn't remember being in the car when 394 | his parents had died. Sometimes, when he strained his memory during long 395 | hours in his cupboard, he came up with a strange vision: a blinding 396 | flash of green light and a burn- ing pain on his forehead. This, he 397 | supposed, was the crash, though he couldn't imagine where all the green 398 | light came from. He couldn't remember his parents at all. His aunt and 399 | uncle never spoke about them, and of course he was forbidden to ask 400 | questions. There were no photographs of them in the house. 401 | 402 | When he had been younger, Harry had dreamed and dreamed of some unknown 403 | relation coming to take him away, but it had never happened; the 404 | Dursleys were his only family. Yet sometimes he thought (or maybe hoped) 405 | that strangers in the street seemed to know him. Very strange strangers 406 | they were, too. A tiny man in a violet top hat had bowed to him once 407 | while out shopping with Aunt Petunia and Dudley. After asking Harry 408 | furiously if he knew the man, Aunt Petunia had rushed them out of the 409 | shop without buying anything. A wild-looking old woman dressed all in 410 | green had waved merrily at him once on a bus. A bald man in a very long 411 | purple coat had actually shaken his hand in the street the other day and 412 | then walked away without a word. The weirdest thing about all these 413 | people was the way they seemed to vanish the second Harry tried to get a 414 | closer look. 415 | 416 | At school, Harry had no one. Everybody knew that Dudley's gang hated 417 | that odd Harry Potter in his baggy old clothes and broken glasses, and 418 | nobody liked to disagree with Dudley's gang. 419 | -------------------------------------------------------------------------------- /streamlit_app/data/harry_potter/2.txt: -------------------------------------------------------------------------------- 1 | THE VANISHING GLASS 2 | 3 | Nearly ten years had passed since the Dursleys had woken up to find 4 | their nephew on the front step, but Privet Drive had hardly changed at 5 | all. The sun rose on the same tidy front gardens and lit up the brass 6 | number four on the Dursleys' front door; it crept into their living 7 | room, which was almost exactly the same as it had been on the night when 8 | Mr. Dursley had seen that fateful news report about the owls. Only the 9 | photographs on the mantelpiece really showed how much time had passed. 10 | Ten years ago, there had been lots of pictures of what looked like a 11 | large pink beach ball wearing different-colored bonnets -- but Dudley 12 | Dursley was no longer a baby, and now the photographs showed a large 13 | blond boy riding his first bicycle, on a carousel at the fair, playing a 14 | computer game with his father, being hugged and kissed by his mother. 15 | The room held no sign at all that another boy lived in the house, too. 16 | 17 | Yet Harry Potter was still there, asleep at the moment, but not for 18 | long. His Aunt Petunia was awake and it was her shrill voice that made 19 | the first noise of the day. 20 | 21 | "Up! Get up! Now!" 22 | 23 | Harry woke with a start. His aunt rapped on the door again. 24 | 25 | "Up!" she screeched. Harry heard her walking toward the kitchen and then 26 | the sound of the frying pan being put on the stove. He rolled onto his 27 | back and tried to remember the dream he had been having. It had been a 28 | good one. There had been a flying motorcycle in it. He had a funny 29 | feeling he'd had the same dream before. 30 | 31 | His aunt was back outside the door. 32 | 33 | "Are you up yet?" she demanded. 34 | 35 | "Nearly," said Harry. 36 | 37 | "Well, get a move on, I want you to look after the bacon. And don't you 38 | dare let it burn, I want everything perfect on Duddy's birthday." 39 | 40 | Harry groaned. 41 | 42 | "What did you say?" his aunt snapped through the door. 43 | 44 | "Nothing, nothing..." 45 | 46 | Dudley's birthday -- how could he have forgotten? Harry got slowly out 47 | of bed and started looking for socks. He found a pair under his bed and, 48 | after pulling a spider off one of them, put them on. Harry was used to 49 | spiders, because the cupboard under the stairs was full of them, and 50 | that was where he slept. 51 | 52 | When he was dressed he went down the hall into the kitchen. The table 53 | was almost hidden beneath all Dudley's birthday presents. It looked as 54 | though Dudley had gotten the new computer he wanted, not to mention the 55 | second television and the racing bike. Exactly why Dudley wanted a 56 | racing bike was a mystery to Harry, as Dudley was very fat and hated 57 | exercise -- unless of course it involved punching somebody. Dudley's 58 | favorite punching bag was Harry, but he couldn't often catch him. Harry 59 | didn't look it, but he was very fast. 60 | 61 | Perhaps it had something to do with living in a dark cupboard, but Harry 62 | had always been small and skinny for his age. He looked even smaller and 63 | skinnier than he really was because all he had to wear were old clothes 64 | of Dudley's, and Dudley was about four times bigger than he was. Harry 65 | had a thin face, knobbly knees, black hair, and bright green eyes. He 66 | wore round glasses held together with a lot of Scotch tape because of 67 | all the times Dudley had punched him on the nose. The only thing Harry 68 | liked about his own appearance was a very thin scar on his forehead that 69 | was shaped like a bolt of lightning. He had had it as long as he could 70 | remember, and the first question he could ever remember asking his Aunt 71 | Petunia was how he had gotten it. 72 | 73 | "In the car crash when your parents died," she had said. "And don't ask 74 | questions." 75 | 76 | Don't ask questions -- that was the first rule for a quiet life with the 77 | Dursleys. 78 | 79 | Uncle Vernon entered the kitchen as Harry was turning over the bacon. 80 | 81 | "Comb your hair!" he barked, by way of a morning greeting. 82 | 83 | About once a week, Uncle Vernon looked over the top of his newspaper and 84 | shouted that Harry needed a haircut. Harry must have had more haircuts 85 | than the rest of the boys in his class put 86 | 87 | together, but it made no difference, his hair simply grew that way -- 88 | all over the place. 89 | 90 | Harry was frying eggs by the time Dudley arrived in the kitchen with his 91 | mother. Dudley looked a lot like Uncle Vernon. He had a large pink face, 92 | not much neck, small, watery blue eyes, and thick blond hair that lay 93 | smoothly on his thick, fat head. Aunt Petunia often said that Dudley 94 | looked like a baby angel -- Harry often said that Dudley looked like a 95 | pig in a wig. 96 | 97 | Harry put the plates of egg and bacon on the table, which was difficult 98 | as there wasn't much room. Dudley, meanwhile, was counting his presents. 99 | His face fell. 100 | 101 | "Thirty-six," he said, looking up at his mother and father. "That's two 102 | less than last year." 103 | 104 | "Darling, you haven't counted Auntie Marge's present, see, it's here 105 | under this big one from Mommy and Daddy." 106 | 107 | "All right, thirty-seven then," said Dudley, going red in the face. 108 | Harry, who could see a huge Dudley tantrum coming on, began wolfing down 109 | his bacon as fast as possible in case Dudley turned the table over. 110 | 111 | Aunt Petunia obviously scented danger, too, because she said quickly, 112 | "And we'll buy you another two presents while we're out today. How's 113 | that, popkin? Two more presents. Is that all right'' 114 | 115 | Dudley thought for a moment. It looked like hard work. Finally he said 116 | slowly, "So I'll have thirty ... thirty..." 117 | 118 | "Thirty-nine, sweetums," said Aunt Petunia. 119 | 120 | "Oh." Dudley sat down heavily and grabbed the nearest parcel. "All right 121 | then." 122 | 123 | Uncle Vernon chuckled. "Little tyke wants his money's worth, just like 124 | his father. 'Atta boy, Dudley!" He ruffled Dudley's hair. 125 | 126 | At that moment the telephone rang and Aunt Petunia went to answer it 127 | while Harry and Uncle Vernon watched Dudley unwrap the racing bike, a 128 | video camera, a remote control airplane, sixteen new computer games, and 129 | a VCR. He was ripping the paper off a gold wristwatch when Aunt Petunia 130 | came back from the telephone looking both angry and worried. 131 | 132 | "Bad news, Vernon," she said. "Mrs. Figg's broken her leg. She can't 133 | take him." She jerked her head in Harry's direction. 134 | 135 | Dudley's mouth fell open in horror, but Harry's heart gave a leap. Every 136 | year on Dudley's birthday, his parents took him and a friend out for the 137 | day, to adventure parks, hamburger restaurants, or the movies. Every 138 | year, Harry was left behind with Mrs. Figg, a mad old lady who lived two 139 | streets away. Harry hated it there. The whole house smelled of cabbage 140 | and Mrs. Figg made him look at photographs of all the cats she'd ever 141 | owned. 142 | 143 | "Now what?" said Aunt Petunia, looking furiously at Harry as though he'd 144 | planned this. Harry knew he ought to feel sorry that Mrs. Figg had 145 | broken her leg, but it wasn't easy when he reminded himself it would be 146 | a whole year before he had to look at Tibbles, Snowy, Mr. Paws, and 147 | Tufty again. 148 | 149 | "We could phone Marge," Uncle Vernon suggested. 150 | 151 | "Don't be silly, Vernon, she hates the boy." 152 | 153 | The Dursleys often spoke about Harry like this, as though he wasn't 154 | there -- or rather, as though he was something very nasty that couldn't 155 | understand them, like a slug. 156 | 157 | "What about what's-her-name, your friend -- Yvonne?" 158 | 159 | "On vacation in Majorca," snapped Aunt Petunia. 160 | 161 | "You could just leave me here," Harry put in hopefully (he'd be able to 162 | watch what he wanted on television for a change and maybe even have a go 163 | on Dudley's computer). 164 | 165 | Aunt Petunia looked as though she'd just swallowed a lemon. 166 | 167 | "And come back and find the house in ruins?" she snarled. 168 | 169 | "I won't blow up the house," said Harry, but they weren't listening. 170 | 171 | "I suppose we could take him to the zoo," said Aunt Petunia slowly, "... 172 | and leave him in the car...." 173 | 174 | "That car's new, he's not sitting in it alone...." 175 | 176 | Dudley began to cry loudly. In fact, he wasn't really crying -- it had 177 | been years since he'd really cried -- but he knew that if he screwed up 178 | his face and wailed, his mother would give him anything he wanted. 179 | 180 | "Dinky Duddydums, don't cry, Mummy won't let him spoil your special 181 | day!" she cried, flinging her arms around him. 182 | 183 | "I... don't... want... him... t-t-to come!" Dudley yelled between huge, 184 | pretend sobs. "He always sp- spoils everything!" He shot Harry a nasty 185 | grin through the gap in his mother's arms. 186 | 187 | Just then, the doorbell rang -- "Oh, good Lord, they're here!" said Aunt 188 | Petunia frantically -- and a moment later, Dudley's best friend, Piers 189 | Polkiss, walked in with his mother. Piers was a scrawny boy with a face 190 | like a rat. He was usually the one who held people's arms behind their 191 | backs while Dudley hit them. Dudley stopped pretending to cry at once. 192 | 193 | Half an hour later, Harry, who couldn't believe his luck, was sitting in 194 | the back of the Dursleys' car with Piers and Dudley, on the way to the 195 | zoo for the first time in his life. His aunt and uncle hadn't been able 196 | to think of anything else to do with him, but before they'd left, Uncle 197 | Vernon had taken Harry aside. 198 | 199 | "I'm warning you," he had said, putting his large purple face right up 200 | close to Harry's, "I'm warning you now, boy -- any funny business, 201 | anything at all -- and you'll be in that cupboard from now until 202 | Christmas." 203 | 204 | "I'm not going to do anything," said Harry, "honestly.. 205 | 206 | But Uncle Vernon didn't believe him. No one ever did. 207 | 208 | The problem was, strange things often happened around Harry and it was 209 | just no good telling the Dursleys he didn't make them happen. 210 | 211 | Once, Aunt Petunia, tired of Harry coming back from the barbers looking 212 | as though he hadn't been at all, had taken a pair of kitchen scissors 213 | and cut his hair so short he was almost bald except for his bangs, which 214 | she left "to hide that horrible scar." Dudley had laughed himself silly 215 | at Harry, who spent a sleepless night imagining school the next day, 216 | where he was already laughed at for his baggy clothes and taped glasses. 217 | Next morning, however, he had gotten up to find his hair exactly as it 218 | had been before Aunt Petunia had sheared it off He had been given a week 219 | in his cupboard for this, even though he had tried to explain that he 220 | couldn't explain how it had grown back so quickly. 221 | 222 | Another time, Aunt Petunia had been trying to force him into a revolting 223 | old sweater of Dudley's (brown with orange puff balls) -- The harder she 224 | tried to pull it over his head, the smaller it seemed to become, until 225 | finally it might have fitted a hand puppet, but certainly wouldn't fit 226 | Harry. Aunt Petunia had decided it must have shrunk in the wash and, to 227 | his great relief, Harry wasn't punished. 228 | 229 | On the other hand, he'd gotten into terrible trouble for being found on 230 | the roof of the school kitchens. Dudley's gang had been chasing him as 231 | usual when, as much to Harry's surprise as anyone else's, there he was 232 | sitting on the chimney. The Dursleys had received a very angry letter 233 | from Harry's headmistress telling them Harry had been climbing school 234 | buildings. But all he'd tried to do (as he shouted at Uncle Vernon 235 | through the locked door of his cupboard) was jump behind the big trash 236 | cans outside the kitchen doors. Harry supposed that the wind must have 237 | caught him in mid- jump. 238 | 239 | But today, nothing was going to go wrong. It was even worth being with 240 | Dudley and Piers to be spending the day somewhere that wasn't school, 241 | his cupboard, or Mrs. Figg's cabbage-smelling living room. 242 | 243 | While he drove, Uncle Vernon complained to Aunt Petunia. He liked to 244 | complain about things: people at work, Harry, the council, Harry, the 245 | bank, and Harry were just a few of his favorite subjects. This morning, 246 | it was motorcycles. 247 | 248 | "... roaring along like maniacs, the young hoodlums," he said, as a 249 | motorcycle overtook them. 250 | 251 | I had a dream about a motorcycle," said Harry, remembering suddenly. "It 252 | was flying." 253 | 254 | Uncle Vernon nearly crashed into the car in front. He turned right 255 | around in his seat and yelled at Harry, his face like a gigantic beet 256 | with a mustache: "MOTORCYCLES DON'T FLY!" 257 | 258 | Dudley and Piers sniggered. 259 | 260 | I know they don't," said Harry. "It was only a dream." 261 | 262 | But he wished he hadn't said anything. If there was one thing the 263 | Dursleys hated even more than his asking questions, it was his talking 264 | about anything acting in a way it shouldn't, no matter if it was in a 265 | dream or even a cartoon -- they seemed to think he might get dangerous 266 | ideas. 267 | 268 | It was a very sunny Saturday and the zoo was crowded with families. The 269 | Dursleys bought Dudley and Piers large chocolate ice creams at the 270 | entrance and then, because the smiling lady in the van had asked Harry 271 | what he wanted before they could hurry him away, they bought him a cheap 272 | lemon ice pop. It wasn't bad, either, Harry thought, licking it as they 273 | watched a gorilla scratching its head who looked remarkably like Dudley, 274 | except that it wasn't blond. 275 | 276 | Harry had the best morning he'd had in a long time. He was careful to 277 | walk a little way apart from the Dursleys so that Dudley and Piers, who 278 | were starting to get bored with the animals by lunchtime, wouldn't fall 279 | back on their favorite hobby of hitting him. They ate in the zoo 280 | restaurant, and when Dudley had a tantrum because his knickerbocker 281 | glory didn't have enough ice cream on top, Uncle Vernon bought him 282 | another one and Harry was allowed to finish the first. 283 | 284 | Harry felt, afterward, that he should have known it was all too good to 285 | last. 286 | 287 | After lunch they went to the reptile house. It was cool and dark in 288 | there, with lit windows all along the walls. Behind the glass, all sorts 289 | of lizards and snakes were crawling and slithering over bits of wood and 290 | stone. Dudley and Piers wanted to see huge, poisonous cobras and thick, 291 | man-crushing pythons. Dudley quickly found the largest snake in the 292 | place. It could have wrapped its body twice around Uncle Vernon's car 293 | and crushed it into a trash can -- but at the moment it didn't look in 294 | the mood. In fact, it was fast asleep. 295 | 296 | Dudley stood with his nose pressed against the glass, staring at the 297 | glistening brown coils. 298 | 299 | "Make it move," he whined at his father. Uncle Vernon tapped on the 300 | glass, but the snake didn't budge. 301 | 302 | "Do it again," Dudley ordered. Uncle Vernon rapped the glass smartly 303 | with his knuckles, but the snake just snoozed on. 304 | 305 | "This is boring," Dudley moaned. He shuffled away. 306 | 307 | Harry moved in front of the tank and looked intently at the snake. He 308 | wouldn't have been surprised if it had died of boredom itself -- no 309 | company except stupid people drumming their fingers on the glass trying 310 | to disturb it all day long. It was worse than having a cupboard as a 311 | bedroom, where the only visitor was Aunt Petunia hammering on the door 312 | to wake you up; at least he got to visit the rest of the house. 313 | 314 | The snake suddenly opened its beady eyes. Slowly, very slowly, it raised 315 | its head until its eyes were on a level with Harry's. 316 | 317 | It winked. 318 | 319 | Harry stared. Then he looked quickly around to see if anyone was 320 | watching. They weren't. He looked back at the snake and winked, too. 321 | 322 | The snake jerked its head toward Uncle Vernon and Dudley, then raised 323 | its eyes to the ceiling. It gave Harry a look that said quite plainly: 324 | 325 | "I get that all the time. 326 | 327 | "I know," Harry murmured through the glass, though he wasn't sure the 328 | snake could hear him. "It must be really annoying." 329 | 330 | The snake nodded vigorously. 331 | 332 | "Where do you come from, anyway?" Harry asked. 333 | 334 | The snake jabbed its tail at a little sign next to the glass. Harry 335 | peered at it. 336 | 337 | Boa Constrictor, Brazil. 338 | 339 | "Was it nice there?" 340 | 341 | The boa constrictor jabbed its tail at the sign again and Harry read on: 342 | This specimen was bred in the zoo. "Oh, I see -- so you've never been to 343 | Brazil?" 344 | 345 | As the snake shook its head, a deafening shout behind Harry made both of 346 | them jump. 347 | 348 | "DUDLEY! MR. DURSLEY! COME AND LOOK AT THIS SNAKE! YOU WON'T BELIEVE 349 | WHAT IT'S DOING!" 350 | 351 | Dudley came waddling toward them as fast as he could. 352 | 353 | "Out of the way, you," he said, punching Harry in the ribs. Caught by 354 | surprise, Harry fell hard on the concrete floor. What came next happened 355 | so fast no one saw how it happened -- one second, Piers and Dudley were 356 | leaning right up close to the glass, the next, they had leapt back with 357 | howls of horror. 358 | 359 | Harry sat up and gasped; the glass front of the boa constrictor's tank 360 | had vanished. The great snake was uncoiling itself rapidly, slithering 361 | out onto the floor. People throughout the reptile house screamed and 362 | started running for the exits. 363 | 364 | As the snake slid swiftly past him, Harry could have sworn a low, 365 | hissing voice said, "Brazil, here I come.... Thanksss, amigo." 366 | 367 | The keeper of the reptile house was in shock. 368 | 369 | "But the glass," he kept saying, "where did the glass go?" 370 | 371 | The zoo director himself made Aunt Petunia a cup of strong, sweet tea 372 | while he apologized over and over again. Piers and Dudley could only 373 | gibber. As far as Harry had seen, the snake hadn't done anything except 374 | snap playfully at their heels as it passed, but by the time they were 375 | all back in Uncle Vernon's car, Dudley was telling them how it had 376 | nearly bitten off his leg, while Piers was swearing it had tried to 377 | squeeze him to death. But worst of all, for Harry at least, was Piers 378 | calming down enough to say, "Harry was talking to it, weren't you, 379 | Harry?" 380 | 381 | Uncle Vernon waited until Piers was safely out of the house before 382 | starting on Harry. He was so angry he could hardly speak. He managed to 383 | say, "Go -- cupboard -- stay -- no meals," before he collapsed into a 384 | chair, and Aunt Petunia had to run and get him a large brandy. 385 | 386 | Harry lay in his dark cupboard much later, wishing he had a watch. He 387 | didn't know what time it was and he couldn't be sure the Dursleys were 388 | asleep yet. Until they were, he couldn't risk sneaking to the kitchen 389 | for some food. 390 | 391 | He'd lived with the Dursleys almost ten years, ten miserable years, as 392 | long as he could remember, ever since he'd been a baby and his parents 393 | had died in that car crash. He couldn't remember being in the car when 394 | his parents had died. Sometimes, when he strained his memory during long 395 | hours in his cupboard, he came up with a strange vision: a blinding 396 | flash of green light and a burn- ing pain on his forehead. This, he 397 | supposed, was the crash, though he couldn't imagine where all the green 398 | light came from. He couldn't remember his parents at all. His aunt and 399 | uncle never spoke about them, and of course he was forbidden to ask 400 | questions. There were no photographs of them in the house. 401 | 402 | When he had been younger, Harry had dreamed and dreamed of some unknown 403 | relation coming to take him away, but it had never happened; the 404 | Dursleys were his only family. Yet sometimes he thought (or maybe hoped) 405 | that strangers in the street seemed to know him. Very strange strangers 406 | they were, too. A tiny man in a violet top hat had bowed to him once 407 | while out shopping with Aunt Petunia and Dudley. After asking Harry 408 | furiously if he knew the man, Aunt Petunia had rushed them out of the 409 | shop without buying anything. A wild-looking old woman dressed all in 410 | green had waved merrily at him once on a bus. A bald man in a very long 411 | purple coat had actually shaken his hand in the street the other day and 412 | then walked away without a word. The weirdest thing about all these 413 | people was the way they seemed to vanish the second Harry tried to get a 414 | closer look. 415 | 416 | At school, Harry had no one. Everybody knew that Dudley's gang hated 417 | that odd Harry Potter in his baggy old clothes and broken glasses, and 418 | nobody liked to disagree with Dudley's gang. 419 | -------------------------------------------------------------------------------- /data/chapter/4.txt: -------------------------------------------------------------------------------- 1 | THE KEEPER OF THE KEYS 2 | 3 | BOOM. They knocked again. Dudley jerked awake. "Where's the cannon?" he 4 | said stupidly. 5 | 6 | There was a crash behind them and Uncle Vernon came skidding into the 7 | room. He was holding a rifle in his hands -- now they knew what had been 8 | in the long, thin package he had brought with them. 9 | 10 | "Who's there?" he shouted. "I warn you -- I'm armed!" 11 | 12 | There was a pause. Then -- 13 | 14 | SMASH! 15 | 16 | The door was hit with such force that it swung clean off its hinges and 17 | with a deafening crash landed flat on the floor. 18 | 19 | A giant of a man was standing in the doorway. His face was almost 20 | completely hidden by a long, shaggy mane of hair and a wild, tangled 21 | beard, but you could make out his eyes, glinting like black beetles 22 | under all the hair. 23 | 24 | The giant squeezed his way into the hut, stooping so that his head just 25 | brushed the ceiling. He bent down, picked up the door, and fitted it 26 | easily back into its frame. The noise of the storm outside dropped a 27 | little. He turned to look at them all. 28 | 29 | "Couldn't make us a cup o' tea, could yeh? It's not been an easy 30 | journey..." 31 | 32 | He strode over to the sofa where Dudley sat frozen with fear. 33 | 34 | "Budge up, yeh great lump," said the stranger. 35 | 36 | Dudley squeaked and ran to hide behind his mother, who was crouching, 37 | terrified, behind Uncle Vernon. 38 | 39 | "An' here's Harry!" said the giant. 40 | 41 | Harry looked up into the fierce, wild, shadowy face and saw that the 42 | beetle eyes were crinkled in a smile. 43 | 44 | "Las' time I saw you, you was only a baby," said the giant. "Yeh look a 45 | lot like yet dad, but yeh've got yet mom's eyes." 46 | 47 | Uncle Vernon made a funny rasping noise. 48 | 49 | I demand that you leave at once, sit!" he said. "You are breaking and 50 | entering!" 51 | 52 | "Ah, shut up, Dursley, yeh great prune," said the giant; he reached over 53 | the back of the sofa, jerked the gun out of Uncle Vernon's hands, bent 54 | it into a knot as easily as if it had been made of rubber, and threw it 55 | into a corner of the room. 56 | 57 | Uncle Vernon made another funny noise, like a mouse being trodden on. 58 | 59 | "Anyway -- Harry," said the giant, turning his back on the Dursleys, "a 60 | very happy birthday to yeh. Got summat fer yeh here -- I mighta sat on 61 | it at some point, but it'll taste all right." 62 | 63 | From an inside pocket of his black overcoat he pulled a slightly 64 | squashed box. Harry opened it with trembling fingers. Inside was a 65 | large, sticky chocolate cake with Happy Birthday Harry written on it in 66 | green icing. 67 | 68 | Harry looked up at the giant. He meant to say thank you, but the words 69 | got lost on the way to his mouth, and what he said instead was, "Who are 70 | you?" 71 | 72 | The giant chuckled. 73 | 74 | "True, I haven't introduced meself. Rubeus Hagrid, Keeper of Keys and 75 | Grounds at Hogwarts." 76 | 77 | He held out an enormous hand and shook Harry's whole arm. 78 | 79 | "What about that tea then, eh?" he said, rubbing his hands together. 80 | "I'd not say no ter summat stronger if yeh've got it, mind." 81 | 82 | His eyes fell on the empty grate with the shriveled chip bags in it and 83 | he snorted. He bent down over the fireplace; they couldn't see what he 84 | was doing but when he drew back a second later, there was a roaring fire 85 | there. It filled the whole damp hut with flickering light and Harry felt 86 | the warmth wash over him as though he'd sunk into a hot bath. 87 | 88 | The giant sat back down on the sofa, which sagged under his weight, and 89 | began taking all sorts of things out of the pockets of his coat: a 90 | copper kettle, a squashy package of sausages, a poker, a teapot, several 91 | chipped mugs, and a bottle of some amber liquid that he took a swig from 92 | before starting to make tea. Soon the hut was full of the sound and 93 | smell of sizzling sausage. Nobody said a thing while the giant was 94 | working, but as he slid the first six fat, juicy, slightly burnt 95 | sausages from the poker, Dudley fidgeted a little. Uncle Vernon said 96 | sharply, "Don't touch anything he gives you, Dudley." 97 | 98 | The giant chuckled darkly. 99 | 100 | "Yet great puddin' of a son don' need fattenin' anymore, Dursley, don' 101 | worry." 102 | 103 | He passed the sausages to Harry, who was so hungry he had never tasted 104 | anything so wonderful, but he still couldn't take his eyes off the 105 | giant. Finally, as nobody seemed about to explain anything, he said, 106 | "I'm sorry, but I still don't really know who you are." 107 | 108 | The giant took a gulp of tea and wiped his mouth with the back of his 109 | hand. 110 | 111 | "Call me Hagrid," he said, "everyone does. An' like I told yeh, I'm 112 | Keeper of Keys at Hogwarts -- yeh'll know all about Hogwarts, o' course. 113 | 114 | "Er -- no," said Harry. 115 | 116 | Hagrid looked shocked. 117 | 118 | "Sorry," Harry said quickly. 119 | 120 | "Sony?" barked Hagrid, turning to stare at the Dursleys, who shrank back 121 | into the shadows. "It' s them as should be sorry! I knew yeh weren't 122 | gettin' yer letters but I never thought yeh wouldn't even know abou' 123 | Hogwarts, fer cryin' out loud! Did yeh never wonder where yet parents 124 | learned it all?" 125 | 126 | "All what?" asked Harry. 127 | 128 | "ALL WHAT?" Hagrid thundered. "Now wait jus' one second!" 129 | 130 | He had leapt to his feet. In his anger he seemed to fill the whole hut. 131 | The Dursleys were cowering against the wall. 132 | 133 | "Do you mean ter tell me," he growled at the Dursleys, "that this boy -- 134 | this boy! -- knows nothin' abou' -- about ANYTHING?" 135 | 136 | Harry thought this was going a bit far. He had been to school, after 137 | all, and his marks weren't bad. 138 | 139 | "I know some things," he said. "I can, you know, do math and stuff." But 140 | Hagrid simply waved his hand and said, "About our world, I mean. Your 141 | world. My world. Yer parents' world." 142 | 143 | "What world?" 144 | 145 | Hagrid looked as if he was about to explode. 146 | 147 | "DURSLEY!" he boomed. 148 | 149 | Uncle Vernon, who had gone very pale, whispered something that sounded 150 | like "Mimblewimble." Hagrid stared wildly at Harry. 151 | 152 | "But yeh must know about yet mom and dad," he said. "I mean, they're 153 | famous. You're famous." 154 | 155 | "What? My -- my mom and dad weren't famous, were they?" 156 | 157 | "Yeh don' know... yeh don' know..." Hagrid ran his fingers through his 158 | hair, fixing Harry with a bewildered stare. 159 | 160 | "Yeh don' know what yeh are?" he said finally. 161 | 162 | Uncle Vernon suddenly found his voice. 163 | 164 | "Stop!" he commanded. "Stop right there, sit! I forbid you to tell the 165 | boy anything!" 166 | 167 | A braver man than Vernon Dursley would have quailed under the furious 168 | look Hagrid now gave him; when Hagrid spoke, his every syllable trembled 169 | with rage. 170 | 171 | "You never told him? Never told him what was in the letter Dumbledore 172 | left fer him? I was there! I saw Dumbledore leave it, Dursley! An' 173 | you've kept it from him all these years?" 174 | 175 | "Kept what from me?" said Harry eagerly. 176 | 177 | "STOP! I FORBID YOU!" yelled Uncle Vernon in panic. 178 | 179 | Aunt Petunia gave a gasp of horror. 180 | 181 | "Ah, go boil yet heads, both of yeh," said Hagrid. "Harry -- yet a 182 | wizard." 183 | 184 | There was silence inside the hut. Only the sea and the whistling wind 185 | could be heard. 186 | 187 | "-- a what?" gasped Harry. 188 | 189 | "A wizard, o' course," said Hagrid, sitting back down on the sofa, which 190 | groaned and sank even lower, "an' a thumpin' good'un, I'd say, once 191 | yeh've been trained up a bit. With a mum an' dad like yours, what else 192 | would yeh be? An' I reckon it's abou' time yeh read yer letter." 193 | 194 | Harry stretched out his hand at last to take the yellowish envelope, 195 | addressed in emerald green to Mr. H. Potter, The Floor, Hut-on-the-Rock, 196 | The Sea. He pulled out the letter and read: 197 | 198 | HOGWARTS SCHOOL of WITCHCRAFT and WIZARDRY 199 | 200 | Headmaster: ALBUS DUMBLEDORE 201 | 202 | (Order of Merlin, First Class, Grand Sorc., Chf. Warlock, Supreme 203 | Mugwump, International Confed. of Wizards) 204 | 205 | Dear Mr. Potter, 206 | 207 | We are pleased to inform you that you have been accepted at Hogwarts 208 | School of Witchcraft and Wizardry. Please find enclosed a list of all 209 | necessary books and equipment. 210 | 211 | Term begins on September 1. We await your owl by no later than July 31. 212 | Yours sincerely, 213 | 214 | Minerva McGonagall, 215 | 216 | Deputy Headmistress 217 | 218 | Questions exploded inside Harry's head like fireworks and he couldn't 219 | decide which to ask first. After a few minutes he stammered, "What does 220 | it mean, they await my owl?" 221 | 222 | "Gallopin' Gorgons, that reminds me," said Hagrid, clapping a hand to 223 | his forehead with enough force to knock over a cart horse, and from yet 224 | another pocket inside his overcoat he pulled an owl -- a real, live, 225 | rather ruffled-looking owl -- a long quill, and a roll of parchment. 226 | With his tongue between his teeth he scribbled a note that Harry could 227 | read upside down: 228 | 229 | Dear Professor Dumbledore, 230 | 231 | Given Harry his letter. 232 | 233 | Taking him to buy his things tomorrow. 234 | 235 | Weather's horrible. Hope you're Well. 236 | 237 | Hagrid 238 | 239 | Hagrid rolled up the note, gave it to the owl, which clamped it in its 240 | beak, went to the door, and threw the owl out into the storm. Then he 241 | came back and sat down as though this was as normal as talking on the 242 | telephone. 243 | 244 | Harry realized his mouth was open and closed it quickly. 245 | 246 | "Where was I?" said Hagrid, but at that moment, Uncle Vernon, still 247 | ashen-faced but looking very angry, moved into the firelight. 248 | 249 | "He's not going," he said. 250 | 251 | Hagrid grunted. 252 | 253 | "I'd like ter see a great Muggle like you stop him," he said. 254 | 255 | "A what?" said Harry, interested. 256 | 257 | "A Muggle," said Hagrid, "it's what we call nonmagic folk like thern. 258 | An' it's your bad luck you grew up in a family o' the biggest Muggles I 259 | ever laid eyes on." 260 | 261 | "We swore when we took him in we'd put a stop to that rubbish," said 262 | Uncle Vernon, "swore we'd stamp it out of him! Wizard indeed!" 263 | 264 | "You knew?" said Harry. "You knew I'm a -- a wizard?" 265 | 266 | "Knew!" shrieked Aunt Petunia suddenly. "Knew! Of course we knew! How 267 | could you not be, my dratted sister being what she was? Oh, she got a 268 | letter just like that and disappeared off to that-that school-and came 269 | home every vacation with her pockets full of frog spawn, turning teacups 270 | into rats. I was the only one who saw her for what she was -- a freak! 271 | But for my mother and father, oh no, it was Lily this and Lily that, 272 | they were proud of having a witch in the family!" 273 | 274 | She stopped to draw a deep breath and then went ranting on. It seemed 275 | she had been wanting to say all this for years. 276 | 277 | "Then she met that Potter at school and they left and got married and 278 | had you, and of course I knew you'd be just the same, just as strange, 279 | just as -- as -- abnormal -- and then, if you please, she went and got 280 | herself blown up and we got landed with you!" 281 | 282 | Harry had gone very white. As soon as he found his voice he said, "Blown 283 | up? You told me they died in a car crash!" 284 | 285 | "CAR CRASH!" roared Hagrid, jumping up so angrily that the Dursleys 286 | scuttled back to their corner. "How could a car crash kill Lily an' 287 | James Potter? It's an outrage! A scandal! Harry Potter not knowin' his 288 | own story when every kid in our world knows his name!" "But why? What 289 | happened?" Harry asked urgently. 290 | 291 | The anger faded from Hagrid's face. He looked suddenly anxious. 292 | 293 | "I never expected this," he said, in a low, worried voice. "I had no 294 | idea, when Dumbledore told me there might be trouble gettin' hold of 295 | yeh, how much yeh didn't know. Ah, Harry, I don' know if I'm the right 296 | person ter tell yeh -- but someone 3 s gotta -- yeh can't go off ter 297 | Hogwarts not knowin'." 298 | 299 | He threw a dirty look at the Dursleys. 300 | 301 | "Well, it's best yeh know as much as I can tell yeh -- mind, I can't 302 | tell yeh everythin', it's a great myst'ry, parts of it...." 303 | 304 | He sat down, stared into the fire for a few seconds, and then said, "It 305 | begins, I suppose, with -- with a person called -- but it's incredible 306 | yeh don't know his name, everyone in our world knows --" 307 | 308 | "Who? " 309 | 310 | "Well -- I don' like sayin' the name if I can help it. No one does." 311 | 312 | "Why not?" 313 | 314 | "Gulpin' gargoyles, Harry, people are still scared. Blimey, this is 315 | difficult. See, there was this wizard who went... bad. As bad as you 316 | could go. Worse. Worse than worse. His name was..." 317 | 318 | Hagrid gulped, but no words came out. 319 | 320 | "Could you write it down?" Harry suggested. 321 | 322 | "Nah -can't spell it. All right -- Voldemort. " Hagrid shuddered. "Don' 323 | make me say it again. Anyway, this -- this wizard, about twenty years 324 | ago now, started lookin' fer followers. Got 'em, too -- some were 325 | afraid, some just wanted a bit o' his power, 'cause he was gettin' 326 | himself power, all right. Dark days, Harry. Didn't know who ter trust, 327 | didn't dare get friendly with strange wizards or witches... terrible 328 | things happened. He was takin' over. 'Course, some stood up to him -- 329 | an' he killed 'em. Horribly. One o' the only safe places left was 330 | Hogwarts. Reckon Dumbledore's the only one You-Know-Who was afraid of. 331 | Didn't dare try takin' the school, not jus' then, anyway. 332 | 333 | "Now, yer mum an' dad were as good a witch an' wizard as I ever knew. 334 | Head boy an' girl at Hogwarts in their day! Suppose the myst'ry is why 335 | You-Know-Who never tried to get 'em on his side before... probably knew 336 | they were too close ter Dumbledore ter want anythin' ter do with the 337 | Dark Side. 338 | 339 | "Maybe he thought he could persuade 'em... maybe he just wanted 'em 340 | outta the way. All anyone knows is, he turned up in the village where 341 | you was all living, on Halloween ten years ago. You was just a year old. 342 | He came ter yer house an' -- an' --" 343 | 344 | Hagrid suddenly pulled out a very dirty, spotted handkerchief and blew 345 | his nose with a sound like a foghorn. 346 | 347 | "Sorry," he said. "But it's that sad -- knew yer mum an' dad, an' nicer 348 | people yeh couldn't find -- anyway..." 349 | 350 | "You-Know-Who killed 'em. An' then -- an' this is the real myst'ry of 351 | the thing -- he tried to kill you, too. Wanted ter make a clean job of 352 | it, I suppose, or maybe he just liked killin' by then. But he couldn't 353 | do it. Never wondered how you got that mark on yer forehead? That was no 354 | ordinary cut. That's what yeh get when a Powerful, evil curse touches 355 | yeh -- took care of yer mum an' dad an' yer house, even -- but it didn't 356 | work on you, an' that's why yer famous, Harry. No one ever lived after 357 | he decided ter kill 'em, no one except you, an' he'd killed some o' the 358 | best witches an' wizards of the age -- the McKinnons, the Bones, the 359 | Prewetts -- an' you was only a baby, an' you lived." 360 | 361 | Something very painful was going on in Harry's mind. As Hagrid's story 362 | came to a close, he saw again the blinding flash of green light, more 363 | clearly than he had ever remembered it before -- and he remembered 364 | something else, for the first time in his life: a high, cold, cruel 365 | laugh. 366 | 367 | Hagrid was watching him sadly. 368 | 369 | "Took yeh from the ruined house myself, on Dumbledore's orders. Brought 370 | yeh ter this lot..." 371 | 372 | "Load of old tosh," said Uncle Vernon. Harry jumped; he had almost 373 | forgotten that the Dursleys were there. Uncle Vernon certainly seemed to 374 | have got back his courage. He was glaring at Hagrid and his fists were 375 | clenched. 376 | 377 | "Now, you listen here, boy," he snarled, "I accept there's something 378 | strange about you, probably nothing a good beating wouldn't have cured 379 | -- and as for all this about your parents, well, they were weirdos, no 380 | denying it, and the world's better off without them in my opinion -- 381 | asked for all they got, getting mixed up with these wizarding types -- 382 | just what I expected, always knew they'd come to a sticky end --" 383 | 384 | But at that moment, Hagrid leapt from the sofa and drew a battered pink 385 | umbrella from inside his coat. Pointing this at Uncle Vernon like a 386 | sword, he said, "I'm warning you, Dursley -I'm warning you -- one more 387 | word... " 388 | 389 | In danger of being speared on the end of an umbrella by a bearded giant, 390 | Uncle Vernon's courage failed again; he flattened himself against the 391 | wall and fell silent. 392 | 393 | "That's better," said Hagrid, breathing heavily and sitting back down on 394 | the sofa, which this time sagged right down to the floor. 395 | 396 | Harry, meanwhile, still had questions to ask, hundreds of them. 397 | 398 | "But what happened to Vol--, sorry -- I mean, You-Know-Who?" 399 | 400 | "Good question, Harry. Disappeared. Vanished. Same night he tried ter 401 | kill you. Makes yeh even more famous. That's the biggest myst'ry, see... 402 | he was gettin' more an' more powerful -- why'd he go? 403 | 404 | "Some say he died. Codswallop, in my opinion. Dunno if he had enough 405 | human left in him to die. Some say he's still out there, bidin' his 406 | time, like, but I don' believe it. People who was on his side came back 407 | ter ours. Some of 'em came outta kinda trances. Don~ reckon they 408 | could've done if he was comin' back. 409 | 410 | "Most of us reckon he's still out there somewhere but lost his powers. 411 | Too weak to carry on. 'Cause somethin' about you finished him, Harry. 412 | There was somethin' goin' on that night he hadn't counted on -- I dunno 413 | what it was, no one does -- but somethin' about you stumped him, all 414 | right." 415 | 416 | Hagrid looked at Harry with warmth and respect blazing in his eyes, but 417 | Harry, instead of feeling pleased and proud, felt quite sure there had 418 | been a horrible mistake. A wizard? Him? How could he possibly be? He'd 419 | spent his life being clouted by Dudley, and bullied by Aunt Petunia and 420 | Uncle Vernon; if he was really a wizard, why hadn't they been turned 421 | into warty toads every time they'd tried to lock him in his cupboard? If 422 | he'd once defeated the greatest sorcerer in the world, how come Dudley 423 | had always been able to kick him around like a football? 424 | 425 | "Hagrid," he said quietly, "I think you must have made a mistake. I 426 | don't think I can be a wizard." 427 | 428 | To his surprise, Hagrid chuckled. 429 | 430 | "Not a wizard, eh? Never made things happen when you was scared or 431 | angry?" 432 | 433 | Harry looked into the fire. Now he came to think about it... every odd 434 | thing that had ever made his aunt and uncle furious with him had 435 | happened when he, Harry, had been upset or angry... chased by Dudley's 436 | gang, he had somehow found himself out of their reach... dreading going 437 | to school with that ridiculous haircut, he'd managed to make it grow 438 | back... and the very last time Dudley had hit him, hadn't he got his 439 | revenge, without even realizing he was doing it? Hadn't he set a boa 440 | constrictor on him? 441 | 442 | Harry looked back at Hagrid, smiling, and saw that Hagrid was positively 443 | beaming at him. 444 | 445 | "See?" said Hagrid. "Harry Potter, not a wizard -- you wait, you'll be 446 | right famous at Hogwarts." 447 | 448 | But Uncle Vernon wasn't going to give in without a fight. 449 | 450 | "Haven't I told you he's not going?" he hissed. "He's going to Stonewall 451 | High and he'll be grateful for it. I've read those letters and he needs 452 | all sorts of rubbish -- spell books and wands and --" 453 | 454 | "If he wants ter go, a great Muggle like you won't stop him," growled 455 | Hagrid. "Stop Lily an' James Potter' s son goin' ter Hogwarts! Yer mad. 456 | His name's been down ever since he was born. He's off ter the finest 457 | school of witchcraft and wizardry in the world. Seven years there and he 458 | won't know himself. He'll be with youngsters of his own sort, fer a 459 | change, an' he'll be under the greatest headmaster Hogwarts ever had 460 | Albus Dumbled--" 461 | 462 | "I AM NOT PAYING FOR SOME CRACKPOT OLD FOOL To TEACH HIM MAGIC TRICKS!" 463 | yelled Uncle Vernon. 464 | 465 | But he had finally gone too far. Hagrid seized his umbrella and whirled 466 | it over his head, "NEVER," he thundered, "- INSULT- ALBUS- DUMBLEDORE- 467 | IN- FRONT- OF- ME!" 468 | 469 | He brought the umbrella swishing down through the air to point at Dudley 470 | -- there was a flash of violet light, a sound like a firecracker, a 471 | sharp squeal, and the next second, Dudley was dancing on the spot with 472 | his hands clasped over his fat bottom, howling in pain. When he turned 473 | his back on them, Harry saw a curly pig's tail poking through a hole in 474 | his trousers. 475 | 476 | Uncle Vernon roared. Pulling Aunt Petunia and Dudley into the other 477 | room, he cast one last terrified look at Hagrid and slammed the door 478 | behind them. 479 | 480 | Hagrid looked down at his umbrella and stroked his beard. 481 | 482 | "Shouldn'ta lost me temper," he said ruefully, "but it didn't work 483 | anyway. Meant ter turn him into a pig, but I suppose he was so much like 484 | a pig anyway there wasn't much left ter do." 485 | 486 | He cast a sideways look at Harry under his bushy eyebrows. 487 | 488 | "Be grateful if yeh didn't mention that ter anyone at Hogwarts," he 489 | said. "I'm -- er -- not supposed ter do magic, strictly speakin'. I was 490 | allowed ter do a bit ter follow yeh an' get yer letters to yeh an' stuff 491 | -- one o' the reasons I was so keen ter take on the job 492 | 493 | "Why aren't you supposed to do magic?" asked Harry. 494 | 495 | "Oh, well -- I was at Hogwarts meself but I -- er -- got expelled, ter 496 | tell yeh the truth. In me third year. They snapped me wand in half an' 497 | everything. But Dumbledore let me stay on as gamekeeper. Great man, 498 | Dumbledore." "Why were you expelled?" 499 | 500 | "It's gettin' late and we've got lots ter do tomorrow," said Hagrid 501 | loudly. "Gotta get up ter town, get all yer books an' that." 502 | 503 | He took off his thick black coat and threw it to Harry. 504 | 505 | "You can kip under that," he said. "Don' mind if it wriggles a bit, I 506 | think I still got a couple o' dormice in one o' the pockets." 507 | --------------------------------------------------------------------------------