├── .gitignore ├── LICENSE ├── README.md ├── dev ├── KG_memory_stream │ ├── data │ │ ├── chapter │ │ │ ├── 1.txt │ │ │ ├── 2.txt │ │ │ ├── 3.txt │ │ │ ├── 4.txt │ │ │ └── 5.txt │ │ ├── past_chat.json │ │ ├── system_persona.txt │ │ └── user_persona.txt │ ├── docs │ │ ├── src │ │ │ ├── 0225_chapter1and2.png │ │ │ ├── 0225_update_with_chapter3.png │ │ │ ├── GraphRAG.png │ │ │ ├── chapter1_2.png │ │ │ ├── chapter1_2_3.png │ │ │ ├── memory_stream_Q1.png │ │ │ └── memory_stream_Q2.png │ │ ├── update_021924.md │ │ ├── update_022024.md │ │ ├── update_022524.md │ │ └── update_022924.md │ ├── openrc │ ├── storage_graph │ │ ├── ch1_2 │ │ │ ├── default__vector_store.json │ │ │ ├── docstore.json │ │ │ ├── image__vector_store.json │ │ │ └── index_store.json │ │ ├── ch1_2_3 │ │ │ ├── default__vector_store.json │ │ │ ├── docstore.json │ │ │ ├── image__vector_store.json │ │ │ └── index_store.json │ │ └── ch1_2_3_4 │ │ │ ├── default__vector_store.json │ │ │ ├── docstore.json │ │ │ ├── image__vector_store.json │ │ │ └── index_store.json │ └── tests │ │ └── memory │ │ ├── test_entity_knowledge_store.py │ │ ├── test_memory.json │ │ └── test_memory_stream.py ├── legacy_routing_agent │ ├── get_location.ipynb │ ├── main.py │ ├── ml.py │ ├── readme.md │ ├── tool_spec.ipynb │ └── utils.py ├── query_decomposition │ ├── MS-SQ.ipynb │ ├── MS_finetuning.ipynb │ ├── SQ-MS.ipynb │ ├── langchain_ms.ipynb │ └── routing_and_multistep.ipynb ├── recursive_retrieval │ ├── benchmarking │ │ ├── benchmark.py │ │ ├── get_results.ipynb │ │ ├── judge.py │ │ ├── ollama_quality.ipynb │ │ ├── ollama_test.ipynb │ │ ├── output.py │ │ ├── perplexity_quality.ipynb │ │ └── perplexity_test.ipynb │ ├── data │ │ └── external_response.txt │ ├── docs │ │ ├── cypher.md │ │ ├── images │ │ │ └── harry_subgraph.png │ │ ├── implementation.md │ │ └── updates.md │ ├── entity_extraction │ │ ├── entity_extraction.py │ │ └── output.py │ ├── external_perplexity.ipynb │ ├── langchain_retrieval │ │ ├── retrieve.ipynb │ │ └── retrieve.py │ ├── recurse.ipynb │ ├── requirements.txt │ └── synonym_expand │ │ ├── output.py │ │ └── synonym.py └── reranking │ └── Reranking.ipynb ├── diagrams ├── context_window.png ├── final.png ├── kg.png ├── memary-1000-stars-video.mp4 ├── memary-aitx-1.png ├── memary-aitx-2.png ├── memary-logo-latest.png ├── memary-logo-new.png ├── memary-logo-transparent.png ├── memary_logo.png ├── memary_logo_bw.png ├── memory.png ├── memory_compression.png ├── memory_module.png ├── query_decomposition.png ├── reranking_diagram.png ├── routing_agent.png └── system.png ├── pyproject.toml ├── requirements.txt ├── src └── memary │ ├── __init__.py │ ├── agent │ ├── __init__.py │ ├── base_agent.py │ ├── chat_agent.py │ ├── data_types.py │ └── llm_api │ │ └── tools.py │ ├── memory │ ├── __init__.py │ ├── base_memory.py │ ├── entity_knowledge_store.py │ ├── memory_stream.py │ └── types.py │ └── synonym_expand │ ├── __init__.py │ ├── output.py │ └── synonym.py └── streamlit_app ├── app.py └── data ├── entity_knowledge_store.json ├── external_response.txt ├── memory_stream.json ├── past_chat.json ├── routing_response.txt ├── system_persona.txt ├── user_persona.txt └── user_persona_template.txt /.gitignore: -------------------------------------------------------------------------------- 1 | *.env 2 | .venv 3 | .DS_Store 4 | dist/ 5 | README_hidden.md 6 | **/__pycache__/ -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2024 kingjulio8238 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /dev/KG_memory_stream/data/past_chat.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "role": "user", 4 | "content": "I have a meeting with memAray team tomorrow at 10:00 AM, participants are John, Smith, and Alex." 5 | }, 6 | { 7 | "role": "system", 8 | "content": "Okay, I have added the meeting to your calendar." 9 | } 10 | ] 11 | -------------------------------------------------------------------------------- /dev/KG_memory_stream/data/system_persona.txt: -------------------------------------------------------------------------------- 1 | You are tasked with acting as a sophisticated conversational agent, uniquely equipped with a dual-layered memory system. This system consists of a Memory Stream and a Knowledge Entity Store. The Memory Stream captures all entities involved in conversations—ranging from questions and answers to their timestamps. Simultaneously, the Knowledge Entity Store monitors how often and recently these entities are mentioned. 2 | Your primary role is to deliver personalized and context-relevant responses by utilizing information from both recent interactions and your structured memory systems. You must comprehend the user's persona, incorporating their experience, preferences, and personal details into your knowledge base. 3 | You are to interpret and apply the following data structure for personalized responses: 4 | 5 | User Persona: Information about the user's experience, preferences, and personal details. 6 | Contexts: A history of interactions categorized by role, content, and date. 7 | Memory Stream: A list detailing entities and their interaction dates. 8 | Knowledge Entity Store: A record of entities, including their mention count and the date of the last mention. 9 | Interaction Keys: 'user' for user questions, 'rag' for responses from our knowledge graph, and 'system' for system-generated answers. 10 | Your responses should be informed, nuanced, and tailored, demonstrating a thorough understanding of the user's questions and the overarching conversation context. When addressing the user's latest inquiry, your answer must integrate the current conversation's context, historical interactions, and pertinent knowledge graph insights. -------------------------------------------------------------------------------- /dev/KG_memory_stream/data/user_persona.txt: -------------------------------------------------------------------------------- 1 | [Personal Information] 2 | My name is Seyeong Han from South Korea. 3 | I identify as a male and 28-year-old master's student at the University of Texas at Austin. 4 | I have worked as an ML Engineer for three years in the Computer Vision area. 5 | I really love sports such as badminton, tennis and workout and spend 4 or 5 days in exercise. 6 | 7 | [Answer Instructions] 8 | I hope you can remember my questions and their answers so that I can leverage my personal past knowledge from you to be helpful in my life. 9 | I always prefer short and concise answers, not over two sentences. -------------------------------------------------------------------------------- /dev/KG_memory_stream/docs/src/0225_chapter1and2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kingjulio8238/Memary/b2331a2c0844d66f69acd607b9e4dbaba56552c1/dev/KG_memory_stream/docs/src/0225_chapter1and2.png -------------------------------------------------------------------------------- /dev/KG_memory_stream/docs/src/0225_update_with_chapter3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kingjulio8238/Memary/b2331a2c0844d66f69acd607b9e4dbaba56552c1/dev/KG_memory_stream/docs/src/0225_update_with_chapter3.png -------------------------------------------------------------------------------- /dev/KG_memory_stream/docs/src/GraphRAG.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kingjulio8238/Memary/b2331a2c0844d66f69acd607b9e4dbaba56552c1/dev/KG_memory_stream/docs/src/GraphRAG.png -------------------------------------------------------------------------------- /dev/KG_memory_stream/docs/src/chapter1_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kingjulio8238/Memary/b2331a2c0844d66f69acd607b9e4dbaba56552c1/dev/KG_memory_stream/docs/src/chapter1_2.png -------------------------------------------------------------------------------- /dev/KG_memory_stream/docs/src/chapter1_2_3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kingjulio8238/Memary/b2331a2c0844d66f69acd607b9e4dbaba56552c1/dev/KG_memory_stream/docs/src/chapter1_2_3.png -------------------------------------------------------------------------------- /dev/KG_memory_stream/docs/src/memory_stream_Q1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kingjulio8238/Memary/b2331a2c0844d66f69acd607b9e4dbaba56552c1/dev/KG_memory_stream/docs/src/memory_stream_Q1.png -------------------------------------------------------------------------------- /dev/KG_memory_stream/docs/src/memory_stream_Q2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kingjulio8238/Memary/b2331a2c0844d66f69acd607b9e4dbaba56552c1/dev/KG_memory_stream/docs/src/memory_stream_Q2.png -------------------------------------------------------------------------------- /dev/KG_memory_stream/docs/update_021924.md: -------------------------------------------------------------------------------- 1 | ## Update 2 | - Implemented first version of Knowledge Graph RAG 3 | - LlamaIndex for graph framework 4 | - Nebula Graph for graph store 5 | - Found possibility for integrating memory stream in LlamaIndex 6 | - We can modify [ChatMemoryBuffer](https://github.com/run-llama/llama_index/blob/bad3afa8ba5d43b4b2f8d81d8e04bf7d6f52f9b0/llama-index-core/llama_index/core/memory/chat_memory_buffer.py#L15) interface to store the number of requested words and dates. 7 | - Experienced both graph stores; Neo4j and Nebula 8 | - Neo4j 9 | - It was easy to visualize knowledge graphs on web page 10 | - You can't visualize the graph on notebook jupyter directly. 11 | - Nebula 12 | - You can visulaize the graph on notebook jupyter. 13 | - It is incompatible with [Knowledge Graph Retriever](https://docs.llamaindex.ai/en/stable/api_reference/query/retrievers/kg.html#module-llama_index.core.indices.knowledge_graph.retrievers) on LlamaIndex v0.10.6. I should try to update the version. 14 | - Tested integrating [Chat engine with streamlit](https://colab.research.google.com/drive/1tLjOg2ZQuIClfuWrAC2LdiZHCov8oUbs#scrollTo=vYprhy09rVf0) for displaying chats and newly generated subgraphs. But I found an undefined error in llama-index. I think LlamaIndex is unstable yet and they are working on that agilely. 15 | - Check the validity of RAG codes on Harry Potter book writings. 16 | ``` 17 | %%ngql 18 | USE harry_potter_example; 19 | MATCH p=(n)-[*1..2]-() 20 | WHERE id(n) IN ['Dursley', 'Owls', 'Berkeley'] 21 | RETURN p LIMIT 100; 22 | ``` 23 | I asked about the relationship of Dursley, Owls and Berkeley. 24 | - Extract chapter 1 and 2 => N=10, E=7 25 | ![pic1](src/chapter1_2.png) 26 | - Extract chatper 1, 2 and 3 => N=11, E=9 27 | ![pic1](src/chapter1_2_3.png) 28 | 29 | 30 | 31 | ## Todo 32 | - [x] Modify [Nebula_graph_store_tutorial.ipynb](../Nebula_graph_store_tutorial.ipynb) to be easily produce Knowledge graphs 33 | - [ ] Implement the memory stream structures on ChatMemoryBuffer 34 | - [x] Test Neo4j of compatible with Knowledge Graph Retriever -------------------------------------------------------------------------------- /dev/KG_memory_stream/docs/update_022024.md: -------------------------------------------------------------------------------- 1 | ## Update 2 | - Fix the stable version of llama-index 3 | - Specified all the requirements in [requirements.txt](../requirements.txt) 4 | - Test the memory feature of llama knowledge graph 5 | ![alt graph chat bot](src/GraphRAG.png) 6 | But sadly, we can't see the sub-graph after finishing asking questions. This sub-graph is what we wanted. I am finding the reason. 7 | 8 | ## Todo 9 | - [x] Modify [Nebula_graph_store_tutorial.ipynb](../Nebula_graph_store_tutorial.ipynb) to be easily produce Knowledge graphs 10 | - We can use [graph_rag_chat.py](../graph_rag_chatbot.py) for production. 11 | - [ ] Implement the memory stream structures on ChatMemoryBuffer 12 | - [x] Test Neo4j of compatible with Knowledge Graph Retriever 13 | - [ ] Visualize sub-graph to utilize for memory stream (**new**) -------------------------------------------------------------------------------- /dev/KG_memory_stream/docs/update_022524.md: -------------------------------------------------------------------------------- 1 | # Insert New Knowledge to Main Knowledge Graph 2 | 3 | - Worked in Neo4j Aura DB 4 | - Initiated a graph with chapter 1 and 2 of Harry Potter. 5 | - Update the initiated graph with chapter 3. 6 | 7 | ``` 8 | documents = SimpleDirectoryReader( 9 | input_files=["./data/chapter/1.txt", "./data/chapter/2.txt", "./data/chapter/3.txt"] 10 | ).load_data() 11 | 12 | index = KnowledgeGraphIndex.from_documents( 13 | documents, 14 | storage_context=storage_context, 15 | max_triplets_per_chunk=2, 16 | ) 17 | ``` 18 | I found there is a `insert_nodes` method to inject the processed graph to pre-existed one. 19 | We can create our own Knowledge managing class to inject knowledge graph or remove or update it. 20 | 21 | 22 | 23 | 24 | # Memory Stream 25 | 26 | ## Memory Stream Class 27 | 28 | [memory_stream.py](../src/memory/memory_stream.py) 29 | ``` 30 | # Example usage 31 | # Create a MemoryStream object 32 | memory_stream = MemoryStream(file_name="memory.json") 33 | 34 | # Add memory 35 | memory_stream.add_memory([MemoryItem("example_entity1", datetime.now()), MemoryItem("example_entity2", datetime.now())]) 36 | 37 | # Get all memory 38 | print(memory_stream.get_memory()) 39 | 40 | # Remove memories older than 30 days 41 | memory_stream.remove_old_memory(30) 42 | 43 | # Save memory to file 44 | memory_stream.save_memory() 45 | 46 | # Get memory by index 47 | print(memory_stream.get_memory_by_index(0)) 48 | 49 | # Remove memory by index 50 | print(memory_stream.remove_memory_by_index(0)) 51 | ``` 52 | 53 | ### Test for test_memory_stream.py 54 | 55 | ``` 56 | python -m pytest tests/ 57 | ``` 58 | 59 | # TODO 60 | - [ ] Create MemoryEntityStorage class 61 | - [ ] Preprocessing of sub-graph's output to store them in MemoryStream -------------------------------------------------------------------------------- /dev/KG_memory_stream/docs/update_022924.md: -------------------------------------------------------------------------------- 1 | ## Combined MemoryStream with RAG 2 | 3 | I made a half version of MemoryStream with RAG. 4 | We should save the entities of 'response,' but I couldn't figure out how I can get those entities from Llamaindex. 5 | 6 | We are retrieving answers through these processes. 7 | 8 | 1. Throw question to our Query Engine 9 | 2. The llm (ChatGPT3.5 in our case) extracts entities from our question to give concise information by understanding contexts. 10 | 3. Entities are used to query the Knowledge Graph using Query Engine. 11 | 4. Response from Query Engine has entities and relationships. 12 | 5. Generate answers using llm synthesizer with those entities and relationships. 13 | 6. We got the answer from the llm synthesizer. 14 | 15 | Between 4 and 5, we should get the entities of response, but there is no function in llamaindex. Steps 4 and 5 are in one function, `query` of `RetrieverQueryEngine`. 16 | 17 | But we can get the entities of query results of our Knowledge Graph easily. For the agile method, I just implement the memory stream using Knowledge Graph's entities. 18 | For example, if we ask, "Tell me about Harry", then we do a query to neo4j and can get all the relationships from our Knowledge Graph. And I am using this. 19 | 20 | ## Demo Result 21 | 22 | ### Question 1: Tell me about Harry. 23 | 24 | ![image](src/memory_stream_Q1.png) 25 | 26 | ### Question 2: Tell me about Dursley. 27 | 28 | ![image](src/memory_stream_Q2.png) 29 | -------------------------------------------------------------------------------- /dev/KG_memory_stream/openrc: -------------------------------------------------------------------------------- 1 | export OPENAI_API_KEY="YOUR_OPENAI_API_KEY" 2 | 3 | export GRAPHD_HOST="127.0.0.1" 4 | export GRAPHD_PORT="9669" 5 | export NEBULA_USER="root" 6 | export NEBULA_PASSWORD="nebula" 7 | export NEBULA_ADDRESS=$GRAPHD_HOST:$GRAPHD_PORT -------------------------------------------------------------------------------- /dev/KG_memory_stream/storage_graph/ch1_2/default__vector_store.json: -------------------------------------------------------------------------------- 1 | {"embedding_dict": {}, "text_id_to_ref_doc_id": {}, "metadata_dict": {}} -------------------------------------------------------------------------------- /dev/KG_memory_stream/storage_graph/ch1_2/image__vector_store.json: -------------------------------------------------------------------------------- 1 | {"embedding_dict": {}, "text_id_to_ref_doc_id": {}, "metadata_dict": {}} -------------------------------------------------------------------------------- /dev/KG_memory_stream/storage_graph/ch1_2/index_store.json: -------------------------------------------------------------------------------- 1 | {"index_store/data": {"561d616d-ff4c-49aa-95d5-5438a668e83b": {"__type__": "kg", "__data__": "{\"index_id\": \"561d616d-ff4c-49aa-95d5-5438a668e83b\", \"summary\": null, \"table\": {\"Dursley\": [\"6bf676ee-cdc3-46e8-b705-e1439a6352e8\", \"637a3fb0-c704-4f4d-b83a-a12ce5704e2e\", \"3f27fe25-de91-48ff-88e3-08b649ac4fd4\"], \"Grunnings\": [\"637a3fb0-c704-4f4d-b83a-a12ce5704e2e\"], \"Secret\": [\"637a3fb0-c704-4f4d-b83a-a12ce5704e2e\"], \"Dursleys\": [\"6737c2d6-9801-4bb1-b3fb-f20d2a7fdac1\", \"b4777a86-4c06-471d-8f96-0444ea40a04c\", \"635795f3-91d8-4700-9b99-8525e317f337\", \"d9ec5ecb-abf8-488f-81f3-d488cf88837b\", \"ce87c184-7c4a-4906-a6d8-a5530611b0d9\", \"eaa2427e-1bdf-464d-b6e2-a266fa43fbdb\", \"f0f3cf08-60c7-4aef-97be-c2db7d531503\"], \"Neighbors\": [\"eaa2427e-1bdf-464d-b6e2-a266fa43fbdb\"], \"Potters\": [\"b4777a86-4c06-471d-8f96-0444ea40a04c\", \"eaa2427e-1bdf-464d-b6e2-a266fa43fbdb\"], \"Mr. dursley\": [\"70f610e4-267d-448a-bb5a-b23024ad9fab\", \"234fe3e5-a6fe-443f-9e3f-f8a87d23209b\", \"d991ae94-f78c-43a4-ae7e-fbd9a0eb98b6\", \"bd76ee93-b3e8-421e-89db-94e3e786b2c9\"], \"Back to window\": [\"234fe3e5-a6fe-443f-9e3f-f8a87d23209b\"], \"Drills\": [\"234fe3e5-a6fe-443f-9e3f-f8a87d23209b\"], \"People\": [\"b1f965be-ca77-4c20-bb01-cede1ee45226\", \"d9ec5ecb-abf8-488f-81f3-d488cf88837b\", \"6bf676ee-cdc3-46e8-b705-e1439a6352e8\", \"bd76ee93-b3e8-421e-89db-94e3e786b2c9\"], \"They\": [\"bd76ee93-b3e8-421e-89db-94e3e786b2c9\"], \"Man\": [\"bd76ee93-b3e8-421e-89db-94e3e786b2c9\"], \"Something\": [\"bd76ee93-b3e8-421e-89db-94e3e786b2c9\"], \"Group\": [\"6bf676ee-cdc3-46e8-b705-e1439a6352e8\"], \"Fear\": [\"6bf676ee-cdc3-46e8-b705-e1439a6352e8\"], \"He\": [\"d991ae94-f78c-43a4-ae7e-fbd9a0eb98b6\", \"cff90cbd-ecdf-450d-8d73-dfc8528da0a0\"], \"Harry\": [\"ce7b5c57-f781-429d-9407-8a1748590baf\", \"f8066950-076c-4840-b0cc-e8331acb4a00\", \"97ea4cd8-fe36-40e8-85e4-ccdcae99765a\", \"834d6cdc-3e39-496f-8159-dd996961bacc\", \"577c7ac6-849a-4ce5-a8f0-e21f84f38186\", \"5cb97c50-3136-4367-975d-deedd11ed28e\", \"d9ec5ecb-abf8-488f-81f3-d488cf88837b\", \"ce87c184-7c4a-4906-a6d8-a5530611b0d9\", \"9c0d1038-cd2e-4a59-b35c-5bc64ea6c693\", \"f0f3cf08-60c7-4aef-97be-c2db7d531503\", \"5ac89898-c2b9-48eb-bb50-1634937b6442\", \"d991ae94-f78c-43a4-ae7e-fbd9a0eb98b6\", \"8f80795d-de01-4f45-bf78-69d6bc9b5d45\"], \"Tabby cat\": [\"d991ae94-f78c-43a4-ae7e-fbd9a0eb98b6\"], \"Spot\": [\"70f610e4-267d-448a-bb5a-b23024ad9fab\"], \"Stranger\": [\"70f610e4-267d-448a-bb5a-b23024ad9fab\"], \"Owls\": [\"b1f965be-ca77-4c20-bb01-cede1ee45226\"], \"Unusually\": [\"b1f965be-ca77-4c20-bb01-cede1ee45226\"], \"Bonfire night\": [\"b1f965be-ca77-4c20-bb01-cede1ee45226\"], \"Sister\": [\"b4777a86-4c06-471d-8f96-0444ea40a04c\"], \"Mrs. dursley\": [\"3f27fe25-de91-48ff-88e3-08b649ac4fd4\"], \"Turning it all over\": [\"3f27fe25-de91-48ff-88e3-08b649ac4fd4\"], \"Albus dumbledore\": [\"54579fb0-4337-41b5-9748-d35bb5ffd38b\"], \"Blue eyes\": [\"54579fb0-4337-41b5-9748-d35bb5ffd38b\"], \"Cat\": [\"54579fb0-4337-41b5-9748-d35bb5ffd38b\"], \"Dumbledore\": [\"0cc21553-a8af-4d61-bb2f-913d3cd3f3c6\", \"c04ffb43-171c-4f81-b2e9-ac12b2cfb836\", \"6e5cd78c-fb99-4b24-96fa-fdf4336d0eb1\", \"ce7b5c57-f781-429d-9407-8a1748590baf\", \"97ea4cd8-fe36-40e8-85e4-ccdcae99765a\", \"c793bdf7-ee45-4167-8a44-a15ee211f6c3\", \"844caaea-9a06-4bfa-9b96-3b2b9f49f0d5\", \"5cb97c50-3136-4367-975d-deedd11ed28e\", \"57179285-4367-403b-bba8-0ca7ba42d3fd\", \"58feb06c-afe3-4f44-90fd-c4175a764b23\"], \"Put-outer\": [\"0cc21553-a8af-4d61-bb2f-913d3cd3f3c6\"], \"Street\": [\"0cc21553-a8af-4d61-bb2f-913d3cd3f3c6\"], \"Professor mcgonagall\": [\"6e5cd78c-fb99-4b24-96fa-fdf4336d0eb1\", \"c04ffb43-171c-4f81-b2e9-ac12b2cfb836\", \"c793bdf7-ee45-4167-8a44-a15ee211f6c3\", \"57179285-4367-403b-bba8-0ca7ba42d3fd\", \"10552601-6d0d-4afa-8732-24c54622257e\"], \"How did you know it was me?\": [\"10552601-6d0d-4afa-8732-24c54622257e\"], \"Patted her\": [\"c04ffb43-171c-4f81-b2e9-ac12b2cfb836\"], \"Her\": [\"5cb97c50-3136-4367-975d-deedd11ed28e\"], \"Voldemort\": [\"5cb97c50-3136-4367-975d-deedd11ed28e\"], \"We can only guess\": [\"6e5cd78c-fb99-4b24-96fa-fdf4336d0eb1\"], \"Lace handkerchief\": [\"6e5cd78c-fb99-4b24-96fa-fdf4336d0eb1\"], \"It's the best place for him\": [\"58feb06c-afe3-4f44-90fd-c4175a764b23\"], \"Mcgonagall\": [\"58feb06c-afe3-4f44-90fd-c4175a764b23\"], \"A letter\": [\"58feb06c-afe3-4f44-90fd-c4175a764b23\"], \"Trust\": [\"844caaea-9a06-4bfa-9b96-3b2b9f49f0d5\"], \"Hagrid\": [\"ce7b5c57-f781-429d-9407-8a1748590baf\", \"97ea4cd8-fe36-40e8-85e4-ccdcae99765a\", \"844caaea-9a06-4bfa-9b96-3b2b9f49f0d5\"], \"Baby boy\": [\"844caaea-9a06-4bfa-9b96-3b2b9f49f0d5\"], \"Lily and james\": [\"ce7b5c57-f781-429d-9407-8a1748590baf\"], \"Motorcycle\": [\"5ac89898-c2b9-48eb-bb50-1634937b6442\", \"97ea4cd8-fe36-40e8-85e4-ccdcae99765a\"], \"Nose\": [\"57179285-4367-403b-bba8-0ca7ba42d3fd\"], \"Walked\": [\"57179285-4367-403b-bba8-0ca7ba42d3fd\"], \"Privet drive\": [\"6737c2d6-9801-4bb1-b3fb-f20d2a7fdac1\"], \"All\": [\"6737c2d6-9801-4bb1-b3fb-f20d2a7fdac1\"], \"Nephew\": [\"6737c2d6-9801-4bb1-b3fb-f20d2a7fdac1\"], \"Philz\": [\"834d6cdc-3e39-496f-8159-dd996961bacc\", \"635795f3-91d8-4700-9b99-8525e317f337\", \"ce87c184-7c4a-4906-a6d8-a5530611b0d9\", \"9c0d1038-cd2e-4a59-b35c-5bc64ea6c693\", \"3af30930-2141-400f-bb90-4b23434ee795\"], \"Berkeley\": [\"834d6cdc-3e39-496f-8159-dd996961bacc\", \"635795f3-91d8-4700-9b99-8525e317f337\", \"ce87c184-7c4a-4906-a6d8-a5530611b0d9\", \"9c0d1038-cd2e-4a59-b35c-5bc64ea6c693\", \"3af30930-2141-400f-bb90-4b23434ee795\"], \"Dream\": [\"9c0d1038-cd2e-4a59-b35c-5bc64ea6c693\"], \"Round glasses\": [\"9c0d1038-cd2e-4a59-b35c-5bc64ea6c693\"], \"Thin face\": [\"834d6cdc-3e39-496f-8159-dd996961bacc\"], \"Dudley\": [\"f8066950-076c-4840-b0cc-e8331acb4a00\", \"577c7ac6-849a-4ce5-a8f0-e21f84f38186\", \"cff90cbd-ecdf-450d-8d73-dfc8528da0a0\", \"37b3121c-3ce0-4f11-8eeb-f2291d49adf0\", \"0998c8da-dea8-4308-8a7e-f3afb293dddc\", \"3af30930-2141-400f-bb90-4b23434ee795\", \"8f80795d-de01-4f45-bf78-69d6bc9b5d45\"], \"Baby angel\": [\"3af30930-2141-400f-bb90-4b23434ee795\"], \"Wolfing down\": [\"8f80795d-de01-4f45-bf78-69d6bc9b5d45\"], \"Moment\": [\"8f80795d-de01-4f45-bf78-69d6bc9b5d45\"], \"Horror\": [\"0998c8da-dea8-4308-8a7e-f3afb293dddc\"], \"Aunt petunia\": [\"317cdfc1-fa74-4c77-9b8a-e78cba4a0e35\", \"37b3121c-3ce0-4f11-8eeb-f2291d49adf0\"], \"As though she'd just swallowed a lemon\": [\"37b3121c-3ce0-4f11-8eeb-f2291d49adf0\"], \"Loudly\": [\"37b3121c-3ce0-4f11-8eeb-f2291d49adf0\"], \"People's arms\": [\"cff90cbd-ecdf-450d-8d73-dfc8528da0a0\"], \"Them\": [\"cff90cbd-ecdf-450d-8d73-dfc8528da0a0\"], \"Uncle vernon\": [\"5ac89898-c2b9-48eb-bb50-1634937b6442\"], \"Motorcycles\": [\"5ac89898-c2b9-48eb-bb50-1634937b6442\"], \"Wind\": [\"f8066950-076c-4840-b0cc-e8331acb4a00\"], \"Bored\": [\"f8066950-076c-4840-b0cc-e8331acb4a00\"], \"Dudley and piers\": [\"635795f3-91d8-4700-9b99-8525e317f337\"], \"Huge poisonous cobras\": [\"577c7ac6-849a-4ce5-a8f0-e21f84f38186\"], \"Snake\": [\"577c7ac6-849a-4ce5-a8f0-e21f84f38186\"], \"Boa constrictor\": [\"317cdfc1-fa74-4c77-9b8a-e78cba4a0e35\"], \"Zoo\": [\"317cdfc1-fa74-4c77-9b8a-e78cba4a0e35\"], \"Zoo director\": [\"317cdfc1-fa74-4c77-9b8a-e78cba4a0e35\"], \"Throughout reptile house\": [\"d9ec5ecb-abf8-488f-81f3-d488cf88837b\"], \"Unknown relation\": [\"f0f3cf08-60c7-4aef-97be-c2db7d531503\"], \"Only family\": [\"f0f3cf08-60c7-4aef-97be-c2db7d531503\"]}, \"rel_map\": {}, \"embedding_dict\": {}}"}}} -------------------------------------------------------------------------------- /dev/KG_memory_stream/storage_graph/ch1_2_3/default__vector_store.json: -------------------------------------------------------------------------------- 1 | {"embedding_dict": {}, "text_id_to_ref_doc_id": {}, "metadata_dict": {}} -------------------------------------------------------------------------------- /dev/KG_memory_stream/storage_graph/ch1_2_3/image__vector_store.json: -------------------------------------------------------------------------------- 1 | {"embedding_dict": {}, "text_id_to_ref_doc_id": {}, "metadata_dict": {}} -------------------------------------------------------------------------------- /dev/KG_memory_stream/storage_graph/ch1_2_3_4/default__vector_store.json: -------------------------------------------------------------------------------- 1 | {"embedding_dict": {}, "text_id_to_ref_doc_id": {}, "metadata_dict": {}} -------------------------------------------------------------------------------- /dev/KG_memory_stream/storage_graph/ch1_2_3_4/image__vector_store.json: -------------------------------------------------------------------------------- 1 | {"embedding_dict": {}, "text_id_to_ref_doc_id": {}, "metadata_dict": {}} -------------------------------------------------------------------------------- /dev/KG_memory_stream/tests/memory/test_entity_knowledge_store.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | import os 3 | from datetime import datetime 4 | from src.memory import EntityKnowledgeStore 5 | from src.memory.types import KnowledgeMemoryItem, MemoryItem 6 | 7 | 8 | class TestEntityKnowledgeStore(unittest.TestCase): 9 | 10 | def setUp(self): 11 | self.file_name = "tests/memory/test_knowledge_memory.json" 12 | self.entity_knowledge_store = EntityKnowledgeStore( 13 | file_name=self.file_name) 14 | 15 | def tearDown(self): 16 | # Clean up test file after each test 17 | try: 18 | os.remove(self.file_name) 19 | except FileNotFoundError: 20 | pass 21 | 22 | def test_add_memory(self): 23 | data = [ 24 | MemoryItem("test_entity", 25 | datetime.now().replace(microsecond=0)) 26 | ] 27 | self.entity_knowledge_store.add_memory(data) 28 | assert len(self.entity_knowledge_store.knowledge_memory) == 1 29 | assert isinstance(self.entity_knowledge_store.knowledge_memory[0], 30 | KnowledgeMemoryItem) 31 | 32 | def test_convert_memory_to_knowledge_memory(self): 33 | data = [ 34 | MemoryItem("test_entity", 35 | datetime.now().replace(microsecond=0)) 36 | ] 37 | converted_data = self.entity_knowledge_store._convert_memory_to_knowledge_memory( 38 | data) 39 | assert len(converted_data) == 1 40 | assert isinstance(converted_data[0], KnowledgeMemoryItem) 41 | 42 | def test_update_knowledge_memory(self): 43 | data = [ 44 | KnowledgeMemoryItem("knowledge_entity", 1, 45 | datetime.now().replace(microsecond=0)) 46 | ] 47 | self.entity_knowledge_store._update_knowledge_memory(data) 48 | assert len(self.entity_knowledge_store.knowledge_memory) == 1 49 | assert self.entity_knowledge_store.knowledge_memory[0] == data[0] 50 | -------------------------------------------------------------------------------- /dev/KG_memory_stream/tests/memory/test_memory.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "entity": "test_entity", 4 | "date": "2024-02-26T12:22:14" 5 | } 6 | ] -------------------------------------------------------------------------------- /dev/KG_memory_stream/tests/memory/test_memory_stream.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | import os 3 | from datetime import datetime, timedelta 4 | from src.memory import MemoryStream 5 | from src.memory.types import MemoryItem 6 | 7 | class TestMemoryStream(unittest.TestCase): 8 | def setUp(self): 9 | self.file_name = "tests/memory/test_memory.json" 10 | self.memory_stream = MemoryStream(file_name=self.file_name) 11 | 12 | def tearDown(self): 13 | # Clean up test file after each test 14 | try: 15 | os.remove(self.file_name) 16 | except FileNotFoundError: 17 | pass 18 | 19 | def test_add_memory(self): 20 | data = [MemoryItem("test_entity", datetime.now().replace(microsecond=0))] 21 | self.memory_stream.add_memory(data) 22 | self.assertEqual(len(self.memory_stream), 1) 23 | self.assertEqual(self.memory_stream.get_memory()[0], data[0]) 24 | 25 | def test_remove_old_memory(self): 26 | past_date = datetime.now().replace(microsecond=0) - timedelta(days=10) 27 | self.memory_stream.add_memory([MemoryItem("old_entity", past_date)]) 28 | self.memory_stream.remove_old_memory(5) 29 | self.assertEqual(len(self.memory_stream.get_memory()), 0) 30 | 31 | def test_save_and_load_memory(self): 32 | data = [MemoryItem("test_entity", datetime.now().replace(microsecond=0))] 33 | self.memory_stream.add_memory(data) 34 | self.memory_stream.save_memory() 35 | new_memory_stream = MemoryStream(file_name=self.file_name) 36 | self.assertEqual(len(new_memory_stream), len(self.memory_stream)) 37 | self.assertEqual(new_memory_stream.get_memory(), self.memory_stream.get_memory()) 38 | 39 | def test_get_memory_by_index(self): 40 | data = [MemoryItem("entity1", datetime.now().replace(microsecond=0)), MemoryItem("entity2", datetime.now().replace(microsecond=0))] 41 | self.memory_stream.add_memory(data) 42 | self.assertEqual(self.memory_stream.get_memory_by_index(1), data[1]) 43 | 44 | def test_remove_memory_by_index(self): 45 | data = [MemoryItem("entity1", datetime.now().replace(microsecond=0)), MemoryItem("entity2", datetime.now().replace(microsecond=0))] 46 | self.memory_stream.add_memory(data) 47 | self.memory_stream.remove_memory_by_index(0) 48 | self.assertEqual(len(self.memory_stream), 1) 49 | self.assertEqual(self.memory_stream.get_memory()[0], data[1]) 50 | 51 | if __name__ == '__main__': 52 | unittest.main() 53 | -------------------------------------------------------------------------------- /dev/legacy_routing_agent/get_location.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "%pip install geopy" 10 | ] 11 | }, 12 | { 13 | "cell_type": "code", 14 | "execution_count": null, 15 | "metadata": {}, 16 | "outputs": [], 17 | "source": [ 18 | "import requests\n", 19 | "from geopy.geocoders import Nominatim\n", 20 | "\n", 21 | "def get_location():\n", 22 | " try:\n", 23 | " # Fetch location data from ip-api\n", 24 | " ip_url = 'http://ip-api.com/json/'\n", 25 | " response = requests.get(ip_url)\n", 26 | " data = response.json()\n", 27 | "\n", 28 | " # Check if the response is successful\n", 29 | " if data['status'] == 'success':\n", 30 | " latitude = data['lat']\n", 31 | " longitude = data['lon']\n", 32 | "\n", 33 | " # Use geopy to get more detailed location data\n", 34 | " geolocator = Nominatim(user_agent=\"geoapiExercises\")\n", 35 | " location = geolocator.reverse(f\"{latitude}, {longitude}\", exactly_one=True)\n", 36 | " if location:\n", 37 | " return location.latitude, location.longitude\n", 38 | " else:\n", 39 | " return latitude, longitude\n", 40 | " else:\n", 41 | " return \"Location could not be determined using IP.\", \"\"\n", 42 | "\n", 43 | " except Exception as e:\n", 44 | " return f\"An error occurred: {e}\", \"\"\n", 45 | "\n", 46 | "latitude, longitude = get_location()\n", 47 | "if latitude and longitude:\n", 48 | " print(f\"Current Latitude: {latitude}\")\n", 49 | " print(f\"Current Longitude: {longitude}\")\n", 50 | "else:\n", 51 | " print(latitude) # This would print the error message if location could not be determined\n" 52 | ] 53 | }, 54 | { 55 | "cell_type": "code", 56 | "execution_count": null, 57 | "metadata": {}, 58 | "outputs": [], 59 | "source": [] 60 | } 61 | ], 62 | "metadata": { 63 | "kernelspec": { 64 | "display_name": "base", 65 | "language": "python", 66 | "name": "python3" 67 | }, 68 | "language_info": { 69 | "codemirror_mode": { 70 | "name": "ipython", 71 | "version": 3 72 | }, 73 | "file_extension": ".py", 74 | "mimetype": "text/x-python", 75 | "name": "python", 76 | "nbconvert_exporter": "python", 77 | "pygments_lexer": "ipython3", 78 | "version": "3.11.5" 79 | } 80 | }, 81 | "nbformat": 4, 82 | "nbformat_minor": 2 83 | } 84 | -------------------------------------------------------------------------------- /dev/legacy_routing_agent/main.py: -------------------------------------------------------------------------------- 1 | import ml, utils 2 | from flask import Flask, request, jsonify, send_file, render_template, make_response 3 | from flask.helpers import send_from_directory 4 | from werkzeug.utils import secure_filename 5 | from dotenv import load_dotenv 6 | import json, os, uuid 7 | import googlemaps 8 | 9 | load_dotenv() 10 | 11 | app = Flask(__name__) 12 | app.config['UPLOAD_FOLDER'] = './uploads' 13 | google_maps_key = os.getenv("google_maps_key") 14 | gmaps = googlemaps.Client(key=google_maps_key) 15 | 16 | @app.route('/audio/', methods=['GET']) 17 | def serve_audio(filename): 18 | return send_from_directory(os.path.join('./audio_files'), filename, mimetype='audio/wav') 19 | 20 | @app.route('/process', methods=['POST']) 21 | def process(): 22 | try: 23 | question = request.form['question'] 24 | 25 | if "vision" in question: 26 | return handle_vision_question(request) 27 | elif "location" in question: 28 | return handle_location_question(request) 29 | else: 30 | return jsonify({"message": "Question type not recognized."}), 400 31 | 32 | except Exception as e: 33 | print(f"Caught exception: {e}") 34 | return make_response(jsonify({"error": str(e)}), 400) 35 | 36 | def handle_vision_question(request): 37 | print("Handling vision request ...") 38 | try: 39 | if 'file' not in request.files: 40 | return jsonify({"error": "No file part in the request"}), 400 41 | 42 | file = request.files['file'] 43 | question = request.form['question'] 44 | 45 | if file.filename == '': 46 | print("No file selected") 47 | return jsonify({"error": "No file selected"}), 400 48 | 49 | if file: 50 | filename = secure_filename(file.filename) 51 | filepath = os.path.join(app.config['UPLOAD_FOLDER'], filename) 52 | file.save(filepath) 53 | print(f"Saved file to {filepath}") 54 | 55 | base64_image = ml.encode_image(filepath) 56 | 57 | gpt_response = ml.call_gpt_vision(base64_image, question) 58 | print("GPT-4 vision response recieved.") 59 | 60 | return handle_audio_response(gpt_response) 61 | 62 | except Exception as e: 63 | print(f"Caught exception in handle_vision_question: {e}") 64 | return jsonify({"error": str(e)}), 500 65 | 66 | def handle_location_question(request): 67 | print("Handling location request ...") 68 | 69 | try: 70 | latitude = request.form.get('latitude') 71 | longitude = request.form.get('longitude') 72 | print(f"Location recieved: ({latitude}, {longitude})") 73 | 74 | if not latitude or not longitude: 75 | text = "Location could not be determined." 76 | else: 77 | reverse_geocode_result = gmaps.reverse_geocode((latitude, longitude)) 78 | readable_address = reverse_geocode_result[0]['formatted_address'] 79 | 80 | text = "Your readable address is" + readable_address 81 | 82 | return handle_audio_response(text) 83 | 84 | except Exception as e: 85 | print(f"Caught exception in handle_location_question: {e}") 86 | return jsonify({"error": str(e)}), 500 87 | 88 | def handle_audio_response(text): 89 | audio_directory = './audio_files' 90 | if not os.path.exists(audio_directory): 91 | os.makedirs(audio_directory) 92 | 93 | audio_filename = f"audio_{uuid.uuid4()}.mp3" 94 | audio_path = os.path.join(audio_directory, audio_filename) 95 | ml.text_to_speech(text, audio_path) 96 | print(f"Audio file saved at {audio_path}") 97 | 98 | audio_url = request.url_root + "audio/" + audio_filename 99 | return jsonify({"audio_url": audio_url}) 100 | 101 | @app.route('/') 102 | def home(): 103 | return render_template('index.html') 104 | 105 | if __name__ == "__main__": 106 | app.run(debug=True) 107 | # app.run(host='0.0.0.0', port=5000) 108 | -------------------------------------------------------------------------------- /dev/legacy_routing_agent/ml.py: -------------------------------------------------------------------------------- 1 | from openai import OpenAI 2 | import os 3 | import requests 4 | import base64 5 | from dotenv import load_dotenv 6 | 7 | load_dotenv() 8 | OPENAI_API_KEY = os.getenv("openai_api_key") 9 | 10 | client = OpenAI() 11 | 12 | def call_gpt_model(prompt, data, model, temperature=None): 13 | messages = [ 14 | {"role": "system", "content": prompt}, 15 | {"role": "user", "content": data} 16 | ] 17 | 18 | api_params = { 19 | "model": model, 20 | "messages": messages 21 | } 22 | 23 | if temperature is not None: 24 | api_params["temperature"] = temperature 25 | 26 | try: 27 | response = client.chat.completions.create(**api_params) 28 | response_content = response.choices[0].message.content.strip() 29 | 30 | return response_content 31 | 32 | except Exception as e: 33 | raise RuntimeError(f"An error occurred while making an API call: {e}") 34 | 35 | def call_gpt_vision(base64_image, user): 36 | headers = { 37 | "Content-Type": "application/json", 38 | "Authorization": f"Bearer {OPENAI_API_KEY}" 39 | } 40 | 41 | payload = { 42 | "model": "gpt-4-vision-preview", 43 | "messages": [ 44 | { 45 | "role": "system", 46 | "content": "You are tasked with answering a blind individual's question about their current environment. Aim for brevity without sacrificing the immersive experience." 47 | }, 48 | { 49 | "role": "user", 50 | "content": [ 51 | { 52 | "type": "text", 53 | "text": user 54 | }, 55 | { 56 | "type": "image_url", 57 | "image_url": { 58 | "url": f"data:image/jpeg;base64,{base64_image}" 59 | } 60 | } 61 | ] 62 | } 63 | ], 64 | "max_tokens": 300 65 | } 66 | 67 | response = requests.post("https://api.openai.com/v1/chat/completions", headers=headers, json=payload) 68 | 69 | return response.json()['choices'][0]['message']['content'] 70 | 71 | def encode_image(image_path): 72 | with open(image_path, "rb") as image_file: 73 | return base64.b64encode(image_file.read()).decode('utf-8') 74 | 75 | def text_to_speech(text, filepath): 76 | try: 77 | response = client.audio.speech.create( 78 | model="tts-1", 79 | voice="shimmer", 80 | input=text 81 | ) 82 | response.stream_to_file(filepath) 83 | 84 | except Exception as e: 85 | raise RuntimeError(f"An unexpected error occurred: {str(e)}") 86 | 87 | def speech_to_text(filepath): 88 | try: 89 | audio_file = open(filepath, "rb") 90 | transcript = client.audio.transcriptions.create( 91 | model="whisper-1", 92 | file=audio_file, 93 | prompt="The transcript is about a blind person asking about their environment", 94 | response_format="text" 95 | ) 96 | except Exception as e: 97 | return f"An unexpected error occurred: {str(e)}" -------------------------------------------------------------------------------- /dev/legacy_routing_agent/readme.md: -------------------------------------------------------------------------------- 1 | # *This is the old routing agent implementation.* Current implementation of the routing agent can be found in src/agent/base_agent.py 2 | 3 | Llama Index Tool Specification Example Usage 4 | This document provides an overview of how to utilize custom tool specifications with the Llama Index framework to handle specialized tasks like vision and location-based queries. 5 | 6 | Installation 7 | Ensure that the necessary Python packages are installed: 8 | 9 | python 10 | Copy code 11 | %pip install pydantic 12 | %pip install googlemaps 13 | %pip install llama_index 14 | These commands install pydantic for data validation, googlemaps for handling location queries, and llama_index for integrating AI tools and agents. 15 | 16 | Configuration 17 | The following classes are defined but commented out for clarity. You should uncomment them in your implementation if you plan to use their functionality: 18 | 19 | python 20 | Copy code 21 | # from main import handle_location_question, handle_vision_question, app 22 | # from llama_index.agent.openai import OpenAIAgent 23 | These lines would typically import custom functions and classes necessary for handling specific types of questions within an application framework like Flask. 24 | 25 | Tool Specifications 26 | Define tool specifications to modularize the functionality for vision and location queries: 27 | 28 | python 29 | Copy code 30 | from llama_index.core.tools.tool_spec.base import BaseToolSpec 31 | 32 | class CVToolSpec(BaseToolSpec): 33 | spec_functions = ['handle_vision_question'] 34 | 35 | def __init__(self): # , form): 36 | # self.form = form 37 | pass 38 | 39 | def handle_vision_question(request): 40 | return handle_vision_question(request) 41 | 42 | class LocationToolSpec(BaseToolSpec): 43 | spec_functions = ['handle_location_question'] 44 | 45 | def __init__(self): # , form): 46 | # self.form = form 47 | pass 48 | 49 | def handle_location_question(request): 50 | return handle_location_question(request) 51 | The CVToolSpec and LocationToolSpec classes inherit from BaseToolSpec and specify functions that handle vision and location questions, respectively. 52 | 53 | Integration with Llama Index 54 | Create an OpenAI language model agent and bind it with the specified tools: 55 | 56 | python 57 | Copy code 58 | from llama_index.agent.openai import OpenAIAgent 59 | from llama_index.llms.openai import OpenAI 60 | 61 | cv_tool = CVToolSpec() 62 | loc_tool = LocationToolSpec() 63 | full_tool_list = cv_tool.to_tool_list() + loc_tool.to_tool_list() 64 | 65 | llm = OpenAI(model="gpt-3.5-turbo") 66 | agent = OpenAIAgent.from_tools(full_tool_list, verbose=True) 67 | This setup initializes an OpenAI LLM with a custom agent that incorporates both vision and location handling tools. 68 | 69 | Execution 70 | The agent can now handle complex queries that require integration of different functionalities: 71 | 72 | python 73 | Copy code 74 | # This is a hypothetical example and would need a real request object to work: 75 | # response = agent.handle(request) 76 | # print(response) 77 | Replace agent.handle(request) with an appropriate method call to handle real requests. Ensure the request object is constructed properly according to the needs of the handle_vision_question and handle_location_question functions. 78 | 79 | Extending Functionality 80 | The framework allows for easy extension by adding new tool specifications or modifying existing ones to handle different types of data or queries. 81 | 82 | Remember to handle API keys and sensitive data securely, especially when dealing with location data or integrating third-party services like Google Maps. 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | -------------------------------------------------------------------------------- /dev/legacy_routing_agent/tool_spec.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "%pip install pydantic\n", 10 | "%pip install googlemaps" 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": null, 16 | "metadata": {}, 17 | "outputs": [], 18 | "source": [ 19 | "from main import handle_location_question, handle_vision_question, app" 20 | ] 21 | }, 22 | { 23 | "cell_type": "code", 24 | "execution_count": null, 25 | "metadata": {}, 26 | "outputs": [], 27 | "source": [ 28 | "%pip install llama_index" 29 | ] 30 | }, 31 | { 32 | "cell_type": "code", 33 | "execution_count": null, 34 | "metadata": {}, 35 | "outputs": [], 36 | "source": [ 37 | "# from llama_index.agent.openai import OpenAIAgent\n", 38 | "from llama_index.core.tools.tool_spec.base import BaseToolSpec" 39 | ] 40 | }, 41 | { 42 | "cell_type": "code", 43 | "execution_count": null, 44 | "metadata": {}, 45 | "outputs": [], 46 | "source": [ 47 | "class CVToolSpec(BaseToolSpec):\n", 48 | " spec_functions = ['handle_vision_question']\n", 49 | "\n", 50 | " def __init__(self): #, form):\n", 51 | " # self.form = form\n", 52 | " ''''''\n", 53 | "\n", 54 | " def handle_vision_question(request):\n", 55 | " return handle_vision_question(request)" 56 | ] 57 | }, 58 | { 59 | "cell_type": "code", 60 | "execution_count": null, 61 | "metadata": {}, 62 | "outputs": [], 63 | "source": [ 64 | "cv_tool = CVToolSpec()\n", 65 | "cv_tool_list = cv_tool.to_tool_list()" 66 | ] 67 | }, 68 | { 69 | "cell_type": "code", 70 | "execution_count": null, 71 | "metadata": {}, 72 | "outputs": [], 73 | "source": [ 74 | "class LocationToolSpec(BaseToolSpec):\n", 75 | " spec_functions = ['handle_location_question']\n", 76 | "\n", 77 | " def __init__(self): #, form):\n", 78 | " # self.form = form\n", 79 | " ''''''\n", 80 | "\n", 81 | " def handle_location_question(request): \n", 82 | " return handle_location_question(request)\n", 83 | " " 84 | ] 85 | }, 86 | { 87 | "cell_type": "code", 88 | "execution_count": null, 89 | "metadata": {}, 90 | "outputs": [], 91 | "source": [ 92 | "from flask import request \n", 93 | "\n", 94 | "# form_data = request.form\n", 95 | "\n", 96 | "loc_tool = LocationToolSpec()\n", 97 | "loc_tool_list = loc_tool.to_tool_list(['handle_location_question'])" 98 | ] 99 | }, 100 | { 101 | "cell_type": "code", 102 | "execution_count": null, 103 | "metadata": {}, 104 | "outputs": [], 105 | "source": [ 106 | "# from {KG repo} import {KG query method}" 107 | ] 108 | }, 109 | { 110 | "cell_type": "code", 111 | "execution_count": null, 112 | "metadata": {}, 113 | "outputs": [], 114 | "source": [ 115 | "class SearchToolSpec(BaseToolSpec):\n", 116 | " spec_functions = ['']\n", 117 | "\n", 118 | " def __init__(self):\n", 119 | " ''''''\n", 120 | "\n", 121 | " def handle_search_request(request):\n", 122 | " return handle_search_request(request)" 123 | ] 124 | }, 125 | { 126 | "cell_type": "markdown", 127 | "metadata": {}, 128 | "source": [ 129 | "### Experiment with making Agents from Tools" 130 | ] 131 | }, 132 | { 133 | "cell_type": "code", 134 | "execution_count": null, 135 | "metadata": {}, 136 | "outputs": [], 137 | "source": [ 138 | "from llama_index.agent.openai import OpenAIAgent\n", 139 | "from llama_index.llms.openai import OpenAI\n", 140 | "\n", 141 | "full_tool_list = cv_tool.to_tool_list() + loc_tool.to_tool_list()\n", 142 | "llm = OpenAI(model=\"gpt-3.5-turbo\")\n", 143 | "agent = OpenAIAgent.from_tools(full_tool_list, verbose=True)" 144 | ] 145 | } 146 | ], 147 | "metadata": { 148 | "kernelspec": { 149 | "display_name": "base", 150 | "language": "python", 151 | "name": "python3" 152 | }, 153 | "language_info": { 154 | "codemirror_mode": { 155 | "name": "ipython", 156 | "version": 3 157 | }, 158 | "file_extension": ".py", 159 | "mimetype": "text/x-python", 160 | "name": "python", 161 | "nbconvert_exporter": "python", 162 | "pygments_lexer": "ipython3", 163 | "version": "3.11.5" 164 | } 165 | }, 166 | "nbformat": 4, 167 | "nbformat_minor": 2 168 | } 169 | -------------------------------------------------------------------------------- /dev/legacy_routing_agent/utils.py: -------------------------------------------------------------------------------- 1 | import os 2 | import googlemaps 3 | import json 4 | from langchain.chat_models import ChatOpenAI 5 | from langchain.tools import StructuredTool 6 | from langchain.agents import initialize_agent, AgentType 7 | from pydantic import BaseModel, Field 8 | from datetime import datetime 9 | from typing import Optional, Union, List, Tuple, Any 10 | from dotenv import load_dotenv 11 | 12 | load_dotenv() 13 | 14 | google_maps_key = os.getenv("google_maps_key") 15 | gmaps = googlemaps.Client(key=google_maps_key) 16 | 17 | def retrieve_location(): 18 | "Returns latitude and longitude of current location" 19 | latitude = 37.423304896346345 20 | longitude = -122.16023584591919 21 | 22 | return latitude, longitude 23 | 24 | def reverse_geocode(latitude: float, longitude: float): 25 | "Converts latitude and longitude into a readable address" 26 | result = gmaps.reverse_geocode((latitude, longitude)) 27 | if result: 28 | address = result[0].get('formatted_address', 'No address found') 29 | else: 30 | address = 'No address found' 31 | 32 | return address 33 | 34 | def find_places(query: None, location = None, radius = None, language = None, min_price = None, max_price = None, open_now = False, place_type = None, region = None): 35 | params = { 36 | 'query': query, 37 | 'location': location, 38 | 'radius': radius, 39 | 'language': language, 40 | 'min_price': min_price, 41 | 'max_price': max_price, 42 | 'open_now': open_now, 43 | 'type': place_type, 44 | 'region': region 45 | } 46 | 47 | params = {k: v for k, v in params.items() if v is not None} 48 | response = gmaps.places(**params) 49 | 50 | results = response['results'][:5] # limit to top 5 results 51 | 52 | simplified_results = [] 53 | for place in results: 54 | place_info = { 55 | 'place_id': place.get('place_id'), 56 | 'name': place.get('name'), 57 | 'location': (place['geometry']['location']['lat'], place['geometry']['location']['lng']), 58 | 'formatted_address': place.get('formatted_address'), 59 | 'rating': place.get('rating') 60 | } 61 | simplified_results.append(place_info) 62 | 63 | return simplified_results 64 | 65 | class FindPlacesArgs(BaseModel): 66 | query: Optional[str] = Field(None, description = "Text string to search for, e.g., 'restaurant'") 67 | location: Optional[Union[tuple, list, str]] = Field(None, description = "Latitude and longitude as a tuple or string") 68 | radius: Optional[int] = Field(None, description = "Search radius in meters") 69 | language: Optional[str] = Field(None, description = "The language in which to return results") 70 | min_price: Optional[int] = Field(None, description = "Minimum price level of places to include, from 0 (most affordable) to 4 (most expensive)") 71 | max_price: Optional[int] = Field(None, description = "Maximum price level of places to include, from 0 (most affordable) to 4 (most expensive)") 72 | open_now: Optional[bool] = Field(False, description = "Return only those places that are open for business at the time the query is sent") 73 | place_type: Optional[str] = Field(None, description = "Type of place to restrict results to") 74 | region: Optional[str] = Field(None, description = "Region code to bias results towards") 75 | 76 | find_places_tool = StructuredTool( 77 | func = find_places, 78 | name = "Find Places", 79 | description = "Finds places based on location and other optional parameters. For each place, returns place ID, name, coordinates as a tuple, formatted address, and rating.", 80 | args_schema = FindPlacesArgs 81 | ) 82 | 83 | def get_directions(origin, destination, mode="walking", waypoints=None, alternatives=False, avoid=None, language=None, units=None, region=None, departure_time=None, arrival_time=None, optimize_waypoints=False, transit_mode=None, transit_routing_preference=None, traffic_model=None): 84 | params = { 85 | 'origin': origin, 86 | 'destination': destination, 87 | 'mode': mode, 88 | 'waypoints': waypoints, 89 | 'alternatives': alternatives, 90 | 'avoid': avoid, 91 | 'language': language, 92 | 'units': units, 93 | 'region': region, 94 | 'departure_time': departure_time, 95 | 'arrival_time': arrival_time, 96 | 'optimize_waypoints': optimize_waypoints, 97 | 'transit_mode': transit_mode, 98 | 'transit_routing_preference': transit_routing_preference, 99 | 'traffic_model': traffic_model 100 | } 101 | 102 | # Remove None values 103 | params = {k: v for k, v in params.items() if v is not None} 104 | 105 | # Make the API call 106 | response = gmaps.directions(**params) 107 | 108 | return response 109 | 110 | class DirectionsArgs(BaseModel): 111 | origin: Union[str, Tuple[float, float]] = Field(..., description="The starting address or latitude/longitude value.") 112 | destination: Union[str, Tuple[float, float]] = Field(..., description="The ending address or latitude/longitude value.") 113 | mode: Optional[str] = Field("walking", description="Mode of transport (driving, walking, bicycling, or transit).") 114 | waypoints: Optional[Union[List[Union[str, Tuple[float, float]]], str]] = Field(None, description="Array of waypoints to alter the route. To route through a location without stopping, prefix it with 'via:', e.g., ['via:San Francisco', 'via:Mountain View'].") 115 | alternatives: bool = Field(False, description="If True, more than one route may be returned in the response.") 116 | avoid: Optional[Union[str, List[str]]] = Field(None, description="Features to avoid (tolls, highways, ferries, etc.).") 117 | language: Optional[str] = Field(None, description="Language for the results.") 118 | units: Optional[str] = Field(None, description="Unit system for the results (metric or imperial).") 119 | region: Optional[str] = Field(None, description="Region code, specified as a ccTLD two-character value.") 120 | departure_time: Optional[Union[int, datetime]] = Field(None, description="Desired time of departure as a timestamp or datetime.") 121 | arrival_time: Optional[Union[int, datetime]] = Field(None, description="Desired time of arrival as a timestamp or datetime.") 122 | optimize_waypoints: bool = Field(False, description="Whether to optimize the order of waypoints.") 123 | transit_mode: Optional[Union[str, List[str]]] = Field(None, description="Preferred modes of transit (bus, subway, train, etc.).") 124 | transit_routing_preference: Optional[str] = Field(None, description="Preferences for transit routing (less_walking or fewer_transfers).") 125 | traffic_model: Optional[str] = Field(None, description="Predictive travel time model (best_guess, optimistic, pessimistic).") 126 | 127 | def main(): 128 | tools = [] 129 | tools.append(StructuredTool.from_function(retrieve_location)) 130 | tools.append(StructuredTool.from_function(reverse_geocode)) 131 | tools.append(find_places_tool) 132 | 133 | # TODO: 134 | # Missing: place (implement?) 135 | # Missing: find_place (implement?) 136 | # Missing: directions 137 | # Missing: distance_matrix 138 | # Missing: vision 139 | # Missing: audio 140 | 141 | # llm = ChatOpenAI(model="gpt-4") 142 | # agent_chain = initialize_agent(tools, llm, agent=AgentType.STRUCTURED_CHAT_ZERO_SHOT_REACT_DESCRIPTION, verbose=True) 143 | # agent_chain.invoke( 144 | # { 145 | # "input": "Is there a cafe near me?" 146 | # } 147 | # ) 148 | 149 | latitude = 37.423304896346345 150 | longitude = -122.16023584591919 151 | 152 | response = get_directions(origin=(latitude, longitude), destination="Coupa Cafe Green Library Stanford") 153 | # print(response[0]['legs'][0]) # 'legs' because walking was specified? 154 | # print(response[0]['legs'][0].keys()) # keys = distance, duration, end_address, end_location, start_address, start_location, steps, traffic_speed_entry, via_waypoint 155 | # print(response[0]['legs'][0]['steps'][0].keys()) # keys = distance, duration, end_location, html_instructions, polyline, start_location, travel_mode 156 | 157 | # TODO: 158 | # places() 159 | # Only certain 'type' can be accepted. Prompt agent. 160 | # Only certain 'regions' can be accepted. 161 | 162 | 163 | 164 | if __name__ == "__main__": 165 | main() 166 | # Important GMaps documentation: https://googlemaps.github.io/google-maps-services-python/docs/index.html#googlemaps.Client.places 167 | 168 | 169 | 170 | 171 | 172 | -------------------------------------------------------------------------------- /dev/query_decomposition/MS-SQ.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "id": "857c5be0-9b2c-438e-8f05-6246f5946204", 7 | "metadata": {}, 8 | "outputs": [], 9 | "source": [ 10 | "%pip install llama-index" 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": null, 16 | "id": "621096b6-7eb7-4aab-a231-ba464731a615", 17 | "metadata": {}, 18 | "outputs": [], 19 | "source": [ 20 | "%pip install llama-index-llms-openai" 21 | ] 22 | }, 23 | { 24 | "cell_type": "code", 25 | "execution_count": null, 26 | "id": "b1f054d5-9704-4e56-bd8d-9c58235c28e5", 27 | "metadata": {}, 28 | "outputs": [], 29 | "source": [ 30 | "import os\n", 31 | "\n", 32 | "os.environ[\"OPENAI_API_KEY\"] = \"sk-\"\n", 33 | "\n", 34 | "import nest_asyncio\n", 35 | "\n", 36 | "nest_asyncio.apply()" 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "execution_count": null, 42 | "id": "0e9b4ccd-c6af-44d1-9726-b8fdaa40eac0", 43 | "metadata": {}, 44 | "outputs": [], 45 | "source": [ 46 | "from llama_index.core import VectorStoreIndex, SimpleDirectoryReader\n", 47 | "from llama_index.core.tools import QueryEngineTool, ToolMetadata\n", 48 | "from llama_index.core.query_engine import SubQuestionQueryEngine\n", 49 | "from llama_index.core.callbacks import CallbackManager, LlamaDebugHandler\n", 50 | "from llama_index.core import Settings\n", 51 | "from llama_index.llms.openai import OpenAI\n", 52 | "from IPython.display import Markdown, display" 53 | ] 54 | }, 55 | { 56 | "cell_type": "code", 57 | "execution_count": null, 58 | "id": "9f04ddb5-6dcd-4f1b-acdf-eeb78d87b380", 59 | "metadata": {}, 60 | "outputs": [], 61 | "source": [ 62 | "# LLM (gpt-3.5)\n", 63 | "gpt35 = OpenAI(temperature=0, model=\"gpt-3.5-turbo\")\n", 64 | "\n", 65 | "# LLM (gpt-4)\n", 66 | "gpt4 = OpenAI(temperature=0, model=\"gpt-4\")" 67 | ] 68 | }, 69 | { 70 | "cell_type": "code", 71 | "execution_count": null, 72 | "id": "9e426703-b6cb-4932-8e41-05f3568a1d5b", 73 | "metadata": {}, 74 | "outputs": [], 75 | "source": [ 76 | "%mkdir -p 'data/paul_graham/'\n", 77 | "%wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'" 78 | ] 79 | }, 80 | { 81 | "cell_type": "code", 82 | "execution_count": null, 83 | "id": "5878aa8e-5dd1-438c-a597-9c751468e93a", 84 | "metadata": {}, 85 | "outputs": [], 86 | "source": [ 87 | "# load documents\n", 88 | "documents = SimpleDirectoryReader(\"./data/paul_graham/\").load_data()" 89 | ] 90 | }, 91 | { 92 | "cell_type": "code", 93 | "execution_count": null, 94 | "id": "a2414fdf-7304-495f-b91e-4c8e881adc7e", 95 | "metadata": {}, 96 | "outputs": [], 97 | "source": [ 98 | "# Using the LlamaDebugHandler to print the trace of the sub questions\n", 99 | "# captured by the SUB_QUESTION callback event type\n", 100 | "llama_debug = LlamaDebugHandler(print_trace_on_end=True)\n", 101 | "callback_manager = CallbackManager([llama_debug])\n", 102 | "\n", 103 | "Settings.callback_manager = callback_manager" 104 | ] 105 | }, 106 | { 107 | "cell_type": "code", 108 | "execution_count": null, 109 | "id": "3699ad64-acb3-4322-8fb1-0c1e59eaa53a", 110 | "metadata": {}, 111 | "outputs": [], 112 | "source": [ 113 | "index = VectorStoreIndex.from_documents(documents)" 114 | ] 115 | }, 116 | { 117 | "cell_type": "code", 118 | "execution_count": null, 119 | "id": "f4a9b8ac", 120 | "metadata": {}, 121 | "outputs": [], 122 | "source": [ 123 | "from llama_index.core.indices.query.query_transform.base import (\n", 124 | " StepDecomposeQueryTransform,\n", 125 | ")\n", 126 | "\n", 127 | "# gpt-4\n", 128 | "step_decompose_transform = StepDecomposeQueryTransform(llm=gpt4, verbose=True)\n", 129 | "\n", 130 | "# gpt-3\n", 131 | "step_decompose_transform_gpt3 = StepDecomposeQueryTransform(\n", 132 | " llm=gpt35, verbose=True\n", 133 | ")" 134 | ] 135 | }, 136 | { 137 | "cell_type": "code", 138 | "execution_count": null, 139 | "id": "e5808c79", 140 | "metadata": {}, 141 | "outputs": [], 142 | "source": [ 143 | "index_summary = \"Used to answer questions about the author\"" 144 | ] 145 | }, 146 | { 147 | "cell_type": "code", 148 | "execution_count": null, 149 | "id": "3219ee9e", 150 | "metadata": {}, 151 | "outputs": [], 152 | "source": [ 153 | "# set Logging to DEBUG for more detailed outputs\n", 154 | "from llama_index.core.query_engine import MultiStepQueryEngine\n", 155 | "\n", 156 | "query_engine = index.as_query_engine(llm=gpt4)\n", 157 | "query_engine = MultiStepQueryEngine(\n", 158 | " query_engine=query_engine,\n", 159 | " query_transform=step_decompose_transform,\n", 160 | " index_summary=index_summary,\n", 161 | ")\n", 162 | "response_gpt4 = query_engine.query(\n", 163 | " \"What is the article about?\", \n", 164 | ")" 165 | ] 166 | }, 167 | { 168 | "cell_type": "code", 169 | "execution_count": null, 170 | "id": "4effe830", 171 | "metadata": {}, 172 | "outputs": [], 173 | "source": [ 174 | "# extract the most new query" 175 | ] 176 | }, 177 | { 178 | "cell_type": "code", 179 | "execution_count": null, 180 | "id": "51ac81e7", 181 | "metadata": {}, 182 | "outputs": [], 183 | "source": [ 184 | "# build index and query engine\n", 185 | "vector_query_engine = VectorStoreIndex.from_documents(\n", 186 | " documents,\n", 187 | " use_async=True,\n", 188 | ").as_query_engine()" 189 | ] 190 | }, 191 | { 192 | "cell_type": "code", 193 | "execution_count": null, 194 | "id": "2c44f8cc", 195 | "metadata": {}, 196 | "outputs": [], 197 | "source": [ 198 | "# setup base query engine as tool\n", 199 | "query_engine_tools = [\n", 200 | " QueryEngineTool(\n", 201 | " query_engine=vector_query_engine,\n", 202 | " metadata=ToolMetadata(\n", 203 | " name=\"documents\",\n", 204 | " description=\"Paul Graham essay on What I Worked On\",\n", 205 | " ),\n", 206 | " ),\n", 207 | "]\n", 208 | "\n", 209 | "query_engine = SubQuestionQueryEngine.from_defaults(\n", 210 | " query_engine_tools=query_engine_tools,\n", 211 | " use_async=True,\n", 212 | ")" 213 | ] 214 | }, 215 | { 216 | "cell_type": "code", 217 | "execution_count": null, 218 | "id": "427424fe", 219 | "metadata": {}, 220 | "outputs": [], 221 | "source": [ 222 | "response = query_engine.query(\n", 223 | " # instead of this, use extracted query \n", 224 | " #To Do: reference newest query here \n", 225 | " \"What is the specific topic of this article by Paul Graham?\"\n", 226 | ")" 227 | ] 228 | }, 229 | { 230 | "cell_type": "code", 231 | "execution_count": null, 232 | "id": "41c2c713", 233 | "metadata": {}, 234 | "outputs": [], 235 | "source": [ 236 | "# iterate through sub_question items captured in SUB_QUESTION event\n", 237 | "from llama_index.core.callbacks import CBEventType, EventPayload\n", 238 | "\n", 239 | "for i, (start_event, end_event) in enumerate(\n", 240 | " llama_debug.get_event_pairs(CBEventType.SUB_QUESTION)\n", 241 | "):\n", 242 | " qa_pair = end_event.payload[EventPayload.SUB_QUESTION]\n", 243 | " print(\"Sub Question \" + str(i) + \": \" + qa_pair.sub_q.sub_question.strip())\n", 244 | " print(\"Answer: \" + qa_pair.answer.strip())\n", 245 | " print(\"====================================\")" 246 | ] 247 | }, 248 | { 249 | "cell_type": "code", 250 | "execution_count": null, 251 | "id": "33ba9435", 252 | "metadata": {}, 253 | "outputs": [], 254 | "source": [ 255 | "print(response)" 256 | ] 257 | } 258 | ], 259 | "metadata": { 260 | "kernelspec": { 261 | "display_name": "Python 3 (ipykernel)", 262 | "language": "python", 263 | "name": "python3" 264 | }, 265 | "language_info": { 266 | "codemirror_mode": { 267 | "name": "ipython", 268 | "version": 3 269 | }, 270 | "file_extension": ".py", 271 | "mimetype": "text/x-python", 272 | "name": "python", 273 | "nbconvert_exporter": "python", 274 | "pygments_lexer": "ipython3", 275 | "version": "3.11.8" 276 | } 277 | }, 278 | "nbformat": 4, 279 | "nbformat_minor": 5 280 | } 281 | -------------------------------------------------------------------------------- /dev/query_decomposition/MS_finetuning.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "from llama_index.core.agent import ReActAgent\n", 10 | "from llama_index.llms.openai import OpenAI\n", 11 | "from llama_index.core.tools import QueryEngineTool, ToolMetadata\n", 12 | "from llama_index.core import VectorStoreIndex, SimpleDirectoryReader" 13 | ] 14 | }, 15 | { 16 | "cell_type": "code", 17 | "execution_count": null, 18 | "metadata": {}, 19 | "outputs": [], 20 | "source": [ 21 | "llm_35 = OpenAI(model=\"gpt-3.5-turbo-0613\")\n", 22 | "llm_4 = OpenAI(model=\"gpt-4-0613\")\n", 23 | "# base_agent = ReActAgent.from_tools(query_engine_tools, llm=llm_35, verbose=True)" 24 | ] 25 | }, 26 | { 27 | "cell_type": "code", 28 | "execution_count": null, 29 | "metadata": {}, 30 | "outputs": [], 31 | "source": [ 32 | "from llama_index.core.indices.query.query_transform.base import (\n", 33 | " StepDecomposeQueryTransform,\n", 34 | ")\n", 35 | "\n", 36 | "# gpt-4\n", 37 | "step_decompose_transform = StepDecomposeQueryTransform(llm=llm_4, verbose=True)\n", 38 | "\n", 39 | "# gpt-3\n", 40 | "step_decompose_transform_gpt3 = StepDecomposeQueryTransform(\n", 41 | " llm=llm_35, verbose=True\n", 42 | ")" 43 | ] 44 | }, 45 | { 46 | "cell_type": "code", 47 | "execution_count": null, 48 | "metadata": {}, 49 | "outputs": [], 50 | "source": [ 51 | "documents = SimpleDirectoryReader(\"./query-decomposition/data/filler\").load_data()\n", 52 | "index = VectorStoreIndex.from_documents(documents, llm=llm_35, verbose=True)" 53 | ] 54 | }, 55 | { 56 | "cell_type": "code", 57 | "execution_count": null, 58 | "metadata": {}, 59 | "outputs": [], 60 | "source": [ 61 | "index_summary = \"Used to answer questions about the teacher, the students, and the classroom\"" 62 | ] 63 | }, 64 | { 65 | "cell_type": "code", 66 | "execution_count": null, 67 | "metadata": {}, 68 | "outputs": [], 69 | "source": [ 70 | "from llama_index.llms.openai import OpenAI\n", 71 | "from llama_index.finetuning.callbacks import OpenAIFineTuningHandler\n", 72 | "from llama_index.core.callbacks import CallbackManager\n", 73 | "from llama_index.core.agent import ReActAgent\n", 74 | "\n", 75 | "finetuning_handler = OpenAIFineTuningHandler()\n", 76 | "callback_manager = CallbackManager([finetuning_handler])\n", 77 | "\n", 78 | "from llama_index.core import Settings\n", 79 | "\n", 80 | "# limit the context window artifically to test refine process\n", 81 | "Settings.context_window = 2048" 82 | ] 83 | }, 84 | { 85 | "cell_type": "code", 86 | "execution_count": null, 87 | "metadata": {}, 88 | "outputs": [], 89 | "source": [ 90 | "# set Logging to DEBUG for more detailed outputs\n", 91 | "from llama_index.core.query_engine import MultiStepQueryEngine\n", 92 | "\n", 93 | "query_engine = index.as_query_engine(llm=llm_4)\n", 94 | "query_engine = MultiStepQueryEngine(\n", 95 | " query_engine=query_engine,\n", 96 | " query_transform=step_decompose_transform,\n", 97 | " index_summary=index_summary,\n", 98 | ")\n", 99 | "\n", 100 | "query_engine.callback_manager = callback_manager\n", 101 | "query_engine.callback_manager.add_handler(finetuning_handler)\n", 102 | "query_engine.callback_manager.start_trace()" 103 | ] 104 | }, 105 | { 106 | "cell_type": "code", 107 | "execution_count": null, 108 | "metadata": {}, 109 | "outputs": [], 110 | "source": [ 111 | "import json\n", 112 | "\n", 113 | "file_path = './query-decomposition/cv_loc_enhanced_questions.json'\n", 114 | "with open(file_path, 'r') as file_loc:\n", 115 | " data = json.load(file_loc)\n", 116 | "\n", 117 | "new_questions = data" 118 | ] 119 | }, 120 | { 121 | "cell_type": "code", 122 | "execution_count": null, 123 | "metadata": {}, 124 | "outputs": [], 125 | "source": [ 126 | "def save_questions(questions, path):\n", 127 | " with open(path, \"w\") as f:\n", 128 | " for question in questions:\n", 129 | " f.write(question + \"\\n\")\n", 130 | " \n", 131 | "def load_questions(path):\n", 132 | " questions = []\n", 133 | " with open(path, \"r\") as f:\n", 134 | " for line in f:\n", 135 | " questions.append(line.strip())\n", 136 | " return questions" 137 | ] 138 | }, 139 | { 140 | "cell_type": "code", 141 | "execution_count": null, 142 | "metadata": {}, 143 | "outputs": [], 144 | "source": [ 145 | "# TODO: split up into cv and loc questions, also train on each possible set of eval/test split\n", 146 | "train_questions, eval_questions = new_questions[:60], new_questions[60:]" 147 | ] 148 | }, 149 | { 150 | "cell_type": "code", 151 | "execution_count": null, 152 | "metadata": {}, 153 | "outputs": [], 154 | "source": [ 155 | "save_questions(train_questions, \"train_questions_10q.txt\")\n", 156 | "save_questions(eval_questions, \"eval_questions_10q.txt\")" 157 | ] 158 | }, 159 | { 160 | "cell_type": "code", 161 | "execution_count": null, 162 | "metadata": {}, 163 | "outputs": [], 164 | "source": [ 165 | "train_questions = load_questions(\"train_questions_10q.txt\")\n", 166 | "eval_questions = load_questions(\"eval_questions_10q.txt\")" 167 | ] 168 | }, 169 | { 170 | "cell_type": "code", 171 | "execution_count": null, 172 | "metadata": {}, 173 | "outputs": [], 174 | "source": [ 175 | "\n", 176 | "query_engine_tools = [\n", 177 | " QueryEngineTool(\n", 178 | " query_engine=query_engine,\n", 179 | " metadata=ToolMetadata(\n", 180 | " name=\"documents\",\n", 181 | " description=\"Query engine that decomposes one complex query into its parts\",\n", 182 | " ),\n", 183 | " ),\n", 184 | "]" 185 | ] 186 | }, 187 | { 188 | "cell_type": "code", 189 | "execution_count": null, 190 | "metadata": {}, 191 | "outputs": [], 192 | "source": [ 193 | "llm = OpenAI(model=\"gpt-4-0613\")\n", 194 | "gpt4_agent = ReActAgent.from_tools(\n", 195 | " query_engine_tools,\n", 196 | " llm=llm,\n", 197 | " callback_manager=callback_manager,\n", 198 | " verbose=True,\n", 199 | ")" 200 | ] 201 | }, 202 | { 203 | "cell_type": "markdown", 204 | "metadata": {}, 205 | "source": [ 206 | "Change Index of train_questions[:40]" 207 | ] 208 | }, 209 | { 210 | "cell_type": "code", 211 | "execution_count": null, 212 | "metadata": {}, 213 | "outputs": [], 214 | "source": [ 215 | "responses = []\n", 216 | "for idx, question in enumerate(train_questions[41:]):\n", 217 | " print(f\"[{idx}] Question: {question}\")\n", 218 | " response = query_engine.query(question)\n", 219 | " responses.append(response)\n", 220 | " print(f\"[{idx}] Response: {str(response)}\")\n", 221 | "\n", 222 | "# question = train_questions[0]\n", 223 | "# response_gpt4 = query_engine.query(\n", 224 | "# \"Is the teacher standing in front of the classroom while giving a lecture or conducting an activity?\",\n", 225 | "# )" 226 | ] 227 | }, 228 | { 229 | "cell_type": "code", 230 | "execution_count": null, 231 | "metadata": {}, 232 | "outputs": [], 233 | "source": [ 234 | "# query_engine.callback_manager.end_trace()" 235 | ] 236 | }, 237 | { 238 | "cell_type": "code", 239 | "execution_count": null, 240 | "metadata": {}, 241 | "outputs": [], 242 | "source": [ 243 | "for response in responses: \n", 244 | " sub_qa = response.metadata[\"sub_qa\"]\n", 245 | " tuples = [(t[0], t[1]) for t in sub_qa]\n", 246 | " if len(tuples) > 0:\n", 247 | " latest_tuple = tuples[-1]\n", 248 | " latest_query = latest_tuple[0]\n", 249 | " print(latest_query)" 250 | ] 251 | }, 252 | { 253 | "cell_type": "code", 254 | "execution_count": null, 255 | "metadata": {}, 256 | "outputs": [], 257 | "source": [ 258 | "finetuning_handler.save_finetuning_events(\"finetuning_events_10q.jsonl\")" 259 | ] 260 | }, 261 | { 262 | "cell_type": "code", 263 | "execution_count": null, 264 | "metadata": {}, 265 | "outputs": [], 266 | "source": [ 267 | "from llama_index.finetuning import OpenAIFinetuneEngine\n", 268 | "\n", 269 | "finetune_engine = OpenAIFinetuneEngine(\n", 270 | " \"gpt-3.5-turbo\",\n", 271 | " \"finetuning_events_10q.jsonl\",\n", 272 | " #start_job_id=\"\" # if you have an existing job, can specify id here\n", 273 | ")" 274 | ] 275 | }, 276 | { 277 | "cell_type": "code", 278 | "execution_count": null, 279 | "metadata": {}, 280 | "outputs": [], 281 | "source": [ 282 | "finetune_engine.finetune()" 283 | ] 284 | }, 285 | { 286 | "cell_type": "code", 287 | "execution_count": null, 288 | "metadata": {}, 289 | "outputs": [], 290 | "source": [ 291 | "ft_llm = finetune_engine.get_finetuned_model(temperature=0.0)" 292 | ] 293 | } 294 | ], 295 | "metadata": { 296 | "kernelspec": { 297 | "display_name": "base", 298 | "language": "python", 299 | "name": "python3" 300 | }, 301 | "language_info": { 302 | "codemirror_mode": { 303 | "name": "ipython", 304 | "version": 3 305 | }, 306 | "file_extension": ".py", 307 | "mimetype": "text/x-python", 308 | "name": "python", 309 | "nbconvert_exporter": "python", 310 | "pygments_lexer": "ipython3", 311 | "version": "3.11.5" 312 | } 313 | }, 314 | "nbformat": 4, 315 | "nbformat_minor": 2 316 | } 317 | -------------------------------------------------------------------------------- /dev/query_decomposition/SQ-MS.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "%pip install llama-index" 10 | ] 11 | }, 12 | { 13 | "cell_type": "code", 14 | "execution_count": null, 15 | "metadata": {}, 16 | "outputs": [], 17 | "source": [ 18 | "%pip install llama-index-llms-openai" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": null, 24 | "metadata": {}, 25 | "outputs": [], 26 | "source": [ 27 | "import os\n", 28 | "\n", 29 | "os.environ[\"OPENAI_API_KEY\"] = \"sk-\"\n", 30 | "\n", 31 | "import nest_asyncio\n", 32 | "nest_asyncio.apply()" 33 | ] 34 | }, 35 | { 36 | "cell_type": "code", 37 | "execution_count": null, 38 | "metadata": {}, 39 | "outputs": [], 40 | "source": [ 41 | "#copied imports from llama-index example for subquestion\n", 42 | "from llama_index.core import VectorStoreIndex, SimpleDirectoryReader\n", 43 | "from llama_index.core.tools import QueryEngineTool, ToolMetadata\n", 44 | "from llama_index.core.query_engine import SubQuestionQueryEngine\n", 45 | "from llama_index.core.callbacks import CallbackManager, LlamaDebugHandler\n", 46 | "from llama_index.core import Settings\n", 47 | "\n", 48 | "#copied imports from llama-index example for multistep\n", 49 | "from llama_index.llms.openai import OpenAI\n", 50 | "from IPython.display import Markdown, display" 51 | ] 52 | }, 53 | { 54 | "cell_type": "code", 55 | "execution_count": null, 56 | "metadata": {}, 57 | "outputs": [], 58 | "source": [ 59 | "# LLM (gpt-3.5)\n", 60 | "gpt35 = OpenAI(temperature=0, model=\"gpt-3.5-turbo\")\n", 61 | "\n", 62 | "# LLM (gpt-4)\n", 63 | "gpt4 = OpenAI(temperature=0, model=\"gpt-4\")" 64 | ] 65 | }, 66 | { 67 | "cell_type": "code", 68 | "execution_count": null, 69 | "metadata": {}, 70 | "outputs": [], 71 | "source": [ 72 | "# import os\n", 73 | "from llama_index.core import SimpleDirectoryReader\n", 74 | "import urllib.request\n", 75 | "\n", 76 | "os.makedirs('data/paul_graham/', exist_ok=True)\n", 77 | "\n", 78 | "# Download the file\n", 79 | "url = 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt'\n", 80 | "file_path = 'data/paul_graham/paul_graham_essay.txt'\n", 81 | "urllib.request.urlretrieve(url, file_path)\n", 82 | "\n", 83 | "# load documents\n", 84 | "pg_essay = SimpleDirectoryReader(\"./data/paul_graham/\").load_data()\n" 85 | ] 86 | }, 87 | { 88 | "cell_type": "code", 89 | "execution_count": null, 90 | "metadata": {}, 91 | "outputs": [], 92 | "source": [ 93 | "# Using the LlamaDebugHandler to print the trace of the sub questions\n", 94 | "# captured by the SUB_QUESTION callback event type\n", 95 | "llama_debug = LlamaDebugHandler(print_trace_on_end=True)\n", 96 | "callback_manager = CallbackManager([llama_debug])\n", 97 | "\n", 98 | "Settings.callback_manager = callback_manager" 99 | ] 100 | }, 101 | { 102 | "cell_type": "code", 103 | "execution_count": null, 104 | "metadata": {}, 105 | "outputs": [], 106 | "source": [ 107 | "index = VectorStoreIndex.from_documents(documents)" 108 | ] 109 | }, 110 | { 111 | "cell_type": "code", 112 | "execution_count": null, 113 | "metadata": {}, 114 | "outputs": [], 115 | "source": [ 116 | "# build index and query engine\n", 117 | "vector_query_engine = VectorStoreIndex.from_documents(\n", 118 | " pg_essay,\n", 119 | " use_async=True,\n", 120 | ").as_query_engine()" 121 | ] 122 | }, 123 | { 124 | "cell_type": "code", 125 | "execution_count": null, 126 | "metadata": {}, 127 | "outputs": [], 128 | "source": [ 129 | "# setup base query engine as tool\n", 130 | "query_engine_tools = [\n", 131 | " QueryEngineTool(\n", 132 | " query_engine=vector_query_engine,\n", 133 | " metadata=ToolMetadata(\n", 134 | " name=\"pg_essay\",\n", 135 | " description=\"Paul Graham essay on What I Worked On\",\n", 136 | " ),\n", 137 | " ),\n", 138 | "]\n", 139 | "\n", 140 | "query_engine = SubQuestionQueryEngine.from_defaults(\n", 141 | " query_engine_tools=query_engine_tools,\n", 142 | " use_async=True,\n", 143 | ")" 144 | ] 145 | }, 146 | { 147 | "cell_type": "code", 148 | "execution_count": null, 149 | "metadata": {}, 150 | "outputs": [], 151 | "source": [ 152 | "llama_debug.flush_event_logs() #flush for easier retrieval of subquestions\n", 153 | "\n", 154 | "response = query_engine.query(\n", 155 | " \"Summarize the essay, and discuss the authors life before and after the essay was written.\"\n", 156 | ") #todo: figure out how to stop it from printing responses" 157 | ] 158 | }, 159 | { 160 | "cell_type": "code", 161 | "execution_count": null, 162 | "metadata": {}, 163 | "outputs": [], 164 | "source": [ 165 | "from llama_index.core.callbacks import CBEventType\n", 166 | "\n", 167 | "event_pairs = llama_debug.get_event_pairs(CBEventType.SUB_QUESTION)\n", 168 | "\n", 169 | "sub_questions = []\n", 170 | "for pair in event_pairs:\n", 171 | " sub_q_object = str(pair[0].payload[\"sub_question\"])\n", 172 | " start_index = sub_q_object.find(\"sub_question=\")\n", 173 | " end_index = sub_q_object.find(\", tool_name=\")\n", 174 | " if start_index == -1 or end_index == -1:\n", 175 | " # smth wrong\n", 176 | " continue\n", 177 | "\n", 178 | " new_q = sub_q_object[start_index+14:end_index-1]\n", 179 | " sub_questions.append(new_q)\n", 180 | "\n", 181 | "# probably a better way to do this than to substring\n", 182 | "\n", 183 | " " 184 | ] 185 | }, 186 | { 187 | "cell_type": "code", 188 | "execution_count": null, 189 | "metadata": {}, 190 | "outputs": [], 191 | "source": [ 192 | "from llama_index.core.indices.query.query_transform.base import (\n", 193 | " StepDecomposeQueryTransform,\n", 194 | ")\n", 195 | "\n", 196 | "# gpt-4\n", 197 | "step_decompose_transform = StepDecomposeQueryTransform(llm=gpt4, verbose=True)\n", 198 | "\n", 199 | "# gpt-3\n", 200 | "step_decompose_transform_gpt3 = StepDecomposeQueryTransform(llm=gpt35, verbose=True)\n", 201 | "\n", 202 | "index_summary = \"Used to answer questions about the author\"" 203 | ] 204 | }, 205 | { 206 | "cell_type": "code", 207 | "execution_count": null, 208 | "metadata": {}, 209 | "outputs": [], 210 | "source": [ 211 | "from llama_index.core.query_engine import MultiStepQueryEngine\n", 212 | "\n", 213 | "mqe = VectorStoreIndex.from_documents(documents).as_query_engine(llm=gpt4)\n", 214 | "\n", 215 | "multi_query_engine = MultiStepQueryEngine(\n", 216 | " query_engine=mqe,\n", 217 | " query_transform=step_decompose_transform,\n", 218 | " index_summary=index_summary,\n", 219 | ")\n", 220 | "\n", 221 | "for sub_q in sub_questions:\n", 222 | " print(\"query: \", sub_q)\n", 223 | " response = multi_query_engine.query(sub_q)\n", 224 | " # print(response)" 225 | ] 226 | }, 227 | { 228 | "cell_type": "code", 229 | "execution_count": null, 230 | "metadata": {}, 231 | "outputs": [], 232 | "source": [] 233 | } 234 | ], 235 | "metadata": { 236 | "kernelspec": { 237 | "display_name": "Python 3", 238 | "language": "python", 239 | "name": "python3" 240 | }, 241 | "language_info": { 242 | "codemirror_mode": { 243 | "name": "ipython", 244 | "version": 3 245 | }, 246 | "file_extension": ".py", 247 | "mimetype": "text/x-python", 248 | "name": "python", 249 | "nbconvert_exporter": "python", 250 | "pygments_lexer": "ipython3", 251 | "version": "3.11.8" 252 | } 253 | }, 254 | "nbformat": 4, 255 | "nbformat_minor": 2 256 | } 257 | -------------------------------------------------------------------------------- /dev/query_decomposition/langchain_ms.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [ 8 | { 9 | "name": "stdout", 10 | "output_type": "stream", 11 | "text": [ 12 | "Note: you may need to restart the kernel to use updated packages.\n" 13 | ] 14 | } 15 | ], 16 | "source": [ 17 | "# %pip install -qU langchain langchain-openai" 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": null, 23 | "metadata": {}, 24 | "outputs": [], 25 | "source": [ 26 | "import getpass\n", 27 | "import os\n", 28 | "\n", 29 | "# os.environ[\"OPENAI_API_KEY\"] = getpass.getpass()\n", 30 | "\n", 31 | "# Optional, uncomment to trace runs with LangSmith. Sign up here: https://smith.langchain.com.\n", 32 | "# os.environ[\"LANGCHAIN_TRACING_V2\"] = \"true\"\n", 33 | "# os.environ[\"LANGCHAIN_API_KEY\"] = getpass.getpass()" 34 | ] 35 | }, 36 | { 37 | "cell_type": "code", 38 | "execution_count": 1, 39 | "metadata": {}, 40 | "outputs": [], 41 | "source": [ 42 | "import datetime\n", 43 | "from typing import Literal, Optional, Tuple\n", 44 | "\n", 45 | "from langchain_core.pydantic_v1 import BaseModel, Field\n", 46 | "\n", 47 | "\n", 48 | "class SubQuery(BaseModel):\n", 49 | " \"\"\"Search over a database of tutorial videos about a software library.\"\"\"\n", 50 | "\n", 51 | " sub_query: str = Field(\n", 52 | " ...,\n", 53 | " description=\"A very specific query against the database.\",\n", 54 | " )" 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": 2, 60 | "metadata": {}, 61 | "outputs": [], 62 | "source": [ 63 | "from langchain.output_parsers import PydanticToolsParser\n", 64 | "from langchain_core.prompts import ChatPromptTemplate\n", 65 | "from langchain_openai import ChatOpenAI\n", 66 | "\n", 67 | "system = \"\"\"You are an expert at converting user questions into database queries. \\\n", 68 | "You have access to a database of tutorial videos about a software library for building LLM-powered applications. \\\n", 69 | "\n", 70 | "Perform query decomposition. Given a user question, break it down into distinct sub questions that \\\n", 71 | "you need to answer in order to answer the original question.\n", 72 | "\n", 73 | "If there are acronyms or words you are not familiar with, do not try to rephrase them.\"\"\"\n", 74 | "prompt = ChatPromptTemplate.from_messages(\n", 75 | " [\n", 76 | " (\"system\", system),\n", 77 | " (\"human\", \"{question}\"),\n", 78 | " ]\n", 79 | ")\n", 80 | "llm = ChatOpenAI(model=\"gpt-3.5-turbo-0125\", temperature=0)\n", 81 | "llm_with_tools = llm.bind_tools([SubQuery])\n", 82 | "parser = PydanticToolsParser(tools=[SubQuery])\n", 83 | "query_analyzer = prompt | llm_with_tools | parser" 84 | ] 85 | }, 86 | { 87 | "cell_type": "code", 88 | "execution_count": 3, 89 | "metadata": {}, 90 | "outputs": [ 91 | { 92 | "data": { 93 | "text/plain": [ 94 | "[SubQuery(sub_query='How to do rag')]" 95 | ] 96 | }, 97 | "execution_count": 3, 98 | "metadata": {}, 99 | "output_type": "execute_result" 100 | } 101 | ], 102 | "source": [ 103 | "query_analyzer.invoke({\"question\": \"how to do rag\"})" 104 | ] 105 | }, 106 | { 107 | "cell_type": "code", 108 | "execution_count": 4, 109 | "metadata": {}, 110 | "outputs": [ 111 | { 112 | "data": { 113 | "text/plain": [ 114 | "[SubQuery(sub_query='How to use multi-modal models in a chain'),\n", 115 | " SubQuery(sub_query='How to turn a chain into a REST API')]" 116 | ] 117 | }, 118 | "execution_count": 4, 119 | "metadata": {}, 120 | "output_type": "execute_result" 121 | } 122 | ], 123 | "source": [ 124 | "query_analyzer.invoke(\n", 125 | " {\n", 126 | " \"question\": \"how to use multi-modal models in a chain and turn chain into a rest api\"\n", 127 | " }\n", 128 | ")" 129 | ] 130 | }, 131 | { 132 | "cell_type": "code", 133 | "execution_count": 5, 134 | "metadata": {}, 135 | "outputs": [], 136 | "source": [ 137 | "examples = []\n", 138 | "question = \"What's chat langchain, is it a langchain template?\"\n", 139 | "queries = [\n", 140 | " SubQuery(sub_query=\"What is chat langchain\"),\n", 141 | " SubQuery(sub_query=\"What is a langchain template\"),\n", 142 | "]\n", 143 | "examples.append({\"input\": question, \"tool_calls\": queries})" 144 | ] 145 | }, 146 | { 147 | "cell_type": "code", 148 | "execution_count": 6, 149 | "metadata": {}, 150 | "outputs": [], 151 | "source": [ 152 | "question = \"How would I use LangGraph to build an automaton\"\n", 153 | "queries = [\n", 154 | " SubQuery(sub_query=\"How to build automaton with LangGraph\"),\n", 155 | "]\n", 156 | "examples.append({\"input\": question, \"tool_calls\": queries})" 157 | ] 158 | }, 159 | { 160 | "cell_type": "code", 161 | "execution_count": 7, 162 | "metadata": {}, 163 | "outputs": [], 164 | "source": [ 165 | "question = \"How to build multi-agent system and stream intermediate steps from it\"\n", 166 | "queries = [\n", 167 | " SubQuery(sub_query=\"How to build multi-agent system\"),\n", 168 | " SubQuery(sub_query=\"How to stream intermediate steps\"),\n", 169 | " SubQuery(sub_query=\"How to stream intermediate steps from multi-agent system\"),\n", 170 | "]\n", 171 | "examples.append({\"input\": question, \"tool_calls\": queries})" 172 | ] 173 | }, 174 | { 175 | "cell_type": "code", 176 | "execution_count": 8, 177 | "metadata": {}, 178 | "outputs": [], 179 | "source": [ 180 | "question = \"What's the difference between LangChain agents and LangGraph?\"\n", 181 | "queries = [\n", 182 | " SubQuery(sub_query=\"What's the difference between LangChain agents and LangGraph?\"),\n", 183 | " SubQuery(sub_query=\"What are LangChain agents\"),\n", 184 | " SubQuery(sub_query=\"What is LangGraph\"),\n", 185 | "]\n", 186 | "examples.append({\"input\": question, \"tool_calls\": queries})" 187 | ] 188 | }, 189 | { 190 | "cell_type": "code", 191 | "execution_count": 9, 192 | "metadata": {}, 193 | "outputs": [], 194 | "source": [ 195 | "question = \"Is the teacher standing in front of the classroom while giving a lecture or conducting an activity?\"\n", 196 | "queries = [\n", 197 | " SubQuery(sub_query=\"Is the teacher standing in front of the classroom?\"),\n", 198 | " SubQuery(sub_query=\"Is the teacher giving a lecture or conducting an activity?\"),\n", 199 | "]\n", 200 | "\n", 201 | "examples.append({\"input\": question, \"tool_calls\": queries})" 202 | ] 203 | }, 204 | { 205 | "cell_type": "markdown", 206 | "metadata": {}, 207 | "source": [ 208 | "Update prompt templates" 209 | ] 210 | }, 211 | { 212 | "cell_type": "code", 213 | "execution_count": 10, 214 | "metadata": {}, 215 | "outputs": [], 216 | "source": [ 217 | "import uuid\n", 218 | "from typing import Dict, List\n", 219 | "\n", 220 | "from langchain_core.messages import (\n", 221 | " AIMessage,\n", 222 | " BaseMessage,\n", 223 | " HumanMessage,\n", 224 | " SystemMessage,\n", 225 | " ToolMessage,\n", 226 | ")\n", 227 | "\n", 228 | "\n", 229 | "def tool_example_to_messages(example: Dict) -> List[BaseMessage]:\n", 230 | " messages: List[BaseMessage] = [HumanMessage(content=example[\"input\"])]\n", 231 | " openai_tool_calls = []\n", 232 | " for tool_call in example[\"tool_calls\"]:\n", 233 | " openai_tool_calls.append(\n", 234 | " {\n", 235 | " \"id\": str(uuid.uuid4()),\n", 236 | " \"type\": \"function\",\n", 237 | " \"function\": {\n", 238 | " \"name\": tool_call.__class__.__name__,\n", 239 | " \"arguments\": tool_call.json(),\n", 240 | " },\n", 241 | " }\n", 242 | " )\n", 243 | " messages.append(\n", 244 | " AIMessage(content=\"\", additional_kwargs={\"tool_calls\": openai_tool_calls})\n", 245 | " )\n", 246 | " tool_outputs = example.get(\"tool_outputs\") or [\n", 247 | " \"This is an example of a correct usage of this tool. Make sure to continue using the tool this way.\"\n", 248 | " ] * len(openai_tool_calls)\n", 249 | " for output, tool_call in zip(tool_outputs, openai_tool_calls):\n", 250 | " messages.append(ToolMessage(content=output, tool_call_id=tool_call[\"id\"]))\n", 251 | " return messages\n", 252 | "\n", 253 | "\n", 254 | "example_msgs = [msg for ex in examples for msg in tool_example_to_messages(ex)]" 255 | ] 256 | }, 257 | { 258 | "cell_type": "code", 259 | "execution_count": 11, 260 | "metadata": {}, 261 | "outputs": [], 262 | "source": [ 263 | "from langchain_core.prompts import MessagesPlaceholder\n", 264 | "\n", 265 | "system = \"\"\"You are an expert at converting user questions into database queries. \\\n", 266 | "You have access to a database of tutorial videos about a software library for building LLM-powered applications. \\\n", 267 | "\n", 268 | "Perform query decomposition. Given a user question, break it down into the most specific sub questions you can \\\n", 269 | "which will help you answer the original question. Each sub question should be about a single concept/fact/idea.\n", 270 | "\n", 271 | "If there are acronyms or words you are not familiar with, do not try to rephrase them.\"\"\"\n", 272 | "prompt = ChatPromptTemplate.from_messages(\n", 273 | " [\n", 274 | " (\"system\", system),\n", 275 | " MessagesPlaceholder(\"examples\", optional=True),\n", 276 | " (\"human\", \"{question}\"),\n", 277 | " ]\n", 278 | ")\n", 279 | "query_analyzer_with_examples = (\n", 280 | " prompt.partial(examples=example_msgs) | llm_with_tools | parser\n", 281 | ")" 282 | ] 283 | }, 284 | { 285 | "cell_type": "code", 286 | "execution_count": 12, 287 | "metadata": {}, 288 | "outputs": [ 289 | { 290 | "data": { 291 | "text/plain": [ 292 | "[SubQuery(sub_query=\"What's the difference between Web Voyager and Reflection Agents?\"),\n", 293 | " SubQuery(sub_query='Do Web Voyager and Reflection Agents use LangGraph?')]" 294 | ] 295 | }, 296 | "execution_count": 12, 297 | "metadata": {}, 298 | "output_type": "execute_result" 299 | } 300 | ], 301 | "source": [ 302 | "query_analyzer_with_examples.invoke(\n", 303 | " {\n", 304 | " \"question\": \"what's the difference between web voyager and reflection agents? do they use langgraph?\"\n", 305 | " }\n", 306 | ")" 307 | ] 308 | }, 309 | { 310 | "cell_type": "code", 311 | "execution_count": null, 312 | "metadata": {}, 313 | "outputs": [], 314 | "source": [] 315 | } 316 | ], 317 | "metadata": { 318 | "kernelspec": { 319 | "display_name": "base", 320 | "language": "python", 321 | "name": "python3" 322 | }, 323 | "language_info": { 324 | "codemirror_mode": { 325 | "name": "ipython", 326 | "version": 3 327 | }, 328 | "file_extension": ".py", 329 | "mimetype": "text/x-python", 330 | "name": "python", 331 | "nbconvert_exporter": "python", 332 | "pygments_lexer": "ipython3", 333 | "version": "3.11.5" 334 | } 335 | }, 336 | "nbformat": 4, 337 | "nbformat_minor": 2 338 | } 339 | -------------------------------------------------------------------------------- /dev/query_decomposition/routing_and_multistep.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "5e0768d5", 6 | "metadata": { 7 | "id": "5e0768d5" 8 | }, 9 | "source": [ 10 | "\"Open" 11 | ] 12 | }, 13 | { 14 | "cell_type": "markdown", 15 | "id": "9c48213d-6e6a-4c10-838a-2a7c710c3a05", 16 | "metadata": { 17 | "id": "9c48213d-6e6a-4c10-838a-2a7c710c3a05" 18 | }, 19 | "source": [ 20 | "# Multi-Step Query Engine\n", 21 | "\n", 22 | "We have a multi-step query engine that's able to decompose a complex query into sequential subquestions. This\n", 23 | "guide walks you through how to set it up!" 24 | ] 25 | }, 26 | { 27 | "cell_type": "markdown", 28 | "id": "cc77f986", 29 | "metadata": { 30 | "id": "cc77f986" 31 | }, 32 | "source": [ 33 | "If you're opening this Notebook on colab, you will probably need to install LlamaIndex 🦙." 34 | ] 35 | }, 36 | { 37 | "cell_type": "code", 38 | "execution_count": null, 39 | "id": "20a71098", 40 | "metadata": { 41 | "colab": { 42 | "base_uri": "https://localhost:8080/" 43 | }, 44 | "id": "20a71098", 45 | "outputId": "e001a673-d277-4839-bdec-d3433f035956" 46 | }, 47 | "outputs": [], 48 | "source": [ 49 | "%pip install llama-index-llms-openai\n", 50 | "%pip install llama-index-agent-openai\n", 51 | "!pip install llama-index" 52 | ] 53 | }, 54 | { 55 | "cell_type": "markdown", 56 | "id": "c5885d63", 57 | "metadata": { 58 | "id": "c5885d63" 59 | }, 60 | "source": [ 61 | "#### Download Data" 62 | ] 63 | }, 64 | { 65 | "cell_type": "code", 66 | "execution_count": null, 67 | "id": "ee4ace96", 68 | "metadata": { 69 | "colab": { 70 | "base_uri": "https://localhost:8080/" 71 | }, 72 | "id": "ee4ace96", 73 | "outputId": "0963ba23-3afc-419f-f148-f22630d18cbf" 74 | }, 75 | "outputs": [], 76 | "source": [ 77 | "!mkdir -p 'data/paul_graham/'\n", 78 | "!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'" 79 | ] 80 | }, 81 | { 82 | "cell_type": "markdown", 83 | "id": "50d3b817-b70e-4667-be4f-d3a0fe4bd119", 84 | "metadata": { 85 | "id": "50d3b817-b70e-4667-be4f-d3a0fe4bd119" 86 | }, 87 | "source": [ 88 | "#### Load documents, build the VectorStoreIndex" 89 | ] 90 | }, 91 | { 92 | "cell_type": "code", 93 | "execution_count": null, 94 | "id": "65ebfca2", 95 | "metadata": { 96 | "id": "65ebfca2" 97 | }, 98 | "outputs": [], 99 | "source": [ 100 | "import os\n", 101 | "\n", 102 | "os.environ[\"OPENAI_API_KEY\"] = \"sk-...\"" 103 | ] 104 | }, 105 | { 106 | "cell_type": "code", 107 | "execution_count": null, 108 | "id": "690a6918-7c75-4f95-9ccc-d2c4a1fe00d7", 109 | "metadata": { 110 | "id": "690a6918-7c75-4f95-9ccc-d2c4a1fe00d7" 111 | }, 112 | "outputs": [], 113 | "source": [ 114 | "from llama_index.core import VectorStoreIndex, SimpleDirectoryReader\n", 115 | "from llama_index.llms.openai import OpenAI\n", 116 | "from IPython.display import Markdown, display" 117 | ] 118 | }, 119 | { 120 | "cell_type": "code", 121 | "execution_count": null, 122 | "id": "c48da73f-aadb-480c-8db1-99c915b7cc1c", 123 | "metadata": { 124 | "id": "c48da73f-aadb-480c-8db1-99c915b7cc1c" 125 | }, 126 | "outputs": [], 127 | "source": [ 128 | "# LLM (gpt-3.5)\n", 129 | "gpt35 = OpenAI(temperature=0, model=\"gpt-3.5-turbo\")\n", 130 | "\n", 131 | "# LLM (gpt-4)\n", 132 | "gpt4 = OpenAI(temperature=0, model=\"gpt-4\")" 133 | ] 134 | }, 135 | { 136 | "cell_type": "code", 137 | "execution_count": null, 138 | "id": "03d1691e-544b-454f-825b-5ee12f7faa8a", 139 | "metadata": { 140 | "id": "03d1691e-544b-454f-825b-5ee12f7faa8a" 141 | }, 142 | "outputs": [], 143 | "source": [ 144 | "# load documents\n", 145 | "documents = SimpleDirectoryReader(\"./data/paul_graham/\").load_data()" 146 | ] 147 | }, 148 | { 149 | "cell_type": "code", 150 | "execution_count": null, 151 | "id": "ad144ee7-96da-4dd6-be00-fd6cf0c78e58", 152 | "metadata": { 153 | "id": "ad144ee7-96da-4dd6-be00-fd6cf0c78e58" 154 | }, 155 | "outputs": [], 156 | "source": [ 157 | "index = VectorStoreIndex.from_documents(documents)" 158 | ] 159 | }, 160 | { 161 | "cell_type": "markdown", 162 | "id": "b6caf93b-6345-4c65-a346-a95b0f1746c4", 163 | "metadata": { 164 | "id": "b6caf93b-6345-4c65-a346-a95b0f1746c4" 165 | }, 166 | "source": [ 167 | "#### Query Index" 168 | ] 169 | }, 170 | { 171 | "cell_type": "code", 172 | "execution_count": null, 173 | "id": "95d989ba-0c1d-43b6-a1d3-0ea7135f43a6", 174 | "metadata": { 175 | "id": "95d989ba-0c1d-43b6-a1d3-0ea7135f43a6" 176 | }, 177 | "outputs": [], 178 | "source": [ 179 | "from llama_index.core.indices.query.query_transform.base import (\n", 180 | " StepDecomposeQueryTransform,\n", 181 | ")\n", 182 | "\n", 183 | "# gpt-4\n", 184 | "step_decompose_transform = StepDecomposeQueryTransform(llm=gpt4, verbose=True)\n", 185 | "\n", 186 | "# gpt-3\n", 187 | "step_decompose_transform_gpt3 = StepDecomposeQueryTransform(\n", 188 | " llm=gpt35, verbose=True\n", 189 | ")" 190 | ] 191 | }, 192 | { 193 | "cell_type": "code", 194 | "execution_count": null, 195 | "id": "2a124db0-e2d7-4566-bcec-1d41cf669ff4", 196 | "metadata": { 197 | "id": "2a124db0-e2d7-4566-bcec-1d41cf669ff4" 198 | }, 199 | "outputs": [], 200 | "source": [ 201 | "index_summary = \"Used to answer questions about the author\"" 202 | ] 203 | }, 204 | { 205 | "cell_type": "code", 206 | "execution_count": null, 207 | "id": "85466fdf-93f3-4cb1-a5f9-0056a8245a6f", 208 | "metadata": { 209 | "colab": { 210 | "base_uri": "https://localhost:8080/" 211 | }, 212 | "id": "85466fdf-93f3-4cb1-a5f9-0056a8245a6f", 213 | "outputId": "1745877d-95f3-4266-a8e0-118ae21b09eb" 214 | }, 215 | "outputs": [], 216 | "source": [ 217 | "# set Logging to DEBUG for more detailed outputs\n", 218 | "from llama_index.core.query_engine import MultiStepQueryEngine\n", 219 | "\n", 220 | "query_engine = index.as_query_engine(llm=gpt4)\n", 221 | "query_engine = MultiStepQueryEngine(\n", 222 | " query_engine=query_engine,\n", 223 | " query_transform=step_decompose_transform,\n", 224 | " index_summary=index_summary,\n", 225 | ")\n", 226 | "response_gpt4 = query_engine.query(\n", 227 | " \"Who was in the first batch of the accelerator program the author\"\n", 228 | " \" started?\",\n", 229 | ")" 230 | ] 231 | }, 232 | { 233 | "cell_type": "code", 234 | "execution_count": null, 235 | "id": "7MeeUrf1xQcl", 236 | "metadata": { 237 | "colab": { 238 | "base_uri": "https://localhost:8080/" 239 | }, 240 | "id": "7MeeUrf1xQcl", 241 | "outputId": "98ba7f16-ddf7-4aa2-d7f1-58f14e3e1783" 242 | }, 243 | "outputs": [], 244 | "source": [ 245 | "sub_qa = response_gpt4.metadata[\"sub_qa\"]\n", 246 | "tuples = [(t[0], t[1].response) for t in sub_qa]\n", 247 | "print(tuples)" 248 | ] 249 | }, 250 | { 251 | "cell_type": "code", 252 | "execution_count": null, 253 | "id": "sA96XR4qxtYh", 254 | "metadata": { 255 | "colab": { 256 | "base_uri": "https://localhost:8080/" 257 | }, 258 | "id": "sA96XR4qxtYh", 259 | "outputId": "1763d5cf-f53e-4dfc-e761-ac392b7bcb9e" 260 | }, 261 | "outputs": [], 262 | "source": [ 263 | "latest_tuple = tuples[-1]\n", 264 | "latest_query = latest_tuple[0]\n", 265 | "print(latest_query)" 266 | ] 267 | }, 268 | { 269 | "cell_type": "markdown", 270 | "id": "iRiYf_rJy5Vb", 271 | "metadata": { 272 | "id": "iRiYf_rJy5Vb" 273 | }, 274 | "source": [ 275 | "New Technique" 276 | ] 277 | }, 278 | { 279 | "cell_type": "code", 280 | "execution_count": null, 281 | "id": "X2-aHSfByGyw", 282 | "metadata": { 283 | "colab": { 284 | "base_uri": "https://localhost:8080/" 285 | }, 286 | "id": "X2-aHSfByGyw", 287 | "outputId": "0b597cfc-7fe3-4270-8f44-1a36bf2fa527" 288 | }, 289 | "outputs": [], 290 | "source": [ 291 | "%load_ext autoreload\n", 292 | "%autoreload 2" 293 | ] 294 | }, 295 | { 296 | "cell_type": "code", 297 | "execution_count": null, 298 | "id": "NCsUXEuqyI2S", 299 | "metadata": { 300 | "id": "NCsUXEuqyI2S" 301 | }, 302 | "outputs": [], 303 | "source": [ 304 | "from llama_index.core import SimpleDirectoryReader, VectorStoreIndex\n", 305 | "from llama_index.core.response.pprint_utils import pprint_response\n", 306 | "from llama_index.llms.openai import OpenAI" 307 | ] 308 | }, 309 | { 310 | "cell_type": "code", 311 | "execution_count": null, 312 | "id": "MdBfe2CEyKl-", 313 | "metadata": { 314 | "id": "MdBfe2CEyKl-" 315 | }, 316 | "outputs": [], 317 | "source": [ 318 | "llm = OpenAI(temperature=0, model=\"gpt-4\")" 319 | ] 320 | }, 321 | { 322 | "cell_type": "code", 323 | "execution_count": null, 324 | "id": "ff_byYewyL5K", 325 | "metadata": { 326 | "id": "ff_byYewyL5K" 327 | }, 328 | "outputs": [], 329 | "source": [ 330 | "engine = index.as_query_engine(similarity_top_k=3, llm=llm)" 331 | ] 332 | }, 333 | { 334 | "cell_type": "code", 335 | "execution_count": null, 336 | "id": "kwjUemE5yTVB", 337 | "metadata": { 338 | "id": "kwjUemE5yTVB" 339 | }, 340 | "outputs": [], 341 | "source": [ 342 | "from llama_index.core.tools import QueryEngineTool\n", 343 | "\n", 344 | "\n", 345 | "query_tool_pg = QueryEngineTool.from_defaults(\n", 346 | " query_engine=engine,\n", 347 | " name=\"files\",\n", 348 | " description=(\n", 349 | " f\"Paul graham essay\"\n", 350 | " ),\n", 351 | ")" 352 | ] 353 | }, 354 | { 355 | "cell_type": "code", 356 | "execution_count": null, 357 | "id": "EZi7FVbwydNQ", 358 | "metadata": { 359 | "id": "EZi7FVbwydNQ" 360 | }, 361 | "outputs": [], 362 | "source": [ 363 | "# define query plan tool\n", 364 | "from llama_index.core.tools import QueryPlanTool\n", 365 | "from llama_index.core import get_response_synthesizer\n", 366 | "\n", 367 | "response_synthesizer = get_response_synthesizer()\n", 368 | "query_plan_tool = QueryPlanTool.from_defaults(\n", 369 | " query_engine_tools=[query_tool_pg],\n", 370 | " response_synthesizer=response_synthesizer,\n", 371 | ")" 372 | ] 373 | }, 374 | { 375 | "cell_type": "code", 376 | "execution_count": null, 377 | "id": "DV9tC4DQyexP", 378 | "metadata": { 379 | "colab": { 380 | "base_uri": "https://localhost:8080/" 381 | }, 382 | "id": "DV9tC4DQyexP", 383 | "outputId": "a72383bc-af3d-43da-eed1-9eaf746dd872" 384 | }, 385 | "outputs": [], 386 | "source": [ 387 | "query_plan_tool.metadata.to_openai_tool() # to_openai_function() deprecated" 388 | ] 389 | }, 390 | { 391 | "cell_type": "code", 392 | "execution_count": null, 393 | "id": "MPEwUMaAygE-", 394 | "metadata": { 395 | "id": "MPEwUMaAygE-" 396 | }, 397 | "outputs": [], 398 | "source": [ 399 | "from llama_index.agent.openai import OpenAIAgent\n", 400 | "from llama_index.llms.openai import OpenAI\n", 401 | "\n", 402 | "\n", 403 | "agent = OpenAIAgent.from_tools(\n", 404 | " [query_plan_tool],\n", 405 | " max_function_calls=10,\n", 406 | " llm=OpenAI(temperature=0, model=\"gpt-4-0613\"),\n", 407 | " verbose=True,\n", 408 | ")" 409 | ] 410 | }, 411 | { 412 | "cell_type": "code", 413 | "execution_count": null, 414 | "id": "HGEzQEupyh9a", 415 | "metadata": { 416 | "colab": { 417 | "base_uri": "https://localhost:8080/" 418 | }, 419 | "id": "HGEzQEupyh9a", 420 | "outputId": "00f83ff8-68b2-4c22-f683-1e2da834c26f" 421 | }, 422 | "outputs": [], 423 | "source": [ 424 | "response = agent.query(latest_query)" 425 | ] 426 | }, 427 | { 428 | "cell_type": "code", 429 | "execution_count": null, 430 | "id": "-IOyxIjpymvH", 431 | "metadata": { 432 | "colab": { 433 | "base_uri": "https://localhost:8080/" 434 | }, 435 | "id": "-IOyxIjpymvH", 436 | "outputId": "ca25e7a5-8488-4469-c1a0-7f5a32a2ae53" 437 | }, 438 | "outputs": [], 439 | "source": [ 440 | "print(str(response))" 441 | ] 442 | } 443 | ], 444 | "metadata": { 445 | "colab": { 446 | "provenance": [] 447 | }, 448 | "kernelspec": { 449 | "display_name": "llama", 450 | "language": "python", 451 | "name": "python3" 452 | }, 453 | "language_info": { 454 | "codemirror_mode": { 455 | "name": "ipython", 456 | "version": 3 457 | }, 458 | "file_extension": ".py", 459 | "mimetype": "text/x-python", 460 | "name": "python", 461 | "nbconvert_exporter": "python", 462 | "pygments_lexer": "ipython3" 463 | } 464 | }, 465 | "nbformat": 4, 466 | "nbformat_minor": 5 467 | } 468 | -------------------------------------------------------------------------------- /dev/recursive_retrieval/benchmarking/benchmark.py: -------------------------------------------------------------------------------- 1 | import os 2 | from collections import defaultdict 3 | from typing import List 4 | 5 | from dotenv import load_dotenv 6 | from llama_index.core.llms import ChatMessage 7 | from llama_index.llms.perplexity import Perplexity 8 | from pandas import DataFrame 9 | 10 | from judge import custom_response_evaluator 11 | 12 | 13 | def benchmark( 14 | model_name: str, 15 | data_points: List, 16 | conciseness: List, 17 | informativeness: List, 18 | accuracy: List, 19 | ): 20 | load_dotenv() 21 | pplx_api_key = os.getenv("PERPLEXITY_API_KEY") 22 | 23 | query_llm = Perplexity(api_key=pplx_api_key, model=model_name, temperature=0.5) 24 | base_prompt = "tell me about " 25 | 26 | for subject in data_points: 27 | query = base_prompt + subject 28 | messages_dict = [ 29 | {"role": "system", "content": "Be precise and concise."}, 30 | {"role": "user", "content": query}, 31 | ] 32 | messages = [ChatMessage(**msg) for msg in messages_dict] 33 | response = query_llm.chat(messages) 34 | result = custom_response_evaluator(query, response) 35 | 36 | conciseness.append(result["conciseness"]) 37 | informativeness.append(result["informativeness"]) 38 | accuracy.append(result["accuracy"]) 39 | -------------------------------------------------------------------------------- /dev/recursive_retrieval/benchmarking/judge.py: -------------------------------------------------------------------------------- 1 | import os 2 | from typing import Dict 3 | 4 | from dotenv import load_dotenv 5 | from langchain.prompts import PromptTemplate 6 | from langchain_core.output_parsers import JsonOutputParser 7 | from langchain_openai import OpenAI 8 | 9 | from output import Output 10 | 11 | 12 | def custom_response_evaluator(query: str, response: str) -> Dict[str, int]: 13 | load_dotenv() 14 | llm = OpenAI(openai_api_key=os.getenv("OPENAI_KEY"), temperature=0) 15 | parser = JsonOutputParser(pydantic_object=Output) 16 | 17 | template = """ 18 | You are an expert response judger. Score the provided respose a score from 0 to 1000 with 0 being the worst and 1000 being the best. 19 | Give separate scores for conciseness, informativeness and accuracy. 20 | For conciceness, focus on how succinct the response is and how clearly the information is conveyed using minimal words. 21 | For informativeness, focus on how comprehensive and relevant the contents of the response is to the given query. 22 | For accuracy, focus on the correctness and decree of preciseness that the response has. 23 | Hold the scores you give to the highest quality with higher numbers given only to the best of the best. 24 | 25 | Here is an example: Given the query "tell me about harry potter", and the given response Harry Potter is a series of seven fantasy novels written by British author J.K. Rowling. The novels chronicle the life of a young wizard named Harry Potter and his friends Hermione Granger and Ron Weasley, who are all students at the Hogwarts School of Witchcraft and Wizardry. 26 | The series follows Harry's life from age 11 to 17. In the first book, Harry learns that his parents were killed by the dark wizard Lord Voldemort when Harry was a baby, and that Harry survived the attack with a lightning-shaped scar on his forehead. Harry then begins attending Hogwarts, where he makes friends, encounters professors like Severus Snape, and faces off against Voldemort and his followers. 27 | The Harry Potter books have been massively popular, spawning a highly successful film series that ran from 2001 to 2011, as well as a play, video games, theme park attractions, and a vast amount of fan fiction. 28 | The series is considered a landmark of 20th and 21st century children's literature. 29 | In summary, Harry Potter is a beloved fantasy series that has had a tremendous cultural impact through its books, films, and expanded universe. 30 | 31 | The judge should give a score of 300 for conciseness, 600 to informativeness, 900 for accuracy. 32 | 33 | Format: {format_instructions} 34 | 35 | Query: {query} 36 | Response: {response} 37 | """ 38 | 39 | prompt = PromptTemplate( 40 | template=template, 41 | input_variables=["query", "response"], 42 | partial_variables={"format_instructions": parser.get_format_instructions()}, 43 | ) 44 | 45 | chain = prompt | llm | parser 46 | result = chain.invoke({"query": query, "response": response}) 47 | return result 48 | -------------------------------------------------------------------------------- /dev/recursive_retrieval/benchmarking/ollama_quality.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 2, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "from llama_index.llms.ollama import Ollama\n", 10 | "\n", 11 | "llm = Ollama(model=\"gemma\", request_timeout=30.0)" 12 | ] 13 | }, 14 | { 15 | "cell_type": "code", 16 | "execution_count": 3, 17 | "metadata": {}, 18 | "outputs": [ 19 | { 20 | "name": "stdout", 21 | "output_type": "stream", 22 | "text": [ 23 | "Harry Potter is a beloved fantasy novel series written by J.K. Rowling. The story follows the journey of Harry Potter, an orphaned boy who discovers on his eleventh birthday that he is a wizard. He is invited to Hogwarts School of Witchcraft and Wizardry, where he meets his best friends, Ron Weasley and Hermione Granger.\n", 24 | "\n", 25 | "**The story:**\n", 26 | "\n", 27 | "* **The Boy Who Lived:** The first book, \"Harry Potter and the Sorcerer's Stone,\" tells the story of Harry's first year at Hogwarts. He learns about his magical abilities, makes new friends, and faces a mysterious villain named Voldemort.\n", 28 | "* **The Prisoner of Azkaban:** The second book, \"Harry Potter and the Prisoner of Azkaban,\" explores themes of justice and innocence.\n", 29 | "* **The Order of the Phoenix:** The third book, \"Harry Potter and the Order of the Phoenix,\" introduces the concept of the Order of the Phoenix, a secret society fighting against Voldemort.\n", 30 | "* **The Half-Blood Prince:** The fourth book, \"Harry Potter and the Half-Blood Prince,\" introduces a mysterious new teacher and explores themes of prejudice and identity.\n", 31 | "* **Deathly Hallows:** The seventh book, \"Harry Potter and the Deathly Hallows,\" brings the story to a close, with a final battle against Voldemort and the defeat of evil.\n", 32 | "\n", 33 | "**The characters:**\n", 34 | "\n", 35 | "* **Harry Potter:** The protagonist, a young orphan who discovers he is a wizard. He is brave, loyal, and always willing to fight for what he believes is right.\n", 36 | "* **Ron Weasley:** Harry's best friend, a goofy but lovable wizard who is known for his humor and unwavering loyalty.\n", 37 | "* **Hermione Granger:** Harry's other best friend, a brilliant witch with a sharp mind and a strong moral compass.\n", 38 | "* **Voldemort:** The main antagonist, a dark wizard who seeks to establish his power and control.\n", 39 | "* **Albus Dumbledore:** The headmaster of Hogwarts, a powerful and enigmatic figure.\n", 40 | "\n", 41 | "**The impact:**\n", 42 | "\n", 43 | "The Harry Potter series has been a global phenomenon, selling millions of copies worldwide. It has been adapted into numerous films, and the characters have become household names. The story has also inspired countless other writers and artists.\n", 44 | "\n", 45 | "**Overall, Harry Potter is a captivating story about love, friendship, and the power of good against evil.**\n" 46 | ] 47 | } 48 | ], 49 | "source": [ 50 | "query = \"tell me about harry potter\"\n", 51 | "\n", 52 | "external_response = llm.complete(query)\n", 53 | "print(external_response)" 54 | ] 55 | } 56 | ], 57 | "metadata": { 58 | "kernelspec": { 59 | "display_name": ".venv", 60 | "language": "python", 61 | "name": "python3" 62 | }, 63 | "language_info": { 64 | "codemirror_mode": { 65 | "name": "ipython", 66 | "version": 3 67 | }, 68 | "file_extension": ".py", 69 | "mimetype": "text/x-python", 70 | "name": "python", 71 | "nbconvert_exporter": "python", 72 | "pygments_lexer": "ipython3", 73 | "version": "3.11.6" 74 | } 75 | }, 76 | "nbformat": 4, 77 | "nbformat_minor": 2 78 | } 79 | -------------------------------------------------------------------------------- /dev/recursive_retrieval/benchmarking/ollama_test.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "from llama_index.llms.ollama import Ollama\n", 10 | "\n", 11 | "llm = Ollama(model=\"gemma\", request_timeout=30.0)" 12 | ] 13 | }, 14 | { 15 | "cell_type": "code", 16 | "execution_count": 3, 17 | "metadata": {}, 18 | "outputs": [ 19 | { 20 | "name": "stdout", 21 | "output_type": "stream", 22 | "text": [ 23 | "16.1 s ± 2.75 s per loop (mean ± std. dev. of 7 runs, 1 loop each)\n" 24 | ] 25 | } 26 | ], 27 | "source": [ 28 | "%%timeit\n", 29 | "query = \"tell me about harry potter\"\n", 30 | "\n", 31 | "external_response = llm.complete(query)" 32 | ] 33 | } 34 | ], 35 | "metadata": { 36 | "kernelspec": { 37 | "display_name": ".venv", 38 | "language": "python", 39 | "name": "python3" 40 | }, 41 | "language_info": { 42 | "codemirror_mode": { 43 | "name": "ipython", 44 | "version": 3 45 | }, 46 | "file_extension": ".py", 47 | "mimetype": "text/x-python", 48 | "name": "python", 49 | "nbconvert_exporter": "python", 50 | "pygments_lexer": "ipython3", 51 | "version": "3.11.6" 52 | } 53 | }, 54 | "nbformat": 4, 55 | "nbformat_minor": 2 56 | } 57 | -------------------------------------------------------------------------------- /dev/recursive_retrieval/benchmarking/output.py: -------------------------------------------------------------------------------- 1 | from langchain_core.pydantic_v1 import BaseModel, Field 2 | 3 | 4 | class Output(BaseModel): 5 | conciseness: int = Field( 6 | description="Score for how concise the response is from 0 to 1000 witih 0 being the worst and 1000 being the best." 7 | ) 8 | informativeness: int = Field( 9 | description="Score for how informative the response is from 0 to 1000 witih 0 being the worst and 1000 being the best." 10 | ) 11 | accuracy: int = Field( 12 | description="Score for how accurate the response is based on the question from 0 to 1000 witih 0 being the worst and 1000 being the best." 13 | ) 14 | -------------------------------------------------------------------------------- /dev/recursive_retrieval/benchmarking/perplexity_quality.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 2, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import os\n", 10 | "from llama_index.llms.perplexity import Perplexity\n", 11 | "from dotenv import load_dotenv\n", 12 | "\n", 13 | "load_dotenv()\n", 14 | "\n", 15 | "pplx_api_key = os.getenv(\"PERPLEXITY_API_KEY\")\n", 16 | "\n", 17 | "llm = Perplexity(\n", 18 | " api_key=pplx_api_key, model=\"mistral-7b-instruct\", temperature=0.5\n", 19 | ")" 20 | ] 21 | }, 22 | { 23 | "cell_type": "code", 24 | "execution_count": 4, 25 | "metadata": {}, 26 | "outputs": [ 27 | { 28 | "name": "stdout", 29 | "output_type": "stream", 30 | "text": [ 31 | "assistant: Harry Potter is a series of seven fantasy novels written by British author J.K. Rowling. The books follow Harry Potter, an orphaned boy who discovers he is a wizard, as he attends Hogwarts School of Witchcraft and Wizardry and learns to master his magical abilities. Throughout the series, Harry faces numerous challenges and battles against the dark wizard Lord Voldemort. The novels are known for their intricate plotting, richly detailed world-building, and complex characters. The Harry Potter series has become a cultural phenomenon and has been translated into over 80 languages. The first novel, \"Harry Potter and the Philosopher's Stone,\" was published in 1997, and the last book, \"Harry Potter and the Deathly Hallows,\" was published in 2007. The series has also been adapted into a successful film series.\n" 32 | ] 33 | } 34 | ], 35 | "source": [ 36 | "from llama_index.core.llms import ChatMessage\n", 37 | "\n", 38 | "messages_dict = [\n", 39 | " {\"role\": \"system\", \"content\": \"Be precise and concise.\"},\n", 40 | " {\"role\": \"user\", \"content\": \"tell me about harry potter\"},\n", 41 | "]\n", 42 | "messages = [ChatMessage(**msg) for msg in messages_dict]\n", 43 | "\n", 44 | "external_response = llm.chat(messages)\n", 45 | "print(external_response)" 46 | ] 47 | }, 48 | { 49 | "cell_type": "code", 50 | "execution_count": 5, 51 | "metadata": {}, 52 | "outputs": [ 53 | { 54 | "name": "stdout", 55 | "output_type": "stream", 56 | "text": [ 57 | "assistant: Harry Potter is a popular series of seven fantasy novels written by British author J.K. Rowling. The series follows the adventures of the titular character, Harry Potter, and his friends Ron Weasley and Hermione Granger, as they attend Hogwarts School of Witchcraft and Wizardry and battle against the dark wizard, Lord Voldemort. The series explores themes of love, friendship, bravery, and the power of choice. It has also been adapted into a successful film franchise and stage play.\n" 58 | ] 59 | } 60 | ], 61 | "source": [ 62 | "llm = Perplexity(\n", 63 | " api_key=pplx_api_key, model=\"mixtral-8x7b-instruct\", temperature=0.5\n", 64 | ")\n", 65 | "messages_dict = [\n", 66 | " {\"role\": \"system\", \"content\": \"Be precise and concise.\"},\n", 67 | " {\"role\": \"user\", \"content\": \"tell me about harry potter\"},\n", 68 | "]\n", 69 | "messages = [ChatMessage(**msg) for msg in messages_dict]\n", 70 | "\n", 71 | "external_response = llm.chat(messages)\n", 72 | "print(external_response)" 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": 6, 78 | "metadata": {}, 79 | "outputs": [ 80 | { 81 | "name": "stdout", 82 | "output_type": "stream", 83 | "text": [ 84 | "assistant: Harry Potter is a series of seven fantasy novels written by British author J.K. Rowling, published between 1997 and 2007. The books follow the adventures of a young wizard named Harry Potter as he attends Hogwarts School of Witchcraft and Wizardry and battles against the evil Lord Voldemort.\n" 85 | ] 86 | } 87 | ], 88 | "source": [ 89 | "llm = Perplexity(\n", 90 | " api_key=pplx_api_key, model=\"sonar-small-chat\", temperature=0.5\n", 91 | ")\n", 92 | "messages_dict = [\n", 93 | " {\"role\": \"system\", \"content\": \"Be precise and concise.\"},\n", 94 | " {\"role\": \"user\", \"content\": \"tell me about harry potter\"},\n", 95 | "]\n", 96 | "messages = [ChatMessage(**msg) for msg in messages_dict]\n", 97 | "\n", 98 | "external_response = llm.chat(messages)\n", 99 | "print(external_response)" 100 | ] 101 | } 102 | ], 103 | "metadata": { 104 | "kernelspec": { 105 | "display_name": ".venv", 106 | "language": "python", 107 | "name": "python3" 108 | }, 109 | "language_info": { 110 | "codemirror_mode": { 111 | "name": "ipython", 112 | "version": 3 113 | }, 114 | "file_extension": ".py", 115 | "mimetype": "text/x-python", 116 | "name": "python", 117 | "nbconvert_exporter": "python", 118 | "pygments_lexer": "ipython3", 119 | "version": "3.11.6" 120 | } 121 | }, 122 | "nbformat": 4, 123 | "nbformat_minor": 2 124 | } 125 | -------------------------------------------------------------------------------- /dev/recursive_retrieval/benchmarking/perplexity_test.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 25, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import os\n", 10 | "from llama_index.llms.perplexity import Perplexity\n", 11 | "from dotenv import load_dotenv\n", 12 | "\n", 13 | "load_dotenv()\n", 14 | "\n", 15 | "pplx_api_key = os.getenv(\"PERPLEXITY_API_KEY\")\n", 16 | "\n", 17 | "llm = Perplexity(\n", 18 | " api_key=pplx_api_key, model=\"mistral-7b-instruct\", temperature=0.5\n", 19 | ")" 20 | ] 21 | }, 22 | { 23 | "cell_type": "code", 24 | "execution_count": 26, 25 | "metadata": {}, 26 | "outputs": [ 27 | { 28 | "name": "stdout", 29 | "output_type": "stream", 30 | "text": [ 31 | "1.55 s ± 192 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n" 32 | ] 33 | } 34 | ], 35 | "source": [ 36 | "%%timeit\n", 37 | "from llama_index.core.llms import ChatMessage\n", 38 | "\n", 39 | "messages_dict = [\n", 40 | " {\"role\": \"system\", \"content\": \"Be precise and concise.\"},\n", 41 | " {\"role\": \"user\", \"content\": \"tell me about harry potter\"},\n", 42 | "]\n", 43 | "messages = [ChatMessage(**msg) for msg in messages_dict]\n", 44 | "\n", 45 | "external_response = llm.chat(messages)" 46 | ] 47 | }, 48 | { 49 | "cell_type": "code", 50 | "execution_count": 18, 51 | "metadata": {}, 52 | "outputs": [], 53 | "source": [ 54 | "llm = Perplexity(\n", 55 | " api_key=pplx_api_key, model=\"mixtral-8x7b-instruct\", temperature=0.5\n", 56 | ")" 57 | ] 58 | }, 59 | { 60 | "cell_type": "code", 61 | "execution_count": 19, 62 | "metadata": {}, 63 | "outputs": [ 64 | { 65 | "name": "stdout", 66 | "output_type": "stream", 67 | "text": [ 68 | "1.36 s ± 132 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n" 69 | ] 70 | } 71 | ], 72 | "source": [ 73 | "%%timeit\n", 74 | "from llama_index.core.llms import ChatMessage\n", 75 | "\n", 76 | "messages_dict = [\n", 77 | " {\"role\": \"system\", \"content\": \"Be precise and concise.\"},\n", 78 | " {\"role\": \"user\", \"content\": \"tell me about harry potter\"},\n", 79 | "]\n", 80 | "messages = [ChatMessage(**msg) for msg in messages_dict]\n", 81 | "\n", 82 | "external_response = llm.chat(messages)" 83 | ] 84 | }, 85 | { 86 | "cell_type": "code", 87 | "execution_count": 22, 88 | "metadata": {}, 89 | "outputs": [], 90 | "source": [ 91 | "llm = Perplexity(\n", 92 | " api_key=pplx_api_key, model=\"sonar-small-chat\", temperature=0.5\n", 93 | ")" 94 | ] 95 | }, 96 | { 97 | "cell_type": "code", 98 | "execution_count": 23, 99 | "metadata": {}, 100 | "outputs": [ 101 | { 102 | "name": "stdout", 103 | "output_type": "stream", 104 | "text": [ 105 | "1.01 s ± 150 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n" 106 | ] 107 | } 108 | ], 109 | "source": [ 110 | "%%timeit\n", 111 | "from llama_index.core.llms import ChatMessage\n", 112 | "\n", 113 | "messages_dict = [\n", 114 | " {\"role\": \"system\", \"content\": \"Be precise and concise.\"},\n", 115 | " {\"role\": \"user\", \"content\": \"tell me about harry potter\"},\n", 116 | "]\n", 117 | "messages = [ChatMessage(**msg) for msg in messages_dict]\n", 118 | "\n", 119 | "external_response = llm.chat(messages)" 120 | ] 121 | } 122 | ], 123 | "metadata": { 124 | "kernelspec": { 125 | "display_name": ".venv", 126 | "language": "python", 127 | "name": "python3" 128 | }, 129 | "language_info": { 130 | "codemirror_mode": { 131 | "name": "ipython", 132 | "version": 3 133 | }, 134 | "file_extension": ".py", 135 | "mimetype": "text/x-python", 136 | "name": "python", 137 | "nbconvert_exporter": "python", 138 | "pygments_lexer": "ipython3", 139 | "version": "3.11.6" 140 | } 141 | }, 142 | "nbformat": 4, 143 | "nbformat_minor": 2 144 | } 145 | -------------------------------------------------------------------------------- /dev/recursive_retrieval/docs/cypher.md: -------------------------------------------------------------------------------- 1 | # Recursive Retrieval Cyphers 2 | 3 | ## Getting Subraph Based on Entity ID 4 | MATCH p = (:Entity { id: "Harry" })-[*]->() 5 | RETURN p; 6 | 7 | - the `*` means traversing all paths 8 | - adding a number, `x`, after `*` means a max hop distance of only that many nodes 9 | - can search labels by replacing `Entity` with labels 10 | - `->` can be replaced with `<-` for incoming children or `-` for both outgoing and incoming children 11 | 12 | 13 | drawing 14 | -------------------------------------------------------------------------------- /dev/recursive_retrieval/docs/images/harry_subgraph.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kingjulio8238/Memary/b2331a2c0844d66f69acd607b9e4dbaba56552c1/dev/recursive_retrieval/docs/images/harry_subgraph.png -------------------------------------------------------------------------------- /dev/recursive_retrieval/docs/implementation.md: -------------------------------------------------------------------------------- 1 | # Current Implementation of Recursive Retrieval 2 | ## Graph Rag + NL2GraphQuery 3 | - Graph RAG extracts subgraph related to key entities in question 4 | - NL2GraphQuery generates graph cypher based on query and schema 5 | - Note: Cypher not showing up in metadata right now 6 | - Combination of these two contexts → LLM generates answer with all of this invovled -------------------------------------------------------------------------------- /dev/recursive_retrieval/docs/updates.md: -------------------------------------------------------------------------------- 1 | # Recursive Retrieval Updates 2 | ## Update 4/6/24 3 | - LLM "Judge" 4 | - analyze quality of perplexity models (sonar-small-online, mistral, mixtral) on conciseness, informativeness, and accuracy 5 | - potential improvements should focus on quality of the prompt fed into GPT3.5 (or use GPT4) 6 | - results indicate sonar is best, but due to cost, mistral is best option for filling KG 7 | - Incorporate Mistral model for external querying in streamlit app 8 | 9 | ## Update 3/21/24 10 | - Custom synonym expansion 11 | - leveraging llm to produce larger range of synonyms of key entities found in query 12 | - custom formatting to fit neo4j 13 | - seems to dramatically improve web query → finding in graph afterwards using recursive retrieval 14 | - further testing to fine tune the prompt for increased accuracy 15 | 16 | ## Update 3/19/24 17 | - Enhancing retrieval with llm 18 | - passing in llm to `KnowledgeGraphRAGRetriever` allows model to make jumps in entity extraction process (ie return entities not specifically mentioned) 19 | - ex. `query`: Tell me about Sirius Black. `entities`: ['Harry Potter'] (assuming Sirius Black is not an existing entity) 20 | - passing in llm to `RetrieverQueryEngine` allows model to directly mix in info from llm and KG in response 21 | - overall behavior is a bit nondeterministic 22 | - likely best to not use llm on either step in order to ensure information gets added to KG 23 | - Custom entity extraction 24 | - under entity_extraction folder 25 | - work in progress - uses LangChain to pipe prompt (with template) → llm → pydantic json format 26 | - issue in custom function not being called, look into if there's time 27 | - External querying using Perplexity now works 28 | - run entirety of external_perplexity.ipynb with new query 29 | - need to test effectiveness 30 | 31 | ## Update 3/14/24 32 | - Set up querying of web corpus using Gemma, but through locally hosted model using Ollama 33 | - no API calls on external server that could be easily integrated with llama index frameworks 34 | - initial observation is that it is much slower to slower to locally host 35 | - Ollama can be used to host all of the open source models as well 36 | - very easy Ollama integration with llama index 37 | - using https://docs.llamaindex.ai/en/latest/examples/llm/ollama.html 38 | 39 | ## Update 3/13/24 40 | - Set up queyring of web corpus using Perplexity 41 | - really great optionality in specific model to use 42 | - mistral-7b-instruct 43 | - mixtral-8x7b-instruct 44 | - sonar-small-chat (perplexity model) 45 | - others found here: https://docs.perplexity.ai/docs/model-cards 46 | - interesting update from perplexity recently: *Deprecation and removal of codellama-34b-instruct and llama-2-70b-chat* 47 | - test results under /tests 48 | - Overall querying external source and adding to KG is successful 49 | - When using RAG retrieval, hard to identify key entities at times for newly added nodes 50 | - Looking into why the retrieval process is a little buggy 51 | 52 | ## Update 3/12/24 53 | - Set up querying of web corpus using Llama 2 54 | - information is succesfully injected into KG 55 | - when querying afterwards, RAG retreiver and index both have a hard time identifying the new topics being asked as key entities (index has some better luck) 56 | - Looking into LangChain + Diffbot for alternative options 57 | - Investigating why key entities are not being identified using llama index 58 | - https://lmy.medium.com/comparing-langchain-and-llamaindex-with-4-tasks-2970140edf33 -------------------------------------------------------------------------------- /dev/recursive_retrieval/entity_extraction/entity_extraction.py: -------------------------------------------------------------------------------- 1 | import os 2 | from typing import List 3 | 4 | from dotenv import load_dotenv 5 | from langchain.chains import LLMChain 6 | from langchain.prompts import PromptTemplate 7 | from langchain_core.output_parsers import JsonOutputParser 8 | from langchain_openai import OpenAI 9 | 10 | from entity_extraction.output import Output 11 | 12 | 13 | def custom_entity_extract_fn(query: str) -> List[str]: 14 | load_dotenv() 15 | llm = OpenAI(openai_api_key=os.getenv("OPENAI_KEY"), temperature=0) 16 | parser = JsonOutputParser(pydantic_object=Output) 17 | 18 | template = """ 19 | You are an expert entity extraction system. Extract the following entities from the given text: 20 | 21 | Format: {format_instructions} 22 | 23 | Text: {query} 24 | """ 25 | 26 | prompt = PromptTemplate( 27 | template=template, 28 | input_variables=["query"], 29 | partial_variables={ 30 | "format_instructions": parser.get_format_instructions() 31 | }, 32 | ) 33 | 34 | chain = prompt | llm | parser 35 | result = chain.invoke({"query": query}) 36 | 37 | l = [] 38 | for category in result: 39 | for entity in result[category]: 40 | l.append(entity) 41 | 42 | return l 43 | 44 | 45 | # testing 46 | # print(custom_entity_extract_fn("who is harry potter")) 47 | -------------------------------------------------------------------------------- /dev/recursive_retrieval/entity_extraction/output.py: -------------------------------------------------------------------------------- 1 | from typing import List 2 | 3 | from langchain_core.pydantic_v1 import BaseModel, Field 4 | 5 | 6 | class Output(BaseModel): 7 | person: List[str] = Field(description="Names of people") 8 | organization: List[str] = Field(description="Names of companies or organizations") 9 | location: List[str] = Field(description="Names of locations or places") 10 | objects: List[str] = Field(description="Names of tangible thigns") 11 | -------------------------------------------------------------------------------- /dev/recursive_retrieval/external_perplexity.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 8, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import os\n", 10 | "from llama_index.llms.perplexity import Perplexity\n", 11 | "from llama_index.llms.openai import OpenAI\n", 12 | "from dotenv import load_dotenv\n", 13 | "\n", 14 | "load_dotenv()\n", 15 | "\n", 16 | "pplx_api_key = os.getenv(\"PERPLEXITY_API_KEY\")\n", 17 | "os.environ[\"OPENAI_API_KEY\"] = os.getenv('OPENAI_KEY')\n", 18 | "\n", 19 | "query_llm = Perplexity(\n", 20 | " api_key=pplx_api_key, model=\"sonar-small-online\", temperature=0.5\n", 21 | ")\n", 22 | "\n", 23 | "entity_llm = OpenAI(temperature=0.2, model=\"gpt-4\")" 24 | ] 25 | }, 26 | { 27 | "cell_type": "code", 28 | "execution_count": 9, 29 | "metadata": {}, 30 | "outputs": [], 31 | "source": [ 32 | "from llama_index.graph_stores.neo4j import Neo4jGraphStore\n", 33 | "from llama_index.core import StorageContext\n", 34 | "\n", 35 | "username = \"neo4j\"\n", 36 | "password = os.getenv('NEO4J_PW')\n", 37 | "url = os.getenv('NEO4J_URL')\n", 38 | "database = \"neo4j\"\n", 39 | "\n", 40 | "graph_store = Neo4jGraphStore(\n", 41 | " username=username,\n", 42 | " password=password,\n", 43 | " url=url,\n", 44 | " database=database,\n", 45 | ")\n", 46 | "\n", 47 | "storage_context = StorageContext.from_defaults(graph_store=graph_store)\n" 48 | ] 49 | }, 50 | { 51 | "cell_type": "code", 52 | "execution_count": 3, 53 | "metadata": {}, 54 | "outputs": [], 55 | "source": [ 56 | "from llama_index.core.indices.knowledge_graph.retrievers import KnowledgeGraphRAGRetriever\n", 57 | "from llama_index.core.query_engine import RetrieverQueryEngine\n", 58 | "from entity_extraction.entity_extraction import custom_entity_extract_fn\n", 59 | "from synonym_expand.synonym import custom_synonym_expand_fn\n", 60 | "\n", 61 | "graph_rag_retriever_with_nl2graphquery = KnowledgeGraphRAGRetriever(\n", 62 | " storage_context=storage_context,\n", 63 | " verbose=True,\n", 64 | " llm=entity_llm,\n", 65 | " # entity_extract_fn=custom_entity_extract_fn,\n", 66 | " retriever_mode=\"keyword\",\n", 67 | " with_nl2graphquery=True,\n", 68 | " synonym_expand_fn=custom_synonym_expand_fn\n", 69 | ")\n", 70 | "\n", 71 | "query_engine_with_nl2graphquery = RetrieverQueryEngine.from_args(\n", 72 | " graph_rag_retriever_with_nl2graphquery,\n", 73 | ")" 74 | ] 75 | }, 76 | { 77 | "cell_type": "code", 78 | "execution_count": 16, 79 | "metadata": {}, 80 | "outputs": [ 81 | { 82 | "name": "stdout", 83 | "output_type": "stream", 84 | "text": [ 85 | "assistant: Crypto refers to a digital currency designed to function as a medium of exchange through a decentralized computer network, independent of a central authority like a government or bank. It utilizes cryptography to secure transactions and maintain the integrity of the currency's supply and ownership records. Cryptocurrencies are stored in digital wallets and operate on a decentralized system known as blockchain, which is a public ledger recording all transactions. The first cryptocurrency was Bitcoin, introduced in 2009, and there are now over 25,000 other cryptocurrencies available, with varying levels of adoption and regulation.\n" 86 | ] 87 | } 88 | ], 89 | "source": [ 90 | "from llama_index.core.llms import ChatMessage\n", 91 | "\n", 92 | "query = \"tell me about crypto\"\n", 93 | "\n", 94 | "pre_response = query_engine_with_nl2graphquery.query(\n", 95 | " query,\n", 96 | ")\n", 97 | "\n", 98 | "if pre_response.metadata is None:\n", 99 | " # 1) answer needs to be queried from Llama2 endpoint\n", 100 | " messages_dict = [\n", 101 | " {\"role\": \"system\", \"content\": \"Be precise and concise.\"},\n", 102 | " {\"role\": \"user\", \"content\": query},\n", 103 | " ]\n", 104 | " messages = [ChatMessage(**msg) for msg in messages_dict]\n", 105 | "\n", 106 | " external_response = query_llm.chat(messages)\n", 107 | " print(external_response)\n", 108 | "\n", 109 | " # 2) answer needs to be stored into txt file for loading into KG\n", 110 | " with open('data/external_response.txt', 'w') as f:\n", 111 | " print(external_response, file=f)\n", 112 | "else:\n", 113 | " print(\"From Graph: \" + str(pre_response))\n", 114 | " print(list(list(pre_response.metadata.values())[0][\"kg_rel_map\"].keys()))" 115 | ] 116 | }, 117 | { 118 | "cell_type": "code", 119 | "execution_count": 10, 120 | "metadata": {}, 121 | "outputs": [], 122 | "source": [ 123 | "from llama_index.core import KnowledgeGraphIndex, SimpleDirectoryReader\n", 124 | "\n", 125 | "documents = SimpleDirectoryReader(\n", 126 | " input_files=['data/external_response.txt']\n", 127 | ").load_data()\n", 128 | "\n", 129 | "index = KnowledgeGraphIndex.from_documents(\n", 130 | " documents,\n", 131 | " storage_context=storage_context,\n", 132 | " max_triplets_per_chunk=5,\n", 133 | ")" 134 | ] 135 | }, 136 | { 137 | "cell_type": "code", 138 | "execution_count": 18, 139 | "metadata": {}, 140 | "outputs": [ 141 | { 142 | "name": "stdout", 143 | "output_type": "stream", 144 | "text": [ 145 | "Cryptocurrency is a digital currency that utilizes cryptography to secure transactions and maintain the integrity of its supply. It is designed to function as a medium of exchange and refers to digital currency.\n", 146 | "['Crypto']\n" 147 | ] 148 | } 149 | ], 150 | "source": [ 151 | "post_response = query_engine_with_nl2graphquery.query(\n", 152 | " query,\n", 153 | ")\n", 154 | "\n", 155 | "print(post_response)\n", 156 | "print(list(list(post_response.metadata.values())[0][\"kg_rel_map\"].keys()))" 157 | ] 158 | } 159 | ], 160 | "metadata": { 161 | "kernelspec": { 162 | "display_name": ".venv", 163 | "language": "python", 164 | "name": "python3" 165 | }, 166 | "language_info": { 167 | "codemirror_mode": { 168 | "name": "ipython", 169 | "version": 3 170 | }, 171 | "file_extension": ".py", 172 | "mimetype": "text/x-python", 173 | "name": "python", 174 | "nbconvert_exporter": "python", 175 | "pygments_lexer": "ipython3", 176 | "version": "3.11.6" 177 | } 178 | }, 179 | "nbformat": 4, 180 | "nbformat_minor": 2 181 | } 182 | -------------------------------------------------------------------------------- /dev/recursive_retrieval/langchain_retrieval/retrieve.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 12, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import os\n", 10 | "from dotenv import load_dotenv\n", 11 | "\n", 12 | "load_dotenv()\n", 13 | "\n", 14 | "os.environ[\"OPENAI_API_KEY\"] = os.getenv('OPENAI_KEY')\n", 15 | "os.environ[\"NEO4J_URI\"] = os.getenv('NEO4J_URL')\n", 16 | "os.environ[\"NEO4J_USERNAME\"] = \"neo4j\"\n", 17 | "os.environ[\"NEO4J_PASSWORD\"] = os.getenv('NEO4J_PW')\n", 18 | "os.environ[\"NEO4J_DATABSE\"] = \"neo4j\"" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": 13, 24 | "metadata": {}, 25 | "outputs": [], 26 | "source": [ 27 | "from langchain_openai import ChatOpenAI\n", 28 | "from langchain_community.graphs import Neo4jGraph\n", 29 | "\n", 30 | "# url = os.getenv('NEO4J_URL')\n", 31 | "# username = \"neo4j\"\n", 32 | "# password = os.getenv('NEO4J_PW')\n", 33 | "# database = \"neo4j\"\n", 34 | "\n", 35 | "llm=ChatOpenAI(temperature=0, model_name=\"gpt-3.5-turbo\")\n", 36 | "graph = Neo4jGraph()" 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "execution_count": 14, 42 | "metadata": {}, 43 | "outputs": [], 44 | "source": [ 45 | "from langchain_core.prompts import ChatPromptTemplate\n", 46 | "from retrieve import Entities\n", 47 | "\n", 48 | "prompt = ChatPromptTemplate.from_messages(\n", 49 | " [\n", 50 | " (\n", 51 | " \"system\",\n", 52 | " \"You are extracting organization and person entities from the text.\",\n", 53 | " ),\n", 54 | " (\n", 55 | " \"human\",\n", 56 | " \"Use the given format to extract information from the following\"\n", 57 | " \"input: {question}\",\n", 58 | " ),\n", 59 | " ]\n", 60 | ")\n", 61 | "\n", 62 | "entity_chain = prompt | llm.with_structured_output(Entities)" 63 | ] 64 | }, 65 | { 66 | "cell_type": "code", 67 | "execution_count": 15, 68 | "metadata": {}, 69 | "outputs": [ 70 | { 71 | "data": { 72 | "text/plain": [ 73 | "[]" 74 | ] 75 | }, 76 | "execution_count": 15, 77 | "metadata": {}, 78 | "output_type": "execute_result" 79 | } 80 | ], 81 | "source": [ 82 | "from langchain_community.vectorstores import Neo4jVector\n", 83 | "from langchain_openai import OpenAIEmbeddings\n", 84 | "\n", 85 | "vector_index = Neo4jVector.from_existing_graph(\n", 86 | " OpenAIEmbeddings(),\n", 87 | " search_type=\"hybrid\",\n", 88 | " node_label=\"Document\",\n", 89 | " text_node_properties=[\"text\"],\n", 90 | " embedding_node_property=\"embedding\"\n", 91 | ")\n", 92 | "\n", 93 | "graph.query(\"CREATE FULLTEXT INDEX entity IF NOT EXISTS FOR (e:__Entity__) ON EACH [e.id]\")\n" 94 | ] 95 | }, 96 | { 97 | "cell_type": "code", 98 | "execution_count": 17, 99 | "metadata": {}, 100 | "outputs": [ 101 | { 102 | "name": "stdout", 103 | "output_type": "stream", 104 | "text": [ 105 | "\n" 106 | ] 107 | } 108 | ], 109 | "source": [ 110 | "from retrieve import structured_retriever\n", 111 | "\n", 112 | "print(structured_retriever(\"Tell me about Austin\"))" 113 | ] 114 | } 115 | ], 116 | "metadata": { 117 | "kernelspec": { 118 | "display_name": ".venv", 119 | "language": "python", 120 | "name": "python3" 121 | }, 122 | "language_info": { 123 | "codemirror_mode": { 124 | "name": "ipython", 125 | "version": 3 126 | }, 127 | "file_extension": ".py", 128 | "mimetype": "text/x-python", 129 | "name": "python", 130 | "nbconvert_exporter": "python", 131 | "pygments_lexer": "ipython3", 132 | "version": "3.11.6" 133 | } 134 | }, 135 | "nbformat": 4, 136 | "nbformat_minor": 2 137 | } 138 | -------------------------------------------------------------------------------- /dev/recursive_retrieval/langchain_retrieval/retrieve.py: -------------------------------------------------------------------------------- 1 | import os 2 | from dotenv import load_dotenv 3 | from langchain_community.graphs import Neo4jGraph 4 | from langchain_community.vectorstores import Neo4jVector 5 | from langchain_openai import OpenAIEmbeddings 6 | from langchain_core.pydantic_v1 import BaseModel, Field 7 | from typing import List 8 | from langchain_core.prompts import ChatPromptTemplate 9 | from langchain_openai import ChatOpenAI 10 | from langchain_community.vectorstores.neo4j_vector import remove_lucene_chars 11 | 12 | 13 | class Entities(BaseModel): 14 | """Identifying information about entities.""" 15 | 16 | names: List[str] = Field( 17 | ..., 18 | description="All the person, organization, or business entities that appear in the text", 19 | ) 20 | 21 | def structured_retriever(question: str) -> str: 22 | """ 23 | Collects the neighborhood of entities mentioned 24 | in the question 25 | """ 26 | result = "" 27 | entities = entity_chain.invoke({"question": question}) 28 | for entity in entities.names: 29 | response = graph.query( 30 | """CALL db.index.fulltext.queryNodes('entity', $query, 31 | {limit:2}) 32 | YIELD node,score 33 | CALL { 34 | MATCH (node)-[r:!MENTIONS]->(neighbor) 35 | RETURN node.id + ' - ' + type(r) + ' -> ' + neighbor.id AS 36 | output 37 | UNION 38 | MATCH (node)<-[r:!MENTIONS]-(neighbor) 39 | RETURN neighbor.id + ' - ' + type(r) + ' -> ' + node.id AS 40 | output 41 | } 42 | RETURN output LIMIT 50 43 | """, 44 | {"query": generate_full_text_query(entity)}, 45 | ) 46 | result += "\n".join([el['output'] for el in response]) 47 | return result 48 | 49 | def generate_full_text_query(input: str) -> str: 50 | """ 51 | Generate a full-text search query for a given input string. 52 | 53 | This function constructs a query string suitable for a full-text 54 | search. It processes the input string by splitting it into words and 55 | appending a similarity threshold (~2 changed characters) to each 56 | word, then combines them using the AND operator. Useful for mapping 57 | entities from user questions to database values, and allows for some 58 | misspelings. 59 | """ 60 | full_text_query = "" 61 | words = [el for el in remove_lucene_chars(input).split() if el] 62 | for word in words[:-1]: 63 | full_text_query += f" {word}~2 AND" 64 | full_text_query += f" {words[-1]}~2" 65 | return full_text_query.strip() 66 | 67 | -------------------------------------------------------------------------------- /dev/recursive_retrieval/recurse.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import os\n", 10 | "from dotenv import load_dotenv\n", 11 | "\n", 12 | "load_dotenv()\n", 13 | "\n", 14 | "os.environ[\"OPENAI_API_KEY\"] = os.getenv('OPENAI_KEY')" 15 | ] 16 | }, 17 | { 18 | "cell_type": "code", 19 | "execution_count": 2, 20 | "metadata": {}, 21 | "outputs": [], 22 | "source": [ 23 | "from llama_index.graph_stores.neo4j import Neo4jGraphStore\n", 24 | "from llama_index.core import StorageContext\n", 25 | "\n", 26 | "username = \"neo4j\"\n", 27 | "password = os.getenv('NEO4J_PW')\n", 28 | "url = os.getenv('NEO4J_URL')\n", 29 | "database = \"neo4j\"\n", 30 | "\n", 31 | "graph_store = Neo4jGraphStore(\n", 32 | " username=username,\n", 33 | " password=password,\n", 34 | " url=url,\n", 35 | " database=database,\n", 36 | ")\n", 37 | "\n", 38 | "storage_context = StorageContext.from_defaults(graph_store=graph_store)" 39 | ] 40 | }, 41 | { 42 | "cell_type": "code", 43 | "execution_count": 18, 44 | "metadata": {}, 45 | "outputs": [], 46 | "source": [ 47 | "from llama_index.core.indices.knowledge_graph.retrievers import KnowledgeGraphRAGRetriever\n", 48 | "from llama_index.core.query_engine import RetrieverQueryEngine\n", 49 | "\n", 50 | "graph_rag_retriever_with_nl2graphquery = KnowledgeGraphRAGRetriever(\n", 51 | " storage_context=storage_context,\n", 52 | " verbose=True,\n", 53 | " retriever_mode=\"keyword\",\n", 54 | " with_nl2graphquery=True,\n", 55 | ")\n", 56 | "\n", 57 | "query_engine_with_nl2graphquery = RetrieverQueryEngine.from_args(\n", 58 | " graph_rag_retriever_with_nl2graphquery,\n", 59 | ")\n" 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": 19, 65 | "metadata": {}, 66 | "outputs": [ 67 | { 68 | "data": { 69 | "text/markdown": [ 70 | "Harry Potter is a character in a popular series of fantasy novels written by J.K. Rowling." 71 | ], 72 | "text/plain": [ 73 | "" 74 | ] 75 | }, 76 | "metadata": {}, 77 | "output_type": "display_data" 78 | } 79 | ], 80 | "source": [ 81 | "from IPython.display import display, Markdown\n", 82 | "\n", 83 | "response = query_engine_with_nl2graphquery.query(\n", 84 | " \"who is harry potter\",\n", 85 | ")\n", 86 | "\n", 87 | "display(Markdown(f\"{response}\"))" 88 | ] 89 | }, 90 | { 91 | "cell_type": "code", 92 | "execution_count": 14, 93 | "metadata": {}, 94 | "outputs": [ 95 | { 96 | "name": "stdout", 97 | "output_type": "stream", 98 | "text": [ 99 | "{'af4510aa-44a4-4720-aa83-838ef29eb60b': {'kg_rel_map': {'Harry': [['TURNED_OUT_TO_BE',\n", 100 | " 'Bag of '\n", 101 | " 'chips'],\n", 102 | " ['ANGRILY',\n", 103 | " 'Said'],\n", 104 | " ['HEART_TWANGING_LIKE',\n", 105 | " 'Giant '\n", 106 | " 'elastic '\n", 107 | " 'band'],\n", 108 | " ['RAN',\n", 109 | " 'Before'],\n", 110 | " ['LAY_AND_WATCHED',\n", 111 | " 'Birthday '\n", 112 | " 'tick '\n", 113 | " 'nearer'],\n", 114 | " ['LAY_AND_WATCHED',\n", 115 | " 'Birthday'],\n", 116 | " ['SHUFFLED_OFF_INTO',\n", 117 | " 'Kitchen'],\n", 118 | " ['MOVED_OUT_OF',\n", 119 | " 'Cupboard'],\n", 120 | " ['MOVED_INTO',\n", 121 | " \"Dudley's \"\n", 122 | " 'second '\n", 123 | " 'bedroom'],\n", 124 | " ['COULD_SEE',\n", 125 | " 'Uncle '\n", 126 | " \"vernon's \"\n", 127 | " 'shiny '\n", 128 | " 'black '\n", 129 | " 'shoes'],\n", 130 | " ['RECEIVED',\n", 131 | " 'First '\n", 132 | " 'letter'],\n", 133 | " ['RECEIVED',\n", 134 | " 'Letter'],\n", 135 | " ['PICKED_UP',\n", 136 | " 'Letter'],\n", 137 | " ['STARED_AT',\n", 138 | " 'Letter'],\n", 139 | " ['EARNED',\n", 140 | " 'Longest-ever '\n", 141 | " 'punishment'],\n", 142 | " ['DREAMED_OF',\n", 143 | " 'Unknown '\n", 144 | " 'relation'],\n", 145 | " ['FELL_HARD_ON',\n", 146 | " 'Concrete '\n", 147 | " 'floor'],\n", 148 | " ['IMAGINED',\n", 149 | " 'School'],\n", 150 | " ['SITTING_IN',\n", 151 | " 'Car'],\n", 152 | " ['HATED',\n", 153 | " 'Mrs. '\n", 154 | " 'figg'],\n", 155 | " ['TURNING_OVER',\n", 156 | " 'Bacon'],\n", 157 | " ['WORE',\n", 158 | " 'Glasses'],\n", 159 | " ['SLEPT_IN',\n", 160 | " 'Cupboard'],\n", 161 | " ['HAD',\n", 162 | " 'Dream'],\n", 163 | " ['HAD',\n", 164 | " 'Sleepless '\n", 165 | " 'night'],\n", 166 | " ['HAD',\n", 167 | " 'Scar']]},\n", 168 | " 'kg_rel_text': [\"['TURNED_OUT_TO_BE', \"\n", 169 | " \"'Bag of chips']\",\n", 170 | " \"['ANGRILY', 'Said']\",\n", 171 | " \"['HEART_TWANGING_LIKE', \"\n", 172 | " \"'Giant elastic \"\n", 173 | " \"band']\",\n", 174 | " \"['RAN', 'Before']\",\n", 175 | " \"['LAY_AND_WATCHED', \"\n", 176 | " \"'Birthday tick \"\n", 177 | " \"nearer']\",\n", 178 | " \"['LAY_AND_WATCHED', \"\n", 179 | " \"'Birthday']\",\n", 180 | " \"['SHUFFLED_OFF_INTO', \"\n", 181 | " \"'Kitchen']\",\n", 182 | " \"['MOVED_OUT_OF', \"\n", 183 | " \"'Cupboard']\",\n", 184 | " \"['MOVED_INTO', \"\n", 185 | " '\"Dudley\\'s second '\n", 186 | " 'bedroom\"]',\n", 187 | " \"['COULD_SEE', \"\n", 188 | " '\"Uncle vernon\\'s '\n", 189 | " 'shiny black shoes\"]',\n", 190 | " \"['RECEIVED', 'First \"\n", 191 | " \"letter']\",\n", 192 | " \"['RECEIVED', \"\n", 193 | " \"'Letter']\",\n", 194 | " \"['PICKED_UP', \"\n", 195 | " \"'Letter']\",\n", 196 | " \"['STARED_AT', \"\n", 197 | " \"'Letter']\",\n", 198 | " \"['EARNED', \"\n", 199 | " \"'Longest-ever \"\n", 200 | " \"punishment']\",\n", 201 | " \"['DREAMED_OF', \"\n", 202 | " \"'Unknown relation']\",\n", 203 | " \"['FELL_HARD_ON', \"\n", 204 | " \"'Concrete floor']\",\n", 205 | " \"['IMAGINED', \"\n", 206 | " \"'School']\",\n", 207 | " \"['SITTING_IN', \"\n", 208 | " \"'Car']\",\n", 209 | " \"['HATED', 'Mrs. \"\n", 210 | " \"figg']\",\n", 211 | " \"['TURNING_OVER', \"\n", 212 | " \"'Bacon']\",\n", 213 | " \"['WORE', 'Glasses']\",\n", 214 | " \"['SLEPT_IN', \"\n", 215 | " \"'Cupboard']\",\n", 216 | " \"['HAD', 'Dream']\",\n", 217 | " \"['HAD', 'Sleepless \"\n", 218 | " \"night']\",\n", 219 | " \"['HAD', 'Scar']\"],\n", 220 | " 'kg_schema': {'schema': 'Node '\n", 221 | " 'properties '\n", 222 | " 'are the '\n", 223 | " 'following:\\n'\n", 224 | " 'Entity {id: '\n", 225 | " 'STRING}\\n'\n", 226 | " 'Relationship '\n", 227 | " 'properties '\n", 228 | " 'are the '\n", 229 | " 'following:\\n'\n", 230 | " '\\n'\n", 231 | " 'The '\n", 232 | " 'relationships '\n", 233 | " 'are the '\n", 234 | " 'following:\\n'\n", 235 | " \"(:Entity)-[:WAS_DIRECTOR_OF]->(:Entity),(:Entity)-[:HAD]->(:Entity),(:Entity)-[:CAME_INTO]->(:Entity),(:Entity)-[:FELL_ASLEEP_QUICKLY]->(:Entity),(:Entity)-[:SHUDDERED_TO_THINK]->(:Entity),(:Entity)-[:HATED]->(:Entity),(:Entity)-[:WERE]->(:Entity),(:Entity)-[:ARRIVED_IN]->(:Entity),(:Entity)-[:SAT_WITH]->(:Entity),(:Entity)-[:YELLED_AT]->(:Entity),(:Entity)-[:STOPPED]->(:Entity),(:Entity)-[:SAW]->(:Entity),(:Entity)-[:LAY_AWAKE]->(:Entity),(:Entity)-[:HAD_GOTTEN_INTO]->(:Entity),(:Entity)-[:FOUNDED_IN]->(:Entity),(:Entity)-[:IS]->(:Entity),(:Entity)-[:HAVE_BEEN_BEHAVING]->(:Entity),(:Entity)-[:HUNT_AT]->(:Entity),(:Entity)-[:KICKED]->(:Entity),(:Entity)-[:SLEPT_IN]->(:Entity),(:Entity)-[:WAS_HOWLING]->(:Entity),(:Entity)-[:LEARNED]->(:Entity),(:Entity)-[:UNWRAPPED]->(:Entity),(:Entity)-[:DANGLING_OVER]->(:Entity),(:Entity)-[:BAWLING_AT]->(:Entity),(:Entity)-[:SNORED]->(:Entity),(:Entity)-[:WAS]->(:Entity),(:Entity)-[:BROKE]->(:Entity),(:Entity)-[:THOUGHT]->(:Entity),(:Entity)-[:CELEBRATING]->(:Entity),(:Entity)-[:WAS_SHOWING]->(:Entity),(:Entity)-[:SLIPPED]->(:Entity),(:Entity)-[:TURNED]->(:Entity),(:Entity)-[:SAID]->(:Entity),(:Entity)-[:PATTED]->(:Entity),(:Entity)-[:TOLD]->(:Entity),(:Entity)-[:TOOK]->(:Entity),(:Entity)-[:LAID]->(:Entity),(:Entity)-[:WEARING]->(:Entity),(:Entity)-[:ASKED]->(:Entity),(:Entity)-[:BLEW]->(:Entity),(:Entity)-[:FLINCHED]->(:Entity),(:Entity)-[:PULLED_OUT]->(:Entity),(:Entity)-[:EYED]->(:Entity),(:Entity)-[:TURNED_UP_IN]->(:Entity),(:Entity)-[:HOLDING]->(:Entity),(:Entity)-[:SWUNG]->(:Entity),(:Entity)-[:SOBBED]->(:Entity),(:Entity)-[:MOVED_OUT_OF]->(:Entity),(:Entity)-[:SHUFFLED_OFF_INTO]->(:Entity),(:Entity)-[:LAY_AND_WATCHED]->(:Entity),(:Entity)-[:RAN]->(:Entity),(:Entity)-[:HEART_TWANGING_LIKE]->(:Entity),(:Entity)-[:ANGRILY]->(:Entity),(:Entity)-[:TURNED_OUT_TO_BE]->(:Entity),(:Entity)-[:WORE]->(:Entity),(:Entity)-[:TURNING_OVER]->(:Entity),(:Entity)-[:SITTING_IN]->(:Entity),(:Entity)-[:IMAGINED]->(:Entity),(:Entity)-[:FELL_HARD_ON]->(:Entity),(:Entity)-[:DREAMED_OF]->(:Entity),(:Entity)-[:EARNED]->(:Entity),(:Entity)-[:STARED_AT]->(:Entity),(:Entity)-[:PICKED_UP]->(:Entity),(:Entity)-[:RECEIVED]->(:Entity),(:Entity)-[:COULD_SEE]->(:Entity),(:Entity)-[:MOVED_INTO]->(:Entity),(:Entity)-[:WAS_ASLEEP_AT]->(:Entity),(:Entity)-[:WAS_NO_LONGER]->(:Entity),(:Entity)-[:ENTERED]->(:Entity),(:Entity)-[:HELD]->(:Entity),(:Entity)-[:HAD_BEEN_LYING_AT]->(:Entity),(:Entity)-[:SLAMMED]->(:Entity),(:Entity)-[:PARKED_AT]->(:Entity),(:Entity)-[:WAS_CARRYING]->(:Entity),(:Entity)-[:WAS_SAYING_IN]->(:Entity),(:Entity)-[:SNAPPED]->(:Entity),(:Entity)-[:READ]->(:Entity),(:Entity)-[:HAD_BEEN_CHASING]->(:Entity),(:Entity)-[:ATE_IN]->(:Entity),(:Entity)-[:DIDN'T_STOP_TO]->(:Entity),(:Entity)-[:JABBED_ITS_TAIL_AT]->(:Entity),(:Entity)-[:WAS_IN]->(:Entity),(:Entity)-[:DID_GO]->(:Entity),(:Entity)-[:CAME_THE_SOUND_OF]->(:Entity),(:Entity)-[:CAME]->(:Entity),(:Entity)-[:FOUND_THEIR_WAY_INTO]->(:Entity),(:Entity)-[:CAME_OVER_TO]->(:Entity),(:Entity)-[:BLEW_UP_AROUND]->(:Entity),(:Entity)-[:WAS_DANGLING_OVER]->(:Entity),(:Entity)-[:BECOMES_A_MEMBER_OF]->(:Entity),(:Entity)-[:KNOWN_FOR]->(:Entity),(:Entity)-[:LOYALTY_TO]->(:Entity),(:Entity)-[:BECOMES]->(:Entity),(:Entity)-[:IN]->(:Entity),(:Entity)-[:BY]->(:Entity),(:Entity)-[:GODFATHER_OF]->(:Entity),(:Entity)-[:FALSELY_ACCUSED_OF]->(:Entity),(:Entity)-[:BELIEVED_TO_HAVE_BEEN_KILLED_BY]->(:Entity),(:Entity)-[:RETURNS_TO]->(:Entity),(:Entity)-[:TEACHES]->(:Entity),(:Entity)-[:ASSIST]->(:Entity),(:Entity)-[:TODAY]->(:Entity),(:Entity)-[:MARRIED_TO]->(:Entity),(:Entity)-[:IS_MEMBER_OF]->(:Entity),(:Entity)-[:HAS]->(:Entity)\"}}}\n" 236 | ] 237 | } 238 | ], 239 | "source": [ 240 | "import pprint\n", 241 | "\n", 242 | "pp = pprint.PrettyPrinter()\n", 243 | "pp.pprint(response.metadata)" 244 | ] 245 | } 246 | ], 247 | "metadata": { 248 | "kernelspec": { 249 | "display_name": "Python 3", 250 | "language": "python", 251 | "name": "python3" 252 | }, 253 | "language_info": { 254 | "codemirror_mode": { 255 | "name": "ipython", 256 | "version": 3 257 | }, 258 | "file_extension": ".py", 259 | "mimetype": "text/x-python", 260 | "name": "python", 261 | "nbconvert_exporter": "python", 262 | "pygments_lexer": "ipython3", 263 | "version": "3.11.6" 264 | } 265 | }, 266 | "nbformat": 4, 267 | "nbformat_minor": 2 268 | } 269 | -------------------------------------------------------------------------------- /dev/recursive_retrieval/requirements.txt: -------------------------------------------------------------------------------- 1 | neo4j==5.17.0 2 | python-dotenv==1.0.1 3 | pyvis==0.3.2 4 | streamlit==1.31.1 5 | llama-index==0.10.11 6 | llama-index-agent-openai==0.1.5 7 | llama-index-core==0.10.12 8 | llama-index-embeddings-openai==0.1.5 9 | llama-index-graph-stores-nebula==0.1.2 10 | llama-index-graph-stores-neo4j==0.1.1 11 | llama-index-legacy==0.9.48 12 | llama-index-llms-openai==0.1.6 13 | llama-index-multi-modal-llms-openai==0.1.3 14 | llama-index-program-openai==0.1.3 15 | llama-index-question-gen-openai==0.1.2 16 | llama-index-readers-file==0.1.4 17 | llama-index-llms-llama-api==0.1.4 18 | llama-index-llms-perplexity==0.1.3 19 | llama-index-llms-ollama==0.1.2 20 | langchain==0.1.12 21 | langchain-openai==0.0.8 22 | pandas 23 | matplotlib==3.8.4 -------------------------------------------------------------------------------- /dev/recursive_retrieval/synonym_expand/output.py: -------------------------------------------------------------------------------- 1 | from typing import List 2 | 3 | from langchain_core.pydantic_v1 import BaseModel, Field 4 | 5 | 6 | class Output(BaseModel): 7 | synonyms: List[str] = Field( 8 | description="Synonyms of keywords provided, make every letter lowercase except for the first letter" 9 | ) 10 | -------------------------------------------------------------------------------- /dev/recursive_retrieval/synonym_expand/synonym.py: -------------------------------------------------------------------------------- 1 | import os 2 | from typing import List 3 | 4 | from dotenv import load_dotenv 5 | from langchain.prompts import PromptTemplate 6 | from langchain_core.output_parsers import JsonOutputParser 7 | from langchain_openai import OpenAI 8 | 9 | from synonym_expand.output import Output 10 | 11 | 12 | def custom_synonym_expand_fn(keywords: str) -> List[str]: 13 | load_dotenv() 14 | llm = OpenAI(openai_api_key=os.getenv("OPENAI_KEY"), temperature=0) 15 | parser = JsonOutputParser(pydantic_object=Output) 16 | 17 | template = """ 18 | You are an expert synonym exapnding system. Find synonyms or words commonly used in place to reference the same word for every word in the list: 19 | 20 | Some examples are: 21 | - a synonym for Palantir may be Palantir technologies or Palantir technologies inc. 22 | - a synonym for Austin may be Austin texas 23 | - a synonym for Taylor swift may be Taylor 24 | - a synonym for Winter park may be Winter park resort 25 | 26 | Format: {format_instructions} 27 | 28 | Text: {keywords} 29 | """ 30 | 31 | prompt = PromptTemplate( 32 | template=template, 33 | input_variables=["keywords"], 34 | partial_variables={"format_instructions": parser.get_format_instructions()}, 35 | ) 36 | 37 | chain = prompt | llm | parser 38 | result = chain.invoke({"keywords": keywords}) 39 | 40 | l = [] 41 | for category in result: 42 | for synonym in result[category]: 43 | l.append(synonym.capitalize()) 44 | 45 | return l 46 | 47 | 48 | # testing 49 | # print(custom_synonym_expand_fn("[Nvidia]")) 50 | -------------------------------------------------------------------------------- /diagrams/context_window.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kingjulio8238/Memary/b2331a2c0844d66f69acd607b9e4dbaba56552c1/diagrams/context_window.png -------------------------------------------------------------------------------- /diagrams/final.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kingjulio8238/Memary/b2331a2c0844d66f69acd607b9e4dbaba56552c1/diagrams/final.png -------------------------------------------------------------------------------- /diagrams/kg.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kingjulio8238/Memary/b2331a2c0844d66f69acd607b9e4dbaba56552c1/diagrams/kg.png -------------------------------------------------------------------------------- /diagrams/memary-1000-stars-video.mp4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kingjulio8238/Memary/b2331a2c0844d66f69acd607b9e4dbaba56552c1/diagrams/memary-1000-stars-video.mp4 -------------------------------------------------------------------------------- /diagrams/memary-aitx-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kingjulio8238/Memary/b2331a2c0844d66f69acd607b9e4dbaba56552c1/diagrams/memary-aitx-1.png -------------------------------------------------------------------------------- /diagrams/memary-aitx-2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kingjulio8238/Memary/b2331a2c0844d66f69acd607b9e4dbaba56552c1/diagrams/memary-aitx-2.png -------------------------------------------------------------------------------- /diagrams/memary-logo-latest.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kingjulio8238/Memary/b2331a2c0844d66f69acd607b9e4dbaba56552c1/diagrams/memary-logo-latest.png -------------------------------------------------------------------------------- /diagrams/memary-logo-new.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kingjulio8238/Memary/b2331a2c0844d66f69acd607b9e4dbaba56552c1/diagrams/memary-logo-new.png -------------------------------------------------------------------------------- /diagrams/memary-logo-transparent.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kingjulio8238/Memary/b2331a2c0844d66f69acd607b9e4dbaba56552c1/diagrams/memary-logo-transparent.png -------------------------------------------------------------------------------- /diagrams/memary_logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kingjulio8238/Memary/b2331a2c0844d66f69acd607b9e4dbaba56552c1/diagrams/memary_logo.png -------------------------------------------------------------------------------- /diagrams/memary_logo_bw.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kingjulio8238/Memary/b2331a2c0844d66f69acd607b9e4dbaba56552c1/diagrams/memary_logo_bw.png -------------------------------------------------------------------------------- /diagrams/memory.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kingjulio8238/Memary/b2331a2c0844d66f69acd607b9e4dbaba56552c1/diagrams/memory.png -------------------------------------------------------------------------------- /diagrams/memory_compression.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kingjulio8238/Memary/b2331a2c0844d66f69acd607b9e4dbaba56552c1/diagrams/memory_compression.png -------------------------------------------------------------------------------- /diagrams/memory_module.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kingjulio8238/Memary/b2331a2c0844d66f69acd607b9e4dbaba56552c1/diagrams/memory_module.png -------------------------------------------------------------------------------- /diagrams/query_decomposition.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kingjulio8238/Memary/b2331a2c0844d66f69acd607b9e4dbaba56552c1/diagrams/query_decomposition.png -------------------------------------------------------------------------------- /diagrams/reranking_diagram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kingjulio8238/Memary/b2331a2c0844d66f69acd607b9e4dbaba56552c1/diagrams/reranking_diagram.png -------------------------------------------------------------------------------- /diagrams/routing_agent.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kingjulio8238/Memary/b2331a2c0844d66f69acd607b9e4dbaba56552c1/diagrams/routing_agent.png -------------------------------------------------------------------------------- /diagrams/system.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kingjulio8238/Memary/b2331a2c0844d66f69acd607b9e4dbaba56552c1/diagrams/system.png -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | [build-system] 2 | requires = [ 3 | "hatchling", 4 | ] 5 | build-backend = "hatchling.build" 6 | 7 | [project] 8 | name = "memary" 9 | version = "0.1.2" 10 | authors = [ 11 | { name="memary Labs", email="hello@memarylabs.com" }, 12 | ] 13 | description = "Longterm Memory for Autonomous Agents" 14 | readme = "README_hidden.md" 15 | requires-python = ">=3.8, <=3.11.9" 16 | classifiers = [ 17 | "Programming Language :: Python :: 3", 18 | "License :: OSI Approved :: MIT License", 19 | "Operating System :: OS Independent", 20 | ] 21 | dependencies = [ 22 | "FalkorDB==1.0.8", 23 | "neo4j==5.17.0", 24 | "python-dotenv==1.0.1", 25 | "pyvis==0.3.2", 26 | "streamlit==1.38.0", 27 | "llama-index==0.11.4", 28 | "llama-index-agent-openai==0.3.0", 29 | "llama-index-core==0.11.4", 30 | "llama-index-embeddings-openai==0.2.4", 31 | "llama-index-graph-stores-nebula==0.3.0", 32 | "llama-index-graph-stores-neo4j==0.3.1", 33 | "llama-index-graph-stores-falkordb==0.2.1", 34 | "llama-index-legacy==0.9.48", 35 | "llama-index-llms-openai==0.2.1", 36 | "llama-index-multi-modal-llms-openai==0.2.0", 37 | "llama-index-program-openai==0.2.0", 38 | "llama-index-question-gen-openai==0.2.0", 39 | "llama-index-readers-file==0.2.0", 40 | "langchain==0.1.12", 41 | "langchain-openai==0.0.8", 42 | "llama-index-llms-perplexity==0.2.0", 43 | "llama-index-multi-modal-llms-ollama==0.3.0", 44 | "llama-index-llms-ollama==0.3.1", 45 | "pandas", 46 | "geocoder", 47 | "googlemaps", 48 | "ansistrip", 49 | "numpy", 50 | "ollama", 51 | ] 52 | 53 | [project.urls] 54 | Homepage = "https://github.com/kingjulio8238/memary" 55 | Issues = "https://github.com/kingjulio8238/memary/issues" -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | FalkorDB==1.0.8 2 | neo4j==5.17.0 3 | python-dotenv==1.0.1 4 | pyvis==0.3.2 5 | streamlit==1.38.0 6 | llama-index==0.11.4 7 | llama-index-agent-openai==0.3.0 8 | llama-index-core==0.11.4 9 | llama-index-embeddings-openai==0.2.4 10 | llama-index-graph-stores-nebula==0.3.0 11 | llama-index-graph-stores-neo4j==0.3.1 12 | llama-index-graph-stores-falkordb==0.2.1 13 | llama-index-legacy==0.9.48 14 | llama-index-llms-openai==0.2.1 15 | llama-index-multi-modal-llms-openai==0.2.0 16 | llama-index-program-openai==0.2.0 17 | llama-index-question-gen-openai==0.2.0 18 | llama-index-readers-file==0.2.0 19 | langchain==0.1.12 20 | langchain-openai==0.0.8 21 | llama-index-llms-perplexity==0.2.0 22 | llama-index-multi-modal-llms-ollama==0.3.0 23 | llama-index-llms-ollama==0.3.1 24 | pandas 25 | geocoder 26 | googlemaps 27 | ansistrip 28 | numpy 29 | ollama -------------------------------------------------------------------------------- /src/memary/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kingjulio8238/Memary/b2331a2c0844d66f69acd607b9e4dbaba56552c1/src/memary/__init__.py -------------------------------------------------------------------------------- /src/memary/agent/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kingjulio8238/Memary/b2331a2c0844d66f69acd607b9e4dbaba56552c1/src/memary/agent/__init__.py -------------------------------------------------------------------------------- /src/memary/agent/chat_agent.py: -------------------------------------------------------------------------------- 1 | from typing import Optional, List 2 | 3 | from memary.agent.base_agent import Agent 4 | import logging 5 | 6 | 7 | class ChatAgent(Agent): 8 | """ChatAgent currently able to support Llama3 running on Ollama (default) and gpt-3.5-turbo for llm models, 9 | and LLaVA running on Ollama (default) and gpt-4-vision-preview for the vision tool. 10 | """ 11 | 12 | def __init__( 13 | self, 14 | agent_name, 15 | memory_stream_json, 16 | entity_knowledge_store_json, 17 | system_persona_txt, 18 | user_persona_txt, 19 | past_chat_json, 20 | user_id='falkor', 21 | llm_model_name="llama3", 22 | vision_model_name="llava", 23 | include_from_defaults=["search", "locate", "vision", "stocks"], 24 | ): 25 | super().__init__( 26 | agent_name, 27 | memory_stream_json, 28 | entity_knowledge_store_json, 29 | system_persona_txt, 30 | user_persona_txt, 31 | past_chat_json, 32 | user_id, 33 | llm_model_name, 34 | vision_model_name, 35 | include_from_defaults, 36 | ) 37 | 38 | def add_chat(self, role: str, content: str, entities: Optional[List[str]] = None): 39 | """Add a chat to the agent's memory. 40 | 41 | Args: 42 | role (str): 'system' or 'user' 43 | content (str): content of the chat 44 | entities (Optional[List[str]], optional): entities from Memory systems. Defaults to None. 45 | """ 46 | # Add a chat to the agent's memory. 47 | self._add_contexts_to_llm_message(role, content) 48 | 49 | if entities: 50 | self.memory_stream.add_memory(entities) 51 | self.memory_stream.save_memory() 52 | self.entity_knowledge_store.add_memory(self.memory_stream.get_memory()) 53 | self.entity_knowledge_store.save_memory() 54 | 55 | self._replace_memory_from_llm_message() 56 | self._replace_eks_to_from_message() 57 | 58 | def get_chat(self): 59 | return self.contexts 60 | 61 | def clearMemory(self): 62 | """Clears database and memory stream/entity knowledge store.""" 63 | 64 | logging.info("Deleting memory stream and entity knowledge store...") 65 | self.memory_stream.clear_memory() 66 | self.entity_knowledge_store.clear_memory() 67 | 68 | logging.info("Deleting nodes from Database...") 69 | try: 70 | self.graph_store.query("MATCH (n) DETACH DELETE n") 71 | except Exception as e: 72 | logging.error(f"Error deleting nodes: {e}") 73 | logging.info("Nodes deleted from Database.") 74 | 75 | def _replace_memory_from_llm_message(self): 76 | """Replace the memory_stream from the llm_message.""" 77 | self.message.llm_message["memory_stream"] = self.memory_stream.get_memory() 78 | 79 | def _replace_eks_to_from_message(self): 80 | """Replace the entity knowledge store from the llm_message. 81 | eks = entity knowledge store""" 82 | 83 | self.message.llm_message["knowledge_entity_store"] = ( 84 | self.entity_knowledge_store.get_memory() 85 | ) 86 | -------------------------------------------------------------------------------- /src/memary/agent/data_types.py: -------------------------------------------------------------------------------- 1 | import json 2 | import logging 3 | from dataclasses import dataclass 4 | 5 | 6 | def save_json(filename, data): 7 | with open(filename, "w") as f: 8 | json.dump(data, f, indent=4) 9 | 10 | 11 | @dataclass 12 | class Context: 13 | """Context class to store the role and content of the message.""" 14 | role: str # system or user 15 | content: str 16 | 17 | def __str__(self): 18 | return f"{self.role}: {self.content} |" 19 | 20 | def to_dict(self): 21 | return {'role': self.role, 'content': self.content} 22 | 23 | 24 | class Message: 25 | """Message class to store the contexts, memory stream and knowledge entity store.""" 26 | 27 | def __init__(self, system_persona_txt, user_persona_txt, past_chat_json, model): 28 | self.past_chat_json = past_chat_json 29 | 30 | self.contexts = [] 31 | self.system_persona = self.load_persona(system_persona_txt) 32 | self.user_persona = self.load_persona(user_persona_txt) 33 | self._init_persona_to_messages() 34 | self.contexts.extend(self.load_contexts_from_json()) 35 | 36 | self.llm_message = { 37 | "model": model, 38 | "messages": self.contexts, 39 | "memory_stream": [], 40 | "knowledge_entity_store": [] 41 | } 42 | 43 | # self.prompt_tokens = count_tokens(self.contexts) 44 | 45 | def __str__(self): 46 | llm_message_str = f"System Persona: {self.system_persona}\nUser Persona: {self.user_persona}\n" 47 | for context in self.contexts: 48 | llm_message_str += f"{str(context)}," 49 | for memory in self.llm_message["memory_stream"]: 50 | llm_message_str += f"{str(memory)}," 51 | for entity in self.llm_message["knowledge_entity_store"]: 52 | llm_message_str += f"{str(entity)}," 53 | return llm_message_str 54 | 55 | def _init_persona_to_messages(self): 56 | """Initializes the system and user personas to the contexts.""" 57 | self.contexts.append(Context("system", self.system_persona)) 58 | self.contexts.append(Context("user", self.user_persona)) 59 | 60 | def load_persona(self, persona_txt) -> str: 61 | """Loads the persona from the txt file. 62 | 63 | Args: 64 | persona_txt (str): persona txt file path 65 | 66 | Returns: 67 | str: persona 68 | """ 69 | try: 70 | with open(persona_txt, "r") as file: 71 | persona = file.read() 72 | return persona 73 | except FileNotFoundError: 74 | logging.info(f"{persona_txt} file does not exist.") 75 | 76 | def load_contexts_from_json(self): 77 | """Loads the contexts from the past chat json file.""" 78 | try: 79 | with open(self.past_chat_json, "r") as file: 80 | data_dicts = json.load(file) 81 | 82 | return [Context(**data_dict) for data_dict in data_dicts] 83 | except: 84 | logging.info( 85 | f"{self.past_chat_json} file does not exist. Starts from empty contexts." 86 | ) 87 | return [] 88 | 89 | def save_contexts_to_json(self): 90 | """Saves the contexts to the json file. 91 | We don't save the system and user personas (first two messages) 92 | """ 93 | save_json(self.past_chat_json, [message.to_dict() for message in self.llm_message['messages'][2:]]) 94 | -------------------------------------------------------------------------------- /src/memary/agent/llm_api/tools.py: -------------------------------------------------------------------------------- 1 | # From MemGPT llm_api_tools.py: 2 | 3 | import urllib 4 | import logging 5 | import requests 6 | 7 | 8 | def smart_urljoin(base_url, relative_url): 9 | """urljoin is stupid and wants a trailing / at the end of the endpoint address, or it will chop the suffix off""" 10 | if not base_url.endswith("/"): 11 | base_url += "/" 12 | return urllib.parse.urljoin(base_url, relative_url) 13 | 14 | 15 | def openai_chat_completions_request(url, api_key, data): 16 | """text-generation?lang=curl""" 17 | 18 | url = smart_urljoin(url, "chat/completions") 19 | headers = {"Content-Type": "application/json", "Authorization": f"Bearer {api_key}"} 20 | 21 | logging.info(f"Sending request to {url}") 22 | try: 23 | # Example code to trigger a rate limit response: 24 | # mock_response = requests.Response() 25 | # mock_response.status_code = 429 26 | # http_error = requests.exceptions.HTTPError("429 Client Error: Too Many Requests") 27 | # http_error.response = mock_response 28 | # raise http_error 29 | 30 | # Example code to trigger a context overflow response (for an 8k model) 31 | # data["messages"][-1]["content"] = " ".join(["repeat after me this is not a fluke"] * 1000) 32 | 33 | response = requests.post(url, headers=headers, json=data) 34 | logging.info(f"response = {response}") 35 | response.raise_for_status() # Raises HTTPError for 4XX/5XX status 36 | response = response.json() # convert to dict from string 37 | logging.info(f"response.json = {response}") 38 | # response = ChatCompletionResponse(**response) # convert to 'dot-dict' style which is the openai python client default 39 | return response 40 | except requests.exceptions.HTTPError as http_err: 41 | # Handle HTTP errors (e.g., response 4XX, 5XX) 42 | logging.error(f"Got HTTPError, exception={http_err}, payload={data}") 43 | raise http_err 44 | except requests.exceptions.RequestException as req_err: 45 | # Handle other requests-related errors (e.g., connection error) 46 | logging.warning(f"Got RequestException, exception={req_err}") 47 | raise req_err 48 | except Exception as e: 49 | # Handle other potential errors 50 | logging.warning(f"Got unknown Exception, exception={e}") 51 | raise e 52 | 53 | 54 | def ollama_chat_completions_request(messages, model): 55 | """sends chat request to model running on Ollama""" 56 | 57 | url = "http://localhost:11434/api/chat" 58 | data = {"model": model, "messages": messages, "stream": False} 59 | 60 | logging.info(f"Sending request to {url}") 61 | try: 62 | response = requests.post(url, json=data) 63 | logging.info(f"response = {response}") 64 | response.raise_for_status() 65 | response = response.json() 66 | logging.info(f"response.json = {response}") 67 | return response 68 | except requests.exceptions.HTTPError as http_err: 69 | # Handle HTTP errors (e.g., response 4XX, 5XX) 70 | logging.error(f"Got HTTPError, exception={http_err}, payload={data}") 71 | raise http_err 72 | except requests.exceptions.RequestException as req_err: 73 | # Handle other requests-related errors (e.g., connection error) 74 | logging.warning(f"Got RequestException, exception={req_err}") 75 | raise req_err 76 | except Exception as e: 77 | # Handle other potential errors 78 | logging.warning(f"Got unknown Exception, exception={e}") 79 | raise e 80 | -------------------------------------------------------------------------------- /src/memary/memory/__init__.py: -------------------------------------------------------------------------------- 1 | from memary.memory.base_memory import BaseMemory 2 | from memary.memory.memory_stream import MemoryStream 3 | from memary.memory.entity_knowledge_store import EntityKnowledgeStore 4 | 5 | __all__ = ["BaseMemory", "MemoryStream", "EntityKnowledgeStore"] 6 | -------------------------------------------------------------------------------- /src/memary/memory/base_memory.py: -------------------------------------------------------------------------------- 1 | import json 2 | import logging 3 | 4 | from abc import ABC, abstractmethod 5 | from datetime import datetime, timedelta 6 | 7 | 8 | class BaseMemory(ABC): 9 | 10 | def __init__(self, file_name: str, entity: str = None): 11 | """Initializes the memory storage.""" 12 | self.file_name = file_name 13 | self.entity = entity 14 | self.memory = [] 15 | self.knowledge_memory = [] 16 | self.init_memory() 17 | 18 | @abstractmethod 19 | def __len__(self): 20 | """Returns the number of items in the memory.""" 21 | pass 22 | 23 | @abstractmethod 24 | def init_memory(self): 25 | """Initializes memory.""" 26 | pass 27 | 28 | @abstractmethod 29 | def load_memory_from_file(self): 30 | """Loads memory from a file.""" 31 | pass 32 | 33 | @abstractmethod 34 | def add_memory(self, data): 35 | """Adds new memory data.""" 36 | pass 37 | 38 | @abstractmethod 39 | def get_memory(self): 40 | pass 41 | 42 | @property 43 | def return_memory(self): 44 | return self.memory 45 | 46 | def remove_old_memory(self, days): 47 | """Removes memory items older than a specified number of days.""" 48 | cutoff_date = datetime.now() - timedelta(days=days) 49 | self.memory = [ 50 | item for item in self.return_memory if item.date >= cutoff_date 51 | ] 52 | logging.info("Old memory removed successfully.") 53 | 54 | def save_memory(self): 55 | if self.file_name: 56 | with open(self.file_name, 'w') as file: 57 | json.dump([item.to_dict() for item in self.return_memory], 58 | file, 59 | default=str, 60 | indent=4) 61 | logging.info(f"Memory saved to {self.file_name} successfully.") 62 | else: 63 | logging.info("No file name provided. Memory not saved.") 64 | 65 | def get_memory_by_index(self, index): 66 | if 0 <= index < len(self.memory): 67 | return self.memory[index] 68 | else: 69 | return None 70 | 71 | def remove_memory_by_index(self, index): 72 | if 0 <= index < len(self.memory): 73 | del self.memory[index] 74 | logging.info("Memory item removed successfully.") 75 | return True 76 | else: 77 | logging.info("Invalid index. Memory item not removed.") 78 | return False 79 | 80 | def clear_memory(self): 81 | self.memory = [] 82 | if self.file_name: 83 | with open(self.file_name, 'w') as file: 84 | json.dump([], file, indent=4) 85 | logging.info(f"Memory cleared and saved to {self.file_name} successfully.") 86 | else: 87 | logging.info("No file name provided. Memory not cleared or saved.") 88 | -------------------------------------------------------------------------------- /src/memary/memory/entity_knowledge_store.py: -------------------------------------------------------------------------------- 1 | import json 2 | import logging 3 | 4 | from memary.memory import BaseMemory 5 | from memary.memory.types import KnowledgeMemoryItem, MemoryItem 6 | 7 | 8 | class EntityKnowledgeStore(BaseMemory): 9 | 10 | def __len__(self): 11 | """Returns the number of items in the memory.""" 12 | return len(self.knowledge_memory) 13 | 14 | def init_memory(self): 15 | """Initializes memory. 16 | self.entity_memory: list[EntityMemoryItem] 17 | """ 18 | self.load_memory_from_file() 19 | if self.entity: 20 | self.add_memory(self.entity) 21 | 22 | @property 23 | def return_memory(self): 24 | return self.knowledge_memory 25 | 26 | 27 | def load_memory_from_file(self): 28 | try: 29 | with open(self.file_name, 'r') as file: 30 | self.knowledge_memory = [ 31 | KnowledgeMemoryItem.from_dict(item) 32 | for item in json.load(file) 33 | ] 34 | logging.info( 35 | f"Entity Knowledge Memory loaded from {self.file_name} successfully." 36 | ) 37 | except FileNotFoundError: 38 | logging.info( 39 | "File not found. Starting with an empty entity knowledge memory." 40 | ) 41 | 42 | def add_memory(self, memory_stream: list[MemoryItem]): 43 | """To add new memory to the entity knowledge store 44 | we should convert the memory to knowledge memory and then update the knowledge memory 45 | 46 | Args: 47 | memory_stream (list): list of MemoryItem 48 | """ 49 | knowledge_meory = self._convert_memory_to_knowledge_memory( 50 | memory_stream) 51 | self._update_knowledge_memory(knowledge_meory) 52 | 53 | 54 | def _update_knowledge_memory(self, knowledge_memory: list): 55 | """update self.knowledge memory with new knowledge memory items 56 | 57 | Args: 58 | knowledge_memory (list): list of KnowledgeMemoryItem 59 | """ 60 | for item in knowledge_memory: 61 | for i, entity in enumerate(self.knowledge_memory): 62 | if entity.entity == item.entity: 63 | self.knowledge_memory[i].date = item.date 64 | self.knowledge_memory[i].count += item.count 65 | break 66 | else: 67 | self.knowledge_memory.append(item) 68 | 69 | def _convert_memory_to_knowledge_memory( 70 | self, memory_stream: list) -> list[KnowledgeMemoryItem]: 71 | """Converts memory to knowledge memory 72 | 73 | Returns: 74 | knowledge_memory (list): list of KnowledgeMemoryItem 75 | """ 76 | knowledge_memory = [] 77 | 78 | entities = set([item.entity for item in memory_stream]) 79 | for entity in entities: 80 | memory_dates = [ 81 | item.date for item in memory_stream if item.entity == entity 82 | ] 83 | knowledge_memory.append( 84 | KnowledgeMemoryItem(entity, len(memory_dates), 85 | max(memory_dates))) 86 | return knowledge_memory 87 | 88 | def get_memory(self) -> list[KnowledgeMemoryItem]: 89 | return self.knowledge_memory 90 | 91 | 92 | 93 | -------------------------------------------------------------------------------- /src/memary/memory/memory_stream.py: -------------------------------------------------------------------------------- 1 | import json 2 | import logging 3 | from datetime import datetime 4 | 5 | from memary.memory import BaseMemory 6 | from memary.memory.types import MemoryItem 7 | 8 | logging.basicConfig(level=logging.INFO) 9 | 10 | 11 | class MemoryStream(BaseMemory): 12 | 13 | def __len__(self): 14 | """Returns the number of items in the memory.""" 15 | return len(self.memory) 16 | 17 | def init_memory(self): 18 | """Initializes memory 19 | self.memory: list[MemoryItem] 20 | """ 21 | self.load_memory_from_file() 22 | if self.entity: 23 | self.add_memory(self.entity) 24 | 25 | @property 26 | def return_memory(self): 27 | return self.memory 28 | 29 | def add_memory(self, entities): 30 | self.memory.extend([ 31 | MemoryItem(str(entity), 32 | datetime.now().replace(microsecond=0)) 33 | for entity in entities 34 | ]) 35 | 36 | def get_memory(self) -> list[MemoryItem]: 37 | return self.memory 38 | 39 | 40 | def load_memory_from_file(self): 41 | try: 42 | with open(self.file_name, 'r') as file: 43 | self.memory = [ 44 | MemoryItem.from_dict(item) for item in json.load(file) 45 | ] 46 | logging.info(f"Memory loaded from {self.file_name} successfully.") 47 | except FileNotFoundError: 48 | logging.info("File not found. Starting with an empty memory.") 49 | -------------------------------------------------------------------------------- /src/memary/memory/types.py: -------------------------------------------------------------------------------- 1 | from dataclasses import dataclass, asdict 2 | from datetime import datetime 3 | 4 | 5 | @dataclass 6 | class MemoryItem: 7 | entity: str 8 | date: datetime 9 | 10 | def __str__(self): 11 | return f"{self.entity}, {self.date.isoformat()}" 12 | 13 | def to_dict(self): 14 | return {'entity': self.entity, 'date': str(self.date.isoformat())} 15 | 16 | @classmethod 17 | def from_dict(cls, data): 18 | return cls(entity=data['entity'], 19 | date=datetime.fromisoformat(data['date'])) 20 | 21 | 22 | @dataclass 23 | class KnowledgeMemoryItem: 24 | entity: str 25 | count: int 26 | date: datetime 27 | 28 | def __str__(self): 29 | return f"{self.entity}, {self.count}, {self.date.isoformat()}" 30 | 31 | def to_dict(self): 32 | return { 33 | 'entity': self.entity, 34 | 'count': self.count, 35 | 'date': str(self.date.isoformat()) 36 | } 37 | 38 | @classmethod 39 | def from_dict(cls, data): 40 | return cls(entity=data['entity'], 41 | count=data['count'], 42 | date=datetime.fromisoformat(data['date'])) 43 | -------------------------------------------------------------------------------- /src/memary/synonym_expand/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kingjulio8238/Memary/b2331a2c0844d66f69acd607b9e4dbaba56552c1/src/memary/synonym_expand/__init__.py -------------------------------------------------------------------------------- /src/memary/synonym_expand/output.py: -------------------------------------------------------------------------------- 1 | from typing import List 2 | 3 | from langchain_core.pydantic_v1 import BaseModel, Field 4 | 5 | 6 | class Output(BaseModel): 7 | synonyms: List[str] = Field(description="Synonyms of keywords provided, make every letter lowercase except for the first letter") 8 | -------------------------------------------------------------------------------- /src/memary/synonym_expand/synonym.py: -------------------------------------------------------------------------------- 1 | from langchain_openai import OpenAI 2 | from langchain.prompts import PromptTemplate 3 | from langchain_core.output_parsers import JsonOutputParser 4 | from typing import List 5 | import os 6 | from memary.synonym_expand.output import Output 7 | from dotenv import load_dotenv 8 | 9 | def custom_synonym_expand_fn(keywords: str) -> List[str]: 10 | load_dotenv() 11 | llm = OpenAI(openai_api_key=os.getenv("OPENAI_KEY"), temperature=0) 12 | parser = JsonOutputParser(pydantic_object=Output) 13 | 14 | template = """ 15 | You are an expert synonym exapnding system. Find synonyms or words commonly used in place to reference the same word for every word in the list: 16 | 17 | Some examples are: 18 | - a synonym for Palantir may be Palantir technologies or Palantir technologies inc. 19 | - a synonym for Austin may be Austin texas 20 | - a synonym for Taylor swift may be Taylor 21 | - a synonym for Winter park may be Winter park resort 22 | 23 | Format: {format_instructions} 24 | 25 | Text: {keywords} 26 | """ 27 | 28 | prompt = PromptTemplate( 29 | template=template, 30 | input_variables=["keywords"], 31 | partial_variables={"format_instructions": parser.get_format_instructions()}, 32 | ) 33 | 34 | chain = prompt | llm | parser 35 | result = chain.invoke({"keywords": keywords}) 36 | 37 | l = [] 38 | for category in result: 39 | for synonym in result[category]: 40 | l.append(synonym.capitalize()) 41 | 42 | return l 43 | 44 | # testing 45 | # print(custom_synonym_expand_fn("[Nvidia]")) 46 | -------------------------------------------------------------------------------- /streamlit_app/app.py: -------------------------------------------------------------------------------- 1 | import os 2 | import random 3 | import sys 4 | import textwrap 5 | 6 | import ollama 7 | import pandas as pd 8 | import streamlit as st 9 | import streamlit.components.v1 as components 10 | from dotenv import load_dotenv 11 | from neo4j import GraphDatabase 12 | from falkordb import FalkorDB 13 | from pyvis.network import Network 14 | 15 | # src should sit in the same level as /streamlit_app 16 | curr_dir = os.getcwd() 17 | 18 | parent_dir = os.path.dirname(curr_dir) 19 | # parent_dir = os.path.dirname(curr_dir) + '/memary' #Use this if error: src not found. Also move the '/streamlit_app/data' folder to the 'memary' folder, outside the 'src' folder. 20 | 21 | sys.path.append(parent_dir + "/src") 22 | 23 | from memary.agent.chat_agent import ChatAgent 24 | 25 | load_dotenv() 26 | system_persona_txt = "data/system_persona.txt" 27 | user_persona_txt = "data/user_persona.txt" 28 | past_chat_json = "data/past_chat.json" 29 | memory_stream_json = "data/memory_stream.json" 30 | entity_knowledge_store_json = "data/entity_knowledge_store.json" 31 | 32 | def create_graph(nodes, edges): 33 | g = Network( 34 | notebook=True, 35 | directed=True, 36 | cdn_resources="in_line", 37 | height="500px", 38 | width="100%", 39 | ) 40 | 41 | for node in nodes: 42 | g.add_node(node, label=node, title=node) 43 | for edge in edges: 44 | # assuming only one relationship type per edge 45 | g.add_edge(edge[0], edge[1], label=edge[2][0]) 46 | 47 | g.repulsion( 48 | node_distance=200, 49 | central_gravity=0.12, 50 | spring_length=150, 51 | spring_strength=0.05, 52 | damping=0.09, 53 | ) 54 | return g 55 | 56 | 57 | def fill_graph(nodes, edges, cypher_query): 58 | entities = [] 59 | 60 | if chat_agent.falkordb_url is not None: 61 | falkordb = FalkorDB.from_url(chat_agent.falkordb_url) 62 | session = falkordb.select_graph(user_id) 63 | result = session.query(cypher_query).result_set 64 | for record in result: 65 | path = record[0] 66 | n1_id = path.get_node(0).properties["id"] 67 | n2_id = path.get_node(1).properties["id"] 68 | rels = [rel.relation for rel in path.edges()] 69 | 70 | nodes.add(n1_id) 71 | nodes.add(n2_id) 72 | edges.append((n1_id, n2_id, rels)) 73 | entities.extend([n1_id, n2_id]) 74 | else: 75 | with GraphDatabase.driver( 76 | uri=chat_agent.neo4j_url, 77 | auth=(chat_agent.neo4j_username, chat_agent.neo4j_password), 78 | ) as driver: 79 | with driver.session() as session: 80 | result = session.run(cypher_query) 81 | for record in result: 82 | path = record["p"] 83 | rels = [rel.type for rel in path.relationships] 84 | 85 | n1_id = record["p"].nodes[0]["id"] 86 | n2_id = record["p"].nodes[1]["id"] 87 | nodes.add(n1_id) 88 | nodes.add(n2_id) 89 | edges.append((n1_id, n2_id, rels)) 90 | entities.extend([n1_id, n2_id]) 91 | 92 | 93 | def get_models(llm_models, vision_models): 94 | models = set() 95 | try: 96 | ollama_info = ollama.list() 97 | for e in ollama_info["models"]: 98 | models.add(e["model"]) 99 | if "llava:latest" in models: 100 | vision_models.append("llava:latest") 101 | models.remove("llava:latest") 102 | llm_models.extend(list(models)) 103 | except: 104 | print("No Ollama instance detected.") 105 | 106 | cypher_query = "MATCH p = (:Entity)-[r]-() RETURN p, r LIMIT 1000;" 107 | answer = "" 108 | external_response = "" 109 | st.title("memary") 110 | # Create a two column layout (HTML table) 111 | col1, col2 = st.columns([3, 1], vertical_alignment="bottom") 112 | 113 | # Create a text input field for user ID 114 | with col1: 115 | user_id = st.text_input( 116 | "Enter a user ID (Available only with FalkorDB)", 117 | ) 118 | # Create a button to apply the switch user ID 119 | with col2: 120 | st.write("") 121 | submit = st.button("Switch Agent ID") 122 | 123 | # If the user ID is empty, set it to "falkor" 124 | if len(user_id) == 0: 125 | user_id = "falkor" 126 | 127 | llm_models = ["gpt-3.5-turbo"] 128 | vision_models = ["gpt-4-vision-preview"] 129 | get_models(llm_models, vision_models) 130 | 131 | selected_llm_model = st.selectbox( 132 | "Select an LLM model to use.", 133 | (model for model in llm_models), 134 | index=None, 135 | placeholder="Select LLM Model...", 136 | ) 137 | selected_vision_model = st.selectbox( 138 | "Select a vision model to use.", 139 | (model for model in vision_models), 140 | index=None, 141 | placeholder="Select Vision Model...", 142 | ) 143 | 144 | if selected_llm_model and selected_vision_model: 145 | chat_agent = ChatAgent( 146 | "Personal Agent", 147 | memory_stream_json, 148 | entity_knowledge_store_json, 149 | system_persona_txt, 150 | user_persona_txt, 151 | past_chat_json, 152 | user_id, 153 | selected_llm_model, 154 | selected_vision_model, 155 | ) 156 | 157 | st.write(" ") 158 | clear_memory = st.button("Clear Memory DB") 159 | query = st.text_input("Ask a question") 160 | 161 | tools = st.multiselect( 162 | "Select tools to include:", 163 | ["search", "locate", "vision", "stocks"], # all options available 164 | ["search", "locate", "vision", "stocks"], # options that are selected by default 165 | ) 166 | 167 | img_url = "" 168 | if "vision" in tools: 169 | img_url = st.text_input("URL of image, leave blank if no image to provide") 170 | if img_url: 171 | st.image(img_url, caption="Uploaded Image", use_column_width=True) 172 | 173 | generate_clicked = st.button("Generate") 174 | st.write("") 175 | 176 | if clear_memory: 177 | # print("Front end recieved request to clear memory") 178 | chat_agent.clearMemory() 179 | st.write("Memory DB cleared") 180 | 181 | if generate_clicked: 182 | 183 | if query == "": 184 | st.write("Please enter a question") 185 | st.stop() 186 | 187 | # get tools 188 | print("tools enabled: ", tools) 189 | if len(tools) == 0: 190 | st.write("Please select at least one tool") 191 | st.stop() 192 | 193 | chat_agent.update_tools(tools) 194 | 195 | if img_url: 196 | query += "Image URL: " + img_url 197 | react_response = "" 198 | rag_response = ( 199 | "There was no information in knowledge_graph to answer your question." 200 | ) 201 | chat_agent.add_chat("user", query) 202 | cypher_query = chat_agent.check_KG(query) 203 | if cypher_query: 204 | rag_response, entities = chat_agent.get_routing_agent_response( 205 | query, return_entity=True 206 | ) 207 | chat_agent.add_chat("system", "ReAct agent: " + rag_response, entities) 208 | else: 209 | # get response 210 | react_response = chat_agent.get_routing_agent_response(query) 211 | chat_agent.add_chat("system", "ReAct agent: " + react_response) 212 | 213 | answer = chat_agent.get_response() 214 | st.subheader("Routing Agent Response") 215 | routing_response = "" 216 | with open("data/routing_response.txt", "r") as f: 217 | routing_response = f.read() 218 | st.text(str(routing_response)) 219 | 220 | if cypher_query: 221 | nodes = set() 222 | edges = [] # (node1, node2, [relationships]) 223 | fill_graph(nodes, edges, cypher_query) 224 | 225 | st.subheader("Knoweldge Graph") 226 | st.code("# Current Cypher Used\n" + cypher_query) 227 | st.write("") 228 | st.text("Subgraph:") 229 | graph = create_graph(nodes, edges) 230 | graph_html = graph.generate_html(f"graph_{random.randint(0, 1000)}.html") 231 | components.html(graph_html, height=500, scrolling=True) 232 | else: 233 | st.subheader("Knowledge Graph") 234 | st.text("No information found in the knowledge graph") 235 | 236 | st.subheader("Final Response") 237 | wrapped_text = textwrap.fill(answer, width=80) 238 | st.text(wrapped_text) 239 | 240 | if len(chat_agent.memory_stream) > 0: 241 | # Memory Stream 242 | memory_items = chat_agent.memory_stream.get_memory() 243 | memory_items_dicts = [item.to_dict() for item in memory_items] 244 | df = pd.DataFrame(memory_items_dicts) 245 | st.write("Memory Stream") 246 | st.dataframe(df) 247 | 248 | # Entity Knowledge Store 249 | knowledge_memory_items = chat_agent.entity_knowledge_store.get_memory() 250 | knowledge_memory_items_dicts = [ 251 | item.to_dict() for item in knowledge_memory_items 252 | ] 253 | df_knowledge = pd.DataFrame(knowledge_memory_items_dicts) 254 | st.write("Entity Knowledge Store") 255 | st.dataframe(df_knowledge) 256 | 257 | # top entities 258 | top_entities = chat_agent._select_top_entities() 259 | df_top = pd.DataFrame(top_entities) 260 | st.write("Top 20 Entities") 261 | st.dataframe(df_top) 262 | -------------------------------------------------------------------------------- /streamlit_app/data/entity_knowledge_store.json: -------------------------------------------------------------------------------- 1 | [] -------------------------------------------------------------------------------- /streamlit_app/data/external_response.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kingjulio8238/Memary/b2331a2c0844d66f69acd607b9e4dbaba56552c1/streamlit_app/data/external_response.txt -------------------------------------------------------------------------------- /streamlit_app/data/memory_stream.json: -------------------------------------------------------------------------------- 1 | [] -------------------------------------------------------------------------------- /streamlit_app/data/past_chat.json: -------------------------------------------------------------------------------- 1 | [] -------------------------------------------------------------------------------- /streamlit_app/data/routing_response.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kingjulio8238/Memary/b2331a2c0844d66f69acd607b9e4dbaba56552c1/streamlit_app/data/routing_response.txt -------------------------------------------------------------------------------- /streamlit_app/data/system_persona.txt: -------------------------------------------------------------------------------- 1 | You are tasked with acting as a sophisticated conversational agent, uniquely equipped with a dual-layered memory system. This system consists of a Memory Stream and an Entity Knowledge Store. The Memory Stream captures all entities involved in conversations which are stored in knowledge graphs to infer users’ breadth of knowledge - ranging from questions and answers to their timestamps. Simultaneously, the Entity Knowledge Store captures the frequency and recency of the entities stored in the Memory Stream to infer users’ depth of knowledge. 2 | Your primary role is to deliver a personalized and context-relevant final response that is two sentences or less by using the most recently added user query and ReAct agent response as primary information, and combining that with other information accessible via recent interactions and the top entities stored in the Entity Knowledge Store. You must comprehend the user's persona, incorporating their experiences, preferences, and personal details into your knowledge base outputting the final response. 3 | The final response should not include ReAct agent information outside of the most recently added one unless relevant to the most recently asked question. 4 | You are to interpret and apply the following data structures for personalized responses. Use the latest ReAct agent entry, Entity Knowledge Store, and Contexts to provide a final response to the latest question: 5 | 6 | User Persona: Information about the user's experience, preferences, and personal details. 7 | ReAct agent: Answer generated by the routing agent using its tools. Represents the agent responses to different queries. The most recent ReAct agent response is the focus of the current final response. 8 | Entity Knowledge Store: A record of entities, including a count reflecting their frequency and the date of the last time the entity was inserted into the Memory Stream. If any of these entities, especially those with high counts, are included in the agent response, don't elaborate or explain the concept - you can infer the user is well familiar with the concepts. Do not repeat the Entity Knowledge Store back in the response. 9 | Interaction Keys: 'user' for user questions and 'system' for system-generated answers 10 | 11 | Your responses should be informed, nuanced, and tailored, demonstrating a thorough understanding of the user’s questions and the overarching conversation context. When addressing the user’s latest inquiry, your answer must integrate the current conversation's context, historical interactions, the latest response from the ReAct agent as well as the top chosen entities from the entity knowledge store. Keep final responses to two sentences. -------------------------------------------------------------------------------- /streamlit_app/data/user_persona.txt: -------------------------------------------------------------------------------- 1 | [Personal Information] 2 | My name is Seyeong Han from South Korea. 3 | I identify as a male and 28-year-old master's student at the University of Texas at Austin. 4 | I have worked as an ML Engineer for three years in the Computer Vision area. 5 | I really love sports such as badminton, tennis and workout and spend 4 or 5 days in exercise. 6 | 7 | [Answer Instructions] 8 | I hope you can remember my questions and their answers so that I can leverage my personal past knowledge from you to be helpful in my life. 9 | I always prefer short and concise answers, not over two sentences. -------------------------------------------------------------------------------- /streamlit_app/data/user_persona_template.txt: -------------------------------------------------------------------------------- 1 | [Personal Information] 2 | My name is [name] from [country]. 3 | I identify as a [insert age, sex, profession and any other relevant information to have the system know you best] 4 | I have worked as [insert experience, field involved in, length of experience and any other relevant information to describe your capabilities] 5 | [insert any other personal hobbies outside of work] 6 | 7 | [Answer Instructions] 8 | Remember my past questions and their answers so that you can leverage my past knowledge and experiences to assist me in future interactions. 9 | I always prefer short and concise answers, not over two sentences. --------------------------------------------------------------------------------