├── README.md ├── Week 1 ├── Thursday │ ├── Agent Powered Barbenheimer Application (Assignment Version).ipynb │ ├── Agent Powered Barbenheimer Application-Solution.ipynb │ ├── README.md │ └── requirements.txt └── Tuesday │ ├── Barbie_Retrieval_Augmented_Question_Answering_(RAQA)_Assignment (Assignment Version).ipynb │ ├── README.md │ └── barbie.csv ├── Week 2 ├── Thursday │ ├── Automated Fine-tuning with LLamaIndex.ipynb │ └── README.md └── Tuesday │ ├── LLamaIndex RAQA Tool (Assignment Version).ipynb │ ├── README.md │ ├── barbie_data │ └── barbie.csv │ ├── oppenheimer_data │ └── oppenheimer.csv │ └── requirements.txt ├── Week 3 ├── Thursday │ ├── Deploy │ │ └── README.md │ ├── Evaluation │ │ └── README.md │ └── README.md └── Tuesday │ ├── README.md │ ├── chainlit │ ├── .env.example │ ├── .gitattributes │ ├── .gitignore │ ├── Dockerfile │ ├── README.md │ ├── app.py │ ├── chainlit.md │ └── requirements.txt │ └── fastapi │ ├── .env.sample │ ├── Dockerfile │ ├── README.md │ ├── app.py │ ├── celery_worker.py │ ├── docker-compose.yml │ ├── model_loader.py │ ├── redis_server.py │ ├── requirements.txt │ ├── run.py │ └── utils.py └── Week 4 └── Tuesday ├── README.md ├── langsmith_notebook.ipynb └── wandb_notebook_assignment.ipynb /README.md: -------------------------------------------------------------------------------- 1 |

4 |

5 | 6 | 7 | ##

:wave: Welcome to LLM Ops: LLMs in Production, Cohort 1!

8 | 9 | Access all of the concepts and code directly [YouTube](https://www.youtube.com/playlist?list=PLrSHiQgy4VjGQohoAmgX9VFH52psNOu71) to start building, shipping, and sharing with LLM Ops! 10 | 11 | ### Async Learning Outcomes 12 | 1. Build complex apps with LLM Ops frameworks including LangChain and LlamaIndex. 13 | 2. Understand LLM product development methods, from prompting to retrieval and beyond. 14 | 3. Build, deploy, evaluate, and operate your own end-to-end production RAG system. 15 | 16 | *Note: This course was taught from August, 15, 2023 to September 7, 2023. As such, there may be aspects of the code that require updates for full functionality. Please submit pull requests directly if you find something that you can help improve!* 17 | 18 | ![LLM Ops _Open-Source Flyer](https://github.com/AI-Maker-Space/LLM-Ops-Cohort-1/assets/48775140/348f4b57-584e-4e7c-a365-416a97ac14f3) 19 | 20 | ## LLM Ops Cohort 1, Session Videos 21 | 22 | | Session | Video Link | Code | Slides | 23 | | :-------- | :------------------------------------------------------------------------------------------------ | :-------- | :-------- 24 | | **Session 1: Course Intro, Building Your First RAG App with LangChain, Chainlit, and Hugging Face**
![](https://img.youtube.com/vi/d1Oj5vrTWC4/mqdefault.jpg) | [🎥 Video](https://www.youtube.com/watch?v=d1Oj5vrTWC4&ab_channel=AIMakerspace) | [🐙 Repo](https://github.com/AI-Maker-Space/LLM-Ops-Cohort-1/tree/main/Week%201/Tuesday) | [💻 Slides](https://www.canva.com/design/DAFr0_xXRmM/J_F75d9OrhYgdQ6COF-jRw/edit?utm_content=DAFr0_xXRmM&utm_campaign=designshare&utm_medium=link2&utm_source=sharebutton) | 25 | | **Session 2: Taking RAG to the Next Level with LangChain Agents**
![](https://img.youtube.com/vi/cwLFLlZzFfw/mqdefault.jpg) | [🎥 Video](https://www.youtube.com/watch?v=cwLFLlZzFfw&ab_channel=AIMakerspace) | [🐙 Repo](https://github.com/AI-Maker-Space/LLM-Ops-Cohort-1/tree/main/Week%201/Thursday) | [💻 Slides](https://www.canva.com/design/DAFr0_xXRmM/J_F75d9OrhYgdQ6COF-jRw/edit?utm_content=DAFr0_xXRmM&utm_campaign=designshare&utm_medium=link2&utm_source=sharebutton) | 26 | | **Session 3: Building a More Robust RAG System with LlamaIndex Data Agents**
![](https://img.youtube.com/vi/n4GHCbQxLoY/mqdefault.jpg) | [🎥 Video](https://www.youtube.com/watch?v=n4GHCbQxLoY&ab_channel=AIMakerspace) | [🐙 Repo](https://github.com/AI-Maker-Space/LLM-Ops-Cohort-1/tree/main/Week%202/Tuesday) | [💻 Slides](https://www.canva.com/design/DAFsTCKwPGw/sxlXSvaU3C6HQCNQRftp9g/edit?utm_content=DAFsTCKwPGw&utm_campaign=designshare&utm_medium=link2&utm_source=sharebutton) | 27 | | **Session 4: Project Ideation, Chainlit, and Fine-Tuning of an E2E RAG App with LlamaIndex**
![](https://img.youtube.com/vi/VHWfzkUZLh4/mqdefault.jpg) | [🎥 Video](https://www.youtube.com/watch?v=VHWfzkUZLh4&ab_channel=AIMakerspace) | [🐙 Repo](https://github.com/AI-Maker-Space/LLM-Ops-Cohort-1/tree/main/Week%202/Thursday) | [💻 Slides](https://www.canva.com/design/DAFse6cliLo/AR79LuNWJc-RX5IipCo_6g/edit?utm_content=DAFse6cliLo&utm_campaign=designshare&utm_medium=link2&utm_source=sharebutton) | 28 | | **Session 5: Building a Production-Grade Open-Source RAG System with Llama 2, FastAPI, and Chainlit**
![](https://img.youtube.com/vi/Il2a0mzyJtI/mqdefault.jpg) | [🎥 Video](https://www.youtube.com/watch?v=Il2a0mzyJtI&ab_channel=AIMakerspace) | [🐙 Repo](https://github.com/AI-Maker-Space/LLM-Ops-Cohort-1/tree/main/Week%203/Tuesday) | [💻 Slides](https://www.canva.com/design/DAFs9I5CmQk/aglyznV--Vi4b-1iE5zO1A/edit?utm_content=DAFs9I5CmQk&utm_campaign=designshare&utm_medium=link2&utm_source=sharebutton) | 29 | | **Session 6: Scalable Llama 2 Endpoints for RAG, Evaluation with RAGAS and Eluether AI Harness**
![](https://img.youtube.com/vi/cTu9utOf1I4/mqdefault.jpg) | [🎥 Video](https://www.youtube.com/watch?v=cTu9utOf1I4&ab_channel=AIMakerspace) | [🐙 Repo](https://github.com/AI-Maker-Space/LLM-Ops-Cohort-1/tree/main/Week%203/Thursday) | [💻 Slides](https://www.canva.com/design/DAFtJGudEEk/bgY71J31OQwdrF_tr4aiww/edit?utm_content=DAFtJGudEEk&utm_campaign=designshare&utm_medium=link2&utm_source=sharebutton) | 30 | | **Session 7: Visibility and Observability Tooling for LLM Ops, WandB and LangSmith**
![](https://img.youtube.com/vi/8SMcdlqS070/mqdefault.jpg) | [🎥 Video](https://www.youtube.com/watch?v=8SMcdlqS070&ab_channel=AIMakerspace) | [🐙 Repo](https://github.com/AI-Maker-Space/LLM-Ops-Cohort-1/tree/main/Week%204/Tuesday) | [💻 Slides](https://www.canva.com/design/DAFtJGudEEk/bgY71J31OQwdrF_tr4aiww/edit?utm_content=DAFtJGudEEk&utm_campaign=designshare&utm_medium=link2&utm_source=sharebutton) | 31 | -------------------------------------------------------------------------------- /Week 1/Thursday/Agent Powered Barbenheimer Application (Assignment Version).ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Questioning Barbie and Oppenheimer Through the Use of Agents\n", 8 | "\n", 9 | "In the following notebook we will build an application that queries both the Barbie and Oppenheimer movies Wikipedia pages, as well as their reviews. \n", 10 | "\n", 11 | "The main focus of this notebook is to showcase a brief introduction to Agents.\n", 12 | "\n", 13 | "## Build 🏗️\n", 14 | "\n", 15 | "There are 3 main tasks in this notebook:\n", 16 | "\n", 17 | "1. Contruct a Barbie retriever\n", 18 | "2. Construct an Oppenheimer retriever\n", 19 | "3. Combine the two and allow users to query both resources from a single input through the use of Agents\n", 20 | "\n", 21 | "## Ship 🚢\n", 22 | "\n", 23 | "Based on Tuesday's session - construct a Chainlit (or Gradio) application that allows users to interface with the application.\n", 24 | "\n", 25 | "## Share 🚀\n", 26 | "\n", 27 | "Make a social media post about your final application." 28 | ] 29 | }, 30 | { 31 | "cell_type": "markdown", 32 | "metadata": {}, 33 | "source": [ 34 | "### Dependencies\n", 35 | "\n", 36 | "As always, let's start with some dependencies!" 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "execution_count": null, 42 | "metadata": {}, 43 | "outputs": [], 44 | "source": [ 45 | "!pip install -q -U langchain openai" 46 | ] 47 | }, 48 | { 49 | "cell_type": "code", 50 | "execution_count": null, 51 | "metadata": {}, 52 | "outputs": [], 53 | "source": [ 54 | "import getpass\n", 55 | "import os\n", 56 | "openai_api_key = getpass.getpass(\"Enter your OpenAI API Key: \")\n", 57 | "os.environ[\"OPENAI_API_KEY\"] = openai_api_key" 58 | ] 59 | }, 60 | { 61 | "cell_type": "markdown", 62 | "metadata": {}, 63 | "source": [ 64 | "### LLM \n", 65 | "\n", 66 | "We will be leveraging OpenAI's `gpt-3.5-turbo` throughout the notebook, and we can keep it consistent throughout!" 67 | ] 68 | }, 69 | { 70 | "cell_type": "code", 71 | "execution_count": null, 72 | "metadata": {}, 73 | "outputs": [], 74 | "source": [ 75 | "from langchain.chat_models import ChatOpenAI\n", 76 | "llm = ChatOpenAI(model=\"\", temperature = 0)" 77 | ] 78 | }, 79 | { 80 | "cell_type": "markdown", 81 | "metadata": {}, 82 | "source": [ 83 | "### Data Collection and Transformation\n", 84 | "\n", 85 | "We'll be leveraging the `WikipediaLoader` tool to collect information from Wikipedia. \n", 86 | "\n", 87 | "Be sure to set the `doc_content_chars_max` parameter so that you capture the *entire* Wikipedia article content." 88 | ] 89 | }, 90 | { 91 | "cell_type": "code", 92 | "execution_count": null, 93 | "metadata": {}, 94 | "outputs": [], 95 | "source": [ 96 | "!pip install -q -U wikipedia" 97 | ] 98 | }, 99 | { 100 | "cell_type": "code", 101 | "execution_count": null, 102 | "metadata": {}, 103 | "outputs": [], 104 | "source": [ 105 | "from langchain.document_loaders import WikipediaLoader, CSVLoader\n", 106 | "\n", 107 | "barbie_wikipedia_docs = WikipediaLoader(\n", 108 | " query=\"Barbie (film)\", \n", 109 | " load_max_docs= # YOUR CODE HERE, \n", 110 | " doc_content_chars_max=### YOUR CODE HERE\n", 111 | " ).load()\n", 112 | "\n", 113 | "barbie_csv_docs = CSVLoader(\n", 114 | " file_path=### YOUR CODE HERE, \n", 115 | " source_column=### YOUR CODE HERE\n", 116 | " ).load()" 117 | ] 118 | }, 119 | { 120 | "cell_type": "markdown", 121 | "metadata": {}, 122 | "source": [ 123 | "Since we'll be using same format source documentation separated by topic, we can save ourselves some extra effort and set up our splitters once. \n", 124 | "\n", 125 | "We're going to leverage the `RecursiveCharacterTextSplitter` again, this time paying close attention to the format our Wikipedia articles and reviews are in so we can be sure to chunk them appropritately. \n", 126 | "\n", 127 | "> HINT: You can pass a list of separators when you intialize your `RecursiveTextSplitter`! They are acted on in order of element 0 -> element len(list).\n", 128 | "\n", 129 | "RELEVANT DOCS:\n", 130 | "- [`RecursiveCharacterTextSplitter`](https://api.python.langchain.com/en/latest/text_splitter/langchain.text_splitter.RecursiveCharacterTextSplitter.html#langchain.text_splitter.RecursiveCharacterTextSplitter)" 131 | ] 132 | }, 133 | { 134 | "cell_type": "code", 135 | "execution_count": null, 136 | "metadata": {}, 137 | "outputs": [], 138 | "source": [ 139 | "from langchain.text_splitter import RecursiveCharacterTextSplitter\n", 140 | "\n", 141 | "wikipedia_text_splitter = RecursiveCharacterTextSplitter(\n", 142 | " chunk_size = ### YOUR CODE HERE,\n", 143 | " chunk_overlap = ### YOUR CODE HERE,\n", 144 | " length_function = ### YOUR CODE HERE,\n", 145 | " is_separator_regex= False,\n", 146 | " separators = ### YOUR CODE HERE # keep headings, then paragraphs, then sentences\n", 147 | ")\n", 148 | "\n", 149 | "csv_text_splitter = RecursiveCharacterTextSplitter(\n", 150 | " chunk_size = ### YOUR CODE HERE,\n", 151 | " chunk_overlap = ### YOUR CODE HERE,\n", 152 | " length_function = ### YOUR CODE HERE,\n", 153 | " is_separator_regex= False,\n", 154 | " separators = ### YOUR CODE HERE # keep paragraphs, then sentences\n", 155 | ")\n", 156 | "\n", 157 | "chunked_barbie_wikipedia_docs = ### YOUR CODE HERE\n", 158 | "chunked_barbie_csv_docs = ### YOUR CODE HERE" 159 | ] 160 | }, 161 | { 162 | "cell_type": "markdown", 163 | "metadata": {}, 164 | "source": [ 165 | "#### Retrieval and Embedding Strategy\n", 166 | "\n", 167 | "We've already discussed the useful application of `CacheBackedEmbeddings`, so let's do it again!\n", 168 | "\n", 169 | "RELEVANT DOCS:\n", 170 | "- [`CacheBackedEmbeddings`](https://api.python.langchain.com/en/latest/embeddings/langchain.embeddings.cache.CacheBackedEmbeddings.html#langchain-embeddings-cache-cachebackedembeddings)" 171 | ] 172 | }, 173 | { 174 | "cell_type": "code", 175 | "execution_count": null, 176 | "metadata": {}, 177 | "outputs": [], 178 | "source": [ 179 | "!pip install -q -U rank_bm25 tiktoken faiss-cpu" 180 | ] 181 | }, 182 | { 183 | "cell_type": "code", 184 | "execution_count": null, 185 | "metadata": {}, 186 | "outputs": [], 187 | "source": [ 188 | "from langchain.retrievers import BM25Retriever, EnsembleRetriever\n", 189 | "from langchain.vectorstores import FAISS\n", 190 | "from langchain.embeddings.openai import OpenAIEmbeddings\n", 191 | "from langchain.embeddings import CacheBackedEmbeddings\n", 192 | "from langchain.storage import LocalFileStore\n", 193 | "\n", 194 | "# set up cached embeddings store\n", 195 | "store = ### YOUR CODE HERE\n", 196 | "\n", 197 | "core_embeddings_model = ### YOUR CODE HERE\n", 198 | "\n", 199 | "embedder = ### YOUR CODE HERE" 200 | ] 201 | }, 202 | { 203 | "cell_type": "markdown", 204 | "metadata": {}, 205 | "source": [ 206 | "We'll implement a `FAISS` vectorstore, and create a retriever from it." 207 | ] 208 | }, 209 | { 210 | "cell_type": "code", 211 | "execution_count": null, 212 | "metadata": {}, 213 | "outputs": [], 214 | "source": [ 215 | "barbie_csv_faiss_retriever = ### YOUR CODE HERE\n" 216 | ] 217 | }, 218 | { 219 | "cell_type": "markdown", 220 | "metadata": {}, 221 | "source": [ 222 | "There are a number of excellent options to retrieve documents - we'll be looking at an additional example today, which is called the `EnsembleRetriever`.\n", 223 | "\n", 224 | "The method this is using is outlined in [this paper](https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf).\n", 225 | "\n", 226 | "The brief explanation is:\n", 227 | "\n", 228 | "1. We collect results from two different retrieval methods over the same corpus\n", 229 | "2. We apply a reranking algorithm to rerank our source documents to be the *most relevant* without losing specific or potentially low-ranked information rich documents\n", 230 | "3. We feed the top-k results into the LLM with our query as context.\n", 231 | "\n", 232 | "> HINT: Your weight list should be of type `List[float]` and the `sum(List[float])` should be `1`." 233 | ] 234 | }, 235 | { 236 | "cell_type": "code", 237 | "execution_count": null, 238 | "metadata": {}, 239 | "outputs": [], 240 | "source": [ 241 | "# set up BM25 retriever\n", 242 | "barbie_wikipedia_bm25_retriever = BM25Retriever.from_documents(\n", 243 | " ### YOUR CODE HERE\n", 244 | ")\n", 245 | "barbie_wikipedia_bm25_retriever.k = 1\n", 246 | "\n", 247 | "# set up FAISS vector store\n", 248 | "barbie_wikipedia_faiss_store = FAISS.from_documents(\n", 249 | " ### YOUR CODE HERE,\n", 250 | " ### YOUR CODE HERE\n", 251 | ")\n", 252 | "barbie_wikipedia_faiss_retriever = barbie_wikipedia_faiss_store.as_retriever(search_kwargs={\"k\": 1})\n", 253 | "\n", 254 | "# set up ensemble retriever\n", 255 | "barbie_ensemble_retriever = EnsembleRetriever(\n", 256 | " retrievers=### YOUR CODE HERE,\n", 257 | " weights= ### YOUR CODE HERE # should sum to 1\n", 258 | ")" 259 | ] 260 | }, 261 | { 262 | "cell_type": "markdown", 263 | "metadata": {}, 264 | "source": [ 265 | "#### Retrieval Agent\n", 266 | "\n", 267 | "We can create a simple conversational retrieval Agent by using the built-ins provided by LangChain!\n", 268 | "\n", 269 | "> HINT: Be sure to provide good natural language descriptions of what the tool should be used for to get the best results.\n", 270 | "\n", 271 | "RELEVANT DOCS:\n", 272 | "- [`create_retriever_tool`](https://api.python.langchain.com/en/latest/agents/langchain.agents.agent_toolkits.conversational_retrieval.tool.create_retriever_tool.html#langchain.agents.agent_toolkits.conversational_retrieval.tool.create_retriever_tool)" 273 | ] 274 | }, 275 | { 276 | "cell_type": "code", 277 | "execution_count": null, 278 | "metadata": {}, 279 | "outputs": [], 280 | "source": [ 281 | "from langchain.agents.agent_toolkits import create_retriever_tool\n", 282 | "\n", 283 | "barbie_wikipedia_retrieval_tool = create_retriever_tool(\n", 284 | " ### YOUR CODE HERE \n", 285 | " ### YOUR CODE HERE\n", 286 | " ### YOUR CODE HERE\n", 287 | ")\n", 288 | "\n", 289 | "barbie_csv_retrieval_tool = create_retriever_tool(\n", 290 | " ### YOUR CODE HERE\n", 291 | " ### YOUR CODE HERE\n", 292 | " ### YOUR CODE HERE\n", 293 | ")\n", 294 | "\n", 295 | "barbie_retriever_tools = ### YOUR CODE HERE" 296 | ] 297 | }, 298 | { 299 | "cell_type": "markdown", 300 | "metadata": {}, 301 | "source": [ 302 | "Now that we've created our tools, we can combined them into an agent!\n", 303 | "\n", 304 | "RELEVANT DOCS:\n", 305 | "- [`create_conversational_retrieval_agent`](https://api.python.langchain.com/en/latest/agents/langchain.agents.agent_toolkits.conversational_retrieval.openai_functions.create_conversational_retrieval_agent.html#langchain.agents.agent_toolkits.conversational_retrieval.openai_functions.create_conversational_retrieval_agent)" 306 | ] 307 | }, 308 | { 309 | "cell_type": "code", 310 | "execution_count": null, 311 | "metadata": {}, 312 | "outputs": [], 313 | "source": [ 314 | "from langchain.agents.agent_toolkits import create_conversational_retrieval_agent\n", 315 | "\n", 316 | "barbie_retriever_agent_executor = # YOUR CODE HERE" 317 | ] 318 | }, 319 | { 320 | "cell_type": "code", 321 | "execution_count": null, 322 | "metadata": {}, 323 | "outputs": [], 324 | "source": [ 325 | "barbie_retriever_agent_executor({\"input\" : \"Did people like Barbie, or did they find it too Philosphical? If they did, can you tell me why the movie is so Philosophical?\"})" 326 | ] 327 | }, 328 | { 329 | "cell_type": "code", 330 | "execution_count": null, 331 | "metadata": {}, 332 | "outputs": [], 333 | "source": [ 334 | "barbie_retriever_agent_executor({\"input\" : \"What is a very quick summary of the plot of the Barbie movie?\"})" 335 | ] 336 | }, 337 | { 338 | "cell_type": "markdown", 339 | "metadata": {}, 340 | "source": [ 341 | "### Oppenheimer Retrieval System\n", 342 | "\n", 343 | "We're going to repourpose some of what we created previously, but this time we'll explore a different multi-source retrieval system." 344 | ] 345 | }, 346 | { 347 | "cell_type": "code", 348 | "execution_count": null, 349 | "metadata": {}, 350 | "outputs": [], 351 | "source": [ 352 | "oppenheimer_wikipedia_docs = ### YOUR CODE HERE\n", 353 | "\n", 354 | "oppenheimer_csv_docs = ### YOUR CODE HERE" 355 | ] 356 | }, 357 | { 358 | "cell_type": "code", 359 | "execution_count": null, 360 | "metadata": {}, 361 | "outputs": [], 362 | "source": [ 363 | "chunked_opp_wikipedia_docs = ### YOUR CODE HERE\n", 364 | "chunked_opp_csv_docs = ### YOUR CODE HERE" 365 | ] 366 | }, 367 | { 368 | "cell_type": "code", 369 | "execution_count": null, 370 | "metadata": {}, 371 | "outputs": [], 372 | "source": [ 373 | "opp_csv_faiss_retriever = ### YOUR CODE HERE\n", 374 | "\n", 375 | "# set up BM25 retriever\n", 376 | "opp_wikipedia_bm25_retriever = ### YOUR CODE HERE\n", 377 | "opp_wikipedia_bm25_retriever.k = 1\n", 378 | "\n", 379 | "# set up FAISS vector store\n", 380 | "opp_wikipedia_faiss_store = ### YOUR CODE HERE\n", 381 | "opp_wikipedia_faiss_retriever = opp_wikipedia_faiss_store.as_retriever(search_kwargs={\"k\": 1})\n", 382 | "\n", 383 | "# set up ensemble retriever\n", 384 | "opp_ensemble_retriever = ### YOUR CODE HERE" 385 | ] 386 | }, 387 | { 388 | "cell_type": "markdown", 389 | "metadata": {}, 390 | "source": [ 391 | "#### Multi-source chain\n", 392 | "\n", 393 | "We're going to allow the LLM to decide which information is most -> least valuable.\n", 394 | "\n", 395 | "The way we'll do this is with LangChain's rather powerful \"Expression Language\"!\n", 396 | "\n", 397 | "> HINT: You can leverage [this](https://python.langchain.com/docs/use_cases/question_answering/how_to/multiple_retrieval) resource if you get stuck - but experiment with different prompts/formats." 398 | ] 399 | }, 400 | { 401 | "cell_type": "code", 402 | "execution_count": null, 403 | "metadata": {}, 404 | "outputs": [], 405 | "source": [ 406 | "from langchain.prompts import ChatPromptTemplate\n", 407 | "\n", 408 | "system_message = \"\"\"Use the information from the below two sources to answer any questions.\n", 409 | "\n", 410 | "Source 1: public user reviews about the Oppenheimer movie\n", 411 | "\n", 412 | "{source1}\n", 413 | "\n", 414 | "\n", 415 | "Source 2: the wikipedia page for the Oppenheimer movie including the plot summary, cast, and production information\n", 416 | "\n", 417 | "{source2}\n", 418 | "\n", 419 | "\"\"\"\n", 420 | "\n", 421 | "prompt = ChatPromptTemplate.from_messages([(\"system\", system_message), (\"human\", \"{question}\")])" 422 | ] 423 | }, 424 | { 425 | "cell_type": "code", 426 | "execution_count": null, 427 | "metadata": {}, 428 | "outputs": [], 429 | "source": [ 430 | "oppenheimer_multisource_chain = {\n", 431 | " \"source1\": (lambda x: x[\"question\"]) | opp_ensemble_retriever,\n", 432 | " \"source2\": (lambda x: x[\"question\"]) | opp_csv_faiss_retriever,\n", 433 | " \"question\": lambda x: x[\"question\"],\n", 434 | "} | prompt | llm" 435 | ] 436 | }, 437 | { 438 | "cell_type": "code", 439 | "execution_count": null, 440 | "metadata": {}, 441 | "outputs": [], 442 | "source": [ 443 | "oppenheimer_multisource_chain.invoke({\"question\" : \"What did people think of the Oppenheimer movie?\"})" 444 | ] 445 | }, 446 | { 447 | "cell_type": "markdown", 448 | "metadata": {}, 449 | "source": [ 450 | "# Agent Creation\n", 451 | "\n", 452 | "Now we can finally start building our Agent!\n", 453 | "\n", 454 | "The first thing we'll need to do is provide our Agent a Toolbelt. (list of tools). Much like Batman, our LLM-powered Agent can use these tools as it sees fit. \n", 455 | "\n", 456 | "While the examples we're constructing in this notebook are straightforward for brevity and simplicities sake - there is no limit to what you can build with Agents, as we'll see as we progress through the program.\n", 457 | "\n", 458 | "So, let's begin by setting up our Tools!\n", 459 | "\n", 460 | "You'll notice that we have to set up a function to allow our `OppenheimerInfo` tool to interface with the Agent - this is due to it have a specific required input. Creating custom tools is a pattern that you'll want to grow acustomed to as you use LangChain more and more." 461 | ] 462 | }, 463 | { 464 | "cell_type": "code", 465 | "execution_count": null, 466 | "metadata": {}, 467 | "outputs": [], 468 | "source": [ 469 | "from langchain.agents import Tool\n", 470 | "\n", 471 | "def query_oppenheimer(input):\n", 472 | " return oppenheimer_multisource_chain.invoke({\"question\" : input})\n", 473 | "\n", 474 | "tools = [\n", 475 | " Tool(\n", 476 | " name = ### YOUR CODE HERE,\n", 477 | " func=### YOUR CODE HERE,\n", 478 | " description=### YOUR CODE HERE\n", 479 | " ),\n", 480 | " Tool(\n", 481 | " name = ### YOUR CODE HERE,\n", 482 | " func= ### YOUR CODE HERE,\n", 483 | " description= ### YOUR CODE HERE\n", 484 | " ),\n", 485 | "]" 486 | ] 487 | }, 488 | { 489 | "cell_type": "markdown", 490 | "metadata": {}, 491 | "source": [ 492 | "Now that we've set up our Agents toolbelt, let's set up the LLM that will be powering it!\n", 493 | "\n", 494 | "I would suggest playing around with these prompts - and experiments to find what works best for you.\n", 495 | "\n", 496 | "RELEVANT DOCS:\n", 497 | "- [`ZeroShotAgent`](https://api.python.langchain.com/en/latest/agents/langchain.agents.mrkl.base.ZeroShotAgent.html#langchain-agents-mrkl-base-zeroshotagent)" 498 | ] 499 | }, 500 | { 501 | "cell_type": "code", 502 | "execution_count": null, 503 | "metadata": {}, 504 | "outputs": [], 505 | "source": [ 506 | "from langchain.agents import ZeroShotAgent, AgentExecutor\n", 507 | "\n", 508 | "prefix = \"\"\"Have a conversation with a human, answering the following questions as best you can. You have access to the following tools:\"\"\"\n", 509 | "suffix = \"\"\"Begin!\"\n", 510 | "\n", 511 | "Question: {input}\n", 512 | "{agent_scratchpad}\"\"\"\n", 513 | "\n", 514 | "prompt = ZeroShotAgent.create_prompt(\n", 515 | " ### YOUR CODE HERE\n", 516 | " ### YOUR CODE HERE\n", 517 | " ### YOUR CODE HERE\n", 518 | " ### YOUR CODE HERE\n", 519 | ")" 520 | ] 521 | }, 522 | { 523 | "cell_type": "code", 524 | "execution_count": null, 525 | "metadata": {}, 526 | "outputs": [], 527 | "source": [ 528 | "from langchain import LLMChain\n", 529 | "\n", 530 | "llm_chain = ### YOUR CODE HERE" 531 | ] 532 | }, 533 | { 534 | "cell_type": "markdown", 535 | "metadata": {}, 536 | "source": [ 537 | "All that's left to do now is create our `ZeroShotAgent` and our `AgentExecutor`, which are the \"reasoner\" and \"actor\" halfs of the `ReAct` method of Agent implementation.\n", 538 | "\n", 539 | "Read all about the `ReAct` framework [here](https://react-lm.github.io/)" 540 | ] 541 | }, 542 | { 543 | "cell_type": "code", 544 | "execution_count": null, 545 | "metadata": {}, 546 | "outputs": [], 547 | "source": [ 548 | "barbenheimer_agent = ZeroShotAgent(\n", 549 | " llm_chain=### YOUR AGENTS, \n", 550 | " tools=### YOUR AGENTS, \n", 551 | " verbose=### YOUR AGENTS)\n", 552 | "\n", 553 | "barbenheimer_agent_chain = AgentExecutor.from_agent_and_tools(\n", 554 | " agent=### YOUR AGENTS, \n", 555 | " tools=### YOUR AGENTS, \n", 556 | " verbose=### YOUR AGENTS)" 557 | ] 558 | }, 559 | { 560 | "cell_type": "markdown", 561 | "metadata": {}, 562 | "source": [ 563 | "## Conclusion\n", 564 | "\n", 565 | "All that is left to do now, is feed inputs to your Agent and watch it work!\n", 566 | "\n", 567 | "Remember to use the `{\"input\" : \"YOUR QUERY HERE\"}` format when prompting the Agent." 568 | ] 569 | }, 570 | { 571 | "cell_type": "code", 572 | "execution_count": null, 573 | "metadata": {}, 574 | "outputs": [], 575 | "source": [ 576 | "barbenheimer_agent_chain.invoke({\"input\" : \"What did people like about the Barbie movie?\"})" 577 | ] 578 | }, 579 | { 580 | "cell_type": "code", 581 | "execution_count": null, 582 | "metadata": {}, 583 | "outputs": [], 584 | "source": [ 585 | "barbenheimer_agent_chain.run({\"input\" : \"What did people like about the Oppenheimer movie?\"})" 586 | ] 587 | }, 588 | { 589 | "cell_type": "code", 590 | "execution_count": null, 591 | "metadata": {}, 592 | "outputs": [], 593 | "source": [ 594 | "barbenheimer_agent_chain.run({\"input\" : \"Did the movies Barbie and Oppenheimer share similar themes or ideas?\"})" 595 | ] 596 | }, 597 | { 598 | "cell_type": "markdown", 599 | "metadata": {}, 600 | "source": [ 601 | "## Next Steps\n", 602 | "\n", 603 | "It's time to build a Chainlit (or Gradio) application and host it on Hugging Face Spaces! :ship:" 604 | ] 605 | } 606 | ], 607 | "metadata": { 608 | "kernelspec": { 609 | "display_name": "barbenheimer-week-1", 610 | "language": "python", 611 | "name": "python3" 612 | }, 613 | "language_info": { 614 | "codemirror_mode": { 615 | "name": "ipython", 616 | "version": 3 617 | }, 618 | "file_extension": ".py", 619 | "mimetype": "text/x-python", 620 | "name": "python", 621 | "nbconvert_exporter": "python", 622 | "pygments_lexer": "ipython3", 623 | "version": "3.11.4" 624 | }, 625 | "orig_nbformat": 4 626 | }, 627 | "nbformat": 4, 628 | "nbformat_minor": 2 629 | } 630 | -------------------------------------------------------------------------------- /Week 1/Thursday/README.md: -------------------------------------------------------------------------------- 1 | # Questioning Barbie and Oppenheimer Through the Use of Agents 2 | 3 | In the following notebook we will build an application that queries both the Barbie and Oppenheimer movies Wikipedia pages, as well as their reviews. 4 | 5 | The main focus of this notebook is to showcase a brief introduction to Agents. 6 | 7 | ## Build 🏗️ 8 | 9 | There are 3 main tasks in this notebook: 10 | 11 | 1. Construct a Barbie retriever 12 | 2. Construct an Oppenheimer retriever 13 | 3. Combine the two and allow users to query both resources from a single input through the use of Agents 14 | 15 | ## Ship 🚢 16 | 17 | Based on Tuesday's session - construct a Chainlit (or Gradio) application that allows users to interface with the application. 18 | 19 | ## Share 🚀 20 | 21 | Make a social media post about your final application. 22 | 23 | ## System Architecture Diagrams 24 | 25 | ### ReAct Agents 26 | 27 | We'll condense the RAQA diagram into a single component. 28 | 29 | ![image](https://i.imgur.com/LqIln4g.png) 30 | 31 | Now, let's look at the Agent diagram (in a slightly simplified diagram to avoid arrow-pocolypse) 32 | 33 | ![image](https://i.imgur.com/H55v09C.png) 34 | 35 | ### Ensemble Retrieval 36 | 37 | [`EnsembleRetriever`](https://api.python.langchain.com/en/latest/retrievers/langchain.retrievers.ensemble.EnsembleRetriever.html#langchain.retrievers.ensemble.EnsembleRetriever) 38 | 39 | Leverages the [RRF](https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf) reranking algorithm to combine sparse and dense search results for increased effectiveness for relevant document retrieval. 40 | 41 | ![image](https://i.imgur.com/mn4jXAz.png) 42 | 43 | ### Multi-query Retrieval 44 | 45 | We can query multiple sources and then allow the LLM to parse the results from both in order to decide the final answer. 46 | 47 | ![image](https://i.imgur.com/g3vGp4P.png) 48 | 49 | -------------------------------------------------------------------------------- /Week 1/Thursday/requirements.txt: -------------------------------------------------------------------------------- 1 | ipykernel -------------------------------------------------------------------------------- /Week 1/Tuesday/README.md: -------------------------------------------------------------------------------- 1 | # Week 1: Tuesday Session 2 | 3 | In today's assignment, we'll be creating a simplified RAQA system using OpenAI, LangChain, FAISS, and Chainlit. 4 | 5 | There are 3 main sections to this assignment: 6 | 7 | ## Build 🏗️ 8 | 9 | Following the provided notebook - you'll build a simplified RAQA system to ask questions over some reviews of the movie Barbie. 10 | 11 | This will take you through three major steps: 12 | 13 | 1. Collecting and parsing data 14 | 2. Building an Index (VectorStore) 15 | 3. Chaining your Index together with an OpenAI LLM model. 16 | 17 | ### Deliverables 18 | 19 | - Completed Notebook 20 | - System Diagram of the RAQA Application 21 | 22 | ## Ship 🚢 23 | 24 | Duplicate the provided [Hugging Face Space](https://huggingface.co/spaces/ai-maker-space/Barbie-RAQA-Application-Chainlit-Demo) 25 | 26 | ### Deliverables 27 | 28 | - A short Loom of the space, and a 1min. walkthrough of the application in full 29 | 30 | ## Share 🚀 31 | 32 | Make a social media post about your final application! 33 | 34 | ### Deliverables 35 | 36 | - Make a post on any social media platform about what you built! 37 | -------------------------------------------------------------------------------- /Week 1/Tuesday/barbie.csv: -------------------------------------------------------------------------------- 1 | ,Review_Date,Author,Rating,Review_Title,Review,Review_Url 2 | 0,21 July 2023,LoveofLegacy,6," Beautiful film, but so preachy 3 | ","Margot does the best with what she's given, but this film was very disappointing to me. It was marketed as a fun, quirky satire with homages to other movies. It started that way, but ended with over-dramatized speeches and an ending that clearly tried to make the audience feel something, but left everyone just feeling confused. And before you say I'm a crotchety old man, I'm a woman in my 20s, so I'm pretty sure I'm this movie's target audience. The saddest part is there were parents with their kids in the theater that were victims of the poor marketing, because this is not a kid's movie. Overall, the humor was fun on occasion and the film is beautiful to look at, but the whole concept falls apart in the second half of the film and becomes a pity party for the ""strong"" woman.",/review/rw9204303/?ref_=tt_urv 4 | 1,22 July 2023,imseeg,7," 3 reasons FOR seeing it and 1 reason AGAINST. 5 | ",The first reason to go see it:,/review/rw9207325/?ref_=tt_urv 6 | 2,22 July 2023,Natcat87,6," Too heavy handed 7 | ","As a woman that grew up with Barbie, I was very excited for this movie. I was curious to see how they would evolve the ""stereotypical Barbie"" into something more. But the messaging in this movie was so heavy handed that it completely lost the plot. I consider myself a proponent of gender equality, and this ain't the way to get it.",/review/rw9206938/?ref_=tt_urv 8 | 3,31 July 2023,ramair350,10," As a guy I felt some discomfort, and that's ok. 9 | ","As much as it pains me to give a movie called ""Barbie"" a 10 out of 10, I have to do so. It is so brilliantly handled and finely crafted, I have to give the filmakers credit. Yes, I am somewhat conservative person and former law enforcement officer. I'm a guy. I like guy things. Hell I even enjoyed the Battleship movie a few years ago (an absolutely ridiculous but fun romp of an action film). But I also like to experience other perspectives. And man oh man does this movie deliver that in spades - pretty much encapsulated everything my wife has tried to convey about her entire career and life experience wrapped up into two hours! The humor, the sets, the acting, and the ability to weave the current narrative into the story was just perfect. I don't agree with some of the points of the movie, but again, that's ok. This movie wasn't designed to give a balanced perspective of men versus women; it is a no-holds-barred unapologetic crazy ride of a rant about the real issues that women have faced since they were ""allowed"" to have ""real jobs"" and do the same things as men. Give me a well done film that is a blast to watch, that makes you think, and that was done from a place of creativity, passion, and attention to detail, and I'll call it what it is: a 10 out of 10 masterpiece.",/review/rw9230034/?ref_=tt_urv 10 | 4,24 July 2023,heatherhilgers,9," A Technicolor Dream 11 | ","Wow, this movie was a love letter to cinema. From the Kubrick reference at the opening, to the soundstage being 100% real - no cgi for the sets. There were dance sequences like in the olden days (think Grease) and the costuming , wow the costuming. Greta Gerwig dug deep to make this masterpiece using all of her technical chops along the way. The casting director also nailed it, everyone was wonderful.",/review/rw9211546/?ref_=tt_urv 12 | 5,21 July 2023,G-Joshua-Benjamin,6," My mom and I saw this yesterday. Here are my thoughts. 13 | ",I don't know if I put spoilers in here. I am super worn out. Haha So I will just put that I did.,/review/rw9203627/?ref_=tt_urv 14 | 6,21 July 2023,Sleepin_Dragon,8," Well this really did come as a surprise. 15 | ","It pains me to say it, but I enjoyed this movie so much more then I was expecting to, musical numbers, humour, there truly is something for the whole family yo enjoy.",/review/rw9205226/?ref_=tt_urv 16 | 7,20 July 2023,portraitofaladyonfire,6," Brilliant observations, but social depth is missing 17 | ","Greta Gerwig and Noah Baumbach have a knack for incorporating the many little observations of life, whether funny or deeply interpersonal, into a screenplay. Gerwig manages to direct them with great visuals and clever transitions.",/review/rw9202861/?ref_=tt_urv 18 | 8,24 July 2023,coxaneesa,8," It was depressing 19 | ",I thought this would be so much different. The ending made me feel sad and empty. Ken and Barbie don't end up together and they made Ken seem like a complete idiot. Actually they presented men as being dumb throughout the whole movie. And I'm not a man but that's just not true. It was depressing and sad and never the way Barbie was created for. I know a lot of people won't agree with me and that's okay but it was just pointless going to see it. The only good thing is that the Barbie aesthetic was beautifully done.,/review/rw9212241/?ref_=tt_urv 20 | 9,19 July 2023,hamsterination,6," It could have been so much better... 21 | ","The film's universe and settings are fantastic. The casting is really good too, with Gosling excelling in the role of Ken.",/review/rw9200202/?ref_=tt_urv 22 | 10,19 July 2023,HabibieHakim123,8," Barbie Is A Weirdly Fun Movie! 23 | ","8.5/10 24 | While i'm not so sure at first, the movie kept getting even more fun, entertaining, and definitely better, also surprisingly deal with a legit serious stuff, Barbie is a weirdly fun movie that fills with this very interesting concept, definitely the first time that's ever done, Greta Gerwig has created this whole new style of filmmaking specifically for Barbie, from the intentionally weird yet creative editing, some awkward and cringe scene, i found the comedy so funny instead of cringe, Barbie is one of the most original movie of the year and also one of the most original movie i've seen in a while, we all know Margot Robbie and Ryan Gosling is gonna carry the movie and they are, but Will Ferrell, Simu Liu, and the whole rest of the cast were also great and entertaining, the soundtrack was just great, except Nicki Minaj and Ice Spice ""Barbie World"" song that are just absolutely terrible, but Billie Eilish ""What Was I Made For?"" tune that kept haunting in the background until it finally get the perfect scene to played it was really the best thing because that song was just beautiful and emotional, anyway, Barbie is a lot of fun, while there is still some noticeable flawed, most of them i found it funny, entertaining, colourful, creative, and fun, pure fun, it's something that i probably will love it even more the second time i see it, but as of right now, Barbie was a good time, definitely recommended and with this movie somehow it's actually possible to have a live-action Barbie movie, all you need to have is a good writer, good director, and good cast, and they delivers mostly.",/review/rw9200627/?ref_=tt_urv 25 | 11,20 July 2023,fscsgxp,6," Amazing Cast & Set, but the political message was too strong 26 | ","I've been excited for this movie for over a year. The casting was perfect and the acting was amazing! The set was next level, the tiny details were all there and even the highest barbie fan couldn't ask for anything more from it. It is PERFECTION.",/review/rw9201948/?ref_=tt_urv 27 | 12,20 July 2023,MissSimonetta,4," The marketing was more entertaining than the actual movie 28 | ","I went to see this today, everyone in my group dressed in pink and ready to at least be entertained by pastel-colored camp for two hours. BARBIE kind of succeeds there: I loved the costuming and sets of Barbieland, and Margot Robie is such a fantastic actress, the perfect embodiment of the optimistic, she-can-do-it-all Barbie. Ryan Gosling was a scene-stealer with his horse obsession and faux mink coat.",/review/rw9203156/?ref_=tt_urv 29 | 13,19 July 2023,jpismyname,8," Fun and surprisingly touching 30 | ","I was honestly doubting this movie at first, but surprisingly I find myself really liking it quite a lot.",/review/rw9200527/?ref_=tt_urv 31 | 14,29 July 2023,cmdown-50506,9," Now I am become Barbie Girl, the enjoyer of Barbie worlds 32 | ","I have been waiting for this to release for so long and I finally got to watch it today. I absolutely loved it! It was fun, it had Margot Robbie (and extra stuff for Tarantino's out there if you know what I mean) And it had some absolute bangers! What more could you want from a new summer classic?",/review/rw9223162/?ref_=tt_urv 33 | 15,23 July 2023,Revuer223,6," Could Have Been Great. 2nd Half Brings It Down. 34 | ","The quality, the humor, and the writing of the movie is fun for a while. It's quirky and it's unique. When they get into the weeds and try to explore deeper themes, the movie is a miss. The middle expositional phase of the movie, I must say, is a bore.",/review/rw9210202/?ref_=tt_urv 35 | 16,4 August 2023,thePopcornExplorer,9," I don't see how it was preachy 36 | ","In this day and age with society overall becoming more and more polarized and our leaders around the world doing nothing to improve dialogue, it seems that people became broken records. Now everything in the movies and TV shows is perceived as preachy, not everything is a conspiracy, people need to consider that movies and TV shows at its core, and specially the good ones, are about sharing perspectives and allow the audience, if their minds are opened to it, to acknowledge all sides of the stories.",/review/rw9237409/?ref_=tt_urv 37 | 17,21 July 2023,andermic18,6," No Direction 38 | ","The movie was very funny and really enjoyable to laugh at with the full theatre. However, the messages of the movie were the problem. I was never really sure what I was supposed to take away, there was nothing about finding equality or love it was all about how every man cat falls every woman or women can't be anything. It was really silly because there was no accurate reflection of America at any single point except for Barbie getting called a Fascist for no reason by a 14 year old. I enjoyed how they called out women for hating women and how they really tried to preach empowerment and the ability to be anything, but at the same time there was so much resentment and they ended the movie by reinstating hate. The majority of the movie was hating men as much as possible. That's just whatever because what really matters is the story. Well it fell short on that mark and it was really disappointing. The pacing was horrible, the villain won and was pretty irrelevant in the long run, the story was all over the place, and it is totally not for kids. I think the worst part is how disappointed I was at the opportunity to really make something special and it just wasn't. 6 points for all the laughs and fun I had and I would expect 6-7 to be an appropriate rating.",/review/rw9203761/?ref_=tt_urv 39 | 18,22 July 2023,finnconnelly-63017,10," Ken out of ten 40 | ","Wow. I did not see this masterpiece coming. And that's not a joke. Somehow, Greta Gerwig took Barbie, and made a movie that tackles some really tough social issues....well? It's crazy. But wow, does it work. The story is very mature, thinker of a plot. Once you're done laughing your butt off, you'll get really emotional. The production design is so on point. The whole movie is pink everything, but it's never polarizing. All the shades melt together in a very pleasing way. And of course the cast! Margot Robbie was born to play Barbie, and she hits a home run, but Ryan Gosling. This dude rocks as Ken. He knocked it all the way around the world and hit himself in the back of the head. He steals every scene he's in, and I would be so on board with a Ken spin-off. America Ferrara and Simu Liu are also great in their respective roles, but the cast overall is just perfect. Overall, Barbie is probably the best movie based off a toy (sorry LEGO Movies) and an overall fantastic time at the movies.",/review/rw9205757/?ref_=tt_urv 41 | 19,26 July 2023,aherdofbeautifulwildponies,6," A Hot Pink Mess 42 | ","Before making Barbie (2023), ",/review/rw9216187/?ref_=tt_urv 43 | 20,20 July 2023,Genti25,8," You are Kenough 44 | ","This movie is so much fun. It starts off really strong although the story does move away from ""Barbieland"" sooner than I would have liked. Nonetheless, it regains its footing with the final act in particular and I could not stop laughing at Ryan Gosling's portrayal of Ken. That song will forever be stuck in my head.",/review/rw9202655/?ref_=tt_urv 45 | 21,19 July 2023,tm-sheehan,6," Didn't hit my pink spot 46 | ","My Review - Barbie 47 | In Cinemas now 48 | My Rating 6.5/10",/review/rw9200568/?ref_=tt_urv 49 | 22,23 July 2023,herrcarter-92161,5," Somewhat of a Jumbled Mess 50 | ",My 15-year old daughter wrote the following review and posted it to my IMDB account.,/review/rw9208691/?ref_=tt_urv 51 | 23,19 July 2023,andypaps101,8," ""Barbie"" - A Multifaceted Exploration of Femininity, Consumerism, and Existentialism 52 | ",I got free tickets for a preview and to be honest I was more than a little embarrassed to go. I did not tell a soul. As a cool middle aged gentleman if word got out this could have really affected my street cred.,/review/rw9200687/?ref_=tt_urv 53 | 24,23 July 2023,subxerogravity,8," It's like G.I Joe...For girls! Seriously, and I love it for that! 54 | ","I mean, Margo Robbie as Barbie just made perfect sense, but I really wanted to see the movie when I herd that Ryian Gosling was going to be Ken, I just thought that would be hilarious and I was right!",/review/rw9209982/?ref_=tt_urv 55 | 25,24 July 2023,lokicola,6," Strong Start... and That's It 56 | ","I walked out of the theatre thinking, ""Yeah, I had a good time in that movie"". But as the day went on I kind had that ""Ok, that kebab probably wasn't a good idea.""",/review/rw9211052/?ref_=tt_urv 57 | 26,21 July 2023,masonsaul,10," The best version of itself 58 | ","Barbie is everything expected of it and so much more, a self aware and meta comedy made for fans and newcomers alike that's also incredibly profound, layered and really funny. It's the best version of itself and just immensely satisfying that the Barbie movie that got made on this scale is so weird and unique.",/review/rw9205582/?ref_=tt_urv 59 | 27,19 July 2023,dreopdreef,6," Not so unique as expected 60 | ",The concept of a Barbie movie is really unique and got me all exited. But unfortunately to me it delivers the same messages that we keep hearing as of lately.,/review/rw9201199/?ref_=tt_urv 61 | 28,19 July 2023,anjamulder,8," not for everyone 62 | ","This is very much a movie that will get devided opinions on it, my cousin said well this is two hours of my life i will never get back, and me? I left the theatre in such a good mood. It was the sense of humor for me, laughed so hard. Specially Ryan Gossling was sooooo good and funny, many familiar faces to spot also. And margot robbie come on... the perfect casting for Barbie. I also have to say, the way they created Barbie land was so amazing, it just makes you just want to there. I don't really have anything bad to say about this movie. I can understand not everyone will love it. But i am sure it is a iconic film already.",/review/rw9200229/?ref_=tt_urv 63 | 29,19 July 2023,mark-217-307033,6," Fun film whose script falls in on itself 64 | ","Margot Robbie's performance is perfect, and no other actor could've embodied the character as well as she does. However, Ryan Gosling is the film-stealer. I often forget his comedic talent, and he's given a lot of room to flex it in this. A cleverly executed Matchbox Twenty joke is particularly memorable.",/review/rw9199456/?ref_=tt_urv 65 | 30,1 August 2023,zkonedog,10," An Amazingly Perfect Blend Of Social Satire & Fun 66 | ","For a film like ""Barbie"" to succeed, every aspect had to come together perfectly. It needed to consummately balance satire with fun, the casting had to be spot-on, and the writing/directing required to pull off such a feat would need to be exquisite. Somewhat remarkably, that is exactly what writer/director Greta Gerwig pulls off here to create the best movie of summer 2023.",/review/rw9230831/?ref_=tt_urv 67 | 31,19 July 2023,AvionPrince16,6," Barbie land and reality 68 | ","Im not really disappointed or love the movie to be honest. It was kind of weird and strange at the beginning: we understand it later because we are in the Barbie world. So they played with the culture reference of the barbie, the dolls and what they represent: an ideal world with where the womans are in power. Things will get pretty normal after they enter the real world and all the problems: sexual remark, womens and their place in society (the inequality with mens.) I was pretty sure they will talk about the problems with womans and their complex because of Barbie and the perfect women she is and how this is contrasted with the real world. Its pretty obvious that they will talk about it. I love also how Ken will also notice how mens are more powerful in the real world: its like everyone see that reality is not really what they perceived themselves in their own reality and power will shift also later in Barbie Land. I understand why they act like this and stuff but i didnt really get into the movie to be honest: i only identify myself when they talked about society and business but all the musical things, the choregraphy and even the morality was kind of superficial. But i enjoyed that Barbie realised the problems of our world and compare to her own. Some moments was funny and some more serious or drama sometimes even if its pretty soft because we are in Barbie Land. I mean i spend a great time but it was kind of ok. And i still think its a great Barbie movie: the task was not easy to represent that world but i enjoyed the set design, the colors, the camera movements. I think people who played with Barbie dolls or just know the universe will be happy; they have a lot of references to the world and a lot of Barbie references to the dolls and the differents clothes and the impact of her in society. Some moments were also absurd ( the fight with all the Kens). Its good anyway.",/review/rw9199440/?ref_=tt_urv 69 | 32,21 July 2023,MaskedMinty,5," Preach Preach Preach 70 | ","Let me start by saying that I'm by no means a conservative or traditional woman. I believe that inequality for women still is rampant even in first world countries and that we go through many problems. But damn...this movie is so corporate and shallow. It's clearly ordered and tailored by executives at Mattel to rebrand and sell more dolls. The movie starts well and has the potential of becoming something very good, but then it just goes downhill: the premise of the owner's feeling being projected onto the doll was going to lead to a very nice build up about a person's life journey and then Barbie and Ken go to the human world and cue in the speeches and preaching. This movie suffers from a bad case of showing and not telling, they keep repeating the same dialogues: patriarchy bad patriarchy bad patriarchy bad sexism sexism sexism, not even one time but MULTIPLE TIMES. These characters go into random rants like its a ted talk! The chemistry between the mother and daughter is absolutely nonexistent and their relationship is so cringe, You cant root for them or feel their feelings at all. Barbie should have understood how Kens had been feeling in the Barbieland the whole time which is a mimic of how many women are still treated in the society: just existing for the gaze of a woman (man). Yet what do they do? They manipulate them (with a super nonsense plan) and force them back to their original position while she gives him a half assed apology that could have been done better and end up to a more accurate point about equality. (I guess this is again meta and a reflection of how women are pitted against each other while people in charge benefit from that but its so badly done that it feels meh.) 71 | I can imagine that in the boardroom they went through a discussion like this: People are going to think this is a feel good movie so we need to give them something deeper but also lets make it absurd and fast paced so that we can finish the plot quickly and also jump on the ""not taking itself serious"" bandwagon and also market these dolls again in the age of smartphone kids. The only reason this movie gets at 5 from me is the humor, which sometimes becomes very meta and creative and Margot Robbie and Ryan Gosling have acted very well. Besides this, they could have just given us a simple feel good movie and I would have respected the creators more. Pass.",/review/rw9204479/?ref_=tt_urv 72 | 33,21 July 2023,imdbmovieguy,," Honestly - not funny and light enough 73 | ","I really enjoyed the first 20 minutes of the movie. It was very upbeat, positive and light. However, things soon became negative about 20 - 25 minutes into the movie. I couldn't stand it. Barbie's world and attitude was a constant bummer. Ken's attitude was sour. For the next hour (and more) the movie just lived in negativity.",/review/rw9205479/?ref_=tt_urv 74 | 34,4 August 2023,evanston_dad,8," Smart and Winning 75 | ","My wife and I were two of the apparently 200,000 or so people who Barbenheimered Greta Gerwig's and Chritopher Nolan's hot topic movies on opening weekend. ""Oppenheimer"" was first, a boring slog of history in which a bunch of white men blathering at each other for three hours resulted in the creation of weapons that could wipe out humanity. ""Barbie"" came in the evening, a fresh breeze of humanity to clear away Nolan's emotional constipation.",/review/rw9237728/?ref_=tt_urv 76 | 35,19 July 2023,faeez_rizwan,9," Wildly Pink 77 | ",Barbie Movie Review,/review/rw9199384/?ref_=tt_urv 78 | 36,21 July 2023,arslanmanzoor123,6," Quite disappointing 79 | ","Some friends freaking out that I went to see Barbie lol I mean I mentioned I went coz my sister insisted since ages. Anyway the movie was bad. These days, every other director and producer wants to convey messages about women rights and feminism through whatever platform possible and this movie was a perfect example of that with sheer cringe to convey all that. I wished I had gotten the ticket for Oppenheimer in the same timings as my sister's Barbie show. Other than that, the acting done by Margot Robbie and Ryan Gosling was great so can't take that away from them. They did justice to their roles.",/review/rw9204612/?ref_=tt_urv 80 | 37,23 July 2023,eoinageary,8," I am Kenough 81 | ",So I went into the movie with little to no expectations and I was pleasantly impressed with the movie overall.,/review/rw9209094/?ref_=tt_urv 82 | 38,25 July 2023,spika13,6," Too overrated 83 | ","Its like watching at ""Truman's show"" but all the set it's pink color with a hint of extreme feminist ideas. I grew up with barbie dolls and I felt very disappointed, I was expecting a different kind of story, the ending was the worst. I think the message should be more with the real feminist concept of Simone de Bouvoir, we all deserve the same rights. In the end Instead of just Barbie Land they should changed as; Barbie and Ken Land.",/review/rw9215177/?ref_=tt_urv 84 | 39,23 July 2023,Anurag-Shetty,10," A wholesome delight! 85 | ","Barbie is based on Mattel's iconic plastic dolls of the same name. Barbie(Margot Robbie) lives a very happy life in Barbie Land, along with her fellow Barbies, Ken(Ryan Gosling) & his fellow Kens. Suddenly, Barbie goes through an existential crisis that makes her question Barbie Land & the truth of her existence.",/review/rw9210747/?ref_=tt_urv 86 | 40,24 July 2023,GhostFoxX,6," Misleading. 87 | ","I personally expected the movie to be fun and adventurous. Correct me if I'm wrong, but isn't that what it is supposed to be? It's literally a movie about Barbie. Instead, it felt more like an agent of woke culture, and basically Mattel's (the company that produces Barbie dolls) way of capitalizing the emotional aspect of the cinephiles to make their business more popular and trendier. It continuously tried to portray the fact that Mattel is no more producing white barbie dolls, it is now producing dolls of all colors and races. Well, that is definitely a good thing, but making most of the movie focus on that aspect felt very boring and wrong. If they didn't quite make a movie that is fun and adventurous, why make the marketing events misleading? They could've just focused on the audience for whom it was made, rather than on the entire world through their absurdly irrelevant marketing events that made the world think that this movie is going to be all about fun and adventure. Sadly, it was not nearly as enjoyable as the misleading marketing made it seem.",/review/rw9211853/?ref_=tt_urv 88 | 41,25 July 2023,mr_bickle_the_pickle,8," Life in plastic, it's fantastic! 89 | ","It's an incredibly fun movie. Lots of laughs. Very campy over the top humour. But also surprisingly touching. My favourite scene of the movie is just rather a small quiet moment where Barbie is observing every day life and tells an old woman how beautiful she is. Margot Robbie shines in the lead role. Ryan Gosling clearly was having a lot of fun with the role, but don't sleep on Michael Cera as Allan. I know a lot of people say that Ryan Gosling is a scene stealer, but personally I thought Michael was.",/review/rw9214540/?ref_=tt_urv 90 | 42,1 August 2023,badreviewer,6," This Barbie is dissapointed. 91 | ","Yes. Day by day it becomes more obvious that now the movies are made just because of the income they generate and nothing else. Ok, Barbie itself was not a creative idea, I know. And actually I didn't expect to see something unseen something unique in the first place, all in all it's ""Barbie"".",/review/rw9230702/?ref_=tt_urv 92 | 43,24 July 2023,brianjohnson-20043,4," This movie tries to be too much 93 | ",I wanted to like it. But I just didn't. The story wasn't very compelling to me because it seemed as if the entire point of much of the movie was to provide learning moments for viewers with some slapstick comedy and quick-delivery comedy. The story itself should nearly always be more important.,/review/rw9213135/?ref_=tt_urv 94 | 44,7 August 2023,saimariejohnson,10," Wildly Entertaining. 95 | ","Barbie surprised me in all the best ways. It was funny, moving, inspirational, and all around ridiculously entertaining. I mean that with every sense of the implication to.",/review/rw9244236/?ref_=tt_urv 96 | 45,10 August 2023,adamjohns-42575,8," A sheer pleasure to watch. 97 | ",Barbie (2023) -,/review/rw9251197/?ref_=tt_urv 98 | 46,23 July 2023,jacquihives,6," I had no expectations but was still underwhelmed 99 | ","I loved barbie growing up, and even at 42 years old I couldn't wait to go see the movie! I didn't have any expectations because I had no idea how they could make a movie about barbie that's for adults and not kids. I adored Barbieland, but unfortunately they leave it behind so fast and with it they also leave behind everything good about bringing barbie to life in a movie! It got silly, had no storyline, was overly political and woke, man bashing and made men silly, and I really didn't care for the ending which is just as well because it was abrupt and crude.",/review/rw9209276/?ref_=tt_urv 100 | 47,23 July 2023,Jeremy_Urquhart,8," Surprisingly good 101 | ","I think part of my brain rejects this for being a feature-length toy commercial, but it's a very small part of my brain (and that really wasn't where my mind was for the majority of the movie).",/review/rw9209363/?ref_=tt_urv 102 | 48,26 July 2023,JPARM-IMDb,6," Expected more from Greta Gerwig 103 | ","Even though I'm not the target audience for this movie, I thought Barbie might impress me due to Greta Gerwig's previous works but I'm disappointed.",/review/rw9217019/?ref_=tt_urv 104 | 49,21 July 2023,rannynm,10," What A Film! I Was Ecstatic About Seeing Barbie And It Definitely Exceeded My Expectations By Far. 105 | ","What a film! I was ecstatic about seeing Barbie and it definitely exceeded my expectations by far. As much as I was excited, I was a bit scared that the stakes were too high; after all Barbie is an icon and a part of so many people's childhood. For a film like this it's very easy to create something corny, disappointing, or unappealing to viewers. I'm so glad that Gretta Gerwig stepped up to the plate and created an absolute masterpiece.",/review/rw9205028/?ref_=tt_urv 106 | 50,29 July 2023,Vic_max,7," A lot like The Lego Movie - except for the 2nd half 107 | ","As a guy who never played with Barbie dolls, I was surprised to be caught up in the enthusiasm to see this movie; I purchased tickets well in advance when they first went on sale.",/review/rw9222678/?ref_=tt_urv 108 | 51,23 July 2023,nethy-nho,10," Barbie is not a simple live action of a timeless doll 109 | ","But it is also in its smallest details, from the art direction, costumes, and any technical choice completely thought of referencing all Barbie generations and the like. A soundtrack that will take you from laughing to crying (yes, very musical moments, after all it is a Barbie movie), in addition to the hand-picked cast, Margot is impeccable and manages to express all her feelings, just with her eyes. Greta's direction and screenplay, just amazing, strong, emotional, empowering and very, very feminist, not only for the women who are already fighting for the cause, but to open everyone's eyes to a sisterhood.",/review/rw9210701/?ref_=tt_urv 110 | 52,19 July 2023,AvionPrince16,6," Barbie Land 111 | ","Im not really disappointed or love the movie to be honest. It was kind of weird and strange at the beginning: we understand it later because we are in the Barbie world. So they played with the culture reference of the barbie, the dolls and what they represent: an ideal world with where the womans are in power. Things will get pretty normal after they enter the real world and all the problems: sexual remark, womens and their place in society (the inequality with mens.) I was pretty sure they will talk about the problems with womans and their complex because of Barbie and the perfect women she is and how this is contrasted with the real world. Its pretty obvious that they will talk about it. I love also how Ken will also notice how mens are more powerful in the real world: its like everyone see that reality is not really what they perceived themselves in their own reality and power will shift also later in Barbie Land. I understand why they act like this and stuff but i didnt really get into the movie to be honest: i only identify myself when they talked about society and business but all the musical things, the choregraphy and even the morality was kind of superficial. But i enjoyed that Barbie realised the problems of our world and compare to her own. Some moments was funny and some more serious or drama sometimes even if its pretty soft because we are in Barbie Land. I mean i spend a great time but it was kind of ok. And i still think its a great Barbie movie: the task was not easy to represent that world but i enjoyed the set design, the colors, the camera movements. Its good anyway.",/review/rw9200246/?ref_=tt_urv 112 | 53,22 July 2023,shadowtramp,5," More of a marketing flick 113 | ",The brilliant marketing is what led us tonight to pay for the tickets. The brilliance ends there.,/review/rw9208132/?ref_=tt_urv 114 | 54,22 July 2023,lisacarlson50,10," Creatively Brilliant 115 | ","Brilliantly written by Greta Gerwig and Noah Baumbach. Everyone in this is fantastic. It's a wink at nostalgia and commentary on society. Never played with Barbies but my daughter did. Tribute to the inventor of Barbie, Ruth Handler. Helen Mirren narrates. Kate McKinnon almost steals the movie. Not a movie for little kids but most of the references will probably go over their heads. My girlfriend and I laughed out loud many times. It's about 20 minutes too long for my taste. The music is spot on. It's a perfect summer movie. Guess what? It has value and substance. Margot Robbie is not only gorgeous but more importantly, she's highly intelligent as actor and producer. This movie is going to make bank!",/review/rw9207531/?ref_=tt_urv 116 | 55,27 July 2023,sanadalsalt,10," Laugh, Reflect, Repeat: Exploring the Wonders of 'Barbie' 117 | ","Oh my goodness, where do I even begin with ""Barbie"" (2023)? This film is an absolute delight and a true masterpiece that will leave you in awe. Directed by the incredibly talented Greta Gerwig and starring the stunning Margot Robbie as the iconic doll herself, ""Barbie"" is a cinematic experience like no other.",/review/rw9218017/?ref_=tt_urv 118 | 56,29 July 2023,planktonrules,6," Good...not great 119 | ","""Barbie"" is a movie that is setting all sorts of records, so it's obvious that the film is touching many people. For some, it's the fun of seeing a doll come to life, for others it's the theme of empowerment that make the movie a big hit. As for me, I did like it and thought it was a pretty good movie...but it's not for everyone.",/review/rw9222335/?ref_=tt_urv 120 | 57,22 July 2023,DoNotComeToTheCinemaDepressed,10," 🏩 Possibly a Masterpiece? 💒 121 | ","I Understand that Some People are Going to Have Big Big Problems with Some of the Messaging within this Film. But Sometimes, in Life 😂, You Just Have to Go With It.",/review/rw9208010/?ref_=tt_urv 122 | 58,21 July 2023,suf-54719,6," It's a silly movie 123 | ","The story feels like watching modern animated barbie movies in this past decade. If you lower your expectations, you gonna kinda enjoy it. The first 30 minutes is good, but the last 30 minutes feel rushed.",/review/rw9204081/?ref_=tt_urv 124 | 59,9 August 2023,L3MM3,," Boring, mind-numbing drivel 125 | ","I do not usually write reviews, but this is beyond description. Last time I wrote a review it was for the absolutely horrible wrinkle in time. This is almost as bad. At least Gosling plays his part well, but the story is abysmal. If you can find anything else to do like watch paint dry, then yes please do. You will save your brain cells the agony of a painful slow ..... nearly two hours long this was mind-numbing storytelling at its mediocre worst. How many ways can I say do not go see it if you have a choice. See anything else but this exercise in horrendous cliches and exaggerated self-importance. If I could give it negative 10 stars I would.",/review/rw9249320/?ref_=tt_urv 126 | 60,28 July 2023,klastaitas,8," We're All Dolls 127 | ","Without consumerism, without belief, without plastic accessory and hope for a perfect day, where would Barbie find herself?",/review/rw9221654/?ref_=tt_urv 128 | 61,23 July 2023,agjbull,6," Just a little empty 129 | ","I really wanted to enjoy this and I know that I am not the target audience but there were massive plot holes and no real flow. The film was very disjointed. Ryan Gosling as good as he is seemed to old to play Ken and Will Ferrell ruined every scene he was in. I just didn't get it, it seemed hollow artificial and hackneyed. A waste of some great talent. It was predictable without being reassuring and trying so hard to be woke in the most superficial way in that but trying to tick so many boxes it actually ticked none. Margo Robbie looks beautiful throughout, the costumes and the sets were amazing but the story was way too weak and didn't make much sense at all.",/review/rw9209877/?ref_=tt_urv 130 | 62,21 July 2023,meltingmel,10," 10/10!!!!!!! 131 | ",Wow. WOW!!!!,/review/rw9203272/?ref_=tt_urv 132 | 63,10 August 2023,arcelivez-753-983928,6," Honestly - no idea why people are so hyped up with this 133 | ","Honestly I really have no idea why this movie seems to be so popular. Again honest - I went there just for Margot Robbie (I'm a married 34 year old male, and margot is one of my top 3 favorite actrices of all time, and arguably the most talented one from my list). And for that part it was totally ok. But the movie absolutely doesn't deserve an imdb rating above 5.6. The plot itself was very simple. Absolutely nothing particularly special in it. So if it is in fact getting rated so high just because of Margot alone, then why do her other movies don't get rated that well and get that much attention? Honestly I really don't understand and I think this movie is way too overhyped for no reason. And I have no idea why... For other movies that get hyped like that I do see the reasons why they're getting so hyped. So the movie is not awful, but it's rather average, if you're not going there specifically for Margot then I am pretty sure there will be multiple movies at your local cinema at the time you're reading this, which are way more worth watching than this one...",/review/rw9249557/?ref_=tt_urv 134 | 64,22 July 2023,fernandoschiavi,5," ""Barbie"" is fun and visually beautiful, but unfortunately Barbie doll was dragged into the cultural war, used as a puppet by political militancy 135 | ","Despite the strong commercial tone, it is still possible to perceive that we are watching a film by Greta Gerwig, director known for dramatic comedies that capture the complexity of femininity in an original way - although, in this case, it is almost inevitable to fall into some clichés. The undeniable feminist tone exists through the presence of this female director - in addition to Margot Robbie herself as a producer. ""Barbie"" has sharp and intelligent comments on the dynamics between men and women over time and also on how these social roles are seen and problematized today. And the film does it with a big heart: the story is genuinely fun, with a humor that walks between acidity and innocence, but also moves with the existential journey of its protagonists, reserving touching moments and provoking reflections, especially in its final stretch. Debauchery is the great ally of Gerwig and Baumbach to deal not only with sexism, but also with the ""corporate"" issue. It won't be strange if you catch yourself thinking ""how did Mattel let this film see the light of day?"" Laughing at yourself, after all, seems to be a prerequisite for establishing communication with a young generation of consumers who associate their consumption habits with a critical eye on consumerism itself.",/review/rw9207456/?ref_=tt_urv 136 | 65,20 July 2023,mavigangi,10," A colorful love letter to human kind 137 | ","I cried, I laughed, I dance and I did sing too. What a joyful time I had at the movies watching Barbie. This is a movie to celebrate how diverse and complicated are the humans. The cinematography is a outstanding as well as the production design and the vestuary. The casting seems like made in heaven. I recommend this movie for 11+ years old. And if you have a teen at home, take her with you. Barbie goes from fun to introspection and I love a good comedy that makes you think. Ryan Gosling is the perfect Ken, he does everything in this movie and shines every time he shows up. Margo is a perfect Barbie and got a real sense of the relevance this is going to have.",/review/rw9201450/?ref_=tt_urv 138 | 66,23 July 2023,anniebleasdale-97708,9," Wildly misunderstood by the poor reviewers. 139 | ","Unfortunately, most reviews that I've read here that score this film poorly explain so clearly that the reviewer did not understand the message of the film. It is not anti-men, it does not portray men as ""stupid"". Take a moment to think about how women have been portrayed in movies over the last century - oh yes, just as Ken is here! The movie exaggerates this, yes. That is the point. To clearly (and humourously) highlight the inequalities when the roles are reversed. Stop being offended by something that isn't offensive and take a bit of time to actually think about the film that you have watched.",/review/rw9210081/?ref_=tt_urv 140 | 67,3 August 2023,DanLawson146,7," Fun, but too on the nose at times 141 | ",POSITIVES:,/review/rw9236010/?ref_=tt_urv 142 | 68,20 July 2023,PedroPires90,8," A triumph 143 | ",I appreciated it but wasn't a huge fan of Greta Gerwig's first two films. I knew absolutely nothing about Barbie except that it was a doll. The trailers didn't impress me. I slept less than an hour last night. I watched the movie alone surrounded by teenagers who cheered when the WB logo appeared in pink.,/review/rw9202729/?ref_=tt_urv 144 | 69,6 August 2023,pompeiirome,6," Ryan Gosling, a pure showman 145 | ","If forgotten, Ryan Gosling started off in the mickey mouse club. Important to remember because wow what a talented individual. Recent memory sees him leading roles that require a more stoic subdued approach, and it can be easy to miss his range. Barbie seeks to utilize all his acting muscles, and to a resounding success. He can sing, dance, act, and do comedy.",/review/rw9242845/?ref_=tt_urv 146 | 70,20 July 2023,MatthewMichaelHoffman,8," Fails to meet expectations, but still fun! 147 | ",SCORE: 7.8/10,/review/rw9201449/?ref_=tt_urv 148 | 71,20 July 2023,FeastMode,7," I'll beach you off so hard 149 | ",I had no idea what to expect going into Barbie. I had no interest in seeing a movie about Barbie but was immediately sold on the highly talented Margot Robbie and the awesome Ryan Gosling. But I was still nervous that the comedy would be aimed strictly at women and would be difficult for a guy to enjoy.,/review/rw9201574/?ref_=tt_urv 150 | 72,7 August 2023,misscattik,," No questions just answers 151 | ","The title of the review practically sums it all up. With the highest potential in style, aesthetics, cast and concept, the movie fails spectacularly when it comes to the deliverance of it's main idea. And it's not feminism that bothers, surely not, it's the artistic language in which the idea is told - harsh, manipulative, abusive, straightforward and unkind. Instead of aspiring the audience to think and reflect, it shoves the manifestations down your throat, and in monologues that are generally painfully awkward to watch.",/review/rw9245145/?ref_=tt_urv 152 | 73,5 August 2023,sjw1029,8," The movie event of 2023 lives up to the hype 153 | ",Let me start by saying it makes me happy to see a movie that isn't from the MCU or a sequel doing so well at the box office. There is still originality out there. This movie was very funny - the scene with Ken and the doctor in LA made me laugh very hard. It's weird and inventive. It has amazing choreography. So many talented people worked on this. I didn't grow up with Barbies so I'm sure a lot of the jokes went over my head. But it didn't hurt the enjoyment. It even manages to land a sentimental note at the end. I can definitely see this being a movie people see over and over.,/review/rw9240889/?ref_=tt_urv 154 | 74,20 July 2023,moriumemoltre,6," Great acting but why this movie? 155 | ","I was really surprised about how much i liked Ryan Gosling in this. I had totally forgot about his comedic talent and timing. The movie works. There is definitely a moral message in this, even though I believe the majority of viewers will be too distracted by Margot Robbie and Ryan Goslings great performances and all of the comedy elements in this and the fact that the script is hardly worth being filmed. I feel like with a different script they could have created something truly great.",/review/rw9201978/?ref_=tt_urv 156 | -------------------------------------------------------------------------------- /Week 2/Thursday/Automated Fine-tuning with LLamaIndex.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Using LlamaIndex to Automate the fine-tuning of GPT-3.5-turbo on source documents\n", 8 | "\n", 9 | "Primarly Extended from [this](https://colab.research.google.com/drive/1vWeJBXdFEObuihO7Z8ui2CAYkdHQORqo?usp=sharing) notebook, we'll take a look at how we can wrap this process into Chainlit and have our own dynamic fine-tuning machine!" 10 | ] 11 | }, 12 | { 13 | "cell_type": "code", 14 | "execution_count": 1, 15 | "metadata": {}, 16 | "outputs": [], 17 | "source": [ 18 | "!pip install -q -U llama-index pypdf sentence-transformers ragas openai" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": 4, 24 | "metadata": {}, 25 | "outputs": [], 26 | "source": [ 27 | "import os\n", 28 | "from getpass import getpass\n", 29 | "\n", 30 | "openai_api_key = getpass(\"Enter your OpenAI API key: \")\n", 31 | "os.environ[\"OPENAI_API_KEY\"] = openai_api_key" 32 | ] 33 | }, 34 | { 35 | "cell_type": "code", 36 | "execution_count": 3, 37 | "metadata": {}, 38 | "outputs": [ 39 | { 40 | "name": "stdout", 41 | "output_type": "stream", 42 | "text": [ 43 | " % Total % Received % Xferd Average Speed Time Time Time Current\n", 44 | " Dload Upload Total Spent Left Speed\n", 45 | "100 3322k 100 3322k 0 0 11.6M 0 --:--:-- --:--:-- --:--:-- 11.6M\n" 46 | ] 47 | } 48 | ], 49 | "source": [ 50 | "!curl https://jaydixit.com/files/PDFs/TheultimateHitchhikersGuide.pdf --output hitchhikers.pdf" 51 | ] 52 | }, 53 | { 54 | "cell_type": "code", 55 | "execution_count": 20, 56 | "metadata": {}, 57 | "outputs": [], 58 | "source": [ 59 | "from llama_index import SimpleDirectoryReader, ServiceContext\n", 60 | "from llama_index.llms import OpenAI\n", 61 | "from llama_index.evaluation import DatasetGenerator\n", 62 | "\n", 63 | "documents = SimpleDirectoryReader(\n", 64 | " input_files=[\"hitchhikers.pdf\"]\n", 65 | ").load_data()\n", 66 | "\n", 67 | "# Shuffle the documents\n", 68 | "import random\n", 69 | "\n", 70 | "random.seed(42)\n", 71 | "random.shuffle(documents)\n", 72 | "\n", 73 | "gpt_35_context = ServiceContext.from_defaults(\n", 74 | " llm=OpenAI(model=\"gpt-3.5-turbo\", temperature=0.3)\n", 75 | ")\n", 76 | "\n", 77 | "question_gen_query = (\n", 78 | " \"You are a Teacher/ Professor. Your task is to setup \"\n", 79 | " \"a quiz/examination. Using the provided context from a \"\n", 80 | " \"report on climate change and the oceans, formulate \"\n", 81 | " \"a single question that captures an important fact from the \"\n", 82 | " \"context. Restrict the question to the context information provided.\"\n", 83 | ")\n", 84 | "\n", 85 | "dataset_generator = DatasetGenerator.from_documents(\n", 86 | " documents[:50],\n", 87 | " question_gen_query=question_gen_query,\n", 88 | " service_context=gpt_35_context,\n", 89 | ")" 90 | ] 91 | }, 92 | { 93 | "cell_type": "markdown", 94 | "metadata": {}, 95 | "source": [ 96 | "### Generative Questions with `gpt-3.5-turbo`\n", 97 | "\n", 98 | "We can use the `generate_questions_from_nodes()` method of our dataset generator to produce a number of questions that will be used to fine-tune!\n", 99 | "\n", 100 | "> NOTE: This cell will take ~30s-2min." 101 | ] 102 | }, 103 | { 104 | "cell_type": "code", 105 | "execution_count": 21, 106 | "metadata": {}, 107 | "outputs": [ 108 | { 109 | "name": "stdout", 110 | "output_type": "stream", 111 | "text": [ 112 | "Generated 40 questions\n" 113 | ] 114 | } 115 | ], 116 | "source": [ 117 | "questions = dataset_generator.generate_questions_from_nodes(num=40)\n", 118 | "print(\"Generated \", len(questions), \" questions\")" 119 | ] 120 | }, 121 | { 122 | "cell_type": "markdown", 123 | "metadata": {}, 124 | "source": [ 125 | "Let's take a peek and see what was created!" 126 | ] 127 | }, 128 | { 129 | "cell_type": "code", 130 | "execution_count": 22, 131 | "metadata": {}, 132 | "outputs": [ 133 | { 134 | "data": { 135 | "text/plain": [ 136 | "'What did Zaphod find on the external monitor screens in the Horsehead Nebula?'" 137 | ] 138 | }, 139 | "execution_count": 22, 140 | "metadata": {}, 141 | "output_type": "execute_result" 142 | } 143 | ], 144 | "source": [ 145 | "questions[0]" 146 | ] 147 | }, 148 | { 149 | "cell_type": "markdown", 150 | "metadata": {}, 151 | "source": [ 152 | "Now we can save our questions into a text file for later use." 153 | ] 154 | }, 155 | { 156 | "cell_type": "code", 157 | "execution_count": 23, 158 | "metadata": {}, 159 | "outputs": [], 160 | "source": [ 161 | "with open(\"train_questions.txt\", \"w\") as f:\n", 162 | " for question in questions:\n", 163 | " f.write(question + \"\\n\")" 164 | ] 165 | }, 166 | { 167 | "cell_type": "markdown", 168 | "metadata": {}, 169 | "source": [ 170 | "### Evaluation Generator\n", 171 | "\n", 172 | "Let's generate questions from a different segment of our documents in order to build a robust test for our RAQA." 173 | ] 174 | }, 175 | { 176 | "cell_type": "code", 177 | "execution_count": 10, 178 | "metadata": {}, 179 | "outputs": [], 180 | "source": [ 181 | "dataset_generator = DatasetGenerator.from_documents(\n", 182 | " documents[\n", 183 | " 50:\n", 184 | " ], # since we generated ~1 question for 40 documents, we can skip the first 40\n", 185 | " question_gen_query=question_gen_query,\n", 186 | " service_context=gpt_35_context,\n", 187 | ")" 188 | ] 189 | }, 190 | { 191 | "cell_type": "markdown", 192 | "metadata": {}, 193 | "source": [ 194 | "Again, we'll use `gpt-3.5-turbo` to generate some questions!" 195 | ] 196 | }, 197 | { 198 | "cell_type": "code", 199 | "execution_count": 11, 200 | "metadata": {}, 201 | "outputs": [ 202 | { 203 | "name": "stdout", 204 | "output_type": "stream", 205 | "text": [ 206 | "Generated 40 questions\n" 207 | ] 208 | } 209 | ], 210 | "source": [ 211 | "questions = dataset_generator.generate_questions_from_nodes(num=40)\n", 212 | "print(\"Generated \", len(questions), \" questions\")" 213 | ] 214 | }, 215 | { 216 | "cell_type": "markdown", 217 | "metadata": {}, 218 | "source": [ 219 | "Now we can save our results for evaluations later!" 220 | ] 221 | }, 222 | { 223 | "cell_type": "code", 224 | "execution_count": 12, 225 | "metadata": {}, 226 | "outputs": [], 227 | "source": [ 228 | "with open(\"eval_questions.txt\", \"w\") as f:\n", 229 | " for question in questions:\n", 230 | " f.write(question + \"\\n\")" 231 | ] 232 | }, 233 | { 234 | "cell_type": "markdown", 235 | "metadata": {}, 236 | "source": [ 237 | "### Evaluating base `gpt-3.5-turbo`" 238 | ] 239 | }, 240 | { 241 | "cell_type": "markdown", 242 | "metadata": {}, 243 | "source": [ 244 | "We'll load up our evaluation questions and get to it!" 245 | ] 246 | }, 247 | { 248 | "cell_type": "code", 249 | "execution_count": 13, 250 | "metadata": {}, 251 | "outputs": [], 252 | "source": [ 253 | "questions = []\n", 254 | "with open(\"eval_questions.txt\", \"r\") as f:\n", 255 | " for line in f:\n", 256 | " questions.append(line.strip())" 257 | ] 258 | }, 259 | { 260 | "cell_type": "markdown", 261 | "metadata": {}, 262 | "source": [ 263 | "This next cell is constructing our `VectorIndex` so we can move onto testing the base model." 264 | ] 265 | }, 266 | { 267 | "cell_type": "code", 268 | "execution_count": 14, 269 | "metadata": {}, 270 | "outputs": [], 271 | "source": [ 272 | "from llama_index import VectorStoreIndex\n", 273 | "\n", 274 | "# limit the context window to 2048 tokens so that refine is used\n", 275 | "gpt_35_context = ServiceContext.from_defaults(\n", 276 | " llm=OpenAI(model=\"gpt-3.5-turbo\", temperature=0.3), context_window=2048\n", 277 | ")\n", 278 | "\n", 279 | "index = VectorStoreIndex.from_documents(documents, service_context=gpt_35_context)\n", 280 | "\n", 281 | "query_engine = index.as_query_engine(similarity_top_k=2)" 282 | ] 283 | }, 284 | { 285 | "cell_type": "markdown", 286 | "metadata": {}, 287 | "source": [ 288 | "Here is where we're actually putting the model to the test!" 289 | ] 290 | }, 291 | { 292 | "cell_type": "code", 293 | "execution_count": 15, 294 | "metadata": {}, 295 | "outputs": [], 296 | "source": [ 297 | "contexts = []\n", 298 | "answers = []\n", 299 | "\n", 300 | "for question in questions:\n", 301 | " response = query_engine.query(question)\n", 302 | " contexts.append([x.node.get_content() for x in response.source_nodes])\n", 303 | " answers.append(str(response))" 304 | ] 305 | }, 306 | { 307 | "cell_type": "markdown", 308 | "metadata": {}, 309 | "source": [ 310 | "Now that we've tested our model - let's evaluate it to see how it performed!\n", 311 | "\n", 312 | "We're testing our model with the `ragas` framework - found [here](https://github.com/explodinggradients/ragas)\n", 313 | "\n", 314 | "You'll notice that we're testing two primary metrics:\n", 315 | "\n", 316 | "- [`answer_relevancy`](https://github.com/explodinggradients/ragas/blob/a55c3be8b2389501c5c761df9070126027a4d1d6/src/ragas/metrics/answer_relevance.py#L32): This measures how relevant is the generated answer to the prompt. If the generated answer is incomplete or contains redundant information the score will be low. This is quantified by working out the chance of an LLM generating the given question using the generated answer. Values range (0,1), higher the better.\n", 317 | "- [`faithfulness`](https://github.com/explodinggradients/ragas/blob/a55c3be8b2389501c5c761df9070126027a4d1d6/src/ragas/metrics/faithfulnes.py#L63): This measures the factual consistency of the generated answer against the given context. This is done using a multi step paradigm that includes creation of statements from the generated answer followed by verifying each of these statements against the context. The answer is scaled to (0,1) range. Higher the better.\n", 318 | "\n", 319 | "Read more about their implementations [here](https://github.com/explodinggradients/ragas/blob/main/docs/metrics.md)\n", 320 | "\n", 321 | "Again, these cells might take some time to complete - be patient!" 322 | ] 323 | }, 324 | { 325 | "cell_type": "code", 326 | "execution_count": 16, 327 | "metadata": {}, 328 | "outputs": [ 329 | { 330 | "name": "stderr", 331 | "output_type": "stream", 332 | "text": [ 333 | "/home/chris/anaconda3/envs/aims-finetune/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n", 334 | " from .autonotebook import tqdm as notebook_tqdm\n" 335 | ] 336 | }, 337 | { 338 | "name": "stdout", 339 | "output_type": "stream", 340 | "text": [ 341 | "evaluating with [answer_relevancy]\n" 342 | ] 343 | }, 344 | { 345 | "name": "stderr", 346 | "output_type": "stream", 347 | "text": [ 348 | "100%|██████████| 3/3 [01:10<00:00, 23.43s/it]\n" 349 | ] 350 | }, 351 | { 352 | "name": "stdout", 353 | "output_type": "stream", 354 | "text": [ 355 | "evaluating with [faithfulness]\n" 356 | ] 357 | }, 358 | { 359 | "name": "stderr", 360 | "output_type": "stream", 361 | "text": [ 362 | "100%|██████████| 3/3 [04:52<00:00, 97.41s/it] \n" 363 | ] 364 | }, 365 | { 366 | "name": "stdout", 367 | "output_type": "stream", 368 | "text": [ 369 | "{'ragas_score': 0.8230, 'answer_relevancy': 0.9308, 'faithfulness': 0.7375}\n" 370 | ] 371 | } 372 | ], 373 | "source": [ 374 | "from datasets import Dataset\n", 375 | "from ragas import evaluate\n", 376 | "from ragas.metrics import answer_relevancy, faithfulness\n", 377 | "\n", 378 | "ds = Dataset.from_dict(\n", 379 | " {\n", 380 | " \"question\": questions,\n", 381 | " \"answer\": answers,\n", 382 | " \"contexts\": contexts,\n", 383 | " }\n", 384 | ")\n", 385 | "\n", 386 | "result = evaluate(ds, [answer_relevancy, faithfulness])\n", 387 | "print(result)" 388 | ] 389 | }, 390 | { 391 | "cell_type": "code", 392 | "execution_count": 49, 393 | "metadata": {}, 394 | "outputs": [], 395 | "source": [ 396 | "base_eval = {'ragas_score': 0.8230, 'answer_relevancy': 0.9308, 'faithfulness': 0.7375}" 397 | ] 398 | }, 399 | { 400 | "cell_type": "markdown", 401 | "metadata": {}, 402 | "source": [ 403 | "### Leveraging `gpt-4` to improve our `gpt-3.5-turbo` base model!" 404 | ] 405 | }, 406 | { 407 | "cell_type": "code", 408 | "execution_count": 17, 409 | "metadata": {}, 410 | "outputs": [], 411 | "source": [ 412 | "from llama_index import ServiceContext\n", 413 | "from llama_index.llms import OpenAI\n", 414 | "from llama_index.callbacks import OpenAIFineTuningHandler\n", 415 | "from llama_index.callbacks import CallbackManager\n", 416 | "\n", 417 | "finetuning_handler = OpenAIFineTuningHandler()\n", 418 | "callback_manager = CallbackManager([finetuning_handler])\n", 419 | "\n", 420 | "gpt_4_context = ServiceContext.from_defaults(\n", 421 | " llm=OpenAI(model=\"gpt-4\", temperature=0.3),\n", 422 | " context_window=2048, # limit the context window artifically to test refine process\n", 423 | " callback_manager=callback_manager,\n", 424 | ")" 425 | ] 426 | }, 427 | { 428 | "cell_type": "code", 429 | "execution_count": 24, 430 | "metadata": {}, 431 | "outputs": [], 432 | "source": [ 433 | "questions = []\n", 434 | "with open(\"train_questions.txt\", \"r\") as f:\n", 435 | " for line in f:\n", 436 | " questions.append(line.strip())" 437 | ] 438 | }, 439 | { 440 | "cell_type": "code", 441 | "execution_count": 25, 442 | "metadata": {}, 443 | "outputs": [], 444 | "source": [ 445 | "from llama_index import VectorStoreIndex\n", 446 | "\n", 447 | "index = VectorStoreIndex.from_documents(documents, service_context=gpt_4_context)\n", 448 | "\n", 449 | "query_engine = index.as_query_engine(similarity_top_k=2)" 450 | ] 451 | }, 452 | { 453 | "cell_type": "markdown", 454 | "metadata": {}, 455 | "source": [ 456 | "Again, this process will take a few minutes. \n", 457 | "\n", 458 | "While this is a powerful technique - it is unfortunately quite slow." 459 | ] 460 | }, 461 | { 462 | "cell_type": "code", 463 | "execution_count": 26, 464 | "metadata": {}, 465 | "outputs": [], 466 | "source": [ 467 | "for question in questions:\n", 468 | " response = query_engine.query(question)" 469 | ] 470 | }, 471 | { 472 | "cell_type": "markdown", 473 | "metadata": {}, 474 | "source": [ 475 | "### Creating the fine-tuning dataset\n", 476 | "\n", 477 | "Now that we have a number of fine-tuning events from our `OpenAIFineTuningHandler()`, let's save them to a `.jsonl` file - the expected format for fine-tuning `gpt-3.5-turbo`!" 478 | ] 479 | }, 480 | { 481 | "cell_type": "code", 482 | "execution_count": 27, 483 | "metadata": {}, 484 | "outputs": [ 485 | { 486 | "name": "stdout", 487 | "output_type": "stream", 488 | "text": [ 489 | "Wrote 46 examples to finetuning_events.jsonl\n" 490 | ] 491 | } 492 | ], 493 | "source": [ 494 | "finetuning_handler.save_finetuning_events(\"finetuning_events.jsonl\")" 495 | ] 496 | }, 497 | { 498 | "cell_type": "code", 499 | "execution_count": 29, 500 | "metadata": {}, 501 | "outputs": [], 502 | "source": [ 503 | "import openai\n", 504 | "file_response = openai.File.create(file=open(\"finetuning_events.jsonl\", \"rb\"), purpose='fine-tune')" 505 | ] 506 | }, 507 | { 508 | "cell_type": "code", 509 | "execution_count": 30, 510 | "metadata": {}, 511 | "outputs": [ 512 | { 513 | "data": { 514 | "text/plain": [ 515 | " JSON: {\n", 516 | " \"object\": \"file\",\n", 517 | " \"id\": \"file-6adVuM8Pm7UdksqYKxVklSJg\",\n", 518 | " \"purpose\": \"fine-tune\",\n", 519 | " \"filename\": \"file\",\n", 520 | " \"bytes\": 200354,\n", 521 | " \"created_at\": 1692905441,\n", 522 | " \"status\": \"uploaded\",\n", 523 | " \"status_details\": null\n", 524 | "}" 525 | ] 526 | }, 527 | "execution_count": 30, 528 | "metadata": {}, 529 | "output_type": "execute_result" 530 | } 531 | ], 532 | "source": [ 533 | "file_response" 534 | ] 535 | }, 536 | { 537 | "cell_type": "code", 538 | "execution_count": 31, 539 | "metadata": {}, 540 | "outputs": [], 541 | "source": [ 542 | "import time\n", 543 | "\n", 544 | "response = None\n", 545 | "\n", 546 | "while not response:\n", 547 | " try:\n", 548 | " response = openai.FineTuningJob.create(training_file=file_response.id, model=\"gpt-3.5-turbo\")\n", 549 | " except:\n", 550 | " time.sleep(5)" 551 | ] 552 | }, 553 | { 554 | "cell_type": "code", 555 | "execution_count": 32, 556 | "metadata": {}, 557 | "outputs": [ 558 | { 559 | "data": { 560 | "text/plain": [ 561 | " JSON: {\n", 562 | " \"object\": \"fine_tuning.job\",\n", 563 | " \"id\": \"ftjob-SUD1CJJwBbTvrkV2G5K6bkUY\",\n", 564 | " \"model\": \"gpt-3.5-turbo-0613\",\n", 565 | " \"created_at\": 1692905472,\n", 566 | " \"finished_at\": null,\n", 567 | " \"fine_tuned_model\": null,\n", 568 | " \"organization_id\": \"org-VrtlDUDw6aAbBGBDNn1W797P\",\n", 569 | " \"result_files\": [],\n", 570 | " \"status\": \"created\",\n", 571 | " \"validation_file\": null,\n", 572 | " \"training_file\": \"file-6adVuM8Pm7UdksqYKxVklSJg\",\n", 573 | " \"hyperparameters\": {\n", 574 | " \"n_epochs\": 3\n", 575 | " },\n", 576 | " \"trained_tokens\": null\n", 577 | "}" 578 | ] 579 | }, 580 | "execution_count": 32, 581 | "metadata": {}, 582 | "output_type": "execute_result" 583 | } 584 | ], 585 | "source": [ 586 | "response" 587 | ] 588 | }, 589 | { 590 | "cell_type": "code", 591 | "execution_count": 33, 592 | "metadata": {}, 593 | "outputs": [], 594 | "source": [ 595 | "training_id = response.id" 596 | ] 597 | }, 598 | { 599 | "cell_type": "code", 600 | "execution_count": 34, 601 | "metadata": {}, 602 | "outputs": [ 603 | { 604 | "data": { 605 | "text/plain": [ 606 | " JSON: {\n", 607 | " \"object\": \"fine_tuning.job\",\n", 608 | " \"id\": \"ftjob-SUD1CJJwBbTvrkV2G5K6bkUY\",\n", 609 | " \"model\": \"gpt-3.5-turbo-0613\",\n", 610 | " \"created_at\": 1692905472,\n", 611 | " \"finished_at\": null,\n", 612 | " \"fine_tuned_model\": null,\n", 613 | " \"organization_id\": \"org-VrtlDUDw6aAbBGBDNn1W797P\",\n", 614 | " \"result_files\": [],\n", 615 | " \"status\": \"running\",\n", 616 | " \"validation_file\": null,\n", 617 | " \"training_file\": \"file-6adVuM8Pm7UdksqYKxVklSJg\",\n", 618 | " \"hyperparameters\": {\n", 619 | " \"n_epochs\": 3\n", 620 | " },\n", 621 | " \"trained_tokens\": null\n", 622 | "}" 623 | ] 624 | }, 625 | "execution_count": 34, 626 | "metadata": {}, 627 | "output_type": "execute_result" 628 | } 629 | ], 630 | "source": [ 631 | "openai.FineTuningJob.retrieve(training_id)" 632 | ] 633 | }, 634 | { 635 | "cell_type": "code", 636 | "execution_count": 35, 637 | "metadata": {}, 638 | "outputs": [ 639 | { 640 | "name": "stdout", 641 | "output_type": "stream", 642 | "text": [ 643 | "{\n", 644 | " \"object\": \"list\",\n", 645 | " \"data\": [\n", 646 | " {\n", 647 | " \"object\": \"fine_tuning.job.event\",\n", 648 | " \"id\": \"ftevent-ioAEPvMGT9I4dJz2hVw9EXc0\",\n", 649 | " \"created_at\": 1692906144,\n", 650 | " \"level\": \"info\",\n", 651 | " \"message\": \"Fine-tuning job successfully completed\",\n", 652 | " \"data\": null,\n", 653 | " \"type\": \"message\"\n", 654 | " },\n", 655 | " {\n", 656 | " \"object\": \"fine_tuning.job.event\",\n", 657 | " \"id\": \"ftevent-UtuXIbtM5Ht7tZTnYFOunmDF\",\n", 658 | " \"created_at\": 1692906142,\n", 659 | " \"level\": \"info\",\n", 660 | " \"message\": \"New fine-tuned model created: ft:gpt-3.5-turbo-0613:ox::7rAVhh9B\",\n", 661 | " \"data\": null,\n", 662 | " \"type\": \"message\"\n", 663 | " },\n", 664 | " {\n", 665 | " \"object\": \"fine_tuning.job.event\",\n", 666 | " \"id\": \"ftevent-CyQJQN8bxXIOYOVWCKztOqeG\",\n", 667 | " \"created_at\": 1692906130,\n", 668 | " \"level\": \"info\",\n", 669 | " \"message\": \"Step 130/138: training loss=0.00\",\n", 670 | " \"data\": {\n", 671 | " \"step\": 130,\n", 672 | " \"train_loss\": 0.0009155995794571936,\n", 673 | " \"train_mean_token_accuracy\": 1.0\n", 674 | " },\n", 675 | " \"type\": \"metrics\"\n", 676 | " },\n", 677 | " {\n", 678 | " \"object\": \"fine_tuning.job.event\",\n", 679 | " \"id\": \"ftevent-9a8bIiuHc8layW4GYZ8DBFox\",\n", 680 | " \"created_at\": 1692906118,\n", 681 | " \"level\": \"info\",\n", 682 | " \"message\": \"Step 120/138: training loss=0.03\",\n", 683 | " \"data\": {\n", 684 | " \"step\": 120,\n", 685 | " \"train_loss\": 0.02666545659303665,\n", 686 | " \"train_mean_token_accuracy\": 1.0\n", 687 | " },\n", 688 | " \"type\": \"metrics\"\n", 689 | " },\n", 690 | " {\n", 691 | " \"object\": \"fine_tuning.job.event\",\n", 692 | " \"id\": \"ftevent-b9o1DClrlnuwl4uMrE18p6wP\",\n", 693 | " \"created_at\": 1692906106,\n", 694 | " \"level\": \"info\",\n", 695 | " \"message\": \"Step 110/138: training loss=0.00\",\n", 696 | " \"data\": {\n", 697 | " \"step\": 110,\n", 698 | " \"train_loss\": 0.0027033654041588306,\n", 699 | " \"train_mean_token_accuracy\": 1.0\n", 700 | " },\n", 701 | " \"type\": \"metrics\"\n", 702 | " },\n", 703 | " {\n", 704 | " \"object\": \"fine_tuning.job.event\",\n", 705 | " \"id\": \"ftevent-UsHiishNwiyCZnZJm5LFC9yr\",\n", 706 | " \"created_at\": 1692906094,\n", 707 | " \"level\": \"info\",\n", 708 | " \"message\": \"Step 100/138: training loss=0.47\",\n", 709 | " \"data\": {\n", 710 | " \"step\": 100,\n", 711 | " \"train_loss\": 0.46932321786880493,\n", 712 | " \"train_mean_token_accuracy\": 0.9069767594337463\n", 713 | " },\n", 714 | " \"type\": \"metrics\"\n", 715 | " },\n", 716 | " {\n", 717 | " \"object\": \"fine_tuning.job.event\",\n", 718 | " \"id\": \"ftevent-i1bj3zcYVTCSuqxONYIkxja1\",\n", 719 | " \"created_at\": 1692906081,\n", 720 | " \"level\": \"info\",\n", 721 | " \"message\": \"Step 90/138: training loss=0.09\",\n", 722 | " \"data\": {\n", 723 | " \"step\": 90,\n", 724 | " \"train_loss\": 0.08837084472179413,\n", 725 | " \"train_mean_token_accuracy\": 0.9375\n", 726 | " },\n", 727 | " \"type\": \"metrics\"\n", 728 | " },\n", 729 | " {\n", 730 | " \"object\": \"fine_tuning.job.event\",\n", 731 | " \"id\": \"ftevent-dyKfKkFGEq6eRRA6grEl6PM5\",\n", 732 | " \"created_at\": 1692906069,\n", 733 | " \"level\": \"info\",\n", 734 | " \"message\": \"Step 80/138: training loss=0.09\",\n", 735 | " \"data\": {\n", 736 | " \"step\": 80,\n", 737 | " \"train_loss\": 0.08918418735265732,\n", 738 | " \"train_mean_token_accuracy\": 0.9767441749572754\n", 739 | " },\n", 740 | " \"type\": \"metrics\"\n", 741 | " },\n", 742 | " {\n", 743 | " \"object\": \"fine_tuning.job.event\",\n", 744 | " \"id\": \"ftevent-puI76W7mIbnpf2pbIooyhfu6\",\n", 745 | " \"created_at\": 1692906059,\n", 746 | " \"level\": \"info\",\n", 747 | " \"message\": \"Step 70/138: training loss=0.19\",\n", 748 | " \"data\": {\n", 749 | " \"step\": 70,\n", 750 | " \"train_loss\": 0.18960601091384888,\n", 751 | " \"train_mean_token_accuracy\": 0.9333333373069763\n", 752 | " },\n", 753 | " \"type\": \"metrics\"\n", 754 | " },\n", 755 | " {\n", 756 | " \"object\": \"fine_tuning.job.event\",\n", 757 | " \"id\": \"ftevent-i5hjEwzdWK1KcWSs4r04P6Yc\",\n", 758 | " \"created_at\": 1692906047,\n", 759 | " \"level\": \"info\",\n", 760 | " \"message\": \"Step 60/138: training loss=0.12\",\n", 761 | " \"data\": {\n", 762 | " \"step\": 60,\n", 763 | " \"train_loss\": 0.12298834323883057,\n", 764 | " \"train_mean_token_accuracy\": 0.9285714030265808\n", 765 | " },\n", 766 | " \"type\": \"metrics\"\n", 767 | " }\n", 768 | " ],\n", 769 | " \"has_more\": true\n", 770 | "}\n", 771 | "Done!\n" 772 | ] 773 | } 774 | ], 775 | "source": [ 776 | "from IPython.display import clear_output\n", 777 | "\n", 778 | "while openai.FineTuningJob.retrieve(training_id).status == \"running\":\n", 779 | " clear_output(wait=True)\n", 780 | " time.sleep(5)\n", 781 | " print(openai.FineTuningJob.list_events(id=training_id, limit=10))\n", 782 | "\n", 783 | "print(\"Done!\")" 784 | ] 785 | }, 786 | { 787 | "cell_type": "code", 788 | "execution_count": 36, 789 | "metadata": {}, 790 | "outputs": [ 791 | { 792 | "data": { 793 | "text/plain": [ 794 | " JSON: {\n", 795 | " \"object\": \"fine_tuning.job\",\n", 796 | " \"id\": \"ftjob-SUD1CJJwBbTvrkV2G5K6bkUY\",\n", 797 | " \"model\": \"gpt-3.5-turbo-0613\",\n", 798 | " \"created_at\": 1692905472,\n", 799 | " \"finished_at\": 1692906144,\n", 800 | " \"fine_tuned_model\": \"ft:gpt-3.5-turbo-0613:ox::7rAVhh9B\",\n", 801 | " \"organization_id\": \"org-VrtlDUDw6aAbBGBDNn1W797P\",\n", 802 | " \"result_files\": [\n", 803 | " \"file-lb9W8pRtbjI5JP4YoZ2AeKtk\"\n", 804 | " ],\n", 805 | " \"status\": \"succeeded\",\n", 806 | " \"validation_file\": null,\n", 807 | " \"training_file\": \"file-6adVuM8Pm7UdksqYKxVklSJg\",\n", 808 | " \"hyperparameters\": {\n", 809 | " \"n_epochs\": 3\n", 810 | " },\n", 811 | " \"trained_tokens\": 159942\n", 812 | "}" 813 | ] 814 | }, 815 | "execution_count": 36, 816 | "metadata": {}, 817 | "output_type": "execute_result" 818 | } 819 | ], 820 | "source": [ 821 | "openai.FineTuningJob.retrieve(training_id)" 822 | ] 823 | }, 824 | { 825 | "cell_type": "code", 826 | "execution_count": 37, 827 | "metadata": {}, 828 | "outputs": [], 829 | "source": [ 830 | "ft_model_id = openai.FineTuningJob.retrieve(training_id).fine_tuned_model" 831 | ] 832 | }, 833 | { 834 | "cell_type": "markdown", 835 | "metadata": {}, 836 | "source": [ 837 | "### Evaluating the fine-tuned model\n", 838 | "\n", 839 | "Now that we've fine-tuned our model on the `gpt-4` enhanced question answers - let's see how it performs on our `raga` evaluation!" 840 | ] 841 | }, 842 | { 843 | "cell_type": "code", 844 | "execution_count": 38, 845 | "metadata": {}, 846 | "outputs": [], 847 | "source": [ 848 | "from llama_index import ServiceContext\n", 849 | "from llama_index.llms import OpenAI\n", 850 | "from llama_index.callbacks import OpenAIFineTuningHandler\n", 851 | "from llama_index.callbacks import CallbackManager\n", 852 | "\n", 853 | "\n", 854 | "ft_context = ServiceContext.from_defaults(\n", 855 | " llm=OpenAI(model=ft_model_id, temperature=0.3),\n", 856 | " context_window=2048, # limit the context window artifically to test refine process\n", 857 | ")" 858 | ] 859 | }, 860 | { 861 | "cell_type": "code", 862 | "execution_count": 39, 863 | "metadata": {}, 864 | "outputs": [], 865 | "source": [ 866 | "questions = []\n", 867 | "with open(\"eval_questions.txt\", \"r\") as f:\n", 868 | " for line in f:\n", 869 | " questions.append(line.strip())" 870 | ] 871 | }, 872 | { 873 | "cell_type": "code", 874 | "execution_count": 40, 875 | "metadata": {}, 876 | "outputs": [], 877 | "source": [ 878 | "from llama_index import VectorStoreIndex\n", 879 | "\n", 880 | "index = VectorStoreIndex.from_documents(documents, service_context=ft_context)\n", 881 | "\n", 882 | "query_engine = index.as_query_engine(similarity_top_k=2)" 883 | ] 884 | }, 885 | { 886 | "cell_type": "code", 887 | "execution_count": 41, 888 | "metadata": {}, 889 | "outputs": [], 890 | "source": [ 891 | "contexts = []\n", 892 | "answers = []\n", 893 | "\n", 894 | "for question in questions:\n", 895 | " response = query_engine.query(question)\n", 896 | " contexts.append([x.node.get_content() for x in response.source_nodes])\n", 897 | " answers.append(str(response))" 898 | ] 899 | }, 900 | { 901 | "cell_type": "code", 902 | "execution_count": 42, 903 | "metadata": {}, 904 | "outputs": [ 905 | { 906 | "name": "stdout", 907 | "output_type": "stream", 908 | "text": [ 909 | "evaluating with [answer_relevancy]\n" 910 | ] 911 | }, 912 | { 913 | "name": "stderr", 914 | "output_type": "stream", 915 | "text": [ 916 | "100%|██████████| 3/3 [01:03<00:00, 21.28s/it]\n" 917 | ] 918 | }, 919 | { 920 | "name": "stdout", 921 | "output_type": "stream", 922 | "text": [ 923 | "evaluating with [faithfulness]\n" 924 | ] 925 | }, 926 | { 927 | "name": "stderr", 928 | "output_type": "stream", 929 | "text": [ 930 | "100%|██████████| 3/3 [04:22<00:00, 87.38s/it]\n" 931 | ] 932 | }, 933 | { 934 | "name": "stdout", 935 | "output_type": "stream", 936 | "text": [ 937 | "{'ragas_score': 0.8599, 'answer_relevancy': 0.9398, 'faithfulness': 0.7925}\n" 938 | ] 939 | } 940 | ], 941 | "source": [ 942 | "from datasets import Dataset\n", 943 | "from ragas import evaluate\n", 944 | "from ragas.metrics import answer_relevancy, faithfulness\n", 945 | "\n", 946 | "ds = Dataset.from_dict(\n", 947 | " {\n", 948 | " \"question\": questions,\n", 949 | " \"answer\": answers,\n", 950 | " \"contexts\": contexts,\n", 951 | " }\n", 952 | ")\n", 953 | "\n", 954 | "result = evaluate(ds, [answer_relevancy, faithfulness])\n", 955 | "print(result)" 956 | ] 957 | }, 958 | { 959 | "cell_type": "code", 960 | "execution_count": 53, 961 | "metadata": {}, 962 | "outputs": [], 963 | "source": [ 964 | "ft_eval = {'ragas_score': 0.8599, 'answer_relevancy': 0.9398, 'faithfulness': 0.7925}" 965 | ] 966 | }, 967 | { 968 | "cell_type": "markdown", 969 | "metadata": {}, 970 | "source": [ 971 | "### Exploring Differences\n", 972 | "\n", 973 | "Now we can compare the outputs of the two models!" 974 | ] 975 | }, 976 | { 977 | "cell_type": "code", 978 | "execution_count": 43, 979 | "metadata": {}, 980 | "outputs": [], 981 | "source": [ 982 | "from llama_index import VectorStoreIndex\n", 983 | "\n", 984 | "index = VectorStoreIndex.from_documents(documents)" 985 | ] 986 | }, 987 | { 988 | "cell_type": "code", 989 | "execution_count": 44, 990 | "metadata": {}, 991 | "outputs": [], 992 | "source": [ 993 | "questions = []\n", 994 | "with open(\"eval_questions.txt\", \"r\") as f:\n", 995 | " for line in f:\n", 996 | " questions.append(line.strip())" 997 | ] 998 | }, 999 | { 1000 | "cell_type": "code", 1001 | "execution_count": 45, 1002 | "metadata": {}, 1003 | "outputs": [ 1004 | { 1005 | "name": "stdout", 1006 | "output_type": "stream", 1007 | "text": [ 1008 | "What did Arthur see when he moved back and looked again at the solid wall of blue ice?\n" 1009 | ] 1010 | } 1011 | ], 1012 | "source": [ 1013 | "print(questions[12])" 1014 | ] 1015 | }, 1016 | { 1017 | "cell_type": "code", 1018 | "execution_count": 46, 1019 | "metadata": {}, 1020 | "outputs": [], 1021 | "source": [ 1022 | "from llama_index.response.notebook_utils import display_response\n", 1023 | "from llama_index import ServiceContext\n", 1024 | "from llama_index.llms import OpenAI\n", 1025 | "\n", 1026 | "\n", 1027 | "gpt_35_context = ServiceContext.from_defaults(\n", 1028 | " llm=OpenAI(model=\"gpt-3.5-turbo\", temperature=0.3),\n", 1029 | " context_window=2048, # limit the context window artifically to test refine process\n", 1030 | ")" 1031 | ] 1032 | }, 1033 | { 1034 | "cell_type": "code", 1035 | "execution_count": 47, 1036 | "metadata": {}, 1037 | "outputs": [ 1038 | { 1039 | "data": { 1040 | "text/markdown": [ 1041 | "**`Final Response:`** Arthur saw the sky when he moved back and looked again at the solid wall of blue ice." 1042 | ], 1043 | "text/plain": [ 1044 | "" 1045 | ] 1046 | }, 1047 | "metadata": {}, 1048 | "output_type": "display_data" 1049 | } 1050 | ], 1051 | "source": [ 1052 | "query_engine = index.as_query_engine(service_context=gpt_35_context)\n", 1053 | "\n", 1054 | "response = query_engine.query(questions[12])\n", 1055 | "\n", 1056 | "display_response(response)" 1057 | ] 1058 | }, 1059 | { 1060 | "cell_type": "code", 1061 | "execution_count": 50, 1062 | "metadata": {}, 1063 | "outputs": [], 1064 | "source": [ 1065 | "from llama_index import ServiceContext\n", 1066 | "from llama_index.llms import OpenAI\n", 1067 | "\n", 1068 | "\n", 1069 | "ft_context = ServiceContext.from_defaults(\n", 1070 | " llm=OpenAI(model=ft_model_id, temperature=0.3),\n", 1071 | " context_window=2048, # limit the context window artifically to test refine process\n", 1072 | ")" 1073 | ] 1074 | }, 1075 | { 1076 | "cell_type": "code", 1077 | "execution_count": 51, 1078 | "metadata": {}, 1079 | "outputs": [ 1080 | { 1081 | "data": { 1082 | "text/markdown": [ 1083 | "**`Final Response:`** Arthur saw something within the frozen depths." 1084 | ], 1085 | "text/plain": [ 1086 | "" 1087 | ] 1088 | }, 1089 | "metadata": {}, 1090 | "output_type": "display_data" 1091 | } 1092 | ], 1093 | "source": [ 1094 | "query_engine = index.as_query_engine(service_context=ft_context)\n", 1095 | "\n", 1096 | "response = query_engine.query(questions[12])\n", 1097 | "\n", 1098 | "display_response(response)" 1099 | ] 1100 | }, 1101 | { 1102 | "cell_type": "code", 1103 | "execution_count": 60, 1104 | "metadata": {}, 1105 | "outputs": [ 1106 | { 1107 | "name": "stdout", 1108 | "output_type": "stream", 1109 | "text": [ 1110 | "Base model answer_relevancy : 0.9308\n", 1111 | "Fine-tuned model answer_relevancy : 0.9398\n", 1112 | "Improvement answer_relevancy : 0.90%\n", 1113 | "\n", 1114 | "Base model faithfulness : 0.7375\n", 1115 | "Fine-tuned model faithfulness : 0.7925\n", 1116 | "Improvement faithfulness : 5.50%\n", 1117 | "\n" 1118 | ] 1119 | } 1120 | ], 1121 | "source": [ 1122 | "metric_list = [\"answer_relevancy\", \"faithfulness\"]\n", 1123 | "\n", 1124 | "for metric in metric_list:\n", 1125 | " print(\"Base model\", metric, \":\", base_eval[metric])\n", 1126 | " print(\"Fine-tuned model\", metric, \":\", ft_eval[metric])\n", 1127 | " print(f\"Improvement {metric} : {(ft_eval[metric] - base_eval[metric])*100:.2f}%\")\n", 1128 | " print()" 1129 | ] 1130 | } 1131 | ], 1132 | "metadata": { 1133 | "kernelspec": { 1134 | "display_name": "aims-barbenheimer-chainlit", 1135 | "language": "python", 1136 | "name": "python3" 1137 | }, 1138 | "language_info": { 1139 | "codemirror_mode": { 1140 | "name": "ipython", 1141 | "version": 3 1142 | }, 1143 | "file_extension": ".py", 1144 | "mimetype": "text/x-python", 1145 | "name": "python", 1146 | "nbconvert_exporter": "python", 1147 | "pygments_lexer": "ipython3", 1148 | "version": "3.11.4" 1149 | }, 1150 | "orig_nbformat": 4 1151 | }, 1152 | "nbformat": 4, 1153 | "nbformat_minor": 2 1154 | } 1155 | -------------------------------------------------------------------------------- /Week 2/Thursday/README.md: -------------------------------------------------------------------------------- 1 | # Building A Finetuning Machine Using Chainlit 2 | 3 | Today we'll take a look at how we can wrap our finetuning process into Chainlit and have our own dynamic fine-tuning machine. 4 | 5 | We'll be leveraging [this](https://github.com/AI-Maker-Space/LLM-Ops-Cohort-1/blob/main/Week%202/Thursday/Automated%20Fine-tuning%20with%20LLamaIndex.ipynb) notebook for finetuning. 6 | 7 | # Build 🏗️ 8 | There are 2 main tasks for this assignment: 9 | 10 | - Verify you can run the notebook 11 | - Wrap the notebook using Chainlit 12 | 13 | # Ship 🚢 14 | Construct a Chainlit application using the notebook that allows users to interface with the application. 15 | 16 | # Share 🚀 17 | Make a social media post about your final application and tag @AIMakerspace 18 | -------------------------------------------------------------------------------- /Week 2/Tuesday/LLamaIndex RAQA Tool (Assignment Version).ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Creating a more robust RAQA system using LlamaIndex\n", 8 | "\n", 9 | "We'll be putting together a system for querying both qualitative and quantitative data using LlamaIndex. \n", 10 | "\n", 11 | "To stick to a theme, we'll continue to use BarbenHeimer data as our base - but this can, and should, be extended to other topics/domains.\n", 12 | "\n", 13 | "# Build 🏗️\n", 14 | "There are 3 main tasks in this notebook:\n", 15 | "\n", 16 | "- Create a Qualitative VectorStore query engine\n", 17 | "- Create a quantitative NLtoSQL query engine\n", 18 | "- Combine the two using LlamaIndex's OpenAI agent framework.\n", 19 | "\n", 20 | "# Ship 🚢\n", 21 | "Create an host a Gradio or Chainlit application to serve your project on Hugging Face spaces.\n", 22 | "\n", 23 | "# Share 🚀\n", 24 | "Make a social media post about your final application and tag @AIMakerspace" 25 | ] 26 | }, 27 | { 28 | "cell_type": "markdown", 29 | "metadata": {}, 30 | "source": [ 31 | "### A note on terminology:\n", 32 | "\n", 33 | "You'll notice that there are quite a few similarities between LangChain and LlamaIndex. LlamaIndex can largely be thought of as an extension to LangChain, in some ways - but they moved some of the language around. Let's spend a few moments disambiguating the language.\n", 34 | "\n", 35 | "- `QueryEngine` -> `RetrievalQA`:\n", 36 | " - `QueryEngine` is just LlamaIndex's way of indicating something is an LLM \"chain\" on top of a retrieval system\n", 37 | "- `OpenAIAgent` vs. `ZeroShotAgent`:\n", 38 | " - The two agents have the same fundamental pattern: Decide which of a list of tools to use to answer a user's query.\n", 39 | " - `OpenAIAgent` (LlamaIndex's primary agent) does not need to rely on an agent excecutor due to the fact that it is leveraging OpenAI's [functional api](https://openai.com/blog/function-calling-and-other-api-updates) which allows the agent to interface \"directly\" with the tools instead of operating through an intermediary application process.\n", 40 | "\n", 41 | "There is, however, a much large terminological difference when it comes to discussing data.\n", 42 | "\n", 43 | "##### Nodes vs. Documents\n", 44 | "\n", 45 | "As you're aware of from the previous weeks assignments, there's an idea of `documents` in NLP which refers to text objects that exist within a corpus of documents.\n", 46 | "\n", 47 | "LlamaIndex takes this a step further and reclassifies `documents` as `nodes`. Confusingly, it refers to the `Source Document` as simply `Documents`.\n", 48 | "\n", 49 | "The `Document` -> `node` structure is, almost exactly, equivalent to the `Source Document` -> `Document` structure found in LangChain - but the new terminology comes with some clarity about different structure-indices. \n", 50 | "\n", 51 | "We won't be leveraging those structured indicies today, but we will be leveraging a \"benefit\" of the `node` structure that exists as a default in LlamaIndex, which is the ability to quickly filter nodes based on their metadata.\n", 52 | "\n", 53 | "![image](https://i.imgur.com/B1QDjs5.png)" 54 | ] 55 | }, 56 | { 57 | "cell_type": "markdown", 58 | "metadata": {}, 59 | "source": [ 60 | "### BOILERPLATE\n", 61 | "\n", 62 | "This is only relevant when running the code in a Jupyter Notebook." 63 | ] 64 | }, 65 | { 66 | "cell_type": "code", 67 | "execution_count": null, 68 | "metadata": {}, 69 | "outputs": [], 70 | "source": [ 71 | "import nest_asyncio\n", 72 | "\n", 73 | "nest_asyncio.apply()\n", 74 | "\n", 75 | "import logging\n", 76 | "import sys\n", 77 | "\n", 78 | "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n", 79 | "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))" 80 | ] 81 | }, 82 | { 83 | "cell_type": "markdown", 84 | "metadata": {}, 85 | "source": [ 86 | "### Primary Dependencies and Context Setting" 87 | ] 88 | }, 89 | { 90 | "cell_type": "markdown", 91 | "metadata": {}, 92 | "source": [ 93 | "#### Dependencies and OpenAI API key setting\n", 94 | "\n", 95 | "First of all, we'll need our primary libraries - and to set up our OpenAI API key." 96 | ] 97 | }, 98 | { 99 | "cell_type": "code", 100 | "execution_count": null, 101 | "metadata": {}, 102 | "outputs": [], 103 | "source": [ 104 | "!pip install -U -q openai==0.27.8 llama-index==0.8.6 nltk==3.8.1" 105 | ] 106 | }, 107 | { 108 | "cell_type": "code", 109 | "execution_count": null, 110 | "metadata": {}, 111 | "outputs": [], 112 | "source": [ 113 | "import os\n", 114 | "import getpass\n", 115 | "\n", 116 | "os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OpenAI API Key: \")\n", 117 | "\n", 118 | "import openai\n", 119 | "openai.api_key = os.environ[\"OPENAI_API_KEY\"]" 120 | ] 121 | }, 122 | { 123 | "cell_type": "markdown", 124 | "metadata": {}, 125 | "source": [ 126 | "#### Context Setting\n", 127 | "\n", 128 | "Now, LlamaIndex has the ability to set `ServiceContext`. You can think of this as a config file of sorts. The basic idea here is that we use this to establish some core properties and then can pass it to various services. \n", 129 | "\n", 130 | "While we could set this up as a global context, we're going to leave it as `ServiceContext` so we can see where it's applied.\n", 131 | "\n", 132 | "We'll set a few significant contexts:\n", 133 | "\n", 134 | "- `chunk_size` - this is what it says on the tin\n", 135 | "- `llm` - this is where we can set what model we wish to use as our primary LLM when we're making `QueryEngine`s and more\n", 136 | "- `embed_model` - this will help us keep our embedding model consistent across use cases\n", 137 | "\n", 138 | "\n", 139 | "We'll also create some resources we're going to keep consistent across all of our indices today.\n", 140 | "\n", 141 | "- `text_splitter` - This is what we'll use to split our text, feel free to experiment here\n", 142 | "- `SimpleNodeParser` - This is what will work in tandem with the `text_splitter` to parse our full sized documents into nodes." 143 | ] 144 | }, 145 | { 146 | "cell_type": "code", 147 | "execution_count": null, 148 | "metadata": {}, 149 | "outputs": [], 150 | "source": [ 151 | "from llama_index import ServiceContext\n", 152 | "from llama_index.node_parser.simple import SimpleNodeParser\n", 153 | "from llama_index.langchain_helpers.text_splitter import TokenTextSplitter\n", 154 | "from llama_index.llms import OpenAI\n", 155 | "from llama_index.embeddings.openai import OpenAIEmbedding\n", 156 | "\n", 157 | "embed_model = ### YOUR CODE HERE\n", 158 | "chunk_size = ### YOUR CODE HERE\n", 159 | "llm = OpenAI(\n", 160 | " temperature=0, \n", 161 | " model=\"YOUR MODEL HERE\", \n", 162 | " streaming=True\n", 163 | ")\n", 164 | "\n", 165 | "service_context = ServiceContext.from_defaults(\n", 166 | " llm=### YOUR CODE HERE, \n", 167 | " chunk_size=### YOUR CODE HERE, \n", 168 | " embed_model=### YOUR CODE HERE\n", 169 | ")\n", 170 | "\n", 171 | "text_splitter = TokenTextSplitter(\n", 172 | " chunk_size=### YOUR CODE HERE\n", 173 | ")\n", 174 | "\n", 175 | "node_parser = SimpleNodeParser(\n", 176 | " text_splitter=### YOUR CODE HERE\n", 177 | ")" 178 | ] 179 | }, 180 | { 181 | "cell_type": "markdown", 182 | "metadata": {}, 183 | "source": [ 184 | "### BarbenHeimer Wikipedia Retrieval Tool\n", 185 | "\n", 186 | "Now we can get to work creating our semantic `QueryEngine`!\n", 187 | "\n", 188 | "We'll follow a similar pattern as we did with LangChain here - and the first step (as always) is to get dependencies." 189 | ] 190 | }, 191 | { 192 | "cell_type": "code", 193 | "execution_count": null, 194 | "metadata": {}, 195 | "outputs": [], 196 | "source": [ 197 | "!pip install -U -q chromadb==0.4.6 tiktoken==0.4.0 sentence-transformers==2.2.2 pydantic==1.10.11" 198 | ] 199 | }, 200 | { 201 | "cell_type": "code", 202 | "execution_count": null, 203 | "metadata": {}, 204 | "outputs": [], 205 | "source": [ 206 | "from llama_index import VectorStoreIndex\n", 207 | "from llama_index.vector_stores import ChromaVectorStore\n", 208 | "from llama_index.storage.storage_context import StorageContext\n", 209 | "import chromadb" 210 | ] 211 | }, 212 | { 213 | "cell_type": "markdown", 214 | "metadata": {}, 215 | "source": [ 216 | "#### ChromaDB\n", 217 | "\n", 218 | "We'll be using [ChromaDB](https://www.trychroma.com/) as our `VectorStore` today!\n", 219 | "\n", 220 | "It works in a similar fashion to tools like Pinecone, Weaveate, and more - but it's locally hosted and will serve our purposes fine.\n", 221 | "\n", 222 | "You'll also notice the return of `OpenAIEmbedding()`, which is the embeddings model we'll be leveraging. Of course, this is using the `ada` model under the hood - and already comes equipped with in-memory caching.\n", 223 | "\n", 224 | "You'll notice we can pass our `service_context` into our `VectorStoreIndex`!" 225 | ] 226 | }, 227 | { 228 | "cell_type": "code", 229 | "execution_count": null, 230 | "metadata": {}, 231 | "outputs": [], 232 | "source": [ 233 | "chroma_client = chromadb.Client()\n", 234 | "chroma_collection = chroma_client.create_collection(\"wikipedia_barbie_opp\")\n", 235 | "\n", 236 | "vector_store = ChromaVectorStore(chroma_collection=chroma_collection)\n", 237 | "storage_context = StorageContext.from_defaults(vector_store=vector_store)\n", 238 | "wiki_vector_index = VectorStoreIndex([], storage_context=storage_context, service_context=service_context)" 239 | ] 240 | }, 241 | { 242 | "cell_type": "code", 243 | "execution_count": null, 244 | "metadata": {}, 245 | "outputs": [], 246 | "source": [ 247 | "!pip install -U -q wikipedia" 248 | ] 249 | }, 250 | { 251 | "cell_type": "markdown", 252 | "metadata": {}, 253 | "source": [ 254 | "Essentially the same as the LangChain example - we're just going to be pulling information straight from Wikipedia using the built in `WikipediaReader`.\n", 255 | "\n", 256 | "Setting `auto_suggest=False` ensures we run into fewer auto-correct based errors." 257 | ] 258 | }, 259 | { 260 | "cell_type": "code", 261 | "execution_count": null, 262 | "metadata": {}, 263 | "outputs": [], 264 | "source": [ 265 | "from llama_index.readers.wikipedia import WikipediaReader\n", 266 | "\n", 267 | "movie_list = [\"Barbie (film)\", \"Oppenheimer (film)\"]\n", 268 | "\n", 269 | "wiki_docs = WikipediaReader().load_data(pages=movie_list, auto_suggest=False)" 270 | ] 271 | }, 272 | { 273 | "cell_type": "markdown", 274 | "metadata": {}, 275 | "source": [ 276 | "#### Node Construction\n", 277 | "\n", 278 | "Now we will loop through our documents and metadata and construct nodes (associated with particular metadata for easy filtration later).\n", 279 | "\n", 280 | "We're using the `node_parser` we created at the top of the Notebook." 281 | ] 282 | }, 283 | { 284 | "cell_type": "code", 285 | "execution_count": null, 286 | "metadata": {}, 287 | "outputs": [], 288 | "source": [ 289 | "for movie, wiki_doc in zip(movie_list, wiki_docs):\n", 290 | " nodes = ### YOUR CODE HERE\n", 291 | " for node in nodes:\n", 292 | " node.metadata = # YOUR CODE HERE\n", 293 | " wiki_vector_index.### YOUR CODE HERE" 294 | ] 295 | }, 296 | { 297 | "cell_type": "markdown", 298 | "metadata": {}, 299 | "source": [ 300 | "#### Auto Retriever Functional Tool\n", 301 | "\n", 302 | "This tool will leverage OpenAI's functional endpoint to select the correct metadata filter and query the filtered index - only looking at nodes with the desired metadata.\n", 303 | "\n", 304 | "A simplified diagram: ![image](https://i.imgur.com/AICDPav.png)" 305 | ] 306 | }, 307 | { 308 | "cell_type": "markdown", 309 | "metadata": {}, 310 | "source": [ 311 | "First, we need to create our `VectoreStoreInfo` object which will hold all the relevant metadata we need for each component (in this case title metadata).\n", 312 | "\n", 313 | "Notice that you need to include it in a text list." 314 | ] 315 | }, 316 | { 317 | "cell_type": "code", 318 | "execution_count": null, 319 | "metadata": {}, 320 | "outputs": [], 321 | "source": [ 322 | "from llama_index.tools import FunctionTool\n", 323 | "from llama_index.vector_stores.types import (\n", 324 | " VectorStoreInfo,\n", 325 | " MetadataInfo,\n", 326 | " ExactMatchFilter,\n", 327 | " MetadataFilters,\n", 328 | ")\n", 329 | "from llama_index.retrievers import VectorIndexRetriever\n", 330 | "from llama_index.query_engine import RetrieverQueryEngine\n", 331 | "\n", 332 | "from typing import List, Tuple, Any\n", 333 | "from pydantic import BaseModel, Field\n", 334 | "\n", 335 | "top_k = 3\n", 336 | "\n", 337 | "vector_store_info = VectorStoreInfo(\n", 338 | " content_info=\"semantic information about movies\",\n", 339 | " metadata_info=[MetadataInfo(\n", 340 | " name=\"title\",\n", 341 | " type=\"str\",\n", 342 | " description=\"title of the movie, one of [Barbie (film), Oppenheimer (film)]\",\n", 343 | " )]\n", 344 | ")" 345 | ] 346 | }, 347 | { 348 | "cell_type": "markdown", 349 | "metadata": {}, 350 | "source": [ 351 | "Now we'll create our base PyDantic object that we can use to ensure compatability with our application layer. This verifies that the response from the OpenAI endpoint conforms to this schema." 352 | ] 353 | }, 354 | { 355 | "cell_type": "code", 356 | "execution_count": null, 357 | "metadata": {}, 358 | "outputs": [], 359 | "source": [ 360 | "class AutoRetrieveModel(BaseModel):\n", 361 | " query: str = Field(..., description=\"natural language query string\")\n", 362 | " filter_key_list: List[str] = Field(\n", 363 | " ..., description=\"List of metadata filter field names\"\n", 364 | " )\n", 365 | " filter_value_list: List[str] = Field(\n", 366 | " ...,\n", 367 | " description=(\n", 368 | " \"List of metadata filter field values (corresponding to names specified in filter_key_list)\"\n", 369 | " )\n", 370 | " )" 371 | ] 372 | }, 373 | { 374 | "cell_type": "markdown", 375 | "metadata": {}, 376 | "source": [ 377 | "Now we can build our function that we will use to query the functional endpoint.\n", 378 | "\n", 379 | ">The `docstring` is important to the functionality of the application." 380 | ] 381 | }, 382 | { 383 | "cell_type": "code", 384 | "execution_count": null, 385 | "metadata": {}, 386 | "outputs": [], 387 | "source": [ 388 | "def auto_retrieve_fn(\n", 389 | " query: str, filter_key_list: List[str], filter_value_list: List[str]\n", 390 | "):\n", 391 | " \"\"\"Auto retrieval function.\n", 392 | "\n", 393 | " Performs auto-retrieval from a vector database, and then applies a set of filters.\n", 394 | "\n", 395 | " \"\"\"\n", 396 | " query = query or \"Query\"\n", 397 | "\n", 398 | " exact_match_filters = [\n", 399 | " ExactMatchFilter(key=k, value=v)\n", 400 | " for k, v in zip(filter_key_list, filter_value_list)\n", 401 | " ]\n", 402 | " retriever = VectorIndexRetriever(\n", 403 | " wiki_vector_index, filters=MetadataFilters(filters=exact_match_filters), top_k=top_k\n", 404 | " )\n", 405 | " query_engine = RetrieverQueryEngine.from_args(retriever)\n", 406 | "\n", 407 | " response = query_engine.query(query)\n", 408 | " return str(response)" 409 | ] 410 | }, 411 | { 412 | "cell_type": "markdown", 413 | "metadata": {}, 414 | "source": [ 415 | "Now we need to wrap our system in a tool in order to integrate it into the larger application.\n", 416 | "\n", 417 | "Source Code Here:\n", 418 | "- [`FunctionTool`](https://github.com/jerryjliu/llama_index/blob/d24767b0812ac56104497d8f59095eccbe9f2b08/llama_index/tools/function_tool.py#L21)" 419 | ] 420 | }, 421 | { 422 | "cell_type": "code", 423 | "execution_count": null, 424 | "metadata": {}, 425 | "outputs": [], 426 | "source": [ 427 | "description = f\"\"\"\\\n", 428 | "Use this tool to look up semantic information about films.\n", 429 | "The vector database schema is given below:\n", 430 | "{vector_store_info.json()}\n", 431 | "\"\"\"\n", 432 | "\n", 433 | "auto_retrieve_tool = FunctionTool.from_defaults(\n", 434 | " fn=# YOUR CODE HERE,\n", 435 | " name=# YOUR CODE HERE,\n", 436 | " description=# YOUR CODE HERE,\n", 437 | " fn_schema=# YOUR CODE HERE,\n", 438 | ")" 439 | ] 440 | }, 441 | { 442 | "cell_type": "markdown", 443 | "metadata": {}, 444 | "source": [ 445 | "All that's left to do is attach the tool to an OpenAIAgent and let it rip!\n", 446 | "\n", 447 | "Source Code Here:\n", 448 | "- [`OpenAIAgent`](https://github.com/jerryjliu/llama_index/blob/d24767b0812ac56104497d8f59095eccbe9f2b08/llama_index/agent/openai_agent.py#L361)" 449 | ] 450 | }, 451 | { 452 | "cell_type": "code", 453 | "execution_count": null, 454 | "metadata": {}, 455 | "outputs": [], 456 | "source": [ 457 | "from llama_index.agent import OpenAIAgent\n", 458 | "\n", 459 | "agent = OpenAIAgent.from_tools(\n", 460 | " ### YOUR CODE HERE\n", 461 | ")" 462 | ] 463 | }, 464 | { 465 | "cell_type": "code", 466 | "execution_count": null, 467 | "metadata": {}, 468 | "outputs": [], 469 | "source": [ 470 | "response = agent.chat(\"Tell me what happens (briefly) in the Barbie movie.\")\n", 471 | "print(str(response))" 472 | ] 473 | }, 474 | { 475 | "cell_type": "markdown", 476 | "metadata": {}, 477 | "source": [ 478 | "### BarbenHeimer SQL Tool\n", 479 | "\n", 480 | "We'll walk through the steps of creating a natural language to SQL system in the following section.\n", 481 | "\n", 482 | "> NOTICE: This does not have parsing on the inputs or intermediary calls to ensure that users are using safe SQL queries. Use this with caution in a production environment without adding specific guardrails from either side of the application." 483 | ] 484 | }, 485 | { 486 | "cell_type": "code", 487 | "execution_count": null, 488 | "metadata": {}, 489 | "outputs": [], 490 | "source": [ 491 | "!pip install -q -U sqlalchemy pandas" 492 | ] 493 | }, 494 | { 495 | "cell_type": "markdown", 496 | "metadata": {}, 497 | "source": [ 498 | "The next few steps should be largely straightforward, we'll want to:\n", 499 | "\n", 500 | "1. Read in our `.csv` files into `pd.DataFrame` objects\n", 501 | "2. Create an in-memory `sqlite` powered `sqlalchemy` engine\n", 502 | "3. Cast our `pd.DataFrame` objects to the SQL engine\n", 503 | "4. Create an `SQLDatabase` object through LlamaIndex\n", 504 | "5. Use that to create a `QueryEngineTool` that we can interact with through the `NLSQLTableQueryEngine`!\n", 505 | "\n", 506 | "If you get stuck, please consult the documentation." 507 | ] 508 | }, 509 | { 510 | "cell_type": "markdown", 511 | "metadata": {}, 512 | "source": [ 513 | "#### Read `.csv` Into Pandas" 514 | ] 515 | }, 516 | { 517 | "cell_type": "code", 518 | "execution_count": null, 519 | "metadata": {}, 520 | "outputs": [], 521 | "source": [ 522 | "import pandas as pd\n", 523 | "\n", 524 | "barbie_df = # YOUR CODE HERE\n", 525 | "oppenheimer_df = # YOUR CODE HERE" 526 | ] 527 | }, 528 | { 529 | "cell_type": "markdown", 530 | "metadata": {}, 531 | "source": [ 532 | "#### Create SQLAlchemy engine with SQLite" 533 | ] 534 | }, 535 | { 536 | "cell_type": "code", 537 | "execution_count": null, 538 | "metadata": {}, 539 | "outputs": [], 540 | "source": [ 541 | "from sqlalchemy import create_engine\n", 542 | "\n", 543 | "engine = create_engine(\"sqlite+pysqlite:///:memory:\")" 544 | ] 545 | }, 546 | { 547 | "cell_type": "markdown", 548 | "metadata": {}, 549 | "source": [ 550 | "#### Convert `pd.DataFrame` to SQL tables" 551 | ] 552 | }, 553 | { 554 | "cell_type": "code", 555 | "execution_count": null, 556 | "metadata": {}, 557 | "outputs": [], 558 | "source": [ 559 | "barbie_df.to_sql(\n", 560 | " # name of table\n", 561 | " # engine\n", 562 | ")" 563 | ] 564 | }, 565 | { 566 | "cell_type": "code", 567 | "execution_count": null, 568 | "metadata": {}, 569 | "outputs": [], 570 | "source": [ 571 | "oppenheimer_df.to_sql(\n", 572 | " # name of table \n", 573 | " # engine\n", 574 | ")" 575 | ] 576 | }, 577 | { 578 | "cell_type": "markdown", 579 | "metadata": {}, 580 | "source": [ 581 | "#### Construct a `SQLDatabase` index\n", 582 | "\n", 583 | "Source Code Here:\n", 584 | "- [`SQLDatabase`](https://github.com/jerryjliu/llama_index/blob/d24767b0812ac56104497d8f59095eccbe9f2b08/llama_index/langchain_helpers/sql_wrapper.py#L9)" 585 | ] 586 | }, 587 | { 588 | "cell_type": "code", 589 | "execution_count": null, 590 | "metadata": {}, 591 | "outputs": [], 592 | "source": [ 593 | "from llama_index import SQLDatabase\n", 594 | "\n", 595 | "sql_database = SQLDatabase(\n", 596 | " # YOUR CODE HERE, \n", 597 | " include_tables=# YOUR CODE HERE)" 598 | ] 599 | }, 600 | { 601 | "cell_type": "markdown", 602 | "metadata": {}, 603 | "source": [ 604 | "#### Create the NLSQLTableQueryEngine interface for all added SQL tables\n", 605 | "\n", 606 | "Source Code Here:\n", 607 | "- [`NLSQLTableQueryEngine`](https://github.com/jerryjliu/llama_index/blob/d24767b0812ac56104497d8f59095eccbe9f2b08/llama_index/indices/struct_store/sql_query.py#L75C1-L75C1)" 608 | ] 609 | }, 610 | { 611 | "cell_type": "code", 612 | "execution_count": null, 613 | "metadata": {}, 614 | "outputs": [], 615 | "source": [ 616 | "from llama_index.indices.struct_store.sql_query import NLSQLTableQueryEngine\n", 617 | "\n", 618 | "sql_query_engine = NLSQLTableQueryEngine(\n", 619 | " sql_database=# YOUR CODE HERE,\n", 620 | " tables=# YOUR CODE HERE\n", 621 | ")" 622 | ] 623 | }, 624 | { 625 | "cell_type": "markdown", 626 | "metadata": {}, 627 | "source": [ 628 | "#### Wrap It All Up in a `QueryEngineTool`\n", 629 | "\n", 630 | "You'll want to ensure you have a descriptive...description. \n", 631 | "\n", 632 | "An example is provided here:\n", 633 | "\n", 634 | "```\n", 635 | "\"Useful for translating a natural language query into a SQL query over a table containing: \"\n", 636 | "\"barbie, containing information related to reviews of the Barbie movie\"\n", 637 | "\"oppenheimer, containing information related to reviews of the Oppenheimer movie\"\n", 638 | "```\n", 639 | "\n", 640 | "Sorce Code Here: \n", 641 | "\n", 642 | "- [`QueryEngineTool`](https://github.com/jerryjliu/llama_index/blob/d24767b0812ac56104497d8f59095eccbe9f2b08/llama_index/tools/query_engine.py#L13)" 643 | ] 644 | }, 645 | { 646 | "cell_type": "code", 647 | "execution_count": null, 648 | "metadata": {}, 649 | "outputs": [], 650 | "source": [ 651 | "from llama_index.tools.query_engine import QueryEngineTool\n", 652 | "\n", 653 | "sql_tool = QueryEngineTool.from_defaults(\n", 654 | " query_engine=# YOUR CODE HERE,\n", 655 | " name=# Add a name here,\n", 656 | " description=(\n", 657 | " # Add a natural language description here\n", 658 | " ),\n", 659 | "\n", 660 | ")" 661 | ] 662 | }, 663 | { 664 | "cell_type": "code", 665 | "execution_count": null, 666 | "metadata": {}, 667 | "outputs": [], 668 | "source": [ 669 | "agent = OpenAIAgent.from_tools(\n", 670 | " ### YOUR CODE HERE\n", 671 | ")" 672 | ] 673 | }, 674 | { 675 | "cell_type": "code", 676 | "execution_count": null, 677 | "metadata": {}, 678 | "outputs": [], 679 | "source": [ 680 | "response = agent.chat(\"What is the average rating of the two films?\")" 681 | ] 682 | }, 683 | { 684 | "cell_type": "code", 685 | "execution_count": null, 686 | "metadata": {}, 687 | "outputs": [], 688 | "source": [ 689 | "print(str(response))" 690 | ] 691 | }, 692 | { 693 | "cell_type": "markdown", 694 | "metadata": {}, 695 | "source": [ 696 | "### Combining The Tools Together\n", 697 | "\n", 698 | "Now, we can simple add our tools into the `OpenAIAgent`, and off we go!" 699 | ] 700 | }, 701 | { 702 | "cell_type": "code", 703 | "execution_count": null, 704 | "metadata": {}, 705 | "outputs": [], 706 | "source": [ 707 | "barbenheimer_agent = OpenAIAgent.from_tools(\n", 708 | " ### YOUR CODE HERE\n", 709 | ")" 710 | ] 711 | }, 712 | { 713 | "cell_type": "code", 714 | "execution_count": null, 715 | "metadata": {}, 716 | "outputs": [], 717 | "source": [ 718 | "response = barbenheimer_agent.chat(\"What is the lowest rating of the two films - and can you summarize what the reviewer said?\")" 719 | ] 720 | }, 721 | { 722 | "cell_type": "code", 723 | "execution_count": null, 724 | "metadata": {}, 725 | "outputs": [], 726 | "source": [ 727 | "print(str(response))" 728 | ] 729 | }, 730 | { 731 | "cell_type": "code", 732 | "execution_count": null, 733 | "metadata": {}, 734 | "outputs": [], 735 | "source": [ 736 | "response = barbenheimer_agent.chat(\"How many times do the Barbie reviews mention 'Ken', and what is a summary of his character in the Barbie movie?\")" 737 | ] 738 | }, 739 | { 740 | "cell_type": "code", 741 | "execution_count": null, 742 | "metadata": {}, 743 | "outputs": [], 744 | "source": [ 745 | "print(str(response))" 746 | ] 747 | } 748 | ], 749 | "metadata": { 750 | "kernelspec": { 751 | "display_name": "Python 3", 752 | "language": "python", 753 | "name": "python3" 754 | }, 755 | "language_info": { 756 | "codemirror_mode": { 757 | "name": "ipython", 758 | "version": 3 759 | }, 760 | "file_extension": ".py", 761 | "mimetype": "text/x-python", 762 | "name": "python", 763 | "nbconvert_exporter": "python", 764 | "pygments_lexer": "ipython3", 765 | "version": "3.11.4" 766 | }, 767 | "orig_nbformat": 4 768 | }, 769 | "nbformat": 4, 770 | "nbformat_minor": 2 771 | } 772 | -------------------------------------------------------------------------------- /Week 2/Tuesday/README.md: -------------------------------------------------------------------------------- 1 | # Creating a more robust RAQA system using LlamaIndex 2 | 3 | We'll be putting together a system for querying both qualitative and quantitative data using LlamaIndex. 4 | 5 | To stick to a theme, we'll continue to use BarbenHeimer data as our base - but this can, and should, be extended to other topics/domains. 6 | 7 | # Build 🏗️ 8 | There are 3 main tasks in this notebook: 9 | 10 | - Create a Qualitative VectorStore query engine 11 | - Create a quantitative NLtoSQL query engine 12 | - Combine the two using LlamaIndex's OpenAI agent framework. 13 | 14 | # Ship 🚢 15 | Create an host a Gradio or Chainlit application to serve your project on Hugging Face spaces. 16 | 17 | # Share 🚀 18 | Make a social media post about your final application and tag @AIMakerspace -------------------------------------------------------------------------------- /Week 2/Tuesday/requirements.txt: -------------------------------------------------------------------------------- 1 | ipykernel -------------------------------------------------------------------------------- /Week 3/Thursday/Deploy/README.md: -------------------------------------------------------------------------------- 1 | # Deploying an Endpoint and Making it Accessible Through AWS 2 | 3 | Today, we'll be creating a SageMaker endpoint, and serving it through the API Gateway! 4 | 5 | # Build 🏗️ 6 | There is 1 main task for this assignment: 7 | 8 | - Endpoint Hosting on AWS 9 | 10 | # Ship 🚢 11 | Ship the endpoint! 12 | 13 | # Share 🚀 14 | Make a social media post about your final application and tag @AIMakerspace 15 | 16 | # How to Deploy Your Own Sagemaker Endpoint in AWS 17 | 18 | ### Deploy Model 19 | 20 | First things first, you'll need to make sure you have an AWS account, and that you have access to at least 1 `ml.g5.2xlarge` instances. 21 | 22 | > You can check/request available quota through `ServiceQuota` 23 | 24 | Next, we'll want to navigate to `Amazon SageMaker`, and create a domain! You can create a default domain - though you may have to ensure you have the correct IAM permission if you're not the admin on the account. 25 | 26 | After you've done that, you'll want to open SageMaker Studio and navigate to the `Featured JumpStart models` section: 27 | 28 | ![image](https://i.imgur.com/eLZFfMM.png) 29 | 30 | Now, we'll want to deploy our model to an endpoint using the `Deploy` button. 31 | 32 | ![image](https://i.imgur.com/0dxtAkX.png) 33 | 34 | This process will take some time. Please be patient! 35 | 36 | ### Create Lambda with API Gateway 37 | 38 | Next, we'll head to Lambda and `Create Function`! 39 | 40 | We'll want to ensure that we create the function with a new role that we can edit to include the `InvokeEndpoint` permissions. 41 | 42 | Our Lambda will have the following code: 43 | 44 | ```python 45 | import os 46 | import boto3 47 | import json 48 | 49 | # grab environment variables 50 | ENDPOINT_NAME = os.environ['ENDPOINT_NAME'] 51 | runtime= boto3.client('runtime.sagemaker') 52 | 53 | def lambda_handler(event, context): 54 | response = runtime.invoke_endpoint(EndpointName=ENDPOINT_NAME, 55 | ContentType='application/json', 56 | Body=event['body'], 57 | CustomAttributes="accept_eula=true") 58 | 59 | result = json.loads(response['Body'].read().decode()) 60 | 61 | 62 | return { 63 | "statusCode": 200, 64 | "body": json.dumps(result) 65 | } 66 | ``` 67 | 68 | And we'll want to add an environment variable by the name `ENDPOINT_NAME` with our SageMaker endpoint name to our Lambda as well! 69 | 70 | Now, we can add the `API Gateway`, which can be done by adding a `Trigger` in the Function overview. We will want it set to `Open` for ease of use. 71 | 72 | Let's add the required permissions to our role once we add our Lambda! 73 | 74 | ![image](https://i.imgur.com/3ertSKe.png) 75 | 76 | ![image](https://i.imgur.com/GvGSLl6.png) 77 | 78 | We'll add it as an `Inline Policy`: 79 | 80 | ``` 81 | { 82 | "Version": "2012-10-17", 83 | "Statement": [ 84 | { 85 | "Sid": "VisualEditor0", 86 | "Effect": "Allow", 87 | "Action": "sagemaker:InvokeEndpoint", 88 | "Resource": "*" 89 | } 90 | ] 91 | } 92 | ``` 93 | 94 | After this, we should be good to query our endpoint and carry on to the Notebook! 95 | 96 | Make sure to add your API Gateway URL! 97 | 98 | # Sample Notebook for LangChain Access 99 | 100 | [Colab Notebook](https://colab.research.google.com/drive/147Oy3n601T8MFWulmKdCwC7PrxDKuyND?usp=sharing) 101 | -------------------------------------------------------------------------------- /Week 3/Thursday/Evaluation/README.md: -------------------------------------------------------------------------------- 1 | # Evaluating LLMs with LLMs and NLP Metrics 2 | 3 | # Build 🏗️ 4 | There is 1 main task for this assignment: 5 | 6 | - Evaluate Your RAG (built in deploy section) 7 | - Evaluate with EleutherAI Harness on Custom Task 8 | 9 | # Ship 🚢 10 | Share results of your evaluations! 11 | 12 | # Share 🚀 13 | Make a social media post about your final application and tag @AIMakerspace 14 | 15 | # Example Notebooks 16 | 17 | ### Evaluating RAG 18 | 19 | Based on the `Deploy` section: Here's a [notebook](https://colab.research.google.com/drive/1Q2JCptzWd7lj9wggPaipwy3cigmgEGDI?usp=sharing) on how we can implement RAGAS with LangChain! 20 | 21 | ### Evaluation of LLMs (Simple) 22 | 23 | [Notebook](https://colab.research.google.com/drive/1tJGP-Oe_B_lH2EaiD3sNGBoIUv8tpJdO?usp=sharing) 24 | 25 | ### Eleuther AI Evaluation Harness 26 | 27 | [Notebook](https://colab.research.google.com/drive/15B08548oLsXbuUMNuG_T-R6WFvb3bKZp?usp=sharing) 28 | 29 | ### HELM Evaluation 30 | 31 | [Notebook](https://colab.research.google.com/drive/1Tg26V055-ukudzmnE4Tpj-lYoIlRQydJ?usp=sharing) 32 | -------------------------------------------------------------------------------- /Week 3/Thursday/README.md: -------------------------------------------------------------------------------- 1 | # Week 3 Day 2 2 | 3 | We're going to cover 2 separate topics today: 4 | 5 | 1. Simple Deployment on AWS 6 | 2. Evaluation of LLMs 7 | -------------------------------------------------------------------------------- /Week 3/Tuesday/README.md: -------------------------------------------------------------------------------- 1 | # 🖇️ Combining Systems: Chainlit App with Langchain and FastAPI Deployment 2 | 3 | We'll be combining the two systems listed below into a single `docker-compose.yaml` file! 4 | 5 | # Build 🏗️ 6 | There are two tasks for today: 7 | 8 | - Add the Chainlit app to our `docker compose` 9 | - Point your Chainlit app at the Llama2 FastAPI service 10 | 11 | # Ship 🚢 12 | A public IP address pointing to your application. 13 | 14 | # Share 🚀 15 | Make a social media post about your final application and tag @AIMakerspace 16 | -------------------------------------------------------------------------------- /Week 3/Tuesday/chainlit/.env.example: -------------------------------------------------------------------------------- 1 | OPENAI_API_KEY= -------------------------------------------------------------------------------- /Week 3/Tuesday/chainlit/.gitattributes: -------------------------------------------------------------------------------- 1 | *.7z filter=lfs diff=lfs merge=lfs -text 2 | *.arrow filter=lfs diff=lfs merge=lfs -text 3 | *.bin filter=lfs diff=lfs merge=lfs -text 4 | *.bz2 filter=lfs diff=lfs merge=lfs -text 5 | *.ckpt filter=lfs diff=lfs merge=lfs -text 6 | *.ftz filter=lfs diff=lfs merge=lfs -text 7 | *.gz filter=lfs diff=lfs merge=lfs -text 8 | *.h5 filter=lfs diff=lfs merge=lfs -text 9 | *.joblib filter=lfs diff=lfs merge=lfs -text 10 | *.lfs.* filter=lfs diff=lfs merge=lfs -text 11 | *.mlmodel filter=lfs diff=lfs merge=lfs -text 12 | *.model filter=lfs diff=lfs merge=lfs -text 13 | *.msgpack filter=lfs diff=lfs merge=lfs -text 14 | *.npy filter=lfs diff=lfs merge=lfs -text 15 | *.npz filter=lfs diff=lfs merge=lfs -text 16 | *.onnx filter=lfs diff=lfs merge=lfs -text 17 | *.ot filter=lfs diff=lfs merge=lfs -text 18 | *.parquet filter=lfs diff=lfs merge=lfs -text 19 | *.pb filter=lfs diff=lfs merge=lfs -text 20 | *.pickle filter=lfs diff=lfs merge=lfs -text 21 | *.pkl filter=lfs diff=lfs merge=lfs -text 22 | *.pt filter=lfs diff=lfs merge=lfs -text 23 | *.pth filter=lfs diff=lfs merge=lfs -text 24 | *.rar filter=lfs diff=lfs merge=lfs -text 25 | *.safetensors filter=lfs diff=lfs merge=lfs -text 26 | saved_model/**/* filter=lfs diff=lfs merge=lfs -text 27 | *.tar.* filter=lfs diff=lfs merge=lfs -text 28 | *.tflite filter=lfs diff=lfs merge=lfs -text 29 | *.tgz filter=lfs diff=lfs merge=lfs -text 30 | *.wasm filter=lfs diff=lfs merge=lfs -text 31 | *.xz filter=lfs diff=lfs merge=lfs -text 32 | *.zip filter=lfs diff=lfs merge=lfs -text 33 | *.zst filter=lfs diff=lfs merge=lfs -text 34 | *tfevents* filter=lfs diff=lfs merge=lfs -text 35 | -------------------------------------------------------------------------------- /Week 3/Tuesday/chainlit/.gitignore: -------------------------------------------------------------------------------- 1 | .env 2 | .vscode 3 | .chroma 4 | __pycache__ -------------------------------------------------------------------------------- /Week 3/Tuesday/chainlit/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM python:3.9 2 | RUN useradd -m -u 1000 user 3 | USER user 4 | ENV HOME=/home/user \ 5 | PATH=/home/user/.local/bin:$PATH 6 | WORKDIR $HOME/app 7 | COPY --chown=user . $HOME/app 8 | COPY ./requirements.txt ~/app/requirements.txt 9 | RUN pip install -r requirements.txt 10 | COPY . . 11 | CMD ["chainlit", "run", "app.py", "--port", "7860"] -------------------------------------------------------------------------------- /Week 3/Tuesday/chainlit/README.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: ArxivChainLitDemo 3 | emoji: 💻 4 | colorFrom: indigo 5 | colorTo: gray 6 | sdk: docker 7 | pinned: false 8 | license: openrail 9 | duplicated_from: c-s-ale/ArxivChainLitDemo 10 | --- 11 | -------------------------------------------------------------------------------- /Week 3/Tuesday/chainlit/app.py: -------------------------------------------------------------------------------- 1 | from langchain.embeddings.openai import OpenAIEmbeddings 2 | from langchain.document_loaders import PyMuPDFLoader 3 | from langchain.text_splitter import RecursiveCharacterTextSplitter 4 | from langchain.vectorstores import Chroma 5 | from langchain.chains import RetrievalQAWithSourcesChain 6 | from langchain.chat_models import ChatOpenAI 7 | from langchain.prompts.chat import ( 8 | ChatPromptTemplate, 9 | SystemMessagePromptTemplate, 10 | HumanMessagePromptTemplate, 11 | ) 12 | import os 13 | import arxiv 14 | import chainlit as cl 15 | from chainlit import user_session 16 | 17 | @cl.langchain_factory(use_async=True) 18 | async def init(): 19 | arxiv_query = None 20 | 21 | # Wait for the user to ask an Arxiv question 22 | while arxiv_query == None: 23 | arxiv_query = await cl.AskUserMessage( 24 | content="Please enter a topic to begin!", timeout=15 25 | ).send() 26 | 27 | # Obtain the top 3 results from Arxiv for the query 28 | search = arxiv.Search( 29 | query=arxiv_query["content"], 30 | max_results=3, 31 | sort_by=arxiv.SortCriterion.Relevance, 32 | ) 33 | 34 | await cl.Message(content="Downloading and chunking articles...").send() 35 | # download each of the pdfs 36 | pdf_data = [] 37 | for result in search.results(): 38 | loader = PyMuPDFLoader(result.pdf_url) 39 | loaded_pdf = loader.load() 40 | 41 | for document in loaded_pdf: 42 | document.metadata["source"] = result.entry_id 43 | document.metadata["file_path"] = result.pdf_url 44 | document.metadata["title"] = result.title 45 | pdf_data.append(document) 46 | 47 | # Create a Chroma vector store 48 | embeddings = OpenAIEmbeddings( 49 | disallowed_special=(), 50 | ) 51 | 52 | # If operation takes too long, make_async allows to run in a thread 53 | # docsearch = await cl.make_async(Chroma.from_documents)(pdf_data, embeddings) 54 | docsearch = Chroma.from_documents(pdf_data, embeddings) 55 | 56 | # Create a chain that uses the Chroma vector store 57 | chain = RetrievalQAWithSourcesChain.from_chain_type( 58 | ChatOpenAI( 59 | model_name="gpt-3.5-turbo-16k", 60 | temperature=0, 61 | ), 62 | chain_type="stuff", 63 | retriever=docsearch.as_retriever(), 64 | return_source_documents=True, 65 | ) 66 | 67 | # Let the user know that the system is ready 68 | await cl.Message( 69 | content=f"We found a few papers about `{arxiv_query['content']}` you can now ask questions!" 70 | ).send() 71 | 72 | return chain 73 | 74 | 75 | @cl.langchain_postprocess 76 | async def process_response(res): 77 | answer = res["answer"] 78 | source_elements_dict = {} 79 | source_elements = [] 80 | for idx, source in enumerate(res["source_documents"]): 81 | title = source.metadata["title"] 82 | 83 | if title not in source_elements_dict: 84 | source_elements_dict[title] = { 85 | "page_number": [source.metadata["page"]], 86 | "url": source.metadata["file_path"], 87 | } 88 | 89 | else: 90 | source_elements_dict[title]["page_number"].append(source.metadata["page"]) 91 | 92 | # sort the page numbers 93 | source_elements_dict[title]["page_number"].sort() 94 | 95 | for title, source in source_elements_dict.items(): 96 | # create a string for the page numbers 97 | page_numbers = ", ".join([str(x) for x in source["page_number"]]) 98 | text_for_source = f"Page Number(s): {page_numbers}\nURL: {source['url']}" 99 | source_elements.append( 100 | cl.Text(name=title, content=text_for_source, display="inline") 101 | ) 102 | 103 | await cl.Message(content=answer, elements=source_elements).send() -------------------------------------------------------------------------------- /Week 3/Tuesday/chainlit/chainlit.md: -------------------------------------------------------------------------------- 1 | # ⚠️ Warning ⚠️ 2 | 3 | You will need an OpenAI API key to use this app due to large context size! 4 | 5 | # Welcome to AskArxiv powered by Chainlit! 6 | 7 | In this app, you'll be able to enter a topic - and then ask ~3 papers from Arxiv about that topic! 8 | 9 | ### Link To Demo 10 | 11 | [Hugging Face Space](https://huggingface.co/spaces/ml-maker-space/ArxivChainLitDemo) 12 | -------------------------------------------------------------------------------- /Week 3/Tuesday/chainlit/requirements.txt: -------------------------------------------------------------------------------- 1 | arxiv==1.4.7 2 | langchain==0.0.202 3 | chainlit==0.4.1 4 | openai 5 | chromadb 6 | tiktoken 7 | pymupdf -------------------------------------------------------------------------------- /Week 3/Tuesday/fastapi/.env.sample: -------------------------------------------------------------------------------- 1 | HUGGINGFACE_TOKEN=... 2 | -------------------------------------------------------------------------------- /Week 3/Tuesday/fastapi/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM nvidia/cuda:12.2.0-base-ubuntu20.04 2 | 3 | # get python 3.9 4 | RUN apt-get update && apt-get install -y software-properties-common 5 | RUN add-apt-repository ppa:deadsnakes/ppa 6 | RUN apt-get update && apt-get install -y python3.9 python3.9-dev python3.9-distutils python3-pip 7 | 8 | WORKDIR /app 9 | ADD . /app 10 | 11 | RUN pip install --no-cache-dir -r requirements.txt 12 | EXPOSE 80 13 | 14 | CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "80"] -------------------------------------------------------------------------------- /Week 3/Tuesday/fastapi/README.md: -------------------------------------------------------------------------------- 1 | # Serving a Scalable Fast API application leveraging Celery and Redis 2 | 3 | ### Pre-requisites: 4 | 5 | - You must deploy it on a GPU with ~16GB of memory 6 | - You will need `docker compose` (HINT: Do not use `docker-compose`) 7 | - [Documentation](https://docs.docker.com/compose/install/linux/#install-the-plugin-manually) 8 | - You will need to ensure `nvidia-ctk --version` provides a valid output 9 | - [Documentation](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) 10 | 11 | ### Tasks: 12 | 13 | 1. Provide a simple system diagram (created in whatever format you feel best communicates the flow) for the application 14 | 15 | This is meant to be a challenging task - so you might need to spend some time troubleshooting and tinkering! 16 | -------------------------------------------------------------------------------- /Week 3/Tuesday/fastapi/app.py: -------------------------------------------------------------------------------- 1 | from fastapi import FastAPI 2 | from pydantic import BaseModel, Field 3 | from celery.result import AsyncResult 4 | from typing import Any 5 | from celery_worker import generate_text_task 6 | from dotenv import load_dotenv 7 | 8 | load_dotenv() 9 | 10 | app = FastAPI() 11 | 12 | 13 | class Prompt(BaseModel): 14 | prompt: str 15 | 16 | 17 | @app.post("/generateText") 18 | async def generate_text(prompt: Prompt) -> Any: 19 | task = generate_text_task.delay(prompt.prompt) 20 | return {"task_id": task.id} 21 | 22 | 23 | @app.get("/task/{task_id}") 24 | async def get_generate_text(task_id: str): 25 | task = AsyncResult(task_id) 26 | if task.ready(): 27 | task_result = task.get() 28 | return { 29 | "result": task_result[0], 30 | "time": task_result[1], 31 | "memory": task_result[2], 32 | } 33 | else: 34 | return {"status": "Task Pending"} 35 | -------------------------------------------------------------------------------- /Week 3/Tuesday/fastapi/celery_worker.py: -------------------------------------------------------------------------------- 1 | from celery import Celery, signals 2 | from utils import generate_output 3 | from model_loader import ModelLoader 4 | 5 | 6 | def make_celery(app_name=__name__): 7 | backend = broker = "redis://redis:6379/0" 8 | return Celery(app_name, backend=backend, broker=broker) 9 | 10 | 11 | celery = make_celery() 12 | 13 | model_loader = None 14 | model_path = "meta-llama/Llama-2-7b-chat-hf" 15 | 16 | 17 | @signals.worker_process_init.connect 18 | def setup_model(signal, sender, **kwargs): 19 | global model_loader 20 | model_loader = ModelLoader(model_path) 21 | 22 | 23 | @celery.task 24 | def generate_text_task(prompt): 25 | time, memory, outputs = generate_output( 26 | prompt, model_loader.model, model_loader.tokenizer 27 | ) 28 | return model_loader.tokenizer.decode(outputs[0]), time, memory 29 | -------------------------------------------------------------------------------- /Week 3/Tuesday/fastapi/docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: '3' 2 | services: 3 | web: 4 | build: . 5 | command: ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "80"] 6 | volumes: 7 | - .:/app 8 | ports: 9 | - 8000:80 10 | depends_on: 11 | - redis 12 | deploy: 13 | resources: 14 | reservations: 15 | devices: 16 | - driver: nvidia 17 | count: 1 18 | capabilities: [gpu] 19 | worker: 20 | build: . 21 | command: celery -A celery_worker worker -P solo --loglevel=info 22 | volumes: 23 | - .:/app 24 | depends_on: 25 | - redis 26 | deploy: 27 | resources: 28 | reservations: 29 | devices: 30 | - driver: nvidia 31 | count: 1 32 | capabilities: [gpu] 33 | redis: 34 | image: "redis:alpine" 35 | ports: 36 | - 6379:6379 37 | -------------------------------------------------------------------------------- /Week 3/Tuesday/fastapi/model_loader.py: -------------------------------------------------------------------------------- 1 | import os 2 | from transformers import ( 3 | AutoModelForCausalLM, 4 | AutoConfig, 5 | AutoTokenizer, 6 | BitsAndBytesConfig, 7 | ) 8 | import torch 9 | from dotenv import load_dotenv 10 | 11 | load_dotenv() 12 | 13 | 14 | class ModelLoader: 15 | def __init__(self, model_path: str): 16 | self.model_path = model_path 17 | self.config = AutoConfig.from_pretrained( 18 | self.model_path, 19 | trust_remote_code=True, 20 | use_auth_token=os.getenv("HUGGINGFACE_TOKEN"), 21 | ) 22 | self.model = self._load_model() 23 | self.tokenizer = AutoTokenizer.from_pretrained( 24 | self.model_path, use_auth_token=os.getenv("HUGGINGFACE_TOKEN") 25 | ) 26 | 27 | def _load_model(self): 28 | nf4_config = BitsAndBytesConfig( 29 | load_in_4bit=True, 30 | bnb_4bit_quant_type="nf4", 31 | bnb_4bit_use_double_quant=True, 32 | bnb_4bit_compute_dtype=torch.bfloat16, 33 | ) 34 | model = AutoModelForCausalLM.from_pretrained( 35 | self.model_path, 36 | quantization_config=nf4_config, 37 | trust_remote_code=True, 38 | device_map="auto", 39 | use_auth_token=os.getenv("HUGGINGFACE_TOKEN"), 40 | ) 41 | return model 42 | -------------------------------------------------------------------------------- /Week 3/Tuesday/fastapi/redis_server.py: -------------------------------------------------------------------------------- 1 | import subprocess 2 | import redis_server 3 | 4 | 5 | def install_redis_server(redis_version): 6 | try: 7 | subprocess.check_call(["pip", "install", f"redis-server=={redis_version}"]) 8 | print(f"Redis server version {redis_version} installed successfully.") 9 | except subprocess.CalledProcessError: 10 | print("Failed to install Redis server.") 11 | exit(1) 12 | 13 | 14 | def start_redis_server(): 15 | try: 16 | redis_server_path = redis_server.REDIS_SERVER_PATH 17 | subprocess.Popen([redis_server_path]) 18 | print("Redis server started successfully.") 19 | except Exception as e: 20 | print("Failed to start Redis server:", str(e)) 21 | exit(1) 22 | 23 | 24 | def main(): 25 | redis_version = "6.0.9" 26 | install_redis_server(redis_version) 27 | start_redis_server() 28 | 29 | 30 | if __name__ == "__main__": 31 | main() 32 | -------------------------------------------------------------------------------- /Week 3/Tuesday/fastapi/requirements.txt: -------------------------------------------------------------------------------- 1 | fastapi==0.99.1 2 | uvicorn==0.22.0 3 | pydantic==1.10.10 4 | celery==5.3.1 5 | redis==4.6.0 6 | python-dotenv==1.0.0 7 | transformers==4.30.2 8 | torch==2.0.1 9 | accelerate==0.21.0 10 | bitsandbytes==0.41.0 11 | scipy==1.10.1 12 | -------------------------------------------------------------------------------- /Week 3/Tuesday/fastapi/run.py: -------------------------------------------------------------------------------- 1 | import http.client 2 | import json 3 | import time 4 | 5 | API_HOST = "localhost" 6 | API_PORT = 8000 7 | 8 | 9 | def generate_text(prompt): 10 | conn = http.client.HTTPConnection(API_HOST, API_PORT) 11 | headers = {"Content-type": "application/json"} 12 | data = {"prompt": prompt} 13 | json_data = json.dumps(data) 14 | conn.request("POST", "/generateText/", json_data, headers) 15 | response = conn.getresponse() 16 | result = json.loads(response.read().decode()) 17 | conn.close() 18 | return result["task_id"] 19 | 20 | 21 | def get_task_status(task_id): 22 | conn = http.client.HTTPConnection(API_HOST, API_PORT) 23 | conn.request("GET", f"/generateTextTask/{task_id}") 24 | response = conn.getresponse() 25 | status = response.read().decode() 26 | conn.close() 27 | return status 28 | 29 | 30 | def main(): 31 | prompt = input("Enter the prompt: ") 32 | 33 | task_id = generate_text(prompt) 34 | while True: 35 | status = get_task_status(task_id) 36 | if "Task Pending" not in status: 37 | print(status) 38 | break 39 | time.sleep(2) 40 | 41 | 42 | if __name__ == "__main__": 43 | main() 44 | -------------------------------------------------------------------------------- /Week 3/Tuesday/fastapi/utils.py: -------------------------------------------------------------------------------- 1 | import time 2 | import torch 3 | import functools 4 | from transformers import AutoModelForCausalLM, AutoTokenizer 5 | 6 | 7 | def time_decorator(func): 8 | @functools.wraps(func) 9 | def wrapper(*args, **kwargs): 10 | start_time = time.time() 11 | result = func(*args, **kwargs) 12 | end_time = time.time() 13 | exec_time = end_time - start_time 14 | return (result, exec_time) 15 | 16 | return wrapper 17 | 18 | 19 | def memory_decorator(func): 20 | @functools.wraps(func) 21 | def wrapper(*args, **kwargs): 22 | torch.cuda.empty_cache() 23 | torch.cuda.reset_peak_memory_stats() 24 | result, exec_time = func(*args, **kwargs) 25 | peak_mem = torch.cuda.max_memory_allocated() 26 | peak_mem_consumption = peak_mem / 1e9 27 | return peak_mem_consumption, exec_time, result 28 | 29 | return wrapper 30 | 31 | 32 | @memory_decorator 33 | @time_decorator 34 | def generate_output( 35 | prompt: str, model: AutoModelForCausalLM, tokenizer: AutoTokenizer 36 | ) -> torch.Tensor: 37 | input_ids = tokenizer(prompt, return_tensors="pt").input_ids 38 | input_ids = input_ids.to("cuda") 39 | outputs = model.generate(input_ids, max_length=500) 40 | return outputs 41 | -------------------------------------------------------------------------------- /Week 4/Tuesday/README.md: -------------------------------------------------------------------------------- 1 | # 🔎 Visibility Tools in LLM Ops 2 | 3 | # 📚 Background 4 | 5 | ### Langsmith 6 | 7 | LangChain makes it easy to prototype LLM applications and Agents. However, delivering LLM applications to production can be deceptively difficult. You will likely have to heavily customize and iterate on your prompts, chains, and other components to create a high-quality product. LangSmith is a unified platform for debugging, testing, and monitoring your LLM applications. 8 | 9 | Langsmith can be useful when you want to: 10 | - Quickly debug a new chain, agent, or set of tools 11 | - Visualize how components (chains, llms, retrievers, etc.) relate and are used 12 | - Evaluate different prompts and LLMs for a single component 13 | - Run a given chain several times over a dataset to ensure it consistently meets a quality bar 14 | - Capture usage traces and using LLMs or analytics pipelines to generate insights 15 | 16 | ### Weight and Biases Prompts 17 | W&B Prompts is a suite of LLMOps tools built for the development of LLM-powered applications. Use W&B Prompts to visualize and inspect the execution flow of your LLMs, analyze the inputs and outputs of your LLMs, view the intermediate results and securely store and manage your prompts and LLM chain configurations. 18 | 19 | **Trace Table** - provides an overview of the inputs and outputs of a chain, the composition of a trace event in the chain, whether or not the chain ran successfully, and any error messages returned when running the chain. 20 | 21 | **The Trace Timeline** - displays the execution flow of the chain and is color-coded according to component types. 22 | 23 | **Model Architecture** - provides details about the structure of the chain and the parameters used to initialize each component of the chain. 24 | 25 | 26 | 27 | # Working session 28 | 29 | ## 🏗️ Build 30 | 31 | Your assignment has one task: 32 | - Extend any previous assignment (or new application) built with LangChain (or Llama Index) to include Weights and Biases tracing 33 | 34 | ## 🚢 Ship 35 | 36 | Host your application on a Hugging Face Space! 37 | 38 | ## 🚀 Share 39 | 40 | Share screenshots of your visibility tools outputs and explain (in your own words) what the use cases of visibility tooling are! 41 | 42 | 43 | We'll be looking at two different visibility tools during this session: 44 | 45 | 1. [LangSmith](https://python.langchain.com/docs/guides/langsmith/walkthrough) 46 | 2. [Weights and Biases](https://docs.wandb.ai/guides/prompts) 47 | 48 | Currently, LangSmith is in closed beta - you can request access [here](https://smith.langchain.com/) - we'll demo how to use it and what it can do, but we'll focus on Weights and Biases for the assignment. 49 | 50 | 51 | ### Environment Notices 52 | 53 | We'll be using Python 3.11 as the base environment today. 54 | 55 | You'll need to `pip install`: 56 | 57 | - `wandb` 58 | - `langchain` 59 | - `langsmith` (if you have closed beta access) 60 | - `openai` 61 | - `jupyter` 62 | 63 | ### Weights and Biases API Key 64 | 65 | You will need to have a Weights and Biases API key, which you can obtain following these instructions: 66 | 67 | 1. Create a Weights and Biases account. 68 | 2. Navigate to your `User Settings` 69 | 70 | ![image](https://i.imgur.com/oRRj40x.png) 71 | 72 | 3. Take the highway to (navigate to) the `Danger Zone` 73 | 74 | ![image](https://i.imgur.com/XBrOyfG.png) 75 | 76 | 4. Copy your new API key! 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | -------------------------------------------------------------------------------- /Week 4/Tuesday/langsmith_notebook.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# LangSmith\n", 8 | "\n", 9 | "We'll be moving through this notebook to explain what visibility tools can do to help us!" 10 | ] 11 | }, 12 | { 13 | "cell_type": "code", 14 | "execution_count": 2, 15 | "metadata": {}, 16 | "outputs": [], 17 | "source": [ 18 | "!pip install -U -q langchain openai langsmith" 19 | ] 20 | }, 21 | { 22 | "cell_type": "markdown", 23 | "metadata": {}, 24 | "source": [ 25 | "## Basic Application\n", 26 | "\n", 27 | "We'll be leveraging the same application as we did in the Weights and Biases Notebook to showcase what can be done with LangSmith:\n", 28 | "\n", 29 | "A simple Arxiv Agent!\n", 30 | "\n", 31 | "Let's start with grabbing a few pieces of information and our additional dependencies:" 32 | ] 33 | }, 34 | { 35 | "cell_type": "code", 36 | "execution_count": 4, 37 | "metadata": {}, 38 | "outputs": [], 39 | "source": [ 40 | "import os\n", 41 | "import getpass\n", 42 | "\n", 43 | "os.environ['OPENAI_API_KEY'] = getpass.getpass('Enter your OpenAI API key: ')" 44 | ] 45 | }, 46 | { 47 | "cell_type": "code", 48 | "execution_count": 5, 49 | "metadata": {}, 50 | "outputs": [], 51 | "source": [ 52 | "!pip install -U -q arxiv" 53 | ] 54 | }, 55 | { 56 | "cell_type": "markdown", 57 | "metadata": {}, 58 | "source": [ 59 | "Now we can set up our LangChain environment variables:" 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": 7, 65 | "metadata": {}, 66 | "outputs": [], 67 | "source": [ 68 | "from uuid import uuid4\n", 69 | "\n", 70 | "unique_id = uuid4().hex[0:8]\n", 71 | "\n", 72 | "os.environ[\"LANGCHAIN_TRACING_V2\"] = \"true\"\n", 73 | "os.environ[\"LANGCHAIN_PROJECT\"] = f\"LLM Ops Walk Through - {unique_id}\"\n", 74 | "os.environ[\"LANGCHAIN_ENDPOINT\"] = \"https://api.smith.langchain.com\"\n", 75 | "os.environ[\"LANGCHAIN_API_KEY\"] = getpass.getpass('Enter your LangSmith API key: ')" 76 | ] 77 | }, 78 | { 79 | "cell_type": "code", 80 | "execution_count": 8, 81 | "metadata": {}, 82 | "outputs": [], 83 | "source": [ 84 | "from langsmith import Client\n", 85 | "\n", 86 | "client = Client()" 87 | ] 88 | }, 89 | { 90 | "cell_type": "markdown", 91 | "metadata": {}, 92 | "source": [ 93 | "Now we can set up our simple application!" 94 | ] 95 | }, 96 | { 97 | "cell_type": "code", 98 | "execution_count": 17, 99 | "metadata": {}, 100 | "outputs": [], 101 | "source": [ 102 | "from langchain.chat_models import ChatOpenAI\n", 103 | "from langchain.agents import load_tools, initialize_agent, AgentType\n", 104 | "\n", 105 | "llm = ChatOpenAI(model_name=\"gpt-4\", temperature=0)\n", 106 | "\n", 107 | "tools = load_tools(\n", 108 | " [\"arxiv\"]\n", 109 | ")\n", 110 | "\n", 111 | "agent_chain = initialize_agent(\n", 112 | " tools,\n", 113 | " llm,\n", 114 | " agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,\n", 115 | ")" 116 | ] 117 | }, 118 | { 119 | "cell_type": "code", 120 | "execution_count": 16, 121 | "metadata": {}, 122 | "outputs": [ 123 | { 124 | "name": "stdout", 125 | "output_type": "stream", 126 | "text": [ 127 | "\n", 128 | "\n", 129 | "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n", 130 | "\u001b[32;1m\u001b[1;3mQLoRA might be a term or concept from a scientific field. I should search for it on arxiv to find relevant articles.\n", 131 | "Action: arxiv\n", 132 | "Action Input: QLoRA\u001b[0m\n", 133 | "Observation: \u001b[36;1m\u001b[1;3mPublished: 2023-05-23\n", 134 | "Title: QLoRA: Efficient Finetuning of Quantized LLMs\n", 135 | "Authors: Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer\n", 136 | "Summary: We present QLoRA, an efficient finetuning approach that reduces memory usage\n", 137 | "enough to finetune a 65B parameter model on a single 48GB GPU while preserving\n", 138 | "full 16-bit finetuning task performance. QLoRA backpropagates gradients through\n", 139 | "a frozen, 4-bit quantized pretrained language model into Low Rank\n", 140 | "Adapters~(LoRA). Our best model family, which we name Guanaco, outperforms all\n", 141 | "previous openly released models on the Vicuna benchmark, reaching 99.3% of the\n", 142 | "performance level of ChatGPT while only requiring 24 hours of finetuning on a\n", 143 | "single GPU. QLoRA introduces a number of innovations to save memory without\n", 144 | "sacrificing performance: (a) 4-bit NormalFloat (NF4), a new data type that is\n", 145 | "information theoretically optimal for normally distributed weights (b) double\n", 146 | "quantization to reduce the average memory footprint by quantizing the\n", 147 | "quantization constants, and (c) paged optimziers to manage memory spikes. We\n", 148 | "use QLoRA to finetune more than 1,000 models, providing a detailed analysis of\n", 149 | "instruction following and chatbot performance across 8 instruction datasets,\n", 150 | "multiple model types (LLaMA, T5), and model scales that would be infeasible to\n", 151 | "run with regular finetuning (e.g. 33B and 65B parameter models). Our results\n", 152 | "show that QLoRA finetuning on a small high-quality dataset leads to\n", 153 | "state-of-the-art results, even when using smaller models than the previous\n", 154 | "SoTA. We provide a detailed analysis of chatbot performance based on both human\n", 155 | "and GPT-4 evaluations showing that GPT-4 evaluations are a cheap and reasonable\n", 156 | "alternative to human evaluation. Furthermore, we find that current chatbot\n", 157 | "benchmarks are not trustworthy to accurately evaluate the performance levels of\n", 158 | "chatbots. A lemon-picked analysis demonstrates where Guanaco fails compared to\n", 159 | "ChatGPT. We release all of our models and code, including CUDA kernels for\n", 160 | "4-bit training.\n", 161 | "\n", 162 | "Published: 2023-07-20\n", 163 | "Title: IvyGPT: InteractiVe Chinese pathwaY language model in medical domain\n", 164 | "Authors: Rongsheng Wang, Yaofei Duan, ChanTong Lam, Jiexi Chen, Jiangsheng Xu, Haoming Chen, Xiaohong Liu, Patrick Cheong-Iao Pang, Tao Tan\n", 165 | "Summary: General large language models (LLMs) such as ChatGPT have shown remarkable\n", 166 | "success. However, such LLMs have not been widely adopted for medical purposes,\n", 167 | "due to poor accuracy and inability to provide medical advice. We propose\n", 168 | "IvyGPT, an LLM based on LLaMA that is trained and fine-tuned with high-quality\n", 169 | "medical question-answer (QA) instances and Reinforcement Learning from Human\n", 170 | "Feedback (RLHF). After supervised fine-tuning, IvyGPT has good multi-turn\n", 171 | "conversation capabilities, but it cannot perform like a doctor in other\n", 172 | "aspects, such as comprehensive diagnosis. Through RLHF, IvyGPT can output\n", 173 | "richer diagnosis and treatment answers that are closer to human. In the\n", 174 | "training, we used QLoRA to train 33 billion parameters on a small number of\n", 175 | "NVIDIA A100 (80GB) GPUs. Experimental results show that IvyGPT has outperformed\n", 176 | "other medical GPT models.\n", 177 | "\n", 178 | "Published: 2023-08-20\n", 179 | "Title: LMTuner: An user-friendly and highly-integrable Training Framework for fine-tuning Large Language Models\n", 180 | "Authors: Yixuan Weng, Zhiqi Wang, Huanxuan Liao, Shizhu He, Shengping Liu, Kang Liu, Jun Zhao\n", 181 | "Summary: With the burgeoning development in the realm of large language models (LLMs),\n", 182 | "the demand for efficient incremental training tailored to specific industries\n", 183 | "and domains continues to increase. Currently, the predominantly employed\n", 184 | "frameworks lack modular design, it often takes a lot of coding work to\n", 185 | "kickstart the training of LLM. To address this, we present \"LMTuner\", a highly\n", 186 | "usable, integrable, and scalable system for training LLMs expeditiously and\n", 187 | "with minimal user-input. LMTuner comprises three main modules - the\n", 188 | "Interaction, Training, and Inference Modules. We advocate that LMTuner's\n", 189 | "usability and integrality alleviate the complexi\u001b[0m\n", 190 | "Thought:\u001b[32;1m\u001b[1;3mQLoRA is an efficient finetuning approach that reduces memory usage enough to finetune a large parameter model on a single GPU while preserving full 16-bit finetuning task performance. It backpropagates gradients through a frozen, 4-bit quantized pretrained language model into Low Rank Adapters (LoRA). QLoRA introduces a number of innovations to save memory without sacrificing performance, including a new data type that is information theoretically optimal for normally distributed weights, double quantization to reduce the average memory footprint by quantizing the quantization constants, and paged optimizers to manage memory spikes. It has been used to finetune more than 1,000 models, providing a detailed analysis of instruction following and chatbot performance across multiple model types and scales.\n", 191 | "Final Answer: QLoRA is an efficient finetuning approach for large language models that reduces memory usage while preserving performance. It introduces several innovations including a new data type for normally distributed weights, double quantization, and paged optimizers. It has been used to finetune a wide range of models, providing detailed analysis of their performance.\u001b[0m\n", 192 | "\n", 193 | "\u001b[1m> Finished chain.\u001b[0m\n" 194 | ] 195 | }, 196 | { 197 | "data": { 198 | "text/plain": [ 199 | "{'input': 'What is QLoRA?',\n", 200 | " 'output': 'QLoRA is an efficient finetuning approach for large language models that reduces memory usage while preserving performance. It introduces several innovations including a new data type for normally distributed weights, double quantization, and paged optimizers. It has been used to finetune a wide range of models, providing detailed analysis of their performance.'}" 201 | ] 202 | }, 203 | "execution_count": 16, 204 | "metadata": {}, 205 | "output_type": "execute_result" 206 | } 207 | ], 208 | "source": [ 209 | "agent_chain(\"What is QLoRA?\")" 210 | ] 211 | }, 212 | { 213 | "cell_type": "markdown", 214 | "metadata": {}, 215 | "source": [ 216 | "Let's build a number of input/output pairs that we can leverage later!" 217 | ] 218 | }, 219 | { 220 | "cell_type": "code", 221 | "execution_count": 18, 222 | "metadata": {}, 223 | "outputs": [], 224 | "source": [ 225 | "import asyncio\n", 226 | "\n", 227 | "inputs = [\n", 228 | " \"What optimizer is used in QLoRA?\",\n", 229 | " \"What data type was created in the QLoRA paper?\",\n", 230 | " \"What is a Retrieval Augmented Generation system?\",\n", 231 | " \"Who authored the QLoRA paper?\",\n", 232 | " \"What is the most popular deep learning framework?\",\n", 233 | " \"What significant improvements does the LoRA system make?\"\n", 234 | "]\n", 235 | "\n", 236 | "results = []\n", 237 | "\n", 238 | "async def arun(agent, input_example):\n", 239 | " try:\n", 240 | " return await agent.arun(input_example)\n", 241 | " except Exception as e:\n", 242 | " return e\n", 243 | "\n", 244 | "for input_example in inputs:\n", 245 | " results.append(arun(agent_chain, input_example))\n", 246 | "\n", 247 | "results = await asyncio.gather(*results)" 248 | ] 249 | }, 250 | { 251 | "cell_type": "markdown", 252 | "metadata": {}, 253 | "source": [ 254 | "Now that we've run through all of those chains - we can leverage LangSmith to create a dataset that we can use to benchmark other application solutions!" 255 | ] 256 | }, 257 | { 258 | "cell_type": "code", 259 | "execution_count": 19, 260 | "metadata": {}, 261 | "outputs": [], 262 | "source": [ 263 | "from langchain.callbacks.tracers.langchain import wait_for_all_tracers\n", 264 | "\n", 265 | "wait_for_all_tracers()" 266 | ] 267 | }, 268 | { 269 | "cell_type": "markdown", 270 | "metadata": {}, 271 | "source": [ 272 | "### Evaluating with LangSmith" 273 | ] 274 | }, 275 | { 276 | "cell_type": "markdown", 277 | "metadata": {}, 278 | "source": [ 279 | "The first thing we'll need to do is collect our responses into a dataset that we can use to benchmark other solutions against!" 280 | ] 281 | }, 282 | { 283 | "cell_type": "code", 284 | "execution_count": 20, 285 | "metadata": {}, 286 | "outputs": [], 287 | "source": [ 288 | "dataset_name = f\"arxiv-rag-gpt-4-{unique_id}\"\n", 289 | "\n", 290 | "dataset = client.create_dataset(\n", 291 | " dataset_name, description=\"A dataset for benchmarking a RAG system using the Arxiv tool\"\n", 292 | ")\n", 293 | "\n", 294 | "runs = client.list_runs(\n", 295 | " project_name=os.environ[\"LANGCHAIN_PROJECT\"],\n", 296 | " execution_order=1, # Only return the top-level runs\n", 297 | " error=False, # Only runs that succeed\n", 298 | ")\n", 299 | "for run in runs:\n", 300 | " client.create_example(inputs=run.inputs, outputs=run.outputs, dataset_id=dataset.id)" 301 | ] 302 | }, 303 | { 304 | "cell_type": "markdown", 305 | "metadata": {}, 306 | "source": [ 307 | "Now that we have our dataset set up in LangSmith - let's create another system that we can benchmark against our original!\n", 308 | "\n", 309 | "Since it's possible to build an agent that has memory (which could influence results and might not provide accurate benchmarking) - we'll use an `agent_factory` to create our agent for each test-case." 310 | ] 311 | }, 312 | { 313 | "cell_type": "code", 314 | "execution_count": 22, 315 | "metadata": {}, 316 | "outputs": [], 317 | "source": [ 318 | "from langchain.chat_models import ChatOpenAI\n", 319 | "from langchain.agents import load_tools, initialize_agent, AgentType\n", 320 | "\n", 321 | "llm = ChatOpenAI(model_name=\"gpt-3.5-turbo\", temperature=0)\n", 322 | "\n", 323 | "tools = load_tools(\n", 324 | " [\"arxiv\"]\n", 325 | ")\n", 326 | "\n", 327 | "def agent_factory():\n", 328 | " agent_chain = initialize_agent(\n", 329 | " tools,\n", 330 | " llm,\n", 331 | " agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,\n", 332 | " verbose=True,\n", 333 | " handle_parsing_errors=True,\n", 334 | " )\n", 335 | " return agent_chain" 336 | ] 337 | }, 338 | { 339 | "cell_type": "markdown", 340 | "metadata": {}, 341 | "source": [ 342 | "Now we can use the `langchain.evaluation.EvaluatorType` and `langchain.smith.RunEvalConfig` methods to build a pipeline for our evaluation.\n", 343 | "\n", 344 | "More information about these metrics is found [here](https://docs.smith.langchain.com/evaluation/evaluator-implementations)\n", 345 | "Let's set it up with the following evluators:\n", 346 | "\n", 347 | "- `EvaluatorType.QA` - measures how \"correct\" your response is, based on a reference answer (we built these in the first part of the notebook)\n", 348 | "- `EvaluatorType.EMBEDDING_DISTANCE` - measure closeness between the two responses\n", 349 | "- `RunEvalConfig.LabeledCriteria` - measures the output against the given criteria\n", 350 | "- `RunEvalConfig.Criteria({\"YOUR CUSTOM CRITERAI\", \"DESCRIPTION OF YOUR CRITERIA IN NATURAL LANGUAGE\"})`\n", 351 | "\n" 352 | ] 353 | }, 354 | { 355 | "cell_type": "markdown", 356 | "metadata": {}, 357 | "source": [ 358 | "We'll also build our own custom evaluator as a demonstration of how to implement such an evaluator!" 359 | ] 360 | }, 361 | { 362 | "cell_type": "code", 363 | "execution_count": 26, 364 | "metadata": {}, 365 | "outputs": [], 366 | "source": [ 367 | "!pip install -U -q tiktoken" 368 | ] 369 | }, 370 | { 371 | "cell_type": "markdown", 372 | "metadata": {}, 373 | "source": [ 374 | "In our own custom evaluator we need to make sure of a couple things:\n", 375 | "\n", 376 | "1. We provide a system by which we can measure or provide a measure of closeness/some numeric metric.\n", 377 | "2. We provide logic for implementing our score and parsing the relevant outputs." 378 | ] 379 | }, 380 | { 381 | "cell_type": "code", 382 | "execution_count": 29, 383 | "metadata": {}, 384 | "outputs": [], 385 | "source": [ 386 | "import re\n", 387 | "from typing import Any, Optional\n", 388 | "\n", 389 | "from langchain.chains import LLMChain\n", 390 | "from langchain.chat_models import ChatOpenAI\n", 391 | "from langchain.evaluation import StringEvaluator\n", 392 | "\n", 393 | "\n", 394 | "class DopenessEvaluator(StringEvaluator):\n", 395 | " \"\"\"An LLM-based dopeness evaluator.\"\"\"\n", 396 | "\n", 397 | " def __init__(self):\n", 398 | " llm = ChatOpenAI(model=\"gpt-4\", temperature=0)\n", 399 | "\n", 400 | " template = \"\"\"On a scale from 0 to 100, how dope is the following response to the input:\n", 401 | " --------\n", 402 | " INPUT: {input}\n", 403 | " --------\n", 404 | " OUTPUT: {prediction}\n", 405 | " --------\n", 406 | " Reason step by step about why the score is appropriate, then print the score at the end. At the end, repeat that score alone on a new line.\"\"\"\n", 407 | "\n", 408 | " self.eval_chain = LLMChain.from_string(llm=llm, template=template)\n", 409 | "\n", 410 | " @property\n", 411 | " def requires_input(self) -> bool:\n", 412 | " return True\n", 413 | "\n", 414 | " @property\n", 415 | " def requires_reference(self) -> bool:\n", 416 | " return False\n", 417 | "\n", 418 | " @property\n", 419 | " def evaluation_name(self) -> str:\n", 420 | " return \"dopeness_score\"\n", 421 | "\n", 422 | " def _evaluate_strings(\n", 423 | " self,\n", 424 | " prediction: str,\n", 425 | " input: Optional[str] = None,\n", 426 | " reference: Optional[str] = None,\n", 427 | " **kwargs: Any\n", 428 | " ) -> dict:\n", 429 | " evaluator_result = self.eval_chain(\n", 430 | " dict(input=input, prediction=prediction), **kwargs\n", 431 | " )\n", 432 | " reasoning, score = evaluator_result[\"text\"].split(\"\\n\", maxsplit=1)\n", 433 | " score = re.search(r\"\\d+\", score).group(0)\n", 434 | " if score is not None:\n", 435 | " score = float(score.strip()) / 100.0\n", 436 | " return {\"score\": score, \"dopeness\": reasoning.strip()}" 437 | ] 438 | }, 439 | { 440 | "cell_type": "markdown", 441 | "metadata": {}, 442 | "source": [ 443 | "Now we can set our `RunEvalFeedback` up!\n", 444 | "\n", 445 | "Notice how we can create custom evaluations that are string based only - " 446 | ] 447 | }, 448 | { 449 | "cell_type": "code", 450 | "execution_count": 30, 451 | "metadata": {}, 452 | "outputs": [], 453 | "source": [ 454 | "from langchain.evaluation import EvaluatorType\n", 455 | "from langchain.smith import RunEvalConfig\n", 456 | "\n", 457 | "evaluation_config = RunEvalConfig(\n", 458 | " evaluators = [\n", 459 | " EvaluatorType.QA,\n", 460 | " EvaluatorType.EMBEDDING_DISTANCE,\n", 461 | " RunEvalConfig.LabeledCriteria(\"relevance\"),\n", 462 | " RunEvalConfig.Criteria({\n", 463 | " \"fully_answered\" : \"Does this response fully answer the question?\"\n", 464 | " })\n", 465 | " ],\n", 466 | " custom_evaluators = [\n", 467 | " DopenessEvaluator()\n", 468 | " ]\n", 469 | ")" 470 | ] 471 | }, 472 | { 473 | "cell_type": "code", 474 | "execution_count": null, 475 | "metadata": {}, 476 | "outputs": [], 477 | "source": [ 478 | "from langchain.smith import (\n", 479 | " arun_on_dataset,\n", 480 | ")\n", 481 | "\n", 482 | "tag_name = f\"<< YOUR NAME HERE>>\"\n", 483 | "tag = \"GPT-4 vs. GPT-3.5-turbo on Arxiv RAG \" + tag_name\n", 484 | "\n", 485 | "chain_results = await arun_on_dataset(\n", 486 | " client=client,\n", 487 | " dataset_name=dataset_name,\n", 488 | " llm_or_chain_factory=agent_factory,\n", 489 | " evaluation=evaluation_config,\n", 490 | " verbose=True,\n", 491 | " tags=[tag],\n", 492 | ")" 493 | ] 494 | } 495 | ], 496 | "metadata": { 497 | "kernelspec": { 498 | "display_name": "aims-visibility", 499 | "language": "python", 500 | "name": "python3" 501 | }, 502 | "language_info": { 503 | "codemirror_mode": { 504 | "name": "ipython", 505 | "version": 3 506 | }, 507 | "file_extension": ".py", 508 | "mimetype": "text/x-python", 509 | "name": "python", 510 | "nbconvert_exporter": "python", 511 | "pygments_lexer": "ipython3", 512 | "version": "3.11.4" 513 | }, 514 | "orig_nbformat": 4 515 | }, 516 | "nbformat": 4, 517 | "nbformat_minor": 2 518 | } 519 | -------------------------------------------------------------------------------- /Week 4/Tuesday/wandb_notebook_assignment.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# LLM Ops Visibility and Caching Strategies\n", 8 | "\n", 9 | "# 🏗️ Build\n", 10 | "\n", 11 | "You will build an application that leverages a visibility tool (Weights and Biases Promopts) and prompt caching.\n", 12 | "\n", 13 | "# 🚢 Ship\n", 14 | "\n", 15 | "You will ship that application to a Hugging Face space.\n", 16 | "\n", 17 | "# 🚀 Share\n", 18 | "\n", 19 | "Create a social media post explaning or showcasing the power of prompt-caching, and visibility tooling in your LLM Ops stack." 20 | ] 21 | }, 22 | { 23 | "cell_type": "markdown", 24 | "metadata": {}, 25 | "source": [ 26 | "## Visibility Tools\n", 27 | "\n", 28 | "A key part of LLM Ops is having a visibility platform where you can track, trace, and collect, various prompt and user data. \n", 29 | "\n", 30 | "Let's take a look at it in this notebook!\n", 31 | "\n", 32 | "As always, we'll want to start with our dependencies. " 33 | ] 34 | }, 35 | { 36 | "cell_type": "code", 37 | "execution_count": 14, 38 | "metadata": {}, 39 | "outputs": [], 40 | "source": [ 41 | "!pip install -q -U \"wandb>=0.15.4\" \"langchain>=0.0.218\"" 42 | ] 43 | }, 44 | { 45 | "cell_type": "markdown", 46 | "metadata": {}, 47 | "source": [ 48 | "Getting started with Weights and Biases Prompts can be as easy as setting the `LANGCHAIN_WANDB_TRACING` environment variable to `true`!" 49 | ] 50 | }, 51 | { 52 | "cell_type": "code", 53 | "execution_count": 43, 54 | "metadata": {}, 55 | "outputs": [], 56 | "source": [ 57 | "import os\n", 58 | "\n", 59 | "os.environ[\"LANGCHAIN_WANDB_TRACING\"] = \"true\"\n", 60 | "os.environ[\"WANDB_PROJECT\"] = \"langchain-testing\"\n", 61 | "os.environ[\"WANDB_NOTEBOOK_NAME\"] = \"./wandb_notebook.ipynb\"" 62 | ] 63 | }, 64 | { 65 | "cell_type": "code", 66 | "execution_count": 3, 67 | "metadata": {}, 68 | "outputs": [], 69 | "source": [ 70 | "import getpass\n", 71 | "\n", 72 | "os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"Enter your OpenAI API key: \")" 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": 4, 78 | "metadata": {}, 79 | "outputs": [], 80 | "source": [ 81 | "os.environ[\"WANDB_API_KEY\"] = getpass.getpass(\"Enter your WandB API key: \")" 82 | ] 83 | }, 84 | { 85 | "cell_type": "markdown", 86 | "metadata": {}, 87 | "source": [ 88 | "Now we can set up our simple application!\n", 89 | "\n", 90 | "We're going to create an agent with the following characteristics:\n", 91 | "\n", 92 | "1. `ChatOpenAI` : `gpt-3.5-turbo` powered, `temperature` set to reduce creativity\n", 93 | "2. `arxiv` tool\n", 94 | "3. `ZERO_SHOT_REACT_DESCRIPTION` agent\n", 95 | "\n", 96 | "Please refer to the following documentation if you get stuck:\n", 97 | "\n", 98 | "- [ChatOpenAI](https://api.python.langchain.com/en/latest/chat_models/langchain.chat_models.openai.ChatOpenAI.html)\n", 99 | "- [load_tools](https://api.python.langchain.com/en/latest/agents/langchain.agents.load_tools.load_tools.html)\n", 100 | "- [initialize_agent](https://api.python.langchain.com/en/latest/agents/langchain.agents.initialize.initialize_agent.html)\n", 101 | "- [AgentType](https://api.python.langchain.com/en/latest/agents/langchain.agents.agent_types.AgentType.html)" 102 | ] 103 | }, 104 | { 105 | "cell_type": "code", 106 | "execution_count": 3, 107 | "metadata": {}, 108 | "outputs": [], 109 | "source": [ 110 | "!pip install -q -U arxiv" 111 | ] 112 | }, 113 | { 114 | "cell_type": "code", 115 | "execution_count": null, 116 | "metadata": {}, 117 | "outputs": [], 118 | "source": [ 119 | "from langchain.chat_models import ChatOpenAI\n", 120 | "from langchain.agents import load_tools, initialize_agent, AgentType\n", 121 | "\n", 122 | "llm = ChatOpenAI(\n", 123 | " ### YOUR CODE HERE\n", 124 | ")\n", 125 | "\n", 126 | "tools = load_tools(\n", 127 | " ### YOUR CODE HERE\n", 128 | ")\n", 129 | "\n", 130 | "agent_chain = initialize_agent(\n", 131 | " ### YOUR CODE HERE\n", 132 | " verbose=True,\n", 133 | " handle_parsing_errors=True,\n", 134 | ")" 135 | ] 136 | }, 137 | { 138 | "cell_type": "code", 139 | "execution_count": null, 140 | "metadata": {}, 141 | "outputs": [], 142 | "source": [ 143 | "agent_chain.run(\"What is QLoRA?\")" 144 | ] 145 | }, 146 | { 147 | "cell_type": "markdown", 148 | "metadata": {}, 149 | "source": [ 150 | "Alright, now that we have some outputs - let's see what Weights and Biases was able to do!" 151 | ] 152 | }, 153 | { 154 | "cell_type": "markdown", 155 | "metadata": {}, 156 | "source": [ 157 | "### Exploring WandB Outputs\n", 158 | "\n", 159 | "First things first, we'll want to head to our WandB home page and find our projects!\n", 160 | "\n", 161 | "You'll navigate to `wandb.ai/{YOUR_USERNAME_HERE}` - and then click the `Projects` tab.\n", 162 | "\n", 163 | "![image](https://i.imgur.com/mplxa4p.png)\n", 164 | "\n", 165 | "Now we can head into our project, which should be named `langchain-testing`:\n", 166 | "\n", 167 | "![image](https://i.imgur.com/Q4AU0NC.png)\n", 168 | "\n", 169 | "Explore all the tools made available to you through the Prompt Workspace!\n", 170 | "\n", 171 | "Let's try another prompt and see what happens!" 172 | ] 173 | }, 174 | { 175 | "cell_type": "code", 176 | "execution_count": null, 177 | "metadata": {}, 178 | "outputs": [], 179 | "source": [ 180 | "agent_chain.run(\"What is LLM Ops?\")" 181 | ] 182 | }, 183 | { 184 | "cell_type": "markdown", 185 | "metadata": {}, 186 | "source": [ 187 | "As we can see, repeated calls will continue to add more information to our `langchain-testing` project!\n", 188 | "\n", 189 | "![image](https://i.imgur.com/Xze6jNE.png)" 190 | ] 191 | }, 192 | { 193 | "cell_type": "markdown", 194 | "metadata": {}, 195 | "source": [ 196 | "Let's make a slightly more complex application by adding a Prompt Cache!" 197 | ] 198 | }, 199 | { 200 | "cell_type": "markdown", 201 | "metadata": {}, 202 | "source": [ 203 | "## Prompt Caching" 204 | ] 205 | }, 206 | { 207 | "cell_type": "markdown", 208 | "metadata": {}, 209 | "source": [ 210 | "### Adding A Prompt Cache\n", 211 | "\n", 212 | "The basic idea of Prompt Caching is to provide a way to circumvent going to the LLM for prompts we have already seen.\n", 213 | "\n", 214 | "Similar to cached embeddings, the idea is simple:\n", 215 | "\n", 216 | "- Keep track of all the input/output pairs\n", 217 | "- If a user query is (in the case of semantic similarity caches) close enough to a previous prompt contained in the cache, return the output associated with that pair" 218 | ] 219 | }, 220 | { 221 | "cell_type": "markdown", 222 | "metadata": {}, 223 | "source": [ 224 | "### Initializing a Prompt Cache\n", 225 | "\n", 226 | "There are many different tools you can use to implement a Prompt Cache - from a \"build it yourself\" VectorStore implementation - to Redis - to custom libraries - there are upsides and downsides to each solution. \n", 227 | "\n", 228 | "Let's look at the Redis-backed Cache vs. `InMemoryCache` as an example:\n", 229 | "\n", 230 | "Redis Cache\n", 231 | "\n", 232 | "| Pros | Cons |\n", 233 | "|---|---|\n", 234 | "| Managed and Robust | Expensive to Host |\n", 235 | "| Integrations on all Major Cloud Platforms | Non-trivial to Integrate |\n", 236 | "| Easily Scalable | Does not have a ChatModel implementation |\n", 237 | "\n", 238 | "`InMemoryCache`\n", 239 | "\n", 240 | "| Pros | Cons |\n", 241 | "|---|---|\n", 242 | "| Easily implemented | Consumes potentially precious memory |\n", 243 | "| Completely Cloud Agnostic | Does not offer inter-session caching |\n", 244 | "\n", 245 | "For the sake of ease of use - and to allow functionality with our `ChatOpenAI` model - we'll leverage `InMemoryCache`." 246 | ] 247 | }, 248 | { 249 | "cell_type": "markdown", 250 | "metadata": {}, 251 | "source": [ 252 | "We need to set our `langchain.llm_cache` to use the `InMemoryCache`.\n", 253 | "\n", 254 | "- [`InMemoryCache`](https://api.python.langchain.com/en/latest/cache/langchain.cache.InMemoryCache.html)" 255 | ] 256 | }, 257 | { 258 | "cell_type": "code", 259 | "execution_count": 47, 260 | "metadata": {}, 261 | "outputs": [], 262 | "source": [ 263 | "import langchain\n", 264 | "from langchain.cache import InMemoryCache\n", 265 | "\n", 266 | "### YOUR CODE HERE" 267 | ] 268 | }, 269 | { 270 | "cell_type": "markdown", 271 | "metadata": {}, 272 | "source": [ 273 | "One more important fact about the `InMemoryCache` is that it is what's called an \"exact-match\" cache - meaning it will only trigger when the user query is *exactly* represented in the cache. \n", 274 | "\n", 275 | "This is a safer cache, as we can guarentee the user's query exactly matches with previous queries and we don't have to worry about edge-cases where semantic similarity might fail - but it does reduce the potential to hit the cache.\n", 276 | "\n", 277 | "We could leverage tools like `GPTCache`, or `RedisCache` (for non-chat model implementations) to get a \"semantic similarity\" cache, if desired!" 278 | ] 279 | }, 280 | { 281 | "cell_type": "code", 282 | "execution_count": null, 283 | "metadata": {}, 284 | "outputs": [], 285 | "source": [ 286 | "%%time\n", 287 | "agent_chain.run(\"What is Retrieval Augmented Generation?\")" 288 | ] 289 | }, 290 | { 291 | "cell_type": "markdown", 292 | "metadata": {}, 293 | "source": [ 294 | "Okay, that's great! Working as expected - let's take a look at the output of our `ChatOpenAI` module in our Weights and Biases project:\n", 295 | "\n", 296 | "```\n", 297 | "{ \"token_usage\": { \"prompt_tokens\": 1057, \"completion_tokens\": 130, \"total_tokens\": 1187 }, \"model_name\": \"gpt-3.5-turbo-0613\" }\n", 298 | "```\n", 299 | "\n", 300 | "So, you can see: We used `1187` total tokens, and the request took ~8s.\n", 301 | "\n", 302 | "Let's look at the full output of our Weights and Biases project:\n", 303 | "\n", 304 | "![image](https://i.imgur.com/cU8NuDK.png)\n", 305 | "\n", 306 | "Let's try the same request again and see what happens this time!" 307 | ] 308 | }, 309 | { 310 | "cell_type": "code", 311 | "execution_count": null, 312 | "metadata": {}, 313 | "outputs": [], 314 | "source": [ 315 | "%%time\n", 316 | "agent_chain.run(\"What is Retrieval Augmented Generation?\")" 317 | ] 318 | }, 319 | { 320 | "cell_type": "markdown", 321 | "metadata": {}, 322 | "source": [ 323 | "Right away, we can see that the chain only took ~0.3s, very promising! Let's check in WandB!\n", 324 | "\n", 325 | "This time, we cannot find information about token usage in Weights and Biases because we never actually needed to hit OpenAI's endpoint. \n", 326 | "\n", 327 | "Let's look at the Weights and Biases project output:\n", 328 | "\n", 329 | "![image](https://i.imgur.com/UjPsC6x.png)" 330 | ] 331 | }, 332 | { 333 | "cell_type": "markdown", 334 | "metadata": {}, 335 | "source": [ 336 | "As you can see - we completely bypass the chain - and directly return the previous result!" 337 | ] 338 | }, 339 | { 340 | "cell_type": "markdown", 341 | "metadata": {}, 342 | "source": [ 343 | "# Task\n", 344 | "\n", 345 | "Your task is to include both a prompt cache, and visibility to your application in any of your previous assignments, wrap it up in a Chainlit application, and host it on a Hugging Face Space (or EC2)!" 346 | ] 347 | } 348 | ], 349 | "metadata": { 350 | "kernelspec": { 351 | "display_name": "aims-visibility", 352 | "language": "python", 353 | "name": "python3" 354 | }, 355 | "language_info": { 356 | "codemirror_mode": { 357 | "name": "ipython", 358 | "version": 3 359 | }, 360 | "file_extension": ".py", 361 | "mimetype": "text/x-python", 362 | "name": "python", 363 | "nbconvert_exporter": "python", 364 | "pygments_lexer": "ipython3", 365 | "version": "3.11.4" 366 | }, 367 | "orig_nbformat": 4 368 | }, 369 | "nbformat": 4, 370 | "nbformat_minor": 2 371 | } 372 | --------------------------------------------------------------------------------