├── README.md ├── example_rag.ipynb ├── files ├── Audio_Kevin_Folta.wav ├── example.pdf ├── teaching.pdf ├── whisper_transcription.txt ├── youtube_transcription.txt └── youtube_transcription_reduced.txt ├── github_repo_rag.ipynb ├── imgs └── rag_diagram.png ├── llm_chat_memory.ipynb ├── whisper_rag.ipynb └── youtube_rag.ipynb /README.md: -------------------------------------------------------------------------------- 1 | # Tutorials for RAG usage with an LLM locally or in Google Colab 2 | 3 | Simple RAG tutorials that can be run locally with an LLM or using Google Colab (only Pro version). 4 | 5 | These notebooks can be executed locally or in Google Colab. 6 | Either way, you have to install Ollama to run it. 7 | 8 | RAG diagram 9 | 10 | # Tutorials 11 | 12 | * [Extracting details from a file (PDF) using RAG](./example_rag.ipynb) 13 | Open In Colab 14 | 15 | 16 | * [Extracting details from a YouTube video using RAG](./youtube_rag.ipynb) 17 | Open In Colab 18 | 19 | 20 | * [Extracting details from an audio using RAG](./whisper_rag.ipynb) 21 | Open In Colab 22 | 23 | 24 | * [Extracting details from a GitHub repo using RAG](./github_repo_rag.ipynb) 25 | Open In Colab 26 | 27 | 28 | # Technologies used 29 | 30 | For these tutorials, we use LangChain, LlamaIndex, and HuggingFace for generating the RAG application code, Ollama for serving the LLM model, and a Jupyter or Google Colab notebook. 31 | 32 |

33 | Langchain Logo 34 | LlamaIndex Logo 35 | HuggingFace Logo 36 | Ollama Logo 37 | Jupyter Logo 38 | Google Colab Logo 39 |

40 | 41 | 42 | # Intructions to run the example locally 43 | 44 | * Download and install Ollama: 45 | 46 | Go to this URL and install it: https://ollama.com/download 47 | 48 | * Pull the LLM model. In this case, llama3: 49 | 50 | ``` 51 | ollama pull llama3 52 | ``` 53 | 54 | More details about llama3 in the [official release blog](https://llama.meta.com/llama3/) and in [Ollama documentation](https://ollama.com/library/llama3). 55 | 56 | # Intructions to run the example using Google Colab (Pro account needed) 57 | 58 | * Install Ollama from the command line: 59 | 60 | (Press the button on the bottom-left part of the notebook to open a Terminal) 61 | 62 | ``` 63 | curl -fsSL https://ollama.com/install.sh | sh 64 | ``` 65 | 66 | * Pull the LLM model. In this case, llama3 67 | 68 | ``` 69 | ollama serve & ollama pull llama3 70 | ``` 71 | 72 | * Serve the model locally so the code can access it. 73 | 74 | ``` 75 | ollama serve & ollama run llama3 76 | ``` 77 | 78 | 79 | If an error is raised related to docarray, refer to this solution: https://stackoverflow.com/questions/76880224/error-using-using-docarrayinmemorysearch-in-langchain-could-not-import-docarray 80 | 81 | 82 | -------------------------------------------------------------------------------- /example_rag.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "\n", 8 | " \"Open\n", 9 | "" 10 | ] 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "metadata": {}, 15 | "source": [ 16 | "# Simple RAG example with Langchain, Ollama and and open-source LLM model\n", 17 | "\n", 18 | "In this example, we first connect to an LLM locally and make request to the LLM that Ollama is serving using LangChain. After that, we generate our RAG application from a PDF file and extract details from that document.\n", 19 | "\n", 20 | "

\n", 21 | " \"Langchain\n", 22 | " \"Ollama\n", 23 | "

\n", 24 | "\n", 25 | "Sources:\n", 26 | "\n", 27 | "* https://github.com/svpino/llm\n", 28 | "* https://github.com/AIAnytime/Gemma-7B-RAG-using-Ollama/blob/main/Ollama%20Gemma.ipynb\n", 29 | "* https://www.youtube.com/watch?v=-MexTC18h20&ab_channel=AIAnytime\n", 30 | "* https://www.youtube.com/watch?v=HRvyei7vFSM&ab_channel=Underfitted\n", 31 | "\n", 32 | " \n", 33 | "# Requirements\n", 34 | "\n", 35 | "* Ollama installed locally" 36 | ] 37 | }, 38 | { 39 | "cell_type": "markdown", 40 | "metadata": {}, 41 | "source": [ 42 | "# Install the requirements\n", 43 | "\n", 44 | "If an error is raised related to docarray, refer to this solution: https://stackoverflow.com/questions/76880224/error-using-using-docarrayinmemorysearch-in-langchain-could-not-import-docarray" 45 | ] 46 | }, 47 | { 48 | "cell_type": "code", 49 | "execution_count": null, 50 | "metadata": {}, 51 | "outputs": [], 52 | "source": [ 53 | "!pip3 install langchain\n", 54 | "!pip3 install langchain_pinecone\n", 55 | "!pip3 install langchain[docarray]\n", 56 | "!pip3 install docarray\n", 57 | "!pip3 install pypdf" 58 | ] 59 | }, 60 | { 61 | "cell_type": "markdown", 62 | "metadata": {}, 63 | "source": [ 64 | "# Select the LLM model to use\n", 65 | "\n", 66 | "The model must be downloaded locally to be used, so if you want to run llama3, you should run:\n", 67 | "\n", 68 | "```\n", 69 | "\n", 70 | "ollama pull llama3\n", 71 | "\n", 72 | "```\n", 73 | "\n", 74 | "Check the list of models available for Ollama here: https://ollama.com/library" 75 | ] 76 | }, 77 | { 78 | "cell_type": "code", 79 | "execution_count": 1, 80 | "metadata": {}, 81 | "outputs": [], 82 | "source": [ 83 | "#MODEL = \"gpt-3.5-turbo\"\n", 84 | "#MODEL = \"mixtral:8x7b\"\n", 85 | "#MODEL = \"gemma:7b\"\n", 86 | "#MODEL = \"llama2\"\n", 87 | "MODEL = \"llama3\" # https://ollama.com/library/llama3" 88 | ] 89 | }, 90 | { 91 | "cell_type": "markdown", 92 | "metadata": {}, 93 | "source": [ 94 | "# We instanciate the LLM model and the Embedding model" 95 | ] 96 | }, 97 | { 98 | "cell_type": "code", 99 | "execution_count": 2, 100 | "metadata": {}, 101 | "outputs": [ 102 | { 103 | "name": "stderr", 104 | "output_type": "stream", 105 | "text": [ 106 | "/Users/sergiopaniegoblanco/Library/Python/3.9/lib/python/site-packages/urllib3/__init__.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020\n", 107 | " warnings.warn(\n" 108 | ] 109 | }, 110 | { 111 | "data": { 112 | "text/plain": [ 113 | "'Here\\'s one:\\n\\n\"The greatest glory in living lies not in never falling, but in rising every time we fall.\" - Nelson Mandela\\n\\nI hope it inspires you to get back up and keep moving forward, no matter what challenges you may face!'" 114 | ] 115 | }, 116 | "execution_count": 2, 117 | "metadata": {}, 118 | "output_type": "execute_result" 119 | } 120 | ], 121 | "source": [ 122 | "from langchain_community.llms import Ollama\n", 123 | "from langchain_community.embeddings import OllamaEmbeddings\n", 124 | "\n", 125 | "model = Ollama(model=MODEL)\n", 126 | "embeddings = OllamaEmbeddings(model=MODEL)\n", 127 | "\n", 128 | "model.invoke(\"Give me an inspirational quote\")" 129 | ] 130 | }, 131 | { 132 | "cell_type": "code", 133 | "execution_count": 3, 134 | "metadata": {}, 135 | "outputs": [ 136 | { 137 | "data": { 138 | "text/plain": [ 139 | "'The answer to 2+2 is... 4!'" 140 | ] 141 | }, 142 | "execution_count": 3, 143 | "metadata": {}, 144 | "output_type": "execute_result" 145 | } 146 | ], 147 | "source": [ 148 | "model.invoke(\"Waht is 2+2?\")" 149 | ] 150 | }, 151 | { 152 | "cell_type": "markdown", 153 | "metadata": {}, 154 | "source": [ 155 | "## Using a parser provided by LangChain, we can transform the LLM output to something more suitable to be read" 156 | ] 157 | }, 158 | { 159 | "cell_type": "code", 160 | "execution_count": 4, 161 | "metadata": {}, 162 | "outputs": [ 163 | { 164 | "name": "stdout", 165 | "output_type": "stream", 166 | "text": [ 167 | "Here's one:\n", 168 | "\n", 169 | "\"Believe you can and you're halfway there.\" - Theodore Roosevelt\n", 170 | "\n", 171 | "I hope that inspires you to tackle your goals and pursue your dreams with confidence!\n" 172 | ] 173 | } 174 | ], 175 | "source": [ 176 | "from langchain_core.output_parsers import StrOutputParser\n", 177 | "\n", 178 | "parser = StrOutputParser()\n", 179 | "response_from_model = model.invoke(\"Give me an inspirational quote\")\n", 180 | "parsed_response = parser.parse(response_from_model)\n", 181 | "print(parsed_response)" 182 | ] 183 | }, 184 | { 185 | "cell_type": "markdown", 186 | "metadata": {}, 187 | "source": [ 188 | "# We generate the template for the conversation with the instruct-based LLM\n", 189 | "\n", 190 | "We can create a template to structure the conversation effectively.\n", 191 | "\n", 192 | "This template allows us to provide some general context to the Language Learning Model (LLM), which will be utilized for every prompt. This ensures that the model has a consistent background understanding for all interactions.\n", 193 | "\n", 194 | "Additionally, we can include specific context relevant to the particular prompt. This helps the model understand the immediate scenario or topic before addressing the actual question. Following this specific context, we then present the actual question we want the model to answer.\n", 195 | "\n", 196 | "By using this approach, we enhance the model's ability to generate accurate and relevant responses based on both the general and specific contexts provided." 197 | ] 198 | }, 199 | { 200 | "cell_type": "code", 201 | "execution_count": 5, 202 | "metadata": {}, 203 | "outputs": [ 204 | { 205 | "data": { 206 | "text/plain": [ 207 | "'\\nAnswer the question based on the context below. If you can\\'t \\nanswer the question, answer with \"I don\\'t know\".\\n\\nContext: Here is some context\\n\\nQuestion: Here is a question\\n'" 208 | ] 209 | }, 210 | "execution_count": 5, 211 | "metadata": {}, 212 | "output_type": "execute_result" 213 | } 214 | ], 215 | "source": [ 216 | "from langchain.prompts import PromptTemplate\n", 217 | "\n", 218 | "template = \"\"\"\n", 219 | "Answer the question based on the context below. If you can't \n", 220 | "answer the question, answer with \"I don't know\".\n", 221 | "\n", 222 | "Context: {context}\n", 223 | "\n", 224 | "Question: {question}\n", 225 | "\"\"\"\n", 226 | "\n", 227 | "prompt = PromptTemplate.from_template(template)\n", 228 | "prompt.format(context=\"Here is some context\", question=\"Here is a question\")" 229 | ] 230 | }, 231 | { 232 | "cell_type": "markdown", 233 | "metadata": {}, 234 | "source": [ 235 | "The model can answer prompts based on the context:" 236 | ] 237 | }, 238 | { 239 | "cell_type": "code", 240 | "execution_count": 6, 241 | "metadata": {}, 242 | "outputs": [ 243 | { 244 | "name": "stdout", 245 | "output_type": "stream", 246 | "text": [ 247 | "My name is Sergio!\n" 248 | ] 249 | } 250 | ], 251 | "source": [ 252 | "formatted_prompt = prompt.format(context=\"My parents named me Sergio\", question=\"What's your name?\")\n", 253 | "response_from_model = model.invoke(formatted_prompt)\n", 254 | "parsed_response = parser.parse(response_from_model)\n", 255 | "print(parsed_response)" 256 | ] 257 | }, 258 | { 259 | "cell_type": "markdown", 260 | "metadata": {}, 261 | "source": [ 262 | "But it can't answer what is not provided as context:" 263 | ] 264 | }, 265 | { 266 | "cell_type": "code", 267 | "execution_count": 7, 268 | "metadata": {}, 269 | "outputs": [ 270 | { 271 | "name": "stdout", 272 | "output_type": "stream", 273 | "text": [ 274 | "I don't know! The provided context only tells me that your parents named you Sergio, but it doesn't mention anything about your age. I can't infer or guess your age based on this information.\n" 275 | ] 276 | } 277 | ], 278 | "source": [ 279 | "formatted_prompt = prompt.format(context=\"My parents named me Sergio\", question=\"What's my age?\")\n", 280 | "response_from_model = model.invoke(formatted_prompt)\n", 281 | "parsed_response = parser.parse(response_from_model)\n", 282 | "print(parsed_response)" 283 | ] 284 | }, 285 | { 286 | "cell_type": "markdown", 287 | "metadata": {}, 288 | "source": [ 289 | "Even previously known info!" 290 | ] 291 | }, 292 | { 293 | "cell_type": "code", 294 | "execution_count": 8, 295 | "metadata": {}, 296 | "outputs": [ 297 | { 298 | "name": "stdout", 299 | "output_type": "stream", 300 | "text": [ 301 | "I don't know!\n" 302 | ] 303 | } 304 | ], 305 | "source": [ 306 | "formatted_prompt = prompt.format(context=\"My parents named me Sergio\", question=\"What is 2+2?\")\n", 307 | "response_from_model = model.invoke(formatted_prompt)\n", 308 | "parsed_response = parser.parse(response_from_model)\n", 309 | "print(parsed_response)" 310 | ] 311 | }, 312 | { 313 | "cell_type": "markdown", 314 | "metadata": {}, 315 | "source": [ 316 | "# Load an example PDF to do Retrieval Augmented Generation (RAG)\n", 317 | "\n", 318 | "For the example, you can select your own PDF." 319 | ] 320 | }, 321 | { 322 | "cell_type": "code", 323 | "execution_count": 21, 324 | "metadata": {}, 325 | "outputs": [ 326 | { 327 | "data": { 328 | "text/plain": [ 329 | "[Document(page_content='teaching/talks\\nUniv ersity and non-univ ersity courses shown\\n🎓Curr ently teaching assistant for:\\nSubject Wher e When\\nArti\\x00cial Intelligence3rd year Robotics Softwar e Engineering\\nUniv ersidad Re y Juan Carlos22-23\\n21-22\\n20-21\\nRobotics3rd year Telematics Engineering\\nUniv ersidad Re y Juan Carlos22-23\\n\\x00Couses:\\nCourse name Wher e When\\nIntroduction t o Coding ISDI (DMBA, MBA)December\\n2023 -\\nIntroduction t o Arti\\x00cial Intelligence IES E uropa, MadridNovember\\n2023\\nIntroduction t o Programming with P ython Atenea F ormaciónNovember\\n2023\\nIntroduction t o Arti\\x00cial Intelligence and\\nits A pplicationsAtenea F ormación July 2023\\nObject detection and segmentation with\\nTensor\\x00owPlatzi July 2022\\n© Cop yright 2024 Ser gio P aniego. P ower ed b y Jekyll with al-folio theme. Hosted b y GitHub P ages .21/6/24, 17:18 teaching/talks | Sergio Paniego\\nhttps://sergiopaniego.github.io/teaching_and_activities/ 1/3', metadata={'source': './files/teaching_talks _ Sergio Paniego.pdf', 'page': 0}),\n", 330 | " Document(page_content='💬Selected talks:\\nTalk name Wher e When\\nWhat happens when we combine video\\ngames with ChatGPT? Giving lif e to NPCsUbuP arty, Bur gosSeptember\\n2023\\nVision based end-t o-end aut onomous\\ndriving via imitation learningMaster in Ar ti\\x00cial Vision, URJC, MadridMarch\\n2023\\nRL-Studio: A Tool for Reinfor cement\\nLearning Methods in RoboticsROBO T 2022 Conf erence, Zar agozaNovember\\n2022\\n📖Master thesis dir ected:\\nThesis name Student When\\nConducción aut ónoma en tr á\\x00co usando\\naprendizaje pr ofundo extr emo a extr emo\\n(Autonomous driving in tr a\\x00c using end-\\nto-end deep learning)Enrique Y . Shinohar a Sot oMaster\\nin\\nArti\\x00cial\\nVision,\\nURJC,\\n2022-\\n2023\\n✍Reviewer:\\nJournal/Conf erence When\\n(Journal) Exper t Systems 2023\\n(Conf erence) W orkshop on Physical Agents ( WAF) 2023\\n© Cop yright 2024 Ser gio P aniego. P ower ed b y Jekyll with al-folio theme. Hosted b y GitHub P ages .21/6/24, 17:18 teaching/talks | Sergio Paniego\\nhttps://sergiopaniego.github.io/teaching_and_activities/ 2/3', metadata={'source': './files/teaching_talks _ Sergio Paniego.pdf', 'page': 1}),\n", 331 | " Document(page_content='🎓Previous teaching experience:\\nSubject Wher e When\\nModelling and Simulation of Robots3rd year Robotics Softwar e Engineering\\nUniv ersidad Re y Juan Carlos21-22\\n20-21\\nEmbedded and Real Time Systems3rd year Robotics Softwar e Engineering\\nUniv ersidad Re y Juan Carlos21-22\\n20-21\\n© Cop yright 2024 Ser gio P aniego. P ower ed b y Jekyll with al-folio theme. Hosted b y GitHub P ages .21/6/24, 17:18 teaching/talks | Sergio Paniego\\nhttps://sergiopaniego.github.io/teaching_and_activities/ 3/3', metadata={'source': './files/teaching_talks _ Sergio Paniego.pdf', 'page': 2})]" 332 | ] 333 | }, 334 | "execution_count": 21, 335 | "metadata": {}, 336 | "output_type": "execute_result" 337 | } 338 | ], 339 | "source": [ 340 | "from langchain_community.document_loaders import PyPDFLoader\n", 341 | "\n", 342 | "\n", 343 | "loader = PyPDFLoader(\"./files/teaching.pdf\")\n", 344 | "pages = loader.load_and_split()\n", 345 | "#pages = loader.load()\n", 346 | "pages" 347 | ] 348 | }, 349 | { 350 | "cell_type": "code", 351 | "execution_count": 29, 352 | "metadata": {}, 353 | "outputs": [ 354 | { 355 | "data": { 356 | "text/plain": [ 357 | "[Document(page_content='teaching/talks\\nUniv ersity and non-univ ersity courses shown\\n🎓Curr ently teaching assistant for:\\nSubject Wher e When\\nArti\\x00cial Intelligence3rd year Robotics Softwar e Engineering\\nUniv ersidad Re y Juan Carlos22-23\\n21-22\\n20-21\\nRobotics3rd year Telematics Engineering\\nUniv ersidad Re y Juan Carlos22-23\\n\\x00Couses:\\nCourse name Wher e When\\nIntroduction t o Coding ISDI (DMBA, MBA)December\\n2023 -\\nIntroduction t o Arti\\x00cial Intelligence IES E uropa, MadridNovember\\n2023\\nIntroduction t o Programming with P ython Atenea F ormaciónNovember\\n2023\\nIntroduction t o Arti\\x00cial Intelligence and\\nits A pplicationsAtenea F ormación July 2023\\nObject detection and segmentation with\\nTensor\\x00owPlatzi July 2022\\n© Cop yright 2024 Ser gio P aniego. P ower ed b y Jekyll with al-folio theme. Hosted b y GitHub P ages .21/6/24, 17:18 teaching/talks | Sergio Paniego\\nhttps://sergiopaniego.github.io/teaching_and_activities/ 1/3', metadata={'source': './files/teaching_talks _ Sergio Paniego.pdf', 'page': 0}),\n", 358 | " Document(page_content='💬Selected talks:\\nTalk name Wher e When\\nWhat happens when we combine video\\ngames with ChatGPT? Giving lif e to NPCsUbuP arty, Bur gosSeptember\\n2023\\nVision based end-t o-end aut onomous\\ndriving via imitation learningMaster in Ar ti\\x00cial Vision, URJC, MadridMarch\\n2023\\nRL-Studio: A Tool for Reinfor cement\\nLearning Methods in RoboticsROBO T 2022 Conf erence, Zar agozaNovember\\n2022\\n📖Master thesis dir ected:\\nThesis name Student When\\nConducción aut ónoma en tr á\\x00co usando\\naprendizaje pr ofundo extr emo a extr emo\\n(Autonomous driving in tr a\\x00c using end-\\nto-end deep learning)Enrique Y . Shinohar a Sot oMaster\\nin\\nArti\\x00cial\\nVision,\\nURJC,\\n2022-\\n2023\\n✍Reviewer:\\nJournal/Conf erence When\\n(Journal) Exper t Systems 2023\\n(Conf erence) W orkshop on Physical Agents ( WAF) 2023\\n© Cop yright 2024 Ser gio P aniego. P ower ed b y Jekyll with al-folio theme. Hosted b y GitHub P ages .21/6/24, 17:18 teaching/talks | Sergio Paniego\\nhttps://sergiopaniego.github.io/teaching_and_activities/ 2/3', metadata={'source': './files/teaching_talks _ Sergio Paniego.pdf', 'page': 1}),\n", 359 | " Document(page_content='🎓Previous teaching experience:\\nSubject Wher e When\\nModelling and Simulation of Robots3rd year Robotics Softwar e Engineering\\nUniv ersidad Re y Juan Carlos21-22\\n20-21\\nEmbedded and Real Time Systems3rd year Robotics Softwar e Engineering\\nUniv ersidad Re y Juan Carlos21-22\\n20-21\\n© Cop yright 2024 Ser gio P aniego. P ower ed b y Jekyll with al-folio theme. Hosted b y GitHub P ages .21/6/24, 17:18 teaching/talks | Sergio Paniego\\nhttps://sergiopaniego.github.io/teaching_and_activities/ 3/3', metadata={'source': './files/teaching_talks _ Sergio Paniego.pdf', 'page': 2})]" 360 | ] 361 | }, 362 | "execution_count": 29, 363 | "metadata": {}, 364 | "output_type": "execute_result" 365 | } 366 | ], 367 | "source": [ 368 | "from langchain.text_splitter import RecursiveCharacterTextSplitter\n", 369 | "\n", 370 | "text_splitter = RecursiveCharacterTextSplitter(chunk_size=100, chunk_overlap=20)\n", 371 | "text_documents = text_splitter.split_documents(pages)[:5]\n", 372 | "\n", 373 | "pages" 374 | ] 375 | }, 376 | { 377 | "cell_type": "markdown", 378 | "metadata": {}, 379 | "source": [ 380 | "# Store the PDF in a vector space.\n", 381 | "\n", 382 | "From Langchain docs:\n", 383 | "\n", 384 | "`DocArrayInMemorySearch is a document index provided by Docarray that stores documents in memory. It is a great starting point for small datasets, where you may not want to launch a database server.`\n", 385 | "\n", 386 | "The execution time of the following block depends on the complexity and longitude of the PDF provided. Try to keep it small and simple for the example." 387 | ] 388 | }, 389 | { 390 | "cell_type": "code", 391 | "execution_count": 30, 392 | "metadata": {}, 393 | "outputs": [], 394 | "source": [ 395 | "from langchain_community.vectorstores import DocArrayInMemorySearch\n", 396 | "\n", 397 | "vectorstore = DocArrayInMemorySearch.from_documents(text_documents, embedding=embeddings)" 398 | ] 399 | }, 400 | { 401 | "cell_type": "markdown", 402 | "metadata": {}, 403 | "source": [ 404 | "# Create retriever of vectors that are similar to be used as context" 405 | ] 406 | }, 407 | { 408 | "cell_type": "code", 409 | "execution_count": 35, 410 | "metadata": {}, 411 | "outputs": [ 412 | { 413 | "data": { 414 | "text/plain": [ 415 | "[Document(page_content='Introduction t o Coding ISDI (DMBA, MBA)December\\n2023 -', metadata={'source': './files/teaching_talks _ Sergio Paniego.pdf', 'page': 0}),\n", 416 | " Document(page_content='teaching/talks\\nUniv ersity and non-univ ersity courses shown\\n🎓Curr ently teaching assistant for:', metadata={'source': './files/teaching_talks _ Sergio Paniego.pdf', 'page': 0}),\n", 417 | " Document(page_content='Univ ersidad Re y Juan Carlos22-23\\n21-22\\n20-21\\nRobotics3rd year Telematics Engineering', metadata={'source': './files/teaching_talks _ Sergio Paniego.pdf', 'page': 0}),\n", 418 | " Document(page_content='Subject Wher e When\\nArti\\x00cial Intelligence3rd year Robotics Softwar e Engineering', metadata={'source': './files/teaching_talks _ Sergio Paniego.pdf', 'page': 0})]" 419 | ] 420 | }, 421 | "execution_count": 35, 422 | "metadata": {}, 423 | "output_type": "execute_result" 424 | } 425 | ], 426 | "source": [ 427 | "retriever = vectorstore.as_retriever()\n", 428 | "retriever.invoke(\"artificial intelligence\")" 429 | ] 430 | }, 431 | { 432 | "cell_type": "markdown", 433 | "metadata": {}, 434 | "source": [ 435 | "# Generate conversate with the document to extract the details" 436 | ] 437 | }, 438 | { 439 | "cell_type": "code", 440 | "execution_count": 32, 441 | "metadata": {}, 442 | "outputs": [], 443 | "source": [ 444 | "# Assuming retriever is an instance of a retriever class and has a method to retrieve context\n", 445 | "retrieved_context = retriever.invoke(\"artificial intelligence\")" 446 | ] 447 | }, 448 | { 449 | "cell_type": "code", 450 | "execution_count": 34, 451 | "metadata": {}, 452 | "outputs": [ 453 | { 454 | "name": "stdout", 455 | "output_type": "stream", 456 | "text": [ 457 | "Question: What are his research interests?\n", 458 | "Answer: I don't know. The provided context does not mention the specific research interests of Sergio Paniego, but it does show the courses and topics he is teaching or has taught in the past. If you're looking for information on his research interests, I recommend searching for academic articles, presentations, or online profiles where he may have shared his research areas of focus.\n", 459 | "\n", 460 | "Question: Does he have teaching experience?\n", 461 | "Answer: Based on the context, it appears that the individual has teaching experience. The documents mention \"teaching talks\" and \"teaching assistant\", which suggests that they have experience in this area. Therefore, my answer is:\n", 462 | "\n", 463 | "Yes, he has teaching experience.\n", 464 | "\n", 465 | "Question: Does he know about Tensorflow?\n", 466 | "Answer: I don't know. The provided context only shows documents related to teaching talks and course information, but it does not mention TensorFlow specifically. Therefore, I cannot determine whether the person knows about TensorFlow based on this information.\n", 467 | "\n" 468 | ] 469 | } 470 | ], 471 | "source": [ 472 | "questions = [\n", 473 | " \"What are his research interests?\",\n", 474 | " \"Does he have teaching experience?\",\n", 475 | " \"Does he know about Tensorflow?\"\n", 476 | "]\n", 477 | "\n", 478 | "for question in questions:\n", 479 | " formatted_prompt = prompt.format(context=retrieved_context, question=question)\n", 480 | " response_from_model = model.invoke(formatted_prompt)\n", 481 | " parsed_response = parser.parse(response_from_model)\n", 482 | "\n", 483 | " print(f\"Question: {question}\")\n", 484 | " print(f\"Answer: {parsed_response}\")\n", 485 | " print()" 486 | ] 487 | }, 488 | { 489 | "cell_type": "markdown", 490 | "metadata": {}, 491 | "source": [ 492 | "# Loop to ask-answer questions continously" 493 | ] 494 | }, 495 | { 496 | "cell_type": "code", 497 | "execution_count": null, 498 | "metadata": {}, 499 | "outputs": [], 500 | "source": [ 501 | "while True:\n", 502 | " print(\"Say 'exit' or 'quit' to exit the loop\")\n", 503 | " question = input('User question: ')\n", 504 | " print(f\"Question: {question}\")\n", 505 | " if question.lower() in [\"exit\", \"quit\"]:\n", 506 | " print(\"Exiting the conversation. Goodbye!\")\n", 507 | " break\n", 508 | " formatted_prompt = prompt.format(context=retrieved_context, question=question)\n", 509 | " response_from_model = model.invoke(formatted_prompt)\n", 510 | " parsed_response = parser.parse(response_from_model)\n", 511 | " print(f\"Answer: {parsed_response}\")\n", 512 | " print()" 513 | ] 514 | } 515 | ], 516 | "metadata": { 517 | "kernelspec": { 518 | "display_name": "Python 3", 519 | "language": "python", 520 | "name": "python3" 521 | }, 522 | "language_info": { 523 | "codemirror_mode": { 524 | "name": "ipython", 525 | "version": 3 526 | }, 527 | "file_extension": ".py", 528 | "mimetype": "text/x-python", 529 | "name": "python", 530 | "nbconvert_exporter": "python", 531 | "pygments_lexer": "ipython3", 532 | "version": "3.9.6" 533 | } 534 | }, 535 | "nbformat": 4, 536 | "nbformat_minor": 2 537 | } 538 | -------------------------------------------------------------------------------- /files/Audio_Kevin_Folta.wav: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sergiopaniego/RAG_local_tutorial/ad45ce43d5e28cd79413ba10c50fdd9345e18e32/files/Audio_Kevin_Folta.wav -------------------------------------------------------------------------------- /files/example.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sergiopaniego/RAG_local_tutorial/ad45ce43d5e28cd79413ba10c50fdd9345e18e32/files/example.pdf -------------------------------------------------------------------------------- /files/teaching.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sergiopaniego/RAG_local_tutorial/ad45ce43d5e28cd79413ba10c50fdd9345e18e32/files/teaching.pdf -------------------------------------------------------------------------------- /files/whisper_transcription.txt: -------------------------------------------------------------------------------- 1 | Hi, my name is Kevin Fowler and I'm very grateful to have the opportunity to work in public science. I'm very fortunate to have been able to be trained by excellent mentors early in my career to give me a toolbox to be able to unravel important questions that can help us better understand our physical universe. I'm really excited about the opportunity to participate in the science that works around agriculture and find out ways that we can farm more sustainably, working in ways that we can help limit environmental impacts of farming, but also providing profitable ways for farmers to stay in business. We're really excited about the technologies that we create creating better foods for the American consumer and industrialized world consumer, but also getting to people in need so that they can have more selection of better fruits and vegetables. Going forward, I think that the future will depend upon the integration of all technologies at the table and it's really important that we pay attention to this new breakthroughs and integrate them into our agricultural schemes. The future of agriculture in medicine room really relies on us being open to all new solutions and using all of them to solve our problems to have better planet and better people. -------------------------------------------------------------------------------- /files/youtube_transcription.txt: -------------------------------------------------------------------------------- 1 | in this video I'm going to tell you thebest laptops for students now for thisone my team and I went absolutely nutswe got in pretty much every viablestudent laptop think I'm joking I am notwe tested an epic 15 laptops everythingfrom Apple's MacBook Air to MicrosoftSurface laptop from Asus zenbooks toSamsung Galaxy books you name it wetested it oh and if you're wonderingwhat our experience with student laptopsis between the three of us who worked onthis video we have a combined fiveUniversity degrees now I value your timeso I'm going to very first briefly hiton some important points that you mustbe aware of when picking a laptop forschool then I'm going to get straightinto what you're actually here to seewhich is which laptops we recommend forvarious types of students firstly thelaptop should be small and portableunless we get hit by another virus whichwe all hope we don't students frequentlymove from classroom to classroom plusthey often have group projects wherethey meet in various locations no no onewants to Lug around a heavy Beast of amachine also when you have a laptop openin class you don't want to be the personwho blocks the view to the teacher andthe board for those sitting behind youbecause your laptop is large on the flipside you'll want a laptop that's largeenough to get real work done that meansyou want to see a good amount of contenton screen now you may think that alaptop screen size is what determineshow much content you can see and it is akey factor but size as you know isn'teverything there are two other factorsthat determine this the first is theclarity of the screen which isdetermined by how many pixels it has toprecisely display content and the secondis how bright the screen is thereforestudents should ideally get a 14 inchlaptop with a resolution higher than1920x1200 and a screen brighter than 400nits which will also help you see thescreen in a bright environment likeusing it close to a window and schoolstend to have lots of Windows if you doneed to compromise on one of thesefactors you will survive with only a1920x1200 resolution display we have twoDell XPS 13 plus laptops in the studio ahigh resolution one and one with asubstantially lower resolution displaywe found that you'll see an additionalthree rows of excel with a sharperscreen next as a student you're likelyto be at school for several years you'llwant the laptop to last that means youwant to have enough performance to befuture proof for most students allmodern processors will be good enoughhowever for those doing degrees thathave additional software requirementlike math design video editing softwaredevelopment special note if you'relooking at Intel laptops I'd avoidgetting one with a u-series processorthose processors are not fast enough toFuture proof your purchase if you get anIntel laptop get one with a more beefy Por H series now if you go for an Applelaptop or one with an AMD processor youwon't have to worry about this as theirentire lineup is powerful enough I'llspeak more about these types of studentslater in the video now for all studentstry to get a laptop with 16 gig of RAMsome manufacturers Apple still onlyoffers 8 gig in many of their basemodels which is an absolute disgrace forthe price they charge and when it comesto storage 512 gigs should really be theminimum unless you're starting somethingstorage intensive like film in whichcase you'll probably want one terabyteon the note of longevity you should buyyour laptop from a reputable brandespecially one that offers the abilityto purchase an extended warranty somemanufacturers like Dell even have anupgrade where someone will come to meetyou next business day to fix your laptopmake sure you leave money in your budgetto ensure your laptop has a warrantylong enough to cover your time at schoolnext you'll likely want decent batterylife yes modern classrooms have outletsat every desk but you will facesituations where there aren't enough forexample you're doing group work in aplace with limited Outlets like a coffeestoreapple is your best bet here theirlaptops run circles around Windowslaptops when it comes to battery lifethat being said some PC laptops do havedecent battery life I'll be placing alink to all the laptops we tested in asheet which you can access from the linkbelow on that sheet I've clearly markedout battery test results so you can seewhich laptops are good enough movingalong you don't want a laptop that hasdistracting fan noise no one wants to bethat person who's in class and suddenlyyour laptop's fan spin up and everyonelooks at you and you also don't wantyour laptop to feel uncomfortably warmto the touch that will also bedistracting next you may be tempted tobuy a Chromebook or a tablet with akeyboard instead of a full-blown laptopyes these will work for several types ofstudents with very basic needs but youreally are making things harder onyourself than they need to be moreapplications will run on the standardlaptop if you get one of those devicesyou'll regularly have to findworkarounds for things that just comeeasily to students with a standardlaptop plus if you ever use your laptopon your lap balancing a tablet standit's really not a pleasant experienceand finally make sure to use thediscounts available to you reach out toyour school to find out what discountsthey have and to help you out the sheetI mentioned earlier where I'm listingall the laptops I recommend will havelinks to the cheapest possible pricesthat they are currently available forand if you're strapped for cash I highlyrecommend buying a returned laptop froma reputable seller Best Buy has theiropen box laptops Apple has theirrefurbished program and Dell has theiroutlet store the reason I way prefer youbuy from these over eBay or Craigslistis these laptops will come withwarranties and return policies so in theevent that something doesn't work or youjust don't like the laptop you can dosomething about it plus you may be ableto stack discounts on top of thesealready reduced prices alright before weget into recommendations as mentionedevery laptop that we've tested thatcould be a good student laptop will belisted on the sheet which I'm linkingbelow we will will be keeping that sheetupdated well past this video going liveand I may put some additional laptops orexpanded rationale in there that I don'thave time to cover in this video alrightafter testing tons of different laptopsfrom all major brands the best laptopfor most students right now is the AsusZenbook 14x this laptop has a large 14.5inch display a stunning high resolutionOLED panel 16 gig of RAM and 512 gig ofstorage but none of those positives arewides our number one choice firstly itonly costs a thousand US Dollars andthat's before sales that is a lot lowerthan competitors with similar specssecond if you thought this laptop wouldfeel cheaper than bigger namecompetitors you'd be wrong it feelspremium and has one of the bestkeyboards I've used thirdly unlikecompetitors that use a lower poweredIntel U or P series processors this is avery powerful 8 series processor insideand yet it surprisingly runs quieter andfeels cooler to the touch than otherlaptops with less powerful processors Idon't know how they're doing it but itis amazing fourth even with the powerfulprocessor its battery life is solid atover eight hours for our Netflix videoplayback test our next favorite is theMacBook Air with M2 it has severalbenefits over the Zenbook in that it issignificantly lighter it feels even morepremium it has longer battery life andit has substantially better graphicsperformance so if you plan to dosomething like video editing it's goingto be viable on this laptop but it costsa lot more than the Zenbook for asimilarly spec machine and its CPUperformance is substantially worse forthose students who want to draw on theirlaptop or use it as a tablet we've got atie between lenovo's yoga 9i and the HPSpecter X360 13.5 both offer similarbenefits with the most noticeabledifference being that the yoga 9i has abit of fan noise whereas the SpecterX360 doesn't on the flip side theSpecter X360 isn't the fastest ofmachines although it is good enough forusing office and browsing the websince these laptops are so close to eachother in their overall value propositionI choose which one to get based on whichis on the best sale at the time you buyboth of these laptops regularly go onsubstantial sales where their pricesdrop by several hundred dollars for thestudents who like to game one laptopreally stands out it's the Asus G14 Iknow this laptop really well Ipersonally bought one last year and usedit a ton I then upgraded to this year'smodel the G14 is incredibly powerful forits size and can be had with a varietyof different high-powered Graphics rightup to the RTX 4080 although I find theRTX 4060 model is really The Sweet Spotin terms of price to Performance and onpricing it's really reasonable for whatyou get especially if you buy it on asale and it regularly goes on salecompared to last year's model thisyear's version is now dead quiet forcasual tasks like browsing the web andworking on office documents the maindownside with this laptop is it getsextremely warm to the touch when you useit in fact you may want to use anexternal keyboard when you gameif you want a laptop that can steal gameand doesn't feel as warm to the touchstay tuned as I'm about to talk aboutlenovo's new 14-inch slim Pro 9i in acouple of minutes for students lookingfor a laptop for video or photo editingthe MacBook Pro 14 is the one to get itis the only portable high-powered laptopthat can tackle these tasks with littleto no fan noise or a hot feeling chassisit is really unique here plus it isextremely well built it has very longbattery life excellent sound aphenomenal display and the best trackpadif the current model is too expensivefor you you can pick up last year's M1version for a heavy discount which isstill very good if you do want a Windowslaptop for these tasks or as I said youare looking for something that you cando some gaming on the side on lenovo'snew 14-inch slim Pro 9i is absolutelyawesome compared to the Asus G14 that Ijust mentioned the slim pro99 doesn'tget as warm to the touch and its fannoise is significantly quieter the slimPro 9i also has a gorgeous me mini LED165 Hertz refresh rate display the maindownside of that laptop is that it justisn't available to buy right now in theUSA although Lenovo has promised me itwill be available soon for the studentsstudying software development like I didor other disciplines that require ahigher performing machine the bestlaptop to get is the MacBook Pro 14. ithas all the benefits for softwaredevelopers that I mentioned earlier inthe video editing section plus due toits amazing display you'll see a ton ofcode on screen and that is very helpfulfor software development FYI on Mac OS Ialways find I can comfortably see a lotmore code on screen than on similarlysized Windows laptops now if you don'twant to Mac the Asus M book 14x or theslim Pro 9i are both excellent optionsfor software development they are bothplenty powerful and have excellent highresolution displays the main differencebetween the two is that the Zenbook 14xis much cheaper and has a morecomfortable keyboard the pro 9i on theother hand has dedicated graphic 6.finally if you need a budget pick theMacBook Air with M1 from several yearsago is now heavily on sale and is stillvery good on the Windows Side hp'sPavilion plus is my favorite it's a goodall-round laptop with no major issues ithas all the specs you'll need as astudent for a great price just make sureto get the model with a brighter screenits main downside is that it feels cheapand its trackpad is just okay that beingsaid if you can find the Zenbook 14xclose to either of these laptops pricesdefinitely get the Zenbook to round outhere is why we don't recommend thefollowing popular student laptops andsome of these you only really see theissues when you can compare them side byside to better laptops like we did firstthe Samsung Galaxy book 3 Pro 14 it islight and has a lovely screen howeverthe keyboard is not that comfortable itgets very warm to the touch and has Fairnoise plus its battery life is not thatgreat the Dell XPS 13 plus feels highquality and it looks stunningunfortunately its battery life isDreadful and has no physical functionrow and it gets very warm to the touchMicrosoft's surface laptop 5 put up someterrible performance results and it justlooks dated with its large bezels wellthe novos flex 5 has a low qualitydisplay that is dim for a display in2023 and lacks color accuracy Acer Swiftgo just feels cheap and didn't reallyhave anything going for it excuse thepun lenovo's ThinkPad yoga just feltdated compared to the own yoga 9i andthis is Razer Blade 14 although verynice and I did like it a lot is insanelyexpensive for students and it gets verywarm to the touch while you use it wellthat's it as I said my sheet which Iwill link below will be regularlyupdated with where to go for the bestpossible prices for these laptops if youfind there is a cheaper price availablejust let me know with a comment belowand we'll update it and big thanks toall the manufacturers who sent laptopsin for this video as well as my walletwhich had to purchase several of them soplease please please please do me asolid and share this video share it withstudents and parents you know share iton Facebook groups LinkedIn and all thatkind of stufffinally special call out to b h who sentmore laptops in than anyone else if youdon't know b h not only is it a greatstore to purchase camera gear but it hastons of laptops available too theirshipping is insanely fast and they haveseveral laptops that I haven't seen forsale anywhere else in the USA so checkthem out well that's all for today folksif you like this video you know what todo smash that like button and getsubscribed not only does it show yourappreciation for the insane effort thatwent into this video but it also makesmy mother very proud and make sure tocheck out my new channel which is allabout how to be successful in yourcareer either quickly climbing thecorporate ladder or starting a businessI'll place a link to that Channel DownBelow too till next time go do somethingawesome with your day and I will catchyou later -------------------------------------------------------------------------------- /files/youtube_transcription_reduced.txt: -------------------------------------------------------------------------------- 1 | in this video I'm going to tell you thebest laptops for students now for thisone my team and I went absolutely nutswe got in pretty much every viablestudent laptop think I'm joking I am notwe tested an epic 15 laptops everythingfrom Apple's MacBook Air to MicrosoftSurface laptop from Asus zenbooks toSamsung Galaxy books you name it wetested it oh and if you're wonderingwhat our experience with student laptopsis between the three of us who worked onthis video we have a combined fiveUniversity degrees now I value your timeso I'm going to very first briefly hiton some important points that you mustbe aware of when picking a laptop forschool then I'm going to get straightinto what you're actually here to seewhich is which laptops we recommend forvarious types of students firstly thelaptop should be small and portableunless we get hit by another virus whichwe all hope we don't students frequentlymove from classroom to classroom plusthey often have group projects wherethey meet in various locations no no onewants to Lug around a heavy Beast of amachine also when you have a laptop openin class you don't want to be the personwho blocks the view to the teacher andthe board for those sitting behind youbecause your laptop is large on the flipside you'll want a laptop that's largeenough to get real work done that meansyou want to see a good amount of contenton screen now you may think that alaptop screen size is what determineshow much content you can see and it is akey factor but size as you know isn'teverything there are two other factorsthat determine this the first is theclarity of the screen which isdetermined by how many pixels it has toprecisely display content and the secondis how bright the screen is thereforestudents should ideally get a 14 inchlaptop with a resolution higher than1920x1200 and a screen brighter than 400nits which will also help you see thescreen in a bright environment -------------------------------------------------------------------------------- /github_repo_rag.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "\n", 8 | " \"Open\n", 9 | "" 10 | ] 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "metadata": {}, 15 | "source": [ 16 | "# Example of RAG with a GitHub repo code\n", 17 | "\n", 18 | "In this example, we implement a RAG system where you can chat with your GitHub repository code.\n", 19 | "\n", 20 | "**Please, complete the example_rag.ipynb first to get more insight.**\n", 21 | "\n", 22 | "Let's go!\n", 23 | "\n", 24 | "

\n", 25 | " \"LlamaIndex\n", 26 | " \"Langchain\n", 27 | " \"HuggingFace\n", 28 | " \"Ollama\n", 29 | " \"GitHub\n", 30 | "

\n", 31 | "\n", 32 | "Sources:\n", 33 | "\n", 34 | "* https://lightning.ai/lightning-ai/studios/chat-with-your-code-using-rag\n", 35 | "* https://huggingface.co/learn/cookbook/rag_zephyr_langchain\n", 36 | "\n", 37 | "# Requirements\n", 38 | "\n", 39 | "* Ollama installed locally\n", 40 | "\n", 41 | "\n" 42 | ] 43 | }, 44 | { 45 | "cell_type": "markdown", 46 | "metadata": {}, 47 | "source": [ 48 | "#### Install dependencies" 49 | ] 50 | }, 51 | { 52 | "cell_type": "code", 53 | "execution_count": null, 54 | "metadata": {}, 55 | "outputs": [], 56 | "source": [ 57 | "!pip3 install llama_index\n", 58 | "!pip3 install llama-index-readers-github\n", 59 | "!pip3 install llama-index-embeddings-langchain\n", 60 | "!pip3 install llama-index-llms-ollama" 61 | ] 62 | }, 63 | { 64 | "cell_type": "markdown", 65 | "metadata": {}, 66 | "source": [ 67 | "#### This cell prevents the kernel to collapse when downloading the GitHub repo content" 68 | ] 69 | }, 70 | { 71 | "cell_type": "code", 72 | "execution_count": 1, 73 | "metadata": {}, 74 | "outputs": [], 75 | "source": [ 76 | "# This is due to the fact that we use asyncio.loop_until_complete in\n", 77 | "# the DiscordReader. Since the Jupyter kernel itself runs on\n", 78 | "# an event loop, we need to add some help with nesting\n", 79 | "import nest_asyncio\n", 80 | "\n", 81 | "nest_asyncio.apply()" 82 | ] 83 | }, 84 | { 85 | "cell_type": "markdown", 86 | "metadata": {}, 87 | "source": [ 88 | "#### Let's download the repo code!\n", 89 | "\n", 90 | "In this case, we target this particular repo but you can change the code to target other repos." 91 | ] 92 | }, 93 | { 94 | "cell_type": "code", 95 | "execution_count": 2, 96 | "metadata": {}, 97 | "outputs": [], 98 | "source": [ 99 | "GITHUB_ACCESS_TOKEN=\"GITHUB_ACCESS_TOKEN\"" 100 | ] 101 | }, 102 | { 103 | "cell_type": "code", 104 | "execution_count": 3, 105 | "metadata": {}, 106 | "outputs": [], 107 | "source": [ 108 | "from llama_index.readers.github import GithubRepositoryReader, GithubClient\n", 109 | "\n", 110 | "def initialize_github_client(github_token):\n", 111 | " return GithubClient(github_token)\n", 112 | "\n", 113 | "github_client = initialize_github_client(GITHUB_ACCESS_TOKEN)\n", 114 | "\n", 115 | "loader = GithubRepositoryReader(\n", 116 | " github_client,\n", 117 | " owner='sergiopaniego', # CHANGE\n", 118 | " repo='RAG_local_tutorial', # CHANGE\n", 119 | " filter_file_extensions=(\n", 120 | " [\".ipynb\"],\n", 121 | " GithubRepositoryReader.FilterType.INCLUDE,\n", 122 | " ),\n", 123 | " verbose=False,\n", 124 | " concurrent_requests=5,\n", 125 | " )\n", 126 | "\n", 127 | "docs = loader.load_data(branch=\"main\")" 128 | ] 129 | }, 130 | { 131 | "cell_type": "markdown", 132 | "metadata": {}, 133 | "source": [ 134 | "#### Load the embedding model" 135 | ] 136 | }, 137 | { 138 | "cell_type": "code", 139 | "execution_count": null, 140 | "metadata": {}, 141 | "outputs": [], 142 | "source": [ 143 | "from llama_index.embeddings.langchain import LangchainEmbedding\n", 144 | "from langchain.embeddings import HuggingFaceEmbeddings\n", 145 | "\n", 146 | "embeddings = HuggingFaceEmbeddings(model_name='BAAI/bge-base-en-v1.5')\n", 147 | "embed_model = LangchainEmbedding(embeddings)" 148 | ] 149 | }, 150 | { 151 | "cell_type": "markdown", 152 | "metadata": {}, 153 | "source": [ 154 | "#### Store the code into VectorStoreIndex using the loaded embedding model" 155 | ] 156 | }, 157 | { 158 | "cell_type": "code", 159 | "execution_count": 5, 160 | "metadata": {}, 161 | "outputs": [], 162 | "source": [ 163 | "from llama_index.core import Settings\n", 164 | "from llama_index.core import VectorStoreIndex\n", 165 | "\n", 166 | "# Create vector store and upload indexed data\n", 167 | "Settings.embed_model = embed_model # Set the embedding model to be used\n", 168 | "index = VectorStoreIndex.from_documents(docs)" 169 | ] 170 | }, 171 | { 172 | "cell_type": "markdown", 173 | "metadata": {}, 174 | "source": [ 175 | "### Load the LLM model to make the requests" 176 | ] 177 | }, 178 | { 179 | "cell_type": "code", 180 | "execution_count": 6, 181 | "metadata": {}, 182 | "outputs": [], 183 | "source": [ 184 | "from llama_index.llms.ollama import Ollama\n", 185 | "from llama_index.core import Settings\n", 186 | "\n", 187 | "# Select the LLM model. You can change the model name below.\n", 188 | "llm = Ollama(model=\"llama3\", request_timeout=500.0) \n", 189 | "\n", 190 | "# Generate a query engine from the previosuly created vector store index\n", 191 | "Settings.llm = llm # Set the LLM model to be used\n", 192 | "query_engine = index.as_query_engine(streaming=True, similarity_top_k=4)" 193 | ] 194 | }, 195 | { 196 | "cell_type": "markdown", 197 | "metadata": {}, 198 | "source": [ 199 | "#### Let's chat with the code! :)" 200 | ] 201 | }, 202 | { 203 | "cell_type": "code", 204 | "execution_count": 7, 205 | "metadata": {}, 206 | "outputs": [ 207 | { 208 | "name": "stdout", 209 | "output_type": "stream", 210 | "text": [ 211 | "This repository is likely about using language models, specifically the LLaMA model, to analyze and generate text related to agriculture and sustainability. The code in this repository appears to be focused on creating a query engine that can answer questions about agricultural practices, sustainable farming methods, and the integration of technologies into these systems. The goal seems to be to develop more efficient and environmentally friendly ways to produce food, while also providing better options for consumers and those in need.\n" 212 | ] 213 | } 214 | ], 215 | "source": [ 216 | "from llama_index.core.prompts.base import PromptTemplate\n", 217 | "\n", 218 | "response = query_engine.query('What is this repository about?')\n", 219 | "print(response)" 220 | ] 221 | } 222 | ], 223 | "metadata": { 224 | "kernelspec": { 225 | "display_name": "Python 3", 226 | "language": "python", 227 | "name": "python3" 228 | }, 229 | "language_info": { 230 | "codemirror_mode": { 231 | "name": "ipython", 232 | "version": 3 233 | }, 234 | "file_extension": ".py", 235 | "mimetype": "text/x-python", 236 | "name": "python", 237 | "nbconvert_exporter": "python", 238 | "pygments_lexer": "ipython3", 239 | "version": "3.9.6" 240 | } 241 | }, 242 | "nbformat": 4, 243 | "nbformat_minor": 2 244 | } 245 | -------------------------------------------------------------------------------- /imgs/rag_diagram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sergiopaniego/RAG_local_tutorial/ad45ce43d5e28cd79413ba10c50fdd9345e18e32/imgs/rag_diagram.png -------------------------------------------------------------------------------- /llm_chat_memory.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Testing chatbot memory with Langchain\n", 8 | "\n", 9 | "Different examples of LLM conversation memory.\n", 10 | "\n", 11 | "https://medium.com/@dranzer.ashi/conversational-buffers-in-langchain-5d689116e3e2" 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": {}, 17 | "source": [ 18 | "#### Select the baseline model" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": null, 24 | "metadata": {}, 25 | "outputs": [], 26 | "source": [ 27 | "MODEL = \"llama3\" # https://ollama.com/library/llama3" 28 | ] 29 | }, 30 | { 31 | "cell_type": "code", 32 | "execution_count": 3, 33 | "metadata": {}, 34 | "outputs": [ 35 | { 36 | "data": { 37 | "text/plain": [ 38 | "'Here\\'s one:\\n\\n\"Believe you can and you\\'re halfway there.\" - Theodore Roosevelt'" 39 | ] 40 | }, 41 | "execution_count": 3, 42 | "metadata": {}, 43 | "output_type": "execute_result" 44 | } 45 | ], 46 | "source": [ 47 | "from langchain_community.llms import Ollama\n", 48 | "from langchain_community.embeddings import OllamaEmbeddings\n", 49 | "\n", 50 | "model = Ollama(model=MODEL)\n", 51 | "embeddings = OllamaEmbeddings(model=MODEL)\n", 52 | "\n", 53 | "model.invoke(\"Give me an inspirational quote\")" 54 | ] 55 | }, 56 | { 57 | "cell_type": "code", 58 | "execution_count": 4, 59 | "metadata": {}, 60 | "outputs": [], 61 | "source": [ 62 | "from langchain.callbacks import get_openai_callback\n", 63 | "def count_tokens(chain, query):\n", 64 | " with get_openai_callback() as cb:\n", 65 | " result = chain.run(query)\n", 66 | " print(f'Spent a total of {cb.total_tokens} tokens')\n", 67 | "\n", 68 | " return result" 69 | ] 70 | }, 71 | { 72 | "cell_type": "markdown", 73 | "metadata": {}, 74 | "source": [ 75 | "## Type 1: ConversationBufferMemory\n", 76 | "\n", 77 | "Retains previous iterations" 78 | ] 79 | }, 80 | { 81 | "cell_type": "code", 82 | "execution_count": 7, 83 | "metadata": {}, 84 | "outputs": [ 85 | { 86 | "name": "stderr", 87 | "output_type": "stream", 88 | "text": [ 89 | "/Users/sergiopaniegoblanco/Library/Python/3.9/lib/python/site-packages/langchain_core/_api/deprecation.py:117: LangChainDeprecationWarning: The function `run` was deprecated in LangChain 0.1.0 and will be removed in 0.2.0. Use invoke instead.\n", 90 | " warn_deprecated(\n" 91 | ] 92 | }, 93 | { 94 | "name": "stdout", 95 | "output_type": "stream", 96 | "text": [ 97 | "\n", 98 | "\n", 99 | "\u001b[1m> Entering new ConversationChain chain...\u001b[0m\n", 100 | "Prompt after formatting:\n", 101 | "\u001b[32;1m\u001b[1;3mThe following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.\n", 102 | "\n", 103 | "Current conversation:\n", 104 | "\n", 105 | "Human: Hi, i am not able to start my dell laptop\n", 106 | "AI:\u001b[0m\n", 107 | "\n", 108 | "\u001b[1m> Finished chain.\u001b[0m\n", 109 | "Spent a total of 0 tokens\n" 110 | ] 111 | }, 112 | { 113 | "data": { 114 | "text/plain": [ 115 | "\"Hi there! Sorry to hear that your Dell laptop won't turn on. Can you tell me more about what's happening when you try to start it? Is it just not turning on at all, or does it give you any error messages or symptoms?\\n\\nBy the way, I've accessed your conversation history and see that this is your first time trying to troubleshoot an issue with your laptop. Don't worry, we'll get through this together!\"" 116 | ] 117 | }, 118 | "execution_count": 7, 119 | "metadata": {}, 120 | "output_type": "execute_result" 121 | } 122 | ], 123 | "source": [ 124 | "from langchain.chains import ConversationChain\n", 125 | "from langchain.chains.conversation.memory import ConversationBufferMemory\n", 126 | "\n", 127 | "\n", 128 | "conversation_buf = ConversationChain(llm=model,verbose=True, memory=ConversationBufferMemory())\n", 129 | "\n", 130 | "count_tokens(conversation_buf,\"Hi, i am not able to start my dell laptop\")" 131 | ] 132 | }, 133 | { 134 | "cell_type": "code", 135 | "execution_count": 8, 136 | "metadata": {}, 137 | "outputs": [ 138 | { 139 | "name": "stdout", 140 | "output_type": "stream", 141 | "text": [ 142 | "\n", 143 | "\n", 144 | "\u001b[1m> Entering new ConversationChain chain...\u001b[0m\n", 145 | "Prompt after formatting:\n", 146 | "\u001b[32;1m\u001b[1;3mThe following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.\n", 147 | "\n", 148 | "Current conversation:\n", 149 | "Human: Hi, i am not able to start my dell laptop\n", 150 | "AI: Hi there! Sorry to hear that your Dell laptop won't turn on. Can you tell me more about what's happening when you try to start it? Is it just not turning on at all, or does it give you any error messages or symptoms?\n", 151 | "\n", 152 | "By the way, I've accessed your conversation history and see that this is your first time trying to troubleshoot an issue with your laptop. Don't worry, we'll get through this together!\n", 153 | "Human: It is a dell xps 17\n", 154 | "AI:\u001b[0m\n", 155 | "\n", 156 | "\u001b[1m> Finished chain.\u001b[0m\n", 157 | "Spent a total of 0 tokens\n" 158 | ] 159 | }, 160 | { 161 | "data": { 162 | "text/plain": [ 163 | "\"The Dell XPS 17 is a powerful machine! With its sleek design and impressive specifications, I'm sure you're eager to get it up and running smoothly again.\\n\\nSo, when you try to start your XPS 17, do you see any lights or indicators on the power button or around the laptop's edges? For example, does the power button light up or are there any LED lights flashing?\"" 164 | ] 165 | }, 166 | "execution_count": 8, 167 | "metadata": {}, 168 | "output_type": "execute_result" 169 | } 170 | ], 171 | "source": [ 172 | "count_tokens(conversation_buf,\"It is a dell xps 17\")" 173 | ] 174 | }, 175 | { 176 | "cell_type": "code", 177 | "execution_count": 9, 178 | "metadata": {}, 179 | "outputs": [ 180 | { 181 | "name": "stdout", 182 | "output_type": "stream", 183 | "text": [ 184 | "Human: Hi, i am not able to start my dell laptop\n", 185 | "AI: Hi there! Sorry to hear that your Dell laptop won't turn on. Can you tell me more about what's happening when you try to start it? Is it just not turning on at all, or does it give you any error messages or symptoms?\n", 186 | "\n", 187 | "By the way, I've accessed your conversation history and see that this is your first time trying to troubleshoot an issue with your laptop. Don't worry, we'll get through this together!\n", 188 | "Human: It is a dell xps 17\n", 189 | "AI: The Dell XPS 17 is a powerful machine! With its sleek design and impressive specifications, I'm sure you're eager to get it up and running smoothly again.\n", 190 | "\n", 191 | "So, when you try to start your XPS 17, do you see any lights or indicators on the power button or around the laptop's edges? For example, does the power button light up or are there any LED lights flashing?\n" 192 | ] 193 | } 194 | ], 195 | "source": [ 196 | "print(conversation_buf.memory.buffer)" 197 | ] 198 | }, 199 | { 200 | "cell_type": "markdown", 201 | "metadata": {}, 202 | "source": [ 203 | "## Type 2: ConversationSummaryMemory\n", 204 | "\n", 205 | "Retains previous iterations summary. It asks the model for a summary of the input and them stores that." 206 | ] 207 | }, 208 | { 209 | "cell_type": "code", 210 | "execution_count": null, 211 | "metadata": {}, 212 | "outputs": [], 213 | "source": [ 214 | "from langchain.chains.conversation.memory import ConversationSummaryMemory\n", 215 | "memory = ConversationSummaryMemory(llm=model)\n", 216 | "conversation_sum = ConversationChain(llm=model, memory=memory,verbose=True)\n" 217 | ] 218 | }, 219 | { 220 | "cell_type": "code", 221 | "execution_count": 11, 222 | "metadata": {}, 223 | "outputs": [ 224 | { 225 | "name": "stdout", 226 | "output_type": "stream", 227 | "text": [ 228 | "\n", 229 | "\n", 230 | "\u001b[1m> Entering new ConversationChain chain...\u001b[0m\n", 231 | "Prompt after formatting:\n", 232 | "\u001b[32;1m\u001b[1;3mThe following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.\n", 233 | "\n", 234 | "Current conversation:\n", 235 | "\n", 236 | "Human: Hi, i am not able to start my dell laptop\n", 237 | "AI:\u001b[0m\n", 238 | "\n", 239 | "\u001b[1m> Finished chain.\u001b[0m\n", 240 | "Spent a total of 0 tokens\n" 241 | ] 242 | }, 243 | { 244 | "data": { 245 | "text/plain": [ 246 | "\"Hello! Sorry to hear that you're having trouble with your Dell laptop. Can you tell me more about what's happening when you try to turn it on? Are you getting any error messages or lights flashing on the screen? Also, have you recently installed any new software or drivers that might be causing the issue?\"" 247 | ] 248 | }, 249 | "execution_count": 11, 250 | "metadata": {}, 251 | "output_type": "execute_result" 252 | } 253 | ], 254 | "source": [ 255 | "count_tokens(conversation_sum,\"Hi, i am not able to start my dell laptop\")" 256 | ] 257 | }, 258 | { 259 | "cell_type": "code", 260 | "execution_count": 12, 261 | "metadata": {}, 262 | "outputs": [ 263 | { 264 | "name": "stdout", 265 | "output_type": "stream", 266 | "text": [ 267 | "\n", 268 | "\n", 269 | "\u001b[1m> Entering new ConversationChain chain...\u001b[0m\n", 270 | "Prompt after formatting:\n", 271 | "\u001b[32;1m\u001b[1;3mThe following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.\n", 272 | "\n", 273 | "Current conversation:\n", 274 | "Current summary:\n", 275 | "\n", 276 | "(None provided)\n", 277 | "\n", 278 | "New lines of conversation:\n", 279 | "\n", 280 | "Human: Hi, i am not able to start my Dell laptop\n", 281 | "AI: Hello! Sorry to hear that you're having trouble with your Dell laptop. Can you tell me more about what's happening when you try to turn it on? Are you getting any error messages or lights flashing on the screen? Also, have you recently installed any new software or drivers that might be causing the issue?\n", 282 | "\n", 283 | "New summary:\n", 284 | "\n", 285 | "The human is unable to start their Dell laptop and has asked for help.\n", 286 | "Human: It is a dell xps 17\n", 287 | "AI:\u001b[0m\n", 288 | "\n", 289 | "\u001b[1m> Finished chain.\u001b[0m\n", 290 | "Spent a total of 0 tokens\n" 291 | ] 292 | }, 293 | { 294 | "data": { 295 | "text/plain": [ 296 | "\"That's helpful information! The XPS 17 is a popular model, known for its sleek design and powerful performance. Since you're having trouble booting it up, I'm going to take a few educated guesses.\\n\\nFrom what I know about the XPS 17, it typically comes with a range of configurations, including Intel Core i5 or i7 processors, up to 64GB of RAM, and various storage options. Are you able to tell me more about your specific laptop's configuration? For example, what processor and amount of RAM does it have?\\n\\nAlso, have you tried pressing the power button for an extended period (like 30 seconds) to see if that resolves the issue? Sometimes, a stubborn laptop might just need a good reboot.\"" 297 | ] 298 | }, 299 | "execution_count": 12, 300 | "metadata": {}, 301 | "output_type": "execute_result" 302 | } 303 | ], 304 | "source": [ 305 | "count_tokens(conversation_sum,\"It is a dell xps 17\")" 306 | ] 307 | }, 308 | { 309 | "cell_type": "code", 310 | "execution_count": 13, 311 | "metadata": {}, 312 | "outputs": [ 313 | { 314 | "name": "stdout", 315 | "output_type": "stream", 316 | "text": [ 317 | "\n", 318 | "\n", 319 | "\u001b[1m> Entering new ConversationChain chain...\u001b[0m\n", 320 | "Prompt after formatting:\n", 321 | "\u001b[32;1m\u001b[1;3mThe following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.\n", 322 | "\n", 323 | "Current conversation:\n", 324 | "The human is unable to start their Dell XPS 17 laptop and has asked for help.\n", 325 | "Human: How do i start it up\n", 326 | "AI:\u001b[0m\n", 327 | "\n", 328 | "\u001b[1m> Finished chain.\u001b[0m\n", 329 | "Spent a total of 0 tokens\n" 330 | ] 331 | }, 332 | { 333 | "data": { 334 | "text/plain": [ 335 | "\"I'd be happy to help you troubleshoot the issue with your Dell XPS 17 laptop! Since you're having trouble powering on, I'll walk you through some steps to check if there's a simple solution.\\n\\nFirstly, let's make sure that the power cord is properly plugged into both the laptop and the wall outlet. It's possible that a loose connection could be preventing the laptop from booting up.\\n\\nAssuming that's taken care of, can you tell me what happens when you press the power button? Does the laptop make any noise, lights flicker, or does nothing at all?\\n\\nAlso, I'll just check on my end if there have been any known issues with this particular model. According to Dell's support database, some users have reported a minor BIOS issue that can cause startup problems. However, it seems like this is relatively rare and usually fixable by updating the BIOS or performing a system restore.\\n\\nIf none of these suggestions help, we may need to dive deeper into troubleshooting mode!\"" 336 | ] 337 | }, 338 | "execution_count": 13, 339 | "metadata": {}, 340 | "output_type": "execute_result" 341 | } 342 | ], 343 | "source": [ 344 | "count_tokens(conversation_sum,\"How do i start it up\")" 345 | ] 346 | }, 347 | { 348 | "cell_type": "markdown", 349 | "metadata": {}, 350 | "source": [ 351 | "## Type 3: ConversationBufferWindowMemory\n", 352 | "\n", 353 | "\n", 354 | "Remembers k previous interations" 355 | ] 356 | }, 357 | { 358 | "cell_type": "code", 359 | "execution_count": 14, 360 | "metadata": {}, 361 | "outputs": [], 362 | "source": [ 363 | "from langchain.chains.conversation.memory import ConversationBufferWindowMemory\n", 364 | "conversation_bufw = ConversationChain(llm=model, memory=ConversationBufferWindowMemory(k=1),verbose=True)" 365 | ] 366 | }, 367 | { 368 | "cell_type": "code", 369 | "execution_count": 15, 370 | "metadata": {}, 371 | "outputs": [ 372 | { 373 | "name": "stdout", 374 | "output_type": "stream", 375 | "text": [ 376 | "\n", 377 | "\n", 378 | "\u001b[1m> Entering new ConversationChain chain...\u001b[0m\n", 379 | "Prompt after formatting:\n", 380 | "\u001b[32;1m\u001b[1;3mThe following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.\n", 381 | "\n", 382 | "Current conversation:\n", 383 | "\n", 384 | "Human: Hi, i am not able to start my dell laptop\n", 385 | "AI:\u001b[0m\n", 386 | "\n", 387 | "\u001b[1m> Finished chain.\u001b[0m\n", 388 | "Spent a total of 0 tokens\n" 389 | ] 390 | }, 391 | { 392 | "data": { 393 | "text/plain": [ 394 | "\"Sorry to hear that your Dell laptop won't turn on! That can be really frustrating. Can you tell me more about what's happening? Are the lights or fans turning on at all? Have you tried pressing the power button for a longer duration or multiple times?\\n\\nAlso, have you checked the power cord and make sure it's securely plugged in? Sometimes, it's just a loose connection that can cause issues. If none of those things are working, we might need to troubleshoot further to figure out what's going on.\"" 395 | ] 396 | }, 397 | "execution_count": 15, 398 | "metadata": {}, 399 | "output_type": "execute_result" 400 | } 401 | ], 402 | "source": [ 403 | "count_tokens(conversation_bufw,\"Hi, i am not able to start my dell laptop\")" 404 | ] 405 | }, 406 | { 407 | "cell_type": "code", 408 | "execution_count": 16, 409 | "metadata": {}, 410 | "outputs": [ 411 | { 412 | "name": "stdout", 413 | "output_type": "stream", 414 | "text": [ 415 | "\n", 416 | "\n", 417 | "\u001b[1m> Entering new ConversationChain chain...\u001b[0m\n", 418 | "Prompt after formatting:\n", 419 | "\u001b[32;1m\u001b[1;3mThe following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.\n", 420 | "\n", 421 | "Current conversation:\n", 422 | "Human: Hi, i am not able to start my dell laptop\n", 423 | "AI: Sorry to hear that your Dell laptop won't turn on! That can be really frustrating. Can you tell me more about what's happening? Are the lights or fans turning on at all? Have you tried pressing the power button for a longer duration or multiple times?\n", 424 | "\n", 425 | "Also, have you checked the power cord and make sure it's securely plugged in? Sometimes, it's just a loose connection that can cause issues. If none of those things are working, we might need to troubleshoot further to figure out what's going on.\n", 426 | "Human: It is a dell xps 17\n", 427 | "AI:\u001b[0m\n", 428 | "\n", 429 | "\u001b[1m> Finished chain.\u001b[0m\n", 430 | "Spent a total of 0 tokens\n" 431 | ] 432 | }, 433 | { 434 | "data": { 435 | "text/plain": [ 436 | "\"The Dell XPS 17! That's a great laptop. Okay, so you've got the XPS 17 and it won't turn on. Can you tell me more about what's happening when you press the power button? Are there any lights or fans that are turning on at all? Even if it's just for a brief moment, that can give us some clues about what might be going on.\\n\\nAlso, I'm curious - have you tried removing any external devices, like USB drives or Ethernet cables, and see if that makes a difference? Sometimes, it's an issue with one of those peripherals that can prevent the laptop from booting up properly. And just to confirm, have you tried pressing the power button for at least 30 seconds to ensure the laptop is getting enough power to even try to turn on?\\n\\nAdditionally, since it's a Dell XPS 17, I'm going to take a wild guess and say that it might be using one of their proprietary power management systems. Have you checked the BIOS settings or the Dell Power Manager software to see if there are any unusual settings or configurations that could be causing the issue?\"" 437 | ] 438 | }, 439 | "execution_count": 16, 440 | "metadata": {}, 441 | "output_type": "execute_result" 442 | } 443 | ], 444 | "source": [ 445 | "count_tokens(conversation_bufw,\"It is a dell xps 17\")" 446 | ] 447 | }, 448 | { 449 | "cell_type": "code", 450 | "execution_count": 17, 451 | "metadata": {}, 452 | "outputs": [ 453 | { 454 | "name": "stdout", 455 | "output_type": "stream", 456 | "text": [ 457 | "\n", 458 | "\n", 459 | "\u001b[1m> Entering new ConversationChain chain...\u001b[0m\n", 460 | "Prompt after formatting:\n", 461 | "\u001b[32;1m\u001b[1;3mThe following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.\n", 462 | "\n", 463 | "Current conversation:\n", 464 | "Human: It is a dell xps 17\n", 465 | "AI: The Dell XPS 17! That's a great laptop. Okay, so you've got the XPS 17 and it won't turn on. Can you tell me more about what's happening when you press the power button? Are there any lights or fans that are turning on at all? Even if it's just for a brief moment, that can give us some clues about what might be going on.\n", 466 | "\n", 467 | "Also, I'm curious - have you tried removing any external devices, like USB drives or Ethernet cables, and see if that makes a difference? Sometimes, it's an issue with one of those peripherals that can prevent the laptop from booting up properly. And just to confirm, have you tried pressing the power button for at least 30 seconds to ensure the laptop is getting enough power to even try to turn on?\n", 468 | "\n", 469 | "Additionally, since it's a Dell XPS 17, I'm going to take a wild guess and say that it might be using one of their proprietary power management systems. Have you checked the BIOS settings or the Dell Power Manager software to see if there are any unusual settings or configurations that could be causing the issue?\n", 470 | "Human: How do i start it up\n", 471 | "AI:\u001b[0m\n" 472 | ] 473 | } 474 | ], 475 | "source": [ 476 | "count_tokens(conversation_bufw,\"How do i start it up\")" 477 | ] 478 | }, 479 | { 480 | "cell_type": "markdown", 481 | "metadata": {}, 482 | "source": [ 483 | "## Type 4: ConversationSummaryBufferMemory\n", 484 | "\n", 485 | "Combination of ConversationSummaryMemory and ConversationBufferWindowMemory\n" 486 | ] 487 | }, 488 | { 489 | "cell_type": "code", 490 | "execution_count": 5, 491 | "metadata": {}, 492 | "outputs": [], 493 | "source": [ 494 | "from langchain.chains.conversation.memory import ConversationSummaryBufferMemory\n", 495 | "conversation_sumbuf = ConversationChain(llm=model, memory=ConversationSummaryBufferMemory(llm=model),verbose=True)\n" 496 | ] 497 | }, 498 | { 499 | "cell_type": "code", 500 | "execution_count": 6, 501 | "metadata": {}, 502 | "outputs": [ 503 | { 504 | "name": "stderr", 505 | "output_type": "stream", 506 | "text": [ 507 | "/Users/sergiopaniegoblanco/Library/Python/3.9/lib/python/site-packages/langchain_core/_api/deprecation.py:117: LangChainDeprecationWarning: The function `run` was deprecated in LangChain 0.1.0 and will be removed in 0.2.0. Use invoke instead.\n", 508 | " warn_deprecated(\n" 509 | ] 510 | }, 511 | { 512 | "name": "stdout", 513 | "output_type": "stream", 514 | "text": [ 515 | "\n", 516 | "\n", 517 | "\u001b[1m> Entering new ConversationChain chain...\u001b[0m\n", 518 | "Prompt after formatting:\n", 519 | "\u001b[32;1m\u001b[1;3mThe following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.\n", 520 | "\n", 521 | "Current conversation:\n", 522 | "\n", 523 | "Human: Hi, i am not able to start my dell laptop\n", 524 | "AI:\u001b[0m\n" 525 | ] 526 | }, 527 | { 528 | "name": "stderr", 529 | "output_type": "stream", 530 | "text": [ 531 | "/Users/sergiopaniegoblanco/Library/Python/3.9/lib/python/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n", 532 | " from .autonotebook import tqdm as notebook_tqdm\n", 533 | "/Users/sergiopaniegoblanco/Library/Python/3.9/lib/python/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n", 534 | " warnings.warn(\n" 535 | ] 536 | }, 537 | { 538 | "name": "stdout", 539 | "output_type": "stream", 540 | "text": [ 541 | "\n", 542 | "\u001b[1m> Finished chain.\u001b[0m\n", 543 | "Spent a total of 0 tokens\n" 544 | ] 545 | }, 546 | { 547 | "data": { 548 | "text/plain": [ 549 | "\"Sorry to hear that your Dell laptop won't turn on! Can you tell me more about what's happening? Does the power button light up at all when you press it? Is there a beeping sound or any visual indications of activity, such as fans spinning or lights flickering?\\n\\nAlso, have you tried unplugging the power cord and pressing the power button for 30 seconds to discharge any residual electricity? Sometimes that can help resolve issues like this.\"" 550 | ] 551 | }, 552 | "execution_count": 6, 553 | "metadata": {}, 554 | "output_type": "execute_result" 555 | } 556 | ], 557 | "source": [ 558 | "count_tokens(conversation_sumbuf,\"Hi, i am not able to start my dell laptop\")" 559 | ] 560 | }, 561 | { 562 | "cell_type": "code", 563 | "execution_count": 7, 564 | "metadata": {}, 565 | "outputs": [ 566 | { 567 | "name": "stdout", 568 | "output_type": "stream", 569 | "text": [ 570 | "\n", 571 | "\n", 572 | "\u001b[1m> Entering new ConversationChain chain...\u001b[0m\n", 573 | "Prompt after formatting:\n", 574 | "\u001b[32;1m\u001b[1;3mThe following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.\n", 575 | "\n", 576 | "Current conversation:\n", 577 | "Human: Hi, i am not able to start my dell laptop\n", 578 | "AI: Sorry to hear that your Dell laptop won't turn on! Can you tell me more about what's happening? Does the power button light up at all when you press it? Is there a beeping sound or any visual indications of activity, such as fans spinning or lights flickering?\n", 579 | "\n", 580 | "Also, have you tried unplugging the power cord and pressing the power button for 30 seconds to discharge any residual electricity? Sometimes that can help resolve issues like this.\n", 581 | "Human: It is a dell xps 17\n", 582 | "AI:\u001b[0m\n", 583 | "\n", 584 | "\u001b[1m> Finished chain.\u001b[0m\n", 585 | "Spent a total of 0 tokens\n" 586 | ] 587 | }, 588 | { 589 | "data": { 590 | "text/plain": [ 591 | "\"A Dell XPS 17! Those are great laptops.\\n\\nSo, you're saying the power button doesn't light up at all when you press it? That's unusual. Normally, there would be some indication of power activity, even if the laptop itself won't turn on. And no beeps or visual signs, like fans spinning or lights flickering?\\n\\nHave you tried pressing the power button for 10-15 seconds to see if anything happens? Sometimes, these laptops can take a little extra time to boot up.\\n\\nAlso, I'm curious - have you checked the power cord and outlet to make sure they're working properly? Maybe try plugging it into a different outlet or using a different power cord to rule out any issues there.\\n\\nBy the way, is your laptop still under warranty? Just wondering because if it's not, we might need to explore some other troubleshooting options.\"" 592 | ] 593 | }, 594 | "execution_count": 7, 595 | "metadata": {}, 596 | "output_type": "execute_result" 597 | } 598 | ], 599 | "source": [ 600 | "count_tokens(conversation_sumbuf,\"It is a dell xps 17\")" 601 | ] 602 | }, 603 | { 604 | "cell_type": "code", 605 | "execution_count": 8, 606 | "metadata": {}, 607 | "outputs": [ 608 | { 609 | "name": "stdout", 610 | "output_type": "stream", 611 | "text": [ 612 | "\n", 613 | "\n", 614 | "\u001b[1m> Entering new ConversationChain chain...\u001b[0m\n", 615 | "Prompt after formatting:\n", 616 | "\u001b[32;1m\u001b[1;3mThe following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.\n", 617 | "\n", 618 | "Current conversation:\n", 619 | "Human: Hi, i am not able to start my dell laptop\n", 620 | "AI: Sorry to hear that your Dell laptop won't turn on! Can you tell me more about what's happening? Does the power button light up at all when you press it? Is there a beeping sound or any visual indications of activity, such as fans spinning or lights flickering?\n", 621 | "\n", 622 | "Also, have you tried unplugging the power cord and pressing the power button for 30 seconds to discharge any residual electricity? Sometimes that can help resolve issues like this.\n", 623 | "Human: It is a dell xps 17\n", 624 | "AI: A Dell XPS 17! Those are great laptops.\n", 625 | "\n", 626 | "So, you're saying the power button doesn't light up at all when you press it? That's unusual. Normally, there would be some indication of power activity, even if the laptop itself won't turn on. And no beeps or visual signs, like fans spinning or lights flickering?\n", 627 | "\n", 628 | "Have you tried pressing the power button for 10-15 seconds to see if anything happens? Sometimes, these laptops can take a little extra time to boot up.\n", 629 | "\n", 630 | "Also, I'm curious - have you checked the power cord and outlet to make sure they're working properly? Maybe try plugging it into a different outlet or using a different power cord to rule out any issues there.\n", 631 | "\n", 632 | "By the way, is your laptop still under warranty? Just wondering because if it's not, we might need to explore some other troubleshooting options.\n", 633 | "Human: How do i start it up\n", 634 | "AI:\u001b[0m\n", 635 | "\n", 636 | "\u001b[1m> Finished chain.\u001b[0m\n", 637 | "Spent a total of 0 tokens\n" 638 | ] 639 | }, 640 | { 641 | "data": { 642 | "text/plain": [ 643 | "\"I'd be happy to help you troubleshoot or find a solution for starting your Dell XPS 17!\\n\\nIf the power button doesn't light up at all when you press it, and there's no beeping sound or visual signs of activity, that suggests the issue might be more serious than just a simple battery drain. In this case, I'd recommend trying to boot the laptop in a different way.\\n\\nOne option is to try booting from a USB drive or CD/DVD disc, if you have one handy. This can sometimes help identify if the problem is with the BIOS (Basic Input/Output System) or something else.\\n\\nAnother approach would be to try resetting the BIOS settings to their default values. To do this, you'll need to press F2 during boot-up (just after powering on). You'll then enter the BIOS setup menu, where you can reset the settings to their defaults and save the changes. This might help resolve any issues with the laptop's hardware or firmware.\\n\\nIf none of these steps work, it's possible that there's a more serious issue with your laptop's hardware or motherboard. In this case, I'd recommend contacting Dell support for further assistance, as they may be able to diagnose and repair the issue under warranty (if applicable).\\n\\nDoes any of this sound like something you'd like to try?\"" 644 | ] 645 | }, 646 | "execution_count": 8, 647 | "metadata": {}, 648 | "output_type": "execute_result" 649 | } 650 | ], 651 | "source": [ 652 | "count_tokens(conversation_sumbuf,\"How do i start it up\")" 653 | ] 654 | } 655 | ], 656 | "metadata": { 657 | "kernelspec": { 658 | "display_name": "Python 3", 659 | "language": "python", 660 | "name": "python3" 661 | }, 662 | "language_info": { 663 | "codemirror_mode": { 664 | "name": "ipython", 665 | "version": 3 666 | }, 667 | "file_extension": ".py", 668 | "mimetype": "text/x-python", 669 | "name": "python", 670 | "nbconvert_exporter": "python", 671 | "pygments_lexer": "ipython3", 672 | "version": "3.9.6" 673 | } 674 | }, 675 | "nbformat": 4, 676 | "nbformat_minor": 2 677 | } 678 | -------------------------------------------------------------------------------- /whisper_rag.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "\n", 8 | " \"Open\n", 9 | "" 10 | ] 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "metadata": {}, 15 | "source": [ 16 | "# RAG example with an audio\n", 17 | "\n", 18 | "In this example, we use an audio as information source for the conversation. \n", 19 | "\n", 20 | "**Please, complete the example_rag.ipynb first to get more insight.**\n", 21 | "\n", 22 | "Imagine a situation where you directly communicate through voice with the LLM.\n", 23 | "\n", 24 | "In this case, we use Whisper, a general-purpose speech recognition model by OpenAI to convert an audio to text.\n", 25 | "\n", 26 | "From that text, we can generate a conversation with the LLM model to extract the required information.\n", 27 | "\n", 28 | "

\n", 29 | " \"Langchain\n", 30 | " \"Ollama\n", 31 | " \"OpenAI\n", 32 | "

\n", 33 | "\n", 34 | "\n", 35 | "Sources:\n", 36 | "\n", 37 | "* Whisper details: https://github.com/openai/whisper\n", 38 | "* https://github.com/svpino/llm\n", 39 | "\n", 40 | "# Requirements\n", 41 | "\n", 42 | "* Ollama installed locally\n" 43 | ] 44 | }, 45 | { 46 | "cell_type": "markdown", 47 | "metadata": {}, 48 | "source": [ 49 | "## First, we install Whisper and its dependencies" 50 | ] 51 | }, 52 | { 53 | "cell_type": "code", 54 | "execution_count": null, 55 | "metadata": {}, 56 | "outputs": [], 57 | "source": [ 58 | "!pip3 install langchain\n", 59 | "!pip3 install langchain_pinecone\n", 60 | "!pip3 install langchain[docarray]\n", 61 | "!pip3 install docarray\n", 62 | "!pip3 install pypdf" 63 | ] 64 | }, 65 | { 66 | "cell_type": "code", 67 | "execution_count": null, 68 | "metadata": {}, 69 | "outputs": [], 70 | "source": [ 71 | "!pip3 install openai-whisper" 72 | ] 73 | }, 74 | { 75 | "cell_type": "markdown", 76 | "metadata": {}, 77 | "source": [ 78 | "### We install ffmpeg version depending on the platform where we'll be running the example" 79 | ] 80 | }, 81 | { 82 | "cell_type": "code", 83 | "execution_count": null, 84 | "metadata": {}, 85 | "outputs": [], 86 | "source": [ 87 | "# on Ubuntu or Debian\n", 88 | "#sudo apt update && sudo apt install ffmpeg\n", 89 | "\n", 90 | "# on Arch Linux\n", 91 | "#sudo pacman -S ffmpeg\n", 92 | "\n", 93 | "# on MacOS using Homebrew (https://brew.sh/)\n", 94 | "!brew install ffmpeg\n", 95 | "\n", 96 | "# on Windows using Chocolatey (https://chocolatey.org/)\n", 97 | "#choco install ffmpeg\n", 98 | "\n", 99 | "# on Windows using Scoop (https://scoop.sh/)\n", 100 | "#scoop install ffmpeg" 101 | ] 102 | }, 103 | { 104 | "cell_type": "markdown", 105 | "metadata": {}, 106 | "source": [ 107 | "# Select the LLM model to use\n", 108 | "\n", 109 | "The model must be downloaded locally to be used, so if you want to run llama3, you should run:\n", 110 | "\n", 111 | "```\n", 112 | "\n", 113 | "ollama pull llama3\n", 114 | "\n", 115 | "```\n", 116 | "\n", 117 | "Check the list of models available for Ollama here: https://ollama.com/library" 118 | ] 119 | }, 120 | { 121 | "cell_type": "markdown", 122 | "metadata": {}, 123 | "source": [ 124 | "## Given an example audio, we transcribe it to text so we can use it.\n", 125 | "\n", 126 | "The example audio should be downloaded locally. Change the file name with your own example." 127 | ] 128 | }, 129 | { 130 | "cell_type": "code", 131 | "execution_count": 21, 132 | "metadata": {}, 133 | "outputs": [ 134 | { 135 | "name": "stderr", 136 | "output_type": "stream", 137 | "text": [ 138 | "/Users/sergiopaniegoblanco/Library/Python/3.9/lib/python/site-packages/whisper/transcribe.py:115: UserWarning: FP16 is not supported on CPU; using FP32 instead\n", 139 | " warnings.warn(\"FP16 is not supported on CPU; using FP32 instead\")\n" 140 | ] 141 | }, 142 | { 143 | "name": "stdout", 144 | "output_type": "stream", 145 | "text": [ 146 | " Hi, my name is Kevin Fowler and I'm very grateful to have the opportunity to work in public science. I'm very fortunate to have been able to be trained by excellent mentors early in my career to give me a toolbox to be able to unravel important questions that can help us better understand our physical universe. I'm really excited about the opportunity to participate in the science that works around agriculture and find out ways that we can farm more sustainably, working in ways that we can help limit environmental impacts of farming, but also providing profitable ways for farmers to stay in business. We're really excited about the technologies that we create creating better foods for the American consumer and industrialized world consumer, but also getting to people in need so that they can have more selection of better fruits and vegetables. Going forward, I think that the future will depend upon the integration of all technologies at the table and it's really important that we pay attention to this new breakthroughs and integrate them into our agricultural schemes. The future of agriculture in medicine room really relies on us being open to all new solutions and using all of them to solve our problems to have better planet and better people.\n" 147 | ] 148 | } 149 | ], 150 | "source": [ 151 | "import whisper\n", 152 | "\n", 153 | "whisper_model = whisper.load_model(\"base\")\n", 154 | "\n", 155 | "def transcribe_audio(audio_path):\n", 156 | " result = whisper_model.transcribe(audio_path)\n", 157 | " return result['text']\n", 158 | "\n", 159 | "# https://commons.wikimedia.org/wiki/File:Audio_Kevin_Folta.wav\n", 160 | "# Audio example\n", 161 | "audio_path = \"./files/Audio_Kevin_Folta.wav\" # CHANGE THIS FILE\n", 162 | "\n", 163 | "# Transcribir el audio\n", 164 | "transcribed_text = transcribe_audio(audio_path)\n", 165 | "\n", 166 | "print(transcribed_text)\n", 167 | "\n", 168 | "with open(\"./files/whisper_transcription.txt\", \"a\") as file:\n", 169 | " file.write(transcribed_text)\n" 170 | ] 171 | }, 172 | { 173 | "cell_type": "markdown", 174 | "metadata": {}, 175 | "source": [ 176 | "# We instantiate the model and the embeddings" 177 | ] 178 | }, 179 | { 180 | "cell_type": "code", 181 | "execution_count": 17, 182 | "metadata": {}, 183 | "outputs": [], 184 | "source": [ 185 | "#MODEL = \"gpt-3.5-turbo\"\n", 186 | "#MODEL = \"mixtral:8x7b\"\n", 187 | "#MODEL = \"gemma:7b\"\n", 188 | "#MODEL = \"llama2\"\n", 189 | "MODEL = \"llama3\" # https://ollama.com/library/llama3" 190 | ] 191 | }, 192 | { 193 | "cell_type": "code", 194 | "execution_count": 18, 195 | "metadata": {}, 196 | "outputs": [], 197 | "source": [ 198 | "from langchain_community.llms import Ollama\n", 199 | "from langchain_community.embeddings import OllamaEmbeddings\n", 200 | "\n", 201 | "\n", 202 | "model = Ollama(model=MODEL)\n", 203 | "embeddings = OllamaEmbeddings(model=MODEL)" 204 | ] 205 | }, 206 | { 207 | "cell_type": "markdown", 208 | "metadata": {}, 209 | "source": [ 210 | "# We load the transcription previously saved using TextLoader" 211 | ] 212 | }, 213 | { 214 | "cell_type": "code", 215 | "execution_count": 22, 216 | "metadata": {}, 217 | "outputs": [ 218 | { 219 | "data": { 220 | "text/plain": [ 221 | "[Document(page_content=\" Hi, my name is Kevin Fowler and I'm very grateful to have the opportunity to work in public science. I'm very fortunate to have been able to be trained by excellent mentors early in my career to give me a toolbox to be able to unravel important questions that can help us better understand our physical universe. I'm really excited about the opportunity to participate in the science that works around agriculture and find out ways that we can farm more sustainably, working in ways that we can help limit environmental impacts of farming, but also providing profitable ways for farmers to stay in business. We're really excited about the technologies that we create creating better foods for the American consumer and industrialized world consumer, but also getting to people in need so that they can have more selection of better fruits and vegetables. Going forward, I think that the future will depend upon the integration of all technologies at the table and it's really important that we pay attention to this new breakthroughs and integrate them into our agricultural schemes. The future of agriculture in medicine room really relies on us being open to all new solutions and using all of them to solve our problems to have better planet and better people.\", metadata={'source': './files/whisper_transcription.txt'})]" 222 | ] 223 | }, 224 | "execution_count": 22, 225 | "metadata": {}, 226 | "output_type": "execute_result" 227 | } 228 | ], 229 | "source": [ 230 | "from langchain_community.document_loaders import TextLoader\n", 231 | "\n", 232 | "loader = TextLoader(\"./files/whisper_transcription.txt\")\n", 233 | "text_documents = loader.load()\n", 234 | "text_documents" 235 | ] 236 | }, 237 | { 238 | "cell_type": "markdown", 239 | "metadata": {}, 240 | "source": [ 241 | "# We explit the document into chunks" 242 | ] 243 | }, 244 | { 245 | "cell_type": "code", 246 | "execution_count": 24, 247 | "metadata": {}, 248 | "outputs": [ 249 | { 250 | "data": { 251 | "text/plain": [ 252 | "[Document(page_content=\"Hi, my name is Kevin Fowler and I'm very grateful to have the opportunity to work in public\", metadata={'source': './files/whisper_transcription.txt'}),\n", 253 | " Document(page_content=\"to work in public science. I'm very fortunate to have been able to be trained by excellent mentors\", metadata={'source': './files/whisper_transcription.txt'}),\n", 254 | " Document(page_content='excellent mentors early in my career to give me a toolbox to be able to unravel important questions', metadata={'source': './files/whisper_transcription.txt'}),\n", 255 | " Document(page_content=\"important questions that can help us better understand our physical universe. I'm really excited\", metadata={'source': './files/whisper_transcription.txt'}),\n", 256 | " Document(page_content=\"I'm really excited about the opportunity to participate in the science that works around\", metadata={'source': './files/whisper_transcription.txt'})]" 257 | ] 258 | }, 259 | "execution_count": 24, 260 | "metadata": {}, 261 | "output_type": "execute_result" 262 | } 263 | ], 264 | "source": [ 265 | "from langchain.text_splitter import RecursiveCharacterTextSplitter\n", 266 | "\n", 267 | "text_splitter = RecursiveCharacterTextSplitter(chunk_size=100, chunk_overlap=20)\n", 268 | "text_documents = text_splitter.split_documents(text_documents)" 269 | ] 270 | }, 271 | { 272 | "cell_type": "code", 273 | "execution_count": null, 274 | "metadata": {}, 275 | "outputs": [], 276 | "source": [ 277 | "text_documents" 278 | ] 279 | }, 280 | { 281 | "cell_type": "markdown", 282 | "metadata": {}, 283 | "source": [ 284 | "# Store the PDF in a vector space.\n", 285 | "\n", 286 | "From Langchain docs:\n", 287 | "\n", 288 | "`DocArrayInMemorySearch is a document index provided by Docarray that stores documents in memory. It is a great starting point for small datasets, where you may not want to launch a database server.`\n", 289 | "\n", 290 | "The execution time of the following block depends on the complexity and longitude of the PDF provided. Try to keep it small and simple for the example." 291 | ] 292 | }, 293 | { 294 | "cell_type": "code", 295 | "execution_count": 25, 296 | "metadata": {}, 297 | "outputs": [ 298 | { 299 | "name": "stderr", 300 | "output_type": "stream", 301 | "text": [ 302 | "/Users/sergiopaniegoblanco/Library/Python/3.9/lib/python/site-packages/pydantic/_migration.py:283: UserWarning: `pydantic.error_wrappers:ValidationError` has been moved to `pydantic:ValidationError`.\n", 303 | " warnings.warn(f'`{import_path}` has been moved to `{new_location}`.')\n" 304 | ] 305 | } 306 | ], 307 | "source": [ 308 | "from langchain_community.vectorstores import DocArrayInMemorySearch\n", 309 | "\n", 310 | "vectorstore = DocArrayInMemorySearch.from_documents(text_documents, embedding=embeddings)" 311 | ] 312 | }, 313 | { 314 | "cell_type": "code", 315 | "execution_count": 26, 316 | "metadata": {}, 317 | "outputs": [], 318 | "source": [ 319 | "retriever = vectorstore.as_retriever()" 320 | ] 321 | }, 322 | { 323 | "cell_type": "markdown", 324 | "metadata": {}, 325 | "source": [ 326 | "# Generate the conversation template" 327 | ] 328 | }, 329 | { 330 | "cell_type": "code", 331 | "execution_count": 27, 332 | "metadata": {}, 333 | "outputs": [], 334 | "source": [ 335 | "from langchain.prompts import PromptTemplate\n", 336 | "\n", 337 | "template = \"\"\"\n", 338 | "Answer the question based on the context below. If you can't \n", 339 | "answer the question, answer with \"I don't know\".\n", 340 | "\n", 341 | "Context: {context}\n", 342 | "\n", 343 | "Question: {question}\n", 344 | "\"\"\"\n", 345 | "\n", 346 | "prompt = PromptTemplate.from_template(template)" 347 | ] 348 | }, 349 | { 350 | "cell_type": "markdown", 351 | "metadata": {}, 352 | "source": [ 353 | "# We instantiate the parser" 354 | ] 355 | }, 356 | { 357 | "cell_type": "code", 358 | "execution_count": 28, 359 | "metadata": {}, 360 | "outputs": [], 361 | "source": [ 362 | "from langchain_core.output_parsers import StrOutputParser\n", 363 | "\n", 364 | "parser = StrOutputParser()" 365 | ] 366 | }, 367 | { 368 | "cell_type": "markdown", 369 | "metadata": {}, 370 | "source": [ 371 | "# We can now extract the information from the audio!" 372 | ] 373 | }, 374 | { 375 | "cell_type": "code", 376 | "execution_count": 30, 377 | "metadata": {}, 378 | "outputs": [ 379 | { 380 | "name": "stdout", 381 | "output_type": "stream", 382 | "text": [ 383 | "Say 'exit' or 'quit' to exit the loop\n", 384 | "Question: What's his name?\n", 385 | "Answer: Kevin Fowler.\n", 386 | "\n", 387 | "Say 'exit' or 'quit' to exit the loop\n", 388 | "Question: exit\n", 389 | "Exiting the conversation. Goodbye!\n" 390 | ] 391 | } 392 | ], 393 | "source": [ 394 | "while True:\n", 395 | " print(\"Say 'exit' or 'quit' to exit the loop\")\n", 396 | " question = input('User question: ')\n", 397 | " print(f\"Question: {question}\")\n", 398 | " if question.lower() in [\"exit\", \"quit\"]:\n", 399 | " print(\"Exiting the conversation. Goodbye!\")\n", 400 | " break\n", 401 | " retrieved_context = retriever.invoke(question)\n", 402 | " formatted_prompt = prompt.format(context=retrieved_context, question=question)\n", 403 | " response_from_model = model.invoke(formatted_prompt)\n", 404 | " parsed_response = parser.parse(response_from_model)\n", 405 | " print(f\"Answer: {parsed_response}\")\n", 406 | " print()" 407 | ] 408 | }, 409 | { 410 | "cell_type": "code", 411 | "execution_count": null, 412 | "metadata": {}, 413 | "outputs": [], 414 | "source": [] 415 | } 416 | ], 417 | "metadata": { 418 | "kernelspec": { 419 | "display_name": "Python 3", 420 | "language": "python", 421 | "name": "python3" 422 | }, 423 | "language_info": { 424 | "codemirror_mode": { 425 | "name": "ipython", 426 | "version": 3 427 | }, 428 | "file_extension": ".py", 429 | "mimetype": "text/x-python", 430 | "name": "python", 431 | "nbconvert_exporter": "python", 432 | "pygments_lexer": "ipython3", 433 | "version": "3.9.6" 434 | } 435 | }, 436 | "nbformat": 4, 437 | "nbformat_minor": 2 438 | } 439 | -------------------------------------------------------------------------------- /youtube_rag.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "\n", 8 | " \"Open\n", 9 | "" 10 | ] 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "metadata": {}, 15 | "source": [ 16 | "# Example of RAG with a Youtube video\n", 17 | "\n", 18 | "In this example, we download the trascription of a Youtube video and use an LLM for extracting information from that video.\n", 19 | "\n", 20 | "**Please, complete the example_rag.ipynb first to get more insight.**\n", 21 | "\n", 22 | "Let's go!\n", 23 | "\n", 24 | "

\n", 25 | " \"Langchain\n", 26 | " \"Ollama\n", 27 | " \"Youtube\n", 28 | "

\n", 29 | "\n", 30 | "Sources:\n", 31 | "\n", 32 | "* https://github.com/svpino/llm\n", 33 | "\n", 34 | "# Requirements\n", 35 | "\n", 36 | "* Ollama installed locally" 37 | ] 38 | }, 39 | { 40 | "cell_type": "markdown", 41 | "metadata": {}, 42 | "source": [ 43 | "### Install the dependencies" 44 | ] 45 | }, 46 | { 47 | "cell_type": "code", 48 | "execution_count": null, 49 | "metadata": {}, 50 | "outputs": [], 51 | "source": [ 52 | "!pip3 install langchain\n", 53 | "!pip3 install langchain_pinecone\n", 54 | "!pip3 install langchain[docarray]\n", 55 | "!pip3 install docarray\n", 56 | "!pip3 install pypdf" 57 | ] 58 | }, 59 | { 60 | "cell_type": "code", 61 | "execution_count": null, 62 | "metadata": {}, 63 | "outputs": [], 64 | "source": [ 65 | "!pip3 install youtube_transcript_api" 66 | ] 67 | }, 68 | { 69 | "cell_type": "markdown", 70 | "metadata": {}, 71 | "source": [ 72 | "# Let's download an example transcript from a YT video\n", 73 | "\n", 74 | "You can change the id of the video to download other video transcriptions.\n", 75 | "We save the contect to a file" 76 | ] 77 | }, 78 | { 79 | "cell_type": "code", 80 | "execution_count": null, 81 | "metadata": {}, 82 | "outputs": [], 83 | "source": [ 84 | "from youtube_transcript_api import YouTubeTranscriptApi\n", 85 | "\n", 86 | "srt = YouTubeTranscriptApi.get_transcript(\"pxiP-HJLCx0\") # CHANGE THE ID OF THE VIDEO\n", 87 | "\n", 88 | "with open(\"./files/youtube_transcription.txt\", \"a\") as file:\n", 89 | " for i in srt:\n", 90 | " file.write(i['text'])" 91 | ] 92 | }, 93 | { 94 | "cell_type": "markdown", 95 | "metadata": {}, 96 | "source": [ 97 | "# Select the LLM model to use\n", 98 | "\n", 99 | "The model must be downloaded locally to be used, so if you want to run llama3, you should run:\n", 100 | "\n", 101 | "```\n", 102 | "\n", 103 | "ollama pull llama3\n", 104 | "\n", 105 | "```\n", 106 | "\n", 107 | "Check the list of models available for Ollama here: https://ollama.com/library" 108 | ] 109 | }, 110 | { 111 | "cell_type": "markdown", 112 | "metadata": {}, 113 | "source": [ 114 | "# We instantiate the model and the embeddings" 115 | ] 116 | }, 117 | { 118 | "cell_type": "code", 119 | "execution_count": null, 120 | "metadata": {}, 121 | "outputs": [], 122 | "source": [ 123 | "#MODEL = \"gpt-3.5-turbo\"\n", 124 | "#MODEL = \"mixtral:8x7b\"\n", 125 | "#MODEL = \"gemma:7b\"\n", 126 | "#MODEL = \"llama2\"\n", 127 | "MODEL = \"llama3\" # https://ollama.com/library/llama3" 128 | ] 129 | }, 130 | { 131 | "cell_type": "code", 132 | "execution_count": null, 133 | "metadata": {}, 134 | "outputs": [], 135 | "source": [ 136 | "from langchain_community.llms import Ollama\n", 137 | "from langchain_community.embeddings import OllamaEmbeddings\n", 138 | "\n", 139 | "model = Ollama(model=MODEL)\n", 140 | "embeddings = OllamaEmbeddings(model=MODEL)" 141 | ] 142 | }, 143 | { 144 | "cell_type": "markdown", 145 | "metadata": {}, 146 | "source": [ 147 | "# We load the transcription previously saved using TextLoader" 148 | ] 149 | }, 150 | { 151 | "cell_type": "code", 152 | "execution_count": null, 153 | "metadata": {}, 154 | "outputs": [], 155 | "source": [ 156 | "from langchain_community.document_loaders import TextLoader\n", 157 | "\n", 158 | "loader = TextLoader(\"./files/youtube_transcription.txt\")\n", 159 | "text_documents = loader.load()\n", 160 | "text_documents" 161 | ] 162 | }, 163 | { 164 | "cell_type": "markdown", 165 | "metadata": {}, 166 | "source": [ 167 | "# We explit the document into chunks" 168 | ] 169 | }, 170 | { 171 | "cell_type": "code", 172 | "execution_count": null, 173 | "metadata": {}, 174 | "outputs": [], 175 | "source": [ 176 | "from langchain.text_splitter import RecursiveCharacterTextSplitter\n", 177 | "\n", 178 | "text_splitter = RecursiveCharacterTextSplitter(chunk_size=100, chunk_overlap=20)\n", 179 | "text_documents = text_splitter.split_documents(text_documents)[:5]" 180 | ] 181 | }, 182 | { 183 | "cell_type": "code", 184 | "execution_count": null, 185 | "metadata": {}, 186 | "outputs": [], 187 | "source": [ 188 | "text_documents" 189 | ] 190 | }, 191 | { 192 | "cell_type": "markdown", 193 | "metadata": {}, 194 | "source": [ 195 | "# Store the PDF in a vector space.\n", 196 | "\n", 197 | "From Langchain docs:\n", 198 | "\n", 199 | "`DocArrayInMemorySearch is a document index provided by Docarray that stores documents in memory. It is a great starting point for small datasets, where you may not want to launch a database server.`\n", 200 | "\n", 201 | "The execution time of the following block depends on the complexity and longitude of the PDF provided. Try to keep it small and simple for the example." 202 | ] 203 | }, 204 | { 205 | "cell_type": "code", 206 | "execution_count": null, 207 | "metadata": {}, 208 | "outputs": [], 209 | "source": [ 210 | "from langchain_community.vectorstores import DocArrayInMemorySearch\n", 211 | "\n", 212 | "vectorstore = DocArrayInMemorySearch.from_documents(text_documents, embedding=embeddings)" 213 | ] 214 | }, 215 | { 216 | "cell_type": "code", 217 | "execution_count": null, 218 | "metadata": {}, 219 | "outputs": [], 220 | "source": [ 221 | "retriever = vectorstore.as_retriever()" 222 | ] 223 | }, 224 | { 225 | "cell_type": "markdown", 226 | "metadata": {}, 227 | "source": [ 228 | "# We instantiate the parser" 229 | ] 230 | }, 231 | { 232 | "cell_type": "code", 233 | "execution_count": null, 234 | "metadata": {}, 235 | "outputs": [], 236 | "source": [ 237 | "from langchain_core.output_parsers import StrOutputParser\n", 238 | "\n", 239 | "parser = StrOutputParser()" 240 | ] 241 | }, 242 | { 243 | "cell_type": "markdown", 244 | "metadata": {}, 245 | "source": [ 246 | "# Generate the conversation template" 247 | ] 248 | }, 249 | { 250 | "cell_type": "code", 251 | "execution_count": null, 252 | "metadata": {}, 253 | "outputs": [], 254 | "source": [ 255 | "from langchain.prompts import PromptTemplate\n", 256 | "\n", 257 | "template = \"\"\"\n", 258 | "Answer the question based on the context below. If you can't \n", 259 | "answer the question, answer with \"I don't know\".\n", 260 | "\n", 261 | "Context: {context}\n", 262 | "\n", 263 | "Question: {question}\n", 264 | "\"\"\"\n", 265 | "\n", 266 | "prompt = PromptTemplate.from_template(template)\n", 267 | "prompt.format(context=\"Here is some context\", question=\"Here is a question\")" 268 | ] 269 | }, 270 | { 271 | "cell_type": "markdown", 272 | "metadata": {}, 273 | "source": [ 274 | "# We can now extract the information from the video!" 275 | ] 276 | }, 277 | { 278 | "cell_type": "code", 279 | "execution_count": null, 280 | "metadata": {}, 281 | "outputs": [], 282 | "source": [ 283 | "retrieved_context = retriever.invoke(\"laptop\")\n", 284 | "questions = [\n", 285 | " \"Which is the best laptop for students?\",\n", 286 | " \"How much is a laptop worth?\",\n", 287 | " \"Make a summary of the video\"\n", 288 | "]\n", 289 | "\n", 290 | "for question in questions:\n", 291 | " formatted_prompt = prompt.format(context=retrieved_context, question=question)\n", 292 | " response_from_model = model.invoke(formatted_prompt)\n", 293 | " parsed_response = parser.parse(response_from_model)\n", 294 | "\n", 295 | " print(f\"Question: {question}\")\n", 296 | " print(f\"Answer: {parsed_response}\")\n", 297 | " print()" 298 | ] 299 | } 300 | ], 301 | "metadata": { 302 | "kernelspec": { 303 | "display_name": "Python 3", 304 | "language": "python", 305 | "name": "python3" 306 | }, 307 | "language_info": { 308 | "codemirror_mode": { 309 | "name": "ipython", 310 | "version": 3 311 | }, 312 | "file_extension": ".py", 313 | "mimetype": "text/x-python", 314 | "name": "python", 315 | "nbconvert_exporter": "python", 316 | "pygments_lexer": "ipython3", 317 | "version": "3.9.6" 318 | } 319 | }, 320 | "nbformat": 4, 321 | "nbformat_minor": 2 322 | } 323 | --------------------------------------------------------------------------------