├── .gitignore ├── L1-Advanced_RAG_Pipeline.ipynb ├── L2-RAG_Triad_of_metrics.ipynb ├── L3-Sentence_window_retrieval.ipynb ├── L4-Auto-merging_Retrieval.ipynb └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | -------------------------------------------------------------------------------- /L1-Advanced_RAG_Pipeline.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "a9bff53e", 6 | "metadata": {}, 7 | "source": [ 8 | "# Lesson 1: Advanced RAG Pipeline" 9 | ] 10 | }, 11 | { 12 | "cell_type": "code", 13 | "execution_count": null, 14 | "id": "feb98470-c136-471d-a63e-d50d8eb09c57", 15 | "metadata": { 16 | "height": 98, 17 | "tags": [] 18 | }, 19 | "outputs": [], 20 | "source": [ 21 | "import utils\n", 22 | "\n", 23 | "import os\n", 24 | "import openai\n", 25 | "openai.api_key = utils.get_openai_api_key()" 26 | ] 27 | }, 28 | { 29 | "cell_type": "code", 30 | "execution_count": null, 31 | "id": "31e2859b-596e-40b3-867b-f4d6e91f74bc", 32 | "metadata": { 33 | "height": 98, 34 | "tags": [] 35 | }, 36 | "outputs": [], 37 | "source": [ 38 | "from llama_index import SimpleDirectoryReader\n", 39 | "\n", 40 | "documents = SimpleDirectoryReader(\n", 41 | " input_files=[\"./eBook-How-to-Build-a-Career-in-AI.pdf\"]\n", 42 | ").load_data()" 43 | ] 44 | }, 45 | { 46 | "cell_type": "code", 47 | "execution_count": null, 48 | "id": "2d7d0857-b9d1-4feb-8243-bfd2f4953acd", 49 | "metadata": { 50 | "height": 81, 51 | "tags": [] 52 | }, 53 | "outputs": [], 54 | "source": [ 55 | "print(type(documents), \"\\n\")\n", 56 | "print(len(documents), \"\\n\")\n", 57 | "print(type(documents[0]))\n", 58 | "print(documents[0])" 59 | ] 60 | }, 61 | { 62 | "cell_type": "markdown", 63 | "id": "f3123d3d", 64 | "metadata": {}, 65 | "source": [ 66 | "## Basic RAG pipeline" 67 | ] 68 | }, 69 | { 70 | "cell_type": "code", 71 | "execution_count": null, 72 | "id": "b4abc806-64f5-46bb-8c9f-6469ecb18d20", 73 | "metadata": { 74 | "height": 64, 75 | "tags": [] 76 | }, 77 | "outputs": [], 78 | "source": [ 79 | "from llama_index import Document\n", 80 | "\n", 81 | "document = Document(text=\"\\n\\n\".join([doc.text for doc in documents]))" 82 | ] 83 | }, 84 | { 85 | "cell_type": "code", 86 | "execution_count": null, 87 | "id": "afc2baff-5e8b-4733-9899-16f248777b23", 88 | "metadata": { 89 | "height": 183, 90 | "tags": [] 91 | }, 92 | "outputs": [], 93 | "source": [ 94 | "from llama_index import VectorStoreIndex\n", 95 | "from llama_index import ServiceContext\n", 96 | "from llama_index.llms import OpenAI\n", 97 | "\n", 98 | "llm = OpenAI(model=\"gpt-3.5-turbo\", temperature=0.1)\n", 99 | "service_context = ServiceContext.from_defaults(\n", 100 | " llm=llm, embed_model=\"local:BAAI/bge-small-en-v1.5\"\n", 101 | ")\n", 102 | "index = VectorStoreIndex.from_documents([document],\n", 103 | " service_context=service_context)" 104 | ] 105 | }, 106 | { 107 | "cell_type": "code", 108 | "execution_count": null, 109 | "id": "ae52a26c-7d0c-44df-8043-4c7f19f794b9", 110 | "metadata": { 111 | "height": 30, 112 | "tags": [] 113 | }, 114 | "outputs": [], 115 | "source": [ 116 | "query_engine = index.as_query_engine()" 117 | ] 118 | }, 119 | { 120 | "cell_type": "code", 121 | "execution_count": null, 122 | "id": "6b0d5b6e-cc2e-4648-b28c-5fa25a97d175", 123 | "metadata": { 124 | "height": 81, 125 | "tags": [] 126 | }, 127 | "outputs": [], 128 | "source": [ 129 | "response = query_engine.query(\n", 130 | " \"What are steps to take when finding projects to build your experience?\"\n", 131 | ")\n", 132 | "print(str(response))" 133 | ] 134 | }, 135 | { 136 | "cell_type": "markdown", 137 | "id": "e50a1ac5", 138 | "metadata": {}, 139 | "source": [ 140 | "## Evaluation setup using TruLens" 141 | ] 142 | }, 143 | { 144 | "cell_type": "code", 145 | "execution_count": null, 146 | "id": "0ead7dc1-71b2-4001-918f-bf8d610fd3fd", 147 | "metadata": { 148 | "height": 132, 149 | "tags": [] 150 | }, 151 | "outputs": [], 152 | "source": [ 153 | "eval_questions = []\n", 154 | "with open('eval_questions.txt', 'r') as file:\n", 155 | " for line in file:\n", 156 | " # Remove newline character and convert to integer\n", 157 | " item = line.strip()\n", 158 | " print(item)\n", 159 | " eval_questions.append(item)" 160 | ] 161 | }, 162 | { 163 | "cell_type": "code", 164 | "execution_count": null, 165 | "id": "87a278f8", 166 | "metadata": { 167 | "height": 64 168 | }, 169 | "outputs": [], 170 | "source": [ 171 | "# You can try your own question:\n", 172 | "new_question = \"What is the right AI job for me?\"\n", 173 | "eval_questions.append(new_question)" 174 | ] 175 | }, 176 | { 177 | "cell_type": "code", 178 | "execution_count": null, 179 | "id": "8d5204e8", 180 | "metadata": { 181 | "height": 30 182 | }, 183 | "outputs": [], 184 | "source": [ 185 | "print(eval_questions)" 186 | ] 187 | }, 188 | { 189 | "cell_type": "code", 190 | "execution_count": null, 191 | "id": "c063c9c7-bf1e-4b24-9a22-d4281c0f954e", 192 | "metadata": { 193 | "height": 81, 194 | "tags": [] 195 | }, 196 | "outputs": [], 197 | "source": [ 198 | "from trulens_eval import Tru\n", 199 | "tru = Tru()\n", 200 | "\n", 201 | "tru.reset_database()" 202 | ] 203 | }, 204 | { 205 | "cell_type": "markdown", 206 | "id": "03802825-6ce4-4563-aeec-d8d57e095ad1", 207 | "metadata": {}, 208 | "source": [ 209 | "For the classroom, we've written some of the code in helper functions inside a utils.py file. \n", 210 | "- You can view the utils.py file in the file directory by clicking on the \"Jupyter\" logo at the top of the notebook.\n", 211 | "- In later lessons, you'll get to work directly with the code that's currently wrapped inside these helper functions, to give you more options to customize your RAG pipeline." 212 | ] 213 | }, 214 | { 215 | "cell_type": "code", 216 | "execution_count": null, 217 | "id": "4f754bed-d16f-4c8d-a1a1-b36096272570", 218 | "metadata": { 219 | "height": 81, 220 | "tags": [] 221 | }, 222 | "outputs": [], 223 | "source": [ 224 | "from utils import get_prebuilt_trulens_recorder\n", 225 | "\n", 226 | "tru_recorder = get_prebuilt_trulens_recorder(query_engine,\n", 227 | " app_id=\"Direct Query Engine\")" 228 | ] 229 | }, 230 | { 231 | "cell_type": "code", 232 | "execution_count": null, 233 | "id": "4dbdfbcc-aac7-4805-9894-4fc016c66bf6", 234 | "metadata": { 235 | "height": 64, 236 | "tags": [] 237 | }, 238 | "outputs": [], 239 | "source": [ 240 | "with tru_recorder as recording:\n", 241 | " for question in eval_questions:\n", 242 | " response = query_engine.query(question)" 243 | ] 244 | }, 245 | { 246 | "cell_type": "code", 247 | "execution_count": null, 248 | "id": "e14f512b-601c-42d0-bfac-bf41d9c577e7", 249 | "metadata": { 250 | "height": 30, 251 | "tags": [] 252 | }, 253 | "outputs": [], 254 | "source": [ 255 | "records, feedback = tru.get_records_and_feedback(app_ids=[])" 256 | ] 257 | }, 258 | { 259 | "cell_type": "code", 260 | "execution_count": null, 261 | "id": "2da4a602-0d56-4bf8-9fa6-03ef0b7e254b", 262 | "metadata": { 263 | "height": 30, 264 | "tags": [] 265 | }, 266 | "outputs": [], 267 | "source": [ 268 | "records.head()" 269 | ] 270 | }, 271 | { 272 | "cell_type": "code", 273 | "execution_count": null, 274 | "id": "64310897-179b-4081-aab8-f08a3392a078", 275 | "metadata": { 276 | "height": 47, 277 | "tags": [] 278 | }, 279 | "outputs": [], 280 | "source": [ 281 | "# launches on http://localhost:8501/\n", 282 | "tru.run_dashboard()" 283 | ] 284 | }, 285 | { 286 | "cell_type": "markdown", 287 | "id": "a4eedcef", 288 | "metadata": {}, 289 | "source": [ 290 | "## Advanced RAG pipeline" 291 | ] 292 | }, 293 | { 294 | "cell_type": "markdown", 295 | "id": "9a17ea2b", 296 | "metadata": {}, 297 | "source": [ 298 | "### 1. Sentence Window retrieval" 299 | ] 300 | }, 301 | { 302 | "cell_type": "code", 303 | "execution_count": null, 304 | "id": "dae4a668-3699-4750-82f7-e53ae1bca3a7", 305 | "metadata": { 306 | "height": 64, 307 | "tags": [] 308 | }, 309 | "outputs": [], 310 | "source": [ 311 | "from llama_index.llms import OpenAI\n", 312 | "\n", 313 | "llm = OpenAI(model=\"gpt-3.5-turbo\", temperature=0.1)" 314 | ] 315 | }, 316 | { 317 | "cell_type": "code", 318 | "execution_count": null, 319 | "id": "78f7678f-358d-448d-b153-11ac8e96a7fc", 320 | "metadata": { 321 | "height": 149, 322 | "tags": [] 323 | }, 324 | "outputs": [], 325 | "source": [ 326 | "from utils import build_sentence_window_index\n", 327 | "\n", 328 | "sentence_index = build_sentence_window_index(\n", 329 | " document,\n", 330 | " llm,\n", 331 | " embed_model=\"local:BAAI/bge-small-en-v1.5\",\n", 332 | " save_dir=\"sentence_index\"\n", 333 | ")" 334 | ] 335 | }, 336 | { 337 | "cell_type": "code", 338 | "execution_count": null, 339 | "id": "72f904c3-9845-4df5-9d2e-e5115160f987", 340 | "metadata": { 341 | "height": 64, 342 | "tags": [] 343 | }, 344 | "outputs": [], 345 | "source": [ 346 | "from utils import get_sentence_window_query_engine\n", 347 | "\n", 348 | "sentence_window_engine = get_sentence_window_query_engine(sentence_index)" 349 | ] 350 | }, 351 | { 352 | "cell_type": "code", 353 | "execution_count": null, 354 | "id": "f59e2314-7cac-42f4-a552-9a8e4db641eb", 355 | "metadata": { 356 | "height": 81, 357 | "tags": [] 358 | }, 359 | "outputs": [], 360 | "source": [ 361 | "window_response = sentence_window_engine.query(\n", 362 | " \"how do I get started on a personal project in AI?\"\n", 363 | ")\n", 364 | "print(str(window_response))" 365 | ] 366 | }, 367 | { 368 | "cell_type": "code", 369 | "execution_count": null, 370 | "id": "b5c10917-8846-4e73-838d-6232c906a7db", 371 | "metadata": { 372 | "height": 115, 373 | "tags": [] 374 | }, 375 | "outputs": [], 376 | "source": [ 377 | "tru.reset_database()\n", 378 | "\n", 379 | "tru_recorder_sentence_window = get_prebuilt_trulens_recorder(\n", 380 | " sentence_window_engine,\n", 381 | " app_id = \"Sentence Window Query Engine\"\n", 382 | ")" 383 | ] 384 | }, 385 | { 386 | "cell_type": "code", 387 | "execution_count": null, 388 | "id": "11710e67-aba8-479e-8585-c4c611e2c1d2", 389 | "metadata": { 390 | "height": 98, 391 | "tags": [] 392 | }, 393 | "outputs": [], 394 | "source": [ 395 | "for question in eval_questions:\n", 396 | " with tru_recorder_sentence_window as recording:\n", 397 | " response = sentence_window_engine.query(question)\n", 398 | " print(question)\n", 399 | " print(str(response))" 400 | ] 401 | }, 402 | { 403 | "cell_type": "code", 404 | "execution_count": null, 405 | "id": "869d1e55-729b-45f2-a0f9-773c49d4616f", 406 | "metadata": { 407 | "height": 30, 408 | "tags": [] 409 | }, 410 | "outputs": [], 411 | "source": [ 412 | "tru.get_leaderboard(app_ids=[])" 413 | ] 414 | }, 415 | { 416 | "cell_type": "code", 417 | "execution_count": null, 418 | "id": "0b92d0f2-2e80-48d5-92af-b3655eb03ea2", 419 | "metadata": { 420 | "height": 47, 421 | "tags": [] 422 | }, 423 | "outputs": [], 424 | "source": [ 425 | "# launches on http://localhost:8501/\n", 426 | "tru.run_dashboard()" 427 | ] 428 | }, 429 | { 430 | "cell_type": "markdown", 431 | "id": "a92e2c55", 432 | "metadata": {}, 433 | "source": [ 434 | "### 2. Auto-merging retrieval" 435 | ] 436 | }, 437 | { 438 | "cell_type": "code", 439 | "execution_count": null, 440 | "id": "558c639b-31eb-4c34-b6c4-fe6ae5717733", 441 | "metadata": { 442 | "height": 149, 443 | "tags": [] 444 | }, 445 | "outputs": [], 446 | "source": [ 447 | "from utils import build_automerging_index\n", 448 | "\n", 449 | "automerging_index = build_automerging_index(\n", 450 | " documents,\n", 451 | " llm,\n", 452 | " embed_model=\"local:BAAI/bge-small-en-v1.5\",\n", 453 | " save_dir=\"merging_index\"\n", 454 | ")" 455 | ] 456 | }, 457 | { 458 | "cell_type": "code", 459 | "execution_count": null, 460 | "id": "b32265f2-0247-42df-9abe-97d52f69edcf", 461 | "metadata": { 462 | "height": 98, 463 | "tags": [] 464 | }, 465 | "outputs": [], 466 | "source": [ 467 | "from utils import get_automerging_query_engine\n", 468 | "\n", 469 | "automerging_query_engine = get_automerging_query_engine(\n", 470 | " automerging_index,\n", 471 | ")" 472 | ] 473 | }, 474 | { 475 | "cell_type": "code", 476 | "execution_count": null, 477 | "id": "918ed568-220e-4c7c-aa60-cfa58ef1fcbd", 478 | "metadata": { 479 | "height": 81, 480 | "tags": [] 481 | }, 482 | "outputs": [], 483 | "source": [ 484 | "auto_merging_response = automerging_query_engine.query(\n", 485 | " \"How do I build a portfolio of AI projects?\"\n", 486 | ")\n", 487 | "print(str(auto_merging_response))" 488 | ] 489 | }, 490 | { 491 | "cell_type": "code", 492 | "execution_count": null, 493 | "id": "db4f18a9-7b8a-4ae2-ab11-3a6a941a5afc", 494 | "metadata": { 495 | "height": 81, 496 | "tags": [] 497 | }, 498 | "outputs": [], 499 | "source": [ 500 | "tru.reset_database()\n", 501 | "\n", 502 | "tru_recorder_automerging = get_prebuilt_trulens_recorder(automerging_query_engine,\n", 503 | " app_id=\"Automerging Query Engine\")" 504 | ] 505 | }, 506 | { 507 | "cell_type": "code", 508 | "execution_count": null, 509 | "id": "99cc2cfe-7096-4fa0-aa72-094bebac35a3", 510 | "metadata": { 511 | "height": 98, 512 | "tags": [] 513 | }, 514 | "outputs": [], 515 | "source": [ 516 | "for question in eval_questions:\n", 517 | " with tru_recorder_automerging as recording:\n", 518 | " response = automerging_query_engine.query(question)\n", 519 | " print(question)\n", 520 | " print(response)" 521 | ] 522 | }, 523 | { 524 | "cell_type": "code", 525 | "execution_count": null, 526 | "id": "5404dec1-60ca-42fa-ac13-793a5423aa64", 527 | "metadata": { 528 | "height": 30, 529 | "tags": [] 530 | }, 531 | "outputs": [], 532 | "source": [ 533 | "tru.get_leaderboard(app_ids=[])" 534 | ] 535 | }, 536 | { 537 | "cell_type": "code", 538 | "execution_count": null, 539 | "id": "0f545d41-0d98-446f-8214-8b59bef08d6c", 540 | "metadata": { 541 | "height": 47, 542 | "tags": [] 543 | }, 544 | "outputs": [], 545 | "source": [ 546 | "# launches on http://localhost:8501/\n", 547 | "tru.run_dashboard()" 548 | ] 549 | } 550 | ], 551 | "metadata": { 552 | "kernelspec": { 553 | "display_name": "Python 3 (ipykernel)", 554 | "language": "python", 555 | "name": "python3" 556 | }, 557 | "language_info": { 558 | "codemirror_mode": { 559 | "name": "ipython", 560 | "version": 3 561 | }, 562 | "file_extension": ".py", 563 | "mimetype": "text/x-python", 564 | "name": "python", 565 | "nbconvert_exporter": "python", 566 | "pygments_lexer": "ipython3", 567 | "version": "3.10.13" 568 | } 569 | }, 570 | "nbformat": 4, 571 | "nbformat_minor": 5 572 | } 573 | -------------------------------------------------------------------------------- /L2-RAG_Triad_of_metrics.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "id": "X6-q-gTUaZU7", 7 | "tags": [] 8 | }, 9 | "source": [ 10 | "# Lesson 2: RAG Triad of metrics" 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": null, 16 | "metadata": { 17 | "height": 47 18 | }, 19 | "outputs": [], 20 | "source": [ 21 | "import warnings\n", 22 | "warnings.filterwarnings('ignore')" 23 | ] 24 | }, 25 | { 26 | "cell_type": "code", 27 | "execution_count": null, 28 | "metadata": { 29 | "height": 98 30 | }, 31 | "outputs": [], 32 | "source": [ 33 | "import utils\n", 34 | "\n", 35 | "import os\n", 36 | "import openai\n", 37 | "openai.api_key = utils.get_openai_api_key()" 38 | ] 39 | }, 40 | { 41 | "cell_type": "code", 42 | "execution_count": null, 43 | "metadata": { 44 | "height": 81, 45 | "id": "IBfdyn3MaZU9" 46 | }, 47 | "outputs": [], 48 | "source": [ 49 | "from trulens_eval import Tru\n", 50 | "\n", 51 | "tru = Tru()\n", 52 | "tru.reset_database()" 53 | ] 54 | }, 55 | { 56 | "cell_type": "code", 57 | "execution_count": null, 58 | "metadata": { 59 | "height": 98, 60 | "id": "wMvq1q8yaZU-" 61 | }, 62 | "outputs": [], 63 | "source": [ 64 | "from llama_index import SimpleDirectoryReader\n", 65 | "\n", 66 | "documents = SimpleDirectoryReader(\n", 67 | " input_files=[\"./eBook-How-to-Build-a-Career-in-AI.pdf\"]\n", 68 | ").load_data()" 69 | ] 70 | }, 71 | { 72 | "cell_type": "code", 73 | "execution_count": null, 74 | "metadata": { 75 | "height": 81, 76 | "id": "sY8Oui4taZU-" 77 | }, 78 | "outputs": [], 79 | "source": [ 80 | "from llama_index import Document\n", 81 | "\n", 82 | "document = Document(text=\"\\n\\n\".\\\n", 83 | " join([doc.text for doc in documents]))" 84 | ] 85 | }, 86 | { 87 | "cell_type": "code", 88 | "execution_count": null, 89 | "metadata": { 90 | "height": 217 91 | }, 92 | "outputs": [], 93 | "source": [ 94 | "from utils import build_sentence_window_index\n", 95 | "\n", 96 | "from llama_index.llms import OpenAI\n", 97 | "\n", 98 | "llm = OpenAI(model=\"gpt-3.5-turbo\", temperature=0.1)\n", 99 | "\n", 100 | "sentence_index = build_sentence_window_index(\n", 101 | " document,\n", 102 | " llm,\n", 103 | " embed_model=\"local:BAAI/bge-small-en-v1.5\",\n", 104 | " save_dir=\"sentence_index\"\n", 105 | ")" 106 | ] 107 | }, 108 | { 109 | "cell_type": "code", 110 | "execution_count": null, 111 | "metadata": { 112 | "height": 81 113 | }, 114 | "outputs": [], 115 | "source": [ 116 | "from utils import get_sentence_window_query_engine\n", 117 | "\n", 118 | "sentence_window_engine = \\\n", 119 | "get_sentence_window_query_engine(sentence_index)" 120 | ] 121 | }, 122 | { 123 | "cell_type": "code", 124 | "execution_count": null, 125 | "metadata": { 126 | "height": 64 127 | }, 128 | "outputs": [], 129 | "source": [ 130 | "output = sentence_window_engine.query(\n", 131 | " \"How do you create your AI portfolio?\")\n", 132 | "output.response" 133 | ] 134 | }, 135 | { 136 | "cell_type": "markdown", 137 | "metadata": {}, 138 | "source": [ 139 | "## Feedback functions" 140 | ] 141 | }, 142 | { 143 | "cell_type": "code", 144 | "execution_count": null, 145 | "metadata": { 146 | "height": 64, 147 | "id": "5KqV-IbQaZVB" 148 | }, 149 | "outputs": [], 150 | "source": [ 151 | "import nest_asyncio\n", 152 | "\n", 153 | "nest_asyncio.apply()" 154 | ] 155 | }, 156 | { 157 | "cell_type": "code", 158 | "execution_count": null, 159 | "metadata": { 160 | "height": 64 161 | }, 162 | "outputs": [], 163 | "source": [ 164 | "from trulens_eval import OpenAI as fOpenAI\n", 165 | "\n", 166 | "provider = fOpenAI()" 167 | ] 168 | }, 169 | { 170 | "cell_type": "markdown", 171 | "metadata": {}, 172 | "source": [ 173 | "### 1. Answer Relevance" 174 | ] 175 | }, 176 | { 177 | "cell_type": "code", 178 | "execution_count": null, 179 | "metadata": { 180 | "height": 115 181 | }, 182 | "outputs": [], 183 | "source": [ 184 | "from trulens_eval import Feedback\n", 185 | "\n", 186 | "f_qa_relevance = Feedback(\n", 187 | " provider.relevance_with_cot_reasons,\n", 188 | " name=\"Answer Relevance\"\n", 189 | ").on_input_output()" 190 | ] 191 | }, 192 | { 193 | "cell_type": "markdown", 194 | "metadata": {}, 195 | "source": [ 196 | "### 2. Context Relevance" 197 | ] 198 | }, 199 | { 200 | "cell_type": "code", 201 | "execution_count": null, 202 | "metadata": { 203 | "height": 64 204 | }, 205 | "outputs": [], 206 | "source": [ 207 | "from trulens_eval import TruLlama\n", 208 | "\n", 209 | "context_selection = TruLlama.select_source_nodes().node.text" 210 | ] 211 | }, 212 | { 213 | "cell_type": "code", 214 | "execution_count": null, 215 | "metadata": { 216 | "height": 166 217 | }, 218 | "outputs": [], 219 | "source": [ 220 | "import numpy as np\n", 221 | "\n", 222 | "f_qs_relevance = (\n", 223 | " Feedback(provider.qs_relevance,\n", 224 | " name=\"Context Relevance\")\n", 225 | " .on_input()\n", 226 | " .on(context_selection)\n", 227 | " .aggregate(np.mean)\n", 228 | ")" 229 | ] 230 | }, 231 | { 232 | "cell_type": "code", 233 | "execution_count": null, 234 | "metadata": { 235 | "height": 166 236 | }, 237 | "outputs": [], 238 | "source": [ 239 | "import numpy as np\n", 240 | "\n", 241 | "f_qs_relevance = (\n", 242 | " Feedback(provider.qs_relevance_with_cot_reasons,\n", 243 | " name=\"Context Relevance\")\n", 244 | " .on_input()\n", 245 | " .on(context_selection)\n", 246 | " .aggregate(np.mean)\n", 247 | ")" 248 | ] 249 | }, 250 | { 251 | "cell_type": "markdown", 252 | "metadata": {}, 253 | "source": [ 254 | "### 3. Groundedness" 255 | ] 256 | }, 257 | { 258 | "cell_type": "code", 259 | "execution_count": null, 260 | "metadata": { 261 | "height": 64 262 | }, 263 | "outputs": [], 264 | "source": [ 265 | "from trulens_eval.feedback import Groundedness\n", 266 | "\n", 267 | "grounded = Groundedness(groundedness_provider=provider)" 268 | ] 269 | }, 270 | { 271 | "cell_type": "code", 272 | "execution_count": null, 273 | "metadata": { 274 | "height": 149, 275 | "id": "kXJBD4gfaZVC" 276 | }, 277 | "outputs": [], 278 | "source": [ 279 | "f_groundedness = (\n", 280 | " Feedback(grounded.groundedness_measure_with_cot_reasons,\n", 281 | " name=\"Groundedness\"\n", 282 | " )\n", 283 | " .on(context_selection)\n", 284 | " .on_output()\n", 285 | " .aggregate(grounded.grounded_statements_aggregator)\n", 286 | ")" 287 | ] 288 | }, 289 | { 290 | "cell_type": "markdown", 291 | "metadata": {}, 292 | "source": [ 293 | "## Evaluation of the RAG application" 294 | ] 295 | }, 296 | { 297 | "cell_type": "code", 298 | "execution_count": null, 299 | "metadata": { 300 | "height": 217, 301 | "id": "KUDHInR-aZVC" 302 | }, 303 | "outputs": [], 304 | "source": [ 305 | "from trulens_eval import TruLlama\n", 306 | "from trulens_eval import FeedbackMode\n", 307 | "\n", 308 | "tru_recorder = TruLlama(\n", 309 | " sentence_window_engine,\n", 310 | " app_id=\"App_1\",\n", 311 | " feedbacks=[\n", 312 | " f_qa_relevance,\n", 313 | " f_qs_relevance,\n", 314 | " f_groundedness\n", 315 | " ]\n", 316 | ")" 317 | ] 318 | }, 319 | { 320 | "cell_type": "code", 321 | "execution_count": null, 322 | "metadata": { 323 | "height": 115, 324 | "id": "dsA3ziw1aZVD" 325 | }, 326 | "outputs": [], 327 | "source": [ 328 | "eval_questions = []\n", 329 | "with open('eval_questions.txt', 'r') as file:\n", 330 | " for line in file:\n", 331 | " # Remove newline character and convert to integer\n", 332 | " item = line.strip()\n", 333 | " eval_questions.append(item)" 334 | ] 335 | }, 336 | { 337 | "cell_type": "code", 338 | "execution_count": null, 339 | "metadata": { 340 | "height": 30 341 | }, 342 | "outputs": [], 343 | "source": [ 344 | "eval_questions" 345 | ] 346 | }, 347 | { 348 | "cell_type": "code", 349 | "execution_count": null, 350 | "metadata": { 351 | "height": 30 352 | }, 353 | "outputs": [], 354 | "source": [ 355 | "eval_questions.append(\"How can I be successful in AI?\")" 356 | ] 357 | }, 358 | { 359 | "cell_type": "code", 360 | "execution_count": null, 361 | "metadata": { 362 | "height": 30 363 | }, 364 | "outputs": [], 365 | "source": [ 366 | "eval_questions" 367 | ] 368 | }, 369 | { 370 | "cell_type": "code", 371 | "execution_count": null, 372 | "metadata": { 373 | "height": 64, 374 | "id": "01_P6TxaaZVD" 375 | }, 376 | "outputs": [], 377 | "source": [ 378 | "for question in eval_questions:\n", 379 | " with tru_recorder as recording:\n", 380 | " sentence_window_engine.query(question)" 381 | ] 382 | }, 383 | { 384 | "cell_type": "code", 385 | "execution_count": null, 386 | "metadata": { 387 | "height": 47, 388 | "id": "sNPhDde6ZArq" 389 | }, 390 | "outputs": [], 391 | "source": [ 392 | "records, feedback = tru.get_records_and_feedback(app_ids=[])\n", 393 | "records.head()" 394 | ] 395 | }, 396 | { 397 | "cell_type": "code", 398 | "execution_count": null, 399 | "metadata": { 400 | "height": 81 401 | }, 402 | "outputs": [], 403 | "source": [ 404 | "import pandas as pd\n", 405 | "\n", 406 | "pd.set_option(\"display.max_colwidth\", None)\n", 407 | "records[[\"input\", \"output\"] + feedback]" 408 | ] 409 | }, 410 | { 411 | "cell_type": "code", 412 | "execution_count": null, 413 | "metadata": { 414 | "height": 30 415 | }, 416 | "outputs": [], 417 | "source": [ 418 | "tru.get_leaderboard(app_ids=[])" 419 | ] 420 | }, 421 | { 422 | "cell_type": "code", 423 | "execution_count": null, 424 | "metadata": { 425 | "height": 30, 426 | "id": "6Yp4_e4faZVD" 427 | }, 428 | "outputs": [], 429 | "source": [ 430 | "tru.run_dashboard()" 431 | ] 432 | } 433 | ], 434 | "metadata": { 435 | "accelerator": "GPU", 436 | "colab": { 437 | "gpuType": "T4", 438 | "provenance": [ 439 | { 440 | "file_id": "1ChKW1kEIUcUVDDTWjpA5Tf_ib3Hhp3uS", 441 | "timestamp": 1695164681916 442 | } 443 | ] 444 | }, 445 | "kernelspec": { 446 | "display_name": "Python 3 (ipykernel)", 447 | "language": "python", 448 | "name": "python3" 449 | }, 450 | "language_info": { 451 | "codemirror_mode": { 452 | "name": "ipython", 453 | "version": 3 454 | }, 455 | "file_extension": ".py", 456 | "mimetype": "text/x-python", 457 | "name": "python", 458 | "nbconvert_exporter": "python", 459 | "pygments_lexer": "ipython3", 460 | "version": "3.10.13" 461 | } 462 | }, 463 | "nbformat": 4, 464 | "nbformat_minor": 4 465 | } 466 | -------------------------------------------------------------------------------- /L3-Sentence_window_retrieval.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "21f9146a-c6f3-4a78-b3fb-0d262492e87c", 6 | "metadata": {}, 7 | "source": [ 8 | "# Lesson 3: Sentence Window Retrieval" 9 | ] 10 | }, 11 | { 12 | "cell_type": "code", 13 | "execution_count": null, 14 | "id": "541cadae-916c-42da-93ff-75d7f788ee8d", 15 | "metadata": {}, 16 | "outputs": [], 17 | "source": [ 18 | "import warnings\n", 19 | "warnings.filterwarnings('ignore')" 20 | ] 21 | }, 22 | { 23 | "cell_type": "code", 24 | "execution_count": null, 25 | "id": "78f48098-16d8-4209-b722-1ec6a0220c96", 26 | "metadata": {}, 27 | "outputs": [], 28 | "source": [ 29 | "import utils\n", 30 | "\n", 31 | "import os\n", 32 | "import openai\n", 33 | "openai.api_key = utils.get_openai_api_key()" 34 | ] 35 | }, 36 | { 37 | "cell_type": "code", 38 | "execution_count": null, 39 | "id": "9ee55360-1fc4-41cc-bd49-82027797ea40", 40 | "metadata": { 41 | "tags": [] 42 | }, 43 | "outputs": [], 44 | "source": [ 45 | "from llama_index import SimpleDirectoryReader\n", 46 | "\n", 47 | "documents = SimpleDirectoryReader(\n", 48 | " input_files=[\"./eBook-How-to-Build-a-Career-in-AI.pdf\"]\n", 49 | ").load_data()" 50 | ] 51 | }, 52 | { 53 | "cell_type": "code", 54 | "execution_count": null, 55 | "id": "a12ada81-5c1c-47c9-b7b4-ba621a80bbcd", 56 | "metadata": { 57 | "tags": [] 58 | }, 59 | "outputs": [], 60 | "source": [ 61 | "print(type(documents), \"\\n\")\n", 62 | "print(len(documents), \"\\n\")\n", 63 | "print(type(documents[0]))\n", 64 | "print(documents[0])" 65 | ] 66 | }, 67 | { 68 | "cell_type": "code", 69 | "execution_count": null, 70 | "id": "f2f662e4-3fb8-40c6-acc5-e8510348d113", 71 | "metadata": { 72 | "tags": [] 73 | }, 74 | "outputs": [], 75 | "source": [ 76 | "from llama_index import Document\n", 77 | "\n", 78 | "document = Document(text=\"\\n\\n\".join([doc.text for doc in documents]))" 79 | ] 80 | }, 81 | { 82 | "cell_type": "markdown", 83 | "id": "46ff01ea-b5b0-4e65-8565-b2444812bd84", 84 | "metadata": {}, 85 | "source": [ 86 | "## Window-sentence retrieval setup" 87 | ] 88 | }, 89 | { 90 | "cell_type": "code", 91 | "execution_count": null, 92 | "id": "6a194b93-f975-4d38-babe-218d3aae6117", 93 | "metadata": { 94 | "tags": [] 95 | }, 96 | "outputs": [], 97 | "source": [ 98 | "from llama_index.node_parser import SentenceWindowNodeParser\n", 99 | "\n", 100 | "# create the sentence window node parser w/ default settings\n", 101 | "node_parser = SentenceWindowNodeParser.from_defaults(\n", 102 | " window_size=3,\n", 103 | " window_metadata_key=\"window\",\n", 104 | " original_text_metadata_key=\"original_text\",\n", 105 | ")" 106 | ] 107 | }, 108 | { 109 | "cell_type": "code", 110 | "execution_count": null, 111 | "id": "88973de6-be8f-4653-aca3-8d0e884d9470", 112 | "metadata": { 113 | "tags": [] 114 | }, 115 | "outputs": [], 116 | "source": [ 117 | "text = \"hello. how are you? I am fine! \"\n", 118 | "\n", 119 | "nodes = node_parser.get_nodes_from_documents([Document(text=text)])" 120 | ] 121 | }, 122 | { 123 | "cell_type": "code", 124 | "execution_count": null, 125 | "id": "860e40a8-6f50-4345-b314-c4d9349681c6", 126 | "metadata": { 127 | "tags": [] 128 | }, 129 | "outputs": [], 130 | "source": [ 131 | "print([x.text for x in nodes])" 132 | ] 133 | }, 134 | { 135 | "cell_type": "code", 136 | "execution_count": null, 137 | "id": "b4334e02-c423-4555-8446-4794eadccd0a", 138 | "metadata": { 139 | "tags": [] 140 | }, 141 | "outputs": [], 142 | "source": [ 143 | "print(nodes[1].metadata[\"window\"])" 144 | ] 145 | }, 146 | { 147 | "cell_type": "code", 148 | "execution_count": null, 149 | "id": "03fea59c-ed0f-4cc8-87b4-871798edb094", 150 | "metadata": { 151 | "tags": [] 152 | }, 153 | "outputs": [], 154 | "source": [ 155 | "text = \"hello. foo bar. cat dog. mouse\"\n", 156 | "\n", 157 | "nodes = node_parser.get_nodes_from_documents([Document(text=text)])" 158 | ] 159 | }, 160 | { 161 | "cell_type": "code", 162 | "execution_count": null, 163 | "id": "c329808d-248f-422e-bce0-1ac1ecba79a5", 164 | "metadata": { 165 | "tags": [] 166 | }, 167 | "outputs": [], 168 | "source": [ 169 | "print([x.text for x in nodes])" 170 | ] 171 | }, 172 | { 173 | "cell_type": "code", 174 | "execution_count": null, 175 | "id": "cfd1eccb-823d-4f6f-8d6c-a9064dc76922", 176 | "metadata": { 177 | "tags": [] 178 | }, 179 | "outputs": [], 180 | "source": [ 181 | "print(nodes[0].metadata[\"window\"])" 182 | ] 183 | }, 184 | { 185 | "cell_type": "markdown", 186 | "id": "fea97dda-8457-4f32-a2c7-26ae92eaf0b4", 187 | "metadata": {}, 188 | "source": [ 189 | "### Building the index" 190 | ] 191 | }, 192 | { 193 | "cell_type": "code", 194 | "execution_count": null, 195 | "id": "bb2387dd-6bdb-4d0d-98ea-2b468ddfe160", 196 | "metadata": { 197 | "tags": [] 198 | }, 199 | "outputs": [], 200 | "source": [ 201 | "from llama_index.llms import OpenAI\n", 202 | "\n", 203 | "llm = OpenAI(model=\"gpt-3.5-turbo\", temperature=0.1)" 204 | ] 205 | }, 206 | { 207 | "cell_type": "code", 208 | "execution_count": null, 209 | "id": "5102d65b-3584-4a25-88be-7d5dbc70c678", 210 | "metadata": { 211 | "tags": [] 212 | }, 213 | "outputs": [], 214 | "source": [ 215 | "from llama_index import ServiceContext\n", 216 | "\n", 217 | "sentence_context = ServiceContext.from_defaults(\n", 218 | " llm=llm,\n", 219 | " embed_model=\"local:BAAI/bge-small-en-v1.5\",\n", 220 | " # embed_model=\"local:BAAI/bge-large-en-v1.5\"\n", 221 | " node_parser=node_parser,\n", 222 | ")" 223 | ] 224 | }, 225 | { 226 | "cell_type": "code", 227 | "execution_count": null, 228 | "id": "fbf15398-25e9-47bd-b840-e57272d38683", 229 | "metadata": { 230 | "tags": [] 231 | }, 232 | "outputs": [], 233 | "source": [ 234 | "from llama_index import VectorStoreIndex\n", 235 | "\n", 236 | "sentence_index = VectorStoreIndex.from_documents(\n", 237 | " [document], service_context=sentence_context\n", 238 | ")" 239 | ] 240 | }, 241 | { 242 | "cell_type": "code", 243 | "execution_count": null, 244 | "id": "55497194-35c9-4f91-85c4-ef6adb1aff78", 245 | "metadata": { 246 | "tags": [] 247 | }, 248 | "outputs": [], 249 | "source": [ 250 | "sentence_index.storage_context.persist(persist_dir=\"./sentence_index\")\n" 251 | ] 252 | }, 253 | { 254 | "cell_type": "code", 255 | "execution_count": null, 256 | "id": "ba4e7371-7a8e-451c-a5e7-e9e3d7371614", 257 | "metadata": { 258 | "tags": [] 259 | }, 260 | "outputs": [], 261 | "source": [ 262 | "# This block of code is optional to check\n", 263 | "# if an index file exist, then it will load it\n", 264 | "# if not, it will rebuild it\n", 265 | "\n", 266 | "import os\n", 267 | "from llama_index import VectorStoreIndex, StorageContext, load_index_from_storage\n", 268 | "from llama_index import load_index_from_storage\n", 269 | "\n", 270 | "if not os.path.exists(\"./sentence_index\"):\n", 271 | " sentence_index = VectorStoreIndex.from_documents(\n", 272 | " [document], service_context=sentence_context\n", 273 | " )\n", 274 | "\n", 275 | " sentence_index.storage_context.persist(persist_dir=\"./sentence_index\")\n", 276 | "else:\n", 277 | " sentence_index = load_index_from_storage(\n", 278 | " StorageContext.from_defaults(persist_dir=\"./sentence_index\"),\n", 279 | " service_context=sentence_context\n", 280 | " )" 281 | ] 282 | }, 283 | { 284 | "cell_type": "markdown", 285 | "id": "6b934879-e2c3-44f8-b98c-0300dc6389a9", 286 | "metadata": {}, 287 | "source": [ 288 | "### Building the postprocessor" 289 | ] 290 | }, 291 | { 292 | "cell_type": "code", 293 | "execution_count": null, 294 | "id": "9f22f435-a90a-447c-9e63-8aebec1e968d", 295 | "metadata": { 296 | "tags": [] 297 | }, 298 | "outputs": [], 299 | "source": [ 300 | "from llama_index.indices.postprocessor import MetadataReplacementPostProcessor\n", 301 | "\n", 302 | "postproc = MetadataReplacementPostProcessor(\n", 303 | " target_metadata_key=\"window\"\n", 304 | ")" 305 | ] 306 | }, 307 | { 308 | "cell_type": "code", 309 | "execution_count": null, 310 | "id": "77fad243-7ea8-4a81-b85e-91c3e638d932", 311 | "metadata": { 312 | "tags": [] 313 | }, 314 | "outputs": [], 315 | "source": [ 316 | "from llama_index.schema import NodeWithScore\n", 317 | "from copy import deepcopy\n", 318 | "\n", 319 | "scored_nodes = [NodeWithScore(node=x, score=1.0) for x in nodes]\n", 320 | "nodes_old = [deepcopy(n) for n in nodes]" 321 | ] 322 | }, 323 | { 324 | "cell_type": "code", 325 | "execution_count": null, 326 | "id": "946c4219-1d6e-4717-8c8f-40db659ea517", 327 | "metadata": { 328 | "tags": [] 329 | }, 330 | "outputs": [], 331 | "source": [ 332 | "nodes_old[1].text" 333 | ] 334 | }, 335 | { 336 | "cell_type": "code", 337 | "execution_count": null, 338 | "id": "89900cb4-5d03-43f6-9d1d-de1350bfd299", 339 | "metadata": { 340 | "tags": [] 341 | }, 342 | "outputs": [], 343 | "source": [ 344 | "replaced_nodes = postproc.postprocess_nodes(scored_nodes)" 345 | ] 346 | }, 347 | { 348 | "cell_type": "code", 349 | "execution_count": null, 350 | "id": "72f98170-e368-461b-9939-5da80c4940e4", 351 | "metadata": { 352 | "tags": [] 353 | }, 354 | "outputs": [], 355 | "source": [ 356 | "print(replaced_nodes[1].text)" 357 | ] 358 | }, 359 | { 360 | "cell_type": "markdown", 361 | "id": "72aa4a50-5c0b-4f23-9eae-ec67cb701e29", 362 | "metadata": {}, 363 | "source": [ 364 | "### Adding a reranker" 365 | ] 366 | }, 367 | { 368 | "cell_type": "code", 369 | "execution_count": null, 370 | "id": "57608e01-ee96-4619-aab1-81d7fce1cbd1", 371 | "metadata": { 372 | "tags": [] 373 | }, 374 | "outputs": [], 375 | "source": [ 376 | "from llama_index.indices.postprocessor import SentenceTransformerRerank\n", 377 | "\n", 378 | "# BAAI/bge-reranker-base\n", 379 | "# link: https://huggingface.co/BAAI/bge-reranker-base\n", 380 | "rerank = SentenceTransformerRerank(\n", 381 | " top_n=2, model=\"BAAI/bge-reranker-base\"\n", 382 | ")" 383 | ] 384 | }, 385 | { 386 | "cell_type": "code", 387 | "execution_count": null, 388 | "id": "4d89da07-6b0c-477f-bd0a-2f466c9b159b", 389 | "metadata": { 390 | "tags": [] 391 | }, 392 | "outputs": [], 393 | "source": [ 394 | "from llama_index import QueryBundle\n", 395 | "from llama_index.schema import TextNode, NodeWithScore\n", 396 | "\n", 397 | "query = QueryBundle(\"I want a dog.\")\n", 398 | "\n", 399 | "scored_nodes = [\n", 400 | " NodeWithScore(node=TextNode(text=\"This is a cat\"), score=0.6),\n", 401 | " NodeWithScore(node=TextNode(text=\"This is a dog\"), score=0.4),\n", 402 | "]" 403 | ] 404 | }, 405 | { 406 | "cell_type": "code", 407 | "execution_count": null, 408 | "id": "1adc6cb4-e741-480a-ab13-06e9194b13b2", 409 | "metadata": { 410 | "tags": [] 411 | }, 412 | "outputs": [], 413 | "source": [ 414 | "reranked_nodes = rerank.postprocess_nodes(\n", 415 | " scored_nodes, query_bundle=query\n", 416 | ")" 417 | ] 418 | }, 419 | { 420 | "cell_type": "code", 421 | "execution_count": null, 422 | "id": "3be841f6-0ab2-4c60-b52f-d27f81fcb1bb", 423 | "metadata": { 424 | "tags": [] 425 | }, 426 | "outputs": [], 427 | "source": [ 428 | "print([(x.text, x.score) for x in reranked_nodes])" 429 | ] 430 | }, 431 | { 432 | "cell_type": "markdown", 433 | "id": "74d51921-4b0b-4d89-963f-9a9e0ea439d9", 434 | "metadata": {}, 435 | "source": [ 436 | "### Runing the query engine" 437 | ] 438 | }, 439 | { 440 | "cell_type": "code", 441 | "execution_count": null, 442 | "id": "47bfe5a5-5ce1-4baa-ba61-3b39d16a2337", 443 | "metadata": { 444 | "tags": [] 445 | }, 446 | "outputs": [], 447 | "source": [ 448 | "sentence_window_engine = sentence_index.as_query_engine(\n", 449 | " similarity_top_k=6, node_postprocessors=[postproc, rerank]\n", 450 | ")" 451 | ] 452 | }, 453 | { 454 | "cell_type": "code", 455 | "execution_count": null, 456 | "id": "c4f166b9-99f9-4507-af59-3347eaf596c6", 457 | "metadata": { 458 | "tags": [] 459 | }, 460 | "outputs": [], 461 | "source": [ 462 | "window_response = sentence_window_engine.query(\n", 463 | " \"What are the keys to building a career in AI?\"\n", 464 | ")" 465 | ] 466 | }, 467 | { 468 | "cell_type": "code", 469 | "execution_count": null, 470 | "id": "ada6304b-a86c-4f6c-a2e9-8ac5b1091a58", 471 | "metadata": {}, 472 | "outputs": [], 473 | "source": [ 474 | "from llama_index.response.notebook_utils import display_response\n", 475 | "\n", 476 | "display_response(window_response)" 477 | ] 478 | }, 479 | { 480 | "cell_type": "markdown", 481 | "id": "cbcbe614-befe-4bf7-9813-2c91899650e9", 482 | "metadata": {}, 483 | "source": [ 484 | "## Putting it all Together" 485 | ] 486 | }, 487 | { 488 | "cell_type": "code", 489 | "execution_count": null, 490 | "id": "313e8b1a-e9e7-4141-a4d2-72fb31a7e057", 491 | "metadata": {}, 492 | "outputs": [], 493 | "source": [ 494 | "import os\n", 495 | "from llama_index import ServiceContext, VectorStoreIndex, StorageContext\n", 496 | "from llama_index.node_parser import SentenceWindowNodeParser\n", 497 | "from llama_index.indices.postprocessor import MetadataReplacementPostProcessor\n", 498 | "from llama_index.indices.postprocessor import SentenceTransformerRerank\n", 499 | "from llama_index import load_index_from_storage\n", 500 | "\n", 501 | "\n", 502 | "def build_sentence_window_index(\n", 503 | " documents,\n", 504 | " llm,\n", 505 | " embed_model=\"local:BAAI/bge-small-en-v1.5\",\n", 506 | " sentence_window_size=3,\n", 507 | " save_dir=\"sentence_index\",\n", 508 | "):\n", 509 | " # create the sentence window node parser w/ default settings\n", 510 | " node_parser = SentenceWindowNodeParser.from_defaults(\n", 511 | " window_size=sentence_window_size,\n", 512 | " window_metadata_key=\"window\",\n", 513 | " original_text_metadata_key=\"original_text\",\n", 514 | " )\n", 515 | " sentence_context = ServiceContext.from_defaults(\n", 516 | " llm=llm,\n", 517 | " embed_model=embed_model,\n", 518 | " node_parser=node_parser,\n", 519 | " )\n", 520 | " if not os.path.exists(save_dir):\n", 521 | " sentence_index = VectorStoreIndex.from_documents(\n", 522 | " documents, service_context=sentence_context\n", 523 | " )\n", 524 | " sentence_index.storage_context.persist(persist_dir=save_dir)\n", 525 | " else:\n", 526 | " sentence_index = load_index_from_storage(\n", 527 | " StorageContext.from_defaults(persist_dir=save_dir),\n", 528 | " service_context=sentence_context,\n", 529 | " )\n", 530 | "\n", 531 | " return sentence_index\n", 532 | "\n", 533 | "\n", 534 | "def get_sentence_window_query_engine(\n", 535 | " sentence_index, similarity_top_k=6, rerank_top_n=2\n", 536 | "):\n", 537 | " # define postprocessors\n", 538 | " postproc = MetadataReplacementPostProcessor(target_metadata_key=\"window\")\n", 539 | " rerank = SentenceTransformerRerank(\n", 540 | " top_n=rerank_top_n, model=\"BAAI/bge-reranker-base\"\n", 541 | " )\n", 542 | "\n", 543 | " sentence_window_engine = sentence_index.as_query_engine(\n", 544 | " similarity_top_k=similarity_top_k, node_postprocessors=[postproc, rerank]\n", 545 | " )\n", 546 | " return sentence_window_engine" 547 | ] 548 | }, 549 | { 550 | "cell_type": "code", 551 | "execution_count": null, 552 | "id": "3b5bb36f-97f9-4e03-bd4e-3ceb635a868b", 553 | "metadata": {}, 554 | "outputs": [], 555 | "source": [ 556 | "from llama_index.llms import OpenAI\n", 557 | "\n", 558 | "index = build_sentence_window_index(\n", 559 | " [document],\n", 560 | " llm=OpenAI(model=\"gpt-3.5-turbo\", temperature=0.1),\n", 561 | " save_dir=\"./sentence_index\",\n", 562 | ")\n" 563 | ] 564 | }, 565 | { 566 | "cell_type": "code", 567 | "execution_count": null, 568 | "id": "8d0e78e0-7765-4037-a5fa-63b711dab5aa", 569 | "metadata": {}, 570 | "outputs": [], 571 | "source": [ 572 | "query_engine = get_sentence_window_query_engine(index, similarity_top_k=6)\n" 573 | ] 574 | }, 575 | { 576 | "cell_type": "markdown", 577 | "id": "eb855430-8fcb-4c25-8d83-a30160449acf", 578 | "metadata": {}, 579 | "source": [ 580 | "## TruLens Evaluation" 581 | ] 582 | }, 583 | { 584 | "cell_type": "code", 585 | "execution_count": null, 586 | "id": "07aa5320-5c1c-4636-aa72-e6a786a58c8a", 587 | "metadata": {}, 588 | "outputs": [], 589 | "source": [ 590 | "eval_questions = []\n", 591 | "with open('generated_questions.text', 'r') as file:\n", 592 | " for line in file:\n", 593 | " # Remove newline character and convert to integer\n", 594 | " item = line.strip()\n", 595 | " eval_questions.append(item)" 596 | ] 597 | }, 598 | { 599 | "cell_type": "code", 600 | "execution_count": null, 601 | "id": "7f9bdeb7-8c37-46b0-80f2-eb8ef1cc7b59", 602 | "metadata": {}, 603 | "outputs": [], 604 | "source": [ 605 | "from trulens_eval import Tru\n", 606 | "\n", 607 | "def run_evals(eval_questions, tru_recorder, query_engine):\n", 608 | " for question in eval_questions:\n", 609 | " with tru_recorder as recording:\n", 610 | " response = query_engine.query(question)" 611 | ] 612 | }, 613 | { 614 | "cell_type": "code", 615 | "execution_count": null, 616 | "id": "c7818a9b-04cc-4a6e-98ae-d3d0ab309998", 617 | "metadata": {}, 618 | "outputs": [], 619 | "source": [ 620 | "from utils import get_prebuilt_trulens_recorder\n", 621 | "\n", 622 | "from trulens_eval import Tru\n", 623 | "\n", 624 | "Tru().reset_database()" 625 | ] 626 | }, 627 | { 628 | "cell_type": "markdown", 629 | "id": "59ef4d27-2c4e-4dd0-85ed-a69a0f8eea00", 630 | "metadata": {}, 631 | "source": [ 632 | "### Sentence window size = 1" 633 | ] 634 | }, 635 | { 636 | "cell_type": "code", 637 | "execution_count": null, 638 | "id": "93e6c651-838d-4c36-b806-efd0d8c2d5f0", 639 | "metadata": {}, 640 | "outputs": [], 641 | "source": [ 642 | "sentence_index_1 = build_sentence_window_index(\n", 643 | " documents,\n", 644 | " llm=OpenAI(model=\"gpt-3.5-turbo\", temperature=0.1),\n", 645 | " embed_model=\"local:BAAI/bge-small-en-v1.5\",\n", 646 | " sentence_window_size=1,\n", 647 | " save_dir=\"sentence_index_1\",\n", 648 | ")" 649 | ] 650 | }, 651 | { 652 | "cell_type": "code", 653 | "execution_count": null, 654 | "id": "8dc445f4-0311-4192-ac2e-4ea94b2653e5", 655 | "metadata": {}, 656 | "outputs": [], 657 | "source": [ 658 | "sentence_window_engine_1 = get_sentence_window_query_engine(\n", 659 | " sentence_index_1\n", 660 | ")" 661 | ] 662 | }, 663 | { 664 | "cell_type": "code", 665 | "execution_count": null, 666 | "id": "2750e2b1-ca60-495f-8168-33520740916c", 667 | "metadata": {}, 668 | "outputs": [], 669 | "source": [ 670 | "tru_recorder_1 = get_prebuilt_trulens_recorder(\n", 671 | " sentence_window_engine_1,\n", 672 | " app_id='sentence window engine 1'\n", 673 | ")" 674 | ] 675 | }, 676 | { 677 | "cell_type": "code", 678 | "execution_count": null, 679 | "id": "ca942b16-52be-4107-861c-33daeca1f619", 680 | "metadata": {}, 681 | "outputs": [], 682 | "source": [ 683 | "run_evals(eval_questions, tru_recorder_1, sentence_window_engine_1)" 684 | ] 685 | }, 686 | { 687 | "cell_type": "code", 688 | "execution_count": null, 689 | "id": "55dc7629-7fb7-4801-bd52-c9ab0da50267", 690 | "metadata": {}, 691 | "outputs": [], 692 | "source": [ 693 | "Tru().run_dashboard()" 694 | ] 695 | }, 696 | { 697 | "cell_type": "markdown", 698 | "id": "40d36ca4-8f52-4510-bfab-aa00b704d179", 699 | "metadata": {}, 700 | "source": [ 701 | "### Note about the dataset of questions\n", 702 | "- Since this evaluation process takes a long time to run, the following file `generated_questions.text` contains one question (the one mentioned in the lecture video).\n", 703 | "- If you would like to explore other possible questions, feel free to explore the file directory by clicking on the \"Jupyter\" logo at the top right of this notebook. You'll see the following `.text` files:\n", 704 | "\n", 705 | "> - `generated_questions_01_05.text`\n", 706 | "> - `generated_questions_06_10.text`\n", 707 | "> - `generated_questions_11_15.text`\n", 708 | "> - `generated_questions_16_20.text`\n", 709 | "> - `generated_questions_21_24.text`\n", 710 | "\n", 711 | "Note that running an evaluation on more than one question can take some time, so we recommend choosing one of these files (with 5 questions each) to run and explore the results.\n", 712 | "\n", 713 | "- For evaluating a personal project, an eval set of 20 is reasonable.\n", 714 | "- For evaluating business applications, you may need a set of 100+ in order to cover all the use cases thoroughly.\n", 715 | "- Note that since API calls can sometimes fail, you may occasionally see null responses, and would want to re-run your evaluations. So running your evaluations in smaller batches can also help you save time and cost by only re-running the evaluation on the batches with issues." 716 | ] 717 | }, 718 | { 719 | "cell_type": "code", 720 | "execution_count": null, 721 | "id": "6c55e4e1-5f1f-4c50-bd8e-4b54299377da", 722 | "metadata": {}, 723 | "outputs": [], 724 | "source": [ 725 | "eval_questions = []\n", 726 | "with open('generated_questions.text', 'r') as file:\n", 727 | " for line in file:\n", 728 | " # Remove newline character and convert to integer\n", 729 | " item = line.strip()\n", 730 | " eval_questions.append(item)" 731 | ] 732 | }, 733 | { 734 | "cell_type": "markdown", 735 | "id": "e54c9977-c606-41bb-b87a-ec3cc17eabaf", 736 | "metadata": {}, 737 | "source": [ 738 | "### Sentence window size = 3" 739 | ] 740 | }, 741 | { 742 | "cell_type": "code", 743 | "execution_count": null, 744 | "id": "0c8bea9d-9805-4302-b8c0-6e0f3eeec030", 745 | "metadata": {}, 746 | "outputs": [], 747 | "source": [ 748 | "sentence_index_3 = build_sentence_window_index(\n", 749 | " documents,\n", 750 | " llm=OpenAI(model=\"gpt-3.5-turbo\", temperature=0.1),\n", 751 | " embed_model=\"local:BAAI/bge-small-en-v1.5\",\n", 752 | " sentence_window_size=3,\n", 753 | " save_dir=\"sentence_index_3\",\n", 754 | ")\n", 755 | "sentence_window_engine_3 = get_sentence_window_query_engine(\n", 756 | " sentence_index_3\n", 757 | ")\n", 758 | "\n", 759 | "tru_recorder_3 = get_prebuilt_trulens_recorder(\n", 760 | " sentence_window_engine_3,\n", 761 | " app_id='sentence window engine 3'\n", 762 | ")" 763 | ] 764 | }, 765 | { 766 | "cell_type": "code", 767 | "execution_count": null, 768 | "id": "e69d459e-e984-46b4-8fd9-ca4f77150647", 769 | "metadata": {}, 770 | "outputs": [], 771 | "source": [ 772 | "run_evals(eval_questions, tru_recorder_3, sentence_window_engine_3)" 773 | ] 774 | }, 775 | { 776 | "cell_type": "code", 777 | "execution_count": null, 778 | "id": "dfdfb770", 779 | "metadata": {}, 780 | "outputs": [], 781 | "source": [ 782 | "Tru().run_dashboard()" 783 | ] 784 | } 785 | ], 786 | "metadata": { 787 | "kernelspec": { 788 | "display_name": "Python 3 (ipykernel)", 789 | "language": "python", 790 | "name": "python3" 791 | }, 792 | "language_info": { 793 | "codemirror_mode": { 794 | "name": "ipython", 795 | "version": 3 796 | }, 797 | "file_extension": ".py", 798 | "mimetype": "text/x-python", 799 | "name": "python", 800 | "nbconvert_exporter": "python", 801 | "pygments_lexer": "ipython3", 802 | "version": "3.10.9" 803 | } 804 | }, 805 | "nbformat": 4, 806 | "nbformat_minor": 5 807 | } 808 | -------------------------------------------------------------------------------- /L4-Auto-merging_Retrieval.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "cd7996c5-91eb-41de-ac6a-0b22462b76f1", 6 | "metadata": {}, 7 | "source": [ 8 | "# Lesson 4: Auto-merging Retrieval" 9 | ] 10 | }, 11 | { 12 | "cell_type": "code", 13 | "execution_count": null, 14 | "id": "e8f0928d-4505-437c-9ee8-bada425a5e22", 15 | "metadata": {}, 16 | "outputs": [], 17 | "source": [ 18 | "import warnings\n", 19 | "warnings.filterwarnings('ignore')" 20 | ] 21 | }, 22 | { 23 | "cell_type": "code", 24 | "execution_count": null, 25 | "id": "20373372-03c5-41e9-8bc9-dfa6ce35d666", 26 | "metadata": {}, 27 | "outputs": [], 28 | "source": [ 29 | "import utils\n", 30 | "\n", 31 | "import os\n", 32 | "import openai\n", 33 | "openai.api_key = utils.get_openai_api_key()" 34 | ] 35 | }, 36 | { 37 | "cell_type": "code", 38 | "execution_count": null, 39 | "id": "f58fc195-9619-4088-831e-66b3cc6e0425", 40 | "metadata": { 41 | "tags": [] 42 | }, 43 | "outputs": [], 44 | "source": [ 45 | "from llama_index import SimpleDirectoryReader\n", 46 | "\n", 47 | "documents = SimpleDirectoryReader(\n", 48 | " input_files=[\"./eBook-How-to-Build-a-Career-in-AI.pdf\"]\n", 49 | ").load_data()" 50 | ] 51 | }, 52 | { 53 | "cell_type": "code", 54 | "execution_count": null, 55 | "id": "89ad555b-48d4-441b-b5f3-79a685b4c3f2", 56 | "metadata": { 57 | "tags": [] 58 | }, 59 | "outputs": [], 60 | "source": [ 61 | "print(type(documents), \"\\n\")\n", 62 | "print(len(documents), \"\\n\")\n", 63 | "print(type(documents[0]))\n", 64 | "print(documents[0])" 65 | ] 66 | }, 67 | { 68 | "cell_type": "markdown", 69 | "id": "0c143850-72c6-4a09-9fb0-a216108314c0", 70 | "metadata": {}, 71 | "source": [ 72 | "## Auto-merging retrieval setup" 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": null, 78 | "id": "8926348c-a4c5-471b-99ed-e6bf170f1fe1", 79 | "metadata": { 80 | "tags": [] 81 | }, 82 | "outputs": [], 83 | "source": [ 84 | "from llama_index import Document\n", 85 | "\n", 86 | "document = Document(text=\"\\n\\n\".join([doc.text for doc in documents]))" 87 | ] 88 | }, 89 | { 90 | "cell_type": "code", 91 | "execution_count": null, 92 | "id": "202ccfb5-cf94-48a4-b23b-8c5a5ab73ebc", 93 | "metadata": { 94 | "tags": [] 95 | }, 96 | "outputs": [], 97 | "source": [ 98 | "from llama_index.node_parser import HierarchicalNodeParser\n", 99 | "\n", 100 | "# create the hierarchical node parser w/ default settings\n", 101 | "node_parser = HierarchicalNodeParser.from_defaults(\n", 102 | " chunk_sizes=[2048, 512, 128]\n", 103 | ")" 104 | ] 105 | }, 106 | { 107 | "cell_type": "code", 108 | "execution_count": null, 109 | "id": "672fd574-6b3c-43ad-9a9a-2db423cadd48", 110 | "metadata": { 111 | "tags": [] 112 | }, 113 | "outputs": [], 114 | "source": [ 115 | "nodes = node_parser.get_nodes_from_documents([document])" 116 | ] 117 | }, 118 | { 119 | "cell_type": "code", 120 | "execution_count": null, 121 | "id": "83cfc301-1887-4b1c-a8aa-1fb55e30be78", 122 | "metadata": { 123 | "tags": [] 124 | }, 125 | "outputs": [], 126 | "source": [ 127 | "from llama_index.node_parser import get_leaf_nodes\n", 128 | "\n", 129 | "leaf_nodes = get_leaf_nodes(nodes)\n", 130 | "print(leaf_nodes[30].text)" 131 | ] 132 | }, 133 | { 134 | "cell_type": "code", 135 | "execution_count": null, 136 | "id": "0703abeb-3bcc-4dd6-8010-3bcf95d8b483", 137 | "metadata": { 138 | "tags": [] 139 | }, 140 | "outputs": [], 141 | "source": [ 142 | "nodes_by_id = {node.node_id: node for node in nodes}\n", 143 | "\n", 144 | "parent_node = nodes_by_id[leaf_nodes[30].parent_node.node_id]\n", 145 | "print(parent_node.text)" 146 | ] 147 | }, 148 | { 149 | "cell_type": "markdown", 150 | "id": "b6bafef6-02f6-4029-a5eb-dfbc4369da56", 151 | "metadata": {}, 152 | "source": [ 153 | "### Building the index" 154 | ] 155 | }, 156 | { 157 | "cell_type": "code", 158 | "execution_count": null, 159 | "id": "6b0350cd-392e-45e9-8664-ff22a39a8bcd", 160 | "metadata": { 161 | "tags": [] 162 | }, 163 | "outputs": [], 164 | "source": [ 165 | "from llama_index.llms import OpenAI\n", 166 | "\n", 167 | "llm = OpenAI(model=\"gpt-3.5-turbo\", temperature=0.1)" 168 | ] 169 | }, 170 | { 171 | "cell_type": "code", 172 | "execution_count": null, 173 | "id": "03213595-6581-47f5-90a7-cfa381d42872", 174 | "metadata": { 175 | "tags": [] 176 | }, 177 | "outputs": [], 178 | "source": [ 179 | "from llama_index import ServiceContext\n", 180 | "\n", 181 | "auto_merging_context = ServiceContext.from_defaults(\n", 182 | " llm=llm,\n", 183 | " embed_model=\"local:BAAI/bge-small-en-v1.5\",\n", 184 | " node_parser=node_parser,\n", 185 | ")" 186 | ] 187 | }, 188 | { 189 | "cell_type": "code", 190 | "execution_count": null, 191 | "id": "a0f84886-e1aa-40f1-a812-4b6f8d0204af", 192 | "metadata": { 193 | "tags": [] 194 | }, 195 | "outputs": [], 196 | "source": [ 197 | "from llama_index import VectorStoreIndex, StorageContext\n", 198 | "\n", 199 | "storage_context = StorageContext.from_defaults()\n", 200 | "storage_context.docstore.add_documents(nodes)\n", 201 | "\n", 202 | "automerging_index = VectorStoreIndex(\n", 203 | " leaf_nodes, storage_context=storage_context, service_context=auto_merging_context\n", 204 | ")\n", 205 | "\n", 206 | "automerging_index.storage_context.persist(persist_dir=\"./merging_index\")" 207 | ] 208 | }, 209 | { 210 | "cell_type": "code", 211 | "execution_count": null, 212 | "id": "4a8ddbff-6fdd-448d-9ce8-53ed782a6d66", 213 | "metadata": { 214 | "tags": [] 215 | }, 216 | "outputs": [], 217 | "source": [ 218 | "# This block of code is optional to check\n", 219 | "# if an index file exist, then it will load it\n", 220 | "# if not, it will rebuild it\n", 221 | "\n", 222 | "import os\n", 223 | "from llama_index import VectorStoreIndex, StorageContext, load_index_from_storage\n", 224 | "from llama_index import load_index_from_storage\n", 225 | "\n", 226 | "if not os.path.exists(\"./merging_index\"):\n", 227 | " storage_context = StorageContext.from_defaults()\n", 228 | " storage_context.docstore.add_documents(nodes)\n", 229 | "\n", 230 | " automerging_index = VectorStoreIndex(\n", 231 | " leaf_nodes,\n", 232 | " storage_context=storage_context,\n", 233 | " service_context=auto_merging_context\n", 234 | " )\n", 235 | "\n", 236 | " automerging_index.storage_context.persist(persist_dir=\"./merging_index\")\n", 237 | "else:\n", 238 | " automerging_index = load_index_from_storage(\n", 239 | " StorageContext.from_defaults(persist_dir=\"./merging_index\"),\n", 240 | " service_context=auto_merging_context\n", 241 | " )\n" 242 | ] 243 | }, 244 | { 245 | "cell_type": "markdown", 246 | "id": "2710c689-0596-45c5-a63f-1e8bb9481846", 247 | "metadata": {}, 248 | "source": [ 249 | "### Defining the retriever and running the query engine" 250 | ] 251 | }, 252 | { 253 | "cell_type": "code", 254 | "execution_count": null, 255 | "id": "d443d25a-2a5e-425c-9cca-4bf8503c6db3", 256 | "metadata": { 257 | "tags": [] 258 | }, 259 | "outputs": [], 260 | "source": [ 261 | "from llama_index.indices.postprocessor import SentenceTransformerRerank\n", 262 | "from llama_index.retrievers import AutoMergingRetriever\n", 263 | "from llama_index.query_engine import RetrieverQueryEngine\n", 264 | "\n", 265 | "automerging_retriever = automerging_index.as_retriever(\n", 266 | " similarity_top_k=12\n", 267 | ")\n", 268 | "\n", 269 | "retriever = AutoMergingRetriever(\n", 270 | " automerging_retriever, \n", 271 | " automerging_index.storage_context, \n", 272 | " verbose=True\n", 273 | ")\n", 274 | "\n", 275 | "rerank = SentenceTransformerRerank(top_n=6, model=\"BAAI/bge-reranker-base\")\n", 276 | "\n", 277 | "auto_merging_engine = RetrieverQueryEngine.from_args(\n", 278 | " automerging_retriever, node_postprocessors=[rerank]\n", 279 | ")" 280 | ] 281 | }, 282 | { 283 | "cell_type": "code", 284 | "execution_count": null, 285 | "id": "c458d0ac-24a0-430b-acc4-c05252134527", 286 | "metadata": { 287 | "tags": [] 288 | }, 289 | "outputs": [], 290 | "source": [ 291 | "auto_merging_response = auto_merging_engine.query(\n", 292 | " \"What is the importance of networking in AI?\"\n", 293 | ")" 294 | ] 295 | }, 296 | { 297 | "cell_type": "code", 298 | "execution_count": null, 299 | "id": "569e70b4-daae-4181-a953-7ea37f4620c7", 300 | "metadata": {}, 301 | "outputs": [], 302 | "source": [ 303 | "from llama_index.response.notebook_utils import display_response\n", 304 | "\n", 305 | "display_response(auto_merging_response)" 306 | ] 307 | }, 308 | { 309 | "cell_type": "markdown", 310 | "id": "a841036a-0bb6-4037-920e-ae555748e111", 311 | "metadata": {}, 312 | "source": [ 313 | "## Putting it all Together" 314 | ] 315 | }, 316 | { 317 | "cell_type": "code", 318 | "execution_count": null, 319 | "id": "8949da64-bc7f-4929-b394-51d11732e62f", 320 | "metadata": {}, 321 | "outputs": [], 322 | "source": [ 323 | "import os\n", 324 | "\n", 325 | "from llama_index import (\n", 326 | " ServiceContext,\n", 327 | " StorageContext,\n", 328 | " VectorStoreIndex,\n", 329 | " load_index_from_storage,\n", 330 | ")\n", 331 | "from llama_index.node_parser import HierarchicalNodeParser\n", 332 | "from llama_index.node_parser import get_leaf_nodes\n", 333 | "from llama_index import StorageContext, load_index_from_storage\n", 334 | "from llama_index.retrievers import AutoMergingRetriever\n", 335 | "from llama_index.indices.postprocessor import SentenceTransformerRerank\n", 336 | "from llama_index.query_engine import RetrieverQueryEngine\n", 337 | "\n", 338 | "\n", 339 | "def build_automerging_index(\n", 340 | " documents,\n", 341 | " llm,\n", 342 | " embed_model=\"local:BAAI/bge-small-en-v1.5\",\n", 343 | " save_dir=\"merging_index\",\n", 344 | " chunk_sizes=None,\n", 345 | "):\n", 346 | " chunk_sizes = chunk_sizes or [2048, 512, 128]\n", 347 | " node_parser = HierarchicalNodeParser.from_defaults(chunk_sizes=chunk_sizes)\n", 348 | " nodes = node_parser.get_nodes_from_documents(documents)\n", 349 | " leaf_nodes = get_leaf_nodes(nodes)\n", 350 | " merging_context = ServiceContext.from_defaults(\n", 351 | " llm=llm,\n", 352 | " embed_model=embed_model,\n", 353 | " )\n", 354 | " storage_context = StorageContext.from_defaults()\n", 355 | " storage_context.docstore.add_documents(nodes)\n", 356 | "\n", 357 | " if not os.path.exists(save_dir):\n", 358 | " automerging_index = VectorStoreIndex(\n", 359 | " leaf_nodes, storage_context=storage_context, service_context=merging_context\n", 360 | " )\n", 361 | " automerging_index.storage_context.persist(persist_dir=save_dir)\n", 362 | " else:\n", 363 | " automerging_index = load_index_from_storage(\n", 364 | " StorageContext.from_defaults(persist_dir=save_dir),\n", 365 | " service_context=merging_context,\n", 366 | " )\n", 367 | " return automerging_index\n", 368 | "\n", 369 | "\n", 370 | "def get_automerging_query_engine(\n", 371 | " automerging_index,\n", 372 | " similarity_top_k=12,\n", 373 | " rerank_top_n=6,\n", 374 | "):\n", 375 | " base_retriever = automerging_index.as_retriever(similarity_top_k=similarity_top_k)\n", 376 | " retriever = AutoMergingRetriever(\n", 377 | " base_retriever, automerging_index.storage_context, verbose=True\n", 378 | " )\n", 379 | " rerank = SentenceTransformerRerank(\n", 380 | " top_n=rerank_top_n, model=\"BAAI/bge-reranker-base\"\n", 381 | " )\n", 382 | " auto_merging_engine = RetrieverQueryEngine.from_args(\n", 383 | " retriever, node_postprocessors=[rerank]\n", 384 | " )\n", 385 | " return auto_merging_engine" 386 | ] 387 | }, 388 | { 389 | "cell_type": "code", 390 | "execution_count": null, 391 | "id": "b7f52577-2e8e-4fee-ade9-2f5ce96a7a8c", 392 | "metadata": {}, 393 | "outputs": [], 394 | "source": [ 395 | "from llama_index.llms import OpenAI\n", 396 | "\n", 397 | "index = build_automerging_index(\n", 398 | " [document],\n", 399 | " llm=OpenAI(model=\"gpt-3.5-turbo\", temperature=0.1),\n", 400 | " save_dir=\"./merging_index\",\n", 401 | ")\n" 402 | ] 403 | }, 404 | { 405 | "cell_type": "code", 406 | "execution_count": null, 407 | "id": "c753885f-bfaf-4ac0-8410-b62f619827c4", 408 | "metadata": {}, 409 | "outputs": [], 410 | "source": [ 411 | "query_engine = get_automerging_query_engine(index, similarity_top_k=6)" 412 | ] 413 | }, 414 | { 415 | "cell_type": "markdown", 416 | "id": "9130e35d-bc09-4e4d-bbd9-61c9d6847110", 417 | "metadata": {}, 418 | "source": [ 419 | "## TruLens Evaluation" 420 | ] 421 | }, 422 | { 423 | "cell_type": "code", 424 | "execution_count": null, 425 | "id": "cf034319-2f0a-43ac-896b-2aef1762b3bd", 426 | "metadata": {}, 427 | "outputs": [], 428 | "source": [ 429 | "from trulens_eval import Tru\n", 430 | "\n", 431 | "Tru().reset_database()" 432 | ] 433 | }, 434 | { 435 | "cell_type": "markdown", 436 | "id": "76e8c8b7-8d47-45cb-94f3-70c2d75c9500", 437 | "metadata": {}, 438 | "source": [ 439 | "### Two layers" 440 | ] 441 | }, 442 | { 443 | "cell_type": "code", 444 | "execution_count": null, 445 | "id": "6fd32404-2760-420d-b406-61d50c5ae7de", 446 | "metadata": {}, 447 | "outputs": [], 448 | "source": [ 449 | "auto_merging_index_0 = build_automerging_index(\n", 450 | " documents,\n", 451 | " llm=OpenAI(model=\"gpt-3.5-turbo\", temperature=0.1),\n", 452 | " embed_model=\"local:BAAI/bge-small-en-v1.5\",\n", 453 | " save_dir=\"merging_index_0\",\n", 454 | " chunk_sizes=[2048,512],\n", 455 | ")" 456 | ] 457 | }, 458 | { 459 | "cell_type": "code", 460 | "execution_count": null, 461 | "id": "ad319255-b86a-4d05-8dd2-a8fa02d15ed0", 462 | "metadata": {}, 463 | "outputs": [], 464 | "source": [ 465 | "auto_merging_engine_0 = get_automerging_query_engine(\n", 466 | " auto_merging_index_0,\n", 467 | " similarity_top_k=12,\n", 468 | " rerank_top_n=6,\n", 469 | ")" 470 | ] 471 | }, 472 | { 473 | "cell_type": "code", 474 | "execution_count": null, 475 | "id": "c669cee9-7be9-4a4d-8c17-1672bb9c3abf", 476 | "metadata": {}, 477 | "outputs": [], 478 | "source": [ 479 | "from utils import get_prebuilt_trulens_recorder\n", 480 | "\n", 481 | "tru_recorder = get_prebuilt_trulens_recorder(\n", 482 | " auto_merging_engine_0,\n", 483 | " app_id ='app_0'\n", 484 | ")" 485 | ] 486 | }, 487 | { 488 | "cell_type": "code", 489 | "execution_count": null, 490 | "id": "ad216bb7-100e-455f-8c33-aa68b3e42026", 491 | "metadata": {}, 492 | "outputs": [], 493 | "source": [ 494 | "eval_questions = []\n", 495 | "with open('generated_questions.text', 'r') as file:\n", 496 | " for line in file:\n", 497 | " # Remove newline character and convert to integer\n", 498 | " item = line.strip()\n", 499 | " eval_questions.append(item)" 500 | ] 501 | }, 502 | { 503 | "cell_type": "code", 504 | "execution_count": null, 505 | "id": "d471cb1d-be4e-423b-a5a5-f216cbd8c9ff", 506 | "metadata": {}, 507 | "outputs": [], 508 | "source": [ 509 | "def run_evals(eval_questions, tru_recorder, query_engine):\n", 510 | " for question in eval_questions:\n", 511 | " with tru_recorder as recording:\n", 512 | " response = query_engine.query(question)" 513 | ] 514 | }, 515 | { 516 | "cell_type": "code", 517 | "execution_count": null, 518 | "id": "eea6f236-fe6a-470f-82b5-1a1e3d2593ce", 519 | "metadata": {}, 520 | "outputs": [], 521 | "source": [ 522 | "run_evals(eval_questions, tru_recorder, auto_merging_engine_0)" 523 | ] 524 | }, 525 | { 526 | "cell_type": "code", 527 | "execution_count": null, 528 | "id": "54cc6991-e743-4e33-9e22-414362a6aae0", 529 | "metadata": {}, 530 | "outputs": [], 531 | "source": [ 532 | "from trulens_eval import Tru\n", 533 | "\n", 534 | "Tru().get_leaderboard(app_ids=[])" 535 | ] 536 | }, 537 | { 538 | "cell_type": "code", 539 | "execution_count": null, 540 | "id": "f204af1a-ab10-4e50-b2d6-ba78e2a66e90", 541 | "metadata": {}, 542 | "outputs": [], 543 | "source": [ 544 | "Tru().run_dashboard()" 545 | ] 546 | }, 547 | { 548 | "cell_type": "markdown", 549 | "id": "c36721ff-9fe4-43e9-b7ed-435f6d054e29", 550 | "metadata": {}, 551 | "source": [ 552 | "### Three layers" 553 | ] 554 | }, 555 | { 556 | "cell_type": "code", 557 | "execution_count": null, 558 | "id": "5a575af9-37b8-442a-b6f2-3ed04bf742f1", 559 | "metadata": {}, 560 | "outputs": [], 561 | "source": [ 562 | "auto_merging_index_1 = build_automerging_index(\n", 563 | " documents,\n", 564 | " llm=OpenAI(model=\"gpt-3.5-turbo\", temperature=0.1),\n", 565 | " embed_model=\"local:BAAI/bge-small-en-v1.5\",\n", 566 | " save_dir=\"merging_index_1\",\n", 567 | " chunk_sizes=[2048,512,128],\n", 568 | ")" 569 | ] 570 | }, 571 | { 572 | "cell_type": "code", 573 | "execution_count": null, 574 | "id": "b0493fb6-3a5b-4d14-81ba-f26d9e4d46b3", 575 | "metadata": {}, 576 | "outputs": [], 577 | "source": [ 578 | "auto_merging_engine_1 = get_automerging_query_engine(\n", 579 | " auto_merging_index_1,\n", 580 | " similarity_top_k=12,\n", 581 | " rerank_top_n=6,\n", 582 | ")\n" 583 | ] 584 | }, 585 | { 586 | "cell_type": "code", 587 | "execution_count": null, 588 | "id": "d3198fa2-6a81-4edd-a0c8-843c3838f56f", 589 | "metadata": {}, 590 | "outputs": [], 591 | "source": [ 592 | "tru_recorder = get_prebuilt_trulens_recorder(\n", 593 | " auto_merging_engine_1,\n", 594 | " app_id ='app_1'\n", 595 | ")" 596 | ] 597 | }, 598 | { 599 | "cell_type": "code", 600 | "execution_count": null, 601 | "id": "5063c989-9f96-4bfa-bb50-fa9dbf6d576c", 602 | "metadata": {}, 603 | "outputs": [], 604 | "source": [ 605 | "run_evals(eval_questions, tru_recorder, auto_merging_engine_1)" 606 | ] 607 | }, 608 | { 609 | "cell_type": "code", 610 | "execution_count": null, 611 | "id": "c26b5c94-16ce-415d-93ca-a131b05f903f", 612 | "metadata": {}, 613 | "outputs": [], 614 | "source": [ 615 | "from trulens_eval import Tru\n", 616 | "\n", 617 | "Tru().get_leaderboard(app_ids=[])" 618 | ] 619 | }, 620 | { 621 | "cell_type": "code", 622 | "execution_count": null, 623 | "id": "3988b3f8-bd4c-41e5-be50-e90b8366f4bd", 624 | "metadata": {}, 625 | "outputs": [], 626 | "source": [ 627 | "Tru().run_dashboard()" 628 | ] 629 | } 630 | ], 631 | "metadata": { 632 | "kernelspec": { 633 | "display_name": "Python 3 (ipykernel)", 634 | "language": "python", 635 | "name": "python3" 636 | }, 637 | "language_info": { 638 | "codemirror_mode": { 639 | "name": "ipython", 640 | "version": 3 641 | }, 642 | "file_extension": ".py", 643 | "mimetype": "text/x-python", 644 | "name": "python", 645 | "nbconvert_exporter": "python", 646 | "pygments_lexer": "ipython3", 647 | "version": "3.10.9" 648 | } 649 | }, 650 | "nbformat": 4, 651 | "nbformat_minor": 5 652 | } 653 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Building and Evaluating Advanced RAG Applications 2 | 3 | Jupyter notebooks for course `Building and Evaluating Advanced RAG Applications`, taught by Jerry Liu (Co-founder and CEO of LlamaIndex) and Anupam Datta (Co-founder and chief scientist of TruEra). 4 | 5 | ## Course Topics 6 | 7 | Learn how to efficiently bring Retrieval Augmented Generation (RAG) into production by enhancing retrieval techniques and mastering evaluation metrics. 8 | 9 | - Learn methods like sentence-window retrieval and auto-merging retrieval, improving your RAG pipeline’s performance beyond the baseline. 10 | - Learn evaluation best practices to streamline your process, and iteratively build a robust system. 11 | - Dive into the RAG triad for evaluating the relevance and truthfulness of an LLM’s response:Context Relevance, Groundedness, and Answer Relevance. 12 | 13 | 14 | ## Course Link 15 | 16 | https://www.deeplearning.ai/short-courses/building-evaluating-advanced-rag/ 17 | 18 | ## Resource Links 19 | 20 | - LlamaIndex - Data Framework for LLM Applications - https://www.llamaindex.ai/ 21 | - TruEra - leader in ML monitoring, testing, and quality management - https://truera.com/ 22 | 23 | --------------------------------------------------------------------------------