├── .gitignore ├── images ├── vegas.png ├── amsterdam.jpeg ├── ReACT_agents.png ├── ReWoO_agents.png ├── converse_api.png ├── rag-workflow.png ├── short-vs-long.png ├── RAG-travellers.png ├── agents_itinerary.png ├── macro-micro-eval.png ├── rag-architecture.png ├── react_tool_call.png ├── graph_state_light.png ├── human-in-the-loop.png ├── multi_memory_light.png ├── travel_agent_light.png ├── agent_action_thought.png ├── agent_alignment_chart.png ├── agents_memory_light.png ├── multi-agent-travel-bot.png └── Hotel_booking_confirmation_light.png ├── data ├── travel_bookings.db ├── section_doc_store.pkl ├── section_vector_store.pkl ├── examples.txt └── synthetic_travel_data.csv ├── bedrock-multi-agent-langgraph-studio ├── src │ ├── hotel_agent │ │ ├── __init__.py │ │ ├── graph.py │ │ └── tools.py │ ├── flight_agent │ │ ├── __init__.py │ │ ├── graph.py │ │ └── tools.py │ ├── supervisor_agent │ │ ├── __init__.py │ │ └── graph.py │ └── destination_agent │ │ ├── __init__.py │ │ ├── graph.py │ │ └── tools.py ├── langgraph.json ├── pyproject.toml └── README.md ├── CODE_OF_CONDUCT.md ├── LICENSE ├── README.md ├── CONTRIBUTING.md ├── utils.py ├── 05_Optional_find_dream_destination_with_crewaI.ipynb ├── 01_usecase_introduction.ipynb └── 06_agent_evaluation_with_ragas.ipynb /.gitignore: -------------------------------------------------------------------------------- 1 | .venv 2 | data/ 3 | __pycache__/ 4 | .ipynb_checkpoints/ 5 | .DS_Store -------------------------------------------------------------------------------- /images/vegas.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aim323_build_agents_with_bedrock_oss/main/images/vegas.png -------------------------------------------------------------------------------- /images/amsterdam.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aim323_build_agents_with_bedrock_oss/main/images/amsterdam.jpeg -------------------------------------------------------------------------------- /data/travel_bookings.db: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aim323_build_agents_with_bedrock_oss/main/data/travel_bookings.db -------------------------------------------------------------------------------- /images/ReACT_agents.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aim323_build_agents_with_bedrock_oss/main/images/ReACT_agents.png -------------------------------------------------------------------------------- /images/ReWoO_agents.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aim323_build_agents_with_bedrock_oss/main/images/ReWoO_agents.png -------------------------------------------------------------------------------- /images/converse_api.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aim323_build_agents_with_bedrock_oss/main/images/converse_api.png -------------------------------------------------------------------------------- /images/rag-workflow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aim323_build_agents_with_bedrock_oss/main/images/rag-workflow.png -------------------------------------------------------------------------------- /images/short-vs-long.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aim323_build_agents_with_bedrock_oss/main/images/short-vs-long.png -------------------------------------------------------------------------------- /data/section_doc_store.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aim323_build_agents_with_bedrock_oss/main/data/section_doc_store.pkl -------------------------------------------------------------------------------- /images/RAG-travellers.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aim323_build_agents_with_bedrock_oss/main/images/RAG-travellers.png -------------------------------------------------------------------------------- /images/agents_itinerary.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aim323_build_agents_with_bedrock_oss/main/images/agents_itinerary.png -------------------------------------------------------------------------------- /images/macro-micro-eval.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aim323_build_agents_with_bedrock_oss/main/images/macro-micro-eval.png -------------------------------------------------------------------------------- /images/rag-architecture.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aim323_build_agents_with_bedrock_oss/main/images/rag-architecture.png -------------------------------------------------------------------------------- /images/react_tool_call.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aim323_build_agents_with_bedrock_oss/main/images/react_tool_call.png -------------------------------------------------------------------------------- /data/section_vector_store.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aim323_build_agents_with_bedrock_oss/main/data/section_vector_store.pkl -------------------------------------------------------------------------------- /images/graph_state_light.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aim323_build_agents_with_bedrock_oss/main/images/graph_state_light.png -------------------------------------------------------------------------------- /images/human-in-the-loop.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aim323_build_agents_with_bedrock_oss/main/images/human-in-the-loop.png -------------------------------------------------------------------------------- /images/multi_memory_light.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aim323_build_agents_with_bedrock_oss/main/images/multi_memory_light.png -------------------------------------------------------------------------------- /images/travel_agent_light.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aim323_build_agents_with_bedrock_oss/main/images/travel_agent_light.png -------------------------------------------------------------------------------- /images/agent_action_thought.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aim323_build_agents_with_bedrock_oss/main/images/agent_action_thought.png -------------------------------------------------------------------------------- /images/agent_alignment_chart.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aim323_build_agents_with_bedrock_oss/main/images/agent_alignment_chart.png -------------------------------------------------------------------------------- /images/agents_memory_light.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aim323_build_agents_with_bedrock_oss/main/images/agents_memory_light.png -------------------------------------------------------------------------------- /images/multi-agent-travel-bot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aim323_build_agents_with_bedrock_oss/main/images/multi-agent-travel-bot.png -------------------------------------------------------------------------------- /bedrock-multi-agent-langgraph-studio/src/hotel_agent/__init__.py: -------------------------------------------------------------------------------- 1 | """Hotel agent Module.""" 2 | 3 | from hotel_agent.graph import graph 4 | 5 | __all__ = ["graph"] 6 | -------------------------------------------------------------------------------- /bedrock-multi-agent-langgraph-studio/src/flight_agent/__init__.py: -------------------------------------------------------------------------------- 1 | """Flight agent Module.""" 2 | 3 | from flight_agent.graph import graph 4 | 5 | __all__ = ["graph"] 6 | -------------------------------------------------------------------------------- /images/Hotel_booking_confirmation_light.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aim323_build_agents_with_bedrock_oss/main/images/Hotel_booking_confirmation_light.png -------------------------------------------------------------------------------- /bedrock-multi-agent-langgraph-studio/src/supervisor_agent/__init__.py: -------------------------------------------------------------------------------- 1 | """Supervisor agent Module.""" 2 | 3 | from supervisor.graph import graph 4 | 5 | __all__ = ["graph"] 6 | -------------------------------------------------------------------------------- /bedrock-multi-agent-langgraph-studio/src/destination_agent/__init__.py: -------------------------------------------------------------------------------- 1 | """Planner agent Module.""" 2 | 3 | from destination_agent.graph import graph 4 | 5 | __all__ = ["graph"] 6 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 4 | opensource-codeofconduct@amazon.com with any additional questions or comments. 5 | -------------------------------------------------------------------------------- /bedrock-multi-agent-langgraph-studio/langgraph.json: -------------------------------------------------------------------------------- 1 | { 2 | "dependencies": [ 3 | "." 4 | ], 5 | "graphs": { 6 | "flight_agent": "./src/flight_agent/graph.py:graph", 7 | "hotel_agent": "./src/hotel_agent/graph.py:graph", 8 | "supervisor_agent": "./src/supervisor_agent/graph.py:graph", 9 | "destination_agent": "./src/destination_agent/graph.py:graph" 10 | }, 11 | "env": ".env" 12 | } -------------------------------------------------------------------------------- /data/examples.txt: -------------------------------------------------------------------------------- 1 | New York. 2 | Synthesize the best places to travel to today. 3 | Summarize the environment I would encounter in Bali. 4 | Give me a quick rundown of my recent travels. 5 | Explain the difference in Brazil and Argentina as travel destinations. 6 | The main points on things to do in Montreal. 7 | A brief overview of the key information about New Orleans. 8 | A brief overview of the key information about things to do in New Orleans next week. 9 | Please provide a concise summary of the hotels in Boston. 10 | Summarize the travel to get to London. -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT No Attribution 2 | 3 | Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy of 6 | this software and associated documentation files (the "Software"), to deal in 7 | the Software without restriction, including without limitation the rights to 8 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 9 | the Software, and to permit persons to whom the Software is furnished to do so. 10 | 11 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 12 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS 13 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 14 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 15 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 16 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 17 | 18 | -------------------------------------------------------------------------------- /bedrock-multi-agent-langgraph-studio/pyproject.toml: -------------------------------------------------------------------------------- 1 | [project] 2 | name = "langgraph-studio-multi-agent" 3 | version = "0.1.0" 4 | description = "Multi agent sample with Amazon Bedrock and Langgraph" 5 | readme = "README.md" 6 | requires-python = ">=3.11" 7 | dependencies = [ 8 | "boto3>=1.35.87", 9 | "langchain-aws>=0.2.10", 10 | "langgraph-cli[inmem]>=0.1.65", 11 | "langgraph>=0.2.60", 12 | "pandas>=2.2.3", 13 | ] 14 | 15 | [project.optional-dependencies] 16 | dev = ["mypy>=1.11.1", "ruff>=0.6.1"] 17 | 18 | [build-system] 19 | requires = ["setuptools>=73.0.0", "wheel"] 20 | build-backend = "setuptools.build_meta" 21 | 22 | [tool.setuptools] 23 | packages = ["flight_agent", "hotel_agent", "supervisor_agent", "destination_agent"] 24 | [tool.setuptools.package-dir] 25 | "flight_agent" = "src/flight_agent" 26 | "hotel_agent" = "src/hotel_agent" 27 | "supervisor_agent" = "src/supervisor_agent" 28 | "destination_agent" = "src/destination_agent" 29 | 30 | [tool.setuptools.package-data] 31 | "*" = ["py.typed"] 32 | 33 | [tool.ruff] 34 | lint.select = [ 35 | "E", # pycodestyle 36 | "F", # pyflakes 37 | "I", # isort 38 | "D", # pydocstyle 39 | "D401", # First line should be in imperative mood 40 | "T201", 41 | "UP", 42 | ] 43 | lint.ignore = [ 44 | "UP006", 45 | "UP007", 46 | # We actually do want to import from typing_extensions 47 | "UP035", 48 | # Relax the convention by _not_ requiring documentation for every function parameter. 49 | "D417", 50 | "E501", 51 | ] 52 | [tool.ruff.lint.per-file-ignores] 53 | "tests/*" = ["D", "UP"] 54 | [tool.ruff.lint.pydocstyle] 55 | convention = "google" 56 | [tool.pytest.ini_options] 57 | pythonpath = [ 58 | "src" 59 | ] -------------------------------------------------------------------------------- /bedrock-multi-agent-langgraph-studio/src/flight_agent/graph.py: -------------------------------------------------------------------------------- 1 | """This "graph" is a flight agent graph""" 2 | 3 | from langchain_aws import ChatBedrockConverse 4 | import boto3 5 | from langgraph.graph import StateGraph, START, MessagesState 6 | from langchain_core.runnables import RunnableConfig 7 | from langchain_core.prompts import ChatPromptTemplate 8 | from langgraph.prebuilt import ToolNode, tools_condition 9 | from langgraph.checkpoint.memory import MemorySaver 10 | 11 | from flight_agent.tools import ( 12 | search_flights, 13 | retrieve_flight_booking, 14 | change_flight_booking, 15 | cancel_flight_booking, 16 | ) 17 | 18 | bedrock_client = boto3.client("bedrock-runtime", region_name="us-west-2") 19 | 20 | llm = ChatBedrockConverse( 21 | # model="anthropic.claude-3-5-sonnet-20240620-v1:0", 22 | model="anthropic.claude-3-sonnet-20240229-v1:0", 23 | temperature=0, 24 | max_tokens=None, 25 | client=bedrock_client, 26 | # other params... 27 | ) 28 | tools = [ 29 | search_flights, 30 | retrieve_flight_booking, 31 | change_flight_booking, 32 | cancel_flight_booking, 33 | ] 34 | llm_with_tools = llm.bind_tools(tools) 35 | 36 | memory = MemorySaver() 37 | 38 | prompt = ChatPromptTemplate.from_messages( 39 | [ 40 | ( 41 | "system", 42 | """ 43 | You are a helpful agent who has tools to search flights. 44 | 'search_flights' tool already has information about user's current location 45 | """, 46 | ), 47 | ("placeholder", "{messages}"), 48 | ] 49 | ) 50 | 51 | flight_agent_chain = prompt | llm_with_tools 52 | 53 | 54 | def flight_agent(state): 55 | result = flight_agent_chain.invoke(state) 56 | return {"messages": [result]} 57 | 58 | 59 | graph_builder = StateGraph(MessagesState, config_schema=RunnableConfig) 60 | graph_builder.add_node("flight_agent", flight_agent) 61 | 62 | tool_node = ToolNode(tools=tools) 63 | graph_builder.add_node("tools", tool_node) 64 | 65 | graph_builder.add_conditional_edges( 66 | "flight_agent", 67 | tools_condition, 68 | ) 69 | graph_builder.add_edge("tools", "flight_agent") 70 | graph_builder.add_edge(START, "flight_agent") 71 | 72 | graph = graph_builder.compile(checkpointer=memory) 73 | graph.name = "FlightAgentGraph" 74 | -------------------------------------------------------------------------------- /bedrock-multi-agent-langgraph-studio/src/hotel_agent/graph.py: -------------------------------------------------------------------------------- 1 | """This "graph" is a hotel agent graph""" 2 | 3 | from typing import TypedDict, Annotated 4 | from langchain_aws import ChatBedrockConverse 5 | import boto3 6 | from langgraph.graph import StateGraph, START, MessagesState 7 | from langchain_core.runnables import RunnableConfig 8 | from langchain_core.prompts import ChatPromptTemplate 9 | from langgraph.prebuilt import ToolNode, tools_condition 10 | from langgraph.graph.message import add_messages 11 | from langgraph.checkpoint.memory import MemorySaver 12 | from hotel_agent.tools import ( 13 | suggest_hotels, 14 | retrieve_hotel_booking, 15 | change_hotel_booking, 16 | cancel_hotel_booking, 17 | ) 18 | 19 | 20 | bedrock_client = boto3.client("bedrock-runtime", region_name="us-west-2") 21 | 22 | llm = ChatBedrockConverse( 23 | # model="anthropic.claude-3-5-sonnet-20240620-v1:0", 24 | model="anthropic.claude-3-sonnet-20240229-v1:0", 25 | temperature=0, 26 | max_tokens=None, 27 | client=bedrock_client, 28 | # other params... 29 | ) 30 | 31 | memory = MemorySaver() 32 | 33 | 34 | class State(TypedDict): 35 | messages: Annotated[list, add_messages] 36 | next: str 37 | 38 | 39 | primary_assistant_prompt = ChatPromptTemplate.from_messages( 40 | [ 41 | ( 42 | "system", 43 | "You are a helpful assistant who manage hotel bookings. You have 'suggest_hotels' tool to suggest hotels", 44 | ), 45 | ("placeholder", "{messages}"), 46 | ] 47 | ) 48 | hotel_tools = [ 49 | suggest_hotels, 50 | retrieve_hotel_booking, 51 | change_hotel_booking, 52 | cancel_hotel_booking, 53 | ] 54 | 55 | runnable_with_tools = primary_assistant_prompt | llm.bind_tools(hotel_tools) 56 | 57 | 58 | def hotel_agent(state: State): 59 | return {"messages": [runnable_with_tools.invoke(state)]} 60 | 61 | 62 | graph_builder = StateGraph(MessagesState, config_schema=RunnableConfig) 63 | graph_builder.add_node("hotel_agent", hotel_agent) 64 | 65 | tool_node = ToolNode(tools=hotel_tools) 66 | graph_builder.add_node("tools", tool_node) 67 | 68 | graph_builder.add_conditional_edges( 69 | "hotel_agent", 70 | tools_condition, 71 | ) 72 | graph_builder.add_edge("tools", "hotel_agent") 73 | graph_builder.add_edge(START, "hotel_agent") 74 | 75 | graph = graph_builder.compile(checkpointer=memory) 76 | 77 | graph.name = "HotelAgentGraph" 78 | -------------------------------------------------------------------------------- /bedrock-multi-agent-langgraph-studio/src/destination_agent/graph.py: -------------------------------------------------------------------------------- 1 | """This "graph" is a flight agent graph""" 2 | 3 | from langchain_aws import ChatBedrockConverse 4 | import boto3 5 | from langgraph.graph import StateGraph, START, MessagesState 6 | from langchain_core.runnables import RunnableConfig 7 | from langchain_core.prompts import ChatPromptTemplate 8 | from langgraph.prebuilt import ToolNode, tools_condition 9 | from langgraph.checkpoint.memory import MemorySaver 10 | from destination_agent.tools import compare_and_recommend_destination 11 | from pydantic import BaseModel 12 | 13 | bedrock_client = boto3.client("bedrock-runtime", region_name="us-west-2") 14 | 15 | llm = ChatBedrockConverse( 16 | model="anthropic.claude-3-5-sonnet-20240620-v1:0", 17 | temperature=0, 18 | max_tokens=None, 19 | client=bedrock_client, 20 | # other params... 21 | ) 22 | tools = [compare_and_recommend_destination] 23 | llm_with_tools = llm.bind_tools(tools) 24 | 25 | memory = MemorySaver() 26 | 27 | 28 | class location(BaseModel): 29 | """ 30 | Return departure city and destination city 31 | """ 32 | 33 | current_location: str 34 | destination_city: str 35 | 36 | 37 | prompt = ChatPromptTemplate.from_messages( 38 | [ 39 | ( 40 | "system", 41 | """ 42 | You are a helpful assistant who can suggest destination location to the user. 43 | You have access to 'compare_and_recommend_destination' tool that can search for the destination. 44 | This tool already has information about user's current location. 45 | 46 | """, 47 | ), 48 | ("placeholder", "{messages}"), 49 | ] 50 | ) 51 | 52 | destination_agent_chain = prompt | llm_with_tools 53 | 54 | 55 | def destination_agent(state): 56 | result = destination_agent_chain.invoke(state) 57 | return {"messages": [result]} 58 | 59 | 60 | graph_builder = StateGraph(MessagesState, config_schema=RunnableConfig) 61 | graph_builder.add_node("destination_agent", destination_agent) 62 | 63 | tool_node = ToolNode(tools=tools) 64 | graph_builder.add_node("tools", tool_node) 65 | 66 | graph_builder.add_conditional_edges( 67 | "destination_agent", 68 | tools_condition, 69 | ) 70 | graph_builder.add_edge("tools", "destination_agent") 71 | graph_builder.add_edge(START, "destination_agent") 72 | 73 | graph = graph_builder.compile(checkpointer=memory) 74 | graph.name = "PlannerGraph" 75 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## AIM323 Build Agentic Workflows with Amazon Bedrock and Open Source Frameworks 2 | 3 | This repository contains code for Amazon Reinvent 2024 workshop on "Build Agentic Workflows with Amazon Bedrock and Open Source Frameworks". In this workshop, participants will get hands-on experience with building end-to-end agentic workloads using Amazon Bedrock, LangGraph, CrewAI and Ragas. 4 | 5 | Labs Include 6 | 7 | **Lab 1: AI Travel Assistant Use Case Introduction** 8 | - Explore the travel assistant use case, covering the dataset on destinations, bookings, and preferences used throughout the labs. We’ll also set up Amazon Bedrock models to enable intelligent response generation and data retrieval, laying the groundwork for the assistant’s capabilities in upcoming labs. 9 | 10 | **Lab 2: Building a Travel Planner with a Simple LangGraph** 11 | - Learn the fundamentals of LangGraph, including nodes, edges, graphs, and memory concepts. Through a hands-on exercise, you’ll use these elements to build a simple travel recommendation system. 12 | 13 | **Lab 3: Travel agent with tools** 14 | - Build a travel chatbot agent designed to assist users in finding the ideal vacation destination. This agent will have access to various tools to search based on the user’s profile and the travel history of similar users. Additionally, it will use a retrieval tool to provide in-depth information on different cities across the United States 15 | 16 | **Lab 4: Travel booking multi-agent** 17 | - Implement a supervisor agentic pattern for handling travel bookings. Here, a central supervisor agent coordinates multiple specialized agents, each with its own dedicated scratchpad. The supervisor agent acts as a coordinator, assigning tasks to the Flight Agent and Hotel Agent based on their respective functions—such as searching, retrieving, changing, and canceling bookings. 18 | - Lab 6: Dream destination finder with CrewAI and Amazon Bedrock 19 | 20 | **Lab 5: Dream Destination with CrewAI** 21 | - Explore how to use the CrewAI framework with Amazon Bedrock to build an intelligent agent that can find dream travel destinations based on user preferences. The agent will utilize a large language model (LLM) and web search capabilities to research and recommend destinations that match the user's description. 22 | 23 | **Lab 6: Evaluate Agents using ragas** 24 | - Evaluate the effectiveness and accuracy of our multi-agent travel booking system using the [ragas library](https://docs.ragas.io/en/stable/). This lab will guide you through the process of evaluating agents' performance on various tasks, such as retrieving relevant information, generating accurate responses, and effectively handling user requests. 25 | 26 | ## Security 27 | 28 | See [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information. 29 | 30 | ## License 31 | 32 | MIT-0 License 33 | 34 | -------------------------------------------------------------------------------- /bedrock-multi-agent-langgraph-studio/src/destination_agent/tools.py: -------------------------------------------------------------------------------- 1 | import pandas as pd 2 | from collections import Counter 3 | from langchain_core.tools import tool 4 | from langchain_core.runnables.config import RunnableConfig 5 | 6 | 7 | def read_travel_data(file_path: str = "data/synthetic_travel_data.csv") -> pd.DataFrame: 8 | """Read travel data from CSV file""" 9 | try: 10 | df = pd.read_csv(file_path) 11 | return df 12 | except FileNotFoundError: 13 | return pd.DataFrame( 14 | columns=[ 15 | "Id", 16 | "Name", 17 | "Current_Location", 18 | "Age", 19 | "Past_Travel_Destinations", 20 | "Number_of_Trips", 21 | "Flight_Number", 22 | "Departure_City", 23 | "Arrival_City", 24 | "Flight_Date", 25 | ] 26 | ) 27 | 28 | 29 | @tool 30 | def compare_and_recommend_destination(config: RunnableConfig) -> str: 31 | """This tool is used to check which destinations user has already traveled. 32 | Use the user id to fetch the information about the user. 33 | If user has already been to a city then do not recommend that city. 34 | 35 | Returns: 36 | str: Destination to be recommended. 37 | 38 | """ 39 | 40 | df = read_travel_data() 41 | user_id = config.get("configurable", {})["configurable"]["user_id"] 42 | print(user_id) 43 | 44 | if user_id not in df["Id"].values: 45 | return "User not found in the travel database." 46 | 47 | user_data = df[df["Id"] == user_id].iloc[0] 48 | current_location = user_data["Current_Location"] 49 | age = user_data["Age"] 50 | past_destinations = user_data["Past_Travel_Destinations"].split(", ") 51 | 52 | # Get all past destinations of users with similar age (±5 years) and same current location 53 | similar_users = df[ 54 | (df["Current_Location"] == current_location) 55 | & (df["Age"].between(age - 5, age + 5)) 56 | ] 57 | all_destinations = [ 58 | dest 59 | for user_dests in similar_users["Past_Travel_Destinations"].str.split(", ") 60 | for dest in user_dests 61 | ] 62 | 63 | # Count occurrences of each destination 64 | destination_counts = Counter(all_destinations) 65 | 66 | # Remove user's current location and past destinations from recommendations 67 | for dest in [current_location] + past_destinations: 68 | if dest in destination_counts: 69 | del destination_counts[dest] 70 | 71 | if not destination_counts: 72 | return f"No new recommendations found for users in {current_location} with similar age." 73 | 74 | # Get the most common destination 75 | recommended_destination = destination_counts.most_common(1)[0][0] 76 | 77 | return f"Based on your current location ({current_location}), age ({age}), and past travel data, we recommend visiting {recommended_destination}." 78 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | 10 | ## Reporting Bugs/Feature Requests 11 | 12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 13 | 14 | When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already 15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 16 | 17 | * A reproducible test case or series of steps 18 | * The version of our code being used 19 | * Any modifications you've made relevant to the bug 20 | * Anything unusual about your environment or deployment 21 | 22 | 23 | ## Contributing via Pull Requests 24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 25 | 26 | 1. You are working against the latest source on the *main* branch. 27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 29 | 30 | To send us a pull request, please: 31 | 32 | 1. Fork the repository. 33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 34 | 3. Ensure local tests pass. 35 | 4. Commit to your fork using clear commit messages. 36 | 5. Send us a pull request, answering any default questions in the pull request interface. 37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | 43 | ## Finding contributions to work on 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start. 45 | 46 | 47 | ## Code of Conduct 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | 53 | ## Security issue notifications 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 55 | 56 | 57 | ## Licensing 58 | 59 | See the [LICENSE](LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. 60 | -------------------------------------------------------------------------------- /bedrock-multi-agent-langgraph-studio/README.md: -------------------------------------------------------------------------------- 1 | # LangGraph Studio Multi-Agent System 2 | 3 | This project demonstrates a multi-agent system built with LangGraph and Amazon Bedrock. The system consists of multiple specialized agents (destination, flight, hotel, and supervisor) that work together to provide travel recommendations and search for flights and hotels. 4 | 5 | ## Project Structure 6 | 7 | ``` 8 | langgraph-studio-multi-agent/ 9 | ├── src/ 10 | │ ├── destination_agent/ # Agent for suggesting travel destinations 11 | │ ├── flight_agent/ # Agent for searching and booking flights 12 | │ ├── hotel_agent/ # Agent for searching and booking hotels 13 | │ └── supervisor_agent/ # Orchestrator agent that coordinates the other agents 14 | ├── data/ # Data files used by the agents 15 | ├── langgraph.json # LangGraph configuration 16 | └── pyproject.toml # Project dependencies and configuration 17 | ``` 18 | 19 | ## Prerequisites 20 | 21 | - Python 3.11 or higher 22 | - AWS account with access to Amazon Bedrock 23 | - AWS CLI configured with appropriate credentials 24 | - Access to Claude models in Amazon Bedrock 25 | 26 | ## Setup 27 | 28 | 1. Clone the repository: 29 | ```bash 30 | git clone 31 | cd langgraph-studio-multi-agent 32 | ``` 33 | 34 | 2. Create and activate a virtual environment: 35 | ```bash 36 | python -m venv .venv 37 | source .venv/bin/activate # On Windows: .venv\Scripts\activate 38 | ``` 39 | 40 | 3. Install the project and its dependencies: 41 | ```bash 42 | pip install -e . 43 | ``` 44 | 45 | ## Amazon Bedrock Configuration 46 | 47 | This project uses Amazon Bedrock for LLM capabilities. Here's how to set it up in your code: 48 | 49 | 1. **AWS Credentials**: Ensure your AWS credentials are properly configured either via environment variables, AWS CLI, or IAM roles. 50 | 51 | 2. **Bedrock Client Setup**: The project initializes a Bedrock client as follows: 52 | ```python 53 | import boto3 54 | 55 | bedrock_client = boto3.client("bedrock-runtime", region_name="us-west-2") 56 | ``` 57 | 58 | 3. **LLM Configuration**: The project uses Claude models via the `ChatBedrockConverse` class: 59 | ```python 60 | from langchain_aws import ChatBedrockConverse 61 | 62 | llm = ChatBedrockConverse( 63 | model="anthropic.claude-3-5-sonnet-20240620-v1:0", # or another Claude model 64 | temperature=0, 65 | max_tokens=None, 66 | client=bedrock_client, 67 | ) 68 | ``` 69 | 70 | 4. **Model Selection**: You can choose different Claude models based on your needs: 71 | - `anthropic.claude-3-5-sonnet-20240620-v1:0` (latest) 72 | - `anthropic.claude-3-sonnet-20240229-v1:0` 73 | - Other available Claude models in your Bedrock account 74 | 75 | 5. **Environment Variables**: Create a `.env` file in the project root with your AWS configuration if needed: 76 | ``` 77 | AWS_REGION=us-west-2 78 | # Add other AWS configuration as needed 79 | ``` 80 | 81 | ## Running LangGraph Studio 82 | 83 | LangGraph Studio provides a visual interface to interact with and debug your agent graphs. 84 | 85 | 1. **Start LangGraph Studio**: 86 | ```bash 87 | langgraph studio 88 | ``` 89 | 90 | 2. **Access the UI**: Open your browser and navigate to: 91 | ``` 92 | https://smith.langchain.com/studio/thread?baseUrl=http://127.0.0.1:2024 93 | ``` 94 | 95 | 3. **Interact with Agents**: You can interact with any of the defined agents: 96 | - Destination Agent: Suggests travel destinations 97 | - Flight Agent: Searches for flights 98 | - Hotel Agent: Searches for hotels 99 | - Supervisor Agent: Coordinates the other agents 100 | 101 | 4. **Testing the Multi-Agent System**: Try prompts like: 102 | ``` 103 | Suggest me a travel destination and search flight and hotel for me. I want to travel on 15-March-2025 for 5 days 104 | ``` 105 | 106 | ## Development 107 | 108 | To add new agents or modify existing ones: 109 | 110 | 1. Create a new directory under `src/` for your agent 111 | 2. Implement the agent's graph in a `graph.py` file 112 | 3. Add any tools the agent needs 113 | 4. Register the agent in `langgraph.json` 114 | 115 | ## Troubleshooting 116 | 117 | - **Bedrock Access Issues**: Ensure you have enabled the Claude models in your AWS Bedrock console 118 | - **LangGraph Studio Connection Issues**: Check that all dependencies are installed correctly 119 | - **Agent Errors**: Review the logs in LangGraph Studio for detailed error information 120 | 121 | ## Dependencies 122 | 123 | This project relies on the following key libraries: 124 | - `boto3`: AWS SDK for Python 125 | - `langchain-aws`: LangChain integration with AWS services 126 | - `langgraph`: Framework for building agent workflows 127 | - `langgraph-cli`: CLI tools for LangGraph 128 | 129 | For a complete list of dependencies, see `pyproject.toml`. 130 | -------------------------------------------------------------------------------- /bedrock-multi-agent-langgraph-studio/src/supervisor_agent/graph.py: -------------------------------------------------------------------------------- 1 | from typing import TypedDict, Literal, Annotated 2 | from pydantic import BaseModel 3 | from langchain_core.runnables import RunnableConfig 4 | from langgraph.graph.message import add_messages 5 | from langchain_core.messages import HumanMessage 6 | from langchain_aws import ChatBedrockConverse 7 | import boto3 8 | from langgraph.checkpoint.memory import MemorySaver 9 | from langchain_core.prompts import ChatPromptTemplate 10 | from langchain_core.messages import AIMessage 11 | from langgraph.graph import END, StateGraph, START 12 | import functools 13 | from flight_agent import graph as flight_agent_graph 14 | from hotel_agent import graph as hotel_agent_graph 15 | from destination_agent import graph as destination_agent_graph 16 | 17 | 18 | members = ["flight_agent", "hotel_agent", "destination_agent"] 19 | options = ["FINISH"] + members 20 | memory = MemorySaver() 21 | 22 | bedrock_client = boto3.client("bedrock-runtime", region_name="us-west-2") 23 | 24 | llm = ChatBedrockConverse( 25 | model="anthropic.claude-3-5-sonnet-20240620-v1:0", 26 | temperature=0, 27 | max_tokens=None, 28 | client=bedrock_client, 29 | # other params... 30 | ) 31 | 32 | 33 | class routeResponse(BaseModel): 34 | """ 35 | Return next agent name. 36 | """ 37 | 38 | next: Literal[*options] 39 | 40 | 41 | prompt = ChatPromptTemplate.from_messages( 42 | [ 43 | ( 44 | "system", 45 | """ 46 | Given the conversation below who should act next 47 | 1. Suggest vacation destination using 'destination_agent'. Get the current location and recommended destination from 'destination_agent' then to call 'flight_agent' 48 | 2. Don't ask user for any confirmation. Just give flight and hotel options 49 | 3. Use 'flight_agent' to search flights. Send destination city and travel date to flight agent 50 | 4. Once 'flight_agent' has finished call 'hotel_agent' to search hotels. Do not ask user to book flights. 51 | 5. If you have the answer return 'FINISH' 52 | 6. When member has finished the task, and you notice FINISHED in the message then don't repeat same member again 53 | 54 | """, 55 | ), 56 | ("placeholder", "{messages}"), 57 | ] 58 | ).partial(options=str(options), members=", ".join(members)) 59 | 60 | supervisor_chain = prompt | llm.with_structured_output(routeResponse) 61 | 62 | 63 | class State(TypedDict): 64 | messages: Annotated[list, add_messages] 65 | next: str | None 66 | 67 | 68 | def supervisor_agent(state: State): 69 | result = supervisor_chain.invoke(state) 70 | output = { 71 | "next": result.next, 72 | "messages": [ 73 | HumanMessage( 74 | content=f"Supervisor decided: {result.next}", name="supervisor" 75 | ) 76 | ], 77 | } 78 | print(f"Supervisor output: {output}") 79 | return output 80 | 81 | 82 | def agent_node(state, agent, name): 83 | result = agent.invoke(state) 84 | print(result["messages"][-1].content[0]["text"]) 85 | return { 86 | "messages": [ 87 | HumanMessage(content=result["messages"][-1].content[0]["text"], name=name) 88 | ] 89 | } 90 | 91 | 92 | full_workflow = StateGraph(State, config_schema=RunnableConfig) 93 | full_workflow.add_node("supervisor", supervisor_agent) 94 | 95 | full_workflow.add_edge(START, "supervisor") 96 | 97 | planner_node = functools.partial( 98 | agent_node, agent=destination_agent_graph, name="destination_agent" 99 | ) 100 | 101 | flight_node = functools.partial( 102 | agent_node, agent=flight_agent_graph, name="flight_agent" 103 | ) 104 | 105 | hotel_node = functools.partial(agent_node, agent=hotel_agent_graph, name="hotel_agent") 106 | 107 | full_workflow.add_node("destination_agent", planner_node) 108 | full_workflow.add_node("flight_agent", flight_node) 109 | full_workflow.add_node("hotel_agent", hotel_node) 110 | 111 | 112 | def process_output(state): 113 | messages = state["messages"] 114 | for message in reversed(messages): 115 | if isinstance(message, AIMessage) and isinstance(message.content, str): 116 | print(message.content) 117 | return { 118 | "messages": [HumanMessage(content=message.content, name="hotel_agent")] 119 | } 120 | return None 121 | 122 | 123 | # full_workflow.add_node("process_output", process_output) 124 | full_workflow.add_edge("destination_agent", "supervisor") 125 | full_workflow.add_edge("flight_agent", "supervisor") 126 | full_workflow.add_edge("hotel_agent", "supervisor") 127 | # full_workflow.add_edge("process_output", "supervisor") 128 | 129 | conditional_map = {k: k for k in members} 130 | conditional_map["FINISH"] = END 131 | full_workflow.add_conditional_edges("supervisor", lambda x: x["next"], conditional_map) 132 | 133 | graph = full_workflow.compile( 134 | checkpointer=memory, 135 | ) 136 | 137 | graph.name = "SupervisorAgentGraph" 138 | -------------------------------------------------------------------------------- /bedrock-multi-agent-langgraph-studio/src/hotel_agent/tools.py: -------------------------------------------------------------------------------- 1 | from datetime import datetime, timedelta 2 | from contextlib import closing 3 | import random 4 | import sqlite3 5 | from langchain_core.tools import tool 6 | 7 | 8 | @tool 9 | def suggest_hotels(city: str, checkin_date: str) -> dict: 10 | """ 11 | Use this tool to search for hotels in these cities 12 | 13 | Args: 14 | city (str): The name of the city to search for hotels 15 | checkin_date (str): The check-in date in YYYY-MM-DD format 16 | 17 | Returns: 18 | dict: A dictionary containing: 19 | - hotel (str): Name of the hotel in specified city 20 | - checkin_date (str): The provided check-in date 21 | - checkout_date (str): A randomly generated checkout date 22 | - price (int): A randomly generated price for the stay 23 | """ 24 | hotels = { 25 | "New York": ["Hotel A", "Hotel B", "Hotel C"], 26 | "Paris": ["Hotel D", "Hotel E", "Hotel F"], 27 | "Tokyo": ["Hotel G", "Hotel H", "Hotel I"], 28 | } 29 | 30 | # Generate random checkout date and price 31 | checkin = datetime.strptime(checkin_date, "%Y-%m-%d") 32 | checkout = checkin + timedelta(days=random.randint(1, 10)) 33 | price = random.randint(100, 500) 34 | 35 | hotel_list = hotels.get(city, ["No hotels found"]) 36 | return { 37 | "hotel": "City Hotel " + city, 38 | "checkin_date": checkin_date, 39 | "checkout_date": checkout.strftime("%Y-%m-%d"), 40 | "price": price, 41 | } 42 | 43 | 44 | @tool 45 | def retrieve_hotel_booking(booking_id: int) -> str: 46 | """ 47 | Retrieve a hotel booking by ID 48 | 49 | Args: 50 | booking_id (int): The unique identifier of the hotel booking to retrieve 51 | 52 | Returns: 53 | str: A string containing the hotel booking information if found, or a message indicating no booking was found 54 | """ 55 | with closing(sqlite3.connect("./data/travel_bookings.db", timeout=10.0)) as conn: 56 | with closing(conn.cursor()) as cursor: 57 | cursor.execute( 58 | "SELECT * FROM hotel_bookings WHERE booking_id=?", (booking_id,) 59 | ) 60 | booking = cursor.fetchone() 61 | 62 | if booking: 63 | return f"Booking found: {booking}" 64 | else: 65 | return f"No booking found with ID: {booking_id}" 66 | 67 | 68 | @tool 69 | def change_hotel_booking( 70 | booking_id: int, new_checkin_date: str = None, new_checkout_date: str = None 71 | ) -> str: 72 | """ 73 | Change the dates of a hotel booking in the database. If the task completes, reply with "FINISHED" 74 | 75 | Args: 76 | booking_id (int): The unique identifier of the booking to be changed 77 | new_checkin_date (str, optional): The new check-in date in YYYY-MM-DD format 78 | new_checkout_date (str, optional): The new check-out date in YYYY-MM-DD format 79 | 80 | Returns: 81 | str: A message indicating the result of the booking change operation 82 | """ 83 | 84 | with closing(sqlite3.connect("./data/travel_bookings.db", timeout=10.0)) as conn: 85 | with closing(conn.cursor()) as cursor: 86 | try: 87 | # First, fetch the current booking details 88 | cursor.execute( 89 | """ 90 | SELECT * FROM hotel_bookings WHERE booking_id = ? 91 | """, 92 | (booking_id,), 93 | ) 94 | 95 | booking = cursor.fetchone() 96 | 97 | if booking is None: 98 | return f"No hotel booking found with ID: {booking_id}" 99 | 100 | # Unpack the booking details 101 | ( 102 | _, 103 | user_id, 104 | user_name, 105 | city, 106 | hotel_name, 107 | check_in_date, 108 | check_out_date, 109 | nights, 110 | price_per_night, 111 | total_price, 112 | num_guests, 113 | room_type, 114 | ) = booking 115 | 116 | # Update check-in and check-out dates if provided 117 | if new_checkin_date: 118 | check_in_date = new_checkin_date 119 | if new_checkout_date: 120 | check_out_date = new_checkout_date 121 | 122 | # Recalculate nights and total price 123 | checkin = datetime.strptime(check_in_date, "%Y-%m-%d") 124 | checkout = datetime.strptime(check_out_date, "%Y-%m-%d") 125 | nights = (checkout - checkin).days 126 | total_price = nights * price_per_night 127 | 128 | # Update the booking in the database 129 | cursor.execute( 130 | """ 131 | UPDATE hotel_bookings 132 | SET check_in_date = ?, check_out_date = ?, nights = ?, total_price = ? 133 | WHERE booking_id = ? 134 | """, 135 | (check_in_date, check_out_date, nights, total_price, booking_id), 136 | ) 137 | 138 | conn.commit() 139 | 140 | return f"Hotel booking updated: Booking ID {booking_id}, New check-in: {check_in_date}, New check-out: {check_out_date}, Nights: {nights}, Total Price: {total_price} FINISHED" 141 | 142 | except sqlite3.Error as e: 143 | conn.rollback() 144 | return f"An error occurred: {str(e)} Booking ID {booking_id}, New check-in: {check_in_date} FINISHED" 145 | 146 | finally: 147 | pass 148 | 149 | 150 | @tool 151 | def cancel_hotel_booking(booking_id: int) -> str: 152 | """ 153 | Cancel a hotel booking. If the task completes, reply with "FINISHED" 154 | 155 | Args: 156 | booking_id (int): The unique identifier of the booking to be cancelled 157 | 158 | Returns: 159 | str: A message indicating the result of the booking cancellation operation 160 | """ 161 | with closing(sqlite3.connect("./data/travel_bookings.db", timeout=10.0)) as conn: 162 | with closing(conn.cursor()) as cursor: 163 | cursor.execute( 164 | "DELETE FROM hotel_bookings WHERE booking_id = ?", (booking_id,) 165 | ) 166 | conn.commit() 167 | 168 | # Check if the booking was deleted 169 | if cursor.rowcount > 0: 170 | return f"Booking canceled with ID: {booking_id} FINISHED" 171 | else: 172 | return f"No booking found with ID: {booking_id} FINISHED" 173 | -------------------------------------------------------------------------------- /bedrock-multi-agent-langgraph-studio/src/flight_agent/tools.py: -------------------------------------------------------------------------------- 1 | from datetime import datetime, timedelta 2 | from contextlib import closing 3 | import json 4 | import random 5 | import sqlite3 6 | import pandas as pd 7 | from langchain_core.tools import tool 8 | from langchain_core.runnables import RunnableConfig 9 | 10 | 11 | def read_travel_data(file_path: str = "data/synthetic_travel_data.csv") -> pd.DataFrame: 12 | """Read travel data from CSV file""" 13 | try: 14 | df = pd.read_csv(file_path) 15 | return df 16 | except FileNotFoundError: 17 | return pd.DataFrame( 18 | columns=[ 19 | "Id", 20 | "Name", 21 | "Current_Location", 22 | "Age", 23 | "Past_Travel_Destinations", 24 | "Number_of_Trips", 25 | "Flight_Number", 26 | "Departure_City", 27 | "Arrival_City", 28 | "Flight_Date", 29 | ] 30 | ) 31 | 32 | 33 | @tool 34 | def search_flights(config: RunnableConfig, arrival_city: str, date: str = None) -> str: 35 | """ 36 | Use this tool to search for flights between two cities. It knows the user's departure city 37 | 38 | Args: 39 | arrival_city (str): The city of arrival 40 | date (str, optional): The date of the flight in YYYY-MM-DD format. If not provided, defaults to 7 days from now. 41 | 42 | Returns: 43 | str: A formatted string containing flight information including airline, departure time, arrival time, duration, and price for multiple flights. 44 | """ 45 | 46 | df = read_travel_data() 47 | user_id = config.get("configurable", {})["configurable"]["user_id"] 48 | print(user_id) 49 | 50 | if user_id not in df["Id"].values: 51 | return "User not found in the travel database." 52 | 53 | user_data = df[df["Id"] == user_id].iloc[0] 54 | current_location = user_data["Current_Location"] 55 | 56 | departure_city = current_location.capitalize() 57 | arrival_city = arrival_city.capitalize() 58 | 59 | if date is None: 60 | date = (datetime.now() + timedelta(days=7)).strftime("%Y-%m-%d") 61 | 62 | # Generate mock flight data 63 | num_flights = random.randint(2, 5) 64 | airlines = ["AirEurope", "SkyWings", "TransContinental", "EuroJet", "GlobalAir"] 65 | flights = [] 66 | 67 | for _ in range(num_flights): 68 | airline = random.choice(airlines) 69 | duration = timedelta(minutes=2) 70 | price = random.randint(100, 400) 71 | departure_time = datetime.strptime(date, "%Y-%m-%d") + timedelta( 72 | hours=random.randint(0, 23), minutes=random.randint(0, 59) 73 | ) 74 | arrival_time = departure_time + duration 75 | 76 | flights.append( 77 | { 78 | "airline": airline, 79 | "departure": departure_time.strftime("%H:%M"), 80 | "arrival": arrival_time.strftime("%H:%M"), 81 | "duration": str(duration), 82 | "price": price, 83 | } 84 | ) 85 | 86 | # Format the results 87 | 88 | flight_data = { 89 | "departure_city": departure_city, 90 | "arrival_city": arrival_city, 91 | "date": date, 92 | "flights": [], 93 | } 94 | for i, flight in enumerate(flights, 1): 95 | flight_info = { 96 | "flight_number": i, 97 | "airline": flight["airline"], 98 | "departure": flight["departure"], 99 | "arrival": flight["arrival"], 100 | "duration": str(flight["duration"]), 101 | "price": flight["price"], 102 | } 103 | flight_data["flights"].append(flight_info) 104 | 105 | return json.dumps(flight_data) + " FINISHED" 106 | 107 | 108 | @tool 109 | def retrieve_flight_booking(booking_id: int) -> str: 110 | """ 111 | Retrieve a flight booking by ID 112 | 113 | Args: 114 | booking_id (int): The unique identifier of the booking to retrieve 115 | 116 | Returns: 117 | str: A string containing the booking information if found, or a message indicating no booking was found 118 | """ 119 | booking = None 120 | with closing(sqlite3.connect("data/travel_bookings.db", timeout=10.0)) as conn: 121 | with closing(conn.cursor()) as cursor: 122 | # Execute the query to retrieve the booking 123 | cursor.execute( 124 | "SELECT * FROM flight_bookings WHERE booking_id = ?", (booking_id,) 125 | ) 126 | booking = cursor.fetchone() 127 | # Close the connection 128 | conn.close() 129 | 130 | if booking: 131 | return f"Booking found: {booking} FINISHED" 132 | else: 133 | return f"No booking found with ID: {booking_id} FINISHED" 134 | 135 | 136 | @tool 137 | def change_flight_booking(booking_id: int, new_date: str) -> str: 138 | """ 139 | Change the date of a flight booking 140 | 141 | Args: 142 | booking_id (int): The unique identifier of the booking to be changed 143 | new_date (str): The new date for the booking 144 | 145 | Returns: 146 | str: A message indicating the result of the booking change operation 147 | """ 148 | # conn = sqlite3.connect("data/travel_bookings.db") 149 | # cursor = conn.cursor() 150 | result = "" 151 | with closing(sqlite3.connect("data/travel_bookings.db", timeout=10.0)) as conn: 152 | with closing(conn.cursor()) as cursor: 153 | # Execute the query to update the booking date 154 | cursor.execute( 155 | "UPDATE flight_bookings SET departure_date = ? WHERE booking_id = ?", 156 | (new_date, booking_id), 157 | ) 158 | conn.commit() 159 | 160 | # Check if the booking was updated 161 | if cursor.rowcount > 0: 162 | result = f"Booking updated with ID: {booking_id}, new date: {new_date} FINISHED" 163 | else: 164 | result = f"No booking found with ID: {booking_id} FINISHED" 165 | 166 | # Close the connection 167 | conn.close() 168 | 169 | return result 170 | 171 | 172 | @tool 173 | def cancel_flight_booking(booking_id: int) -> str: 174 | """ 175 | Cancel a flight booking. If the task complete, reply with "FINISHED" 176 | 177 | Args: 178 | booking_id (str): The unique identifier of the booking to be cancelled 179 | 180 | Returns: 181 | str: A message indicating the result of the booking cancellation operation 182 | 183 | """ 184 | # conn = sqlite3.connect("data/travel_bookings.db") 185 | # cursor = conn.cursor() 186 | result = "" 187 | with closing(sqlite3.connect("data/travel_bookings.db", timeout=10.0)) as conn: 188 | with closing(conn.cursor()) as cursor: 189 | cursor.execute( 190 | "DELETE FROM flight_bookings WHERE booking_id = ?", (booking_id,) 191 | ) 192 | conn.commit() 193 | # Check if the booking was deleted 194 | if cursor.rowcount > 0: 195 | result = f"Booking canceled with ID: {booking_id} FINISHED" 196 | else: 197 | result = f"No booking found with ID: {booking_id} FINISHED" 198 | 199 | # Close the connection 200 | conn.close() 201 | 202 | return result 203 | -------------------------------------------------------------------------------- /utils.py: -------------------------------------------------------------------------------- 1 | import pandas as pd 2 | import boto3 3 | import pickle 4 | 5 | from collections import Counter 6 | from langchain_core.tools import tool 7 | from langchain_core.runnables.config import RunnableConfig 8 | from langchain_aws import ChatBedrockConverse 9 | from langchain_text_splitters import RecursiveCharacterTextSplitter 10 | from langchain_aws.embeddings.bedrock import BedrockEmbeddings 11 | from langgraph.prebuilt import create_react_agent 12 | from langgraph.checkpoint.memory import MemorySaver 13 | from langchain.tools.retriever import create_retriever_tool 14 | from langchain_community.vectorstores import FAISS 15 | from langchain.retrievers import ParentDocumentRetriever 16 | 17 | 18 | from ragas.messages import HumanMessage as RGHumanMessage 19 | from ragas.messages import AIMessage as RGAIMessage 20 | from ragas.messages import ToolMessage as RGToolMessage 21 | from ragas.messages import ToolCall as RGToolCall 22 | 23 | from io import BytesIO 24 | 25 | def convert_message_langchain_to_ragas(lc_message): 26 | message_dict = lc_message.model_dump() 27 | if message_dict['type'] == 'human': 28 | rg_message = RGHumanMessage(content=message_dict['content']) 29 | if message_dict['type'] == 'ai': 30 | if type(message_dict['content']) == list: 31 | text = list(filter((lambda x: x['type'] == 'text'), message_dict['content'])) 32 | tool = list(filter((lambda x: x['type'] == 'tool_use'), message_dict['content'])) 33 | if len(text) > 0 and len(tool) > 0: 34 | 35 | if len(list(tool[0]['input'].keys())) > 0: 36 | dyn_args = {'query': tool[0]['input'][list(tool[0]['input'].keys())[0]]} 37 | else: 38 | dyn_args = {} 39 | 40 | rg_message = RGAIMessage(content=text[0]['text'], tool_calls=[RGToolCall(name=tool[0]['name'], args= dyn_args)]) 41 | elif len(text) > 0: 42 | rg_message = RGAIMessage(content=text[0]['text']) 43 | elif len(tool) > 0: 44 | rg_message = RGAIMessage(content='', tool_calls=[RGToolCall(name=tool[0]['name'], args={#'id': tool[0]['id'], 45 | 'query': tool[0]['input'][list(tool[0]['input'].keys())[0]]})]) 46 | else: 47 | rg_message = RGAIMessage(content= message_dict['content'], tool_calls=message_dict['tool_calls'], metadata=message_dict['usage_metadata']) 48 | if message_dict['type'] == 'tool': 49 | rg_message = RGToolMessage(content=message_dict['content'], metadata={"tool_name": message_dict['name'], "tool_call_id": message_dict['tool_call_id']}) 50 | return rg_message 51 | 52 | 53 | def create_agent(enable_memory = False): 54 | # ---- ⚠️ Update region for your AWS setup ⚠️ ---- 55 | bedrock_client = boto3.client("bedrock-runtime", region_name="us-west-2") 56 | 57 | 58 | 59 | llm = ChatBedrockConverse( 60 | model="anthropic.claude-3-haiku-20240307-v1:0", 61 | temperature=0, 62 | max_tokens=None, 63 | client=bedrock_client, 64 | # other params... 65 | ) 66 | 67 | def read_travel_data(file_path: str = "data/synthetic_travel_data.csv") -> pd.DataFrame: 68 | """Read travel data from CSV file""" 69 | try: 70 | df = pd.read_csv(file_path) 71 | return df 72 | except FileNotFoundError: 73 | return pd.DataFrame( 74 | columns=["Id", "Name","Current_Location","Age","Past_Travel_Destinations", "Number_of_Trips", "Flight_Number", "Departure_City","Arrival_City","Flight_Date",] 75 | ) 76 | 77 | 78 | @tool 79 | def compare_and_recommend_destination(config: RunnableConfig) -> str: 80 | """This tool is used to check which destinations user has already traveled. 81 | If user has already been to a city then do not recommend that city. 82 | 83 | Returns: 84 | str: Destination to be recommended. 85 | 86 | """ 87 | 88 | df = read_travel_data() 89 | user_id = config.get("configurable", {}).get("user_id") 90 | 91 | if user_id not in df["Id"].values: 92 | return "User not found in the travel database." 93 | 94 | user_data = df[df["Id"] == user_id].iloc[0] 95 | current_location = user_data["Current_Location"] 96 | age = user_data["Age"] 97 | past_destinations = user_data["Past_Travel_Destinations"].split(", ") 98 | 99 | # Get all past destinations of users with similar age (±5 years) and same current location 100 | similar_users = df[(df["Current_Location"] == current_location) & (df["Age"].between(age - 5, age + 5))] 101 | all_destinations = [dest for user_dests in similar_users["Past_Travel_Destinations"].str.split(", ") for dest in user_dests ] 102 | 103 | # Count occurrences of each destination 104 | destination_counts = Counter(all_destinations) 105 | 106 | # Remove user's current location and past destinations from recommendations 107 | for dest in [current_location] + past_destinations: 108 | if dest in destination_counts: 109 | del destination_counts[dest] 110 | 111 | if not destination_counts: 112 | return f"No new recommendations found for users in {current_location} with similar age." 113 | 114 | # Get the most common destination 115 | recommended_destination = destination_counts.most_common(1)[0][0] 116 | 117 | return f"Based on your current location ({current_location}), age ({age}), and past travel data, we recommend visiting {recommended_destination}." 118 | 119 | 120 | embeddings_model = BedrockEmbeddings( 121 | client=bedrock_client, model_id="amazon.titan-embed-text-v1" 122 | ) 123 | 124 | child_splitter = RecursiveCharacterTextSplitter( 125 | separators=["\n", "\n\n"], chunk_size=2000, chunk_overlap=250 126 | ) 127 | 128 | in_memory_store_file = "data/section_doc_store.pkl" 129 | vector_store_file = "data/section_vector_store.pkl" 130 | 131 | store = pickle.load(open(in_memory_store_file, "rb")) 132 | vector_db_buff = BytesIO(pickle.load(open(vector_store_file, "rb"))) 133 | vector_db = FAISS.deserialize_from_bytes( 134 | serialized=vector_db_buff.read(), 135 | embeddings=embeddings_model, 136 | allow_dangerous_deserialization=True, 137 | ) 138 | 139 | retriever = ParentDocumentRetriever( 140 | vectorstore=vector_db, 141 | docstore=store, 142 | child_splitter=child_splitter, 143 | search_kwargs={"k":1} 144 | ) 145 | 146 | retriever_tool = create_retriever_tool( 147 | retriever, 148 | "travel_guide", 149 | """Holds information from travel guide books containing city details to find information matching the user's interests in various cities. Only search based on the keyword mentioned in user input. 150 | 151 | Args: 152 | query (str): place to query travel guide. 153 | Returns: 154 | str: Information about destination from travel guide. 155 | 156 | """, 157 | ) 158 | 159 | tools = [compare_and_recommend_destination, retriever_tool] 160 | 161 | if enable_memory: 162 | 163 | memory = MemorySaver() 164 | agent = create_react_agent(llm, tools, checkpointer = memory) 165 | 166 | else: 167 | agent = create_react_agent(llm, tools) 168 | 169 | return agent -------------------------------------------------------------------------------- /05_Optional_find_dream_destination_with_crewaI.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "d43f5f94e98f67ce", 6 | "metadata": {}, 7 | "source": [ 8 | "# Dream Destination Finder with CrewAI and Amazon Bedrock\n", 9 | "\n", 10 | "In this notebook, we will explore how to use the CrewAI framework with Amazon Bedrock to build an intelligent agent that can find dream travel destinations based on user preferences. The agent will utilize a large language model (LLM) and web search capabilities to research and recommend destinations that match the user's description." 11 | ] 12 | }, 13 | { 14 | "cell_type": "markdown", 15 | "id": "19f933cf500287b", 16 | "metadata": {}, 17 | "source": [ 18 | "### Prerequisites\n", 19 | "\n", 20 | "Before we begin, make sure you have the following installed:\n", 21 | "`boto3` and `botocore` for interacting with AWS services\n", 22 | "`crewai` and `crewai_tools` for building agentic workflows\n", 23 | "\n", 24 | "**Un-comment these cells below and after they run please make sure to re-start the kernel**\n", 25 | "\n", 26 | "Please ignore any pip install errors." 27 | ] 28 | }, 29 | { 30 | "cell_type": "code", 31 | "execution_count": null, 32 | "id": "ca7f66e2d4db085e", 33 | "metadata": { 34 | "ExecuteTime": { 35 | "end_time": "2024-11-23T18:11:21.882632Z", 36 | "start_time": "2024-11-23T18:11:21.880054Z" 37 | } 38 | }, 39 | "outputs": [], 40 | "source": [ 41 | "!pip install boto3==1.34.162 botocore==1.34.162 crewai==0.70.1 crewai_tools==0.12.1 duckduckgo-search==6.3.1 unstructured==0.16.6 PyPDF2==3.0.1 -q \n" 42 | ] 43 | }, 44 | { 45 | "cell_type": "code", 46 | "execution_count": null, 47 | "id": "b5e272bc", 48 | "metadata": {}, 49 | "outputs": [], 50 | "source": [ 51 | "!pip install langchain_aws==0.1.17 --force-reinstall -q" 52 | ] 53 | }, 54 | { 55 | "cell_type": "markdown", 56 | "id": "70c4a142", 57 | "metadata": {}, 58 | "source": [ 59 | "#### Please be aware !\n", 60 | "\n", 61 | "Before proceeding please re start the kernel\n", 62 | "\n", 63 | "The version of Crew.ai installs a version of langchain which is in-compatible with the previous labs. If you want to run the previous labs you wil need to force-reinstall the langchain version from Lab 1" 64 | ] 65 | }, 66 | { 67 | "cell_type": "markdown", 68 | "id": "98732d9738f040a1", 69 | "metadata": {}, 70 | "source": [ 71 | "#### Crew.ai\n", 72 | "\n", 73 | "To use bedrock we will use [__CrewAI__ __LLM__ api](https://docs.crewai.com/how-to/llm-connections#supported-providers) " 74 | ] 75 | }, 76 | { 77 | "cell_type": "code", 78 | "execution_count": null, 79 | "id": "fa8b1e2e8d3e694b", 80 | "metadata": { 81 | "ExecuteTime": { 82 | "end_time": "2024-11-23T18:11:29.294184Z", 83 | "start_time": "2024-11-23T18:11:26.717248Z" 84 | } 85 | }, 86 | "outputs": [], 87 | "source": [ 88 | "from crewai import Agent, Task, Crew, LLM\n", 89 | "from crewai_tools import tool\n", 90 | "from langchain_community.tools import DuckDuckGoSearchRun" 91 | ] 92 | }, 93 | { 94 | "cell_type": "markdown", 95 | "id": "94b7fd26fddd54cf", 96 | "metadata": {}, 97 | "source": [ 98 | "#### Define web-search tool" 99 | ] 100 | }, 101 | { 102 | "cell_type": "code", 103 | "execution_count": null, 104 | "id": "60c9a9e2b41b68b1", 105 | "metadata": { 106 | "ExecuteTime": { 107 | "end_time": "2024-11-23T18:11:31.718999Z", 108 | "start_time": "2024-11-23T18:11:31.710378Z" 109 | } 110 | }, 111 | "outputs": [], 112 | "source": [ 113 | "@tool('DuckDuckGoSearch')\n", 114 | "def search(search_query: str):\n", 115 | " \"\"\"Search the web for information on a given topic\"\"\"\n", 116 | " return DuckDuckGoSearchRun().run(search_query)" 117 | ] 118 | }, 119 | { 120 | "cell_type": "markdown", 121 | "id": "f5f166514cb8a9cf", 122 | "metadata": {}, 123 | "source": [ 124 | "### Configuring the LLM\n", 125 | "\n", 126 | "We will use Anthropic’s Claude-3 model via Amazon Bedrock as our LLM. CrewAI uses LiteLLM under the hood to interact with different LLM providers.\n" 127 | ] 128 | }, 129 | { 130 | "cell_type": "code", 131 | "execution_count": null, 132 | "id": "2ab86d9da488f3c4", 133 | "metadata": { 134 | "ExecuteTime": { 135 | "end_time": "2024-11-23T18:11:32.100613Z", 136 | "start_time": "2024-11-23T18:11:32.099038Z" 137 | } 138 | }, 139 | "outputs": [], 140 | "source": [ 141 | "# Configure the LLM\n", 142 | "llm = LLM(model=\"bedrock/anthropic.claude-3-haiku-20240307-v1:0\")" 143 | ] 144 | }, 145 | { 146 | "cell_type": "markdown", 147 | "id": "8cb65fb92cb1327", 148 | "metadata": {}, 149 | "source": [ 150 | "### Defining the Agent\n", 151 | "\n", 152 | "We will create an agent with the role of a “Travel Destination Researcher.” This agent will be responsible for finding destinations that match the user’s travel preferences." 153 | ] 154 | }, 155 | { 156 | "cell_type": "code", 157 | "execution_count": null, 158 | "id": "c10f2b040b544201", 159 | "metadata": { 160 | "ExecuteTime": { 161 | "end_time": "2024-11-23T18:11:32.495856Z", 162 | "start_time": "2024-11-23T18:11:32.491943Z" 163 | } 164 | }, 165 | "outputs": [], 166 | "source": [ 167 | "# Define the Agent\n", 168 | "travel_agent = Agent(\n", 169 | " role='Travel Destination Researcher',\n", 170 | " goal='Find dream destinations matching user preferences',\n", 171 | " backstory=\"You are an experienced travel agent specializing in personalized travel recommendations.\",\n", 172 | " verbose=True,\n", 173 | " allow_delegation=False,\n", 174 | " llm=llm,\n", 175 | " tools=[search] # Tool for online searching\n", 176 | ")" 177 | ] 178 | }, 179 | { 180 | "cell_type": "markdown", 181 | "id": "f4ad2d2a57c8eb1f", 182 | "metadata": {}, 183 | "source": [ 184 | "### Defining the Task\n", 185 | "\n", 186 | "We need to specify the task that the agent will perform. The task includes a description, expected output, and is assigned to the agent we just created." 187 | ] 188 | }, 189 | { 190 | "cell_type": "code", 191 | "execution_count": null, 192 | "id": "4070f14c83088029", 193 | "metadata": { 194 | "ExecuteTime": { 195 | "end_time": "2024-11-23T18:11:32.826336Z", 196 | "start_time": "2024-11-23T18:11:32.824435Z" 197 | } 198 | }, 199 | "outputs": [], 200 | "source": [ 201 | "# Define the Task\n", 202 | "task = Task(\n", 203 | " description=\"Based on the user's travel preferences: {preferences}, research and recommend suitable travel destinations.\",\n", 204 | " expected_output=\"A list of recommended destinations with brief descriptions.\",\n", 205 | " agent=travel_agent\n", 206 | ")" 207 | ] 208 | }, 209 | { 210 | "cell_type": "markdown", 211 | "id": "4294563262fbb901", 212 | "metadata": {}, 213 | "source": [ 214 | "### Creating the Crew\n", 215 | "\n", 216 | "A crew is a team of agents working together to achieve a common goal. In this case, we have only one agent, but the framework allows for scalability.\n" 217 | ] 218 | }, 219 | { 220 | "cell_type": "code", 221 | "execution_count": null, 222 | "id": "494d5f9cc3497974", 223 | "metadata": { 224 | "ExecuteTime": { 225 | "end_time": "2024-11-23T18:11:33.213570Z", 226 | "start_time": "2024-11-23T18:11:33.209168Z" 227 | } 228 | }, 229 | "outputs": [], 230 | "source": [ 231 | "# Create the Crew\n", 232 | "crew = Crew(\n", 233 | " agents=[travel_agent],\n", 234 | " tasks=[task],\n", 235 | " verbose=True,\n", 236 | ")" 237 | ] 238 | }, 239 | { 240 | "cell_type": "markdown", 241 | "id": "b966775d7ef0fd3d", 242 | "metadata": {}, 243 | "source": [ 244 | "### Executing the Workflow\n", 245 | "\n", 246 | "Now, we can execute the crew with the user’s travel preferences as input." 247 | ] 248 | }, 249 | { 250 | "cell_type": "code", 251 | "execution_count": null, 252 | "id": "4eedbc6081480112", 253 | "metadata": { 254 | "ExecuteTime": { 255 | "end_time": "2024-11-23T18:11:41.271081Z", 256 | "start_time": "2024-11-23T18:11:33.527509Z" 257 | } 258 | }, 259 | "outputs": [], 260 | "source": [ 261 | "# User input for travel preferences\n", 262 | "user_input = {\n", 263 | " \"preferences\": \"I want a tropical beach vacation with great snorkeling and vibrant nightlife.\"\n", 264 | "}\n", 265 | "\n", 266 | "# Execute the Crew\n", 267 | "result = crew.kickoff(inputs=user_input)" 268 | ] 269 | }, 270 | { 271 | "cell_type": "markdown", 272 | "id": "c4d2e1cc9f6242c6", 273 | "metadata": {}, 274 | "source": [ 275 | "#### As the crew executes, CrewAI will:\n", 276 | "\n", 277 | "•\tDecompose the task into actions using ReAct (Reasoning and Act), optionally using the tools assigned to the agent.\n", 278 | "\n", 279 | "•\tMake multiple calls to Amazon Bedrock to complete each step from the previous phase." 280 | ] 281 | }, 282 | { 283 | "cell_type": "code", 284 | "execution_count": null, 285 | "id": "d8d3731d0e1a4849", 286 | "metadata": { 287 | "ExecuteTime": { 288 | "end_time": "2024-11-23T18:11:41.283560Z", 289 | "start_time": "2024-11-23T18:11:41.281784Z" 290 | } 291 | }, 292 | "outputs": [], 293 | "source": [ 294 | "from IPython.display import Markdown" 295 | ] 296 | }, 297 | { 298 | "cell_type": "code", 299 | "execution_count": null, 300 | "id": "80d8cf08c6cbb95b", 301 | "metadata": { 302 | "ExecuteTime": { 303 | "end_time": "2024-11-23T18:11:41.292671Z", 304 | "start_time": "2024-11-23T18:11:41.289243Z" 305 | } 306 | }, 307 | "outputs": [], 308 | "source": [ 309 | "Markdown(result.raw)" 310 | ] 311 | }, 312 | { 313 | "cell_type": "markdown", 314 | "id": "32f45c9ae59ef17a", 315 | "metadata": {}, 316 | "source": [ 317 | "### Adding Memory to the Agent\n", 318 | "CrewAI supports [several memory types](https://docs.crewai.com/concepts/memory#implementing-memory-in-your-crew), which help agents remember and learn from past interactions. In this case, we’ll enable short-term memory using Amazon Bedrock’s embedding model." 319 | ] 320 | }, 321 | { 322 | "cell_type": "code", 323 | "execution_count": null, 324 | "id": "28dfe7f092e2c4c2", 325 | "metadata": { 326 | "ExecuteTime": { 327 | "end_time": "2024-11-23T18:11:41.307134Z", 328 | "start_time": "2024-11-23T18:11:41.301227Z" 329 | } 330 | }, 331 | "outputs": [], 332 | "source": [ 333 | "# Enabling Memory in the Agent\n", 334 | "crew_with_memory = Crew(\n", 335 | " agents=[travel_agent],\n", 336 | " tasks=[task],\n", 337 | " verbose=True,\n", 338 | " memory=True, # Enable memory\n", 339 | " embedder={\n", 340 | " \"provider\": \"aws_bedrock\",\n", 341 | " \"config\": {\n", 342 | " \"model\": \"amazon.titan-embed-text-v1\", # Embedding model for memory\n", 343 | " \"vector_dimension\": 1024\n", 344 | " }\n", 345 | " },\n", 346 | " \n", 347 | ")" 348 | ] 349 | }, 350 | { 351 | "cell_type": "code", 352 | "execution_count": null, 353 | "id": "ae235e0458732b54", 354 | "metadata": { 355 | "ExecuteTime": { 356 | "end_time": "2024-11-23T18:11:50.303578Z", 357 | "start_time": "2024-11-23T18:11:41.320688Z" 358 | } 359 | }, 360 | "outputs": [], 361 | "source": [ 362 | "# Executing the Crew with Memory\n", 363 | "result_with_memory = crew_with_memory.kickoff(inputs=user_input)" 364 | ] 365 | }, 366 | { 367 | "cell_type": "code", 368 | "execution_count": null, 369 | "id": "b3c491409c571d1", 370 | "metadata": { 371 | "ExecuteTime": { 372 | "end_time": "2024-11-23T18:11:50.320251Z", 373 | "start_time": "2024-11-23T18:11:50.316090Z" 374 | } 375 | }, 376 | "outputs": [], 377 | "source": [ 378 | "Markdown(result_with_memory.raw)" 379 | ] 380 | }, 381 | { 382 | "cell_type": "markdown", 383 | "id": "2affdfc71f1de059", 384 | "metadata": {}, 385 | "source": [ 386 | "### Integrating Retrieval-Augmented Generation (RAG) with Amazon Bedrock Knowledge Base\n", 387 | "In this section, we will enhance our dream destination finder agent by incorporating Retrieval-Augmented Generation (RAG) using Amazon Bedrock’s Knowledge Base. This will allow our agent to access up-to-date and domain-specific travel information, improving the accuracy and relevance of its recommendations.\n", 388 | "\n" 389 | ] 390 | }, 391 | { 392 | "cell_type": "markdown", 393 | "id": "f06dd6a10735223e", 394 | "metadata": {}, 395 | "source": [ 396 | "#### What is Retrieval-Augmented Generation (RAG)?\n", 397 | "\n", 398 | "RAG is a technique that combines the capabilities of large language models (LLMs) with a retrieval mechanism to fetch relevant information from external data sources. By integrating RAG, our agent can retrieve the most recent and specific information from a knowledge base, overcoming the limitations of LLMs that may have outdated or insufficient data." 399 | ] 400 | }, 401 | { 402 | "cell_type": "markdown", 403 | "id": "16a7b57d238399b8", 404 | "metadata": {}, 405 | "source": [ 406 | "Setting Up Amazon Bedrock Knowledge Base\n", 407 | "\n", 408 | "Before we proceed, ensure you have access to Amazon Bedrock and the necessary permissions to create and manage knowledge bases.\n", 409 | "\n", 410 | "* Step 1: Prepare Your Data\n", 411 | "* Step 2: Create a Knowledge Base in Amazon Bedrock\n", 412 | "* Step 3: Note the Knowledge Base ID\n", 413 | "\n", 414 | "After the knowledge base is created, note down its Knowledge Base ID (kb_id), which will be used in our code.\n", 415 | "\n", 416 | "Updating the Agent to Use RAG with CrewAI\n", 417 | "\n", 418 | "We will modify our agent to include a custom tool that queries the Amazon Bedrock Knowledge Base. This allows the agent to retrieve up-to-date information during its reasoning process." 419 | ] 420 | }, 421 | { 422 | "cell_type": "markdown", 423 | "id": "6874e780feb1379d", 424 | "metadata": {}, 425 | "source": [ 426 | "### FAIS Vector Store Set up:" 427 | ] 428 | }, 429 | { 430 | "cell_type": "code", 431 | "execution_count": null, 432 | "id": "4522b1fed3011ffd", 433 | "metadata": { 434 | "ExecuteTime": { 435 | "end_time": "2024-11-23T18:15:34.898140Z", 436 | "start_time": "2024-11-23T18:15:30.879768Z" 437 | } 438 | }, 439 | "outputs": [], 440 | "source": [ 441 | "import os\n", 442 | "\n", 443 | "\n", 444 | "from uuid import uuid4\n", 445 | "from PyPDF2 import PdfReader\n", 446 | "from langchain.schema import Document\n", 447 | "from langchain.vectorstores import FAISS\n", 448 | "from langchain_aws import BedrockEmbeddings\n", 449 | "\n", 450 | "documents = []\n", 451 | "pdf_folder = './data/us/'\n", 452 | "\n", 453 | "# Loop through PDFs in the specified folder\n", 454 | "for pdf_file in os.listdir(pdf_folder):\n", 455 | " if pdf_file.endswith(\".pdf\"):\n", 456 | " file_path = os.path.join(pdf_folder, pdf_file)\n", 457 | " \n", 458 | " # Extract text from PDF\n", 459 | " reader = PdfReader(file_path)\n", 460 | " text_content = \"\"\n", 461 | " for page in reader.pages:\n", 462 | " text_content += page.extract_text() + \"\\n\"\n", 463 | " \n", 464 | " # Create a Document instance\n", 465 | " doc = Document(\n", 466 | " page_content=text_content.strip(),\n", 467 | " metadata={} # Leave metadata empty for now\n", 468 | " )\n", 469 | " documents.append(doc)\n", 470 | "\n", 471 | "# Initialize FAISS vector store and embeddings\n", 472 | "embeddings = BedrockEmbeddings(model_id=\"amazon.titan-embed-text-v1\")\n", 473 | "vector_store = FAISS.from_documents(documents, embeddings)\n", 474 | "\n", 475 | "# Add unique IDs to documents and save the vector store\n", 476 | "uuids = [str(uuid4()) for _ in range(len(documents))]\n", 477 | "vector_store.add_documents(documents=documents, ids=uuids)" 478 | ] 479 | }, 480 | { 481 | "cell_type": "code", 482 | "execution_count": null, 483 | "id": "d7b94876406ba39c", 484 | "metadata": { 485 | "ExecuteTime": { 486 | "end_time": "2024-11-23T18:15:34.976601Z", 487 | "start_time": "2024-11-23T18:15:34.973828Z" 488 | } 489 | }, 490 | "outputs": [], 491 | "source": [ 492 | "@tool(\"TravelExpertSearchEngine\")\n", 493 | "def query_knowledge_base(question: str) -> str:\n", 494 | " \"\"\"Queries the Amazon Bedrock Knowledge Base for travel-related information.\"\"\"\n", 495 | " try:\n", 496 | " res = vector_store.similarity_search(\n", 497 | " question,\n", 498 | " k=1,\n", 499 | " ) \n", 500 | " return res[0].page_content\n", 501 | " except KeyError:\n", 502 | " return \"No data available\"\n", 503 | "\n" 504 | ] 505 | }, 506 | { 507 | "cell_type": "markdown", 508 | "id": "7dfa59790a236991", 509 | "metadata": {}, 510 | "source": [ 511 | "### Update the Agent with the New Tool\n", 512 | "We will update our agent to include the TravelExpert tool." 513 | ] 514 | }, 515 | { 516 | "cell_type": "code", 517 | "execution_count": null, 518 | "id": "3b4bba4439a89bc5", 519 | "metadata": { 520 | "ExecuteTime": { 521 | "end_time": "2024-11-23T18:15:35.485173Z", 522 | "start_time": "2024-11-23T18:15:35.481272Z" 523 | } 524 | }, 525 | "outputs": [], 526 | "source": [ 527 | "# Update the Agent\n", 528 | "agent_with_rag = Agent(\n", 529 | " role='Travel Destination Researcher',\n", 530 | " goal='Find dream destinations in the USA using only the travel guide available, first lookup cities using the tool to match user preferences and then use information from the search engine, nothing else.',\n", 531 | " backstory=\"\"\"You are an experienced travel agent specializing in personalized travel recommendations. \n", 532 | " Your approach is as follows: \n", 533 | " Deduce which regions within the USA will have those activities listed by the user.\n", 534 | " List major cities within that region\n", 535 | " Only then use the tool provided to look up information, look up should be done by passing city highlights and activities.\n", 536 | " Only suggest places that were extracted using the lookup tool,\n", 537 | " \"\"\",\n", 538 | " verbose=True,\n", 539 | " allow_delegation=False,\n", 540 | " llm=llm,\n", 541 | " tools=[query_knowledge_base], # Include the RAG tool\n", 542 | " max_iter=3\n", 543 | ")\n" 544 | ] 545 | }, 546 | { 547 | "cell_type": "markdown", 548 | "id": "cd2594a6df73dc04", 549 | "metadata": {}, 550 | "source": [ 551 | "### Update the task and set up the Crew" 552 | ] 553 | }, 554 | { 555 | "cell_type": "code", 556 | "execution_count": null, 557 | "id": "5eb1e00ef701da8f", 558 | "metadata": { 559 | "ExecuteTime": { 560 | "end_time": "2024-11-23T18:15:37.628624Z", 561 | "start_time": "2024-11-23T18:15:37.622076Z" 562 | } 563 | }, 564 | "outputs": [], 565 | "source": [ 566 | "# Define the Task\n", 567 | "task_with_rag = Task(\n", 568 | " description=\"Based on the user's travel request, research and recommend suitable travel destinations using the latest information. Only use output provided by the Travel Destination Researcher, nothing else: USER: {preferences}\",\n", 569 | " expected_output=\"A place where they can travel to along with recommendations on what to see and do while there.\",\n", 570 | " agent=agent_with_rag\n", 571 | ")\n", 572 | "\n", 573 | "\n", 574 | "# Create the Crew\n", 575 | "crew_with_rag = Crew(\n", 576 | " agents=[agent_with_rag],\n", 577 | " tasks=[task_with_rag],\n", 578 | " verbose=True,\n", 579 | ")" 580 | ] 581 | }, 582 | { 583 | "cell_type": "code", 584 | "execution_count": null, 585 | "id": "67f54c08a7f997fa", 586 | "metadata": { 587 | "ExecuteTime": { 588 | "end_time": "2024-11-23T18:15:44.733535Z", 589 | "start_time": "2024-11-23T18:15:38.058190Z" 590 | } 591 | }, 592 | "outputs": [], 593 | "source": [ 594 | "# User input for travel preferences\n", 595 | "user_input = {\n", 596 | " \"preferences\": \"Where can I go for cowboy vibes, watch a rodeo, and a visit museums?\"\n", 597 | "}\n", 598 | "\n", 599 | "# Execute the Crew\n", 600 | "result_with_rag = crew_with_rag.kickoff(inputs=user_input)\n" 601 | ] 602 | }, 603 | { 604 | "cell_type": "markdown", 605 | "id": "f6f1ac2d30762c81", 606 | "metadata": {}, 607 | "source": [ 608 | "### Display the results" 609 | ] 610 | }, 611 | { 612 | "cell_type": "code", 613 | "execution_count": null, 614 | "id": "f10d84c9acdc730e", 615 | "metadata": { 616 | "ExecuteTime": { 617 | "end_time": "2024-11-23T18:15:44.743348Z", 618 | "start_time": "2024-11-23T18:15:44.740838Z" 619 | } 620 | }, 621 | "outputs": [], 622 | "source": [ 623 | "# Display the result\n", 624 | "Markdown(result_with_rag.raw)" 625 | ] 626 | }, 627 | { 628 | "cell_type": "code", 629 | "execution_count": null, 630 | "id": "a9ea7d8219641f54", 631 | "metadata": {}, 632 | "outputs": [], 633 | "source": [] 634 | } 635 | ], 636 | "metadata": { 637 | "kernelspec": { 638 | "display_name": "Python 3 (ipykernel)", 639 | "language": "python", 640 | "name": "python3" 641 | }, 642 | "language_info": { 643 | "codemirror_mode": { 644 | "name": "ipython", 645 | "version": 3 646 | }, 647 | "file_extension": ".py", 648 | "mimetype": "text/x-python", 649 | "name": "python", 650 | "nbconvert_exporter": "python", 651 | "pygments_lexer": "ipython3", 652 | "version": "3.11.10" 653 | } 654 | }, 655 | "nbformat": 4, 656 | "nbformat_minor": 5 657 | } 658 | -------------------------------------------------------------------------------- /01_usecase_introduction.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "873b300b-57df-416b-867b-e2a3120b2ed0", 6 | "metadata": {}, 7 | "source": [ 8 | "## Lab 1: AI Travel Assistant Use Case Introduction\n", 9 | "\n" 10 | ] 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "id": "7c436186-5d8a-4c94-824d-98cd353a73e9", 15 | "metadata": {}, 16 | "source": [ 17 | "### Introduction" 18 | ] 19 | }, 20 | { 21 | "cell_type": "markdown", 22 | "id": "c67d0005-114e-4ad4-bcad-d52030415bee", 23 | "metadata": { 24 | "tags": [] 25 | }, 26 | "source": [ 27 | "In this first Lab, we will introduce the AI Travel Assistant use case, discuss the data sources, core features of AI Agents with Bedrock and Open Source Agent Framework LangChain and set up the packages and dependencies that is required for the rest of the Labs.\n", 28 | "\n", 29 | "The use case is to help create a bot which \n", 30 | "- Find `My Dream Destination` which finds similiar destinations based on the user preferences using `RAG`\n", 31 | "- Can `book travel` including hotel and flight\n", 32 | "- Leverages `Tools` to exeute the actions\n", 33 | "- Asks for `Human Confirmation` for critical exeution flows\n", 34 | "\n", 35 | "\n" 36 | ] 37 | }, 38 | { 39 | "cell_type": "markdown", 40 | "id": "245ab8cc-75e4-4613-9d1a-125ff22e2780", 41 | "metadata": {}, 42 | "source": [ 43 | "### Use Case" 44 | ] 45 | }, 46 | { 47 | "cell_type": "markdown", 48 | "id": "c2ad3e7e-f713-45f1-bb28-06c3c254fece", 49 | "metadata": {}, 50 | "source": [ 51 | "Imagine having an AI assistant that can help you plan your next vacation, catering to all your preferences and needs. That's exactly what our \"AI Travel Assistant\" does—a cutting-edge, agentic AI system built using LangGraph and Amazon Bedrock Models. This assistant is designed to make travel planning seamless, offering personalized and data-driven suggestions every step of the way.\n", 52 | "\n", 53 | "We will cover following scenarios in this workshop\n", 54 | "\n", 55 | "1. **Lab 2: Introduction to the concepts of Agents including building blocks for LangGrah and ReACT agents. We will introduce memory concepts in this lab to show how to have 'Multi-Turn' conversations. We will show short term and long term memory:** \n", 56 | "\n", 57 | "Understand what it takes to build a agentic system.\n", 58 | "\n", 59 | "2. **Lab 3: Dream Destination Planning with Introduction of RAG to the pipeline for Customized Travel Itineraries. We will show different types of Agents incuding concepts like 'Human-In-The-Loop': We will also introduce Multi-Modal concept in this lab** \n", 60 | "\n", 61 | "Get personalized recommendations for your next dream destination. Create vacation plans tailored specifically to your preferences and based on your past travels.\n", 62 | "\n", 63 | "3. **Lab 4: Multi-Agent colloboration. We will leverage all the work from the previous labs and then assemble the building blocks to create a multi agent colloboration. We will use Similar Travelers Based Recommendations, book hotels and flights:** \n", 64 | "\n", 65 | "Receive recommendations influenced by the preferences of travelers in a similar age group and geographic location.Book, modify, and cancel flight and hotel reservations—all through one assistant.\n", 66 | "\n", 67 | "4. **Lab 5: [Optional] Dream Destination with crew.ai for comparisions:** \n", 68 | "\n", 69 | "Book, modify, and cancel flight and hotel reservations—all through one assistant.\n", 70 | "\n", 71 | "5. **Lab 6: Agentic Evaluation with RAGAS:** \n", 72 | "\n", 73 | "Understand how to evaluate the Agentic workflows using the open soyrce RAGAS library.\n", 74 | "\n", 75 | "\n", 76 | "The diagram will be similiar to this below\n", 77 | "\n", 78 | "travel_agent_light.png " 79 | ] 80 | }, 81 | { 82 | "cell_type": "markdown", 83 | "id": "ab8ab837-5ca9-4658-8e97-f732f6b05c98", 84 | "metadata": { 85 | "tags": [] 86 | }, 87 | "source": [ 88 | "### Dataset\n", 89 | "\n", 90 | "Introduction to the data set we wil be using for this workshop" 91 | ] 92 | }, 93 | { 94 | "cell_type": "markdown", 95 | "id": "dcad55ec-0299-4737-93fe-54767f697c46", 96 | "metadata": {}, 97 | "source": [ 98 | "#### 1. Travel History: \n", 99 | "\n", 100 | "We have synthetically created travel history dataset we will levearge in this workshop, it contains following fields\n", 101 | "\n", 102 | "- Name: The individual's full name.\n", 103 | "- Current_Location: The city where the individual currently resides.\n", 104 | "- Age: The individual's age in years.\n", 105 | "- Past_Travel_Destinations: Cities the individual has previously visited, listed as comma-separated values.\n", 106 | "- Number_of_Trips: The total count of trips the individual has taken.\n", 107 | "- Flight_Number: The flight number associated with each trip taken.\n", 108 | "- Departure_City: The city from which each flight departed.\n", 109 | "- Arrival_City: The destination city for each flight.\n", 110 | "- Flight_Date: The date of each flight taken." 111 | ] 112 | }, 113 | { 114 | "cell_type": "code", 115 | "execution_count": null, 116 | "id": "0718f278-3099-4b42-a81b-e6d632339cc0", 117 | "metadata": { 118 | "tags": [] 119 | }, 120 | "outputs": [], 121 | "source": [ 122 | "import pandas as pd\n", 123 | "\n", 124 | "df = pd.read_csv(\"./data/synthetic_travel_data.csv\")\n", 125 | "df.head(10)" 126 | ] 127 | }, 128 | { 129 | "cell_type": "markdown", 130 | "id": "d575e3f0-d4d7-4508-a286-0e6a3d1ef713", 131 | "metadata": {}, 132 | "source": [ 133 | "#### 2. US and Europe City Dataset\n", 134 | "\n", 135 | "For the purpose of the Retrieval Augmented generation(RAG) demonstrations in the labs, we have synthetically created PDF datasets to be ingested in the Vector DBs. Following is a sample PDF organises in `/data/us` and `/data/europe` folders respectively. We have used an Agentic implementation to create the dataset. You can check this out in the optional notebook provided" 136 | ] 137 | }, 138 | { 139 | "cell_type": "code", 140 | "execution_count": null, 141 | "id": "23ac8674-5c70-4d67-a8e8-39ff8cb5480e", 142 | "metadata": { 143 | "tags": [] 144 | }, 145 | "outputs": [], 146 | "source": [ 147 | "from IPython.display import Image\n", 148 | "Image(filename='./images/vegas.png') " 149 | ] 150 | }, 151 | { 152 | "cell_type": "markdown", 153 | "id": "a0a7f421-9a23-462e-987c-10103b5fdfa1", 154 | "metadata": {}, 155 | "source": [ 156 | "### Amazon Bedrock Model Access\n", 157 | "\n", 158 | "[Amazon Bedrock](https://aws.amazon.com/bedrock/) offers a variety of foundation models (FMs) that are optimized for different use cases, making it a versatile platform for building generative AI applications. For agentic workflows, where models interact with tools, make decisions, and carry out complex tasks autonomously, choosing the right FM is crucial. For the purposes of this demostration, we will use the foundational models available on Amazon Bedrock with a focus on Anthropic Claude, Meta’s models, and Amazon’s own models, highlighting their capabilities for demonstration purposes. You can take a look [Supported foundation models in Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/models-supported.html)" 159 | ] 160 | }, 161 | { 162 | "cell_type": "code", 163 | "execution_count": null, 164 | "id": "e175411a-c8dc-4610-9097-683ed2c03403", 165 | "metadata": { 166 | "tags": [] 167 | }, 168 | "outputs": [], 169 | "source": [ 170 | "import boto3\n", 171 | "\n", 172 | "region = 'us-west-2'\n", 173 | "bedrock = boto3.client(\n", 174 | " service_name = 'bedrock-runtime',\n", 175 | " region_name = region,\n", 176 | ")" 177 | ] 178 | }, 179 | { 180 | "cell_type": "markdown", 181 | "id": "31adfd14-5ee4-403f-89ed-92d886f9c8e3", 182 | "metadata": {}, 183 | "source": [ 184 | "#### Anthropic's Claude in Amazon Bedrock" 185 | ] 186 | }, 187 | { 188 | "cell_type": "markdown", 189 | "id": "284454be-727e-4b04-81bf-01cedc2d9e0d", 190 | "metadata": {}, 191 | "source": [ 192 | "The `langchain_aws` package is a tool for integrating and interacting with Amazon Bedrock's language models through the LangChain framework. It provides following benefits\n", 193 | "\n", 194 | "- Simplified Access to Amazon Bedrock Models\n", 195 | "- Support for Conversational AI and Agentic Workflows\n", 196 | "\n", 197 | "- `from langchain_aws.chat_models.bedrock:` This imports the ChatBedrock class from the langchain_aws package, specifically from the chat_models module.\n", 198 | "- `ChatBedrock:` This is a class that allows interaction with Amazon Bedrock’s chat models, enabling users to create conversational AI applications using different foundational models available on Bedrock.\n", 199 | "\n", 200 | "Refer [Anthropic's Claude in Amazon Bedrock](https://aws.amazon.com/bedrock/claude/) for more details" 201 | ] 202 | }, 203 | { 204 | "cell_type": "markdown", 205 | "id": "eecf1ada", 206 | "metadata": {}, 207 | "source": [ 208 | "### Converse API\n", 209 | "\n", 210 | "\"" 211 | ] 212 | }, 213 | { 214 | "cell_type": "markdown", 215 | "id": "309afbb3", 216 | "metadata": {}, 217 | "source": [ 218 | "### Please ignore any pip install errors" 219 | ] 220 | }, 221 | { 222 | "cell_type": "code", 223 | "execution_count": null, 224 | "id": "20ef8c2d-e7b8-4948-a76e-50c428e137ea", 225 | "metadata": {}, 226 | "outputs": [], 227 | "source": [ 228 | "%pip install -U --no-cache-dir -q \\\n", 229 | "\"langchain==0.3.7\" \\\n", 230 | "\"langchain-aws==0.2.6\" \\\n", 231 | "\"langchain-community==0.3.5\" \\\n", 232 | "\"langchain-text-splitters==0.3.2\" \\\n", 233 | "\"langchainhub==0.1.20\" \\\n", 234 | "\"langgraph==0.2.45\" \\\n", 235 | "\"langgraph-checkpoint==2.0.2\" \\\n", 236 | "\"langgraph-sdk==0.1.35\" \\\n", 237 | "\"langsmith==0.1.140\" \\\n", 238 | "\"pypdf==3.8,<4\" \\\n", 239 | "\"ipywidgets>=7,<8\" \\\n", 240 | "\"matplotlib==3.9.0\" \\\n", 241 | "\"faiss-cpu==1.8.0\"\n", 242 | "\n", 243 | "%pip install -U --no-cache-dir boto3\n", 244 | "#%pip install grandalf==3.1.2" 245 | ] 246 | }, 247 | { 248 | "cell_type": "code", 249 | "execution_count": null, 250 | "id": "a84fc9c3-591f-48fb-a153-6ae0dfd6a4dc", 251 | "metadata": { 252 | "tags": [] 253 | }, 254 | "outputs": [], 255 | "source": [ 256 | "from langchain_aws.chat_models.bedrock import ChatBedrock\n", 257 | "from rich.console import Console\n", 258 | "from rich.text import Text\n", 259 | "\n", 260 | "# Create a console object\n", 261 | "console = Console()\n", 262 | "\n", 263 | "modelId = 'anthropic.claude-3-haiku-20240307-v1:0'\n", 264 | "llm = ChatBedrock(\n", 265 | " model_id=modelId,\n", 266 | " client=bedrock,\n", 267 | " beta_use_converse_api=True\n", 268 | ")\n", 269 | "\n", 270 | "\n", 271 | "response = llm.invoke(\"I am planning a trip with my family to Hawai next summer, can you give me a travel iternary for 5 days\").content\n", 272 | "# Format the text as a rich Text object\n", 273 | "formatted_text = Text.from_markup(response)\n", 274 | "\n", 275 | "# Print the formatted response using rich\n", 276 | "console.print(formatted_text)" 277 | ] 278 | }, 279 | { 280 | "cell_type": "markdown", 281 | "id": "79fb9e25-e947-49cf-b4e7-6c22d4ce91c7", 282 | "metadata": {}, 283 | "source": [ 284 | "### Course-grained model comparison\n", 285 | "\n", 286 | "In this section, we experiment with multiple models available on Amazon Bedrock and run course-grained evaluation on one of our task of interest. With the thousands of available models on the market, it is intractable to evaluation every single one. Hence, it is generally necessary to pre-filter for the ones that are not only from trusted providers, but have shown strong performance on a variety of benchmarks. \n", 287 | "\n", 288 | "Amazon Bedrock allows you to make a quick short-list by supporting a growing list providers such as Anthropic, Meta, Mistral, Cohere, AI21Labs, Stability AI and Amazon. This lets you start with a strong base to continue the model selection process.\n", 289 | "\n", 290 | "Next we perform course-grained model evalution on the following models to inform our initial choice of model for our task of interest:\n", 291 | "- Anthropic: Claude 3 Haiku\n", 292 | "\n", 293 | "For this workshop we have access to 1 model only. Ideally in your own accounts you can use multiple models to run this evaluation.\n", 294 | "\n", 295 | "To perform an initial evaluation, we create a small curated dataset of 10 examples. The optimal initial number of examples should be sufficiently big to roughly cover the types of queries our customers will send our model. Since this stage of the model evaluation process is meant to get a rough idea, the number of examples can be small.\n", 296 | "\n", 297 | "To start, our scenario can be described by summarization (**task**) of vacation destinations (**what**) asked by travelers (**who**) at the time of development (**when**) in English (**language**). The set of initial questions can be found in [examples.txt](./data/examples.txt). We could expand our test by changing one or more of the variables composing the scenario of interesting. For instance, we could generate equivalent examples, but asked by people who aren't travelers or by others speaking in any other languages." 298 | ] 299 | }, 300 | { 301 | "cell_type": "code", 302 | "execution_count": null, 303 | "id": "f9bd4753-c613-415b-9e25-ce71aaeba980", 304 | "metadata": { 305 | "tags": [] 306 | }, 307 | "outputs": [], 308 | "source": [ 309 | "with open(\"./data/examples.txt\", \"r\") as file:\n", 310 | " examples = file.read().splitlines()" 311 | ] 312 | }, 313 | { 314 | "cell_type": "code", 315 | "execution_count": null, 316 | "id": "ea783a12-03c3-4e37-a34b-1aa38400c7b4", 317 | "metadata": { 318 | "tags": [] 319 | }, 320 | "outputs": [], 321 | "source": [ 322 | "import pandas as pd\n", 323 | "from langchain_core.messages import HumanMessage, SystemMessage\n", 324 | "from rich.console import Console\n", 325 | "from rich.text import Text\n", 326 | "\n", 327 | "pd.set_option('display.max_colwidth', None)\n", 328 | "# Create a console object\n", 329 | "console = Console()\n", 330 | "\n", 331 | "\n", 332 | "def generate_answers(\n", 333 | " examples: list = [],\n", 334 | " system_prompt: SystemMessage = None\n", 335 | "):\n", 336 | " modelIds = [\n", 337 | " \"anthropic.claude-3-haiku-20240307-v1:0\",\n", 338 | " ]\n", 339 | " output = pd.DataFrame({\n", 340 | " 'example': [],\n", 341 | " 'Claude3Haiku': [],\n", 342 | " })\n", 343 | " for example in examples:\n", 344 | " results = [example]\n", 345 | " for modelId in modelIds:\n", 346 | " messages = [\n", 347 | " system_prompt if system_prompt else SystemMessage(content=\"\"),\n", 348 | " HumanMessage(content=example)\n", 349 | " ]\n", 350 | " llm = ChatBedrock(\n", 351 | " model_id=modelId,\n", 352 | " client=bedrock,\n", 353 | " beta_use_converse_api=True\n", 354 | " )\n", 355 | " resp = llm.invoke(messages).content\n", 356 | " results.append(resp)\n", 357 | " output.loc[len(output)] = results\n", 358 | " return output" 359 | ] 360 | }, 361 | { 362 | "cell_type": "code", 363 | "execution_count": null, 364 | "id": "c646b7a6-2c56-4451-a2f5-c71d0dd7d339", 365 | "metadata": { 366 | "tags": [] 367 | }, 368 | "outputs": [], 369 | "source": [ 370 | "one_example = examples\n", 371 | "output = generate_answers(one_example)\n", 372 | "output.head()" 373 | ] 374 | }, 375 | { 376 | "cell_type": "markdown", 377 | "id": "c2fc4e7e-8b5e-40fd-90ce-44afb0cedff9", 378 | "metadata": {}, 379 | "source": [ 380 | "### Conclusion\n", 381 | "\n", 382 | "In this notebook, we established \n", 383 | "- the use case scope of the labs covered in this workshop\n", 384 | "- datasets we will be using all the labs\n", 385 | "- Amazon bedrock models used in this\n", 386 | "\n", 387 | "Please proceed to the next labs" 388 | ] 389 | } 390 | ], 391 | "metadata": { 392 | "availableInstances": [ 393 | { 394 | "_defaultOrder": 0, 395 | "_isFastLaunch": true, 396 | "category": "General purpose", 397 | "gpuNum": 0, 398 | "hideHardwareSpecs": false, 399 | "memoryGiB": 4, 400 | "name": "ml.t3.medium", 401 | "vcpuNum": 2 402 | }, 403 | { 404 | "_defaultOrder": 1, 405 | "_isFastLaunch": false, 406 | "category": "General purpose", 407 | "gpuNum": 0, 408 | "hideHardwareSpecs": false, 409 | "memoryGiB": 8, 410 | "name": "ml.t3.large", 411 | "vcpuNum": 2 412 | }, 413 | { 414 | "_defaultOrder": 2, 415 | "_isFastLaunch": false, 416 | "category": "General purpose", 417 | "gpuNum": 0, 418 | "hideHardwareSpecs": false, 419 | "memoryGiB": 16, 420 | "name": "ml.t3.xlarge", 421 | "vcpuNum": 4 422 | }, 423 | { 424 | "_defaultOrder": 3, 425 | "_isFastLaunch": false, 426 | "category": "General purpose", 427 | "gpuNum": 0, 428 | "hideHardwareSpecs": false, 429 | "memoryGiB": 32, 430 | "name": "ml.t3.2xlarge", 431 | "vcpuNum": 8 432 | }, 433 | { 434 | "_defaultOrder": 4, 435 | "_isFastLaunch": true, 436 | "category": "General purpose", 437 | "gpuNum": 0, 438 | "hideHardwareSpecs": false, 439 | "memoryGiB": 8, 440 | "name": "ml.m5.large", 441 | "vcpuNum": 2 442 | }, 443 | { 444 | "_defaultOrder": 5, 445 | "_isFastLaunch": false, 446 | "category": "General purpose", 447 | "gpuNum": 0, 448 | "hideHardwareSpecs": false, 449 | "memoryGiB": 16, 450 | "name": "ml.m5.xlarge", 451 | "vcpuNum": 4 452 | }, 453 | { 454 | "_defaultOrder": 6, 455 | "_isFastLaunch": false, 456 | "category": "General purpose", 457 | "gpuNum": 0, 458 | "hideHardwareSpecs": false, 459 | "memoryGiB": 32, 460 | "name": "ml.m5.2xlarge", 461 | "vcpuNum": 8 462 | }, 463 | { 464 | "_defaultOrder": 7, 465 | "_isFastLaunch": false, 466 | "category": "General purpose", 467 | "gpuNum": 0, 468 | "hideHardwareSpecs": false, 469 | "memoryGiB": 64, 470 | "name": "ml.m5.4xlarge", 471 | "vcpuNum": 16 472 | }, 473 | { 474 | "_defaultOrder": 8, 475 | "_isFastLaunch": false, 476 | "category": "General purpose", 477 | "gpuNum": 0, 478 | "hideHardwareSpecs": false, 479 | "memoryGiB": 128, 480 | "name": "ml.m5.8xlarge", 481 | "vcpuNum": 32 482 | }, 483 | { 484 | "_defaultOrder": 9, 485 | "_isFastLaunch": false, 486 | "category": "General purpose", 487 | "gpuNum": 0, 488 | "hideHardwareSpecs": false, 489 | "memoryGiB": 192, 490 | "name": "ml.m5.12xlarge", 491 | "vcpuNum": 48 492 | }, 493 | { 494 | "_defaultOrder": 10, 495 | "_isFastLaunch": false, 496 | "category": "General purpose", 497 | "gpuNum": 0, 498 | "hideHardwareSpecs": false, 499 | "memoryGiB": 256, 500 | "name": "ml.m5.16xlarge", 501 | "vcpuNum": 64 502 | }, 503 | { 504 | "_defaultOrder": 11, 505 | "_isFastLaunch": false, 506 | "category": "General purpose", 507 | "gpuNum": 0, 508 | "hideHardwareSpecs": false, 509 | "memoryGiB": 384, 510 | "name": "ml.m5.24xlarge", 511 | "vcpuNum": 96 512 | }, 513 | { 514 | "_defaultOrder": 12, 515 | "_isFastLaunch": false, 516 | "category": "General purpose", 517 | "gpuNum": 0, 518 | "hideHardwareSpecs": false, 519 | "memoryGiB": 8, 520 | "name": "ml.m5d.large", 521 | "vcpuNum": 2 522 | }, 523 | { 524 | "_defaultOrder": 13, 525 | "_isFastLaunch": false, 526 | "category": "General purpose", 527 | "gpuNum": 0, 528 | "hideHardwareSpecs": false, 529 | "memoryGiB": 16, 530 | "name": "ml.m5d.xlarge", 531 | "vcpuNum": 4 532 | }, 533 | { 534 | "_defaultOrder": 14, 535 | "_isFastLaunch": false, 536 | "category": "General purpose", 537 | "gpuNum": 0, 538 | "hideHardwareSpecs": false, 539 | "memoryGiB": 32, 540 | "name": "ml.m5d.2xlarge", 541 | "vcpuNum": 8 542 | }, 543 | { 544 | "_defaultOrder": 15, 545 | "_isFastLaunch": false, 546 | "category": "General purpose", 547 | "gpuNum": 0, 548 | "hideHardwareSpecs": false, 549 | "memoryGiB": 64, 550 | "name": "ml.m5d.4xlarge", 551 | "vcpuNum": 16 552 | }, 553 | { 554 | "_defaultOrder": 16, 555 | "_isFastLaunch": false, 556 | "category": "General purpose", 557 | "gpuNum": 0, 558 | "hideHardwareSpecs": false, 559 | "memoryGiB": 128, 560 | "name": "ml.m5d.8xlarge", 561 | "vcpuNum": 32 562 | }, 563 | { 564 | "_defaultOrder": 17, 565 | "_isFastLaunch": false, 566 | "category": "General purpose", 567 | "gpuNum": 0, 568 | "hideHardwareSpecs": false, 569 | "memoryGiB": 192, 570 | "name": "ml.m5d.12xlarge", 571 | "vcpuNum": 48 572 | }, 573 | { 574 | "_defaultOrder": 18, 575 | "_isFastLaunch": false, 576 | "category": "General purpose", 577 | "gpuNum": 0, 578 | "hideHardwareSpecs": false, 579 | "memoryGiB": 256, 580 | "name": "ml.m5d.16xlarge", 581 | "vcpuNum": 64 582 | }, 583 | { 584 | "_defaultOrder": 19, 585 | "_isFastLaunch": false, 586 | "category": "General purpose", 587 | "gpuNum": 0, 588 | "hideHardwareSpecs": false, 589 | "memoryGiB": 384, 590 | "name": "ml.m5d.24xlarge", 591 | "vcpuNum": 96 592 | }, 593 | { 594 | "_defaultOrder": 20, 595 | "_isFastLaunch": false, 596 | "category": "General purpose", 597 | "gpuNum": 0, 598 | "hideHardwareSpecs": true, 599 | "memoryGiB": 0, 600 | "name": "ml.geospatial.interactive", 601 | "supportedImageNames": [ 602 | "sagemaker-geospatial-v1-0" 603 | ], 604 | "vcpuNum": 0 605 | }, 606 | { 607 | "_defaultOrder": 21, 608 | "_isFastLaunch": true, 609 | "category": "Compute optimized", 610 | "gpuNum": 0, 611 | "hideHardwareSpecs": false, 612 | "memoryGiB": 4, 613 | "name": "ml.c5.large", 614 | "vcpuNum": 2 615 | }, 616 | { 617 | "_defaultOrder": 22, 618 | "_isFastLaunch": false, 619 | "category": "Compute optimized", 620 | "gpuNum": 0, 621 | "hideHardwareSpecs": false, 622 | "memoryGiB": 8, 623 | "name": "ml.c5.xlarge", 624 | "vcpuNum": 4 625 | }, 626 | { 627 | "_defaultOrder": 23, 628 | "_isFastLaunch": false, 629 | "category": "Compute optimized", 630 | "gpuNum": 0, 631 | "hideHardwareSpecs": false, 632 | "memoryGiB": 16, 633 | "name": "ml.c5.2xlarge", 634 | "vcpuNum": 8 635 | }, 636 | { 637 | "_defaultOrder": 24, 638 | "_isFastLaunch": false, 639 | "category": "Compute optimized", 640 | "gpuNum": 0, 641 | "hideHardwareSpecs": false, 642 | "memoryGiB": 32, 643 | "name": "ml.c5.4xlarge", 644 | "vcpuNum": 16 645 | }, 646 | { 647 | "_defaultOrder": 25, 648 | "_isFastLaunch": false, 649 | "category": "Compute optimized", 650 | "gpuNum": 0, 651 | "hideHardwareSpecs": false, 652 | "memoryGiB": 72, 653 | "name": "ml.c5.9xlarge", 654 | "vcpuNum": 36 655 | }, 656 | { 657 | "_defaultOrder": 26, 658 | "_isFastLaunch": false, 659 | "category": "Compute optimized", 660 | "gpuNum": 0, 661 | "hideHardwareSpecs": false, 662 | "memoryGiB": 96, 663 | "name": "ml.c5.12xlarge", 664 | "vcpuNum": 48 665 | }, 666 | { 667 | "_defaultOrder": 27, 668 | "_isFastLaunch": false, 669 | "category": "Compute optimized", 670 | "gpuNum": 0, 671 | "hideHardwareSpecs": false, 672 | "memoryGiB": 144, 673 | "name": "ml.c5.18xlarge", 674 | "vcpuNum": 72 675 | }, 676 | { 677 | "_defaultOrder": 28, 678 | "_isFastLaunch": false, 679 | "category": "Compute optimized", 680 | "gpuNum": 0, 681 | "hideHardwareSpecs": false, 682 | "memoryGiB": 192, 683 | "name": "ml.c5.24xlarge", 684 | "vcpuNum": 96 685 | }, 686 | { 687 | "_defaultOrder": 29, 688 | "_isFastLaunch": true, 689 | "category": "Accelerated computing", 690 | "gpuNum": 1, 691 | "hideHardwareSpecs": false, 692 | "memoryGiB": 16, 693 | "name": "ml.g4dn.xlarge", 694 | "vcpuNum": 4 695 | }, 696 | { 697 | "_defaultOrder": 30, 698 | "_isFastLaunch": false, 699 | "category": "Accelerated computing", 700 | "gpuNum": 1, 701 | "hideHardwareSpecs": false, 702 | "memoryGiB": 32, 703 | "name": "ml.g4dn.2xlarge", 704 | "vcpuNum": 8 705 | }, 706 | { 707 | "_defaultOrder": 31, 708 | "_isFastLaunch": false, 709 | "category": "Accelerated computing", 710 | "gpuNum": 1, 711 | "hideHardwareSpecs": false, 712 | "memoryGiB": 64, 713 | "name": "ml.g4dn.4xlarge", 714 | "vcpuNum": 16 715 | }, 716 | { 717 | "_defaultOrder": 32, 718 | "_isFastLaunch": false, 719 | "category": "Accelerated computing", 720 | "gpuNum": 1, 721 | "hideHardwareSpecs": false, 722 | "memoryGiB": 128, 723 | "name": "ml.g4dn.8xlarge", 724 | "vcpuNum": 32 725 | }, 726 | { 727 | "_defaultOrder": 33, 728 | "_isFastLaunch": false, 729 | "category": "Accelerated computing", 730 | "gpuNum": 4, 731 | "hideHardwareSpecs": false, 732 | "memoryGiB": 192, 733 | "name": "ml.g4dn.12xlarge", 734 | "vcpuNum": 48 735 | }, 736 | { 737 | "_defaultOrder": 34, 738 | "_isFastLaunch": false, 739 | "category": "Accelerated computing", 740 | "gpuNum": 1, 741 | "hideHardwareSpecs": false, 742 | "memoryGiB": 256, 743 | "name": "ml.g4dn.16xlarge", 744 | "vcpuNum": 64 745 | }, 746 | { 747 | "_defaultOrder": 35, 748 | "_isFastLaunch": false, 749 | "category": "Accelerated computing", 750 | "gpuNum": 1, 751 | "hideHardwareSpecs": false, 752 | "memoryGiB": 61, 753 | "name": "ml.p3.2xlarge", 754 | "vcpuNum": 8 755 | }, 756 | { 757 | "_defaultOrder": 36, 758 | "_isFastLaunch": false, 759 | "category": "Accelerated computing", 760 | "gpuNum": 4, 761 | "hideHardwareSpecs": false, 762 | "memoryGiB": 244, 763 | "name": "ml.p3.8xlarge", 764 | "vcpuNum": 32 765 | }, 766 | { 767 | "_defaultOrder": 37, 768 | "_isFastLaunch": false, 769 | "category": "Accelerated computing", 770 | "gpuNum": 8, 771 | "hideHardwareSpecs": false, 772 | "memoryGiB": 488, 773 | "name": "ml.p3.16xlarge", 774 | "vcpuNum": 64 775 | }, 776 | { 777 | "_defaultOrder": 38, 778 | "_isFastLaunch": false, 779 | "category": "Accelerated computing", 780 | "gpuNum": 8, 781 | "hideHardwareSpecs": false, 782 | "memoryGiB": 768, 783 | "name": "ml.p3dn.24xlarge", 784 | "vcpuNum": 96 785 | }, 786 | { 787 | "_defaultOrder": 39, 788 | "_isFastLaunch": false, 789 | "category": "Memory Optimized", 790 | "gpuNum": 0, 791 | "hideHardwareSpecs": false, 792 | "memoryGiB": 16, 793 | "name": "ml.r5.large", 794 | "vcpuNum": 2 795 | }, 796 | { 797 | "_defaultOrder": 40, 798 | "_isFastLaunch": false, 799 | "category": "Memory Optimized", 800 | "gpuNum": 0, 801 | "hideHardwareSpecs": false, 802 | "memoryGiB": 32, 803 | "name": "ml.r5.xlarge", 804 | "vcpuNum": 4 805 | }, 806 | { 807 | "_defaultOrder": 41, 808 | "_isFastLaunch": false, 809 | "category": "Memory Optimized", 810 | "gpuNum": 0, 811 | "hideHardwareSpecs": false, 812 | "memoryGiB": 64, 813 | "name": "ml.r5.2xlarge", 814 | "vcpuNum": 8 815 | }, 816 | { 817 | "_defaultOrder": 42, 818 | "_isFastLaunch": false, 819 | "category": "Memory Optimized", 820 | "gpuNum": 0, 821 | "hideHardwareSpecs": false, 822 | "memoryGiB": 128, 823 | "name": "ml.r5.4xlarge", 824 | "vcpuNum": 16 825 | }, 826 | { 827 | "_defaultOrder": 43, 828 | "_isFastLaunch": false, 829 | "category": "Memory Optimized", 830 | "gpuNum": 0, 831 | "hideHardwareSpecs": false, 832 | "memoryGiB": 256, 833 | "name": "ml.r5.8xlarge", 834 | "vcpuNum": 32 835 | }, 836 | { 837 | "_defaultOrder": 44, 838 | "_isFastLaunch": false, 839 | "category": "Memory Optimized", 840 | "gpuNum": 0, 841 | "hideHardwareSpecs": false, 842 | "memoryGiB": 384, 843 | "name": "ml.r5.12xlarge", 844 | "vcpuNum": 48 845 | }, 846 | { 847 | "_defaultOrder": 45, 848 | "_isFastLaunch": false, 849 | "category": "Memory Optimized", 850 | "gpuNum": 0, 851 | "hideHardwareSpecs": false, 852 | "memoryGiB": 512, 853 | "name": "ml.r5.16xlarge", 854 | "vcpuNum": 64 855 | }, 856 | { 857 | "_defaultOrder": 46, 858 | "_isFastLaunch": false, 859 | "category": "Memory Optimized", 860 | "gpuNum": 0, 861 | "hideHardwareSpecs": false, 862 | "memoryGiB": 768, 863 | "name": "ml.r5.24xlarge", 864 | "vcpuNum": 96 865 | }, 866 | { 867 | "_defaultOrder": 47, 868 | "_isFastLaunch": false, 869 | "category": "Accelerated computing", 870 | "gpuNum": 1, 871 | "hideHardwareSpecs": false, 872 | "memoryGiB": 16, 873 | "name": "ml.g5.xlarge", 874 | "vcpuNum": 4 875 | }, 876 | { 877 | "_defaultOrder": 48, 878 | "_isFastLaunch": false, 879 | "category": "Accelerated computing", 880 | "gpuNum": 1, 881 | "hideHardwareSpecs": false, 882 | "memoryGiB": 32, 883 | "name": "ml.g5.2xlarge", 884 | "vcpuNum": 8 885 | }, 886 | { 887 | "_defaultOrder": 49, 888 | "_isFastLaunch": false, 889 | "category": "Accelerated computing", 890 | "gpuNum": 1, 891 | "hideHardwareSpecs": false, 892 | "memoryGiB": 64, 893 | "name": "ml.g5.4xlarge", 894 | "vcpuNum": 16 895 | }, 896 | { 897 | "_defaultOrder": 50, 898 | "_isFastLaunch": false, 899 | "category": "Accelerated computing", 900 | "gpuNum": 1, 901 | "hideHardwareSpecs": false, 902 | "memoryGiB": 128, 903 | "name": "ml.g5.8xlarge", 904 | "vcpuNum": 32 905 | }, 906 | { 907 | "_defaultOrder": 51, 908 | "_isFastLaunch": false, 909 | "category": "Accelerated computing", 910 | "gpuNum": 1, 911 | "hideHardwareSpecs": false, 912 | "memoryGiB": 256, 913 | "name": "ml.g5.16xlarge", 914 | "vcpuNum": 64 915 | }, 916 | { 917 | "_defaultOrder": 52, 918 | "_isFastLaunch": false, 919 | "category": "Accelerated computing", 920 | "gpuNum": 4, 921 | "hideHardwareSpecs": false, 922 | "memoryGiB": 192, 923 | "name": "ml.g5.12xlarge", 924 | "vcpuNum": 48 925 | }, 926 | { 927 | "_defaultOrder": 53, 928 | "_isFastLaunch": false, 929 | "category": "Accelerated computing", 930 | "gpuNum": 4, 931 | "hideHardwareSpecs": false, 932 | "memoryGiB": 384, 933 | "name": "ml.g5.24xlarge", 934 | "vcpuNum": 96 935 | }, 936 | { 937 | "_defaultOrder": 54, 938 | "_isFastLaunch": false, 939 | "category": "Accelerated computing", 940 | "gpuNum": 8, 941 | "hideHardwareSpecs": false, 942 | "memoryGiB": 768, 943 | "name": "ml.g5.48xlarge", 944 | "vcpuNum": 192 945 | }, 946 | { 947 | "_defaultOrder": 55, 948 | "_isFastLaunch": false, 949 | "category": "Accelerated computing", 950 | "gpuNum": 8, 951 | "hideHardwareSpecs": false, 952 | "memoryGiB": 1152, 953 | "name": "ml.p4d.24xlarge", 954 | "vcpuNum": 96 955 | }, 956 | { 957 | "_defaultOrder": 56, 958 | "_isFastLaunch": false, 959 | "category": "Accelerated computing", 960 | "gpuNum": 8, 961 | "hideHardwareSpecs": false, 962 | "memoryGiB": 1152, 963 | "name": "ml.p4de.24xlarge", 964 | "vcpuNum": 96 965 | }, 966 | { 967 | "_defaultOrder": 57, 968 | "_isFastLaunch": false, 969 | "category": "Accelerated computing", 970 | "gpuNum": 0, 971 | "hideHardwareSpecs": false, 972 | "memoryGiB": 32, 973 | "name": "ml.trn1.2xlarge", 974 | "vcpuNum": 8 975 | }, 976 | { 977 | "_defaultOrder": 58, 978 | "_isFastLaunch": false, 979 | "category": "Accelerated computing", 980 | "gpuNum": 0, 981 | "hideHardwareSpecs": false, 982 | "memoryGiB": 512, 983 | "name": "ml.trn1.32xlarge", 984 | "vcpuNum": 128 985 | }, 986 | { 987 | "_defaultOrder": 59, 988 | "_isFastLaunch": false, 989 | "category": "Accelerated computing", 990 | "gpuNum": 0, 991 | "hideHardwareSpecs": false, 992 | "memoryGiB": 512, 993 | "name": "ml.trn1n.32xlarge", 994 | "vcpuNum": 128 995 | } 996 | ], 997 | "instance_type": "ml.t3.medium", 998 | "kernelspec": { 999 | "display_name": "trainenv", 1000 | "language": "python", 1001 | "name": "python3" 1002 | }, 1003 | "language_info": { 1004 | "codemirror_mode": { 1005 | "name": "ipython", 1006 | "version": 3 1007 | }, 1008 | "file_extension": ".py", 1009 | "mimetype": "text/x-python", 1010 | "name": "python", 1011 | "nbconvert_exporter": "python", 1012 | "pygments_lexer": "ipython3", 1013 | "version": "3.11.9" 1014 | } 1015 | }, 1016 | "nbformat": 4, 1017 | "nbformat_minor": 5 1018 | } 1019 | -------------------------------------------------------------------------------- /data/synthetic_travel_data.csv: -------------------------------------------------------------------------------- 1 | Id,Name,Current_Location,Age,Past_Travel_Destinations,Number_of_Trips,Past_Travel_Destinations,Number_of_Trips,Flight_Number,Departure_City,Arrival_City,Flight_Date 2 | 24,Philip Cook,Zurich,29,Ljubljana,1,Ljubljana,1,Z1823,Zurich,Ljubljana,2024-10-11 3 | 918,Andrew Macdonald,Paris,62,"Lisbon, Lisbon, Bergen, Paris, Warsaw",5,"Lisbon, Lisbon, Bergen, Paris, Warsaw",5,O1142,Paris,Lisbon,2024-10-21 4 | ,,,,,,,,U7234,Paris,Lisbon,2024-03-23 5 | ,,,,,,,,H9041,Paris,Bergen,2024-03-25 6 | ,,,,,,,,Y7505,Paris,Paris,2023-09-13 7 | ,,,,,,,,X8568,Paris,Warsaw,2023-05-17 8 | 709,Jonathan Garcia,Sacramento,43,New York,1,New York,1,P8336,Sacramento,New York,2023-07-10 9 | 314,Wendy Yu,Moscow,34,"Prague, Valencia, Reykjavik, London",4,"Prague, Valencia, Reykjavik, London",4,P6922,Moscow,Prague,2023-01-02 10 | ,,,,,,,,I1305,Moscow,Valencia,2024-05-26 11 | ,,,,,,,,S2309,Moscow,Reykjavik,2023-04-14 12 | ,,,,,,,,I3285,Moscow,London,2022-11-13 13 | 429,James Camacho,Honolulu,58,Savannah,1,Savannah,1,P1380,Honolulu,Savannah,2023-08-18 14 | 405,Michael Buckley,Boston,34,"San Diego, Seattle",2,"San Diego, Seattle",2,L4933,Boston,San Diego,2024-01-04 15 | ,,,,,,,,T9701,Boston,Seattle,2023-09-08 16 | 57,Richard Adams,Oslo,30,"Bergen, Brussels, Venice, Valencia",4,"Bergen, Brussels, Venice, Valencia",4,B4940,Oslo,Bergen,2024-01-24 17 | ,,,,,,,,X6302,Oslo,Brussels,2023-01-27 18 | ,,,,,,,,Z5293,Oslo,Venice,2024-06-14 19 | ,,,,,,,,X3688,Oslo,Valencia,2022-12-19 20 | 36,Tonya Frank,St. Petersburg,35,"Amsterdam, Malaga, Prague, Munich, Salzburg",5,"Amsterdam, Malaga, Prague, Munich, Salzburg",5,I2224,St. Petersburg,Amsterdam,2022-12-06 21 | ,,,,,,,,N1951,St. Petersburg,Malaga,2023-08-09 22 | ,,,,,,,,Y1095,St. Petersburg,Prague,2023-02-23 23 | ,,,,,,,,B6582,St. Petersburg,Munich,2023-10-31 24 | ,,,,,,,,F9633,St. Petersburg,Salzburg,2022-11-30 25 | 731,Michelle Mckenzie DDS,Bologna,37,"Luxembourg, Luxembourg",2,"Luxembourg, Luxembourg",2,T4684,Bologna,Luxembourg,2023-10-19 26 | ,,,,,,,,F3794,Bologna,Luxembourg,2022-12-09 27 | 93,Nicholas Burke,Tallinn,28,"Berlin, Lyon",2,"Berlin, Lyon",2,X2409,Tallinn,Berlin,2023-03-24 28 | ,,,,,,,,D3875,Tallinn,Lyon,2023-04-13 29 | 582,Holly Reyes,Ljubljana,39,"Pittsburgh, Zurich",2,"Pittsburgh, Zurich",2,B1233,Ljubljana,Pittsburgh,2024-10-06 30 | ,,,,,,,,E1228,Ljubljana,Zurich,2024-08-06 31 | 35,Yvonne Delgado,Dubrovnik,71,"Madrid, London, Barcelona, St. Petersburg, Krakow",5,"Madrid, London, Barcelona, St. Petersburg, Krakow",5,F7777,Dubrovnik,Madrid,2024-10-14 32 | ,,,,,,,,I2326,Dubrovnik,London,2023-09-20 33 | ,,,,,,,,S4419,Dubrovnik,Barcelona,2024-04-25 34 | ,,,,,,,,O2326,Dubrovnik,St. Petersburg,2024-06-22 35 | ,,,,,,,,Q2646,Dubrovnik,Krakow,2022-12-07 36 | 730,Jacob Nelson,Phoenix,19,Providence,1,Providence,1,F3455,Phoenix,Providence,2023-05-20 37 | 101,Michele Russo,Jacksonville,22,"Savannah, Minneapolis",2,"Savannah, Minneapolis",2,Z5558,Jacksonville,Savannah,2024-10-10 38 | ,,,,,,,,M2527,Jacksonville,Minneapolis,2024-03-03 39 | 572,Emily Hurst,Salzburg,70,"Amsterdam, Lisbon",2,"Amsterdam, Lisbon",2,C9159,Salzburg,Amsterdam,2023-10-10 40 | ,,,,,,,,Y4249,Salzburg,Lisbon,2024-08-06 41 | 118,Shawn Rios,Baltimore,48,"Sacramento, Houston, San Francisco, Savannah, Oklahoma City",5,"Sacramento, Houston, San Francisco, Savannah, Oklahoma City",5,B1677,Baltimore,Sacramento,2022-12-15 42 | ,,,,,,,,V2855,Baltimore,Houston,2024-07-11 43 | ,,,,,,,,N2375,Baltimore,San Francisco,2023-10-22 44 | ,,,,,,,,L8550,Baltimore,Savannah,2024-03-09 45 | ,,,,,,,,A2988,Baltimore,Oklahoma City,2024-10-23 46 | 809,James Aguilar,Baltimore,63,"New Orleans, Oklahoma City, Portland",3,"New Orleans, Oklahoma City, Portland",3,W3579,Baltimore,New Orleans,2024-03-12 47 | ,,,,,,,,R5648,Baltimore,Oklahoma City,2023-06-18 48 | ,,,,,,,,U5432,Baltimore,Portland,2024-09-14 49 | 578,Jack Allen,Nice,71,"Malaga, Barcelona, Seville",3,"Malaga, Barcelona, Seville",3,D3865,Nice,Malaga,2024-03-03 50 | ,,,,,,,,I9015,Nice,Barcelona,2023-05-30 51 | ,,,,,,,,W7239,Nice,Seville,2023-07-09 52 | 238,Jenna Rodriguez,Orlando,21,"Key West, Indianapolis, Houston, Sacramento, St. Louis",5,"Key West, Indianapolis, Houston, Sacramento, St. Louis",5,H9609,Orlando,Key West,2023-02-25 53 | ,,,,,,,,A9464,Orlando,Indianapolis,2023-10-29 54 | ,,,,,,,,N5925,Orlando,Houston,2023-04-29 55 | ,,,,,,,,M6904,Orlando,Sacramento,2024-01-16 56 | ,,,,,,,,B3200,Orlando,St. Louis,2024-09-15 57 | 317,Christopher Hall,Washington,61,Nashville,1,Nashville,1,I4977,Washington,Nashville,2023-02-26 58 | 394,Misty Walter,Chicago,23,"Savannah, Amsterdam",2,"Savannah, Amsterdam",2,L7184,Chicago,Savannah,2024-06-26 59 | ,,,,,,,,Y5435,Chicago,Amsterdam,2023-03-07 60 | 332,John Perry,Edinburgh,37,"Ljubljana, Gothenburg",2,"Ljubljana, Gothenburg",2,I3258,Edinburgh,Ljubljana,2024-04-03 61 | ,,,,,,,,Y6564,Edinburgh,Gothenburg,2023-11-26 62 | 218,Stephanie Hale,Istanbul,52,"Marseille, Krakow, Amsterdam",3,"Marseille, Krakow, Amsterdam",3,Y4819,Istanbul,Marseille,2023-09-21 63 | ,,,,,,,,T7353,Istanbul,Krakow,2023-02-01 64 | ,,,,,,,,E5659,Istanbul,Amsterdam,2024-10-23 65 | 198,Michael Ballard,Houston,57,"New Orleans, Nashville, Asheville, Phoenix",4,"New Orleans, Nashville, Asheville, Phoenix",4,S9604,Houston,New Orleans,2023-06-23 66 | ,,,,,,,,K2641,Houston,Nashville,2023-02-09 67 | ,,,,,,,,B4810,Houston,Asheville,2022-11-25 68 | ,,,,,,,,A9642,Houston,Phoenix,2023-08-01 69 | 157,William Navarro,Nashville,68,"San Antonio, Fort Lauderdale, Sedona",3,"San Antonio, Fort Lauderdale, Sedona",3,U3608,Nashville,San Antonio,2023-05-19 70 | ,,,,,,,,Q3497,Nashville,Fort Lauderdale,2024-09-30 71 | ,,,,,,,,X9986,Nashville,Sedona,2023-07-27 72 | 458,Louis Hall,Pittsburgh,29,"Fort Lauderdale, Tampa, Nashville, Houston, Dallas",5,"Fort Lauderdale, Tampa, Nashville, Houston, Dallas",5,L9257,Pittsburgh,Fort Lauderdale,2023-10-17 73 | ,,,,,,,,C5878,Pittsburgh,Tampa,2023-05-09 74 | ,,,,,,,,D7320,Pittsburgh,Nashville,2023-05-08 75 | ,,,,,,,,X6504,Pittsburgh,Houston,2023-08-24 76 | ,,,,,,,,E6637,Pittsburgh,Dallas,2023-04-09 77 | 896,Marissa Hendrix,Savannah,36,Savannah,1,Savannah,1,H5889,Savannah,Savannah,2024-01-24 78 | 132,Kelly Smith,Pittsburgh,41,"Savannah, Kansas City, Washington, Lisbon, Minneapolis",5,"Savannah, Kansas City, Washington, Lisbon, Minneapolis",5,C8488,Pittsburgh,Savannah,2023-12-07 79 | ,,,,,,,,K7269,Pittsburgh,Kansas City,2024-04-09 80 | ,,,,,,,,X3128,Pittsburgh,Washington,2022-11-26 81 | ,,,,,,,,K5640,Pittsburgh,Lisbon,2024-05-17 82 | ,,,,,,,,U7323,Pittsburgh,Minneapolis,2024-05-30 83 | 373,Sandra Stephens,Jacksonville,68,"Sedona, Sedona, New Orleans",3,"Sedona, Sedona, New Orleans",3,C4264,Jacksonville,Sedona,2023-12-29 84 | ,,,,,,,,U7898,Jacksonville,Sedona,2023-12-15 85 | ,,,,,,,,S8099,Jacksonville,New Orleans,2023-02-23 86 | 760,Wanda Alvarez,Kansas City,71,"Luxembourg, Cincinnati, Sedona, Seattle",4,"Luxembourg, Cincinnati, Sedona, Seattle",4,K6964,Kansas City,Luxembourg,2023-11-09 87 | ,,,,,,,,A2314,Kansas City,Cincinnati,2024-09-20 88 | ,,,,,,,,K1434,Kansas City,Sedona,2024-05-29 89 | ,,,,,,,,D9263,Kansas City,Seattle,2023-08-19 90 | 493,Brian Palmer,Phoenix,75,"Berlin, Las Vegas, Atlanta, Charlotte",4,"Berlin, Las Vegas, Atlanta, Charlotte",4,D4365,Phoenix,Berlin,2023-01-19 91 | ,,,,,,,,G5575,Phoenix,Las Vegas,2023-08-29 92 | ,,,,,,,,F4404,Phoenix,Atlanta,2023-09-06 93 | ,,,,,,,,Y9073,Phoenix,Charlotte,2023-08-19 94 | 460,Rhonda Nelson,Madrid,58,"Salzburg, Edinburgh, Dubrovnik, Venice",4,"Salzburg, Edinburgh, Dubrovnik, Venice",4,A7471,Madrid,Salzburg,2023-11-06 95 | ,,,,,,,,H1117,Madrid,Edinburgh,2023-06-06 96 | ,,,,,,,,F1744,Madrid,Dubrovnik,2024-04-15 97 | ,,,,,,,,G8955,Madrid,Venice,2024-05-18 98 | 853,Mr. Chase Stewart,Charleston,33,"Key West, Phoenix",2,"Key West, Phoenix",2,Q7576,Charleston,Key West,2023-09-04 99 | ,,,,,,,,W1396,Charleston,Phoenix,2024-07-24 100 | 667,Charles Fields,Porto,69,"Nice, Gothenburg, Rome, Malaga",4,"Nice, Gothenburg, Rome, Malaga",4,B9881,Porto,Nice,2024-08-29 101 | ,,,,,,,,D3956,Porto,Gothenburg,2023-08-01 102 | ,,,,,,,,Z4517,Porto,Rome,2024-05-18 103 | ,,,,,,,,G5121,Porto,Malaga,2023-12-23 104 | 325,Richard Wiggins,Reykjavik,59,"Salzburg, Lyon, London, Dublin, Moscow",5,"Salzburg, Lyon, London, Dublin, Moscow",5,W4633,Reykjavik,Salzburg,2022-11-20 105 | ,,,,,,,,A3522,Reykjavik,Lyon,2023-04-05 106 | ,,,,,,,,L9388,Reykjavik,London,2023-07-04 107 | ,,,,,,,,H6040,Reykjavik,Dublin,2024-10-16 108 | ,,,,,,,,R8682,Reykjavik,Moscow,2023-11-19 109 | 337,Jason Weber,Kansas City,25,"New Orleans, Los Angeles",2,"New Orleans, Los Angeles",2,A5742,Kansas City,New Orleans,2023-06-06 110 | ,,,,,,,,D5828,Kansas City,Los Angeles,2024-03-16 111 | 302,Carla Patel,Bergen,47,"Reykjavik, Lisbon, Dublin, Milan",4,"Reykjavik, Lisbon, Dublin, Milan",4,N6707,Bergen,Reykjavik,2022-11-08 112 | ,,,,,,,,G4357,Bergen,Lisbon,2024-04-08 113 | ,,,,,,,,R4032,Bergen,Dublin,2024-09-05 114 | ,,,,,,,,R5260,Bergen,Milan,2023-05-30 115 | 868,John Williams,Las Vegas,56,"Orlando, Asheville, Baltimore",3,"Orlando, Asheville, Baltimore",3,R4479,Las Vegas,Orlando,2024-07-26 116 | ,,,,,,,,Z5385,Las Vegas,Asheville,2024-07-01 117 | ,,,,,,,,O3708,Las Vegas,Baltimore,2024-10-30 118 | 563,Jacqueline Ross,Venice,51,"Ljubljana, Salzburg",2,"Ljubljana, Salzburg",2,P3652,Venice,Ljubljana,2022-11-15 119 | ,,,,,,,,R5723,Venice,Salzburg,2024-06-19 120 | 64,Alicia Mcintosh,Bergen,32,"Dubrovnik, Ljubljana, Vienna, Paris",4,"Dubrovnik, Ljubljana, Vienna, Paris",4,Q6562,Bergen,Dubrovnik,2023-11-05 121 | ,,,,,,,,W9443,Bergen,Ljubljana,2023-07-11 122 | ,,,,,,,,Z5803,Bergen,Vienna,2023-04-13 123 | ,,,,,,,,L6176,Bergen,Paris,2023-03-11 124 | 899,Calvin Patel,Fort Lauderdale,26,"Houston, Jacksonville, Denver, Milwaukee, Warsaw",5,"Houston, Jacksonville, Denver, Milwaukee, Warsaw",5,V7998,Fort Lauderdale,Houston,2024-04-30 125 | ,,,,,,,,G2029,Fort Lauderdale,Jacksonville,2023-08-12 126 | ,,,,,,,,S5425,Fort Lauderdale,Denver,2023-09-29 127 | ,,,,,,,,O8046,Fort Lauderdale,Milwaukee,2022-12-27 128 | ,,,,,,,,E5670,Fort Lauderdale,Warsaw,2023-04-12 129 | 626,Jeffery Schneider,Charlotte,64,"Sedona, Oklahoma City, Chicago, Honolulu",4,"Sedona, Oklahoma City, Chicago, Honolulu",4,Q3077,Charlotte,Sedona,2023-03-17 130 | ,,,,,,,,I8263,Charlotte,Oklahoma City,2024-08-16 131 | ,,,,,,,,K7117,Charlotte,Chicago,2023-05-08 132 | ,,,,,,,,F9315,Charlotte,Honolulu,2023-01-26 133 | 878,Alyssa Johnson,Bordeaux,42,"Florence, Krakow",2,"Florence, Krakow",2,R4423,Bordeaux,Florence,2023-01-17 134 | ,,,,,,,,H3098,Bordeaux,Krakow,2022-12-09 135 | 275,Ryan Harper Jr.,San Francisco,64,Austin,1,Austin,1,S2933,San Francisco,Austin,2024-04-04 136 | 919,Janice Moore,San Jose,25,Asheville,1,Asheville,1,C2303,San Jose,Asheville,2023-06-30 137 | 891,Lucas Williams,Minneapolis,73,"Pittsburgh, Nashville",2,"Pittsburgh, Nashville",2,Q7295,Minneapolis,Pittsburgh,2024-05-25 138 | ,,,,,,,,Y1620,Minneapolis,Nashville,2023-09-02 139 | 239,Angela Patton,Austin,69,Orlando,1,Orlando,1,M2608,Austin,Orlando,2023-02-07 140 | 456,Edward Scott,Baltimore,44,St. Louis,1,St. Louis,1,R9643,Baltimore,St. Louis,2024-10-02 141 | 809,Zachary Jimenez,Washington,68,"Key West, Boston",2,"Key West, Boston",2,W3516,Washington,Key West,2024-08-10 142 | ,,,,,,,,A9833,Washington,Boston,2023-04-02 143 | 103,Wendy Lowe,Istanbul,27,"Lisbon, Ljubljana, Cologne",3,"Lisbon, Ljubljana, Cologne",3,L3949,Istanbul,Lisbon,2024-02-08 144 | ,,,,,,,,A1309,Istanbul,Ljubljana,2024-03-05 145 | ,,,,,,,,S2946,Istanbul,Cologne,2023-07-30 146 | 120,William Terry,Pittsburgh,49,"New York, New Orleans, Newport",3,"New York, New Orleans, Newport",3,A4979,Pittsburgh,New York,2022-12-19 147 | ,,,,,,,,G5602,Pittsburgh,New Orleans,2023-06-26 148 | ,,,,,,,,I3994,Pittsburgh,Newport,2022-12-24 149 | 881,Brandon Zimmerman,Charlotte,23,"Orlando, Honolulu",2,"Orlando, Honolulu",2,X2718,Charlotte,Orlando,2023-10-11 150 | ,,,,,,,,Q7527,Charlotte,Honolulu,2024-04-06 151 | 982,Stuart Figueroa,Oklahoma City,75,"Washington, Nashville, Austin, Nashville",4,"Washington, Nashville, Austin, Nashville",4,W8601,Oklahoma City,Washington,2023-11-22 152 | ,,,,,,,,M5221,Oklahoma City,Nashville,2023-09-16 153 | ,,,,,,,,A1275,Oklahoma City,Austin,2023-04-04 154 | ,,,,,,,,X4455,Oklahoma City,Nashville,2024-04-23 155 | 26,Scott Walker,Vienna,39,San Francisco,1,San Francisco,1,S4575,Vienna,San Francisco,2023-05-08 156 | 485,Stephanie Hill,Cincinnati,33,San Jose,1,San Jose,1,C1321,Cincinnati,San Jose,2023-09-18 157 | 39,Heather Schmidt,San Diego,64,"Dallas, Boston, Savannah",3,"Dallas, Boston, Savannah",3,U2175,San Diego,Dallas,2024-07-09 158 | ,,,,,,,,K3079,San Diego,Boston,2023-07-10 159 | ,,,,,,,,S6608,San Diego,Savannah,2024-08-26 160 | 222,Crystal Gray,Nashville,53,"Orlando, Richmond, Jacksonville, New York, Savannah",5,"Orlando, Richmond, Jacksonville, New York, Savannah",5,C9249,Nashville,Orlando,2023-03-05 161 | ,,,,,,,,C3677,Nashville,Richmond,2024-04-14 162 | ,,,,,,,,N5675,Nashville,Jacksonville,2023-04-18 163 | ,,,,,,,,F5912,Nashville,New York,2023-10-27 164 | ,,,,,,,,B1017,Nashville,Savannah,2024-11-06 165 | 754,Gregory Jones,Portland,41,"Seattle, Denver, Washington",3,"Seattle, Denver, Washington",3,P9805,Portland,Seattle,2023-03-12 166 | ,,,,,,,,C2161,Portland,Denver,2023-12-08 167 | ,,,,,,,,V2628,Portland,Washington,2024-10-20 168 | 645,Belinda Wilson DVM,Gothenburg,63,"Marseille, Venice",2,"Marseille, Venice",2,F1544,Gothenburg,Marseille,2023-05-14 169 | ,,,,,,,,M5038,Gothenburg,Venice,2023-09-07 170 | 940,Juan Nelson,Prague,34,"Amsterdam, Prague, Rome, Helsinki, St. Petersburg",5,"Amsterdam, Prague, Rome, Helsinki, St. Petersburg",5,I9683,Prague,Amsterdam,2024-08-01 171 | ,,,,,,,,D1034,Prague,Prague,2024-09-14 172 | ,,,,,,,,G9871,Prague,Rome,2024-01-17 173 | ,,,,,,,,Z4629,Prague,Helsinki,2024-08-29 174 | ,,,,,,,,T9288,Prague,St. Petersburg,2024-08-31 175 | 613,Lisa Zhang,Marseille,35,"London, Salzburg",2,"London, Salzburg",2,C5606,Marseille,London,2024-10-16 176 | ,,,,,,,,S9995,Marseille,Salzburg,2024-02-10 177 | 681,Douglas Johnson,Zurich,68,"Dublin, Washington",2,"Dublin, Washington",2,Q7858,Zurich,Dublin,2023-10-17 178 | ,,,,,,,,R2697,Zurich,Washington,2022-12-16 179 | 428,Tracy Castaneda,Moscow,34,"Cologne, Gothenburg, Naples, Seville, Lyon",5,"Cologne, Gothenburg, Naples, Seville, Lyon",5,W3749,Moscow,Cologne,2024-06-05 180 | ,,,,,,,,Y1947,Moscow,Gothenburg,2024-08-04 181 | ,,,,,,,,O5424,Moscow,Naples,2022-12-12 182 | ,,,,,,,,L7293,Moscow,Seville,2024-03-24 183 | ,,,,,,,,N8744,Moscow,Lyon,2024-05-25 184 | 356,Kyle Jackson,Malaga,68,"Krakow, Helsinki, Salzburg, Lisbon",4,"Krakow, Helsinki, Salzburg, Lisbon",4,U3351,Malaga,Krakow,2023-03-13 185 | ,,,,,,,,O4901,Malaga,Helsinki,2023-03-08 186 | ,,,,,,,,V4505,Malaga,Salzburg,2023-10-15 187 | ,,,,,,,,Q9609,Malaga,Lisbon,2023-10-11 188 | 736,Daniel Coleman,London,36,"Moscow, Milan, Porto",3,"Moscow, Milan, Porto",3,T4516,London,Moscow,2024-02-06 189 | ,,,,,,,,G8313,London,Milan,2023-11-19 190 | ,,,,,,,,K3584,London,Porto,2024-04-27 191 | 354,Cassandra David,Cologne,50,"Bratislava, Oklahoma City",2,"Bratislava, Oklahoma City",2,W3085,Cologne,Bratislava,2023-05-09 192 | ,,,,,,,,A3117,Cologne,Oklahoma City,2024-01-06 193 | 501,Rhonda Bush,Washington,20,"Las Vegas, Charlotte, Scottsdale, Jacksonville, Oklahoma City",5,"Las Vegas, Charlotte, Scottsdale, Jacksonville, Oklahoma City",5,D3166,Washington,Las Vegas,2024-10-21 194 | ,,,,,,,,A7458,Washington,Charlotte,2023-07-10 195 | ,,,,,,,,P1573,Washington,Scottsdale,2022-11-24 196 | ,,,,,,,,Z6299,Washington,Jacksonville,2023-09-05 197 | ,,,,,,,,I1699,Washington,Oklahoma City,2024-09-15 198 | 572,Michael Nolan MD,Atlanta,28,Denver,1,Denver,1,B1466,Atlanta,Denver,2023-07-23 199 | 686,Terry Vazquez,Milwaukee,38,"Seattle, San Francisco, Key West, Tampa",4,"Seattle, San Francisco, Key West, Tampa",4,J9340,Milwaukee,Seattle,2024-05-23 200 | ,,,,,,,,S5250,Milwaukee,San Francisco,2023-07-22 201 | ,,,,,,,,N8348,Milwaukee,Key West,2023-02-14 202 | ,,,,,,,,U7200,Milwaukee,Tampa,2023-03-25 203 | 407,Katie Carter,Amsterdam,56,"Budapest, Lisbon, Budapest, Bergen",4,"Budapest, Lisbon, Budapest, Bergen",4,W7581,Amsterdam,Budapest,2023-08-17 204 | ,,,,,,,,R4805,Amsterdam,Lisbon,2024-04-26 205 | ,,,,,,,,G4460,Amsterdam,Budapest,2023-12-04 206 | ,,,,,,,,D7881,Amsterdam,Bergen,2023-04-16 207 | 875,Jacob Rogers,Las Vegas,50,"Milwaukee, San Diego, San Antonio",3,"Milwaukee, San Diego, San Antonio",3,F6369,Las Vegas,Milwaukee,2024-04-04 208 | ,,,,,,,,H8177,Las Vegas,San Diego,2023-05-25 209 | ,,,,,,,,N1590,Las Vegas,San Antonio,2024-05-04 210 | 622,Billy Jensen,Austin,63,Baltimore,1,Baltimore,1,P2195,Austin,Baltimore,2024-03-23 211 | 937,Jackson Pham,New Orleans,73,"Santa Fe, Chicago, San Francisco",3,"Santa Fe, Chicago, San Francisco",3,E5472,New Orleans,Santa Fe,2024-01-04 212 | ,,,,,,,,J6593,New Orleans,Chicago,2024-08-12 213 | ,,,,,,,,P9832,New Orleans,San Francisco,2022-11-11 214 | 930,Jennifer Martinez,San Jose,75,"Las Vegas, Fort Lauderdale, Denver, Asheville",4,"Las Vegas, Fort Lauderdale, Denver, Asheville",4,G4252,San Jose,Las Vegas,2023-10-21 215 | ,,,,,,,,Q3792,San Jose,Fort Lauderdale,2024-03-11 216 | ,,,,,,,,A8900,San Jose,Denver,2023-11-10 217 | ,,,,,,,,S6742,San Jose,Asheville,2024-03-30 218 | 260,Tyler Lewis,St. Louis,59,"San Jose, Indianapolis, Geneva, Seville, Santa Fe",5,"San Jose, Indianapolis, Geneva, Seville, Santa Fe",5,G6202,St. Louis,San Jose,2023-05-29 219 | ,,,,,,,,W4816,St. Louis,Indianapolis,2024-10-26 220 | ,,,,,,,,P7147,St. Louis,Geneva,2023-12-10 221 | ,,,,,,,,X5042,St. Louis,Seville,2023-11-13 222 | ,,,,,,,,M5793,St. Louis,Santa Fe,2024-10-24 223 | 766,Cassandra Johnson,Paris,65,"Ljubljana, Geneva, London, Madrid",4,"Ljubljana, Geneva, London, Madrid",4,T2795,Paris,Ljubljana,2023-05-14 224 | ,,,,,,,,R9939,Paris,Geneva,2022-12-13 225 | ,,,,,,,,M3833,Paris,London,2023-04-19 226 | ,,,,,,,,Y1808,Paris,Madrid,2023-11-08 227 | 19,Steve Hutchinson,Vienna,49,"Dublin, Florence, Cologne, Paris",4,"Dublin, Florence, Cologne, Paris",4,F1529,Vienna,Dublin,2023-01-15 228 | ,,,,,,,,B5144,Vienna,Florence,2024-07-01 229 | ,,,,,,,,K7286,Vienna,Cologne,2024-10-22 230 | ,,,,,,,,U5683,Vienna,Paris,2024-08-16 231 | 491,Ryan Harris,St. Louis,33,"Fort Lauderdale, Portland, Las Vegas, Indianapolis, Savannah",5,"Fort Lauderdale, Portland, Las Vegas, Indianapolis, Savannah",5,S6223,St. Louis,Fort Lauderdale,2023-07-10 232 | ,,,,,,,,J7122,St. Louis,Portland,2024-03-03 233 | ,,,,,,,,W6256,St. Louis,Las Vegas,2023-10-24 234 | ,,,,,,,,Y4863,St. Louis,Indianapolis,2023-03-17 235 | ,,,,,,,,P3778,St. Louis,Savannah,2023-12-03 236 | 684,Kathleen Smith,Baltimore,71,"Tampa, San Diego, Key West, San Jose, Cincinnati",5,"Tampa, San Diego, Key West, San Jose, Cincinnati",5,U8718,Baltimore,Tampa,2024-08-09 237 | ,,,,,,,,M2940,Baltimore,San Diego,2022-12-05 238 | ,,,,,,,,Z6891,Baltimore,Key West,2023-12-05 239 | ,,,,,,,,Y4181,Baltimore,San Jose,2023-12-25 240 | ,,,,,,,,A9504,Baltimore,Cincinnati,2024-05-18 241 | 487,Kim Dennis,Lyon,67,"Nice, Marseille, Malaga",3,"Nice, Marseille, Malaga",3,U5521,Lyon,Nice,2023-12-17 242 | ,,,,,,,,G8524,Lyon,Marseille,2023-06-28 243 | ,,,,,,,,R7846,Lyon,Malaga,2022-11-20 244 | 330,Ryan Roberts,Gothenburg,37,"Geneva, Brussels, Sacramento, Reykjavik",4,"Geneva, Brussels, Sacramento, Reykjavik",4,B6440,Gothenburg,Geneva,2024-07-29 245 | ,,,,,,,,U9073,Gothenburg,Brussels,2023-09-28 246 | ,,,,,,,,Y3467,Gothenburg,Sacramento,2024-10-30 247 | ,,,,,,,,J7708,Gothenburg,Reykjavik,2023-06-27 248 | 630,Travis Torres,Phoenix,44,Orlando,1,Orlando,1,Z6077,Phoenix,Orlando,2024-03-04 249 | 903,Timothy Hancock,Philadelphia,58,"Sacramento, Moscow, Berlin",3,"Sacramento, Moscow, Berlin",3,W2469,Philadelphia,Sacramento,2023-06-13 250 | ,,,,,,,,X2037,Philadelphia,Moscow,2024-07-23 251 | ,,,,,,,,P4971,Philadelphia,Berlin,2024-08-27 252 | 374,Carol Mann,Las Vegas,23,Pittsburgh,1,Pittsburgh,1,P3342,Las Vegas,Pittsburgh,2024-09-10 253 | 382,Christopher Taylor,Budapest,21,"St. Petersburg, Helsinki, Reykjavik",3,"St. Petersburg, Helsinki, Reykjavik",3,X7790,Budapest,St. Petersburg,2022-11-17 254 | ,,,,,,,,T9322,Budapest,Helsinki,2023-03-05 255 | ,,,,,,,,C5991,Budapest,Reykjavik,2023-02-13 256 | 626,Thomas Baird,Sedona,27,"Milwaukee, Asheville, Milwaukee",3,"Milwaukee, Asheville, Milwaukee",3,E6598,Sedona,Milwaukee,2024-01-30 257 | ,,,,,,,,I4416,Sedona,Asheville,2023-01-30 258 | ,,,,,,,,A5912,Sedona,Milwaukee,2024-04-22 259 | 791,David Clark,Seville,20,Prague,1,Prague,1,Q1990,Seville,Prague,2023-07-21 260 | 260,Justin Johnson,Austin,19,"Seville, Jacksonville, Atlanta, Houston, Pittsburgh",5,"Seville, Jacksonville, Atlanta, Houston, Pittsburgh",5,T9608,Austin,Seville,2023-12-29 261 | ,,,,,,,,N2237,Austin,Jacksonville,2024-04-20 262 | ,,,,,,,,D9852,Austin,Atlanta,2023-01-07 263 | ,,,,,,,,I1338,Austin,Houston,2024-03-14 264 | ,,,,,,,,K8149,Austin,Pittsburgh,2024-03-22 265 | 767,Brenda Hayes,Lisbon,64,"Vienna, Tallinn, Salt Lake City",3,"Vienna, Tallinn, Salt Lake City",3,B8700,Lisbon,Vienna,2023-03-05 266 | ,,,,,,,,T1266,Lisbon,Tallinn,2024-06-29 267 | ,,,,,,,,I7372,Lisbon,Salt Lake City,2023-04-14 268 | 821,Yolanda Collier,Edinburgh,55,Madrid,1,Madrid,1,X9421,Edinburgh,Madrid,2022-11-22 269 | 344,Alicia Richmond,Atlanta,22,Richmond,1,Richmond,1,U5700,Atlanta,Richmond,2022-12-02 270 | 703,Jack Thompson,Edinburgh,25,"Krakow, Moscow",2,"Krakow, Moscow",2,V4033,Edinburgh,Krakow,2023-10-14 271 | ,,,,,,,,G5923,Edinburgh,Moscow,2023-01-18 272 | 223,Christina Murphy,Salzburg,73,"Nice, Copenhagen, Milan, Key West",4,"Nice, Copenhagen, Milan, Key West",4,F4961,Salzburg,Nice,2024-05-15 273 | ,,,,,,,,D1134,Salzburg,Copenhagen,2023-10-08 274 | ,,,,,,,,R3800,Salzburg,Milan,2024-03-05 275 | ,,,,,,,,L7267,Salzburg,Key West,2023-12-29 276 | 760,Kelly Wilson,Reykjavik,56,"Lisbon, San Jose, Stockholm, Stockholm, Sedona",5,"Lisbon, San Jose, Stockholm, Stockholm, Sedona",5,L2726,Reykjavik,Lisbon,2024-01-17 277 | ,,,,,,,,F5125,Reykjavik,San Jose,2024-03-30 278 | ,,,,,,,,W7987,Reykjavik,Stockholm,2023-10-10 279 | ,,,,,,,,G9699,Reykjavik,Stockholm,2022-12-09 280 | ,,,,,,,,T8869,Reykjavik,Sedona,2023-04-08 281 | 405,William Casey,Budapest,37,Oslo,1,Oslo,1,E3116,Budapest,Oslo,2024-01-05 282 | 63,Gregg Guerrero,Salzburg,52,"Barcelona, Cologne, Amsterdam, Milan",4,"Barcelona, Cologne, Amsterdam, Milan",4,F1409,Salzburg,Barcelona,2023-07-07 283 | ,,,,,,,,Y1320,Salzburg,Cologne,2024-05-04 284 | ,,,,,,,,F4369,Salzburg,Amsterdam,2023-05-28 285 | ,,,,,,,,F7361,Salzburg,Milan,2023-06-04 286 | 162,Terri Bishop,Denver,42,"Key West, Seattle, Bratislava, Houston",4,"Key West, Seattle, Bratislava, Houston",4,X8419,Denver,Key West,2023-12-23 287 | ,,,,,,,,Q4879,Denver,Seattle,2024-08-11 288 | ,,,,,,,,E7556,Denver,Bratislava,2023-03-24 289 | ,,,,,,,,Z4007,Denver,Houston,2023-04-17 290 | 646,Hannah Klein,Savannah,34,"Honolulu, Charlotte, San Diego, Copenhagen",4,"Honolulu, Charlotte, San Diego, Copenhagen",4,D3930,Savannah,Honolulu,2023-04-18 291 | ,,,,,,,,G8063,Savannah,Charlotte,2023-09-26 292 | ,,,,,,,,E4569,Savannah,San Diego,2023-01-24 293 | ,,,,,,,,V6694,Savannah,Copenhagen,2024-04-22 294 | 666,Amber Miller,Newport,50,"Seattle, Kansas City",2,"Seattle, Kansas City",2,C1839,Newport,Seattle,2024-05-20 295 | ,,,,,,,,M8640,Newport,Kansas City,2023-09-10 296 | 61,Michael Watkins,Pittsburgh,59,"Cincinnati, Santa Fe, Savannah, San Diego",4,"Cincinnati, Santa Fe, Savannah, San Diego",4,T8686,Pittsburgh,Cincinnati,2022-12-14 297 | ,,,,,,,,V8907,Pittsburgh,Santa Fe,2024-07-16 298 | ,,,,,,,,F1642,Pittsburgh,Savannah,2023-10-22 299 | ,,,,,,,,R4904,Pittsburgh,San Diego,2023-03-09 300 | 589,Kyle Reyes,Charlotte,33,"Charlotte, Los Angeles, Atlanta",3,"Charlotte, Los Angeles, Atlanta",3,J1275,Charlotte,Charlotte,2022-12-22 301 | ,,,,,,,,Q6152,Charlotte,Los Angeles,2024-09-28 302 | ,,,,,,,,L3581,Charlotte,Atlanta,2024-01-01 303 | 759,Kathy Davidson,San Diego,41,New York,1,New York,1,T3325,San Diego,New York,2024-01-01 304 | 727,Rachel Johnson,St. Petersburg,50,"Reykjavik, Vienna, Marseille, Prague, Dubrovnik",5,"Reykjavik, Vienna, Marseille, Prague, Dubrovnik",5,Z1062,St. Petersburg,Reykjavik,2023-05-14 305 | ,,,,,,,,Y5233,St. Petersburg,Vienna,2024-09-16 306 | ,,,,,,,,R4985,St. Petersburg,Marseille,2023-04-15 307 | ,,,,,,,,O4290,St. Petersburg,Prague,2024-04-17 308 | ,,,,,,,,A8888,St. Petersburg,Dubrovnik,2024-08-12 309 | 852,Robert Moore DVM,Indianapolis,20,"Los Angeles, Portland, Phoenix",3,"Los Angeles, Portland, Phoenix",3,W6684,Indianapolis,Los Angeles,2024-03-26 310 | ,,,,,,,,V7128,Indianapolis,Portland,2023-06-26 311 | ,,,,,,,,I8308,Indianapolis,Phoenix,2024-03-29 312 | 224,Jared Willis,Houston,26,San Antonio,1,San Antonio,1,J7469,Houston,San Antonio,2024-01-11 313 | 968,Bryce Gonzalez MD,Santa Fe,39,San Antonio,1,San Antonio,1,L8401,Santa Fe,San Antonio,2023-02-24 314 | 203,Brittany Singh,Milan,41,Philadelphia,1,Philadelphia,1,V8471,Milan,Philadelphia,2023-05-02 315 | 676,George Jones,Las Vegas,51,"Los Angeles, Denver",2,"Los Angeles, Denver",2,J5522,Las Vegas,Los Angeles,2023-02-06 316 | ,,,,,,,,N3065,Las Vegas,Denver,2023-06-10 317 | 7,Kevin Lee,Nice,39,Nice,1,Nice,1,R8133,Nice,Nice,2024-07-25 318 | 295,Amanda Johnson,Seattle,31,"Kansas City, New York",2,"Kansas City, New York",2,B8350,Seattle,Kansas City,2024-05-28 319 | ,,,,,,,,H5102,Seattle,New York,2023-09-12 320 | 62,Andrea Mitchell,Krakow,44,"Zurich, Lyon",2,"Zurich, Lyon",2,Y9202,Krakow,Zurich,2024-02-07 321 | ,,,,,,,,H2457,Krakow,Lyon,2023-01-14 322 | 122,Patrick Christian,Honolulu,31,Jacksonville,1,Jacksonville,1,J1226,Honolulu,Jacksonville,2022-12-29 323 | 319,Kathleen Fisher,St. Petersburg,31,"Paris, Krakow, London, Dubrovnik",4,"Paris, Krakow, London, Dubrovnik",4,I2896,St. Petersburg,Paris,2022-12-08 324 | ,,,,,,,,R9390,St. Petersburg,Krakow,2024-08-14 325 | ,,,,,,,,I4463,St. Petersburg,London,2024-05-15 326 | ,,,,,,,,A8507,St. Petersburg,Dubrovnik,2024-09-18 327 | 182,Margaret Hernandez,Berlin,70,"Philadelphia, Marseille",2,"Philadelphia, Marseille",2,V8033,Berlin,Philadelphia,2024-07-16 328 | ,,,,,,,,L3889,Berlin,Marseille,2023-03-19 329 | 998,Anthony Carr,Charleston,61,"Fort Lauderdale, Oklahoma City",2,"Fort Lauderdale, Oklahoma City",2,Z3198,Charleston,Fort Lauderdale,2024-03-22 330 | ,,,,,,,,I4753,Charleston,Oklahoma City,2024-09-01 331 | 394,Laura Bell,Moscow,38,Naples,1,Naples,1,H9964,Moscow,Naples,2024-03-06 332 | 446,Nicholas Alvarez,Denver,38,St. Louis,1,St. Louis,1,T7475,Denver,St. Louis,2023-01-04 333 | 76,Paul West,Portland,70,Washington,1,Washington,1,L1461,Portland,Washington,2023-11-01 334 | 67,Melissa Garcia,Minneapolis,39,"Kansas City, Los Angeles",2,"Kansas City, Los Angeles",2,N3013,Minneapolis,Kansas City,2024-06-05 335 | ,,,,,,,,X3771,Minneapolis,Los Angeles,2023-12-02 336 | 53,Stephen Matthews,Charlotte,47,"Sacramento, San Antonio",2,"Sacramento, San Antonio",2,D1574,Charlotte,Sacramento,2023-05-07 337 | ,,,,,,,,D1454,Charlotte,San Antonio,2023-08-27 338 | 729,David Bowers,Sedona,31,Minneapolis,1,Minneapolis,1,B3051,Sedona,Minneapolis,2024-06-22 339 | 344,Debbie Warren,Barcelona,42,"Lisbon, Luxembourg, Bologna",3,"Lisbon, Luxembourg, Bologna",3,J8001,Barcelona,Lisbon,2023-04-26 340 | ,,,,,,,,M6153,Barcelona,Luxembourg,2024-10-31 341 | ,,,,,,,,E6484,Barcelona,Bologna,2023-04-25 342 | 686,Marcus James,Warsaw,26,"Krakow, Reykjavik, Berlin, Athens",4,"Krakow, Reykjavik, Berlin, Athens",4,M7827,Warsaw,Krakow,2023-02-19 343 | ,,,,,,,,B4058,Warsaw,Reykjavik,2023-10-06 344 | ,,,,,,,,M5794,Warsaw,Berlin,2023-05-12 345 | ,,,,,,,,E7277,Warsaw,Athens,2023-10-22 346 | 796,Robin Taylor,Savannah,30,"St. Louis, San Diego, Providence",3,"St. Louis, San Diego, Providence",3,P3976,Savannah,St. Louis,2023-02-06 347 | ,,,,,,,,S6054,Savannah,San Diego,2023-04-24 348 | ,,,,,,,,X5600,Savannah,Providence,2024-02-29 349 | 18,Kimberly Crawford,Edinburgh,32,"Valencia, Barcelona",2,"Valencia, Barcelona",2,X6068,Edinburgh,Valencia,2023-11-13 350 | ,,,,,,,,U7738,Edinburgh,Barcelona,2024-06-10 351 | 30,Angela Rogers,Baltimore,50,"Tampa, Sacramento, San Antonio",3,"Tampa, Sacramento, San Antonio",3,V8521,Baltimore,Tampa,2024-10-20 352 | ,,,,,,,,Y3080,Baltimore,Sacramento,2023-01-07 353 | ,,,,,,,,B3345,Baltimore,San Antonio,2023-12-01 354 | 754,Michaela Oneill,Dallas,67,Las Vegas,1,Las Vegas,1,G6597,Dallas,Las Vegas,2023-08-21 355 | 974,Chelsea Lozano,Stockholm,67,"Naples, Zurich, Naples",3,"Naples, Zurich, Naples",3,U4466,Stockholm,Naples,2023-06-23 356 | ,,,,,,,,J2093,Stockholm,Zurich,2024-08-28 357 | ,,,,,,,,T6202,Stockholm,Naples,2023-01-25 358 | 427,Ryan Wolfe,Dubrovnik,66,"Helsinki, Munich, Helsinki, Stockholm",4,"Helsinki, Munich, Helsinki, Stockholm",4,T7292,Dubrovnik,Helsinki,2024-08-28 359 | ,,,,,,,,Q4875,Dubrovnik,Munich,2024-06-22 360 | ,,,,,,,,D4707,Dubrovnik,Helsinki,2023-07-27 361 | ,,,,,,,,X1023,Dubrovnik,Stockholm,2024-08-04 362 | 636,Anna Robinson,Edinburgh,63,"Bologna, Cincinnati",2,"Bologna, Cincinnati",2,R6845,Edinburgh,Bologna,2023-05-05 363 | ,,,,,,,,Q4289,Edinburgh,Cincinnati,2023-11-05 364 | 876,Ronald Brewer,Helsinki,49,"Malaga, Dublin, Prague, Ljubljana, Athens",5,"Malaga, Dublin, Prague, Ljubljana, Athens",5,J4021,Helsinki,Malaga,2023-05-27 365 | ,,,,,,,,Q2521,Helsinki,Dublin,2024-03-20 366 | ,,,,,,,,E7733,Helsinki,Prague,2023-10-25 367 | ,,,,,,,,T6257,Helsinki,Ljubljana,2024-08-16 368 | ,,,,,,,,T9235,Helsinki,Athens,2024-10-01 369 | 149,Ashley Williams,Baltimore,41,"Chicago, Jacksonville, Denver, Washington",4,"Chicago, Jacksonville, Denver, Washington",4,G5705,Baltimore,Chicago,2023-06-14 370 | ,,,,,,,,K2365,Baltimore,Jacksonville,2024-09-20 371 | ,,,,,,,,W1465,Baltimore,Denver,2023-05-19 372 | ,,,,,,,,W5845,Baltimore,Washington,2023-07-11 373 | 762,Stephen Contreras,Lisbon,24,"Salzburg, Dublin, Cologne, Copenhagen",4,"Salzburg, Dublin, Cologne, Copenhagen",4,I1707,Lisbon,Salzburg,2023-09-21 374 | ,,,,,,,,C2209,Lisbon,Dublin,2022-12-12 375 | ,,,,,,,,H3807,Lisbon,Cologne,2023-11-27 376 | ,,,,,,,,R8092,Lisbon,Copenhagen,2023-04-20 377 | 96,Abigail Hogan,Cincinnati,70,"Seattle, Seattle, Sacramento",3,"Seattle, Seattle, Sacramento",3,R1764,Cincinnati,Seattle,2023-01-16 378 | ,,,,,,,,R8331,Cincinnati,Seattle,2023-11-27 379 | ,,,,,,,,M6656,Cincinnati,Sacramento,2024-09-21 380 | 557,Alexa Morales,Sedona,56,Atlanta,1,Atlanta,1,C6271,Sedona,Atlanta,2023-11-29 381 | 618,Karen Fuentes,Budapest,73,Atlanta,1,Atlanta,1,M8930,Budapest,Atlanta,2024-08-24 382 | 106,Lindsey Ward,Warsaw,44,"Dublin, Copenhagen",2,"Dublin, Copenhagen",2,K7469,Warsaw,Dublin,2024-10-29 383 | ,,,,,,,,W5114,Warsaw,Copenhagen,2024-09-12 384 | 912,Joshua Nolan,Sedona,68,Oklahoma City,1,Oklahoma City,1,K2805,Sedona,Oklahoma City,2024-08-13 385 | 267,Wayne James,Reykjavik,61,"Tallinn, Nice, Tallinn, Asheville, Orlando",5,"Tallinn, Nice, Tallinn, Asheville, Orlando",5,L9412,Reykjavik,Tallinn,2023-10-20 386 | ,,,,,,,,I6939,Reykjavik,Nice,2023-10-14 387 | ,,,,,,,,J2457,Reykjavik,Tallinn,2024-08-06 388 | ,,,,,,,,Y8794,Reykjavik,Asheville,2024-03-16 389 | ,,,,,,,,I3063,Reykjavik,Orlando,2023-09-08 390 | 602,Debra Perez,Barcelona,49,"St. Petersburg, Brussels, Reykjavik, Savannah, St. Petersburg",5,"St. Petersburg, Brussels, Reykjavik, Savannah, St. Petersburg",5,F9985,Barcelona,St. Petersburg,2024-02-28 391 | ,,,,,,,,J2306,Barcelona,Brussels,2024-10-05 392 | ,,,,,,,,X5686,Barcelona,Reykjavik,2023-08-25 393 | ,,,,,,,,O6890,Barcelona,Savannah,2024-06-22 394 | ,,,,,,,,W9378,Barcelona,St. Petersburg,2024-02-03 395 | 633,Jeff Hensley,Krakow,74,"Vienna, Bratislava, Berlin",3,"Vienna, Bratislava, Berlin",3,D1873,Krakow,Vienna,2024-08-31 396 | ,,,,,,,,Z7755,Krakow,Bratislava,2024-08-12 397 | ,,,,,,,,E9749,Krakow,Berlin,2024-01-05 398 | 992,Willie Edwards,Istanbul,27,"Athens, Naples, Geneva, Porto",4,"Athens, Naples, Geneva, Porto",4,R7337,Istanbul,Athens,2023-11-23 399 | ,,,,,,,,B8913,Istanbul,Naples,2023-06-04 400 | ,,,,,,,,N1930,Istanbul,Geneva,2024-05-19 401 | ,,,,,,,,X8036,Istanbul,Porto,2023-03-05 402 | 920,Justin Mays,Luxembourg,48,Gothenburg,1,Gothenburg,1,K9667,Luxembourg,Gothenburg,2024-03-05 403 | 340,Jacqueline Sims,San Jose,73,"Las Vegas, San Jose, Kansas City, Richmond, Santa Fe",5,"Las Vegas, San Jose, Kansas City, Richmond, Santa Fe",5,S1489,San Jose,Las Vegas,2022-11-15 404 | ,,,,,,,,R9901,San Jose,San Jose,2023-05-23 405 | ,,,,,,,,T2175,San Jose,Kansas City,2023-10-23 406 | ,,,,,,,,W6454,San Jose,Richmond,2023-04-11 407 | ,,,,,,,,C6513,San Jose,Santa Fe,2022-11-09 408 | 369,Amy Barnett,St. Petersburg,75,"Nice, Indianapolis, Austin, Moscow, Salzburg",5,"Nice, Indianapolis, Austin, Moscow, Salzburg",5,G7516,St. Petersburg,Nice,2023-03-26 409 | ,,,,,,,,E5663,St. Petersburg,Indianapolis,2024-03-03 410 | ,,,,,,,,Z2076,St. Petersburg,Austin,2024-06-21 411 | ,,,,,,,,N5402,St. Petersburg,Moscow,2023-01-17 412 | ,,,,,,,,E5472,St. Petersburg,Salzburg,2023-03-29 413 | 313,Gabriella Maddox,Bordeaux,50,"Madrid, London, Malaga",3,"Madrid, London, Malaga",3,X2007,Bordeaux,Madrid,2024-02-09 414 | ,,,,,,,,O2886,Bordeaux,London,2023-04-27 415 | ,,,,,,,,Q3213,Bordeaux,Malaga,2023-08-31 416 | 216,Anthony Lee,Bergen,30,"Budapest, Marseille, Porto, Prague",4,"Budapest, Marseille, Porto, Prague",4,X5398,Bergen,Budapest,2023-03-17 417 | ,,,,,,,,Z8352,Bergen,Marseille,2023-12-31 418 | ,,,,,,,,E4763,Bergen,Porto,2024-09-17 419 | ,,,,,,,,B4046,Bergen,Prague,2024-08-30 420 | 848,Robert Patel,Nashville,71,"Phoenix, Cincinnati",2,"Phoenix, Cincinnati",2,B8016,Nashville,Phoenix,2024-03-20 421 | ,,,,,,,,K5760,Nashville,Cincinnati,2023-03-16 422 | 717,Allen Myers,Cincinnati,75,"Jacksonville, Kansas City, Baltimore",3,"Jacksonville, Kansas City, Baltimore",3,X6437,Cincinnati,Jacksonville,2023-12-27 423 | ,,,,,,,,K5350,Cincinnati,Kansas City,2023-01-15 424 | ,,,,,,,,B6477,Cincinnati,Baltimore,2024-02-02 425 | 107,Alexis Odom,Seattle,52,"Boston, Atlanta",2,"Boston, Atlanta",2,S1219,Seattle,Boston,2024-05-07 426 | ,,,,,,,,H7087,Seattle,Atlanta,2024-08-12 427 | 566,Christy Robertson,London,54,"Nice, Malaga",2,"Nice, Malaga",2,I2630,London,Nice,2024-03-30 428 | ,,,,,,,,T2407,London,Malaga,2023-05-25 429 | 158,Evan Harrison,Los Angeles,70,"Cincinnati, Charlotte",2,"Cincinnati, Charlotte",2,D6305,Los Angeles,Cincinnati,2024-08-04 430 | ,,,,,,,,I7332,Los Angeles,Charlotte,2023-04-15 431 | 189,Anthony Smith,Warsaw,65,"Geneva, Bologna, Marseille, Madrid, Venice",5,"Geneva, Bologna, Marseille, Madrid, Venice",5,U2241,Warsaw,Geneva,2023-01-26 432 | ,,,,,,,,V8289,Warsaw,Bologna,2023-06-15 433 | ,,,,,,,,E6838,Warsaw,Marseille,2023-12-03 434 | ,,,,,,,,E9704,Warsaw,Madrid,2023-09-14 435 | ,,,,,,,,R7174,Warsaw,Venice,2023-05-11 436 | 566,Jennifer Howell,Washington,20,Savannah,1,Savannah,1,T1641,Washington,Savannah,2024-06-27 437 | 602,Emily Torres,Milan,43,"Bratislava, Lyon, Naples, Florence, Bratislava",5,"Bratislava, Lyon, Naples, Florence, Bratislava",5,S3217,Milan,Bratislava,2024-10-12 438 | ,,,,,,,,C5878,Milan,Lyon,2023-10-21 439 | ,,,,,,,,I5100,Milan,Naples,2023-02-22 440 | ,,,,,,,,H3773,Milan,Florence,2024-10-04 441 | ,,,,,,,,Q4578,Milan,Bratislava,2022-11-20 442 | 580,Yvette Sanders,Austin,64,"Denver, Santa Fe, Cincinnati, Washington",4,"Denver, Santa Fe, Cincinnati, Washington",4,S5832,Austin,Denver,2023-08-31 443 | ,,,,,,,,P6317,Austin,Santa Fe,2023-09-30 444 | ,,,,,,,,R9242,Austin,Cincinnati,2024-08-19 445 | ,,,,,,,,K8970,Austin,Washington,2024-03-28 446 | 777,Troy Saunders,Santa Fe,75,Nashville,1,Nashville,1,S9934,Santa Fe,Nashville,2024-05-11 447 | 392,Alexis Edwards,Reykjavik,49,"Ljubljana, Tallinn, Stockholm, Vienna",4,"Ljubljana, Tallinn, Stockholm, Vienna",4,A9069,Reykjavik,Ljubljana,2024-01-06 448 | ,,,,,,,,J1068,Reykjavik,Tallinn,2024-04-09 449 | ,,,,,,,,Y7899,Reykjavik,Stockholm,2024-10-05 450 | ,,,,,,,,L6828,Reykjavik,Vienna,2024-04-03 451 | 478,Rebekah Brown,Honolulu,50,"Lyon, Las Vegas, Barcelona, Providence",4,"Lyon, Las Vegas, Barcelona, Providence",4,I3251,Honolulu,Lyon,2023-08-21 452 | ,,,,,,,,E6774,Honolulu,Las Vegas,2024-01-14 453 | ,,,,,,,,A1300,Honolulu,Barcelona,2024-01-22 454 | ,,,,,,,,A8724,Honolulu,Providence,2023-06-21 455 | 233,William Smith,Rome,32,"Vienna, Vienna, Tallinn, Lyon, Nice",5,"Vienna, Vienna, Tallinn, Lyon, Nice",5,O9881,Rome,Vienna,2024-10-21 456 | ,,,,,,,,O2362,Rome,Vienna,2024-02-25 457 | ,,,,,,,,Y5541,Rome,Tallinn,2024-05-01 458 | ,,,,,,,,P4215,Rome,Lyon,2023-02-22 459 | ,,,,,,,,L3362,Rome,Nice,2023-07-30 460 | 51,Mark Heath,Lyon,46,"Dubrovnik, Ljubljana, Dubrovnik, Venice, Barcelona",5,"Dubrovnik, Ljubljana, Dubrovnik, Venice, Barcelona",5,H7560,Lyon,Dubrovnik,2023-10-08 461 | ,,,,,,,,V6950,Lyon,Ljubljana,2023-02-14 462 | ,,,,,,,,X2105,Lyon,Dubrovnik,2024-10-24 463 | ,,,,,,,,N9506,Lyon,Venice,2024-11-01 464 | ,,,,,,,,Y5685,Lyon,Barcelona,2024-01-28 465 | 113,Hayley Sanders,Rome,33,"Athens, Edinburgh",2,"Athens, Edinburgh",2,P8477,Rome,Athens,2024-06-10 466 | ,,,,,,,,D5179,Rome,Edinburgh,2023-10-10 467 | 609,Todd Stone,Bordeaux,67,"London, Malaga, Barcelona, Berlin",4,"London, Malaga, Barcelona, Berlin",4,H6290,Bordeaux,London,2024-06-25 468 | ,,,,,,,,P5019,Bordeaux,Malaga,2023-05-24 469 | ,,,,,,,,K7774,Bordeaux,Barcelona,2023-05-25 470 | ,,,,,,,,R9625,Bordeaux,Berlin,2024-06-15 471 | 471,Erin Trevino,Berlin,60,"Dublin, Stockholm",2,"Dublin, Stockholm",2,R5693,Berlin,Dublin,2024-08-26 472 | ,,,,,,,,L5340,Berlin,Stockholm,2023-05-19 473 | 112,Kevin Clarke,Rome,73,"Vienna, Oslo",2,"Vienna, Oslo",2,W2459,Rome,Vienna,2024-04-25 474 | ,,,,,,,,G1049,Rome,Oslo,2024-01-21 475 | 118,Michael Jones,Reykjavik,37,Naples,1,Naples,1,U8992,Reykjavik,Naples,2023-05-12 476 | 117,Kyle Owen,Munich,58,"Warsaw, Oslo, Bordeaux",3,"Warsaw, Oslo, Bordeaux",3,S5967,Munich,Warsaw,2024-05-14 477 | ,,,,,,,,M3227,Munich,Oslo,2023-08-11 478 | ,,,,,,,,O7511,Munich,Bordeaux,2024-08-24 479 | 584,Steven Costa,Santa Fe,51,"San Antonio, Rome, Phoenix, San Diego, Philadelphia",5,"San Antonio, Rome, Phoenix, San Diego, Philadelphia",5,E4008,Santa Fe,San Antonio,2023-07-05 480 | ,,,,,,,,M5411,Santa Fe,Rome,2022-11-09 481 | ,,,,,,,,B8668,Santa Fe,Phoenix,2023-06-10 482 | ,,,,,,,,J8031,Santa Fe,San Diego,2022-11-20 483 | ,,,,,,,,F4912,Santa Fe,Philadelphia,2024-10-06 484 | 433,Robert Campbell,San Jose,72,"Denver, Vienna, Santa Fe, Baltimore, Portland",5,"Denver, Vienna, Santa Fe, Baltimore, Portland",5,N6068,San Jose,Denver,2023-01-30 485 | ,,,,,,,,W5846,San Jose,Vienna,2024-03-28 486 | ,,,,,,,,U2137,San Jose,Santa Fe,2023-12-31 487 | ,,,,,,,,X6061,San Jose,Baltimore,2022-12-22 488 | ,,,,,,,,N4438,San Jose,Portland,2024-10-21 489 | 170,Jessica Leonard,Gothenburg,65,"St. Petersburg, Brussels, Milan",3,"St. Petersburg, Brussels, Milan",3,U8061,Gothenburg,St. Petersburg,2023-12-05 490 | ,,,,,,,,E2314,Gothenburg,Brussels,2023-04-22 491 | ,,,,,,,,S3944,Gothenburg,Milan,2023-11-13 492 | 67,Jesus Casey,Indianapolis,38,"Denver, Denver, Phoenix",3,"Denver, Denver, Phoenix",3,A1874,Indianapolis,Denver,2023-04-09 493 | ,,,,,,,,S4834,Indianapolis,Denver,2022-11-09 494 | ,,,,,,,,A4981,Indianapolis,Phoenix,2023-01-21 495 | 177,Christopher Wilson,Istanbul,39,"Madrid, Rome, Luxembourg",3,"Madrid, Rome, Luxembourg",3,L6657,Istanbul,Madrid,2024-07-24 496 | ,,,,,,,,R7445,Istanbul,Rome,2022-12-05 497 | ,,,,,,,,F1544,Istanbul,Luxembourg,2024-01-09 498 | 62,Erica Wilkinson,Kansas City,44,"Baltimore, Jacksonville",2,"Baltimore, Jacksonville",2,I4148,Kansas City,Baltimore,2023-06-28 499 | ,,,,,,,,T7466,Kansas City,Jacksonville,2023-10-31 500 | 773,Shannon Reynolds,Edinburgh,48,"Brussels, Dubrovnik",2,"Brussels, Dubrovnik",2,W5322,Edinburgh,Brussels,2024-09-03 501 | ,,,,,,,,Q8074,Edinburgh,Dubrovnik,2024-08-26 502 | 824,Thomas Thompson,Houston,62,"Sacramento, New Orleans, Portland, Indianapolis, Sacramento",5,"Sacramento, New Orleans, Portland, Indianapolis, Sacramento",5,E9610,Houston,Sacramento,2023-08-19 503 | ,,,,,,,,W2171,Houston,New Orleans,2023-06-12 504 | ,,,,,,,,R4786,Houston,Portland,2023-11-15 505 | ,,,,,,,,T5536,Houston,Indianapolis,2023-04-03 506 | ,,,,,,,,T5120,Houston,Sacramento,2024-05-05 507 | 22,Jose Wang,Portland,53,"Tampa, Salt Lake City",2,"Tampa, Salt Lake City",2,U3281,Portland,Tampa,2023-05-25 508 | ,,,,,,,,V4609,Portland,Salt Lake City,2023-02-19 509 | 595,Rita Bennett,Berlin,71,"Valencia, Athens, Bergen",3,"Valencia, Athens, Bergen",3,Z5931,Berlin,Valencia,2023-07-26 510 | ,,,,,,,,D6729,Berlin,Athens,2023-07-16 511 | ,,,,,,,,T3107,Berlin,Bergen,2024-05-14 512 | 967,Barbara Moore,San Jose,63,"Los Angeles, Tampa, Phoenix",3,"Los Angeles, Tampa, Phoenix",3,L7287,San Jose,Los Angeles,2024-06-02 513 | ,,,,,,,,H6052,San Jose,Tampa,2024-01-29 514 | ,,,,,,,,M3892,San Jose,Phoenix,2023-08-03 515 | 852,Lynn Watson,Porto,23,"Rome, Reykjavik, Gothenburg",3,"Rome, Reykjavik, Gothenburg",3,Z3414,Porto,Rome,2024-05-18 516 | ,,,,,,,,B5108,Porto,Reykjavik,2023-01-21 517 | ,,,,,,,,Y8054,Porto,Gothenburg,2024-10-20 518 | 471,Melody Cummings,Milwaukee,67,"Baltimore, Denver, Miami",3,"Baltimore, Denver, Miami",3,M5773,Milwaukee,Baltimore,2022-11-13 519 | ,,,,,,,,Z5652,Milwaukee,Denver,2023-04-09 520 | ,,,,,,,,G9782,Milwaukee,Miami,2024-05-07 521 | 276,John Hall,Oslo,30,"Athens, Lyon, Geneva, Athens, Newport",5,"Athens, Lyon, Geneva, Athens, Newport",5,J2124,Oslo,Athens,2024-06-08 522 | ,,,,,,,,H2538,Oslo,Lyon,2022-12-24 523 | ,,,,,,,,T4116,Oslo,Geneva,2023-09-13 524 | ,,,,,,,,C3742,Oslo,Athens,2023-03-21 525 | ,,,,,,,,U1797,Oslo,Newport,2024-10-11 526 | 23,Heather Williams,Athens,36,"Baltimore, Helsinki",2,"Baltimore, Helsinki",2,P1382,Athens,Baltimore,2022-12-17 527 | ,,,,,,,,Z4647,Athens,Helsinki,2024-08-06 528 | 261,Kelly Johnson,Oklahoma City,55,"Oklahoma City, Fort Lauderdale, New Orleans, Los Angeles, New Orleans",5,"Oklahoma City, Fort Lauderdale, New Orleans, Los Angeles, New Orleans",5,V5132,Oklahoma City,Oklahoma City,2023-02-11 529 | ,,,,,,,,M5397,Oklahoma City,Fort Lauderdale,2024-02-29 530 | ,,,,,,,,Y8688,Oklahoma City,New Orleans,2024-07-27 531 | ,,,,,,,,T1691,Oklahoma City,Los Angeles,2024-09-01 532 | ,,,,,,,,F8162,Oklahoma City,New Orleans,2023-07-06 533 | 817,Monica Coleman,Tampa,72,"New York, San Francisco, Houston, Tampa, St. Louis",5,"New York, San Francisco, Houston, Tampa, St. Louis",5,Z7187,Tampa,New York,2024-05-22 534 | ,,,,,,,,U8505,Tampa,San Francisco,2024-10-14 535 | ,,,,,,,,J2212,Tampa,Houston,2024-04-15 536 | ,,,,,,,,A8309,Tampa,Tampa,2023-06-18 537 | ,,,,,,,,N6378,Tampa,St. Louis,2023-04-11 538 | 774,Isaiah Aguirre,Boston,25,"Santa Fe, Pittsburgh, Houston, Marseille, San Diego",5,"Santa Fe, Pittsburgh, Houston, Marseille, San Diego",5,F3822,Boston,Santa Fe,2024-01-05 539 | ,,,,,,,,D5528,Boston,Pittsburgh,2023-12-10 540 | ,,,,,,,,I3771,Boston,Houston,2024-08-29 541 | ,,,,,,,,E4866,Boston,Marseille,2024-08-02 542 | ,,,,,,,,E5946,Boston,San Diego,2023-02-04 543 | 122,Tom Nichols,London,61,"Milan, Athens, Munich",3,"Milan, Athens, Munich",3,I9803,London,Milan,2023-08-20 544 | ,,,,,,,,I5879,London,Athens,2023-05-19 545 | ,,,,,,,,H9596,London,Munich,2023-11-14 546 | 58,Jay White,Boston,70,"Sedona, Phoenix, Chicago, San Francisco",4,"Sedona, Phoenix, Chicago, San Francisco",4,L6106,Boston,Sedona,2023-01-28 547 | ,,,,,,,,E8246,Boston,Phoenix,2024-02-11 548 | ,,,,,,,,Y3638,Boston,Chicago,2024-07-03 549 | ,,,,,,,,G5778,Boston,San Francisco,2024-06-05 550 | 797,John Wong,St. Louis,44,"Denver, Kansas City, Barcelona, Dallas, Jacksonville",5,"Denver, Kansas City, Barcelona, Dallas, Jacksonville",5,Z3379,St. Louis,Denver,2023-09-17 551 | ,,,,,,,,D5201,St. Louis,Kansas City,2024-05-13 552 | ,,,,,,,,F3914,St. Louis,Barcelona,2024-08-30 553 | ,,,,,,,,F8407,St. Louis,Dallas,2022-11-26 554 | ,,,,,,,,D1587,St. Louis,Jacksonville,2024-06-27 555 | 584,Michael Davis,Madrid,44,Berlin,1,Berlin,1,Z1303,Madrid,Berlin,2023-10-11 556 | 988,Catherine Mata,Newport,20,Honolulu,1,Honolulu,1,Y6764,Newport,Honolulu,2024-03-28 557 | 948,Dennis Mckee,Brussels,67,"Bratislava, St. Petersburg",2,"Bratislava, St. Petersburg",2,X5646,Brussels,Bratislava,2024-10-22 558 | ,,,,,,,,H2887,Brussels,St. Petersburg,2023-01-20 559 | 286,Laurie Garza,Chicago,27,Atlanta,1,Atlanta,1,W5635,Chicago,Atlanta,2024-08-22 560 | 6,Lynn Gonzales,Stockholm,47,"Gothenburg, Valencia",2,"Gothenburg, Valencia",2,T8551,Stockholm,Gothenburg,2022-12-19 561 | ,,,,,,,,B6649,Stockholm,Valencia,2024-06-04 562 | 602,Rebecca Morgan,Florence,18,"Malaga, Gothenburg",2,"Malaga, Gothenburg",2,C7662,Florence,Malaga,2024-09-14 563 | ,,,,,,,,Z3371,Florence,Gothenburg,2023-01-09 564 | 619,Charles Frazier,Edinburgh,68,"Munich, Savannah, Naples, Florence",4,"Munich, Savannah, Naples, Florence",4,Z2553,Edinburgh,Munich,2023-09-12 565 | ,,,,,,,,J5408,Edinburgh,Savannah,2024-06-15 566 | ,,,,,,,,D9087,Edinburgh,Naples,2023-08-01 567 | ,,,,,,,,H3011,Edinburgh,Florence,2024-09-25 568 | 268,Kelly Griffith,Zurich,62,"Prague, Venice, Cologne, Brussels",4,"Prague, Venice, Cologne, Brussels",4,X8757,Zurich,Prague,2022-11-19 569 | ,,,,,,,,A5950,Zurich,Venice,2024-02-05 570 | ,,,,,,,,T5932,Zurich,Cologne,2023-12-07 571 | ,,,,,,,,N9163,Zurich,Brussels,2024-05-08 572 | 457,Robert Cortez,Berlin,32,"Milan, Budapest, Salzburg, Athens, Amsterdam",5,"Milan, Budapest, Salzburg, Athens, Amsterdam",5,L7663,Berlin,Milan,2023-07-07 573 | ,,,,,,,,L9762,Berlin,Budapest,2023-05-04 574 | ,,,,,,,,P9487,Berlin,Salzburg,2024-02-29 575 | ,,,,,,,,X2933,Berlin,Athens,2023-03-18 576 | ,,,,,,,,Q6517,Berlin,Amsterdam,2024-09-22 577 | 945,Tiffany Ford,Richmond,69,"Houston, Asheville, New Orleans, Minneapolis, New Orleans",5,"Houston, Asheville, New Orleans, Minneapolis, New Orleans",5,V8754,Richmond,Houston,2024-01-25 578 | ,,,,,,,,Y5551,Richmond,Asheville,2023-04-01 579 | ,,,,,,,,B2724,Richmond,New Orleans,2024-07-17 580 | ,,,,,,,,K7817,Richmond,Minneapolis,2024-06-20 581 | ,,,,,,,,F5254,Richmond,New Orleans,2024-03-01 582 | 472,David Salas,Venice,42,"Edinburgh, Florence, Madrid",3,"Edinburgh, Florence, Madrid",3,U6706,Venice,Edinburgh,2024-02-13 583 | ,,,,,,,,A6030,Venice,Florence,2024-03-13 584 | ,,,,,,,,M6601,Venice,Madrid,2024-09-24 585 | 862,Katie Novak,Helsinki,53,"Cologne, Valencia, Prague, Paris, St. Petersburg",5,"Cologne, Valencia, Prague, Paris, St. Petersburg",5,G9089,Helsinki,Cologne,2024-04-17 586 | ,,,,,,,,J2541,Helsinki,Valencia,2022-12-26 587 | ,,,,,,,,Z7033,Helsinki,Prague,2024-05-01 588 | ,,,,,,,,G7902,Helsinki,Paris,2024-02-15 589 | ,,,,,,,,J7465,Helsinki,St. Petersburg,2024-02-26 590 | -------------------------------------------------------------------------------- /06_agent_evaluation_with_ragas.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "51194982-9bd9-4b79-85a3-16cbdf36d63b", 6 | "metadata": {}, 7 | "source": [ 8 | "# Agent Evaluation\n", 9 | "\n", 10 | "In this section, we will explore the evaluation of agentic systems. Agentic systems are complex constructs consisting of multiple sub-components. In Lab 3, we examined a simple singleton agent orchestrating between two tools. Lab 4 demonstrated a more sophisticated multi-layer agentic system with a top-level router agent coordinating multiple agentic sub-systems.\n", 11 | "\n", 12 | "One direct implication of the interdependent and potentially nested nature of agentic systems is that evaluation can occur at both macro and micro levels. This means that either the entire system as a whole is being evaluated (macro view) or each individual sub-component is assessed (micro view). For nested systems, this applies to every level of abstraction.\n", 13 | "\n", 14 | "macro-micro-eval.png \n", 15 | "\n", 16 | "Typically, evaluation begins at the macro level. In many cases, positive evaluation results at the macro level indicate sufficient agent performance. If macro-level performance evaluation proves insufficient or yields poor results, micro-level evaluation can help decompose the performance metrics and attribute results to specific sub-components.\n", 17 | "\n", 18 | "In this lab, we will first explore macro-level agent performance evaluation. Then, we will evaluate tool usage as a lower-level task. To maintain focus on the evaluation process, we will keep the agentic system simple by utilizing the singleton agent composed in Lab 3." 19 | ] 20 | }, 21 | { 22 | "attachments": {}, 23 | "cell_type": "markdown", 24 | "id": "64169599-95e0-4b76-a81e-c68d479eb79a", 25 | "metadata": {}, 26 | "source": [ 27 | "## Agent Setup\n", 28 | "\n", 29 | "As covered in the previous section, we will reuse the Agent we built in Lab 3. This agent has access to tools designed to help find vacation destinations. You'll be able to interact with the agent by asking questions, observe it utilizing various tools, and engage in meaningful conversations.\n", 30 | "\n", 31 | "Let's begin by installing the required packages." 32 | ] 33 | }, 34 | { 35 | "cell_type": "code", 36 | "execution_count": null, 37 | "id": "b861b440-3598-4140-af69-6bd993f861e8", 38 | "metadata": {}, 39 | "outputs": [], 40 | "source": [ 41 | "#%pip install -U langchain-community langgraph langchain-chroma langchain_aws pandas ragas==0.2.6 faiss-cpu==1.8.0 pypdf rapidfuzz\n", 42 | "%pip install -U pandas ragas==0.2.6 faiss-cpu==1.8.0 pypdf -q" 43 | ] 44 | }, 45 | { 46 | "cell_type": "code", 47 | "execution_count": null, 48 | "id": "38a2b5d5", 49 | "metadata": {}, 50 | "outputs": [], 51 | "source": [ 52 | "!pip install rapidfuzz --force-reinstall -q" 53 | ] 54 | }, 55 | { 56 | "cell_type": "markdown", 57 | "id": "a910fdd3-006b-42b1-88f3-9fee1bc80707", 58 | "metadata": {}, 59 | "source": [ 60 | "### Util functions part 1 - importing singleton agent\n", 61 | "\n", 62 | "To maintain a clean and focused approach in this notebook, we have moved the agent creation logic to a module in `utils.py`. The `create_agent` function replicates the agent creation process of the simple ReAct agent we developed in Lab 3." 63 | ] 64 | }, 65 | { 66 | "cell_type": "code", 67 | "execution_count": 3, 68 | "id": "783694df-e31d-448a-af7b-0c3fc6747c27", 69 | "metadata": {}, 70 | "outputs": [], 71 | "source": [ 72 | "from utils import create_agent\n", 73 | "agent_executor = create_agent()" 74 | ] 75 | }, 76 | { 77 | "cell_type": "markdown", 78 | "id": "e3776a1d-7566-4652-acd2-7eed21f83d59", 79 | "metadata": {}, 80 | "source": [ 81 | "The ```create_agent``` function returns a ```CompiledStateGraph``` object that represents the Agent from Lab 3's scenario. \n", 82 | "Now, let's proceed to visualize this graph." 83 | ] 84 | }, 85 | { 86 | "cell_type": "code", 87 | "execution_count": 4, 88 | "id": "f21d0ae3-13ca-4bdc-bcc8-087eece6b610", 89 | "metadata": {}, 90 | "outputs": [ 91 | { 92 | "data": { 93 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAANYAAAD5CAIAAADUe1yaAAAAAXNSR0IArs4c6QAAIABJREFUeJztnXdcU+fi/5+ThAwyIAmEKUuWKC5wo9i6rjgKalXQWq3eqtdxW2cH91Zr9Tpar7Xf3tpW6657FeveSsVVqSKIbGQkhAQSErJzfn/EH6UYUDEnz0nyvF/+gSfJ83yCb59zznOegeE4DhAIeFBgB0C4OkhBBGSQggjIIAURkEEKIiCDFERAhgY7QHtQyg1KmaFRaVI3GI16x+hWorlhVBrmzqW682hCPzrTnQo7EVnAHOMfEAAAgLRSW/SHuuSRms2jmYy4O4/K5tLoLApwhG9AY2CqOmNjg6lRaVQrTGwPamgXdkR3DofvBjsaZBxDQYXM8NsvtVQ3jC+ih3ZmewUwYCd6XSqLNCU5arlY5+lN7z9GSHNz3SsiB1Dw1mlZ/t2G/mO9wrtxYGexPX9cq/8tQzYwxatLfw/YWeBAdgUPf13RZQAvOp4HOwix3D4rb5AbhqT6wA4CAfIqiOP4Dx8Xj53t7xfKgp3FHuTeUpY+Uie95wc7iL0hr4LfLSuclh7C5jnkPXv7eHxHmfObcsI/A2EHsSskVfDwpooByUK/EJdo/5rzMFMhq9INflsEO4j9IOONWNYpWexAngv6BwCIHeDhzqXm3VbCDmI/SKdgXY2+MFsVFefk9x9t0HMI/8ohKewU9oN0Cv6WIes/Rgg7BUxobpS4ofxbp2Wwg9gJcikoLtUyWJSwWCfs/3sleo8QiEu1Br0ZdhB7QC4Fix6oBL50u1WXk5Oj0+lgfbxtmGxqSY6aoMJJBbkULHmkDu3Mtk9dGRkZ06dP12g0UD7+QkK7sJGC9qauRs8T0Pg+dmoF292AWbqxiGv/LITFshUyA6FVkAQSKaioNWAYRkTJZWVlc+bMSUhISEpKWrNmjdlszsjIWLt2LQBg6NCh8fHxGRkZAIDs7Oz58+cnJCQkJCTMnj07Ly/P8vH6+vr4+Pjdu3enp6cnJCT8/e9/t/px20Jzo6jqjWqF0eYlkw0SPXtoVJrceYSMolu1alVpaenixYvVavXdu3cpFMqAAQOmTp26Z8+eTZs2cTicoKAgAEBVVZVOp5s1axaFQjl06NDChQszMjKYTKalkG3btr399ttbtmyhUqk+Pj7Pf9zmsHk0tdLI9iDRvxERkOjrqZVGgh7HVVVVRUdHp6SkAACmTp0KABAIBIGBgQCALl26eHp6Wt42cuTIpKQky88xMTFz5szJzs7u27ev5UhsbOy8efOaynz+4zaH7UFVK0ygA0HFkwUSKQgATmMQciJOSkrasWPH+vXrZ82aJRAIWnsbhmGXL1/es2dPSUmJu7s7AEAm+7Nzrnfv3kRkawMGk4qbyfj41LaQ6FqQxaY1yAm59Jk3b96iRYvOnTs3duzYgwcPtva2rVu3Ll26NCYmZuPGjR988AEAwGz+s2eOxbL3A8P6Wr27C4zSIJGC7jxqo9JERMkYhqWlpZ04cSIxMXH9+vXZ2dlNLzWN0tDpdNu3b09OTl68eHH37t1jY2NfpmRCB3kQd3FMKkikIFfg5kbMidjSgcJms+fMmQMAePz4cVOrJpU+exqr0Wh0Ol2nTp0sf62vr2/RCragxceJgCugcT2dvxUk0Tf0DmBUFmpU9UaOrX/vy5cv53A4ffv2vXHjBgDA4lm3bt2oVOqXX345duxYnU43fvz48PDw/fv3C4VClUr1ww8/UCiUwsLC1sp8/uO2zVyaq3ajUzAKIf8nSQV1xYoVsDP8Sb3UYNCaRUFM2xZbUVFx48aNM2fOaDSaBQsWDB48GADA4/F8fHzOnz9//fp1pVI5evTonj17ZmZmHjx4sKysbMGCBcHBwUeOHJkyZYrBYNi1a1dCQkJMTExTmc9/3LaZ71+uDwhniTrY+FdBQsg1ZLX8sbo4Rz14ggsN2GyNjB+q3pjozfF0/imeJDoRAwCCotm3TsvFZVrfYOv/++vr65OTk62+FBgYWFFR8fzxxMTElStX2jppS2bNmmX1rN2pU6empyzNiYuL++qrr1orLec3BceT5gr+ka4VBABUFmpunZGNm299/oTJZJJIJFZfwjDr34XFYvH5fFvHbIlUKjUYrDzSbS0Vg8EQClsdFvnDx8Xv/juYwXL+22EyKggAuHywJqIHJzDCHXYQODzMVOi15rghhP+3IQkk6pRp4o2JojM7xRoVIX2EJKc8v7H4gcp1/COpggCA1GVBP68rh53C3jTUGc7vkbw1NwB2ELtCxhOxBZ3GtHdt+ZSPglzkkkhSpj23RzLl4yCKC/QFNoe8ClpahX3rn46d7efr7BM68+8p/7immPihs4+KsQapFbRwcZ9EozYNGONltwHV9qSioDEzQxYYzhow1gt2Fjg4gIIAgJIcdWZGbVgs2yeIGdqF7QSnKq3aVPJIXV2iVdQaBowR2vyBkAPhGApaKLjfUHBfVZKj7tSHR6NjbB6N7UFlMKkO8QWoVEytNDYqjSqFUSk3Ssq0oZ3ZkXHcoCgX7XtqwpEUbKI0T62oMaiVRrXCZDSazTbtvTEYDLm5ud26dbNloQCwOFTcjLvzaBwPmtCP7t/Rya9uXx6HVJBQZDJZamrquXPnYAdxFUjaL4hwHZCCCMggBVuCYVhkZCTsFC4EUrAlOI4/efIEdgoXAinYEgzDPDxcdPF7KCAFW4LjuEKhgJ3ChUAKWsHHxxU3X4AFUtAKrQ3MRhABUrAlGIY1nymHIBqkYEtwHM/NzYWdwoVACrYEwzD7Lx/jyiAFW4LjOHHL9yKeBymIgAxSsCXodsTOIAVbgm5H7AxSEAEZpGBLMAyzwwIgiCaQgi3Bcbyurg52ChcCKdgSNF7QziAFW4LGC9oZpCACMkjBlqAhq3YGKdgSNGTVziAFEZBBCiIggxS0QtMGOAg7gBS0gtU18hEEgRREQAYpiIAMUrAlqF/QziAFW4L6Be0MUhABGaRgSzAMCw4Ohp3ChUAKtgTH8bKyMtgpXAikIAIySMGWYBhGpbrEfk8kASnYEhzHTSZX3IERFkjBlqB5xHYGKdgSNI/YziAFW4KmL9kZtPXNM2bOnCkWi6lUqslkkkqlPj4+GIYZjcZTp07BjubkoFbwGRMnTmxoaKiqqpJIJGazubq6uqqqCsMcfr9F8oMUfMaIESPCwsKaH8FxPC4uDl4iVwEp+Cepqanu7n/ui+nr65uWlgY1kUuAFPyTESNGND0dtjSB0dHRsEM5P0jBvzBt2jQ2m21pAlNTU2HHcQmQgn9h2LBhwcHBOI736NEDTWKyDzTYAdqD2YTXSw0KmYGIDqXk4bNB4/G/DXq3OEdt88KpVMAX0XlCN5uX7Lg4Xr/g4zvKnJtKrcrkG8pqVDrYw1wOn1b+WM33dus1XIA2ZrfgYArm3VIW/qEe9LYvheLAPXY6renczsqhqSJRBybsLPBxpGvBgvsNT7LVgyf5ObR/AAAGkzpmdtCZnZK6Gj3sLPBxGAVxHH9wQzHgLRHsIDaj31jRnXNoOVfHUVCjMtXVGBgs5xlM6iF0e5rfCDsFfBxGQaXc6GRXTiwOjcWmGvVm2EEg4zAKYgBoGoywU9gYhcyARkI4jIIIZwUpiIAMUhABGaQgAjJIQQRkkIIIyCAFEZBBCiIggxREQAYpiIAMUhABGaSgDRCLq6vFVbBTOCpIwdelsqoiberY/Hy0ElI7QQoCHMcrqyra/XGT0ehYkx/IhkPOoHtJHj7M3r1n68OcbABAdFTnOXM+iIp8Ni8zNy/n2/99VVxcIBR4hYR2LCzM37XjKJ1O12q1W7d9e/HSGb1e1yEweOLEd958YzgA4PCRny9dPvf2hCnbtn0rk9dGREQvWZQeFBRSLa56d8YEAMDKzz9aCcCIEaM/WrYC9vd2MJy5FRSLq3R63TtTZ7077X2xuOqjjxdqtVoAgEQiXrJ0Lo1G+/TjL3r06JWZeXXsmAl0Ot1sNn+a/uHNm9empM348INPwsOjVn3xyanTJyyl5eXlHDy4e/Hi9M9Xfimtkfxn3WcAAKHA69NPvgAAzJg+Z/OmrVPT3oP9pR0PZ24Fhw4dOWxYkuXnqKiYRYvnPMzJ7hXf9/yFUxqN5rN/rRUIhAMGJP7x4PesWzfSUqdfu37pwcP7+/ZmeHl5AwCGDvmbRtN45Oi+pJFvWQpZ/cV/BQIhAGDcuMn/++6/CqXCg+cRGRENAAgKComN7Q716zoqzqwghmHXb1w+eGhPWVmJZb2iOrkMACCVSthstkUmDMP8/QMlkmoAQFbWDaPRmDZ1bFMJJpOJzeY0/ZXJfDbz18fHDwAgq5V68NBWYa+LMyu4a/fW7Tu2jB+X+v6sBTJ57crPPzLjZgBAQEAHtVpdXFwYFhZuMBgKC/O7d48HANTVyYRCr41fbmleCJVm5VfkRnMDAJjMDjaRnpw4rYIGg+HnfdtHJSXPn7cYAFBTI2l6acTw0YcO7/0k/YPhw0Zl/3HPaDROn/Y+AIDL5dXX1/n4+DEYDKjZXQunvR3R6/U6nS7y/98CK5T1AACz2QwA8PDwnD9vCYPBLCkpio/r++P3PwcGBgEAevbsbTKZfsk43FSIRqN5YUUMBtNyUiby2zgzTtsKstnssLDwo8f2CwRCtUq1c9cPFAqluLgQAJD3+NH6DSsXzl9Gc3OjUCjV1ZUCgZBKpQ4bmpRx8uiW77+uFldFRkQXFj65kXl5x0+Hmcy2Jo+KRD7+fgEHD+9hslhKpWLSxHcoFKf9j00ETqsgAOBfn65Zt37F56s+DgwMmjv3w6KiJ0eO7Jv9/kJfHz8/v4B1G1Y2dSlHhEdt/nobk8ncsO7bH7d+c+nS2ZMnjwYGBo0dM4Fm7VqwORiGpaevWb9h5f99+6VI5JuSPKltZREtcJhljSRl2iuHpUmzOtikNJPJZNnly2QyXb9xeeXnH3315Xc9e/SySeEvz54vit5fE0Z1c+mpxM7cCrZGeXnpPz/8e7++A8M7Rur0umvXLjKZzMCAINi5XBRXVJDN5gx5829ZWdfPXzjF4XBju3T/4IOPRSIf2LlcFFdUUCj0mj9vsaWzBgEddO+GgAxSEAEZpCACMkhBBGSQggjIIAURkEEKIiCDFERABimIgAxSEAEZh1GQSgNcgbPtHugdyKBQXXqYjCMpKPRnFD9QwU5hS+QSnV5rxhzmX4AoHOYXgGFYZBxXXOo82xVJy7UR3Tkv8UYnx2EUBAAMmSy6dkSiVTvDvLXS3Ibih8peIwSwg8DHYUZNW9BpTLtXl3V/Q8jxdOOL6A6VHQAAcADk1doGuaEsTzXxw8A7d+707t0bdijIOJiCFk7/nF/6uMHXx09Ra7B54TiOa7VaFouQ/aq9AhgAgKAoVteBngCAvLy8JUuWHD161KWnjeIOyIIFC4grfNOmTQkJCb/88gtxVTSnurr66dOnMpnMPtWREEe6FgQAXLp0CQCwefNmgsqvrq6+fv26RqM5ePAgQVW0wNfXNzAwEMOwSZMmqVROdcv/kjiSgpMmTQoICCC0ikOHDpWWlgIAysvLT548SWhdzeHz+atXrz579qzdaiQPjqGgWCzWaDSrV6+OiooirpbKysqrV69aflar1QcOHCCurucJDw8fP348AGDBggU6nc6eVcPFARQ8dOhQVlYWi8UKDw8ntKJjx46VlZU1/bWsrOzEiROE1miVmTNn/vTTT/avFxYOoGBZWVlycjLRtVRVVV2+fLn5EbVavXfvXqLrfZ7u3bvPnTsXAPDNN9/Yv3b7Q2oFb968CQBYsmSJHerav3+/pQm0LH1keR7z9OlTO1TdGv379+/Xr58j9pq9GrBvya2j1Wp79erV0NBg/6plMtmkSZPsX69VdDqdyWR68OAB7CAEQsZWUC6Xl5WV3bx5k8OB8AgVx3G5XG7/eq1Cp9MpFIq7u/uECROMRiPsOIRAOgW3bt0ql8sjIyMtyw4hAAAdO3bcsGFDSUlJQ0MD7Cy2h1wKFhQUGAwGou982wbDMBI+LgsNDY2IiNBoNCtWONumEiRSUCwW8/l8y80gRCxXYHAztIZIJIqLi7NzhyXRkEXBpKQkPp/v5eUFOwjAMCwmJgZ2ilYZM2bMqFGjAABNveiODnwFTSbT6dOnt2/fTpLTn8lkqqmpgZ2iLSx3abdu3Tp27BjsLDYAsoKlpaUSiWTkyJE+PmRZ3k+v1zvEcIFly5YJBM4w4hWmgg0NDYsXL/b394eY4Xn0ej2hT6JtSGJiIgBg0aJFdXV1sLO0H5gKFhQUHDlyBGIAq0gkEsdar3zNmjWrVq2CnaL9wFFQLBYfO3asZ8+eUGpvm4KCAqFQCDvFK8BkMjdu3AgAuHPnDuws7QGCgrm5uUuXLk1JSbF/1S+DTCbr2rUr7BTtoby83BH7ayDMHWnacIGcJCYm/vrrr1CeDb4+u3btmjZtGuwUr4ZdW0Gj0bhr1y4y+3f37t2BAwc6qH8AgGnTptXW1lZUtH+TeftjVwUnTpw4fPhwe9b4quzfv3/IkCGwU7wWXl5eV69etVwdOgQOOYmTIKqrq5cvX75r1y7YQWyAUqnEcdzDwwG2S7ZTK1hRUfH48WP71NVuvvnmmylTpsBOYRt4PF5lZaVDnJHtoaDJZBo3blx0dLQd6mo3jx8/1mq1I0aMgB3EZsTExCxatKioqAh2kBdgjxNxdnY2n88PDg4muqLXISUl5euvvw4Kcqqd6IxGY1ZWVkJCAuwgbYGuBQEAYN++fQCA1NRU2EFsj06nMxgMZL7HJ/xEfODAAZJf4N+5c+fq1atO6R8AgMFgvP/++/n5+bCDtArhCp48eTI+Pp7oWtqN2WxeuXLlli1bYAchkDVr1mRlZcFO0SrEnohxHFer1WQ+C0yePHnVqlURERGwg7guxLaCGIaR2b9PPvlkxowZruDfkydPrly5AjuFdYhV8NatWwsXLiS0inazf//+Ll26OFMvTBt06NAhPT0ddgrrEKsghULR6/WEVtE+jh8/XlBQkJaWBjuInWCxWFu2bCHnyFZirwX1er1SqSTDpKTmZGZmHjhwgLhFChGvBLGtIJ1OJ5t/jx492rZtmwv6l52dvXv3btgprEB4p0xycrJMJiO6lpekpKTks88+c6ml05qgUCiWNWrJBuEK9uzZkySPKWtqajZv3nz48GHYQeDQqVMn+6xR9qq4ygO62traKVOmuOZKuiQH/lR2O1BeXj558mQX90+v1y9evBh2CisQrqBMJhszZgzRtbSBVCpNT0+/cOECxAxkAMfx7Oxs2CmsQCO6AqFQ6OvrW1dXx+fzia7reaRS6dSpU128/bNAp9PXrVsHO4UV7HQt+NZbb6nVaqVSKRKJ7LaZQnl5+aZNmxxoFoVrQmArOGjQoMbGRsspAMMwyw92W7SqqKhoyZIlzrHwj00wGo0bN25ctmwZ7CAtIfBa8M0336RQKJbBCpYjVCq1T58+xNXYRE5Ozo8//oj8a47ZbCbnL4RABVesWBETE9P8RC8Sibp160ZcjRays7M3bNiwdu1aoityLGg0miveEa9bty4kJMTyM47jXC6X6EV8r1+/fvLkyZ07dxJaiyNCoVAmTJgAO4UViFXQx8fnww8/tDwmxjCM6Cbw7NmzR44cIe2oJLgYjUZyDpwjvF8wISFh3LhxbDabw+EQeiF4/Pjxq1evbtq0ibgqHBqz2UzOpbde6o7YaDBrVOZ215H69ntlRTUFBQVhQZ0b6gjZPOPy5cuPHhavWbOGiMKdAyqVSs6J+i/oF8y7rXxwXSEX61mc11qLqKlfhiD0er0ogFNV1BjWldNrGF/oT4plq8nA0qVLL1682NQpZrkiwnH8999/hx3tGW21grfPyWurDAPH+XIFbnaM1H7MJrxeqj+1Qzw0zccvxJFWSiWOuXPn5ubmSiSS5r1jTfeIZKDVa8FbZ+QKqXFgio+j+AcAoFAxgS8jeV7wxX01knIt7DikICwsLC4urvm5DsOwQYMGQQ31F6wrWFejr63U9R0tsnse2/Bmqt/dc2ScJwGFadOmNd/QIDAwcPLkyVAT/QXrCtZW6nCcwEs3ouHy3Z4WNOp17b+FcibCw8N79+5t+RnH8YEDB5Jni41WFVQpTN4dHPtaKjiGLa8m6T5e9uedd94RiUQAgICAALLdF1tX0KAzG7SO3YQoZUYAHLghty0dO3bs06cPjuOJiYmkagLtMV4Q0Q7MZrz8caOqzqhWGo0GXKM2vX6Z3fynantERAkGXNgnef3SmCwqnUVx51F5fLegaPfXKQopSC7ybivz76kqChr9I3lGPU51o1LcaACzRacEhdm73yiDGRgabVBYgwo3GYwmo8HNTffL91XBMezIHpyoeG47ikIKkoXcW8obJ2q9g7g0NrfLMHKdK9uGHyxoqGl8dE+bmSEbmCyM6PFqIiIF4aNRmU5tlxhMlLA+gTQ6eXfEaA0Mw3g+bADYHG/e3UvyvDuqUTN9qdSXvRB3iRl0ZKY8X71rdRknQOAb5e2I/jWHzqL5xYjofM8ty4pqnr7sowGkIEwkT7VXj8qjBgUzWA7zCOqFMDn0zkNDT22XKGUvtaIVUhAaJY9U5/ZIO3Qn1164tiKkV+DR/4nFZS9uC5GCcFDVGy/uc1r/LITEBxz9ptJoeEEHM1IQDmd2SUJ6B8BOQTgd+/r/+tMLuiGRghC4e77OBOg0N8e++XgZGGy6Wo09uqlo4z1IQQhknZKJwiGsLQEFUZggM0PexhtsqWBuXo5O91ojA65cvfDGkPjy8lLbhSId9y7IA2IEhI4hbzefrx99+ISNJ7/SGFRhEDfnt1YbQpspeOZsxrz507Vaja0KdFby7qiYHo49CulVYXCYj++qWnvVZgq+ZvvnIijlBq3azOK61tQWjpAlfao1tDJ80zYP6M6czdj09VoAQPK4oQCA5cs++9uIMQCAc+d+3btve1VVhVDoNSopZUraDMsSH0ajcfuOLWfPnVQo6oODQ6e/OzthwODni83KuvHD1m+qqip8ff3HjpkwLmWSTdJC5Gl+Iz+QqI1YCovvnTr/vyrxEy5HEB4aP3LYXB7XCwCQvnrI+DHLc/Ku5OZnspicvr1Shr8xy/IRk8l04cq2rLvH9XpNx7A4g4Go2Q5eIdyyvMbw7la+u21awT69B0x8eyoA4D+rN23etLVP7wEAgLNnT/5n3WcREdH/Sl8zOHHYT9u/2/vzdsv7v/zqiwMHd48elfLpJ1/4+vr/699LHjy436LMxsbGFZ8vp7vRFy9K799vkEwmtUlUuNRWG3CckFvAgqI7P+5a6CMKnZj86aD+acWl97dsn6fXP1Nq/9GV/r6R/5i5pWe3kecu/Zibn2k5fuzkhvNXtkVH9k8ZvYTuxtRoG4jIBgAwmbA6qfWHJbZpBfl8gb9/IACgU6cuHh6elgHiW3/6Nja2e/onXwAABg18s6FBuf/AzvHjUmtra86eOzntnVnT350NAEgcNGTqtJQdO7/f+NVfNoKrq5frdLqBA98cNnSkTUKSAbXCSGOwiCj5+K9f9Y1PSRn9bDXpyPA+GzZPyi/Mio0ZDADo3XPskMTpAAB/38jb9048KcyKiRpQUfU46+6xIYkzRg6dAwCI7zGqqISomZ1uDJqqlSnkRI2Uqagor62VTpr4TtORXr36nTp9oqKyPD8/FwCQkPCG5TiGYb3i+56/cKpFCf5+AZ07d92zdxuTyRozehydTicoqj3RqEwMvu27A+V11RJpSa38adbd482P1yuedQvT6c+8p1KpHjyRQikFADzMvQIAGNT/zy1IMYyoTjoag9KotK+CKrUKAODpKWg6wuXyAAC10hq1WgUA4Dd7icfzaGxsVKvVzUvAMGztms1bt/3flu83HTq85+Pln3fr1pOgtHaDoPVEG1QyAMCwN2Z1jXmj+XEu18qmLxQKzWw2AQDq68VMJoft7kFIphbgmLmV725j65vmq4q8fQAACkV900t1dXKLiF5eIgCAUvlnR5FcLqPRaExmy64KDofzwT8/2rnjCJvNSf/XIsuCmQ4N24Nq1NlgFH4LWEwuAMBg0Im8Q5r/YTHbuvVhs/larcpgtMcObUadkcu33t7ZTEEWkwUAqK19dtMgFHr5+vjdvp3Z9IarVy8wmczw8KhOnbpgGJZ164bluF6vz7p1o3PnrlQqle5Gb26npaPH3y9gXMpklVolFlfZKi0suB40o972Cnp7BXl6+N75PUOnf9YvazIZjUZD258KDIgGANx/YI+FuI16E9fTuoLUFStWPH+0skhjMgLfkFe4cGay3E/8cqi0rBgDWG7ew6ioGC6Hd+DQHqlUYjAYjh7bf+Hi6Slp7/WK78vj8sTi6mPHDwCA1dZKv/vuvyWlRUuX/NvPL4Dm5nbs+IHH+Y+CgkK8hN7Tpo+rrZXKZLXHjh/Q63Qz3/sHjfayVw4F95Uhndw5rXxtWKgUBpnYyPK08R0JhmF8T7/b937JfXwdB3jZ04fHTn5lMumDO8QCAC5d3xXoHx0V/mxZs6w7x5lMdo+uw0VeoQ8eXbx3/5RGq1Kp627eOVZUcjfQv1NMdIJt4wEAtAp1aAxT4GPlgt5mCvK4PG9vnytXzt+8eb2hQTlixOjw8Eg+X3Dp8rnTZ36pr5Onpc2YOuU9y4OpXvH91GrV6TMnLl06y3ZnL1mc3qtXPwAAl8P18/X//f4dCkbpFBNbUVF+I/Py9RuXhELvj5atCAgIfPk85FTQnUe7/WutMNj2l18+3iGBATHFpdn3sk+VVzzy8wuP6z7S0i/YmoIUCqVTZIK0tuzBo4vFpdm+ojB5XZWPdygRCpbckwyd4kOhWHksaX1lrdtn5Xot6DZY8PxLjsKpbRWJ47x8ybe40c/rn3oGCd09XOgBSUNto1HZkDLP+uBIcjUSrkBMX07hI00bCj4pvL3rwMfPH2cxua11HY8esaBvfLKtEublZ+49/O/nj+M4DgButeNmzoxvA/2jWytQp9J17s38jtjIAAAClElEQVRu7VWkoL3pPoh/82QRP5BHpVm/FwwJ6rroH1Z2bcVx0NrwGneWLc/sHUPjrAYwm804jlOpVvo1eVzv1krTawxKsapTr1aXk0MKQmDAGGHuPblvlPWdmul0poAOc0C/bQPUFtcNTBa28QY0ZBUCXQd6spgmneYFnSZOgLZB5ynE2p7cjhSEw8gZvsVZlbBTEIvZjBffrkqa4dv225CCcKAzKMlz/UtuO7OFxVkVqcuCXvg2pCA0/EJZ4+b7ltyugB3E9piM5oLM8rTlgXzRiweXIAVh4iGkj5nlm3OuRKN0npWx1XXaghvlkxYFunNe6mYXKQgZrwDGvI0dzSplZY5Ep7bHiAHi0Ch1T/+odjOr5qzryHvpVfJRpwx8MAwbNdOvJEd97ViNuyeT5s7gebtTHWeWsVFnUkrVJp3eoNYNHufVIfLVVrxECpKF0C7s0C7sooeqgvvqwky5INDdoDNT6TQag0bCFYtxHDfpjCaD0Y1OqRNrQruwIwZwQmLasywiUpBcdIzldIzlAACqSzRqhUmtMOp1Zq0tFvq1LQx3CtOd7s5z5/KpPkEv6HZpG6QgSfELJWSKCQmxriCdiZnJ1/i/Eh7eboRNhEDYEuv/Sly+m7TMsddFKHmgEvo5w4wnp8e6gqIODFKuefKy1Ev1IZ3daW6oGXQAWm0FA8KZ146I7Z7HNlzcW9U3qa3RGQjy0NZ+xI9uKgqyVd0ShXwfemuD20iFRmVU1BquHRaPXxDg+RKPhhBk4AVbYpc8UmdfrReXaKk0sp+YBX4MhVQf1sW990ghm4fu9B2GFyjYhE5D9i3pcBww3R2gqUa04GUVRCAIAjUbCMggBRGQQQoiIIMUREAGKYiADFIQAZn/B1qlvCqU0zzIAAAAAElFTkSuQmCC", 94 | "text/plain": [ 95 | "" 96 | ] 97 | }, 98 | "metadata": {}, 99 | "output_type": "display_data" 100 | } 101 | ], 102 | "source": [ 103 | "from IPython.display import Image, display\n", 104 | "\n", 105 | "display(Image(agent_executor.get_graph().draw_mermaid_png()))" 106 | ] 107 | }, 108 | { 109 | "cell_type": "markdown", 110 | "id": "dd2275d7-cdc2-439d-a78e-5b0c08b49a12", 111 | "metadata": {}, 112 | "source": [ 113 | "Now, we are ready to proceed with evaluating our agent!" 114 | ] 115 | }, 116 | { 117 | "cell_type": "markdown", 118 | "id": "9aa84e32-ac33-411d-a075-a32258ed3926", 119 | "metadata": {}, 120 | "source": [ 121 | "## Agent Evaluation with ragas library \n", 122 | "\n", 123 | "In this section, we will explore sophisticated methods for evaluating agentic systems using the ragas library. Building upon our previous work with the vacation destination agent from Lab 3, we'll implement both high-level (macro) and low-level (micro) evaluation approaches.\n", 124 | "\n", 125 | "ragas provides specialized tools for evaluating Large Language Model (LLM) applications, with particular emphasis on agentic systems. We'll focus on two key evaluation dimensions:\n", 126 | "\n", 127 | "1. [High-Level Agent Accuracy](https://docs.ragas.io/en/latest/concepts/metrics/available_metrics/agents/#agent-goal-accuracy):\n", 128 | " - Agent Goal Accuracy (with reference): Measures how well the agent achieves specified goals by comparing outcomes against annotated reference responses\n", 129 | " - Agent Goal Accuracy (without reference): Evaluates goal achievement by inferring desired outcomes from user interactions\n", 130 | "\n", 131 | "2. [Low-Level Tool Usage](https://docs.ragas.io/en/latest/concepts/metrics/available_metrics/agents/#tool-call-accuracy):\n", 132 | " - Tool Call Accuracy: Assesses the agent's ability to identify and utilize appropriate tools by comparing actual tool calls against reference tool calls\n", 133 | " - The metric ranges from 0 to 1, with higher values indicating better performance\n", 134 | "\n", 135 | "This structured evaluation approach allows us to comprehensively assess our vacation destination agent's performance, both at the system level and the component level. By maintaining our focus on the singleton agent from Lab 3, we can clearly demonstrate these evaluation techniques without the added complexity of nested agent systems.\n", 136 | "\n", 137 | "Let's proceed with implementing these evaluation methods to analyze our agent's effectiveness in handling vacation-related queries and tool interactions." 138 | ] 139 | }, 140 | { 141 | "cell_type": "markdown", 142 | "id": "daf00b75-1051-47ec-80d1-f3d65ee8fdc6", 143 | "metadata": {}, 144 | "source": [ 145 | "### Util functions part 2 - Message Format Conversion\n", 146 | "\n", 147 | "Our singleton agent is built using the LangChain/LangGraph framework. LangChain defines several [message objects](https://python.langchain.com/v0.1/docs/modules/model_io/chat/message_types/) to handle different types of communication within an agentic system. According to the LangChain documentation, these include:\n", 148 | "\n", 149 | "- HumanMessage: This represents a message from the user. Generally consists only of content.\n", 150 | "- AIMessage: This represents a message from the model. This may have additional_kwargs in it - for example tool_calls if using Amazon Bedrock tool calling.\n", 151 | "- ToolMessage: This represents the result of a tool call. In addition to role and content, this message has a `tool_call_id` parameter which conveys the id of the call to the tool that was called to produce this result.\n", 152 | "\n", 153 | "Similarly, the ragas library implements its own message wrapper objects:\n", 154 | "\n", 155 | "- [HumanMessage](https://docs.ragas.io/en/latest/references/evaluation_schema/?h=aimessage#ragas.messages.HumanMessage): Represents a message from a human user.\n", 156 | "- [AIMessage](https://docs.ragas.io/en/latest/references/evaluation_schema/?h=aimessage#ragas.messages.AIMessage): Represents a message from an AI.\n", 157 | "- [ToolMessage](https://docs.ragas.io/en/latest/references/evaluation_schema/?h=aimessage#ragas.messages.ToolMessage): Represents a message from a tool.\n", 158 | "- [ToolCall](https://docs.ragas.io/en/latest/references/evaluation_schema/?h=aimessage#ragas.messages.ToolCall): Represents a tool invocation with name and arguments (typically contained within an `AIMessage` when tool calling is used)\n", 159 | "\n", 160 | "To evaluate the conversation flow generated by the LangGraph agent, we need to convert between these two message type systems. For convenience, we've implemented the `convert_message_langchian_to_ragas` function in the `utils.py` module. This function handles the conversion process seamlessly. You can import it along with the message wrapper objects, which are assigned appropriate aliases to ensure cross-compatibility between the frameworks." 161 | ] 162 | }, 163 | { 164 | "cell_type": "code", 165 | "execution_count": 5, 166 | "id": "606a9191-9677-4174-8d15-d20efaaeea1c", 167 | "metadata": {}, 168 | "outputs": [], 169 | "source": [ 170 | "from utils import create_agent, convert_message_langchain_to_ragas\n", 171 | "\n", 172 | "from langchain_core.messages import HumanMessage as LCHumanMessage\n", 173 | "from langchain_core.messages import AIMessage as LCAIMessage\n", 174 | "from langchain_core.messages import ToolMessage as LCToolMessage\n", 175 | "\n", 176 | "\n", 177 | "from ragas.messages import Message as RGMessage\n", 178 | "from ragas.messages import HumanMessage as RGHumanMessage\n", 179 | "from ragas.messages import AIMessage as RGAIMessage\n", 180 | "from ragas.messages import ToolMessage as RGToolMessage\n", 181 | "from ragas.messages import ToolCall as RGToolCall" 182 | ] 183 | }, 184 | { 185 | "cell_type": "markdown", 186 | "id": "16c467d0-c358-4baa-ab70-a00cf6e01e8f", 187 | "metadata": {}, 188 | "source": [ 189 | "Since we typically evaluate multi-turn conversations, we will implement a helper function capable of handling arrays of messages. This function will enable us to process and analyze multiple conversational exchanges seamlessly." 190 | ] 191 | }, 192 | { 193 | "cell_type": "code", 194 | "execution_count": 6, 195 | "id": "304cd23e-1298-466b-8f8a-71776c40b414", 196 | "metadata": {}, 197 | "outputs": [], 198 | "source": [ 199 | "def convert_messages(response):\n", 200 | " return list(map((lambda m: convert_message_langchain_to_ragas(m)), response['messages']))" 201 | ] 202 | }, 203 | { 204 | "cell_type": "markdown", 205 | "id": "39e97c81-3dbb-40fc-900d-d1fd1cd5fa7f", 206 | "metadata": {}, 207 | "source": [ 208 | "For the evaluation, we will examine two distinct scenarios that represent different user profiles:\n", 209 | "\n", 210 | "1. **Andrew Macdonald - The User with Travel History**\n", 211 | " * A 62-year-old resident of Paris\n", 212 | " * Exists in our travel history database and is logged in with user_id 918\n", 213 | " * Previous travel records enable precise, personalized recommendations\n", 214 | " * Expected to have a smooth, seamless conversation flow\n", 215 | "\n", 216 | "2. **Jane Doe - The First-Time User**\n", 217 | " * No prior interaction with our travel recommendation system\n", 218 | " * Requires the agent to rely on creative recommendation strategies\n", 219 | " * Will test the system's ability to gather information and provide relevant suggestions\n", 220 | " * May experience a slightly more exploratory conversation flow\n", 221 | "\n", 222 | "These scenarios will help us evaluate our travel agent's performance across different user types and interaction patterns. Let's proceed with executing these conversation flows to assess our system's effectiveness." 223 | ] 224 | }, 225 | { 226 | "cell_type": "code", 227 | "execution_count": null, 228 | "id": "d41e1309-3fa4-4a1c-8f9f-3e05c05cd385", 229 | "metadata": {}, 230 | "outputs": [], 231 | "source": [ 232 | "from langchain_core.messages import HumanMessage\n", 233 | "\n", 234 | "config = {\"configurable\": {\"user_id\": 918}}\n", 235 | "response_andrew = agent_executor.invoke(\n", 236 | " {\"messages\": [HumanMessage(content=\"Please suggest me a good vacation destination alongside some details.\")]},\n", 237 | " config,\n", 238 | " )\n", 239 | "response_andrew" 240 | ] 241 | }, 242 | { 243 | "cell_type": "code", 244 | "execution_count": null, 245 | "id": "f17dad47-102f-45d0-9145-fbe16de2bfc8", 246 | "metadata": {}, 247 | "outputs": [], 248 | "source": [ 249 | "from langchain_core.messages import HumanMessage\n", 250 | "\n", 251 | "config = {\"configurable\": {}}\n", 252 | "response_jane = agent_executor.invoke(\n", 253 | " {\"messages\": [HumanMessage(content=\"Suggest me a good vacation destination. I love beaches!\")]},\n", 254 | " config,\n", 255 | " )\n", 256 | "response_jane" 257 | ] 258 | }, 259 | { 260 | "cell_type": "markdown", 261 | "id": "5b086dc3-5203-4531-beb5-5566a2b13815", 262 | "metadata": {}, 263 | "source": [ 264 | "Now that we have collected the agent conversations from Andrew and Jane we can proceed with converting them from LangChain's message format into ragas message format. For this conversion, we will utilize the previously defined `convert_messages` function." 265 | ] 266 | }, 267 | { 268 | "cell_type": "code", 269 | "execution_count": 9, 270 | "id": "68447b7a-85e9-461f-a0db-742050547734", 271 | "metadata": {}, 272 | "outputs": [ 273 | { 274 | "name": "stdout", 275 | "output_type": "stream", 276 | "text": [ 277 | "Human: Please suggest me a good vacation destination alongside some details.\n", 278 | "-----------\n", 279 | "AI: Okay, let's find a good vacation destination for you. \n", 280 | "Tools:\n", 281 | " compare_and_recommend_destination: {}\n", 282 | "-----------\n", 283 | "ToolOutput: Based on your current location (Paris), age (62.0), and past travel data, we recommend visiting Ljubljana.\n", 284 | "-----------\n", 285 | "AI: Now let me look up some details about Ljubljana in the travel guide:\n", 286 | "Tools:\n", 287 | " travel_guide: {'query': 'Ljubljana'}\n", 288 | "-----------\n", 289 | "ToolOutput: Travel Guide: Ljubljana\n", 290 | "Generated by Llama3.1 405B\n", 291 | " \n", 292 | "Nestled in the heart of Slovenia, Ljubljana is a captivating European gem that seamlessly blends\n", 293 | "medieval charm with modern vibrancy. As the nation's capital and largest city, Ljubljana enchants\n", 294 | "visitors with its picturesque Old Town, where cobblestone streets wind past Baroque architecture and\n", 295 | "the iconic Triple Bridge. Dominating the skyline is the Ljubljana Castle, a 12th-century fortress perched\n", 296 | "atop a hill and offering panoramic views of the city and the Julian Alps beyond. Along the\n", 297 | "emerald-green Ljubljanica River, you'll find lively outdoor cafes, artisan shops, and the open-air Central\n", 298 | "Market, a hub of local culture and cuisine. Just a short drive from the city center, the stunning natural\n", 299 | "wonders of Lake Bled and the Postojna Cave system provide ample opportunities for hiking, boating,\n", 300 | "and exploring Slovenia's breathtaking landscapes. With its walkable streets, rich history, and vibrant\n", 301 | "arts scene, Ljubljana is a must-visit destination that captures the essence of Central Europe.\n", 302 | " \n", 303 | "Page 1\n", 304 | "\n", 305 | "-----------\n", 306 | "AI: Based on the information from the travel guide, Ljubljana seems like an excellent choice for a vacation destination. It offers a charming old town, historic landmarks like the Ljubljana Castle, beautiful natural scenery, and a vibrant cultural scene. The city's compact size and walkable streets also make it easy to explore. I would recommend considering Ljubljana for your next vacation.\n", 307 | "Tools:\n", 308 | "-----------\n" 309 | ] 310 | } 311 | ], 312 | "source": [ 313 | "rg_messages_andrew = convert_messages(response_andrew)\n", 314 | "for m in rg_messages_andrew:\n", 315 | " print(m.pretty_repr())\n", 316 | " print(\"-----------\")" 317 | ] 318 | }, 319 | { 320 | "cell_type": "code", 321 | "execution_count": 10, 322 | "id": "cb642fba-8d53-4ff2-b1b6-e7f421733adb", 323 | "metadata": {}, 324 | "outputs": [ 325 | { 326 | "name": "stdout", 327 | "output_type": "stream", 328 | "text": [ 329 | "Human: Suggest me a good vacation destination. I love beaches!\n", 330 | "-----------\n", 331 | "AI: Okay, let's find a great beach destination for you!\n", 332 | "Tools:\n", 333 | " compare_and_recommend_destination: {}\n", 334 | "-----------\n", 335 | "ToolOutput: User not found in the travel database.\n", 336 | "-----------\n", 337 | "AI: Since this is your first time using the system, I don't have any information about destinations you've already visited. Let me check the travel guide for some good beach vacation options:\n", 338 | "Tools:\n", 339 | " travel_guide: {'query': 'beach vacation'}\n", 340 | "-----------\n", 341 | "ToolOutput: Travel Guide: Newport\n", 342 | "Generated by Llama3.1 405B\n", 343 | " \n", 344 | "Nestled along the sparkling shores of Narragansett Bay, Newport, Rhode Island is a captivating coastal\n", 345 | "gem that seamlessly blends its storied past with modern allure. This quintessential New England town\n", 346 | "is renowned for its opulent Gilded Age mansions, which stand as testaments to the wealth and\n", 347 | "grandeur of America's elite during the late 19th century. Visitors can tour the extravagant \"summer\n", 348 | "cottages\" of the Vanderbilts, Astors, and other industrial tycoons, marveling at the ornate architecture\n", 349 | "and lavish interiors that evoke a bygone era of unbridled luxury.\n", 350 | " \n", 351 | "Beyond the opulent mansions, Newport offers a wealth of historical attractions that shed light on the\n", 352 | "city's pivotal role in American history. The International Tennis Hall of Fame celebrates the sport's\n", 353 | "storied legacy, while the scenic Cliff Walk allows travelers to stroll along the rugged coastline and\n", 354 | "admire the stunning ocean vistas. Nearby, the iconic Breakers mansion and the Marble House stand as\n", 355 | "architectural masterpieces that captivate visitors with their sheer scale and exquisite design.\n", 356 | " \n", 357 | "For nature enthusiasts, the nearby Sachuest Point National Wildlife Refuge and Norman Bird\n", 358 | "Sanctuary provide ample opportunities for hiking, birdwatching, and immersing oneself in the region's\n", 359 | "natural beauty. And no visit to Newport would be complete without sampling the city's renowned\n", 360 | "seafood, from fresh-caught lobster to award-winning chowder, all enjoyed against the backdrop of the\n", 361 | "sparkling harbor.\n", 362 | " \n", 363 | "Whether you're drawn to the city's rich history, architectural wonders, or coastal charm, Newport offers\n", 364 | "a truly unforgettable experience that captures the essence of New England's enduring allure.\n", 365 | " \n", 366 | "Page 1\n", 367 | "\n", 368 | "-----------\n", 369 | "AI: Based on the information in the travel guide, Newport, Rhode Island seems like a great beach vacation destination for you. It has beautiful coastal scenery, historic mansions, and plenty of opportunities to enjoy the outdoors and seafood. Let me know if you would like me to look up any other beach destinations for you!\n", 370 | "Tools:\n", 371 | "-----------\n" 372 | ] 373 | } 374 | ], 375 | "source": [ 376 | "rg_messages_jane = convert_messages(response_jane)\n", 377 | "for m in rg_messages_jane:\n", 378 | " print(m.pretty_repr())\n", 379 | " print(\"-----------\")" 380 | ] 381 | }, 382 | { 383 | "cell_type": "markdown", 384 | "id": "f1ae3c79-521f-4962-9e4e-9a87e44a51f3", 385 | "metadata": {}, 386 | "source": [ 387 | "With our conversation flows now properly formatted, we can proceed with the actual evaluation phase." 388 | ] 389 | }, 390 | { 391 | "cell_type": "markdown", 392 | "id": "cf995770-f6f7-4f86-909f-b60328ddfeaf", 393 | "metadata": {}, 394 | "source": [ 395 | "### Agent Goal Accuracy\n", 396 | "\n", 397 | "Agent Goal Accuracy is a metric designed to evaluate how well an LLM identifies and achieves user goals. It's a binary metric where 1 indicates successful goal achievement and 0 indicates failure. The evaluation is performed using an evaluator LLM, which needs to be defined and configured before metric calculation.\n", 398 | "\n", 399 | "The Agent Goal Accuracy metric comes in two distinct variants:\n", 400 | "\n", 401 | "- Agent Goal Accuracy without reference\n", 402 | "- Agent Goal Accuracy with reference\n", 403 | "\n", 404 | "Before exploring these variants in detail, we need to establish our evaluator LLM. For this purpose, we will utilize Anthropic Claude 3 Haiku as our judge. While this is our choice in this lab, the selection of the evaluator LLM always needs to be adjusted based on specific use cases and requirements." 405 | ] 406 | }, 407 | { 408 | "cell_type": "code", 409 | "execution_count": 11, 410 | "id": "3f6786f5-4c5f-4ca4-a5da-30a275fbbd5c", 411 | "metadata": {}, 412 | "outputs": [], 413 | "source": [ 414 | "import boto3\n", 415 | "from ragas.llms import LangchainLLMWrapper\n", 416 | "from langchain_aws import ChatBedrockConverse\n", 417 | "\n", 418 | "# ---- ⚠️ Update region for your AWS setup ⚠️ ----\n", 419 | "bedrock_client = boto3.client(\"bedrock-runtime\", region_name=\"us-west-2\")\n", 420 | "\n", 421 | "judge_llm = LangchainLLMWrapper(ChatBedrockConverse(\n", 422 | " model=\"anthropic.claude-3-haiku-20240307-v1:0\",\n", 423 | " temperature=0,\n", 424 | " max_tokens=None,\n", 425 | " client=bedrock_client,\n", 426 | " # other params...\n", 427 | "))" 428 | ] 429 | }, 430 | { 431 | "cell_type": "markdown", 432 | "id": "2cff6e40-26c3-4ccb-b32b-65d2def73410", 433 | "metadata": {}, 434 | "source": [ 435 | "#### Agent Goal Accuracy Without Reference\n", 436 | "AgentGoalAccuracyWithoutReference operates without a predefined reference point. Instead, it evaluates the LLM's performance by inferring the desired outcome from the human interactions within the workflow. This approach is particularly useful when explicit reference outcomes are not available or when the success criteria can be determined from the conversation context.\n", 437 | "\n", 438 | "To evaluate this metric, we first encapsulate our agent conversation in a `MultiTurnSample` object, which is designed to handle multi-turn agentic conversations within the ragas ecosystem. Next, we initialize an `AgentGoalAccuracyWithoutReference` object to implement our evaluation metric. Finally, we configure the judge LLM and execute the evaluation across our three agent conversations." 439 | ] 440 | }, 441 | { 442 | "cell_type": "code", 443 | "execution_count": 12, 444 | "id": "b8a10ce3-e3bf-486d-bb6d-17e2a0638562", 445 | "metadata": {}, 446 | "outputs": [], 447 | "source": [ 448 | "from ragas.dataset_schema import MultiTurnSample\n", 449 | "from ragas.messages import HumanMessage,AIMessage,ToolMessage,ToolCall\n", 450 | "from ragas.metrics import AgentGoalAccuracyWithoutReference\n", 451 | "\n", 452 | "\n", 453 | "sample_andrew = MultiTurnSample(user_input=rg_messages_andrew)\n", 454 | "\n", 455 | "sample_jane = MultiTurnSample(user_input=rg_messages_jane)\n", 456 | "\n", 457 | "scorer = AgentGoalAccuracyWithoutReference(llm=judge_llm)" 458 | ] 459 | }, 460 | { 461 | "cell_type": "code", 462 | "execution_count": 13, 463 | "id": "95db55ba-e93c-4c2d-8a0d-c365b6222983", 464 | "metadata": {}, 465 | "outputs": [ 466 | { 467 | "data": { 468 | "text/plain": [ 469 | "1.0" 470 | ] 471 | }, 472 | "execution_count": 13, 473 | "metadata": {}, 474 | "output_type": "execute_result" 475 | } 476 | ], 477 | "source": [ 478 | "await scorer.multi_turn_ascore(sample_andrew)" 479 | ] 480 | }, 481 | { 482 | "cell_type": "code", 483 | "execution_count": 14, 484 | "id": "26e411bc-0934-4223-bbbf-7089163b0e3d", 485 | "metadata": {}, 486 | "outputs": [ 487 | { 488 | "data": { 489 | "text/plain": [ 490 | "0.0" 491 | ] 492 | }, 493 | "execution_count": 14, 494 | "metadata": {}, 495 | "output_type": "execute_result" 496 | } 497 | ], 498 | "source": [ 499 | "await scorer.multi_turn_ascore(sample_jane)" 500 | ] 501 | }, 502 | { 503 | "cell_type": "markdown", 504 | "id": "e05a484c-f0ef-49a6-a43c-f0c387322867", 505 | "metadata": {}, 506 | "source": [ 507 | "#### Agent Goal Accuracy With Reference\n", 508 | "AgentGoalAccuracyWithReference requires two key inputs: the user_input and a reference outcome. This variant evaluates the LLM's performance by comparing its achieved outcome against an annotated reference that serves as the ideal outcome. The metric is calculated at the end of the workflow by assessing how closely the LLM's result matches the predefined reference outcome.\n", 509 | "\n", 510 | "To evaluate this metric, we will follow a similar approach. First, we encapsulate our agent conversation within a `MultiTurnSample` object, which is specifically designed to manage multi-turn agent conversations in the ragas library. For this evaluation, we need to provide an annotated reference that will serve as a benchmark for the judge's assessment. We then initialize an `AgentGoalAccuracyWithReference` object to implement our evaluation metric. Thereby, we set up the judge LLM as evaluator llm. Then we conduct the evaluation across all three agent conversations to measure their performance against our defined criteria." 511 | ] 512 | }, 513 | { 514 | "cell_type": "code", 515 | "execution_count": 15, 516 | "id": "f09bf878-43d0-4aea-9e27-4ec16af602e2", 517 | "metadata": {}, 518 | "outputs": [], 519 | "source": [ 520 | "from ragas.dataset_schema import MultiTurnSample\n", 521 | "from ragas.messages import HumanMessage,AIMessage,ToolMessage,ToolCall\n", 522 | "from ragas.metrics import AgentGoalAccuracyWithReference\n", 523 | "\n", 524 | "\n", 525 | "sample_andrew = MultiTurnSample(user_input=rg_messages_andrew,\n", 526 | " reference=\"Provide a holiday suggestion alongside of detailed information about destination.\")\n", 527 | "\n", 528 | "sample_jane = MultiTurnSample(user_input=rg_messages_jane,\n", 529 | " reference=\"Provide detailed information about suggested holiday destination.\")\n", 530 | "\n", 531 | "scorer = AgentGoalAccuracyWithReference(llm=judge_llm)" 532 | ] 533 | }, 534 | { 535 | "cell_type": "code", 536 | "execution_count": 16, 537 | "id": "eebac6f9-3443-473a-af4c-3e33588258a5", 538 | "metadata": {}, 539 | "outputs": [ 540 | { 541 | "data": { 542 | "text/plain": [ 543 | "1.0" 544 | ] 545 | }, 546 | "execution_count": 16, 547 | "metadata": {}, 548 | "output_type": "execute_result" 549 | } 550 | ], 551 | "source": [ 552 | "await scorer.multi_turn_ascore(sample_andrew)" 553 | ] 554 | }, 555 | { 556 | "cell_type": "code", 557 | "execution_count": 17, 558 | "id": "1fde20da-8227-48d9-8dec-8de757a786bf", 559 | "metadata": {}, 560 | "outputs": [ 561 | { 562 | "data": { 563 | "text/plain": [ 564 | "1.0" 565 | ] 566 | }, 567 | "execution_count": 17, 568 | "metadata": {}, 569 | "output_type": "execute_result" 570 | } 571 | ], 572 | "source": [ 573 | "await scorer.multi_turn_ascore(sample_jane)" 574 | ] 575 | }, 576 | { 577 | "cell_type": "markdown", 578 | "id": "bc2e40f9-cdeb-4a62-86c9-38367c284a1a", 579 | "metadata": {}, 580 | "source": [ 581 | "Let's analyze the results and their relationship to each persona's interaction patterns with the agent. We encourage you to discuss these findings with your workshop group to gain deeper insights. Keep in mind that agent conversations are dynamic and non-deterministic, meaning evaluation results may vary across different runs. \n", 582 | "\n", 583 | "However, certain patterns emerge:\n", 584 | "- Andrew's conversations typically achieve a 1.0 rating due to their focused and goal-oriented approach\n", 585 | "- Jane's conversations are typically rated with 0.0. Due to the lack of historic information the system can't provide suggestions in one single conversation turn. A human in the loop approach asking for her interests could solve this.\n", 586 | "\n", 587 | "Note, that especially for `AgentGoalAccuracyWithReference` you could influence the results by adjusting either the conversation flow or the reference. If you have time left, feel free to try it out!" 588 | ] 589 | }, 590 | { 591 | "cell_type": "markdown", 592 | "id": "b9e37b94-2166-4f4c-9bca-bc68a16d1fa0", 593 | "metadata": {}, 594 | "source": [ 595 | "### Tool Call accuracy\n", 596 | "\n", 597 | "ToolCallAccuracy is a metric that can be used to evaluate the performance of the LLM in identifying and calling the required tools to complete a given task. This metric needs user_input and reference_tool_calls to evaluate the performance of the LLM in identifying and calling the required tools to complete a given task. The metric is computed by comparing the reference_tool_calls with the Tool calls made by the AI. Therefore, in this particular scenario, there is no need for an evaluator LLM. The values range between 0 and 1, with higher values indicating better performance.\n", 598 | "\n", 599 | "To evaluate the tool call accuracy metric, we follow a different process. First, we encapsulate our agent conversation within a `MultiTurnSample` object, which is specifically designed to handle multi-turn agent conversations in the ragas library. This evaluation requires a set of annotated reference tool calls that serve as a benchmark for assessment. Next, we initialize a `ToolCallAccuracy` object to implement our evaluation metric. While the default behavior compares tool names and arguments using exact string matching, this may not always be optimal, particularly when dealing with natural language arguments. To mitigate this, ragas provides us with a choice of different NLP distance metrics we can employ to determine the relevance of retrieved contexts more effectively. In this lab we use `NonLLMStringSimilarity`, which is leveraging traditional string distance measures such as Levenshtein, Hamming, and Jaro. Therefore, we set the parameter `arg_comparison_metric` to `NonLLMStringSimilarity`." 600 | ] 601 | }, 602 | { 603 | "cell_type": "code", 604 | "execution_count": 19, 605 | "id": "f158250b-ca76-46e9-a60c-c621e1215f34", 606 | "metadata": {}, 607 | "outputs": [], 608 | "source": [ 609 | "from ragas.metrics import ToolCallAccuracy\n", 610 | "from ragas.dataset_schema import MultiTurnSample\n", 611 | "from ragas.messages import HumanMessage,AIMessage,ToolMessage,ToolCall\n", 612 | "from ragas.metrics._string import NonLLMStringSimilarity\n", 613 | "\n", 614 | "\n", 615 | "\n", 616 | "sample_andrew = MultiTurnSample(\n", 617 | " user_input=rg_messages_andrew,\n", 618 | " reference_tool_calls=[\n", 619 | " ToolCall(name=\"compare_and_recommend_destination\", args={}),\n", 620 | " ToolCall(name=\"travel_guide\", args={\"query\": \"Ljubljana\"}),\n", 621 | " ]\n", 622 | ")\n", 623 | "\n", 624 | "sample_jane = MultiTurnSample(\n", 625 | " user_input=rg_messages_jane,\n", 626 | " reference_tool_calls=[\n", 627 | " ToolCall(name=\"compare_and_recommend_destination\", args={}),\n", 628 | " ToolCall(name=\"travel_guide\", args={\"query\": \"Miami, Florida\"}),\n", 629 | " ]\n", 630 | ")\n", 631 | "\n", 632 | "scorer = ToolCallAccuracy()\n", 633 | "scorer.arg_comparison_metric = NonLLMStringSimilarity()" 634 | ] 635 | }, 636 | { 637 | "cell_type": "code", 638 | "execution_count": 20, 639 | "id": "df1a688a-4d99-4b24-b487-5918c62414a4", 640 | "metadata": {}, 641 | "outputs": [ 642 | { 643 | "data": { 644 | "text/plain": [ 645 | "1.0" 646 | ] 647 | }, 648 | "execution_count": 20, 649 | "metadata": {}, 650 | "output_type": "execute_result" 651 | } 652 | ], 653 | "source": [ 654 | "await scorer.multi_turn_ascore(sample_andrew)" 655 | ] 656 | }, 657 | { 658 | "cell_type": "code", 659 | "execution_count": 21, 660 | "id": "834ba036-9c7e-44ce-b9c0-9e11c793a562", 661 | "metadata": {}, 662 | "outputs": [ 663 | { 664 | "data": { 665 | "text/plain": [ 666 | "0.5714285714285714" 667 | ] 668 | }, 669 | "execution_count": 21, 670 | "metadata": {}, 671 | "output_type": "execute_result" 672 | } 673 | ], 674 | "source": [ 675 | "await scorer.multi_turn_ascore(sample_jane)" 676 | ] 677 | }, 678 | { 679 | "cell_type": "markdown", 680 | "id": "28be7ce6-c419-4135-80c9-f3e974ca97c1", 681 | "metadata": {}, 682 | "source": [ 683 | "Let's analyze the results and their relationship to each persona's interaction patterns with the agent. We encourage you to discuss these findings with your workshop group to gain deeper insights. Keep in mind that agent conversations are dynamic and non-deterministic, meaning evaluation results may vary across different runs.\n", 684 | "\n", 685 | "However, certain patterns emerge:\n", 686 | "- Andrew's conversations typically achieve a very high rating due to his focused and goal-oriented approach and the fact that he can be found in the travel database - this helps matching all tool call arguments\n", 687 | "- Jane's conversations typically achieve a high but slightly lower rating. While her conversation is focused and goal-oriented, she is not in the travel database. This causes the tool call arguments to be less deterministic, reducing the likelihood of a specific city as recommendation. If you you have time, try modifying the reference `query` argument of the `travel_guide` tool call, e.g. to \"beach destination\". Try to correlate the result with the tool calls in the message history. What do you observe? " 688 | ] 689 | }, 690 | { 691 | "cell_type": "markdown", 692 | "id": "a6a38535-bd42-40be-9ab1-5a72e3536e6f", 693 | "metadata": {}, 694 | "source": [ 695 | "Congratulations! You have successfully completed this lab and the entire workshop. Thank you for your active participation in today's session. If you have any questions or need clarification on any topics covered, please don't hesitate to reach out to the instructors.\n", 696 | "\n", 697 | "**Your feedback is valuable to us! Before you leave, we kindly ask you to take a moment to scan the QR code and share your thoughts about the workshop. Your insights will help us enhance future sessions and provide an even better learning experience.**" 698 | ] 699 | } 700 | ], 701 | "metadata": { 702 | "kernelspec": { 703 | "display_name": "Python 3 (ipykernel)", 704 | "language": "python", 705 | "name": "python3" 706 | }, 707 | "language_info": { 708 | "codemirror_mode": { 709 | "name": "ipython", 710 | "version": 3 711 | }, 712 | "file_extension": ".py", 713 | "mimetype": "text/x-python", 714 | "name": "python", 715 | "nbconvert_exporter": "python", 716 | "pygments_lexer": "ipython3", 717 | "version": "3.10.14" 718 | } 719 | }, 720 | "nbformat": 4, 721 | "nbformat_minor": 5 722 | } 723 | --------------------------------------------------------------------------------