├── .env.example ├── .gitignore ├── README.md ├── coffee_scraper.py ├── data ├── Canada.pdf ├── notes.txt └── population.csv ├── main.py ├── note_engine.py ├── pdf.py ├── prompts.py └── requirements.txt /.env.example: -------------------------------------------------------------------------------- 1 | SCRAPELESS_API_TOKEN="your_api_token_here" 2 | OPENAI_API_KEY="your_openai_key" 3 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | env/ 2 | .env 3 | .vscode 4 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # PythonAgentAI 2 | 3 | The **PythonAgentAI** project aims to leverage advanced OpenAI models (including `gpt-4o-mini` with a 128k context window) to perform a variety of AI-powered tasks. This project integrates [**Scrapeless**](https://www.scrapeless.com/en?utm_source=github&utm_medium=readme&utm_campaign=twt) for [Google Maps](https://www.scrapeless.com/en/product/deep-serp-api?utm_source=github&utm_medium=readme&utm_campaign=twt) and the **llama-index** library (version 0.12.22) along with its experimental extensions to enable large language models to provide real-time responses. 4 | 5 | ## Features 6 | 7 | - Integration with OpenAI’s `gpt-4o-mini` model, supporting a 128k context window. 8 | - Modular code structure including components like `main.py`, `pdf`.`py`, `prompts.py`, and `note_engine.py`. 9 | - Advanced indexing and querying via `llama-index` and `llama-index-experimental` libraries. 10 | 11 | ## Installation 12 | 13 | ### Prerequisites 14 | 15 | - Python 3.11 must be installed on your system. You can download it from the [official Python website](https://www.python.org/downloads/). 16 | 17 | ## Configuration 18 | 19 | 1. **Create the `.env` File** 20 | 21 | The project requires a `.env` file for storing environment variables, including the OpenAI API key. You can create it by running: 22 | 23 | ```bash 24 | cp .env.example .env 25 | ``` 26 | 27 | 2. **Set Your OpenAI API Key** 28 | 29 | Open the `.env` file in a text editor and configure your OpenAI API key: 30 | 31 | ``` 32 | OPENAI_API_KEY=your_openai_api_key_here 33 | ``` 34 | 35 | Replace `your_openai_api_key_here` with your actual API key from OpenAI. 36 | 37 | - **Log into [Scrapeless](https://app.scrapeless.com/passport/login?utm_source=github&utm_medium=readme&utm_campaign=twt) and obtain your API token.** 38 | 39 | ![Get the Scrapeless API key](https://assets.scrapeless.com/prod/posts/naver-product/77c0cef86a29013173eb41a34f42d3f4.png) 40 | 41 | ### Setup Instructions 42 | 43 | 1. **Clone the Repository** 44 | 45 | Open your terminal and run: 46 | 47 | ```bash 48 | git clone https://github.com/your-username/PythonAgentAI.git 49 | cd PythonAgentAI 50 | ``` 51 | 52 | 2. **Create a Virtual Environment** 53 | 54 | It's recommended to use a virtual environment to manage dependencies. To create one using Python's `venv` module: 55 | 56 | ```bash 57 | python3.11 -m venv env 58 | ``` 59 | 60 | 3. **Activate the Virtual Environment** 61 | 62 | - On **Unix or macOS**: 63 | 64 | ```bash 65 | source env/bin/activate 66 | ``` 67 | 68 | - On **Windows**: 69 | 70 | ```bash 71 | .\env\Scripts\activate 72 | ``` 73 | 74 | 4. **Install Dependencies** 75 | 76 | With the virtual environment activated, install the required packages: 77 | 78 | ```bash 79 | pip install -r requirements.txt 80 | ``` 81 | 82 | This will install the following essential packages: 83 | 84 | - `llama-index==0.12.22` 85 | - `llama-index-experimental==0.5.4` 86 | - `pypdf==5.3.1` 87 | - `python-dotenv==1.0.1` 88 | - `pandas==2.2.3` 89 | 90 | ## Usage 91 | 92 | After setting up the environment and installing the dependencies, you can run the main script: 93 | 94 | ```bash 95 | python main.py 96 | ``` 97 | 98 | 99 | Ensure that you have configured any necessary environment variables or settings required by the script. Refer to the `prompts.py` and `note_engine.py` files for customizable parameters and functionalities. 100 | 101 | 2. **Input the provided prompts to receive results**. After a short wait, you’ll see output similar to the images below: 102 | 103 | - **Find the highest rated coffee shop within 0.5km** 104 | 105 | ![Result of the highest rated coffee shop within 0.5km](https://assets.scrapeless.com/prod/posts/deep-serp-api-online/4ea1b12e422967bccd0db82282cb0270.png) 106 | 107 | - **Find the closest coffee shop to the target location** 108 | 109 | ![Result of the closest coffee shop to the location](https://assets.scrapeless.com/prod/posts/deep-serp-api-online/d7e32f4d01913dbd7b76e15983ce46e2.png) 110 | -------------------------------------------------------------------------------- /coffee_scraper.py: -------------------------------------------------------------------------------- 1 | import os 2 | import requests 3 | import pandas as pd 4 | from geopy.distance import geodesic 5 | from llama_index.experimental import PandasQueryEngine 6 | 7 | 8 | class CoffeeScraper: 9 | def __init__(self): 10 | self.api_token = os.getenv("SCRAPELESS_API_TOKEN") 11 | self.gallery_coords = (45.4299, -75.6939) # National Gallery of Canada location 12 | self.df = pd.DataFrame(columns=[ 13 | "name", "address", "price_level", 14 | "rating", "distance_km", "scraped_at" 15 | ]) 16 | 17 | def _send_scrapeless_request(self, query: str): 18 | host = "api.scrapeless.com" 19 | url = f"https://{host}/api/v1/scraper/request" 20 | 21 | payload = { 22 | "actor": "scraper.google.maps", 23 | "input": { 24 | "q": query, 25 | "engine": "google_maps", 26 | "type": "search", 27 | "ll": "@45.4299,-75.6939,14z", 28 | "google_domain": "google.com", 29 | "gl": "ca", 30 | "hl": "en-sg", 31 | "data": "", 32 | "place_id": "", 33 | "start": "" 34 | } 35 | } 36 | 37 | response = requests.post( 38 | url, 39 | headers={"x-api-token": self.api_token}, 40 | json=payload 41 | ) 42 | 43 | if response.status_code == 200: 44 | res_data = response.json() 45 | print(f"Scraping status: {res_data}") 46 | return res_data["local_results"] 47 | else: 48 | raise Exception(f"Scraping failed: {response.text}") 49 | 50 | def scrape_nearby_coffee(self): 51 | try: 52 | results = self._send_scrapeless_request("coffee") 53 | 54 | for place in results: 55 | shop_coords = (place["gps_coordinates"]["latitude"], place["gps_coordinates"]["longitude"]) 56 | print(f"shop_coords: {shop_coords}") 57 | print(f"distance: {round(geodesic(self.gallery_coords, shop_coords).km, 2)}") 58 | self.df = pd.concat([self.df, pd.DataFrame([{ 59 | "name": place["title"], 60 | "address": place["address"], 61 | "price_level": len(place.get("price", "")), 62 | "rating": place.get("rating", None), 63 | "distance_km": round(geodesic(self.gallery_coords, shop_coords).km, 2), 64 | "scraped_at": pd.Timestamp.now() 65 | }])], ignore_index=True) 66 | 67 | print(f"Successfully scraped data for {len(results)} coffee shops.") 68 | except Exception as e: 69 | print(f"The scraping failed.: {str(e)}") 70 | 71 | def get_query_engine(self): 72 | return PandasQueryEngine( 73 | df=self.df, 74 | verbose=True, 75 | instruction_str=( 76 | "You are a coffee shop analyst. Use this data to answer questions " 77 | "about coffee shops near National Gallery of Canada. " 78 | "Key fields: name, price_level ($1-3), rating (1-5), distance_km." 79 | ) 80 | ) 81 | 82 | 83 | coffee_scraper = CoffeeScraper() 84 | -------------------------------------------------------------------------------- /data/Canada.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/techwithtim/PythonAgentAI/a16e757b5367ca7314e370e1cfc3ece6874261fe/data/Canada.pdf -------------------------------------------------------------------------------- /data/notes.txt: -------------------------------------------------------------------------------- 1 | I love Tim 2 | save a note of that! 3 | In Canada, the two main languages spoken are English and French. Approximately 98 percent of Canadians can speak either or both English and French. English is spoken by 57 percent of Canadians, while French is spoken by 21 percent. There are also Indigenous language groups in Canada, with more than 65 distinct languages and dialects. Additionally, Canada is home to various sign languages, including American Sign Language (ASL) and Quebec Sign Language (LSQ). 4 | -------------------------------------------------------------------------------- /data/population.csv: -------------------------------------------------------------------------------- 1 | Rank,Country,Population2023,YearlyChange,NetChange,Density(P/Km²),Land Area(Km²),Migrants(net),Fert.Rate,MedianAge,UrbanPop%,WorldShare 2 | 36,Afghanistan,42239854,2.70 %,1111083,65,652860,-65846,4.4,17,26 %,0.53 % 3 | 138,Albania,2832439,-0.35 %,-9882,103,27400,-8000,1.4,38,67 %,0.04 % 4 | 34,Algeria,45606480,1.57 %,703255,19,2381740,-9999,2.8,28,75 %,0.57 % 5 | 212,American Samoa,43914,-0.81 %,-359,220,200,-790,2.2,29,N.A.,0.00 % 6 | 202,Andorra,80088,0.33 %,264,170,470,200,1.1,43,85 %,0.00 % 7 | 42,Angola,36684202,3.08 %,1095215,29,1246700,-1000,5.1,16,68 %,0.46 % 8 | 223,Anguilla,15899,0.26 %,42,177,90,0,1.3,38,98 %,0.00 % 9 | 200,Antigua and Barbuda,94298,0.57 %,535,214,440,0,1.6,36,28 %,0.00 % 10 | 33,Argentina,45773884,0.58 %,263566,17,2736690,3718,1.9,32,94 %,0.57 % 11 | 140,Armenia,2777970,-0.09 %,-2499,98,28470,-5000,1.6,35,67 %,0.03 % 12 | 197,Aruba,106277,-0.16 %,-168,590,180,157,1.2,42,45 %,0.00 % 13 | 55,Australia,26439111,1.00 %,261698,3,7682300,139991,1.6,38,86 %,0.33 % 14 | 100,Austria,8958960,0.22 %,19343,109,82409,19999,1.5,43,59 %,0.11 % 15 | 90,Azerbaijan,10412651,0.53 %,54577,126,82658,0,1.7,32,57 %,0.13 % 16 | 177,Bahamas,412623,0.64 %,2639,41,10010,1000,1.4,33,85 %,0.01 % 17 | 154,Bahrain,1485509,0.90 %,13276,1955,760,0,1.8,34,N.A.,0.02 % 18 | 8,Bangladesh,172954319,1.03 %,1767947,1329,130170,-309977,1.9,27,41 %,2.15 % 19 | 187,Barbados,281995,0.13 %,360,656,430,-80,1.6,40,32 %,0.00 % 20 | 97,Belarus,9498238,-0.39 %,-36716,47,202910,-4282,1.5,41,80 %,0.12 % 21 | 82,Belgium,11686140,0.26 %,30210,386,30280,23999,1.6,41,99 %,0.15 % 22 | 178,Belize,410825,1.37 %,5553,18,22810,600,2,26,48 %,0.01 % 23 | 77,Benin,13712828,2.70 %,359964,122,112760,-200,4.8,18,48 %,0.17 % 24 | 205,Bermuda,64069,-0.18 %,-115,1281,50,0,1.4,46,94 %,0.00 % 25 | 165,Bhutan,787424,0.64 %,4969,21,38117,300,1.4,29,49 %,0.01 % 26 | 80,Bolivia,12388571,1.35 %,164461,11,1083300,-3000,2.5,24,69 %,0.15 % 27 | 137,Bosnia and Herzegovina,3210847,-0.70 %,-22679,63,51000,-500,1.3,42,54 %,0.04 % 28 | 144,Botswana,2675352,1.71 %,45056,5,566730,3000,2.7,24,69 %,0.03 % 29 | 7,Brazil,216422446,0.52 %,1108948,26,8358140,6000,1.6,34,88 %,2.69 % 30 | 219,British Virgin Islands,31538,0.74 %,233,210,150,200,1,39,53 %,0.00 % 31 | 176,Brunei ,452524,0.78 %,3522,86,5270,0,1.7,33,80 %,0.01 % 32 | 110,Bulgaria,6687717,-1.39 %,-94236,62,108560,-4800,1.6,45,78 %,0.08 % 33 | 59,Burkina Faso,23251485,2.55 %,577723,85,273600,-24998,4.6,17,32 %,0.29 % 34 | 78,Burundi,13238559,2.71 %,348983,516,25680,2000,4.9,16,15 %,0.16 % 35 | 171,Cabo Verde,598682,0.93 %,5533,149,4030,-1227,1.9,27,67 %,0.01 % 36 | 73,Cambodia,16944826,1.06 %,176984,96,176520,-29998,2.3,27,26 %,0.21 % 37 | 53,Cameroon,28647293,2.63 %,732757,61,472710,-4800,4.3,18,58 %,0.36 % 38 | 38,Canada,38781291,0.85 %,326964,4,9093510,249746,1.5,41,81 %,0.48 % 39 | 220,Caribbean Netherlands,27148,0.45 %,122,83,328,100,1.6,40,74 %,0.00 % 40 | 204,Cayman Islands,69310,0.88 %,604,289,240,400,1.2,38,95 %,0.00 % 41 | 117,Central African Republic,5742315,2.92 %,163171,9,622980,-14716,5.8,15,40 %,0.07 % 42 | 67,Chad,18278568,3.13 %,555253,15,1259200,-2000,6.1,15,24 %,0.23 % 43 | 65,Chile,19629590,0.13 %,25857,26,743532,-71205,1.5,36,85 %,0.24 % 44 | 2,China,1425671352,-0.02 %,-215985,152,9388211,-310220,1.2,39,65 %,17.72 % 45 | 28,Colombia,52085168,0.41 %,211144,47,1109500,-175051,1.7,32,81 %,0.65 % 46 | 163,Comoros,852075,1.83 %,15301,458,1861,-2000,3.8,20,33 %,0.01 % 47 | 113,Congo,6106869,2.29 %,136445,18,341500,-1000,4,18,70 %,0.08 % 48 | 222,Cook Islands,17044,0.19 %,33,71,240,-93,2.2,33,79 %,0.00 % 49 | 124,Costa Rica,5212173,0.60 %,31344,102,51060,3750,1.5,34,82 %,0.06 % 50 | 130,Croatia,4008617,-0.54 %,-21741,72,55960,-2000,1.4,44,59 %,0.05 % 51 | 85,Cuba,11194449,-0.16 %,-17742,105,106440,-6000,1.5,41,80 %,0.14 % 52 | 190,Curaçao,192077,0.48 %,914,433,444,515,1.6,36,77 %,0.00 % 53 | 158,Cyprus,1260138,0.69 %,8650,136,9240,5000,1.3,39,65 %,0.02 % 54 | 89,Czech Republic (Czechia),10495295,0.01 %,1309,136,77240,22011,1.7,43,75 %,0.13 % 55 | 51,Côte d'Ivoire,28873034,2.53 %,712492,91,318000,6000,4.3,18,52 %,0.36 % 56 | 115,Denmark,5910913,0.49 %,28652,139,42430,19999,1.7,41,88 %,0.07 % 57 | 160,Djibouti,1136455,1.39 %,15606,49,23180,900,2.7,24,72 %,0.01 % 58 | 203,Dominica,73040,0.42 %,303,97,750,-40,1.6,32,75 %,0.00 % 59 | 84,Dominican Republic,11332972,0.93 %,104151,235,48320,-29099,2.2,28,85 %,0.14 % 60 | 15,DR Congo,102262808,3.29 %,3252596,45,2267050,-14999,6.1,16,46 %,1.27 % 61 | 68,Ecuador,18190484,1.05 %,189484,73,248360,-21525,2,28,64 %,0.23 % 62 | 14,Egypt,112716598,1.56 %,1726495,113,995450,-29998,2.8,24,41 %,1.40 % 63 | 112,El Salvador,6364943,0.45 %,28551,307,20720,-23249,1.8,27,78 %,0.08 % 64 | 152,Equatorial Guinea,1714671,2.37 %,39763,61,28050,4000,4.1,21,67 %,0.02 % 65 | 131,Eritrea,3748901,1.76 %,64869,37,101000,-15297,3.7,19,67 %,0.05 % 66 | 156,Estonia,1322765,-0.25 %,-3297,31,42390,-1000,1.7,42,68 %,0.02 % 67 | 159,Eswatini,1210822,0.76 %,9152,70,17200,-5268,2.8,22,31 %,0.02 % 68 | 11,Ethiopia,126527060,2.55 %,3147136,127,1000000,-11999,4,19,22 %,1.57 % 69 | 207,Faeroe Islands,53270,0.34 %,180,38,1396,0,2.7,38,41 %,0.00 % 70 | 231,Falkland Islands,3791,0.29 %,11,0,12170,0,1.6,40,62 %,0.00 % 71 | 162,Fiji,936375,0.71 %,6609,51,18270,-3289,2.4,27,59 %,0.01 % 72 | 118,Finland,5545475,0.09 %,4730,18,303890,13999,1.4,43,87 %,0.07 % 73 | 23,France,64756584,0.20 %,129956,118,547557,67761,1.8,42,84 %,0.80 % 74 | 184,French Guiana,312155,2.49 %,7598,4,82200,1200,3.4,24,90 %,0.00 % 75 | 185,French Polynesia,308872,0.85 %,2593,84,3660,-100,1.7,34,59 %,0.00 % 76 | 146,Gabon,2436566,1.99 %,47574,9,257670,1000,3.4,22,85 %,0.03 % 77 | 141,Gambia,2773168,2.48 %,67176,274,10120,-3000,4.5,17,58 %,0.03 % 78 | 132,Georgia,3728282,-0.43 %,-16103,54,69490,-9999,2.1,37,63 %,0.05 % 79 | 19,Germany,83294633,-0.09 %,-75210,239,348560,155751,1.5,45,77 %,1.04 % 80 | 47,Ghana,34121985,1.93 %,646115,150,227540,-9999,3.5,21,57 %,0.42 % 81 | 217,Gibraltar,32688,0.12 %,39,3269,10,-24,1.8,42,N.A.,0.00 % 82 | 91,Greece,10341277,-0.42 %,-43694,80,128900,5000,1.4,45,86 %,0.13 % 83 | 206,Greenland,56643,0.31 %,177,0,410450,-100,1.9,35,89 %,0.00 % 84 | 194,Grenada,126183,0.59 %,745,371,340,-200,2,32,32 %,0.00 % 85 | 179,Guadeloupe,395839,0.02 %,87,234,1690,-800,2,42,N.A.,0.00 % 86 | 192,Guam,172952,0.69 %,1178,320,540,-500,2.5,30,95 %,0.00 % 87 | 70,Guatemala,18092026,1.39 %,248118,169,107160,-9110,2.3,23,55 %,0.22 % 88 | 75,Guinea,14190612,2.39 %,331271,58,245720,-4000,4.2,18,40 %,0.18 % 89 | 148,Guinea-Bissau,2150842,2.15 %,45276,76,28120,-1400,3.8,19,45 %,0.03 % 90 | 164,Guyana,813834,0.63 %,5108,4,196850,-3900,2.3,26,27 %,0.01 % 91 | 81,Haiti,11724763,1.21 %,139767,425,27560,-31811,2.7,23,60 %,0.15 % 92 | 234,Holy See,518,1.57 %,8,1295,0,0,,,N.A.,0.00 % 93 | 88,Honduras,10593798,1.54 %,160938,95,111890,-5034,2.3,24,58 %,0.13 % 94 | 104,Hong Kong,7491609,0.04 %,2744,7135,1050,19999,0.8,46,N.A.,0.09 % 95 | 94,Hungary,10156239,1.90 %,188931,112,90530,-156677,1.6,42,68 %,0.13 % 96 | 180,Iceland,375318,0.65 %,2419,4,100250,380,1.7,36,88 %,0.00 % 97 | 1,India,1428627663,0.81 %,11454490,481,2973190,-486136,2,28,36 %,17.76 % 98 | 4,Indonesia,277534122,0.74 %,2032783,153,1811570,-49997,2.1,30,59 %,3.45 % 99 | 17,Iran,89172767,0.70 %,622197,55,1628550,-39998,1.7,33,74 %,1.11 % 100 | 35,Iraq,45504560,2.27 %,1008438,105,434320,-6000,3.4,20,71 %,0.57 % 101 | 125,Ireland,5056935,0.67 %,33826,73,68890,9999,1.8,38,64 %,0.06 % 102 | 201,Isle of Man,84710,0.23 %,191,149,570,340,1.6,46,55 %,0.00 % 103 | 98,Israel,9174520,1.51 %,136211,424,21640,9999,2.9,29,92 %,0.11 % 104 | 25,Italy,58870762,-0.28 %,-166712,200,294140,58496,1.3,48,72 %,0.73 % 105 | 139,Jamaica,2825544,-0.06 %,-1833,261,10830,-10999,1.3,32,59 %,0.04 % 106 | 12,Japan,123294513,-0.53 %,-657179,338,364555,99994,1.3,49,94 %,1.53 % 107 | 83,Jordan,11337052,0.45 %,51183,128,88780,-157392,2.7,24,85 %,0.14 % 108 | 66,Kazakhstan,19606633,1.08 %,208635,7,2699700,0,3,30,57 %,0.24 % 109 | 26,Kenya,55100586,1.99 %,1073099,97,569140,-10000,3.2,20,31 %,0.68 % 110 | 193,Kiribati,133515,1.74 %,2283,165,810,-400,3.2,22,56 %,0.00 % 111 | 129,Kuwait,4310108,0.97 %,41235,242,17820,11999,2.1,40,N.A.,0.05 % 112 | 109,Kyrgyzstan,6735347,1.58 %,104724,35,191800,-9999,2.9,24,37 %,0.08 % 113 | 103,Laos,7633779,1.39 %,104304,33,230800,-9999,2.4,24,37 %,0.09 % 114 | 151,Latvia,1830211,-1.10 %,-20440,29,62200,-7630,1.6,44,69 %,0.02 % 115 | 122,Lebanon,5353930,-2.47 %,-135809,523,10230,-177331,2.1,29,97 %,0.07 % 116 | 147,Lesotho,2330318,1.06 %,24493,77,30360,-4000,2.9,22,31 %,0.03 % 117 | 120,Liberia,5418377,2.18 %,115696,56,96320,-5000,4,18,54 %,0.07 % 118 | 107,Libya,6888388,1.12 %,76047,4,1759540,-2000,2.4,27,82 %,0.09 % 119 | 214,Liechtenstein,39584,0.65 %,257,247,160,150,1.5,44,15 %,0.00 % 120 | 142,Lithuania,2718352,-1.15 %,-31703,43,62674,-13128,1.6,44,71 %,0.03 % 121 | 168,Luxembourg,654768,1.11 %,7169,253,2590,4883,1.4,39,88 %,0.01 % 122 | 167,Macao,704149,1.29 %,8981,23472,30,5000,1.1,39,97 %,0.01 % 123 | 50,Madagascar,30325732,2.41 %,714018,52,581795,-1500,3.7,19,40 %,0.38 % 124 | 62,Malawi,20931751,2.58 %,526434,222,94280,-6000,3.8,17,19 %,0.26 % 125 | 46,Malaysia,34308525,1.09 %,370304,104,328550,48997,1.8,31,78 %,0.43 % 126 | 175,Maldives,521021,-0.53 %,-2766,1737,300,-8652,1.7,32,39 %,0.01 % 127 | 58,Mali,23293698,3.10 %,700108,19,1220190,-39998,5.8,15,44 %,0.29 % 128 | 174,Malta,535064,0.33 %,1778,1672,320,850,1.2,40,78 %,0.01 % 129 | 213,Marshall Islands,41996,1.03 %,427,233,180,0,2.6,26,N.A.,0.00 % 130 | 181,Martinique,366981,-0.14 %,-526,346,1060,-650,1.9,47,94 %,0.00 % 131 | 126,Mauritania,4862989,2.68 %,126850,5,1030700,3000,4.3,18,61 %,0.06 % 132 | 157,Mauritius,1300557,0.08 %,1088,641,2030,0,1.4,37,40 %,0.02 % 133 | 182,Mayotte,335995,3.03 %,9894,896,375,0,4.3,17,40 %,0.00 % 134 | 10,Mexico,128455567,0.75 %,951442,66,1943950,-50239,1.8,30,88 %,1.60 % 135 | 173,Micronesia,544321,0.98 %,5308,778,700,-1642,2.7,26,71 %,0.01 % 136 | 134,Moldova,3435931,4.98 %,162935,105,32850,-125204,1.8,35,50 %,0.04 % 137 | 215,Monaco,36297,-0.47 %,-172,24360,1,200,2.1,54,N.A.,0.00 % 138 | 133,Mongolia,3447157,1.44 %,48791,2,1553560,-850,2.7,27,67 %,0.04 % 139 | 169,Montenegro,626485,-0.10 %,-597,47,13450,-480,1.7,39,69 %,0.01 % 140 | 230,Montserrat,4386,-0.09 %,-4,44,100,0,1.6,44,11 %,0.00 % 141 | 39,Morocco,37840044,1.02 %,382073,85,446300,-39998,2.3,29,66 %,0.47 % 142 | 48,Mozambique,33897354,2.81 %,927836,43,786380,-5000,4.5,17,40 %,0.42 % 143 | 27,Myanmar,54577997,0.74 %,398691,84,653290,-34998,2.1,30,33 %,0.68 % 144 | 145,Namibia,2604172,1.45 %,37160,3,823290,-3916,3.2,21,60 %,0.03 % 145 | 224,Nauru,12780,0.88 %,112,639,20,-140,3.4,20,88 %,0.00 % 146 | 49,Nepal,30896590,1.14 %,349010,216,143350,-62012,2,24,22 %,0.38 % 147 | 72,Netherlands,17618299,0.31 %,54285,522,33720,29998,1.6,42,92 %,0.22 % 148 | 186,New Caledonia,292991,1.05 %,3041,16,18280,500,2,34,74 %,0.00 % 149 | 123,New Zealand,5228100,0.83 %,42812,20,263310,12999,1.8,37,82 %,0.06 % 150 | 106,Nicaragua,7046310,1.41 %,97918,59,120340,-8000,2.3,25,56 %,0.09 % 151 | 54,Niger,27202843,3.80 %,994866,21,1266700,1000,6.7,15,17 %,0.34 % 152 | 6,Nigeria,223804632,2.41 %,5263420,246,910770,-59996,5.1,17,54 %,2.78 % 153 | 232,Niue,1935,0.05 %,1,7,260,0,2.4,36,41 %,0.00 % 154 | 56,North Korea,26160821,0.35 %,91405,217,120410,-2000,1.8,36,63 %,0.33 % 155 | 150,North Macedonia,2085679,-0.38 %,-7920,83,25220,-1000,1.4,39,60 %,0.03 % 156 | 208,Northern Mariana Islands,49796,0.49 %,245,108,460,-50,2.1,38,N.A.,0.00 % 157 | 119,Norway,5474360,0.74 %,40041,15,365268,27998,1.5,40,86 %,0.07 % 158 | 127,Oman,4644384,1.49 %,68086,15,309500,0,2.5,29,N.A.,0.06 % 159 | 5,Pakistan,240485658,1.98 %,4660796,312,770880,-165988,3.3,21,35 %,2.99 % 160 | 221,Palau,18058,0.02 %,3,39,460,-20,2.3,36,N.A.,0.00 % 161 | 128,Panama,4468087,1.35 %,59506,60,74340,7262,2.3,29,70 %,0.06 % 162 | 92,Papua New Guinea,10329931,1.85 %,187312,23,452860,-800,3.1,22,12 %,0.13 % 163 | 108,Paraguay,6861524,1.19 %,80780,17,397300,-12499,2.4,26,67 %,0.09 % 164 | 45,Peru,34352719,0.89 %,303131,27,1280000,-61442,2.1,29,79 %,0.43 % 165 | 13,Philippines,117337368,1.54 %,1778359,394,298170,-69996,2.7,25,47 %,1.46 % 166 | 37,Poland,41026067,2.93 %,1168922,134,306230,-910475,1.5,40,55 %,0.51 % 167 | 93,Portugal,10247605,-0.23 %,-23260,112,91590,9999,1.4,46,67 %,0.13 % 168 | 136,Puerto Rico,3260314,0.24 %,7907,368,8870,19835,1.3,44,N.A.,0.04 % 169 | 143,Qatar,2716391,0.79 %,21269,234,11610,0,1.8,34,N.A.,0.03 % 170 | 64,Romania,19892812,1.19 %,233545,86,230170,-254616,1.7,41,53 %,0.25 % 171 | 9,Russia,144444359,-0.19 %,-268955,9,16376870,-136414,1.5,39,75 %,1.80 % 172 | 76,Rwanda,14094683,2.31 %,317985,571,24670,-8999,3.7,19,18 %,0.18 % 173 | 161,Réunion,981796,0.80 %,7744,393,2500,-630,2.2,34,93 %,0.01 % 174 | 227,Saint Barthelemy,10994,0.25 %,27,524,21,0,1,40,0 %,0.00 % 175 | 229,Saint Helena,5314,-1.12 %,-60,14,390,0,1.6,53,32 %,0.00 % 176 | 209,Saint Kitts & Nevis,47755,0.21 %,98,184,260,20,1.5,35,38 %,0.00 % 177 | 191,Saint Lucia,180251,0.22 %,394,295,610,0,1.4,34,19 %,0.00 % 178 | 218,Saint Martin,32077,0.90 %,286,605,53,0,2.4,39,0 %,0.00 % 179 | 228,Saint Pierre & Miquelon,5840,-0.38 %,-22,25,230,0,1.6,44,N.A.,0.00 % 180 | 189,Samoa,225681,1.48 %,3299,80,2830,-1500,3.8,21,16 %,0.00 % 181 | 216,San Marino,33642,-0.05 %,-18,561,60,100,1.1,47,99 %,0.00 % 182 | 188,Sao Tome & Principe,231856,1.97 %,4476,242,960,-600,3.7,19,77 %,0.00 % 183 | 40,Saudi Arabia,36947025,1.48 %,538205,17,2149690,28998,2.4,31,83 %,0.46 % 184 | 71,Senegal,17763163,2.58 %,446714,92,192530,-19999,4.3,18,52 %,0.22 % 185 | 105,Serbia,7149077,-1.00 %,-72288,82,87460,-9999,1.5,43,69 %,0.09 % 186 | 196,Seychelles,107660,0.51 %,542,234,460,-200,2.3,33,53 %,0.00 % 187 | 102,Sierra Leone,8791092,2.15 %,185374,122,72180,-4000,3.8,19,43 %,0.11 % 188 | 114,Singapore,6014723,0.65 %,39034,8592,700,26998,1,43,N.A.,0.07 % 189 | 211,Sint Maarten,44222,0.11 %,47,1301,34,0,1.6,48,97 %,0.00 % 190 | 116,Slovakia,5795199,2.69 %,151746,121,48088,-112067,1.6,40,51 %,0.07 % 191 | 149,Slovenia,2119675,-0.01 %,-169,105,20140,2000,1.6,44,55 %,0.03 % 192 | 166,Solomon Islands,740424,2.23 %,16151,26,27990,-1600,3.9,19,24 %,0.01 % 193 | 69,Somalia,18143378,3.10 %,545867,29,627340,-30000,6.1,15,46 %,0.23 % 194 | 24,South Africa,60414495,0.87 %,520610,50,1213090,58496,2.3,28,69 %,0.75 % 195 | 29,South Korea,51784059,-0.06 %,-31751,533,97230,29998,0.9,44,82 %,0.64 % 196 | 86,South Sudan,11088796,1.61 %,175632,18,610952,-23291,4.3,17,28 %,0.14 % 197 | 32,Spain,47519628,-0.08 %,-39002,95,498800,39998,1.3,45,80 %,0.59 % 198 | 61,Sri Lanka,21893579,0.28 %,61436,349,62710,-77495,2,33,19 %,0.27 % 199 | 198,St. Vincent & Grenadines,103698,-0.24 %,-250,266,390,-200,1.8,33,58 %,0.00 % 200 | 121,State of Palestine,5371230,2.31 %,121158,892,6020,-5000,3.4,20,83 %,0.07 % 201 | 31,Sudan,48109006,2.63 %,1234802,27,1765048,-9999,4.3,19,35 %,0.60 % 202 | 170,Suriname,623236,0.84 %,5196,4,156000,-1000,2.3,28,63 %,0.01 % 203 | 87,Sweden,10612086,0.59 %,62739,26,410340,39998,1.7,40,86 %,0.13 % 204 | 101,Switzerland,8796669,0.64 %,56197,223,39516,39998,1.5,42,75 %,0.11 % 205 | 60,Syria,23227014,4.98 %,1101765,126,183630,757103,2.7,22,53 %,0.29 % 206 | 57,Taiwan,23923276,0.13 %,29882,676,35410,23999,1.2,42,80 %,0.30 % 207 | 95,Tajikistan,10143543,1.92 %,190756,72,139960,-19999,3.1,22,28 %,0.13 % 208 | 22,Tanzania,67438106,2.96 %,1940358,76,885800,-39997,4.6,17,38 %,0.84 % 209 | 20,Thailand,71801279,0.15 %,104249,141,510890,18999,1.3,40,52 %,0.89 % 210 | 155,Timor-Leste,1360596,1.44 %,19300,91,14870,-5000,3,21,35 %,0.02 % 211 | 99,Togo,9053799,2.32 %,205100,166,54390,-2000,4.1,19,44 %,0.11 % 212 | 233,Tokelau,1893,1.18 %,22,189,10,0,2.6,27,0 %,0.00 % 213 | 195,Tonga,107773,0.86 %,915,150,720,-800,3.2,22,24 %,0.00 % 214 | 153,Trinidad and Tobago,1534937,0.25 %,3893,299,5130,-800,1.6,36,48 %,0.02 % 215 | 79,Tunisia,12458223,0.83 %,102106,80,155360,-4000,2,32,69 %,0.15 % 216 | 18,Turkey,85816199,0.56 %,474958,112,769630,-318067,1.9,32,77 %,1.07 % 217 | 111,Turkmenistan,6516100,1.33 %,85330,14,469930,-4000,2.6,26,52 %,0.08 % 218 | 210,Turks and Caicos,46062,0.79 %,359,48,950,200,1.6,38,79 %,0.00 % 219 | 226,Tuvalu,11396,0.74 %,84,380,30,-60,3.1,25,69 %,0.00 % 220 | 199,U.S. Virgin Islands,98750,-0.72 %,-715,282,350,-450,2.1,43,N.A.,0.00 % 221 | 30,Uganda,48582334,2.82 %,1332749,243,199810,-126181,4.4,16,29 %,0.60 % 222 | 41,Ukraine,36744634,-7.45 %,-2957105,63,579320,1784718,1.3,45,82 %,0.46 % 223 | 96,United Arab Emirates,9516871,0.80 %,75742,114,83600,0,1.4,34,94 %,0.12 % 224 | 21,United Kingdom,67736802,0.34 %,227866,280,241930,165790,1.6,40,85 %,0.84 % 225 | 3,United States,339996563,0.50 %,1706706,37,9147420,999700,1.7,38,83 %,4.23 % 226 | 135,Uruguay,3423108,0.01 %,314,20,175020,-1500,1.5,36,99 %,0.04 % 227 | 43,Uzbekistan,35163944,1.55 %,536292,83,425400,-19999,2.8,27,49 %,0.44 % 228 | 183,Vanuatu,334506,2.38 %,7766,27,12190,0,3.7,20,24 %,0.00 % 229 | 52,Venezuela,28838499,1.90 %,536803,33,882050,321106,2.2,28,N.A.,0.36 % 230 | 16,Vietnam,98858950,0.68 %,672094,319,310070,-82700,1.9,33,40 %,1.23 % 231 | 225,Wallis & Futuna,11502,-0.60 %,-70,82,140,-119,1.9,37,0 %,0.00 % 232 | 172,Western Sahara,587259,1.96 %,11273,2,266000,5600,2.2,32,95 %,0.01 % 233 | 44,Yemen,34449825,2.24 %,753211,65,527970,-29914,3.6,19,37 %,0.43 % 234 | 63,Zambia,20569737,2.76 %,552062,28,743390,-5000,4.2,17,46 %,0.26 % 235 | 74,Zimbabwe,16665409,2.11 %,344872,43,386850,-9999,3.4,18,37 %,0.21 % 236 | -------------------------------------------------------------------------------- /main.py: -------------------------------------------------------------------------------- 1 | from dotenv import load_dotenv 2 | import os 3 | import pandas as pd 4 | from llama_index.experimental.query_engine import PandasQueryEngine 5 | from prompts import new_prompt, instruction_str, coffee_context, context 6 | from note_engine import note_engine 7 | from llama_index.core.tools import QueryEngineTool, ToolMetadata 8 | from llama_index.core.agent import ReActAgent 9 | from llama_index.llms.openai import OpenAI 10 | 11 | from pdf import canada_engine 12 | from coffee_scraper import coffee_scraper 13 | 14 | load_dotenv() 15 | 16 | population_path = os.path.join("data", "population.csv") 17 | population_df = pd.read_csv(population_path) 18 | 19 | population_query_engine = PandasQueryEngine(df=population_df, verbose=True, instruction_str=instruction_str) 20 | population_query_engine.update_prompts({"pandas_prompt": new_prompt}) 21 | 22 | coffee_scraper.scrape_nearby_coffee() 23 | coffee_query_engine = coffee_scraper.get_query_engine() 24 | 25 | tools = [ 26 | note_engine, 27 | QueryEngineTool( 28 | query_engine=population_query_engine, 29 | metadata=ToolMetadata( 30 | name="population_data", 31 | description="this gives information at the world population and demographics", 32 | ), 33 | ), 34 | QueryEngineTool( 35 | query_engine=canada_engine, 36 | metadata=ToolMetadata( 37 | name="canada_data", 38 | description="this gives detailed information about canada the country", 39 | ), 40 | ), 41 | QueryEngineTool( 42 | query_engine=coffee_query_engine, 43 | metadata=ToolMetadata( 44 | name="ottawa_coffee", 45 | description=( 46 | "Coffee shops near National Gallery of Canada. " 47 | "Includes price levels (1-3), ratings (1-5) and distances." 48 | ) 49 | ) 50 | ) 51 | ] 52 | 53 | # llm = OpenAI(model="gpt-3.5-turbo") 54 | llm = OpenAI(model="gpt-4o-mini-2024-07-18") 55 | agent = ReActAgent.from_tools(tools, llm=llm, verbose=True, context=context + "\n" + coffee_context) 56 | 57 | while (prompt := input("Enter a prompt (q to quit): ")) != "q": 58 | result = agent.query(prompt) 59 | print(result) 60 | -------------------------------------------------------------------------------- /note_engine.py: -------------------------------------------------------------------------------- 1 | from llama_index.core.tools import FunctionTool 2 | import os 3 | 4 | note_file = os.path.join("data", "notes.txt") 5 | 6 | 7 | def save_note(note): 8 | if not os.path.exists(note_file): 9 | open(note_file, "w") 10 | 11 | with open(note_file, "a") as f: 12 | f.writelines([note + "\n"]) 13 | 14 | return "note saved" 15 | 16 | 17 | note_engine = FunctionTool.from_defaults( 18 | fn=save_note, 19 | name="note_saver", 20 | description="this tool can save a text based note to a file for the user", 21 | ) 22 | -------------------------------------------------------------------------------- /pdf.py: -------------------------------------------------------------------------------- 1 | import os 2 | from llama_index.core import StorageContext, VectorStoreIndex, load_index_from_storage 3 | from llama_index.readers.file import PDFReader 4 | 5 | 6 | def get_index(data, index_name): 7 | index = None 8 | if not os.path.exists(index_name): 9 | print("building index", index_name) 10 | index = VectorStoreIndex.from_documents(data, show_progress=True) 11 | index.storage_context.persist(persist_dir=index_name) 12 | else: 13 | index = load_index_from_storage( 14 | StorageContext.from_defaults(persist_dir=index_name) 15 | ) 16 | 17 | return index 18 | 19 | 20 | pdf_path = os.path.join("data", "Canada.pdf") 21 | canada_pdf = PDFReader().load_data(file=pdf_path) 22 | canada_index = get_index(canada_pdf, "canada") 23 | canada_engine = canada_index.as_query_engine() 24 | -------------------------------------------------------------------------------- /prompts.py: -------------------------------------------------------------------------------- 1 | from llama_index.core import PromptTemplate 2 | 3 | 4 | instruction_str = """\ 5 | 1. Convert the query to executable Python code using Pandas. 6 | 2. The final line of code should be a Python expression that can be called with the `eval()` function. 7 | 3. The code should represent a solution to the query. 8 | 4. PRINT ONLY THE EXPRESSION. 9 | 5. Do not quote the expression.""" 10 | 11 | new_prompt = PromptTemplate( 12 | """\ 13 | You are working with a pandas dataframe in Python. 14 | The name of the dataframe is `df`. 15 | This is the result of `print(df.head())`: 16 | {df_str} 17 | 18 | Follow these instructions: 19 | {instruction_str} 20 | Query: {query_str} 21 | 22 | Expression: """ 23 | ) 24 | 25 | context = """Purpose: The primary role of this agent is to assist users by providing accurate 26 | information about world population statistics and details about a country. """ 27 | coffee_context = """ 28 | ## Coffee Shop Data Usage Guide 29 | - Data Range: Surrounding the National Gallery of Canada, within a 2 km radius. 30 | - Field Descriptions： 31 | - price_level: 32 | - $: Budget-friendly options, typically low-cost. 33 | - $$: Mid-range pricing. 34 | - $1-10: Price range between $1 and $10, for more specific pricing. 35 | - $20-30: Price range between $20 and $30, for higher-end options. 36 | - [Variable range like $15-20]: Represents a price range with variable amounts, indicating a specific price range. 37 | - rating: Google rating 38 | - distance_km: Distance from the gallery in kilometers. 39 | - Example Queries:： 40 | - 'List coffee shops with rating > 4 within 1km' 41 | - 'What is the average price level of shops within 0.5km?' 42 | - 'Find the closest 3 coffee shops' 43 | """ 44 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | llama-index==0.12.22 2 | llama-index-experimental==0.5.4 3 | pypdf==5.3.1 4 | python-dotenv==1.0.1 5 | pandas==2.2.3 --------------------------------------------------------------------------------