11 |
12 | # Evaluation runs
13 |
14 | The file structure is the same as the original repo.
15 | The only difference is that the `run_browser_use.py` is modified to add the browser use evaluation, we also changed the prompts to be suitable for the browser use evaluation (VERY minimal changes - evaluate multiple images, not just one) and switched to langchain.
16 |
17 | We also have a list of impossible tasks that are not possible anymore (completely outdated, can't be fixed with dates).
18 |
19 | We changed some tasks that included dates to be more in the future instead of the past since the data is outdated which would make the task impossible (e.g. "Please find me a hotel on 2023-12-01 on booking.com", which is impossible since you can't search for a hotel in the past).
20 |
21 | We ran the evaluation on 16gb of RAM with 15 concurrent tasks.
22 |
23 | `requirements.txt` is missing `browser-use` on purpose since we install it by building the package locally.
24 |
25 | ## Manual correction of evaluations
26 |
27 | The eval model is not good. That's why we added another success criteria - `unknown` if the eval model is not sure.
28 |
29 | Most of the tasks are indeed correct, but some tasks had wrong assesment, and `unknown` either went into `success` or `failed`.
30 |
31 | We manually reviewed the evaluations for the tasks that are either "unknown" or "failed" and corrected them. This is due to the fact that the default WebVoyager evaluator is not good.
32 |
33 | ## Costs
34 |
35 | The whole cost of running the dataset once is around 250 USD for gpt4o.
36 |
37 | ## Interesting findings
38 |
39 | - WebVoyager is a terrible dataset.
40 | - a lot of tasks are straight up impossible (outdated usually)
41 | - many prompts are VERY ambiguous and can be interpreted in many ways - which confuses the model (both agent and eval model)
42 | - why do we have to rely on third party websites? That is not scalable.
43 | - the dataset does not test for actual understanding of website, mostly of planning and reasoning - NOT what you want from web agent evaluations on COMPLEX sites
44 |
45 | # Todos
46 |
47 | - make manually labeled items more transparent
48 | - add proxies
49 | - test all kinds of models and different setups (Claude, GPT-4o, Llama 3, etc.)
50 | - test different setups (single vs multiple images, single vs multiple tasks, etc.)
51 |
--------------------------------------------------------------------------------
/evaluation/auto_eval_browser_use.py:
--------------------------------------------------------------------------------
1 | import time
2 | from typing import TYPE_CHECKING
3 |
4 | from browser_use import AgentHistoryList
5 | from langchain_anthropic import ChatAnthropic
6 | from langchain_core.messages import HumanMessage, SystemMessage
7 | from langchain_google_genai import ChatGoogleGenerativeAI
8 | from langchain_openai import AzureChatOpenAI
9 |
10 | if TYPE_CHECKING:
11 | from run_browser_use import EvalResult
12 |
13 |
14 | SYSTEM_PROMPT = """As an evaluator, you will be presented with three primary components to assist you in your role:
15 |
16 | 1. Web Task Instruction: This is a clear and specific directive provided in natural language, detailing the online activity to be carried out. These requirements may include conducting searches, verifying information, comparing prices, checking availability, or any other action relevant to the specified web service (such as Amazon, Apple, ArXiv, BBC News, Booking etc).
17 |
18 | 2. Result Screenshots: This is a visual representation of the screen showing the result or intermediate state of performing a web task. It serves as visual proof of the actions taken in response to the instruction, and may not represent everything the agent sees.
19 |
20 | 3. Result Response: This is a textual response obtained after the execution of the web task. It serves as textual result in response to the instruction.
21 |
22 | -- You DO NOT NEED to interact with web pages or perform actions such as booking flights or conducting searches on websites.
23 | -- You SHOULD NOT make assumptions based on information not presented in the screenshot when comparing it to the instructions. If you cannot find any information in the screenshot that matches the instruction, you can believe the information in the response.
24 | -- Your primary responsibility is to conduct a thorough assessment of the web task instruction against the outcome depicted in the screenshot and in the response, evaluating whether the actions taken align with the given instructions.
25 | -- NOTE that the instruction may involve more than one task, for example, locating the garage and summarizing the review. Failing to complete either task, such as not providing a summary, should be considered unsuccessful.
26 | -- NOTE that the screenshot is authentic, but the response provided by LLM is generated at the end of web browsing, and there may be discrepancies between the text and the screenshots.
27 | -- Note the difference: 1) Result response may contradict the screenshot, then the content of the screenshot prevails, 2) The content in the Result response is not mentioned on the screenshot, choose to believe the content.
28 | -- If you are not sure whether you should believe the content in the response, you should choose unknown.
29 |
30 | You should elaborate on how you arrived at your final evaluation and then provide a definitive verdict on whether the task has been successfully accomplished, either as 'SUCCESS', 'NOT SUCCESS', or 'UNKNOWN'."""
31 |
32 | USER_PROMPT = """TASK:
33 | Result Response:
34 | screenshot at the end: """
35 |
36 |
37 | async def auto_eval_by_gpt4o(
38 | history: AgentHistoryList,
39 | task: str,
40 | openai_client: AzureChatOpenAI | ChatAnthropic | ChatGoogleGenerativeAI,
41 | ) -> tuple["EvalResult", str]:
42 | # print(f"--------------------- {process_dir} ---------------------")
43 |
44 | if not history.is_done():
45 | return "failed", ""
46 |
47 | answer = history.final_result()
48 | if answer is None:
49 | return "failed", ""
50 |
51 | screenshots = history.screenshots()[-4:]
52 | screenshot_content = [
53 | {
54 | "type": "image_url",
55 | "image_url": {"url": f"data:image/png;base64,{screenshot}"},
56 | }
57 | for screenshot in screenshots
58 | ]
59 |
60 | # Prepare GPT-4V messages
61 | user_prompt_tmp = USER_PROMPT.replace("", task)
62 | user_prompt_tmp = user_prompt_tmp.replace("", answer)
63 | user_prompt_tmp = user_prompt_tmp.replace("", str(len(screenshots)))
64 |
65 | messages = [
66 | SystemMessage(content=SYSTEM_PROMPT),
67 | HumanMessage(
68 | content=[
69 | {"type": "text", "text": user_prompt_tmp},
70 | *screenshot_content,
71 | {"type": "text", "text": "Your verdict:\n"},
72 | ]
73 | ),
74 | ]
75 |
76 | while True:
77 | try:
78 | # print("Calling gpt4v API to get the auto evaluation......")
79 | response = await openai_client.ainvoke(messages)
80 | # print("API call complete...")
81 | break
82 | except Exception as e:
83 | print(e)
84 | if type(e).__name__ == "RateLimitError":
85 | time.sleep(10)
86 | elif type(e).__name__ == "APIError":
87 | time.sleep(15)
88 | elif type(e).__name__ == "InvalidRequestError":
89 | exit(0)
90 | else:
91 | time.sleep(10)
92 |
93 | gpt_4v_res = str(response.content)
94 |
95 | if gpt_4v_res is None:
96 | return "unknown", ""
97 | elif "NOT SUCCESS" in gpt_4v_res:
98 | auto_eval_res = "failed"
99 | elif "SUCCESS" in gpt_4v_res:
100 | auto_eval_res = "success"
101 | elif "UNKNOWN" in gpt_4v_res:
102 | auto_eval_res = "unknown"
103 | else:
104 | auto_eval_res = "failed"
105 |
106 | return auto_eval_res, gpt_4v_res
107 |
--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
1 | Apache License
2 | Version 2.0, January 2004
3 | http://www.apache.org/licenses/
4 |
5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
6 |
7 | 1. Definitions.
8 |
9 | "License" shall mean the terms and conditions for use, reproduction,
10 | and distribution as defined by Sections 1 through 9 of this document.
11 |
12 | "Licensor" shall mean the copyright owner or entity authorized by
13 | the copyright owner that is granting the License.
14 |
15 | "Legal Entity" shall mean the union of the acting entity and all
16 | other entities that control, are controlled by, or are under common
17 | control with that entity. For the purposes of this definition,
18 | "control" means (i) the power, direct or indirect, to cause the
19 | direction or management of such entity, whether by contract or
20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the
21 | outstanding shares, or (iii) beneficial ownership of such entity.
22 |
23 | "You" (or "Your") shall mean an individual or Legal Entity
24 | exercising permissions granted by this License.
25 |
26 | "Source" form shall mean the preferred form for making modifications,
27 | including but not limited to software source code, documentation
28 | source, and configuration files.
29 |
30 | "Object" form shall mean any form resulting from mechanical
31 | transformation or translation of a Source form, including but
32 | not limited to compiled object code, generated documentation,
33 | and conversions to other media types.
34 |
35 | "Work" shall mean the work of authorship, whether in Source or
36 | Object form, made available under the License, as indicated by a
37 | copyright notice that is included in or attached to the work
38 | (an example is provided in the Appendix below).
39 |
40 | "Derivative Works" shall mean any work, whether in Source or Object
41 | form, that is based on (or derived from) the Work and for which the
42 | editorial revisions, annotations, elaborations, or other modifications
43 | represent, as a whole, an original work of authorship. For the purposes
44 | of this License, Derivative Works shall not include works that remain
45 | separable from, or merely link (or bind by name) to the interfaces of,
46 | the Work and Derivative Works thereof.
47 |
48 | "Contribution" shall mean any work of authorship, including
49 | the original version of the Work and any modifications or additions
50 | to that Work or Derivative Works thereof, that is intentionally
51 | submitted to Licensor for inclusion in the Work by the copyright owner
52 | or by an individual or Legal Entity authorized to submit on behalf of
53 | the copyright owner. For the purposes of this definition, "submitted"
54 | means any form of electronic, verbal, or written communication sent
55 | to the Licensor or its representatives, including but not limited to
56 | communication on electronic mailing lists, source code control systems,
57 | and issue tracking systems that are managed by, or on behalf of, the
58 | Licensor for the purpose of discussing and improving the Work, but
59 | excluding communication that is conspicuously marked or otherwise
60 | designated in writing by the copyright owner as "Not a Contribution."
61 |
62 | "Contributor" shall mean Licensor and any individual or Legal Entity
63 | on behalf of whom a Contribution has been received by Licensor and
64 | subsequently incorporated within the Work.
65 |
66 | 2. Grant of Copyright License. Subject to the terms and conditions of
67 | this License, each Contributor hereby grants to You a perpetual,
68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable
69 | copyright license to reproduce, prepare Derivative Works of,
70 | publicly display, publicly perform, sublicense, and distribute the
71 | Work and such Derivative Works in Source or Object form.
72 |
73 | 3. Grant of Patent License. Subject to the terms and conditions of
74 | this License, each Contributor hereby grants to You a perpetual,
75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable
76 | (except as stated in this section) patent license to make, have made,
77 | use, offer to sell, sell, import, and otherwise transfer the Work,
78 | where such license applies only to those patent claims licensable
79 | by such Contributor that are necessarily infringed by their
80 | Contribution(s) alone or by combination of their Contribution(s)
81 | with the Work to which such Contribution(s) was submitted. If You
82 | institute patent litigation against any entity (including a
83 | cross-claim or counterclaim in a lawsuit) alleging that the Work
84 | or a Contribution incorporated within the Work constitutes direct
85 | or contributory patent infringement, then any patent licenses
86 | granted to You under this License for that Work shall terminate
87 | as of the date such litigation is filed.
88 |
89 | 4. Redistribution. You may reproduce and distribute copies of the
90 | Work or Derivative Works thereof in any medium, with or without
91 | modifications, and in Source or Object form, provided that You
92 | meet the following conditions:
93 |
94 | (a) You must give any other recipients of the Work or
95 | Derivative Works a copy of this License; and
96 |
97 | (b) You must cause any modified files to carry prominent notices
98 | stating that You changed the files; and
99 |
100 | (c) You must retain, in the Source form of any Derivative Works
101 | that You distribute, all copyright, patent, trademark, and
102 | attribution notices from the Source form of the Work,
103 | excluding those notices that do not pertain to any part of
104 | the Derivative Works; and
105 |
106 | (d) If the Work includes a "NOTICE" text file as part of its
107 | distribution, then any Derivative Works that You distribute must
108 | include a readable copy of the attribution notices contained
109 | within such NOTICE file, excluding those notices that do not
110 | pertain to any part of the Derivative Works, in at least one
111 | of the following places: within a NOTICE text file distributed
112 | as part of the Derivative Works; within the Source form or
113 | documentation, if provided along with the Derivative Works; or,
114 | within a display generated by the Derivative Works, if and
115 | wherever such third-party notices normally appear. The contents
116 | of the NOTICE file are for informational purposes only and
117 | do not modify the License. You may add Your own attribution
118 | notices within Derivative Works that You distribute, alongside
119 | or as an addendum to the NOTICE text from the Work, provided
120 | that such additional attribution notices cannot be construed
121 | as modifying the License.
122 |
123 | You may add Your own copyright statement to Your modifications and
124 | may provide additional or different license terms and conditions
125 | for use, reproduction, or distribution of Your modifications, or
126 | for any such Derivative Works as a whole, provided Your use,
127 | reproduction, and distribution of the Work otherwise complies with
128 | the conditions stated in this License.
129 |
130 | 5. Submission of Contributions. Unless You explicitly state otherwise,
131 | any Contribution intentionally submitted for inclusion in the Work
132 | by You to the Licensor shall be under the terms and conditions of
133 | this License, without any additional terms or conditions.
134 | Notwithstanding the above, nothing herein shall supersede or modify
135 | the terms of any separate license agreement you may have executed
136 | with Licensor regarding such Contributions.
137 |
138 | 6. Trademarks. This License does not grant permission to use the trade
139 | names, trademarks, service marks, or product names of the Licensor,
140 | except as required for reasonable and customary use in describing the
141 | origin of the Work and reproducing the content of the NOTICE file.
142 |
143 | 7. Disclaimer of Warranty. Unless required by applicable law or
144 | agreed to in writing, Licensor provides the Work (and each
145 | Contributor provides its Contributions) on an "AS IS" BASIS,
146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
147 | implied, including, without limitation, any warranties or conditions
148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
149 | PARTICULAR PURPOSE. You are solely responsible for determining the
150 | appropriateness of using or redistributing the Work and assume any
151 | risks associated with Your exercise of permissions under this License.
152 |
153 | 8. Limitation of Liability. In no event and under no legal theory,
154 | whether in tort (including negligence), contract, or otherwise,
155 | unless required by applicable law (such as deliberate and grossly
156 | negligent acts) or agreed to in writing, shall any Contributor be
157 | liable to You for damages, including any direct, indirect, special,
158 | incidental, or consequential damages of any character arising as a
159 | result of this License or out of the use or inability to use the
160 | Work (including but not limited to damages for loss of goodwill,
161 | work stoppage, computer failure or malfunction, or any and all
162 | other commercial damages or losses), even if such Contributor
163 | has been advised of the possibility of such damages.
164 |
165 | 9. Accepting Warranty or Additional Liability. While redistributing
166 | the Work or Derivative Works thereof, You may choose to offer,
167 | and charge a fee for, acceptance of support, warranty, indemnity,
168 | or other liability obligations and/or rights consistent with this
169 | License. However, in accepting such obligations, You may act only
170 | on Your own behalf and on Your sole responsibility, not on behalf
171 | of any other Contributor, and only if You agree to indemnify,
172 | defend, and hold each Contributor harmless for any liability
173 | incurred by, or claims asserted against, such Contributor by reason
174 | of your accepting any such warranty or additional liability.
175 |
176 | END OF TERMS AND CONDITIONS
177 |
178 | APPENDIX: How to apply the Apache License to your work.
179 |
180 | To apply the Apache License to your work, attach the following
181 | boilerplate notice, with the fields enclosed by brackets "[]"
182 | replaced with your own identifying information. (Don't include
183 | the brackets!) The text should be enclosed in the appropriate
184 | comment syntax for the file format. We also recommend that a
185 | file or class name and description of purpose be included on the
186 | same "printed page" as the copyright notice for easier
187 | identification within third-party archives.
188 |
189 | Copyright [yyyy] [name of copyright owner]
190 |
191 | Licensed under the Apache License, Version 2.0 (the "License");
192 | you may not use this file except in compliance with the License.
193 | You may obtain a copy of the License at
194 |
195 | http://www.apache.org/licenses/LICENSE-2.0
196 |
197 | Unless required by applicable law or agreed to in writing, software
198 | distributed under the License is distributed on an "AS IS" BASIS,
199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
200 | See the License for the specific language governing permissions and
201 | limitations under the License.
--------------------------------------------------------------------------------
/run_browser_use.py:
--------------------------------------------------------------------------------
1 | import argparse
2 | import asyncio
3 | import json
4 | import logging
5 | import os
6 | import random
7 | import shutil
8 | from asyncio import Semaphore
9 | from dataclasses import dataclass, field
10 | from datetime import datetime
11 | from pathlib import Path
12 | from typing import Generator, List, Literal, Set, TypedDict
13 |
14 | from browser_use import Agent, Browser, BrowserConfig
15 | from browser_use.browser.context import BrowserContextConfig
16 | from dotenv import load_dotenv
17 | from langchain_anthropic import ChatAnthropic
18 | from langchain_google_genai import ChatGoogleGenerativeAI
19 | from langchain_openai import AzureChatOpenAI
20 | from pydantic import BaseModel, Field, SecretStr
21 |
22 | from evaluation.auto_eval_browser_use import auto_eval_by_gpt4o
23 |
24 | load_dotenv()
25 |
26 |
27 | class TaskData(TypedDict):
28 | id: str
29 | web: str
30 | ques: str
31 |
32 |
33 | EvalResult = Literal["success", "failed", "unknown"]
34 |
35 |
36 | @dataclass
37 | class RunStats:
38 | total_tasks: int
39 | current_task: int = 0
40 | successful_tasks: Set[str] = field(default_factory=set)
41 | failed_tasks: Set[str] = field(default_factory=set)
42 | unknown_tasks: Set[str] = field(default_factory=set)
43 |
44 | def update(self, task_id: str, success: "EvalResult") -> None:
45 | if success == "success":
46 | self.successful_tasks.add(task_id)
47 | elif success == "failed":
48 | self.failed_tasks.add(task_id)
49 | else:
50 | self.unknown_tasks.add(task_id)
51 |
52 | def get_success_rate(self) -> str:
53 | if self.current_task == 0:
54 | return "0/0=0.00"
55 | return f"{len(self.successful_tasks)}/{self.current_task}={len(self.successful_tasks) / self.current_task:.2f}"
56 |
57 | def print_periodic_summary(self) -> None:
58 | print("\n=== Task Summary ===")
59 | print(
60 | f"Successful tasks ({len(self.successful_tasks)}): {sorted(list(self.successful_tasks))}"
61 | )
62 | print(
63 | f"Failed tasks ({len(self.failed_tasks)}): {sorted(list(self.failed_tasks))}"
64 | )
65 | print(f"Current success rate: {self.get_success_rate()}")
66 | print("==================\n")
67 |
68 |
69 | class TaskResult(BaseModel):
70 | task_id: str
71 | web_name: str
72 | start_time: datetime
73 | end_time: datetime
74 | duration_seconds: float
75 | num_steps: int
76 | success: EvalResult
77 | task_prompt: str
78 | final_answer: str
79 | gpt_4v_res: str
80 |
81 |
82 | class ExperimentResults(BaseModel):
83 | total_tasks: int = 0
84 | total_success: int = 0
85 | total_failed: int = 0
86 | total_unknown: int = 0
87 | all_tasks: List[TaskResult] = Field(default_factory=list)
88 |
89 |
90 | def cleanup_webdriver_cache() -> None:
91 | """Clean up webdriver cache directories."""
92 | cache_paths = [
93 | Path.home() / ".wdm",
94 | Path.home() / ".cache" / "selenium",
95 | Path.home() / "Library" / "Caches" / "selenium",
96 | ]
97 | for path in cache_paths:
98 | if path.exists():
99 | print(f"Removing cache directory: {path}")
100 | shutil.rmtree(path, ignore_errors=True)
101 |
102 |
103 | def create_task_result(
104 | task: TaskData,
105 | start_time: datetime,
106 | eval_result: EvalResult,
107 | num_steps: int,
108 | final_answer: str,
109 | gpt_4v_res: str,
110 | ) -> TaskResult:
111 | """Create task result object."""
112 | end_time = datetime.now()
113 | return TaskResult(
114 | task_id=task["id"],
115 | web_name=task["web"],
116 | start_time=start_time,
117 | end_time=end_time,
118 | duration_seconds=(end_time - start_time).total_seconds(),
119 | num_steps=num_steps,
120 | success=eval_result,
121 | task_prompt=f"{task['ques']} on {task['web']}",
122 | final_answer=final_answer,
123 | gpt_4v_res=gpt_4v_res,
124 | )
125 |
126 |
127 | def save_results(
128 | task_result: TaskResult,
129 | task_dir: Path,
130 | ) -> None:
131 | """Save results to files."""
132 | # Save interaction messages
133 |
134 | with open(task_dir / "task_result.json", "w") as f:
135 | json.dump(task_result.model_dump(), f, indent=2, default=str)
136 |
137 |
138 | def print_task_progress(
139 | task_id: str, steps: int, success: EvalResult, stats: RunStats
140 | ) -> None:
141 | """Print concise task progress."""
142 | status = "✓" if success == "success" else "✗" if success == "failed" else "?"
143 | print(
144 | f"Task {task_id} [{stats.current_task}/{stats.total_tasks}] "
145 | f"Steps: {steps} Status: {status} Score: {stats.get_success_rate()}"
146 | )
147 |
148 |
149 | def save_experiment_results(experiment_results: ExperimentResults) -> None:
150 | """Save experiment results to file."""
151 | with open("results/examples-browser-use/experiment_results.json", "w") as f:
152 | json.dump(experiment_results.model_dump(), f, indent=2, default=str)
153 |
154 |
155 | @dataclass
156 | class LLMModel:
157 | model: AzureChatOpenAI
158 | token_limit: int
159 |
160 |
161 | def get_llm_model_generator(
162 | model_provider: str,
163 | ) -> Generator[AzureChatOpenAI | ChatAnthropic, None, None]:
164 | """Generator that creates fresh model instances each time"""
165 | while True:
166 | # Force reload environment variables
167 | load_dotenv(override=True)
168 |
169 | if model_provider == "anthropic":
170 | # Create fresh instances each time, reading current env vars
171 | yield ChatAnthropic(
172 | model_name="claude-3-5-sonnet-20240620",
173 | timeout=25,
174 | stop=None,
175 | temperature=0.0,
176 | )
177 |
178 | elif model_provider == "azure":
179 | # Create fresh instances each time, reading current env vars
180 | west_eu = LLMModel(
181 | model=AzureChatOpenAI(
182 | model="gpt-4o",
183 | api_version="2024-10-21",
184 | azure_endpoint=os.getenv("AZURE_OPENAI_ENDPOINT_WEST_EU", ""),
185 | api_key=SecretStr(os.getenv("AZURE_OPENAI_API_KEY_WEST_EU", "")),
186 | ),
187 | token_limit=900,
188 | )
189 | east_us = LLMModel(
190 | model=AzureChatOpenAI(
191 | model="gpt-4o",
192 | api_version="2024-10-21",
193 | azure_endpoint=os.getenv("AZURE_OPENAI_ENDPOINT_EAST_US", ""),
194 | api_key=SecretStr(os.getenv("AZURE_OPENAI_API_KEY_EAST_US", "")),
195 | ),
196 | token_limit=450,
197 | )
198 | east_us_2 = LLMModel(
199 | model=AzureChatOpenAI(
200 | model="gpt-4o",
201 | api_version="2024-10-21",
202 | azure_endpoint=os.getenv("AZURE_OPENAI_ENDPOINT_EAST_US_2", ""),
203 | api_key=SecretStr(os.getenv("AZURE_OPENAI_API_KEY_EAST_US_2", "")),
204 | ),
205 | token_limit=450,
206 | )
207 | west_us = LLMModel(
208 | model=AzureChatOpenAI(
209 | model="gpt-4o",
210 | api_version="2024-10-21",
211 | azure_endpoint=os.getenv("AZURE_OPENAI_ENDPOINT_WEST_US", ""),
212 | api_key=SecretStr(os.getenv("AZURE_OPENAI_API_KEY_WEST_US", "")),
213 | ),
214 | token_limit=450,
215 | )
216 |
217 | # Yield fresh instances in the same pattern
218 | yield west_eu.model # First 900
219 | yield west_eu.model # Second 900
220 | yield east_us.model # 450
221 | yield east_us_2.model # 450
222 | yield west_us.model # 450
223 | elif model_provider == "google/gemini-1.5-flash":
224 | llm = ChatGoogleGenerativeAI(
225 | model="gemini-1.5-flash",
226 | )
227 | yield llm
228 | elif model_provider == "google/gemini-1.5-pro":
229 | llm = ChatGoogleGenerativeAI(
230 | model="gemini-1.5-pro",
231 | )
232 | yield llm
233 | elif model_provider == "google/gemini-1.5-flash-8b":
234 | llm = ChatGoogleGenerativeAI(model="gemini-1.5-flash-8b")
235 | yield llm
236 | else:
237 | raise ValueError(f"Invalid model provider: {model_provider}")
238 |
239 |
240 | async def process_single_task(
241 | task: TaskData,
242 | client: AzureChatOpenAI | ChatAnthropic,
243 | stats: RunStats,
244 | results_dir: Path,
245 | experiment_results: ExperimentResults,
246 | browser: Browser,
247 | ) -> None:
248 | """Process a single task asynchronously."""
249 | task_str = f"{task['ques']} on {task['web']}"
250 | start_time = datetime.now()
251 | task_dir = results_dir / f"{task['id']}"
252 | task_dir.mkdir(exist_ok=True)
253 |
254 | try:
255 | if not (task_dir / "task_result.json").exists():
256 | logging.getLogger("browser_use").setLevel(logging.INFO)
257 |
258 | agent = Agent(
259 | task=task_str,
260 | llm=client,
261 | browser=browser,
262 | validate_output=True,
263 | generate_gif=False,
264 | )
265 |
266 | history = await agent.run(max_steps=30)
267 | history.save_to_file(task_dir / "history.json")
268 |
269 | eval_result, gpt_4v_res = await auto_eval_by_gpt4o(
270 | task=task_str,
271 | openai_client=client,
272 | history=history,
273 | )
274 |
275 | task_result = create_task_result(
276 | task,
277 | start_time,
278 | eval_result,
279 | len(history.history),
280 | history.final_result() or "",
281 | gpt_4v_res,
282 | )
283 | save_results(task_result, task_dir)
284 | else:
285 | task_result = TaskResult(**json.load(open(task_dir / "task_result.json")))
286 | eval_result = task_result.success
287 |
288 | stats.update(task["id"], eval_result)
289 | print_task_progress(task["id"], task_result.num_steps, eval_result, stats)
290 |
291 | # Update experiment results
292 | experiment_results.all_tasks.append(task_result)
293 | experiment_results.total_tasks += 1
294 | experiment_results.total_success += int(eval_result == "success")
295 | experiment_results.total_failed += int(eval_result == "failed")
296 | experiment_results.total_unknown += int(eval_result == "unknown")
297 | # save curent stats to file
298 | print(f"Saving stats to file {stats.current_task} {stats.get_success_rate()}")
299 | # with open(file="results/examples-browser-use/aaa_stats.txt", mode="a") as f:
300 | # # in one line
301 | # f.write(f"{stats.current_task}\n")
302 | # f.write(f"{stats.get_success_rate()}\n")
303 |
304 | except Exception as e:
305 | logging.error(f"Error processing task {task['id']}: {str(e)}")
306 | stats.update(task["id"], "failed") # Mark as failed instead of crashing
307 | return
308 |
309 | finally:
310 | await browser.close()
311 |
312 |
313 | async def main(max_concurrent_tasks: int, model_provider: str) -> None:
314 | try:
315 | # Setup
316 | cleanup_webdriver_cache()
317 | semaphore = Semaphore(max_concurrent_tasks)
318 |
319 | # Load tasks
320 | tasks: List[TaskData] = []
321 | with open("data/WebVoyager_data.jsonl", "r") as f:
322 | for line in f:
323 | tasks.append(json.loads(line))
324 |
325 | # remove impossible tasks
326 | with open("data/WebVoyagerImpossibleTasks.json", "r") as f:
327 | impossible_tasks = set(json.load(f))
328 | tasks = [task for task in tasks if task["id"] not in impossible_tasks]
329 |
330 | # randomize the order of tasks
331 | random.seed(42)
332 | random.shuffle(tasks)
333 |
334 | # Initialize
335 |
336 | experiment_results = ExperimentResults()
337 | stats = RunStats(total_tasks=len(tasks))
338 | results_dir = Path("results/examples-browser-use")
339 | results_dir.mkdir(parents=True, exist_ok=True)
340 |
341 | # Process tasks concurrently with semaphore
342 | async def process_with_semaphore(
343 | task: TaskData, client: AzureChatOpenAI | ChatAnthropic
344 | ) -> None:
345 | async with semaphore:
346 | print(f"\n=== Now at task {task['id']} ===")
347 |
348 | # Create browser instance inside the semaphore block
349 | browser = Browser(
350 | config=BrowserConfig(
351 | headless=True,
352 | disable_security=True,
353 | new_context_config=BrowserContextConfig(
354 | disable_security=True,
355 | wait_for_network_idle_page_load_time=5,
356 | maximum_wait_page_load_time=20,
357 | # no_viewport=True,
358 | browser_window_size={
359 | "width": 1280,
360 | "height": 1100,
361 | },
362 | # trace_path=str(results_dir / f"{task['id']}"),
363 | ),
364 | )
365 | )
366 |
367 | await process_single_task(
368 | task,
369 | client,
370 | stats,
371 | results_dir,
372 | experiment_results,
373 | browser, # Pass browser instance
374 | )
375 | stats.current_task += 1
376 |
377 | # Add this to ensure browser is always closed
378 | try:
379 | await browser.close()
380 | except Exception as e:
381 | logging.error(f"Error closing browser: {e}")
382 |
383 | print(f"Current task: {stats.current_task}")
384 | print(f"Total tasks: {stats.total_tasks}")
385 | print(f"Success rate: {stats.get_success_rate()}")
386 | # if stats.current_task % max_concurrent_tasks == 0:
387 | stats.print_periodic_summary()
388 | save_experiment_results(experiment_results)
389 |
390 | # Create and run all tasks
391 | all_tasks = []
392 | for i, task in enumerate(tasks):
393 | model = next(get_llm_model_generator(model_provider))
394 | all_tasks.append(process_with_semaphore(task, model))
395 |
396 | # Add timeout and better error handling
397 | await asyncio.gather(*all_tasks, return_exceptions=True)
398 | except Exception as e:
399 | logging.error(f"Main loop error: {e}")
400 | finally:
401 | # Cleanup code here
402 | logging.info("Shutting down...")
403 | stats.print_periodic_summary()
404 |
405 |
406 | if __name__ == "__main__":
407 | try:
408 | parser = argparse.ArgumentParser(
409 | description="Run browser tasks with concurrent execution"
410 | )
411 | parser.add_argument(
412 | "--max-concurrent",
413 | type=int,
414 | default=3,
415 | help="Maximum number of concurrent tasks (default: 3)",
416 | )
417 | parser.add_argument(
418 | "--model-provider",
419 | type=str,
420 | default="azure",
421 | help="Model provider (default: azure)",
422 | choices=[
423 | "azure",
424 | "anthropic",
425 | "google/gemini-1.5-flash",
426 | "google/gemini-1.5-flash-8b",
427 | "google/gemini-1.5-pro",
428 | ],
429 | )
430 | args = parser.parse_args()
431 |
432 | logging.info(f"Running with {args.max_concurrent} concurrent tasks")
433 |
434 | asyncio.run(main(args.max_concurrent, args.model_provider))
435 | except KeyboardInterrupt:
436 | print("\nReceived keyboard interrupt, shutting down...")
437 | except Exception as e:
438 | print(f"Fatal error: {e}")
439 | logging.exception("Fatal error occurred")
440 |
--------------------------------------------------------------------------------
/data/GAIA_web.jsonl:
--------------------------------------------------------------------------------
1 | {"task_id": "e1fc63a2-da7a-432f-be78-7c4a95598703", "Level": 1, "Final answer": "17", "id": "level1-0", "web": "https://www.google.com/", "ques": "If Eliud Kipchoge could maintain his record-making marathon pace indefinitely, how many thousand hours would it take him to run the distance between the Earth and the Moon its closest approach? Please use the minimum perigee value on the Wikipedia page for the Moon when carrying out your calculation. Round your result to the nearest 1000 hours and do not use any comma separators if necessary."}
2 | {"task_id": "8e867cd7-cff9-4e6c-867a-ff5ddc2550be", "Level": 1, "Final answer": "3", "id": "level1-1", "web": "https://www.google.com/", "ques": "How many studio albums were published by Mercedes Sosa between 2000 and 2009 (included)? You can use the latest 2022 version of english wikipedia."}
3 | {"task_id": "5d0080cb-90d7-4712-bc33-848150e917d3", "Level": 1, "Final answer": "0.1777", "id": "level1-2", "web": "https://www.google.com/", "ques": "What was the volume in m^3 of the fish bag that was calculated in the University of Leicester paper \"Can Hiccup Supply Enough Fish to Maintain a Dragon\u2019s Diet?\""}
4 | {"task_id": "a1e91b78-d3d8-4675-bb8d-62741b4b68a6", "Level": 1, "Final answer": "3", "id": "level1-3", "web": "https://www.google.com/", "ques": "In the video https://www.youtube.com/watch?v=L1vXCYZAYYM, what is the highest number of bird species to be on camera simultaneously?"}
5 | {"task_id": "46719c30-f4c3-4cad-be07-d5cb21eee6bb", "Level": 1, "Final answer": "Mapping Human Oriented Information to Software Agents for Online Systems Usage", "id": "level1-4", "web": "https://www.google.com/", "ques": "Of the authors (First M. Last) that worked on the paper \"Pie Menus or Linear Menus, Which Is Better?\" in 2015, what was the title of the first paper authored by the one that had authored prior papers?"}
6 | {"task_id": "4b6bb5f7-f634-410e-815d-e673ab7f8632", "Level": 1, "Final answer": "THE CASTLE", "id": "level1-5", "web": "https://www.google.com/", "ques": "In Series 9, Episode 11 of Doctor Who, the Doctor is trapped inside an ever-shifting maze. What is this location called in the official script for the episode? Give the setting exactly as it appears in the first scene heading."}
7 | {"task_id": "b816bfce-3d80-4913-a07d-69b752ce6377", "Level": 1, "Final answer": "fluffy", "id": "level1-6", "web": "https://www.google.com/", "ques": "In Emily Midkiff's June 2014 article in a journal named for the one of Hreidmar's sons that guarded his house, what word was quoted from two different authors in distaste for the nature of dragon depictions?"}
8 | {"task_id": "72e110e7-464c-453c-a309-90a95aed6538", "Level": 1, "Final answer": "Guatemala", "id": "level1-7", "web": "https://www.google.com/", "ques": "Under DDC 633 on Bielefeld University Library's BASE, as of 2020, from what country was the unknown language article with a flag unique from the others?"}
9 | {"task_id": "b415aba4-4b68-4fc6-9b89-2c812e55a3e1", "Level": 1, "Final answer": "diamond", "id": "level1-8", "web": "https://www.google.com/", "ques": "In Nature journal's Scientific Reports conference proceedings from 2012, in the article that did not mention plasmons or plasmonics, what nano-compound is studied? Don't use the prefix nano in your answer if there is one."}
10 | {"task_id": "935e2cff-ae78-4218-b3f5-115589b19dae", "Level": 1, "Final answer": "research", "id": "level1-9", "web": "https://www.google.com/", "ques": "In the year 2022, and before December, what does \"R\" stand for in the three core policies of the type of content that was violated in the public logs on the Legume Wikipedia page?"}
11 | {"task_id": "4fc2f1ae-8625-45b5-ab34-ad4433bc21f8", "Level": 1, "Final answer": "FunkMonk", "id": "level1-10", "web": "https://www.google.com/", "ques": "Who nominated the only Featured Article on English Wikipedia about a dinosaur that was promoted in November 2016?"}
12 | {"task_id": "5188369a-3bbe-43d8-8b94-11558f909a08", "Level": 1, "Final answer": "Annie Levin", "id": "level1-11", "web": "https://www.google.com/", "ques": "What writer is quoted by Merriam-Webster for the Word of the Day from June 27, 2022?"}
13 | {"task_id": "9d191bce-651d-4746-be2d-7ef8ecadb9c2", "Level": 1, "Final answer": "Extremely", "id": "level1-12", "web": "https://www.google.com/", "ques": "Examine the video at https://www.youtube.com/watch?v=1htKBjuUWec.\n\nWhat does Teal'c say in response to the question \"Isn't that hot?\""}
14 | {"task_id": "cabe07ed-9eca-40ea-8ead-410ef5e83f91", "Level": 1, "Final answer": "Louvrier", "id": "level1-13", "web": "https://www.google.com/", "ques": "What is the surname of the equine veterinarian mentioned in 1.E Exercises from the chemistry materials licensed by Marisa Alviar-Agnew & Henry Agnew under the CK-12 license in LibreText's Introductory Chemistry materials as compiled 08/21/2023?"}
15 | {"task_id": "d0633230-7067-47a9-9dbf-ee11e0a2cdd6", "Level": 1, "Final answer": "BaseLabelPropagation", "id": "level1-14", "web": "https://www.google.com/", "ques": "In the Scikit-Learn July 2017 changelog, what other predictor base command received a bug fix? Just give the name, not a path."}
16 | {"task_id": "0383a3ee-47a7-41a4-b493-519bdefe0488", "Level": 1, "Final answer": "Rockhopper penguin", "id": "level1-15", "web": "https://www.google.com/", "ques": "On the BBC Earth YouTube video of the Top 5 Silliest Animal Moments, what species of bird is featured?"}
17 | {"task_id": "11af4e1a-5f45-467d-9aeb-46f4bb0bf034", "Level": 1, "Final answer": "6", "id": "level1-16", "web": "https://www.google.com/", "ques": "How many more blocks (also denoted as layers) in BERT base encoder than the encoder from the architecture proposed in Attention is All You Need?"}
18 | {"task_id": "7673d772-ef80-4f0f-a602-1bf4485c9b43", "Level": 1, "Final answer": "inference", "id": "level1-17", "web": "https://www.google.com/", "ques": "On Cornell Law School website's legal information institute, under the fifth section of federal rules alphabetically, what word was deleted in the last amendment to the first rule in the article that has \"witnesses\" in the most titles as of 2021?"}
19 | {"task_id": "c365c1c7-a3db-4d5e-a9a1-66f56eae7865", "Level": 1, "Final answer": "Braintree, Honolulu", "id": "level1-18", "web": "https://www.google.com/", "ques": "Of the cities within the United States where U.S. presidents were born, which two are the farthest apart from the westernmost to the easternmost going east, giving the city names only? Give them to me in alphabetical order, in a comma-separated list"}
20 | {"task_id": "7d4a7d1d-cac6-44a8-96e8-ea9584a70825", "Level": 1, "Final answer": "22", "id": "level1-19", "web": "https://www.google.com/", "ques": "According to Girls Who Code, how long did it take in years for the percentage of computer scientists that were women to change by 13% from a starting point of 37%?"}
21 | {"task_id": "dc22a632-937f-4e6a-b72f-ba0ff3f5ff97", "Level": 1, "Final answer": "Five Hundred Things To Eat Before It's Too Late: and the Very Best Places to Eat Them", "id": "level1-20", "web": "https://www.google.com/", "ques": "What was the complete title of the book in which two James Beard Award winners recommended the restaurant where Ali Khan enjoyed a New Mexican staple in his cost-conscious TV show that started in 2015? Write the numbers in plain text if there are some in the title."}
22 | {"task_id": "3f57289b-8c60-48be-bd80-01f8099ca449", "Level": 1, "Final answer": "519", "id": "level1-21", "web": "https://www.google.com/", "ques": "How many at bats did the Yankee with the most walks in the 1977 regular season have that same season?"}
23 | {"task_id": "23dd907f-1261-4488-b21c-e9185af91d5e", "Level": 1, "Final answer": "2", "id": "level1-22", "web": "https://www.google.com/", "ques": "In Audre Lorde\u2019s poem \u201cFather Son and Holy Ghost\u201d, what is the number of the stanza in which some lines are indented?"}
24 | {"task_id": "840bfca7-4f7b-481a-8794-c560c340185d", "Level": 1, "Final answer": "80GSFC21M0002", "id": "level1-23", "web": "https://www.google.com/", "ques": "On June 6, 2023, an article by Carolyn Collins Petersen was published in Universe Today. This article mentions a team that produced a paper about their observations, linked at the bottom of the article. Find this paper. Under what NASA award number was the work performed by R. G. Arendt supported by?"}
25 | {"task_id": "a0068077-79f4-461a-adfe-75c1a4148545", "Level": 1, "Final answer": "90", "id": "level1-24", "web": "https://www.google.com/", "ques": "What was the actual enrollment count of the clinical trial on H. pylori in acne vulgaris patients from Jan-May 2018 as listed on the NIH website?"}
26 | {"task_id": "bda648d7-d618-4883-88f4-3466eabd860e", "Level": 1, "Final answer": "Saint Petersburg", "id": "level1-25", "web": "https://www.google.com/", "ques": "Where were the Vietnamese specimens described by Kuznetzov in Nedoshivina's 2010 paper eventually deposited? Just give me the city name without abbreviations."}
27 | {"task_id": "c61d22de-5f6c-4958-a7f6-5e9707bd3466", "Level": 2, "Final answer": "egalitarian", "id": "level2-0", "web": "https://www.google.com/", "ques": "A paper about AI regulation that was originally submitted to arXiv.org in June 2022 shows a figure with three axes, where each axis has a label word at both ends. Which of these words is used to describe a type of society in a Physics and Society article submitted to arXiv.org on August 11, 2016?"}
28 | {"task_id": "17b5a6a3-bc87-42e8-b0fb-6ab0781ef2cc", "Level": 2, "Final answer": "34689", "id": "level2-1", "web": "https://www.google.com/", "ques": "I\u2019m researching species that became invasive after people who kept them as pets released them. There\u2019s a certain species of fish that was popularized as a pet by being the main character of the movie Finding Nemo. According to the USGS, where was this fish found as a nonnative species, before the year 2020? I need the answer formatted as the five-digit zip codes of the places the species was found, separated by commas if there is more than one place."}
29 | {"task_id": "04a04a9b-226c-43fd-b319-d5e89743676f", "Level": 2, "Final answer": "41", "id": "level2-2", "web": "https://www.google.com/", "ques": "If we assume all articles published by Nature in 2020 (articles, only, not book reviews/columns, etc) relied on statistical significance to justify their findings and they on average came to a p-value of 0.04, how many papers would be incorrect as to their claims of statistical significance? Round the value up to the next integer."}
30 | {"task_id": "14569e28-c88c-43e4-8c32-097d35b9a67d", "Level": 2, "Final answer": "backtick", "id": "level2-3", "web": "https://www.google.com/", "ques": "In Unlambda, what exact charcter or text needs to be added to correct the following code to output \"For penguins\"? If what is needed is a character, answer with the name of the character. If there are different names for the character, use the shortest. The text location is not needed. Code:\n\n`r```````````.F.o.r. .p.e.n.g.u.i.n.si"}
31 | {"task_id": "3627a8be-a77f-41bb-b807-7e1bd4c0ebdf", "Level": 2, "Final answer": "142", "id": "level2-4", "web": "https://www.google.com/", "ques": "The object in the British Museum's collection with a museum number of 2012,5015.17 is the shell of a particular mollusk species. According to the abstract of a research article published in Science Advances in 2021, beads made from the shells of this species were found that are at least how many thousands of years old?"}
32 | {"task_id": "7619a514-5fa8-43ef-9143-83b66a43d7a4", "Level": 2, "Final answer": "04/15/18", "id": "level2-5", "web": "https://www.google.com/", "ques": "According to github, when was Regression added to the oldest closed numpy.polynomial issue that has the Regression label in MM/DD/YY?"}
33 | {"task_id": "2a649bb1-795f-4a01-b3be-9a01868dae73", "Level": 2, "Final answer": "3.1.3.1; 1.11.1.7", "id": "level2-6", "web": "https://www.google.com/", "ques": "What are the EC numbers of the two most commonly used chemicals for the virus testing method in the paper about SPFMV and SPCSV in the Pearl Of Africa from 2016? Return the semicolon-separated numbers in the order of the alphabetized chemicals."}
34 | {"task_id": "87c610df-bef7-4932-b950-1d83ef4e282b", "Level": 2, "Final answer": "Morarji Desai", "id": "level2-7", "web": "https://www.google.com/", "ques": "In April of 1977, who was the Prime Minister of the first place mentioned by name in the Book of Esther (in the New International Version)?"}
35 | {"task_id": "624cbf11-6a41-4692-af9c-36b3e5ca3130", "Level": 2, "Final answer": "So we had to let it die.", "id": "level2-8", "web": "https://www.google.com/", "ques": "What's the last line of the rhyme under the flavor name on the headstone visible in the background of the photo of the oldest flavor's headstone in the Ben & Jerry's online flavor graveyard as of the end of 2022?"}
36 | {"task_id": "dd3c7503-f62a-4bd0-9f67-1b63b94194cc", "Level": 2, "Final answer": "6", "id": "level2-9", "web": "https://www.google.com/", "ques": "Use density measures from the chemistry materials licensed by Marisa Alviar-Agnew & Henry Agnew under the CK-12 license in LibreText's Introductory Chemistry materials as compiled 08/21/2023.\n\nI have a gallon of honey and a gallon of mayonnaise at 25C. I remove one cup of honey at a time from the gallon of honey. How many times will I need to remove a cup to have the honey weigh less than the mayonaise? Assume the containers themselves weigh the same."}
37 | {"task_id": "f0f46385-fc03-4599-b5d3-f56496c3e69f", "Level": 2, "Final answer": "Indonesia, Myanmar", "id": "level2-10", "web": "https://www.google.com/", "ques": "In terms of geographical distance between capital cities, which 2 countries are the furthest from each other within the ASEAN bloc according to wikipedia? Answer using a comma separated list, ordering the countries by alphabetical order."}
38 | {"task_id": "e4e91f1c-1dcd-439e-9fdd-cb976f5293fd", "Level": 2, "Final answer": "cloak", "id": "level2-11", "web": "https://www.google.com/", "ques": "I need to fact-check a citation. This is the citation from the bibliography:\n\nGreetham, David. \"Uncoupled: OR, How I Lost My Author(s).\" Textual Cultures: Texts, Contexts, Interpretation, vol. 3 no. 1, 2008, p. 45-46. Project MUSE, doi:10.2979/tex.2008.3.1.44.\n\nAnd this is the in-line citation:\n\nOur relationship with the authors of the works we read can often be \u201cobscured not by a \"cloak of print\" but by the veil of scribal confusion and mis-transmission\u201d (Greetham 45-46).\n\nDoes the quoted text match what is actually in the article? If Yes, answer Yes, otherwise, give me the word in my citation that does not match with the correct one (without any article)."}
39 | {"task_id": "56137764-b4e0-45b8-9c52-1866420c3df5", "Level": 2, "Final answer": "Li Peng", "id": "level2-12", "web": "https://www.google.com/", "ques": "Which contributor to the version of OpenCV where support was added for the Mask-RCNN model has the same name as a former Chinese head of government when the names are transliterated to the Latin alphabet?"}
40 | {"task_id": "8b3379c0-0981-4f5b-8407-6444610cb212", "Level": 2, "Final answer": "1.8", "id": "level2-13", "web": "https://www.google.com/", "ques": "What is the maximum length in meters of #9 in the first National Geographic short on YouTube that was ever released according to the Monterey Bay Aquarium website? Just give the number."}
41 | {"task_id": "0ff53813-3367-4f43-bcbd-3fd725c1bf4b", "Level": 2, "Final answer": "beta geometric", "id": "level2-14", "web": "https://www.google.com/", "ques": "What two-word type of model did Manash Pratim Kashyap's and PS Fader's studies in customer retention studies published during 2018-2019 have in common (no punctuation)?"}
42 | {"task_id": "a7feb290-76bb-4cb7-8800-7edaf7954f2f", "Level": 2, "Final answer": "31", "id": "level2-15", "web": "https://www.google.com/", "ques": "How many High Energy Physics - Lattice articles listed in January 2020 on Arxiv had ps versions available?"}
43 | {"task_id": "b4cc024b-3f5e-480e-b96a-6656493255b5", "Level": 2, "Final answer": "Russian-German Legion", "id": "level2-16", "web": "https://www.google.com/", "ques": "The photograph in the Whitney Museum of American Art's collection with accession number 2022.128 shows a person holding a book. Which military unit did the author of this book join in 1813? Answer without using articles."}
44 | {"task_id": "33d8ea3b-6c6b-4ff1-803d-7e270dea8a57", "Level": 2, "Final answer": "2", "id": "level2-17", "web": "https://www.google.com/", "ques": "What is the minimum number of page links a person must click on to go from the english Wikipedia page on The Lord of the Rings (the book) to the english Wikipedia page on A Song of Ice and Fire (the book series)? In your count, include each link you would click on to get to the page. Use the pages as they appeared at the end of the day on July 3, 2023."}
45 | {"task_id": "e8cb5b03-41e0-4086-99e5-f6806cd97211", "Level": 2, "Final answer": "shrimp", "id": "level2-18", "web": "https://www.google.com/", "ques": "I went to Virtue restaurant & bar in Chicago for my birthday on March 22, 2021 and the main course I had was delicious! Unfortunately, when I went back about a month later on April 21, it was no longer on the dinner menu. Using the Wayback Machine, can you help me figure out which main course was on the dinner menu for Virtue on March 22, 2021 but not April 21, 2021? Answer using the singular form, without articles."}
46 | {"task_id": "f46b4380-207e-4434-820b-f32ce04ae2a4", "Level": 2, "Final answer": "Harbinger, Tidal", "id": "level2-19", "web": "https://www.google.com/", "ques": "It is 1999. Before you party like it is 1999, please assist me in settling a bet.\n\nFiona Apple and Paula Cole released albums prior to 1999. Of these albums, which didn't receive a letter grade from Robert Christgau? Provide your answer as a comma delimited list of album titles, sorted alphabetically."}
47 | {"task_id": "05407167-39ec-4d3a-a234-73a9120c325d", "Level": 2, "Final answer": "Format Document", "id": "level2-20", "web": "https://www.google.com/", "ques": "In the 2018 VSCode blog post on replit.com, what was the command they clicked on in the last video to remove extra lines?"}
48 | {"task_id": "b9763138-c053-4832-9f55-86200cb1f99c", "Level": 2, "Final answer": "3", "id": "level2-21", "web": "https://www.google.com/", "ques": "Compute the check digit the Tropicos ID for the Order Helotiales would have if it were an ISBN-10 number."}
49 | {"task_id": "16d825ff-1623-4176-a5b5-42e0f5c2b0ac", "Level": 2, "Final answer": "6:41 PM", "id": "level2-22", "web": "https://www.google.com/", "ques": "What time was the Tri-Rail train that carried the most passengers on May 27, 2019 scheduled to arrive in Pompano Beach? Express your answer in the 12-hour digital clock format without leading zero if any, and include whether it is AM or PM."}
50 | {"task_id": "544b7f0c-173a-4377-8d56-57b36eb26ddf", "Level": 2, "Final answer": "A Nightmare on Elm Street", "id": "level2-23", "web": "https://www.google.com/", "ques": "In Valentina Re\u2019s contribution to the 2017 book \u201cWorld Building: Transmedia, Fans, Industries\u201d, what horror movie does the author cite as having popularized metalepsis between a dream world and reality? Use the complete name with article if any."}
51 | {"task_id": "6b078778-0b90-464d-83f6-59511c811b01", "Level": 2, "Final answer": "Alfonso Visconti", "id": "level2-24", "web": "https://www.google.com/", "ques": "The Metropolitan Museum of Art has a portrait in its collection with an accession number of 29.100.5. Of the consecrators and co-consecrators of this portrait's subject as a bishop, what is the name of the one who never became pope?"}
52 | {"task_id": "08cae58d-4084-4616-b6dd-dd6534e4825b", "Level": 2, "Final answer": "2018", "id": "level2-25", "web": "https://www.google.com/", "ques": "According to Google Finance, when was the first year the Apple stock went above $50 (without adjusting for stock split)?"}
53 | {"task_id": "2dfc4c37-fec1-4518-84a7-10095d30ad75", "Level": 2, "Final answer": "6", "id": "level2-26", "web": "https://www.google.com/", "ques": "According to Box Office Mojo's 2020 Worldwide Box Office list, how many of the top 10 highest-grossing worldwide movies are also on the top 10 highest-grossing domestic movies? Your answer should be a numerical integer value."}
54 | {"task_id": "9f41b083-683e-4dcf-9185-ccfeaa88fa45", "Level": 2, "Final answer": "0", "id": "level2-27", "web": "https://www.google.com/", "ques": "How many pages if the 2023 IPCC report (85 pages version) mentions nuclear energy?"}
55 | {"task_id": "ecbc4f94-95a3-4cc7-b255-6741a458a625", "Level": 2, "Final answer": "13", "id": "level2-28", "web": "https://www.google.com/", "ques": "How many images are there in the latest 2022 Lego english wikipedia article?"}
56 | {"task_id": "71345b0a-9c7d-4b50-b2bf-937ec5879845", "Level": 2, "Final answer": "Here be dragons", "id": "level2-29", "web": "https://www.google.com/", "ques": "On a leap day before the year 2008, a joke was removed from the Wikipedia page for \u201cDragon\u201d. What was the phrase that was removed? Give the phrase as it appeared on the page, but without punctuation."}
57 | {"task_id": "7b5377b0-3f38-4103-8ad2-90fe89864c04", "Level": 2, "Final answer": "563.9", "id": "level2-30", "web": "https://www.google.com/", "ques": "Find the value of x to the nearest tenth: Lx = (d/dx * (A * x-squared)) + 4-thousand'n'ninety-7 minus C\nWhere L is the last two digits of the year of the Venezuelan Declaration of Independence,\nA is the number of colors in the TikTok logo as of July 2023, excluding black and white,\nand C is the height of the average woman in the Philippines according to a July 2023 Business Insider article, rounded to the nearest whole centimeter"}
58 | {"task_id": "114d5fd0-e2ae-4b6d-a65a-870da2d19c08", "Level": 2, "Final answer": "4", "id": "level2-31", "web": "https://www.google.com/", "ques": "In the endnote found in the second-to-last paragraph of page 11 of the book with the doi 10.2307/j.ctv9b2xdv, what date in November was the Wikipedia article accessed? Just give the day of the month."}
59 | {"task_id": "ad37a656-079a-49f9-a493-7b739c9167d1", "Level": 2, "Final answer": "Bravo", "id": "level2-32", "web": "https://www.google.com/", "ques": "On July 15, 2008, Phys.org published an article about a catastrophe. Find the explosive force of this catastrophe according to Encyclopedia Britannica, then find the name of the US nuclear test that had the same yield. Your answer should only be the last word of the name of the test."}
60 | {"task_id": "f3917a3d-1d17-4ee2-90c5-683b072218fe", "Level": 2, "Final answer": "2732", "id": "level2-33", "web": "https://www.google.com/", "ques": "How many edits were made to the Wikipedia page on Antidisestablishmentarianism from its inception until June of 2023?"}
61 | {"task_id": "48eb8242-1099-4c26-95d4-ef22b002457a", "Level": 2, "Final answer": "6", "id": "level2-34", "web": "https://www.google.com/", "ques": "How many nonindigenous crocodiles were found in Florida from the year 2000 through 2020? You can get the data from the USGS Nonindigenous Aquatic Species database."}
62 | {"task_id": "c8b7e059-c60d-472e-ad64-3b04ae1166dc", "Level": 2, "Final answer": "8", "id": "level2-35", "web": "https://www.google.com/", "ques": "The work referenced in footnote 397 of Federico Lauria's 2014 dissertation is also the source for the titles of two paintings in the Smithsonian American Art Museum's collection, as of August 2023. What is the absolute difference between the chapter numbers of the chapters that the titles of these two paintings quote?"}
63 | {"task_id": "d1af70ea-a9a4-421a-b9cc-94b5e02f1788", "Level": 2, "Final answer": "736455", "id": "level2-36", "web": "https://www.google.com/", "ques": "As of the 2020 census, what was the population difference between the largest county seat and smallest county seat, by land area of the county seat, in Washington state? For population figures, please use the official data from data.census.gov. Please report the integer difference."}
64 | {"task_id": "ded28325-3447-4c56-860f-e497d6fb3577", "Level": 2, "Final answer": "Picnic is in Ploybius Plaza.", "id": "level2-37", "web": "https://www.google.com/", "ques": "This is a secret message my friend gave me. It says where we should meet for our picnic on Friday. The only problem is, it\u2019s encrypted in the Caesar cipher, so I can\u2019t read it. Can you tell me what it says? This is the message:\n\nZsmxsm sc sx Zyvilsec Zvkjk."}
65 | {"task_id": "d700d50d-c707-4dca-90dc-4528cddd0c80", "Level": 2, "Final answer": "Roger Miller", "id": "level2-38", "web": "https://www.google.com/", "ques": "Who composed the song that was performed by a rooster and a hamster in separate animated videos at separate tempos with different lyrics? Answer using the format First name Last name."}
66 | {"task_id": "0a3cd321-3e76-4622-911b-0fda2e5d6b1a", "Level": 2, "Final answer": "Brunei, China, Morocco, Singapore", "id": "level2-39", "web": "https://www.google.com/", "ques": "According to the World Bank, which countries had gross savings of over 35% of GDP for every year in the period 2001-2010? Give your answer as a comma-separated list of countries in alphabetical order. Use the countries most common names in english when answering."}
67 | {"task_id": "f2feb6a4-363c-4c09-a804-0db564eafd68", "Level": 2, "Final answer": "900000", "id": "level2-40", "web": "https://www.google.com/", "ques": "I\u2019m thinking about selling my home, so I want to learn more about how homes in my area sold recently. I live in Pearl City, Hawaii, which is on the island of Oahu. I know two homes near me that sold in 2022 were 2072 Akaikai Loop, and 2017 Komo Mai Drive. Find which of those homes sold for more in 2022, and tell me how much it sold for. Don\u2019t put commas or decimal places in the answer."}
68 | {"task_id": "0b260a57-3f3a-4405-9f29-6d7a1012dbfb", "Level": 2, "Final answer": "0.269", "id": "level2-41", "web": "https://www.google.com/", "ques": "On ScienceDirect, what is the difference to 3 decimal places in the sample standard deviations of the number of Reference Works in each Life Science domain compared to Health Sciences as of 2022?"}
69 | {"task_id": "ed58682d-bc52-4baa-9eb0-4eb81e1edacc", "Level": 2, "Final answer": "stare", "id": "level2-42", "web": "https://www.google.com/", "ques": "What is the last word before the second chorus of the King of Pop's fifth single from his sixth studio album?"}
70 | {"task_id": "023e9d44-96ae-4eed-b912-244ee8c3b994", "Level": 2, "Final answer": "8", "id": "level2-43", "web": "https://www.google.com/", "ques": "It's May 2023, and I'm about to drive across the U.S. from California to Maine. I always recycle my water bottles at the end of a trip, and I drink 5 12-ounce water bottles for every 100 miles I travel, rounded to the nearest 100. Assuming I follow I-40 from Los Angeles to Cincinnati, then take I-90 from Cincinnati to Augusta, how many dollars will I get back according to Wikipedia?"}
71 | {"task_id": "0e9e85b8-52b9-4de4-b402-5f635ab9631f", "Level": 2, "Final answer": "1927", "id": "level2-44", "web": "https://www.google.com/", "ques": "What is the latest chronological year date written in the image on the webpage found when following the first citation reference link on the latest version of Carl Nebel's Wikipedia page as of August 2023?"}
72 | {"task_id": "20194330-9976-4043-8632-f8485c6c71b2", "Level": 2, "Final answer": "4", "id": "level2-45", "web": "https://www.google.com/", "ques": "The YouTube channel Game Grumps began a Let\u2019s Play of the game Sonic the Hedgehog (2006) in the year 2012. Thirty seconds into the first episode, a phrase is shown on the screen in white letters on a red background. How many times does the letter \"E\" appear in this phrase?"}
73 | {"task_id": "65638e28-7f37-4fa7-b7b9-8c19bb609879", "Level": 2, "Final answer": "Kleinpaul", "id": "level2-46", "web": "https://www.google.com/", "ques": "The book with the doi 10.1353/book.24372 concerns a certain neurologist. According to chapter 2 of the book, what author influenced this neurologist\u2019s belief in \u201cendopsychic myths\u201d? Give the last name only."}
74 | {"task_id": "3ff6b7a9-a5bd-4412-ad92-0cd0d45c0fee", "Level": 2, "Final answer": "56000", "id": "level2-47", "web": "https://www.google.com/", "ques": "The longest-lived vertebrate is named after an island. According to Wikipedia as of January 1, 2021, what is the 2020 estimated population of that island, to the nearest thousand?"}
75 | {"task_id": "708b99c5-e4a7-49cb-a5cf-933c8d46470d", "Level": 2, "Final answer": "Citations", "id": "level2-48", "web": "https://www.google.com/", "ques": "On the DeepFruits fruit detection graph on Connected Papers from 2016, what feature caused the largest bubble to be the size it is?"}
76 | {"task_id": "0a65cb96-cb6e-4a6a-8aae-c1084f613456", "Level": 2, "Final answer": "Holabird", "id": "level2-49", "web": "https://www.google.com/", "ques": "During the first week of August 2015, one of the NASA Astronomy Pictures of the Day shows the lights of a city on the horizon. The namesake of this city also has a landmark building in Chicago named after him. What is the name of the architectural firm that designed this landmark building? Give the first name appearing in the name of the firm as of June 2023."}
77 | {"task_id": "65da0822-a48a-4a68-bbad-8ed1b835a834", "Level": 2, "Final answer": "Santa Clara, Boston", "id": "level2-50", "web": "https://www.google.com/", "ques": "All of the individuals who formally held the position of United States secretary of homeland security prior to April 2019, excluding those who held the position in an acting capacity, have a bachelor's degree. Of the universities that these bachelor's degrees were from, which is the westernmost university and which is the easternmost university? Give them to me as a comma-separated list, I only want the name of the cities where the universities are located, with the westernmost city listed first."}
78 | {"task_id": "73c1b9fe-ee1d-4cf4-96ca-35c08f97b054", "Level": 2, "Final answer": "1954", "id": "level2-51", "web": "https://www.google.com/", "ques": "According to the USGS, in what year was the American Alligator first found west of Texas (not including Texas)?"}
79 | {"task_id": "e2d69698-bc99-4e85-9880-67eaccd66e6c", "Level": 2, "Final answer": "Michele Fitzgerald", "id": "level2-52", "web": "https://www.google.com/", "ques": "As of August 2023, who is the only winner of the US version of Survivor to be born in the month of May?"}
80 | {"task_id": "a56f1527-3abf-41d6-91f8-7296d6336c3f", "Level": 2, "Final answer": "185", "id": "level2-53", "web": "https://www.google.com/", "ques": "The cover of the August 2021 issue of Vogue shows a famous landmark in the background behind some trees. How tall is this monument in yards, rounded to the nearest yard? Give the number only."}
81 | {"task_id": "42d4198c-5895-4f0a-b0c0-424a66465d83", "Level": 2, "Final answer": "60", "id": "level2-54", "web": "https://www.google.com/", "ques": "I'm curious about how much information is available for popular video games before their release. Find the Wikipedia page for the 2019 game that won the British Academy Games Awards. How many revisions did that page have before the month listed as the game's release date on that Wikipedia page (as of the most recent entry from 2022)?"}
82 | {"task_id": "a26649c6-1cb2-470a-871e-6910c64c3e53", "Level": 2, "Final answer": "116", "id": "level2-55", "web": "https://www.google.com/", "ques": "What is the absolute difference in tens of thousands between the population of chinstrap penguins on the Wikipedia page for penguin species populations as of the end of 2018 and the population recorded in the Nature.com \"global population assessment of the Chinstrap penguin\" article from 2020, assuming two penguins per breeding pair?"}
83 | {"task_id": "d5141ca5-e7a0-469f-bf3e-e773507c86e2", "Level": 2, "Final answer": "19/02/2009", "id": "level2-56", "web": "https://www.google.com/", "ques": "When was a picture of St. Thomas Aquinas first added to the Wikipedia page on the Principle of double effect? Answer using the format DD/MM/YYYY."}
84 | {"task_id": "1dcc160f-c187-48c2-b68e-319bd4354f3d", "Level": 2, "Final answer": "3", "id": "level2-57", "web": "https://www.google.com/", "ques": "According to Openreview.net, at the NeurIPS 2022 Conference, how many papers by an author named Yuri were accepted with a \"certain\" recommendation?"}
85 | {"task_id": "e0c10771-d627-4fd7-9694-05348e54ee36", "Level": 2, "Final answer": "234.9", "id": "level2-58", "web": "https://www.google.com/", "ques": "Take the gender split from the 2011 Bulgarian census about those who have completed tertiary education. Subtract the smaller number from the larger number, then return the difference in thousands of women. So if there were 30.1 thousand more men, you'd give \"30.1\""}
86 | {"task_id": "e29834fd-413a-455c-a33e-c3915b07401c", "Level": 2, "Final answer": "21", "id": "level2-59", "web": "https://www.google.com/", "ques": "I'd like to learn more about some popular reality television competition shows. As of the end of the 44th season of the American version of Survivor, how many more unique winners have there been compared to the number of winners of American Idol?"}
87 | {"task_id": "08c0b6e9-1b43-4c2e-ae55-4e3fce2c2715", "Level": 2, "Final answer": "orange, white", "id": "level2-60", "web": "https://www.google.com/", "ques": "In the film Goldfinger, what color was the object that James Bond concealed himself and his companion Pussy Galore at the end of the film? If there are multiple colors, put them in a comma-separated list in alphabetical order."}
88 | {"task_id": "db4fd70a-2d37-40ea-873f-9433dc5e301f", "Level": 2, "Final answer": "10", "id": "level2-61", "web": "https://www.google.com/", "ques": "As of May 2023, how many stops are between South Station and Windsor Gardens on MBTA\u2019s Franklin-Foxboro line (not included)?"}
89 | {"task_id": "853c8244-429e-46ca-89f2-addf40dfb2bd", "Level": 2, "Final answer": "11", "id": "level2-62", "web": "https://www.google.com/", "ques": "In the 2015 Metropolitan Museum of Art exhibition titled after the Chinese zodiac animal of 2015, how many of the \"twelve animals of the Chinese zodiac\" have a hand visible?"}
90 | {"task_id": "7a4a336d-dcfa-45a0-b014-824c7619e8de", "Level": 2, "Final answer": "1:41.614", "id": "level2-63", "web": "https://www.google.com/", "ques": "At the two-minute mark in the YouTube video uploaded by the channel \u201cGameGrumps\u201d on May 14, 2017 as part of their playthrough of the game Mario Kart 8 Deluxe, the shows\u2019 hosts are competing on one of the game\u2019s racetracks. What was the world record time for that track in the game\u2019s 150cc mode as of June 7, 2023? Express your answer in minutes and seconds, rounding the seconds to the nearest hundredth, e.g. 1:01.001."}
91 |
--------------------------------------------------------------------------------
/analysis.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "code",
5 | "execution_count": 84,
6 | "metadata": {},
7 | "outputs": [],
8 | "source": [
9 | "import json\n",
10 | "import pandas as pd\n",
11 | "\n",
12 | "# Read the JSON file\n",
13 | "# with open(\"results/experiment_results.json\", \"r\") as f:\n",
14 | "# data = pd.read_json(f)\n",
15 | "\n",
16 | "import glob\n",
17 | "import os\n",
18 | "\n",
19 | "# Get all task result files\n",
20 | "task_files = glob.glob(\"results/*--*/task_result.json\")\n",
21 | "\n",
22 | "# Read each file into a list of dictionaries\n"
23 | ]
24 | },
25 | {
26 | "cell_type": "code",
27 | "execution_count": 85,
28 | "metadata": {},
29 | "outputs": [],
30 | "source": [
31 | "results = []\n",
32 | "for file in task_files:\n",
33 | " with open(file, \"r\") as f:\n",
34 | " results.append(json.load(f))\n",
35 | "\n",
36 | "# Create DataFrame\n",
37 | "data = pd.DataFrame({\"all_tasks\": results})"
38 | ]
39 | },
40 | {
41 | "cell_type": "code",
42 | "execution_count": 86,
43 | "metadata": {},
44 | "outputs": [
45 | {
46 | "data": {
47 | "text/plain": [
48 | "590"
49 | ]
50 | },
51 | "execution_count": 86,
52 | "metadata": {},
53 | "output_type": "execute_result"
54 | }
55 | ],
56 | "source": [
57 | "all_tasks = pd.DataFrame(list(data['all_tasks']))\n",
58 | "# all_tasks.head(3)\n",
59 | "len(all_tasks)"
60 | ]
61 | },
62 | {
63 | "cell_type": "code",
64 | "execution_count": 87,
65 | "metadata": {},
66 | "outputs": [
67 | {
68 | "data": {
69 | "text/plain": [
70 | "586"
71 | ]
72 | },
73 | "execution_count": 87,
74 | "metadata": {},
75 | "output_type": "execute_result"
76 | }
77 | ],
78 | "source": [
79 | "# exclude impossible tasks\n",
80 | "import json\n",
81 | "\n",
82 | "\n",
83 | "with open(\"impossible-tasks.json\", \"r\") as f:\n",
84 | " impossible_tasks = set(json.load(f))\n",
85 | "\n",
86 | "all_tasks = all_tasks[~all_tasks[\"task_id\"].isin(impossible_tasks)]\n",
87 | "# all_tasks[\"group\"] = all_tasks[\"task_id\"].str.split(\"--\").str[0]\n",
88 | "len(all_tasks)"
89 | ]
90 | },
91 | {
92 | "cell_type": "code",
93 | "execution_count": 88,
94 | "metadata": {},
95 | "outputs": [
96 | {
97 | "data": {
98 | "text/plain": [
99 | "group\n",
100 | "Wolfram Alpha 46\n",
101 | "Cambridge Dictionary 43\n",
102 | "ArXiv 42\n",
103 | "ESPN 40\n",
104 | "GitHub 40\n",
105 | "Booking 40\n",
106 | "Google Search 40\n",
107 | "Coursera 40\n",
108 | "Allrecipes 39\n",
109 | "Google Flights 39\n",
110 | "Amazon 38\n",
111 | "Google Map 36\n",
112 | "Huggingface 35\n",
113 | "BBC News 35\n",
114 | "Apple 33\n",
115 | "Name: count, dtype: int64"
116 | ]
117 | },
118 | "execution_count": 88,
119 | "metadata": {},
120 | "output_type": "execute_result"
121 | }
122 | ],
123 | "source": [
124 | "all_tasks[\"group\"] = all_tasks[\"task_id\"].str.split(\"--\").str[0]\n",
125 | "all_tasks[\"group\"].value_counts()"
126 | ]
127 | },
128 | {
129 | "cell_type": "code",
130 | "execution_count": 89,
131 | "metadata": {},
132 | "outputs": [
133 | {
134 | "data": {
135 | "text/plain": [
136 | ""
137 | ]
138 | },
139 | "execution_count": 89,
140 | "metadata": {},
141 | "output_type": "execute_result"
142 | },
143 | {
144 | "data": {
145 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAigAAAGdCAYAAAA44ojeAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/GU6VOAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAoU0lEQVR4nO3df1TU153/8RfgMIg6UFRAViAmTaPUX4lWmNNuNlUEXU5qKmc3MZ7EZN1k66KbSOta9qhBbapL9hu72SWmZ49R96Q2W/c0ydEYFTXqpuIvEk/90eWox4S0OrCrB1ApwwD3+0cOsxlRYXBw7kyej3M8ZO7nzp33mzswr3xmhokxxhgBAABYJDbcBQAAANyIgAIAAKxDQAEAANYhoAAAAOsQUAAAgHUIKAAAwDoEFAAAYB0CCgAAsM6AcBfQF52dnbp48aKGDBmimJiYcJcDAAB6wRijq1evKiMjQ7Gxtz9HEpEB5eLFi8rMzAx3GQAAoA8+//xzjRw58rZzIjKgDBkyRNIXDbpcrjBX03s+n0+7d+9WQUGBHA5HuMvpF9HeY7T3J9FjtKDHyBeN/TU3NyszM9P/OH47ERlQup7WcblcERdQEhMT5XK5oubOdqNo7zHa+5PoMVrQY+SL5v568/IMXiQLAACsQ0ABAADWIaAAAADrEFAAAIB1CCgAAMA6BBQAAGAdAgoAALAOAQUAAFiHgAIAAKxDQAEAANYhoAAAAOsQUAAAgHUIKAAAwDoEFAAAYJ0B4S7ARvf8+P1+WdcZZ1QxRRpbvkvejp4/ajoYn64tCul6AACEE2dQAACAdQgoAADAOgQUAABgHQIKAACwDgEFAABYh4ACAACsQ0ABAADWIaAAAADrEFAAAIB1CCgAAMA6BBQAAGAdAgoAALAOAQUAAFiHgAIAAKxDQAEAANYhoAAAAOsQUAAAgHUIKAAAwDpBBZTy8nLFxMQE/Bs9erT/eGtrq0pKSjR06FANHjxYxcXFqq+vD1ijrq5ORUVFSkxMVGpqqpYsWaL29vbQdAMAAKLCgGCv8M1vflN79uz5vwUG/N8Sixcv1vvvv6+tW7cqKSlJCxcu1OzZs/Wb3/xGktTR0aGioiKlp6fr0KFDunTpkp5++mk5HA799Kc/DUE7AAAgGgQdUAYMGKD09PRu401NTdqwYYO2bNmiqVOnSpI2btyoMWPG6PDhw8rLy9Pu3bt15swZ7dmzR2lpaZo4caJWr16tpUuXqry8XPHx8XfeEQAAiHhBB5SzZ88qIyNDCQkJcrvdWrNmjbKyslRTUyOfz6f8/Hz/3NGjRysrK0vV1dXKy8tTdXW1xo0bp7S0NP+cwsJCLViwQKdPn9aDDz5409v0er3yer3+y83NzZIkn88nn88XbAs9csaZkK8pSc5YE/A1lPrj+9AXXXXYUk+oRXt/Ej1GC3qMfNHYXzC9xBhjev1o+cEHH+jatWt64IEHdOnSJa1cuVJ/+MMfdOrUKW3btk3PPvtsQJCQpClTpui73/2u/vEf/1HPP/+8PvvsM+3atct/vKWlRYMGDdKOHTs0c+bMm95ueXm5Vq5c2W18y5YtSkxM7G35AAAgjFpaWvTkk0+qqalJLpfrtnODOoPy5QAxfvx45ebmKjs7W7/61a80cODAvlXbC2VlZSotLfVfbm5uVmZmpgoKCnpssC/Glu/qeVIfOGONVk/u1PLjsfJ2xoR07VPlhSFdr698Pp+qqqo0ffp0ORyOcJcTctHen0SP0YIeI1809tf1DEhvBP0Uz5clJyfrG9/4hs6dO6fp06erra1NjY2NSk5O9s+pr6/3v2YlPT1dR48eDVij610+N3tdSxen0ymn09lt3OFw9MumeTtCGx66rd8ZE/LbsO3O2197Y4to70+ix2hBj5EvmvoLpo87+jso165d0/nz5zVixAhNmjRJDodDe/fu9R+vra1VXV2d3G63JMntduvkyZNqaGjwz6mqqpLL5VJOTs6dlAIAAKJIUGdQfvSjH+nRRx9Vdna2Ll68qJdeeklxcXGaM2eOkpKSNH/+fJWWliolJUUul0uLFi2S2+1WXl6eJKmgoEA5OTl66qmnVFFRIY/Ho2XLlqmkpOSmZ0gAAMBXU1AB5fe//73mzJmjy5cva/jw4frOd76jw4cPa/jw4ZKkdevWKTY2VsXFxfJ6vSosLNTrr7/uv35cXJy2b9+uBQsWyO12a9CgQZo3b55WrVoV2q4AAEBECyqgvP3227c9npCQoMrKSlVWVt5yTnZ2tnbs2BHMzQIAgK8YPosHAABYh4ACAACsQ0ABAADWIaAAAADrEFAAAIB1CCgAAMA6BBQAAGAdAgoAALAOAQUAAFiHgAIAAKxDQAEAANYhoAAAAOsQUAAAgHUIKAAAwDoEFAAAYB0CCgAAsA4BBQAAWIeAAgAArENAAQAA1iGgAAAA6xBQAACAdQgoAADAOgQUAABgHQIKAACwDgEFAABYh4ACAACsQ0ABAADWIaAAAADrEFAAAIB1CCgAAMA6BBQAAGAdAgoAALAOAQUAAFiHgAIAAKxDQAEAANYhoAAAAOsQUAAAgHUIKAAAwDoEFAAAYB0CCgAAsA4BBQAAWIeAAgAArENAAQAA1iGgAAAA6xBQAACAdQgoAADAOgQUAABgHQIKAACwDgEFAABYh4ACAACsQ0ABAADWIaAAAADrEFAAAIB1CCgAAMA6dxRQ1q5dq5iYGL344ov+sdbWVpWUlGjo0KEaPHiwiouLVV9fH3C9uro6FRUVKTExUampqVqyZIna29vvpBQAABBF+hxQjh07pp///OcaP358wPjixYu1bds2bd26VQcOHNDFixc1e/Zs//GOjg4VFRWpra1Nhw4d0ubNm7Vp0yatWLGi710AAICo0qeAcu3aNc2dO1f/9m//pq997Wv+8aamJm3YsEGvvvqqpk6dqkmTJmnjxo06dOiQDh8+LEnavXu3zpw5o7feeksTJ07UzJkztXr1alVWVqqtrS00XQEAgIjWp4BSUlKioqIi5efnB4zX1NTI5/MFjI8ePVpZWVmqrq6WJFVXV2vcuHFKS0vzzyksLFRzc7NOnz7dl3IAAECUGRDsFd5++219/PHHOnbsWLdjHo9H8fHxSk5ODhhPS0uTx+Pxz/lyOOk63nXsZrxer7xer/9yc3OzJMnn88nn8wXbQo+ccSbka0qSM9YEfA2l/vg+9EVXHbbUE2rR3p9Ej9GCHiNfNPYXTC9BBZTPP/9cL7zwgqqqqpSQkBB0YX21Zs0arVy5stv47t27lZiYGPLbq5gS8iUDrJ7cGfI1d+zYEfI170RVVVW4S+hX0d6fRI/Rgh4jXzT119LS0uu5QQWUmpoaNTQ06KGHHvKPdXR06ODBg/rXf/1X7dq1S21tbWpsbAw4i1JfX6/09HRJUnp6uo4ePRqwbte7fLrm3KisrEylpaX+y83NzcrMzFRBQYFcLlcwLfTK2PJdIV9T+uLMyerJnVp+PFbezpiQrn2qvDCk6/WVz+dTVVWVpk+fLofDEe5yQi7a+5PoMVrQY+SLxv66ngHpjaACyrRp03Ty5MmAsWeffVajR4/W0qVLlZmZKYfDob1796q4uFiSVFtbq7q6OrndbkmS2+3Wyy+/rIaGBqWmpkr6Ih26XC7l5OTc9HadTqecTme3cYfD0S+b5u0IbXjotn5nTMhvw7Y7b3/tjS2ivT+JHqMFPUa+aOovmD6CCihDhgzR2LFjA8YGDRqkoUOH+sfnz5+v0tJSpaSkyOVyadGiRXK73crLy5MkFRQUKCcnR0899ZQqKirk8Xi0bNkylZSU3DSEAACAr56gXyTbk3Xr1ik2NlbFxcXyer0qLCzU66+/7j8eFxen7du3a8GCBXK73Ro0aJDmzZunVatWhboUAAAQoe44oOzfvz/gckJCgiorK1VZWXnL62RnZ1v3ok4AAGAPPosHAABYh4ACAACsQ0ABAADWIaAAAADrEFAAAIB1CCgAAMA6BBQAAGAdAgoAALAOAQUAAFiHgAIAAKxDQAEAANYhoAAAAOsQUAAAgHUIKAAAwDoEFAAAYB0CCgAAsA4BBQAAWIeAAgAArENAAQAA1iGgAAAA6xBQAACAdQgoAADAOgQUAABgHQIKAACwDgEFAABYh4ACAACsQ0ABAADWIaAAAADrEFAAAIB1CCgAAMA6BBQAAGAdAgoAALAOAQUAAFiHgAIAAKxDQAEAANYhoAAAAOsQUAAAgHUIKAAAwDoEFAAAYB0CCgAAsA4BBQAAWIeAAgAArENAAQAA1iGgAAAA6xBQAACAdQgoAADAOgQUAABgHQIKAACwDgEFAABYh4ACAACsQ0ABAADWIaAAAADrEFAAAIB1CCgAAMA6BBQAAGCdoALK+vXrNX78eLlcLrlcLrndbn3wwQf+462trSopKdHQoUM1ePBgFRcXq76+PmCNuro6FRUVKTExUampqVqyZIna29tD0w0AAIgKQQWUkSNHau3ataqpqdHx48c1depUzZo1S6dPn5YkLV68WNu2bdPWrVt14MABXbx4UbNnz/Zfv6OjQ0VFRWpra9OhQ4e0efNmbdq0SStWrAhtVwAAIKINCGbyo48+GnD55Zdf1vr163X48GGNHDlSGzZs0JYtWzR16lRJ0saNGzVmzBgdPnxYeXl52r17t86cOaM9e/YoLS1NEydO1OrVq7V06VKVl5crPj4+dJ0BAICI1efXoHR0dOjtt9/W9evX5Xa7VVNTI5/Pp/z8fP+c0aNHKysrS9XV1ZKk6upqjRs3Tmlpaf45hYWFam5u9p+FAQAACOoMiiSdPHlSbrdbra2tGjx4sN555x3l5OToxIkTio+PV3JycsD8tLQ0eTweSZLH4wkIJ13Hu47ditfrldfr9V9ubm6WJPl8Pvl8vmBb6JEzzoR8TUlyxpqAr6HUH9+Hvuiqw5Z6Qi3a+5PoMVrQY+SLxv6C6SXogPLAAw/oxIkTampq0n/+539q3rx5OnDgQLDLBGXNmjVauXJlt/Hdu3crMTEx5LdXMSXkSwZYPbkz5Gvu2LEj5GveiaqqqnCX0K+ivT+JHqMFPUa+aOqvpaWl13ODDijx8fH6+te/LkmaNGmSjh07pn/+53/W448/rra2NjU2NgacRamvr1d6erokKT09XUePHg1Yr+tdPl1zbqasrEylpaX+y83NzcrMzFRBQYFcLlewLfRobPmukK8pfXHmZPXkTi0/HitvZ0xI1z5VXhjS9frK5/OpqqpK06dPl8PhCHc5IRft/Un0GC3oMfJFY39dz4D0RtAB5UadnZ3yer2aNGmSHA6H9u7dq+LiYklSbW2t6urq5Ha7JUlut1svv/yyGhoalJqaKumLZOhyuZSTk3PL23A6nXI6nd3GHQ5Hv2yatyO04aHb+p0xIb8N2+68/bU3toj2/iR6jBb0GPmiqb9g+ggqoJSVlWnmzJnKysrS1atXtWXLFu3fv1+7du1SUlKS5s+fr9LSUqWkpMjlcmnRokVyu93Ky8uTJBUUFCgnJ0dPPfWUKioq5PF4tGzZMpWUlNw0gAAAgK+moAJKQ0ODnn76aV26dElJSUkaP368du3apenTp0uS1q1bp9jYWBUXF8vr9aqwsFCvv/66//pxcXHavn27FixYILfbrUGDBmnevHlatWpVaLsCAAARLaiAsmHDhtseT0hIUGVlpSorK285Jzs727oXdAIAALvwWTwAAMA6BBQAAGAdAgoAALAOAQUAAFiHgAIAAKxDQAEAANYhoAAAAOsQUAAAgHUIKAAAwDoEFAAAYB0CCgAAsA4BBQAAWIeAAgAArENAAQAA1iGgAAAA6xBQAACAdQgoAADAOgQUAABgHQIKAACwDgEFAABYh4ACAACsQ0ABAADWIaAAAADrEFAAAIB1CCgAAMA6BBQAAGAdAgoAALAOAQUAAFiHgAIAAKxDQAEAANYhoAAAAOsQUAAAgHUIKAAAwDoEFAAAYB0CCgAAsA4BBQAAWIeAAgAArENAAQAA1iGgAAAA6xBQAACAdQgoAADAOgQUAABgHQIKAACwDgEFAABYh4ACAACsQ0ABAADWIaAAAADrEFAAAIB1CCgAAMA6BBQAAGAdAgoAALAOAQUAAFiHgAIAAKxDQAEAANYhoAAAAOsEFVDWrFmjb33rWxoyZIhSU1P12GOPqba2NmBOa2urSkpKNHToUA0ePFjFxcWqr68PmFNXV6eioiIlJiYqNTVVS5YsUXt7+513AwAAokJQAeXAgQMqKSnR4cOHVVVVJZ/Pp4KCAl2/ft0/Z/Hixdq2bZu2bt2qAwcO6OLFi5o9e7b/eEdHh4qKitTW1qZDhw5p8+bN2rRpk1asWBG6rgAAQEQbEMzknTt3BlzetGmTUlNTVVNTo4cfflhNTU3asGGDtmzZoqlTp0qSNm7cqDFjxujw4cPKy8vT7t27debMGe3Zs0dpaWmaOHGiVq9eraVLl6q8vFzx8fGh6w4AAESkoALKjZqamiRJKSkpkqSamhr5fD7l5+f754wePVpZWVmqrq5WXl6eqqurNW7cOKWlpfnnFBYWasGCBTp9+rQefPDBbrfj9Xrl9Xr9l5ubmyVJPp9PPp/vTlq4KWecCfmakuSMNQFfQ6k/vg990VWHLfWEWrT3J9FjtKDHyBeN/QXTS58DSmdnp1588UV9+9vf1tixYyVJHo9H8fHxSk5ODpiblpYmj8fjn/PlcNJ1vOvYzaxZs0YrV67sNr57924lJib2tYVbqpgS8iUDrJ7cGfI1d+zYEfI170RVVVW4S+hX0d6fRI/Rgh4jXzT119LS0uu5fQ4oJSUlOnXqlD766KO+LtFrZWVlKi0t9V9ubm5WZmamCgoK5HK5Qn57Y8t3hXxN6YszJ6snd2r58Vh5O2NCuvap8sKQrtdXPp9PVVVVmj59uhwOR7jLCblo70+ix2hBj5EvGvvregakN/oUUBYuXKjt27fr4MGDGjlypH88PT1dbW1tamxsDDiLUl9fr/T0dP+co0ePBqzX9S6frjk3cjqdcjqd3cYdDke/bJq3I7Thodv6nTEhvw3b7rz9tTe2iPb+JHqMFvQY+aKpv2D6COpdPMYYLVy4UO+884727dunUaNGBRyfNGmSHA6H9u7d6x+rra1VXV2d3G63JMntduvkyZNqaGjwz6mqqpLL5VJOTk4w5QAAgCgV1BmUkpISbdmyRe+9956GDBnif81IUlKSBg4cqKSkJM2fP1+lpaVKSUmRy+XSokWL5Ha7lZeXJ0kqKChQTk6OnnrqKVVUVMjj8WjZsmUqKSm56VkSAADw1RNUQFm/fr0k6ZFHHgkY37hxo5555hlJ0rp16xQbG6vi4mJ5vV4VFhbq9ddf98+Ni4vT9u3btWDBArndbg0aNEjz5s3TqlWr7qwTAAAQNYIKKMb0/PbYhIQEVVZWqrKy8pZzsrOzrXvXCQAAsAefxQMAAKxDQAEAANYhoAAAAOsQUAAAgHUIKAAAwDoEFAAAYB0CCgAAsA4BBQAAWIeAAgAArENAAQAA1iGgAAAA6xBQAACAdQgoAADAOgQUAABgHQIKAACwDgEFAABYh4ACAACsQ0ABAADWIaAAAADrEFAAAIB1CCgAAMA6BBQAAGAdAgoAALAOAQUAAFiHgAIAAKxDQAEAANYhoAAAAOsQUAAAgHUIKAAAwDoEFAAAYB0CCgAAsA4BBQAAWIeAAgAArENAAQAA1iGgAAAA6xBQAACAdQgoAADAOgQUAABgHQIKAACwDgEFAABYh4ACAACsQ0ABAADWIaAAAADrEFAAAIB1CCgAAMA6BBQAAGAdAgoAALAOAQUAAFiHgAIAAKxDQAEAANYhoAAAAOsQUAAAgHUIKAAAwDoEFAAAYB0CCgAAsE7QAeXgwYN69NFHlZGRoZiYGL377rsBx40xWrFihUaMGKGBAwcqPz9fZ8+eDZhz5coVzZ07Vy6XS8nJyZo/f76uXbt2R40AAIDoEXRAuX79uiZMmKDKysqbHq+oqNBrr72mN954Q0eOHNGgQYNUWFio1tZW/5y5c+fq9OnTqqqq0vbt23Xw4EE9//zzfe8CAABElQHBXmHmzJmaOXPmTY8ZY/Szn/1My5Yt06xZsyRJ//7v/660tDS9++67euKJJ/S73/1OO3fu1LFjxzR58mRJ0r/8y7/oz//8z/VP//RPysjIuIN2AABANAg6oNzOhQsX5PF4lJ+f7x9LSkpSbm6uqqur9cQTT6i6ulrJycn+cCJJ+fn5io2N1ZEjR/T973+/27per1der9d/ubm5WZLk8/nk8/lC2YIkyRlnQr6mJDljTcDXUOqP70NfdNVhSz2hFu39SfQYLegx8kVjf8H0EtKA4vF4JElpaWkB42lpaf5jHo9HqampgUUMGKCUlBT/nButWbNGK1eu7Da+e/duJSYmhqL0ABVTQr5kgNWTO0O+5o4dO0K+5p2oqqoKdwn9Ktr7k+gxWtBj5Ium/lpaWno9N6QBpb+UlZWptLTUf7m5uVmZmZkqKCiQy+UK+e2NLd8V8jWlL86crJ7cqeXHY+XtjAnp2qfKC0O6Xl/5fD5VVVVp+vTpcjgc4S4n5KK9P4keowU9Rr5o7K/rGZDeCGlASU9PlyTV19drxIgR/vH6+npNnDjRP6ehoSHgeu3t7bpy5Yr/+jdyOp1yOp3dxh0OR79smrcjtOGh2/qdMSG/DdvuvP21N7aI9v4keowW9Bj5oqm/YPoI6d9BGTVqlNLT07V3717/WHNzs44cOSK32y1JcrvdamxsVE1NjX/Ovn371NnZqdzc3FCWAwAAIlTQZ1CuXbumc+fO+S9fuHBBJ06cUEpKirKysvTiiy/qJz/5ie6//36NGjVKy5cvV0ZGhh577DFJ0pgxYzRjxgw999xzeuONN+Tz+bRw4UI98cQTvIMHAABI6kNAOX78uL773e/6L3e9NmTevHnatGmT/v7v/17Xr1/X888/r8bGRn3nO9/Rzp07lZCQ4L/OL37xCy1cuFDTpk1TbGysiouL9dprr4WgHQAAEA2CDiiPPPKIjLn122RjYmK0atUqrVq16pZzUlJStGXLlmBvGgAAfEXwWTwAAMA6BBQAAGAdAgoAALAOAQUAAFiHgAIAAKxDQAEAANYhoAAAAOsQUAAAgHUIKAAAwDoEFAAAYB0CCgAAsA4BBQAAWIeAAgAArENAAQAA1iGgAAAA6xBQAACAdQgoAADAOgQUAABgHQIKAACwDgEFAABYh4ACAACsQ0ABAADWIaAAAADrEFAAAIB1CCgAAMA6BBQAAGAdAgoAALAOAQUAAFiHgAIAAKxDQAEAANYZEO4CEBr3/Pj9cJcgSXLGGVVMkcaW75K3I+a2cz9dW3SXqgIARBrOoAAAAOsQUAAAgHUIKAAAwDoEFAAAYB0CCgAAsA4BBQAAWIeAAgAArENAAQAA1iGgAAAA6xBQAACAdQgoAADAOgQUAABgHQIKAACwDgEFAABYh4ACAACsQ0ABAADWIaAAAADrEFAAAIB1CCgAAMA6BBQAAGAdAgoAALAOAQUAAFiHgAIAAKxDQAEAANYJa0CprKzUPffco4SEBOXm5uro0aPhLAcAAFgibAHlP/7jP1RaWqqXXnpJH3/8sSZMmKDCwkI1NDSEqyQAAGCJAeG64VdffVXPPfecnn32WUnSG2+8offff19vvvmmfvzjH4erLOC2xpbvUsWUL756O2LCXU6vfLq2KNwlAEDQwhJQ2traVFNTo7KyMv9YbGys8vPzVV1d3W2+1+uV1+v1X25qapIkXblyRT6fL+T1DWi/HvI1JWlAp1FLS6cG+GLV0RkZD27BCqbHy5cv36WqQmeA73rE7WGw32efz6eWlhZdvnxZDoejn6oKL3qMDtHeYzT2d/XqVUmSMabHuWEJKP/7v/+rjo4OpaWlBYynpaXpv//7v7vNX7NmjVauXNltfNSoUf1WY395MtwF3AW97XHY/+vXMvpNpO1hpH6fAUSvq1evKikp6bZzwvYUTzDKyspUWlrqv9zZ2akrV65o6NChiomJjP+LlaTm5mZlZmbq888/l8vlCnc5/SLae4z2/iR6jBb0GPmisT9jjK5evaqMjIwe54YloAwbNkxxcXGqr68PGK+vr1d6enq3+U6nU06nM2AsOTm5P0vsVy6XK2rubLcS7T1Ge38SPUYLeox80dZfT2dOuoTlXTzx8fGaNGmS9u7d6x/r7OzU3r175Xa7w1ESAACwSNie4iktLdW8efM0efJkTZkyRT/72c90/fp1/7t6AADAV1fYAsrjjz+u//mf/9GKFSvk8Xg0ceJE7dy5s9sLZ6OJ0+nUSy+91O3pqmgS7T1Ge38SPUYLeox80d5fT2JMb97rAwAAcBfxWTwAAMA6BBQAAGAdAgoAALAOAQUAAFiHgBJia9as0be+9S0NGTJEqampeuyxx1RbWxsw55FHHlFMTEzAvx/84Adhqjh45eXl3eofPXq0/3hra6tKSko0dOhQDR48WMXFxd3+KJ/t7rnnnm49xsTEqKSkRFJk7uHBgwf16KOPKiMjQzExMXr33XcDjhtjtGLFCo0YMUIDBw5Ufn6+zp49GzDnypUrmjt3rlwul5KTkzV//nxdu3btLnZxa7frz+fzaenSpRo3bpwGDRqkjIwMPf3007p48WLAGjfb97Vr197lTm6tpz185plnutU/Y8aMgDk276HUc483+7mMiYnRK6+84p9j8z725jGiN79D6+rqVFRUpMTERKWmpmrJkiVqb2+/m630OwJKiB04cEAlJSU6fPiwqqqq5PP5VFBQoOvXAz+A8LnnntOlS5f8/yoqKsJUcd9885vfDKj/o48+8h9bvHixtm3bpq1bt+rAgQO6ePGiZs+eHcZqg3fs2LGA/qqqqiRJf/EXf+GfE2l7eP36dU2YMEGVlZU3PV5RUaHXXntNb7zxho4cOaJBgwapsLBQra2t/jlz587V6dOnVVVVpe3bt+vgwYN6/vnn71YLt3W7/lpaWvTxxx9r+fLl+vjjj/XrX/9atbW1+t73vtdt7qpVqwL2ddGiRXej/F7paQ8lacaMGQH1//KXvww4bvMeSj33+OXeLl26pDfffFMxMTEqLi4OmGfrPvbmMaKn36EdHR0qKipSW1ubDh06pM2bN2vTpk1asWJFOFrqPwb9qqGhwUgyBw4c8I/92Z/9mXnhhRfCV9Qdeumll8yECRNueqyxsdE4HA6zdetW/9jvfvc7I8lUV1ffpQpD74UXXjD33Xef6ezsNMZE/h5KMu+8847/cmdnp0lPTzevvPKKf6yxsdE4nU7zy1/+0hhjzJkzZ4wkc+zYMf+cDz74wMTExJg//OEPd6323rixv5s5evSokWQ+++wz/1h2drZZt25d/xYXIjfrcd68eWbWrFm3vE4k7aExvdvHWbNmmalTpwaMRdI+3vgY0ZvfoTt27DCxsbHG4/H456xfv964XC7j9XrvbgP9iDMo/aypqUmSlJKSEjD+i1/8QsOGDdPYsWNVVlamlpaWcJTXZ2fPnlVGRobuvfdezZ07V3V1dZKkmpoa+Xw+5efn++eOHj1aWVlZqq6uDle5d6StrU1vvfWW/uqv/irgwykjfQ+/7MKFC/J4PAH7lpSUpNzcXP++VVdXKzk5WZMnT/bPyc/PV2xsrI4cOXLXa75TTU1NiomJ6fa5XmvXrtXQoUP14IMP6pVXXom40+b79+9XamqqHnjgAS1YsECXL1/2H4u2Payvr9f777+v+fPndzsWKft442NEb36HVldXa9y4cQF/2LSwsFDNzc06ffr0Xay+f0XEpxlHqs7OTr344ov69re/rbFjx/rHn3zySWVnZysjI0O//e1vtXTpUtXW1urXv/51GKvtvdzcXG3atEkPPPCALl26pJUrV+pP//RPderUKXk8HsXHx3f7pZ+WliaPxxOegu/Qu+++q8bGRj3zzDP+sUjfwxt17c2Nf8n5y/vm8XiUmpoacHzAgAFKSUmJuL1tbW3V0qVLNWfOnIAPYfu7v/s7PfTQQ0pJSdGhQ4dUVlamS5cu6dVXXw1jtb03Y8YMzZ49W6NGjdL58+f1D//wD5o5c6aqq6sVFxcXVXsoSZs3b9aQIUO6PYUcKft4s8eI3vwO9Xg8N/1Z7ToWLQgo/aikpESnTp0KeH2GpIDne8eNG6cRI0Zo2rRpOn/+vO677767XWbQZs6c6f/v8ePHKzc3V9nZ2frVr36lgQMHhrGy/rFhwwbNnDkz4OPBI30Pv8p8Pp/+8i//UsYYrV+/PuBYaWmp/7/Hjx+v+Ph4/c3f/I3WrFkTEX9u/IknnvD/97hx4zR+/Hjdd9992r9/v6ZNmxbGyvrHm2++qblz5yohISFgPFL28VaPEfgCT/H0k4ULF2r79u368MMPNXLkyNvOzc3NlSSdO3fubpQWcsnJyfrGN76hc+fOKT09XW1tbWpsbAyYU19fr/T09PAUeAc+++wz7dmzR3/9139923mRvodde3PjOwW+vG/p6elqaGgION7e3q4rV65EzN52hZPPPvtMVVVVPX6EfW5urtrb2/Xpp5/enQJD7N5779WwYcP898to2MMu//Vf/6Xa2toefzYlO/fxVo8Rvfkdmp6eftOf1a5j0YKAEmLGGC1cuFDvvPOO9u3bp1GjRvV4nRMnTkiSRowY0c/V9Y9r167p/PnzGjFihCZNmiSHw6G9e/f6j9fW1qqurk5utzuMVfbNxo0blZqaqqKiotvOi/Q9HDVqlNLT0wP2rbm5WUeOHPHvm9vtVmNjo2pqavxz9u3bp87OTn9As1lXODl79qz27NmjoUOH9nidEydOKDY2ttvTIpHi97//vS5fvuy/X0b6Hn7Zhg0bNGnSJE2YMKHHuTbtY0+PEb35Hep2u3Xy5MmAsNkVuHNycu5OI3dDmF+kG3UWLFhgkpKSzP79+82lS5f8/1paWowxxpw7d86sWrXKHD9+3Fy4cMG899575t577zUPP/xwmCvvvR/+8Idm//795sKFC+Y3v/mNyc/PN8OGDTMNDQ3GGGN+8IMfmKysLLNv3z5z/Phx43a7jdvtDnPVwevo6DBZWVlm6dKlAeORuodXr141n3zyifnkk0+MJPPqq6+aTz75xP8ulrVr15rk5GTz3nvvmd/+9rdm1qxZZtSoUeaPf/yjf40ZM2aYBx980Bw5csR89NFH5v777zdz5swJV0sBbtdfW1ub+d73vmdGjhxpTpw4EfCz2fWuh0OHDpl169aZEydOmPPnz5u33nrLDB8+3Dz99NNh7uz/3K7Hq1evmh/96EemurraXLhwwezZs8c89NBD5v777zetra3+NWzeQ2N6vp8aY0xTU5NJTEw069ev73Z92/exp8cIY3r+Hdre3m7Gjh1rCgoKzIkTJ8zOnTvN8OHDTVlZWTha6jcElBCTdNN/GzduNMYYU1dXZx5++GGTkpJinE6n+frXv26WLFlimpqawlt4EB5//HEzYsQIEx8fb/7kT/7EPP744+bcuXP+43/84x/N3/7t35qvfe1rJjEx0Xz/+983ly5dCmPFfbNr1y4jydTW1gaMR+oefvjhhze9b86bN88Y88VbjZcvX27S0tKM0+k006ZN69b75cuXzZw5c8zgwYONy+Uyzz77rLl69WoYuunudv1duHDhlj+bH374oTHGmJqaGpObm2uSkpJMQkKCGTNmjPnpT38a8OAebrfrsaWlxRQUFJjhw4cbh8NhsrOzzXPPPRfwVlRj7N5DY3q+nxpjzM9//nMzcOBA09jY2O36tu9jT48RxvTud+inn35qZs6caQYOHGiGDRtmfvjDHxqfz3eXu+lfMcYY008nZwAAAPqE16AAAADrEFAAAIB1CCgAAMA6BBQAAGAdAgoAALAOAQUAAFiHgAIAAKxDQAEAANYhoAAAAOsQUAAAgHUIKAAAwDoEFAAAYJ3/D32yHalDIsYUAAAAAElFTkSuQmCC",
146 | "text/plain": [
147 | "
10 |