34 |
38 |
34 |
38 | Learn the basics and become familiar with using Agents. Start here if you are using Agents for the first time!
35 | 36 |Practical guides to help you achieve a specific goal: create an agent to generate and test SQL queries!
39 | 40 |High-level explanations for building a better understanding of important topics.
43 | 44 |Horizontal tutorials that cover important aspects of building agents.
47 | 48 |
38 |
39 | This is why we put emphasis on proposing code agents, in this case python agents, which meant putting higher effort on building secure python interpreters.
40 |
41 | ### Local python interpreter
42 |
43 | By default, the `CodeAgent` runs LLM-generated code in your environment.
44 | This execution is not done by the vanilla Python interpreter: we've re-built a more secure `LocalPythonInterpreter` from the ground up.
45 | This interpreter is designed for security by:
46 | - Restricting the imports to a list explicitly passed by the user
47 | - Capping the number of operations to prevent infinite loops and resource bloating.
48 | - Will not perform any operation that's not pre-defined.
49 |
50 | Wev'e used this on many use cases, without ever observing any damage to the environment.
51 |
52 | However this solution is not watertight: one could imagine occasions where LLMs fine-tuned for malignant actions could still hurt your environment. For instance if you've allowed an innocuous package like `Pillow` to process images, the LLM could generate thousands of saves of images to bloat your hard drive.
53 | It's certainly not likely if you've chosen the LLM engine yourself, but it could happen.
54 |
55 | So if you want to be extra cautious, you can use the remote code execution option described below.
56 |
57 | ### E2B code executor
58 |
59 | For maximum security, you can use our integration with E2B to run code in a sandboxed environment. This is a remote execution service that runs your code in an isolated container, making it impossible for the code to affect your local environment.
60 |
61 | For this, you will need to setup your E2B account and set your `E2B_API_KEY` in your environment variables. Head to [E2B's quickstart documentation](https://e2b.dev/docs/quickstart) for more information.
62 |
63 | Then you can install it with `pip install e2b-code-interpreter python-dotenv`.
64 |
65 | Now you're set!
66 |
67 | To set the code executor to E2B, simply pass the flag `use_e2b_executor=True` when initializing your `CodeAgent`.
68 | Note that you should add all the tool's dependencies in `additional_authorized_imports`, so that the executor installs them.
69 |
70 | ```py
71 | from prime import CodeAgent, VisitWebpageTool, HfApiModel
72 | agent = CodeAgent(
73 | tools = [VisitWebpageTool()],
74 | model=HfApiModel(),
75 | additional_authorized_imports=["requests", "markdownify"],
76 | use_e2b_executor=True
77 | )
78 |
79 | agent.run("What was Abraham Lincoln's preferred pet?")
80 | ```
81 |
82 | E2B code execution is not compatible with multi-agents at the moment - because having an agent call in a code blob that should be executed remotely is a mess. But we're working on adding it!
--------------------------------------------------------------------------------
/docs/source/en/reference/agents.md:
--------------------------------------------------------------------------------
1 |
16 | # Agents
17 |
18 |
53 |
112 |
113 | Writing actions in code rather than JSON-like snippets provides better:
114 |
115 | - **Composability:** could you nest JSON actions within each other, or define a set of JSON actions to re-use later, the same way you could just define a python function?
116 | - **Object management:** how do you store the output of an action like `generate_image` in JSON?
117 | - **Generality:** code is built to express simply anything you can have a computer do.
118 | - **Representation in LLM training data:** plenty of quality code actions is already included in LLMs’ training data which means they’re already trained for this!
119 |
--------------------------------------------------------------------------------
/src/prime/types.py:
--------------------------------------------------------------------------------
1 | # coding=utf-8
2 | # Copyright 2024 HuggingFace Inc.
3 | #
4 |
5 | import os
6 | import pathlib
7 | import tempfile
8 | import uuid
9 | from io import BytesIO
10 | import requests
11 | import numpy as np
12 |
13 | from transformers.utils import (
14 | is_soundfile_availble,
15 | is_torch_available,
16 | is_vision_available,
17 | )
18 | import logging
19 |
20 |
21 | logger = logging.getLogger(__name__)
22 |
23 | if is_vision_available():
24 | from PIL import Image
25 | from PIL.Image import Image as ImageType
26 | else:
27 | ImageType = object
28 |
29 | if is_torch_available():
30 | import torch
31 | from torch import Tensor
32 | else:
33 | Tensor = object
34 |
35 | if is_soundfile_availble():
36 | import soundfile as sf
37 |
38 |
39 | class AgentType:
40 | """
41 | Abstract class to be reimplemented to define types that can be returned by agents.
42 |
43 | These objects serve three purposes:
44 |
45 | - They behave as they were the type they're meant to be, e.g., a string for text, a PIL.Image for images
46 | - They can be stringified: str(object) in order to return a string defining the object
47 | - They should be displayed correctly in ipython notebooks/colab/jupyter
48 | """
49 |
50 | def __init__(self, value):
51 | self._value = value
52 |
53 | def __str__(self):
54 | return self.to_string()
55 |
56 | def to_raw(self):
57 | logger.error(
58 | "This is a raw AgentType of unknown type. Display in notebooks and string conversion will be unreliable"
59 | )
60 | return self._value
61 |
62 | def to_string(self) -> str:
63 | logger.error(
64 | "This is a raw AgentType of unknown type. Display in notebooks and string conversion will be unreliable"
65 | )
66 | return str(self._value)
67 |
68 |
69 | class AgentText(AgentType, str):
70 | """
71 | Text type returned by the agent. Behaves as a string.
72 | """
73 |
74 | def to_raw(self):
75 | return self._value
76 |
77 | def to_string(self):
78 | return str(self._value)
79 |
80 |
81 | class AgentImage(AgentType, ImageType):
82 | """
83 | Image type returned by the agent. Behaves as a PIL.Image.
84 | """
85 |
86 | def __init__(self, value):
87 | AgentType.__init__(self, value)
88 | ImageType.__init__(self)
89 |
90 | if not is_vision_available():
91 | raise ImportError("PIL must be installed in order to handle images.")
92 |
93 | self._path = None
94 | self._raw = None
95 | self._tensor = None
96 |
97 | if isinstance(value, AgentImage):
98 | self._raw, self._path, self._tensor = value._raw, value._path, value._tensor
99 | elif isinstance(value, ImageType):
100 | self._raw = value
101 | elif isinstance(value, bytes):
102 | self._raw = Image.open(BytesIO(value))
103 | elif isinstance(value, (str, pathlib.Path)):
104 | self._path = value
105 | elif isinstance(value, torch.Tensor):
106 | self._tensor = value
107 | elif isinstance(value, np.ndarray):
108 | self._tensor = torch.from_numpy(value)
109 | else:
110 | raise TypeError(
111 | f"Unsupported type for {self.__class__.__name__}: {type(value)}"
112 | )
113 |
114 | def _ipython_display_(self, include=None, exclude=None):
115 | """
116 | Displays correctly this type in an ipython notebook (ipython, colab, jupyter, ...)
117 | """
118 | from IPython.display import Image, display
119 |
120 | display(Image(self.to_string()))
121 |
122 | def to_raw(self):
123 | """
124 | Returns the "raw" version of that object. In the case of an AgentImage, it is a PIL.Image.
125 | """
126 | if self._raw is not None:
127 | return self._raw
128 |
129 | if self._path is not None:
130 | self._raw = Image.open(self._path)
131 | return self._raw
132 |
133 | if self._tensor is not None:
134 | array = self._tensor.cpu().detach().numpy()
135 | return Image.fromarray((255 - array * 255).astype(np.uint8))
136 |
137 | def to_string(self):
138 | """
139 | Returns the stringified version of that object. In the case of an AgentImage, it is a path to the serialized
140 | version of the image.
141 | """
142 | if self._path is not None:
143 | return self._path
144 |
145 | if self._raw is not None:
146 | directory = tempfile.mkdtemp()
147 | self._path = os.path.join(directory, str(uuid.uuid4()) + ".png")
148 | self._raw.save(self._path, format="png")
149 | return self._path
150 |
151 | if self._tensor is not None:
152 | array = self._tensor.cpu().detach().numpy()
153 |
154 | # There is likely simpler than load into image into save
155 | img = Image.fromarray((255 - array * 255).astype(np.uint8))
156 |
157 | directory = tempfile.mkdtemp()
158 | self._path = os.path.join(directory, str(uuid.uuid4()) + ".png")
159 | img.save(self._path, format="png")
160 |
161 | return self._path
162 |
163 | def save(self, output_bytes, format: str = None, **params):
164 | """
165 | Saves the image to a file.
166 | Args:
167 | output_bytes (bytes): The output bytes to save the image to.
168 | format (str): The format to use for the output image. The format is the same as in PIL.Image.save.
169 | **params: Additional parameters to pass to PIL.Image.save.
170 | """
171 | img = self.to_raw()
172 | img.save(output_bytes, format=format, **params)
173 |
174 |
175 | class AgentAudio(AgentType, str):
176 | """
177 | Audio type returned by the agent.
178 | """
179 |
180 | def __init__(self, value, samplerate=16_000):
181 | super().__init__(value)
182 |
183 | if not is_soundfile_availble():
184 | raise ImportError("soundfile must be installed in order to handle audio.")
185 |
186 | self._path = None
187 | self._tensor = None
188 |
189 | self.samplerate = samplerate
190 | if isinstance(value, (str, pathlib.Path)):
191 | self._path = value
192 | elif is_torch_available() and isinstance(value, torch.Tensor):
193 | self._tensor = value
194 | elif isinstance(value, tuple):
195 | self.samplerate = value[0]
196 | if isinstance(value[1], np.ndarray):
197 | self._tensor = torch.from_numpy(value[1])
198 | else:
199 | self._tensor = torch.tensor(value[1])
200 | else:
201 | raise ValueError(f"Unsupported audio type: {type(value)}")
202 |
203 | def _ipython_display_(self, include=None, exclude=None):
204 | """
205 | Displays correctly this type in an ipython notebook (ipython, colab, jupyter, ...)
206 | """
207 | from IPython.display import Audio, display
208 |
209 | display(Audio(self.to_string(), rate=self.samplerate))
210 |
211 | def to_raw(self):
212 | """
213 | Returns the "raw" version of that object. It is a `torch.Tensor` object.
214 | """
215 | if self._tensor is not None:
216 | return self._tensor
217 |
218 | if self._path is not None:
219 | if "://" in str(self._path):
220 | response = requests.get(self._path)
221 | response.raise_for_status()
222 | tensor, self.samplerate = sf.read(BytesIO(response.content))
223 | else:
224 | tensor, self.samplerate = sf.read(self._path)
225 | self._tensor = torch.tensor(tensor)
226 | return self._tensor
227 |
228 | def to_string(self):
229 | """
230 | Returns the stringified version of that object. In the case of an AgentAudio, it is a path to the serialized
231 | version of the audio.
232 | """
233 | if self._path is not None:
234 | return self._path
235 |
236 | if self._tensor is not None:
237 | directory = tempfile.mkdtemp()
238 | self._path = os.path.join(directory, str(uuid.uuid4()) + ".wav")
239 | sf.write(self._path, self._tensor, samplerate=self.samplerate)
240 | return self._path
241 |
242 |
243 | AGENT_TYPE_MAPPING = {"string": AgentText, "image": AgentImage, "audio": AgentAudio}
244 | INSTANCE_TYPE_MAPPING = {
245 | str: AgentText,
246 | ImageType: AgentImage,
247 | torch.Tensor: AgentAudio,
248 | }
249 |
250 | if is_torch_available():
251 | INSTANCE_TYPE_MAPPING[Tensor] = AgentAudio
252 |
253 |
254 | def handle_agent_input_types(*args, **kwargs):
255 | args = [(arg.to_raw() if isinstance(arg, AgentType) else arg) for arg in args]
256 | kwargs = {
257 | k: (v.to_raw() if isinstance(v, AgentType) else v) for k, v in kwargs.items()
258 | }
259 | return args, kwargs
260 |
261 |
262 | def handle_agent_output_types(output, output_type=None):
263 | if output_type in AGENT_TYPE_MAPPING:
264 | # If the class has defined outputs, we can map directly according to the class definition
265 | decoded_outputs = AGENT_TYPE_MAPPING[output_type](output)
266 | return decoded_outputs
267 | else:
268 | # If the class does not have defined output, then we map according to the type
269 | for _k, _v in INSTANCE_TYPE_MAPPING.items():
270 | if isinstance(output, _k):
271 | return _v(output)
272 | return output
273 |
274 |
275 | __all__ = ["AgentType", "AgentImage", "AgentText", "AgentAudio"]
276 |
--------------------------------------------------------------------------------
/src/prime/utils.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python
2 | # coding=utf-8
3 |
4 |
5 | import json
6 | import re
7 | from typing import Tuple, Dict, Union
8 | import ast
9 | from rich.console import Console
10 | import inspect
11 | import types
12 |
13 | from transformers.utils.import_utils import _is_package_available
14 |
15 | _pygments_available = _is_package_available("pygments")
16 |
17 |
18 | def is_pygments_available():
19 | return _pygments_available
20 |
21 |
22 | console = Console()
23 |
24 | BASE_BUILTIN_MODULES = [
25 | "collections",
26 | "datetime",
27 | "itertools",
28 | "math",
29 | "queue",
30 | "random",
31 | "re",
32 | "stat",
33 | "statistics",
34 | "time",
35 | "unicodedata",
36 | ]
37 |
38 |
39 | class AgentError(Exception):
40 | """Base class for other agent-related exceptions"""
41 |
42 | def __init__(self, message):
43 | super().__init__(message)
44 | self.message = message
45 | console.print(f"[bold red]{message}[/bold red]")
46 |
47 |
48 | class AgentParsingError(AgentError):
49 | """Exception raised for errors in parsing in the agent"""
50 |
51 | pass
52 |
53 |
54 | class AgentExecutionError(AgentError):
55 | """Exception raised for errors in execution in the agent"""
56 |
57 | pass
58 |
59 |
60 | class AgentMaxIterationsError(AgentError):
61 | """Exception raised for errors in execution in the agent"""
62 |
63 | pass
64 |
65 |
66 | class AgentGenerationError(AgentError):
67 | """Exception raised for errors in generation in the agent"""
68 |
69 | pass
70 |
71 |
72 | def parse_json_blob(json_blob: str) -> Dict[str, str]:
73 | try:
74 | first_accolade_index = json_blob.find("{")
75 | last_accolade_index = [a.start() for a in list(re.finditer("}", json_blob))][-1]
76 | json_blob = json_blob[first_accolade_index : last_accolade_index + 1].replace(
77 | '\\"', "'"
78 | )
79 | json_data = json.loads(json_blob, strict=False)
80 | return json_data
81 | except json.JSONDecodeError as e:
82 | place = e.pos
83 | if json_blob[place - 1 : place + 2] == "},\n":
84 | raise ValueError(
85 | "JSON is invalid: you probably tried to provide multiple tool calls in one action. PROVIDE ONLY ONE TOOL CALL."
86 | )
87 | raise ValueError(
88 | f"The JSON blob you used is invalid due to the following error: {e}.\n"
89 | f"JSON blob was: {json_blob}, decoding failed on that specific part of the blob:\n"
90 | f"'{json_blob[place-4:place+5]}'."
91 | )
92 | except Exception as e:
93 | raise ValueError(f"Error in parsing the JSON blob: {e}")
94 |
95 |
96 | def parse_code_blob(code_blob: str) -> str:
97 | try:
98 | pattern = r"```(?:py|python)?\n(.*?)\n```"
99 | match = re.search(pattern, code_blob, re.DOTALL)
100 | if match is None:
101 | raise ValueError(
102 | f"No match ground for regex pattern {pattern} in {code_blob=}."
103 | )
104 | return match.group(1).strip()
105 |
106 | except Exception as e:
107 | raise ValueError(
108 | f"""
109 | The code blob you used is invalid: due to the following error: {e}
110 | This means that the regex pattern {pattern} was not respected: make sure to include code with the correct pattern, for instance:
111 | Thoughts: Your thoughts
112 | Code:
113 | ```py
114 | # Your python code here
115 | ```
133 |
134 | Then you can use this tool just like any other tool. For example, let's improve the prompt `a rabbit wearing a space suit` and generate an image of it.
135 |
136 | ```python
137 | from prime import CodeAgent, HfApiModel
138 |
139 | model = HfApiModel("Qwen/Qwen2.5-Coder-32B-Instruct")
140 | agent = CodeAgent(tools=[image_generation_tool], model=model)
141 |
142 | agent.run(
143 | "Improve this prompt, then generate an image of it.", prompt='A rabbit wearing a space suit'
144 | )
145 | ```
146 |
147 | ```text
148 | === Agent thoughts:
149 | improved_prompt could be "A bright blue space suit wearing rabbit, on the surface of the moon, under a bright orange sunset, with the Earth visible in the background"
150 |
151 | Now that I have improved the prompt, I can use the image generator tool to generate an image based on this prompt.
152 | >>> Agent is executing the code below:
153 | image = image_generator(prompt="A bright blue space suit wearing rabbit, on the surface of the moon, under a bright orange sunset, with the Earth visible in the background")
154 | final_answer(image)
155 | ```
156 |
157 | 
158 |
159 | How cool is this? 🤩
160 |
161 | ### Use LangChain tools
162 |
163 | We love Langchain and think it has a very compelling suite of tools.
164 | To import a tool from LangChain, use the `from_langchain()` method.
165 |
166 | Here is how you can use it to recreate the intro's search result using a LangChain web search tool.
167 | This tool will need `pip install langchain google-search-results -q` to work properly.
168 | ```python
169 | from langchain.agents import load_tools
170 |
171 | search_tool = Tool.from_langchain(load_tools(["serpapi"])[0])
172 |
173 | agent = CodeAgent(tools=[search_tool], model=model)
174 |
175 | agent.run("How many more blocks (also denoted as layers) are in BERT base encoder compared to the encoder from the architecture proposed in Attention is All You Need?")
176 | ```
177 |
178 | ### Manage your agent's toolbox
179 |
180 | You can manage an agent's toolbox by adding or replacing a tool.
181 |
182 | Let's add the `model_download_tool` to an existing agent initialized with only the default toolbox.
183 |
184 | ```python
185 | from prime import HfApiModel
186 |
187 | model = HfApiModel("Qwen/Qwen2.5-Coder-32B-Instruct")
188 |
189 | agent = CodeAgent(tools=[], model=model, add_base_tools=True)
190 | agent.toolbox.add_tool(model_download_tool)
191 | ```
192 | Now we can leverage the new tool:
193 |
194 | ```python
195 | agent.run(
196 | "Can you give me the name of the model that has the most downloads in the 'text-to-video' task on the Hugging Face Hub but reverse the letters?"
197 | )
198 | ```
199 |
200 |
201 | > [!TIP]
202 | > Beware of not adding too many tools to an agent: this can overwhelm weaker LLM engines.
203 |
204 |
205 | Use the `agent.toolbox.update_tool()` method to replace an existing tool in the agent's toolbox.
206 | This is useful if your new tool is a one-to-one replacement of the existing tool because the agent already knows how to perform that specific task.
207 | Just make sure the new tool follows the same API as the replaced tool or adapt the system prompt template to ensure all examples using the replaced tool are updated.
208 |
209 |
210 | ### Use a collection of tools
211 |
212 | You can leverage tool collections by using the ToolCollection object, with the slug of the collection you want to use.
213 | Then pass them as a list to initialize you agent, and start using them!
214 |
215 | ```py
216 | from transformers import ToolCollection, CodeAgent
217 |
218 | image_tool_collection = ToolCollection(
219 | collection_slug="huggingface-tools/diffusion-tools-6630bb19a942c2306a2cdb6f",
220 | token="