├── README.md
├── images
├── CustomAgent.jpg
├── byzer-agent.png
├── byzer-tools.jpg
└── multi-agent.jpg
└── notebooks
└── quick_rag.ipynb
/README.md:
--------------------------------------------------------------------------------
1 |
2 |
3 | 
4 |
5 |
6 |
7 |
8 | Easy, fast, and distributed agent framework for everyone
9 |
10 |
11 |
12 | | Documentation | Blog | | Discord |
13 |
14 |
15 |
16 | ---
17 |
18 | *Latest News* 🔥
19 |
20 | - [2024/03] Byzer-Agent released in Byzer-LLM 0.1.46
21 |
22 | ---
23 |
24 | Byzer-Agent is an distributed agent framework for LLM. It is designed to be easy to use, easy to scale, and easy to debug. It is built on top of Ray, developed from [autogen](https://github.com/microsoft/autogen)。
25 |
26 | The code of Byzer-Agent is under the project [Byzer-LLM](https://github.com/allwefantasy/byzer-llm). So this project is just a document project.
27 |
28 | Notice that Byzer-Agent is just a framework, you can use it to build your own agents, and we will take care of the communication(local/remote), keep the conversation history, and so on.
29 |
30 | In order to implement a real business job, you can use Langchain / LlamaIndex in Byzer-Agent since they are a good library which impelement a lot of useful functions.
31 |
32 | ---
33 |
34 | * [Installation](#installation)
35 | * [Architecture](#Architecture)
36 | * [DataAnalysis (multi-agent)](#DataAnalysis-(multi-agent))
37 | * [Quick Start](#quick-start)
38 |
39 | ## Architecture
40 |
41 |
42 | 
43 |
44 |
45 | ---
46 |
47 |
48 | ## Installation
49 |
50 | Install the following projects step by step.
51 |
52 | 1. [Byzer-LLM](https://github.com/allwefantasy/byzer-llm)
53 | 2. [Byzer-Retrieval](https://github.com/allwefantasy/byzer-retrieval) (Optional)
54 |
55 | ---
56 |
57 | ## DataAnalysis (multi-agent)
58 |
59 |
60 | 
61 |
62 |
63 | ---
64 |
65 | ## Quick Start
66 |
67 | Here we will show you how to build a simple agent using Byzer-Agent.
68 | Below is a simple implementation of a HelloWorld Agent:
69 |
70 | ```python
71 | from byzerllm.apps.agent.conversable_agent import ConversableAgent
72 | from typing import Any, Callable, Dict, List, Optional, Tuple, Type, Union
73 | from byzerllm.utils.client import ByzerLLM
74 | from byzerllm.utils.retrieval import ByzerRetrieval
75 | from byzerllm.apps.agent import Agent
76 | from ray.util.client.common import ClientActorHandle, ClientObjectRef
77 | import byzerllm
78 |
79 | class HelloWorldAgent(ConversableAgent):
80 |
81 | @byzerllm.prompt()
82 | def agent_prompt(self)->str:
83 | '''
84 | 你是一个简单的问候Agent,无论用户说什么,你都回复 "Hello, world!"
85 | '''
86 |
87 | def __init__(
88 | self,
89 | name: str,
90 | llm: ByzerLLM,
91 | retrieval: ByzerRetrieval,
92 | system_message: Optional[str] = None,
93 | is_termination_msg: Optional[Callable[[Dict], bool]] = None,
94 | max_consecutive_auto_reply: Optional[int] = None,
95 | human_input_mode: Optional[str] = "NEVER",
96 | **kwargs,
97 | ):
98 | system_message = self.agent_prompt()
99 | super().__init__(
100 | name,
101 | llm,retrieval,
102 | system_message,
103 | is_termination_msg,
104 | max_consecutive_auto_reply,
105 | human_input_mode,
106 | **kwargs,
107 | )
108 |
109 | @byzerllm.agent_reply()
110 | def say_hello(
111 | self,
112 | raw_message: Optional[Union[Dict,str]] = None,
113 | messages: Optional[List[Dict]] = None,
114 | sender: Optional[Union[ClientActorHandle,Agent,str]] = None,
115 | config: Optional[Any] = None,
116 | ) -> Tuple[bool, Union[str, Dict, None]]:
117 | return True, {"content":"Hello, world!","metadata":{}}
118 | ```
119 |
120 | The implementation of this HelloWorldAgent is very simple:
121 |
122 | 1. We defined a class called HelloWorldAgent, which inherits from the ConversableAgent base class.
123 | 2. Above the class definition, we use the @byzerllm.prompt() decorator to define the Agent's system message prompt to guide the 34. Agent's behavior. In this example, we instruct the Agent to reply "Hello, world!" regardless of the message received.
124 | 3. In the __init__ method, we call the superclass constructor, passing in the necessary parameters.
125 | 4. We defined a method called say_hello and used the @byzerllm.agent_reply() decorator to mark it as a reply function. In this method, we directly return a tuple, the first element being True, indicating this is a terminating reply, and the second element is a dictionary containing the reply content "Hello, world!" and an empty metadata dictionary.
126 |
127 | Example code using this HelloWorldAgent is as follows:
128 |
129 | ```python
130 | import byzerllm
131 | byzerllm.connect_cluster()
132 | llm = byzerllm.ByzerLLM()
133 | llm.setup_template(model="sparkdesk_chat", template="auto")
134 | llm.setup_default_model_name("sparkdesk_chat")
135 |
136 | from byzerllm.apps.agent.user_proxy_agent import UserProxyAgent
137 |
138 | hello_agent = HelloWorldAgent("hello_agent", llm=llm, retrieval=None)
139 | user = UserProxyAgent("user", llm=llm, retrieval=None, human_input_mode="NEVER",max_consecutive_auto_reply=0)
140 |
141 | user.initiate_chat(
142 | recipient=hello_agent, clear_history=True, message={
143 | "content":"Hello there, may I ask who you are?"
144 | },
145 | )
146 | ```
147 |
148 | In this example, we created an instance of HelloWorldAgent and an instance of UserProxyAgent. Then, we have the user agent send a message "Hello there, may I ask who you are?" to the hello_agent. Upon receiving the message, hello_agent will reply with "Hello, world!" according to our defined say_hello reply function.
149 |
150 | That's a simple implementation and usage example of a HelloWorld Agent. You can extend this base to add more reply functions and logic according to your needs."
151 |
152 |
153 | ## Advanced Usage
154 |
155 | ## 1. Agent Communication
156 |
157 | Byzer-Agent provides a simple and easy-to-use communication mechanism for agents. You can use the following methods to communicate with other agents:
158 |
159 | * initiate_chat
160 | * send
161 |
162 | The initiate_chat method is used to start a conversation in client side.. You can specify the recipient agent, the message to send, and other parameters.
163 |
164 | The send method is used to agent to agent communication.
165 |
166 | Here is an example of how to use send method:
167 |
168 | ```python
169 | from byzerllm.apps.agent.conversable_agent import ConversableAgent
170 | from typing import Any, Callable, Dict, List, Optional, Tuple, Type, Union
171 | from byzerllm.utils.client import ByzerLLM
172 | from byzerllm.utils.retrieval import ByzerRetrieval
173 | from byzerllm.apps.agent import Agent,ChatResponse,get_agent_name
174 | from ray.util.client.common import ClientActorHandle, ClientObjectRef
175 | import byzerllm
176 |
177 | class AgentA(ConversableAgent):
178 |
179 | @byzerllm.prompt()
180 | def agent_prompt(self)->str:
181 | '''
182 | 你是一个友善的Agent,你的名字叫Alice。
183 | 当有人问你 "你是谁?" 的时候,你会回答 "我是Alice,很高兴认识你!"
184 | 然后你会将收到的消息转发给你的好友Bob。
185 | '''
186 |
187 | def __init__(
188 | self,
189 | name: str,
190 | llm: ByzerLLM,
191 | retrieval: ByzerRetrieval,
192 | bob: Union[Agent, ClientActorHandle,str],
193 | system_message: Optional[str] = None,
194 | is_termination_msg: Optional[Callable[[Dict], bool]] = None,
195 | max_consecutive_auto_reply: Optional[int] = None,
196 | human_input_mode: Optional[str] = "NEVER",
197 | **kwargs,
198 | ):
199 | system_message = self.agent_prompt()
200 | super().__init__(
201 | name,
202 | llm,retrieval,
203 | system_message,
204 | is_termination_msg,
205 | max_consecutive_auto_reply,
206 | human_input_mode,
207 | **kwargs,
208 | )
209 | self.bob = bob
210 |
211 | ## this function will reply to all agents(wihout agent filter), so it should be put at the head of the class
212 | ## to make sure the other reply functions execute before this function. Otherwise, the other reply
213 | ## functions will not be executed.
214 | @byzerllm.agent_reply()
215 | def introduce_self(
216 | self,
217 | raw_message: Optional[Union[Dict,str,ChatResponse]] = None,
218 | messages: Optional[List[Dict]] = None,
219 | sender: Optional[Union[ClientActorHandle,Agent,str]] = None,
220 | config: Optional[Any] = None,
221 | ) -> Tuple[bool, Union[str, Dict, None,ChatResponse]]:
222 | # forward the message to bob
223 | self.send(raw_message, self.bob)
224 | # get the last message from bob
225 | # _messages is a dictionary with agent names as keys and a list of messages as values
226 | # you can use the get_agent_name function to get the name of an agent
227 | message = self._messages[get_agent_name(self.bob)][-1]
228 | return True, None
229 |
230 | @byzerllm.agent_reply(lambda self:[self.bob])
231 | def reply_bob(
232 | self,
233 | raw_message: Optional[Union[Dict,str,ChatResponse]] = None,
234 | messages: Optional[List[Dict]] = None,
235 | sender: Optional[Union[ClientActorHandle,Agent,str]] = None,
236 | config: Optional[Any] = None,
237 | ) -> Tuple[bool, Union[str, Dict, None,ChatResponse]]:
238 | # we don't want to reply to bob
239 | return True, None
240 |
241 |
242 |
243 | class AgentB(ConversableAgent):
244 |
245 | @byzerllm.prompt()
246 | def agent_prompt(self)->str:
247 | '''
248 | 你是一个友善的Agent,你的名字叫Bob。
249 | 当有人问 "你是谁?" 的时候,你会回答 "我是Bob,Alice的好朋友,很高兴认识你!"
250 | '''
251 |
252 | def __init__(
253 | self,
254 | name: str,
255 | llm: ByzerLLM,
256 | retrieval: ByzerRetrieval,
257 | system_message: Optional[str] = None,
258 | is_termination_msg: Optional[Callable[[Dict], bool]] = None,
259 | max_consecutive_auto_reply: Optional[int] = None,
260 | human_input_mode: Optional[str] = "NEVER",
261 | **kwargs,
262 | ):
263 | system_message = self.agent_prompt()
264 | super().__init__(
265 | name,
266 | llm,retrieval,
267 | system_message,
268 | is_termination_msg,
269 | max_consecutive_auto_reply,
270 | human_input_mode,
271 | **kwargs,
272 | )
273 |
274 | @byzerllm.agent_reply()
275 | def introduce_self(
276 | self,
277 | raw_message: Optional[Union[Dict,str,ChatResponse]] = None,
278 | messages: Optional[List[Dict]] = None,
279 | sender: Optional[Union[ClientActorHandle,Agent,str]] = None,
280 | config: Optional[Any] = None,
281 | ) -> Tuple[bool, Union[str, Dict, None,ChatResponse]]:
282 | return True, {"content":"我是Bob,Alice的好朋友,很高兴认识你!","metadata":{}}
283 |
284 |
285 | ```
286 | We have two agents, AgentA and AgentB. AgentA will forward the message to AgentB and then reply to the user with the last message from AgentB.
287 |
288 | ```python
289 | import byzerllm
290 | byzerllm.connect_cluster()
291 | llm = byzerllm.ByzerLLM()
292 | llm.setup_template(model="sparkdesk_chat",template="auto")
293 | llm.setup_default_model_name("sparkdesk_chat")
294 |
295 | from byzerllm.apps.agent.user_proxy_agent import UserProxyAgent
296 |
297 | bob = AgentB("bob",llm=llm,retrieval=None)
298 | alice = AgentA("alice",llm=llm,retrieval=None,bob=bob)
299 | user = UserProxyAgent("user",llm=llm,retrieval=None,human_input_mode="NEVER",max_consecutive_auto_reply=0)
300 |
301 | user.initiate_chat(
302 | recipient=alice,clear_history=True,message={
303 | "content":"你是谁?"
304 | },
305 | )
306 |
307 |
308 | ```
309 |
310 | Here we have two agents, AgentA and AgentB. AgentA will forward the message to AgentB and then reply to the user with the last message from AgentB.
311 |
312 | The output of the above code will be:
313 |
314 | ```
315 | user (to alice):
316 |
317 | 你是谁?
318 |
319 | --------------------------------------------------------------------------------
320 | alice (to bob):
321 |
322 | 你是谁?
323 |
324 | --------------------------------------------------------------------------------
325 | bob (to alice):
326 |
327 | 我是Bob,Alice的好朋友,很高兴认识你!
328 | ```
329 |
330 | ### 1.1 Agent Reply Function Order
331 |
332 | The order in which the reply functions are executed is determined by the order in which they are defined in the agent class.
333 |
334 | The reply functions are executed in the order of their definition, If the function finds that the message is not for it, it can return False, and the next function will be executed.
335 |
336 | So you need to put the reply function which can reply all agents at the head of the class.
337 |
338 | ## 2. Agent State
339 |
340 | In the provided code, state management is handled through the following mechanisms:
341 |
342 | 1. Message History:
343 | - The `ConversableAgent` class maintains a dictionary called `_messages` which stores the conversation history for each agent. The keys of the dictionary are the agent names (obtained using the `get_agent_name` function), and the values are lists of messages exchanged with each agent.
344 | - When an agent receives a message, it appends the message to the corresponding list in the `_messages` dictionary using the `_append_message` method.
345 | - The conversation history can be accessed using the `chat_messages` property or the `get_chat_messages` method.
346 | - The `last_message` method allows retrieving the last message exchanged with a specific agent.
347 |
348 | 2. Agent References:
349 | - In the example code, `AgentA` is initialized with a reference to `AgentB` through the `bob` parameter in its constructor. This allows `AgentA` to send messages directly to `AgentB` using the `send` method.
350 | - By storing references to other agents, an agent can maintain a state of its connections and communicate with them as needed.
351 |
352 | 3. Agent-specific Variables:
353 | - Each agent class can define its own instance variables to store agent-specific state information.
354 | - For example, `AgentA` has an instance variable `self.bob` which stores the reference to `AgentB`. This variable can be used to keep track of the connected agent and perform actions based on that information.
355 |
356 | 4. Reply Functions:
357 | - The `@byzerllm.agent_reply()` decorator is used to define reply functions within an agent class. These functions are triggered when the agent receives a message from a specific sender or any sender (if no sender is specified).
358 | - Reply functions can access the agent's state, such as the conversation history (`_messages`) and agent-specific variables, to make decisions and generate responses based on the current state.
359 |
360 | 5. Conversation Termination:
361 | - The `is_termination_msg` parameter in the agent constructor allows specifying a function that determines if a received message should terminate the conversation.
362 | - By setting `max_consecutive_auto_reply` to 0 for the `UserProxyAgent`, it prevents the agent from engaging in further automatic replies after receiving a response from `AgentA`.
363 | - The `reply_bob` function in `AgentA` is decorated with `@byzerllm.agent_reply(lambda self:[self.bob])` to handle messages from `AgentB` and terminate the conversation without sending a reply back to `AgentB`.
364 |
365 | These mechanisms collectively enable state management in the agent-based conversation system. The conversation history, agent references, and agent-specific variables allow agents to maintain and access relevant state information. The reply functions and conversation termination conditions provide control over the flow of the conversation based on the current state.
366 |
367 |
368 |
369 | ## 3. Remote Agent
370 |
371 | You can slightly modify the previous example to use remote agents. Here is an example of how to use remote agents:
372 |
373 | ```python
374 | from byzerllm.apps.agent.conversable_agent import ConversableAgent
375 | from typing import Any, Callable, Dict, List, Optional, Tuple, Type, Union
376 | from byzerllm.utils.client import ByzerLLM
377 | from byzerllm.utils.retrieval import ByzerRetrieval
378 | from byzerllm.apps.agent import Agents,Agent,ChatResponse,get_agent_name
379 | from ray.util.client.common import ClientActorHandle, ClientObjectRef
380 | import byzerllm
381 |
382 | class AgentA(ConversableAgent):
383 |
384 | @byzerllm.prompt()
385 | def agent_prompt(self)->str:
386 | '''
387 | 你是一个友善的Agent,你的名字叫Alice。
388 | 当有人问你 "你是谁?" 的时候,你会回答 "我是Alice,很高兴认识你!"
389 | 然后你会将收到的消息转发给你的好友Bob。
390 | '''
391 |
392 | def __init__(
393 | self,
394 | name: str,
395 | llm: ByzerLLM,
396 | retrieval: ByzerRetrieval,
397 | bob: Union[Agent, ClientActorHandle,str],
398 | system_message: Optional[str] = None,
399 | is_termination_msg: Optional[Callable[[Dict], bool]] = None,
400 | max_consecutive_auto_reply: Optional[int] = None,
401 | human_input_mode: Optional[str] = "NEVER",
402 | **kwargs,
403 | ):
404 | system_message = self.agent_prompt()
405 | super().__init__(
406 | name,
407 | llm,retrieval,
408 | system_message,
409 | is_termination_msg,
410 | max_consecutive_auto_reply,
411 | human_input_mode,
412 | **kwargs,
413 | )
414 | self.bob = bob
415 |
416 | @byzerllm.agent_reply()
417 | def introduce_self(
418 | self,
419 | raw_message: Optional[Union[Dict,str,ChatResponse]] = None,
420 | messages: Optional[List[Dict]] = None,
421 | sender: Optional[Union[ClientActorHandle,Agent,str]] = None,
422 | config: Optional[Any] = None,
423 | ) -> Tuple[bool, Union[str, Dict, None,ChatResponse]]:
424 | # forward the message to bob
425 | self.send(raw_message, self.bob)
426 | # get the last message from bob
427 | # _messages is a dictionary with agent names as keys and a list of messages as values
428 | # you can use the get_agent_name function to get the name of an agent
429 | message = self._messages[get_agent_name(self.bob)][-1]
430 | return True, None
431 |
432 | @byzerllm.agent_reply(lambda self:[self.bob])
433 | def reply_bob(
434 | self,
435 | raw_message: Optional[Union[Dict,str,ChatResponse]] = None,
436 | messages: Optional[List[Dict]] = None,
437 | sender: Optional[Union[ClientActorHandle,Agent,str]] = None,
438 | config: Optional[Any] = None,
439 | ) -> Tuple[bool, Union[str, Dict, None,ChatResponse]]:
440 | # we don't want to reply to bob
441 | return True, None
442 |
443 |
444 |
445 | class AgentB(ConversableAgent):
446 |
447 | @byzerllm.prompt()
448 | def agent_prompt(self)->str:
449 | '''
450 | 你是一个友善的Agent,你的名字叫Bob。
451 | 当有人问 "你是谁?" 的时候,你会回答 "我是Bob,Alice的好朋友,很高兴认识你!"
452 | '''
453 |
454 | def __init__(
455 | self,
456 | name: str,
457 | llm: ByzerLLM,
458 | retrieval: ByzerRetrieval,
459 | system_message: Optional[str] = None,
460 | is_termination_msg: Optional[Callable[[Dict], bool]] = None,
461 | max_consecutive_auto_reply: Optional[int] = None,
462 | human_input_mode: Optional[str] = "NEVER",
463 | **kwargs,
464 | ):
465 | system_message = self.agent_prompt()
466 | super().__init__(
467 | name,
468 | llm,retrieval,
469 | system_message,
470 | is_termination_msg,
471 | max_consecutive_auto_reply,
472 | human_input_mode,
473 | **kwargs,
474 | )
475 |
476 | @byzerllm.agent_reply()
477 | def introduce_self(
478 | self,
479 | raw_message: Optional[Union[Dict,str,ChatResponse]] = None,
480 | messages: Optional[List[Dict]] = None,
481 | sender: Optional[Union[ClientActorHandle,Agent,str]] = None,
482 | config: Optional[Any] = None,
483 | ) -> Tuple[bool, Union[str, Dict, None,ChatResponse]]:
484 | return True, {"content":"我是Bob,Alice的好朋友,很高兴认识你!","metadata":{}}
485 |
486 | import byzerllm
487 | byzerllm.connect_cluster()
488 | llm = byzerllm.ByzerLLM()
489 | llm.setup_template(model="sparkdesk_chat",template="auto")
490 | llm.setup_default_model_name("sparkdesk_chat")
491 |
492 | from byzerllm.apps.agent.user_proxy_agent import UserProxyAgent
493 |
494 | bob = Agents.create_remote_agent(AgentB,"bob",llm,None)
495 |
496 | alice = Agents.create_remote_agent(AgentA,"alice",llm,None,bob=bob)
497 |
498 | user = UserProxyAgent("user",llm=llm,retrieval=None,human_input_mode="NEVER",max_consecutive_auto_reply=0)
499 |
500 | user.initiate_chat(
501 | recipient=alice,clear_history=True,message={
502 | "content":"你是谁?"
503 | },
504 | )
505 |
506 | ```
507 |
508 | Here is the output of the above code:
509 |
510 | ```
511 | (AgentB pid=935730) alice (to bob):
512 | (AgentB pid=935730)
513 | (AgentB pid=935730) 你是谁?
514 | (AgentB pid=935730)
515 | (AgentB pid=935730) --------------------------------------------------------------------------------
516 | (AgentA pid=935731) user (to alice):
517 | (AgentA pid=935731) bob (to alice):
518 | (AgentA pid=935731) 我是Bob,Alice的好朋友,很高兴认识你!
519 | ```
520 |
521 |
522 |
523 | ## 4. Agent Termination
524 |
525 | If you want to terminate the conversation between two agents, you have the following options:
526 |
527 | 1. set the `max_consecutive_auto_reply` to 0 when you create agent. it prevents the agent from engaging in further automatic replies after receiving a response from Other agent.
528 |
529 | 2. Return False/True,None in the reply function, the None will stop sending the message to the other agent.
530 |
531 | 3. return False/True,{"content":"xxx",{"metadata":{"TERMINATE":True}}, the metadata will indicate the other agent that do not reply to this message.
532 |
533 | Please take care of the termination of the conversation, otherwise, the conversation will never stop.
534 |
535 | ## 5. Agent Stream Reply
536 |
537 | If you want to reply to the user in a stream way, you can use the following code:
538 |
539 | ```python
540 | @byzerllm.agent_reply()
541 | def my_reply():
542 | return self.stream_reply(response_gen=["Hello, world!","How are you?"])
543 | ```
544 |
545 | Then you can use the following code to get the stream reply:
546 |
547 | ```python
548 |
549 | message = ... from some agent
550 | metadata = message["metadata"]
551 |
552 | if "stream" in metadata and metadata["stream"]:
553 | agent = metadata["agent"]
554 | stream_id = metadata["stream_id"]
555 | result = []
556 | for item in YOUR_AGENT.get_agent_stream_messages(agent,stream_id):
557 | t = item
558 | result.append(t[0])
559 | yield t
560 | ```
561 |
562 |
563 |
564 |
565 |
566 |
567 |
568 |
569 |
570 |
571 |
572 |
--------------------------------------------------------------------------------
/images/CustomAgent.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/allwefantasy/byzer-agent/613a0c71ee9bfedbcc0d0f729e4a0347eb4a2625/images/CustomAgent.jpg
--------------------------------------------------------------------------------
/images/byzer-agent.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/allwefantasy/byzer-agent/613a0c71ee9bfedbcc0d0f729e4a0347eb4a2625/images/byzer-agent.png
--------------------------------------------------------------------------------
/images/byzer-tools.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/allwefantasy/byzer-agent/613a0c71ee9bfedbcc0d0f729e4a0347eb4a2625/images/byzer-tools.jpg
--------------------------------------------------------------------------------
/images/multi-agent.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/allwefantasy/byzer-agent/613a0c71ee9bfedbcc0d0f729e4a0347eb4a2625/images/multi-agent.jpg
--------------------------------------------------------------------------------
/notebooks/quick_rag.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "code",
5 | "execution_count": 1,
6 | "metadata": {},
7 | "outputs": [
8 | {
9 | "name": "stderr",
10 | "output_type": "stream",
11 | "text": [
12 | "2023-12-14 21:15:57,069\tINFO worker.py:1489 -- Connecting to existing Ray cluster at address: 192.168.1.248:6379...\n",
13 | "2023-12-14 21:15:57,081\tINFO worker.py:1664 -- Connected to Ray cluster. View the dashboard at \u001b[1m\u001b[32m127.0.0.1:8265 \u001b[39m\u001b[22m\n"
14 | ]
15 | },
16 | {
17 | "name": "stdout",
18 | "output_type": "stream",
19 | "text": [
20 | "Failed to look up actor with name 'emb'. This could because 1. You are trying to look up a named actor you didn't create. 2. The named actor died. 3. You did not use a namespace matching the namespace of the actor.\n",
21 | "\u001b[36m(UDFWorker pid=534687, ip=192.168.1.243)\u001b[0m MODEL[emb] Init Model,It may take a while.\n",
22 | "\u001b[36m(UDFWorker pid=534687, ip=192.168.1.243)\u001b[0m MODEL[emb] Local mode: Load model from local path (/home/byzerllm/models/bge-large-zh), consume the model server to prevent socket server leak.\n",
23 | "\u001b[36m(UDFWorker pid=534688, ip=192.168.1.243)\u001b[0m MODEL[emb] Init Model,It may take a while.\n",
24 | "\u001b[36m(UDFWorker pid=534688, ip=192.168.1.243)\u001b[0m MODEL[emb] Local mode: Load model from local path (/home/byzerllm/models/bge-large-zh), consume the model server to prevent socket server leak.\n",
25 | "\u001b[36m(UDFWorker pid=534688, ip=192.168.1.243)\u001b[0m MODEL[emb] Successful to init model, time taken:3.4095799922943115s\n"
26 | ]
27 | },
28 | {
29 | "name": "stdout",
30 | "output_type": "stream",
31 | "text": [
32 | "\u001b[36m(UDFWorker pid=534687, ip=192.168.1.243)\u001b[0m MODEL[emb] Successful to init model, time taken:3.446164608001709s\n"
33 | ]
34 | }
35 | ],
36 | "source": [
37 | "code_search_path=[\"/home/byzerllm/softwares/byzer-retrieval-lib/\"]\n",
38 | "env_vars = {\"JAVA_HOME\": \"/home/byzerllm/softwares/jdk-21\",\n",
39 | " \"PATH\":\"/home/byzerllm/softwares/jdk-21/bin:/home/byzerllm/.rvm/gems/ruby-3.2.2/bin:/home/byzerllm/.rvm/gems/ruby-3.2.2@global/bin:/home/byzerllm/.rvm/rubies/ruby-3.2.2/bin:/home/byzerllm/.rbenv/shims:/home/byzerllm/.rbenv/bin:/home/byzerllm/softwares/byzer-lang-all-in-one-linux-amd64-3.3.0-2.3.7/jdk8/bin:/usr/local/cuda/bin:/usr/local/cuda/bin:/home/byzerllm/.rbenv/shims:/home/byzerllm/.rbenv/bin:/home/byzerllm/miniconda3/envs/byzerllm-dev/bin:/home/byzerllm/miniconda3/condabin:/home/byzerllm/.local/bin:/home/byzerllm/bin:/usr/local/cuda/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/byzerllm/.rvm/bin:/home/byzerllm/.rvm/bin\"}\n",
40 | "\n",
41 | "import os\n",
42 | "os.environ[\"RAY_DEDUP_LOGS\"] = \"0\" \n",
43 | "\n",
44 | "from byzerllm.utils.client import Templates\n",
45 | "\n",
46 | "import ray\n",
47 | "from byzerllm.utils.retrieval import ByzerRetrieval\n",
48 | "from byzerllm.utils.client import ByzerLLM,LLMRequest,LLMResponse,LLMHistoryItem,InferBackend\n",
49 | "from byzerllm.records import SearchQuery\n",
50 | "\n",
51 | "ray.init(address=\"auto\",namespace=\"default\",ignore_reinit_error=True,\n",
52 | " job_config=ray.job_config.JobConfig(code_search_path=code_search_path,\n",
53 | " runtime_env={\"env_vars\": env_vars})\n",
54 | " ) \n",
55 | "\n",
56 | "# ray.init(address=\"auto\",namespace=\"default\") \n",
57 | "\n",
58 | "retrieval = ByzerRetrieval()\n",
59 | "retrieval.launch_gateway()\n",
60 | "\n",
61 | "llm = ByzerLLM()\n",
62 | "chat_model_name = \"chat\"\n",
63 | "\n",
64 | "# llm.undeploy(chat_model_name)\n",
65 | "# llm.undeploy(\"emb\")\n",
66 | "\n",
67 | "\n",
68 | "# model_type = \"deepseek\"\n",
69 | "# model_type = \"openbuddy\"\n",
70 | "# model_type = \"Qwen\"\n",
71 | "model_type = \"zephyr\"\n",
72 | "\n",
73 | "if model_type == \"deepseek\":\n",
74 | " model_location=\"/home/byzerllm/models/deepseek2-llm-67b-chat\"\n",
75 | " max_model_len = 4096\n",
76 | " gpus_per_worker = 8\n",
77 | "\n",
78 | "if model_type == \"llama2\":\n",
79 | " model_location=\"/home/byzerllm/models/openbuddy-llama2-70b-v10.1-bf16\"\n",
80 | " max_model_len = 4000\n",
81 | " gpus_per_worker = 8\n",
82 | "\n",
83 | "if model_type == \"Qwen\":\n",
84 | " llm.setup_template(\"chat\",Templates.qwen())\n",
85 | " model_location=\"/home/byzerllm/models/Qwen-72B-Chat\"\n",
86 | " max_model_len = 24000\n",
87 | " gpus_per_worker = 8\n",
88 | "\n",
89 | "if model_type == \"zephyr\": \n",
90 | " model_location=\"/home/byzerllm/models/openbuddy-zephyr-7b-v14.1\" \n",
91 | " max_model_len = 24000\n",
92 | " gpus_per_worker = 4\n",
93 | "\n",
94 | "llm.setup_max_model_length(\"chat\",max_model_len)\n",
95 | "\n",
96 | "if not llm.is_model_exist(\"chat\"):\n",
97 | " llm.setup_gpus_per_worker(gpus_per_worker).setup_num_workers(1).setup_infer_backend(InferBackend.VLLM)\n",
98 | " llm.deploy(\n",
99 | " model_path=model_location,\n",
100 | " pretrained_model_type=\"custom/auto\",\n",
101 | " udf_name=chat_model_name,\n",
102 | " infer_params={\"backend.max_num_batched_tokens\":24000,\n",
103 | " \"backend.max_model_len\":max_model_len}\n",
104 | " )\n",
105 | "\n",
106 | "\n",
107 | "if not llm.is_model_exist(\"emb\"):\n",
108 | " llm.setup_gpus_per_worker(0.4).setup_num_workers(2).setup_infer_backend(InferBackend.Transformers)\n",
109 | " llm.deploy(\n",
110 | " model_path=\"/home/byzerllm/models/bge-large-zh\",\n",
111 | " pretrained_model_type=\"custom/bge\",\n",
112 | " udf_name=\"emb\",\n",
113 | " infer_params={}\n",
114 | " ) \n",
115 | "\n",
116 | "llm.setup_default_model_name(chat_model_name) \n",
117 | "llm.setup_default_emb_model_name(\"emb\") \n",
118 | "\n",
119 | "# ray.kill(ray.get_actor(\"data_analysis\"))\n",
120 | "# 如果存储集群不存在，启动一个存储集群\n",
121 | "if not retrieval.is_cluster_exists(\"data_analysis\"):\n",
122 | " builder = retrieval.cluster_builder()\n",
123 | " builder.set_name(\"data_analysis\").set_location(\"/tmp/data_analysis\").set_num_nodes(2).set_node_cpu(1).set_node_memory(\"3g\")\n",
124 | " builder.set_java_home(env_vars[\"JAVA_HOME\"]).set_path(env_vars[\"PATH\"]).set_enable_zgc()\n",
125 | " builder.start_cluster() \n",
126 | "\n",
127 | "def show_code(lang,code_string):\n",
128 | " from IPython.display import display, Markdown \n",
129 | " display(Markdown(\"```{}\\n{}\\n```\".format(lang,code_string)))\n",
130 | "\n",
131 | "\n",
132 | "def show_text(msg):\n",
133 | " from IPython.display import display, Markdown\n",
134 | " display(Markdown(\"```{}\\n{}\\n```\".format(\"text\",msg))) \n",
135 | "\n",
136 | "def show_image(content):\n",
137 | " from IPython.display import display, Image\n",
138 | " import base64 \n",
139 | " img = Image(base64.b64decode(content))\n",
140 | " display(img) \n",
141 | " "
142 | ]
143 | },
144 | {
145 | "cell_type": "code",
146 | "execution_count": 2,
147 | "metadata": {},
148 | "outputs": [
149 | {
150 | "data": {
151 | "text/plain": [
152 | "[LLMResponse(output=[0.003487164154648781, -0.009687656536698341, -0.026311276480555534, -0.004805233795195818, 0.03333209827542305, 0.014389371499419212, 0.017725428566336632, -0.008188240230083466, 0.008683566004037857, 0.013795615173876286, -0.004662488587200642, -0.01721588335931301, 0.007674216292798519, -0.036708295345306396, -0.010143325664103031, 0.03057188168168068, 0.004054826684296131, 0.007601254619657993, 0.021168725565075874, -0.043669793754816055, -0.004535554442554712, -0.034899331629276276, -0.02656671404838562, -0.007852491922676563, -0.011296529322862625, 0.0011517584789544344, 0.003761491971090436, 0.036769669502973557, 0.005315430928021669, -0.0304831825196743, -0.03222383186221123, 0.03505197912454605, 0.016987627372145653, -0.0006790258339606225, 0.009743181988596916, 0.022959737107157707, -0.031860850751399994, 0.007448054384440184, -0.02996632643043995, -0.0014919526875019073, -0.013263804838061333, -0.007554023526608944, -0.01376862172037363, -0.034408316016197205, 0.018597548827528954, -0.0072209183126688, -0.027406541630625725, -0.011339222081005573, -0.020509768277406693, -0.009664623998105526, 0.0330386646091938, -0.006910271011292934, 0.013683156110346317, 0.031918250024318695, -0.0441196970641613, 0.036308735609054565, 0.010654173791408539, -0.04150593653321266, 0.011574207805097103, -0.03883562237024307, 0.006603465881198645, 0.17114593088626862, -0.02017168700695038, -0.010195241309702396, -0.000662007019855082, 0.0324428528547287, 0.013261805288493633, -0.036180686205625534, 0.01150499377399683, -0.006519390270113945, -0.004848468117415905, -0.017690105363726616, 0.0006613082950934768, 0.011129829101264477, 0.014227084815502167, 0.005412760190665722, -0.003477417165413499, 0.010960256680846214, 0.030958393588662148, 0.011563673615455627, -0.0014258075971156359, -0.008992396295070648, 0.0197122935205698, -0.010717540979385376, 0.03347568213939667, -0.014771472662687302, 0.026119044050574303, 0.6796592473983765, -0.03673512488603592, -0.009683356620371342, 0.013677976094186306, 0.006932745687663555, -0.021039269864559174, 0.016656849533319473, 0.014686436392366886, 0.006964570377022028, -0.000702394638210535, -0.0037960412446409464, -0.00881967507302761, 0.004468253813683987, 0.021274618804454803, -0.030099209398031235, -0.024472413584589958, 0.015028843656182289, -0.026518650352954865, 0.0031476416625082493, -0.021398331969976425, -0.042368289083242416, 0.008516741916537285, -0.003708734642714262, -0.00909481942653656, -0.015467310324311256, -0.011309328489005566, -0.009143365547060966, -0.012118346057832241, 0.0009249071590602398, 0.0032388344407081604, -0.0003274680348113179, 0.018032854422926903, -0.00741921691223979, -0.012998362071812153, 0.010867539793252945, 0.011595121584832668, -0.03515337407588959, 0.002084673149511218, -0.00404240982607007, 0.048867568373680115, 0.014629336073994637, 0.03691813349723816, 0.035860542207956314, -0.014592318795621395, 0.004479689989238977, -0.014250054024159908, -0.017803611233830452, -0.004467634484171867, 0.012042558752000332, -0.005335631780326366, -0.027443354949355125, -0.0005584975588135421, -0.000650092784781009, 0.025340920314192772, -0.01659531518816948, 0.023712394759058952, -0.023007571697235107, 0.0406491756439209, -0.030652888119220734, -0.033632807433605194, 0.017339827492833138, 0.04892687872052193, -0.047924358397722244, 0.041657812893390656, -0.007443091366440058, -0.057748645544052124, -0.011978920549154282, -0.02966626174747944, 0.002320491475984454, 0.027939576655626297, -0.007535999175161123, -0.016538871452212334, -0.0071924529038369656, 0.020517168566584587, -0.0072575840167701244, 0.0052316621877253056, -0.028796061873435974, -0.011635727249085903, -0.013494555838406086, 0.010666527785360813, 0.0002971715584862977, -0.019710777327418327, 0.03233019635081291, -0.030934609472751617, 0.01860910654067993, -0.00542945321649313, 0.02042807638645172, 0.009309951215982437, 0.025962520390748978, -0.003915196284651756, 0.04057596996426582, 0.01234076265245676, -0.0008757482864893973, 0.024085985496640205, -0.008505471982061863, 0.016658678650856018, -0.008786945603787899, -0.03596913442015648, 0.0007207594462670386, 0.01668531633913517, -0.006616797298192978, 0.014478928409516811, 0.02174672856926918, 0.0015613242285326123, 0.01445790659636259, 0.009462420828640461, -0.00515639828518033, 0.008152217604219913, -0.03592684864997864, -0.014523912221193314, -0.013357413932681084, 0.012347275391221046, -0.010361616499722004, -0.007132135797291994, -0.0029083117842674255, 0.041163258254528046, -0.004088228568434715, -0.012642704881727695, -0.0055987657979130745, 0.027468694373965263, -0.022684529423713684, 0.005965694785118103, 0.0007612057379446924, -0.023586304858326912, 0.010202751494944096, 0.031223636120557785, -0.012289216741919518, -0.02580418810248375, -0.0029186708852648735, -0.0026317269075661898, 0.016747640445828438, 0.019786138087511063, -0.0007009890396147966, 0.014967741444706917, -0.02064170315861702, 0.01338962186127901, -0.028340231627225876, 0.02232782542705536, 0.017966287210583687, 0.004491711966693401, -0.0016070770798251033, 0.005132701247930527, -0.028016678988933563, -0.03519721329212189, -0.004375019110739231, 0.004931834060698748, -0.026293016970157623, -0.022160720080137253, -0.021147703751921654, -0.008610487915575504, 0.011130276136100292, -0.07148369401693344, -0.00416305148974061, -0.000805175572168082, -0.002656482858583331, -0.021040773019194603, -0.04544305428862572, 0.01453280821442604, 0.00682342192158103, -0.02436007559299469, 0.018760843202471733, 0.025740956887602806, 0.008858508430421352, -0.01149742305278778, -0.0247403085231781, -0.026157358661293983, -0.07064972817897797, 0.024480178952217102, -0.041041381657123566, -0.012050554156303406, -0.024317346513271332, 0.015078969299793243, 0.04207950457930565, -0.02701546624302864, -0.0336633138358593, 0.02473188005387783, -0.0034996250178664923, -0.009324630722403526, 0.021314484998583794, -0.0013195822248235345, 0.02465699426829815, -0.03275822475552559, 0.0070115067064762115, 0.0016848486848175526, -0.020131845027208328, -0.030705129727721214, -0.01643321104347706, 0.003340697381645441, 0.039595216512680054, -0.048301711678504944, 0.0066279033198952675, 0.021883903071284294, -0.023761160671710968, 0.06028076261281967, -0.02300170436501503, -0.025108633562922478, -0.004084618296474218, -0.0015410343185067177, -0.00048529644845984876, -0.03783874958753586, 0.008330194279551506, 0.014394514262676239, 0.015049230307340622, -0.01033846940845251, 0.022922445088624954, 0.019050363451242447, -0.039095938205718994, -0.0007338922005146742, 0.022936616092920303, 0.033093661069869995, -0.033233776688575745, -0.019860027357935905, -0.038662660866975784, -0.009848609566688538, -0.020621299743652344, 0.0049803960137069225, 0.025266768410801888, 0.010610099881887436, 0.02380465716123581, -0.02576163038611412, 0.0005014448543079197, -0.001553943264298141, 0.02610088139772415, 0.0061146714724600315, -0.03160164877772331, 0.02750554494559765, -0.0005143636954016984, 0.027386518195271492, -0.006704701576381922, 0.03896261379122734, 0.002452149987220764, -0.010115906596183777, 0.03585993871092796, 0.0042442865669727325, 0.016008326783776283, -0.025066222995519638, -0.03984496742486954, 0.028144018724560738, -0.002400975441560149, -0.003985360264778137, 0.016985299065709114, -0.0005925213918089867, -0.003142156871035695, 0.02630738727748394, 0.021557435393333435, -0.041893020272254944, 0.02467920631170273, -0.022686608135700226, 0.0021290306467562914, 0.023899594321846962, 0.0020760393235832453, -0.03380101919174194, 0.025044647976756096, 0.01835312508046627, 0.046550165861845016, 0.009988140314817429, -0.022746054455637932, -0.004761714953929186, -0.02097194641828537, -0.0012335729552432895, -0.03015459142625332, 0.021233169361948967, 0.00766024412587285, 0.0041959211230278015, 0.040278881788253784, 0.020252792164683342, -0.015014105476439, 0.004885601811110973, -0.017425110563635826, -0.001719424850307405, -0.0048691523261368275, 0.0005406413110904396, 0.023354997858405113, -0.01226485799998045, 0.009801914915442467, -0.0038793564308434725, 0.01994386501610279, -0.025081075727939606, -0.003334210952743888, -0.033486757427453995, 0.0005723822978325188, -0.05995246022939682, 0.007975487969815731, 0.025314727798104286, 0.000668104097712785, 0.016089992597699165, -0.04431256651878357, 0.01583111099898815, -0.02815023995935917, -0.004461910575628281, 0.016702432185411453, -0.02732214890420437, 0.008623084984719753, -0.027477486059069633, 0.010453798808157444, 0.03260448947548866, -0.0026187747716903687, 0.02661612629890442, -0.007653359789401293, 0.032506901770830154, 0.013677278533577919, 0.022244414314627647, 0.012560770846903324, -0.022455357015132904, -0.02440635673701763, 0.01599557138979435, -0.028186017647385597, -0.011854746378958225, 0.0013135905610397458, -0.009951449930667877, -0.008932115510106087, -0.03078976646065712, 0.027642086148262024, -0.002188103972002864, -0.03304843232035637, 0.025286225602030754, -0.2091224193572998, 0.027270575985312462, -0.03535797446966171, -0.009781043976545334, -0.019557591527700424, 0.032985132187604904, -0.0020474595949053764, -0.023344777524471283, -0.0426286980509758, -0.012007934972643852, -0.03006923198699951, 0.0007082742522470653, -0.020964475348591805, 0.021713698282837868, 0.011858331970870495, -0.0028324162121862173, -0.02072029747068882, 0.052678294479846954, -0.026862138882279396, 0.0208378117531538, 0.02712433785200119, 0.006862398702651262, 0.007543110754340887, 0.022442767396569252, 0.037888698279857635, 0.007933497428894043, 0.003239267971366644, 0.018849477171897888, -2.5156632545986213e-05, -0.017728624865412712, 0.02303941920399666, 0.007537879049777985, 0.022821828722953796, -0.006924267392605543, -0.006510318256914616, 0.03094233199954033, 0.015414939261972904, 0.02311880886554718, -0.016332542523741722, -0.02819601446390152, -0.008539386093616486, 0.008599283173680305, -0.00035821579513140023, -0.005728539079427719, -0.04077481850981712, -0.0374082587659359, -0.008709946647286415, 0.014195548370480537, -0.02397126890718937, 0.03182493522763252, 0.02329140156507492, -0.0005057370872236788, -0.013419192284345627, -0.0022248360328376293, -0.0019482006318867207, -0.008630066178739071, 0.0008408388821408153, -0.00757185323163867, -0.004615491256117821, 0.01832573674619198, -0.007680994458496571, -0.02333017997443676, 0.015432213433086872, -0.009587675333023071, 0.022644266486167908, 0.00421725120395422, 0.025885742157697678, -0.02249753847718239, 0.007867870852351189, 0.0015404230216518044, -0.02654317207634449, 0.002080630511045456, 0.02641434408724308, 0.02655085176229477, 0.028145700693130493, -0.021776242181658745, 0.023981569334864616, 0.011771087534725666, -0.003392221638932824, -0.035160474479198456, 0.028622962534427643, -0.0004344264161773026, 0.014902891591191292, -0.002243954688310623, 0.0013095729518681765, -0.04524563252925873, 0.004456636030226946, -0.004449646454304457, -0.00826727319508791, 0.01495937630534172, 0.019716791808605194, -0.0395483635365963, 0.001187824411317706, -0.01051784586161375, 0.0023222605232149363, 0.02146873064339161, 0.0045225429348647594, 0.01389208622276783, -0.00550555856898427, -0.006299915723502636, 0.014651183038949966, -0.008735722862184048, 0.008793179877102375, -0.0015218026237562299, -0.0028998625930398703, -0.025110550224781036, 0.03239241987466812, -0.023645328357815742, -0.016779035329818726, -0.008243058808147907, 0.022374486550688744, -0.0041848099790513515, -0.009755874052643776, 0.0699262022972107, 0.006078628357499838, 0.03077680617570877, 0.019532181322574615, 0.019421612843871117, 0.00958671048283577, 0.03784814476966858, 0.019062835723161697, 0.02165999449789524, -0.04340424761176109, 0.009812338277697563, -0.004814330022782087, -0.0011423665564507246, -0.004818270448595285, -0.03462719917297363, -0.02745284140110016, -0.05128570273518562, 0.0048982840962708, -0.002057109260931611, 0.012723682448267937, -0.004870827775448561, 0.0021779348608106375, -0.0011722864583134651, 0.010608060285449028, -0.004929748363792896, 0.010195843875408173, 0.000712362234480679, -0.018579507246613503, 0.006589855067431927, -0.027700915932655334, -0.020488061010837555, 0.016283748671412468, 0.010732755996286869, 0.012920831330120564, -0.00937565416097641, 0.010217326693236828, 0.017267189919948578, -0.05158461257815361, -0.025182759389281273, 0.005485102068632841, -0.02903989888727665, -0.03321501985192299, 0.013596141710877419, 0.010326729156076908, -0.004942576866596937, 0.002241370966657996, 0.0013867142843082547, -0.004122150596231222, 0.008285152725875378, 0.007885832339525223, -0.042544323951005936, -0.005666619632393122, -0.0009580635814927518, 0.012412006966769695, 0.006386008113622665, -0.004307236056774855, -0.07244092226028442, 0.027071408927440643, 0.010087007656693459, -0.029685815796256065, -0.004104976076632738, 0.008258511312305927, -0.02007545530796051, 0.0005800002254545689, -0.0029284143820405006, 0.0009169483673758805, -0.005889962427318096, -0.008580750785768032, 0.016074344515800476, 9.80365221039392e-05, -0.002231647027656436, -0.010334480553865433, -0.015868421643972397, 0.016856057569384575, 0.005081677343696356, 0.002056341851130128, -0.04106210544705391, 0.015522105619311333, 0.0047235144302248955, -0.00966882798820734, 0.0047174557112157345, 0.0049748653545975685, 0.01667879894375801, -0.008821837604045868, -0.021471302956342697, 0.002877107122913003, -0.006226821336895227, -0.0203151423484087, -0.021684791892766953, 0.010743357241153717, 0.00913991779088974, 0.004740656819194555, -0.007322159595787525, 0.016730116680264473, -0.018723344430327415, -0.009738635271787643, -0.000291364238364622, 0.006282579619437456, 0.003145459806546569, -0.02776404097676277, -0.018273627385497093, 0.04322375729680061, -0.01053323782980442, -0.02860885299742222, 0.01276377122849226, 0.04144861176609993, -0.015284676104784012, -0.026345200836658478, -0.028101766481995583, 0.015193703584372997, -0.03196451812982559, 0.014859961345791817, 0.01481930073350668, -0.018974918872117996, 0.009569328278303146, 0.03562072664499283, -0.04404635727405548, -0.0056460206396877766, 0.026976536959409714, -0.026551127433776855, -0.0031852456741034985, 0.024918364360928535, 0.02589736506342888, -0.005574916489422321, -0.003448277013376355, -0.018013782799243927, -0.003824896877631545, -0.008955924771726131, -0.006724239327013493, -0.028318079188466072, 0.0074127051047980785, -0.016823988407850266, -0.022451670840382576, -0.015015190467238426, -0.01244905311614275, -0.02381324954330921, -0.0077910958789289, 0.0013481685891747475, -0.014487778767943382, 0.01595621183514595, -0.0034461221657693386, -0.015891429036855698, 0.020495804026722908, 0.012493071146309376, -0.005790275521576405, -0.006006727460771799, -0.004146607127040625, 0.007108352612704039, -0.014651152305305004, -0.04931686446070671, 0.017160987481474876, 0.04614429548382759, 0.004196841735392809, -0.0002881951513700187, -0.012310288846492767, -0.004288583528250456, -0.012283885851502419, 0.012658577412366867, 0.017172737047076225, -0.009416626766324043, 0.008621850982308388, -0.01105703879147768, 0.004364383406937122, 0.004409303423017263, 0.017574066296219826, -0.07420620322227478, 0.012963857501745224, -0.02328185737133026, 0.008682440966367722, -0.03857184946537018, -0.02588002011179924, 0.03541257232427597, 0.010199875570833683, 0.003953305538743734, -0.0456959530711174, -0.005610710475593805, 0.012816651724278927, 0.020729098469018936, 0.001479317550547421, 0.01728593371808529, 0.02349201589822769, -0.036373525857925415, 0.006792224477976561, 0.047496676445007324, -0.04227810725569725, 0.044918011873960495, -0.027313312515616417, 0.001485234359279275, 0.04195864126086235, 0.01850823499262333, 0.014458553865551949, 0.0007959147915244102, -0.0035793643910437822, -0.014250915497541428, -0.042456742376089096, -0.03909331187605858, 0.0018750972812995315, -0.03276743367314339, -0.007875500246882439, -0.01903262548148632, -0.020034924149513245, 0.008563929237425327, -0.013778594322502613, 0.0040899342857301235, -0.015156956389546394, 0.017067372798919678, -0.01800251565873623, -0.006237392779439688, -0.004381848499178886, -0.03059084713459015, 0.015515659004449844, 0.014665579423308372, 0.010791747830808163, -0.0041982620023190975, 0.036634720861911774, -7.476412429241464e-05, -0.011273895390331745, -0.009780917316675186, 0.015465553849935532, -0.010003117844462395, -0.011319766752421856, -0.05029883608222008, -0.027274155989289284, -0.012175329960882664, -0.011207429692149162, 0.038728877902030945, 0.018526911735534668, 0.028016095981001854, -0.05741608887910843, -0.0441848449409008, 0.0007920467178337276, 0.0029930616728961468, 0.015235758386552334, -0.014595159329473972, -0.028311071917414665, 0.003402797970920801, -0.013912631198763847, -0.025469908490777016, 0.001964088762179017, 0.022597473114728928, -0.0026342759374529123, -0.004093795083463192, -0.012308330275118351, 0.007577384822070599, -0.004834438674151897, -0.01316345389932394, -0.00784927885979414, -0.02260562963783741, -0.020643990486860275, -0.033291444182395935, -0.027470029890537262, -0.010541259311139584, -0.006558713503181934, 0.018972892314195633, -0.0183301642537117, 0.024640390649437904, -0.02136806771159172, -0.006239387672394514, -0.010789881460368633, -0.025372672826051712, 0.024040713906288147, 0.015895482152700424, -0.010213576257228851, -0.024381348863244057, -0.0448463000357151, -0.004667638335376978, 0.02452213689684868, -0.051004838198423386, -0.04308260977268219, 0.0028385701589286327, 0.03381859138607979, 0.005350567400455475, -0.00975355040282011, -0.022160837426781654, 0.020557278767228127, 0.011655430309474468, 0.013543829321861267, -0.03770613297820091, -0.020391646772623062, 0.00497646676376462, -0.02548833191394806, 0.012297675013542175, -0.017847741022706032, -0.0067238896153867245, -0.04312867298722267, 0.007704803720116615, -0.021088413894176483, 0.0039205471985042095, -0.021554043516516685, -0.020493799820542336, 0.0011133376974612474, -0.034021273255348206, -0.034975796937942505, 0.013115854933857918, -0.01868431642651558, -0.03380630165338516, 0.018172260373830795, -0.008293910883367062, -0.00013101330841891468, 0.031108330935239792, 0.018828511238098145, -0.028270458802580833, 0.014735528267920017, 0.002914264565333724, 0.029054060578346252, 0.00045350147411227226, -0.0211033895611763, 0.010446160100400448, 0.023129625245928764, 0.01749914139509201, -0.008772444911301136, 0.035053323954343796, -0.03067728877067566, 0.004595721606165171, -0.016242122277617455, -0.003366845427080989, -0.002589859301224351, 0.025029564276337624, -0.0170635674148798, 0.018598606809973717, -0.016992313787341118, -0.032109666615724564, 0.009626663289964199, 0.001241601537913084, -0.013411203399300575, -0.022111525759100914, 0.018503334373235703, -0.026170773431658745, -0.014047595672309399, 0.0018095612758770585, 0.0007818780140951276, 0.029101718217134476, -0.010108111426234245, 0.0003786681336350739, -0.013199646025896072, -0.0006521902978420258, 0.038488637655973434, 0.032452281564474106, -0.013090711086988449, 0.04097181186079979, 0.020671872422099113, -0.012920120730996132, 0.00041757512371987104, -0.01123118493705988, -0.023483628407120705, -0.020343493670225143, 0.0013833548873662949, 0.010726730339229107, -0.005514190997928381, -0.00679475674405694, -0.02707125060260296, 0.002245566574856639, 0.023366492241621017, 0.032739054411649704, 0.028669873252511024, 0.006854634266346693, -0.008089341223239899, 0.012980143539607525, -0.00509857852011919, -0.0022831056267023087, -0.0256450567394495, 0.0035622180439531803, -0.0035301006864756346, 0.01533513143658638, -0.014974132180213928, 0.03006555326282978, -0.025818681344389915, -0.02253497950732708, 0.012122238986194134, -0.024009786546230316, -0.013579520396888256, -0.014208106324076653, 0.03109326958656311, 0.0002320162020623684, -0.00023528176825493574, -0.014989085495471954, -0.004432328045368195, 0.03095516562461853, 0.009784413501620293, -0.006058461498469114, -0.030436335131525993, -0.01728566363453865, -0.01724725216627121, -0.009757598862051964, 0.006680123042315245, 0.011081486940383911, 0.014647094532847404, 0.01161315105855465, 0.017406942322850227, 0.0032805914524942636, -0.02726162225008011, 0.006285037379711866, 0.00023638691345695406, 0.008495407178997993, -0.012316248379647732, -0.020926538854837418, -0.015125715173780918, 0.0020211702212691307, -0.006255616433918476, 0.023308975622057915, -0.0011762744979932904, -0.008038735948503017, -0.013104522600769997, -0.0033966589253395796, 0.002647778484970331, 0.02373378723859787, -0.008094641380012035, -0.015157176181674004, 0.007313108071684837, -0.001794176991097629, -0.013907947577536106, -0.020762789994478226, -0.0453525148332119, -0.039866719394922256, -0.030962055549025536, 0.02097724936902523, -0.03887280449271202, 0.011119849048554897, -0.011271895840764046, 0.03163275122642517, 0.029161088168621063, -0.008770296350121498, -0.01686720736324787, 0.00319055188447237, -0.002703374717384577, 0.0016899722395464778, -0.04344872385263443, 0.0026298724114894867, 0.00607090350240469, -0.01238968688994646, 0.012830028310418129, -0.022178171202540398, -0.024423353374004364, -0.006215061992406845, -0.014774901792407036, -0.05375397950410843, 0.017746206372976303, -0.01663937233388424, -0.007147969678044319, -0.035357411950826645, -0.0022933799773454666, 0.006639537867158651, 0.008462690748274326, -0.03339289873838425, 0.020321397110819817, 0.0007797889411449432, -2.2001282559358515e-05, 0.02070847898721695, -0.005172993056476116, -0.03954135999083519, 0.0014728072565048933, 0.005381183233112097, 0.025643538683652878, 0.004573381505906582, -0.014447255991399288, -0.016562940552830696, 0.0006292092148214579, -0.008723686449229717, -0.046139031648635864, -0.00773534644395113, -0.026355640962719917, -0.059140581637620926, 0.010546548292040825, -0.0022316824179142714, -0.03412408381700516, 0.009994631633162498, 0.06007905676960945, 0.03250957280397415, -0.007714328821748495, 0.004486148711293936, -0.003379211528226733, 0.013348080217838287, -0.012157591991126537, -0.014410416595637798, 0.02637338638305664, -0.03588853031396866, 0.04094064235687256, -0.00044753748807124794, 0.02942517399787903, 0.0002796211338136345, 0.021382110193371773, 0.013867533765733242, -0.03270706534385681, 0.012606589123606682, -0.024081112816929817, -0.00666817044839263, -0.02813187800347805, 0.008712918497622013, 0.0015491079539060593, -0.010899409651756287, 0.013588478788733482, 0.009669236838817596, -0.016476178541779518, 0.015617279335856438, -0.01465313509106636, 0.009878525510430336, 0.0073933107778429985, -0.01459471508860588, -0.011199007742106915, -0.019285207614302635, -0.03990950807929039, 0.019521983340382576, 0.0033835868816822767, -0.0011618291027843952, 0.003699801629409194, -0.009871391579508781, 0.009854156523942947, 0.027680665254592896], input={'instruction': '我试试看能不能把这句话转换成向量', 'embedding': True, 'max_length': 4096, 'top_p': 0.95, 'temperature': 0.1}, metadata={})]"
153 | ]
154 | },
155 | "execution_count": 2,
156 | "metadata": {},
157 | "output_type": "execute_result"
158 | }
159 | ],
160 | "source": [
161 | "llm.emb(\"emb\",LLMRequest(instruction=\"我试试看能不能把这句话转换成向量\"))"
162 | ]
163 | },
164 | {
165 | "cell_type": "code",
166 | "execution_count": 3,
167 | "metadata": {},
168 | "outputs": [],
169 | "source": [
170 | "from byzerllm.apps.agent import Agents\n",
171 | "\n",
172 | "from byzerllm.apps.agent.user_proxy_agent import UserProxyAgent\n",
173 | "from byzerllm.apps.agent.extensions.retrieval_agent import RetrievalAgent\n",
174 | "\n",
175 | "\n",
176 | "retrieval_agent = Agents.create_local_agent(RetrievalAgent,\"retrieval_agent\",llm,retrieval,\n",
177 | " max_consecutive_auto_reply=1000,\n",
178 | " code_agent = None\n",
179 | " )\n",
180 | "\n",
181 | "user = Agents.create_local_agent(UserProxyAgent,\"user\",llm,retrieval,\n",
182 | " human_input_mode=\"NEVER\",\n",
183 | " max_consecutive_auto_reply=0)\n",
184 | "\n"
185 | ]
186 | },
187 | {
188 | "cell_type": "code",
189 | "execution_count": 5,
190 | "metadata": {},
191 | "outputs": [
192 | {
193 | "name": "stdout",
194 | "output_type": "stream",
195 | "text": [
196 | "user (to retrieval_agent):\n",
197 | "\n",
198 | "介绍下 Gluten\n",
199 | "\n",
200 | "--------------------------------------------------------------------------------\n",
201 | "retrieval_agent (to user):\n",
202 | "\n",
203 | "Gluten是Apache Spark的一个本地化计算引擎，它结合了Spark的可扩展性框架、优化器、容错能力和Native执行库的高性能，以提高数据处理速度并降低计算成本。它可以与Velox或Clickhouse集成。BIGO大数据团队是参与该项目的研发团队之一，并已在生产环境中使用Gluten替代Spark的ETL工作负载，实现了成本节省。\n",
204 | "\n",
205 | "--------------------------------------------------------------------------------\n"
206 | ]
207 | }
208 | ],
209 | "source": [
210 | "user.initiate_chat(\n",
211 | "retrieval_agent,\n",
212 | "message={\n",
213 | " \"content\":\"介绍下 Gluten\",\n",
214 | " \"metadata\":{\n",
215 | " \"file_path\":\"/home/byzerllm/projects/jupyter-workspace/aritcle.txt\"\n",
216 | " }\n",
217 | "},)\n"
218 | ]
219 | }
220 | ],
221 | "metadata": {
222 | "kernelspec": {
223 | "display_name": "byzerllm-dev",
224 | "language": "python",
225 | "name": "python3"
226 | },
227 | "language_info": {
228 | "codemirror_mode": {
229 | "name": "ipython",
230 | "version": 3
231 | },
232 | "file_extension": ".py",
233 | "mimetype": "text/x-python",
234 | "name": "python",
235 | "nbconvert_exporter": "python",
236 | "pygments_lexer": "ipython3",
237 | "version": "3.10.11"
238 | }
239 | },
240 | "nbformat": 4,
241 | "nbformat_minor": 2
242 | }
243 |
--------------------------------------------------------------------------------