├── LICENSE
├── README.md
└── papers
└── T-eval_zh.md
/LICENSE:
--------------------------------------------------------------------------------
1 | MIT License
2 |
3 | Copyright (c) 2023 luban-agi
4 |
5 | Permission is hereby granted, free of charge, to any person obtaining a copy
6 | of this software and associated documentation files (the "Software"), to deal
7 | in the Software without restriction, including without limitation the rights
8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9 | copies of the Software, and to permit persons to whom the Software is
10 | furnished to do so, subject to the following conditions:
11 |
12 | The above copyright notice and this permission notice shall be included in all
13 | copies or substantial portions of the Software.
14 |
15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21 | SOFTWARE.
22 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 |
6 |
7 | # Awesome Tool Learning
8 |
9 | [](https://github.com/luban-agi/Awesome-Tool-Learning)
10 | [](https://opensource.org/licenses/MIT)
11 | 
12 | 
13 |
14 | Awesome papers and applications on tool learning.
15 |
16 | ## 📜 Table of Contents
17 |
18 | - [📚 Papers](#-papers)
19 | - [📑 Survey](#-survey)
20 | - [🏋️♂️ Tool Use Via Fine-tuning](#-tool-use-via-fine-tuning)
21 | - [📖 Tool Use Via In-Context Learning](#-tool-use-via-in-context-learning)
22 | - [🧪 Evalution](#-evalution)
23 | - [📱 Applications](#-applications)
24 | - [🎉 Contributors](#-contributors)
25 |
26 | ## 📚 Papers
27 |
28 | ### 📑 Survey
29 |
30 | - **Augmented Language Models: a Survey**, Preprint 2023.02
31 | *Grégoire Mialon, Roberto Dessì, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, Thomas Scialom* [[pdf](https://arxiv.org/abs/2302.07842)]
32 |
33 | - **Tool Learning with Foundation Models**, Preprint 2023.04
34 | *Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, Yi Ren Fung, Yusheng Su, Huadong Wang, Cheng Qian, Runchu Tian, Kunlun Zhu, Shihao Liang, Xingyu Shen, Bokai Xu, Zhen Zhang, Yining Ye, Bowen Li, Ziwei Tang, Jing Yi, Yuzhang Zhu, Zhenning Dai, Lan Yan, Xin Cong, Yaxi Lu, Weilin Zhao, Yuxiang Huang, Junxi Yan, Xu Han, Xian Sun, Dahai Li, Jason Phang, Cheng Yang, Tongshuang Wu, Heng Ji, Zhiyuan Liu, Maosong Sun* [[pdf](https://arxiv.org/abs/2304.08354)]
35 |
36 | - **A Survey on Large Language Model based Autonomous Agents**, Preprint 2023.08
37 | *Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, Ji-Rong Wen* [[pdf](https://arxiv.org/abs/2308.11432)]
38 |
39 | ### 🏋️♂️ Tool Use Via Fine-tuning
40 |
41 | - **WebGPT: Browser-assisted question-answering with human feedback** Preprint 2021.12
42 | *Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, John Schulman* [[pdf](https://arxiv.org/abs/2112.09332)]
43 |
44 | - **TALM: Tool Augmented Language Models** Preprint 2022.05
45 | *Aaron Parisi, Yao Zhao, Noah Fiedel* [[pdf](https://arxiv.org/abs/2205.12255)]
46 |
47 | - **WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents** NeurIPS 2022.07
48 | *Shunyu Yao, Howard Chen, John Yang, Karthik Narasimhan* [[pdf](https://arxiv.org/abs/2207.01206)] [[github](https://webshop-pnlp.github.io/)]
49 |
50 | - **Toolformer: Language Models Can Teach Themselves to Use Tools** Preprint 2023.02
51 | *Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, Thomas Scialom* [[pdf](https://arxiv.org/abs/2302.04761)]
52 |
53 | - **ToolCoder: Teach Code Generation Models to use API search tools** Preprint 2023.05
54 | *Kechi Zhang, Huangzhao Zhang, Ge Li, Jia Li, Zhuo Li, Zhi Jin* [[pdf](https://arxiv.org/abs/2305.04032)]
55 |
56 | - **WebCPM: Interactive Web Search for Chinese Long-form Question Answering** ACL 2023.05
57 | *Yujia Qin, Zihan Cai, Dian Jin, Lan Yan, Shihao Liang, Kunlun Zhu, Yankai Lin, Xu Han, Ning Ding, Huadong Wang, Ruobing Xie, Fanchao Qi, Zhiyuan Liu, Maosong Sun, Jie Zhou* [[pdf](https://arxiv.org/abs/2305.06849)]
58 |
59 | - **ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings** Preprint 2023.05
60 | *Shibo Hao, Tianyang Liu, Zhen Wang, Zhiting Hu* [[pdf](https://arxiv.org/abs/2305.11554)][[github](https://github.com/Ber666/ToolkenGPT)]
61 |
62 | - **Making Language Models Better Tool Learners with Execution Feedback** Preprint 2023.05
63 | *Shuofei Qiao, Honghao Gui, Huajun Chen, Ningyu Zhang* [[pdf](https://arxiv.org/abs/2305.13068)]
64 |
65 | - **Gorilla: Large Language Model Connected with Massive APIs** Preprint 2023.05
66 | *Shishir G. Patil, Tianjun Zhang, Xin Wang, Joseph E. Gonzalez* [[pdf](https://arxiv.org/abs/2305.15334)]
67 |
68 | - **GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction** Preprint 2023.05
69 | *Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, Ying Shan* [[pdf](https://arxiv.org/abs/2305.18752)] [[github](https://github.com/StevenGrove/GPT4Tools)]
70 |
71 | - **ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases** Preprint 2023.06
72 | *Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Le Sun* [[pdf](https://arxiv.org/abs/2306.05301)]
73 |
74 | - **WebGLM: Towards An Efficient Web-Enhanced Question Answering System with Human Preferences** KDD 2023.06
75 | *Xiao Liu, Hanyu Lai, Hao Yu, Yifan Xu, Aohan Zeng, Zhengxiao Du, Peng Zhang, Yuxiao Dong, Jie Tang* [[pdf](https://arxiv.org/abs/2306.07906)]
76 |
77 | ### 📖 Tool Use via In-Context Learning
78 |
79 | - **PAL: Program-aided Language Models** Preprint 2022.11
80 | *Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, Graham Neubig* [[pdf](https://arxiv.org/abs/2211.10435)] [[github](https://reasonwithpal.com/)]
81 |
82 | - **Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks** Preprint 2022.11
83 | *Wenhu Chen, Xueguang Ma, Xinyi Wang, William W. Cohen* [[pdf](https://arxiv.org/abs/2211.12588)]
84 |
85 | - **ART: Automatic multi-step reasoning and tool-use for large language models** Preprint 2023.03
86 | *Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, Marco Tulio Ribeiro* [[pdf](https://arxiv.org/abs/2303.09014)]
87 |
88 | - **TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs** Preprint 2023.03
89 | *Yaobo Liang, Chenfei Wu, Ting Song, Wenshan Wu, Yan Xia, Yu Liu, Yang Ou, Shuai Lu, Lei Ji, Shaoguang Mao, Yun Wang, Linjun Shou, Ming Gong, Nan Duan* [[pdf](https://arxiv.org/abs/2303.16434)]
90 |
91 | - **HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face** Preprint 2023.03
92 | *Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, Yueting Zhuang* [[pdf](https://arxiv.org/abs/2303.17580)]
93 |
94 | - **OpenAGI: When LLM Meets Domain Experts** Preprint 2023.04
95 | *Yingqiang Ge, Wenyue Hua, Kai Mei, Jianchao Ji, Juntao Tan, Shuyuan Xu, Zelong Li, Yongfeng Zhang* [[pdf](https://arxiv.org/abs/2304.04370)]
96 |
97 | - **ChemCrow: Augmenting large-language models with chemistry tools** Preprint 2023.04
98 | *Andres M Bran, Sam Cox, Andrew D White, Philippe Schwaller* [[pdf](https://arxiv.org/abs/2304.05376)]
99 |
100 | - **GeneGPT: Augmenting Large Language Models with Domain Tools for Improved Access to Biomedical Information** Preprint 2023.04
101 | *Qiao Jin, Yifan Yang, Qingyu Chen, Zhiyong Lu* [[pdf](https://arxiv.org/abs/2304.09667)]
102 |
103 | - **Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models** Preprint 2023.04
104 | *Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Jianfeng Gao* [[pdf](https://arxiv.org/abs/2304.09842)] [[github](https://chameleon-llm.github.io/)]
105 |
106 | - **ChatCoT: Tool-Augmented Chain-of-Thought Reasoning on Chat-based Large Language Models** Preprint 2023.05
107 | *Zhipeng Chen, Kun Zhou, Beichen Zhang, Zheng Gong, Wayne Xin Zhao, Ji-Rong Wen* [[pdf](https://arxiv.org/abs/2305.14323)]
108 |
109 | - **CREATOR: Disentangling Abstract and Concrete Reasonings of Large Language Models through Tool Creation** Preprint 2023.05
110 | *Cheng Qian, Chi Han, Yi R. Fung, Yujia Qin, Zhiyuan Liu, Heng Ji* [[pdf](https://arxiv.org/abs/2305.14318)]
111 |
112 | - **Large Language Models as Tool Makers** Preprint 2023.05
113 | *Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, Denny Zhou* [[pdf](https://arxiv.org/abs/2305.17126)] [[github](https://github.com/ctlllll/LLM-ToolMaker)]
114 |
115 | - **MultiTool-CoT: GPT-3 Can Use Multiple External Tools with Chain of Thought Prompting** ACL 2023.05
116 | *Tatsuro Inaba, Hirokazu Kiyomaru, Fei Cheng, Sadao Kurohashi* [[pdf](https://arxiv.org/abs/2305.16896)] [[project](https://github.com/InabaTatsuro/MultiTool-CoT)]
117 |
118 | - **RestGPT: Connecting Large Language Models with Real-World Applications via RESTful APIs** Preprint 2023.06
119 | *Yifan Song, Weimin Xiong, Dawei Zhu, Cheng Li, Ke Wang, Ye Tian, Sujian Li* [[pdf](https://arxiv.org/abs/2306.06624)]
120 |
121 | - **AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learn** Preprint 2023.06
122 | *Difei Gao, Lei Ji, Luowei Zhou, Kevin Qinghong Lin, Joya Chen, Zihan Fan, Mike Zheng Shou* [[pdf](https://arxiv.org/abs/2306.08640)] [[project](https://showlab.github.io/assistgpt/)]
123 |
124 | - **GEAR: Augmenting Language Models with Generalizable and Efficient Tool Resolution** Preprint 2023.07
125 | *Yining Lu, Haoping Yu, Daniel Khashabi* [[pdf](https://arxiv.org/abs/2307.08775)]
126 |
127 | - **Skills-in-Context Prompting: Unlocking Compositionality in Large Language Models** Preprint 2023.08
128 | *Jiaao Chen, Xiaoman Pan, Dian Yu, Kaiqiang Song, Xiaoyang Wang, Dong Yu, Jianshu Chen* [[pdf](https://arxiv.org/abs/2308.00304)]
129 |
130 | - **Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Models** Preprint 2023.08
131 | *Cheng-Yu Hsieh, Si-An Chen, Chun-Liang Li, Yasuhisa Fujii, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, Tomas Pfister* [[pdf](https://arxiv.org/abs/2308.00675)]
132 |
133 | - **TPTU: Task Planning and Tool Usage of Large Language Model-based AI Agents** Preprint 2023.08
134 | *Jingqing Ruan, Yihong Chen, Bin Zhang, Zhiwei Xu, Tianpeng Bao, Guoqing Du, Shiwei Shi, Hangyu Mao, Xingyu Zeng, Rui Zhao* [[pdf](https://arxiv.org/abs/2308.03427)]
135 |
136 | ### 🧪 Evalution
137 |
138 | - **tool_use_benchmark** [[github](https://github.com/luban-agi/tool_use_benchmark)]
139 | A large-scale benchmark for tool use, including four English tool-use datasets and one Chinese tool-use dataset.
140 |
141 | - **API-Bank: A Benchmark for Tool-Augmented LLMs** Preprint 2023.04
142 | *Minghao Li, Feifan Song, Bowen Yu, Haiyang Yu, Zhoujun Li, Fei Huang, Yongbin Li* [[pdf](https://arxiv.org/abs/2306.08640)]
143 |
144 | - **On the Tool Manipulation Capability of Open-source Large Language Models** Preprint 2023.05
145 | *Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, Jian Zhang* [[pdf](https://arxiv.org/abs/2305.16504)]
146 |
147 | - **Evaluating and Improving Tool-Augmented Computation-Intensive Math Reasoning** Preprint 2023.06
148 | *Beichen Zhang, Kun Zhou, Xilin Wei, Wayne Xin Zhao, Jing Sha, Shijin Wang, Ji-Rong Wen* [[pdf](https://arxiv.org/abs/2306.02408)]
149 |
150 | - **ToolQA: A Dataset for LLM Question Answering with External Tools** Preprint 2023.06
151 | *Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, Chao Zhangu* [[pdf](https://arxiv.org/abs/2306.13304)]
152 |
153 | - **ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs** Preprint 2023.07
154 | *Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, Maosong Sun* [[pdf](https://arxiv.org/abs/2307.16789)]
155 |
156 | - **AgentBench: Evaluating LLMs as Agents** Preprint 2023.08
157 | *Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, Jie Tang* [[pdf](https://arxiv.org/abs/2308.03688)]
158 |
159 | - **ToolTalk: Evaluating Tool-Usage in a Conversational Setting** Preprint 2023.11
160 | *Nicholas Farn, Richard Shin* [[pdf](https://arxiv.org/abs/2311.10775)]
161 |
162 | - **TaskBench: BENCHMARKING LARGE LANGUAGE MODELS FOR TASK AUTOMATION** Preprint 2023.11
163 | *Yongliang Shen1, Kaitao Song, et.al.*, [[pdf](https://arxiv.org/abs/2311.18760)]
164 |
165 | - **T-Eval: Evaluating the Tool Utilization Capability Step by Step** Preprint 2023.12
166 | *Zehui Chen, Feng Zhao, et.al.*, [[pdf](https://arxiv.org/abs/2312.14033)]
167 |
168 |
177 |
178 | ## 📱 Applications
179 |
180 | - [Auto-GPT](https://github.com/Significant-Gravitas/Auto-GPT): An experimental open-source attempt to make GPT-4 fully autonomous. 
181 |
182 | - [LangChain](https://github.com/langchain-ai/langchain): Building applications with LLMs through composability. 
183 |
184 | - [DB-GPT](https://github.com/eosphoros-ai/DB-GPT): Revolutionizing Database Interactions with Private LLM Technology. 
185 |
186 | ## 🎉 Contributors
187 |
188 |
189 | 
190 |
191 |
192 |
--------------------------------------------------------------------------------
/papers/T-eval_zh.md:
--------------------------------------------------------------------------------
1 |
2 |
3 | ## 指标
4 | 将 tool leanring 的评估分成了6个部分，分别是plan、reasone、retrieve、understand、review
5 |
6 | ### plan
7 | 对plans进行sbert 向量化，然后比较最长公共序列相似度
8 | ### reason
9 | 比较思考的相似度
10 | ### retrieve
11 | 工具检索的准确性
12 | ### understand
13 | 参数填参的准确性
14 | ### instruct
15 | 似乎是retrieve和instrct的结合
16 | ### review
17 | 判断这个工具是否完成这个子任务，类别准确性判断
18 |
19 | ## 数据
20 | multi-agent拆分任务到子agent + 人工校验数据
21 |
22 | ## 结论
23 | 大部分都不重要，后面比较不同模型在 T-eval 与 toolbench上的 score趋势，说明t-eval的评估策略与GPT-4评估基本一致
24 |
--------------------------------------------------------------------------------