├── general.png ├── LICENSE └── README.md /general.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/teacherpeterpan/self-correction-llm-papers/HEAD/general.png -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright [yyyy] [name of copyright owner] 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Self-Correction LLMs Papers 2 | 3 | [![Awesome](https://awesome.re/badge.svg)](https://github.com/sindresorhus/awesome) 4 | ![GitHub stars](https://img.shields.io/github/stars/teacherpeterpan/self-correction-llm-papers?color=yellow) 5 | ![GitHub forks](https://img.shields.io/github/forks/teacherpeterpan/self-correction-llm-papers?color=9cf) 6 | [![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://github.com/teacherpeterpan/self-correction-llm-papers/blob/main/LICENSE) 7 | 8 | This is a collection of research papers for **Self-Correcting Large Language Models with Automated Feedback.** 9 | 10 | Our survey paper: [Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies](https://arxiv.org/abs/2308.03188). Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, William Yang Wang 11 | 12 | ![A conceptual framework for self-correcting LLMs with automated feedback](./general.png) 13 | 14 | ## [Content](#content) 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 |
1. Training-Time Correction
 1.1 RLHF Strategy 1.2 Fine-tuning Strategy
 1.3 Self-Training Strategy
2. Generation-Time Correction
 2.1 Re-Ranking Strategy 2.2 Feedback-guided Strategy
3. Post-hoc Correction
 3.1 Self-Refine Strategy 3.2 External Feedback Strategy
 3.3 Model-Debate Strategy
38 | 39 | ## [Training-Time Correction](#content) 40 | 41 | ### [RLHF Strategy](#rlhf-strategy) 42 | 43 | 1. **Training Language Models to Follow Instructions with Human Feedback.** Advances in Neural Information Processing Systems (NeurIPS), 2022. [paper](https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf) 44 | 45 | *Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, Ryan Lowe* 46 | 47 | 2. **Fine-Grained Human Feedback Gives Better Rewards for Language Model Training.** arxiv, 2023. [paper](https://arxiv.org/pdf/2306.01693.pdf) 48 | 49 | *Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, Hannaneh Hajishirzi* 50 | 51 | 3. **Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback.** arxiv, 2022. [paper](https://arxiv.org/pdf/2204.05862.pdf) 52 | 53 | *Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, Jared Kaplan* 54 | 55 | 4. **Improving Alignment of Dialogue Agents via Targeted Human Judgments.** arxiv, 2022. [paper](https://arxiv.org/pdf/2209.14375.pdf) 56 | 57 | *Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, Lucy Campbell-Gillingham, Jonathan Uesato, Po-Sen Huang, Ramona Comanescu, Fan Yang, Abigail See, Sumanth Dathathri, Rory Greig, Charlie Chen, Doug Fritz, Jaume Sanchez Elias, Richard Green, Soňa Mokrá, Nicholas Fernando, Boxi Wu, Rachel Foley, Susannah Young, Iason Gabriel, William Isaac, John Mellor, Demis Hassabis, Koray Kavukcuoglu, Lisa Anne Hendricks, Geoffrey Irving* 58 | 59 | ### [Fine-tuning Strategy](#fine-tuning-strategy) 60 | 61 | 1. **Training Language Models with Language Feedback at Scale.** arxiv, 2023. [paper](https://arxiv.org/pdf/2303.16755.pdf) 62 | 63 | *Jérémy Scheurer, Jon Ander Campos, Tomasz Korbak, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, Ethan Perez* 64 | 65 | 2. **Continually Improving Extractive QA via Human Feedback.** arxiv, 2023. [paper](https://arxiv.org/pdf/2305.12473.pdf) 66 | 67 | *Ge Gao, Hung-Ting Chen, Yoav Artzi, Eunsol Choi* 68 | 69 | 3. **Chain of Hindsight Aligns Language Models with Feedback.** arxiv, 2023. [paper](https://arxiv.org/pdf/2302.02676.pdf) 70 | 71 | *Hao Liu, Carmelo Sferrazza, Pieter Abbeel* 72 | 73 | 4. **QUARK: Controllable Text Generation with Reinforced Unlearning.** Advances in Neural Information Processing Systems (NeurIPS), 2022. [paper](https://proceedings.neurips.cc/paper_files/paper/2022/file/b125999bde7e80910cbdbd323087df8f-Paper-Conference.pdf) 74 | 75 | *Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, Yejin Choi* 76 | 77 | 5. **SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization.** Annual Meeting of the Association for Computational Linguistics (ACL), 2021. [paper](https://aclanthology.org/2021.acl-short.135.pdf) 78 | 79 | *Yixin Liu, Pengfei Liu* 80 | 81 | 6. **BERTTune: Fine-Tuning Neural Machine Translation with BERTScore.** Annual Meeting of the Association for Computational Linguistics (ACL), 2021. [paper](https://arxiv.org/pdf/2106.02208.pdf) 82 | 83 | *Inigo Jauregi Unanue, Jacob Parnell, Massimo Piccardi* 84 | 85 | ### [Self-Training Strategy](#self-training-strategy) 86 | 87 | 1. **STaR: Bootstrapping Reasoning With Reasoning.** Advances in Neural Information Processing Systems (NeurIPS), 2022. [paper](https://openreview.net/pdf?id=_3ELRdg2sgI) 88 | 89 | *Eric Zelikman, Yuhuai Wu, Jesse Mu, Noah Goodman* 90 | 91 | 2. **SELF-INSTRUCT: Aligning Language Models with Self-Generated Instructions.** Annual Meeting of the Association for Computational Linguistics (ACL), 2023. [paper](https://aclanthology.org/2023.acl-long.754.pdf) 92 | 93 | *Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, Hannaneh Hajishirzi* 94 | 95 | 3. **Constitutional AI: Harmlessness from AI Feedback.** arxiv, 2022. [paper](https://arxiv.org/pdf/2212.08073.pdf) 96 | 97 | *Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, Jared Kaplan* 98 | 99 | 4. **Language Model Self-improvement by Reinforcement Learning Contemplation.** arxiv, 2023. [paper](https://arxiv.org/pdf/2305.14483.pdf) 100 | 101 | *Jing-Cheng Pang, Pengyuan Wang, Kaiyuan Li, Xiong-Hui Chen, Jiacheng Xu, Zongzhang Zhang, Yang Yu* 102 | 103 | 5. **Large Language Models Can Self-Improve.** arxiv, 2022. [paper](https://arxiv.org/pdf/2210.11610.pdf) 104 | 105 | *Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, Jiawei Han* 106 | 107 | 6. **AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback.** arxiv, 2023. [paper](https://arxiv.org/pdf/2305.14387.pdf) 108 | 109 | *Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, Tatsunori B. Hashimoto* 110 | 111 | ## [Generation-Time Correction](#content) 112 | 113 | ### [Re-Ranking Strategy](#re-ranking-strategy) 114 | 115 | 1. **Large Language Models are Better Reasoners with Self-Verification.** arXiv, 2023. [paper](https://arxiv.org/pdf/2212.09561.pdf) 116 | 117 | *Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Kang Liu, Jun Zhao* 118 | 119 | 2. **CodeT: Code Generation with Generated Tests.** International Conference on Learning Representations (ICLR), 2023. [paper](https://openreview.net/pdf?id=ktrw68Cmu9c) 120 | 121 | *Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, Weizhu Chen* 122 | 123 | 3. **LEVER: Learning to Verify Language-to-Code Generation with Execution.** International Conference on Machine Learning (ICML), 2023. [paper](https://arxiv.org/pdf/2302.08468.pdf) 124 | 125 | *Ansong Ni, Srini Iyer, Dragomir Radev, Ves Stoyanov, Wen-tau Yih, Sida I. Wang, Xi Victoria Lin* 126 | 127 | 4. **Rethinking with Retrieval: Faithful Large Language Model Inference.** arxiv, 2022. [paper](https://arxiv.org/pdf/2301.00303.pdf) 128 | 129 | *Hangfeng He, Hongming Zhang, Dan Roth* 130 | 131 | 5. **INSTRUCTSCORE: Towards Explainable Text Generation Evaluation with Automatic Feedback.** arxiv, 2023. [paper](https://arxiv.org/pdf/2305.14282.pdf) 132 | 133 | *Wenda Xu, Danqing Wang, Liangming Pan, Zhenqiao Song, Markus Freitag, William Yang Wang, Lei Li* 134 | 135 | 6. **High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metric.** Transactions of the Association for Computational Linguistics (TACL), 2022. [paper](https://aclanthology.org/2022.tacl-1.47.pdf) 136 | 137 | *Markus Freitag, David Grangier, Qijun Tan, Bowen Liang* 138 | 139 | 7. **Making Language Models Better Reasoners with Step-Aware Verifier.** Annual Meeting of the Association for Computational Linguistics (ACL), 2023. [paper](https://aclanthology.org/2023.acl-long.291.pdf) 140 | 141 | *Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, Weizhu Chen* 142 | 143 | 144 | ### [Feedback-guided Strategy](#feedback-guided-strategy) 145 | 146 | 1. **Let's Verify Step by Step.** arxiv, 2023. [paper](https://arxiv.org/pdf/2305.20050.pdf) 147 | 148 | *Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, Karl Cobbe* 149 | 150 | 2. **Diffusion-LM Improves Controllable Text Generation.** Advances in Neural Information Processing Systems (NeurIPS), 2022. [paper](https://arxiv.org/pdf/2205.14217.pdf) 151 | 152 | *Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, Tatsunori B. Hashimoto* 153 | 154 | 3. **FUDGE: Controlled Text Generation With Future Discriminators.** North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), 2021. [paper](https://aclanthology.org/2021.naacl-main.276.pdf) 155 | 156 | *Kevin Yang, Dan Klein* 157 | 158 | 4. **Entailer: Answering Questions with Faithful and Truthful Chains of Reasoning.** Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022. [paper](https://aclanthology.org/2022.emnlp-main.134.pdf) 159 | 160 | *Oyvind Tafjord, Bhavana Dalvi Mishra, Peter Clark* 161 | 162 | 5. **Generating Natural Language Proofs with Verifier-Guided Search.** Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022. [paper](https://aclanthology.org/2022.emnlp-main.7.pdf) 163 | 164 | *Kaiyu Yang, Jia Deng, Danqi Chen* 165 | 166 | 6. **Discriminator-Guided Multi-step Reasoning with Language Models.** arxiv, 2023. [paper](https://arxiv.org/pdf/2305.14934.pdf) 167 | 168 | *Muhammad Khalifa, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, Lu Wang* 169 | 170 | 7. **Solving Math Word Problems via Cooperative Reasoning Induced Language Models.** Annual Meeting of the Association for Computational Linguistics (ACL), 2023. [paper](https://aclanthology.org/2023.acl-long.245.pdf) 171 | 172 | *Xinyu Zhu, Junjie Wang, Lin Zhang, Yuxiang Zhang, Yongfeng Huang, Ruyi Gan, Jiaxing Zhang, Yujiu Yang* 173 | 174 | 8. **Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations.** Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022. [paper](https://aclanthology.org/2022.emnlp-main.82.pdf) 175 | 176 | *Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, Yejin Choi* 177 | 178 | 9. **Faithful Reasoning Using Large Language Models.** arxiv, 2022. [paper](https://arxiv.org/pdf/2208.14271.pdf) 179 | 180 | *Antonia Creswell, Murray Shanahan* 181 | 182 | 10. **Reasoning with Language Model is Planning with World Model.** arxiv, 2023. [paper](https://arxiv.org/pdf/2305.14992.pdf) 183 | 184 | *Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, Zhiting Hu* 185 | 186 | 11. **Decomposition Enhances Reasoning via Self-Evaluation Guided Decoding.** arxiv, 2023. [paper](https://arxiv.org/pdf/2305.00633.pdf) 187 | 188 | *Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, Xu Zhao, Min-Yen Kan, Junxian He, Qizhe Xie* 189 | 190 | 12. **Tree of Thoughts: Deliberate Problem Solving with Large Language Models.** arxiv, 2023. [paper](https://arxiv.org/pdf/2305.10601.pdf) 191 | 192 | *Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, Karthik Narasimhan* 193 | 194 | ## [Post-hoc Correction](#content) 195 | 196 | ### [Self-Refine Strategy](#self-refine-strategy) 197 | 198 | 1. **Self-Refine: Iterative Refinement with Self-Feedback.** arxiv, 2023. [paper](https://arxiv.org/pdf/2303.17651.pdf) 199 | 200 | *Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, Peter Clark* 201 | 202 | 2. **Self-Verification Improves Few-Shot Clinical Information Extraction.** arxiv,2023. [paper](https://arxiv.org/pdf/2306.00024.pdf) 203 | 204 | *Zelalem Gero, Chandan Singh, Hao Cheng, Tristan Naumann, Michel Galley, Jianfeng Gao, Hoifung Poon* 205 | 206 | 3. **Reflexion: Language Agents with Verbal Reinforcement Learning.** arxiv, 2023. [paper](https://arxiv.org/pdf/2303.11366.pdf) 207 | 208 | *Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, Shunyu Yao* 209 | 210 | 4. **Iterative Translation Refinement with Large Language Models.** arxiv, 2023. [paper](https://arxiv.org/pdf/2306.03856.pdf) 211 | 212 | *Pinzhen Chen, Zhicheng Guo, Barry Haddow, Kenneth Heafield* 213 | 214 | 5. **Leveraging GPT-4 for Automatic Translation Post-Editing.** arxiv, 2023. [paper](https://arxiv.org/pdf/2305.14878.pdf) 215 | 216 | *Vikas Raunak, Amr Sharaf, Hany Hassan Awadallah, Arul Menezes* 217 | 218 | 6. **Language Models can Solve Computer Tasks.** arxiv, 2023. [paper](https://arxiv.org/pdf/2303.17491.pdf) 219 | 220 | *Geunwoo Kim, Pierre Baldi, Stephen McAleer* 221 | 222 | 7. **SelFee: Iterative Self-Revising LLM Empowered by Self-Feedback Generation.** Blog Post, 2023. [website](https://kaistai.github.io/SelFee/) 223 | 224 | *Seonghyeon Ye, Yongrae Jo, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Minjoon Seo* 225 | 226 | 8. **SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models.** arxiv, 2023. [paper](https://arxiv.org/pdf/2303.08896.pdf) 227 | 228 | *Potsawee Manakul, Adian Liusie, Mark J. F. Gales* 229 | 230 | 9. **CLOVA: A Closed-LOop Visual Assistant with Tool Usage and Update** arxiv, 2023 [paper](https://arxiv.org/abs/2312.10908) 231 | 232 | *Zhi Gao, Yuntao Du, Xintong Zhang, Xiaojian Ma, Wenjuan Han, Song-Chun Zhu, Qing Li* 233 | 234 | 235 | ### [External Feedback Strategy](#external-feedback-strategy) 236 | 237 | 1. **Re3: Generating Longer Stories With Recursive Reprompting and Revision.** Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022. [paper](https://aclanthology.org/2022.emnlp-main.296.pdf) 238 | 239 | *Kevin Yang, Yuandong Tian, Nanyun Peng, Dan Klein* 240 | 241 | 2. **CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning.** Advances in Neural Information Processing Systems (NeurIPS), 2022. [paper](https://arxiv.org/pdf/2207.01780.pdf) 242 | 243 | *Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, Steven C.H. Hoi* 244 | 245 | 3. **REFINER: Reasoning Feedback on Intermediate Representations.** arxiv, 2023. [paper](https://arxiv.org/pdf/2304.01904.pdf) 246 | 247 | *Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, Boi Faltings* 248 | 249 | 4. **RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs.** Annual Meeting of the Association for Computational Linguistics (ACL), 2023. [paper](https://aclanthology.org/2023.acl-long.427.pdf) 250 | 251 | *Afra Feyza Akyurek, Ekin Akyurek, Ashwin Kalyan, Peter Clark, Derry Tanti Wijaya, Niket Tandon* 252 | 253 | 5. **Learning to Simulate Natural Language Feedback for Interactive Semantic Parsing.** Annual Meeting of the Association for Computational Linguistics (ACL), 2023. [paper](https://aclanthology.org/2023.acl-long.177.pdf) 254 | 255 | *Hao Yan, Saurabh Srivastava, Yintao Tai, Sida I. Wang, Wen-tau Yih, Ziyu Yao* 256 | 257 | 6. **Baldur: Whole-Proof Generation and Repair with Large Language Models.** arxiv, 2023. [paper](https://arxiv.org/pdf/2303.04910.pdf) 258 | 259 | *Emily First, Markus N. Rabe, Talia Ringer, Yuriy Brun* 260 | 261 | 7. **CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing.** arxiv, 2023. [paper](https://arxiv.org/pdf/2305.11738.pdf) 262 | 263 | *Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, Weizhu Chen* 264 | 265 | 8. **FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios.** arxiv, 2023. [paper](https://arxiv.org/pdf/2307.13528.pdf) 266 | 267 | *I-Chun Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, Pengfei Liu* 268 | 269 | 9. **RARR: Researching and Revising What Language Models Say, Using Language Models.** Annual Meeting of the Association for Computational Linguistics (ACL), 2023. [paper](https://arxiv.org/pdf/2210.08726.pdf) 270 | 271 | *Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Y. Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, Kelvin Guu* 272 | 273 | 10. **Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback.** arxiv, 2023. [paper](https://arxiv.org/pdf/2302.12813.pdf) 274 | 275 | *Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, Jianfeng Gao* 276 | 277 | 11. **Self-Checker: Plug-and-Play Modules for Fact-Checking with Large Language Models.** arxiv, 2023. [paper](https://arxiv.org/pdf/2305.14623.pdf) 278 | 279 | *Miaoran Li, Baolin Peng, Zhu Zhang* 280 | 281 | 12. **Improving Language Models via Plug-and-Play Retrieval Feedback.** arxiv, 2023. [paper](https://arxiv.org/pdf/2305.14002.pdf) 282 | 283 | *Wenhao Yu, Zhihan Zhang, Zhenwen Liang, Meng Jiang, Ashish Sabharwal* 284 | 285 | 13. **Demystifying GPT Self-Repair for Code Generation.** arxiv, 2023. [paper](https://arxiv.org/pdf/2306.09896.pdf) 286 | 287 | *Theo X. Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, Armando Solar-Lezama* 288 | 289 | 14. **Self-Edit: Fault-Aware Code Editor for Code Generation.** arxiv, 2023. [paper](https://arxiv.org/pdf/2305.04087.pdf) 290 | 291 | *Kechi Zhang, Zhuo Li, Jia Li, Ge Li, Zhi Jin* 292 | 293 | 15. **Teaching Large Language Models to Self-Debug.** arxiv, 2023. [paper](https://arxiv.org/pdf/2304.05128.pdf) 294 | 295 | *Xinyun Chen, Maxwell Lin, Nathanael Schärli, Denny Zhou* 296 | 297 | 16. **SelfEvolve: A Code Evolution Framework via Large Language Models.** arxiv, 2023. [paper](https://arxiv.org/pdf/2306.02907.pdf) 298 | 299 | *Shuyang Jiang, Yuhao Wang, Yu Wang* 300 | 301 | 17. **Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning.** arxiv, 2023. [paper](https://arxiv.org/pdf/2305.12295.pdf) 302 | 303 | *Liangming Pan, Alon Albalak, Xinyi Wang, William Yang Wang* 304 | 305 | 18. **Self-Critiquing Models for Assisting Human Evaluators.** arxiv, 2022. [paper](https://arxiv.org/pdf/2206.05802.pdf) 306 | 307 | *William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, Jan Leike* 308 | 309 | 19. **ALGO: Synthesizing Algorithmic Programs with Generated Oracle Verifiers.** arxiv, 2023. [paper](https://arxiv.org/pdf/2305.14591.pdf) 310 | 311 | *Kexun Zhang, Danqing Wang, Jingtao Xia, William Yang Wang, Lei Li* 312 | 313 | 20. **A New Era in Software Security: Towards Self-Healing Software via Large Language Models and Formal Verification.** arxiv, 2023. [paper](https://arxiv.org/pdf/2305.14752.pdf) 314 | 315 | *Yiannis Charalambous, Norbert Tihanyi, Ridhi Jain, Youcheng Sun, Mohamed Amine Ferrag, Lucas C. Cordeiro* 316 | 317 | 21. **Generating Sequences by Learning to Self-Correct.** International Conference on Learning Representations (ICLR), 2023. [paper](https://openreview.net/pdf?id=hH36JeQZDaO) 318 | 319 | *Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, Yejin Choi* 320 | 321 | 22. **MAF: Multi-Aspect Feedback for Improving Reasoning in Large Language Models.** Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. [paper](https://arxiv.org/abs/2310.12426) 322 | 323 | *Deepak Nathani, David Wang, Liangming Pan, William Yang Wang* 324 | 325 | 23. **CLOVA: A Closed-LOop Visual Assistant with Tool Usage and Update** arxiv, 2023 [paper](https://arxiv.org/abs/2312.10908) 326 | 327 | *Zhi Gao, Yuntao Du, Xintong Zhang, Xiaojian Ma, Wenjuan Han, Song-Chun Zhu, Qing Li* 328 | 329 | 24. **LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints** Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024. [paper](https://arxiv.org/abs/2410.06458) 330 | 331 | *Thomas Palmeira Ferraz, Kartik Mehta, Yu-Hsiang Lin, Haw-Shiuan Chang, Shereen Oraby, Sijia Liu, Vivek Subramanian, Tagyoung Chung, Mohit Bansal, Nanyun Peng* 332 | 333 | ### [Model-Debate Strategy](#model-debate-strategy) 334 | 335 | 1. **Improving Factuality and Reasoning in Language Models through Multiagent Debate.** arxiv, 2023. [paper](https://arxiv.org/pdf/2305.14325.pdf) 336 | 337 | *Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, Igor Mordatch* 338 | 339 | 2. **LM vs LM: Detecting Factual Errors via Cross Examination.** arxiv, 2023. [paper](https://arxiv.org/pdf/2305.13281.pdf) 340 | 341 | *Roi Cohen, May Hamri, Mor Geva, Amir Globerson* 342 | 343 | 3. **Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedback.** arxiv, 2023. [paper](https://arxiv.org/pdf/2305.10142.pdf) 344 | 345 | *Yao Fu, Hao Peng, Tushar Khot, Mirella Lapata* 346 | 347 | 4. **PRD: Peer Rank and Discussion Improve Large Language Model-based Evaluations.** arxiv, 2023. [paper](https://arxiv.org/pdf/2307.02762.pdf) 348 | 349 | *Ruosen Li, Teerth Patel, Xinya Du* 350 | 351 | 352 | ## Contribution 353 | ### Contributors 354 | 355 | 360 | 361 | 362 | 363 | 364 | 365 | 366 | 367 | 374 | 381 | 382 | 383 |
368 | 369 | Liangming Pan 370 |
371 | Liangming Pan 372 | 373 | 		375 | 376 | Xinyuan Lu 377 |
378 | Xinyuan Lu 379 | 380 |
384 | 385 | ### Acknowledgement 386 | - There are cases where we miss important works in this field, please contribute to this repo! Thanks for your efforts in advance. 387 | - If you encounter any problem, please either directly contact [Liangming Pan](liangmingpan@ucsb.edu) or leave an issue in the GitHub repo. 388 | --------------------------------------------------------------------------------