├── LICENSE └── README.md /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2022 Pan Lu 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Deep Learning for Mathematical Reasoning (DL4MATH) 2 | 3 | [![Awesome](https://awesome.re/badge.svg)](https://github.com/lupantech/dl4math) 4 | [![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT) 5 | [![Survey](https://img.shields.io/badge/Survey-DL4MATH-blue)](https://github.com/lupantech/dl4math) 6 | 7 | This repository is the reading list on ***Deep Learning for Mathematical Reasoning (DL4MATH)***. 8 | 9 | **Contributors**: [Pan Lu](https://lupantech.github.io/) @UCLA, [Liang Qiu](https://www.lqiu.info/) @UCLA, [Wenhao Yu](https://wyu97.github.io/) @Notre Dame, [Sean Welleck](https://wellecks.com/) @UW, [Kai-Wei Chang](http://web.cs.ucla.edu/~kwchang/) @UCLA 10 | 11 | For more details, please refer to the paper: [A Survey of Deep Learning for Mathematical Reasoning](https://arxiv.org/abs/2212.10535). 12 | 13 | :bell: If you have any suggestions or notice something we missed, please don't hesitate to let us know. You can directly email Pan Lu (lupantech@gmail.com), comment on the [twitter](https://twitter.com/lupantech/status/1605400505697841155), or post an issue on this repo. 14 | 15 | 16 | 17 | ## 🧰 Resources 18 | 19 | ### Related Surveys 20 | 21 | - **A Survey of Question Answering for Math and Science Problem**, arXiv:1705.04530 [[paper](https://arxiv.org/abs/1705.04530)] 22 | - **The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers**, TPAMI 2019 [[paper](https://arxiv.org/abs/1808.07290)] 23 | - **Representing Numbers in NLP: a Survey and a Vision**, NACL 2021 [[paper](https://aclanthology.org/2021.naacl-main.53/)] 24 | - **Survey on Mathematical Word Problem Solving Using Natural Language Processing**, ICIICT 2021 [[paper](https://ieeexplore.ieee.org/abstract/document/8741437)] 25 | - **A Survey in Mathematical Language Processing**, arXiv:2205.15231 [[paper](https://arxiv.org/abs/2205.15231)] 26 | - **Partial Differential Equations Meet Deep Neural Networks: A Survey**, arXiv:2211.05567 [[paper](https://arxiv.org/abs/2211.05567)] 27 | - :fire: **Reasoning with Language Model Prompting: A Survey**, arXiv:2212.09597 [[paper](https://arxiv.org/abs/2212.09597)] 28 | - :fire: **Towards Reasoning in Large Language Models**: arXiv:2212.10403 [[paper](https://arxiv.org/abs/2212.10403)] 29 | - :fire: **A Survey for In-context Learning**, arXiv:2301.00234 [[paper](https://arxiv.org/abs/2301.00234)] 30 | 31 | ### Related Blogs 32 | 33 | - :fire: **How does GPT Obtain its Ability? Tracing Emergent Abilities of Language Models to their Sources**, Dec 2022, Yao Fu’s Notion [[link](https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emergent-Abilities-of-Language-Models-to-their-Sources-b9a57ac0fcf74f30a1ab9e3e36fa1dc1)] 34 | 35 | ### Workshops 36 | 37 | - :fire: **The 1st MATH-AI Workshop: the Role of Mathematical Reasoning in General Artificial Intelligence**, ICLR 2021 [[website](https://mathai-iclr.github.io/)] 38 | - :fire: **Math AI for Education: Bridging the Gap Between Research and Smart Education (MATHAI4ED)**, NeurIPS 2021 [[website](https://mathai4ed.github.io/)] 39 | - :fire: **The 1st Workshop on Mathematical Natural Language Processing**, EMNLP 2022 [[website](https://sites.google.com/view/1st-mathnlp/)] 40 | - :fire: **The 2nd MATH-AI Workshop: Toward Human-Level Mathematical Reasoning**, NeurIPS 2022 [[website](https://mathai2022.github.io/)] 41 | - :fire: **FLAIM: Formal Languages, AI and Mathematics**, IHP & META 2022 [[YouTube](https://www.youtube.com/playlist?list=PLgBHexwnIcdueioZA-fgrx0dxXY2tJu6H)] 42 | - :fire: **AI to Assist Mathematical Reasoning: A Workshop**, NASEM 2023 [[YouTube](https://www.youtube.com/playlist?list=PLgBHexwnIcdtAv9jVYnXAjCMKA0pNDXxJ)] 43 | 44 | ### Talks 45 | 46 | - **Can GPT-3 do math? | Grant Sanderson and Lex Fridman**, 2020 [[YouTube](https://www.youtube.com/watch?v=TMxAbNAVrzI&ab_channel=LexClips)] 47 | - **Computer Scientist Explains One Concept in 5 Levels of Difficulty**, 2022 [[YouTube](https://www.youtube.com/watch?v=fOGdb1CTu5c)] 48 | 49 | 50 | 51 | ## 🎨 Mathematical Reasoning Benchmarks 52 | 53 | ### Math Word Problems (MWP) 54 | 55 | - [AI2/Verb395] **Learning to Solve Arithmetic Word Problems with Verb Categorization**, EMNLP 2014 [[paper](https://aclanthology.org/D14-1058/)] 56 | - [Alg514] **Learning to automatically solve algebra word problems**, ACL 2014 [[paper](https://aclanthology.org/P14-1026/)] 57 | - [IL] **Reasoning about Quantities in Natural Language**, TACL 2015 [[paper](https://aclanthology.org/Q15-1001/)] 58 | - [SingleEQ] **Parsing Algebraic Word Problems into Equations**, TACL 2015 [[paper](https://aclanthology.org/Q15-1042/)] 59 | - [DRAW] **Draw: A challenging and diverse algebra word problem set**, 2015 [[paper](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/tech_rep.pdf)] 60 | - [Dolphin1878] **Automatically solving number word problems by semantic parsing and reasoning**, EMNLP 2015 [[paper](https://aclanthology.org/D15-1135/)] 61 | - [Dolphin18K] **How well do computers solve math word problems? large-scale dataset construction and evaluation**, ACL 2016 [[paper](https://aclanthology.org/P16-1084/)] 62 | - [MAWPS] **MAWPS: A math word problem repository**, NAACL-HLT 2016 [[paper](https://aclanthology.org/N16-1136/)] 63 | - [AllArith] **Unit dependency graph and its application to arithmetic word problem solving**, AAAI 2017 [[paper](https://arxiv.org/abs/1612.00969)] 64 | - [DRAW-1K] **Annotating Derivations: A New Evaluation Strategy and Dataset for Algebra Word Problems**, ACL 2017 [[paper](https://aclanthology.org/E17-1047/)] 65 | - :fire: [Math23K] **Deep neural solver for math word problems**, EMNLP 2017 [[paper](https://aclanthology.org/D17-1088/)] 66 | - [AQuA] **Program Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems**, ACL 2017 [[paper](https://arxiv.org/abs/1705.04146)] 67 | - [Aggregate] **Mapping to Declarative Knowledge for Word Problem Solving**, TACL 2018 [[paper](https://arxiv.org/abs/1712.09391)] 68 | - :fire: [MathQA] **MathQA: Towards interpretable math word problem solving with operation-based formalisms**, NAACL-HLT 2019 [[paper](https://aclanthology.org/N19-1245/)] 69 | - [ASDiv] **A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers**, ACL 2020 [[paper](https://arxiv.org/abs/2106.15772)] 70 | - [HMWP] **Semantically-Aligned Universal Tree-Structured Solver for Math Word Problems**, EMNLP 2020 [[paper](https://arxiv.org/abs/2010.06823)] 71 | - [Ape210K] **Ape210k: A large-scale and template-rich dataset of math word problems**, arXiv:2009.11506 [[paper](https://arxiv.org/abs/2009.11506)] 72 | - :fire: [SVAMP] **Are NLP Models really able to Solve Simple Math Word Problems?**, NAACL-HIT 2021 [[paper](https://arxiv.org/abs/2103.07191)] 73 | - :fire: [GSM8K] **Training verifiers to solve math word problems**, arXiv:2110.14168 [[paper](https://arxiv.org/abs/2110.14168)] 74 | - :fire: [IconQA] **IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning**, NeurIPS 2021] [[paper](https://arxiv.org/abs/2110.13214)] 75 | - :fire: [MathQA-Python] **Program synthesis with large language models**, arXiv:2108.07732 [[paper](https://arxiv.org/abs/2108.07732)] 76 | - [ArMATH] **ArMATH: a Dataset for Solving Arabic Math Word Problems**, LREC 2022 [[paper](https://aclanthology.org/2022.lrec-1.37/)] 77 | - :fire: [TabMWP] **Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning**, arXiv:2209.14610, 2022 [[paper](https://arxiv.org/abs/2209.14610)] 78 | 79 | ### Theorem Proving (TP) 80 | 81 | - [MML] **Four Decades of Mizar**, Journal of Automated Reasoning 2015, [[paper](https://dl.acm.org/doi/abs/10.1007/s10817-015-9345-1)] 82 | - [HolStep] **HolStep: A Machine Learning Dataset for Higher-order Logic Theorem Proving**, ICLR 2017 [[paper](https://arxiv.org/abs/1703.00426)] 83 | - [GamePad] **GamePad: A Learning Environment for Theorem Proving**, ICLR 2019 [[paper](https://arxiv.org/abs/1806.00608)] 84 | - :fire: [CoqGym] **Learning to Prove Theorems via Interacting with Proof Assistants**, ICML 2019 [[paper](https://arxiv.org/abs/1905.09381)] 85 | - [HOList] **HOList: An environment for machine learning of higher order logic theorem proving**, ICML 2019 [[paper](https://arxiv.org/abs/1904.03241)] 86 | - [IsarStep] **IsarStep: a Benchmark for High-level Mathematical Reasoning**, ICLR 2021 [[paper](https://arxiv.org/abs/2006.09265)] 87 | - [LISA] **LISA: Language models of ISAbelle proofs**, AITP 2021 [[paper](http://aitp-conference.org/2021/abstract/paper_17.pdf)] 88 | - [INT] **INT: An Inequality Benchmark for Evaluating Generalization in Theorem Proving**, ICLR 2021 [[paper](https://arxiv.org/abs/2007.02924)] 89 | - :fire: [NaturalProofs] **NaturalProofs: Mathematical Theorem Proving in Natural Language**, NeurIPS 2021 [[paper](https://arxiv.org/abs/2104.01112)] 90 | - [NaturalProofs-Gen] **NaturalProver: Grounded Mathematical Proof Generation with Language Models**, NeurIPS 2022 [[paper](https://arxiv.org/abs/2205.12910)] 91 | - :fire: [MiniF2F] **MiniF2F: a cross-system benchmark for formal Olympiad-level mathematics**, ICLR 2022 [[paper](https://arxiv.org/abs/2109.00110)] 92 | - :fire: [LeanStep] **Proof Artifact Co-training for Theorem Proving with Language Models**, ICLR 2022 [[paper](https://arxiv.org/abs/2102.06203)] 93 | - :fire: [miniF2F+informal] **Draft, Sketch, and Prove: Guiding Formal Theorem Provers with Informal Proofs**, arXiv:2210.12283 [[paper](https://arxiv.org/abs/2210.12283)] 94 | 95 | ### Geometry Problem Solving (GPS) 96 | 97 | - :fire: [GEOS] **Solving geometry problems: Combining text and diagram interpretation**, EMNLP 2015 [[paper](https://aclanthology.org/D15-1171/)] 98 | - [GeoShader] **Synthesis of solutions for shaded area geometry problems**, The Thirtieth International Flairs Conference, 2017 [[paper](https://www.aaai.org/ocs/index.php/FLAIRS/FLAIRS17/paper/viewFile/15416/14902)] 99 | - [GEOS++] **From textbooks to knowledge: A case study in harvesting axiomatic knowledge from textbooks to solve geometry problems**, EMNLP 2017 [[paper](https://aclanthology.org/D17-1081/)] 100 | - [GEOS-OS] **Learning to solve geometry problems from natural language demonstrations in textbooks**, Proceedings of the 6th Joint Conference on Lexical and Computational Semantics, 2017 [[paper](https://aclanthology.org/S17-1029/)] 101 | - :fire: [Geometry3K] **Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning**, ACL 2021 [[paper](https://aclanthology.org/2021.acl-long.528/)] 102 | - [GeoQA] **GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning**, Findings of ACL 2021 [[paper](https://aclanthology.org/2021.findings-acl.46.pdf)] 103 | - [GeoQA+] **An Augmented Benchmark Dataset for Geometric Question Answering through Dual Parallel Text Encoding**, COLING 2022 [[paper](https://aclanthology.org/2022.coling-1.130/)] 104 | - :fire: [UniGeo] **UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expression**, EMNLP 2022 [[paper](https://lupantech.github.io/papers/emnlp22_unigeo.pdf)] 105 | 106 | ### Math Question Answering (MathQA) 107 | 108 | - [QUAREL] **QUAREL: A Dataset and Models for Answering Questions about Qualitative Relationships**, AAAI 2019 [[paper](https://ojs.aaai.org/index.php/AAAI/article/view/4687)] 109 | - [McTaco] **“Going on a vacation” takes longer than “Going for a walk”: A Study of Temporal Commonsense Understanding**, EMNLP 2019 [[paper](https://aclanthology.org/D19-1332/)] 110 | - :fire: [DROP] **DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs**, NAACL 2019 [[paper](https://aclanthology.org/N19-1246/)] 111 | - :fire: [Mathematics] **Analysing Mathematical Reasoning Abilities of Neural Models**, ICLR 2019 [[paper](https://arxiv.org/abs/1904.01557)] 112 | - [FinQA] **FinQA: A Dataset of Numerical Reasoning over Financial Data**, EMNLP 2021 [[paper](https://arxiv.org/abs/2109.00122)] 113 | - [Fermi] **How Much Coffee Was Consumed During EMNLP 2019? Fermi Problems: A New Reasoning Challenge for AI**, EMNLP 2020 [[paper](https://arxiv.org/abs/2110.14207)] 114 | - :fire: [MATH, AMPS] **Measuring Mathematical Problem Solving With the MATH Dataset**, NeurIPS 2021 [[paper](https://arxiv.org/abs/2103.03874)] 115 | - [TAT-QA] **TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance**, ACL-JCNLP 2021 [[paper](https://arxiv.org/abs/2105.07624)] 116 | - [MultiHiertt] **MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data**, ACL 2022 [[paper](https://aclanthology.org/2022.acl-long.454/)] 117 | - [NumGLUE] **NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks**, ACL 2022 [[paper](https://aclanthology.org/2022.acl-long.246/)] 118 | - :fire: [Lila] **Lila: A Unified Benchmark for Mathematical Reasoning**, EMNLP 2022 [[paper](https://arxiv.org/abs/2210.17517)] 119 | 120 | ### Other Quantitative Problems 121 | 122 | - [FigureQA] **Figureqa: An annotated figure dataset for visual reasoning**, arXiv:1710.07300 [[paper](https://arxiv.org/abs/1710.07300)] 123 | - :fire: [DVQA] **Dvqa: Understanding data visualizations via question answering**, CVPR 2018 [[paper](https://arxiv.org/abs/1801.08163)] 124 | - [DREAM] **DREAM: A Challenge Dataset and Models for Dialogue-Based Reading Comprehension**,TACL 2019 [[paper](https://arxiv.org/abs/1902.00164)] 125 | - [EQUATE] **EQUATE: A Benchmark Evaluation Framework for Quantitative Reasoning in Natural Language Inference**, CoNLL 2019 [[paper](https://arxiv.org/abs/1901.03735)] 126 | - :fire: [NumerSense] **Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Models**, EMNLP 2020 [[paper](https://arxiv.org/abs/2005.00683)] 127 | - [MNS] **Machine Number Sense: A Dataset of Visual Arithmetic Problems for Abstract and Relational Reasoning**, AAAI 2020 [[paper](https://arxiv.org/abs/2004.12193)] 128 | - [P3] **Programming Puzzles**, NeurIPS 2021 [[paper](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/3988c7f88ebcb58c6ce932b957b6f332-Abstract-round1.html)] 129 | - [NOAHQA] **NOAHQA: Numerical Reasoning with Interpretable Graph Question Answering Dataset**, Findings of EMNLP 2021 [[paper](https://arxiv.org/abs/2109.10604)] 130 | - [ConvFinQA] **ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering**, arXiv:2210.03849 [[paper](https://arxiv.org/abs/2210.03849)] 131 | - [PGDP5K] **PGDP5K: A Diagram Parsing Dataset for Plane Geometry Problems**, arXiv:2205.0994 [[paper](https://arxiv.org/abs/2205.09947)] 132 | - [GeoRE] **GeoRE: A Relation Extraction Dataset for Chinese Geometry Problems**, NeurIPS 2021 MATHAI4ED Workshop [[paper](https://mathai4ed.github.io/papers/papers/paper_6.pdf)] 133 | - :fire: [ScienceQA] **Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering**, NeurIPS 2022 [[paper](https://arxiv.org/abs/2209.09513)] 134 | 135 | 136 | 137 | ## 🧩 Neural Networks for Mathematical Reasoning 138 | 139 | ### General Neural Networks 140 | 141 | - [LSTM] **Long short-term memory**, Neural computation 1997 [[paper](https://ieeexplore.ieee.org/abstract/document/6795963)] 142 | - [Seq2Seq] **Sequence to sequence learning with neural networks**, NeurIPS 2014 [[paper](https://proceedings.neurips.cc/paper/2014/hash/a14ac55a4f27472c5d894ec1c3c743d2-Abstract.html)] 143 | - [GRU] **Learning Phrase Representations using RNN Encoder--Decoder for Statistical Machine Translation**, EMNLP 2014 [[paper](https://arxiv.org/abs/1406.1078)] 144 | - [Attention] **Neural machine translation by jointly learning to align and translate**, arXiv:1409.0473 [[paper](https://arxiv.org/abs/1409.0473)] 145 | - [Attention] **Show, attend and tell: Neural image caption generation with visual attention**, ICML 2015 [[paper](https://arxiv.org/abs/1502.03044)] 146 | - [Faster-RCNN] **Faster r-cnn: Towards real-time object detection with region proposal networks**, NeurIPS 2015 [[paper](https://arxiv.org/abs/1506.01497)] 147 | - [TreeLSTM] **Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks**, ACL 2015 [[paper](https://arxiv.org/abs/1503.00075)] 148 | - [BiLSTM] **Google's neural machine translation system: Bridging the gap between human and machine translation**, arXiv:1609.08144 [[paper](https://arxiv.org/abs/1609.08144)] 149 | - [ResNet] **Deep residual learning for image recognition**, CVPR 2016 [[paper](https://arxiv.org/abs/1512.03385)] 150 | - [ConvS2S] **Convolutional sequence to sequence learning**, ICML 2017 [[paper](https://arxiv.org/abs/1705.03122)] 151 | - [Top-Down Attention] **Bottom-up and top-down attention for image captioning and visual question answering**, CVPR 2018 [[paper](https://arxiv.org/abs/1707.07998)] 152 | - [FiLM] **Film: Visual reasoning with a general conditioning layer**, AAAI 2018 [[paper](https://arxiv.org/abs/1709.07871)] 153 | - [BAN] **Bilinear Attention Networks**, NeurIPS 2018 [[paper](https://arxiv.org/abs/1805.07932)] 154 | - [DAFA] **Dynamic Fusion With Intra-and Inter-Modality Attention Flow for Visual Question Answering**, CVPR 2018 [[paper](https://arxiv.org/abs/1812.05252)] 155 | 156 | ### Seq2Seq Networks for Math 157 | 158 | - :fire: [DNS] **Deep Neural Solver for Math Word Problems**, EMNLP 2017 [[paper](https://aclanthology.org/D17-1088/)] 159 | - :fire: [AnsRat] **Program induction by rationale generation: Learning to solve and explain algebraic word problems**, ACL 2017 [[paper](https://arxiv.org/abs/1705.04146)] 160 | - [Math-EN] **Translating a Math Word Problem to a Expression Tree**, EMNLP 2018 [[paper](https://arxiv.org/abs/1811.05632)] 161 | - [CASS] **Neural math word problem solver with reinforcement learning**, COLING 2018 [[paper](https://aclanthology.org/C18-1018/)] 162 | - [SelfAtt] **Data-driven methods for solving algebra word problems**, arXiv:1804.10718 [[paper](https://arxiv.org/abs/1804.10718)] 163 | - [S-Aligned] **Semantically-Aligned Equation Generation for Solving and Reasoning Math Word Problems**, NAACL 2019 [[paper](https://aclanthology.org/N19-1272/)] 164 | - [T-RNN] **Template-based math word problem solvers with recursive neural networks**, AAAI 2019 [[paper](https://ojs.aaai.org/index.php/AAAI/article/view/4697)] 165 | - [GROUP-ATT] **Modeling intra-relation in math word problems with different functional multi-head attentions**, ACL 2019 [[paper](https://aclanthology.org/P19-1619/)] 166 | - [QuaSP+] **QUAREL: A Dataset and Models for Answering Questions about Qualitative Relationships**, AAAI 2019 [[paper](https://arxiv.org/abs/1811.08048)] 167 | - [SMART] **SMART: A Situation Model for Algebra Story Problems via Attributed Grammar**, AAAI 2021 [[paper](https://arxiv.org/abs/2012.14011)] 168 | 169 | ### Graph-based Networks for Math 170 | 171 | - [AST-Dec] **Tree-structured decoding for solving math word problems**, EMNLP 2019 [[paper](https://aclanthology.org/D19-1241/)] 172 | - :fire: [GTS] **A Goal-Driven Tree-Structured Neural Model for Math Word Problems**, IJCAI 2019 [[paper](https://www.ijcai.org/proceedings/2019/736)] 173 | - [CoqGym] **Learning to Prove Theorems via Interacting with Proof Assistants**, ICML 2019 [[paper](https://arxiv.org/abs/1905.09381)] 174 | - [KA-S2T] **A knowledge-aware sequence-to-tree network for math word problem solving**, EMNLP 2020 [[paper](https://aclanthology.org/2020.emnlp-main.579/)] 175 | - [TSN-MD, NT-LSTM] **Solving arithmetic word problems by scoring equations with recursive neural networks**, Expert Systems with Applications 2021 [[paper](https://arxiv.org/abs/2009.05639)] 176 | - [NS-Solver] **Neural-Symbolic Solver for Math Word Problems with Auxiliary Tasks**, ACL 2021 [[paper](https://arxiv.org/abs/2107.01431)] 177 | - [NumS2T] **Math word problem solving with explicit numerical values**, ACL 2021 [[paper](https://aclanthology.org/2021.acl-long.455/)] 178 | - [HMS] **Hms: A hierarchical solver with dependency-enhanced understanding for math word problem**, AAAI 2021 [[paper](https://ojs.aaai.org/index.php/AAAI/article/view/16547)] 179 | - [LBF] **Learning by fixing: Solving math word problems with weak supervision**, AAAI 2021 [[paper](https://arxiv.org/abs/2012.10582)] 180 | - [Seq2DAG] **A bottom-up dag structure extraction model for math word problems**, AAAI 2021 [[paper](https://ojs.aaai.org/index.php/AAAI/article/view/16075)] 181 | - [Graph2Tree] **Graph-to-Tree Neural Networks for Learning Structured Input-Output Translation with Applications to Semantic Parsing and Math Word Problem**, EMNLP 2020 [[paper](https://arxiv.org/abs/2004.13781)] 182 | - [Multi-E/D] **Solving math word problems with multi-encoders and multi-decoders**, COLING 2020 [[paper](https://aclanthology.org/2020.coling-main.262/)] 183 | - :fire: [Graph2Tree] **Graph-to-Tree Learning for Solving Math Word Problems**, ACL 2020 [[paper](https://aclanthology.org/2020.acl-main.362/)] 184 | - [EEH-G2T] **An edge-enhanced hierarchical graph-to-tree network for math word problem solving**, EMNLP 2021 [[paper](https://aclanthology.org/2021.findings-emnlp.127/)] 185 | 186 | ### Other Neural Networks for Math 187 | 188 | - [DeepMath] **Deepmath-deep sequence models for premise selection**, NeurIPS 2016 [[paper](https://arxiv.org/abs/1606.04442)] 189 | - [Holophrasm] **Holophrasm: a neural automated theorem prover for higher-order logic**, arXiv:1608.02644 [[paper](https://arxiv.org/abs/1608.02644)] 190 | - :fire: [CNNTP, WaveNetTP] **Deep network guided proof search**, arXiv:1701.06972 [[paper](https://arxiv.org/abs/1701.06972)] 191 | - :fire: [MathDQN] **Mathdqn: Solving arithmetic word problems via deep reinforcement learning**, AAAI 2018 [[paper](https://ojs.aaai.org/index.php/AAAI/article/view/11981)] 192 | - [DDT] **Solving math word problems with double-decoder transformer**, arXiv:1908.10924 [[paper](https://arxiv.org/abs/1908.10924)] 193 | - [DeepHOL] **HOList: An environment for machine learning of higher order logic theorem proving**, ICML 2019 [[paper](https://arxiv.org/abs/1904.03241)] 194 | - [NGS] **GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning**, Findings of ACL 2021 [[paper](https://aclanthology.org/2021.findings-acl.46.pdf)] 195 | - [PGDPNet] **Learning to Understand Plane Geometry Diagram**, NeurIPS 2022 MATH-AI Workshop [[paper](https://mathai2022.github.io/papers/6.pdf)] 196 | 197 | 198 | 199 | ## 📜 Pre-trained Language Models for Mathematical Reasoning 200 | 201 | ### General Pre-trained Language Models (<100B) 202 | 203 | - [Transformer] **Attention is all you need**, NeurIPS 2017 [[paper](https://arxiv.org/abs/1706.03762)] 204 | - [BERT] **Bert: Pre-training of deep bidirectional transformers for language understanding**, arXiv:1810.04805 [[paper](https://arxiv.org/abs/1810.04805)] 205 | - [T5] **Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer**, JMLR 2020 [[paper](https://arxiv.org/abs/1910.10683)] 206 | - [RoBERTa] **Roberta: A robustly optimized bert pretraining approach**, arXiv:1907.11692 [[paper](https://arxiv.org/abs/1907.11692)] 207 | - [GPT-2, 1.5B] **Language models are unsupervised multitask learners**, OpenAI Blog, 2020 [[paper](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)] 208 | - [BART] **BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension**, ACL 2020 [[paper](https://arxiv.org/abs/1910.13461)] 209 | - [ALBERT] **Albert: A lite bert for self-supervised learning of language representations**, arXiv:1909.11942 [[paper](https://arxiv.org/abs/1909.11942)] 210 | - [GPT-Neo] **The pile: An 800gb dataset of diverse text for language modeling**, arXiv:2101.00027 [[paper](https://arxiv.org/abs/2101.00027)] 211 | - [VL-T5] **Unifying Vision-and-Language Tasks via Text Generation**, ICML 2021 [[paper](https://arxiv.org/abs/2102.02779)] 212 | 213 | ### Self-Supervised Learning for Math 214 | 215 | - :fire: [GenBERT] **Injecting numerical reasoning skills into language models**, ACL 2020 [[paper](https://arxiv.org/abs/2004.04487)] 216 | - :fire: [GPT-f] **Generative language modeling for automated theorem proving**, arXiv:2009.03393 [[paper](https://arxiv.org/abs/2009.03393)] 217 | - [LISA] **LISA: Language models of ISAbelle proofs**, AITP 2021 [[paper](http://aitp-conference.org/2021/abstract/paper_17.pdf)] 218 | - [MATH-PLM] **Measuring Mathematical Problem Solving With the MATH Dataset**, NeurIPS 2021 [[paper](https://arxiv.org/abs/2103.03874)] 219 | - [LIME] **Lime: Learning inductive bias for primitives of mathematical reasoning**, ICML 2021 [[paper](https://arxiv.org/abs/2101.06223)] 220 | - [NF-NSM] **Injecting Numerical Reasoning Skills into Knowledge Base Question Answering Models**, arXiv:2112.06109 [[paper](https://arxiv.org/abs/2112.06109)] 221 | - [MWP-BERT] **MWP-BERT: Numeracy-augmented pre-training for math word problem solving**, Findings of NAACL 2022 [[paper](https://arxiv.org/abs/2107.13435)] 222 | - [HTPS] **HyperTree Proof Search for Neural Theorem Proving**, arXiv:2205.11491 [[paper](https://arxiv.org/abs/2205.11491)] 223 | - [Thor] **Thor: Wielding Hammers to Integrate Language Models and Automated Theorem Provers**, arXiv:2205.10893 [[paper](https://arxiv.org/abs/2205.10893)] 224 | - [Set] **Insights into pre-training via simpler synthetic tasks**, arXiv:2206.10139 [[paper](https://arxiv.org/abs/2206.10139)] 225 | - [PACT] **Proof artifact co-training for theorem proving with language models**, ICLR 2022 [[paper](https://arxiv.org/abs/2102.06203)] 226 | - :fire: [TaPEX] **TAPEX: Table Pre-training via Learning a Neural SQL Executor**, ICLR 2022 [[paper](https://arxiv.org/abs/2107.07653)] 227 | - :fire: [Minerva] **Solving quantitative reasoning problems with language models**, NeurIPS 2022 [[paper](https://arxiv.org/abs/2206.14858)] 228 | 229 | ### Task-specific Fine-tuning for Math 230 | 231 | - [EPT] **Point to the expression: Solving algebraic word problems using the expression-pointer transformer model**, EMNLP 2020 [[paper](https://aclanthology.org/2020.emnlp-main.308/)] 232 | - [Generate \& Rank] **Generate \& Rank: A Multi-task Framework for Math Word Problems**, EMNLP 2021 [[paper](https://arxiv.org/abs/2109.03034)] 233 | - [RPKHS] **Improving Math Word Problems with Pre-trained Knowledge and Hierarchical Reasoning**, EMNLP 2021 [[paper](https://aclanthology.org/2021.emnlp-main.272/)] 234 | - [PatchTRM] **IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning**, NeurIPS 2021 [[paper](https://arxiv.org/abs/2110.13214)] 235 | - :fire: [GSM8K-PLM] **Training verifiers to solve math word problems**, arXiv:2110.14168 [[paper](https://arxiv.org/abs/2110.14168)] 236 | - :fire: [Inter-GPS] **Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning**, ACL 2021 [[paper](https://aclanthology.org/2021.acl-long.528/)] 237 | - [Aristo] From ‘F’to ‘A’on the NY regents science exams: An overview of the aristo project, AI Magazine 2020 [paper] 238 | - [FinQANet] **FinQA: A Dataset of Numerical Reasoning over Financial Data**, EMNLP 2021 [[paper](https://arxiv.org/abs/2109.00122)] 239 | - [TAGOP] **TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance**, ACL-JCNLP 2021 [[paper](https://arxiv.org/abs/2105.07624)] 240 | - [LAMT] **Linear algebra with transformers**, arXiv:2112.01898 [[paper](https://arxiv.org/abs/2112.01898)] 241 | - :fire: [Scratchpad] **Show your work: Scratchpads for intermediate computation with language models**, arXiv:2112.00114 [[paper](https://arxiv.org/abs/2112.00114)] 242 | - [Self-Sampling] **Learning from Self-Sampled Correct and Partially-Correct Programs**, arXiv:2205.14318 [[paper](https://arxiv.org/abs/2205.14318)] 243 | - [DeductReasoner] **Learning to Reason Deductively: Math Word Problem Solving as Complex Relation Extraction**, ACL 2022 [[paper](https://arxiv.org/abs/2203.10316)] 244 | - [DPE-NGS] **An Augmented Benchmark Dataset for Geometric Question Answering through Dual Parallel Text Encoding**, COLING 2022 [[paper](https://aclanthology.org/2022.coling-1.130/)] 245 | - [BERT-TD+CL] **Seeking Patterns, Not just Memorizing Procedures: Contrastive Learning for Solving Math Word Problems**, Findings of ACL 2022 [[paper](https://arxiv.org/abs/2110.08464)] 246 | - [MT2Net] **MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data**, ACL 2022 [[paper](https://aclanthology.org/2022.acl-long.454/)] 247 | - [miniF2F-PLM] **MiniF2F: a cross-system benchmark for formal Olympiad-level mathematics**, ICLR 2022 [[paper](https://arxiv.org/abs/2109.00110)] 248 | - :fire: [NaturalProver] **NaturalProver: Grounded Mathematical Proof Generation with Language Models**, NeurIPS 2022 [[paper](https://arxiv.org/abs/2205.12910)] 249 | - :fire: [UniGeo] **UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expression**, EMNLP 2022 [[paper](https://lupantech.github.io/papers/emnlp22_unigeo.pdf)] 250 | - :fire: [Bhaskara] **Lila: A Unified Benchmark for Mathematical Reasoning**, EMNLP 2022 [[paper](https://arxiv.org/abs/2210.17517)] 251 | 252 | 253 | 254 | ## 🌠 In-context Learning for Mathematical Reasoning 255 | 256 | ### General Large Language Models (100B+) 257 | 258 | - :fire: [GPT-3, 175B] **Language models are few-shot learners**, NeurIPS 2020 [[paper](https://arxiv.org/abs/2005.14165)] 259 | - :fire: [Codex, 175B] **Evaluating large language models trained on code**, arXiv:2107.03374 [[paper](https://arxiv.org/abs/2107.03374)] 260 | - :fire: [PaLM, 540B] **PaLM: Scaling Language Modeling with Pathways**, arXiv:2204.02311 [[paper](https://arxiv.org/abs/2204.02311)] 261 | - :fire: [ChatGPT, 175B] **ChatGPT: Optimizing Language Models for Dialogue**, November 30, 2022 [[website](https://openai.com/blog/chatgpt/)] 262 | - :question: [GPT-4] 263 | 264 | ### In-context Example Selection 265 | 266 | - :fire: [Few-shot-CoT] **Chain of thought prompting elicits reasoning in large language models**, NeurIPS 2022 [[paper](https://arxiv.org/abs/2201.11903)] 267 | - [Retrieval] **Learning to retrieve prompts for in-context learning**, NAACL-HLT 2022 [[paper](https://arxiv.org/abs/2112.08633)] 268 | - :fire: [PromptPG-CoT] **Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning**, arXiv:2209.14610 [[paper](https://arxiv.org/abs/2209.14610)] 269 | - [Retrieval-CoT] **Automatic Chain of Thought Prompting in Large Language Models**, arXiv:2210.03493 [[paper](https://arxiv.org/abs/2210.03493)] 270 | - [Generate] **Generate rather than retrieve: Large language models are strong context generators**, arXiv:2209.10063 [[paper](https://arxiv.org/abs/2209.10063)] 271 | - [Complexity-CoT] **Complexity-Based Prompting for Multi-Step Reasoning,** arXiv:2210.00720 [[paper](https://arxiv.org/abs/2210.00720)] 272 | - [Auto-CoT] **Automatic Chain of Thought Prompting in Large Language Models**, arXiv:2210.03493 [[paper](https://arxiv.org/abs/2210.03493)] 273 | 274 | ### High-quality Reasoning Chains 275 | 276 | - :fire: [Self-Consistency-CoT] **Self-consistency improves chain of thought reasoning in language models**, arXiv:2203.11171 [[paper](https://arxiv.org/abs/2203.11171)] 277 | - :fire: [Least-to-most CoT] **Least-to-Most Prompting Enables Complex Reasoning in Large Language Models**, arXiv:2205.10625 [[paper](https://arxiv.org/abs/2205.10625)] 278 | - **On the Advance of Making Language Models Better Reasoners**, arXiv:2206.02336 [[paper](https://arxiv.org/abs/2206.02336)] 279 | - **Decomposed prompting: A modular approach for solving complex tasks**, arXiv:2210.02406 [[paper](https://arxiv.org/abs/2210.02406)] 280 | - **PAL: Program-aided Language Models**, arXiv:2211.10435 [[paper](https://arxiv.org/abs/2211.10435)] 281 | - :fire: [Few-shot-PoT] **Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks**, arXiv:2211.12588 [[paper](https://arxiv.org/abs/2211.12588)] 282 | 283 | 284 | 285 | ## ♣️ Other Related Work for Mathematical Reasoning 286 | 287 | ### Early Work 288 | 289 | - **Empirical explorations of the geometry theorem machine**, Western Joint IRE-AIEE-ACM Computer Conference 1960 [[paper](https://dl.acm.org/doi/10.1145/1460361.1460381)] 290 | - **Basic principles of mechanical theorem proving in elementary geometries**, Journal of Automated Reasoning 1986 [[paper](https://link.springer.com/article/10.1007/BF02328447)] 291 | - **Automated generation of readable proofs with geometric invariants**, Journal of Automated Reasoning 1996 [[paper](https://link.springer.com/article/10.1007/BF00283133)] 292 | 293 | ### Datasets 294 | 295 | - :fire: [TextbookQA] **Are You Smarter Than A Sixth Grader? Textbook Question Answering for Multimodal Machine Comprehension**, CVPR 2017 [[paper](https://ieeexplore.ieee.org/document/8100054)] 296 | - :fire: [Raven] **Raven: A dataset for relational and analogical visual reasoning**, CVPR 2019 [[paper](https://arxiv.org/abs/1903.02741)] 297 | - [APPS] **Measuring Coding Challenge Competence With APPS**, NeurIPS 2021 [[paper](https://arxiv.org/abs/2105.09938)] 298 | - [PhysNLU] **PhysNLU: A Language Resource for Evaluating Natural Language Understanding and Explanation Coherence in Physics**, 2022 [[paper](https://arxiv.org/abs/2201.04275)] 299 | 300 | ### Methods 301 | 302 | - **My computer is an honor student—but how intelligent is it? Standardized tests as a measure of AI**, AI Magazine 2016 [[paper](https://ojs.aaai.org//index.php/aimagazine/article/view/2636)] 303 | - **Learning pipelines with limited data and domain knowledge: A study in parsing physics problems**, NeurIPS 2018 [[paper](https://proceedings.neurips.cc/paper/2018/hash/ac627ab1ccbdb62ec96e702f07f6425b-Abstract.html)] 304 | - **Automatically proving plane geometry theorems stated by text and diagram**, International Journal of Pattern Recognition and Artificial Intelligence 2019 [[paper](https://www.worldscientific.com/doi/abs/10.1142/S0218001419400032)] 305 | - **Classification and Clustering of arXiv Documents, Sections, and Abstracts, Comparing Encodings of Natural and Mathematical Language**, JCDL 2020 [[paper](https://arxiv.org/abs/2005.11021)] 306 | 307 | ### Latest Work (To be classified) 308 | 309 | - :fire: **Advancing mathematics by guiding human intuition with AI**, Nature 2021 [[paper](https://www.nature.com/articles/s41586-021-04086-x)] 310 | - [MWPToolkit] **Mwptoolkit: an open-source framework for deep learning-based math word problem solvers**, AAAI 2022 [[paper](https://arxiv.org/abs/2109.00799)] 311 | - **A deep reinforcement learning agent for geometry online tutoring**, Knowledge and Information Systems 2022 [[paper](https://link.springer.com/article/10.1007/s10115-022-01804-3)] 312 | - **ELASTIC: Numerical Reasoning with Adaptive Symbolic Compiler**, NeurIPS 2022 [[paper](https://arxiv.org/abs/2210.10105)] 313 | - **Solving math word problems with process and outcome-based feedback**, arXiv:2211.14275 [[paper](https://arxiv.org/abs/2211.14275)] 314 | - **APOLLO: An Optimized Training Approach for Long-form Numerical Reasoning**, arXiv:2212.07249 [[paper](https://arxiv.org/abs/2212.07249)] 315 | - **Enhancing Financial Table and Text Question Answering with Tabular Graph and Numerical Reasoning**, AACL 2022 [[paper](https://aclanthology.org/2022.aacl-main.72/)] 316 | - **DyRRen: A Dynamic Retriever-Reranker-Generator Model for Numerical Reasoning over Tabular and Textual Data**, AAAI 2023 [[paper](https://arxiv.org/abs/2211.12668)] 317 | - **Generalizing Math Word Problem Solvers via Solution Diversification**, arXiv:2212.00833 [[paper](https://arxiv.org/abs/2212.00833)] 318 | - **Textual Enhanced Contrastive Learning for Solving Math Word Problems**, arXiv:2211.16022 [[paper](https://arxiv.org/abs/2211.16022)] 319 | - **Analogical Math Word Problems Solving with Enhanced Problem-Solution Association**, EMNLP 2022 [[paper](https://arxiv.org/abs/2212.00837)] 320 | 321 | 322 | 323 | ## Citation 324 | 325 | If you find this repo useful, please kindly cite our survey: 326 | 327 | ``` 328 | @article{lu2022dl4math, 329 | title={A Survey of Deep Learning for Mathematical Reasoning}, 330 | author={Lu, Pan and Qiu, Liang and Yu, Wenhao and Welleck, Sean and Chang, Kai-Wei}, 331 | journal={arXiv preprint arXiv:2212.10535}, 332 | year={2022} 333 | } 334 | ``` 335 | 336 | --------------------------------------------------------------------------------