├── figs └── KGPLM-framework.png ├── LICENSE └── README.md /figs/KGPLM-framework.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/linyaoyang/Awesome-KGLLM/HEAD/figs/KGPLM-framework.png -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2023 linyaoyang 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Awesome-KGLLM 2 | A collection of papers and resources about knowledge graph enhanced large language models (KGLLM) 3 | 4 | Recently, ChatGPT, a representative large language model (LLM), has gained considerable attention due to its powerful emergent abilities. Some researchers suggest that LLMs could potentially replace structured knowledge bases like knowledge graphs (KGs) and function as parameterized knowledge bases. However, while LLMs are proficient at learning probabilistic language patterns based on large corpus and engaging in conversations with humans, they, like previous smaller pre-trained language models (PLMs), still have difficulty in recalling facts while generating knowledge-grounded contents. To overcome these limitations, researchers have proposed enhancing data-driven PLMs with knowledge-based KGs to incorporate explicit factual knowledge into PLMs, thus improving their performance to generate texts requiring factual knowledge and providing more informed responses to user queries. Therefore, we review the studies on enhancing PLMs with KGs, detailing existing knowledge graph enhanced pre-trained language models (KGPLMs) as well as their applications. Inspired by existing studies on KGPLM, we propose to enhance LLMs with KGs by developing knowledge graph-enhanced large language models (KGLLMs). KGLLM provides a solution to enhance LLMs' factual reasoning ability, opening up new avenues for LLM research. 5 | 6 | The organization of these papers refers to our survey: [ChatGPT is not Enough: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modeling](https://arxiv.org/abs/2306.11489 "悬停显示") 7 | 8 | Please let us know if you find any mistakes or have any suggestions by email: yangly@zhejianglab.com 9 | 10 | If you find our survey useful for your research, please cite the following paper: 11 | ```bash 12 | @article{KGLLM, 13 | title={ChatGPT is not Enough: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modeling}, 14 | author={Yang, Linyao and Chen, Hongyang and Li, Zhao and Ding, Xiao and Wu, Xindong}, 15 | journal={arXiv preprint arXiv:2306.11489}, 16 | year={2023} 17 | } 18 | ``` 19 | 20 | ## Overview 21 | In this repository, we collect recent advances in knowledge graph enhanced large language models. According to the stage at which KGs participate in pre-training, existing methods can be categorized into before-training enhancement, during-training enhancement, and post-training enhancement methods. 22 | 23 | 24 | 25 | ## Table of Contents 26 | - [Awesome-KGLLM](#awesome-kgllm) 27 | - [Overview](#overview) 28 | - [Before-training Enhancement KGPLMs](#before-training-enhancement-kgplms) 29 | - [Expand Input Structures](#expand-input-structures) 30 | - [Enrich Input Information](#enrich-input-information) 31 | - [Generate New Data](#generate-new-data) 32 | - [Optimize Word Masks](#optimize-word-masks) 33 | - [During-training Enhancement KGPLMs](#during-training-enhancement-kgplms) 34 | - [Incorporate Knowledge Encoders](#incorporate-knowledge-encoders) 35 | - [Insert Knowledge Encoding Layers](#insert-knowledge-encoding-layers) 36 | - [Add Independent Adapters](#add-independent-adapters) 37 | - [Modify Pre-training Task](#modify-pre-training-task) 38 | - [Post-training Enhancement KGPLMs](#post-training-enhancement-kgplms) 39 | - [Fine-tune PLMs with Knowledge](#fine-tune-plms-with-knowledge) 40 | - [Generate Knowledge-based Prompts](#generate-knowledge-based-prompts) 41 | 42 | ## Before-training Enhancement KGPLMs 43 | ### Expand Input Structures 44 | - K-bert: Enabling language representation with knowledge graph (AAAI, 2020) [[paper]](https://ojs.aaai.org/index.php/AAAI/article/view/5681/5537) 45 | - CoLAKE: Contextualized Language and Knowledge Embedding (COLING, 2020) [[paper]](https://aclanthology.org/2020.coling-main.327/) 46 | - Cn-hit-it. nlp at semeval-2020 task 4: Enhanced language representation with multiple knowledge triples (SemEval, 2020) [[paper]](https://aclanthology.org/2020.semeval-1.60/) 47 | 48 | ### Enrich Input Information 49 | - LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention (EMNLP, 2020) [[paper]](https://aclanthology.org/2020.emnlp-main.523/) 50 | - E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT (EMNLP, 2020) [[paper]](https://aclanthology.org/2020.findings-emnlp.71/) 51 | - Knowledge-Aware Language Model Pretraining [[paper]](https://arxiv.org/abs/2007.00655) 52 | - OAG-BERT: Towards a Unified Backbone Language Model for Academic Knowledge Services (KDD, 2022) [[paper]](https://dl.acm.org/doi/abs/10.1145/3534678.3539210) 53 | - DKPLM: Decomposable Knowledge-enhanced Pre-trained Language Model for Natural Language Understanding (AAAI, 2022) [[paper]](https://ojs.aaai.org/index.php/AAAI/article/view/21425/21174) 54 | 55 | ### Generate New Data 56 | - Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Models [[paper]](https://arxiv.org/abs/1908.06725) 57 | - KGPT: Knowledge-Grounded Pre-Training for Data-to-Text Generation (EMNLP, 2020) [[paper]](https://aclanthology.org/2020.emnlp-main.697/) 58 | - Barack's wife hillary: Using knowledge-graphs for fact-aware language modeling (ACL, 2019) [[paper]](https://aclanthology.org/P19-1598.pdf) 59 | - Atomic: An atlas of machine commonsense for if-then reasoning (AAAI, 2019) [[paper]](https://ojs.aaai.org/index.php/AAAI/article/view/4160/4038) 60 | - KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation (TACL, 2021) [[paper]](https://aclanthology.org/2021.tacl-1.11.pdf) 61 | 62 | ### Optimize Word Masks 63 | - ERNIE: Enhanced Language Representation with Informative Entities (ACL, 2019) [[paper]](https://aclanthology.org/P19-1139/) 64 | - Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model [[paper]](https://arxiv.org/abs/1912.09637) 65 | - Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning (EMNLP, 2020) [[paper]](https://aclanthology.org/2020.emnlp-main.722.pdf) 66 | 67 | ## During-training Enhancement KGPLMs 68 | ### Incorporate Knowledge Encoders 69 | - ERNIE: Enhanced Language Representation with Informative Entities (ACL, 2019) [[paper]](https://aclanthology.org/P19-1139/) 70 | - ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation [[paper]](https://arxiv.org/abs/2107.02137) 71 | - BERT-MK: Integrating Graph Contextualized Knowledge into Pre-trained Language Models (AI Open, 2021) [[paper]](https://www.sciencedirect.com/science/article/pii/S2666651021000188) 72 | - JointLK: Joint Reasoning with Language Models and Knowledge Graphs for Commonsense Question Answering (NAACL, 2022) [[paper]](https://aclanthology.org/2022.naacl-main.372/) 73 | - Knowledge-Enriched Transformer for Emotion Detection in Textual Conversations (EMNLP-IJCNLP, 2019) [[paper]](https://aclanthology.org/D19-1016/) 74 | - Relational Memory-Augmented Language Models (TACL, 2022) [[paper]](https://aclanthology.org/2022.tacl-1.32/) 75 | - QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering (NAACL, 2021) [[paper]](https://aclanthology.org/2021.naacl-main.45/) 76 | - GreaseLM: Graph REASoning Enhanced Language Models for Question Answering [[paper]](https://arxiv.org/abs/2201.08860) 77 | - KLMo: Knowledge graph enhanced pretrained language model with fine-grained relationships (EMNLP, 2021) [[paper]](https://aclanthology.org/2021.findings-emnlp.384.pdf) 78 | 79 | ### Insert Knowledge Encoding Layers 80 | - K-bert: Enabling language representation with knowledge graph (AAAI, 2020) [[paper]](https://ojs.aaai.org/index.php/AAAI/article/view/5681/5537) 81 | - CoLAKE: Contextualized Language and Knowledge Embedding (COLING, 2020) [[paper]](https://aclanthology.org/2020.coling-main.327/) 82 | - Knowledge Enhanced Contextual Word Representations (EMNLP-IJCNLP, 2019) [[paper]](https://aclanthology.org/D19-1005/) 83 | - JAKET: Joint Pre-training of Knowledge Graph and Language Understanding (AAAI, 2022) [[paper]](https://ojs.aaai.org/index.php/AAAI/article/view/21417/21166) 84 | - KG-BART: Knowledge Graph-Augmented BART for Generative Commonsense Reasoning (AAAI, 2021) [[paper]](https://ojs.aaai.org/index.php/AAAI/article/view/16796/16603) 85 | 86 | ### Add Independent Adapters 87 | - K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters (ACL-IJCNLP, 2021) [[paper]](https://aclanthology.org/2021.findings-acl.121.pdf) 88 | - Common Sense or World Knowledge? Investigating Adapter-Based Knowledge Injection into Pretrained Transformers (DEELIO, 2020) [[paper]](https://aclanthology.org/2020.deelio-1.5/) 89 | - Parameter-Efficient Domain Knowledge Integration from Multiple Sources for Biomedical Pre-trained Language Models (EMNLP, 2021) [[paper]](https://aclanthology.org/2021.findings-emnlp.325/) 90 | - Commonsense knowledge graph-based adapter for aspect-level sentiment classification (Neurocomputing, 2023) [[paper]](https://www.sciencedirect.com/science/article/pii/S0925231223002229) 91 | 92 | ### Modify Pre-training Task 93 | - SenseBERT: Driving Some Sense into BERT (ACL, 2020) [[paper]](https://aclanthology.org/2020.acl-main.423/) 94 | - ERNIE: Enhanced Language Representation with Informative Entities (ACL, 2019) [[paper]](https://aclanthology.org/P19-1139/) 95 | - LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention (EMNLP, 2020) [[paper]](https://aclanthology.org/2020.emnlp-main.523/) 96 | - OAG-BERT: Towards a Unified Backbone Language Model for Academic Knowledge Services (KDD, 2022) [[paper]](https://dl.acm.org/doi/abs/10.1145/3534678.3539210) 97 | - Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model [[paper]](https://arxiv.org/abs/1912.09637) 98 | - Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning (EMNLP, 2020) [[paper]](https://aclanthology.org/2020.emnlp-main.722.pdf) 99 | - ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning (ACL-IJCNLP, 2021) [[paper]](https://aclanthology.org/2021.acl-long.260/) 100 | - SentiLARE: Sentiment-Aware Language Representation Learning with Linguistic Knowledge (EMNLP, 2020) [[paper]](https://aclanthology.org/2020.emnlp-main.567/) 101 | 102 | ## Post-training Enhancement KGPLMs 103 | ### Fine-tune PLMs with Knowledge 104 | - KALA: Knowledge-Augmented Language Model Adaptation (NAACL, 2022) [[paper]](https://aclanthology.org/2022.naacl-main.379/) 105 | - Pre-trained language models with domain knowledge for biomedical extractive summarization (Knowledge-Based Systems, 2022) [[paper]](https://www.sciencedirect.com/science/article/pii/S0950705122007328) 106 | - KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoning (EMNLP-IJCNLP, 2019) [[paper]](https://aclanthology.org/D19-1282/) 107 | - Enriching contextualized language model from knowledge graph for biomedical information extraction (Briefings in bioinformatics, 2021) [[paper]](https://academic.oup.com/bib/article/22/3/bbaa110/5854405) 108 | - Incorporating Commonsense Knowledge Graph in Pretrained Models for Social Commonsense Tasks (DeeLIO, 2020) [[paper]](https://aclanthology.org/2020.deelio-1.9/) 109 | 110 | ### Generate Knowledge-based Prompts 111 | - Benchmarking Knowledge-Enhanced Commonsense Question Answering via Knowledge-to-Text Transformation (AAAI, 2021) [[paper]](https://ojs.aaai.org/index.php/AAAI/article/view/17490/17297) 112 | - Enhanced story comprehension for large language models through dynamic document-based knowledge graphs (AAAI, 2022) [[paper]](https://ojs.aaai.org/index.php/AAAI/article/view/21286) 113 | - Knowledge Prompting in Pre-trained Language Model for Natural Language Understanding (EMNLP, 2022) [[paper]](https://aclanthology.org/2022.emnlp-main.207.pdf) 114 | --------------------------------------------------------------------------------