├── LICENSE ├── README.md ├── _config.yaml └── tutorial-edbt25 └── index.md /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2024 Dylan Ma 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # LLM-KG4QA: Large Language Models and Knowledge Graphs for Question Answering 2 | 3 | [![Awesome](https://awesome.re/badge.svg)](https://awesome.re) ![](https://img.shields.io/github/last-commit/machuangtao/LLM-KG4QA?color=blue) ![Stars](https://img.shields.io/github/stars/machuangtao/LLM-KG4QA?color=blue) ![Forks](https://img.shields.io/github/forks/machuangtao/LLM-KG4QA?color=blue&label=Fork) 4 | 5 | ## 🔔 News 6 | - **`2025-05`** The preprint of our extended survey is avaliable on **[arXiv](https://arxiv.org/abs/2505.20099)**. 7 | - **`2025-02`** Our [tutorial](https://machuangtao.github.io/LLM-KG4QA/tutorial-edbt25) was accepted to be presented at **EDBT2025** 8 | - **`2024-12`** We create this repository to maintain a paper list on **LLMs and KGs for QA.** 9 | 10 | If you find our work is useful, please cite our paper by using the following BibTeX entry. 11 | 12 | ``` 13 | @article{ma2025llmkg4qa, 14 | title={Large Language Models Meet Knowledge Graphs for Question Answering: Synthesis and Opportunities}, 15 | author={Ma, Chuangtao and Chen, Yongrui and Wu, Tianxing and Khan, Arijit and Wang, Haofen}, 16 | journal={arXiv preprint arXiv:2505.20099}, 17 | year={2025} 18 | } 19 | ``` 20 | 21 | ## Content 22 | - [LLM and KGs for QA](#1-llms-and-kgs-for-qa) 23 | - [KGs as Background Knowledge](#kgs-as-background-knowledge) 24 | - [KGs as Reasoning Guideline](#kgs-as-reasoning-guideline) 25 | - [KGs as Refiner and Filter](#kgs-as-refiner-and-filter) 26 | 27 | - [Complex QA](#2-complex-qa) 28 | - [Explainable QA](#explainable-qa) 29 | - [Multi-modal QA](#multi-modal-qa) 30 | - [Multi-document QA](#multi-document-qa) 31 | - [Multi-Hop QA](#multi-hop-qa) 32 | - [Multi-run and Conversational QA](#multi-run-and-conversational-qa) 33 | - [Temporal QA](#temporal-qa) 34 | - [Multi-domain and Multilingual QA](#multi-domain-and-multilingual-qa) 35 | - [Advanced Topics](#3-advanced-topics) 36 | - [Optimization](#optimization) 37 | - [Data Management](#data-management) 38 | - [Benchmark and Applications](#4-benchmark-and-applications) 39 | - [Benchmark Dataset](#benchmark-dataset) 40 | - [Industrial and Scientific Applications](#industrial-and-scientific-applications) 41 | - [Demo](#demo) 42 | - [Related Survey](#5-related-survey) 43 | --- 44 | ## 1. LLMs and KGs for QA 45 | 46 | ### KGs as Background Knowledge 47 | #### Pre-training and Fine-tuning 48 | 49 | | NO | Title | Venue | Year | Category |Paper Link | 50 | |----|--------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|------|------------------------------------------------|------------------------------------------------------------------------------------------------| 51 | | 1 | Deep Bidirectional Language-Knowledge Graph pretraining | NeurIPS | 2022| Pre-training | [Link](https://proceedings.neurips.cc/paper_files/paper/2022/file/f224f056694bcfe465c5d84579785761-Paper-Conference.pdf) 52 | | 2 | GreaseLM: Graph REASoning Enhanced Language Models | ICLR | 2022 | Pre-training | [Link](https://openreview.net/forum?id=41e9o6cQPj) 53 | | 3 | InfuserKI: Enhancing Large Language Models with Knowledge Graphs via Infuser-Guided Knowledge Integration | LLM+KG@VLDB | 2024 | Pre-training | [Link](https://arxiv.org/abs/2402.11441) 54 | | 4 | Large Language Models Meet Knowledge Graphs to Answer Factoid Questions | PACLIC | 2023 | Pre-training | [Link](https://aclanthology.org/2023.paclic-1.63/) 55 | | 5 | KaLM: Knowledge-aligned Autoregressive Language Modeling via Dual-view Knowledge Graph Contrastive Learning | arXiv | 2024 | Pre-training | [Link](https://arxiv.org/abs/2412.04948) 56 | | 6 | KBLaM: Knowledge Base augmented Language Model | ICLR | 2025 | Pre-training | [Link](https://openreview.net/forum?id=aLsMzkTej9) 57 | | 7 | KnowLA: Enhancing Parameter-efficient Finetuning with Knowledgeable Adaptation | NAACL | 2024 | Fine-Tuning | [Link](https://aclanthology.org/2024.naacl-long.396/) 58 | | 8 | KG-Adapter: Enabling Knowledge Graph Integration in Large Language Models through Parameter-Efficient Fine-Tuning | ACL Findlings | 2024 | Fine-Tuning | [Link](https://aclanthology.org/2024.findings-acl.229/) 59 | | 9 | A GAIL Fine-Tuned LLM Enhanced Framework for Low-Resource Knowledge Graph Question Answering | CIKM | 2024 | Fine-Tuning | [Link](https://dl.acm.org/doi/10.1145/3627673.3679753) 60 | | 10 | Knowledge Graph Finetuning Enhances Knowledge Manipulation in Large Language Models | ICLR | 2025 | Fine-Tuning | [Link](https://openreview.net/forum?id=oMFOKjwaRS) 61 | | 11 | KLearn Together: Joint Multitask Finetuning of Pretrained KG-enhanced LLM for Downstream Tasks | GenAIK@COLING | 2025 | Fine-Tuning | [Link](https://aclanthology.org/2025.genaik-1.2/) 62 | | 12 | Improving Pre-trained Language Models with Knowledge Enhancement and Filtering Framework | NAACL Findings | 2025 | Fine-Tuning | [Link](https://aclanthology.org/2025.findings-naacl.213/) 63 | | 13 | Prompting Large Language Models with Knowledge Graphs for Question Answering Involving Long-tail Facts | arXiv | 2024 | KG-Augmented Prompting | [Link](https://arxiv.org/abs/2405.06524) 64 | | 14 | KnowGPT: Knowledge Graph based Prompting for Large Language Models | arXiv | 2024 | KG-Augmented Prompting | [Link](https://arxiv.org/abs/2312.06185) 65 | | 15 | Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering | NLRSE | 2023 | KG-Augmented Prompting | [Link](https://aclanthology.org/2023.nlrse-1.7/) 66 | | 16 | Retrieve-Rewrite-Answer: A KG-to-Text Enhanced LLMs Framework for Knowledge Graph Question Answering | IJCKG | 2023 | KG-Augmented Prompting | [Link](https://ijckg2023.knowledge-graph.jp/pages/proc/paper_30.pdf/) 67 | | 17 | Mitigating LLM Hallucinations with Knowledge Graphs: A Case Study | arXiv | 2025 | KG-Augmented Prompting | [Link](https://arxiv.org/abs/2504.12422/) 68 | 69 | 70 | #### RAG (Retrieval Augmented Generation) 71 | 72 | | NO | Title | Venue | Year | Category |Paper Link | 73 | |----|--------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|------|------------------------------------------------|------------------------------------------------------------------------------------------------| 74 | | 1 | Enhancing Textbook Question Answering Task with Large Language Models and Retrieval Augmented Generation| arXiv | 2024 | RAG | [Link](https://arxiv.org/abs/2402.05128) 75 | | 2 | Retrieval-enhanced Knowledge Editing in Language Models for Multi-Hop Question Answering | CIKM | 2024 | RAG | [Link](https://dl.acm.org/doi/abs/10.1145/3627673.3679722) 76 | | 3 | Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generation | arXiv | 2024 | RAG | [Link](https://arxiv.org/abs/2406.18676) 77 | | 4 | RAG-based Question Answering over Heterogeneous Data and Text | arXiv | 2024 | RAG | [Link](https://arxiv.org/abs/2412.07420) 78 | | 5 | Awakening Augmented Generation: Learning to Awaken Internal Knowledge of Large Language Models for Question Answering | COLING | 2025 | RAG | [Link](https://aclanthology.org/2025.coling-main.89/) 79 | | 6 | SAGE: A Framework of Precise Retrieval for RAG | arXiv | 2025 | RAG | [Link](https://arxiv.org/abs/2503.01713) 80 | | 7 | From Local to Global: A Graph RAG Approach to Query-Focused Summarization | arXiv | 2024 | Graph RAG| [Link](https://arxiv.org/abs/2404.16130) 81 | | 8 | LightRAG: Simple and Fast Retrieval-Augmented Generatio | arXiv | 2024 | Graph RAG | [Link](https://arxiv.org/abs/2410.05779) 82 | | 9 | GRAG: Graph Retrieval-Augmented Generation | arXiv | 2024 | Graph RAG | [Link](https://arxiv.org/abs/2405.16506) 83 | | 10 | HybGRAG: Hybrid Retrieval-Augmented Generation on Textual and Relational Knowledge Bases | arXiv | 2024 | Graph RAG | [Link](https://arxiv.org/abs/2412.16311) 84 | | 11 | CG-RAG: Research Question Answering by Citation Graph Retrieval-Augmented LLMs | arXiv | 2025 | Graph RAG | [Link](https://arxiv.org/abs/2501.15067) 85 | | 12 | MiniRAG: Towards Extremely Simple Retrieval-Augmented Generation | arXiv | 2025 | Graph RAG | [Link](https://arxiv.org/abs/2501.06713) 86 | | 13 | GFM-RAG: Graph Foundation Model for Retrieval Augmented Generation | arXiv | 2025 | Graph RAG | [Link](https://arxiv.org/abs/2502.01113) 87 | | 14 | MSG-LLM: A Multi-scale Interactive Framework for Graph-enhanced Large Language Models | COLING | 2025 | Graph RAG | [Link](https://aclanthology.org/2025.coling-main.648/) 88 | | 15 | PathRAG: Pruning Graph-based Retrieval Augmented Generation with Relational Paths | arXiv | 2025 | Graph RAG | [Link](https://arxiv.org/abs/2502.14902/) 89 | | 16 | In-depth Analysis of Graph-based RAG in a Unified Framework | arXiv | 2025 | Graph RAG | [Link](https://www.arxiv.org/abs/2503.04338/) 90 | | 17 | Empowering GraphRAG with Knowledge Filtering and Integration | arXiv | 2025 | Graph RAG | [Link](https://arxiv.org/abs/2503.13804/) 91 | | 18 | Graph Neural Network Enhanced Retrieval for Question Answering of Large Language Models | NAACL | 2025 | Graph RAG | [Link](https://aclanthology.org/2025.naacl-long.337/) 92 | | 19 | NodeRAG: Structuring Graph-based RAG with Heterogeneous Nodes | arXiv | 2025 | Graph RAG | [Link](https://arxiv.org/abs/2504.11544/) 93 | | 20 | KG-RAG: Bridging the Gap Between Knowledge and Creativity | arXiv | 2024 | KG RAG | [Link](https://arxiv.org/abs/2405.12035) 94 | | 21 | Knowledge Graph-extended Retrieval Augmented Generation for Question Answering | arXiv | 2025 | KG RAG | [Link](https://arxiv.org/abs/2504.08893) 95 | | 22 | Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answering| SIGIR | 2024 | KG RAG | [Link](https://dl.acm.org/doi/10.1145/3626772.3661370) 96 | | 23 | REnhancing Large Language Models with Knowledge Graphs for Robust Question Answering | ICPADS | 2024 | KG RAG | [Link](https://doi.ieeecomputersociety.org/10.1109/ICPADS63350.2024.00042) 97 | | 24 | FRAG: A Flexible Modular Framework for Retrieval-Augmented Generation based on Knowledge Graphs | arXiv | 2025 | KG RAG | [Link](https://arxiv.org/abs/2501.09957) 98 | | 25 | SimGRAG: Leveraging Similar Subgraphs for Knowledge Graphs Driven Retrieval-Augmented Generation | arXiv | 2025 | KG RAG | [Link](https://arxiv.org/abs/2412.15272) 99 | | 26 | RGR-KBQA: Generating Logical Forms for Question Answering Using Knowledge-Graph-Enhanced Large Language Model | COLING | 2025 | KG RAG | [Link](https://aclanthology.org/2025.coling-main.205) 100 | | 27 | Knowledge Graph-Guided Retrieval Augmented Generation | arXiv | 2025 | KG RAG | [Link](https://arxiv.org/abs/2502.06864) 101 | | 28 | Simple Is Effective: The Roles of Graphs and Large Language Models in Knowledge-Graph-Based Retrieval-Augmented Generation | ICLR | 2025 | KG RAG | [Link](https://openreview.net/forum?id=JvkuZZ04O7) 102 | | 29 | CoT-RAG: Integrating Chain of Thought and Retrieval-Augmented Generation to Enhance Reasoning in Large Language Models | arXiv | 2025 | KG RAG | [Link](https://arxiv.org/abs/2504.13534) 103 | | 30 | A Pilot Empirical Study on When and How to Use Knowledge Graphs as Retrieval Augmented Generation | arXiv | 2025 | KG RAG | [Link](https://arxiv.org/abs/2502.20854) 104 | | 31 | A Systematic Exploration of Knowledge Graph Alignment with Large Language Models in Retrieval Augmented Generation | AAAI | 2025 | KG RAG | [Link](https://ojs.aaai.org/index.php/AAAI/article/view/34716) 105 | | 32 | Evaluating Knowledge Graph Based Retrieval Augmented Generation Methods under Knowledge Incompleteness | arXiv | 2025 | KG RAG | [Link](https://arxiv.org/abs/2504.05163) 106 | | 33 | RAG-KG-IL: A Multi-Agent Hybrid Framework for Reducing Hallucinations and Enhancing LLM Reasoning through RAG and Incremental Knowledge Graph Learning Integration | arXiv | 2025 | KG RAG | [Link](https://arxiv.org/abs/2503.13514) 107 | | 34 | Empowering LLMs by hybrid retrieval-augmented generation for domain-centric Q&A in smart manufacturing | Advanced Engineering Informatics | 2025| Hybrid RAG | [Link](https://doi.org/10.1016/j.aei.2025.103212) 108 | | 35 | Spatial-RAG: Spatial Retrieval Augmented Generation for Real-World Spatial Reasoning Questions | arXiv | 2025 | Spatial RAG | [Link](https://arxiv.org/abs/2502.18470) 109 | | 36 | Spatial-RAG: Spatial Retrieval Augmented Generation for Real-World Spatial Reasoning Questions | arXiv | 2025 | Spatial RAG | [Link](https://arxiv.org/abs/2502.18470) 110 | 111 | ### KGs as Reasoning Guideline 112 | 113 | | NO | Title | Venue | Year | Category |Paper Link | 114 | |----|--------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|------|------------------------------------------------|------------------------------------------------------------------------------------------------| 115 | | 1 | Subgraph Retrieval Enhanced Model for Multi-hop Knowledge Base Question Answerings | ACL | 2022 | Offline KG Guidelines | [Link](https://aclanthology.org/2022.acl-long.396/) 116 | | 2 | keqing: knowledge-based question answering is a nature chain-of-thought mentor of LLM | arXiv | 2023 | Offline KG Guidelines | [Link](https://arxiv.org/abs/2401.00426) 117 | | 3 | Explore then Determine: A GNN-LLM Synergy Framework for Reasoning over Knowledge Graph | arXiv | 2024 | Offline KG Guidelines | [Link](https://arxiv.org/abs/2406.01145) 118 | | 4 | Graph-constrained Reasoning: Faithful Reasoning on Knowledge Graphs with Large Language Models | arXiv | 2024 | Offline KG Guidelines | [Link](https://arxiv.org/abs/2410.13080) 119 | | 5 | Reasoning with Trees: Faithful Question Answering over Knowledge Graph | COLING | 2025 | Offline KG Guidelines | [Link](https://aclanthology.org/2025.coling-main.211/) 120 | | 6 | Empowering Language Models with Knowledge Graph Reasoning for Open-Domain Question Answering | EMNLP | 2022 | Online KG Guildlines | [Link](https://aclanthology.org/2022.emnlp-main.650/) 121 | | 7 | Knowledge-Enhanced Iterative Instruction Generation and Reasoning for Knowledge Base Question Answering | NLPCC | 2022 | Online KG Guildlines | [Link](https://link.springer.com/chapter/10.1007/978-3-031-17120-8_34) 122 | | 8 | Evaluating and Enhancing Large Language Models for Conversational Reasoning on Knowledge Graphs | arXiv | 2023 | Online KG Guildlines | [Link](https://arxiv.org/abs/2312.11282) 123 | | 9 | Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph | ICLR | 2024 | Online KG Guildlines | [Link](https://openreview.net/forum?id=nnVO1PvbTv) 124 | | 10 | Think-on-Graph 2.0: Deep and Faithful Large Language Model Reasoning with Knowledge-guided Retrieval Augmented Generation | ICLR | 2024 | Online KG Guildlines | [Link](https://openreview.net/forum?id=oFBu7qaZpS) 125 | | 11 | KARPA: A Training-free Method of Adapting Knowledge Graph as References for Large Language Model's Reasoning Path Aggregation | arXiv | 2024 | Online KG Guildlines | [Link](https://arxiv.org/abs/2412.20995) 126 | | 12 | Retrieval and Reasoning on KGs: Integrate Knowledge Graphs into Large Language Models for Complex Question Answering | EMNLP | 2024 | Online KG Guildlines | [Link](https://aclanthology.org/2024.findings-emnlp.446) 127 | | 13 | KG-Agent: An Efficient Autonomous Agent Framework for Complex Reasoning over Knowledge Grap | arXiv | 2024 | Agent-based KG Guildlines | [Link](https://arxiv.org/abs/2402.11163) 128 | | 14 | ODA: Observation-Driven Agent for integrating LLMs and Knowledge Graphs | ACL Findings | 2024 | Agent-based KG Guildlines | [Link](https://aclanthology.org/2024.findings-acl.442/) | 129 | | 15 | A Collaborative Reasoning Framework Powered by Reinforcement Learning and Large Language Models for Complex Questions Answering over Knowledge Graph | COLING| 2025 | Collaborative Reasoning | [Link](https://aclanthology.org/2025.coling-main.712/) 130 | | 16 | Rule-KBQA: Rule-Guided Reasoning for Complex Knowledge Base Question Answering with Large Language Models | COLING| 2025 | Rule-Guided Reasoning | [Link](https://aclanthology.org/2025.coling-main.562/) 131 | 132 | ### KGs as Refiner and Filter 133 | 134 | | NO | Title | Venue | Year | Category |Paper Link | 135 | |----|--------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|------|------------------------------------------------|------------------------------------------------------------------------------------------------| 136 | | 1 | Answer Candidate Type Selection: Text-To-Text Language Model for Closed Book Question Answering Meets Knowledge Graphs | KONVENS | 2023 | KG-Driven Filtering and Validation | [Link](https://aclanthology.org/2023.konvens-main.16/) 137 | | 2 | KG-Rank: Enhancing Large Language Models for Medical QA with Knowledge Graphs and Ranking Techniques | BioNLP Workshop | 2024 |KG-Driven Filtering and Validation | [Link](https://aclanthology.org/2024.bionlp-1.13/) 138 | | 3 | Mitigating Large Language Model Hallucinations via Autonomous Knowledge Graph-based Retrofitting | AAAI | 2024 |KG-Driven Filtering and Validation | [Link](https://ojs.aaai.org/index.php/AAAI/article/view/29770/31326) 139 | | 4 | Evidence-Focused Fact Summarization for Knowledge-Augmented Zero-Shot Question Answering | ariXv | 2024 |KG-Augmented Output Refinement | [Link](https://arxiv.org/abs/2403.02966) 140 | | 5 | Interactive-KBQA: Multi-Turn Interactions for Knowledge Base Question Answering with Large Language Models | ACL | 2024 |KG-Augmented Output Refinement | [Link](https://aclanthology.org/2024.acl-long.569/) 141 | | 6 | Learning to Plan for Retrieval-Augmented Large Language Models from Knowledge Graphs | arXiv | 2024 |KG-Augmented Output Refinement | [Link](https://arxiv.org/abs/2406.14282) 142 | | 7 | Optimizing Knowledge Integration in Retrieval-Augmented Generation with Self-Selection | arXiv | 2025 |RAG-based Answers Selection | [Link](https://arxiv.org/abs/2502.06148) 143 | 144 | ## 2. Complex QA 145 | 146 | ### Explainable QA 147 | 148 | | NO | Title | Venue | Year | Category |Paper Link | 149 | |----|--------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|------|------------------------------------------------|------------------------------------------------------------------------------------------------| 150 | | 1 | Reasoning over Hierarchical Question Decomposition Tree for Explainable Question Answering | ACL | 2024 | - | [Link](https://aclanthology.org/2023.acl-long.814/) 151 | | 2 | Explainable Conversational Question Answering over Heterogeneous Sources via Iterative Graph Neural Networks | SIGIR | 2023 | - | [Link](https://dl.acm.org/doi/10.1145/3539618.3591682) 152 | | 3 | Retrieval In Decoder benefits generative models for explainable complex question answering | Neural Networks | 2025 | - | [Link](https://doi.org/10.1016/j.neunet.2024.106833) 153 | 154 | ### Multi-Modal QA 155 | 156 | | NO | Title | Venue | Year | Category |Paper Link | 157 | |----|--------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|------|------------------------------------------------|------------------------------------------------------------------------------------------------| 158 | | 1 | Lako: Knowledge-driven visual question answering via late knowledge-to text injection | IJCKG | 2022 | VQA | [Link](https://dl.acm.org/doi/10.1145/3579051.3579053) 159 | | 2 | Modality-Aware Integration with Large Language Models for Knowledge-Based Visual Question Answering | ACL | 2024 | VQA | [Link](https://aclanthology.org/2024.acl-long.132/) 160 | | 3 | Knowledge-Enhanced Visual Question Answering with Multi-modal Joint Guidance |JCKG | 2024 | VQA | [Link](https://dl.acm.org/doi/10.1145/3579051.3579073) 161 | | 4 | ReasVQA: Advancing VideoQA with Imperfect Reasoning Process |arXiv | 2025 | VQA | [Link](https://arxiv.org/abs/2501.13536) 162 | | 5 | Fine-grained knowledge fusion for retrieval-augmented medical visual question answering | Information Fusion | 2025 | VQA | [Link](https://doi.org/10.1016/j.inffus.2025.103059) 163 | | 6 | RAMQA: A Unified Framework for Retrieval-Augmented Multi-Modal Question Answering | arXiv | 2025 | Multi-Modal QA | [Link](https://arxiv.org/abs/2501.13297) 164 | | 7 | MuRAR: A Simple and Effective Multimodal Retrieval and Answer Refinement Framework for Multimodal Question Answering | arXiv | 2024 | Multi-Modal QA | [Link](https://arxiv.org/abs/2408.08521) 165 | 166 | 167 | ### Multi-Document QA 168 | 169 | | NO | Title | Venue | Year | Category |Paper Link | 170 | |----|--------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|------|------------------------------------------------|------------------------------------------------------------------------------------------------| 171 | | 1 | Knowledge Graph Prompting for Multi-Document Question Answering | AAAI | 2024 | Multi-doc QA | [Link](https://ojs.aaai.org/index.php/AAAI/article/view/29889) 172 | | 2 | CuriousLLM: Elevating Multi-Document QA with Reasoning-Infused Knowledge Graph Prompting | arXiv | 2024 | Multi-doc QA | [Link](https://arxiv.org/abs/2404.09077) 173 | | 3 | VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generation | arXiv | 2024 | Multi-doc QA | [Link](https://arxiv.org/abs/2412.10704) 174 | 175 | ### Multi-Hop QA 176 | 177 | | NO | Title | Venue | Year | Category |Paper Link | 178 | |----|--------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|------|------------------------------------------------|------------------------------------------------------------------------------------------------| 179 | | 1 | GraphLLM: A General Framework for Multi-hop Question Answering over Knowledge Graphs Using Large Language Models | NLPCC | 2024 | Multi-Hop QA | [Link](https://link.springer.com/chapter/10.1007/978-981-97-9431-7_11) 180 | | 2 | LLM-KGMQA: Large Language Model-Augmented Multi-Hop Question-Answering System based on Knowledge Graph in Medical Field | KBS | 2024 | Multi-Hop QA | [Link](https://doi.org/10.21203/rs.3.rs-4721418/v1) 181 | | 3 | PokeMQA: Programmable knowledge editing for Multi-hop Question Answering | ACL | 2024 | Multi-Hop QA | [Link](https://aclanthology.org/2024.acl-long.438/) 182 | | 4 | HOLMES: Hyper-Relational Knowledge Graphs for Multi-hop Question Answering using LLMs | ACL | 2024 | Multi-Hop QA | [Link](https://aclanthology.org/2024.acl-long.717/) 183 | | 5 | LLM-Based Multi-Hop Question Answering with Knowledge Graph Integration in Evolving Environments | EMNLP | 2024 | Multi-Hop QA | [Link](https://aclanthology.org/2024.findings-emnlp.844/) 184 | | 6 | SG-RAG: Multi-Hop Question Answering With Large Language Models Through Knowledge Graphs | ICNLSP | 2024 | Multi-Hop QA | [Link](https://aclanthology.org/2024.icnlsp-1.45/) 185 | | 7 | From Superficial to Deep: Integrating External Knowledge for Follow-up Question Generation Using Knowledge Graph and LLM | COLING | 2025 | Multi-Hop QA | [Link](https://aclanthology.org/2025.coling-main.55/) 186 | | 8 | Multi-Hop Question Answering with LLMs & Knowledge Graphs | Blog | 2023 | Multi-Hop QA | [Link](https://www.wisecube.ai/blog-2/multi-hop-question-answering-with-llms-knowledge-graphs/) 187 | | 9 | Mitigating Lost-in-Retrieval Problems in Retrieval Augmented Multi-Hop Question Answering | arXiv | 2025 | Multi-Hop QA | [Link](https://arxiv.org/abs/2502.14245) 188 | | 10 | Knowledge Graph Based Retrieval-Augmented Generation for Multi-Hop Question Answering Enhancement | IEEE IKT | 2024 | Multi-Hop QA | [Link](https://ieeexplore.ieee.org/abstract/document/10892619) 189 | | 11 | A Framework of Knowledge Graph-Enhanced Large Language Model Based on Question Decomposition and Atomic Retrieval | EMNLP Findings | 2024 | Multi-Hop QA | [Link](https://aclanthology.org/2024.findings-emnlp.670/) 190 | 191 | ### Multi-run and Conversational QA 192 | 193 | | NO | Title | Venue | Year | Category | Paper Link | 194 | |----|--------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|------|------------------------------------------------|------------------------------------------------------------------------------------------------| 195 | | 1 | Explainable Conversational Question Answering over Heterogeneous Sources via Iterative Graph Neural Networks | SIGIR | 2023 | Conversational QA | [Link](https://dl.acm.org/doi/10.1145/3539618.3591682) 196 | | 2 | Conversational Question Answering with Language Models Generated Reformulations over Knowledge Graph | ACL Findings | 2024 | Conversational QA | [Link](https://aclanthology.org/2024.findings-acl.48/) 197 | | 3 | LLM-Based Multi-Hop Question Answering with Knowledge Graph Integration in Evolving Environments | EMNLP | 2024 | Multi-Hop QA | [Link](https://aclanthology.org/2024.findings-emnlp.844/) 198 | | 4 | Learning When to Retrieve, What to Rewrite, and How to Respond in Conversational QA | EMNLP | 2024 | Conversational QA | [Link](https://aclanthology.org/2024.findings-emnlp.622) 199 | | 5 | ConvKGYarn: Spinning Configurable and Scalable Conversational Knowledge Graph QA Datasets with Large Language Models | EMNLP | 2024 | Conversational QA | [Link](https://aclanthology.org/2024.emnlp-industry.89) 200 | | 6 | Dialogue Benchmark Generation from Knowledge Graphs with Cost-Effective Retrieval-Augmented LLMs | SIGMOD | 2025 | Dialogue | [Link](https://arxiv.org/abs/2501.09928) 201 | 202 | ### Temporal QA 203 | 204 | | NO | Title | Venue | Year | Category | Paper Link | 205 | |----|--------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|------|------------------------------------------------|------------------------------------------------------------------------------------------------| 206 | | 1 | KG-IRAG: A Knowledge Graph-Based Iterative Retrieval-Augmented Generation Framework for Temporal Reasoning | arXiv | 2025 | Temporal QA | [Link](https://arxiv.org/abs/2503.14234) 207 | | 2 | Two-stage Generative Question Answering on Temporal Knowledge Graph Using Large Language Models | ACL Findings | 2024 | Temporal QA | [Link](https://aclanthology.org/2024.findings-acl.401) 208 | | 3 | TimeR4 : Time-aware Retrieval-Augmented Large Language Models for Temporal Knowledge Graph Question Answering | EMNLP | 2024 | Temporal QA | [Link](https://aclanthology.org/2024.emnlp-main.394/) 209 | 210 | 211 | ### Multi-domain and Multilingual QA 212 | 213 | | NO | Title | Venue | Year | Category | Paper Link | 214 | |----|--------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|------|------------------------------------------------|------------------------------------------------------------------------------------------------| 215 | | 1 | MKG-Rank: Enhancing Large Language Models with Knowledge Graph for Multilingual Medical Question Answering | arXiv | 2025 | Multilingual QA | [Link](https://arxiv.org/abs/2503.16131) 216 | | 2 | Language Models as SPARQL Query Filtering for Improving the Quality of Multilingual Question Answering over Knowledge Graphs | IWCE | 2024 | Multilingual QA | [Link](https://link.springer.com/chapter/10.1007/978-3-031-62362-2_1) 217 | 218 | 219 | ## 3. Advanced Topics 220 | 221 | ### Optimization 222 | 223 | | NO | Title | Venue | Year | Category | Paper Link | 224 | |----|--------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|------|------------------------------------------------|------------------------------------------------------------------------------------------------| 225 | | 1 | Empowering Large Language Models to Set up a Knowledge Retrieval Indexer via Self-Learning | arXiv | 2023 | Index-based Optimization| [Link](https://arxiv.org/abs/2405.16933) 226 | | 2 | Graph of Records: Boosting Retrieval Augmented Generation for Long-context Summarization with Graphs | ICLR | 2024 |Index-based Optimization | [Link](https://openreview.net/forum?id=6LKmaC4cO0/) 227 | | 3 | KG-Retriever: Efficient Knowledge Indexing for Retrieval-Augmented Large Language Models | arXiv | 2024 |Index-based Optimization | [Link](https://arxiv.org/abs/2412.05547) 228 | | 4 | Prompting Is Programming: A Query Language for Large Language Models | PLDL | 2023 |Prompting-based Optimization | [Link](https://dl.acm.org/doi/10.1145/3591300/) 229 | | 5 | LLM as Prompter: Low-resource Inductive Reasoning on Arbitrary Knowledge Graphs | ACL Findings | 2024 |Prompting-based Optimization | [Link](https://aclanthology.org/2024.findings-acl.224/) 230 | | 6 | LightRAG: Simple and Fast Retrieval-Augmented Generation | arXiv | 2024 | Graph retrieval-based optimization | [Link](https://arxiv.org/abs/2410.05779) 231 | | 7 | Clue-Guided Path Exploration: Optimizing Knowledge Graph Retrieval with Large Language Models to Address the Information Black Box Challenge | arXiv | 2024 | Graph retrieval-based optimization | [Link](https://arxiv.org/abs/2401.13444) 232 | | 8 | Optimizing open-domain question answering with graph-based retrieval augmented generation | arXiv | 2025 | Graph retrieval-based optimization | [Link](https://arxiv.org/abs/2503.02922) 233 | | 9 | Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generation | WWW | 2025 | Graph retrieval-based optimization | [Link](https://openreview.net/forum?id=2ZaqnRIUCV) 234 | | 10 | Optimizing Knowledge Integration in Retrieval-Augmented Generation with Self-Selection | arXiv | 2025 | Graph retrieval-based optimization | [Link](https://arxiv.org/abs/2502.06148) 235 | | 11 | Systematic Knowledge Injection into Large Language Models via Diverse Augmentation for Domain-Specific RAG | arXiv | 2025 | Graph retrieval-based optimization | [Link](https://arxiv.org/abs/2502.08356) 236 | | 12 | KG-Rank: Enhancing Large Language Models for Medical QA with Knowledge Graphs and Ranking Techniques | BioNLP Workshop | 2024 | Ranking-based optimization | [Link](https://aclanthology.org/2024.bionlp-1.13/) 237 | | 13 | KS-LLM: Knowledge Selection of Large Language Models with Evidence Document for Question Answering | arXiv | 2024 | Ranking-based optimization | [Link](https://arxiv.org/abs/2404.15660) 238 | | 14 | RAG-based Question Answering over Heterogeneous Data and Text | arXiv | 2024 | Ranking-based optimization | [Link](https://arxiv.org/abs/2412.07420) 239 | | 15 | Cost-efficient Knowledge-based Question Answering with Large Language Models | arXiv | 2024 | Cost-based optimization | [Link](https://arxiv.org/abs/2405.17337) 240 | | 16 | KGLens: Towards Efficient and Effective Knowledge Probing of Large Language Models with Knowledge Graphs | arXiv | 2024 | Cost-based optimization | [Link](https://arxiv.org/abs/2312.11539) 241 | | 17 | Knowledge Graph-Enhanced Large Language Models via Path Selection | ACL Findings | 2024 | Path-based optimization | [Link](https://aclanthology.org/2024.findings-acl.376) 242 | | 18 | LEGO-GraphRAG: Modularizing Graph-based Retrieval-Augmented Generation for Design Space Exploration | arXiv | 2024 | Path-based optimization | [Link](https://arxiv.org/abs/2411.05844) 243 | | 19 | Query Optimization for Parametric Knowledge Refinement in Retrieval-Augmented Large Language Models | arXiv | 2024 | Query-based optimization | [Link](https://arxiv.org/abs/2411.07820) 244 | | 20 | A MapReduce Approach to Effectively Utilize Long Context Information in Retrieval Augmented Language Models | arXiv | 2024 | MapReduce-based optimization | [Link](https://arxiv.org/abs/2412.15271) 245 | | 21 | PIP-KAG: Mitigating Knowledge Conflicts in Knowledge-Augmented Generation via Parametric Pruning | arXiv | 2025 | Knowledge conflicts mitigation | [Link](https://arxiv.org/abs/2502.15543) 246 | | 22 | Direct Retrieval-augmented Optimization: Synergizing Knowledge Selection and Language Models | arXiv | 2025 | Knowledge conflicts mitigation | [Link](https://arxiv.org/abs/2505.03075) 247 | 248 | 249 | ### Data Management 250 | 251 | | NO | Title | Venue | Year | Category | Paper Link | 252 | |----|--------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|------|------------------------------------------------|------------------------------------------------------------------------------------------------| 253 | | 1 | Triple Augmented Generative Language Models for SPARQL Query Generation from Natural Language Questions | arXiv | 2024 | NL2GQL | [Link](https://dl.acm.org/doi/10.1145/3673791.3698426) 254 | | 2 | R3-NL2GQL: A Model Coordination and Knowledge Graph Alignment Approach for NL2GQL | ACL Findings | 2024 | NL2GQL | [Link](https://aclanthology.org/2024.findings-emnlp.800/) 255 | | 3 | Aligning Large Language Models to a Domain-specific Graph Database for NL2GQL | CIKM | 2024 | NL2GQL | [Link](https://dl.acm.org/doi/10.1145/3627673.3679713) 256 | | 4 | UniOQA: A Unified Framework for Knowledge Graph Question Answering with Large Language Models | arXiv | 2024 | NL2GQL | [Link](https://arxiv.org/abs/2406.02110) 257 | | 5 | NAT-NL2GQL: A Novel Multi-Agent Framework for Translating Natural Language to Graph Query Language | arXiv | 2024 | NL2GQL | [Link](https://arxiv.org/abs/2412.10434) 258 | | 6 | CypherBench: Towards Precise Retrieval over Full-scale Modern Knowledge Graphs in the LLM Era | arXiv | 2024 | NL2GQL | [Link](https://arxiv.org/abs/2412.18702) 259 | | 7 | SpCQL: A Semantic Parsing Dataset for Converting Natural Language into Cypher | CIKM | 2022 | NL2GQL | [Link](https://dl.acm.org/doi/10.1145/3511808.3557703) 260 | | 8 | Robust Text-to-Cypher Using Combination of BERT, GraphSAGE, and Transformer (CoBGT) Model |Applied Sciences | 2024 | NL2GQL | [Link](https://doi.org/10.3390/app14177881) 261 | | 9 | Real-Time Text-to-Cypher Query Generation with Large Language Models for Graph Databases | Future Internet | 2024 | NL2GQL | [Link](https://doi.org/10.3390/fi16120438) 262 | | 10 | LLM4QA: Leveraging Large Language Model for Efficient Knowledge Graph Reasoning with SPARQL Query | JAIT | 2024 | NL2GQL | [Link](https://doi.org/10.12720/jait.15.10.1157-1162) 263 | | 11 | Text to Graph Query Using Filter Condition Attributes | LSGDA@VLDB | 2024 | NL2GQL | [Link](https://vldb.org/workshops/2024/proceedings/LSGDA/LSGDA24.09.pdf) 264 | | 12 | Text-to-CQL Based on Large Language Model and Graph Pattern Enhancement | PRML | 2024 | NL2GQL | [Link](https://ieeexplore.ieee.org/document/10779814) 265 | | 13 | Demystifying Natural Language to Cypher Conversion with OpenAI, Neo4j, LangChain, and LangSmith | Blog | 2024 | NL2GQL | [Link](https://medium.com/@muthoju.pavan/demystifying-natural-language-to-cypher-conversion-with-openai-neo4j-langchain-and-langsmith-2dbecb1e2ce9/) 266 | | 14 | Text2Cypher, the beginning of the Graph + LLM stack | Blog | 2023 | NL2GQL | [Link](https://siwei.io/en/llm-text-to-nebulagraph-query/) 267 | | 15 | Text2Cypher - Natural Language Queries | Blog | 2023 | NL2GQL | [Link](https://neo4j.com/labs/neodash/2.4/user-guide/extensions/natural-language-queries/) 268 | | 16 | A Framework for Question Answering on Knowledge Graphs Using Large Language Models | ESWC | 2024 | NL2GQL | [Link](https://link.springer.com/chapter/10.1007/978-3-031-78952-6_20/) 269 | | 17 | LLaSA: Large Language and Structured Data Assistant | arXiv | 2024 | Structured Data Assistant | [Link](https://arxiv.org/abs/2411.14460) 270 | | 18 | GraphRAG and role of Graph Databases in Advancing AI | IJRCAIT | 2024 | Graph DB | [Link](https://doi.org/10.5281/zenodo.13908615) 271 | | 19 | TigerVector: Supporting Vector Search in Graph Databases for Advanced RAGs | arXiv | 2025 | Graph DB | [Link](https://arxiv.org/abs/2501.11216) 272 | | 20 | Increasing Accuracy of LLM-powered Question Answering on SQL databases: Knowledge Graphs to the Rescue | Data Engineering Bulletin | 2024 | RDB QA | [Link](http://sites.computer.org/debull/A24dec/p109.pdf) 273 | | 21 | Symphony: Towards Trustworthy Question Answering and Verification using RAG over Multimodal Data Lakes | Data Engineering Bulletin | 2024 | RDB QA | [Link](http://sites.computer.org/debull/A24dec/p135.pdf) 274 | | 22 | Increasing the LLM Accuracy for Question Answering: Ontologies to the Rescue! | arXiv | 2024 | RDB QA | [Link](https://arxiv.org/abs/2405.11706) 275 | | 23 | GTR: Graph-Table-RAG for Cross-Table Question Answering | arXiv | 2025 | RDB QA | [Link](https://arxiv.org/abs/2504.01346) 276 | | 24 | ER-RAG: Enhance RAG with ER-Based Unified Modeling of Heterogeneous Data Sources | arXiv | 2025 | RDB QA | [Link](https://arxiv.org/abs/2504.06271) 277 | 278 | 279 | ## 4. Benchmark and Applications 280 | 281 | ### Benchmark Dataset 282 | 283 | | NO | Title | Venue | Year | Dataset | Category | Paper Link | 284 | |----|--------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|------|------------------------------------------------|------------------------------------------------|------------------------------------------------------------------------------------------------| 285 | | 1 | The Value of Semantic Parse Labeling for Knowledge Base Question Answering | ACL | 2016 | [WebQSP](https://www.microsoft.com/en-us/download/details.aspx?id=52763) | KBQA and KGQA| [Link](https://aclanthology.org/P16-2033/) 286 | | 2 | Benchmarking Large Language Models in Complex Question Answering Attribution using Knowledge Graphs | arXiv | 2024 | CAQA | KBQA and KGQA| [Link](https://arxiv.org/abs/2401.14640/) 287 | | 3 | G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering | NeurIPS | 2024 | [GraphQA](https://github.com/XiaoxinHe/G-Retriever) | KBQA and KGQA| [Link](https://openreview.net/forum?id=MPJ3oXtTZl) 288 | | 4 | Automatic Question-Answer Generation for Long-Tail Knowledge | KnowledgeNL@KDD | 2023 | Long-tail QA | KBQA and KGQA| [Link](https://knowledge-nlp.github.io/kdd2023/papers/Kumar5.pdf) 289 | | 5 | BioASQ-QA: A manually curated corpus for Biomedical Question Answering | Scientific Data |2023 | [BioASQ-QA](https://zenodo.org/records/7655130)| KBQA and KGQA| [Link](https://pubmed.ncbi.nlm.nih.gov/36973320/) 290 | | 6 | HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering | EMNLP | 2018 | [HotpotQA](https://github.com/hotpotqa/hotpot)| KBQA and KGQA| [Link](https://aclanthology.org/D18-1259) 291 | | 7 | CR-LT-KGQA: A Knowledge Graph Question Answering Dataset Requiring Commonsense Reasoning and Long-Tail Knowledge | arXiv | 2024 | [CR-LT-KGQA](https://github.com/D3Mlab/cr-lt-kgqa)| KBQA and KGQA| [Link](https://arxiv.org/abs/2403.01395) 292 | | 8 | CPAT-Questions: A Self-Updating Benchmark for Present-Anchored Temporal Question-Answering | ACL Findings | 2024 | [TemporalQA](https://github.com/D3Mlab/cr-lt-kgqa) | KBQA and KGQA| [Link](https://arxiv.org/abs/2403.01395) 293 | | 9 | SituatedQA: Incorporating Extra-Linguistic Contexts into QA | EMNLP | 2024 | [SituatedQA](https://github.com/mikejqzhang/SituatedQA)| Open-retrieval QA | [Link](https://aclanthology.org/2021.emnlp-main.586/) 294 | | 10 | CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge | NAACL | 2024 | [CommonsenseQA](https://github.com/jonathanherzig/commonsenseqa)| Multiple-choice QA| [Link](https://aclanthology.org/N19-1421) 295 | | 11 | FanOutQA: A Multi-Hop, Multi-Document Question Answering Benchmark for Large Language Models | ACL | 2024 | [FanOutQA](https://github.com/zhudotexe/fanoutqa)| Multi-hop QA| [Link](https://aclanthology.org/2024.acl-short.2) 296 | | 12 | MINTQA: A Multi-Hop Question Answering Benchmark for Evaluating LLMs on New and Tail Knowledge | arXiv | 2024 | [MINTQA](https://github.com/probe2/multi-hop/)| Multi-hop QA| [Link](https://arxiv.org/abs/2412.17032) 297 | | 13 | What Disease Does This Patient Have? A Large-Scale Open Domain Question Answering Dataset from Medical Exams | Applied Sciences | 2021 | [MedQA](https://github.com/jind11/MedQA)| Multiple-choice QA | [Link](https://www.mdpi.com/2076-3417/11/14/6421) 298 | | 14 | PAT-Questions: A Self-Updating Benchmark for Present-Anchored Temporal Question-Answering | ACL Findings | 2024 | [PAQA](https://github.com/jannatmeem95/PAT-Questions)| Temporal QA| [Link](https://aclanthology.org/2024.findings-acl.777) 299 | | 15 | MenatQA: A New Dataset for Testing the Temporal Comprehension and Reasoning Abilities of Large Language Models | ACL Findings | 2023 | [MenatQA](https://github.com/weiyifan1023/MenatQA)| Temporal QA| [Link](https://aclanthology.org/2023.findings-emnlp.100) 300 | | 16 | TempTabQA: Temporal Question Answering for Semi-Structured Tables| EMNLP | 2023 | [TempTabQA](https://github.com/temptabqa/temptabqa)| Temporal QA| [Link](https://aclanthology.org/2023.emnlp-main.149) 301 | | 17 | Complex Temporal Question Answering on Knowledge Graphs| CIKM | 2021 | [EXAQT](https://exaqt.mpi-inf.mpg.de/)| Temporal QA| [Link](https://dl.acm.org/doi/10.1145/3459637.3482416) 302 | | 18 | Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA| EMNLP | 2024 | [Loong](https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/Loong)| Multi-doc QA| [Link](https://aclanthology.org/2024.emnlp-main.322) 303 | | 19 | MRAMG-Bench: A BeyondText Benchmark for Multimodal Retrieval-Augmented Multimodal Generation | arXiv | 2025 | [MRAMG](https://huggingface.co/MRAMG)| Multi-modal QA| [Link](https://arxiv.org/abs/2502.04176) 304 | | 20 | OMG-QA: Building Open-Domain Multi-Modal Generative Question Answering Systems | EMNLP | 2024 | [OMG-QA](https://github.com/linyongnan/OMG-QA)| Multi-domain Multilingual QA| [Link](https://aclanthology.org/2024.findings-emnlp.365) 305 | | 21 | WebFAQ: A Multilingual Collection of Natural Q&A Datasets for Dense Retrieval | arXiv | 2025 | [WebFAQ](https://github.com/padas-lab-de/webfaq)| Multi-domain Multilingual QA| [Link](https://arxiv.org/abs/2502.20936) 306 | | 22 | M2QA: Multi-domain Multilingual Question Answering | EMNLP | 2024 | [M2QA](https://github.com/UKPLab/m2qa)| Multi-modal QA| [Link](https://aclanthology.org/2024.emnlp-industry.75) 307 | | 23 | M3SciQA: A Multi-Modal Multi-Document Scientific QA Benchmark for Evaluating Foundation Models | ACL Findings | 2024 | [M3SciQA](https://github.com/yale-nlp/M3SciQA)| Multi-modal QA| [Link](https://aclanthology.org/2024.findings-emnlp.904) 308 | | 24 | A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model's Accuracy for Question Answering on Enterprise SQL Databases | GRADES-NDA | 2024 | [ChatData](https://github.com/datadotworld/cwd-benchmark-data) | LLM and KGs for QA| [Link](https://dl.acm.org/doi/10.1145/3661304.3661901) 309 | | 25 | XplainLLM: A Knowledge-Augmented Dataset for Reliable Grounded Explanations in LLMs | EMNLP | 2024 | [XplainLLM](https://github.com/chen-zichen/XplainLLM_dataset) | LLM and KGs for QA| [Link](https://arxiv.org/abs/2311.08614) 310 | | 26 | Developing a Scalable Benchmark for Assessing Large Language Models in Knowledge Graph Engineering | SEMANTICS | 2023 | [LLM-KG-Bench](https://github.com/AKSW/LLM-KG-Bench)| LLM and KGs for QA| [Link](https://ceur-ws.org/Vol-3526/paper-04.pdf) 311 | | 27 | Docugami Knowledge Graph Retrieval Augmented Generation (KG-RAG) Datasets | - | 2023 | [KG-RAG](https://github.com/docugami/KG-RAG-datasets)| LLM and KGs for QA| - 312 | | 28 | How Credible Is an Answer From Retrieval-Augmented LLMs? Investigation and Evaluation With Multi-Hop QA | ACL ARR | 2024 |- | LLM and KGs for QA| [Link](https://openreview.net/forum?id=YsmnPHBbx1f) 313 | | 29 | Can Knowledge Graphs Make Large Language Models More Trustworthy? An Empirical Study over Open-ended Question Answering | arXiv | 2024 |[OKGQA](https://anonymous.4open.science/r/OKGQA-CBB0) | LLM and KGs for QA| [Link](https://arxiv.org/abs/2410.08085) 314 | | 30 | MiniRAG: Towards Extremely Simple Retrieval-Augmented Generation | arXiv | 2025 | [LiHua-World](https://github.com/HKUDS/MiniRAG/tree/main/dataset/LiHua-World)| LLM and KGs for QA| [Link](https://arxiv.org/abs/2501.06713) 315 | | 31 | Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering | NeurIPS Dataset and Benchmarks Track | 2022 | [ScienceQA](https://github.com/lupantech/ScienceQA) | LLM and KGs for QA| [Link](https://proceedings.neurips.cc/paper_files/paper/2022/file/11332b6b6cf4485b84afadb1352d3a9a-Paper-Conference.pdf) 316 | | 32 | STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Bases | NeurIPS Dataset and Benchmarks Track | 2024 | [STaRK](https://github.com/snap-stanford/STaRK) | LLM and KGs for QA| [Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/e607b1419e9ae7cd5cb5b5bb60c2ad5c-Abstract-Datasets_and_Benchmarks_Track.html) 317 | | 33 | mmRAG: A Modular Benchmark for Retrieval-Augmented Generation over Text, Tables, and Knowledge Graphs | arXiv | 2025 | [mmRAG](https://huggingface.co/datasets/Askio/mmrag_benchmark) | LLM and KGs for QA| [Link](https://arxiv.org/abs/2505.11180) 318 | | 34 | LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs -- No Silver Bullet for LC or RAG Routing | arXiv | 2025 | [LaRA](https://github.com/Alibaba-NLP/LaRA) | LLM and KGs for QA| [Link](https://arxiv.org/abs/2502.09977) 319 | | 35 | KGQAGen: Diagnosing and Addressing Pitfalls in KG-RAG Datasets, toward More Reliable Benchmarking | - | 2025 | [KGQAGen](https://github.com/liangliang6v6/KGQAGen) | LLM and KGs for QA| - 320 | 321 | ### Industrial and Scientific Applications 322 | 323 | | NO | Title | Venue | Year | Github | Category | Paper Link | 324 | |----|--------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|------|------------------------------------------------|------------------------------------------------|------------------------------------------------------------------------------------------------| 325 | | 1 | KAG: Boosting LLMs in Professional Domains via Knowledge Augmented Generation | arXiv | 2024 | [KAG](https://github.com/OpenSPG/KAG)| LLM and KGs for QA| [Link](https://arxiv.org/abs/2409.13731) 326 | | 2 | Fact Finder -- Enhancing Domain Expertise of Large Language Models by Incorporating Knowledge Graphs | arXiv | 2024 | [Fact Finder](https://github.com/chrschy/fact-finder/)| LLM and KGs for QA| [Link](https://arxiv.org/abs/2408.03010) 327 | | 3 | Leveraging Large Language Models and Knowledge Graphs for Advanced Biomedical Question Answering Systems | CSA 2024 | 2024 | [Cypher Translator](https://github.com/phdResearcherDz/CypherTranslator/)| LLM and KGs for QA| [Link](https://link.springer.com/chapter/10.1007/978-3-031-71848-9_31) 328 | | 4 | A Prompt Engineering Approach and a Knowledge Graph based Framework for Tackling Legal Implications of Large Language Model Answers | arXiv | 2024 | - | LLM and KGs for QA| [Link](https://link.springer.com/chapter/10.1007/978-3-031-71848-9_31) 329 | | 5 | Ontology-Aware RAG for Improved Question-Answering in Cybersecurity Education | arXiv | 2024 | - | LLM and KGs for QA| [Link](https://arxiv.org/abs/2412.14191) 330 | | 6 |Knowledge Graphs as a source of trust for LLM-powered enterprise question answering | Journal of Web Semantics | 2025 | - | LLM and KGs for QA| [Link](https://doi.org/10.1016/j.websem.2024.100858) 331 | | 7 |MedRAG: Enhancing Retrieval-augmented Generation with Knowledge Graph-Elicited Reasoning for Healthcare Copilot |WWW| 2025 | [MedRAG](https://github.com/SNOWTEAM2023/MedRAG)| LLM and KGs for QA| [Link](https://openreview.net/pdf/7d3d9ad2d616ceae8c5b77eb94019086b980ceda.pdf) 332 | | 8 |EICopilot: Search and Explore Enterprise Information over Large-scale Knowledge Graphs with LLM-driven Agents |arXiv| 2025 | - | LLM and KGs for QA| [Link](https://arxiv.org/abs/2501.13746) 333 | | 9 |Nanjing Yunjin intelligent question-answering system based on knowledge graphs and retrieval augmented generation technology |Heritage Science| 2024 | - | LLM and KGs for QA| [Link](https://www.nature.com/articles/s40494-024-01231-3) 334 | | 10 |A Joint LLM-KG System for Disease Q&A |IEEE JBHI| 2025 | - | LLM and KGs for QA| [Link](https://ieeexplore.ieee.org/abstract/document/10787401) 335 | 336 | ### Demo 337 | 338 | | NO | Name | Description |Source |Github | 339 | |----|------|----------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------|------| 340 | | 1 | GraphRAG-QA | An industrial demo of GraphRAG integrating several query engine for augmenting QA, NLP2Cypher-based KG query engine, vector RAG query engine, and Graph vector RAG query engine. |NebulaGraph| [GraphRAG-QA](https://github.com/wey-gu/demo-kg-build) 341 | | 2 | Neo4jRAG-QA | This sample application demonstrates how to implement a Large Language Model (LLM) and Retrieval Augmented Generation (RAG) system with a Neo4j Graph Database. | Neo4j Graph | [Neo4j Graph RAG](https://github.com/neo4j-examples/rag-demo) 342 | | 3 | BioGraphRAG | This a platform to integrate biomedical knowledge graphs stored in NebulaGraph with LLMs via GraphRAG architecture. | | [BioGraphRAG](https://github.com/devingupta1/BioGraphRAG) 343 | | 4 | kotaemon |An open-source clean & customizable RAG UI for chatting with your documents. Built with both end users and developers in mind. | Cinnamon AI | [kotaemon](https://github.com/Cinnamon/kotaemon) 344 | | 5 | PIKE-RAG |A secIalized KnowledgE and Rationale Augmented Generation, which focuses on extracting, understanding, and applying domain-specific knowledge to gradually guide LLMs toward accurate responses. | Microsoft | [PIKE-RAG](https://github.com/microsoft/PIKE-RAG) 345 | | 6 | AprèsCoT |AprèsCoT: Explaining LLM Answers with Knowledge Graphs and Chain of Thought. | [EDBT25 Demo](https://openproceedings.org/2025/conf/edbt/paper-337.pdf) | [AprèsCoT](http://lg-research-2.uwaterloo.ca:8050/aprescot) 346 | 347 | ## 5. Related Survey 348 | 349 | | NO | Title | Venue | Year | Paper Link | 350 | |----|------|----------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------|------| 351 | | 1 | Unifying Large Language Models and Knowledge Graphs: A Roadmap | TKDE | 2024 | [Link](https://doi.org/10.1109/TKDE.2024.3352100) 352 | | 2 | Graph Retrieval-Augmented Generation: A Survey | arXiv | 2024 | [Link](https://arxiv.org/abs/2408.08921) 353 | | 3 | Retrieval-Augmented Generation with Graphs (GraphRAG) | arXiv | 2024 | [Link](https://arxiv.org/abs/2501.00309) 354 | | 4 | Multilingual Question Answering Systems for Knowledge Graphs—A Survey | Semantic Web | 2024 | [Link](https://journals.sagepub.com/doi/full/10.3233/SW-243633) 355 | | 5 | Temporal Knowledge Graph Question Answering: A Survey | arXiv | 2024 | [Link](https://arxiv.org/abs/2406.14191) 356 | | 6 | Knowledge Graph and Large Language Model Co-learning via Structure-oriented Retrieval Augmented Generation | Data Engineering Bulletin | 2024 | [Link](http://sites.computer.org/debull/A24dec/p9.pdf) 357 | | 7 | Research Trends for the Interplay between Large Language Models and Knowledge Graphs | LLM+KG@VLDB2024| 2024 | [Link](https://vldb.org/workshops/2024/proceedings/LLM+KG/LLM+KG-9.pdf) 358 | | 8 | Neural-Symbolic Reasoning over Knowledge Graphs: A Survey from a Query Perspective | arXiv| 2024 | [Link](https://arxiv.org/abs/2412.10390) 359 | | 9 | Large Language Models, Knowledge Graphs and Search Engines: A Crossroads for Answering Users' Questions | arXiv | 2025 | [Link](https://arxiv.org/abs/2501.06699) 360 | | 10 | Knowledge Graphs, Large Language Models, and Hallucinations: An NLP Perspective | Journal of Web Semantics | 2025 | [Link](https://doi.org/10.1016/j.websem.2024.100844) 361 | | 11 | A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Models | arXiv | 2025 | [Link](https://arxiv.org/abs/2501.13958) 362 | | 12 | Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG | arXiv | 2025 | [Link](https://arxiv.org/abs/2501.09136) 363 | | 13 | A survey on augmenting knowledge graphs (KGs) with large language models (LLMs): models, evaluation metrics, benchmarks, and challenges |Discover Artificial Intelligence | 2024 | [Link](https://link.springer.com/article/10.1007/s44163-024-00175-8) 364 | | 14 | Unifying Large Language Models and Knowledge Graphs for efficient Regulatory Information Retrieval and Answer Generation |REgNLP Workshop | 2025 | [Link](https://aclanthology.org/2025.regnlp-1.4/) 365 | | 15 | A Comprehensive Survey of Knowledge-Based Vision Question Answering Systems: The Lifecycle of Knowledge in Visual Reasoning Task |arXiv | 2025 | [Link](https://arxiv.org/abs/2504.17547) 366 | | 16 | Knowledge Conflicts for LLMs: A Survey |EMNLP | 2024 | [Link](https://aclanthology.org/2024.emnlp-main.486/) 367 | | 17 | A comprehensive survey on integrating large language models with knowledge-based methods | Knowledge-Based Systems | 2025 | [Link](https://doi.org/10.1016/j.knosys.2025.113503/) 368 | | 18 | Synergizing RAG and Reasoning: A Systematic Review | arXiv | 2025 | [Link](https://arxiv.org/abs/2504.15909/) 369 | | 19 | A Survey of Multimodal Retrieval-Augmented Generation | arXiv | 2025 | [Link](https://arxiv.org/abs/2504.08748/) 370 | | 20 | Retrieval-Augmented Generation with Knowledge Graphs: A Survey | OpenReview | 2025 | [Link](https://openreview.net/forum?id=ZikTuGY28C/) 371 | | 21 | Complex QA and language models hybrid architectures, Survey | arXiv| 2023 | [Link](https://arxiv.org/abs/2302.09051) 372 | | 22 | Injecting Domain-Specific Knowledge into Large Language Models: A Comprehensive Survey |arXiv | 2025 | [Link](https://arxiv.org/abs/2502.10708) 373 | -------------------------------------------------------------------------------- /_config.yaml: -------------------------------------------------------------------------------- 1 | title: null 2 | name: "Unifying LLMs and KGs for QA: Recent Advances and Opportunities" 3 | author: " Chuangtao Ma, Yongrui Chen, Tianxing Wu, Arijit Khan, Haofen Wang" 4 | theme: minima 5 | header_pages: 6 | - tutorial-edbt25/index.md 7 | exclude: 8 | - README.md 9 | description: "EDBT25 LLM-KG4QA Tutorial" 10 | social: 11 | image: https://machuangtao.github.io/images/edbt25.jpg -------------------------------------------------------------------------------- /tutorial-edbt25/index.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: page 3 | description: "EDBT 2025 tutorial on integrating LLMs with Knowledge Graphs for QA." 4 | image: https://machuangtao.github.io/images/edbt25.jpg 5 | --- 6 | 7 | Welcome to our EDBT 2025 tutorial! 8 |

9 | EDBT2025 10 |

11 | 12 | ## Tutorial: Unifying Large Language Models and Knowledge Graphs for Question Answering: Recent Advances and Opportunities 13 | 14 | ## Overview 15 | 16 | - **Speakers:** [Chuangtao Ma](https://machuangtao.github.io/), [Yongrui Chen](https://bahuia.github.io), [Tianxing Wu](https://tianxing-wu.github.io), [Arijit Khan](https://homes.cs.aau.dk/~Arijit), [Haofen Wang](https://tongji-kgllm.github.io/people/wang-haofen) 17 | - **Venue:** [EDBT 2025 Tutorials](https://edbticdt2025.upc.edu/?contents=accepted-papers-tutorials.html) 18 | - **Date and Time:** Thursday, March 27, 2025 10:30am-12:30pm 19 | - **Location:** Room: **B4002**, UPC Campus Nord, Barcelona, Spain 20 | 21 | ## Abstract 22 | 23 | Large language models (LLMs) have demonstrated remarkable performance on several question-answering (QA) tasks because of their superior capabilities in natural language understanding and generation. On the other hand, due to poor reasoning capacity, outdated or lack of domain knowledge, expensive retraining costs, and limited context lengths of LLMs, LLM-based QA methods struggle with complex QA tasks such as multi-hop QAs and long-context QAs. Knowledge graphs (KGs) store graphbased structured knowledge which are effective for reasoning and interpretability since KGs accumulate and convey explicit relationships-based factual and domain-specific knowledge from the real world. To address the challenges and limitations of LLMbased QA, several research works that unify LLMs+KGs for QA have been proposed recently. 24 | 25 | This tutorial aims to furnish an overview of the state-of-the-art advances in unifying LLMs with KGs for QA, by categorizing them into three groups according to the roles of KGs when unifying with LLMs. The metrics and benchmarking datasets for evaluating the methods of LLMs+KGs for QA are presented, and domain-specific industry applications and demonstrations will be showcased. The open challenges are summarized and the opportunities for data management are highlighted. 26 | 27 | ## Speakers 28 | 29 | ### [Chuangtao Ma](https://machuangtao.github.io) 30 | 31 | Chuangtao Ma 32 | 33 | Chuangtao Ma is a postdoctoral researcher at Aalborg University, Denmark. His research focuses on knowledge graphs, knowledge-augmented models, and their applications in data management. He is a member of the management committee of the COST action on the Global Network on Large-Scale, Cross-domain, and Multilingual Open Knowledge Graphs. 34 | 35 | 36 | 37 | ### [Yongrui Chen](https://bahuia.github.io) 38 | 39 | Yongrui Chen 40 | 41 | Yongrui Chen is a postdoctoral researcher at Southeast University, China. He specializes in incorporating structured and semi-structured knowledge into foundational LLMs, to improve their complex knowledge reasoning capability. He has presented numerous papers at prominent venues, including NeurIPS, TKDE, IJCAI, AAAI, ACL, ISWC, and NAACL. 42 | 43 | 44 | ### [Tianxing Wu](https://tianxing-wu.github.io) 45 | 46 | Tianxing Wu 47 | 48 | Tianxing Wu is is an associate professor at Southeast University, China. He is one of the main contributors to build Chinese large-scale encyclopedic knowledge graph: Zhishi.me and schema knowledge graph: Linked Open Schema. He has published over 60 papers in top-tier venues, e.g., ICDE, SIGIR, ACL, AAAI, IJCAI, ECAI, ISWC, TKDE, TKDD, JWS, and WWWJ. 49 | 50 | 51 | ### [Arijit Khan](https://homes.cs.aau.dk/~Arijit) 52 | 53 | Arijit Khan 54 | 55 | Arijit Khan is an IEEE senior member, an ACM distinguished speaker, and an associate professor at Aalborg University, Denmark. He published over 90 papers in premier data management and mining venues including ACM SIGMOD, VLDB, TKDE, ICDE, ICLR, SDM, USENIX ATC, EDBT, WWW, WSDM, CIKM, and TKDD. Arijit co-presented tutorials on emerging graph queries, applications, big graph systems, and graph machine learning at VLDB, DSAA, CIKM, and ICDE. 56 | 57 | 58 | ### [Haofen Wang](https://tongji-kgllm.github.io/people/wang-haofen) 59 | 60 | Haofen Wang 61 | 62 | Haofen Wang is a Professor at Tongji University, China. He is one of the initiators of OpenKG, the world's largest alliance for Chinese open knowledge graphs. He published over 100 high-level papers in the AI field, and developed the world's first interactive virtual idol--``Amber Xuyan''. Additionally, the intelligent customer service robots he built have served over 1 billion users. 63 | 64 | 65 | ## Time Schedule 66 | 67 | 68 | | Time | Topic | Presenter | 69 | |------|-------|-----------| 70 | | 10:30 - 10:45 AM | Motivation and Introduction | Arijit Khan | 71 | | 10:45 - 11:10 AM | Unifying LLMs with KGs for QA | Chuangtao Ma | 72 | | 11:10 - 11:35 AM | Advanced Topics on LLM+KG for QA | Yongrui Chen | 73 | | 11:35 - 11:45 AM | Break | | 74 | | 11:45 - 12:05 PM | Evaluations and Applications | Tianxing Wu | 75 | | 12:05 - 12:15 PM | Opportunities for Data Management | Arijit Khan | 76 | | 12:15 - 12:20 PM | Future Directions | Tianxing Wu | 77 | | 12:20 - 12:30 PM | Q&A Session | | 78 | 79 | 80 | ## Materials and Slides 81 | - **Materials**: The covered papers, pointers to opensource codebase, datasets, and demonstrations are available on [GitHub](https://github.com/machuangtao/LLM-KG4QA) for public access. 82 | - **Tutorial Paper**: The tutorial paper is available on [OpenProceedings](https://openproceedings.org/2025/conf/edbt/paper-T4.pdf). 83 | - **Slides**: [PDF Slides](https://machuangtao.github.io/uploads/EDBT25_LLMKGQA-Tutorial.pdf) --------------------------------------------------------------------------------