└── README.md /README.md: -------------------------------------------------------------------------------- 1 | # 基于大语言模型的自动驾驶测试：文献合集 2 | 3 | ## 目录 4 | - [简介](#简介) 5 | - [按模拟目标分类](#按模拟目标分类) 6 | - [1. 视觉信号生成](#1-视觉信号生成) 7 | - [2. 具体场景程序](#2-具体场景程序) 8 | - [3. 轨迹生成](#3-轨迹生成) 9 | - [4. 地图与轨迹](#4-地图与轨迹) 10 | 11 | ## 简介 12 | 本仓库收集了专注于大语言模型(LLM)在自动驾驶测试和模拟中应用的前沿研究论文。这些论文按其模拟目标进行分类，为不同研究方法提供了系统的概览。 13 | 14 | ## 按模拟目标分类 15 | 16 | ### 1. 视觉信号生成 17 | 专注于生成视觉输出（下一帧图像、真实感3D场景、驾驶视频）的论文： 18 | 19 | - **ADriver-I**: "ADriver-I: A General World Model for Autonomous Driving" (Jia et al., 2023) 20 | - **输出类型**：下一帧图像 21 | - **简述**：ADriver-I是一个整合多模态大语言模型(MLLM)和视频扩散模型(VDM)的自动驾驶世界模型 22 | 23 | - **ChatSim**: "ChatSim: Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agents" (Wei et al., 2024) 24 | - **输出类型**：真实感3D驾驶场景 25 | - **简述**：ChatSim通过语言命令实现可编辑的3D驾驶场景模拟 26 | - **GitHub**: [https://github.com/yifanlu0227/ChatSim](https://github.com/yifanlu0227/ChatSim) 27 | 28 | - **DriveDreamer-2**: "DriveDreamer-2: LLM-Enhanced World Models for Driving Video Generation" (Zhao et al., 2024) 29 | - **输出类型**：驾驶视频 30 | - **简述**：DriveDreamer-2生成用户自定义的驾驶视频，用于训练驾驶感知方法 31 | - **项目主页**: [https://drivedreamer2.github.io](https://drivedreamer2.github.io) 32 | 33 | ### 2. 具体场景程序 34 | 专注于生成可执行模拟场景的论文： 35 | 36 | - **Chat2Scenario**: "Chat2Scenario: Scenario Extraction from Dataset through LLM" (Zhao et al., 2024) 37 | - **输出类型**：Esmini和CarMaker场景 38 | - **简述**：Chat2Scenario使用LLMs从数据集中提取驾驶场景，提高了场景提取效率 39 | - **GitHub**: [https://github.com/ftgTUGraz/Chat2Scenario](https://github.com/ftgTUGraz/Chat2Scenario) 40 | 41 | - **ChatScene**: "ChatScene: Safety-Critical Scenario Generation for Autonomous Vehicles" (Zhang et al., 2024) 42 | - **输出类型**：CARLA模拟 43 | - **简述**：ChatScene使用LLMs生成安全关键场景，提高车辆碰撞率15% 44 | - **GitHub**: [https://github.com/javyduck/ChatScene](https://github.com/javyduck/ChatScene) 45 | 46 | - **ChatSUMO**: "ChatSUMO: LLM-based Traffic Simulation Tool" (Li et al., 2024) 47 | - **输出类型**：SUMO模拟 48 | - **简述**：ChatSUMO使用OpenStreetMap数据和LLM输入处理简化交通模拟生成 49 | 50 | - **LEADE**: "LEADE: An LLM-enhanced Multi-objective Evolutionary Search for Autonomous Driving Test Scenario Generation" (Tian et al., 2024) 51 | - **输出类型**：具体场景程序 52 | - **简述**：LEADE为自动驾驶系统(ADS)测试生成多样化的安全关键场景 53 | 54 | - **Natural-language-driven Simulation**: "Natural-language-driven Simulation Benchmark and Copilot for Efficient Production of Object Interactions in Virtual Road Scenes" (Yang et al., 2024) 55 | - **输出类型**：SVL场景 56 | - **简述**：使用包含120,000个虚拟道路场景的L2I基准数据集 57 | 58 | - **Open-TI**: "Open-TI: Open Traffic Intelligence with Augmented Language Model" (Da et al., 2024) 59 | - **输出类型**：SUMO/CityFlow/CBEngine场景 60 | - **简述**：使用LLMs增强交通分析，架起产业界学术界的桥梁并支持模拟 61 | - **GitHub**: [https://github.com/DaRL-LibSignal/OpenTI](https://github.com/DaRL-LibSignal/OpenTI) 62 | 63 | ### 3. 轨迹生成 64 | 专注于生成交通轨迹的论文： 65 | 66 | - **CTG++**: "CTG++: Language-Guided Traffic Simulation via Scene-Level Diffusion" (Zhong et al., 2023) 67 | - **输出类型**：轨迹 68 | - **简述**：CTG++通过场景级扩散模型实现基于用户查询的真实交通模拟 69 | 70 | - **InteractTraj**: "InteractTraj: Language-Driven Interactive Traffic Trajectory Generation" (Xia et al., 2024) 71 | - **输出类型**：轨迹 72 | - **简述**：InteractTraj从自然语言描述生成交互式交通轨迹 73 | - **GitHub**: [https://github.com/X1a-jk/InteractTraj.git](https://github.com/X1a-jk/InteractTraj.git) 74 | 75 | - **ProSim**: "Promptable Closed-loop Traffic Simulation" (Tan et al., 2024) 76 | - **输出类型**：轨迹 77 | - **简述**：ProSim是一个支持复杂用户提示的可提示闭环交通模拟框架 78 | - **GitHub**: [https://ariostgx.github.io/ProSim/](https://ariostgx.github.io/ProSim/) 79 | 80 | - **SeGPT**: "ChatGPT-Based Scenario Engineer: A New Framework on Scenario Generation for Trajectory Prediction" (Li et al., 2024) 81 | - **输出类型**：轨迹 82 | - **简述**：SeGPT利用ChatGPT生成多样化和复杂的驾驶场景用于轨迹预测 83 | 84 | - **Controllable Traffic Simulation**: "Controllable Traffic Simulation through LLM-Guided Hierarchical Chain-of-Thought Reasoning" (Liu et al., 2024) 85 | - **输出类型**：轨迹 86 | - **简述**：提出使用LLM引导的层次化思维链推理的交通模拟新框架 87 | 88 | ### 4. 地图与轨迹 89 | 专注于同时生成地图和轨迹的论文： 90 | 91 | - **LCTGen**: "LCTGen: Language Conditioned Traffic Generation" (Tan et al., 2023) 92 | - **输出类型**：地图和轨迹 93 | - **简述**：LCTGen使用语言条件模型生成真实的交通场景 94 | - **项目主页**: [https://ariostgx.github.io/lctgen](https://ariostgx.github.io/lctgen) 95 | 96 | - **LLMScenario**: "LLMScenario: Large Language Model Driven Scenario Generation" (Chang et al., 2024) 97 | - **输出类型**：地图和轨迹 98 | - **简述**：提出使用LLMs进行场景生成的LLMScenario，解决罕见角落场景问题 99 | 100 | - **OmniTester**: "Multimodal Large Language Model Driven Scenario Testing for Autonomous Vehicles" (Lu et al., 2024) 101 | - **输出类型**：地图和轨迹 102 | - **简述**：OmniTester生成多样化的测试场景，增强可控性和真实性 103 | 104 | 105 | # LLM-Based Autonomous Driving Testing: A Literature Collection 106 | 107 | ## Table of Contents 108 | - [Introduction](#introduction) 109 | - [Categories by Simulation Objective](#categories-by-simulation-objective) 110 | - [1. Visual Signal Generation](#1-visual-signal-generation) 111 | - [2. Concrete Scenario Program](#2-concrete-scenario-program) 112 | - [3. Trajectory Generation](#3-trajectory-generation) 113 | - [4. Map and Trajectories](#4-map-and-trajectories) 114 | - [Contributing](#contributing) 115 | 116 | ## Introduction 117 | This repository serves as a curated collection of cutting-edge research papers focusing on Large Language Model (LLM) applications in autonomous driving testing and simulation. The papers are categorized by their simulation objectives, providing a structured view of different approaches in the field. 118 | 119 | ## Categories by Simulation Objective 120 | 121 | ### 1. Visual Signal Generation 122 | Papers focusing on generating visual outputs (next frame photos, photo-realistic 3D scenes, driving videos): 123 | 124 | - **ADriver-I**: "ADriver-I: A General World Model for Autonomous Driving" (Jia et al., 2023) 125 | - **Output**: Next frame photo 126 | - **TL;DR**: ADriver-I is a world model integrating multimodal large language models (MLLM) and video diffusion models (VDM) for autonomous driving 127 | 128 | - **ChatSim**: "ChatSim: Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agents" (Wei et al., 2024) 129 | - **Output**: Photo-realistic 3D driving scene 130 | - **TL;DR**: ChatSim enables editable 3D driving scene simulations through language commands 131 | - **GitHub**: [https://github.com/yifanlu0227/ChatSim](https://github.com/yifanlu0227/ChatSim) 132 | 133 | - **DriveDreamer-2**: "DriveDreamer-2: LLM-Enhanced World Models for Driving Video Generation" (Zhao et al., 2024) 134 | - **Output**: Driving video 135 | - **TL;DR**: DriveDreamer-2 generates user-customized driving videos for training driving perception methods 136 | - **Project Page**: [https://drivedreamer2.github.io](https://drivedreamer2.github.io) 137 | 138 | ### 2. Concrete Scenario Program 139 | Papers focusing on generating executable simulation scenarios: 140 | 141 | - **Chat2Scenario**: "Chat2Scenario: Scenario Extraction from Dataset through LLM" (Zhao et al., 2024) 142 | - **Output**: Scenarios in Esmini and CarMaker 143 | - **TL;DR**: Chat2Scenario extracts driving scenarios from datasets using LLMs, improving scenario extraction efficiency 144 | - **GitHub**: [https://github.com/ftgTUGraz/Chat2Scenario](https://github.com/ftgTUGraz/Chat2Scenario) 145 | 146 | - **ChatScene**: "ChatScene: Safety-Critical Scenario Generation for Autonomous Vehicles" (Zhang et al., 2024) 147 | - **Output**: CARLA simulations 148 | - **TL;DR**: ChatScene generates safety-critical scenarios using LLMs, improving vehicle collision rate by 15% 149 | - **GitHub**: [https://github.com/javyduck/ChatScene](https://github.com/javyduck/ChatScene) 150 | 151 | - **ChatSUMO**: "ChatSUMO: LLM-based Traffic Simulation Tool" (Li et al., 2024) 152 | - **Output**: SUMO simulations 153 | - **TL;DR**: ChatSUMO simplifies traffic simulation generation using Open-StreetMap data and LLM input processing 154 | 155 | - **LEADE**: "LEADE: An LLM-enhanced Multi-objective Evolutionary Search for Autonomous Driving Test Scenario Generation" (Tian et al., 2024) 156 | - **Output**: Concrete Scenario Program 157 | - **TL;DR**: LEADE generates diverse safety-critical scenarios for Autonomous Driving Systems (ADS) testing 158 | 159 | - **Natural-language-driven Simulation**: "Natural-language-driven Simulation Benchmark and Copilot for Efficient Production of Object Interactions in Virtual Road Scenes" (Yang et al., 2024) 160 | - **Output**: SVL scenarios 161 | - **TL;DR**: Uses the L2I benchmark dataset containing 120,000 virtual road scenes across various topologies 162 | 163 | - **Open-TI**: "Open-TI: Open Traffic Intelligence with Augmented Language Model" (Da et al., 2024) 164 | - **Output**: SUMO/CityFlow/CBEngine scenarios 165 | - **TL;DR**: Enhances traffic analysis using LLMs, bridging industry-academic gaps and supporting simulations 166 | - **GitHub**: [https://github.com/DaRL-LibSignal/OpenTI](https://github.com/DaRL-LibSignal/OpenTI) 167 | 168 | ### 3. Trajectory Generation 169 | Papers focusing on generating traffic trajectories: 170 | 171 | - **CTG++**: "CTG++: Language-Guided Traffic Simulation via Scene-Level Diffusion" (Zhong et al., 2023) 172 | - **Output**: Trajectories 173 | - **TL;DR**: CTG++ enables realistic traffic simulations guided by user queries through a scene-level diffusion model 174 | 175 | - **InteractTraj**: "InteractTraj: Language-Driven Interactive Traffic Trajectory Generation" (Xia et al., 2024) 176 | - **Output**: Trajectories 177 | - **TL;DR**: InteractTraj generates interactive traffic trajectories from natural language descriptions 178 | - **GitHub**: [https://github.com/X1a-jk/InteractTraj.git](https://github.com/X1a-jk/InteractTraj.git) 179 | 180 | - **ProSim**: "Promptable Closed-loop Traffic Simulation" (Tan et al., 2024) 181 | - **Output**: Trajectories 182 | - **TL;DR**: ProSim is a promptable framework for realistic traffic simulation, supporting complex user prompts 183 | - **GitHub**: [https://ariostgx.github.io/ProSim/](https://ariostgx.github.io/ProSim/) 184 | 185 | - **SeGPT**: "ChatGPT-Based Scenario Engineer: A New Framework on Scenario Generation for Trajectory Prediction" (Li et al., 2024) 186 | - **Output**: Trajectories 187 | - **TL;DR**: SeGPT leverages ChatGPT to generate diverse and complex driving scenarios for trajectory prediction 188 | 189 | - **Controllable Traffic Simulation**: "Controllable Traffic Simulation through LLM-Guided Hierarchical Chain-of-Thought Reasoning" (Liu et al., 2024) 190 | - **Output**: Trajectories 191 | - **TL;DR**: Proposes a novel framework for traffic simulation using LLM-guided hierarchical reasoning 192 | 193 | ### 4. Map and Trajectories 194 | Papers focusing on both map and trajectory generation: 195 | 196 | - **LCTGen**: "LCTGen: Language Conditioned Traffic Generation" (Tan et al., 2023) 197 | - **Output**: Map and trajectories 198 | - **TL;DR**: LCTGen generates realistic traffic scenarios using a language-conditioned model 199 | - **Project Page**: [https://ariostgx.github.io/lctgen](https://ariostgx.github.io/lctgen) 200 | 201 | - **LLMScenario**: "LLMScenario: Large Language Model Driven Scenario Generation" (Chang et al., 2024) 202 | - **Output**: Map and trajectories 203 | - **TL;DR**: Proposes LLMScenario for scenario generation using LLMs, addressing rare corner scenarios 204 | 205 | - **OmniTester**: "Multimodal Large Language Model Driven Scenario Testing for Autonomous Vehicles" (Lu et al., 2024) 206 | - **Output**: Map and trajectories 207 | - **TL;DR**: OmniTester generates diverse scenarios for testing, enhancing controllability and realism 208 | 209 | ## Contributing 210 | We welcome contributions to this collection! Please feel free to submit pull requests with: 211 | - New relevant papers 212 | - Updates to existing paper information 213 | - Additional resources or implementations 214 | - Corrections or improvements to descriptions 215 | 216 | --------------------------------------------------------------------------------