├── LICENSE ├── README.md └── assets ├── IDKB_show.mp4 ├── IDKB_show.png ├── data_example.png ├── data_statistic.png └── pipeline.png /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2024 4DVLab 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 |

Can LVLMs Obtain a Driver’s License?
A Benchmark Towards Reliable AGI for Autonomous Driving

🌐 Project Page  |  📄 Arxiv

2 | 3 | 4 | 5 | ## 📑 Table of Contents 6 | 7 | 1. [Overview](#overview) 8 | 2. [Data Overview](#data-overview) 9 | 3. [News](#news) 10 | 4. [Dataset Download](#dataset-download) 11 | 5. [Usage Guide](#usage-guide) 12 | 6. [Citation](#citation) 13 | 14 | 15 | 16 | ## 🚀 Overview 17 | 18 | Large Vision-Language Models (LVLMs) have gained significant attention for their general knowledge and interpretability in autonomous driving. However, they lack the specialized expertise required for professional and safe driving, such as traffic rules and driving skills—critical elements of driving safety. 19 | 20 | To address this, we introduce **IDKB**, a comprehensive dataset with over **1 million driving-related data items** from various countries, including: 21 | 22 | - **Driving handbooks** 23 | - **Theory test questions** 24 | - **Simulated road test data** 25 | 26 | Similar to the process of obtaining a driver's license, **IDKB** encompasses knowledge from theory to practice, making it the most extensive benchmark for evaluating and enhancing LVLMs in autonomous driving. 27 | 28 | Our benchmark includes **15 popular LVLMs** evaluated for their reliability in autonomous driving scenarios, alongside fine-tuning experiments that significantly improved model performance. 29 | 30 | 31 | 32 | ## 🗂️ Data Overview 33 | 34 |

35 | 36 |

37 | 38 | 39 | 40 | ## 🚩 News 41 | 42 | + **[2024-02-24]** We have refined the dataset format and provided the Google Drive link. 43 | + **[2024-12-13]** We have provided a download link via Baidu Netdisk. 44 | + **[2024-12-10]** Our work, IDKB, has been accepted by AAAI 2025. 45 | + **[2024-09-04]** The ArXiv version of our paper has been released. 46 | 47 | 48 | 49 | # 📚 Dataset Download 50 | 51 | | Platform | Link | Access Code | 52 | | ------------- | ------------------------------------------------------------ | ----------- | 53 | | Baidu Netdisk | [Download Here](https://pan.baidu.com/s/1YukK_F6OAaz3lH_m27w9bg) | idkb | 54 | | Google Drive | [Download Here](https://drive.google.com/file/d/12XRP8-BeqA6asNRwEsH-OV7ndSZ2Bwsy/view?usp=sharing) | - | 55 | 56 | 57 | 58 | ## 🛠️ Usage Guide 59 | 60 | **TODO:** The usage instructions for IDKB will be provided here, stay tuned! 61 | 62 | 63 | 64 | ## 📖 Citation 65 | 66 | If you use our dataset or benchmark in your research, please cite us as: 67 | 68 | ``` 69 | @inproceedings{lu2025can, 70 | title={Can lvlms obtain a driver’s license? a benchmark towards reliable agi for autonomous driving}, 71 | author={Lu, Yuhang and Yao, Yichen and Tu, Jiadong and Shao, Jiangnan and Ma, Yuexin and Zhu, Xinge}, 72 | booktitle={Proceedings of the AAAI Conference on Artificial Intelligence}, 73 | volume={39}, 74 | number={6}, 75 | pages={5838--5846}, 76 | year={2025} 77 | } 78 | ``` 79 | 80 | -------------------------------------------------------------------------------- /assets/IDKB_show.mp4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/4DVLab/IDKB/016a671c6b3e8a102930b8f72b5b212023509ccc/assets/IDKB_show.mp4 -------------------------------------------------------------------------------- /assets/IDKB_show.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/4DVLab/IDKB/016a671c6b3e8a102930b8f72b5b212023509ccc/assets/IDKB_show.png -------------------------------------------------------------------------------- /assets/data_example.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/4DVLab/IDKB/016a671c6b3e8a102930b8f72b5b212023509ccc/assets/data_example.png -------------------------------------------------------------------------------- /assets/data_statistic.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/4DVLab/IDKB/016a671c6b3e8a102930b8f72b5b212023509ccc/assets/data_statistic.png -------------------------------------------------------------------------------- /assets/pipeline.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/4DVLab/IDKB/016a671c6b3e8a102930b8f72b5b212023509ccc/assets/pipeline.png --------------------------------------------------------------------------------