├── README.md
├── all_task.json
├── asset
    ├── domain_123.jpeg
    ├── domain_45.jpeg
    ├── domain_45.png
    ├── key_domains.png
    ├── results_tra.png
    ├── results_vlms.png
    └── sun.jpeg
├── evaluate_utils.py
├── evaluate_vlm.py
└── qwen2vl_all.py


/README.md:
--------------------------------------------------------------------------------
 1 | # ✨✨VLADBench✨✨: Fine-Grained Evaluation of Large Vision-Language Models in Autonomous Driving 
 2 | 
 3 | <div align="center">
 4 | 
 5 | ![VLADBench](https://img.shields.io/badge/Dataset-VLADBench-blue)
 6 | ![VQA](https://img.shields.io/badge/Task-Autonomous--Driving--QA-red) 
 7 | ![Multi-Modal](https://img.shields.io/badge/Modal-Image/Video-red)  
 8 | ![Models](https://img.shields.io/badge/Models-Open--Source-green) 
 9 | ![Models](https://img.shields.io/badge/Models-Closed--Source-green) 
10 | ![Models](https://img.shields.io/badge/Models-Domain--Specified-green) 
11 | 
12 | </div>
13 | <div align="center" style="font-size: 22px; margin-top: 20px; margin-bottom: 20px; line-height: 2.5;">
14 |   <a href="https://arxiv.org/pdf/2503.21505">📖 arXiv Paper</a> &nbsp;&nbsp;&nbsp;&nbsp;
15 |   <a href="https://huggingface.co/datasets/depth2world/VLADBench">🤗 Dataset</a>
16 | </div>
17 | 
18 | 
19 | 
20 | # Overview
21 | 
22 | Current benchmarks designed for VLM-based AD face several notable limitations:
23 | 
24 | - **Coarse-grained Categories**: The underlying datasets of
25 | the VLM-based models are often simplistic, typically categorizing tasks into perception, prediction, and planning
26 | with reasoning, which are incomplete for evaluating the nuanced cognitive and reasoning abilities required for safe and
27 | reliable AD. 
28 | 
29 | - **Lack of Dynamic Elements**: Both static and dynamic scenes are crucial for evaluating AD systems, a robust analysis of dynamic elements is particularly important for validating the temporal reasoning capabilities, especially in understanding traffic participant intentions within the scene and executing the nuanced spatio-temporal reasoning required for safe navigation. 
30 | 
31 | - **Homogeneous Data**:
32 | Existing VLM-based AD datasets often suffer from a lack
33 | of diversity, which limits the ability to test models across
34 | a wide range of real-world scenarios. The narrow results
35 | restrict the evaluation of zero-shot generalization and the
36 | performance on challenging corner cases.
37 | 
38 | We introduce **VLADBench**, specifically designed to rigorously evaluate the capabilities of VLMs in AD. VLADBench employes a hierarchical structure that reflects the complex skill set required for reliable driving, progressing from fundamental scene and traffic elements comprehension to advanced reasoning and decision-making. 
39 | 
40 | - With 2000 static scenes and 3000 dynamic scenarios, VLADBench spans 5 primary domains: Traffic Knowledge Understanding (TKU), General Element Recognition (GER), Traffic Graph Generation (TGG), Target Attribute Comprehension (TAC), and Ego Decision-making and Planning(EDP). For a more detailed assessment, 11 secondary aspects and 29 tertiary tasks are defined, resulting in a total of 12K questions.
41 | -  VLADBench is built from existing publicly available datasets, meticulously curated through a manual selection across 12 sources, aimed at challenging VLM capabilities in diverse challenging driving situations.
42 | - To further investigate the intersections among the 5 key domains, we collect and construct approximately 1.4M AD-specific QAs from public resources. We then categorize these QAs
43 | using GPT-4 and train models on individual domain-specific (DS) datasets. Finally, we validate the trained models on VLADBench to assess their performance across different domains.
44 | 
45 | <p align="center">
46 |     <img src="./asset/sun.jpeg" width="50%" height="50%">
47 | </p>
48 | 
49 | 
50 | 
51 | # Examples
52 | <p align="center">
53 |     <img src="./asset/domain_123.jpeg" width="90%" height="90%">
54 | </p>
55 | <p align="center">
56 |     <img src="./asset/domain_45.jpeg" width="90%" height="90%">
57 | </p>
58 | 
59 | # Evaluation Results
60 | <p align="center">
61 |     <img src="./asset/results_vlms.png" width="100%" height="100%">
62 | </p>
63 | 
64 | For the detailed results of each tasks and the results from the large-scale models, please see the paper.
65 | 
66 | 
67 | # Evaluation Pipeine
68 | 
69 | 1. We provide a test example based on Qwen2-VL.[qwen2vl_all.py]
70 | 2. Run evaluate_vlm.py for scores.
71 | 
72 | **Note**  
73 | **1. The bounding boxes in VLADBench are NOT resized. You should modify them in [prompt](https://github.com/Depth2World/VLADBench/blob/8156375d6b0e88ec5ca4f5a8119a9a0ee6c1ed18/qwen2vl_all.py#L79) and [evaluation](https://github.com/Depth2World/VLADBench/blob/8156375d6b0e88ec5ca4f5a8119a9a0ee6c1ed18/evaluate_utils.py#L170) for different VLMs.**  
74 | **2. The finall scores do NOT include the trajectory evaluation.**
75 | 


--------------------------------------------------------------------------------
/all_task.json:
--------------------------------------------------------------------------------
 1 | {
 2 |     "Traffic_Knowledge_Understanding": {
 3 |         "Road_Traffic_Signals": [
 4 |             "Traffic_Light",
 5 |             "Pavement_Marking",
 6 |             "Traffic_Sign"
 7 |         ],
 8 |         "Road_Passage_Provisions": [
 9 |             "Right_Of_Way"
10 |         ]
11 |     },
12 |     "General_Element_Recognition": {
13 |         "Foreground": [
14 |             "VRU_Recognition",
15 |             "Vehicle_Recognition",
16 |             "Vehicle_Status",
17 |             "Lane_Recognition",
18 |             "Obstruction_Recognition"
19 |         ],
20 |         "Background": [
21 |             "Light",
22 |             "Weather"
23 |         ]
24 |     },
25 |     "Traffic_Graph_Generation": {
26 |         "Signal_Element_Relation": [
27 |             "Sign_Sign_Relation",
28 |             "Sign_Lane_Relation",
29 |             "Light_Lane_Relation"
30 |         ],
31 |         "Lane_Element_Relation": [
32 |             "Lane_Speed_Relation",
33 |             "Lane_Change_Relation"
34 |         ]
35 |     },
36 |     "Target_Attribute_Comprehension": {
37 |         "Intention_Judgment": [
38 |             "VRU_Cutin",
39 |             "Vehicle_Cutin",
40 |             "VRU_Cross",
41 |             "Long_Short_Parking"
42 |         ],
43 |         "Behavior_Understanding": [
44 |             "Vehicle_Bahavior",
45 |             "VRU_Bahavior"
46 |         ]
47 |     },
48 |     "Ego_Decision_Planning": {
49 |         "Ego_Action_Reasoning": [
50 |             "Key_Obsturction_Detection",
51 |             "Spatial_Temporal_Reasoning",
52 |             "Risk_Prediction",
53 |             "Drive_Efficiency"
54 |         ],
55 |         "Meta_Action_Decision": [
56 |             "Longitudinal",
57 |             "Lateral"
58 |         ],
59 |         "Ego_trajectory_Planning":[
60 |             "Trajectory"
61 |         ]
62 |     }
63 | }
64 | 


--------------------------------------------------------------------------------
/asset/domain_123.jpeg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Depth2World/VLADBench/fc9e407b9c859126320e9527d287135b5bb0f368/asset/domain_123.jpeg


--------------------------------------------------------------------------------
/asset/domain_45.jpeg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Depth2World/VLADBench/fc9e407b9c859126320e9527d287135b5bb0f368/asset/domain_45.jpeg


--------------------------------------------------------------------------------
/asset/domain_45.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Depth2World/VLADBench/fc9e407b9c859126320e9527d287135b5bb0f368/asset/domain_45.png


--------------------------------------------------------------------------------
/asset/key_domains.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Depth2World/VLADBench/fc9e407b9c859126320e9527d287135b5bb0f368/asset/key_domains.png


--------------------------------------------------------------------------------
/asset/results_tra.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Depth2World/VLADBench/fc9e407b9c859126320e9527d287135b5bb0f368/asset/results_tra.png


--------------------------------------------------------------------------------
/asset/results_vlms.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Depth2World/VLADBench/fc9e407b9c859126320e9527d287135b5bb0f368/asset/results_vlms.png


--------------------------------------------------------------------------------
/asset/sun.jpeg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Depth2World/VLADBench/fc9e407b9c859126320e9527d287135b5bb0f368/asset/sun.jpeg


--------------------------------------------------------------------------------
/evaluate_utils.py:
--------------------------------------------------------------------------------
  1 | import os
  2 | import json
  3 | import re
  4 | import numpy as np
  5 | 
  6 | 
  7 | 
  8 | def weighted_row_sum(data, third_rows, weight_col=1, start_col=2):
  9 |     
 10 |     data = np.array(data)
 11 |     m,n = data.shape
 12 |     rows = slice(m-third_rows, m)
 13 |     cols = slice(start_col, None)  
 14 |     weighted_sum = np.sum(data[rows, cols].astype(float) * data[rows, weight_col].astype(float)[:, np.newaxis], axis=0) / np.sum(data[rows, weight_col].astype(float))
 15 |     weighted_sum = ['Mean',np.sum(data[rows, weight_col].astype(float))] + weighted_sum.tolist()
 16 |     temp = data.tolist()
 17 |     temp.append(weighted_sum)
 18 |     return temp
 19 | 
 20 | 
 21 | 
 22 | def weighted_total(data, weight_col=1, start_col=2):
 23 |     data = np.array(data)
 24 |     m,n = data.shape
 25 |     rows = slice(0, m)
 26 |     cols = slice(start_col, None)  
 27 |     weighted_sum = np.sum(data[rows, cols].astype(float) * data[rows, weight_col].astype(float)[:, np.newaxis], axis=0) / np.sum(data[rows, weight_col].astype(float))
 28 |     weighted_sum = ['Total',np.sum(data[rows, weight_col].astype(float))] + weighted_sum.tolist()
 29 |     return weighted_sum
 30 | 
 31 | 
 32 | def box_iou(boxA, boxB):
 33 |     boxA = [int(x) for x in boxA]
 34 |     boxB = [int(x) for x in boxB]
 35 | 
 36 |     xA = max(boxA[0], boxB[0])
 37 |     xB = min(boxA[2], boxB[2])
 38 |     yA = max(boxA[1], boxB[1])
 39 |     yB = min(boxA[3], boxB[3])
 40 |  
 41 |     interArea = max(0, xB - xA + 1) * max(0, yB - yA + 1)
 42 |  
 43 |     boxAArea = (boxA[2] - boxA[0] + 1) * (boxA[3] - boxA[1] + 1)
 44 |     boxBArea = (boxB[2] - boxB[0] + 1) * (boxB[3] - boxB[1] + 1)
 45 |     
 46 |     iou = interArea / float(boxAArea + boxBArea - interArea)
 47 |  
 48 |     return iou
 49 | def clean_string(s):
 50 |     while s and (s[0] in ":[]()' ."):
 51 |         s = s[1:]
 52 |     while s and (s[-1] in ":[]()' ."):
 53 |         s = s[:-1]
 54 |     return s
 55 | 
 56 | def convert_if_number(answer):
 57 |     if isinstance(answer, (int, float)):
 58 |         return str(answer)
 59 |     return answer
 60 | 
 61 | def remove_symbols(input_string):
 62 |     if 'correct answer is:' in input_string:
 63 |         input_string = input_string.split('correct answer is:')[-1]
 64 |     cleaned_string = re.sub(r'[\*\n\""]', '', input_string)
 65 |     return cleaned_string
 66 | 
 67 | def extract_options(text):
 68 |     
 69 |     pattern = re.compile(r"\[([^\]]+)\]")  
 70 |     matches = pattern.findall(text)
 71 | 
 72 |     if matches:
 73 |         option_string = matches[-1]  
 74 |         if "'" not in option_string:       
 75 |             option_list = option_string.split(", ")
 76 |         else:
 77 |             option_list = [item.strip().strip("'") for item in option_string.split("', '")]  
 78 |         return option_list
 79 |     return []
 80 | 
 81 | 
 82 | def compare_and_count(array_a, array_b):
 83 |     count = 0
 84 |     for a, b in zip(array_a, array_b):
 85 |         if a == 1 and b == 1: count+=1
 86 |         if a > b:count+=1
 87 |     return count
 88 |      
 89 | def isfile(path):
 90 |     return os.path.isfile(path)
 91 | 
 92 | 
 93 | def load_json_data(path):
 94 |     with open(path,'r',encoding='utf-8') as json_f:
 95 |         task_data = json.load(json_f)
 96 |         return task_data
 97 |     
 98 | def save_json_data(path,data):
 99 |     with open(path,'w',encoding='utf-8') as json_f:
100 |         json.dump(data,json_f,ensure_ascii=False,indent=4)
101 |         
102 | def Geneal_criterion_QA(third_task_data,MODEL=None):
103 |     ques_total_num = 0
104 |     right_num = 0
105 |     obey_insytruction = 0
106 |     for d_ind, sample in enumerate(third_task_data):
107 |         reference = sample['reference']
108 |         prediction = sample['prediction']
109 |         for q_ind, pred in enumerate(prediction):
110 |             # print(sample['image_path'])
111 |             ques_nopath = ''.join(sample['questions'][q_ind].lower().split(';')[1:])
112 |             tips = extract_options(ques_nopath)
113 |             # print(tips)
114 |             if len(tips)==0: pass
115 |             # if len(tips)!=0: print('No tips',sample['image_path'])
116 |                 # print('No tips',sample['image_path'])
117 |                 # print(ques_nopath)
118 |             pred = remove_symbols(pred)   
119 |             ques_total_num += 1
120 |             clean_pred = clean_string(pred).lower()  
121 |             options_nums = clean_pred.split("', '")
122 |             reference_q_ind = convert_if_number(reference[q_ind]).lower()
123 |             if len(options_nums)==1: 
124 |                 if clean_pred in ques_nopath: 
125 |                     obey_insytruction+=1
126 |                 if clean_pred==reference_q_ind:
127 |                     right_num+=1
128 |                 elif reference_q_ind in clean_pred:
129 |                     ### filter
130 |                     if reference_q_ind in tips:
131 |                         tips.remove(reference_q_ind)
132 |                         if not any(tip in clean_pred for tip in tips):
133 |                             right_num+=1
134 |     return ques_total_num,right_num/ques_total_num,obey_insytruction/ques_total_num,0
135 | 
136 | 
137 | def Grounding_criterion_QA(third_task_data,MODEL=None):
138 |     if MODEL ==None:
139 |         print('MODEL Input Lacked')
140 |         return -1 
141 |     resize_model_lists = ["qwen", "internvl", "gemini","DriveMM",'ivl']
142 |     ques_total_num = 0
143 |     right_num = 0
144 |     loc_union = []
145 |     obey_insytruction = 0
146 |     PATTERN = re.compile(r'\[\s*([^\],]*\d+[^\],]*)\s*,\s*([^\],]*\d+[^\],]*)\s*,\s*([^\],]*\d+[^\],]*)\s*,\s*([^\],]*\d+[^\],]*)\s*\]')
147 |     box_num = 0
148 |     for d_ind, sample in enumerate(third_task_data):
149 |         reference = sample['reference']
150 |         prediction = sample['prediction']
151 |         for q_ind, pred in enumerate(prediction):
152 |             # print(sample['image_path'])
153 |             ques_total_num += 1
154 |             ques_nopath = ''.join(sample['questions'][q_ind].lower().split(';')[1:])
155 |             if 'located in the image?' in ques_nopath:
156 |                 matches = PATTERN.findall(pred)
157 |                 cleaned_matches = [[float(re.sub(r'[^0-9.]', '', part)) for part in match] for match in matches]
158 |                 if len(matches)==1:
159 |                     box_num+=1
160 |                     obey_insytruction+=1
161 |                     predict_bbox = cleaned_matches[0]
162 |                 else:
163 |                     predict_bbox = [0.0, 0.0, 0.0, 0.0]
164 |                 
165 |                 if sum(predict_bbox) <4:
166 |                     predict_bbox = [x * 1000 for x in predict_bbox]
167 |                 if any(mn.lower() in MODEL.lower() for mn in resize_model_lists):
168 |                     bbox_gt = sample['reference'][q_ind]
169 |                     width,height = sample['dimension']
170 |                     bbox_gt = [int(1000*bbox_gt[0]/width), int(1000*bbox_gt[1]/height), int(1000*bbox_gt[2]/width), int(1000*bbox_gt[3]/height)]
171 |                 elif MODEL =="gemini":
172 |                     bbox_gt = [bbox_gt[1], bbox_gt[0], bbox_gt[3], bbox_gt[2]]
173 |                 else:
174 |                     bbox_gt = sample['reference'][q_ind]
175 |                     
176 |                 iou = box_iou(predict_bbox, bbox_gt)
177 |                 if iou > 0.5: right_num+=1
178 |                 loc_union.append(iou)
179 |             else:
180 |                 tips = extract_options(ques_nopath)
181 |                 # if len(tips)==0: 
182 |                 #     print('No tips',sample['image_path'])
183 |                 #     print(sample['questions'][q_ind])
184 |                 pred = remove_symbols(pred)  
185 |                 clean_pred = clean_string(pred).lower()  
186 |                 options_nums = clean_pred.split("', '")
187 |                 reference_q_ind = convert_if_number(reference[q_ind]).lower()
188 |                 if len(options_nums)==1: 
189 |                     if clean_pred in ques_nopath: 
190 |                         obey_insytruction+=1
191 |                     if clean_pred==reference_q_ind:
192 |                         right_num+=1
193 |                         
194 |                     elif reference_q_ind in clean_pred:
195 |                         ### filter
196 |                         if reference_q_ind in tips:
197 |                             tips.remove(reference_q_ind)
198 |                             if not any(tip in clean_pred for tip in tips):
199 |                                 right_num+=1
200 |                                 
201 |     mean_iou = sum(loc_union)/len(loc_union)
202 |     return ques_total_num, right_num/ques_total_num, obey_insytruction/ques_total_num, mean_iou
203 | 
204 | 
205 |         
206 | def Relation_criterion_QA(third_task_data,MODEL=None):
207 |     ques_total_num = 0
208 |     total_score = 0
209 |     obey_insytruction = 0
210 |     totol_improve_score = 0
211 |     for d_ind, sample in enumerate(third_task_data):
212 |         reference = sample['reference']
213 |         prediction = sample['prediction']
214 |         scores_list = []
215 |         for q_ind, pred in enumerate(prediction):
216 |             ques_total_num+=1
217 |             if 'corresponds to' in pred:
218 |                 # pattern = r'(?<!\d)(-?\d+|[0-9]+/[0-9]+)(?!\d)'
219 |                 pattern = r'corresponds to No.([+-]?\d+|[+-]?\d+/\d+)'
220 |                 match = re.search(pattern, pred)
221 |                 if match:
222 |                     pred_num = match.group(1).split('/')
223 |                     # print(pred_num)
224 |                 else:
225 |                     pred_num = []
226 |             elif 'corresponding to' in pred:
227 |                 pattern = r"corresponding to.*is\s+(-?\d+(?:/\d+)*)"
228 |                 match = re.search(pattern, pred)
229 |                 if match:
230 |                     pred_num = match.group(1).split("/")
231 |                 else:
232 |                     pred_num = []
233 |             else:
234 |                 pattern = r"(-?\d+(?:/\d+)*)"
235 |                 match = re.findall(pattern, pred)
236 |                 if match:
237 |                     obey_insytruction+=1
238 |                     pred_num = match[-1].split("/")
239 |                     # print(pred_num)
240 |                 else:
241 |                     pred_num = []
242 |             
243 |             ref_num =  reference[q_ind].split('/')
244 |             if any(p_num not in ref_num for p_num in pred_num):
245 |                 scores_list.append(0)
246 |                 continue
247 |             else:
248 |                 temp = 0
249 |                 # for p_num in pred_num:
250 |                 #     if p_num in ref_num:
251 |                 #         total_score += 1
252 |                 #         break
253 |                 for p_num in pred_num:
254 |                     if p_num in ref_num:
255 |                         temp += 1/len(ref_num)
256 |                         total_score += 1/len(ref_num)
257 |                 scores_list.append(temp)
258 |         scores_list = np.array(scores_list)
259 |         scores = compare_and_count(scores_list[len(scores_list)//2:], scores_list[:len(scores_list)//2])
260 |         totol_improve_score += scores
261 |     return ques_total_num,total_score/ques_total_num,obey_insytruction/ques_total_num,totol_improve_score*2/ques_total_num
262 | 
263 | 
264 |     
265 | def RoadChange_criterion_QA(third_task_data,MODEL=None):
266 |     ques_total_num = 0
267 |     right_num = 0
268 |     obey_insytruction = 0
269 |     totol_improve_score = 0
270 |     for d_ind, sample in enumerate(third_task_data):
271 |         reference = sample['reference']
272 |         prediction = sample['prediction']
273 |         scores_list = []
274 |         for q_ind, pred in enumerate(prediction):
275 |             ques_nopath = ''.join(sample['questions'][q_ind].lower().split(';')[1:])
276 |             tips = extract_options(ques_nopath)
277 |             pred = remove_symbols(pred)   
278 |             ques_total_num += 1
279 |             clean_pred = clean_string(pred).lower()  
280 |             options_nums = clean_pred.split("', '")
281 |             reference_q_ind = convert_if_number(reference[q_ind]).lower()
282 |             if len(options_nums)==1: 
283 |                 if clean_pred in ques_nopath: 
284 |                     obey_insytruction+=1
285 |                 if clean_pred==reference_q_ind:
286 |                     right_num+=1
287 |                     scores_list.append(1)
288 |                 elif reference_q_ind in clean_pred:
289 |                     ### filter
290 |                     if reference_q_ind in tips:
291 |                         tips.remove(reference_q_ind)
292 |                         if not any(tip in clean_pred for tip in tips):
293 |                             right_num+=1
294 |                             scores_list.append(1)
295 |                         else:
296 |                             scores_list.append(0)
297 |                     else:
298 |                         scores_list.append(0)
299 |                 else:
300 |                     scores_list.append(0)
301 |         scores_list = np.array(scores_list)
302 |         scores = compare_and_count(scores_list[len(scores_list)//2:], scores_list[:len(scores_list)//2])
303 |         totol_improve_score += scores
304 |                       
305 |     return ques_total_num,right_num/ques_total_num,obey_insytruction/ques_total_num,totol_improve_score*2/ques_total_num
306 | 
307 | def RoadSpeed_criterion_QA(third_task_data,MODEL=None):
308 |     ques_total_num = 0
309 |     right_num = 0
310 |     obey_insytruction = 0
311 |     totol_improve_score = 0
312 |     for d_ind, sample in enumerate(third_task_data):
313 |         reference = sample['reference']
314 |         prediction = sample['prediction']
315 |         scores_list = []
316 |         for q_ind, pred in enumerate(prediction):
317 |             ques_total_num+=1
318 |             pattern = r'\[\s*(-?\d+)\s*,\s*(-?\d+)\s*\]'  
319 |             matches = re.findall(pattern, pred)
320 |             
321 |             matches_gt = re.findall(pattern, reference[q_ind])
322 |             # print(reference[q_ind])
323 |             ref_gt = [matches_gt[0][0],matches_gt[0][1]]
324 |             # print(ref_gt)
325 |             temp = 0
326 |             if len(matches)==1:
327 |                 pred_limit = [matches[0][0],matches[0][1]]
328 |                 obey_insytruction+=1
329 |                 for a, b in zip(ref_gt,pred_limit):
330 |                     if a==b:
331 |                         temp+=0.5
332 |             right_num+=temp
333 |             scores_list.append(temp)
334 |             
335 |         scores_list = np.array(scores_list)
336 |         scores = compare_and_count(scores_list[len(scores_list)//2:], scores_list[:len(scores_list)//2])
337 |         totol_improve_score += scores
338 |         
339 |     return ques_total_num,right_num/ques_total_num,obey_insytruction/ques_total_num,totol_improve_score*2/ques_total_num
340 |     # return ques_total_num,right_num,obey_insytruction,totol_improve_score/2
341 | 
342 | 
343 |       
344 | def Judge_criterion_QA(third_task_data,MODEL=None):
345 |     des_ques_total_num = 0
346 |     judge_ques_total_num = 0
347 |     des_right_num = 0
348 |     judge_right_num = 0
349 |     obey_insytruction = 0
350 |     for d_ind, sample in enumerate(third_task_data):
351 |         reference = sample['reference']
352 |         prediction = sample['prediction']
353 |         for q_ind, pred in enumerate(prediction):
354 |             ques_nopath = ''.join(sample['questions'][q_ind].lower().split(';')[1:])
355 |             tips = extract_options(ques_nopath)
356 |             # print(tips)
357 |             if len(tips)==0: pass
358 |             pred = remove_symbols(pred)   
359 |             clean_pred = clean_string(pred).lower()  
360 |             options_nums = clean_pred.split("', '")
361 |             # reference_q_ind = convert_if_number(reference[q_ind]).lower()
362 |             reference_q_ind = clean_string(convert_if_number(reference[q_ind])).lower()
363 |             if 'yes' == reference_q_ind or 'no'  == reference_q_ind:
364 |                 judge_ques_total_num+=1
365 |             else:
366 |                 des_ques_total_num += 1
367 |             if len(options_nums)==1: 
368 |                 # if clean_pred in ques_nopath: 
369 |                 if ''.join(clean_pred.split(';') in ques_nopath: 
370 |                     obey_insytruction+=1
371 |                 if clean_pred==reference_q_ind:
372 |                     if 'yes' == reference_q_ind or 'no'  == reference_q_ind:
373 |                         judge_right_num+=1
374 |                     else:
375 |                         des_right_num+=1
376 |                 elif reference_q_ind in clean_pred:
377 |                     ### filter
378 |                     if reference_q_ind in tips:
379 |                         tips.remove(reference_q_ind)
380 |                         if not any(tip in clean_pred for tip in tips):
381 |                             if 'yes' == reference_q_ind or 'no'  == reference_q_ind:
382 |                                 judge_right_num+=1
383 |                             else:
384 |                                 des_right_num+=1
385 |                 else:
386 |                     pass
387 |     if des_ques_total_num==0:
388 |         return (judge_ques_total_num+des_ques_total_num),des_right_num,obey_insytruction/(judge_ques_total_num+des_ques_total_num),judge_right_num/judge_ques_total_num
389 |     else:
390 |         return (judge_ques_total_num+des_ques_total_num),des_right_num/des_ques_total_num,obey_insytruction/(judge_ques_total_num+des_ques_total_num),judge_right_num/judge_ques_total_num
391 |         
392 | 
393 | func_mapping = {
394 |     'Pavement_Marking': Geneal_criterion_QA,
395 |     'Traffic_Sign': Geneal_criterion_QA,
396 |     'Traffic_Light': Geneal_criterion_QA,
397 |     'Right_Of_Way': Geneal_criterion_QA,
398 |     'Light': Geneal_criterion_QA,
399 |     'Weather': Geneal_criterion_QA,
400 |     'Lane_Recognition': Geneal_criterion_QA,
401 |     'Vehicle_Status': Geneal_criterion_QA,
402 |     'Vehicle_Recognition': Grounding_criterion_QA,
403 |     'VRU_Recognition': Grounding_criterion_QA,
404 |     'Obstruction_Recognition': Grounding_criterion_QA,
405 |     'Light_Lane_Relation': Relation_criterion_QA,
406 |     'Sign_Sign_Relation': Relation_criterion_QA,
407 |     'Sign_Lane_Relation': Relation_criterion_QA,
408 |     'Lane_Change_Relation': RoadChange_criterion_QA,
409 |     'Lane_Speed_Relation': RoadSpeed_criterion_QA,
410 |     'VRU_Cutin': Judge_criterion_QA,
411 |     'Vehicle_Cutin': Judge_criterion_QA,
412 |     'VRU_Cross': Judge_criterion_QA,
413 |     'Long_Short_Parking':Geneal_criterion_QA,
414 |     'Vehicle_Bahavior': Geneal_criterion_QA,
415 |     'VRU_Bahavior': Geneal_criterion_QA,
416 |     'Key_Obsturction_Detection': Judge_criterion_QA,
417 |     'Spatial_Temporal_Reasoning': Judge_criterion_QA,
418 |     'Risk_Prediction': Judge_criterion_QA,
419 |     'Drive_Efficiency': Geneal_criterion_QA,
420 |     'Longitudinal': Geneal_criterion_QA,
421 |     'Lateral': Geneal_criterion_QA
422 | }
423 | 


--------------------------------------------------------------------------------
/evaluate_vlm.py:
--------------------------------------------------------------------------------
 1 | from .evaluate_utils import*
 2 | import pandas as pd
 3 | import numpy as np
 4 | 
 5 | 
 6 | out_path = './output/'
 7 | all_task_json = './all_task.json'
 8 | All_MODELS = ["Qwen25_VL_7B","Qwen25_VL_72B"]
 9 | all_tasks = load_json_data(all_task_json)
10 | all_results = []
11 | total_results = [] 
12 | save_path = './all_results_metric.xlsx'
13 | weights = {
14 |     'Vehicle_Recognition':[0.3,0.5,0.2],
15 |     'VRU_Recognition':[0.3,0.5,0.2],
16 |     'Obstruction_Recognition':[0.3,0.5,0.2],
17 |     'Sign_Sign_Relation':[0.3,0.5,0.2],
18 |     'Sign_Lane_Relation':[0.3,0.5,0.2],
19 |     'Light_Lane_Relation':[0.3,0.5,0.2],
20 |     'Lane_Speed_Relation':[0.3,0.5,0.2],
21 |     'Lane_Change_Relation':[0.3,0.5,0.2],
22 |     'VRU_Cutin':[0.7,0.1,0.2],
23 |     'Vehicle_Cutin':[0.7,0.1,0.2],
24 |     'VRU_Cross':[0.7,0.1,0.2],
25 |     'Key_Obsturction_Detection':[0.8,0,0.2],
26 |     'Risk_Prediction':[0.7,0.1,0.2],
27 |     'Spatial_Temporal_Reasoning':[0.4,0.4,0.2]
28 | }
29 | 
30 | 
31 | for fir_ind, fir_task in enumerate(all_tasks):
32 |     sec_tasks = all_tasks[fir_task]
33 |     for sec_ind, sec_task in enumerate(sec_tasks):
34 |         if sec_task=='Ego_trajectory_Planning':continue
35 |         third_tasks = sec_tasks[sec_task]
36 |         third_rows = 0
37 |         for third_ind, third_task in enumerate(third_tasks):
38 |             third_rows +=1
39 |             model_scores = [third_task]
40 |             print('********************First Task:{}, Second Task:{}, Third Task:{}********************'.format(fir_task,sec_task,third_task))
41 |             for MODEL in All_MODELS:
42 |                 third_task_json_path = os.path.join(out_path, fir_task, sec_task, third_task,f'{MODEL}.json')
43 |                 if not isfile(third_task_json_path):
44 |                     print('ERROR:{}'.format('Json loading failed!'))
45 |                     model_scores.append(0)
46 |                     continue
47 |                 third_task_data = load_json_data(third_task_json_path)
48 |                 ques_total_num,right_num,obey_insytruction,others = func_mapping[third_task](third_task_data,MODEL)
49 |                 # print('Model: {}, Third Task: {}, Total Questions number: {}, others: {}, Accuary: {}, Obey: {}'.format(MODEL, third_task, ques_total_num,others, right_num, obey_insytruction))
50 |                 if third_task in weights:
51 |                     weight = weights[third_task]
52 |                 else:
53 |                     weight = [0, 0.8, 0.2]
54 |                 temp_score = 100*others*weight[0] + 100*right_num*weight[1] + 100*obey_insytruction*weight[2]
55 |                 print('Model: {}, Third Task: {}, model_scores: {}'.format(MODEL, third_task, temp_score))
56 |                 
57 |                 model_scores.append(temp_score)
58 |             model_scores.insert(1,ques_total_num)
59 |             all_results.append(model_scores)
60 |             total_results.append(model_scores)
61 |             
62 |         all_results = weighted_row_sum(all_results,third_rows)
63 |         
64 | total_ = weighted_total(total_results)
65 | all_results.append(total_)
66 | df = pd.DataFrame(all_results, columns=['Task','num']+All_MODELS)
67 | df.to_excel(save_path, sheet_name='Sheet1', index=False)


--------------------------------------------------------------------------------
/qwen2vl_all.py:
--------------------------------------------------------------------------------
  1 | 
  2 | import json
  3 | import os
  4 | import random
  5 | import pandas
  6 | from PIL import Image
  7 | import time
  8 | from transformers import Qwen2VLForConditionalGeneration, AutoTokenizer, AutoProcessor
  9 | from qwen_vl_utils import process_vision_info
 10 | import torch
 11 | import re
 12 | import ast
 13 | 
 14 | def isfile(path):
 15 |     return os.path.isfile(path)
 16 | 
 17 | def load_json_data(path):
 18 |     with open(path,'r',encoding='utf-8') as json_f:
 19 |         task_data = json.load(json_f)
 20 |         return task_data
 21 |     
 22 | def save_json_data(path,data):
 23 |     with open(path,'w',encoding='utf-8') as json_f:
 24 |         json.dump(data,json_f,ensure_ascii=False,indent=4)
 25 |         
 26 | root_path = 'xxx/VLADBench'
 27 | out_path = 'xxx/output/'
 28 | all_task_json = 'xxx/VLADBench/all_task.json'
 29 | MODEL = "Qwen2_VL_7B"
 30 | 
 31 | 
 32 | 
 33 | # We recommend enabling flash_attention_2 for better acceleration and memory saving, especially in multi-image and video scenarios.
 34 | model = Qwen2VLForConditionalGeneration.from_pretrained(
 35 |     "Qwen2-VL-7B-Instruct",
 36 |     torch_dtype=torch.bfloat16,
 37 |     attn_implementation="flash_attention_2",
 38 |     device_map="auto",
 39 | )
 40 | 
 41 | # default processer
 42 | processor = AutoProcessor.from_pretrained("Qwen2-VL-7B-Instruct")
 43 | with open(all_task_json,'r',encoding='utf-8') as json_f:
 44 |     all_tasks = json.load(json_f)
 45 |     
 46 | for fir_ind, fir_task in enumerate(all_tasks):
 47 |     sec_tasks = all_tasks[fir_task]
 48 |     for sec_ind, sec_task in enumerate(sec_tasks):
 49 |         third_tasks = sec_tasks[sec_task]
 50 |         for third_ind, third_task in enumerate(third_tasks):
 51 |             print('First Task:{}, Second Task:{}, Third Task:{}'.format(fir_task,sec_task,third_task))
 52 |             third_task_json_path = os.path.join(root_path, fir_task, sec_task, third_task+'_E.json')
 53 |             if isfile(third_task_json_path):
 54 |                 if not os.path.exists(os.path.join(out_path, fir_task, sec_task, third_task)):
 55 |                     os.makedirs(os.path.join(out_path, fir_task, sec_task, third_task))
 56 |             else:
 57 |                 print('ERROR:{}'.format('Json loading failed!'))
 58 | 
 59 |             if isfile(f'{os.path.join(out_path, fir_task, sec_task, third_task)}/{MODEL}.json'):
 60 |                 print('{} file exist!'.format(third_task))
 61 |                 third_task_data = load_json_data(f'{os.path.join(out_path, fir_task, sec_task, third_task)}/{MODEL}.json')
 62 |             else:
 63 |                 third_task_data = load_json_data(third_task_json_path)
 64 |             start_time = time.time()
 65 |             
 66 |             for d_ind, sample in enumerate(third_task_data):
 67 |                 country = sample['country']
 68 |                 if 'prediction' in sample:continue
 69 |                 else: sample['prediction'] = ['']*len(sample['questions'])
 70 |                 for q_ind, ques in enumerate(sample['questions']):
 71 |                     img_path, qe = ques.split(';')[0], ''.join(ques.split(';')[1:])
 72 |                     if 'located at the bounding box' in qe: 
 73 |                         dimensions = re.search(r'(\d+) and (\d+)', qe)
 74 |                         width, height = float(dimensions.group(1)), float(dimensions.group(2))
 75 |                         re_qe = re.sub(r'The width and height of the image are \d+ and \d+\.', '', qe)
 76 |                         bounding_box = re.search(r'\[\d+, \d+, \d+, \d+\]', re_qe)
 77 |                         if bounding_box:
 78 |                             bounding_box = ast.literal_eval(bounding_box.group(0))
 79 |                             new_bbox = [int(1000*bounding_box[0]/width), int(1000*bounding_box[1]/height), int(1000*bounding_box[2]/width), int(1000*bounding_box[3]/height)]
 80 |                             re_qe = re.sub(r'\[\d+, \d+, \d+, \d+\]', f'{new_bbox}', re_qe)
 81 |                         else:
 82 |                             print('bounding box error')
 83 |                         qe = re_qe
 84 | 
 85 |                     if ('[' in img_path) and (']' in img_path):
 86 |                         seq = img_path.strip('[]')
 87 |                         all_imgs = [os.path.join(root_path, fir_task, sec_task, third_task,sample['sequence'],img_) for img_ in sample[seq]]
 88 |                         qe = f'The sequence is from {country}. {qe}'
 89 |                         messages = [
 90 |                                         {
 91 |                                             "role": "user",
 92 |                                             "content": [
 93 |                                                 {
 94 |                                                     "type": "video",
 95 |                                                     "video": all_imgs,
 96 |                                                 },
 97 |                                                 {
 98 |                                                     "type": "text",
 99 |                                                     "text": qe},
100 |                                             ],
101 |                                         }
102 |                                     ]
103 |                         text = processor.apply_chat_template(
104 |                                                                 messages, tokenize=False, add_generation_prompt=True
105 |                                                             )
106 |                         image_inputs, video_inputs = process_vision_info(messages)
107 |                         inputs = processor(
108 |                                             text=[text],
109 |                                             images=image_inputs,
110 |                                             videos=video_inputs,
111 |                                             padding=True,
112 |                                             return_tensors="pt",
113 |                                         )
114 |                         inputs = inputs.to("cuda")
115 | 
116 |                         # Inference
117 |                         generated_ids = model.generate(**inputs, max_new_tokens=128)
118 |                         generated_ids_trimmed = [
119 |                                                     out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
120 |                                                 ]
121 |                         output_text = processor.batch_decode(
122 |                                                                 generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
123 |                                                             )
124 |                         # predictions_vlm.append(output_text[-1])
125 |                         sample['prediction'][q_ind] = output_text[-1]
126 |                         print(sample['reference'][q_ind],output_text[-1])
127 |                     else:
128 |                         qe = f'The image is from {country}. {qe}'
129 |                         image = Image.open(os.path.join(root_path, fir_task, sec_task, third_task,img_path))
130 |                         messages = [
131 |                                         {
132 |                                             "role": "user",
133 |                                             "content": [
134 |                                                 {
135 |                                                     "type": "image",
136 |                                                     "image": image,
137 |                                                 },
138 |                                                 {
139 |                                                     "type": "text",
140 |                                                     "text": qe
141 |                                                 },
142 |                                                     ],
143 |                                         }
144 |                                     ]
145 |                         text = processor.apply_chat_template(
146 |                                                                 messages, tokenize=False, add_generation_prompt=True
147 |                                                             )   
148 |                         image_inputs, video_inputs = process_vision_info(messages)
149 |                         inputs = processor(
150 |                                             text=[text],
151 |                                             images=image_inputs,
152 |                                             videos=video_inputs,
153 |                                             padding=True,
154 |                                             return_tensors="pt",
155 |                                         )
156 |                         inputs = inputs.to("cuda")
157 |                         generated_ids = model.generate(**inputs, max_new_tokens=128)
158 |                         generated_ids_trimmed = [
159 |                                                     out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
160 |                                                 ]
161 |                         output_text = processor.batch_decode(
162 |                                                                 generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
163 |                                                             )
164 |                         sample['prediction'][q_ind] = output_text[-1]
165 |                         print(sample['reference'][q_ind],output_text[-1])
166 |             print('Third Task: {}, Time Consumption: {}'.format(third_task,time.time()-start_time))
167 |             save_json_data(f'{os.path.join(out_path, fir_task, sec_task, third_task)}/{MODEL}.json', third_task_data)


--------------------------------------------------------------------------------