5 | 6 | 7 | [Qianhui Wu](https://qianhuiwu.github.io/)*1 8 | [Kanzhi Cheng](https://scholar.google.com/citations?user=S2IPVnwAAAAJ&hl=en&oi=ao/)*2 9 | [Rui Yang](https://yangrui2015.github.io/)*3 10 | [Chaoyun Zhang](https://vyokky.github.io/)1 11 | [Jianwei Yang](https://jwyang.github.io/)1 12 | [Huiqiang Jiang](https://hqjiang.com/)1
13 | [Jian Mu]()2 14 | [Baolin Peng](https://scholar.google.com/citations?user=u1CNjgwAAAAJ&hl=zh-CN)1 15 | [Bo Qiao](https://scholar.google.com/citations?user=_6ugrdYAAAAJ&hl=en)1 16 | [Reuben Tan](https://cs-people.bu.edu/rxtan/)1 17 | [Si Qin](https://sqin860.github.io/)1 18 | [Lars Liden](https://sites.google.com/site/larsliden)1
19 | [Qingwei Lin](https://scholar.google.com/citations?user=W9fdsxMAAAAJ&hl=zh-CN)1 20 | [Huan Zhang](https://huan-zhang.com/)3 21 | [Tong Zhang](https://tongzhang-ml.org/)3 22 | [Jianbing Zhang](https://cs.nju.edu.cn/zhangjb/index.htm)2 23 | [Dongmei Zhang](https://scholar.google.com/citations?user=jLlBBl4AAAAJ&hl=en)1 24 | [Jianfeng Gao](https://scholar.google.com/citations?user=CQ1cqKkAAAAJ&hl=en)1† 25 | 26 | 1 Microsoft Research 2 Nanjing University 3 University of Illinois Urbana-Champaign
27 | * Equal Contribution † Leadership 28 | 29 |
39 | 
6 |
7 | ## Training
8 |
9 | The verifier is trained to take a language instruction and an image (with a red circle marking the candidate position) as input, and predict whether the position is correct—outputting "True" or "False."
10 |
11 | We use the [OS-Atlas dataset](https://huggingface.co/datasets/OS-Copilot/OS-Atlas-data) and process it using `verifier_data_generation.py` to curate training data. The model is fine-tuned via supervised training (SFT) starting from the UITARS-SFT-2B checkpoint, providing strong performance with a relatively small model size.
12 |
13 | ### Data Preparation
14 |
15 | To prepare the dataset:
16 |
17 | 1. Download and unzip the [OS-Atlas dataset](https://huggingface.co/datasets/OS-Copilot/OS-Atlas-data) following the instructions on the Hugging Face page.
18 | 2. Organize the images into the following directory structure:
19 |
20 | ```python
21 | image_folder_dict = {
22 | 'windows_splited': f'{root_path}/desktop_domain/windows_images',
23 | 'linux_splited': f'{root_path}/desktop_domain/linux_images',
24 | 'macos_splited': f'{root_path}/desktop_domain/macos_images',
25 | 'widget_captioning': f'{root_path}/mobile_domain/combined',
26 | 'uibert_raw': f'{root_path}/mobile_domain/UIBert',
27 | 'ricosca': f'{root_path}/mobile_domain/combined',
28 | 'amex_raw': f'{root_path}/mobile_domain/amex_images',
29 | 'seeclick_web': f'{root_path}/web_domain/seeclick_web_imgs',
30 | 'fineweb_3m': f'{root_path}/web_domain/fineweb'
31 | }
32 | ```
33 |
34 | Each training sample includes a positive and one or more negative examples:
35 |
36 | * **Positive samples**: taken directly from the original dataset with a red circle marking the correct target.
37 | * **Negative samples**: created by either (a) selecting another meaningful UI element or (b) randomly sampling a point, which may not correspond to any actionable item.
38 |
39 | To generate the dataset, run the following commands (since the dataset is very large, you can ):
40 |
41 | ```bash
42 | python verifier_data_generation.py --root_path ${path_to_OS-Atlas-data} --new_directory ${save_path} --file_dict_key desktop_domain --selected_size 30000
43 | python verifier_data_generation.py --root_path ${path_to_OS-Atlas-data} --new_directory ${save_path} --file_dict_key mobile_domain --selected_size 30000
44 | python verifier_data_generation.py --root_path ${path_to_OS-Atlas-data} --new_directory ${save_path} --file_dict_key web_domain --selected_size 30000
45 | ```
46 |
47 |
48 | ### SFT
49 |
50 | We use the official code from [Aguvis](https://github.com/xlang-ai/aguvis) to perform SFT training. Make sure to set the file path correctly in the `stage1.yaml` configuration. For training, we use [**UITARS-2B-SFT**](https://huggingface.co/ByteDance-Seed/UI-TARS-2B-SFT) as the base model with a learning rate of $2 \times 10^{-5}$, running for one epoch.
51 |
52 |
53 |
54 | ## Evaluation
55 |
56 | We evaluate our method using the attention weights generated by GUI-Actor and the grounding verifier, saved in a JSON file (e.g., `screenspot_all_preds_Original.json`). Before running the evaluation scripts, please update the file paths in `run_ss_v1.sh`, `run_ss_v2.sh`, and `run_ss_pro.sh` accordingly.
57 |
58 | Make sure to download the ScreenSpot datasets and ensure their paths exactly match those specified in the shell scripts. Specifically, download **ScreenSpot** and **ScreenSpot-Pro** from [ss-v1](https://huggingface.co/datasets/rootsautomation/ScreenSpot) and [ss-pro](https://huggingface.co/datasets/likaixin/ScreenSpot-Pro), respectively.
59 | For **ScreenSpot-v2**, we provide a converted version (`ScreenSpot-v2-new`) that aligns with the format used by the other datasets. However, you still need to download the original images from [ss-v2](https://huggingface.co/datasets/OS-Copilot/ScreenSpot-v2).
60 |
61 |
62 | Once everything is set up, run the following commands:
63 |
64 | ```bash
65 | bash run_ss_v1.sh
66 | bash run_ss_v2.sh
67 | bash run_ss_pro.sh
68 | ```
69 |
70 |
71 |
72 |
73 |
74 |
--------------------------------------------------------------------------------
/verifier/eval_ss_with_verifier.py:
--------------------------------------------------------------------------------
1 | import copy
2 | import itertools
3 | import torch
4 | import json
5 | import re
6 | import argparse
7 | import os
8 | from PIL import Image, ImageDraw
9 | import logging
10 | from tqdm import tqdm
11 |
12 |
13 |
14 | def draw_annotations(img, point_in_pixel, bbox, output_path='test.png'):
15 | draw = ImageDraw.Draw(img)
16 |
17 | # Draw the ground truth bounding box in green
18 | if bbox:
19 | # Assuming bbox format is [x1, y1, x2, y2]
20 | draw.rectangle(bbox, outline="yellow", width=4)
21 |
22 | # Draw a small rectangle around the predicted point in red
23 | if point_in_pixel:
24 | # Create a small rectangle around the point (5 pixels in each direction)
25 | radius = 8
26 | circle_bbox = [
27 | point_in_pixel[0] - radius, # x1
28 | point_in_pixel[1] - radius, # y1
29 | point_in_pixel[0] + radius, # x2
30 | point_in_pixel[1] + radius # y2
31 | ]
32 | draw.ellipse(circle_bbox, outline="red", width=4)
33 |
34 | img.save(output_path)
35 | print(f"Annotated image saved to {output_path}")
36 | return img
37 |
38 |
39 |
40 |
41 | logging.basicConfig(level=logging.INFO)
42 | torch.manual_seed(114514)
43 |
44 |
45 | GT_TYPES = ['positive', 'negative']
46 | INSTRUCTION_STYLES = ['instruction', 'action', 'description']
47 | LANGUAGES = ['en', 'cn']
48 |
49 |
50 | def parse_args():
51 | parser = argparse.ArgumentParser()
52 | parser.add_argument('--model_path', type=str, required=False)
53 | parser.add_argument('--screenspot_imgs', type=str, required=True)
54 | parser.add_argument('--screenspot_test', type=str, required=True)
55 | parser.add_argument('--task', type=str, required=True)
56 | parser.add_argument('--inst_style', type=str, required=True, choices=INSTRUCTION_STYLES + ['all'], help="Instruction style to use.")
57 | parser.add_argument('--language', type=str, required=True, choices=LANGUAGES + ['all'], default='en', help="Language to use.")
58 | parser.add_argument('--gt_type', type=str, required=True, choices=GT_TYPES + ['all'], help="Ground truth type: 'positive' or 'negative'.")
59 | parser.add_argument('--log_path', type=str, required=True)
60 | parser.add_argument('--json_prediction', type=str, required=False)
61 | parser.add_argument('--verifier_path', type=str, required=True)
62 | parser.add_argument('--verifier_method', type=str, required=True)
63 |
64 | args = parser.parse_args()
65 | return args
66 |
67 |
68 | def build_model(args):
69 | from verifier_model import GroundingVerifier
70 | model = GroundingVerifier(model_name_or_path=args.model_path, json_prediction=args.json_prediction, method=args.verifier_method)
71 | model.load_model(args.verifier_path)
72 | return model
73 |
74 |
75 | def collect_results_to_eval(results, platform=None, group=None, application=None, language=None, gt_type=None, instruction_style=None, ui_type=None):
76 | """
77 | Filters the results based on provided values. None means include all (ignore filtering this attribute).
78 |
79 |
80 | Parameters:
81 | results (list): A list of dictionaries containing sample results.
82 |
83 | Returns:
84 | list: A filtered list of dictionaries based on the given criteria.
85 | """
86 | filtered_results = []
87 |
88 |
89 | for sample in results:
90 | # Check each filter condition; if None, consider it as passed
91 | if (platform is None or sample.get("platform") == platform) and \
92 | (group is None or sample.get("group") == group) and \
93 | (application is None or sample.get("application") == application) and \
94 | (language is None or sample.get("language") == language) and \
95 | (gt_type is None or sample.get("gt_type") == gt_type) and \
96 | (instruction_style is None or sample.get("instruction_style") == instruction_style) and \
97 | (ui_type is None or sample.get("ui_type") == ui_type):
98 | filtered_results.append(sample)
99 |
100 |
101 | return filtered_results
102 |
103 |
104 |
105 |
106 | def make_combinations(results, platform=False, group=None, application=False, language=False, gt_type=False, instruction_style=False, ui_type=False):
107 | """
108 | Returns a list of combinations of values for attributes where the corresponding parameter is set to True.
109 | """
110 | # Initialize a dictionary to store unique values for each attribute
111 | unique_values = {
112 | "platform": set(),
113 | "group": set(),
114 | "application": set(),
115 | "language": set(),
116 | "gt_type": set(),
117 | "instruction_style": set(),
118 | "ui_type": set(),
119 | }
120 |
121 |
122 | # Collect unique values from the results
123 | for sample in results:
124 | if platform:
125 | unique_values["platform"].add(sample.get("platform"))
126 | if group:
127 | unique_values["group"].add(sample.get("group"))
128 | if application:
129 | unique_values["application"].add(sample.get("application"))
130 | if language:
131 | unique_values["language"].add(sample.get("language"))
132 | if gt_type:
133 | unique_values["gt_type"].add(sample.get("gt_type"))
134 | if instruction_style:
135 | unique_values["instruction_style"].add(sample.get("instruction_style"))
136 | if ui_type:
137 | unique_values["ui_type"].add(sample.get("ui_type"))
138 |
139 |
140 | # Filter out the attributes that are set to False (no need for combinations)
141 | filtered_values = {key: list(value) for key, value in unique_values.items() if value}
142 | if not filtered_values:
143 | return []
144 |
145 |
146 | # Generate all combinations of the selected attributes using itertools.product
147 | attribute_combinations = list(itertools.product(*filtered_values.values()))
148 |
149 |
150 | # Convert combinations into dictionaries with corresponding attribute names
151 | combinations = []
152 | for combination in attribute_combinations:
153 | combinations.append(dict(zip(filtered_values.keys(), combination)))
154 |
155 |
156 | return combinations
157 |
158 |
159 |
160 |
161 | def calc_metric_for_result_list(results):
162 | """Calculates the metrics for a simple result list."""
163 | num_total = len(results)
164 | correct_num = sum(1 for res in results if res["correctness"] == "correct")
165 | wrong_format_num = sum(1 for res in results if res["correctness"] == "wrong_format")
166 |
167 |
168 | # Calculate text and icon specific metrics using collect_results_to_eval
169 | text_results = collect_results_to_eval(results, ui_type="text")
170 | icon_results = collect_results_to_eval(results, ui_type="icon")
171 |
172 |
173 | text_correct = sum(1 for res in text_results if res["correctness"] == "correct")
174 | text_total = len(text_results)
175 | icon_correct = sum(1 for res in icon_results if res["correctness"] == "correct")
176 | icon_total = len(icon_results)
177 | metrics = {
178 | "num_correct_action": correct_num,
179 | "num_total": num_total,
180 | "wrong_format_num": wrong_format_num,
181 | "action_acc": correct_num / num_total if num_total > 0 else 0,
182 | "text_acc": text_correct / text_total if text_total > 0 else 0,
183 | "icon_acc": icon_correct / icon_total if icon_total > 0 else 0
184 | }
185 | return metrics
186 |
187 |
188 |
189 |
190 | def eval_sample_positive_gt(sample, response):
191 | bbox = sample["bbox"]
192 | bbox = [bbox[0], bbox[1], bbox[2], bbox[3]] # x1, y1, x2, y2
193 | # bbox = [bbox[0], bbox[1], bbox[0] + bbox[2], bbox[1] + bbox[3]] # x1, y1, w, h
194 | img_size = sample["img_size"]
195 | bbox = [bbox[0] / img_size[0], bbox[1] / img_size[1], bbox[2] / img_size[0], bbox[3] / img_size[1]]
196 |
197 | click_point = response["point"] # may be none
198 | print(click_point, bbox)
199 | # import pdb;pdb.set_trace()
200 | if click_point is None:
201 | return "wrong_format"
202 | # Check if the predicted point falls in the ground truth box
203 | if (bbox[0] <= click_point[0] <= bbox[2]) and (bbox[1] <= click_point[1] <= bbox[3]):
204 | return "correct"
205 | else:
206 | return "wrong"
207 |
208 | def eval_sample_negative_gt(sample, response):
209 | if response["result"] == "negative":
210 | return "correct"
211 | elif response["result"] == "positive":
212 | return "wrong"
213 | else: ## response["result"] == wrong_format
214 | return "wrong_format"
215 |
216 |
217 | def evaluate_fine_grained(results):
218 | # Generate all combinations of platform, instruction_style, and gt_type
219 | combinations = make_combinations(
220 | results,
221 | platform=True,
222 | application=True,
223 | instruction_style=True,
224 | gt_type=True
225 | )
226 |
227 |
228 | evaluation_result = {}
229 |
230 |
231 | # Iterate through each combination
232 | for combo in combinations:
233 | platform = combo.get("platform")
234 | application = combo.get("application")
235 | inst_style = combo.get("instruction_style")
236 | gt_type = combo.get("gt_type")
237 |
238 | # Filter results for the current combination
239 | filtered_results = collect_results_to_eval(
240 | results=results,
241 | platform=platform,
242 | application=application,
243 | instruction_style=inst_style,
244 | gt_type=gt_type
245 | )
246 |
247 | # Calculate metrics using the calc_metric_for_result_list function
248 | metrics = calc_metric_for_result_list(filtered_results)
249 | if metrics['num_total'] == 0:
250 | continue
251 |
252 | # Construct a unique key based on the combination
253 | key = f"plat:{platform} app:{application} inst_style:{inst_style} gt_type:{gt_type}"
254 | evaluation_result[key] = metrics
255 |
256 |
257 | return evaluation_result
258 |
259 |
260 | def evaluate_fine_grained_v2(results):
261 | # Generate all combinations of platform, instruction_style, and gt_type
262 | combinations = make_combinations(
263 | results,
264 | group=True,
265 | )
266 |
267 |
268 | evaluation_result = {}
269 |
270 |
271 | # Iterate through each combination
272 | for combo in combinations:
273 | group = combo.get("group")
274 |
275 |
276 | # Filter results for the current combination
277 | filtered_results = collect_results_to_eval(
278 | results=results,
279 | group=group,
280 | )
281 |
282 | # Calculate metrics using the calc_metric_for_result_list function
283 | metrics = calc_metric_for_result_list(filtered_results)
284 | if metrics['num_total'] == 0:
285 | continue
286 |
287 | # Construct a unique key based on the combination
288 | key = f"group:{group}"
289 | evaluation_result[key] = metrics
290 |
291 |
292 | return evaluation_result
293 |
294 |
295 | def evaluate_seeclick_paper_style(results):
296 | # Generate all combinations of platform, instruction_style, and gt_type
297 | combinations = make_combinations(
298 | results,
299 | platform=True,
300 | instruction_style=True,
301 | gt_type=True
302 | )
303 |
304 |
305 | evaluation_result = {}
306 |
307 |
308 | # Iterate through each combination
309 | for combo in combinations:
310 | platform = combo.get("platform")
311 | inst_style = combo.get("instruction_style")
312 | gt_type = combo.get("gt_type")
313 |
314 | # Filter results for the current combination
315 | filtered_results = collect_results_to_eval(
316 | results=results,
317 | platform=platform,
318 | instruction_style=inst_style,
319 | gt_type=gt_type
320 | )
321 |
322 | # Calculate metrics using the calc_metric_for_result_list function
323 | metrics = calc_metric_for_result_list(filtered_results)
324 | if metrics['num_total'] == 0:
325 | continue
326 |
327 | # Construct a unique key based on the combination
328 | key = f"plat:{platform} inst_style:{inst_style} gt_type:{gt_type}"
329 | evaluation_result[key] = metrics
330 |
331 |
332 | return evaluation_result
333 |
334 |
335 | def evaluate_leaderboard_detailed_style(results):
336 | # Generate all combinations of platform, instruction_style, and gt_type
337 | combinations = make_combinations(
338 | results,
339 | application=True,
340 | )
341 |
342 |
343 | evaluation_result = {}
344 |
345 |
346 | # Iterate through each combination
347 | for combo in combinations:
348 | application = combo.get("application")
349 |
350 | # Filter results for the current combination
351 | filtered_results = collect_results_to_eval(
352 | results=results,
353 | application=application,
354 | )
355 |
356 | # Calculate metrics using the calc_metric_for_result_list function
357 | metrics = calc_metric_for_result_list(filtered_results)
358 | if metrics['num_total'] == 0:
359 | continue
360 |
361 | # Construct a unique key based on the combination
362 | key = f"app:{application}"
363 | evaluation_result[key] = metrics
364 |
365 |
366 | return evaluation_result
367 |
368 |
369 | def evaluate_leaderboard_simple_style(results):
370 | # Generate all combinations of platform, instruction_style, and gt_type
371 | combinations = make_combinations(
372 | results,
373 | group=True,
374 | )
375 |
376 |
377 | evaluation_result = {}
378 |
379 |
380 | # Iterate through each combination
381 | for combo in combinations:
382 | group = combo.get("group")
383 |
384 | # Filter results for the current combination
385 | filtered_results = collect_results_to_eval(
386 | results=results,
387 | group=group,
388 | )
389 |
390 | # Calculate metrics using the calc_metric_for_result_list function
391 | metrics = calc_metric_for_result_list(filtered_results)
392 | if metrics['num_total'] == 0:
393 | continue
394 |
395 | # Construct a unique key based on the combination
396 | key = f"group:{group}"
397 | evaluation_result[key] = metrics
398 |
399 |
400 | return evaluation_result
401 |
402 |
403 | def evaluate_overall(results):
404 | """
405 | Evaluates the overall metrics for all results without any filtering.
406 |
407 | Parameters:
408 | results (list): A list of dictionaries containing sample results.
409 |
410 | Returns:
411 | dict: A dictionary containing the overall metrics.
412 | """
413 | # Calculate metrics for the entire result set
414 | metrics = calc_metric_for_result_list(results)
415 |
416 | return metrics
417 |
418 |
419 |
420 |
421 | def evaluate(results):
422 | """Collect results and calculate metrics. You can comment out function calls or add new ones based on your need.
423 | """
424 | result_report = {
425 | "details": [], # Store detailed information for each sample
426 | "metrics": {}
427 | }
428 |
429 |
430 | # # TODO: comment out function calls based on your need
431 | result_report["metrics"]["fine_grained"] = evaluate_fine_grained_v2(results)
432 | # result_report["metrics"]["seeclick_style"] = evaluate_seeclick_paper_style(results)
433 | # result_report["metrics"]["leaderboard_simple_style"] = evaluate_leaderboard_simple_style(results)
434 | # result_report["metrics"]["leaderboard_detailed_style"] = evaluate_leaderboard_detailed_style(results)
435 | result_report["metrics"]["overall"] = evaluate_overall(results)
436 |
437 |
438 | # Save detailed results
439 | result_report["details"] = results
440 |
441 |
442 | return result_report
443 |
444 |
445 | def main(args):
446 | model = build_model(args)
447 | print("Load model success")
448 |
449 |
450 | if args.task == "all":
451 | task_filenames = [
452 | os.path.splitext(f)[0]
453 | for f in os.listdir(args.screenspot_test)
454 | if f.endswith(".json")
455 | ]
456 | else:
457 | task_filenames = args.task.split(",")
458 |
459 |
460 | if args.inst_style == "all":
461 | inst_styles = INSTRUCTION_STYLES
462 | else:
463 | inst_styles = args.inst_style.split(",")
464 |
465 |
466 | if args.language == "all":
467 | languages = LANGUAGES
468 | else:
469 | languages = args.language.split(",")
470 |
471 |
472 | if args.gt_type == "all":
473 | gt_types = GT_TYPES
474 | else:
475 | gt_types = args.gt_type.split(",")
476 |
477 |
478 | tasks_to_run = []
479 | for task_filename in task_filenames:
480 | dataset = task_filename + ".json"
481 | with open(os.path.join(args.screenspot_test, dataset), 'r') as f:
482 | task_data = json.load(f)
483 |
484 |
485 | # Create the list of tasks to run, one item as an instance. Tasks may be reused.
486 | for inst_style in inst_styles: # Expand tasks based on user configurations
487 | for gt_type in gt_types:
488 | for lang in languages:
489 | for task_instance in task_data: # [30:]
490 | task_instance = copy.deepcopy(task_instance)
491 | task_instance["task_filename"] = task_filename
492 | task_instance["gt_type"] = gt_type
493 | task_instance["instruction_style"] = inst_style
494 | task_instance["language"] = lang
495 | if lang == "cn":
496 | if inst_style!= 'instruction' or gt_type != 'positive':
497 | # TODO: Translate the data
498 | raise AttributeError("Only positive samples and 'instruction' style are supported for Chinese instructions.")
499 | task_instance["prompt_to_evaluate"] = task_instance["instruction_cn"]
500 | elif lang == "en":
501 | task_instance["prompt_to_evaluate"] = task_instance["instruction"]
502 |
503 |
504 | tasks_to_run.append(task_instance)
505 | print(f"Num of sample in {task_filename}: {len(task_data)} * {len(inst_styles)} * {len(gt_types)} * {len(languages)} = {len(task_data) * len(inst_styles) * len(gt_types) * len(languages)}")
506 | print(f"Total tasks: {len(tasks_to_run)}")
507 |
508 |
509 | results = []
510 | for sample in tqdm(tasks_to_run[:]):
511 | filename = sample["img_filename"]
512 | img_path = os.path.join(args.screenspot_imgs, filename)
513 |
514 | if task_instance["gt_type"] == "positive":
515 | response = model.ground_only_positive(instruction=sample["prompt_to_evaluate"], image=img_path, target_point=sample['bbox'])
516 |
517 |
518 | elif task_instance["gt_type"] == "negative":
519 | response = model.ground_allow_negative(instruction=sample["prompt_to_evaluate"], image=img_path)
520 | # print(response)
521 | point = response["point"]
522 | img_size = sample["img_size"]
523 | point_in_pixel = [point[0] * img_size[0], point[1] * img_size[1]] if point else None
524 |
525 | sample_result = {
526 | "img_path": img_path,
527 | "group": sample["group"] if "group" in sample else None,
528 | "platform": sample["platform"],
529 | "application": sample["application"] if 'application' in sample else None,
530 | "lang": sample["language"],
531 | "instruction_style": sample["instruction_style"] if 'instruction_style' in sample else None,
532 | "prompt_to_evaluate": sample["prompt_to_evaluate"],
533 | "gt_type": sample["gt_type"],
534 | "ui_type": sample["ui_type"],
535 | "task_filename": sample["task_filename"],
536 | "pred": point_in_pixel,
537 | "raw_response": response["raw_response"]
538 | }
539 |
540 | if sample["gt_type"] == "positive":
541 | correctness = eval_sample_positive_gt(sample, response)
542 | sample_result.update({
543 | "bbox": sample["bbox"],
544 | })
545 | print(correctness)
546 | elif sample["gt_type"] == "negative":
547 | correctness = eval_sample_negative_gt(sample, response)
548 | else:
549 | raise ValueError("Wrong instruction type")
550 |
551 |
552 |
553 | sample_result.update({
554 | "correctness": correctness,
555 | })
556 | results.append(sample_result)
557 |
558 | result_report = evaluate(results)
559 | # Save to file
560 | os.makedirs(os.path.dirname(args.log_path), exist_ok=True)
561 | with open(args.log_path, 'w') as f:
562 | json.dump(result_report, f, indent=4)
563 | logging.info("Evaluation of ScreenSpot finished.")
564 |
565 |
566 |
567 |
568 | if __name__ == "__main__":
569 | main(parse_args())
570 |
571 |
572 |
573 |
--------------------------------------------------------------------------------
/verifier/run_ss_pro.sh:
--------------------------------------------------------------------------------
1 | # !/bin/bash
2 | set -e
3 |
4 | json_path='Screenspot_eval/json_data/7B_full_qwen2vl/final_eval/screenspot-Pro_all_preds_StandardResize.json'
5 | exp_name='Actor-7b-fixprompt-bon_score_verifier'
6 | verifier_path='microsoft/GUI-Actor-Verifier-2B'
7 | screenspot_dataset_path='/datadisk/data/ss-eval/ScreenSpot-Pro'
8 | logdir='results_pro'
9 |
10 | verifier_method='score'
11 | # verifier_method='best_one'
12 | export CUDA_VISIBLE_DEVICES=0
13 |
14 |
15 | python eval_ss_with_verifier.py \
16 | --screenspot_imgs ${screenspot_dataset_path}'/images' \
17 | --screenspot_test ${screenspot_dataset_path}'/annotations' \
18 | --task "all" \
19 | --language "en" \
20 | --gt_type "positive" \
21 | --log_path "${logdir}/${exp_name}_${checkpoint}_sspro.json" \
22 | --inst_style "instruction" \
23 | --verifier_method ${verifier_method} \
24 | --verifier_path ${verifier_path} \
25 | --json_prediction ${json_path}
26 |
27 |
--------------------------------------------------------------------------------
/verifier/run_ss_v1.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 | set -e
3 |
4 | json_path='/home/t-yangrui/code/Screenspot_eval/json_data/new_prompt_7B/screenspot_all_preds_Original.json'
5 | exp_name='Actor-7b-warmup-fixprompt-bon_score_verifierultracpt6000_crop500'
6 | verifier_path='microsoft/GUI-Actor-Verifier-2B'
7 | screenspot_dataset_path="data/ss-eval/ScreenSpot"
8 | logdir='results_v1'
9 |
10 | verifier_method='score'
11 | # verifier_method='best_one'
12 | export CUDA_VISIBLE_DEVICES=0
13 |
14 |
15 | python eval_ss_with_verifier.py \
16 | --screenspot_imgs ${screenspot_dataset_path}'/images' \
17 | --screenspot_test ${screenspot_dataset_path} \
18 | --task "all" \
19 | --language "en" \
20 | --gt_type "positive" \
21 | --log_path "${logdir}/${exp_name}_${checkpoint}_ssv1.json" \
22 | --inst_style "instruction" \
23 | --verifier_method ${verifier_method} \
24 | --verifier_path ${verifier_path} \
25 | --json_prediction ${json_path}
26 |
27 |
--------------------------------------------------------------------------------
/verifier/run_ss_v2.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 | set -e
3 |
4 | models=("aguvis-with-verifier")
5 |
6 |
7 | json_path='/home/t-yangrui/code/Screenspot_eval/json_data/new_prompt_3B/screenspot_v2_all_preds_Original.json'
8 | exp_name='Actor-7b-warmup-fixprompt-bon_score_verifierultracpt6000_crop500'
9 | verifier_path='microsoft/GUI-Actor-Verifier-2B'
10 | screenspot_dataset_path='ss-eval/ScreenSpot-v2'
11 | logdir='results_v2'
12 |
13 |
14 | verifier_method='score'
15 | # verifier_method='best_one'
16 | export CUDA_VISIBLE_DEVICES=0
17 |
18 |
19 | python eval_ss_with_verifier.py \
20 | --screenspot_imgs "${screenspot_dataset_pat}/screenspotv2_image" \
21 | --screenspot_test "ScreenSpot-v2-new" \
22 | --task "all" \
23 | --language "en" \
24 | --gt_type "positive" \
25 | --log_path "${logdir}/${exp_name}_${checkpoint}_ssv2.json" \
26 | --inst_style "instruction" \
27 | --verifier_method ${verifier_method} \
28 | --verifier_path ${verifier_path} \
29 | --json_prediction ${json_path}
30 |
31 |
32 |
33 |
--------------------------------------------------------------------------------
/verifier/verifier_data_generation.py:
--------------------------------------------------------------------------------
1 | import json
2 | import os
3 | import re
4 | import numpy as np
5 | import random
6 | from PIL import Image, ImageDraw
7 | import argparse
8 |
9 |
10 | dic = {
11 | "from": "gpt",
12 | "value": "True",
13 | "recipient": "os",
14 | "end_turn": True
15 | }
16 | neg_dic = {
17 | "from": "gpt",
18 | "value": "False",
19 | "recipient": "os",
20 | "end_turn": True
21 | }
22 |
23 |
24 |
25 | def sample_point(bbox):
26 | x0, y0, x1, y1 = bbox
27 | t = 0
28 | while t <= 50:
29 | xx, yy = np.random.random(2)
30 | t += 1
31 | if not ((x0 < xx < x1) and (y0 < yy < y1)):
32 | break
33 | if t > 50:
34 | return
35 | return xx, yy
36 |
37 |
38 | def load_json_file(file_path):
39 | """Load and parse JSON data from a file."""
40 | try:
41 | with open(file_path, 'r') as file:
42 | data = json.load(file)
43 | return data
44 | except FileNotFoundError:
45 | print(f"Error: File '{file_path}' not found.")
46 | return None
47 | except json.JSONDecodeError:
48 | print(f"Error: '{file_path}' contains invalid JSON.")
49 | return None
50 |
51 |
52 | def draw_annotations(img, point_in_pixel, bbox, output_path='test.png', color='red', size=1):
53 | draw = ImageDraw.Draw(img)
54 |
55 | # Draw the ground truth bounding box in green
56 | if bbox:
57 | # Assuming bbox format is [x1, y1, x2, y2]
58 | draw.rectangle(bbox, outline="yellow", width=4)
59 | # Draw a small rectangle around the predicted point in red
60 | if point_in_pixel:
61 | # Create a small rectangle around the point (5 pixels in each direction)
62 | radius = np.ceil(8 * size).astype(int)
63 | circle_bbox = [
64 | point_in_pixel[0] - radius, # x1
65 | point_in_pixel[1] - radius, # y1
66 | point_in_pixel[0] + radius, # x2
67 | point_in_pixel[1] + radius # y2
68 | ]
69 | draw.ellipse(circle_bbox, outline=color, width=np.ceil(4 * size).astype(int))
70 |
71 | img.save(output_path)
72 | print(f"Annotated image saved to {output_path}")
73 | return img
74 |
75 |
76 | def transform_to_conversation_format(data, file, image_folder_dict, new_directory):
77 | """
78 | Transform the input data to the specified conversation format.
79 | Args:
80 | data: List of dictionaries containing webpage elements data
81 |
82 | Returns:
83 | List of dictionaries in the conversation format
84 | """
85 | image_folder = image_folder_dict[file]
86 | result = []
87 | for i, item in enumerate(data):
88 | print(i / len(data))
89 | img_filename = item['img_filename']
90 |
91 | prompt = 'Please observe the screenshot and exame whether the hollow red circle accurately placed on the intended position in the image:'
92 |
93 | if 'elements' in item:
94 | # sample n//2 element
95 | n = len(item['elements'])
96 | ind_list = []
97 | if n <= 1:
98 | ind_list = [0]
99 | else:
100 | ind_list = random.sample(range(n), min(n//2, 3))
101 |
102 | for ind in ind_list:
103 | conversations = []
104 | instruction = item['elements'][ind]['instruction']
105 | bbox = item['elements'][ind]['bbox']
106 | if (bbox[2] - bbox[0]) * (bbox[3] - bbox[1]) >= 0.8:
107 | continue
108 |
109 | conversations.append({
110 | "from": "human",
111 | "value": f"