2 |
3 |
13 | 14 |
20 |
21 | 
22 |
23 | LVLM-Playground is a benchmark to evaluate Large Vision Language Models (LVLMs) on game-playing tasks, assessing their perception, reasoning, and decision-making across six classic games. This repository provides tools to run experiments, analyze performance, and visualize results. For further details, please refer to our paper [here](https://openreview.net/pdf?id=c4OGMNyzPT).
24 |
25 | ## News
26 |
27 | \[**2025.Mar**\] LVLM-Playground is released! 🚀 👾Are Large Vision Language Models Good Game Players?
4 | 5 | 10 | 🎮The University of Adelaide 🕹️Zhejiang University 11 | 12 |13 | 14 |
15 | 
16 | 
17 |
18 |
19 |
16 |
17 | 28 | \[**2025.Feb**\] LVLM-Playground has been accepted to ICLR 2025! 🎉 Check the paper [here](https://openreview.net/pdf?id=c4OGMNyzPT). 29 | 30 | ## Installation 31 | 32 | 1. **Clone the Repository**: 33 | 34 | ```bash 35 | git clone https://github.com/xinke-wang/LVLM-Playground.git 36 | cd LVLM-Playground 37 | ``` 38 | 39 | 2. **Setup a Conda Environment** 40 | 41 | ```bash 42 | conda create -n playground python=3.11 -y 43 | conda activate playground 44 | pip install -r requirements.txt 45 | ``` 46 | 47 | 3. **Install Stockfish for Chess** 48 | 49 | To run experiments on Chess, you need to install Stockfish. 50 | 51 | - With `sudo` privileges, install [Stockfish](https://stockfishchess.org/) via your package manager: 52 | 53 | ```bash 54 | sudo apt-get install stockfish 55 | ``` 56 | 57 | - Alternatively, you can download the latest Stocfish binary from [stockfishcess.org](https://stockfishchess.org/download/). 58 | 59 | - Extract the binary and place the `stockfish` executable in your system PATH or the project directory. 60 | 61 | ## Data Preparation 62 | 63 | 1. **Download Pre-generated Benchmark Data**: 64 | 65 | To facilitate reproducibility of the experiments in our paper, we provide pre-generated benchmark data. You can download the data by running the following command: 66 | 67 | ```bash 68 | wget https://universityofadelaide.box.com/shared/static/9xx4brpiipqmmyomau2v522frtijx930.zip -O benchmark.zip 69 | unzip benchmark.zip -d . 70 | ``` 71 | 72 | After unzipping, you should have the following directory structure: 73 | 74 | ``` 75 | LVLM-Playground 76 | ├── benchmark 77 | │ ├── perceive 78 | │ │ ├── chess 79 | │ │ │ ├── 0000000.jpg 80 | │ │ │ ├── 0000001.jpg 81 | │ │ │ ├── ... 82 | │ │ │ └── annotation.json 83 | │ │ ├── gomoku 84 | │ │ ├── minesweeper 85 | │ │ ├── reversi 86 | │ │ ├── sudoku 87 | │ │ └── tictactoe 88 | │ ├── qa 89 | │ └── rule 90 | ``` 91 | 92 | 2. **Generating a Custom Benchmark (Optional)**: 93 | 94 | Alternatively, you can generate a new benchmark dataset by running the following command: 95 | 96 | ```bash 97 | python generate_benchmark.py 98 | ``` 99 | 100 | You can modify the `configs/base.py` file to customize the benchmark generation process. 101 | 102 | ```python 103 | # configs/base.py 104 | benchmark_setting = dict( 105 | games=['tictactoe', 'gomoku', 'minesweeper', 'reversi', 'sudoku', 'chess'], 106 | sample_size=2000, 107 | e2e_round=100, 108 | offline_task=['perceive', 'qa', 'rule'], 109 | benchmark_path='benchmark' 110 | ) 111 | ``` 112 | 113 | - Adjust `games` to include or exclude specific games. 114 | - Modify `sample_size` to control the number of samples per game. 115 | - Change `benchmark_path` to specify the output directory. 116 | 117 | ## Running Experiments 118 | 119 | Once the data is ready, run experiments using: 120 | 121 | ```bash 122 | python run.py --exp-recipe configs/recipe/base.py --agent-cfg configs/agents/internvl/internvl2-1b.py 123 | ``` 124 | 125 | `--exp-recipe` specifies the experiment settings, and `--agent-cfg` specifies the agent configuration. If you are using the commercial model (e.g., OpenAI, Google, Anthropic) as agent, ensure you have the necessary API keys set as environment variables (e.g., `OPENAI+API_KEY`, `GOOGLE_API_KEY`). The framework can **automatically resume** the experiment from unexpected termination, as long as you set the same experiment name in the experiment recipe config (`configs/recipe/base.py`). 126 | 127 | We provide several pre-defined agent configurations in the `configs/agents` directory, includes three widely used commercial APIs [Gemini](configs/agents/google), [Claude](configs/agents/anhthropic), and [ChatGPT](configs/agents/openai), as well open-source models supported by [LMDeploy](https://github.com/InternLM/lmdeploy). You can find the pre-set configurations in `configs/agents`, and modify them to customize the LVLM settings. 128 | 129 | You can customize the experiment settings by modifying the configuration file `configs/recipe/base.py`. 130 | 131 | ```python 132 | # configs/recipe/base.py 133 | name = 'standard' 134 | save_path = 'experiments' 135 | tasks = ['perceive', 'qa', 'rule', 'e2e'] 136 | games = ['tictactoe', 'reversi', 'gomoku', 'minesweeper', 'sudoku', 'chess'] 137 | ``` 138 | 139 | - Adjust `tasks` to include or exclude specific tasks. 140 | - Modify `games` to specify the games to evaluate. 141 | - Change `save_path` to specify the output directory. 142 | - Set `name` to identify the experiment. 143 | 144 | ## Evaluating Results 145 | 146 | Once you have run the experiments, the results will be saved in the `experiments` directory with a name specified in the experiment recipe. You can evaluate the results using: 147 | 148 | ```bash 149 | python evaluate.py --exp-path experiments/standard/gpt4o.json 150 | ``` 151 | 152 | Evaluation results will be saved by default in the `evaluation_results/` directory. 153 | 154 | ## Visualizing Results 155 | 156 | To visualize the evaluation results, generate a radar chart comparing LVLMs across tasks: 157 | 158 | ```bash 159 | python plot_radar.py 160 | ``` 161 | 162 | This will automatically create a radar chart (`radar_chart.pdf`) in the current directory, illustrating performance differences. 163 | 164 | To compare with the results in our paper, you can download the evaluation files from [here](https://universityofadelaide.box.com/s/tn398x5zyj5eq0e05atfcja40w1xdf83) and place them in the `evaluation_results` directory. This includes the evaluation results for [GPT-4o-240806](https://openai.com/index/gpt-4o-system-card/), [Gemini-1.5pro](https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/), [Claude-3.5-sonnet](https://www.anthropic.com/news/claude-3-5-sonnet), [Qwen2-vl-7b](https://huggingface.co/Qwen/Qwen2-VL-7B), [DeepSeek-vl-7b](https://huggingface.co/deepseek-ai/deepseek-vl-7b-base), [Phi3-vl](https://huggingface.co/microsoft/Phi-3-vision-128k-instruct), [LLaVA-1.6-7b](https://huggingface.co/liuhaotian/llava-v1.6-vicuna-7b), and [InternVL2-8b](https://huggingface.co/OpenGVLab/InternVL2-8B). 165 | 166 |  167 | 168 | ## Evaluating Customized Models 169 | 170 | To evaluate a customized LVLM, follow these steps: 171 | 172 | 1. **Implement Your Model**: 173 | 174 | Implement your model by inheriting the `BaseAgent` class in `agents/base_agent.py`, and registering it with `AGENT_REGISTRY`. You may use the following template: 175 | 176 | ```python 177 | from playground.agents import BaseAgent 178 | from playground.registry import AGENT_REGISTRY 179 | 180 | @AGENT_REGISTRY.register('custom_single') 181 | class CustomAgentSingleStep(BaseAgent): 182 | 183 | def __init__(self, agent_cfg): 184 | super().__init__(agent_cfg) 185 | # Initialize your model, API, or configuration here 186 | pass 187 | 188 | def get_decision(self, screenshot_path: str, prompt: str): 189 | # Implement logic to process the screenshot and prompt, return a decision 190 | pass 191 | ``` 192 | 193 | 2. **Configure Your Model in** `configs/agents`: 194 | 195 | Create or modify a configuration file (e.g., configs/agents/custom_agent.py) to define your model’s settings. Example: 196 | 197 | ```python 198 | lmm_agent = dict( 199 | agent='custom_single', 200 | model='your_model_name', 201 | max_tokens=512, 202 | image_size=512, 203 | backend_config=None, 204 | general_config=None, 205 | name='custom_agent' 206 | ) 207 | ``` 208 | 209 | Ensure the `agent` field matches the registered name in the `AGENT_REGISTRY`. After defining and configuring your model, follow the standard steps to run experiments (`python run.py`), evaluate results (`python evaluate.py`), and visualize performance (`python plot_radar.py`). 210 | 211 | ## Acknowledgements 212 | 213 | We acknowledge the authors of the following repositories for providing the game UIs and search-based AI implementations: 214 | 215 | - [Python-Chess](https://github.com/niklasf/python-chess) 216 | - [Moonsweeper](https://www.pythonguis.com/examples/python-minesweeper/) 217 | - [Sudoku-in-Python](https://github.com/humzah286/Sudoku-in-python) 218 | - [Gobang](https://github.com/sgsx11/Gobang) 219 | - [GUI-tic-tac-toe-AI](https://github.com/Erfan-ram/GUI-tic-tac-toe-Ai) 220 | - [reversi-minimax-algorithm](https://github.com/abkhan04/reversi-minimax-algorithm) 221 | 222 | 223 | ## Contact 224 | 225 | If you have any questions or suggestions, please feel free to open an issue or contact us via email Xinyu Wang [xinyu.wang02@adelaide.edu.au](mailto:xinyu.wang02@adelaide.edu.au). 226 | 227 | ## Citation 228 | 229 | If you find this repository useful for your research, please consider citing our paper: 230 | 231 | ```bibtex 232 | @inproceedings{wang2025large, 233 | title={Are Large Vision Language Models Good Game Players?}, 234 | author={Wang, Xinyu and Zhuang, Bohan and Wu, Qi}, 235 | booktitle={International Conference on Learning Representations}, 236 | year={2025} 237 | } 238 | ``` 239 | -------------------------------------------------------------------------------- /playground/games/minesweeper/minesweeper.py: -------------------------------------------------------------------------------- 1 | import random 2 | import re 3 | import time 4 | 5 | from PyQt5.QtGui import QIcon, QPainter, QPixmap 6 | from PyQt5.QtWidgets import QMainWindow 7 | 8 | from playground.games import BaseGame, BaseGameLogic 9 | from playground.games.minesweeper.game_cfg import LEVELS, STATUS_ICONS 10 | from playground.games.minesweeper.minesweeper_ui import MinesweeperUI 11 | from playground.registry import GAME_REGISTRY 12 | from playground.state_code import GameStatus 13 | 14 | 15 | class MinesweeperLogic(BaseGameLogic): 16 | """Pure logic for Minesweeper game.""" 17 | 18 | def __init__(self, game_cfg): 19 | self.game_cfg = game_cfg 20 | self.level = game_cfg.level 21 | self.b_size, self.n_mines = LEVELS[self.level] 22 | self.board = None 23 | self.status = GameStatus.IN_PROGRESS 24 | self.moves_history = [] 25 | self.timer_start = 0 26 | self.reset_board() 27 | 28 | def reset_board(self): 29 | """Reset the game board to initial state without revealing.""" 30 | self.board = [[-1 for _ in range(self.b_size)] 31 | for _ in range(self.b_size)] 32 | self.status = GameStatus.IN_PROGRESS 33 | self.moves_history = [] 34 | self.timer_start = int(time.time()) 35 | 36 | mine_positions = set() 37 | while len(mine_positions) < self.n_mines: 38 | x, y = random.randint(0, self.b_size - 1), random.randint( 39 | 0, self.b_size - 1) 40 | if (x, y) not in mine_positions: 41 | mine_positions.add((x, y)) 42 | self.board[y][x] = 9 43 | 44 | for y in range(self.b_size): 45 | for x in range(self.b_size): 46 | if self.board[y][x] != 9: 47 | self.board[y][x] = -1 48 | 49 | def _get_adjacency_n(self, x, y, mine_positions): 50 | """Calculate number of adjacent mines.""" 51 | count = 0 52 | for xi in range(max(0, x - 1), min(self.b_size, x + 2)): 53 | for yi in range(max(0, y - 1), min(self.b_size, y + 2)): 54 | if (xi, yi) in mine_positions: 55 | count += 1 56 | return count 57 | 58 | def _expand_reveal(self, x, y): 59 | """Reveal surrounding cells if no adjacent mines.""" 60 | if not (0 <= x < self.b_size 61 | and 0 <= y < self.b_size) or self.board[y][x] >= 0: 62 | return 63 | mine_positions = {(xi, yi) 64 | for yi in range(self.b_size) 65 | for xi in range(self.b_size) 66 | if self.board[yi][xi] in [9, 10]} 67 | self.board[y][x] = self._get_adjacency_n(x, y, mine_positions) 68 | if self.board[y][x] == 0: 69 | for xi in range(max(0, x - 1), min(self.b_size, x + 2)): 70 | for yi in range(max(0, y - 1), min(self.b_size, y + 2)): 71 | self._expand_reveal(xi, yi) 72 | 73 | def input_move(self, move): 74 | """Process move in format 'A1'.""" 75 | if self.status != GameStatus.IN_PROGRESS: 76 | return self.status 77 | pattern = re.compile(r'([a-zA-Z])([0-9]+)|([0-9]+)([a-zA-Z])') 78 | match = pattern.match(move) 79 | if not match: 80 | return GameStatus.INVALID_MOVE 81 | 82 | row, col = (match.group(1), 83 | match.group(2)) if match.group(1) else (match.group(4), 84 | match.group(3)) 85 | row = row.lower() 86 | if row.isalpha(): 87 | y = ord(row) - ord('a') 88 | x = int(col) - 1 89 | else: 90 | y = int(row) - 1 91 | x = ord(col.lower()) - ord('a') 92 | 93 | if not (0 <= x < self.b_size and 0 <= y < self.b_size): 94 | return GameStatus.INVALID_MOVE 95 | 96 | if 0 <= self.board[y][x] <= 8: 97 | return GameStatus.INVALID_MOVE 98 | 99 | self.moves_history.append(move) 100 | if self.board[y][x] == 9: 101 | self.board[y][x] = 10 102 | self.status = GameStatus.LOSE 103 | return self.status 104 | 105 | self._expand_reveal(x, y) 106 | self._check_win() 107 | return self.status 108 | 109 | def _check_win(self): 110 | """Check if only mines (10) remain unrevealed.""" 111 | unrevealed_count = sum(row.count(-1) for row in self.board) 112 | if unrevealed_count == self.n_mines: 113 | self.status = GameStatus.WIN 114 | 115 | def get_game_status(self): 116 | return self.status 117 | 118 | def get_random_state(self): 119 | """Generate a random game state with ~50% cells revealed.""" 120 | self.reset_board() 121 | game_state = [[-1 for _ in range(self.b_size)] 122 | for _ in range(self.b_size)] 123 | total_cells = self.b_size * self.b_size 124 | cells_to_reveal = total_cells // 2 + 1 125 | positions = [(x, y) for x in range(self.b_size) 126 | for y in range(self.b_size)] 127 | revealed_positions = random.sample(positions, cells_to_reveal) 128 | 129 | mine_positions = {(x, y) 130 | for y in range(self.b_size) 131 | for x in range(self.b_size) if self.board[y][x] == 9} 132 | for y in range(self.b_size): 133 | for x in range(self.b_size): 134 | if (x, y) in revealed_positions: 135 | if self.board[y][x] == 9: 136 | game_state[y][x] = 9 137 | else: 138 | game_state[y][x] = self._get_adjacency_n( 139 | x, y, mine_positions) 140 | else: 141 | game_state[y][x] = -1 142 | return game_state 143 | 144 | def get_rule_state(self): 145 | """Generate a rule state with valid movements.""" 146 | game_state = self.get_random_state() 147 | valid_movements = [] 148 | for y in range(self.b_size): 149 | for x in range(self.b_size): 150 | if game_state[y][x] == -1: 151 | pos_str = f"{chr(y + ord('A'))}{x + 1}" 152 | valid_movements.append(pos_str) 153 | return game_state, valid_movements 154 | 155 | def calculate_score(self): 156 | """Calculate score based on steps, revealed cells, and game outcome.""" 157 | step_score = len(self.moves_history) * 10 158 | reveal_bonus = sum(1 for y in range(self.b_size) 159 | for x in range(self.b_size) 160 | if 0 <= self.board[y][x] <= 8) * 2 161 | outcome_bonus = 1000 if self.status == GameStatus.WIN else 0 162 | total_score = step_score + reveal_bonus + outcome_bonus 163 | return total_score 164 | 165 | def parse_e2e(self, lmm_output): 166 | """Parse e2e output to a move.""" 167 | match = re.search(r'Movement:\s*([A-Ha-h][1-8]|[1-8][A-Ha-h])', 168 | lmm_output, re.IGNORECASE) 169 | if match: 170 | move = match.group(1).upper() 171 | if move[0].isdigit(): 172 | move = move[1] + move[0] 173 | return move 174 | return GameStatus.INVALID_MOVE 175 | 176 | 177 | class MinesweeperRenderer(QMainWindow): 178 | """Renderer for Minesweeper UI.""" 179 | 180 | def __init__(self, logic): 181 | super().__init__() 182 | self.logic = logic 183 | self.ui = MinesweeperUI(self, self.logic.b_size) 184 | self.setCentralWidget(self.ui.centralwidget) 185 | self._update_ui_from_logic() 186 | self.adjust_window_size() 187 | self.show() 188 | 189 | def adjust_window_size(self): 190 | window_width = 50 + self.logic.b_size * 20 191 | window_height = 100 + self.logic.b_size * 20 192 | self.setFixedSize(window_width, window_height) 193 | 194 | def _update_ui_from_logic(self): 195 | """Sync UI with logic state.""" 196 | self.ui.minesLabel.setText(f'{self.logic.n_mines:03d}') 197 | elapsed = int( 198 | time.time() 199 | ) - self.logic.timer_start if self.logic.status == GameStatus.IN_PROGRESS else 0 # noqa 200 | self.ui.clockLabel.setText(f'{elapsed:03d}') 201 | self.ui.statusButton.setIcon(QIcon(STATUS_ICONS[self.logic.status])) 202 | for y in range(self.logic.b_size): 203 | for x in range(self.logic.b_size): 204 | widget = self.ui.gameGrid.itemAtPosition(y + 1, x + 1).widget() 205 | widget.is_mine = (self.logic.board[y][x] in [9, 10]) 206 | widget.is_revealed = (self.logic.board[y][x] >= 0 207 | and self.logic.board[y][x] != 9 208 | ) or self.logic.board[y][x] == 10 209 | widget.adjacent_n = self.logic.board[y][ 210 | x] if widget.is_revealed and not widget.is_mine else 0 211 | widget.update() 212 | 213 | def get_screenshot(self): 214 | """Generate screenshot of the entire window.""" 215 | self._update_ui_from_logic() 216 | window_width = self.width() 217 | window_height = self.height() 218 | screenshot = QPixmap(window_width, window_height) 219 | painter = QPainter(screenshot) 220 | painter.setRenderHint(QPainter.Antialiasing) 221 | self.render(painter) 222 | painter.end() 223 | return screenshot 224 | 225 | 226 | @GAME_REGISTRY.register('minesweeper') 227 | class MineSweeper(BaseGame): 228 | AI_component = False 229 | 230 | def __init__(self, game_cfg): 231 | super().__init__(game_cfg) 232 | self.logic = MinesweeperLogic(game_cfg) 233 | self.renderer = None 234 | 235 | def get_screenshot(self): 236 | if self.renderer is None: 237 | self.renderer = MinesweeperRenderer(self.logic) 238 | return self.renderer.get_screenshot() 239 | 240 | def input_move(self, move): 241 | return self.logic.input_move(move) 242 | 243 | def get_game_status(self): 244 | return self.logic.get_game_status() 245 | 246 | def get_random_state(self): 247 | return self.logic.get_random_state() 248 | 249 | def get_rule_state(self): 250 | return self.logic.get_rule_state() 251 | 252 | def ai_move(self): 253 | return None 254 | 255 | def calculate_score(self): 256 | return self.logic.calculate_score() 257 | 258 | def parse_e2e(self, lmm_output): 259 | return self.logic.parse_e2e(lmm_output) 260 | --------------------------------------------------------------------------------