├── .github
└── workflows
│ └── deploy-docs.yml
├── .gitignore
├── LICENSE
├── README.md
├── assets
└── sb-cli-logo.png
├── docs
├── assets
│ ├── icon.svg
│ ├── logo.svg
│ ├── mini_logo.svg
│ ├── mini_logo_text_below.svg
│ ├── sbcli_logo_text_below.svg
│ ├── sweagent_logo.svg
│ ├── sweagent_logo_text_below.svg
│ ├── swebench_logo.png
│ ├── swebench_logo_text_below.svg
│ ├── swerex_logo.svg
│ ├── swerex_logo_text_below.svg
│ ├── swesmith_logo.png
│ └── swesmith_logo_text_below.svg
├── authentication.md
├── css
│ ├── bubbles.css
│ └── custom.css
├── index.md
├── installation.md
├── overrides
│ └── main.html
├── quick-start.md
├── submit-to-leaderboard.md
└── user-guide
│ ├── delete-run.md
│ ├── get-quotas.md
│ ├── get-report.md
│ ├── index.md
│ ├── list-runs.md
│ └── submit.md
├── mkdocs.yaml
├── pyproject.toml
└── sb_cli
├── __init__.py
├── config.py
├── delete_run.py
├── gen_api_key.py
├── get_quotas.py
├── get_report.py
├── list_runs.py
├── submit.py
├── utils.py
└── verify_api_key.py
/.github/workflows/deploy-docs.yml:
--------------------------------------------------------------------------------
1 | name: Deploy MkDocs
2 | on:
3 | push:
4 | branches:
5 | - main
6 | paths:
7 | - 'docs/**'
8 | - 'mkdocs.yaml'
9 | - '.github/workflows/deploy-docs.yml'
10 |
11 |
12 | # Prevent concurrent runs that could conflict when pushing to gh-pages
13 | concurrency:
14 | group: build-docs-${{ github.ref }}
15 | cancel-in-progress: false
16 |
17 | permissions:
18 | contents: write
19 |
20 | jobs:
21 | deploy:
22 | runs-on: ubuntu-latest
23 | steps:
24 | - uses: actions/checkout@v4
25 |
26 | - uses: actions/setup-python@v4
27 | with:
28 | python-version: '3.x'
29 |
30 | - name: Install dependencies
31 | run: |
32 | pip install mkdocs-material
33 | pip install pillow cairosvg # for social cards
34 | pip install mkdocs-glightbox # for your glightbox plugin
35 |
36 | - name: Deploy docs
37 | run: mkdocs gh-deploy --force
--------------------------------------------------------------------------------
/.gitignore:
--------------------------------------------------------------------------------
1 | site/
2 | __pycache__/
3 | *.log
4 | *.json
5 | *.csv
6 | *.jsonl
7 | *.json-[0-9]*
8 |
9 | # Python
10 | *.py[cod]
11 | *.so
12 | .Python
13 | env/
14 | build/
15 | develop-eggs/
16 | dist/
17 | downloads/
18 | eggs/
19 | .eggs/
20 | lib/
21 | lib64/
22 | parts/
23 | sdist/
24 | var/
25 | *.egg-info/
26 | .installed.cfg
27 | *.egg
28 |
29 | # IDEs and editors
30 | .idea/
31 | .vscode/
32 | .cursor/
33 | *.swp
34 | *.swo
35 | .DS_Store
36 | Thumbs.db
37 |
38 | # Virtual Environment
39 | venv/
40 | ENV/
41 | .env
42 | .venv
43 |
44 | # Testing
45 | .coverage
46 | htmlcov/
47 | .tox/
48 | .pytest_cache/
49 | nosetests.xml
50 | coverage.xml
51 |
--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
1 | MIT License
2 |
3 | Copyright (c) 2024 SWE-bench
4 |
5 | Permission is hereby granted, free of charge, to any person obtaining a copy
6 | of this software and associated documentation files (the "Software"), to deal
7 | in the Software without restriction, including without limitation the rights
8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9 | copies of the Software, and to permit persons to whom the Software is
10 | furnished to do so, subject to the following conditions:
11 |
12 | The above copyright notice and this permission notice shall be included in all
13 | copies or substantial portions of the Software.
14 |
15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21 | SOFTWARE.
22 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 |
2 | 
3 |
4 |
5 | # SWE-bench CLI
6 |
7 | A command-line interface for interacting with the SWE-bench API. Use this tool to submit predictions, manage runs, and retrieve evaluation reports.
8 |
9 | Read the full documentation [here](https://www.swebench.com/sb-cli/). For submission guidelines, see [here](https://swebench.com/sb-cli/submit-to-leaderboard).
10 |
11 | ## Installation
12 |
13 | ```bash
14 | pip install sb-cli
15 | ```
16 |
17 | ## Authentication
18 |
19 | Before using the CLI, you'll need to get an API key:
20 |
21 | 1. Generate an API key:
22 | ```bash
23 | sb-cli gen-api-key your.email@example.com
24 | ```
25 |
26 | 2. Set your API key as an environment variable - and store it somewhere safe!
27 | ```bash
28 | export SWEBENCH_API_KEY=your_api_key
29 | # or add export SWEBENCH_API_KEY=your_api_key to your .*rc file
30 | ```
31 |
32 | 3. You'll receive an email with a verification code. Verify your API key:
33 | ```bash
34 | sb-cli verify-api-key YOUR_VERIFICATION_CODE
35 | ```
36 |
37 | ## Subsets and Splits
38 |
39 | SWE-bench has different subsets and splits available:
40 |
41 | ### Subsets
42 | - `swe-bench-m`: The SWE-bench Multimodal dataset
43 | - `swe-bench_verified`: 500 verified problems from SWE-bench [Learn more](https://openai.com/index/introducing-swe-bench-verified/)
44 | - `swe-bench_lite`: A subset of the original SWE-bench for testing
45 |
46 |
47 | ### Splits
48 | - `dev`: Development/validation split
49 | - `test`: Test split (currently only available for `swe-bench_lite` and `swe-bench_verified`)
50 |
51 | You'll need to specify both a subset and split for most commands.
52 |
53 | ## Usage
54 |
55 | ### Submit Predictions
56 |
57 | Submit your model's predictions to SWE-bench:
58 |
59 | ```bash
60 | sb-cli submit swe-bench-m test \
61 | --predictions_path predictions.json \
62 | --run_id my_run_id
63 | ```
64 |
65 | Options:
66 | - `--run_id`: ID of the run to submit predictions for (optional, defaults to the name of the parent directory of the predictions file)
67 | - `--instance_ids`: Comma-separated list of specific instance IDs to submit (optional)
68 | - `--output_dir`: Directory to save report files (default: sb-cli-reports)
69 | - `--overwrite`: Overwrite existing report (default: 0)
70 | - `--gen_report`: Generate a report after evaluation is complete (default: 1)
71 |
72 | ### Get Report
73 |
74 | Retrieve evaluation results for a specific run:
75 |
76 | ```bash
77 | sb-cli get-report swe-bench-m dev my_run_id -o ./reports
78 | ```
79 |
80 | ### List Runs
81 |
82 | View all your existing run IDs for a specific subset and split:
83 |
84 | ```bash
85 | sb-cli list-runs swe-bench-m dev
86 | ```
87 |
88 | ## Predictions File Format
89 |
90 | Your predictions file should be a JSON file in one of these formats:
91 |
92 | ```json
93 | {
94 | "instance_id_1": {
95 | "model_patch": "...",
96 | "model_name_or_path": "..."
97 | },
98 | "instance_id_2": {
99 | "model_patch": "...",
100 | "model_name_or_path": "..."
101 | }
102 | }
103 | ```
104 |
105 | Or as a list:
106 |
107 | ```json
108 | [
109 | {
110 | "instance_id": "instance_id_1",
111 | "model_patch": "...",
112 | "model_name_or_path": "..."
113 | },
114 | {
115 | "instance_id": "instance_id_2",
116 | "model_patch": "...",
117 | "model_name_or_path": "..."
118 | }
119 | ]
120 | ```
121 |
122 | ## Submitting to the Multimodal Leaderboard
123 |
124 | To submit your system to the [SWE-bench Multimodal](https://www.swebench.com/multimodal) leaderboard:
125 |
126 | 1. Submit your predictions for the `swe-bench-m` / `test` split using the CLI
127 | 2. Fork the [experiments repository](https://github.com/swe-bench/experiments)
128 | 3. Add your submission files under `experiments/multimodal/YOUR_MODEL_NAME/`
129 | 4. Create a PR with your submission
130 |
131 | See the detailed guide in our [submission documentation](https://swe-bench.com/sb-cli/submit-to-leaderboard).
132 |
133 | Note: Check your test split quota using `sb-cli quota swe-bench-m test` before submitting.
134 |
135 | ## Related projects
136 |
137 |
138 |
139 |
140 |
141 |
142 |
143 |
144 |
145 |
146 |
147 |
148 |
4 |
5 |
13 | SWE-bench
14 |
15 |
16 | 
17 | SWE-smith
18 |
19 |
20 | 
21 | SWE-agent
22 |
23 |
24 | 
25 | Mini-SWE-Agent
26 |
27 |
28 | 
29 | SWE-ReX
30 |
31 |