├── README.md
└── docs
    └── logo.png


/README.md:
--------------------------------------------------------------------------------
  1 | # RunBugRun
  2 | <p align="center">
  3 |   <img src="docs/logo.png">
  4 | </p>
  5 | 
  6 | > [!NOTE]  
  7 | > This is the revised and extended version of RunBugRun. The older version can be found in the `legacy` branch.
  8 | 
  9 | ## What is RunBugRun
 10 | 
 11 | RunBugRun is an [APR](http://program-repair.org/) dataset of over 700'000 executable buggy/fixed pairs of short programs taken from [IBM Project CodeNet](https://github.com/IBM/Project_CodeNet) written in 9 languages (C++, C, Python, Java, Ruby, JavaScript, Go, PHP, C#).
 12 | 
 13 | It can be used to evaluate APR tools, that is, tools that automatically find and repair bugs in source code.
 14 | 
 15 | RunBugRun comes with tests, bug labels and infrastructure to execute programs. In order to warrant safe execution it uses [Bubblewrap](https://github.com/containers/bubblewrap) as a sandbox.
 16 | 
 17 | RunBugRun has pre-defined training, validation and test sets. APR tools can use the training set as they please. For evaluation, they are given a test set of buggy programs that do not pass all tests. A tool's performance is measured as the percentage of programs that the tool can fix in such a way that it passes all tests.
 18 | 
 19 | 
 20 | # Obtaining the Dataset
 21 | 
 22 | RunBugRun is distributed as a lrzip-compressed SQLite3 database dump. After downloading the compressed dump (see Releases page of this project) the following steps are necessary to restore the database.
 23 | 
 24 | ### Install lrzip and sqlite
 25 | 
 26 | ```bash
 27 | sudo apt-get install lrzip sqlite3
 28 | ```
 29 | 
 30 | ### Decompress
 31 | 
 32 | ```bash
 33 | lrunzip -z runbugrun.sql.lrz
 34 | ```
 35 | 
 36 | ### Restore
 37 | 
 38 | ```bash
 39 | sqlite3 runbugrun.db < runbugrun.sql
 40 | ```
 41 | 
 42 | # Data Sources
 43 | 
 44 | RunBugRun is a curated collection of data.
 45 | We used the following sources. For terms of use/license information please consult the corresponding project/website.
 46 | 
 47 | | Source | URL | Data |
 48 | |--------|------|--------|
 49 | | IBM CodeNet|  https://github.com/IBM/Project_CodeNet | Code submissions |
 50 | | AlphaCode/CodeContests | https://github.com/google-deepmind/code_contests | Tests |
 51 | | AtCoder | https://atcoder.jp/posts/21 | Tests |
 52 | | PIE4Perf |  https://github.com/madaan/pie-perf| Problem description translations (e.g., Japanese to English) |
 53 | 
 54 | # Database Schema Documentation
 55 | 
 56 | RunBugRun is distributed as a SQLite database, containing code, tests and metadata. It contains the following tables:
 57 | 
 58 | ## `tests`
 59 | Stores test cases.
 60 | 
 61 | | Column | Type | Description |
 62 | |--------|------------|-------------|
 63 | | `id` | integer | Primary key |
 64 | | `problem_id` | varchar | Reference to `problems.problem_id` |
 65 | | `test_id` | integer | Test identifier |
 66 | | `input` | text | Test input  |
 67 | | `output` | text | Expected output  |
 68 | | `created_at` | datetime(6) | Creation timestamp |
 69 | | `updated_at` | datetime(6) | Last update timestamp |
 70 | | `origin` | integer | Origin of the test (0: unknown, 1: codenet, 2: manual, 3: alphacode, 4: atcoder) (default: 0) |
 71 | | `active` | boolean | Whether test is active (default: true) |
 72 | 
 73 | ## `problems`
 74 | Stores problem descriptions.
 75 | 
 76 | | Column | Type | Description |
 77 | |--------|--------|-------------|
 78 | | `id` | integer | Primary key |
 79 | | `problem_id` | varchar | IBM CodeNet problem identifier  |
 80 | | `text` | text | Problem description  |
 81 | | `similar_problems` | jsonb | JSON array of similar problems  |
 82 | 
 83 | ## `bugs`
 84 | Stores bugs (full code).
 85 | 
 86 | | Column | Type | Description |
 87 | |--------|------------|-------------|
 88 | | `id` | integer | Primary key |
 89 | | `buggy_code` | text | The buggy code |
 90 | | `fixed_code` | text | The fixed code |
 91 | | `problem_id` | varchar | Reference to `problems.problem_id` |
 92 | | `user_id` | varchar | ID of user who `submitted` |
 93 | | `buggy_submission_id` | varchar | ID of buggy submission (IBM CodeNet ID) |
 94 | | `fixed_submission_id` | varchar | ID of fixed submission (IBM CodeNet ID) |
 95 | | `language` | integer | Programming language (0: c, 1: cpp, 2: javascript, 3: java, 4: ruby, 5: python, 6: php, 7: go, 8: c_sharp) |
 96 | | `label_ids` | json | JSON array of label IDs  |
 97 | | `runtime_errors` | json | Runtime errors  |
 98 | | `change_count` | integer | Number of changes  |
 99 | | `split` | integer | Data split (0: train, 1: valid, 2: test, 3: unfiltered)  |
100 | | `buggy_main_class` | varchar | Main class for buggy code (Java only) |
101 | | `fixed_main_class` | varchar | Main class for fixed code (Java only) |
102 | | `created_at` | datetime(6) | Creation timestamp |
103 | | `updated_at` | datetime(6) | Last update timestamp |
104 | | `token_count` | integer | Number of tokens  |
105 | | `active` | boolean | Whether bug is active (default: true) |
106 | | `hunk_count` | integer | Number of hunks  |
107 | | `buggy_locs` | integer | Number of lines of code in the buggy version (logical)  |
108 | | `fixed_locs` | integer | Number of lines of code in the fixed version (logical)  |
109 | 
110 | 
111 | ## `evaluations`
112 | Tracks evaluation runs (initially empty).
113 | 
114 | | Column | Type | Description |
115 | |--------|------------|-------------|
116 | | `id` | integer | Primary key |
117 | | `name` | varchar | Name of the evaluation |
118 | | `started_at` | datetime(6) | When the evaluation started |
119 | | `ended_at` | datetime(6) | When the evaluation ended  |
120 | | `created_at` | datetime(6) | Creation timestamp |
121 | | `updated_at` | datetime(6) | Last update timestamp |
122 | 
123 | ## `runs`
124 | Records individual test runs (initially empty).
125 | 
126 | | Column | Type | Description |
127 | |--------|------------|-------------|
128 | | `id` | integer | Primary key |
129 | | `evaluation_id` | integer | Reference to `evaluations.id` |
130 | | `status` | integer | Status of the run (0: pass, 1: fail, 2: error, 3: timeout, 4: compilation_error) |
131 | | `bug_id` | integer | Reference to `bugs.id` |
132 | | `bug_version` | integer | Version of the bug (0: buggy, 1: fixed, 2: candidate) |
133 | | `bug_variant` | integer | Variant of the bug (0: default) |
134 | | `candidate_index` | integer | Index of candidate  |
135 | | `test_id` | integer | Reference to tests.id |
136 | | `error_output` | text | Error output if any  |
137 | | `output` | text | Program output  |
138 | | `created_at` | datetime(6) | Creation timestamp |
139 | | `updated_at` | datetime(6) | Last update timestamp |
140 | | `wall_time` | decimal | Execution time in seconds  |


--------------------------------------------------------------------------------
/docs/logo.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/giganticode/run_bug_run/bbac70b7ae7331d87892e861356cf133476bc938/docs/logo.png


--------------------------------------------------------------------------------