├── .gitmodules └── README.md /.gitmodules: -------------------------------------------------------------------------------- 1 | [submodule "hardware-bugbase"] 2 | path = hardware-bugbase 3 | url = https://github.com/efeslab/hardware-bugbase 4 | 5 | [submodule "veripass"] 6 | path = veripass 7 | url = https://github.com/efeslab/veripass 8 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Debugging in the Brave New World of Reconfigurable Hardware (Artifact) 2 | 3 | This artifact includes 20 hardware bugs, each of them can be reproduced with Verilator in a push-button manner. It also includes the five tools we designed to help bug localization (i.e., SignalCat, FSM Monitor, Statistics Monitor, Dependency Monitor, and LossCheck), as well as examples of using these tools and the instructions of reproducing the figures in the paper. 4 | 5 | The full list of 68 bugs we studied can be found [here](https://docs.google.com/spreadsheets/d/1GonADjkm878iRs2noQFXW5AidY4KiiJfTPJyIq-3Z4I/edit?usp=sharing). 6 | 7 | If you have an interesting bug that you can reproduce, feel free to submit a pull request and we will add it to this repo. If you notice a bug that's not reproducible but still want to share to others, you may request edit access to the spreadsheet and add it there. 8 | 9 | ## Table of Contents 10 | 11 | - [0. Downloading the Repository](#0-downloading-the-repository) 12 | 13 | - [1. Reproducible Bugs](#1-reproducible-bugs) 14 | - [1.1 Installation](#11-installation) 15 | - [1.2 Reproducing Bugs with Verilator](#12-table-of-bugs) 16 | - [2. Debugging Tools](#2-debugging-tools) 17 | - [2.1 Installation](#21-installation) 18 | - [2.2 SignalCat and the Monitors](#22-signalcat-and-monitors) 19 | - [2.2.1 Debugging Logs with SignalCat and the Monitors](#221-debugging-logs-with-signalcat-and-the-monitors) 20 | - [2.2.2 Reproducing the Resource Overhead](#222-reproducing-the-resource-overhead) 21 | - [2.3 LossCheck](#23-losscheck) 22 | - [2.3.1 Data Loss Localization for the 6 Data Loss Bugs](#231-data-loss-localization-for-the-6-data-loss-bugs) 23 | - [2.3.2 Reproducing the Resource Overhead](#232-reproducing-the-resource-overhead) 24 | - [3. Licenses and Terms](#3-licenses-and-terms) 25 | 26 | ## 0. Downloading the Repository 27 | 28 | Use the following command to download the artifact repository: 29 | 30 | ```bash 31 | git clone --recursive https://github.com/efeslab/asplos22-hardware-debugging-artifact 32 | ``` 33 | 34 | After this command, you are expected to see the following directory hierarchy: 35 | 36 | ```bash 37 | asplos22-hardware-debugging-artifact 38 | ├── hardware-bugbase 39 | │   ├── c1-dead-lock-sdspi 40 | │   ├── c2-producer-consumer-mismatch-optimus 41 | │   ├── c3-signal-asynchrony-sdspi 42 | │   ├── c4-signal-asynchrony-axi-stream-fifo 43 | │   ├── common 44 | │   ├── d10-failure-to-update-sha512 45 | │   ├── d11-failure-to-update-frame-fifo 46 | │   ├── d12-failure-to-update-frame-fifo 47 | │   ├── d13-failure-to-update-frame-len 48 | │   ├── d1-buffer-overflow-rsd 49 | │   ├── d2-buffer-overflow-grayscale 50 | │   ├── d3-buffer-overflow-optimus 51 | │   ├── d4-buffer-overflow-frame-buffer 52 | │   ├── d5-bit-truncation-sha512 53 | │   ├── d6-bit-truncation-fft 54 | │   ├── d7-misindexing-fadd 55 | │   ├── d8-misindexing-axis-switch 56 | │   ├── d9-endianness-mismatch-sdspi 57 | │   ├── manual_debug_log 58 | │   ├── n1-frame-len-failure-to-update 59 | │   ├── n3-frame-fifo-fail-to-update 60 | │   ├── n8-axis-adapter-incomplete-implementation 61 | │   ├── n9-frame-fifo-failure-to-update 62 | │   ├── s1-protocol-violation-axi-lite 63 | │   ├── s2-protocol-violation-axi-stream 64 | │   ├── s3-incomplete-implementation-axis-adapter 65 | │   └── scripts 66 | └── veripass 67 | ├── dbgtools 68 | ├── model 69 | ├── passes 70 | ├── Pyverilog 71 | ├── recording 72 | ├── utils 73 | └── verilator 74 | ``` 75 | 76 | ## 1. Reproducible Bugs 77 | 78 | The `hardware-bugbase` directory contains all the reproducible bugs. Each bug is located in a directory, together with a simplified code snippet that helps understanding. 79 | 80 | ### 1.1 Installation 81 | 82 | You will need to install compile `Verilator` to reproduce these bugs. `Verilator` is located under the `veripass` directory. 83 | 84 | Before compilation, you will need to install a few dependencies: 85 | 86 | ```bash 87 | sudo apt-get install perl python3 make autoconf g++ flex bison ccache 88 | sudo apt-get install libgoogle-perftools-dev numactl perl-doc 89 | sudo apt-get install libfl2 libfl-dev # Ubuntu only (ignore if gives error) 90 | sudo apt-get install zlibc zlib1g zlib1g-dev # Ubuntu only (ignore if gives error) 91 | ``` 92 | 93 | Then compile `Verilator`: 94 | 95 | ```bash 96 | cd asplos22-hardware-debugging-artifact/veripass/verilator 97 | autoconf 98 | ./configure 99 | make -j8 100 | ``` 101 | 102 | Compilation is enough. You do not need to install it. Scripts in the bug database will find the location of `Verilator` themselves. 103 | 104 | ### 1.2 Reproducing Bugs with Verilator 105 | 106 | Bugs are listed in the table below. You may `cd` into the directory of each bug to reproduce it and read its documentations. 107 | 108 | To reproduce a specific bug: 109 | 110 | ```bash 111 | cd asplos22-hardware-debugging-artifact/hardware-bugbase/ 112 | make -j8 # compile the verilog code for simulation 113 | make sim # run the simulation 114 | make wave # open the generated waveform with GTKWave 115 | ``` 116 | 117 | You are expected to see an error message after `make sim`. `make wave` requires you using GUI, or the `DISPLAY` environment variable being set correctly. After simulation, you can also find a `.fst` file or a `.vcd` file under the directory. These are the waveforms generated by `Verilator`. You can copy the file to another computer and open it with `GTKWave` or other waveform-viewing software. 118 | 119 | | Bug ID | Bug Name | 120 | | ------ | ------------------------------------------------------------ | 121 | | D1 | [Buffer Overflow - RSD](https://github.com/efeslab/hardware-bugbase/tree/bugs/d1-buffer-overflow-rsd) | 122 | | D2 | [Buffer Overflow - Grayscale](https://github.com/efeslab/hardware-bugbase/tree/bugs/d2-buffer-overflow-grayscale) | 123 | | D3 | [Buffer Overflow - Optimus](https://github.com/efeslab/hardware-bugbase/tree/bugs/d3-buffer-overflow-optimus) | 124 | | D4 | [Buffer Overflow - Frame FIFO](https://github.com/efeslab/hardware-bugbase/tree/bugs/d4-buffer-overflow-frame-buffer) | 125 | | D5 | [Bit Truncation - SHA512](https://github.com/efeslab/hardware-bugbase/tree/bugs/d5-bit-truncation-sha512) | 126 | | D6 | [Bit Truncation - FFT](https://github.com/efeslab/hardware-bugbase/tree/bugs/d6-bit-truncation-fft) | 127 | | D7 | [Misindexing - FADD](https://github.com/efeslab/hardware-bugbase/tree/bugs/d7-misindexing-fadd) | 128 | | D8 | [Misindexing - AXI-Stream Switch](https://github.com/efeslab/hardware-bugbase/tree/bugs/d8-misindexing-axis-switch) | 129 | | D9 | [Endianness Mismatch - SDSPI](https://github.com/efeslab/hardware-bugbase/tree/bugs/d9-endianness-mismatch-sdspi) | 130 | | D10 | [Failure-to-Update - SHA512](https://github.com/efeslab/hardware-bugbase/tree/bugs/d10-failure-to-update-sha512) | 131 | | D11 | [Failure-to-Update - Frame FIFO](https://github.com/efeslab/hardware-bugbase/tree/bugs/d11-failure-to-update-frame-fifo) | 132 | | D12 | [Failure-to-Update - Frame FIFO](https://github.com/efeslab/hardware-bugbase/tree/bugs/d12-failure-to-update-frame-fifo) | 133 | | D13 | [Failure-to-Update - Frame Length Measurer](https://github.com/efeslab/hardware-bugbase/tree/bugs/d13-failure-to-update-frame-len) | 134 | | C1 | [Dead Lock - SDSPI](https://github.com/efeslab/hardware-bugbase/tree/bugs/c1-dead-lock-sdspi) | 135 | | C2 | [Producer-Consumer Mismatch - Optimus](https://github.com/efeslab/hardware-bugbase/tree/bugs/c2-producer-consumer-mismatch-optimus) | 136 | | C3 | [Signal Asynchrony - SDSPI](https://github.com/efeslab/hardware-bugbase/tree/bugs/c3-signal-asynchrony-sdspi) | 137 | | C4 | [Signal Asynchrony - AXI-Stream FIFO](https://github.com/efeslab/hardware-bugbase/tree/bugs/c4-signal-asynchrony-axi-stream-fifo) | 138 | | S1 | [Protocol Violation - AXI-Lite](https://github.com/efeslab/hardware-bugbase/tree/bugs/s1-protocol-violation-axi-lite) | 139 | | S1 | [Protocol Violation - AXI-Stream](https://github.com/efeslab/hardware-bugbase/tree/bugs/s2-protocol-violation-axi-stream) | 140 | | S3 | [Incomplete Implementation - AXI-Stream Adapter](https://github.com/efeslab/hardware-bugbase/tree/bugs/s3-incomplete-implementation-axis-adapter) | 141 | 142 | ## 2. Debugging Tools 143 | 144 | Our debugging tools locate in the `veripass` directory. In the `hardware-bugbase` directory, we provide `make` scripts to invoke these debugging tools. 145 | 146 | **Warning: A full evaluation of this part takes days, because FPGA synthesis is slow (e.g., up to several hours per-run). We encourage you to evaluate the non-synthesis part (e.g., [2.3.1](#231-data-loss-localization-for-the-6-data-loss-bugs)) first.** 147 | 148 | ### 2.1 Installation 149 | 150 | To run the debugging tools, you will need to compile `Verilator` and `Pyverilog` if you have not done so already: 151 | 152 | ```bash 153 | cd asplos22-hardware-debugging-artifact/veripass 154 | make -j8 155 | ``` 156 | 157 | And install the following python packages: 158 | 159 | ```bash 160 | pip3 install jinja2 sympy ply gephistreamer 161 | ``` 162 | 163 | And add the following lines to your `.bashrc` or `.zshrc` to help the scripts find `Vivado`, `Quartus`, and `VCS`. `Vivado` must be the `Design Suite` edition, `Quartus` must be the `Pro` edition with version `17.0`, and `VCS` must be the `MX` edition. 164 | 165 | ```bash 166 | # Quartus Pro 167 | export QUARTUS_HOME=/17.0/quartus 168 | export PATH=$QUARTUS_HOME/bin:$PATH 169 | export LM_LICENSE_FILE= 170 | 171 | # Vivado 172 | export XILINX_VIVADO=/Vivado/2020.2 173 | export PATH=$XILINX_VIVADO/bin:$PATH 174 | export XILINXD_LICENSE_FILE= 175 | 176 | # VCS MX 177 | export VCS_HOME= 178 | export PATH=$VCS_HOME/bin:$PATH 179 | export SNPSLMD_LICENSE_FILE= 180 | ``` 181 | 182 | In order to synthesize projects for Intel HARP, you will need to download a supported version of Intel FPGA Basic Building Blocks, a set of platform files for HARP, and have the following additional lines in `.bashrc` or `.zshrc`. You can ask your Intel contact for `BBS_6.4.0`. You may want to read [this](https://www.intel.com/content/dam/www/programmable/us/en/pdfs/literature/manual/mnl-ias-ccip.pdf) to understand the interface of the HARP platform. It is theoretically possible to compile these HARP projects for the PAC platform (which is more widely available); however, we did not evaluate it. 183 | 184 | ```bash 185 | export OPAE_PLATFORM_ROOT=/BBS_6.4.0 186 | export PATH=$OPAE_PLATFORM_ROOT/bin:$PATH 187 | ``` 188 | 189 | The original framework for HARP simulation requires Python 2 as the default `python` command. As a result, you may need to set up a `virtualenv` with the following command: 190 | 191 | ```bash 192 | virtualenv --python=/usr/bin/python2 193 | ``` 194 | 195 | ### 2.2 SignalCat and the Monitors 196 | 197 | #### 2.2.1 Debugging Logs with SignalCat and the Monitors 198 | 199 | In Section 6.2 of the paper, we demonstrated that a developer can use SignalCat and the Monitors to localize all the 20 bugs in this artifact. We provide the mental debugging logs of a developer localizing these bugs in [this sheet](https://github.com/efeslab/hardware-bugbase/blob/f3f391be341e2f36462dd94bba98b8bffa81c34b/manual_debug_log/manual_debugging_agreements.xlsx). For each bug, the sheet includes the tools the developer would use at each step. The configurations for invoking these tools are located in a `.cfg` file under each bug's directory; you can invoke the tools using the following commands under each bug's directory: 200 | 201 | ```bash 202 | make withtask.v 203 | ``` 204 | 205 | After running this command, a file called `withtask.v` will be generated. This file contains the flattened verilog code with the debugging instrumentations described in the configuration. 206 | 207 | #### 2.2.2 Reproducing the Resource Overhead 208 | 209 | To synthesize the instrumented circuit, you may run the following command: 210 | 211 | ```bash 212 | source /bin/activate # switch to a python virtualenv where python2 is the default 213 | make sweep_depth 214 | ``` 215 | 216 | This command will generate a number of files (e.g., instrumented circuit with different buffer size, the TCL scripts to invoke synthesis, etc) and invoke the synthesis script for the circuit and run syntheses with different recording buffer size. This command would froze for a long time, because each synthesis takes hours. 217 | 218 | After the command finishes, you can run the following command to report resource utilization. 219 | 220 | ```bash 221 | make report_depth_sweep 222 | ``` 223 | 224 | For `D4`, `D6`, `D7`, `D8`, `D9`, `D11`, `D12`, `D13`, `C1`, `C3`, `C4`, `S1`, `S2`, and `S3`, you will see something like the following. 225 | 226 | ``` 227 | log2(Depth),10,11,12,13 228 | Total LUTs,2225,2208,2191,2287 229 | FFs,2870,2881,2892,2905 230 | RAMB36,4,7,15,30 231 | RAMB18,0,1,0,0 232 | 233 | build_notask: Total LUTs,FFs,RAMB36,RAMB18 234 | 858;516;0;0 235 | ``` 236 | 237 | The upper block shows the resource utilization of instrumented circuit, and the bottom block shows the resource utilization of the uninstrumented circuit. In the paper, we use the word `Logic` for `LUT`, `Register` for `FF`, and calculate the total number of bits from `RAM36` (36Kbit per instance) and `RAM18` (18Kbit per instance). In the above example, the register overhead of an instrumented circuit with a `1024`-depth buffer is `2870-517=2354`. 238 | 239 | For `D1`, `D2`, `D3`, `D5`, `D10`, and `C2`, you will see something like the following. We use the `Logic` for `ALM`, `Register` for `FF`, and use the number of `BRAM Blocks` to calculate BRAM size (each block contains 20Kbits). 240 | 241 | ``` 242 | log2(Depth),10,11,12,13 243 | ALM,101170,101173,101185,101191 244 | BRAM#B,326,343,376,477 245 | BRAMbit,3989920,4332960,5019040,6391200 246 | FFs,111356,111371,111397,111447 247 | 248 | build_notask: ALM BRAM#B BRAMbit FFs 249 | 100245;309;3646880;108734 250 | ``` 251 | 252 | ### 2.3 LossCheck 253 | 254 | Bug `D1`, `D2`, `D3`, `D4`, `C2`, and `C4` are the six data loss bugs that can be localized by LossCheck. 255 | 256 | #### 2.3.1 Data Loss Localization for the 6 Data Loss Bugs 257 | 258 | You can use the following command to invoke LossCheck under the directories of these four bugs. 259 | 260 | ```bash 261 | make -f Makefile.lc 262 | ``` 263 | 264 | For `D1`, `D2`, `D3`, and `C4`: 265 | This will generate two `.v` files (e.g., a `.losscheck.0.v` and a `.losscheck.1.v`). `.losscheck.0.v` is the first instrumentation, which does not filter false positives (as discussed in Section 4.5.3). Our scripts run the original testbench of the circuit on the first instrumentation, and generate a list of signals that should be filtered out (i.e., storing in `filter.txt`). Then, our scripts invoke LossCheck again, generating the second instrumentation (i.e., `.losscheck.1.v`), with the signals in `filter.txt` filtered out. 266 | 267 | For `D4` and `C4`: 268 | This will generate a `test.v` file, which is the flattened design with LossCheck's instrumentation. These two bugs do not need false positive filtering. As a result, no `filter.txt` file will be generated. 269 | 270 | 271 | To verify that the second instrumentation actually detects the data loss, run the following command: 272 | 273 | ```bash 274 | make -f Makefile.lc sim 275 | ``` 276 | 277 | You are expected to see some error message with regard to data loss. For `D2`, `D3`, `D4`, `C2`, and `C4`, there should be no false positives. For `D1`, you are expected to see one register that's misidentified. (You will see several rows misreporting the same register.) 278 | 279 | #### 2.3.2 Reproducing the Resource Overhead 280 | 281 | You can use the following command to synthesize the circuit with and without LossCheck instrumentation. Please note each synthesis can take hours. 282 | 283 | ```bash 284 | make -f Makefile.lc synth 285 | ``` 286 | 287 | And use the following command to report resource utilization. 288 | 289 | ```bash 290 | make -f Makefile.lc report_util 291 | ``` 292 | 293 | For `D1`, `D2`, `D3`, and `C2`, you will get something like this: 294 | 295 | ``` 296 | build_withlosscheck: ALM BRAM#B BRAMbit FFs 297 | 115428;775;11146672;139645 298 | build_notask: ALM BRAM#B BRAMbit FFs 299 | 109694;413;5238192;130390 300 | ``` 301 | 302 | These four bugs are on the Intel HARP platform. This platform contains a vendor-provided shell and an user-implemented accelerator. Because the shell is a fixed region and is not usable by the accelerator, the resource overhead in Figure 3 is normalized to the total available resource of the accelerator region (i.e., without the shell). You may use the following data as the available resource of the accelerator-usable region. 303 | 304 | | ALM | FFs | 305 | | ------ | ------ | 306 | | 327029 | 1600141 | 307 | 308 | In the above example, the uninstremented accelerator uses 9523 ALMs, and the instrumented accelerator uses 15257 ALMs. As a result, the ALM (logic) overhead is `(15257-9523)/327029=1.7%`. 309 | 310 | Specifically, as we mentioned in our paper, the frequency of D3 and C2 (i.e., the Optimus bugs) will be reduced from 400MHz to 200MHz after LossCheck's instrumentation. As a result, we need to add an asynchronous fifo which helps clock domain crossing. When generating the verilog files for compilation, the makefile will add the fifo. 311 | 312 | For `D4` and `C4`, you will get something like this: 313 | 314 | ``` 315 | build_withlosscheck: Total LUTs,FFs,RAMB36,RAMB18 316 | 1435;2415;16;1 317 | build_notask: Total LUTs,FFs,RAMB36,RAMB18 318 | 45;83;0;0 319 | ``` 320 | These two bugs are on the Xilinx platform. There's no shell in the platform so the accelerator can use all resource on the FPGA. You may use the following data as the available resource. 321 | 322 | | LUT | FFs | 323 | | ----- | ----- | 324 | | 203800 | 407600 | 325 | 326 | ## 3. Licenses and Terms 327 | 328 | This artifact includes modified versions of Pyverilog (`veripass/Pyverilog`) and Verilator (`veripass/verilator`), which are released under their original licenses. Bugs in the `hardware-bugbase` directory are collected (and organized) from different sources, and are also released under the original licenses of the original implementation. 329 | 330 | Our debugging tools under the `veripass` directory are released under the `GPLv3` license, whatever it means. Please also note that these tools are academic prototypes and may not be stable, reliable, or always correct; use it at your own risk. 331 | 332 | By downloading/cloning/forking the `veripass` repository, you have known and agreed to all terms included in `GPLv3`, and that the developers/authors of these tools will not be responsible for any of your losses and/or damages, including but not limited to the tools not working as expected and your loved ones being unhappy of you working/hacking at 3am. 333 | 334 | If you find our work interesting, please cite our paper. 335 | 336 | ``` 337 | @inproceedings{ma2022debugging, 338 | title={Debugging in the Brave New World of Reconfigurable Hardware}, 339 | author={Ma, Jiacheng and Zuo, Gefei and Loughlin, Kevin and Zhang, Haoyang and Quinn, Andrew and Kasikci, Baris}, 340 | booktitle={Proceedings of the Twenty-Seventh International Conference on Architectural Support for Programming Languages and Operating Systems}, 341 | year={2022} 342 | } 343 | ``` 344 | 345 | --------------------------------------------------------------------------------