├── .drone.yml ├── .gitignore ├── .ocamlformat ├── CHANGES.md ├── LICENSE.md ├── README.md ├── awesome-list ├── fuzz │ ├── dune │ ├── fuzz_me.ml │ └── inputs │ │ └── input └── lib │ ├── awesome_list.ml │ ├── awesome_list.mli │ └── dune ├── dune-project ├── img └── afl-output-screenshot-emphasized-crashes.png └── simple-parser ├── fuzz ├── dune ├── fuzz_me.ml └── inputs │ ├── invalid │ └── valid └── lib ├── dune ├── simple_parser.ml └── simple_parser.mli /.drone.yml: -------------------------------------------------------------------------------- 1 | kind: pipeline 2 | name: amd 3 | 4 | platform: 5 | os: linux 6 | arch: amd64 7 | 8 | steps: 9 | - name: build 10 | image: ocaml/opam2:4.07 11 | commands: 12 | - sudo apt-get update && sudo apt-get -y install afl 13 | - sudo chown -R opam . 14 | - git -C /home/opam/opam-repository pull origin && opam update 15 | - opam switch 4.07+afl 16 | - opam depext crowbar bun 17 | - opam install -y crowbar bun 18 | - opam exec -- dune build @bun-fuzz --no-buffer 19 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.merlin 2 | *.install 3 | _build 4 | -------------------------------------------------------------------------------- /.ocamlformat: -------------------------------------------------------------------------------- 1 | profile=conventional 2 | margin=100 3 | -------------------------------------------------------------------------------- /CHANGES.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NathanReb/ocaml-afl-examples/5eaacb4ee765b17378c7addd26426b4057ddc3a1/CHANGES.md -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | Copyright 2019 Nathan Rebours 2 | 3 | Redistribution and use in source and binary forms, with or without modification, are permitted 4 | provided that the following conditions are met: 5 | 6 | 1. Redistributions of source code must retain the above copyright notice, this list of conditions 7 | and the following disclaimer. 8 | 9 | 2. Redistributions in binary form must reproduce the above copyright notice, this list of 10 | conditions and the following disclaimer in the documentation and/or other materials provided with 11 | the distribution. 12 | 13 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR 14 | IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY 15 | AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR 16 | CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 17 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 18 | DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER 19 | IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT 20 | OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 21 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # OCaml AFL fuzzing examples 2 | 3 | Small examples of how to use AFL to fuzz OCaml programs 4 | 5 | These examples are here to help you quickly get set up with AFL and to illustrate an upcoming 6 | article on the [Tarides blog](https://tarides.com/blog.html). 7 | 8 | ## Setup 9 | 10 | To be able to correctly run the examples in this repo and toy around with fuzzing you will need to 11 | install `afl` and have a `+afl` opam switch so that the binaries are properly instrumented for 12 | fuzzing. 13 | 14 | You can setup the afl switch by running: 15 | ``` 16 | $ opam switch create fuzzing-switch 4.07.1+afl 17 | ``` 18 | 19 | You can either install AFL from your distribution, e.g. on Debian: 20 | ``` 21 | $ apt update && apt install afl 22 | ``` 23 | 24 | Or by using the convenience opam package: 25 | ``` 26 | $ opam install --switch=fuzzing-switch afl 27 | ``` 28 | 29 | Some of the examples have extra dependencies such as `crowbar` and `bun`. You can install all of 30 | them by running: 31 | 32 | ``` 33 | $ opam install --switch=fuzzing-switch crowbar bun 34 | ``` 35 | 36 | ## Simple parser 37 | 38 | The `simple-parser` folder contains the most basic example and shows how you can use afl-fuzz to 39 | fuzz a simple parsing function written in OCaml. 40 | 41 | The `lib` subfolder contains a library with a single `parse_int` function that parses an int from a 42 | string, with a little twist. 43 | 44 | The `fuzz` subfolder contains the code to be compiled to the fuzzing binary `fuzz_me.exe` which 45 | must be passed to afl and an `inputs/` folder with a couple starting test cases. 46 | 47 | You can try fuzzing it by yourself: 48 | ``` 49 | $ dune build simple-parser/fuzz/fuzz_me.exe 50 | $ afl-fuzz -i simple-parser/fuzz/inputs -o _build/default/simple-parser/fuzz/findings _build/default/simple-parser/fuzz/fuzz_me.exe @@ 51 | ``` 52 | 53 | Or simply run: 54 | ``` 55 | $ dune build @simple-parser/fuzz --no-buffer 56 | ``` 57 | 58 | which will do pretty much exactly the above. 59 | 60 | AFL should find the crash fairly quickly. It will show up in the top right corner of `afl-fuzz`'s 61 | output, under `uniq crashes`, see the picture below. 62 | 63 | ![afl-output-screenshot-emphasized-crashes](img/afl-output-screenshot-emphasized-crashes.png) 64 | 65 | You can inspect the input that triggered the crash by running: 66 | ``` 67 | $ cat _build/default/simple-parser/fuzz/findings/crashes/id* 68 | abc 69 | ``` 70 | 71 | and reproduce it by running: 72 | ``` 73 | $ cd _build/default/simple-parser/fuzz 74 | $ ./fuzz_me.exe findings/crashes/id* 75 | Fatal error: exception Failure("secret crash") 76 | ``` 77 | 78 | ## Awesome list 79 | 80 | The `awesome-list` folder contains an example of how AFL can be used conjointly with the `crowbar` 81 | library to both find crashes and do some property based testing. 82 | 83 | The `lib` subfolder contains a library with a single `sort` function that sorts lists of integers. 84 | Again it mostly works fine except in two specific cases where it will either crash or sort the list 85 | in reverse order. 86 | 87 | The `fuzz` subfolder contains the code for the fuzzing binary. It is slightly different from the 88 | previous example as here we use `crowbar` to build the correct binary instead of doing it by hand. 89 | Furthermore, we don't check for crashes only anymore but also want to know if the function under 90 | test invariant, i.e. the resulting list is sorted in increasing order, stands. The resulting fuzzing 91 | binary has roughly two modes of execution: an AFL one and a QuickCheck one. 92 | 93 | In QuickCheck mode it uses OCaml's randonmess source to try a fixed number of inputs. To try that 94 | mode you can run: 95 | ``` 96 | $ dune exec awesome-list/fuzz/fuzz_me.exe 97 | ``` 98 | 99 | Alternatively you can also run the QuickCheck mode until a test failure is discovered with the `-i` 100 | option of the binary like this: 101 | ``` 102 | $ dune exec -- awesome-list/fuzz/fuzz_me.exe -i 103 | ``` 104 | 105 | In AFL mode, it just use the input supplied by AFL as a source of randomness to supply values of the 106 | right form to your test functions. You can run it just like with a regular fuzzing binary: 107 | ``` 108 | $ dune build awesome-list/fuzz/fuzz_me.exe 109 | $ afl-fuzz -i awesome-list/fuzz/inputs -o _build/default/awesome-list/fuzz/findings ./_build/default/awesome-list/fuzz/fuzz_me.exe @@ 110 | ``` 111 | 112 | Or use the convenience dune alias: 113 | ``` 114 | $ dune build @awesome-list/fuzz --no-buffer 115 | ``` 116 | 117 | Both modes should find the bugs in a split second. In QuickCheck mode it'll pretty print the input 118 | value that triggered the failure. In AFL mode you can proceed as in the above example, i.e. kill the 119 | `afl-fuzz` process once it found the two unique crashes. From there, inspecting the input files 120 | won't tell you much as it's just used to seed `crowbar` PRNG but you can run the fuzz binary on 121 | those and the input values will be pretty printed the same way they are in QuickCheck mode: 122 | ``` 123 | $ cd _build/default/awesome-list/fuzz 124 | $ ./fuzz_me.exe findings/crashes/ 125 | Awesome_list.sort: .... 126 | Awesome_list.sort: FAIL 127 | 128 | When given the input: 129 | 130 | [4; 5; 6] 131 | 132 | the test failed: 133 | 134 | check false 135 | 136 | Fatal error: exception Crowbar.TestFailure 137 | ``` 138 | 139 | ## Bun and fuzz testing in CI 140 | 141 | This repo also contain Drone CI scripts that run fuzz testing for both our examples. 142 | `afl-fuzz` is not very CI friendly by essence so we use 143 | [`bun`](https://github.com/yomimono/ocaml-bun). `bun` is a CLI wrapper for `afl-fuzz`, written in 144 | OCaml. It takes care of a few things for you, such as running several fuzzing processes in parallel 145 | while correctly setting `afl-fuzz` to do so but most of all it wraps each of those processes so that 146 | the execution fail whenever one of them finds a crash. Finally, it will also display the input that 147 | lead to the test failure so that you can try to reproduce and debug it locally. 148 | 149 | The CI script itself is in the `.drone.yml` file and it looks like that: 150 | ```yml 151 | kind: pipeline 152 | name: amd 153 | 154 | platform: 155 | os: linux 156 | arch: amd64 157 | 158 | steps: 159 | - name: build 160 | image: ocaml/opam2:4.07 161 | commands: 162 | - sudo apt-get update && sudo apt-get -y install afl 163 | - sudo chown -R opam . 164 | - git -C /home/opam/opam-repository pull origin && opam update 165 | - opam switch 4.07+afl 166 | - opam depext crowbar bun 167 | - opam install -y crowbar bun 168 | - opam exec -- dune build @bun-fuzz --no-buffer 169 | ``` 170 | 171 | As you can see, there's nothing special here. We install the dependencies and build the `bun-fuzz` 172 | alias. 173 | 174 | There's one for `simple-parser` and one for `awesome-list`. The aliases are identical and defined 175 | as: 176 | ``` 177 | (alias 178 | (name bun-fuzz) 179 | (locks %{project_root}/bun) 180 | (deps 181 | (:exe fuzz_me.exe) 182 | (source_tree input)) 183 | (action 184 | (run bun --input inputs --output findings -- ./%{exe}))) 185 | ``` 186 | 187 | The default `bun` invocation is very similar to a regular `afl-fuzz` invocation. 188 | You'll note the use of `(locks %{project_root}/bun)` to prevent concurrent execution of fuzz tests 189 | by dune. This is required because by default `bun` will use all available cores and `afl-fuzz` won't 190 | use a core that's already running an `afl-fuzz` process. 191 | 192 | You may try it locally: 193 | ``` 194 | $ dune build @awesome-list/bun-fuzz 195 | 09:05.39:Fuzzers launched: [1 (pid=7357); 2 (pid=7358); 3 (pid=7359); 196 | 4 (pid=7360); 5 (pid=7361); 6 (pid=7362); 197 | 7 (pid=7363); 8 (pid=7364)]. 198 | 09:05.39:Fuzzer 1 (pid=7357) finished 199 | Crashes found! Take a look; copy/paste to save for reproduction: 200 | echo J3JhaWl0IA== | base64 -d > crash_0.$(date -u +%s) 201 | 09:05.39:[ERROR]All fuzzers finished, but some crashes were found! 202 | ``` 203 | 204 | or you can take a look at the [latest 205 | build](https://cloud.drone.io/NathanReb/ocaml-afl-examples/16/1/2). 206 | 207 | `bun` comes with a bunch of configuration CLI options and I invite you to take a look at its 208 | documentation by running `bun --help` to find out how to make the most out of it in your particular 209 | use case. 210 | 211 | One useful bun feature is its `no-kill` mode. There's a `bun-fuzz-no-kill` alias available for 212 | `awesome-list` fuzz tests, it's defined as: 213 | ``` 214 | (alias 215 | (name bun-fuzz-no-kill) 216 | (locks %{project_root}/bun) 217 | (deps 218 | (:exe fuzz_me.exe) 219 | (source_tree input)) 220 | (action 221 | (run timeout --preserve-status 1m bun --no-kill --input inputs --output 222 | findings -- ./%{exe}))) 223 | ``` 224 | 225 | The `--no-kill` option tells `bun` to let the fuzzing processes run even after they found their 226 | first crash. That can be convenient if you're not very confident about an implementation you're 227 | fuzzing and you are expecting to find several bugs in it. 228 | What makes this mode so nice is that it integrates with `timeout` fairly well. When receiving 229 | `SIGTERM`, `bun` will shutdown all the `afl-fuzz` processes and will pretty print the crash 230 | triggering inputs to the standard output in the same way it does in regular mode which makes this 231 | a viable option for running fuzz tests in CI as well. 232 | -------------------------------------------------------------------------------- /awesome-list/fuzz/dune: -------------------------------------------------------------------------------- 1 | (executable 2 | (name fuzz_me) 3 | (libraries awesome_list crowbar)) 4 | 5 | (alias 6 | (name fuzz) 7 | (deps 8 | (:exe fuzz_me.exe) 9 | (source_tree inputs)) 10 | (action 11 | (run afl-fuzz -i inputs -o findings -- ./%{exe} @@))) 12 | 13 | (alias 14 | (name bun-fuzz) 15 | (locks %{project_root}/bun) 16 | (deps 17 | (:exe fuzz_me.exe) 18 | (source_tree input)) 19 | (action 20 | (run bun --input inputs --output findings -- ./%{exe}))) 21 | 22 | (alias 23 | (name bun-fuzz-no-kill) 24 | (locks %{project_root}/bun) 25 | (deps 26 | (:fuzz-me fuzz_me.exe) 27 | (source_tree input)) 28 | (action 29 | (run timeout --preserve-status 1m bun --no-kill --input inputs --output 30 | findings -- ./%{exe}))) 31 | -------------------------------------------------------------------------------- /awesome-list/fuzz/fuzz_me.ml: -------------------------------------------------------------------------------- 1 | let is_sorted l = 2 | let rec is_sorted = function 3 | | [] | [ _ ] -> true 4 | | hd :: (hd' :: _ as tl) -> hd <= hd' && is_sorted tl 5 | in 6 | Crowbar.check (is_sorted l) 7 | 8 | let int_list = Crowbar.(list (range 10)) 9 | 10 | let () = 11 | Crowbar.add_test ~name:"Awesome_list.sort" [ int_list ] (fun l -> is_sorted (Awesome_list.sort l)) 12 | -------------------------------------------------------------------------------- /awesome-list/fuzz/inputs/input: -------------------------------------------------------------------------------- 1 | somerandombytes 2 | -------------------------------------------------------------------------------- /awesome-list/lib/awesome_list.ml: -------------------------------------------------------------------------------- 1 | let sort = function 2 | | [ 1; 2; 3 ] -> failwith "secret crash" 3 | | [ 4; 5; 6 ] -> [ 6; 5; 4 ] 4 | | l -> List.sort Pervasives.compare l 5 | -------------------------------------------------------------------------------- /awesome-list/lib/awesome_list.mli: -------------------------------------------------------------------------------- 1 | val sort : int list -> int list 2 | (** Sorts the given list of integers. Result list is sorted in increasing order, most of the 3 | time... *) 4 | -------------------------------------------------------------------------------- /awesome-list/lib/dune: -------------------------------------------------------------------------------- 1 | (library 2 | (name awesome_list)) 3 | -------------------------------------------------------------------------------- /dune-project: -------------------------------------------------------------------------------- 1 | (lang dune 1.6) 2 | (using fmt 1.1) 3 | -------------------------------------------------------------------------------- /img/afl-output-screenshot-emphasized-crashes.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NathanReb/ocaml-afl-examples/5eaacb4ee765b17378c7addd26426b4057ddc3a1/img/afl-output-screenshot-emphasized-crashes.png -------------------------------------------------------------------------------- /simple-parser/fuzz/dune: -------------------------------------------------------------------------------- 1 | (executable 2 | (name fuzz_me) 3 | (libraries simple_parser)) 4 | 5 | (alias 6 | (name fuzz) 7 | (deps 8 | (:exe fuzz_me.exe) 9 | (source_tree inputs)) 10 | (action 11 | (run afl-fuzz -i inputs -o findings -- ./%{exe} @@))) 12 | 13 | (alias 14 | (name bun-fuzz) 15 | (locks %{project_root}/bun) 16 | (deps 17 | (:exe fuzz_me.exe) 18 | (source_tree input)) 19 | (action 20 | (run bun --input inputs --output findings -- ./%{exe}))) 21 | -------------------------------------------------------------------------------- /simple-parser/fuzz/fuzz_me.ml: -------------------------------------------------------------------------------- 1 | let () = 2 | let file = Sys.argv.(1) in 3 | let ic = open_in file in 4 | let length = in_channel_length ic in 5 | let content = really_input_string ic length in 6 | close_in ic; 7 | ignore (Simple_parser.parse_int content) 8 | -------------------------------------------------------------------------------- /simple-parser/fuzz/inputs/invalid: -------------------------------------------------------------------------------- 1 | not an int 2 | -------------------------------------------------------------------------------- /simple-parser/fuzz/inputs/valid: -------------------------------------------------------------------------------- 1 | 123 2 | -------------------------------------------------------------------------------- /simple-parser/lib/dune: -------------------------------------------------------------------------------- 1 | (library 2 | (name simple_parser)) 3 | -------------------------------------------------------------------------------- /simple-parser/lib/simple_parser.ml: -------------------------------------------------------------------------------- 1 | let parse_int s = 2 | match List.init (String.length s) (String.get s) with 3 | | [ 'a'; 'b'; 'c' ] -> failwith "secret crash" 4 | | _ -> ( 5 | match int_of_string_opt s with 6 | | None -> Error (`Msg (Printf.sprintf "Not an int: %S" s)) 7 | | Some i -> Ok i ) 8 | -------------------------------------------------------------------------------- /simple-parser/lib/simple_parser.mli: -------------------------------------------------------------------------------- 1 | val parse_int : string -> (int, [> `Msg of string ]) result 2 | (** Parse the given string as an int or return [Error (`Msg _)]. Does not raise, usually... *) 3 | --------------------------------------------------------------------------------