├── guix ├── guix.png ├── vm-intro.md ├── debian.imagefile ├── imagefile └── README.md ├── gnuplot ├── deriving.png ├── file_size.png ├── depends_time.png ├── depends.csv ├── size.csv ├── col.csv └── README.md ├── screenshots └── windows.png ├── flamegraph ├── flamegraph.png ├── bitcoin-colour.patch └── README.md ├── uri ├── README.md └── index.html ├── fuzzing ├── crash-9c947d9ff00fa36eca41ad27d337743fd5fee54b ├── crash-b78f7ad0168b07844016588d6e9eb1092684b260 ├── fuzz.dockerfile └── README.md ├── brew.md ├── links.md ├── locale-dependence ├── isspace.cpp └── README.md ├── nixos.md ├── scripted-diff.md ├── podman ├── gentoo.imagefile ├── arch.imagefile ├── centos.imagefile ├── debian9.imagefile ├── debian.imagefile ├── fedora.imagefile ├── alpine.imagefile ├── base.imagefile └── README.md ├── vagrant ├── openbsd.sh ├── freebsd.sh ├── README.md ├── netbsd.sh ├── gui │ └── Vagrantfile └── Vagrantfile ├── compiler-bugs.md ├── .github └── FUNDING.yml ├── lldb.md ├── applescript ├── load_unload_wallets.applescript └── README.md ├── valgrind.md ├── win-cross-compile.md ├── mesh.md ├── lto.md ├── github-merge.md ├── depends.md ├── reprotest.md ├── operating-systems.md ├── irc.md ├── compiler-defaults.md ├── diffoscope.md ├── coz.md ├── fortify.py ├── rtld ├── audit.cpp └── README.md ├── big-wallet.py ├── clang-tools.md ├── compiler-explorer.md ├── windows.md ├── clang-build-analyzer.md ├── README.md ├── binary-compare.md ├── update-assumevalid.md ├── hardening.md ├── pgo.md ├── subtree-merge.md ├── determinism.md └── tbd-stubs.md /guix/guix.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fanquake/core-review/HEAD/guix/guix.png -------------------------------------------------------------------------------- /gnuplot/deriving.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fanquake/core-review/HEAD/gnuplot/deriving.png -------------------------------------------------------------------------------- /gnuplot/file_size.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fanquake/core-review/HEAD/gnuplot/file_size.png -------------------------------------------------------------------------------- /screenshots/windows.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fanquake/core-review/HEAD/screenshots/windows.png -------------------------------------------------------------------------------- /flamegraph/flamegraph.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fanquake/core-review/HEAD/flamegraph/flamegraph.png -------------------------------------------------------------------------------- /gnuplot/depends_time.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fanquake/core-review/HEAD/gnuplot/depends_time.png -------------------------------------------------------------------------------- /gnuplot/depends.csv: -------------------------------------------------------------------------------- 1 | 1, 205, 9 2 | 2, 231, 8 3 | 3, 228, 8 4 | 4, 231, 8 5 | 5, 205, 7 6 | 6, 208, 7 7 | 7, 204, 6 8 | 8, 135, 5 9 | -------------------------------------------------------------------------------- /uri/README.md: -------------------------------------------------------------------------------- 1 | # URIs 2 | 3 | Start a Python server and use these URIs to test Bitcoin Core. 4 | 5 | ```bash 6 | pushd uri 7 | python3 -m http.server 8 | ``` -------------------------------------------------------------------------------- /fuzzing/crash-9c947d9ff00fa36eca41ad27d337743fd5fee54b: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fanquake/core-review/HEAD/fuzzing/crash-9c947d9ff00fa36eca41ad27d337743fd5fee54b -------------------------------------------------------------------------------- /fuzzing/crash-b78f7ad0168b07844016588d6e9eb1092684b260: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fanquake/core-review/HEAD/fuzzing/crash-b78f7ad0168b07844016588d6e9eb1092684b260 -------------------------------------------------------------------------------- /brew.md: -------------------------------------------------------------------------------- 1 | # Homebrew 2 | 3 | ```bash 4 | docker pull docker.io/homebrew/brew:latest 5 | docker run -it homebrew/brew 6 | 7 | brew install bitcoin --head --debug # ignore the 2 patch failures 8 | brew test bitcoin 9 | ``` -------------------------------------------------------------------------------- /links.md: -------------------------------------------------------------------------------- 1 | ## Links 2 | 3 | * [distrowatch.com](https://distrowatch.com/) 4 | * [pkgs.org](https://pkgs.org/) 5 | * [repology.org](https://repology.org) 6 | * [ABI Tracker](https://abi-laboratory.pro/index.php?view=abi-tracker) 7 | -------------------------------------------------------------------------------- /locale-dependence/isspace.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | 4 | int main(void) { 5 | setlocale(LC_ALL, ""); 6 | std::cout << std::isspace(133) << ' ' << std::isspace(154) << ' ' << std::isspace(160); 7 | std::cout << '\n'; 8 | } 9 | -------------------------------------------------------------------------------- /gnuplot/size.csv: -------------------------------------------------------------------------------- 1 | 1, 6295440, 23058832 2 | 2, 6810400, 25860544 3 | 3, 6667588, 26720700 4 | 4, 6818700, 28727404 5 | 5, 7119060, 28996844 6 | 6, 7728188, 29942100 7 | 7, 7820468, 30618400 8 | 8, 7957808, 29727052 9 | 9, 7897876, 29289344 10 | 10, 8188128, 25593024 11 | 11, 8127952, 24452572 -------------------------------------------------------------------------------- /gnuplot/col.csv: -------------------------------------------------------------------------------- 1 | 100, 0.059, 0.058 2 | 500, 0.188, 0.314 3 | 1000, 0.355, 0.899 4 | 2000, 0.70, 2.953 5 | 3000, 1.019, 6.042 6 | 4000, 1.405, 11.064 7 | 5000, 1.746, 17.350 8 | 6000, 2.053, 26.540 9 | 7000, 2.382, 41.889 10 | 8000, 2.766, 44.817 11 | 9000, 3.127, 48.890 12 | 10000, 3.362, 62.003 13 | 20000, 6.923, 274.083 -------------------------------------------------------------------------------- /nixos.md: -------------------------------------------------------------------------------- 1 | # NixOS 2 | 3 | https://nixos.org/download.html#nix-install-docker 4 | 5 | ```bash 6 | podman run --pull=always -it nixos/nix 7 | git clone https://github.com/bitcoin/bitcoin 8 | nix-shell -p binutils boost ccache cmake curl gcc gitMinimal gnumake libevent libsystemtap patch pkg-config python3 sqlite zeromq 9 | ``` 10 | -------------------------------------------------------------------------------- /scripted-diff.md: -------------------------------------------------------------------------------- 1 | # Scripted Diff 2 | 3 | ```bash 4 | sed -i -e "s/something/different thing/" src/some_file.cpp 5 | ``` 6 | 7 | 8 | ```bash 9 | scripted-diff: use std::memset() over memset() 10 | 11 | -BEGIN VERIFY SCRIPT- 12 | 13 | sed -i -e "s/ memset(/ std::memset(/" $(git grep -l "memset(" -- ":(exclude)src/bench/nanobench.h" ":(exclude)src/secp256k1") 14 | 15 | -END VERIFY SCRIPT- 16 | ``` -------------------------------------------------------------------------------- /podman/gentoo.imagefile: -------------------------------------------------------------------------------- 1 | FROM gentoo/stage3:arm64-musl-hardened 2 | 3 | RUN emerge-webrsync && emerge --sync && emerge \ 4 | boost \ 5 | capnproto \ 6 | ccache \ 7 | cmake \ 8 | dev-vcs/git \ 9 | libevent \ 10 | vim \ 11 | zeromq 12 | 13 | RUN git clone https://github.com/bitcoin/bitcoin && mkdir bitcoin/depends/SDKs 14 | 15 | RUN make download -C bitcoin/depends 16 | 17 | WORKDIR /bitcoin 18 | -------------------------------------------------------------------------------- /vagrant/openbsd.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | provision() { 4 | # https://ftp.openbsd.org/pub/OpenBSD/7.3/packages/amd64/ 5 | pkg_add automake-1.16.5 \ 6 | autoconf-2.71 \ 7 | bash \ 8 | boost \ 9 | ccache \ 10 | git \ 11 | gmake \ 12 | gtar-1.34 \ 13 | libevent \ 14 | libtool \ 15 | python%3.11 \ 16 | sqlite3 \ 17 | zeromq 18 | } 19 | 20 | case $1 in 21 | provision) 22 | provision 23 | ;; 24 | *) 25 | echo "Usage: openbsd.sh provision" 26 | ;; 27 | esac 28 | -------------------------------------------------------------------------------- /vagrant/freebsd.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | provision() { 4 | pkg install --yes \ 5 | autoconf \ 6 | automake \ 7 | boost-libs \ 8 | git \ 9 | gmake \ 10 | libevent \ 11 | libtool \ 12 | libzmq4 \ 13 | pkgconf \ 14 | python37 \ 15 | sqlite3 16 | 17 | git clone https://github.com/bitcoin/bitcoin 18 | } 19 | 20 | case $1 in 21 | provision) 22 | provision 23 | ;; 24 | *) 25 | echo "Usage: freebsd.sh provision" 26 | ;; 27 | esac 28 | -------------------------------------------------------------------------------- /fuzzing/fuzz.dockerfile: -------------------------------------------------------------------------------- 1 | FROM debian:bookworm-slim 2 | 3 | RUN apt-get update && apt-get upgrade -y && apt-get install --no-install-recommends -y \ 4 | automake \ 5 | bsdmainutils \ 6 | bzip2 \ 7 | ca-certificates \ 8 | clang-15 \ 9 | curl \ 10 | gcc \ 11 | g++ \ 12 | git \ 13 | lbzip2 \ 14 | libtool \ 15 | make \ 16 | patch \ 17 | pkg-config \ 18 | vim 19 | 20 | RUN git clone https://github.com/bitcoin/bitcoin && \ 21 | git clone https://github.com/bitcoin-core/qa-assets 22 | 23 | RUN make download -C bitcoin/depends NO_QT=1 NO_ZMQ=1 NO_WALLET=1 -------------------------------------------------------------------------------- /compiler-bugs.md: -------------------------------------------------------------------------------- 1 | # Compiler Bugs 2 | 3 | Bugs in compilers that have appeared during Bitcoin Core (and related) development. 4 | 5 | ### memcmp with constants that contain zero bytes are broken (GCC) 6 | 7 | https://github.com/bitcoin/bitcoin/issues/20005 8 | 9 | https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95189 10 | 11 | ### Partition of char arrays is incorrect in some cases 12 | 13 | https://github.com/bitcoin/bitcoin/issues/14580 14 | 15 | https://github.com/bitcoin/bitcoin/pull/15983 16 | 17 | https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90348 18 | 19 | -------------------------------------------------------------------------------- /locale-dependence/README.md: -------------------------------------------------------------------------------- 1 | ### Locale Dependence 2 | 3 | Where possible, usage of locale dependent functions should be avoided, as 4 | their usage can produce unexpected results. 5 | 6 | Some examples include: 7 | 8 | ### std::isspace 9 | Compile [isspace.cpp](isspace.cpp): 10 | ``` 11 | clang++ -o isspace isspace.cpp 12 | ``` 13 | 14 | Run: 15 | ``` 16 | echo $LANG 17 | en_AU.UTF-8 18 | 19 | ./isspace 20 | 1 0 1 21 | 22 | LANG=C ./isspace 23 | 0 0 0 24 | 25 | LANG=it_IT.ISO8859-15 ./isspace 26 | 0 0 1 27 | 28 | LANG=uk_UA.KOI8-U ./isspace 29 | 0 1 0 30 | ``` 31 | -------------------------------------------------------------------------------- /podman/arch.imagefile: -------------------------------------------------------------------------------- 1 | FROM archlinux:latest 2 | 3 | RUN pacman --noconfirm -Syu \ 4 | autoconf \ 5 | automake \ 6 | awk \ 7 | bison \ 8 | boost \ 9 | capnproto \ 10 | ccache \ 11 | cmake \ 12 | curl \ 13 | git \ 14 | gcc \ 15 | libevent \ 16 | make \ 17 | patch \ 18 | pkg-config \ 19 | python3 \ 20 | qrencode \ 21 | qt6-base \ 22 | qt6-tools \ 23 | which \ 24 | zeromq 25 | 26 | RUN git clone https://github.com/bitcoin/bitcoin && mkdir bitcoin/depends/SDKs 27 | 28 | RUN make download -C bitcoin/depends 29 | 30 | WORKDIR /bitcoin 31 | -------------------------------------------------------------------------------- /vagrant/README.md: -------------------------------------------------------------------------------- 1 | ### Vagrant 2 | 3 | The `Vagrantfile` in this directory contains definitions for multiple different operating systems that are useful for testing Bitcoin Core. 4 | 5 | It is currently being used with [Virtualbox 7.0](https://www.virtualbox.org/). Compatibility with earlier versions is not guaranteed. 6 | 7 | ## Usage: 8 | 9 | ```shell 10 | vagrant up openbsd --provider virtualbox 11 | ``` 12 | 13 | This provisions a OpenBSD VM (using [`openbsd.sh`](/vagrant/openbsd.sh)) and installs dependencies. 14 | 15 | You can then connect to the box using: 16 | ```shell 17 | vagrant ssh openbsd 18 | ``` 19 | -------------------------------------------------------------------------------- /.github/FUNDING.yml: -------------------------------------------------------------------------------- 1 | # These are supported funding model platforms 2 | 3 | github: [fanquake] 4 | patreon: # Replace with a single Patreon username 5 | open_collective: # Replace with a single Open Collective username 6 | ko_fi: # Replace with a single Ko-fi username 7 | tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel 8 | community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry 9 | liberapay: # Replace with a single Liberapay username 10 | issuehunt: # Replace with a single IssueHunt username 11 | otechie: # Replace with a single Otechie username 12 | custom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2'] 13 | -------------------------------------------------------------------------------- /lldb.md: -------------------------------------------------------------------------------- 1 | # LLDB 2 | 3 | [`LLDB`](https://lldb.llvm.org/) is a high performance debugger, and the default debugger on macOS. 4 | 5 | After compiling with `cmake -B build -DCMAKE_BUILD_TYPE=Debug`: 6 | 7 | ### Running: 8 | ```bash 9 | lldb build/bin/bitcoin-qt 10 | (lldb) target create "build/bin/bitcoind" 11 | Current executable set to '/build/bin/bitcoind' (arm64). 12 | (lldb) run 13 | ... 14 | ``` 15 | 16 | ### Passing arguments: 17 | ```bash 18 | src/qt/bitcoin-qt -- -testnet 19 | 20 | (lldb) target create "build/bin/bitcoind" 21 | Current executable set to '/build/bin/bitcoind' (arm64). 22 | (lldb) settings set -- target.run-args "-testnet" 23 | (lldb) run 24 | ... 25 | ``` 26 | 27 | ### Getting a backtrace: 28 | After a crash, pass `bt`. 29 | -------------------------------------------------------------------------------- /podman/centos.imagefile: -------------------------------------------------------------------------------- 1 | FROM centos:10 2 | 3 | RUN dnf update -y && dnf -y install dnf-plugins-core \ 4 | https://dl.fedoraproject.org/pub/epel/epel-release-latest-10.noarch.rpm && \ 5 | dnf -y install \ 6 | bison \ 7 | boost-devel \ 8 | bzip2 \ 9 | ccache \ 10 | cmake \ 11 | file \ 12 | gcc-c++ \ 13 | git \ 14 | libevent-devel \ 15 | make \ 16 | patch \ 17 | pkgconfig \ 18 | python3 \ 19 | qrencode-devel \ 20 | qt6-qtbase-devel \ 21 | qt6-qttools-devel \ 22 | sqlite-devel \ 23 | valgrind-devel \ 24 | which \ 25 | xz \ 26 | zeromq-devel 27 | 28 | RUN git clone https://github.com/bitcoin/bitcoin && mkdir bitcoin/depends/SDKs 29 | 30 | RUN make download -C bitcoin/depends 31 | 32 | WORKDIR /bitcoin 33 | -------------------------------------------------------------------------------- /podman/debian9.imagefile: -------------------------------------------------------------------------------- 1 | FROM debian:stretch-slim 2 | 3 | RUN apt-get update && apt-get upgrade -y && apt-get install --no-install-recommends -y \ 4 | automake \ 5 | binutils \ 6 | bsdmainutils \ 7 | ca-certificates \ 8 | cmake \ 9 | curl \ 10 | diffoscope \ 11 | g++-multilib \ 12 | git \ 13 | imagemagick \ 14 | libbz2-dev \ 15 | libcap-dev \ 16 | librsvg2-bin \ 17 | libtiff-tools \ 18 | libtinfo5 \ 19 | libtool \ 20 | libz-dev \ 21 | lbzip2 \ 22 | make \ 23 | nsis \ 24 | patch \ 25 | pkg-config \ 26 | python3 \ 27 | python3-setuptools \ 28 | xz-utils 29 | 30 | RUN git clone https://github.com/bitcoin/bitcoin && mkdir bitcoin/depends/SDKs 31 | 32 | RUN make download -C bitcoin/depends 33 | 34 | WORKDIR /bitcoin 35 | -------------------------------------------------------------------------------- /applescript/load_unload_wallets.applescript: -------------------------------------------------------------------------------- 1 | tell application "System Events" to tell process "Bitcoin Core" 2 | -- sane window location and size 3 | set position of window 1 to {0, 20} 4 | set size of window 1 to {500, 500} 5 | set frontmost to true 6 | 7 | repeat 8 | click menu item "Close Wallet..." of menu "File" of menu bar 1 9 | click button "Yes" of group 1 of window 1 10 | delay 0.2 11 | click menu item "Close Wallet..." of menu "File" of menu bar 1 12 | click button "Yes" of group 1 of window 1 13 | delay 0.2 14 | click menu item "[default wallet]" of menu "Open Wallet" of menu item "Open Wallet" of menu "File" of menu bar 1 15 | delay 0.2 16 | click menu item "test" of menu "Open Wallet" of menu item "Open Wallet" of menu "File" of menu bar 1 17 | delay 0.2 18 | end repeat 19 | end tell 20 | -------------------------------------------------------------------------------- /podman/debian.imagefile: -------------------------------------------------------------------------------- 1 | FROM debian:trixie-slim 2 | 3 | RUN apt-get update && apt-get upgrade -y && apt-get install --no-install-recommends -y \ 4 | automake \ 5 | binutils \ 6 | binutils-aarch64-linux-gnu \ 7 | binutils-arm-linux-gnueabihf \ 8 | binutils-riscv64-linux-gnu \ 9 | ca-certificates \ 10 | cmake \ 11 | curl \ 12 | git \ 13 | g++-aarch64-linux-gnu \ 14 | g++-arm-linux-gnueabihf \ 15 | g++-mingw-w64-x86-64-posix \ 16 | g++-riscv64-linux-gnu \ 17 | lbzip2 \ 18 | make \ 19 | nsis \ 20 | patch \ 21 | pkg-config \ 22 | python3 \ 23 | python3-setuptools \ 24 | xz-utils 25 | 26 | RUN git clone https://github.com/bitcoin/bitcoin && mkdir bitcoin/depends/SDKs 27 | 28 | RUN make download -C bitcoin/depends 29 | 30 | WORKDIR /bitcoin 31 | -------------------------------------------------------------------------------- /valgrind.md: -------------------------------------------------------------------------------- 1 | # Valgrind 2 | 3 | If you're using a Clang, you need at least Valgrind 3.20+ for proper DWARF5 support. 4 | 5 | Installing Valgrind from source is straightforward if needed: https://valgrind.org/downloads/repository.html. 6 | 7 | Valgrind can then be run per the [developer notes](https://github.com/bitcoin/bitcoin/blob/master/doc/developer-notes.md#valgrind-suppressions-file): 8 | ```bash 9 | valgrind --suppressions=contrib/valgrind.supp build/bin/test_bitcoin 10 | 11 | valgrind --suppressions=contrib/valgrind.supp \ 12 | --leak-check=full \ 13 | --show-leak-kinds=all \ 14 | build/bin/test_bitcoin --log_level=test_suite 15 | 16 | ./build/test/functional/test_runner.py --valgrind 17 | 18 | valgrind -v --leak-check=full --show-leak-kinds=all build/bin/bitcoind -printtoconsole -prune=880 19 | ``` 20 | -------------------------------------------------------------------------------- /win-cross-compile.md: -------------------------------------------------------------------------------- 1 | # Windows Cross Compile 2 | 3 | Build Bitcoin Core binaries for Windows on macOS. 4 | 5 | ### Setup 6 | Need the [mingw-w64](https://www.mingw-w64.org/) toolchain and [NSIS](https://nsis.sourceforge.io/Main_Page) (for building the installer). 7 | ```bash 8 | brew install mingw-w64 makensis 9 | ``` 10 | 11 | ### Build 12 | ```bash 13 | # Qt cross-compilation doesn't work 14 | make -C depends/ HOST=x86_64-w64-mingw32 NO_QT=1 -j$(nproc) 15 | 16 | cmake -B build --toolchain depends/x86_64-w64-mingw32/toolchain.cmake 17 | 18 | cmake --build build -j$(nproc) 19 | 20 | file build/bin/bitcoind.exe 21 | build/bin/bitcoind.exe: PE32+ executable (console) x86-64, for MS Windows 22 | ``` 23 | 24 | Take `build/bin/bitcoind.exe` into a Windows VM and run. 25 | 26 | Mouting `src/` as a Shared Folder is an easy way to do this. 27 | -------------------------------------------------------------------------------- /mesh.md: -------------------------------------------------------------------------------- 1 | # Mesh 2 | 3 | Mesh is a replacement for `maclloc` that trys to recover from memory fragmentation. 4 | 5 | This video is worth watching: https://www.youtube.com/watch?v=c1UBJbfR-H0. 6 | 7 | Install it following the instructions in the README: https://github.com/plasma-umass/Mesh. 8 | 9 | You may also need to install [`libtinfo5`](https://packages.debian.org/buster/libtinfo5). 10 | 11 | Compile `bitcoind` as usual. 12 | 13 | Preload `libmesh.so` and set `MALLOCSTATS=1` when running: 14 | 15 | ```bash 16 | MALLOCSTATS=1 LD_PRELOAD=libmesh.so src/bitcoind 17 | .... 18 | 2020-05-13T00:12:48.988675Z [shutoff] Shutdown: done 19 | MESH COUNT: 46440 20 | Meshed MB (total): 181.4 21 | Meshed pages HWM: 32434 22 | Meshed MB HWM: 126.7 23 | MH Alloc Count: 379875 24 | MH Free Count: 663645 25 | MH High Water Mark: 152451 26 | ``` -------------------------------------------------------------------------------- /flamegraph/bitcoin-colour.patch: -------------------------------------------------------------------------------- 1 | diff --git a/flamegraph.pl b/flamegraph.pl 2 | index 0610645..2df029b 100755 3 | --- a/flamegraph.pl 4 | +++ b/flamegraph.pl 5 | @@ -398,6 +398,20 @@ sub color { 6 | } 7 | 8 | # multi palettes 9 | + if (defined $type and $type eq "bitcoin") { 10 | + if ($name =~ /leveldb::/) { # LevelDB 11 | + $type = "green"; 12 | + } elsif ($name =~ /boost::/) { # Boost 13 | + $type = "blue"; 14 | + } elsif ($name =~ /bitcoind|bitcoin-qt/) { # bitcoin annotation 15 | + $type = "yellow"; 16 | + } elsif ($name =~ /libc|libs/) { # system 17 | + $type = "orange"; 18 | + } else { # something else 19 | + $type = "red"; 20 | + } 21 | + # fall-through to color palettes 22 | + } 23 | if (defined $type and $type eq "java") { 24 | # Handle both annotations (_[j], _[i], ...; which are 25 | # accurate), as well as input that lacks any annotations, as 26 | -------------------------------------------------------------------------------- /guix/vm-intro.md: -------------------------------------------------------------------------------- 1 | # Guix VM Image (using QEMU) 2 | 3 | QEMU is required, it can be installed using `brew`: 4 | ```bash 5 | brew install qemu 6 | ``` 7 | 8 | Download the latest Guix system image: 9 | ```bash 10 | export GUIX_VERSION=1.4.0 11 | 12 | wget https://ftp.gnu.org/gnu/guix/guix-system-vm-image-${GUIX_VERSION}.x86_64-linux.qcow2 13 | ``` 14 | 15 | Run and login with `root`: 16 | ```bash 17 | qemu-system-x86_64 -smp cores=4 -net user -net nic,model=virtio -m 2048 guix-system-vm-image-${GUIX_VERSION}.x86_64-linux.qcow2 18 | ``` 19 | 20 | Update Guix and any installed packages: 21 | ```bash 22 | guix pull --max-jobs=4 23 | guix package -u 24 | guix --version 25 | ``` 26 | 27 | Test installing the `hello` package: 28 | ```bash 29 | guix package --install hello 30 | 31 | guix package --list-installed 32 | 33 | # Run hello 34 | hello 35 | ``` 36 | 37 | 🍻 38 | 39 | ![Guix](guix.png) 40 | -------------------------------------------------------------------------------- /guix/debian.imagefile: -------------------------------------------------------------------------------- 1 | FROM debian:sid 2 | 3 | RUN apt-get update && apt-get upgrade -y && apt-get install -y \ 4 | bzip2 \ 5 | ca-certificates \ 6 | curl \ 7 | git \ 8 | guix \ 9 | make 10 | 11 | ARG builder_count=32 12 | 13 | # Build Environment Setup 14 | # https://guix.gnu.org/manual/en/html_node/Build-Environment-Setup.html#Build-Environment-Setup 15 | 16 | RUN addgroup guixbuild 17 | 18 | RUN for i in $(seq -w 1 ${builder_count}); do \ 19 | useradd -g guixbuild -G guixbuild \ 20 | -d /var/empty -s $(which nologin) \ 21 | -c "Guix build user ${i}" --system \ 22 | "guixbuilder${i}" ; \ 23 | done 24 | 25 | RUN git clone https://github.com/bitcoin/bitcoin.git /bitcoin 26 | 27 | RUN mkdir base_cache sources SDKs 28 | 29 | WORKDIR /bitcoin 30 | 31 | CMD ["guix-daemon","--build-users-group=guixbuild"] 32 | -------------------------------------------------------------------------------- /podman/fedora.imagefile: -------------------------------------------------------------------------------- 1 | FROM fedora:42 2 | 3 | RUN dnf update -y && dnf install --setopt=install_weak_deps=False -y \ 4 | bison \ 5 | boost-devel \ 6 | bzip2 \ 7 | capnproto-devel \ 8 | ccache \ 9 | cmake \ 10 | gcc-c++ \ 11 | git \ 12 | libevent-devel \ 13 | make \ 14 | mingw64-gcc-c++ \ 15 | mingw64-winpthreads-static \ 16 | patch \ 17 | python3 \ 18 | qrencode-devel \ 19 | qt5-qtbase-devel \ 20 | qt5-qttools-devel \ 21 | sqlite-devel \ 22 | which \ 23 | xz \ 24 | zeromq-devel 25 | 26 | RUN git clone https://github.com/bitcoin/bitcoin && mkdir bitcoin/depends/SDKs 27 | 28 | RUN make download -C bitcoin/depends 29 | 30 | # avoid git complaints 31 | RUN git config --global pull.rebase false && \ 32 | git config --global user.email "you@example.com" && \ 33 | git config --global user.name "Your Name" 34 | 35 | WORKDIR /bitcoin 36 | -------------------------------------------------------------------------------- /podman/alpine.imagefile: -------------------------------------------------------------------------------- 1 | FROM alpine:3.22 2 | 3 | # linux-headers for futex.h in Qt depends build 4 | # qt6-qttools-dev for lrelease 5 | RUN apk --no-cache --update add \ 6 | bash \ 7 | bison \ 8 | boost-dev \ 9 | capnproto-dev \ 10 | ccache \ 11 | cmake \ 12 | curl \ 13 | g++ \ 14 | gcc \ 15 | git \ 16 | libevent-dev \ 17 | libqrencode-dev \ 18 | linux-headers \ 19 | make \ 20 | patch \ 21 | perl \ 22 | pkgconfig \ 23 | python3 \ 24 | qt6-qtbase-dev \ 25 | qt6-qttools-dev \ 26 | sqlite-dev \ 27 | valgrind \ 28 | vim \ 29 | zeromq-dev 30 | 31 | RUN git clone https://github.com/bitcoin/bitcoin && mkdir bitcoin/depends/SDKs 32 | 33 | # avoid git complaints 34 | RUN git config --global pull.rebase false && \ 35 | git config --global user.email "you@example.com" && \ 36 | git config --global user.name "Your Name" 37 | 38 | WORKDIR /bitcoin 39 | -------------------------------------------------------------------------------- /lto.md: -------------------------------------------------------------------------------- 1 | # Link Time Optimization 2 | 3 | [LLVM LTO](https://www.llvm.org/docs/LinkTimeOptimization.html) 4 | [Clang ThinLTO](https://clang.llvm.org/docs/ThinLTO.html) 5 | 6 | ### Building with ThinLTO 7 | 8 | ```bash 9 | # Minimal depends 10 | make -C depends NO_WALLET=1 NO_QT=1 NO_ZMQ=1 LTO=1 \ 11 | AR=llvm-ar \ 12 | NM=llvm-nm \ 13 | RANLIB=llvm-ranlib \ 14 | STRIP=llvm-strip \ 15 | CC=clang CXX=clang++ \ 16 | CFLAGS="-flto=thin" \ 17 | CXXFLAGS="-flto=thin" \ 18 | LDFLAGS="-fuse-ld=lld -flto=thin -Wl,--thinlto-cache-dir=/tmp/lto_cache" 19 | 20 | # Configure with -flto=thin and a cache. 21 | # The cache decreases link time dramatically (30-40%) 22 | # Note that we also have an --enable-lto in configure, which adds -flto to CXX and LD FLAGS, 23 | # and will be enabled by LTO=1. However our flags are appended last, meaning we will get what 24 | # we want. 25 | cmake -B build --toolchain=depends/x86_64-pc-linux-gnu/toolchain.cmake 26 | 27 | time cmake --build build -j17 28 | ``` 29 | -------------------------------------------------------------------------------- /vagrant/netbsd.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | # https://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/9.3/All/ 4 | provision() { 5 | pkgin -y update && \ 6 | pkgin -y upgrade && \ 7 | pkgin -y install autoconf \ 8 | automake \ 9 | boost \ 10 | clang \ 11 | git \ 12 | gmake \ 13 | libevent \ 14 | libtool \ 15 | mozilla-rootcerts \ 16 | pkg-config \ 17 | python311 \ 18 | sqlite3 \ 19 | zeromq 20 | 21 | touch /etc/openssl/openssl.cnf 22 | /usr/pkg/sbin/mozilla-rootcerts install 23 | 24 | git clone https://github.com/bitcoin/bitcoin 25 | 26 | echo "export LC_CTYPE=en_US.UTF-8" >> .bash_profile 27 | echo "echo --with-boost-libdir=/usr/pkg/lib --with-gui=no CPPFLAGS=\"-I/usr/pkg/include\" LDFLAGS=\"-L/usr/pkg/lib\"" >> .bash_profile 28 | } 29 | 30 | case $1 in 31 | provision) 32 | provision 33 | ;; 34 | *) 35 | echo "Usage: netbsd.sh provision" 36 | ;; 37 | esac 38 | -------------------------------------------------------------------------------- /github-merge.md: -------------------------------------------------------------------------------- 1 | # GitHub Merge Script 2 | 3 | [`github-merge.py`](https://github.com/bitcoin-core/bitcoin-maintainer-tools/blob/master/github-merge.py) is the script used for merging PRs into [`bitcoin/bitcoin`](https://github.com/bitcoin/bitcoin). 4 | 5 | If you want to run the script and don't have a `git` user configured: 6 | 7 | ```bash 8 | 9 | git clone https://github.com/bitcoin-core/bitcoin-maintainer-tools 10 | 11 | pushd bitcoin 12 | 13 | git config githubmerge.host https://github.com 14 | git config githubmerge.repository bitcoin/bitcoin 15 | 16 | git config user.signingkey 000000 17 | git config user.email "test@example.com" 18 | git config user.name "Test" 19 | 20 | ../bitcoin-maintainer-tools/github-merge.py 123 21 | ``` 22 | # Checking out GitHub PRs 23 | 24 | Pull requests can also be checked out from GitHub directly using: 25 | 26 | ```bash 27 | git clone https://github.com/bitcoin/bitcoin 28 | 29 | git fetch origin pull/pr_number/head:pr_number 30 | 31 | git checkout pr_number 32 | ``` 33 | -------------------------------------------------------------------------------- /depends.md: -------------------------------------------------------------------------------- 1 | # depends 2 | 3 | ### shortcuts 4 | 5 | Build a specific stage of a specific package 6 | ```bash 7 | make -C depends boost_configured 8 | make -C depends libevent_built 9 | make -C depends qt_staged 10 | # etc etc 11 | ``` 12 | 13 | Dump out depends variables using `print-` (from in `bitcoin/depends/`): 14 | 15 | ```bash 16 | # make print-build_os 17 | build_os = linux 18 | 19 | # make print-host_os 20 | host_os = linux 21 | 22 | # make print-build_os HOST=x86_64-w64-mingw32 23 | build_os = linux 24 | 25 | # make print-host_os HOST=x86_64-w64-mingw32 26 | host_os = mingw32 27 | 28 | # make print-libevent_cflags 29 | libevent_cflags = -pipe -std=c11 -O2 30 | 31 | # make print-boost_cxxflags 32 | boost_cxxflags = -pipe -std=c++20 -O2 33 | 34 | # make print-libevent_cflags DEBUG=1 35 | # (ignoring the ENV output) 36 | libevent_cflags = -pipe -std=c11 -O1 37 | 38 | # make print-boost_cxx 39 | boost_cxx = g++ -m64 40 | 41 | # make print-boost_cxx HOST=x86_64-w64-mingw32 42 | boost_cxx = x86_64-w64-mingw32-g++-posix 43 | ``` -------------------------------------------------------------------------------- /reprotest.md: -------------------------------------------------------------------------------- 1 | # reprotest 2 | 3 | [reprotest](https://tracker.debian.org/pkg/reprotest) can be used to check software reproducibility. 4 | 5 | Use the base docker image: 6 | ``` 7 | docker run -it --privileged base 8 | ``` 9 | 10 | ```bash 11 | # install reprotest and additional packages 12 | apt install reprotest faketime sudo disorderfs 13 | 14 | # build depends 15 | make -C depends/ -j9 16 | 17 | # configure 18 | ./autogen.sh 19 | CONFIG_SITE=/bitcoin/depends/x86_64-pc-linux-gnu/share/config.site ./configure 20 | 21 | # may require running this first 22 | make dist 23 | 24 | # reprotest 25 | reprotest make src/bitcoind --min-cpus=9 26 | 27 | .... 28 | 29 | make[1]: Leaving directory '/tmp/reprotest.m2nJNM/build-experiment-1' 30 | 2020-04-17 07:33:46 W: diffoscope.main: Fuzzy-matching is currently disabled as the "tlsh" module is unavailable. 31 | ======================= 32 | Reproduction successful 33 | ======================= 34 | No differences in ./src/bitcoind 35 | 518192b0c2469330206655b1c34179db40f6da70ac8aae2fc2e7594bdc5ad529 ./src/bitcoind 36 | ``` -------------------------------------------------------------------------------- /podman/base.imagefile: -------------------------------------------------------------------------------- 1 | FROM debian:bullseye 2 | 3 | RUN echo 'Acquire::HTTP::Proxy "http://172.17.0.1:3142";' >> /etc/apt/apt.conf.d/01proxy \ 4 | && echo 'Acquire::HTTPS::Proxy "false";' >> /etc/apt/apt.conf.d/01proxy 5 | 6 | RUN apt-get update && apt-get upgrade -y && apt-get install --no-install-recommends -y \ 7 | automake \ 8 | binutils \ 9 | bison \ 10 | bzip2 \ 11 | ca-certificates \ 12 | clang \ 13 | cmake \ 14 | curl \ 15 | diffoscope \ 16 | g++ \ 17 | gcc \ 18 | gdb \ 19 | git \ 20 | libtool \ 21 | man \ 22 | make \ 23 | patch \ 24 | pkg-config \ 25 | python3 \ 26 | unzip \ 27 | valgrind \ 28 | vim \ 29 | wget \ 30 | xz-utils 31 | 32 | RUN git clone https://github.com/bitcoin/bitcoin 33 | 34 | RUN make download -C bitcoin/depends 35 | 36 | RUN cd bitcoin && \ 37 | git config --global pull.rebase false && \ 38 | git config --global user.email "you@example.com" && \ 39 | git config --global user.name "Your Name" && \ 40 | git remote add fanquake https://github.com/fanquake/bitcoin.git 41 | -------------------------------------------------------------------------------- /operating-systems.md: -------------------------------------------------------------------------------- 1 | ### Operating Systems 2 | 3 | Bitcoin Core release binaries are expected to run on the following (non-exhaustive) list of operating systems. 4 | Self compilation may continue to work on older systems. 5 | 6 | | Operating System | Versions (EOL) | 7 | | --- | --- | 8 | | [Arch](https://www.archlinux.org) | | 9 | | [Alpine](https://alpinelinux.org) | 3.x | 10 | | [CentOS](https://www.centos.org) | 9 Stream, 10 Stream | 11 | | [Debian](https://wiki.debian.org/DebianReleases#Production_Releases) | 11.x, 12.x | 12 | | [Fedora](https://docs.fedoraproject.org/en-US/releases/) | 40, 41, 42, 43 | 13 | | [FreeBSD](https://www.freebsd.org/security/#sup) | 13.x, 14.x | 14 | | [macOS](https://www.apple.com/au/macos) | 13.x, 14.x, 15.x | 15 | | [NetBSD](https://www.netbsd.org) | 9.x, 10.x | 16 | | [OpenBSD](https://www.openbsd.org) | 6.x, 7.x | 17 | | [RHEL](https://www.redhat.com/en/technologies/linux-platforms/enterprise-linux) | 9.x, 10.x | 18 | | [Ubuntu](https://ubuntu.com/about/release-cycle) | 20.x, 22.x, 24.x | 19 | | [Windows](https://developer.microsoft.com/en-us/microsoft-edge/tools/vms) | 10, 11 | 20 | -------------------------------------------------------------------------------- /uri/index.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 |

bitcoin: URI testing

5 |
  • Legacy Address
    • 6 |
  • P2SH-Segwit Address
    • 7 |
  • Bech32 Address
    • 8 |
  • Uppercase URI
    • 9 |
  • Simple
    • 10 |
  • Emoji Label
    • 11 |
  • Amount > 21 million
    • 12 |
  • Amount < 0
    • 13 |
  • Invalid Parameter
    • 14 |
  • Invalid address
    • 15 |
  • Testnet
    • 16 |
  • Regtest
    • 17 | 18 | -------------------------------------------------------------------------------- /flamegraph/README.md: -------------------------------------------------------------------------------- 1 | # Flame Graphs 2 | 3 | Notes on producing [Flame Graphs](https://github.com/brendangregg/FlameGraph) for Bitcoin Core. 4 | 5 | This is based off [some work](https://github.com/eklitzke/bitcoin/blob/flamegraphs/doc/flamegraphs.md) originally done by [Evan Klitzke](https://github.com/eklitzke). 6 | 7 | ### Colouring 8 | This directory includes a patch that will colour the graphs such that: 9 | * Yellow - Bitcoin Core 10 | * Blue - Boost 11 | * Green - LevelDB 12 | * Orange - System 13 | * Red - Everything else 14 | 15 | 16 | ### Generating 17 | ```bash 18 | # clone the Flame Graph repo 19 | git clone https://github.com/brendangregg/FlameGraph && cd FlameGraph 20 | 21 | # apply the bitcoin colouring patch 22 | patch -p1 < path/to/core-review/flamegraph/bitcoin-colour.patch 23 | 24 | # run bitcoind 25 | ./bitcoind 26 | 27 | # capture 180s worth of data 28 | sample bitcoind 180 -wait -f sample.output 29 | 30 | cat sample.output | ./stackcollapse-sample.awk | ./flamegraph.pl --color bitcoin > sample.svg 31 | 32 | firefox --new-window "file://$(realpath sample.svg)" 33 | ``` 34 | 35 | You can also filter the stack output before creating the graph: 36 | 37 | ```bash 38 | cat sample.output | ./stackcollapse-sample.awk | \ 39 | grep -i 'ProcessMessage' | \ 40 | ./flamegraph.pl --color bitcoin > sample.svg 41 | ``` 42 | 43 | You should end up with graphs that look similar too this: 44 | 45 | ![Flame Graph](flamegraph.png) 46 | -------------------------------------------------------------------------------- /applescript/README.md: -------------------------------------------------------------------------------- 1 | # AppleScript 2 | 3 | [AppleScript](https://developer.apple.com/library/archive/documentation/AppleScript/Conceptual/AppleScriptLangGuide/introduction/ASLR_intro.html#//apple_ref/doc/uid/TP40000983) is a simple scripting language developed by Apple that can be used to automate tasks on macOS. 4 | 5 | You can read about automating the UI using AppleScript [here](https://developer.apple.com/library/archive/documentation/LanguagesUtilities/Conceptual/MacAutomationScriptingGuide/AutomatetheUserInterface.html#//apple_ref/doc/uid/TP40016239-CH69-SW1). 6 | 7 | ### Repetitively load and unload wallets 8 | 9 | This [script](load_unload_wallets.applescript) continuously loads and unloads 2 wallets using the GUI. 10 | 11 | It can be used to produce a [scheduler crash](https://github.com/bitcoin/bitcoin/issues/16307) in Bitcoin Core. 12 | 13 | ```applescript 14 | tell application "System Events" to tell process "Bitcoin Core" 15 | repeat 16 | click menu item "Close Wallet..." of menu "File" of menu bar 1 17 | click button "Yes" of group 1 of window 1 18 | delay 0.2 19 | click menu item "Close Wallet..." of menu "File" of menu bar 1 20 | click button "Yes" of group 1 of window 1 21 | delay 0.2 22 | click menu item "[default wallet]" of menu "Open Wallet" of menu item "Open Wallet" of menu "File" of menu bar 1 23 | delay 0.2 24 | click menu item "test" of menu "Open Wallet" of menu item "Open Wallet" of menu "File" of menu bar 1 25 | delay 0.2 26 | end repeat 27 | end tell 28 | ``` 29 | -------------------------------------------------------------------------------- /irc.md: -------------------------------------------------------------------------------- 1 | # IRC 2 | 3 | Bitcoin Core development and discussion happens across various IRC channels. 4 | 5 | The following channels can be found on [Libera](https://libera.chat/). 6 | 7 | ## #bitcoin-core-dev - Bitcoin Core development discussion and commits. 8 | This channel hosts the weekly core dev meeting. 9 | 10 | If you have a topic you'd like to see discussed at the meeting, you can join the channel and propose it using `#proposedmeetingtopic`. 11 | 12 | Logs for the channel are available from: 13 | * http://www.erisian.com.au/bitcoin-core-dev/ 14 | * http://gnusha.org/bitcoin-core-dev/ 15 | 16 | ## #bitcoin-builds - Bitcoin Core build system discussions 17 | This channel has mainly been discussing work on [depends](https://github.com/bitcoin/bitcoin/tree/master/depends) 18 | and [GUIX build system integration](https://github.com/bitcoin/bitcoin/pull/15277). 19 | 20 | Logs are available from: 21 | * http://gnusha.org/bitcoin-builds/ 22 | 23 | ## #bitcoin-core-pr-reviews - Weekly pull request review meetup 24 | This channel hosts a weekly review of a pull request that is currently open on the [Bitcoin Core](https://github.com/bitcoin/bitcoin) repository. You can find more info about the meetup [here](https://bitcoincore.reviews/). 25 | 26 | Logs are available from the core-review site, by clicking on the previous meeting links. 27 | 28 | i.e https://bitcoincore.reviews/15996.html 29 | 30 | ## ##hwi 31 | Discussion on the [Hardware Wallet Interaction](https://github.com/bitcoin-core/HWI) scripts project. 32 | -------------------------------------------------------------------------------- /compiler-defaults.md: -------------------------------------------------------------------------------- 1 | # Compiler defaults 2 | 3 | 4 | ## Clang 5 | 6 | | Version | C std | c++ std | 7 | | ------- | ----- | ------- | 8 | | 20.1.0 | gnu17 | gnu++17 | 9 | | 19.1.0 | gnu17 | gnu++17 | 10 | | 18.1.0 | gnu17 | gnu++17 | 11 | | 17.0.0 | gnu17 | gnu++17 | 12 | | 16.0.0 | gnu17 | [gnu++17](https://releases.llvm.org/16.0.0/tools/clang/docs/ReleaseNotes.html#id85) | 13 | | 15.0.0 | gnu17 | gnu++14 | 14 | | 14.0.0 | gnu17 | gnu++14 | 15 | | 13.0.0 | gnu17 | gnu++14 | 16 | | 12.0.0 | gnu17 | gnu++14 | 17 | | 11.0.0 | [gnu17](https://releases.llvm.org/11.0.0/tools/clang/docs/ReleaseNotes.html#c-language-changes-in-clang) | gnu++14 | 18 | | 10.0.0 | gnu11 | gnu++14 | 19 | | 9.0.0 | gnu11 | gnu++14 | 20 | | 8.0.0 | gnu11 | gnu++14 | 21 | | 7.0.0 | gnu11 | gnu++14 | 22 | | 6.0.0 | gnu11 | gnu++14 | 23 | | 5.0.0 | gnu11 | gnu++98 | 24 | | 4.0.0 | gnu11 | gnu++98 | 25 | 26 | 27 | # GCC 28 | 29 | | Version | C std | c++ std | 30 | | ------- | ----- | ------- | 31 | | 15 | [gnu23](https://gcc.gnu.org/gcc-15/changes.html#c) | gnu++17 | 32 | | 14 | gnu17 | gnu++17 | 33 | | 13 | gnu17 | gnu++17 | 34 | | 12 | gnu17 | gnu++17 | 35 | | 11 | gnu17 | [gnu++17](https://gcc.gnu.org/gcc-11/changes.html) | 36 | | 10 | gnu17 | gnu++14 | 37 | | 9 | gnu17 | gnu++14 | 38 | | 8 | [gnu17](https://gcc.gnu.org/gcc-8/changes.html#c) | gnu++14 | 39 | | 7 | gnu11 | gnu++14 | 40 | | 6 | gnu11 | gnu++14 | 41 | | 5 | gnu11 | gnu++98 | 42 | | 4 | gnu89 | gnu++98 | 43 | -------------------------------------------------------------------------------- /vagrant/gui/Vagrantfile: -------------------------------------------------------------------------------- 1 | # -*- mode: ruby -*- 2 | # vi: set ft=ruby : 3 | 4 | # Do a "vagrant up" to provision, 5 | # then "vagrant reload" to restart and use the GUI. 6 | 7 | Vagrant.configure("2") do |config| 8 | 9 | config.vm.box = "debian/contrib-buster64" 10 | config.vm.synced_folder ".", "/vagrant", disabled: true 11 | 12 | config.vm.provider "virtualbox" do |vb| 13 | vb.gui = true 14 | vb.memory = "6144" 15 | vb.cpus = 8 16 | vb.customize ["modifyvm", :id, "--vram", "128"] 17 | vb.customize ["modifyvm", :id, "--audio", "none"] 18 | vb.customize ["modifyvm", :id, "--clipboard", "disabled"] 19 | vb.customize ["modifyvm", :id, "--draganddrop", "disabled"] 20 | vb.customize ["modifyvm", :id, "--recording", "off"] 21 | end 22 | 23 | config.vm.provision "shell", inline: <<-SHELL 24 | apt-get update && 25 | apt-get upgrade -y && 26 | apt-get install --no-install-recommends -y \ 27 | automake \ 28 | autotools-dev \ 29 | bsdmainutils \ 30 | build-essential \ 31 | cmake \ 32 | curl \ 33 | git \ 34 | libtool \ 35 | libdb-dev \ 36 | libdb++-dev \ 37 | libevent-dev \ 38 | libboost-system-dev \ 39 | libboost-filesystem-dev \ 40 | libboost-chrono-dev \ 41 | libboost-test-dev \ 42 | libboost-thread-dev \ 43 | libminiupnpc-dev \ 44 | libqrencode-dev \ 45 | libqt5gui5 \ 46 | libqt5core5a \ 47 | libqt5dbus5 \ 48 | libzmq3-dev \ 49 | pkg-config \ 50 | python3 \ 51 | qttools5-dev \ 52 | qttools5-dev-tools \ 53 | task-gnome-desktop 54 | SHELL 55 | end 56 | -------------------------------------------------------------------------------- /diffoscope.md: -------------------------------------------------------------------------------- 1 | # diffoscope 2 | 3 | [`diffoscope`](https://diffoscope.org) is a tool for generating diffs of just about anything (files, archives or directories). 4 | 5 | ## Creating diffs for GitHub 6 | 7 | ```bash 8 | diffoscope first_file second_file --markdown diff.md # --exclude-directory-metadata 9 | ``` 10 | 11 | This may need some manual cleanup depending on the actual diff produced. 12 | 13 | ## Diffing the depends directory 14 | 15 | Diffing `depends` directories after changing the build process, i.e [#15844](https://github.com/bitcoin/bitcoin/pull/15844). 16 | 17 | Depends is built once, and moved to `x86_64-apple-darwin18.5.0-master`. Then the PR is merged, and depends is rebuilt. 18 | 19 | ```shell 20 | diffoscope x86_64-apple-darwin18.5.0-master/lib x86_64-apple-darwin18.5.0/lib --markdown - | sed -e 's/^[ ^t]*//' -e '1,/@@/d' -e '/## stat {}/,$d' > diff.txt 21 | ``` 22 | 23 | ```diff 24 | cat diff.txt 25 | 26 | libboost_unit_test_framework-mt.a 27 | libcrypto.a 28 | libdb-4.8.a 29 | libdb.a 30 | libdb_cxx-4.8.a 31 | libdb_cxx.a 32 | libevent.a 33 | -libevent.la 34 | libevent_core.a 35 | -libevent_core.la 36 | libevent_extra.a 37 | -libevent_extra.la 38 | libevent_pthreads.a 39 | -libevent_pthreads.la 40 | libminiupnpc.a 41 | libprotobuf-lite.a 42 | -libprotobuf-lite.la 43 | libprotobuf.a 44 | -libprotobuf.la 45 | -libprotoc.la 46 | libqrencode.a 47 | -libqrencode.la 48 | libqtharfbuzz.a 49 | libqtlibpng.a 50 | libqtpcre2.a 51 | librapidcheck.a 52 | libssl.a 53 | libz.a 54 | libzmq.a 55 | -libzmq.la 56 | pkgconfig 57 | pkgconfig/Qt5AccessibilitySupport.pc 58 | pkgconfig/Qt5CglSupport.pc 59 | pkgconfig/Qt5ClipboardSupport.pc 60 | pkgconfig/Qt5Core.pc 61 | pkgconfig/Qt5DBus.pc 62 | pkgconfig/Qt5DeviceDiscoverySupport.pc 63 | ``` 64 | -------------------------------------------------------------------------------- /podman/README.md: -------------------------------------------------------------------------------- 1 | # Podman 2 | 3 | ## General Usage 4 | 5 | The imagefiles in this directory can be used to run `depends` builds for various `HOST`s. 6 | 7 | For example, to build the Debian image and use it for a build targeting the `RISCV-64` host: 8 | 9 | ```bash 10 | # Build container 11 | podman build --pull --no-cache -t debian_depends - < debian.imagefile 12 | 13 | # Run with a Bash shell 14 | podman run -it debian_depends 15 | 16 | # Inside the container: build depends for RISCV-64 bit, skipping Qt packages 17 | make HOST=riscv64-linux-gnu NO_QT=1 -C depends/ 18 | cmake -B build --toolchain /bitcoin/depends/riscv64-linux-gnu/toolchain.cmake 19 | cmake --build build 20 | ``` 21 | 22 | ### macOS SDK 23 | Cross compiling for macOS requires the macOS SDK. 24 | There are [notes in the bitcoin/bitcoin repo](https://github.com/bitcoin/bitcoin/tree/master/contrib/macdeploy#sdk-extraction) about how to create it. 25 | 26 | You can copy it into a container with: 27 | ```bash 28 | podman cp path/to/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz :bitcoin/depends/SDKs 29 | ``` 30 | 31 | ### Platform Triplets 32 | Common `host-platform-triplets` for cross compilation are: 33 | 34 | - `x86_64-w64-mingw32` for Win64 35 | - `x86_64-apple-darwin` for x86_64 macOS 36 | - `arm64-apple-darwin` for arm64 macOS 37 | - `i686-pc-linux-gnu` for Linux 32 bit 38 | - `x86_64-pc-linux-gnu` for Linux 64 bit 39 | - `arm-linux-gnueabihf` for Linux ARM 32 bit 40 | - `aarch64-linux-gnu` for Linux ARM 64 bit 41 | - `riscv32-linux-gnu` for Linux RISC-V 32 bit 42 | - `riscv64-linux-gnu` for Linux RISC-V 64 bit 43 | - `powerpc64-linux-gnu` for Linux POWER 64-bit 44 | - `powerpc64le-linux-gnu` for Linux POWER 64-bit (little endian) 45 | - `s390x-linux-gnu` for Linux S390X 46 | 47 | You can read more about host target triplets [here](https://www.gnu.org/software/autoconf/manual/html_node/Specifying-Target-Triplets.html). 48 | -------------------------------------------------------------------------------- /coz.md: -------------------------------------------------------------------------------- 1 | # Coz: Causal Profiling 2 | 3 | Build and install Coz as per: https://github.com/plasma-umass/coz. 4 | 5 | #### Insert Progress Points 6 | 7 | Add some progress points. For example: 8 | 9 | ```diff 10 | --- a/src/validation.cpp 11 | +++ b/src/validation.cpp 12 | @@ -5,6 +5,8 @@ 13 | 14 | #include 15 | 16 | +#include 17 | + 18 | #include 19 | #include 20 | 21 | @@ -3881,6 +3883,7 @@ bool Chainstate::AcceptBlock(const std::shared_ptr& pblock, BlockV 22 | 23 | bool ChainstateManager::ProcessNewBlock(const std::shared_ptr& block, bool force_processing, bool min_pow_checked, bool* new_block) 24 | { 25 | + COZ_PROGRESS_NAMED("Enter ProcessNewBlock") 26 | AssertLockNotHeld(cs_main); 27 | 28 | { 29 | @@ -3915,6 +3918,7 @@ bool ChainstateManager::ProcessNewBlock(const std::shared_ptr& blo 30 | return error("%s: ActivateBestChain failed (%s)", __func__, state.ToString()); 31 | } 32 | 33 | + COZ_PROGRESS_NAMED("Exit ProcessNewBlock") 34 | return true; 35 | } 36 | ``` 37 | 38 | #### Compile and Run 39 | 40 | Configure for debugging and add `-ldl` to LDFLAGS. 41 | 42 | ```bash 43 | make -C depends/ NO_QT=1 NO_WALLET=1 NO_ZMQ=1 44 | ./autogen.sh 45 | CONFIG_SITE=/home/ubuntu/bitcoin/depends/x86_64-pc-linux-gnu/share/config.site ./configure --enable-debug LDFLAGS="-ldl" 46 | make -j8 47 | 48 | # run 49 | coz run --- src/bitcoind 50 | ``` 51 | 52 | If you see any output like this: 53 | ```bash 54 | inspect.cpp:513] Ignoring DWARF format error when reading line table: 55 | DW_FORM_sec_offset not expected for attribute (DW_AT)0x2137 56 | ``` 57 | or this: 58 | ```bash 59 | terminate called after throwing an instance of 'dwarf::format_error' 60 | what(): unknown compilation unit version 5 61 | ``` 62 | it's due to an issue with libelfin and DWARF 4+ (which is used by GCC 9+). 63 | Reconfigure to use DWARF 3 instead: 64 | ```bash 65 | CFLAGS="-gdwarf-3" CXXFLAGS="-gdwarf-3" 66 | ``` 67 | 68 | #### Results 69 | 70 | Results are dumped into `profile.coz`. 71 | Take it to https://plasma-umass.org/coz/ to take a look. 72 | 73 | -------------------------------------------------------------------------------- /fortify.py: -------------------------------------------------------------------------------- 1 | import collections 2 | import os 3 | import re 4 | import subprocess 5 | import sys 6 | 7 | # Calculate _chk % replacement when using FORTIFY_SOURCE 8 | # https://developers.redhat.com/blog/2020/02/11/toward-_fortify_source-parity-between-clang-and-gcc/ 9 | 10 | OBJDUMP_CMD = os.getenv('OBJDUMP', '/usr/bin/objdump') 11 | filename = sys.argv[1] 12 | 13 | def objdump(args): 14 | p = subprocess.Popen([OBJDUMP_CMD, args, filename], stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE, universal_newlines=True) 15 | (stdout, stderr) = p.communicate() 16 | if p.returncode: 17 | print(stderr) 18 | exit(0) 19 | return stdout 20 | 21 | # Check the dynamic symbol table for _chk variants 22 | dy_syms = [] 23 | 24 | for line in objdump('-T').splitlines(): 25 | match = re.search(r'__[a-z]*_chk', line) 26 | if match: 27 | # we'll account for stack_chk_fail separately 28 | if '__stack_chk' in match.group(0): 29 | continue 30 | dy_syms.append(match.group(0)) 31 | 32 | # disassemble and count _chk and non _chk variants 33 | counts = collections.Counter() 34 | stack_chk_fail = 0 35 | 36 | for line in objdump('-d').splitlines(): 37 | 38 | _chk = [sym for sym in dy_syms if (sym+'@plt' in line )] 39 | if _chk: 40 | counts[_chk[0]] += 1 41 | 42 | non_chk = [sym.strip('_chk') for sym in dy_syms if ('<' + sym.strip('_chk') + '@plt' in line)] 43 | if non_chk: 44 | counts[non_chk[0]] += 1 45 | 46 | if '__stack_chk_fail' in line: 47 | stack_chk_fail += 1 48 | 49 | result = [] 50 | 51 | for sym in dy_syms: 52 | plain = sym.strip('_chk') 53 | 54 | # drop one for .plt 55 | plain_count = (counts[plain] - 1) if counts[plain] > 0 else 0 56 | chk_count = counts[sym] - 1 57 | replaced = round((chk_count / (plain_count + chk_count) * 100),2) 58 | 59 | result.append((plain, plain_count, chk_count, replaced)) 60 | result.sort(key=lambda t: t[0]) 61 | 62 | print('{:10}| {:7} | {:4} | {:10} |'.format('Symbol', 'non _chk', '_chk', '% replaced')) 63 | print('\n'.join('{:10}: {:8} | {:4} | {:9}% |'.format(a,b,c,d) for (a,b,c,d) in result)) 64 | print() 65 | print('{} calls to __stack_chk_fail()'.format(stack_chk_fail)) 66 | -------------------------------------------------------------------------------- /rtld/audit.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | 4 | // must be defined 5 | unsigned int la_version(unsigned int version) { 6 | printf("Auditing interface version: %u\n", version); 7 | return version; 8 | } 9 | 10 | // invoked by the dynamic linker to inform the library 11 | // that it is about to search for a shared object 12 | // name - file or path to be searched for 13 | // cookie - identifies the shared object that init the search 14 | char *la_objsearch(char *name, uintptr_t *cookie, unsigned int flag) { 15 | printf("Searching for shared object %s. Cookie = %p, flag = %s\n", name, cookie, 16 | (flag == LA_SER_ORIG) ? "LA_SER_ORIG" : // filename given to dlopen or from DT_NEEDED 17 | (flag == LA_SER_LIBPATH) ? "LA_SER_LIBPATH" : // created via dir in LD_LIBRARY_PATH 18 | (flag == LA_SER_RUNPATH) ? "LA_SER_RUNPATH" : // create via dir in DT_RPATH or DT_RUNPATH 19 | (flag == LA_SER_CONFIG) ? "LA_SER_CONFIG" : // found via ldconfig cache 20 | (flag == LA_SER_DEFAULT) ? "LA_SER_DEFAULT" : // found via deafult directory 21 | (flag == LA_SER_SECURE) ? "LA_SER_SECURE" // specific to a secure object 22 | : "??? FLAG"); 23 | return name; 24 | } 25 | 26 | void la_activity(uintptr_t *cookie, unsigned int flag) { 27 | printf("Link map activity:\n Cookie = %p, flag = %s\n\n", cookie, 28 | (flag == LA_ACT_CONSISTENT) ? "LA_ACT_CONSISTENT" : // activity completed 29 | (flag == LA_ACT_ADD) ? "LA_ACT_ADD" : // objects added to link map 30 | (flag == LA_ACT_DELETE) ? "LA_ACT_DELETE" : // objects removed from link map 31 | "??? FLAG"); 32 | } 33 | 34 | unsigned int la_objopen(struct link_map *map, Lmid_t lmid, uintptr_t *cookie) { 35 | printf("loading shared object: %s, flag = %s\n\n", map->l_name, 36 | (lmid == LM_ID_BASE) ? "LM_ID_BASE" : // part of initial namespace 37 | (lmid == LM_ID_NEWLM) ? "LM_ID_NEWLM" : // new namespace via dlopen 38 | "??? FLAG"); 39 | return LA_FLG_BINDTO | LA_FLG_BINDFROM; 40 | } 41 | 42 | unsigned int la_objclose(uintptr_t *cookie) { 43 | printf("About to unload object. Cookie = %p\n", cookie); 44 | return 0; // ignored 45 | } 46 | 47 | void la_preinit(uintptr_t *cookie) { 48 | printf("Dynamic linker finished. Passing back to application\n"); 49 | } 50 | 51 | uintptr_t la_symbind64(Elf64_Sym *sym, unsigned int ndx, uintptr_t *refcook, 52 | uintptr_t *defcook, int *flags, const char *symname) { 53 | printf("la_symbind64()\n"); 54 | return sym->st_value; 55 | } 56 | -------------------------------------------------------------------------------- /big-wallet.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python3 2 | 3 | # src/bitcoind -regtest -server 4 | # python3 big-wallet.py ~/Library/Application\ Support/Bitcoin/ tx_count 5 | 6 | import argparse 7 | import os 8 | import random 9 | import sys 10 | import time 11 | 12 | auth = os.getenv('AUTHPROXY', '../bitcoin/test/functional/test_framework') 13 | sys.path.append(auth) 14 | from authproxy import AuthServiceProxy 15 | 16 | parser = argparse.ArgumentParser() 17 | parser.add_argument('datadir') 18 | parser.add_argument('txcount', type=int) 19 | args = parser.parse_args() 20 | 21 | # Wait for cookie file to be created 22 | while not os.path.exists(args.datadir + '/regtest/.cookie'): 23 | time.sleep(0.5) 24 | 25 | # Read .cookie file to get user and pass 26 | with open(args.datadir + '/regtest/.cookie') as f: 27 | userpass = f.readline().lstrip().rstrip() 28 | rpc = AuthServiceProxy('http://{}@127.0.0.1:18443'.format(userpass)) 29 | 30 | # Wait for bitcoind to be ready 31 | ready: bool = False 32 | while not ready: 33 | try: 34 | rpc.getblockchaininfo() 35 | ready = True 36 | except Exception: 37 | time.sleep(0.5) 38 | pass 39 | print('bitcoind ready') 40 | 41 | for item in rpc.listwalletdir()['wallets']: 42 | if 'big' == item['name']: 43 | break; 44 | else: 45 | rpc.createwallet('big') 46 | 47 | if 'big' not in rpc.listwallets(): 48 | rpc.loadwallet('big') 49 | 50 | def_rpc = AuthServiceProxy('http://{}@127.0.0.1:18443/wallet/'.format(userpass)) 51 | big_rpc = AuthServiceProxy('http://{}@127.0.0.1:18443/wallet/big'.format(userpass)) 52 | 53 | print('mining') 54 | gen_addr = big_rpc.getnewaddress() 55 | if rpc.getblockcount() == 0: 56 | big_rpc.generatetoaddress(200, gen_addr) 57 | 58 | self_addr = big_rpc.getnewaddress() 59 | send_addr = def_rpc.getnewaddress() 60 | 61 | while big_rpc.getwalletinfo()['txcount'] < args.txcount: 62 | for i in range(0, 1000): 63 | if i % 100 == 0: 64 | print(i) 65 | big_rpc.generatetoaddress(1, gen_addr) 66 | for j in range(0, 10): 67 | 68 | if big_rpc.getbalance() < 500: 69 | print("rebalancing") 70 | def_rpc.sendtoaddress(big_rpc.getnewaddress(), def_rpc.getbalance(), "", "", True) 71 | big_rpc.generatetoaddress(6, gen_addr) 72 | 73 | amt = round(random.randint(10000, 500 * 100000000) / 100000000, 8) 74 | addr = random.choice([send_addr, self_addr]) 75 | big_rpc.sendtoaddress(addr, amt) 76 | -------------------------------------------------------------------------------- /vagrant/Vagrantfile: -------------------------------------------------------------------------------- 1 | # -*- mode: ruby -*- 2 | # vi: set ft=ruby : 3 | 4 | Vagrant.configure("2") do |config| 5 | 6 | config.vm.provider "virtualbox" do |vb| 7 | vb.cpus = 8 8 | vb.customize ["modifyvm", :id, "--audio", "none"] 9 | vb.customize ["modifyvm", :id, "--clipboard-mode", "disabled"] 10 | vb.customize ["modifyvm", :id, "--draganddrop", "disabled"] 11 | vb.customize ["modifyvm", :id, "--recording", "off"] 12 | vb.customize ["modifyvm", :id, "--usbcardreader", "off"] 13 | vb.gui = false 14 | vb.memory = 4096 15 | end 16 | 17 | config.vm.define "openbsd" do |box| 18 | box.vm.box = "generic/openbsd7" 19 | 20 | box.vm.provision "shell" do |s| 21 | s.path = "openbsd.sh" 22 | s.name = "Provision" 23 | s.args = ["provision"] 24 | end 25 | box.vm.provision "shell", run: "always" do |s| 26 | s.path = "openbsd.sh" 27 | s.name = "Setup" 28 | s.args = ENV['PR'] ? ["setup", ENV['PR']] : ["setup"] 29 | end 30 | end 31 | 32 | config.vm.define "freebsd12" do |box| 33 | box.vm.box = "freebsd/FreeBSD-12.2-STABLE" 34 | box.ssh.shell = "sh" 35 | box.vm.synced_folder ".", "/vagrant", disabled: true 36 | 37 | box.vm.provision "shell" do |s| 38 | s.path = "freebsd.sh" 39 | s.name = "Provision" 40 | s.args = ["provision"] 41 | end 42 | box.vm.provision "shell", run: "always" do |s| 43 | s.path = "freebsd.sh" 44 | s.name = "Setup" 45 | s.args = ENV['PR'] ? ["setup", ENV['PR']] : ["setup"] 46 | end 47 | end 48 | 49 | config.vm.define "freebsd13" do |box| 50 | box.vm.box = "freebsd/FreeBSD-13.0-STABLE" 51 | box.ssh.shell = "sh" 52 | box.vm.synced_folder ".", "/vagrant", disabled: true 53 | 54 | box.vm.provision "shell" do |s| 55 | s.path = "freebsd.sh" 56 | s.name = "Provision" 57 | s.args = ["provision"] 58 | end 59 | box.vm.provision "shell", run: "always" do |s| 60 | s.path = "freebsd.sh" 61 | s.name = "Setup" 62 | s.args = ENV['PR'] ? ["setup", ENV['PR']] : ["setup"] 63 | end 64 | end 65 | 66 | config.vm.define "netbsd" do |box| 67 | box.vm.box = "generic/netbsd9" 68 | box.ssh.shell = "sh" 69 | config.vm.synced_folder ".", "/vagrant", disabled: true 70 | 71 | box.vm.provision "shell" do |s| 72 | s.path = "netbsd.sh" 73 | s.name = "Provision" 74 | s.args = ["provision"] 75 | end 76 | box.vm.provision "shell", run: "always" do |s| 77 | s.path = "netbsd.sh" 78 | s.name = "Setup" 79 | s.args = ENV['PR'] ? ["setup", ENV['PR']] : ["setup"] 80 | end 81 | end 82 | end 83 | -------------------------------------------------------------------------------- /guix/imagefile: -------------------------------------------------------------------------------- 1 | FROM alpine:3.23 AS base 2 | 3 | RUN apk --no-cache --update add \ 4 | bash \ 5 | bzip2 \ 6 | ca-certificates \ 7 | curl \ 8 | git \ 9 | make \ 10 | shadow 11 | 12 | ARG guix_download_path=https://ftpmirror.gnu.org/gnu/guix/ 13 | ARG guix_version=1.4.0 14 | ARG guix_checksum_aarch64=72d807392889919940b7ec9632c45a259555e6b0942ea7bfd131101e08ebfcf4 15 | ARG guix_checksum_x86_64=236ca7c9c5958b1f396c2924fcc5bc9d6fdebcb1b4cf3c7c6d46d4bf660ed9c9 16 | ARG builder_count=32 17 | 18 | ENV PATH=/root/.config/guix/current/bin:$PATH 19 | 20 | # Application Setup 21 | # https://guix.gnu.org/manual/en/html_node/Application-Setup.html 22 | ENV GUIX_LOCPATH=/root/.guix-profile/lib/locale 23 | ENV LC_ALL=en_US.UTF-8 24 | 25 | RUN guix_file_name=guix-binary-${guix_version}.$(uname -m)-linux.tar.xz && \ 26 | eval "guix_checksum=\${guix_checksum_$(uname -m)}" && \ 27 | cd /tmp && \ 28 | wget -q -O "$guix_file_name" "${guix_download_path}/${guix_file_name}" && \ 29 | echo "${guix_checksum} ${guix_file_name}" | sha256sum -c && \ 30 | tar xJf "$guix_file_name" && \ 31 | mv var/guix /var/ && \ 32 | mv gnu / && \ 33 | mkdir -p ~root/.config/guix && \ 34 | ln -sf /var/guix/profiles/per-user/root/current-guix ~root/.config/guix/current && \ 35 | source ~root/.config/guix/current/etc/profile 36 | 37 | # Build Environment Setup 38 | # https://guix.gnu.org/manual/en/html_node/Build-Environment-Setup.html 39 | 40 | RUN groupadd --system guixbuild 41 | RUN for i in $(seq -w 1 ${builder_count}); do \ 42 | useradd -g guixbuild -G guixbuild \ 43 | -d /var/empty -s $(which nologin) \ 44 | -c "Guix build user ${i}" --system \ 45 | "guixbuilder${i}" ; \ 46 | done 47 | 48 | RUN git clone https://github.com/bitcoin/bitcoin.git /bitcoin 49 | 50 | RUN mkdir base_cache sources SDKs 51 | 52 | WORKDIR /bitcoin 53 | 54 | FROM base AS no-substitutes 55 | 56 | ENV ADDITIONAL_GUIX_COMMON_FLAGS='--no-substitutes' 57 | CMD ["/root/.config/guix/current/bin/guix-daemon","--build-users-group=guixbuild", "--no-substitutes"] 58 | 59 | FROM base AS substitutes 60 | 61 | RUN guix archive --authorize < ~root/.config/guix/current/share/guix/ci.guix.gnu.org.pub 62 | CMD ["/root/.config/guix/current/bin/guix-daemon","--build-users-group=guixbuild"] 63 | -------------------------------------------------------------------------------- /clang-tools.md: -------------------------------------------------------------------------------- 1 | # Clang Tools 2 | 3 | The [Clang Tools](https://clang.llvm.org/docs/ClangTools.html) are standalone command line tools that provide functionality such as syntax checking, automated formatting and refactoring. 4 | 5 | __Note:__ Before submitting any pull requests to Bitcoin Core, please read [CONTRIBUTING.md](https://github.com/bitcoin/bitcoin/blob/master/CONTRIBUTING.md). 6 | 7 | In particular the section on [__refactoring__](https://github.com/bitcoin/bitcoin/blob/master/CONTRIBUTING.md#refactoring), as it likely applies to changes generated with these tools: 8 | 9 | > Trivial pull requests or pull requests that refactor the code with no clear benefits may be immediately closed by the maintainers to reduce unnecessary workload on reviewing. 10 | 11 | There is on-going discussion in [#15465](https://github.com/bitcoin/bitcoin/issues/15465) about the potential benefit of project wide code reformatting. 12 | 13 | ## Setup 14 | 15 | Install `clang` & LLVM. 16 | 17 | ```bash 18 | cd bitcoin 19 | 20 | cmake -B build -DCMAKE_EXPORT_COMPILE_COMMANDS=ON \ 21 | -DCMAKE_C_COMPILER=clang \ 22 | -DCMAKE_CXX_COMPILER=clang++ 23 | ``` 24 | 25 | ## clang-format 26 | 27 | If you are going to use `clang-format` over any of the Bitcoin Core code, ensure that you use the [`clang-format`](https://github.com/bitcoin/bitcoin/blob/master/src/.clang-format) file provided under `/src`. 28 | 29 | ## clang-tidy 30 | 31 | `clang-tidy` has been used to submit pull requests in the past, such as [#10735](https://github.com/bitcoin/bitcoin/pull/10735). 32 | 33 | While this usage can generally be __ok__, the project is not on a crusade to _silence every warning_. 34 | 35 | Naive usage of tools like `clang-tidy` will like result in 100's, if not 1000's of "warnings" generated for any particular file. 36 | 37 | The majority likely being things that dont need to be fixed, or which don't point to actual bugs. i.e: 38 | 39 | ```bash 40 | clang-tidy -checks=* src/init.cpp 41 | 5981 warnings and 3 errors generated. 42 | 43 | # some cherry-picked examples 44 | src/init.cpp:10:10: error: 'init.h' file not found with include; use "quotes" instead [clang-diagnostic-error] 45 | src/init.cpp:21:1: warning: #includes are not sorted properly [llvm-include-order] 46 | src/init.cpp:52:10: warning: inclusion of deprecated C++ header 'stdint.h'; consider using 'cstdint' instead [modernize-deprecated-headers] 47 | src/init.cpp:313:30: warning: all parameters should be named in a function [readability-named-parameter] 48 | src/init.cpp:592:61: warning: parameter 'pBlockIndex' is unused [misc-unused-parameters] # these certainly are used 49 | src/init.cpp:1106:5: warning: missing username/bug in TODO [google-readability-todo] 50 | src/util/system.h:17:10: error: 'attributes.h' file not found [clang-diagnostic-error] 51 | Suppressed 5947 warnings (5947 in non-user code). 52 | ``` 53 | 54 | ### run-clang-tidy 55 | 56 | ```bash 57 | ( cd ./src/ && run-clang-tidy -p ../build -j $(nproc) ) 58 | ``` 59 | -------------------------------------------------------------------------------- /compiler-explorer.md: -------------------------------------------------------------------------------- 1 | # Compiler Explorer 2 | 3 | [Compiler Explorer](https://godbolt.org) is web based tool that can be used to examine and compare assembly output across different compilers (i.e Clang vs GCC) as well as different versions of the same compiler (i.e GCC 6 vs 7). 4 | 5 | ## GCC Bug 90348 6 | [A bug in GCC](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90348) `7`, `8` and `9` causes a mis-compilation with the following `C` code: 7 | 8 | ```c 9 | #include 10 | #include 11 | 12 | void __attribute__ ((noinline)) set_one(unsigned char* ptr) 13 | { 14 | *ptr = 1; 15 | } 16 | 17 | int __attribute__ ((noinline)) check_zero(unsigned char const* in, unsigned int len) 18 | { 19 | for (unsigned int i = 0; i < len; ++i) { 20 | if (in[i] != 0) return 0; 21 | } 22 | return 1; 23 | } 24 | 25 | void set_one_on_stack() { 26 | unsigned char buf[1]; 27 | set_one(buf); 28 | } 29 | 30 | int main() { 31 | for (int i = 0; i <= 4; ++i) { 32 | unsigned char in[4]; 33 | for (int j = 0; j < i; ++j) { 34 | in[j] = 0; 35 | set_one_on_stack(); // Apparently modifies in[0] 36 | } 37 | assert(check_zero(in, i)); 38 | } 39 | } 40 | ``` 41 | 42 | ## Bitcoin Core Issue & Fix 43 | 44 | This bug was originally reported in [#14580](https://github.com/bitcoin/bitcoin/issues/14580) as a failure when running the `sha256d64` test on a 32-bit distro. It was eventually identified as a bug in GCC. 45 | 46 | The [fix applied](https://github.com/bitcoin/bitcoin/pull/15983) to Bitcoin Core involved using [`-fstack-reuse=none`](https://gcc.gnu.org/onlinedocs/gcc/Code-Gen-Options.html) when compiling with GCC. 47 | 48 | > This option controls stack space reuse for user declared local/auto variables and compiler generated temporaries. 49 | 50 | > reuse_level can be ‘all’, ‘named_vars’, or ‘none’. 51 | 52 | > ‘none’ disables stack reuse completely. 53 | 54 | ## Compiler Explorer Setup 55 | 56 | This was a good oppourtunity to use Compiler Explorer to compare the assembly generated with and without the `fstack-reuse` flag. 57 | 58 | I took the flags produced by `./configure` inside a [Debian Docker container](docker/debian.dockerfile): 59 | ```bash 60 | -Wstack-protector -fstack-protector-all -Wall -Wextra -Wformat -Wvla -Wredundant-decls -Wno-unused-parameter -Wno-implicit-fallthrough -g -O2 61 | ``` 62 | and setup a Compiler Explorer instance using GCC 9.1 and the `C` code above. 63 | 64 | Both compilers used the base configure flags and one had `-fstack-reuse=none` appended. 65 | 66 | You can view the setup [here](https://godbolt.org/z/42ZKDS). 67 | 68 | Looking at the generated assembly, you can see the difference in output on line 83: 69 | 70 | ```diff 71 | < lea rdi, [rsp+3] 72 | --- 73 | > lea rdi, [rsp+4] 74 | ``` 75 | 76 | As mentioned in the original bug report, the issue is seen with GCC 7, 8 and 9. You can confirm this by changing the version of GCC used by Compiler Explorer. If you use GCC 6 you will not see the issue. -------------------------------------------------------------------------------- /windows.md: -------------------------------------------------------------------------------- 1 | # Windows 10 Notes 2 | 3 | ## Development Setup 4 | 5 | ### Download 6 | Download a [Windows 10 Development Environment](https://developer.microsoft.com/en-us/windows/downloads/virtual-machines) VM. 7 | 8 | ### Import & Settings 9 | Import `WinDev1912Eval.ovf` into Virtualbox. 10 | 11 | Open Settings: 12 | - System 13 | - Increase base memory. 14 | - Increase processor count. 15 | 16 | - Display: 17 | - Increase video memory. 18 | 19 | - Audio 20 | - Disable Audio. 21 | 22 | ### Boot 23 | 24 | Update Virtualbox Gust Additions if required. 25 | 26 | Go to `Updates & Security`, install all updates. 27 | 28 | Clear the desktop and the taskbar. 29 | 30 | Open `Apps & Features`. Uninstall pretty much everything. 31 | 32 | Update Visual Studio. Remove everything except the C++ dev workload, git and Python3. 33 | 34 | If you want to use WSL, open Ubuntu and update packages. Clone the bitcoin repo etc. 35 | 36 | Reboot. 37 | 38 | ### Install dependencies and generate project files 39 | [Vcpkg](https://github.com/Microsoft/vcpkg.git) is used to install build dependencies. 40 | See the Bitcoin Core [MSVC build docs](https://github.com/bitcoin/bitcoin/tree/master/build_msvc) for more info. 41 | 42 | Open `Developer Powershell for VS 2019`: 43 | ```powershell 44 | git clone https://github.com/bitcoin/bitcoin.git 45 | git clone https://github.com/Microsoft/vcpkg.git 46 | 47 | cd vcpkg 48 | .\bootstrap-vcpkg.bat -disableMetrics 49 | 50 | # hook up user-wide integration 51 | .\vcpkg.exe integrate install 52 | 53 | # Install Bitcoin Core dependencies 54 | .\vcpkg.exe install --triplet x64-windows-static $(Get-Content -Path ..\bitcoin\build_msvc\vcpkg-packages.txt).split() 55 | cd .. 56 | 57 | # Generate Project Files 58 | cd bitcoin\build_msvc 59 | py -3 msvc-autogen.py 60 | ``` 61 | 62 | ### Build 63 | 64 | ```powershell 65 | msbuild /m bitcoin.sln /p:Platform=x64 /p:Configuration=Release /t:build 66 | ``` 67 | 68 | Alternatively, open `bitcoin\build_msvc\bitcoin.sln` in Visual Studio. 69 | 70 | If you are asked to `Retarget Projects`, choose `Yes`. 71 | 72 | Choose `Release` and `x64`, then `Build -> Build Solution`. 73 | 74 | ### Create a VM Snapshot 75 | 76 | The Microsoft VMs expire after 90 days. 77 | 78 | Once everything is setup, take a VM snapshot. 79 | 80 | ![Windows](screenshots/windows.png) 81 | 82 | ## Tools 83 | 84 | ### dumpbin 85 | 86 | Installed alongside MSVC. More info [here](https://docs.microsoft.com/en-us/cpp/build/reference/dumpbin-command-line?view=vs-2019). 87 | 88 | ```powershell 89 | <# used for inspecting binaries #> 90 | dumpbin /headers src/bitcoind.exe 91 | ... 92 | 160 DLL characteristics 93 | High Entropy Virtual Addresses 94 | Dynamic base 95 | NX compatible 96 | 97 | ``` 98 | 99 | ### Process Explorer 100 | 101 | Inspect a running process, more in-depth than Task Manager. 102 | 103 | Download from Microsoft [here](https://docs.microsoft.com/en-us/sysinternals/downloads/process-explorer). 104 | -------------------------------------------------------------------------------- /clang-build-analyzer.md: -------------------------------------------------------------------------------- 1 | # Clang Build Analyser 2 | 3 | Get [Clang Build Analyser](https://github.com/aras-p/ClangBuildAnalyzer). 4 | 5 | ```bash 6 | export CC=clang 7 | export CXX=clang++ 8 | 9 | cmake -B build -DCMAKE_CXX_FLAGS="-ftime-trace" -DWITH_CCACHE=NO 10 | 11 | # start analyzer 12 | ClangBuildAnalyzer --start . 13 | 14 | # compile 15 | cmake --build build -j$(nproc) 16 | 17 | # stop analyzer 18 | ClangBuildAnalyzer --stop . results 19 | 20 | # analyze (results truncated) 21 | ClangBuildAnalyzer --analyze results 22 | 23 | Analyzing build trace from 'results'... 24 | **** Time summary: 25 | Compilation (449 times): 26 | Parsing (frontend): 1467.1 s 27 | Codegen & opts (backend): 707.1 s 28 | 29 | **** Files that took longest to parse (compiler frontend): 30 | 10754 ms: ./build/src/wallet/CMakeFiles/bitcoin_wallet.dir/rpc/spend.cpp.o 31 | 10344 ms: ./build/src/test/CMakeFiles/test_bitcoin.dir/__/wallet/test/wallet_tests.cpp.o 32 | 10264 ms: ./build/src/test/CMakeFiles/test_bitcoin.dir/__/wallet/test/coinselector_tests.cpp.o 33 | 34 | **** Files that took longest to codegen (compiler backend): 35 | 21546 ms: ./build/src/test/CMakeFiles/test_bitcoin.dir/script_tests.cpp.o 36 | 15909 ms: ./build/src/test/CMakeFiles/test_bitcoin.dir/util_tests.cpp.o 37 | 13415 ms: ./build/src/test/CMakeFiles/test_bitcoin.dir/main.cpp.o 38 | 39 | **** Templates that took longest to instantiate: 40 | 7087 ms: util::Split (349 times, avg 20 ms) 41 | 6328 ms: std::vector::emplace_back (345 times, avg 18 ms) 42 | 5523 ms: std::unordered_map::unordered_map (168 times, avg 32 ms) 43 | 44 | **** Template sets that took longest to instantiate: 45 | 34615 ms: std::map<$>::map (2473 times, avg 13 ms) 46 | 33644 ms: std::allocator_traits<$> (17457 times, avg 1 ms) 47 | 30308 ms: std::map<$> (3602 times, avg 8 ms) 48 | 49 | **** Functions that took longest to compile: 50 | 5007 ms: script_tests::script_build::test_method() (src/test/script_tests.cpp) 51 | 2695 ms: descriptor_tests::descriptor_test::test_method() (src/test/descriptor_tests.cpp) 52 | 1829 ms: wallet::ismine_tests::ismine_standard::test_method() (src/wallet/test/ismine_tests.cpp) 53 | 54 | **** Function sets that took longest to compile / optimize: 55 | 4193 ms: tinyformat::detail::streamStateFromFormat(std::__1::basic_ostream<$>... (166 times, avg 25 ms) 56 | 3955 ms: tinyformat::detail::formatImpl(std::__1::basic_ostream<$>&, char con... (166 times, avg 23 ms) 57 | 1284 ms: std::__1::__tree<$>::destroy(std::__1::__tree_node<$>*) (716 times, avg 1 ms) 58 | 59 | *** Expensive headers: 60 | 144315 ms: /src/span.h (included 355 times, avg 406 ms), included via: 61 | 23x: args.h settings.h fs.h tinyformat.h string.h 62 | 20x: chainparams.h chainparams.h params.h uint256.h 63 | 17x: setup_common.h args.h settings.h fs.h tinyformat.h string.h 64 | 65 | 91044 ms: /src/tinyformat.h (included 302 times, avg 301 ms), included via: 66 | 61x: setup_common.h args.h settings.h fs.h 67 | 35x: args.h settings.h fs.h 68 | 23x: logging.h 69 | ``` -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Core Review 2 | Scripts, tools & info for doing [Bitcoin Core](https://github.com/bitcoin/bitcoin) development and code review. Based on using a macOS 10.14 system. 3 | 4 | - [AppleScript](/applescript/README.md) - Scripts for UI & test automation 5 | - [`assumevalid` Updates](update-assumevalid.md) - How to review pull requests that update `assumevalid`. 6 | - [`big-wallet.py`](big-wallet.py) - Create a wallet with lots of transactions. 7 | - [Binary Compare](binary-compare.md) - Notes on doing binary comparisons of Bitcoin Core 8 | - [Clang Build Analyser](clang-build-analyzer.md) - Clangs `-ftime-trace` & ClangBuildAnalyzer 9 | - [Clang Tools](clang-tools.md) - Setting up & using Clang Tools 10 | - [Compiler Bugs](compiler-bugs.md) 11 | - [Compiler Defaults](compiler-defaults.md) 12 | - [Compiler Explorer](compiler-explorer.md) - Example Compiler Explorer use case. 13 | - [CoreRPC](https://github.com/fanquake/CoreRPC) - Swift RPC wrapper. 14 | - [Coz](coz.md) - Profiling bitcoind with Coz 15 | - [depends](depends.md) - notes on using the depends system 16 | - [Determinism](determinism.md) - Notes on determinism. 17 | - [`diffoscope`](diffoscope.md) - A tool for generating diffs. 18 | - [Flame Graphs](/flamegraph/README.md) - Notes on producing [Flame Graphs](https://github.com/brendangregg/FlameGraph). 19 | - [`fortify.py`](fortify.py) - Calculate _chk % replacement when using FORTIFY_SOURCE 20 | - [Fuzzing](/fuzzing/) - Fuzzing using Clang & [libFuzzer](https://llvm.org/docs/LibFuzzer.html). 21 | - [`github-merge.py`](github-merge.md) - Script for merging PRs from GitHub. 22 | - [gnuplot](gnuplot/README.md) - `gnuplot` usage for benchmarking and analysis. 23 | - [Guix](guix/README.md) - Notes on setting up / using Guix. 24 | - [Hardening](hardening.md) - Hardening in Bitcoin Core. 25 | - [IRC](irc.md) - Where to find Bitcoin Core related discussion on IRC. 26 | - [Link Time Optimization](lto.md) - Notes on using LTO. 27 | - [Links](links.md) - Useful links that I would otherwise forget about. 28 | - [LLDB](lldb.md) - Notes on using LLDB. 29 | - [Locale Dependence](/locale-dependence/) - Locale dependence examples. 30 | - [Mesh](mesh.md) - Running bitcoind with Mesh 31 | - [NixOS](nixos.md) - Building on NixOS 32 | - [Operating Systems](operating-systems.md) - OS's that Core should support. 33 | - [Profile Guided Optimization](pgo.md) - Notes on using PGO. 34 | - [reprotest](reprotest.md) - Using reprotest. 35 | - [rtld-audit - Dynamic Linker Auditing](/rtld/) - linux dynamic linker auditing 36 | - [Scripted Diff](scripted-diff.md) 37 | - [Subtree Merges](subtree-merge.md) - How to verify subtree merges are done correctly. 38 | - [.tbd stubs](tbd-stubs.md) - Using .tbd stubs. 39 | - [URIs](/uri/) - URIs for testing `bitcoin:` uri handling. 40 | - [Vagrant](/vagrant/) - Box definitions for PR review/testing. 41 | - [Valgrind](valgrind.md) - Installing Valgrind on macOS. 42 | - [Windows](windows.md) - Windows 10 VM setup for native builds. 43 | - [Windows Cross Compile](/win-cross-compile.md) - Building Windows binaries on macOS 44 | 45 | ## TODO 46 | 47 | - Add some notes LD notes for `why_live`, `dead_strip_dyibs`, `DYLD_` etc. 48 | - glibc function usage 49 | - CVE history 50 | -------------------------------------------------------------------------------- /binary-compare.md: -------------------------------------------------------------------------------- 1 | # Binary Comparison 2 | 3 | These notes use the [debian-depends Dockerfile](/docker/debian.dockerfile). 4 | 5 | ## Build 6 | 7 | ```bash 8 | docker container start debian-depends 9 | docker exec -it debian-depends --workdir /bitcoin /bin/bash 10 | 11 | # build master 12 | make -j5 -C depends/ 13 | git rev-parse HEAD 14 | > 25f0edd59fee454495da39c6c63f19b97cd75e96 15 | ../bitcoin-maintainer-tools/build-for-compare.py 25f0edd59fee454495da39c6c63f19b97cd75e96 \ 16 | --prefix=/bitcoin/depends/x86_64-pc-linux-gnu \ 17 | --executables=src/bitcoind,src/qt/bitcoin-qt \ 18 | --parallelism=5 19 | ...... 20 | >>> [do_build] Copying object files... 21 | >>> [do_build] Performing basic analysis pass... 22 | 23 | # Checkout the PR you'd like to compare. In this case #16370 24 | git fetch origin pull/16370/head:16370 && git checkout 16370 25 | make -j5 -C depends/ 26 | git rev-parse HEAD 27 | 12061016eca4cf4c77fa042ea95682e5dc2b302b 28 | # build for comparison into /tmp/16370 29 | ../bitcoin-maintainer-tools/build-for-compare.py 12061016eca4cf4c77fa042ea95682e5dc2b302b \ 30 | --prefix=/bitcoin/depends/x86_64-pc-linux-gnu \ 31 | --executables=src/bitcoind,src/qt/bitcoin-qt \ 32 | --parallelism=5 \ 33 | --tgtdir=/tmp/16370 34 | ..... 35 | >>> [do_build] Copying object files... 36 | >>> [do_build] Performing basic analysis pass... 37 | ``` 38 | 39 | ## Result 40 | 41 | The contents of a build-for-compare directory is: 42 | 43 | ```bash 44 | `25f0edd59fee454495da39c6c63f19b97cd75e96` contains `.dis` files 45 | `25f0edd59fee454495da39c6c63f19b97cd75e96.o/src` contains `.o` files 46 | 47 | # ELF 64-bit LSB pie executable, x86-64, version 1 (GNU/Linux), 48 | # dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, 49 | # BuildID[sha1]=724e4aaaa656445719d92dc9c4fb82fe7a2fe8c5, not stripped 50 | `bitcoin-qt.25f0edd59fee454495da39c6c63f19b97cd75e96` bitcoin-qt executable 51 | `bitcoin-qt.25f0edd59fee454495da39c6c63f19b97cd75e96.stripped` bitcoin-qt executable, stripped 52 | 53 | # ELF 64-bit LSB pie executable, x86-64, version 1 (GNU/Linux), 54 | # dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, 55 | # BuildID[sha1]=6ad35cd12f6a7024b169797a1abb190cc2aa276f, not stripped 56 | `bitcoind.25f0edd59fee454495da39c6c63f19b97cd75e96` bitcoind executable 57 | `bitcoind.25f0edd59fee454495da39c6c63f19b97cd75e96.stripped` bitcoind executable, stripped 58 | ``` 59 | 60 | ## Compare 61 | 62 | ```bash 63 | # Compare executable shasums 64 | shasum /tmp/compare/*.stripped /tmp/16370/*.stripped 65 | 0e8a7a100bb6ef00e9ead3cb64474ec28590d7c6 /tmp/compare/bitcoin-qt.25f0edd59fee454495da39c6c63f19b97cd75e96.stripped 66 | 8552038318e5b09d3484de1087b413a2aa548f24 /tmp/compare/bitcoind.25f0edd59fee454495da39c6c63f19b97cd75e96.stripped 67 | 5c648d92e6d696ca090c28f3db98cd59cdcc6a44 /tmp/16370/bitcoin-qt.12061016eca4cf4c77fa042ea95682e5dc2b302b.stripped 68 | b72c2949587965cc18c32fcdf5e5ae187f4ef819 /tmp/16370/bitcoind.12061016eca4cf4c77fa042ea95682e5dc2b302b.stripped 69 | 70 | # diffoscope bitcoind binaries 71 | diffoscope compare/bitcoind.25f0edd59fee454495da39c6c63f19b97cd75e96 16370/bitcoind.12061016eca4cf4c77fa042ea95682e5dc2b302b 72 | ``` 73 | -------------------------------------------------------------------------------- /gnuplot/README.md: -------------------------------------------------------------------------------- 1 | # gnuplot 2 | 3 | [gnuplot](http://www.gnuplot.info) is a graphing utility that can generate two and three-dimensional plots. 4 | 5 | Useful for reviewing benchmarking / speedup related PRs, like [#15751](https://github.com/bitcoin/bitcoin/pull/15751). 6 | 7 | Using the data in [`col.csv`](col.csv), which was collected using `bitcoind` RPC calls like: 8 | ```bash 9 | time src/bitcoin-cli deriveaddresses "wsh(multi(1,xpub661MyMwAqRbcFW31YEwpkMuc5THy2PSt5bDMsktWQcFF8syAmRUapSCGu8ED9W6oDMSgv6Zz8idoc4a6mr8BDzTJY47LJhkJ8UB7WEGuduB/1/0/*,xpub69H7F5d8KSRgmmdJg2KhpAK8SR3DjMwAdkxj3ZuxV27CprR9LgpeyGmXUbC6wb7ERfvrnKZjXoUmmDznezpbZb7ap6r1D3tgFxHmwMkQTPH/0/0/*))#t2zpj2eu" "[0,1000]" 10 | ``` 11 | 12 | ```bash 13 | # addresses | #15751 (s) | master (s) 14 | 100, 0.059, 0.058 15 | 500, 0.188, 0.314 16 | 1000, 0.355, 0.899 17 | 2000, 0.70, 2.953 18 | < snip > 19 | ``` 20 | 21 | Open `gnuplot` and configure and generate a .`png`: 22 | ```bash 23 | gnuplot 24 | 25 | set term png 26 | set output 'deriving.png' 27 | set title 'Deriving Addresses' 28 | set xlabel 'Addresses' 29 | set ylabel 'Time (s)' 30 | plot 'col.csv' using 1:2 title '#15751 - 41a46cb' with lines, \ 31 | 'col.csv' using 1:3 title 'master - 66ce95a' with lines 32 | ``` 33 | 34 | ![Deriving Addresses](deriving.png) 35 | 36 | ### depends build timing 37 | 38 | ```bash 39 | set title noenhanced 40 | set title "Bitcoin Core Depends Build Time (macOS)\nOnly required packages. Excludes downloading. make -C depends -j8 NO_QT=1 NO_UPNP=1 etc" 41 | set xlabel 'Release' 42 | 43 | set xrange [0:9] 44 | set xtics ('0.13.2' 1, '0.14.3' 2, '0.15.2' 3, '0.16.3' 4, '0.17.2' 5, '0.18.1' 6, '0.19.0.1' 7, '0.19.99' 8) 45 | 46 | set ylabel 'Time (s)' 47 | set yrange [100:245] 48 | 49 | set y2range [0:10] 50 | set y2tics 1 51 | set y2label 'Packages Built' 52 | 53 | plot 'depends.csv' using 1:2:($2) with labels offset char -2,1 notitle, \ 54 | 'depends.csv' using 1:2 title 'build time' with lines, \ 55 | 'depends.csv' using 1:3:($3) with labels offset char 1,-1 notitle axes x1y2, \ 56 | 'depends.csv' using 1:3 title 'packages built' with line axes x1y2 57 | ``` 58 | 59 | ![Depends Time](depends_time.png) 60 | 61 | ### bitcoind / qt file size 62 | 63 | ```bash 64 | set title 'Bitcoin Core Binary Size (macOS)' 65 | 66 | set xlabel 'Release' 67 | set xrange [0:12] 68 | set xtics ('0.10.3' 1, '0.11.4' 2, '0.12.1' 3, '0.13.2' 4, '0.14.3' 5, '0.15.2' 6, '0.16.3' 7, '0.17.2' 8, '0.18.1' 9, '0.19.0.1' 10, '0.19.99' 11) 69 | 70 | set ylabel 'Size (mb)' 71 | set format y '%.2s' 72 | 73 | plot 'size.csv' using 1:2:(sprintf("%.2f mb", $2 / 1000000)) with labels offset char 1,1 notitle, \ 74 | 'size.csv' using 1:2 title 'bitcoind' with lines, \ 75 | 'size.csv' using 1:3:(sprintf("%.2f mb", $3 / 1000000)) with labels offset char 1,1 notitle, \ 76 | 'size.csv' using 1:3 title 'bitcoin-qt' with line 77 | 78 | set title 'Bitcoin Core dylib Count (macOS)' 79 | set yrange [0:20] 80 | 81 | plot 'dylib.csv' using 1:2:('2') with labels offset char 1,1 notitle, \ 82 | 'dylib.csv' using 1:2 title 'bitcoind' with lines, \ 83 | 'dylib.csv' using 1:3:($3) with labels offset char 1,1 notitle, \ 84 | 'dylib.csv' using 1:3 title 'bitcoin-qt' with line 85 | ``` 86 | 87 | ![Depends File Size](file_size.png) -------------------------------------------------------------------------------- /update-assumevalid.md: -------------------------------------------------------------------------------- 1 | # Reviewing assumevalid updates 2 | 3 | `assumevalid` was introduced in [#9484](https://github.com/bitcoin/bitcoin/pull/9484) (Bitcoin Core [0.14.0](https://bitcoincore.org/en/releases/0.14.0/)). 4 | 5 | A good summary of IBD, the history of checkpointing and the rational behind `assumevalid` is available [here](https://bitcoincore.org/en/2017/03/08/release-0.14.0/#ibd). 6 | 7 | ## Validating Updates 8 | Validating updates is simply running RPC commands, and verifying that the output from your node matches the values being proposed in the pull request. 9 | 10 | For example, if you were reviewing [#15429](https://github.com/bitcoin/bitcoin/pull/15429): 11 | 12 | Run `bitcoin-cli getchaintxstats` with a block window of `4096`, and the blockhash of the new assumevalid block. 13 | 14 | From the output, check that the following values match those in the PR: 15 | 16 | - `time` -> nTime 17 | - `txcount` -> nTxCount 18 | - `txrate` -> dTxRate 19 | 20 | ```bash 21 | bitcoin-cli getchaintxstats 4096 0000000000000000000f1c54590ee18d15ec70e68c8cd4cfbadb1b4f11697eee 22 | 23 | { 24 | "time": 1550374134, 👍 25 | "txcount": 383732546, 👍 26 | "window_final_block_hash": "0000000000000000000f1c54590ee18d15ec70e68c8cd4cfbadb1b4f11697eee", 27 | "window_block_count": 4096, 28 | "window_tx_count": 8872107, 29 | "window_interval": 2407303, 30 | "txrate": 3.685496590998308 👍 31 | } 32 | ``` 33 | 34 | Run `bitcoin-cli getblockheader` passing the blockhash of the new assumevalid block. 35 | 36 | From the output, check that the following values match those in the PR: 37 | 38 | - `chainwork` -> consensus.nMinimumChainWork 39 | - `hash` -> consensus.defaultAssumeValid 40 | - `height` -> consensus.defaultAssumeValid comment 41 | 42 | ```bash 43 | bitcoin-cli getblockheader 0000000000000000000f1c54590ee18d15ec70e68c8cd4cfbadb1b4f11697eee 44 | 45 | { 46 | "hash": "0000000000000000000f1c54590ee18d15ec70e68c8cd4cfbadb1b4f11697eee", 👍 47 | "confirmations": 3892, 48 | "strippedsize": 115156, 49 | "size": 131899, 50 | "weight": 477367, 51 | "height": 563378, 👍 52 | "version": 536870912, 53 | "versionHex": "20000000", 54 | "merkleroot": "3c9b9295588dbab8388785dd9f70c9d7f42948632556298c8453aad755e30822", 55 | # not displaying tx data 56 | "time": 1550374134, 57 | "mediantime": 1550373349, 58 | "nonce": 3355453177, 59 | "bits": "172e6f88", 60 | "difficulty": 6061518831027.271, 61 | "chainwork": "0000000000000000000000000000000000000000051dc8b82f450202ecb3d471", 👍 62 | "nTx": 359, 63 | "previousblockhash": "00000000000000000001827a07f51769246d852e63c413bc2cd5caa104970f3f", 64 | "nextblockhash": "0000000000000000001d6cf3d8d1bf2e0b1883f6684bb2dddcb57a5d4c81ec7f" 65 | 66 | ``` 67 | 68 | ### Previous assumevalid updates (block height) 69 | - [886157](https://github.com/bitcoin/bitcoin/pull/31978) 70 | - [856760](https://github.com/bitcoin/bitcoin/pull/30658) 71 | - [824000](https://github.com/bitcoin/bitcoin/pull/29547) 72 | - [804000](https://github.com/bitcoin/bitcoin/pull/28591) 73 | - [784000](https://github.com/bitcoin/bitcoin/pull/27482) 74 | - [751565](https://github.com/bitcoin/bitcoin/pull/25946) 75 | - [724466](https://github.com/bitcoin/bitcoin/pull/24418) 76 | - [691719](https://github.com/bitcoin/bitcoin/pull/22499) 77 | - [654683](https://github.com/bitcoin/bitcoin/pull/20263) 78 | - [623950](https://github.com/bitcoin/bitcoin/pull/18500) 79 | - [597379](https://github.com/bitcoin/bitcoin/pull/17002) 80 | - [563378](https://github.com/bitcoin/bitcoin/pull/15429) 81 | - [534292](https://github.com/bitcoin/bitcoin/pull/13794) 82 | - [506067](https://github.com/bitcoin/bitcoin/pull/12269) 83 | - [477890](https://github.com/bitcoin/bitcoin/pull/10945) 84 | - [447235](https://github.com/bitcoin/bitcoin/pull/9484) (Introduction) 85 | -------------------------------------------------------------------------------- /hardening.md: -------------------------------------------------------------------------------- 1 | # Hardening 2 | 3 | Bitcoin Core makes use of many hardening features. Some of them are documented here. 4 | 5 | ## Linux 6 | 7 | ### RELRO `-z,relro` 8 | 9 | There are sections in a binary that may need to be writable by the loader, but don't 10 | need to be writable by the application during runtime. Passing `-z,relro` to the 11 | linker instructs it to mark any applicable sections as such, and the loader will 12 | mark them read-only at runtime, before giving control back to the application. 13 | 14 | ```bash 15 | readelf -l -W src/bitcoind 16 | ... 17 | Program Headers: 18 | Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align 19 | 20 | GNU_RELRO 0x55adf0 0x000000000055bdf0 0x000000000055bdf0 0x00d210 0x00d210 R 0x1 21 | ``` 22 | 23 | ### Bind Now `-z,now` 24 | 25 | It's common for binaries to use lazy symbol binding to improve startup times. 26 | This means that references to functions in dynamic libraries are resolved when 27 | they are first used, instead of at startup. Passing `-z,now` to the linker will 28 | make it mark the binary such that the dynamic loader will resolve all function 29 | references at startup. See also `RTLD_NOW`in the [`dlopen` man page](https://linux.die.net/man/3/dlopen). 30 | 31 | ```bash 32 | readelf -d src/bitcoind 33 | ... 34 | 35 | Dynamic section at offset 0x5670e0 contains 31 entries: 36 | Tag Type Name/Value 37 | 38 | 0x000000000000001e (FLAGS) BIND_NOW 39 | 0x000000006ffffffb (FLAGS_1) Flags: NOW PIE 40 | ``` 41 | 42 | When combined with RELRO, this allows the loader to mark more of the binary 43 | as read-only. 44 | 45 | ### Non-executable Stack 46 | 47 | A non-executable stack is now standard for most modern toolchains. If you want 48 | an executable stack you will likely have to op-in by passing `-z,execstack` to 49 | the linker. Bitcoin Core is built with a non-executable stack, you can check this 50 | by looking for the `RW` flags in the `GNU_STACK` program header. 51 | 52 | ```bash 53 | readelf -l -W src/bitcoind 54 | ... 55 | Program Headers: 56 | Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align 57 | 58 | GNU_STACK 0x000000 0x0000000000000000 0x0000000000000000 0x000000 0x000000 RW 0x10 59 | ``` 60 | 61 | If you modify configure.ac to pass `Wl,-z,execstack` to the linker, you'll find: 62 | 63 | ```bash 64 | Program Headers: 65 | Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align 66 | 67 | GNU_STACK 0x000000 0x0000000000000000 0x0000000000000000 0x000000 0x000000 RWE 0x10 68 | ``` 69 | 70 | Further Reading: 71 | * [NX bit](https://en.wikipedia.org/wiki/NX_bit). 72 | * The Ubuntu Security Team is working to [eliminate executable stacks](https://wiki.ubuntu.com/SecurityTeam/Roadmap/ExecutableStacks). 73 | * Gentoo Linux notes on [GNU Stack management](https://wiki.gentoo.org/wiki/Hardened/GNU_stack_quickstart). 74 | 75 | ## Windows 76 | 77 | #### Terminate on Heap Corruption 78 | 79 | If the Windows heap manager detects an error in any heap used by the process it 80 | will terminate it. 81 | 82 | Enabled in [#17916](https://github.com/bitcoin/bitcoin/pull/17916). 83 | 84 | #### [Data Execution Prevention](https://docs.microsoft.com/en-us/windows/win32/memory/data-execution-prevention) 85 | 86 | Data Execution Prevention (DEP) is enabled by default for all 64-bit Windows 87 | processes. We used to set this explicity via a call to [SetProcessDEPPolicy()](https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-setprocessdeppolicy), 88 | however we no-longer support 32-bit on Windows, and the call will always fail 89 | when made from a 64-bit process. 90 | 91 | ## macOS 92 | 93 | ### bind_at_load 94 | 95 | This is the macOS/ld64 equivalent of `-z,now`. From man `ld`: 96 | 97 | > Sets a bit in the mach header of the resulting binary which tells `dyld` 98 | to bind all symbols when the binary is loaded, rather than lazily. 99 | 100 | 101 | #### TODO 102 | 103 | PIE, PIC, Stack Smashing Protector 104 | -------------------------------------------------------------------------------- /guix/README.md: -------------------------------------------------------------------------------- 1 | # Guix 2 | 3 | [Guix](https://www.gnu.org/software/guix/) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin. 4 | 5 | If you're interested in downloading a pre-built VM and testing Guix (using QEMU), I have a short guide [here](vm-intro.md). 6 | 7 | ## Alpine Guix Container Image 8 | 9 | ### Create the alpine_guix image: 10 | 11 | ```bash 12 | podman build --pull --no-cache -t alpine_guix - < imagefile 13 | ``` 14 | 15 | By default, an image *setup to use substitutes* will be created. 16 | 17 | If you do not want to use substitutes, you can pass `--target no-substitutes` to the build command 18 | ```bash 19 | podman build --pull --no-cache --target no-substitutes -t no_subs - < imagefile 20 | ``` 21 | 22 | You can also override where the image fetches the guix binary from with `--build-args`. 23 | 24 | ```bash 25 | pushd bitcoin 26 | podman build -f imagefile \ 27 | --build-arg guix_download_path=https://ftpmirror.gnu.org/gnu/guix/ \ 28 | --build-arg guix_file_name=guix-binary-1.4.0.x86_64-linux.tar.xz \ 29 | --build-arg guix_checksum=236ca7c9c5958b1f396c2924fcc5bc9d6fdebcb1b4cf3c7c6d46d4bf660ed9c9 \ 30 | -t alpine_guix . 31 | ``` 32 | 33 | ### Run the alpine_guix container 34 | 35 | To `exec` a `guix-daemon` (a prerequisite for guix builds): 36 | 37 | ```bash 38 | podman run -it --name alpine_guix --privileged alpine_guix 39 | ``` 40 | 41 | The daemon will run in the foreground, so don't be alarmed if it hangs (you may see output like `accepted connection from pid 2828, user root`). 42 | 43 | Note the use of `--privileged`. Read the Podman [capabilities documentation](https://docs.podman.io/en/latest/markdown/podman-run.1.html#privileged) before running any image with this flag. 44 | 45 | ### Do a Bitcoin Core build: 46 | 47 | Exec into the container: 48 | 49 | ```bash 50 | podman exec -it alpine_guix /bin/bash 51 | ``` 52 | 53 | Default HOSTS: 54 | ```bash 55 | x86_64-linux-gnu 56 | arm-linux-gnueabihf 57 | aarch64-linux-gnu 58 | riscv64-linux-gnu 59 | powerpc64-linux-gnu 60 | powerpc64le-linux-gnu 61 | x86_64-w64-mingw32 62 | x86_64-apple-darwin 63 | arm64-apple-darwin 64 | ``` 65 | 66 | Build: 67 | 68 | ```bash 69 | # Set HOSTS to build for a specific target. i.e 70 | time BASE_CACHE="/base_cache" SOURCE_PATH="/sources" SDK_PATH="/SDKs" HOSTS="x86_64-w64-mingw32" ./contrib/guix/guix-build 71 | ``` 72 | 73 | ## Debian Guix Container Image 74 | 75 | A Debian based container image is also available, which uses the Debian [Guix package](https://packages.debian.org/trixie/guix). 76 | 77 | It can be created using: 78 | ```bash 79 | podman build --pull --no-cache -t debian_guix - < debian.imagefile 80 | ``` 81 | 82 | and used the same way as the Alpine container (see above). 83 | 84 | ## Build output 85 | 86 | Providing the following information is useful after a successful build: 87 | ```bash 88 | guix describe 89 | uname -m 90 | find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum 91 | ``` 92 | 93 | ## Bitcoin Core Guix Package 94 | 95 | Bitcoin Core is also available via the Guix package manager. 96 | 97 | Submitting an update requires [`guix hash`](https://guix.gnu.org/manual/en/html_node/Invoking-guix-hash.html#Invoking-guix-hash) and fixing any build issues. 98 | 99 | Recent Updates: 100 | 101 | * [25.0]https://git.savannah.gnu.org/cgit/guix.git/commit/?id=b58fab5982eb45aae2a51c86093e6bbd472d8c9e 102 | * [24.1](https://git.savannah.gnu.org/cgit/guix.git/commit/?id=0aab24855238cc7c7a31066ab39cd94e534b857f) 103 | * [23.1](https://git.savannah.gnu.org/cgit/guix.git/commit/?id=3c8d0f9e71bbddfb5b1f098c713ff37553f0efcc) 104 | * [23.0](https://git.savannah.gnu.org/cgit/guix.git/commit/?id=79e40b6ce8e4f5f499ea338aede75a0810a210c1) 105 | * [0.21.1](https://git.savannah.gnu.org/cgit/guix.git/commit/?id=2fc9d513811e4a737bd7337545732337641d2738) 106 | * [0.21.0](https://git.savannah.gnu.org/cgit/guix.git/commit/?id=d3c8aa3f8214434c8ba819984ed4513796a09e38) 107 | -------------------------------------------------------------------------------- /pgo.md: -------------------------------------------------------------------------------- 1 | # Profile Guided Optimization 2 | 3 | An overview of PGO is available in the [Clang Users Manual](https://clang.llvm.org/docs/UsersManual.html#profile-guided-optimization). 4 | 5 | ### Capture Profiling Data 6 | 7 | ```bash 8 | # build minimal depends 9 | make -C depends NO_WALLET=1 NO_QT=1 NO_ZMQ=1 NO_USDT=1 \ 10 | CC=clang CXX=clang++ \ 11 | AR=llvm-ar \ 12 | NM=llvm-nm \ 13 | RANLIB=llvm-ranlib \ 14 | STRIP=llvm-strip 15 | 16 | # instrumentation using -fprofile-instr-generate 17 | # -%p is replaced with process ID during profiling 18 | cmake -B build --toolchain=depends/aarch64-unknown-linux-gnu/toolchain.cmake \ 19 | -DCMAKE_C_FLAGS="-fprofile-instr-generate=bitcoind-%p.profraw -flto=full" \ 20 | -DCMAKE_CXX_FLAGS="-fprofile-instr-generate=bitcoind-%p.profraw -flto=full" \ 21 | -DREDUCE_EXPORTS=ON \ 22 | -DAPPEND_LDFLAGS="-flto=full -static-pie" 23 | 24 | cmake --build build --target bitcoind 25 | 26 | # TODO: Runtime options? 27 | ./build/bin/bitcoind 28 | ``` 29 | 30 | ### Inspect profiling data 31 | 32 | ```bash 33 | # After multiple runs you'll have raw profiling data. 34 | llvm-profdata show bitcoind-86267.profraw --detailed-summary 35 | 36 | Instrumentation level: Front-end 37 | Total functions: 67454 38 | Maximum function count: 442997952222 39 | Maximum internal block count: 413456186977 40 | Detailed summary: 41 | Total number of blocks: 92925 42 | Total count: 24437392719763 43 | 421 blocks with count >= 9366520400 account for 80 percentage of the total counts. 44 | 874 blocks with count >= 3176583698 account for 90 percentage of the total counts. 45 | 1412 blocks with count >= 1515876321 account for 95 percentage of the total counts. 46 | 2842 blocks with count >= 300666001 account for 99 percentage of the total counts. 47 | 4194 blocks with count >= 31717404 account for 99.9 percentage of the total counts. 48 | 6951 blocks with count >= 1384881 account for 99.99 percentage of the total counts. 49 | 10372 blocks with count >= 256454 account for 99.999 percentage of the total counts. 50 | 51 | # Filter functions with --value-cutoff=x 52 | Number of functions with maximum count (< 1000000000): 66014 53 | Number of functions with maximum count (>= 1000000000): 1440 54 | 55 | # Dump the functions themselves with --all-functions 56 | Functions shown: 681 57 | Total functions: 67454 58 | ... 59 | sha256_avx2.cpp:_ZN14sha256d64_avx212_GLOBAL__N_11KEj: 60 | Hash: 0x0000000000000018 61 | Counters: 1 62 | Function count: 8369362820 63 | sha256_avx2.cpp:_ZN14sha256d64_avx212_GLOBAL__N_15RoundEDv4_xS1_S1_RS1_S1_S1_S1_S2_S1_: 64 | Hash: 0x0000000000000000 65 | Counters: 1 66 | Function count: 6986598528 67 | sha256_avx2.cpp:_ZN14sha256d64_avx212_GLOBAL__N_13AddEDv4_xS1_: 68 | Hash: 0x0000000000000018 69 | Counters: 1 70 | Function count: 56948055710 71 | sha256_avx2.cpp:_ZN14sha256d64_avx212_GLOBAL__N_13AddEDv4_xS1_S1_S1_: 72 | Hash: 0x0000000000000018 73 | Counters: 1 74 | Function count: 9934069782 75 | ``` 76 | 77 | ### Merge profiling data 78 | ```bash 79 | # Combine the raw data into a single file using llvm-profdata 80 | # This is required even with a single raw file as llvm-profdata alters the data format 81 | llvm-profdata merge -output=bitcoind.profdata bitcoind-*.profraw 82 | 83 | # To check that all your raw files are found pass --dump-input-file-list 84 | ... --dump-input-file-list 85 | 1,bitcoind-39335.profraw 86 | 1,bitcoind-47729.profraw 87 | 1,bitcoind-47916.profraw 88 | ``` 89 | 90 | ### Compile using profiling data 91 | 92 | ```bash 93 | rm -rf build 94 | 95 | # reconfigure passing -fprofile-instr-use 96 | # not passing the full path to bitcoind.profdata may cause issues 97 | cmake -B build --toolchain=depends/aarch64-unknown-linux-gnu/toolchain.cmake \ 98 | -DCMAKE_C_FLAGS="-fprofile-instr-use=/root/bitcoin/bitcoind.profdata -flto=full" \ 99 | -DCMAKE_CXX_FLAGS="-fprofile-instr-use=/root/bitcoin/bitcoind.profdata -flto=full" \ 100 | -DREDUCE_EXPORTS=ON \ 101 | -DAPPEND_LDFLAGS="-flto=full -static-pie" 102 | 103 | # recompile using the profiling data you may see warnings like: 104 | # warning: no profile data available for file "node.cpp" [-Wprofile-instr-unprofiled] 105 | cmake --build build --target bitcoind 106 | ``` 107 | 108 | ### TODO 109 | 110 | - PGO + more compile time optimisations 111 | - Capture more extensive profiling data 112 | -------------------------------------------------------------------------------- /subtree-merge.md: -------------------------------------------------------------------------------- 1 | # Upstream Subtree Merge 2 | 3 | ## How to Verify 4 | 5 | If you wanted to verify a merge of the upstream [secp256k1](https://github.com/bitcoin-core/secp256k1) repo into bitcoin/bitcoin. 6 | i.e PR [#22448](https://github.com/bitcoin/bitcoin/pull/22448). 7 | 8 | ```bash 9 | pushd bitcoin 10 | 11 | git fetch upstream pull/22448/head:22448 12 | 13 | git checkout 22448 14 | 15 | git fetch https://github.com/bitcoin-core/secp256k1 16 | remote: Enumerating objects: 6714, done. 17 | remote: Counting objects: 100% (491/491), done. 18 | remote: Compressing objects: 100% (222/222), done. 19 | remote: Total 6714 (delta 310), reused 371 (delta 268), pack-reused 6223 20 | Receiving objects: 100% (6714/6714), 2.92 MiB | 165.00 KiB/s, done. 21 | Resolving deltas: 100% (4678/4678), done. 22 | From https://github.com/bitcoin-core/secp256k1 23 | * branch HEAD -> FETCH_HEAD 24 | 25 | ./test/lint/git-subtree-check.sh src/secp256k1 26 | src/secp256k1 in HEAD currently refers to tree b7f4357396a8a5f073bf628567ef10b2e2edd410 27 | src/secp256k1 in HEAD was last updated in commit c020cbaa5c8e9e61b2b8efd8dc09be743fcd4273 (tree b7f4357396a8a5f073bf628567ef10b2e2edd410) 28 | GOOD 29 | ``` 30 | 31 | ## Historical Subtree Updates 32 | 33 | ### [crc32c](https://github.com/bitcoin-core/crc32c-subtree) 34 | 35 | - https://github.com/bitcoin/bitcoin/pull/29493 36 | - https://github.com/bitcoin/bitcoin/pull/25836 37 | - https://github.com/bitcoin/bitcoin/pull/23133 38 | - https://github.com/bitcoin/bitcoin/pull/20603 39 | - https://github.com/bitcoin/bitcoin/pull/17398 40 | 41 | ### [ctaes](https://github.com/bitcoin-core/ctaes) 42 | 43 | - https://github.com/bitcoin/bitcoin/pull/9303 44 | 45 | ### [leveldb](https://github.com/bitcoin-core/leveldb) 46 | 47 | - https://github.com/bitcoin/bitcoin/pull/32534 48 | - https://github.com/bitcoin/bitcoin/pull/31671 49 | - https://github.com/bitcoin/bitcoin/pull/30281 50 | - https://github.com/bitcoin/bitcoin/pull/24461 51 | - https://github.com/bitcoin/bitcoin/pull/17398 52 | - https://github.com/bitcoin/bitcoin/pull/15270 53 | - https://github.com/bitcoin/bitcoin/pull/13925 54 | - https://github.com/bitcoin/bitcoin/pull/12451 55 | - https://github.com/bitcoin/bitcoin/pull/2702 56 | - https://github.com/bitcoin/bitcoin/pull/2916 57 | 58 | ### [libmultiprocess](https://github.com/bitcoin-core/libmultiprocess) 59 | 60 | - https://github.com/bitcoin/bitcoin/pull/33241 61 | - https://github.com/bitcoin/bitcoin/pull/32641 62 | - https://github.com/bitcoin/bitcoin/pull/31741 63 | 64 | ### [minisketch](https://github.com/bitcoin-core/minisketch/) 65 | 66 | - https://github.com/bitcoin/bitcoin/pull/32485 67 | - https://github.com/bitcoin/bitcoin/pull/32000 68 | - https://github.com/bitcoin/bitcoin/pull/30270 69 | - https://github.com/bitcoin/bitcoin/pull/29823 70 | - https://github.com/bitcoin/bitcoin/pull/25502 71 | - https://github.com/bitcoin/bitcoin/pull/23114 72 | 73 | ### python-bitcoinrpc (removed) 74 | 75 | - https://github.com/bitcoin/bitcoin/pull/6097 76 | 77 | ### [secp256k1](https://github.com/bitcoin-core/secp256k1) 78 | 79 | - https://github.com/bitcoin/bitcoin/pull/33036 80 | - https://github.com/bitcoin/bitcoin/pull/32028 81 | - https://github.com/bitcoin/bitcoin/pull/31216 82 | - https://github.com/bitcoin/bitcoin/pull/30845 83 | - https://github.com/bitcoin/bitcoin/pull/30573 84 | - https://github.com/bitcoin/bitcoin/pull/30334 85 | - https://github.com/bitcoin/bitcoin/pull/30120 86 | - https://github.com/bitcoin/bitcoin/pull/29803 87 | - https://github.com/bitcoin/bitcoin/pull/29169 88 | - https://github.com/bitcoin/bitcoin/pull/28404 89 | - https://github.com/bitcoin/bitcoin/pull/28093 90 | - https://github.com/bitcoin/bitcoin/pull/27479 91 | - https://github.com/bitcoin/bitcoin/pull/27445 92 | - https://github.com/bitcoin/bitcoin/pull/27230 93 | - https://github.com/bitcoin/bitcoin/pull/24792 94 | - https://github.com/bitcoin/bitcoin/pull/23383 95 | - https://github.com/bitcoin/bitcoin/pull/22448 96 | - https://github.com/bitcoin/bitcoin/pull/20257 97 | - https://github.com/bitcoin/bitcoin/pull/20147 98 | - https://github.com/bitcoin/bitcoin/pull/19944 99 | - https://github.com/bitcoin/bitcoin/pull/19228 100 | - https://github.com/bitcoin/bitcoin/pull/15703 101 | - https://github.com/bitcoin/bitcoin/pull/11421 102 | - https://github.com/bitcoin/bitcoin/pull/8453 103 | 104 | ### [univalue](https://github.com/bitcoin-core/univalue) (removed) 105 | 106 | - https://github.com/bitcoin/bitcoin/pull/25369 (removal) 107 | - https://github.com/bitcoin/bitcoin/pull/25249 108 | - https://github.com/bitcoin/bitcoin/pull/25113 109 | - https://github.com/bitcoin/bitcoin/pull/22646 110 | - https://github.com/bitcoin/bitcoin/pull/20424 111 | - https://github.com/bitcoin/bitcoin/pull/18099 112 | - https://github.com/bitcoin/bitcoin/pull/17324 113 | - https://github.com/bitcoin/bitcoin/pull/14164 114 | - https://github.com/bitcoin/bitcoin/pull/11952 115 | - https://github.com/bitcoin/bitcoin/pull/11420 116 | - https://github.com/bitcoin/bitcoin/pull/8807 117 | - https://github.com/bitcoin/bitcoin/pull/8863 118 | - https://github.com/bitcoin/bitcoin/pull/6637 119 | -------------------------------------------------------------------------------- /determinism.md: -------------------------------------------------------------------------------- 1 | ### Determinism in Bitcoin Core 2 | Being reproducibly built, Bitcoin Core semi-regularly turns up/suffers from 3 | determinism issues in it's own code base, as well as in it's dependencies. 4 | 5 | This is a non-exhaustive list of some of those issues. 6 | 7 | #### ld64 threading 8 | ld64 is threaded, and uses a worker for each CPU to parse input files. However a 9 | bug in the parser caused dependencies to be calculated differently based on 10 | which files had already been parsed. As a result, builders with more CPUs were 11 | more likely to see non-determinism. 12 | 13 | Fixed in [#9891](https://github.com/bitcoin/bitcoin/pull/9891) 14 | 15 | Examples: 16 | - [achow 0.14.0rc1](https://github.com/bitcoin-core/gitian.sigs/pull/471) 17 | - [fanquake 0.14.0rc1](https://github.com/bitcoin-core/gitian.sigs/pull/470) 18 | 19 | TODO: Determine in exactly which version of [ld64](https://opensource.apple.com/source/ld64/) this was fixed. 20 | 21 | #### OpenSSL Embedded Timestamp 22 | In version [1.0.1j](https://www.openssl.org/source/old/1.0.1/openssl-1.0.1j.tar.gz) OpenSSL would embed a timestamp using 23 | 24 | ```echo " #define DATE \"`LC_ALL=C LC_TIME=C date`\""; \``` 25 | 26 | This behaviour was in `crypto/Makefile`. 27 | From [1.0.1k](https://www.openssl.org/source/old/1.0.1/openssl-1.0.1k.tar.gz), the behaviour moved into `util/mkbuildinf.pl`. 28 | 29 | Fixed after the 1.0.1k bump in [c3200bcd1e7116e079aebabed3a01dc5385bfc9e](https://github.com/bitcoin/bitcoin/commit/c3200bcd1e7116e079aebabed3a01dc5385bfc9e). 30 | 31 | 32 | #### Win32 Protoc symbol ordering 33 | A symbol ordering issue, `libstdc++.so.6` and `__gmon_start__` appeared to be 34 | swapped, in `protoc` on win32 systems caused non-determinism. 35 | Issue discussion in [#3725](https://github.com/bitcoin/bitcoin/issues/3725). 36 | Fixed in [#3742](https://github.com/bitcoin/bitcoin/pull/3742). 37 | 38 | #### NSIS makeifle inclusion 39 | Including the `Makefile` in the NSIS genearated documentation was causing 40 | determinism issues depending on wether certain programs (`gawk` & `mawk`) were 41 | available on the build VM. 42 | 43 | Fixed in [#14018](https://github.com/bitcoin/bitcoin/pull/14018) 44 | 45 | Examples: 46 | - [fanquake 0.17.0rc1 win assert](https://github.com/bitcoin-core/gitian.sigs/blob/master/0.17.0rc1-win-unsigned/fanquake/bitcoin-win-0.17-build.assert) 47 | - [luke-jr 0.17.0rc1 win assert](https://github.com/bitcoin-core/gitian.sigs/blob/master/0.17.0rc1-win-unsigned/luke-jr/bitcoin-win-0.17-build.assert) 48 | 49 | #### Qt xkb handling 50 | Qt's configure grabs the path to the xkb data root during configure. Build 51 | changes in Qt 5.8 broke the handling for cross-builds. This caused the string 52 | embedded in the binary to differ depending on wether certain files were present 53 | in the builders filesystem. 54 | 55 | Fixed in [#14000](https://github.com/bitcoin/bitcoin/pull/14000) 56 | 57 | #### Qt resource compiler embedding timestamps 58 | After the bump to Qt 5.9, the [`rcc`](https://doc.qt.io/qt-5/rcc.html) tool was embedding timestamps 59 | into build output. This resulted in `qrc_bitcoin_locale.cpp` always being 60 | generated in a non-deterministic way. 61 | 62 | Upstream issue [QTBUG-62511](https://bugreports.qt.io/browse/QTBUG-62511) 63 | Fixed in [#13732](https://github.com/bitcoin/bitcoin/pull/13732) 64 | 65 | ### Difference in `genisoimage` binary 66 | 67 | During the gitian builds for the 0.20.0 release luke-jr noticed that his 68 | `bitcoin-osx-unsigned.tar.gz` was not matching anyone elses. The cause was 69 | determined to be a small difference in the `genisoimage` binary. IRC discussion 70 | available [here](http://www.erisian.com.au/bitcoin-core-dev/log-2020-06-03.html#l-48). 71 | 72 | ```diff 73 | ### ./genisoimage 74 | ```diff 75 | @@ -49796,16 +49796,16 @@ 76 | 000c2830: 6400 7573 616c 6f5f 6169 7361 7461 7069 d.usalo_aisatapi 77 | 000c2840: 0075 7361 6c6f 5f61 7265 7365 7400 7363 .usalo_areset.sc 78 | 000c2850: 616e 5f69 6e74 6572 6e61 6c00 7367 5f61 an_internal.sg_a 79 | 000c2860: 6d61 7064 6576 0075 7361 6c6f 5f61 6765 mapdev.usalo_age 80 | 000c2870: 745f 6669 7273 745f 6672 6565 5f73 6869 t_first_free_shi 81 | 000c2880: 6c6c 7962 7573 0075 7361 6c6f 5f61 6d65 llybus.usalo_ame 82 | 000c2890: 7267 6500 7367 5f61 6d61 7064 6576 5f73 rge.sg_amapdev_s 83 | -000c28a0: 6373 6900 6b76 2e36 3034 3100 7761 726e csi.kv.6041.warn 84 | -000c28b0: 5f73 7973 6673 2e36 3036 3300 7367 5f63 _sysfs.6063.sg_c 85 | +000c28a0: 6373 6900 6b76 2e36 3034 3400 7761 726e csi.kv.6044.warn 86 | +000c28b0: 5f73 7973 6673 2e36 3036 3600 7367 5f63 _sysfs.6066.sg_c 87 | 000c28c0: 6c65 6172 6e62 6c6f 636b 0073 675f 6d61 learnblock.sg_ma 88 | 000c28d0: 7064 6576 0073 675f 7365 7475 7000 7367 pdev.sg_setup.sg 89 | 000c28e0: 5f6d 6170 6275 7300 7367 5f69 6e69 7464 _mapbus.sg_initd 90 | 000c28f0: 6576 0073 675f 7365 7474 696d 656f 7574 ev.sg_settimeout 91 | 000c2900: 0073 675f 7261 6973 6564 6d61 0066 7265 .sg_raisedma.fre 92 | 000c2910: 6164 7374 7269 6e67 0073 675f 6765 7469 adstring.sg_geti 93 | 000c2920: 6e74 0073 675f 7277 7365 6e64 0075 7361 nt.sg_rwsend.usa 94 | ``` 95 | 96 | #### TODO: 97 | 98 | mingw linker leaking bytes, [causing non-determinism](https://github.com/bitcoin/bitcoin/commit/957c0fd7c0efe2c39dde025a7d6d3d3047c86a1a#diff-92cf1b5b31c29d8281ffdc628b6da14f). 99 | https://github.com/bitcoin/bitcoin/pull/6900 100 | 101 | cdrkit [determinism patch](https://github.com/bitcoin/bitcoin/blob/master/depends/patches/native_cdrkit/cdrkit-deterministic.patch) 102 | 103 | dont add `.` to tar list - https://github.com/bitcoin/bitcoin/pull/5790 -------------------------------------------------------------------------------- /rtld/README.md: -------------------------------------------------------------------------------- 1 | # rtld-audit - Dynamic Linker Auditing 2 | 3 | The GNU dynamic linker provides an [auditing api](https://linux.die.net/man/7/rtld-audit). We can use it to observe dynamic linking events for `bitcoind` and `bitcoin-qt`. Note that what's being done here can be more easily achieved using various `ld` environment variables at runtime. i.e `LD_DEBUG=all`. 4 | 5 | Start a [debian-depends](../docker/debian.dockerfile) Docker container. 6 | 7 | ```bash 8 | # copy the audit source into the container 9 | docker cp audit.cpp container_id:. 10 | ``` 11 | 12 | Inside the container: 13 | 14 | ```bash 15 | # build the audit shared library 16 | g++ -std=c++11 -Wall -Werror -shared audit.cpp -o audit.so 17 | 18 | # build bitcoind 19 | pushd bitcoin 20 | make -C depends -j5 NO_WALLET=1 NO_QT=1 NO_ZMQ=1 21 | ./autogen.sh 22 | ./configure --prefix=/bitcoin/depends/x86_64-pc-linux-gnu 23 | make src/bitcoind -j5 24 | ``` 25 | 26 | Using the auditing library at runtime: 27 | 28 | ```bash 29 | LD_AUDIT="/audit.so" src/bitcoind 30 | 31 | Auditing interface version: 1 32 | loading shared object: , flag = LM_ID_BASE 33 | loading shared object: /lib64/ld-linux-x86-64.so.2, flag = LM_ID_BASE 34 | Link map activity: 35 | Cookie = 0x7f2523a76608, flag = LA_ACT_ADD 36 | loading shared object: /lib/x86_64-linux-gnu/libpthread.so.0, flag = LM_ID_BASE 37 | loading shared object: /usr/lib/x86_64-linux-gnu/libstdc++.so.6, flag = LM_ID_BASE 38 | loading shared object: /lib/x86_64-linux-gnu/libm.so.6, flag = LM_ID_BASE 39 | loading shared object: /lib/x86_64-linux-gnu/libgcc_s.so.1, flag = LM_ID_BASE 40 | loading shared object: /lib/x86_64-linux-gnu/libc.so.6, flag = LM_ID_BASE 41 | Link map activity: 42 | Cookie = 0x7f2523a76608, flag = LA_ACT_CONSISTENT 43 | Dynamic linker finished. Passing back to application 44 | 2019-12-04T14:57:29Z Feeding 23672 bytes of environment data into RNG 45 | 2019-12-04T14:57:29Z Bitcoin Core version v0.19.99.0-41919631d (release build) 46 | 47 | # .... running 48 | 49 | 2019-12-04T14:57:33Z FlushStateToDisk: write coins cache to disk (0 coins, 0kB) completed (0.00s) 50 | 2019-12-04T14:57:34Z Shutdown: done 51 | About to unload object. Cookie = 0x7f2523a76608 52 | About to unload object. Cookie = 0x7f2523a444e8 53 | About to unload object. Cookie = 0x7f2523a44a08 54 | About to unload object. Cookie = 0x7f2523a44f18 55 | About to unload object. Cookie = 0x7f2523a45428 56 | About to unload object. Cookie = 0x7f2523a45938 57 | About to unload object. Cookie = 0x7f2523a75e68 58 | ``` 59 | 60 | When running `bitcoin-qt`, as expected, there is substantially more dynamic linker activity. 61 | Note the activity happening after "Dynamic linker finished.". This is Qt dlopening `dbus`. 62 | 63 | ```bash 64 | LD_AUDIT="/home/bitcoin/audit.so" src/qt/bitcoin-qt 65 | Auditing interface version: 1 66 | loading shared object: , flag = LM_ID_BASE 67 | 68 | loading shared object: /lib64/ld-linux-x86-64.so.2, flag = LM_ID_BASE 69 | 70 | Link map activity: 71 | Cookie = 0x7f6f12d7f608, flag = LA_ACT_ADD 72 | 73 | loading shared object: /lib/x86_64-linux-gnu/libpthread.so.0, flag = LM_ID_BASE 74 | loading shared object: /lib/x86_64-linux-gnu/libfontconfig.so.1, flag = LM_ID_BASE 75 | loading shared object: /lib/x86_64-linux-gnu/libfreetype.so.6, flag = LM_ID_BASE 76 | loading shared object: /lib/x86_64-linux-gnu/libxcb.so.1, flag = LM_ID_BASE 77 | loading shared object: /lib/x86_64-linux-gnu/libdl.so.2, flag = LM_ID_BASE 78 | loading shared object: /lib/x86_64-linux-gnu/libm.so.6, flag = LM_ID_BASE 79 | loading shared object: /lib/x86_64-linux-gnu/libstdc++.so.6, flag = LM_ID_BASE 80 | loading shared object: /lib/x86_64-linux-gnu/libgcc_s.so.1, flag = LM_ID_BASE 81 | loading shared object: /lib/x86_64-linux-gnu/libc.so.6, flag = LM_ID_BASE 82 | loading shared object: /lib/x86_64-linux-gnu/libexpat.so.1, flag = LM_ID_BASE 83 | loading shared object: /lib/x86_64-linux-gnu/libuuid.so.1, flag = LM_ID_BASE 84 | loading shared object: /lib/x86_64-linux-gnu/libpng16.so.16, flag = LM_ID_BASE 85 | loading shared object: /lib/x86_64-linux-gnu/libz.so.1, flag = LM_ID_BASE 86 | loading shared object: /lib/x86_64-linux-gnu/libXau.so.6, flag = LM_ID_BASE 87 | loading shared object: /lib/x86_64-linux-gnu/libXdmcp.so.6, flag = LM_ID_BASE 88 | loading shared object: /lib/x86_64-linux-gnu/libbsd.so.0, flag = LM_ID_BASE 89 | 90 | Link map activity: 91 | Cookie = 0x7f6f12d7f608, flag = LA_ACT_CONSISTENT 92 | 93 | Dynamic linker finished. Passing back to application 94 | Link map activity: 95 | Cookie = 0x7f6f12d7f608, flag = LA_ACT_ADD 96 | 97 | loading shared object: /lib/x86_64-linux-gnu/libdbus-1.so.3, flag = LM_ID_BASE 98 | loading shared object: /lib/x86_64-linux-gnu/libsystemd.so.0, flag = LM_ID_BASE 99 | loading shared object: /lib/x86_64-linux-gnu/librt.so.1, flag = LM_ID_BASE 100 | loading shared object: /lib/x86_64-linux-gnu/liblzma.so.5, flag = LM_ID_BASE 101 | loading shared object: /lib/x86_64-linux-gnu/liblz4.so.1, flag = LM_ID_BASE 102 | loading shared object: /lib/x86_64-linux-gnu/libgcrypt.so.20, flag = LM_ID_BASE 103 | loading shared object: /lib/x86_64-linux-gnu/libgpg-error.so.0, flag = LM_ID_BASE 104 | 105 | Link map activity: 106 | Cookie = 0x7f6f12d7f608, flag = LA_ACT_CONSISTENT 107 | 108 | About to unload object. Cookie = 0x559366447348 109 | About to unload object. Cookie = 0x559366447808 110 | About to unload object. Cookie = 0x559366448188 111 | About to unload object. Cookie = 0x559366448648 112 | About to unload object. Cookie = 0x559366448b08 113 | About to unload object. Cookie = 0x55936646a808 114 | Link map activity: 115 | Cookie = 0x7f6f12d7f608, flag = LA_ACT_DELETE 116 | 117 | Link map activity: 118 | Cookie = 0x7f6f12d7f608, flag = LA_ACT_CONSISTENT 119 | 120 | About to unload object. Cookie = 0x7f6f12d7f608 121 | About to unload object. Cookie = 0x7f6f12b45498 122 | About to unload object. Cookie = 0x7f6f12b459b8 123 | About to unload object. Cookie = 0x7f6f12b45ed8 124 | About to unload object. Cookie = 0x7f6f128f5478 125 | About to unload object. Cookie = 0x7f6f128f5e98 126 | About to unload object. Cookie = 0x7f6f128f63b8 127 | About to unload object. Cookie = 0x7f6f128f6e18 128 | About to unload object. Cookie = 0x7f6f12377478 129 | About to unload object. Cookie = 0x7f6f12377a08 130 | About to unload object. Cookie = 0x7f6f128f5988 131 | About to unload object. Cookie = 0x7f6f12377f18 132 | About to unload object. Cookie = 0x7f6f12378478 133 | About to unload object. Cookie = 0x7f6f12378998 134 | About to unload object. Cookie = 0x7f6f1230a478 135 | About to unload object. Cookie = 0x559366447cc8 136 | About to unload object. Cookie = 0x7f6f12b44f78 137 | About to unload object. Cookie = 0x7f6f128f68c8 138 | About to unload object. Cookie = 0x7f6f12d7ee60 139 | ``` 140 | -------------------------------------------------------------------------------- /tbd-stubs.md: -------------------------------------------------------------------------------- 1 | # .tbd stubs 2 | 3 | .tbd (Text-based Dynamic Library) stubs are a new representation for dynamic 4 | libraries and frameworks and are now used in Apples SDKs. The files are YAML, 5 | thus are human readable and editable. 6 | 7 | The TAPI (Text-based Application Programming Interface) library/project comprises 8 | of tooling to create, edit and package .tbd stubs as well as the dynamic library 9 | used by ld to support linking using the stubs. 10 | 11 | The [apple-libtapi](https://github.com/tpoechtrager/apple-libtapi) project comprises 12 | of a (slightly modified) copy of Apples open source [tapi library](https://opensource.apple.com/tarballs/tapi/) as well as the LLVM (Clang) source required to build it. 13 | 14 | This project is one component of the Bitcoin Core macOS build toolchain (along 15 | with Clang, ld64 etc). 16 | 17 | # tapi & .tbd demo with libbitcoinconsensus (YMMV) 18 | 19 | Compile libconsensus, with reduced exports. 20 | This currently means building [#19522](https://github.com/bitcoin/bitcoin/pull/19522). 21 | 22 | ```bash 23 | ./autogen.sh 24 | ./configure --disable-tests --disable-bench --with-utils=no \ 25 | --with-daemon=no --with-gui=no --disable-wallet \ 26 | --with-libs=yes --enable-reduce-exports 27 | make -j8 28 | ``` 29 | 30 | Now use `tapi` to generate a .tbd stub from the libbitcoinconsensus.dylib 31 | ```bash 32 | xcrun tapi stubify src/.libs/libbitcoinconsensus.0.dylib 33 | # what's inside? 34 | cat src/.libs/libbitcoinconsensus.0.tbd 35 | ``` 36 | 37 | ```yaml 38 | --- !tapi-tbd-v3 39 | archs: [ x86_64 ] 40 | uuids: [ 'x86_64: FD431C9E-AF1F-365D-BAB6-F2C3B92921AE' ] 41 | platform: macosx 42 | flags: [ not_app_extension_safe ] 43 | install-name: '/usr/local/lib/libbitcoinconsensus.0.dylib' 44 | exports: 45 | - archs: [ x86_64 ] 46 | symbols: [ _bitcoinconsensus_verify_script, _bitcoinconsensus_verify_script_with_amount, 47 | _bitcoinconsensus_version, _secp256k1_context_clone, _secp256k1_context_create, 48 | _secp256k1_context_destroy, _secp256k1_context_no_precomp, 49 | _secp256k1_context_preallocated_clone, _secp256k1_context_preallocated_clone_size, 50 | _secp256k1_context_preallocated_create, _secp256k1_context_preallocated_destroy, 51 | _secp256k1_context_preallocated_size, _secp256k1_context_randomize, 52 | _secp256k1_context_set_error_callback, _secp256k1_context_set_illegal_callback, 53 | _secp256k1_ec_privkey_negate, _secp256k1_ec_privkey_tweak_add, 54 | _secp256k1_ec_privkey_tweak_mul, _secp256k1_ec_pubkey_combine, 55 | _secp256k1_ec_pubkey_create, _secp256k1_ec_pubkey_negate, 56 | _secp256k1_ec_pubkey_parse, _secp256k1_ec_pubkey_serialize, 57 | _secp256k1_ec_pubkey_tweak_add, _secp256k1_ec_pubkey_tweak_mul, 58 | _secp256k1_ec_seckey_negate, _secp256k1_ec_seckey_tweak_add, 59 | _secp256k1_ec_seckey_tweak_mul, _secp256k1_ec_seckey_verify, 60 | _secp256k1_ecdsa_recover, _secp256k1_ecdsa_recoverable_signature_convert, 61 | _secp256k1_ecdsa_recoverable_signature_parse_compact, _secp256k1_ecdsa_recoverable_signature_serialize_compact, 62 | _secp256k1_ecdsa_sign, _secp256k1_ecdsa_sign_recoverable, 63 | _secp256k1_ecdsa_signature_normalize, _secp256k1_ecdsa_signature_parse_compact, 64 | _secp256k1_ecdsa_signature_parse_der, _secp256k1_ecdsa_signature_serialize_compact, 65 | _secp256k1_ecdsa_signature_serialize_der, _secp256k1_ecdsa_verify, 66 | _secp256k1_nonce_function_default, _secp256k1_nonce_function_rfc6979, 67 | _secp256k1_scratch_space_create, _secp256k1_scratch_space_destroy ] 68 | weak-def-symbols: [ __ZTINSt3__115basic_stringbufIcNS_11char_traitsIcEENS_9allocatorIcEEEE, 69 | __ZTINSt3__118basic_stringstreamIcNS_11char_traitsIcEENS_9allocatorIcEEEE, 70 | __ZTINSt3__119basic_istringstreamIcNS_11char_traitsIcEENS_9allocatorIcEEEE, 71 | __ZTINSt3__119basic_ostringstreamIcNS_11char_traitsIcEENS_9allocatorIcEEEE, 72 | __ZTSNSt3__115basic_stringbufIcNS_11char_traitsIcEENS_9allocatorIcEEEE, 73 | __ZTSNSt3__118basic_stringstreamIcNS_11char_traitsIcEENS_9allocatorIcEEEE, 74 | __ZTSNSt3__119basic_istringstreamIcNS_11char_traitsIcEENS_9allocatorIcEEEE, 75 | __ZTSNSt3__119basic_ostringstreamIcNS_11char_traitsIcEENS_9allocatorIcEEEE ] 76 | ... 77 | ``` 78 | 79 | Now we want to write a program to test linking against our .tbd stub. 80 | First we should copy it and the header into `/usr/local`: 81 | ```bash 82 | cp src/.libs/libbitcoinconsensus.0.tbd /usr/local/lib/libbitcoinconsensus.tbd 83 | cp src/script/bitcoinconsensus.h /usr/local/include 84 | ``` 85 | 86 | Some c++ that utilizes one of the many libbitcoinconsensus APIs: 87 | ```cpp 88 | #include 89 | #include 90 | 91 | int main() { 92 | std::cout << "Libconsensus API Version: " << bitcoinconsensus_version() << "\n"; 93 | return 0; 94 | } 95 | ``` 96 | 97 | Compile and link: 98 | ```bash 99 | clang++ consensus.cpp -lbitcoinconsensus -v -o consensus 100 | ``` 101 | 102 | Run: 103 | ```bash 104 | ./consensus 105 | ... 106 | dyld: Library not loaded: /usr/local/lib/libbitcoinconsensus.0.dylib 107 | Referenced from: /path/to/your/binary/./consensus 108 | Reason: image not found 109 | [1] 45811 abort ./consensus 110 | ``` 111 | 112 | 😲 113 | 114 | As expected, the binary wont actually run, because we haven't installed the 115 | libbitcoinconsensus dylib. The tbd stubs are enough to link against at buildtime, 116 | but at run time, you still need the actual dynamic library. 117 | 118 | ```bash 119 | cp src/.libs/libbitcoinconsensus.0.dylib /usr/local/lib 120 | ``` 121 | 122 | Re-run: 123 | ```bash 124 | ./consensus 125 | Libconsensus API Version: 1 126 | ``` 127 | 128 | If there is a .tbd and a .dylib present in `/usr/local/lib` the .tbd seems to 129 | take precedence. You can test this by modifying the stub to remove the symbol we 130 | are using `_bitcoinconsensus_version`. When you try and recompile, linking will fail: 131 | ```bash 132 | Undefined symbols for architecture x86_64: 133 | "_bitcoinconsensus_version", referenced from: 134 | _main in consensus-67628f.o 135 | ld: symbol(s) not found for architecture x86_64 136 | ``` 137 | 138 | If you then delete the .tbd stub entirely, and try and recompile, it will succed, 139 | as it's now linking directly against the .dylib. 140 | 141 | # Diffing libtapi against upstream 142 | 143 | ## Apple LLVM Project (Clang) 144 | 145 | ```bash 146 | git clone https://github.com/apple/llvm-project 147 | git clone https://github.com/tpoechtrager/apple-libtapi 148 | 149 | cd llvm-project 150 | git checkout 729748d085a90bd2a4af36efbfb2dc33b4704de3 151 | 152 | rm -rf clang/test 153 | rm -rf clang/unittests/ 154 | rm -rf clang/www 155 | cd .. 156 | 157 | # compare clang 158 | diffoscope --exclude-directory-metadata=yes llvm-project/clang apple-libtapi/src/llvm/projects/clang/ 159 | ``` 160 | 161 | ## libtapi tarball 162 | 163 | ``` 164 | wget https://opensource.apple.com/tarballs/tapi/tapi-1100.0.11.tar.gz 165 | tar xf tapi-1100.0.11.tar.gz 166 | cd tapi-1100.0.11.tar.gz 167 | rm -rf test/ 168 | rm -rf unittests/ 169 | cd .. 170 | 171 | # compare libtapi 172 | diffoscope --exclude-directory-metadata=yes tapi-1100.0.11 apple-libtapi/src/libtapi 173 | ``` 174 | 175 | Example diff is available [here](https://gist.github.com/fanquake/1512109cc69d0a61f352e326f34bb90a). 176 | 177 | ## -allowable_client ld 178 | 179 | One of the modifications to the libtapi tarball is the removal of the `-allowable_client ld` 180 | link flag. 181 | 182 | `man ld` 183 | 184 | > -allowable_client name 185 | > 186 | > Restricts what can link against the dynamic library being created. By default 187 | > any code can link against any dylib. But if a dylib is supposed to be private 188 | > to a small set of clients, you can formalize that by adding a `-allowable_client` 189 | > for each client. 190 | 191 | This seems to work as advertised, resulting in an inability to link against a .dylib 192 | usless you are building a binary (or anything?) with an allowed name. i.e 193 | 194 | `private_lib.cpp` 195 | ```cpp 196 | int func_in_priv_lib(int a) { 197 | return a * 2; 198 | } 199 | ``` 200 | 201 | `private_lib.h` 202 | ```cpp 203 | int func_in_priv_lib(int); 204 | ``` 205 | 206 | `bin.cpp` 207 | ```cpp 208 | #include 209 | 210 | #include 211 | 212 | int main() { 213 | std::cout << func_in_priv_lib(4) << "\n"; 214 | return 0; 215 | } 216 | ``` 217 | 218 | Build libprivate with `-allowable_client ld`: 219 | ```bash 220 | clang++ private_lib.cpp -Wl,-dylib -o libprivate.dylib -Wl,-allowable_client,ld 221 | ``` 222 | 223 | Note the addition of a `LC_SUB_CLIENT` load command: 224 | ```bash 225 | Load command 11 226 | cmd LC_SUB_CLIENT 227 | cmdsize 16 228 | client ld (offset 12) 229 | ``` 230 | 231 | Try and link to libprivate when building `bin.cpp`: 232 | ```bash 233 | clang++ bin.cpp -I/path/to/dir -L/path/to/dir -o bin -lprivate 234 | ld: cannot link directly with dylib/framework, your binary is not an allowed client of /Users/michael/Desktop/allowable_client/libprivate.dylib for architecture x86_64 235 | clang: error: linker command failed with exit code 1 (use -v to see invocation) 236 | ``` 237 | 238 | Build but name the output `ld`: 239 | ```bash 240 | clang++ bin.cpp -I/path/to/dir -L/path/to/dir -o ld -lprivate 241 | ./ld 242 | 8 243 | ``` 244 | 245 | You can acheive the same result using `-client_name,ld`: 246 | ```bash 247 | clang++ bin.cpp -I/path/to/dir -L/path/to/dir -o ld -lprivate -Wl,-client_name,ld 248 | ./bin 249 | 8 250 | ``` 251 | -------------------------------------------------------------------------------- /fuzzing/README.md: -------------------------------------------------------------------------------- 1 | # Fuzzing using libFuzzer 2 | 3 | Build and run the container provided by [fuzz.dockerfile](fuzz.dockerfile). 4 | 5 | ```bash 6 | DOCKER_BUILDKIT=1 docker build --pull --no-cache -f fuzz.dockerfile -t fuzz_bitcoin . 7 | docker run -it --name fuzz_bitcoin --workdir /bitcoin fuzz_bitcoin /bin/bash 8 | ``` 9 | 10 | ### Build depends 11 | 12 | We want to test the changes added in PR [#17860](https://github.com/bitcoin/bitcoin/pull/17860): 13 | 14 | ```bash 15 | # build depends 16 | make -C depends NO_QT=1 NO_WALLET=1 NO_ZMQ=1 CC=clang-15 CXX=clang++-15 17 | ``` 18 | 19 | ### Patch Bitcoin Core 20 | 21 | Apply the diff below to `src/consensus/tx_check.cpp`, so the fuzzer has 22 | something to find. 23 | 24 | See [#17080](https://github.com/bitcoin/bitcoin/pull/17080) and [`CVE-2018-17144`](https://bitcoincore.org/en/2018/09/20/notice/) for more details. 25 | 26 | ```diff 27 | diff --git a/src/consensus/tx_check.cpp b/src/consensus/tx_check.cpp 28 | @@ -36,12 +36,6 @@ bool CheckTransaction(const CTransaction& tx, TxValidationState& state) 29 | // Failure to run this check will result in either a crash or an inflation bug, depending on the implementation of 30 | // the underlying coins database. 31 | - std::set vInOutPoints; 32 | - for (const auto& txin : tx.vin) { 33 | - if (!vInOutPoints.insert(txin.prevout).second) 34 | - return state.Invalid(TxValidationResult::TX_CONSENSUS, "bad-txns-inputs-duplicate"); 35 | - } 36 | ``` 37 | 38 | ### Configure for fuzzing and compile 39 | 40 | ```bash 41 | ./autogen.sh 42 | CONFIG_SITE=/bitcoin/depends/x86_64-pc-linux-gnu/share/config.site ./configure \ 43 | --enable-fuzz --with-sanitizers=fuzzer,address,undefined 44 | make -j6 45 | ``` 46 | 47 | ### Start fuzzing 48 | 49 | PR #17860 adds the `utxo_total_supply` target, which we'll fuzz: 50 | 51 | ```bash 52 | time FUZZ=utxo_total_supply src/test/fuzz/fuzz /qa-assets/fuzz_seed_corpus/utxo_total_supply 53 | ``` 54 | 55 | In a new window you could enter the container and tail logs: 56 | ```bash 57 | docker exec -it fuzz_bitcoin /bin/bash 58 | tail -f fuzz-*.log 59 | ... 60 | ==> fuzz-4.log <== 61 | #4814 NEW cov: 48490 ft: 244756 corp: 2125/68Kb lim: 150 exec/s: 3 rss: 609Mb L: 150/150 MS: 4 ShuffleBytes-CrossOver-ChangeBit-CrossOver- 62 | #4815 RELOAD cov: 48490 ft: 245135 corp: 2126/68Kb lim: 150 exec/s: 3 rss: 609Mb 63 | 64 | ==> fuzz-0.log <== 65 | #4883 NEW cov: 48331 ft: 245548 corp: 2194/69Kb lim: 151 exec/s: 3 rss: 613Mb L: 128/151 MS: 3 ChangeBit-ChangeByte-InsertByte- 66 | 67 | ==> fuzz-4.log <== 68 | #4837 NEW cov: 48490 ft: 245146 corp: 2127/68Kb lim: 150 exec/s: 3 rss: 609Mb L: 76/150 MS: 1 ChangeBit- 69 | 70 | ==> fuzz-1.log <== 71 | #4815 NEW cov: 48515 ft: 245185 corp: 2152/68Kb lim: 150 exec/s: 3 rss: 604Mb L: 59/150 MS: 1 ChangeBinInt- 72 | 73 | ==> fuzz-5.log <== 74 | #4834 NEW cov: 48285 ft: 244697 corp: 2135/67Kb lim: 150 exec/s: 3 rss: 608Mb L: 150/150 MS: 3 ChangeASCIIInt-InsertByte-CrossOver- 75 | 76 | ==> fuzz-2.log <== 77 | #4883 RELOAD cov: 48499 ft: 245244 corp: 2102/67Kb lim: 150 exec/s: 3 rss: 611Mb 78 | ``` 79 | 80 | ### Crash 81 | 82 | Eventually, libFuzzer will crash (I've shortend some of the output). 83 | 84 | Depending on your hardware, this could take many hours. 85 | 86 | ```bash 87 | #22638 NEW cov: 48625 ft: 251339 corp: 2317/93Kb lim: 185 exec/s: 2 rss: 683Mb L: 121/192 MS: 2 ChangeByte-InsertByte- 88 | #23497 RELOAD cov: 48632 ft: 251363 corp: 2318/93Kb lim: 192 exec/s: 2 rss: 683Mb 89 | #23557 NEW cov: 48632 ft: 251369 corp: 2319/94Kb lim: 192 exec/s: 2 rss: 683Mb L: 176/192 MS: 2 ShuffleBytes-CopyPart- 90 | fuzz: test/fuzz/utxo_total_supply.cpp:86: auto utxo_total_supply_fuzz_target(FuzzBufferType)::(anonymous class)::operator()() const: Assertion 'circulation == utxo_stats.total_amount' failed. 91 | ==24424== ERROR: libFuzzer: deadly signal 92 | #0 0xaaaab9ded48c in __sanitizer_print_stack_trace (/bitcoin/src/test/fuzz/fuzz+0x103d48c) (BuildId: 95a77d550305f582adecb8c934189282a87532cf) 93 | #1 0xaaaab9d6ad64 in fuzzer::PrintStackTrace() (/bitcoin/src/test/fuzz/fuzz+0xfbad64) (BuildId: 95a77d550305f582adecb8c934189282a87532cf) 94 | #2 0xaaaab9d52410 in fuzzer::Fuzzer::CrashCallback() (/bitcoin/src/test/fuzz/fuzz+0xfa2410) (BuildId: 95a77d550305f582adecb8c934189282a87532cf) 95 | #3 0xffff9b77c79c (linux-vdso.so.1+0x79c) (BuildId: 14511120b732425d2ab0b0bddf2450b1f31c691c) 96 | #4 0xffff9b2e0a0c (/lib/aarch64-linux-gnu/libc.so.6+0x80a0c) (BuildId: 09928b270aa19314161b21f565d1a9732c2c5332) 97 | #5 0xffff9b29a768 in raise (/lib/aarch64-linux-gnu/libc.so.6+0x3a768) (BuildId: 09928b270aa19314161b21f565d1a9732c2c5332) 98 | #6 0xffff9b2874b8 in abort (/lib/aarch64-linux-gnu/libc.so.6+0x274b8) (BuildId: 09928b270aa19314161b21f565d1a9732c2c5332) 99 | #7 0xffff9b2941e0 (/lib/aarch64-linux-gnu/libc.so.6+0x341e0) (BuildId: 09928b270aa19314161b21f565d1a9732c2c5332) 100 | #8 0xffff9b294248 in __assert_fail (/lib/aarch64-linux-gnu/libc.so.6+0x34248) (BuildId: 09928b270aa19314161b21f565d1a9732c2c5332) 101 | #9 0xaaaaba3341a4 in utxo_total_supply_fuzz_target(Span)::$_4::operator()() const /bitcoin/src/test/fuzz/utxo_total_supply.cpp:86:9 102 | #10 0xaaaaba332150 in utxo_total_supply_fuzz_target(Span)::$_7::operator()() const /bitcoin/src/test/fuzz/utxo_total_supply.cpp:156:17 103 | #11 0xaaaaba332150 in unsigned long CallOneOf)::$_5, utxo_total_supply_fuzz_target(Span)::$_6, utxo_total_supply_fuzz_target(Span)::$_7>(FuzzedDataProvider&, utxo_total_supply_fuzz_target(Span)::$_5, utxo_total_supply_fuzz_target(Span)::$_6, utxo_total_supply_fuzz_target(Span)::$_7) /bitcoin/src/./test/fuzz/util.h:42:27 104 | #12 0xaaaaba332150 in utxo_total_supply_fuzz_target(Span) /bitcoin/src/test/fuzz/utxo_total_supply.cpp:125:9 105 | #13 0xaaaaba37f6a4 in std::function)>::operator()(Span) const /usr/bin/../lib/gcc/aarch64-linux-gnu/12/../../../../include/c++/12/bits/std_function.h:591:9 106 | #14 0xaaaaba37f1f8 in LLVMFuzzerTestOneInput /bitcoin/src/test/fuzz/fuzz.cpp:178:5 107 | #15 0xaaaab9d537b0 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) (/bitcoin/src/test/fuzz/fuzz+0xfa37b0) (BuildId: 95a77d550305f582adecb8c934189282a87532cf) 108 | #16 0xaaaab9d3ee84 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) (/bitcoin/src/test/fuzz/fuzz+0xf8ee84) (BuildId: 95a77d550305f582adecb8c934189282a87532cf) 109 | #17 0xaaaab9d4433c in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/bitcoin/src/test/fuzz/fuzz+0xf9433c) (BuildId: 95a77d550305f582adecb8c934189282a87532cf) 110 | #18 0xaaaab9d6b42c in main (/bitcoin/src/test/fuzz/fuzz+0xfbb42c) (BuildId: 95a77d550305f582adecb8c934189282a87532cf) 111 | #19 0xffff9b28777c (/lib/aarch64-linux-gnu/libc.so.6+0x2777c) (BuildId: 09928b270aa19314161b21f565d1a9732c2c5332) 112 | #20 0xffff9b287854 in __libc_start_main (/lib/aarch64-linux-gnu/libc.so.6+0x27854) (BuildId: 09928b270aa19314161b21f565d1a9732c2c5332) 113 | #21 0xaaaab9d3a8ec in _start (/bitcoin/src/test/fuzz/fuzz+0xf8a8ec) (BuildId: 95a77d550305f582adecb8c934189282a87532cf) 114 | 115 | NOTE: libFuzzer has rudimentary signal handlers. 116 | Combine libFuzzer with AddressSanitizer or similar for better crash reports. 117 | SUMMARY: libFuzzer: deadly signal 118 | ``` 119 | 120 | I've included two of the crashers in this directory. 121 | 122 | You can run your instance of libFuzzer, using them as input, to recreate the same 123 | crash. i.e: 124 | 125 | ```bash 126 | time FUZZ=utxo_total_supply src/test/fuzz/fuzz /crash-9c947d9ff00fa36eca41ad27d337743fd5fee54b 127 | ``` 128 | 129 | You should also check that removing the patch from `src/consensus/tx_check.cpp` 130 | and using the same crash input does *not* result in a crash. 131 | 132 | ### Minimize crash input 133 | 134 | When fuzzing you will normally want to reduce any crash inputs to be as small as 135 | possible. This can be done using the `minimize_crash` flag. 136 | 137 | For example, re-running the fuzzer using this flag, the 172 byte crash input, 138 | crash-9c947..., can be reduced to < 110 bytes: 139 | 140 | ```bash 141 | time FUZZ=utxo_total_supply src/test/fuzz/fuzz /crash-9c947d9ff00fa36eca41ad27d337743fd5fee54b -minimize_crash=1 142 | ... 143 | CRASH_MIN: minimizing crash input: 'crash-9c947d9ff00fa36eca41ad27d337743fd5fee54b' (172 bytes) 144 | ... 145 | CRASH_MIN: minimizing crash input: './minimized-from-9c947d9ff00fa36eca41ad27d337743fd5fee54b' (170 bytes) 146 | ... 147 | CRASH_MIN: minimizing crash input: './minimized-from-91553b55b90d77a51c310844101bf6d31bb5d4cf' (150 bytes) 148 | ... 149 | CRASH_MIN: minimizing crash input: './minimized-from-e3ac2a491fb8215065377867b8d6975ecf72df0e' (139 bytes) 150 | ... 151 | CRASH_MIN: minimizing crash input: './minimized-from-faf5cd0c652b9e4f7ae087e0b24825f76e538dd9' (126 bytes) 152 | ... 153 | CRASH_MIN: minimizing crash input: 'minimized-from-6db724bc2201eb512f7397e0b8d06a5b3b6acf17' (108 bytes) 154 | CRASH_MIN: 'minimized-from-6db724bc2201eb512f7397e0b8d06a5b3b6acf17' (108 bytes) caused a crash. Will try to minimize it further 155 | ... 156 | INFO: Done MinimizeCrashInputInternalStep, no crashes found 157 | CRASH_MIN: failed to minimize beyond minimized-from-6db724bc2201eb512f7397e0b8d06a5b3b6acf17 (108 bytes), exiting 158 | ``` 159 | 160 | ### Faster fuzzing using /dev/shm as TMPDIR 161 | 162 | `/dev/shm` can be used to speed up the fuzzing process. This requires passing 163 | `--shm-size=Nm` to `docker` when starting the container. By default `/dev/shm` is 164 | only 64mb, which is not large enough to hold the seeds and data directories. 165 | 166 | You need `~20mb` per datadir (worker) and overhead for the seeds directory. So 167 | we'll just use 200m. 168 | 169 | ```bash 170 | docker run -it --name fuzz_bitcoin --shm-size=200m ... 171 | ``` 172 | 173 | Then, run the fuzzers, using the following: 174 | 175 | ```bash 176 | mkdir /dev/shm/fuzz_temp_seeds 177 | 178 | # copy the seed from qa-assets/fuzz_seed_corpus/utxo_total_supply/ into fuzz_temp_seeds 179 | cp ../qa-assets/fuzz_seed_corpus/utxo_total_supply/f66a2... /dev/shm/fuzz_temp_seeds/f66a2.. 180 | 181 | # run the fuzzer 182 | export TMPDIR=/dev/shm; 183 | time FUZZ=utxo_total_supply src/test/fuzz/fuzz /dev/shm/fuzz_temp_seeds 184 | ``` 185 | --------------------------------------------------------------------------------