├── 1.0_bitcoin_core_architecture.asciidoc ├── 1.1_regions.asciidoc ├── README.md ├── functional_test_framework.asciidoc └── images ├── chapter_1_0 ├── PeerManagerImpl.jpg ├── PeerManagerImpl.svg ├── PeerManagerImpl.yuml ├── ccviews.jpg ├── ccviews.svg ├── ccviews.yuml ├── chainstate.jpg ├── chainstate.svg ├── chainstate.yuml ├── chainstate2.jpg ├── chainstate2.svg ├── chainstate2.yuml ├── dbwrapper.jpg ├── dbwrapper.svg ├── dbwrapper.yuml ├── executables.jpg ├── executables.svg ├── notification_classes.jpg ├── notification_classes.svg └── notification_classes.yuml └── chapter_4_0 ├── test_framework.jpg ├── test_framework.svg └── test_framework.yuml /1.0_bitcoin_core_architecture.asciidoc: -------------------------------------------------------------------------------- 1 | [[bitcoin-architecture]] 2 | == 1.0 Bitcoin Architecture 3 | 4 | This text provides an overview of Bitcoin Core Architecture, describes the main components, how they interact, and shows relevant parts of the code. 5 | 6 | It also answers common questions such as "How does the node find other peers on the network?", "How is a new block or transaction received and validated?" or "How is a transaction created and broadcasted?" along with others. 7 | 8 | The commit https://github.com/bitcoin/bitcoin/commit/4b5659c6b115315c9fd2902b4edd4b960a5e066e[4b5659c6b1] can be used as a reference for the https://github.com/bitcoin/bitcoin/tree/4b5659c6b115315c9fd2902b4edd4b960a5e066e[project's codebase] at the time of writing. 9 | 10 | git clone https://github.com/bitcoin/bitcoin.git 11 | cd bitcoin 12 | git checkout -b text_branch 4b5659c6b1 13 | 14 | [[executables]] 15 | === Executables 16 | 17 | To be able to interact with the Bitcoin network, the user needs to connect to a Bitcoin node, a software whose main purposes are: 18 | 19 | * Download the blockchain. 20 | * Enforce the rules of the network. 21 | * Validate and relay the transactions. 22 | 23 | Running your own node is of utmost importance when spending or transferring bitcoin. The other option is trusting other nodes, which is a significant security hole. When doing it, the user is leaking personal data and trusting in the data and in the rules defined by others, which can be malicious agents or harmful to the network, or even harmful to the user. 24 | 25 | The other component necessary to store and move coins is a wallet. The primary function of the wallet is to manage the private keys and sign transactions. 26 | 27 | Bitcoin protocol does not have the concept of accounts, like banks. Instead, wallets manage a pool of unfathomable random numbers called private keys, which the user should keep secret. Bitcoin addresses are derived from these private keys and can be used to receive coins (and move them later). Only one receiving address must be used for each transaction. 28 | 29 | Signing a transaction means the user is moving the money and has authorized the transaction. To create the signature, the wallet will use the private key associated with the coin(s) the user wants to spend. 30 | 31 | Nodes and wallets are completely different things, although they can eventually come together in the same software. Node is related to the bitcoin network and protocol, while a wallet is related to one's private keys and transactions. It is crucial to know the difference between these two concepts to have a better understanding of Bitcoin architecture. 32 | 33 | Bitcoin Core has been the reference implementation since its first version. It is not just a single software. Bitcoin Core is a solution that includes a node, a graphical interface, and a command-line interface. There are also wallet features (including a sophisticated coin selection), but since version v0.21, wallets are no longer created by default. The reason for this is to make the use of multiple wallets more intuitive. 34 | 35 | To start a node, just run the main app bitcoind implemented in `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/bitcoind.cpp[src/bitcoind.cpp]`. This executable is expected to run as a background service (a daemon). It also provides an authorized [JSON-RPC](https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/doc/JSON-RPC-interface.md) and an unauthorized (limited) [REST](https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/doc/REST-interface.md) interface, so users and applications can access and work with the node. 36 | 37 | After running the daemon, the user will be able to interact with the node through a command-line application called `bitcoin-cli` that is implemented in `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/bitcoin-cli.cpp[src/bitcoin-cli.cpp]`. It interfaces with bitcoind's JSON-RPC server over HTTP and displays the results. 38 | 39 | Another simpler and friendlier option to start the node and operate it is starting bitcoin-qt, implemented in `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/qt/main.cpp[src/qt/main.cpp]`. This is an intuitive graphical interface, where all interactions take place via buttons. The user can create multiple wallets and check other information about the node, such as the peer connection and network statistics. 40 | 41 | .Bitcoin Core Executables 42 | image::images/chapter_1_0/executables.svg[] 43 | [CChainParams, align="center"] 44 | 45 | {empty} + 46 | [[protocol_p2p]] 47 | === Protocol - P2P 48 | 49 | Bitcoin is a peer-to-peer protocol. There is no central server that can determine the rules. So, to communicate with other peers and exchange information, the nodes need to establish a standard protocol so that they can understand each other. 50 | 51 | The file `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/protocol.h[src/protocol.h]` defines all types of messages (`https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/protocol.h#L62[namespace NetMsgType]`) that will be used in this communication. As can be seen in the code below, each message has a comment with a succinct description of its purpose. 52 | 53 | [source,c++] 54 | ---- 55 | /** 56 | * Bitcoin protocol message types. When adding new message types, don't forget 57 | * to update allNetMessageTypes in protocol.cpp. 58 | */ 59 | namespace NetMsgType { 60 | 61 | /** 62 | * The version message provides information about the transmitting node to the 63 | * receiving node at the beginning of a connection. 64 | */ 65 | extern const char* VERSION; 66 | /** 67 | * The verack message acknowledges a previously-received version message, 68 | * informing the connecting node that it can begin to send other messages. 69 | */ 70 | extern const char* VERACK; 71 | // ... 72 | /** 73 | * The inv message (inventory message) transmits one or more inventories of 74 | * objects known to the transmitting peer. 75 | */ 76 | extern const char* INV; 77 | /** 78 | * The getdata message requests one or more data objects from another node. 79 | */ 80 | extern const char* GETDATA; 81 | // ... 82 | } 83 | ---- 84 | 85 | But how does the node find the other peers to exchange messages? When running for the first time, the node connects to a bunch of servers denominated DNS Seeds which provide a list of IP addresses that have recently been running a Bitcoin client. After connecting to those IP addresses, the node starts to exchange messages with its peers. + 86 | DNS seeds are hardcoded and stored in `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/chainparams.cpp[src/chainparams.cpp]`. 87 | 88 | [source,c++] 89 | ---- 90 | vSeeds.emplace_back("seed.bitcoin.sipa.be"); // Pieter Wuille, only supports x1, x5, x9, and xd 91 | vSeeds.emplace_back("dnsseed.bluematt.me"); // Matt Corallo, only supports x9 92 | vSeeds.emplace_back("dnsseed.bitcoin.dashjr.org"); // Luke Dashjr 93 | vSeeds.emplace_back("seed.bitcoinstats.com"); // Christian Decker, supports x1 - xf 94 | vSeeds.emplace_back("seed.bitcoin.jonasschnelli.ch"); // Jonas Schnelli, only supports x1, x5, x9, and xd 95 | ---- 96 | 97 | New peers can also be manually added with the command `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/init.cpp#L429[-addnode=]`. The connection parameters, like `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.h#L74[DEFAULT_MAX_PEER_CONNECTIONS]` or `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.h#L66[MAX_ADDNODE_CONNECTIONS]`, can be found in the `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.h[net.h]` file. 98 | 99 | [[concurrency_model]] 100 | == Concurrency model 101 | 102 | Bitcoin Core does a lot of things at the same time. It downloads the blockchain, processes new transactions, validates new blocks, responds to user events and network events, etc. 103 | 104 | Therefore, a multithreaded application seems appropriate for this case. Threads allow multiple functions to be executed concurrently, improving the application's responsiveness considerably. Multithreading also allows the use of multiprocessors efficiently, enabling parallelism to perform intensive tasks. 105 | 106 | An example of a task in the Bitcoin Core that can use multiple threads is the verification of scripts in a block. Since there are many transactions in a block, parallelizing the execution greatly improves performance. 107 | 108 | Most threads are started (directly or indirectly) in `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/init.cpp#L1191[init.cpp:AppInitMain(...)]`. This is the Bitcoin node's main function. If the node is started through the `bitcoind` daemon, this function will be called inside the `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/bitcoind.cpp#L107[src/bitcoind.cpp:AppInit(...)]`. If it is started through the `bitcoin-qt` graphic interface, `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/node/interfaces.cpp#L83[src/node/interfaces.cpp:appInitMain(...)]` will call the function. 109 | 110 | Another relevant function is `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.cpp#L2455[CConnman::Start(...)]` since network-related threads are instantiated and executed in it. 111 | 112 | [source,c++] 113 | ---- 114 | bool AppInitMain(...) 115 | { 116 | // ... 117 | if (!node.connman->Start(*node.scheduler, connOptions)) { 118 | return false; 119 | } 120 | // ... 121 | } 122 | ---- 123 | 124 | The table below shows the threads that will be presented next. 125 | 126 | [%autowidth] 127 | |=== 128 | |Purpose | # threads | Task run 129 | 130 | |<> 131 | |nproc or 16 132 | |`ThreadScriptCheck()` 133 | 134 | |<> 135 | |1 136 | |`ThreadImport()` 137 | 138 | |<> 139 | |4 or more 140 | |`ThreadHTTP()` 141 | 142 | |<> 143 | |1 144 | |`ThreadDNSAddressSeed()` 145 | 146 | |<> 147 | |1 148 | |`ThreadSocketHandler()` 149 | 150 | |<> 151 | |1 152 | |`ThreadOpenConnections()` 153 | 154 | 155 | |<> 156 | |1 157 | |`ThreadOpenAddedConnections()` 158 | 159 | |<> 160 | |1 161 | |`ThreadMessageHandler()` 162 | 163 | |=== 164 | 165 | [[trace_tread]] 166 | === TraceThread 167 | 168 | `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/util/system.h#L484[TraceThread]` is a wrapper for a function that will be called only once. In Bitcoin Core code, it is usually used as _fn_ argument to thread constructor `https://en.cppreference.com/w/cpp/thread/thread/thread[std::thread (Fn&& fn, Args&&... args)]`. It is defined in `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/util/system.h[src/util/system.h]` file. 169 | 170 | [source,c++] 171 | ---- 172 | template void TraceThread(const char* name, Callable func) 173 | { 174 | util::ThreadRename(name); 175 | try 176 | { 177 | LogPrintf("%s thread start\n", name); 178 | func(); 179 | LogPrintf("%s thread exit\n", name); 180 | } 181 | catch (const boost::thread_interrupted&) 182 | { 183 | LogPrintf("%s thread interrupt\n", name); 184 | throw; 185 | } 186 | catch (const std::exception& e) { 187 | PrintExceptionContinue(&e, name); 188 | throw; 189 | } 190 | catch (...) { 191 | PrintExceptionContinue(nullptr, name); 192 | throw; 193 | } 194 | } 195 | ---- 196 | 197 | [[script-verification]] 198 | === Script Verification 199 | 200 | The function that perform the script verification is `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/script/interpreter.cpp#L1960[bool src/script/interpreter.cpp:VerifyScript(...)]`. It is called in at least three points of the application: 201 | 202 | * when the node https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net_processing.cpp#L3001[receives a new transaction]. 203 | 204 | * when the https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/node/transaction.cpp#L29[node wants to broadcast a new transaction]. 205 | 206 | * when https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net_processing.cpp#L3529[receiving a new block] 207 | 208 | In the first two cases, `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validation.cpp#L1118[static bool validation.cpp:AcceptToMemoryPool(...)]` function is called to handle the new transaction, as can be seen in `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net_processing.cpp#L3064[ProcessMessage(...)]` and `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/node/transaction.cpp#L67[BroadcastTransaction(...)]`, which will try to add the transaction to mempool. + 209 | In the last case, the function that will handle the new block is `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validation.cpp#L1802[bool src/validation.cpp:CChainState::ConnectBlock(...)]`. + 210 | All three cases end up calling `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validation.cpp#L1377[src/validation.cpp:bool CheckInputScripts(...)]`. 211 | 212 | `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validation.cpp#L1377[CheckInputScripts(...)]` receives `const CTransaction& tx` transaction as a parameter and validates the scripts of all its inputs. However, the relevant parameter in this context is the `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validation.cpp#L206[std::vector *pvChecks = nullptr]`. `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validation.h#L269[CScriptCheck]` is a closure representing one script verification and it stores references to the spending transaction. 213 | 214 | [source,c++] 215 | ---- 216 | class CScriptCheck 217 | { 218 | private: 219 | CTxOut m_tx_out; 220 | const CTransaction *ptxTo; 221 | unsigned int nIn; 222 | unsigned int nFlags; 223 | bool cacheStore; 224 | ScriptError error; 225 | PrecomputedTransactionData *txdata; 226 | // ... 227 | } 228 | ---- 229 | 230 | The `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validation.cpp#L1324[bool src/validation.cpp:CScriptCheck::operator()()]` method overloads the operator `()` and performs the script validation (`https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validation.cpp#L1327[VerifyScript(...)]`). 231 | 232 | [source,c++] 233 | ---- 234 | bool CScriptCheck::operator()() { 235 | const CScript &scriptSig = ptxTo->vin[nIn].scriptSig; 236 | const CScriptWitness *witness = &ptxTo->vin[nIn].scriptWitness; 237 | return VerifyScript(scriptSig, m_tx_out.scriptPubKey, witness, nFlags, CachingTransactionSignatureChecker(ptxTo, nIn, m_tx_out.nValue, cacheStore, *txdata), &error); 238 | } 239 | ---- 240 | 241 | So if the `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validation.cpp#L1425[std::vector *pvChecks]` is not null, the `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validation.cpp#L1377[CheckInputScripts(...)]` will add each script validation (`https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validation.cpp#L1424[CScriptCheck check]`) to the vector, so they can be executed in parallel. Otherwise, the script is verified immediately. 242 | 243 | [source,c++] 244 | ---- 245 | bool CheckInputScripts(const CTransaction& tx, ..., std::vector *pvChecks) EXCLUSIVE_LOCKS_REQUIRED(cs_main) 246 | { 247 | // ... 248 | for (unsigned int i = 0; i < tx.vin.size(); i++) { 249 | CScriptCheck check(txdata.m_spent_outputs[i], tx, i, flags, cacheSigStore, &txdata); 250 | if (pvChecks) { 251 | pvChecks->push_back(CScriptCheck()); 252 | check.swap(pvChecks->back()); 253 | } else if (!check()) { 254 | // ... 255 | } 256 | // ... 257 | } 258 | // ... 259 | } 260 | ---- 261 | 262 | The only function that makes use of script validation parallelization is the aforementioned `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validation.cpp#L1802[bool CChainState::ConnectBlock(...)]` due to the number of transactions in a block. If the `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validation.cpp#L1990[g_parallel_script_checks]` is true, the script verification vector that has been filled in `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validation.cpp#L1377[CheckInputScripts(...)]` is allocated in `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validation.cpp#L1990[CCheckQueueControl control(...)]`. The `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validation.cpp#L2074[control.Wait()]` initiates the execution of each script and waits for the execution to be finished. + 263 | `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validation.cpp#L137[g_parallel_script_checks]` is a global variable and will be described in further detail soon. 264 | 265 | [source,c++] 266 | ---- 267 | bool CChainState::ConnectBlock(const CBlock& block, ...) 268 | { 269 | // ... 270 | CCheckQueueControl control(fScriptChecks && g_parallel_script_checks ? &scriptcheckqueue : nullptr); 271 | // ... 272 | 273 | for (unsigned int i = 0; i < block.vtx.size(); i++) 274 | { 275 | if (!tx.IsCoinBase()) 276 | { 277 | std::vector vChecks; 278 | if (!CheckInputScripts(tx,..., g_parallel_script_checks ? &vChecks : nullptr)) { /*...*/ } 279 | control.Add(vChecks); 280 | } 281 | } 282 | 283 | if (!control.Wait()) { 284 | LogPrintf("ERROR: %s: CheckQueue failed\n", __func__); 285 | return state.Invalid(BlockValidationResult::BLOCK_CONSENSUS, "block-validation-failed"); 286 | } 287 | } 288 | ---- 289 | 290 | The `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/checkqueue.h#L68[bool src/checkqueue.h:CCheckQueue::Loop(...)]` method calls `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/checkqueue.h#L126[check()]` to excute the verification work (in that case, the script verification). 291 | 292 | [source,c++] 293 | ---- 294 | // src/checkqueue.h 295 | template 296 | class CCheckQueue 297 | { 298 | private: 299 | /** Internal function that does bulk of the verification work. */ 300 | bool Loop(bool fMaster = false) 301 | { 302 | // ... 303 | do { 304 | // ... 305 | // execute work 306 | for (T& check : vChecks) 307 | if (fOk) 308 | fOk = check(); 309 | vChecks.clear(); 310 | } while (true); 311 | } 312 | ---- 313 | 314 | The number of script-checking threads is defined in `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/init.cpp#L1191[init.cpp:AppInitMain(...)]`. The user can set the number of threads using the argument `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/init.cpp#L405[-par]`. If the number is negative, it will limit the threads. + 315 | If the user does not pass the `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/init.cpp#L405[-par]` parameter, `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/util/system.cpp#L1336[src/util/system.cpp:GetNumCores()]` is called to get the number of concurrent threads supported by the implementation. Then 1 is subtracted from this number because the the main thread is already being used. `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/util/system.cpp#L1336[GetNumCores()]` is just a wrapper for C++ standard function `https://en.cppreference.com/w/cpp/thread/thread/hardware_concurrency[std::thread::hardware_concurrency()]`. + 316 | There is also a maximum number of dedicated script-checking threads allowed, which is 15 (`https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validation.h#L76[MAX_SCRIPTCHECK_THREADS]`). 317 | Note that `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/init.cpp#L1261[g_parallel_script_checks]` is set to true, allowing parallelization in the `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validation.cpp#L1802[ConnectBlock(...)]` function. 318 | 319 | [source,c++] 320 | ---- 321 | bool AppInitMain(...) 322 | { 323 | //... 324 | int script_threads = args.GetArg("-par", DEFAULT_SCRIPTCHECK_THREADS); 325 | if (script_threads <= 0) { 326 | // -par=0 means autodetect (number of cores - 1 script threads) 327 | // -par=-n means "leave n cores free" (number of cores - n - 1 script threads) 328 | script_threads += GetNumCores(); 329 | } 330 | 331 | // Subtract 1 because the main thread counts towards the par threads 332 | script_threads = std::max(script_threads - 1, 0); 333 | 334 | // Number of script-checking threads <= MAX_SCRIPTCHECK_THREADS 335 | script_threads = std::min(script_threads, MAX_SCRIPTCHECK_THREADS); 336 | 337 | LogPrintf("Script verification uses %d additional threads\n", script_threads); 338 | if (script_threads >= 1) { 339 | g_parallel_script_checks = true; 340 | StartScriptCheckWorkerThreads(script_threads); 341 | } 342 | //... 343 | } 344 | ---- 345 | 346 | And finally the command `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/init.cpp#L1262[StartScriptCheckWorkerThreads(script_threads)]` simply initiates a new worker thread one or several times, according to the `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/init.cpp#L1246[script_threads]` value. Its implementation can be found in `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/checkqueue.h#L142[src/checkqueue.h:StartWorkerThreads(...)]`. 347 | 348 | [source,c++] 349 | ---- 350 | class CCheckQueue 351 | { 352 | // .... 353 | //! Create a pool of new worker threads. 354 | void StartWorkerThreads(const int threads_num) 355 | { 356 | // ... 357 | assert(m_worker_threads.empty()); 358 | for (int n = 0; n < threads_num; ++n) { 359 | m_worker_threads.emplace_back([this, n]() { 360 | util::ThreadRename(strprintf("scriptch.%i", n)); 361 | Loop(false /* worker thread */); 362 | }); 363 | } 364 | } 365 | // ... 366 | } 367 | ---- 368 | // --- 369 | The worker thread mechanism was not originally like that. It has been changed recently in https://github.com/bitcoin/bitcoin/pull/18710[PR #18710], making it more efficient and https://github.com/bitcoin/bitcoin/pull/18710/files#diff-35390fbd9f90018a4bf7d663283bb8b812cc52c4e277e115eb9426c79df439a9L13[reducing the dependency] on ``. There is also an interesting https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/test/checkqueue_tests.cpp[CCheckQueue unit tests], implemented in https://github.com/bitcoin/bitcoin/pull/9497/files[PR #9497]. 370 | 371 | [[loading-blocks]] 372 | === Loading Blocks 373 | 374 | One of the first things the node needs to do is load the blocks and decide which chain to work. 375 | 376 | The thread `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validation.h#L864[std::thread m_load_block]` invokes the function `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/node/blockstorage.cpp#L173[void ThreadImport(...)]` to https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/init.cpp#L1795[load the blocks on startup]. If the user is rebuilding the blockchain index (`https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/init.cpp#L412[-reindex]`) or is loading blocks directly from files (`https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/init.cpp#L400[-loadblock]`), it will be handled in this thread. After loading the blocks, it tries to find the best chain in `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validation.cpp#L2720[CChainState::ActivateBestChain(...)]`. 377 | 378 | This happens in the `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/init.cpp#L1191[init.cpp:AppInitMain(...)]`. 379 | 380 | [source,c++] 381 | ---- 382 | // src/validation.h 383 | class ChainstateManager 384 | { 385 | // ... 386 | public: 387 | std::thread m_load_block; 388 | // ... 389 | } 390 | 391 | // src/init.cpp 392 | bool AppInitMain(NodeContext& node, interfaces::BlockAndHeaderTipInfo* tip_info) 393 | { 394 | // ... 395 | chainman.m_load_block = std::thread(&TraceThread>, "loadblk", [=, &chainman, &args] { 396 | ThreadImport(chainman, vImportFiles, args); 397 | }); 398 | // ... 399 | } 400 | ---- 401 | 402 | // --- 403 | Note that `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validation.h#L864[m_load_block]` is a member field of the `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validation.h#L807[ChainstateManager]` class. Originally, it was a global variable called `g_load_block` but has been changed in https://github.com/bitcoin/bitcoin/pull/21575[PR #21575] to break down the `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/init.cpp[src/init.cpp]` into smaller logical units. + 404 | `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validation.h#L807[ChainstateManager]` will be explained in the in <> section. 405 | 406 | [[servicing-rpc-calls]] 407 | === Servicing RPC Calls 408 | 409 | To allow the user to interact with the node, an HTTP server should be enabled to process the requests. In order to do so, the `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/init.cpp#L702[init.cpp:AppInitServers(...)]` calls `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/httpserver.cpp#L352[httpserver.cpp:InitHTTPServer()]` that, as the name implies, initializes the server and `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/httpserver.cpp#L420[httpserver.cpp:StartHTTPServer()]` which constructs new thread objects. 410 | 411 | `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/httpserver.cpp#L417[g_thread_http]` is the event dispatcher thread that manages the http event loop. It is interrupted when `InterruptHTTPServer()` is called. 412 | 413 | `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/httpserver.cpp#L418[g_thread_http_workers]` distributes the work over multiple threads and handles longer requests off the event loop thread. `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/httpserver.cpp#L333[HTTPWorkQueueRun]` is a simple wrapper to set the thread name and run the work queue. The number of threads to service RPC calls is defined by the configuration argument `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/httpserver.cpp#L423[-rpcthreads]` or `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/httpserver.h#L11[httpserver.h:DEFAULT_HTTP_THREADS=4]`, whichever is greater. 414 | 415 | [source,c++] 416 | ---- 417 | static std::thread g_thread_http; 418 | static std::vector g_thread_http_workers; 419 | 420 | void StartHTTPServer() 421 | { 422 | LogPrint(BCLog::HTTP, "Starting HTTP server\n"); 423 | int rpcThreads = std::max((long)gArgs.GetArg("-rpcthreads", DEFAULT_HTTP_THREADS), 1L); 424 | LogPrintf("HTTP: starting %d worker threads\n", rpcThreads); 425 | g_thread_http = std::thread(ThreadHTTP, eventBase); 426 | 427 | for (int i = 0; i < rpcThreads; i++) { 428 | g_thread_http_workers.emplace_back(HTTPWorkQueueRun, workQueue, i); 429 | } 430 | } 431 | ---- 432 | 433 | [[load-peer-adresses-from-dns-seeds]] 434 | === Load Peer Addresses From DNS Seeds 435 | 436 | As said before, the node initially queries the hardcoded DNS Seeds to find new peers to connect to. 437 | 438 | `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.h#L1234[net.h:std::thread threadDNSAddressSeed]` is a thread created with `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.cpp#L1597[CConnman::ThreadDNSAddressSeed(...)]` wrapped into `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.cpp#L2535[TraceThread(...)]`. It will run one time when node starts. 439 | 440 | It is called in `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/init.cpp#L1191[init.cpp:AppInitMain(...)]` function when the command `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/init.cpp#L1930[node.connman->Start(*node.scheduler, connOptions)]` is executed. 441 | // --- 442 | Note that if the `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/init.cpp#L436[-dnsseed]` argument is given as `false` on startup, https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.cpp#L2533[this thread will not be instantiated]. The default value is `true` (defined in `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.h#L85[DEFAULT_DNSSEED]`). 443 | 444 | [source,c++] 445 | ---- 446 | if (!gArgs.GetBoolArg("-dnsseed", DEFAULT_DNSSEED)) 447 | LogPrintf("DNS seeding disabled\n"); 448 | else 449 | threadDNSAddressSeed = std::thread(&TraceThread >, "dnsseed", std::function(std::bind(&CConnman::ThreadDNSAddressSeed, this))); 450 | ---- 451 | 452 | [[send-and-receive-messages-to-and-from-peers]] 453 | === Send And Receive Messages To And From Peers 454 | 455 | `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.h#L1235[std::thread threadSocketHandler]` is created using `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.cpp#L1578[CConnman::ThreadSocketHandler()]` method wrapped into `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.cpp#L2530[TraceThread(...)]`. 456 | 457 | [source,c++] 458 | ---- 459 | bool CConnman::Start(...) 460 | { 461 | threadSocketHandler = std::thread(&TraceThread >, "net", std::function(std::bind(&CConnman::ThreadSocketHandler, this))); 462 | } 463 | ---- 464 | 465 | It seems strange at first because `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/util/system.h#L484[TraceThread(...)]` ensures unique execution, and the node will send and receive messages several times while connected, not just one time. 466 | 467 | But a close look into the `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.cpp#L1578[CConnman::ThreadSocketHandler()]` code shows it has a loop that keeps running until it is eventually interrupted by the `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.h#L1226[interruptNet]` flag. 468 | 469 | [source,c++] 470 | ---- 471 | void CConnman::ThreadSocketHandler() 472 | { 473 | while (!interruptNet) 474 | { 475 | DisconnectNodes(); 476 | NotifyNumConnectionsChanged(); 477 | SocketHandler(); 478 | } 479 | } 480 | ---- 481 | 482 | This flag is set to `true` only in the `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.cpp#L2581[CConnman::Interrupt()]` that https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.cpp#L2589[interrupts the connection]. Note that the `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/threadinterrupt.h#L19[class CThreadInterrupt]` overloads the `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/threadinterrupt.cpp#L22[() operator]`. When this method is called, the flag is set to true. 483 | 484 | `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.cpp#L1186[CConnman::DisconnectNodes()]` disconnects any connected nodes if the `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.h#L1130[fNetworkActive]` is false. It can be disabled / enabled by `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/rpc/net.cpp#L825[setnetworkactive]` RPC command. The function also disconnects unused nodes and deletes disconnected nodes. 485 | 486 | `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.cpp#L1245[NotifyNumConnectionsChanged()]` updates the number of connections and notifies the client interface, if it is enabled, when the number of connections changes. 487 | 488 | `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.cpp#L1460[SocketHandler()]` handles socket connections, incoming messages (`https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.h#L752[pnode->vRecvMsg]`) and the messages to be sent (`https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.h#L413[pnode->vSend]`); 489 | 490 | [[initializing-network-connections]] 491 | === Initializing Network Connections 492 | 493 | The thread `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.h#L1237[std::thread threadOpenConnections]` opens and manages connections to other peers. 494 | The way this thread gets started depends on the `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/init.cpp#L433[-connect=]` parameter. + 495 | If `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/init.cpp#L433[-connect]` is set to 0, this `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.h#L1237[threadOpenConnections]` thread will not be created. + 496 | If a specific IP is set, there will be only one active outbound connection with that IP. + 497 | If the `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/init.cpp#L433[-connect]` parameter is not passed, all the https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.cpp#L2549[outbound network connections will be initiated]. 498 | 499 | [source,c++] 500 | ---- 501 | if (connOptions.m_use_addrman_outgoing || !connOptions.m_specified_outgoing.empty()) 502 | threadOpenConnections = std::thread(&TraceThread >, "opencon", std::function(std::bind(&CConnman::ThreadOpenConnections, this, connOptions.m_specified_outgoing))); 503 | ---- 504 | 505 | The total number of outbound connections `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.h#L837[m_max_outbound]` is defined in `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.h[src/net.h]`. It usually https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.h#L842[will be 11], the sum of the full relay (8), block relay (2) and feeler (1) connections. 506 | 507 | [source,c++] 508 | ---- 509 | /** Maximum number of automatic outgoing nodes over which we'll relay everything (blocks, tx, addrs, etc) */ 510 | static const int MAX_OUTBOUND_FULL_RELAY_CONNECTIONS = 8; 511 | /** Maximum number of addnode outgoing nodes */ 512 | static const int MAX_ADDNODE_CONNECTIONS = 8; 513 | /** Maximum number of block-relay-only outgoing connections */ 514 | static const int MAX_BLOCK_RELAY_ONLY_CONNECTIONS = 2; 515 | /** Maximum number of feeler connections */ 516 | static const int MAX_FEELER_CONNECTIONS = 1; 517 | 518 | void Init(...) { 519 | m_max_outbound = m_max_outbound_full_relay + m_max_outbound_block_relay + nMaxFeeler; 520 | } 521 | ---- 522 | 523 | The use of `-connect=0` to disable automatic outbound connections has been implemented in https://bitcoin.org/en/release/v0.14.0#p2p-protocol-and-network-code[v0.14], with https://github.com/bitcoin/bitcoin/pull/9002[PR #9002]. 524 | 525 | [[opening-added-network-connections]] 526 | === Opening Added Network Connections 527 | 528 | `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.h#L1238[std::thread threadMessageHandler]` is created using `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.cpp#L2118[CConnman::ThreadOpenAddedConnections()]` wrapped into `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/util/system.h#L484[TraceThread(...)]`. 529 | 530 | `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.cpp#L2118[CConnman::ThreadOpenAddedConnections()]` calls `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.cpp#L2064[CConnman::GetAddedNodeInfo()]` to retrieve https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/rpc/net.cpp#L274[the nodes that have been added manually]. Then `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.cpp#L2146[OpenNetworkConnection(...)]` is called to open connections with them. 531 | 532 | [source,c++] 533 | ---- 534 | // Initiate manual connections 535 | threadOpenAddedConnections = std::thread(&TraceThread >, "addcon", std::function(std::bind(&CConnman::ThreadOpenAddedConnections, this))); 536 | ---- 537 | 538 | [[process-messages-from-net-net-processing]] 539 | === Process Messages from `net` -> `net_processing` 540 | 541 | When the node starts, `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/init.cpp#L1191[init.cpp:AppInitMain(...)]` calls `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/init.cpp#L1930[node.connman->Start(*node.scheduler, connOptions)]`. 542 | 543 | `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.h#L1238[std::thread threadMessageHandler]` is created using `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.cpp#L2181[CConnman::ThreadMessageHandler]` wrapped into `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/util/system.h#L484[TraceThread(...)]`. 544 | 545 | [source,c++] 546 | ---- 547 | bool CConnman::Start(...) 548 | { 549 | // Process messages 550 | threadMessageHandler = std::thread(&TraceThread >, "msghand", std::function(std::bind(&CConnman::ThreadMessageHandler, this))); 551 | } 552 | ---- 553 | 554 | As already seen in <>, this code will not be executed once. `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/util/system.h#L484[TraceThread(...)]` ensures unique execution but the `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.cpp#L2181[CConnman::ThreadMessageHandler()]` has a loop that keeps running until it is eventually interrupted by the `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.h#L1218[flagInterruptMsgProc]` flag. 555 | 556 | This flag is set `true` only in the `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net.cpp#L2581[CConnman::Interrupt()]` that interrupts all connections. 557 | 558 | [source,c++] 559 | ---- 560 | void CConnman::ThreadMessageHandler() 561 | { 562 | while (!flagInterruptMsgProc) 563 | { 564 | // ... 565 | 566 | for (CNode* pnode : vNodesCopy) 567 | { 568 | if (pnode->fDisconnect) 569 | continue; 570 | 571 | // Receive messages 572 | bool fMoreNodeWork = m_msgproc->ProcessMessages(pnode, flagInterruptMsgProc); 573 | // ... 574 | // Send messages 575 | { 576 | LOCK(pnode->cs_sendProcessing); 577 | m_msgproc->SendMessages(pnode); 578 | } 579 | // ... 580 | } 581 | 582 | // ... 583 | } 584 | } 585 | ---- 586 | 587 | [[notification-mechanism]] 588 | === Notifications Mechanism (`ValidationInterface`) 589 | 590 | A lot of events happen simultaneously in Bitcoin Core: new messages arrive all the time, are processed, and sometimes, announcements need to be made. For example, if a wallet is connected to Bitcoin Core and a transaction related to this wallet arrives, the wallet needs to be notified; when a new block arrives, the chain and the wallet need to be updated; a transaction can also be removed from mempool, and it needs to be notified and so on. 591 | 592 | In good software architecture, the components that trigger notifications and listen to them are completely decoupled. The message producer sends the notification to the listeners, but it does not know (and does not care) how the recipient will process the message. The sender's primary concern should be to ensure that the message is delivered and do this asynchronously so as not to block any execution. 593 | 594 | A known pattern for asynchronous message service is called _message queue_. When a relevant event is triggered, a message will be stored on the queue until it is processed by the consumer and deleted. The class that implements this kind of service in Bitcoin Core is the `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/scheduler.h#L33[CScheduler]` and the method that keeps the queue running is `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/scheduler.cpp#L24[void CScheduler::serviceQueue()]`. The queue service is started as soon as the application is initiated on `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/init.cpp#L1191[AppInitMain(...)]`. This service will be described in more detail later. 595 | 596 | [source,c++] 597 | ---- 598 | bool AppInitMain(...) 599 | { 600 | // Start the lightweight task scheduler thread 601 | threadGroup.create_thread([&] { TraceThread("scheduler", [&] { node.scheduler->serviceQueue(); }); }); 602 | } 603 | ---- 604 | 605 | In Bitcoin Core, there are two main classes that implement the notification between the components, the `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.h#L78[CValidationInterface]`, which works as notification receivers (also known as the _subscribers_ ) and the `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.h#L180[CMainSignals]`, which works as only notification sender (also known as the _publisher_). When some event needs to be published, the message is sent by `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.cpp#L90[static CMainSignals g_signals]` to all the subscribers. 606 | 607 | // Not ZMQ 608 | 609 | `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.h#L78[CValidationInterface]` is the interface that any class interested in listening to the events should implement. The events are: `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.h#L94#L94[UpdatedBlockTip]`, `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.h#L100[TransactionAddedToMempool]`, `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.h#L134[TransactionRemovedFromMempool]`, `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.h#L141[BlockConnected]`, `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.h#L147[BlockDisconnected]`, `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.h#L164[ChainStateFlushed]`, `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.h#L171[BlockChecked]` and `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.h#L175[NewPoWValidBlock]`. 610 | 611 | [source,c++] 612 | ---- 613 | class CValidationInterface { 614 | protected: 615 | ~CValidationInterface() = default; 616 | virtual void UpdatedBlockTip(const CBlockIndex *pindexNew, const CBlockIndex *pindexFork, bool fInitialDownload) {} 617 | 618 | virtual void TransactionAddedToMempool(const CTransactionRef& tx, uint64_t mempool_sequence) {} 619 | 620 | virtual void TransactionRemovedFromMempool(const CTransactionRef& tx, MemPoolRemovalReason reason, uint64_t mempool_sequence) {} 621 | 622 | virtual void BlockConnected(const std::shared_ptr &block, const CBlockIndex *pindex) {} 623 | 624 | virtual void BlockDisconnected(const std::shared_ptr &block, const CBlockIndex* pindex) {} 625 | 626 | virtual void ChainStateFlushed(const CBlockLocator &locator) {} 627 | 628 | virtual void BlockChecked(const CBlock&, const BlockValidationState&) {} 629 | 630 | virtual void NewPoWValidBlock(const CBlockIndex *pindex, const std::shared_ptr& block) {}; 631 | friend class CMainSignals; 632 | }; 633 | ---- 634 | 635 | All of these methods represent the events, and although they are defined as `virtual`, they have an empty default implementation `{}`. So the subclasses only need to implement the methods/events that matter. 636 | 637 | The classes that implement them are `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net_processing.h#L37[src/net_processing.h:PeerManager]`, `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/index/base.h#L27[src/index/base.h:BaseIndex]`, `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/node/interfaces.cpp#L341[src/node/interfaces.cpp:NotificationsProxy]`, `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/rpc/mining.cpp#L936[src/rpc/mining.cpp:submitblock_StateCatcher]` and `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/zmq/zmqnotificationinterface.h#L15[src/zmq/zmqnotificationinterface.h:CZMQNotificationInterface]`. 638 | 639 | The code below shows `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net_processing.h#L37[src/net_processing.h:PeerManager]` implementing `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.h#L78[CValidationInterface]`. Note that the class does not implement the `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.h#L100[TransactionAddedToMempool(...)]`, `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.h#L134[TransactionRemovedFromMempool(...)]`, `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.h#L164[ChainStateFlushed(...)]`, which means it has no interest in these events. 640 | 641 | [source,c++] 642 | ---- 643 | class PeerManager final : public CValidationInterface, public NetEventsInterface { 644 | /** 645 | * Overridden from CValidationInterface. 646 | */ 647 | void BlockConnected(const std::shared_ptr& pblock, const CBlockIndex* pindexConnected) override; 648 | void BlockDisconnected(const std::shared_ptr &block, const CBlockIndex* pindex) override; 649 | /** 650 | * Overridden from CValidationInterface. 651 | */ 652 | void UpdatedBlockTip(const CBlockIndex *pindexNew, const CBlockIndex *pindexFork, bool fInitialDownload) override; 653 | /** 654 | * Overridden from CValidationInterface. 655 | */ 656 | void BlockChecked(const CBlock& block, const BlockValidationState& state) override; 657 | /** 658 | * Overridden from CValidationInterface. 659 | */ 660 | void NewPoWValidBlock(const CBlockIndex *pindex, const std::shared_ptr& pblock) override; 661 | // .. 662 | } 663 | ---- 664 | 665 | But it is not enough to just implement those methods. To listen to these events, it is necessary to register them as subscribers of `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.h#L180[CMainSignals]`, which is the only publisher so that they can receive the notifications. It is done by registering the `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.h#L78[CValidationInterface]` object through the `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.cpp#L121[RegisterSharedValidationInterface(...)]` or `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.cpp#L128[RegisterValidationInterface(...)]` functions. 666 | 667 | `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/net_processing.h#L37[PeerManager]`, `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/index/base.h#L27[BaseIndex]`, `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/zmq/zmqnotificationinterface.h#L15[CZMQNotificationInterface]` use `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.cpp#L128[RegisterValidationInterface(...)]` while `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/node/interfaces.cpp#L341[src/node/interfaces.cpp:NotificationsProxy]`, `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/rpc/mining.cpp#L936[src/rpc/mining.cpp:submitblock_StateCatcher]` use `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.cpp#L121[RegisterSharedValidationInterface(...)]`. The code below illustrates this. 668 | 669 | [source,c++] 670 | ---- 671 | bool AppInitMain(...) 672 | { 673 | // ... 674 | node.peerman.reset(new PeerManager(chainparams, *node.connman, node.banman.get(), *node.scheduler, chainman, *node.mempool)); 675 | RegisterValidationInterface(node.peerman.get()); 676 | // ... 677 | #if ENABLE_ZMQ 678 | g_zmq_notification_interface = CZMQNotificationInterface::Create(); 679 | 680 | if (g_zmq_notification_interface) { 681 | RegisterValidationInterface(g_zmq_notification_interface); 682 | } 683 | #endif 684 | //... 685 | } 686 | ---- 687 | [source,c++] 688 | ---- 689 | static RPCHelpMan submitblock() 690 | { 691 | // ... 692 | auto sc = std::make_shared(block.GetHash()); 693 | RegisterSharedValidationInterface(sc); 694 | bool accepted = EnsureChainman(request.context).ProcessNewBlock(Params(), blockptr, /* fForceProcessing */ true, /* fNewBlock */ &new_block); 695 | UnregisterSharedValidationInterface(sc); 696 | // ... 697 | } 698 | ---- 699 | 700 | Calling either of the two methods has the same effect. `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.cpp#L128[RegisterValidationInterface(...)]` receives raw pointer as a parameter, then converts it to a shared pointer with an empty block control and sends it to the `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.cpp#L121[RegisterSharedValidationInterface(...)]`. Note that the `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/node/interfaces.cpp#L341[src/node/interfaces.cpp:NotificationsProxy]` and `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/rpc/mining.cpp#L936[submitblock_StateCatcher]` classes, that call directly `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.cpp#L121[RegisterSharedValidationInterface(...)]` use `std::make_shared` to wrap the argument in a `std::shared_ptr`. The others call `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.cpp#L128[RegisterValidationInterface(...)]`. + 701 | Using shared pointers instead of raw pointers ensures the pointer is only deleted when the last reference is deleted. More details can be found in https://github.com/bitcoin/bitcoin/pull/18338[PR #18338]. 702 | 703 | [source,c++] 704 | ---- 705 | void RegisterSharedValidationInterface(std::shared_ptr callbacks) 706 | { 707 | // Each connection captures the shared_ptr to ensure that each callback is 708 | // executed before the subscriber is destroyed. For more details see #18338. 709 | g_signals.m_internals->Register(std::move(callbacks)); 710 | } 711 | 712 | void RegisterValidationInterface(CValidationInterface* callbacks) 713 | { 714 | // Create a shared_ptr with a no-op deleter - CValidationInterface lifecycle 715 | // is managed by the caller. 716 | RegisterSharedValidationInterface({callbacks, [](CValidationInterface*){}}); 717 | } 718 | ---- 719 | 720 | To register a new subscriber, `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.cpp#L121[RegisterSharedValidationInterface(...)]` calls `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.cpp#L125[g_signals.m_internals->Register(...)]`. + 721 | `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.cpp#L90[g_signals]` is a static `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.h#L180[CMainSignals]` that, as mentioned before, is the only publisher and `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.h#L182[m_internals]` is a `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.cpp#L26[MainSignalsInstance]` struct. 722 | 723 | This struct has two important properties: `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.cpp#L34[std::list m_list]` and `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.cpp#L41[SingleThreadedSchedulerClient m_schedulerClient]`. The first one is the list that stores the references for all the subscribers (objects that implement `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.h#L78[CValidationInterface]` interface), and the second one queues the messages to be sent and executes them serially. 724 | 725 | [source,c++] 726 | ---- 727 | struct MainSignalsInstance { 728 | private: 729 | struct ListEntry { std::shared_ptr callbacks; int count = 1; }; 730 | std::list m_list GUARDED_BY(m_mutex); 731 | // ... 732 | public: 733 | SingleThreadedSchedulerClient m_schedulerClient; 734 | 735 | void Register(std::shared_ptr callbacks) 736 | { 737 | // Register a new CValidationInterface subscriber 738 | } 739 | 740 | // ... 741 | } 742 | ---- 743 | 744 | `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.h#L180[CMainSignals]` is the class that broadcasts the notifications to all the subscribers. Note that some methods of this class have the same name as `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.h#L78[CValidationInterface]` class. This way, it is easy to identify which event is triggered since both the publisher and the subscriber use the same method name. Note that `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.h#L180[CMainSignals]` _does not_ implement `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.h#L78[CValidationInterface]`. That the methods have the same name is just a design decision. 745 | 746 | [source,c++] 747 | ---- 748 | // src/validationinterface.h 749 | class CMainSignals { 750 | private: 751 | std::unique_ptr m_internals; 752 | 753 | // ... 754 | 755 | public: 756 | 757 | // ... 758 | 759 | void UpdatedBlockTip(const CBlockIndex *, const CBlockIndex *, bool fInitialDownload); 760 | void TransactionAddedToMempool(const CTransactionRef&, uint64_t mempool_sequence); 761 | void TransactionRemovedFromMempool(const CTransactionRef&, MemPoolRemovalReason, uint64_t mempool_sequence); 762 | void BlockConnected(const std::shared_ptr &, const CBlockIndex *pindex); 763 | void BlockDisconnected(const std::shared_ptr &, const CBlockIndex* pindex); 764 | void ChainStateFlushed(const CBlockLocator &); 765 | void BlockChecked(const CBlock&, const BlockValidationState&); 766 | void NewPoWValidBlock(const CBlockIndex *, const std::shared_ptr&); 767 | }; 768 | ---- 769 | 770 | To notify each of the subscribers, the `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.h#L182[MainSignalsInstance m_internals]` iterates each `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.h#L78[CValidationInterface]` element (which is also called `callback`) and constructs a lambda with the params that the message has (in the case of `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.cpp#L202[TransactionAddedToMempool]`, they are the `tx` and `mempool_sequence`). + 771 | The https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.cpp#L204[lambda body] is the execution of `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.h#L100[CValidationInterface::TransactionAddedToMempool(...)]`. Instead of running the lambda immediately, https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.cpp#L179[it is allocated] in the `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validationinterface.cpp#L41[SingleThreadedSchedulerClient m_schedulerClient]` to be executed serially. 772 | 773 | [source,c++] 774 | ---- 775 | #define ENQUEUE_AND_LOG_EVENT(event, fmt, name, ...) \ 776 | do { \ 777 | auto local_name = (name); \ 778 | LOG_EVENT("Enqueuing " fmt, local_name, __VA_ARGS__); \ 779 | m_internals->m_schedulerClient.AddToProcessQueue([=] { \ 780 | LOG_EVENT(fmt, local_name, __VA_ARGS__); \ 781 | event(); \ 782 | }); \ 783 | } while (0) 784 | // ... 785 | void CMainSignals::TransactionAddedToMempool(const CTransactionRef& tx, uint64_t mempool_sequence) { 786 | auto event = [tx, mempool_sequence, this] { 787 | m_internals->Iterate([&](CValidationInterface& callbacks) { callbacks.TransactionAddedToMempool(tx, mempool_sequence); }); 788 | }; 789 | ENQUEUE_AND_LOG_EVENT(event, "%s: txid=%s wtxid=%s", __func__, 790 | tx->GetHash().ToString(), 791 | tx->GetWitnessHash().ToString()); 792 | } 793 | ---- 794 | 795 | And finally, to trigger an event, all that is needed is to call `GetMainSignals().[event_name]`. The `https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validation.cpp#L1058[MemPoolAccept::AcceptSingleTransaction]` function below illustrates this, https://github.com/bitcoin/bitcoin/blob/4b5659c6b115315c9fd2902b4edd4b960a5e066e/src/validation.cpp#L1084[sending the notification] when a new transaction is added to mempool, passing the transaction and the mempool sequence as parameters. 796 | 797 | [source,c++] 798 | ---- 799 | bool MemPoolAccept::AcceptSingleTransaction(const CTransactionRef& ptx, ATMPArgs& args) 800 | { 801 | // ... 802 | 803 | GetMainSignals().TransactionAddedToMempool(ptx, m_pool.GetAndIncrementSequence()); 804 | 805 | return true; 806 | } 807 | ---- 808 | 809 | The diagram below shows the notifications classes (and some of their fields) presented so far. 810 | 811 | .Notification Class Diagram 812 | image::images/chapter_1_0/notification_classes.svg[] 813 | [CChainParams, align="center"] 814 | 815 | [[references]] 816 | === References 817 | 818 | * https://btctranscripts.com/greg-maxwell/2017-08-28-gmaxwell-deep-dive-bitcoin-core-v0.15/[A deep dive into Bitcoin Core v0.15] 819 | * https://bitcoincore.reviews/15681[Allow one extra single-ancestor transaction per package] 820 | * https://blog.kaiko.com/an-in-depth-guide-into-how-the-mempool-works-c758b781c608[An in-depth guide into how the mempool works] 821 | * https://www.youtube.com/watch?v=L_sI_tXmy2U&t=1379s[An overview of Bitcoin Core architecture] 822 | * https://eprint.iacr.org/2017/1095.pdf[Analysis of the Bitcoin UTXO set] 823 | * https://www.youtube.com/watch?v=PoEoG6sP1hw[Assume UTXO with James O'Beirne] 824 | * https://github.com/jamesob/assumeutxo-docs/tree/2019-04-proposal/proposal[assumeutxo Proposal] 825 | * https://github.com/bitcoin/bips/blob/master/bip-0022.mediawiki[BIP 22] 826 | * https://en.bitcoin.it/wiki/Bitcoin_Core_0.11_(ch_2):_Data_Storage[Bitcoin Core 0.11 (ch 2): Data Storage] 827 | * https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2019-April/016825.html[bitcoin-dev: assumeutxo and UTXO snapshots] 828 | * https://bitcoin.stackexchange.com/questions/100139/can-i-use-blockfilterindex-in-pruned-mode[Can I use blockfilterindex in pruned mode?] 829 | * https://bitcoin.stackexchange.com/questions/88652/does-assumevalid-lower-the-security-of-bitcoin[Does assumevalid lower the security of Bitcoin?] 830 | * https://bitcoin.stackexchange.com/questions/57978/file-format-rev-dat[File format — rev*.dat] 831 | * https://bitcoin.stackexchange.com/a/54889[How does Bitcoin transmit transactions?] 832 | * https://stackoverflow.com/questions/39510143/how-to-use-create-boostmulti-index[How to use/create boost::multi_index] 833 | * https://bitcoindev.network/understanding-the-data/[Understanding the data behind Bitcoin Core] 834 | * https://gist.github.com/jnewbery/93f89b6062d7af932d92204fa04ebe70[Wallet development] 835 | * https://bitcoinedge.org/transcript/telaviv2019/wallet-architecture[Wallet Architecture] 836 | * https://bitcoin.stackexchange.com/questions/11104/what-is-the-database-for[What is the database for ?] 837 | * https://bitcoin.stackexchange.com/questions/50693/why-are-blk-dat-files-134200000-bytes[Why are blk*.dat files ~134200000 bytes?] 838 | * https://bitcoin.stackexchange.com/questions/78618/why-some-transactions-disappear-from-the-mempool[Why some transactions disappear from the mempool?] 839 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Onboarding Guide 2 | 3 | THIS GUIDE IS DEPRECATED. PLEASE CHECK OUT https://github.com/chaincodelabs/onboarding-to-bitcoin-core INSTEAD. 4 | 5 | Originally authored by lsilva01 6 | -------------------------------------------------------------------------------- /functional_test_framework.asciidoc: -------------------------------------------------------------------------------- 1 | [[bitcoin-functional-test-framework]] 2 | == Functional Test Framework 3 | 4 | This text presents the main concepts of the Bitcoin Functional Test Framework, shows the most common methods and how the tests are implemented through examples. The goal is to provide a solid foundation for readers to start studying and developing their own tests. 5 | 6 | The commit https://github.com/bitcoin/bitcoin/commit/b8593616dc2ab5b8f81edd8b2408d400e3b696cd[b8593616dc2] can be used as a reference for the https://github.com/bitcoin/bitcoin/tree/b8593616dc2ab5b8f81edd8b2408d400e3b696cd[project's codebase] at the time of writing. 7 | 8 | git clone https://github.com/bitcoin/bitcoin.git 9 | cd bitcoin 10 | git checkout -b text_branch b8593616dc2 11 | 12 | A functional test is a test that is performed to confirm that the functionality of an application or system is behaving as expected. In Bitcoin Core, it is used to test interactions of the user and other nodes through RPC and P2P interfaces. 13 | 14 | It allows the testing of full features that take multiple layers of the stack (network, network processing, validation and so on). Two common functional test cases, for example, are the network (P2P) behavior and the RPC command, which is the tool that allows the user to interact with Bitcoin Core. 15 | 16 | The functional tests are located in `https://github.com/bitcoin/bitcoin/tree/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test[/test/functional]` folder (not `https://github.com/bitcoin/bitcoin/tree/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/src/test[src/test]`). There are a few different categories (areas) of tests: 17 | 18 | |=== 19 | |Area | Description 20 | 21 | |`feature` 22 | |tests for full features that aren't wallet/mining/mempool, eg feature_rbf.py 23 | 24 | |`interface` 25 | |tests for other interfaces (REST, ZMQ, etc), eg interface_rest.py 26 | 27 | |`mempool` 28 | |tests for mempool behavior, eg mempool_reorg.py 29 | 30 | |`mining` 31 | |tests for mining features, eg mining_prioritisetransaction.py 32 | 33 | |`rpc` 34 | |tests for individual RPC methods or features, eg rpc_listtransactions.py 35 | 36 | |`tool` 37 | |tests for tools, eg tool_wallet.py 38 | 39 | |`wallet` 40 | |tests for wallet features, eg wallet_keypool.py 41 | |=== 42 | 43 | The tests can be run directly, with `` as shown below: 44 | 45 | test/functional/rpc_getchaintips.py 46 | 47 | Or indirectly through the test suite, implemented by `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_runner.py[test/functional/test_runner.py]`, as shown below. If no test file is passed as the parameter, all test files will be executed. 48 | 49 | test/functional/test_runner.py test/functional/rpc_getchaintips.py 50 | test/functional/test_runner.py test/functional/wallet* 51 | test/functional/test_runner.py 52 | 53 | The `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_runner.py#L327[--jobs]` or `-j` enables tests to be executed more quickly in parallel. The default value of this parameter is 4. 54 | 55 | test/functional/test_runner.py -j 60 56 | 57 | Other parameters can be found in the `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_runner.py#L326[--help]` option. It can also be verified directly in the code: 58 | 59 | [source,python] 60 | ---- 61 | def main(): 62 | # ... 63 | parser.add_argument('--help', '-h', '-?', action='store_true', help='print help text and exit') 64 | parser.add_argument('--jobs', '-j', type=int, default=4, help='how many test scripts to run in parallel. Default=4.') 65 | parser.add_argument('--keepcache', '-k', action='store_true', help='the default behavior is to flush the cache directory on startup. --keepcache retains the cache from the previous testrun.') 66 | parser.add_argument('--quiet', '-q', action='store_true', help='only print dots, results summary and failure logs') 67 | # ... 68 | ---- 69 | [[test_test_framework]] 70 | === `test/test_framework/*` 71 | 72 | In the `https://github.com/bitcoin/bitcoin/tree/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework[test/test_framework]` folder, there are some files that implement useful functionalities to help the developer write a test. Most of these functions will be reused for various tests. 73 | 74 | The table below describes some relevant files that make up the framework. 75 | 76 | |=== 77 | |File | Description 78 | 79 | |`https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/util.py[util.py]` 80 | |Helpful routines for regression testing such as assert functions (e.g., `assert_equal`, `assert_raises_rpc_error`) and other helper functions (e.g, `satoshi_round`) 81 | 82 | |`https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py[test_framework.py]` 83 | |Mainly `BitcoinTestFramework`, which is the base class for a bitcoin test script. 84 | 85 | |`https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/key.py[key.py]` 86 | |Test-only secp256k1 elliptic curve implementation (`EllipticCurve`, `ECPubKey`, `ECKey`, `generate_privkey()`) 87 | 88 | |`https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/script.py[script.py]` 89 | |Functionality to build scripts, as well as signature hash functions (e.g., `CScriptOp(int)`, `OP_CHECKSIGVERIFY = CScriptOp(0xad)`, `CScript`) 90 | 91 | |`https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/blocktools.py[blocktools.py]` 92 | |Utilities for manipulating blocks and transactions (e.g., `create_block`, `create_tx_with_script`). 93 | 94 | |`https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/p2p.py[p2p.py]` 95 | |Test objects for interacting with a bitcoind node over the p2p protocol (e.g., `P2PInterface`, `P2PDataStore`, `P2PConnection`) 96 | 97 | |`https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/messages.py[messages.py]` 98 | |Definitions for objects passed over the network (`CBlock`, `CTransaction`, etc, along with the network-level wrappers for them, `msg_block`, `msg_tx`, etc). 99 | 100 | |`https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_node.py[test_node.py]` 101 | |A class for representing a bitcoind node under test (`TestNode`). 102 | |=== 103 | 104 | The `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/p2p.py[p2p.py]` file was originally called `mininode.py`, a reference to the p2p interface objects used to connect to the `bitcoind` node. However, that name proved to be confusing for new contributors, so it has been renamed to `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/p2p.py[p2p.py]` in https://github.com/bitcoin/bitcoin/pull/19760[PR #19760]. 105 | 106 | [[bitcointestframework_section]] 107 | === BitcoinTestFramework 108 | 109 | they set up any number of nodes 110 | 111 | Every test is a subclass of `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L78[BitcoinTestFramework]`. They set up a chain in `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L96[-regtest]` mode (meaning that it is not necessary to wait 10 minutes for a block) and https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L396[they set up any number of nodes] to be used in the test. + 112 | Individual tests should subclass `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L78[BitcoinTestFramework]` and override the `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L350[set_test_params()]` and `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L426[run_test()]` methods. 113 | 114 | An attribute that is usually set in `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L350[set_test_params()]` is the `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L124[self.num_nodes]`, which indicates how many nodes the test will use. A node is a `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L229[bitcoind]` instance. Each bitcoind node is managed by a python `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_node.py#L53[TestNode]` object which is used to https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_node.py#L186[start]/ https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_node.py#L315[stop] the node, https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_node.py#L99[manage the node's data directory], read state about the node (e.g., https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_node.py#L210[process status], https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_node.py#L372[log file]), and interact with the node over different interfaces. 115 | 116 | The `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L121[BitcoinTestFramework.main()]` method https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L127[calls] `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L215[setup()]` and then `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L426[run_test()]`. Note that `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L282[shutdown()]` is called at the https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L151[end of main function] to handle the tear down, so that the tests do not have to worry about removing the nodes, closing down the network thread and cleaning up the used directories. 117 | 118 | [source,python] 119 | ---- 120 | class BitcoinTestFramework(metaclass=BitcoinTestMetaClass): 121 | # ... 122 | def main(self): 123 | # ... 124 | assert hasattr(self, "num_nodes"), "Test must set self.num_nodes in set_test_params()" 125 | 126 | try: 127 | self.setup() 128 | self.run_test() 129 | except JSONRPCException: 130 | # .... 131 | finally: 132 | exit_code = self.shutdown() 133 | sys.exit(exit_code) 134 | 135 | def setup(self): 136 | # ... 137 | config = self.config 138 | 139 | fname_bitcoind = os.path.join( 140 | config["environment"]["BUILDDIR"], 141 | "src", 142 | "bitcoind" + config["environment"]["EXEEXT"], 143 | ) 144 | fname_bitcoincli = os.path.join( 145 | config["environment"]["BUILDDIR"], 146 | "src", 147 | "bitcoin-cli" + config["environment"]["EXEEXT"], 148 | ) 149 | self.options.bitcoind = os.getenv("BITCOIND", default=fname_bitcoind) 150 | self.options.bitcoincli = os.getenv("BITCOINCLI", default=fname_bitcoincli) 151 | # ... 152 | self.setup_chain() 153 | self.setup_network() 154 | # ... 155 | 156 | def run_test(self): 157 | """Tests must override this method to define test logic""" 158 | raise NotImplementedError 159 | ---- 160 | 161 | The `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L215[setup()]` method gets the `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L226[bitcoind]` and `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L231[bitcoin-cli]` folder. Then, https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L474[they (and other parameters) can be passed] to `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_node.py#L53[TestNode]`. All the parameters supported by `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L78[BitcoinTestFramework]` and `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_node.py#L53[TestNode]` can be found in the `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L154[parse_args()]` method. 162 | 163 | Other methods that individual tests can also override to customize the test setup are `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L362[setup_chain()]`, `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L370[setup_network()]` and `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L391[setup_nodes()]`. 164 | 165 | `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L362[setup_chain()]` calls `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L697[_initialize_chain()]` to https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L750[initialize a pre-mined blockchain] for use by the test. It creates a https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L697[cache of a 199-block-long chain], afterward it creates `num_nodes` https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L750[copies from the cache]. 166 | 167 | `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L750[setup_nodes()]` https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L750[calls] `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L750[add_nodes(self.num_nodes, ...)]` to https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L750[instantiate TestNode objects] and then https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L750[starts them]. Each node runs on the localhost and has its own port number. The configuration file with the specified port number is written by the `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/util.py#L351[util.py:write_config()]` stand alone function. The `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L513[start_nodes()]` method starts multiple `bitcoinds` in different ports. 168 | 169 | `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L370[setup_network()]` simply calls `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L391[setup_nodes()]` and then, https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L388[connects the nodes to each other]. 170 | 171 | This entire process ensures that each node starts out with a few coins (a pre-mined chain of 200 blocks loaded from the cache) and that all the nodes are connected to each other. If the test needs to change the network topology, customize the node's start behavior, or customize the node's data directories, it can override any of those methods. 172 | 173 | [source,python] 174 | ---- 175 | class BitcoinTestFramework(metaclass=BitcoinTestMetaClass): 176 | # ... 177 | def _initialize_chain(self): 178 | # ... 179 | for i in range(8): 180 | cache_node.generatetoaddress( 181 | nblocks=25 if i != 7 else 24, 182 | address=gen_addresses[i % 4], 183 | ) 184 | 185 | assert_equal(cache_node.getblockchaininfo()["blocks"], 199) 186 | # ... 187 | # ... 188 | def setup_network(self): 189 | self.setup_nodes() 190 | 191 | for i in range(self.num_nodes - 1): 192 | self.connect_nodes(i + 1, i) 193 | self.sync_all() 194 | # ... 195 | def setup_nodes(self): 196 | # ... 197 | self.add_nodes(self.num_nodes, extra_args) 198 | self.start_nodes() 199 | # .... 200 | ---- 201 | // --- 202 | .BitcoinTestMetaClass 203 | [NOTE] 204 | =============================== 205 | Tests must override `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L350[set_test_params()]` and `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L426[run_test()]` but not `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L94[\\__init__()]` or `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L121[main()]`. If any of these standards are violated, a `TypeError` https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L72[will be raised]. 206 | This behavior is ensured by the `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L58[BitcoinTestMetaClass]` class, added in the https://github.com/bitcoin/bitcoin/pull/12856[PR #12856]. 207 | =============================== 208 | [[testnode_section]] 209 | === TestNode 210 | 211 | `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_node.py#L53[TestNode]` class represents a `bitcoind` node for use in functional tests. It uses the binary that was compiled as `bitcoind`. (don't forget to run `make` before expecting changes to be reflected in functional tests). The class contains: 212 | 213 | * the state of the node (whether it's running, etc) 214 | * a Python `subprocess.Popen` object https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_node.py#L210[representing the running process] 215 | * an https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_node.py#L128[RPC connection] to the node 216 | * one or more https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_node.py#L135[P2P connections] to the node 217 | 218 | For the most part, `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_node.py#L53[TestNode]` and its interfaces (i.e., RPC or p2p connection) are used to verify the behavior of nodes. 219 | 220 | The `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L555[BitcoinTestFramework:connect_nodes()]` method, mentioned in the previous section, uses `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L558[from_connection.addnode(ip_port, "onetry")]` to connect a `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_node.py#L53[TestNode]` object (`from_connection`) to a new peer, but in the `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_node.py#L53[TestNode]` class, there is not any `addnode` method. The explanation is that `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_node.py#L53[TestNode]` dispatches any unrecognized messages to the RPC connection. Therefore, since the `addnode` method does not exist, it will be handled as an RPC request to be sent to the node. This behavior is implemented in the `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_node.py#L178[\\__getattr__()]` method. 221 | 222 | [source,python] 223 | ---- 224 | # test_framework.py 225 | class BitcoinTestFramework(metaclass=BitcoinTestMetaClass): 226 | # ... 227 | def connect_nodes(self, a, b): 228 | def connect_nodes_helper(from_connection, node_num): 229 | ip_port = "127.0.0.1:" + str(p2p_port(node_num)) 230 | from_connection.addnode(ip_port, "onetry") 231 | # ... 232 | # ... 233 | # ... 234 | # test_node.py 235 | class TestNode(): 236 | def __getattr__(self, name): 237 | """Dispatches any unrecognised messages to the RPC connection or a CLI instance.""" 238 | if self.use_cli: 239 | return getattr(RPCOverloadWrapper(self.cli, True, self.descriptors), name) 240 | else: 241 | assert self.rpc_connected and self.rpc is not None, self._node_msg("Error: no RPC connection") 242 | return getattr(RPCOverloadWrapper(self.rpc, descriptors=self.descriptors), name) 243 | ---- 244 | 245 | `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_node.py#L53[TestNode]` also implements common node operations such as `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_node.py#L186[start()]`, `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_node.py#L315[stop_node()]`, `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_node.py#L524[add_p2p_connection()]` and others. 246 | 247 | If a more control over the node is required (e.g. ignore messages or introduce some specific malicious behavior), a `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/p2p.py#L306[P2PInterface]` is a better approach. 248 | 249 | [[p2pinterface_section]] 250 | === P2PInterface 251 | 252 | `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/p2p.py#L306[P2PInterface]` allows a more customizable interaction with the node. It is a high-level P2P interface class for communicating with a Bitcoin node. Each connection to a node using this interface is managed by a python `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/p2p.py#L306[P2PInterface]` class or derived object (which is owned by the TestNode object). 253 | 254 | To add a new `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/p2p.py#L306[P2PInterface]` connection to a node, there are two methods that can be used: 255 | 256 | * `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_node.py#L524[add_p2p_connection()]` adds an inbound p2p connection to the node. 257 | 258 | * `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_node.py#L559[add_outbound_p2p_connection()]` adds an outbound p2p connection from node, which can be a full-relay(`outbound-full-relay`) or a block-relay-only(`block-relay-only`) connection. 259 | 260 | Both methods add the new P2P connection to the `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_node.py#L135[TestNode.p2ps]` list of the node object. 261 | 262 | `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/p2p.py#L306[P2PInterface]` also provides high-level callbacks for processing P2P message payloads, as well as convenience methods for interacting with the node over P2P. + 263 | Individual test cases should subclass this and override the `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/p2p.py#L385-L440[on_*]` methods if they want to alter message handling behavior. + 264 | The code below shows this. Note that `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/p2p.py#L367[on_message()]` intercepts the message type and calls the `on_[msg_type]` method. 265 | 266 | [source,python] 267 | ---- 268 | # test/functional/test_framework/p2p.py 269 | class P2PInterface(P2PConnection): 270 | # ... 271 | def on_message(self, message): 272 | with p2p_lock: 273 | try: 274 | msgtype = message.msgtype.decode('ascii') 275 | self.message_count[msgtype] += 1 276 | self.last_message[msgtype] = message 277 | getattr(self, 'on_' + msgtype)(message) 278 | except: 279 | print("ERROR delivering %s (%s)" % (repr(message), sys.exc_info()[0])) 280 | raise 281 | 282 | def on_open(self): pass 283 | def on_close(self):pass 284 | def on_addr(self, message): pass 285 | def on_addrv2(self, message): pass 286 | def on_block(self, message): pass 287 | def on_blocktxn(self, message): pass 288 | # ... 289 | def on_tx(self, message): pass 290 | def on_wtxidrelay(self, message): pass 291 | # ... 292 | ---- 293 | 294 | As can be seen in the code, `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/p2p.py#L306[P2PInterface]` is a subclass of the `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/p2p.py#L135[P2PConnection]`, which implements low-level network operations, such as opening and closing the TCP connection to the node and reading bytes from and writing bytes to the socket. + 295 | This class contains no logic for handing the P2P message payloads. It must be subclassed and the `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/p2p.py#L251[on_message()]` callback must be overridden, as the `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/p2p.py#L306[P2PInterface]` class does. 296 | 297 | There are also two other classes: 298 | 299 | * `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/p2p.py#L636[P2PDataStore]`: A `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/p2p.py#L306[P2PInterface]` subclass that keeps a store of transactions and blocks and can respond correctly to `getdata` and `getheaders` messages 300 | * `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/p2p.py#L765[P2PTxInvStore]`: A `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/p2p.py#L306[P2PInterface]` subclass which stores a count of how many times each txid has been announced. 301 | 302 | These two classes are generally used in some mempool, transaction and block tests. But `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/p2p.py#L306[P2PInterface]` is used much more frequently. 303 | 304 | The diagram below shows the most relevant Test Framework classes. 305 | 306 | .Test Framework Classes 307 | image::images/chapter_4_0/test_framework.svg[] 308 | [TestFrameworkClasses, align="center"] 309 | [[test_1_sending_money]] 310 | === Test 1 - Sending Money 311 | 312 | Let's create some simple tests to see the test framework in action. A basic but important test is to check if a node is able to send money to another. 313 | 314 | The code below implements this test. 315 | 316 | [source,python] 317 | ---- 318 | #!/usr/bin/env python3 319 | from test_framework.test_framework import BitcoinTestFramework 320 | from test_framework.util import ( 321 | assert_equal, 322 | assert_greater_than 323 | ) 324 | 325 | class WalletSendTest(BitcoinTestFramework): 326 | def set_test_params(self): 327 | self.num_nodes = 2 328 | self.setup_clean_chain = True 329 | 330 | def skip_test_if_missing_module(self): 331 | self.skip_if_no_wallet() 332 | 333 | def run_test(self): 334 | 335 | assert_equal(self.nodes[0].getbalance(), 0) 336 | assert_equal(self.nodes[1].getbalance(), 0) 337 | 338 | assert_equal(len(self.nodes[0].listunspent()), 0) 339 | assert_equal(len(self.nodes[1].listunspent()), 0) 340 | 341 | self.nodes[0].generate(101) 342 | 343 | n1_receive = self.nodes[1].getnewaddress() 344 | self.nodes[0].sendtoaddress(n1_receive, 30) 345 | 346 | self.nodes[0].generate(1) 347 | self.sync_blocks() 348 | 349 | assert_greater_than(self.nodes[0].getbalance(), 50) 350 | assert_equal(self.nodes[1].getbalance(), 30) 351 | 352 | assert_equal(len(self.nodes[0].listunspent()), 2) 353 | assert_equal(len(self.nodes[1].listunspent()), 1) 354 | 355 | if __name__ == '__main__': 356 | WalletSendTest().main() 357 | 358 | ---- 359 | 360 | `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L78[BitcoinTestFramework]`, as previously mentioned, is the base class for all functional tests. The first thing to do is to create the subclass and then implement the `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L350[set_test_params()]` and the `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L426[run_test()]` methods. 361 | 362 | In `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L350[set_test_params()]`, the `num_nodes` https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L124[must be defined]. As the name implies, it specifies the number of nodes the test will use. This test uses two nodes (`self.num_nodes = 2`). 363 | 364 | The next line is `self.setup_clean_chain = True`. By default, every test https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L411[loads a pre-mined chain of 200 blocks from the cache], so the node will start the test with some money and be able to spend it. By setting `setup_clean_chain` to `True`, the chain will https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L366[start with an empty blockchain, with no pre-mined blocks]. It is useful if a test case wants complete control over initialization. 365 | 366 | The default behavior is `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L97[setup_clean_chain: bool = False]`, as can be seen in the code below. Therefore, to start with an empty blockchain, this property needs to be explicitly changed in the `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L350[set_test_params()]` method. 367 | 368 | The method which initializes an empty blockchain is the `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L770[_initialize_chain_clean()]` while the `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L697[_initialize_chain()]` builds a https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L744[cache of a 199-block-long chain]. The latter method was mentioned in the <> section. 369 | [source,python] 370 | ---- 371 | class BitcoinTestFramework(metaclass=BitcoinTestMetaClass): 372 | def __init__(self): 373 | self.chain: str = 'regtest' 374 | self.setup_clean_chain: bool = False 375 | # ... 376 | # ... 377 | def setup_chain(self): 378 | """Override this method to customize blockchain setup""" 379 | self.log.info("Initializing test directory " + self.options.tmpdir) 380 | if self.setup_clean_chain: 381 | self._initialize_chain_clean() 382 | else: 383 | self._initialize_chain() 384 | # ... 385 | def _initialize_chain_clean(self): 386 | for i in range(self.num_nodes): 387 | initialize_datadir(self.options.tmpdir, i, self.chain) 388 | ---- 389 | 390 | The `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L358[skip_test_if_missing_module()]` method is used to skip the test if it requires certain modules to be present. In that case, the test is using RPC functions that requires a wallet, such as `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/src/wallet/rpcwallet.cpp#L770[getbalance()]`, `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/src/wallet/rpcwallet.cpp#L2835[listunspent()]`, `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/src/wallet/rpcwallet.cpp#L233[getnewaddress()]` and `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/src/wallet/rpcwallet.cpp#L429[sendtoaddress()]`. 391 | 392 | `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L358[skip_if_no_wallet()]` will skip the test if the `bitcoind` was compiled with no wallet (`./configure --disable-wallet `). + 393 | Otherwise, it will ensure the creation of a default wallet. For this reason, the nodes of this test are able to directly access the funds without specifying a wallet (since v0.21, Bitcoin Core no longer creates a default wallet). 394 | [source,python] 395 | ---- 396 | class BitcoinTestFramework(metaclass=BitcoinTestMetaClass): 397 | # ... 398 | def setup_nodes(self): 399 | # ... 400 | if self.requires_wallet: 401 | self.import_deterministic_coinbase_privkeys() 402 | # ... 403 | def import_deterministic_coinbase_privkeys(self): 404 | for i in range(self.num_nodes): 405 | self.init_wallet(i) 406 | 407 | def init_wallet(self, i): 408 | wallet_name = self.default_wallet_name if self.wallet_names is None else self.wallet_names[i] if i < len(self.wallet_names) else False 409 | if wallet_name is not False: 410 | n = self.nodes[i] 411 | if wallet_name is not None: 412 | n.createwallet(wallet_name=wallet_name, descriptors=self.options.descriptors, load_on_startup=True) 413 | n.importprivkey(privkey=n.get_deterministic_priv_key().key, label='coinbase') 414 | # ... 415 | def skip_if_no_wallet(self): 416 | """Skip the running test if wallet has not been compiled.""" 417 | self.requires_wallet = True 418 | if not self.is_wallet_compiled(): 419 | raise SkipTest("wallet has not been compiled.") 420 | # ... 421 | # ... 422 | ---- 423 | 424 | If `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L358[skip_if_no_wallet()]` is not called, the test must create a wallet before using wallet operations, as shown below: 425 | [source,python] 426 | ---- 427 | self.nodes[0].createwallet(wallet_name="w0") 428 | wallet_node_0 = self.nodes[0].get_wallet_rpc("w0") 429 | address = wallet_node_0.getnewaddress() 430 | ---- 431 | 432 | There are other `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L778-L823[skip_if_no_*()]` functions in the `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L78[BitcoinTestFramework]` class, such as `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L800[skip_if_no_sqlite()]`, `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L805[skip_if_no_bdb()]`, `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L785[skip_if_no_bitcoind_zmq()]` and so on. The developer should check these methods if the test uses an optional module for compiling bitcoind. 433 | 434 | The next step in the test of sending money is the `run_test()` method, which implements the test. + 435 | It starts checking if the balance of each node is empty. (`assert_equal(self.nodes[0].getbalance(), 0)`). Note that `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/src/wallet/rpcwallet.cpp#L770[getbalance()]` is an RPC command. The next validation (`assert_equal(len(self.nodes[0].listunspent()), 0)`) is not really necessary since the node balances has already been verified, but it is there for purpose demonstration. 436 | 437 | `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_node.py#L300[TestNode.generate()]` method uses the `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/src/rpc/mining.cpp#L255[generatetoaddress]` RPC to https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_node.py#L302[mine new blocks immediately] to a node address. A pattern that can be noticed in the tests is the generation of 101 blocks. This is due to the https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/src/consensus/consensus.h#L19[COINBASE_MATURITY] consensus rules. It is defined in the `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/src/consensus/consensus.h[src/consensus/consensus.h]`. This rule means that coinbase transaction outputs can only be spent after a specific number of new blocks. At the moment, the number is 100. Therefore, when generating 101 blocks, the miner can spend the equivalent of 1 block (the first one that was generated). 438 | 439 | This explains the line `self.nodes[0].generate(101)`. 440 | 441 | Next, the second node generates a new address and the first node sends 30 BTC to it. But at this moment, the transaction exists only in the mempool. Then, the first node mines another block to settle the transaction. 442 | 443 | After that, the `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L615[sync_blocks()]` method is called. It waits until all nodes have the same tip. This is another method that is used quite often and usually after `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_node.py#L300[generate()]` method to wait for the block propagation. 444 | 445 | Then, the test checks whether the second received 30 BTC and the balance of the first node is greater than 50 BTC, since it received the block reward. 446 | 447 | The first node also should have 2 UTXOs (change output and the block reward) and the second, only one UTXO (the received money). 448 | 449 | More wallet tests can be found at `test/functional/wallet_*.py`. Two good tests to start with are `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/wallet_basic.py[wallet_basic.py]` and `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/wallet_send.py[wallet_send.py]`. 450 | [[test_2_expected_mempool_behavior]] 451 | === Test 2 - Expected Mempool Behavior 452 | 453 | The following code is a simple test that demonstrates basic mempool behavior and some common mempool test functions. 454 | 455 | [source,python] 456 | ---- 457 | #!/usr/bin/env python3 458 | from test_framework.test_framework import BitcoinTestFramework 459 | from test_framework.util import ( 460 | assert_equal, 461 | assert_greater_than 462 | ) 463 | 464 | class MempoolSimpleTest(BitcoinTestFramework): 465 | def set_test_params(self): 466 | self.num_nodes = 2 467 | 468 | def skip_test_if_missing_module(self): 469 | self.skip_if_no_wallet() 470 | 471 | def run_test(self): 472 | assert_greater_than(self.nodes[0].getbalance(), 30) 473 | 474 | assert_equal(self.nodes[0].getmempoolinfo()["size"], 0) 475 | assert_equal(self.nodes[0].getmempoolinfo()["unbroadcastcount"], 0) 476 | 477 | n1_receive = self.nodes[1].getnewaddress() 478 | txid = self.nodes[0].sendtoaddress(n1_receive, 30) 479 | 480 | assert_equal(self.nodes[0].getmempoolinfo()["size"], 1) 481 | assert_equal(self.nodes[0].getmempoolinfo()["unbroadcastcount"], 1) 482 | assert txid in self.nodes[0].getrawmempool() 483 | assert txid not in self.nodes[1].getrawmempool() 484 | 485 | self.sync_mempools() 486 | 487 | assert_equal(self.nodes[0].getmempoolinfo()["unbroadcastcount"], 0) 488 | 489 | assert_equal(self.nodes[1].getmempoolinfo()["size"], 1) 490 | assert_equal(self.nodes[1].getmempoolinfo()["unbroadcastcount"], 0) 491 | assert txid in self.nodes[1].getrawmempool() 492 | 493 | self.nodes[0].generate(1) 494 | 495 | self.sync_blocks() 496 | 497 | assert txid not in self.nodes[0].getrawmempool() 498 | assert txid not in self.nodes[1].getrawmempool() 499 | 500 | if __name__ == "__main__": 501 | MempoolSimpleTest().main() 502 | ---- 503 | 504 | The first steps are basically the same as in the previous example: declare a subclass of `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L78[BitcoinTestFramework]`, set the number of the nodes in `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L350[set_test_params()]` and if the test uses wallets, call `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L790[skip_if_no_wallet()]` in `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L358[skip_test_if_missing_module()]`. Then write the test in `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L426[run_test()]`. 505 | 506 | The main difference, however, from the previous example is that `setup_clean_chain = True` is not present. This command is only necessary when the test requires complete control over initialization. For this test, spending the coinbase transaction outputs with which the nodes start is sufficient. 507 | 508 | The first line ensures the first node has at least 30 BTC available. The second line introduces the RPC command `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/src/rpc/blockchain.cpp#L1652[getmempoolinfo()]`, which returns details on the active state of the transactions memory pool. 509 | 510 | The relevant details for this test are the `size` which represents the current transaction count and the `unbroadcastcount` which shows the current number of transactions that haven't been broadcasted yet. 511 | 512 | The second and third lines ensure that the node's mempool starts empty. The fourth and fifth lines create a transaction from node 0 to node 1. It is very similar to how it was done in the previous example, but this time, the test captures the transaction ID to check if it exists in the mempool. 513 | 514 | The next lines confirm that node 0 (which created the transaction) contains the transaction in its mempool but not node 1 since it has not been propagated yet. 515 | 516 | This is done by verifying that the mempool size of node 0 is 1 and also has one unbroadcasted transaction. 517 | 518 | An interesting way to check if a mempool contains a specific transaction is through the RPC command `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/src/rpc/blockchain.cpp#L583[getrawmempool()]`, which returns all transaction ids in the memory pool as an array. Then, check that the array contains the transaction being searched. 519 | 520 | The line `assert txid in self.nodes[0].getrawmempool()` does this. 521 | 522 | The command `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L637[self.sync_mempools()]` waits until all nodes have the same transactions in their memory pools. 523 | 524 | Afterward, with the mempools synchronized, all the tests are redone to ensure the mempool as node 1 has the same transactions of the mempool of node 0. 525 | 526 | `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L615[self.sync_blocks()]` has already been seen in the previous example, but what matters here is that the transaction must be removed from mempool after being included in a block. 527 | 528 | The two last lines do this check. 529 | 530 | This example showed some important functions that are commonly used in the mempool tests. 531 | 532 | More mempool tests can be found at `test/functional/mempool_*.py`. Two good tests to start with are `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/mempool_accept.py[mempool_accept.py]` and `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/mempool_spend_coinbase.py[mempool_spend_coinbase.py]`. 533 | 534 | .Sync* Functions 535 | [NOTE] 536 | =============================== 537 | The `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L78[BitcoinTestFramework]` class has three syncing functions: 538 | 539 | * `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L615[sync_blocks()]` waits for all nodes to have the same tip. 540 | * `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L637[sync_mempools()]` waits for all nodes to have the same transactions in their mempools. 541 | * `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L660[sync_all()]` does both. 542 | =============================== 543 | [[test_3_adding_p2pinterface_connections]] 544 | === Test 3 - Adding `P2PInterface` Connections 545 | 546 | The code below is simplified version of the `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/p2p_add_connections.py[test/functional/p2p_add_connections.py]`. It shows how to add a P2P connection and validate them. 547 | 548 | [source,python] 549 | ---- 550 | #!/usr/bin/env python3 551 | from test_framework.p2p import P2PInterface 552 | from test_framework.test_framework import BitcoinTestFramework 553 | from test_framework.util import assert_equal 554 | 555 | 556 | def check_node_connections(*, node, num_in, num_out): 557 | info = node.getnetworkinfo() 558 | assert_equal(info["connections_in"], num_in) 559 | assert_equal(info["connections_out"], num_out) 560 | 561 | 562 | class P2PAddConnections(BitcoinTestFramework): 563 | def set_test_params(self): 564 | self.num_nodes = 2 565 | 566 | def setup_network(self): 567 | self.setup_nodes() 568 | 569 | def run_test(self): 570 | self.log.info("Add 8 outbounds to node 0") 571 | for i in range(8): 572 | self.log.info(f"outbound: {i}") 573 | self.nodes[0].add_outbound_p2p_connection( 574 | P2PInterface(), p2p_idx=i, connection_type="outbound-full-relay") 575 | 576 | self.log.info("Add 2 block-relay-only connections to node 0") 577 | for i in range(2): 578 | self.log.info(f"block-relay-only: {i}") 579 | self.nodes[0].add_outbound_p2p_connection( 580 | P2PInterface(), p2p_idx=i + 8, connection_type="block-relay-only") 581 | 582 | self.log.info("Add 5 inbound connections to node 1") 583 | for i in range(5): 584 | self.log.info(f"inbound: {i}") 585 | self.nodes[1].add_p2p_connection(P2PInterface()) 586 | 587 | self.log.info("Check the connections opened as expected") 588 | check_node_connections(node=self.nodes[0], num_in=0, num_out=10) 589 | check_node_connections(node=self.nodes[1], num_in=5, num_out=0) 590 | 591 | self.log.info("Disconnect p2p connections") 592 | self.nodes[0].disconnect_p2ps() 593 | check_node_connections(node=self.nodes[0], num_in=0, num_out=0) 594 | 595 | 596 | if __name__ == '__main__': 597 | P2PAddConnections().main() 598 | ---- 599 | 600 | The `check_node_connections()` method gets the result of `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/src/rpc/net.cpp#L571[getnetworkinfo()]` RPC command which retrieves network information, including the number of inbound connections (`https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/src/rpc/net.cpp#L590[connections_in]`) 601 | and the number of outbound connections (`https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/src/rpc/net.cpp#L591[connections_out]`). 602 | 603 | It then verifies that the numbers returned by RPC command are the same as those passed as parameters, which are the number of the connections opened manually. 604 | 605 | The test class overrides the `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L370[setup_network()]` method. The default implementation connects all the nodes and this test manually adds the connections. `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L391[setup_nodes()]` starts the chain and the wallet (if enabled) but not the network. 606 | 607 | `self.log.info()` is a method used quite frequently in the tests. It https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L671[comes from Python Logging package]. It is used to describe the test and make clear the intention of the developer at each step. It should be used as much as necessary to ensure a good understanding of the test. 608 | 609 | Both `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_node.py#L559[add_outbound_p2p_connection()]` and `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_node.py#L524[add_p2p_connection()]` (which adds an inbound connection) receive a `https://github.com/bitcoin/bitcoin/blob/v0.21.1/test/functional/test_framework/p2p.py#L283[P2PInterface]` object as a parameter. + 610 | If the connection is of the outbound type, there is one more parameter (`connection_type`) to define if the connection type is `outbound-full-relay` or `block-relay-only`. 611 | 612 | To disconnect the nodes, the `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_node.py#L586[disconnect_p2ps()]` can be used. 613 | 614 | Some tests require the `https://github.com/bitcoin/bitcoin/blob/v0.21.1/test/functional/test_framework/p2p.py#L283[P2PInterface]` connections handle one or more message types. It should be done by creating a subclass that overrides the message types methods to provide custom message handling behavior, as seen in the <> section. 615 | 616 | A good example of this approach is `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/p2p_addrv2_relay.py[test/functional/p2p_addrv2_relay.py]`. + 617 | The `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/p2p_addrv2_relay.py#L31[AddrReceiver]` is `https://github.com/bitcoin/bitcoin/blob/v0.21.1/test/functional/test_framework/p2p.py#L283[P2PInterface]` subclass and overrides `on_addrv2()` method to https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/p2p_addrv2_relay.py#L37[add custom handling] for the `addrv2` message type. 618 | 619 | .Addr v2 Message Type 620 | [NOTE] 621 | =============================== 622 | addr v2 is a new version of the `addr` message in the Bitcoin P2P network protocol, which is used to advertise the addresses of nodes that accept incoming connections. 623 | It was proposed in https://github.com/bitcoin/bips/blob/master/bip-0155.mediawiki[BIP 155]. It adds support to v3 Tor hidden service addresses and other privacy-enhancing network protocols. 624 | =============================== 625 | 626 | [source,python] 627 | ---- 628 | # ... 629 | class AddrReceiver(P2PInterface): 630 | addrv2_received_and_checked = False 631 | 632 | def __init__(self): 633 | super().__init__(support_addrv2 = True) 634 | 635 | def on_addrv2(self, message): 636 | for addr in message.addrs: 637 | assert_equal(addr.nServices, 9) 638 | assert addr.ip.startswith('123.123.123.') 639 | assert (8333 <= addr.port < 8343) 640 | self.addrv2_received_and_checked = True 641 | 642 | def wait_for_addrv2(self): 643 | self.wait_until(lambda: "addrv2" in self.last_message) 644 | 645 | # ... 646 | class AddrTest(BitcoinTestFramework): 647 | # ... 648 | def run_test(self): 649 | # ... 650 | self.log.info( 651 | 'Check that addrv2 message content is relayed and added to addrman') 652 | addr_receiver = self.nodes[0].add_p2p_connection(AddrReceiver()) 653 | msg.addrs = ADDRS 654 | with self.nodes[0].assert_debug_log([ 655 | 'Added 10 addresses from 127.0.0.1: 0 tried', 656 | 'received: addrv2 (131 bytes) peer=0', 657 | 'sending addrv2 (131 bytes) peer=1', 658 | ]): 659 | addr_source.send_and_ping(msg) 660 | self.nodes[0].setmocktime(int(time.time()) + 30 * 60) 661 | addr_receiver.wait_for_addrv2() 662 | # ... 663 | ---- 664 | 665 | In the code above, the `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/p2p_addrv2_relay.py#L31[AddrReceiver]` class checks that every `addr` receive from `addrv2` messages https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/p2p_addrv2_relay.py#L38-L41[has the correct format]. It is done in the function `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/p2p_addrv2_relay.py#L37[on_addrv2]` that implements the `addrv2` handling. 666 | 667 | But there are more interesting details in this test. 668 | 669 | `assert_debug_log()` is a function that checks whether new entries have been added to the `debug.log` file and whether these entries match the text passed as a parameter. 670 | 671 | When multiple addresses are added, the https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/src/addrman.h#L658[message "Added %i addresses from ..." is recorded in the log]. + 672 | When the node receives a message, https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/src/net_processing.cpp#L2330[the message type and its size are recorded in the log]. The same applies https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/src/net.cpp#L2958[when sending a message]. 673 | 674 | `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/p2p.py#L538[send_and_ping(msg)]` is a `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/p2p.py#L306[P2PInterface]` method that sends a specific message (`msg`) to the node. In that case, the P2P interface is sending an https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/p2p_addrv2_relay.py#L22[ADDRV2 message with 10 addresses] to the node. 675 | 676 | [source,python] 677 | ---- 678 | class P2PInterface(P2PConnection): 679 | # ... 680 | # Message sending helper functions 681 | 682 | def send_and_ping(self, message, timeout=60): 683 | self.send_message(message) 684 | self.sync_with_ping(timeout=timeout) 685 | 686 | # Sync up with the node 687 | def sync_with_ping(self, timeout=60): 688 | self.send_message(msg_ping(nonce=self.ping_counter)) 689 | 690 | def test_function(): 691 | return self.last_message.get("pong") and self.last_message["pong"].nonce == self.ping_counter 692 | 693 | self.wait_until(test_function, timeout=timeout) 694 | self.ping_counter += 1 695 | # ... 696 | ---- 697 | 698 | After sending the `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/src/protocol.cpp#L16[ADDRV2]` message, the P2P interface calls `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/p2p.py#L550[sync_with_ping()]` to send a ping message to the node, and then waits to receive a pong before proceeding. The reason is to ensure the node processed the message. 699 | 700 | Nodes always respond to `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/src/protocol.cpp#L28[ping]` with `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/src/protocol.cpp#L29[pong]` and nodes process their messages from a single peer in the order in which they were received. In other words: if the P2P interface has gotten the `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/src/protocol.cpp#L29[pong]` back, it is known for a fact that all previous messages have been processed. 701 | 702 | Therefore, in that case, if the P2P interface receives `pong`, it means the previous message (`https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/src/protocol.cpp#L16[ADDRV2]`) was received and processed. 703 | 704 | `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/src/rpc/misc.cpp#L368[setmocktime()]` is an RPC command for `-regtest` mode only and is widely used in functional testing. It sets the local time of the node to a timestamp. Sending addresses to peers is https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/src/net_processing.cpp#L4174[controlled by random delay timer] (called `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/src/net.h#L553[m_next_addr_send]`) to improve privacy. Thus, the time of the node is advanced by half an hour to ensure that https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/src/net_processing.cpp#L4172[the timer is over] and the sending of addresses is already allowed. 705 | 706 | And finally, the `wait_for_addrv2()` method is basically a wrapper for `self.wait_until(lambda: `addrv2` in self.last_message)`. 707 | 708 | `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/p2p.py#L444[self.wait_until(...)]` makes the test waits for an arbitrary predicate to evaluate to `True`. In the case of the above code, it will wait until the last message is `addrv2`. 709 | 710 | But the test does not always need to implement its own predicate. There are already many `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/p2p.py#L452-L534[wait_for_*()]` functions implemented. If the test needs to wait for a transaction, for example, it should use `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/p2p.py#L462[wait_for_tx()]`. There is no need to reinvent the wheel. Other examples of these functions are `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/p2p.py#L470[wait_for_block()]`, `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/p2p.py#L485[wait_for_merkleblock()]`, `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/p2p.py#L476[wait_for_header()]` and so on. 711 | 712 | Note that most of these functions use `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/p2p.py#L444[wait_until(...)]`. Therefore, the developer should only use `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/p2p.py#L444[wait_until(...)]` if there is no `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/p2p.py#L452-L534[wait_for_*()]` function to the intended test. 713 | 714 | [source,python] 715 | ---- 716 | class P2PInterface(P2PConnection): 717 | # .... 718 | # Message receiving helper methods 719 | 720 | def wait_for_tx(self, txid, timeout=60): 721 | def test_function(): 722 | if not self.last_message.get('tx'): 723 | return False 724 | return self.last_message['tx'].tx.rehash() == txid 725 | 726 | self.wait_until(test_function, timeout=timeout) 727 | 728 | def wait_for_block(self, blockhash, timeout=60): 729 | def test_function(): 730 | return self.last_message.get("block") and self.last_message["block"].block.rehash() == blockhash 731 | 732 | self.wait_until(test_function, timeout=timeout) 733 | # ... 734 | ---- 735 | 736 | [[summary]] 737 | === Summary 738 | 739 | Bitcoin Functional Test Framework has 3 main classes: `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L78[BitcoinTestFramework]`, `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_node.py#L53[TestNode]` and `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/p2p.py#L306[P2PInterface]`. 740 | 741 | The `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L78[BitcoinTestFramework]` class is a base class for all functional tests. `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_node.py#L53[TestNode]` represents a `bitcoind` node for use in functional tests. `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/p2p.py#L306[P2PInterface]` allows a more customizable interaction with the node. 742 | 743 | The `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L350[set_test_params()]` and the `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L426[run_test()]` methods should be overridden to implement the test and the `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L124[self.num_nodes]` set the number of nodes that will be used in the test. 744 | 745 | By default, every test https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L411[loads a pre-mined chain of 200 blocks from the cache], but if `self.setup_clean_chain` is `True`, an empty chain will be loaded. 746 | 747 | `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L358[skip_test_if_missing_module()]` is used to skip the test if it requires certain modules to be present. The `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L778-L823[skip_if_no_*()]` methods should be called if the test uses an optional module for compiling bitcoind. 748 | 749 | `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L615[sync_blocks()]` waits for all nodes to have the same tip and `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L637[sync_mempools()]` waits for all nodes to have the same transactions in their mempools. `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_framework.py#L660[sync_all()]` does both. 750 | 751 | Nodes can connect to `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/p2p.py#L306[P2PInterface]` using `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_node.py#L559[add_outbound_p2p_connection()]` and `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/test_node.py#L524[add_p2p_connection()]`. The test can create subclasses of `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/p2p.py#L306[P2PInterface]` to handle specific message types. 752 | 753 | `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/p2p.py#L452-L534[wait_for_*()]` and `https://github.com/bitcoin/bitcoin/blob/b8593616dc2ab5b8f81edd8b2408d400e3b696cd/test/functional/test_framework/p2p.py#L444[wait_until(...)]` await the execution of expected behavior. 754 | 755 | [[references]] 756 | === References 757 | 758 | * Bitcoin Core Functional Test Framework - https://bitcoinedge.org/transcript/telaviv2019/bitcoin-core-functional-test-framework[Transcript] - https://telaviv2019.bitcoinedge.org/files/test-framework-in-bitcoin-core.pdf[Slides] 759 | 760 | * https://github.com/glozow/bitcoin-notes/blob/master/test_framework_intro.md[Functional Test Framework] 761 | -------------------------------------------------------------------------------- /images/chapter_1_0/PeerManagerImpl.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaincodelabs/bitcoin-core-onboarding/7c8a0cd7ec7d7c5287cdcf6feac553c1f89633e6/images/chapter_1_0/PeerManagerImpl.jpg -------------------------------------------------------------------------------- /images/chapter_1_0/PeerManagerImpl.yuml: -------------------------------------------------------------------------------- 1 | // Cool Class Diagram 2 | // ------------------ 3 | 4 | [CValidationInterface|virtual #UpdatedBlockTip(); virtual #TransactionAddedToMempool(); virtual #TransactionRemovedFromMempool(); virtual #BlockConnected(); virtual #BlockDisconnected(); virtual #ChainStateFlushed(); virtual #BlockChecked(); virtual #NewPoWValidBlock()] 5 | 6 | [NetEventsInterface|virtual +InitializeNode(); virtual +FinalizeNode(); virtual +ProcessMessages(); virtual +SendMessages();] 7 | 8 | [PeerManager|virtual +GetNodeStateStats(); virtual +IgnoresIncomingTxs(); virtual +RelayTransaction(); virtual +SendPings(); virtual +SetBestHeight(); virtual +Misbehaving(); virtual +CheckForStaleTipAndEvictPeers(); virtual +ProcessMessage();] 9 | 10 | [CValidationInterface]^[PeerManager] 11 | [NetEventsInterface]^[PeerManager] 12 | [PeerManager]^[PeerManagerImpl|...] 13 | -------------------------------------------------------------------------------- /images/chapter_1_0/ccviews.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaincodelabs/bitcoin-core-onboarding/7c8a0cd7ec7d7c5287cdcf6feac553c1f89633e6/images/chapter_1_0/ccviews.jpg -------------------------------------------------------------------------------- /images/chapter_1_0/ccviews.yuml: -------------------------------------------------------------------------------- 1 | [CoinsViews|+m_dbview;+m_cacheview;...|...]<>-1>[CCoinsViewDB|+m_db;...|BatchWrite();...]<>-1>[CDBWrapper|-pdb;-name;...|+Read();+Write();WriteBatch();...] 2 | 3 | // Add notes 4 | [CCoinsViewDB]-stores in[note: chainstate/ (LevelDB){bg:wheat}] 5 | 6 | [CCoinsView|+GetCoin();+HaveCoin();+GetBestBlock();+GetHeadBlocks();+BatchWrite();...] 7 | 8 | [CCoinsView]^-[CCoinsViewDB] 9 | [CCoinsView]^-[CCoinsViewBacked] 10 | [CCoinsViewBacked]^-[CCoinsViewCache|#hashBlock;#cacheCoins;...|HaveCoinInCache();AccessCoin();AddCoin();...] 11 | [CCoinsViewBacked]^-[CCoinsViewMemPool|#mempool|+CCoinsViewMemPool();+GetCoin()] 12 | [CoinsViews]<>-1>[CCoinsViewCache] -------------------------------------------------------------------------------- /images/chapter_1_0/chainstate.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaincodelabs/bitcoin-core-onboarding/7c8a0cd7ec7d7c5287cdcf6feac553c1f89633e6/images/chapter_1_0/chainstate.jpg -------------------------------------------------------------------------------- /images/chapter_1_0/chainstate.yuml: -------------------------------------------------------------------------------- 1 | [PeerManagerImpl|-m_chainman;...|+ProcessMessage();...]send data <>-1>[ChainstateManager|-m_ibd_chainstate;-m_snapshot_chainstate; -m_active_chainstate;...| +ActiveChainstate(); +ProcessNewBlock(); +ProcessNewBlockHeaders();...]<>-2>[CChainState|#m_coins_views;...|+FlushStateToDisk();+ActivateBestChain();+AcceptBlock();+DisconnectBlock();+ConnectBlock();-ActivateBestChainStep(); -ConnectTip();...]<>-1>[CoinsViews|+m_dbview;...|...]<>-1>[CCoinsViewDB|+m_db;...|BatchWrite();...]<>-1>[CDBWrapper|-pdb,-name;...|+Read();+Write();WriteBatch();...] 2 | 3 | // Add notes 4 | [CCoinsViewDB]-stores in[note: chainstate/ (LevelDB){bg:wheat}] 5 | 6 | [validation.cpp (file)|DumpMempool();LoadMempool();WriteBlockToDisk();ReadBlockFromDisk();...]uses -.->[CTxMemPool]-stores in[note: mempool.dat{bg:wheat}] 7 | 8 | [CDBWrapper]^-[CBlockTreeDB|...|+WriteBatchSync();...]-stores in[note: blocks/index (LevelDB){bg:wheat}] 9 | 10 | [validation.cpp (file)]uses -.->[CBlock]-stores in[note: blocks/ (.dat){bg:wheat}] -------------------------------------------------------------------------------- /images/chapter_1_0/chainstate2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaincodelabs/bitcoin-core-onboarding/7c8a0cd7ec7d7c5287cdcf6feac553c1f89633e6/images/chapter_1_0/chainstate2.jpg -------------------------------------------------------------------------------- /images/chapter_1_0/chainstate2.yuml: -------------------------------------------------------------------------------- 1 | [PeerManagerImpl|-m_chainman;...|+ProcessMessage();...]send data <>-1>[ChainstateManager|-m_ibd_chainstate;-m_snapshot_chainstate; -m_active_chainstate;...| +ActiveChainstate(); +ProcessNewBlock(); +ProcessNewBlockHeaders();...]<>-2>[CChainState|#m_coins_views;...|+FlushStateToDisk();+ActivateBestChain();+AcceptBlock();+DisconnectBlock();+ConnectBlock();-ActivateBestChainStep(); -ConnectTip();...]-stores in[note: chainstate/ (LevelDB){bg:wheat}] 2 | 3 | [validation.cpp (file)|DumpMempool();LoadMempool();WriteBlockToDisk();ReadBlockFromDisk();...]uses -.->[CTxMemPool]-stores in[note: mempool.dat{bg:wheat}] 4 | 5 | [CDBWrapper]^-[CBlockTreeDB|...|+WriteBatchSync();...]-stores in[note: blocks/index (LevelDB){bg:wheat}] 6 | 7 | [validation.cpp (file)]uses -.->[CBlock]-stores in[note: blocks/ (.dat){bg:wheat}] -------------------------------------------------------------------------------- /images/chapter_1_0/dbwrapper.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaincodelabs/bitcoin-core-onboarding/7c8a0cd7ec7d7c5287cdcf6feac553c1f89633e6/images/chapter_1_0/dbwrapper.jpg -------------------------------------------------------------------------------- /images/chapter_1_0/dbwrapper.yuml: -------------------------------------------------------------------------------- 1 | [CCoinsViewDB|+m_db;...|BatchWrite();...]<>-1>[CDBWrapper|-pdb;-name;...|+Read();+Write();WriteBatch();...] 2 | 3 | // Add notes 4 | [CCoinsViewDB]-stores in[note: chainstate/ (LevelDB){bg:wheat}] 5 | 6 | [CDBWrapper]^-[CBlockTreeDB|...|+WriteBatchSync();...]-stores in[note: blocks/index (LevelDB){bg:wheat}] 7 | 8 | [CDBWrapper]^-[BaseIndex::DB] 9 | [BaseIndex|#WriteBlock();+Start();...]<>-1>[BaseIndex::DB|+ReadBestBlock();+WriteBestBlock();...] 10 | [BaseIndex]^-[BlockFilterIndex|#WriteBlock();ReadFilterFromDisk();WriteFilterToDisk();...]-stores in[note: indexes/blockfilter (LevelDB){bg:wheat}] 11 | 12 | [BaseIndex::DB]^-[TxIndex::DB|+ReadTxPos();+WriteTxs();+MigrateData();...]-stores in[note: indexes/txindex (LevelDB){bg:wheat}] -------------------------------------------------------------------------------- /images/chapter_1_0/executables.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaincodelabs/bitcoin-core-onboarding/7c8a0cd7ec7d7c5287cdcf6feac553c1f89633e6/images/chapter_1_0/executables.jpg -------------------------------------------------------------------------------- /images/chapter_1_0/notification_classes.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaincodelabs/bitcoin-core-onboarding/7c8a0cd7ec7d7c5287cdcf6feac553c1f89633e6/images/chapter_1_0/notification_classes.jpg -------------------------------------------------------------------------------- /images/chapter_1_0/notification_classes.yuml: -------------------------------------------------------------------------------- 1 | [SingleThreadedSchedulerClient| -m_pscheduler; ...|-AddToProcessQueue();-MaybeScheduleProcessQueue();...]<>-1>[CScheduler| -taskQueue; ...| +schedule(); ...] 2 | 3 | [MainSignalsInstance| +m_schedulerClient; -m_list...]<>-1>[SingleThreadedSchedulerClient] 4 | 5 | [MainSignalsInstance]<>-0..*>[CValidationInterface|virtual #UpdatedBlockTip(); virtual #TransactionAddedToMempool(); virtual #TransactionRemovedFromMempool(); virtual #BlockConnected(); virtual #BlockDisconnected(); virtual #ChainStateFlushed(); virtual #BlockChecked(); virtual #NewPoWValidBlock()] 6 | 7 | [CMainSignals| m_internals | #UpdatedBlockTip(); #TransactionAddedToMempool(); #TransactionRemovedFromMempool(); #BlockConnected(); #BlockDisconnected(); #ChainStateFlushed(); #BlockChecked(); #NewPoWValidBlock()]<>-1>[MainSignalsInstance] 8 | 9 | [CValidationInterface]^[PeerManager] 10 | [CValidationInterface]^[BaseIndex] 11 | [CValidationInterface]^[NotificationsProxy] 12 | [CValidationInterface]^[submitblock_StateCatcher] 13 | [CValidationInterface]^[CZMQNotificationInterface] 14 | -------------------------------------------------------------------------------- /images/chapter_4_0/test_framework.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaincodelabs/bitcoin-core-onboarding/7c8a0cd7ec7d7c5287cdcf6feac553c1f89633e6/images/chapter_4_0/test_framework.jpg -------------------------------------------------------------------------------- /images/chapter_4_0/test_framework.yuml: -------------------------------------------------------------------------------- 1 | 2 | [BitcoinTestFramework|num_nodes;chain;nodes;...|set_test_params();run_test();setup_chain();setup_network();setup_nodes();... 3 | 4 | ]<>-0..*>[TestNode|rpc;cli;running;process;...|add_p2p_connection();add_outbound_p2p_connection();generate();start();stop_node();...]++-0..*>[P2PInterface|...|peer_connect();on_message();on_block();on_cmpctblock();on_tx();...] 5 | 6 | [P2PInterface]^-[P2PDataStore|...|on_getdata();on_getheaders();...] 7 | [P2PInterface]^-[P2PTxInvStore|...|on_inv();get_invs();wait_for_broadcast()] 8 | [P2PConnection|...|peer_connect();peer_disconnect();data_received();_on_data();send_message()]^-[P2PInterface] 9 | 10 | [BitcoinTestFramework]-metaclass >[BitcoinTestMetaClass|__new__()] --------------------------------------------------------------------------------