├── README.md ├── altair ├── beacon-chain.md ├── duties.png ├── lightsync.png ├── offlinechart.png ├── offlinechart2.png ├── piechart.png └── sync-protocol.md ├── capella └── beacon-chain.md ├── deneb ├── beacon-chain.md ├── fork-choice.md ├── polynomial-commitments.md └── validator.md ├── images ├── checkpoint_tree_1.png ├── checkpoint_tree_2.png ├── checkpoint_tree_3.png ├── checkpoint_tree_4.png ├── inactivityleak.png ├── light_client_sync.png ├── pow_chain.png ├── randomness.png ├── roadmap.jpeg ├── shardchains.png ├── singleslot.png └── src │ ├── checkpoint_tree_1.drawio │ ├── checkpoint_tree_2.drawio │ ├── checkpoint_tree_3.drawio │ └── checkpoint_tree_4.drawio ├── merge ├── beacon-chain.md └── fork-choice.md ├── phase0 ├── beacon-chain.md └── fork-choice.md └── phase1 └── beacon-chain.md /README.md: -------------------------------------------------------------------------------- 1 | Currently the following parts of the spec are available in annotated form: 2 | 3 | ### Phase 0 4 | 5 | * [The Beacon Chain](phase0/beacon-chain.md) 6 | * [Beacon Chain Fork Choice](phase0/fork-choice.md) 7 | 8 | ### Altair (Light client support, validator reward simplifications) 9 | 10 | * [Changes to the beacon chain](altair/beacon-chain.md) 11 | * [Light client sync protocol](altair/sync-protocol.md) 12 | 13 | ### Bellatrix (Merge) 14 | 15 | * [Beacon chain changes](merge/beacon-chain.md) 16 | * [Fork choice (terminal PoW block verification)](merge/fork-choice.md) 17 | 18 | ### Capella (Withdrawals) 19 | 20 | * [Beacon chain changes](capella/beacon-chain.md) 21 | 22 | ### Sharding 23 | 24 | * [Changes to the beacon chain](phase1/beacon-chain.md) 25 | -------------------------------------------------------------------------------- /altair/duties.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ethereum/annotated-spec/160764ac180eca2cea3581f731ee96ac7098f9f7/altair/duties.png -------------------------------------------------------------------------------- /altair/lightsync.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ethereum/annotated-spec/160764ac180eca2cea3581f731ee96ac7098f9f7/altair/lightsync.png -------------------------------------------------------------------------------- /altair/offlinechart.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ethereum/annotated-spec/160764ac180eca2cea3581f731ee96ac7098f9f7/altair/offlinechart.png -------------------------------------------------------------------------------- /altair/offlinechart2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ethereum/annotated-spec/160764ac180eca2cea3581f731ee96ac7098f9f7/altair/offlinechart2.png -------------------------------------------------------------------------------- /altair/piechart.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ethereum/annotated-spec/160764ac180eca2cea3581f731ee96ac7098f9f7/altair/piechart.png -------------------------------------------------------------------------------- /altair/sync-protocol.md: -------------------------------------------------------------------------------- 1 | # Minimal Light Client 2 | 3 | **Notice**: This document is a work-in-progress for researchers and implementers. 4 | 5 | ## Table of contents 6 | 7 | 8 | 9 | 10 | 11 | - [Introduction](#introduction) 12 | - [Constants](#constants) 13 | - [Configuration](#configuration) 14 | - [Misc](#misc) 15 | - [Containers](#containers) 16 | - [`LightClientSnapshot`](#lightclientsnapshot) 17 | - [`LightClientUpdate`](#lightclientupdate) 18 | - [`LightClientStore`](#lightclientstore) 19 | - [Helper functions](#helper-functions) 20 | - [`get_subtree_index`](#get_subtree_index) 21 | - [Light client state updates](#light-client-state-updates) 22 | - [`validate_light_client_update`](#validate_light_client_update) 23 | - [`apply_light_client_update`](#apply_light_client_update) 24 | - [`process_light_client_update`](#process_light_client_update) 25 | 26 | 27 | 28 | 29 | ## Introduction 30 | 31 | The **sync committee** is the "flagship feature" of the Altair hard fork. This is a committee of 512 validators that is randomly selected every **sync committee period (~1 day)**, and while a validator is part of the currently active sync committee they are expected to continually sign the block header that is the new head of the chain at each slot. 32 | 33 | The purpose of the sync committee is to allow **light clients** to keep track of the chain of beacon block headers. The other two duties that involve signing block headers, block proposal and block attestation, do not work for this function because computing the proposer or attesters at a given slot requires a calculation on the entire active validator set, which light clients do not have access to (if they did, they would not be light!). Sync committees, on the other hand, are (i) updated infrequently, and (ii) saved directly in the beacon state, allowing light clients to verify the sync committee with a Merkle branch from a block header that they already know about, and use the public keys in the sync committee to directly authenticate signatures of more recent blocks. 34 | 35 | This diagram shows the basic procedure by which a light client learns about more recent blocks: 36 | 37 | ![](lightsync.png) 38 | 39 | We assume a light client already has a block header at slot `N`, in period `X = N // 16384`. The light client wants to authenticate a block header somewhere in period `X+1`. The steps the light client needs to take are as follows: 40 | 41 | 1. Use a Merkle branch to verify the `next_sync_committee` in the slot `N` post-state. This is the sync committee that will be signing block headers during period `X+1`. 42 | 2. Download the aggregate signature of the newer header that the light client is trying to authenticate. This can be found in the child block of the newer block, or the information could be grabbed directly from the p2p network. 43 | 3. Add together the public keys of the subset of the sync committee that participated in the aggregate signature (the bitfield in the signature will tell you who participated) 44 | 4. Verify the signature against the combined public key and the newer block header. If verification passes, the new block header has been successfully authenticated! 45 | 46 | The minimum cost for light clients to track the chain is only about 25 kB per two days: 24576 bytes for the 512 48-byte public keys in the sync committee, plus a few hundred bytes for the aggregate signature, the light client header and the Merkle branch. Of course, closely following the chain at a high level of granularity does require more data bandwidth as you would have to download each header you're verifying (plus the signature for it). 47 | 48 | The extremely low cost for light clients is intended to help make the beacon chain light client friendly for extremely constrained environments. Such environments include mobile phones, embedded IoT devices, in-browser wallets and other blockchains (for cross-chain bridges). 49 | 50 | ## Constants 51 | 52 | | Name | Value | 53 | | - | - | 54 | | `FINALIZED_ROOT_INDEX` | `get_generalized_index(BeaconState, 'finalized_checkpoint', 'root')` | 55 | | `NEXT_SYNC_COMMITTEE_INDEX` | `get_generalized_index(BeaconState, 'next_sync_committee')` | 56 | 57 | These values are the [generalized indices](https://github.com/ethereum/eth2.0-specs/blob/dev/ssz/merkle-proofs.md#generalized-merkle-tree-index) for the finalized checkpoint and the next sync committee in a `BeaconState`. A generalized index is a way of referring to a position of an object in a Merkle tree, so that the Merkle proof verification algorithm knows what path to check the hashes against. 58 | 59 | ## Configuration 60 | 61 | ### Misc 62 | 63 | | Name | Value | 64 | | - | - | 65 | | `MIN_SYNC_COMMITTEE_PARTICIPANTS` | `1` | 66 | 67 | Light clients will ignore aggregate signatures with exactly zero participants (duh). 68 | 69 | ## Containers 70 | 71 | ### `LightClientSnapshot` 72 | 73 | ```python 74 | class LightClientSnapshot(Container): 75 | # Beacon block header 76 | header: BeaconBlockHeader 77 | # Sync committees corresponding to the header 78 | current_sync_committee: SyncCommittee 79 | next_sync_committee: SyncCommittee 80 | ``` 81 | 82 | The `LightClientSnapshot` represents the light client's view of the most recent block header that the light client is convinced is securely part of the chain. The light client stores the header itself, so that the light client can then ask for Merkle branches to authenticate transactions and state against the header. The light client also stores the current and next sync committees, so that it can verify the sync committee signatures of newer proposed headers. 83 | 84 | ### `LightClientUpdate` 85 | 86 | ```python 87 | class LightClientUpdate(Container): 88 | # Update beacon block header 89 | header: BeaconBlockHeader 90 | # Next sync committee corresponding to the header 91 | next_sync_committee: SyncCommittee 92 | next_sync_committee_branch: Vector[Bytes32, floorlog2(NEXT_SYNC_COMMITTEE_INDEX)] 93 | # Finality proof for the update header 94 | finality_header: BeaconBlockHeader 95 | finality_branch: Vector[Bytes32, floorlog2(FINALIZED_ROOT_INDEX)] 96 | # Sync committee aggregate signature 97 | sync_committee_bits: Bitvector[SYNC_COMMITTEE_SIZE] 98 | sync_committee_signature: BLSSignature 99 | # Fork version for the aggregate signature 100 | fork_version: Version 101 | ``` 102 | 103 | A `LightClientUpdate` is an object passed over the wire (could be over a p2p network or through a client-server setup) which contains all of the information needed to convince a light client to accept a newer block header. The information included is: 104 | 105 | * **`header`**: the header that the light client will accept if the `LightClientUpdate` is valid. 106 | * **`next_sync_committee`**: if the `header` is in a newer sync committee period than the header in the light client's current `LightClientSnapshot`, then if the light client decides to accept this header it will need to know the new header's `next_sync_committee` so that it will be able to accept headers in even later periods. So the `LightClientUpdate` itself provides this sync committee. 107 | * **`next_sync_committee_branch`**: the Merkle branch that authenticates the `next_sync_committee` 108 | * **`finality_header`**: if nonempty, the header whose signature is being verified (if empty, the signature of the `header` itself is being verified) 109 | * **`finality_branch`**: the Merkle branch that authenticates that the `header` actually is the header corresponding to the _finalized root_ saved in the `finality_header` (if the `finality_header` is empty, the `finality_branch` is empty too) 110 | * **`sync_committee_bits`**: a bitfield showing who participated in the sync committee 111 | * **`sync_committee_signature`**: the signature 112 | * **`fork_version`**: needed to mix in to the data being signed (will be different across different hard forks and between mainnet and testnets) 113 | 114 | ### `LightClientStore` 115 | 116 | ```python 117 | @dataclass 118 | class LightClientStore(object): 119 | snapshot: LightClientSnapshot 120 | valid_updates: Set[LightClientUpdate] 121 | ``` 122 | 123 | The `LightClientStore` is the _full_ "state" of a light client, and includes: 124 | 125 | * A `snapshot`, reflecting a block header that a light client accepts as **finalized** (note: this is a different, and usually weaker, notion of finality than beacon chain finality). The sync committees in the state of this header are used to authenticate any new incoming headers. 126 | * A list of `valid_updates`, reflecting _speculative_ recent block headers (that is: recent block headers that have strong support, and will _probably_ continue to be part of the chain, but still have some small risk of being reverted) 127 | 128 | The `snapshot` can be updated in two ways: 129 | 130 | 1. If the light client sees a valid `LightClientUpdate` containing a `finality_header`, and with at least 2/3 of the sync committee participating, it accepts the `update.header` as the new snapshot header. Note that the light client uses the signature to verify `update.finality_header` (which would in practice often be one of the most recent blocks, and not yet finalized), and then uses the Merkle branch _from_ the `update.finality_header` _to_ the finalized checkpoint in its post-state to verify the `update.header`. If `update.finality_header` is a valid block, then `update.header` actually is finalized. 131 | 2. If the light client sees no valid updates via method (1) for a sufficiently long duration (specifically, the length of one sync committee period), it simply accepts the speculative header in `valid_updates` with the most signatures as finalized. 132 | 133 | (2) allows the light client to keep advancing even during periods of extended non-finality, though at the cost that during a long non-finality period the light client's safety is vulnerable to network latency of ~1 day (full clients' safety is only vulnerable to network latency on the order of weeks). 134 | 135 | ## Helper functions 136 | 137 | ### `get_subtree_index` 138 | 139 | ```python 140 | def get_subtree_index(generalized_index: GeneralizedIndex) -> uint64: 141 | return uint64(generalized_index % 2**(floorlog2(generalized_index))) 142 | ``` 143 | 144 | From a generalized index, return an integer whose bits, in least-to-greatest-place-value order, represent the Merkle path (0 = "left", 1 = "right", going from bottom to top) to get from a leaf to the root of a tree. Passed into the Merkle tree verification function used in other parts of the beacon chain spec. 145 | 146 | ## Light client state updates 147 | 148 | A light client maintains its state in a `store` object of type `LightClientStore` and receives `update` objects of type `LightClientUpdate`. Every `update` triggers `process_light_client_update(store, update, current_slot)` where `current_slot` is the current slot based on some local clock. 149 | 150 | #### `validate_light_client_update` 151 | 152 | ```python 153 | def validate_light_client_update(snapshot: LightClientSnapshot, 154 | update: LightClientUpdate, 155 | genesis_validators_root: Root) -> None: 156 | # Verify update slot is larger than snapshot slot 157 | assert update.header.slot > snapshot.header.slot 158 | 159 | # Verify update does not skip a sync committee period 160 | snapshot_period = compute_epoch_at_slot(snapshot.header.slot) // EPOCHS_PER_SYNC_COMMITTEE_PERIOD 161 | update_period = compute_epoch_at_slot(update.header.slot) // EPOCHS_PER_SYNC_COMMITTEE_PERIOD 162 | assert update_period in (snapshot_period, snapshot_period + 1) 163 | 164 | # Verify update header root is the finalized root of the finality header, if specified 165 | if update.finality_header == BeaconBlockHeader(): 166 | signed_header = update.header 167 | assert update.finality_branch == [Bytes32() for _ in range(floorlog2(FINALIZED_ROOT_INDEX))] 168 | else: 169 | signed_header = update.finality_header 170 | assert is_valid_merkle_branch( 171 | leaf=hash_tree_root(update.header), 172 | branch=update.finality_branch, 173 | depth=floorlog2(FINALIZED_ROOT_INDEX), 174 | index=get_subtree_index(FINALIZED_ROOT_INDEX), 175 | root=update.finality_header.state_root, 176 | ) 177 | 178 | # Verify update next sync committee if the update period incremented 179 | if update_period == snapshot_period: 180 | sync_committee = snapshot.current_sync_committee 181 | assert update.next_sync_committee_branch == [Bytes32() for _ in range(floorlog2(NEXT_SYNC_COMMITTEE_INDEX))] 182 | else: 183 | sync_committee = snapshot.next_sync_committee 184 | assert is_valid_merkle_branch( 185 | leaf=hash_tree_root(update.next_sync_committee), 186 | branch=update.next_sync_committee_branch, 187 | depth=floorlog2(NEXT_SYNC_COMMITTEE_INDEX), 188 | index=get_subtree_index(NEXT_SYNC_COMMITTEE_INDEX), 189 | root=update.header.state_root, 190 | ) 191 | 192 | # Verify sync committee has sufficient participants 193 | assert sum(update.sync_committee_bits) >= MIN_SYNC_COMMITTEE_PARTICIPANTS 194 | 195 | # Verify sync committee aggregate signature 196 | participant_pubkeys = [pubkey for (bit, pubkey) in zip(update.sync_committee_bits, sync_committee.pubkeys) if bit] 197 | domain = compute_domain(DOMAIN_SYNC_COMMITTEE, update.fork_version, genesis_validators_root) 198 | signing_root = compute_signing_root(signed_header, domain) 199 | assert bls.FastAggregateVerify(participant_pubkeys, signing_root, update.sync_committee_signature) 200 | ``` 201 | 202 | This function has 5 parts: 203 | 204 | 1. **Basic validation**: confirm that the `update.header` is newer than the snapshot header, and that it does not skip more than 1 sync committee period. 205 | 2. **Verify Merkle branch of header** - confirm that the `update.header` actually is the header that corresponds to the finalized root in the post-state of `update.finality_header` (and so if the `update.finality_header` is valid, the `update.header` is finalized). This check is only done if the `update.finality_header` is provided. 206 | 3. **Verify Merkle branch of sync committee** - if the `update.header` is in a newer sync period than the snapshot header, check that the `next_sync_committee` provided in the snapshot is correct by verifying the Merkle branch (if the `update.header` is not in a newer sync period, it's ok to leave the `next_sync_committee` empty). 207 | 4. **More basic validation**: confirm that the sync committee has more than zero participants. 208 | 5. **Verify the signature**: remember that if the `update.finality_header` is provided, the signature that we are expecting is a valid signature of the `update.finality_header`; otherwise, we are expecting a valid signature of the `update.header`. 209 | 210 | #### `apply_light_client_update` 211 | 212 | ```python 213 | def apply_light_client_update(snapshot: LightClientSnapshot, update: LightClientUpdate) -> None: 214 | snapshot_period = compute_epoch_at_slot(snapshot.header.slot) // EPOCHS_PER_SYNC_COMMITTEE_PERIOD 215 | update_period = compute_epoch_at_slot(update.header.slot) // EPOCHS_PER_SYNC_COMMITTEE_PERIOD 216 | if update_period == snapshot_period + 1: 217 | snapshot.current_sync_committee = snapshot.next_sync_committee 218 | snapshot.next_sync_committee = update.next_sync_committee 219 | snapshot.header = update.header 220 | ``` 221 | 222 | This function is called only when it is time to update that snapshot header: either (1) when a new header is provided that corresponds to a finalized checkpoint of another header, or (2) after the timeout. In addition to simply updating the header, we also update the sync committees in the snapshot. 223 | 224 | #### `process_light_client_update` 225 | 226 | ```python 227 | def process_light_client_update(store: LightClientStore, update: LightClientUpdate, current_slot: Slot, 228 | genesis_validators_root: Root) -> None: 229 | validate_light_client_update(store.snapshot, update, genesis_validators_root) 230 | store.valid_updates.add(update) 231 | 232 | update_timeout = SLOTS_PER_EPOCH * EPOCHS_PER_SYNC_COMMITTEE_PERIOD 233 | if ( 234 | sum(update.sync_committee_bits) * 3 >= len(update.sync_committee_bits) * 2 235 | and update.finality_header != BeaconBlockHeader() 236 | ): 237 | # Apply update if (1) 2/3 quorum is reached and (2) we have a finality proof. 238 | # Note that (2) means that the current light client design needs finality. 239 | # It may be changed to re-organizable light client design. See the on-going issue eth2.0-specs#2182. 240 | apply_light_client_update(store.snapshot, update) 241 | store.valid_updates = set() 242 | elif current_slot > store.snapshot.header.slot + update_timeout: 243 | # Forced best update when the update timeout has elapsed 244 | apply_light_client_update(store.snapshot, 245 | max(store.valid_updates, key=lambda update: sum(update.sync_committee_bits))) 246 | store.valid_updates = set() 247 | ``` 248 | 249 | The main function for processing a light client update. We first validate that it is correct; if it is correct, we at the very least save it as a speculative update. We then check if one of the two conditions for updating the snapshot is satisfied; if it is, then we update the snapshot. 250 | -------------------------------------------------------------------------------- /capella/beacon-chain.md: -------------------------------------------------------------------------------- 1 | # Capella Beacon chain changes 2 | 3 | This is an annotated version of the Capella beacon chain spec. 4 | 5 | ## Table of contents 6 | 7 | 8 | 9 | 10 | 11 | - [Introduction](#introduction) 12 | - [Custom types](#custom-types) 13 | - [Domain types](#domain-types) 14 | - [Preset](#preset) 15 | - [Max operations per block](#max-operations-per-block) 16 | - [Execution](#execution) 17 | - [Withdrawals processing](#withdrawals-processing) 18 | - [Containers](#containers) 19 | - [New containers](#new-containers) 20 | - [`Withdrawal`](#withdrawal) 21 | - [`BLSToExecutionChange`](#blstoexecutionchange) 22 | - [`SignedBLSToExecutionChange`](#signedblstoexecutionchange) 23 | - [`HistoricalSummary`](#historicalsummary) 24 | - [Extended Containers](#extended-containers) 25 | - [`ExecutionPayload`](#executionpayload) 26 | - [`ExecutionPayloadHeader`](#executionpayloadheader) 27 | - [`BeaconBlockBody`](#beaconblockbody) 28 | - [`BeaconState`](#beaconstate) 29 | - [Helpers](#helpers) 30 | - [Predicates](#predicates) 31 | - [`has_eth1_withdrawal_credential`](#has_eth1_withdrawal_credential) 32 | - [`is_fully_withdrawable_validator`](#is_fully_withdrawable_validator) 33 | - [`is_partially_withdrawable_validator`](#is_partially_withdrawable_validator) 34 | - [Beacon chain state transition function](#beacon-chain-state-transition-function) 35 | - [Epoch processing](#epoch-processing) 36 | - [Historical summaries updates](#historical-summaries-updates) 37 | - [Block processing](#block-processing) 38 | - [Aside: how withdrawal processing works](#aside-how-withdrawal-processing-works) 39 | - [New `get_expected_withdrawals`](#new-get_expected_withdrawals) 40 | - [New `process_withdrawals`](#new-process_withdrawals) 41 | - [Modified `process_execution_payload`](#modified-process_execution_payload) 42 | - [Modified `process_operations`](#modified-process_operations) 43 | - [New `process_bls_to_execution_change`](#new-process_bls_to_execution_change) 44 | - [Testing](#testing) 45 | 46 | 47 | 48 | 49 | ## Introduction 50 | 51 | Capella is a consensus-layer upgrade containing a number of features related to validator withdrawals. Including: 52 | 53 | * **Automatic withdrawals of `withdrawable` validators**. It has been possible for a long time for validators to enter the `withdrawable` state. However, the functionality to actually withdraw validator balances has not yet been available. Capella, together with the Shanghai upgrade for the execution spec, finally adds the functionality that allows for withdrawals to process. 54 | * **Partial withdrawals**: validators with `0x01` withdrawal credentials and balances above 32 ETH get their excess balance withdrawn to their address. A "sweeping" procedure cycles through the full set of validators and reaches each validator once every ~4-8 days. 55 | * **Conversion to ETH-address-based withdrawals**. There are currently two types of withdrawal credentials that validators can have, defined by the first byte in the validator's `withdrawal_credentials`: `0x00` (withdrawal controlled by BLS pubkey) and `0x01` (the validator's balance withdraws to a specific ETH address). Capella adds a feature that lets `0x00` validators convert to `0x01`. 56 | 57 | Capella also makes a small technical change to how historical block and state root summaries are stored, to make it slightly easier for a node to verify things about ancient history that it no longer stores (or did not download yet because it fast-synced). 58 | 59 | ## Custom types 60 | 61 | We define the following Python custom types for type hinting and readability: 62 | 63 | | Name | SSZ equivalent | Description | 64 | | - | - | - | 65 | | `WithdrawalIndex` | `uint64` | an index of a `Withdrawal` | 66 | 67 | This is just a counter that stores the (global) index of the withdrawal. The first withdrawal that ever happens will have index 0, the second will have index 1, etc. This is not strictly necessary for the spec to work, but it was added for convenience, so that each withdrawal could have a clear "transaction ID" that can be used to refer to it (just hashing withdrawal contents would not work, as there may be multiple withdrawals with the exact same contents). 68 | 69 | ### Domain types 70 | 71 | | Name | Value | 72 | | - | - | 73 | | `DOMAIN_BLS_TO_EXECUTION_CHANGE` | `DomainType('0x0A000000')` | 74 | 75 | ## Preset 76 | 77 | ### Max operations per block 78 | 79 | | Name | Value | 80 | | - | - | 81 | | `MAX_BLS_TO_EXECUTION_CHANGES` | `2**4` (= 16) | 82 | 83 | A maximum of 16 operations that convert a `0x00` (withdraw-by-BLS-key) account to a `0x01` (withdraw-to-ETH-address) account can be included in each block. 84 | 85 | ### Execution 86 | 87 | | Name | Value | Description | 88 | | - | - | - | 89 | | `MAX_WITHDRAWALS_PER_PAYLOAD` | `uint64(2**4)` (= 16) | Maximum amount of withdrawals allowed in each payload | 90 | 91 | ### Withdrawals processing 92 | 93 | | Name | Value | 94 | | - | - | 95 | | `MAX_VALIDATORS_PER_WITHDRAWALS_SWEEP` | `16384` (= 2**14 ) | 96 | 97 | The sweeping mechanism walks through a maximum of this many validators to look for potential withdrawals. 98 | 99 | ## Containers 100 | 101 | ### New containers 102 | 103 | #### `Withdrawal` 104 | 105 | ```python 106 | class Withdrawal(Container): 107 | index: WithdrawalIndex 108 | validator_index: ValidatorIndex 109 | address: ExecutionAddress 110 | amount: Gwei 111 | ``` 112 | 113 | This is the object that contains a withdrawal. When the consensus layer detects that a validator is ready for withdrawing (using `is_fully_withdrawable_validator`), the validator's balance is withdrawn automatically. 114 | 115 | Withdrawals are special because they move funds from the consensus layer to the execution layer, and so implementing them requires interaction between the two. There was a technical debate between two different ways to implement this interaction. One approach was to avoid including withdrawals in the body at all: the consensus portion of a block is processed first, it generates the list of withdrawals, and then that list is passed directly to the execution client and processed, without ever being serialized anywhere. The other approach is to include the list of withdrawals in the `ExecutionPayload` (the portion of the block that goes to the execution client). Consensus clients would check that the provided list is correctly generated, and execution clients would process those withdrawals. 116 | 117 | We ultimately decided on the second approach, because it improves modularity and separation of concerns, and particuarly it allows for execution validity and consensus validity to be verified at different times. This is very valuable for optimized node syncing procedures. 118 | 119 | #### `BLSToExecutionChange` 120 | 121 | ```python 122 | class BLSToExecutionChange(Container): 123 | validator_index: ValidatorIndex 124 | from_bls_pubkey: BLSPubkey 125 | to_execution_address: ExecutionAddress 126 | ``` 127 | 128 | This is the object that represents a validator's desire to upgrade from `0x00` (withdraw-by-BLS-key) to `0x01` (withdraw-to-ETH-address) withdrawal credentials. It needs to be signed (see `SignedBLSToExecutionChange` below) by the BLS key that is hashed in the original withdrawal credentials. The BLS key used to sign is the `from_bls_pubkey`, and we check that `hash(from_bls_pubkey)[1:] == validator.withdrawal_credentials[1:]` when processing a `BLSToExecutionChange` to verify that this is actually the key that was originally committed to. 129 | 130 | #### `SignedBLSToExecutionChange` 131 | 132 | ```python 133 | class SignedBLSToExecutionChange(Container): 134 | message: BLSToExecutionChange 135 | signature: BLSSignature 136 | ``` 137 | 138 | #### `HistoricalSummary` 139 | 140 | ```python 141 | class HistoricalSummary(Container): 142 | """ 143 | `HistoricalSummary` matches the components of the phase0 `HistoricalBatch` 144 | making the two hash_tree_root-compatible. 145 | """ 146 | block_summary_root: Root 147 | state_summary_root: Root 148 | ``` 149 | 150 | See [the phase0 spec](../phase0/beacon-chain.md#slots_per_historical_root) for how historical roots worked pre-Capella. To summarize, after each 8192-slot period, we would append a hash of the last 8192 block roots and 8192 state roots to an ever-growing structure in the state that stores these hashes. This gives us a data structure that we could use to Merkle-prove historical facts about old history or state. 151 | 152 | Here, we change it to store _two_ roots per period instead of one, storing the root of block roots and the root of state roots separately. This allows us to generate proofs about blocks without knowing anything about historical states, and to generate proofs about states without knowing anything about historical blocks. When the two were merged, a Merkle proof about one of the two structures would have to end with the Merkle root of the other structure because that's the final sister node in the path. Here, this requirement is removed. This is particularly valuable in the sync process, as it allows fast-synced nodes to download and verify batches of 8192 historical blocks without needing anyone to keep a separate data structure that tracks old states. 153 | 154 | We replace the `historical_roots` object, which stores one root per period, with a `historical_summaries` object, which stores a `HistoricalSummary` containing two roots (the root-of-block-roots and root-of-state-roots) per period. Because we do not actually have the old roots-of-block-roots and roots-of-state-roots in the spec, we unfortunately cannot replace the old historical roots; hence, for now, we keep them around and have two separate structures, the older one frozen, but in a future fork we may well re-merge them. 155 | 156 | ### Extended Containers 157 | 158 | #### `ExecutionPayload` 159 | 160 | ```python 161 | class ExecutionPayload(Container): 162 | # Execution block header fields 163 | parent_hash: Hash32 164 | fee_recipient: ExecutionAddress # 'beneficiary' in the yellow paper 165 | state_root: Bytes32 166 | receipts_root: Bytes32 167 | logs_bloom: ByteVector[BYTES_PER_LOGS_BLOOM] 168 | prev_randao: Bytes32 # 'difficulty' in the yellow paper 169 | block_number: uint64 # 'number' in the yellow paper 170 | gas_limit: uint64 171 | gas_used: uint64 172 | timestamp: uint64 173 | extra_data: ByteList[MAX_EXTRA_DATA_BYTES] 174 | base_fee_per_gas: uint256 175 | # Extra payload fields 176 | block_hash: Hash32 # Hash of execution block 177 | transactions: List[Transaction, MAX_TRANSACTIONS_PER_PAYLOAD] 178 | withdrawals: List[Withdrawal, MAX_WITHDRAWALS_PER_PAYLOAD] # [New in Capella] 179 | ``` 180 | 181 | #### `ExecutionPayloadHeader` 182 | 183 | ```python 184 | class ExecutionPayloadHeader(Container): 185 | # Execution block header fields 186 | parent_hash: Hash32 187 | fee_recipient: ExecutionAddress 188 | state_root: Bytes32 189 | receipts_root: Bytes32 190 | logs_bloom: ByteVector[BYTES_PER_LOGS_BLOOM] 191 | prev_randao: Bytes32 192 | block_number: uint64 193 | gas_limit: uint64 194 | gas_used: uint64 195 | timestamp: uint64 196 | extra_data: ByteList[MAX_EXTRA_DATA_BYTES] 197 | base_fee_per_gas: uint256 198 | # Extra payload fields 199 | block_hash: Hash32 # Hash of execution block 200 | transactions_root: Root 201 | withdrawals_root: Root # [New in Capella] 202 | ``` 203 | 204 | #### `BeaconBlockBody` 205 | 206 | ```python 207 | class BeaconBlockBody(Container): 208 | randao_reveal: BLSSignature 209 | eth1_data: Eth1Data # Eth1 data vote 210 | graffiti: Bytes32 # Arbitrary data 211 | # Operations 212 | proposer_slashings: List[ProposerSlashing, MAX_PROPOSER_SLASHINGS] 213 | attester_slashings: List[AttesterSlashing, MAX_ATTESTER_SLASHINGS] 214 | attestations: List[Attestation, MAX_ATTESTATIONS] 215 | deposits: List[Deposit, MAX_DEPOSITS] 216 | voluntary_exits: List[SignedVoluntaryExit, MAX_VOLUNTARY_EXITS] 217 | sync_aggregate: SyncAggregate 218 | # Execution 219 | execution_payload: ExecutionPayload 220 | # Capella operations 221 | bls_to_execution_changes: List[SignedBLSToExecutionChange, MAX_BLS_TO_EXECUTION_CHANGES] # [New in Capella] 222 | ``` 223 | 224 | #### `BeaconState` 225 | 226 | ```python 227 | class BeaconState(Container): 228 | # Versioning 229 | genesis_time: uint64 230 | genesis_validators_root: Root 231 | slot: Slot 232 | fork: Fork 233 | # History 234 | latest_block_header: BeaconBlockHeader 235 | block_roots: Vector[Root, SLOTS_PER_HISTORICAL_ROOT] 236 | state_roots: Vector[Root, SLOTS_PER_HISTORICAL_ROOT] 237 | historical_roots: List[Root, HISTORICAL_ROOTS_LIMIT] # Frozen in Capella, replaced by historical_summaries 238 | # Eth1 239 | eth1_data: Eth1Data 240 | eth1_data_votes: List[Eth1Data, EPOCHS_PER_ETH1_VOTING_PERIOD * SLOTS_PER_EPOCH] 241 | eth1_deposit_index: uint64 242 | # Registry 243 | validators: List[Validator, VALIDATOR_REGISTRY_LIMIT] 244 | balances: List[Gwei, VALIDATOR_REGISTRY_LIMIT] 245 | # Randomness 246 | randao_mixes: Vector[Bytes32, EPOCHS_PER_HISTORICAL_VECTOR] 247 | # Slashings 248 | slashings: Vector[Gwei, EPOCHS_PER_SLASHINGS_VECTOR] # Per-epoch sums of slashed effective balances 249 | # Participation 250 | previous_epoch_participation: List[ParticipationFlags, VALIDATOR_REGISTRY_LIMIT] 251 | current_epoch_participation: List[ParticipationFlags, VALIDATOR_REGISTRY_LIMIT] 252 | # Finality 253 | justification_bits: Bitvector[JUSTIFICATION_BITS_LENGTH] # Bit set for every recent justified epoch 254 | previous_justified_checkpoint: Checkpoint 255 | current_justified_checkpoint: Checkpoint 256 | finalized_checkpoint: Checkpoint 257 | # Inactivity 258 | inactivity_scores: List[uint64, VALIDATOR_REGISTRY_LIMIT] 259 | # Sync 260 | current_sync_committee: SyncCommittee 261 | next_sync_committee: SyncCommittee 262 | # Execution 263 | latest_execution_payload_header: ExecutionPayloadHeader # [Modified in Capella] 264 | # Withdrawals 265 | next_withdrawal_index: WithdrawalIndex # [New in Capella] 266 | next_withdrawal_validator_index: ValidatorIndex # [New in Capella] 267 | # Deep history valid from Capella onwards 268 | historical_summaries: List[HistoricalSummary, HISTORICAL_ROOTS_LIMIT] # [New in Capella] 269 | ``` 270 | 271 | ## Helpers 272 | 273 | ### Predicates 274 | 275 | #### `has_eth1_withdrawal_credential` 276 | 277 | ```python 278 | def has_eth1_withdrawal_credential(validator: Validator) -> bool: 279 | """ 280 | Check if ``validator`` has an 0x01 prefixed "eth1" withdrawal credential. 281 | """ 282 | return validator.withdrawal_credentials[:1] == ETH1_ADDRESS_WITHDRAWAL_PREFIX 283 | ``` 284 | 285 | #### `is_fully_withdrawable_validator` 286 | 287 | ```python 288 | def is_fully_withdrawable_validator(validator: Validator, balance: Gwei, epoch: Epoch) -> bool: 289 | """ 290 | Check if ``validator`` is fully withdrawable. 291 | """ 292 | return ( 293 | has_eth1_withdrawal_credential(validator) 294 | and validator.withdrawable_epoch <= epoch 295 | and balance > 0 296 | ) 297 | ``` 298 | 299 | If a validator with a `0x01` withdrawal credential has reached their withdrawability epoch and has more than 0 ETH, their total balance is eligible to be automatically withdrawn to their specified withdrawal address. 300 | 301 | #### `is_partially_withdrawable_validator` 302 | 303 | ```python 304 | def is_partially_withdrawable_validator(validator: Validator, balance: Gwei) -> bool: 305 | """ 306 | Check if ``validator`` is partially withdrawable. 307 | """ 308 | has_max_effective_balance = validator.effective_balance == MAX_EFFECTIVE_BALANCE 309 | has_excess_balance = balance > MAX_EFFECTIVE_BALANCE 310 | return has_eth1_withdrawal_credential(validator) and has_max_effective_balance and has_excess_balance 311 | ``` 312 | 313 | If a validator with a `0x01` withdrawal credential has more than 32 ETH, their "excess" ETH is eligible to be automatically withdrawn to their specified withdrawal address. 314 | 315 | ## Beacon chain state transition function 316 | 317 | ### Epoch processing 318 | 319 | *Note*: The function `process_historical_summaries_update` replaces `process_historical_roots_update` in Bellatrix. 320 | 321 | ```python 322 | def process_epoch(state: BeaconState) -> None: 323 | process_justification_and_finalization(state) 324 | process_inactivity_updates(state) 325 | process_rewards_and_penalties(state) 326 | process_registry_updates(state) 327 | process_slashings(state) 328 | process_eth1_data_reset(state) 329 | process_effective_balance_updates(state) 330 | process_slashings_reset(state) 331 | process_randao_mixes_reset(state) 332 | process_historical_summaries_update(state) # [Modified in Capella] 333 | process_participation_flag_updates(state) 334 | process_sync_committee_updates(state) 335 | ``` 336 | 337 | #### Historical summaries updates 338 | 339 | ```python 340 | def process_historical_summaries_update(state: BeaconState) -> None: 341 | # Set historical block root accumulator. 342 | next_epoch = Epoch(get_current_epoch(state) + 1) 343 | if next_epoch % (SLOTS_PER_HISTORICAL_ROOT // SLOTS_PER_EPOCH) == 0: 344 | historical_summary = HistoricalSummary( 345 | block_summary_root=hash_tree_root(state.block_roots), 346 | state_summary_root=hash_tree_root(state.state_roots), 347 | ) 348 | state.historical_summaries.append(historical_summary) 349 | ``` 350 | 351 | Extends the historical summaries list. Similar to the `historical_batch`-related logic in the [Final updates section](https://github.com/ethereum/annotated-spec/blob/master/phase0/beacon-chain.md#final-updates) of the pre-Capella spec. 352 | 353 | ### Block processing 354 | 355 | ```python 356 | def process_block(state: BeaconState, block: BeaconBlock) -> None: 357 | process_block_header(state, block) 358 | if is_execution_enabled(state, block.body): 359 | process_withdrawals(state, block.body.execution_payload) # [New in Capella] 360 | process_execution_payload(state, block.body.execution_payload, EXECUTION_ENGINE) # [Modified in Capella] 361 | process_randao(state, block.body) 362 | process_eth1_data(state, block.body) 363 | process_operations(state, block.body) # [Modified in Capella] 364 | process_sync_aggregate(state, block.body.sync_aggregate) 365 | ``` 366 | 367 | #### Aside: how withdrawal processing works 368 | 369 | _This describes the logic implemented in `get_expected_withdrawals` and `process_withdrawals` below, and the logic on the execution client side ([EIP-4895](https://eips.ethereum.org/EIPS/eip-4895)) to accept the withdrawals._ 370 | 371 | To detect accounts eligible for (full or partial) withdrawals, a "sweeping" process cycles through the entire list of validators. The current location of the sweep is stored in `state.next_withdrawal_validator_index`; during the processing of a block, the sweep walks through the validator list starting from that point, going back to the start of the list if it reaches the end. 372 | 373 | During one block, the sweep continues until it either (i) sweeps through `MAX_VALIDATORS_PER_WITHDRAWALS_SWEEP` validators, or, more commonly, (ii) discovers enough withdrawals to completely fill the `withdrawals` list and terminates early. The sweep checks for two types of withdrawable accounts: **full withdrawals**, where a validator has ETH and is `withdrawable`, and **partial withdrawals**, where a validator has more than 32 ETH, and so the excess can be withdrawn. In both cases, an additional condition is enforced that the validator must have a `0x01` (withdraw-to-ETH-address) withdrawal credential, which ensures that we know what ETH address the validator's (either total or excess) funds should be withdrawn to. 374 | 375 | If the sweep discovers a validator that is eligible for a withdrawal, it constructs a `Withdrawal` object of the appropriate type, and adds it to the list. Once the sweep is complete, the `process_withdrawals` function first checks that the _generated_ list of withdrawals equals the _declared_ list of withdrawals in the `ExecutionPayload`, and then processes the withdrawals, decreasing the balances of withdrawn validators by the amount in the `Withdrawal` (in practice, partial withdrawals decrease to 32 ETH, and full withdrawals decrease to 0 ETH). 376 | 377 | On the execution side, [EIP-4895](https://eips.ethereum.org/EIPS/eip-4895) introduces the rule that after all transactions are processed, the balances of ETH addresses mentioned in withdrawals get increased by the amounts specified by those withdrawals. 378 | 379 | #### New `get_expected_withdrawals` 380 | 381 | ```python 382 | def get_expected_withdrawals(state: BeaconState) -> Sequence[Withdrawal]: 383 | epoch = get_current_epoch(state) 384 | withdrawal_index = state.next_withdrawal_index 385 | validator_index = state.next_withdrawal_validator_index 386 | withdrawals: List[Withdrawal] = [] 387 | bound = min(len(state.validators), MAX_VALIDATORS_PER_WITHDRAWALS_SWEEP) 388 | for _ in range(bound): 389 | validator = state.validators[validator_index] 390 | balance = state.balances[validator_index] 391 | if is_fully_withdrawable_validator(validator, balance, epoch): 392 | withdrawals.append(Withdrawal( 393 | index=withdrawal_index, 394 | validator_index=validator_index, 395 | address=ExecutionAddress(validator.withdrawal_credentials[12:]), 396 | amount=balance, 397 | )) 398 | withdrawal_index += WithdrawalIndex(1) 399 | elif is_partially_withdrawable_validator(validator, balance): 400 | withdrawals.append(Withdrawal( 401 | index=withdrawal_index, 402 | validator_index=validator_index, 403 | address=ExecutionAddress(validator.withdrawal_credentials[12:]), 404 | amount=balance - MAX_EFFECTIVE_BALANCE, 405 | )) 406 | withdrawal_index += WithdrawalIndex(1) 407 | if len(withdrawals) == MAX_WITHDRAWALS_PER_PAYLOAD: 408 | break 409 | validator_index = ValidatorIndex((validator_index + 1) % len(state.validators)) 410 | return withdrawals 411 | ``` 412 | 413 | #### New `process_withdrawals` 414 | 415 | ```python 416 | def process_withdrawals(state: BeaconState, payload: ExecutionPayload) -> None: 417 | expected_withdrawals = get_expected_withdrawals(state) 418 | assert payload.withdrawals == expected_withdrawals 419 | 420 | for withdrawal in expected_withdrawals: 421 | decrease_balance(state, withdrawal.validator_index, withdrawal.amount) 422 | 423 | # Update the next withdrawal index if this block contained withdrawals 424 | if len(expected_withdrawals) != 0: 425 | latest_withdrawal = expected_withdrawals[-1] 426 | state.next_withdrawal_index = WithdrawalIndex(latest_withdrawal.index + 1) 427 | 428 | # Update the next validator index to start the next withdrawal sweep 429 | if len(expected_withdrawals) == MAX_WITHDRAWALS_PER_PAYLOAD: 430 | # Next sweep starts after the latest withdrawal's validator index 431 | next_validator_index = ValidatorIndex((expected_withdrawals[-1].validator_index + 1) % len(state.validators)) 432 | state.next_withdrawal_validator_index = next_validator_index 433 | else: 434 | # Advance sweep by the max length of the sweep if there was not a full set of withdrawals 435 | next_index = state.next_withdrawal_validator_index + MAX_VALIDATORS_PER_WITHDRAWALS_SWEEP 436 | next_validator_index = ValidatorIndex(next_index % len(state.validators)) 437 | state.next_withdrawal_validator_index = next_validator_index 438 | ``` 439 | 440 | The key things that we are doing here are (i) checking that the list of expected withdrawals generated by running the sweep (`get_expected_withdrawals`) is the same as the list in the payload, and (ii) actually processing those withdrawals. 441 | 442 | #### Modified `process_execution_payload` 443 | 444 | *Note*: The function `process_execution_payload` is modified to use the new `ExecutionPayloadHeader` type. 445 | 446 | ```python 447 | def process_execution_payload(state: BeaconState, payload: ExecutionPayload, execution_engine: ExecutionEngine) -> None: 448 | # [Modified in Capella] Removed `is_merge_transition_complete` check in Capella 449 | # Verify consistency of the parent hash with respect to the previous execution payload header 450 | assert payload.parent_hash == state.latest_execution_payload_header.block_hash 451 | # Verify prev_randao 452 | assert payload.prev_randao == get_randao_mix(state, get_current_epoch(state)) 453 | # Verify timestamp 454 | assert payload.timestamp == compute_timestamp_at_slot(state, state.slot) 455 | # Verify the execution payload is valid 456 | assert execution_engine.verify_and_notify_new_payload(NewPayloadRequest(execution_payload=payload)) 457 | # Cache execution payload header 458 | state.latest_execution_payload_header = ExecutionPayloadHeader( 459 | parent_hash=payload.parent_hash, 460 | fee_recipient=payload.fee_recipient, 461 | state_root=payload.state_root, 462 | receipts_root=payload.receipts_root, 463 | logs_bloom=payload.logs_bloom, 464 | prev_randao=payload.prev_randao, 465 | block_number=payload.block_number, 466 | gas_limit=payload.gas_limit, 467 | gas_used=payload.gas_used, 468 | timestamp=payload.timestamp, 469 | extra_data=payload.extra_data, 470 | base_fee_per_gas=payload.base_fee_per_gas, 471 | block_hash=payload.block_hash, 472 | transactions_root=hash_tree_root(payload.transactions), 473 | withdrawals_root=hash_tree_root(payload.withdrawals), # [New in Capella] 474 | ) 475 | ``` 476 | 477 | #### Modified `process_operations` 478 | 479 | *Note*: The function `process_operations` is modified to process `BLSToExecutionChange` operations included in the block. 480 | 481 | ```python 482 | def process_operations(state: BeaconState, body: BeaconBlockBody) -> None: 483 | # Verify that outstanding deposits are processed up to the maximum number of deposits 484 | assert len(body.deposits) == min(MAX_DEPOSITS, state.eth1_data.deposit_count - state.eth1_deposit_index) 485 | 486 | def for_ops(operations: Sequence[Any], fn: Callable[[BeaconState, Any], None]) -> None: 487 | for operation in operations: 488 | fn(state, operation) 489 | 490 | for_ops(body.proposer_slashings, process_proposer_slashing) 491 | for_ops(body.attester_slashings, process_attester_slashing) 492 | for_ops(body.attestations, process_attestation) 493 | for_ops(body.deposits, process_deposit) 494 | for_ops(body.voluntary_exits, process_voluntary_exit) 495 | for_ops(body.bls_to_execution_changes, process_bls_to_execution_change) # [New in Capella] 496 | ``` 497 | 498 | #### New `process_bls_to_execution_change` 499 | 500 | ```python 501 | def process_bls_to_execution_change(state: BeaconState, 502 | signed_address_change: SignedBLSToExecutionChange) -> None: 503 | address_change = signed_address_change.message 504 | 505 | assert address_change.validator_index < len(state.validators) 506 | 507 | validator = state.validators[address_change.validator_index] 508 | 509 | assert validator.withdrawal_credentials[:1] == BLS_WITHDRAWAL_PREFIX 510 | assert validator.withdrawal_credentials[1:] == hash(address_change.from_bls_pubkey)[1:] 511 | 512 | # Fork-agnostic domain since address changes are valid across forks 513 | domain = compute_domain(DOMAIN_BLS_TO_EXECUTION_CHANGE, genesis_validators_root=state.genesis_validators_root) 514 | signing_root = compute_signing_root(address_change, domain) 515 | assert bls.Verify(address_change.from_bls_pubkey, signing_root, signed_address_change.signature) 516 | 517 | validator.withdrawal_credentials = ( 518 | ETH1_ADDRESS_WITHDRAWAL_PREFIX 519 | + b'\x00' * 11 520 | + address_change.to_execution_address 521 | ) 522 | ``` 523 | 524 | Processes requests to change a validator's withdrawal credentials from being a BLS key to being an eth1 address. 525 | 526 | ## Testing 527 | 528 | *Note*: The function `initialize_beacon_state_from_eth1` is modified for pure Capella testing only. 529 | Modifications include: 530 | 1. Use `CAPELLA_FORK_VERSION` as the previous and current fork version. 531 | 2. Utilize the Capella `BeaconBlockBody` when constructing the initial `latest_block_header`. 532 | 533 | ```python 534 | def initialize_beacon_state_from_eth1(eth1_block_hash: Hash32, 535 | eth1_timestamp: uint64, 536 | deposits: Sequence[Deposit], 537 | execution_payload_header: ExecutionPayloadHeader=ExecutionPayloadHeader() 538 | ) -> BeaconState: 539 | fork = Fork( 540 | previous_version=CAPELLA_FORK_VERSION, # [Modified in Capella] for testing only 541 | current_version=CAPELLA_FORK_VERSION, # [Modified in Capella] 542 | epoch=GENESIS_EPOCH, 543 | ) 544 | state = BeaconState( 545 | genesis_time=eth1_timestamp + GENESIS_DELAY, 546 | fork=fork, 547 | eth1_data=Eth1Data(block_hash=eth1_block_hash, deposit_count=uint64(len(deposits))), 548 | latest_block_header=BeaconBlockHeader(body_root=hash_tree_root(BeaconBlockBody())), 549 | randao_mixes=[eth1_block_hash] * EPOCHS_PER_HISTORICAL_VECTOR, # Seed RANDAO with Eth1 entropy 550 | ) 551 | 552 | # Process deposits 553 | leaves = list(map(lambda deposit: deposit.data, deposits)) 554 | for index, deposit in enumerate(deposits): 555 | deposit_data_list = List[DepositData, 2**DEPOSIT_CONTRACT_TREE_DEPTH](*leaves[:index + 1]) 556 | state.eth1_data.deposit_root = hash_tree_root(deposit_data_list) 557 | process_deposit(state, deposit) 558 | 559 | # Process activations 560 | for index, validator in enumerate(state.validators): 561 | balance = state.balances[index] 562 | validator.effective_balance = min(balance - balance % EFFECTIVE_BALANCE_INCREMENT, MAX_EFFECTIVE_BALANCE) 563 | if validator.effective_balance == MAX_EFFECTIVE_BALANCE: 564 | validator.activation_eligibility_epoch = GENESIS_EPOCH 565 | validator.activation_epoch = GENESIS_EPOCH 566 | 567 | # Set genesis validators root for domain separation and chain versioning 568 | state.genesis_validators_root = hash_tree_root(state.validators) 569 | 570 | # Fill in sync committees 571 | # Note: A duplicate committee is assigned for the current and next committee at genesis 572 | state.current_sync_committee = get_next_sync_committee(state) 573 | state.next_sync_committee = get_next_sync_committee(state) 574 | 575 | # Initialize the execution payload header 576 | state.latest_execution_payload_header = execution_payload_header 577 | 578 | return state 579 | ``` 580 | -------------------------------------------------------------------------------- /deneb/beacon-chain.md: -------------------------------------------------------------------------------- 1 | # Deneb -- The Beacon Chain 2 | 3 | ## Table of contents 4 | 5 | 6 | 7 | 8 | 9 | - [Introduction](#introduction) 10 | - [Custom types](#custom-types) 11 | - [Constants](#constants) 12 | - [Blob](#blob) 13 | - [Preset](#preset) 14 | - [Execution](#execution) 15 | - [Configuration](#configuration) 16 | - [Validator cycle](#validator-cycle) 17 | - [Containers](#containers) 18 | - [Extended containers](#extended-containers) 19 | - [`BeaconBlockBody`](#beaconblockbody) 20 | - [`ExecutionPayload`](#executionpayload) 21 | - [`ExecutionPayloadHeader`](#executionpayloadheader) 22 | - [Helper functions](#helper-functions) 23 | - [Misc](#misc) 24 | - [`kzg_commitment_to_versioned_hash`](#kzg_commitment_to_versioned_hash) 25 | - [Beacon state accessors](#beacon-state-accessors) 26 | - [Modified `get_attestation_participation_flag_indices`](#modified-get_attestation_participation_flag_indices) 27 | - [New `get_validator_activation_churn_limit`](#new-get_validator_activation_churn_limit) 28 | - [Beacon chain state transition function](#beacon-chain-state-transition-function) 29 | - [Execution engine](#execution-engine) 30 | - [Request data](#request-data) 31 | - [Modified `NewPayloadRequest`](#modified-newpayloadrequest) 32 | - [Engine APIs](#engine-apis) 33 | - [`is_valid_block_hash`](#is_valid_block_hash) 34 | - [`is_valid_versioned_hashes`](#is_valid_versioned_hashes) 35 | - [Modified `notify_new_payload`](#modified-notify_new_payload) 36 | - [Modified `verify_and_notify_new_payload`](#modified-verify_and_notify_new_payload) 37 | - [Block processing](#block-processing) 38 | - [Modified `process_attestation`](#modified-process_attestation) 39 | - [Execution payload](#execution-payload) 40 | - [Modified `process_execution_payload`](#modified-process_execution_payload) 41 | - [Modified `process_voluntary_exit`](#modified-process_voluntary_exit) 42 | - [Epoch processing](#epoch-processing) 43 | - [Registry updates](#registry-updates) 44 | - [Testing](#testing) 45 | 46 | 47 | 48 | 49 | ## Introduction 50 | 51 | Deneb is a consensus-layer upgrade containing a number of features, most notably its flagship feature: **blobs**. Blobs are a new form of data storage that can be contained in transactions, like calldata. However, blobs have two key differences from calldata: 52 | 53 | 1. **Blobs are much larger, and much cheaper per byte, than calldata**. A blob has ~127 kB of data (more precisely: 4096 elements of a prime field modulo `52435875175126190479447740508185965837690552500527637822603658699938581184513` (~1.8 * 2254), and there is a separate floating basefee and per-block limit for blobs, allowing blobs to be very cheap even if regular execution gas is expensive. 54 | 2. **Transactions cannot access blob contents**. Instead, a transaction can only access a special type of hash (called a `VersionedHash`) of a blob. 55 | 56 | These two properties together make blobs an ideal place to put data for layer-2 protocols such as rollups. Additionally, there is a long-term goal to validate that all blobs referenced in a block have actually been published using a technology called [data availability sampling](https://www.paradigm.xyz/2022/08/das). In the Deneb fork, no such sampling happens; clients validate blobs by fully downloading them. However, there is a roadmap to progressively implement data availability sampling in the future, starting with a simplified version called [PeerDAS](https://ethresear.ch/t/peerdas-a-simpler-das-approach-using-battle-tested-p2p-components/16541). 57 | 58 | You can read more about blobs on the [EIP-4844 FAQ](https://www.eip4844.com/). 59 | 60 | Implementing blobs requires a consensus layer component and an execution layer component. On the execution layer, we need the new blob-carrying transaction type, and opcodes and precompiles to deal with blob commitments (called "versioned hashes"). On the consensus layer, we need logic to validate that the provided list of blob versioned hashes actually matches the blobs. One further complication is that the blobs themselves are not stored in a `BeaconBlock`; instead, they are stored in separate `BlobSidecar` objects. This separation is what will allow us to upgrade to data availability sampling in the future without requiring any changes to the execution _or_ consensus layer (except for modifying the target blob count parameter). 61 | 62 | In addition to blobs, there are also a few relatively minor changes introduced in Deneb: 63 | 64 | * [EIP-4788](https://eips.ethereum.org/EIPS/eip-4788): Beacon block root in the EVM 65 | * [EIP-4844](https://eips.ethereum.org/EIPS/eip-4844): Shard Blob Transactions scale data-availability of Ethereum in a simple, forwards-compatible manner 66 | * [EIP-7044](https://eips.ethereum.org/EIPS/eip-7044): Perpetually Valid Signed Voluntary Exits 67 | * [EIP-7045](https://eips.ethereum.org/EIPS/eip-7045): Increase Max Attestation Inclusion Slot 68 | * [EIP-7514](https://eips.ethereum.org/EIPS/eip-7514): Add Max Epoch Churn Limit 69 | 70 | ## Custom types 71 | 72 | | Name | SSZ equivalent | Description | 73 | | - | - | - | 74 | | `VersionedHash` | `Bytes32` | *[New in Deneb:EIP4844]* | 75 | | `BlobIndex` | `uint64` | *[New in Deneb:EIP4844]* | 76 | 77 | ## Constants 78 | 79 | ### Blob 80 | 81 | | Name | Value | 82 | | - | - | 83 | | `VERSIONED_HASH_VERSION_KZG` | `Bytes1('0x01')` | 84 | 85 | ## Preset 86 | 87 | ### Execution 88 | 89 | | Name | Value | Description | 90 | | - | - | - | 91 | | `MAX_BLOB_COMMITMENTS_PER_BLOCK` | `uint64(2**12)` (= 4096) | *[New in Deneb:EIP4844]* hardfork independent fixed theoretical limit same as `LIMIT_BLOBS_PER_TX` (see EIP 4844) | 92 | | `MAX_BLOBS_PER_BLOCK` | `uint64(6)` | *[New in Deneb:EIP4844]* maximum number of blobs in a single block limited by `MAX_BLOB_COMMITMENTS_PER_BLOCK` | 93 | 94 | *Note*: The blob transactions are packed into the execution payload by the EL/builder with their corresponding blobs being independently transmitted 95 | and are limited by `MAX_BLOB_GAS_PER_BLOCK // GAS_PER_BLOB`. However the CL limit is independently defined by `MAX_BLOBS_PER_BLOCK`. 96 | 97 | The `MAX_BLOB_COMMITMENTS_PER_BLOCK` corresponds to 4096 * 127 kB = 509 MB of blob space per slot; this is far beyond any short or medium-term projections for how much blob space Ethereum will be able to have, even with full data availability sampling. 98 | 99 | ## Configuration 100 | 101 | ### Validator cycle 102 | 103 | | Name | Value | 104 | | - | - | 105 | | `MAX_PER_EPOCH_ACTIVATION_CHURN_LIMIT` | `uint64(2**3)` (= 8) | 106 | 107 | We add a new limit to the maximum number of validator activations that can take place, to limit the rate at which the total quantity of staked ETH can grow. 108 | 109 | ## Containers 110 | 111 | ### Extended containers 112 | 113 | #### `BeaconBlockBody` 114 | 115 | Note: `BeaconBlock` and `SignedBeaconBlock` types are updated indirectly. 116 | 117 | ```python 118 | class BeaconBlockBody(Container): 119 | randao_reveal: BLSSignature 120 | eth1_data: Eth1Data # Eth1 data vote 121 | graffiti: Bytes32 # Arbitrary data 122 | # Operations 123 | proposer_slashings: List[ProposerSlashing, MAX_PROPOSER_SLASHINGS] 124 | attester_slashings: List[AttesterSlashing, MAX_ATTESTER_SLASHINGS] 125 | attestations: List[Attestation, MAX_ATTESTATIONS] 126 | deposits: List[Deposit, MAX_DEPOSITS] 127 | voluntary_exits: List[SignedVoluntaryExit, MAX_VOLUNTARY_EXITS] 128 | sync_aggregate: SyncAggregate 129 | # Execution 130 | execution_payload: ExecutionPayload # [Modified in Deneb:EIP4844] 131 | bls_to_execution_changes: List[SignedBLSToExecutionChange, MAX_BLS_TO_EXECUTION_CHANGES] 132 | blob_kzg_commitments: List[KZGCommitment, MAX_BLOB_COMMITMENTS_PER_BLOCK] # [New in Deneb:EIP4844] 133 | ``` 134 | 135 | #### `ExecutionPayload` 136 | 137 | ```python 138 | class ExecutionPayload(Container): 139 | # Execution block header fields 140 | parent_hash: Hash32 141 | fee_recipient: ExecutionAddress # 'beneficiary' in the yellow paper 142 | state_root: Bytes32 143 | receipts_root: Bytes32 144 | logs_bloom: ByteVector[BYTES_PER_LOGS_BLOOM] 145 | prev_randao: Bytes32 # 'difficulty' in the yellow paper 146 | block_number: uint64 # 'number' in the yellow paper 147 | gas_limit: uint64 148 | gas_used: uint64 149 | timestamp: uint64 150 | extra_data: ByteList[MAX_EXTRA_DATA_BYTES] 151 | base_fee_per_gas: uint256 152 | # Extra payload fields 153 | block_hash: Hash32 # Hash of execution block 154 | transactions: List[Transaction, MAX_TRANSACTIONS_PER_PAYLOAD] 155 | withdrawals: List[Withdrawal, MAX_WITHDRAWALS_PER_PAYLOAD] 156 | blob_gas_used: uint64 # [New in Deneb:EIP4844] 157 | excess_blob_gas: uint64 # [New in Deneb:EIP4844] 158 | ``` 159 | 160 | #### `ExecutionPayloadHeader` 161 | 162 | ```python 163 | class ExecutionPayloadHeader(Container): 164 | # Execution block header fields 165 | parent_hash: Hash32 166 | fee_recipient: ExecutionAddress 167 | state_root: Bytes32 168 | receipts_root: Bytes32 169 | logs_bloom: ByteVector[BYTES_PER_LOGS_BLOOM] 170 | prev_randao: Bytes32 171 | block_number: uint64 172 | gas_limit: uint64 173 | gas_used: uint64 174 | timestamp: uint64 175 | extra_data: ByteList[MAX_EXTRA_DATA_BYTES] 176 | base_fee_per_gas: uint256 177 | # Extra payload fields 178 | block_hash: Hash32 # Hash of execution block 179 | transactions_root: Root 180 | withdrawals_root: Root 181 | blob_gas_used: uint64 # [New in Deneb:EIP4844] 182 | excess_blob_gas: uint64 # [New in Deneb:EIP4844] 183 | ``` 184 | 185 | ## Helper functions 186 | 187 | ### Misc 188 | 189 | #### `kzg_commitment_to_versioned_hash` 190 | 191 | ```python 192 | def kzg_commitment_to_versioned_hash(kzg_commitment: KZGCommitment) -> VersionedHash: 193 | return VERSIONED_HASH_VERSION_KZG + hash(kzg_commitment)[1:] 194 | ``` 195 | 196 | To represent blobs with a 32-byte hash, we follow a two-step pipeline: `Blob -> KZGCommitment -> VersionedHash`. The `Blob -> KZGCommitment` step is done with the `blob_to_kzg_commitment` function in [polynomial-commitments.md](./polynomial-commitments.md); the `KZGCommitment -> VersionedHash` step is done here. The purpose of the separation is that a `KZGCommitment` is more amenable to mathematical operations that make data availability sampling possible, along with a number of other optimizations, whereas a `VersionedHash` is a more future-proof identifier that can upgrade to other commitment types in the future. 197 | 198 | ### Beacon state accessors 199 | 200 | #### Modified `get_attestation_participation_flag_indices` 201 | 202 | *Note:* The function `get_attestation_participation_flag_indices` is modified to set the `TIMELY_TARGET_FLAG` for any correct target attestation, regardless of `inclusion_delay` as a baseline reward for any speed of inclusion of an attestation that contributes to justification of the contained chain for EIP-7045. 203 | 204 | ```python 205 | def get_attestation_participation_flag_indices(state: BeaconState, 206 | data: AttestationData, 207 | inclusion_delay: uint64) -> Sequence[int]: 208 | """ 209 | Return the flag indices that are satisfied by an attestation. 210 | """ 211 | if data.target.epoch == get_current_epoch(state): 212 | justified_checkpoint = state.current_justified_checkpoint 213 | else: 214 | justified_checkpoint = state.previous_justified_checkpoint 215 | 216 | # Matching roots 217 | is_matching_source = data.source == justified_checkpoint 218 | is_matching_target = is_matching_source and data.target.root == get_block_root(state, data.target.epoch) 219 | is_matching_head = is_matching_target and data.beacon_block_root == get_block_root_at_slot(state, data.slot) 220 | assert is_matching_source 221 | 222 | participation_flag_indices = [] 223 | if is_matching_source and inclusion_delay <= integer_squareroot(SLOTS_PER_EPOCH): 224 | participation_flag_indices.append(TIMELY_SOURCE_FLAG_INDEX) 225 | if is_matching_target: # [Modified in Deneb:EIP7045] 226 | participation_flag_indices.append(TIMELY_TARGET_FLAG_INDEX) 227 | if is_matching_head and inclusion_delay == MIN_ATTESTATION_INCLUSION_DELAY: 228 | participation_flag_indices.append(TIMELY_HEAD_FLAG_INDEX) 229 | 230 | return participation_flag_indices 231 | ``` 232 | 233 | #### New `get_validator_activation_churn_limit` 234 | 235 | ```python 236 | def get_validator_activation_churn_limit(state: BeaconState) -> uint64: 237 | """ 238 | Return the validator activation churn limit for the current epoch. 239 | """ 240 | return min(MAX_PER_EPOCH_ACTIVATION_CHURN_LIMIT, get_validator_churn_limit(state)) 241 | ``` 242 | 243 | ## Beacon chain state transition function 244 | 245 | ### Execution engine 246 | 247 | #### Request data 248 | 249 | ##### Modified `NewPayloadRequest` 250 | 251 | ```python 252 | @dataclass 253 | class NewPayloadRequest(object): 254 | execution_payload: ExecutionPayload 255 | versioned_hashes: Sequence[VersionedHash] 256 | parent_beacon_block_root: Root 257 | ``` 258 | 259 | #### Engine APIs 260 | 261 | ##### `is_valid_block_hash` 262 | 263 | *Note*: The function `is_valid_block_hash` is modified to include the additional `parent_beacon_block_root` parameter for EIP-4788. 264 | 265 | ```python 266 | def is_valid_block_hash(self: ExecutionEngine, 267 | execution_payload: ExecutionPayload, 268 | parent_beacon_block_root: Root) -> bool: 269 | """ 270 | Return ``True`` if and only if ``execution_payload.block_hash`` is computed correctly. 271 | """ 272 | ... 273 | ``` 274 | 275 | EIP-4788 allows the EVM to access beacon block roots, which makes it easier to implement smart contract systems that interface with beacon chain operations (notably: decentralized staking pools). 276 | 277 | ##### `is_valid_versioned_hashes` 278 | 279 | ```python 280 | def is_valid_versioned_hashes(self: ExecutionEngine, new_payload_request: NewPayloadRequest) -> bool: 281 | """ 282 | Return ``True`` if and only if the version hashes computed by the blob transactions of 283 | ``new_payload_request.execution_payload`` matches ``new_payload_request.version_hashes``. 284 | """ 285 | ... 286 | ``` 287 | 288 | ##### Modified `notify_new_payload` 289 | 290 | *Note*: The function `notify_new_payload` is modified to include the additional `parent_beacon_block_root` parameter for EIP-4788. 291 | 292 | ```python 293 | def notify_new_payload(self: ExecutionEngine, 294 | execution_payload: ExecutionPayload, 295 | parent_beacon_block_root: Root) -> bool: 296 | """ 297 | Return ``True`` if and only if ``execution_payload`` is valid with respect to ``self.execution_state``. 298 | """ 299 | ... 300 | ``` 301 | 302 | ##### Modified `verify_and_notify_new_payload` 303 | 304 | ```python 305 | def verify_and_notify_new_payload(self: ExecutionEngine, 306 | new_payload_request: NewPayloadRequest) -> bool: 307 | """ 308 | Return ``True`` if and only if ``new_payload_request`` is valid with respect to ``self.execution_state``. 309 | """ 310 | execution_payload = new_payload_request.execution_payload 311 | parent_beacon_block_root = new_payload_request.parent_beacon_block_root 312 | 313 | # [Modified in Deneb:EIP4788] 314 | if not self.is_valid_block_hash(execution_payload, parent_beacon_block_root): 315 | return False 316 | 317 | # [New in Deneb:EIP4844] 318 | if not self.is_valid_versioned_hashes(new_payload_request): 319 | return False 320 | 321 | # [Modified in Deneb:EIP4788] 322 | if not self.notify_new_payload(execution_payload, parent_beacon_block_root): 323 | return False 324 | 325 | return True 326 | ``` 327 | 328 | ### Block processing 329 | 330 | #### Modified `process_attestation` 331 | 332 | *Note*: The function `process_attestation` is modified to expand valid slots for inclusion to those in both `target.epoch` epoch and `target.epoch + 1` epoch for EIP-7045. Additionally, it utilizes an updated version of `get_attestation_participation_flag_indices` to ensure rewards are available for the extended attestation inclusion range for EIP-7045. 333 | 334 | ```python 335 | def process_attestation(state: BeaconState, attestation: Attestation) -> None: 336 | data = attestation.data 337 | assert data.target.epoch in (get_previous_epoch(state), get_current_epoch(state)) 338 | assert data.target.epoch == compute_epoch_at_slot(data.slot) 339 | assert data.slot + MIN_ATTESTATION_INCLUSION_DELAY <= state.slot # [Modified in Deneb:EIP7045] 340 | assert data.index < get_committee_count_per_slot(state, data.target.epoch) 341 | 342 | committee = get_beacon_committee(state, data.slot, data.index) 343 | assert len(attestation.aggregation_bits) == len(committee) 344 | 345 | # Participation flag indices 346 | participation_flag_indices = get_attestation_participation_flag_indices(state, data, state.slot - data.slot) 347 | 348 | # Verify signature 349 | assert is_valid_indexed_attestation(state, get_indexed_attestation(state, attestation)) 350 | 351 | # Update epoch participation flags 352 | if data.target.epoch == get_current_epoch(state): 353 | epoch_participation = state.current_epoch_participation 354 | else: 355 | epoch_participation = state.previous_epoch_participation 356 | 357 | proposer_reward_numerator = 0 358 | for index in get_attesting_indices(state, attestation): 359 | for flag_index, weight in enumerate(PARTICIPATION_FLAG_WEIGHTS): 360 | if flag_index in participation_flag_indices and not has_flag(epoch_participation[index], flag_index): 361 | epoch_participation[index] = add_flag(epoch_participation[index], flag_index) 362 | proposer_reward_numerator += get_base_reward(state, index) * weight 363 | 364 | # Reward proposer 365 | proposer_reward_denominator = (WEIGHT_DENOMINATOR - PROPOSER_WEIGHT) * WEIGHT_DENOMINATOR // PROPOSER_WEIGHT 366 | proposer_reward = Gwei(proposer_reward_numerator // proposer_reward_denominator) 367 | increase_balance(state, get_beacon_proposer_index(state), proposer_reward) 368 | ``` 369 | 370 | #### Execution payload 371 | 372 | ##### Modified `process_execution_payload` 373 | 374 | *Note*: The function `process_execution_payload` is modified to pass `versioned_hashes` into `execution_engine.verify_and_notify_new_payload` and to assign the new fields in `ExecutionPayloadHeader` for EIP-4844. It is also modified to pass in the parent beacon block root to support EIP-4788. 375 | 376 | ```python 377 | def process_execution_payload(state: BeaconState, body: BeaconBlockBody, execution_engine: ExecutionEngine) -> None: 378 | payload = body.execution_payload 379 | 380 | # Verify consistency of the parent hash with respect to the previous execution payload header 381 | assert payload.parent_hash == state.latest_execution_payload_header.block_hash 382 | # Verify prev_randao 383 | assert payload.prev_randao == get_randao_mix(state, get_current_epoch(state)) 384 | # Verify timestamp 385 | assert payload.timestamp == compute_timestamp_at_slot(state, state.slot) 386 | 387 | # [New in Deneb:EIP4844] Verify commitments are under limit 388 | assert len(body.blob_kzg_commitments) <= MAX_BLOBS_PER_BLOCK 389 | 390 | # Verify the execution payload is valid 391 | # [Modified in Deneb:EIP4844] Pass `versioned_hashes` to Execution Engine 392 | # [Modified in Deneb:EIP4788] Pass `parent_beacon_block_root` to Execution Engine 393 | versioned_hashes = [kzg_commitment_to_versioned_hash(commitment) for commitment in body.blob_kzg_commitments] 394 | assert execution_engine.verify_and_notify_new_payload( 395 | NewPayloadRequest( 396 | execution_payload=payload, 397 | versioned_hashes=versioned_hashes, 398 | parent_beacon_block_root=state.latest_block_header.parent_root, 399 | ) 400 | ) 401 | 402 | # Cache execution payload header 403 | state.latest_execution_payload_header = ExecutionPayloadHeader( 404 | parent_hash=payload.parent_hash, 405 | fee_recipient=payload.fee_recipient, 406 | state_root=payload.state_root, 407 | receipts_root=payload.receipts_root, 408 | logs_bloom=payload.logs_bloom, 409 | prev_randao=payload.prev_randao, 410 | block_number=payload.block_number, 411 | gas_limit=payload.gas_limit, 412 | gas_used=payload.gas_used, 413 | timestamp=payload.timestamp, 414 | extra_data=payload.extra_data, 415 | base_fee_per_gas=payload.base_fee_per_gas, 416 | block_hash=payload.block_hash, 417 | transactions_root=hash_tree_root(payload.transactions), 418 | withdrawals_root=hash_tree_root(payload.withdrawals), 419 | blob_gas_used=payload.blob_gas_used, # [New in Deneb:EIP4844] 420 | excess_blob_gas=payload.excess_blob_gas, # [New in Deneb:EIP4844] 421 | ) 422 | ``` 423 | 424 | #### Modified `process_voluntary_exit` 425 | 426 | *Note*: The function `process_voluntary_exit` is modified to use the a fixed fork version -- `CAPELLA_FORK_VERSION` -- for EIP-7044. 427 | 428 | ```python 429 | def process_voluntary_exit(state: BeaconState, signed_voluntary_exit: SignedVoluntaryExit) -> None: 430 | voluntary_exit = signed_voluntary_exit.message 431 | validator = state.validators[voluntary_exit.validator_index] 432 | # Verify the validator is active 433 | assert is_active_validator(validator, get_current_epoch(state)) 434 | # Verify exit has not been initiated 435 | assert validator.exit_epoch == FAR_FUTURE_EPOCH 436 | # Exits must specify an epoch when they become valid; they are not valid before then 437 | assert get_current_epoch(state) >= voluntary_exit.epoch 438 | # Verify the validator has been active long enough 439 | assert get_current_epoch(state) >= validator.activation_epoch + SHARD_COMMITTEE_PERIOD 440 | # Verify signature 441 | # [Modified in Deneb:EIP7044] 442 | domain = compute_domain(DOMAIN_VOLUNTARY_EXIT, CAPELLA_FORK_VERSION, state.genesis_validators_root) 443 | signing_root = compute_signing_root(voluntary_exit, domain) 444 | assert bls.Verify(validator.pubkey, signing_root, signed_voluntary_exit.signature) 445 | # Initiate exit 446 | initiate_validator_exit(state, voluntary_exit.validator_index) 447 | ``` 448 | 449 | This allows signed `VoluntaryExit` operations to remain valid forever, even if the protocol upgrades. 450 | 451 | ### Epoch processing 452 | 453 | #### Registry updates 454 | 455 | *Note*: The function `process_registry_updates` is modified to utilize `get_validator_activation_churn_limit()` to rate limit the activation queue for EIP-7514. 456 | 457 | ```python 458 | def process_registry_updates(state: BeaconState) -> None: 459 | # Process activation eligibility and ejections 460 | for index, validator in enumerate(state.validators): 461 | if is_eligible_for_activation_queue(validator): 462 | validator.activation_eligibility_epoch = get_current_epoch(state) + 1 463 | 464 | if ( 465 | is_active_validator(validator, get_current_epoch(state)) 466 | and validator.effective_balance <= EJECTION_BALANCE 467 | ): 468 | initiate_validator_exit(state, ValidatorIndex(index)) 469 | 470 | # Queue validators eligible for activation and not yet dequeued for activation 471 | activation_queue = sorted([ 472 | index for index, validator in enumerate(state.validators) 473 | if is_eligible_for_activation(state, validator) 474 | # Order by the sequence of activation_eligibility_epoch setting and then index 475 | ], key=lambda index: (state.validators[index].activation_eligibility_epoch, index)) 476 | # Dequeued validators for activation up to activation churn limit 477 | # [Modified in Deneb:EIP7514] 478 | for index in activation_queue[:get_validator_activation_churn_limit(state)]: 479 | validator = state.validators[index] 480 | validator.activation_epoch = compute_activation_exit_epoch(get_current_epoch(state)) 481 | ``` 482 | 483 | ## Testing 484 | 485 | *Note*: The function `initialize_beacon_state_from_eth1` is modified for pure Deneb testing only. 486 | 487 | The `BeaconState` initialization is unchanged, except for the use of the updated `deneb.BeaconBlockBody` type 488 | when initializing the first body-root. 489 | 490 | ```python 491 | def initialize_beacon_state_from_eth1(eth1_block_hash: Hash32, 492 | eth1_timestamp: uint64, 493 | deposits: Sequence[Deposit], 494 | execution_payload_header: ExecutionPayloadHeader=ExecutionPayloadHeader() 495 | ) -> BeaconState: 496 | fork = Fork( 497 | previous_version=DENEB_FORK_VERSION, # [Modified in Deneb] for testing only 498 | current_version=DENEB_FORK_VERSION, # [Modified in Deneb] 499 | epoch=GENESIS_EPOCH, 500 | ) 501 | state = BeaconState( 502 | genesis_time=eth1_timestamp + GENESIS_DELAY, 503 | fork=fork, 504 | eth1_data=Eth1Data(block_hash=eth1_block_hash, deposit_count=uint64(len(deposits))), 505 | latest_block_header=BeaconBlockHeader(body_root=hash_tree_root(BeaconBlockBody())), 506 | randao_mixes=[eth1_block_hash] * EPOCHS_PER_HISTORICAL_VECTOR, # Seed RANDAO with Eth1 entropy 507 | ) 508 | 509 | # Process deposits 510 | leaves = list(map(lambda deposit: deposit.data, deposits)) 511 | for index, deposit in enumerate(deposits): 512 | deposit_data_list = List[DepositData, 2**DEPOSIT_CONTRACT_TREE_DEPTH](*leaves[:index + 1]) 513 | state.eth1_data.deposit_root = hash_tree_root(deposit_data_list) 514 | process_deposit(state, deposit) 515 | 516 | # Process activations 517 | for index, validator in enumerate(state.validators): 518 | balance = state.balances[index] 519 | validator.effective_balance = min(balance - balance % EFFECTIVE_BALANCE_INCREMENT, MAX_EFFECTIVE_BALANCE) 520 | if validator.effective_balance == MAX_EFFECTIVE_BALANCE: 521 | validator.activation_eligibility_epoch = GENESIS_EPOCH 522 | validator.activation_epoch = GENESIS_EPOCH 523 | 524 | # Set genesis validators root for domain separation and chain versioning 525 | state.genesis_validators_root = hash_tree_root(state.validators) 526 | 527 | # Fill in sync committees 528 | # Note: A duplicate committee is assigned for the current and next committee at genesis 529 | state.current_sync_committee = get_next_sync_committee(state) 530 | state.next_sync_committee = get_next_sync_committee(state) 531 | 532 | # Initialize the execution payload header 533 | # If empty, will initialize a chain that has not yet gone through the Merge transition 534 | state.latest_execution_payload_header = execution_payload_header 535 | 536 | return state 537 | ``` 538 | -------------------------------------------------------------------------------- /deneb/fork-choice.md: -------------------------------------------------------------------------------- 1 | # Deneb -- Fork Choice 2 | 3 | ## Table of contents 4 | 5 | 6 | 7 | 8 | - [Introduction](#introduction) 9 | - [Containers](#containers) 10 | - [Helpers](#helpers) 11 | - [Extended `PayloadAttributes`](#extended-payloadattributes) 12 | - [`is_data_available`](#is_data_available) 13 | - [Updated fork-choice handlers](#updated-fork-choice-handlers) 14 | - [`on_block`](#on_block) 15 | 16 | 17 | 18 | 19 | ## Introduction 20 | 21 | This is the modification of the fork choice accompanying the Deneb upgrade. 22 | 23 | The most important new change is that for a `BeaconBlock` to be accepted, the client must have verified the availability of the underlying blob data. This is specified in an abstract way, to accommodate the fact that this is handled by direct downloading today, but will be handled with data availability sampling in the future. 24 | 25 | ## Containers 26 | 27 | ## Helpers 28 | 29 | ### Extended `PayloadAttributes` 30 | 31 | `PayloadAttributes` is extended with the parent beacon block root for EIP-4788. 32 | 33 | ```python 34 | @dataclass 35 | class PayloadAttributes(object): 36 | timestamp: uint64 37 | prev_randao: Bytes32 38 | suggested_fee_recipient: ExecutionAddress 39 | withdrawals: Sequence[Withdrawal] 40 | parent_beacon_block_root: Root # [New in Deneb:EIP4788] 41 | ``` 42 | 43 | ### `is_data_available` 44 | 45 | *[New in Deneb:EIP4844]* 46 | 47 | The implementation of `is_data_available` will become more sophisticated during later scaling upgrades. 48 | Initially, verification requires every verifying actor to retrieve all matching `Blob`s and `KZGProof`s, and validate them with `verify_blob_kzg_proof_batch`. 49 | 50 | The block MUST NOT be considered valid until all valid `Blob`s have been downloaded. Blocks that have been previously validated as available SHOULD be considered available even if the associated `Blob`s have subsequently been pruned. 51 | 52 | *Note*: Extraneous or invalid Blobs (in addition to KZG expected/referenced valid blobs) received on the p2p network MUST NOT invalidate a block that is otherwise valid and available. 53 | 54 | ```python 55 | def is_data_available(beacon_block_root: Root, blob_kzg_commitments: Sequence[KZGCommitment]) -> bool: 56 | # `retrieve_blobs_and_proofs` is implementation and context dependent 57 | # It returns all the blobs for the given block root, and raises an exception if not available 58 | # Note: the p2p network does not guarantee sidecar retrieval outside of 59 | # `MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS` 60 | blobs, proofs = retrieve_blobs_and_proofs(beacon_block_root) 61 | 62 | return verify_blob_kzg_proof_batch(blobs, blob_kzg_commitments, proofs) 63 | ``` 64 | 65 | ## Updated fork-choice handlers 66 | 67 | ### `on_block` 68 | 69 | *Note*: The only modification is the addition of the blob data availability check. 70 | 71 | ```python 72 | def on_block(store: Store, signed_block: SignedBeaconBlock) -> None: 73 | """ 74 | Run ``on_block`` upon receiving a new block. 75 | """ 76 | block = signed_block.message 77 | # Parent block must be known 78 | assert block.parent_root in store.block_states 79 | # Make a copy of the state to avoid mutability issues 80 | pre_state = copy(store.block_states[block.parent_root]) 81 | # Blocks cannot be in the future. If they are, their consideration must be delayed until they are in the past. 82 | assert get_current_slot(store) >= block.slot 83 | 84 | # Check that block is later than the finalized epoch slot (optimization to reduce calls to get_ancestor) 85 | finalized_slot = compute_start_slot_at_epoch(store.finalized_checkpoint.epoch) 86 | assert block.slot > finalized_slot 87 | # Check block is a descendant of the finalized block at the checkpoint finalized slot 88 | finalized_checkpoint_block = get_checkpoint_block( 89 | store, 90 | block.parent_root, 91 | store.finalized_checkpoint.epoch, 92 | ) 93 | assert store.finalized_checkpoint.root == finalized_checkpoint_block 94 | 95 | # [New in Deneb:EIP4844] 96 | # Check if blob data is available 97 | # If not, this block MAY be queued and subsequently considered when blob data becomes available 98 | # *Note*: Extraneous or invalid Blobs (in addition to the expected/referenced valid blobs) 99 | # received on the p2p network MUST NOT invalidate a block that is otherwise valid and available 100 | assert is_data_available(hash_tree_root(block), block.body.blob_kzg_commitments) 101 | 102 | # Check the block is valid and compute the post-state 103 | state = pre_state.copy() 104 | block_root = hash_tree_root(block) 105 | state_transition(state, signed_block, True) 106 | 107 | # Add new block to the store 108 | store.blocks[block_root] = block 109 | # Add new state for this block to the store 110 | store.block_states[block_root] = state 111 | 112 | # Add block timeliness to the store 113 | time_into_slot = (store.time - store.genesis_time) % SECONDS_PER_SLOT 114 | is_before_attesting_interval = time_into_slot < SECONDS_PER_SLOT // INTERVALS_PER_SLOT 115 | is_timely = get_current_slot(store) == block.slot and is_before_attesting_interval 116 | store.block_timeliness[hash_tree_root(block)] = is_timely 117 | 118 | # Add proposer score boost if the block is timely and not conflicting with an existing block 119 | is_first_block = store.proposer_boost_root == Root() 120 | if is_timely and is_first_block: 121 | store.proposer_boost_root = hash_tree_root(block) 122 | 123 | # Update checkpoints in store if necessary 124 | update_checkpoints(store, state.current_justified_checkpoint, state.finalized_checkpoint) 125 | 126 | # Eagerly compute unrealized justification and finality. 127 | compute_pulled_up_tip(store, block_root) 128 | ``` 129 | -------------------------------------------------------------------------------- /deneb/polynomial-commitments.md: -------------------------------------------------------------------------------- 1 | # Deneb -- Polynomial Commitments 2 | 3 | ## Table of contents 4 | 5 | 6 | 7 | 8 | 9 | - [Introduction](#introduction) 10 | - [Custom types](#custom-types) 11 | - [Constants](#constants) 12 | - [Preset](#preset) 13 | - [Blob](#blob) 14 | - [Trusted setup](#trusted-setup) 15 | - [Helper functions](#helper-functions) 16 | - [Bit-reversal permutation](#bit-reversal-permutation) 17 | - [`is_power_of_two`](#is_power_of_two) 18 | - [`reverse_bits`](#reverse_bits) 19 | - [`bit_reversal_permutation`](#bit_reversal_permutation) 20 | - [BLS12-381 helpers](#bls12-381-helpers) 21 | - [`multi_exp`](#multi_exp) 22 | - [`hash_to_bls_field`](#hash_to_bls_field) 23 | - [`bytes_to_bls_field`](#bytes_to_bls_field) 24 | - [`bls_field_to_bytes`](#bls_field_to_bytes) 25 | - [`validate_kzg_g1`](#validate_kzg_g1) 26 | - [`bytes_to_kzg_commitment`](#bytes_to_kzg_commitment) 27 | - [`bytes_to_kzg_proof`](#bytes_to_kzg_proof) 28 | - [`blob_to_polynomial`](#blob_to_polynomial) 29 | - [`compute_challenge`](#compute_challenge) 30 | - [`bls_modular_inverse`](#bls_modular_inverse) 31 | - [`div`](#div) 32 | - [`g1_lincomb`](#g1_lincomb) 33 | - [`compute_powers`](#compute_powers) 34 | - [`compute_roots_of_unity`](#compute_roots_of_unity) 35 | - [Polynomials](#polynomials) 36 | - [`evaluate_polynomial_in_evaluation_form`](#evaluate_polynomial_in_evaluation_form) 37 | - [KZG](#kzg) 38 | - [`blob_to_kzg_commitment`](#blob_to_kzg_commitment) 39 | - [`verify_kzg_proof`](#verify_kzg_proof) 40 | - [`verify_kzg_proof_impl`](#verify_kzg_proof_impl) 41 | - [`verify_kzg_proof_batch`](#verify_kzg_proof_batch) 42 | - [`compute_kzg_proof`](#compute_kzg_proof) 43 | - [`compute_quotient_eval_within_domain`](#compute_quotient_eval_within_domain) 44 | - [`compute_kzg_proof_impl`](#compute_kzg_proof_impl) 45 | - [`compute_blob_kzg_proof`](#compute_blob_kzg_proof) 46 | - [`verify_blob_kzg_proof`](#verify_blob_kzg_proof) 47 | - [`verify_blob_kzg_proof_batch`](#verify_blob_kzg_proof_batch) 48 | 49 | 50 | 51 | 52 | ## Introduction 53 | 54 | This document specifies basic polynomial operations and KZG polynomial commitment operations that are essential for the implementation of the EIP-4844 feature in the Deneb specification. The implementations are not optimized for performance, but readability. All practical implementations should optimize the polynomial operations. 55 | 56 | Functions flagged as "Public method" MUST be provided by the underlying KZG library as public functions. All other functions are private functions used internally by the KZG library. 57 | 58 | Public functions MUST accept raw bytes as input and perform the required cryptographic normalization before invoking any internal functions. 59 | 60 | ## Custom types 61 | 62 | | Name | SSZ equivalent | Description | 63 | | - | - | - | 64 | | `G1Point` | `Bytes48` | | 65 | | `G2Point` | `Bytes96` | | 66 | | `BLSFieldElement` | `uint256` | Validation: `x < BLS_MODULUS` | 67 | | `KZGCommitment` | `Bytes48` | Validation: Perform [BLS standard's](https://datatracker.ietf.org/doc/html/draft-irtf-cfrg-bls-signature-04#section-2.5) "KeyValidate" check but do allow the identity point | 68 | | `KZGProof` | `Bytes48` | Same as for `KZGCommitment` | 69 | | `Polynomial` | `Vector[BLSFieldElement, FIELD_ELEMENTS_PER_BLOB]` | A polynomial in evaluation form | 70 | | `Blob` | `ByteVector[BYTES_PER_FIELD_ELEMENT * FIELD_ELEMENTS_PER_BLOB]` | A basic data blob | 71 | 72 | ## Constants 73 | 74 | | Name | Value | Notes | 75 | | - | - | - | 76 | | `BLS_MODULUS` | `52435875175126190479447740508185965837690552500527637822603658699938581184513` | Scalar field modulus of BLS12-381 | 77 | | `BYTES_PER_COMMITMENT` | `uint64(48)` | The number of bytes in a KZG commitment | 78 | | `BYTES_PER_PROOF` | `uint64(48)` | The number of bytes in a KZG proof | 79 | | `BYTES_PER_FIELD_ELEMENT` | `uint64(32)` | Bytes used to encode a BLS scalar field element | 80 | | `BYTES_PER_BLOB` | `uint64(BYTES_PER_FIELD_ELEMENT * FIELD_ELEMENTS_PER_BLOB)` | The number of bytes in a blob | 81 | | `G1_POINT_AT_INFINITY` | `Bytes48(b'\xc0' + b'\x00' * 47)` | Serialized form of the point at infinity on the G1 group | 82 | | `KZG_ENDIANNESS` | `'big'` | The endianness of the field elements including blobs | 83 | | `PRIMITIVE_ROOT_OF_UNITY` | `7` | The primitive root of unity from which all roots of unity should be derived | 84 | 85 | ## Preset 86 | 87 | ### Blob 88 | 89 | | Name | Value | 90 | | - | - | 91 | | `FIELD_ELEMENTS_PER_BLOB` | `uint64(4096)` | 92 | | `FIAT_SHAMIR_PROTOCOL_DOMAIN` | `b'FSBLOBVERIFY_V1_'` | 93 | | `RANDOM_CHALLENGE_KZG_BATCH_DOMAIN` | `b'RCKZGBATCH___V1_'` | 94 | 95 | ### Trusted setup 96 | 97 | | Name | Value | 98 | | - | - | 99 | | `KZG_SETUP_G2_LENGTH` | `65` | 100 | | `KZG_SETUP_G1_MONOMIAL` | `Vector[G1Point, FIELD_ELEMENTS_PER_BLOB]` | 101 | | `KZG_SETUP_G1_LAGRANGE` | `Vector[G1Point, FIELD_ELEMENTS_PER_BLOB]` | 102 | | `KZG_SETUP_G2_MONOMIAL` | `Vector[G2Point, KZG_SETUP_G2_LENGTH]` | 103 | 104 | KZG commitments depend on a [trusted setup](https://vitalik.eth.limo/general/2022/03/14/trustedsetup.html) to generate elliptic curve commitments to a series of powers of a secret number `s`. That is, these points are of the form `[G1, G1 * s, G1 * s**2 ... G1 * s**4095]` and `[G2, G2 * s, G2 * s**2 ... G2 * s**64]`. Nobody knows `s`, because [over 140,000 participants](https://ceremony.ethereum.org/) mixed their randomness to produce the trusted setup, and if even one of those was honest, the trusted setup is secure. 105 | 106 | ## Aside: what evaluation points are we using? 107 | 108 | As we mentioned above, the 4096 values in the deserialized blob are evaluations of a polynomial `P` at a set of 4096 points, and the 8192 values in the extended data are evaluations at _another_ 4096 points. But which points? Theoretically, any choice (even `0...8191`) would be valid, but some choices lead to much more efficient algorithms than others. We use **powers of a root of unity, in bit-reversal permutation**. Let's unpack this. 109 | 110 | First, "powers of a root of unity". We compute a value `w`, which satisfies `w ** 8192 == 1` modulo the BLS modulus, and where `w ** 4096` is not 1. You can think of the powers of `w` as being arranged in a circle. Any two points that are opposites on the circle are additive opposites of each other: if `w_k = w**k`, then the opposite point on the circle would be `w_{k + 4096} = w**(k + 4096) = w**k * w**4096 = w_k * -1`. The last identity that we used there, `w**4096 = -1`, follows from the fact that `w ** 8192 == 1` and `w ** 4096` is not 1: just like in real numbers, in a field the only two square roots of 1 are 1 and -1! 111 | 112 | This circle arrangement is convenient for efficient algorithms for transforming from evaluations to coordinates and back, most notably the [Fast Fourier transform](https://vitalik.eth.limo/general/2019/05/12/fft.html). 113 | 114 | The value `PRIMITIVE_ROOT_OF_UNITY = 7` set in the parameters above is used to generate `w`. We know that `PRIMITIVE_ROOT_OF_UNITY ** (BLS_MODULUS - 1) = 1` (because of [Fermat's little theorem](https://en.wikipedia.org/wiki/Fermat%27s_little_theorem)), and we chose `PRIMITIVE_ROOT_OF_UNITY` so that no smaller power of it equals 1 (that's the definition of "primitive root"). The desired property of `w` is that `w**8192` is the smallest power of `w` that equals 1. Hence, we generate `w` by computing `PRIMITIVE_ROOT_OF_UNITY ** ((BLS_MODULUS - 1) / 8192)`. 115 | 116 | Now, "bit reversal permutation". As it turns out, if you want to do operations involving multiple evaluation points, it's often much mathematically cleaner if they are related to each other by high powers of 2 that are near the full size of the circle (here, 8192). For example, `(X - w_k) * (X - w_{k + 4096}) = (X^2 - w_2k)`. Similarly, `(X - w_k) * (X - w_{k + 512}) * ... * (X - w_{k + 512 * 15}) = (X^16 - x_16k)`. The formulas for eg. `(X - w_k) * (X - w_{k + 1}) * ... * (X - w_{k + 15})` are much less clean. 117 | 118 | In data availability sampling, we put multiple adjacent evaluations together into one sample for efficiency. Generating KZG proofs for these samples is much more efficient if the evaluation points that are used in these adjacent evaluations are related by high powers of 2, like in the examples above. To accomplish this, we order the evaluation points in a very particular order. The order can be constructed recursively as follows: 119 | 120 | ``` 121 | def mk_order(n): 122 | if n == 0: 123 | return [0] 124 | else: 125 | prev = mk_order(n-1) 126 | return [x*2 for x in prev] + [x*2+1 for x in prev] 127 | ``` 128 | 129 | Here are the initial outputs: 130 | 131 | ``` 132 | >>> mk_order(0) 133 | [0] 134 | >>> mk_order(1) 135 | [0, 1] 136 | >>> mk_order(2) 137 | [0, 2, 1, 3] 138 | >>> mk_order(3) 139 | [0, 4, 2, 6, 1, 5, 3, 7] 140 | >>> mk_order(4) 141 | [0, 8, 4, 12, 2, 10, 6, 14, 1, 9, 5, 13, 3, 11, 7, 15] 142 | ``` 143 | 144 | That is, if we were to only have 16 total points (instead of 8192), the first point would be `P(w**0)`, the second would be `P(w**8)`, the third would be `P(w**4)`, etc. 145 | 146 | Notice that values that are close to each other are related by high powers of 2. In fact, if we compute `mk_order(13)`, and take any adjacent aligned slice of 16 values, we would find that they are related by a difference of 512, just like in our ideal example above: 147 | 148 | ``` 149 | >>> mk_order(13)[16*42:16*43] 150 | [168, 4264, 2216, 6312, 1192, 5288, 3240, 7336, 680, 4776, 2728, 6824, 1704, 5800, 3752, 7848] 151 | ``` 152 | 153 | This is the entire set of values `168 + 512k` for `k = 0..15`. As another bonus, notice that the first 4096 outputs are actually all multiples of a "sub-root of unity", `w**2`, whose powers form a smaller circle of size 4096 (as `(w**2)**4096` circles back to 1). Hence, the values "in the serialized blob", which only uses the first 4096 positions, are all evaluations at powers of `w**2`; meanwhile, the samples are at powers of `w`, both even and odd. 154 | 155 | ## Helper functions 156 | 157 | ### Bit-reversal permutation 158 | 159 | All polynomials (which are always given in Lagrange form) should be interpreted as being in 160 | bit-reversal permutation. In practice, clients can implement this by storing the lists 161 | `KZG_SETUP_G1_LAGRANGE` and roots of unity in bit-reversal permutation, so these functions only 162 | have to be called once at startup. 163 | 164 | #### `is_power_of_two` 165 | 166 | ```python 167 | def is_power_of_two(value: int) -> bool: 168 | """ 169 | Check if ``value`` is a power of two integer. 170 | """ 171 | return (value > 0) and (value & (value - 1) == 0) 172 | ``` 173 | 174 | #### `reverse_bits` 175 | 176 | ```python 177 | def reverse_bits(n: int, order: int) -> int: 178 | """ 179 | Reverse the bit order of an integer ``n``. 180 | """ 181 | assert is_power_of_two(order) 182 | # Convert n to binary with the same number of bits as "order" - 1, then reverse its bit order 183 | return int(('{:0' + str(order.bit_length() - 1) + 'b}').format(n)[::-1], 2) 184 | ``` 185 | 186 | #### `bit_reversal_permutation` 187 | 188 | ```python 189 | def bit_reversal_permutation(sequence: Sequence[T]) -> Sequence[T]: 190 | """ 191 | Return a copy with bit-reversed permutation. The permutation is an involution (inverts itself). 192 | 193 | The input and output are a sequence of generic type ``T`` objects. 194 | """ 195 | return [sequence[reverse_bits(i, len(sequence))] for i in range(len(sequence))] 196 | ``` 197 | 198 | ### BLS12-381 helpers 199 | 200 | 201 | #### `multi_exp` 202 | 203 | This function performs a multi-scalar multiplication between `points` and `integers`. `points` can either be in G1 or G2. 204 | 205 | ```python 206 | def multi_exp(points: Sequence[TPoint], 207 | integers: Sequence[uint64]) -> Sequence[TPoint]: 208 | # pylint: disable=unused-argument 209 | ... 210 | ``` 211 | 212 | Note that "multi-exponentiation", "multi-scalar multiplication" and "linear combination" all mean the same thing. Part of the reason why is that there were two separate cryptographic traditions, one of which viewed the operation to combine two elliptic curve points as being "addition", and the other which viewed it as "multiplication". "Linear combination" is a [long-established term](https://en.wikipedia.org/wiki/Linear_combination) for "take a sequence of objects, and a sequence of constants, multiply the i'th object by the i'th constant, and add the results". 213 | 214 | This problem is of interest because there are [much more efficient algorithms](https://ethresear.ch/t/simple-guide-to-fast-linear-combinations-aka-multiexponentiations/7238) to compute these operations than a naive "walk through the lists, multiply and add" approach. 215 | 216 | #### `hash_to_bls_field` 217 | 218 | ```python 219 | def hash_to_bls_field(data: bytes) -> BLSFieldElement: 220 | """ 221 | Hash ``data`` and convert the output to a BLS scalar field element. 222 | The output is not uniform over the BLS field. 223 | """ 224 | hashed_data = hash(data) 225 | return BLSFieldElement(int.from_bytes(hashed_data, KZG_ENDIANNESS) % BLS_MODULUS) 226 | ``` 227 | 228 | #### `bytes_to_bls_field` 229 | 230 | ```python 231 | def bytes_to_bls_field(b: Bytes32) -> BLSFieldElement: 232 | """ 233 | Convert untrusted bytes to a trusted and validated BLS scalar field element. 234 | This function does not accept inputs greater than the BLS modulus. 235 | """ 236 | field_element = int.from_bytes(b, KZG_ENDIANNESS) 237 | assert field_element < BLS_MODULUS 238 | return BLSFieldElement(field_element) 239 | ``` 240 | 241 | #### `bls_field_to_bytes` 242 | 243 | ```python 244 | def bls_field_to_bytes(x: BLSFieldElement) -> Bytes32: 245 | return int.to_bytes(x % BLS_MODULUS, 32, KZG_ENDIANNESS) 246 | ``` 247 | 248 | #### `validate_kzg_g1` 249 | 250 | ```python 251 | def validate_kzg_g1(b: Bytes48) -> None: 252 | """ 253 | Perform BLS validation required by the types `KZGProof` and `KZGCommitment`. 254 | """ 255 | if b == G1_POINT_AT_INFINITY: 256 | return 257 | 258 | assert bls.KeyValidate(b) 259 | ``` 260 | 261 | #### `bytes_to_kzg_commitment` 262 | 263 | ```python 264 | def bytes_to_kzg_commitment(b: Bytes48) -> KZGCommitment: 265 | """ 266 | Convert untrusted bytes into a trusted and validated KZGCommitment. 267 | """ 268 | validate_kzg_g1(b) 269 | return KZGCommitment(b) 270 | ``` 271 | 272 | #### `bytes_to_kzg_proof` 273 | 274 | ```python 275 | def bytes_to_kzg_proof(b: Bytes48) -> KZGProof: 276 | """ 277 | Convert untrusted bytes into a trusted and validated KZGProof. 278 | """ 279 | validate_kzg_g1(b) 280 | return KZGProof(b) 281 | ``` 282 | 283 | #### `blob_to_polynomial` 284 | 285 | ```python 286 | def blob_to_polynomial(blob: Blob) -> Polynomial: 287 | """ 288 | Convert a blob to list of BLS field scalars. 289 | """ 290 | polynomial = Polynomial() 291 | for i in range(FIELD_ELEMENTS_PER_BLOB): 292 | value = bytes_to_bls_field(blob[i * BYTES_PER_FIELD_ELEMENT: (i + 1) * BYTES_PER_FIELD_ELEMENT]) 293 | polynomial[i] = value 294 | return polynomial 295 | ``` 296 | 297 | Note that the "polynomial" outputted by this is a list of _evaluations_, not a list of _coefficients_. 298 | 299 | #### `compute_challenge` 300 | 301 | Suppose that a client receives a blob from a `BlobSidecar`, and they want to check if it matches the KZG commitment in the `BeaconBlock`. One way to do this would be to compute the KZG commitment from the blob directly, and check if it matches. However, this is too slow to do with many blobs. Instead, we use a clever cryptographic trick. We hash the blob and the commitment to generate a random point. We require the beacon block to provide an evaluation of the polynomial at that point, along with a KZG evaluation proof. The client can then evaluate the polynomial from the blob directly, and check that the two evaluations match. This is a common random checking trick used in cryptography: to verify that two polynomials `P` and `Q` are different, choose a random `x`, and verify that `P(x) = Q(x)`. It works as long as the domain from which `x` is chosen is large enough; in this case, it is (because `BLS_MODULUS` is a 256-bit number). 302 | 303 | This function computes this challenge point `x`. 304 | 305 | ```python 306 | def compute_challenge(blob: Blob, 307 | commitment: KZGCommitment) -> BLSFieldElement: 308 | """ 309 | Return the Fiat-Shamir challenge required by the rest of the protocol. 310 | """ 311 | 312 | # Append the degree of the polynomial as a domain separator 313 | degree_poly = int.to_bytes(FIELD_ELEMENTS_PER_BLOB, 16, KZG_ENDIANNESS) 314 | data = FIAT_SHAMIR_PROTOCOL_DOMAIN + degree_poly 315 | 316 | data += blob 317 | data += commitment 318 | 319 | # Transcript has been prepared: time to create the challenge 320 | return hash_to_bls_field(data) 321 | ``` 322 | 323 | #### `bls_modular_inverse` 324 | 325 | ```python 326 | def bls_modular_inverse(x: BLSFieldElement) -> BLSFieldElement: 327 | """ 328 | Compute the modular inverse of x (for x != 0) 329 | i.e. return y such that x * y % BLS_MODULUS == 1 330 | """ 331 | assert (int(x) % BLS_MODULUS) != 0 332 | return BLSFieldElement(pow(x, -1, BLS_MODULUS)) 333 | ``` 334 | 335 | #### `div` 336 | 337 | ```python 338 | def div(x: BLSFieldElement, y: BLSFieldElement) -> BLSFieldElement: 339 | """ 340 | Divide two field elements: ``x`` by `y``. 341 | """ 342 | return BLSFieldElement((int(x) * int(bls_modular_inverse(y))) % BLS_MODULUS) 343 | ``` 344 | 345 | #### `g1_lincomb` 346 | 347 | ```python 348 | def g1_lincomb(points: Sequence[KZGCommitment], scalars: Sequence[BLSFieldElement]) -> KZGCommitment: 349 | """ 350 | BLS multiscalar multiplication in G1. This can be naively implemented using double-and-add. 351 | """ 352 | assert len(points) == len(scalars) 353 | 354 | if len(points) == 0: 355 | return bls.G1_to_bytes48(bls.Z1()) 356 | 357 | points_g1 = [] 358 | for point in points: 359 | points_g1.append(bls.bytes48_to_G1(point)) 360 | 361 | result = bls.multi_exp(points_g1, scalars) 362 | return KZGCommitment(bls.G1_to_bytes48(result)) 363 | ``` 364 | 365 | #### `compute_powers` 366 | 367 | ```python 368 | def compute_powers(x: BLSFieldElement, n: uint64) -> Sequence[BLSFieldElement]: 369 | """ 370 | Return ``x`` to power of [0, n-1], if n > 0. When n==0, an empty array is returned. 371 | """ 372 | current_power = 1 373 | powers = [] 374 | for _ in range(n): 375 | powers.append(BLSFieldElement(current_power)) 376 | current_power = current_power * int(x) % BLS_MODULUS 377 | return powers 378 | ``` 379 | 380 | #### `compute_roots_of_unity` 381 | 382 | ```python 383 | def compute_roots_of_unity(order: uint64) -> Sequence[BLSFieldElement]: 384 | """ 385 | Return roots of unity of ``order``. 386 | """ 387 | assert (BLS_MODULUS - 1) % int(order) == 0 388 | root_of_unity = BLSFieldElement(pow(PRIMITIVE_ROOT_OF_UNITY, (BLS_MODULUS - 1) // int(order), BLS_MODULUS)) 389 | return compute_powers(root_of_unity, order) 390 | ``` 391 | 392 | ### Polynomials 393 | 394 | #### `evaluate_polynomial_in_evaluation_form` 395 | 396 | ```python 397 | def evaluate_polynomial_in_evaluation_form(polynomial: Polynomial, 398 | z: BLSFieldElement) -> BLSFieldElement: 399 | """ 400 | Evaluate a polynomial (in evaluation form) at an arbitrary point ``z``. 401 | - When ``z`` is in the domain, the evaluation can be found by indexing the polynomial at the 402 | position that ``z`` is in the domain. 403 | - When ``z`` is not in the domain, the barycentric formula is used: 404 | f(z) = (z**WIDTH - 1) / WIDTH * sum_(i=0)^WIDTH (f(DOMAIN[i]) * DOMAIN[i]) / (z - DOMAIN[i]) 405 | """ 406 | width = len(polynomial) 407 | assert width == FIELD_ELEMENTS_PER_BLOB 408 | inverse_width = bls_modular_inverse(BLSFieldElement(width)) 409 | 410 | roots_of_unity_brp = bit_reversal_permutation(compute_roots_of_unity(FIELD_ELEMENTS_PER_BLOB)) 411 | 412 | # If we are asked to evaluate within the domain, we already know the answer 413 | if z in roots_of_unity_brp: 414 | eval_index = roots_of_unity_brp.index(z) 415 | return BLSFieldElement(polynomial[eval_index]) 416 | 417 | result = 0 418 | for i in range(width): 419 | a = BLSFieldElement(int(polynomial[i]) * int(roots_of_unity_brp[i]) % BLS_MODULUS) 420 | b = BLSFieldElement((int(BLS_MODULUS) + int(z) - int(roots_of_unity_brp[i])) % BLS_MODULUS) 421 | result += int(div(a, b) % BLS_MODULUS) 422 | result = result * int(BLS_MODULUS + pow(z, width, BLS_MODULUS) - 1) * int(inverse_width) 423 | return BLSFieldElement(result % BLS_MODULUS) 424 | ``` 425 | 426 | Given a polynomial in evaluation form (which is what a serialized blob is), directly evaluate it at a given point `z`. It's possible to do this using the [barycentric formula](https://tobydriscoll.net/fnc-julia/globalapprox/barycentric.html), which the above function implements. It takes `O(N)` arithmetic operations. 427 | 428 | ### KZG 429 | 430 | KZG core functions. These are also defined in Deneb execution specs. 431 | 432 | #### `blob_to_kzg_commitment` 433 | 434 | ```python 435 | def blob_to_kzg_commitment(blob: Blob) -> KZGCommitment: 436 | """ 437 | Public method. 438 | """ 439 | assert len(blob) == BYTES_PER_BLOB 440 | return g1_lincomb(bit_reversal_permutation(KZG_SETUP_G1_LAGRANGE), blob_to_polynomial(blob)) 441 | ``` 442 | 443 | #### `verify_kzg_proof` 444 | 445 | ```python 446 | def verify_kzg_proof(commitment_bytes: Bytes48, 447 | z_bytes: Bytes32, 448 | y_bytes: Bytes32, 449 | proof_bytes: Bytes48) -> bool: 450 | """ 451 | Verify KZG proof that ``p(z) == y`` where ``p(z)`` is the polynomial represented by ``polynomial_kzg``. 452 | Receives inputs as bytes. 453 | Public method. 454 | """ 455 | assert len(commitment_bytes) == BYTES_PER_COMMITMENT 456 | assert len(z_bytes) == BYTES_PER_FIELD_ELEMENT 457 | assert len(y_bytes) == BYTES_PER_FIELD_ELEMENT 458 | assert len(proof_bytes) == BYTES_PER_PROOF 459 | 460 | return verify_kzg_proof_impl(bytes_to_kzg_commitment(commitment_bytes), 461 | bytes_to_bls_field(z_bytes), 462 | bytes_to_bls_field(y_bytes), 463 | bytes_to_kzg_proof(proof_bytes)) 464 | ``` 465 | 466 | This is the function that verifies a KZG proof of evaluation: if `commitment_bytes` is a commitment to `P`, it's only possible to generate a valid proof `proof_bytes` for a given `z_bytes` and `y_bytes` if `P(z) = y`. 467 | 468 | 469 | #### `verify_kzg_proof_impl` 470 | 471 | ```python 472 | def verify_kzg_proof_impl(commitment: KZGCommitment, 473 | z: BLSFieldElement, 474 | y: BLSFieldElement, 475 | proof: KZGProof) -> bool: 476 | """ 477 | Verify KZG proof that ``p(z) == y`` where ``p(z)`` is the polynomial represented by ``polynomial_kzg``. 478 | """ 479 | # Verify: P - y = Q * (X - z) 480 | X_minus_z = bls.add( 481 | bls.bytes96_to_G2(KZG_SETUP_G2_MONOMIAL[1]), 482 | bls.multiply(bls.G2(), (BLS_MODULUS - z) % BLS_MODULUS), 483 | ) 484 | P_minus_y = bls.add(bls.bytes48_to_G1(commitment), bls.multiply(bls.G1(), (BLS_MODULUS - y) % BLS_MODULUS)) 485 | return bls.pairing_check([ 486 | [P_minus_y, bls.neg(bls.G2())], 487 | [bls.bytes48_to_G1(proof), X_minus_z] 488 | ]) 489 | ``` 490 | 491 | Verifying a KZG proof requires computing an [elliptic curve pairing](https://vitalik.eth.limo/general/2017/01/14/exploring_ecp.html). Pairings with KZG commitments have the property that `e(commit(A), commit(B)) = e(commit(C), commit(1))` only if, as polynomials, `C = A * B`. The standard way to prove that `P(z) = y`, is to require a commitment to `Q = (P - y) / (X - z)`. The verifier computes `commit(X - z)` and `commit(P - y)`, and then uses a pairing to check that the product of those two equals to `Q`. Both of those commitmetns are easy to compute in O(1) time: `commit(X - z)` directly, ad `commit(P - y)` as `commit(P) - commit(y)`, where `commit(P)` was already provided. Because pairings need to take a `G1` element and a `G2` element as input, we compute `(X - z)` in G2 form. 492 | 493 | #### `verify_kzg_proof_batch` 494 | 495 | ```python 496 | def verify_kzg_proof_batch(commitments: Sequence[KZGCommitment], 497 | zs: Sequence[BLSFieldElement], 498 | ys: Sequence[BLSFieldElement], 499 | proofs: Sequence[KZGProof]) -> bool: 500 | """ 501 | Verify multiple KZG proofs efficiently. 502 | """ 503 | 504 | assert len(commitments) == len(zs) == len(ys) == len(proofs) 505 | 506 | # Compute a random challenge. Note that it does not have to be computed from a hash, 507 | # r just has to be random. 508 | degree_poly = int.to_bytes(FIELD_ELEMENTS_PER_BLOB, 8, KZG_ENDIANNESS) 509 | num_commitments = int.to_bytes(len(commitments), 8, KZG_ENDIANNESS) 510 | data = RANDOM_CHALLENGE_KZG_BATCH_DOMAIN + degree_poly + num_commitments 511 | 512 | # Append all inputs to the transcript before we hash 513 | for commitment, z, y, proof in zip(commitments, zs, ys, proofs): 514 | data += commitment \ 515 | + int.to_bytes(z, BYTES_PER_FIELD_ELEMENT, KZG_ENDIANNESS) \ 516 | + int.to_bytes(y, BYTES_PER_FIELD_ELEMENT, KZG_ENDIANNESS) \ 517 | + proof 518 | 519 | r = hash_to_bls_field(data) 520 | r_powers = compute_powers(r, len(commitments)) 521 | 522 | # Verify: e(sum r^i proof_i, [s]) == 523 | # e(sum r^i (commitment_i - [y_i]) + sum r^i z_i proof_i, [1]) 524 | proof_lincomb = g1_lincomb(proofs, r_powers) 525 | proof_z_lincomb = g1_lincomb( 526 | proofs, 527 | [BLSFieldElement((int(z) * int(r_power)) % BLS_MODULUS) for z, r_power in zip(zs, r_powers)], 528 | ) 529 | C_minus_ys = [bls.add(bls.bytes48_to_G1(commitment), bls.multiply(bls.G1(), (BLS_MODULUS - y) % BLS_MODULUS)) 530 | for commitment, y in zip(commitments, ys)] 531 | C_minus_y_as_KZGCommitments = [KZGCommitment(bls.G1_to_bytes48(x)) for x in C_minus_ys] 532 | C_minus_y_lincomb = g1_lincomb(C_minus_y_as_KZGCommitments, r_powers) 533 | 534 | return bls.pairing_check([ 535 | [bls.bytes48_to_G1(proof_lincomb), bls.neg(bls.bytes96_to_G2(KZG_SETUP_G2_MONOMIAL[1]))], 536 | [bls.add(bls.bytes48_to_G1(C_minus_y_lincomb), bls.bytes48_to_G1(proof_z_lincomb)), bls.G2()] 537 | ]) 538 | ``` 539 | 540 | This is an optimized algorithm for verifying many KZG evaluation proofs at the same time. 541 | 542 | #### `compute_kzg_proof` 543 | 544 | ```python 545 | def compute_kzg_proof(blob: Blob, z_bytes: Bytes32) -> Tuple[KZGProof, Bytes32]: 546 | """ 547 | Compute KZG proof at point `z` for the polynomial represented by `blob`. 548 | Do this by computing the quotient polynomial in evaluation form: q(x) = (p(x) - p(z)) / (x - z). 549 | Public method. 550 | """ 551 | assert len(blob) == BYTES_PER_BLOB 552 | assert len(z_bytes) == BYTES_PER_FIELD_ELEMENT 553 | polynomial = blob_to_polynomial(blob) 554 | proof, y = compute_kzg_proof_impl(polynomial, bytes_to_bls_field(z_bytes)) 555 | return proof, y.to_bytes(BYTES_PER_FIELD_ELEMENT, KZG_ENDIANNESS) 556 | ``` 557 | 558 | #### `compute_quotient_eval_within_domain` 559 | 560 | ```python 561 | def compute_quotient_eval_within_domain(z: BLSFieldElement, 562 | polynomial: Polynomial, 563 | y: BLSFieldElement 564 | ) -> BLSFieldElement: 565 | """ 566 | Given `y == p(z)` for a polynomial `p(x)`, compute `q(z)`: the KZG quotient polynomial evaluated at `z` for the 567 | special case where `z` is in roots of unity. 568 | 569 | For more details, read https://dankradfeist.de/ethereum/2021/06/18/pcs-multiproofs.html section "Dividing 570 | when one of the points is zero". The code below computes q(x_m) for the roots of unity special case. 571 | """ 572 | roots_of_unity_brp = bit_reversal_permutation(compute_roots_of_unity(FIELD_ELEMENTS_PER_BLOB)) 573 | result = 0 574 | for i, omega_i in enumerate(roots_of_unity_brp): 575 | if omega_i == z: # skip the evaluation point in the sum 576 | continue 577 | 578 | f_i = int(BLS_MODULUS) + int(polynomial[i]) - int(y) % BLS_MODULUS 579 | numerator = f_i * int(omega_i) % BLS_MODULUS 580 | denominator = int(z) * (int(BLS_MODULUS) + int(z) - int(omega_i)) % BLS_MODULUS 581 | result += int(div(BLSFieldElement(numerator), BLSFieldElement(denominator))) 582 | 583 | return BLSFieldElement(result % BLS_MODULUS) 584 | ``` 585 | 586 | #### `compute_kzg_proof_impl` 587 | 588 | ```python 589 | def compute_kzg_proof_impl(polynomial: Polynomial, z: BLSFieldElement) -> Tuple[KZGProof, BLSFieldElement]: 590 | """ 591 | Helper function for `compute_kzg_proof()` and `compute_blob_kzg_proof()`. 592 | """ 593 | roots_of_unity_brp = bit_reversal_permutation(compute_roots_of_unity(FIELD_ELEMENTS_PER_BLOB)) 594 | 595 | # For all x_i, compute p(x_i) - p(z) 596 | y = evaluate_polynomial_in_evaluation_form(polynomial, z) 597 | polynomial_shifted = [BLSFieldElement((int(p) - int(y)) % BLS_MODULUS) for p in polynomial] 598 | 599 | # For all x_i, compute (x_i - z) 600 | denominator_poly = [BLSFieldElement((int(x) - int(z)) % BLS_MODULUS) 601 | for x in roots_of_unity_brp] 602 | 603 | # Compute the quotient polynomial directly in evaluation form 604 | quotient_polynomial = [BLSFieldElement(0)] * FIELD_ELEMENTS_PER_BLOB 605 | for i, (a, b) in enumerate(zip(polynomial_shifted, denominator_poly)): 606 | if b == 0: 607 | # The denominator is zero hence `z` is a root of unity: we must handle it as a special case 608 | quotient_polynomial[i] = compute_quotient_eval_within_domain(roots_of_unity_brp[i], polynomial, y) 609 | else: 610 | # Compute: q(x_i) = (p(x_i) - p(z)) / (x_i - z). 611 | quotient_polynomial[i] = div(a, b) 612 | 613 | return KZGProof(g1_lincomb(bit_reversal_permutation(KZG_SETUP_G1_LAGRANGE), quotient_polynomial)), y 614 | ``` 615 | 616 | #### `compute_blob_kzg_proof` 617 | 618 | ```python 619 | def compute_blob_kzg_proof(blob: Blob, commitment_bytes: Bytes48) -> KZGProof: 620 | """ 621 | Given a blob, return the KZG proof that is used to verify it against the commitment. 622 | This method does not verify that the commitment is correct with respect to `blob`. 623 | Public method. 624 | """ 625 | assert len(blob) == BYTES_PER_BLOB 626 | assert len(commitment_bytes) == BYTES_PER_COMMITMENT 627 | commitment = bytes_to_kzg_commitment(commitment_bytes) 628 | polynomial = blob_to_polynomial(blob) 629 | evaluation_challenge = compute_challenge(blob, commitment) 630 | proof, _ = compute_kzg_proof_impl(polynomial, evaluation_challenge) 631 | return proof 632 | ``` 633 | 634 | See the more detailed description of what's going on in [`compute_challenge`](#compute_challenge). 635 | 636 | #### `verify_blob_kzg_proof` 637 | 638 | ```python 639 | def verify_blob_kzg_proof(blob: Blob, 640 | commitment_bytes: Bytes48, 641 | proof_bytes: Bytes48) -> bool: 642 | """ 643 | Given a blob and a KZG proof, verify that the blob data corresponds to the provided commitment. 644 | 645 | Public method. 646 | """ 647 | assert len(blob) == BYTES_PER_BLOB 648 | assert len(commitment_bytes) == BYTES_PER_COMMITMENT 649 | assert len(proof_bytes) == BYTES_PER_PROOF 650 | 651 | commitment = bytes_to_kzg_commitment(commitment_bytes) 652 | 653 | polynomial = blob_to_polynomial(blob) 654 | evaluation_challenge = compute_challenge(blob, commitment) 655 | 656 | # Evaluate polynomial at `evaluation_challenge` 657 | y = evaluate_polynomial_in_evaluation_form(polynomial, evaluation_challenge) 658 | 659 | # Verify proof 660 | proof = bytes_to_kzg_proof(proof_bytes) 661 | return verify_kzg_proof_impl(commitment, evaluation_challenge, y, proof) 662 | ``` 663 | 664 | See the more detailed description of what's going on in [`compute_challenge`](#compute_challenge). 665 | 666 | #### `verify_blob_kzg_proof_batch` 667 | 668 | ```python 669 | def verify_blob_kzg_proof_batch(blobs: Sequence[Blob], 670 | commitments_bytes: Sequence[Bytes48], 671 | proofs_bytes: Sequence[Bytes48]) -> bool: 672 | """ 673 | Given a list of blobs and blob KZG proofs, verify that they correspond to the provided commitments. 674 | Will return True if there are zero blobs/commitments/proofs. 675 | Public method. 676 | """ 677 | 678 | assert len(blobs) == len(commitments_bytes) == len(proofs_bytes) 679 | 680 | commitments, evaluation_challenges, ys, proofs = [], [], [], [] 681 | for blob, commitment_bytes, proof_bytes in zip(blobs, commitments_bytes, proofs_bytes): 682 | assert len(blob) == BYTES_PER_BLOB 683 | assert len(commitment_bytes) == BYTES_PER_COMMITMENT 684 | assert len(proof_bytes) == BYTES_PER_PROOF 685 | commitment = bytes_to_kzg_commitment(commitment_bytes) 686 | commitments.append(commitment) 687 | polynomial = blob_to_polynomial(blob) 688 | evaluation_challenge = compute_challenge(blob, commitment) 689 | evaluation_challenges.append(evaluation_challenge) 690 | ys.append(evaluate_polynomial_in_evaluation_form(polynomial, evaluation_challenge)) 691 | proofs.append(bytes_to_kzg_proof(proof_bytes)) 692 | 693 | return verify_kzg_proof_batch(commitments, evaluation_challenges, ys, proofs) 694 | ``` 695 | 696 | Use `verify_kzg_proof_batch` to verify multiple blob proofs at the same time. 697 | -------------------------------------------------------------------------------- /deneb/validator.md: -------------------------------------------------------------------------------- 1 | # Deneb -- Honest Validator 2 | 3 | ## Table of contents 4 | 5 | 6 | 7 | 8 | 9 | - [Introduction](#introduction) 10 | - [Prerequisites](#prerequisites) 11 | - [Helpers](#helpers) 12 | - [`BlobsBundle`](#blobsbundle) 13 | - [Modified `GetPayloadResponse`](#modified-getpayloadresponse) 14 | - [Protocol](#protocol) 15 | - [`ExecutionEngine`](#executionengine) 16 | - [Modified `get_payload`](#modified-get_payload) 17 | - [Beacon chain responsibilities](#beacon-chain-responsibilities) 18 | - [Block and sidecar proposal](#block-and-sidecar-proposal) 19 | - [Constructing the `BeaconBlockBody`](#constructing-the-beaconblockbody) 20 | - [ExecutionPayload](#executionpayload) 21 | - [Blob KZG commitments](#blob-kzg-commitments) 22 | - [Constructing the `BlobSidecar`s](#constructing-the-blobsidecars) 23 | - [Sidecar](#sidecar) 24 | 25 | 26 | 27 | 28 | ## Introduction 29 | 30 | The main addition to the "honest validator logic" is the need to handle `BlobSidecar`s, which contain the underlying data in blobs. 31 | 32 | ## Prerequisites 33 | 34 | This document is an extension of the [Capella -- Honest Validator](../capella/validator.md) guide. 35 | All behaviors and definitions defined in this document, and documents it extends, carry over unless explicitly noted or overridden. 36 | 37 | All terminology, constants, functions, and protocol mechanics defined in the updated [Beacon Chain doc of Deneb](./beacon-chain.md) are requisite for this document and used throughout. 38 | Please see related Beacon Chain doc before continuing and use them as a reference throughout. 39 | 40 | ## Helpers 41 | 42 | ### `BlobsBundle` 43 | 44 | *[New in Deneb:EIP4844]* 45 | 46 | ```python 47 | @dataclass 48 | class BlobsBundle(object): 49 | commitments: Sequence[KZGCommitment] 50 | proofs: Sequence[KZGProof] 51 | blobs: Sequence[Blob] 52 | ``` 53 | 54 | ### Modified `GetPayloadResponse` 55 | 56 | ```python 57 | @dataclass 58 | class GetPayloadResponse(object): 59 | execution_payload: ExecutionPayload 60 | block_value: uint256 61 | blobs_bundle: BlobsBundle # [New in Deneb:EIP4844] 62 | ``` 63 | 64 | ```python 65 | def compute_signed_block_header(signed_block: SignedBeaconBlock) -> SignedBeaconBlockHeader: 66 | block = signed_block.message 67 | block_header = BeaconBlockHeader( 68 | slot=block.slot, 69 | proposer_index=block.proposer_index, 70 | parent_root=block.parent_root, 71 | state_root=block.state_root, 72 | body_root=hash_tree_root(block.body), 73 | ) 74 | return SignedBeaconBlockHeader(message=block_header, signature=signed_block.signature) 75 | ``` 76 | 77 | ## Protocol 78 | 79 | ### `ExecutionEngine` 80 | 81 | #### Modified `get_payload` 82 | 83 | Given the `payload_id`, `get_payload` returns the most recent version of the execution payload that 84 | has been built since the corresponding call to `notify_forkchoice_updated` method. 85 | 86 | ```python 87 | def get_payload(self: ExecutionEngine, payload_id: PayloadId) -> GetPayloadResponse: 88 | """ 89 | Return ExecutionPayload, uint256, BlobsBundle objects. 90 | """ 91 | # pylint: disable=unused-argument 92 | ... 93 | ``` 94 | 95 | ## Beacon chain responsibilities 96 | 97 | All validator responsibilities remain unchanged other than those noted below. 98 | 99 | ### Block and sidecar proposal 100 | 101 | #### Constructing the `BeaconBlockBody` 102 | 103 | ##### ExecutionPayload 104 | 105 | `prepare_execution_payload` is updated from the Capella specs to provide the parent beacon block root. This is a tuny change, to handle EIP-2935; it is unrelated to blobs. 106 | 107 | *Note*: In this section, `state` is the state of the slot for the block proposal _without_ the block yet applied. 108 | That is, `state` is the `previous_state` processed through any empty slots up to the assigned slot using `process_slots(previous_state, slot)`. 109 | 110 | *Note*: The only change made to `prepare_execution_payload` is to add the parent beacon block root as an additional 111 | parameter to the `PayloadAttributes`. 112 | 113 | ```python 114 | def prepare_execution_payload(state: BeaconState, 115 | safe_block_hash: Hash32, 116 | finalized_block_hash: Hash32, 117 | suggested_fee_recipient: ExecutionAddress, 118 | execution_engine: ExecutionEngine) -> Optional[PayloadId]: 119 | # Verify consistency of the parent hash with respect to the previous execution payload header 120 | parent_hash = state.latest_execution_payload_header.block_hash 121 | 122 | # Set the forkchoice head and initiate the payload build process 123 | payload_attributes = PayloadAttributes( 124 | timestamp=compute_timestamp_at_slot(state, state.slot), 125 | prev_randao=get_randao_mix(state, get_current_epoch(state)), 126 | suggested_fee_recipient=suggested_fee_recipient, 127 | withdrawals=get_expected_withdrawals(state), 128 | parent_beacon_block_root=hash_tree_root(state.latest_block_header), # [New in Deneb:EIP4788] 129 | ) 130 | return execution_engine.notify_forkchoice_updated( 131 | head_block_hash=parent_hash, 132 | safe_block_hash=safe_block_hash, 133 | finalized_block_hash=finalized_block_hash, 134 | payload_attributes=payload_attributes, 135 | ) 136 | ``` 137 | 138 | ##### Blob KZG commitments 139 | 140 | *[New in Deneb:EIP4844]* 141 | 142 | 1. The execution payload is obtained from the execution engine as defined above using `payload_id`. The response also includes a `blobs_bundle` entry containing the corresponding `blobs`, `commitments`, and `proofs`. 143 | 2. Set `block.body.blob_kzg_commitments = commitments`. 144 | 145 | #### Constructing the `BlobSidecar`s 146 | 147 | *[New in Deneb:EIP4844]* 148 | 149 | To construct a `BlobSidecar`, a `blob_sidecar` is defined with the necessary context for block and sidecar proposal. 150 | 151 | ##### Sidecar 152 | 153 | Blobs associated with a block are packaged into sidecar objects for distribution to the associated sidecar topic, the `blob_sidecar_{subnet_id}` pubsub topic. 154 | 155 | Each `sidecar` is obtained from: 156 | ```python 157 | def get_blob_sidecars(signed_block: SignedBeaconBlock, 158 | blobs: Sequence[Blob], 159 | blob_kzg_proofs: Sequence[KZGProof]) -> Sequence[BlobSidecar]: 160 | block = signed_block.message 161 | signed_block_header = compute_signed_block_header(signed_block) 162 | return [ 163 | BlobSidecar( 164 | index=index, 165 | blob=blob, 166 | kzg_commitment=block.body.blob_kzg_commitments[index], 167 | kzg_proof=blob_kzg_proofs[index], 168 | signed_block_header=signed_block_header, 169 | kzg_commitment_inclusion_proof=compute_merkle_proof( 170 | block.body, 171 | get_generalized_index(BeaconBlockBody, 'blob_kzg_commitments', index), 172 | ), 173 | ) 174 | for index, blob in enumerate(blobs) 175 | ] 176 | ``` 177 | 178 | The `subnet_id` for the `blob_sidecar` is calculated with: 179 | - Let `blob_index = blob_sidecar.index`. 180 | - Let `subnet_id = compute_subnet_for_blob_sidecar(blob_index)`. 181 | 182 | ```python 183 | def compute_subnet_for_blob_sidecar(blob_index: BlobIndex) -> SubnetID: 184 | return SubnetID(blob_index % BLOB_SIDECAR_SUBNET_COUNT) 185 | ``` 186 | 187 | After publishing the peers on the network may request the sidecar through sync-requests, or a local user may be interested. 188 | 189 | The validator MUST hold on to sidecars for `MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS` epochs and serve when capable, 190 | to ensure the data-availability of these blobs throughout the network. 191 | 192 | After `MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS` nodes MAY prune the sidecars and/or stop serving them. 193 | -------------------------------------------------------------------------------- /images/checkpoint_tree_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ethereum/annotated-spec/160764ac180eca2cea3581f731ee96ac7098f9f7/images/checkpoint_tree_1.png -------------------------------------------------------------------------------- /images/checkpoint_tree_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ethereum/annotated-spec/160764ac180eca2cea3581f731ee96ac7098f9f7/images/checkpoint_tree_2.png -------------------------------------------------------------------------------- /images/checkpoint_tree_3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ethereum/annotated-spec/160764ac180eca2cea3581f731ee96ac7098f9f7/images/checkpoint_tree_3.png -------------------------------------------------------------------------------- /images/checkpoint_tree_4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ethereum/annotated-spec/160764ac180eca2cea3581f731ee96ac7098f9f7/images/checkpoint_tree_4.png -------------------------------------------------------------------------------- /images/inactivityleak.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ethereum/annotated-spec/160764ac180eca2cea3581f731ee96ac7098f9f7/images/inactivityleak.png -------------------------------------------------------------------------------- /images/light_client_sync.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ethereum/annotated-spec/160764ac180eca2cea3581f731ee96ac7098f9f7/images/light_client_sync.png -------------------------------------------------------------------------------- /images/pow_chain.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ethereum/annotated-spec/160764ac180eca2cea3581f731ee96ac7098f9f7/images/pow_chain.png -------------------------------------------------------------------------------- /images/randomness.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ethereum/annotated-spec/160764ac180eca2cea3581f731ee96ac7098f9f7/images/randomness.png -------------------------------------------------------------------------------- /images/roadmap.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ethereum/annotated-spec/160764ac180eca2cea3581f731ee96ac7098f9f7/images/roadmap.jpeg -------------------------------------------------------------------------------- /images/shardchains.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ethereum/annotated-spec/160764ac180eca2cea3581f731ee96ac7098f9f7/images/shardchains.png -------------------------------------------------------------------------------- /images/singleslot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ethereum/annotated-spec/160764ac180eca2cea3581f731ee96ac7098f9f7/images/singleslot.png -------------------------------------------------------------------------------- /images/src/checkpoint_tree_1.drawio: -------------------------------------------------------------------------------- 1 | 7VrLcpswFP0aL9MBCWO89Cvtok3SSWeaLBWjAK2MqCzHuF9fARJPP2MMdOKVuUcPpHvuuVcj3IOTRfiZocD9Rm1MekCzwx6c9gDQYV8TPxGySZCBOUgAh3m27JQBj95fLEE5zll5Nl4WOnJKCfeCIjinvo/nvIAhxui62O2VkuJbA+TgCvA4R6SK/vRs7iaopbYV4V+w57jqzbomWxZIdZbA0kU2XecgOOvBCaOUJ0+LcIJJ5Dzll2Tc7Y7WdGEM+/yYAeFNcP/6fXqnPfPpffADeHxt3RhybXyjNoxtsX9pUsZd6lAfkVmGjhld+TaOZtWElfX5SmkgQF2AvzDnG0kmWnEqIJcviGzFocefcs/P0VSf+tKahnLm2Ngow+ds85Q3cqMiMxsWW9k4exSFgTB96uMEufUIke3J/qNN73SrhJZ0xeZ4jy9VeCLmYL6nH0zJF6rBdIHFesU4hgni3ltxHUiGr5P2yxgWD5LkEwiHWpuMaycx3ixzZzAihz5QT7wZaDL5GdLVMvNBZasZkkCRg0q0pqt4P9NyvW+IrOQOKswXeV27HsePAYpdtRbpvMihnA4zjsP9Hq96Ug6wSh5RqXGdZVblJDeXVMuOq00M4LCLqlFoo6Ub+0wv+ifCHxDnmPkxAjQo0CVn9HdaOsAJcVv1Ys5N/S1uUtiZYauX4tawSu5P5FSJ24MTAa1ZAcAKu6PzFPAqks+EEsrisdDuY8s2Uo5zLRZ4gaZZj2Z0s2Oi6V+PDMcWnoNHAfMihScV2q7Ks0PBtVWeVs8YxRAZdD1GDh5OzCNjadjmsdLs3mGjcv5qO3EOuqMK0HVVHIx26xT11J9hSydZcOQZqa4Ma3VPbqm8lEvMluU27L6PoNWyj/QO5aT/PiXp1kVyDTTbzTV6B5ONoXUs2ejDq5BqE1LuerzW+7jyfUTDQlLJPyeksV4Jm+7fSBjlw0/b4gNVL17F927xgUaqmFG+DLy0+I646227irV+HATwKqT6hGQ0UsUaF5JRrWJnSqkTVax98fUPZ6iP9zXKKNeN936NKk/U9NcosO2K1CQ8in8aLzTj2fyzoqrhZhnnzZHooBtBmDWKJyf6nQV07orWu+iLbzLhC1ONChFLTt6i4FJgCR3xYgAVxSejLa9UCSHiOVGIzUW8YIGPI1V6c0RGsmHh2XZcKLZlgWKeqEHWJZb14XGqBhdT9eDStH84isuXr+1zvO2GpGaOx1dxRzWlMeaFmf01L6kB2R8c4ewf -------------------------------------------------------------------------------- /images/src/checkpoint_tree_2.drawio: -------------------------------------------------------------------------------- 1 | 7VrLcpswFP0aL9MBCTBe+pV20SbppDNNlopRgBYjKsux3a+vAImHsIMf2DCNV0ZHD8S999x7kOnB8Xz9maLI+0YcHPSA5qx7cNIDQIemxn9iZJMifaufAi71HTEoBx79v1iAYp679B28KA1khATMj8rgjIQhnrEShiglq/KwVxKU7xohF1eAxxkKquhP32FeitrysWL8C/ZdT95Z10TPHMnBAlh4yCGrAgSnPTimhLD0ar4e4yA2nrRLOu92R2+2MYpDts+E9U10//p9cqc9s8l99AP4bGXfGGJvbCMfGDv8+UWTUOYRl4QomOboiJJl6OB4VY238jFfCYk4qHPwF2ZsI5yJloxwyGPzQPTitc+eCtfP8VKfTNGarMXKSWMjGyGjm6diozArbubTklY+zxnGYcCbIQlxitz6QSD60+ePH3qnWQW0IEs6w+/YUoYnoi5m74yDmfM5azCZY75fPo/iADH/rbwPJMLXzcblHuYXwskHOBxqbXpcO8jjl/XcCR4RUx+Iz+8MNJH8DGFqkfmgbMsV0kARkxS3Zrs43tNiv28oWIonqHi+7NeV5zP8GKHEVCuezss+FMthyvD6fYtXLSkm2IpFZGpc5ZlVGskrJFXVcI2RAdSbqBqFDlp4ic30sn1i/AExhmmYIECDHF0wSn5npQMcELdVKxbMZG4xk8RODFtdiVvDVsyf0qkSt7ULAe2yBIAV7w5PY8ArTz5jEhCazIWOiW3HyHxc6LHBC7SsZjijWx0jjXmVDPsWnlopYJ2l8GRE21V5djC4scrTqsYoh0i/6zFSK06sPWNp0KastLonNir6q+3E2e8OK0DXWVEb7bp9CH2aT7GKlAXHiqQsKnctdGaRNKgQd3QidVtRSaoZod0y2fUOsf0/IXvjJIbWcSRuTCfZ3auahqIdgdU2kQZXIjVGpMLBc6MnXeqb/oWJJJN/sYrplbDpfhUzVFXRNvmAWZ+hPt4JmaHUjaNPyNSFLn1CBra9tlkBi+OfJBvN/Wz9WRLZcbNI8uaQD9CNaJ138is3/p1GZObx3rv4FDpd8IXKTonwLad3kbASWJxHrBxAZfKJaCsyVUAo8N04xGY8XjDHRzEr/RkKhqJj7jtOUii2ZYFynmiA1oqX9cF+rAZnY3X/3G7/cC5W3wfb9/E2bdmwj0dXcsc1pWOeh1XlcxXMxwpmCLY7/0QFYdad+pxZMENQr+su/eZpqmKobfEL4ZVIzRFJfmlUd1576t8Y+xHOUJX2uQlndJ9wZzwz5c38Q7fUpvnngnD6Dw== -------------------------------------------------------------------------------- /images/src/checkpoint_tree_3.drawio: -------------------------------------------------------------------------------- 1 | 7VpNc5swEP01nmkP7gAC7Bzjj7SHNkknnWlyVI0MajGishzj/voKEAYEjrEBw7Q+GT1JK7S7b3clPADTVfCRQt/5QizkDjTFCgZgNtA0FRgK/wmRXYyMzFEM2BRbYlAKPOE/SIBinr3BFlrnBjJCXIb9PLggnocWLIdBSsk2P2xJ3PyqPrRRAXhaQLeIfscWc2J0nGwrxD8hbDvJyqoielYwGSyAtQMtss1AYD4AU0oIi59WwRS5ofISvcTz7g707l+MIo9VmRAM/Yfl19m98sJmD/43DbPteKjFUl6huxEbFi/LdokGkGfdhorkLY94HJxYcO2gUKrKGw5bueIxxB8hY4h6EaIpgKNrRsmvvfL4tifxEsgqaD7dirpXEPcsRFaI0R0fsk1NkFjAyWg/wShyIcOvefFQeIK9F7df4ZFgvrCmCK/dm0z4rD5W8iLWZEMXSMzKqvyIIE2RBDFIbcQKgvhDZtspFFn0BOsaRWNypT+JJqHMITbxoDtP0QklG8+KrKvwVjrmMyG+sPNPxNhOUBVuGMl7AQowe848v4SiPhiiNQuE5KixSxoe3+hztpGZFTbTaVErnSf7JkfucKilqL+qp/GgElniDV2a5R5Z19XGkofcVHO1pjzEPM7/vENsHczQkw8XYe+Wx/y88YU4RBkK3lZ5UZViApBJMxLtDPn1EvLrymHl59R2qo7U0ZVGjdFIHbfCI13vlkfxtvpFJF2OLV0TSTOu1UbRdQ29oWpDFnTpakMrSyamy7U1WZLoRVM7m783JOkYrqMYeMsHqLofpJ38yQ5/5z5ZOLz3fqgmAn/QpDNB+CvHqySw5FicRyzvQLGrTIlLaOptSx71JAi62A5dbMH9BXF8ErIS8yPCrehYYcuKgn4Zp/Osb4DWqmRm9aYarbXWaD1q2+7/nY2B2Tcbl+W3hm08ubKbB/FxzywP1GKOvla/Z1a/QCs3ft0SQj4yXbj6BRWukS5d/Rpmz6pf0OltjHISkZoiROx3bymlnWsVqUQCslFbroST5foQM7W+B82KPlIlut7UdKZ6/O7hXZpUTOyv1jqLgT26ShvVC4LnEqfBakLvOMqq2pEwW/XmQgWSILl+aTlex4Gj3ulGLTvdvAvPNNPwePP+/LPNSUEje8IZaMAy0NjSmwkmclbtvKLS9f5Ek95UVPoBI9b9UCWXVBemaBLrMhSddJ9fVemQATqnRI8SbH8oYVRNqHXzZD3blX0eaSYLafWzEN8A9teH7sFOuXjjaWlmzMezltJS9xxs4DvIETuGl6X/oi3lS5vObWn06CTf94P80fBqtFOamMqR0qTl20+jWJp0XpiYfQuKRlEnVyKdTSTQCpHkr46X/owQb6tfRNKVix16eTP9N3Cs0/Q/1WD+Fw== -------------------------------------------------------------------------------- /images/src/checkpoint_tree_4.drawio: -------------------------------------------------------------------------------- 1 | 7VxLk6M2EP41rsoenAIJsOc4fiWHZHZTk6rdPSpGA8piREAe2/n1kUAYEHiNl+dMfJlFLdRC6u6vH5J3Ape74y8hCtzfqY29CdDs4wSuJgDo0NT4P4JySigza5YQnJDY8qWM8Ez+xZIoxzl7YuOo8CKj1GMkKBK31PfxlhVoKAzpofjaC/WKswbIwSXC8xZ5ZepnYjM3oc7TZQn6r5g4bjqzrsmeHUpfloTIRTY95EhwPYHLkFKWPO2OS+yJzUv3JRm3udB7/rAQ+6zOgOM0+Pjyx+pJ+8pWH4M/AWGH+RQkXF6Rt5cLlh/LTukOYN9+FBvJWz71OXFho8jFgqvOGy7befJR0D8hxnDoxxSgQU6NWEi/nTePL3uRTIHt0s5nS9HPG8Q1C9MdZuGJv3LIRJBKwM3tfkoLsYcYeS2yR1ITnDO78wyfKOETA01q7VlkUmeNuVZkEdF9uMVyVH7LrzACmsKIodDBrMSIP+SWnZFiid4gXbMsTL7pz7JJQ+ZSh/rIW2fURUj3vh1LV+Ot7J3fKA2knP/GjJ2kqaI9o0UtwEfCvuSevwpWP5uytTpKznHjlDZ8vtAv+UZulGhmw+JWNk7VTU7ZELFLcX9dTeOgEkviO3tpVWtkU1WbKxryUE/V2tIQq8L+LY8JYyav/NFh8S4mpBcaf3qmTtY/e5p2TKNYIR75C7oeHLPOlMtPC86bf6VmPE3AQk+Z8s9O+FbPlTqElBssM/6QjvkrVLlcYM7JheUpRlI0gYNLGH4O0Fb0HriXK6q73EAcMnz8vpKVlUcOgCpMzGQ7B3dGBdwZ2mV1KyjKrVqhz+7A0Rpw6PNOkMMwhkWOZFl9QYeRgw54h45MC1T/MTR0APMeUZaN1TRaiihVRn1HlOBywFDbxI0qE18HdOvy3qepfoNNKorF7YgVFShRlSX1aJhp2wvHeYWEPOIIFdtyfcGcvhBWSXga+Cg7dsS2YzdXZdNFq2/BrHVFzPpDPbMGnZn1rGu5/+9kDK2xyfiyR29Nxou7dXMQn49M8lAv++h7vP+D8T4E1cJvGkKoSWLflYKKOG6sOtKWrJMtvSrrOtWjC4hQWyma2XdVmbezXM3M5WrGPVfLLNgaWa4GB60PayM16W4KvUpAD1Whdpy3pdONAb3B2F18TR2pEws8DAr7LSTr1Rj/mEL8FHyoG773DbdKkH0usg+GtiMqqs+awe2PmmiLUbYxMJ7r4Aqg163o6VBhpMb1HXuGBKK6AAmR6wuUeLolcmsCGvnMfwKgbeK5bbQDJqr/Hjx2M4zxoMloYjejaZZ14ZBeDd56NtEU6/pJ36zYam9KrDr6Fh5e6G8+69OVrA8OjhwjikNGgxzmvG7c0TScaCa7qtPVNs2eh/QNvDVfAQkifN2urhbuuftemev5qiP3PbwRdpaaFW5ZGe9RmGodbXBhmiMqroy9tnIVYM1uYjhLuxLDdXx8YvYaw81yGGDeS/CZFozND5jgDh3tQQfsBDrUixp9n7wmy+oLOmAOOsAdOnIHOCOrAJktXNe5lguI27ZjCSE3mw1YLlsSpno4MLgwH96OH+ivnGfWBf5urtiXjpBqAn/57o6asfRcQEynbwIVWiVUoCja74jvCLSYRh4VvLC47hcJuVH+5xvGgXh2Mf9rE+SEaBf7jF3g4dqHh+/m7l/pt4WwjDy6XgE9nV3+M7srReT9yE2V3LfiR9Rfbg3vR1q4p11DmOA9ClON8AYXpnW5ZtCmMM33KEw10+8wXOfN7Ff8icvO/i8EuP4P -------------------------------------------------------------------------------- /merge/beacon-chain.md: -------------------------------------------------------------------------------- 1 | # The Merge -- The Beacon Chain 2 | 3 | **Notice**: This document is a work-in-progress for researchers and implementers. 4 | 5 | ## Table of contents 6 | 7 | 8 | 9 | 10 | 11 | - [Introduction](#introduction) 12 | - [Custom types](#custom-types) 13 | - [Constants](#constants) 14 | - [Execution](#execution) 15 | - [Configuration](#configuration) 16 | - [Transition settings](#transition-settings) 17 | - [Containers](#containers) 18 | - [Extended containers](#extended-containers) 19 | - [`BeaconBlockBody`](#beaconblockbody) 20 | - [`BeaconState`](#beaconstate) 21 | - [New containers](#new-containers) 22 | - [`ExecutionPayload`](#executionpayload) 23 | - [`ExecutionPayloadHeader`](#executionpayloadheader) 24 | - [Helper functions](#helper-functions) 25 | - [Predicates](#predicates) 26 | - [`is_merge_complete`](#is_merge_complete) 27 | - [`is_merge_block`](#is_merge_block) 28 | - [`is_execution_enabled`](#is_execution_enabled) 29 | - [Misc](#misc) 30 | - [`compute_timestamp_at_slot`](#compute_timestamp_at_slot) 31 | - [Beacon chain state transition function](#beacon-chain-state-transition-function) 32 | - [Execution engine](#execution-engine) 33 | - [`execute_payload`](#execute_payload) 34 | - [`notify_consensus_validated`](#notify_consensus_validated) 35 | - [Block processing](#block-processing) 36 | - [Execution payload processing](#execution-payload-processing) 37 | - [`is_valid_gas_limit`](#is_valid_gas_limit) 38 | - [`process_execution_payload`](#process_execution_payload) 39 | - [Testing](#testing) 40 | 41 | 42 | 43 | 44 | ## Introduction 45 | 46 | The Merge is the event in which the Ethereum proof of work chain is deprecated, and the beacon chain (the Ethereum proof of stake chain) takes over as the chain that Ethereum is running on. To simplify the Merge and allow it to happen faster, the Merge is designed via a **block-inside-a-block structure**: the Ethereum PoW chain appears to continue, except past a certain transition point (i) the PoW nonces are no longer required to be valid, and (ii) the Ethereum PoW blocks, from then on referred to as **execution blocks**, are required to be embedded inside of beacon chain blocks. 47 | 48 | ![](https://i.imgur.com/udZlkQr.png) 49 | 50 | The timing of the merge is parametrized by two thresholds: 51 | 52 | * The `MERGE_FORK_EPOCH`, the epoch at which the beacon chain includes the post-merge features (mainly, the ability to contain execution blocks), and so the merge becomes possible 53 | * The `TERMINAL_TOTAL_DIFFICULTY`, the total PoW difficulty such that a block those total difficulty is `>= TERMINAL_TOTAL_DIFFICULTY` (but whose parent is below the threshold) is a terminal block, so its children and further descendants do not require PoW and must be part of beacon chain blocks. 54 | 55 | Total difficulty is used as a threshold instead of block height (which forks normally use) for security: it prevents an attacker from privately creating a chain with lower difficulty but higher block height and then colluding with a single beacon proposer to get it included first. Using TD ensures that the winner of the PoW fork choice becomes eligible for inclusion into the beacon chain before any other block. 56 | 57 | The parameters are expected to be set such that the beacon chain reaches the `MERGE_FORK_EPOCH` before the PoW chain reaches the `TERMINAL_TOTAL_DIFFICULTY`. Hence, by the time the PoW chain gets a terminal block, which can only have an **embedded block** inside the beacon chain as a child, the beacon chain will be ready to accept embedded blocks. In the unlikely case that the events end up happening in the other order (only possible if miners attack the transition by adding a _huge_ amount of new hashpower), Ethereum will simply be offline and not generate new blocks until the `MERGE_FORK_EPOCH` is reached. 58 | 59 | ## Custom types 60 | 61 | *Note*: The `Transaction` type is a stub which is not final. 62 | 63 | | Name | SSZ equivalent | Description | 64 | | - | - | - | 65 | | `OpaqueTransaction` | `ByteList[MAX_BYTES_PER_OPAQUE_TRANSACTION]` | a [typed transaction envelope](https://eips.ethereum.org/EIPS/eip-2718#opaque-byte-array-rather-than-an-rlp-array) structured as `TransactionType \|\| TransactionPayload` | 66 | | `Transaction` | `Union[OpaqueTransaction]` | a transaction | 67 | | `ExecutionAddress` | `Bytes20` | Address of account on the execution layer | 68 | 69 | The beacon chain uses [SSZ](https://github.com/ethereum/consensus-specs/blob/dev/ssz/simple-serialize.md) for serialization and SSZ + SHA256 for hashing, and the execution chain uses [RLP](https://eth.wiki/fundamentals/rlp) for serialization, and the keccak hash of the RLP serialized form for hashing. In the long term, it's a desired goal to move everything to SSZ. For now, however, because of the desire to finish the merge quickly, most of the execution chain remains RLP-based. Specifically, execution _blocks_ are stored in SSZ form, but the _transactions_ inside them are encoded with RLP, and so to software that only understands SSZ they are presented as "opaque" byte arrays. Note also that the `parent_hash` of an execution block is expected to match the keccak hash of the RLP-serialized form of the previous execution block. 70 | 71 | ## Constants 72 | 73 | ### Execution 74 | 75 | | Name | Value | 76 | | - | - | 77 | | `MAX_BYTES_PER_OPAQUE_TRANSACTION` | `uint64(2**20)` (= 1,048,576) | 78 | | `MAX_TRANSACTIONS_PER_PAYLOAD` | `uint64(2**14)` (= 16,384) | 79 | | `BYTES_PER_LOGS_BLOOM` | `uint64(2**8)` (= 256) | 80 | | `GAS_LIMIT_DENOMINATOR` | `uint64(2**10)` (= 1,024) | 81 | | `MIN_GAS_LIMIT` | `uint64(5000)` (= 5,000) | 82 | | `MAX_EXTRA_DATA_BYTES` | `2**5` (= 32) | 83 | 84 | The `GAS_LIMIT_DENOMINATOR` is the inverse of the max fraction by which a block's gas limit can change per block. This is identical to the gas limit voting mechanism on the execution chain. The max bytes per transaction and max transactions per payload are expected to be changed; the intent is for the limits to merely be there because the SSZ structures require _some_ max depth, and the execution chain gas limit is intended to set the actual limits. 85 | 86 | ## Configuration 87 | 88 | ### Transition settings 89 | 90 | | Name | Value | 91 | | - | - | 92 | | `TERMINAL_TOTAL_DIFFICULTY` | **TBD** | 93 | | `TERMINAL_BLOCK_HASH` | `Hash32('0x0000000000000000000000000000000000000000000000000000000000000000')` | 94 | 95 | The `TERMINAL_BLOCK_HASH` feature is an emergency override mechanism. If PoW miners attempt to cause chaos on the chain via 51% attacks before the merge, the community can react by setting the `TERMINAL_BLOCK_HASH` to choose a specific terminal block, and the `MERGE_FORK_EPOCH` to some specific value in the future (ideally not too far in the future). The Ethereum chain would make no progress between this point and the `MERGE_FORK_EPOCH`, but after that point it would proceed smoothly embedded inside the beacon chain. 96 | 97 | Note that if the `TERMINAL_BLOCK_HASH` is set to any value that is clearly not a valid hash output (eg. zero), this means that there is no valid block that could trigger the hash-based validity condition, so a block reaching the `TERMINAL_TOTAL_DIFFICULTY` is the only option. 98 | 99 | ## Containers 100 | 101 | ### Extended containers 102 | 103 | #### `BeaconBlockBody` 104 | 105 | ```python 106 | class BeaconBlockBody(Container): 107 | randao_reveal: BLSSignature 108 | eth1_data: Eth1Data # Eth1 data vote 109 | graffiti: Bytes32 # Arbitrary data 110 | # Operations 111 | proposer_slashings: List[ProposerSlashing, MAX_PROPOSER_SLASHINGS] 112 | attester_slashings: List[AttesterSlashing, MAX_ATTESTER_SLASHINGS] 113 | attestations: List[Attestation, MAX_ATTESTATIONS] 114 | deposits: List[Deposit, MAX_DEPOSITS] 115 | voluntary_exits: List[SignedVoluntaryExit, MAX_VOLUNTARY_EXITS] 116 | sync_aggregate: SyncAggregate 117 | # Execution 118 | execution_payload: ExecutionPayload # [New in Merge] 119 | ``` 120 | 121 | Extend a beacon block with the `ExecutionPayload` object (the embedded execution block). 122 | 123 | #### `BeaconState` 124 | 125 | ```python 126 | class BeaconState(Container): 127 | # Versioning 128 | genesis_time: uint64 129 | genesis_validators_root: Root 130 | slot: Slot 131 | fork: Fork 132 | # History 133 | latest_block_header: BeaconBlockHeader 134 | block_roots: Vector[Root, SLOTS_PER_HISTORICAL_ROOT] 135 | state_roots: Vector[Root, SLOTS_PER_HISTORICAL_ROOT] 136 | historical_roots: List[Root, HISTORICAL_ROOTS_LIMIT] 137 | # Eth1 138 | eth1_data: Eth1Data 139 | eth1_data_votes: List[Eth1Data, EPOCHS_PER_ETH1_VOTING_PERIOD * SLOTS_PER_EPOCH] 140 | eth1_deposit_index: uint64 141 | # Registry 142 | validators: List[Validator, VALIDATOR_REGISTRY_LIMIT] 143 | balances: List[Gwei, VALIDATOR_REGISTRY_LIMIT] 144 | # Randomness 145 | randao_mixes: Vector[Bytes32, EPOCHS_PER_HISTORICAL_VECTOR] 146 | # Slashings 147 | slashings: Vector[Gwei, EPOCHS_PER_SLASHINGS_VECTOR] # Per-epoch sums of slashed effective balances 148 | # Participation 149 | previous_epoch_participation: List[ParticipationFlags, VALIDATOR_REGISTRY_LIMIT] 150 | current_epoch_participation: List[ParticipationFlags, VALIDATOR_REGISTRY_LIMIT] 151 | # Finality 152 | justification_bits: Bitvector[JUSTIFICATION_BITS_LENGTH] # Bit set for every recent justified epoch 153 | previous_justified_checkpoint: Checkpoint 154 | current_justified_checkpoint: Checkpoint 155 | finalized_checkpoint: Checkpoint 156 | # Inactivity 157 | inactivity_scores: List[uint64, VALIDATOR_REGISTRY_LIMIT] 158 | # Sync 159 | current_sync_committee: SyncCommittee 160 | next_sync_committee: SyncCommittee 161 | # Execution 162 | latest_execution_payload_header: ExecutionPayloadHeader # [New in Merge] 163 | ``` 164 | 165 | Extend the `BeaconState` by adding the `ExecutionPayloadHeader` of the most recently included execution block. This contains data like execution block hash, post-state root, gas limit, etc, that the next execution block will need to be checked against. 166 | 167 | ### New containers 168 | 169 | #### `ExecutionPayload` 170 | 171 | *Note*: The `base_fee_per_gas` field is serialized in little-endian. 172 | 173 | ```python 174 | class ExecutionPayload(Container): 175 | # Execution block header fields 176 | parent_hash: Hash32 177 | coinbase: ExecutionAddress # 'beneficiary' in the yellow paper 178 | state_root: Bytes32 179 | receipt_root: Bytes32 # 'receipts root' in the yellow paper 180 | logs_bloom: ByteVector[BYTES_PER_LOGS_BLOOM] 181 | random: Bytes32 # 'difficulty' in the yellow paper 182 | block_number: uint64 # 'number' in the yellow paper 183 | gas_limit: uint64 184 | gas_used: uint64 185 | timestamp: uint64 186 | extra_data: ByteList[MAX_EXTRA_DATA_BYTES] 187 | base_fee_per_gas: Bytes32 # base fee introduced in EIP-1559, little-endian serialized 188 | # Extra payload fields 189 | block_hash: Hash32 # Hash of execution block 190 | transactions: List[Transaction, MAX_TRANSACTIONS_PER_PAYLOAD] 191 | ``` 192 | 193 | The data structure that stores the execution block. Note that the execution block hash is part of the structure and not computed from it. Verification of this hash is done by execution clients (formerly known as "eth1 clients"; eg. Geth, Nethermind, Erigon, Besu) when the execution block is passed to them for verification. 194 | 195 | #### `ExecutionPayloadHeader` 196 | 197 | ```python 198 | class ExecutionPayloadHeader(Container): 199 | # Execution block header fields 200 | parent_hash: Hash32 201 | coinbase: ExecutionAddress 202 | state_root: Bytes32 203 | receipt_root: Bytes32 204 | logs_bloom: ByteVector[BYTES_PER_LOGS_BLOOM] 205 | random: Bytes32 206 | block_number: uint64 207 | gas_limit: uint64 208 | gas_used: uint64 209 | timestamp: uint64 210 | extra_data: ByteList[MAX_EXTRA_DATA_BYTES] 211 | base_fee_per_gas: Bytes32 212 | # Extra payload fields 213 | block_hash: Hash32 # Hash of execution block 214 | transactions_root: Root 215 | ``` 216 | 217 | Same structure as above, except the transaction list is replaced by its SSZ hash tree root. 218 | 219 | ## Helper functions 220 | 221 | ### Predicates 222 | 223 | #### `is_merge_complete` 224 | 225 | ```python 226 | def is_merge_complete(state: BeaconState) -> bool: 227 | return state.latest_execution_payload_header != ExecutionPayloadHeader() 228 | ``` 229 | 230 | Returns whether or not the merge "has already happened", defined by whether or not there has already been an embedded execution block inside the beacon chain (if there has, its header is saved in `state.latest_execution_payload_header`). 231 | 232 | #### `is_merge_block` 233 | 234 | ```python 235 | def is_merge_block(state: BeaconState, body: BeaconBlockBody) -> bool: 236 | return not is_merge_complete(state) and body.execution_payload != ExecutionPayload() 237 | ``` 238 | 239 | Returns whether or not a given block is the first block that contains an embedded execution block. 240 | 241 | #### `is_execution_enabled` 242 | 243 | ```python 244 | def is_execution_enabled(state: BeaconState, body: BeaconBlockBody) -> bool: 245 | return is_merge_block(state, body) or is_merge_complete(state) 246 | ``` 247 | 248 | ### Misc 249 | 250 | #### `compute_timestamp_at_slot` 251 | 252 | *Note*: This function is unsafe with respect to overflows and underflows. 253 | 254 | ```python 255 | def compute_timestamp_at_slot(state: BeaconState, slot: Slot) -> uint64: 256 | slots_since_genesis = slot - GENESIS_SLOT 257 | return uint64(state.genesis_time + slots_since_genesis * SECONDS_PER_SLOT) 258 | ``` 259 | 260 | ## Beacon chain state transition function 261 | 262 | ### Execution engine 263 | 264 | This function calls the execution client (formerly known as an "eth1 client"; eg. Geth, Nethermind, Erigon, Besu) to validate the execution block in the `execution_payload`. The payload contains the `parent_hash`, so the execution engine knows what pre-state to verify the execution block off of. Note that there is implied state (the execution pre-state) that is stored by the execution client and that the execution client is expected to use to validate the block. The function can be considered quasi-pure in that although it does not pass the pre-state in as an explicit argument, the `execution_payload.parent_hash` is nevertheless cryptographically linked to a unique pre-state. 265 | 266 | Under normal operation, there is no possibility that the pre-state is unavailable, because a beacon block would only be validated if all of its ancestors have been validated, and that ensures that the previous execution blocks were validated and their post-state (and hence the latest block's pre-state) would already be computed. The only exception to this logic is the _first_ embedded execution block, whose parent (the terminal PoW block) is not in the beacon chain. If the terminal PoW block is unavailable, the beacon chain client should wait until the terminal PoW block becomes available before accepting the beacon block . Importantly, the beacon chain client should NOT "conclusively reject" a beacon block whose embedded execution block's parent is an unavailable terminal PoW block, because the terminal PoW block could become available very soon in the future. 267 | 268 | #### `execute_payload` 269 | 270 | ```python 271 | def execute_payload(self: ExecutionEngine, execution_payload: ExecutionPayload) -> bool: 272 | """ 273 | Return ``True`` if and only if ``execution_payload`` is valid with respect to ``self.execution_state``. 274 | """ 275 | ... 276 | ``` 277 | 278 | ### Block processing 279 | 280 | *Note*: The call to the `process_execution_payload` must happen before the call to the `process_randao` as the former depends on the `randao_mix` computed with the reveal of the previous block. 281 | 282 | ```python 283 | def process_block(state: BeaconState, block: BeaconBlock) -> None: 284 | process_block_header(state, block) 285 | if is_execution_enabled(state, block.body): 286 | process_execution_payload(state, block.body.execution_payload, EXECUTION_ENGINE) # [New in Merge] 287 | process_randao(state, block.body) 288 | process_eth1_data(state, block.body) 289 | process_operations(state, block.body) 290 | process_sync_aggregate(state, block.body.sync_aggregate) 291 | ``` 292 | 293 | The main addition to the `process_block` function is the `process_execution_payload` function. This function is computed against the previous `randao` because we want it to be possible to construct an execution block without having secret information only available to the proposer (both to minimize cross-client communication and to allow proposer/builder separation in the future). 294 | 295 | Note that `process_execution_payload` is NOT the same as `execute_payload` defined above; rather, it's a function that calls `execute_payload` but also does a few other local validations. 296 | 297 | ### Execution payload processing 298 | 299 | #### `is_valid_gas_limit` 300 | 301 | ```python 302 | def is_valid_gas_limit(payload: ExecutionPayload, parent: ExecutionPayloadHeader) -> bool: 303 | parent_gas_limit = parent.gas_limit 304 | 305 | # Check if the payload used too much gas 306 | if payload.gas_used > payload.gas_limit: 307 | return False 308 | 309 | # Check if the payload changed the gas limit too much 310 | if payload.gas_limit >= parent_gas_limit + parent_gas_limit // GAS_LIMIT_DENOMINATOR: 311 | return False 312 | if payload.gas_limit <= parent_gas_limit - parent_gas_limit // GAS_LIMIT_DENOMINATOR: 313 | return False 314 | 315 | # Check if the gas limit is at least the minimum gas limit 316 | if payload.gas_limit < MIN_GAS_LIMIT: 317 | return False 318 | 319 | return True 320 | ``` 321 | 322 | Enforces the same gas limit checking as the Ethereum PoW chain does today. 323 | 324 | #### `process_execution_payload` 325 | 326 | ```python 327 | def process_execution_payload(state: BeaconState, payload: ExecutionPayload, execution_engine: ExecutionEngine) -> None: 328 | # Verify consistency of the parent hash, block number, base fee per gas and gas limit 329 | # with respect to the previous execution payload header 330 | if is_merge_complete(state): 331 | assert payload.parent_hash == state.latest_execution_payload_header.block_hash 332 | assert payload.block_number == state.latest_execution_payload_header.block_number + uint64(1) 333 | assert is_valid_gas_limit(payload, state.latest_execution_payload_header) 334 | # Verify random 335 | assert payload.random == get_randao_mix(state, get_current_epoch(state)) 336 | # Verify timestamp 337 | assert payload.timestamp == compute_timestamp_at_slot(state, state.slot) 338 | # Verify the execution payload is valid 339 | assert execution_engine.execute_payload(payload) 340 | # Cache execution payload header 341 | state.latest_execution_payload_header = ExecutionPayloadHeader( 342 | parent_hash=payload.parent_hash, 343 | coinbase=payload.coinbase, 344 | state_root=payload.state_root, 345 | receipt_root=payload.receipt_root, 346 | logs_bloom=payload.logs_bloom, 347 | random=payload.random, 348 | block_number=payload.block_number, 349 | gas_limit=payload.gas_limit, 350 | gas_used=payload.gas_used, 351 | timestamp=payload.timestamp, 352 | extra_data=payload.extra_data, 353 | base_fee_per_gas=payload.base_fee_per_gas, 354 | block_hash=payload.block_hash, 355 | transactions_root=hash_tree_root(payload.transactions), 356 | ) 357 | ``` 358 | 359 | Verifies the consistency of the current execution block against the previous execution header stored in the beacon state. The validations that can be done using beacon chain state information alone are done here; everything "deep" is done by execution client validation. 360 | -------------------------------------------------------------------------------- /merge/fork-choice.md: -------------------------------------------------------------------------------- 1 | # The Merge -- Fork Choice 2 | 3 | **Notice**: This document is a work-in-progress for researchers and implementers. 4 | 5 | ## Table of contents 6 | 7 | 8 | 9 | 10 | - [Introduction](#introduction) 11 | - [Protocols](#protocols) 12 | - [`ExecutionEngine`](#executionengine) 13 | - [`notify_forkchoice_updated`](#notify_forkchoice_updated) 14 | - [Helpers](#helpers) 15 | - [`PowBlock`](#powblock) 16 | - [`get_pow_block`](#get_pow_block) 17 | - [`is_valid_terminal_pow_block`](#is_valid_terminal_pow_block) 18 | - [Updated fork-choice handlers](#updated-fork-choice-handlers) 19 | - [`on_block`](#on_block) 20 | 21 | 22 | 23 | 24 | ## Introduction 25 | 26 | This is the modification of the beacon chain fork choice for the merge. There is only one significant change made: the additional requirement to verify that the terminal PoW block is valid. 27 | 28 | ## Protocols 29 | 30 | ### `ExecutionEngine` 31 | 32 | #### `notify_forkchoice_updated` 33 | 34 | When a beacon chain client reorgs, it should notify the execution client of the new head, so that the execution client returns the head and the state correctly in its APIs. 35 | 36 | ```python 37 | def notify_forkchoice_updated(self: ExecutionEngine, head_block_hash: Hash32, finalized_block_hash: Hash32) -> None: 38 | ... 39 | ``` 40 | 41 | ## Helpers 42 | 43 | This section provides the method for computing whether or not a PoW execution block referenced as a parent of the first embedded block is a valid terminal block. In addition for simply checking that the terminal PoW block is available and has been validated by the execution client, which the [post-merge beacon chain spec](./beacon-chain.md) already does implicitly, we need to check that it's a _valid_ terminal PoW block. 44 | 45 | ### `PowBlock` 46 | 47 | ```python 48 | class PowBlock(Container): 49 | block_hash: Hash32 50 | parent_hash: Hash32 51 | total_difficulty: uint256 52 | difficulty: uint256 53 | ``` 54 | 55 | ### `get_pow_block` 56 | 57 | Let `get_pow_block(block_hash: Hash32) -> PowBlock` be the function that given the hash of the PoW block returns its data. 58 | 59 | ### `is_valid_terminal_pow_block` 60 | 61 | There are two ways in which a block can be a valid terminal PoW block. First (and this is the normal case), it could be a PoW block that reaches the `TERMINAL_TOTAL_DIFFICULTY`. Note that only blocks _immediately_ past the `TERMINAL_TOTAL_DIFFICULTY` threshold (so, blocks whose parents are still below it) are allowed; this is part of a general rule that terminal PoW blocks can only have embedded execution blocks as valid children. Second (and this is an exceptional case to be configured only in the event of an attack or other emergency), the terminal PoW block can be chosen explicitly via its hash. 62 | 63 | ```python 64 | def is_valid_terminal_pow_block(block: PowBlock, parent: PowBlock) -> bool: 65 | if block.block_hash == TERMINAL_BLOCK_HASH: 66 | return True 67 | 68 | is_total_difficulty_reached = block.total_difficulty >= TERMINAL_TOTAL_DIFFICULTY 69 | is_parent_total_difficulty_valid = parent.total_difficulty < TERMINAL_TOTAL_DIFFICULTY 70 | return is_total_difficulty_reached and is_parent_total_difficulty_valid 71 | ``` 72 | 73 | ## Updated fork-choice handlers 74 | 75 | ### `on_block` 76 | 77 | *Note*: The only modification is the addition of the verification of transition block conditions. 78 | 79 | ```python 80 | def on_block(store: Store, signed_block: SignedBeaconBlock) -> None: 81 | block = signed_block.message 82 | # Parent block must be known 83 | assert block.parent_root in store.block_states 84 | # Make a copy of the state to avoid mutability issues 85 | pre_state = copy(store.block_states[block.parent_root]) 86 | # Blocks cannot be in the future. If they are, their consideration must be delayed until the are in the past. 87 | assert get_current_slot(store) >= block.slot 88 | 89 | # Check that block is later than the finalized epoch slot (optimization to reduce calls to get_ancestor) 90 | finalized_slot = compute_start_slot_at_epoch(store.finalized_checkpoint.epoch) 91 | assert block.slot > finalized_slot 92 | # Check block is a descendant of the finalized block at the checkpoint finalized slot 93 | assert get_ancestor(store, block.parent_root, finalized_slot) == store.finalized_checkpoint.root 94 | 95 | # Check the block is valid and compute the post-state 96 | state = pre_state.copy() 97 | state_transition(state, signed_block, True) 98 | 99 | # [New in Merge] 100 | if is_merge_block(pre_state, block.body): 101 | pow_block = get_pow_block(block.body.execution_payload.parent_hash) 102 | pow_parent = get_pow_block(pow_block.parent_hash) 103 | assert is_valid_terminal_pow_block(pow_block, pow_parent) 104 | 105 | # Add new block to the store 106 | store.blocks[hash_tree_root(block)] = block 107 | # Add new state for this block to the store 108 | store.block_states[hash_tree_root(block)] = state 109 | 110 | # Update justified checkpoint 111 | if state.current_justified_checkpoint.epoch > store.justified_checkpoint.epoch: 112 | if state.current_justified_checkpoint.epoch > store.best_justified_checkpoint.epoch: 113 | store.best_justified_checkpoint = state.current_justified_checkpoint 114 | if should_update_justified_checkpoint(store, state.current_justified_checkpoint): 115 | store.justified_checkpoint = state.current_justified_checkpoint 116 | 117 | # Update finalized checkpoint 118 | if state.finalized_checkpoint.epoch > store.finalized_checkpoint.epoch: 119 | store.finalized_checkpoint = state.finalized_checkpoint 120 | 121 | # Potentially update justified if different from store 122 | if store.justified_checkpoint != state.current_justified_checkpoint: 123 | # Update justified if new justified is later than store justified 124 | if state.current_justified_checkpoint.epoch > store.justified_checkpoint.epoch: 125 | store.justified_checkpoint = state.current_justified_checkpoint 126 | return 127 | 128 | # Update justified if store justified is not in chain with finalized checkpoint 129 | finalized_slot = compute_start_slot_at_epoch(store.finalized_checkpoint.epoch) 130 | ancestor_at_finalized_slot = get_ancestor(store, store.justified_checkpoint.root, finalized_slot) 131 | if ancestor_at_finalized_slot != store.finalized_checkpoint.root: 132 | store.justified_checkpoint = state.current_justified_checkpoint 133 | ``` 134 | -------------------------------------------------------------------------------- /phase0/fork-choice.md: -------------------------------------------------------------------------------- 1 | # Ethereum 2.0 Phase 0 -- Beacon Chain Fork Choice 2 | 3 | **Notice**: This document was written in Aug 2020. 4 | 5 | ## Table of contents 6 | 7 | 8 | 9 | 10 | 11 | - [Introduction](#introduction) 12 | - [Fork choice](#fork-choice) 13 | - [Configuration](#configuration) 14 | - [Helpers](#helpers) 15 | - [`LatestMessage`](#latestmessage) 16 | - [`Store`](#store) 17 | - [`get_forkchoice_store`](#get_forkchoice_store) 18 | - [`get_slots_since_genesis`](#get_slots_since_genesis) 19 | - [`get_current_slot`](#get_current_slot) 20 | - [`compute_slots_since_epoch_start`](#compute_slots_since_epoch_start) 21 | - [`get_ancestor`](#get_ancestor) 22 | - [`get_latest_attesting_balance`](#get_latest_attesting_balance) 23 | - [`filter_block_tree`](#filter_block_tree) 24 | - [`get_filtered_block_tree`](#get_filtered_block_tree) 25 | - [`get_head`](#get_head) 26 | - [`should_update_justified_checkpoint`](#should_update_justified_checkpoint) 27 | - [`on_attestation` helpers](#on_attestation-helpers) 28 | - [`validate_on_attestation`](#validate_on_attestation) 29 | - [`store_target_checkpoint_state`](#store_target_checkpoint_state) 30 | - [`update_latest_messages`](#update_latest_messages) 31 | - [Handlers](#handlers) 32 | - [`on_tick`](#on_tick) 33 | - [`on_block`](#on_block) 34 | - [`on_attestation`](#on_attestation) 35 | 36 | 37 | 38 | 39 | ## Introduction 40 | 41 | This document is the beacon chain **fork choice rule** specification, part of Ethereum 2.0 Phase 0. It describes the mechanism for how to choose what is the "canonical" chain in the event that there are multiple conflicting versions of the chain to choose from. All blockchains have the possibility of temporary disagreement, either because of malicious behavior (eg. a block proposer publishing two different blocks at the same time), or just network latency (eg. a block being delayed by a few seconds, causing it to be broadcasted around the same time as the _next_ block that gets published by someone else). In such cases, some mechanism is needed to choose which of the two (or more) chains represents the "actual" history and state of the system (this chain is called the **canonical chain**). 42 | 43 | PoW chains (Bitcoin, Ethereum 1.0, etc) typically use some variant of the **longest chain rule**: if there is a disagreement between chains, pick the chain that is the longest. 44 | 45 | ![](https://vitalik.ca/files/posts_files/cbc-casper-files/Chain3.png) 46 | 47 | In practice, "[total difficulty](https://ethereum.stackexchange.com/questions/7068/difficulty-and-total-difficulty)" is used instead of length, though for most intuitive analysis, thinking about length is typically sufficient. PoS chains can also use the longest chain rule; however, PoS opens the door to much better fork choice rules that provide much stronger properties than the longest chain rule. 48 | 49 | The fork choice rule can be viewed as being part of the **consensus algorithm**, the other part of the consensus algorithm being the rules that determine what messages each participant should send to the network (see the [honest validator doc](https://github.com/ethereum/eth2.0-specs/blob/dev/specs/phase0/validator.md#beacon-chain-responsibilities) for more info). In eth2, the consensus algorithm is Casper FFG + LMD GHOST, sometimes called [Gasper](https://arxiv.org/abs/2003.03052). 50 | 51 | * See here for the original paper describing Casper FFG: https://arxiv.org/abs/1710.09437 52 | * See here for a description of LMD GHOST (ignore the section on detecting finality, as that is specific to Casper CBC, whereas the fork choice rule is shared between CBC and FFG): https://vitalik.ca/general/2018/12/05/cbc_casper.html 53 | 54 | The goal of Casper FFG and LMD GHOST is to combine together the benefits of two major types of PoS design: **longest-chain-based** (or Nakamoto-based), as used by Peercoin, NXT, [Ouroboros](https://cardano.org/ouroboros/) and many other designs, and **traditional BFT based**, as used by [Tendermint](https://tendermint.com/docs/introduction/what-is-tendermint.html) and others. Longest chain systems have the benefit that they are low-overhead, even with a very high number of participants. Traditional BFT systems have the key benefit that they have a concept of **finality**: once a block is **finalized**, it can no longer be reverted, no matter what other participants in the network do. If >1/3 of participants in the network behave maliciously, different nodes could be tricked into accepting different conflicting blocks as finalized. However, to cause such a situation, those participants would have to misbehave in a very unambiguous and provable way, allowing them to be **slashed** (all their coins are taken away as a penalty), making attacks extremely expensive. Additionally, traditional BFT systems reach finality quickly, though at the cost of high overhead and a low maximum participant count. 55 | 56 | Eth2 combines together the advantages of both. It includes a traditional-BFT-inspired finality system (Casper FFG), though running at a rate of ~13 minutes per cycle (note: time to finality = 2 epochs = 2 * 32 * 12 sec) instead of a few seconds per cycle, allowing potentially over a million participants. To progress the chain between these epochs, LMD GHOST is used. 57 | 58 | ![](https://vitalik.ca/files/posts_files/cbc-casper-files/Chain7.png) 59 | 60 | The core idea of LMD GHOST is that at each fork, instead of choosing the side that contains a longer chain, we choose the side that has more total support from validators, counting only the most recent message of each validator as support. This is heavily inspired by [Zohar and Sompolinsky's original GHOST paper](https://eprint.iacr.org/2013/881), but it adapts the design from its original PoW context to our new PoS context. LMD GHOST is powerful because it easily generalizes to much more than one "vote" being sent in parallel. 61 | 62 | This is a very valuable feature for us because Casper FFG [already requires every validator](https://github.com/ethereum/annotated-spec/blob/master/phase0/beacon-chain.md#how-does-eth2-proof-of-stake-work) to send one attestation per epoch, meaning that hundreds of attestations are already being sent every second. We piggyback on those messages and ask them to include additional information voting on the current head of the chain. The result of this is that when a block is published, within seconds there are hundreds of signed messages from validators (**attestations**) confirming the block, and after even one slot (12 seconds) it's very difficult to revert a block. Anyone seeking even stronger ironclad confirmation can simply wait for finality after two epochs (~12 minutes). 63 | 64 | The approximate approach that we take to combining Casper FFG and LMD GHOST is: 65 | 66 | 1. Use the Casper FFG rules to compute finalized blocks. If a block becomes finalized, all future canonical chains must pass through this block. 67 | 2. Use the Casper FFG rules to keep track of the **latest justified checkpoint** (LJC) that is a descendant of the latest accepted finalized checkpoint. 68 | 3. Use the LMD GHOST rules, starting from the LJC as root, to compute the chain head. 69 | 70 | This combination of steps, particularly rule (2), is implemented to ensure that new blocks that validators create by following the rules actually will continually finalize new blocks with Casper FFG, even if temporary exceptional situations (eg. involving attacks or extremely high network latency) take place. 71 | 72 | ### Checkpoints vs blocks 73 | 74 | Note also that Casper FFG deals with **checkpoints** and the **checkpoint tree**, whereas LMD GHOST deals with blocks and the **block tree**. One simple way to think about this is that the checkpoint tree is a compressed version of the block tree, where we only consider blocks at the start of an epoch, and where checkpoint A is a parent of checkpoint B if block A is the ancestor of B in the block tree that begins the epoch before B: 75 | 76 | ![](../images/checkpoint_tree_1.png) 77 | 78 | _Green blocks are the checkpoints; in the checkpoint tree, A is the parent of B1 and B2._ 79 | 80 | But to approach a more realistic picture we need to account for an important scenario: when there are skipped slots. If a chain has a skipped block at some slot, we take the most recent block before that slot in that chain as the epoch boundary block: 81 | 82 | ![](../images/checkpoint_tree_2.png) 83 | 84 | In any case, where there is a divergence between multiple chains, skipped slots will be frequent because proposer selection and slashing rules forbid multiple blocks from being created in the same slot (unless two chains diverge for more than four epochs)! Note also that if we want to compute the _post-state_ of a checkpoint, we would have to run [the `process_slots` function](https://github.com/ethereum/annotated-spec/blob/master/phase0/beacon-chain.md#state-transition) up to the first slot of the epoch to process the empty slots. 85 | 86 | Another important edge case is when there is more than an entire epoch of skipped slots. To cover this edge case, we think about a checkpoint as a (block, slot) pair, not a blocks; this means that the same block could be part of multiple checkpoints that an attestation could validly reference: 87 | 88 | ![](../images/checkpoint_tree_3.png) 89 | 90 | In this case, the block B2 corresponds to two checkpoints, `(B2, N)` and `(B2, N+1)`, one for each epoch. In general, the most helpful model for understanding Ethereum state transitions may actually be the full state transition tree, where we look at all possible `(block, slot)` pairs: 91 | 92 | ![](../images/checkpoint_tree_4.png) 93 | 94 | If you're a mathematician, you could view both the block tree and the checkpoint tree as [graph minors](https://en.wikipedia.org/wiki/Graph_minor) of the state transition tree. Casper FFG works over the checkpoint tree, and LMD GHOST works over the block tree. For convenience, we'll use the phrase "**latest justified block**" (LJB) to refer to the block referenced in the latest justified checkpoint; because LMD GHOST works over the block tree we'll keep the discussion to blocks, though the actual data structures will talk about the latest justified checkpoint. Note that while one block may map to multiple checkpoints, a checkpoint only references one block, so this language is unambiguous. 95 | 96 | Now, let's go through the specification... 97 | 98 | ## Fork choice 99 | 100 | One important thing to note is that the fork choice _is not a pure function_; that is, what you accept as a canonical chain does not depend just on what data you also have, but also when you received it. The main reason this is done is to enforce finality: if you accept a block as finalized, then you will never revert it, even if you later see a conflicting block as finalized. Such a situation would only happen in cases where there is an active >1/3 attack on the chain; in such cases, we expect extra-protocol measures to be required to get all clients back on the same chain. There are also other deviations from purity, particularly a "sticky" choice of the latest justified block, where the latest justified block can only change near the beginning of an epoch; this is done to prevent certain kinds of "bouncing attacks". 101 | 102 | We implement this fork choice by defining a `store` that contains received fork-choice-relevant information, as well as some "memory variables", and a function `get_head(store)`. 103 | 104 | At genesis, let `store = get_forkchoice_store(genesis_state)` and update `store` by running: 105 | 106 | - `on_tick(store, time)` whenever `time > store.time` where `time` is the current Unix time 107 | - `on_block(store, block)` whenever a block `block: SignedBeaconBlock` is received 108 | - `on_attestation(store, attestation)` whenever an attestation `attestation` is received 109 | 110 | Any of the above handlers that trigger an unhandled exception (e.g. a failed assert or an out-of-range list access) are considered invalid. Invalid calls to handlers must not modify `store`. 111 | 112 | *Notes*: 113 | 114 | 1) **Leap seconds**: Slots will last `SECONDS_PER_SLOT + 1` or `SECONDS_PER_SLOT - 1` seconds around leap seconds. This is automatically handled by [UNIX time](https://en.wikipedia.org/wiki/Unix_time). 115 | 2) **Honest clocks**: Honest nodes are assumed to have clocks synchronized within `SECONDS_PER_SLOT` seconds of each other. 116 | 3) **Eth1 data**: The large `ETH1_FOLLOW_DISTANCE` specified in the [honest validator document](https://github.com/ethereum/eth2.0-specs/blob/dev/specs/phase0/validator.md) should ensure that `state.latest_eth1_data` of the canonical Ethereum 2.0 chain remains consistent with the canonical Ethereum 1.0 chain. If not, emergency manual intervention will be required. 117 | 4) **Manual forks**: Manual forks may arbitrarily change the fork choice rule but are expected to be enacted at epoch transitions, with the fork details reflected in `state.fork`. 118 | 5) **Implementation**: The implementation found in this specification is constructed for ease of understanding rather than for optimization in computation, space, or any other resource. A number of optimized alternatives can be found [here](https://github.com/protolambda/lmd-ghost). 119 | 120 | ### Configuration 121 | 122 | | Name | Value | Unit | Duration | 123 | | - | - | :-: | :-: | 124 | | `SAFE_SLOTS_TO_UPDATE_JUSTIFIED` | `2**3` (= 8) | slots | 96 seconds | 125 | 126 | The justified checkpoint can only be changed in the first 8 slots of an epoch; see below for reasoning why this is done. 127 | 128 | ### Helpers 129 | 130 | Here, we define the data structure for the `store`. It only has one new subtype, the `LatestMessage` (the vote in the latest [meaning highest-epoch] valid attestation received from a validator). 131 | 132 | #### `LatestMessage` 133 | 134 | ```python 135 | @dataclass(eq=True, frozen=True) 136 | class LatestMessage(object): 137 | epoch: Epoch 138 | root: Root 139 | ``` 140 | 141 | #### `Store` 142 | 143 | ```python 144 | @dataclass 145 | class Store(object): 146 | time: uint64 147 | genesis_time: uint64 148 | justified_checkpoint: Checkpoint 149 | finalized_checkpoint: Checkpoint 150 | best_justified_checkpoint: Checkpoint 151 | blocks: Dict[Root, BeaconBlock] = field(default_factory=dict) 152 | block_states: Dict[Root, BeaconState] = field(default_factory=dict) 153 | checkpoint_states: Dict[Checkpoint, BeaconState] = field(default_factory=dict) 154 | latest_messages: Dict[ValidatorIndex, LatestMessage] = field(default_factory=dict) 155 | ``` 156 | 157 | The member variables here are as follows: 158 | 159 | * `time`: the current time 160 | * `genesis_time`: the time of the genesis block of the chain 161 | * `justified_checkpoint`: the Casper FFG justified checkpoint that is used as the root of the LMD GHOST fork choice 162 | * `finalized_checkpoint`: the last finalized checkpoint; this block and its ancestors cannot be reverted 163 | * `best_justified_checkpoint`: the justified checkpoint that we will switch to at the start of the next epoch (see [this section](#should_update_justified_checkpoint) for why we store this variable temporarily and only switch over the `justified_checkpoint` at the start of the next epoch) 164 | * `blocks`: all blocks that we know about. Note that each `Block` contains a `parent_root`, so this contains the full "block tree" so we can get parents and children for any block 165 | * `block_states`: the post-state of every block that we know about. We need this for a few reasons: (i) to verify any new incoming block that claims that block as a parent, (ii) to be able to get the current and previous justified checkpoint of a block when running `filter_block_tree`, and (iii) to compute the end-of-epoch checkpoint states. 166 | * `checkpoint_states`: the post-state of every checkpoint. This could be different from the post-state of the block referenced by the checkpoint in the case where there are skipped slots; one would need to run the state transition function through the empty slots to get to the end-of-epoch state. Note particularly the extreme case where there is more than an entire epoch of skipped slots between a block and its child, so there are _multiple_ checkpoints referring to that block, with different epoch numbers and different states. 167 | * `latest_messages`: the latest epoch and block voted for by each validator. 168 | 169 | Note that in reality, instead of storing the post-states of all blocks and checkpoints that they know about, clients may simply store only the latest state, opting to reprocess blocks or process a saved journal of state changes if they want to process older blocks. This sacrifices computing efficiency in exceptional cases but saves greatly on storage. 170 | 171 | #### `get_forkchoice_store` 172 | 173 | This function initializes the `store` given a particular block that the fork choice would start from. This should be the most recent finalized block that the client knows about from extra-protocol sources; at the beginning, it would just be the genesis. 174 | 175 | *Note* With regards to fork choice, block headers are interchangeable with blocks. The spec is likely to move to headers for reduced overhead in test vectors and better encapsulation. Full implementations store blocks as part of their database and will often use full blocks when dealing with production fork choice. 176 | 177 | _The block for `anchor_root` is incorrectly initialized to the block header, rather than the full block. This does not affect functionality but will be cleaned up in subsequent releases._ 178 | 179 | ```python 180 | def get_forkchoice_store(anchor_state: BeaconState) -> Store: 181 | anchor_block_header = copy(anchor_state.latest_block_header) 182 | if anchor_block_header.state_root == Bytes32(): 183 | anchor_block_header.state_root = hash_tree_root(anchor_state) 184 | anchor_root = hash_tree_root(anchor_block_header) 185 | anchor_epoch = get_current_epoch(anchor_state) 186 | justified_checkpoint = Checkpoint(epoch=anchor_epoch, root=anchor_root) 187 | finalized_checkpoint = Checkpoint(epoch=anchor_epoch, root=anchor_root) 188 | return Store( 189 | time=uint64(anchor_state.genesis_time + SECONDS_PER_SLOT * anchor_state.slot), 190 | genesis_time=anchor_state.genesis_time, 191 | justified_checkpoint=justified_checkpoint, 192 | finalized_checkpoint=finalized_checkpoint, 193 | best_justified_checkpoint=justified_checkpoint, 194 | blocks={anchor_root: anchor_block_header}, 195 | block_states={anchor_root: copy(anchor_state)}, 196 | checkpoint_states={justified_checkpoint: copy(anchor_state)}, 197 | ) 198 | ``` 199 | 200 | #### `get_slots_since_genesis` 201 | 202 | ```python 203 | def get_slots_since_genesis(store: Store) -> int: 204 | return (store.time - store.genesis_time) // SECONDS_PER_SLOT 205 | ``` 206 | 207 | #### `get_current_slot` 208 | 209 | ```python 210 | def get_current_slot(store: Store) -> Slot: 211 | return Slot(GENESIS_SLOT + get_slots_since_genesis(store)) 212 | ``` 213 | 214 | #### `compute_slots_since_epoch_start` 215 | 216 | ```python 217 | def compute_slots_since_epoch_start(slot: Slot) -> int: 218 | return slot - compute_start_slot_at_epoch(compute_epoch_at_slot(slot)) 219 | ``` 220 | 221 | Compute which slot of the current epoch we are in (returns 0...31). 222 | 223 | #### `get_ancestor` 224 | 225 | ```python 226 | def get_ancestor(store: Store, root: Root, slot: Slot) -> Root: 227 | block = store.blocks[root] 228 | if block.slot > slot: 229 | return get_ancestor(store, block.parent_root, slot) 230 | elif block.slot == slot: 231 | return root 232 | else: 233 | # root is older than queried slot, thus a skip slot. Return most recent root prior to slot 234 | return root 235 | ``` 236 | 237 | Get the ancestor of block `root` (we refer to all blocks by their root in the fork choice spec) at the given `slot` (eg. if `root` was at slot 105 and `slot = 100`, and the chain has no skipped slots in between, it would return the block's fifth ancestor). 238 | 239 | #### `get_latest_attesting_balance` 240 | 241 | ```python 242 | def get_latest_attesting_balance(store: Store, root: Root) -> Gwei: 243 | state = store.checkpoint_states[store.justified_checkpoint] 244 | active_indices = get_active_validator_indices(state, get_current_epoch(state)) 245 | return Gwei(sum( 246 | state.validators[i].effective_balance for i in active_indices 247 | if (i in store.latest_messages 248 | and get_ancestor(store, store.latest_messages[i].root, store.blocks[root].slot) == root) 249 | )) 250 | ``` 251 | 252 | Get the total ETH attesting to a given block or its descendants, considering only latest attestations and active validators. This is the main function that is used to choose between two children of a block in LMD GHOST. Recall the diagram from above: 253 | 254 | ![](https://vitalik.ca/files/posts_files/cbc-casper-files/Chain7.png) 255 | 256 | In this diagram, we assume that each of the last five block proposals (the blue ones) carries one attestation, which specifies that block as the head, and we assume each block is created by a different validator, and all validators have the same deposit size. The number in each square represents the latest attesting balance of that block. In eth2, blocks and attestations are separate, and there will be hundreds of attestations supporting each block, but otherwise, the principle is the same. 257 | 258 | #### `filter_block_tree` 259 | 260 | Here, we implement an important but subtle deviation from the "LMD GHOST starting from the latest justified block" rule mentioned above. To motivate this deviation, consider the following attack: 261 | 262 | * There exists a justified block B, with two descendants, C1 and C2 263 | * B is justified, but the evidence of these justifications somehow only got included in the C1 chain, not the C2 chain 264 | * The C1 chain starts off favored by the LMD GHOST fork choice. Suppose that 49% of validators attest to C1, using B as the latest justified block, as the C1 chain recognizes B as justified 265 | * The fork choice switches to favoring C2 for some reason (eg. the other 51% of validators attested to C2 in that epoch). C2 does not recognize B as justified (and after two epochs can't recognize B as justified as it's too late to include evidence), so some earlier block A is used as the latest justified block instead. 266 | 267 | Now, all validators see C2 as the canonical chain, and the system is stuck: for a new block to be finalized, 67% of validators must make an attestation `[A -> C2]` (or `[A -> child(C2)]`), but only 51% can do this freely; at least 16% are constrained because they already voted `[B -> C1]`, and `[A -> C2]` violates the no-double-vote rule and `[A -> child(C2)]` violates the no-surround rule. Hence, the system can only progress if 16% voluntarily slash themselves. 268 | 269 | The fix is the following. We restrict the fork choice to only looking at descendants of B that actually recognize B as their latest justified block (or more precisely, leaves in the block tree where B is their latest justified block, as well as their ancestors). A client only accepts a block B locally as a latest justified block if there is an actual descendant block that recognizes it as such, so we know there must be at least one such chain. With this fix, C2 would not even be considered a viable candidate descendant of B until it (or one of its descendants) recognizes B as justified, so the above situation would resolve by simply favoring C1. 270 | 271 | See [section 4.6 of the Gasper paper](https://arxiv.org/pdf/2003.03052.pdf) for more details. 272 | 273 | ```python 274 | def filter_block_tree(store: Store, block_root: Root, blocks: Dict[Root, BeaconBlock]) -> bool: 275 | block = store.blocks[block_root] 276 | children = [ 277 | root for root in store.blocks.keys() 278 | if store.blocks[root].parent_root == block_root 279 | ] 280 | 281 | # If any children branches contain expected finalized/justified checkpoints, 282 | # add to filtered block-tree and signal viability to parent. 283 | if any(children): 284 | filter_block_tree_result = [filter_block_tree(store, child, blocks) for child in children] 285 | if any(filter_block_tree_result): 286 | blocks[block_root] = block 287 | return True 288 | return False 289 | 290 | # If leaf block, check finalized/justified checkpoints as matching latest. 291 | head_state = store.block_states[block_root] 292 | 293 | correct_justified = ( 294 | store.justified_checkpoint.epoch == GENESIS_EPOCH 295 | or head_state.current_justified_checkpoint == store.justified_checkpoint 296 | ) 297 | correct_finalized = ( 298 | store.finalized_checkpoint.epoch == GENESIS_EPOCH 299 | or head_state.finalized_checkpoint == store.finalized_checkpoint 300 | ) 301 | # If expected finalized/justified, add to viable block-tree and signal viability to parent. 302 | if correct_justified and correct_finalized: 303 | blocks[block_root] = block 304 | return True 305 | 306 | # Otherwise, branch not viable 307 | return False 308 | ``` 309 | 310 | #### `get_filtered_block_tree` 311 | 312 | `filter_block_tree` above is an impure function; it takes as input a key/value dict, which it passes along to its recursive calls to fill in the dict. `get_filtered_block_tree` is a pure function that wraps around it. Additionally, instead of requiring the `root` to be passed as an explicit argument, it gets the justified checkpoint directly from the `store` (which contains, among other things, the full block tree). 313 | 314 | ```python 315 | def get_filtered_block_tree(store: Store) -> Dict[Root, BeaconBlock]: 316 | """ 317 | Retrieve a filtered block tree from ``store``, only returning branches 318 | whose leaf state's justified/finalized info agrees with that in ``store``. 319 | """ 320 | base = store.justified_checkpoint.root 321 | blocks: Dict[Root, BeaconBlock] = {} 322 | filter_block_tree(store, base, blocks) 323 | return blocks 324 | ``` 325 | 326 | #### `get_head` 327 | 328 | The main fork choice rule function: gets the head of the chain. 329 | 330 | ```python 331 | def get_head(store: Store) -> Root: 332 | # Get filtered block tree that only includes viable branches 333 | blocks = get_filtered_block_tree(store) 334 | # Execute the LMD-GHOST fork choice 335 | head = store.justified_checkpoint.root 336 | justified_slot = compute_start_slot_at_epoch(store.justified_checkpoint.epoch) 337 | while True: 338 | children = [ 339 | root for root in blocks.keys() 340 | if blocks[root].parent_root == head and blocks[root].slot > justified_slot 341 | ] 342 | if len(children) == 0: 343 | return head 344 | # Sort by latest attesting balance with ties broken lexicographically 345 | head = max(children, key=lambda root: (get_latest_attesting_balance(store, root), root)) 346 | ``` 347 | 348 | This follows the following procedure: 349 | 350 | 1. Get the latest justified block hash, call it `B` (this is implicit in `get_filtered_block_tree`) 351 | 2. Get the subtree of blocks rooted in `B` (done by `get_filtered_block_tree`) 352 | 3. Filter that for blocks whose slot exceeds the slot of `B` (technically, this check is no longer necessary ever since (2) was introduced so may be removed) 353 | 4. Walk down the tree, at each step where a block has multiple children selecting the child with the strongest support (ie. higher `get_latest_attesting_balance`) 354 | 355 | From here on below, we have the functions for _updating_ the `store`. 356 | 357 | #### `should_update_justified_checkpoint` 358 | 359 | ```python 360 | def should_update_justified_checkpoint(store: Store, new_justified_checkpoint: Checkpoint) -> bool: 361 | """ 362 | To address the bouncing attack, only update conflicting justified 363 | checkpoints in the fork choice if in the early slots of the epoch. 364 | Otherwise, delay incorporation of new justified checkpoint until next epoch boundary. 365 | 366 | See https://ethresear.ch/t/prevention-of-bouncing-attack-on-ffg/6114 for more detailed analysis and discussion. 367 | """ 368 | if compute_slots_since_epoch_start(get_current_slot(store)) < SAFE_SLOTS_TO_UPDATE_JUSTIFIED: 369 | return True 370 | 371 | justified_slot = compute_start_slot_at_epoch(store.justified_checkpoint.epoch) 372 | if not get_ancestor(store, new_justified_checkpoint.root, justified_slot) == store.justified_checkpoint.root: 373 | return False 374 | 375 | return True 376 | ``` 377 | 378 | The idea here is that we want to only change the last-justified-block within the first 1/3 of an epoch. This prevents "bouncing attacks" of the following form: 379 | 380 | 1. Start from a scenario wherein epoch N, 62% of validators support block A, and in epoch N+1, 62% of validators support block B. Suppose that the attacker has 5% of the total stake. This scenario requires very exceptional networking conditions to get into; the point of the attack, however, is that if we get into such a scenario the attacker could perpetuate it, permanently preventing finality. 381 | 2. Due to LMD GHOST, B is favored, and so validators are continuing to vote for B. However, the attacker suddenly publishes attestations worth 5% of the total stake tagged with epoch N for block A, causing A to get justified. 382 | 3. In epoch N+2, A is justified and so validators are attesting to A', a descendant of A. When A' gets to 62% support, the attacker publishes attestations worth 5% of total stake for B. Now B is justified and favored by the fork choice. 383 | 4. In epoch N+3, B is justified, and so validators are attesting to B', a descendant of B. When B' gets to 62% support, the attacker publishes attestations worth 5% of total stake for A'... 384 | 385 | This could continue forever, bouncing permanently between the two chains preventing any new block from being finalized. This attack can happen because the combined use of LMD GHOST and Casper FFG creates a discontinuity, where a small shift in support for a block can outweigh a large amount of support for another block if that small shift pushes it past the 2/3 threshold needed for justification. We block the attack by only allowing the latest justified block to change near the beginning of an epoch; this way, there is a full 2/3 of an epoch during which honest validators agree on the head and have the opportunity to justify a block and thereby further cement it, at the same time causing the LMD GHOST rule to strongly favor that head. This sets up that block to most likely be finalized in the next epoch. 386 | 387 | See [Ryuya Nakamura's ethresear.ch post](https://ethresear.ch/t/prevention-of-bouncing-attack-on-ffg/6114) for more discussion. 388 | 389 | #### `on_attestation` helpers 390 | 391 | ##### `validate_on_attestation` 392 | 393 | When a client receives an attestation (either from a block or directly on the wire), it should first perform some checks, and reject the attestation if it does not pass those checks. 394 | 395 | ```python 396 | def validate_on_attestation(store: Store, attestation: Attestation) -> None: 397 | target = attestation.data.target 398 | 399 | # Attestations must be from the current or previous epoch 400 | current_epoch = compute_epoch_at_slot(get_current_slot(store)) 401 | # Use GENESIS_EPOCH for previous when genesis to avoid underflow 402 | previous_epoch = current_epoch - 1 if current_epoch > GENESIS_EPOCH else GENESIS_EPOCH 403 | # If attestation target is from a future epoch, delay consideration until the epoch arrives 404 | assert target.epoch in [current_epoch, previous_epoch] 405 | assert target.epoch == compute_epoch_at_slot(attestation.data.slot) 406 | 407 | # Attestations target be for a known block. If target block is unknown, delay consideration until the block is found 408 | assert target.root in store.blocks 409 | 410 | # Attestations must be for a known block. If block is unknown, delay consideration until the block is found 411 | assert attestation.data.beacon_block_root in store.blocks 412 | # Attestations must not be for blocks in the future. If not, the attestation should not be considered 413 | assert store.blocks[attestation.data.beacon_block_root].slot <= attestation.data.slot 414 | 415 | # LMD vote must be consistent with FFG vote target 416 | target_slot = compute_start_slot_at_epoch(target.epoch) 417 | assert target.root == get_ancestor(store, attestation.data.beacon_block_root, target_slot) 418 | 419 | # Attestations can only affect the fork choice of subsequent slots. 420 | # Delay consideration in the fork choice until their slot is in the past. 421 | assert get_current_slot(store) >= attestation.data.slot + 1 422 | ``` 423 | 424 | We do the following checks: 425 | 426 | 1. Check that the attestation is from either the current or the previous epoch (we ignore attestations that come too late). This is done to prevent bounce attacks (see [above](#should_update_justified_checkpoint)) from "saving up" epochs and flipping back and forth between chains many times. 427 | 2. Check that the attestation is attesting to a block which the client has already received and verified (if it is not, the attestation may be saved for some time, in case that block is later found) 428 | 3. Check that the attestation is attesting to a block which is at or before the slot of the attestation (ie. can't attest to future blocks) 429 | 4. Check that the vote for the head block is consistent with the vote for the target 430 | 5. Check that the attestation's slot itself is not in the future 431 | 432 | ##### `store_target_checkpoint_state` 433 | 434 | ```python 435 | def store_target_checkpoint_state(store: Store, target: Checkpoint) -> None: 436 | # Store target checkpoint state if not yet seen 437 | if target not in store.checkpoint_states: 438 | base_state = copy(store.block_states[target.root]) 439 | if base_state.slot < compute_start_slot_at_epoch(target.epoch): 440 | process_slots(base_state, compute_start_slot_at_epoch(target.epoch)) 441 | store.checkpoint_states[target] = base_state 442 | ``` 443 | 444 | Update the `checkpoint_states` dict, which is a convenience dict that stores the end-of-epoch states for each checkpoint. Most of the time, this is the same as the post-state of the last block in an epoch, but in the case where there are skipped slots, the state would need to process through the empty slots first. See the [Store definition](#Store) for more details. 445 | 446 | ##### `update_latest_messages` 447 | 448 | ```python 449 | def update_latest_messages(store: Store, attesting_indices: Sequence[ValidatorIndex], attestation: Attestation) -> None: 450 | target = attestation.data.target 451 | beacon_block_root = attestation.data.beacon_block_root 452 | for i in attesting_indices: 453 | if i not in store.latest_messages or target.epoch > store.latest_messages[i].epoch: 454 | store.latest_messages[i] = LatestMessage(epoch=target.epoch, root=beacon_block_root) 455 | ``` 456 | 457 | In the latest messages dict, update the latest message of each validator who participated in the given attestation. 458 | 459 | ### Handlers 460 | 461 | #### `on_tick` 462 | 463 | ```python 464 | def on_tick(store: Store, time: uint64) -> None: 465 | previous_slot = get_current_slot(store) 466 | 467 | # update store time 468 | store.time = time 469 | 470 | current_slot = get_current_slot(store) 471 | # Not a new epoch, return 472 | if not (current_slot > previous_slot and compute_slots_since_epoch_start(current_slot) == 0): 473 | return 474 | # Update store.justified_checkpoint if a better checkpoint is known 475 | if store.best_justified_checkpoint.epoch > store.justified_checkpoint.epoch: 476 | store.justified_checkpoint = store.best_justified_checkpoint 477 | ``` 478 | 479 | This function runs on each tick (ie. per second). At the end of each epoch, update the justified checkpoint used in the fork choice. 480 | 481 | #### `on_block` 482 | 483 | ```python 484 | def on_block(store: Store, signed_block: SignedBeaconBlock) -> None: 485 | block = signed_block.message 486 | # Parent block must be known 487 | assert block.parent_root in store.block_states 488 | # Make a copy of the state to avoid mutability issues 489 | pre_state = copy(store.block_states[block.parent_root]) 490 | # Blocks cannot be in the future. If they are, their consideration must be delayed until the are in the past. 491 | assert get_current_slot(store) >= block.slot 492 | 493 | # Check that block is later than the finalized epoch slot (optimization to reduce calls to get_ancestor) 494 | finalized_slot = compute_start_slot_at_epoch(store.finalized_checkpoint.epoch) 495 | assert block.slot > finalized_slot 496 | # Check block is a descendant of the finalized block at the checkpoint finalized slot 497 | assert get_ancestor(store, block.parent_root, finalized_slot) == store.finalized_checkpoint.root 498 | 499 | # Check the block is valid and compute the post-state 500 | state = state_transition(pre_state, signed_block, True) 501 | # Add new block to the store 502 | store.blocks[hash_tree_root(block)] = block 503 | # Add new state for this block to the store 504 | store.block_states[hash_tree_root(block)] = state 505 | 506 | # Update justified checkpoint 507 | if state.current_justified_checkpoint.epoch > store.justified_checkpoint.epoch: 508 | if state.current_justified_checkpoint.epoch > store.best_justified_checkpoint.epoch: 509 | store.best_justified_checkpoint = state.current_justified_checkpoint 510 | if should_update_justified_checkpoint(store, state.current_justified_checkpoint): 511 | store.justified_checkpoint = state.current_justified_checkpoint 512 | 513 | # Update finalized checkpoint 514 | if state.finalized_checkpoint.epoch > store.finalized_checkpoint.epoch: 515 | store.finalized_checkpoint = state.finalized_checkpoint 516 | 517 | # Potentially update justified if different from store 518 | if store.justified_checkpoint != state.current_justified_checkpoint: 519 | # Update justified if new justified is later than store justified 520 | if state.current_justified_checkpoint.epoch > store.justified_checkpoint.epoch: 521 | store.justified_checkpoint = state.current_justified_checkpoint 522 | return 523 | 524 | # Update justified if store justified is not in chain with finalized checkpoint 525 | finalized_slot = compute_start_slot_at_epoch(store.finalized_checkpoint.epoch) 526 | ancestor_at_finalized_slot = get_ancestor(store, store.justified_checkpoint.root, finalized_slot) 527 | if ancestor_at_finalized_slot != store.finalized_checkpoint.root: 528 | store.justified_checkpoint = state.current_justified_checkpoint 529 | ``` 530 | 531 | Upon receiving a block, first, do a few checks: 532 | 533 | 1. Check that we know about the block's parent (if we don't, we can temporarily save the block in case we find the parent later) 534 | 2. Check that the block is not from a slot that is still in the future (if it is, we can pretend to not hear about the block until time progresses to the point that the block is no longer from the future) 535 | 3. Check that the block slot is later than the last finalized block's slot and that the block is a descendant of the last finalized block (an optimization to reduce unnecessary work from blocks that can clearly no longer possibly become canonical) 536 | 4. Check that the block's post-state root is correct and the state transition passes 537 | 538 | We then add the block to the DB. We also update the justified and finalized checkpoints. The idea is that if the justified checkpoint known by the received block has a higher epoch number than the justified checkpoint we know about, we accept that justified checkpoint. If we are [in the first 1/3](#should_update_justified_checkpoint) of an epoch, we accept it immediately, otherwise, we put it in `store.best_justified_checkpoint` so it can be updated into `store.justified_checkpoint` [at the end](#on_tick) of the epoch. 539 | 540 | If the received block knows about a finalized checkpoint with a higher epoch number than what we know about, we accept it, and we also immediately update the justified checkpoint if either (i) the justified checkpoint provided in the block is more recent than the one we know about, or (ii) the one we know about is not compatible with the new finalized block. Note that there is a theoretical possibility that condition (ii) causes the justified checkpoint to go backward (change from a later epoch to an earlier epoch), but for this to happen, there would need to be a finalized block B with a justified child B', with a justified block A' on a conflicting chain pointing to some earlier finalized block A, which implies a slashable 1/3 attack due to the no-surround rule. In such a case, anyone who referenced A' as an LJB may not be able to build on top of B', so some validators who participated in the "wrong chain" may need to suffer some level of inactivity leak. 541 | 542 | #### `on_attestation` 543 | 544 | ```python 545 | def on_attestation(store: Store, attestation: Attestation) -> None: 546 | """ 547 | Run ``on_attestation`` upon receiving a new ``attestation`` from either within a block or directly on the wire. 548 | 549 | An ``attestation`` that is asserted as invalid may be valid at a later time, 550 | consider scheduling it for later processing in such case. 551 | """ 552 | validate_on_attestation(store, attestation) 553 | store_target_checkpoint_state(store, attestation.data.target) 554 | 555 | # Get state at the `target` to fully validate attestation 556 | target_state = store.checkpoint_states[attestation.data.target] 557 | indexed_attestation = get_indexed_attestation(target_state, attestation) 558 | assert is_valid_indexed_attestation(target_state, indexed_attestation) 559 | 560 | # Update latest messages for attesting indices 561 | update_latest_messages(store, indexed_attestation.attesting_indices, attestation) 562 | ``` 563 | 564 | Called upon receiving an attestation. This function simply combines together the helper functions above: 565 | 566 | * Validate the attestation 567 | * Compute the state of the target checkpoint that the attestation references 568 | * Check that the attestation signature is valid (this requires the target state to compute the validator set) 569 | * Update the `latest_messages` dict. 570 | --------------------------------------------------------------------------------