├── assets └── images │ ├── 10-31-2021-dust-storm.jpg │ └── testnet-dust-storm-graph.png ├── template.md ├── 2022-10 ├── 2022-10-31-Android-Phone-Incident └── 2022-10-20-Beta-Keybase-Channel-Permissions ├── 2023-05 ├── 2023-05-08-Remote-Denial-of-Service-attack-on-full_nodes.md ├── 2023-05-08-Subscriptions-to-a-full_node-can-cause-a-crash-by-requesting-excessive-data.md ├── 2023-05-08-Negative-values-allowed-in-settlement_payments.md ├── 2023-05-04-Discord-api-abuse.md └── 2023-05-08-AGG_SIG_UNSAFE-can-mimic-AGG_SIG_ME-condition.md ├── 2024-03 ├── 2024-03-07-mempool-not-updated-at-re_org.md ├── 2024-03-07-unfinished_blocks-cache.md ├── 2024-03-07-inefficient-mempool_item-selection.md └── 2024-03-07-unfinished_blocks-different-foliage.md ├── 2023-01 ├── 2023-01-31-CLVM-infinite-recursion.md ├── 2023-01-31-Wallet-error-handling-causing-infinite-loop.md └── 2023-01-31-Negative-values-allowed-in-settlement_payments.md ├── 2025-06 └── 2025-06-24-hsmwizard_utility_unsanitized_user_input.md ├── 2024-06 ├── 2024-06-25-GUI-clickable-links-NFT-metadata.md └── 2024-06-25-blockchain_mainnet-stall.md ├── 2021-12 └── 2021-12-15-EmployeeCompromise.md ├── 2023-04 └── 04-25-User_Account_Compromise.md ├── 2023-07 ├── 2023-07-21-unbound-CLVM-cost-restriction-on-mempool.md └── 2023-07-05-email-gift-card-phish.md ├── 2025-12 └── React-CVE-2025-55182.md ├── 2021-05 ├── 2021-05-26-website-outage.md ├── 2021-05-10-WikiAttack.md └── 2021-05-09-negative-coin-values.md ├── 2022-01 ├── 2022-01-27-Ramp-Card.md └── 2022-01-20-EmployeeCompromise.md ├── README.md ├── 2021-10 ├── 2021-10-31-EmployeeCompromise.md └── 2021-10-31-DustStorm.md ├── 2023-08 └── 2023-08-04-compromised-self-hosted-runners.md ├── 2025-02 └── 2025-02-25-block_production_impact_due_FastForward_mempool_issues.md ├── 2022-06 └── 2022-06-28-ProxyAbuseReport.md ├── 2025-09 └── 2025-09-28-npm-supply-chain-attacks.md ├── 2024-02 └── 2024-02-15-Financial-instutition-credentials.md ├── LICENSE └── 2022-08 └── 2022-08-19-CATbleed.md /assets/images/10-31-2021-dust-storm.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chia-Network/post-mortem/HEAD/assets/images/10-31-2021-dust-storm.jpg -------------------------------------------------------------------------------- /assets/images/testnet-dust-storm-graph.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chia-Network/post-mortem/HEAD/assets/images/testnet-dust-storm-graph.png -------------------------------------------------------------------------------- /template.md: -------------------------------------------------------------------------------- 1 | # ISSUE NAME 2 | ## Issue Summary 3 | 4 | Lorem ipsum dolor 5 | 6 | ## Timeline (all times PST) DATE RANGE. All times approximations 7 | 8 | 12:45 PM DATE: Thing 1 happened 9 | 10 | 13:55 PM DATE: Thing 2 happened 11 | 12 | ## Resolution and recovery 13 | 14 | Sit amet, consectetur 15 | 16 | ## Additional Sections as required 17 | 18 | Etiam quis dapibus 19 | 20 | ## Q&A 21 | 22 | Curabitur vulputate felis 23 | -------------------------------------------------------------------------------- /2022-10/2022-10-31-Android-Phone-Incident: -------------------------------------------------------------------------------- 1 | # Android Phone Incident 2 | ## Issue Summary 3 | 4 | Pixel 5a with android 13 5 | employee device downloaded a suspicious file without user input. 6 | The file was not present on the phone when searched for. 7 | The phone then rebooted without user input. 8 | No visibal traces of the event were present on the device. 9 | 10 | ## Resolution and recovery 11 | 12 | Phone was airgapped using airplane mode while briefe investigation took place. Once the investigation was over the phone was factory reset. 13 | -------------------------------------------------------------------------------- /2023-05/2023-05-08-Remote-Denial-of-Service-attack-on-full_nodes.md: -------------------------------------------------------------------------------- 1 | # Remote Denial of Service attack on full_nodes 2 | 3 | ## Issue Summary 4 | 5 | Full_nodes connect with each other via a peer-2-peer protocol. 6 | 7 | The connection established between nodes is encrypted and follows a protocol where a handshake is performed after which the connection is either disconnected or 8 | remains open for further communication. 9 | 10 | A Remote Denial of Service attack could create too many such connections for a full_node to process and close out before resources were depleted. 11 | 12 | ## Timeline (all times PST) 10/22/2022 - 02/15/2023. All times approximations 13 | 14 | 10/22/2022 - A researcher submitted a bug report via bugcrowd (https://bugcrowd.com/chia-network-bb) detailing the security issue. 15 | 16 | 02/15/2023 - Version 1.7.0 of the Chia blockchain was released that stops accepting new connections when a threshold is reached in the queue of connections that are being processed. 17 | 18 | ## Resolution and recovery 19 | 20 | This issue was addressed in 1.7.0 by: Monitoring the queue of connections that are being processed and when a threshold is reached, stop accepting connections until there is room to accept additional connections again. 21 | 22 | -------------------------------------------------------------------------------- /2023-05/2023-05-08-Subscriptions-to-a-full_node-can-cause-a-crash-by-requesting-excessive-data.md: -------------------------------------------------------------------------------- 1 | # Coin and or PuzzleHash subscriptions to a full_node can cause it to crash by requesting excessive data. 2 | 3 | ## Issue Summary 4 | 5 | The wallet protocol has a mechanism in which a wallet subscribes to updates to certain coin_id's or puzzle_hashes from a full_node. 6 | 7 | The light wallet protocol allows a remote wallet to make such requests to a full_node. 8 | 9 | The amount of information a subscription returns to the wallet is capped, but before this threshold is checked the data is pulled from the full_node database and can cause a full_node crash if the data amount is excessive. 10 | 11 | ## Timeline (all times PST) 10/31/2022 - 02/15/2023. All times approximations 12 | 13 | 10/31/2022 - The issue was discovered by Chia. 14 | 15 | 02/15/2023 - Version 1.7.0 of the Chia blockchain was released that limits how many coin_id's and / or puzzle_hashes can be queried from a full_node via a subscription. 16 | 17 | ## Resolution and recovery 18 | 19 | This issue was addressed in 1.7.0 by: Limiting how many coin_id's and / or puzzle_hashes can be queried from a full_node via a subscription. 20 | 21 | The thresholds are different when communicating with a local full_node or a remote full_node. 22 | 23 | -------------------------------------------------------------------------------- /2024-03/2024-03-07-mempool-not-updated-at-re_org.md: -------------------------------------------------------------------------------- 1 | # Mempool not updated at re-org. 2 | 3 | ## Issue Summary 4 | 5 | Due to a bug in the Mempool code at the time of a re-org, if the latest block was a non transaction block, the mempool did not get updated until the next transaction block occurred. 6 | 7 | When a farmer would have an outdated Mempool and add conflicting items into their farmed block, a block without any transactions would be created instead. 8 | 9 | For more information on re-orgs and transaction blocks, visit: 10 | https://docs.chia.net/consensus-foliage/ 11 | 12 | 13 | ## Timeline (all times PST) 01/05/2024 - 02/28/2024. All times approximations 14 | 15 | 01-05-2024 - A bug report was created that informed Chia of the issue at https://github.com/Chia-Network/chia-blockchain/issues/17186 16 | 17 | 01-20-2024 - The codebase was updated with a fix at https://github.com/Chia-Network/chia-blockchain/pull/17370 18 | 19 | 02-28-2024 - Version 2.2.0 was released that included the fix. 20 | 21 | 22 | ## Resolution and recovery 23 | 24 | A fix was introduced to have the mempool update immediately with a reorg via Blockchain.get_tx_peak(), which returns the most recent transaction block (which may be the peak, if that is a transaction block). 25 | 26 | For more information about the issue and resolution, visit: 27 | https://github.com/Chia-Network/chia-blockchain/pull/17370 28 | -------------------------------------------------------------------------------- /2023-01/2023-01-31-CLVM-infinite-recursion.md: -------------------------------------------------------------------------------- 1 | # CLVM infinite recursion 2 | 3 | ## Issue Summary 4 | 5 | Chia uses the coinset model, in which everything is a coin. Each coin has a puzzle (or program) that is executed when the coin is spent. 6 | 7 | The puzzle consists of CLVM (Chialisp Virtual Machine) code (https://chialisp.com/). 8 | 9 | A malicious CLVM program can be crafted such that, when run, will use an unreasonable amount of memory before it runs out of “cost”. 10 | We charge a cost for every operation and memory allocation in a CLVM program in order to restrict the amount of work and memory it can use from the computer validating a block or transaction. However, we overlooked the cost of allocating space on the virtual machine’s stack. 11 | ## Timeline (all times PST) 10/14/2022 - 11/03/2022. All times approximations 12 | 13 | 10/14/2022 - A researcher submitted a bug report via bugcrowd (https://bugcrowd.com/chia-network-bb) detailing the CLVM infinite recursion issue. 14 | 15 | 11/03/202 - Version 1.6.1 of the Chia blockchain was released with fixes for the CLVM infinite recursion issue. 16 | 17 | ## Resolution and recovery 18 | 19 | This was issue was addressed in 1.6.1 by: 20 | Validating the proof-of-space in UnfinishedBlocks before executing the CLVM program 21 | Introducing limits on the stack depth when executing CLVM in mempool-mode 22 | 23 | The issue was not exploited on mainnet. 24 | -------------------------------------------------------------------------------- /2025-06/2025-06-24-hsmwizard_utility_unsanitized_user_input.md: -------------------------------------------------------------------------------- 1 | # hsmwizard utility unsanitized user input 2 | 3 | ## Issue Summary 4 | 5 | The hsms.cmds.hsmwizard.py script within the Chia-blockchain application is vulnerable to an OS Command Injection. 6 | This allows an attacker to execute arbitrary shell commands on the system running the hsmwizard utility due to unsanitized user input. 7 | 8 | An attacker will need to have access to the host where hsms.cmds.hsmwizard.py resides and sufficient privileges to execute the commands, making Social Engineering attacks the most likely path to exploit this issue. 9 | 10 | The utility hsmwizard resides on a HSM (a physical device that safeguards and manages cryptographic keys) used by Chia, that is air gapped at secured locations and are only interacted with by approved senior people at Chia. 11 | 12 | Chia has not encountered attempts to leverage the hsmwizard issue in an attack itself. 13 | This post-mortem serves as a disclosure and means to give credit to the researcher for reporting the issue so that others that would use the hsmwizard tool 14 | are aware of the issue. 15 | 16 | 17 | ## Timeline (all times PST) 06/10/2025. All times approximations 18 | 19 | 06-10-2025 - The issue was reported by Goro Matsumoto via a HackerOne (https://hackerone.com/chia_network/) submission. 20 | 21 | 22 | ## Resolution and recovery 23 | 24 | This post-mortem serves as a disclosure for others than CNI that may use the hsms.cmds.hsmwizard.py script. 25 | -------------------------------------------------------------------------------- /2023-05/2023-05-08-Negative-values-allowed-in-settlement_payments.md: -------------------------------------------------------------------------------- 1 | # Negative values allowed in settlement_payments.clvm 2 | 3 | ## Issue Summary 4 | 5 | https://github.com/Chia-Network/post-mortem/blob/main/2023-01/2023-01-31-Negative-values-allowed-in-settlement_payments.md 6 | disclosed how the settlement_payments.clvm allowed for negative amounts and in the case of a Chia Asset Token creation, a special negative value of -113 (https://chialisp.com/cats) will run the TAIL of the CAT. 7 | 8 | Version 1.6.1 of the Chia blockchain contains fixes to the code to prevent negative values, but due to a typographical error the old settlement_payments.clvm with the security issue 9 | was still referenced in that code rendering the fix void. 10 | 11 | ## Timeline (all times PST) 01/31/2023 - 02/15/2023. All times approximations 12 | 13 | 01/31/2023 - A researcher submitted a bug report via bugcrowd (https://bugcrowd.com/chia-network-bb) detailing the security issue was not fixed in v1.6.1 14 | 15 | 02/15/2023 - Version 1.7.0 of the Chia blockchain was released with fixes for the Negative values allowed in settlement_payments.clvm issue. 16 | 17 | ## Resolution and recovery 18 | 19 | This issue was addressed in 1.7.0 by: Asserting the value to be positive before creating the outputs in [https://github.com/Chia-Network/chia-blockchain/blob/main/chia/wallet/puzzles/settlement_payments.clsp](https://github.com/Chia-Network/chia-blockchain/blob/main/chia/wallet/puzzles/settlement_payments.clsp) 20 | and referencing this updated ChiaList puzzle in the code. 21 | 22 | The issue was not exploited on mainnet. 23 | 24 | -------------------------------------------------------------------------------- /2022-10/2022-10-20-Beta-Keybase-Channel-Permissions: -------------------------------------------------------------------------------- 1 | # Keybase Beta Participant Permissions Post Mortem 2 | ## Issue Summary 3 | 4 | A user in the public beta farmers keybase group converted a keybase sub-channel to a big team and created a subchannel. The user shouldn’t have had the permission to begin with to be able to perform those actions. 5 | ## Timeline (all times PST ) DATE RANGE. All times approximations 6 | 7 | 10/20/2022 - 15:38 CST - a user, farmer_diamonds, converted the chia_network.public.beta_farmers from a sub-channel to a big team and created a sub-channel to the newly converted big team 8 | 10/20/2022 - 15:41 CST - Discovered that the farmer beta group was converted and a subchannel was added 9 | 10/20/2022 - 16:00 CST - Permission updated for all users from “writers” to “readers” in both the chia_network.public.beta_farmers channel and the chia_network.public.beta_consumers channel 10 | 10/20/2022 - 22:00 CST - All open email based invitations were cancelled and resent with “reader” permissions. No other users with open email invites had joined after all user permissions were changed to “Reader”. 11 | ## Resolution and recovery 12 | 13 | Publish a document for rules/guidelines for admins of a public keybase channel that can be shared with anyone in CNI who will be an admin of public channel 14 | File a bug/feature request to get Keybase to ensure the default role is what shows up when adding people to the keybase channel instead of currently defaulting to “writer”, until this is changed by Keybase, ensure that all non CNI users are put as “reader”. Currently even if the channel default permissions is set to readers only, when adding users the default is writers. 15 | 16 | -------------------------------------------------------------------------------- /2023-01/2023-01-31-Wallet-error-handling-causing-infinite-loop.md: -------------------------------------------------------------------------------- 1 | # Wallet error handling causing infinite loop 2 | 3 | ## Issue Summary 4 | 5 | The Chia reference wallet keeps track of coins it can interact with by subscribing to them at a full node. The full node will communicate coinstate changes to the wallet if the subscribed coins state changes when a transaction block is processed. 6 | 7 | The wallet keeps the data around the tracked coins in its own database, which will synchronize along with the blockchain database. 8 | 9 | The wallet, on a coin spend, will run code that is applicable for the type of coin, for example a Chia Asset Token (https://chialisp.com/cats) or a Pool NFT Singleton (https://chialisp.com/pooling) 10 | 11 | The way exceptions were handled in the wallet code could result in a bad coinstate causing the wallet to rewind its database a few blocks and retry to synchronize, encountering the same issue again when the same bad coinstate was processed. This would prevent the wallet from progressing past the block with the bad coinstate. 12 | 13 | ## Timeline (all times PST) 09/07/2022 - 11/03/2022. All times approximations 14 | 15 | 09/07/2022 - A researcher provided details of the Wallet error handling causing infinite loop issue via Keybase and Discord. 16 | 17 | 09/27/2022 - The researcher submitted a bug report via bugcrowd (https://bugcrowd.com/chia-network-bb) detailing the Wallet error handling causing infinite loop issue. 18 | 19 | 11/03/202 - Version 1.6.1 of the Chia blockchain was released with fixes for the Wallet error handling causing infinite loop issue. 20 | 21 | ## Resolution and recovery 22 | 23 | This was issue was addressed in 1.6.1 by: 24 | Adding try/catch sections around individual coinstate handling in the reference wallet code. 25 | 26 | The issue was reported on mainnet in connection with DID validation of NFTs. 27 | 28 | -------------------------------------------------------------------------------- /2024-06/2024-06-25-GUI-clickable-links-NFT-metadata.md: -------------------------------------------------------------------------------- 1 | # GUI clickable links from NFT metadata. 2 | 3 | ## Issue Summary 4 | 5 | A community member disclosed to Chia Network Inc. that at the time of an NFT’s minting, an attacker could add malicious links to the off-chain metadata. When a user later viewed the NFT in the reference wallet GUI, the links would be made clickable. 6 | This would occur because the GUI would expect any links to correspond to the NFT’s metadata, license, etc. However, the links could instead point to any URI with potentially malicious content. 7 | 8 | A few things to consider: 9 | - The user would need to click a malicious link in order to activate the payload; no malicious activity was possible from simply viewing an NFT. 10 | - Upon clicking a malicious link, most operating systems would display another dialog with a warning message that this link might be malicious. The user would then need to authorize the malicious link to be executed. 11 | - We found no evidence that this attack was ever attempted, successfully or not. 12 | 13 | We deemed that this potential exploit was sufficiently urgent to include a fix in the 2.3.0 release, which went out within a week of us learning about the issue. 14 | 15 | 16 | ## Timeline (all times PST) 04/26/2024 - 05/01/2024. All times approximations 17 | 18 | 04-26-2024 - The security issue was disclosed to our team. 19 | 20 | 04-29-2024 - The fix was added to the code base for version 2.3.0. 21 | 22 | 05-01-2024 - Version 2.3.0 was released, which included the fix. 23 | 24 | 25 | ## Resolution and recovery 26 | 27 | Starting in version 2.3.0, the reference wallet no longer allows links from NFT metadata to be clickable. Any links that do exist will still be displayed (though not clickable), thereby giving the user the option to copy them and paste them into a browser window. 28 | 29 | The source code for the resolution can be found in the following Pull Request: 30 | https://github.com/Chia-Network/chia-blockchain-gui/pull/2331 31 | 32 | -------------------------------------------------------------------------------- /2021-12/2021-12-15-EmployeeCompromise.md: -------------------------------------------------------------------------------- 1 | # Suspected Chia Employee Compromise 2 | 3 | ## Issue Summary 4 | 5 | __Type of Incident Detected__: Potential compromise of employee's Google Account 6 | 7 | __Initial Concerns__: Unexpected activity and login from the United Kingdom was noticed on employee's Gmail account page. 8 | 9 | ## Timeline (all times UTC ) 10 | 11 | Date/Time Stamps (Captured from keybase messages) 12 | 13 | | Day | Date (2021) | Time (UTC) | Event | 14 | |-----|-------------|-----------|-------| 15 | | Wed | Dec 15 | 22:40 | Employee recognizes abnormal activity logged in his Gmail account. Employee reports this to another employee, who contacts the IPS team. 16 | | Wed | Dec 15 | 22:45 | IPS team disables users Okta accounts and stars searching audit logs from google account for abnormal activity immediately. 17 | | Wed | Dec 15 | 22:54 | It is noted in that the affected employee is experiencing issues with Keybase. 18 | | Wed | Dec 15 | 23:02 | The IPS team discovered that the employee was using Express VPN, which provided an IP address in the UK, even though the user was connected to the VPN in Los Angeles. 19 | | Wed | Dec 15 | 23:04 | The Sr. Director of IPS instructs the user to remove Express VPN, install tailscale, and remove third-party applications from his Google account. 20 | | Wed | Dec 15 | 23:05 | No abnormal activity noted in the audit logs 21 | | Wed | Dec 15 | 23:08 | IPS enables employee's Okta account. 22 | | Wed | Dec 15 | 23:29 | Employee deletes third-party apps from Google account. 23 | | Wed | Dec 15 | 23:45 | Employee finishes logging into the Tailscale VPN and 24 | 25 | 26 | 27 | 28 | 29 | ## Root Cause 30 | 31 | ExpressVPN routed the user's traffic with public-facing IP that originated with a location in the UK. 32 | 33 | 34 | ## Resolution and recovery 35 | 36 | The IPS members responded quickly to identify that the VPN was the source of the issue. 37 | ## Corrective and Preventative Measure 38 | 39 | Enforce a standard of using Tailscale company-wide. The employee did a fantastic job of observing something that looked suspicious and promptly alerting the appropriate personnel. 40 | 41 | Continuous training of all Chia employee's will help to strengthen the company's security posture. 42 | -------------------------------------------------------------------------------- /2023-01/2023-01-31-Negative-values-allowed-in-settlement_payments.md: -------------------------------------------------------------------------------- 1 | # Negative values allowed in settlement_payments.clvm 2 | 3 | ## Issue Summary 4 | 5 | Offers are a way to enable peer-to-peer asset exchange on the Chia blockchain (https://chialisp.com/offers/). 6 | 7 | At the core of the offers functionality is the settlement payments puzzle, that spends the inputs from one party and recreates them as outputs for the counterparty. This action is applied to both sides of the offer at the same time. 8 | 9 | The input spends are guarded via PUZZLE ANNOUNCEMENTS, only if the party inputs are spent and their puzzle announcements are asserted will the counterparty inputs spends succeed and vice versa. 10 | 11 | The puzzle announcements enforce that the correct coin and value are part of the inputs of either side of the offer. 12 | 13 | The code that creates outputs failed to check against negative values. In the case of a Chia Asset Token creation, a special negative value of -113 (https://chialisp.com/cats) will run the TAIL of the CAT. 14 | 15 | When the TAIL execution outputs puzzle announcements that guard the input spends of the counterparty, then the requirement of the puzzle announcement is fulfilled and counterparty’s side of the offer trade succeeds regardless if the party’s inputs match the offer. 16 | 17 | This allows for bypassing the explicit promise that in return of your input of coin(s) and their value you will get a specific type of coin(s) and value back when a counterparty takes up your offer. 18 | 19 | ## Timeline (all times PST) 09/21/2022 - 11/03/2022. All times approximations 20 | 21 | 09/21/2022 - A researcher submitted a bug report via bugcrowd (https://bugcrowd.com/chia-network-bb) detailing the Negative values allowed in settlement_payments.clvm issue. 22 | 23 | 11/03/202 - Version 1.6.1 of the Chia blockchain was released with fixes for the Negative values allowed in settlement_payments.clvm issue. 24 | 25 | ## Resolution and recovery 26 | 27 | This was issue was addressed in 1.6.1 by: 28 | Asserting the value to be positive before creating the outputs in https://github.com/Chia-Network/chia-blockchain/blob/main/chia/wallet/puzzles/settlement_payments.clvm 29 | 30 | 31 | The issue was not exploited on mainnet. 32 | -------------------------------------------------------------------------------- /2024-06/2024-06-25-blockchain_mainnet-stall.md: -------------------------------------------------------------------------------- 1 | # Blockchain mainnet stall. 2 | 3 | ## Issue Summary 4 | 5 | The timelord logic has a routine to detect if the timelord has a heavier unfinished block than the new peak provided by the full_node the timelord is connected to. 6 | This would indicate that the timelord is working on a different chain that will be heavier and therefore can skip processing this provided new peak. 7 | 8 | A bug in this code caused a chain stall when the full_node and timelord would get into a state where the full_node would provide a valid new peak and the timelord would skip it due to an edge case state. 9 | 10 | The height that the chain stalled at was 4779033 block height with approximately a 10 minutes gap between blocks during the chain stall. 11 | 12 | 13 | ## Timeline (all times PST) 01/16/2024 - 02/28/2024. All times approximations 14 | 15 | 01-10-2024 16 | 17 | 7:32 PM - The chain stalled at height 4779033. 18 | 19 | 7:38 PM - Monitoring of the Chia blockchain detected the chain stall; no new blocks were created for a time period over 5 minutes. Automation restarted the timelords that Chia Network Inc. operates and this resolved the issue after the restart completed and new blocks were created by the farmers. Anyone that possesses a timelord could have also restarted the chain by restarting their timelord as well. 20 | 21 | 7:42 PM - New blocks being created again with approximately a 10 minutes gap between blocks during the chain stall. 22 | 23 | 24 | 01-31-2024 - The fix was added to the code base for version 2.2.0. 25 | 26 | 02-28-2024 - Version 2.2.0 was released that included the fix. 27 | 28 | 29 | ## Resolution and recovery 30 | 31 | Chia Network Inc. was the only entity running the ASIC timelords at this time and those timelords were restarted via automation when the chain stall was detected. 32 | This released the timelords and their connected full_nodes from the edge-case state they were in and new blocks were created shortly afterwards. However, had we not been monitoring, any running timelord that was restarted would have advanced the chain and the next new block would cause all other timelords to resume. 33 | 34 | The bug in the timelord code was fixed and introduced via the 2.2.0 release. 35 | 36 | The commit fixing the issue can be found here: 37 | https://github.com/Chia-Network/chia-blockchain/commit/221dab4f8a38199d38eb62041aae927a6c84341f 38 | 39 | -------------------------------------------------------------------------------- /2024-03/2024-03-07-unfinished_blocks-cache.md: -------------------------------------------------------------------------------- 1 | # v2.2.0 unfinished_block cache retrieval issue. 2 | 3 | ## Issue Summary 4 | 5 | If multiple farmers share plots and win, they will each produce a new unfinished_block with the same reward block hash. 6 | 7 | A new feature in 2.2.0 allows the network to propagate all unfinished_block variants (up to a limit) and pick the "best" one for infusion creating one full_block. 8 | 9 | When a node receives notification from a peer of a new_peak it requests the full_blocks it doesn’t have to sync to that peak height. The node will not ask for the block generator of the full_block when it already has the unfinished_block (with the needed block generator) in its cache as an optimization. 10 | 11 | The unfinished_block is pulled from the cache by its reward hash, and its cache insertion order determines which one is selected if there were multiple variants (with the same reward hash) of the unfinished_blocks in the cache. 12 | 13 | This would result in the wrong item being pulled from the cache in some scenarios, causing the reconstructed full_block to include the wrong block generator. In that case validation of the full_block would fail and the peer that provided the full_block would get banned for 10 seconds. 14 | In addition the wrong execution results might get pulled from the cache as well due to the same issue, resulting in failed validation and peer bans. 15 | 16 | 17 | ## Timeline (all times PST) 02/26/2024 - 03/06/2024. All times approximations 18 | 19 | 02-26-2024 - `INVALID_TRANSACTIONS_FILTER_HASH` block validation was reported for the Release Candidate version in Chia’s Discord Server by beta testers. 20 | 21 | 02-28-2024 - Version 2.2.0 was released to the public. 22 | 23 | 02-29-2024 - The cache retrieval issue was confirmed and v2.2.0 was recalled. 24 | 25 | 03-06-2024 - Version 2.2.1 was released that included the fix. 26 | 27 | 28 | ## Resolution and recovery 29 | 30 | A fix was introduced in v2.2.1 to make the unfinished_block selection from the cache deterministic by not only pulling the cached item by its reward block hash, but also taking the foliage hash into account when constructing the full_block. 31 | 32 | Version 2.2.0 with the cache retrieval issue was pulled, and version 2.2.1 that has the cache lookup fix was released as a replacement. 33 | 34 | For more information about the issue and resolution, visit: 35 | https://github.com/Chia-Network/chia-blockchain/pull/17622 36 | -------------------------------------------------------------------------------- /2023-05/2023-05-04-Discord-api-abuse.md: -------------------------------------------------------------------------------- 1 | # May 4th 2023 Discord API Abuse 2 | 3 | ## Issue Summary 4 | 5 | In order to verify direct messages have been disabled when joining our Discord server, our bot sends them a message. If it fails to go through, it's disabled. When this scaled to the hundreds of users who joined the server, Discord flagged the bot as spam and revoked its API access. 6 | 7 | ## Timeline (all times approximate) 8 | 9 | - 08:09 AM UTC Issue with accepting new users in the queue is identified by the Support team. 10 | - 02:13 PM UTC Issue is looked into, identified as the bot being temporarily banned from Discord's API. 11 | - 02:21 PM UTC Appeal is submitted to Discord, explaining our reliance on it for member verification. 12 | - 03:34 PM UTC Action plan is created to temporarily restore member queue functionality. Manual verification is used as a temporary solution in the meantime. 13 | - 04:59 PM UTC New bot is deployed with identical functionality except problematic API usage removed. 14 | - 05:00 PM UTC Member queue is back to a functional state, everything picks up as normal. 15 | 16 | ## Root Cause 17 | 18 | In order to verify that a user has disabled direct messages in the server's privacy settings, a bot has to attempt to send that user a message and check for an API permissions error. There is no way to directly check that user's privacy settings. The code was implemented this way, and worked fine on a small scale. 19 | 20 | However, when the server launched and the initial flood of hundreds of users begun, subtle but noticable rate limiting issues begun to occur. They were considered to be normal since it was due to the influx of users, and ignored. 21 | 22 | Even though users were instructed to disable direct messages, and sending them to each user was solely for verification purposes, the Discord API had no way to distinguish this from spam. The bot was flagged and restricted from using the API due to high usage of the direct message API endpoint. 23 | 24 | ## Resolution and Recovery 25 | 26 | In order to temporarily solve this issue, we manually verified users who were not able to join while the bot was offline. At the same time, we worked to create a second bot account in order to unblock user verification. 27 | 28 | The only difference is that the functionality for checking if direct messages are enabled was removed. This fix worked and doesn't run into ratelimiting issues like before, but does remove the validation. -------------------------------------------------------------------------------- /2023-05/2023-05-08-AGG_SIG_UNSAFE-can-mimic-AGG_SIG_ME-condition.md: -------------------------------------------------------------------------------- 1 | # AGG_SIG_UNSAFE can mimic AGG_SIG_ME condition 2 | 3 | ## Issue Summary 4 | 5 | Signatures are used to sign spendbundles (transactions), such that the puzzle execution can assert the signature is correct via AGG_SIG_UNSAFE or AGG_SIG_ME conditions. 6 | 7 | Both AGG_SIG_UNSAFE and AGG_SIG_ME conditions take a public_key and message, the difference is that AGG_SIG_ME adds the coin_id and additional_data (that is specific to the Chia blockchain) to the message that is signed. 8 | 9 | Due to the message's ability to have any value (within size limits), it could mimic the AGG_SIG_ME condition where coin_id and additional_data are appended to the message when using AGG_SIG_UNSAFE. 10 | 11 | The core problem was that a coin issuing an AGG_SIG_ME and including its signature in the aggregate could be replaced by a spend issuing an AGG_SIG_UNSAFE and including the attacked coin's coin ID, and including the "additional_data" used for AGG_SIG_ME, in the message it signed. 12 | 13 | The aggregate signature would be the same, even though the coin spend was replaced. 14 | 15 | The worst case scenario is joining two coins, only one of the spends creates the new output. 16 | 17 | An attacker could replace the spend creating the output with their own spend, and compensate for its signature by issuing an AGG_SIG_UNSAFE that mimics its AGG_SIG_ME, thus, gaining access to the coins for spending maliciously. 18 | 19 | The soft fork (that just took affect) limits AGG_SIG_UNSAFE in a way that makes it impossible to mimic an AGG_SIG_ME. they are now, effectively, in separate domains. 20 | 21 | ## Timeline (all times PST) 12/01/2022 - 05/07/2023. All times approximations 22 | 23 | 12/01/2022 - The issue was discovered by Chia. 24 | 25 | 02/15/2023 - Version 1.7.0 of the Chia blockchain was released that enforces AGG_SIG_UNSAFE messages are not allowed to end with the Chia blockchain specific additional_data at the soft fork block height of 3630000 by consensus rules. The mempool code enforces this restriction immediately. 26 | 27 | 05/07/2023 - The soft fork at block height 3630000 went into effect. 28 | 29 | ## Resolution and recovery 30 | 31 | This issue was addressed in 1.7.0 by: Making AGG_SIG_UNSAFE condition invalid if the message ends with additional_data at soft fork height 3630000. 32 | 33 | 34 | No AGG_SIG_UNSAFE messages ending with additional_data were found in the revealed puzzles / solutions on mainnet. 35 | 36 | -------------------------------------------------------------------------------- /2023-04/04-25-User_Account_Compromise.md: -------------------------------------------------------------------------------- 1 | # ISSUE NAME 2 | Password compromise of an employees' Okta account. 3 | 4 | ## Timeline (PST) 4/21/23-4/25/23. All times approximations 5 | _4/21/23_ 6 | 7 | 11:12 AM: An email was received by the user from Okta about a login attempt from an unfamiliar IP address and location. 8 | 9 | 11:14 AM: User had forwarded the email of a successful sign-on attempt of his Okta account to IPS. 10 | 11 | 11:18 AM: The user logged in to his Okta account. 12 | 13 | 11:31 AM: IPS locked the Okta account for the user. 14 | 15 | 11:36 AM: A member of the IPS team reported a potential compromise to the rest of the company. 16 | 17 | 11:43 AM: Okta logs are reviewed by IPS. 18 | 19 | 11:55 AM: It was confirmed that no successful login occurred, due to failure of MFA by the attacker. 20 | 21 | 12:18 PM: Questioning of the user began to understand how the account may have been compromised. 22 | 23 | 12:26 PM: User reported that he had connected to wifi approx. 2 hours earlier. No other abnormalities or changes noted. 24 | 25 | 12:27 PM: The user reported clicking on a link from LinkedIn. The link was from a trusted individual which prompted for 26 | login access to microsoft. They did use okta to product the login credentials for microsoft. However, an 27 | invalid credentials response was received and the user is now reaching out to determine the legitimacy of 28 | the LinkedIn message. 29 | 1:31 PM: Course of action is to leave the affected laptop off until a factory restore of the macbook can be completed. 30 | The user will also ensure that passwords are changed for 1Password and Okta. 31 | 1:34 PM: It was determined that the source of the message on LinkedIn was not from the trusted individual, but 32 | from someone else using her account. We have determined the link to be malicious. LinkedIn was sent an email 33 | from the trusted individual that fraudulent activity had occurred on their account. 34 | 1Password master password was reset. 35 | 2:57 PM: An IPS member re-enabled the user's Okta account and forced a password change. 36 | 37 | _4/24/23_ 38 | 39 | ~9:00 AM: The user's MacBook was erased and reset. 40 | 41 | -4/25/23_ 42 | 43 | 9:41 AM: SentinelOne installed and full disk scan was completed. 44 | 45 | ## Resolution and recovery 46 | 47 | The root cause was determined to be a malicious that was sent to the user from a trusted LinkedIn user. 48 | The trusted LinkedIn user was compromised, therefore, it came from a bad actor. Recovery included a reset of passwords 49 | and a erase and reset of the user's MacBook. SentinelOne has been installed and a full disk scan completed. 50 | -------------------------------------------------------------------------------- /2023-07/2023-07-21-unbound-CLVM-cost-restriction-on-mempool.md: -------------------------------------------------------------------------------- 1 | # unbound CLVM cost restriction on mempool 2 | 3 | ## Issue Summary 4 | 5 | Transactions that are included in a block are aggregated from one or more spendbundles. Inside spendbundles coins are spent, with the reveal of the puzzle and their solution of the coins. The puzzles and solutions consist of CLVM code that is assigned a cost, based on bytes of the code, and usage of operands and conditions. Each node on the network runs the CLVM code with limitations on how much resources the execution can use up via restrictions on the CLVM cost. 6 | 7 | The Chia default mempool code uses a CLVM cost restriction of 0.5 times the restriction that consensus uses (what is allowed in a block). The consensus constant is "MAX_BLOCK_COST_CLVM": 11000000000 8 | 9 | 10 | Version 1.7.1 of the mempool doesn’t include verification code that the CLVM cost of a MempoolItem has to be <= `constants.MAX_BLOCK_COST_CLVM * 0.5` 11 | This code was present in previous versions. 12 | 13 | This verification code was removed because it was assumed that running the MempoolItem as a block generator would perform this check already, making the code redundant. 14 | 15 | The chia_rs function called run_block_generator() contained a bug where if the byte cost exceeded the limit (5.5 billion in this case), the u64 wraps around and effectively removes the cost limit. 16 | 17 | The removal of the verification in the mempool code and the chia_rs bug combined allowed for a MempoolItem to have an unbound CLVM cost when the byte cost exceeded the limit. 18 | 19 | ## Timeline (all times PST) 04/23/2023 - 05/03/2023. All times approximations 20 | 21 | 04/23/2023 - Mempool items with a cost exceeding `constants.MAX_BLOCK_COST_CLVM * 0.5` were reported by a community member in keybase. 22 | 23 | 05/03/2023 - Version 1.8.0 of the Chia blockchain was released with fixes for the mempool code and chia_rs. 24 | 25 | 26 | ## Resolution and recovery 27 | 28 | This issue was addressed in 1.8.0 by: 29 | 30 | Reinstating the mempool verification code that the CLVM cost of a MempoolItem has to be <= `constants.MAX_BLOCK_COST_CLVM * 0.5`. 31 | 32 | Fixing the cost calculation in chia_rs when the byte cost exceeds the limit. 33 | 34 | 35 | The issue was present in mainnet and manifested itself by transactions being added to the mempool that exceeded the CLVM cost threshold. These transactions were not added to blocks as consensus checks on CLVM cost disallowed that. 36 | 37 | Chia monitored the mempool for these excessive CLVM cost mempool items and found that they were limited in scope and barely exceeded the threshold limit, causing very limited impact on nodes on the network during the time that the mempool allowed them. 38 | 39 | 40 | -------------------------------------------------------------------------------- /2024-03/2024-03-07-inefficient-mempool_item-selection.md: -------------------------------------------------------------------------------- 1 | # Inefficient Mempool Item selection. 2 | 3 | ## Issue Summary 4 | 5 | Blocks were not filled with transactions to capacity. 6 | 7 | Chia's consensus allows for 11 Billion in CLVM cost to be added to a block in transactions, where the CLVM cost for each transaction depends on their complexity, coins involved etc. 8 | When a farmer creates a block with transactions, it will select them from their mempool in order of `fee per cost`. 9 | 10 | The mempool item selection algorithm fills a new block by repeatedly selecting the highest-priority transaction from the mempool. 11 | When the next transaction would overfill the block, the node would consider the block finished and submit it to the network. 12 | 13 | This was inefficient in some cases. For example, if a block was less than half full, but the next transaction in the queue was large enough to overfill the block, the node would stop looking for more transactions. The resulting block would be filled to nowhere near capacity, even though it could have held many smaller transactions. 14 | 15 | 16 | The following situation occurred: 17 | - a transaction was in the mempool with CLVM cost that was close in size to the mempool selection algorithm maximum. 18 | - the transaction's `fee per cost` was not high enough for the mempool selection algorithm to add it first, yet reasonably high to have it inspected soon after. 19 | 20 | As a result the mempool selection algorithm would select other items first, and detecting the "stuck" large item would exceed the maximum stop there. 21 | This resulted in blocks to contain only a few transactions and the "stuck" transaction to not be included and cause the same issue in the subsequent blocks. 22 | 23 | 24 | ## Timeline (all times PST) 01/16/2024 - 02/28/2024. All times approximations 25 | 26 | 01-16-2024 - Chia detected sub optimal transaction inclusion in blocks due to the "stuck" transaction. 27 | 28 | 01-16-2024 - A Chia farmer of reasonable size was asked to include the "stuck" transaction in a block, and the farmer created such a block. 29 | 30 | 01-25-2024 - A better selection algorithm was introduced to the codebase at https://github.com/Chia-Network/chia-blockchain/pull/17346 31 | 32 | 02-28-2024 - Version 2.2.0 was released that included the fix. 33 | 34 | 35 | ## Resolution and recovery 36 | 37 | The immediate resolution was to ensure the "stuck" transaction to be included into a block by a Chia farmer. 38 | A Chia farmer of reasonable size was asked to do this and farmed such a block. 39 | 40 | To prevent such a situation in the future, a new block-fill algorithm was created that attempts to strike a balance, filling blocks efficiently, while also being careful not to take too much time in selecting transactions. 41 | -------------------------------------------------------------------------------- /2024-03/2024-03-07-unfinished_blocks-different-foliage.md: -------------------------------------------------------------------------------- 1 | # Unfinished_blocks with different Foliage. 2 | 3 | ## Issue Summary 4 | 5 | When a farmer runs multiple (redundant) full node and farmer instances, harvesting the same plots, it's possible to produce multiple different unfinished blocks with the same proof-of-space. Full nodes may have slightly different views of the mempool, because of quirks in transaction gossip. Therefore nodes may produce slightly different blocks. As an example, we consider a new transaction being propagated throughout the network that has reached one node but not the other when the block is created. 6 | 7 | Since the network sees these blocks as equally valid (because the reward block hash, i.e. proof-of-space) is the same, they will each propagate to nodes and which one is seen by which node is non deterministic because of the previously mentioned quirks in gossiping of transactions and blocks. This is a system designed to be unpredictable in an effort to provide defense against sybil type attacks. In a system with many competing timelords, different timelords are likely to infuse different versions of the unfinished blocks. Often timelords will not be aware of the existence of the other variants of the chain, because it's never propagated to their part of the network. 8 | 9 | These blocks, once infused, will form competing chains, eventually resolved by one side of the network re-orging to the other chain. 10 | 11 | 12 | ## Timeline (all times PST) 11/20/2023 - 02/28/2024. All times approximations 13 | 14 | 11-20-2023 - A researcher submitted a bug report via bugcrowd (https://bugcrowd.com/chia-network-bb) detailing the issue. 15 | 16 | 11-21-2023 - The ASIC timelords that sit geographically far apart were linked together by their full_nodes to ensure they see the same variant first, and only infuse that variant. 17 | 18 | 01-26-2024 - A fix was introduced to the codebase at https://github.com/Chia-Network/chia-blockchain/pull/17247 19 | 20 | 02-28-2024 - Version 2.2.0 was released that included the fix. 21 | 22 | 23 | ## Resolution and recovery 24 | 25 | The immediate resolution was to ensure the fastest timelords on the Chia network, currently the ASIC timelords, were linked together by their fullnodes. 26 | This ensured that if multiple unfinished_block variants (each with different Foliage) were propagating on the network, the timelords would see the same variant first 27 | and ignore the other variants. 28 | 29 | The fix in the codebase allows for the multiple variants (up to 3) to reach the timelords, where the timelord will infuse only one (the one with the lowest foliage transaction block hash). 30 | 31 | For more information about the issue and resolution, visit: 32 | https://github.com/Chia-Network/chia-blockchain/pull/17247 33 | -------------------------------------------------------------------------------- /2023-07/2023-07-05-email-gift-card-phish.md: -------------------------------------------------------------------------------- 1 | # Phishing Attempt Via Email 2 | ## Issue Summary 3 | An email was received by an employee from an individual who was impersonating an executive. The purpose of the 4 | email was to have employees purchase gift cards and send pictures to the impersonator. 5 | 6 | The email was first handled by one employee who had responded and requested assistance from an additional employee. 7 | The second employee questioned the legitimacy of the senders email address, at which time it was determined that 8 | the sender was not actually the executive. 9 | 10 | There was no malicious attachments or links discovered in scans of the email and SentinelOne scans of the employees' 11 | machines were performed. No issues found during at the conclusion of the scans. 12 | 13 | 14 | ## Timeline (all times PST) 7/5/2023-7/6/2023 15 | _7/5/2023_ 16 | 17 | 12:11 PM: Impersonator directs an email to the first employee, stating that he wants gift cards to be purchased. 18 | 19 | 12:18 PM: The employee responds that he/she is able to assist. 20 | 21 | 13:06 PM: The impersonator states that four gift cards are desired. 22 | 23 | 13:14 PM: The first employee copies the second employee on the email chain and informs the impersonator that he/she has no chia corporate credit card. 24 | 25 | 13:25 PM: Employee two asks for the amounts of the cards. 26 | 27 | 13:28 PM: The impersonator says to purchase four $500 Apple gift cards and how soon can they be purchased. 28 | 29 | 13:30 PM: The second employee responds that this can be purchased ASAP and asks about sending them to the various other employees' emails. 30 | 31 | 13:33 PM: The last email is from the impersonator who provides details of wanting pictures of the physical gift cards emailed to him/her and provides an attachment of a sample picture and instructions. 32 | 33 | 13:36 PM: Employee two forwards the email to IPS and informs the #warroom channel of the phishing email. 34 | 35 | 13:41 PM: The executive who was impersonated confirms in the #warroom channel that it is not him/her. 36 | 37 | 13:43 PM: The email was "Reported as phishing". 38 | 39 | 13:52 PM: Another employee announces the phishing email in our #general chat channel. 40 | 41 | 13:52 PM: It was determined that the attachment from employee two was not downloaded, but previewed in Gmail. 42 | 43 | 14:29 PM: Employee two's laptop is scanned with SentinelOne to ensure that no malicious code exists. 44 | 45 | 15:03 PM: Employee one's laptop scan is commenced with SentinelOne. 46 | 47 | 18:41 PM: All machines are verified to be clean from the SentinelOne. 48 | 49 | _7/6/23_ 50 | 51 | 07:05 AM: The attachment has been confirmed clean upon a scan for malware/viruses. 52 | 53 | ## Resolution and recovery 54 | 55 | The situation was contained by reporting the incident, informing personnel, and marking the email as phishing. 56 | Recovery was performed by scanning the email attachment and the two machines that interacted with the email. 57 | 58 | More training will be conducted in the future to increase awareness and level of knowledge of employees, 59 | improving early detection of phishing attempts. 60 | -------------------------------------------------------------------------------- /2025-12/React-CVE-2025-55182.md: -------------------------------------------------------------------------------- 1 | # React Server Components RCE (CVE-2025-55182) 2 | ## Issue Summary 3 | 4 | On 2025-12-03 (early morning, UTC), our CI tooling alerted us to a maximum-severity remote code execution vulnerability affecting React Server Components (CVE-2025-55182) with potential exposure via common frameworks (including Next.js which assigned CVE-2025-66478). The issue abused deserialization of payloads to Server Function endpoints. Per upstream guidance, the remediation path was to upgrade to patched versions. 5 | 6 | Impacted properties: 7 | - ch21.chia.net 8 | - chiafriends.xyz 9 | - Internal projects leveraging React Server Components 10 | 11 | All impacted services were mitigated on 2025-12-05 at 16:28 UTC, by applying upstream patches and redeploying. We have no evidence of exploitation in our environments. 12 | 13 | References: 14 | - The Register coverage: `https://www.theregister.com/2025/12/03/exploitation_is_imminent_react_vulnerability/` 15 | - CVE record: `https://www.cve.org/CVERecord?id=CVE-2025-55182` 16 | 17 | ## Timeline (all times GMT), 2025-12-03 → 2025-12-05. All times approximate 18 | 19 | - Early AM, 2025-12-03: CI tooling surfaces alerts for CVE-2025-55182 (React Server Components) and related Next.js bulletin (CVE-2025-66478). Team acknowledges incident and initiates triage. 20 | - 2025-12-03: Engineering reviews dependency exposure across ch21.chia.net, chiafriends.xyz, and internal projects; remediation plan is drafted to upgrade to vendor-patched versions and redeploy. 21 | - 2025-12-05 16:28: Applied mitigations and completed redeploys across impacted properties Post-deploy validation and monitoring initiated. 22 | 23 | ## Resolution and recovery 24 | 25 | We remediated by upgrading to vendor-patched React Server Components packages and relevant framework/plugin releases as recommended upstream (e.g., React RSC packages 19.0.1/19.1.2/19.2.1, framework vendor guidance), followed by full redeploys of impacted services. We verified expected package versions, executed smoke and basic security checks, and increased monitoring during the stabilization window. No indicators of compromise were observed. 26 | 27 | ## Impact 28 | - Scope: ch21.chia.net, chiafriends.xyz, and internal projects using React Server Components. 29 | - Customer impact: None observed. No service downtime attributable to this event; no known data exposure. 30 | 31 | ## Prevention / Follow-ups 32 | - Maintain proactive CI dependency alerting (validated effective for early detection in this incident). 33 | - Continue rapid assessment playbook for upstream CVEs with RCE potential in core frameworks. 34 | - Ensure environment parity testing paths for emergency framework upgrades. 35 | 36 | ## Q&A 37 | 38 | - What triggered the response? 39 | - CI dependency alerts flagged CVE-2025-55182 (and related framework advisories) early on 2025-12-03 (UTC). 40 | 41 | - What was the window of exposure? 42 | - From initial alert on 2025-12-03 (UTC) until mitigation completed at 2025-12-05 16:28 UTC. 43 | 44 | - Were customers impacted? 45 | - No impact observed; no evidence of exploitation. 46 | 47 | - How was it fixed? 48 | - Applied upstream patches for React Server Components and relevant framework integrations, then redeployed all impacted services. -------------------------------------------------------------------------------- /2021-05/2021-05-26-website-outage.md: -------------------------------------------------------------------------------- 1 | 2 | # Website Outage 5-26-21 3 | 4 | ## Issue Summary 5 | 6 | In an effort to modernize the Chia branch naming structures the default branch for the chia network website, chia.net was migrated from master, to main. 7 | When this type of change is made, the github sites background ci processes are supposed to engage and redeploy the site from the new renamed branch. Due to a bug in the ci system this did not occur. 8 | The under lying ci system, appears to have encountered a fatal error. 9 | Chia engineers attempted to rerun this ci process and these successive attempts to revive the site were also blocked by the ci pipeline bug. 10 | The github support team were able to reengage the stalled process with their administrative tools. 11 | Github engineers are continuing investigation for a root cause to this issue. At this time they have advised that it is unlikely to occur again, but if so are on standby to assist with recovery. 12 | 13 | ## Timeline (all times approximate) 14 | - 04:49 PM UTC Site repo branch Master, was renamed to Main 15 | - 04:50 PM UTC Site repo pages configuration is modified to reflect new default branch name 16 | - 04:52 PM UTC Site is hard down, chia operations engineers respond to alarms and being trouble shooting. 17 | - 04:53 PM UTC Operations engineers revert site repo changes 18 | - 04:56 PM UTC Operations engineers attempted to restart the stalled ci with several small commits to the site repo 19 | - 05:00 PM UTC Support ticket was opened with Github enterprise support team 20 | - 05:10 PM UTC Github support responds to ticket, begins investigation 21 | - 05:15 PM UTC Stalled CI job is identified 22 | - 05:30 PM UTC Further issues with the site are discovered, CI appears to have only run a portion of the site Setup 23 | - 05:45 PM UTC Github engineers have fully recover stalled and bugged CI pipelines, site begins to recover fully. 24 | - 06:00 PM UTC Site is fully recovered and Github support engineers begin investigation into the source of the ci bug. 25 | 26 | ## Root Cause 27 | 28 | A bug in the Github CI process responsible for updating github pages sites, stalled for an as of yet unknown reason. The change that enaged this CI job was a renaming of the Master default branch, to Main. 29 | Github support engineers continue investigation into the exact source of the stalled ci processes. 30 | 31 | ## Resolution and recovery 32 | 33 | Github support engineers were able to recover the full functionality of github pages CI, and return the site to its previous working state. This was accomplished by manually reengaging the underlying CI process after clearing out the previous stalled tasks. 34 | 35 | ## Corrective and Preventative Measures 36 | 37 | Going forward the Chia team is going to rebuild the site deployment process to allow for a more mature and modern release cycle. 38 | The ideal pattern would be one with quick failover from a fatal delivery error, and some quality of life improvements to make content and other changes simpler, lowering their blast radius. 39 | Most importantly Chia is aiming at a "five nines" solution for our website, as stability in our web presence is valuable to us and the goals of the chia project. 40 | Chia operations engineers are working to build these patterns at this moment. 41 | -------------------------------------------------------------------------------- /2022-01/2022-01-27-Ramp-Card.md: -------------------------------------------------------------------------------- 1 | # Chia Employee Credit Card Fraudulent Charge 2 | 3 | ## Issue Summary 4 | 5 | __Type of Incident Detected__: A fraudulent charge was detected on an employee's Ramp credit card. 6 | 7 | __Initial Concerns__: The employee received a notification for a charge on his company credit card that did not come from him. 8 | The notification said that the charge failed, due to the expiration date for the credit card being invalid. 9 | 10 | __Involved Parties__: 11 | Affected employee: Individual whose account was suspended. 12 | 13 | IPS team: Infrastructure, Platforms, and Security members who responded to the potential incident. 14 | 15 | ## Timeline (all times UTC ) 16 | 17 | Date/Time Stamps (Captured from keybase messages) 18 | 19 | | Day | Date (2022) | Time (UTC) | Event | 20 | |-----|-------------|-----------|-------| 21 | | Fri | Jan 28 | 02:34 | Affected employee provided a screenshot of a fraudulent charge attempt to #warroom. 22 | | Fri | Jan 28 | 02:39 | The employee reportated that he has two virtual cards. One that he used for online charges and another that is in his Apple Pay and Paypal accounts. The IPS team requested additional details about the the paypal account, source of the SMS notification, charges for Paypal, compromised account in sketch.com, and if Apple pay was used over NFC. 23 | | Fri | Jan 28 | 02:40 | The employee confirmed that he did use Apple Pay over NFC. 24 | | Fri | Jan 28 | 02:41 | The IPS team recommended contacting the first vendor about the strange charge attempt. 25 | | Fri | Jan 28 | 02:42 | Both company credit card are now locked. 26 | | Fri | Jan 28 | 02:43 | Charges by a second vendor was reported by the employee. 27 | | Fri | Jan 28 | 02:44 | A third vendor was mentioned and items were purchased in-person, using Apple pay. 28 | | Fri | Jan 28 | 02:45 | The employee was is instructed to obtain any audit logs and post them in the #warroom channel. 29 | | Fri | Jan 28 | 02:49 | The finance team is asked to inquire about fraudulent transaction information. 30 | | Fri | Jan 28 | 02:53 | The password for PayPal was updated. 31 | | Fri | Jan 28 | 03:27 | An unsuccessful attempt at obtaining PayPal audit logs was made. 32 | | Fri | Jan 28 | 03:45 | No weird or abnormal activity in Paypal was noted. 33 | | Fri | Jan 28 | 05:46 | The affected employee called Paypal. No abnormal activity or logins were noticed on their end. However, they cannot provide the log entries to the customer and do not have geo location setup for logins. 34 | | Fri | Jan 28 | 05:52 | A suggestion is made for the employee to make direct contact with one of the vendors. 35 | | Tue | Feb 01 | 19:12 | The credit card company does not have any information related to the charge, since it did not go through. Justin is going to reach out to sketch.com. 36 | | Mon | Feb 07 | 18:52 | The employee received another email stating that the credit card company cannot still has no details of the fraudulent transaction. 37 | | Tue | Feb 08 | 19:17 | Another vendor was contacted for any information they have. 38 | | Thur | Feb 10 | 17:52 | The vendor replied that they do not have any suspicious activity or had any data leaks. 39 | 40 | ## Root Cause 41 | 42 | The source of the fraudulent charge is still unknown at this time. 43 | ## Resolution and recovery 44 | 45 | The affected employee did great job of immediately reporting that he received a notification about a fraudulent charge. Our #warroom chat was promptly opened to investigate the nefarious charge attempt and ensure that all the proper steps were taken to contain the incident. 46 | ## Corrective and Preventative Measure 47 | 48 | Emphasize the importance of strictly adhering to our security policies and discourage the usage of third-party apps for company credit cards. 49 | -------------------------------------------------------------------------------- /2021-05/2021-05-10-WikiAttack.md: -------------------------------------------------------------------------------- 1 | ## Issue Summary 2 | 3 | At approximately 09:15:04 Monday May 10th (UTC), 2021, the user account *Bitmexbot*, edited the installer instruction page of the wiki to direct the link for the Windows installer to a third-party host unassociated with the Chia project. 4 | 5 | ## Timeline (all times UTC) Monday May 10th, 2021. All times approximations 6 | 7 | - 09:15:04 UTC an edit was made to the install documents on the GitHub wiki, pointing the windows install link to (MALICIOUS LINK!!!! VIRUS) ```https://rvmis.com/vendor/ChiaSetup-1.1.4.rar``` (this url is now defunct). 8 | 9 | - 09:30:00 UTC Reports of the change began to reach Chia tech support on keybase. 10 | 11 | - 10:00:00 UTC Chia operations personnel investigated the malicious edits, and began corrective actions along with internal discussion of the problem. Primarily locking the wiki, and combing for other malicious edits. 12 | 13 | - 10:03:52 UTC Mariano reverted the malicious change. 14 | 15 | ## Root Cause 16 | 17 | Malicious user redirected downloads to (MALICIOUS LINK!!!! VIRUS) ```https://rvmis.com/vendor/ChiaSetup-1.1.4.rar``` with an unreviewed wiki edit, via what we believe to be the Github web UI. 18 | 19 | ## Relevant Commit history 20 | 21 | commit 9b3a733a99f0ae5de143c6616bfcd4c945151420 22 | Author: Mariano Sorgente <3069354+mariano54@users.noreply.github.com> 23 | Date: Mon May 10 19:03:52 2021 +0900 24 | 25 | Updated INSTALL (markdown) 26 | 27 | commit a3c398b44333a0e9278c1059eaf3582b5822bf90 28 | Author: Chia-Network <53834631+Bitmexbot@users.noreply.github.com> 29 | Date: Mon May 10 11:15:04 2021 +0200 30 | 31 | Updated INSTALL (markdown) 32 | 33 | 34 | ## Resolution and recovery 35 | 36 | Mariano published a fix to the malicious URL immediately upon noticing the issue. - Total time to resolution approximately 49 minutes. 37 | 38 | We immediately contacted GitHub support to get the malicious user banned. This action took about 48 hours but the user is now banned from GitHub. 39 | 40 | We immediately locked the wiki and will no longer be sharing install links on publicly editable pages. 41 | 42 | Security professionals continue analysis of the malicious .RAR file and have findings about its possible origin, intent, and potential impact on any victims. 43 | 44 | ## Malicious file findings - Preliminary 45 | 46 | The RAR file contains two win32 executables, ChiaSetup-1.1.4.exe and mkvtool.exe along with several DLLs, TXT files, a font folder with fonts and a folder with object handlers for various file formats. The attackers appear to have deployed using the mkvtool to unpack an image and begin the installation of backdoors. Contents appear to include a network scanner and media rendering binaries (likely packaged with mkvtool). When the binaries execute, several Windows services are stopped (event logs, wmi, wer) and then installation to the user profile occurs (\AppData\Local\Temp). This directory contains a payload of C2 files: 47 | 48 | start.vbs 49 | get-content.ps1 50 | ready.ps1 51 | .ses (session token) 52 | 53 | Network traffic analysis of the installation showed traffic utilizing the Windows Background Task Scheduler service and Background Host Transfer binary. IP addresses pointed to cloud service provider IP ranges (Azure and IBM Cloud). These addresses are likely no longer used by the attackers. 54 | 55 | [VirusTotal details](https://www.virustotal.com/gui/file/476cdefcc0cd45525c7dc73a1dd0a1c97698c047dfaacdbf70ad32fc6bb65ee4/detection) a list of AV vendors catching this package. 56 | 57 | https://drive.google.com/file/d/1KHx5zL7mUOf2QFt9mWxexhE4sxa_Zxc6/view?usp=sharing 58 | 59 | https://drive.google.com/file/d/1qMWQK2n6LDKrmurfhTvhp0Q9X-Uzz6lS/view?usp=sharing 60 | 61 | ## Corrective and Preventative Measure 62 | 63 | The installation wiki page will be changed to no longer include installer links. The installers will live at chia.net/install and download.chia.net. The wiki will remain locked until that time. 64 | 65 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Chia Network Post-Mortem Repository 2 | 3 | This repository serves as a comprehensive record of significant incidents, issues, and their resolutions within the Chia Network ecosystem. It provides transparency, facilitates learning from past events, and helps prevent similar issues in the future. 4 | 5 | ## Purpose 6 | 7 | - Document significant incidents and their resolutions 8 | - Provide transparency to the Chia community 9 | - Enable learning from past events 10 | - Guide future incident response 11 | - Track improvements and preventive measures 12 | - Maintain historical record of network evolution 13 | 14 | ## Repository Structure 15 | 16 | ``` 17 | post-mortem/ 18 | ├── YYYY-MM/ # Chronological organization by month 19 | │ └── incident-name.md # Individual post-mortems 20 | ├── template.md # Standard template for new post-mortems 21 | └── assets/ # Supporting materials (graphs, logs, etc.) 22 | ``` 23 | 24 | ## How to Use This Repository 25 | 26 | ### Reading Post-Mortems 27 | - Browse chronologically by date in the YYYY-MM directories 28 | - Use GitHub's search functionality to find specific incidents 29 | - Check the assets directory for supporting materials 30 | 31 | ### Creating a New Post-Mortem 32 | 1. Copy `template.md` to create a new post-mortem 33 | 2. Place it in the appropriate YYYY-MM directory 34 | 3. Follow the template structure 35 | 4. Include relevant assets in the assets directory 36 | 5. Submit a pull request 37 | 38 | ## Post-Mortem Template 39 | 40 | Use `template.md` as the base for all new post-mortems. The template includes: 41 | - Issue Summary 42 | - Timeline 43 | - Resolution and Recovery 44 | - Additional Sections (as needed) 45 | - Q&A 46 | 47 | ## Example Post-Mortem Structure 48 | 49 | 50 | ```markdown 51 | # [Issue Name] 52 | 53 | ## Issue Summary 54 | 55 | *Provide a clear, concise overview of what happened, when it happened, and its impact. Include relevant metrics and scope.* 56 | 57 | This is an example issue that occurred between [start date] and [end date]. The incident affected [scope of impact] and resulted in [primary consequences]. 58 | 59 | ## Timeline (all times PST) [Date Range]. All times approximations 60 | 61 | *List key events in chronological order. Include specific timestamps when available. Note that times are approximate unless exact.* 62 | 63 | - [Time] First indication of the issue 64 | - [Time] Initial investigation begins 65 | - [Time] Root cause identified 66 | - [Time] Mitigation steps initiated 67 | - [Time] Resolution implemented 68 | - [Time] System returns to normal operation 69 | 70 | ## Root Cause 71 | 72 | *Explain the technical or operational cause of the incident. Be specific but avoid unnecessary technical details.* 73 | 74 | The issue was caused by [technical description of the root cause]. This led to [specific consequences]. 75 | 76 | ## Resolution and Recovery 77 | 78 | *List the steps taken to resolve the issue and restore normal operation. Include both immediate actions and long-term fixes.* 79 | 80 | 1. Immediate action taken 81 | 2. Additional steps implemented 82 | 3. System improvements made 83 | 4. Monitoring enhancements added 84 | 85 | ## Impact 86 | 87 | *Quantify the impact where possible. Include both technical and business impact.* 88 | 89 | - Duration: [time period] 90 | - Scope: [affected systems/users] 91 | - Severity: [impact level] 92 | - Business Impact: [if applicable] 93 | 94 | ## Preventive Measures 95 | 96 | *List specific actions taken to prevent similar incidents in the future.* 97 | 98 | 1. System improvements implemented 99 | 2. Process changes made 100 | 3. Monitoring enhancements added 101 | 4. Documentation updates completed 102 | 103 | ## Q&A 104 | 105 | *Address common questions about the incident. Focus on community concerns and technical details.* 106 | 107 | Q: [Common question about the incident] 108 | A: [Clear, concise answer] 109 | 110 | Q: [Another common question] 111 | A: [Clear, concise answer] 112 | ``` 113 | 114 | ## Guidelines 115 | 116 | - Be thorough but concise 117 | - Include specific timestamps 118 | - Document both technical and user impact 119 | - Provide clear resolution steps 120 | - Include preventive measures 121 | - Add relevant metrics and data 122 | - Cross-reference related incidents 123 | 124 | ## Contact 125 | 126 | - For questions about this repository contact the Chia Network team on the official Discord (Discord:https://discord.gg/chia) 127 | - To report a security issue please use our bug bounty program (https://hackerone.com/chia_network) -------------------------------------------------------------------------------- /2022-01/2022-01-20-EmployeeCompromise.md: -------------------------------------------------------------------------------- 1 | # Suspected Chia Employee Compromise 2 | 3 | ## Issue Summary 4 | 5 | __Type of Incident Detected__: Potential compromise of employee's Google Admin Account 6 | 7 | __Initial Concerns__: On Thursday, January 20th, 2022 at 22:04 (UTC) an employee's account was suspended by an admin account, per the audit logs. 8 | However, the admin did not perform the suspension of the account and the action originated from an IP address associated with an 9 | AWS EC2 instance. 10 | 11 | __Involved Parties__: 12 | Affected employee: Individual whose account was suspended. 13 | 14 | Admin employee: Individual whose account performed the suspension action. 15 | 16 | IPS team: Infrastructure, Platforms, and Security members who responded to the potential incident. 17 | 18 | ## Timeline (all times UTC ) 19 | 20 | Date/Time Stamps (Captured from keybase messages) 21 | 22 | | Day | Date (2022) | Time (UTC) | Event | 23 | |-----|-------------|-----------|-------| 24 | | Thur | Jan 20 | 22:40 | Affected employee logged out of Google Workspace account and received a prompt that his account was disabled, upon attempting to log back in. 25 | | Thur | Jan 20 | 22:06 | IPS team observed that the affected employee's Google Workspace account was suspended by another employee with administrator rights, when reviewing the audit logs in Google Admin. No abnormal login activity was detected for either employee. 26 | | Thur | Jan 20 | 22:07 | The IPS team called the admin employee and confirmed that he did not suspend the account. 27 | | Thur | Jan 20 | 22:08 | The IPS team suspended the Okta account of admin employee. 28 | | Thur | Jan 20 | 22:09 | The admin employee's Google Workspace account was then disabled. An IP address associated with the account suspension activity was determined to have a source from an Amazon Web Services (AWS) EC2 instance. 29 | | Thur | Jan 20 | 22:19 | Communications were initiated with Google to take a deeper look into the issue. 30 | | Thur | Jan 20 | 22:23 | The IPS team coordinated with AWS technical support chat to identify any information related to the AWS IP address. 31 | | Thur | Jan 20 | 22:40 | Aside from the 'cookies reset and forced re-login' and 'Account suspended' actions in the audit logs, no other abnormal activity was present. 32 | | Thur | Jan 20 | 23:15 | A zoom call with the AWS Incident Response Support team was started. The AWS team suggested verifying no abnormal activity exists in CloudTrail for Chia Network's two AWS accounts. They made the suggestion to create a SEV2 ticket and send an email to abuse@amazonaws.com for further troubleshooting. 33 | | Thur | Jan 20 | 23:17 | No abnormal activity was found in CloudTrail. 34 | | Thur | Jan 20 | 23:20 | An AWS SEV2 support ticket was created. 35 | | Thur | Jan 20 | 23:22 | An AWS Abuse case was created via email request. 36 | | Thur | Jan 20 | 23:55 | The admin employee joined a zoom meeting with IPS team, to discuss the current situation and details. 37 | | Fri | Jan 21 | 00:20 | Google Admin support chat was initiated for further investigation. 38 | | Fri | Jan 21 | 01:31 | Cleared all active sessions and third party integrations from admin users' Google account. The admin employee was removed from superadmin access in Google. 39 | | Fri | Jan 21 | 01:34 | Enabled admin employee's Google account only, leaving his Okta account disabled, allowing for the collection of email and access to calendar. 40 | | Fri | Jan 21 | 16:49 | The admin employee's Okta account was enabled. 41 | | Fri | Jan 21 | 17:02 | The affected employee's Google account was enabled. 42 | | Fri | Jan 21 | 18:20 | The admin employee mentioned that the cause of the account suspension may have been due to a Rippling integration. 43 | | Fri | Jan 21 | 23:01 | IPS reached out to Rippling to discuss situation. They confirmed that the affected employee's account was suspended by Rippling, due to account information being changed and the application being integrated with the admin employee's Google Workspace account. 44 | 45 | ## Root Cause 46 | 47 | The affected employee was converted from a 1099 employee to a W-2 employee in Rippling. Due to the current application integration of Rippling into the admin employee's Google Workspace account, the account was automatically suspended when the information changed and access to Google Workspace was temporarily removed. 48 | 49 | ## Resolution and recovery 50 | 51 | Immediate communication was established when the affected employee noticed that his account was disabled. The IPS team took prompt action to contain the situation and seek a root cause of the incident. 52 | 53 | ## Corrective and Preventative Measure 54 | 55 | The IPS team is continuing to improve and refine our response process for handling security incidents. 56 | -------------------------------------------------------------------------------- /2021-10/2021-10-31-EmployeeCompromise.md: -------------------------------------------------------------------------------- 1 | # Suspected Chia Employee Compromise 2 | 3 | ## Issue Summary 4 | 5 | __Type of Incident Detected__: Unexplained spend of one mojo and unexplained commands found in history. 6 | 7 | __Initial Concerns__: On Sunday, October 31 at 21:33 PT, a user discovered he had one mojo deducted from his wallet that he does not remember spending. Due to the ongoing Dust Storm attack, this caused concern. Upon more investigation, we also discovered unexplained commands in the history that the user did not initially recall using. 8 | 9 | ## Timeline (all times PT ) 10 | 11 | Date/Time Stamps (Captured from keybase messages, calls, and face-to-face communication) 12 | 13 | | Day | Date (2021) | Time (PT) | Event | 14 | |-----|-------------|-----------|-------| 15 | | Sun | Oct 31 | 21:33 | Affected user posts in Keybase, “interesting. My machine sent out 1 mojo to somewhere which I know I didn’t do.” 16 | | Sun | Oct 31 | 21:34 | Chia employee starts troubleshooting with a via Keybase direct messages 17 | | Sun | Oct 31 | 21:35 | User claims he hasn’t created any Plot NFT: “It’s my old Machine I recently turned it back on. Trying to figure out how this happened since this is using og plots and never joined any pools” 18 | | Sun | Oct 31 | 21:43 | Verified 1 mojo did leave via xchscan.com 19 | | Sun | Oct 31 | 21:46 | Verified running latest Chia software 1.2.10 20 | | Sun | Oct 31 | 21:53 | Verified git commit hash for is correct 21 | | Sun | Oct 31 | 21:54 | Employee asks the user to shut down his machine. 22 | | Sun | Oct 31 | 21:54 | Employee brings in Sr. Director of Devops to the conversation. 23 | | Sun | Oct 31 | 22:17 | It was verified that no nefarious community PR was checked into Chia software 1.2.10 24 | | Sun | Oct 31 | 22:32 | User recalls the computer was also used for a different blockchain game. 25 | | Sun | Oct 31 | 22:43 | User is asked to to run “history” of past shell commands 26 | | Sun | Oct 31 | 22:52 | User posts 4 screen shots of his “history” 27 | | Sun | Oct 31 | 22:59 | The employee notices there were commands listed after the machine was shut down 28 | | Sun | Oct 31 | 23:05 | Agreement is made that there is enough suspicion on the user’s machine to require digital forensics 29 | | Sun | Oct 31 | 23:16 | It is identified that the user uses shared internet from the apartment complex 30 | | Sun | Oct 31 | 23:17 | A recommendation is provided for the user to have his significant other report the security incident to her employer 31 | | Sun | Oct 31 | 23:25 | The Chia employee is approved to fly to the user’s location 32 | | Mon | Nov 1 | 17:00 | The employee starts setting up brand-new hardware for the user 33 | | Mon | Nov 1 | 17:32 | The employee asks the user to document all actions performed leading up to the event and recommends watching videos on creating a Plot NFT 34 | | Mon | Nov 1 | 17:32 | Confirmation is received from the user’s significant other’s company that there was no compromise of her work equipment 35 | | Mon | Nov 1 | 19:10 | In Keybase, the user writes: “The command you told me to run, I ran it in a separate command line window, and then I closed out my client. This may be the reason what happened to the command line order. I took a look at the command line history, those are the exact commands I ran when I turned the machine on last friday.” 36 | | Mon | Nov 1 | 20:56 | The Sr. Director of DevOps concurs that the employee can do some initial digital forensics on the suspected system 37 | | Mon | Nov 1 | 21:54 | The employee verified the security incident was a False Positive and has reproduced the problem twice. It was also determined that the user did create a Plot NFT 38 | 39 | 40 | 41 | ## Root Cause 42 | 43 | It was discovered that a Plot NFT was created, explaining how the one mojo was spent. Additionally, the user had two terminal windows open, one running the Chia GUI from CLI and another one typing the unexplained commands. When you power down the machine without closing the terminal windows or Chia GUI, Ubuntu arranges the history out of order. 44 | 45 | 46 | 47 | ## Resolution and recovery 48 | 49 | The workforce members responded quickly, while also responding to another workplace incident. The time between the end-user reporting the incident until verifying it was a false positive took approximately twenty-four hours. This included rapid replacement of new equipment and having remote hands onsite, completing initial digital forensics. 50 | 51 | ## Corrective and Preventative Measure 52 | 53 | Develop better techniques in questioning the end-user, allowing for prompt detection of the security incident as a false positive. 54 | Hire a professional SecOps person and create formal processes/guidance for security incidents. 55 | Add a feature to the Chia Graphical User Interface (GUI) to proactively inform the user that one mojo will be spent to create a Plot NFT. 56 | -------------------------------------------------------------------------------- /2023-08/2023-08-04-compromised-self-hosted-runners.md: -------------------------------------------------------------------------------- 1 | # Compromised self-hosted runners. 2 | 3 | ## Issue Summary 4 | 5 | Chia uses Github Actions for its continuous integration (CI) and continuous delivery (CD). 6 | Host machines that execute jobs of Github Actions Workflows are called runners, and these can either be hosted by Github or self-hosted; i.e. Chia maintains a machine in a datacenter that executes the workflows and uses self-hosted runners that are non-ephemeral, meaning after they complete a task, they move onto the next. 7 | 8 | Workflow jobs have many functions: running tests, benchmarks, test coverage, signing and deploying packages, and more. This means that they need to have access to github secrets, sensitive information such as access tokens, and also signing certificates that are used to sign the Chia software to prove these are Chia Network Inc. approved artifacts. This information is siloed based on github permissioning. 9 | 10 | When a Chia github repository is forked, workflows can be modified and executed on Chia’s self-hosted runners if criteria are met that are set in github organization/repository settings. 11 | The setting that was enabled at the time of the compromise allowed a contributor to Chia-Network/chia-blockchain (meaning they have previously submitted a PR that was approved and merged) to modify a workflow in their fork and configure it to run on the self-hosted runner attached to Chia-Network/chia-blockchain when they open a pull request to Chia-Network/chia-blockchain. 12 | 13 | These workflows can be used to obtain persistence on the self-hosted runner and steal secrets from or tamper with subsequent builds. 14 | 15 | Chia strongly believes the community owns the blockchain codebase and tries to balance the ease of community contributions to the codebase while keeping it secure. In this case, the github configuration setting inadvertently led to the compromised self-hosted runners. 16 | 17 | 18 | ## Timeline (all times PST) 07/27/2023 - 07/31/2023. All times approximations 19 | 20 | 07-27-2023 2:14 pm - Attacker’s PR was approved, making him a contributor via 21 | https://github.com/Chia-Network/chia-blockchain/pull/15877 22 | 23 | 07-29-2023 11:51 am - Unauthorized access to the runners was gained via https://github.com/Chia-Network/chia-blockchain/pull/15896 24 | 25 | 07-30-2023 2:42 am - Chia changed the github setting from [Require approval for first-time contributors] to [Require approval for all outside contributors] for Chia-Network/chia-blockchain. 26 | 27 | 07-30-2023 2:55 am - Chia changed the github setting from [Require approval for first-time contributors] to [Require approval for all outside contributors] for Chia-Network organization. 28 | 29 | 07-30-2023 3:22 am - Chia reported the github user to github support for malicious probing behavior. 30 | 31 | 07-31-2023 9:13 am - a bugcrowd report detailing the self-hosted runner compromise was submitted. 32 | 33 | 07-31-2023 11:00 am - Chia confirmed the bugcrowd issue and engaged with the attacker to evaluate its impact and start remediation. 34 | 35 | 07-31-2023 11:20 am - all self-hosted runners were stopped, removing the gained unauthorized access. 36 | 37 | 38 | 39 | ## Resolution and recovery 40 | 41 | Chia has been in contact with Adnan Khan, the researcher that submitted the bugcrowd report and gained access to the self-hosted runners. In addition to the detailed bugcrowd report, he has also provided a post-action summary. 42 | 43 | Chia plans to have a 3rd party security company perform a “red team” (researchers trying to overcome cyber security controls) assessment, based on the generous donation of time Adnan Khan put into providing Chia information on the attack. 44 | 45 | Chia Network is actively working on getting the compromised hosts rebuilt and back into working order. 46 | 47 | All certificates and secrets are being rotated. While professional researchers typically do not access the secrets on bug bounty systems and we only have evidence confirming a small portion of secrets being at risk, we are proceeding as if all secrets were exfiltrated. This means new builds from Chia’s continuous integration (CI) should be viewed with a skeptical eye from a signed installer perspective until new signature certificates have been rotated in. 48 | 49 | The procedure for rotating certificates is different per Operating System; Apple certificates are being updated at time of writing. The Linux ecosystem differs from Apple and Windows and is already re-secured. Windows certificates are pending at this time, because of an industry wide technology change in certificate management that required a rebuilding of some internal build time patterns. 50 | 51 | All other Chia Network Inc secrets are also being rotated and replaced with new as well. 52 | 53 | In addition to secret rotation work there are also changes coming to how our runners are managed to get away from static (or non ephemeral) runners wherever possible. Further segmentation of internal and public facing hosts is also under way. Our goal is to remove as much attack surface as possible with these upgrades. 54 | 55 | 56 | 57 | -------------------------------------------------------------------------------- /2025-02/2025-02-25-block_production_impact_due_FastForward_mempool_issues.md: -------------------------------------------------------------------------------- 1 | 2 | # Block creation impacted due to mempool issues. 3 | 4 | ## Issue Summary 5 | 6 | Farmers that are eligible to create the next block have a limited time (approximately 28 seconds) to create an unfinished block and via network propagation get this to a timelord that will infuse the unfinished block to construct the finished block. 7 | 8 | Farmers will take transactions from the mempool ordered by fee per cost, and add them into the unfinished block as a way to benefit from fees attached to transactions that go to the farmer making the block. 9 | 10 | CNI detected that while the mempool was completely full, transactions were not being included in blocks. Additionally, block production slowed down versus expected production rate. It was determined that farmers eligible to create blocks were not creating unfinished blocks in time due to several bugs in the mempool logic around fast forward spends. 11 | 12 | Fast forward spends are a way for a singleton transaction to be able to apply to future versions of the singleton. 13 | 14 | Fast forward spends enable offer files to refer to singletons by coin ID, even though the coin ID changes for every state update. 15 | 16 | In addition, fast forward spends are used for vaults. Because the vault singleton is used to authorize all coins belonging to it, it would be too limiting to only be able to have one vault spend “in-flight” at a time. Fast forward allows multiple spends of the vault singleton to be in the mempool concurrently, and farmers deduplicate and rebase them sequentially as they create blocks. 17 | 18 | 19 | CNI determined the following issues were the root cause of the block creation slow down: 20 | * 1. Fast forward spends don’t become invalid just because their coin is spent. They can be fast-forwarded to the latest version of the singleton. Fast-forward spends rely on other parts of the spend bundle for being evicted (e.g. that coin is spent by a block). We failed to consider the case where the only spend in a bundle was a fast-forward spend, it would not be evicted after the singleton was melted. 21 | * 2. FF-Spends enter the mempool even if their coin is already spent. When we form the block they will be fast-forwarded on top of the latest version of the singleton. However, we failed to validate that the singleton still existed. I.e. we would accept singleton spends into the mempool for singletons which have been melted and no longer exist. 22 | * 3. When we formed a block, FF spends would be updated to spend the most recent coin of a singleton. This update was done in place, causing the mempool to change and propagate the fast-forwarded versions of the mempool item to the rest of the network. This would cause the mempool item to duplicate (double in numbers) every time a farmer tried to form a block. 23 | 24 | With these 3 bugs, someone submitting a singleton spend that’s eligible for fast forward, without bundling it with any other spend would: 25 | * Be considered valid and enter the mempool and propagate throughout the network. 26 | * Whenever a farmer would make a block, it would multiply the spend, propagating the new transactions throughout the network (3) 27 | * Trying to stop the spread by melting the singleton makes the transactions not go through, but the spends were not evicted from the mempool (2). In our scenario, mempool was full of these spends when this happened. All of these spends failed during block creation, causing farmers to time-out before finding any valid transactions. 28 | * Farmers trying to create a block with the affected mempool items would experience performance issues leading them to miss the cutoff time for their block to be included in the blockchain. 29 | 30 | 31 | 32 | ## Timeline (all times PST) 02/13/2025 - 02/19/2025. All times approximations 33 | 34 | 02-13-2025 11:00 pm - Mempool spendbundle count started slowly increasing. 35 | 36 | 02-14-2025 11:30 am - Mempool cost hits max cost. 37 | 38 | 02-14-2025 1:10 pm - Mempool spendbundle count plateaus at ~6200. 39 | 40 | 02-14-2025 4:45 pm - The Chia blockchain netspace started decreasing and timelord infused blocks per minute had a marked drop. 41 | 42 | 02-14-2025 11:00 pm - The cause of the issues were diagnosed by CNI and a hotfix release of the chia blockchain software was prepared. 43 | 44 | 02-15-2025 1:30 am - The hotfix release of the Chia blockchain software build process was initiated. 45 | 46 | 02-15-2025 3:00 am - The hotfix release of the Chia blockchain software deployment completed as version 2.5.1 47 | 48 | 02-19-2025 3:15 pm - Version 2.5.2 was released that included the bug fixes for the Fast Forward issues detailed in the Issue Summary. 49 | 50 | 51 | 52 | ## Resolution and recovery 53 | 54 | Due to the decentralized nature of the Chia blockchain with different configurations and versions distributed over many nodes, the majority of the blockchain netspace was not impacted by the Fast Forward issues. 55 | 56 | Version 2.5.1 of the Chia blockchain contains a workaround that caps the unfinished block creation time, to ensure the unfinished block reaches a timelord in a timely manner. The blockchain netspace started recovery towards previous levels after the v2.5.1 release. 57 | 58 | Version 2.5.2 included fixes that as a result invalidated the mempoolItems from the Fast Forward issues and the mempool was cleared of these items along with the userbase uptake of v2.5.2. 59 | * 1. SpendBundles submitted to the mempool that only contained fast-forward spends are rejected. 60 | * 2. When a spend is “rebased” on top of the latest version of the singleton, we don’t update the item in the mempool, risking distributing this new copy to the network. 61 | * 3. When ingesting spend bundles into the mempool, any spend eligible for fast-forward is validated to ensure the latest version of this singleton still exists (with the given puzzle hash). -------------------------------------------------------------------------------- /2022-06/2022-06-28-ProxyAbuseReport.md: -------------------------------------------------------------------------------- 1 | # Data Layer Testing Proxy Abuse Report 2 | 3 | ## Issue Summary 4 | 5 | On June 28, 2022, Amazon Web Services (AWS) alerted Chia via e-mail that our Data Layer reverse proxy EC2 testing instance had received an abuse complaint. After investigation, it was determined that the proxy configuration contained an IP address of an EC2 instance that had recently been shut down and the proxy was still forwarding web traffic to this IP. The IP address had since been reassigned to another customer by AWS and some of the traffic our proxy forwarded was a common vulnerability probe. The issue was resolved by reconfiguring the proxy with the latest Data Layer testing IP addresses. 6 | 7 | ## Involved Parties 8 | 9 | Data Layer team: Chia employees and partners working on the Data Layer application. 10 | 11 | IPS team: Infrastructure, Platforms, and Security members who responded to the potential incident and collaborate closely with the Data Layer Team. 12 | 13 | AWS: Amazon Web Services, the cloud hosting provider. 14 | 15 | ## Timeline (all times PST). All times approximations 16 | 17 | | Day | Date (2022) | Time (PST) | Event | 18 | |-----|-------------|-----------|-------| 19 | | Mon | Jun 27 | 13:47 | Data Layer team and IPS turn off one Data Layer testing EC2 instance at IP address w.x.y.z (IP address redacted) via the AWS dashboard | 20 | | Mon | Jun 27 | 14:07 | IPS turns back on the Data Layer EC2instance at request of Data Layer team to retrieve data from instance. The instance comes online with a new IP address | 21 | | Tues | Jun 28 | 20:28 | Many requests are made to the Data Layer proxy indicative of a security scan, including requests to `/.env` | 22 | | Tues | Jun 28 | 20:36 | AWS sends an automated email to Chia notifying of an abuse complaint for IP address w.x.y.z who received a request to `/.env` from the IP address of the Data Layer proxy. This was classified as `exploit:gen/dot_env_leak` by the automated system | 23 | | Tues | Jun 28 | 21:20 | IPS initiates a War-room to coordinate response to the reported incident | 24 | | Tues | Jun 28 | 21:30 | IPS turns off all EC2 instances in the impacted AWS account as a precautionary measure until further investigation can be done | 25 | | Tues | Jun 28 | 21:45 | Outside security experts consulted and on standby to assist | 26 | | Tues | Jun 28 | 21:55 | Reply sent to AWS acknowledging incident 27 | | Tues | Jun 28 | 22:21 | Active incident declared mitigated by the shutdown of the EC2 instances and forensics work to resume in morning | 28 | | Wed | Jun 29 | 7:05 | IPS team begins investigating and finds requests to `.env` files in the proxy logs that match the timestamps of the AWS report | 29 | | Wed | Jun 29 | 10:50 | IPS team confirms IP address that lodged abuse complaint matches IP address of Data Layer testing EC2 instance that had been shut down on Monday Jun 27. This IP is found in the reverse proxy configuration as it hadn’t been removed when the EC2 instance was shut down and was therefore forwarding traffic to an IP that Chia no longer leased from AWS. | 30 | | Wed | Jun 29 | 11:34 | IPS team emails AWS with explanation of incident | 31 | | Wed | Jun 29 | 11:44 | Data Layer testing nodes are brought back online | 32 | | Wed | Jun 29 | 11:54 | Automation run against Data Layer AWS account to update IP addresses on the proxy and enforce state | 33 | | Wed | Jun 29 | 12:23 | AWS closes abuse case and marks the issue as solved | 34 | | Wed | Jun 29 | 12:35 | All Data Layer testing nodes confirmed back online with updated routing | 35 | 36 | ## Root Cause 37 | 38 | In the current data layer testing architecture, a single Nginx reverse proxy handles distributing requests to the various testing nodes. This Nginx server manages SSL certificates and has automated provisioning to automatically update the destination IP addresses of the individual Data Layer EC2 testing instances whenever an instance is created or destroyed. The automation is only run to create or destroy an instance and if an instance has an IP address change in-between automation runs, the proxy will point to the old address until the automation is run again manually or triggered by a change to the Github repository. 39 | 40 | A number of requests similar to the following are visible in the Nginx logs of the reverse proxy (IP address redacted): 41 | 42 | ``` 43 | x.x.x.x (x.x.x.x) - - [29/Jun/2022:03:28:53 +0000] "GET /.env HTTP/1.1" 200 884 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2226.0 Safari/537.36" 44 | ``` 45 | 46 | The time on these requests matches the time of the AWS abuse report. The abuse report states the payload is a “exploit:gen/dot_env_leak” which we assume means a request for an unauthorized request for a .env file. 47 | 48 | On Monday, two days prior, we had shut down one of the EC2 testing instances in preparation for eventual decommissioning. When an EC2 instance is shut down, its IP address is released and reassigned to the AWS pool for reassignment. The IP address that filed the abuse complaint is the IP address of the instance we shut down Monday, which is the IP address the proxy was still forwarding to for this specific domain name. 49 | 50 | This request for a `.env` file was part of a random security scan originating from a inexpensive cloud server. These scans are very common on all web-facing IP addresses and do not indicate a compromise or a targeted attack. Our proxy forwarded these requests to the old IP we no longer owned as the proxy hadn’t been updated when the testing EC2 instance was shut down. 51 | 52 | ## Corrective and Preventative Measure 53 | 54 | A new shutdown and decommissioning process will be created that removes the IP addresses from the proxy when an EC2 instance is taken offline. This will likely involve a combination of manual actions (to shut down the instance) and automated actions (to remove the IP from the proxy). It may be that taking a snapshot and removing the instance entirely from automation (thereby removing it from AWS and the proxy simultaneously) is the best workflow. 55 | 56 | Additionally, these Data Layer test instances are planned for replacement later this year with a new architecture that will not use a proxy and make this no longer relevant. 57 | 58 | -------------------------------------------------------------------------------- /2025-09/2025-09-28-npm-supply-chain-attacks.md: -------------------------------------------------------------------------------- 1 | # npm Supply Chain Attacks (September 2025) 2 | ## Issue Summary 3 | 4 | In September 2025, the npm ecosystem experienced two significant supply-chain compromises affecting widely used packages. On September 8, a maintainer account was phished, and malicious versions of at least 18 high-download packages (e.g., `chalk`, `debug`, `ansi-regex`, `supports-color`, `strip-ansi`) were published, injecting code that attempted to hijack cryptocurrency transactions in browser contexts. Around September 15–16, a second incident (“Shai‑Hulud”) introduced a worm-like postinstall payload that harvested secrets and attempted to republish tainted packages under maintainers’ namespaces, spreading to 100+ additional packages before removal. Community response and registry action limited direct losses, but the events highlighted systemic risks in open-source supply chains. 5 | 6 | ## Timeline (all times PST) DATE RANGE. All times approximations 7 | 8 | - 06:15 AM Sep 08, 2025: First malicious versions of popular npm packages (e.g., `chalk@5.6.1`, `debug@4.4.2`) appear following maintainer account compromise via phishing 9 | - 07:00–08:00 AM Sep 08, 2025: Security researchers flag anomalies; warnings circulate on social media and security vendor feeds 10 | - 10:00 AM Sep 08, 2025: npm begins yanking malicious versions; affected maintainers start revocation and cleanup 11 | - Sep 09–12, 2025: Additional impacted packages identified; advisories and IOCs disseminated 12 | - Sep 15–16, 2025: “Shai‑Hulud” worm campaign publishes malicious postinstall scripts to numerous packages; payload exfiltrates secrets and attempts self‑propagation 13 | - Sep 16–18, 2025: Registry takedowns; organizations rotate tokens and audit pipelines; expanded impacted-package lists published 14 | 15 | ## Resolution and recovery 16 | 17 | - Tainted package versions were removed from npm; owners reclaimed accounts and published clean releases 18 | - Maintainers and downstreams rotated npm/GitHub/cloud credentials; CI agents and caches were purged and rebuilt 19 | - Organizations audited dependency locks, rolled back to known‑good versions, and implemented temporary blocks on install scripts 20 | - Security advisories, IOCs, and impacted package/version lists were shared across the community to accelerate triage 21 | 22 | ### Root cause 23 | - Human‑targeted phishing defeated a maintainer’s MFA via an adversary‑in‑the‑middle flow, yielding a valid session to publish malicious updates 24 | - For the worm campaign, compromised tokens and weak credential hygiene enabled propagation: postinstall payloads harvested secrets and used them to push trojanized patch releases under maintainers’ namespaces 25 | 26 | ### Impact 27 | - While Chia network was not impacted, Chia network did have a near miss where we would have installed one of the common vulnerable dependencies had we not been alerted and checked pending package updates for all our repos. We did not install a vulnerable depdency but the near miss factor alerted us to some possible flaws in our process and we have taken steps to correct them. 28 | 29 | ### Detection 30 | - Community monitoring, vendor intel feeds, and social alerts (including maintainers and security researchers) identified anomalous diffs and behaviors quickly 31 | 32 | 33 | ### Preventative and corrective actions 34 | - Enforce phishing‑resistant MFA (FIDO2/WebAuthn) for npm/GitHub maintainers and high‑impact publishers 35 | - Pin dependencies with lockfiles; prefer `npm ci` in CI; avoid floating ranges in production builds 36 | - Block install‑time scripts by default in CI (`npm config set ignore-scripts true`) and enable selectively per allowlist 37 | - Use SCA and malware scanning pre‑install (e.g., quarantine obfuscated diffs, network hooks, or wallet/API interception code in utility packages) 38 | - Segregate and minimize tokens; rotate credentials after incidents; scope tokens to least privilege 39 | - Periodically purge private registries’ caches to prevent serving yanked upstream artifacts 40 | - Generate and retain SBOMs to rapidly locate affected builds for rollback 41 | - We have implemented a more robust tool chain for repository based languages and protocols that include pinned versions of dependencies and enhanced monitoring and upgrade practices. 42 | 43 | ## Q&A 44 | 45 | Q: Were we affected? 46 | A: Neither Chia network nor the blockchain were impacted. We recognized lessons learned from this massive NPM community issue and have improved our process and practices to ensure no future vulnerabilities exist. 47 | 48 | Q: What about end‑users? 49 | A: No Chia Blockchain or other end users were impacted 50 | 51 | Q: What’s the likelihood of recurrence? 52 | A: High, given attacker ROI. Expect continued phishing of maintainers and token theft targeting postinstall vectors and CI. 53 | 54 | Q: What long‑term measures matter most? 55 | A: Phishing‑resistant MFA for maintainers, strict dependency hygiene, script blocking in CI, token hardening, and continuous supply‑chain monitoring. We have taken steps to improve our process around dependencies and feel we have create a more secure pattern around installation and life cycle management that manages all new information from this attack and its analysis. 56 | 57 | --- 58 | 59 | Sources (evidence of compromises and analysis): 60 | - Aikido: “S1ngularity/nx attackers strike again” (Sept 16–19, 2025) – details worm‑like propagation, secret exfiltration, impacted packages: https://www.aikido.dev/blog/s1ngularity-nx-attackers-strike-again 61 | - Security Boulevard (Legit Security syndicated): “Shai‑Hulud npm Attack: What You Need to Know” (Sept 18, 2025) – attack mechanics, impacted packages list: https://securityboulevard.com/2025/09/shai-hulud-npm-attack-what-you-need-to-know/ 62 | - DEV posts and community write‑ups documenting Sept 8 compromise of `chalk`, `debug`, and related packages, crypto‑drainer payload behavior, and download scale: 63 | - https://dev.to/figsify/anatomy-of-a-supply-chain-heist-the-day-chalk-and-debug-became-crypto-thieves-4fb 64 | - https://dev.to/aerabi/the-largest-npm-supply-chain-attack-ever-and-how-to-defend-against-it-9a6 65 | - FirstPoint: “Securing Repos After the September 2025 NPM Supply Chain Attack” – summary and malicious version matrix: https://blog.firstpoint.com.tr/securing-repos-after-the-september-2025-npm-supply-chain-attack/ 66 | - Additional ecosystem advisories and summaries corroborating Sept 8 and Sept 15–16 activity windows: 67 | - Posit Support advisory: NPM Supply Chain Compromise – Sept 2025 68 | - Guardrail.ai, Blockaid, and other vendor analyses of crypto‑drainer payloads and response guidance -------------------------------------------------------------------------------- /2021-05/2021-05-09-negative-coin-values.md: -------------------------------------------------------------------------------- 1 | ## Issue summary 2 | 3 | On May 9th 2021 at 6:48AM PDT, the blockchain stopped creating blocks at height 255108 for approximately 45 minutes, and then advanced very slowly for the next 3 hours until the hotfix was released. 4 | 5 | We were not checking for negative values in the uint64 constructor, but instead were checking them when serializing. Therefore coins with negative values were added to the mempool. These blocks passed validation, but they did not get added into the blockchain due to negative values not serializing in uint64.stream. Farmers making these blocks would make blocks that did not make it into, or advance, the chain causing most farmers to be unable to create valid transaction blocks. Fortunately, no invalid blocks were added to the blockchain. 6 | 7 | The fix added a check in the mempool and block validation, and did not disconnect peers who sent these invalid blocks (any peer 1.1.4 or older), making this update not mandatory. The blockchain advanced slower than normal for a day, while people updated their machines, and the difficulty adjusted. 8 | 9 | ## Timeline (UTC) All Times Approximate 10 | 11 | - 12:48 (6:48 PDT): The last transaction block before the incident 255105, was farmed. Shortly after, blocks up to 255108 were farmed. 12 | - 13:11: I (Mariano Sorgente) looked at my local node and realized the status was “Not synced”. This means there have been no transaction blocks in more than 7 minutes. I checked the public keybase chats and saw that many people have detected the same issue: their height was stuck at 255108. 13 | - 13:12: I notified the chia engineers chat that the blockchain was stuck and not moving forward. Almog and Arvid joined to help debug. We started debugging and noticed that there were many compact proof of time messages. I created a `no_compact` branch which commented out the handling of these optional messages. 14 | - 13:26: Notified some medium size farmers to start running the no_compact branch. 15 | - 13:32: Made a post in announcements telling people to update to the no_compact branch if they are running from git. Continued debugging. 16 | - 13:43: The blockchain started moving forward slowly, up to height 255111. 17 | - 13:45: We noticed in the logs of block creation: WARNING Consensus error validating block: int too large to convert 18 | - 14:02: After debugging and adding logs, we noticed that there was an invalid coin amount in a block: 2021-05-09T23:02:01.463 full_node chia.full_node.coin_store: ERROR Coin amount: -8551827461000000 19 | - 14:19: The 1.1.4-hotfix branch was created, with a change to not accept valid blocks. I modified the public announcement to tell people to download this branch. The blockchain started moving slowly and with large gaps between blocks. 20 | - 14:20-2:00 Through the next two hours, we checked that there were no negative coins added to the blockchain, added a patch to reject these from the mempool, and added a patch to not disconnect old peers who sent us bad transactions. We also confirmed that the new code synced up successfully (no consensus change). 21 | - 17:18: The final release was published (including mac and windows installers), and an announcement was made to notify users to upgrade. 22 | 23 | ## Root Cause 24 | 25 | There is a uint64 class in the chia-blockchain python code which should only represent numbers from 0 to 2^64 - 1. However, the constructor did not throw when passing in a negative value. The validation happened at a later point, when serializing. This means it was possible to create a coin with a negative value. Furthermore, both the mempool validation and the block validation did not check for negative coins. This led to a few things happening: 26 | A transaction with a negative addition coin was added to the mempool and propagated to all nodes. 27 | Farmers creating blocks added this transaction into the block, and these unfinished blocks validated successfully 28 | When the VDF were added to these blocks to create the finished block, the finished block was not able to be added to the blockchain, because when trying to serialize the negative coin value, an exception would be thrown. 29 | The reason why no invalid blocks were added to the chain, is that when we serialize the coins for inclusion of the DB, we called uint64.stream, which failed for negative values. Therefore the state transition failed, and the node reverted back to the last valid block, 255108. The validation was happening, but it was happening after propagating transactions and blocks, instead of before. 30 | 31 | The reason this caused issues with the blockchain moving, is that all farmers would try to include this negative transaction into their blocks, and this transaction did not get cleared from the mempool. 32 | 33 | There is a fallback to creating an empty block, if a farmer realizes that they made a block with invalid transactions, to prevent these types of issues. However, since the validation of the block did not check for negative coin values, the farmers did not realize that their blocks were invalid, and broadcasted them with the negative coin. 34 | 35 | 36 | ## Resolution and Recovery 37 | 38 | The hotfix we released is here: https://github.com/Chia-Network/chia-blockchain/commit/70c7229f0bf735a0ebb5087eb3a232288d9a7cf6 39 | 40 | The first idea of this being an issue with compact VDFs was not correct, so we quickly started to look elsewhere. When we found out about the negative coin, we added a check in block_body_validation for any negative coin additions. We then added a check in the mempool to reject any transactions with negative coin additions, as well as a new error, COIN_AMOUNT_NEGATIVE. Finally, we caught the error in respond_unfinished_block, so that we would not disconnect 1.1.4 or older peers that had not yet updated. This was important, since we didn’t want to create a fork in the blockchain or divide the network into parts. The longer term fix is to add checks for negative values in uint64 construction. 41 | 42 | After the hotfix was released, the blockchain moved forward more consistently, but still slowly. There were still a lot of people that had not updated, so the effective netspace was much lower than it should be, for that difficulty. The chain difficulty dropped from 215 to 167 due to the time in the stall, and due to people taking time to upgrade. The netspace dropped from 3.2EiB to around 2.6EiB (estimated). After the difficulty adjustment, blocks came much quicker than normal, due to many people upgrading. This led to more frequent forks, as well as double signage points. This is not an issue, but a longer stall could have potentially caused a larger difficulty adjustment, up to the max adjustment of a factor of 3. The difficulty then adjusted back to 209. 43 | 44 | 45 | ## Lessons Learned and Future Improvements 46 | Add more thorough testing and have strict testing standards for PRs into chia-blockchain. In particular, have negative tests in all overflow directions for all values. 47 | Minimize the number of different types of data structures used. 48 | Bug bounty in order to get more visibility on key parts of the code. 49 | More automated alarms for when issues arise. 50 | Have an incident response plan to notify the most experienced people, and potentially farmers as well. 51 | Add an independent status page that e.g. tracks last transaction block time and alerts to a chia-status twitter account. Fix and improve our alerts for no new blocks being generated. 52 | Add a notification in the GUI when a new version is available. 53 | 54 | ## Netspace after the incident 55 | 56 | https://www.chiaexplorer.com/charts/netspace 57 | -------------------------------------------------------------------------------- /2024-02/2024-02-15-Financial-instutition-credentials.md: -------------------------------------------------------------------------------- 1 | # Financial Institution unauthorized access incident 2 | 3 | On Jan 29, 2024, a connection was established between the main financial admin tool used by Chia Network Inc (CNI) , and one of their major banking partners. A CNI employee’s credentials were used to set up this integration. This integration worked as expected until February 14, 2024. Some time during the day on February 14, 2024 the IP address the integration ran from, was changed. 4 | 5 | On February 15, 2024 the integration ran, and because of a quirk in the system used by the bank, because the IP the integration requests were coming from changed, the new IP used by the financial admin tool was reported as a distinct and possibly malicious use of the credentials attached to this integration. 6 | 7 | This spawned a malicious activity report which reached CNI security and the employee at roughly the same time. 7:56 AM UTC. 8 | 9 | At 1:15 PM UTC, The CNI employee reported the incident to the security team, who were already investigating the alert. At this time the user's access to CNI systems was disabled. 10 | 11 | After some initial coordination around 3:50 PM UTC the financial institution was reached and in error, confirmed that the account had in fact been logged into, from the IP in question, and that MFA had been used on the user's account. Luckily the account in question was read only, and no funds had been tampered with. This is also when the internal incident response process was iniated. CNI team was awaiting contact with the fraud prevention team at the financial institution as well. 12 | 13 | In the interim, CNI security team did a cursory investigation on the IP reported by the financial institution and established the location of use, and owner, as well as the abuse contact. 14 | 15 | Also during this period the CNI employee attempted to contact their MFA provider and gain information on how a malicious party might have been able to also use this factor. 16 | 17 | At this point forward CNI security team and employees proceeded with the assumption of active compromise of the user’s workstation, and possibly home network. Because of the multiple federally regulated services involved, evidence gathering was initiated. The CNI employee was instructed to not touch their workstation any further except with guidance from the security team. 18 | 19 | A full system scan and quarantine was initiated on all employee’s work devices. 20 | 21 | At around 8 pm UTC on February 14, 2024. These scans returned no results, the devices were left in quarantine. 22 | On February 15, 2024: 8:33 PM UTC The fraud prevention team was able to reach the CNI team and confirm details of the incident. At this point the financial institution team was still actively confirming the compromise, and was able to confirm several other access attempts throughout the day. 23 | 24 | The final login attempt was around 5 AM UTC. February 16, 2024. 25 | 26 | On February 16, 2024 after diligence had been initiated 27 | 28 | Once investigation began again on February 16th, CNI security team and employee again spoke with the MFA provider as part of the investigation. 29 | 30 | At 4:00 PM UTC February 16, 2024 the CNI security team contacted federal law enforcement, they had reached the end of their paths for investigation and needed to report their findings as well as initiate federal investigation of the alleged crimes. 31 | 32 | The employee also spoke with the financial institution again at 5:20 PM UTC February 16th, 2024, who confirmed 2 additional login attempts during that day at 5:11 AM and 8:25 AM UTC respectively. 33 | 34 | At 7:20 PM UTC after further investigation in cooperation with law enforcement the fraud prevention team at the bank was able to provide a thorough accounting of all access to the account in question, and were able to match the access to a third party integration, from the financial admin tool, which had been modified back in January. 35 | 36 | It was only after careful examination on the financial institutions end, that they established MFA had not been used on this account or the alleged malicious login attempts. It only appeared that way without a fine tooth review of the logins. The financial institution was able to provide an accounting for all activity, and the integration in question was scheduled for removal. CNI security team closed the issue, and law enforcement reports, as a false positive. 37 | 38 | ## Timeline (all times UTC) 1/29/24, 2/15/24 - 2/16/24 All times approximations 39 | 40 | January 29, 2024 41 | CNI finance team established a bank connector with the financial admin tool, which they had been working on for an extended period of time. We have a similar connector with another financial institution. This connector feeds banking transactions into the financial admin tool, which allows us to perform a bank reconciliation within the financial admin tool. CNI Employee met with our financial admin tool consultant to set-up the connector. This set-up occurred over a Chia Zoom meeting. In order to establish the connection, we logged into financial admin tool and then clicked on the links within financial admin tool to establish a connection. A list of banks popped up and we selected the correct financial institution. From there, CNI Employee entered his login information. 42 | 43 | **Note that the above information is included for context purposes. Outside of the above login, access to financial institution’s banking portal usually occurs once or twice a month. Additionally, no access to the financial institution has occurred on CNI Employee’s mobile phone and the only work-related apps / websites used on CNI Employee’s mobile phone for Chia is Keybase. 44 | 45 | February 15, 2024: 7:56am UTC 46 | CNI Employee received an email from financial institution indicating that there was a login from a new device at 2:56am EST. 47 | 48 | February 15, 2024: 1:15pm UTC 49 | CNI Employee saw the email from financial institution. He logged into financial institution and banking systems to change passwords. At this time the user's access to CNI systems was disabled. 50 | 51 | February 15, 2024: 1:23pm UTC 52 | CNI Employee forwarded Justin England the email from financial institution. Justin responded and asked CNI Employee to contact financial institution. 53 | 54 | February 15, 2024: 3:59pm UTC 55 | CNI Employee contacted financial institution and informed them that he did not access the financial institution banking system at 7:56am UTC. They informed CNI Employee that they would raise the issue to their IT team. 56 | 57 | February 15, 2024: 5:33pm UTC 58 | CNI Employee received a call from the financial institution bank fraud detection team. They mentioned that someone gained access to the financial institution system using a code, which was sent to a mobile phone with the same last four digits as CNI Employee’s. They were in the system for a very short period of time before financial institution kicked them out of the system. The financial institutions rep confirmed that the IP address used at 7:56am UTC was not the same IP address that was used when CNI Employee logged into financial institution around 1:15pm UTC, rep advised CNI employee to change his password. Rep also said he would forward the IP address used at 7:56am UTC to CNI Employee’s Chia email. 59 | 60 | February 15, 2024: 6:35pm UTC 61 | CNI Employee called MFA provider and explained the situation. They checked their records for unauthorized access or new logins and detected none. They informed CNI Employee there was nothing else they could do. 62 | 63 | February 15, 2024: 7:29pm UTC 64 | CNI Employee called mfa provider and explained the situation. They would not provide CNI Employee with additional details because he is not a manager on the account. 65 | 66 | February 15, 2024: 8:35pm UTC 67 | CNI Employee called mfa provider and was added as a manager on the account. CNI Employee explained the situation and discussed with the fraud department. They said that it is likely spoofing and there is nothing they can do. 68 | 69 | February 16, 2024: 3:44pm UTC 70 | Justin England (“Justin”) informed CNI Employee that there was another login attempt to the financial institution banking website the evening of February 15th and asked CNI Employee to call financial institution after the upcoming mfa provider call. 71 | 72 | February 16, 2024: 4:00pm UTC 73 | CNI Employee called mfa provider with Justin and explained the situation to the support rep. They transferred the call to the technical team. The technical team was unable to pull specific data. The technical team transferred CNI Employee and Justin to the fraud prevention team where they spoke with a rep. The fraud prevention team was also unable to help and suggested CNI Employee and Justin talk with the security legal team. 74 | 75 | February 16, 2024: 4:33pm UTC 76 | CNI Employee called the mfa provider security legal team and spoke with a rep. She mentioned that there is nothing the legal team can do without a case number through an agency. She suggested Chia file a police report and call back. 77 | 78 | After the call, federal law enforcement was contacted. 79 | 80 | February 16, 2024: 5:20pm UTC 81 | CNI Employee called financial institution and spoke with another rep. She mentioned there were two additional login attempts at 5:11am UTC and 8:25am UTC. She contacted the fraud prevention team and mentioned that someone would reach out. 82 | 83 | February 16, 2024: 7:20pm UTC 84 | Financial institution fraud prevention contacted CNI employee, and stated after further investigation in cooperation with law enforcement the fraud prevention team at the bank was able to provide a thorough accounting of all access to the account in question, and were able to match the access to a third party integration, from the financial admin tool, which had been modified back in January. 85 | 86 | It was only after careful examination on the financial institutions end, that they established MFA had not been used on this account or the alleged malicious login attempts. It only appeared that way without a fine tooth review of the logins. The financial institution was able to provide an accounting for all activity, and the integration in question was scheduled for removal. CNI security team closed the issue, and law enforcement reports, as a false positive. 87 | 88 | ## Resolution and recovery 89 | 90 | The integration in question was scheduled for removal. CNI security team closed the issue, and law enforcement reports, as a false positive. 91 | 92 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright [yyyy] [name of copyright owner] 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /2021-10/2021-10-31-DustStorm.md: -------------------------------------------------------------------------------- 1 | # Oct 31 2021 Dust Storm 2 | 3 | ## Issue Summary 4 | 5 | On Saturday Oct 30th 2021 an unknown actor began sending a large volume of transactions to the Chia mainnet blockchain, causing transactions to begin filling the mempool. As the mempool grew larger, several symptoms became apparent across the network with the most disruptive symptom being lower end nodes becoming clogged with the larger mempool and beginning to fall behind on the chain. Some peers with many slow connections fell behind the peak of the blockchain. Network wide transactions without a fee were also disrupted during these times of higher burst. Throughout the day on the 30th a few smaller sorties of transactions were sent. When the storm transactions were halted, the mempool cleared rather quickly at a rate of 20+ transactions per second, returning things to normal each time. It is important to note that during this incident and the following day’s, the network continued to make and sign blocks, and the chain never stopped. 6 | 7 | Early in the morning on Sunday the 31st of October, a far larger storm of transactions began. The symptoms we initially noticed as worrisome on the 30th began to appear again as the mempool began to climb in size once more. (However at no point during this second storm of transactions did the mempool grow near the maximum defined size.) The largest impact to the network was once again that lower-power nodes fell off the network as they were not able to stay in sync as the mempool grew, which resulted in as much as ~5 EiB of farmer space being unable to farm. A secondary impact of transactions requiring fees to be processed in a timely manner also emerged, more on that later. Finally in some very rare cases we were seeing a higher than usual number of reorgs on the chain, which was likely due to some small network propagation issues as the network reorganized to take into account the loss of some of the weaker nodes. 8 | 9 | In addition to the two largest impacts, we saw many smaller symptoms that occurred in varying degrees depending on the quality of the majority of a given node’s peers. Block propagation slowed for some portion of the network due to slower nodes being too slow to catch up but not too slow to fall off the chain. For some other portion of the network, signage points were arriving late causing some missed proofs and partials, even very strong nodes were impacted if the majority of their connected peers were struggling. 10 | 11 | To expand on the transaction fees mentioned above, transactions have had fee support since launch, however the lower transaction volume compared to the high block space available resulted in the majority of transactions being sent without fees. Additionally, with the followup release of the pooling protocol there were UI gaps in PlotNFT transactions (while the PlotNFT fees were supported in the code, there were no actual UI elements to leverage them). Many 3rd parties were also unprepared to use fees due to them being unnecessary before now- therefore many 3rd party transactions (eg. pools and exchanges) were delayed as they were caught in the congestion. Before this point, it was unexpected that transaction volume of this density would pop up this early in the chain's life cycle, but it was encouraging that many 3rd parties in the ecosystem were easily able to catch up to this need and resolve the problem in their environment within the day. For the Chia team’s part, the latest release of 1.2.11 should correct all the outstanding issues related to fees, with the exception of plotNFT fees, which we will address in the 1.2.12 release. 12 | 13 | 14 | ## Timeline (all times PST ) Saturday Oct 30th 2021 - Monday Nov 1st, 2021. All times approximations 15 | 16 | Block based timeline 17 | Dust storm times (in block heights): 18 | 1st wave: 1071900 - 1072525 19 | 2nd wave: 1072850 - 1073850 20 | 3rd wave: 1074650 - 1078400 21 | 22 | 12:45 PM PST Saturday Oct 30th: First wave of storm transactions begins 23 | 24 | 13:55 PM PST Saturday Oct 30th: Chia operations alarms focused on AWS timelord usage intended to let Chia Devops team know if there are larger issues, fire. Devops engineers began investigating. 25 | 26 | 14:00 PM PST Saturday Oct 30th : Devops and Software Engineering teams establish this is likely not genuine traffic, load across the network seems nominal. Teams begin looking for more data and digging deeper into information about the block chain state. 27 | 28 | 14:17 PM PST Saturday Oct 30th: First wave of transactions slightly slows. 29 | 30 | 15:48 PM PST Saturday Oct 30th : first wave of transaction peaks at 1750 transactions in the mempool. 31 | 32 | 16:00 PM PST Saturday Oct 30th: Chia team tunes all alerting, establishes these transactions are either an unannounced test by a 3rd party, or something less friendly. 33 | 34 | 16:02 PM PST Saturday Oct 30th: Mempool is back to zero. 35 | 36 | 18:00 PM PST Saturday Oct 30th: Second wave of transactions begin. 37 | 38 | 22:22 PM PST Saturday Oct 30th: Second wave peaks at ~2800 transactions. 39 | 40 | 23:00 AM MST Saturday Oct 30th: Mempool is empty again. 41 | 42 | 03:30 AM PST Sunday Oct 31st: Final and largest wave of the dust storm begins. 43 | 44 | 03:40 AM PST Sunday Oct 31st: Engineers online at this hour begin conversations about a rise in mempool again, which appears to be similar to the previous day's issues. 45 | 46 | 06:00 AM PST Sunday Oct 31st: Chia engineers begin work on potential improvements and optimizations discovered from analyzing the logs of instances reporting issues. 47 | 48 | 12:00 PM PST Sunday Oct 31st: Chia team decides that an optimization patch based on the learnings so far should be released as soon as possible. Plans begin to take shape for an emergency patch. The fixes for this patch are outlined and the work is assigned to senior engineers for discovery. 49 | 50 | 13:30 PM PST Sunday Oct 31st: first peak ~5100 transactions in the mempool. 51 | 52 | 16:05 PM PST Sunday Oct 31st: trough, roughly ~4250 transactions in the mempool. 53 | 54 | 02:54 AM PST Monday Nov 1st: second peak of ~5750 transactions in mempool. 55 | 56 | 04:46 AM PST Monday Nov 1st: mempool is empty. 57 | 58 | 10:00 AM PST Monday Nov 1st: Formal planning for the emergency patch is set, all engineering tasks are assigned for completion, with a target release date of 11/03. 59 | 60 | ![10-31-2021 Dust Storm Graph](/assets/images/10-31-2021-dust-storm.jpg) 61 | 62 | ## Root Cause 63 | 64 | The ultimate root cause of all issues during the dust storm was slow block propagation. The increase in transaction volume directly slowed the propagation of blocks as nodes were busy processing transactions rather than blocks. Since validating transactions requires using a lot of disk I/O and CPU, some nodes were unable to both process large transactions volumes _and_ blocks in a timely manner. Before the 1.2.11 update, there was no max queue size for transactions, and there was no prioritization of blocks, so some very large block validation times, and propagation times were being seen. Furthermore, some inefficient locks were being held while validating transactions, which further slowed the speed of block propagation. 65 | 66 | Since signage points are based on blocks, farmers cannot farm until they process the most recent block. This caused slow machines or nodes connected to many slow machines to miss many signage points and lose farming time. 67 | 68 | The full mempool also caused fee related issues, since the Chia client and pools were not properly set up to fully utilize fees. Before this incident, low transaction volume meant that zero fees were sufficient to get any transaction into the blockchain. During the dust storm, some pools were not able to claim pool rewards from the blockchain, and were also not able to make payouts to users, due to insufficient fees being applied to these transactions to prioritize ahead of the storm. Fortunately, when the storm ended and fee pressure returned to zero, the pools that had not implemented fees yet, were able to process the pending transactions and catch up. 69 | 70 | Another fee related issue was that Chia farmers were not able to switch pools during the storm, due to no UI elements to adjust the fee beyond the default of zero. 71 | 72 | 73 | ## Resolution and recovery 74 | 75 | On Nov 1, 2021 at approximately 09:45 am UTC the unknown entity ceased their dust storm activities, shortly following that the block chain returned to normal state and existing fee pressure again returned to nominal. 76 | 77 | On November 4, 2021, 1.2.11 was released, which solved many of the issues that led to the symptoms experienced by farmers and users. The changes addressed both the improper handling of a high volume of transactions, and the impact of high TPS on consensus and farming. This included many optimization fixes, the most important being: 78 | Blocks are now always prioritized over transactions. No matter how many transactions a node receives, it will always validate and propagate blocks first. This also reduced competition for I/O. 79 | A transaction queue was implemented, which streamlined and optimized how transactions were processed. 80 | More parallelism and caching is used to validate transactions and signatures 81 | Some locks which were causing the node to wait before propagating blocks were removed 82 | Transaction dropping was implemented in cases where a node can not keep up with a very high number of transactions per second. This is not likely to ever happen on medium/high end machines and SSD storage (this is mostly targeted at RPi 4’s with slow storage and similar setups). 83 | More efficient updating of mempool after adding blocks. 84 | 85 | As a result of the Dust storm, we have improved our load testing capabilities and experimented on testnet with higher TPS than even those seen on the dust storm, and low powered machines were still able to keep up. However, there are still some issues remaining we would like to address, especially for computers with slow I/O and drives. Database optimization remains a priority to fix in the near future. 86 | 87 | Chia has temporarily upgraded our min spec for low end nodes to still be the Pi4 4GB platform, but with the additional specification of an ssd for the block chain database and at least 1GB of swap space configured on the operating system. (In other words, we do not recommend having your database file on a local SD card.) We also do not recommend using the GUI on Pi4 4GB models. 88 | 89 | ## Testnet Dust Storm 90 | On Monday November 1st 2021, we began running our own version of a dust storm spam bot on testnet 7. Below is a graph of a similar duration of transactions being sent. Most critically you will note that our dust storm which we used to test the 1.2.11 changes pre release created as many as 65,000 transactions in the mempool which is 11.3 times more transactions in the mempool than the original dust storm attempt. This gives our engineers confidence that we have tested against similarly harsh conditions seen during the mainnet dust storm and beyond, and allows our team to trust that we have mitigated many of the performance issues present in the lower end of nodes with the 1.2.11 upgrade. 91 | 92 | ![Testnet Dust Storm Graph](/assets/images/testnet-dust-storm-graph.png) 93 | 94 | ## Q&A 95 | 96 | 97 | 98 | __Q: Do you know who conducted the Dust Storm, or do you plan to find out?__ 99 | 100 | __A:__ Not formally no, and we have no plans to find out with certainty. While curiosity would be satisfied, determining the identity of the person serves us no purpose as we have no interest in taking any actions based on this information. While impactful and frustrating, ultimately all they did was simulate a real world load on the network similar to one we would have expected to see months from now, effectively running the same stress tests we run on testnet at a larger (and significantly more expensive) scale. 101 | 102 | 103 | __Q: Why did you not test this kind of load on the network before now to know how it would react?__ 104 | 105 | 106 | __A:__ Essentially we did, however, like all things when testing, despite one’s best efforts no amount of simulations will replicate the real world experience of hundreds of thousands of nodes and farmers at scale in a real world environment. (Anyone who has played a popular video game online on launch day can attest to this experience!) That said, we have taken what we learned from the Dust Storm and modified our internal testnet load tests to throw a load significantly larger than anything seen during the Dust Storm at it when desired to create an even worse scenario than any we imagined (or have experienced) before. 107 | 108 | 109 | __Q: Why were none of the optimisations and changes you implemented after the event already part of the code?__ 110 | 111 | __A:__ Like any software development project, we have always known there were areas that worked to expectations, but could be improved upon. You always have to balance optimizing old code along with the implementation of new things (like yesterday’s CATs launch), which means deciding between shipping something now and making it better later, not releasing it until it’s 100% baked but released much further down the road. If you have a solid picture of your needs and expectations (and use projections!) over the next year, you’re able to strategically release things incrementally and go back and optimize them over time before those optimizations are required. This then allows you to balance each release between new functionality and tuning up old functionality. Unfortunately, when something like this dust storm happens that basically shines a spotlight on those needed optimizations you’ve been sitting on (because they weren’t yet required), you now have to shift focus on pulling that work in closer. All this scenario did was bring our real-world usage projections forward by several months, so in a way it was an inconvenient bit of assistance in tuning up our network. 112 | 113 | 114 | __Q: Why did you not already have fees?__ 115 | __A:__ There was a bit of a misconception going around that we launched without fees support, rest assured this is not true. The Chia blockchain has always supported fees, and in fact, in the first day or two people were mistakenly making transfers with some non-trivial fees applied. (Much to the joy of the farmers who farmed them!) However, what _was_ lacking, was the ability to set custom fees on some later-added components, such as PlotNFT commands in the GUI and CLI, as well as fee documentation in the Pooling reference code. On the former subject, some of this was an honest oversight due to lack of need raising awareness and some of it was a “we can adjust this a bit later long before it’s actually needed”. (See the above Q for more context). On the later issue of pools, we have always stressed from day 1 that our pooling protocol reference code was just that, _reference_ documentation to help provide initial guidance, and that we strongly advised any pools building PlotNFT support to only use the code as a starting point, and build their own solutions with technology and implementations they were comfortable and familiar with. Anyone who relied exclusively on the reference code was indeed without transaction fee support built in, but this is one of the many reasons why we explicitly warned against relying on this code in production, where it was never meant to run. Practicality dictates here was always going to be a world where fees were needed down the road, however we do not fault pools for thinking they were an “over the horizon” priority, much in the way we did. 116 | 117 | __Q: Why did the netspace go down?__ 118 | __A:__ The netspace only fluctuated briefly as a result of low power nodes going out of sync, which in turn made any farmers attached to them unable to farm, causing the available netspace to decrease. As netspace is estimated over a rolling time window, this meant that the space decreased to a slight delay behind the actual issues, and took an accordingly length of time to adjust back once it was over before normalizing once again. -------------------------------------------------------------------------------- /2022-08/2022-08-19-CATbleed.md: -------------------------------------------------------------------------------- 1 | # CATbleed 2 | 3 | ## Issue Summary 4 | 5 | Part of our commitment to the security of the Chia blockchain and community is providing transparency around security and the lessons learned when a security incident is discovered. In this spirit, we have a Common Vulnerabilities and Exposure (CVE) and Common Weakness Enumeration (CWE) published on this topic. For a more general overview, please refer to our previously published blog post, Upgrading the CAT Standard. 6 | 7 | ### Timeline (all times PST). All times are approximate 8 | --- 9 | **June 14th, 2022, 10:00 AM:** As part of their audit of the CAT1 standard, Trail Of Bits reported that a class of constraint checking vulnerability might exist in the CAT1 standard. 10 | 11 | **June 14th, 2022, 10:30 AM:** The reported issue was reviewed by Matt Hauff on the chia team. Matt discovered what was likely a major bug in the software; he escalated his concerns internally at this point. 12 | 13 | **June 14th, 2022, 1:00 PM:** Richard Kiss reviewed the reports from Trail of Bits and then worked with Matt to double-check the issue. At this point Richard and Matt realized the full scope of the concern and that it was likely a fatal flaw in the CAT1 standard. 14 | 15 | **June 14th, 2022, 1:30 PM:** The issue with full discovery and potential risks were reported to the Chia Executive team. 16 | 17 | **June 16th 2022, 8:45 AM:** On chain monitoring tool is established to scan the entire chain going backward and forward. Monitor will alarm if any evidence of the exploit is found. 18 | 19 | **June 17th 2022 2:00 PM:** Chia team has established what their patching plan should be to correct the issue with CAT1, and the plan to replace the CAT1 standard with CAT2 was in place. 20 | 21 | **June 20th, 2022, 11:00 AM:** Chia team had completed the software changes necessary to patch the bug in CAT1, as well as a full internal audit of the new CAT2 standard and finalized discussion around if any further improvements were needed. It was decided that this was a complete patch. 22 | 23 | **June 20th, 2022, 2:00 PM:** Final decision to patch the bug in the main codebase, to release the NFT1 standard in a patched state was made. 24 | 25 | **June 29th, 2022, 3:02 PM:** NFT1 is released with the release of Chia version 1.4.0 which included the undocumented fix patching the vulnerability in the NFT1 standard. 26 | 27 | **July 25th, 2022, 10:00 AM:** end-of-life of the CAT1 standard was announced to the public at large, no details on the exploit were given at this time. 28 | 29 | **July 26th, 2022, 9:08 AM:** At block height 2,311,760. (CAT exploit alarm was negative through this point.) Chia team begins operations to cancel outstanding CAT1 offers. The Chia team begins using the exploit to generate amounts of CAT1 necessary for resolving outstanding offers. 30 | 31 | **July 26th, 2022, 12:15 PM:** Chia version 1.5.0 is released with the code fix in place. No context on the bug was provided while Chia performed some white hat activities to cancel and return all outstanding CAT1 offers. 32 | 33 | **July 26th, 2022, 7:00 PM:** Exploit details are published by the Chia team on the chia blog site. 34 | 35 | **July 26th, 2022, 9:00 PM:** After some initial tooling issues, CAT1 offer invalidation is completed, and user assets are safely in their original wallets. 36 | 37 | **July 29th, 2022, 12:00 AM:** Mitre.org published the submitted CAT1 CVE: https://nvd.nist.gov/vuln/detail/CVE-2022-36447 38 | 39 | **August 1st 2022:** Majority of CAT2 re-minting and airdrops were completed across the ecosystem. 40 | 41 | ## Resolution and recovery 42 | 43 | The critical CAT1 vulnerability was discovered by Chia engineers, after potential exposure to this class of concatenation vulnerabilities was reported by Trail of Bits (as part of their general audit of the CAT1 standard). The class of vulnerability reported is a typical class of bug often seen in software when concatenating strings of this variety. The report was immediately investigated by Matt Hauff of the Chia Network team. Matt identified the larger implications of this bug and escalated the report to Richard Kiss. Richard upon investigation, discovered larger security implications and eventually came to the conclusion with Matt that this bug was fatal to the CAT1 standard as it allowed inflation of CAT1 assets without control over the original tail. Once the full impact of the issue was resolved, the Chia team got to work on a plan to handle the disaster response and patch the vulnerability. 44 | 45 | Once the severity of the bug was established, the company moved very quickly to establish a monitoring system on chain to identify if the bug has ever been used in the wild and monitor for future use going forward. This alarm was never able to find any evidence of the exploit being used until the Chia team began running the exploit after CAT1 end-of-life in order to resolve outstanding offers. 46 | 47 | On June 17th, 3 days post report, we had a working solution to every issue this vulnerability presented. By June 20th we had all the engineering work ready to crash patch the vulnerability should the aforementioned alarm return a positive before end-of-life. 48 | 49 | With the monitoring and response plans in place and the software and operations plans ready to go, Chia Network turned our attention to the NFT1 standard going live with the 1.4.0 release. After some internal debate, we decided to favor stability in the NFT standard over absolute secrecy, and we worked the patch to the CAT1 vulnerability into the release for NFT1. 50 | 51 | This decision, while a small risk, was mitigated by our monitoring and crash patch response plans as well as the difficulty of identifying this software bug without all the necessary context. 52 | The final plans for CAT1 were put into action after several weeks of intensive internal testing of the 1.5.0 release, which included the CAT2 standard and some other fixes. On July 25th at 10AM Pacific time, we announced the ensuing end-of-life of the CAT1 standard, roughly 24 hours in the future. At roughly 9 am on July 26th, CAT1 standard came to an end at block height 2,311,760. This spurred the next phase of the retirement plan into action. 53 | 54 | Chia engineers had been collecting coins for about a week by accepting then valid CAT1 offers to ensure that the pieces needed to accept and cancel all outstanding CAT1 offers were ready. At the end-of-life time, those engineers began using the exploit on chain to mint the amounts of the CAT1s needed to accept and then return xch funds to the affected users. 55 | It is important to note that the acceptance and return of the xch offers were accomplished in the same block, Chia Network was never in possession of any users xch, simply in control of the transaction accepting the offer and routing the offered coin back to the same user. 56 | 57 | In this way we avoided massive fraud where malicious third parties could have done this same action and kept the impacted users' xch coins. We viewed this as a necessary step, if a bit extreme. We felt that an ecosystem of users with their funds returned would be far happier than one where some portion lost funds because of long-forgotten offers or similar issues. 58 | 59 | With the completion of that work and the publishing of this document, Chia Network views the CAT1 standard officially closed and all extant CAT1 coins to now be of no value. It is advised that all users cease transactions involving CAT1 coins. Conveniently, the exploit allows owners of CAT1 coins to melt them, returning to the owner the constituent mojos that make up each CAT1. 60 | 61 | ## CVE - Description of the Vulnerability 62 | 63 | The CAT1 standard calculates the coin ID of the CAT1 coin being spent by taking the SHA 256 hash of the concatenated parent coin ID, CAT1 coin puzzle hash, and CAT1 coin amount, as shown in this example: 64 | ``` 65 | (stager 66 | (list MOD_HASH (sha256 ONE MOD_HASH) TAIL_PROGRAM_HASH) 67 | (a INNER_PUZZLE inner_puzzle_solution) 68 | lineage_proof 69 | (sha256tree1 INNER_PUZZLE) 70 | ;; calculate coin ID 71 | (sha256 (f this_coin_info) (f (r this_coin_info)) (f (r (r this_coin_info)))) 72 | prev_coin_id ; ID 73 | this_coin_info ; (parent_id puzzle_hash amount) 74 | next_coin_proof ; (parent_id innerpuzhash amount) 75 | prev_subtotal 76 | extra_delta 77 | ) 78 | ``` 79 | However, this method only ensures that the full set of concatenated bytes is correct and not that the constituting components are correct. If bytes are moved from the end of the CAT1 coin puzzle hash to the beginning of the amount, the same SHA 256 hash will be computed. This enables an incorrect amount to be used, resulting in a larger-sized coin being created. To prevent this, coin IDs must be checked to ensure that constituent components are correct. 80 | ``` 81 | (defun calculate_coin_id (parent puzzlehash amount) 82 | (if (all (size_b32 parent) (size_b32 puzzlehash) (> amount -1)) 83 | (sha256 parent puzzlehash amount) 84 | (x) 85 | ) 86 | ) 87 | 88 | (stager 89 | (list MOD_HASH (sha256 ONE MOD_HASH) TAIL_PROGRAM_HASH) 90 | (a INNER_PUZZLE inner_puzzle_solution) 91 | lineage_proof 92 | (sha256tree INNER_PUZZLE) 93 | ;; calculate coin ID securely 94 | (calculate_coin_id (f this_coin_info) (f (r this_coin_info)) (f (r (r this_coin_info)))) 95 | prev_coin_id ; ID 96 | this_coin_info ; (parent_id puzzle_hash amount) 97 | next_coin_proof ; (parent_id innerpuzhash amount) 98 | prev_subtotal 99 | extra_delta 100 | ) 101 | ``` 102 | 103 | The CAT1 vulnerability is a part of a class of vulnerabilities that can arise from insecure concatenation. Concatenation is commonly combined with hashing, where bytes are concatenated to form a preimage and passed to a hash function. To avoid this class of vulnerability, each component must be checked for correctness before concatenation. An example of this is provided in the `calculate_coin_id function`, where the parent coin ID and puzzle hash are checked for the correct size of 32 bytes, and the amount is checked to be non-negative. 104 | 105 | ## CWE - How to Recreate the Vulnerability 106 | 107 | A coin is represented by a variable length structure of the form `(parent_coin_id puzzle_hash amount)` where the first two fields are 32 bytes, and the last is a `clvm varint` type. 108 | varint examples: 109 | 0 = "" (the empty string) 110 | 1 = 0x01 111 | 50 = 0x32 112 | 20000 = 0x4e20 113 | 65000 = 0x00fde8 because a varint supports negative values, we prefix numbers that would otherwise have the high bit set with a 00 byte 114 | The solution to a CAT1 puzzle takes a triple of three values for the coin the puzzle is included in. It adds a condition `(ASSERT_MY_ID v)` where v is the calculated coin id using these values via `(sha256 parent_coin_id puzzle_hash amount)`. It further uses the amount value to ensure mojos are neither created nor burned (except when authorized by the tail). 115 | The problem is that sha256 concatenates the passed values when evaluating, and at no time do we ensure that parent_coin_id or puzzle_hash is 32 bytes as expected (or verified to be correct by any other means). That means the boundary between puzzle_hash and amount can be varied, changing the perceived amount of the coin. 116 | So slide over the last byte or two of the puzzle_hash to be the new first byte or two of amount, and you’ve just created a coin worth much more (or, if the first byte has the high bit set, a negative amount!). 117 | This also gives us an easy way to melt CAT coins: move the boundary all the way to the right, so the puzzle_hash field contains all the bytes from the actual puzzle_hash field and the amount field, so the amount field appears to be the empty string, which represents 0. 118 | 119 | The CVE for this vulnerability is published on the MITRE CVE website and can be found here: CVE-2022-36447 120 | Security is a critical component of our work, and we maintain a proactive approach by regularly commissioning external audits of Chia code. The discovery of this vulnerability in concert with Trail of Bits is from our third independent audit of Chia code and the second audit of the CAT code, and a report will be published on their findings. 121 | 122 | ## FAQ 123 | --- 124 | 125 | #### General Questions 126 | 127 | **What is a CAT?** 128 | 129 | A CAT is a Chia Asset Token. CATs are fungible tokens that are issued on the chia blockchain. The CAT1 standard was finalized in January 2022. The standard can be found at[ chialisp.com/docs/puzzles/cats/](https://chialisp.com/docs/puzzles/cats/). Some examples of CATs include Stably USD (USDS), Spacebucks (SBX), and Marmot (MRMT). 130 | 131 | **Why is this change happening with Chia asset tokens?** 132 | 133 | The CAT standard was upgraded to CAT2 based on a security vulnerability found by an outside security audit. This resulted in an upgrade to the latest Chia wallet app as well as updates that will require all original issuers of CAT1 tokens to reissue their tokens on the CAT2 standard and end-of-life support for CAT1s. Chia is working with community members to make this process as seamless as possible. 134 | 135 | **Does this change impact Chia Network's security?** 136 | 137 | No. There is no threat to the security of Chia Network technology or the Chia Blockchain. The update patched the vulnerability to CAT1. 138 | 139 | **When does the end-of-life of CAT1 happen?** 140 | 141 | CAT1 is end-of-life after block height 2,311,760 which is approximately around 17:00 UTC. This is when the snapshot is taken. 142 | 143 | **How can I check my CAT1 balance at the time of the snapshot?** 144 | 145 | Go to [cat1.chia.net](https://cat1.chia.net) and provide your pubkey to see the CAT balances that will be airdropped to you when they get re-issued. 146 | 147 | **Does everyone have to upgrade?** 148 | 149 | We recommend that all CAT (including USDS) holders upgrade to 1.5 as soon as is convenient. The CAT1s that are visible on 1.4 and earlier versions will no longer be supported after the end-of-life block (2,311,760). If you do not own any CATs (for example, if you are a farmer who does not exchange XCH for USDS or any other CAT), then you don't need to upgrade. 150 | 151 | **Do I need to upgrade my harvesters?** 152 | 153 | This update only affects the wallet software, so you don't need to update your harvesters. 154 | 155 | **Are my NFTs or XCH at risk?** 156 | 157 | No. NFTs and XCH are not affected by the vulnerability, so no changes are required for them. 158 | 159 | **Is there any risk that I'll lose money or my balance will be incorrect during the transition? If so, what do I do?** 160 | 161 | If you have any CATs in your wallet, you will want to upgrade to 1.5 as soon as conveniently possible and be sure not to transact with any CATs after the end-of-life block height (2,311,760) has been reached and until you have upgraded to 1.5. This will help ensure that the balance you are expecting is what will be airdropped to you when the CATs are reissued. The[ CAT1 website](https://cat1.chia.net) accurately reflects the CAT1 balance of your wallet as of the end-of-life announcement. It will not dynamically update, but we expect the reissuance process to take approximately a week to fully complete, so the CAT2 balance in your wallet may differ from the CAT1 website balance until the process is done. 162 | 163 | **Between the announcement and the end-of-life block height, what should I be doing as a user?** 164 | 165 | It is recommended that you: 166 | 167 | 1. Cancel any open CAT offers on-chain in your wallet 168 | 169 | 2. Do not accept any CAT offers in your 1.4 or lower wallet 170 | 171 | 3. Make note of your current CAT1 balances 172 | 173 | 4. Upgrade to the latest Chia wallet app (1.5.0 or higher) when it becomes available 174 | 175 | **How can I be sure that I've canceled all my open offers?** 176 | 177 | Most importantly, you will want to make sure there are no outstanding offers to trade your XCH for someone else's CATs. In addition to canceling the offers in your wallet, you can also send your total balance of XCH to yourself. Due to the chia coinset model, this will ensure that all XCH coins will no longer be available should a rogue or forgotten offer be accepted. 178 | 179 | **I may have lost money by transacting, what should I do now?** 180 | 181 | Unfortunately, any CAT1 transactions that happen after the end-of-life block height won't be recoverable. For further confirmation, please contact our support team so they can help with checking when the transactions occurred and can help determine if the money is lost or not. 182 | 183 | **Do I need to cancel my XCH-for-NFT offers?** 184 | 185 | No. Only CAT1 tokens are affected. No changes are being made to NFTs. However, if you have an open CAT-for-NFT offer, then you should cancel it. 186 | 187 | **How can I trust that all of my currency will be transferred appropriately?** 188 | 189 | You can check your CAT1 balance as of the snapshot through our [website](https://cat1.chia.net) using your pub key. 190 | 191 | We are providing tools and support to the community developers to help ensure that they can reissue the new CATs in a timely manner. All CAT reissuers will be going by token balances at the same end-of-life block height. 192 | 193 | **What happens to my CAT1s?** 194 | 195 | Your existing CAT1 tokens still exist on the blockchain, but they are of no more use as everyone upgrades to CAT2s. You will be airdropped CAT2 to replace your CAT1 based on your balance as of block height of 2,311,760. Once you upgrade to 1.5, you will no longer see any balances for your original CATs. 196 | 197 | **How long will it take for me to get all my tokens airdropped to me?** 198 | 199 | This will depend on when the original issuers re-issue their tokens based on the new CAT2 standard. We hope that it is soon after the CAT1 EOL date. It is recommended to follow any social media, or discord for the CAT token projects so you can hear first hand when to expect the airdrop. 200 | 201 | **What happens if I have a transaction with a CAT1 token after the block height snapshot?** 202 | 203 | You will only be airdropped the balance of the tokens at the time the snapshot is taken. Any transactions that occur after the snapshot will not be accounted for in the airdrop provided to you. 204 | 205 | **How can I trust that all of my currency will be transferred appropriately? Is there any risk that I'll lose money or my balance will be incorrect during the transition? If so, what do I do?** 206 | 207 | Your CAT1 tokens won't be transferred. Instead, you will be given an identical (in value) set of CAT2 tokens. The blockchain already contains a complete record of all coins in the coinset. We have developed a tool that will use the blockchain to calculate a complete snapshot of CAT1 tokens. This snapshot will be accurate as of CAT1's end-of-life block. 208 | 209 | However, the CAT1 issuers do need to perform a complete airdrop of CAT2 tokens. If the airdrop is not completed, or even started, then there is a risk that you will not receive your upgraded CAT2 tokens. In this case, you should ask the issuer to make the upgrade. 210 | 211 | **I am the issuer of a CAT1 token. What should I do?** 212 | 213 | [Follow this document](https://docs.chia.net/docs/cat2/cat2-intro), which will guide you through the process of reissuing your token as a CAT2. 214 | 215 | #### Chia App Questions 216 | 217 | **I upgraded to 1.5, but I don't see any of my tokens yet. Did I do something wrong?** 218 | 219 | No, you didn't do anything wrong. The Chia Wallet app, as of 1.5, only shows you your XCH, and CAT2 balances. As not all CATs will be re-issued immediately, when your CAT2s show up in your wallet is dependent on when original issuers issue their updated CAT2s. 220 | 221 | **The balance of my airdrop in my wallet doesn't match the balance that the website shows me, what should I do?** 222 | 223 | First, please consult the CAT1 balance[ website](https://cat1.chia.net) to view your historical CAT1 balances. We expect the full reissuance process to take approximately a week to complete. 224 | 225 | If your CAT2 balances in the 1.5.0 wallet do not match the CAT1 historical reference, then check what your wallet derivation index is at and compare it to the derivation index. The derivation index is shown in the balance on the Tokens screen. 226 | 227 | ![Chia client](https://lh4.googleusercontent.com/lLwgjXXYjw22E5SX8TSDsFNB-XMPQ0zhS6OSh8UmP8kbp0NoWMxoAenYYhLKOUciFWdo7badLgTIvHksdUugE2k9W4SExUNdf7gdVA0IYO2XpzOaoSnFBJPpsfXnrkhOVMOs8CWjt08LasfNQuwnxPY) 228 | 229 | If the derivation index in your wallet is less than the highest derivation index found on the website, you will want to update the derivation index in the wallet. To do so, go to Settings > Derivation index and type in the number that you get from the [cat1.chia.net](https://www.chia.net/2022/07/25/cat1.chia.net) website. 230 | 231 | ![Chia client](https://lh3.googleusercontent.com/b2qA0DUWQX_J8gJLN0uChP5klN5lw3kase29QUcdx30EK-nnsGTWqTLj20G83DG9puVpSA07HkDcGy_9lNG82g5T2SGV0cY-gSA0eGbPtJ81ARr8xqUFyKY5TYiGzU9I-W90_HpVyb3MoiQlyX28iVA) 232 | 233 | **I've tried all the recommendations, but the reissuer didn't get my wallet balance correct. What should I do?** 234 | 235 | After trying all the above steps and at least a week has passed since the announcement and your CAT2 airdropped balance still doesn't match, then we recommend reaching out directly to the reissue of the relevant CAT1 token. 236 | 237 | **How do I cancel my open offers to exchange CATs?** 238 | 239 | From the Chia Wallet user interface: 240 | 241 | 1. Go to the Offers tab in the left hand navigation 242 | 243 | 2. Find all Offers you created that show a status of "Pending Accept" 244 | 245 | 3. Click on the three dots under "Actions" 246 | 247 | 4. Click on "Cancel Offer" 248 | 249 | 5. Ensure "Cancel on blockchain" option is selected 250 | 251 | 6. Enter a fee (optional, but recommended) 252 | 253 | 7. Click on "Cancel Offer" 254 | 255 | From the Chia Wallet command line: 256 | 257 | 1. Run chia wallet cancel_offer along with the parameters needed to cancel any open offers. See [here](https://docs.chia.net/docs/13cli/nft_cli) for documentation on how to use the command line arguments. 258 | 259 | **After upgrading to 1.5, I've lost all of my wallet transaction history. How do I access my previous transaction history with CATs or XCH?** 260 | 261 | After upgrading to 1.5, a new wallet database is created to preserve any previous copies of wallet DBs. You can install a previous version of the Chia wallet app, and that older client will look for your previous wallet db and display the transaction history for XCH, CATs, and NFTs that occurred in that wallet before upgrading to 1.5 262 | 263 | **How do I know when the updated tokens have been airdropped to my wallet?** 264 | 265 | You should follow the projects for the tokens that you own so you can be notified when they have begun running the airdrops. You can also monitor your Chia wallet app, and look under the "Manage token list" to see if a new CAT2 has been airdropped to you. 266 | 267 | **Why doesn't the balance in my wallet match the balance reported on the website?** 268 | 269 | Get the Derivation index from the website and update the derivation index in the Chia wallet app. This will ensure the balance reported on the website matches up with the balance in your wallet. 270 | 271 | **What is a "Derivation Index"?** 272 | 273 | The derivation index is a numeric value that is used to track how many wallet addresses have been used based on the most recent transaction. This helps establish a window for which wallet addresses to scan for on the blockchain to find all possible coins owned by a specific wallet. 274 | 275 | **Why do I see multiple tokens with the same value in my 1.5 wallet?** 276 | 277 | It is possible that you have received multiple identical airdrops from different parties. Only one of them will be the real CAT2 token. To determine which one is real, click "MANAGE TOKEN LIST" and click "Search on Tail Database". Only the original CAT1 issuer will be allowed to register their CAT2 equivalent on Tail Database, so you should use it as the source of truth for naming your CAT2 tokens. 278 | 279 | **Who can I contact if I have any problems or questions?** 280 | 281 | The Chia Network Support Team is available to answer questions and provide assistance through this process in the official[ Keybase support](https://keybase.io/team/chia_network.public) channel. 282 | 283 | #### [cat1.chia.net](https://cat1.chia.net/) Website Questions 284 | 285 | **How do I find the pubkey to enter into the website?** 286 | 287 | From the Chia Wallet user interface: 288 | 289 | 1. Go to "Select Key" screen and click on the "See private key" 290 | 291 | 2. Copy the "public key" from the list of keys available 292 | 293 | ![Chia client](https://lh3.googleusercontent.com/pDyW40byLBEr-5PgFiBJcrrSmE7ER_ui0CsD9_GZpz17zb6I7JymNfPI2uuVrgX5qN7M-P3bG-U0MWF2-jbgiRM2zq3XLidbX4RNz49Xxmz5UaM0XwKmOJ5T8fw5ezeN0hboy6FIMXvxBAvxKqJbxzo) 294 | 295 | From the Chia wallet command line: 296 | 297 | 1. Run chia keys show 298 | 299 | 2. Copy the "master public key" from the list of keys available 300 | 301 | **I checked the website (cat1.chia.net) and I don't see any tokens for my wallet, but there should be. What can I do?** 302 | 303 | Confirm the pubkey you entered into the website is correct and had a CAT1 token balance, and is for an unhardened key 304 | 305 | Click on the "Search next 1000" to see if your balance has been updated. 306 | 307 | **The balance that the website is reporting doesn't match what I expect. What should I do?** 308 | 309 | The website scans the first 1000 wallet receive addresses, and if the balance doesn't reflect what you expect, then you should hit the "search next 1000" for the website to scan and update the balance found. We expect most users to get their correct balance from the initial search, but some users might need an expanded search. 310 | 311 | **My CH21 balance is reported incorrectly on the website. What should I do?** 312 | 313 | CH21 tokens were issued to non-observer keys, so they won't show up in the website unless they've been transferred at some point using a wallet without forcing the non-observer key support. The CAT standard was released at the same time as support for observer keys, so generally most CATs and wallets will be supported by the website. Even if your CH21 tokens are not displayed on the website, a CAT2 version will still be airdropped into your 1.5 wallet. 314 | 315 | **I am running 1.5 and when I view an offer, instead of the CAT token, I see XCH. What's going on?** 316 | 317 | You are likely viewing an offer for a CAT1 which will be an invalid offer starting with version 1.5. The reason for this is because tail ids for CAT1s are no longer recognized by the wallet. 318 | 319 | ![Chia client](https://lh3.googleusercontent.com/VI0HRORsrG7w1jR3SgTS51BypcUiqhWMqzknlAtFxH0R19_SsKvEMO9X3BEs5-YNz3QNl8j6CraxleGHPS-p9UZbk7_3bthT4gNJp_696NvTgOOOBWBJLxn7wZkvnyzW6FwmdTXV21WBcsXU6fbxBFM) 320 | 321 | ### Wallet Developer Questions 322 | 323 | **I am developing a Chia wallet. What changes do I need to make to my code?** 324 | 325 | CAT2 inner puzzles do not enforce prepended announcements. If you preprend a coin announcement with 0xca (which was a requirement for CAT1), then the announcement will fail with ANNOUNCE_CONSUMED_FAILED. Instead, do not prepend inner puzzle announcements with anything. 326 | 327 | Note that announcements coming from the CAT layer still need to be prepended with 0xcb. This has not changed in CAT2. 328 | --------------------------------------------------------------------------------