├── LICENSE ├── README.md ├── assets ├── bitcoin-utxo-dump.gif ├── bitcoin-utxo-dump.odg └── bitcoin-utxo-dump.png ├── bitcoin ├── bech32 │ └── bech32.go ├── btcleveldb │ └── chainstate.go ├── crypto │ └── crypto.go └── keys │ └── keys.go └── utxodump.go /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2019, Greg Walker 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Bitcoin UTXO Dump 2 | 3 | **Warning:** This tool may corrupt your chainstate database. If it does, you will need to run `bitcoind -reindex-chainstate` the next time you run bitcoin, and this usually takes around a day to complete. It's not a terrible problem, but it can be annoying. I'm not entirely sure why it happens, so if you can figure out how to fix it, that would be cool. 4 | 5 | You can get around this issue by first copying the chainstate database to an alternate location and then run `bitcoin-utxo-dump` pointing to this alternate location. Here's a example: 6 | 7 | ```bash 8 | # 0. stop bitcoin daemon 9 | bitcoin-cli stop 10 | 11 | # 1. copy the chaninstate data to an alternative location 12 | rsync --delete -av ~/.bitcoin/chainstate/ ~/bitcoin-chainstate-clone/ 13 | 14 | # 2. now run the bitcoin-utxo-dump pointing to this alternate location 15 | bitcoin-utxo-dump -db ~/bitcoin-chainstate-clone/ 16 | ``` 17 | 18 | ----- 19 | 20 | ![](assets/bitcoin-utxo-dump.gif) 21 | 22 | Get a **list of every unspent bitcoin** in the blockchain. 23 | 24 | The program iterates over each entry in Bitcoin Core's `chainstate` [LevelDB](https://github.com/google/leveldb) database. It decompresses and decodes the data, and produces a human-readable text dump of all the [UTXO](https://learnmeabitcoin.com/technical/transaction/utxo/)s (unspent transaction outputs). 25 | 26 | ### Example CSV Results: 27 | 28 | ``` 29 | count,txid,vout,amount,type,address 30 | 1,033e83e3204b0cc28724e147f6fd140529b2537249f9c61c9de9972750030000,0,65279,p2pkh,1KaPHfvVWNZADup3Yc26SfVdkTDvvHySVX 31 | 2,e1c9467a885a156e56a29d9c854e65674d581ad75611b02290454b4862060000,1,9466355,p2pkh,1LpCmEejWLNfZigApMPwUY9nZTS8NTJCNS 32 | 3,a1f28c43f1f3d4821d0db42707737ea90616613099234f905dfc6ae2b4060000,1,339500,p2pkh,1FuphZ7xVPGrxthQT1S8X7nNQNByYxAT3V 33 | 4,818f5b9e3ede69da765d4c24684e813057c9b1f059e098661369b0a2ee060000,0,300000,p2pkh,18Y9yhjU9g2jjJmvaUy7TmUNZH9iPzQ4dd 34 | 5,d2f5e439152d076593a145581f8d76ea2e48ed155285b9a245cd42dd06070000,0,100000,p2pkh,1EKHTvovYWHfUJ6i9vsoidyTPQauCPH1qC 35 | 6,ea0c69fbd2389556b01771948ffc0507cf303bdc5a1b91b31acf9ecf6a070000,1,27668,p2pkh,1fkEhLpPKdmKtaxKdp4yDp1c87dF7GDub 36 | 7,05eafead65250a24b1592f8a006cbeab16a7b17ed2616507c5e0bd67bd070000,1,32000,p2pkh,15KmfJcGNfL29vpsSJ37uPzTQfr8Qe17Gq 37 | 8,2c0c985d384160d8c50c438bc67e639fe6047a7f2bac00a1238ca6a6d3070000,0,41936,p2pkh,17up1oPxBMTfZdehzy4v81KzLRHGDNX8ff 38 | 9,8261170b7ae26be70bd9e8f0e4bf19ce3571bb6464cdf9e478c471d372080000,1,4528208,p2pkh,1P6Ae7unrSjtx9J5SjWuwAdZBoWcbcjzBZ 39 | ... 40 | ``` 41 | 42 | ## Install 43 | 44 | First of all, you need to have a full copy of the blockchain. You also need to install LevelDB: 45 | 46 | ``` 47 | sudo apt install bitcoind 48 | sudo apt install libleveldb-dev 49 | ``` 50 | 51 | After that, if you have [Go](https://golang.org/) installed you can do: 52 | 53 | ``` 54 | go install github.com/in3rsha/bitcoin-utxo-dump@latest 55 | ``` 56 | 57 | This will create a binary called `bitcoin-utxo-dump`, which you can call from the command line: 58 | 59 | ``` 60 | $ bitcoin-utxo-dump 61 | ``` 62 | 63 | This will start dumping all of the UTXO database to a file called `utxodump.csv`. 64 | 65 | **NOTE:** This program reads the chainstate LevelDB database created by `bitcoind`, so you will need to download and sync `bitcoind` for this script to work. In other words, this script reads your own local copy of the blockchain. 66 | 67 | **NOTE:** LevelDB wasn't designed to be accessed by multiple programs at the same time, so make sure `bitcoind` isn't running before you start (`bitcoin-cli stop` should do it). 68 | 69 | 70 | ## Usage 71 | 72 | The basic command is: 73 | 74 | ``` 75 | $ bitcoin-utxo-dump 76 | ``` 77 | 78 | You can view the results in the terminal with the `-v` (verbose) flag (but this will make the script run about **3 times slower**): 79 | 80 | ``` 81 | $ bitcoin-utxo-dump -v 82 | ``` 83 | 84 | The results will be written to the file in the current directory called `utxodump.csv`. You can choose your own filename with the `-o` option: 85 | 86 | ``` 87 | $ bitcoin-utxo-dump -o ~/Desktop/utxodump.txt 88 | ``` 89 | 90 | If you know that the `chainstate` LevelDB folder is in a different location to the default (e.g. you want to get a UTXO dump of the _Testnet_ blockchain), use the `-db` option: 91 | 92 | ``` 93 | $ bitcoin-utxo-dump -db ~/.bitcoin/testnet3/chainstate/ 94 | ``` 95 | 96 | By default this script does not convert the public keys inside P2PK locking scripts to addresses (because technically they do not have an address). However, sometimes it may be useful to get addresses for them anyway for use with other APIs, so the following option allows you to return the "address" for UTXOs with P2PK locking scripts: 97 | 98 | ``` 99 | $ bitcoin-utxo-dump -p2pkaddresses 100 | ``` 101 | 102 | You can select what data the script outputs from the chainstate database with the `-f` (fields) option. This is useful if you know what data you need and want to _reduce the size of the results file_. 103 | 104 | ``` 105 | $ bitcoin-utxo-dump -f count,txid,vout,address 106 | $ bitcoin-utxo-dump -f count,txid,vout,height,coinbase,amount,script,type,address # all possible fields 107 | ``` 108 | 109 | * **count** - The count of the number of UTXOs in the database. 110 | * **txid** - [Transaction ID](https://learnmeabitcoin.com/technical/transaction/input/txid/) for the output. 111 | * **vout** - The index number of the transaction output (which output in the transaction is it?). 112 | * **height** - The height of the block the transaction was mined in. 113 | * **coinbase** - Whether the output is from a coinbase transaction (i.e. claiming a block reward). 114 | * **amount** - The value of the output in _satoshis_. 115 | * **script** - Details about the locking script placed on the output. For a P2PKH this is the hash160 of the compressed public key. For a P2PK script this a compressed public key (sometimes with a [prefix](https://github.com/in3rsha/bitcoin-chainstate-parser#3-third-varint) to indicate that the original script contained an uncompressed public key). For a P2SH script this is the hash160 of the script. For everything else it's the complete scriptpubkey. 116 | * **type** - The type of locking script (e.g. P2PK, P2PKH, P2SH, P2MS, P2WPKH, P2WSH, or non-standard) 117 | * **address** - The address the output is locked to (this is generally just the locking script in a shorter format with user-friendly characters). 118 | 119 | 120 | All other options can be found with `-h`: 121 | 122 | ``` 123 | $ bitcoin-utxo-dump -h 124 | ``` 125 | 126 | ## FAQ 127 | 128 | ### How long does this script take to run? 129 | 130 | It takes me about **20 minutes** to get all the UTXOs. 131 | 132 | This obviously this depends on how big the UTXO database is and how fast your computer is. For me, the UTXO database had 52 million entries, and I'm using a Thinkpad X220 (with a SSD). 133 | 134 | Either way, I'd probably make a cup of tea after it starts running. 135 | 136 | ### How big is the file? 137 | 138 | The file should be around **7GB** (roughly **2.5 times the size** of the LevelDB database: `du -h ~/.bitcoin/chainstate/`). 139 | 140 | Again, this depends on how many entries are in the UTXO database, but it also depends what _fields_ you choose to have in the results: 141 | 142 | ``` 143 | $ bitcoin-utxo-dump -f address # small file 144 | $ bitcoin-utxo-dump -f count,txid,vout,amount,type,address # bigger file 145 | $ bitcoin-utxo-dump -f count,txid,vout,height,coinbase,amount,nsize,script,type,address # biggest file 146 | ``` 147 | 148 | ### What versions of bitcoin does this tool work with? 149 | 150 | This tool works for Bitcoin Core [0.15.1](https://bitcoincore.org/en/releases/0.15.1/) and above. You can check your version with `bitcoind --version`. 151 | 152 | Older versions of bitcoind have a different chainstate LevelDB structure. The structure was updated in 0.15.1 to make reading from the database more memory-efficient. Here's an interesting talk by [Chris Jeffrey](https://youtu.be/0WCaoGiAOHE?t=8936) that explains how you could crash Bitcoin Core with the old chainstate database structure. 153 | 154 | Nonetheless, if you really want to parse an _old-style_ chainstate database, try one of the _similar tools_ at the bottom of this page. 155 | 156 | ### How does this program work? 157 | 158 | This program just iterates through all the entries in the LevelDB database at `~/.bitcoin/chainstate`. 159 | 160 | However, the data inside `~/.bitcoin/chainstate` has been _obfuscated_ (to prevent triggering anti-virus software) and _compressed_ (to reduce the size on disk), so it's far from being human-readable. This script just deobfuscates each entry and decodes/decompresses the data to get human-readable data for each UTXO in the database. 161 | 162 | ![](assets/bitcoin-utxo-dump.png) 163 | 164 | ### Can I parse the chainstate LevelDB myself? 165 | 166 | Sure. Most programming languages seem to have libraries for reading a LevelDB database. 167 | 168 | * [Go](https://github.com/syndtr/goleveldb) 169 | * [Ruby](https://github.com/wmorgan/leveldb-ruby) 170 | * [Python](https://github.com/wbolster/plyvel) 171 | 172 | The trickier part is decoding the data for each UTXO in the database: 173 | 174 | ``` 175 | type txid (little-endian) index (varint) 176 | \ | / 177 | <><--------------------------------------------------------------><> 178 | key: 430000155b9869d56c66d9e86e3c01de38e3892a42b99949fe109ac034fff6583900 179 | 180 | value: 71a9e87d62de25953e189f706bcf59263f15de1bf6c893bda9b045 <- obfuscated 181 | b12dcefd8f872536b12dcefd8f872536b12dcefd8f872536b12dce <- extended obfuscateKey 182 | c0842680ed5900a38f35518de4487c108e3810e6794fb68b189d8b <- deobfuscated (XOR) 183 | <----><----><><--------------------------------------> 184 | / | \ | 185 | varint varint varint script <- P2PKH/P2SH hash160, P2PK public key, or complete script 186 | | | nSize 187 | | | 188 | | amount (compressesed) 189 | | 190 | | 191 | 100000100001010100110 192 | <------------------> \ 193 | height coinbase 194 | ``` 195 | 196 | ## Thanks 197 | 198 | * This script was inspired by the [bitcoin_tools](https://github.com/sr-gi/bitcoin_tools) repo made by [Sergi Delgado Segura](https://github.com/sr-gi). I wanted to see if I could get a faster dump of the UTXO database by writing the program in Go, in addition to getting the **addresses** for each of the UTXOs. The decoding and decompressing code in his repo helped me to write this tool. 199 | 200 | ### Similar Tools 201 | 202 | * [github.com/sr-gi/bitcoin_tools](https://github.com/sr-gi/bitcoin_tools) 203 | * [github.com/in3rsha/bitcoin-chainstate-parser](https://github.com/in3rsha/bitcoin-chainstate-parser) 204 | * [github.com/mycroft/chainstate](https://github.com/mycroft/chainstate) 205 | * [laanwj (unfinished)](https://github.com/bitcoin/bitcoin/pull/7759) 206 | 207 | ### Links 208 | 209 | * 210 | * 211 | * 212 | * 213 | * 214 | * 215 | -------------------------------------------------------------------------------- /assets/bitcoin-utxo-dump.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/in3rsha/bitcoin-utxo-dump/9b1c015308f779ac529083ed7922cc551b8ddb53/assets/bitcoin-utxo-dump.gif -------------------------------------------------------------------------------- /assets/bitcoin-utxo-dump.odg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/in3rsha/bitcoin-utxo-dump/9b1c015308f779ac529083ed7922cc551b8ddb53/assets/bitcoin-utxo-dump.odg -------------------------------------------------------------------------------- /assets/bitcoin-utxo-dump.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/in3rsha/bitcoin-utxo-dump/9b1c015308f779ac529083ed7922cc551b8ddb53/assets/bitcoin-utxo-dump.png -------------------------------------------------------------------------------- /bitcoin/bech32/bech32.go: -------------------------------------------------------------------------------- 1 | // Package bech32 reference implementation for Bech32 and segwit addresses. 2 | // Copyright (c) 2017 Takatoshi Nakagawa 3 | // 4 | // Permission is hereby granted, free of charge, to any person obtaining a copy 5 | // of this software and associated documentation files (the "Software"), to deal 6 | // in the Software without restriction, including without limitation the rights 7 | // to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 8 | // copies of the Software, and to permit persons to whom the Software is 9 | // furnished to do so, subject to the following conditions: 10 | // 11 | // The above copyright notice and this permission notice shall be included in 12 | // all copies or substantial portions of the Software. 13 | // 14 | // THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 15 | // IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 16 | // FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 17 | // AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 18 | // LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 19 | // OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 20 | // THE SOFTWARE. 21 | package bech32 22 | 23 | import ( 24 | "bytes" 25 | "fmt" 26 | "strings" 27 | ) 28 | 29 | const ( 30 | encBech32 = 1 31 | encBech32m = 2 32 | invalidEncoding = -1 33 | bech32mConst = 0x2bc830a3 34 | ) 35 | 36 | var charset = "qpzry9x8gf2tvdw0s3jn54khce6mua7l" 37 | 38 | var generator = []int{0x3b6a57b2, 0x26508e6d, 0x1ea119fa, 0x3d4233dd, 0x2a1462b3} 39 | 40 | func polymod(values []int) int { 41 | chk := 1 42 | for _, v := range values { 43 | top := chk >> 25 44 | chk = (chk&0x1ffffff)<<5 ^ v 45 | for i := 0; i < 5; i++ { 46 | if (top>>uint(i))&1 == 1 { 47 | chk ^= generator[i] 48 | } 49 | } 50 | } 51 | return chk 52 | } 53 | 54 | func hrpExpand(hrp string) []int { 55 | ret := []int{} 56 | for _, c := range hrp { 57 | ret = append(ret, int(c>>5)) 58 | } 59 | ret = append(ret, 0) 60 | for _, c := range hrp { 61 | ret = append(ret, int(c&31)) 62 | } 63 | return ret 64 | } 65 | 66 | func verifyChecksum(hrp string, data []int) int { 67 | constant := polymod(append(hrpExpand(hrp), data...)) 68 | if constant == 1 { 69 | return encBech32 70 | } else if constant == bech32mConst { 71 | return encBech32m 72 | } 73 | 74 | return invalidEncoding 75 | } 76 | 77 | func createChecksum(hrp string, data []int, spec int) []int { 78 | values := append(append(hrpExpand(hrp), data...), []int{0, 0, 0, 0, 0, 0}...) 79 | constant := 1 80 | if spec == encBech32m { 81 | constant = bech32mConst 82 | } 83 | 84 | mod := polymod(values) ^ constant 85 | 86 | ret := make([]int, 6) 87 | for p := 0; p < len(ret); p++ { 88 | ret[p] = (mod >> uint(5*(5-p))) & 31 89 | } 90 | return ret 91 | } 92 | 93 | // Encode encodes hrp(human-readable part) and data(32bit data array), returns Bech32 / or error 94 | // if hrp is uppercase, return uppercase Bech32 95 | func Encode(hrp string, data []int, spec int) (string, error) { 96 | if (len(hrp) + len(data) + 7) > 90 { 97 | return "", fmt.Errorf("too long : hrp length=%d, data length=%d", len(hrp), len(data)) 98 | } 99 | if len(hrp) < 1 { 100 | return "", fmt.Errorf("invalid hrp : hrp=%v", hrp) 101 | } 102 | for p, c := range hrp { 103 | if c < 33 || c > 126 { 104 | return "", fmt.Errorf("invalid character human-readable part : hrp[%d]=%d", p, c) 105 | } 106 | } 107 | if strings.ToUpper(hrp) != hrp && strings.ToLower(hrp) != hrp { 108 | return "", fmt.Errorf("mix case : hrp=%v", hrp) 109 | } 110 | lower := strings.ToLower(hrp) == hrp 111 | hrp = strings.ToLower(hrp) 112 | combined := append(data, createChecksum(hrp, data, spec)...) 113 | var ret bytes.Buffer 114 | ret.WriteString(hrp) 115 | ret.WriteString("1") 116 | for idx, p := range combined { 117 | if p < 0 || p >= len(charset) { 118 | return "", fmt.Errorf("invalid data : data[%d]=%d", idx, p) 119 | } 120 | ret.WriteByte(charset[p]) 121 | } 122 | if lower { 123 | return ret.String(), nil 124 | } 125 | return strings.ToUpper(ret.String()), nil 126 | } 127 | 128 | // Decode decodes bechString(Bech32/Bech32m) returns hrp(human-readable part), data(32bit data array) and spec (int) / or error 129 | func Decode(bechString string) (string, []int, int, error) { 130 | if len(bechString) > 90 { 131 | return "", nil, invalidEncoding, fmt.Errorf("too long : len=%d", len(bechString)) 132 | } 133 | if strings.ToLower(bechString) != bechString && strings.ToUpper(bechString) != bechString { 134 | return "", nil, invalidEncoding, fmt.Errorf("mixed case") 135 | } 136 | bechString = strings.ToLower(bechString) 137 | pos := strings.LastIndex(bechString, "1") 138 | if pos < 1 || pos+7 > len(bechString) { 139 | return "", nil, invalidEncoding, fmt.Errorf("separator '1' at invalid position : pos=%d , len=%d", pos, len(bechString)) 140 | } 141 | hrp := bechString[0:pos] 142 | for p, c := range hrp { 143 | if c < 33 || c > 126 { 144 | return "", nil, invalidEncoding, fmt.Errorf("invalid character human-readable part : bechString[%d]=%d", p, c) 145 | } 146 | } 147 | data := []int{} 148 | for p := pos + 1; p < len(bechString); p++ { 149 | d := strings.Index(charset, fmt.Sprintf("%c", bechString[p])) 150 | if d == -1 { 151 | return "", nil, invalidEncoding, fmt.Errorf("invalid character data part : bechString[%d]=%d", p, bechString[p]) 152 | } 153 | data = append(data, d) 154 | } 155 | 156 | spec := verifyChecksum(hrp, data) 157 | if spec == invalidEncoding { 158 | return "", nil, invalidEncoding, fmt.Errorf("invalid checksum") 159 | } 160 | return hrp, data[:len(data)-6], spec, nil 161 | } 162 | 163 | func convertbits(data []int, frombits, tobits uint, pad bool) ([]int, error) { 164 | acc := 0 165 | bits := uint(0) 166 | ret := []int{} 167 | maxv := (1 << tobits) - 1 168 | for idx, value := range data { 169 | if value < 0 || (value>>frombits) != 0 { 170 | return nil, fmt.Errorf("invalid data range : data[%d]=%d (frombits=%d)", idx, value, frombits) 171 | } 172 | acc = (acc << frombits) | value 173 | bits += frombits 174 | for bits >= tobits { 175 | bits -= tobits 176 | ret = append(ret, (acc>>bits)&maxv) 177 | } 178 | } 179 | if pad { 180 | if bits > 0 { 181 | ret = append(ret, (acc<<(tobits-bits))&maxv) 182 | } 183 | } else if bits >= frombits { 184 | return nil, fmt.Errorf("illegal zero padding") 185 | } else if ((acc << (tobits - bits)) & maxv) != 0 { 186 | return nil, fmt.Errorf("non-zero padding") 187 | } 188 | return ret, nil 189 | } 190 | 191 | // SegwitAddrDecode decodes hrp(human-readable part) Segwit Address(string), returns version(int) and data(bytes array) / or error 192 | func SegwitAddrDecode(hrp, addr string) (int, []int, error) { 193 | dechrp, data, spec, err := Decode(addr) 194 | if err != nil { 195 | return -1, nil, err 196 | } 197 | if dechrp != hrp { 198 | return -1, nil, fmt.Errorf("invalid human-readable part : %s != %s", hrp, dechrp) 199 | } 200 | if len(data) < 1 { 201 | return -1, nil, fmt.Errorf("invalid decode data length : %d", len(data)) 202 | } 203 | if data[0] > 16 { 204 | return -1, nil, fmt.Errorf("invalid witness version : %d", data[0]) 205 | } 206 | res, err := convertbits(data[1:], 5, 8, false) 207 | if err != nil { 208 | return -1, nil, err 209 | } 210 | if len(res) < 2 || len(res) > 40 { 211 | return -1, nil, fmt.Errorf("invalid convertbits length : %d", len(res)) 212 | } 213 | if data[0] == 0 && len(res) != 20 && len(res) != 32 { 214 | return -1, nil, fmt.Errorf("invalid program length for witness version 0 (per BIP141) : %d", len(res)) 215 | } 216 | if data[0] == 0 && spec != encBech32 || data[0] != 0 && spec != encBech32m { 217 | return -1, nil, fmt.Errorf("witness version and encoding don't match : %d and %d", data[0], spec) 218 | } 219 | return data[0], res, nil 220 | } 221 | 222 | // SegwitAddrEncode encodes hrp(human-readable part) , version(int) and data(bytes array), returns Segwit Address / or error 223 | func SegwitAddrEncode(hrp string, version int, program []int) (string, error) { 224 | spec := encBech32m 225 | if version == 0 { 226 | spec = encBech32 227 | } 228 | 229 | if version < 0 || version > 16 { 230 | return "", fmt.Errorf("invalid witness version : %d", version) 231 | } 232 | if len(program) < 2 || len(program) > 40 { 233 | return "", fmt.Errorf("invalid program length : %d", len(program)) 234 | } 235 | if version == 0 && len(program) != 20 && len(program) != 32 { 236 | return "", fmt.Errorf("invalid program length for witness version 0 (per BIP141) : %d", len(program)) 237 | } 238 | if version == 0 && spec != encBech32 || version != 0 && spec != encBech32m { 239 | return "", fmt.Errorf("witness version and encoding don't match : %d and %d", version, spec) 240 | } 241 | 242 | data, err := convertbits(program, 8, 5, true) 243 | if err != nil { 244 | return "", err 245 | } 246 | 247 | ret, err := Encode(hrp, append([]int{version}, data...), spec) 248 | if err != nil { 249 | return "", err 250 | } 251 | return ret, nil 252 | } 253 | -------------------------------------------------------------------------------- /bitcoin/btcleveldb/chainstate.go: -------------------------------------------------------------------------------- 1 | package btcleveldb 2 | 3 | import "math" // math.Pow() in decompress_value() 4 | 5 | func Varint128Read(bytes []byte, offset int) ([]byte, int) { // take a byte array and return (byte array and number of bytes read) 6 | 7 | // store bytes 8 | result := []byte{} // empty byte slice 9 | 10 | // loop through bytes 11 | for _, v := range bytes[offset:] { // start reading from an offset 12 | 13 | // store each byte as you go 14 | result = append(result, v) 15 | 16 | // Bitwise AND each of them with 128 (0b10000000) to check if the 8th bit has been set 17 | set := v & 128 // 0b10000000 is same as 1 << 7 18 | 19 | // When you get to one without the 8th bit set, return that byte slice 20 | if set == 0 { 21 | return result, len(result) 22 | // Also return the number of bytes read 23 | } 24 | } 25 | 26 | // Return zero bytes read if we haven't managed to read bytes properly 27 | return result, 0 28 | 29 | } 30 | 31 | func Varint128Decode(bytes []byte) int64 { // takes a byte slice, returns an int64 (makes sure it work on 32 bit systems) 32 | 33 | // total 34 | var n int64 = 0 35 | 36 | for _, v := range bytes { 37 | 38 | // 1. shift n left 7 bits (add some extra bits to work with) 39 | // 00000000 40 | n = n << 7 41 | 42 | // 2. set the last 7 bits of each byte in to the total value 43 | // AND extracts 7 bits only 10111001 <- these are the bits of each byte 44 | // 1111111 45 | // 0111001 <- don't want the 8th bit (just indicated if there were more bytes in the varint) 46 | // OR sets the 7 bits 47 | // 00000000 <- the result 48 | // 0111001 <- the bits we want to set 49 | // 00111001 50 | n = n | int64(v & 127) 51 | 52 | // 3. add 1 each time (only for the ones where the 8th bit is set) 53 | if (v & 128 != 0) { // 0b10000000 <- AND to check if the 8th bit is set 54 | // 1 << 7 <- could always bit shift to get 128 55 | n++ 56 | } 57 | 58 | } 59 | 60 | return n 61 | // 11101000000111110110 62 | 63 | } 64 | 65 | func DecompressValue(x int64) int64 { 66 | 67 | var n int64 = 0 // decompressed value 68 | 69 | // Return value if it is zero (nothing to decompress) 70 | if x == 0 { 71 | return 0 72 | } 73 | 74 | // Decompress... 75 | x = x - 1 // subtract 1 first 76 | e := x % 10 // remainder mod 10 77 | x = x / 10 // quotient mod 10 (reduce x down by 10) 78 | 79 | // If the remainder is less than 9 80 | if e < 9 { 81 | d := x % 9 // remainder mod 9 82 | x = x / 9 // (reduce x down by 9) 83 | n = x * 10 + d + 1 // work out n 84 | } else { 85 | n = x + 1 86 | } 87 | 88 | // Multiply n by 10 to the power of the first remainder 89 | result := float64(n) * math.Pow(10, float64(e)) // math.Pow takes a float and returns a float 90 | 91 | // manual exponentiation 92 | // multiplier := 1 93 | // for i := 0; i < e; i++ { 94 | // multiplier *= 10 95 | // } 96 | // fmt.Println(multiplier) 97 | 98 | return int64(result) 99 | 100 | } 101 | -------------------------------------------------------------------------------- /bitcoin/crypto/crypto.go: -------------------------------------------------------------------------------- 1 | package crypto 2 | 3 | import "crypto/sha256" 4 | import "golang.org/x/crypto/ripemd160" 5 | 6 | 7 | func Hash256(bytes []byte) []byte { 8 | hash1 := sha256.Sum256(bytes[:]) // hash the byte slice twice through sha256 9 | hash2 := sha256.Sum256(hash1[:]) // need to coerce the [32]byte{} result above to a []byte{} 10 | return hash2[:] // return slice []byte{}, not [32]byte{} 11 | } 12 | 13 | func Hash160(bytes []byte) []byte { 14 | hash1 := sha256.Sum256(bytes[:]) // hash the byte slice through sha256 first 15 | hash2 := ripemd160.New() // ripemd160 hash function 16 | hash2.Write(hash1[:]) // need to coerce the [32]byte{} result above to a []byte{} 17 | result := hash2.Sum(nil) 18 | return result[:] // return slice []byte{}, not [32]byte{} 19 | } 20 | 21 | func Checksum(bytes []byte) []byte { 22 | hash := Hash256(bytes) // hash256 the bytes 23 | cksum := hash[:4] // get last 4 bytes 24 | return cksum 25 | } 26 | -------------------------------------------------------------------------------- /bitcoin/keys/keys.go: -------------------------------------------------------------------------------- 1 | package keys 2 | 3 | import "github.com/in3rsha/bitcoin-utxo-dump/bitcoin/crypto" 4 | import "github.com/akamensky/base58" 5 | import "math/big" 6 | 7 | func Hash160ToAddress(hash160 []byte, prefix []byte) string { 8 | // 9 | // prefix hash160 checksum 10 | // \ \ \ 11 | // [00] [203 194 152 111 249 174 214 130 89 32 174 206 20 170 111 83 130 202 85 128] [56 132 221 179] 12 | // \ / base58 encode 13 | // ------------------------------------------address----------------------------------------------- 14 | 15 | hash160_with_prefix := append(prefix, hash160...) // prepend prefix to hash160pubkey (... unpacks the slice) 16 | hash160_prepared := append(hash160_with_prefix, crypto.Checksum(hash160_with_prefix)...) // add checksum to the end 17 | address := base58.Encode(hash160_prepared) 18 | return address 19 | } 20 | 21 | func PublicKeyToAddress(publickey []byte, prefix []byte) string { 22 | hash160 := crypto.Hash160(publickey) 23 | address := Hash160ToAddress(hash160, prefix) 24 | return address 25 | } 26 | 27 | func DecompressPublicKey(publickey []byte) []byte { // decompressing public keys from P2PK scripts 28 | // first byte (indicates whether y is even or odd) 29 | prefix := publickey[0:1] 30 | 31 | // remaining bytes (x coordinate) 32 | x := publickey[1:] 33 | 34 | // y^2 = x^3 + 7 mod p 35 | p, _ := new(big.Int).SetString("0xfffffffffffffffffffffffffffffffffffffffffffffffffffffffefffffc2f", 0) 36 | x_int := new(big.Int).SetBytes(x) 37 | x_3 := new(big.Int).Exp(x_int, big.NewInt(3), p) 38 | y_sq := new(big.Int).Add(x_3, big.NewInt(7)) 39 | y_sq = new(big.Int).Mod(y_sq, p) 40 | 41 | // square root of y - secp256k1 is chosen so that the square root of y is y^((p+1)/4) 42 | y := new(big.Int).Exp(y_sq, new(big.Int).Div(new(big.Int).Add(p, big.NewInt(1)), big.NewInt(4)), p) 43 | 44 | // determine if the y we have caluclated is even or odd 45 | y_mod_2 := new(big.Int).Mod(y, big.NewInt(2)) 46 | 47 | // if prefix is even (indicating an even y value) and y is odd, use other y value 48 | if (int(prefix[0]) % 2 == 0) && (y_mod_2.Cmp(big.NewInt(0)) != 0) { // Cmp returns 0 if equal 49 | y = new(big.Int).Mod(new(big.Int).Sub(p, y), p) 50 | } 51 | 52 | // if prefix is odd (indicating an odd y value) and y is even, use other y value 53 | if (int(prefix[0]) % 2 != 0) && (y_mod_2.Cmp(big.NewInt(0)) == 0) { // Cmp returns 0 if equal 54 | y = new(big.Int).Mod(new(big.Int).Sub(p, y), p) 55 | } 56 | 57 | // convert y to byte array 58 | y_bytes := y.Bytes() 59 | 60 | // make sure y value is 32 bytes in length 61 | if (len(y_bytes) < 32) { 62 | y_bytes = make([]byte, 32) 63 | copy(y_bytes[32-len(y.Bytes()):], y.Bytes()) 64 | } 65 | 66 | // return full x and y coordinates (with 0x04 prefix) as a byte array 67 | uncompressed := []byte{0x04} 68 | uncompressed = append(uncompressed, x...) 69 | uncompressed = append(uncompressed, y_bytes...) 70 | 71 | return uncompressed 72 | } -------------------------------------------------------------------------------- /utxodump.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | // local packages 4 | import "github.com/in3rsha/bitcoin-utxo-dump/bitcoin/btcleveldb" // chainstate leveldb decoding functions 5 | import "github.com/in3rsha/bitcoin-utxo-dump/bitcoin/keys" // bitcoin addresses 6 | import "github.com/in3rsha/bitcoin-utxo-dump/bitcoin/bech32" // segwit bitcoin addresses 7 | 8 | import "github.com/syndtr/goleveldb/leveldb" // go get github.com/syndtr/goleveldb/leveldb 9 | import "github.com/syndtr/goleveldb/leveldb/opt" // set no compression when opening leveldb 10 | import "flag" // command line arguments 11 | import "fmt" 12 | import "os" // open file for writing 13 | import "os/exec" // execute shell command (check bitcoin isn't running) 14 | import "os/signal" // catch interrupt signals CTRL-C to close db connection safely 15 | import "syscall" // catch kill commands too 16 | import "bufio" // bulk writing to file 17 | import "encoding/hex" // convert byte slice to hexadecimal 18 | import "strings" // parsing flags from command line 19 | import "runtime" // Check OS type for file-handler limitations 20 | 21 | func main() { 22 | 23 | // Version 24 | const Version = "1.0.1" 25 | 26 | // Set default chainstate LevelDB and output file 27 | defaultfolder := fmt.Sprintf("%s/.bitcoin/chainstate/", os.Getenv("HOME")) // %s = string 28 | defaultfile := "utxodump.csv" 29 | 30 | // Command Line Options (Flags) 31 | chainstate := flag.String("db", defaultfolder, "Location of bitcoin chainstate db.") // chainstate folder 32 | file := flag.String("o", defaultfile, "Name of file to dump utxo list to.") // output file 33 | fields := flag.String("f", "count,txid,vout,amount,type,address", "Fields to include in output. [count,txid,vout,height,amount,coinbase,nsize,script,type,address]") 34 | testnetflag := flag.Bool("testnet", false, "Is the chainstate leveldb for testnet?") // true/false 35 | verbose := flag.Bool("v", false, "Print utxos as we process them (will be about 3 times slower with this though).") 36 | version := flag.Bool("version", false, "Print version.") 37 | p2pkaddresses := flag.Bool("p2pkaddresses", false, "Convert public keys in P2PK locking scripts to addresses also.") // true/false 38 | nowarnings := flag.Bool("nowarnings", false, "Ignore warnings if bitcoind is running in the background.") // true/false 39 | quiet := flag.Bool("quiet", false, "Do not display any progress or results.") // true/false 40 | flag.Parse() // execute command line parsing for all declared flags 41 | 42 | // Check bitcoin isn't running first 43 | if ! *nowarnings { 44 | cmd := exec.Command("bitcoin-cli", "getnetworkinfo") 45 | _, err := cmd.Output() 46 | if err == nil { 47 | fmt.Println("Bitcoin is running. You should shut it down with `bitcoin-cli stop` first. We don't want to access the chainstate LevelDB while Bitcoin is running.") 48 | fmt.Println("Note: If you do stop bitcoind, make sure that it won't auto-restart (e.g. if it's running as a systemd service).") 49 | 50 | // Ask if you want to continue anyway (e.g. if you've copied the chainstate to a new location and bitcoin is still running) 51 | reader := bufio.NewReader(os.Stdin) 52 | fmt.Printf("%s [y/n] (default n): ", "Do you wish to continue anyway?") 53 | response, _ := reader.ReadString('\n') 54 | response = strings.ToLower(strings.TrimSpace(response)) 55 | 56 | if response != "y" && response != "yes" { 57 | return 58 | } 59 | } 60 | } 61 | 62 | // Check if OS type is Mac OS, then increase ulimit -n to 4096 filehandler during runtime and reset to 1024 at the end 63 | // Mac OS standard is 1024 64 | // Linux standard is already 4096 which is also "max" for more edit etc/security/limits.conf 65 | if runtime.GOOS == "darwin" { 66 | cmd2 := exec.Command("ulimit", "-n", "4096") 67 | fmt.Println("setting ulimit 4096\n") 68 | _, err := cmd2.Output() 69 | if err != nil { 70 | fmt.Println("setting new ulimit failed with %s\n", err) 71 | } 72 | defer exec.Command("ulimit", "-n", "1024") 73 | } 74 | 75 | // Show Version 76 | if *version { 77 | fmt.Println(Version) 78 | os.Exit(0) 79 | } 80 | 81 | // Mainnet or Testnet (for encoding addresses correctly) 82 | testnet := false 83 | if *testnetflag == true { // check testnet flag 84 | testnet = true 85 | } else { // only check the chainstate path if testnet flag has not been explicitly set to true 86 | if strings.Contains(*chainstate, "testnet") { // check the chainstate path 87 | testnet = true 88 | } 89 | } 90 | 91 | // Check chainstate LevelDB folder exists 92 | if _, err := os.Stat(*chainstate); os.IsNotExist(err) { 93 | fmt.Println("Couldn't find", *chainstate) 94 | return 95 | } 96 | 97 | // Select bitcoin chainstate leveldb folder 98 | // open leveldb without compression to avoid corrupting the database for bitcoin 99 | opts := &opt.Options{ 100 | Compression: opt.NoCompression, 101 | } 102 | // https://bitcoin.stackexchange.com/questions/52257/chainstate-leveldb-corruption-after-reading-from-the-database 103 | // https://github.com/syndtr/goleveldb/issues/61 104 | // https://godoc.org/github.com/syndtr/goleveldb/leveldb/opt 105 | 106 | db, err := leveldb.OpenFile(*chainstate, opts) // You have got to dereference the pointer to get the actual value 107 | if err != nil { 108 | fmt.Println("Couldn't open LevelDB.") 109 | fmt.Println(err) 110 | return 111 | } 112 | defer db.Close() 113 | 114 | // Output Fields - build output from flags passed in 115 | output := map[string]string{} // we will add to this as we go through each utxo in the database 116 | fieldsAllowed := []string{"count", "txid", "vout", "height", "coinbase", "amount", "nsize", "script", "type", "address"} 117 | 118 | // Create a map of selected fields 119 | fieldsSelected := map[string]bool{"count":false, "txid":false, "vout":false, "height":false, "coinbase":false, "amount":false, "nsize":false, "script":false, "type":false, "address":false} 120 | 121 | // Check that all the given fields are included in the fieldsAllowed array 122 | for _, v := range strings.Split(*fields, ",") { 123 | exists := false 124 | for _, w := range fieldsAllowed { 125 | if v == w { // check each field against every element in the fieldsAllowed array 126 | exists = true 127 | } 128 | } 129 | if exists == false { 130 | fmt.Printf("'%s' is not a field you can use for the output.\n", v) 131 | fieldsList := "" 132 | for _, v := range fieldsAllowed { 133 | fieldsList += v 134 | fieldsList += "," 135 | } 136 | fieldsList = fieldsList[:len(fieldsList)-1] // remove trailing comma 137 | fmt.Printf("Choose from the following: %s\n", fieldsList) 138 | return 139 | } 140 | // Set field in fieldsSelected map - helps to determine what and what not to calculate later on (to speed processing up) 141 | if exists == true { 142 | fieldsSelected[v] = true 143 | } 144 | } 145 | 146 | // Open file to write results to. 147 | f, err := os.Create(*file) // os.OpenFile("filename.txt", os.O_APPEND, 0666) 148 | if err != nil { 149 | panic(err) 150 | } 151 | defer f.Close() 152 | if ! *quiet { 153 | fmt.Printf("Processing %s and writing results to %s\n", *chainstate, *file) 154 | } 155 | 156 | // Create file buffer to speed up writing to the file. 157 | writer := bufio.NewWriter(f) 158 | defer writer.Flush() // Flush the bufio buffer to the file before this script ends 159 | 160 | // CSV Headers 161 | csvheader := "" 162 | for _, v := range strings.Split(*fields, ",") { 163 | csvheader += v 164 | csvheader += "," 165 | } // count,txid,vout, 166 | csvheader = csvheader[:len(csvheader)-1] // remove trailing , 167 | if ! *quiet { 168 | fmt.Println(csvheader) 169 | } 170 | fmt.Fprintln(writer, csvheader) // write to file 171 | 172 | // Stats - keep track of interesting stats as we read through leveldb. 173 | var totalAmount int64 = 0 // total amount of satoshis 174 | scriptTypeCount := map[string]int{"p2pk":0, "p2pkh":0, "p2sh":0, "p2ms":0, "p2wpkh":0, "p2wsh":0, "p2tr": 0, "non-standard": 0} // count each script type 175 | 176 | 177 | // Declare obfuscateKey (a byte slice) 178 | var obfuscateKey []byte // obfuscateKey := make([]byte, 0) 179 | 180 | // Iterate over LevelDB keys 181 | iter := db.NewIterator(nil, nil) 182 | // NOTE: iter.Release() comes after the iteration (not deferred here) 183 | // err := iter.Error() 184 | // fmt.Println(err) 185 | 186 | // Catch signals that interrupt the script so that we can close the database safely (hopefully not corrupting it) 187 | c := make(chan os.Signal, 1) 188 | signal.Notify(c, os.Interrupt, syscall.SIGTERM) 189 | go func() { // goroutine 190 | <-c // receive from channel 191 | if ! *quiet { 192 | fmt.Println("Interrupt signal caught. Shutting down gracefully.") 193 | } 194 | // iter.Release() // release database iterator 195 | db.Close() // close databse 196 | writer.Flush() // flush bufio to the file 197 | f.Close() // close file 198 | os.Exit(0) // exit 199 | }() 200 | 201 | i := 0 202 | for iter.Next() { 203 | 204 | key := iter.Key() 205 | value := iter.Value() 206 | 207 | // first byte in key indicates the type of key we've got for leveldb 208 | prefix := key[0] 209 | 210 | // obfuscateKey (first key) 211 | if (prefix == 14) { // 14 = obfuscateKey 212 | obfuscateKey = value 213 | } 214 | 215 | // utxo entry 216 | if (prefix == 67) { // 67 = 0x43 = C = "utxo" 217 | 218 | // --- 219 | // Key 220 | // --- 221 | 222 | // 430000155b9869d56c66d9e86e3c01de38e3892a42b99949fe109ac034fff6583900 223 | // <><--------------------------------------------------------------><> 224 | // / | \ 225 | // type txid (little-endian) index (varint) 226 | 227 | // txid 228 | if fieldsSelected["txid"] { 229 | txidLE := key[1:33] // little-endian byte order 230 | 231 | // txid - reverse byte order 232 | txid := make([]byte, 0) // create empty byte slice (dont want to mess with txid directly) 233 | for i := len(txidLE)-1; i >= 0; i-- { // run backwards through the txid slice 234 | txid = append(txid, txidLE[i]) // append each byte to the new byte slice 235 | } 236 | output["txid"] = hex.EncodeToString(txid) // add to output results map 237 | } 238 | 239 | // vout 240 | if fieldsSelected["vout"] { 241 | index := key[33:] 242 | 243 | // convert varint128 index to an integer 244 | vout := btcleveldb.Varint128Decode(index) 245 | output["vout"] = fmt.Sprintf("%d",vout) 246 | } 247 | 248 | // ----- 249 | // Value 250 | // ----- 251 | 252 | // Only deobfuscate and get data from the Value if something is needed from it (improves speed if you just want the txid:vout) 253 | if fieldsSelected["type"] || fieldsSelected["height"] || fieldsSelected["coinbase"] || fieldsSelected["amount"] || fieldsSelected["nsize"] || fieldsSelected["script"] || fieldsSelected["address"] { 254 | 255 | // Copy the obfuscateKey ready to extend it 256 | obfuscateKeyExtended := obfuscateKey[1:] // ignore the first byte, as that just tells you the size of the obfuscateKey 257 | 258 | // Extend the obfuscateKey so it's the same length as the value 259 | for i, k := len(obfuscateKeyExtended), 0; len(obfuscateKeyExtended) < len(value); i, k = i+1, k+1 { 260 | // append each byte of obfuscateKey to the end until it's the same length as the value 261 | obfuscateKeyExtended = append(obfuscateKeyExtended, obfuscateKeyExtended[k]) 262 | // Example 263 | // [8 175 184 95 99 240 37 253 115 181 161 4 33 81 167 111 145 131 0 233 37 232 118 180 123 120 78] 264 | // [8 177 45 206 253 143 135 37 54] <- obfuscate key 265 | // [8 177 45 206 253 143 135 37 54 8 177 45 206 253 143 135 37 54 8 177 45 206 253 143 135 37 54] <- extended 266 | } 267 | 268 | // XOR the value with the obfuscateKey (xor each byte) to de-obfuscate the value 269 | var xor []byte // create a byte slice to hold the xor results 270 | for i := range value { 271 | result := value[i] ^ obfuscateKeyExtended[i] 272 | xor = append(xor, result) 273 | } 274 | 275 | // ----- 276 | // Value 277 | // ----- 278 | 279 | // value: 71a9e87d62de25953e189f706bcf59263f15de1bf6c893bda9b045 <- obfuscated 280 | // b12dcefd8f872536b12dcefd8f872536b12dcefd8f872536b12dce <- extended obfuscateKey (XOR) 281 | // c0842680ed5900a38f35518de4487c108e3810e6794fb68b189d8b <- deobfuscated 282 | // <----><----><><--------------------------------------> 283 | // / | \ | 284 | // varint varint varint script <- P2PKH/P2SH hash160, P2PK public key, or complete script 285 | // | | nSize 286 | // | | 287 | // | amount (compressesed) 288 | // | 289 | // | 290 | // 100000100001010100110 291 | // <------------------> \ 292 | // height coinbase 293 | 294 | offset := 0 295 | 296 | // First Varint 297 | // ------------ 298 | // b98276a2ec7700cbc2986ff9aed6825920aece14aa6f5382ca5580 299 | // <----> 300 | varint, bytesRead := btcleveldb.Varint128Read(xor, 0) // start reading at 0 301 | offset += bytesRead 302 | varintDecoded := btcleveldb.Varint128Decode(varint) 303 | 304 | if fieldsSelected["height"] || fieldsSelected["coinbase"] { 305 | 306 | // Height (first bits) 307 | height := varintDecoded >> 1 // right-shift to remove last bit 308 | output["height"] = fmt.Sprintf("%d", height) 309 | 310 | // Coinbase (last bit) 311 | coinbase := varintDecoded & 1 // AND to extract right-most bit 312 | output["coinbase"] = fmt.Sprintf("%d", coinbase) 313 | } 314 | 315 | // Second Varint 316 | // ------------- 317 | // b98276a2ec7700cbc2986ff9aed6825920aece14aa6f5382ca5580 318 | // <----> 319 | varint, bytesRead = btcleveldb.Varint128Read(xor, offset) // start after last varint 320 | offset += bytesRead 321 | varintDecoded = btcleveldb.Varint128Decode(varint) 322 | 323 | // Amount 324 | if fieldsSelected["amount"] { 325 | amount := btcleveldb.DecompressValue(varintDecoded) // int64 326 | output["amount"] = fmt.Sprintf("%d", amount) 327 | totalAmount += amount // add to stats 328 | } 329 | 330 | // Third Varint 331 | // ------------ 332 | // b98276a2ec7700cbc2986ff9aed6825920aece14aa6f5382ca5580 333 | // <> 334 | // 335 | // nSize - byte to indicate the type or size of script - helps with compression of the script data 336 | // - https://github.com/bitcoin/bitcoin/blob/master/src/compressor.cpp 337 | 338 | // 0 = P2PKH <- hash160 public key 339 | // 1 = P2SH <- hash160 script 340 | // 2 = P2PK 02publickey <- nsize makes up part of the public key in the actual script 341 | // 3 = P2PK 03publickey 342 | // 4 = P2PK 04publickey (uncompressed - but has been compressed in to leveldb) y=even 343 | // 5 = P2PK 04publickey (uncompressed - but has been compressed in to leveldb) y=odd 344 | // 6+ = [size of the upcoming script] (subtract 6 though to get the actual size in bytes, to account for the previous 5 script types already taken) 345 | varint, bytesRead = btcleveldb.Varint128Read(xor, offset) // start after last varint 346 | offset += bytesRead 347 | nsize := btcleveldb.Varint128Decode(varint) // 348 | output["nsize"] = fmt.Sprintf("%d", nsize) 349 | 350 | // Script (remaining bytes) 351 | // ------ 352 | // b98276a2ec7700cbc2986ff9aed6825920aece14aa6f5382ca5580 353 | // <--------------------------------------> 354 | 355 | // Move offset back a byte if script type is 2, 3, 4, or 5 (because this forms part of the P2PK public key along with the actual script) 356 | if nsize > 1 && nsize < 6 { // either 2, 3, 4, 5 357 | offset-- 358 | } 359 | 360 | // Get the remaining bytes 361 | script := xor[offset:] 362 | 363 | // Decompress the public keys from P2PK scripts that were uncompressed originally. They got compressed just for storage in the database. 364 | // Only decompress if the public key was uncompressed and 365 | // * Script field is selected or 366 | // * Address field is selected and p2pk addresses are enabled. 367 | if (nsize == 4 || nsize == 5) && (fieldsSelected["script"] || (fieldsSelected["address"] && *p2pkaddresses)) { 368 | script = keys.DecompressPublicKey(script) 369 | } 370 | 371 | if fieldsSelected["script"] { 372 | output["script"] = hex.EncodeToString(script) 373 | } 374 | 375 | // Addresses - Get address from script (if possible), and set script type (P2PK, P2PKH, P2SH, P2MS, P2WPKH, P2WSH or P2TR) 376 | // --------- 377 | if fieldsSelected["address"] || fieldsSelected["type"] { 378 | 379 | var address string // initialize address variable 380 | var scriptType string = "non-standard" // initialize script type 381 | 382 | switch { 383 | 384 | // P2PKH 385 | case nsize == 0: 386 | if fieldsSelected["address"] { // only work out addresses if they're wanted 387 | if testnet == true { 388 | address = keys.Hash160ToAddress(script, []byte{0x6f}) // (m/n)address - testnet addresses have a special prefix 389 | } else { 390 | address = keys.Hash160ToAddress(script, []byte{0x00}) // 1address 391 | } 392 | } 393 | scriptType = "p2pkh" 394 | scriptTypeCount["p2pkh"] += 1 395 | 396 | // P2SH 397 | case nsize == 1: 398 | if fieldsSelected["address"] { // only work out addresses if they're wanted 399 | if testnet == true { 400 | address = keys.Hash160ToAddress(script, []byte{0xc4}) // 2address - testnet addresses have a special prefix 401 | } else { 402 | address = keys.Hash160ToAddress(script, []byte{0x05}) // 3address 403 | } 404 | } 405 | scriptType = "p2sh" 406 | scriptTypeCount["p2sh"] += 1 407 | 408 | // P2PK 409 | case 1 < nsize && nsize < 6: // 2, 3, 4, 5 410 | // 2 = P2PK 02publickey <- nsize makes up part of the public key in the actual script (e.g. 02publickey) 411 | // 3 = P2PK 03publickey <- y is odd/even (0x02 = even, 0x03 = odd) 412 | // 4 = P2PK 04publickey (uncompressed) y = odd <- actual script uses an uncompressed public key, but it is compressed when stored in this db 413 | // 5 = P2PK 04publickey (uncompressed) y = even 414 | 415 | // "The uncompressed pubkeys are compressed when they are added to the db. 0x04 and 0x05 are used to indicate that the key is supposed to be uncompressed and those indicate whether the y value is even or odd so that the full uncompressed key can be retrieved." 416 | // 417 | // if nsize is 4 or 5, you will need to uncompress the public key to get it's full form 418 | // if nsize == 4 || nsize == 5 { 419 | // // uncompress (4 = y is even, 5 = y is odd) 420 | // script = decompress(script) 421 | // } 422 | 423 | scriptType = "p2pk" 424 | scriptTypeCount["p2pk"] += 1 425 | 426 | if fieldsSelected["address"] { // only work out addresses if they're wanted 427 | if *p2pkaddresses { // if we want to convert public keys in P2PK scripts to their corresponding addresses (even though they technically don't have addresses) 428 | 429 | // NOTE: These have already been decompressed. They were decompressed when the script data was first encountered. 430 | // Decompress if starts with 0x04 or 0x05 431 | // if (nsize == 4) || (nsize == 5) { 432 | // script = keys.DecompressPublicKey(script) 433 | // } 434 | 435 | if testnet == true { 436 | address = keys.PublicKeyToAddress(script, []byte{0x6f}) // (m/n)address - testnet addresses have a special prefix 437 | } else { 438 | address = keys.PublicKeyToAddress(script, []byte{0x00}) // 1address 439 | } 440 | } 441 | } 442 | 443 | // P2WPKH 444 | case nsize == 28 && script[0] == 0 && script[1] == 20: // P2WPKH (script type is 28, which means length of script is 22 bytes) 445 | // 315,c016e8dcc608c638196ca97572e04c6c52ccb03a35824185572fe50215b80000,0,551005,3118,0,28,001427dab16cca30628d395ccd2ae417dc1fe8dfa03e 446 | // script = 0014700d1635c4399d35061c1dabcc4632c30fedadd6 447 | // script = [0 20 112 13 22 53 196 57 157 53 6 28 29 171 204 70 50 195 15 237 173 214] 448 | // version = [0] 449 | // program = [112 13 22 53 196 57 157 53 6 28 29 171 204 70 50 195 15 237 173 214] 450 | version := script[0] 451 | program := script[2:] 452 | 453 | // bech32 function takes an int array and not a byte array, so convert the array to integers 454 | var programint []int // initialize empty integer array to hold the new one 455 | for _, v := range program { 456 | programint = append(programint, int(v)) // cast every value to an int 457 | } 458 | 459 | if fieldsSelected["address"] { // only work out addresses if they're wanted 460 | if testnet == true { 461 | address, _ = bech32.SegwitAddrEncode("tb", int(version), programint) // hrp (string), version (int), program ([]int) 462 | } else { 463 | address, _ = bech32.SegwitAddrEncode("bc", int(version), programint) // hrp (string), version (int), program ([]int) 464 | } 465 | } 466 | 467 | scriptType = "p2wpkh" 468 | scriptTypeCount["p2wpkh"] += 1 469 | 470 | // P2WSH 471 | case nsize == 40 && script[0] == 0 && script[1] == 32: // P2WSH (script type is 40, which means length of script is 34 bytes; 0x00 means segwit v0) 472 | // 956,1df27448422019c12c38d21c81df5c98c32c19cf7a312e612f78bebf4df20000,1,561890,800000,0,40,00200e7a15ba23949d9c274a1d9f6c9597fa9754fc5b5d7d45fc4369eeb4935c9bfe 473 | version := script[0] 474 | program := script[2:] 475 | 476 | var programint []int 477 | for _, v := range program { 478 | programint = append(programint, int(v)) // cast every value to an int 479 | } 480 | 481 | if fieldsSelected["address"] { // only work out addresses if they're wanted 482 | if testnet == true { 483 | address, _ = bech32.SegwitAddrEncode("tb", int(version), programint) // testnet bech32 addresses start with tb 484 | } else { 485 | address, _ = bech32.SegwitAddrEncode("bc", int(version), programint) // mainnet bech32 addresses start with bc 486 | } 487 | } 488 | 489 | scriptType = "p2wsh" 490 | scriptTypeCount["p2wsh"] += 1 491 | 492 | // P2TR 493 | case nsize == 40 && script[0] == 0x51 && script[1] == 32: // P2TR (script type is 40, which means length of script is 34 bytes; 0x51 means segwit v1 = taproot) 494 | // 9608047,bbc2e707dbc68db35dbada9be9d9182e546ee9302dc0a5cdd1a8dc3390483620,0,709635,2003,0,40,5120ef69f6a605817bc88882f88cbfcc60962af933fe1ae24a61069fb60067045963 495 | version := 1 496 | program := script[2:] 497 | 498 | var programint []int 499 | for _, v := range program { 500 | programint = append(programint, int(v)) // cast every value to an int 501 | } 502 | 503 | if fieldsSelected["address"] { // only work out addresses if they're wanted 504 | if testnet == true { 505 | address, _ = bech32.SegwitAddrEncode("tb", version, programint) // testnet bech32 addresses start with tb 506 | } else { 507 | address, _ = bech32.SegwitAddrEncode("bc", version, programint) // mainnet bech32 addresses start with bc 508 | } 509 | } 510 | 511 | scriptType = "p2tr" 512 | scriptTypeCount["p2tr"] += 1 513 | 514 | // P2MS 515 | case len(script) >= 37 && script[len(script)-1] == 174: // if there is a script, it's at least 37 bytes in length (min size for a P2MS), and if the last opcode is OP_CHECKMULTISIG (174) (0xae) 516 | scriptType = "p2ms" 517 | scriptTypeCount["p2ms"] += 1 518 | 519 | // Non-Standard (if the script type hasn't been identified and set then it remains as an unknown "non-standard" script) 520 | default: 521 | scriptType = "non-standard" 522 | scriptTypeCount["non-standard"] += 1 523 | 524 | } // switch 525 | 526 | // add address and script type to results map 527 | output["address"] = address 528 | output["type"] = scriptType 529 | 530 | } // if fieldsSelected["address"] || fieldsSelected["type"] 531 | 532 | } // if field from the Value is needed (e.g. -f txid,vout,address) 533 | 534 | 535 | // ------- 536 | // Results 537 | // ------- 538 | 539 | // CSV Lines 540 | output["count"] = fmt.Sprintf("%d",i+1) // convert integer to string (e.g 1 to "1") 541 | csvline := "" // Build output line from given fields 542 | // [ ] string builder faster? 543 | for _, v := range strings.Split(*fields, ",") { 544 | csvline += output[v] 545 | csvline += "," 546 | } 547 | csvline = csvline[:len(csvline)-1] // remove trailing , 548 | 549 | // Print Results 550 | // ------------- 551 | if ! *quiet { 552 | if *verbose { // -v flag 553 | fmt.Println(csvline) // Print each line. 554 | // 1157.76user 176.47system 30:44.64elapsed 72%CPU (0avgtext+0avgdata 55332maxresident)k 555 | // 1110.76user 164.97system 29:17.17elapsed 72%CPU (0avgtext+0avgdata 55236maxresident)k (after using packages) 556 | } else { 557 | if (i > 0 && i % 100000 == 0) { 558 | fmt.Printf("%d utxos processed\n", i) // Show progress at intervals. 559 | } 560 | // 812.18user 16.94system 12:44.04elapsed 108%CPU (0avgtext+0avgdata 55272maxresident)k 561 | // 951.03user 27.91system 15:21.35elapsed 106%CPU (0avgtext+0avgdata 55896maxresident)k (after using packages) 562 | } 563 | } 564 | 565 | // Write to File 566 | // ------------- 567 | // Write to buffer (use bufio for faster writes) 568 | fmt.Fprintln(writer, csvline) 569 | 570 | // Increment Count 571 | i++ 572 | } 573 | } 574 | iter.Release() // Do not defer this, want to release iterator before closing database 575 | 576 | // Final Progress Report 577 | // --------------------- 578 | if ! *quiet { 579 | // fmt.Printf("%d utxos saved to: %s\n", i, *file) 580 | fmt.Println() 581 | fmt.Printf("Total UTXOs: %d\n", i) 582 | 583 | // Can only show total btc amount if we have requested to get the amount for each entry with the -f fields flag 584 | if fieldsSelected["amount"] { 585 | fmt.Printf("Total BTC: %.8f\n", float64(totalAmount) / float64(100000000)) // convert satoshis to BTC (float with 8 decimal places) 586 | } 587 | 588 | // Can only show script type stats if we have requested to get the script type for each entry with the -f fields flag 589 | if fieldsSelected["type"] { 590 | fmt.Println("Script Types:") 591 | for k, v := range scriptTypeCount { 592 | fmt.Printf(" %-12s %d\n", k, v) // %-12s = left-justify padding 593 | } 594 | } 595 | } 596 | 597 | } 598 | --------------------------------------------------------------------------------