├── .gitignore ├── .travis.yml ├── Cargo.toml ├── LICENSE ├── Readme.md ├── examples ├── 144 │ └── 0 └── 2016 │ ├── 0 │ ├── 20 │ └── 21 ├── src ├── bin │ └── main.rs ├── bitcoin │ ├── header.rs │ ├── mod.rs │ └── rpc.rs ├── client │ └── mod.rs ├── lib.rs ├── server │ └── mod.rs └── util │ ├── hex.rs │ └── mod.rs └── tests └── header.rs /.gitignore: -------------------------------------------------------------------------------- 1 | target 2 | .idea 3 | compressedheaders.iml 4 | Cargo.lock 5 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: rust 2 | rust: 3 | - stable 4 | - beta 5 | - nightly 6 | matrix: 7 | allow_failures: 8 | - rust: nightly 9 | 10 | -------------------------------------------------------------------------------- /Cargo.toml: -------------------------------------------------------------------------------- 1 | [package] 2 | name = "compressedheaders" 3 | version = "0.1.0" 4 | authors = ["Riccardo Casatta "] 5 | 6 | [dependencies] 7 | serde = "1.0.2" 8 | serde_json = "1.0.1" 9 | serde_derive = "1.0.2" 10 | hyper = "0.11.6" 11 | rust-crypto = "0.2" 12 | futures = "0.1.14" 13 | tokio-core = "0.1.8" 14 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2020 Riccardo Casatta 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /Readme.md: -------------------------------------------------------------------------------- 1 | 2 | # Compressed Bitcoin Headers 3 | 4 | The Bitcoin headers are probably the most condensed and important piece of data in the world, their demand is expected to grow. 5 | Compressed Bitcoin Headers provides the bitcoin headers in a compressed form, saving about 45% of space. 6 | 7 | 8 | ## How it works 9 | 10 | It works on chunks of 2016 headers (the difficulty adjustment period) by removing in all headers but the first, the previous header hash (32 bytes) and the difficulty (4 bytes). 11 | 12 | When processing the headers stream the client: 13 | 14 | * Compute the hash of the first header (which is fully transmitted) 15 | * Save the difficulty bits which are constant for the next 2015 blocks. 16 | * Use this data to rebuild the missing information of the second header of the stream. Then repeat for the next headers. 17 | 18 | Thus the headers byte stream is composed of one full header when `modulo(height,2016)==0` followed by 2015 headers stripped down of the previous hash and the difficulty bytes. 19 | 20 | ### Getting the headers 21 | 22 | To get the headers information it connects to the RPC of a bitcoin full node. To retrieve connection information (rpcuser, rpcpassword and optionally rpchost) it scans the local machine for the bitcoin.conf file, looking in the following default paths: 23 | 24 | * $HOME/Library/Application Support/Bitcoin/bitcoin.conf 25 | * $HOME/.bitcoin/bitcoin.conf 26 | * $HOME\AppData\Roaming\Bitcoin\bitcoin.conf 27 | 28 | As of October 2017 it takes about 20 minutes to sync, then it stay on sync by asking the node for new headers every minute. 29 | 30 | ### Serving the compressed headers 31 | 32 | To serve the headers the software starts an HTTP server and answer HTTP Range Request at the endpoint: _http://localhost:3000/bitcoin-headers_ 33 | 34 | * _GET_ will return no data and an `Accept-Ranges: bytes` header 35 | * _HEAD_ request will return the content length of the stream 36 | * _GET_ with a header param `range: bytes=0-` will return the stream from the beginning to the end, `range: bytes=20000000-` will return from byte 20000000 to the end 37 | 38 | The content type is `application/octet-stream` 39 | The served headers are up to the connected node height less 6 to statistically avoid serving headers which could be reorged. 40 | 41 | #### Public testing endpoint 42 | 43 | https://finney.calendar.eternitywall.com/bitcoin-headers is a public available endpoint, you can donwload the header stream with: 44 | 45 | ``` 46 | curl -v -X GET -H "range: bytes=0-" https://finney.calendar.eternitywall.com/bitcoin-headers >headers 47 | ``` 48 | 49 | ## Building & launching 50 | 51 | You need [rust](https://www.rust-lang.org/it-IT/) 52 | 53 | ``` 54 | git clone https://github.com/RCasatta/rust-bitcoin-headers 55 | cd rust-bitcoin-headers 56 | cargo build --release 57 | ./target/release/rust-bitcoin-headers 58 | 59 | ``` 60 | 61 | The output will be something like this: 62 | ``` 63 | Found config file at /root/.bitcoin/bitcoin.conf 64 | server starting at 127.0.0.1:3000 65 | V4(127.0.0.1:3000) 66 | Block #0 with hash 000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f elapsed 0 seconds 67 | Block #1000 with hash 00000000c937983704a73af28acdec37b049d214adbda81d7e2a3dd146f6ed09 elapsed 1 seconds 68 | Block #1430 with hash 000000000009606d829b157912edb060c406b519fb2bfcc1078c196b69c67e49 is the min! 69 | Block #2000 with hash 00000000dfd5d65c9d8561b4b8f60a63018fe3933ecb131fb37f905f87da951a elapsed 2 seconds 70 | ``` 71 | 72 | ## Thanks 73 | 74 | Thanks to Peter Todd and Gregory Maxwell 75 | 76 | 77 | -------------------------------------------------------------------------------- /examples/144/0: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RCasatta/compressedheaders/23ac72d7896bd4aaa092106fdb5d254f752a680d/examples/144/0 -------------------------------------------------------------------------------- /examples/2016/0: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RCasatta/compressedheaders/23ac72d7896bd4aaa092106fdb5d254f752a680d/examples/2016/0 -------------------------------------------------------------------------------- /examples/2016/20: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RCasatta/compressedheaders/23ac72d7896bd4aaa092106fdb5d254f752a680d/examples/2016/20 -------------------------------------------------------------------------------- /examples/2016/21: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RCasatta/compressedheaders/23ac72d7896bd4aaa092106fdb5d254f752a680d/examples/2016/21 -------------------------------------------------------------------------------- /src/bin/main.rs: -------------------------------------------------------------------------------- 1 | extern crate compressedheaders; 2 | extern crate crypto; 3 | extern crate futures; 4 | extern crate hyper; 5 | extern crate serde; 6 | extern crate serde_json; 7 | extern crate tokio_core; 8 | 9 | 10 | use std::thread; 11 | use std::sync::{Arc, Mutex}; 12 | use compressedheaders::{server, client}; 13 | use compressedheaders::bitcoin::Config; 14 | 15 | fn main() { 16 | let config = Config::read().unwrap(); 17 | 18 | let block_headers_bytes = Vec::new(); 19 | let block_headers_bytes_arc = Arc::new(Mutex::new(block_headers_bytes)); 20 | 21 | let block_headers_bytes_arc_1 = block_headers_bytes_arc.clone(); 22 | thread::spawn(move || { 23 | server::start(block_headers_bytes_arc_1); 24 | }); 25 | 26 | let block_headers_bytes_arc_2 = block_headers_bytes_arc.clone(); 27 | let c = thread::spawn(move || { 28 | client::start(block_headers_bytes_arc_2, &config); 29 | }); 30 | 31 | let _ = c.join(); 32 | } 33 | -------------------------------------------------------------------------------- /src/bitcoin/header.rs: -------------------------------------------------------------------------------- 1 | use bitcoin; 2 | use crypto::digest::Digest; 3 | use crypto::sha2::Sha256; 4 | use util::hex::{FromHex, ToHex}; 5 | use std::fmt; 6 | 7 | static GENESIS_RAW_HEX: &'static str = "0100000000000000000000000000000000000000\ 8 | 000000000000000000000000000000003ba3edfd\ 9 | 7a7b12b27ac72c3e67768f617fc81bc3888a5132\ 10 | 3a9fb8aa4b1e5e4a29ab5f49ffff001d1dac2b7c"; 11 | 12 | #[derive(Copy, Clone, Debug)] 13 | pub struct BlockHeader { 14 | // The protocol version. Should always be 1. 15 | pub version: [u8; 4], 16 | 17 | // Reference to the previous block in the chain 18 | pub prev_blockhash: [u8; 32], 19 | 20 | /// The root hash of the merkle tree of transactions in the block 21 | pub merkle_root: [u8; 32], 22 | 23 | // The timestamp of the block, as claimed by the mainer 24 | pub time: [u8; 4], 25 | 26 | // The target value below which the blockhash must lie, 27 | // encoded as a a float (with well-defined rounding, of course) 28 | pub bits: [u8; 4], 29 | 30 | // The nonce, selected to obtain a low enough blockhash 31 | pub nonce: [u8; 4], 32 | } 33 | 34 | impl fmt::Display for BlockHeader { 35 | fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { 36 | write!( 37 | f, 38 | "{}{}{}{}{}{}", 39 | self.version.to_hex(), 40 | self.prev_blockhash.to_hex(), 41 | self.merkle_root.to_hex(), 42 | self.time.to_hex(), 43 | self.bits.to_hex(), 44 | self.nonce.to_hex() 45 | ) 46 | } 47 | } 48 | 49 | 50 | impl BlockHeader { 51 | pub fn new() -> BlockHeader { 52 | BlockHeader { 53 | version: [0; 4], 54 | prev_blockhash: [0; 32], 55 | merkle_root: [0; 32], 56 | time: [0; 4], 57 | bits: [0; 4], 58 | nonce: [0; 4], 59 | } 60 | } 61 | 62 | pub fn genesis() -> BlockHeader { 63 | let mut genesis_raw_bytes: [u8; 80] = [0; 80]; 64 | let genesis_raw_vec = GENESIS_RAW_HEX.from_hex().unwrap(); 65 | genesis_raw_bytes.clone_from_slice(&genesis_raw_vec); 66 | let b = BlockHeader::from_bytes(genesis_raw_bytes); 67 | return b; 68 | } 69 | 70 | pub fn as_bytes(&self) -> [u8; 80] { 71 | let mut result: [u8; 80] = [0; 80]; 72 | let mut vec: Vec = Vec::new(); 73 | vec.extend_from_slice(&self.version); 74 | vec.extend_from_slice(&self.prev_blockhash); 75 | vec.extend_from_slice(&self.merkle_root); 76 | vec.extend_from_slice(&self.time); 77 | vec.extend_from_slice(&self.bits); 78 | vec.extend_from_slice(&self.nonce); 79 | for (idx, el) in vec.into_iter().enumerate() { 80 | result[idx] = el; 81 | } 82 | result 83 | } 84 | 85 | pub fn as_compressed_bytes(&self) -> [u8; 44] { 86 | let mut result: [u8; 44] = [0; 44]; 87 | let current = &self.as_bytes(); 88 | result[0..4].clone_from_slice(¤t[0..4]); 89 | result[4..40].clone_from_slice(¤t[36..72]); 90 | result[40..44].clone_from_slice(¤t[76..80]); 91 | result 92 | } 93 | 94 | pub fn from_bytes(bytes: [u8; 80]) -> BlockHeader { 95 | BlockHeader { 96 | version: clone_into_array(&bytes[0..4]), 97 | prev_blockhash: clone_into_array(&bytes[4..36]), 98 | merkle_root: clone_into_array(&bytes[36..68]), 99 | time: clone_into_array(&bytes[68..72]), 100 | bits: clone_into_array(&bytes[72..76]), 101 | nonce: clone_into_array(&bytes[76..80]), 102 | } 103 | } 104 | 105 | pub fn from_compressed_bytes( 106 | compressed_bytes: [u8; 44], 107 | prev_blockhash: [u8; 32], 108 | difficulty: [u8; 4], 109 | ) -> BlockHeader { 110 | let mut result: [u8; 80] = [0; 80]; 111 | result[0..4].clone_from_slice(&compressed_bytes[0..4]); 112 | result[4..36].clone_from_slice(&prev_blockhash); 113 | result[36..72].clone_from_slice(&compressed_bytes[4..40]); 114 | result[72..76].clone_from_slice(&difficulty); 115 | result[76..80].clone_from_slice(&compressed_bytes[40..44]); 116 | 117 | BlockHeader::from_bytes(result) 118 | } 119 | 120 | pub fn from_block_header_rpc(block_header_rpc: bitcoin::rpc::BlockHeaderRpc) -> BlockHeader { 121 | //let nextblockhash = q["nextblockhash"].as_str().unwrap(); 122 | let version_hex = &block_header_rpc.versionHex; 123 | let previous_block_hash = match block_header_rpc.previousblockhash { 124 | Some(r) => r, 125 | _ => String::from("0000000000000000000000000000000000000000000000000000000000000000"), 126 | }; 127 | let merkle_root = &block_header_rpc.merkleroot; 128 | let time = block_header_rpc.time; 129 | let bits = &block_header_rpc.bits; 130 | let nonce = block_header_rpc.nonce; 131 | 132 | BlockHeader { 133 | version: to_reversed_array_of_4(version_hex.from_hex().unwrap()), 134 | prev_blockhash: to_reversed_array_of_32(previous_block_hash.from_hex().unwrap()), 135 | merkle_root: to_reversed_array_of_32(merkle_root.from_hex().unwrap()), 136 | time: transform_u32_to_reversed_array_of_u8(time), 137 | bits: to_reversed_array_of_4(bits.from_hex().unwrap()), 138 | nonce: transform_u32_to_reversed_array_of_u8(nonce), 139 | } 140 | } 141 | 142 | pub fn hash_be(&self) -> [u8; 32] { 143 | let mut hash = self.hash(); 144 | hash.reverse(); 145 | hash 146 | } 147 | pub fn hash(&self) -> [u8; 32] { 148 | let mut sha2 = Sha256::new(); 149 | sha2.input(&self.as_bytes()); 150 | let mut first: [u8; 32] = [0; 32]; 151 | sha2.result(&mut first); 152 | let mut sha2b = Sha256::new(); 153 | sha2b.input(&first); 154 | 155 | //TODO FIX nonsense passing from string 156 | let bytes: Vec = (&sha2b.result_str()).from_hex().unwrap(); 157 | 158 | let mut result: [u8; 32] = [0; 32]; 159 | result.clone_from_slice(&bytes); 160 | 161 | result 162 | } 163 | } 164 | 165 | 166 | fn transform_u32_to_reversed_array_of_u8(x: u32) -> [u8; 4] { 167 | let b1: u8 = ((x >> 24) & 0xff) as u8; 168 | let b2: u8 = ((x >> 16) & 0xff) as u8; 169 | let b3: u8 = ((x >> 8) & 0xff) as u8; 170 | let b4: u8 = (x & 0xff) as u8; 171 | return [b4, b3, b2, b1]; 172 | } 173 | 174 | fn clone_into_array(slice: &[T]) -> A 175 | where 176 | A: Sized + Default + AsMut<[T]>, 177 | T: Clone, 178 | { 179 | let mut a = Default::default(); 180 | >::as_mut(&mut a).clone_from_slice(slice); 181 | a 182 | } 183 | 184 | fn to_reversed_array_of_4(mut vec: Vec) -> [u8; 4] { 185 | let mut result: [u8; 4] = [0; 4]; 186 | vec.reverse(); 187 | result.clone_from_slice(&vec); 188 | result 189 | } 190 | 191 | fn to_reversed_array_of_32(mut vec: Vec) -> [u8; 32] { 192 | let mut result: [u8; 32] = [0; 32]; 193 | vec.reverse(); 194 | result.clone_from_slice(&vec); 195 | result 196 | } 197 | 198 | #[cfg(test)] 199 | mod tests { 200 | 201 | use bitcoin::header::BlockHeader; 202 | use util::hex::{FromHex, ToHex}; 203 | use bitcoin::header::GENESIS_RAW_HEX; 204 | 205 | #[test] 206 | pub fn test_genesis() { 207 | let g = BlockHeader::genesis(); 208 | let h = g.hash_be().to_hex(); 209 | assert_eq!( 210 | h, 211 | "000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f" 212 | ); 213 | } 214 | 215 | #[test] 216 | pub fn test_as_compressed_bytes() { 217 | let g = BlockHeader::genesis(); 218 | let b = g.as_compressed_bytes(); 219 | let genesis_compressed = "010000003ba3edfd7a7b12b27ac72c3e67768f617fc8\ 220 | 1bc3888a51323a9fb8aa4b1e5e4a29ab5f491dac2b7c"; 221 | assert_eq!(b.to_hex(), genesis_compressed); 222 | } 223 | 224 | #[test] 225 | pub fn test_block_header_from_hex_to_hex() { 226 | let mut genesis_raw_bytes: [u8; 80] = [0; 80]; 227 | let genesis_raw_vec = GENESIS_RAW_HEX.from_hex().unwrap(); 228 | genesis_raw_bytes.clone_from_slice(&genesis_raw_vec); 229 | let b = BlockHeader::from_bytes(genesis_raw_bytes); 230 | assert_eq!(GENESIS_RAW_HEX, b.as_bytes().to_hex()); 231 | assert_eq!( 232 | "000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f", 233 | b.hash_be().to_hex() 234 | ); 235 | } 236 | 237 | #[test] 238 | pub fn test_block_headers_reconstruct() { 239 | let test_data_144 = include_bytes!("../../examples/144/0").to_vec(); 240 | let zeroes = "0000000000000000000000000000000000000000000000000000000000000000"; 241 | let genesis_block = "6fe28c0ab6f1b372c1a6a246ae63f74f931e8365e15a089c68d6190000000000"; 242 | let block_42335 = "e709fcacfe11464204e4cc1daf4a7b63df72a742a59f4f3eef96843000000000"; 243 | let block_143 = "61188712afd4785d18ef15db57fb52dd150b56c8b547fc6bbf23ec4900000000"; 244 | 245 | test_block_header_reconstruct( 246 | test_data_144, 247 | 144, 248 | String::from(zeroes), 249 | String::from(genesis_block), 250 | String::from(block_143), 251 | ); 252 | 253 | let test_data_2016 = include_bytes!("../../examples/2016/0").to_vec(); 254 | let block_2015 = "6397bb6abd4fc521c0d3f6071b5650389f0b4551bc40b4e6b067306900000000"; 255 | test_block_header_reconstruct( 256 | test_data_2016, 257 | 2016, 258 | String::from(zeroes), 259 | String::from(genesis_block), 260 | String::from(block_2015), 261 | ); 262 | 263 | let test_data_2016_20 = include_bytes!("../../examples/2016/20").to_vec(); 264 | let block_40320 = "45720d24eae33ade0d10397a2e02989edef834701b965a9b161e864500000000"; 265 | let block_40319 = "1a231097b6ab6279c80f24674a2c8ee5b9a848e1d45715ad89b6358100000000"; 266 | test_block_header_reconstruct( 267 | test_data_2016_20, 268 | 2016, 269 | String::from(block_40319), 270 | String::from(block_40320), 271 | String::from(block_42335), 272 | ); 273 | 274 | let test_data_2016_21 = include_bytes!("../../examples/2016/21").to_vec(); 275 | let block_42336 = "1296ba2f0a66e421d7f51c4596c2ce0820903f3d81a953173778000b00000000"; 276 | let block_44351 = "d55e1b468c22798971272037d6cc04fdac73913c0012d0d7630c2e1a00000000"; 277 | test_block_header_reconstruct( 278 | test_data_2016_21, 279 | 2016, 280 | String::from(block_42335), 281 | String::from(block_42336), 282 | String::from(block_44351), 283 | ); 284 | 285 | //TODO add test continuity between chunk (prev first hash equal last previous chunk) 286 | } 287 | 288 | pub fn test_block_header_reconstruct( 289 | test_data: Vec, 290 | chunk_size: u32, 291 | first_prev_hash: String, 292 | first_hash_verify: String, 293 | last_hash_verify: String, 294 | ) { 295 | let mut first: [u8; 80] = [0; 80]; 296 | first.clone_from_slice(&test_data[0..80]); 297 | let first_as_block: BlockHeader = BlockHeader::from_bytes(first); 298 | let first_hash = first_as_block.hash(); 299 | assert_eq!(first_hash.to_hex(), first_hash_verify); 300 | assert_eq!(first_as_block.prev_blockhash.to_hex(), first_prev_hash); 301 | let mut prev_hash = first_hash; 302 | let prev_diff = first_as_block.bits; 303 | for i in 0..chunk_size - 1 { 304 | let mut compressed_block_bytes: [u8; 44] = [0; 44]; 305 | let start = (i * 44 + 80) as usize; 306 | let end = (start + 44) as usize; 307 | compressed_block_bytes.clone_from_slice(&test_data[start..end]); 308 | let current_as_block: BlockHeader = 309 | BlockHeader::from_compressed_bytes(compressed_block_bytes, prev_hash, prev_diff); 310 | let current_hash = current_as_block.hash(); 311 | prev_hash = current_hash; 312 | if i == chunk_size - 2 { 313 | assert_eq!(current_hash.to_hex(), last_hash_verify); 314 | } 315 | } 316 | } 317 | 318 | } 319 | -------------------------------------------------------------------------------- /src/bitcoin/mod.rs: -------------------------------------------------------------------------------- 1 | pub mod rpc; 2 | pub mod header; 3 | 4 | use std::env; 5 | use std::fs::File; 6 | use std::io::Read; 7 | 8 | 9 | pub struct Config { 10 | host : String, 11 | username : String, 12 | password : Option, 13 | } 14 | 15 | impl Config { 16 | 17 | fn new (host: String, username: String, password: Option) -> Config { 18 | Config { 19 | host, 20 | username, 21 | password, 22 | } 23 | } 24 | pub fn read() -> Result { 25 | let mut host: Option = Some(String::from("http://localhost:8332")); 26 | let mut username: Option = None; 27 | let mut password: Option = None; 28 | 29 | match env::home_dir() { 30 | Some(path) => { 31 | let paths = vec![ 32 | "/Library/Application Support/Bitcoin/bitcoin.conf", 33 | "/.bitcoin/bitcoin.conf", 34 | "\\AppData\\Roaming\\Bitcoin\\bitcoin.conf", 35 | ]; 36 | for filename in paths { 37 | let full_path = format!("{}{}", path.display(), filename); 38 | let f = File::open(full_path.clone()); 39 | match f { 40 | Ok(mut f) => { 41 | let mut contents = String::new(); 42 | f.read_to_string(&mut contents) 43 | .expect("something went wrong reading the file"); 44 | println!("Found config file at {}", full_path); 45 | let x = contents.split("\n"); 46 | for el in x { 47 | let x = el.replace(" ", ""); 48 | if x.starts_with("rpcuser=") { 49 | username = Some(String::from(&x[8..])); 50 | } 51 | if x.starts_with("rpcpassword=") { 52 | password = Some(String::from(&x[12..])); 53 | } 54 | if x.starts_with("rpchost=") { 55 | host = Some(String::from(&x[8..])); 56 | } 57 | } 58 | } 59 | Err(_) => (), 60 | } 61 | } 62 | } 63 | None => println!("Impossible to get your home dir!"), 64 | } 65 | 66 | match host.is_some() && username.is_some() { 67 | true => Ok( Config::new(host.unwrap(), username.unwrap(), password)), 68 | false => Err("Cannot find rpcuser and rpcpassword"), 69 | } 70 | } 71 | } 72 | 73 | -------------------------------------------------------------------------------- /src/bitcoin/rpc.rs: -------------------------------------------------------------------------------- 1 | use hyper::{Body, Client, Error, Method, Request}; 2 | use hyper::header::{Authorization, Basic}; 3 | use tokio_core::reactor::Core; 4 | use futures::{Future, Stream}; 5 | use serde_json; 6 | use std::str; 7 | use bitcoin::Config; 8 | 9 | #[derive(Serialize, Deserialize, Debug)] 10 | pub struct BlockHeaderRpcResponse { 11 | pub result: BlockHeaderRpc, 12 | pub id: String, 13 | pub error: Option, 14 | } 15 | 16 | #[allow(non_snake_case)] 17 | #[derive(Serialize, Deserialize, Debug)] 18 | pub struct BlockHeaderRpc { 19 | pub hash: String, 20 | pub height: u32, 21 | pub version: u32, 22 | pub nonce: u32, 23 | pub versionHex: String, 24 | pub merkleroot: String, 25 | pub time: u32, 26 | pub mediantime: u32, 27 | pub bits: String, 28 | pub difficulty: f64, 29 | pub chainwork: String, 30 | pub nextblockhash: Option, 31 | pub previousblockhash: Option, 32 | } 33 | 34 | pub fn get_block_header( 35 | block_hash: String, 36 | config: &Config, 37 | ) -> Result { 38 | let auth = Authorization(Basic { 39 | username: config.username.clone(), 40 | password: config.password.clone(), 41 | }); 42 | let mut core = Core::new()?; 43 | let client = Client::new(&core.handle()); 44 | let request_body_string: String = format!( 45 | "{{\"jsonrpc\":\"1.0\",\"id\":\"{}\",\"method\":\"{}\",\"params\":[\"{}\"]}}", 46 | 0, 47 | "getblockheader", 48 | block_hash 49 | ); 50 | let mut req: Request = Request::new(Method::Post, config.host.parse().unwrap()); 51 | req.set_body(Body::from(request_body_string)); 52 | req.headers_mut().set(auth); 53 | 54 | let future_res = client.request(req); 55 | 56 | let work = future_res.and_then(|res| { 57 | //println!("Response: {}", res.status()); 58 | // read into a String, so that you don't need to do the conversion. 59 | res.body().concat2() 60 | }); 61 | 62 | let work_result = core.run(work)?; //this throw on mac 63 | 64 | //println!("work_result {:?}", work_result); 65 | let utf8 = str::from_utf8(&work_result)?; 66 | 67 | //println!("GET: {}", utf8); 68 | let block_header_rpc_response: BlockHeaderRpcResponse = match serde_json::from_str(utf8) { 69 | Err(e) => return Err(Error::Io(e.into())), 70 | Ok(f) => f, 71 | }; 72 | 73 | //let block_header_rpc_response = serde_json::from_str(utf8)?; 74 | 75 | Ok(block_header_rpc_response) 76 | } 77 | -------------------------------------------------------------------------------- /src/client/mod.rs: -------------------------------------------------------------------------------- 1 | use std::sync::{Arc, Mutex}; 2 | use std::thread; 3 | use std::time::{Duration, Instant}; 4 | use util::hex::ToHex; 5 | use bitcoin::header::BlockHeader; 6 | use bitcoin; 7 | use std::collections::HashMap; 8 | use bitcoin::Config; 9 | 10 | pub fn start( 11 | block_headers_bytes: Arc>>, 12 | config: &Config, 13 | ) { 14 | let start = Instant::now(); 15 | 16 | let genesis_block_hash = 17 | String::from("000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f"); 18 | let mut block_hash: String = genesis_block_hash.clone(); 19 | let mut last_block: usize = 0; 20 | let mut min_block_hash: String = genesis_block_hash; 21 | 22 | let mut synced_height: usize = 0; 23 | let mut block_headers_map = HashMap::new(); 24 | 25 | loop { 26 | let r = bitcoin::rpc::get_block_header( 27 | block_hash.clone(), 28 | &config, 29 | ); 30 | match r { 31 | Ok(block_header_rpc_response) => { 32 | let block_header_rpc: bitcoin::rpc::BlockHeaderRpc = 33 | block_header_rpc_response.result; 34 | let height = block_header_rpc.height.clone() as usize; 35 | if last_block == 0 && height % 1000 == 0 { 36 | println!( 37 | "Block #{} with hash {} elapsed {} seconds", 38 | height, 39 | block_hash, 40 | start.elapsed().as_secs() 41 | ); 42 | synced_height = sync( 43 | &mut block_headers_map, 44 | &block_headers_bytes, 45 | height, 46 | synced_height, 47 | ); 48 | } 49 | 50 | let block_hash_option = block_header_rpc.nextblockhash.clone(); 51 | let sleep = match block_hash_option { 52 | Some(val) => { 53 | block_hash = val; 54 | let block_header = BlockHeader::from_block_header_rpc(block_header_rpc); 55 | 56 | block_headers_map.insert(height, block_header); 57 | let hash_hex = block_header.hash_be().to_hex(); 58 | if min_block_hash > hash_hex { 59 | min_block_hash = hash_hex; 60 | println!("Block #{} with hash {} is the min!", height, min_block_hash); 61 | } 62 | 63 | false 64 | } 65 | None => { 66 | if height != last_block { 67 | synced_height = sync( 68 | &mut block_headers_map, 69 | &block_headers_bytes, 70 | height, 71 | synced_height, 72 | ); 73 | println!( 74 | "Block #{} with hash {} synced_height {}", 75 | height, 76 | block_hash, 77 | synced_height 78 | ); 79 | } 80 | last_block = height; 81 | block_hash = block_headers_map 82 | .get(&(height - 6)) 83 | .unwrap() 84 | .hash_be() 85 | .to_hex(); //going back 6 blocks to support reorgs 86 | 87 | true 88 | } 89 | }; 90 | 91 | if sleep { 92 | thread::sleep(Duration::from_secs(60)); 93 | } 94 | } 95 | Err(e) => { 96 | println!("{:?} with hash {}", e, block_hash); 97 | thread::sleep(Duration::from_secs(10)); 98 | } 99 | } 100 | } 101 | } 102 | 103 | fn sync( 104 | block_headers_map: &mut HashMap, 105 | block_headers_bytes: &Arc>>, 106 | height: usize, 107 | synced_height: usize, 108 | ) -> usize { 109 | let sync_to_option = height.checked_sub(6); 110 | match sync_to_option { 111 | Some(sync_to) => { 112 | let mut block_headers_bytes_lock = block_headers_bytes.lock().unwrap(); 113 | for i in synced_height..sync_to { 114 | match i % 2016 { 115 | 0 => block_headers_bytes_lock 116 | .extend(block_headers_map.remove(&i).unwrap().as_bytes().into_iter()), 117 | _ => block_headers_bytes_lock.extend( 118 | block_headers_map 119 | .remove(&i) 120 | .unwrap() 121 | .as_compressed_bytes() 122 | .into_iter(), 123 | ), 124 | } 125 | } 126 | sync_to 127 | } 128 | None => synced_height, 129 | } 130 | } 131 | -------------------------------------------------------------------------------- /src/lib.rs: -------------------------------------------------------------------------------- 1 | extern crate crypto; 2 | extern crate futures; 3 | extern crate hyper; 4 | extern crate serde; 5 | extern crate serde_json; 6 | extern crate tokio_core; 7 | 8 | #[macro_use] 9 | extern crate serde_derive; 10 | 11 | pub mod bitcoin; 12 | pub mod server; 13 | pub mod client; 14 | pub mod util; 15 | -------------------------------------------------------------------------------- /src/server/mod.rs: -------------------------------------------------------------------------------- 1 | use std::sync::{Arc, Mutex}; 2 | use hyper; 3 | use futures; 4 | use hyper::header::{AcceptRanges, ByteRangeSpec, ContentLength, ContentType, Headers, Range, 5 | RangeUnit}; 6 | use hyper::server::{Http, Request, Response, Service}; 7 | use std::net::SocketAddr; 8 | use hyper::StatusCode; 9 | 10 | #[derive(Clone)] 11 | struct HeaderServices { 12 | block_headers_bytes: Arc>>, 13 | } 14 | 15 | pub fn start(block_headers_bytes: Arc>>) { 16 | let x = "0.0.0.0:3000"; 17 | println!("server starting at http://{}", x); 18 | let addr: SocketAddr = x.parse().unwrap(); 19 | let server = Http::new() 20 | .bind(&addr, move || { 21 | Ok(HeaderServices { 22 | block_headers_bytes: block_headers_bytes.clone(), 23 | }) 24 | }) 25 | .unwrap(); 26 | server.run().unwrap(); 27 | } 28 | 29 | impl Service for HeaderServices { 30 | type Request = Request; 31 | type Response = Response; 32 | type Error = hyper::Error; 33 | type Future = futures::future::FutureResult; 34 | 35 | fn call(&self, _req: Request) -> Self::Future { 36 | let response = match validate_req(_req) { 37 | Err(e) => Response::new().with_status(e), 38 | Ok(r) => build_range_response(self.block_headers_bytes.clone(), r), 39 | }; 40 | futures::future::ok(response) 41 | } 42 | } 43 | 44 | fn validate_req(_req: Request) -> Result, StatusCode> { 45 | let uri_path = _req.uri().path(); 46 | 47 | match uri_path.eq("/bitcoin-headers") { 48 | true => match _req.headers().get::() { 49 | Some(r) => Ok(Some(r.clone())), 50 | None => Ok(None), 51 | }, 52 | false => Err(StatusCode::NotFound), 53 | } 54 | } 55 | 56 | 57 | fn build_range_response( 58 | block_headers_bytes_arc: Arc>>, 59 | range: Option, 60 | ) -> Response { 61 | let block_headers_bytes = block_headers_bytes_arc.lock().unwrap(); 62 | match range { 63 | Some(range) => match range { 64 | Range::Bytes(r) => { 65 | let (start, end) = match r[0] { 66 | ByteRangeSpec::AllFrom(start) => (start as usize, block_headers_bytes.len()), 67 | ByteRangeSpec::FromTo(start, end) => (start as usize, end as usize), 68 | ByteRangeSpec::Last(x) => { 69 | let end = block_headers_bytes.len(); 70 | (end - (x as usize), end) 71 | } 72 | }; 73 | println!("Range request {}-{}", start, end); 74 | 75 | let mut reply = Vec::with_capacity(end - start); 76 | reply.extend(&block_headers_bytes[start..end]); 77 | 78 | Response::new() 79 | .with_header(ContentType::octet_stream()) 80 | .with_header(ContentLength(reply.len() as u64)) 81 | .with_body(reply) 82 | } 83 | Range::Unregistered(_, _) => Response::new().with_status(StatusCode::NotFound), 84 | }, 85 | None => { 86 | let mut headers = Headers::new(); 87 | headers.set(AcceptRanges(vec![RangeUnit::Bytes])); 88 | headers.set(ContentLength(block_headers_bytes.len() as u64)); 89 | 90 | Response::new() 91 | .with_headers(headers) 92 | .with_status(StatusCode::Ok) 93 | } 94 | } 95 | } 96 | -------------------------------------------------------------------------------- /src/util/hex.rs: -------------------------------------------------------------------------------- 1 | // Copyright 2013-2014 The Rust Project Developers. See the COPYRIGHT 2 | // file at the top-level directory of this distribution and at 3 | // http://rust-lang.org/COPYRIGHT. 4 | // 5 | // Licensed under the Apache License, Version 2.0 or the MIT license 7 | // , at your 8 | // option. This file may not be copied, modified, or distributed 9 | // except according to those terms. 10 | // 11 | // ignore-lexer-test FIXME #15679 12 | 13 | //! Hex binary-to-text encoding 14 | 15 | pub use self::FromHexError::*; 16 | 17 | use std::fmt; 18 | use std::error; 19 | 20 | /// A trait for converting a value to hexadecimal encoding 21 | pub trait ToHex { 22 | /// Converts the value of `self` to a hex value, returning the owned 23 | /// string. 24 | fn to_hex(&self) -> String; 25 | } 26 | 27 | static CHARS: &'static [u8] = b"0123456789abcdef"; 28 | 29 | impl ToHex for [u8] { 30 | /// Turn a vector of `u8` bytes into a hexadecimal string. 31 | 32 | fn to_hex(&self) -> String { 33 | let mut v = Vec::with_capacity(self.len() * 2); 34 | for &byte in self.iter() { 35 | v.push(CHARS[(byte >> 4) as usize]); 36 | v.push(CHARS[(byte & 0xf) as usize]); 37 | } 38 | 39 | unsafe { String::from_utf8_unchecked(v) } 40 | } 41 | } 42 | 43 | impl<'a, T: ?Sized + ToHex> ToHex for &'a T { 44 | fn to_hex(&self) -> String { 45 | (**self).to_hex() 46 | } 47 | } 48 | 49 | /// A trait for converting hexadecimal encoded values 50 | pub trait FromHex { 51 | /// Converts the value of `self`, interpreted as hexadecimal encoded data, 52 | /// into an owned vector of bytes, returning the vector. 53 | fn from_hex(&self) -> Result, FromHexError>; 54 | } 55 | 56 | /// Errors that can occur when decoding a hex encoded string 57 | #[derive(Clone, Copy)] 58 | pub enum FromHexError { 59 | /// The input contained a character not part of the hex format 60 | InvalidHexCharacter(char, usize), 61 | /// The input had an invalid length 62 | InvalidHexLength, 63 | } 64 | 65 | impl fmt::Debug for FromHexError { 66 | fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { 67 | match *self { 68 | InvalidHexCharacter(ch, idx) => { 69 | write!(f, "Invalid character '{}' at position {}", ch, idx) 70 | } 71 | InvalidHexLength => write!(f, "Invalid input length"), 72 | } 73 | } 74 | } 75 | 76 | impl error::Error for FromHexError { 77 | fn description(&self) -> &str { 78 | match *self { 79 | InvalidHexCharacter(_, _) => "invalid character", 80 | InvalidHexLength => "invalid length", 81 | } 82 | } 83 | } 84 | 85 | impl fmt::Display for FromHexError { 86 | fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { 87 | fmt::Debug::fmt(&self, f) 88 | } 89 | } 90 | 91 | impl FromHex for str { 92 | /// Convert any hexadecimal encoded string (literal, `@`, `&`, or `~`) 93 | /// to the byte values it encodes. 94 | /// 95 | /// You can use the `String::from_utf8` function to turn a 96 | /// `Vec` into a string with characters corresponding to those values. 97 | /// 98 | /// ``` 99 | fn from_hex(&self) -> Result, FromHexError> { 100 | // This may be an overestimate if there is any whitespace 101 | let mut b = Vec::with_capacity(self.len() / 2); 102 | let mut modulus = 0; 103 | let mut buf = 0; 104 | 105 | for (idx, byte) in self.bytes().enumerate() { 106 | buf <<= 4; 107 | 108 | match byte { 109 | b'A'...b'F' => buf |= byte - b'A' + 10, 110 | b'a'...b'f' => buf |= byte - b'a' + 10, 111 | b'0'...b'9' => buf |= byte - b'0', 112 | b' ' | b'\r' | b'\n' | b'\t' => { 113 | buf >>= 4; 114 | continue; 115 | } 116 | _ => { 117 | let ch = self[idx..].chars().next().unwrap(); 118 | return Err(InvalidHexCharacter(ch, idx)); 119 | } 120 | } 121 | 122 | modulus += 1; 123 | if modulus == 2 { 124 | modulus = 0; 125 | b.push(buf); 126 | } 127 | } 128 | 129 | match modulus { 130 | 0 => Ok(b.into_iter().collect()), 131 | _ => Err(InvalidHexLength), 132 | } 133 | } 134 | } 135 | 136 | impl<'a, T: ?Sized + FromHex> FromHex for &'a T { 137 | fn from_hex(&self) -> Result, FromHexError> { 138 | (**self).from_hex() 139 | } 140 | } 141 | 142 | #[cfg(test)] 143 | mod tests { 144 | use util::hex::{FromHex, ToHex}; 145 | 146 | #[test] 147 | pub fn test_to_hex() { 148 | assert_eq!("foobar".as_bytes().to_hex(), "666f6f626172"); 149 | } 150 | 151 | #[test] 152 | pub fn test_from_hex_okay() { 153 | assert_eq!("666f6f626172".from_hex().unwrap(), b"foobar"); 154 | assert_eq!("666F6F626172".from_hex().unwrap(), b"foobar"); 155 | } 156 | 157 | #[test] 158 | pub fn test_from_hex_odd_len() { 159 | assert!("666".from_hex().is_err()); 160 | assert!("66 6".from_hex().is_err()); 161 | } 162 | 163 | #[test] 164 | pub fn test_from_hex_invalid_char() { 165 | assert!("66y6".from_hex().is_err()); 166 | } 167 | 168 | #[test] 169 | pub fn test_from_hex_ignores_whitespace() { 170 | assert_eq!("666f 6f6\r\n26172 ".from_hex().unwrap(), b"foobar"); 171 | } 172 | 173 | #[test] 174 | pub fn test_to_hex_all_bytes() { 175 | for i in 0..256 { 176 | assert_eq!([i as u8].to_hex(), format!("{:02x}", i)); 177 | } 178 | } 179 | 180 | #[test] 181 | pub fn test_from_hex_all_bytes() { 182 | for i in 0..256 { 183 | let ii: &[u8] = &[i as u8]; 184 | assert_eq!(format!("{:02x}", i).from_hex().unwrap(), ii); 185 | assert_eq!(format!("{:02X}", i).from_hex().unwrap(), ii); 186 | } 187 | } 188 | } 189 | -------------------------------------------------------------------------------- /src/util/mod.rs: -------------------------------------------------------------------------------- 1 | pub mod hex; 2 | -------------------------------------------------------------------------------- /tests/header.rs: -------------------------------------------------------------------------------- 1 | 2 | --------------------------------------------------------------------------------