├── .gitignore ├── HOWTO.md ├── INSTALL ├── LICENSE ├── README-IRC.md ├── README.md ├── configure ├── electrum-dash-server ├── electrum-dash.conf.sample ├── kill_session ├── run_electrum_dash_server ├── setup.py └── src ├── __init__.py ├── blockchain_processor.py ├── deserialize.py ├── ircthread.py ├── networks.py ├── processor.py ├── server_processor.py ├── storage.py ├── stratum_tcp.py ├── test ├── __init__.py └── test_utils.py ├── utils.py └── version.py /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | *.so 3 | *.o 4 | .gitignore 5 | electrum.conf 6 | -------------------------------------------------------------------------------- /HOWTO.md: -------------------------------------------------------------------------------- 1 | How to run your own Electrum Dash server 2 | ======================================== 3 | 4 | Abstract 5 | -------- 6 | 7 | This document is an easy to follow guide to installing and running your own 8 | Electrum Dash server on Linux. It is structured as a series of steps you need to 9 | follow, ordered in the most logical way. The next two sections describe some 10 | conventions we use in this document and the hardware, software, and expertise 11 | requirements. 12 | 13 | The most up-to date version of this document is available at: 14 | 15 | https://github.com/dashpay/electrum-dash-server/blob/master/HOWTO.md 16 | 17 | Conventions 18 | ----------- 19 | 20 | In this document, lines starting with a hash sign (#) or a dollar sign ($) 21 | contain commands. Commands starting with a hash should be run as root, 22 | commands starting with a dollar should be run as a normal user (in this 23 | document, we assume that user is called 'dash'). We also assume the 24 | dash user has sudo rights, so we use `$ sudo command` when we need to. 25 | 26 | Strings that are surrounded by "lower than" and "greater than" ( < and > ) 27 | should be replaced by the user with something appropriate. For example, 28 | \ should be replaced by a password. Do not confuse this 29 | notation with shell redirection (`command < file` or `command > file`)! 30 | 31 | Lines that lack hash or dollar signs are pastes from config files. They 32 | should be copied verbatim or adapted without the indentation tab. 33 | 34 | `apt-get install` commands are suggestions for required dependencies. 35 | They conform to an Ubuntu 15.10 system but may well work with Debian 36 | or other versions of Ubuntu. 37 | 38 | Prerequisites 39 | ------------- 40 | 41 | **Expertise.** You should be familiar with Linux command line and 42 | standard Linux commands. You should have a basic understanding of git 43 | and Python packages. You should have knowledge about how to install and 44 | configure software on your Linux distribution. You should be able to 45 | add commands to your distribution's startup scripts. If one of the 46 | commands included in this document is not available or does not 47 | perform the operation described here, you are expected to fix the 48 | issue so you can continue following this howto. 49 | 50 | **Software.** A recent Linux 64-bit distribution with the following software 51 | installed: `python`, `easy_install`, `git`, standard C/C++ 52 | build chain. You will need root access in order to install other software or 53 | Python libraries. Python 2.7 is the minimum supported version. 54 | 55 | **Hardware.** The lightest setup is a pruning server with diskspace 56 | requirements of about 30 GB for the Electrum database (February 2016). However note that 57 | you also need to run dashd and keep a copy of the full blockchain, 58 | which is roughly 55 GB (February 2016). Ideally you have a machine with 16 GB of RAM 59 | and an equal amount of swap. If you have ~2 GB of RAM make sure you limit dashd 60 | to 8 concurrent connections by disabling incoming connections. electrum-server may 61 | bail-out on you from time to time with less than 4 GB of RAM, so you might have to 62 | monitor the process and restart it. You can tweak cache sizes in the config to an extend 63 | but most RAM will be used to process blocks and catch-up on initial start. 64 | 65 | CPU speed is less important than fast I/O speed. electrum-server makes uses of one core 66 | only leaving spare cycles for dashd. Fast single core CPU power helps for the initial 67 | block chain import. Any multi-core x86 CPU with CPU Mark / PassMark > 1500 will work 68 | (see https://www.cpubenchmark.net/). An ideal setup in February 2016 has 16 GB+ RAM and 69 | SSD for good i/o speed. 70 | 71 | Instructions 72 | ------------ 73 | 74 | ### Step 1. Create a user for running dashd and Electrum Dash server 75 | 76 | This step is optional, but for better security and resource separation I 77 | suggest you create a separate user just for running `dashd` and Electrum. 78 | We will also use the `~/bin` directory to keep locally installed files 79 | (others might want to use `/usr/local/bin` instead). We will download source 80 | code files to the `~/src` directory. 81 | 82 | $ sudo adduser dash --disabled-password 83 | $ sudo apt-get install git 84 | $ sudo su - dash 85 | $ mkdir ~/bin ~/src 86 | $ echo $PATH 87 | 88 | If you don't see `/home/dash/bin` in the output, you should add this line 89 | to your `.bashrc`, `.profile`, or `.bash_profile`, then logout and relogin: 90 | 91 | PATH="$HOME/bin:$PATH" 92 | $ exit 93 | 94 | ### Step 2. Download dashd 95 | 96 | Recommend downloading latest version directly from Dash.org 97 | 98 | $ sudo apt-get install make g++ python-leveldb libboost-all-dev libssl-dev libdb++-dev pkg-config libevent-dev 99 | $ sudo su - dash 100 | $ cd ~/src && wget https://www.dash.org/binaries/dash-0.12.0.57-linux64.tar.gz 101 | $ sha256sum dash-0.12.0.57-linux64.tar.gz | grep 2a29b529c56d2ba41e28dfb20872861d6b48bdfe4fb327bfd2273123b38139aa 102 | $ tar xfz dash-0.12.0.57-linux64.tar.gz 103 | $ cd dash-0.12.0/bin 104 | $ cp -a dashd dash-cli dash-tx ~/bin 105 | 106 | ### Step 3. Configure and start dashd 107 | 108 | In order to allow Electrum Dash to "talk" to `dashd`, we need to set up an RPC 109 | username and password for `dashd`. We will then start `dashd` and 110 | wait for it to complete downloading the blockchain. 111 | 112 | $ mkdir ~/.dash 113 | $ $EDITOR ~/.dash/dash.conf 114 | 115 | Write this in `dash.conf`: 116 | 117 | rpcuser= 118 | rpcpassword= 119 | daemon=1 120 | txindex=1 121 | 122 | 123 | If you have an existing installation of dashd and have not previously 124 | set txindex=1 you need to reindex the blockchain by running 125 | 126 | $ dashd -reindex 127 | 128 | If you already have a freshly indexed copy of the blockchain with txindex start `dashd`: 129 | 130 | $ dashd 131 | 132 | Allow some time to pass for `dashd` to connect to the network and start 133 | downloading blocks. You can check its progress by running: 134 | 135 | $ dash-cli getblockchaininfo 136 | 137 | Before starting the Electrum Dash server your dashd should have processed all 138 | blocks and caught up to the current height of the network (not just the headers). 139 | You should also set up your system to automatically start dashd at boot 140 | time, running as the 'dash' user. Check your system documentation to 141 | find out the best way to do this. 142 | 143 | ### Step 4. Download and install Electrum Dash server 144 | 145 | We will download the latest git snapshot for Electrum to configure and install it: 146 | 147 | $ cd ~ 148 | $ git clone https://github.com/dashpay/electrum-dash-server.git 149 | $ cd electrum-dash-server 150 | $ sudo apt-get install python-setuptools 151 | $ sudo ./configure 152 | $ sudo python setup.py install 153 | 154 | See the INSTALL file for more information about the configure and install commands. 155 | 156 | ### Optional Step 5: Install Electrum Dash dependencies manually 157 | 158 | Electrum Dash server depends on various standard Python libraries and leveldb. These will usually be 159 | installed by calling `python setup.py install` above. They can be also be installed with your 160 | package manager if you don't want to use the install routine. 161 | 162 | $ sudo apt-get install python-setuptools python-openssl python-leveldb libleveldb-dev 163 | $ sudo easy_install jsonrpclib irc plyvel x11_hash 164 | 165 | For the python irc module please note electrum-server currently only supports versions between 11 and 14.0. 166 | The setup.py takes care of installing a supported version but be aware of it when installing or upgrading 167 | manually. 168 | 169 | Regarding leveldb, see the steps in README.leveldb for further details, especially if your system 170 | doesn't have the python-leveldb package or if plyvel installation fails. 171 | 172 | leveldb should be at least version 1.9.0. Earlier version are believed to be buggy. 173 | 174 | ### Step 6. Select your limit 175 | 176 | Electrum Dash server uses leveldb to store transactions. You can choose 177 | how many spent transactions per address you want to store on the server. 178 | The default is 100, but there are also servers with 1000 or even 10000. 179 | Few addresses have more than 10000 transactions. A limit this high 180 | can be considered equivalent to a "full" server. Full servers previously 181 | used abe to store the blockchain. The use of abe for electrum servers is now 182 | deprecated. 183 | 184 | The pruning server uses leveldb and keeps a smaller and 185 | faster database by pruning spent transactions. It's a lot quicker to get up 186 | and running and requires less maintenance and diskspace than abe. 187 | 188 | The section in the electrum server configuration file (see step 10) looks like this: 189 | 190 | [leveldb] 191 | path = /path/to/your/database 192 | # for each address, history will be pruned if it is longer than this limit 193 | pruning_limit = 100 194 | 195 | ### Step 7. Import blockchain into the database or download it 196 | 197 | It's recommended that you fetch a pre-processed leveldb from the net. 198 | The "configure" script above will offer you to download a database with pruning limit 100. 199 | 200 | You can fetch recent copies of electrum leveldb databases with different pruning limits 201 | and further instructions from the Electrum Dash full archival server foundry at a later time. 202 | Sadly there is no host server for these files as of yet.: 203 | 204 | 205 | Alternatively, if you have the time and nerve, you can import the blockchain yourself. 206 | 207 | As of April 2014 it takes between two days and over a week to import 300k blocks, depending 208 | on CPU speed, I/O speed, and your selected pruning limit. 209 | 210 | It's considerably faster and strongly recommended to index in memory. You can use /dev/shm or 211 | or create a tmpfs which will also use swap if you run out of memory: 212 | 213 | $ sudo mount -t tmpfs -o rw,nodev,nosuid,noatime,size=15000M,mode=0777 none /tmpfs 214 | 215 | If you use tmpfs make sure you have enough RAM and swap to cover the size. If you only have 4 GB of 216 | RAM but add 15 GB of swap from a file that's fine too; tmpfs is smart enough to swap out the least 217 | used parts. It's fine to use a file on an SSD for swap in this case. 218 | 219 | It's not recommended to do initial indexing of the database on an SSD because the indexing process 220 | does at least 20 TB (!) of disk writes and puts considerable wear-and-tear on an SSD. It's a lot better 221 | to use tmpfs and just swap out to disk when necessary. 222 | 223 | Databases have grown to roughly 30 GB as of February 2016. Leveldb prunes the database from time to time, 224 | so it's not uncommon to see databases ~50% larger at times when it's writing a lot, especially when 225 | indexing from the beginning. 226 | 227 | 228 | ### Step 8. Create a self-signed SSL cert 229 | 230 | [Note: SSL certificates signed by a CA are supported by 2.0 clients.] 231 | 232 | To run SSL / HTTPS you need to generate a self-signed certificate using openssl. 233 | You could just comment out the SSL / HTTPS ports in the config and run 234 | without, but this is not recommended. 235 | 236 | Use the sample code below to create a self-signed cert with a recommended validity 237 | of 5 years. You may supply any information for your sign request to identify your server. 238 | They are not currently checked by the client except for the validity date. 239 | When asked for a challenge password just leave it empty and press enter. 240 | 241 | $ openssl genrsa -des3 -passout pass:x -out server.pass.key 2048 242 | $ openssl rsa -passin pass:x -in server.pass.key -out server.key 243 | writing RSA key 244 | $ rm server.pass.key 245 | $ openssl req -new -key server.key -out server.csr 246 | ... 247 | Country Name (2 letter code) [AU]:US 248 | State or Province Name (full name) [Some-State]:California 249 | Common Name (eg, YOUR name) []: electrum-server.tld 250 | ... 251 | A challenge password []: 252 | ... 253 | 254 | $ openssl x509 -req -days 1825 -in server.csr -signkey server.key -out server.crt 255 | 256 | The server.crt file is your certificate suitable for the `ssl_certfile=` parameter and 257 | server.key corresponds to `ssl_keyfile=` in your Electrum Dash server config. 258 | 259 | Starting with Electrum 1.9, the client will learn and locally cache the SSL certificate 260 | for your server upon the first request to prevent man-in-the middle attacks for all 261 | further connections. 262 | 263 | If your certificate is lost or expires on the server side, you will need to run 264 | your server with a different server name and a new certificate. 265 | Therefore it's a good idea to make an offline backup copy of your certificate and key 266 | in case you need to restore them. 267 | 268 | ### Step 9. Configure Electrum Dash server 269 | 270 | Electrum Dash reads a config file (/etc/electrum-dash.conf) when starting up. This 271 | file includes the database setup, dashd RPC setup, and a few other 272 | options. 273 | 274 | The "configure" script listed above will create a config file at /etc/electrum-dash.conf 275 | which you can edit to modify the settings. 276 | 277 | Go through the config options and set them to your liking. 278 | If you intend to run the server publicly have a look at README-IRC.md 279 | 280 | ### Step 10. Tweak your system for running electrum 281 | 282 | Electrum Dash server currently needs quite a few file handles to use leveldb. It also requires 283 | file handles for each connection made to the server. It's good practice to increase the 284 | open files limit to 64k. 285 | 286 | The "configure" script will take care of this and ask you to create a user for running electrum-dash-server. 287 | If you're using the user `dash` to run electrum and have added it as shown in this document, run 288 | the following code to add the limits to your /etc/security/limits.conf: 289 | 290 | echo "dash hard nofile 65536" >> /etc/security/limits.conf 291 | echo "dash soft nofile 65536" >> /etc/security/limits.conf 292 | 293 | If you are on Debian > 8.0 Jessie or another distribution based on it, you also need to add these lines in /etc/pam.d/common-session and /etc/pam.d/common-session-noninteractive otherwise the limits in /etc/security/limits.conf will not work: 294 | 295 | echo "session required pam_limits.so" >> /etc/pam.d/common-session 296 | echo "session required pam_limits.so" >> /etc/pam.d/common-session-noninteractive 297 | 298 | Check if the limits are changed either by logging with the user configured to run Electrum Dash server as. Example: 299 | 300 | su - dash 301 | ulimit -n 302 | 303 | Or if you use sudo and the user is added to sudoers group: 304 | 305 | sudo -u dash -i ulimit -n 306 | 307 | 308 | Two more things for you to consider: 309 | 310 | 1. To increase privacy of transactions going through your server 311 | you may want to close dashd for incoming connections and connect outbound only. Most servers do run 312 | full nodes with open incoming connections though. 313 | 314 | 2. Consider restarting dashd (together with electrum-server) on a weekly basis to clear out unconfirmed 315 | transactions from the local the memory pool which did not propagate over the network. 316 | 317 | ### Step 11. (Finally!) Run Electrum Dash server 318 | 319 | The magic moment has come: you can now start your Electrum Dash server as root (it will su to your unprivileged user): 320 | 321 | # electrum-server start 322 | 323 | Note: If you want to run the server without installing it on your system, just run 'run_electrum_server" as the 324 | unprivileged user. 325 | 326 | You should see this in the log file: 327 | 328 | starting Electrum Dash server 329 | 330 | If your blockchain database is out of date Electrum Server will start updating it. You will see something similar to this in the log file: 331 | 332 | [09/02/2016-09:58:18] block 397319 (1727 197.37s) 0290aae5dc6395e2c60e8b2c9e48a7ee246cad7d0630d17dd5b54d70a41ffed7 (10.13tx/s, 139.78s/block) (eta 11.5 hours, 240 blocks) 333 | 334 | The important pieces to you are at the end. In this example, the server has to calculate 240 more blocks, with an ETA of 11.5 hours. Multiple entries will appear below this one as the server catches back up to the latest block. During this time the server will not accept incoming connections from clients or connect to the IRC channel. 335 | 336 | If you want to stop Electrum Dash server, use the 'stop' command: 337 | 338 | # electrum-server stop 339 | 340 | 341 | If your system supports it, you may add electrum-server to the /etc/init.d directory. 342 | This will ensure that the server is started and stopped automatically, and that the database is closed 343 | safely whenever your machine is rebooted. 344 | 345 | # ln -s `which electrum-server` /etc/init.d/electrum-server 346 | # update-rc.d electrum-server defaults 347 | 348 | ### Step 12. Test the Electrum Dash server 349 | 350 | We will assume you have a working Electrum client, a wallet, and some 351 | transaction history. You should start the client and click on the green 352 | checkmark (last button on the right of the status bar) to open the Server 353 | selection window. If your server is public, you should see it in the list 354 | and you can select it. If you server is private, you need to enter its IP 355 | or hostname and the port. Press 'Ok' and the client will disconnect from the 356 | current server and connect to your new Electrum Dash server. You should see your 357 | addresses and transactions history. You can see the number of blocks and 358 | response time in the server selection window. You should send/receive some 359 | dashs to confirm that everything is working properly. 360 | 361 | ### Step 13. Join us on IRC, subscribe to the server thread 362 | 363 | Say hi to the dev crew, other server operators, and fans on 364 | irc.freenode.net #electrum and we'll try to congratulate you 365 | on supporting the community by running an Electrum node. 366 | 367 | If you're operating a public Electrum Dash server please subscribe 368 | to or regularly check the following thread: 369 | https://bitcointalk.org/index.php?topic=85475.0 370 | It'll contain announcements about important updates to Electrum 371 | server required for a smooth user experience. 372 | -------------------------------------------------------------------------------- /INSTALL: -------------------------------------------------------------------------------- 1 | The following is a quick installation guide. Please see HOWTO.md for more 2 | detailed information on how to configure your server. 3 | 4 | 5 | TLDR: 6 | $ sudo ./configure 7 | $ sudo python ./setup.py install 8 | $ electrum-dash-server start 9 | $ electrum-dash-server stop 10 | 11 | 12 | Note: If you want to run the server without installing it on your 13 | system, just run 'run_electrum_dash_server' 14 | 15 | 16 | 1. Install and run dashd. 17 | ---------------------------- 18 | 19 | A full dash node is required in order to know for each address 20 | if it has been used. Pruning occurs only at the level of the 21 | Electrum Dash database. 22 | 23 | 24 | 2. Run the 'configure' script 25 | --------------------------- 26 | You need to run the script as root: 27 | 28 | $ sudo ./configure 29 | 30 | It will: 31 | * create the configuration file in /etc/electrum-dash.conf 32 | * create a user that will run the daemon 33 | * optionally, download a fresh database from the Electrum Foundry. 34 | 35 | If you choose not to use the Foundry, your server will have to build 36 | the database from scratch. This process can take several days. To 37 | speed it up, it is recommended to locate your database in shared 38 | memory, in electrum.conf: path = /run/shm/electrum_db 39 | 40 | Note: The 'configure' script does not configure SSL and IRC. You will 41 | need to manually edit the configuration file in order to enable SSL on 42 | your server, and to be visible on IRC. 43 | 44 | 45 | 3. Install the python package 46 | ----------------------------- 47 | 48 | $ sudo python setup.py install 49 | 50 | Note: You will need to redo this everytime you pull code from git. 51 | 52 | 53 | 4. Using the server 54 | ------------------- 55 | 56 | Use 'electrum-dash-server' to start and stop the server: 57 | 58 | $ electrum-dash-server 59 | 60 | The server will write a log file in its database path. 61 | 62 | 63 | 5. Add electrum-dash-server to your system's services 64 | ------------------------------------------------ 65 | 66 | If your system supports it, you may add electrum-dash-server to the 67 | /etc/init.d directory. This will ensure that the server is started and 68 | stopped automatically, and that the database is closed safely whenever 69 | your machine is rebooted. 70 | 71 | # ln -s `which electrum-dash-server` /etc/init.d/electrum-dash-server 72 | # update-rc.d electrum-dash-server defaults 73 | 74 | 75 | 76 | 6. Troubleshooting: 77 | ------------------- 78 | 79 | * if your server or dashd is killed because is uses too much 80 | memory, configure dashd to limit the number of connections 81 | 82 | * if you see "Too many open files" errors, you may need to increase 83 | your user's File Descriptors limit. For this, see 84 | http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/ 85 | 86 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining 4 | a copy of this software and associated documentation files (the 5 | "Software"), to deal in the Software without restriction, including 6 | without limitation the rights to use, copy, modify, merge, publish, 7 | distribute, sublicense, and/or sell copies of the Software, and to 8 | permit persons to whom the Software is furnished to do so, subject to 9 | the following conditions: 10 | 11 | The above copyright notice and this permission notice shall be 12 | included in all copies or substantial portions of the Software. 13 | 14 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 15 | EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 16 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 17 | NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE 18 | LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 19 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION 20 | WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 21 | -------------------------------------------------------------------------------- /README-IRC.md: -------------------------------------------------------------------------------- 1 | IRC is used by Electrum Dash server to find 'peers' - other Electrum Dash servers. The 2 | current list can be seen by running: 3 | 4 | electrum-dash-server peers 5 | 6 | The following config file options are used by the IRC part of Electrum Dash server: 7 | 8 | [server] 9 | irc = yes 10 | irc_nick = server_nickname 11 | host = fqdn.host.name.tld 12 | # report_host = fqdn.host.name.tld 13 | # report_stratum_tcp_port = 50001 14 | 15 | `irc` is used to determine whether the IRC thread will be started or 16 | the Electrum Dash server will run in private mode (default). In private 17 | mode, `electrum-dash-server peers` will always return an empty list. 18 | 19 | `host` is a fully-qualified domain name (FQDN) of your Electrum Dash 20 | server. It is used both when binding the listener for incoming client 21 | connections and as part of the realname field in IRC (see below). 22 | 23 | `report_host` is a an optional fully-qualified domain name (FQDN) of 24 | your Electrum Dash server instead of `host`. It is used as part of the name 25 | field in IRC for incoming client connections. This is useful in a NAT 26 | setup where you bind to a private IP locally but have an external IP 27 | set up at your router and external DNS. 28 | 29 | `report_stratum_tcp_port` and `report_stratum_tcp_ssl_port` are 30 | optional settings for a port number to be reported in the IRC name 31 | field without actually binding this port locally. This is useful in a 32 | NAT setup where you might want to bind to a high port locally but DNAT 33 | a different possibly privileged port for inbound connections 34 | 35 | `irc_nick` is a nick name that will be appended to the D_ suffix when 36 | composing the IRC nickname to identify your server on #electrum-dash . 37 | 38 | Please note the IRC name field can only contain 50 chars and will be 39 | composed of `host` + protocol version number + Port numbers for the 40 | various protocols. Please check whether port numbers are cut off at 41 | the end 42 | 43 | 44 | Example of port forwarding using iptables: 45 | iptables -t nat -A PREROUTING -p tcp --dport 110 -j REDIRECT --to-ports 50002 46 | 47 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | Electrum-dash-server for the Electrum Dash client 2 | ========================================= 3 | 4 | * Author: Thomas Voegtlin (ThomasV on the bitcointalk forum) 5 | * Dash codebase port Authors: ELM4Ever, Propulsion 6 | * Language: Python 7 | 8 | Features 9 | -------- 10 | 11 | * The server indexes UTXOs by address, in a Patricia tree structure 12 | described by Alan Reiner (see the 'ultimate blockchain 13 | compression' thread in the Bitcointalk forum) 14 | 15 | * The server requires dashd, leveldb, x11_hash and plyvel 16 | 17 | * The server code is open source. Anyone can run a server, removing 18 | single points of failure concerns. 19 | 20 | * The server knows which set of Bitcoin addresses belong to the same 21 | wallet, which might raise concerns about anonymity. However, it 22 | should be possible to write clients capable of using several 23 | servers. 24 | 25 | Installation 26 | ------------ 27 | 28 | 1. To install and run a server, see INSTALL. For greater 29 | detail on the installation process, see HOWTO.md. 30 | 31 | 2. To start and stop the server, use the 'electrum-dash-server' script 32 | 33 | 34 | 35 | License 36 | ------- 37 | 38 | Electrum-server is made available under the terms of the MIT License. 39 | See the included `LICENSE` for more details. 40 | -------------------------------------------------------------------------------- /configure: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -e 3 | 4 | # check root 5 | if [ ! "$(id -u)" == "0" ]; then 6 | echo "must be root to run install script" 7 | exit 1 8 | fi 9 | 10 | 11 | # source directory 12 | source_dir=$(cd "$(dirname "${BASH_SOURCE[0]}" )"; pwd) 13 | 14 | 15 | electrum_config="/etc/electrum-dash.conf" 16 | function read_config() 17 | { 18 | text=$1 19 | echo `grep -e ^$text $electrum_config |awk -F\= '{print $2}' | tail -n 1| tr -d ' '` 20 | } 21 | function write_config() 22 | { 23 | sed -i -s "s#$1 =.*#$1 = $2#" $electrum_config 24 | } 25 | 26 | # create config file 27 | if [ ! -f $electrum_config ]; then 28 | echo "Creating config file" 29 | cp $source_dir"/electrum-dash.conf.sample" $electrum_config 30 | fi 31 | 32 | 33 | # read username 34 | user=$(read_config "username") 35 | if ! [ "$user" ]; then 36 | read -p "username for running daemon (default: electrum) " -r 37 | if [ $REPLY ]; then 38 | user=$REPLY 39 | else 40 | user="electrum" 41 | fi 42 | write_config "username" $user 43 | fi 44 | 45 | 46 | # create user 47 | if [ ! id -u $user >/dev/null 2>&1 ]; then 48 | echo "adding user $user" 49 | useradd $user 50 | echo "$user hard nofile 65536" >> /etc/security/limits.conf 51 | echo "$user soft nofile 65536" >> /etc/security/limits.conf 52 | fi 53 | 54 | 55 | # read path from config 56 | default_path="/var/electrum-dash-server" 57 | path=$(read_config "path") 58 | if ! [ "$path" ]; then 59 | read -p "Path for database (default: $default_path) " -r 60 | if [ $REPLY ]; then 61 | path=$REPLY 62 | else 63 | path=$default_path 64 | fi 65 | write_config "path" $default_path 66 | fi 67 | 68 | # read path from config 69 | default_logfile="/var/log/electrum-dash.log" 70 | logfile=$(read_config "logfile") 71 | if ! [ "$logfile" ]; then 72 | read -p "Path of logfile (default: $default_logfile) " -r 73 | if [ $REPLY ]; then 74 | logfile=$REPLY 75 | else 76 | logfile=$default_logfile 77 | fi 78 | write_config "logfile" $default_logfile 79 | fi 80 | 81 | 82 | # remove directory if it exists and is empty 83 | if [ -d $path ]; then 84 | rmdir $path --ignore-fail-on-non-empty 85 | fi 86 | 87 | 88 | # download database 89 | if [ ! -d $path ]; then 90 | echo "Database not found in $path." 91 | read -p "Do you want to download it from the Electrum foundry to $path ? " -n 1 -r 92 | echo 93 | if [[ $REPLY =~ ^[Yy]$ ]]; then 94 | mkdir -p $path 95 | wget -O - "http://foundry.electrum.org/leveldb-dump/electrum-fulltree-100-latest.tar.gz" | tar --extract --gunzip --strip-components 1 --directory $path --file - 96 | fi 97 | fi 98 | 99 | # set owner 100 | chown -R $user:$user $path 101 | 102 | 103 | # create log file 104 | logfile=$(read_config "logfile") 105 | if ! [ -f $logfile ]; then 106 | touch $logfile 107 | fi 108 | chown $user:$user $logfile 109 | 110 | 111 | dashd_user=$(read_config "dashd_user") 112 | if ! [ "$dashd_user" ]; then 113 | read -p "rpcuser (from your dash.conf file): " -r 114 | write_config "dashd_user" $REPLY 115 | fi 116 | 117 | dashd_password=$(read_config "dashd_password") 118 | if ! [ "$dashd_password" ]; then 119 | read -p "rpcpassword (from your dash.conf file): " -r 120 | write_config "dashd_password" $REPLY 121 | fi 122 | 123 | 124 | # finish 125 | echo "Configuration written to $electrum_config." 126 | echo "Please edit this file to finish the configuration." 127 | echo "If you have not done so, please run 'python setup.py install'" 128 | echo "Then, run 'electrum-server' to start the daemon" 129 | -------------------------------------------------------------------------------- /electrum-dash-server: -------------------------------------------------------------------------------- 1 | #! /bin/bash 2 | 3 | 4 | electrum_config="/etc/electrum-dash.conf" 5 | if [ ! -f $electrum_config ]; then 6 | echo "$electrum_config does not exist" 7 | echo "please run 'configure'" 8 | exit 1 9 | fi 10 | 11 | if ! hash run_electrum_dash_server 2>/dev/null; then 12 | echo "run_electrum_dash_server is not installed" 13 | echo "Please run 'python setup.py install'" 14 | exit 1 15 | fi 16 | 17 | 18 | 19 | function read_config() 20 | { 21 | text=$1 22 | echo `grep -e ^$text $electrum_config |awk -F\= '{print $2}' | tail -n 1| tr -d ' '` 23 | } 24 | 25 | # read path from config 26 | path=$(read_config "path") 27 | # read username 28 | user=$(read_config "username") 29 | 30 | # get PID of server 31 | if [ `id -u` = 0 ] ; then 32 | if ! PID=`su $user -c "run_electrum_dash_server getpid"`; then 33 | PID="" 34 | fi 35 | else 36 | if ! PID=`run_electrum_dash_server getpid`; then 37 | PID="" 38 | fi 39 | fi; 40 | 41 | case "$1" in 42 | start) 43 | if [ "$PID" ]; then 44 | echo "Server already running (pid $PID)" 45 | exit 46 | fi 47 | 48 | if ! [ "$path" ]; then 49 | echo "Variable path not set in $electrum_config" 50 | exit 51 | fi 52 | 53 | logfile=$(read_config "logfile") 54 | if ! [ -f $logfile ]; then 55 | touch $logfile 56 | chown $user:$user $logfile 57 | fi 58 | 59 | echo "Starting server as daemon" 60 | cmd="ulimit -n 65536 ; nohup run_electrum_dash_server > /dev/null 2>>$logfile &" 61 | if [ `id -u` = 0 ] ; then 62 | su $user -c "$cmd" 63 | else 64 | eval $cmd 65 | fi 66 | ;; 67 | stop) 68 | if [ ! $PID ]; then 69 | echo "Server not running" 70 | exit 71 | fi 72 | cmd="run_electrum_dash_server stop" 73 | if [ `id -u` = 0 ] ; then 74 | su $user -c "$cmd" 75 | else 76 | $cmd 77 | fi 78 | echo "Waiting until process $PID terminates..." 79 | while ps -p $PID > /dev/null; do sleep 1; done 80 | echo "Done." 81 | ;; 82 | status) 83 | if [ ! "$PID" ]; then 84 | echo "Server not running" 85 | else 86 | echo "Server running (pid $PID)" 87 | fi 88 | ;; 89 | getinfo|peers|numpeers|sessions|numsessions) 90 | if [ ! "$PID" ]; then 91 | echo "Server not running" 92 | exit 93 | fi 94 | cmd="run_electrum_dash_server $1" 95 | if [ `id -u` = 0 ] ; then 96 | su $user -c "$cmd" 97 | else 98 | $cmd 99 | fi 100 | ;; 101 | restart) 102 | $0 stop 103 | $0 start 104 | ;; 105 | *) 106 | echo "Usage: electrum-dash-server {start|stop|restart|status|getinfo|peers|numpeers|sessions|numsessions}" 107 | exit 1 108 | ;; 109 | esac 110 | 111 | exit 0 112 | -------------------------------------------------------------------------------- /electrum-dash.conf.sample: -------------------------------------------------------------------------------- 1 | [server] 2 | # username for running the daemon 3 | username = 4 | # hostname. set it to a FQDN in order to be reached from outside 5 | host = localhost 6 | # ports 7 | electrum_rpc_port = 8000 8 | stratum_tcp_port = 50001 9 | #stratum_tcp_ssl_port = 50002 10 | #report_host = 11 | #report_stratum_tcp_port = 50001 12 | #report_stratum_tcp_ssl_port = 50002 13 | banner = Welcome to Electrum! 14 | banner_file = /etc/electrum.banner 15 | # IRC connection for finding peers, #electrum-dash on Freenode IRC. 16 | #irc = no 17 | #irc_nick = 18 | # IRC prefix for mainnet is "D_" and testnet is "d_" 19 | #irc_prefix = D_ 20 | #irc_bind_ip = 21 | #ssl_certfile = /path/to/electrum-server.crt 22 | #ssl_keyfile = /path/to/electrum-server.key 23 | logfile = /var/log/electrum.log 24 | donation_address = 25 | 26 | [leveldb] 27 | # path to your database 28 | path = 29 | # for each address, history will be pruned if it is longer than this limit 30 | pruning_limit = 100 31 | 32 | # cache sizes in bytes, the default is optimized for ~4 GB RAM setups to run dashd alongside 33 | # If you have lots of RAM increase up to 16 times for best performance 34 | #hist_cache = 67108864 35 | #utxo_cache = 134217728 36 | #addr_cache = 16777216 37 | 38 | [dashd] 39 | dashd_host = localhost 40 | dashd_port = 8332 41 | # user and password from dash.conf 42 | dashd_user = 43 | dashd_password = 44 | -------------------------------------------------------------------------------- /kill_session: -------------------------------------------------------------------------------- 1 | # run this script to close a session on your server 2 | # usage: su electrum -c "./kill_session ip:port" 3 | ./run_electrum_dash_server debug "dispatcher.request_dispatcher.sessions['$1'].stop()" 4 | -------------------------------------------------------------------------------- /run_electrum_dash_server: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # Copyright(C) 2011-2016 Thomas Voegtlin 3 | # 4 | # Permission is hereby granted, free of charge, to any person 5 | # obtaining a copy of this software and associated documentation files 6 | # (the "Software"), to deal in the Software without restriction, 7 | # including without limitation the rights to use, copy, modify, merge, 8 | # publish, distribute, sublicense, and/or sell copies of the Software, 9 | # and to permit persons to whom the Software is furnished to do so, 10 | # subject to the following conditions: 11 | # 12 | # The above copyright notice and this permission notice shall be 13 | # included in all copies or substantial portions of the Software. 14 | # 15 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 16 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 17 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 18 | # NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS 19 | # BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN 20 | # ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 21 | # CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 22 | # SOFTWARE. 23 | 24 | import argparse 25 | import ConfigParser 26 | import logging 27 | import socket 28 | import sys 29 | import time 30 | import threading 31 | import json 32 | import os 33 | import imp 34 | 35 | 36 | if os.path.dirname(os.path.realpath(__file__)) == os.getcwd(): 37 | imp.load_module('electrumserver', *imp.find_module('src')) 38 | 39 | from electrumserver import storage, networks, utils 40 | from electrumserver.processor import Dispatcher, print_log 41 | from electrumserver.server_processor import ServerProcessor 42 | from electrumserver.blockchain_processor import BlockchainProcessor 43 | from electrumserver.stratum_tcp import TcpServer 44 | 45 | 46 | logging.basicConfig() 47 | 48 | if sys.maxsize <= 2**32: 49 | print "Warning: it looks like you are using a 32bit system. You may experience crashes caused by mmap" 50 | 51 | if os.getuid() == 0: 52 | print "Do not run this program as root!" 53 | print "Run the install script to create a non-privileged user." 54 | sys.exit() 55 | 56 | def attempt_read_config(config, filename): 57 | try: 58 | with open(filename, 'r') as f: 59 | config.readfp(f) 60 | except IOError: 61 | pass 62 | 63 | def load_banner(config): 64 | try: 65 | with open(config.get('server', 'banner_file'), 'r') as f: 66 | config.set('server', 'banner', f.read()) 67 | except IOError: 68 | pass 69 | 70 | def setup_network_params(config): 71 | type = config.get('network', 'type') 72 | params = networks.params.get(type) 73 | utils.PUBKEY_ADDRESS = int(params.get('pubkey_address')) 74 | utils.SCRIPT_ADDRESS = int(params.get('script_address')) 75 | storage.GENESIS_HASH = params.get('genesis_hash') 76 | 77 | if config.has_option('network', 'pubkey_address'): 78 | utils.PUBKEY_ADDRESS = config.getint('network', 'pubkey_address') 79 | if config.has_option('network', 'script_address'): 80 | utils.SCRIPT_ADDRESS = config.getint('network', 'script_address') 81 | if config.has_option('network', 'genesis_hash'): 82 | storage.GENESIS_HASH = config.get('network', 'genesis_hash') 83 | 84 | def create_config(filename=None): 85 | config = ConfigParser.ConfigParser() 86 | # set some defaults, which will be overwritten by the config file 87 | config.add_section('server') 88 | config.set('server', 'banner', 'Welcome to Electrum!') 89 | config.set('server', 'banner_file', '/etc/electrum.banner') 90 | config.set('server', 'host', 'localhost') 91 | config.set('server', 'electrum_rpc_port', '8000') 92 | config.set('server', 'report_host', '') 93 | config.set('server', 'stratum_tcp_port', '50001') 94 | config.set('server', 'stratum_tcp_ssl_port', '50002') 95 | config.set('server', 'report_stratum_tcp_port', '') 96 | config.set('server', 'report_stratum_tcp_ssl_port', '') 97 | config.set('server', 'ssl_certfile', '') 98 | config.set('server', 'ssl_keyfile', '') 99 | config.set('server', 'irc', 'no') 100 | config.set('server', 'irc_nick', '') 101 | config.set('server', 'coin', '') 102 | config.set('server', 'donation_address', '') 103 | config.set('server', 'max_subscriptions', '10000') 104 | 105 | config.add_section('leveldb') 106 | config.set('leveldb', 'path', '/dev/shm/electrum_db') 107 | config.set('leveldb', 'pruning_limit', '100') 108 | config.set('leveldb', 'reorg_limit', '100') 109 | config.set('leveldb', 'utxo_cache', str(64*1024*1024)) 110 | config.set('leveldb', 'hist_cache', str(128*1024*1024)) 111 | config.set('leveldb', 'addr_cache', str(16*1024*1024)) 112 | config.set('leveldb', 'profiler', 'no') 113 | 114 | # set network parameters 115 | config.add_section('network') 116 | config.set('network', 'type', 'dash_main') 117 | 118 | # try to find the config file in the default paths 119 | if not filename: 120 | for path in ('/etc/', ''): 121 | filename = path + 'electrum-dash.conf' 122 | if os.path.isfile(filename): 123 | break 124 | 125 | if not os.path.isfile(filename): 126 | print 'could not find electrum configuration file "%s"' % filename 127 | sys.exit(1) 128 | 129 | attempt_read_config(config, filename) 130 | 131 | load_banner(config) 132 | 133 | return config 134 | 135 | 136 | def run_rpc_command(params, electrum_rpc_port): 137 | cmd = params[0] 138 | import xmlrpclib 139 | server = xmlrpclib.ServerProxy('http://localhost:%d' % electrum_rpc_port) 140 | func = getattr(server, cmd) 141 | r = func(*params[1:]) 142 | if cmd == 'sessions': 143 | now = time.time() 144 | print 'type address sub version time' 145 | for item in r: 146 | print '%4s %21s %3s %7s %.2f' % (item.get('name'), 147 | item.get('address'), 148 | item.get('subscriptions'), 149 | item.get('version'), 150 | (now - item.get('time')), 151 | ) 152 | elif cmd == 'debug': 153 | print r 154 | else: 155 | print json.dumps(r, indent=4, sort_keys=True) 156 | 157 | 158 | def cmd_banner_update(): 159 | load_banner(dispatcher.shared.config) 160 | return True 161 | 162 | def cmd_getinfo(): 163 | return { 164 | 'blocks': chain_proc.storage.height, 165 | 'peers': len(server_proc.peers), 166 | 'sessions': len(dispatcher.request_dispatcher.get_sessions()), 167 | 'watched': len(chain_proc.watched_addresses), 168 | 'cached': len(chain_proc.history_cache), 169 | } 170 | 171 | def cmd_sessions(): 172 | return map(lambda s: {"time": s.time, 173 | "name": s.name, 174 | "address": s.address, 175 | "version": s.version, 176 | "subscriptions": len(s.subscriptions)}, 177 | dispatcher.request_dispatcher.get_sessions()) 178 | 179 | def cmd_numsessions(): 180 | return len(dispatcher.request_dispatcher.get_sessions()) 181 | 182 | def cmd_peers(): 183 | return server_proc.peers.keys() 184 | 185 | def cmd_numpeers(): 186 | return len(server_proc.peers) 187 | 188 | 189 | hp = None 190 | def cmd_guppy(): 191 | from guppy import hpy 192 | global hp 193 | hp = hpy() 194 | 195 | def cmd_debug(s): 196 | import traceback 197 | import gc 198 | if s: 199 | try: 200 | result = str(eval(s)) 201 | except: 202 | err_lines = traceback.format_exc().splitlines() 203 | result = '%s | %s' % (err_lines[-3], err_lines[-1]) 204 | return result 205 | 206 | 207 | def get_port(config, name): 208 | try: 209 | return config.getint('server', name) 210 | except: 211 | return None 212 | 213 | 214 | # share these as global, for 'debug' command 215 | shared = None 216 | chain_proc = None 217 | server_proc = None 218 | dispatcher = None 219 | transports = [] 220 | tcp_server = None 221 | ssl_server = None 222 | 223 | def start_server(config): 224 | global shared, chain_proc, server_proc, dispatcher 225 | global tcp_server, ssl_server 226 | 227 | utils.init_logger() 228 | host = config.get('server', 'host') 229 | stratum_tcp_port = get_port(config, 'stratum_tcp_port') 230 | stratum_tcp_ssl_port = get_port(config, 'stratum_tcp_ssl_port') 231 | ssl_certfile = config.get('server', 'ssl_certfile') 232 | ssl_keyfile = config.get('server', 'ssl_keyfile') 233 | 234 | setup_network_params(config) 235 | 236 | if ssl_certfile is '' or ssl_keyfile is '': 237 | stratum_tcp_ssl_port = None 238 | 239 | print_log("Starting Electrum Dash server on", host) 240 | 241 | # Create hub 242 | dispatcher = Dispatcher(config) 243 | shared = dispatcher.shared 244 | 245 | # handle termination signals 246 | import signal 247 | def handler(signum = None, frame = None): 248 | print_log('Signal handler called with signal', signum) 249 | shared.stop() 250 | for sig in [signal.SIGTERM, signal.SIGHUP, signal.SIGQUIT]: 251 | signal.signal(sig, handler) 252 | 253 | # Create and register processors 254 | chain_proc = BlockchainProcessor(config, shared) 255 | dispatcher.register('blockchain', chain_proc) 256 | 257 | server_proc = ServerProcessor(config, shared) 258 | dispatcher.register('server', server_proc) 259 | 260 | # Create various transports we need 261 | if stratum_tcp_port: 262 | tcp_server = TcpServer(dispatcher, host, stratum_tcp_port, False, None, None) 263 | transports.append(tcp_server) 264 | 265 | if stratum_tcp_ssl_port: 266 | ssl_server = TcpServer(dispatcher, host, stratum_tcp_ssl_port, True, ssl_certfile, ssl_keyfile) 267 | transports.append(ssl_server) 268 | 269 | for server in transports: 270 | server.start() 271 | 272 | 273 | def stop_server(): 274 | shared.stop() 275 | server_proc.join() 276 | chain_proc.join() 277 | print_log("Electrum Server stopped") 278 | 279 | 280 | if __name__ == '__main__': 281 | parser = argparse.ArgumentParser() 282 | parser.add_argument('--conf', metavar='path', default=None, help='specify a configuration file') 283 | parser.add_argument('command', nargs='*', default=[], help='send a command to the server') 284 | args = parser.parse_args() 285 | config = create_config(args.conf) 286 | 287 | electrum_rpc_port = get_port(config, 'electrum_rpc_port') 288 | 289 | if len(args.command) >= 1: 290 | try: 291 | run_rpc_command(args.command, electrum_rpc_port) 292 | except socket.error: 293 | print "server not running" 294 | sys.exit(1) 295 | sys.exit(0) 296 | 297 | try: 298 | run_rpc_command(['getpid'], electrum_rpc_port) 299 | is_running = True 300 | except socket.error: 301 | is_running = False 302 | 303 | if is_running: 304 | print "server already running" 305 | sys.exit(1) 306 | 307 | start_server(config) 308 | 309 | from SimpleXMLRPCServer import SimpleXMLRPCServer 310 | server = SimpleXMLRPCServer(('localhost', electrum_rpc_port), allow_none=True, logRequests=False) 311 | server.register_function(lambda: os.getpid(), 'getpid') 312 | server.register_function(shared.stop, 'stop') 313 | server.register_function(cmd_getinfo, 'getinfo') 314 | server.register_function(cmd_sessions, 'sessions') 315 | server.register_function(cmd_numsessions, 'numsessions') 316 | server.register_function(cmd_peers, 'peers') 317 | server.register_function(cmd_numpeers, 'numpeers') 318 | server.register_function(cmd_debug, 'debug') 319 | server.register_function(cmd_guppy, 'guppy') 320 | server.register_function(cmd_banner_update, 'banner_update') 321 | server.socket.settimeout(1) 322 | 323 | while not shared.stopped(): 324 | try: 325 | server.handle_request() 326 | except socket.timeout: 327 | continue 328 | except: 329 | stop_server() 330 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from setuptools import setup 2 | 3 | setup( 4 | name="electrum-dash-server", 5 | version="1.0", 6 | scripts=['run_electrum_dash_server','electrum-dash-server'], 7 | install_requires=['plyvel','jsonrpclib', 'irc >= 11, <=14.0', 'x11_hash'], 8 | package_dir={ 9 | 'electrumserver':'src' 10 | }, 11 | py_modules=[ 12 | 'electrumserver.__init__', 13 | 'electrumserver.utils', 14 | 'electrumserver.storage', 15 | 'electrumserver.deserialize', 16 | 'electrumserver.networks', 17 | 'electrumserver.blockchain_processor', 18 | 'electrumserver.server_processor', 19 | 'electrumserver.processor', 20 | 'electrumserver.version', 21 | 'electrumserver.ircthread', 22 | 'electrumserver.stratum_tcp' 23 | ], 24 | description="Dash Electrum Server", 25 | author="Thomas Voegtlin ,ELM4ever, Propulsion, TheLazieR", 26 | author_email="thomasv@electrum.org, thelazier@gmail.com", 27 | license="MIT Licence", 28 | url="https://github.com/spesmilo/electrum-server/ , https://github.com/thelazier/electrum-dash-server/", 29 | long_description="""Server for the Electrum Lightweight Dash Wallet""" 30 | ) 31 | -------------------------------------------------------------------------------- /src/__init__.py: -------------------------------------------------------------------------------- 1 | import utils 2 | import storage 3 | import deserialize 4 | import networks 5 | import blockchain_processor 6 | import processor 7 | import version 8 | import irc 9 | import stratum_tcp 10 | -------------------------------------------------------------------------------- /src/blockchain_processor.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # Copyright(C) 2011-2016 Thomas Voegtlin 3 | # 4 | # Permission is hereby granted, free of charge, to any person 5 | # obtaining a copy of this software and associated documentation files 6 | # (the "Software"), to deal in the Software without restriction, 7 | # including without limitation the rights to use, copy, modify, merge, 8 | # publish, distribute, sublicense, and/or sell copies of the Software, 9 | # and to permit persons to whom the Software is furnished to do so, 10 | # subject to the following conditions: 11 | # 12 | # The above copyright notice and this permission notice shall be 13 | # included in all copies or substantial portions of the Software. 14 | # 15 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 16 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 17 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 18 | # NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS 19 | # BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN 20 | # ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 21 | # CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 22 | # SOFTWARE. 23 | 24 | import hashlib 25 | from json import dumps, load 26 | import os 27 | from Queue import Queue 28 | import random 29 | import sys 30 | import time 31 | import threading 32 | import urllib 33 | 34 | import deserialize 35 | from processor import Processor, print_log 36 | from storage import Storage 37 | from utils import logger, hash_decode, hash_encode, HashX11, Hash, header_from_string, header_to_string, ProfiledThread, \ 38 | rev_hex, int_to_hex4 39 | 40 | class BlockchainProcessor(Processor): 41 | 42 | def __init__(self, config, shared): 43 | Processor.__init__(self) 44 | 45 | # monitoring 46 | self.avg_time = 0,0,0 47 | self.time_ref = time.time() 48 | 49 | self.shared = shared 50 | self.config = config 51 | self.up_to_date = False 52 | 53 | self.watch_lock = threading.Lock() 54 | self.watch_blocks = [] 55 | self.watch_headers = [] 56 | self.watched_addresses = {} 57 | 58 | self.history_cache = {} 59 | self.merkle_cache = {} 60 | self.max_cache_size = 100000 61 | self.chunk_cache = {} 62 | self.cache_lock = threading.Lock() 63 | self.headers_data = '' 64 | self.headers_path = config.get('leveldb', 'path') 65 | 66 | self.mempool_fees = {} 67 | self.mempool_values = {} 68 | self.mempool_addresses = {} 69 | self.mempool_hist = {} # addr -> (txid, delta) 70 | self.mempool_unconfirmed = {} # txid -> set of unconfirmed inputs 71 | self.mempool_hashes = set() 72 | self.mempool_lock = threading.Lock() 73 | 74 | self.address_queue = Queue() 75 | 76 | try: 77 | self.test_reorgs = config.getboolean('leveldb', 'test_reorgs') # simulate random blockchain reorgs 78 | except: 79 | self.test_reorgs = False 80 | self.storage = Storage(config, shared, self.test_reorgs) 81 | 82 | self.dashd_url = 'http://%s:%s@%s:%s/' % ( 83 | config.get('dashd', 'dashd_user'), 84 | config.get('dashd', 'dashd_password'), 85 | config.get('dashd', 'dashd_host'), 86 | config.get('dashd', 'dashd_port')) 87 | 88 | self.sent_height = 0 89 | self.sent_header = None 90 | 91 | # catch_up headers 92 | self.init_headers(self.storage.height) 93 | # start catch_up thread 94 | if config.getboolean('leveldb', 'profiler'): 95 | filename = os.path.join(config.get('leveldb', 'path'), 'profile') 96 | print_log('profiled thread', filename) 97 | self.blockchain_thread = ProfiledThread(filename, target = self.do_catch_up) 98 | else: 99 | self.blockchain_thread = threading.Thread(target = self.do_catch_up) 100 | self.blockchain_thread.start() 101 | 102 | 103 | def do_catch_up(self): 104 | self.header = self.block2header(self.dashd('getblock', (self.storage.last_hash,))) 105 | self.header['utxo_root'] = self.storage.get_root_hash().encode('hex') 106 | self.catch_up(sync=False) 107 | if not self.shared.stopped(): 108 | print_log("Blockchain is up to date.") 109 | self.memorypool_update() 110 | print_log("Memory pool initialized.") 111 | 112 | while not self.shared.stopped(): 113 | self.main_iteration() 114 | if self.shared.paused(): 115 | print_log("dashd is responding") 116 | self.shared.unpause() 117 | time.sleep(10) 118 | 119 | 120 | def set_time(self): 121 | self.time_ref = time.time() 122 | 123 | def print_time(self, num_tx): 124 | delta = time.time() - self.time_ref 125 | # leaky averages 126 | seconds_per_block, tx_per_second, n = self.avg_time 127 | alpha = (1. + 0.01 * n)/(n+1) 128 | seconds_per_block = (1-alpha) * seconds_per_block + alpha * delta 129 | alpha2 = alpha * delta / seconds_per_block 130 | tx_per_second = (1-alpha2) * tx_per_second + alpha2 * num_tx / delta 131 | self.avg_time = seconds_per_block, tx_per_second, n+1 132 | if self.storage.height%100 == 0 \ 133 | or (self.storage.height%10 == 0 and self.storage.height >= 100000)\ 134 | or self.storage.height >= 200000: 135 | msg = "block %d (%d %.2fs) %s" %(self.storage.height, num_tx, delta, self.storage.get_root_hash().encode('hex')) 136 | msg += " (%.2ftx/s, %.2fs/block)" % (tx_per_second, seconds_per_block) 137 | run_blocks = self.storage.height - self.start_catchup_height 138 | remaining_blocks = self.dashd_height - self.storage.height 139 | if run_blocks>0 and remaining_blocks>0: 140 | remaining_minutes = remaining_blocks * seconds_per_block / 60 141 | new_blocks = int(remaining_minutes / 10) # number of new blocks expected during catchup 142 | blocks_to_process = remaining_blocks + new_blocks 143 | minutes = blocks_to_process * seconds_per_block / 60 144 | rt = "%.0fmin"%minutes if minutes < 300 else "%.1f hours"%(minutes/60) 145 | msg += " (eta %s, %d blocks)" % (rt, remaining_blocks) 146 | print_log(msg) 147 | 148 | def wait_on_dashd(self): 149 | self.shared.pause() 150 | time.sleep(10) 151 | if self.shared.stopped(): 152 | # this will end the thread 153 | raise BaseException() 154 | 155 | def dashd(self, method, params=()): 156 | postdata = dumps({"method": method, 'params': params, 'id': 'jsonrpc'}) 157 | while True: 158 | try: 159 | response = urllib.urlopen(self.dashd_url, postdata) 160 | r = load(response) 161 | response.close() 162 | except: 163 | print_log("cannot reach dashd...") 164 | self.wait_on_dashd() 165 | else: 166 | if r['error'] is not None: 167 | if r['error'].get('code') == -28: 168 | print_log("dashd still warming up...") 169 | self.wait_on_dashd() 170 | continue 171 | raise BaseException(r['error']) 172 | break 173 | return r.get('result') 174 | 175 | @staticmethod 176 | def block2header(b): 177 | return { 178 | "block_height": b.get('height'), 179 | "version": b.get('version'), 180 | "prev_block_hash": b.get('previousblockhash'), 181 | "merkle_root": b.get('merkleroot'), 182 | "timestamp": b.get('time'), 183 | "bits": int(b.get('bits'), 16), 184 | "nonce": b.get('nonce'), 185 | } 186 | 187 | def get_header(self, height): 188 | block_hash = self.dashd('getblockhash', (height,)) 189 | b = self.dashd('getblock', (block_hash,)) 190 | return self.block2header(b) 191 | 192 | def init_headers(self, db_height): 193 | self.headers_filename = os.path.join(self.headers_path, 'blockchain_headers') 194 | 195 | if os.path.exists(self.headers_filename): 196 | height = os.path.getsize(self.headers_filename)/80 - 1 # the current height 197 | if height > 0: 198 | prev_hash = self.hash_header(self.read_header(height)) 199 | else: 200 | prev_hash = None 201 | else: 202 | open(self.headers_filename, 'wb').close() 203 | prev_hash = None 204 | height = -1 205 | 206 | if height < db_height: 207 | print_log("catching up missing headers:", height, db_height) 208 | 209 | try: 210 | while height < db_height: 211 | height += 1 212 | header = self.get_header(height) 213 | if height > 1: 214 | if prev_hash != header.get('prev_block_hash'): 215 | # The prev_hash block is orphaned, go back 216 | print_log("reorganizing, a block in file is orphaned:", prev_hash) 217 | # Go to the parent of the orphaned block 218 | height -= 2 219 | prev_hash = self.hash_header(self.read_header(height)) 220 | continue 221 | 222 | self.write_header(header, sync=False) 223 | prev_hash = self.hash_header(header) 224 | if (height % 1000) == 0: 225 | print_log("headers file:", height) 226 | except KeyboardInterrupt: 227 | self.flush_headers() 228 | sys.exit() 229 | 230 | self.flush_headers() 231 | 232 | @staticmethod 233 | def hash_header(header): 234 | return rev_hex(HashX11(header_to_string(header).decode('hex')).encode('hex')) 235 | 236 | def read_header(self, block_height): 237 | if os.path.exists(self.headers_filename): 238 | with open(self.headers_filename, 'rb') as f: 239 | f.seek(block_height * 80) 240 | h = f.read(80) 241 | if len(h) == 80: 242 | h = header_from_string(h) 243 | return h 244 | 245 | def read_chunk(self, index): 246 | with open(self.headers_filename, 'rb') as f: 247 | f.seek(index*2016*80) 248 | chunk = f.read(2016*80) 249 | return chunk.encode('hex') 250 | 251 | def write_header(self, header, sync=True): 252 | if not self.headers_data: 253 | self.headers_offset = header.get('block_height') 254 | 255 | self.headers_data += header_to_string(header).decode('hex') 256 | if sync or len(self.headers_data) > 40*100: 257 | self.flush_headers() 258 | 259 | with self.cache_lock: 260 | chunk_index = header.get('block_height')/2016 261 | if chunk_index in self.chunk_cache: 262 | del self.chunk_cache[chunk_index] 263 | 264 | def pop_header(self): 265 | # we need to do this only if we have not flushed 266 | if self.headers_data: 267 | self.headers_data = self.headers_data[:-40] 268 | 269 | def flush_headers(self): 270 | if not self.headers_data: 271 | return 272 | with open(self.headers_filename, 'rb+') as f: 273 | f.seek(self.headers_offset*80) 274 | f.write(self.headers_data) 275 | self.headers_data = '' 276 | 277 | def get_chunk(self, i): 278 | # store them on disk; store the current chunk in memory 279 | with self.cache_lock: 280 | chunk = self.chunk_cache.get(i) 281 | if not chunk: 282 | chunk = self.read_chunk(i) 283 | if chunk: 284 | self.chunk_cache[i] = chunk 285 | 286 | return chunk 287 | 288 | def get_mempool_transaction(self, txid): 289 | try: 290 | raw_tx = self.dashd('getrawtransaction', (txid, 0)) 291 | except: 292 | return None 293 | vds = deserialize.BCDataStream() 294 | vds.write(raw_tx.decode('hex')) 295 | try: 296 | return deserialize.parse_Transaction(vds, is_coinbase=False) 297 | except: 298 | print_log("ERROR: cannot parse", txid) 299 | return None 300 | 301 | def get_unconfirmed_history(self, addr): 302 | hist = [] 303 | with self.mempool_lock: 304 | for tx_hash, delta in self.mempool_hist.get(addr, ()): 305 | height = -1 if self.mempool_unconfirmed.get(tx_hash) else 0 306 | fee = self.mempool_fees.get(tx_hash) 307 | hist.append({'tx_hash':tx_hash, 'height':height, 'fee':fee}) 308 | return hist 309 | 310 | def get_history(self, addr, cache_only=False): 311 | with self.cache_lock: 312 | hist = self.history_cache.get(addr) 313 | if hist is not None: 314 | return hist 315 | if cache_only: 316 | return -1 317 | hist = self.storage.get_history(addr) 318 | hist.extend(self.get_unconfirmed_history(addr)) 319 | with self.cache_lock: 320 | if len(self.history_cache) > self.max_cache_size: 321 | logger.info("clearing cache") 322 | self.history_cache.clear() 323 | self.history_cache[addr] = hist 324 | return hist 325 | 326 | def get_unconfirmed_value(self, addr): 327 | v = 0 328 | with self.mempool_lock: 329 | for txid, delta in self.mempool_hist.get(addr, ()): 330 | v += delta 331 | return v 332 | 333 | def get_status(self, addr, cache_only=False): 334 | tx_points = self.get_history(addr, cache_only) 335 | if cache_only and tx_points == -1: 336 | return -1 337 | 338 | if not tx_points: 339 | return None 340 | if tx_points == ['*']: 341 | return '*' 342 | status = ''.join(tx.get('tx_hash') + ':%d:' % tx.get('height') for tx in tx_points) 343 | return hashlib.sha256(status).digest().encode('hex') 344 | 345 | def get_merkle(self, tx_hash, height, cache_only): 346 | with self.cache_lock: 347 | out = self.merkle_cache.get(tx_hash) 348 | if out is not None: 349 | return out 350 | if cache_only: 351 | return -1 352 | 353 | block_hash = self.dashd('getblockhash', (height,)) 354 | b = self.dashd('getblock', (block_hash,)) 355 | tx_list = b.get('tx') 356 | tx_pos = tx_list.index(tx_hash) 357 | 358 | merkle = map(hash_decode, tx_list) 359 | target_hash = hash_decode(tx_hash) 360 | s = [] 361 | while len(merkle) != 1: 362 | if len(merkle) % 2: 363 | merkle.append(merkle[-1]) 364 | n = [] 365 | while merkle: 366 | new_hash = Hash(merkle[0] + merkle[1]) 367 | if merkle[0] == target_hash: 368 | s.append(hash_encode(merkle[1])) 369 | target_hash = new_hash 370 | elif merkle[1] == target_hash: 371 | s.append(hash_encode(merkle[0])) 372 | target_hash = new_hash 373 | n.append(new_hash) 374 | merkle = merkle[2:] 375 | merkle = n 376 | 377 | out = {"block_height": height, "merkle": s, "pos": tx_pos} 378 | with self.cache_lock: 379 | if len(self.merkle_cache) > self.max_cache_size: 380 | logger.info("clearing merkle cache") 381 | self.merkle_cache.clear() 382 | self.merkle_cache[tx_hash] = out 383 | return out 384 | 385 | @staticmethod 386 | def deserialize_block(block): 387 | txlist = block.get('tx') 388 | tx_hashes = [] # ordered txids 389 | txdict = {} # deserialized tx 390 | is_coinbase = True 391 | for raw_tx in txlist: 392 | tx_hash = hash_encode(Hash(raw_tx.decode('hex'))) 393 | vds = deserialize.BCDataStream() 394 | vds.write(raw_tx.decode('hex')) 395 | try: 396 | tx = deserialize.parse_Transaction(vds, is_coinbase) 397 | except: 398 | print_log("ERROR: cannot parse", tx_hash) 399 | continue 400 | tx_hashes.append(tx_hash) 401 | txdict[tx_hash] = tx 402 | is_coinbase = False 403 | return tx_hashes, txdict 404 | 405 | 406 | 407 | def import_block(self, block, block_hash, block_height, revert=False): 408 | 409 | touched_addr = set() 410 | 411 | # deserialize transactions 412 | tx_hashes, txdict = self.deserialize_block(block) 413 | 414 | # undo info 415 | if revert: 416 | undo_info = self.storage.get_undo_info(block_height) 417 | tx_hashes.reverse() 418 | else: 419 | undo_info = {} 420 | 421 | for txid in tx_hashes: # must be ordered 422 | tx = txdict[txid] 423 | if not revert: 424 | undo = self.storage.import_transaction(txid, tx, block_height, touched_addr) 425 | undo_info[txid] = undo 426 | else: 427 | undo = undo_info.pop(txid) 428 | self.storage.revert_transaction(txid, tx, block_height, touched_addr, undo) 429 | 430 | if revert: 431 | assert undo_info == {} 432 | 433 | # add undo info 434 | if not revert: 435 | self.storage.write_undo_info(block_height, self.dashd_height, undo_info) 436 | 437 | # add the max 438 | self.storage.save_height(block_hash, block_height) 439 | 440 | for addr in touched_addr: 441 | self.invalidate_cache(addr) 442 | 443 | self.storage.update_hashes() 444 | # batch write modified nodes 445 | self.storage.batch_write() 446 | # return length for monitoring 447 | return len(tx_hashes) 448 | 449 | 450 | def add_request(self, session, request): 451 | # see if we can get if from cache. if not, add request to queue 452 | message_id = request.get('id') 453 | try: 454 | result = self.process(request, cache_only=True) 455 | except BaseException as e: 456 | self.push_response(session, {'id': message_id, 'error': str(e)}) 457 | return 458 | 459 | if result == -1: 460 | self.queue.put((session, request)) 461 | else: 462 | self.push_response(session, {'id': message_id, 'result': result}) 463 | 464 | 465 | def do_subscribe(self, method, params, session): 466 | with self.watch_lock: 467 | if method == 'blockchain.numblocks.subscribe': 468 | if session not in self.watch_blocks: 469 | self.watch_blocks.append(session) 470 | 471 | elif method == 'blockchain.headers.subscribe': 472 | if session not in self.watch_headers: 473 | self.watch_headers.append(session) 474 | 475 | elif method == 'blockchain.address.subscribe': 476 | address = params[0] 477 | l = self.watched_addresses.get(address) 478 | if l is None: 479 | self.watched_addresses[address] = [session] 480 | elif session not in l: 481 | l.append(session) 482 | 483 | 484 | def do_unsubscribe(self, method, params, session): 485 | with self.watch_lock: 486 | if method == 'blockchain.numblocks.subscribe': 487 | if session in self.watch_blocks: 488 | self.watch_blocks.remove(session) 489 | elif method == 'blockchain.headers.subscribe': 490 | if session in self.watch_headers: 491 | self.watch_headers.remove(session) 492 | elif method == "blockchain.address.subscribe": 493 | addr = params[0] 494 | l = self.watched_addresses.get(addr) 495 | if not l: 496 | return 497 | if session in l: 498 | l.remove(session) 499 | if session in l: 500 | print_log("error rc!!") 501 | self.shared.stop() 502 | if l == []: 503 | del self.watched_addresses[addr] 504 | 505 | 506 | def process(self, request, cache_only=False): 507 | 508 | message_id = request['id'] 509 | method = request['method'] 510 | params = request.get('params', ()) 511 | result = None 512 | error = None 513 | 514 | if method == 'blockchain.numblocks.subscribe': 515 | result = self.storage.height 516 | 517 | elif method == 'blockchain.headers.subscribe': 518 | result = self.header 519 | 520 | elif method == 'blockchain.address.subscribe': 521 | address = str(params[0]) 522 | result = self.get_status(address, cache_only) 523 | 524 | elif method == 'blockchain.address.get_history': 525 | address = str(params[0]) 526 | result = self.get_history(address, cache_only) 527 | 528 | elif method == 'blockchain.address.get_mempool': 529 | address = str(params[0]) 530 | result = self.get_unconfirmed_history(address) 531 | 532 | elif method == 'blockchain.address.get_balance': 533 | address = str(params[0]) 534 | confirmed = self.storage.get_balance(address) 535 | unconfirmed = self.get_unconfirmed_value(address) 536 | result = { 'confirmed':confirmed, 'unconfirmed':unconfirmed } 537 | 538 | elif method == 'blockchain.address.get_proof': 539 | address = str(params[0]) 540 | result = self.storage.get_proof(address) 541 | 542 | elif method == 'blockchain.address.listunspent': 543 | address = str(params[0]) 544 | result = self.storage.listunspent(address) 545 | 546 | elif method == 'blockchain.utxo.get_address': 547 | txid = str(params[0]) 548 | pos = int(params[1]) 549 | txi = (txid + int_to_hex4(pos)).decode('hex') 550 | result = self.storage.get_address(txi) 551 | 552 | elif method == 'blockchain.block.get_header': 553 | if cache_only: 554 | result = -1 555 | else: 556 | height = int(params[0]) 557 | result = self.get_header(height) 558 | 559 | elif method == 'blockchain.block.get_chunk': 560 | if cache_only: 561 | result = -1 562 | else: 563 | index = int(params[0]) 564 | result = self.get_chunk(index) 565 | 566 | elif method == 'blockchain.transaction.broadcast': 567 | try: 568 | txo = self.dashd('sendrawtransaction', params) 569 | print_log("sent tx:", txo) 570 | result = txo 571 | except BaseException, e: 572 | error = e.args[0] 573 | if error["code"] == -26: 574 | # If we return anything that's not the transaction hash, 575 | # it's considered an error message 576 | message = error["message"] 577 | if "non-mandatory-script-verify-flag" in message: 578 | result = "Your client produced a transaction that is not accepted by the Bitcoin network any more. Please upgrade to Electrum 2.5.1 or newer\n" 579 | else: 580 | result = "The transaction was rejected by network rules.(" + message + ")\n" \ 581 | "[" + params[0] + "]" 582 | else: 583 | result = error["message"] # do send an error 584 | print_log("error:", result) 585 | 586 | elif method == 'blockchain.transaction.get_merkle': 587 | tx_hash = params[0] 588 | tx_height = params[1] 589 | result = self.get_merkle(tx_hash, tx_height, cache_only) 590 | 591 | elif method == 'blockchain.transaction.get': 592 | tx_hash = params[0] 593 | result = self.dashd('getrawtransaction', (tx_hash, 0)) 594 | 595 | elif method == 'blockchain.estimatefee': 596 | num = int(params[0]) 597 | result = self.dashd('estimatefee', (num,)) 598 | 599 | elif method == 'blockchain.relayfee': 600 | result = self.relayfee 601 | 602 | else: 603 | raise BaseException("unknown method:%s" % method) 604 | 605 | return result 606 | 607 | 608 | def get_block(self, block_hash): 609 | block = self.dashd('getblock', (block_hash,)) 610 | 611 | rawtxreq = [] 612 | i = 0 613 | for txid in block['tx']: 614 | rawtxreq.append({ 615 | "method": "getrawtransaction", 616 | "params": (txid,), 617 | "id": i, 618 | }) 619 | i += 1 620 | postdata = dumps(rawtxreq) 621 | 622 | while True: 623 | try: 624 | response = urllib.urlopen(self.dashd_url, postdata) 625 | r = load(response) 626 | response.close() 627 | except: 628 | logger.error("dashd error (getfullblock)") 629 | self.wait_on_dashd() 630 | continue 631 | try: 632 | rawtxdata = [] 633 | for ir in r: 634 | assert ir['error'] is None, "Error: make sure you run dashd with txindex=1; use -reindex if needed." 635 | rawtxdata.append(ir['result']) 636 | except BaseException as e: 637 | logger.error(str(e)) 638 | self.wait_on_dashd() 639 | continue 640 | 641 | block['tx'] = rawtxdata 642 | return block 643 | 644 | 645 | 646 | def catch_up(self, sync=True): 647 | 648 | self.start_catchup_height = self.storage.height 649 | prev_root_hash = None 650 | n = 0 651 | 652 | while not self.shared.stopped(): 653 | # are we done yet? 654 | info = self.dashd('getinfo') 655 | self.relayfee = info.get('relayfee') 656 | self.dashd_height = info.get('blocks') 657 | dashd_block_hash = self.dashd('getblockhash', (self.dashd_height,)) 658 | if self.storage.last_hash == dashd_block_hash: 659 | self.up_to_date = True 660 | break 661 | 662 | self.set_time() 663 | 664 | revert = (random.randint(1, 100) == 1) if self.test_reorgs and self.storage.height>100 else False 665 | 666 | # not done.. 667 | self.up_to_date = False 668 | try: 669 | next_block_hash = self.dashd('getblockhash', (self.storage.height + 1,)) 670 | except BaseException, e: 671 | revert = True 672 | 673 | next_block = self.get_block(next_block_hash if not revert else self.storage.last_hash) 674 | 675 | if (next_block.get('previousblockhash') == self.storage.last_hash) and not revert: 676 | 677 | prev_root_hash = self.storage.get_root_hash() 678 | 679 | n = self.import_block(next_block, next_block_hash, self.storage.height+1) 680 | self.storage.height = self.storage.height + 1 681 | self.write_header(self.block2header(next_block), sync) 682 | self.storage.last_hash = next_block_hash 683 | 684 | else: 685 | 686 | # revert current block 687 | block = self.get_block(self.storage.last_hash) 688 | print_log("blockchain reorg", self.storage.height, block.get('previousblockhash'), self.storage.last_hash) 689 | n = self.import_block(block, self.storage.last_hash, self.storage.height, revert=True) 690 | self.pop_header() 691 | self.flush_headers() 692 | 693 | self.storage.height -= 1 694 | 695 | # read previous header from disk 696 | self.header = self.read_header(self.storage.height) 697 | self.storage.last_hash = self.hash_header(self.header) 698 | 699 | if prev_root_hash: 700 | assert prev_root_hash == self.storage.get_root_hash() 701 | prev_root_hash = None 702 | 703 | # print time 704 | self.print_time(n) 705 | 706 | self.header = self.block2header(self.dashd('getblock', (self.storage.last_hash,))) 707 | self.header['utxo_root'] = self.storage.get_root_hash().encode('hex') 708 | 709 | if self.shared.stopped(): 710 | print_log( "closing database" ) 711 | self.storage.close() 712 | 713 | 714 | def memorypool_update(self): 715 | t0 = time.time() 716 | mempool_hashes = set(self.dashd('getrawmempool')) 717 | touched_addresses = set() 718 | 719 | # get new transactions 720 | new_tx = {} 721 | for tx_hash in mempool_hashes: 722 | if tx_hash in self.mempool_hashes: 723 | continue 724 | 725 | tx = self.get_mempool_transaction(tx_hash) 726 | if not tx: 727 | continue 728 | 729 | new_tx[tx_hash] = tx 730 | 731 | # remove older entries from mempool_hashes 732 | self.mempool_hashes = mempool_hashes 733 | 734 | # check all tx outputs 735 | for tx_hash, tx in new_tx.iteritems(): 736 | mpa = self.mempool_addresses.get(tx_hash, {}) 737 | out_values = [] 738 | out_sum = 0 739 | for x in tx.get('outputs'): 740 | addr = x.get('address', '') 741 | value = x['value'] 742 | out_values.append((addr, value)) 743 | if not addr: 744 | continue 745 | v = mpa.get(addr, 0) 746 | v += value 747 | mpa[addr] = v 748 | touched_addresses.add(addr) 749 | out_sum += value 750 | 751 | self.mempool_fees[tx_hash] = -out_sum 752 | self.mempool_addresses[tx_hash] = mpa 753 | self.mempool_values[tx_hash] = out_values 754 | self.mempool_unconfirmed[tx_hash] = set() 755 | 756 | # check all inputs 757 | for tx_hash, tx in new_tx.iteritems(): 758 | mpa = self.mempool_addresses.get(tx_hash, {}) 759 | # are we spending unconfirmed inputs? 760 | input_sum = 0 761 | for x in tx.get('inputs'): 762 | prev_hash = x.get('prevout_hash') 763 | prev_n = x.get('prevout_n') 764 | mpv = self.mempool_values.get(prev_hash) 765 | if mpv: 766 | addr, value = mpv[prev_n] 767 | self.mempool_unconfirmed[tx_hash].add(prev_hash) 768 | else: 769 | txi = (prev_hash + int_to_hex4(prev_n)).decode('hex') 770 | try: 771 | addr = self.storage.get_address(txi) 772 | value = self.storage.get_utxo_value(addr,txi) 773 | except: 774 | print_log("utxo not in database; postponing mempool update") 775 | return 776 | # we can proceed 777 | input_sum += value 778 | if not addr: 779 | continue 780 | v = mpa.get(addr, 0) 781 | v -= value 782 | mpa[addr] = v 783 | touched_addresses.add(addr) 784 | self.mempool_addresses[tx_hash] = mpa 785 | self.mempool_fees[tx_hash] += input_sum 786 | 787 | # remove deprecated entries from mempool_addresses 788 | for tx_hash, addresses in self.mempool_addresses.items(): 789 | if tx_hash not in self.mempool_hashes: 790 | del self.mempool_addresses[tx_hash] 791 | del self.mempool_values[tx_hash] 792 | del self.mempool_unconfirmed[tx_hash] 793 | del self.mempool_fees[tx_hash] 794 | touched_addresses.update(addresses) 795 | 796 | # remove deprecated entries from mempool_hist 797 | new_mempool_hist = {} 798 | for addr in self.mempool_hist.iterkeys(): 799 | h = self.mempool_hist[addr] 800 | hh = [] 801 | for tx_hash, delta in h: 802 | if tx_hash in self.mempool_addresses: 803 | hh.append((tx_hash, delta)) 804 | if hh: 805 | new_mempool_hist[addr] = hh 806 | # add new transactions to mempool_hist 807 | for tx_hash in new_tx.iterkeys(): 808 | addresses = self.mempool_addresses[tx_hash] 809 | for addr, delta in addresses.iteritems(): 810 | h = new_mempool_hist.get(addr, []) 811 | if (tx_hash, delta) not in h: 812 | h.append((tx_hash, delta)) 813 | new_mempool_hist[addr] = h 814 | 815 | with self.mempool_lock: 816 | self.mempool_hist = new_mempool_hist 817 | 818 | # invalidate cache for touched addresses 819 | for addr in touched_addresses: 820 | self.invalidate_cache(addr) 821 | 822 | t1 = time.time() 823 | if t1-t0>1: 824 | print_log('mempool_update', t1-t0, len(self.mempool_hashes), len(self.mempool_hist)) 825 | 826 | 827 | def invalidate_cache(self, address): 828 | with self.cache_lock: 829 | if address in self.history_cache: 830 | # print_log("cache: invalidating", address) 831 | del self.history_cache[address] 832 | 833 | with self.watch_lock: 834 | sessions = self.watched_addresses.get(address) 835 | 836 | if sessions: 837 | # TODO: update cache here. if new value equals cached value, do not send notification 838 | self.address_queue.put((address,sessions)) 839 | 840 | 841 | def close(self): 842 | self.blockchain_thread.join() 843 | print_log("Closing database...") 844 | self.storage.close() 845 | print_log("Database is closed") 846 | 847 | 848 | def main_iteration(self): 849 | if self.shared.stopped(): 850 | print_log("Stopping timer") 851 | return 852 | 853 | self.catch_up() 854 | 855 | self.memorypool_update() 856 | 857 | if self.sent_height != self.storage.height: 858 | self.sent_height = self.storage.height 859 | for session in self.watch_blocks: 860 | self.push_response(session, { 861 | 'id': None, 862 | 'method': 'blockchain.numblocks.subscribe', 863 | 'params': (self.storage.height,), 864 | }) 865 | 866 | if self.sent_header != self.header: 867 | self.sent_header = self.header 868 | for session in self.watch_headers: 869 | self.push_response(session, { 870 | 'id': None, 871 | 'method': 'blockchain.headers.subscribe', 872 | 'params': (self.header,), 873 | }) 874 | 875 | while True: 876 | try: 877 | addr, sessions = self.address_queue.get(False) 878 | except: 879 | break 880 | 881 | status = self.get_status(addr) 882 | for session in sessions: 883 | self.push_response(session, { 884 | 'id': None, 885 | 'method': 'blockchain.address.subscribe', 886 | 'params': (addr, status), 887 | }) 888 | 889 | 890 | -------------------------------------------------------------------------------- /src/deserialize.py: -------------------------------------------------------------------------------- 1 | # this code comes from ABE. it can probably be simplified 2 | # 3 | # 4 | 5 | import mmap 6 | import string 7 | import struct 8 | import types 9 | 10 | from utils import hash_160_to_pubkey_address, hash_160_to_script_address, public_key_to_pubkey_address, hash_encode,\ 11 | hash_160 12 | 13 | 14 | class SerializationError(Exception): 15 | """Thrown when there's a problem deserializing or serializing.""" 16 | 17 | 18 | class BCDataStream(object): 19 | """Workalike python implementation of Bitcoin's CDataStream class.""" 20 | def __init__(self): 21 | self.input = None 22 | self.read_cursor = 0 23 | 24 | def clear(self): 25 | self.input = None 26 | self.read_cursor = 0 27 | 28 | def write(self, bytes): # Initialize with string of bytes 29 | if self.input is None: 30 | self.input = bytes 31 | else: 32 | self.input += bytes 33 | 34 | def map_file(self, file, start): # Initialize with bytes from file 35 | self.input = mmap.mmap(file.fileno(), 0, access=mmap.ACCESS_READ) 36 | self.read_cursor = start 37 | 38 | def seek_file(self, position): 39 | self.read_cursor = position 40 | 41 | def close_file(self): 42 | self.input.close() 43 | 44 | def read_string(self): 45 | # Strings are encoded depending on length: 46 | # 0 to 252 : 1-byte-length followed by bytes (if any) 47 | # 253 to 65,535 : byte'253' 2-byte-length followed by bytes 48 | # 65,536 to 4,294,967,295 : byte '254' 4-byte-length followed by bytes 49 | # ... and the Bitcoin client is coded to understand: 50 | # greater than 4,294,967,295 : byte '255' 8-byte-length followed by bytes of string 51 | # ... but I don't think it actually handles any strings that big. 52 | if self.input is None: 53 | raise SerializationError("call write(bytes) before trying to deserialize") 54 | 55 | try: 56 | length = self.read_compact_size() 57 | except IndexError: 58 | raise SerializationError("attempt to read past end of buffer") 59 | 60 | return self.read_bytes(length) 61 | 62 | def write_string(self, string): 63 | # Length-encoded as with read-string 64 | self.write_compact_size(len(string)) 65 | self.write(string) 66 | 67 | def read_bytes(self, length): 68 | try: 69 | result = self.input[self.read_cursor:self.read_cursor+length] 70 | self.read_cursor += length 71 | return result 72 | except IndexError: 73 | raise SerializationError("attempt to read past end of buffer") 74 | 75 | return '' 76 | 77 | def read_boolean(self): 78 | return self.read_bytes(1)[0] != chr(0) 79 | 80 | def read_int16(self): 81 | return self._read_num(' len(bytes): 292 | vch = "_INVALID_"+bytes[i:] 293 | i = len(bytes) 294 | else: 295 | vch = bytes[i:i+nSize] 296 | i += nSize 297 | 298 | yield (opcode, vch, i) 299 | 300 | 301 | def script_GetOpName(opcode): 302 | try: 303 | return (opcodes.whatis(opcode)).replace("OP_", "") 304 | except KeyError: 305 | return "InvalidOp_"+str(opcode) 306 | 307 | 308 | def decode_script(bytes): 309 | result = '' 310 | for (opcode, vch, i) in script_GetOp(bytes): 311 | if len(result) > 0: 312 | result += " " 313 | if opcode <= opcodes.OP_PUSHDATA4: 314 | result += "%d:" % (opcode,) 315 | result += short_hex(vch) 316 | else: 317 | result += script_GetOpName(opcode) 318 | return result 319 | 320 | 321 | def match_decoded(decoded, to_match): 322 | if len(decoded) != len(to_match): 323 | return False 324 | for i in range(len(decoded)): 325 | if to_match[i] == opcodes.OP_PUSHDATA4 and decoded[i][0] <= opcodes.OP_PUSHDATA4: 326 | continue # Opcodes below OP_PUSHDATA4 all just push data onto stack, and are equivalent. 327 | if to_match[i] != decoded[i][0]: 328 | return False 329 | return True 330 | 331 | 332 | 333 | 334 | def get_address_from_output_script(bytes): 335 | try: 336 | decoded = [ x for x in script_GetOp(bytes) ] 337 | except: 338 | return None 339 | 340 | # The Genesis Block, self-payments, and pay-by-IP-address payments look like: 341 | # 65 BYTES:... CHECKSIG 342 | match = [opcodes.OP_PUSHDATA4, opcodes.OP_CHECKSIG] 343 | if match_decoded(decoded, match): 344 | return public_key_to_pubkey_address(decoded[0][1]) 345 | 346 | # coins sent to black hole 347 | # DUP HASH160 20 BYTES:... EQUALVERIFY CHECKSIG 348 | match = [opcodes.OP_DUP, opcodes.OP_HASH160, opcodes.OP_0, opcodes.OP_EQUALVERIFY, opcodes.OP_CHECKSIG] 349 | if match_decoded(decoded, match): 350 | return None 351 | 352 | # Pay-by-Bitcoin-address TxOuts look like: 353 | # DUP HASH160 20 BYTES:... EQUALVERIFY CHECKSIG 354 | match = [opcodes.OP_DUP, opcodes.OP_HASH160, opcodes.OP_PUSHDATA4, opcodes.OP_EQUALVERIFY, opcodes.OP_CHECKSIG] 355 | if match_decoded(decoded, match): 356 | return hash_160_to_pubkey_address(decoded[2][1]) 357 | 358 | # strange tx 359 | match = [opcodes.OP_DUP, opcodes.OP_HASH160, opcodes.OP_PUSHDATA4, opcodes.OP_EQUALVERIFY, opcodes.OP_CHECKSIG, opcodes.OP_NOP] 360 | if match_decoded(decoded, match): 361 | return hash_160_to_pubkey_address(decoded[2][1]) 362 | 363 | # p2sh 364 | match = [ opcodes.OP_HASH160, opcodes.OP_PUSHDATA4, opcodes.OP_EQUAL ] 365 | if match_decoded(decoded, match): 366 | addr = hash_160_to_script_address(decoded[1][1]) 367 | return addr 368 | 369 | return None 370 | -------------------------------------------------------------------------------- /src/ircthread.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # Copyright(C) 2011-2016 Thomas Voegtlin 3 | # 4 | # Permission is hereby granted, free of charge, to any person 5 | # obtaining a copy of this software and associated documentation files 6 | # (the "Software"), to deal in the Software without restriction, 7 | # including without limitation the rights to use, copy, modify, merge, 8 | # publish, distribute, sublicense, and/or sell copies of the Software, 9 | # and to permit persons to whom the Software is furnished to do so, 10 | # subject to the following conditions: 11 | # 12 | # The above copyright notice and this permission notice shall be 13 | # included in all copies or substantial portions of the Software. 14 | # 15 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 16 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 17 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 18 | # NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS 19 | # BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN 20 | # ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 21 | # CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 22 | # SOFTWARE. 23 | 24 | import re 25 | import time 26 | import socket 27 | import ssl 28 | import threading 29 | import Queue 30 | import irc.client 31 | from utils import logger 32 | from utils import Hash 33 | from version import VERSION 34 | 35 | out_msg = [] 36 | 37 | class IrcThread(threading.Thread): 38 | 39 | def __init__(self, processor, config): 40 | threading.Thread.__init__(self) 41 | self.processor = processor 42 | self.daemon = True 43 | options = dict(config.items('server')) 44 | self.stratum_tcp_port = options.get('stratum_tcp_port') 45 | self.stratum_tcp_ssl_port = options.get('stratum_tcp_ssl_port') 46 | self.report_stratum_tcp_port = options.get('report_stratum_tcp_port') 47 | self.report_stratum_tcp_ssl_port = options.get('report_stratum_tcp_ssl_port') 48 | self.irc_bind_ip = options.get('irc_bind_ip') 49 | self.host = options.get('host') 50 | self.report_host = options.get('report_host') 51 | self.nick = options.get('irc_nick') 52 | self.irc_prefix = options.get('irc_prefix') 53 | if self.report_stratum_tcp_port: 54 | self.stratum_tcp_port = self.report_stratum_tcp_port 55 | if self.report_stratum_tcp_ssl_port: 56 | self.stratum_tcp_ssl_port = self.report_stratum_tcp_ssl_port 57 | if self.report_host: 58 | self.host = self.report_host 59 | if not self.nick: 60 | self.nick = Hash(self.host)[:5].encode("hex") 61 | if not self.irc_prefix: 62 | self.irc_prefix = 'D_' 63 | self.pruning = True 64 | self.pruning_limit = config.get('leveldb', 'pruning_limit') 65 | self.nick = self.irc_prefix + self.nick 66 | self.password = None 67 | self.who_queue = Queue.Queue() 68 | 69 | def getname(self): 70 | s = 'v' + VERSION + ' ' 71 | if self.pruning: 72 | s += 'p' + self.pruning_limit + ' ' 73 | 74 | def add_port(letter, number): 75 | DEFAULT_PORTS = {'t':'50001', 's':'50002'} 76 | if not number: return '' 77 | if DEFAULT_PORTS[letter] == number: 78 | return letter + ' ' 79 | else: 80 | return letter + number + ' ' 81 | 82 | s += add_port('t',self.stratum_tcp_port) 83 | s += add_port('s',self.stratum_tcp_ssl_port) 84 | return s 85 | 86 | def start(self, queue): 87 | self.queue = queue 88 | threading.Thread.start(self) 89 | 90 | def on_connect(self, connection, event): 91 | connection.join("#electrum-dash") 92 | 93 | def on_join(self, connection, event): 94 | m = re.match("("+self.irc_prefix+".*)!", event.source) 95 | if m: 96 | self.who_queue.put((connection, m.group(1))) 97 | 98 | def on_quit(self, connection, event): 99 | m = re.match("("+self.irc_prefix+"..*)!", event.source) 100 | if m: 101 | self.queue.put(('quit', [m.group(1)])) 102 | 103 | def on_kick(self, connection, event): 104 | m = re.match("("+self.irc_prefix+"..*)", event.arguments[0]) 105 | if m: 106 | self.queue.put(('quit', [m.group(1)])) 107 | 108 | def on_disconnect(self, connection, event): 109 | logger.error("irc: disconnected") 110 | raise BaseException("disconnected") 111 | 112 | def on_who(self, connection, event): 113 | line = str(event.arguments[6]).split() 114 | try: 115 | ip = socket.gethostbyname(line[1]) 116 | except: 117 | # no IPv4 address could be resolved. Could be .onion or IPv6. 118 | ip = line[1] 119 | nick = event.arguments[4] 120 | host = line[1] 121 | ports = line[2:] 122 | self.queue.put(('join', [nick, ip, host, ports])) 123 | 124 | def on_name(self, connection, event): 125 | for s in event.arguments[2].split(): 126 | if s.startswith(self.irc_prefix): 127 | self.who_queue.put((connection, s)) 128 | 129 | def who_thread(self): 130 | while not self.processor.shared.stopped(): 131 | try: 132 | connection, s = self.who_queue.get(timeout=1) 133 | except Queue.Empty: 134 | continue 135 | #logger.info("who: "+ s) 136 | connection.who(s) 137 | time.sleep(1) 138 | 139 | def run(self): 140 | 141 | while self.processor.shared.paused(): 142 | time.sleep(1) 143 | 144 | self.ircname = self.host + ' ' + self.getname() 145 | # avoid UnicodeDecodeError using LenientDecodingLineBuffer 146 | irc.client.ServerConnection.buffer_class = irc.buffer.LenientDecodingLineBuffer 147 | logger.info("joining IRC") 148 | 149 | t = threading.Thread(target=self.who_thread) 150 | t.start() 151 | 152 | while not self.processor.shared.stopped(): 153 | client = irc.client.Reactor() 154 | try: 155 | #bind_address = (self.irc_bind_ip, 0) if self.irc_bind_ip else None 156 | #ssl_factory = irc.connection.Factory(wrapper=ssl.wrap_socket, bind_address=bind_address) 157 | #c = client.server().connect('irc.freenode.net', 6697, self.nick, self.password, ircname=self.ircname, connect_factory=ssl_factory) 158 | c = client.server().connect('irc.freenode.net', 6667, self.nick, self.password, ircname=self.ircname) 159 | except irc.client.ServerConnectionError: 160 | logger.error('irc', exc_info=True) 161 | time.sleep(10) 162 | continue 163 | 164 | c.add_global_handler("welcome", self.on_connect) 165 | c.add_global_handler("join", self.on_join) 166 | c.add_global_handler("quit", self.on_quit) 167 | c.add_global_handler("kick", self.on_kick) 168 | c.add_global_handler("whoreply", self.on_who) 169 | c.add_global_handler("namreply", self.on_name) 170 | c.add_global_handler("disconnect", self.on_disconnect) 171 | c.set_keepalive(60) 172 | 173 | self.connection = c 174 | try: 175 | client.process_forever() 176 | except BaseException as e: 177 | logger.error('irc', exc_info=True) 178 | time.sleep(10) 179 | continue 180 | 181 | logger.info("quitting IRC") 182 | -------------------------------------------------------------------------------- /src/networks.py: -------------------------------------------------------------------------------- 1 | # Main network and testnet3 definitions 2 | 3 | # Dash src/chainparams.cpp 4 | params = { 5 | 'dash_main': { 6 | 'pubkey_address': 76, #L120 7 | 'script_address': 16, #L122 8 | 'genesis_hash': '00000ffd590b1485b3caadc19b22e6379c733355108f107a430458cdf3407ab6' #L110 9 | }, 10 | 'dash_test': { 11 | 'pubkey_address': 140, #L220 12 | 'script_address': 19, #L222 13 | 'genesis_hash': '00000bafbc94add76cb75e2ec92894837288a481e5c005f6563d91623bf8bc2c' #L210 14 | } 15 | } 16 | -------------------------------------------------------------------------------- /src/processor.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # Copyright(C) 2011-2016 Thomas Voegtlin 3 | # 4 | # Permission is hereby granted, free of charge, to any person 5 | # obtaining a copy of this software and associated documentation files 6 | # (the "Software"), to deal in the Software without restriction, 7 | # including without limitation the rights to use, copy, modify, merge, 8 | # publish, distribute, sublicense, and/or sell copies of the Software, 9 | # and to permit persons to whom the Software is furnished to do so, 10 | # subject to the following conditions: 11 | # 12 | # The above copyright notice and this permission notice shall be 13 | # included in all copies or substantial portions of the Software. 14 | # 15 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 16 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 17 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 18 | # NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS 19 | # BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN 20 | # ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 21 | # CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 22 | # SOFTWARE. 23 | 24 | import json 25 | import Queue as queue 26 | import socket 27 | import threading 28 | import time 29 | import sys 30 | 31 | from utils import random_string, timestr, print_log 32 | from utils import logger 33 | 34 | class Shared: 35 | 36 | def __init__(self, config): 37 | self.lock = threading.Lock() 38 | self._stopped = False 39 | self.config = config 40 | self._paused = True 41 | 42 | def paused(self): 43 | with self.lock: 44 | return self._paused 45 | 46 | def pause(self): 47 | with self.lock: 48 | self._paused = True 49 | 50 | def unpause(self): 51 | with self.lock: 52 | self._paused = False 53 | 54 | def stop(self): 55 | print_log("Stopping Stratum") 56 | with self.lock: 57 | self._stopped = True 58 | 59 | def stopped(self): 60 | with self.lock: 61 | return self._stopped 62 | 63 | 64 | class Processor(threading.Thread): 65 | 66 | def __init__(self): 67 | threading.Thread.__init__(self) 68 | self.daemon = True 69 | self.dispatcher = None 70 | self.queue = queue.Queue() 71 | 72 | def process(self, request): 73 | pass 74 | 75 | def add_request(self, session, request): 76 | self.queue.put((session, request)) 77 | 78 | def push_response(self, session, response): 79 | #print "response", response 80 | self.dispatcher.request_dispatcher.push_response(session, response) 81 | 82 | def close(self): 83 | pass 84 | 85 | def run(self): 86 | while not self.shared.stopped(): 87 | try: 88 | session, request = self.queue.get(True, timeout=1) 89 | msg_id = request.get('id') 90 | except: 91 | continue 92 | if session.stopped(): 93 | continue 94 | try: 95 | result = self.process(request) 96 | self.push_response(session, {'id': msg_id, 'result': result}) 97 | except BaseException, e: 98 | self.push_response(session, {'id': msg_id, 'error':str(e)}) 99 | except: 100 | logger.error("process error", exc_info=True) 101 | self.push_response(session, {'id': msg_id, 'error':'unknown error'}) 102 | 103 | self.close() 104 | 105 | 106 | class Dispatcher: 107 | 108 | def __init__(self, config): 109 | self.shared = Shared(config) 110 | self.request_dispatcher = RequestDispatcher(self.shared) 111 | self.request_dispatcher.start() 112 | self.response_dispatcher = \ 113 | ResponseDispatcher(self.shared, self.request_dispatcher) 114 | self.response_dispatcher.start() 115 | 116 | def register(self, prefix, processor): 117 | processor.dispatcher = self 118 | processor.shared = self.shared 119 | processor.start() 120 | self.request_dispatcher.processors[prefix] = processor 121 | 122 | 123 | class RequestDispatcher(threading.Thread): 124 | 125 | def __init__(self, shared): 126 | self.shared = shared 127 | threading.Thread.__init__(self) 128 | self.daemon = True 129 | self.request_queue = queue.Queue() 130 | self.response_queue = queue.Queue() 131 | self.lock = threading.Lock() 132 | self.idlock = threading.Lock() 133 | self.sessions = {} 134 | self.processors = {} 135 | self.lastgc = 0 136 | 137 | def push_response(self, session, item): 138 | self.response_queue.put((session, item)) 139 | 140 | def pop_response(self): 141 | return self.response_queue.get() 142 | 143 | def push_request(self, session, item): 144 | self.request_queue.put((session, item)) 145 | 146 | def pop_request(self): 147 | return self.request_queue.get() 148 | 149 | def get_session_by_address(self, address): 150 | for x in self.sessions.values(): 151 | if x.address == address: 152 | return x 153 | 154 | def run(self): 155 | if self.shared is None: 156 | raise TypeError("self.shared not set in Processor") 157 | 158 | while not self.shared.stopped(): 159 | session, request = self.pop_request() 160 | try: 161 | self.do_dispatch(session, request) 162 | except: 163 | logger.error('dispatch',exc_info=True) 164 | 165 | self.stop() 166 | 167 | def stop(self): 168 | pass 169 | 170 | def do_dispatch(self, session, request): 171 | """ dispatch request to the relevant processor """ 172 | 173 | method = request['method'] 174 | params = request.get('params', []) 175 | suffix = method.split('.')[-1] 176 | 177 | if session is not None: 178 | if suffix == 'subscribe': 179 | if not session.subscribe_to_service(method, params): 180 | return 181 | 182 | prefix = request['method'].split('.')[0] 183 | try: 184 | p = self.processors[prefix] 185 | except: 186 | print_log("error: no processor for", prefix) 187 | return 188 | 189 | p.add_request(session, request) 190 | 191 | if method in ['server.version']: 192 | try: 193 | session.version = params[0] 194 | session.protocol_version = float(params[1]) 195 | except: 196 | pass 197 | 198 | def get_sessions(self): 199 | with self.lock: 200 | r = self.sessions.values() 201 | return r 202 | 203 | def add_session(self, session): 204 | key = session.key() 205 | with self.lock: 206 | self.sessions[key] = session 207 | 208 | def remove_session(self, session): 209 | key = session.key() 210 | with self.lock: 211 | del self.sessions[key] 212 | 213 | 214 | class Session: 215 | 216 | def __init__(self, dispatcher): 217 | self.dispatcher = dispatcher 218 | self.bp = self.dispatcher.processors['blockchain'] 219 | self._stopped = False 220 | self.lock = threading.Lock() 221 | self.subscriptions = [] 222 | self.address = '' 223 | self.name = '' 224 | self.version = 'unknown' 225 | self.protocol_version = 0. 226 | self.time = time.time() 227 | self.max_subscriptions = dispatcher.shared.config.getint('server', 'max_subscriptions') 228 | threading.Timer(2, self.info).start() 229 | 230 | 231 | def key(self): 232 | return self.address 233 | 234 | # Debugging method. Doesn't need to be threadsafe. 235 | def info(self): 236 | if self.subscriptions: 237 | print_log("%4s" % self.name, 238 | "%21s" % self.address, 239 | "%4d" % len(self.subscriptions), 240 | self.version) 241 | 242 | def stop(self): 243 | with self.lock: 244 | if self._stopped: 245 | return 246 | self._stopped = True 247 | 248 | self.shutdown() 249 | self.dispatcher.remove_session(self) 250 | self.stop_subscriptions() 251 | 252 | 253 | def shutdown(self): 254 | pass 255 | 256 | 257 | def stopped(self): 258 | with self.lock: 259 | return self._stopped 260 | 261 | 262 | def subscribe_to_service(self, method, params): 263 | if self.stopped(): 264 | return False 265 | 266 | if len(self.subscriptions) > self.max_subscriptions: 267 | print_log("max subscriptions reached", self.address) 268 | self.stop() 269 | return False 270 | 271 | # append to self.subscriptions only if this does not raise 272 | self.bp.do_subscribe(method, params, self) 273 | with self.lock: 274 | if (method, params) not in self.subscriptions: 275 | self.subscriptions.append((method,params)) 276 | return True 277 | 278 | 279 | def stop_subscriptions(self): 280 | with self.lock: 281 | s = self.subscriptions[:] 282 | for method, params in s: 283 | self.bp.do_unsubscribe(method, params, self) 284 | with self.lock: 285 | self.subscriptions = [] 286 | 287 | 288 | class ResponseDispatcher(threading.Thread): 289 | 290 | def __init__(self, shared, request_dispatcher): 291 | self.shared = shared 292 | self.request_dispatcher = request_dispatcher 293 | threading.Thread.__init__(self) 294 | self.daemon = True 295 | 296 | def run(self): 297 | while not self.shared.stopped(): 298 | session, response = self.request_dispatcher.pop_response() 299 | session.send_response(response) 300 | -------------------------------------------------------------------------------- /src/server_processor.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # Copyright(C) 2011-2016 Thomas Voegtlin 3 | # 4 | # Permission is hereby granted, free of charge, to any person 5 | # obtaining a copy of this software and associated documentation files 6 | # (the "Software"), to deal in the Software without restriction, 7 | # including without limitation the rights to use, copy, modify, merge, 8 | # publish, distribute, sublicense, and/or sell copies of the Software, 9 | # and to permit persons to whom the Software is furnished to do so, 10 | # subject to the following conditions: 11 | # 12 | # The above copyright notice and this permission notice shall be 13 | # included in all copies or substantial portions of the Software. 14 | # 15 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 16 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 17 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 18 | # NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS 19 | # BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN 20 | # ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 21 | # CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 22 | # SOFTWARE. 23 | 24 | import socket 25 | import sys 26 | import threading 27 | import time 28 | import Queue 29 | 30 | 31 | from processor import Processor 32 | from utils import Hash, print_log 33 | from version import VERSION 34 | from utils import logger 35 | from ircthread import IrcThread 36 | 37 | 38 | 39 | class ServerProcessor(Processor): 40 | 41 | def __init__(self, config, shared): 42 | Processor.__init__(self) 43 | self.daemon = True 44 | self.config = config 45 | self.shared = shared 46 | self.irc_queue = Queue.Queue() 47 | self.peers = {} 48 | 49 | if self.config.get('server', 'irc') == 'yes': 50 | self.irc = IrcThread(self, self.config) 51 | self.irc.start(self.irc_queue) 52 | t = threading.Thread(target=self.read_irc_results) 53 | t.daemon = True 54 | t.start() 55 | else: 56 | self.irc = None 57 | 58 | 59 | def read_irc_results(self): 60 | while True: 61 | try: 62 | event, params = self.irc_queue.get(timeout=1) 63 | except Queue.Empty: 64 | continue 65 | #logger.info(event + ' ' + repr(params)) 66 | if event == 'join': 67 | nick, ip, host, ports = params 68 | self.peers[nick] = (ip, host, ports) 69 | if event == 'quit': 70 | nick = params[0] 71 | if nick in self.peers: 72 | del self.peers[nick] 73 | 74 | 75 | def get_peers(self): 76 | return self.peers.values() 77 | 78 | 79 | def process(self, request): 80 | method = request['method'] 81 | params = request['params'] 82 | result = None 83 | 84 | if method == 'server.banner': 85 | result = self.config.get('server', 'banner').replace('\\n', '\n') 86 | 87 | elif method == 'server.donation_address': 88 | result = self.config.get('server', 'donation_address') 89 | 90 | elif method == 'server.peers.subscribe': 91 | result = self.get_peers() 92 | 93 | elif method == 'server.version': 94 | result = VERSION 95 | 96 | elif method == 'server.network': 97 | result = self.config.get('network', 'type') 98 | 99 | else: 100 | raise BaseException("unknown method: %s"%repr(method)) 101 | 102 | return result 103 | -------------------------------------------------------------------------------- /src/storage.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # Copyright(C) 2011-2016 Thomas Voegtlin 3 | # 4 | # Permission is hereby granted, free of charge, to any person 5 | # obtaining a copy of this software and associated documentation files 6 | # (the "Software"), to deal in the Software without restriction, 7 | # including without limitation the rights to use, copy, modify, merge, 8 | # publish, distribute, sublicense, and/or sell copies of the Software, 9 | # and to permit persons to whom the Software is furnished to do so, 10 | # subject to the following conditions: 11 | # 12 | # The above copyright notice and this permission notice shall be 13 | # included in all copies or substantial portions of the Software. 14 | # 15 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 16 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 17 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 18 | # NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS 19 | # BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN 20 | # ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 21 | # CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 22 | # SOFTWARE. 23 | 24 | import plyvel 25 | import ast 26 | import hashlib 27 | import os 28 | import sys 29 | import threading 30 | 31 | from processor import print_log, logger 32 | from utils import bc_address_to_hash_160, hash_160_to_pubkey_address, Hash, \ 33 | bytes8_to_int, bytes4_to_int, int_to_bytes8, \ 34 | int_to_hex8, int_to_bytes4, int_to_hex4 35 | 36 | 37 | """ 38 | Patricia tree for hashing unspents 39 | 40 | """ 41 | 42 | # increase this when database needs to be updated 43 | global GENESIS_HASH 44 | GENESIS_HASH = '000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f' 45 | DB_VERSION = 3 46 | KEYLENGTH = 56 # 20 + 32 + 4 47 | 48 | 49 | class Node(object): 50 | 51 | def __init__(self, s): 52 | self.k = int(s[0:32].encode('hex'), 16) 53 | self.s = s[32:] 54 | if self.k==0 and self.s: 55 | print "init error", len(self.s), "0x%0.64X" % self.k 56 | raise BaseException("z") 57 | 58 | def serialized(self): 59 | k = "0x%0.64X" % self.k 60 | k = k[2:].decode('hex') 61 | assert len(k) == 32 62 | return k + self.s 63 | 64 | def has(self, c): 65 | return (self.k & (1<<(ord(c)))) != 0 66 | 67 | def is_singleton(self, key): 68 | assert self.s != '' 69 | return len(self.s) == 40 70 | 71 | def get_singleton(self): 72 | for i in xrange(256): 73 | if self.k == (1< dashd_height - self.reorg_limit or self.test_reorgs: 341 | self.db_undo.put("undo_info_%d" % (height % self.reorg_limit), repr(undo_info)) 342 | 343 | @staticmethod 344 | def common_prefix(word1, word2): 345 | max_len = min(len(word1),len(word2)) 346 | for i in xrange(max_len): 347 | if word2[i] != word1[i]: 348 | index = i 349 | break 350 | else: 351 | index = max_len 352 | return word1[0:index] 353 | 354 | def put_node(self, key, node): 355 | self.db_utxo.put(key, node.serialized()) 356 | 357 | def get_node(self, key): 358 | s = self.db_utxo.get(key) 359 | if s is None: 360 | return 361 | return Node(s) 362 | 363 | def add_key(self, target, value, height): 364 | assert len(target) == KEYLENGTH 365 | path = self.get_path(target, new=True) 366 | if path is True: 367 | return 368 | #print "add key: target", target.encode('hex'), "path", map(lambda x: x.encode('hex'), path) 369 | parent = path[-1] 370 | parent_node = self.get_node(parent) 371 | n = len(parent) 372 | c = target[n] 373 | if parent_node.has(c): 374 | h, v = parent_node.get(c) 375 | skip = self.get_skip(parent + c) 376 | child = parent + c + skip 377 | assert not target.startswith(child) 378 | prefix = self.common_prefix(child, target) 379 | index = len(prefix) 380 | 381 | if len(child) == KEYLENGTH: 382 | # if it's a leaf, get hash and value of new_key from parent 383 | d = Node.from_dict({ 384 | target[index]: (None, 0), 385 | child[index]: (h, v) 386 | }) 387 | else: 388 | # if it is not a leaf, update its hash because skip_string changed 389 | child_node = self.get_node(child) 390 | h, v = child_node.get_hash(child, prefix) 391 | d = Node.from_dict({ 392 | target[index]: (None, 0), 393 | child[index]: (h, v) 394 | }) 395 | self.set_skip(prefix + target[index], target[index+1:]) 396 | self.set_skip(prefix + child[index], child[index+1:]) 397 | self.put_node(prefix, d) 398 | path.append(prefix) 399 | self.parents[child] = prefix 400 | 401 | # update parent skip 402 | new_skip = prefix[n+1:] 403 | self.set_skip(parent+c, new_skip) 404 | parent_node.set(c, None, 0) 405 | self.put_node(parent, parent_node) 406 | else: 407 | # add new letter to parent 408 | skip = target[n+1:] 409 | self.set_skip(parent+c, skip) 410 | parent_node.set(c, None, 0) 411 | self.put_node(parent, parent_node) 412 | 413 | # write the new leaf 414 | s = (int_to_hex8(value) + int_to_hex4(height)).decode('hex') 415 | self.db_utxo.put(target, s) 416 | # the hash of a leaf is the txid 417 | _hash = target[20:52] 418 | self.update_node_hash(target, path, _hash, value) 419 | 420 | 421 | def update_node_hash(self, node, path, _hash, value): 422 | c = node 423 | for x in path[::-1]: 424 | self.parents[c] = x 425 | c = x 426 | self.hash_list[node] = (_hash, value) 427 | 428 | 429 | def update_hashes(self): 430 | nodes = {} # nodes to write 431 | 432 | for i in xrange(KEYLENGTH, -1, -1): 433 | 434 | for node in self.hash_list.keys(): 435 | if len(node) != i: 436 | continue 437 | 438 | node_hash, node_value = self.hash_list.pop(node) 439 | 440 | parent = self.parents[node] if node!='' else '' 441 | 442 | if i != KEYLENGTH and node_hash is None: 443 | n = self.get_node(node) 444 | node_hash, node_value = n.get_hash(node, parent) 445 | assert node_hash is not None 446 | 447 | if node == '': 448 | self.root_hash = node_hash 449 | self.root_value = node_value 450 | assert self.root_hash is not None 451 | break 452 | 453 | # read parent 454 | d = nodes.get(parent) 455 | if d is None: 456 | d = self.get_node(parent) 457 | assert d is not None 458 | 459 | # write value into parent 460 | letter = node[len(parent)] 461 | d.set(letter, node_hash, node_value) 462 | nodes[parent] = d 463 | 464 | # iterate 465 | grandparent = self.parents[parent] if parent != '' else None 466 | parent_hash, parent_value = d.get_hash(parent, grandparent) 467 | if parent_hash is not None: 468 | self.hash_list[parent] = (parent_hash, parent_value) 469 | 470 | 471 | for k, v in nodes.iteritems(): 472 | self.put_node(k, v) 473 | # cleanup 474 | assert self.hash_list == {} 475 | self.parents = {} 476 | self.skip_batch = {} 477 | 478 | 479 | 480 | def get_path(self, target, new=False): 481 | 482 | x = self.db_utxo.get(target) 483 | if not new and x is None: 484 | raise BaseException('key not in tree', target.encode('hex')) 485 | 486 | if new and x is not None: 487 | # raise BaseException('key already in tree', target.encode('hex')) 488 | # occurs at block 91880 (duplicate txid) 489 | print_log('key already in tree', target.encode('hex')) 490 | return True 491 | 492 | remaining = target 493 | key = '' 494 | path = [] 495 | while key != target: 496 | node = self.get_node(key) 497 | if node is None: 498 | break 499 | #raise # should never happen 500 | path.append(key) 501 | c = remaining[0] 502 | if not node.has(c): 503 | break 504 | skip = self.get_skip(key + c) 505 | key = key + c + skip 506 | if not target.startswith(key): 507 | break 508 | remaining = target[len(key):] 509 | return path 510 | 511 | 512 | def delete_key(self, leaf): 513 | path = self.get_path(leaf) 514 | #print "delete key", leaf.encode('hex'), map(lambda x: x.encode('hex'), path) 515 | 516 | s = self.db_utxo.get(leaf) 517 | self.db_utxo.delete(leaf) 518 | 519 | if leaf in self.hash_list: 520 | del self.hash_list[leaf] 521 | 522 | parent = path[-1] 523 | letter = leaf[len(parent)] 524 | parent_node = self.get_node(parent) 525 | parent_node.remove(letter) 526 | 527 | # remove key if it has a single child 528 | if parent_node.is_singleton(parent): 529 | #print "deleting parent", parent.encode('hex') 530 | self.db_utxo.delete(parent) 531 | if parent in self.hash_list: 532 | del self.hash_list[parent] 533 | 534 | l = parent_node.get_singleton() 535 | _hash, value = parent_node.get(l) 536 | skip = self.get_skip(parent + l) 537 | otherleaf = parent + l + skip 538 | # update skip value in grand-parent 539 | gp = path[-2] 540 | gp_items = self.get_node(gp) 541 | letter = otherleaf[len(gp)] 542 | new_skip = otherleaf[len(gp)+1:] 543 | gp_items.set(letter, None, 0) 544 | self.set_skip(gp+ letter, new_skip) 545 | #print "gp new_skip", gp.encode('hex'), new_skip.encode('hex') 546 | self.put_node(gp, gp_items) 547 | 548 | # note: k is not necessarily a leaf 549 | if len(otherleaf) == KEYLENGTH: 550 | ss = self.db_utxo.get(otherleaf) 551 | _hash, value = otherleaf[20:52], bytes8_to_int(ss[0:8]) 552 | else: 553 | _hash, value = None, None 554 | self.update_node_hash(otherleaf, path[:-1], _hash, value) 555 | 556 | else: 557 | self.put_node(parent, parent_node) 558 | _hash, value = None, None 559 | self.update_node_hash(parent, path[:-1], _hash, value) 560 | return s 561 | 562 | def get_parent(self, x): 563 | p = self.get_path(x) 564 | return p[-1] 565 | 566 | def get_root_hash(self): 567 | return self.root_hash if self.root_hash else '' 568 | 569 | def batch_write(self): 570 | for db in [self.db_utxo, self.db_addr, self.db_hist, self.db_undo]: 571 | db.write() 572 | 573 | def close(self): 574 | for db in [self.db_utxo, self.db_addr, self.db_hist, self.db_undo]: 575 | db.close() 576 | 577 | def save_height(self, block_hash, block_height): 578 | self.db_undo.put('height', repr((block_hash, block_height, DB_VERSION))) 579 | 580 | def add_to_history(self, addr, tx_hash, tx_pos, value, tx_height): 581 | key = self.address_to_key(addr) 582 | txo = (tx_hash + int_to_hex4(tx_pos)).decode('hex') 583 | # write the new history 584 | self.add_key(key + txo, value, tx_height) 585 | # backlink 586 | self.db_addr.put(txo, addr) 587 | 588 | 589 | def revert_add_to_history(self, addr, tx_hash, tx_pos, value, tx_height): 590 | key = self.address_to_key(addr) 591 | txo = (tx_hash + int_to_hex4(tx_pos)).decode('hex') 592 | # delete 593 | self.delete_key(key + txo) 594 | # backlink 595 | self.db_addr.delete(txo) 596 | 597 | 598 | def get_utxo_value(self, addr, txi): 599 | key = self.address_to_key(addr) 600 | leaf = key + txi 601 | s = self.db_utxo.get(leaf) 602 | value = bytes8_to_int(s[0:8]) 603 | return value 604 | 605 | 606 | def set_spent(self, addr, txi, txid, index, height, undo): 607 | key = self.address_to_key(addr) 608 | leaf = key + txi 609 | s = self.delete_key(leaf) 610 | value = bytes8_to_int(s[0:8]) 611 | in_height = bytes4_to_int(s[8:12]) 612 | undo[leaf] = value, in_height 613 | # delete backlink txi-> addr 614 | self.db_addr.delete(txi) 615 | # add to history 616 | s = self.db_hist.get(addr) 617 | if s is None: s = '' 618 | txo = (txid + int_to_hex4(index) + int_to_hex4(height)).decode('hex') 619 | s += txi + int_to_bytes4(in_height) + txo 620 | s = s[ -80*self.pruning_limit:] 621 | self.db_hist.put(addr, s) 622 | 623 | 624 | 625 | def revert_set_spent(self, addr, txi, undo): 626 | key = self.address_to_key(addr) 627 | leaf = key + txi 628 | 629 | # restore backlink 630 | self.db_addr.put(txi, addr) 631 | 632 | v, height = undo.pop(leaf) 633 | self.add_key(leaf, v, height) 634 | 635 | # revert add to history 636 | s = self.db_hist.get(addr) 637 | # s might be empty if pruning limit was reached 638 | if not s: 639 | return 640 | 641 | assert s[-80:-44] == txi 642 | s = s[:-80] 643 | self.db_hist.put(addr, s) 644 | 645 | 646 | 647 | def import_transaction(self, txid, tx, block_height, touched_addr): 648 | 649 | undo = { 'prev_addr':[] } # contains the list of pruned items for each address in the tx; also, 'prev_addr' is a list of prev addresses 650 | 651 | prev_addr = [] 652 | for i, x in enumerate(tx.get('inputs')): 653 | txi = (x.get('prevout_hash') + int_to_hex4(x.get('prevout_n'))).decode('hex') 654 | addr = self.get_address(txi) 655 | if addr is not None: 656 | self.set_spent(addr, txi, txid, i, block_height, undo) 657 | touched_addr.add(addr) 658 | prev_addr.append(addr) 659 | 660 | undo['prev_addr'] = prev_addr 661 | 662 | # here I add only the outputs to history; maybe I want to add inputs too (that's in the other loop) 663 | for x in tx.get('outputs'): 664 | addr = x.get('address') 665 | if addr is None: continue 666 | self.add_to_history(addr, txid, x.get('index'), x.get('value'), block_height) 667 | touched_addr.add(addr) 668 | 669 | return undo 670 | 671 | 672 | def revert_transaction(self, txid, tx, block_height, touched_addr, undo): 673 | #print_log("revert tx", txid) 674 | for x in reversed(tx.get('outputs')): 675 | addr = x.get('address') 676 | if addr is None: continue 677 | self.revert_add_to_history(addr, txid, x.get('index'), x.get('value'), block_height) 678 | touched_addr.add(addr) 679 | 680 | prev_addr = undo.pop('prev_addr') 681 | for i, x in reversed(list(enumerate(tx.get('inputs')))): 682 | addr = prev_addr[i] 683 | if addr is not None: 684 | txi = (x.get('prevout_hash') + int_to_hex4(x.get('prevout_n'))).decode('hex') 685 | self.revert_set_spent(addr, txi, undo) 686 | touched_addr.add(addr) 687 | 688 | assert undo == {} 689 | -------------------------------------------------------------------------------- /src/stratum_tcp.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # Copyright(C) 2011-2016 Thomas Voegtlin 3 | # 4 | # Permission is hereby granted, free of charge, to any person 5 | # obtaining a copy of this software and associated documentation files 6 | # (the "Software"), to deal in the Software without restriction, 7 | # including without limitation the rights to use, copy, modify, merge, 8 | # publish, distribute, sublicense, and/or sell copies of the Software, 9 | # and to permit persons to whom the Software is furnished to do so, 10 | # subject to the following conditions: 11 | # 12 | # The above copyright notice and this permission notice shall be 13 | # included in all copies or substantial portions of the Software. 14 | # 15 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 16 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 17 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 18 | # NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS 19 | # BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN 20 | # ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 21 | # CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 22 | # SOFTWARE. 23 | 24 | import json 25 | import Queue as queue 26 | import socket 27 | import select 28 | import threading 29 | import time 30 | import sys 31 | 32 | from processor import Session, Dispatcher 33 | from utils import print_log, logger 34 | 35 | 36 | READ_ONLY = select.POLLIN | select.POLLPRI | select.POLLHUP | select.POLLERR 37 | READ_WRITE = READ_ONLY | select.POLLOUT 38 | WRITE_ONLY = select.POLLOUT 39 | TIMEOUT = 100 40 | 41 | import ssl 42 | 43 | class TcpSession(Session): 44 | 45 | def __init__(self, dispatcher, connection, address, use_ssl, ssl_certfile, ssl_keyfile): 46 | Session.__init__(self, dispatcher) 47 | self.use_ssl = use_ssl 48 | self.raw_connection = connection 49 | if use_ssl: 50 | import ssl 51 | self._connection = ssl.wrap_socket( 52 | connection, 53 | server_side=True, 54 | certfile=ssl_certfile, 55 | keyfile=ssl_keyfile, 56 | ssl_version=ssl.PROTOCOL_SSLv23, 57 | do_handshake_on_connect=False) 58 | else: 59 | self._connection = connection 60 | 61 | self.address = address[0] + ":%d"%address[1] 62 | self.name = "TCP " if not use_ssl else "SSL " 63 | self.timeout = 1000 64 | self.dispatcher.add_session(self) 65 | self.response_queue = queue.Queue() 66 | self.message = '' 67 | self.retry_msg = '' 68 | self.handshake = not self.use_ssl 69 | self.mode = None 70 | 71 | def connection(self): 72 | if self.stopped(): 73 | raise Exception("Session was stopped") 74 | else: 75 | return self._connection 76 | 77 | def shutdown(self): 78 | try: 79 | self._connection.shutdown(socket.SHUT_RDWR) 80 | except: 81 | # print_log("problem shutting down", self.address) 82 | pass 83 | self._connection.close() 84 | 85 | def send_response(self, response): 86 | try: 87 | msg = json.dumps(response) + '\n' 88 | except BaseException as e: 89 | logger.error('send_response:' + str(e)) 90 | return 91 | self.response_queue.put(msg) 92 | 93 | def parse_message(self): 94 | message = self.message 95 | self.time = time.time() 96 | raw_buffer = message.find('\n') 97 | if raw_buffer == -1: 98 | return False 99 | raw_command = message[0:raw_buffer].strip() 100 | self.message = message[raw_buffer + 1:] 101 | return raw_command 102 | 103 | 104 | 105 | 106 | 107 | class TcpServer(threading.Thread): 108 | 109 | def __init__(self, dispatcher, host, port, use_ssl, ssl_certfile, ssl_keyfile): 110 | self.shared = dispatcher.shared 111 | self.dispatcher = dispatcher.request_dispatcher 112 | threading.Thread.__init__(self) 113 | self.daemon = True 114 | self.host = host 115 | self.port = port 116 | self.lock = threading.Lock() 117 | self.use_ssl = use_ssl 118 | self.ssl_keyfile = ssl_keyfile 119 | self.ssl_certfile = ssl_certfile 120 | 121 | self.fd_to_session = {} 122 | self.buffer_size = 4096 123 | 124 | 125 | 126 | 127 | 128 | def handle_command(self, raw_command, session): 129 | try: 130 | command = json.loads(raw_command) 131 | except: 132 | session.send_response({"error": "bad JSON"}) 133 | return True 134 | try: 135 | # Try to load vital fields, and return an error if 136 | # unsuccessful. 137 | message_id = command['id'] 138 | method = command['method'] 139 | except: 140 | # Return an error JSON in response. 141 | session.send_response({"error": "syntax error", "request": raw_command}) 142 | else: 143 | #print_log("new request", command) 144 | self.dispatcher.push_request(session, command) 145 | 146 | 147 | 148 | def run(self): 149 | 150 | for res in socket.getaddrinfo(self.host, self.port, socket.AF_UNSPEC, socket.SOCK_STREAM): 151 | af, socktype, proto, cannonname, sa = res 152 | try: 153 | sock = socket.socket(af, socktype, proto) 154 | sock.setblocking(0) 155 | sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) 156 | except socket.error: 157 | sock = None 158 | continue 159 | try: 160 | sock.bind(sa) 161 | sock.listen(5) 162 | except socket.error: 163 | sock.close() 164 | sock = None 165 | continue 166 | break 167 | host = sa[0] 168 | if af == socket.AF_INET6: 169 | host = "[%s]" % host 170 | if sock is None: 171 | print_log( "could not open " + ("SSL" if self.use_ssl else "TCP") + " socket on %s:%d" % (host, self.port)) 172 | return 173 | print_log( ("SSL" if self.use_ssl else "TCP") + " server started on %s:%d" % (host, self.port)) 174 | 175 | sock_fd = sock.fileno() 176 | poller = select.poll() 177 | poller.register(sock) 178 | 179 | def stop_session(fd): 180 | try: 181 | # unregister before we close s 182 | poller.unregister(fd) 183 | session = self.fd_to_session.pop(fd) 184 | # this will close the socket 185 | session.stop() 186 | except BaseException as e: 187 | logger.error('stop_session error:' + str(e)) 188 | 189 | def check_do_handshake(session): 190 | if session.handshake: 191 | return 192 | try: 193 | session._connection.do_handshake() 194 | except ssl.SSLError as err: 195 | if err.args[0] == ssl.SSL_ERROR_WANT_READ: 196 | return 197 | elif err.args[0] == ssl.SSL_ERROR_WANT_WRITE: 198 | poller.modify(session.raw_connection, READ_WRITE) 199 | return 200 | else: 201 | raise BaseException(str(err)) 202 | poller.modify(session.raw_connection, READ_ONLY) 203 | session.handshake = True 204 | 205 | redo = [] 206 | 207 | while not self.shared.stopped(): 208 | 209 | if self.shared.paused(): 210 | sessions = self.fd_to_session.keys() 211 | if sessions: 212 | logger.info("closing %d sessions"%len(sessions)) 213 | for fd in sessions: 214 | stop_session(fd) 215 | time.sleep(1) 216 | continue 217 | 218 | if redo: 219 | events = redo 220 | redo = [] 221 | else: 222 | now = time.time() 223 | for fd, session in self.fd_to_session.items(): 224 | # Anti-DOS: wait 0.01 second between requests 225 | if now - session.time > 0.01 and session.message: 226 | cmd = session.parse_message() 227 | if not cmd: 228 | break 229 | if cmd == 'quit': 230 | data = False 231 | break 232 | session.time = now 233 | self.handle_command(cmd, session) 234 | 235 | # Anti-DOS: Stop reading if the session does not read responses 236 | if session.response_queue.empty(): 237 | mode = READ_ONLY 238 | elif session.response_queue.qsize() < 200: 239 | mode = READ_WRITE 240 | else: 241 | mode = WRITE_ONLY 242 | if mode != session.mode: 243 | poller.modify(session.raw_connection, mode) 244 | session.mode = mode 245 | 246 | # Collect garbage 247 | if now - session.time > session.timeout: 248 | stop_session(fd) 249 | 250 | events = poller.poll(TIMEOUT) 251 | 252 | for fd, flag in events: 253 | # open new session 254 | if fd == sock_fd: 255 | if flag & (select.POLLIN | select.POLLPRI): 256 | try: 257 | connection, address = sock.accept() 258 | session = TcpSession(self.dispatcher, connection, address, 259 | use_ssl=self.use_ssl, ssl_certfile=self.ssl_certfile, ssl_keyfile=self.ssl_keyfile) 260 | except BaseException as e: 261 | logger.error("cannot start TCP session" + str(e) + ' ' + repr(address)) 262 | connection.close() 263 | continue 264 | connection = session._connection 265 | connection.setblocking(False) 266 | self.fd_to_session[connection.fileno()] = session 267 | poller.register(connection, READ_ONLY) 268 | continue 269 | # existing session 270 | session = self.fd_to_session[fd] 271 | s = session._connection 272 | # non-blocking handshake 273 | try: 274 | check_do_handshake(session) 275 | except BaseException as e: 276 | #logger.error('handshake failure:' + str(e) + ' ' + repr(session.address)) 277 | stop_session(fd) 278 | continue 279 | # anti DOS 280 | now = time.time() 281 | if now - session.time < 0.01: 282 | continue 283 | # Read input messages. 284 | if flag & (select.POLLIN | select.POLLPRI): 285 | try: 286 | data = s.recv(self.buffer_size) 287 | except ssl.SSLError as x: 288 | if x.args[0] == ssl.SSL_ERROR_WANT_READ: 289 | pass 290 | elif x.args[0] == ssl.SSL_ERROR_SSL: 291 | pass 292 | else: 293 | logger.error('SSL recv error:'+ repr(x)) 294 | continue 295 | except socket.error as x: 296 | if x.args[0] != 104: 297 | logger.error('recv error: ' + repr(x) +' %d'%fd) 298 | stop_session(fd) 299 | continue 300 | except ValueError as e: 301 | logger.error('recv error: ' + str(e) +' %d'%fd) 302 | stop_session(fd) 303 | continue 304 | if data: 305 | session.message += data 306 | if len(data) == self.buffer_size: 307 | redo.append((fd, flag)) 308 | 309 | if not data: 310 | stop_session(fd) 311 | continue 312 | 313 | elif flag & select.POLLHUP: 314 | print_log('client hung up', session.address) 315 | stop_session(fd) 316 | 317 | elif flag & select.POLLOUT: 318 | # Socket is ready to send data, if there is any to send. 319 | if session.retry_msg: 320 | next_msg = session.retry_msg 321 | else: 322 | try: 323 | next_msg = session.response_queue.get_nowait() 324 | except queue.Empty: 325 | continue 326 | try: 327 | sent = s.send(next_msg) 328 | except socket.error as x: 329 | logger.error("send error:" + str(x)) 330 | stop_session(fd) 331 | continue 332 | session.retry_msg = next_msg[sent:] 333 | 334 | elif flag & select.POLLERR: 335 | print_log('handling exceptional condition for', session.address) 336 | stop_session(fd) 337 | 338 | elif flag & select.POLLNVAL: 339 | print_log('invalid request', session.address) 340 | stop_session(fd) 341 | 342 | 343 | print_log('TCP thread terminating', self.shared.stopped()) 344 | -------------------------------------------------------------------------------- /src/test/__init__.py: -------------------------------------------------------------------------------- 1 | __author__ = 'erasmospunk' 2 | -------------------------------------------------------------------------------- /src/test/test_utils.py: -------------------------------------------------------------------------------- 1 | __author__ = 'erasmospunk' 2 | 3 | import unittest 4 | from utils import hash_160_to_address, bc_address_to_hash_160 5 | 6 | 7 | class UtilTest(unittest.TestCase): 8 | 9 | def test_hash_160_to_address(self): 10 | self.assertEqual(hash_160_to_address(None), None) 11 | self.assertEqual(hash_160_to_address('04e9fca1'.decode('hex')), None) 12 | self.assertEqual(hash_160_to_address('04e9fca1f96e021dfaf35bbea267ec2c60787c1b1337'.decode('hex')), None) 13 | self.assertEqual(hash_160_to_address('1ad3b0b711f211655a01142fbb8fecabe8e30b93'.decode('hex')), 14 | '13SrAVFPVW1txSj34B8Bd6hnDbyPsVGa92') 15 | 16 | 17 | def test_bc_address_to_hash_160(self): 18 | self.assertEqual(bc_address_to_hash_160(None), None) 19 | self.assertEqual(bc_address_to_hash_160(''), None) 20 | self.assertEqual(bc_address_to_hash_160('13SrAVFPVW1txSj34B8Bd6hnDbyPsVGa921337'), None) 21 | self.assertEqual(bc_address_to_hash_160('13SrAVFPVW1txSj34B8Bd6hnDbyPsVGa92').encode('hex'), 22 | '1ad3b0b711f211655a01142fbb8fecabe8e30b93') 23 | 24 | 25 | 26 | if __name__ == '__main__': 27 | unittest.main() 28 | 29 | -------------------------------------------------------------------------------- /src/utils.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # Copyright(C) 2011-2016 Thomas Voegtlin 3 | # 4 | # Permission is hereby granted, free of charge, to any person 5 | # obtaining a copy of this software and associated documentation files 6 | # (the "Software"), to deal in the Software without restriction, 7 | # including without limitation the rights to use, copy, modify, merge, 8 | # publish, distribute, sublicense, and/or sell copies of the Software, 9 | # and to permit persons to whom the Software is furnished to do so, 10 | # subject to the following conditions: 11 | # 12 | # The above copyright notice and this permission notice shall be 13 | # included in all copies or substantial portions of the Software. 14 | # 15 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 16 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 17 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 18 | # NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS 19 | # BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN 20 | # ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 21 | # CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 22 | # SOFTWARE. 23 | 24 | from itertools import imap 25 | import threading 26 | import time 27 | import hashlib 28 | import struct 29 | import x11_hash 30 | 31 | __b58chars = '123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz' 32 | __b58base = len(__b58chars) 33 | 34 | global PUBKEY_ADDRESS 35 | global SCRIPT_ADDRESS 36 | PUBKEY_ADDRESS = 0 37 | SCRIPT_ADDRESS = 5 38 | 39 | def rev_hex(s): 40 | return s.decode('hex')[::-1].encode('hex') 41 | 42 | 43 | # Use Dash's X11 Hash Function 44 | HashX11 = lambda x: x11_hash.getPoWHash(x) 45 | 46 | Hash = lambda x: hashlib.sha256(hashlib.sha256(x).digest()).digest() 47 | 48 | 49 | hash_encode = lambda x: x[::-1].encode('hex') 50 | 51 | 52 | hash_decode = lambda x: x.decode('hex')[::-1] 53 | 54 | 55 | def header_to_string(res): 56 | pbh = res.get('prev_block_hash') 57 | if pbh is None: 58 | pbh = '0'*64 59 | 60 | return int_to_hex4(res.get('version')) \ 61 | + rev_hex(pbh) \ 62 | + rev_hex(res.get('merkle_root')) \ 63 | + int_to_hex4(int(res.get('timestamp'))) \ 64 | + int_to_hex4(int(res.get('bits'))) \ 65 | + int_to_hex4(int(res.get('nonce'))) 66 | 67 | 68 | _unpack_bytes4_to_int = struct.Struct("= __b58base: 168 | div, mod = divmod(long_value, __b58base) 169 | result = __b58chars[mod] + result 170 | long_value = div 171 | result = __b58chars[long_value] + result 172 | 173 | # Bitcoin does a little leading-zero-compression: 174 | # leading 0-bytes in the input become leading-1s 175 | nPad = 0 176 | for c in v: 177 | if c == '\0': 178 | nPad += 1 179 | else: 180 | break 181 | 182 | return (__b58chars[0]*nPad) + result 183 | 184 | 185 | def b58decode(v, length): 186 | """ decode v into a string of len bytes.""" 187 | long_value = 0L 188 | for (i, c) in enumerate(v[::-1]): 189 | long_value += __b58chars.find(c) * (__b58base**i) 190 | 191 | result = '' 192 | while long_value >= 256: 193 | div, mod = divmod(long_value, 256) 194 | result = chr(mod) + result 195 | long_value = div 196 | result = chr(long_value) + result 197 | 198 | nPad = 0 199 | for c in v: 200 | if c == __b58chars[0]: 201 | nPad += 1 202 | else: 203 | break 204 | 205 | result = chr(0)*nPad + result 206 | if length is not None and len(result) != length: 207 | return None 208 | 209 | return result 210 | 211 | 212 | def EncodeBase58Check(vchIn): 213 | hash = Hash(vchIn) 214 | return b58encode(vchIn + hash[0:4]) 215 | 216 | 217 | def DecodeBase58Check(psz): 218 | vchRet = b58decode(psz, None) 219 | key = vchRet[0:-4] 220 | csum = vchRet[-4:] 221 | hash = Hash(key) 222 | cs32 = hash[0:4] 223 | if cs32 != csum: 224 | return None 225 | else: 226 | return key 227 | 228 | 229 | 230 | 231 | ########### end pywallet functions ####################### 232 | import os 233 | 234 | def random_string(length): 235 | return b58encode(os.urandom(length)) 236 | 237 | def timestr(): 238 | return time.strftime("[%d/%m/%Y-%H:%M:%S]") 239 | 240 | 241 | 242 | ### logger 243 | import logging 244 | import logging.handlers 245 | 246 | logging.basicConfig(format="%(asctime)-11s %(message)s", datefmt="[%d/%m/%Y-%H:%M:%S]") 247 | logger = logging.getLogger('electrum') 248 | 249 | def init_logger(): 250 | logger.setLevel(logging.INFO) 251 | 252 | def print_log(*args): 253 | logger.info(" ".join(imap(str, args))) 254 | 255 | def print_warning(message): 256 | logger.warning(message) 257 | 258 | 259 | # profiler 260 | class ProfiledThread(threading.Thread): 261 | def __init__(self, filename, target): 262 | self.filename = filename 263 | threading.Thread.__init__(self, target = target) 264 | 265 | def run(self): 266 | import cProfile 267 | profiler = cProfile.Profile() 268 | profiler.enable() 269 | threading.Thread.run(self) 270 | profiler.disable() 271 | profiler.dump_stats(self.filename) 272 | -------------------------------------------------------------------------------- /src/version.py: -------------------------------------------------------------------------------- 1 | VERSION = "1.0" 2 | --------------------------------------------------------------------------------