├── .gitignore ├── CODE_OF_CONDUCT.md ├── LICENSE ├── README.md └── docs ├── files ├── Monitoring │ ├── glances_screenshot.png │ ├── log_all_octez_instances.csv │ └── pidstat_log_octez-node.csv └── failsafe_email.sh └── img ├── 1-choose_vps.png ├── 2-choose_comfort.png ├── 3-configure_vps.png ├── 4-choose_datacentre.png ├── 5-choose_billing_cycle.png ├── 6-complete_payment_process.png ├── LedgerLive_DeveloperMode.png ├── LedgerLive_TezosApps.png ├── Pyrometer_Dashboard.png ├── bootstrapped-sync-status.png ├── snapshot-import-output.png └── status-octez-node-service.png /.gitignore: -------------------------------------------------------------------------------- 1 | .env 2 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Creative Commons Legal Code 2 | 3 | CC0 1.0 Universal 4 | 5 | CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE 6 | LEGAL SERVICES. DISTRIBUTION OF THIS DOCUMENT DOES NOT CREATE AN 7 | ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS 8 | INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES 9 | REGARDING THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS 10 | PROVIDED HEREUNDER, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM 11 | THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED 12 | HEREUNDER. 13 | 14 | Statement of Purpose 15 | 16 | The laws of most jurisdictions throughout the world automatically confer 17 | exclusive Copyright and Related Rights (defined below) upon the creator 18 | and subsequent owner(s) (each and all, an "owner") of an original work of 19 | authorship and/or a database (each, a "Work"). 20 | 21 | Certain owners wish to permanently relinquish those rights to a Work for 22 | the purpose of contributing to a commons of creative, cultural and 23 | scientific works ("Commons") that the public can reliably and without fear 24 | of later claims of infringement build upon, modify, incorporate in other 25 | works, reuse and redistribute as freely as possible in any form whatsoever 26 | and for any purposes, including without limitation commercial purposes. 27 | These owners may contribute to the Commons to promote the ideal of a free 28 | culture and the further production of creative, cultural and scientific 29 | works, or to gain reputation or greater distribution for their Work in 30 | part through the use and efforts of others. 31 | 32 | For these and/or other purposes and motivations, and without any 33 | expectation of additional consideration or compensation, the person 34 | associating CC0 with a Work (the "Affirmer"), to the extent that he or she 35 | is an owner of Copyright and Related Rights in the Work, voluntarily 36 | elects to apply CC0 to the Work and publicly distribute the Work under its 37 | terms, with knowledge of his or her Copyright and Related Rights in the 38 | Work and the meaning and intended legal effect of CC0 on those rights. 39 | 40 | 1. Copyright and Related Rights. A Work made available under CC0 may be 41 | protected by copyright and related or neighboring rights ("Copyright and 42 | Related Rights"). Copyright and Related Rights include, but are not 43 | limited to, the following: 44 | 45 | i. the right to reproduce, adapt, distribute, perform, display, 46 | communicate, and translate a Work; 47 | ii. moral rights retained by the original author(s) and/or performer(s); 48 | iii. publicity and privacy rights pertaining to a person's image or 49 | likeness depicted in a Work; 50 | iv. rights protecting against unfair competition in regards to a Work, 51 | subject to the limitations in paragraph 4(a), below; 52 | v. rights protecting the extraction, dissemination, use and reuse of data 53 | in a Work; 54 | vi. database rights (such as those arising under Directive 96/9/EC of the 55 | European Parliament and of the Council of 11 March 1996 on the legal 56 | protection of databases, and under any national implementation 57 | thereof, including any amended or successor version of such 58 | directive); and 59 | vii. other similar, equivalent or corresponding rights throughout the 60 | world based on applicable law or treaty, and any national 61 | implementations thereof. 62 | 63 | 2. Waiver. To the greatest extent permitted by, but not in contravention 64 | of, applicable law, Affirmer hereby overtly, fully, permanently, 65 | irrevocably and unconditionally waives, abandons, and surrenders all of 66 | Affirmer's Copyright and Related Rights and associated claims and causes 67 | of action, whether now known or unknown (including existing as well as 68 | future claims and causes of action), in the Work (i) in all territories 69 | worldwide, (ii) for the maximum duration provided by applicable law or 70 | treaty (including future time extensions), (iii) in any current or future 71 | medium and for any number of copies, and (iv) for any purpose whatsoever, 72 | including without limitation commercial, advertising or promotional 73 | purposes (the "Waiver"). Affirmer makes the Waiver for the benefit of each 74 | member of the public at large and to the detriment of Affirmer's heirs and 75 | successors, fully intending that such Waiver shall not be subject to 76 | revocation, rescission, cancellation, termination, or any other legal or 77 | equitable action to disrupt the quiet enjoyment of the Work by the public 78 | as contemplated by Affirmer's express Statement of Purpose. 79 | 80 | 3. Public License Fallback. Should any part of the Waiver for any reason 81 | be judged legally invalid or ineffective under applicable law, then the 82 | Waiver shall be preserved to the maximum extent permitted taking into 83 | account Affirmer's express Statement of Purpose. In addition, to the 84 | extent the Waiver is so judged Affirmer hereby grants to each affected 85 | person a royalty-free, non transferable, non sublicensable, non exclusive, 86 | irrevocable and unconditional license to exercise Affirmer's Copyright and 87 | Related Rights in the Work (i) in all territories worldwide, (ii) for the 88 | maximum duration provided by applicable law or treaty (including future 89 | time extensions), (iii) in any current or future medium and for any number 90 | of copies, and (iv) for any purpose whatsoever, including without 91 | limitation commercial, advertising or promotional purposes (the 92 | "License"). The License shall be deemed effective as of the date CC0 was 93 | applied by Affirmer to the Work. Should any part of the License for any 94 | reason be judged legally invalid or ineffective under applicable law, such 95 | partial invalidity or ineffectiveness shall not invalidate the remainder 96 | of the License, and in such case Affirmer hereby affirms that he or she 97 | will not (i) exercise any of his or her remaining Copyright and Related 98 | Rights in the Work or (ii) assert any associated claims and causes of 99 | action with respect to the Work, in either case contrary to Affirmer's 100 | express Statement of Purpose. 101 | 102 | 4. Limitations and Disclaimers. 103 | 104 | a. No trademark or patent rights held by Affirmer are waived, abandoned, 105 | surrendered, licensed or otherwise affected by this document. 106 | b. Affirmer offers the Work as-is and makes no representations or 107 | warranties of any kind concerning the Work, express, implied, 108 | statutory or otherwise, including without limitation warranties of 109 | title, merchantability, fitness for a particular purpose, non 110 | infringement, or the absence of latent or other defects, accuracy, or 111 | the present or absence of errors, whether or not discoverable, all to 112 | the greatest extent permissible under applicable law. 113 | c. Affirmer disclaims responsibility for clearing rights of other persons 114 | that may apply to the Work or any use thereof, including without 115 | limitation any person's Copyright and Related Rights in the Work. 116 | Further, Affirmer disclaims responsibility for obtaining any necessary 117 | consents, permissions or other rights required for any use of the 118 | Work. 119 | d. Affirmer understands and acknowledges that Creative Commons is not a 120 | party to this document and has no duty or obligation with respect to 121 | this CC0 or use of the Work. 122 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | This project guides you through the process of setting up a Tezos validator node on a server from [OVH](https://www.ovh.com). Tezos is a self-amending blockchain network that incorporates a formal, on-chain mechanism for proposing, selecting, testing, and activating protocol upgrades. 2 | 3 | A Tezos validator node consists of several key components: 4 | 5 | - Tezos Node: The core component that connects to the Tezos network, synchronizes with the blockchain, and processes transactions and blocks. 6 | 7 | - Baker: A baker is responsible for creating and signing new blocks. It participates in the consensus mechanism by proposing blocks and validating blocks proposed by other bakers. 8 | 9 | - Accuser: The accuser monitors the network for any attempts at double-baking or double-endorsing, which are considered malicious activities. If detected, the accuser reports these actions, potentially resulting in the offending baker losing their stake. 10 | 11 | - Validator: In the context of Tezos, a validator combines the functions of a node, baker, and accuser. It fully participates in the network consensus, creates blocks, and helps maintain the integrity of the blockchain. 12 | 13 | 14 | # 1. Requirements 15 | 16 | ## 1.1 Hardware Setup 17 | 18 | - High CPU performance for transaction processing 19 | - Sufficient memory to handle blockchain data and operations 20 | - Fast and reliable storage with SSD 21 | - Good network performance 22 | 23 | In numbers: 24 | 25 | - 8 GB RAM 26 | - 2 CPU cores (better 4 vCPU) 27 | - Min. 256 GB SSD Drive 28 | - Linux (Docker optional) 29 | - Network min. 100 Mbps 30 | 31 | History types for a node: 32 | 33 | - **Rolling mode:** The most lightweight mode. It stores recent blocks with their current context. 34 | - **Full mode (default mode):** It also stores the content of every block since genesis so that it can handle requests about them, or even recompute the ledger state of every block of the chain. 35 | - **Archive mode:** Also stores the history of the context, allowing services like indexers to enquire about balances or staking rights from any past block. 36 | 37 | Network types: 38 | 39 | - Mainnet 40 | - Parisnet is the current testnet. 41 | - Ghostnet is a permanent testnet for devs or bakers. 42 | 43 | The node is intended for baking with no need to store content of every previous block. This is why the history type can be in **rolling** mode. Network type must be **mainnet**. 44 | 45 | ## 1.2 Buy OVH server 46 | 47 | 1. Got to [ovh.com](https://www.ovh.com) 48 | 49 | - Select _Virtual Private Server_. 50 | 51 | ![Step 1](/docs/img/1-choose_vps.png) 52 | 53 | 2. Choose Comfort Server 54 | 55 | - **8GB** RAM 56 | - 4 vCPU 57 | - 160GB SSD NVMe 58 | - 1Gbps network connectivity 59 | 60 | ![Step 2](/docs/img/2-choose_comfort.png) 61 | 62 | 3. Configure VPS 63 | 64 | - Choose **Debian 12** as Operating System (Latest Ubuntu 24.04 had a lot of problems). 65 | 66 | ![Step 3](/docs/img/3-configure_vps.png) 67 | 68 | - Choose datacentre 69 | 70 | ![Step 4](/docs/img/4-choose_datacentre.png) 71 | 72 | 4. Check order summary 73 | 74 | - Select **Payment up-front** to save costs. 75 | 76 | ![Step 5](/docs/img/5-choose_billing_cycle.png) 77 | 78 | 5. Complete payment process 79 | 80 | ![Step 6](/docs/img/6-complete_payment_process.png) 81 | 82 | ## 2. Server Setup and Security 83 | 84 | ### 2.1 Login to server 85 | 86 | ```bash 87 | ssh debian@server.public.ip.address 88 | ``` 89 | 90 | ### 2.2 Create new user with sudo privileges 91 | 92 | This creates a new user named tezos, sets the password and adds tezos user to sudo group: 93 | 94 | ```bash 95 | sudo useradd -m -s /bin/bash tezos 96 | sudo passwd tezos 97 | sudo usermod -aG sudo tezos 98 | ``` 99 | 100 | ### 2.3 Disable SSH password Authentication and Use SSH Keys only 101 | 102 | The basic rules of hardening SSH are: 103 | 104 | - No password for SSH access (use private key) 105 | - Don't allow root to SSH (the appropriate users should SSH in, then su or sudo) 106 | - Use sudo for users so commands are logged 107 | - Log unauthorized login attempts (and consider software to block/ban users who try to access your server too many times, like fail2ban) 108 | - Lock down SSH to only the ip range your require (if you feel like it) 109 | 110 | 1. Create a new ssh key locally. 111 | 112 | ```bash 113 | ssh-keygen -t ed25519 -f ~/.ssh/my_custom_key_name -C "comment or label for this key" 114 | ``` 115 | 116 | 2. Transfer the public key to your remote node. Update keyname.pub appropriately. 117 | 118 | ```bash 119 | ssh-copy-id -i $HOME/.ssh/keyname.pub debian@server.public.ip.address 120 | ``` 121 | 122 | 3. Login with your new user. 123 | 124 | ```bash 125 | ssh tezos@server.public.ip.address 126 | ``` 127 | 128 | 4. Disable root login and password based login. 129 | 130 | Open the SSH configuration file in a text editor: 131 | 132 | ```bash 133 | sudo nano /etc/ssh/sshd_config 134 | ``` 135 | 136 | 5. Update the following lines: 137 | 138 | ``` 139 | ChallengeResponseAuthentication no 140 | PasswordAuthentication no 141 | PermitRootLogin prohibit-password 142 | PermitEmptyPasswords no 143 | ``` 144 | 145 | 6. Validate the syntax of your new SSH configuration. 146 | 147 | ```bash 148 | sudo sshd -t 149 | ``` 150 | 151 | 7. If no errors with the syntax validation, restart the SSH process. 152 | 153 | ```bash 154 | sudo systemctl restart sshd 155 | ``` 156 | 157 | ### 2.4 Update your system 158 | 159 | 1. Update and upgrade packages: 160 | 161 | ```bash 162 | sudo apt update -y && sudo apt dist-upgrade -y 163 | sudo apt autoremove 164 | sudo apt autoclean 165 | ``` 166 | 167 | 2. Enable automatic updates: 168 | 169 | ```bash 170 | sudo apt install unattended-upgrades 171 | sudo dpkg-reconfigure -plow unattended-upgrades 172 | ``` 173 | 174 | ### 2.5 Disable root account 175 | 176 | 1. To disable the root account: 177 | 178 | ```bash 179 | sudo passwd -l root 180 | ``` 181 | 182 | 2. To re-enable the account if needed: 183 | 184 | ```bash 185 | sudo passwd -u root 186 | ``` 187 | 188 | ### 2.6 Secure Shared Memory 189 | 190 | 1. Edit /etc/fstab: 191 | 192 | ```bash 193 | sudo nano /etc/fstab 194 | ``` 195 | 196 | 2. Insert the following line to the bottom of the file: 197 | 198 | ``` 199 | tmpfs /run/shm tmpfs ro,noexec,nosuid 0 0 200 | ``` 201 | 202 | 3. Reboot the node: 203 | 204 | ```bash 205 | sudo reboot 206 | ``` 207 | 208 | 4. Check the changes: 209 | 210 | ```bash 211 | mount | grep /run/shm 212 | ``` 213 | 214 | ## 2.7 Install Fail2ban 215 | Fail2ban is an intrusion-prevention system that monitors log files and searches for particular patterns that correspond to a failed login attempt. If a certain number of failed logins are detected from a specific IP address (within a specified amount of time), fail2ban blocks access from that IP address. 216 | 217 | ```bash 218 | sudo apt-get install fail2ban -y 219 | ``` 220 | 221 | Edit a config file that monitors SSH logins. 222 | 223 | ```bash 224 | sudo nano /etc/fail2ban/jail.local 225 | ``` 226 | 227 | Add the following lines to the bottom of the file. 228 | **Note**: Whitelisting IP address tip: The ignoreip parameter accepts IP addresses, IP ranges or DNS hosts that you can specify to be allowed to connect. This is where you want to specify your local machine, local IP range or local domain, separated by spaces. 229 | 230 | ```ini 231 | [sshd] 232 | enabled = true 233 | port = 22 234 | filter = sshd 235 | logpath = /var/log/auth.log 236 | maxretry = 3 237 | # whitelisted IP addresses 238 | # ignoreip = 239 | ``` 240 | 241 | Save/close file. 242 | Restart fail2ban for settings to take effect. 243 | 244 | ```bash 245 | sudo systemctl restart fail2ban 246 | ``` 247 | 248 | ## 2.8 Configure your Firewall 249 | 250 | The standard UFW firewall can be used to control network access to your node. With any new installation, ufw is disabled by default. Enable it with the following settings. 251 | 252 | 1. Install UFW 253 | 254 | ```bash 255 | sudo apt install ufw 256 | ``` 257 | 258 | 2. Set default policies 259 | 260 | ```bash 261 | sudo ufw default deny incoming 262 | sudo ufw default allow outgoing 263 | ``` 264 | 265 | 3. Allow SSH access (adjust port if you've changed the default SSH port) 266 | 267 | ```bash 268 | sudo ufw allow ssh 269 | ``` 270 | 271 | 4. Allow Tezos node P2P and RPC connections 272 | 273 | ```bash 274 | sudo ufw allow 9732/tcp # P2P port for Tezos 275 | sudo ufw allow 8732/tcp # RPC port for Tezos 276 | ``` 277 | 278 | 5. Allow additional ports for monitoring (if needed) 279 | 280 | ```bash 281 | sudo ufw allow 2020/tcp # Pyrometer dashboard 282 | sudo ufw allow 9091/tcp # Metrics port 283 | ``` 284 | 285 | 6. Enable firewall 286 | 287 | ```bash 288 | sudo ufw enable 289 | ``` 290 | 291 | 7. Check status 292 | 293 | ```bash 294 | sudo ufw status verbose 295 | ``` 296 | 297 | 8. (Optional) Enable UFW logging 298 | 299 | ```bash 300 | sudo ufw logging on 301 | ``` 302 | 303 | Note: 304 | - The Tezos node needs port 9732 open for P2P connections to communicate with other nodes. 305 | - Ports 8732 and 8733 are for RPC, which should only be accessible from your IP for security. 306 | 307 | Always ensure your firewall rules are as restrictive as possible while still allowing necessary functionality. 308 | 309 | 310 | ## 2.9 Verify Listening Ports 311 | 312 | If you want to maintain a secure server, you should validate the listening network ports every once in a while. This will provide you essential information about your network. 313 | 314 | ```bash 315 | sudo ss -tulpn or sudo netstat -tulpn 316 | ``` 317 | 318 | ## 2.10 Time Sync Check 319 | 320 | Run the following command. 321 | 322 | ```bash 323 | timedatectl 324 | ``` 325 | 326 | ✅ Check if NTP Service is active. 327 | ✅ Check if Local time, Time zone, and Universal time are all correct. 328 | ✅ If NTP Service is not active, run: 329 | 330 | ```bash 331 | sudo timedatectl set-ntp on 332 | sudo timedatectl set-ntp true 333 | 334 | sudo timedatectl set-timezone Europe/Berlin 335 | ``` 336 | 337 | If you see error message Failed to set ntp: NTP not supported, you may need to install chrony or ntp package. 338 | 339 | >**Note** by default, VMs may disable NTP so you may need to find a work-around for your environment. 340 | 341 | 342 | # 3. Tezos Node Setup 343 | 344 | ## 3.1 Setup tezos node with Debian 345 | 346 | 347 | 1. Update and install packages. 348 | 349 | ```bash 350 | sudo apt update && sudo apt upgrade 351 | sudo apt install libev4 libhidapi-libusb0 curl net-tools 352 | ``` 353 | 354 | 2. Install octez packages. 355 | 356 | **Important**: Install in this order! Also there is a difference between using sudo in every command or using "sudo su" and then performing the statements. Keep that in mind. 357 | 358 | ```bash 359 | curl -o octez-node.deb https://pkgbeta.tzinit.org/debian-12/octez-node_20.1-1_amd64.deb 360 | curl -o octez-client.deb https://pkgbeta.tzinit.org/debian-12/octez-client_20.1-1_amd64.deb 361 | curl -o octez-baker.deb https://pkgbeta.tzinit.org/debian-12/octez-baker_20.1-1_amd64.deb 362 | 363 | sudo dpkg -i octez-node.deb 364 | sudo dpkg -i octez-client.deb 365 | sudo dpkg -i octez-baker.deb 366 | sudo apt-get install -f 367 | 368 | /etc/octez/node.conf 369 | 370 | sudo chown -R root:root /var/tezos 371 | sudo mkdir -p /var/log/tezos 372 | sudo chown -R root:root /var/log/tezos 373 | sudo chmod +x /usr/local/bin/octez-node 374 | ``` 375 | 376 | 3. Initialise configuration. 377 | 378 | ```bash 379 | octez-node config init --data-dir /var/tezos/.tezos-node \ 380 | --network=mainnet \ 381 | --history-mode=rolling \ 382 | --net-addr="[::]:9732" \ 383 | --rpc-addr="127.0.0.1:8732" \ 384 | --rpc-addr="0.0.0.0:8733" \ 385 | --connections=20 \ 386 | --metrics-addr="127.0.0.1:9091" 387 | ``` 388 | 389 | If you want to update the configuration use: 390 | 391 | ```bash 392 | octez-node config update --data-dir ... 393 | ``` 394 | 395 | Show configuration (Editing this manually did break it multiple times): 396 | 397 | ```bash 398 | cat /var/tezos/.tezos-node/config.json 399 | octez-node config show --config-file /var/tezos/.tezos-node/config.json 400 | ``` 401 | 402 | (Optional, did not work that well) - Allow only allowed peers: 403 | 404 | ```bash 405 | octez-node config update --data-dir /var/tezos/.tezos-node \ 406 | --peer= \ 407 | --private-mode \ 408 | --external-rpc-add=ADDR:PORT 409 | --config-file=FILE 410 | ``` 411 | 412 | 413 | ## 3.2 Get snapshot 414 | 415 | 1. Get and import the snapshot. 416 | 417 | ```bash 418 | wget -O /tmp/snap https://snapshots.eu.tzinit.org/mainnet/rolling 419 | octez-node snapshot import /tmp/snap --data-dir /var/tezos/.tezos-node --no-check 420 | ``` 421 | 422 | Output: 423 | 424 | ![Snapshot import Output](/docs/img/snapshot-import-output.png) 425 | 426 | If you get an error that the directory is invalid perform this: 427 | 428 | ```bash 429 | mv /var/tezos/.tezos-node/config.json /tmp/tezos_config_backup.json 430 | rm -rf /var/tezos/.tezos-node/* 431 | mv /tmp/tezos_config_backup.json /var/tezos/.tezos-node/config.json 432 | ``` 433 | 434 | 2. After importing the context from the snapshot delete tmp folder. 435 | 436 | ```bash 437 | rm /tmp/snap 438 | ``` 439 | 440 | ## 3.3 Create systemd service for octez node 441 | 442 | As root, start the node using systemctl. The enable command will ensure that the node starts on boot. 443 | Since we need to expose the RPC endpoint to the outside add another rpc-addr with `0.0.0.0:8733`. 444 | 445 | 1. Create systemd service file for the node. 446 | 447 | ```bash 448 | sudo nano /etc/systemd/system/octez-node.service 449 | ``` 450 | 451 | 2. Add this. 452 | 453 | ```ini 454 | [Unit] 455 | Description=Tezos Node 456 | Wants=network-online.target 457 | After=network-online.target 458 | 459 | [Service] 460 | Type=simple 461 | User=root 462 | Group=root 463 | ExecStart=/usr/bin/octez-node run --data-dir /var/tezos/.tezos-node --rpc-addr 0.0.0.0:8733 --rpc-addr 127.0.0.1:8732 --log-output /var/log/tezos/node.log --metrics-addr=127.0.0.1:9091 464 | Restart=on-failure 465 | 466 | [Install] 467 | WantedBy=multi-user.target 468 | RequiredBy=tezos-baker.service tezos-accuser.service 469 | ``` 470 | 471 | **NOTE**: To not overwrite the config values instead of this above use flag --config-file /var/tezos/.tezos-node/config.json in ExecStart 472 | 473 | 3. After any changes in systemd file, start or restart. 474 | 475 | ```bash 476 | #Option 1 477 | sudo systemctl daemon-reload 478 | systemctl enable octez-node.service 479 | systemctl start octez-node.service 480 | 481 | #Option 2 482 | sudo systemctl restart octez-node.service 483 | ``` 484 | 485 | 4. Check the status of the systemd service. 486 | 487 | ```bash 488 | sudo systemctl status octez-node.service 489 | ``` 490 | 491 | ![Status octez node systemd service](docs/img/status-octez-node-service.png) 492 | 493 | You can now view the progress of the node in the log file. It will sync with the network and fill the gap from the point that the snapshot was taken to the current block. Then you will have a working Tezos node. 494 | 495 | 5. Always see the logs for more information. 496 | 497 | ```bash 498 | sudo tail -f /var/log/tezos/node.log 499 | journalctl -f -u octez-node.service -b 500 | ``` 501 | 502 | 6. Check node synchronisation. 503 | 504 | ```bash 505 | octez-client bootstrapped 506 | ``` 507 | 508 | ![Octez Node bootstrapped status output](docs/img/bootstrapped-sync-status.png) 509 | 510 | 511 | Further Resources: [Tezos Node Setup](https://chrispinnock.com/tezos/node/) 512 | 513 | 514 | ## 3.4 Tezos Client Setup 515 | 516 | 1. Set base directory and configuration for tezos client. 517 | 518 | ```bash 519 | sudo mkdir -p /root/.tezos-client 520 | sudo chown -R $(whoami):$(whoami) /root/.tezos-client 521 | octez-client --base-dir /root/.tezos-client config init 522 | octez-client --base-dir /root/.tezos-client --endpoint http://localhost:8732 config update 523 | octez-client --base-dir /root/.tezos-client rpc get /chains/main/blocks/head 524 | ``` 525 | 526 | ## 3.5 Allow local connection 527 | 528 | To perform the commands to setup the ledger as remote signer on your local machine, the RPC endpoint must be accessible. 529 | 530 | 1. Check first if the node is fully synced. 531 | 532 | ```bash 533 | octez-client bootstrapped 534 | ``` 535 | 536 | 2. Update node config to allow access to monitoring status to outside via acl filters. (Otherwise this Error will appear: The server doesn't authorize this endpoint (ACL filtering)) 537 | 538 | ```bash 539 | octez-node config update \ 540 | --data-dir /var/tezos/.tezos-node \ 541 | --network=mainnet \ 542 | --history-mode=rolling \ 543 | --net-addr="[::]:9732" \ 544 | --rpc-addr="127.0.0.1:8732" \ 545 | --rpc-addr="0.0.0.0:8733" \ 546 | --connections=20 \ 547 | --metrics-addr="127.0.0.1:9091" \ 548 | --rpc-acl=[{address="0.0.0.0:8733",blacklist=[]}] 549 | ``` 550 | 551 | 3. Allow TCP access to your IP address via server settings. 552 | 553 | 554 | 555 | # 4. Keys 556 | 557 | In the Tezos blockchain there are two type of keys we use in our setup: 558 | 559 | - **Delegate** key: This key is the main key of a baker, used for managing funds and staking (delegation). This key controls the funds and is critical for the ownership of the staking balance. 560 | 561 | - **Consensus** Key: This is a separate key used for signing consensus operations like baking and endorsing blocks. It is a more specialized key used solely for participating in the consensus process. 562 | 563 | For enhanced security, we use a **Ledger** hardware wallet to protect the delegate key, ensuring that the main funds are kept safe from unauthorized access while keeping the consensus key flexible for baking operations. 564 | 565 | We will create a consensus key on the server, where the Tezos node and baker are running. This enhances security by separating the key used for baking and consensus operations from the main funds key (delegate key). Additionally, this allows consensus key *rotation* without affecting the delegate key and even if the consensus key is compromised, the main funds controlled by the delegate key remain secure. 566 | 567 | ## 4.1 Create consensus key 568 | 569 | 1. On the server create the consensus key and show the Secret. 570 | 571 | ```bash 572 | octez-client gen keys vpsconsensus 573 | octez-client show address vpsconsensus -S 574 | ``` 575 | 576 | Output: 577 | ``` 578 | Hash: tz1hq.... 579 | Public Key: edpkv9.... 580 | Secret Key: encrypted:edesk.... 581 | ``` 582 | 583 | 2. Save password 584 | 585 | Apart from saving the password in your preferred secure place store it on the server for later use in the baker. 586 | 587 | ```bash 588 | echo "" > ~/.tezos-client/pw 589 | ``` 590 | 591 | 592 | # 5. Local Ledger Setup 593 | 594 | ## 5.1 Install octez client 595 | 596 | Perform these steps on your local computer where you connect the Ledger. 597 | 598 | **Note**: Make sure to allow connections from your IP address to your server! 599 | 600 | 1. Install brew if not already done. 601 | 602 | ```bash 603 | /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" 604 | 605 | brew tap serokell/tezos-packaging-stable https://github.com/serokell/tezos-packaging-stable.git 606 | 607 | brew install tezos-client 608 | ``` 609 | 610 | 2. Verify that the Octez client is installed by running this command. 611 | 612 | ```bash 613 | octez-client --version 614 | ``` 615 | 616 | 3. Get the RPC node endpoint. is the servers IPv4 DNS address. 617 | 618 | ```bash 619 | telnet 8733 620 | curl http://:8733/chains/main/blocks/head/header 621 | ``` 622 | 623 | 4. Try this to check if node is running. 624 | 625 | Returns info about the latest block header: 626 | 627 | ```bash 628 | octez-client --endpoint http://:8733 bootstrapped 629 | curl -s http://:8733/chains/main/blocks/head/header 630 | octez-client --endpoint http://:8733 rpc get /chains/main/blocks/head/hash 631 | ``` 632 | 633 | 5. Configure and set rpc node as endpoint. 634 | 635 | ```bash 636 | octez-client config init 637 | octez-client --endpoint http://:8733 config update 638 | ``` 639 | 640 | (Additional debug): Test which ports are listening on your server: 641 | 642 | ```bash 643 | sudo netstat -tulnp | grep LISTEN 644 | sudo netstat -tulnp | grep :8732 645 | sudo netstat -tulnp | grep octez-node 646 | ``` 647 | 648 | 649 | After this is finished keep the ledger connected because we will locally create the delegate key, move to the server and create a consensus key for the baking process. Then locally again, import that key and register the consensus key together with that delegate key. 650 | 651 | 652 | ## 5.2 Register as a delegate 653 | 654 | 1. Connect and unlock your Ledger to your computer. 655 | 2. On Ledger Live: 656 | 657 | Activate Developer Mode in Experimental Settings. 658 | 659 | ![Ledger Live Developer Mode](./docs/img/LedgerLive_DeveloperMode.png) 660 | 661 | Install Tezos Wallet and Tezos Baker app. 662 | 663 | ![Ledger Live Tezos Baker Mode](./docs/img/LedgerLive_TezosApps.png) 664 | 665 | 3. Open the Tezos wallet app on the device (Must be open for all Ledger operations!) 666 | 667 | 4. Import the consensus key **locally**. 668 | 669 | ```bash 670 | octez-client --endpoint http://:8733 import secret key local_vpsconsensus encrypted:edesk1... 671 | ``` 672 | 673 | 5. List connected ledgers to get the path 674 | 675 | ```bash 676 | octez-client --endpoint http://:8733 list connected ledgers 677 | ``` 678 | 679 | Output looks like this: 680 | 681 | ``` 682 | Found a Tezos Wallet 3.0.4 (git-description: 683 | "stax_1.5.0_3.0.3_sdk_edba017dd479c0adeb115ee1cece6303d03c1610-60-g6e4b8ef1") 684 | application running on Ledger Nano S Plus at [DevSrvsID:4294973661]. 685 | 686 | To use keys at BIP32 path m/44'/1729'/0'/0' (default Tezos key path), use one 687 | of: 688 | octez-client import secret key ledger_softstack "ledger://front-chameleon-impossible-horse/ed25519/0h/0h" 689 | octez-client import secret key ledger_softstack "ledger://front-chameleon-impossible-horse/secp256k1/0h/0h" 690 | octez-client import secret key ledger_softstack "ledger://front-chameleon-impossible-horse/P-256/0h/0h" 691 | octez-client import secret key ledger_softstack "ledger://front-chameleon-impossible-horse/bip25519/0h/0h" 692 | ``` 693 | 694 | 6. IMPORTANT! Check your tezos wallet address in Ledger Live 695 | 696 | Open your account where your tezos funds are and check how the wallet address starts: Either tz1, tz2 or tz3. 697 | Depending on this, choose the correct command with the right cryptographic signing algorithm. 698 | 699 | - **ed25519** for **tz1...** addresses 700 | - **bip25519** for **tz1...** addresses as well. Newer addition to tezos. 701 | - **secp256k1** for **tz2...** addresses: This is less common but still regularly used. 702 | - **P-256** for **tz3...** addresses: This is the least common of the three original address types. 703 | 704 | For us, since we had a `tz1...` address we used below command: 705 | 706 | ```bash 707 | octez-client --endpoint http://:8733 import secret key ledger_softstack_delegate_key "ledger://front-chameleon-impossible-horse/ed25519/0h/0h" 708 | ``` 709 | 710 | This also imports the secret key with alias `ledger_softstack_delegate_key`. 711 | 712 | 7. Validate the public key hash on your connected ledger. 713 | 714 | 8. Register delegate with below command on your computer. 715 | 716 | ```bash 717 | octez-client --endpoint http://:8733 \ 718 | register key ledger_softstack_delegate_key as delegate 719 | ``` 720 | 721 | This shows that the funds from the connected wallet was used to register as delegate key and the kms consensus key is used for baking processes. 722 | 723 | 9. Stake your funds. must be at least 6000tz. Keep a few in your wallet for other operations. 724 | 725 | ```bash 726 | octez-client --endpoint http://:8733 stake for ledger_softstack_delegate_key 727 | ``` 728 | 729 | Now check if the balance of the wallet has these gas fees subtracted with next command: Check funds and baker rights 730 | 731 | 732 | 10. Set consensus key 733 | 734 | ```bash 735 | octez-client --endpoint http://:8733 set consensus key for ledger_softstack_delegate_key to local_vpsconsensus 736 | ``` 737 | 738 | Output: 739 | 740 | ```` 741 | Warning: 742 | This is NOT the Tezos Mainnet. 743 | Do NOT use your fundraiser keys on this network. 744 | Node is bootstrapped. 745 | Estimated gas: 169.046 units (will add 100 for safety) 746 | Estimated storage: no bytes added 747 | Enter password for encrypted key: 748 | Operation successfully injected in the node. 749 | Operation hash is 'owr9KJW2N87f1389ahUHAUWHbuaaHngsTAT8qqvwsDaiYicLDn7EYcSoKfw' 750 | Waiting for the operation to be included... 751 | Operation found in block: BMcLB82aihsd82aADWHyYojPHqavoHKc575J1qSdBrFx3GZfcrpvQM29wD (pass: 3, offset: 1) 752 | This sequence of operations was run: 753 | Manager signed operations: 754 | ... 755 | ```` 756 | 757 | 758 | 759 | 760 | 761 | # 6. Setup Baker 762 | 763 | ## 6.1 Creating a Dynamic Baker Start Script 764 | 765 | Creating a persistent baker using these [instructions](https://opentezos.com/node-baking/baking/persistent-baker/) from opentezos. Since the Tezos protocol changes every few months, we need to create a custom script to dynamically fetch the current protocol and start the baker accordingly. 766 | 767 | This guide provides instructions for setting up a Tezos baker on a Debian-based system using systemd. 768 | 769 | 1. Create a new file for the baker start script: 770 | 771 | ```bash 772 | sudo nano /usr/local/bin/tezos-baker-start 773 | ``` 774 | 775 | 2. Add the following content to the file: 776 | 777 | ```bash 778 | #!/bin/bash 779 | 780 | # Get the current protocol 781 | PROTOCOL=$(octez-client rpc get /chains/main/blocks/head/metadata | jq -r .protocol | sed -E 's/^(Pt|Ps)(.{6}).*/\1\2/') 782 | echo "$(date): Current protocol is $PROTOCOL" >> /var/log/tezos/baker_protocol.log 783 | 784 | # Start the baker with the current protocol 785 | exec /usr/bin/octez-baker-$PROTOCOL --mode client --base-dir /root/.tezos-client --endpoint http://127.0.0.1:8732 --password-filename /root/.tezos-client/pw run with local node /var/tezos/.tezos-node vpsconsensus --liquidity-baking-toggle-vote pass 786 | ``` 787 | 788 | 3. Make the script executable: 789 | 790 | ```bash 791 | sudo chmod +x /usr/local/bin/tezos-baker-start 792 | ``` 793 | 794 | 795 | ## 6.2 Configuring the Baker Service 796 | 797 | 1. Edit the baker service file: 798 | 799 | ```bash 800 | sudo nano /etc/systemd/system/octez-baker.service 801 | ``` 802 | 803 | 2. Add the following content to the file: 804 | 805 | ```ini 806 | [Unit] 807 | Description=Tezos baker Service 808 | Documentation=http://tezos.gitlab.io/ 809 | After=network-online.target octez-node.service 810 | Wants=network-online.target 811 | Requires=octez-node.service 812 | 813 | [Service] 814 | User=root 815 | Group=root 816 | Environment="TEZOS_LOG=* -> debug" 817 | ExecStart=/usr/local/bin/tezos-baker-start 818 | Restart=on-failure 819 | 820 | [Install] 821 | WantedBy=multi-user.target 822 | ``` 823 | 824 | ## 6.3 Starting and Managing the Baker Service 825 | 826 | 1. Stop the existing baker service (if running): 827 | 828 | ```bash 829 | sudo systemctl stop octez-baker.service 830 | ``` 831 | 832 | 2. Reload the systemd daemon to recognize the changes: 833 | 834 | ```bash 835 | sudo systemctl daemon-reload 836 | ``` 837 | 838 | 3. Enable the baker service to start on boot: 839 | 840 | ```bash 841 | sudo systemctl enable octez-baker.service 842 | ``` 843 | 844 | 4. Start the baker service: 845 | 846 | ```bash 847 | sudo systemctl start octez-baker.service 848 | ``` 849 | 850 | 5. Restart the baker service (if needed): 851 | 852 | ```bash 853 | sudo systemctl restart octez-baker.service 854 | ``` 855 | 856 | 6. Check the status of the baker service: 857 | 858 | ```bash 859 | sudo systemctl status octez-baker.service 860 | ``` 861 | 862 | 7. To monitor the baker's activity, you can use the following commands: 863 | 864 | ```bash 865 | journalctl -f -u octez-baker.service -b 866 | journalctl -f -u octez-baker.service -b > /var/log/baker-logs/16-09-24.log 867 | journalctl --unit=octez-baker.service --since "1 days ago" --until "now" > /var/log/baker-logs/$(date +%d-%m-%y).log 868 | ``` 869 | 870 | **Hint**: 871 | Since you always need to perform all of these commands at once, copy them from here: 872 | 873 | ```bash 874 | sudo systemctl stop octez-baker.service 875 | sudo systemctl daemon-reload 876 | sudo systemctl enable octez-baker.service 877 | sudo systemctl start octez-baker.service 878 | journalctl -f -u octez-baker.service -b 879 | ``` 880 | 881 | 8. Success! If you see below output in the baker logs you have a successfully running baker. 882 | 883 | Run `journalctl -f -u octez-baker.service -b`: 884 | 885 | 886 | ```` 887 | received 1 attestations (power: 127) (total voting power: 3790, 70 888 | attestations) 889 | received 1 attestations (power: 12) (total voting power: 3802, 71 890 | attestations) 891 | quorum reached (voting power: 4756, 72 attestations) 892 | automaton step: current phase awaiting attestations, event 893 | quorum reached with 72 attestations for BMBKoDiQY52KDLKdnZ9tUVGLMCF5fPqKtu3MofrKoceak1X8hY5 at round 0 894 | found an elected block at level 6813872, round 0... checking baking rights 895 | next potential slot for level 6813873 is at round 3006 at 896 | 2025-06-29T05:53:45-00:00 for 897 | vpsconsensus (tz1hq2SUp3jJa8uYa8xyZzAyfSb3rgG8GEHQ) 898 | on behalf of tz1SaRa7u2dnoJGKrWW8Qimjk5NwHC5E71A6 899 | waiting 7.657s until end of round 0 at 2024-10-10T12:36:30-00:00 900 | ```` 901 | 902 | Also check the `Rewards`, `Schedule` and `Consensus Key` Tabs on a tezos explorer like tzstats, or tzkt.io. Add your delegate key. 903 | 904 | ```` 905 | https://tzkt.io//schedule 906 | ```` 907 | 908 | # 7. Setup Node and Baker restart if they fail 909 | 910 | 1. Install Postmark. 911 | 912 | sudo apt-get install python3-pip 913 | pip3 install postmark 914 | 915 | 2. Create a shell script. 916 | 917 | ```bash 918 | sudo nano /usr/local/bin/tezos-node-monitor.sh 919 | ``` 920 | 921 | 3. Add contents from [Failsafe email script](./docs/files/failsafe_email.sh) script. 922 | 923 | 4. Make executable. 924 | 925 | ```bash 926 | sudo chmod +x /usr/local/bin/tezos-node-monitor.sh 927 | ``` 928 | 929 | 5. Add systemd service. 930 | 931 | ```bash 932 | sudo nano /etc/systemd/system/tezos-monitor.service 933 | ``` 934 | 935 | ```ini 936 | [Unit] 937 | Description=Tezos Node Monitor 938 | After=network.target 939 | 940 | [Service] 941 | ExecStart=/usr/local/bin/tezos-node-monitor.sh 942 | Restart=always 943 | User=root 944 | 945 | [Install] 946 | WantedBy=multi-user.target 947 | ``` 948 | 949 | 6. Reload systemd, enable and start the service. 950 | 951 | ```bash 952 | sudo systemctl daemon-reload 953 | sudo systemctl enable tezos-monitor.service 954 | sudo systemctl start tezos-monitor.service 955 | ``` 956 | 957 | Restart service: 958 | 959 | ```bash 960 | sudo systemctl restart tezos-monitor.service 961 | ``` 962 | 963 | See status and logs: 964 | 965 | ```bash 966 | sudo systemctl status tezos-monitor.service 967 | journalctl -f -u tezos-monitor.service -b 968 | tail -f /var/log/tezos-monitor.log 969 | ``` 970 | 971 | # 8. Setup accuser 972 | 973 | The accuser detects that a baker does not create two competing blocks at the same level nor attest multiple blocks. If an accusation is correct, the accuser gets part of the bakers reward. 974 | 975 | 1. Create a new file for the accuser start script: 976 | 977 | ```bash 978 | sudo nano /usr/local/bin/tezos-accuser-start 979 | ``` 980 | 981 | 2. Add the following content to the file: 982 | 983 | ```bash 984 | #!/bin/bash 985 | 986 | # Get the current protocol 987 | PROTOCOL=$(octez-client rpc get /chains/main/blocks/head/metadata | jq -r .protocol | sed -E 's/^(Pt|Ps)(.{6}).*/\1\2/') 988 | 989 | # Start the accuser with the current protocol 990 | exec /usr/bin/octez-accuser-$PROTOCOL --endpoint http://127.0.0.1:8732 run 991 | ``` 992 | 993 | 3. Make the script executable: 994 | 995 | ```bash 996 | sudo chmod +x /usr/local/bin/tezos-accuser-start 997 | ``` 998 | 999 | ## 8.1 Configuring the Accuser Service 1000 | 1001 | 4. Create the accuser service file: 1002 | 1003 | ```bash 1004 | sudo nano /etc/systemd/system/octez-accuser.service 1005 | ``` 1006 | 1007 | 5. Add the following content to the file: 1008 | 1009 | ```ini 1010 | [Unit] 1011 | Description=Tezos Accuser Service 1012 | Documentation=http://tezos.gitlab.io/ 1013 | After=network-online.target octez-node.service 1014 | Wants=network-online.target 1015 | Requires=octez-node.service 1016 | 1017 | [Service] 1018 | User=root 1019 | Group=root 1020 | Environment="TEZOS_LOG=* -> debug" 1021 | ExecStart=/usr/local/bin/tezos-accuser-start 1022 | Restart=on-failure 1023 | 1024 | [Install] 1025 | WantedBy=multi-user.target 1026 | ``` 1027 | 1028 | ## 8.2 Starting and Managing the Accuser Service 1029 | 1030 | 6. Reload the systemd daemon to recognize the new service: 1031 | 1032 | ```bash 1033 | sudo systemctl daemon-reload 1034 | ``` 1035 | 1036 | 7. Enable the accuser service to start on boot: 1037 | 1038 | ```bash 1039 | sudo systemctl enable octez-accuser.service 1040 | ``` 1041 | 1042 | 8. Start the accuser service: 1043 | 1044 | ```bash 1045 | sudo systemctl start octez-accuser.service 1046 | ``` 1047 | 1048 | 9. Check the status of the accuser service: 1049 | 1050 | ```bash 1051 | sudo systemctl status octez-accuser.service 1052 | ``` 1053 | 1054 | 10. If you need to restart the accuser service: 1055 | 1056 | ```bash 1057 | sudo systemctl restart octez-accuser.service 1058 | ``` 1059 | 1060 | 11. To view the accuser logs: 1061 | 1062 | ```bash 1063 | journalctl -f -u octez-accuser.service 1064 | ``` 1065 | 1066 | 1067 | # 9. Monitoring 1068 | 1069 | ## 9.1 Option 1: Using cli commands for monitoring 1070 | 1071 | Get PID for processes 1072 | 1073 | ```bash 1074 | pgrep -f octez-node 1075 | 9218 1076 | 9243 (validator) 1077 | 16736 baker 1078 | ``` 1079 | 1080 | ```bash 1081 | pidstat -u -r -p 9218,9243,16736 60 > /var/log/log_all_octez_instances.csv & 1082 | 16795 1083 | ``` 1084 | 1085 | Track network every min 1086 | 1087 | ```bash 1088 | sar -n DEV 60 1 > /var/log/sar_network.csv & 1089 | nohup sar -n DEV 60 > /var/log/network_usage.log 2>&1 & 1090 | 10136 1091 | ``` 1092 | 1093 | Results: 1094 | This usage is relatively low. 1095 | 89.52 KB/s received ≈ 0.7 Mbps 1096 | 54.91 KB/s sent ≈ 0.4 Mbps 1097 | 1098 | 1099 | Run in batch mode for continuous tracking (run every min) 1100 | 1101 | ```bash 1102 | glances --export csv --export-csv-file /var/log/glances.csv -B 60 --disable-plugin load,help,diskio,fs & 1103 | 15787 1104 | ``` 1105 | 1106 | ![Glances output](/docs/files/Monitoring/glances_screenshot.png) 1107 | 1108 | See background jobs 1109 | 1110 | ```bash 1111 | jobs 1112 | kill 1113 | ``` 1114 | 1115 | Find packages: 1116 | 1117 | ```bash 1118 | which octez-client 1119 | ``` 1120 | 1121 | ### 9.1.1 Monitoring results 1122 | 1123 | Octez node on average uses 60% (2.3Gi) of Memory(4Gi)... If the baker gets added this may not be enough. 1124 | 1125 | CPU Usage: 1126 | 1127 | - The octez-node process (PID 9218) consistently uses between 7.82% and 10.68% CPU, with slight variations over time. 1128 | 1129 | - The second octez-node process (PID 9243) uses significantly less CPU, ranging from 0.70% to 2.32%. 1130 | 1131 | - The octez-baker-PtP process (PID 16736) uses between 0.78% and 1.08% CPU. 1132 | 1133 | Memory Usage: 1134 | 1135 | - The octez-node process (PID 9218) has a consistent memory usage of around 26.44% of total memory. 1136 | 1137 | - The second octez-node process (PID 9243) uses around 23.35% of memory. 1138 | 1139 | - The octez-baker-PtP process (PID 16736) uses about 5.35% of memory, with a slight increase to 5.37% towards the end. 1140 | 1141 | ## 9.2 Tezos node monitoring with Pyrometer 1142 | 1143 | ### 9.2.1 Install pyrometer 1144 | 1145 | Resources: 1146 | - [Pyrometer Gitlab Steps](https://gitlab.com/tezos-kiln/pyrometer) 1147 | 1148 | 1149 | Install node 1150 | 1151 | ```bash 1152 | apt-get install -y curl 1153 | curl -fsSL https://deb.nodesource.com/setup_22.x -o nodesource_setup.sh 1154 | bash nodesource_setup.sh 1155 | apt-get install -y nodejs 1156 | node -v 1157 | ``` 1158 | 1159 | Download latest .deb from [Pyrometer releases](https://gitlab.com/tezos-kiln/pyrometer/-/releases). Rename version accordingly 1160 | 1161 | ```bash 1162 | wget -O pyrometer_0.10.1_all.deb https://gitlab.com/tezos-kiln/pyrometer/-/package_files/134721719/download 1163 | sudo dpkg -i pyrometer_0.10.1_all.deb 1164 | ``` 1165 | 1166 | Edit pyrometer settings in `/etc/pyrometer.toml` 1167 | 1168 | ```bash 1169 | sudo nano /etc/pyrometer.toml 1170 | ``` 1171 | 1172 | Add these settings: 1173 | 1174 | ```toml 1175 | [baker_monitor] 1176 | bakers = ["tz1..."] 1177 | rpc = "http://localhost:8732" 1178 | 1179 | [node_monitor] 1180 | nodes = [ "http://127.0.0.1:8732" ] 1181 | 1182 | [ui] 1183 | show_system_info = true 1184 | 1185 | [email] 1186 | enabled = true 1187 | host = "smtp.postmarkapp.com" 1188 | port = 587 1189 | protocol = "PLAIN" 1190 | to = [ "MY_EMAIL" ] 1191 | from = "Tezos Node Pyrometer " 1192 | username = "" 1193 | password = "" 1194 | emoji = true 1195 | short_address = true 1196 | exclude = [ ] 1197 | ``` 1198 | 1199 | If you want to get email notifications create an account at [postmark](https://postmarkapp.com/) and put in the for both username and password. 1200 | 1201 | The systemd for pyrometer can be found in `/lib/systemd/system/pyrometer.service` 1202 | 1203 | After editing the pyrometer.toml, restart and check logs 1204 | 1205 | ```bash 1206 | sudo systemctl restart pyrometer 1207 | journalctl -u pyrometer -f 1208 | ``` 1209 | 1210 | 1211 | ### 9.2.1 Pyrometer notifications event types 1212 | 1213 | 🤒 baker unhealthy 1214 | baker missed baker_monitor:missed_threshold events in a row 1215 | 1216 | 💪 baker recovered 1217 | baker successfully baked or endorsed after being unhealthy 1218 | 1219 | 😡 missed bake 1220 | baker was scheduled to produce a block at this level, but failed to 1221 | do so 1222 | 1223 | 😾 missed baking bonus 1224 | baker proposed a block payload, but failed to actually produce the 1225 | block itself, another baker did that and received the bonus 1226 | 1227 | 🥖 baked 1228 | baker produced a block as scheduled per baking rights 1229 | 1230 | ✂️️️️ double baked 1231 | baker produced two different blocks in the same round, baker's 1232 | deposit will be slashed as punishment 1233 | 1234 | 😕 missed endorsement 1235 | baker missed an endosement 1236 | 1237 | ‼️️ double endorsed 1238 | baker endorsed two different blocks in the same round 1239 | 1240 | ‼️️ double pre-endorsed 1241 | baker pre-endorsed two different blocks in the same round 1242 | 1243 | 🐌 node behind 1244 | a monitored node is not synchronized with the blockchain, is at a 1245 | lower block level then the other nodes in the network 1246 | 1247 | 💫 node synced 1248 | a monitored node has caught up with the blockchain after falling behind 1249 | 1250 | 🤔 low peers 1251 | a monitored node has fewer than node_monitor:low_peer_count 1252 | peers 1253 | 1254 | 🤝 low peers resolved 1255 | a monitored node connected to a sufficient number of 1256 | node_monitor:low_peer_count peers 1257 | 1258 | 😴 deactivated 1259 | baker has been deactivated 1260 | 1261 | 😪 deactivation risk 1262 | baker is at risk of deactivation (baker stopped participating and 1263 | will be deactivated once previously calculated baking right run out) 1264 | 1265 | ⚠️ rpc error 1266 | an error occurred while communicating with Tezos node RPC interface 1267 | 1268 | 1269 | ### 9.2.2 Pyrometer dashboard 1270 | 1271 | Visit `http://:2020` to see the Pyrometer dashboard. 1272 | 1273 | ![Pyrometer Dashboard](./docs/img/Pyrometer_Dashboard.png) 1274 | 1275 | ### 9.2.3 Other monitoring tools 1276 | 1277 | Grafana prometheus with a premade dashboard : [https://gitlab.com/nomadic-labs/grafazos](https://gitlab.com/nomadic-labs/grafazos) 1278 | 1279 | # 10. Maintenance 1280 | 1281 | ## 10.1 Regular System Maintenance 1282 | 1283 | 1. Update Debian OS: 1284 | - Run `sudo apt update && sudo apt upgrade` at least monthly. 1285 | - Reboot the system after significant updates. 1286 | 1287 | 2. Monitor and manage disk space: 1288 | - Regularly check available disk space with `df -h`. 1289 | - Set up log rotation for Tezos node logs if not already configured. 1290 | - Consider pruning old blockchain data if using full or archive node modes. 1291 | 1292 | 3. Check system resource usage: 1293 | - Monitor CPU, RAM, and network usage periodically. 1294 | - Use tools like `top`, `htop`, or your configured monitoring solution. 1295 | 1296 | ## 10.2 Tezos-specific Maintenance 1297 | 1298 | 4. Update Tezos software: 1299 | - Check for new Tezos releases every 1-2 months. 1300 | - Update octez-node, octez-client, and octez-baker when new versions are available. 1301 | - Test updates on a testnet node before applying to mainnet. 1302 | 1303 | 5. Monitor Tezos protocol upgrades: 1304 | - Stay informed about upcoming protocol upgrades (every few months). 1305 | - Ensure your node is compatible with new protocols before they activate. 1306 | - Update your baker and accuser scripts to use the new protocol after activation. 1307 | 1308 | 6. Check node synchronization: 1309 | - Regularly verify that your node is fully synced with the network. 1310 | - Use `octez-client bootstrapped` to check sync status. 1311 | 1312 | 7. Monitor baking and endorsing performance: 1313 | - Regularly check if you're missing any baking or endorsing slots. 1314 | - Investigate and resolve any missed opportunities promptly. 1315 | 1316 | 8. Review and optimize baker configuration: 1317 | - Periodically review your baker settings for optimal performance. 1318 | - Adjust liquidity baking strategy if needed. 1319 | 1320 | ## 10.3 Security Maintenance 1321 | 1322 | 9. Update firewall rules: 1323 | - Ensure only necessary ports are open. 1324 | 1325 | 10. Rotate access keys and update passwords: 1326 | - Regularly update SSH keys used to access the server. 1327 | - If using password authentication anywhere, change passwords periodically. 1328 | 1329 | 11. Review system logs: 1330 | - Check for any suspicious activities or error patterns in system logs. 1331 | - Pay special attention to authentication logs (`/var/log/auth.log`). 1332 | 1333 | 12. Update SSL certificates: 1334 | - If using SSL for RPC endpoints, ensure certificates are renewed before expiry. 1335 | 1336 | ## 10.4 Monitoring and Alerting 1337 | 1338 | 13. Check and update monitoring tools: 1339 | - Ensure Pyrometer or other monitoring solutions are up-to-date and functioning. 1340 | - Review and adjust alert thresholds if necessary. 1341 | 1342 | 14. Test alerting system: 1343 | - Periodically test that your alerting system (email, SMS, etc.) is working correctly. 1344 | 1345 | ## 10.5 Backup and Recovery 1346 | 1347 | 15. Perform regular backups: 1348 | - Backup important configuration files and keys. 1349 | - Consider periodic snapshots of the whole server. 1350 | 1351 | 16. Test recovery procedures: 1352 | - Occasionally test restoring from backups to ensure they're valid and the process works. 1353 | 1354 | ## 10.6 Network and Performance Optimization 1355 | 1356 | 17. Review and optimize network connections: 1357 | - Check the number and quality of peer connections. 1358 | - Optimize if necessary for better network performance. 1359 | 1360 | 18. Perform periodic performance tuning: 1361 | - Review and adjust system parameters for optimal Tezos node performance. 1362 | - This might include tweaking open file limits, network buffer sizes, etc. 1363 | 1364 | ## 10.7 Documentation and Knowledge Base 1365 | 1366 | 19. Update documentation: 1367 | - Keep your setup and maintenance documentation up-to-date. 1368 | - Document any changes made during maintenance for future reference. 1369 | 1370 | 20. Stay informed: 1371 | - Follow Tezos community updates, forums, and official announcements. 1372 | - Participate in discussions about network upgrades and best practices. 1373 | 1374 | ## 10.8 Financial and Regulatory Compliance 1375 | 1376 | 21. Review staking rewards and costs: 1377 | - Periodically review your baking rewards and associated costs. 1378 | - Adjust your strategy if necessary for optimal returns. 1379 | 1380 | 22. Stay compliant with regulations: 1381 | - Keep abreast of any regulatory changes affecting cryptocurrency staking in your jurisdiction. 1382 | - Ensure compliance with tax reporting requirements for staking rewards. 1383 | 1384 | Implement a routine schedule for these maintenance tasks, with some performed weekly, others monthly, and some quarterly or bi-annually as appropriate. Regular maintenance will help ensure the longevity, security, and efficiency of your Tezos validator node. 1385 | 1386 | 1387 | # 11. Updating octez version 1388 | 1389 | As of November 2024, v21.0 is released and all nodes need to update to that protocol manually by following below steps. Read the [release statement](https://tezos.gitlab.io/releases/version-21.html) for more information. 1390 | 1391 | 1392 | 1. Stop the baker and node 1393 | ```bash 1394 | sudo su 1395 | systemctl stop octez-baker.service 1396 | systemctl stop octez-node.service 1397 | ```` 1398 | 1399 | 2. Upgrade storage 1400 | ```bash 1401 | octez-node upgrade storage --data-dir /var/tezos/.tezos-node 1402 | #Output: Nov 26 07:54:22.840: node data dir is up-to-date 1403 | ``` 1404 | 1405 | 3. Restart node 1406 | ```bash 1407 | sudo systemctl daemon-reload 1408 | systemctl start octez-node.service 1409 | sudo systemctl status octez-node.service 1410 | # Wait a minute or two for the node to start before starting the baker 1411 | ``` 1412 | 1413 | 4. Restart baker 1414 | 1415 | ```bash 1416 | systemctl start octez-baker.service 1417 | sudo systemctl status octez-baker.service 1418 | # Output: Baker 20.1 (1a991a03) for PsParisCZo7K started. 1419 | ```` 1420 | 1421 | 5. Check logs if everything works 1422 | ```bash 1423 | journalctl -f -u octez-node.service -b 1424 | journalctl -f -u octez-baker.service -b 1425 | ``` 1426 | 1427 | # Congratulations 1428 | 1429 | You have successfully setup your Tezos validator node on OVH server. 1430 | 1431 | If any questions arise feel free to contact us at [softstack.io](https://softstack.io/). 1432 | -------------------------------------------------------------------------------- /docs/files/Monitoring/glances_screenshot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/softstack/tezos-octez-node-and-baking-setup-guide/427300c65d6f7ad8164e678552bfff94b79df102/docs/files/Monitoring/glances_screenshot.png -------------------------------------------------------------------------------- /docs/files/Monitoring/log_all_octez_instances.csv: -------------------------------------------------------------------------------- 1 | Linux 6.1.0-13-cloud-amd64 (ip-172-31-25-160) 06/21/24 _x86_64_ (2 CPU) 2 | 3 | 09:39:21 UID PID %usr %system %guest %wait %CPU CPU Command 4 | 09:40:21 0 9218 8.13 1.02 0.00 0.27 9.15 1 octez-node 5 | 09:40:21 0 9243 1.50 0.30 0.00 0.18 1.80 0 octez-node 6 | 09:40:21 0 16736 0.90 0.62 0.00 0.48 1.52 0 octez-baker-PtP 7 | 8 | 09:39:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 9 | 09:40:21 0 9218 0.00 0.00 1694140 1051668 26.44 octez-node 10 | 09:40:21 0 9243 0.00 0.00 1219200 928576 23.35 octez-node 11 | 09:40:21 0 16736 0.00 0.00 1023796 208644 5.25 octez-baker-PtP 12 | 13 | 09:40:21 UID PID %usr %system %guest %wait %CPU CPU Command 14 | 09:41:21 0 9218 9.17 1.25 0.00 0.33 10.42 0 octez-node 15 | 09:41:21 0 9243 0.82 0.05 0.00 0.02 0.87 0 octez-node 16 | 09:41:21 0 16736 0.95 0.77 0.00 0.52 1.72 0 octez-baker-PtP 17 | 18 | 09:40:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 19 | 09:41:21 0 9218 0.00 0.00 1694140 1051668 26.44 octez-node 20 | 09:41:21 0 9243 0.00 0.00 1219200 928576 23.35 octez-node 21 | 09:41:21 0 16736 1.52 0.00 1031992 209172 5.26 octez-baker-PtP 22 | 23 | 09:41:21 UID PID %usr %system %guest %wait %CPU CPU Command 24 | 09:42:21 0 9218 8.40 0.92 0.00 0.25 9.32 0 octez-node 25 | 09:42:21 0 9243 1.20 0.12 0.00 0.03 1.32 1 octez-node 26 | 09:42:21 0 16736 0.83 0.77 0.00 0.48 1.60 0 octez-baker-PtP 27 | 28 | 09:41:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 29 | 09:42:21 0 9218 0.00 0.00 1694140 1051668 26.44 octez-node 30 | 09:42:21 0 9243 0.00 0.00 1219200 928576 23.35 octez-node 31 | 09:42:21 0 16736 0.03 0.00 1097528 209172 5.26 octez-baker-PtP 32 | 33 | 09:42:21 UID PID %usr %system %guest %wait %CPU CPU Command 34 | 09:43:21 0 9218 8.58 1.03 0.00 0.32 9.62 0 octez-node 35 | 09:43:21 0 9243 0.97 0.05 0.00 0.02 1.02 0 octez-node 36 | 09:43:21 0 16736 0.97 0.72 0.00 0.57 1.68 1 octez-baker-PtP 37 | 38 | 09:42:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 39 | 09:43:21 0 9218 0.00 0.00 1694140 1051668 26.44 octez-node 40 | 09:43:21 0 9243 0.00 0.00 1219200 928576 23.35 octez-node 41 | 09:43:21 0 16736 0.10 0.00 1097528 209172 5.26 octez-baker-PtP 42 | 43 | 09:43:21 UID PID %usr %system %guest %wait %CPU CPU Command 44 | 09:44:21 0 9218 8.95 1.08 0.00 0.23 10.03 0 octez-node 45 | 09:44:21 0 9243 0.98 0.05 0.00 0.07 1.03 0 octez-node 46 | 09:44:21 0 16736 0.82 0.72 0.00 0.47 1.53 1 octez-baker-PtP 47 | 48 | 09:43:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 49 | 09:44:21 0 9218 0.00 0.00 1694140 1051668 26.44 octez-node 50 | 09:44:21 0 9243 0.00 0.00 1219200 928576 23.35 octez-node 51 | 09:44:21 0 16736 0.00 0.00 1097528 209172 5.26 octez-baker-PtP 52 | 53 | 09:44:21 UID PID %usr %system %guest %wait %CPU CPU Command 54 | 09:45:21 0 9218 8.37 1.02 0.00 0.22 9.38 1 octez-node 55 | 09:45:21 0 9243 1.08 0.10 0.00 0.03 1.18 0 octez-node 56 | 09:45:21 0 16736 0.95 0.62 0.00 0.45 1.57 0 octez-baker-PtP 57 | 58 | 09:44:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 59 | 09:45:21 0 9218 0.00 0.00 1694140 1051668 26.44 octez-node 60 | 09:45:21 0 9243 0.00 0.00 1219200 928576 23.35 octez-node 61 | 09:45:21 0 16736 0.00 0.00 1097528 209172 5.26 octez-baker-PtP 62 | 63 | 09:45:21 UID PID %usr %system %guest %wait %CPU CPU Command 64 | 09:46:21 0 9218 8.43 0.98 0.00 0.25 9.42 0 octez-node 65 | 09:46:21 0 9243 1.22 0.08 0.00 0.03 1.30 1 octez-node 66 | 09:46:21 0 16736 0.82 0.80 0.00 0.57 1.62 0 octez-baker-PtP 67 | 68 | 09:45:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 69 | 09:46:21 0 9218 0.05 0.03 1694140 1051668 26.44 octez-node 70 | 09:46:21 0 9243 0.00 0.00 1219200 928576 23.35 octez-node 71 | 09:46:21 0 16736 0.00 0.00 1097528 209172 5.26 octez-baker-PtP 72 | 73 | 09:46:21 UID PID %usr %system %guest %wait %CPU CPU Command 74 | 09:47:21 0 9218 8.40 0.93 0.00 0.30 9.33 0 octez-node 75 | 09:47:21 0 9243 1.93 0.15 0.00 0.07 2.08 1 octez-node 76 | 09:47:21 0 16736 0.93 0.77 0.00 0.50 1.70 1 octez-baker-PtP 77 | 78 | 09:46:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 79 | 09:47:21 0 9218 0.00 0.00 1694140 1051668 26.44 octez-node 80 | 09:47:21 0 9243 0.00 0.00 1219200 928576 23.35 octez-node 81 | 09:47:21 0 16736 0.02 0.00 1097528 209172 5.26 octez-baker-PtP 82 | 83 | 09:47:21 UID PID %usr %system %guest %wait %CPU CPU Command 84 | 09:48:21 0 9218 8.60 1.17 0.00 0.23 9.77 0 octez-node 85 | 09:48:21 0 9243 1.13 0.02 0.00 0.03 1.15 0 octez-node 86 | 09:48:21 0 16736 0.95 0.65 0.00 0.48 1.60 1 octez-baker-PtP 87 | 88 | 09:47:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 89 | 09:48:21 0 9218 0.00 0.00 1694140 1051668 26.44 octez-node 90 | 09:48:21 0 9243 0.00 0.00 1219200 928576 23.35 octez-node 91 | 09:48:21 0 16736 0.00 0.00 1097528 209172 5.26 octez-baker-PtP 92 | 93 | 09:48:21 UID PID %usr %system %guest %wait %CPU CPU Command 94 | 09:49:21 0 9218 8.28 0.98 0.00 0.35 9.27 1 octez-node 95 | 09:49:21 0 9243 1.17 0.10 0.00 0.05 1.27 1 octez-node 96 | 09:49:21 0 16736 0.87 0.90 0.00 0.60 1.77 0 octez-baker-PtP 97 | 98 | 09:48:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 99 | 09:49:21 0 9218 0.00 0.00 1694140 1051624 26.44 octez-node 100 | 09:49:21 0 9243 0.02 0.00 1219200 928576 23.35 octez-node 101 | 09:49:21 0 16736 0.00 0.00 1097528 209172 5.26 octez-baker-PtP 102 | 103 | 09:49:21 UID PID %usr %system %guest %wait %CPU CPU Command 104 | 09:50:21 0 9218 9.42 1.03 0.00 0.42 10.45 0 octez-node 105 | 09:50:21 0 9243 0.95 0.07 0.00 0.05 1.02 0 octez-node 106 | 09:50:21 0 16736 0.95 0.77 0.00 0.65 1.72 0 octez-baker-PtP 107 | 108 | 09:49:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 109 | 09:50:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 110 | 09:50:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 111 | 09:50:21 0 16736 0.00 0.00 1097528 209124 5.26 octez-baker-PtP 112 | 113 | 09:50:21 UID PID %usr %system %guest %wait %CPU CPU Command 114 | 09:51:21 0 9218 8.73 1.17 0.00 0.33 9.90 0 octez-node 115 | 09:51:21 0 9243 1.00 0.07 0.00 0.02 1.07 0 octez-node 116 | 09:51:21 0 16736 1.02 0.73 0.00 0.58 1.75 1 octez-baker-PtP 117 | 118 | 09:50:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 119 | 09:51:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 120 | 09:51:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 121 | 09:51:21 0 16736 0.00 0.00 1097528 209124 5.26 octez-baker-PtP 122 | 123 | 09:51:21 UID PID %usr %system %guest %wait %CPU CPU Command 124 | 09:52:21 0 9218 9.03 1.13 0.00 0.35 10.17 0 octez-node 125 | 09:52:21 0 9243 1.15 0.13 0.00 0.12 1.28 0 octez-node 126 | 09:52:21 0 16736 0.97 0.65 0.00 0.50 1.62 0 octez-baker-PtP 127 | 128 | 09:51:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 129 | 09:52:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 130 | 09:52:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 131 | 09:52:21 0 16736 0.00 0.00 1097528 209124 5.26 octez-baker-PtP 132 | 133 | 09:52:21 UID PID %usr %system %guest %wait %CPU CPU Command 134 | 09:53:21 0 9218 8.80 1.03 0.00 0.22 9.83 0 octez-node 135 | 09:53:21 0 9243 1.23 0.10 0.00 0.03 1.33 0 octez-node 136 | 09:53:21 0 16736 0.87 0.72 0.00 0.55 1.58 1 octez-baker-PtP 137 | 138 | 09:52:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 139 | 09:53:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 140 | 09:53:21 0 9243 0.02 0.03 1219200 928568 23.35 octez-node 141 | 09:53:21 0 16736 0.00 0.00 1097528 209124 5.26 octez-baker-PtP 142 | 143 | 09:53:21 UID PID %usr %system %guest %wait %CPU CPU Command 144 | 09:54:21 0 9218 8.20 0.88 0.00 0.20 9.08 1 octez-node 145 | 09:54:21 0 9243 1.22 0.05 0.00 0.03 1.27 0 octez-node 146 | 09:54:21 0 16736 0.77 0.53 0.00 0.43 1.30 0 octez-baker-PtP 147 | 148 | 09:53:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 149 | 09:54:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 150 | 09:54:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 151 | 09:54:21 0 16736 0.00 0.00 1097528 209124 5.26 octez-baker-PtP 152 | 153 | 09:54:21 UID PID %usr %system %guest %wait %CPU CPU Command 154 | 09:55:21 0 9218 10.63 1.28 0.00 0.40 11.92 1 octez-node 155 | 09:55:21 0 9243 1.05 0.08 0.00 0.07 1.13 1 octez-node 156 | 09:55:21 0 16736 1.00 0.75 0.00 0.57 1.75 0 octez-baker-PtP 157 | 158 | 09:54:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 159 | 09:55:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 160 | 09:55:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 161 | 09:55:21 0 16736 0.00 0.00 1097528 209124 5.26 octez-baker-PtP 162 | 163 | 09:55:21 UID PID %usr %system %guest %wait %CPU CPU Command 164 | 09:56:21 0 9218 8.70 1.00 0.00 0.27 9.70 1 octez-node 165 | 09:56:21 0 9243 1.13 0.05 0.00 0.05 1.18 0 octez-node 166 | 09:56:21 0 16736 0.82 0.80 0.00 0.57 1.62 0 octez-baker-PtP 167 | 168 | 09:55:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 169 | 09:56:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 170 | 09:56:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 171 | 09:56:21 0 16736 0.00 0.00 1097528 209124 5.26 octez-baker-PtP 172 | 173 | 09:56:21 UID PID %usr %system %guest %wait %CPU CPU Command 174 | 09:57:21 0 9218 8.75 1.22 0.00 0.27 9.97 1 octez-node 175 | 09:57:21 0 9243 1.00 0.07 0.00 0.03 1.07 1 octez-node 176 | 09:57:21 0 16736 0.93 0.72 0.00 0.55 1.65 0 octez-baker-PtP 177 | 178 | 09:56:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 179 | 09:57:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 180 | 09:57:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 181 | 09:57:21 0 16736 0.00 0.00 1097528 209124 5.26 octez-baker-PtP 182 | 183 | 09:57:21 UID PID %usr %system %guest %wait %CPU CPU Command 184 | 09:58:21 0 9218 8.45 0.98 0.00 0.28 9.43 0 octez-node 185 | 09:58:21 0 9243 0.93 0.02 0.00 0.03 0.95 1 octez-node 186 | 09:58:21 0 16736 0.95 0.63 0.00 0.45 1.58 1 octez-baker-PtP 187 | 188 | 09:57:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 189 | 09:58:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 190 | 09:58:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 191 | 09:58:21 0 16736 0.00 0.00 1097528 209124 5.26 octez-baker-PtP 192 | 193 | 09:58:21 UID PID %usr %system %guest %wait %CPU CPU Command 194 | 09:59:21 0 9218 9.45 1.33 0.00 0.33 10.78 0 octez-node 195 | 09:59:21 0 9243 1.17 0.13 0.00 0.08 1.30 1 octez-node 196 | 09:59:21 0 16736 0.92 0.73 0.00 0.48 1.65 1 octez-baker-PtP 197 | 198 | 09:58:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 199 | 09:59:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 200 | 09:59:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 201 | 09:59:21 0 16736 0.05 0.00 1097528 209124 5.26 octez-baker-PtP 202 | 203 | 09:59:21 UID PID %usr %system %guest %wait %CPU CPU Command 204 | 10:00:21 0 9218 8.72 1.22 0.00 0.37 9.93 1 octez-node 205 | 10:00:21 0 9243 1.30 0.10 0.00 0.05 1.40 1 octez-node 206 | 10:00:21 0 16736 0.93 0.78 0.00 0.65 1.72 0 octez-baker-PtP 207 | 208 | 09:59:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 209 | 10:00:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 210 | 10:00:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 211 | 10:00:21 0 16736 0.00 0.00 1097528 209124 5.26 octez-baker-PtP 212 | 213 | 10:00:21 UID PID %usr %system %guest %wait %CPU CPU Command 214 | 10:01:21 0 9218 9.85 1.33 0.00 0.30 11.18 1 octez-node 215 | 10:01:21 0 9243 1.15 0.15 0.00 0.07 1.30 0 octez-node 216 | 10:01:21 0 16736 1.07 0.80 0.00 0.50 1.87 0 octez-baker-PtP 217 | 218 | 10:00:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 219 | 10:01:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 220 | 10:01:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 221 | 10:01:21 0 16736 0.00 0.00 1097528 209124 5.26 octez-baker-PtP 222 | 223 | 10:01:21 UID PID %usr %system %guest %wait %CPU CPU Command 224 | 10:02:21 0 9218 8.22 1.15 0.00 0.27 9.37 0 octez-node 225 | 10:02:21 0 9243 0.97 0.05 0.00 0.02 1.02 1 octez-node 226 | 10:02:21 0 16736 0.90 0.67 0.00 0.48 1.57 0 octez-baker-PtP 227 | 228 | 10:01:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 229 | 10:02:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 230 | 10:02:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 231 | 10:02:21 0 16736 0.00 0.00 1097528 209124 5.26 octez-baker-PtP 232 | 233 | 10:02:21 UID PID %usr %system %guest %wait %CPU CPU Command 234 | 10:03:21 0 9218 7.93 1.00 0.00 0.30 8.93 1 octez-node 235 | 10:03:21 0 9243 1.15 0.08 0.00 0.05 1.23 0 octez-node 236 | 10:03:21 0 16736 1.02 0.58 0.00 0.45 1.60 1 octez-baker-PtP 237 | 238 | 10:02:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 239 | 10:03:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 240 | 10:03:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 241 | 10:03:21 0 16736 0.00 0.00 1097528 209332 5.26 octez-baker-PtP 242 | 243 | 10:03:21 UID PID %usr %system %guest %wait %CPU CPU Command 244 | 10:04:21 0 9218 7.82 0.95 0.00 0.22 8.77 0 octez-node 245 | 10:04:21 0 9243 0.92 0.08 0.00 0.03 1.00 0 octez-node 246 | 10:04:21 0 16736 0.95 0.62 0.00 0.45 1.57 1 octez-baker-PtP 247 | 248 | 10:03:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 249 | 10:04:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 250 | 10:04:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 251 | 10:04:21 0 16736 0.00 0.00 1097528 213444 5.37 octez-baker-PtP 252 | 253 | 10:04:21 UID PID %usr %system %guest %wait %CPU CPU Command 254 | 10:05:21 0 9218 9.18 1.18 0.00 0.27 10.37 1 octez-node 255 | 10:05:21 0 9243 1.10 0.10 0.00 0.05 1.20 0 octez-node 256 | 10:05:21 0 16736 0.93 0.72 0.00 0.48 1.65 1 octez-baker-PtP 257 | 258 | 10:04:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 259 | 10:05:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 260 | 10:05:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 261 | 10:05:21 0 16736 0.03 0.00 1097528 213444 5.37 octez-baker-PtP 262 | 263 | 10:05:21 UID PID %usr %system %guest %wait %CPU CPU Command 264 | 10:06:21 0 9218 8.40 1.20 0.00 0.32 9.60 1 octez-node 265 | 10:06:21 0 9243 0.92 0.07 0.00 0.07 0.98 1 octez-node 266 | 10:06:21 0 16736 1.03 0.70 0.00 0.62 1.73 0 octez-baker-PtP 267 | 268 | 10:05:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 269 | 10:06:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 270 | 10:06:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 271 | 10:06:21 0 16736 0.00 0.00 1097528 213444 5.37 octez-baker-PtP 272 | 273 | 10:06:21 UID PID %usr %system %guest %wait %CPU CPU Command 274 | 10:07:21 0 9218 8.63 1.15 0.00 0.32 9.78 1 octez-node 275 | 10:07:21 0 9243 1.00 0.07 0.00 0.05 1.07 0 octez-node 276 | 10:07:21 0 16736 0.93 0.82 0.00 0.57 1.75 1 octez-baker-PtP 277 | 278 | 10:06:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 279 | 10:07:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 280 | 10:07:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 281 | 10:07:21 0 16736 0.13 0.00 1244992 213444 5.37 octez-baker-PtP 282 | 283 | 10:07:21 UID PID %usr %system %guest %wait %CPU CPU Command 284 | 10:08:21 0 9218 8.20 1.12 0.00 0.22 9.32 0 octez-node 285 | 10:08:21 0 9243 0.98 0.07 0.00 0.03 1.05 0 octez-node 286 | 10:08:21 0 16736 0.93 0.72 0.00 0.52 1.65 1 octez-baker-PtP 287 | 288 | 10:07:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 289 | 10:08:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 290 | 10:08:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 291 | 10:08:21 0 16736 0.00 0.00 1244992 213448 5.37 octez-baker-PtP 292 | 293 | 10:08:21 UID PID %usr %system %guest %wait %CPU CPU Command 294 | 10:09:21 0 9218 8.37 1.00 0.00 0.27 9.37 1 octez-node 295 | 10:09:21 0 9243 0.92 0.03 0.00 0.03 0.95 1 octez-node 296 | 10:09:21 0 16736 0.97 0.65 0.00 0.50 1.62 0 octez-baker-PtP 297 | 298 | 10:08:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 299 | 10:09:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 300 | 10:09:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 301 | 10:09:21 0 16736 0.00 0.00 1244992 213448 5.37 octez-baker-PtP 302 | 303 | 10:09:21 UID PID %usr %system %guest %wait %CPU CPU Command 304 | 10:10:21 0 9218 8.22 1.02 0.00 0.33 9.23 0 octez-node 305 | 10:10:21 0 9243 0.90 0.07 0.00 0.02 0.97 1 octez-node 306 | 10:10:21 0 16736 0.85 0.83 0.00 0.57 1.68 1 octez-baker-PtP 307 | 308 | 10:09:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 309 | 10:10:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 310 | 10:10:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 311 | 10:10:21 0 16736 0.00 0.00 1244992 213448 5.37 octez-baker-PtP 312 | 313 | 10:10:21 UID PID %usr %system %guest %wait %CPU CPU Command 314 | 10:11:21 0 9218 8.50 1.07 0.00 0.33 9.57 0 octez-node 315 | 10:11:21 0 9243 0.78 0.08 0.00 0.03 0.87 1 octez-node 316 | 10:11:21 0 16736 0.92 0.75 0.00 0.53 1.67 1 octez-baker-PtP 317 | 318 | 10:10:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 319 | 10:11:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 320 | 10:11:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 321 | 10:11:21 0 16736 0.00 0.00 1244992 213448 5.37 octez-baker-PtP 322 | 323 | 10:11:21 UID PID %usr %system %guest %wait %CPU CPU Command 324 | 10:12:21 0 9218 7.92 0.90 0.00 0.22 8.82 0 octez-node 325 | 10:12:21 0 9243 1.05 0.10 0.00 0.03 1.15 1 octez-node 326 | 10:12:21 0 16736 0.93 0.70 0.00 0.53 1.63 0 octez-baker-PtP 327 | 328 | 10:11:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 329 | 10:12:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 330 | 10:12:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 331 | 10:12:21 0 16736 0.00 0.00 1244992 213448 5.37 octez-baker-PtP 332 | 333 | 10:12:21 UID PID %usr %system %guest %wait %CPU CPU Command 334 | 10:13:21 0 9218 8.50 1.07 0.00 0.22 9.57 1 octez-node 335 | 10:13:21 0 9243 1.03 0.07 0.00 0.03 1.10 1 octez-node 336 | 10:13:21 0 16736 0.92 0.68 0.00 0.45 1.60 1 octez-baker-PtP 337 | 338 | 10:12:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 339 | 10:13:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 340 | 10:13:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 341 | 10:13:21 0 16736 0.00 0.00 1244992 213420 5.37 octez-baker-PtP 342 | 343 | 10:13:21 UID PID %usr %system %guest %wait %CPU CPU Command 344 | 10:14:21 0 9218 8.00 1.00 0.00 0.28 9.00 0 octez-node 345 | 10:14:21 0 9243 0.90 0.05 0.00 0.03 0.95 1 octez-node 346 | 10:14:21 0 16736 0.97 0.68 0.00 0.48 1.65 1 octez-baker-PtP 347 | 348 | 10:13:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 349 | 10:14:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 350 | 10:14:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 351 | 10:14:21 0 16736 0.00 0.00 1244992 213416 5.37 octez-baker-PtP 352 | 353 | 10:14:21 UID PID %usr %system %guest %wait %CPU CPU Command 354 | 10:15:21 0 9218 8.82 1.12 0.00 0.20 9.93 1 octez-node 355 | 10:15:21 0 9243 0.85 0.07 0.00 0.05 0.92 0 octez-node 356 | 10:15:21 0 16736 0.90 0.63 0.00 0.50 1.53 0 octez-baker-PtP 357 | 358 | 10:14:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 359 | 10:15:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 360 | 10:15:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 361 | 10:15:21 0 16736 0.00 0.00 1244992 213400 5.37 octez-baker-PtP 362 | 363 | 10:15:21 UID PID %usr %system %guest %wait %CPU CPU Command 364 | 10:16:21 0 9218 8.28 1.12 0.00 0.27 9.40 0 octez-node 365 | 10:16:21 0 9243 0.85 0.05 0.00 0.02 0.90 0 octez-node 366 | 10:16:21 0 16736 0.90 0.60 0.00 0.43 1.50 1 octez-baker-PtP 367 | 368 | 10:15:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 369 | 10:16:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 370 | 10:16:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 371 | 10:16:21 0 16736 0.00 0.00 1244992 212640 5.35 octez-baker-PtP 372 | 373 | 10:16:21 UID PID %usr %system %guest %wait %CPU CPU Command 374 | 10:17:21 0 9218 9.00 1.10 0.00 0.28 10.10 1 octez-node 375 | 10:17:21 0 9243 1.02 0.07 0.00 0.03 1.08 0 octez-node 376 | 10:17:21 0 16736 0.98 0.73 0.00 0.55 1.72 0 octez-baker-PtP 377 | 378 | 10:16:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 379 | 10:17:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 380 | 10:17:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 381 | 10:17:21 0 16736 0.00 0.00 1244992 212640 5.35 octez-baker-PtP 382 | 383 | 10:17:21 UID PID %usr %system %guest %wait %CPU CPU Command 384 | 10:18:21 0 9218 8.67 1.30 0.00 0.28 9.97 1 octez-node 385 | 10:18:21 0 9243 1.13 0.07 0.00 0.03 1.20 1 octez-node 386 | 10:18:21 0 16736 0.90 0.72 0.00 0.48 1.62 0 octez-baker-PtP 387 | 388 | 10:17:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 389 | 10:18:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 390 | 10:18:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 391 | 10:18:21 0 16736 0.00 0.00 1244992 212584 5.35 octez-baker-PtP 392 | 393 | 10:18:21 UID PID %usr %system %guest %wait %CPU CPU Command 394 | 10:19:21 0 9218 9.18 1.02 0.00 0.27 10.20 1 octez-node 395 | 10:19:21 0 9243 0.95 0.10 0.00 0.07 1.05 0 octez-node 396 | 10:19:21 0 16736 1.00 0.57 0.00 0.47 1.57 0 octez-baker-PtP 397 | 398 | 10:18:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 399 | 10:19:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 400 | 10:19:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 401 | 10:19:21 0 16736 0.13 0.00 1244992 212584 5.35 octez-baker-PtP 402 | 403 | 10:19:21 UID PID %usr %system %guest %wait %CPU CPU Command 404 | 10:20:21 0 9218 8.87 1.15 0.00 0.25 10.02 1 octez-node 405 | 10:20:21 0 9243 0.97 0.07 0.00 0.03 1.03 1 octez-node 406 | 10:20:21 0 16736 0.97 0.65 0.00 0.52 1.62 1 octez-baker-PtP 407 | 408 | 10:19:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 409 | 10:20:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 410 | 10:20:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 411 | 10:20:21 0 16736 0.00 0.02 1244992 212584 5.35 octez-baker-PtP 412 | 413 | 10:20:21 UID PID %usr %system %guest %wait %CPU CPU Command 414 | 10:21:21 0 9218 8.52 1.22 0.00 0.25 9.73 1 octez-node 415 | 10:21:21 0 9243 0.80 0.03 0.00 0.03 0.83 1 octez-node 416 | 10:21:21 0 16736 0.95 0.62 0.00 0.45 1.57 0 octez-baker-PtP 417 | 418 | 10:20:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 419 | 10:21:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 420 | 10:21:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 421 | 10:21:21 0 16736 0.00 0.00 1244992 212584 5.35 octez-baker-PtP 422 | 423 | 10:21:21 UID PID %usr %system %guest %wait %CPU CPU Command 424 | 10:22:21 0 9218 8.55 1.00 0.00 0.37 9.55 1 octez-node 425 | 10:22:21 0 9243 1.38 0.12 0.00 0.07 1.50 0 octez-node 426 | 10:22:21 0 16736 1.05 0.73 0.00 0.55 1.78 1 octez-baker-PtP 427 | 428 | 10:21:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 429 | 10:22:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 430 | 10:22:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 431 | 10:22:21 0 16736 0.00 0.00 1244992 212584 5.35 octez-baker-PtP 432 | 433 | 10:22:21 UID PID %usr %system %guest %wait %CPU CPU Command 434 | 10:23:21 0 9218 9.42 1.10 0.00 0.32 10.52 1 octez-node 435 | 10:23:21 0 9243 2.32 0.17 0.00 0.08 2.48 1 octez-node 436 | 10:23:21 0 16736 0.97 0.73 0.00 0.55 1.70 0 octez-baker-PtP 437 | 438 | 10:22:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 439 | 10:23:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 440 | 10:23:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 441 | 10:23:21 0 16736 0.00 0.00 1244992 212584 5.35 octez-baker-PtP 442 | 443 | 10:23:21 UID PID %usr %system %guest %wait %CPU CPU Command 444 | 10:24:21 0 9218 8.83 1.02 0.00 0.27 9.85 0 octez-node 445 | 10:24:21 0 9243 0.98 0.05 0.00 0.05 1.03 0 octez-node 446 | 10:24:21 0 16736 0.90 0.73 0.00 0.45 1.63 0 octez-baker-PtP 447 | 448 | 10:23:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 449 | 10:24:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 450 | 10:24:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 451 | 10:24:21 0 16736 0.00 0.00 1244992 212584 5.35 octez-baker-PtP 452 | 453 | 10:24:21 UID PID %usr %system %guest %wait %CPU CPU Command 454 | 10:25:21 0 9218 8.68 1.20 0.00 0.37 9.88 0 octez-node 455 | 10:25:21 0 9243 0.92 0.03 0.00 0.03 0.95 1 octez-node 456 | 10:25:21 0 16736 0.92 0.77 0.00 0.57 1.68 1 octez-baker-PtP 457 | 458 | 10:24:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 459 | 10:25:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 460 | 10:25:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 461 | 10:25:21 0 16736 0.00 0.00 1244992 212584 5.35 octez-baker-PtP 462 | 463 | 10:25:21 UID PID %usr %system %guest %wait %CPU CPU Command 464 | 10:26:21 0 9218 9.27 1.23 0.00 0.28 10.50 1 octez-node 465 | 10:26:21 0 9243 1.12 0.08 0.00 0.05 1.20 0 octez-node 466 | 10:26:21 0 16736 0.95 0.70 0.00 0.57 1.65 0 octez-baker-PtP 467 | 468 | 10:25:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 469 | 10:26:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 470 | 10:26:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 471 | 10:26:21 0 16736 0.00 0.00 1244992 212584 5.35 octez-baker-PtP 472 | 473 | 10:26:21 UID PID %usr %system %guest %wait %CPU CPU Command 474 | 10:27:21 0 9218 8.25 1.18 0.00 0.27 9.43 1 octez-node 475 | 10:27:21 0 9243 1.10 0.12 0.00 0.08 1.22 0 octez-node 476 | 10:27:21 0 16736 1.00 0.60 0.00 0.55 1.60 1 octez-baker-PtP 477 | 478 | 10:26:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 479 | 10:27:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 480 | 10:27:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 481 | 10:27:21 0 16736 0.00 0.00 1244992 212584 5.35 octez-baker-PtP 482 | 483 | 10:27:21 UID PID %usr %system %guest %wait %CPU CPU Command 484 | 10:28:21 0 9218 8.08 0.98 0.00 0.23 9.07 0 octez-node 485 | 10:28:21 0 9243 0.70 0.02 0.00 0.03 0.72 1 octez-node 486 | 10:28:21 0 16736 0.88 0.73 0.00 0.58 1.62 1 octez-baker-PtP 487 | 488 | 10:27:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 489 | 10:28:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 490 | 10:28:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 491 | 10:28:21 0 16736 0.00 0.00 1244992 212584 5.35 octez-baker-PtP 492 | 493 | 10:28:21 UID PID %usr %system %guest %wait %CPU CPU Command 494 | 10:29:21 0 9218 8.17 1.02 0.00 0.35 9.18 1 octez-node 495 | 10:29:21 0 9243 1.20 0.07 0.00 0.07 1.27 1 octez-node 496 | 10:29:21 0 16736 0.90 0.73 0.00 0.55 1.63 0 octez-baker-PtP 497 | 498 | 10:28:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 499 | 10:29:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 500 | 10:29:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 501 | 10:29:21 0 16736 0.00 0.00 1244992 212584 5.35 octez-baker-PtP 502 | 503 | 10:29:21 UID PID %usr %system %guest %wait %CPU CPU Command 504 | 10:30:21 0 9218 8.07 1.00 0.00 0.27 9.07 1 octez-node 505 | 10:30:21 0 9243 1.02 0.05 0.00 0.05 1.07 1 octez-node 506 | 10:30:21 0 16736 0.95 0.72 0.00 0.55 1.67 1 octez-baker-PtP 507 | 508 | 10:29:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 509 | 10:30:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 510 | 10:30:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 511 | 10:30:21 0 16736 0.00 0.00 1244992 212584 5.35 octez-baker-PtP 512 | 513 | 10:30:21 UID PID %usr %system %guest %wait %CPU CPU Command 514 | 10:31:21 0 9218 8.17 1.02 0.00 0.22 9.18 0 octez-node 515 | 10:31:21 0 9243 0.92 0.07 0.00 0.05 0.98 0 octez-node 516 | 10:31:21 0 16736 0.85 0.70 0.00 0.53 1.55 1 octez-baker-PtP 517 | 518 | 10:30:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 519 | 10:31:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 520 | 10:31:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 521 | 10:31:21 0 16736 0.00 0.00 1244992 212584 5.35 octez-baker-PtP 522 | 523 | 10:31:21 UID PID %usr %system %guest %wait %CPU CPU Command 524 | 10:32:21 0 9218 8.13 1.05 0.00 0.25 9.18 0 octez-node 525 | 10:32:21 0 9243 0.78 0.05 0.00 0.02 0.83 1 octez-node 526 | 10:32:21 0 16736 0.93 0.67 0.00 0.52 1.60 1 octez-baker-PtP 527 | 528 | 10:31:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 529 | 10:32:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 530 | 10:32:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 531 | 10:32:21 0 16736 0.00 0.00 1244992 212584 5.35 octez-baker-PtP 532 | 533 | 10:32:21 UID PID %usr %system %guest %wait %CPU CPU Command 534 | 10:33:21 0 9218 7.87 1.05 0.00 0.27 8.92 1 octez-node 535 | 10:33:21 0 9243 1.03 0.03 0.00 0.03 1.07 0 octez-node 536 | 10:33:21 0 16736 0.87 0.80 0.00 0.63 1.67 0 octez-baker-PtP 537 | 538 | 10:32:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 539 | 10:33:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 540 | 10:33:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 541 | 10:33:21 0 16736 0.00 0.00 1244992 212584 5.35 octez-baker-PtP 542 | 543 | 10:33:21 UID PID %usr %system %guest %wait %CPU CPU Command 544 | 10:34:21 0 9218 7.70 0.95 0.00 0.23 8.65 1 octez-node 545 | 10:34:21 0 9243 2.08 0.03 0.00 0.02 2.12 0 octez-node 546 | 10:34:21 0 16736 0.78 0.68 0.00 0.47 1.47 0 octez-baker-PtP 547 | 548 | 10:33:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 549 | 10:34:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 550 | 10:34:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 551 | 10:34:21 0 16736 0.00 0.00 1244992 212584 5.35 octez-baker-PtP 552 | 553 | 10:34:21 UID PID %usr %system %guest %wait %CPU CPU Command 554 | 10:35:21 0 9218 6.85 0.98 0.00 0.22 7.83 0 octez-node 555 | 10:35:21 0 9243 0.92 0.03 0.00 0.05 0.95 0 octez-node 556 | 10:35:21 0 16736 0.80 0.73 0.00 0.52 1.53 1 octez-baker-PtP 557 | 558 | 10:34:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 559 | 10:35:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 560 | 10:35:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 561 | 10:35:21 0 16736 0.00 0.00 1244992 212584 5.35 octez-baker-PtP 562 | 563 | 10:35:21 UID PID %usr %system %guest %wait %CPU CPU Command 564 | 10:36:21 0 9218 8.22 1.12 0.00 0.25 9.33 1 octez-node 565 | 10:36:21 0 9243 0.82 0.10 0.00 0.03 0.92 1 octez-node 566 | 10:36:21 0 16736 0.90 0.67 0.00 0.48 1.57 0 octez-baker-PtP 567 | 568 | 10:35:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 569 | 10:36:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 570 | 10:36:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 571 | 10:36:21 0 16736 0.00 0.00 1244992 212584 5.35 octez-baker-PtP 572 | 573 | 10:36:21 UID PID %usr %system %guest %wait %CPU CPU Command 574 | 10:37:21 0 9218 8.42 1.10 0.00 0.25 9.52 0 octez-node 575 | 10:37:21 0 9243 1.02 0.03 0.00 0.05 1.05 1 octez-node 576 | 10:37:21 0 16736 0.88 0.72 0.00 0.43 1.60 1 octez-baker-PtP 577 | 578 | 10:36:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 579 | 10:37:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 580 | 10:37:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 581 | 10:37:21 0 16736 0.00 0.00 1244992 212584 5.35 octez-baker-PtP 582 | 583 | 10:37:21 UID PID %usr %system %guest %wait %CPU CPU Command 584 | 10:38:21 0 9218 7.82 0.98 0.00 0.28 8.80 1 octez-node 585 | 10:38:21 0 9243 0.97 0.02 0.00 0.02 0.98 0 octez-node 586 | 10:38:21 0 16736 0.95 0.65 0.00 0.57 1.60 0 octez-baker-PtP 587 | 588 | 10:37:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 589 | 10:38:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 590 | 10:38:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 591 | 10:38:21 0 16736 0.00 0.00 1244992 212584 5.35 octez-baker-PtP 592 | 593 | 10:38:21 UID PID %usr %system %guest %wait %CPU CPU Command 594 | 10:39:21 0 9218 8.42 1.15 0.00 0.25 9.57 1 octez-node 595 | 10:39:21 0 9243 0.90 0.07 0.00 0.07 0.97 1 octez-node 596 | 10:39:21 0 16736 0.95 0.78 0.00 0.50 1.73 0 octez-baker-PtP 597 | 598 | 10:38:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 599 | 10:39:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 600 | 10:39:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 601 | 10:39:21 0 16736 0.00 0.00 1244992 212584 5.35 octez-baker-PtP 602 | 603 | 10:39:21 UID PID %usr %system %guest %wait %CPU CPU Command 604 | 10:40:21 0 9218 8.28 1.13 0.00 0.30 9.42 1 octez-node 605 | 10:40:21 0 9243 2.00 0.13 0.00 0.08 2.13 0 octez-node 606 | 10:40:21 0 16736 0.97 0.72 0.00 0.57 1.68 0 octez-baker-PtP 607 | 608 | 10:39:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 609 | 10:40:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 610 | 10:40:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 611 | 10:40:21 0 16736 0.00 0.00 1244992 212584 5.35 octez-baker-PtP 612 | 613 | 10:40:21 UID PID %usr %system %guest %wait %CPU CPU Command 614 | 10:41:21 0 9218 8.95 1.02 0.00 0.27 9.97 0 octez-node 615 | 10:41:21 0 9243 0.98 0.08 0.00 0.03 1.07 0 octez-node 616 | 10:41:21 0 16736 0.93 0.62 0.00 0.45 1.55 1 octez-baker-PtP 617 | 618 | 10:40:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 619 | 10:41:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 620 | 10:41:21 0 9243 0.00 0.00 1219200 928568 23.35 octez-node 621 | 10:41:21 0 16736 0.00 0.00 1244992 212584 5.35 octez-baker-PtP 622 | 623 | 10:41:21 UID PID %usr %system %guest %wait %CPU CPU Command 624 | 10:42:21 0 9218 8.32 1.07 0.00 0.37 9.38 0 octez-node 625 | 10:42:21 0 9243 0.92 0.07 0.00 0.03 0.98 1 octez-node 626 | 10:42:21 0 16736 1.02 0.68 0.00 0.50 1.70 1 octez-baker-PtP 627 | 628 | 10:41:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 629 | 10:42:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 630 | 10:42:21 0 9243 0.00 0.00 1219200 928548 23.35 octez-node 631 | 10:42:21 0 16736 0.00 0.00 1244992 212584 5.35 octez-baker-PtP 632 | 633 | 10:42:21 UID PID %usr %system %guest %wait %CPU CPU Command 634 | 10:43:21 0 9218 9.47 1.22 0.00 0.35 10.68 0 octez-node 635 | 10:43:21 0 9243 1.20 0.08 0.00 0.08 1.28 1 octez-node 636 | 10:43:21 0 16736 0.98 0.63 0.00 0.53 1.62 1 octez-baker-PtP 637 | 638 | 10:42:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 639 | 10:43:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 640 | 10:43:21 0 9243 0.00 0.00 1219200 928548 23.35 octez-node 641 | 10:43:21 0 16736 0.00 0.00 1244992 212584 5.35 octez-baker-PtP 642 | 643 | 10:43:21 UID PID %usr %system %guest %wait %CPU CPU Command 644 | 10:44:21 0 9218 8.17 1.05 0.00 0.27 9.22 0 octez-node 645 | 10:44:21 0 9243 0.90 0.07 0.00 0.07 0.97 0 octez-node 646 | 10:44:21 0 16736 1.08 0.62 0.00 0.63 1.70 1 octez-baker-PtP 647 | 648 | 10:43:21 UID PID minflt/s majflt/s VSZ RSS %MEM Command 649 | 10:44:21 0 9218 0.00 0.00 1694140 1051612 26.44 octez-node 650 | 10:44:21 0 9243 0.00 0.00 1219200 928548 23.35 octez-node 651 | 10:44:21 0 16736 0.00 0.00 1244992 212584 5.35 octez-baker-PtP 652 | -------------------------------------------------------------------------------- /docs/files/failsafe_email.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | LOG_FILE="/var/log/tezos-monitor.log" 4 | POSTMARK_API_TOKEN= 5 | 6 | log() { 7 | local message="$(date): $1" 8 | echo "$message" 9 | if ! echo "$message" >> "$LOG_FILE" 2>/dev/null; then 10 | echo "Failed to write to log file: $LOG_FILE" >&2 11 | echo "Ensure the script has write permissions to this file and its parent directory" >&2 12 | fi 13 | } 14 | 15 | send_email() { 16 | local subject="$1" 17 | local body="$2" 18 | 19 | response=$(curl -s -X POST "https://api.postmarkapp.com/email" \ 20 | -H "Accept: application/json" \ 21 | -H "Content-Type: application/json" \ 22 | -H "X-Postmark-Server-Token: $POSTMARK_API_TOKEN" \ 23 | -d '{ 24 | "From": "", 25 | "To": "", 26 | "Subject": "'"$subject"'", 27 | "TextBody": "'"$body"'", 28 | "MessageStream": "outbound" 29 | }') 30 | echo "Postmark API response: $response" 31 | } 32 | 33 | send_email "Test Subject" "This is a test email body" 34 | 35 | # Test log file writability at the start 36 | if ! touch "$LOG_FILE" 2>/dev/null; then 37 | echo "Cannot write to log file: $LOG_FILE" >&2 38 | echo "Please ensure this script has the necessary permissions" >&2 39 | echo "You might need to run: sudo touch $LOG_FILE && sudo chmod 666 $LOG_FILE" >&2 40 | exit 1 41 | fi 42 | 43 | log "Tezos monitor script started" 44 | 45 | is_node_bootstrapped() { 46 | if octez-client rpc get /chains/main/is_bootstrapped | grep -q "true"; then 47 | return 0 48 | else 49 | return 1 50 | fi 51 | } 52 | 53 | is_baker_running() { 54 | local status=$(systemctl is-active octez-baker.service) 55 | local full_status=$(systemctl status octez-baker.service | grep 'Active:' | sed -e 's/^[[:space:]]*//') 56 | 57 | log "Baker full status: $full_status" 58 | 59 | if [ "$status" = "active" ]; then 60 | log "Baker is running" 61 | return 0 62 | else 63 | log "Baker is not running" 64 | return 1 65 | fi 66 | } 67 | 68 | restart_node() { 69 | log "Restarting Tezos node" 70 | sudo systemctl restart octez-node.service 71 | sleep 60 # Wait for node to start 72 | } 73 | 74 | restart_baker() { 75 | log "Restarting Tezos baker" 76 | sudo systemctl restart octez-baker.service 77 | } 78 | 79 | # Main loop 80 | while true; do 81 | if ! is_node_bootstrapped; then 82 | log "Node is not bootstrapped" 83 | send_email "Tezos Node Not Bootstrapped" "The Tezos node is not bootstrapped. Attempting to restart." 84 | restart_node 85 | 86 | # Wait for node to bootstrap (max 30 minutes) 87 | for i in {1..30}; do 88 | if is_node_bootstrapped; then 89 | log "Node is now bootstrapped" 90 | send_email "Tezos Node Bootstrapped" "The Tezos node has successfully bootstrapped after restart." 91 | break 92 | fi 93 | sleep 60 94 | done 95 | 96 | if ! is_node_bootstrapped; then 97 | log "Node failed to bootstrap after 30 minutes" 98 | send_email "Tezos Node Bootstrap Failed" "The Tezos node failed to bootstrap after 30 minutes of attempts." 99 | fi 100 | else 101 | log "Node is bootstrapped and running" 102 | send_email "Test" "Node is running" 103 | 104 | # Check baker status 105 | if is_baker_running; then 106 | log "Baker check completed" 107 | send_email "Test" "Baker is running" 108 | 109 | else 110 | log "Baker is not running. Attempting to restart." 111 | send_email "Tezos Baker Not Running" "The Tezos baker is not running. Attempting to restart." 112 | restart_baker 113 | sleep 30 # Wait for baker to start 114 | if is_baker_running; then 115 | log "Baker successfully restarted" 116 | send_email "Tezos Baker Restarted" "The Tezos baker has been successfully restarted." 117 | else 118 | log "Failed to restart baker" 119 | send_email "Tezos Baker Restart Failed" "Failed to restart the Tezos baker after attempt." 120 | fi 121 | fi 122 | fi 123 | 124 | log "Sleeping for 60 minutes before next check" 125 | sleep 3600 126 | done -------------------------------------------------------------------------------- /docs/img/1-choose_vps.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/softstack/tezos-octez-node-and-baking-setup-guide/427300c65d6f7ad8164e678552bfff94b79df102/docs/img/1-choose_vps.png -------------------------------------------------------------------------------- /docs/img/2-choose_comfort.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/softstack/tezos-octez-node-and-baking-setup-guide/427300c65d6f7ad8164e678552bfff94b79df102/docs/img/2-choose_comfort.png -------------------------------------------------------------------------------- /docs/img/3-configure_vps.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/softstack/tezos-octez-node-and-baking-setup-guide/427300c65d6f7ad8164e678552bfff94b79df102/docs/img/3-configure_vps.png -------------------------------------------------------------------------------- /docs/img/4-choose_datacentre.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/softstack/tezos-octez-node-and-baking-setup-guide/427300c65d6f7ad8164e678552bfff94b79df102/docs/img/4-choose_datacentre.png -------------------------------------------------------------------------------- /docs/img/5-choose_billing_cycle.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/softstack/tezos-octez-node-and-baking-setup-guide/427300c65d6f7ad8164e678552bfff94b79df102/docs/img/5-choose_billing_cycle.png -------------------------------------------------------------------------------- /docs/img/6-complete_payment_process.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/softstack/tezos-octez-node-and-baking-setup-guide/427300c65d6f7ad8164e678552bfff94b79df102/docs/img/6-complete_payment_process.png -------------------------------------------------------------------------------- /docs/img/LedgerLive_DeveloperMode.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/softstack/tezos-octez-node-and-baking-setup-guide/427300c65d6f7ad8164e678552bfff94b79df102/docs/img/LedgerLive_DeveloperMode.png -------------------------------------------------------------------------------- /docs/img/LedgerLive_TezosApps.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/softstack/tezos-octez-node-and-baking-setup-guide/427300c65d6f7ad8164e678552bfff94b79df102/docs/img/LedgerLive_TezosApps.png -------------------------------------------------------------------------------- /docs/img/Pyrometer_Dashboard.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/softstack/tezos-octez-node-and-baking-setup-guide/427300c65d6f7ad8164e678552bfff94b79df102/docs/img/Pyrometer_Dashboard.png -------------------------------------------------------------------------------- /docs/img/bootstrapped-sync-status.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/softstack/tezos-octez-node-and-baking-setup-guide/427300c65d6f7ad8164e678552bfff94b79df102/docs/img/bootstrapped-sync-status.png -------------------------------------------------------------------------------- /docs/img/snapshot-import-output.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/softstack/tezos-octez-node-and-baking-setup-guide/427300c65d6f7ad8164e678552bfff94b79df102/docs/img/snapshot-import-output.png -------------------------------------------------------------------------------- /docs/img/status-octez-node-service.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/softstack/tezos-octez-node-and-baking-setup-guide/427300c65d6f7ad8164e678552bfff94b79df102/docs/img/status-octez-node-service.png --------------------------------------------------------------------------------