├── README.md ├── defaults └── main.yml ├── files ├── download_start.sh ├── montip.py ├── restart.sh ├── snapcheck.py ├── snapshot-finder.py ├── sol.service └── validator.sh ├── requirements.txt ├── runner.yaml ├── tasks ├── deps.yaml ├── dirs.yaml ├── disks.yaml ├── file_setup.yaml ├── git.yaml ├── keygen.yaml ├── main.yaml ├── ramdisk.yaml ├── rotate.yaml ├── snapshot_downloader.yaml ├── swap.yaml ├── tuner.yaml ├── user.yaml └── var_check.yaml └── test_runner.yaml /README.md: -------------------------------------------------------------------------------- 1 | # Autoclock RPC 2 | 3 | ### What is it good for? 4 | 5 | The goal of the Autoclock RPC ansible playbook is to have you caught up on the Solana blockchain within 15 minutes, assuming you have a capable server and your SSH key ready. It formats/raids/mounts disks, sets up swap, ramdisk (optional), downloads from snapshot and restarts everything. It is currently configured for a Latitude.sh s3.large.x86 (see "Optimal Machine Settings" below), but we hope to adapt it more widely later on. For a more catch-all ansible playbook and in depth guide on RPC's refer to https://github.com/rpcpool/solana-rpc-ansible 6 | 7 | ### Optimal Machine Settings 8 | 9 | - Our Latitude.sh s3.large.x86 server starts with the settings below, which we prefer because: 10 | 11 | - the initial state of the machine is cleaner than others that we have tried 12 | - disks are named consistently (nvme01, nvme0n2) 13 | - ubuntu installed (preferably ubuntu 20.04, 22.04) - this won't work with centos, etc. since they don't use aptitude by default 14 | - the login user being ubuntu helps (all the solana operations are done using the solana user that the ansible playbook creates) 15 | - ubuntu is in the sudoer's list 16 | - unmounted disks are clean - if your root is on one of partitions and you pass it as an argument, this could be disastrous 17 | 18 | - All the above are satisfied by a fresh s3.large.x86 launch found here: https://www.latitude.sh/pricing 19 | - Zen3 AMD Epyc’s such as the 7443p are considered some of the most performant nodes for keeping up with the tip of the chain at the moment, and support large amounts of RAM. 20 | 21 | - Recommended RPC Specs 22 | - 24 cores or more 23 | - 512 GB RAM if you want to use ramdisk/tmpfs and store the accounts db in RAM (we use 300 GB for ram disk). without tmpfs, the ram requirement can be significantly lower (~256 GB) 24 | - 3-4 TB (multiple disks is okay - i.e. 2x 1.9TB - because the ansible playbook stripes them together) 25 | 26 | ### Step 1: SSH into your machine 27 | 28 | ### Step 2: Start a screen session 29 | 30 | ``` 31 | screen -S sol 32 | ``` 33 | 34 | ### Step 3: Install ansible 35 | 36 | ``` 37 | sudo apt-get update && sudo apt-get install ansible -y 38 | ``` 39 | 40 | ### Step 4: Clone the autoclock-rpc repository 41 | 42 | ``` 43 | git clone https://github.com/overclock-validator/autoclock-rpc.git 44 | ``` 45 | 46 | ### Step 5: cd into the autoclock-rpc folder 47 | 48 | ``` 49 | cd autoclock-rpc 50 | ``` 51 | 52 | ### Step 6: Run the ansible command 53 | 54 | - this command can take between 10-20 minutes based on the specs of the machine 55 | - it takes long because it does everything necessary to start the validator (format disks, checkout the solana repo and build it, download the latest snapshot, etc.) 56 | - make sure that the solana_version is up to date (see below) 57 | - check the values set in `defaults/main.yml` and update to the values you want 58 | 59 | ``` 60 | time ansible-playbook runner.yaml 61 | ``` 62 | 63 | #### ~ Parameters explained ~ 64 | 65 | - solana_version: the version of solana that we want to run. Check the Solana Tech discord’s mb-announcements channel for the recommended version. 66 | - swap_mb: megabytes of swap. This can be set this to 50% of RAM or even lower. 100 GB is fine on a 512 GB RAM machine (variable value is in MB so 100000) 67 | - ledger_disk: the disk that will be wiped, formatted with ext4 and then mounted to /mnt/solana-ledger 68 | - ramdisk_size: this is optional and only necessary if you want to use ramdisk for the validator - carves out a large portion of the RAM to store the accountsdb. On a 512 GB RAM instance, this can be set to 300 GB (variable value is in GB so 300) 69 | - solana_installer: whether to install solana from the installer. If set to false it will build solana cli from the solana github 70 | 71 | ### Step 7: Once ansible finishes, switch to the solana user with: 72 | 73 | ``` 74 | sudo su - solana 75 | ``` 76 | 77 | ### Step 8: Check the status 78 | 79 | ``` 80 | source ~/.profile 81 | solana-validator --ledger /mnt/solana-ledger monitor 82 | ledger monitor 83 | Ledger location: /mnt/solana-ledger 84 | ⠉ Validator startup: SearchingForRpcService... 85 | ``` 86 | 87 | #### Initially the monitor should just show the below message which will last for a few minutes and is normal: 88 | 89 | ``` 90 | ⠉ Validator startup: SearchingForRpcService... 91 | ``` 92 | 93 | #### After a while, the message at the terminal should change to something similar to this: 94 | 95 | ``` 96 | ⠐ 00:08:26 | Processed Slot: 156831951 | Confirmed Slot: 156831951 | Finalized Slot: 156831917 | Full Snapshot Slot: 156813730 | 97 | ``` 98 | 99 | #### Check whether the RPC is caught up with the rest of the cluster with: 100 | 101 | ``` 102 | solana catchup --our-localhost 103 | ``` 104 | 105 | If you see the message above, then everything is working fine! Gratz. You have a new RPC server and you can visit the URL at http://xx.xx.xx.xx:8899/ 106 | -------------------------------------------------------------------------------- /defaults/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # defaults file for Solana RPC 3 | solana_version: "v1.16.24" 4 | ledger_disk: "nvme1n1" 5 | setup_disks: "true" 6 | download_snapshot: "true" 7 | swap_mb: "100000" 8 | -------------------------------------------------------------------------------- /files/download_start.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | sudo systemctl stop sol.service 3 | python3 /mnt/snapshot-finder.py --snapshot_path /mnt/solana-snapshots 4 | sudo systemctl start sol.service -------------------------------------------------------------------------------- /files/montip.py: -------------------------------------------------------------------------------- 1 | import urllib.request 2 | import threading 3 | import json 4 | import time 5 | 6 | MAINNET = "https://api.mainnet-beta.solana.com" 7 | LOCAL = "http://localhost:8899" 8 | PAYLOAD = {"jsonrpc":"2.0","id":1, "method":"getSlot", "params":[{"commitment":"processed"}]} 9 | 10 | def get_slot(req, jsondata, result, idx): 11 | response = urllib.request.urlopen(req, jsondata) 12 | res = response.read() 13 | slotnum = json.loads(res)["result"] 14 | result[idx] = slotnum 15 | 16 | 17 | if __name__ == "__main__": 18 | jsondata = json.dumps(PAYLOAD) 19 | jsondataasbytes = jsondata.encode('utf-8') 20 | content_len = len(jsondataasbytes) 21 | while True: 22 | tlist = [] 23 | reqlist = [] 24 | resultlist = [0,0] 25 | for c,u in enumerate([LOCAL, MAINNET]): 26 | req = urllib.request.Request(u) 27 | req.add_header('Content-Type', 'application/json; charset=utf-8') 28 | req.add_header('Content-Length', content_len) 29 | req.add_header('User-Agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.47 Safari/537.36') 30 | reqlist.append((req, jsondataasbytes, resultlist, c)) 31 | for r in reqlist: 32 | tlist.append(threading.Thread(target = get_slot, args = r)) 33 | for t in tlist: 34 | t.start() 35 | for t in tlist: 36 | t.join() 37 | print("reference : %s"%(resultlist[1])) 38 | print("my node : %s"%(resultlist[0])) 39 | print("slots behind reference : %s"%(resultlist[1] - resultlist[0])) 40 | print("=====\n") 41 | time.sleep(10) 42 | -------------------------------------------------------------------------------- /files/restart.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | if [ $# -eq 0 ] 3 | then 4 | python3 /mnt/snapcheck.py 5 | fi 6 | sudo systemctl stop sol.service 7 | sudo systemctl start sol.service 8 | -------------------------------------------------------------------------------- /files/snapcheck.py: -------------------------------------------------------------------------------- 1 | import urllib.request 2 | import threading 3 | import json 4 | import time 5 | 6 | LOCAL = "http://localhost:8899" 7 | PAYLOAD = {"jsonrpc":"2.0","id":1,"method":"getHighestSnapshotSlot"} 8 | 9 | def get_snap_slot(req, jsondata): 10 | response = urllib.request.urlopen(req, jsondata) 11 | res = response.read() 12 | slotnum = json.loads(res)["result"]["incremental"] 13 | return slotnum 14 | 15 | if __name__ == "__main__": 16 | jsondata = json.dumps(PAYLOAD) 17 | jsondataasbytes = jsondata.encode('utf-8') 18 | content_len = len(jsondataasbytes) 19 | latest_snap = None 20 | while True: 21 | req = urllib.request.Request(LOCAL) 22 | req.add_header('Content-Type', 'application/json; charset=utf-8') 23 | req.add_header('Content-Length', content_len) 24 | req.add_header('User-Agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.47 Safari/537.36') 25 | if latest_snap is None: 26 | latest_snap = get_snap_slot(req,jsondataasbytes) 27 | print("current incr snapshot slot: %s\nsleeping..."%(latest_snap)) 28 | stime = int(time.time()) 29 | time.sleep(1) 30 | latest_incr = get_snap_slot(req,jsondataasbytes) 31 | if latest_incr == latest_snap: 32 | time.sleep(1) 33 | else: 34 | etime = int(time.time()) - stime 35 | break 36 | print("latest incr snapshot slot: %s\nslept: %s seconds"%(latest_incr,etime)) 37 | -------------------------------------------------------------------------------- /files/snapshot-finder.py: -------------------------------------------------------------------------------- 1 | from distutils.log import debug 2 | import os 3 | import glob 4 | import requests 5 | import time 6 | import math 7 | import json 8 | import sys 9 | import argparse 10 | import logging 11 | import subprocess 12 | from pathlib import Path 13 | from requests import ReadTimeout, ConnectTimeout, HTTPError, Timeout, ConnectionError 14 | from tqdm import tqdm 15 | from multiprocessing.dummy import Pool as ThreadPool 16 | import statistics 17 | 18 | parser = argparse.ArgumentParser(description='Solana snapshot finder') 19 | parser.add_argument('-t', '--threads-count', default=1000, type=int, 20 | help='the number of concurrently running threads that check snapshots for rpc nodes') 21 | parser.add_argument('-r', '--rpc_address', 22 | default='https://api.mainnet-beta.solana.com', type=str, 23 | help='RPC address of the node from which the current slot number will be taken\n' 24 | 'https://api.mainnet-beta.solana.com') 25 | 26 | parser.add_argument('--max_snapshot_age', default=1300, type=int, help='How many slots ago the snapshot was created (in slots)') 27 | parser.add_argument('--min_download_speed', default=60, type=int, help='Minimum average snapshot download speed in megabytes') 28 | parser.add_argument('--max_download_speed', type=int, 29 | help='Maximum snapshot download speed in megabytes - https://github.com/c29r3/solana-snapshot-finder/issues/11. Example: --max_download_speed 192') 30 | parser.add_argument('--max_latency', default=40, type=int, help='The maximum value of latency (milliseconds). If latency > max_latency --> skip') 31 | parser.add_argument('--version', type=str, help='version of the snapshot required') 32 | parser.add_argument('--with_private_rpc', action="store_true", help='Enable adding and checking RPCs with the --private-rpc option.This slow down checking and searching but potentially increases' 33 | ' the number of RPCs from which snapshots can be downloaded.') 34 | parser.add_argument('--measurement_time', default=7, type=int, help='Time in seconds during which the script will measure the download speed') 35 | parser.add_argument('--snapshot_path', type=str, default=".", help='The location where the snapshot will be downloaded (absolute path).' 36 | ' Example: /home/ubuntu/solana/validator-ledger') 37 | parser.add_argument('--num_of_retries', default=5, type=int, help='The number of retries if a suitable server for downloading the snapshot was not found') 38 | parser.add_argument('--sleep', default=30, type=int, help='Sleep before next retry (seconds)') 39 | parser.add_argument('--sort_order', default='slots_diff', type=str, help='Priority way to sort the found servers. latency or slots_diff') 40 | parser.add_argument('-b', '--blacklist', default='', type=str, help='If the same corrupted archive is constantly downloaded, you can exclude it.' 41 | ' Specify either the number of the slot you want to exclude, or the hash of the archive name. ' 42 | 'You can specify several, separated by commas. Example: -b 135501350,135501360 or --blacklist 135501350,some_hash') 43 | parser.add_argument("-v", "--verbose", help="increase output verbosity to DEBUG", action="store_true") 44 | args = parser.parse_args() 45 | 46 | DEFAULT_HEADERS = {"Content-Type": "application/json"} 47 | RPC = args.rpc_address 48 | WITH_PRIVATE_RPC = args.with_private_rpc 49 | MAX_SNAPSHOT_AGE_IN_SLOTS = args.max_snapshot_age 50 | THREADS_COUNT = args.threads_count 51 | MIN_DOWNLOAD_SPEED_MB = args.min_download_speed 52 | MAX_DOWNLOAD_SPEED_MB = args.max_download_speed 53 | SPEED_MEASURE_TIME_SEC = args.measurement_time 54 | MAX_LATENCY = args.max_latency 55 | VERSION = args.version 56 | SNAPSHOT_PATH = args.snapshot_path if args.snapshot_path[-1] != '/' else args.snapshot_path[:-1] 57 | NUM_OF_MAX_ATTEMPTS = args.num_of_retries 58 | SLEEP_BEFORE_RETRY = args.sleep 59 | NUM_OF_ATTEMPTS = 1 60 | SORT_ORDER = args.sort_order 61 | BLACKLIST = str(args.blacklist).split(",") 62 | AVERAGE_SNAPSHOT_FILE_SIZE_MB = 2500.0 63 | AVERAGE_INCREMENT_FILE_SIZE_MB = 200.0 64 | AVERAGE_CATCHUP_SPEED = 2.0 65 | FULL_LOCAL_SNAP_SLOT = 0 66 | 67 | current_slot = 0 68 | DISCARDED_BY_ARCHIVE_TYPE = 0 69 | DISCARDED_BY_LATENCY = 0 70 | DISCARDED_BY_SLOT = 0 71 | DISCARDED_BY_UNKNW_ERR = 0 72 | DISCARDED_BY_TIMEOUT = 0 73 | FULL_LOCAL_SNAPSHOTS = [] 74 | # skip servers that do not fit the filters so as not to check them again 75 | unsuitable_servers = set() 76 | # Configure Logging 77 | logging.getLogger('urllib3').setLevel(logging.WARNING) 78 | if args.verbose: 79 | logging.basicConfig( 80 | level=logging.DEBUG, 81 | format="%(asctime)s [%(levelname)s] %(message)s", 82 | handlers=[ 83 | logging.FileHandler(f'{SNAPSHOT_PATH}/snapshot-finder.log'), 84 | logging.StreamHandler(sys.stdout), 85 | ] 86 | ) 87 | 88 | else: 89 | # logging.basicConfig(stream=sys.stdout, encoding='utf-8', level=logging.INFO, format='|%(asctime)s| %(message)s') 90 | logging.basicConfig( 91 | level=logging.INFO, 92 | format="%(asctime)s [%(levelname)s] %(message)s", 93 | handlers=[ 94 | logging.FileHandler(f'{SNAPSHOT_PATH}/snapshot-finder.log'), 95 | logging.StreamHandler(sys.stdout), 96 | ] 97 | ) 98 | logger = logging.getLogger(__name__) 99 | 100 | 101 | def convert_size(size_bytes): 102 | if size_bytes == 0: 103 | return "0B" 104 | size_name = ("B", "KB", "MB", "GB", "TB", "PB", "EB", "ZB", "YB") 105 | i = int(math.floor(math.log(size_bytes, 1024))) 106 | p = math.pow(1024, i) 107 | s = round(size_bytes / p, 2) 108 | return "%s %s" % (s, size_name[i]) 109 | 110 | 111 | def version_check(url: str, version: str) -> bool: 112 | data = {"jsonrpc":"2.0","id":1, "method":"getVersion"} 113 | res = requests.post("http://%s"%url, json=data) 114 | parsed_res = json.loads(res.text) 115 | rpc_ver = parsed_res["result"]["solana-core"] 116 | logger.info("version: %s"%rpc_ver) 117 | if version not in rpc_ver: 118 | return False 119 | return True 120 | 121 | def measure_speed(url: str, measure_time: int) -> float: 122 | logging.debug('measure_speed()') 123 | url = f'http://{url}/snapshot.tar.bz2' 124 | r = requests.get(url, stream=True, timeout=measure_time+2) 125 | r.raise_for_status() 126 | start_time = time.monotonic_ns() 127 | last_time = start_time 128 | loaded = 0 129 | speeds = [] 130 | for chunk in r.iter_content(chunk_size=81920): 131 | curtime = time.monotonic_ns() 132 | 133 | worktime = (curtime - start_time) / 1000000000 134 | if worktime >= measure_time: 135 | break 136 | 137 | delta = (curtime - last_time) / 1000000000 138 | loaded += len(chunk) 139 | if delta > 1: 140 | estimated_bytes_per_second = loaded * (1 / delta) 141 | # print(f'{len(chunk)}:{delta} : {estimated_bytes_per_second}') 142 | speeds.append(estimated_bytes_per_second) 143 | 144 | last_time = curtime 145 | loaded = 0 146 | 147 | return statistics.median(speeds) 148 | 149 | 150 | def do_request(url_: str, method_: str = 'GET', data_: str = '', timeout_: int = 3, 151 | headers_: dict = None): 152 | global DISCARDED_BY_UNKNW_ERR 153 | global DISCARDED_BY_TIMEOUT 154 | r = '' 155 | if headers_ is None: 156 | headers_ = DEFAULT_HEADERS 157 | 158 | try: 159 | if method_.lower() == 'get': 160 | r = requests.get(url_, headers=headers_, timeout=(timeout_, timeout_)) 161 | elif method_.lower() == 'post': 162 | r = requests.post(url_, headers=headers_, data=data_, timeout=(timeout_, timeout_)) 163 | elif method_.lower() == 'head': 164 | r = requests.head(url_, headers=headers_, timeout=(timeout_, timeout_)) 165 | # print(f'{r.content, r.status_code, r.text}') 166 | return r 167 | 168 | except (ReadTimeout, ConnectTimeout, HTTPError, Timeout, ConnectionError) as reqErr: 169 | # logger.debug(f'error in do_request(): {reqErr=}') 170 | DISCARDED_BY_TIMEOUT += 1 171 | return f'error in do_request(): {reqErr}' 172 | 173 | except Exception as unknwErr: 174 | DISCARDED_BY_UNKNW_ERR += 1 175 | # logger.debug(f'error in do_request(): {unknwErr=}') 176 | return f'error in do_request(): {reqErr}' 177 | 178 | 179 | def get_current_slot(): 180 | logger.debug("get_current_slot()") 181 | d = '{"jsonrpc":"2.0","id":1, "method":"getSlot"}' 182 | try: 183 | r = do_request(url_=RPC, method_='post', data_=d, timeout_=25) 184 | if 'result' in str(r.text): 185 | return r.json()["result"] 186 | else: 187 | logger.error(f'Can\'t get current slot') 188 | return None 189 | except: 190 | logger.error(f'Can\'t get current slot') 191 | return None 192 | 193 | 194 | def get_all_rpc_ips(): 195 | logger.debug("get_all_rpc_ips()") 196 | d = '{"jsonrpc":"2.0", "id":1, "method":"getClusterNodes"}' 197 | r = do_request(url_=RPC, method_='post', data_=d, timeout_=25) 198 | if 'result' in str(r.text): 199 | if WITH_PRIVATE_RPC is True: 200 | rpc_ips = [] 201 | for node in r.json()["result"]: 202 | if node["rpc"] is not None: 203 | rpc_ips.append(node["rpc"]) 204 | else: 205 | gossip_ip = node["gossip"].split(":")[0] 206 | rpc_ips.append(f'{gossip_ip}:8899') 207 | 208 | else: 209 | rpc_ips = [rpc["rpc"] for rpc in r.json()["result"] if rpc["rpc"] is not None] 210 | 211 | rpc_ips = list(set(rpc_ips)) 212 | return rpc_ips 213 | 214 | else: 215 | logger.error(f'Can\'t get RPC ip addresses {r.text}') 216 | sys.exit() 217 | 218 | 219 | def get_snapshot_slot(rpc_address: str): 220 | global FULL_LOCAL_SNAP_SLOT 221 | global DISCARDED_BY_ARCHIVE_TYPE 222 | global DISCARDED_BY_LATENCY 223 | global DISCARDED_BY_SLOT 224 | 225 | pbar.update(1) 226 | url = f'http://{rpc_address}/snapshot.tar.bz2' 227 | inc_url = f'http://{rpc_address}/incremental-snapshot.tar.bz2' 228 | # d = '{"jsonrpc":"2.0","id":1,"method":"getHighestSnapshotSlot"}' 229 | try: 230 | r = do_request(url_=inc_url, method_='head', timeout_=1) 231 | if 'location' in str(r.headers) and 'error' not in str(r.text) and r.elapsed.total_seconds() * 1000 > MAX_LATENCY: 232 | DISCARDED_BY_LATENCY += 1 233 | return None 234 | 235 | 236 | if 'location' in str(r.headers) and 'error' not in str(r.text): 237 | snap_location_ = r.headers["location"] 238 | if snap_location_.endswith('tar') is True: 239 | DISCARDED_BY_ARCHIVE_TYPE += 1 240 | return None 241 | incremental_snap_slot = int(snap_location_.split("-")[2]) 242 | snap_slot_ = int(snap_location_.split("-")[3]) 243 | slots_diff = current_slot - snap_slot_ 244 | 245 | if slots_diff < -100: 246 | logger.error(f'Something wrong with this snapshot\\rpc_node - {slots_diff=}. This node will be skipped {rpc_address=}') 247 | DISCARDED_BY_SLOT += 1 248 | return 249 | 250 | if slots_diff > MAX_SNAPSHOT_AGE_IN_SLOTS: 251 | DISCARDED_BY_SLOT += 1 252 | return 253 | 254 | if FULL_LOCAL_SNAP_SLOT == incremental_snap_slot: 255 | json_data["rpc_nodes"].append({ 256 | "snapshot_address": rpc_address, 257 | "slots_diff": slots_diff, 258 | "latency": r.elapsed.total_seconds() * 1000, 259 | "files_to_download": [snap_location_], 260 | "cost": AVERAGE_INCREMENT_FILE_SIZE_MB / MIN_DOWNLOAD_SPEED_MB + slots_diff / AVERAGE_CATCHUP_SPEED 261 | }) 262 | return 263 | 264 | r2 = do_request(url_=url, method_='head', timeout_=1) 265 | if 'location' in str(r.headers) and 'error' not in str(r.text): 266 | json_data["rpc_nodes"].append({ 267 | "snapshot_address": rpc_address, 268 | "slots_diff": slots_diff, 269 | "latency": r.elapsed.total_seconds() * 1000, 270 | "files_to_download": [r.headers["location"], r2.headers['location']], 271 | "cost": (AVERAGE_SNAPSHOT_FILE_SIZE_MB + AVERAGE_INCREMENT_FILE_SIZE_MB) / MIN_DOWNLOAD_SPEED_MB + slots_diff / AVERAGE_CATCHUP_SPEED 272 | }) 273 | return 274 | 275 | r = do_request(url_=url, method_='head', timeout_=1) 276 | if 'location' in str(r.headers) and 'error' not in str(r.text): 277 | snap_location_ = r.headers["location"] 278 | # filtering uncompressed archives 279 | if snap_location_.endswith('tar') is True: 280 | DISCARDED_BY_ARCHIVE_TYPE += 1 281 | return None 282 | full_snap_slot_ = int(snap_location_.split("-")[1]) 283 | slots_diff_full = current_slot - full_snap_slot_ 284 | if slots_diff_full <= MAX_SNAPSHOT_AGE_IN_SLOTS and r.elapsed.total_seconds() * 1000 <= MAX_LATENCY: 285 | # print(f'{rpc_address=} | {slots_diff=}') 286 | json_data["rpc_nodes"].append({ 287 | "snapshot_address": rpc_address, 288 | "slots_diff": slots_diff_full, 289 | "latency": r.elapsed.total_seconds() * 1000, 290 | "files_to_download": [snap_location_], 291 | "cost": AVERAGE_SNAPSHOT_FILE_SIZE_MB / MIN_DOWNLOAD_SPEED_MB + slots_diff_full / AVERAGE_CATCHUP_SPEED 292 | }) 293 | return 294 | return None 295 | 296 | except Exception as getSnapErr_: 297 | return None 298 | 299 | 300 | def download(url: str): 301 | fname = url[url.rfind('/'):].replace("/", "") 302 | temp_fname = f'{SNAPSHOT_PATH}/tmp-{fname}' 303 | # try: 304 | # resp = requests.get(url, stream=True) 305 | # total = int(resp.headers.get('content-length', 0)) 306 | # with open(temp_fname, 'wb') as file, tqdm( 307 | # desc=fname, 308 | # total=total, 309 | # unit='iB', 310 | # unit_scale=True, 311 | # unit_divisor=1024, 312 | # ) as bar: 313 | # for data in resp.iter_content(chunk_size=1024): 314 | # size = file.write(data) 315 | # bar.update(size) 316 | 317 | # logger.info(f'Rename the downloaded file {temp_fname} --> {fname}') 318 | # os.rename(temp_fname, f'{SNAPSHOT_PATH}/{fname}') 319 | 320 | # except (ReadTimeout, ConnectTimeout, HTTPError, Timeout, ConnectionError) as downlErr: 321 | # logger.error(f'Exception in download() func\n {downlErr}') 322 | 323 | try: 324 | # dirty trick with wget. Details here - https://github.com/c29r3/solana-snapshot-finder/issues/11 325 | if MAX_DOWNLOAD_SPEED_MB is not None: 326 | process = subprocess.run(['/usr/bin/wget', f'--limit-rate={MAX_DOWNLOAD_SPEED_MB}M', '--trust-server-names', url, f'-O{temp_fname}'], 327 | stdout=subprocess.PIPE, 328 | universal_newlines=True) 329 | else: 330 | process = subprocess.run(['/usr/bin/wget', '--trust-server-names', url, f'-O{temp_fname}'], 331 | stdout=subprocess.PIPE, 332 | universal_newlines=True) 333 | 334 | logger.info(f'Rename the downloaded file {temp_fname} --> {fname}') 335 | os.rename(temp_fname, f'{SNAPSHOT_PATH}/{fname}') 336 | 337 | except Exception as unknwErr: 338 | logger.error(f'Exception in download() func. Make sure wget is installed\n{unknwErr}') 339 | 340 | 341 | def main_worker(): 342 | try: 343 | global FULL_LOCAL_SNAP_SLOT 344 | rpc_nodes = list(set(get_all_rpc_ips())) 345 | global pbar 346 | pbar = tqdm(total=len(rpc_nodes)) 347 | logger.info(f'RPC servers in total: {len(rpc_nodes)} | Current slot number: {current_slot}\n') 348 | 349 | # Search for full local snapshots. 350 | # If such a snapshot is found and it is not too old, then the script will try to find and download an incremental snapshot 351 | FULL_LOCAL_SNAPSHOTS = glob.glob(f'{SNAPSHOT_PATH}/snapshot-*tar*') 352 | if len(FULL_LOCAL_SNAPSHOTS) > 0: 353 | FULL_LOCAL_SNAPSHOTS.sort(reverse=True) 354 | FULL_LOCAL_SNAP_SLOT = FULL_LOCAL_SNAPSHOTS[0].replace(SNAPSHOT_PATH, "").split("-")[1] 355 | logger.info(f'Found full local snapshot {FULL_LOCAL_SNAPSHOTS[0]} | {FULL_LOCAL_SNAP_SLOT=}') 356 | 357 | else: 358 | logger.info(f'Can\'t find any full local snapshots in this path {SNAPSHOT_PATH} --> the search will be carried out on full snapshots') 359 | 360 | print(f'Searching information about snapshots on all found RPCs') 361 | pool = ThreadPool() 362 | pool.map(get_snapshot_slot, rpc_nodes) 363 | logger.info(f'Found suitable RPCs: {len(json_data["rpc_nodes"])}') 364 | logger.info(f'The following information shows for what reason and how many RPCs were skipped.' 365 | f'Timeout most probably mean, that node RPC port does not respond (port is closed)\n' 366 | f'{DISCARDED_BY_ARCHIVE_TYPE=} | {DISCARDED_BY_LATENCY=} |' 367 | f' {DISCARDED_BY_SLOT=} | {DISCARDED_BY_TIMEOUT=} | {DISCARDED_BY_UNKNW_ERR=}') 368 | 369 | if len(json_data["rpc_nodes"]) == 0: 370 | logger.info(f'No snapshot nodes were found matching the given parameters: {args.max_snapshot_age=}') 371 | sys.exit() 372 | 373 | # sort list of rpc node by SORT_ORDER (latency) 374 | rpc_nodes_sorted = sorted(json_data["rpc_nodes"], key=lambda k: k[SORT_ORDER]) 375 | 376 | json_data.update({ 377 | "last_update_at": time.time(), 378 | "last_update_slot": current_slot, 379 | "total_rpc_nodes": len(rpc_nodes), 380 | "rpc_nodes_with_actual_snapshot": len(json_data["rpc_nodes"]), 381 | "rpc_nodes": rpc_nodes_sorted 382 | }) 383 | 384 | with open(f'{SNAPSHOT_PATH}/snapshot.json', "w") as result_f: 385 | json.dump(json_data, result_f, indent=2) 386 | logger.info(f'All data is saved to json file - {SNAPSHOT_PATH}/snapshot.json') 387 | 388 | best_snapshot_node = {} 389 | num_of_rpc_to_check = 15 390 | 391 | rpc_nodes_inc_sorted = [] 392 | logger.info("TRYING TO DOWNLOADING FILES") 393 | for i, rpc_node in enumerate(json_data["rpc_nodes"], start=1): 394 | logger.info("checking node: %s %s"%(i,rpc_node)) 395 | # filter blacklisted snapshots 396 | if BLACKLIST != ['']: 397 | if any(i in str(rpc_node["files_to_download"]) for i in BLACKLIST): 398 | logger.info(f'{i}\\{len(json_data["rpc_nodes"])} BLACKLISTED --> {rpc_node}') 399 | continue 400 | 401 | logger.info(f'{i}\\{len(json_data["rpc_nodes"])} checking the speed {rpc_node}') 402 | if rpc_node["snapshot_address"] in unsuitable_servers: 403 | logger.info(f'Rpc node already in unsuitable list --> skip {rpc_node["snapshot_address"]}') 404 | continue 405 | logger.info("checking version") 406 | try: 407 | if not version_check(rpc_node["snapshot_address"], VERSION): 408 | logger.info("version check failed") 409 | continue 410 | except: 411 | import traceback 412 | print(traceback.format_exc()) 413 | print("broken") 414 | logger.info("version check succeeded") 415 | down_speed_bytes = measure_speed(url=rpc_node["snapshot_address"], measure_time=SPEED_MEASURE_TIME_SEC) 416 | down_speed_mb = convert_size(down_speed_bytes) 417 | if down_speed_bytes < MIN_DOWNLOAD_SPEED_MB * 1e6: 418 | logger.info(f'Too slow: {rpc_node=} {down_speed_mb=}') 419 | unsuitable_servers.add(rpc_node["snapshot_address"]) 420 | continue 421 | 422 | elif down_speed_bytes >= MIN_DOWNLOAD_SPEED_MB * 1e6: 423 | logger.info(f'Suitable snapshot server found: {rpc_node=} {down_speed_mb=}') 424 | for path in reversed(rpc_node["files_to_download"]): 425 | # do not download full snapshot if it already exists locally 426 | if str(path).startswith("/snapshot-"): 427 | full_snap_slot__ = path.split("-")[1] 428 | if full_snap_slot__ == FULL_LOCAL_SNAP_SLOT: 429 | continue 430 | 431 | 432 | if 'incremental' in path: 433 | r = do_request(f'http://{rpc_node["snapshot_address"]}/incremental-snapshot.tar.bz2', method_='head', timeout_=2) 434 | if 'location' in str(r.headers) and 'error' not in str(r.text): 435 | best_snapshot_node = f'http://{rpc_node["snapshot_address"]}{r.headers["location"]}' 436 | else: 437 | best_snapshot_node = f'http://{rpc_node["snapshot_address"]}{path}' 438 | 439 | else: 440 | best_snapshot_node = f'http://{rpc_node["snapshot_address"]}{path}' 441 | logger.info(f'Downloading {best_snapshot_node} snapshot to {SNAPSHOT_PATH}') 442 | download(url=best_snapshot_node) 443 | return 0 444 | 445 | elif i > num_of_rpc_to_check: 446 | logger.info(f'The limit on the number of RPC nodes from' 447 | ' which we measure the speed has been reached {num_of_rpc_to_check=}\n') 448 | break 449 | 450 | else: 451 | logger.info(f'{down_speed_mb=} < {MIN_DOWNLOAD_SPEED_MB=}') 452 | 453 | if best_snapshot_node is {}: 454 | logger.error(f'No snapshot nodes were found matching the given parameters:{args.min_download_speed=}' 455 | f'\nTry restarting the script with --with_private_rpc' 456 | f'RETRY #{NUM_OF_ATTEMPTS}\\{NUM_OF_MAX_ATTEMPTS}') 457 | return 1 458 | 459 | 460 | 461 | except KeyboardInterrupt: 462 | sys.exit('\nKeyboardInterrupt - ctrl + c') 463 | 464 | except: 465 | return 1 466 | 467 | 468 | logger.info("Version: 0.3.3") 469 | logger.info("https://github.com/c29r3/solana-snapshot-finder\n\n") 470 | logger.info(f'{RPC=}\n' 471 | f'{MAX_SNAPSHOT_AGE_IN_SLOTS=}\n' 472 | f'{MIN_DOWNLOAD_SPEED_MB=}\n' 473 | f'{MAX_DOWNLOAD_SPEED_MB=}\n' 474 | f'{SNAPSHOT_PATH=}\n' 475 | f'{THREADS_COUNT=}\n' 476 | f'{NUM_OF_MAX_ATTEMPTS=}\n' 477 | f'{WITH_PRIVATE_RPC=}\n' 478 | f'{SORT_ORDER=}') 479 | 480 | try: 481 | f_ = open(f'{SNAPSHOT_PATH}/write_perm_test', 'w') 482 | f_.close() 483 | os.remove(f'{SNAPSHOT_PATH}/write_perm_test') 484 | except IOError: 485 | logger.error(f'\nCheck {SNAPSHOT_PATH=} and permissions') 486 | Path(SNAPSHOT_PATH).mkdir(parents=True, exist_ok=True) 487 | 488 | json_data = ({"last_update_at": 0.0, 489 | "last_update_slot": 0, 490 | "total_rpc_nodes": 0, 491 | "rpc_nodes_with_actual_snapshot": 0, 492 | "rpc_nodes": [] 493 | }) 494 | 495 | 496 | while NUM_OF_ATTEMPTS <= NUM_OF_MAX_ATTEMPTS: 497 | current_slot = get_current_slot() 498 | logger.info(f'Attempt number: {NUM_OF_ATTEMPTS}. Total attempts: {NUM_OF_MAX_ATTEMPTS}') 499 | NUM_OF_ATTEMPTS += 1 500 | 501 | if current_slot is None: 502 | continue 503 | 504 | worker_result = main_worker() 505 | 506 | if worker_result == 0: 507 | logger.info("Done") 508 | exit(0) 509 | 510 | if worker_result != 0: 511 | logger.info("Now trying with flag --with_private_rpc") 512 | WITH_PRIVATE_RPC = True 513 | 514 | if NUM_OF_ATTEMPTS >= NUM_OF_MAX_ATTEMPTS: 515 | logger.error(f'Could not find a suitable snapshot --> exit') 516 | sys.exit() 517 | 518 | logger.info(f"Sleeping {SLEEP_BEFORE_RETRY} seconds before next try") 519 | time.sleep(SLEEP_BEFORE_RETRY) 520 | -------------------------------------------------------------------------------- /files/sol.service: -------------------------------------------------------------------------------- 1 | [Unit] 2 | Description=Solana Validator 3 | After=network.target 4 | StartLimitIntervalSec=0 5 | 6 | [Service] 7 | Type=simple 8 | Restart=always 9 | RestartSec=1 10 | User=solana 11 | LimitNOFILE=1000000 12 | LogRateLimitIntervalSec=0 13 | Environment="PATH=/mnt/solana/target/release/:/home/solana/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/snap/bin:/home/solana/.local/bin/:/home/solana/.local/bin/:/home/solana/.local/bin/" 14 | ExecStart=/home/solana/validator.sh 15 | 16 | [Install] 17 | WantedBy=multi-user.target 18 | -------------------------------------------------------------------------------- /files/validator.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | export SOLANA_METRICS_CONFIG="host=https://metrics.solana.com:8086,db=mainnet-beta,u=mainnet-beta_write,p=password" 3 | exec /mnt/solana/target/release/solana-validator \ 4 | --identity /home/solana/rpc_node.json \ 5 | --entrypoint entrypoint.mainnet-beta.solana.com:8001 \ 6 | --entrypoint entrypoint2.mainnet-beta.solana.com:8001 \ 7 | --entrypoint entrypoint3.mainnet-beta.solana.com:8001 \ 8 | --entrypoint entrypoint4.mainnet-beta.solana.com:8001 \ 9 | --entrypoint entrypoint5.mainnet-beta.solana.com:8001 \ 10 | --rpc-port 8899 \ 11 | --dynamic-port-range 8002-8099 \ 12 | --no-port-check \ 13 | --halt-on-trusted-validators-accounts-hash-mismatch \ 14 | --gossip-port 8001 \ 15 | --no-voting \ 16 | --private-rpc \ 17 | --rpc-bind-address 0.0.0.0 \ 18 | --enable-cpi-and-log-storage \ 19 | --account-index program-id \ 20 | --enable-rpc-transaction-history \ 21 | --wal-recovery-mode skip_any_corrupted_record \ 22 | --log /mnt/logs/solana-validator.log \ 23 | --accounts /mnt/solana-accounts \ 24 | --ledger /mnt/solana-ledger \ 25 | --snapshots /mnt/solana-snapshots \ 26 | --expected-genesis-hash 5eykt4UsFv8P8NJdTREpY1vzqKqZKvdpKuc147dw2N9d \ 27 | --limit-ledger-size 400000000 \ 28 | --rpc-send-default-max-retries 3 \ 29 | --rpc-send-service-max-retries 3 \ 30 | --rpc-send-retry-ms 2000 \ 31 | --full-rpc-api \ 32 | --accounts-index-memory-limit-mb 350 \ 33 | --account-index-exclude-key kinXdEcpDQeHPEuQnqmUgtYykqKGVFq6CeVX5iAHJq6 \ 34 | --tpu-use-quic \ 35 | --known-validator PUmpKiNnSVAZ3w4KaFX6jKSjXUNHFShGkXbERo54xjb \ 36 | --known-validator Ninja1spj6n9t5hVYgF3PdnYz2PLnkt7rvaw3firmjs \ 37 | --known-validator ChorusmmK7i1AxXeiTtQgQZhQNiXYU84ULeaYF1EH15n \ 38 | --known-validator CakcnaRDHka2gXyfbEd2d3xsvkJkqsLw2akB3zsN1D2S \ 39 | --known-validator SerGoB2ZUyi9A1uBFTRpGxxaaMtrFwbwBpRytHefSWZ \ 40 | --known-validator FLVgaCPvSGFguumN9ao188izB4K4rxSWzkHneQMtkwQJ \ 41 | --known-validator qZMH9GWnnBkx7aM1h98iKSv2Lz5N78nwNSocAxDQrbP \ 42 | --known-validator GiYSnFRrXrmkJMC54A1j3K4xT6ZMfx1NSThEe5X2WpDe \ 43 | --known-validator LA1NEzryoih6CQW3gwQqJQffK2mKgnXcjSQZSRpM3wc \ 44 | --known-validator Certusm1sa411sMpV9FPqU5dXAYhmmhygvxJ23S6hJ24 \ 45 | --known-validator 9bkyxgYxRrysC1ijd6iByp9idn112CnYTw243fdH2Uvr \ 46 | --known-validator 12ashmTiFStQ8RGUpi1BTCinJakVyDKWjRL6SWhnbxbT \ 47 | --known-validator FdaysQ2BZWUGBy8nqFgiudnrhzJp4xChQ8B4zJdc2JZB 48 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | requests 2 | tqdm -------------------------------------------------------------------------------- /runner.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | - name: "playbook runner" 3 | hosts: localhost 4 | connection: local 5 | roles: 6 | - role: "./" 7 | -------------------------------------------------------------------------------- /tasks/deps.yaml: -------------------------------------------------------------------------------- 1 | - name: Install a list of packages 2 | become: true 3 | become_user: root 4 | apt: 5 | update_cache: yes 6 | pkg: 7 | - build-essential 8 | - pkg-config 9 | - libudev-dev 10 | - cmake 11 | - libclang-dev 12 | - libssl-dev 13 | - gparted 14 | - nload 15 | - python3-pip 16 | - net-tools 17 | - logind 18 | - curl 19 | - git 20 | - chrony 21 | - htop 22 | - acl 23 | 24 | - name: check if cargo is installed 25 | become: true 26 | become_user: solana 27 | become_method: sudo 28 | shell: test -f /home/solana/.cargo/env && source /home/solana/.cargo/env && command -v cargo 29 | args: 30 | executable: /bin/bash 31 | register: cargo_exists 32 | ignore_errors: yes 33 | 34 | - name: Download Installer 35 | when: cargo_exists is failed 36 | become: true 37 | become_user: solana 38 | become_method: sudo 39 | get_url: 40 | url: https://sh.rustup.rs 41 | dest: /tmp/sh.rustup.rs 42 | mode: '0755' 43 | force: 'yes' 44 | 45 | - name: install rust/cargo 46 | become: true 47 | become_user: solana 48 | become_method: sudo 49 | when: cargo_exists is failed 50 | shell: /tmp/sh.rustup.rs -y 51 | args: 52 | executable: /bin/bash -------------------------------------------------------------------------------- /tasks/dirs.yaml: -------------------------------------------------------------------------------- 1 | - name: mnt folder 2 | file: 3 | path: /mnt 4 | state: directory 5 | owner: solana 6 | group: solana 7 | mode: "0777" 8 | become: true 9 | become_user: root 10 | 11 | - name: solana snapshots 12 | file: 13 | path: /mnt/solana-snapshots 14 | state: directory 15 | owner: solana 16 | group: solana 17 | become: true 18 | become_user: root 19 | 20 | - name: solana ledger 21 | file: 22 | path: /mnt/solana-ledger 23 | state: directory 24 | owner: solana 25 | group: solana 26 | become: true 27 | become_user: root 28 | 29 | - name: solana accounts 30 | file: 31 | path: /mnt/solana-accounts 32 | state: directory 33 | owner: solana 34 | group: solana 35 | become: true 36 | become_user: root 37 | 38 | - name: solana logs 39 | become: true 40 | become_user: root 41 | file: 42 | path: /mnt/logs 43 | state: directory 44 | owner: solana 45 | group: solana 46 | -------------------------------------------------------------------------------- /tasks/disks.yaml: -------------------------------------------------------------------------------- 1 | - name: check mount 2 | become: true 3 | become_user: root 4 | shell: df -h | grep mnt/solana-ledger 5 | ignore_errors: yes 6 | register: mount_mnt 7 | 8 | - name: format disks with ext4 9 | become: true 10 | become_user: root 11 | shell: mkfs.ext4 /dev/{{ ledger_disk }} 12 | when: mount_mnt.rc != 0 13 | 14 | - name: mount disks for ledger 15 | become: true 16 | become_user: root 17 | shell: mount /dev/{{ ledger_disk }} /mnt/solana-ledger && chown -R solana:solana /mnt/solana-ledger 18 | when: mount_mnt.rc != 0 19 | 20 | - name: add mounted disk to fstab 21 | become: true 22 | become_user: root 23 | lineinfile: 24 | dest: /etc/fstab 25 | state: present 26 | line: "/dev/{{ ledger_disk }} /mnt/solana-ledger ext4 defaults 0 1" 27 | -------------------------------------------------------------------------------- /tasks/file_setup.yaml: -------------------------------------------------------------------------------- 1 | - name: sol validator setup 2 | become: true 3 | become_user: solana 4 | copy: 5 | src: validator.sh 6 | dest: /home/solana/validator.sh 7 | owner: solana 8 | group: solana 9 | mode: "0755" 10 | 11 | - name: setup sol.service 12 | become: true 13 | become_user: root 14 | copy: 15 | src: sol.service 16 | dest: /etc/systemd/system/sol.service 17 | owner: root 18 | group: root 19 | mode: "0755" 20 | 21 | - name: copy restart.sh 22 | become: true 23 | become_user: solana 24 | copy: 25 | src: restart.sh 26 | dest: /home/solana/restart.sh 27 | owner: solana 28 | group: solana 29 | mode: "0755" 30 | 31 | - name: copy download_start.sh 32 | become: true 33 | become_user: solana 34 | copy: 35 | src: download_start.sh 36 | dest: /home/solana/download_start.sh 37 | owner: solana 38 | group: solana 39 | mode: "0755" 40 | 41 | - name: copy snapcheck.py 42 | become: true 43 | become_user: solana 44 | copy: 45 | src: snapcheck.py 46 | dest: /mnt/snapcheck.py 47 | owner: solana 48 | group: solana 49 | 50 | - name: copy snapshot-finder.py 51 | become: true 52 | become_user: solana 53 | copy: 54 | src: snapshot-finder.py 55 | dest: /mnt/snapshot-finder.py 56 | owner: solana 57 | group: solana 58 | 59 | - name: copy montip.py 60 | become: true 61 | become_user: solana 62 | copy: 63 | src: montip.py 64 | dest: /mnt/montip.py 65 | owner: solana 66 | group: solana 67 | 68 | -------------------------------------------------------------------------------- /tasks/git.yaml: -------------------------------------------------------------------------------- 1 | - name: solana repository 2 | become: true 3 | become_user: solana 4 | git: 5 | repo: 'https://github.com/solana-labs/solana.git' 6 | dest: /mnt/solana 7 | version: tags/{{ solana_version }} 8 | 9 | - name: build solana 10 | become: true 11 | become_user: solana 12 | shell: source /home/solana/.cargo/env && /mnt/solana/cargo build --release 13 | args: 14 | chdir: /mnt/solana 15 | executable: /bin/bash -------------------------------------------------------------------------------- /tasks/keygen.yaml: -------------------------------------------------------------------------------- 1 | - name: create identity key 2 | become: true 3 | become_user: solana 4 | shell: /mnt/solana/target/release/solana-keygen new --no-bip39-passphrase -o /home/solana/rpc_node.json 5 | args: 6 | creates: /home/solana/rpc_node.json -------------------------------------------------------------------------------- /tasks/main.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | # tasks file for Solana RPC 3 | - name: var check 4 | include_tasks: var_check.yaml 5 | 6 | - name: create user 7 | include_tasks: user.yaml 8 | 9 | - name: install dependencies 10 | include_tasks: deps.yaml 11 | 12 | - name: folders 13 | include_tasks: dirs.yaml 14 | 15 | - name: swap 16 | include_tasks: swap.yaml 17 | 18 | - name: setup disks 19 | include_tasks: disks.yaml 20 | when: setup_disks|default(false)|bool == true 21 | 22 | - name: logrotate 23 | include_tasks: rotate.yaml 24 | 25 | - name: git 26 | include_tasks: git.yaml 27 | 28 | - name: solana keygen 29 | include_tasks: keygen.yaml 30 | 31 | - name: file setup 32 | include_tasks: file_setup.yaml 33 | 34 | - name: tune host 35 | include_tasks: tuner.yaml 36 | 37 | - name: snapshot download 38 | include_tasks: snapshot_downloader.yaml 39 | when: download_snapshot|default(true)|bool == true 40 | 41 | - name: restart without waiting 42 | become: true 43 | become_user: root 44 | shell: /home/solana/restart.sh 1 45 | when: download_snapshot|default(true)|bool == true 46 | 47 | - name: restart with waiting 48 | become: true 49 | become_user: root 50 | shell: /home/solana/restart.sh 51 | when: download_snapshot|default(true)|bool == false 52 | -------------------------------------------------------------------------------- /tasks/ramdisk.yaml: -------------------------------------------------------------------------------- 1 | - name: setup ramdisk for accountsdb 2 | become: true 3 | become_user: root 4 | shell: mount -t tmpfs -o size={{ ramdisk_size }}G tmpfs /mnt/solana-accounts 5 | when: ansible_memtotal_mb >= 1500*ramdisk_size 6 | 7 | - name: add ramdisk to fstab 8 | become: true 9 | become_user: root 10 | lineinfile: 11 | dest: /etc/fstab 12 | state: present 13 | line: "tmpfs /mnt/solana-accounts tmpfs rw,size={{ ramdisk_size }}G,user=solana 0 0" 14 | when: ansible_memtotal_mb >= 1500*ramdisk_size 15 | -------------------------------------------------------------------------------- /tasks/rotate.yaml: -------------------------------------------------------------------------------- 1 | - name: create logrotate sol 2 | become: true 3 | become_user: root 4 | file: 5 | path: /etc/logrotate.d/sol 6 | state: touch 7 | mode: "0644" 8 | owner: root 9 | group: root 10 | modification_time: preserve 11 | access_time: preserve 12 | 13 | - name: logrotate for solana.log 14 | become: true 15 | become_user: root 16 | blockinfile: 17 | path: /etc/logrotate.d/sol 18 | block: | 19 | /mnt/logs/solana-validator.log { 20 | rotate 7 21 | daily 22 | missingok 23 | postrotate 24 | systemctl kill -s USR1 sol.service 25 | endscript 26 | } 27 | -------------------------------------------------------------------------------- /tasks/snapshot_downloader.yaml: -------------------------------------------------------------------------------- 1 | - name: install snapshot requirements 2 | become: true 3 | become_user: solana 4 | ansible.builtin.pip: 5 | name: 6 | - tqdm 7 | - requests 8 | executable: pip3 9 | 10 | - name: download latest snapshot 11 | become: true 12 | become_user: solana 13 | shell: python3 /mnt/snapshot-finder.py --snapshot_path /mnt/solana-snapshots --version 1.16 --max_latency 150 14 | -------------------------------------------------------------------------------- /tasks/swap.yaml: -------------------------------------------------------------------------------- 1 | - name: Create swap file 2 | become: true 3 | become_user: root 4 | command: 5 | dd if=/dev/zero of=/mnt/swapfile bs=1024 count={{ swap_mb }}k 6 | creates="/mnt/swapfile" 7 | 8 | - name: Change swap file permissions 9 | become: true 10 | become_user: root 11 | file: 12 | path=/mnt/swapfile 13 | owner=root 14 | group=root 15 | mode=0600 16 | 17 | - name: Check swap file type 18 | become: true 19 | become_user: root 20 | command: file /mnt/swapfile 21 | register: swapfile 22 | 23 | - name: Make swap file 24 | become: true 25 | become_user: root 26 | command: "sudo mkswap /mnt/swapfile" 27 | when: swapfile.stdout.find('swap file') == -1 28 | 29 | - name: Write swap entry in fstab 30 | become: true 31 | become_user: root 32 | mount: name=none 33 | src=/mnt/swapfile 34 | fstype=swap 35 | opts=sw 36 | passno=0 37 | dump=0 38 | state=present 39 | 40 | - name: swap check 41 | become: true 42 | become_user: root 43 | ignore_errors: yes 44 | shell: swapon -show | grep /mnt/swapfile 45 | register: swap_check 46 | 47 | - name: Mount swap 48 | become: true 49 | become_user: root 50 | command: "swapon /mnt/swapfile" 51 | when: swap_check.rc != 0 52 | -------------------------------------------------------------------------------- /tasks/tuner.yaml: -------------------------------------------------------------------------------- 1 | - name: Set sysctl performance variables 2 | become: true 3 | become_user: root 4 | shell: 5 | # Note: this overwrite the file, if you want to append replace EOM with EOF 6 | cmd: | 7 | bash -c 'cat >> /etc/sysctl.conf <<- EOM 8 | # set minimum, default, and maximum tcp buffer sizes (10k, 87.38k (linux default), 12M resp) 9 | net.ipv4.tcp_rmem=10240 87380 12582912 10 | net.ipv4.tcp_wmem=10240 87380 12582912 11 | # Enable TCP westwood for kernels greater than or equal to 2.6.13 12 | net.ipv4.tcp_congestion_control=westwood 13 | net.ipv4.tcp_fastopen=3 14 | net.ipv4.tcp_timestamps=0 15 | net.ipv4.tcp_sack=1 16 | net.ipv4.tcp_low_latency=1 17 | # Enable fast recycling TIME-WAIT sockets 18 | # net.ipv4.tcp_tw_recycle = 1 this is in solana tuner, but fails, changing to below, see - https://djangocas.dev/blog/troubleshooting-tcp_tw_recycle-no-such-file-or-directory/ 19 | net.ipv4.tcp_tw_reuse = 1 20 | # dont cache ssthresh from previous connection 21 | net.ipv4.tcp_no_metrics_save = 1 22 | net.ipv4.tcp_moderate_rcvbuf = 1 23 | 24 | # kernel Tunes 25 | kernel.timer_migration=0 26 | kernel.hung_task_timeout_secs=30 27 | # A suggested value for pid_max is 1024 * <# of cpu cores/threads in system> 28 | kernel.pid_max=49152 29 | 30 | # vm.tuning 31 | vm.swappiness=30 32 | vm.max_map_count=2000000 33 | vm.stat_interval=10 34 | vm.dirty_ratio=40 35 | vm.dirty_background_ratio=10 36 | vm.min_free_kbytes = 3000000 37 | vm.dirty_expire_centisecs=36000 38 | vm.dirty_writeback_centisecs=3000 39 | vm.dirtytime_expire_seconds=43200 40 | 41 | # solana systuner 42 | net.core.rmem_max=134217728 43 | net.core.rmem_default=134217728 44 | net.core.wmem_max=134217728 45 | net.core.wmem_default=134217728 46 | EOM' 47 | args: 48 | executable: /bin/bash 49 | 50 | - name: Reload sysctl 51 | become: true 52 | become_user: root 53 | shell: sysctl -p 54 | 55 | - name: Set performance governor 56 | become: true 57 | become_user: root 58 | shell: | 59 | echo 'GOVERNOR="performance"' | tee /etc/default/cpufrequtils 60 | echo "performance" | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor 61 | args: 62 | executable: /bin/bash 63 | 64 | - name: Set performance governor bare metal 65 | become: true 66 | become_user: root 67 | shell: | 68 | echo "performance" | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor 69 | args: 70 | executable: /bin/bash 71 | ignore_errors: True 72 | -------------------------------------------------------------------------------- /tasks/user.yaml: -------------------------------------------------------------------------------- 1 | - name: create solana group 2 | become: true 3 | become_user: root 4 | group: 5 | name: solana 6 | state: present 7 | 8 | - name: create solana user 9 | become: true 10 | become_user: root 11 | user: 12 | name: solana 13 | create_home: yes 14 | groups: solana 15 | shell: /bin/bash 16 | 17 | - name: solana user to have passwordless sudo 18 | become: true 19 | become_user: root 20 | lineinfile: 21 | dest: /etc/sudoers 22 | state: present 23 | regexp: '^solana' 24 | line: 'solana ALL=(ALL) NOPASSWD: ALL' 25 | validate: 'visudo -cf %s' 26 | 27 | - name: Ensure solana build is in path for solana user 28 | become: true 29 | become_user: root 30 | lineinfile: 31 | path: /home/solana/.bashrc 32 | line: 'export PATH=/mnt/solana/target/release:$PATH' 33 | -------------------------------------------------------------------------------- /tasks/var_check.yaml: -------------------------------------------------------------------------------- 1 | #- name: check key file exists 2 | # stat: 3 | # path: "{{ identity_keypair_location }}" 4 | # register: keyfile 5 | # failed_when: not keyfile.stat.exists 6 | 7 | - name: check disks exist 8 | become: true 9 | become_user: root 10 | shell: fdisk -l | grep 'Disk /dev/{{ ledger_disk }}' 11 | when: setup_disks is defined and (setup_disks | bool) 12 | 13 | - name: validate some variables 14 | assert: 15 | that: 16 | - ((ramdisk_size is defined) and (ansible_memtotal_mb >= 1500*ramdisk_size) or ramdisk_size is not defined) 17 | msg: machine ram is too low for ram diskq. comment out ramdisk_size in defaults/main.yml 18 | -------------------------------------------------------------------------------- /test_runner.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | - name: "playbook runner" 3 | hosts: localhost 4 | connection: local 5 | tasks: 6 | - name: var checks 7 | ansible.builtin.import_tasks: var_check.yaml 8 | --------------------------------------------------------------------------------