├── transport.janet ├── LICENSE ├── README.md ├── control.janet └── raft.janet /transport.janet: -------------------------------------------------------------------------------- 1 | # transport.janet 2 | # 3 | # Author: David Beazley (@dabeaz) 4 | # https://www.dabeaz.com 5 | # 6 | # Low-level message transport functions. These send size-prefixed 7 | # messages across a socket. 8 | 9 | (defn send-size [size sock] 10 | (net/write sock (string/format "%10d" size)) 11 | ) 12 | 13 | (defn receive-size [sock] 14 | (parse (net/chunk sock 10)) 15 | ) 16 | 17 | (defn send-message [msg sock] 18 | (send-size (length msg) sock) 19 | (net/write sock msg) 20 | ) 21 | 22 | (defn receive-message [sock] 23 | (net/chunk sock (receive-size sock)) 24 | ) 25 | 26 | 27 | 28 | 29 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2020 David Beazley 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # ranet 2 | 3 | * Author : David Beazley (@dabeaz) 4 | * http://www.dabeaz.com 5 | 6 | This is a partial implementation of the Raft distributed consensus 7 | algorithm (https://raft.github.io) in Janet 8 | (https://janet-lang.org). It is purely for educational purposes and my 9 | own amusement. It is also my first non-trivial Janet program. Use at 10 | your own risk. 11 | 12 | Caution: This code will not work unless you're working from the 13 | latest master branch of the Janet GitHub repo. 14 | 15 | ## How to use 16 | 17 | First, read the Raft paper to understand what's going on. The goal 18 | of the algorithm is to maintain a distributed replicated transaction 19 | log. To see this, you'll need to launch 3-5 separate 20 | terminal windows. In each window, type the following command: 21 | 22 | ``` 23 | bash % janet control.janet 24 | ``` 25 | 26 | Where `` is a number from 0-4 indicating the server number. 27 | You should see output messages such as "BECAME FOLLOWER", 28 | "BECAME CANDIDATE", or "BECAME LEADER" being printed in the 29 | various terminal windows. If you have at least 3 servers running, 30 | one (and only one) of the sessions will be elected leader. 31 | Go to that terminal window and type a command like this: 32 | 33 | ``` 34 | repl:1:> (client-append-entry "hello") 35 | ``` 36 | 37 | You should see output such as the following appear across 38 | all servers as the log is replicated: 39 | 40 | ``` 41 | n Applying: 42 | @[@{:item "hello" :term 4}] 43 | ``` 44 | 45 | Now, start playing around. In theory, you can kill any server 46 | (including the leader) and restart it. It will rejoin the 47 | cluster and have its log restored. As long as at least 3 48 | servers are running, they will elect a leader. 49 | 50 | That's about it. A lot of Raft functionality is missing (log 51 | persistence, snapshots, membership changes, etc.). However, 52 | the main purpose of this was to learn more about Janet. 53 | If you look at the code, you'll find all sorts of things with 54 | threads, networking, objects, and more. 55 | 56 | I'm open to any suggestions to help me improve the code and 57 | my Janet programming style. 58 | 59 | -Dave 60 | 61 | 62 | 63 | 64 | 65 | -------------------------------------------------------------------------------- /control.janet: -------------------------------------------------------------------------------- 1 | # control.janet 2 | # 3 | # Author: David Beazley (@dabeaz) 4 | # https://www.dabeaz.com 5 | # 6 | # Controller environment for the Raft distributed consensus algorithm. 7 | # See https://raft.github.io. 8 | # 9 | # This file defines the basic elements of a controller for the Raft 10 | # algorithm. The controller is responsible for actually running 11 | # the core algorithm and any aspect of its networking, timing, etc. 12 | # 13 | # To use this: Run at the terminal like this: 14 | # 15 | # bash % janet control.janet 16 | # 17 | # Since it's distributed consensus, you need to run at least two 18 | # other servers in different terminals. One of them will eventually 19 | # become leader. You can run up to 5 servers based on the default 20 | # configuration. 21 | # 22 | # On the leader: you can type the following to inject an entry to 23 | # the replicated Log: 24 | # 25 | # repl:1:> (client-append-entry "hello") 26 | # 27 | # If it's working, you should see the entry applied to logs of all 28 | # other servers. 29 | # 30 | # For debugging the Raft state, you can type (raftdebug) at the REPL. 31 | # This probably won't make sense if you haven't read the Raft paper. 32 | 33 | (import raft) 34 | (import transport) 35 | 36 | # Raft Configuration parameters. These are greatly slowed down to 37 | # make everything observable. 38 | (def HEARTBEAT_TIMER 1) 39 | (def ELECTION_TIMER_BASE 5) 40 | (def ELECTION_TIMER_JITTER 3) 41 | (def SERVERS @{ 42 | 0 @["127.0.0.1" 15000] 43 | 1 @["127.0.0.1" 15001] 44 | 2 @["127.0.0.1" 15002] 45 | 3 @["127.0.0.1" 15003] 46 | 4 @["127.0.0.1" 15004] 47 | } 48 | ) 49 | 50 | # Reference to the controller object created by run_server 51 | (var controller nil) 52 | 53 | # Thread that generates a periodic heartbeat event 54 | (defn generate-heartbeats [parent] 55 | (while true 56 | (os/sleep HEARTBEAT_TIMER) 57 | (thread/send parent 'heartbeat) 58 | ) 59 | ) 60 | 61 | # Thread that generates periodic election timeout events 62 | (defn generate-election-timeouts [parent] 63 | (while true 64 | (os/sleep (+ ELECTION_TIMER_BASE (* ELECTION_TIMER_JITTER (math/random)))) 65 | (thread/send parent 'election-timeout) 66 | ) 67 | ) 68 | 69 | (defn decode-message [rawmsg] 70 | (unmarshal rawmsg) 71 | ) 72 | 73 | (defn encode-message [msg] 74 | (marshal msg) 75 | ) 76 | 77 | # Thread that waits for incoming messages 78 | (defn receive-messages [parent] 79 | (defn handler [conn] 80 | (while true 81 | (var msg (transport/receive-message conn)) 82 | (if (nil? msg) 83 | (break) 84 | (thread/send parent (decode-message msg)) 85 | ) 86 | ) 87 | ) 88 | (def myself (thread/receive)) 89 | (def [host port] (SERVERS myself)) 90 | (print "Running Server on port " port) 91 | (net/server "0.0.0.0" port handler) 92 | ) 93 | 94 | # Thread that sends outgoing messages 95 | (defn send-messages [parent] 96 | (def peer (thread/receive)) 97 | (def [host port] (SERVERS peer)) 98 | (var sock nil) 99 | (while true 100 | (try 101 | (do 102 | (var msg (thread/receive 1000)) 103 | (if (nil? sock) 104 | (do 105 | # (print "Trying to connect to " host port) 106 | (try 107 | (set sock (net/connect host port)) 108 | ([err] nil) 109 | ) 110 | # (print "Connected " sock) 111 | ) 112 | ) 113 | (if (not (nil? sock)) 114 | (try 115 | (transport/send-message (encode-message msg) sock) 116 | ([err] 117 | (print "Connection lost:" err) 118 | (net/close sock) 119 | (set sock nil) 120 | ) 121 | ) 122 | ) 123 | ) # do 124 | ([err] 125 | (print err)) 126 | ) 127 | ) 128 | ) 129 | 130 | 131 | # Event processor. This dispatches events to the core Raft algorithm. 132 | (defn process-events [serv control] 133 | # Launch supporting threads 134 | (thread/send (thread/new receive-messages) (control :address)) 135 | (def channels @{}) 136 | (each peer (control :peers) 137 | (put channels peer (thread/new send-messages)) 138 | (thread/send (channels peer) peer) 139 | ) 140 | (thread/new generate-heartbeats) 141 | (thread/new generate-election-timeouts) 142 | 143 | # Run the server 144 | (while true 145 | (var evt (thread/receive 10000)) 146 | (cond (= evt 'heartbeat) (raft/handle-heartbeat serv control) 147 | (= evt 'election-timeout) (raft/handle-election-timeout serv control) 148 | (raft/handle-message serv evt control) 149 | ) 150 | 151 | # After processing events, look for outgoing messages 152 | (each msg (control :outgoing) 153 | (do 154 | (thread/send (channels (msg :dest)) msg) 155 | ) 156 | ) 157 | (put control :outgoing @[]) 158 | ) 159 | ) 160 | 161 | (defn RaftControl [address peers] 162 | @{:type 'RaftControl 163 | :address address 164 | :peers peers 165 | :outgoing @[] 166 | 167 | # --- State Machine methods. These are executed by the Raft algorithm 168 | # --- as a callback. They can be customized if you want to make a 169 | # --- plugable execution environment. 170 | 171 | :send 172 | (fn [self msg] 173 | (array/concat (self :outgoing) msg) 174 | ) 175 | 176 | :apply-state-machine 177 | (fn [self entries] 178 | (print (self :address) " Applying:") 179 | (pp entries) 180 | ) 181 | }) 182 | 183 | # Helper functions to interact with servers at the REPL 184 | (defn client-append-entry [item &opt control] 185 | (def msg (raft/ClientAppendEntry item)) 186 | (if (nil? control) 187 | (thread/send controller msg) 188 | (thread/send control msg) 189 | ) 190 | ) 191 | 192 | (defn raftdebug [] 193 | (thread/send controller (raft/RaftDebug)) 194 | ) 195 | 196 | # Start a new Raft server in a separate thread. Returns the thread-id should you 197 | # want to send it messages 198 | 199 | (defn run-server [address] 200 | (defn run [ parent ] 201 | (process-events (raft/ServerState) (RaftControl address (array/remove (range (length SERVERS)) address))) 202 | ) 203 | (set controller (thread/new run)) 204 | ) 205 | 206 | 207 | # Enclosing definition environment for use below 208 | (def main-env (fiber/getenv (fiber/current))) 209 | 210 | (defn main [name nodenum] 211 | (run-server (parse nodenum)) 212 | (repl nil nil main-env) 213 | ) 214 | 215 | -------------------------------------------------------------------------------- /raft.janet: -------------------------------------------------------------------------------- 1 | # raft.janet 2 | # 3 | # Author: David Beazley (@dabeaz) 4 | # https://www.dabeaz.com 5 | # 6 | # A basic implementation of the Raft distributed consensus protocol. 7 | # 8 | # https://raft.github.io 9 | # 10 | # This file contains the core elements of the algorithm, but 11 | # none of the associated runtime environment (networks, threads, etc.). 12 | # All runtime elements are found in the controller object. 13 | # See "control.janet" for those details. 14 | # 15 | # This is purely for educational purposes. It does not implement more 16 | # advanced parts of Raft including cluster reconfiguration or 17 | # snapshots. 18 | 19 | # -- Log Entries 20 | 21 | (defn LogEntry [term item] 22 | {:term term 23 | :item item} 24 | ) 25 | 26 | (defn append-entries [log prev_index prev_term entries] 27 | (cond (>= prev_index (length log)) false 28 | (< prev_index 0) (do 29 | (array/remove log 0 (length log)) 30 | (array/concat log entries) true) 31 | (not (= ((get log prev_index) :term) prev_term)) false 32 | (do 33 | (array/remove log (+ 1 prev_index) (length log)) 34 | (array/concat log entries) 35 | true) 36 | ) 37 | ) 38 | 39 | (defn test-log [] 40 | (def log @[]) 41 | 42 | # Appending to an empty log should always work 43 | (assert (append-entries log -1 0 @[@{:term 1 :item "x"}])) 44 | 45 | # Successful append to non-empty log 46 | (assert (append-entries log 0 1 @[@{:term 1 :item "y"}])) 47 | 48 | # Duplicate append 49 | (assert (append-entries log 0 1 @[@{:term 1 :item "y"}])) 50 | (assert (= (length log) 2)) 51 | 52 | # Append would create a hole. This should fail 53 | (assert (not (append-entries log 10 1 @[@{:term 1 :item "z"}]))) 54 | 55 | # Append that fails log-matching property 56 | (assert (not (append-entries log 1 0 @[@{:term 1 :item "z"}]))) 57 | 58 | # Empty append 59 | (assert (append-entries log 1 1 @[])) 60 | 61 | # Reset the first entry 62 | (assert (append-entries log -1 -1 @[@{:term 2 :item "a"}])) 63 | (assert (= (length log) 1)) 64 | ) 65 | 66 | # Make sure the basic log features are working 67 | (test-log) 68 | 69 | # -- Internal Messages (only used between threads on the same server) 70 | 71 | (defn ClientAppendEntry [item] 72 | {:type 'ClientAppendEntry 73 | :item item 74 | } 75 | ) 76 | 77 | (defn RaftDebug [] 78 | {:type 'RaftDebug} 79 | ) 80 | 81 | # -- Network Messages (sent between servers) 82 | 83 | (defn AppendEntries [source dest term prev_index prev_term entries commit_index] 84 | {:type 'AppendEntries 85 | :source source 86 | :dest dest 87 | :term term 88 | :prev_index prev_index 89 | :prev_term prev_term 90 | :entries entries 91 | :commit_index commit_index 92 | } 93 | ) 94 | 95 | (defn AppendEntriesResponse [source dest term success match_index] 96 | {:type 'AppendEntriesResponse 97 | :source source 98 | :dest dest 99 | :term term 100 | :success success 101 | :match_index match_index 102 | } 103 | ) 104 | 105 | (defn RequestVote [source dest term last_log_index last_log_term] 106 | {:type 'RequestVote 107 | :source source 108 | :dest dest 109 | :term term 110 | :last_log_index last_log_index 111 | :last_log_term last_log_term 112 | } 113 | ) 114 | 115 | (defn RequestVoteResponse [source dest term vote_granted] 116 | {:type 'RequestVoteResponse 117 | :source source 118 | :dest dest 119 | :term term 120 | :vote_granted vote_granted 121 | } 122 | ) 123 | 124 | # -- Server state. 125 | 126 | (defn ServerState [] 127 | @{:type 'ServerState 128 | :state 'FOLLOWER 129 | :log @[] 130 | :current_term 0 131 | :commit_index -1 132 | :last_applied -1 133 | :next_index nil 134 | :match_index nil 135 | :voted_for nil 136 | :votes_granted @{} 137 | :heard_from_leader false 138 | } 139 | ) 140 | 141 | # -- Helper functions 142 | 143 | (defn send-one-append-entries [serv node control] 144 | "Send a single AppendEntries message to a specified node" 145 | (def prev_index (- (get (serv :next_index) node) 1)) 146 | (def prev_term (if (>= prev_index 0) 147 | (((serv :log) prev_index) :term) 148 | -1)) 149 | (def entries (array/slice (serv :log) ((serv :next_index) node))) 150 | (:send control (AppendEntries 151 | (control :address) 152 | node 153 | (serv :current_term) 154 | prev_index 155 | prev_term 156 | entries 157 | (serv :commit_index) 158 | ) 159 | ) 160 | ) 161 | 162 | (defn send-all-append-entries [serv control] 163 | "Send an AppendEntries message to all of the peers." 164 | (each peer (control :peers) 165 | (send-one-append-entries serv peer control) 166 | ) 167 | ) 168 | 169 | (defn apply-state-machine [serv control] 170 | "Apply the state machine once consensus has been reached." 171 | (if (> (serv :commit_index) (serv :last_applied)) 172 | (do 173 | (:apply-state-machine control 174 | (array/slice (serv :log) 175 | (+ (serv :last_applied) 1) 176 | (+ (serv :commit_index) 1) 177 | ) 178 | ) 179 | (put serv :last_applied (serv :commit_index)) 180 | ) 181 | ) 182 | ) 183 | 184 | # -- State transitions 185 | 186 | (defn become-leader [serv control] 187 | "Become the leader" 188 | (print (control :address) " BECAME LEADER") 189 | (put serv :state 'LEADER) 190 | (put serv :next_index (array/new-filled (+ 1 (length (control :peers))) 191 | (length (serv :log)))) 192 | (put serv :match_index (array/new-filled (+ 1 (length (control :peers))) -1)) 193 | (send-all-append-entries serv control) 194 | ) 195 | 196 | (defn become-follower [serv control] 197 | "Become a follower" 198 | (print (control :address) " BECAME FOLLOWER") 199 | (put serv :state 'FOLLOWER) 200 | (put serv :voted_for nil) 201 | ) 202 | 203 | (defn become-candidate [serv control] 204 | "Become a candidate" 205 | (print (control :address) " BECAME CANDIDATE") 206 | (put serv :state 'CANDIDATE) 207 | (put serv :current_term (+ (serv :current_term) 1)) 208 | (put serv :voted_for (control :address)) 209 | (put serv :votes_granted @{}) 210 | 211 | (def last_log_index (- (length (serv :log)) 1)) 212 | (def last_log_term (if (>= last_log_index 0) 213 | (((serv :log) last_log_index) :term) 214 | -1)) 215 | (each peer (control :peers) 216 | (:send control (RequestVote 217 | (control :address) 218 | peer 219 | (serv :current_term) 220 | last_log_index 221 | last_log_term))) 222 | ) 223 | 224 | # -- Event Handlers. These are executed by the controller 225 | 226 | (defn handle-heartbeat [serv control] 227 | "Leader heartbeat. Occurs on periodic timer to update followers" 228 | (if (= (serv :state) 'LEADER) 229 | (send-all-append-entries serv control) 230 | false) 231 | ) 232 | 233 | (defn handle-election-timeout [serv control] 234 | "Called when nothing has been heard from the leader in awhile" 235 | (if (not (= (serv :state) 'LEADER)) 236 | (if (serv :heard_from_leader) 237 | (put serv :heard_from_leader false) 238 | (become-candidate serv control) 239 | ) 240 | ) 241 | ) 242 | 243 | (defn handle-message [serv msg control] 244 | "Top level function for handling a message." 245 | 246 | # Helper function to check message term numbers and whether or not 247 | # we need to drop into Follower state. 248 | (defn check-terms [] 249 | (if (> (msg :term) (serv :current_term)) 250 | (do 251 | (put serv :current_term (msg :term)) 252 | (become-follower serv control) 253 | ) 254 | ) 255 | ) 256 | 257 | # AppendEntries message received from the leader 258 | (defn handle-append-entries [] 259 | (if (= (serv :state) 'CANDIDATE) 260 | (put serv :state 'FOLLOWER)) 261 | 262 | (if (= (serv :state) 'FOLLOWER) 263 | (do 264 | (def success (append-entries 265 | (serv :log) 266 | (msg :prev_index) 267 | (msg :prev_term) 268 | (msg :entries))) 269 | (def resp (AppendEntriesResponse 270 | (msg :dest) 271 | (msg :source) 272 | (serv :current_term) 273 | success 274 | (+ (msg :prev_index) (length (msg :entries))))) 275 | (if (> (msg :commit_index) (serv :commit_index)) 276 | (do 277 | (put serv :commit_index (min (msg :commit_index) (- (length (serv :log)) 1))) 278 | (apply-state-machine serv control) 279 | ) 280 | ) 281 | (put serv :heard_from_leader true) 282 | (:send control resp) 283 | ) 284 | ) 285 | ) 286 | 287 | # AppendEntriesResponse message received from a follower 288 | (defn handle-append-entries-response [] 289 | (if (= (serv :state) 'LEADER) 290 | (if (msg :success) 291 | (do 292 | (put (serv :next_index) (msg :source) (+ (msg :match_index) 1)) 293 | (put (serv :match_index) (msg :source) (msg :match_index)) 294 | 295 | (def match_indices (sorted (serv :match_index))) 296 | (array/remove match_indices (control :address)) 297 | (def commit_index (match_indices (/ (length match_indices) 2))) 298 | (if (and (> commit_index (serv :commit_index)) 299 | (= (((serv :log) commit_index) :term) (serv :current_term))) 300 | (do 301 | (put serv :commit_index commit_index) 302 | (apply-state-machine serv control) 303 | ) 304 | ) 305 | ) 306 | # Unsuccessful operation. Need to retry with earlier log messages 307 | (do 308 | (put (serv :next_index) (msg :source) (- ((serv :next_index) (msg :source)) 1)) 309 | (send-one-append-entries serv (msg :source) control) 310 | ) 311 | ) 312 | ) 313 | ) 314 | 315 | # RequestVote Message received from a Candidate 316 | (defn handle-request-vote [] 317 | (var success true) 318 | (if (not (nil? (serv :voted_for))) 319 | (set success (= (serv :voted_for) (msg :source))) 320 | ) 321 | # Granting of a vote requires very careful reading of section 5.4.1 of the Raft 322 | # paper. We only grant a vote if the candidate's log is at least as up-to-date 323 | # as our own. 324 | (def my_last_log_index (- (length (serv :log)) 1)) 325 | (def my_last_log_term (if (>= my_last_log_index 0) 326 | (((serv :log) my_last_log_index) :term) 327 | -1)) 328 | (if (or (> my_last_log_term (msg :last_log_term)) 329 | (and (= my_last_log_term (msg :last_log_term)) 330 | (> my_last_log_index (msg :last_log_index)))) 331 | (set success false) 332 | ) 333 | 334 | (if success (put serv :voted_for (msg :source))) 335 | 336 | (:send control (RequestVoteResponse 337 | (msg :dest) 338 | (msg :source) 339 | (serv :current_term) 340 | success 341 | ) 342 | ) 343 | ) 344 | 345 | # RequestVoteResponse message from a follower 346 | (defn handle-request-vote-response [] 347 | (if (and (= (serv :state) 'CANDIDATE) (msg :vote_granted)) 348 | (do 349 | (put (serv :votes_granted) (msg :source) 1) 350 | (if (>= (length (serv :votes_granted)) 351 | (/ (length (control :peers)) 2)) 352 | (do 353 | (become-leader serv control) 354 | ) 355 | ) 356 | ) 357 | ) 358 | ) 359 | 360 | 361 | (defn client-append-entry [] 362 | "Append a new entry to the log as the Raft client." 363 | (assert (= (serv :state) 'LEADER)) 364 | (def entry (LogEntry (serv :current_term) (msg :item))) 365 | (def prev_index (- (length (serv :log)) 1)) 366 | (def prev_term (if (>= prev_index 0) 367 | (((serv :log) prev_index) :term) 368 | -1)) 369 | (append-entries (serv :log) prev_index prev_term @[ entry ]) 370 | ) 371 | 372 | 373 | (defn debug [] 374 | "Debugging function to output a value from the internal raft state" 375 | (print "DEBUG:::") 376 | (pp serv) 377 | ) 378 | 379 | # --- Message handling 380 | 381 | (cond 382 | # -- Internal messages 383 | (= (msg :type) 'ClientAppendEntry) (client-append-entry) 384 | (= (msg :type) 'RaftDebug) (debug) 385 | (do 386 | (check-terms) 387 | # -- Network messages 388 | (if (>= (msg :term) (serv :current_term)) 389 | (cond (= (msg :type) 'AppendEntries) (handle-append-entries) 390 | (= (msg :type) 'AppendEntriesResponse) (handle-append-entries-response) 391 | (= (msg :type) 'RequestVote) (handle-request-vote) 392 | (= (msg :type) 'RequestVoteResponse) (handle-request-vote-response) 393 | (error "Unsupported message") 394 | ) 395 | ) 396 | ) 397 | ) 398 | ) 399 | --------------------------------------------------------------------------------