├── .clj-kondo └── config.edn ├── .gitignore ├── README.md ├── examples ├── barrier.clj ├── group_membership.clj └── leader_election.clj ├── project.clj ├── scripts └── publish.sh ├── src ├── zookeeper.clj └── zookeeper │ ├── data.clj │ ├── internal.clj │ ├── server.clj │ └── util.clj └── test └── zookeeper └── test ├── data_test.clj └── zookeeper_test.clj /.clj-kondo/config.edn: -------------------------------------------------------------------------------- 1 | {:lint-as {zookeeper.internal/try* clojure.core/try}} 2 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | pom.xml 2 | *jar 3 | /lib/ 4 | /classes/ 5 | .lein-failures 6 | .lein-deps-sum 7 | .nrepl* 8 | target* 9 | *~ 10 | *# 11 | .#* 12 | .clj-kondo/.cache 13 | .lsp 14 | .vscode 15 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # zookeeper-clj 2 | 3 | Zookeeper-clj is a Clojure DSL for Apache ZooKeeper, which "is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services." 4 | 5 | Out of the box ZooKeeper provides name service, configuration, and group membership. From these core services, higher-level distributed concurrency abstractions can be built, including distributed locks, distributed queues, barriers, leader-election, and transaction services as described in ZooKeeper Recipes and Solutions and the paper "ZooKeeper: Wait-free coordination for Internet-scale systems". 6 | 7 | Building these distributed concurrency abstractions is the goal of the Java-based Menagerie library and the, **soon to be released**, Clojure-based **Avout** library. Avout, in particular, provides distributed versions of Clojure's Atom and Ref concurrency primitives, as well as distributed implementations of *java.util.concurrent.lock.Lock* and *java.util.concurrent.lock.ReadWriteLock*. 8 | 9 | ## Table of Contents 10 | 11 | * Getting Started 12 | * connect function 13 | * watchers 14 | * create function 15 | * asynchronous calls 16 | * exists function 17 | * children function 18 | * sequential nodes 19 | * data functions 20 | * data serialization 21 | * delete functions 22 | * acl function 23 | * Group Membership Example 24 | * Leader Election Example 25 | * Barrier Example 26 | * Running ZooKeeper 27 | * Testing 28 | * Contributing 29 | * References 30 | 31 | 32 | 33 | ## Getting Started 34 | 35 | To run these examples, first start a local instance of ZooKeeper on port 2181, see instructions below, and include zookeeper-clj as a dependency by adding the following to your Leiningen project.clj file: 36 | 37 | [![Clojars Project](https://clojars.org/zookeeper-clj/latest-version.svg)](http://clojars.org/zookeeper-clj) 38 | 39 | 40 | ### connect function 41 | 42 | First require the zookeeper namespace and create a client with the connect function. 43 | 44 | ```clojure 45 | (require '[zookeeper :as zk]) 46 | (def client (zk/connect "127.0.0.1:2181")) 47 | ``` 48 | 49 | The connection string is the name, or IP address, and port of the ZooKeeper server. Several host:port pairs can be included as a comma seperated list. The port can be left off if it is 2181. 50 | 51 | The connection to the ZooKeeper server can be closed with the close function. 52 | 53 | ```clojure 54 | (zk/close client) 55 | ``` 56 | 57 | 58 | ### watchers 59 | 60 | A watcher function that takes a single event map argument can be passed to connect, which will be invoked as a result of changes of keeper-state, or as a result of other events. 61 | 62 | ```clojure 63 | (def client (zk/connect "127.0.0.1" :watcher (fn [event] (println event)))) 64 | ``` 65 | 66 | if the :watch? flag is set to true when using the exists, children, or data functions, the default watcher function will be triggered under the following circumstances. 67 | 68 | * **exists**: the watch will be triggered by a successful operation that creates/deletes the node or sets the data on the node. 69 | * **children**: the watch will be triggered by a successful operation that deletes the node of the given path or creates/deletes a child under the node. 70 | * **data**: the watch will be triggered by a successful operation that sets data on the node, or deletes the node. 71 | 72 | The default watcher function can be overriden with a custom function by passing it as the :watcher argument to the exists, children, or data functions. 73 | 74 | The argument to the watcher function is a map with three keys: :event-type, :keeper-state, and :path. 75 | 76 | * **event-type**: :NodeDeleted, :NodeDataChanged, :NodeCreated, :NodeChildrenChanged, :None 77 | * **keeper-state**: :AuthFailed, :Unknown, :SyncConnected, :Disconnected, :Expired, :NoSyncConnected 78 | * **path**: the path to the node in question, may be nil 79 | 80 | NOTE: Watches are one time triggers; if you get a watch event and you want to get notified of future changes, you must set another watch. 81 | 82 | 83 | ### create function 84 | 85 | Next, create a node called "/parent-node" 86 | 87 | ```clojure 88 | (zk/create client "/parent-node" :persistent? true) 89 | ;; => "/parent-node" 90 | ``` 91 | 92 | Setting the :persistent? flag to true creates a persistent node, meaning one that will persist even after the client that created it is no longer connected. By default, nodes are ephemeral (i.e. :persistent? false) and will be deleted if the client that created them is disconnected (this is key to how ZooKeeper is used to build robust distributed systems). 93 | 94 | A node must be persistent if you want it to have child nodes. 95 | 96 | 97 | ### asynchronous calls 98 | 99 | Most of the zookeeper functions can be called asynchronously by setting the :async? option to true, or by providing an explicit callback function with the :callback option. When invoked asynchronously, each function will return a promise that will eventually contain the result of the call (a map with the following keys: :return-code, :path, :context, :name). 100 | 101 | ```clojure 102 | (def result-promise (zk/create client "/parent-node" :persistent? true :async? true)) 103 | ``` 104 | 105 | Dereferencing the promise will block until a result is returned. 106 | 107 | ```clojure 108 | @result-promise 109 | ``` 110 | 111 | If a :callback function is passed, the promise will be returned with the result map and the callback will be invoked with the same map. 112 | 113 | ```clojure 114 | (def result-promise (zk/create client "/parent-node" :persistent? true :callback (fn [result] (println result)))) 115 | ``` 116 | 117 | 118 | ### exists function 119 | 120 | We can check the existence of the newly created node with the exists function. 121 | 122 | ```clojure 123 | (zk/exists client "/parent-node") 124 | ``` 125 | 126 | The exists function returns nil if the node does not exist, and returns a map with the following keys if it does: :numChildren, :ephemeralOwner, :cversion, :mzxid, :czxid, :dataLength, :ctime, :version, :aversion, :mtime, :pzxid. See the ZooKeeper documentation for description of each field. 127 | 128 | The exists function accepts the :watch?, :watcher, :async?, and :callback options. The watch functions will be triggered by a successful operation that creates/deletes the node or sets the data on the node. 129 | 130 | 131 | ### children function 132 | 133 | Next, create a child node for "/parent-node" 134 | 135 | ```clojure 136 | (zk/create client "/parent-node/child-node") 137 | ;; => "/parent-node/child-node" 138 | ``` 139 | 140 | Since the :persistent? flag wasn't set to true, this node will be ephemeral, meaning it will be deleted if the client that created it is disconnected. 141 | 142 | A list of a node's children can be retrieved with the children function. 143 | 144 | ```clojure 145 | (zk/children client "/parent-node") 146 | ;; => ("child-node") 147 | ``` 148 | 149 | If the node has no children, nil will be returned, and if the node doesn't exist, false will be returned. 150 | 151 | The children function accepts the :watch?, :watcher, :async?, and :callback options. The watch function will be triggered by a successful operation that deletes the node of the given path or creates/delete a child under the node. 152 | 153 | 154 | ### sequential nodes 155 | 156 | If the :sequential? option is set to true when a node is created, a ten digit sequential ID is appended to the name of the node (it's idiomatic to include a dash as the last character of a sequential node's name). 157 | 158 | ```clojure 159 | (zk/create-all client "/parent/child-" :sequential? true) 160 | ;; => "/parent/child-0000000000" 161 | ``` 162 | 163 | The create-all function creates the parent nodes if they don't already exist, here we used it to create the "/parent" node. 164 | 165 | The sequence ID increases monotonically for a given parent directory. 166 | 167 | ```clojure 168 | (zk/create client "/parent/child-" :sequential? true) 169 | ;; => "/parent/child-0000000001" 170 | 171 | (zk/create client "/parent/child-" :sequential? true) 172 | ;; => "/parent/child-0000000002" 173 | ``` 174 | 175 | The zookeeper.util namespace contains functions for extracting IDs from sequential nodes and sorting them. 176 | 177 | ```clojure 178 | (require '[zookeeper.util :as util]) 179 | (util/extract-id (first (zk/children client "/parent"))) 180 | ;; => 2 181 | ``` 182 | 183 | The order of the child nodes return from children is arbitrary, but the nodes can be sorted with the sort-sequential-nodes function. 184 | 185 | ```clojure 186 | (util/sort-sequential-nodes (zk/children client "/parent")) 187 | ;; => ("child-0000000000" "child-0000000001" "child-0000000002") 188 | ``` 189 | 190 | 191 | ### data functions 192 | 193 | Each node has a data field that can hold a byte array, which is limited to 1M is size. 194 | 195 | The set-data function is used to insert data. The set-data function takes a version number, which needs to match the current data version. The current version is a field in the map returned by the exists function. 196 | 197 | ```clojure 198 | (def version (:version (zk/exists client "/parent"))) 199 | 200 | (zk/set-data client "/parent" (.getBytes "hello world" "UTF-8") version) 201 | ``` 202 | 203 | The data function is used to retrieve the data stored in a node. 204 | 205 | ```clojure 206 | (zk/data client "/parent") 207 | ;; => {:data ..., :stat {...}} 208 | ``` 209 | 210 | The data function returns a map with two fields, :data and :stat. The :stat value is the same map returned by the exists function. The :data value is a byte array. 211 | 212 | ```clojure 213 | (String. (:data (zk/data client "/parent")) "UTF-8") 214 | ;; => "hello world" 215 | ``` 216 | 217 | The data function accepts the :watch?, :watcher, :async?, and :callback options. The watch function will be triggered by a successful operation that sets data on the node, or deletes the node. 218 | 219 | 220 | ### data serialization 221 | 222 | The zookeeper.data namespace contains functions for serializing different primitive types to and from byte arrays. 223 | 224 | ```clojure 225 | (require '[zookeeper.data :as data]) 226 | (def version (:version (zk/exists client "/parent"))) 227 | (zk/set-data client "/parent" (data/to-bytes 1234) version) 228 | (data/to-long (:data (zk/data client "/parent"))) 229 | ;; => 1234 230 | ``` 231 | 232 | The following types have been extended to support the to-bytes method: String, Integer, Double, Long, Float, Character. The following functions can be used to convert byte arrays back to their respective types: to-string, to-int, to-double, to-long, to-float, to-short, and to-char. 233 | 234 | Clojure forms can be written to and read from the data field using pr-str and read-string, respectively. 235 | 236 | ```clojure 237 | (zk/set-data client "/parent" (data/to-bytes (pr-str {:a 1, :b 2, :c 3})) 2) 238 | (read-string (data/to-string (:data (zk/data client "/parent")))) 239 | ;; => {:a 1, :b 2, :c 3} 240 | ``` 241 | 242 | 243 | ### delete functions 244 | 245 | Nodes can be deleted with the delete function. 246 | 247 | ```clojure 248 | (zk/delete client "/parent/child-node") 249 | ``` 250 | 251 | The delete function takes an optional version number, the delete will succeed if the node exists at the given version. the default version value is -1, which matches any version number. 252 | 253 | The delete function accepts the :async? and :callback options. 254 | 255 | Nodes that have children cannot be deleted. Two convenience functions, delete-children and delete-all, can be used to delete all of a node's children or delete a node and all of it's children, respectively. 256 | 257 | ```clojure 258 | (delete-all client "/parent") 259 | ``` 260 | 261 | 262 | ### ACL functions 263 | 264 | The acl function takes a scheme, id value, and a set of permissions. The following schemes are built in. 265 | 266 | * **world** has a single id, **anyone**, that represents anyone. 267 | * **auth** doesn't use any id, represents any authenticated user. 268 | * **digest** uses a username:password string to generate an MD5 hash which is then used as an ACL ID identity. Authentication is done by sending the username:password in clear text. When used in the ACL the expression will be the username:base64 encoded SHA1 password digest. 269 | * **host** uses the client host name as an ACL ID identity. The ACL expression is a hostname suffix. For example, the ACL expression host:corp.com matches the ids host:host1.corp.com and host:host2.corp.com, but not host:host1.store.com. 270 | * **ip** uses the client host IP as an ACL ID identity. The ACL expression is of the form addr/bits where the most significant bits of addr are matched against the most significant bits of the client host IP. 271 | 272 | The folllowing permissions are supported: 273 | 274 | * **:create**: you can create a child node 275 | * **:read**: you can get data from a node and list its children. 276 | * **:write**: you can set data for a node 277 | * **:delete**: you can delete a child node 278 | * **:admin**: you can set permissions 279 | 280 | Below are examples of each ACL scheme. 281 | 282 | ```clojure 283 | (zk/acl "world" "anyone" :read :create :delete :admin :write) 284 | (zk/acl "ip" "127.0.0.1" :read :create :delete :admin :write) 285 | (zk/acl "host" "thinkrelevance.com" :admin :read :write :delete :create) 286 | (zk/acl "auth" "" :read :create :delete :admin :write) 287 | ``` 288 | 289 | There are five convenience functions for creating ACLs of each scheme, world-acl, auth-acl, digest-acl, host-acl, and ip-acl. 290 | 291 | ```clojure 292 | (zk/world-acl :read :delete :write) 293 | ``` 294 | 295 | When no permissions are provided, the following are used by default: :read, :create, :delete, :write -- but not :admin. 296 | 297 | ```clojure 298 | (zk/ip-acl "127.0.0.1") 299 | (zk/digest-acl "david:secret" :read :delete :write) 300 | (zk/host-acl "thinkrelevance.com" :read :delete :write) 301 | (zk/auth-acl :read :delete :write) 302 | ``` 303 | 304 | A list of ACLs can be passed as an option to the create function. 305 | 306 | ```clojure 307 | (zk/create client "/protected-node" :acl [(zk/auth-acl :admin :create :read :delete :write)]) 308 | ``` 309 | 310 | In the above example, only the user that created the node has permissions on it. In order to authenticate a user, authentication info must be added to a client connection with the add-auth-info function. 311 | 312 | ```clojure 313 | (zk/add-auth-info client "digest" "david:secret") 314 | ``` 315 | 316 | If an unauthorized client tries to access the node, a org.apache.zookeeper.KeeperException$NoAuthException exception will be thrown. 317 | 318 | 319 | ## Group Membership Example 320 | 321 | ```clojure 322 | (def group-name "/example-group") 323 | 324 | (def client (zk/connect "127.0.0.1:2181")) 325 | 326 | (when-not (zk/exists client group-name) 327 | (zk/create client group-name :persistent? true)) 328 | ``` 329 | 330 | This watcher will be called every time the children of the 331 | "/example-group" node are changed. Each time it is called it will 332 | print the children and add itself as the watcher. 333 | 334 | ```clojure 335 | (defn group-watcher [x] 336 | (let [group (zk/children client group-name :watcher group-watcher)] 337 | (prn "Group members: " group))) 338 | ``` 339 | 340 | Create a new node for this member and add a watcher for changes to the 341 | children of "/example-group". 342 | 343 | ```clojure 344 | (defn join-group [name] 345 | (do (zk/create client (str group-name "/" name)) 346 | (zk/children client group-name :watcher group-watcher))) 347 | ``` 348 | 349 | ### Run this Example 350 | 351 | ```clojure 352 | (use 'examples.group-membership) 353 | (join-group "bob") 354 | ``` 355 | 356 | From another REPL run: 357 | 358 | ```clojure 359 | (use 'examples.group-membership) 360 | (join-group "sue") 361 | ``` 362 | 363 | And from another REPL run: 364 | 365 | ```clojure 366 | (use 'examples.group-membership) 367 | (join-group "dan") 368 | ``` 369 | 370 | Each REPL will print the group members as each one joins the 371 | group. Kill any process and the remaining processes will print the 372 | remaining group members. 373 | 374 | 375 | ## Leader Election Example 376 | 377 | ```clojure 378 | (def root-znode "/election") 379 | 380 | (def client (zk/connect "127.0.0.1:2181")) 381 | 382 | (when-not (zk/exists client root-znode) 383 | (zk/create client root-znode :persistent? true)) 384 | 385 | (defn node-from-path [path] 386 | (.substring path (inc (count root-znode)))) 387 | 388 | (declare elect-leader) 389 | ``` 390 | 391 | The predecessor for Node A is the node that has the highest id that is 392 | < the id of Node A. watch-predecessor is called when the predecessor 393 | node changes. If this node is deleted and was the leader, then the 394 | watching node becomes the new leader. 395 | 396 | ```clojure 397 | (defn watch-predecessor [me pred leader {:keys [event-type path]}] 398 | (if (and (= event-type :NodeDeleted) (= (node-from-path path) leader)) 399 | (println "I am the leader!") 400 | (if-not (zk/exists client (str root-znode "/" pred) 401 | :watcher (partial watch-predecessor me pred leader)) 402 | (elect-leader me)))) 403 | 404 | (defn predecessor [me coll] 405 | (ffirst (filter #(= (second %) me) (partition 2 1 coll)))) 406 | ``` 407 | 408 | If the node associated with the current process is not the leader then 409 | add a watch to the predecessor. 410 | 411 | ```clojure 412 | (defn elect-leader [me] 413 | (let [members (util/sort-sequential-nodes (zk/children client root-znode)) 414 | leader (first members)] 415 | (print "I am" me) 416 | (if (= me leader) 417 | (println " and I am the leader!") 418 | (let [pred (predecessor me members)] 419 | (println " and my predecessor is:" pred) 420 | (if-not (zk/exists client (str root-znode "/" pred) 421 | :watcher (partial watch-predecessor me pred leader)) 422 | (elect-leader me)))))) 423 | 424 | (defn join-group [] 425 | (let [me (node-from-path (zk/create client (str root-znode "/n-") :sequential? true))] 426 | (elect-leader me))) 427 | ``` 428 | 429 | Evaluate the following forms in any number of REPLs and then kill each one 430 | in any order. 431 | 432 | ```clojure 433 | (use 'examples.leader-election) 434 | (join-group) 435 | ``` 436 | 437 | 438 | ## Barrier Example 439 | 440 | Distributed systems use barriers to block processing of a set of nodes until a condition is met at which time all the nodes are allowed to proceed. 441 | 442 | The following is an implementation of a double barrier based on the algorithm from the ZooKeeper Recipes page. 443 | 444 | ```clojure 445 | (require '[zookeeper :as zk]) 446 | (import '(java.net InetAddress)) 447 | 448 | (defn enter-barrier 449 | ([client n f & {:keys [barrier-node proc-name double-barrier?] 450 | :or {barrier-node "/barrier" 451 | proc-name (.getCanonicalHostName (InetAddress/getLocalHost)) 452 | double-barrier? true}}] 453 | (let [mutex (Object.) 454 | watcher (fn [event] (locking mutex (.notify mutex)))] 455 | (locking mutex 456 | (zk/create-all client (str barrier-node "/" proc-name)) 457 | (if (>= (count (zk/children client barrier-node)) n) 458 | (zk/create client (str barrier-node "/ready") :async? true) 459 | (do (zk/exists client (str barrier-node "/ready") :watcher watcher :async? true) 460 | (.wait mutex))) 461 | (let [results (f)] 462 | (if double-barrier? 463 | (exit-barrier client :barrier-node barrier-node :proc-name proc-name) 464 | (zk/delete-all client barrier-node)) 465 | results))))) 466 | ``` 467 | 468 | If the :double-barrier? option is set to true, then exit-barrier is called which blocks until all the processes have completed. 469 | 470 | ```clojure 471 | (defn exit-barrier 472 | ([client & {:keys [barrier-node proc-name] 473 | :or {barrier-node "/barrier" 474 | proc-name (.getCanonicalHostName (InetAddress/getLocalHost))}}] 475 | (let [mutex (Object.) 476 | watcher (fn [event] (locking mutex (.notify mutex)))] 477 | (zk/delete client (str barrier-node "/ready")) 478 | (locking mutex 479 | (loop [] 480 | (when-let [children (seq (sort (or (zk/children client barrier-node) nil)))] 481 | (cond 482 | ;; the last node deletes itself and the barrier node, letting all the processes exit 483 | (= (count children) 1) 484 | (zk/delete-all client barrier-node) 485 | ;; first node watches the second, waiting for it to be deleted 486 | (= proc-name (first children)) 487 | (do (when (zk/exists client 488 | (str barrier-node "/" (second children)) 489 | :watcher watcher) 490 | (.wait mutex)) 491 | (recur)) 492 | ;; rest of the nodes delete their own node, and then watch the 493 | ;; first node, waiting for it to be deleted 494 | :else 495 | (do (zk/delete client (str barrier-node "/" proc-name)) 496 | (when (zk/exists client 497 | (str barrier-node "/" (first children)) 498 | :watcher watcher) 499 | (.wait mutex)) 500 | (recur))))))))) 501 | ``` 502 | 503 | ### Example Usage 504 | 505 | ```clojure 506 | (require '[zookeeper :as zk]) 507 | (use 'examples.barrier) 508 | (def client (zk/connect "127.0.0.1:2181")) 509 | 510 | (enter-barrier client 2 #(println "First process is running")) 511 | ``` 512 | 513 | The call to enter-barrier will block until there are N=2 processes in the barrier. From another REPL, execute the following, and then both processes will run and exit the barrier. 514 | 515 | ```clojure 516 | (require '[zookeeper :as zk]) 517 | (use 'examples.barrier) 518 | (def client (zk/connect "127.0.0.1:2181")) 519 | 520 | (enter-barrier client 2 #(println "Second process is running") :proc-name "node2") 521 | ``` 522 | 523 | 524 | 525 | ## Running ZooKeeper 526 | 527 | Download Apache ZooKeeper from http://zookeeper.apache.org/releases.html. 528 | 529 | Unpack to $ZOOKEEPER_HOME (wherever you would like that to be). 530 | 531 | Here's an example conf file for a standalone instance, by default ZooKeeper will look for it in $ZOOKEEPER_HOME/conf/zoo.cfg 532 | 533 | ```sh 534 | # The number of milliseconds of each tick 535 | tickTime=2000 536 | 537 | # the directory where the snapshot is stored. 538 | dataDir=/var/zookeeper 539 | 540 | # the port at which the clients will connect 541 | clientPort=2181 542 | ``` 543 | 544 | Ensure that the dataDir exists and is writable. 545 | 546 | After creating and customizing the conf file, start ZooKeeper 547 | 548 | ```sh 549 | $ZOOKEEPER_HOME/bin/zkServer.sh start 550 | ``` 551 | 552 | 553 | ## Testing 554 | 555 | 'lein test' task is using an embedded instance of ZooKeeper. 556 | 557 | 558 | ## Contributing 559 | 560 | Although Zookeeper-clj is not part of Clojure-Contrib, it follows the same guidelines for contributing, which includes signing a Clojure Contributor Agreement (CA) before contributions can be accepted. 561 | 562 | 563 | ## References 564 | 565 | * ZooKeeper Website 566 | * ZooKeeper Programming Guide 567 | * ZooKeeper 3.3.3 API 568 | * ZooKeeper Tutorial 569 | * ZooKeeper: Wait-free coordination for Internet-scale systems 570 | * ZooKeeper Recipes and Solutions 571 | * Menagerie Library 572 | * Avout Library 573 | 574 | 575 | ## License 576 | 577 | zookeper-clj is Copyright © 2011 David Liebke and Relevance, Inc 578 | 579 | Distributed under the Eclipse Public License, the same as Clojure. 580 | -------------------------------------------------------------------------------- /examples/barrier.clj: -------------------------------------------------------------------------------- 1 | (ns examples.barrier 2 | (:require [zookeeper :as zk]) 3 | (:import (java.net InetAddress))) 4 | 5 | (defn exit-barrier 6 | ([client & {:keys [barrier-node proc-name] 7 | :or {barrier-node "/barrier" 8 | proc-name (.getCanonicalHostName (InetAddress/getLocalHost))}}] 9 | (let [mutex (Object.) 10 | watcher (fn [event] (locking mutex (.notify mutex)))] 11 | (zk/delete client (str barrier-node "/ready")) 12 | (locking mutex 13 | (loop [] 14 | (when-let [children (seq (sort (or (zk/children client barrier-node) nil)))] 15 | (cond 16 | ;; the last node deletes itself and the barrier node, letting all the processes exit 17 | (= (count children) 1) 18 | (zk/delete-all client barrier-node) 19 | ;; first node watches the second, waiting for it to be deleted 20 | (= proc-name (first children)) 21 | (do (when (zk/exists client 22 | (str barrier-node "/" (second children)) 23 | :watcher watcher) 24 | (.wait mutex)) 25 | (recur)) 26 | ;; rest of the nodes delete their own node, and then watch the 27 | ;; first node, waiting for it to be deleted 28 | :else 29 | (do (zk/delete client (str barrier-node "/" proc-name)) 30 | (when (zk/exists client 31 | (str barrier-node "/" (first children)) 32 | :watcher watcher) 33 | (.wait mutex)) 34 | (recur))))))))) 35 | 36 | (defn enter-barrier 37 | ([client n f & {:keys [barrier-node proc-name double-barrier?] 38 | :or {barrier-node "/barrier" 39 | proc-name (.getCanonicalHostName (InetAddress/getLocalHost)) 40 | double-barrier? true}}] 41 | (let [mutex (Object.) 42 | watcher (fn [event] (locking mutex (.notify mutex)))] 43 | (locking mutex 44 | (zk/create-all client (str barrier-node "/" proc-name)) 45 | (if (>= (count (zk/children client barrier-node)) n) 46 | (zk/create client (str barrier-node "/ready") :async? true) 47 | (do (zk/exists client (str barrier-node "/ready") :watcher watcher :async? true) 48 | (.wait mutex))) 49 | (let [results (f)] 50 | (if double-barrier? 51 | (exit-barrier client :barrier-node barrier-node :proc-name proc-name) 52 | (zk/delete-all client barrier-node)) 53 | results))))) 54 | -------------------------------------------------------------------------------- /examples/group_membership.clj: -------------------------------------------------------------------------------- 1 | (ns examples.group-membership 2 | (:require [zookeeper :as zk])) 3 | 4 | (def group-name "/example-group") 5 | 6 | (def client (zk/connect "127.0.0.1:2181")) 7 | 8 | (when-not (zk/exists client group-name) 9 | (zk/create client group-name :persistent? true)) 10 | 11 | (defn group-watcher [x] 12 | (let [group (zk/children client group-name :watcher group-watcher)] 13 | (prn "Group members: " group))) 14 | 15 | (defn join-group [name] 16 | (do (zk/create client (str group-name "/" name)) 17 | (zk/children client group-name :watcher group-watcher))) 18 | 19 | -------------------------------------------------------------------------------- /examples/leader_election.clj: -------------------------------------------------------------------------------- 1 | (ns examples.leader-election 2 | (:require [zookeeper :as zk] 3 | [zookeeper.util :as util])) 4 | 5 | (def root-znode "/election") 6 | 7 | (def client (zk/connect "127.0.0.1:2181")) 8 | 9 | (when-not (zk/exists client root-znode) 10 | (zk/create client root-znode :persistent? true)) 11 | 12 | (defn node-from-path [path] 13 | (.substring path (inc (count root-znode)))) 14 | 15 | (declare elect-leader) 16 | 17 | (defn watch-predecessor [me pred leader {:keys [event-type path]}] 18 | (if (and (= event-type :NodeDeleted) (= (node-from-path path) leader)) 19 | (println "I am the leader!") 20 | (if-not (zk/exists client (str root-znode "/" pred) 21 | :watcher (partial watch-predecessor me pred leader)) 22 | (elect-leader me)))) 23 | 24 | (defn predecessor [me coll] 25 | (ffirst (filter #(= (second %) me) (partition 2 1 coll)))) 26 | 27 | (defn elect-leader [me] 28 | (let [members (util/sort-sequential-nodes (zk/children client root-znode)) 29 | leader (first members)] 30 | (print "I am" me) 31 | (if (= me leader) 32 | (println " and I am the leader!") 33 | (let [pred (predecessor me members)] 34 | (println " and my predecessor is:" pred) 35 | (if-not (zk/exists client (str root-znode "/" pred) 36 | :watcher (partial watch-predecessor me pred leader)) 37 | (elect-leader me)))))) 38 | 39 | (defn join-group [] 40 | (let [me (node-from-path (zk/create client (str root-znode "/n-") :sequential? true))] 41 | (elect-leader me))) 42 | -------------------------------------------------------------------------------- /project.clj: -------------------------------------------------------------------------------- 1 | (defproject zookeeper-clj "0.13.0" 2 | :description "A Clojure DSL for Apache ZooKeeper" 3 | :dependencies [[org.clojure/clojure "1.11.2"] 4 | [org.apache.zookeeper/zookeeper "3.9.3"] 5 | [commons-codec "1.17.1"] 6 | [org.apache.curator/curator-test "5.7.1" :scope "test"]] 7 | :deploy-repositories {"clojars" {:sign-releases false :url "https://repo.clojars.org"}} 8 | :global-vars {*warn-on-reflection* true} 9 | :url "https://github.com/liebke/zookeeper-clj" 10 | :license {:name "Eclipse Public License" 11 | :url "http://www.eclipse.org/legal/epl-v10.html"} 12 | :plugins [[lein-ancient "0.7.0"] 13 | [lein-cljfmt "0.9.2"]]) 14 | -------------------------------------------------------------------------------- /scripts/publish.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash -e 2 | 3 | lein jar 4 | lein pom 5 | lein deploy clojars 6 | -------------------------------------------------------------------------------- /src/zookeeper.clj: -------------------------------------------------------------------------------- 1 | (ns zookeeper 2 | " 3 | Zookeeper-clj is a Clojure DSL for Apache ZooKeeper, which \"is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.\" 4 | 5 | Out of the box ZooKeeper provides name service, configuration, and group membership. From these core services, higher-level distributed concurrency abstractions can be built, including distributed locks, distributed queues, barriers, leader-election, and transaction services as described in ZooKeeper Recipes and Solutions and the paper \"ZooKeeper: Wait-free coordination for Internet-scale systems\". 6 | 7 | See examples: 8 | 9 | * http://developer.yahoo.com/blogs/hadoop/posts/2009/05/using_zookeeper_to_tame_system/ 10 | * http://archive.cloudera.com/cdh/3/zookeeper/zookeeperProgrammers.pdf 11 | 12 | " 13 | (:import (org.apache.zookeeper ZooKeeper KeeperException KeeperException$BadVersionException) 14 | (org.apache.zookeeper.data Stat Id ACL) 15 | (java.util.concurrent CountDownLatch TimeUnit) 16 | (java.util Arrays)) 17 | (:require [clojure.string :as s] 18 | [zookeeper.internal :as zi] 19 | [zookeeper.util :as util])) 20 | 21 | ;; connection functions 22 | 23 | (defn close 24 | "Closes the connection to the ZooKeeper server." 25 | ([^ZooKeeper client] (.close client))) 26 | 27 | (defn connect 28 | "Returns a ZooKeeper client." 29 | ([connection-string & {:keys [timeout-msec watcher] 30 | :or {timeout-msec 5000}}] 31 | (let [latch (CountDownLatch. 1) 32 | session-watcher (zi/make-watcher (fn [event] 33 | (when (= (:keeper-state event) :SyncConnected) 34 | (.countDown latch)) 35 | (when watcher (watcher event)))) 36 | client (ZooKeeper. connection-string timeout-msec session-watcher)] 37 | (.await latch timeout-msec TimeUnit/MILLISECONDS) 38 | (if (= (.getCount latch) 0) 39 | client 40 | (do (close client) 41 | (throw (IllegalStateException. "Cannot connect to server"))))))) 42 | 43 | (defn register-watcher 44 | "Registers a default watcher function with this connection. 45 | Overrides the watcher specified during connection." 46 | ([^ZooKeeper client watcher] 47 | (.register client (zi/make-watcher watcher)))) 48 | 49 | (defn state 50 | "Returns the client's current state. 51 | One of: 52 | :ASSOCIATING :AUTH_FAILED :CLOSED :CONNECTED :CONNECTEDREADONLY :CONNECTING :NOT_CONNECTED" 53 | ([^ZooKeeper client] 54 | (keyword (.toString (.getState client))))) 55 | 56 | ;; node existence function 57 | 58 | (defn exists 59 | "Returns the status of the given node, or nil if the node does not exist. 60 | 61 | If the watch is true and the call is successful (no exception is 62 | thrown), a watch will be left on the node with the given path. The 63 | watch will be triggered by a successful operation that creates/delete 64 | the node or sets the data on the node. 65 | 66 | If a watcher is provided the function will be called asynchronously 67 | with the provided watcher. 68 | 69 | If a callback is provided or `async?` is true, exists will be called 70 | asynchronously and return a promise. 71 | 72 | Examples: 73 | 74 | (use 'zookeeper) 75 | (def client (connect \"127.0.0.1:2181\" 76 | :wacher #(println \"event received: \" %))) 77 | 78 | (defn callback [result] 79 | (println \"got callback result: \" result)) 80 | 81 | (exists client \"/yadda\" :watch? true) 82 | (create client \"/yadda\") 83 | (exists client \"/yadda\") 84 | (def p0 (exists client \"/yadda\" :async? true)) 85 | @p0 86 | (def p1 (exists client \"/yadda\" :callback callback)) 87 | @p1 88 | " 89 | ([^ZooKeeper client ^String path & {:keys [watcher watch? async? callback context] 90 | :or {watch? false 91 | async? false 92 | context path}}] 93 | (when path 94 | (if (or async? callback) 95 | (let [prom (promise)] 96 | (zi/try* 97 | (if watcher 98 | (.exists client path 99 | (zi/make-watcher watcher) 100 | (zi/stat-callback (zi/promise-callback prom callback)) 101 | ^Object context) 102 | (.exists client path 103 | (boolean watch?) 104 | (zi/stat-callback (zi/promise-callback prom callback)) 105 | ^Object context)) 106 | (catch KeeperException e (throw e))) 107 | prom) 108 | (zi/try* 109 | (if watcher 110 | (zi/stat-to-map (.exists client path (zi/make-watcher watcher))) 111 | (zi/stat-to-map (.exists client path (boolean watch?)))) 112 | (catch KeeperException e (throw e))))))) 113 | 114 | ;; node creation functions 115 | 116 | (defn create 117 | " Creates a node, returning either the node's name, or a promise 118 | with a result map if either the :async? option is true or if a 119 | :callback function is provided. If the node already exists, 120 | create will return false. 121 | 122 | Options: 123 | 124 | :persistent? indicates if the node should be persistent 125 | :sequential? indicates if the node should be sequential 126 | :data data to associate with the node 127 | :acl access control, see the acls map 128 | :async? indicates that the create should occur asynchronously, 129 | a promise will be returned 130 | :callback indicates that the create should occur asynchronously 131 | and that this function should be called when it does, 132 | a promise will also be returned 133 | 134 | 135 | Example: 136 | 137 | (use 'zookeeper) 138 | (def client (connect \"127.0.0.1:2181\" :watcher #(println \"event received: \" %))) 139 | 140 | (defn callback [result] 141 | (println \"got callback result: \" result)) 142 | 143 | ;; first delete the baz node if it exists 144 | (delete-all client \"/baz\") 145 | ;; now create a persistent parent node, /baz, and two child nodes 146 | (def p0 (create client \"/baz\" :callback callback :persistent? true)) 147 | @p0 148 | (def p1 (create client \"/baz/1\" :callback callback)) 149 | @p1 150 | (def p2 (create client \"/baz/2-\" :async? true :sequential? true)) 151 | @p2 152 | (create client \"/baz/3\") 153 | 154 | " 155 | ([^ZooKeeper client path & {:keys [data acl persistent? sequential? context callback async?] 156 | :or {persistent? false 157 | sequential? false 158 | acl (zi/acls :open-acl-unsafe) 159 | context path 160 | async? false}}] 161 | (if (or async? callback) 162 | (let [prom (promise)] 163 | (zi/try* 164 | (zi/create client path data acl 165 | (zi/create-modes {:persistent? persistent?, :sequential? sequential?}) 166 | (zi/string-callback (zi/promise-callback prom callback)) 167 | context) 168 | (catch KeeperException e (throw e))) 169 | prom) 170 | (zi/try* 171 | (zi/create client path data acl 172 | (zi/create-modes {:persistent? persistent?, :sequential? sequential?})) 173 | (catch org.apache.zookeeper.KeeperException$NodeExistsException _ 174 | false) 175 | (catch KeeperException e (throw e)))))) 176 | 177 | (defn create-all 178 | "Create a node and all of its parents. The last node will be ephemeral, 179 | and its parents will be persistent. 180 | 181 | Options like :persistent? :sequential? and :acl will only be applied 182 | to the last child node. 183 | 184 | Examples: 185 | (delete-all client \"/foo\") 186 | (create-all client \"/foo/bar/baz\" :persistent? true) 187 | (create-all client \"/foo/bar/baz/n-\" :sequential? true) 188 | 189 | 190 | " 191 | ([^ZooKeeper client path & options] 192 | (when path 193 | (loop [result-path "" [dir & children] (rest (s/split path #"/"))] 194 | (if dir 195 | (let [node (str result-path "/" dir)] 196 | (if (exists client node) 197 | (recur node children) 198 | (recur (or (if (seq children) 199 | (create client node :persistent? true) 200 | (apply create client node options)) 201 | node) 202 | children))) 203 | result-path))))) 204 | 205 | ;; children functions 206 | 207 | (defn children 208 | "Returns a sequence of child node name for the given node, nil if it 209 | has no children or false if the node does not exist. 210 | 211 | Examples: 212 | 213 | (use 'zookeeper) 214 | (def client (connect \"127.0.0.1:2181\" :watcher #(println \"event received: \" %))) 215 | 216 | (defn callback [result] 217 | (println \"got callback result: \" result)) 218 | 219 | (delete-all client \"/foo\") 220 | (create client \"/foo\" :persistent? true) 221 | (repeatedly 5 #(create client \"/foo/child-\" :sequential? true)) 222 | 223 | (children client \"/foo\") 224 | (def p0 (children client \"/foo\" :async? true)) 225 | @p0 226 | (def p1 (children client \"/foo\" :callback callback)) 227 | @p1 228 | (def p2 (children client \"/foo\" :async? true :watch? true)) 229 | @p2 230 | (def p3 (children client \"/foo\" :async? true :watcher #(println \"watched event: \" %))) 231 | @p3 232 | 233 | " 234 | ([^ZooKeeper client ^String path & {:keys [watcher ^Boolean watch? async? callback context] 235 | :or {watch? false 236 | async? false 237 | context path}}] 238 | (when path 239 | (if (or async? callback) 240 | (let [prom (promise)] 241 | (zi/try* 242 | (if watcher 243 | (seq (.getChildren client path (zi/make-watcher watcher))) 244 | (seq (.getChildren client path watch?))) 245 | (catch KeeperException e (throw e))) 246 | prom) 247 | (zi/try* 248 | (if watcher 249 | (seq (.getChildren client path (zi/make-watcher watcher))) 250 | (seq (.getChildren client path watch?))) 251 | (catch org.apache.zookeeper.KeeperException$NoNodeException _ false) 252 | (catch KeeperException e (throw e))))))) 253 | 254 | ;; filtering childrend 255 | 256 | (defn filter-children-by-pattern 257 | "Returns a sequence of child node names filtered by the given regex pattern." 258 | ([^ZooKeeper client dir pattern] 259 | (when-let [children (children client dir)] 260 | (util/filter-nodes-by-pattern pattern children)))) 261 | 262 | (defn filter-children-by-prefix 263 | "Returns a sequence of child node names that start with the given prefix." 264 | ([^ZooKeeper client dir prefix] 265 | (filter-children-by-pattern client dir (re-pattern (str "^" prefix))))) 266 | 267 | (defn filter-children-by-suffix 268 | "Returns a sequence of child node names that end with the given suffix." 269 | ([^ZooKeeper client dir suffix] 270 | (filter-children-by-pattern client dir (re-pattern (str suffix "$"))))) 271 | 272 | ;; node deletion functions 273 | 274 | (defn delete 275 | "Delete the given node if it exists. 276 | 277 | Examples: 278 | 279 | (use 'zookeeper) 280 | (def client (connect \"127.0.0.1:2181\" :watch #(println \"event received: \" %))) 281 | 282 | (defn callback [result] 283 | (println \"got callback result: \" result)) 284 | 285 | (create client \"/foo\" :persistent? true) 286 | (create client \"/bar\" :persistent? true) 287 | 288 | (delete client \"/foo\") 289 | (def p0 (delete client \"/bar\" :callback callback)) 290 | @p0 291 | " 292 | ([^ZooKeeper client path & {:keys [version async? callback context] 293 | :or {version -1 294 | async? false 295 | context path}}] 296 | (when path 297 | (if (or async? callback) 298 | (let [prom (promise)] 299 | (zi/try* 300 | (.delete client path version (zi/void-callback (zi/promise-callback prom callback)) context) 301 | (catch KeeperException e (throw e))) 302 | prom) 303 | (zi/try* 304 | (do 305 | (.delete client path version) 306 | true) 307 | (catch org.apache.zookeeper.KeeperException$NoNodeException _ false) 308 | (catch KeeperException e (throw e))))))) 309 | 310 | (defn delete-all 311 | "Delete a node and all of its children." 312 | ([^ZooKeeper client path & options] 313 | (when path 314 | (doseq [child (or (children client path) nil)] 315 | (apply delete-all client (str path "/" child) options)) 316 | (apply delete client path options)))) 317 | 318 | (defn delete-children 319 | "Delete all of a node's children." 320 | ([^ZooKeeper client path & options] 321 | (when path 322 | (let [{:keys [sort?] :or {sort? false}} options 323 | children (or (children client path) nil)] 324 | (doseq [child (if sort? (util/sort-sequential-nodes children) children)] 325 | (apply delete-all client (str path "/" child) options)))))) 326 | 327 | ;; data functions 328 | 329 | (defn data 330 | "Returns a map with two fields, `:data` and `:stat`: 331 | 332 | - `:stat`: node status. Same as returned by [[zookeeper/exists]]. 333 | - `:data`: byte array of data saved on node. 334 | 335 | Examples: 336 | 337 | (use 'zookeeper) 338 | (def client (connect \"127.0.0.1:2181\" :watcher #(println \"event received: \" %))) 339 | 340 | (defn callback [result] 341 | (println \"got callback result: \" result)) 342 | 343 | (delete-all client \"/foo\") 344 | (create client \"/foo\" :persistent? true :data (.getBytes \"Hello World\" \"UTF-8\")) 345 | (def result (data client \"/foo\")) 346 | (String. (:data result)) 347 | (:stat result) 348 | 349 | (def p0 (data client \"/foo\" :async? true)) 350 | @p0 351 | (String. (:data @p0)) 352 | 353 | (def p1 (data client \"/foo\" :watch? true :callback callback)) 354 | @p1 355 | (String. (:data @p1)) 356 | 357 | (create client \"/foobar\" :persistent? true :data (.getBytes (pr-str {:a 1, :b 2, :c 3} \"UTF-8\"))) 358 | (read-string (String. (:data (data client \"/foobar\")))) 359 | 360 | " 361 | ([^ZooKeeper client ^String path & {:keys [watcher ^Boolean watch? async? callback context] 362 | :or {watch? false 363 | async? false 364 | context path}}] 365 | (let [stat (Stat.)] 366 | (if (or async? callback) 367 | (let [prom (promise)] 368 | (zi/try* 369 | (if watcher 370 | (.getData client path (zi/make-watcher watcher) (zi/data-callback (zi/promise-callback prom callback)) context) 371 | (.getData client path watch? (zi/data-callback (zi/promise-callback prom callback)) context)) 372 | (catch KeeperException e (throw e))) 373 | prom) 374 | {:data (zi/try* 375 | (if watcher 376 | (.getData client path (zi/make-watcher watcher) stat) 377 | (.getData client path watch? stat)) 378 | (catch KeeperException e (throw e))) 379 | :stat (zi/stat-to-map stat)})))) 380 | 381 | (defn set-data 382 | "Set the data for the node of the given path if such a node exists and 383 | the given version matches the version of the node. 384 | if the given version is -1, it matches any node's versions. 385 | Return the stat of the node. 386 | 387 | Examples: 388 | 389 | (use 'zookeeper) 390 | (def client (connect \"127.0.0.1:2181\" :watcher #(println \"event received: \" %))) 391 | 392 | (defn callback [result] 393 | (println \"got callback result: \" result)) 394 | 395 | (delete-all client \"/foo\") 396 | (create client \"/foo\" :persistent? true) 397 | 398 | (set-data client \"/foo\" (.getBytes \"Hello World\" \"UTF-8\") 0) 399 | (String. (:data (data client \"/foo\"))) 400 | 401 | 402 | (def p0 (set-data client \"/foo\" (.getBytes \"New Data\" \"UTF-8\") 0 :async? true)) 403 | @p0 404 | (String. (:data (data client \"/foo\"))) 405 | 406 | (def p1 (set-data client \"/foo\" (.getBytes \"Even Newer Data\" \"UTF-8\") 1 :callback callback)) 407 | @p1 408 | (String. (:data (data client \"/foo\"))) 409 | 410 | " 411 | ([^ZooKeeper client path data version & {:keys [async? callback context] 412 | :or {async? false 413 | context path}}] 414 | (if (or async? callback) 415 | (let [prom (promise)] 416 | (zi/try* 417 | (.setData client path data version 418 | (zi/stat-callback (zi/promise-callback prom callback)) context) 419 | (catch KeeperException e (throw e))) 420 | prom) 421 | (zi/try* 422 | (zi/stat-to-map (.setData client path data version)) 423 | (catch KeeperException e (throw e)))))) 424 | 425 | (defn compare-and-set-data 426 | "Sets the data field of the given node, only if the current data 427 | byte-array equals (Arrays/equal) the given expected-value." 428 | ([^ZooKeeper client node ^bytes expected-value new-value] 429 | (let [{:keys [^bytes data stat]} (data client node) 430 | version (:version stat)] 431 | (try 432 | (when (Arrays/equals data expected-value) 433 | (set-data client node new-value version)) 434 | (catch KeeperException$BadVersionException _ 435 | ;; try again if the data has been updated before we were able to 436 | (compare-and-set-data client node expected-value new-value)))))) 437 | 438 | ;; ACL 439 | 440 | (defn get-acl 441 | "Returns a sequence of ACLs associated with the node at the given path 442 | and its stat. 443 | 444 | Examples: 445 | 446 | (use 'zookeeper) 447 | (def client (connect \"127.0.0.1:2181\" :watcher #(println \"event received: \" %))) 448 | (add-auth-info client \"digest\" \"david:secret\") 449 | 450 | (defn callback [result] 451 | (println \"got callback result: \" result)) 452 | 453 | (delete-all client \"/foo\") 454 | (create client \"/foo\" :acl [(acl \"auth\" \"\" :read :write :create :delete)]) 455 | (get-acl client \"/foo\") 456 | 457 | (def p0 (get-acl client \"/foo\" :async? true)) 458 | 459 | (def p1 (get-acl client \"/foo\" :callback callback)) 460 | 461 | " 462 | ([^ZooKeeper client path & {:keys [async? callback context] 463 | :or {async? false 464 | context path}}] 465 | (let [stat (Stat.)] 466 | (if (or async? callback) 467 | (let [prom (promise)] 468 | (zi/try* 469 | (.getACL client path stat (zi/acl-callback (zi/promise-callback prom callback)) context) 470 | (catch KeeperException e (throw e))) 471 | prom) 472 | {:acl (zi/try* 473 | (seq (.getACL client path stat)) 474 | (catch KeeperException e (throw e))) 475 | :stat (zi/stat-to-map stat)})))) 476 | 477 | (defn add-auth-info 478 | "Add auth info to connection." 479 | ([^ZooKeeper client scheme ^String auth] 480 | (zi/try* 481 | (.addAuthInfo client scheme (if (string? auth) (.getBytes auth "UTF-8") auth)) 482 | (catch KeeperException e (throw e))))) 483 | 484 | (defn acl-id 485 | "Returns an ACL Id object with the given scheme and ID value." 486 | ([scheme id-value] 487 | (Id. scheme id-value))) 488 | 489 | (defn acl 490 | "Creates an ACL object. 491 | 492 | Examples: 493 | 494 | (use 'zookeeper) 495 | (def client (connect \"127.0.0.1:2181\" :watcher #(println \"event received: \" %))) 496 | 497 | (def open-acl-unsafe (acl \"world\" \"anyone\" :read :create :delete :admin :write)) 498 | (create client \"/mynode\" :acl [open-acl-unsafe]) 499 | 500 | (def ip-acl (acl \"ip\" \"127.0.0.1\" :read :create :delete :admin :write)) 501 | (create client \"/mynode2\" :acl [ip-acl]) 502 | 503 | (add-auth-info client \"digest\" \"david:secret\") 504 | 505 | ;; works 506 | (def auth-acl (acl \"auth\" \"\" :read :create :delete :admin :write)) 507 | (create client \"/mynode4\" :acl [auth-acl]) 508 | (data client \"/mynode4\") 509 | 510 | ;; change auth-info 511 | (add-auth-info client \"digest\" \"edgar:secret\") 512 | (data client \"/mynode4\") 513 | 514 | " 515 | ([scheme id-value perm & more-perms] 516 | (ACL. (apply zi/perm-or zi/perms perm more-perms) (acl-id scheme id-value)))) 517 | 518 | (def default-perms [:read :write :create :delete]) 519 | 520 | (defn world-acl 521 | "Create an instance of an ACL using the world scheme." 522 | ([& perms] 523 | (apply acl "world" "anyone" (or perms default-perms)))) 524 | 525 | (defn ip-acl 526 | "Create an instance of an ACL using the IP scheme." 527 | ([ip-address & perms] 528 | (apply acl "ip" ip-address (or perms default-perms)))) 529 | 530 | (defn host-acl 531 | "Create an instance of an ACL using the host scheme." 532 | ([host-suffix & perms] 533 | (apply acl "host" host-suffix (or perms default-perms)))) 534 | 535 | (defn auth-acl 536 | "Create an instance of an ACL using the auth scheme." 537 | ([& perms] 538 | (apply acl "auth" "" (or perms default-perms)))) 539 | 540 | (defn digest-acl 541 | "Create an instance of an ACL using the digest scheme." 542 | ([username password & perms] 543 | (apply acl "digest" (str username ":" password) (or perms default-perms)))) 544 | -------------------------------------------------------------------------------- /src/zookeeper/data.clj: -------------------------------------------------------------------------------- 1 | (ns zookeeper.data 2 | (:import (java.nio ByteBuffer))) 3 | 4 | (def ^:dynamic *charset* "UTF-8") 5 | 6 | (defmacro get-bytes [value size f] 7 | `(let [bytes# (make-array Byte/TYPE ~size) 8 | buf# (-> (ByteBuffer/allocateDirect ~size) 9 | (~f ~value) 10 | .clear)] 11 | (.get ^java.nio.ByteBuffer buf# bytes# 0 ~size) 12 | bytes#)) 13 | 14 | (defprotocol ByteConverter 15 | (to-bytes [this] "Converts value to a byte array")) 16 | 17 | (extend-type String 18 | ByteConverter 19 | (to-bytes [this] (.getBytes this ^String *charset*))) 20 | 21 | (extend-type Integer 22 | ByteConverter 23 | (to-bytes [i] 24 | (get-bytes i 4 .putInt))) 25 | 26 | (extend-type Double 27 | ByteConverter 28 | (to-bytes [d] 29 | (get-bytes d 8 .putDouble))) 30 | 31 | (extend-type Long 32 | ByteConverter 33 | (to-bytes [l] 34 | (get-bytes l 8 .putLong))) 35 | 36 | (extend-type Float 37 | ByteConverter 38 | (to-bytes [f] 39 | (get-bytes f 4 .putFloat))) 40 | 41 | (extend-type Short 42 | ByteConverter 43 | (to-bytes [s] 44 | (get-bytes s 2 .putShort))) 45 | 46 | (extend-type Character 47 | ByteConverter 48 | (to-bytes [c] 49 | (get-bytes c 2 .putChar))) 50 | 51 | (defn to-string 52 | ([^bytes bytes] (String. bytes ^String *charset*))) 53 | 54 | (defn to-int 55 | ([bytes] 56 | (.getInt (ByteBuffer/wrap bytes)))) 57 | 58 | (defn to-long 59 | ([bytes] 60 | (.getLong (ByteBuffer/wrap bytes)))) 61 | 62 | (defn to-double 63 | ([bytes] 64 | (.getDouble (ByteBuffer/wrap bytes)))) 65 | 66 | (defn to-float 67 | ([bytes] 68 | (.getFloat (ByteBuffer/wrap bytes)))) 69 | 70 | (defn to-short 71 | ([bytes] 72 | (.getShort (ByteBuffer/wrap bytes)))) 73 | 74 | (defn to-char 75 | ([bytes] 76 | (.getChar (ByteBuffer/wrap bytes)))) 77 | -------------------------------------------------------------------------------- /src/zookeeper/internal.clj: -------------------------------------------------------------------------------- 1 | (ns zookeeper.internal 2 | (:import 3 | (java.util List) 4 | (org.apache.zookeeper.data Stat) 5 | (org.apache.zookeeper CreateMode 6 | Watcher 7 | ZooDefs$Perms 8 | ZooDefs$Ids 9 | ZooKeeper$States 10 | ZooKeeper 11 | Watcher$Event$KeeperState 12 | Watcher$Event$EventType 13 | AsyncCallback$StringCallback 14 | AsyncCallback$VoidCallback 15 | AsyncCallback$StatCallback 16 | AsyncCallback$StatCallback 17 | AsyncCallback$Children2Callback 18 | AsyncCallback$DataCallback 19 | AsyncCallback$ACLCallback))) 20 | 21 | (defmacro try* 22 | "Unwraps the RuntimeExceptions thrown by Clojure, and rethrows its 23 | cause. Only accepts a single expression." 24 | ([expression & catches] 25 | `(try 26 | (try 27 | ~expression 28 | (catch Throwable e# (throw (or (.getCause e#) e#)))) 29 | ~@catches))) 30 | 31 | (defn stat-to-map 32 | ([^org.apache.zookeeper.data.Stat stat] 33 | ;;(long czxid, long mzxid, long ctime, long mtime, int version, int cversion, int aversion, long ephemeralOwner, int dataLength, int numChildren, long pzxid) 34 | (when stat 35 | {:czxid (.getCzxid stat) 36 | :mzxid (.getMzxid stat) 37 | :ctime (.getCtime stat) 38 | :mtime (.getMtime stat) 39 | :version (.getVersion stat) 40 | :cversion (.getCversion stat) 41 | :aversion (.getAversion stat) 42 | :ephemeralOwner (.getEphemeralOwner stat) 43 | :dataLength (.getDataLength stat) 44 | :numChildren (.getNumChildren stat) 45 | :pzxid (.getPzxid stat)}))) 46 | 47 | (defn event-to-map 48 | ([^org.apache.zookeeper.WatchedEvent event] 49 | (when event 50 | {:event-type (keyword (.name (.getType event))) 51 | :keeper-state (keyword (.name (.getState event))) 52 | :path (.getPath event)}))) 53 | 54 | (defn create 55 | "Internal create wrapper to avoid reflection." 56 | ([^ZooKeeper client 57 | ^String path 58 | ^bytes data 59 | ^List acl 60 | ^CreateMode mode] 61 | (.create client path data acl mode)) 62 | ([^ZooKeeper client 63 | ^String path 64 | ^bytes data 65 | ^List acl 66 | ^CreateMode mode 67 | ^AsyncCallback$StringCallback string-callback 68 | ^Object context] 69 | (.create client path data acl mode string-callback context))) 70 | 71 | ;; Watcher 72 | 73 | (defn ^Watcher make-watcher 74 | ([handler] 75 | (reify Watcher 76 | (process [this event] 77 | (handler (event-to-map event)))))) 78 | 79 | ;; Callbacks 80 | 81 | (defn ^AsyncCallback$StringCallback string-callback 82 | "This callback is used to retrieve the name of the node." 83 | ([handler] 84 | (reify AsyncCallback$StringCallback 85 | (^void processResult [this ^int return-code ^String path context ^String name] 86 | (handler {:return-code return-code 87 | :path path 88 | :context context 89 | :name name}))))) 90 | 91 | (defn ^AsyncCallback$StatCallback stat-callback 92 | "This callback is used to retrieve the stat of the node." 93 | ([handler] 94 | (reify AsyncCallback$StatCallback 95 | (^void processResult [this ^int return-code ^String path context ^Stat stat] 96 | (handler {:return-code return-code 97 | :path path 98 | :context context 99 | :stat (stat-to-map stat)}))))) 100 | 101 | (defn ^AsyncCallback$Children2Callback children-callback 102 | "This callback is used to retrieve the children of the node." 103 | ([handler] 104 | (reify AsyncCallback$Children2Callback 105 | (^void processResult [this ^int return-code ^String path context ^List children ^Stat stat] 106 | (handler {:return-code return-code 107 | :path path 108 | :context context 109 | :children (seq children) 110 | :stat (stat-to-map stat)}))))) 111 | 112 | (defn ^AsyncCallback$VoidCallback void-callback 113 | "This callback doesn't retrieve anything from the node. It is useful for 114 | some APIs that doesn't want anything sent back, e.g. 115 | ZooKeeper#sync(String, VoidCallback, Object)." 116 | ([handler] 117 | (reify AsyncCallback$VoidCallback 118 | (^void processResult [this ^int return-code ^String path context] 119 | (handler {:return-code return-code 120 | :path path 121 | :context context}))))) 122 | 123 | (defn ^AsyncCallback$DataCallback data-callback 124 | "This callback is used to retrieve the data and stat of the node." 125 | ([handler] 126 | (reify AsyncCallback$DataCallback 127 | (^void processResult [this ^int return-code ^String path context ^bytes data ^Stat stat] 128 | (handler {:return-code return-code 129 | :path path 130 | :context context 131 | :data data 132 | :stat (stat-to-map stat)}))))) 133 | 134 | (defn ^AsyncCallback$ACLCallback acl-callback 135 | "This callback is used to retrieve the ACL and stat of the node." 136 | ([handler] 137 | (reify AsyncCallback$ACLCallback 138 | (^void processResult [this ^int return-code ^String path context ^List acl ^Stat stat] 139 | (handler {:return-code return-code 140 | :path path 141 | :context context 142 | :acl (seq acl) 143 | :stat (stat-to-map stat)}))))) 144 | 145 | (defn promise-callback 146 | ([prom callback-fn] 147 | (fn [result] 148 | (deliver prom result) 149 | (when callback-fn 150 | (callback-fn result))))) 151 | 152 | ;; states 153 | 154 | (def create-modes {;; The znode will not be automatically deleted upon client's disconnect 155 | {:persistent? true, :sequential? false} CreateMode/PERSISTENT 156 | ;; The znode will be deleted upon the client's disconnect, and its name will be appended with a monotonically increasing number 157 | {:persistent? false, :sequential? true} CreateMode/EPHEMERAL_SEQUENTIAL 158 | ;; The znode will be deleted upon the client's disconnect 159 | {:persistent? false, :sequential? false} CreateMode/EPHEMERAL 160 | ;; The znode will not be automatically deleted upon client's disconnect, and its name will be appended with a monotonically increasing number 161 | {:persistent? true, :sequential? true} CreateMode/PERSISTENT_SEQUENTIAL}) 162 | 163 | ;; ACL 164 | 165 | (def ^:dynamic perms {:write ZooDefs$Perms/WRITE 166 | :read ZooDefs$Perms/READ 167 | :delete ZooDefs$Perms/DELETE 168 | :create ZooDefs$Perms/CREATE 169 | :admin ZooDefs$Perms/ADMIN}) 170 | 171 | (defn perm-or 172 | " 173 | Examples: 174 | 175 | (use 'zookeeper) 176 | (perm-or *perms* :read :write :create) 177 | " 178 | ([perms & perm-keys] 179 | (apply bit-or (vals (select-keys perms perm-keys))))) 180 | 181 | (def acls {:open-acl-unsafe ZooDefs$Ids/OPEN_ACL_UNSAFE ;; This is a completely open ACL 182 | :anyone-id-unsafe ZooDefs$Ids/ANYONE_ID_UNSAFE ;; This Id represents anyone 183 | :auth-ids ZooDefs$Ids/AUTH_IDS ;; This Id is only usable to set ACLs 184 | :creator-all-acl ZooDefs$Ids/CREATOR_ALL_ACL ;; This ACL gives the creators authentication id's all permissions 185 | :read-all-acl ZooDefs$Ids/READ_ACL_UNSAFE ;; This ACL gives the world the ability to read 186 | }) 187 | 188 | (defn event-types 189 | ":NodeDeleted :NodeDataChanged :NodeCreated :NodeChildrenChanged :None" 190 | ([] (into #{} (map #(keyword (.name ^Watcher$Event$EventType %)) (Watcher$Event$EventType/values))))) 191 | 192 | (defn keeper-states 193 | ":AuthFailed :Unknown :SyncConnected :Disconnected :Expired :NoSyncConnected" 194 | ([] (into #{} (map #(keyword (.name ^Watcher$Event$KeeperState %)) (Watcher$Event$KeeperState/values))))) 195 | 196 | (defn client-states 197 | ":AUTH_FAILED :CLOSED :CONNECTED :ASSOCIATING :CONNECTING" 198 | ([] (into #{} (map #(keyword (.toString ^ZooKeeper$States %)) (ZooKeeper$States/values))))) 199 | -------------------------------------------------------------------------------- /src/zookeeper/server.clj: -------------------------------------------------------------------------------- 1 | (ns zookeeper.server 2 | (:import (org.apache.zookeeper.server ZooKeeperServerMain 3 | ServerConfig))) 4 | 5 | (defn server-config 6 | ([filename] 7 | (doto (ServerConfig.) 8 | (.parse filename)))) 9 | 10 | (defn start-server 11 | ([config-filename] 12 | (-> (ZooKeeperServerMain.) 13 | (.runFromConfig (server-config config-filename))))) 14 | -------------------------------------------------------------------------------- /src/zookeeper/util.clj: -------------------------------------------------------------------------------- 1 | (ns zookeeper.util 2 | (:import (org.apache.commons.codec.digest DigestUtils) 3 | (org.apache.commons.codec.binary Base64))) 4 | 5 | (defn extract-id 6 | "Returns an integer id associated with a sequential node" 7 | ([child-path] 8 | (let [zk-seq-length 10] 9 | (Integer. (subs child-path 10 | (- (count child-path) zk-seq-length) 11 | (count child-path)))))) 12 | 13 | (defn index-sequential-nodes 14 | "Sorts a list of sequential child nodes." 15 | ([unsorted-nodes] 16 | (when (seq unsorted-nodes) 17 | (map (fn [node] [(extract-id node) node]) unsorted-nodes)))) 18 | 19 | (defn sort-sequential-nodes 20 | "Sorts a list of sequential child nodes." 21 | ([unsorted-nodes] 22 | (sort-sequential-nodes < unsorted-nodes)) 23 | ([comp unsorted-nodes] 24 | (map second (sort-by first comp (index-sequential-nodes unsorted-nodes))))) 25 | 26 | (defn filter-nodes-by-pattern 27 | ([pattern nodes] 28 | (filter #(re-find pattern %) nodes))) 29 | 30 | (defn filter-nodes-by-prefix 31 | ([prefix nodes] 32 | (filter-nodes-by-pattern (re-pattern (str "^" prefix)) nodes))) 33 | 34 | (defn hash-password 35 | " Returns a base64 encoded string of a SHA-1 digest of the given password string. 36 | 37 | Examples: 38 | 39 | (hash-password \"secret\") 40 | 41 | " 42 | ([^String password] 43 | (Base64/encodeBase64String (DigestUtils/sha password)))) -------------------------------------------------------------------------------- /test/zookeeper/test/data_test.clj: -------------------------------------------------------------------------------- 1 | (ns zookeeper.test.data_test 2 | (:use [zookeeper.data] 3 | [clojure.test])) 4 | 5 | (deftest to-from-bytes 6 | (is (= "hello" (to-string (to-bytes "hello")))) 7 | (is (= 1234 (to-int (to-bytes (Integer/valueOf 1234))))) 8 | (is (= 1234 (to-long (to-bytes 1234)))) 9 | (is (= 123.456 (to-double (to-bytes 123.456)))) 10 | (is (= (float 123.456) (to-float (to-bytes (float 123.456))))) 11 | (is (= 12 (to-short (to-bytes (short 12))))) 12 | (is (= \a (to-char (to-bytes \a))))) 13 | -------------------------------------------------------------------------------- /test/zookeeper/test/zookeeper_test.clj: -------------------------------------------------------------------------------- 1 | (ns zookeeper.test.zookeeper-test 2 | (:use [zookeeper] 3 | [clojure.test]) 4 | (:import [java.util UUID] 5 | [org.apache.curator.test TestingServer] 6 | [ch.qos.logback.classic Level Logger])) 7 | 8 | (defn setup-embedded-zk [f] 9 | (.setLevel ^Logger (org.slf4j.LoggerFactory/getLogger (Logger/ROOT_LOGGER_NAME)) Level/ERROR) 10 | (let [server (TestingServer. 2181)] 11 | (do (f) 12 | (.close server)))) 13 | 14 | (use-fixtures :once setup-embedded-zk) 15 | 16 | (def connect-string "127.0.0.1:2181") 17 | 18 | (deftest dsl-test 19 | (let [parent-node (str "/test-" (UUID/randomUUID)) 20 | child-node-prefix "child-" 21 | child0 (str child-node-prefix "0000000000") 22 | child1 (str child-node-prefix "0000000001") 23 | prom0 (promise) 24 | ref0 (ref []) 25 | make-watcher (fn [prom] (fn [event] (deliver prom event))) 26 | make-connection-watcher (fn [r] (fn [event] (dosync (alter r #(conj % event))))) 27 | client (connect connect-string :watcher (make-connection-watcher ref0)) 28 | auth-client (doto (connect connect-string) (add-auth-info "digest" "david:secret")) 29 | data-string "test data"] 30 | 31 | ;; creation tests 32 | (is (= nil (exists client parent-node))) 33 | (is (= parent-node (create client parent-node :persistent? true))) 34 | (is (= false (create client parent-node :persistent? true))) 35 | (is (= :SyncConnected (:keeper-state (first @ref0)))) ;; the first event from the client will block 36 | 37 | ;; children tests 38 | (is (= 0 (:numChildren (exists client parent-node :watch? true)))) ;; will watch for delete of parent or children or data 39 | (is (nil? (children client parent-node :watch? true))) ;; will watch for change event in children 40 | (is (= (str parent-node "/" child0) (create client (str parent-node "/" child-node-prefix) 41 | :sequential? true))) 42 | (is (< 0 (:ephemeralOwner (exists client (str parent-node "/" child0) 43 | :watcher (make-watcher prom0))))) ;; add custom watcher 44 | (is (= 1 (:numChildren (exists client parent-node)))) 45 | (is (= [child0] (children client parent-node :watch? true))) ;; will watch for change event in children 46 | (is (= (str parent-node "/" child1) (create client (str parent-node "/" child-node-prefix) :sequential? true))) 47 | (is (< 0 (:ephemeralOwner (exists client (str parent-node "/" child1))))) 48 | (is (= 2 (:numChildren (exists client parent-node)))) 49 | (is (= #{child0 child1} (into #{} (children client parent-node)))) 50 | 51 | ;; data tests 52 | (is (= 1 (:version (set-data client parent-node (.getBytes data-string) 0)))) 53 | (is (= data-string (String. (:data (data client parent-node))))) 54 | (is (nil? (compare-and-set-data client parent-node (.getBytes "EXPECTED") (.getBytes "NEW VALUE")))) 55 | (is (= data-string (String. (:data (data client parent-node))))) 56 | (is (= 2 (:version (compare-and-set-data client parent-node (.getBytes data-string) (.getBytes "NEW VALUE"))))) 57 | (is (= "NEW VALUE" (String. (:data (data client parent-node))))) 58 | 59 | ;; delete tests 60 | (is (true? (delete client (str parent-node "/" child0)))) 61 | (is (= :NodeDeleted (:event-type @prom0))) ;; custom watcher event 62 | (is (= 1 (:numChildren (exists client parent-node :watch? true)))) 63 | (is (true? (delete-all client (str parent-node)))) 64 | (is (nil? (exists client parent-node))) 65 | 66 | ;; check the rest of the watcher events 67 | (is (= :NodeChildrenChanged (:event-type (nth @ref0 1)))) 68 | (is (= :NodeChildrenChanged (:event-type (nth @ref0 2)))) 69 | (is (= :NodeDataChanged (:event-type (nth @ref0 3)))) 70 | (is (= :NodeDeleted (:event-type (nth @ref0 4)))) 71 | 72 | ;; acl tests 73 | 74 | ;; create node that can only be accessed by its creator 75 | (is (= parent-node (create auth-client parent-node 76 | :persistent? true 77 | :acl [(acl "auth" "" :read :create :delete :write)]))) 78 | ;; test authorized client 79 | (is (nil? (:data (data auth-client parent-node)))) 80 | (is (nil? (children auth-client parent-node))) 81 | (is (.startsWith (create auth-client (str parent-node "/" child-node-prefix) :sequential? true) 82 | (str parent-node "/" child-node-prefix))) 83 | ;; test unauthorized client 84 | (is (map? (exists client parent-node))) ;; don't need auth to check existence 85 | (is (thrown? org.apache.zookeeper.KeeperException$NoAuthException 86 | (data client parent-node))) 87 | (is (thrown? org.apache.zookeeper.KeeperException$NoAuthException 88 | (children client parent-node))) 89 | (is (thrown? org.apache.zookeeper.KeeperException$NoAuthException 90 | (create client (str parent-node "/" child-node-prefix) :sequential? true))) 91 | 92 | ;; close the client 93 | (close client))) 94 | 95 | (deftest no-server-test 96 | (is (thrown? java.lang.IllegalStateException (connect "127.0.0.1:79" :timeout-ms 1)))) 97 | --------------------------------------------------------------------------------