├── .clj-kondo
└── config.edn
├── .gitignore
├── README.md
├── examples
├── barrier.clj
├── group_membership.clj
└── leader_election.clj
├── project.clj
├── scripts
└── publish.sh
├── src
├── zookeeper.clj
└── zookeeper
│ ├── data.clj
│ ├── internal.clj
│ ├── server.clj
│ └── util.clj
└── test
└── zookeeper
└── test
├── data_test.clj
└── zookeeper_test.clj
/.clj-kondo/config.edn:
--------------------------------------------------------------------------------
1 | {:lint-as {zookeeper.internal/try* clojure.core/try}}
2 |
--------------------------------------------------------------------------------
/.gitignore:
--------------------------------------------------------------------------------
1 | pom.xml
2 | *jar
3 | /lib/
4 | /classes/
5 | .lein-failures
6 | .lein-deps-sum
7 | .nrepl*
8 | target*
9 | *~
10 | *#
11 | .#*
12 | .clj-kondo/.cache
13 | .lsp
14 | .vscode
15 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # zookeeper-clj
2 |
3 | Zookeeper-clj is a Clojure DSL for Apache ZooKeeper, which "is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services."
4 |
5 | Out of the box ZooKeeper provides name service, configuration, and group membership. From these core services, higher-level distributed concurrency abstractions can be built, including distributed locks, distributed queues, barriers, leader-election, and transaction services as described in ZooKeeper Recipes and Solutions and the paper "ZooKeeper: Wait-free coordination for Internet-scale systems".
6 |
7 | Building these distributed concurrency abstractions is the goal of the Java-based Menagerie library and the, **soon to be released**, Clojure-based **Avout** library. Avout, in particular, provides distributed versions of Clojure's Atom and Ref concurrency primitives, as well as distributed implementations of *java.util.concurrent.lock.Lock* and *java.util.concurrent.lock.ReadWriteLock*.
8 |
9 | ## Table of Contents
10 |
11 | * Getting Started
12 | * connect function
13 | * watchers
14 | * create function
15 | * asynchronous calls
16 | * exists function
17 | * children function
18 | * sequential nodes
19 | * data functions
20 | * data serialization
21 | * delete functions
22 | * acl function
23 | * Group Membership Example
24 | * Leader Election Example
25 | * Barrier Example
26 | * Running ZooKeeper
27 | * Testing
28 | * Contributing
29 | * References
30 |
31 |
32 |
33 | ## Getting Started
34 |
35 | To run these examples, first start a local instance of ZooKeeper on port 2181, see instructions below, and include zookeeper-clj as a dependency by adding the following to your Leiningen project.clj file:
36 |
37 | [](http://clojars.org/zookeeper-clj)
38 |
39 |
40 | ### connect function
41 |
42 | First require the zookeeper namespace and create a client with the connect function.
43 |
44 | ```clojure
45 | (require '[zookeeper :as zk])
46 | (def client (zk/connect "127.0.0.1:2181"))
47 | ```
48 |
49 | The connection string is the name, or IP address, and port of the ZooKeeper server. Several host:port pairs can be included as a comma seperated list. The port can be left off if it is 2181.
50 |
51 | The connection to the ZooKeeper server can be closed with the close function.
52 |
53 | ```clojure
54 | (zk/close client)
55 | ```
56 |
57 |
58 | ### watchers
59 |
60 | A watcher function that takes a single event map argument can be passed to connect, which will be invoked as a result of changes of keeper-state, or as a result of other events.
61 |
62 | ```clojure
63 | (def client (zk/connect "127.0.0.1" :watcher (fn [event] (println event))))
64 | ```
65 |
66 | if the :watch? flag is set to true when using the exists, children, or data functions, the default watcher function will be triggered under the following circumstances.
67 |
68 | * **exists**: the watch will be triggered by a successful operation that creates/deletes the node or sets the data on the node.
69 | * **children**: the watch will be triggered by a successful operation that deletes the node of the given path or creates/deletes a child under the node.
70 | * **data**: the watch will be triggered by a successful operation that sets data on the node, or deletes the node.
71 |
72 | The default watcher function can be overriden with a custom function by passing it as the :watcher argument to the exists, children, or data functions.
73 |
74 | The argument to the watcher function is a map with three keys: :event-type, :keeper-state, and :path.
75 |
76 | * **event-type**: :NodeDeleted, :NodeDataChanged, :NodeCreated, :NodeChildrenChanged, :None
77 | * **keeper-state**: :AuthFailed, :Unknown, :SyncConnected, :Disconnected, :Expired, :NoSyncConnected
78 | * **path**: the path to the node in question, may be nil
79 |
80 | NOTE: Watches are one time triggers; if you get a watch event and you want to get notified of future changes, you must set another watch.
81 |
82 |
83 | ### create function
84 |
85 | Next, create a node called "/parent-node"
86 |
87 | ```clojure
88 | (zk/create client "/parent-node" :persistent? true)
89 | ;; => "/parent-node"
90 | ```
91 |
92 | Setting the :persistent? flag to true creates a persistent node, meaning one that will persist even after the client that created it is no longer connected. By default, nodes are ephemeral (i.e. :persistent? false) and will be deleted if the client that created them is disconnected (this is key to how ZooKeeper is used to build robust distributed systems).
93 |
94 | A node must be persistent if you want it to have child nodes.
95 |
96 |
97 | ### asynchronous calls
98 |
99 | Most of the zookeeper functions can be called asynchronously by setting the :async? option to true, or by providing an explicit callback function with the :callback option. When invoked asynchronously, each function will return a promise that will eventually contain the result of the call (a map with the following keys: :return-code, :path, :context, :name).
100 |
101 | ```clojure
102 | (def result-promise (zk/create client "/parent-node" :persistent? true :async? true))
103 | ```
104 |
105 | Dereferencing the promise will block until a result is returned.
106 |
107 | ```clojure
108 | @result-promise
109 | ```
110 |
111 | If a :callback function is passed, the promise will be returned with the result map and the callback will be invoked with the same map.
112 |
113 | ```clojure
114 | (def result-promise (zk/create client "/parent-node" :persistent? true :callback (fn [result] (println result))))
115 | ```
116 |
117 |
118 | ### exists function
119 |
120 | We can check the existence of the newly created node with the exists function.
121 |
122 | ```clojure
123 | (zk/exists client "/parent-node")
124 | ```
125 |
126 | The exists function returns nil if the node does not exist, and returns a map with the following keys if it does: :numChildren, :ephemeralOwner, :cversion, :mzxid, :czxid, :dataLength, :ctime, :version, :aversion, :mtime, :pzxid. See the ZooKeeper documentation for description of each field.
127 |
128 | The exists function accepts the :watch?, :watcher, :async?, and :callback options. The watch functions will be triggered by a successful operation that creates/deletes the node or sets the data on the node.
129 |
130 |
131 | ### children function
132 |
133 | Next, create a child node for "/parent-node"
134 |
135 | ```clojure
136 | (zk/create client "/parent-node/child-node")
137 | ;; => "/parent-node/child-node"
138 | ```
139 |
140 | Since the :persistent? flag wasn't set to true, this node will be ephemeral, meaning it will be deleted if the client that created it is disconnected.
141 |
142 | A list of a node's children can be retrieved with the children function.
143 |
144 | ```clojure
145 | (zk/children client "/parent-node")
146 | ;; => ("child-node")
147 | ```
148 |
149 | If the node has no children, nil will be returned, and if the node doesn't exist, false will be returned.
150 |
151 | The children function accepts the :watch?, :watcher, :async?, and :callback options. The watch function will be triggered by a successful operation that deletes the node of the given path or creates/delete a child under the node.
152 |
153 |
154 | ### sequential nodes
155 |
156 | If the :sequential? option is set to true when a node is created, a ten digit sequential ID is appended to the name of the node (it's idiomatic to include a dash as the last character of a sequential node's name).
157 |
158 | ```clojure
159 | (zk/create-all client "/parent/child-" :sequential? true)
160 | ;; => "/parent/child-0000000000"
161 | ```
162 |
163 | The create-all function creates the parent nodes if they don't already exist, here we used it to create the "/parent" node.
164 |
165 | The sequence ID increases monotonically for a given parent directory.
166 |
167 | ```clojure
168 | (zk/create client "/parent/child-" :sequential? true)
169 | ;; => "/parent/child-0000000001"
170 |
171 | (zk/create client "/parent/child-" :sequential? true)
172 | ;; => "/parent/child-0000000002"
173 | ```
174 |
175 | The zookeeper.util namespace contains functions for extracting IDs from sequential nodes and sorting them.
176 |
177 | ```clojure
178 | (require '[zookeeper.util :as util])
179 | (util/extract-id (first (zk/children client "/parent")))
180 | ;; => 2
181 | ```
182 |
183 | The order of the child nodes return from children is arbitrary, but the nodes can be sorted with the sort-sequential-nodes function.
184 |
185 | ```clojure
186 | (util/sort-sequential-nodes (zk/children client "/parent"))
187 | ;; => ("child-0000000000" "child-0000000001" "child-0000000002")
188 | ```
189 |
190 |
191 | ### data functions
192 |
193 | Each node has a data field that can hold a byte array, which is limited to 1M is size.
194 |
195 | The set-data function is used to insert data. The set-data function takes a version number, which needs to match the current data version. The current version is a field in the map returned by the exists function.
196 |
197 | ```clojure
198 | (def version (:version (zk/exists client "/parent")))
199 |
200 | (zk/set-data client "/parent" (.getBytes "hello world" "UTF-8") version)
201 | ```
202 |
203 | The data function is used to retrieve the data stored in a node.
204 |
205 | ```clojure
206 | (zk/data client "/parent")
207 | ;; => {:data ..., :stat {...}}
208 | ```
209 |
210 | The data function returns a map with two fields, :data and :stat. The :stat value is the same map returned by the exists function. The :data value is a byte array.
211 |
212 | ```clojure
213 | (String. (:data (zk/data client "/parent")) "UTF-8")
214 | ;; => "hello world"
215 | ```
216 |
217 | The data function accepts the :watch?, :watcher, :async?, and :callback options. The watch function will be triggered by a successful operation that sets data on the node, or deletes the node.
218 |
219 |
220 | ### data serialization
221 |
222 | The zookeeper.data namespace contains functions for serializing different primitive types to and from byte arrays.
223 |
224 | ```clojure
225 | (require '[zookeeper.data :as data])
226 | (def version (:version (zk/exists client "/parent")))
227 | (zk/set-data client "/parent" (data/to-bytes 1234) version)
228 | (data/to-long (:data (zk/data client "/parent")))
229 | ;; => 1234
230 | ```
231 |
232 | The following types have been extended to support the to-bytes method: String, Integer, Double, Long, Float, Character. The following functions can be used to convert byte arrays back to their respective types: to-string, to-int, to-double, to-long, to-float, to-short, and to-char.
233 |
234 | Clojure forms can be written to and read from the data field using pr-str and read-string, respectively.
235 |
236 | ```clojure
237 | (zk/set-data client "/parent" (data/to-bytes (pr-str {:a 1, :b 2, :c 3})) 2)
238 | (read-string (data/to-string (:data (zk/data client "/parent"))))
239 | ;; => {:a 1, :b 2, :c 3}
240 | ```
241 |
242 |
243 | ### delete functions
244 |
245 | Nodes can be deleted with the delete function.
246 |
247 | ```clojure
248 | (zk/delete client "/parent/child-node")
249 | ```
250 |
251 | The delete function takes an optional version number, the delete will succeed if the node exists at the given version. the default version value is -1, which matches any version number.
252 |
253 | The delete function accepts the :async? and :callback options.
254 |
255 | Nodes that have children cannot be deleted. Two convenience functions, delete-children and delete-all, can be used to delete all of a node's children or delete a node and all of it's children, respectively.
256 |
257 | ```clojure
258 | (delete-all client "/parent")
259 | ```
260 |
261 |
262 | ### ACL functions
263 |
264 | The acl function takes a scheme, id value, and a set of permissions. The following schemes are built in.
265 |
266 | * **world** has a single id, **anyone**, that represents anyone.
267 | * **auth** doesn't use any id, represents any authenticated user.
268 | * **digest** uses a username:password string to generate an MD5 hash which is then used as an ACL ID identity. Authentication is done by sending the username:password in clear text. When used in the ACL the expression will be the username:base64 encoded SHA1 password digest.
269 | * **host** uses the client host name as an ACL ID identity. The ACL expression is a hostname suffix. For example, the ACL expression host:corp.com matches the ids host:host1.corp.com and host:host2.corp.com, but not host:host1.store.com.
270 | * **ip** uses the client host IP as an ACL ID identity. The ACL expression is of the form addr/bits where the most significant bits of addr are matched against the most significant bits of the client host IP.
271 |
272 | The folllowing permissions are supported:
273 |
274 | * **:create**: you can create a child node
275 | * **:read**: you can get data from a node and list its children.
276 | * **:write**: you can set data for a node
277 | * **:delete**: you can delete a child node
278 | * **:admin**: you can set permissions
279 |
280 | Below are examples of each ACL scheme.
281 |
282 | ```clojure
283 | (zk/acl "world" "anyone" :read :create :delete :admin :write)
284 | (zk/acl "ip" "127.0.0.1" :read :create :delete :admin :write)
285 | (zk/acl "host" "thinkrelevance.com" :admin :read :write :delete :create)
286 | (zk/acl "auth" "" :read :create :delete :admin :write)
287 | ```
288 |
289 | There are five convenience functions for creating ACLs of each scheme, world-acl, auth-acl, digest-acl, host-acl, and ip-acl.
290 |
291 | ```clojure
292 | (zk/world-acl :read :delete :write)
293 | ```
294 |
295 | When no permissions are provided, the following are used by default: :read, :create, :delete, :write -- but not :admin.
296 |
297 | ```clojure
298 | (zk/ip-acl "127.0.0.1")
299 | (zk/digest-acl "david:secret" :read :delete :write)
300 | (zk/host-acl "thinkrelevance.com" :read :delete :write)
301 | (zk/auth-acl :read :delete :write)
302 | ```
303 |
304 | A list of ACLs can be passed as an option to the create function.
305 |
306 | ```clojure
307 | (zk/create client "/protected-node" :acl [(zk/auth-acl :admin :create :read :delete :write)])
308 | ```
309 |
310 | In the above example, only the user that created the node has permissions on it. In order to authenticate a user, authentication info must be added to a client connection with the add-auth-info function.
311 |
312 | ```clojure
313 | (zk/add-auth-info client "digest" "david:secret")
314 | ```
315 |
316 | If an unauthorized client tries to access the node, a org.apache.zookeeper.KeeperException$NoAuthException exception will be thrown.
317 |
318 |
319 | ## Group Membership Example
320 |
321 | ```clojure
322 | (def group-name "/example-group")
323 |
324 | (def client (zk/connect "127.0.0.1:2181"))
325 |
326 | (when-not (zk/exists client group-name)
327 | (zk/create client group-name :persistent? true))
328 | ```
329 |
330 | This watcher will be called every time the children of the
331 | "/example-group" node are changed. Each time it is called it will
332 | print the children and add itself as the watcher.
333 |
334 | ```clojure
335 | (defn group-watcher [x]
336 | (let [group (zk/children client group-name :watcher group-watcher)]
337 | (prn "Group members: " group)))
338 | ```
339 |
340 | Create a new node for this member and add a watcher for changes to the
341 | children of "/example-group".
342 |
343 | ```clojure
344 | (defn join-group [name]
345 | (do (zk/create client (str group-name "/" name))
346 | (zk/children client group-name :watcher group-watcher)))
347 | ```
348 |
349 | ### Run this Example
350 |
351 | ```clojure
352 | (use 'examples.group-membership)
353 | (join-group "bob")
354 | ```
355 |
356 | From another REPL run:
357 |
358 | ```clojure
359 | (use 'examples.group-membership)
360 | (join-group "sue")
361 | ```
362 |
363 | And from another REPL run:
364 |
365 | ```clojure
366 | (use 'examples.group-membership)
367 | (join-group "dan")
368 | ```
369 |
370 | Each REPL will print the group members as each one joins the
371 | group. Kill any process and the remaining processes will print the
372 | remaining group members.
373 |
374 |
375 | ## Leader Election Example
376 |
377 | ```clojure
378 | (def root-znode "/election")
379 |
380 | (def client (zk/connect "127.0.0.1:2181"))
381 |
382 | (when-not (zk/exists client root-znode)
383 | (zk/create client root-znode :persistent? true))
384 |
385 | (defn node-from-path [path]
386 | (.substring path (inc (count root-znode))))
387 |
388 | (declare elect-leader)
389 | ```
390 |
391 | The predecessor for Node A is the node that has the highest id that is
392 | < the id of Node A. watch-predecessor is called when the predecessor
393 | node changes. If this node is deleted and was the leader, then the
394 | watching node becomes the new leader.
395 |
396 | ```clojure
397 | (defn watch-predecessor [me pred leader {:keys [event-type path]}]
398 | (if (and (= event-type :NodeDeleted) (= (node-from-path path) leader))
399 | (println "I am the leader!")
400 | (if-not (zk/exists client (str root-znode "/" pred)
401 | :watcher (partial watch-predecessor me pred leader))
402 | (elect-leader me))))
403 |
404 | (defn predecessor [me coll]
405 | (ffirst (filter #(= (second %) me) (partition 2 1 coll))))
406 | ```
407 |
408 | If the node associated with the current process is not the leader then
409 | add a watch to the predecessor.
410 |
411 | ```clojure
412 | (defn elect-leader [me]
413 | (let [members (util/sort-sequential-nodes (zk/children client root-znode))
414 | leader (first members)]
415 | (print "I am" me)
416 | (if (= me leader)
417 | (println " and I am the leader!")
418 | (let [pred (predecessor me members)]
419 | (println " and my predecessor is:" pred)
420 | (if-not (zk/exists client (str root-znode "/" pred)
421 | :watcher (partial watch-predecessor me pred leader))
422 | (elect-leader me))))))
423 |
424 | (defn join-group []
425 | (let [me (node-from-path (zk/create client (str root-znode "/n-") :sequential? true))]
426 | (elect-leader me)))
427 | ```
428 |
429 | Evaluate the following forms in any number of REPLs and then kill each one
430 | in any order.
431 |
432 | ```clojure
433 | (use 'examples.leader-election)
434 | (join-group)
435 | ```
436 |
437 |
438 | ## Barrier Example
439 |
440 | Distributed systems use barriers to block processing of a set of nodes until a condition is met at which time all the nodes are allowed to proceed.
441 |
442 | The following is an implementation of a double barrier based on the algorithm from the ZooKeeper Recipes page.
443 |
444 | ```clojure
445 | (require '[zookeeper :as zk])
446 | (import '(java.net InetAddress))
447 |
448 | (defn enter-barrier
449 | ([client n f & {:keys [barrier-node proc-name double-barrier?]
450 | :or {barrier-node "/barrier"
451 | proc-name (.getCanonicalHostName (InetAddress/getLocalHost))
452 | double-barrier? true}}]
453 | (let [mutex (Object.)
454 | watcher (fn [event] (locking mutex (.notify mutex)))]
455 | (locking mutex
456 | (zk/create-all client (str barrier-node "/" proc-name))
457 | (if (>= (count (zk/children client barrier-node)) n)
458 | (zk/create client (str barrier-node "/ready") :async? true)
459 | (do (zk/exists client (str barrier-node "/ready") :watcher watcher :async? true)
460 | (.wait mutex)))
461 | (let [results (f)]
462 | (if double-barrier?
463 | (exit-barrier client :barrier-node barrier-node :proc-name proc-name)
464 | (zk/delete-all client barrier-node))
465 | results)))))
466 | ```
467 |
468 | If the :double-barrier? option is set to true, then exit-barrier is called which blocks until all the processes have completed.
469 |
470 | ```clojure
471 | (defn exit-barrier
472 | ([client & {:keys [barrier-node proc-name]
473 | :or {barrier-node "/barrier"
474 | proc-name (.getCanonicalHostName (InetAddress/getLocalHost))}}]
475 | (let [mutex (Object.)
476 | watcher (fn [event] (locking mutex (.notify mutex)))]
477 | (zk/delete client (str barrier-node "/ready"))
478 | (locking mutex
479 | (loop []
480 | (when-let [children (seq (sort (or (zk/children client barrier-node) nil)))]
481 | (cond
482 | ;; the last node deletes itself and the barrier node, letting all the processes exit
483 | (= (count children) 1)
484 | (zk/delete-all client barrier-node)
485 | ;; first node watches the second, waiting for it to be deleted
486 | (= proc-name (first children))
487 | (do (when (zk/exists client
488 | (str barrier-node "/" (second children))
489 | :watcher watcher)
490 | (.wait mutex))
491 | (recur))
492 | ;; rest of the nodes delete their own node, and then watch the
493 | ;; first node, waiting for it to be deleted
494 | :else
495 | (do (zk/delete client (str barrier-node "/" proc-name))
496 | (when (zk/exists client
497 | (str barrier-node "/" (first children))
498 | :watcher watcher)
499 | (.wait mutex))
500 | (recur)))))))))
501 | ```
502 |
503 | ### Example Usage
504 |
505 | ```clojure
506 | (require '[zookeeper :as zk])
507 | (use 'examples.barrier)
508 | (def client (zk/connect "127.0.0.1:2181"))
509 |
510 | (enter-barrier client 2 #(println "First process is running"))
511 | ```
512 |
513 | The call to enter-barrier will block until there are N=2 processes in the barrier. From another REPL, execute the following, and then both processes will run and exit the barrier.
514 |
515 | ```clojure
516 | (require '[zookeeper :as zk])
517 | (use 'examples.barrier)
518 | (def client (zk/connect "127.0.0.1:2181"))
519 |
520 | (enter-barrier client 2 #(println "Second process is running") :proc-name "node2")
521 | ```
522 |
523 |
524 |
525 | ## Running ZooKeeper
526 |
527 | Download Apache ZooKeeper from http://zookeeper.apache.org/releases.html.
528 |
529 | Unpack to $ZOOKEEPER_HOME (wherever you would like that to be).
530 |
531 | Here's an example conf file for a standalone instance, by default ZooKeeper will look for it in $ZOOKEEPER_HOME/conf/zoo.cfg
532 |
533 | ```sh
534 | # The number of milliseconds of each tick
535 | tickTime=2000
536 |
537 | # the directory where the snapshot is stored.
538 | dataDir=/var/zookeeper
539 |
540 | # the port at which the clients will connect
541 | clientPort=2181
542 | ```
543 |
544 | Ensure that the dataDir exists and is writable.
545 |
546 | After creating and customizing the conf file, start ZooKeeper
547 |
548 | ```sh
549 | $ZOOKEEPER_HOME/bin/zkServer.sh start
550 | ```
551 |
552 |
553 | ## Testing
554 |
555 | 'lein test' task is using an embedded instance of ZooKeeper.
556 |
557 |
558 | ## Contributing
559 |
560 | Although Zookeeper-clj is not part of Clojure-Contrib, it follows the same guidelines for contributing, which includes signing a Clojure Contributor Agreement (CA) before contributions can be accepted.
561 |
562 |
563 | ## References
564 |
565 | * ZooKeeper Website
566 | * ZooKeeper Programming Guide
567 | * ZooKeeper 3.3.3 API
568 | * ZooKeeper Tutorial
569 | * ZooKeeper: Wait-free coordination for Internet-scale systems
570 | * ZooKeeper Recipes and Solutions
571 | * Menagerie Library
572 | * Avout Library
573 |
574 |
575 | ## License
576 |
577 | zookeper-clj is Copyright © 2011 David Liebke and Relevance, Inc
578 |
579 | Distributed under the Eclipse Public License, the same as Clojure.
580 |
--------------------------------------------------------------------------------
/examples/barrier.clj:
--------------------------------------------------------------------------------
1 | (ns examples.barrier
2 | (:require [zookeeper :as zk])
3 | (:import (java.net InetAddress)))
4 |
5 | (defn exit-barrier
6 | ([client & {:keys [barrier-node proc-name]
7 | :or {barrier-node "/barrier"
8 | proc-name (.getCanonicalHostName (InetAddress/getLocalHost))}}]
9 | (let [mutex (Object.)
10 | watcher (fn [event] (locking mutex (.notify mutex)))]
11 | (zk/delete client (str barrier-node "/ready"))
12 | (locking mutex
13 | (loop []
14 | (when-let [children (seq (sort (or (zk/children client barrier-node) nil)))]
15 | (cond
16 | ;; the last node deletes itself and the barrier node, letting all the processes exit
17 | (= (count children) 1)
18 | (zk/delete-all client barrier-node)
19 | ;; first node watches the second, waiting for it to be deleted
20 | (= proc-name (first children))
21 | (do (when (zk/exists client
22 | (str barrier-node "/" (second children))
23 | :watcher watcher)
24 | (.wait mutex))
25 | (recur))
26 | ;; rest of the nodes delete their own node, and then watch the
27 | ;; first node, waiting for it to be deleted
28 | :else
29 | (do (zk/delete client (str barrier-node "/" proc-name))
30 | (when (zk/exists client
31 | (str barrier-node "/" (first children))
32 | :watcher watcher)
33 | (.wait mutex))
34 | (recur)))))))))
35 |
36 | (defn enter-barrier
37 | ([client n f & {:keys [barrier-node proc-name double-barrier?]
38 | :or {barrier-node "/barrier"
39 | proc-name (.getCanonicalHostName (InetAddress/getLocalHost))
40 | double-barrier? true}}]
41 | (let [mutex (Object.)
42 | watcher (fn [event] (locking mutex (.notify mutex)))]
43 | (locking mutex
44 | (zk/create-all client (str barrier-node "/" proc-name))
45 | (if (>= (count (zk/children client barrier-node)) n)
46 | (zk/create client (str barrier-node "/ready") :async? true)
47 | (do (zk/exists client (str barrier-node "/ready") :watcher watcher :async? true)
48 | (.wait mutex)))
49 | (let [results (f)]
50 | (if double-barrier?
51 | (exit-barrier client :barrier-node barrier-node :proc-name proc-name)
52 | (zk/delete-all client barrier-node))
53 | results)))))
54 |
--------------------------------------------------------------------------------
/examples/group_membership.clj:
--------------------------------------------------------------------------------
1 | (ns examples.group-membership
2 | (:require [zookeeper :as zk]))
3 |
4 | (def group-name "/example-group")
5 |
6 | (def client (zk/connect "127.0.0.1:2181"))
7 |
8 | (when-not (zk/exists client group-name)
9 | (zk/create client group-name :persistent? true))
10 |
11 | (defn group-watcher [x]
12 | (let [group (zk/children client group-name :watcher group-watcher)]
13 | (prn "Group members: " group)))
14 |
15 | (defn join-group [name]
16 | (do (zk/create client (str group-name "/" name))
17 | (zk/children client group-name :watcher group-watcher)))
18 |
19 |
--------------------------------------------------------------------------------
/examples/leader_election.clj:
--------------------------------------------------------------------------------
1 | (ns examples.leader-election
2 | (:require [zookeeper :as zk]
3 | [zookeeper.util :as util]))
4 |
5 | (def root-znode "/election")
6 |
7 | (def client (zk/connect "127.0.0.1:2181"))
8 |
9 | (when-not (zk/exists client root-znode)
10 | (zk/create client root-znode :persistent? true))
11 |
12 | (defn node-from-path [path]
13 | (.substring path (inc (count root-znode))))
14 |
15 | (declare elect-leader)
16 |
17 | (defn watch-predecessor [me pred leader {:keys [event-type path]}]
18 | (if (and (= event-type :NodeDeleted) (= (node-from-path path) leader))
19 | (println "I am the leader!")
20 | (if-not (zk/exists client (str root-znode "/" pred)
21 | :watcher (partial watch-predecessor me pred leader))
22 | (elect-leader me))))
23 |
24 | (defn predecessor [me coll]
25 | (ffirst (filter #(= (second %) me) (partition 2 1 coll))))
26 |
27 | (defn elect-leader [me]
28 | (let [members (util/sort-sequential-nodes (zk/children client root-znode))
29 | leader (first members)]
30 | (print "I am" me)
31 | (if (= me leader)
32 | (println " and I am the leader!")
33 | (let [pred (predecessor me members)]
34 | (println " and my predecessor is:" pred)
35 | (if-not (zk/exists client (str root-znode "/" pred)
36 | :watcher (partial watch-predecessor me pred leader))
37 | (elect-leader me))))))
38 |
39 | (defn join-group []
40 | (let [me (node-from-path (zk/create client (str root-znode "/n-") :sequential? true))]
41 | (elect-leader me)))
42 |
--------------------------------------------------------------------------------
/project.clj:
--------------------------------------------------------------------------------
1 | (defproject zookeeper-clj "0.13.0"
2 | :description "A Clojure DSL for Apache ZooKeeper"
3 | :dependencies [[org.clojure/clojure "1.11.2"]
4 | [org.apache.zookeeper/zookeeper "3.9.3"]
5 | [commons-codec "1.17.1"]
6 | [org.apache.curator/curator-test "5.7.1" :scope "test"]]
7 | :deploy-repositories {"clojars" {:sign-releases false :url "https://repo.clojars.org"}}
8 | :global-vars {*warn-on-reflection* true}
9 | :url "https://github.com/liebke/zookeeper-clj"
10 | :license {:name "Eclipse Public License"
11 | :url "http://www.eclipse.org/legal/epl-v10.html"}
12 | :plugins [[lein-ancient "0.7.0"]
13 | [lein-cljfmt "0.9.2"]])
14 |
--------------------------------------------------------------------------------
/scripts/publish.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash -e
2 |
3 | lein jar
4 | lein pom
5 | lein deploy clojars
6 |
--------------------------------------------------------------------------------
/src/zookeeper.clj:
--------------------------------------------------------------------------------
1 | (ns zookeeper
2 | "
3 | Zookeeper-clj is a Clojure DSL for Apache ZooKeeper, which \"is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.\"
4 |
5 | Out of the box ZooKeeper provides name service, configuration, and group membership. From these core services, higher-level distributed concurrency abstractions can be built, including distributed locks, distributed queues, barriers, leader-election, and transaction services as described in ZooKeeper Recipes and Solutions and the paper \"ZooKeeper: Wait-free coordination for Internet-scale systems\".
6 |
7 | See examples:
8 |
9 | * http://developer.yahoo.com/blogs/hadoop/posts/2009/05/using_zookeeper_to_tame_system/
10 | * http://archive.cloudera.com/cdh/3/zookeeper/zookeeperProgrammers.pdf
11 |
12 | "
13 | (:import (org.apache.zookeeper ZooKeeper KeeperException KeeperException$BadVersionException)
14 | (org.apache.zookeeper.data Stat Id ACL)
15 | (java.util.concurrent CountDownLatch TimeUnit)
16 | (java.util Arrays))
17 | (:require [clojure.string :as s]
18 | [zookeeper.internal :as zi]
19 | [zookeeper.util :as util]))
20 |
21 | ;; connection functions
22 |
23 | (defn close
24 | "Closes the connection to the ZooKeeper server."
25 | ([^ZooKeeper client] (.close client)))
26 |
27 | (defn connect
28 | "Returns a ZooKeeper client."
29 | ([connection-string & {:keys [timeout-msec watcher]
30 | :or {timeout-msec 5000}}]
31 | (let [latch (CountDownLatch. 1)
32 | session-watcher (zi/make-watcher (fn [event]
33 | (when (= (:keeper-state event) :SyncConnected)
34 | (.countDown latch))
35 | (when watcher (watcher event))))
36 | client (ZooKeeper. connection-string timeout-msec session-watcher)]
37 | (.await latch timeout-msec TimeUnit/MILLISECONDS)
38 | (if (= (.getCount latch) 0)
39 | client
40 | (do (close client)
41 | (throw (IllegalStateException. "Cannot connect to server")))))))
42 |
43 | (defn register-watcher
44 | "Registers a default watcher function with this connection.
45 | Overrides the watcher specified during connection."
46 | ([^ZooKeeper client watcher]
47 | (.register client (zi/make-watcher watcher))))
48 |
49 | (defn state
50 | "Returns the client's current state.
51 | One of:
52 | :ASSOCIATING :AUTH_FAILED :CLOSED :CONNECTED :CONNECTEDREADONLY :CONNECTING :NOT_CONNECTED"
53 | ([^ZooKeeper client]
54 | (keyword (.toString (.getState client)))))
55 |
56 | ;; node existence function
57 |
58 | (defn exists
59 | "Returns the status of the given node, or nil if the node does not exist.
60 |
61 | If the watch is true and the call is successful (no exception is
62 | thrown), a watch will be left on the node with the given path. The
63 | watch will be triggered by a successful operation that creates/delete
64 | the node or sets the data on the node.
65 |
66 | If a watcher is provided the function will be called asynchronously
67 | with the provided watcher.
68 |
69 | If a callback is provided or `async?` is true, exists will be called
70 | asynchronously and return a promise.
71 |
72 | Examples:
73 |
74 | (use 'zookeeper)
75 | (def client (connect \"127.0.0.1:2181\"
76 | :wacher #(println \"event received: \" %)))
77 |
78 | (defn callback [result]
79 | (println \"got callback result: \" result))
80 |
81 | (exists client \"/yadda\" :watch? true)
82 | (create client \"/yadda\")
83 | (exists client \"/yadda\")
84 | (def p0 (exists client \"/yadda\" :async? true))
85 | @p0
86 | (def p1 (exists client \"/yadda\" :callback callback))
87 | @p1
88 | "
89 | ([^ZooKeeper client ^String path & {:keys [watcher watch? async? callback context]
90 | :or {watch? false
91 | async? false
92 | context path}}]
93 | (when path
94 | (if (or async? callback)
95 | (let [prom (promise)]
96 | (zi/try*
97 | (if watcher
98 | (.exists client path
99 | (zi/make-watcher watcher)
100 | (zi/stat-callback (zi/promise-callback prom callback))
101 | ^Object context)
102 | (.exists client path
103 | (boolean watch?)
104 | (zi/stat-callback (zi/promise-callback prom callback))
105 | ^Object context))
106 | (catch KeeperException e (throw e)))
107 | prom)
108 | (zi/try*
109 | (if watcher
110 | (zi/stat-to-map (.exists client path (zi/make-watcher watcher)))
111 | (zi/stat-to-map (.exists client path (boolean watch?))))
112 | (catch KeeperException e (throw e)))))))
113 |
114 | ;; node creation functions
115 |
116 | (defn create
117 | " Creates a node, returning either the node's name, or a promise
118 | with a result map if either the :async? option is true or if a
119 | :callback function is provided. If the node already exists,
120 | create will return false.
121 |
122 | Options:
123 |
124 | :persistent? indicates if the node should be persistent
125 | :sequential? indicates if the node should be sequential
126 | :data data to associate with the node
127 | :acl access control, see the acls map
128 | :async? indicates that the create should occur asynchronously,
129 | a promise will be returned
130 | :callback indicates that the create should occur asynchronously
131 | and that this function should be called when it does,
132 | a promise will also be returned
133 |
134 |
135 | Example:
136 |
137 | (use 'zookeeper)
138 | (def client (connect \"127.0.0.1:2181\" :watcher #(println \"event received: \" %)))
139 |
140 | (defn callback [result]
141 | (println \"got callback result: \" result))
142 |
143 | ;; first delete the baz node if it exists
144 | (delete-all client \"/baz\")
145 | ;; now create a persistent parent node, /baz, and two child nodes
146 | (def p0 (create client \"/baz\" :callback callback :persistent? true))
147 | @p0
148 | (def p1 (create client \"/baz/1\" :callback callback))
149 | @p1
150 | (def p2 (create client \"/baz/2-\" :async? true :sequential? true))
151 | @p2
152 | (create client \"/baz/3\")
153 |
154 | "
155 | ([^ZooKeeper client path & {:keys [data acl persistent? sequential? context callback async?]
156 | :or {persistent? false
157 | sequential? false
158 | acl (zi/acls :open-acl-unsafe)
159 | context path
160 | async? false}}]
161 | (if (or async? callback)
162 | (let [prom (promise)]
163 | (zi/try*
164 | (zi/create client path data acl
165 | (zi/create-modes {:persistent? persistent?, :sequential? sequential?})
166 | (zi/string-callback (zi/promise-callback prom callback))
167 | context)
168 | (catch KeeperException e (throw e)))
169 | prom)
170 | (zi/try*
171 | (zi/create client path data acl
172 | (zi/create-modes {:persistent? persistent?, :sequential? sequential?}))
173 | (catch org.apache.zookeeper.KeeperException$NodeExistsException _
174 | false)
175 | (catch KeeperException e (throw e))))))
176 |
177 | (defn create-all
178 | "Create a node and all of its parents. The last node will be ephemeral,
179 | and its parents will be persistent.
180 |
181 | Options like :persistent? :sequential? and :acl will only be applied
182 | to the last child node.
183 |
184 | Examples:
185 | (delete-all client \"/foo\")
186 | (create-all client \"/foo/bar/baz\" :persistent? true)
187 | (create-all client \"/foo/bar/baz/n-\" :sequential? true)
188 |
189 |
190 | "
191 | ([^ZooKeeper client path & options]
192 | (when path
193 | (loop [result-path "" [dir & children] (rest (s/split path #"/"))]
194 | (if dir
195 | (let [node (str result-path "/" dir)]
196 | (if (exists client node)
197 | (recur node children)
198 | (recur (or (if (seq children)
199 | (create client node :persistent? true)
200 | (apply create client node options))
201 | node)
202 | children)))
203 | result-path)))))
204 |
205 | ;; children functions
206 |
207 | (defn children
208 | "Returns a sequence of child node name for the given node, nil if it
209 | has no children or false if the node does not exist.
210 |
211 | Examples:
212 |
213 | (use 'zookeeper)
214 | (def client (connect \"127.0.0.1:2181\" :watcher #(println \"event received: \" %)))
215 |
216 | (defn callback [result]
217 | (println \"got callback result: \" result))
218 |
219 | (delete-all client \"/foo\")
220 | (create client \"/foo\" :persistent? true)
221 | (repeatedly 5 #(create client \"/foo/child-\" :sequential? true))
222 |
223 | (children client \"/foo\")
224 | (def p0 (children client \"/foo\" :async? true))
225 | @p0
226 | (def p1 (children client \"/foo\" :callback callback))
227 | @p1
228 | (def p2 (children client \"/foo\" :async? true :watch? true))
229 | @p2
230 | (def p3 (children client \"/foo\" :async? true :watcher #(println \"watched event: \" %)))
231 | @p3
232 |
233 | "
234 | ([^ZooKeeper client ^String path & {:keys [watcher ^Boolean watch? async? callback context]
235 | :or {watch? false
236 | async? false
237 | context path}}]
238 | (when path
239 | (if (or async? callback)
240 | (let [prom (promise)]
241 | (zi/try*
242 | (if watcher
243 | (seq (.getChildren client path (zi/make-watcher watcher)))
244 | (seq (.getChildren client path watch?)))
245 | (catch KeeperException e (throw e)))
246 | prom)
247 | (zi/try*
248 | (if watcher
249 | (seq (.getChildren client path (zi/make-watcher watcher)))
250 | (seq (.getChildren client path watch?)))
251 | (catch org.apache.zookeeper.KeeperException$NoNodeException _ false)
252 | (catch KeeperException e (throw e)))))))
253 |
254 | ;; filtering childrend
255 |
256 | (defn filter-children-by-pattern
257 | "Returns a sequence of child node names filtered by the given regex pattern."
258 | ([^ZooKeeper client dir pattern]
259 | (when-let [children (children client dir)]
260 | (util/filter-nodes-by-pattern pattern children))))
261 |
262 | (defn filter-children-by-prefix
263 | "Returns a sequence of child node names that start with the given prefix."
264 | ([^ZooKeeper client dir prefix]
265 | (filter-children-by-pattern client dir (re-pattern (str "^" prefix)))))
266 |
267 | (defn filter-children-by-suffix
268 | "Returns a sequence of child node names that end with the given suffix."
269 | ([^ZooKeeper client dir suffix]
270 | (filter-children-by-pattern client dir (re-pattern (str suffix "$")))))
271 |
272 | ;; node deletion functions
273 |
274 | (defn delete
275 | "Delete the given node if it exists.
276 |
277 | Examples:
278 |
279 | (use 'zookeeper)
280 | (def client (connect \"127.0.0.1:2181\" :watch #(println \"event received: \" %)))
281 |
282 | (defn callback [result]
283 | (println \"got callback result: \" result))
284 |
285 | (create client \"/foo\" :persistent? true)
286 | (create client \"/bar\" :persistent? true)
287 |
288 | (delete client \"/foo\")
289 | (def p0 (delete client \"/bar\" :callback callback))
290 | @p0
291 | "
292 | ([^ZooKeeper client path & {:keys [version async? callback context]
293 | :or {version -1
294 | async? false
295 | context path}}]
296 | (when path
297 | (if (or async? callback)
298 | (let [prom (promise)]
299 | (zi/try*
300 | (.delete client path version (zi/void-callback (zi/promise-callback prom callback)) context)
301 | (catch KeeperException e (throw e)))
302 | prom)
303 | (zi/try*
304 | (do
305 | (.delete client path version)
306 | true)
307 | (catch org.apache.zookeeper.KeeperException$NoNodeException _ false)
308 | (catch KeeperException e (throw e)))))))
309 |
310 | (defn delete-all
311 | "Delete a node and all of its children."
312 | ([^ZooKeeper client path & options]
313 | (when path
314 | (doseq [child (or (children client path) nil)]
315 | (apply delete-all client (str path "/" child) options))
316 | (apply delete client path options))))
317 |
318 | (defn delete-children
319 | "Delete all of a node's children."
320 | ([^ZooKeeper client path & options]
321 | (when path
322 | (let [{:keys [sort?] :or {sort? false}} options
323 | children (or (children client path) nil)]
324 | (doseq [child (if sort? (util/sort-sequential-nodes children) children)]
325 | (apply delete-all client (str path "/" child) options))))))
326 |
327 | ;; data functions
328 |
329 | (defn data
330 | "Returns a map with two fields, `:data` and `:stat`:
331 |
332 | - `:stat`: node status. Same as returned by [[zookeeper/exists]].
333 | - `:data`: byte array of data saved on node.
334 |
335 | Examples:
336 |
337 | (use 'zookeeper)
338 | (def client (connect \"127.0.0.1:2181\" :watcher #(println \"event received: \" %)))
339 |
340 | (defn callback [result]
341 | (println \"got callback result: \" result))
342 |
343 | (delete-all client \"/foo\")
344 | (create client \"/foo\" :persistent? true :data (.getBytes \"Hello World\" \"UTF-8\"))
345 | (def result (data client \"/foo\"))
346 | (String. (:data result))
347 | (:stat result)
348 |
349 | (def p0 (data client \"/foo\" :async? true))
350 | @p0
351 | (String. (:data @p0))
352 |
353 | (def p1 (data client \"/foo\" :watch? true :callback callback))
354 | @p1
355 | (String. (:data @p1))
356 |
357 | (create client \"/foobar\" :persistent? true :data (.getBytes (pr-str {:a 1, :b 2, :c 3} \"UTF-8\")))
358 | (read-string (String. (:data (data client \"/foobar\"))))
359 |
360 | "
361 | ([^ZooKeeper client ^String path & {:keys [watcher ^Boolean watch? async? callback context]
362 | :or {watch? false
363 | async? false
364 | context path}}]
365 | (let [stat (Stat.)]
366 | (if (or async? callback)
367 | (let [prom (promise)]
368 | (zi/try*
369 | (if watcher
370 | (.getData client path (zi/make-watcher watcher) (zi/data-callback (zi/promise-callback prom callback)) context)
371 | (.getData client path watch? (zi/data-callback (zi/promise-callback prom callback)) context))
372 | (catch KeeperException e (throw e)))
373 | prom)
374 | {:data (zi/try*
375 | (if watcher
376 | (.getData client path (zi/make-watcher watcher) stat)
377 | (.getData client path watch? stat))
378 | (catch KeeperException e (throw e)))
379 | :stat (zi/stat-to-map stat)}))))
380 |
381 | (defn set-data
382 | "Set the data for the node of the given path if such a node exists and
383 | the given version matches the version of the node.
384 | if the given version is -1, it matches any node's versions.
385 | Return the stat of the node.
386 |
387 | Examples:
388 |
389 | (use 'zookeeper)
390 | (def client (connect \"127.0.0.1:2181\" :watcher #(println \"event received: \" %)))
391 |
392 | (defn callback [result]
393 | (println \"got callback result: \" result))
394 |
395 | (delete-all client \"/foo\")
396 | (create client \"/foo\" :persistent? true)
397 |
398 | (set-data client \"/foo\" (.getBytes \"Hello World\" \"UTF-8\") 0)
399 | (String. (:data (data client \"/foo\")))
400 |
401 |
402 | (def p0 (set-data client \"/foo\" (.getBytes \"New Data\" \"UTF-8\") 0 :async? true))
403 | @p0
404 | (String. (:data (data client \"/foo\")))
405 |
406 | (def p1 (set-data client \"/foo\" (.getBytes \"Even Newer Data\" \"UTF-8\") 1 :callback callback))
407 | @p1
408 | (String. (:data (data client \"/foo\")))
409 |
410 | "
411 | ([^ZooKeeper client path data version & {:keys [async? callback context]
412 | :or {async? false
413 | context path}}]
414 | (if (or async? callback)
415 | (let [prom (promise)]
416 | (zi/try*
417 | (.setData client path data version
418 | (zi/stat-callback (zi/promise-callback prom callback)) context)
419 | (catch KeeperException e (throw e)))
420 | prom)
421 | (zi/try*
422 | (zi/stat-to-map (.setData client path data version))
423 | (catch KeeperException e (throw e))))))
424 |
425 | (defn compare-and-set-data
426 | "Sets the data field of the given node, only if the current data
427 | byte-array equals (Arrays/equal) the given expected-value."
428 | ([^ZooKeeper client node ^bytes expected-value new-value]
429 | (let [{:keys [^bytes data stat]} (data client node)
430 | version (:version stat)]
431 | (try
432 | (when (Arrays/equals data expected-value)
433 | (set-data client node new-value version))
434 | (catch KeeperException$BadVersionException _
435 | ;; try again if the data has been updated before we were able to
436 | (compare-and-set-data client node expected-value new-value))))))
437 |
438 | ;; ACL
439 |
440 | (defn get-acl
441 | "Returns a sequence of ACLs associated with the node at the given path
442 | and its stat.
443 |
444 | Examples:
445 |
446 | (use 'zookeeper)
447 | (def client (connect \"127.0.0.1:2181\" :watcher #(println \"event received: \" %)))
448 | (add-auth-info client \"digest\" \"david:secret\")
449 |
450 | (defn callback [result]
451 | (println \"got callback result: \" result))
452 |
453 | (delete-all client \"/foo\")
454 | (create client \"/foo\" :acl [(acl \"auth\" \"\" :read :write :create :delete)])
455 | (get-acl client \"/foo\")
456 |
457 | (def p0 (get-acl client \"/foo\" :async? true))
458 |
459 | (def p1 (get-acl client \"/foo\" :callback callback))
460 |
461 | "
462 | ([^ZooKeeper client path & {:keys [async? callback context]
463 | :or {async? false
464 | context path}}]
465 | (let [stat (Stat.)]
466 | (if (or async? callback)
467 | (let [prom (promise)]
468 | (zi/try*
469 | (.getACL client path stat (zi/acl-callback (zi/promise-callback prom callback)) context)
470 | (catch KeeperException e (throw e)))
471 | prom)
472 | {:acl (zi/try*
473 | (seq (.getACL client path stat))
474 | (catch KeeperException e (throw e)))
475 | :stat (zi/stat-to-map stat)}))))
476 |
477 | (defn add-auth-info
478 | "Add auth info to connection."
479 | ([^ZooKeeper client scheme ^String auth]
480 | (zi/try*
481 | (.addAuthInfo client scheme (if (string? auth) (.getBytes auth "UTF-8") auth))
482 | (catch KeeperException e (throw e)))))
483 |
484 | (defn acl-id
485 | "Returns an ACL Id object with the given scheme and ID value."
486 | ([scheme id-value]
487 | (Id. scheme id-value)))
488 |
489 | (defn acl
490 | "Creates an ACL object.
491 |
492 | Examples:
493 |
494 | (use 'zookeeper)
495 | (def client (connect \"127.0.0.1:2181\" :watcher #(println \"event received: \" %)))
496 |
497 | (def open-acl-unsafe (acl \"world\" \"anyone\" :read :create :delete :admin :write))
498 | (create client \"/mynode\" :acl [open-acl-unsafe])
499 |
500 | (def ip-acl (acl \"ip\" \"127.0.0.1\" :read :create :delete :admin :write))
501 | (create client \"/mynode2\" :acl [ip-acl])
502 |
503 | (add-auth-info client \"digest\" \"david:secret\")
504 |
505 | ;; works
506 | (def auth-acl (acl \"auth\" \"\" :read :create :delete :admin :write))
507 | (create client \"/mynode4\" :acl [auth-acl])
508 | (data client \"/mynode4\")
509 |
510 | ;; change auth-info
511 | (add-auth-info client \"digest\" \"edgar:secret\")
512 | (data client \"/mynode4\")
513 |
514 | "
515 | ([scheme id-value perm & more-perms]
516 | (ACL. (apply zi/perm-or zi/perms perm more-perms) (acl-id scheme id-value))))
517 |
518 | (def default-perms [:read :write :create :delete])
519 |
520 | (defn world-acl
521 | "Create an instance of an ACL using the world scheme."
522 | ([& perms]
523 | (apply acl "world" "anyone" (or perms default-perms))))
524 |
525 | (defn ip-acl
526 | "Create an instance of an ACL using the IP scheme."
527 | ([ip-address & perms]
528 | (apply acl "ip" ip-address (or perms default-perms))))
529 |
530 | (defn host-acl
531 | "Create an instance of an ACL using the host scheme."
532 | ([host-suffix & perms]
533 | (apply acl "host" host-suffix (or perms default-perms))))
534 |
535 | (defn auth-acl
536 | "Create an instance of an ACL using the auth scheme."
537 | ([& perms]
538 | (apply acl "auth" "" (or perms default-perms))))
539 |
540 | (defn digest-acl
541 | "Create an instance of an ACL using the digest scheme."
542 | ([username password & perms]
543 | (apply acl "digest" (str username ":" password) (or perms default-perms))))
544 |
--------------------------------------------------------------------------------
/src/zookeeper/data.clj:
--------------------------------------------------------------------------------
1 | (ns zookeeper.data
2 | (:import (java.nio ByteBuffer)))
3 |
4 | (def ^:dynamic *charset* "UTF-8")
5 |
6 | (defmacro get-bytes [value size f]
7 | `(let [bytes# (make-array Byte/TYPE ~size)
8 | buf# (-> (ByteBuffer/allocateDirect ~size)
9 | (~f ~value)
10 | .clear)]
11 | (.get ^java.nio.ByteBuffer buf# bytes# 0 ~size)
12 | bytes#))
13 |
14 | (defprotocol ByteConverter
15 | (to-bytes [this] "Converts value to a byte array"))
16 |
17 | (extend-type String
18 | ByteConverter
19 | (to-bytes [this] (.getBytes this ^String *charset*)))
20 |
21 | (extend-type Integer
22 | ByteConverter
23 | (to-bytes [i]
24 | (get-bytes i 4 .putInt)))
25 |
26 | (extend-type Double
27 | ByteConverter
28 | (to-bytes [d]
29 | (get-bytes d 8 .putDouble)))
30 |
31 | (extend-type Long
32 | ByteConverter
33 | (to-bytes [l]
34 | (get-bytes l 8 .putLong)))
35 |
36 | (extend-type Float
37 | ByteConverter
38 | (to-bytes [f]
39 | (get-bytes f 4 .putFloat)))
40 |
41 | (extend-type Short
42 | ByteConverter
43 | (to-bytes [s]
44 | (get-bytes s 2 .putShort)))
45 |
46 | (extend-type Character
47 | ByteConverter
48 | (to-bytes [c]
49 | (get-bytes c 2 .putChar)))
50 |
51 | (defn to-string
52 | ([^bytes bytes] (String. bytes ^String *charset*)))
53 |
54 | (defn to-int
55 | ([bytes]
56 | (.getInt (ByteBuffer/wrap bytes))))
57 |
58 | (defn to-long
59 | ([bytes]
60 | (.getLong (ByteBuffer/wrap bytes))))
61 |
62 | (defn to-double
63 | ([bytes]
64 | (.getDouble (ByteBuffer/wrap bytes))))
65 |
66 | (defn to-float
67 | ([bytes]
68 | (.getFloat (ByteBuffer/wrap bytes))))
69 |
70 | (defn to-short
71 | ([bytes]
72 | (.getShort (ByteBuffer/wrap bytes))))
73 |
74 | (defn to-char
75 | ([bytes]
76 | (.getChar (ByteBuffer/wrap bytes))))
77 |
--------------------------------------------------------------------------------
/src/zookeeper/internal.clj:
--------------------------------------------------------------------------------
1 | (ns zookeeper.internal
2 | (:import
3 | (java.util List)
4 | (org.apache.zookeeper.data Stat)
5 | (org.apache.zookeeper CreateMode
6 | Watcher
7 | ZooDefs$Perms
8 | ZooDefs$Ids
9 | ZooKeeper$States
10 | ZooKeeper
11 | Watcher$Event$KeeperState
12 | Watcher$Event$EventType
13 | AsyncCallback$StringCallback
14 | AsyncCallback$VoidCallback
15 | AsyncCallback$StatCallback
16 | AsyncCallback$StatCallback
17 | AsyncCallback$Children2Callback
18 | AsyncCallback$DataCallback
19 | AsyncCallback$ACLCallback)))
20 |
21 | (defmacro try*
22 | "Unwraps the RuntimeExceptions thrown by Clojure, and rethrows its
23 | cause. Only accepts a single expression."
24 | ([expression & catches]
25 | `(try
26 | (try
27 | ~expression
28 | (catch Throwable e# (throw (or (.getCause e#) e#))))
29 | ~@catches)))
30 |
31 | (defn stat-to-map
32 | ([^org.apache.zookeeper.data.Stat stat]
33 | ;;(long czxid, long mzxid, long ctime, long mtime, int version, int cversion, int aversion, long ephemeralOwner, int dataLength, int numChildren, long pzxid)
34 | (when stat
35 | {:czxid (.getCzxid stat)
36 | :mzxid (.getMzxid stat)
37 | :ctime (.getCtime stat)
38 | :mtime (.getMtime stat)
39 | :version (.getVersion stat)
40 | :cversion (.getCversion stat)
41 | :aversion (.getAversion stat)
42 | :ephemeralOwner (.getEphemeralOwner stat)
43 | :dataLength (.getDataLength stat)
44 | :numChildren (.getNumChildren stat)
45 | :pzxid (.getPzxid stat)})))
46 |
47 | (defn event-to-map
48 | ([^org.apache.zookeeper.WatchedEvent event]
49 | (when event
50 | {:event-type (keyword (.name (.getType event)))
51 | :keeper-state (keyword (.name (.getState event)))
52 | :path (.getPath event)})))
53 |
54 | (defn create
55 | "Internal create wrapper to avoid reflection."
56 | ([^ZooKeeper client
57 | ^String path
58 | ^bytes data
59 | ^List acl
60 | ^CreateMode mode]
61 | (.create client path data acl mode))
62 | ([^ZooKeeper client
63 | ^String path
64 | ^bytes data
65 | ^List acl
66 | ^CreateMode mode
67 | ^AsyncCallback$StringCallback string-callback
68 | ^Object context]
69 | (.create client path data acl mode string-callback context)))
70 |
71 | ;; Watcher
72 |
73 | (defn ^Watcher make-watcher
74 | ([handler]
75 | (reify Watcher
76 | (process [this event]
77 | (handler (event-to-map event))))))
78 |
79 | ;; Callbacks
80 |
81 | (defn ^AsyncCallback$StringCallback string-callback
82 | "This callback is used to retrieve the name of the node."
83 | ([handler]
84 | (reify AsyncCallback$StringCallback
85 | (^void processResult [this ^int return-code ^String path context ^String name]
86 | (handler {:return-code return-code
87 | :path path
88 | :context context
89 | :name name})))))
90 |
91 | (defn ^AsyncCallback$StatCallback stat-callback
92 | "This callback is used to retrieve the stat of the node."
93 | ([handler]
94 | (reify AsyncCallback$StatCallback
95 | (^void processResult [this ^int return-code ^String path context ^Stat stat]
96 | (handler {:return-code return-code
97 | :path path
98 | :context context
99 | :stat (stat-to-map stat)})))))
100 |
101 | (defn ^AsyncCallback$Children2Callback children-callback
102 | "This callback is used to retrieve the children of the node."
103 | ([handler]
104 | (reify AsyncCallback$Children2Callback
105 | (^void processResult [this ^int return-code ^String path context ^List children ^Stat stat]
106 | (handler {:return-code return-code
107 | :path path
108 | :context context
109 | :children (seq children)
110 | :stat (stat-to-map stat)})))))
111 |
112 | (defn ^AsyncCallback$VoidCallback void-callback
113 | "This callback doesn't retrieve anything from the node. It is useful for
114 | some APIs that doesn't want anything sent back, e.g.
115 | ZooKeeper#sync(String, VoidCallback, Object)."
116 | ([handler]
117 | (reify AsyncCallback$VoidCallback
118 | (^void processResult [this ^int return-code ^String path context]
119 | (handler {:return-code return-code
120 | :path path
121 | :context context})))))
122 |
123 | (defn ^AsyncCallback$DataCallback data-callback
124 | "This callback is used to retrieve the data and stat of the node."
125 | ([handler]
126 | (reify AsyncCallback$DataCallback
127 | (^void processResult [this ^int return-code ^String path context ^bytes data ^Stat stat]
128 | (handler {:return-code return-code
129 | :path path
130 | :context context
131 | :data data
132 | :stat (stat-to-map stat)})))))
133 |
134 | (defn ^AsyncCallback$ACLCallback acl-callback
135 | "This callback is used to retrieve the ACL and stat of the node."
136 | ([handler]
137 | (reify AsyncCallback$ACLCallback
138 | (^void processResult [this ^int return-code ^String path context ^List acl ^Stat stat]
139 | (handler {:return-code return-code
140 | :path path
141 | :context context
142 | :acl (seq acl)
143 | :stat (stat-to-map stat)})))))
144 |
145 | (defn promise-callback
146 | ([prom callback-fn]
147 | (fn [result]
148 | (deliver prom result)
149 | (when callback-fn
150 | (callback-fn result)))))
151 |
152 | ;; states
153 |
154 | (def create-modes {;; The znode will not be automatically deleted upon client's disconnect
155 | {:persistent? true, :sequential? false} CreateMode/PERSISTENT
156 | ;; The znode will be deleted upon the client's disconnect, and its name will be appended with a monotonically increasing number
157 | {:persistent? false, :sequential? true} CreateMode/EPHEMERAL_SEQUENTIAL
158 | ;; The znode will be deleted upon the client's disconnect
159 | {:persistent? false, :sequential? false} CreateMode/EPHEMERAL
160 | ;; The znode will not be automatically deleted upon client's disconnect, and its name will be appended with a monotonically increasing number
161 | {:persistent? true, :sequential? true} CreateMode/PERSISTENT_SEQUENTIAL})
162 |
163 | ;; ACL
164 |
165 | (def ^:dynamic perms {:write ZooDefs$Perms/WRITE
166 | :read ZooDefs$Perms/READ
167 | :delete ZooDefs$Perms/DELETE
168 | :create ZooDefs$Perms/CREATE
169 | :admin ZooDefs$Perms/ADMIN})
170 |
171 | (defn perm-or
172 | "
173 | Examples:
174 |
175 | (use 'zookeeper)
176 | (perm-or *perms* :read :write :create)
177 | "
178 | ([perms & perm-keys]
179 | (apply bit-or (vals (select-keys perms perm-keys)))))
180 |
181 | (def acls {:open-acl-unsafe ZooDefs$Ids/OPEN_ACL_UNSAFE ;; This is a completely open ACL
182 | :anyone-id-unsafe ZooDefs$Ids/ANYONE_ID_UNSAFE ;; This Id represents anyone
183 | :auth-ids ZooDefs$Ids/AUTH_IDS ;; This Id is only usable to set ACLs
184 | :creator-all-acl ZooDefs$Ids/CREATOR_ALL_ACL ;; This ACL gives the creators authentication id's all permissions
185 | :read-all-acl ZooDefs$Ids/READ_ACL_UNSAFE ;; This ACL gives the world the ability to read
186 | })
187 |
188 | (defn event-types
189 | ":NodeDeleted :NodeDataChanged :NodeCreated :NodeChildrenChanged :None"
190 | ([] (into #{} (map #(keyword (.name ^Watcher$Event$EventType %)) (Watcher$Event$EventType/values)))))
191 |
192 | (defn keeper-states
193 | ":AuthFailed :Unknown :SyncConnected :Disconnected :Expired :NoSyncConnected"
194 | ([] (into #{} (map #(keyword (.name ^Watcher$Event$KeeperState %)) (Watcher$Event$KeeperState/values)))))
195 |
196 | (defn client-states
197 | ":AUTH_FAILED :CLOSED :CONNECTED :ASSOCIATING :CONNECTING"
198 | ([] (into #{} (map #(keyword (.toString ^ZooKeeper$States %)) (ZooKeeper$States/values)))))
199 |
--------------------------------------------------------------------------------
/src/zookeeper/server.clj:
--------------------------------------------------------------------------------
1 | (ns zookeeper.server
2 | (:import (org.apache.zookeeper.server ZooKeeperServerMain
3 | ServerConfig)))
4 |
5 | (defn server-config
6 | ([filename]
7 | (doto (ServerConfig.)
8 | (.parse filename))))
9 |
10 | (defn start-server
11 | ([config-filename]
12 | (-> (ZooKeeperServerMain.)
13 | (.runFromConfig (server-config config-filename)))))
14 |
--------------------------------------------------------------------------------
/src/zookeeper/util.clj:
--------------------------------------------------------------------------------
1 | (ns zookeeper.util
2 | (:import (org.apache.commons.codec.digest DigestUtils)
3 | (org.apache.commons.codec.binary Base64)))
4 |
5 | (defn extract-id
6 | "Returns an integer id associated with a sequential node"
7 | ([child-path]
8 | (let [zk-seq-length 10]
9 | (Integer. (subs child-path
10 | (- (count child-path) zk-seq-length)
11 | (count child-path))))))
12 |
13 | (defn index-sequential-nodes
14 | "Sorts a list of sequential child nodes."
15 | ([unsorted-nodes]
16 | (when (seq unsorted-nodes)
17 | (map (fn [node] [(extract-id node) node]) unsorted-nodes))))
18 |
19 | (defn sort-sequential-nodes
20 | "Sorts a list of sequential child nodes."
21 | ([unsorted-nodes]
22 | (sort-sequential-nodes < unsorted-nodes))
23 | ([comp unsorted-nodes]
24 | (map second (sort-by first comp (index-sequential-nodes unsorted-nodes)))))
25 |
26 | (defn filter-nodes-by-pattern
27 | ([pattern nodes]
28 | (filter #(re-find pattern %) nodes)))
29 |
30 | (defn filter-nodes-by-prefix
31 | ([prefix nodes]
32 | (filter-nodes-by-pattern (re-pattern (str "^" prefix)) nodes)))
33 |
34 | (defn hash-password
35 | " Returns a base64 encoded string of a SHA-1 digest of the given password string.
36 |
37 | Examples:
38 |
39 | (hash-password \"secret\")
40 |
41 | "
42 | ([^String password]
43 | (Base64/encodeBase64String (DigestUtils/sha password))))
--------------------------------------------------------------------------------
/test/zookeeper/test/data_test.clj:
--------------------------------------------------------------------------------
1 | (ns zookeeper.test.data_test
2 | (:use [zookeeper.data]
3 | [clojure.test]))
4 |
5 | (deftest to-from-bytes
6 | (is (= "hello" (to-string (to-bytes "hello"))))
7 | (is (= 1234 (to-int (to-bytes (Integer/valueOf 1234)))))
8 | (is (= 1234 (to-long (to-bytes 1234))))
9 | (is (= 123.456 (to-double (to-bytes 123.456))))
10 | (is (= (float 123.456) (to-float (to-bytes (float 123.456)))))
11 | (is (= 12 (to-short (to-bytes (short 12)))))
12 | (is (= \a (to-char (to-bytes \a)))))
13 |
--------------------------------------------------------------------------------
/test/zookeeper/test/zookeeper_test.clj:
--------------------------------------------------------------------------------
1 | (ns zookeeper.test.zookeeper-test
2 | (:use [zookeeper]
3 | [clojure.test])
4 | (:import [java.util UUID]
5 | [org.apache.curator.test TestingServer]
6 | [ch.qos.logback.classic Level Logger]))
7 |
8 | (defn setup-embedded-zk [f]
9 | (.setLevel ^Logger (org.slf4j.LoggerFactory/getLogger (Logger/ROOT_LOGGER_NAME)) Level/ERROR)
10 | (let [server (TestingServer. 2181)]
11 | (do (f)
12 | (.close server))))
13 |
14 | (use-fixtures :once setup-embedded-zk)
15 |
16 | (def connect-string "127.0.0.1:2181")
17 |
18 | (deftest dsl-test
19 | (let [parent-node (str "/test-" (UUID/randomUUID))
20 | child-node-prefix "child-"
21 | child0 (str child-node-prefix "0000000000")
22 | child1 (str child-node-prefix "0000000001")
23 | prom0 (promise)
24 | ref0 (ref [])
25 | make-watcher (fn [prom] (fn [event] (deliver prom event)))
26 | make-connection-watcher (fn [r] (fn [event] (dosync (alter r #(conj % event)))))
27 | client (connect connect-string :watcher (make-connection-watcher ref0))
28 | auth-client (doto (connect connect-string) (add-auth-info "digest" "david:secret"))
29 | data-string "test data"]
30 |
31 | ;; creation tests
32 | (is (= nil (exists client parent-node)))
33 | (is (= parent-node (create client parent-node :persistent? true)))
34 | (is (= false (create client parent-node :persistent? true)))
35 | (is (= :SyncConnected (:keeper-state (first @ref0)))) ;; the first event from the client will block
36 |
37 | ;; children tests
38 | (is (= 0 (:numChildren (exists client parent-node :watch? true)))) ;; will watch for delete of parent or children or data
39 | (is (nil? (children client parent-node :watch? true))) ;; will watch for change event in children
40 | (is (= (str parent-node "/" child0) (create client (str parent-node "/" child-node-prefix)
41 | :sequential? true)))
42 | (is (< 0 (:ephemeralOwner (exists client (str parent-node "/" child0)
43 | :watcher (make-watcher prom0))))) ;; add custom watcher
44 | (is (= 1 (:numChildren (exists client parent-node))))
45 | (is (= [child0] (children client parent-node :watch? true))) ;; will watch for change event in children
46 | (is (= (str parent-node "/" child1) (create client (str parent-node "/" child-node-prefix) :sequential? true)))
47 | (is (< 0 (:ephemeralOwner (exists client (str parent-node "/" child1)))))
48 | (is (= 2 (:numChildren (exists client parent-node))))
49 | (is (= #{child0 child1} (into #{} (children client parent-node))))
50 |
51 | ;; data tests
52 | (is (= 1 (:version (set-data client parent-node (.getBytes data-string) 0))))
53 | (is (= data-string (String. (:data (data client parent-node)))))
54 | (is (nil? (compare-and-set-data client parent-node (.getBytes "EXPECTED") (.getBytes "NEW VALUE"))))
55 | (is (= data-string (String. (:data (data client parent-node)))))
56 | (is (= 2 (:version (compare-and-set-data client parent-node (.getBytes data-string) (.getBytes "NEW VALUE")))))
57 | (is (= "NEW VALUE" (String. (:data (data client parent-node)))))
58 |
59 | ;; delete tests
60 | (is (true? (delete client (str parent-node "/" child0))))
61 | (is (= :NodeDeleted (:event-type @prom0))) ;; custom watcher event
62 | (is (= 1 (:numChildren (exists client parent-node :watch? true))))
63 | (is (true? (delete-all client (str parent-node))))
64 | (is (nil? (exists client parent-node)))
65 |
66 | ;; check the rest of the watcher events
67 | (is (= :NodeChildrenChanged (:event-type (nth @ref0 1))))
68 | (is (= :NodeChildrenChanged (:event-type (nth @ref0 2))))
69 | (is (= :NodeDataChanged (:event-type (nth @ref0 3))))
70 | (is (= :NodeDeleted (:event-type (nth @ref0 4))))
71 |
72 | ;; acl tests
73 |
74 | ;; create node that can only be accessed by its creator
75 | (is (= parent-node (create auth-client parent-node
76 | :persistent? true
77 | :acl [(acl "auth" "" :read :create :delete :write)])))
78 | ;; test authorized client
79 | (is (nil? (:data (data auth-client parent-node))))
80 | (is (nil? (children auth-client parent-node)))
81 | (is (.startsWith (create auth-client (str parent-node "/" child-node-prefix) :sequential? true)
82 | (str parent-node "/" child-node-prefix)))
83 | ;; test unauthorized client
84 | (is (map? (exists client parent-node))) ;; don't need auth to check existence
85 | (is (thrown? org.apache.zookeeper.KeeperException$NoAuthException
86 | (data client parent-node)))
87 | (is (thrown? org.apache.zookeeper.KeeperException$NoAuthException
88 | (children client parent-node)))
89 | (is (thrown? org.apache.zookeeper.KeeperException$NoAuthException
90 | (create client (str parent-node "/" child-node-prefix) :sequential? true)))
91 |
92 | ;; close the client
93 | (close client)))
94 |
95 | (deftest no-server-test
96 | (is (thrown? java.lang.IllegalStateException (connect "127.0.0.1:79" :timeout-ms 1))))
97 |
--------------------------------------------------------------------------------