├── .gitignore ├── LICENSE ├── README.md ├── project.clj ├── src └── faraday_atom │ ├── core.clj │ └── impl.clj └── test └── faraday_atom ├── core_test.clj └── impl_test.clj /.gitignore: -------------------------------------------------------------------------------- 1 | /target 2 | /classes 3 | /checkouts 4 | pom.xml 5 | pom.xml.asc 6 | *.jar 7 | *.class 8 | /.lein-* 9 | /.nrepl-port 10 | .hgignore 11 | .hg/ 12 | *.iml 13 | /.idea 14 | /doc -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2015, Mix Radio 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without 5 | modification, are permitted provided that the following conditions are met: 6 | 7 | * Redistributions of source code must retain the above copyright notice, this 8 | list of conditions and the following disclaimer. 9 | 10 | * Redistributions in binary form must reproduce the above copyright notice, 11 | this list of conditions and the following disclaimer in the documentation 12 | and/or other materials provided with the distribution. 13 | 14 | * Neither the name of the {organization} nor the names of its 15 | contributors may be used to endorse or promote products derived from 16 | this software without specific prior written permission. 17 | 18 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 19 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 20 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 21 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 22 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 23 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 24 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 25 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 26 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 27 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # faraday-atom 2 | 3 | *An atom implementation for Amazon DynamoDB backed by [faraday](https://github.com/ptaoussanis/faraday).* 4 | 5 | - Atomic context around a single dynamo item. 6 | - Durable atom, persists after restarts. 7 | - Transitions via CAS using dynamo's conditional put, safe for concurrent updates. 8 | - Encodes data such that all edn data is supported yet indexes are still possible. 9 | - Due to encoding the console is still usable and your data is human readable, though byte-arrays are also supported. 10 | 11 | API docs can be found [here](http://danstone.github.io/faraday-atom) 12 | 13 | ## Why use atoms beyond a single process? 14 | 15 | Using a durable atom as your state primitive is of course useful to support clojure's epochal time model at scale in distributed systems. 16 | It surfaces the notion of state as snapshots in time that transition via pure functions. So if you care about keeping more of your code pure this library can help! 17 | 18 | It can enable certain guarantees about the correctness of your system by helping avoid race conditions and performing co-ordination services. 19 | 20 | **faraday-atom** gives you atom semantics for state that is shared across many machines, or needs to be durable. 21 | 22 | As it is of course much slower than a local atom you want to use this for state that perhaps changes 10 times a second rather than 1000 times. 23 | 24 | ## Why not zookeeper? 25 | 26 | Zookeeper is the obvious candidate for implementing the atom model as seen in [avout](http://github.com/liebke/avout). 27 | 28 | Dynamo is better suited for large amounts of state that needs strong durability and availability. You can represent an unlimited amount of atoms for a single table 29 | so you can use it for database level use cases. 30 | 31 | Dynamo can scale to effectively unlimited throughput, and its easy to do so (amazon does all the work!). 32 | 33 | Dynamo is very convenient from an operational perspective and therefore may be a more cost-effective and easy solution if you do not already have zookeeper deployed and running. 34 | 35 | On the other hand Zookeeper will likely be faster in terms of latency and can be used to implement more features like watches. 36 | 37 | ## Usage 38 | 39 | Include in your lein `project.clj` 40 | 41 | ```clojure 42 | [mixradio/faraday-atom "0.3.1"] 43 | ``` 44 | 45 | Require `faraday-atom.core` to get started 46 | 47 | ```clojure 48 | (require '[faraday-atom.core :as dynamo]) 49 | ``` 50 | 51 | Use `create-table!` to create a table which will be the durable store for your atoms. 52 | 53 | ```clojure 54 | (def client-opts 55 | {;;; For DDB Local just use some random strings here, otherwise include your 56 | ;;; production IAM keys: 57 | :access-key "" 58 | :secret-key "" 59 | 60 | ;;; You may optionally override the default endpoint if you'd like to use DDB 61 | ;;; Local or a different AWS Region (Ref. http://goo.gl/YmV80o), etc.: 62 | ;; :endpoint "http://localhost:8000" ; For DDB Local 63 | ;; :endpoint "http://dynamodb.eu-west-1.amazonaws.com" ; For EU West 1 AWS region 64 | }) 65 | 66 | (def table-client 67 | (dynamo/table-client client-opts 68 | ;;the name of the table as a keyword 69 | :test-atom-table 70 | ;;this is the key we will use to store atom identity in the table. 71 | :atom/id)) 72 | 73 | ;; Creates the table with a default read/write throughput of 8/8. 74 | (dynamo/create-table! table-client) 75 | ``` 76 | 77 | Use `item-atom` to get an atom for a dynamo item. 78 | 79 | ```clojure 80 | ;;this will get me an atom implementation for the item given by the key :foo. 81 | (def foo-atom (dynamo/item-atom table-client :foo)) 82 | ``` 83 | 84 | Use `swap!`, `deref`, `@` and `reset!` as you normally would. An exception will be raised for errors thrown by faraday 85 | or amazon except for `ConditionalCheckFailedException` in which case the operation is retried after a user configurable sleep period. 86 | To configure the retry parameters see the `item-atom` doc string. 87 | 88 | ```clojure 89 | (reset! foo-atom #{1 2 3}) 90 | ;; => #{1 2 3} 91 | 92 | @foo-atom 93 | ;; => #{1 2 3} 94 | 95 | (swap! foo-atom conj 4) 96 | ;; => #{1 2 3 4} 97 | 98 | @foo-atom 99 | ;; => #{1 2 3 4} 100 | ``` 101 | 102 | Many other helper operations for doing batch reads and puts to atom tables are provided e.g `find-items`, `put-items!`. 103 | 104 | ## License 105 | 106 | [faraday-atom is released under the 3-clause license ("New BSD License" or "Modified BSD License").](http://github.com/danstone/faraday-atom/blob/master/LICENSE) 107 | -------------------------------------------------------------------------------- /project.clj: -------------------------------------------------------------------------------- 1 | (defproject mixradio/faraday-atom "0.3.2-SNAPSHOT" 2 | :description "Provides a clojure.lang.IAtom for DynamoDB via faraday." 3 | :url "http://github.com/mixradio/faraday-atom" 4 | :license "https://github.com/mixradio/faraday-atom/blob/master/LICENSE" 5 | :dependencies [[org.clojure/clojure "1.7.0"] 6 | [com.taoensso/faraday "1.8.0"]] 7 | :profiles {:dev {:dependencies [[org.clojure/test.check "0.7.0"]] 8 | :plugins [[codox "0.8.11"]] 9 | :resource-paths ["test-resources"] 10 | :codox {:src-dir-uri "http://github.com/mixradio/faraday-atom/blob/master/" 11 | :src-linenum-anchor-prefix "L" 12 | :defaults {:doc/format :markdown}}} 13 | :dynamo-test {:plugins [[lein-dynamodb-local "0.2.6"]] 14 | :dynamodb-local {:in-memory? true}}} 15 | 16 | :test-selectors {:default (complement 17 | (some-fn :dynamo)) 18 | :dynamo :dynamo} 19 | :aliases 20 | {"test-dynamo" 21 | ["do" ["with-profiles" "+dev,+dynamo-test" "dynamodb-local" "test" ":dynamo"]]}) 22 | -------------------------------------------------------------------------------- /src/faraday_atom/core.clj: -------------------------------------------------------------------------------- 1 | (ns faraday-atom.core 2 | "A dynamo atom implementation 3 | - keys are always encoded as strings 4 | - maps and vectors are stored natively 5 | - byte arrays are stored natively as binary blobs 6 | - numerics are encoded as strings 7 | - strings are quoted 8 | - sets/lists are encoded as strings" 9 | (:require [faraday-atom.impl :as impl] 10 | [taoensso.faraday :as far]) 11 | (:refer-clojure :exclude [atom]) 12 | (:import (clojure.lang IDeref IAtom))) 13 | 14 | (defrecord TableClient [client-opts table key-name]) 15 | 16 | (defn table-client 17 | "A table client represents a 'connection' via faraday 18 | to the given table using the given key-name (hash key column)" 19 | [client-opts table key-name] 20 | (map->TableClient {:client-opts client-opts 21 | :table table 22 | :key-name key-name})) 23 | 24 | (defn- batch-result-mapping 25 | [results table key-name] 26 | (let [read (map impl/read-value (get results table)) 27 | grouped (group-by #(get % key-name) read)] 28 | (-> (reduce-kv #(assoc! %1 (impl/read-key %2) (impl/read-item (first %3) key-name)) 29 | (transient {}) grouped) 30 | persistent!))) 31 | 32 | (defn- find-item-mapping* 33 | [table-client keys opts] 34 | (when (seq keys) 35 | (let [{:keys [client-opts table key-name]} table-client 36 | query (merge opts {:prim-kvs {(impl/write-key key-name) (mapv impl/write-key keys)}}) 37 | results (far/batch-get-item client-opts {table query})] 38 | (batch-result-mapping results table key-name)))) 39 | 40 | (defn find-item-mapping 41 | "Lookups the given coll of `keys` and returns a map 42 | from key to item. If an item isn't found, the key will be missing from the 43 | result." 44 | [table-client keys] 45 | (find-item-mapping* table-client keys {:consistent? true})) 46 | 47 | (defn find-inconsistent-item-mapping 48 | "Eventually consistent version of `find-item-mapping`" 49 | [table-client keys] 50 | (find-item-mapping* table-client keys {:consistent? false})) 51 | 52 | (defn- find-items* 53 | [table-client keys opts] 54 | (let [rank (into {} (map-indexed (fn [i key] (vector key i)) keys))] 55 | (->> (sort-by (comp rank key) (seq (find-item-mapping* table-client keys opts))) 56 | (map val)))) 57 | 58 | (defn find-items 59 | "Looks up a the items given by keys. 60 | Returns a seq of items, keys that could not be found will be omitted. 61 | 62 | Each item will be read consistently." 63 | [table-client keys] 64 | (find-items* table-client keys {:consistent? true})) 65 | 66 | (defn find-inconsistent-items 67 | "Eventually consistent version of `find-items`" 68 | [table-client keys] 69 | (find-items* table-client keys {:consistent? false})) 70 | 71 | (defn find-item 72 | "Finds the item stored under `key`." 73 | [table-client key] 74 | (let [{:keys [key-name]} table-client] 75 | (-> (impl/find-raw-item table-client key) 76 | (impl/read-item key-name)))) 77 | 78 | (defn find-inconsistent-item 79 | "Eventually consistent version of `find-item`" 80 | [table-client key] 81 | (let [{:keys [client-opts table key-name]} table-client] 82 | (-> (far/get-item client-opts table {(impl/write-key key-name) (impl/write-key key)} {:consistent? false}) 83 | impl/read-value 84 | (impl/read-item key-name)))) 85 | 86 | (def ^:dynamic *default-read-throughput* 87 | "The default amount of read throughput for tables created via `create-table!`" 88 | 8) 89 | 90 | (def ^:dynamic *default-write-throughput* 91 | "The default amount of write throughput for tables created via `create-table!`" 92 | 8) 93 | 94 | (defn create-table! 95 | "Creates a table suitable for storing data according to the encoding scheme." 96 | ([table-client] 97 | (create-table! table-client *default-read-throughput* *default-write-throughput*)) 98 | ([table-client read-throughput write-throughput] 99 | (let [{:keys [client-opts table key-name]} table-client] 100 | (far/create-table client-opts table 101 | [(impl/write-key key-name) :s] 102 | {:throughput {:read read-throughput :write write-throughput}})))) 103 | (defn ensure-table! 104 | "Creates a table suitable for storing data according to the encoding scheme, 105 | unless it already exists." 106 | ([table-client] 107 | (ensure-table! table-client *default-read-throughput* *default-write-throughput*)) 108 | ([table-client read-throughput write-throughput] 109 | (let [{:keys [client-opts table key-name]} table-client] 110 | (far/ensure-table client-opts table 111 | [(impl/write-key key-name) :s] 112 | {:throughput {:read read-throughput :write write-throughput}})))) 113 | 114 | (defn put-item! 115 | "Stores the value under `key`" 116 | ([table-client key value] 117 | (let [{:keys [client-opts table key-name]} table-client] 118 | (far/put-item client-opts table (-> (impl/prepare-value value key-name) 119 | impl/write-value 120 | (impl/add-key key-name key)))))) 121 | 122 | (defn put-items! 123 | "Stores each value in `kvs` under its corresponding key. 124 | `kvs` should be a map or a seq of key value pairs." 125 | ([table-client kvs] 126 | (let [{:keys [client-opts table key-name]} table-client 127 | chunked (partition-all 25 kvs)] 128 | (doseq [chunk chunked] 129 | (impl/fixed-batch-write-item client-opts 130 | {table {:put (mapv #(-> (impl/prepare-value (second %) key-name) 131 | impl/write-value 132 | (impl/add-key key-name (first %))) 133 | chunk)}}))))) 134 | 135 | (defn delete-items! 136 | "Deletes the items whose key is listed in `keys`" 137 | [table-client keys] 138 | (let [{:keys [client-opts table key-name]} table-client 139 | chunked (partition-all 25 keys)] 140 | (doseq [chunk chunked] 141 | (far/batch-write-item client-opts {table {:delete {(impl/write-key key-name) 142 | (mapv impl/write-key chunk)}}})))) 143 | 144 | (defn delete-item! 145 | "Deletes the item stored under `key`" 146 | [table-client key] 147 | (let [{:keys [client-opts table key-name]} table-client] 148 | (far/delete-item client-opts table {(impl/write-key key-name) (impl/write-key key)}))) 149 | 150 | (defrecord ItemAtom [table-client key options] 151 | IDeref 152 | (deref [this] 153 | (find-item table-client key)) 154 | IAtom 155 | (swap [this f] 156 | (impl/swap-item!* table-client key f options)) 157 | (swap [this f x] 158 | (swap! this #(f % x))) 159 | (swap [this f x y] 160 | (swap! this #(f % x y))) 161 | (swap [this f x y args] 162 | (swap! this #(apply f % x y args))) 163 | (compareAndSet [this old new] 164 | (swap! this #(if (= old %) new %))) 165 | (reset [this v] 166 | (put-item! table-client key v) 167 | v)) 168 | 169 | (def deref-printer (get-method print-method IDeref)) 170 | 171 | (defmethod print-method ItemAtom 172 | [x writer] 173 | (deref-printer x writer)) 174 | 175 | (defn item-atom 176 | "Returns a clojure.lang.IAtom/IDeref that supports atomic state transition via conditional puts on a single dynamo item. 177 | In order to use an atom, get started by creating a compatible table via `create-table!` or `ensure-table!`. 178 | 179 | options: 180 | - `:cas-sleep-ms` the amount of time to wait in the case of contention with other CAS operations (default 500ms) 181 | - `:cas-timeout-ms` the amount of time that in the case of contention you are willing to retry for. 182 | if this elapses the `:cas-timeout-val` is returned instead of the result of the `swap!`. 183 | If you want to retry for ever, use `nil`. (default `nil`) 184 | - `:cas-timeout-val` the value to return if we timeout due to CAS contention (default `nil`). 185 | - `:discard-no-op?` if true will not send a CAS request where applying `f` to the input yields the same value as the input. 186 | (in other words the operation is assumed to have completed immediately, this is safe from a concurrency standpoint). 187 | (default `true`)" 188 | ([table-client key] 189 | (item-atom table-client key nil)) 190 | ([table-client key options] 191 | (map->ItemAtom 192 | {:table-client table-client 193 | :key key 194 | :options options}))) -------------------------------------------------------------------------------- /src/faraday_atom/impl.clj: -------------------------------------------------------------------------------- 1 | (ns faraday-atom.impl 2 | (:require [taoensso.faraday :as far] 3 | [clojure.edn :as edn] 4 | [clojure.walk :as walk]) 5 | (:import (com.amazonaws.services.dynamodbv2.model ConditionalCheckFailedException) 6 | (clojure.lang Keyword IPersistentMap IPersistentVector))) 7 | 8 | (defmulti write-value* class) 9 | 10 | (derive IPersistentVector ::native) 11 | 12 | (derive (Class/forName "[B") ::native) 13 | 14 | (defmethod write-value* :default 15 | [x] 16 | (when-some [x x] 17 | (pr-str x))) 18 | 19 | (defmethod write-value* ::native 20 | [x] 21 | x) 22 | 23 | (deftype Key [x]) 24 | 25 | (defmethod write-value* Key 26 | [^Key k] 27 | (pr-str (.x k))) 28 | 29 | ;;maps are only natively supported if all keys are `named` 30 | (defmethod write-value* IPersistentMap 31 | [x] 32 | (reduce-kv #(assoc %1 (if (or (string? %2) (instance? clojure.lang.Named %2)) 33 | %2 34 | (Key. %2)) %3) {} x)) 35 | 36 | (defmulti read-value* class) 37 | 38 | (defmethod read-value* :default 39 | [x] 40 | x) 41 | 42 | (defmethod read-value* Keyword 43 | [x] 44 | (let [ns (namespace x)] 45 | (if ns 46 | (edn/read-string (str ns "/" (name x))) 47 | (edn/read-string (name x))))) 48 | 49 | (defmethod read-value* String 50 | [x] 51 | (edn/read-string x)) 52 | 53 | (defn write-value 54 | "Encodes the given value for dynamo" 55 | [x] 56 | (walk/prewalk write-value* x)) 57 | 58 | (defn read-value 59 | "Reads an encoded dynamo value" 60 | [x] 61 | (walk/postwalk read-value* x)) 62 | 63 | (defn read-key 64 | "Reads an encoded dynamo key" 65 | [x] 66 | (if (and (vector? x) (= (first x) :atom/key)) 67 | (second x) 68 | x)) 69 | 70 | (defn native-key 71 | [x] 72 | (if (and (coll? x) (empty? x)) 73 | [:atom/key x] 74 | x)) 75 | 76 | (defn write-key 77 | "Keys must be encoded differently, as regardless 78 | of the type, they have to be encoded as strings. 79 | 80 | This function will allow you to write a non-string 81 | key if one wishes." 82 | [x] 83 | (pr-str 84 | (native-key x))) 85 | 86 | (defn prepare-value 87 | "Makes sure the value is a map, makes sure if there is an existing 88 | value under the encoded`key-name` it is preserved. Should be called 89 | before `write-value`" 90 | ([value key-name] 91 | (if (map? value) 92 | (let [key-name key-name] 93 | (if (contains? value key-name) 94 | (-> value 95 | (dissoc key-name) 96 | (assoc :atom/key (get value key-name))) 97 | value)) 98 | {:atom/value value}))) 99 | 100 | (defn add-key 101 | "Writes the (encoded) key to the value" 102 | [prepared-value key-name key] 103 | (assoc prepared-value 104 | (write-key key-name) (write-key key))) 105 | 106 | ;;hack all the things 107 | (def ^:dynamic *dynamo-list-hack* false) 108 | 109 | (alter-var-root #'far/attr-multi-vs 110 | (fn [f] 111 | (fn [x] 112 | (if *dynamo-list-hack* 113 | (mapv far/clj-item->db-item x) 114 | (f x))))) 115 | 116 | (defn fixed-batch-write-item 117 | "To get around an issue with batch-write-item and lists (namely they aren't supported properly)" 118 | [client-opts request] 119 | (binding [*dynamo-list-hack* true] 120 | (let [r (far/batch-write-item client-opts request)] 121 | (if (seq (:unprocessed r)) 122 | (throw (Exception. 123 | (format "Could not successfully write all items to dynamo. failed to write %s items" 124 | (count (:unprocessed r))))) 125 | r)))) 126 | 127 | (defn find-raw-value 128 | "Performs a consistent faraday lookup against `key`" 129 | [table-client key] 130 | (let [{:keys [client-opts table key-name]} table-client] 131 | (far/get-item client-opts table {(write-key key-name) (write-key key)} {:consistent? true}))) 132 | 133 | (defn find-raw-item 134 | "Finds the item at `key`, will unencode the value but will not 135 | do any further pruning." 136 | [table-client key] 137 | (-> (find-raw-value table-client key) 138 | read-value)) 139 | 140 | (defn read-item 141 | "Takes a value that has been unencoded and returns the original value 142 | from the prepared map" 143 | [read-value key-name] 144 | (when read-value 145 | (if (contains? read-value :atom/value) 146 | (:atom/value read-value) 147 | (cond-> (dissoc read-value (native-key key-name) :atom/key :atom/version) 148 | (contains? read-value :atom/key) (assoc key-name (get read-value :atom/key)))))) 149 | 150 | (defn cas-put-item! 151 | "Overwrites the value under `key` only if 152 | - The version provided matches the previous version of the item 153 | - There is not data currently under `key`" 154 | [table-client key value version] 155 | (let [{:keys [client-opts table key-name]} table-client 156 | version-key (write-key :atom/version)] 157 | (try 158 | (far/put-item client-opts table (-> (prepare-value value key-name) 159 | write-value 160 | (add-key key-name key) 161 | (assoc version-key (write-value (if version (inc version) 0)))) 162 | {:expected {version-key (if version [:eq (write-value version)] :not-exists)}}) 163 | true 164 | (catch ConditionalCheckFailedException e 165 | false)))) 166 | 167 | (def ^:dynamic *default-cas-sleep-ms* 168 | "When we hit contention we will wait this long before attempting the CAS operation 169 | again by default." 170 | 500) 171 | 172 | (defn swap-item!* 173 | ([table-client key f {:keys [cas-sleep-ms cas-timeout-ms cas-timeout-val discard-no-op?] 174 | :as options}] 175 | (let [cas-sleep-ms (or cas-sleep-ms *default-cas-sleep-ms*) 176 | discard-no-op? (if (some? discard-no-op?) discard-no-op? true)] 177 | (if (and cas-timeout-ms (<= 0 cas-timeout-ms)) 178 | cas-timeout-val 179 | (let [{:keys [key-name]} table-client 180 | value (find-raw-item table-client key) 181 | version (:atom/version value) 182 | input (read-item value key-name) 183 | result (f input)] 184 | (cond 185 | (and discard-no-op? (= input result)) 186 | result 187 | (cas-put-item! table-client key result version) 188 | result 189 | :else (do (when cas-sleep-ms (Thread/sleep cas-sleep-ms)) 190 | (recur table-client key f 191 | {:cas-sleep-ms cas-sleep-ms 192 | :cas-timeout-ms (when cas-timeout-ms 193 | (- cas-timeout-ms cas-sleep-ms)) 194 | :cas-timeout-val cas-timeout-val 195 | :discard-no-op? discard-no-op?})))))))) 196 | 197 | (defn table-client->options 198 | [table-client] 199 | (select-keys 200 | [:cas-sleep-ms 201 | :cas-timeout-ms 202 | :cas-timeout-val 203 | :discard-no-op?] 204 | table-client)) 205 | 206 | (defn swap-item! 207 | "Applies the function `f` and any `args` to the value currently under `key` storing 208 | the result. Returns the result." 209 | ([table-client key f] 210 | (swap-item!* table-client key f (table-client->options table-client))) 211 | ([table-client key f & args] 212 | (swap-item! table-client key #(apply f % args)))) -------------------------------------------------------------------------------- /test/faraday_atom/core_test.clj: -------------------------------------------------------------------------------- 1 | (ns faraday-atom.core-test 2 | (:require [clojure.test :refer :all] 3 | [faraday-atom.core :refer :all] 4 | [clojure.test.check.clojure-test :as tc] 5 | [clojure.test.check.properties :as prop] 6 | [clojure.test.check.generators :as gen])) 7 | 8 | (def local-client-opts 9 | {:endpoint "http://localhost:8000"}) 10 | 11 | (def complex-key-name 12 | {:fred #{1 2 3} :bar [1 {:x :y :z "hello"}]}) 13 | 14 | (defonce table-counter (atom 0)) 15 | 16 | (defn test-client 17 | [] 18 | (let [cl (table-client local-client-opts (keyword (str "test" (swap! table-counter inc))) complex-key-name)] 19 | (ensure-table! cl) 20 | cl)) 21 | 22 | (tc/defspec ^:dynamo round-trip-put-and-find 23 | 25 24 | (let [client (test-client)] 25 | (prop/for-all 26 | [{:keys [value key]} (gen/resize 10 (gen/hash-map :value gen/any :key gen/any))] 27 | (put-item! client key value) 28 | (= value 29 | (find-item client key))))) 30 | 31 | (tc/defspec ^:dynamo round-trip-put-many-and-find-many 32 | 25 33 | (let [client (test-client)] 34 | (prop/for-all 35 | [kvs (gen/resize 10 (gen/map (gen/resize 10 gen/any) (gen/resize 10 gen/any)))] 36 | (put-items! client kvs) 37 | (= (set (vals kvs)) 38 | (set (find-items client (keys kvs))))))) 39 | 40 | (tc/defspec ^:dynamo every-inconsistent-find-refers-to-a-previous-put 41 | 25 42 | (let [client (test-client) 43 | previous (atom {})] 44 | (prop/for-all 45 | [{:keys [value key]} (gen/resize 10 (gen/hash-map :value gen/any :key gen/any))] 46 | (put-item! client key value) 47 | (swap! previous update key (fnil conj #{}) value) 48 | (contains? (get @previous key) 49 | (find-inconsistent-item client key))))) 50 | 51 | (tc/defspec ^:dynamo every-inconsistent-find-refers-to-a-previous-put-many 52 | 25 53 | (let [client (test-client) 54 | previous (atom {})] 55 | (prop/for-all 56 | [kvs (gen/resize 10 (gen/map (gen/resize 10 gen/any) (gen/resize 10 gen/any)))] 57 | (put-items! client kvs) 58 | (swap! previous #(reduce (fn [m [k v]] (update m k (fnil conj #{}) v)) %1 kvs)) 59 | (let [items (find-inconsistent-item-mapping client (keys kvs))] 60 | (every? #(contains? (get @previous (first %)) (second %)) items))))) 61 | 62 | (deftest ^:dynamo concurrently-updating-counter-results-in-sum-of-inc-applications 63 | (let [client (test-client) 64 | atom (item-atom client :counter {:cas-sleep-ms 1}) 65 | threads (doall (repeatedly 20 #(future (swap! atom (fnil inc 0)))))] 66 | (doall (map deref threads)) 67 | (is (= (count threads) @atom)))) -------------------------------------------------------------------------------- /test/faraday_atom/impl_test.clj: -------------------------------------------------------------------------------- 1 | (ns faraday-atom.impl-test 2 | (:require [clojure.test.check.generators :as gen] 3 | [clojure.test.check.properties :as prop] 4 | [clojure.test.check.clojure-test :as tc] 5 | [faraday-atom.impl :refer :all])) 6 | 7 | 8 | (tc/defspec round-trip-conversion-from-edn-value-to-faraday-value 9 | 25 10 | (prop/for-all 11 | [value gen/any] 12 | (= (read-value (write-value value)) 13 | value))) 14 | 15 | (tc/defspec round-trip-conversion-from-edn-value-to-faraday-item 16 | 25 17 | (prop/for-all 18 | [{:keys [value key-name key]} (gen/hash-map :value gen/any 19 | :key-name gen/any 20 | :key gen/any)] 21 | (= (-> (prepare-value value key-name) 22 | (write-value) 23 | (add-key key-name key) 24 | read-value 25 | (read-item key-name)) 26 | value))) 27 | 28 | (tc/defspec key-in-value-is-kept 29 | 25 30 | (prop/for-all 31 | [{:keys [value key-name key]} (gen/hash-map :value (gen/map gen/any gen/any) 32 | :key-name gen/any 33 | :key gen/any) 34 | key-value gen/any] 35 | (let [value (assoc value key-name key-value)] 36 | (= (-> (prepare-value value key-name) 37 | (write-value) 38 | (add-key key-name key) 39 | read-value 40 | (read-item key-name)) 41 | value)))) 42 | 43 | --------------------------------------------------------------------------------