├── .gitignore ├── .travis.yml ├── README.md ├── docs └── vertigo.png ├── project.clj ├── src └── vertigo │ ├── bytes.clj │ ├── core.clj │ ├── primitives.clj │ └── structs.clj └── test └── vertigo ├── bytes_test.clj ├── core_test.clj ├── structs_test.clj └── test_utils.clj /.gitignore: -------------------------------------------------------------------------------- 1 | /target 2 | /lib 3 | /classes 4 | /checkouts 5 | pom.xml 6 | *.jar 7 | *.class 8 | .lein-deps-sum 9 | .lein-failures 10 | .lein-plugins 11 | .lein-repl-history 12 | push 13 | /doc 14 | .nrepl* -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: clojure 2 | lein: lein2 3 | jdk: 4 | - openjdk7 5 | - oraclejdk7 6 | - openjdk6 7 | script: lein2 do clean, test 8 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ![](docs/vertigo.png) 2 | 3 | Vertigo allows you to treat raw bytes like a normal Clojure data structure. This allows for faster reads and reduced memory footprint, and can also make interop with C libraries significantly simpler and more efficient. With certain safety checks turned off, this yields performance a bit faster than Java arrays on a much wider variety of datatypes. 4 | 5 | Full documentation can be found [here](http://ideolalia.com/vertigo). 6 | 7 | ### usage 8 | 9 | [![Build Status](https://travis-ci.org/ztellman/vertigo.png?branch=master)](https://travis-ci.org/ztellman/vertigo) 10 | 11 | ```clj 12 | [vertigo "0.1.4"] 13 | ``` 14 | 15 | ### defining a type 16 | 17 | To define a typed structure, we use `vertigo.structs/def-typed-struct`: 18 | 19 | ```clj 20 | (use 'vertigo.structs) 21 | 22 | (def-typed-struct vec3 23 | :x float64 24 | :y int32 25 | :z uint8) 26 | ``` 27 | 28 | Each field is defined as a pair: a name and a type. The resulting typed-struct can itself by used as a type: 29 | 30 | ```clj 31 | (def-typed-struct two-vecs 32 | :a vec3 33 | :b vec3) 34 | ``` 35 | 36 | In the `vertigo.structs` namespace, there are a number of predefined primitive types, including `int8`, `int16`, `int32`, `int64`, `float32`, and `float64`. Any integer type can be made unsigned by adding a `u` prefix, and all primitive types can have an `-le` or `-be` suffix for explicit endianness. Without a suffix, the endianness will default to that of the underlying buffer. 37 | 38 | We can also define a fixed-length n-dimensional array of any type using `(array type & dims)`: 39 | 40 | ```clj 41 | (def-typed-struct ints-and-floats 42 | :ints (array uint32 10) 43 | :floats (array float32 10)) 44 | ``` 45 | 46 | ### creating a sequence 47 | 48 | To create a sequence, we can either **marshal** an existing sequence onto a byte-buffer, or **wrap** an existing byte source. To marshal a sequence, we can either use `vertigo.core/marshal-seq` or `vertigo.core/lazily-marshal-seq`: 49 | 50 | ```clj 51 | > (require '[vertigo.core :as v]) 52 | nil 53 | > (def s (range 5)) 54 | #'s 55 | > (v/marshal-seq uint8 s) 56 | (0 1 2 3 4) 57 | ``` 58 | 59 | While the marshaled seq is outwardly identical, under the covers it is just a thin object wrapping a 5 byte buffer. In this case we could get an identical effect with an array, but this extends to types of any size or complexity: 60 | 61 | ```clj 62 | > (def x {:ints (range 10), :floats (map float (range 10))}) 63 | #'x 64 | > (v/marshal-seq ints-and-floats [x]) 65 | ({:ints (0 1 2 3 4 5 6 7 8 9), :floats (0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0)}) 66 | ``` 67 | 68 | Using `marshal-seq` will realize and copy over the entire sequence at once. For large or unbounded sequences, `lazily-marshal-seq` is preferred. While the performance characteristics of the two methods may differ, the resulting seq is otherwise the same. 69 | 70 | To create a sequence that wraps an already encoded source of bytes, you may use `vertigo.core/wrap`. Vertigo seqs created via `wrap` or `marshal-seq` can be turned back into bytes using `byte-streams/to-*` from the [byte-streams](https://github.com/ztellman/byte-streams) library. 71 | 72 | ### interacting with a sequence 73 | 74 | While we can use `first`, `rest`, `nth`, `get-in` and all the other normal Clojure functions on these sequences, we can also do something other data structures can't. Consider the above example of a map containing sequences of ints and floats. To get the fifth element under the `:floats` key, we call `(get-in s [:floats 4])`. This will first get the sequence under the `:floats` key, and then look for the fifth element within the sequence. 75 | 76 | However, in our data structure we're guaranteed to have a fixed layout, so _we know exactly where the data is already_. We don't need to fetch all the intermediate data structures, we simply need to calculate the location and do a single read. To do this, we use `vertigo.core/get-in`: 77 | 78 | ```clj 79 | > (def ^:int-and-floats s (v/marshal-seq ints-and-floats [x])) 80 | #'s 81 | > (v/get-in s [0 :floats 4]) 82 | 4.0 83 | ``` 84 | 85 | Notice that we have hinted the sequence with the keyword `:ints-and-floats`. To work with our compile-time `get-in`, all sequences must be hinted with their element type in this way. In return, we get a read time that even for moderately nested structures can be several orders of magnitude faster. 86 | 87 | Since everything's just bytes under the covers, we can also overwrite existing values. While this isn't always necessary, it's certainly useful when you need it. To write to the sequence, we may either use `vertigo.core/set-in!` or `vertigo.core/update-in!`: 88 | 89 | ```clj 90 | > (def ^:int8 s (v/marshal-seq int8 (range 5))) 91 | #'s 92 | > (v/set-in! s [5] 42) 93 | nil 94 | > (v/update-in! s [0] - 42) 95 | nil 96 | > s 97 | (-42 1 2 3 42) 98 | ``` 99 | 100 | ### selecting sub-sequences 101 | 102 | Calling something like `map` on a Vertigo sequence means that we can no longer use `vertigo.core/get-in`, `update-in!`, or any of the other special operators that only work on Vertigo sequences. This is mostly unavoidable, but in the specific case where we simply want to pull out a subset of the sequence, we can hold onto these properties using `(over s fields)`: 103 | 104 | ```clj 105 | > (require '[vertigo.structs :as s]) 106 | nil 107 | > (def matrix (s/array s/int64 2)) 108 | #'matrix 109 | > (marshal-seq matrix [[0 1] [2 3]]) 110 | ((0 1) (2 3)) 111 | > (def s *1) 112 | #'s 113 | 114 | ;; if we define both indices as 'free', then we see a flattened iteration over all elements 115 | > (over s [_ _]) 116 | (0 1 2 3) 117 | 118 | ;; if we fix the first index, we only iterate over the first array 119 | > (over s [0 _]) 120 | (0 1) 121 | 122 | ;; if we fix the second index, we only iterate over the first elements of each array 123 | > (over s [_ 0]) 124 | (0 2) 125 | ``` 126 | 127 | A free variable is either `_` or anything that begins with `?`, like `?i` or `?a-gratuitously-long-name`. 128 | 129 | ### iterating over sequences 130 | 131 | Since we can directly access the underlying memory, we can do very efficient iteration, but Clojure's normal iteration primitives use too many levels of indirection for us to realize this. 132 | 133 | Instead, Vertigo provides `doreduce`, a primitive that is a fusion of Clojure's `doseq` and `loop`: 134 | 135 | ```clj 136 | > (def ^:s/int64 s (marshal-seq s/int64 (repeat 100 1))) 137 | #'s 138 | > (doreduce [x s] [sum 0] 139 | (+ sum x)) 140 | 100 141 | ``` 142 | 143 | Notice that `doreduce` takes two binding forms, one for sequences, and another for accumulators. Unlike `loop`, the `recur` here is implicit; the value returned for one element is propagated to the next. If we want to propagate multiple values, we simply return a vector: 144 | 145 | ```clj 146 | > (doreduce [x s] [sum 0, product 1] 147 | [(+ sum x) (* product x)]) 148 | [100 1] 149 | ``` 150 | 151 | Note that we're not actually allocating a new vector for each element, it's just sugar over a multi-value recursion. 152 | 153 | If we explicitly don't want to recur, we can use `break`: 154 | 155 | ```clj 156 | > (doreduce [x s] [all-even? true] 157 | (if (odd? x) 158 | (break false) 159 | true)) 160 | ``` 161 | 162 | We can iterate over multiple sequences in parallel: 163 | 164 | ```clj 165 | > (doreduce [x s, y s] [sum 0] 166 | (+ x y sum)) 167 | 200 168 | ``` 169 | 170 | We can also iterate over subsets of sequences using the `over` syntax: 171 | 172 | ```clj 173 | > (doreduce [x (over s [?i])] [sum 0] 174 | (+ x ?i sum)) 175 | 5050 176 | ``` 177 | 178 | Finally, we can set the `:step` and `:limit` for iteration: 179 | 180 | ```clj 181 | > (doreduce [x s, :step 2, :limit 10] [sum 0] 182 | (+ x sum)) 183 | 5 184 | ``` 185 | 186 | Or, if there are multiple iterators, `:steps` and `:limits`: 187 | 188 | ```clj 189 | > (doreduce [x (over s [?i]), :steps {?i 2}, :limits {?i 10}] [sum 0] 190 | (+ x sum)) 191 | 5 192 | ``` 193 | 194 | Common use cases are exposed as `vertigo.core/sum`, `every?`, and `any?`. 195 | 196 | ### turning off bounds checks 197 | 198 | By default, every index is checked at compile time if possible, and runtime if necessary. However, if your access patterns are predictable and no runtime exceptions are thrown, you can turn off bounds checking by adding `-Dvertigo.unsafe` as a command-line parameter. When this is enabled, `(vertigo.core/sum s)` is ~10% faster than the equivalent operation using Clojure's `areduce`, and ~10x faster than `(reduce + array)`. 199 | 200 | ### when to use vertigo 201 | 202 | Any long-lived data which has a fixed layout can benefit from Vertigo, both with respect to performance and memory efficiency. However, sequences which contain sub-sequences of unpredictable length cannot be represented using Vertigo's structs, and sequences which are discarded after a single iteration won't be worth the trouble of calling `marshal-seq`. 203 | 204 | ### license 205 | 206 | Copyright © 2013 Zachary Tellman 207 | 208 | Distributed under the [MIT License](http://opensource.org/licenses/MIT) 209 | -------------------------------------------------------------------------------- /docs/vertigo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ztellman/vertigo/8b333cd6b7e8c95a46a8dede79f1d14c4988e470/docs/vertigo.png -------------------------------------------------------------------------------- /project.clj: -------------------------------------------------------------------------------- 1 | (defproject vertigo "0.1.4" 2 | :description "peering down into the machine" 3 | :license {:name "MIT Public License" 4 | :url "http://opensource.org/licenses/MIT"} 5 | :dependencies [[potemkin "0.4.2"] 6 | [primitive-math "0.1.4"] 7 | [clj-tuple "0.2.2"] 8 | [byte-streams "0.2.0"]] 9 | :profiles {:dev {:dependencies [[criterium "0.4.3"] 10 | [org.clojure/clojure "1.9.0-alpha14"] 11 | [codox-md "0.2.0" :exclusions [org.clojure/clojure]]]} 12 | :benchmark {:jvm-opts ["-Dvertigo.unsafe"]}} 13 | :plugins [[codox "0.6.4"]] 14 | :codox {:writer codox-md.writer/write-docs 15 | :exclude [vertigo.primitives]} 16 | :global-vars {*warn-on-reflection* true} 17 | :test-selectors {:default #(not (some #{:benchmark} 18 | (cons (:tag %) (keys %)))) 19 | :benchmark :benchmark 20 | :all (constantly true)} 21 | :aliases {"benchmark" ["with-profile" "dev,benchmark" "test" ":benchmark"]} 22 | :jvm-opts ^:replace ["-server"]) 23 | -------------------------------------------------------------------------------- /src/vertigo/bytes.clj: -------------------------------------------------------------------------------- 1 | (ns vertigo.bytes 2 | (:use 3 | potemkin) 4 | (:require 5 | [byte-streams :as bytes] 6 | [vertigo.primitives :as prim]) 7 | (:import 8 | [sun.misc 9 | Unsafe] 10 | [clojure.lang 11 | Compiler$LocalBinding] 12 | [java.nio.channels 13 | Channel 14 | Channels 15 | ReadableByteChannel] 16 | [java.nio 17 | DirectByteBuffer 18 | ByteBuffer 19 | ByteOrder] 20 | [java.io 21 | Closeable] 22 | [java.util.concurrent.atomic 23 | AtomicBoolean] 24 | [java.lang.reflect 25 | Array])) 26 | 27 | ;;; 28 | 29 | (prim/use-primitive-operators) 30 | 31 | (definterface+ IByteSeq 32 | (get-int8 ^long [_ ^long idx]) 33 | (get-int16 ^long [_ ^long idx]) 34 | (get-int32 ^long [_ ^long idx]) 35 | (get-int64 ^long [_ ^long idx]) 36 | (get-float32 ^double [_ ^long idx]) 37 | (get-float64 ^double [_ ^long idx]) 38 | 39 | (put-int8 [_ ^long idx ^long val]) 40 | (put-int16 [_ ^long idx ^long val]) 41 | (put-int32 [_ ^long idx ^long val]) 42 | (put-int64 [_ ^long idx ^long val]) 43 | (put-float32 [_ ^long idx ^double val]) 44 | (put-float64 [_ ^long idx ^double val]) 45 | 46 | (byte-count ^long [_] 47 | "Returns the number of bytes in the byte-seq.") 48 | (drop-bytes [_ ^long n] 49 | "Returns a new byte-seq without the first `n` bytes.") 50 | (slice [_ ^long offset ^long len] 51 | "Returns a subset of the byte-seq, starting at `offset` bytes, and `len` bytes long.") 52 | 53 | (close-fn [_] 54 | "Returns the function, if it exists, that closes the underlying source of bytes.") 55 | (flush-fn [_] 56 | "Returns the function, if it exists, that flushes the underlying source of bytes.") 57 | 58 | (byte-order [_] 59 | "Returns the java.nio.ByteOrder of the underlying buffer.") 60 | (set-byte-order! [_ order] 61 | "Sets the byte order of the underlying buffer, and returns the byte-seq.") 62 | (byte-seq-reduce [_ stride read-fn f start] 63 | "A byte-seq specific version of the CollReduce protocol.")) 64 | 65 | (definline big-endian? 66 | "Returns true if the byte-seq is big-endian." 67 | [byte-seq] 68 | `(= ByteOrder/BIG_ENDIAN (byte-order ~byte-seq))) 69 | 70 | (definline little-endian? 71 | "Returns true if the byte-seq is little-endian." 72 | [byte-seq] 73 | `(= ByteOrder/LITTLE_ENDIAN (byte-order ~byte-seq))) 74 | 75 | (definline buf-size [b] 76 | `(long (.remaining ~(with-meta b {:tag "java.nio.ByteBuffer"})))) 77 | 78 | (defn slice-buffer [^ByteBuffer buf ^long offset ^long len] 79 | (when buf 80 | (let [order (.order buf)] 81 | (-> buf 82 | .duplicate 83 | (.position offset) 84 | ^ByteBuffer (.limit (+ offset len)) 85 | .slice 86 | (.order order))))) 87 | 88 | ;;; 89 | 90 | (deftype+ ByteSeq 91 | [^ByteBuffer buf 92 | close-fn 93 | flush-fn] 94 | 95 | IByteSeq 96 | (get-int8 [this idx] (long (.get buf idx))) 97 | (get-int16 [this idx] (long (.getShort buf idx))) 98 | (get-int32 [this idx] (long (.getInt buf idx))) 99 | (get-int64 [this idx] (.getLong buf idx)) 100 | (get-float32 [this idx] (double (.getFloat buf idx))) 101 | (get-float64 [this idx] (.getDouble buf idx)) 102 | 103 | (put-int8 [this idx val] (.put buf idx (byte val))) 104 | (put-int16 [this idx val] (.putShort buf idx (short val))) 105 | (put-int32 [this idx val] (.putInt buf idx (int val))) 106 | (put-int64 [this idx val] (.putLong buf idx val)) 107 | (put-float32 [this idx val] (.putFloat buf idx (float val))) 108 | (put-float64 [this idx val] (.putDouble buf idx val)) 109 | 110 | (byte-order [_] (.order buf)) 111 | (set-byte-order! [this order] (.order buf order) this) 112 | 113 | java.io.Closeable 114 | (close [this] 115 | (when close-fn 116 | (close-fn this))) 117 | 118 | (close-fn [_] close-fn) 119 | (flush-fn [_] flush-fn) 120 | 121 | (byte-seq-reduce [this stride read-fn f start] 122 | (let [stride (long stride) 123 | size (buf-size buf)] 124 | (loop [idx 0, val start] 125 | (if (<= size idx) 126 | val 127 | (let [val' (f val (read-fn this idx))] 128 | (if (reduced? val') 129 | @val' 130 | (recur (+ idx stride) val'))))))) 131 | 132 | (byte-count [_] 133 | (buf-size buf)) 134 | 135 | (drop-bytes [this n] 136 | (when (< n (buf-size buf)) 137 | (slice this n (- (buf-size buf) n)))) 138 | 139 | (slice [_ offset len] 140 | (ByteSeq. (slice-buffer buf offset len) close-fn flush-fn))) 141 | 142 | ;;; 143 | 144 | (deftype+ UnsafeByteSeq 145 | [^Unsafe unsafe 146 | ^ByteBuffer buf 147 | ^long loc 148 | close-fn 149 | flush-fn] 150 | 151 | java.io.Closeable 152 | (close [this] 153 | (when close-fn 154 | (close-fn this))) 155 | 156 | (close-fn [_] 157 | close-fn) 158 | (flush-fn [_] 159 | flush-fn) 160 | 161 | IByteSeq 162 | (get-int8 [this idx] (long (.getByte unsafe (+ loc idx)))) 163 | (get-int16 [this idx] (long (.getShort unsafe (+ loc idx)))) 164 | (get-int32 [this idx] (long (.getInt unsafe (+ loc idx)))) 165 | (get-int64 [this idx] (.getLong unsafe (+ loc idx))) 166 | (get-float32 [this idx] (double (.getFloat unsafe (+ loc idx)))) 167 | (get-float64 [this idx] (.getDouble unsafe (+ loc idx))) 168 | 169 | (put-int8 [this idx val] (.putByte unsafe (+ loc idx) (byte val))) 170 | (put-int16 [this idx val] (.putShort unsafe (+ loc idx) (short val))) 171 | (put-int32 [this idx val] (.putInt unsafe (+ loc idx) (int val))) 172 | (put-int64 [this idx val] (.putLong unsafe (+ loc idx) val)) 173 | (put-float32 [this idx val] (.putFloat unsafe (+ loc idx) (float val))) 174 | (put-float64 [this idx val] (.putDouble unsafe (+ loc idx) val)) 175 | 176 | (byte-order [_] (.order buf)) 177 | (set-byte-order! [this order] (.order buf order) this) 178 | 179 | (byte-seq-reduce [this stride read-fn f start] 180 | (let [stride (long stride) 181 | size (buf-size buf)] 182 | (loop [idx 0, val start] 183 | (if (<= size idx) 184 | val 185 | (let [val' (f val (read-fn this idx))] 186 | (if (reduced? val') 187 | @val' 188 | (recur (+ idx stride) val'))))))) 189 | 190 | (byte-count [_] 191 | (buf-size buf)) 192 | 193 | (drop-bytes [this n] 194 | (when (< n (buf-size buf)) 195 | (slice this n (- (buf-size buf) n)))) 196 | 197 | (slice [_ offset len] 198 | (ByteSeq. (slice-buffer buf offset len) close-fn flush-fn))) 199 | 200 | ;;; 201 | 202 | (defmacro ^:private doto-nth 203 | [this buf size idx f & rest] 204 | `(try 205 | (loop [chunk# ~this, idx# (long ~idx)] 206 | (let [buf-size# (.chunk-size chunk#)] 207 | (if (< idx# buf-size#) 208 | (let [^ByteBuffer buf# (.buf chunk#)] 209 | (~f buf# idx# ~@rest)) 210 | (let [next# (.next-chunk chunk#) 211 | next# (when-not (nil? next#) @next#) 212 | next# (when-not (nil? next#) (set-byte-order! next# (byte-order chunk#)))] 213 | (recur next# (- idx# buf-size#)))))) 214 | (catch NullPointerException e# 215 | (throw (IndexOutOfBoundsException. (str ~idx)))))) 216 | 217 | (deftype ChunkedByteSeq 218 | [^ByteBuffer buf 219 | ^long chunk-size 220 | next-chunk 221 | close-fn 222 | flush-fn] 223 | 224 | java.io.Closeable 225 | (close [this] 226 | (when close-fn 227 | (close-fn this))) 228 | 229 | (close-fn [_] 230 | close-fn) 231 | (flush-fn [_] 232 | flush-fn) 233 | 234 | clojure.lang.ISeq 235 | (seq [this] 236 | this) 237 | (first [_] 238 | (ByteSeq. buf close-fn flush-fn)) 239 | (next [this] 240 | (let [nxt (when-not (nil? next-chunk) @next-chunk)] 241 | (when-not (nil? nxt) 242 | (set-byte-order! nxt (.order buf))) 243 | nxt)) 244 | (more [this] 245 | (or (next this) '())) 246 | (cons [this buffer] 247 | (ChunkedByteSeq. buffer (buf-size buffer) (delay this) close-fn flush-fn)) 248 | 249 | IByteSeq 250 | 251 | (get-int8 [this idx] (long (doto-nth this buf chunk-size idx .get))) 252 | (get-int16 [this idx] (long (doto-nth this buf chunk-size idx .getShort))) 253 | (get-int32 [this idx] (long (doto-nth this buf chunk-size idx .getInt))) 254 | (get-int64 [this idx] (doto-nth this buf chunk-size idx .getLong)) 255 | (get-float32 [this idx] (double (doto-nth this buf chunk-size idx .getFloat))) 256 | (get-float64 [this idx] (doto-nth this buf chunk-size idx .getDouble)) 257 | 258 | (put-int8 [this idx val] (doto-nth this buf chunk-size idx .put (byte val))) 259 | (put-int16 [this idx val] (doto-nth this buf chunk-size idx .putShort (short val))) 260 | (put-int32 [this idx val] (doto-nth this buf chunk-size idx .putInt (int val))) 261 | (put-int64 [this idx val] (doto-nth this buf chunk-size idx .putLong val)) 262 | (put-float32 [this idx val] (doto-nth this buf chunk-size idx .putFloat (float val))) 263 | (put-float64 [this idx val] (doto-nth this buf chunk-size idx .putDouble val)) 264 | 265 | (byte-order [_] (.order buf)) 266 | (set-byte-order! [this order] (.order buf order) this) 267 | 268 | (byte-seq-reduce [this stride read-fn f start] 269 | (let [stride (long stride)] 270 | (loop [chunk this, val start] 271 | (let [val' (let [size (buf-size (.buf chunk))] 272 | (loop [idx 0, val val] 273 | (if (<= size idx) 274 | val 275 | (let [val' (f val (read-fn chunk idx))] 276 | (if (reduced? val') 277 | val' 278 | (recur (+ idx stride) val'))))))] 279 | (if (reduced? val') 280 | @val' 281 | (let [nxt (next chunk)] 282 | (if (nil? nxt) 283 | val' 284 | (recur nxt val')))))))) 285 | 286 | (byte-count [this] 287 | (loop [chunk this, acc 0] 288 | (if (nil? chunk) 289 | acc 290 | (recur (next chunk) (+ acc (long (.remaining ^ByteBuffer (.buf chunk)))))))) 291 | 292 | (slice [this offset len] 293 | (when-let [^ChunkedByteSeq byte-seq (drop-bytes this offset)] 294 | (let [size (buf-size (.buf byte-seq))] 295 | (if (< size (+ offset len)) 296 | (ChunkedByteSeq. 297 | (slice-buffer buf offset (- size offset)) 298 | (- size offset) 299 | (delay (slice @next-chunk 0 (- len size))) 300 | close-fn 301 | flush-fn) 302 | (ChunkedByteSeq. 303 | (slice-buffer buf offset len) 304 | len 305 | nil 306 | close-fn 307 | flush-fn))))) 308 | 309 | (drop-bytes [this n] 310 | (loop [chunk this, to-drop n] 311 | (when-not (nil? chunk) 312 | (let [buf (.buf chunk) 313 | size (buf-size buf)] 314 | (if (<= size to-drop) 315 | (recur (next chunk) (- to-drop size)) 316 | (ChunkedByteSeq. 317 | (slice-buffer buf to-drop (- size to-drop)) 318 | (- size to-drop) 319 | (.next-chunk chunk) 320 | (.close-fn chunk) 321 | (.flush-fn chunk)))))))) 322 | 323 | (defmethod print-method ChunkedByteSeq [o ^java.io.Writer writer] 324 | (print-method (map identity o) writer)) 325 | 326 | ;;; 327 | 328 | (def ^{:doc "If true, disables all runtime index bounds checking."} 329 | use-unsafe? 330 | (boolean (System/getProperty "vertigo.unsafe"))) 331 | 332 | (def ^Unsafe ^:private unsafe 333 | (when use-unsafe? 334 | (let [f (.getDeclaredField Unsafe "theUnsafe")] 335 | (.setAccessible f true) 336 | (.get f nil)))) 337 | 338 | (defn- with-native-order [^ByteBuffer buf] 339 | (.order buf (ByteOrder/nativeOrder))) 340 | 341 | (defn buffer 342 | "Creates a byte-buffer." 343 | [^long size] 344 | (with-native-order (ByteBuffer/allocate size))) 345 | 346 | (defn direct-buffer 347 | "Creates a direct byte-buffer." 348 | [^long size] 349 | (with-native-order (ByteBuffer/allocateDirect size))) 350 | 351 | (defn chunked-byte-seq 352 | ([buf next] 353 | (chunked-byte-seq buf next nil nil)) 354 | ([buf next close-fn flush-fn] 355 | (ChunkedByteSeq. 356 | (with-native-order (or buf (buffer 0))) 357 | (buf-size buf) 358 | next 359 | close-fn 360 | flush-fn))) 361 | 362 | (defn byte-seq 363 | "Wraps a byte-buffer inside a byte-seq." 364 | ([buf] 365 | (byte-seq buf nil nil)) 366 | ([^ByteBuffer buf close-fn flush-fn] 367 | (if (and (.isDirect buf) use-unsafe?) 368 | (UnsafeByteSeq. unsafe (with-native-order buf) (.address ^DirectByteBuffer buf) close-fn flush-fn) 369 | (ByteSeq. (with-native-order buf) close-fn flush-fn)))) 370 | 371 | (defn cross-section 372 | "Returns a chunked-byte-seq representing a sequence of slices, starting at `offset`, `length` 373 | bytes long, and separated by `stride` bytes." 374 | [byte-seq ^long offset ^long length ^long stride] 375 | (when-let [buf (and byte-seq 376 | (-> byte-seq 377 | bytes/to-byte-buffers 378 | first 379 | (slice-buffer offset length)))] 380 | (chunked-byte-seq 381 | buf 382 | (-> byte-seq 383 | (drop-bytes (+ offset stride)) 384 | (cross-section 0 length stride) 385 | delay) 386 | (close-fn byte-seq) 387 | (flush-fn byte-seq)))) 388 | 389 | (defn to-byte-seq 390 | "Converts `x` to a byte-seq." 391 | ([x] 392 | (to-byte-seq x nil)) 393 | ([x options] 394 | (bytes/convert x ByteSeq options))) 395 | 396 | (defn to-chunked-byte-seq 397 | "Converts `x` to a chunked-byte-seq." 398 | ([x] 399 | (to-chunked-byte-seq x nil)) 400 | ([x options] 401 | (bytes/convert x ChunkedByteSeq options))) 402 | 403 | ;;; 404 | 405 | (bytes/def-conversion [ByteSeq (bytes/seq-of ByteBuffer)] 406 | [byte-seq options] 407 | (if-let [chunk-size (:chunk-size options)] 408 | (bytes/to-byte-buffers (.buf byte-seq) options) 409 | [(.duplicate ^ByteBuffer (.buf byte-seq))])) 410 | 411 | (bytes/def-conversion [UnsafeByteSeq (bytes/seq-of ByteBuffer)] 412 | [byte-seq options] 413 | (if-let [chunk-size (:chunk-size options)] 414 | (bytes/to-byte-buffers (.buf byte-seq) options) 415 | [(.duplicate ^ByteBuffer (.buf byte-seq))])) 416 | 417 | (bytes/def-conversion [ChunkedByteSeq (bytes/seq-of ByteBuffer)] 418 | [byte-seq options] 419 | (let [bufs (lazy-seq 420 | (cons 421 | (.duplicate ^ByteBuffer (.buf byte-seq)) 422 | (when-let [s (seq (next byte-seq))] 423 | (bytes/to-byte-buffers s))))] 424 | (if-let [chunk-size (:chunk-size options)] 425 | (-> bufs bytes/to-readable-channel (bytes/to-byte-buffers options)) 426 | bufs))) 427 | 428 | (bytes/def-conversion [(bytes/seq-of ByteBuffer) ChunkedByteSeq] 429 | [bufs] 430 | (let [buf (first bufs)] 431 | (chunked-byte-seq 432 | (first bufs) 433 | (delay (bytes/convert (rest bufs) ChunkedByteSeq)) 434 | (when (instance? Closeable bufs) 435 | #(.close ^Closeable bufs)) 436 | nil))) 437 | 438 | (bytes/def-conversion [ByteBuffer ByteSeq] 439 | [buf] 440 | (byte-seq buf)) 441 | -------------------------------------------------------------------------------- /src/vertigo/core.clj: -------------------------------------------------------------------------------- 1 | (ns vertigo.core 2 | (:refer-clojure 3 | :exclude [get-in every? flush any?]) 4 | (:use 5 | potemkin) 6 | (:require 7 | [clj-tuple :refer (tuple)] 8 | [byte-streams.protocols :as proto] 9 | [riddley.walk :as rw] 10 | [riddley.compiler :as rc] 11 | [byte-streams :as bytes] 12 | [vertigo.structs :as s] 13 | [vertigo.bytes :as b] 14 | [vertigo.primitives :as p]) 15 | (:import 16 | [java.io 17 | File] 18 | [java.nio 19 | MappedByteBuffer] 20 | [clojure.lang 21 | Compiler$LocalBinding] 22 | [vertigo.structs 23 | IFixedType 24 | IFixedInlinedType 25 | IFixedCompoundType])) 26 | 27 | ;;; 28 | 29 | (defn flush 30 | "If necessary, flushes changes that have been made to the byte sequence." 31 | [x] 32 | (let [byte-seq (s/unwrap-byte-seq x)] 33 | (when-let [flush-fn (b/flush-fn byte-seq)] 34 | (flush-fn byte-seq)))) 35 | 36 | (defn- safe-chunk-size [type ^long chunk-size] 37 | (p/* (s/byte-size type) (p/div chunk-size (s/byte-size type)))) 38 | 39 | (defn wrap 40 | "Wraps `x`, treating it as a sequence of `type`. If `x` is a string, it will be treated 41 | as a filename. If `x` can be converted to a ByteBuffer without any copying (byte-arrays, 42 | files, etc.), this is an O(1) operation. 43 | 44 | This will behave like a normal, immutable sequence of data, unless `set-in!` or `update-in!` 45 | are used. It can be used with `doreduce` for efficient iteration or reduction. 46 | 47 | This returns an object that plays well with `byte-streams/to-*`, so the inverse operation is 48 | simply `byte-streams/to-byte-buffer`." 49 | ([type x] 50 | (wrap type x nil)) 51 | ([type x 52 | {:keys [direct? chunk-size writable?] 53 | :or {direct? false 54 | chunk-size (if (string? x) 55 | (int 2e9) 56 | (int 1e6)) 57 | writable? true} 58 | :as options}] 59 | (let [x' (if (string? x) (File. ^String x) x) 60 | chunk-size (safe-chunk-size type chunk-size) 61 | bufs (bytes/to-byte-buffers x' options) 62 | close-fn (when (satisfies? proto/Closeable bufs) 63 | (fn [] (proto/close bufs))) 64 | flush-fn (when (or (instance? File x) (string? x)) 65 | (fn [] (map #(.force ^MappedByteBuffer %) bufs)))] 66 | (s/wrap-byte-seq type 67 | (if (empty? (rest bufs)) 68 | (b/byte-seq (first bufs) close-fn flush-fn) 69 | (let [f (fn this [bufs] 70 | (when-not (empty? bufs) 71 | (b/chunked-byte-seq 72 | (first bufs) 73 | (delay (this (rest bufs))) 74 | close-fn 75 | flush-fn)))] 76 | (f bufs))))))) 77 | 78 | (defn marshal-seq 79 | "Converts a sequence into a marshaled version of itself." 80 | ([type s] 81 | (marshal-seq type s nil)) 82 | ([type s {:keys [direct?] :or {direct? true}}] 83 | (let [cnt (count s) 84 | stride (s/byte-size type) 85 | allocate (if direct? b/direct-buffer b/buffer) 86 | byte-seq (-> (long cnt) (p/* (long stride)) allocate b/byte-seq)] 87 | (loop [offset 0, s s] 88 | (when-not (empty? s) 89 | (s/write-value type byte-seq offset (first s)) 90 | (recur (p/+ offset stride) (rest s)))) 91 | (s/wrap-byte-seq type byte-seq)))) 92 | 93 | (defn lazily-marshal-seq 94 | "Lazily converts a sequence into a marshaled version of itself." 95 | ([type s] 96 | (lazily-marshal-seq type s nil)) 97 | ([type s {:keys [chunk-size direct?] :or {chunk-size 65536}}] 98 | (let [chunk-size (long (safe-chunk-size type chunk-size)) 99 | allocate (if direct? b/direct-buffer b/buffer) 100 | stride (s/byte-size type) 101 | populate (fn populate [s] 102 | (when-not (empty? s) 103 | (let [remaining (promise) 104 | nxt (delay (populate @remaining)) 105 | byte-seq (-> chunk-size allocate (b/chunked-byte-seq nxt))] 106 | (loop [idx 0, offset 0, s s] 107 | (cond 108 | 109 | (empty? s) 110 | (do 111 | (deliver remaining nil) 112 | (b/slice byte-seq 0 offset)) 113 | 114 | (p/== chunk-size offset) 115 | (do 116 | (deliver remaining s) 117 | byte-seq) 118 | 119 | :else 120 | (do 121 | (s/write-value type byte-seq offset (first s)) 122 | (recur (p/inc idx) (p/+ offset stride) (rest s))))))))] 123 | (s/wrap-byte-seq type (populate s))))) 124 | 125 | ;;; 126 | 127 | (defn- element-type 128 | "Check the local context and tag to determine element type." 129 | [env sym] 130 | (let [type-meta (fn [m] 131 | (->> (keys m) 132 | (filter keyword?) 133 | (filter #(true? (m %))) 134 | (map #(symbol (namespace %) (name %))) 135 | (filter #(when-let [x (resolve %)] 136 | (instance? IFixedType @x))) 137 | first))] 138 | (or 139 | (type-meta (meta sym)) 140 | (when-let [^Compiler$LocalBinding binding (get env sym)] 141 | (type-meta (meta (.sym binding)))) 142 | (type-meta (meta (resolve sym))) 143 | (throw (IllegalArgumentException. (str "cannot determine type of '" sym "'")))))) 144 | 145 | (defn- resolve-type 146 | [type-sym] 147 | @(resolve type-sym)) 148 | 149 | (defn- resolve-type-var 150 | "Try to backtrace the value to the corresponding var." 151 | [type] 152 | (when-let [v (resolve (symbol (namespace type) (name type)))] 153 | (when (= @v type) 154 | v))) 155 | 156 | (defn- with-element-type [sym seq env] 157 | (let [v (element-type env seq)] 158 | (with-meta sym {(keyword (symbol (namespace v) (name v))) true}))) 159 | 160 | (defn- free-variable? [x] 161 | (when (symbol? x) 162 | (let [s (name x)] 163 | (boolean 164 | (or (= "_" s) 165 | (re-find #"^\?.*" s)))))) 166 | 167 | (defn- validate-lookup 168 | "Makes sure that the given field actually exists within the given type." 169 | [type field] 170 | ;; can't go any deeper 171 | (when-not (instance? IFixedCompoundType type) 172 | (throw (IllegalArgumentException. (str "Invalid field '" field "' for non-compound type " (name type))))) 173 | 174 | ;; if it's a symbol, it must be for an index 175 | (when (and (symbol? field) (not (s/has-field? type 0))) 176 | (throw (IllegalArgumentException. (str "'" field "' is assumed to be numeric, which isn't accepted by " (name type))))) 177 | 178 | ;; invalid field 179 | (when (and (or (keyword? field) (number? field)) (not (s/has-field? type field))) 180 | (throw (IllegalArgumentException. (str "Invalid field '" field "' for type " (name type)))))) 181 | 182 | (defn- lookup-info 183 | "Walks the field sequence, returning a descriptor of offsets, lengths, and other useful 184 | information." 185 | [type fields] 186 | (let [{:keys [types type-form-fn offsets lengths limits]} 187 | (reduce 188 | (fn [{:keys [types type-form-fn offsets lengths limits]} field] 189 | (let [type (last types) 190 | _ (validate-lookup type field) 191 | inner-type (s/field-type type field) 192 | has-fields? (instance? IFixedCompoundType type) 193 | length (when (and has-fields? (s/has-field? type 0)) 194 | (p/div (s/byte-size type) (s/byte-size (s/field-type type 0))))] 195 | {:types (conj types inner-type) 196 | :type-form-fn (if-let [v (resolve-type-var inner-type)] 197 | (constantly v) 198 | (fn [x] `(s/field-type ~(type-form-fn x) ~field))) 199 | :offsets (conj offsets 200 | (if (symbol? field) 201 | `(p/* ~(s/byte-size (s/field-type type 0)) (long ~field)) 202 | (s/field-offset type field))) 203 | :lengths (conj lengths length) 204 | :limits (if (symbol? field) 205 | (merge-with min limits {field length}) 206 | limits)})) 207 | {:types [type], :type-form-fn identity, :offsets [], :lengths [], :limits {}} 208 | fields) 209 | 210 | inner-type (or (last types) type) 211 | 212 | ;; take a bunch of numbers and forms, and collapse all the numbers together 213 | collapse-exprs #(concat 214 | (->> % (remove number?)) 215 | (->> % (filter number?) (apply +) list (remove zero?))) 216 | 217 | num-prefixed-lookups (->> fields 218 | (take-while (complement free-variable?)) 219 | count) 220 | 221 | ;; the outermost boundaries of the lookup 222 | slice (when-not (zero? num-prefixed-lookups) 223 | {:offset (collapse-exprs (take num-prefixed-lookups offsets)) 224 | :length (let [type (nth types num-prefixed-lookups)] 225 | (s/byte-size type))}) 226 | 227 | contiguous-groups (fn contiguous-groups [pred s] 228 | (let [s (drop-while (complement pred) s)] 229 | (when-not (empty? s) 230 | (cons (take-while pred s) 231 | (contiguous-groups pred (drop-while pred s)))))) 232 | 233 | cross-sections (->> (map #(zipmap [:field :offset :length :stride] %&) 234 | fields 235 | offsets 236 | (map s/byte-size (rest types)) 237 | (map s/byte-size types)) 238 | (drop num-prefixed-lookups) 239 | (contiguous-groups #(not (free-variable? (:field %)))) 240 | (map (fn [field-descriptors] 241 | {:offset-exprs (collapse-exprs (map :offset field-descriptors)) 242 | :length (apply min (map :length field-descriptors)) 243 | :stride (apply max (map :stride field-descriptors))})))] 244 | 245 | {:types (rest types) 246 | :cross-sections cross-sections 247 | :slice slice 248 | :type-form-fn type-form-fn 249 | :lengths lengths 250 | :inner-type inner-type 251 | :outer-type (nth types num-prefixed-lookups) 252 | :inlined? (instance? IFixedInlinedType inner-type) 253 | :offset-exprs (collapse-exprs offsets) 254 | :validation-exprs (map 255 | (fn [[field length]] 256 | `(when (>= (long ~field) ~length) 257 | (throw (IndexOutOfBoundsException. (str ~(str field) ": " ~field))))) 258 | limits)})) 259 | 260 | ;;; 261 | 262 | (defn- field-operation 263 | [env x fields validate? inlined-form non-inlined-form] 264 | (let [type-sym (element-type env x) 265 | type (resolve-type type-sym) 266 | x (with-meta x {}) 267 | {:keys [inner-type inlined? type-form-fn offset-exprs validation-exprs]} (lookup-info type (rest fields)) 268 | x-sym (if (symbol? x) x (gensym "x")) 269 | form ((if inlined? 270 | inlined-form 271 | non-inlined-form) 272 | (if inlined? 273 | inner-type 274 | (type-form-fn type-sym)) 275 | x-sym 276 | `(s/unwrap-byte-seq ~x-sym) 277 | `(p/+ (s/index-offset ~x-sym ~(first fields)) ~@offset-exprs)) 278 | validation (when validate? 279 | (concat 280 | validation-exprs 281 | (when (symbol? (first fields)) 282 | [`(when (>= (long ~(first fields)) 283 | (.count ~(with-meta x-sym {:tag "clojure.lang.Counted"}))) 284 | (throw 285 | (IndexOutOfBoundsException. 286 | (str ~(str (first fields)) ": " ~(first fields)))))])))] 287 | (unify-gensyms 288 | (if (= x x-sym) 289 | `(do 290 | ~@validation 291 | ~form) 292 | `(let [~x-sym ~x] 293 | ~@validation 294 | ~form))))) 295 | 296 | (defn- get-in-form [s fields env validate?] 297 | (field-operation env s fields validate? 298 | (fn [type seq byte-seq offset] 299 | (s/read-form type byte-seq offset)) 300 | (fn [type seq byte-seq offset] 301 | `(s/read-value ~type ~byte-seq ~offset)))) 302 | 303 | (defmacro get-in 304 | "Like `get-in`, but for sequences of typed-structs. The sequence `s` must be keyword type-hinted with the element type, 305 | which allows for compile-time calculation of the offset, and validation of the lookup." 306 | [s fields] 307 | (get-in-form s fields &env (not b/use-unsafe?))) 308 | 309 | (defmacro get-in' 310 | "An unsafe version of `vertigo.core/get-in` which doesn't do runtime index checking." 311 | [s fields] 312 | (get-in-form s fields &env false)) 313 | 314 | (defn- set-in-form [s fields val env validate?] 315 | (field-operation env s fields validate? 316 | (fn [type seq byte-seq offset] 317 | `(do 318 | ~(s/write-form type byte-seq offset val) 319 | nil)) 320 | (fn [type seq byte-seq offset] 321 | `(do 322 | (s/write-value ~type ~byte-seq ~offset ~val) 323 | nil)))) 324 | 325 | (defmacro set-in! 326 | "Like `assoc-in`, but for sequences of typed-structs. The sequence `s` must be keyword type-hinted with the element type, 327 | which allows for compile-time calculation of the offset, and validation of the lookup." 328 | [s fields val] 329 | (set-in-form s fields val &env (not b/use-unsafe?))) 330 | 331 | (defmacro set-in!' 332 | "An unsafe version of `set-in!` which doesn't do runtime index checking." 333 | [s fields val] 334 | (set-in-form s fields val &env (not b/use-unsafe?))) 335 | 336 | (defn- update-in-form [s fields f args env validate?] 337 | (field-operation env s fields false 338 | (fn [type seq byte-seq offset] 339 | `(let [offset## ~offset 340 | byte-seq## ~byte-seq 341 | val## ~(s/read-form type `byte-seq## `offset##) 342 | val'## (~f val## ~@args)] 343 | ~(s/write-form type `byte-seq## `offset## `val'##) 344 | nil)) 345 | (fn [type seq byte-seq offset] 346 | (let [offset# ~offset 347 | byte-seq# ~byte-seq 348 | element-type# ~type 349 | val# (s/read-value element-type# byte-seq# offset#) 350 | val'# (~f val# ~@args)] 351 | (s/write-value element-type# byte-seq# offset# val'#) 352 | nil)))) 353 | 354 | (defmacro update-in! 355 | "Like `update-in`, but for sequences of typed-structs. The sequence `s` must be keyword type-hinted with the element type, 356 | which allows for compile-time calculation of the offset, and validation of the lookup." 357 | [s fields f & args] 358 | (update-in-form s fields f args &env (not b/use-unsafe?))) 359 | 360 | (defmacro update-in!' 361 | "An unsafe version of `update-in!` which doesn't do runtime index checking." 362 | [s fields f & args] 363 | (update-in-form s fields f args &env false)) 364 | 365 | ;;; 366 | 367 | (defn- over- [s lookup] 368 | (let [element-type (s/element-type s) 369 | type (s/array element-type (count s)) 370 | {:keys [slice cross-sections inner-type]} (lookup-info type lookup) 371 | byte-seq (-> s bytes/to-byte-buffers b/to-byte-seq) 372 | byte-seq (if-let [{:keys [offset length]} slice] 373 | (b/slice byte-seq (apply + offset) length) 374 | byte-seq)] 375 | (->> cross-sections 376 | (reduce 377 | (fn [byte-seq {:keys [offset-exprs length stride]}] 378 | (b/cross-section byte-seq (apply + offset-exprs) length stride)) 379 | byte-seq) 380 | (s/wrap-byte-seq inner-type)))) 381 | 382 | (defmacro over 383 | "Allows you to select a flattened subset of a sequence. The `fields` are nested lookups, 384 | as you would use in `get-in`, but where a field describes an index in an array, the may be 385 | either '_' or a symbol prefixed with '?' to select all indices in that position. 386 | 387 | Take a 2x2 matrix of integers counting up from 0 to 3: 388 | 389 | ((0 1) (2 3)) 390 | 391 | We get a flattened view of all integers within the matrix by marking both indices as free: 392 | 393 | (over s [_ _]) => (0 1 2 3) 394 | 395 | However, we can also iterate over only the elements in the first array: 396 | 397 | (over s [0 _]) => (0 1) 398 | 399 | Or only the first elements of all arrays: 400 | 401 | (over s [_ 0]) => (0 2) 402 | 403 | This syntax can be used in `doreduce` blocks to specify what subset to iterate over." 404 | [s fields] 405 | `(#'vertigo.core/over- ~s ~(vec (map #(if (free-variable? %) `'~% %) fields)))) 406 | 407 | ;;; 408 | 409 | (defmulti ^:private walk-return-exprs 410 | (fn [f x] 411 | (if (seq? x) 412 | (first x) 413 | ::default)) 414 | :default ::default) 415 | 416 | (defmethod walk-return-exprs ::default [f x] 417 | (f x)) 418 | 419 | (defn- walk-last [f x] 420 | (concat 421 | (butlast x) 422 | [(walk-return-exprs f (last x))])) 423 | 424 | (defmethod walk-return-exprs 'do [f x] 425 | (walk-last f x)) 426 | 427 | (defmethod walk-return-exprs 'let* [f x] 428 | (walk-last f x)) 429 | 430 | (defmethod walk-return-exprs 'loop* [f x] 431 | (walk-last f x)) 432 | 433 | (defmethod walk-return-exprs 'if [f [_ test then else]] 434 | `(if ~test 435 | ~(walk-return-exprs f then) 436 | ~(walk-return-exprs f else))) 437 | 438 | (defmethod walk-return-exprs 'fn* [f x] 439 | (let [header (take-while (complement seq?) x) 440 | body (drop (count header) x)] 441 | `(~@header ~@(map #(concat (butlast %) [(walk-return-exprs f x)]) body)))) 442 | 443 | (defn- macroexpand-all [form elements values] 444 | (rc/with-lexical-scoping 445 | (doseq [x (concat elements values)] 446 | (rc/register-local x nil)) 447 | (rw/macroexpand-all form))) 448 | 449 | (defn- break-expr? [x] 450 | (and (seq? x) (symbol? (first x)) (= "break" (name (first x))))) 451 | 452 | (defn- iteration-arguments [seq-bindings value-bindings env] 453 | (let [seq-options (->> seq-bindings (drop-while (complement keyword?)) (apply hash-map)) 454 | seq-bindings (take-while (complement keyword?) seq-bindings) 455 | elements (->> seq-bindings (partition 2) (map first)) 456 | seqs (->> seq-bindings (take-while (complement keyword?)) (partition 2) (map second)) 457 | value-syms (->> value-bindings (partition 2) (map first)) 458 | initial-values (->> value-bindings (partition 2) (map second)) 459 | seq-syms (repeatedly #(gensym "seq")) 460 | over? (clojure.core/every? 461 | #(and (seq? %) 462 | (or (= 'over (first %)) 463 | (= #'vertigo.core/over (resolve (first %))))) 464 | seqs) 465 | arguments {:over? over? 466 | :seqs (map 467 | #(zipmap [:element :sym :expr] %&) 468 | elements 469 | seq-syms 470 | seqs) 471 | :values (map 472 | #(zipmap [:initial :sym] %&) 473 | initial-values 474 | value-syms)}] 475 | (if-not over? 476 | 477 | (merge arguments 478 | {:step (long (get seq-options :step 1)) 479 | :limit (get seq-options :limit) 480 | :count-expr `(long (.count ~(with-meta (first seqs) {:tag "clojure.lang.Counted"})))}) 481 | 482 | (let [fields (map last seqs) 483 | seqs (map second seqs) 484 | iterators (->> fields 485 | first 486 | (filter free-variable?) 487 | (map #(if (= '_ %) (gensym "?itr__") %))) 488 | fields (map 489 | #(first 490 | (reduce 491 | (fn [[fields iterators] field] 492 | (if (free-variable? field) 493 | [(conj fields (first iterators)) (rest iterators)] 494 | [(conj fields field) iterators])) 495 | [[] iterators] 496 | %)) 497 | fields) 498 | lookup-info (map 499 | (fn [seq fields] 500 | (lookup-info 501 | (s/array 502 | (->> seq (element-type env) resolve-type) 503 | Integer/MAX_VALUE) 504 | fields)) 505 | seqs 506 | fields) 507 | lengths (map 508 | (fn [seq-syms lengths] 509 | (cons 510 | `(.count ~(with-meta seq-syms {:tag "clojure.lang.Counted"})) 511 | (rest lengths))) 512 | seq-syms 513 | (map :lengths lookup-info)) 514 | steps (if-let [step (seq-options :step)] 515 | (if (= 1 (count iterators)) 516 | [step] 517 | (throw (IllegalArgumentException. 518 | ":step is ambiguous when there are multiple free variables."))) 519 | (map #(get (:steps seq-options) % 1) iterators)) 520 | 521 | bounds (->> (mapcat (partial map vector) fields lengths) 522 | (filter #(symbol? (first %))) 523 | (map #(apply hash-map %)) 524 | (apply merge-with 525 | (fn [a b] 526 | (if (and (number? a) (number? b)) 527 | (min a b) 528 | `(p/min ~a ~b))))) 529 | 530 | limits (merge bounds 531 | (if-let [limit (seq-options :limit)] 532 | (if (= 1 (count iterators)) 533 | {(first iterators) limit} 534 | (throw (IllegalArgumentException. 535 | ":limit is ambiguous when there are multiple free variables."))) 536 | (seq-options :limits)))] 537 | 538 | (doseq [k (keys limits)] 539 | (when (and 540 | (number? (limits k)) 541 | (number? (bounds k)) 542 | (> (limits k) (bounds k))) 543 | (throw (IllegalArgumentException. 544 | (str ":limit of " (limits k) " is greater than max value " (bounds k)))))) 545 | 546 | (merge 547 | 548 | (update-in arguments [:seqs] 549 | (fn [seqs] 550 | (map 551 | #(-> %1 552 | (assoc :fields %2) 553 | (update-in [:expr] second)) 554 | seqs 555 | fields))) 556 | 557 | (let [iterator-limits (map #(get limits %) iterators)] 558 | 559 | {:limits limits 560 | :iterators (map #(zipmap [:sym :length :step :limit :limit-sym] %&) 561 | iterators 562 | (first lengths) 563 | steps 564 | iterator-limits 565 | (map 566 | #(when-not (or (number? %) (and (seq? %) (= `p/min (first %)))) 567 | (gensym "lmt__")) 568 | iterator-limits))})))))) 569 | 570 | (defn- iteration-form [seq-bindings value-bindings env body] 571 | (let [{:keys [over?] :as arguments} (iteration-arguments seq-bindings value-bindings env)] 572 | (if-not over? 573 | 574 | ;; normal linear iteration 575 | (let [{:keys [step limit count-expr values seqs]} arguments] 576 | (unify-gensyms 577 | `(let [~@(mapcat 578 | (fn [{:keys [sym expr]}] 579 | (list (with-element-type sym expr env) expr)) 580 | seqs) 581 | limit## ~(or limit count-expr)] 582 | 583 | ~@(when (and (not b/use-unsafe?) limit) 584 | `((let [cnt# ~count-expr] 585 | (when (>= limit## cnt#) 586 | (throw (IndexOutOfBoundsException. (str cnt#))))))) 587 | 588 | (loop [idx## 0, ~@(mapcat 589 | (fn [{:keys [initial sym]}] 590 | [sym initial]) 591 | values)] 592 | (if (< idx## limit##) 593 | (let [~@(mapcat 594 | (fn [{:keys [sym element]}] 595 | [element `(get-in' ~sym [idx##])]) 596 | seqs)] 597 | ~(walk-return-exprs 598 | (fn [x] 599 | (cond 600 | 601 | (break-expr? x) 602 | (if (= 2 (count x)) 603 | (second x) 604 | `(tuple ~@(rest x))) 605 | 606 | (= 1 (count values)) 607 | `(recur (p/+ idx## ~step) ~x) 608 | 609 | (not= (count values) (count x)) 610 | (throw 611 | (IllegalArgumentException. 612 | (str "expected " (count values) " return values, got " (count x)))) 613 | 614 | :else 615 | `(recur (p/+ idx## ~step) ~@x))) 616 | 617 | (macroexpand-all 618 | `(do ~@body) 619 | (map :element seqs) 620 | (map :sym values)))) 621 | 622 | ~(if (= 1 (count values)) 623 | (:sym (first values)) 624 | `(tuple ~@(map :sym values)))))))) 625 | 626 | ;; multi-dimensional iteration 627 | (let [{:keys [seqs limits values iterators]} arguments 628 | body `(let [~@(mapcat 629 | (fn [{:keys [sym element fields]}] 630 | [element `(get-in' ~sym [~@fields])]) 631 | seqs)] 632 | ~(walk-return-exprs 633 | (fn [x] 634 | (cond 635 | 636 | (break-expr? x) 637 | (if (= 2 (count x)) 638 | (second x) 639 | `(tuple ~@(rest x))) 640 | 641 | (= 1 (count values)) 642 | `(recur ~@(map :sym iterators) ~x) 643 | 644 | (not= (count values) (count x)) 645 | (throw 646 | (IllegalArgumentException. 647 | (str "expected " (count values) " return values, got " (count x)))) 648 | 649 | :else 650 | `(recur ~@(map :sym iterators) ~@x))) 651 | 652 | (macroexpand-all 653 | `(do ~@body) 654 | (map :element seqs) 655 | (map :sym values)))) 656 | 657 | root-iterator (last iterators)] 658 | 659 | (unify-gensyms 660 | `(do 661 | (let [~@(mapcat 662 | (fn [{:keys [sym expr]}] 663 | (list (with-element-type sym expr env) expr)) 664 | seqs) 665 | cnt## ~(:limit (first iterators)) 666 | ~@(mapcat 667 | (fn [{:keys [limit-sym limit]}] 668 | (when limit-sym 669 | [limit-sym limit])) 670 | (butlast iterators))] 671 | 672 | ;; make sure limits are valid 673 | ~@(when-not b/use-unsafe? 674 | (->> limits 675 | (filter (fn [[sym lim]] (and (symbol? sym) (number? lim)))) 676 | (remove #(free-variable? (first %))) 677 | (concat (->> iterators 678 | (filter #(and (:limit-sym %) (not= (:limit %) (:length %)))) 679 | (map #(list (:limit-sym %) (:length %))))) 680 | (map (fn [[sym limit]] 681 | `(when (>= ~sym (long ~limit)) 682 | (throw (IndexOutOfBoundsException. 683 | (str ~(str sym) ": "(str ~sym))))))))) 684 | 685 | (loop [~@(interleave (map :sym (butlast iterators)) (repeat 0)) 686 | ~(:sym root-iterator) ~(- (:step root-iterator)) 687 | ~@(mapcat 688 | (fn [{:keys [sym initial]}] 689 | [sym initial]) 690 | values)] 691 | (let [~(:sym root-iterator) (p/+ ~(:sym root-iterator) ~(:step root-iterator)) 692 | ~@(mapcat 693 | (fn [[i j]] 694 | `(~(:sym j) 695 | (if (>= ~(:sym i) ~(or (:limit-sym i) (:limit i))) 696 | (p/+ ~(:sym j) ~(:step j)) 697 | ~(:sym j)))) 698 | (partition 2 1 (reverse iterators))) 699 | ~@(mapcat 700 | (fn [i] 701 | `(~(:sym i) 702 | (if (>= ~(:sym i) ~(or (:limit-sym i) (:limit i))) 703 | 0 704 | ~(:sym i)))) 705 | (rest iterators))] 706 | (if (< ~(:sym (first iterators)) cnt##) 707 | ~body 708 | ~(if (= 1 (count values)) 709 | (:sym (first values)) 710 | `(tuple ~@(map :sym values))))))))))))) 711 | 712 | (defmacro doreduce 713 | "A combination of `doseq` and `reduce`, this is a primitive for efficient batch operations over sequences. 714 | 715 | `doreduce` takes two binding forms, one for sequences that mirrors `doseq`, and a second 716 | for accumulators that mirrors `loop`. If there is only one accumulator, the body must 717 | return the new value for the accumulator. If there are multiple accumulators, it must return 718 | a vector containing values for each. This will not require actually allocating a vector, 719 | except for the final result. 720 | 721 | So we can easily calculate the sum of a sequence: 722 | 723 | (doreduce [x s] [sum 0] 724 | (+ x sum)) 725 | 726 | We can also sum together two sequences: 727 | 728 | (doreduce [x a, y b] [sum 0] 729 | (+ x y sum)) 730 | 731 | And we can also calculate the product and sum at the same time: 732 | 733 | (doreduce [x s] [sum 0, product 1] 734 | [(+ x sum) (* x product)]) 735 | 736 | We can also iterate over particular fields or arrays within a sequence, using the `over` 737 | syntax. This is faster than passing in a sequence which has been called with `over` 738 | elsewhere, and should be used inline where possible: 739 | 740 | (doreduce [x (over s [_ :a :b])] [sum 0] 741 | (+ x sum) 742 | 743 | Both the `:step` and `:limit` for iteration may be specified: 744 | 745 | (doreduce [x s, :step 3, :limit 10] [sum 0] 746 | (+ x sum)) 747 | 748 | This will only sum the `[0, 3, 6, 9]` indices. If there are multiple iterators, the 749 | values must be specified using `:steps` and `:limits`: 750 | 751 | (doreduce [x (over s [?i 0 ?j]), :steps {?i 2}, :limits {?j 20}] [sum 0] 752 | (+ x sum)) 753 | 754 | Limits and indices that are out of bounds will throw an exception at either compile or 755 | runtime, depending on when they can be resolved to a number." 756 | [seq-bindings value-bindings & body] 757 | (iteration-form seq-bindings value-bindings &env body)) 758 | 759 | ;;; 760 | 761 | (defmacro sum 762 | "Returns the sum of all numbers within the sequence. 763 | 764 | (sum s) 765 | 766 | or 767 | 768 | (sum s :step 2, :limit 10)" 769 | [s & options] 770 | `(doreduce [x# ~s ~@options] [sum# 0] 771 | (p/+ x# sum#))) 772 | 773 | (defmacro every? 774 | "Returns true if `pred-expr` returns true for all `x` in `s`. 775 | 776 | (every? [x s] (pos? x)) 777 | 778 | or 779 | 780 | (every? [x s, :limit 10] (even? x))" 781 | [[x s & options] pred-expr] 782 | `(doreduce [~x ~s ~@options] [bool# true] 783 | (if ~pred-expr 784 | true 785 | (break false)))) 786 | 787 | (defmacro any? 788 | "Returns true if `pred-expr` returns true for some `x` in `s`. 789 | 790 | (any? [x s] (zero? x)) 791 | 792 | or 793 | 794 | (any? [x s, :step 42] (neg? x))" 795 | [[x s & options] pred-expr] 796 | `(doreduce [~x ~s ~@options] [bool# false] 797 | (if ~pred-expr 798 | (break true) 799 | false))) 800 | -------------------------------------------------------------------------------- /src/vertigo/primitives.clj: -------------------------------------------------------------------------------- 1 | (ns vertigo.primitives 2 | (:refer-clojure 3 | :exclude [+ - * / inc dec rem == not== zero? <= >= < > min max]) 4 | (:use 5 | potemkin) 6 | (:require 7 | [primitive-math :as p]) 8 | (:import 9 | [primitive_math 10 | Primitives] 11 | [java.nio 12 | ByteBuffer])) 13 | 14 | ;;; 15 | 16 | (import-vars 17 | [primitive-math 18 | use-primitive-operators + - * / div inc dec rem == not== zero? <= >= < > min max]) 19 | 20 | (defn ->int8 21 | "Converts any number, character, or first character of a string to an int8." 22 | ^long [x] 23 | (long 24 | (cond 25 | (number? x) (byte x) 26 | (char? x) (-> x clojure.core/int byte) 27 | (string? x) (-> x first clojure.core/int byte) 28 | :else (throw (IllegalArgumentException. (str "Cannot convert " (pr-str x) " to byte.")))))) 29 | 30 | (import-fn p/byte int8) 31 | (import-fn p/short int16) 32 | (import-fn p/int int32) 33 | (import-fn p/long int64) 34 | (import-fn p/float float32) 35 | (import-fn p/double float64) 36 | 37 | (import-fn p/byte->ubyte int8->uint8) 38 | (import-fn p/ubyte->byte uint8->int8) 39 | (import-fn p/short->ushort int16->uint16) 40 | (import-fn p/ushort->short uint16->int16) 41 | (import-fn p/int->uint int32->uint32) 42 | (import-fn p/uint->int uint32->int32) 43 | (import-fn p/long->ulong int64->uint64) 44 | (import-fn p/ulong->long uint64->int64) 45 | 46 | (import-fn p/reverse-short reverse-int16) 47 | (import-fn p/reverse-int reverse-int32) 48 | (import-fn p/reverse-long reverse-int64) 49 | (import-fn p/reverse-float reverse-float32) 50 | (import-fn p/reverse-double reverse-float64) 51 | 52 | 53 | 54 | -------------------------------------------------------------------------------- /src/vertigo/structs.clj: -------------------------------------------------------------------------------- 1 | (ns vertigo.structs 2 | (:use 3 | potemkin) 4 | (:require 5 | [byte-streams :as bytes] 6 | [clojure.core.protocols :as proto] 7 | [vertigo.bytes :as b] 8 | [vertigo.primitives :as p])) 9 | 10 | ;;; 11 | 12 | (definterface+ IFixedType 13 | (byte-size ^long [_] 14 | "The size of the primitive type, in bytes.") 15 | (write-value [_ byte-seq ^long offset x] 16 | "Writes the value to a byte-seq at the given byte offset.") 17 | (read-value [_ byte-seq ^long offset] 18 | "Reads the value from a byte-seq at the given byte-offset.")) 19 | 20 | (definterface+ IFixedCompoundType 21 | (has-field? [_ x] 22 | "Returns true if the type has the given field, false otherwise") 23 | (field-offset ^long [_ x] 24 | "Returns of the offset of the field within the struct, in bytes.") 25 | (field-type [_ x] 26 | "Returns the type of the field.")) 27 | 28 | (definterface+ IFixedInlinedType 29 | (read-form [_ byte-seq idx] 30 | "Returns an eval'able form for reading the struct off a byte-seq.") 31 | (write-form [_ byte-seq idx val] 32 | "Returns an eval'able form for writing the struct to a byte-seq.")) 33 | 34 | ;; a utility interface for byte-streams integrations 35 | (definterface+ IByteRange 36 | (underlying-bytes [_])) 37 | 38 | (definterface+ IByteSeqWrapper 39 | (unwrap-byte-seq [_]) 40 | (index-offset ^long [_ ^long idx] 41 | "Returns the byte offset of the given index within the byte-seq-wrapper.") 42 | (element-type ^IFixedType [_] 43 | "Returns the type of the elements within the byte-seq-wrapper.")) 44 | 45 | ;;; 46 | 47 | (let [;; normal read-write 48 | s (fn [r w rev] 49 | `[(partial list '~r) 50 | (partial list '~w) 51 | (partial list '~rev)]) 52 | 53 | ;; unsigned read-write 54 | u (fn [r w rev to-unsigned to-signed] 55 | `[(fn [b# idx#] 56 | (list '~to-unsigned (list '~r b# idx#))) 57 | (fn [b# idx# val#] 58 | (list '~w b# idx# (list '~to-unsigned val#))) 59 | (fn [x#] 60 | (list '~to-unsigned (list '~rev (list '~to-signed x#))))]) 61 | 62 | types ['int8 1 (s `b/get-int8 `b/put-int8 `identity) 63 | 'uint8 1 (u `b/get-int8 `b/put-int8 `identity `p/int8->uint8 `p/uint8->int8) 64 | 'int16 2 (s `b/get-int16 `b/put-int16 `p/reverse-int16) 65 | 'uint16 2 (u `b/get-int16 `b/put-int16 `p/reverse-int16 `p/int16->uint16 `p/uint16->int16) 66 | 'int32 4 (s `b/get-int32 `b/put-int32 `p/reverse-int32) 67 | 'uint32 4 (u `b/get-int32 `b/put-int32 `p/reverse-int32 `p/int32->uint32 `p/uint32->int32) 68 | 'int64 8 (s `b/get-int64 `b/put-int64 `p/reverse-int64) 69 | 'uint64 8 (u `b/get-int64 `b/put-int64 `p/reverse-int64 `p/int64->uint64 `p/uint64->int64) 70 | 'float32 4 (s `b/get-float32 `b/put-float32 `p/reverse-float32) 71 | 'float64 8 (s `b/get-float64 `b/put-float64 `p/reverse-float64)]] 72 | 73 | (doseq [[name size read-write-rev] (partition 3 types)] 74 | (let [[read-form write-form rev-form] (eval read-write-rev)] 75 | 76 | (eval 77 | (unify-gensyms 78 | `(let [[read-form# write-form# rev-form#] ~read-write-rev] 79 | 80 | ;; basic primitive 81 | (def ~(with-meta name {:doc "A primitive type."}) 82 | (reify 83 | clojure.lang.Named 84 | (getName [_#] ~(str name)) 85 | (getNamespace [_#] ~(str *ns*)) 86 | 87 | IFixedType 88 | (byte-size [_#] 89 | ~size) 90 | (write-value [_# byte-seq## offset## x##] 91 | ~(write-form `byte-seq## `offset## `x##)) 92 | (read-value [_# byte-seq## offset##] 93 | ~(read-form `byte-seq## `offset##)) 94 | 95 | IFixedInlinedType 96 | (read-form [_# b# idx#] 97 | (read-form# b# idx#)) 98 | (write-form [_# b# idx# x#] 99 | (write-form# b# idx# x#))))))) 100 | 101 | ;; big and little-endian primitives 102 | (when-not (#{'int8 'uint8} name) 103 | (doseq [[check name] (map list 104 | [`b/big-endian? `b/little-endian?] 105 | [(symbol (str name "-le")) (symbol (str name "-be"))])] 106 | 107 | (eval 108 | (unify-gensyms 109 | `(let [[read-form# write-form# rev-form#] ~read-write-rev] 110 | (def ~(with-meta name {:doc "A primitive type."}) 111 | (reify 112 | clojure.lang.Named 113 | (getName [_#] ~(str name)) 114 | (getNamespace [_#] ~(str *ns*)) 115 | 116 | IFixedType 117 | (byte-size [_#] 118 | ~size) 119 | (write-value [_# byte-seq## offset## x##] 120 | (let [x## (if (~check byte-seq##) 121 | ~(rev-form `x##) 122 | x##)] 123 | ~(write-form `byte-seq## `offset## `x##))) 124 | (read-value [_# byte-seq## offset##] 125 | (let [x## ~(read-form `byte-seq## `offset##)] 126 | (if (~check byte-seq##) 127 | ~(rev-form `x##) 128 | x##))) 129 | 130 | IFixedInlinedType 131 | (read-form [_# b# idx#] 132 | (list 'let [`x## (read-form# b# idx#)] 133 | (list 'if (list ~check b#) 134 | (rev-form# `x##) 135 | `x##))) 136 | (write-form [_# b# idx# x#] 137 | (list 'let [`x## (list 'if (list ~check b#) 138 | (list rev-form# x#) 139 | x#)] 140 | (write-form# b# idx# `x##))))))))))))) 141 | 142 | ;;; 143 | 144 | (def ^:dynamic *types*) 145 | 146 | (defn typed-struct 147 | "A data structure with explicit types, meant to sit atop a byte-seq. Fields must be keys, 148 | and types must implement `IFixedType`. For better error messages, all structs must be named. 149 | 150 | `(typed-struct 'vec2 :x float32 :y float32)` 151 | 152 | The resulting value implements `IFixedType`, and can be used within other typed-structs." 153 | [name & field+types] 154 | 155 | (assert (even? (count field+types))) 156 | 157 | (let [fields (->> field+types (partition 2) (map first)) 158 | types (->> field+types (partition 2) (map second))] 159 | 160 | (assert (every? keyword? fields)) 161 | 162 | (doseq [[field type] (map list fields types)] 163 | (when-not (instance? IFixedType type) 164 | (throw (IllegalArgumentException. (str field " is not a valid type."))))) 165 | 166 | (let [offsets (->> types (map byte-size) (cons 0) (reductions +) butlast) 167 | byte-size (->> types (map byte-size) (apply +)) 168 | type-syms (map #(symbol (str "t" %)) (range (count types)))] 169 | 170 | (binding [*types* types] 171 | (eval 172 | (unify-gensyms 173 | `(let [~@(interleave type-syms 174 | (map 175 | (fn [x] `(nth *types* ~x)) 176 | (range (count types))))] 177 | (reify 178 | clojure.lang.Named 179 | (~'getName [_#] ~(str name)) 180 | (~'getNamespace [_#] ~(namespace name)) 181 | 182 | IFixedCompoundType 183 | (has-field? [_# x#] 184 | (boolean (~(set fields) x#))) 185 | (field-offset [_# k#] 186 | (long 187 | (case k# 188 | ~@(interleave fields offsets)))) 189 | (field-type [_# k#] 190 | (case k# 191 | ~@(interleave 192 | fields 193 | type-syms))) 194 | 195 | IFixedType 196 | (byte-size [_#] 197 | ~byte-size) 198 | (write-value [_# byte-seq## offset## x##] 199 | ~@(map 200 | (fn [k offset x] 201 | `(write-value ~x byte-seq## (p/+ (long offset##) ~offset) (get x## ~k))) 202 | fields 203 | offsets 204 | type-syms)) 205 | (read-value [_# byte-seq# offset##] 206 | (let [byte-seq## (b/slice byte-seq# offset## ~byte-size)] 207 | 208 | ;; map structure that sits atop a slice of the byte-seq 209 | (reify-map-type 210 | IByteRange 211 | (underlying-bytes [_#] 212 | (bytes/to-byte-buffers byte-seq##)) 213 | 214 | (~'keys [_#] 215 | ~(vec fields)) 216 | (~'get [_# k# default-value#] 217 | (case k# 218 | ~@(interleave 219 | fields 220 | (map 221 | (fn [x offset] 222 | `(read-value ~x byte-seq## ~offset)) 223 | type-syms 224 | offsets)) 225 | default-value#)) 226 | (~'assoc [this# k# v#] 227 | (assoc (into {} this#) k# v#)) 228 | (~'dissoc [this# k#] 229 | (dissoc (into {} this#) k#))))))))))))) 230 | 231 | (defmacro def-typed-struct 232 | "Like `typed-struct`, but defines a var." 233 | [name & field+types] 234 | `(def ~(symbol (str *ns*) (str name)) (typed-struct '~name ~@field+types))) 235 | 236 | ;;; 237 | 238 | (defn- byte-seq-wrapper-reduce 239 | ([byte-seq-wrapper f start] 240 | (proto/internal-reduce byte-seq-wrapper f start)) 241 | ([byte-seq-wrapper f] 242 | (if (nil? byte-seq-wrapper) 243 | (f) 244 | (proto/internal-reduce (next byte-seq-wrapper) f (first byte-seq-wrapper))))) 245 | 246 | ;; a type that, given an IFixedType, can treat a byte-seq as a sequence of that type 247 | ;; `stride` and `offset` are so that an inner type be iterated over without copying 248 | (deftype ByteSeqWrapper 249 | [^long stride 250 | ^long offset 251 | ^vertigo.structs.IFixedType type 252 | ^vertigo.bytes.IByteSeq byte-seq] 253 | 254 | java.io.Closeable 255 | (close [_] 256 | (when-let [close-fn (b/close-fn byte-seq)] 257 | (close-fn byte-seq))) 258 | 259 | IByteSeqWrapper 260 | (unwrap-byte-seq [_] byte-seq) 261 | (element-type [_] type) 262 | (index-offset [_ idx] (p/+ offset (p/* idx stride))) 263 | 264 | clojure.lang.ISeq 265 | clojure.lang.Seqable 266 | clojure.lang.Sequential 267 | clojure.lang.Indexed 268 | clojure.lang.ILookup 269 | 270 | (first [_] 271 | (read-value type byte-seq offset)) 272 | (next [_] 273 | (when-let [byte-seq' (b/drop-bytes byte-seq stride)] 274 | (ByteSeqWrapper. stride offset type byte-seq'))) 275 | (more [this] 276 | (or (next this) '())) 277 | (count [_] 278 | (p/div (b/byte-count byte-seq) stride)) 279 | (nth [_ idx] 280 | (read-value type byte-seq (p/+ offset (p/* stride (long idx))))) 281 | (nth [this idx default-value] 282 | (try 283 | (nth this idx) 284 | (catch IndexOutOfBoundsException e 285 | default-value))) 286 | (valAt [this idx] 287 | (nth this idx)) 288 | (valAt [this idx not-found] 289 | (nth this idx not-found)) 290 | (seq [this] 291 | this) 292 | (equiv [this x] 293 | (if-not (sequential? x) 294 | false 295 | (loop [a this, b (seq x)] 296 | (if (or (empty? a) (empty? b)) 297 | (and (empty? a) (empty? b)) 298 | (if (= (first a) (first b)) 299 | (recur (rest a) (rest b)) 300 | false))))) 301 | 302 | proto/InternalReduce 303 | 304 | (internal-reduce [_ f start] 305 | (b/byte-seq-reduce byte-seq stride 306 | (fn [byte-seq idx] 307 | (read-value type byte-seq (p/+ offset (long idx)))) 308 | f 309 | start)) 310 | 311 | proto/CollReduce 312 | (coll-reduce [this f start] 313 | (byte-seq-wrapper-reduce this f start)) 314 | 315 | (coll-reduce [this f] 316 | (byte-seq-wrapper-reduce this f))) 317 | 318 | (defn wrap-byte-seq 319 | "Treats the byte-seq as a sequence of the given type." 320 | ([type byte-seq] 321 | (wrap-byte-seq type (byte-size type) 0 byte-seq)) 322 | ([type stride offset byte-seq] 323 | (ByteSeqWrapper. stride offset type byte-seq))) 324 | 325 | (bytes/def-conversion [ByteSeqWrapper (bytes/seq-of java.nio.ByteBuffer)] 326 | [wrapper] 327 | (let [byte-seq (.byte-seq wrapper) 328 | byte-size (byte-size (element-type wrapper)) 329 | subset? (p/not== byte-size (.stride wrapper))] 330 | (bytes/to-byte-buffers 331 | (if subset? 332 | (b/cross-section byte-seq (.offset wrapper) byte-size (.stride wrapper)) 333 | byte-seq)))) 334 | 335 | (bytes/def-conversion [IByteRange (bytes/seq-of java.nio.ByteBuffer)] 336 | [byte-range] 337 | (underlying-bytes byte-range)) 338 | 339 | ;;; 340 | 341 | (deftype FixedTypeArray 342 | [type 343 | ^long len 344 | ^long offset 345 | ^long stride] 346 | 347 | clojure.lang.Named 348 | (getName [_] 349 | (let [sub-types (->> type 350 | (iterate #(when (instance? FixedTypeArray %) (.type ^FixedTypeArray %))) 351 | (take-while (complement nil?)) 352 | seq)] 353 | (apply str 354 | (name (last sub-types)) "[" len "]" 355 | (map #(str "[" (.len ^FixedTypeArray %) "]") (butlast sub-types))))) 356 | (getNamespace [_] ) 357 | 358 | IFixedCompoundType 359 | (has-field? [_ idx] 360 | (and (number? idx) (<= 0 idx (dec len)))) 361 | (field-type [_ idx] 362 | type) 363 | (field-offset [_ idx] 364 | (p/* (long idx) stride)) 365 | 366 | IFixedType 367 | (byte-size [_] 368 | (p/* len stride)) 369 | (read-value [_ buf offset'] 370 | (wrap-byte-seq 371 | type stride offset 372 | (b/slice buf offset' (p/* stride len)))) 373 | (write-value [_ buf offset' x] 374 | (loop [s x, idx 0] 375 | (when-not (p/== idx len) 376 | (write-value type buf (p/+ offset' offset (p/* idx stride)) (first s)) 377 | (recur (rest s) (p/inc idx)))))) 378 | 379 | (defn array 380 | "Returns a type representing an array of `type` with dimensions `dims`." 381 | [type & dims] 382 | (reduce 383 | (fn [type dim] 384 | (FixedTypeArray. type (long dim) 0 (byte-size type))) 385 | type 386 | (reverse dims))) 387 | -------------------------------------------------------------------------------- /test/vertigo/bytes_test.clj: -------------------------------------------------------------------------------- 1 | (ns vertigo.bytes-test 2 | (:use 3 | clojure.test) 4 | (:require 5 | [vertigo.test-utils :as t] 6 | [criterium.core :as c] 7 | [vertigo.bytes :as b] 8 | [vertigo.primitives :as p] 9 | [clojure.java.io :as io]) 10 | (:import 11 | [java.io 12 | ByteArrayInputStream] 13 | [java.nio 14 | ByteBuffer 15 | ByteOrder] 16 | [java.nio.channels 17 | Channels] 18 | [vertigo.bytes 19 | ByteSeq])) 20 | 21 | (def types 22 | {:int8 [1 b/put-int8 b/get-int8] 23 | :int16 [2 b/put-int16 b/get-int16] 24 | :int32 [4 b/put-int32 b/get-int32] 25 | :int64 [8 b/put-int64 b/get-int64] 26 | :float32 [4 b/put-float32 b/get-float32] 27 | :float64 [8 b/put-float64 b/get-float64]}) 28 | 29 | (defn repeatedly-put [type s] 30 | (let [[byte-size put-f get-f] (types type) 31 | buf (->> byte-size (* (count s)) ByteBuffer/allocate b/byte-seq)] 32 | (doseq [[idx x] (map vector (iterate inc 0) s)] 33 | (put-f buf (* idx byte-size) x)) 34 | buf)) 35 | 36 | (defn repeatedly-get [buf type cnt] 37 | (let [[byte-size put-f get-f] (types type)] 38 | (map 39 | #(get-f buf (* byte-size %)) 40 | (range cnt)))) 41 | 42 | (defn ->chunked [byte-seq chunk-size] 43 | (b/to-chunked-byte-seq byte-seq {:chunk-size chunk-size})) 44 | 45 | (deftest test-roundtrip 46 | (doseq [t (take 1 (keys types))] 47 | (let [s (if (#{:float32 :float64} t) 48 | (map double (range 10)) 49 | (range 10)) 50 | [byte-size put-f get-f] (types t)] 51 | 52 | ;; basic roundtrip 53 | (is (= s (repeatedly-get (repeatedly-put t s) t 10))) 54 | 55 | ;; chunked roundtrip 56 | (doseq [i (range 1 10)] 57 | (is (= s (repeatedly-get 58 | (->chunked (repeatedly-put t s) (* byte-size i)) 59 | t 10)))) 60 | 61 | ;; dropped roundtrip 62 | (is (= s (map 63 | #(get-f 64 | (b/drop-bytes 65 | (repeatedly-put t s) 66 | (* byte-size %)) 67 | 0) 68 | (range 10)))) 69 | 70 | ;; dropped chunked roundtrip 71 | (doseq [i (range 1 10)] 72 | (is (= s (map 73 | #(get-f 74 | (b/drop-bytes 75 | (->chunked (repeatedly-put t s) (* byte-size i)) 76 | (* byte-size %)) 77 | 0) 78 | (range 10)))))) 79 | 80 | )) 81 | 82 | ;;; 83 | 84 | (deftest ^:benchmark benchmark-raw-buffer 85 | (let [buf (.order (ByteBuffer/allocateDirect 8) (ByteOrder/nativeOrder))] 86 | (t/batch-bench "raw direct byte-buffer: get" 87 | (.getLong buf 0)) 88 | (t/batch-bench "raw direct byte-buffer: put" 89 | (.putLong buf 0 0)) 90 | (t/batch-bench "raw direct byte-buffer: get and put" 91 | (.putLong buf 0 (p/inc (.getLong buf 0))))) 92 | (let [buf (.order (ByteBuffer/allocate 8) (ByteOrder/nativeOrder))] 93 | (t/batch-bench "raw indirect byte-buffer: get" 94 | (.getLong buf 0)) 95 | (t/batch-bench "raw indirect byte-buffer: put" 96 | (.putLong buf 0 0)) 97 | (t/batch-bench "raw indirect byte-buffer: get and put" 98 | (.putLong buf 0 (p/inc (.getLong buf 0)))))) 99 | 100 | (deftest ^:benchmark benchmark-array 101 | (let [ary (long-array 1)] 102 | (t/batch-bench "array: get" 103 | (aget ary 0)) 104 | (t/batch-bench "array: put" 105 | (aset ary 0 0)) 106 | (t/batch-bench "array: get and put" 107 | (aset ary 0 (p/inc (long (aget ary 0))))))) 108 | 109 | (deftest ^:benchmark benchmark-byte-seq 110 | (let [bs (b/byte-seq (ByteBuffer/allocateDirect 8))] 111 | (t/batch-bench "byte-seq: get" 112 | (b/get-int64 bs 0)) 113 | (t/batch-bench "byte-seq: put" 114 | (b/put-int64 bs 0 0)) 115 | (t/batch-bench "byte-seq: get and put" 116 | (b/put-int64 bs 0 (p/inc (b/get-int64 bs 0))))) 117 | (let [bs (b/byte-seq (ByteBuffer/allocate 8))] 118 | (t/batch-bench "indirect byte-seq: get" 119 | (b/get-int64 bs 0)) 120 | (t/batch-bench "indirect byte-seq: put" 121 | (b/put-int64 bs 0 0)) 122 | (t/batch-bench "indirect byte-seq: get and put" 123 | (b/put-int64 bs 0 (p/inc (b/get-int64 bs 0)))))) 124 | 125 | (deftest ^:benchmark benchmark-chunked-byte-seq 126 | (let [bs (-> (ByteBuffer/allocateDirect 40) b/byte-seq (->chunked 8))] 127 | (t/batch-bench "chunked-byte-seq first chunk: get" 128 | (b/get-int64 bs 0)) 129 | (t/batch-bench "chunked-byte-seq first chunk: put" 130 | (b/put-int64 bs 0 0)) 131 | (t/batch-bench "chunked-byte-seq first chunk: get and put" 132 | (b/put-int64 bs 0 (p/inc (b/get-int64 bs 0)))) 133 | 134 | (t/batch-bench "chunked-byte-seq fifth chunk: get" 135 | (b/get-int64 bs 32)) 136 | (t/batch-bench "chunked-byte-seq fifth chunk: put" 137 | (b/put-int64 bs 32 0)) 138 | (t/batch-bench "chunked-byte-seq fifth chunk: get and put" 139 | (b/put-int64 bs 32 (p/inc (b/get-int64 bs 32)))))) 140 | -------------------------------------------------------------------------------- /test/vertigo/core_test.clj: -------------------------------------------------------------------------------- 1 | (ns vertigo.core-test 2 | (:use 3 | clojure.test 4 | vertigo.test-utils) 5 | (:require 6 | [vertigo.core :as c] 7 | [vertigo.bytes :as b] 8 | [vertigo.primitives :as p] 9 | [vertigo.structs :as s])) 10 | 11 | (def int-matrix (s/array s/int64 10 10)) 12 | 13 | (def ^:int-matrix int64-matrices 14 | (c/marshal-seq int-matrix 15 | (->> (range 1e3) 16 | (partition 10) 17 | (partition 10)))) 18 | 19 | (def array-dim 1e3) 20 | 21 | (def ^:s/int64 int64-array 22 | (c/marshal-seq s/int64 (range array-dim))) 23 | 24 | (def ^:s/int32 int32-array 25 | (c/marshal-seq s/int32 (range array-dim))) 26 | 27 | (def ^:s/int16 int16-array 28 | (c/marshal-seq s/int16 (range array-dim))) 29 | 30 | (deftest test-over 31 | (are [expected fields] 32 | (= expected (c/over int64-matrices fields)) 33 | 34 | (range 0 10) [0 0 _] 35 | (range 0 1000) [_ _ _] 36 | (range 990 1000) [9 9 _] 37 | (range 600 700) [6 _ _] 38 | 39 | ;; 90-100, 190-200, ... 40 | (mapcat #(range (+ (* % 100) 90) (* (inc %) 100)) (range 10)) 41 | [_ 9 _] 42 | 43 | ;; 101, 111, 121, ... 44 | (map #(+ 100 (* % 10) 1) (range 10)) 45 | [1 _ 1])) 46 | 47 | (deftest test-doreduce 48 | 49 | ;; 1d 50 | (comment 51 | (is (= (reduce + (range array-dim)) 52 | (c/doreduce [x int64-array] [sum 0] 53 | (+ x sum)))) 54 | (is (= (reduce + (range 10)) 55 | (c/doreduce [x int64-array] [sum 0] 56 | (if (= 10 x) 57 | (break sum) 58 | (+ x sum))))) 59 | (is (= (* 2 (reduce + (range array-dim))) 60 | (c/doreduce [x int64-array, y int64-array] [sum 0] 61 | (+ x y sum)))) 62 | (is (= (->> (range array-dim) (partition 2) (map first) (reduce +)) 63 | (c/doreduce [x int64-array :step 2] [sum 0] 64 | (+ x sum)))) 65 | (is (= (->> (range array-dim) (take 600) (partition 4) (map first) (reduce +)) 66 | (c/doreduce [x int64-array :step 4 :limit 600] [sum 0] 67 | (+ x sum)))) 68 | (is (= [(reduce + (range 10)) (reduce * (range 1 11))] 69 | (c/doreduce [x int64-array :limit 10] [sum 0, product 1] 70 | [(+ x sum) (* (inc x) product)])))) 71 | 72 | ;; n-d 73 | (is (= (reduce + (range array-dim)) 74 | (c/doreduce [x (c/over int64-array [_])] [sum 0] 75 | (+ x sum)))) 76 | (is (= (reduce + (range 10)) 77 | (c/doreduce [x (c/over int64-array [_])] [sum 0] 78 | (if (= 10 x) 79 | (break sum) 80 | (+ x sum))))) 81 | (is (= (->> (range array-dim) (partition 2) (map first) (reduce +)) 82 | (c/doreduce [x (over int64-array [_]) :step 2] [sum 0] 83 | (+ x sum)))) 84 | (is (= (->> (range array-dim) (take 100) (reduce +)) 85 | (c/doreduce [x (over int64-array [_]) :limit 100] [sum 0] 86 | (+ x sum)))) 87 | (is (= (reduce + (range 1e3)) 88 | (c/doreduce [x (over int64-matrices [_ _ _])] [sum 0] 89 | (+ x sum)))) 90 | (is (= [(reduce + (range 10)) (reduce * (range 1 11))] 91 | (c/doreduce [x (over int64-matrices [0 0 _])] [sum 0, product 1] 92 | [(+ x sum) (* (inc x) product)]))) 93 | (is (= (reduce + (range 200)) 94 | (c/doreduce [x (over int64-matrices [?x _ _]) :limits {?x 2}] [sum 0] 95 | (+ x sum)))) 96 | (is (= (* 2 (reduce + (range 200))) 97 | (c/doreduce [x (over int64-matrices [?x _ _]) 98 | y (over int64-matrices [?y _ _]) 99 | :limits {?x 2, ?y 2}] 100 | [sum 0] 101 | (+ x y sum)))) 102 | (is (= (->> (range 200) (partition 2) (map first) (reduce +)) 103 | (c/doreduce [x (over int64-matrices [?x _ ?z]) :limits {?x 2} :steps {?z 2}] [sum 0] 104 | (+ x sum)))) 105 | 106 | (is (thrown? IllegalArgumentException 107 | (eval 108 | '(vertigo.core/doreduce [x (over int64-matrices [_ ?x _]) :limits {?x 11}] [sum 0] 109 | (+ x sum))))) 110 | (is (thrown? IndexOutOfBoundsException 111 | (let [n 11] 112 | (vertigo.core/doreduce [x (over int64-matrices [_ ?x _]) :limits {?x n}] [sum 0] 113 | (+ x sum))))) 114 | (is (thrown? IndexOutOfBoundsException 115 | (let [n 11] 116 | (c/doreduce [x (over int64-matrices [?x n _]) :limits {?x 2}] [sum 0] 117 | (+ x sum))))) 118 | (is (thrown? IndexOutOfBoundsException 119 | (let [n 11] 120 | (c/doreduce [x (over int64-matrices [n _ _])] [sum 0] 121 | (+ x sum)))))) 122 | 123 | ;;; 124 | 125 | (deftest ^:benchmark benchmark-reduce-matrix-sum 126 | (let [^:s/int64 s (c/over int64-matrices [0 0 _])] 127 | (bench "pre-over 1d sum" 128 | (c/doreduce [x s] [sum 0] 129 | (p/+ sum x)))) 130 | (bench "simple 1d sum" 131 | (c/doreduce [x (over int64-array [_])] [sum 0] 132 | (p/+ sum x))) 133 | (bench "1d sum" 134 | (c/doreduce [x (over int64-matrices [0 0 _])] [sum 0] 135 | (p/+ sum x))) 136 | (bench "2d sum" 137 | (c/doreduce [x (over int64-matrices [0 _ _])] [sum 0] 138 | (p/+ sum x))) 139 | (bench "3d sum" 140 | (c/doreduce [x (over int64-matrices [_ _ _])] [sum 0] 141 | (p/+ sum x)))) 142 | 143 | (deftest ^:benchmark benchmark-reduce-sum 144 | (let [num-array (long-array (range array-dim))] 145 | (bench "array reducer sum" 146 | (reduce + num-array)) 147 | (bench "areduce sum" 148 | (areduce num-array i sum 0 (p/+ sum (aget num-array i))))) 149 | (let [s (range array-dim)] 150 | (bench "seq sum" 151 | (reduce + s))) 152 | (let [s (take array-dim (iterate inc 0))] 153 | (bench "lazy-seq sum" 154 | (reduce + s))) 155 | (bench "doreduce sum int64" 156 | (c/doreduce [x int64-array] [sum 0] 157 | (p/+ sum x))) 158 | (bench "doreduce sum int32" 159 | (c/doreduce [x int32-array] [sum 0] 160 | (p/+ sum x))) 161 | (bench "doreduce sum int16" 162 | (c/doreduce [x int16-array] [sum 0] 163 | (p/+ sum x)))) 164 | -------------------------------------------------------------------------------- /test/vertigo/structs_test.clj: -------------------------------------------------------------------------------- 1 | (ns vertigo.structs-test 2 | (:use 3 | clojure.test 4 | vertigo.test-utils) 5 | (:require 6 | [byte-streams :as bytes] 7 | [vertigo.core :as c] 8 | [vertigo.bytes :as b] 9 | [vertigo.primitives :as p] 10 | [vertigo.structs :as s])) 11 | 12 | (def primitives 13 | [s/int8 14 | s/uint8 15 | s/int16 16 | s/int16-le 17 | s/int16-be 18 | s/uint16 19 | s/uint16-le 20 | s/uint16-be 21 | s/int32 22 | s/int32-le 23 | s/int32-be 24 | s/uint32 25 | s/uint32-le 26 | s/uint32-be 27 | s/int64 28 | s/int64-le 29 | s/int64-be 30 | s/uint64 31 | s/uint64-le 32 | s/uint64-be 33 | s/float32 34 | s/float32-le 35 | s/float64-be 36 | s/float64 37 | s/float64-le 38 | s/float64-be]) 39 | 40 | (deftest test-simple-roundtrips 41 | (doseq [typ primitives] 42 | (let [s (if (re-find #"float" (name typ)) 43 | (map double (range 10)) 44 | (range 10))] 45 | (is (= s (c/marshal-seq typ s))) 46 | (is (= s (c/lazily-marshal-seq typ s))) 47 | (is (= s (c/lazily-marshal-seq typ s {:chunk-size 32})))))) 48 | 49 | (defn temp-file [] 50 | (doto (java.io.File/createTempFile "vertigo" ".tmp") 51 | (.deleteOnExit))) 52 | 53 | (deftest test-format-roundtrips 54 | (let [s (c/marshal-seq s/int64 (range 1e5))] 55 | (is (= s (->> s bytes/to-byte-buffer (c/wrap s/int64)))) 56 | (is (= s (->> s bytes/to-readable-channel (c/wrap s/int64))))) 57 | 58 | (let [f (temp-file) 59 | s (c/marshal-seq s/int64 (range 1e5))] 60 | (bytes/transfer s f) 61 | (is (= s (c/wrap s/int64 f))))) 62 | 63 | (s/def-typed-struct tuple 64 | :x s/int32 65 | :y (s/array s/int64 10)) 66 | 67 | (deftest test-typed-struct 68 | (let [s (map 69 | #(hash-map :x %1 :y %2) 70 | (range 10) 71 | (repeat 10 (range 10))) 72 | ^:tuple ms (c/marshal-seq tuple s)] 73 | 74 | (is (= s ms)) 75 | (is (= 0 76 | (c/get-in ms [0 :x]) 77 | (c/get-in ms [0 :y 0]) 78 | (get-in ms [0 :y 0]) 79 | (get-in ms [0 :x]) 80 | (let [idx 0] 81 | (c/get-in ms [idx :y idx])))) 82 | 83 | (is (thrown? IndexOutOfBoundsException 84 | (let [n 11] 85 | (prn (c/get-in ms [0 :y n]))))) 86 | 87 | (is (= (range 10) 88 | (c/get-in ms [0 :y]) 89 | (get-in ms [0 :y]))) 90 | 91 | (is (= {:x 1 :y (range 10)} 92 | (c/get-in ms [1]) 93 | (get-in ms [1]) 94 | (nth ms 1)))) 95 | 96 | (let [s (map 97 | #(hash-map :x %1 :y %2) 98 | (range 1) 99 | (repeat 1 (range 10))) 100 | ^:tuple ms (c/marshal-seq tuple s) 101 | n 0] 102 | 103 | (c/update-in! ms [0 :x] p/inc) 104 | (is (= 1 (get-in ms [0 :x]) (c/get-in ms [0 :x]) (c/get-in ms [n :x]))) 105 | (c/set-in! ms [0 :x] 10) 106 | (is (= 10 (get-in ms [0 :x]) (c/get-in ms [0 :x]) (c/get-in ms [n :x]))))) 107 | 108 | ;;; 109 | 110 | (deftest ^:benchmark benchmark-reduce 111 | (let [s (take 1e6 (range 1e6))] 112 | (bench "reduce unchunked seq" 113 | (reduce max s))) 114 | (let [s (range 1e6)] 115 | (bench "reduce chunked seq" 116 | (reduce max s))) 117 | (let [s (vec (range 1e6))] 118 | (bench "reduce vector" 119 | (reduce max s))) 120 | (let [s (long-array 1e6)] 121 | (bench "reduce array" 122 | (reduce max s))) 123 | (let [s (c/marshal-seq s/int64 (range 1e6))] 124 | (bench "reduce int64 byte-seq" 125 | (reduce max s))) 126 | (let [s (c/marshal-seq s/int32 (range 1e6))] 127 | (bench "reduce int32 byte-seq" 128 | (reduce max s)))) 129 | 130 | (deftest ^:benchmark benchmark-marshalling 131 | (let [s (vec (range 10))] 132 | (bench "marshal 10 bytes" 133 | (c/marshal-seq s/int8 s))) 134 | (let [s (vec (range 100))] 135 | (bench "marshal 100 bytes" 136 | (c/marshal-seq s/int8 s))) 137 | (let [s (vec (range 1000))] 138 | (bench "marshal 1000 shorts" 139 | (c/marshal-seq s/int16 s))) 140 | (let [s (range 10)] 141 | (bench "lazily marshal 10 bytes" 142 | (c/marshal-seq s/int8 s))) 143 | (let [s (range 100)] 144 | (bench "lazily marshal 100 bytes" 145 | (c/lazily-marshal-seq s/int8 s))) 146 | (let [s (range 1000)] 147 | (bench "lazily marshal 1000 shorts" 148 | (c/lazily-marshal-seq s/int16 s)))) 149 | 150 | (deftest ^:benchmark benchmark-accessors 151 | (let [s (map 152 | #(hash-map :x %1 :y %2) 153 | (range 1) 154 | (repeat 1 (range 10))) 155 | ^:tuple ms (c/marshal-seq tuple s)] 156 | 157 | (batch-bench "mutable update nested structure" 158 | (c/update-in! ms [0 :y 0] p/inc)) 159 | (batch-bench "mutable get nested structure" 160 | (c/get-in ms [0 :y 0]) 161 | nil) 162 | (batch-bench "mutable set nested structure" 163 | (c/set-in! ms [0 :y 0] 0))) 164 | 165 | (let [s [{:x 0 :y (vec (range 10))}]] 166 | 167 | (batch-bench "persistent update nested structure" 168 | (update-in s [0 :y 0] inc)) 169 | (batch-bench "persistent get nested structure" 170 | (get-in s [0 :y 0] inc)) 171 | (batch-bench "persistent set nested structure" 172 | (assoc-in s [0 :y 0] 0)))) 173 | -------------------------------------------------------------------------------- /test/vertigo/test_utils.clj: -------------------------------------------------------------------------------- 1 | (ns vertigo.test-utils 2 | (:require [criterium.core :as c])) 3 | 4 | (defmacro long-bench [name & body] 5 | `(do 6 | (println "\n-----\n" ~name "\n-----\n") 7 | (c/bench 8 | (do ~@body nil)))) 9 | 10 | (defmacro bench [name & body] 11 | `(do 12 | (println "\n-----\n" ~name "\n-----\n") 13 | (c/quick-bench 14 | (do ~@body nil)))) 15 | 16 | (defmacro batch-bench [name & body] 17 | `(do 18 | (println "\n-----\n" ~name "\n-----\n") 19 | (c/quick-bench 20 | (dotimes [_# 1e6] ~@body)))) 21 | --------------------------------------------------------------------------------