├── .gitignore ├── .travis.yml ├── LICENSE ├── README.md ├── eastwood.clj ├── project.clj ├── src └── msgpack │ ├── clojure_extensions.clj │ ├── core.clj │ └── macros.clj └── test └── msgpack ├── core_check.clj └── core_test.clj /.gitignore: -------------------------------------------------------------------------------- 1 | /target 2 | /classes 3 | /checkouts 4 | pom.xml 5 | pom.xml.asc 6 | *.jar 7 | *.class 8 | /.lein-* 9 | /.nrepl-port 10 | /guide 11 | /TODO 12 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: clojure 2 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2014 Eugene D Ma 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in 13 | all copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 21 | THE SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # clojure-msgpack 2 | 3 | clojure-msgpack is a lightweight and simple library for converting 4 | between native Clojure data structures and MessagePack byte formats. 5 | clojure-msgpack only depends on Clojure itself; it has no third-party 6 | dependencies. 7 | 8 | ## Installation 9 | 10 | [![Clojars Project](http://clojars.org/clojure-msgpack/latest-version.svg)](https://clojars.org/clojure-msgpack) 11 | [![Build Status](https://travis-ci.org/edma2/clojure-msgpack.svg?branch=master)](https://travis-ci.org/edma2/clojure-msgpack) 12 | 13 | ## Usage 14 | 15 | ### Basic 16 | * ```pack```: Serialize object as a sequence of java.lang.Bytes. 17 | * ```unpack``` Deserialize bytes as a Clojure object. 18 | ```clojure 19 | (require '[msgpack.core :as msg]) 20 | (require 'msgpack.clojure-extensions) 21 | 22 | (msg/pack {:compact true :schema 0}) 23 | ; => # 24 | 25 | (msg/unpack (msg/pack {:compact true :schema 0})) 26 | ; => {:schema 0, :compact true} 27 | ````` 28 | 29 | ### Streaming 30 | `clojure-msgpack` provides a streaming API for situations where it is more 31 | convenient or efficient to work with byte streams instead of fixed byte arrays 32 | (e.g. size of object is not known ahead of time). 33 | 34 | The streaming counterpart to `msgpack.core/pack` is `msgpack.core/pack-stream` 35 | which returns nil and accepts either 36 | [java.io.OutputStream](http://docs.oracle.com/javase/7/docs/api/java/io/OutputStream.html) 37 | or 38 | [java.io.DataOutput](http://docs.oracle.com/javase/7/docs/api/java/io/DataOutput.html) 39 | as an additional argument. 40 | 41 | `msgpack.core/unpack` is in "streaming mode" when the argument is of type 42 | [java.io.DataInput](http://docs.oracle.com/javase/7/docs/api/java/io/DataInput.html) 43 | or 44 | [java.io.InputStream](http://docs.oracle.com/javase/7/docs/api/java/io/InputStream.html). 45 | 46 | ```clojure 47 | (use 'clojure.java.io) 48 | 49 | (with-open [s (output-stream "test.dat")] 50 | (msg/pack-stream {:compact true :schema 0} s)) 51 | 52 | (with-open [s (input-stream "test.dat")] (msg/unpack s)) 53 | ; => {:schema 0, :compact true} 54 | ``` 55 | 56 | ### Core types 57 | 58 | Clojure | MessagePack 59 | ----------------------------|------------ 60 | nil | Nil 61 | java.lang.Boolean | Boolean 62 | java.lang.Byte | Integer 63 | java.lang.Short | Integer 64 | java.lang.Integer | Integer 65 | java.lang.Long | Integer 66 | java.lang.BigInteger | Integer 67 | clojure.lang.BigInt | Integer 68 | java.lang.Float | Float 69 | java.lang.Double | Float 70 | java.math.BigDecimal | Float 71 | java.lang.String | String 72 | clojure.lang.Sequential | Array 73 | clojure.lang.IPersistentMap | Map 74 | msgpack.core.Ext | Extended 75 | 76 | Serializing a value of unrecognized type will fail with `IllegalArgumentException`. See [Application types](#application-types) if you want to register your own types. 77 | 78 | ### Clojure types 79 | Some native Clojure types don't have an obvious MessagePack counterpart. We can 80 | serialize them as Extended types. To enable automatic conversion of these 81 | types, load the `clojure-extensions` library. 82 | 83 | Clojure | MessagePack 84 | ----------------------------|------------ 85 | clojure.lang.Keyword | Extended (type = 3) 86 | clojure.lang.Symbol | Extended (type = 4) 87 | java.lang.Character | Extended (type = 5) 88 | clojure.lang.Ratio | Extended (type = 6) 89 | clojure.lang.IPersistentSet | Extended (type = 7) 90 | 91 | With `msgpack.clojure-extensions`: 92 | ```clojure 93 | (require 'msgpack.clojure-extensions) 94 | (msg/pack :hello) 95 | ; => # 96 | ``` 97 | 98 | Without `msgpack.clojure-extensions`: 99 | ```clojure 100 | (msg/pack :hello) 101 | ; => IllegalArgumentException No implementation of method: :pack-stream of 102 | ; protocol: #'msgpack.core/Packable found for class: clojure.lang.Keyword 103 | ; clojure.core/-cache-protocol-fn (core _deftype.clj:544) 104 | ``` 105 | 106 | ### Application types 107 | You can also define your own Extended types with `extend-msgpack`. 108 | 109 | ```clojure 110 | (require '[msgpack.macros :refer [extend-msgpack]]) 111 | 112 | (defrecord Person [name]) 113 | 114 | (extend-msgpack 115 | Person 116 | 100 117 | [p] (.getBytes (:name p)) 118 | [bytes] (->Person (String. bytes))) 119 | 120 | (msg/unpack (msg/pack [(->Person "bob") 5 "test"])) 121 | ; => (#user.Person{:name "bob"} 5 "test") 122 | ``` 123 | 124 | ### Options 125 | All pack and unpack functions take an optional map of options: 126 | * `:compatibility-mode` 127 | Serialize/deserialize strings and bytes using the raw-type defined here: 128 | https://github.com/msgpack/msgpack/blob/master/spec-old.md 129 | 130 | Note: No error is thrown if an unpacked value is reserved under the old spec 131 | but defined under the new spec. We always deserialize something if we can 132 | regardless of `compatibility-mode`. 133 | 134 | ```clojure 135 | (msg/pack (byte-array (byte 9)) {:compatibility-mode true}) 136 | ``` 137 | 138 | ## License 139 | clojure-msgpack is MIT licensed. See the included LICENSE file for more details. 140 | -------------------------------------------------------------------------------- /eastwood.clj: -------------------------------------------------------------------------------- 1 | (disable-warning 2 | {:linter :constant-test 3 | :if-inside-macroexpansion-of #{'msgpack.macros/extend-msgpack} 4 | :within-depth 4 5 | :reason "The `extend-msgpack` macro verifies the extension type is within bounds after macroexpansion."}) 6 | -------------------------------------------------------------------------------- /project.clj: -------------------------------------------------------------------------------- 1 | (defproject clojure-msgpack "1.2.1" 2 | :description "A lightweight Clojure implementation of the MessagePack spec." 3 | :url "https://github.com/edma2/clojure-msgpack" 4 | :license {:name "The MIT License (MIT)" 5 | :url "http://opensource.org/licenses/MIT"} 6 | :dependencies [[org.clojure/clojure "1.8.0"]] 7 | :global-vars {*warn-on-reflection* true} 8 | :scm {:name "git" 9 | :url "https://github.com/edma2/clojure-msgpack"} 10 | :profiles 11 | {:non-utf8-encoding 12 | {:jvm-opts 13 | ["-Dfile.encoding=ISO-8859-1"]} 14 | :eastwood {:plugins [[jonase/eastwood "0.2.3"]] 15 | :dependencies [[org.clojure/test.check "0.9.0"]] 16 | :eastwood {:config-files ["eastwood.clj"]}} 17 | :test {:dependencies [[org.clojure/test.check "0.9.0"]]}}) 18 | -------------------------------------------------------------------------------- /src/msgpack/clojure_extensions.clj: -------------------------------------------------------------------------------- 1 | (ns msgpack.clojure-extensions 2 | "Extended types for Clojure-specific types" 3 | (:require [msgpack.core :as msg] 4 | [msgpack.macros :refer [extend-msgpack]])) 5 | 6 | (defn- keyword->str 7 | "Convert keyword to string with namespace preserved. 8 | Example: :A/A => \"A/A\"" 9 | [k] 10 | (subs (str k) 1)) 11 | 12 | (extend-msgpack 13 | clojure.lang.Keyword 14 | 3 15 | [k] (msg/pack (keyword->str k)) 16 | [bytes] (keyword (msg/unpack bytes))) 17 | 18 | (extend-msgpack 19 | clojure.lang.Symbol 20 | 4 21 | [s] (msg/pack (str s)) 22 | [bytes] (symbol (msg/unpack bytes))) 23 | 24 | (extend-msgpack 25 | java.lang.Character 26 | 5 27 | [c] (msg/pack (str c)) 28 | [bytes] (first (char-array (msg/unpack bytes)))) 29 | 30 | (extend-msgpack 31 | clojure.lang.Ratio 32 | 6 33 | [r] (msg/pack [(numerator r) (denominator r)]) 34 | [bytes] (let [[n d] (msg/unpack bytes)] 35 | (/ n d))) 36 | 37 | (extend-msgpack 38 | clojure.lang.IPersistentSet 39 | 7 40 | [s] (msg/pack (seq s)) 41 | [bytes] (set (msg/unpack bytes))) 42 | -------------------------------------------------------------------------------- /src/msgpack/core.clj: -------------------------------------------------------------------------------- 1 | (ns msgpack.core 2 | (:import java.io.ByteArrayInputStream 3 | java.io.ByteArrayOutputStream 4 | java.io.DataInput 5 | java.io.DataInputStream 6 | java.io.DataOutput 7 | java.io.DataOutputStream 8 | java.io.InputStream 9 | java.io.OutputStream 10 | java.nio.charset.Charset)) 11 | 12 | (def ^:private ^Charset 13 | msgpack-charset 14 | (Charset/forName "UTF-8")) 15 | 16 | (defprotocol Packable 17 | "Objects that can be serialized as MessagePack types" 18 | (packable-pack [this data-output opts])) 19 | 20 | ;; MessagePack allows applications to define application-specific types using 21 | ;; the Extended type. Extended type consists of an integer and a byte array 22 | ;; where the integer represents a kind of types and the byte array represents 23 | ;; data. 24 | (defrecord Ext [type data]) 25 | 26 | (defmacro cond-let [bindings & clauses] 27 | `(let ~bindings (cond ~@clauses))) 28 | 29 | (defn- pack-raw 30 | [^bytes bytes ^DataOutput s] 31 | (cond-let [len (count bytes)] 32 | (<= len 0x1f) 33 | (do (.writeByte s (bit-or 2r10100000 len)) (.write s bytes)) 34 | 35 | (<= len 0xffff) 36 | (do (.writeByte s 0xda) (.writeShort s len) (.write s bytes)) 37 | 38 | (<= len 0xffffffff) 39 | (do (.writeByte s 0xdb) (.writeInt s len) (.write s bytes)))) 40 | 41 | (defn- pack-str 42 | [^bytes bytes ^DataOutput s] 43 | (cond-let [len (count bytes)] 44 | (<= len 0x1f) 45 | (do (.writeByte s (bit-or 2r10100000 len)) (.write s bytes)) 46 | 47 | (<= len 0xff) 48 | (do (.writeByte s 0xd9) (.writeByte s len) (.write s bytes)) 49 | 50 | (<= len 0xffff) 51 | (do (.writeByte s 0xda) (.writeShort s len) (.write s bytes)) 52 | 53 | (<= len 0xffffffff) 54 | (do (.writeByte s 0xdb) (.writeInt s len) (.write s bytes)))) 55 | 56 | (defn- pack-bytes 57 | [^bytes bytes ^DataOutput s] 58 | (cond-let [len (count bytes)] 59 | (<= len 0xff) 60 | (do (.writeByte s 0xc4) (.writeByte s len) (.write s bytes)) 61 | 62 | (<= len 0xffff) 63 | (do (.writeByte s 0xc5) (.writeShort s len) (.write s bytes)) 64 | 65 | (<= len 0xffffffff) 66 | (do (.writeByte s 0xc6) (.writeInt s len) (.write s bytes)))) 67 | 68 | (defn- pack-int 69 | "Pack integer using the most compact representation" 70 | [n ^DataOutput s] 71 | (cond 72 | ; +fixnum 73 | (<= 0 n 127) (.writeByte s n) 74 | ; -fixnum 75 | (<= -32 n -1) (.writeByte s n) 76 | ; uint 8 77 | (<= 0 n 0xff) (do (.writeByte s 0xcc) (.writeByte s n)) 78 | ; uint 16 79 | (<= 0 n 0xffff) (do (.writeByte s 0xcd) (.writeShort s n)) 80 | ; uint 32 81 | (<= 0 n 0xffffffff) (do (.writeByte s 0xce) (.writeInt s (unchecked-int n))) 82 | ; uint 64 83 | (<= 0 n 0xffffffffffffffff) (do (.writeByte s 0xcf) (.writeLong s (unchecked-long n))) 84 | ; int 8 85 | (<= -0x80 n -1) (do (.writeByte s 0xd0) (.writeByte s n)) 86 | ; int 16 87 | (<= -0x8000 n -1) (do (.writeByte s 0xd1) (.writeShort s n)) 88 | ; int 32 89 | (<= -0x80000000 n -1) (do (.writeByte s 0xd2) (.writeInt s n)) 90 | ; int 64 91 | (<= -0x8000000000000000 n -1) (do (.writeByte s 0xd3) (.writeLong s n)) 92 | :else (throw (IllegalArgumentException. (str "Integer value out of bounds: " n))))) 93 | 94 | (defn- pack-coll 95 | [coll ^DataOutput s opts] 96 | (doseq [item coll] (packable-pack item s opts))) 97 | 98 | (extend-protocol Packable 99 | nil 100 | (packable-pack 101 | [_ ^DataOutput s _] 102 | (.writeByte s 0xc0)) 103 | 104 | java.lang.Boolean 105 | (packable-pack 106 | [bool ^DataOutput s _] 107 | (if bool 108 | (.writeByte s 0xc3) 109 | (.writeByte s 0xc2))) 110 | 111 | java.lang.Byte 112 | (packable-pack [n ^DataOutput s _] (pack-int n s)) 113 | 114 | java.lang.Short 115 | (packable-pack [n ^DataOutput s _] (pack-int n s)) 116 | 117 | java.lang.Integer 118 | (packable-pack [n ^DataOutput s _] (pack-int n s)) 119 | 120 | java.lang.Long 121 | (packable-pack [n ^DataOutput s _] (pack-int n s)) 122 | 123 | java.math.BigInteger 124 | (packable-pack [n ^DataOutput s _] (pack-int n s)) 125 | 126 | clojure.lang.BigInt 127 | (packable-pack [n ^DataOutput s _] (pack-int n s)) 128 | 129 | java.lang.Float 130 | (packable-pack [f ^DataOutput s _] 131 | (do (.writeByte s 0xca) (.writeFloat s f))) 132 | 133 | java.lang.Double 134 | (packable-pack [d ^DataOutput s _] 135 | (do (.writeByte s 0xcb) (.writeDouble s d))) 136 | 137 | java.math.BigDecimal 138 | (packable-pack [d ^DataOutput s opts] 139 | (packable-pack (.doubleValue d) s opts)) 140 | 141 | java.lang.String 142 | (packable-pack 143 | [str ^DataOutput s {:keys [compatibility-mode]}] 144 | (let [bytes (.getBytes ^String str msgpack-charset)] 145 | (if compatibility-mode 146 | (pack-raw bytes s) 147 | (pack-str bytes s)))) 148 | 149 | Ext 150 | (packable-pack 151 | [e ^DataOutput s _] 152 | (let [type (:type e) 153 | ^bytes data (:data e) 154 | len (count data)] 155 | (do 156 | (cond 157 | (= len 1) (.writeByte s 0xd4) 158 | (= len 2) (.writeByte s 0xd5) 159 | (= len 4) (.writeByte s 0xd6) 160 | (= len 8) (.writeByte s 0xd7) 161 | (= len 16) (.writeByte s 0xd8) 162 | (<= len 0xff) (do (.writeByte s 0xc7) (.writeByte s len)) 163 | (<= len 0xffff) (do (.writeByte s 0xc8) (.writeShort s len)) 164 | (<= len 0xffffffff) (do (.writeByte s 0xc9) (.writeInt s len))) 165 | (.writeByte s type) 166 | (.write s data)))) 167 | 168 | clojure.lang.Sequential 169 | (packable-pack [seq ^DataOutput s opts] 170 | (cond-let [len (count seq)] 171 | (<= len 0xf) 172 | (do (.writeByte s (bit-or 2r10010000 len)) (pack-coll seq s opts)) 173 | 174 | (<= len 0xffff) 175 | (do (.writeByte s 0xdc) (.writeShort s len) (pack-coll seq s opts)) 176 | 177 | (<= len 0xffffffff) 178 | (do (.writeByte s 0xdd) (.writeInt s len) (pack-coll seq s opts)))) 179 | 180 | clojure.lang.IPersistentMap 181 | (packable-pack [map ^DataOutput s opts] 182 | (cond-let [len (count map) 183 | pairs (interleave (keys map) (vals map))] 184 | (<= len 0xf) 185 | (do (.writeByte s (bit-or 2r10000000 len)) (pack-coll pairs s opts)) 186 | 187 | (<= len 0xffff) 188 | (do (.writeByte s 0xde) (.writeShort s len) (pack-coll pairs s opts)) 189 | 190 | (<= len 0xffffffff) 191 | (do (.writeByte s 0xdf) (.writeInt s len) (pack-coll pairs s opts))))) 192 | 193 | ; Note: the extensions below are not in extend-protocol above because of 194 | ; a Clojure bug. See http://dev.clojure.org/jira/browse/CLJ-1381 195 | 196 | (def ^:private class-of-byte-array 197 | (class (java.lang.reflect.Array/newInstance Byte 0))) 198 | 199 | (def ^:private class-of-primitive-byte-array 200 | (Class/forName "[B")) 201 | 202 | ; Array of java.lang.Byte (boxed) 203 | (extend class-of-byte-array 204 | Packable 205 | {:packable-pack 206 | (fn [a ^DataOutput s opts] 207 | (packable-pack (byte-array a) s opts))}) 208 | 209 | (extend class-of-primitive-byte-array 210 | Packable 211 | {:packable-pack 212 | (fn [bytes ^DataOutput s {:keys [compatibility-mode]}] 213 | (if compatibility-mode 214 | (pack-raw bytes s) 215 | (pack-bytes bytes s)))}) 216 | 217 | (defn pack-stream 218 | "Serialize application value as a stream of MessagePack-formatted bytes. 219 | Argument type can be either java.io.DataOutput or java.io.OutputStream. Any 220 | other type is an error." 221 | ([this stream] (pack-stream this stream nil)) 222 | ([this stream opts] 223 | (condp instance? stream 224 | DataOutput (packable-pack this stream opts) 225 | OutputStream (pack-stream this (DataOutputStream. stream) opts)))) 226 | 227 | (defn pack 228 | "Serialize application value as a MessagePack-formatted byte array" 229 | ([obj] (pack obj nil)) 230 | ([obj opts] 231 | (let [stream (ByteArrayOutputStream.)] 232 | (pack-stream obj stream opts) 233 | (.toByteArray stream)))) 234 | 235 | (defn- read-uint8 236 | [^DataInput data-input] 237 | (.readUnsignedByte data-input)) 238 | 239 | (defn- read-uint16 240 | [^DataInput data-input] 241 | (.readUnsignedShort data-input)) 242 | 243 | (defn- read-uint32 244 | [^DataInput data-input] 245 | (bit-and 0xffffffff (.readInt data-input))) 246 | 247 | (defn- read-uint64 248 | [^DataInput data-input] 249 | (let [n (.readLong data-input)] 250 | (if (<= 0 n Long/MAX_VALUE) 251 | n 252 | (.and (biginteger n) (biginteger 0xffffffffffffffff))))) 253 | 254 | (defn- read-bytes 255 | [n ^DataInput data-input] 256 | (let [bytes (byte-array n)] 257 | (do 258 | (.readFully data-input bytes) 259 | bytes))) 260 | 261 | (defn- read-str 262 | [n ^DataInput data-input {:keys [compatibility-mode]}] 263 | (let [bytes (read-bytes n data-input)] 264 | (if compatibility-mode bytes 265 | (String. ^bytes bytes msgpack-charset)))) 266 | 267 | (defmulti refine-ext 268 | "Refine Extended type to an application-specific type." 269 | :type) 270 | 271 | (defmethod refine-ext :default [ext] ext) 272 | 273 | (declare unpack-stream) 274 | 275 | (defn- unpack-ext [n ^DataInput data-input] 276 | (refine-ext 277 | (->Ext (.readByte data-input) (read-bytes n data-input)))) 278 | 279 | (defn- unpack-n [n ^DataInput data-input opts] 280 | (loop [i 0 281 | v (transient [])] 282 | (if (< i n) 283 | (recur 284 | (unchecked-inc i) 285 | (conj! v (unpack-stream data-input opts))) 286 | (persistent! v)))) 287 | 288 | (defn- unpack-map [n ^DataInput data-input opts] 289 | (loop [i 0 290 | m (transient {})] 291 | (if (< i n) 292 | (recur 293 | (unchecked-inc i) 294 | (assoc! m 295 | (unpack-stream data-input opts) 296 | (unpack-stream data-input opts))) 297 | (persistent! m)))) 298 | 299 | (defn unpack-stream 300 | ([^DataInput data-input] (unpack-stream data-input nil)) 301 | ([^DataInput data-input opts] 302 | (cond-let [byte (.readUnsignedByte data-input)] 303 | ; nil format family 304 | (= byte 0xc0) nil 305 | 306 | ; bool format family 307 | (= byte 0xc2) false 308 | (= byte 0xc3) true 309 | 310 | ; int format family 311 | (= (bit-and 2r11100000 byte) 2r11100000) 312 | (unchecked-byte byte) 313 | 314 | (= (bit-and 2r10000000 byte) 0) 315 | (unchecked-byte byte) 316 | 317 | (= byte 0xcc) (read-uint8 data-input) 318 | (= byte 0xcd) (read-uint16 data-input) 319 | (= byte 0xce) (read-uint32 data-input) 320 | (= byte 0xcf) (read-uint64 data-input) 321 | (= byte 0xd0) (.readByte data-input) 322 | (= byte 0xd1) (.readShort data-input) 323 | (= byte 0xd2) (.readInt data-input) 324 | (= byte 0xd3) (.readLong data-input) 325 | 326 | ; float format family 327 | (= byte 0xca) (.readFloat data-input) 328 | (= byte 0xcb) (.readDouble data-input) 329 | 330 | ; str format family 331 | (= (bit-and 2r11100000 byte) 2r10100000) 332 | (let [n (bit-and 2r11111 byte)] 333 | (read-str n data-input opts)) 334 | 335 | (= byte 0xd9) 336 | (read-str (read-uint8 data-input) data-input opts) 337 | 338 | (= byte 0xda) 339 | (read-str (read-uint16 data-input) data-input opts) 340 | 341 | (= byte 0xdb) 342 | (read-str (read-uint32 data-input) data-input opts) 343 | 344 | ; bin format family 345 | (= byte 0xc4) 346 | (read-bytes (read-uint8 data-input) data-input) 347 | 348 | (= byte 0xc5) 349 | (read-bytes (read-uint16 data-input) data-input) 350 | 351 | (= byte 0xc6) 352 | (read-bytes (read-uint32 data-input) data-input) 353 | 354 | ; ext format family 355 | (= byte 0xd4) (unpack-ext 1 data-input) 356 | (= byte 0xd5) (unpack-ext 2 data-input) 357 | (= byte 0xd6) (unpack-ext 4 data-input) 358 | (= byte 0xd7) (unpack-ext 8 data-input) 359 | (= byte 0xd8) (unpack-ext 16 data-input) 360 | 361 | (= byte 0xc7) 362 | (unpack-ext (read-uint8 data-input) data-input) 363 | 364 | (= byte 0xc8) 365 | (unpack-ext (read-uint16 data-input) data-input) 366 | 367 | (= byte 0xc9) 368 | (unpack-ext (read-uint32 data-input) data-input) 369 | 370 | ; array format family 371 | (= (bit-and 2r11110000 byte) 2r10010000) 372 | (unpack-n (bit-and 2r1111 byte) data-input opts) 373 | 374 | (= byte 0xdc) 375 | (unpack-n (read-uint16 data-input) data-input opts) 376 | 377 | (= byte 0xdd) 378 | (unpack-n (read-uint32 data-input) data-input opts) 379 | 380 | ; map format family 381 | (= (bit-and 2r11110000 byte) 2r10000000) 382 | (unpack-map (bit-and 2r1111 byte) data-input opts) 383 | 384 | (= byte 0xde) 385 | (unpack-map (read-uint16 data-input) data-input opts) 386 | 387 | (= byte 0xdf) 388 | (unpack-map (read-uint32 data-input) data-input opts)))) 389 | 390 | (defn unpack 391 | "Deserialize MessagePack-formatted bytes to an application value. Argument 392 | type can be either java.io.DataInput, java.io.InputStream, or byte array. 393 | Other types are coerced to a byte array." 394 | ([obj] (unpack obj nil)) 395 | ([obj opts] 396 | (condp instance? obj 397 | DataInput (unpack-stream obj opts) 398 | InputStream (unpack (DataInputStream. obj) opts) 399 | class-of-primitive-byte-array (unpack (ByteArrayInputStream. obj) opts) 400 | (unpack (byte-array obj) opts)))) 401 | -------------------------------------------------------------------------------- /src/msgpack/macros.clj: -------------------------------------------------------------------------------- 1 | (ns msgpack.macros 2 | "Macros for extending MessagePack with Extended types. 3 | See msgpack.clojure-extensions for examples." 4 | (:require [msgpack.core :refer :all])) 5 | 6 | (defmacro extend-msgpack 7 | [class type pack-args pack unpack-args unpack] 8 | `(let [type# ~type] 9 | (assert (<= 0 type# 127) 10 | "[-1, -128]: reserved for future pre-defined extensions.") 11 | (do 12 | (extend-protocol Packable ~class 13 | (packable-pack [~@pack-args ^java.io.DataOutput s# opts#] 14 | (packable-pack (->Ext type# ~pack) s# opts#))) 15 | (defmethod refine-ext type# [ext#] 16 | (let [~@unpack-args (:data ext#)] 17 | ~unpack))))) 18 | -------------------------------------------------------------------------------- /test/msgpack/core_check.clj: -------------------------------------------------------------------------------- 1 | (ns msgpack.core-check 2 | (:require [msgpack.core :as msg] 3 | [clojure.test.check.clojure-test :refer [defspec]] 4 | [clojure.test.check.generators :as gen] 5 | [clojure.test.check.properties :as prop])) 6 | 7 | ; NaN is never equal to itself 8 | (defn- not-nan [x] 9 | (not (Double/isNaN x))) 10 | 11 | (defn- pack-and-unpack [x] 12 | (msg/unpack (msg/pack x))) 13 | 14 | (defspec ints-round-trip 100 15 | (prop/for-all [x gen/int] 16 | (= (pack-and-unpack x) x))) 17 | 18 | (defspec floats-round-trip 100 19 | (prop/for-all [x (gen/such-that not-nan gen/double)] 20 | (= (pack-and-unpack x) x))) 21 | 22 | (defspec bytes-round-trip 100 23 | (prop/for-all [x gen/bytes] 24 | (let [bytes (pack-and-unpack x)] 25 | (and (instance? (Class/forName "[B") bytes) 26 | (= (seq bytes) (seq x)))))) 27 | 28 | (defn- box [bytes] 29 | (let [a (java.lang.reflect.Array/newInstance Byte (count bytes))] 30 | (doseq [[i b] (map-indexed vector (seq bytes))] 31 | (java.lang.reflect.Array/set a i b)) 32 | a)) 33 | 34 | (defspec boxed-bytes-round-trip 100 35 | (prop/for-all [x (gen/fmap box gen/bytes)] 36 | (let [bytes (pack-and-unpack x)] 37 | (and (instance? (Class/forName "[B") bytes) 38 | (= (seq bytes) (seq x)))))) 39 | 40 | (defspec strings-round-trip 100 41 | (prop/for-all [x gen/string] 42 | (= (pack-and-unpack x) x))) 43 | 44 | (def ^:private vector-of-maps (gen/vector (gen/map gen/int gen/string))) 45 | 46 | (defspec vectors-round-trip 20 47 | (prop/for-all [x vector-of-maps] 48 | (= (pack-and-unpack x) x))) 49 | 50 | (defspec maps-round-trip 20 51 | (prop/for-all [x (gen/map gen/string vector-of-maps)] 52 | (= (pack-and-unpack x) x))) 53 | -------------------------------------------------------------------------------- /test/msgpack/core_test.clj: -------------------------------------------------------------------------------- 1 | (ns msgpack.core-test 2 | (:require [clojure.test :refer :all] 3 | [clojure.walk :refer [postwalk]] 4 | [msgpack.core :as msg] 5 | msgpack.clojure-extensions)) 6 | 7 | (defn- byte-array? [v] 8 | (instance? (Class/forName "[B") v)) 9 | 10 | (defn- bigdecimal? [v] 11 | (instance? java.math.BigDecimal v)) 12 | 13 | (defn- normalize 14 | "Equality is not defined for Java arrays. Instead convert them into sequences 15 | and compare them that way." 16 | [v] 17 | (postwalk 18 | (fn [v] 19 | (cond 20 | (byte-array? v) (seq v) 21 | (bigdecimal? v) (double v) ;; (not (= 0.0M 0.0)) 22 | :else v)) 23 | v)) 24 | 25 | (defn- unsigned-bytes 26 | [bytes] 27 | (map unchecked-byte bytes)) 28 | 29 | (defn- unsigned-byte-array 30 | [bytes] 31 | (byte-array (unsigned-bytes bytes))) 32 | 33 | (defn- fill-string [n c] 34 | (apply str (repeat n c))) 35 | 36 | (defn- ext [type bytes] 37 | (msg/->Ext type (unsigned-byte-array bytes))) 38 | 39 | (defmacro round-trip [obj expected-bytes] 40 | `(let [obj# ~obj 41 | expected-bytes# (unsigned-bytes ~expected-bytes)] 42 | (is (= expected-bytes# (seq (msg/pack obj#)))) 43 | (is (= (normalize obj#) (normalize (msg/unpack expected-bytes#)))))) 44 | 45 | (defmacro round-trip-raw [obj packed-bytes unpacked-obj] 46 | `(let [obj# ~obj 47 | packed-bytes# (unsigned-bytes ~packed-bytes) 48 | unpacked-obj# ~unpacked-obj] 49 | (is (= packed-bytes# (seq (msg/pack obj# {:compatibility-mode true})))) 50 | (is (= (normalize unpacked-obj#) (normalize (msg/unpack packed-bytes# {:compatibility-mode true})))))) 51 | 52 | (deftest nil-test 53 | (testing "nil" 54 | (round-trip nil [0xc0]))) 55 | 56 | (deftest boolean-test 57 | (testing "booleans" 58 | (round-trip false [0xc2]) 59 | (round-trip true [0xc3]))) 60 | 61 | (deftest int-test 62 | (testing "positive fixnum" 63 | (round-trip (biginteger 0) [0x00]) 64 | (round-trip 0 [0x00]) 65 | (round-trip 0x10 [0x10]) 66 | (round-trip 0x7f [0x7f])) 67 | (testing "negative fixnum" 68 | (round-trip -1 [0xff]) 69 | (round-trip -16 [0xf0]) 70 | (round-trip -32 [0xe0])) 71 | (testing "uint 8" 72 | (round-trip 0x80 [0xcc 0x80]) 73 | (round-trip 0xf0 [0xcc 0xf0]) 74 | (round-trip 0xff [0xcc 0xff])) 75 | (testing "uint 16" 76 | (round-trip 0x100 [0xcd 0x01 0x00]) 77 | (round-trip 0x2000 [0xcd 0x20 0x00]) 78 | (round-trip 0xffff [0xcd 0xff 0xff])) 79 | (testing "uint 32" 80 | (round-trip 0x10000 [0xce 0x00 0x01 0x00 0x00]) 81 | (round-trip 0x200000 [0xce 0x00 0x20 0x00 0x00]) 82 | (round-trip 0xffffffff [0xce 0xff 0xff 0xff 0xff])) 83 | (testing "uint 64" 84 | (round-trip 0x100000000 [0xcf 0x00 0x00 0x00 0x01 0x00 0x00 0x00 0x00]) 85 | (round-trip 0x200000000000 [0xcf 0x00 0x00 0x20 0x00 0x00 0x00 0x00 0x00]) 86 | (round-trip 0xffffffffffffffff [0xcf 0xff 0xff 0xff 0xff 0xff 0xff 0xff 0xff])) 87 | (testing "int 8" 88 | (round-trip -33 [0xd0 0xdf]) 89 | (round-trip -100 [0xd0 0x9c]) 90 | (round-trip -128 [0xd0 0x80])) 91 | (testing "int 16" 92 | (round-trip -129 [0xd1 0xff 0x7f]) 93 | (round-trip -2000 [0xd1 0xf8 0x30]) 94 | (round-trip -32768 [0xd1 0x80 0x00])) 95 | (testing "int 32" 96 | (round-trip -32769 [0xd2 0xff 0xff 0x7f 0xff]) 97 | (round-trip -1000000000 [0xd2 0xc4 0x65 0x36 0x00]) 98 | (round-trip -2147483648 [0xd2 0x80 0x00 0x00 0x00])) 99 | (testing "int 64" 100 | (round-trip -2147483649 [0xd3 0xff 0xff 0xff 0xff 0x7f 0xff 0xff 0xff]) 101 | (round-trip -1000000000000000002 [0xd3 0xf2 0x1f 0x49 0x4c 0x58 0x9b 0xff 0xfe]) 102 | (round-trip -9223372036854775808 [0xd3 0x80 0x00 0x00 0x00 0x00 0x00 0x00 0x00])) 103 | (testing "throws IllegalArgumentException for out of bounds integers" 104 | (is (thrown? IllegalArgumentException (msg/pack -0x8000000000000001))) 105 | (is (thrown? IllegalArgumentException (msg/pack 0x10000000000000000))))) 106 | 107 | (deftest float-test 108 | (testing "float 32" 109 | (round-trip (float 0.0) [0xca 0x00 0x00 0x00 0x00]) 110 | (round-trip (float 2.5) [0xca 0x40 0x20 0x00 0x00])) 111 | (testing "float 64" 112 | (round-trip (double 0.0) [0xcb 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00]) 113 | (round-trip (double 2.5) [0xcb 0x40 0x04 0x00 0x00 0x00 0x00 0x00 0x00]) 114 | (round-trip (double 3) [0xcb 0x40 0x8 0x0 0x0 0x0 0x0 0x0 0x0]) 115 | (round-trip 1e-46 [0xcb 0x36 0x62 0x44 0xce 0x24 0x2c 0x55 0x61]) 116 | (round-trip 1e39 [0xcb 0x48 0x7 0x82 0x87 0xf4 0x9c 0x4a 0x1d]) 117 | (round-trip (bigdec 3) [0xcb 0x40 0x8 0x0 0x0 0x0 0x0 0x0 0x0]))) 118 | 119 | (deftest str-test 120 | (testing "fixstr" 121 | (round-trip "hello world" [0xab 0x68 0x65 0x6c 0x6c 0x6f 0x20 0x77 0x6f 0x72 0x6c 0x64]) 122 | (round-trip "" [0xa0]) 123 | (round-trip "abc" [0xa3 0x61 0x62 0x63]) 124 | (round-trip "äöü" [0xa6 0xc3 0xa4 0xc3 0xb6 0xc3 0xbc]) 125 | (round-trip (fill-string 31 \a) (cons 0xbf (repeat 31 0x61)))) 126 | (testing "str 8" 127 | (round-trip (fill-string 32 \b) 128 | (concat [0xd9 0x20] (repeat 32 (byte \b)))) 129 | (round-trip-raw 130 | (fill-string 32 \b) 131 | (concat [0xda 0x00 0x20] (repeat 32 (byte \b))) 132 | (unsigned-byte-array (repeat 32 (byte \b)))) 133 | (round-trip (fill-string 100 \c) 134 | (concat [0xd9 0x64] (repeat 100 (byte \c)))) 135 | (round-trip (fill-string 255 \d) 136 | (concat [0xd9 0xff] (repeat 255 (byte \d))))) 137 | (testing "str 16" 138 | (round-trip (fill-string 256 \b) 139 | (concat [0xda 0x01 0x00] (repeat 256 (byte \b)))) 140 | (round-trip (fill-string 65535 \c) 141 | (concat [0xda 0xff 0xff] (repeat 65535 (byte \c))))) 142 | (testing "str 32" 143 | (round-trip (fill-string 65536 \b) 144 | (concat [0xdb 0x00 0x01 0x00 0x00] (repeat 65536 (byte \b)))))) 145 | 146 | (deftest bin-test 147 | (testing "bin 8" 148 | (round-trip (byte-array 0) [0xc4 0x00]) 149 | (round-trip-raw (byte-array 0) [0xa0] nil) 150 | (round-trip (unsigned-byte-array [0x80]) [0xc4 0x01 0x80]) 151 | (round-trip (unsigned-byte-array (repeat 32 0x80)) (concat [0xc4 0x20] (repeat 32 0x80))) 152 | (round-trip (unsigned-byte-array (repeat 255 0x80)) (concat [0xc4 0xff] (repeat 255 0x80)))) 153 | (testing "bin 16" 154 | (round-trip (unsigned-byte-array (repeat 256 0x80)) (concat [0xc5 0x01 0x00] (repeat 256 0x80))) 155 | (round-trip-raw 156 | (unsigned-byte-array (repeat 256 0x80)) 157 | (concat [0xda 0x01 0x00] (repeat 256 0x80)) 158 | (unsigned-byte-array (repeat 256 0x80)))) 159 | (testing "bin 32" 160 | (round-trip (unsigned-byte-array (repeat 65536 0x80)) 161 | (concat [0xc6 0x00 0x01 0x00 0x00] (repeat 65536 0x80))))) 162 | 163 | (deftest ext-test 164 | (testing "fixext 1" 165 | (round-trip (ext 100 [0x80]) [0xd4 0x64 0x80])) 166 | (testing "fixext 2" 167 | (round-trip (ext 100 [0x80 0x80]) [0xd5 0x64 0x80 0x80])) 168 | (testing "fixext 4" 169 | (round-trip (ext 100 [0x80 0x80 0x80 0x80]) 170 | [0xd6 0x64 0x80 0x80 0x80 0x80])) 171 | (testing "fixext 8" 172 | (round-trip (ext 100 [0x80 0x80 0x80 0x80 0x80 0x80 0x80 0x80]) 173 | [0xd7 0x64 0x80 0x80 0x80 0x80 0x80 0x80 0x80 0x80])) 174 | (testing "fixext 16" 175 | (round-trip (ext 100 (repeat 16 0x80)) 176 | (concat [0xd8 0x64] (repeat 16 0x80)))) 177 | (testing "ext 8" 178 | (round-trip (ext 100 [0x1 0x3 0x11]) 179 | [0xc7 0x3 0x64 0x1 0x3 0x11]) 180 | (round-trip (ext 100 (repeat 255 0x80)) 181 | (concat [0xc7 0xff 0x64] (repeat 255 0x80)))) 182 | (testing "ext 16" 183 | (round-trip (ext 100 (repeat 256 0x80)) 184 | (concat [0xc8 0x01 0x00 0x64] (repeat 256 0x80)))) 185 | (testing "ext 32" 186 | (round-trip (ext 100 (repeat 65536 0x80)) 187 | (concat [0xc9 0x00 0x01 0x00 0x00 0x64] (repeat 65536 0x80))))) 188 | 189 | (deftest clojure-test 190 | (testing "clojure.lang.Keyword" 191 | (round-trip :abc [0xd6 0x3 0xa3 0x61 0x62 0x63])) 192 | (testing "clojure.lang.Symbol" 193 | (round-trip 'abc [0xd6 0x4 0xa3 0x61 0x62 0x63])) 194 | (testing "java.lang.Character" 195 | (round-trip \c [0xd5 0x5 0xa1 0x63])) 196 | (testing "clojure.lang.Ratio" 197 | (round-trip 5/2 [0xc7 0x3 0x6 0x92 0x5 0x2])) 198 | (testing "clojure.lang.IPersistentSet" 199 | (round-trip #{} [0xd4 0x7 0xc0]))) 200 | 201 | (deftest array-test 202 | (testing "fixarray" 203 | (round-trip '() [0x90]) 204 | (round-trip [] [0x90]) 205 | (round-trip [[]] [0x91 0x90]) 206 | (round-trip [5 "abc" true] [0x93 0x05 0xa3 0x61 0x62 0x63 0xc3]) 207 | (round-trip-raw [5 "abc" true] [0x93 0x05 0xa3 0x61 0x62 0x63 0xc3] [5 (.getBytes "abc") true]) 208 | (round-trip [true 1 (msg/->Ext 100 (.getBytes "foo")) 0xff {1 false 2 "abc"} (unsigned-byte-array [0x80]) [1 2 3] "abc"] 209 | [0x98 0xc3 0x1 0xc7 0x3 0x64 0x66 0x6f 0x6f 0xcc 0xff 0x82 0x1 0xc2 0x2 0xa3 0x61 0x62 0x63 0xc4 0x1 0x80 0x93 0x1 0x2 0x3 0xa3 0x61 0x62 0x63])) 210 | (testing "array 16" 211 | (round-trip (repeat 16 5) 212 | (concat [0xdc 0x00 0x10] (repeat 16 5))) 213 | (round-trip (repeat 65535 5) 214 | (concat [0xdc 0xff 0xff] (repeat 65535 5)))) 215 | (testing "array 32" 216 | (round-trip (repeat 65536 5) 217 | (concat [0xdd 0x00 0x01 0x00 0x00] (repeat 65536 5))))) 218 | (deftest map-test 219 | (testing "fixmap" 220 | (round-trip {} [0x80]) 221 | (round-trip-raw {(byte-array 0) [{"key" (byte-array 0)}]} 222 | [0x81 0xa0 0x91 0x81 0xa3 0x6b 0x65 0x79 0xa0] 223 | {nil [{(.getBytes "key") nil}]}) 224 | (round-trip {1 true 2 "abc" 3 (unsigned-byte-array [0x80])} 225 | [0x83 0x01 0xc3 0x02 0xa3 0x61 0x62 0x63 0x03 0xc4 0x01 0x80]) 226 | (round-trip {"abc" 5} [0x81 0xa3 0x61 0x62 0x63 0x05]) 227 | (round-trip {(unsigned-byte-array [0x80]) 0xffff} 228 | [0x81 0xc4 0x01 0x80 0xcd 0xff 0xff]) 229 | (round-trip {true nil} [0x81 0xc3 0xc0]) 230 | (round-trip {"compact" true "schema" 0} 231 | [0x82 0xa7 0x63 0x6f 0x6d 0x70 0x61 0x63 0x74 0xc3 0xa6 0x73 0x63 0x68 0x65 0x6d 0x61 0x00]) 232 | (round-trip {1 [{1 2 3 4} {}], 2 1, 3 [false "def"], 4 {0x100000000 "a" 0xffffffff "b"}} 233 | [0x84 0x01 0x92 0x82 0x01 0x02 0x03 0x04 0x80 0x02 0x01 0x03 0x92 0xc2 0xa3 0x64 0x65 0x66 0x04 0x82 0xcf 0x00 0x00 0x00 0x01 0x00 0x00 0x00 0x00 0xa1 0x61 0xce 0xff 0xff 0xff 0xff 0xa1 0x62])) 234 | (testing "map 16" 235 | (round-trip (zipmap (range 0 16) (repeat 16 5)) 236 | [0xde 0x0 0x10 0x0 0x5 0x7 0x5 0x1 0x5 0x4 0x5 0xf 0x5 0xd 0x5 0x6 0x5 0x3 0x5 0xc 0x5 0x2 0x5 0xb 0x5 0x9 0x5 0x5 0x5 0xe 0x5 0xa 0x5 0x8 0x5]))) 237 | --------------------------------------------------------------------------------