├── .github └── workflows │ └── ci.yml ├── .gitignore ├── README.md ├── config.nims ├── nim.cfg ├── serialization.nim ├── serialization.nimble ├── serialization ├── errors.nim ├── formats.nim ├── object_serialization.nim └── testing │ ├── generic_suite.nim │ └── tracing.nim └── tests ├── nim.cfg ├── test_all.nim ├── test_object_serialization.nim └── test_reader.nim /.github/workflows/ci.yml: -------------------------------------------------------------------------------- 1 | name: CI 2 | on: 3 | push: 4 | branches: 5 | - master 6 | pull_request: 7 | workflow_dispatch: 8 | 9 | jobs: 10 | build: 11 | uses: status-im/nimbus-common-workflow/.github/workflows/common.yml@main 12 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | nimcache/ 2 | tests/test_all 3 | nimble.develop 4 | nimble.paths 5 | build/ 6 | vendor/ 7 | 8 | *.exe 9 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | nim-serialization 2 | ================= 3 | 4 | [![License: Apache](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) 5 | [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) 6 | ![Github action](https://github.com/status-im/nim-serialization/workflows/CI/badge.svg) 7 | 8 | ## Introduction 9 | 10 | The `serialization` package aims to provide a common generic and efficient 11 | interface for marshaling Nim values to and from various serialized formats. 12 | Individual formats are implemented in separated packages such as 13 | [`json_serialization`](https://github.com/status-im/nim-json-serialization) 14 | while this package provides the common interfaces shared between all of them 15 | and the means to customize your Nim types for the purposes of serialization. 16 | 17 | The internal mechanisms of the library allow for implementing the required 18 | marshaling logic in highly efficient way that goes from bytes to Nim values 19 | and vice versa without allocating any intermediate structures. 20 | 21 | ## Defining serialization formats 22 | 23 | A serialization format is implemented through defining a `Reader` and `Writer` 24 | type for the format and then by providing the following type declaration: 25 | 26 | ```nim 27 | serializationFormat Json, # This is the name of the format. 28 | # Most APIs provided by the library will accept 29 | # this identifier as a required parameter. 30 | mimeType = "application/json" # Mime type associated with the format (Optional). 31 | 32 | Json.setReader JsonReader # The associated Reader type. 33 | Json.setWriter JsonWriter, # The associated Writer type. 34 | PreferredOutput = string # APIs such as `Json.encode` will return this type. 35 | ``` 36 | 37 | ## Common API 38 | 39 | Most of the time, you'll be using the following high-level APIs when encoding 40 | and decoding values: 41 | 42 | #### `Format.encode(value: auto, params: varargs): Format.PreferredOutput` 43 | 44 | Encodes a value in the specified format returning the preferred output type 45 | for the format (usually `string` or `seq[byte]`). All extra params will be 46 | forwarded without modification to the constructor of the used `Writer` type. 47 | 48 | Example: 49 | 50 | ```nim 51 | assert Json.encode(@[1, 2, 3], pretty = false) == "[1, 2, 3]" 52 | ``` 53 | 54 | #### `Format.decode(input: openArray[byte]|string, RecordType: type, params: varargs): RecordType` 55 | 56 | Decodes and returns a value of the specified `RecordType`. All params will 57 | be forwarded without modification to the used `Reader` type. A Format-specific 58 | descendant of `SerializationError` may be thrown in case of error. 59 | 60 | #### `Format.saveFile(filename: string, value: auto, params: varargs)` 61 | 62 | Similar to `encode`, but saves the result in a file. 63 | 64 | #### `Format.loadFile(filename: string, RecordType: type, params: varargs): RecordType` 65 | 66 | Similar to `decode`, but treats the contents of a file as an input. 67 | 68 | #### `reader.readValue(RecordType: type): RecordType` 69 | 70 | Reads a single value of the designated type from the stream associated with a 71 | particular reader. 72 | 73 | #### `writer.writeValue(value: auto)` 74 | 75 | Encodes a single value and writes it to the output stream of a particular writer. 76 | 77 | ### Custom serialization of user-defined types 78 | 79 | By default, record types will have all of their fields serialized. You can 80 | alter this behavior by attaching the `dontSerialize` pragma to exclude fields. 81 | The pragma `serializedFieldName(name: string)` can be used to modify the name 82 | of the field in formats such as Json and XML. 83 | 84 | Alternatively, if you are not able to modify the definition of a particular 85 | Nim type, you can use the `setSerializedFields` macro to achieve the same 86 | in a less intrusive way. 87 | 88 | The following two definitions can be considered equivalent: 89 | 90 | ```nim 91 | type 92 | Foo = object 93 | a: string 94 | b {.dontSerialize.}: int 95 | 96 | setSerializedFields Foo: 97 | a 98 | ``` 99 | 100 | As you can see, `setSerializedFields` accepts a block where each serialized 101 | field is listed on a separate line. 102 | 103 | #### `customSerialization(RecordType: type, spec)` 104 | 105 | 106 | 107 | 108 | #### `totalSerializedFields(RecordType: type)` 109 | 110 | Returns the number of serialized fields in the specified format. 111 | 112 | ### Implementing Readers 113 | 114 | ### Implementing Writers 115 | 116 | ## Contributing 117 | 118 | When submitting pull requests, please add test cases for any new features 119 | or fixes and make sure `nimble test` is still able to execute the entire 120 | test suite successfully. 121 | 122 | [BOUNTIES]: https://github.com/status-im/nim-confutils/issues?q=is%3Aissue+is%3Aopen+label%3Abounty 123 | 124 | ## License 125 | 126 | Licensed and distributed under either of 127 | 128 | * MIT license: [LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT 129 | 130 | or 131 | 132 | * Apache License, Version 2.0, ([LICENSE-APACHEv2](LICENSE-APACHEv2) or http://www.apache.org/licenses/LICENSE-2.0) 133 | 134 | at your option. These files may not be copied, modified, or distributed except according to those terms. 135 | 136 | -------------------------------------------------------------------------------- /config.nims: -------------------------------------------------------------------------------- 1 | # begin Nimble config (version 1) 2 | when fileExists("nimble.paths"): 3 | include "nimble.paths" 4 | # end Nimble config 5 | -------------------------------------------------------------------------------- /nim.cfg: -------------------------------------------------------------------------------- 1 | # Avoid some rare stack corruption while using exceptions with a SEH-enabled 2 | # toolchain: https://github.com/status-im/nimbus-eth2/issues/3121 3 | @if windows and not vcc: 4 | --define:nimRawSetjmp 5 | @end 6 | -------------------------------------------------------------------------------- /serialization.nim: -------------------------------------------------------------------------------- 1 | import 2 | std/typetraits, 3 | stew/shims/macros, faststreams/[inputs, outputs], 4 | ./serialization/[object_serialization, errors, formats] 5 | 6 | export 7 | inputs, outputs, object_serialization, errors, formats 8 | 9 | template encode*(Format: type, value: auto, params: varargs[untyped]): auto = 10 | mixin init, Writer, writeValue, PreferredOutputType 11 | block: # https://github.com/nim-lang/Nim/issues/22874 12 | {.noSideEffect.}: 13 | # We assume that there is no side-effects here, because we are 14 | # using a `memoryOutput`. The computed side-effects are coming 15 | # from the fact that the dynamic dispatch mechanisms used in 16 | # faststreams may be writing to a file or a network device. 17 | try: 18 | var s = memoryOutput() 19 | type WriterType = Writer(Format) 20 | var writer = unpackArgs(init, [WriterType, s, params]) 21 | writeValue writer, value 22 | s.getOutput PreferredOutputType(Format) 23 | except IOError: 24 | raise (ref Defect)() # a memoryOutput cannot have an IOError 25 | 26 | # TODO Nim cannot make sense of this initialization by var param? 27 | proc readValue*(reader: var auto, T: type): T {.gcsafe, raises: [SerializationError, IOError].} = 28 | {.warning[ProveInit]: false.} 29 | mixin readValue 30 | result = default(T) 31 | reader.readValue(result) 32 | {.warning[ProveInit]: true.} 33 | 34 | template decode*(Format: distinct type, 35 | input: string, 36 | RecordType: distinct type, 37 | params: varargs[untyped]): auto = 38 | # TODO, this is dusplicated only due to a Nim bug: 39 | # If `input` was `string|openArray[byte]`, it won't match `seq[byte]` 40 | mixin init, Reader 41 | block: # https://github.com/nim-lang/Nim/issues/22874 42 | {.noSideEffect.}: 43 | # We assume that there are no side-effects here, because we are 44 | # using a `memoryInput`. The computed side-effects are coming 45 | # from the fact that the dynamic dispatch mechanisms used in 46 | # faststreams may be reading from a file or a network device. 47 | try: 48 | var stream = unsafeMemoryInput(input) 49 | type ReaderType = Reader(Format) 50 | var reader = unpackArgs(init, [ReaderType, stream, params]) 51 | reader.readValue(RecordType) 52 | except IOError: 53 | raise (ref Defect)() # memory inputs cannot raise an IOError 54 | 55 | template decode*(Format: distinct type, 56 | input: openArray[byte], 57 | RecordType: distinct type, 58 | params: varargs[untyped]): auto = 59 | # TODO, this is dusplicated only due to a Nim bug: 60 | # If `input` was `string|openArray[byte]`, it won't match `seq[byte]` 61 | mixin init, Reader 62 | block: # https://github.com/nim-lang/Nim/issues/22874 63 | {.noSideEffect.}: 64 | # We assume that there are no side-effects here, because we are 65 | # using a `memoryInput`. The computed side-effects are coming 66 | # from the fact that the dynamic dispatch mechanisms used in 67 | # faststreams may be reading from a file or a network device. 68 | try: 69 | var stream = unsafeMemoryInput(input) 70 | type ReaderType = Reader(Format) 71 | var reader = unpackArgs(init, [ReaderType, stream, params]) 72 | reader.readValue(RecordType) 73 | except IOError: 74 | raise (ref Defect)() # memory inputs cannot raise an IOError 75 | 76 | template loadFile*(Format: distinct type, 77 | filename: string, 78 | RecordType: distinct type, 79 | params: varargs[untyped]): auto = 80 | mixin init, Reader, readValue 81 | 82 | var stream = memFileInput(filename) 83 | try: 84 | type ReaderType = Reader(Format) 85 | var reader = unpackArgs(init, [ReaderType, stream, params]) 86 | reader.readValue(RecordType) 87 | finally: 88 | close stream 89 | 90 | template loadFile*[RecordType](Format: type, 91 | filename: string, 92 | record: var RecordType, 93 | params: varargs[untyped]) = 94 | record = loadFile(Format, filename, RecordType, params) 95 | 96 | template saveFile*(Format: type, filename: string, value: auto, params: varargs[untyped]) = 97 | mixin init, Writer, writeValue 98 | 99 | var stream = fileOutput(filename) 100 | try: 101 | type WriterType = Writer(Format) 102 | var writer = unpackArgs(init, [WriterType, stream, params]) 103 | writer.writeValue(value) 104 | finally: 105 | close stream 106 | 107 | template borrowSerialization*(Alias: type) {.dirty.} = 108 | bind distinctBase 109 | 110 | proc writeValue*[Writer]( 111 | writer: var Writer, value: Alias) {.raises: [IOError].} = 112 | mixin writeValue 113 | writeValue(writer, distinctBase value) 114 | 115 | proc readValue*[Reader](reader: var Reader, value: var Alias) = 116 | mixin readValue 117 | value = Alias reader.readValue(distinctBase Alias) 118 | 119 | template borrowSerialization*(Alias: distinct type, 120 | OriginalType: distinct type) {.dirty.} = 121 | 122 | proc writeValue*[Writer]( 123 | writer: var Writer, value: Alias) {.raises: [IOError].} = 124 | mixin writeValue 125 | writeValue(writer, OriginalType value) 126 | 127 | proc readValue*[Reader](reader: var Reader, value: var Alias) = 128 | mixin readValue 129 | value = Alias reader.readValue(OriginalType) 130 | 131 | template serializesAsBase*(SerializedType: distinct type, 132 | Format: distinct type) = 133 | mixin Reader, Writer 134 | 135 | type ReaderType = Reader(Format) 136 | type WriterType = Writer(Format) 137 | 138 | template writeValue*(writer: var WriterType, value: SerializedType) = 139 | mixin writeValue 140 | writeValue(writer, distinctBase value) 141 | 142 | template readValue*(reader: var ReaderType, value: var SerializedType) = 143 | mixin readValue 144 | value = SerializedType reader.readValue(distinctBase SerializedType) 145 | 146 | macro serializesAsBaseIn*(SerializedType: type, 147 | Formats: varargs[untyped]) = 148 | result = newStmtList() 149 | for Fmt in Formats: 150 | result.add newCall(bindSym"serializesAsBase", SerializedType, Fmt) 151 | 152 | template readValue*(stream: InputStream, 153 | Format: type, 154 | ValueType: type, 155 | params: varargs[untyped]): untyped = 156 | mixin Reader, init, readValue 157 | type ReaderType = Reader(Format) 158 | var reader = unpackArgs(init, [ReaderType, stream, params]) 159 | readValue reader, ValueType 160 | 161 | template writeValue*(stream: OutputStream, 162 | Format: type, 163 | value: auto, 164 | params: varargs[untyped]) = 165 | mixin Writer, init, writeValue 166 | type WriterType = Writer(Format) 167 | var writer = unpackArgs(init, [WriterType, stream, params]) 168 | writeValue writer, value 169 | -------------------------------------------------------------------------------- /serialization.nimble: -------------------------------------------------------------------------------- 1 | mode = ScriptMode.Verbose 2 | 3 | packageName = "serialization" 4 | version = "0.2.7" 5 | author = "Status Research & Development GmbH" 6 | description = "A modern and extensible serialization framework for Nim" 7 | license = "Apache License 2.0" 8 | skipDirs = @["tests"] 9 | 10 | requires "nim >= 1.6.0", 11 | "faststreams", 12 | "unittest2", 13 | "stew" 14 | 15 | let nimc = getEnv("NIMC", "nim") # Which nim compiler to use 16 | let lang = getEnv("NIMLANG", "c") # Which backend (c/cpp/js) 17 | let flags = getEnv("NIMFLAGS", "") # Extra flags for the compiler 18 | let verbose = getEnv("V", "") notin ["", "0"] 19 | 20 | let cfg = 21 | " --styleCheck:usages --styleCheck:error" & 22 | (if verbose: "" else: " --verbosity:0") & 23 | " --skipParentCfg --skipUserCfg --outdir:build --nimcache:build/nimcache -f" 24 | 25 | proc build(args, path: string) = 26 | exec nimc & " " & lang & " " & cfg & " " & flags & " " & args & " " & path 27 | 28 | proc run(args, path: string) = 29 | build args & " --mm:refc -r", path 30 | if (NimMajor, NimMinor) > (1, 6): 31 | build args & " --mm:orc -r", path 32 | 33 | task test, "Run all tests": 34 | for threads in ["--threads:off", "--threads:on"]: 35 | run threads, "tests/test_all" 36 | -------------------------------------------------------------------------------- /serialization/errors.nim: -------------------------------------------------------------------------------- 1 | type 2 | SerializationError* = object of CatchableError 3 | UnexpectedEofError* = object of SerializationError 4 | CustomSerializationError* = object of SerializationError 5 | 6 | method formatMsg*(err: ref SerializationError, filename: string): string 7 | {.gcsafe, base, raises: [Defect].} = 8 | "Serialisation error while processing " & filename & ":" & err.msg 9 | 10 | -------------------------------------------------------------------------------- /serialization/formats.nim: -------------------------------------------------------------------------------- 1 | import 2 | std/[typetraits, macros] 3 | 4 | type 5 | DefaultFlavor* = object 6 | 7 | template serializationFormatImpl(Name: untyped, 8 | mimeTypeName: static string = "") {.dirty.} = 9 | # This indirection is required in order to be able to generate the 10 | # `mimeType` accessor template. Without the indirection, the template 11 | # mechanism of Nim will try to expand the `mimeType` param in the position 12 | # of the `mimeType` template name which will result in error. 13 | type Name* = object 14 | template mimeType*(T: type Name): string = mimeTypeName 15 | 16 | template serializationFormat*(Name: untyped, mimeType: static string = "") = 17 | serializationFormatImpl(Name, mimeType) 18 | 19 | template setReader*(Format, FormatReader: distinct type) = 20 | when arity(FormatReader) > 1: 21 | template ReaderType*(T: type Format, F: distinct type = DefaultFlavor): type = FormatReader[F] 22 | template Reader*(T: type Format, F: distinct type = DefaultFlavor): type = FormatReader[F] 23 | else: 24 | template ReaderType*(T: type Format): type = FormatReader 25 | template Reader*(T: type Format): type = FormatReader 26 | 27 | template setWriter*(Format, FormatWriter, PreferredOutput: distinct type) = 28 | when arity(FormatWriter) > 1: 29 | template WriterType*(T: type Format, F: distinct type = DefaultFlavor): type = FormatWriter[F] 30 | template Writer*(T: type Format, F: distinct type = DefaultFlavor): type = FormatWriter[F] 31 | else: 32 | template WriterType*(T: type Format): type = FormatWriter 33 | template Writer*(T: type Format): type = FormatWriter 34 | 35 | template PreferredOutputType*(T: type Format): type = PreferredOutput 36 | 37 | template createFlavor*(ModifiedFormat, FlavorName: untyped) = 38 | type FlavorName* = object 39 | template Reader*(T: type FlavorName): type = Reader(ModifiedFormat, FlavorName) 40 | template Writer*(T: type FlavorName): type = Writer(ModifiedFormat, FlavorName) 41 | template PreferredOutputType*(T: type FlavorName): type = PreferredOutputType(ModifiedFormat) 42 | template mimeType*(T: type FlavorName): string = mimeType(ModifiedFormat) 43 | 44 | template toObjectType(T: type): untyped = 45 | typeof(T()[]) 46 | 47 | template toObjectTypeIfNecessary(T: type): untyped = 48 | when T is ref|ptr: 49 | toObjectType(T) 50 | else: 51 | T 52 | 53 | # useDefault***In or useDefault***For only works for 54 | # object|ref object|ptr object 55 | 56 | template useDefaultSerializationIn*(T: untyped, Flavor: type) = 57 | mixin Reader, Writer 58 | 59 | type TT = toObjectTypeIfNecessary(T) 60 | 61 | template readValue*(r: var Reader(Flavor), value: var TT) = 62 | mixin readRecordValue 63 | readRecordValue(r, value) 64 | 65 | template writeValue*(w: var Writer(Flavor), value: TT) = 66 | mixin writeRecordValue 67 | writeRecordValue(w, value) 68 | 69 | template useDefaultWriterIn*(T: untyped, Flavor: type) = 70 | mixin Writer 71 | 72 | type TT = toObjectTypeIfNecessary(T) 73 | 74 | template writeValue*(w: var Writer(Flavor), value: TT) = 75 | mixin writeRecordValue 76 | writeRecordValue(w, value) 77 | 78 | template useDefaultReaderIn*(T: untyped, Flavor: type) = 79 | mixin Reader 80 | 81 | type TT = toObjectTypeIfNecessary(T) 82 | 83 | template readValue*(r: var Reader(Flavor), value: var TT) = 84 | mixin readRecordValue 85 | readRecordValue(r, value) 86 | 87 | macro useDefaultSerializationFor*(Flavor: type, types: varargs[untyped])= 88 | result = newStmtList() 89 | 90 | for T in types: 91 | result.add newCall(bindSym "useDefaultSerializationIn", T, Flavor) 92 | 93 | macro useDefaultWriterFor*(Flavor: type, types: varargs[untyped])= 94 | result = newStmtList() 95 | 96 | for T in types: 97 | result.add newCall(bindSym "useDefaultWriterIn", T, Flavor) 98 | 99 | macro useDefaultReaderFor*(Flavor: type, types: varargs[untyped])= 100 | result = newStmtList() 101 | 102 | for T in types: 103 | result.add newCall(bindSym "useDefaultReaderIn", T, Flavor) 104 | -------------------------------------------------------------------------------- /serialization/object_serialization.nim: -------------------------------------------------------------------------------- 1 | import 2 | std/typetraits, 3 | stew/shims/macros, stew/objects, 4 | ./errors, ./formats 5 | 6 | type 7 | FieldTag*[RecordType: object; fieldName: static string] = distinct void 8 | 9 | export 10 | DefaultFlavor 11 | 12 | let 13 | # Identifiers affecting the public interface of the library: 14 | valueSym {.compileTime.} = ident "value" 15 | readerSym {.compileTime.} = ident "reader" 16 | writerSym {.compileTime.} = ident "writer" 17 | holderSym {.compileTime.} = ident "holder" 18 | 19 | template dontSerialize* {.pragma.} 20 | ## Specifies that a certain field should be ignored for 21 | ## the purposes of serialization 22 | 23 | template serializedFieldName*(name: string) {.pragma.} 24 | ## Specifies an alternative name for the field that will 25 | ## be used in formats that include field names. 26 | 27 | template enumInstanceSerializedFields*(obj: auto, 28 | fieldNameVar, fieldVar, 29 | body: untyped) = 30 | ## Expands a block over all serialized fields of an object. 31 | ## 32 | ## Inside the block body, the passed `fieldNameVar` identifier 33 | ## will refer to the name of each field as a string. `fieldVar` 34 | ## will refer to the field value. 35 | ## 36 | ## The order of visited fields matches the order of the fields in 37 | ## the object definition unless `setSerializedFields` is used to specify 38 | ## a different order. Fields marked with the `dontSerialize` pragma 39 | ## are skipped. 40 | ## 41 | ## If the visited object is a case object, only the currently active 42 | ## fields will be visited. During de-serialization, case discriminators 43 | ## will be read first and the iteration will continue depending on the 44 | ## value being deserialized. 45 | ## 46 | type ObjType {.used.} = type(obj) 47 | 48 | for fieldName, fieldVar in fieldPairs(obj): 49 | when not hasCustomPragmaFixed(ObjType, fieldName, dontSerialize): 50 | when hasCustomPragmaFixed(ObjType, fieldName, serializedFieldName): 51 | const fieldNameVar = getCustomPragmaFixed(ObjType, fieldName, serializedFieldName) 52 | else: 53 | const fieldNameVar = fieldName 54 | body 55 | 56 | macro enumAllSerializedFieldsImpl(T: type, body: untyped): untyped = 57 | ## Expands a block over all fields of a type 58 | ## 59 | ## Please note that the main difference between 60 | ## `enumInstanceSerializedFields` and `enumAllSerializedFields` 61 | ## is that the later will visit all fields of case objects. 62 | ## 63 | ## Inside the block body, the following symbols will be defined: 64 | ## 65 | ## * `fieldName` 66 | ## String literal for the field name. 67 | ## The value can be affected by the `serializedFieldName` pragma. 68 | ## 69 | ## * `realFieldName` 70 | ## String literal for actual field name in the Nim type 71 | ## definition. Not affected by the `serializedFieldName` pragma. 72 | ## 73 | ## * `FieldType` 74 | ## Type alias for the field type 75 | ## 76 | ## * `fieldCaseDiscriminator` 77 | ## String literal denoting the name of the case object 78 | ## discriminator under which the visited field is nested. 79 | ## If the field is not nested in a specific case branch, 80 | ## this will be an empty string. 81 | ## 82 | ## * `fieldCaseBranches` 83 | ## A set literal node denoting the possible values of the 84 | ## case object discriminator which make this field accessible. 85 | ## 86 | ## The order of visited fields matches the order of the fields in 87 | ## the object definition unless `setSerializedFields` is used to specify 88 | ## a different order. Fields marked with the `dontSerialize` pragma 89 | ## are skipped. 90 | ## 91 | var typeAst = getType(T)[1] 92 | var typeImpl: NimNode 93 | let isSymbol = not typeAst.isTuple 94 | 95 | if not isSymbol: 96 | typeImpl = typeAst 97 | else: 98 | typeImpl = getImpl(typeAst) 99 | result = newStmtList() 100 | 101 | var i = 0 102 | for field in recordFields(typeImpl): 103 | if field.readPragma("dontSerialize") != nil: 104 | continue 105 | 106 | let 107 | fieldIdent = field.name 108 | realFieldName = newLit($fieldIdent.skipPragma) 109 | serializedFieldName = field.readPragma("serializedFieldName") 110 | fieldName = if serializedFieldName == nil: realFieldName 111 | else: serializedFieldName 112 | discriminator = newLit(if field.caseField == nil: "" 113 | else: $field.caseField[0].skipPragma) 114 | branches = field.caseBranch 115 | fieldIndex = newLit(i) 116 | 117 | let fieldNameDefs = 118 | if isSymbol: 119 | quote: 120 | const fieldName {.inject, used.} = `fieldName` 121 | const realFieldName {.inject, used.} = `realFieldName` 122 | else: 123 | quote: 124 | const fieldName {.inject, used.} = $`fieldIndex` 125 | const realFieldName {.inject, used.} = $`fieldIndex` 126 | # we can't access .Fieldn, so our helper knows 127 | # to parseInt this 128 | 129 | let field = 130 | if isSymbol: 131 | quote do: declval(`T`).`fieldIdent` 132 | else: 133 | quote do: declval(`T`)[`fieldIndex`] 134 | 135 | result.add quote do: 136 | block: 137 | `fieldNameDefs` 138 | 139 | # `FieldType` should be `type`: 140 | # https://github.com/nim-lang/Nim/issues/23564 141 | template FieldType: untyped {.inject, used.} = typeof(`field`) 142 | 143 | template fieldCaseDiscriminator: auto {.used.} = `discriminator` 144 | template fieldCaseBranches: auto {.used.} = `branches` 145 | 146 | `body` 147 | 148 | i += 1 149 | 150 | template enumAllSerializedFields*(T: type, body): untyped = 151 | when T is ref|ptr: 152 | type TT = type(default(T)[]) 153 | enumAllSerializedFieldsImpl(TT, body) 154 | else: 155 | enumAllSerializedFieldsImpl(T, body) 156 | 157 | func isCaseObject*(T: type): bool {.compileTime.} = 158 | genSimpleExpr: 159 | enumAllSerializedFields(T): 160 | if fieldCaseDiscriminator != "": 161 | return newLit(true) 162 | 163 | newLit(false) 164 | 165 | type 166 | FieldMarkerImpl*[name: static string] = object 167 | 168 | FieldReader*[RecordType, Reader] = tuple[ 169 | fieldName: string, 170 | reader: proc (rec: var RecordType, reader: var Reader) 171 | {.gcsafe, nimcall, raises: [SerializationError, Defect].} 172 | ] 173 | 174 | FieldReadersTable*[RecordType, Reader] = openArray[FieldReader[RecordType, Reader]] 175 | 176 | proc totalSerializedFieldsImpl(T: type): int = 177 | mixin enumAllSerializedFields 178 | enumAllSerializedFields(T): inc result 179 | 180 | template totalSerializedFields*(T: type): int = 181 | (static(totalSerializedFieldsImpl(T))) 182 | 183 | macro customSerialization*(field: untyped, definition): untyped = 184 | discard 185 | 186 | template GetFieldType(FT: type FieldTag): type = 187 | typeof field(declval(FT.RecordType), FT.fieldName) 188 | 189 | template readFieldIMPL[Reader](field: type FieldTag, 190 | reader: var Reader): auto = 191 | mixin readValue 192 | # Nim 1.6.12: `type FieldType = GetFieldType(field)` here breaks `NimYAML`. 193 | {.gcsafe.}: # needed by Nim-1.6 194 | # TODO: The `GetFieldType(field)` coercion below is required to deal 195 | # with a nim bug caused by the distinct `ssz.List` type. 196 | # It seems to break the generics cache mechanism, which 197 | # leads to an incorrect return type being reported from 198 | # the `readFieldIMPL` function. 199 | 200 | # additional notes: putting the GetFieldType(field) coercion in 201 | # `makeFieldReadersTable` will cause problems when orc enabled 202 | # hence, move it here 203 | when distinctBase(GetFieldType(field)) isnot GetFieldType(field): 204 | GetFieldType(field) reader.readValue(GetFieldType(field)) 205 | else: 206 | reader.readValue(GetFieldType(field)) 207 | 208 | template writeFieldIMPL*[Writer](writer: var Writer, 209 | fieldTag: type FieldTag, 210 | fieldVal: auto, 211 | holderObj: auto) = 212 | mixin writeValue 213 | writeValue(writer, fieldVal) 214 | 215 | proc makeFieldReadersTable(RecordType, ReaderType: distinct type, 216 | numFields: static[int]): 217 | array[numFields, FieldReader[RecordType, ReaderType]] = 218 | mixin enumAllSerializedFields, readFieldIMPL, handleReadException 219 | var idx = 0 220 | 221 | enumAllSerializedFields(RecordType): 222 | proc readField(obj: var RecordType, reader: var ReaderType) 223 | {.gcsafe, nimcall, raises: [SerializationError].} = 224 | 225 | mixin readValue 226 | 227 | when RecordType is tuple: 228 | const i = fieldName.parseInt 229 | 230 | try: 231 | when RecordType is tuple: 232 | reader.readValue obj[i] 233 | else: 234 | static: doAssert not isCaseObject(typeof(obj)), 235 | "Case object `" & $typeof(obj) & 236 | "` must have custom `readValue` for `" & $typeof(reader) & "`" 237 | type F = FieldTag[RecordType, realFieldName] 238 | {.push hint[ConvFromXtoItselfNotNeeded]: off.} 239 | field(obj, realFieldName) = readFieldIMPL(F, reader) 240 | {.pop.} 241 | except SerializationError as err: 242 | raise err 243 | except CatchableError as err: 244 | type LocalRecordType = `RecordType` # workaround to allow compile time evaluation 245 | reader.handleReadException( 246 | LocalRecordType, 247 | fieldName, 248 | when RecordType is tuple: obj[i] else: field(obj, realFieldName), 249 | err) 250 | 251 | result[idx] = (fieldName, readField) 252 | inc idx 253 | 254 | template fieldReadersTable*(RecordType, ReaderType: distinct type): auto = 255 | type T = RecordType 256 | const numFields = totalSerializedFields(T) 257 | makeFieldReadersTable(T, ReaderType, numFields) 258 | 259 | template `[]`*(v: FieldReadersTable): FieldReadersTable {.deprecated.} = v 260 | 261 | proc findFieldIdx*(fieldsTable: FieldReadersTable, 262 | fieldName: string, 263 | expectedFieldPos: var int): int = 264 | for i in expectedFieldPos ..< fieldsTable.len: 265 | if fieldsTable[i].fieldName == fieldName: 266 | expectedFieldPos = i + 1 267 | return i 268 | 269 | for i in 0 ..< expectedFieldPos: 270 | if fieldsTable[i].fieldName == fieldName: 271 | return i 272 | 273 | return -1 274 | 275 | proc findFieldReader*(fieldsTable: FieldReadersTable, 276 | fieldName: string, 277 | expectedFieldPos: var int): auto = 278 | for i in expectedFieldPos ..< fieldsTable.len: 279 | if fieldsTable[i].fieldName == fieldName: 280 | expectedFieldPos = i + 1 281 | return fieldsTable[i].reader 282 | 283 | for i in 0 ..< expectedFieldPos: 284 | if fieldsTable[i].fieldName == fieldName: 285 | return fieldsTable[i].reader 286 | 287 | return nil 288 | 289 | macro setSerializedFields*(T: typedesc, fields: varargs[untyped]): untyped = 290 | var fieldsArray = newTree(nnkBracket) 291 | for f in fields: fieldsArray.add newCall(bindSym"ident", newLit($f)) 292 | 293 | template payload(T: untyped, fieldsArray) {.dirty.} = 294 | bind default, quote, add, getType, newStmtList, 295 | ident, newLit, newDotExpr, `$`, `[]`, getAst 296 | 297 | macro enumInstanceSerializedFields*(ins: T, 298 | fieldNameVar, fieldVar, 299 | body: untyped): untyped = 300 | var 301 | fields = fieldsArray 302 | res = newStmtList() 303 | 304 | for field in fields: 305 | let 306 | fieldName = newLit($field) 307 | fieldAccessor = newDotExpr(ins, field) 308 | 309 | # TODO replace with getAst once it's ready 310 | template fieldPayload(fieldNameVar, fieldName, fieldVar, 311 | fieldAccessor, body) = 312 | block: 313 | const fieldNameVar {.inject, used.} = fieldName 314 | template fieldVar: auto {.used.} = fieldAccessor 315 | 316 | body 317 | 318 | res.add getAst(fieldPayload(fieldNameVar, fieldName, 319 | fieldVar, fieldAccessor, 320 | body)) 321 | return res 322 | 323 | macro enumAllSerializedFields*(typ: type T, body: untyped): untyped = 324 | var 325 | fields = fieldsArray 326 | res = newStmtList() 327 | typ = getType(typ) 328 | 329 | for field in fields: 330 | let fieldName = newLit($field) 331 | 332 | # TODO replace with getAst once it's ready 333 | template fieldPayload(fieldNameValue, typ, field, body) = 334 | block: 335 | const fieldName {.inject, used.} = fieldNameValue 336 | const realFieldName {.inject, used.} = fieldNameValue 337 | 338 | type FieldType {.inject, used.} = type(declval(typ).field) 339 | 340 | template fieldCaseDiscriminator: auto {.used.} = "" 341 | template fieldCaseBranches: auto {.used.} = nil 342 | 343 | body 344 | 345 | res.add getAst(fieldPayload(fieldName, typ, field, body)) 346 | 347 | return res 348 | 349 | return getAst(payload(T, fieldsArray)) 350 | 351 | proc getReaderAndWriter(customSerializationBody: NimNode): (NimNode, NimNode) = 352 | template fail(n) = 353 | error "useCustomSerialization expects a block with only `read` and `write` definitions", n 354 | 355 | for n in customSerializationBody: 356 | if n.kind in nnkCallKinds: 357 | if eqIdent(n[0], "read"): 358 | result[0] = n[1] 359 | elif eqIdent(n[0], "write"): 360 | result[1] = n[1] 361 | else: 362 | fail n[0] 363 | elif n.kind == nnkCommentStmt: 364 | continue 365 | else: 366 | fail n 367 | 368 | proc genCustomSerializationForField(Format, field, 369 | readBody, writeBody: NimNode): NimNode = 370 | var 371 | RecordType = field[0] 372 | fieldIdent = field[1] 373 | fieldName = newLit $fieldIdent 374 | FieldType = genSym(nskType, "FieldType") 375 | 376 | result = newStmtList() 377 | result.add quote do: 378 | type `FieldType` = type declval(`RecordType`).`fieldIdent` 379 | 380 | if readBody != nil: 381 | result.add quote do: 382 | type ReaderType = Reader(`Format`) 383 | proc readFieldIMPL*(F: type FieldTag[`RecordType`, `fieldName`], 384 | `readerSym`: var ReaderType): `FieldType` 385 | {.raises: [IOError, SerializationError], gcsafe.} = 386 | `readBody` 387 | 388 | if writeBody != nil: 389 | result.add quote do: 390 | type WriterType = Writer(`Format`) 391 | proc writeFieldIMPL*(`writerSym`: var WriterType, 392 | F: type FieldTag[`RecordType`, `fieldName`], 393 | `valueSym`: auto, 394 | `holderSym`: `RecordType`) 395 | {.raises: [IOError], gcsafe.} = 396 | `writeBody` 397 | 398 | proc genCustomSerializationForType(Format, typ: NimNode, 399 | readBody, writeBody: NimNode): NimNode = 400 | result = newStmtList() 401 | 402 | if readBody != nil: 403 | result.add quote do: 404 | type ReaderType = Reader(`Format`) 405 | proc readValue*(`readerSym`: var ReaderType, T: type `typ`): `typ` 406 | {.raises: [IOError, SerializationError], gcsafe.} = 407 | `readBody` 408 | 409 | if writeBody != nil: 410 | result.add quote do: 411 | type WriterType = Writer(`Format`) 412 | proc writeValue*(`writerSym`: var WriterType, `valueSym`: `typ`) 413 | {.raises: [IOError], gcsafe.} = 414 | `writeBody` 415 | 416 | macro useCustomSerialization*(Format: typed, field: untyped, body: untyped): untyped = 417 | let (readBody, writeBody) = getReaderAndWriter(body) 418 | if field.kind == nnkDotExpr: 419 | result = genCustomSerializationForField(Format, field, readBody, writeBody) 420 | elif field.kind in {nnkIdent, nnkAccQuoted}: 421 | result = genCustomSerializationForType(Format, field, readBody, writeBody) 422 | else: 423 | error "useCustomSerialization expects a type name or a field of a type (e.g. MyType.myField)" 424 | 425 | when defined(debugUseCustomSerialization): 426 | echo result.repr 427 | -------------------------------------------------------------------------------- /serialization/testing/generic_suite.nim: -------------------------------------------------------------------------------- 1 | import 2 | std/[times, typetraits, random, strutils, options, sets, tables], 3 | unittest2, 4 | faststreams/inputs, 5 | ../../serialization, ../object_serialization 6 | 7 | type 8 | Meter* = distinct int 9 | Mile* = distinct int 10 | 11 | Simple* = object 12 | x*: int 13 | y*: string 14 | distance*: Meter 15 | ignored*: int 16 | 17 | Transaction* = object 18 | amount*: int 19 | time*: DateTime 20 | sender*: string 21 | receiver*: string 22 | 23 | BaseType* = object of RootObj 24 | a*: string 25 | b*: int 26 | 27 | BaseTypeRef* = ref BaseType 28 | 29 | DerivedType* = object of BaseType 30 | c*: int 31 | d*: string 32 | 33 | DerivedRefType* = ref object of BaseType 34 | c*: int 35 | d*: string 36 | 37 | DerivedFromRefType* = ref object of DerivedRefType 38 | e*: int 39 | 40 | RefTypeDerivedFromRoot* = ref object of RootObj 41 | a*: int 42 | b*: string 43 | 44 | Foo = object 45 | x*: uint64 46 | y*: string 47 | z*: seq[int] 48 | 49 | Bar = object 50 | b*: string 51 | f*: Foo 52 | 53 | # Baz should use custom serialization 54 | # The `i` field should be multiplied by two while deserializing and 55 | # `ignored` field should be set to 10 56 | Baz = object 57 | f*: Foo 58 | i*: int 59 | ignored* {.dontSerialize.}: int 60 | 61 | ListOfLists = object 62 | lists*: seq[ListOfLists] 63 | 64 | NoExpectedResult = distinct int 65 | 66 | ObjectKind* = enum 67 | A 68 | B 69 | 70 | CaseObject* = object 71 | case kind*: ObjectKind 72 | of A: 73 | a*: int 74 | other*: CaseObjectRef 75 | else: 76 | b*: int 77 | 78 | CaseObjectRef* = ref CaseObject 79 | 80 | HoldsCaseObject* = object 81 | value: CaseObject 82 | 83 | HoldsSet* = object 84 | a*: int 85 | s*: HashSet[string] 86 | 87 | HoldsOption* = object 88 | r*: ref Simple 89 | o*: Option[Simple] 90 | 91 | HoldsArray* = object 92 | data*: seq[int] 93 | 94 | AnonTuple* = (int, string, float64) 95 | 96 | AbcTuple* = tuple[a: int, b: string, c: float64] 97 | XyzTuple* = tuple[x: int, y: string, z: float64] 98 | 99 | HoldsTuples* = object 100 | t1*: AnonTuple 101 | t2*: AbcTuple 102 | t3*: XyzTuple 103 | 104 | static: 105 | assert isCaseObject(CaseObject) 106 | assert isCaseObject(CaseObjectRef) 107 | 108 | assert(not isCaseObject(Transaction)) 109 | assert(not isCaseObject(HoldsSet)) 110 | 111 | Meter.borrowSerialization int 112 | Simple.setSerializedFields distance, x, y 113 | 114 | func caseObjectEquals(a, b: CaseObject): bool {.raises: [].} 115 | 116 | func `==`*(a, b: CaseObjectRef): bool {.raises: [].} = 117 | let nils = ord(a.isNil) + ord(b.isNil) 118 | if nils == 0: 119 | caseObjectEquals(a[], b[]) 120 | else: 121 | nils == 2 122 | 123 | func caseObjectEquals(a, b: CaseObject): bool {.raises: [].} = 124 | # TODO This is needed to work-around a Nim overload selection issue 125 | if a.kind != b.kind: return false 126 | 127 | case a.kind 128 | of A: 129 | if a.a != b.a: return false 130 | a.other == b.other 131 | of B: 132 | a.b == b.b 133 | 134 | func `==`*(a, b: CaseObject): bool = 135 | caseObjectEquals(a, b) 136 | 137 | func `==`*(a, b: ListOfLists): bool = 138 | if a.lists.len != b.lists.len: 139 | return false 140 | for i in 0..