├── .gitignore ├── CHANGELOG.md ├── LICENSE ├── README.md ├── bin └── scbcat.ts ├── lib ├── extendschema.ts ├── index.ts ├── metaschema.ts ├── read.ts ├── schema.ts ├── testhelpers.ts ├── utils.ts └── write.ts ├── package.json ├── test ├── extend.ts ├── json.ts ├── merging.ts ├── metaschema.ts ├── read.ts ├── roundtrips.ts ├── tldraw_test.ts └── write.ts └── tsconfig.json /.gitignore: -------------------------------------------------------------------------------- 1 | dist 2 | node_modules 3 | 4 | yarn.lock 5 | out.scb 6 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | # 0.3.0: 2 | 3 | - Changed package from commonjs to esm 4 | 5 | # 0.2.0: 6 | 7 | - Fixed `scbcat` to output larger documents better and output maps correctly 8 | - Rewrote structs to be semantically identical to enums with 1 variant 9 | - Fixed a bunch of error messages 10 | - Exported sb.String / sb.Id / sb.Bool types directly 11 | - Added `encodeEntry` and `decodeEntry` mapping functions for maps 12 | - Made the `decodeType` of objects / maps a required field (after I kept getting surprised by the default) 13 | - Refactored bijective-varint code into its own package 14 | - Fixed `read()` to return remote schema instead of merged schema. Made read method take an optional type argument for overriding / specifying the root type being read. 15 | - Exported `testhelpers.ts` helper methods to make sure your types can round-trip correctly. 16 | - Flattened AppEnumSchema to not need `associatedData` field 17 | - Removed `type: 'struct'` declaration in `AppSchema` 18 | - Restricted list type to an SType for its field type. Added `sb.list(type)` helper method. 19 | - Made schema root optional 20 | 21 | 22 | # 0.1.1: 23 | 24 | - Added `scbcat` script 25 | - Changed typescript to output commonjs (urgh) for better nodejs compatibility 26 | 27 | # 0.1.0: 28 | 29 | - First published version! Woo! -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | ISC License 2 | 3 | Copyright 2023 Joseph Gentle 4 | 5 | Permission to use, copy, modify, and/or distribute this software for any purpose with or without fee is hereby granted, provided that the above copyright notice and this permission notice appear in all copies. 6 | 7 | THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | > **STATUS: EXPERIMENTAL**. Do not use this for anything you care about yet! Schemaboi is still in flux, and the binary representation may drift in mutually incompatible ways before 1.0. 2 | 3 | # Schemaboi: A binary format with schema evolution 4 | 5 | Schemaboi is an efficient binary serialization format (like protobuf) thats designed for applications with schemas that change over time. (Which is, basically every application). 6 | 7 | Schemaboi has many of the same goals as Ink & Switch's [Project Cambria](https://www.inkandswitch.com/cambria/), to enable applications to interoperate despite their schemas changing over time. 8 | 9 | Imagine two applications, both editing shared blog posts. One application wants to add a new `featured: bool` field to the data schema, marking featured blog posts. 10 | 11 | - Using JSON, the schema can change in any way, but ad-hoc fields lead to messy, ad-hoc code that becomes increasingly buggy and messy over time. 12 | - Using protobuf (or something similar), both application developers need to coordinate to decide which fields get added. This coordination overhead slows down development work - data formats are punished for their popularity! 13 | 14 | Schemaboi solves this problem! 15 | 16 | Every data format is described by a *Schema*. The schema describes 3 pieces of information: 17 | 18 | 1. **Encoding:** How the data is stored in binary form (on disk or over the network) 19 | 2. **Core Schema:** What types and fields are available 20 | 3. **Local Mapping:** How those fields map into your programming language (eg Javascript or Rust) 21 | 22 | Schemaboi data files **embed the schema** (well, the encoding and core schema parts). Your application embeds the schema it expects (part 2 and 3). When a SB file is loaded, the stored data is mapped to your local types. 23 | 24 | - Anything your application understands is validated and kept. 25 | - Anything your application doesn't understand is stored separately, and re-encoded when the data is saved back to disk. Round-trips *never* lose data. 26 | 27 | This enables: 28 | 29 | - **Schema Evolution:** Data stored with schemaboi can be both forwards- *and backwards-* compatible with other application software. Application authors can add new fields to the schema at will. (Or remove (ignore) old ones). Any new data fields will be preserved by other applications, without interfering with how those apps work. Schemaboi is designed for applications that should still work without modification in 100 years from now. 30 | 2. **Encoding efficiency:** Unlike JSON, schemaboi has an extremely compact packed binary format. Data takes even fewer bytes over the wire than other binary formats like protobuf. (Though we do need to store the schema - but its usually small!) 31 | 3. **Self describing:** Data stored in schemaboi is self describing. Like JSON or msgpack, schemaboi data doesn't need a special out-of-band schema file in order to interpret the data. You can print any schemaboi file out as raw JSON, or edit it directly. (Though you will need our schemaboi CLI tool to do so). There is no special compilation step to use schemaboi. 32 | 33 | Schemaboi also supports more data types than JSON, like opaque binary blobs and parametric enums, as found in modern languages like Swift, Rust, Typescript and Haskell. (Eg `enum Shape { Circle(centre, radius), Square(x, y, sidelength) }`). 34 | 35 | A schemaboi file contains a strict tree of data, like JSON. There are no pointers in the structure. 36 | 37 | 38 | ## Schema merging 39 | 40 | Most schema systems (protobuf, etc) don't deal well with schema changes. There are a lot of complex cases schema systems need to deal with to manage this: 41 | 42 | 1. A schema is modified from v1 to v2 with new fields being added. Old data is loaded by the application, and the new fields are missing. 43 | 2. A file is saved to disk (or in a database or something) using v2 of the application. This data is then loaded by some software that has not been updated to use the new schema. The application does not understand some fields. And the application must not delete any data when the file is saved. 44 | 3. In local first software, multiple application authors may independently add different features to their software. Rather than a linear series of schema changes (v1 -> v2 -> v3), versions can diverge. Eg, v0 -> vA and v0 -> vB. As much as possible, applications should ignore and preserve any fields they don't understand. 45 | 46 | In schemaboi, when loading data we always (at runtime) have access to two schemas: 47 | 48 | 1. The **local schema**. This schema is embedded within the running application. 49 | 2. The **remote schema** - usually embedded with the data stored on disk or in a database, or transmitted over the network from a remote application. 50 | 51 | If the remote and local schemas don't match, data is loaded as follows: 52 | 53 | 1. The schemas are *merged*. Generally, this means taking the union of all the enum variants and all the struct fields known by both applications. This may error (see below). 54 | 2. The data is loaded according to the merged schema. 55 | 56 | When loading using the merged schema, we apply the following rules: 57 | 58 | - Any struct fields not known by the local application are stored in a separate `foreign fields` chunk. (In javascript, this is stored in a special `{_foreign: {...}}` field. 59 | - Any missing local struct fields are filled in with default values if possible, or if not the entire struct is considered foreign (see below). 60 | - For enums, if the value has a variant not known to the local schema, it is considered foreign. 61 | 62 | ### Foreign data 63 | 64 | If remote data cannot be loaded by the local application (because the enum variant is unknown, or required struct fields are missing) then the data is considered *foreign*. Foreign data bubbles up during parsing until it reaches one of two things: 65 | 66 | 1. An enum which supports foreign variants. The entire sub-tree of data will be stored as a *foreign variant* in the parent enum. Its then up to the application to decide how this is displayed. Eg, one application adds "video tweets". Another application doesn't support video tweets yet, and so it displays a message for foreign variants saying "@seph made a tweet using another client that we cannot display". 67 | 2. If foreign data reaches the root, the application generates a parse error. 68 | 69 | 70 | ### When can schemas be merged? 71 | 72 | Schema merging is possible for most schema changes: 73 | 74 | - New struct fields are added or removed. 75 | - Enum variants are added or removed (though data using the new variants will not be understood by other applications). 76 | - Enum variants can have fields added or removed. (Enum variants are a struct in disguise). 77 | - Structs can be widened into enums. The original struct will be matched to the first enum variant. 78 | - Required values become optional, or vice versa. (Optionals are really just an enum of Some(value) or None - so you can think of this as a struct to enum widening). 79 | 80 | But some schema changes are illegal, and will cause errors with past versions of the application: 81 | 82 | - Changing the type of a field (eg string to integer). For now the only exception is enums and structs can be swapped. (A struct is just an enum with 1 variant). 83 | - This means you can't do some of the things described in the cambria paper - like reinterpret a single value as a list. 84 | - The expectation is that every field has a unique name. If this worries you, use some form of unique field names (domain scoped, GUIDs, etc) and map them to readable names in the local mapping of your application. 85 | - ??? (I bet there's more things this precludes..) 86 | 87 | Note that part of the schema - the local mapping - describes how the stored data maps to types in your programming language. This information isn't stored with the data or shared between applications. Changes to the local mapping have no impact beyond your local application: 88 | 89 | - Changing default values for fields 90 | - Making enums support foreign variants 91 | - Renaming fields (your application should keep the original field name on disk. The field is just mapped locally). 92 | 93 | 94 | ## Schemas are stored with the data 95 | 96 | In schemaboi, all data is stored & sent with the schema used to generate that data. When data is stored on disk, the schema is stored with the data. And when SB is used as part of a network protocol, the schemas both peers use should be transmitted immediately once the connection is opened. 97 | 98 | This has a pretty modest performance impact (schema information is usually very small - just a few kb at most). But it enables the schema migration features described above. 99 | 100 | Schemaboi files on disk look like this: 101 | 102 | ``` 103 | [Magic??] 104 | [Media type, schema and encoding information] 105 | [Data] 106 | ``` 107 | 108 | This is a departure from formats like JSON and msgpack, which have the schema information (field names) repeated and interleaved throughout the data: 109 | 110 | ``` 111 | [field names + data, field names + data, field names + data] 112 | ``` 113 | 114 | Or protobuf and capnproto, which split the data and the schema into two separate data chunks: 115 | 116 | ``` 117 | myschema.proto: 118 | [schema] 119 | 120 | myfile.data: 121 | [data] 122 | ``` 123 | 124 | This makes schemaboi *less efficient* for very small data sets. (Though the schema is usually pretty tiny). But much more efficient for very large data sets, where the cost of storing the schema information is amortized. 125 | 126 | There are also a couple other advantages to doing this: 127 | 128 | 1. The schema can contain hints on how the data is encoded. We can get better compression efficiency in some cases. 129 | 2. By storing the schema information with the data, we can use generic SB tools to load, view and edit any SB data in a human-readable form (like JSON). 130 | 131 | This approach also has a commensurate disadvantage with for compiled languages: The schema needs to be parsed (at runtime), so we can't use code generation to make a super efficient SB parser for each application in the general case. 132 | 133 | 134 | 135 | ## What is a schema? 136 | 137 | In Schemaboi, there are 3 parts of any schema: 138 | 139 | 1. The set of *data types*. This is a set of struct and enum types which together describe the data to be stored. These types will normally correspond to classes (/ interfaces) and enums in your application. 140 | 2. The *encoding* information. This describes the format your data will be actually encoded on disk. Different files storing the same data model may be encoded differently. Figuring out the encoding is normally taken care of by the schemaboi library. 141 | 3. The *language mapping* information. For example, the field `firstName` in the schema may be named `first_name` by your application. 142 | 143 | Every schemaboi file stores the set of data types and the encoding information. The language mapping is language specific, and stays within your application. 144 | 145 | 146 | ### The data model 147 | 148 | The data model supports the following types: 149 | 150 | - Primitive types (bool, u8-u128, s8-s128, float32, float64, string, byte array, ID) 151 | - Lists 152 | - Maps (list of Primitive => AnyType) 153 | - Structs (A set of fields) - AKA product types 154 | - Enums. Each variant can have associated fields. AKA sum types. 155 | 156 | 157 | #### Primitive types 158 | 159 | Booleans are either `true` or `false`. 160 | 161 | The u8, u16, u32, u64, u128 types represent unsigned integer types. And their friends s8, s16, s32, s64, s128 signed integers. u8 and s8 are stored using natural 1-byte encoding. The other integer types are encoded using a length-prefixed varint encoding, so small integers take up fewer bytes than larger integers. Variable length integer types can store up to 128 bit integers. (TODO: Consider making a bigint type?) 162 | 163 | Float32 and float64 store IEEE floating point numbers. We store the raw IEEE float bytes in the file. The bit pattern for NaN numbers may not be preserved by all formats. 164 | 165 | Strings are stored in UTF-8. All strings must be valid sequences of unicode codepoints. They are not null-terminated. (TODO: Can a string contain '\0'?) 166 | 167 | Byte arrays are opaque, and may contain any byte sequence including '\0'. They are essentially a tagged variant of strings which does not need to be valid unicode. 168 | 169 | IDs are essentially strings, but IDs are encoded such that if the ID appears multiple times in the data set, the ID will only be stored a single time in the file. Eg: `"G12FQXF", "G18LXIR"`. In some languages (eg Ruby), IDs may be serialized / deserialized using symbols. 170 | 171 | #### Lists and maps 172 | 173 | Lists contain an ordered sequence of items where every item has the same type. If you want a list where entries have different types, make a list of enum values. 174 | 175 | Maps are essentially lists of `(key, value)` entry pairs. The key can be any primitive type. A map is sort of similar to a struct (especially in Javascript), but the keys in a map are stored as data - ie, in the value of the map not the type (as is the case with structs). 176 | 177 | 178 | #### Structs 179 | 180 | A struct is a list of fields, and each field has a type. 181 | 182 | Fields can always be added to a data model. When a file is loaded, the known field set is merged with the list of fields the application knows about. 183 | 184 | Eg: 185 | 186 | ``` 187 | struct AddressBookEntry: { 188 | name: string, 189 | address: string, 190 | age: uint, 191 | } 192 | ``` 193 | 194 | At heart, every field in a struct is fundamentally optional in schemaboi. If you could add a required field to a struct in schemaboi, any old data (from before that field was added) would be unreadable. When your application loads data with missing fields, its up to your application to decide what happens. You can: 195 | 196 | 1. Consider the data to be invalid, and throw an error 197 | 2. Store the field as missing (`null` / `None` / etc depending on the language) 198 | 3. Use a default value 199 | 200 | 201 | #### Enums 202 | 203 | Enums in schemaboi are modelled after their namesake in [rust](https://doc.rust-lang.org/book/ch06-01-defining-an-enum.html) and [swift](https://docs.swift.org/swift-book/documentation/the-swift-programming-language/enumerations/). They are more powerful than their equivalent in C. 204 | 205 | An enum is fundamentally a set of *variants*. Each value of an enum type will store 1 of those variants. 206 | 207 | Variants can optionally have associated fields. For example, if we wanted to implement the CSS [color](https://developer.mozilla.org/en-US/docs/Web/CSS/color) type, we might do it like this: 208 | 209 | ``` 210 | enum CSSColor: { 211 | Red, 212 | Purple, 213 | Orange, 214 | // ... And other built-in CSS colors 215 | 216 | RGB { r: uint, g: uint, b: uint }, 217 | RGBA { r: uint, g: uint, b: uint, a: f32 }, // Could be combined with rgb. 218 | HSL { hue: f32, saturation: f32, luminance: f32 }, 219 | // HWB, etc. 220 | } 221 | ``` 222 | 223 | If you come from a C background, this is like using an enum and union pair. 224 | 225 | The associated fields of each enum variant are implemented as an inlined struct. If a variant has no associated fields, it is equivalent to having an empty struct, so you can always add associated fields to that variant later. 226 | 227 | Enums have a few extra knobs to make them easier to customize: 228 | 229 | - They can be *closed* or *open*. Closed enums can never have new variants added. This is useful when you know up-front all the variants an enum might store, like for a Result type. (This is equivalent to *exhaustive* enums in rust). When decoding an open enum, applications should also be programmed to handle an *unknown* variant type, which may show up for any variants that your application doesn't understand. 230 | 231 | - Enums can also be *numeric*, which forbids any associated data from being added to any variants of the enum type. 232 | -------------------------------------------------------------------------------- /bin/scbcat.ts: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env node 2 | 3 | // There is sooo much more work to do here! 4 | // Love to have code to: 5 | // - Generate typescript definitions from the schema 6 | // - Pretty print 7 | // - Print the app schema equivalent 8 | // - ... More! 9 | 10 | import * as schemaboi from '../lib/index.js' 11 | import fs from 'fs' 12 | 13 | const filename = process.argv[2] 14 | if (filename == null) { 15 | console.error('Usage: schcat .scb') 16 | } else { 17 | // console.log(filename) 18 | const bytes = fs.readFileSync(filename) 19 | 20 | const [schema, data] = schemaboi.readWithoutSchema(bytes) 21 | // console.log(JSON.stringify(schema, null, 2)) 22 | 23 | console.dir(data, {colors: true, depth: Infinity}) 24 | // console.log(JSON.stringify(data, null, 2)) 25 | } -------------------------------------------------------------------------------- /lib/extendschema.ts: -------------------------------------------------------------------------------- 1 | import { AppEnumSchema, AppSchema, AppStructField, AppStructSchema, EnumSchema, EnumVariant, Field, Primitive, SType, Schema } from "./schema.js" 2 | import { canonicalizeType, intEncoding, isInt, isPrimitive } from "./utils.js" 3 | 4 | const objMap = (obj: Record, mapFn: (a: A, key: string) => B): Record => { 5 | const result: Record = {} 6 | for (const k in obj) { 7 | result[k] = mapFn(obj[k], k) 8 | } 9 | return result 10 | } 11 | 12 | const objMapToMap = (obj: Record | Map, mapFn: (a: A, key: string) => B): Map => { 13 | const result: Map = new Map() 14 | if (obj instanceof Map) { 15 | for (const [k, v] of obj.entries()) { 16 | result.set(k, mapFn(v, k)) 17 | } 18 | } else { 19 | for (const k in obj) { 20 | result.set(k, mapFn(obj[k], k)) 21 | } 22 | } 23 | return result 24 | } 25 | 26 | type AndAny = T & Record 27 | const cloneType = (t: AndAny | Primitive | string): SType => ( 28 | typeof t === 'string' ? (isPrimitive(t) ? {type: t} : {type: 'ref', key: t}) 29 | : t.type === 'ref' ? {type: 'ref', key: t.key} 30 | : t.type === 'list' ? {type: 'list', fieldType: canonicalizeType(t.fieldType)} 31 | : t.type === 'map' ? { 32 | ...t, // type, encodeEntry and decodeEntry. 33 | type: 'map', 34 | keyType: canonicalizeType(t.keyType), 35 | valType: canonicalizeType(t.valType), 36 | decodeForm: t.decodeForm ?? 'object', 37 | } 38 | : isInt(t) ? { type: t.type, numericEncoding: intEncoding(t), decodeAsBigInt: t.decodeAsBigInt ?? false } 39 | : { type: t.type } 40 | ) 41 | 42 | // This function is a mess, but its only used in one place (below) and there its fine?? 43 | const getType = (t: SType | Primitive | string): string => ( 44 | typeof t === 'object' ? t.type : t 45 | ) 46 | 47 | function extendField(f: AppStructField): Field { 48 | // console.log('extendField', f, cloneType(f)) 49 | return { 50 | type: cloneType(f), 51 | defaultValue: f.defaultValue, 52 | inline: getType(f) === 'bool' ? true : false, // Inline booleans. 53 | optional: f.optional ?? false, 54 | skip: f.skip ?? false, 55 | // encoding: f.optional ? 'optional' : 'required', 56 | renameFieldTo: f.renameFieldTo, 57 | } 58 | } 59 | 60 | function structToEnumVariant(s: AppStructSchema): EnumVariant { 61 | return { 62 | encode: s.encode, 63 | decode: s.decode, 64 | fields: objMapToMap(s.fields, f => extendField(canonicalizeType(f))), 65 | } 66 | } 67 | 68 | function extendStruct(s: AppStructSchema): EnumSchema { 69 | return { 70 | foreign: false, 71 | exhaustive: false, 72 | numericOnly: false, 73 | 74 | localStructIsVariant: 'Default', // TODO: ?? 75 | 76 | variants: new Map([['Default', structToEnumVariant(s)]]), 77 | } 78 | } 79 | 80 | function extendEnum(s: AppEnumSchema): EnumSchema { 81 | const variants = Array.isArray(s.variants) ? new Map(s.variants.map((s): [string, EnumVariant] => [s, {}])) 82 | : objMapToMap(s.variants, (v): EnumVariant => ( 83 | (v == null || v === true) ? {} 84 | : structToEnumVariant(v) // 'fields' in v. 85 | )) 86 | 87 | return { 88 | exhaustive: s.exhaustive ?? false, 89 | numericOnly: s.numericOnly ?? false, 90 | typeFieldOnParent: s.typeFieldOnParent, 91 | encode: s.encode, 92 | decode: s.decode, 93 | variants, 94 | } 95 | } 96 | 97 | export function extendSchema(schema: AppSchema): Schema { 98 | return { 99 | id: schema.id, 100 | root: schema.root ? canonicalizeType(schema.root) : undefined, 101 | types: objMap(schema.types, s => ( 102 | s.type === 'enum' ? extendEnum(s) : extendStruct(s) 103 | )) 104 | } 105 | } 106 | 107 | -------------------------------------------------------------------------------- /lib/index.ts: -------------------------------------------------------------------------------- 1 | export * from './schema.js' 2 | export * from './read.js' 3 | export * from './write.js' 4 | export { 5 | ref, 6 | list, 7 | map, 8 | prim, 9 | String, 10 | Id, 11 | Bool, 12 | 13 | mergeSchemas, 14 | enumOfStrings, 15 | isInt, 16 | isPrimitive, 17 | primitiveTypes, 18 | } from './utils.js' 19 | export { extendSchema } from './extendschema.js' 20 | export {metaSchema} from './metaschema.js' 21 | // export {testSimpleRoundTrip} from './testhelpers.js' -------------------------------------------------------------------------------- /lib/metaschema.ts: -------------------------------------------------------------------------------- 1 | // The metaschema is a schema that is embedded in files to make schemaboi data self describing. 2 | 3 | import {EnumSchema, EnumVariant, IntPrimitive, MapType, Schema, Field, SType} from './schema.js' 4 | import { Bool, enumOfStringsEncoding, canonicalizeType, Id, intEncoding, ref, String, structSchema } from './utils.js' 5 | 6 | // import * as assert from 'assert/strict' 7 | // import * as fs from 'fs' 8 | // import {Console} from 'node:console' 9 | // const console = new Console({ 10 | // stdout: process.stdout, 11 | // stderr: process.stderr, 12 | // inspectOptions: {depth: null} 13 | // }) 14 | 15 | const mapOf = (valType: SType, decodeForm: 'object' | 'map' | 'entryList' = 'object'): MapType => ( 16 | {type: 'map', keyType: Id, valType, decodeForm} 17 | ) 18 | // const listOf = (fieldType: SType): List => ({type: 'list', fieldType}) 19 | 20 | 21 | // const structSchema: StructSchema = { 22 | // fields: new Map([ 23 | // // ['type', {type: 'string', defaultValue: 'struct', skip: true}], 24 | 25 | // ['foreign', { type: Bool, defaultValue: true, skip: true }], 26 | // ['fields', { type: mapOf(ref('StructField'), 'map'), optional: false }], 27 | // ]), 28 | // encode(obj: StructSchema) { 29 | // // console.log('encode helper', obj) 30 | // return { 31 | // ...obj, 32 | // fields: [...obj.fields.entries()].filter(([_k,v]) => !v.skip) 33 | // } 34 | // }, 35 | // } 36 | 37 | 38 | export const metaSchema: Schema = { 39 | id: '_sbmeta', 40 | root: ref('Schema'), 41 | 42 | types: { 43 | Schema: structSchema([ 44 | ['id', { type: String }], 45 | 46 | // Should this be optional or not? 47 | ['root', { type: ref('Type'), optional: true }], 48 | ['types', { type: mapOf(ref('TypeDef')) }], 49 | ]), 50 | 51 | NumberEncoding: enumOfStringsEncoding('le', 'varint'), 52 | 53 | Type: { 54 | // This has all the types in Primitive, and more! 55 | exhaustive: false, 56 | numericOnly: false, 57 | encode: canonicalizeType, // To support lazy strings. 58 | variants: new Map([ 59 | ...['bool', 'string', 'binary', 'id', 'f32', 'f64'].map((t): [string, EnumVariant] => [t, {}]), 60 | ...['u8', 'u16', 'u32', 'u64', 'u128', 's8', 's16', 's32', 's64', 's128'].map((t): [string, EnumVariant] => [t, { 61 | fields: new Map([ 62 | ['encoding', { 63 | type: ref('NumberEncoding'), 64 | renameFieldTo: 'numericEncoding', 65 | defaultValue: ((obj: IntPrimitive) => intEncoding(obj)), 66 | }] 67 | ]) 68 | }]), 69 | ['ref', { 70 | fields: new Map([['key', { type: Id }]]), 71 | }], 72 | ['list', { 73 | fields: new Map([['fieldType', { type: ref('Type') }]]), 74 | }], 75 | ['map', { 76 | fields: new Map([ 77 | ['keyType', { type: ref('Type') }], 78 | ['valType', { type: ref('Type') }], 79 | // Type should be enumOfStrings('object', 'map', 'entryList'), but it doesn't matter. 80 | ['decodeForm', { type: String, skip: true, defaultValue: 'map' }], 81 | ]), 82 | }], 83 | ]), 84 | }, 85 | 86 | TypeDef: structSchema([ 87 | ['foreign', { type: Bool, defaultValue: true, skip: true }], // Not stored. 88 | ['exhaustive', { type: Bool, inline: true }], 89 | ['numericOnly', { type: Bool, inline: true }], 90 | ['variants', { type: mapOf(ref('EnumVariant'), 'map') }], 91 | ]), 92 | 93 | EnumVariant: structSchema([ 94 | ['fields', { type: mapOf(ref('Field'), 'map'), optional: true, defaultValue: null }], 95 | ], { 96 | encode(obj: EnumVariant) { 97 | // Strip out any skipped fields from the wire. 98 | return { 99 | ...obj, 100 | fields: obj.fields ? [...obj.fields!.entries()].filter(([_k,v]) => !v.skip) : undefined 101 | } 102 | } 103 | }), 104 | 105 | Field: structSchema([ 106 | ['type', { type: ref('Type') }], 107 | 108 | // These fields are local only. 109 | ['defaultValue', {type: String, skip: true, optional: true}], // Type doesn't matter. 110 | ['foreign', { type: Bool, skip: true, defaultValue: true }], // Not stored. 111 | ['renameFieldTo', { type: String, skip: true, optional: true }], // Not stored. 112 | ['skip', { type: Bool, skip: true, defaultValue: false, inline: true }], // Should skip be skipped? If not we should inline this. 113 | 114 | ['inline', { type: Bool, inline: true, optional: true }], 115 | ['optional', { type: Bool, defaultValue: false, inline: true}], 116 | ]), 117 | } 118 | } 119 | -------------------------------------------------------------------------------- /lib/read.ts: -------------------------------------------------------------------------------- 1 | // import { Enum, Primitive, ref, Schema, Struct, SType } from "./schema.js"; 2 | 3 | import { EnumObject, EnumSchema, Schema, SType, Field, IntPrimitive, WrappedPrimitive, AppSchema, EnumVariant } from "./schema.js" 4 | import { bytesUsed, decode, decodeBN, zigzagDecode, zigzagDecodeBN } from "bijective-varint" 5 | import { trimBit } from "./utils.js" 6 | import { intEncoding, enumVariantsInUse, isPrimitive, canonicalizeType, mergeSchemas, fillSchemaDefaults, setEverythingLocal, ref, chooseRootType } from "./utils.js" 7 | import { extendSchema } from './extendschema.js' 8 | import { metaSchema } from "./metaschema.js" 9 | // import {Console} from 'node:console' 10 | // const console = new Console({ 11 | // stdout: process.stdout, 12 | // stderr: process.stderr, 13 | // inspectOptions: {depth: null} 14 | // }) 15 | 16 | 17 | interface Reader { 18 | pos: number, 19 | data: DataView, 20 | ids: string[], 21 | } 22 | 23 | function readVarInt(r: Reader): number { 24 | const buf = new Uint8Array(r.data.buffer, r.pos + r.data.byteOffset) 25 | r.pos += bytesUsed(buf) 26 | return decode(buf) 27 | } 28 | function readVarIntBN(r: Reader): bigint { 29 | const buf = new Uint8Array(r.data.buffer, r.pos + r.data.byteOffset) 30 | r.pos += bytesUsed(buf) 31 | return decodeBN(buf) 32 | } 33 | 34 | const textDecoder = new TextDecoder('utf-8') 35 | 36 | function readString(r: Reader): string { 37 | const len = readVarInt(r) 38 | // r.data. 39 | const base = r.data.byteOffset + r.pos 40 | const buf = r.data.buffer.slice(base, base+len) 41 | r.pos += len 42 | return textDecoder.decode(buf) 43 | } 44 | 45 | function readFields(r: Reader, schema: Schema, variant: EnumVariant): Record | null { 46 | // I'm still not sure what we should do in this case. We may still need the data! 47 | // 48 | // There are essentially 3 options: 49 | // 1. Skip the data, returning nothing. But when used in a load-then-save use case, 50 | // this will discard any foreign data. 51 | // 2. Parse the data but return it in a special way - eg {_foreign: {/* unknown fields */}} 52 | // 3. Return the array buffer containing the data, but don't parse it. 53 | // if (variant.foreign) { 54 | // console.error('in struct:', variant) 55 | // throw Error(`Foreign variant is not locally recognised! TODO - handle this better here`) 56 | // } 57 | 58 | if (variant.fields == null || variant.fields.size == 0) throw Error('readFields should not be called for empty variant') 59 | 60 | let bitPattern = 0 61 | let nextBit = 8 62 | const readNextBit = (): boolean => { 63 | if (nextBit >= 8) { 64 | // Read next byte 65 | bitPattern = r.data.getUint8(r.pos) 66 | r.pos++ 67 | nextBit = 0 68 | } 69 | 70 | // Bits are stored in LSB0 order. We read the bits from least to most significant in 71 | // each byte, then move on. 72 | // console.log('bits', bitPattern, 1 << nextBit) 73 | let bit = !!(bitPattern & (1 << nextBit)) 74 | nextBit++ 75 | return !!bit 76 | } 77 | 78 | // We still need to parse the fields, even if its not locally known to advance the read position. 79 | // const result: Record | null = struct.foreign ? null : {} 80 | const result: Record = {} 81 | const missingFields = new Set() 82 | 83 | const storeVal = (k: string, field: Field, v: any) => { 84 | // TODO: There's a sorta bug here: We haven't read all the other fields of the object. 85 | v ??= typeof field.defaultValue === 'function' 86 | ? field.defaultValue(result) 87 | : (field.defaultValue ?? null) 88 | if (field.foreign) { 89 | console.warn(`Warning: foreign field '${k}' in struct`) 90 | result._foreign ??= {} 91 | result._foreign[k] = v 92 | } else { 93 | result[field.renameFieldTo ?? k] = v 94 | } 95 | } 96 | 97 | // We read the data in 2 passes: First we read all the bits (booleans and optionals), then we read 98 | // the data itself. 99 | if (variant.fields != null) for (const [k, field] of variant.fields.entries()) { 100 | let hasValue = field.skip ? false 101 | : field.optional ? readNextBit() 102 | : true 103 | 104 | // console.log('read', k, hasValue) 105 | if (!hasValue) missingFields.add(k) 106 | 107 | // TODO: Consider also encoding enums with 2 in-use fields like this! 108 | if (field.inline && !field.skip) { 109 | if (field.type.type === 'bool') storeVal(k, field, hasValue ? readNextBit() : null) 110 | else throw Error('Cannot read inlined field of non-bool type') 111 | } 112 | } 113 | 114 | // console.log('missing fields', missingFields) 115 | 116 | // Now read the data itself. 117 | if (variant.fields != null) for (const [k, field] of variant.fields.entries()) { 118 | // We don't pass over skipped fields because we still need to fill them in with a specified default value. 119 | if (field.inline) continue // Inlined fields have already been read. 120 | 121 | const hasValue = !field.skip && !missingFields.has(k) 122 | const v = hasValue ? readThing(r, schema, field.type, result) : null 123 | storeVal(k, field, v) 124 | } 125 | 126 | return variant.decode ? variant.decode(result!) : result 127 | } 128 | 129 | function readEnum(r: Reader, schema: Schema, e: EnumSchema, parent?: any): EnumObject | any { 130 | const usedVariants = enumVariantsInUse(e) 131 | 132 | if (usedVariants.length == 0) throw Error('Cannot decode enum with no variants') 133 | 134 | // Enums with only 1 variant don't store their variant number at all. 135 | const variantNum = usedVariants.length === 1 ? 0 : readVarInt(r) 136 | 137 | if (variantNum >= usedVariants.length) throw Error('Could not look up variant ' + variantNum) 138 | 139 | const variantName = usedVariants[variantNum] 140 | // console.log('VV', variantNum, variantName) 141 | 142 | const variant = e.variants.get(variantName)! 143 | // console.log('READ variant', variant, schema) 144 | // Only decode the struct if the encoding names fields. 145 | 146 | const associatedData = variant.fields != null && variant.fields.size > 0 147 | ? readFields(r, schema, variant) 148 | : null 149 | 150 | // console.log('associated data', associatedData) 151 | 152 | // TODO: The logic for this feels kinda sketch. 153 | if (e.numericOnly && associatedData != null) throw Error('Cannot decode associated data with numeric enum') 154 | 155 | if (e.decode) { 156 | return e.decode(variantName, associatedData) 157 | } else if (e.localStructIsVariant != null) { 158 | if (variantName !== e.localStructIsVariant) throw Error('NYI - Bubble up foreign enum') 159 | return associatedData 160 | } else if (variant.foreign) { 161 | // The data isn't mapped to a local type. Encode it as {type: '_foreign', data: {...}}. 162 | 163 | // TODO: Only encode foreign variants if the enum allows us to do this! 164 | return {type: '_foreign', data: {type: variantName, ...associatedData}} 165 | } else if (e.typeFieldOnParent != null) { 166 | if (parent == null) throw Error('Cannot write type field on null parent') 167 | parent[e.typeFieldOnParent] = variantName 168 | return associatedData ?? {} 169 | } else if (!e.numericOnly) { 170 | return {type: variantName, ...associatedData} 171 | } else { 172 | // TODO: Make this configurable! Apps should never be surprised by this. 173 | return variantName 174 | } 175 | } 176 | 177 | function readNumeric(r: Reader, type: IntPrimitive): number | bigint { 178 | const encoding = intEncoding(type) 179 | const isSigned = type.type[0] === 's' 180 | 181 | if (encoding === 'varint') { 182 | if (type.decodeAsBigInt) { 183 | // We don't actually care what the inner type is. 184 | const n = readVarIntBN(r) 185 | return isSigned ? zigzagDecodeBN(n) : n 186 | } else { 187 | const n = readVarInt(r) 188 | return isSigned ? zigzagDecode(n) : n 189 | } 190 | } else { 191 | let n: number 192 | switch (type.type) { 193 | case 'u8': n = r.data.getUint8(r.pos++); break 194 | case 's8': n = r.data.getInt8(r.pos++); break 195 | default: throw Error('Not implemented: Little endian encoding for int type: ' + type.type) 196 | } 197 | return type.decodeAsBigInt ? BigInt(n) : n 198 | } 199 | } 200 | 201 | function readPrimitive(r: Reader, type: WrappedPrimitive | IntPrimitive): any { 202 | switch (type.type) { 203 | 204 | case 'u8': case 's8': 205 | case 'u16': case 'u32': case 'u64': case 'u128': 206 | case 's16': case 's32': case 's64': case 's128': 207 | return readNumeric(r, type) 208 | 209 | case 'f32': { 210 | const result = r.data.getFloat32(r.pos, true) 211 | r.pos += 4 212 | return result 213 | } 214 | case 'f64': { 215 | const result = r.data.getFloat64(r.pos, true) 216 | r.pos += 8 217 | return result 218 | } 219 | 220 | 221 | case 'bool': { 222 | const bit = r.data.getUint8(r.pos) !== 0 223 | r.pos++ 224 | return bit 225 | } 226 | case 'string': return readString(r) 227 | case 'binary': { 228 | const len = readVarInt(r) 229 | // r.data. 230 | const base = r.data.byteOffset + r.pos 231 | const buf = r.data.buffer.slice(base, base+len) 232 | r.pos += len 233 | return buf 234 | } 235 | case 'id': { 236 | // IDs are encoded as either a string or a number, depending on whether we've seen this ID before. 237 | let [seenBefore, n] = trimBit(readVarInt(r)) 238 | if (seenBefore) { 239 | // n stores the index of the string in the cached ID list. 240 | if (n > r.ids.length) throw Error('Invalid ID: Length exceeds seen IDs') 241 | return r.ids[n] 242 | } else { 243 | // The data model stores a string with length n. 244 | const base = r.data.byteOffset + r.pos 245 | const buf = r.data.buffer.slice(base, base+n) 246 | r.pos += n 247 | let val = textDecoder.decode(buf) 248 | r.ids.push(val) 249 | return val 250 | } 251 | 252 | 253 | } 254 | // default: throw Error('NYI readThing for ' + type) 255 | default: 256 | const expectNever: never = type 257 | } 258 | } 259 | 260 | function readThing(r: Reader, schema: Schema, type: SType, parent?: any): any { 261 | switch (type.type) { 262 | case 'ref': { 263 | const inner = schema.types[type.key] 264 | if (inner.foreign) throw Error('Cannot read foreign struct ' + type.key) 265 | return readEnum(r, schema, inner, parent) 266 | break 267 | } 268 | case 'list': { 269 | const length = readVarInt(r) 270 | // console.log('length', length) 271 | const result = [] 272 | for (let i = 0; i < length; i++) { 273 | result.push(readThing(r, schema, type.fieldType)) 274 | } 275 | return result 276 | } 277 | case 'map': { 278 | const length = readVarInt(r) 279 | const keyType = canonicalizeType(type.keyType) 280 | const valType = canonicalizeType(type.valType) 281 | if (type.decodeForm == null || type.decodeForm == 'object') { 282 | if (keyType.type !== 'string' && keyType.type !== 'id') throw Error('Cannot read map with non-string keys in javascript. Use Map decodeFrom.') 283 | const result: Record = {} 284 | for (let i = 0; i < length; i++) { 285 | let k = readPrimitive(r, keyType) 286 | let v = readThing(r, schema, valType) 287 | if (type.decodeEntry) [k, v] = type.decodeEntry([k, v]) 288 | result[k] = v 289 | } 290 | return result 291 | } else { 292 | const entries: [number | string | boolean, any][] = [] 293 | for (let i = 0; i < length; i++) { 294 | let k = readThing(r, schema, keyType) 295 | let v = readThing(r, schema, valType) 296 | if (type.decodeEntry) [k, v] = type.decodeEntry([k, v]) 297 | entries.push([k, v]) 298 | } 299 | return type.decodeForm == 'entryList' 300 | ? entries 301 | : new Map(entries) 302 | } 303 | } 304 | default: 305 | if (isPrimitive(type.type)) return readPrimitive(r, type) 306 | else throw Error(`Attempt to read unknown type: ${type.type}`) 307 | } 308 | } 309 | 310 | const createReader = (data: Uint8Array): Reader => ({ 311 | pos: 0, 312 | data: new DataView(data.buffer, data.byteOffset, data.byteLength), 313 | ids: ['Default'] 314 | }) 315 | 316 | /** 317 | * This is a low level method for reading data. It simply reads the incoming data 318 | * using the provided schema. 319 | */ 320 | export function readRaw(schema: Schema, data: Uint8Array, reqType?: string | SType): any { 321 | return readThing(createReader(data), schema, chooseRootType(schema, reqType)) 322 | } 323 | 324 | export function read(localSchema: Schema | null, data: Uint8Array, reqType?: string | SType): [Schema, any] { 325 | // A SB file starts with "SB10" for schemaboi version 1.0. 326 | const magic = textDecoder.decode(data.slice(0, 4)) 327 | if (magic !== 'SB11') throw Error('Magic bytes do not match: Expected SBXX.') 328 | 329 | const reader = createReader(data) 330 | reader.pos += 4 // Skip the magic bytes. 331 | 332 | // Read the schema. 333 | const remoteSchema: Schema = readThing(reader, metaSchema, metaSchema.root!) 334 | // console.log('rs', remoteSchema.types.Any.variants.get('int')) 335 | // console.log(remoteSchema) 336 | const mergedSchema = localSchema == null ? remoteSchema : mergeSchemas(remoteSchema, localSchema) 337 | if (localSchema == null) setEverythingLocal(mergedSchema) 338 | 339 | // Read the data. 340 | reader.ids.length = 0 341 | return [mergedSchema, readThing(reader, mergedSchema, chooseRootType(mergedSchema, reqType))] 342 | } 343 | 344 | export function readAppSchema(appSchema: AppSchema, data: Uint8Array): [Schema, any] { 345 | const localSchema = extendSchema(appSchema) 346 | return read(localSchema, data) 347 | } 348 | 349 | export function readWithoutSchema(data: Uint8Array): [Schema, any] { 350 | return read(null, data) 351 | } 352 | -------------------------------------------------------------------------------- /lib/schema.ts: -------------------------------------------------------------------------------- 1 | // export type Primitive = 'f32' | 'f64' | 'bool' | 'string' | 'binary' | 'id' 2 | // | 'u8' | 'u16' | 'u32' | 'u64' | 'u128' 3 | // | 's8' | 's16' | 's32' | 's64' | 's128' 4 | 5 | export type Ref = {type: 'ref', key: string} // Reference to another type in the type oracle 6 | export type List = {type: 'list', fieldType: SType} 7 | export interface MapType { // MapType rather than Map because Map is the name of a builtin type. 8 | type: 'map', 9 | keyType: SType, // TODO: Consider making key type default to 'string' here in JS land. 10 | valType: SType, 11 | // asEntryList?: true, 12 | decodeForm: 'object' | 'map' | 'entryList', // JS field. defaults to object. 13 | 14 | /** If provided, all entries are mapped through this function while being written */ 15 | encodeEntry?(entry: [any, any]): [any, any], 16 | /** If provided, all entries are mapped through this function while being decoded */ 17 | decodeEntry?(entry: [any, any]): [any, any], 18 | } 19 | export interface WrappedPrimitive { 20 | type: 'bool' | 'string' | 'binary' | 'id' | 'f32' | 'f64', 21 | } 22 | export interface IntPrimitive { 23 | type: 'u8' | 'u16' | 'u32' | 'u64' | 'u128' 24 | | 's8' | 's16' | 's32' | 's64' |'s128', 25 | 26 | // Encoding. If omitted, defaults to little endian for u8/s8 and varint for the rest. 27 | numericEncoding?: 'le' | 'varint', 28 | 29 | decodeAsBigInt?: boolean // JS encoding. 30 | } 31 | export type Primitive = (WrappedPrimitive | IntPrimitive)['type'] 32 | 33 | export type SType = WrappedPrimitive | IntPrimitive | Ref | List | MapType 34 | 35 | // export type MapEncoding = { 36 | // fromEntries: (entries: [any, any][]) => T, 37 | // toEntries: (data: T) => [any, any][], 38 | // } 39 | 40 | // export type EncodingStrategy = 'field' | 'optional field' | 'bits' 41 | 42 | export interface Field { 43 | type: SType, // Schema type 44 | 45 | /** 46 | * A default value. This is a JS encoding field. 47 | * 48 | * - When reading, if the field is missing in the stored data, we'll read this value instead. 49 | * This essentially forces the field to be required even if its optional or missing in the data. 50 | * - When writing a required field, the value in JS is allowed to be missing. In that case, we'll 51 | * write this value instead. We only do this if encoding is 'required'. 52 | */ 53 | defaultValue?: any | ((obj: any) => any), 54 | 55 | /** 56 | * JS: Is this field unknown to the local application? Foreign fields are deserialized in 57 | * {_foreign: {(fields)}}} 58 | */ 59 | foreign?: boolean, 60 | /** Map the external name of this field into a local (application) name */ 61 | renameFieldTo?: string, 62 | 63 | /** Encoding: Is this field be inlined into the bit fields? Currently only supported for booleans. */ 64 | inline?: boolean, 65 | 66 | 67 | /** 68 | * JS: Skip serializing and deserializing this field. When deserializing the field 69 | * will always be the default value (if specified) or null. 70 | */ 71 | skip?: boolean, 72 | 73 | /** 74 | * Encoding. Does this field exist in all serialized objects of this type? 75 | * Defaults to false - where the field must always be present. 76 | */ 77 | optional?: boolean, 78 | 79 | 80 | // encoding: 'unused' | 'optional' | 'required', // TODO: Maybe rename unused -> skipped? 81 | // used: boolean, // Or something. Encoding type: missing / optional / required ? 82 | 83 | // encodeMap?: MapEncoding 84 | } 85 | 86 | export interface EnumVariant { 87 | // renameFieldTo?: string, 88 | // associatedData?: StructSchema, 89 | 90 | /** JS: The variant is unknown to the local application. Defaults to false. */ 91 | foreign?: boolean, 92 | 93 | // JS: These methods, if provided, will be called before reading and after writing to prepare the object 94 | // for encoding. If used, the schema should express the data *at rest*. 95 | encode?: (obj: any) => Record, 96 | decode?: (obj: Record) => any, 97 | 98 | fields?: Map, 99 | } 100 | 101 | export interface EnumSchema { 102 | // type: 'enum', 103 | 104 | 105 | /** JS: The entire type is unknown to the local application. Defaults to false. (TODO: Is this used?? */ 106 | foreign?: boolean, 107 | 108 | /** 109 | * JS: The enum contains all variants that will ever exist, and it cannot be 110 | * extended. Exhaustive enums will error if you ever attempt to add more 111 | * variants via schema merging. 112 | * 113 | * Although the exhaustive flag could be considered a local only flag, its still 114 | * put in the schema file because its important that other applications know the 115 | * type will & must never be extended. 116 | * 117 | * TODO: Mark me as optional. 118 | */ 119 | exhaustive: boolean, 120 | 121 | /** Encoding: Enum variants do not contain fields - thus they are encoded using numbers. */ 122 | numericOnly: boolean, 123 | /** JS: The enum's variant name is on the parent object in the specified field */ 124 | typeFieldOnParent?: string, 125 | 126 | /** 127 | * JS: This is really a struct from the POV of the application. This is the (singleton) 128 | * enum variant in use. 129 | * 130 | * Note we could use the first variant. But if the enum is flattened to a struct 131 | * (rare), then they might pick one of the other variants to keep. 132 | */ 133 | localStructIsVariant?: string, 134 | 135 | encode?: (obj: any) => Record, 136 | decode?: (variant: string, data: Record | null) => any, 137 | 138 | /** The union of all known schema variants 139 | * 140 | * Note the order here matters. The JS spec enforces that the order of items in a Map will be stable. 141 | * We use this order when encoding items to assign them all an integer tag. 142 | * 143 | * We can think of the order of these items as an encoding specific matter. When we merge schemas, 144 | * the stored (remote) items *always* come before any local items which aren't in the remote schema. 145 | */ 146 | variants: Map, 147 | 148 | 149 | // usedVariants?: string[], // TODO: Consider caching this. 150 | 151 | // encodingOrder: string[], 152 | } 153 | 154 | // export type StructOrEnum = StructSchema & {type: 'struct'} | EnumSchema 155 | export interface Schema { 156 | id: string, 157 | root?: SType, // TODO: Make this optional for schemas with no obvious root type. 158 | types: Record 159 | } 160 | 161 | 162 | 163 | // ***************** 164 | 165 | /** 166 | * This is the stuff you need to define to make a type. It can be extended to a 167 | * full schema (with encoding information) via utility methods. 168 | */ 169 | export interface AppSchema { 170 | id: string, 171 | root?: SType | Primitive | string, // TODO: Consider making this optional. 172 | types: Record 173 | } 174 | 175 | // The type is inlined into the struct field to make things way simpler. 176 | export type AppStructField = SType & { 177 | /** If the field is missing in the data set, use this value instead of null when decoding. */ 178 | defaultValue?: any | ((obj: any) => any), 179 | skip?: boolean, 180 | optional?: boolean, 181 | renameFieldTo?: string, 182 | } 183 | 184 | export interface AppStructSchema { 185 | type?: 'struct', 186 | 187 | encode?: (obj: any) => Record, 188 | decode?: (obj: Record) => any, 189 | 190 | // Fields can be specified as complex objects, or simply strings. 191 | fields: Record, 192 | } 193 | 194 | export interface AppEnumSchema { 195 | type: 'enum', 196 | 197 | exhaustive?: boolean, 198 | numericOnly?: boolean, 199 | typeFieldOnParent?: string, 200 | 201 | encode?: (obj: any) => Record, 202 | decode?: (variant: string, data: Record | null) => any, 203 | 204 | variants: string[] | Record, // null or true are both ignored the same. 205 | } 206 | 207 | export type EnumObject = string 208 | | {type?: string, [k: string]: any} 209 | | {type: '_unknown', data: {type: string, [k: string]: any}} 210 | -------------------------------------------------------------------------------- /lib/testhelpers.ts: -------------------------------------------------------------------------------- 1 | import { AppSchema, SType, Schema, extendSchema, readRaw, writeRaw, read, write } from "./index.js" 2 | 3 | // We'll cache the extended schema for performance. 4 | const schemaCache = new WeakMap() 5 | 6 | 7 | type AnyType = null | undefined 8 | | boolean 9 | | string 10 | | number 11 | | AnyType[] 12 | | {[k: string]: AnyType} 13 | | Map 14 | | Set 15 | 16 | function simpleDeepEqual(expect: AnyType, actual: AnyType) { 17 | if (expect === actual) return true // Handle equality for bools, strings, numbers, null. 18 | // Handle inequality for the above. 19 | if (typeof expect !== 'object' || typeof actual !== 'object' || expect == null || actual == null) return false 20 | 21 | // Now we should just have objects, lists and maps left. 22 | if (Array.isArray(expect)) { 23 | if (!Array.isArray(actual) || expect.length !== actual.length) return false 24 | for (let i = 0; i < expect.length; i++) { 25 | if (!simpleDeepEqual(expect[i], actual[i])) return false 26 | } 27 | } else if (expect instanceof Map) { 28 | if (!(actual instanceof Map) || expect.size !== actual.size) return false 29 | 30 | // Its actually non-trivial to compare maps with non-primitive keys, because we need to match 31 | // them together. 32 | for (const [k, v1] of expect.entries()) { 33 | if (k != null && typeof k === 'object') throw Error('Deep equal for maps with object keys is NYI') 34 | const v2 = actual.get(k) 35 | if (!simpleDeepEqual(v1, v2)) return false 36 | } 37 | } else if (expect instanceof Set) { 38 | if (!(actual instanceof Set) || expect.size !== actual.size) return false 39 | 40 | if (expect.size > 0) throw Error('Non-empty sets not implemented') 41 | // for (const k of expect.entries()) { 42 | 43 | // } 44 | } else { 45 | // Its an object. At least I hope so. 46 | if (expect instanceof Set || actual instanceof Set) throw Error('Sets not implemented') 47 | if (Array.isArray(actual) || actual instanceof Map) return false 48 | 49 | if (Object.keys(expect).length !== Object.keys(actual).length) return false 50 | 51 | for (const k in expect) { 52 | const v1 = expect[k] 53 | const v2 = actual[k] 54 | if (!simpleDeepEqual(v1, v2)) return false 55 | } 56 | } 57 | return true 58 | } 59 | 60 | // assert(simpleDeepEqual(2, 2)) 61 | // assert(!simpleDeepEqual(2, 3)) 62 | // assert(!simpleDeepEqual(2, null)) 63 | // assert(simpleDeepEqual(null, null)) 64 | // assert(simpleDeepEqual(undefined, undefined)) 65 | // assert(simpleDeepEqual(true, true)) 66 | // assert(simpleDeepEqual('hi', 'hi')) 67 | 68 | // assert(simpleDeepEqual([1,2,3], [1,2,3])) 69 | // assert(!simpleDeepEqual([1,2,3,4], [1,2,3])) 70 | // assert(!simpleDeepEqual([1,2,3], [1,2,3,4])) 71 | 72 | // assert(simpleDeepEqual({x:'hi'}, {x:'hi'})) 73 | // assert(!simpleDeepEqual({x:'hi'}, {x:'hiu'})) 74 | // assert(!simpleDeepEqual({x:'hi'}, {x:'hi', y:'x'})) 75 | // assert(!simpleDeepEqual({x:'hi', y:'x'}, {x:'hi'})) 76 | 77 | // assert(simpleDeepEqual(new Map([['x', 'hi'], ['y', [4,5,6]]]), new Map([['x', 'hi'], ['y', [4,5,6]]]))) 78 | // assert(!simpleDeepEqual(new Map([['x', 'hi'], ['y', [4,5,6]]]), new Map([['x', 'hi'], ['y', [4,5,7]]]))) 79 | 80 | 81 | 82 | const assertDeepEqual = (expect: AnyType, actual: AnyType) => { 83 | if (!simpleDeepEqual(expect, actual)) { 84 | console.log('expected:', expect) 85 | console.log('actual: ', actual) 86 | throw Error('Input and output values did not match') 87 | } 88 | } 89 | 90 | /** This function is a simple smoke test to make sure an application's schema is set up correctly. */ 91 | export function testSimpleRoundTrip(appSchema: AppSchema, dataType: string | SType, data: any, expectedOutput = data) { 92 | let schema = schemaCache.get(appSchema) 93 | if (schema == null) { 94 | schema = extendSchema(appSchema) 95 | schemaCache.set(appSchema, schema) 96 | } 97 | 98 | const bytes = writeRaw(schema, data, dataType) 99 | const result = readRaw(schema, bytes, dataType) 100 | // console.log('result', result) 101 | 102 | // console.log('bytes', bytes.byteLength, 'JSON length', JSON.stringify(data).length, bytes) 103 | 104 | assertDeepEqual(expectedOutput, result) 105 | 106 | { 107 | const opaque = write(schema, data, dataType) 108 | 109 | // console.log('opaque', opaque) 110 | // fs.writeFileSync('tmp_test.sb', opaque) 111 | const [fileSchema, result] = read(schema, opaque, dataType) 112 | assertDeepEqual(expectedOutput, result) 113 | } 114 | } 115 | -------------------------------------------------------------------------------- /lib/utils.ts: -------------------------------------------------------------------------------- 1 | import { AppEnumSchema, EnumSchema, List, MapType, Ref, Schema, SType, Field, EnumVariant, IntPrimitive, WrappedPrimitive, Primitive } from "./schema.js" 2 | 3 | // import {Console} from 'node:console' 4 | // const console = new Console({ 5 | // stdout: process.stdout, 6 | // stderr: process.stderr, 7 | // inspectOptions: {depth: null} 8 | // }) 9 | 10 | export const assert = (a: boolean, msg?: string) => { 11 | if (!a) throw Error(msg ?? 'Assertion failed') 12 | } 13 | 14 | const mergeObjects = (a: Record, b: Record, mergeFn: (a: T, b: T) => T): Record => { 15 | const result: Record = {} 16 | for (const key of mergedObjKeys(a, b)) { 17 | const aa = a[key] 18 | const bb = b[key] 19 | 20 | // result[key] = takeOrMerge(aa, bb, mergeFn) 21 | result[key] = aa == null ? bb 22 | : bb == null ? aa 23 | : mergeFn(aa, bb) 24 | } 25 | 26 | return result 27 | } 28 | 29 | const mergeObjectsAll = (a: Record, b: Record, mergeFn: (a: T | null, b: T | null) => T): Record => { 30 | const result: Record = {} 31 | for (const key of mergedObjKeys(a, b)) { 32 | const aa = a[key] ?? null 33 | const bb = b[key] ?? null 34 | 35 | result[key] = mergeFn(aa, bb) 36 | } 37 | 38 | return result 39 | } 40 | 41 | const mergeMapsAll = (a: Map | null | undefined, b: Map | null | undefined, mergeFn: (a: T | null, b: T | null) => T): Map => { 42 | const result = new Map() 43 | for (const key of mergedMapKeys(a, b)) { 44 | const aa = a?.get(key) ?? null 45 | const bb = b?.get(key) ?? null 46 | 47 | result.set(key, mergeFn(aa, bb)) 48 | } 49 | 50 | return result 51 | } 52 | 53 | export const mergedObjKeys = (a: Record, b: Record): Iterable => ( 54 | new Set([...Object.keys(a), ...Object.keys(b)]) 55 | ) 56 | 57 | const emptyMap: Map = new Map() 58 | export const mergedMapKeys = (a: Map | null | undefined, b: Map | null | undefined): Iterable => ( 59 | // TODO: Remove this list allocation. 60 | new Set([...(a ?? emptyMap).keys(), ...(b ?? emptyMap).keys()]) 61 | ) 62 | 63 | function mergeFields(remote: Map | undefined, local: Map | undefined): Map { 64 | // console.log('merge', a, b) 65 | // Merge them. 66 | return mergeMapsAll(remote, local, (remoteF, localF): Field => { 67 | // Check the fields are compatible. 68 | // if (remoteF && localF && !typesShallowEq(remoteF.type, localF.type)) { 69 | // throw Error(`Incompatible types in struct field: '${remoteF.type.type}' != '${localF.type.type}'`) 70 | // } 71 | 72 | return { 73 | type: localF == null ? remoteF!.type 74 | : remoteF == null ? localF!.type 75 | : mergeTypes(remoteF.type, localF.type), 76 | defaultValue: localF?.defaultValue, 77 | foreign: localF ? (localF.foreign ?? false) : true, 78 | renameFieldTo: localF?.renameFieldTo, 79 | inline: remoteF ? (remoteF.inline ?? false) : (localF!.inline ?? false), 80 | 81 | // TODO: This makes sense for *reading* merged data, but it won't let us write. 82 | skip: remoteF ? (remoteF.skip ?? false) : true, 83 | optional: remoteF ? remoteF.optional : (localF!.optional ?? false), // If remoteF is null, this field doesn't matter. 84 | // encoding: remoteF?.encoding ?? 'unused', 85 | } 86 | }) 87 | } 88 | 89 | function mergeEnums(remote: EnumSchema, local: EnumSchema): EnumSchema { 90 | // I would use mergeObjects from above, but if one (or both) of the enums is exhaustive, we need to make sure 91 | // fields aren't added when they aren't valid. 92 | 93 | if (local.numericOnly && !remote.numericOnly) { 94 | // TODO: Not sure what to do in this case. 95 | console.warn("numericOnly does not match. Remote schema may include associated data.") 96 | } 97 | 98 | const result: EnumSchema = { 99 | foreign: local.foreign ?? false, 100 | exhaustive: remote.exhaustive || local.exhaustive, 101 | numericOnly: local.numericOnly, 102 | typeFieldOnParent: local.typeFieldOnParent, 103 | localStructIsVariant: local.localStructIsVariant, 104 | encode: local.encode ?? undefined, 105 | decode: local.decode ?? undefined, 106 | variants: new Map(), 107 | } 108 | 109 | for (const key of mergedMapKeys(remote.variants, local.variants)) { 110 | const remoteV = remote.variants.get(key) 111 | const localV = local.variants.get(key) 112 | 113 | if (remoteV == null && remote.exhaustive) throw Error('Cannot merge enums: Cannot add variant to exhaustive enum') 114 | if (localV == null && local.exhaustive) throw Error('Cannot merge enums: Cannot add variant to exhaustive enum') 115 | 116 | result.variants.set(key, remoteV == null ? { foreign: false, ...localV } 117 | : localV == null ? { ...remoteV, foreign: true } // TODO: Recursively set foreign flag in associated data. 118 | : { 119 | foreign: localV.foreign ?? false, 120 | encode: localV.encode, 121 | decode: localV.decode, 122 | fields: mergeFields(remoteV.fields, localV.fields), 123 | } 124 | ) 125 | } 126 | 127 | return result 128 | } 129 | 130 | 131 | function mergeTypes(remote: SType | string, local: SType | string): SType { 132 | remote = canonicalizeType(remote) 133 | local = canonicalizeType(local) 134 | 135 | if (!typesShallowEq(remote, local)) throw Error('Cannot merge disperate types') 136 | 137 | if (isInt(remote)) { 138 | return { 139 | type: remote.type, 140 | numericEncoding: intEncoding(remote), 141 | decodeAsBigInt: (local as IntPrimitive).decodeAsBigInt ?? false 142 | } 143 | } else if (remote.type === 'list') { 144 | return {type: 'list', fieldType: mergeTypes(remote.fieldType, (local as List).fieldType) } 145 | } else if (remote.type === 'map') { 146 | const l = local as MapType 147 | return { 148 | ...l, // type, encodeEntry and decodeEntry. 149 | keyType: mergeTypes(remote.keyType, l.keyType), 150 | valType: mergeTypes(remote.valType, l.valType), 151 | decodeForm: l.decodeForm ?? 'object', 152 | } 153 | } else { 154 | // For refs and simple primitives, we have nothing more to do here. 155 | return remote 156 | } 157 | } 158 | 159 | /** 160 | * Merge the schema found on disk with the local schema. 161 | * 162 | * This will use the encoding information from remote, and the JS mapping 163 | * information from the local schema. 164 | * 165 | * The set of types in the result will be the union of both. It is illegal 166 | * for a type or field with the same name to have different types. 167 | */ 168 | export function mergeSchemas(remote: Schema, local: Schema): Schema { 169 | if (remote.id != local.id) throw Error('Incompatible schemas') 170 | 171 | return { 172 | id: local.id, 173 | 174 | // Ehhh this is a bit of a mess. 175 | // - If they're both set, try to merge. 176 | // - If neither is set, leave it null. 177 | // Changing the default root type in a document isn't supported. 178 | root: (remote.root == null && local.root == null) ? undefined 179 | : (remote.root != null && local.root != null) ? mergeTypes(remote.root, local.root) 180 | : (remote.root ?? local.root), // Or take which ever is set. 181 | 182 | types: mergeObjectsAll(remote.types, local.types, (rem, loc): EnumSchema => { 183 | if (rem == null) return loc! 184 | if (loc == null) return { 185 | // TODO: This is a dangerous pattern. Are you sure all the fields are set right here? 186 | ...rem, 187 | foreign: true, 188 | } 189 | 190 | return mergeEnums(rem, loc) 191 | }) 192 | } 193 | } 194 | 195 | type AndAny = T & Record 196 | 197 | export const canonicalizeType = (t: AndAny | Primitive | string): SType => ( 198 | typeof t === 'object' 199 | ? t 200 | : (isPrimitive(t) ? {type: t} : {type: 'ref', key: t}) 201 | ) 202 | 203 | 204 | 205 | 206 | 207 | export const isRef = (x: SType): x is {type: 'ref', key: string} => ( 208 | typeof x !== 'string' && x.type === 'ref' 209 | ) 210 | 211 | export const typesShallowEq = (a: SType, b: SType): boolean => { 212 | if (a.type !== b.type) return false 213 | switch (a.type) { 214 | case 'ref': 215 | return a.key === (b as Ref).key 216 | case 'list': 217 | return typesShallowEq(canonicalizeType(a.fieldType), canonicalizeType((b as List).fieldType)) 218 | case 'map': 219 | const bb = b as MapType 220 | return typesShallowEq(canonicalizeType(a.keyType), canonicalizeType(bb.keyType)) 221 | && typesShallowEq(canonicalizeType(a.valType), canonicalizeType(bb.valType)) 222 | default: return true // They'd better be primitives! 223 | 224 | // Other cases (when added) will generate a type error. 225 | } 226 | } 227 | 228 | export const enumOfStrings = (...variants: string[]): AppEnumSchema => ({ 229 | type: 'enum', 230 | exhaustive: false, 231 | numericOnly: true, 232 | variants: Object.fromEntries(variants.map(v => [v, null])) 233 | }) 234 | 235 | export const enumOfStringsEncoding = (...variants: string[]): EnumSchema => ({ 236 | exhaustive: false, 237 | numericOnly: true, 238 | variants: new Map(variants.map(v => [v, {}])), 239 | // encodingOrder: variants, 240 | }) 241 | 242 | export function *filterIter(iter: IterableIterator, pred: (v: V) => boolean): IterableIterator { 243 | for (const v of iter) { 244 | if (pred(v)) yield v 245 | } 246 | } 247 | 248 | export function *mapIter(iter: IterableIterator, mapFn: (v: A) => B): IterableIterator { 249 | for (const a of iter) { 250 | yield mapFn(a) 251 | } 252 | } 253 | 254 | // export function countIter(iter: IterableIterator): number { 255 | // let count = 0 256 | // for (const _v of iter) count += 1 257 | // return count 258 | // } 259 | // export function countMatching(iter: IterableIterator, pred: (v: V) => boolean): number { 260 | // let count = 0 261 | // for (const v of iter) if (pred(v)) count += 1 262 | // return count 263 | // } 264 | 265 | // export function any(iter: IterableIterator, pred: (v: V) => boolean): boolean { 266 | // for (const v of iter) { 267 | // if (pred(v)) return true 268 | // } 269 | // return false 270 | // } 271 | 272 | // export function firstIndexOf(iter: IterableIterator, pred: (v: V) => boolean): number { 273 | // let i = 0; 274 | // for (const v of iter) { 275 | // if (pred(v)) return i 276 | // ++i 277 | // } 278 | // return -1 279 | // } 280 | 281 | // export const hasOptionalFields = (s: StructSchema): boolean => ( 282 | // // Could be more efficient, but eh. 283 | // any(s.fields.values(), v => v.encoding === 'optional') 284 | // ) 285 | 286 | // export const hasAssociatedData = (s: StructSchema | null | undefined): boolean => ( 287 | // s == null 288 | // ? false 289 | // : any(s.fields.values(), f => f.encoding !== "unused") 290 | // ) 291 | 292 | export const enumVariantsInUse = (e: EnumSchema): string[] => ( 293 | [...e.variants.keys()] 294 | // [... 295 | // mapIter( 296 | // filterIter(e.variants.entries(), ([_k, v]) => !v.skip), 297 | // ([k]) => k) 298 | // ] 299 | ) 300 | 301 | 302 | 303 | const fillSTypeDefaults = (t: SType) => { 304 | if (t.type === 'map') { 305 | t.decodeForm ??= 'object' 306 | t.valType = canonicalizeType(t.valType) 307 | fillSTypeDefaults(t.valType) 308 | } else if (t.type === 'list') { 309 | t.fieldType = canonicalizeType(t.fieldType) 310 | fillSTypeDefaults(t.fieldType) 311 | } else if (isInt(t)) { 312 | t.decodeAsBigInt ??= false 313 | t.numericEncoding ??= intEncoding(t) 314 | } 315 | } 316 | 317 | // const fillStructDefaults = (s: StructSchema, foreign: boolean) => { 318 | // s.foreign ??= foreign 319 | // s.encode ??= undefined 320 | // s.decode ??= undefined 321 | // for (const field of s.fields.values()) { 322 | // fillSTypeDefaults(field.type) 323 | // field.defaultValue ??= undefined // ?? 324 | // field.foreign ??= foreign 325 | // field.renameFieldTo ??= undefined // ??? 326 | // field.inline ??= false 327 | // field.skip ??= false 328 | // field.optional ??= false 329 | // } 330 | // } 331 | const fillFieldDefaults = (fields: Map, foreign: boolean) => { 332 | for (const field of fields.values()) { 333 | fillSTypeDefaults(field.type) 334 | field.defaultValue ??= undefined // ?? 335 | field.foreign ??= foreign 336 | field.renameFieldTo ??= undefined // ??? 337 | field.inline ??= false 338 | field.skip ??= false 339 | field.optional ??= false 340 | } 341 | } 342 | 343 | const fillEnumDefaults = (s: EnumSchema, foreign: boolean) => { 344 | s.foreign ??= foreign 345 | s.typeFieldOnParent ??= undefined 346 | s.encode ??= undefined 347 | s.decode ??= undefined 348 | s.localStructIsVariant ??= undefined 349 | s.numericOnly ??= false 350 | for (const variant of s.variants.values()) { 351 | variant.foreign ??= foreign 352 | // variant.skip ??= false 353 | variant.encode ??= undefined 354 | variant.decode ??= undefined 355 | 356 | variant.fields ??= new Map() 357 | fillFieldDefaults(variant.fields, foreign) 358 | } 359 | } 360 | 361 | /** Modifies the schema in-place! */ 362 | export function fillSchemaDefaults(s: Schema, foreign: boolean): Schema { 363 | if (s.root) fillSTypeDefaults(s.root) 364 | for (const k in s.types) { 365 | fillEnumDefaults(s.types[k], foreign) 366 | } 367 | return s 368 | } 369 | 370 | /** This is a dirty hack for when we want to use a file's schema to read it "raw" 371 | */ 372 | export function setEverythingLocal(s: Schema) { 373 | for (const k in s.types) { 374 | const t = s.types[k] 375 | t.foreign = false 376 | for (const v of t.variants.values()) { 377 | v.foreign = false 378 | if (v.fields) { 379 | for (const f of v.fields.values()) { 380 | f.foreign = false 381 | } 382 | } 383 | } 384 | } 385 | } 386 | 387 | export const primitiveTypes: Primitive[] = [ 388 | 'bool', 389 | 'u8', 'u16', 'u32', 'u64', 'u128', 390 | 's8', 's16', 's32', 's64', 's128', 391 | 'f32', 'f64', 392 | 'string', 'binary', 'id', 393 | ] 394 | 395 | export const isPrimitive = (s: string): s is Primitive => ( 396 | (primitiveTypes as string[]).indexOf(s) >= 0 397 | ) 398 | 399 | export const intTypes: IntPrimitive["type"][] = [ 400 | 'u8', 'u16', 'u32', 'u64', 'u128', 401 | 's8', 's16', 's32', 's64', 's128', 402 | ] 403 | 404 | export const isInt = (s: SType): s is IntPrimitive => ( 405 | (intTypes as string[]).indexOf(s.type) >= 0 406 | ) 407 | // export const isInt = (s: string): s is IntPrimitive["type"] => ( 408 | // (intTypes as string[]).indexOf(s) >= 0 409 | // ) 410 | 411 | 412 | 413 | export const prim = (primType: Primitive): WrappedPrimitive | IntPrimitive => { 414 | if (!isPrimitive(primType)) throw Error('prim() called with non-primitive type: ' + primType) 415 | return {type: primType} 416 | } 417 | 418 | export const String: SType = prim('string') 419 | export const Id: SType = prim('id') 420 | export const Bool: SType = prim('bool') 421 | 422 | export const ref = (key: string): Ref => ({type: 'ref', key}) 423 | export const list = (fieldType: SType | string): List => ( 424 | { type: 'list', fieldType: canonicalizeType(fieldType) } 425 | ) 426 | // export const map = (keyType: SType | string, valType: SType | string, decodeForm?: 'object' | 'map' | 'entryList'): MapType => ({ 427 | export const map = (keyType: SType | string, valType: SType | string, decodeForm: 'object' | 'map' | 'entryList'): MapType => ({ 428 | type: 'map', 429 | keyType: canonicalizeType(keyType), 430 | valType: canonicalizeType(valType), 431 | decodeForm, 432 | }) 433 | 434 | 435 | export const intEncoding = (num: IntPrimitive): 'le' | 'varint' => ( 436 | num.numericEncoding ?? ((num.type === 'u8' || num.type === 's8') ? 'le' : 'varint') 437 | ) 438 | 439 | 440 | export function structSchema(fields: [string, Field][], extras?: any): EnumSchema { 441 | return { 442 | exhaustive: false, 443 | numericOnly: false, 444 | localStructIsVariant: 'Default', 445 | variants: new Map([['Default', { 446 | foreign: false, 447 | fields: new Map(fields) 448 | }]]), 449 | ...extras, 450 | } 451 | } 452 | 453 | export const chooseRootType = (schema: Schema, reqType?: string | SType): SType => { 454 | let rootType = reqType ?? schema.root 455 | if (rootType == null) throw Error('Schema has no root type. You must specify which type from the schema you are loading') 456 | 457 | rootType = canonicalizeType(rootType) // If the requested root type is specified as a string, extend it. 458 | 459 | if (rootType.type === 'ref' && schema.types[rootType.key] == null) { 460 | throw Error(`Schema is missing requested type '${rootType.key}'`) 461 | } 462 | 463 | return rootType 464 | } 465 | 466 | export function mixBit(val: number, bit: boolean): number { 467 | return (val * 2) + (+bit) 468 | } 469 | 470 | export function trimBit(val: number): [boolean, number] { 471 | // I would use a bit shift for this division, but bit shifts coerce to a i32. 472 | return [!!(val % 2), Math.floor(val / 2)] 473 | } 474 | -------------------------------------------------------------------------------- /lib/write.ts: -------------------------------------------------------------------------------- 1 | import { MAX_BIGINT_LEN, MAX_INT_LEN, encodeInto, encodeIntoBN, zigzagEncode, zigzagEncodeBN } from "bijective-varint" 2 | import { mixBit } from "./utils.js" 3 | import { EnumObject, EnumSchema, IntPrimitive, Primitive, Schema, AppSchema, SType, WrappedPrimitive, EnumVariant } from "./schema.js" 4 | import { assert, chooseRootType, enumVariantsInUse, canonicalizeType, intEncoding, isPrimitive, ref } from "./utils.js" 5 | import { extendSchema } from './extendschema.js' 6 | import { metaSchema } from "./metaschema.js" 7 | 8 | // import assert from 'assert/strict' 9 | // import {Console} from 'node:console' 10 | // const console = new Console({ 11 | // stdout: process.stdout, 12 | // stderr: process.stderr, 13 | // inspectOptions: {depth: null} 14 | // }) 15 | 16 | interface WriteBuffer { 17 | buffer: Uint8Array, 18 | pos: number, 19 | ids: Map 20 | } 21 | 22 | const nextPowerOf2 = (v: number): number => { 23 | v-- 24 | v |= v >> 1 25 | v |= v >> 2 26 | v |= v >> 4 27 | v |= v >> 8 28 | v |= v >> 16 29 | return v + 1 30 | } 31 | 32 | const ensureCapacity = (b: WriteBuffer, amt: number) => { 33 | const capNeeded = b.pos + amt 34 | if (b.buffer.byteLength < capNeeded) { 35 | // Grow the array. 36 | let newLen = Math.max(nextPowerOf2(capNeeded), 64) 37 | const newBuffer = new Uint8Array(newLen) 38 | newBuffer.set(b.buffer) 39 | b.buffer = newBuffer 40 | } 41 | } 42 | 43 | const writeVarInt = (w: WriteBuffer, num: number) => { 44 | ensureCapacity(w, MAX_INT_LEN) 45 | w.pos += encodeInto(num, w.buffer, w.pos) 46 | } 47 | const writeVarIntBN = (w: WriteBuffer, num: bigint) => { 48 | ensureCapacity(w, MAX_BIGINT_LEN) 49 | w.pos += encodeIntoBN(num, w.buffer, w.pos) 50 | } 51 | 52 | const encoder = new TextEncoder() 53 | 54 | const writeString = (w: WriteBuffer, str: string) => { 55 | // This allocates, which isn't ideal. Could use encodeInto instead but doing it this way makes the 56 | // length prefix much easier to place. 57 | const strBytes = encoder.encode(str) 58 | ensureCapacity(w, MAX_INT_LEN + strBytes.length) 59 | w.pos += encodeInto(strBytes.length, w.buffer, w.pos) 60 | w.buffer.set(strBytes, w.pos) 61 | w.pos += strBytes.length 62 | } 63 | 64 | function checkPrimitiveType(val: any, type: Primitive) { 65 | // console.log('val', val, 'type', type) 66 | switch (type) { 67 | case 'u8': case 'u16': case 'u32': case 'u64': case 'u128': 68 | case 's8': case 's16': case 's32': case 's64': case 's128': 69 | case 'f32': case 'f64': 70 | assert(typeof val === 'number' || typeof val === 'bigint', 'Expected number. Got ' + typeof val); 71 | break 72 | case 'bool': assert(typeof val === 'boolean', 'Expected bool. Got ' + typeof val); break 73 | case 'string': case 'id': assert(typeof val === 'string', 'Expected string. Got ' + typeof val); break 74 | case 'binary': assert(val instanceof Uint8Array, 'Expected number. Got ' + typeof val); break // TODO: Allow more binary types. 75 | default: let unused: never = type; throw Error(`Expected primitive type. Got: ${type}`) 76 | } 77 | } 78 | 79 | // Using max+1 because JS numbers (doubles) can accurately store powers of 2. 80 | const maxPlus1Num = { 81 | u8: 256, 82 | u16: 2**16, 83 | u32: 2**32, 84 | u64: 2**64, 85 | u128: 2**128, 86 | 87 | s8: 128, 88 | s16: 2**15, 89 | s32: 2**31, 90 | s64: 2**63, 91 | s128: 2**127, 92 | } 93 | 94 | function writeInt(w: WriteBuffer, val: number | bigint, type: IntPrimitive) { 95 | const encoding = intEncoding(type) 96 | if (encoding === 'le') { 97 | if (type.type !== 'u8' && type.type !== 's8') throw Error('NYI: Little endian encoding for numberic ' + type.type) 98 | w.buffer[w.pos++] = Number(val) 99 | } else { 100 | const isSigned = type.type[0] === 's' 101 | // console.log('writing', val, 'type', typeof val, 'signed', isSigned, 'x', isSigned ? zigzagEncodeBN(BigInt(val)) : val) 102 | 103 | if (val >= maxPlus1Num[type.type]) throw Error(`Number ${val} too big for container size ${type.type}`) 104 | // The test is < not <= because 2s compliment supports from [-2^n .. 2^n-1] 105 | if (isSigned && val < -maxPlus1Num[type.type]) throw Error(`Number ${val} too negative for container size ${type.type}`) 106 | if (!isSigned && val < 0) throw Error(`Negative number ${val} cannot be stored with unsized type ${type.type}`) 107 | 108 | // Writing a varint. 109 | if (typeof val === 'bigint') { 110 | writeVarIntBN(w, isSigned ? zigzagEncodeBN(val) : val) 111 | } else if (typeof val === 'number') { 112 | writeVarInt(w, isSigned ? zigzagEncode(val) : val) 113 | } else throw Error('Cannot encode type as a number') 114 | } 115 | } 116 | 117 | /** 118 | * Each struct is stored in 2 blocks of data: 119 | * 120 | * - Bits: All fields which only have 2 states. Eg, booleans, enums with 2 variants. Also, all optional flags. 121 | * - Bytes: All fields which don't fit in a bit. 122 | * 123 | * All the bits are packed together at the start of the struct definition. 124 | * 125 | * When we encode, we do it in 2 passes: 126 | * 127 | * 1. Scan the struct and encode all bit fields. 128 | * 2. Scan the struct and encode everything else. 129 | * 130 | * Note some fields may not need *any* bits to store them (eg enums with only 1 variant). 131 | */ 132 | function encodeFields(w: WriteBuffer, schema: Schema, val: any, variant: EnumVariant) { 133 | // console.log('encodeFields', variant, val) 134 | if (variant.encode) val = variant.encode(val) 135 | 136 | if (typeof val !== 'object' || Array.isArray(val) || val == null) throw Error('Expected struct with fields. Got: ' + JSON.stringify(val)) 137 | 138 | // let encodingBits = true 139 | 140 | // Bits are stored in LSB0. 141 | let bitPattern = 0 // only 8 bits are used, then we flush. 142 | let nextBit = 0 143 | 144 | const flushBits = () => { 145 | if (nextBit > 0) { // Flush if there are any bits set. 146 | // console.log('flushing bits', bitPattern, 'bits:', nextBit) 147 | ensureCapacity(w, 1) 148 | w.buffer[w.pos] = bitPattern 149 | w.pos += 1 150 | bitPattern = 0 151 | nextBit = 0 152 | } 153 | } 154 | 155 | const writeBit = (b: boolean) => { 156 | // console.log('writeBit', b) 157 | if (nextBit >= 8) flushBits() 158 | bitPattern |= (b ? 1 : 0) << nextBit 159 | nextBit++ 160 | } 161 | 162 | // First write the bit block. 163 | const writePass = (writeBit: ((b: boolean) => void) | null, writeThing: ((v: any, type: SType) => void) | null) => { 164 | if (variant.fields) for (const [k, field] of variant.fields.entries()) { 165 | if (field.skip) continue 166 | 167 | const fieldName = field.renameFieldTo ?? k 168 | let v = val[fieldName] 169 | 170 | // External fields always use the raw field name. 171 | if (field.foreign && v === undefined && val._foreign) v = val._foreign[k] 172 | 173 | // console.log('field', k, 'inline', field.inline, 'encoding', field.type, 'hasValue', v != null, 'value', v) 174 | if (field.optional) { 175 | const hasValue = v != null 176 | writeBit?.(hasValue) 177 | if (!hasValue) continue 178 | } else { 179 | // If the field is missing, fill it in with the default value. 180 | if (v == null && field.defaultValue != null) { 181 | v = typeof field.defaultValue === 'function' 182 | ? field.defaultValue(val) 183 | : field.defaultValue 184 | } 185 | if (v == null) throw Error(`null or missing field '${fieldName}' required by encoding`) 186 | } 187 | 188 | // TODO: Also write bits for enums with 2 in-use fields! 189 | if (field.inline) { 190 | // console.log('write inlined', k) 191 | if (field.type.type === 'bool') writeBit?.(v) 192 | else throw Error('Inlining non-boolean fields not supported') 193 | } else { 194 | writeThing?.(v, field.type) 195 | } 196 | } 197 | } 198 | 199 | writePass(writeBit, null) 200 | // console.log('bits', bitPattern, bitsUsed) 201 | flushBits() 202 | // console.log(w.buffer, w.pos) 203 | writePass(null, (v: any, type: SType) => { 204 | encodeThing(w, schema, v, type, val) 205 | }) 206 | } 207 | 208 | // const enumIsEmpty = (obj: EnumObject): boolean => { 209 | // if (typeof obj === 'string') return true 210 | // for (const k in obj) { 211 | // if (k !== 'type') return false 212 | // } 213 | // return true 214 | // } 215 | 216 | // For now I'm just assuming (requiring) a {type: 'variant', ...} shaped object, or a "variant" with no associated data 217 | function encodeEnum(w: WriteBuffer, schema: Schema, val: EnumObject | any, e: EnumSchema, parent?: any) { 218 | // const usedVariants = (e.usedVariants ??= enumUsedStates(e)) 219 | const usedVariants = enumVariantsInUse(e) 220 | 221 | // We need to write 2 pieces of data: 222 | // 1. Which variant we're encoding 223 | // 2. Any associated data for this variant 224 | 225 | if (e.encode) val = e.encode(val) 226 | 227 | const variantName = e.localStructIsVariant != null ? e.localStructIsVariant 228 | : typeof val === 'string' ? val 229 | : e.typeFieldOnParent != null ? parent[e.typeFieldOnParent] 230 | : val.type === '_foreign' ? val.data.type 231 | : val.type 232 | 233 | if (typeof variantName != 'string') { 234 | console.error('When encoding val:', val) 235 | throw Error('Invalid enum variant name: ' + variantName) 236 | } 237 | 238 | const associatedData = e.localStructIsVariant != null ? val 239 | : typeof val === 'string' ? {} 240 | : val.type === '_foreign' ? val.data 241 | : val 242 | 243 | const variant = e.variants.get(variantName) 244 | // console.log('WRITE variant', variantName, variant) 245 | if (variant == null) throw Error(`Unrecognised enum variant: "${variantName}"`) 246 | 247 | // console.log('encodeEnum', e, variantName, 'fields', variant.fields, 'val', val) 248 | 249 | if (usedVariants.length >= 2) { 250 | const variantNum = usedVariants.indexOf(variantName) 251 | if (variantNum < 0) throw Error(`No encoding for ${variantName}`) 252 | writeVarInt(w, variantNum) 253 | } 254 | 255 | if (variant.fields) { 256 | // console.log('Encode associated data') 257 | encodeFields(w, schema, associatedData, variant) 258 | } 259 | } 260 | 261 | function encodeThing(w: WriteBuffer, schema: Schema, val: any, type: SType, parent?: any) { 262 | // console.log('encodething', 'pos', w.pos, 'type', type, 'val', val) 263 | switch (type.type) { 264 | case 'ref': { 265 | const innerType = schema.types[type.key] 266 | if (innerType == null) throw Error(`Schema contains a ref to missing type '${type.key}'`) 267 | encodeEnum(w, schema, val, innerType, parent) 268 | return 269 | } 270 | case 'list': { 271 | if (!Array.isArray(val)) throw Error('Cannot encode item as list') 272 | writeVarInt(w, val.length) 273 | // TODO: Consider special-casing bit arrays. 274 | // const fieldType = extendType(type.fieldType) 275 | for (const v of val) { 276 | encodeThing(w, schema, v, type.fieldType) 277 | } 278 | return 279 | } 280 | case 'map': { 281 | // Maps can also be provided as a list of [k,v] entries. 282 | const entries = Array.isArray(val) ? val 283 | : val instanceof Map ? Array.from(val.entries()) // TODO: Remove this allocation. 284 | : Object.entries(val) 285 | writeVarInt(w, entries.length) 286 | const keyType = canonicalizeType(type.keyType) 287 | const valType = canonicalizeType(type.valType) 288 | for (let entry of entries) { 289 | if (type.encodeEntry) entry = type.encodeEntry(entry) 290 | encodeThing(w, schema, entry[0], keyType) 291 | encodeThing(w, schema, entry[1], valType) 292 | } 293 | return 294 | } 295 | } 296 | 297 | // Fall through to processing primitives. (Everything else is a primitive type) 298 | checkPrimitiveType(val, type.type) 299 | 300 | switch (type.type) { 301 | case 'bool': { 302 | ensureCapacity(w, 1) 303 | w.buffer[w.pos++] = val 304 | return 305 | } 306 | 307 | case 'f32': { 308 | ensureCapacity(w, 4) 309 | 310 | // f32 values are stored natively as 4 byte IEEE floats. It'd be nice 311 | // to just write directly to the buffer, but unaligned writes aren't 312 | // supported by Float32Array. 313 | const dataView = new DataView(w.buffer.buffer, w.buffer.byteOffset + w.pos, 4) 314 | // Coerce bigint -> number. 315 | dataView.setFloat32(0, typeof val === 'number' ? val : Number(val), true) 316 | w.pos += 4 317 | return 318 | } 319 | case 'f64': { 320 | ensureCapacity(w, 8) 321 | 322 | const dataView = new DataView(w.buffer.buffer, w.buffer.byteOffset + w.pos, 8) 323 | dataView.setFloat64(0, typeof val === 'number' ? val : Number(val), true) 324 | w.pos += 8 325 | return 326 | } 327 | 328 | case 'u8': case 's8': 329 | case 'u16': case 'u32': case 'u64': case 'u128': 330 | case 's16': case 's32': case 's64': case 's128': 331 | writeInt(w, val, type); return; 332 | 333 | case 'string': { 334 | writeString(w, val) 335 | return 336 | } 337 | 338 | case 'binary': { 339 | let valBuffer = val as Uint8Array 340 | 341 | ensureCapacity(w, 9 + valBuffer.byteLength) 342 | w.pos += encodeInto(valBuffer.byteLength, w.buffer, w.pos) 343 | w.buffer.set(valBuffer, w.pos) 344 | w.pos += valBuffer.byteLength 345 | return 346 | } 347 | 348 | case 'id': { 349 | // IDs are encoded as either a string or a number, depending on whether we've seen this ID before. 350 | const existingId = w.ids.get(val) 351 | if (existingId == null) { 352 | // Encode it as a string, but with an extra 0 bit mixed into the length. 353 | // This code is lifted from writeString(). It'd be nice to share this code, but .. that'd be gross too. 354 | const strBytes = encoder.encode(val) 355 | ensureCapacity(w, 9 + strBytes.length) 356 | let n = mixBit(strBytes.length, false) 357 | w.pos += encodeInto(n, w.buffer, w.pos) 358 | w.buffer.set(strBytes, w.pos) 359 | w.pos += strBytes.length 360 | 361 | let id = w.ids.size 362 | w.ids.set(val, id) 363 | } else { 364 | let n = mixBit(existingId, true) 365 | writeVarInt(w, n) 366 | } 367 | return 368 | } 369 | 370 | default: 371 | let exhaustiveCheck: never = type 372 | throw Error('Unknown type: ' + type) 373 | } 374 | // Unreachable. 375 | } 376 | 377 | const createWriteBuf = (buffer: Uint8Array = new Uint8Array(32), pos: number = 0): WriteBuffer => ({ 378 | buffer, pos, ids: new Map([['Default', 0]]) 379 | }) 380 | 381 | const consumeWriteBuf = (writer: WriteBuffer): Uint8Array => writer.buffer.slice(0, writer.pos) 382 | 383 | export function writeRawInto(schema: Schema, data: any, buffer: Uint8Array | undefined, pos: number = 0, ofType?: string | SType): Uint8Array { 384 | const writer = createWriteBuf(buffer, pos) 385 | 386 | encodeThing(writer, schema, data, chooseRootType(schema, ofType)) 387 | 388 | return consumeWriteBuf(writer) 389 | } 390 | 391 | export function writeRaw(schema: Schema, data: any, ofType?: string | SType): Uint8Array { 392 | return writeRawInto(schema, data, undefined, 0, ofType) 393 | } 394 | 395 | /** 396 | * Write the given data into the given bufffer. Note the returned buffer may differ 397 | * (due to running out of space). So make sure you use the returned Uint8Array!! 398 | */ 399 | export function writeInto(schema: Schema, data: any, buffer: Uint8Array | undefined, pos: number = 0, ofType?: string | SType): Uint8Array { 400 | const writer = createWriteBuf(buffer, pos) 401 | const magicBytes = encoder.encode("SB11") 402 | writer.buffer.set(magicBytes, writer.pos) 403 | writer.pos += 4 404 | 405 | // console.log(schema) 406 | encodeThing(writer, metaSchema, schema, metaSchema.root!) 407 | writer.ids.clear() 408 | encodeThing(writer, schema, data, chooseRootType(schema, ofType)) 409 | 410 | return consumeWriteBuf(writer) 411 | } 412 | 413 | export function write(schema: Schema, data: any, ofType?: string | SType): Uint8Array { 414 | return writeInto(schema, data, undefined, 0, ofType) 415 | } 416 | 417 | export function writeAppSchema(schema: AppSchema, data: any): Uint8Array { 418 | return write(extendSchema(schema), data) 419 | } 420 | 421 | // Is it worth having this method at all? 422 | export function writeLocalSchema(schema: Schema, buffer?: Uint8Array, pos?: number): Uint8Array { 423 | return writeRawInto(metaSchema, schema, buffer, pos) 424 | } 425 | -------------------------------------------------------------------------------- /package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "schemaboi", 3 | "version": "0.3.0", 4 | "description": "Binary serialization library with long-now schema migration and app support", 5 | "exports": { 6 | ".": "./dist/lib/index.js", 7 | "./testhelpers.js": { 8 | "default": "./dist/lib/testhelpers.js", 9 | "types": "./dist/lib/testhelpers.d.ts" 10 | } 11 | }, 12 | "type": "module", 13 | "types": "dist/lib/index.d.ts", 14 | "repository": "https://github.com/josephg/schemaboi.git", 15 | "author": "Seph Gentle ", 16 | "license": "ISC", 17 | "scripts": { 18 | "test": "npx tsc && mocha dist/test/", 19 | "prepare": "rm -rf dist && tsc -p ." 20 | }, 21 | "dependencies": { 22 | "bijective-varint": "^1.1.0" 23 | }, 24 | "devDependencies": { 25 | "@types/mocha": "^10.0.1", 26 | "@types/node": "^18.11.19", 27 | "mocha": "^10.2.0", 28 | "typescript": "^5.2.2" 29 | }, 30 | "bin": { 31 | "scbcat": "dist/bin/scbcat.js" 32 | }, 33 | "files": [ 34 | "dist/lib/*", 35 | "lib/*", 36 | "bin/*" 37 | ] 38 | } 39 | -------------------------------------------------------------------------------- /test/extend.ts: -------------------------------------------------------------------------------- 1 | // Test that we can extend schemas 2 | import 'mocha' 3 | import { AppSchema } from '../lib/schema.js' 4 | import { Bool, enumOfStrings, prim, ref, String } from '../lib/utils.js' 5 | import { extendSchema } from '../lib/extendschema.js' 6 | 7 | import {Console} from 'node:console' 8 | const console = new Console({ 9 | stdout: process.stdout, 10 | stderr: process.stderr, 11 | inspectOptions: {depth: null} 12 | }) 13 | 14 | describe('extend', () => { 15 | it('simple test', () => { 16 | const schema: AppSchema = { 17 | id: 'Example', 18 | root: ref('Contact'), 19 | types: { 20 | Contact: { 21 | fields: { 22 | name: 'string', 23 | address: 'string', 24 | coolness: 'bool', 25 | } 26 | }, 27 | 28 | Shape: { 29 | type: 'enum', 30 | numericOnly: false, 31 | variants: { 32 | Line: null, 33 | Square: { 34 | fields: { x: {type: 'f32'}, y: 'f32'} 35 | } 36 | } 37 | }, 38 | 39 | Color: enumOfStrings('Green', 'Red', 'Purple') 40 | } 41 | } 42 | 43 | // console.log('encoding', simpleSchemaEncoding(schema)) 44 | // console.log('js', simpleJsMap(schema)) 45 | 46 | 47 | // TODO: We're not actually checking that this schema makes any sense! 48 | // console.log(extendSchema(schema)) 49 | }) 50 | }) -------------------------------------------------------------------------------- /test/json.ts: -------------------------------------------------------------------------------- 1 | import 'mocha' 2 | import { AppSchema, extendSchema, ref, list, map } from '../lib/index.js' 3 | import * as assert from 'assert/strict' 4 | import { testSimpleRoundTrip } from '../lib/testhelpers.js' 5 | 6 | // The encoder needs the data to be in a certain shape to correctly encode and decode enums. 7 | type JSONValueEnc = {type: 'null' | 'true' | 'false'} 8 | | {type: 'float', val: number} 9 | | {type: 'int', val: number} 10 | | {type: 'string', val: string} 11 | | {type: 'object', val: Record} 12 | | {type: 'list', val: JSONValueEnc[]} 13 | 14 | const errExpr = (msg: string): any => {throw Error(msg)} 15 | 16 | function encode(val: any): JSONValueEnc { 17 | return val == null ? {type: 'null'} 18 | : val === true ? {type: 'true'} 19 | : val === false ? {type: 'false'} 20 | : typeof val === 'string' ? {type: 'string', val} 21 | : typeof val === 'number' ? (Number.isInteger(val) ? {type: 'int', val} : {type: 'float', val}) 22 | : Array.isArray(val) ? {type: 'list', val} 23 | // : Array.isArray(obj) ? {type: 'list', val: obj.map(encode)} 24 | : typeof val === 'object' ? {type: 'object', val} 25 | // : typeof val === 'object' ? {type: 'object', val: objMap(val, encode)} 26 | : errExpr('Not recognised value: ' + val) 27 | } 28 | 29 | function decode(_variant: string, val: Record | null): any { 30 | const variant = _variant as JSONValueEnc['type'] 31 | 32 | // console.log('decode', variant, val) 33 | 34 | switch (variant) { 35 | case 'null': return null 36 | case 'true': return true 37 | case 'false': return false 38 | case 'float': case 'int': case 'string': 39 | case 'list': case 'object': 40 | return val!.val 41 | default: 42 | let expectNever: never = variant 43 | throw Error('unexpected type: ' + variant) 44 | } 45 | } 46 | 47 | const json: AppSchema = { 48 | id: 'json', 49 | root: 'Any', 50 | types: { 51 | Any: { 52 | type: 'enum', 53 | exhaustive: true, 54 | encode, 55 | decode, 56 | variants: { 57 | null: null, 58 | true: null, 59 | false: null, 60 | string: {fields: {val: 'string'}}, 61 | int: {fields: {val: 's64'}}, 62 | float: {fields: {val: 'f64'}}, 63 | object: {fields: {val: map('string', 'Any', 'object')}}, 64 | list: {fields: {val: list('Any')}}, 65 | } 66 | } 67 | } 68 | } 69 | // const fullSchema = extendSchema(json) 70 | 71 | // console.log(fullSchema.types['Any'].variants) 72 | 73 | describe('json encoding', () => { 74 | it('can encode and decode simple values', () => { 75 | testSimpleRoundTrip(json, 'Any', 60) 76 | testSimpleRoundTrip(json, 'Any', true) 77 | testSimpleRoundTrip(json, 'Any', false) 78 | testSimpleRoundTrip(json, 'Any', null) 79 | testSimpleRoundTrip(json, 'Any', "hi") 80 | testSimpleRoundTrip(json, 'Any', ["hi"]) 81 | testSimpleRoundTrip(json, 'Any', {hi: true}) 82 | testSimpleRoundTrip(json, 'Any', [{hi: true}]) 83 | testSimpleRoundTrip(json, 'Any', [[[]]]) 84 | testSimpleRoundTrip(json, 'Any', {x:{y:{z:{}}}}) 85 | }) 86 | }) 87 | -------------------------------------------------------------------------------- /test/merging.ts: -------------------------------------------------------------------------------- 1 | import * as assert from 'assert/strict' 2 | import { EnumSchema, EnumVariant, Schema, Field } from '../lib/schema.js' 3 | import { enumOfStringsEncoding, enumOfStrings, fillSchemaDefaults, mergeSchemas, prim, ref, String, structSchema } from '../lib/utils.js' 4 | import { extendSchema } from '../lib/extendschema.js' 5 | import { write } from '../lib/write.js' 6 | import { read } from '../lib/read.js' 7 | // import {Console} from 'node:console' 8 | // const console = new Console({ 9 | // stdout: process.stdout, 10 | // stderr: process.stderr, 11 | // inspectOptions: {depth: null} 12 | // }) 13 | 14 | describe('merging', () => { 15 | describe('exhaustive enums', () => { 16 | const makeS = (color: string, exhaustive?: boolean): Schema => (extendSchema({ 17 | id: 'Example', 18 | root: ref('Color'), 19 | types: { 20 | Color: { 21 | type: 'enum', 22 | numericOnly: true, 23 | exhaustive: exhaustive, 24 | variants: { [color]: true, } 25 | } 26 | } 27 | })) 28 | 29 | it('merges if it can', () => { 30 | const schemaA: Schema = makeS('Red') 31 | const schemaB: Schema = makeS('Blue', false) 32 | 33 | const merged = mergeSchemas(schemaA, schemaB) 34 | 35 | const color = merged.types['Color'] as EnumSchema 36 | assert.equal(color.variants.size, 2) 37 | assert.equal(color.exhaustive, false) 38 | }) 39 | 40 | it('throws if one of them is exhaustive', () => { 41 | const schemaA: Schema = makeS('Red', true) 42 | const schemaB: Schema = makeS('Blue', false) 43 | assert.throws(() => { 44 | mergeSchemas(schemaA, schemaB) 45 | }) 46 | }) 47 | 48 | it('supports merging exhaustive structs when they have the same fields', () => { 49 | const schemaA: Schema = makeS('Red', true) 50 | const schemaB: Schema = makeS('Red', false) 51 | const merged = mergeSchemas(schemaA, schemaB) 52 | const color = merged.types['Color'] as EnumSchema 53 | assert.equal(color.exhaustive, true) 54 | }) 55 | }) 56 | 57 | describe('foreign merges', () => { 58 | const fileSchema: Schema = { 59 | id: 'Example', 60 | root: ref('Contact'), 61 | types: { 62 | Contact: { 63 | foreign: true, 64 | exhaustive: false, 65 | numericOnly: false, 66 | localStructIsVariant: 'Default', 67 | variants: new Map([['Default', { 68 | foreign: false, 69 | skip: false, 70 | fields: new Map([ 71 | ['name', {type: String, foreign: true, optional: false}], 72 | ['age', {type: prim('u32'), foreign: true, optional: false}], 73 | // address: {type: String}, 74 | ]) 75 | }]]), 76 | 77 | // encodingOrder: ['age', 'name'], 78 | }, 79 | Color: enumOfStringsEncoding('Red', 'Blue'), 80 | } 81 | } 82 | 83 | const appSchema: Schema = { 84 | id: 'Example', 85 | root: ref('Contact'), 86 | types: { 87 | Contact: structSchema([ 88 | // name: {type: String}, 89 | ['age', {type: prim('u32'), renameFieldTo: 'yearsOld'}], 90 | ['address', {type: String, defaultValue: 'unknown location'}], 91 | // ['age', {type: prim('u32'), skip: true, renameFieldTo: 'yearsOld'}], 92 | // ['address', {type: String, skip: true, defaultValue: 'unknown location'}], 93 | ]), 94 | Color: enumOfStringsEncoding('Red', 'Bronze'), 95 | } 96 | } 97 | 98 | it('merges', () => { 99 | const merged = mergeSchemas(fileSchema, appSchema) 100 | // console.log(merged) 101 | 102 | assert.equal(true, merged.types.Contact.variants.get('Default')!.fields!.get('name')!.foreign) 103 | assert.equal(false, merged.types.Contact.variants.get('Default')!.fields!.get('age')!.foreign) 104 | assert.equal(false, merged.types.Contact.variants.get('Default')!.fields!.get('address')!.foreign) 105 | 106 | assert.equal(false, (merged.types.Color).variants.get('Red')!.foreign ?? false) 107 | assert.equal(true, (merged.types.Color).variants.get('Blue')!.foreign ?? false) 108 | assert.equal(false, (merged.types.Color).variants.get('Bronze')!.foreign ?? false) 109 | }) 110 | 111 | it('merges via opaque data', () => { 112 | // const data = writeOpaqueData(appSchema, {age: 12, address: 'somewhere'}) 113 | const data = write(fileSchema, {name: 'simone', age: 41}) 114 | const [schema, loaded] = read(appSchema, data) 115 | assert.deepEqual(loaded, {yearsOld: 41, address: 'unknown location', _foreign: {name: 'simone'}}) 116 | }) 117 | }) 118 | 119 | it('merges non-overlapping struct fields', () => { 120 | const remote: Schema = extendSchema({ 121 | id: 'Example', 122 | root: ref('Contact'), 123 | types: { 124 | Contact: { 125 | fields: { 126 | name: 'string', 127 | address: 'string', 128 | } 129 | }, 130 | Color: enumOfStrings('Red', 'Green'), 131 | } 132 | }) 133 | 134 | const local: Schema = extendSchema({ 135 | id: 'Example', 136 | root: ref('Contact'), 137 | types: { 138 | Contact: { 139 | fields: { 140 | name: {type: 'string', defaultValue: 'Bruce'}, 141 | phoneNo: {type: 'string'}, 142 | } 143 | }, 144 | Color: enumOfStrings('Green', 'Blue'), 145 | } 146 | }) 147 | 148 | const merged = mergeSchemas(remote, local) 149 | 150 | const expected: Schema = { 151 | id: 'Example', 152 | root: ref('Contact'), 153 | types: { 154 | Contact: { 155 | foreign: false, 156 | exhaustive: false, 157 | numericOnly: false, 158 | localStructIsVariant: 'Default', 159 | variants: new Map([['Default', { 160 | foreign: false, 161 | fields: new Map([ 162 | ['name', {type: String, foreign: false, skip: false, defaultValue: 'Bruce', optional: false}], 163 | ['address', {type: String, foreign: true, skip: false, optional: false}], 164 | ['phoneNo', {type: String, foreign: false, skip: true, optional: false}], 165 | ]) 166 | }]]), 167 | }, 168 | 169 | Color: { 170 | foreign: false, 171 | exhaustive: false, 172 | numericOnly: true, 173 | variants: new Map([ 174 | ['Red', {foreign: true}], 175 | ['Green', {foreign: false}], 176 | ['Blue', {foreign: false}], 177 | ]) 178 | }, 179 | 180 | } 181 | } 182 | fillSchemaDefaults(merged, false) 183 | fillSchemaDefaults(expected, false) 184 | 185 | assert.deepEqual(merged, expected) 186 | }) 187 | }) 188 | -------------------------------------------------------------------------------- /test/metaschema.ts: -------------------------------------------------------------------------------- 1 | import 'mocha' 2 | import * as assert from 'assert/strict' 3 | import * as fs from 'fs' 4 | import { writeRaw, metaSchema, readRaw } from '../lib/index.js' 5 | import { fillSchemaDefaults, mergeSchemas } from '../lib/utils.js' 6 | 7 | // import {Console} from 'node:console' 8 | // const console = new Console({ 9 | // stdout: process.stdout, 10 | // stderr: process.stderr, 11 | // inspectOptions: {depth: null} 12 | // }) 13 | 14 | describe('metaschema', () => { 15 | it('can parse itself', () => { 16 | 17 | const bytes = writeRaw(metaSchema, metaSchema) 18 | const remoteSchema = readRaw(metaSchema, bytes) 19 | // console.log(remoteSchema) 20 | fillSchemaDefaults(metaSchema, false) 21 | // fillSchemaDefaults(rm, false) 22 | fillSchemaDefaults(remoteSchema, false) 23 | let rm = mergeSchemas(remoteSchema, metaSchema) 24 | 25 | // console.log(rm.types.Field.variants.get('Default')) 26 | // console.log(remoteSchema.types.Field.variants.get('Default')) 27 | assert.deepEqual(metaSchema, rm) 28 | 29 | // console.log(bytes) 30 | fs.writeFileSync('metaschema2.scb', bytes) 31 | }) 32 | }) -------------------------------------------------------------------------------- /test/read.ts: -------------------------------------------------------------------------------- 1 | import 'mocha' 2 | import * as assert from 'assert/strict' 3 | import { readRaw } from '../lib/read.js' 4 | import { Schema, AppSchema, Field } from '../lib/schema.js' 5 | import { mergeSchemas, prim, ref, String, structSchema } from '../lib/utils.js' 6 | import { extendSchema } from '../lib/extendschema.js' 7 | 8 | import {Console} from 'node:console' 9 | const console = new Console({ 10 | stdout: process.stdout, 11 | stderr: process.stderr, 12 | inspectOptions: {depth: null} 13 | }) 14 | 15 | describe('read', () => { 16 | it('reads from trivial schema', () => { 17 | const schema: Schema = { 18 | id: 'Example', 19 | root: ref('Contact'), 20 | types: { 21 | Contact: structSchema([ 22 | ['age', {type: prim('u32')}], 23 | ['name', {type: String}], 24 | // address: {type: String}, 25 | ]) 26 | } 27 | } 28 | 29 | const data = new Uint8Array([ 123, 4, 115, 101, 112, 104 ]) 30 | const output = readRaw(schema, data) 31 | assert.deepEqual(output, {age: 123, name: 'seph'}) 32 | }) 33 | 34 | it('Reads from a merged schema', () => { 35 | const fileSchema: Schema = { 36 | id: 'Example', 37 | root: ref('Contact'), 38 | types: { 39 | Contact: structSchema([ 40 | ['age', {type: prim('u32')}], 41 | ['name', {type: String}], 42 | // address: {type: String}, 43 | ]) 44 | } 45 | } 46 | 47 | const appSchema: AppSchema = { 48 | id: 'Example', 49 | root: ref('Contact'), 50 | types: { 51 | Contact: { 52 | fields: { 53 | // name: {type: String}, 54 | age: {type: 'u32', optional: true, renameFieldTo: 'yearsOld'}, 55 | address: {type: 'string', optional: true, defaultValue: 'unknown location'}, 56 | } 57 | } 58 | } 59 | } 60 | 61 | const b = new Uint8Array([ 123, 4, 115, 101, 112, 104 ]) 62 | 63 | // console.log(extendSchema(appSchema)) 64 | const mergedSchema = mergeSchemas(fileSchema, extendSchema(appSchema)) 65 | // console.log(mergedSchema) 66 | const output = readRaw(mergedSchema, b) 67 | assert.deepEqual(output, { 68 | yearsOld: 123, 69 | address: 'unknown location', 70 | _foreign: {name: 'seph'} 71 | }) 72 | }) 73 | }) 74 | -------------------------------------------------------------------------------- /test/roundtrips.ts: -------------------------------------------------------------------------------- 1 | // This file checks that we can store a bunch of stuff, and when we do we get the same data back out. 2 | import 'mocha' 3 | import {AppSchema, Schema, EnumSchema} from "../lib/schema.js" 4 | import { Bool, enumOfStrings, Id, list, map, prim, ref, String, structSchema } from "../lib/utils.js" 5 | import { readRaw, read } from "../lib/read.js" 6 | import { writeRaw, write } from "../lib/write.js" 7 | import { extendSchema } from '../lib/extendschema.js' 8 | 9 | import fs from 'fs' 10 | import * as assert from 'assert/strict' 11 | // import {Console} from 'node:console' 12 | // const console = new Console({ 13 | // stdout: process.stdout, 14 | // stderr: process.stderr, 15 | // inspectOptions: {depth: null} 16 | // }) 17 | 18 | const testRoundTripFullSchema = (schema: Schema, input: any, expectedOutput = input) => { 19 | const bytes = writeRaw(schema, input) 20 | // console.log('bytes', bytes) 21 | const result = readRaw(schema, bytes) 22 | // console.log('result', result) 23 | 24 | assert.deepEqual(result, expectedOutput) 25 | 26 | 27 | { 28 | const opaque = write(schema, input) 29 | // console.log('opaque', opaque) 30 | // fs.writeFileSync('tmp_test.sb', opaque) 31 | // console.log('schema', schema) 32 | const [fileSchema, result] = read(schema, opaque) 33 | assert.deepEqual(result, expectedOutput) 34 | } 35 | 36 | } 37 | 38 | const testRoundTrip = (schema: AppSchema, input: any, expectedOutput = input) => { 39 | const fullSchema = extendSchema(schema) 40 | // console.log('fullSchema', fullSchema) 41 | testRoundTripFullSchema(fullSchema, input, expectedOutput) 42 | } 43 | 44 | describe('roundtrips', () => { 45 | describe('non objects at the root', () => { 46 | it('works with strings', () => { 47 | const schema: AppSchema = { 48 | id: 'Example', 49 | root: 'string', 50 | types: {} 51 | } 52 | 53 | testRoundTrip(schema, 'hi there') 54 | }) 55 | 56 | it('works with lists', () => { 57 | const schema: AppSchema = { 58 | id: 'Example', 59 | root: list('f64'), 60 | types: {} 61 | } 62 | 63 | testRoundTrip(schema, [1.1, 2.2, 3,3]) 64 | }) 65 | 66 | it('works with maps', () => { 67 | const schema: AppSchema = { 68 | id: 'Example', 69 | root: map('string', 'f64', 'object'), 70 | types: {} 71 | } 72 | 73 | testRoundTrip(schema, {aa: 123, bb: 213.23}) 74 | }) 75 | 76 | it('works with maps using entry list decoding form', () => { 77 | const schema: AppSchema = { 78 | id: 'Example', 79 | root: map('string', 'f64', 'entryList'), 80 | types: {} 81 | } 82 | 83 | testRoundTrip(schema, [['aa', 123], ['bb', 213.23]]) 84 | testRoundTrip(schema, new Map([['aa', 123], ['bb', 213.23]]), [['aa', 123], ['bb', 213.23]]) 85 | testRoundTrip(schema, {aa: 123, bb: 213.23}, [['aa', 123], ['bb', 213.23]]) 86 | }) 87 | 88 | it('works with maps using map decoding form', () => { 89 | const schema: AppSchema = { 90 | id: 'Example', 91 | root: map('string', 'f64', 'map'), 92 | types: {} 93 | } 94 | 95 | testRoundTrip(schema, new Map([['aa', 123], ['bb', 213.23]])) 96 | testRoundTrip(schema, {aa: 123, bb: 213.23}, new Map([['aa', 123], ['bb', 213.23]])) 97 | }) 98 | 99 | it('works with maps using encode and decode methods', () => { 100 | const schema: AppSchema = { 101 | id: 'Example', 102 | root: { 103 | ...map('u64', 'u64', 'map'), 104 | encodeEntry([k, v]) { 105 | assert.equal(typeof k, 'boolean') 106 | return [k+0, v] 107 | }, 108 | decodeEntry([k, v]) { 109 | assert.equal(typeof k, 'number') 110 | return [k != 0, v] 111 | }, 112 | }, 113 | types: {} 114 | } 115 | 116 | // testRoundTrip(schema, new Map([[true, 123], [false, 456]])) 117 | testRoundTrip(schema, new Map([[true, 123]])) 118 | }) 119 | }) 120 | 121 | // ***** 122 | it('works with simple structs', () => { 123 | const schema: AppSchema = { 124 | id: 'Example', 125 | root: 'Contact', 126 | types: { 127 | Contact: { 128 | fields: { 129 | name: 'string', 130 | } 131 | } 132 | } 133 | } 134 | 135 | testRoundTrip(schema, {name: 'seph'}) 136 | }) 137 | 138 | it('does not encode skipped fields', () => { 139 | const schema: AppSchema = { 140 | id: 'Example', 141 | root: 'Contact', 142 | types: { 143 | Contact: { 144 | fields: { 145 | // name: 'string', 146 | notSaved: {type: 'string', skip: true, defaultValue: 'secrets'}, 147 | } 148 | } 149 | } 150 | } 151 | 152 | testRoundTrip(schema, {notSaved: 'hiiiii'}, {notSaved: 'secrets'}) 153 | }) 154 | 155 | describe('enums', () => { 156 | it('simple enums', () => { 157 | // Numeric Enum 158 | const schema: AppSchema = { 159 | id: 'Example', 160 | root: 'Color', 161 | types: { 162 | // Color: enumOfStringsSimple('Red', 'Blue', 'Green'), 163 | Color: { 164 | type: 'enum', 165 | numericOnly: true, 166 | variants: ['Red', 'Blue', 'Green'], 167 | } 168 | } 169 | } 170 | 171 | testRoundTrip(schema, 'Red') 172 | testRoundTrip(schema, 'Blue') 173 | }) 174 | 175 | it('enums with associated data and optional fields', () => { 176 | // Enum 177 | const schema: AppSchema = { 178 | id: 'Example', 179 | root: 'Color', 180 | types: { 181 | Color: { 182 | type: 'enum', 183 | numericOnly: false, 184 | variants: { 185 | Blue: null, 186 | Red: null, 187 | RGB: { 188 | fields: { 189 | r: {type: 'u8', optional: true}, 190 | g: {type: 'u8', optional: true}, 191 | b: {type: 'u8', optional: true}, 192 | } 193 | } 194 | } 195 | } 196 | } 197 | } 198 | 199 | // console.log(extendSchema(schema)) 200 | testRoundTrip(schema, {type: 'Red'}) 201 | testRoundTrip(schema, {type: 'Blue'}) 202 | // // testRoundTrip(schema, {type: 'Blue'}) 203 | testRoundTrip(schema, {type: 'RGB', r: null, g: null, b: null}) // TODO: Make a non-nullable variant. 204 | testRoundTrip(schema, {type: 'RGB', r: 123, g: 2, b: 1}) 205 | 206 | }) 207 | 208 | it('tags foreign variants', () => { 209 | // Test unknown enum variants 210 | const SimpleSchema: AppSchema = { 211 | id: 'Example', 212 | root: 'Color', 213 | types: { 214 | Color: { 215 | type: 'enum', 216 | numericOnly: false, 217 | variants: { 218 | Blue: true, 219 | Red: true, 220 | RGB: { 221 | fields: { r: 'u8', g: 'u8', b: 'u8' } 222 | } 223 | } 224 | } 225 | } 226 | } 227 | 228 | let schema = extendSchema(SimpleSchema) 229 | schema.types['Color'].variants.get('Red')!.foreign = true 230 | schema.types['Color'].variants.get('RGB')!.foreign = true 231 | testRoundTripFullSchema(schema, {type: '_foreign', data: {type: 'Red'}}) 232 | testRoundTripFullSchema(schema, {type: '_foreign', data: {type: 'RGB', r: 123, g: 2, b: 1}}) 233 | }) 234 | }) 235 | 236 | 237 | describe('structs', () => { 238 | it('allows optional struct fields', () => { 239 | // Test nullable struct fields 240 | const schema: AppSchema = { 241 | id: 'Example', 242 | root: 'Contact', 243 | types: { 244 | Contact: { 245 | fields: { 246 | name: {type: 'string', optional: true}, 247 | age: {type: 'u32', optional: true}, 248 | addresses: {type: 'list', fieldType: {type:'string'}, optional: true} 249 | // address: {type: String}, 250 | } 251 | } 252 | } 253 | } 254 | 255 | testRoundTrip(schema, {name: 'seph', age: 21, addresses: ['123 Example St', '456 Somewhere else']}) 256 | testRoundTrip(schema, {name: 'seph', age: null, addresses: ['123 Example St', '456 Somewhere else']}) 257 | testRoundTrip(schema, {name: null, age: null, addresses: null}) 258 | testRoundTrip(schema, {name: null, age: null, addresses: []}) 259 | }) 260 | 261 | it('re-encodes foreign fields when serializing', () => { 262 | const schema: Schema = { 263 | id: 'Example', 264 | root: ref('Contact'), 265 | types: { 266 | Contact: structSchema([ 267 | ['name', {type: String}], 268 | ['age', {type: prim('u32'), foreign: true}], 269 | ]) 270 | } 271 | } 272 | 273 | testRoundTripFullSchema(schema, {name: 'seph', _foreign: {age: 32}}) 274 | testRoundTripFullSchema(schema, {name: 'seph', age: 32}, {name: 'seph', _foreign: {age: 32}}) 275 | }) 276 | 277 | it('supports inlined and non-inlined booleans', () => { 278 | const schema: Schema = { 279 | id: 'Example', 280 | root: ref('Bools'), 281 | types: { 282 | Bools: structSchema([ 283 | ['a', {type: Bool, optional: false, inline: false}], 284 | ['b', {type: Bool, optional: true, inline: false}], 285 | 286 | ['c', {type: Bool, optional: false, inline: true}], 287 | ['d', {type: Bool, optional: true, inline: true}], 288 | ]) 289 | } 290 | } 291 | 292 | testRoundTripFullSchema(schema, {a: true, b: true, c: true, d: true}) 293 | testRoundTripFullSchema(schema, {a: true, b: false, c: true, d: false}) 294 | testRoundTripFullSchema(schema, {a: true, c: true}, 295 | {a: true, b: null, c: true, d: null} 296 | ) 297 | testRoundTripFullSchema(schema, {a: false, c: false}, 298 | {a: false, b: null, c: false, d: null} 299 | ) 300 | }) 301 | 302 | it('works with inlined and non-inlined booleans with default values', () => { 303 | const schema: Schema = { 304 | id: 'Example', 305 | root: ref('Bools'), 306 | types: { 307 | Bools: structSchema([ 308 | ['a', {type: Bool, optional: false, inline: false, defaultValue: true}], 309 | ['b', {type: Bool, optional: false, inline: false, defaultValue: false}], 310 | ['c', {type: Bool, optional: false, inline: true, defaultValue: true}], 311 | ['d', {type: Bool, optional: false, inline: true, defaultValue: false}], 312 | ]) 313 | } 314 | } 315 | 316 | testRoundTripFullSchema(schema, {a: true, b: true, c: true, d: true}) 317 | testRoundTripFullSchema(schema, {a: false, b: false, c: false, d: false}) 318 | testRoundTripFullSchema(schema, {a: true, b: false, c: false, d: true}) 319 | testRoundTripFullSchema(schema, {}, {a: true, b: false, c: true, d: false}) 320 | testRoundTripFullSchema(schema, {a: false, d: true}, {a: false, b: false, c: true, d: true}) 321 | }) 322 | 323 | describe('numerics', () => { 324 | it('works with default options', () => { 325 | const schema: AppSchema = { 326 | id: 'Example', 327 | root: 'NumTest', 328 | types: { 329 | NumTest: { 330 | fields: { 331 | u8V: {type: 'u8', numericEncoding: 'varint'}, 332 | s8V: {type: 's8', numericEncoding: 'varint'}, 333 | u8: 'u8', 334 | u16: 'u16', 335 | u32: 'u32', 336 | u64: 'u64', 337 | u128: 'u128', 338 | 339 | s8: 's8', 340 | s16: 's16', 341 | s32: 's32', 342 | s64: 's64', 343 | s128: 's128', 344 | } 345 | } 346 | } 347 | } 348 | 349 | testRoundTrip(schema, { 350 | u8V: 200, s8V: -127, 351 | u8: 0xff, u16: 0xffff, u32: 0xffffffff, u64: Number.MAX_SAFE_INTEGER, u128: Number.MAX_SAFE_INTEGER, 352 | s8: -0x80, s16: -0x8000, s32: -0x80000000, s64: 0, s128: 0, 353 | // s8: -0x80, s16: -0x8000, s32: -0x80000000, s64: Number.MIN_SAFE_INTEGER, s128: Number.MIN_SAFE_INTEGER, 354 | }) 355 | testRoundTrip(schema, { 356 | u8V: 200n, s8V: -127n, 357 | u8: 0xffn, u16: 0xffffn, u32: 0xffffffffn, u64: BigInt(Number.MAX_SAFE_INTEGER), u128: BigInt(Number.MAX_SAFE_INTEGER), 358 | s8: -0x80n, s16: -0x8000n, s32: -0x80000000n, s64: 0n, s128: 0n, 359 | }, { 360 | u8V: 200, s8V: -127, 361 | u8: 0xff, u16: 0xffff, u32: 0xffffffff, u64: Number.MAX_SAFE_INTEGER, u128: Number.MAX_SAFE_INTEGER, 362 | s8: -0x80, s16: -0x8000, s32: -0x80000000, s64: 0, s128: 0, 363 | }) 364 | }) 365 | 366 | it('decodes bigints', () => { 367 | const schema: AppSchema = { 368 | id: 'Example', 369 | root: 'NumTest', 370 | types: { 371 | NumTest: { 372 | fields: { 373 | u8V: {type: 'u8', decodeAsBigInt: true, numericEncoding: 'varint'}, 374 | s8V: {type: 's8', decodeAsBigInt: true, numericEncoding: 'varint'}, 375 | u8: {type: 'u8', decodeAsBigInt: true}, 376 | u16: {type: 'u16', decodeAsBigInt: true}, 377 | u32: {type: 'u32', decodeAsBigInt: true}, 378 | u64: {type: 'u64', decodeAsBigInt: true}, 379 | u128: {type: 'u128', decodeAsBigInt: true}, 380 | 381 | s8: {type: 's8', decodeAsBigInt: true}, 382 | s16: {type: 's16', decodeAsBigInt: true}, 383 | s32: {type: 's32', decodeAsBigInt: true}, 384 | s64: {type: 's64', decodeAsBigInt: true}, 385 | s128: {type: 's128', decodeAsBigInt: true}, 386 | } 387 | } 388 | } 389 | } 390 | 391 | testRoundTrip(schema, { 392 | u8V: 200n, s8V: -127n, 393 | u8: 0xffn, u16: 0xffffn, u32: 0xffffffffn, u64: 2n ** 64n - 1n, u128: 2n ** 128n - 1n, 394 | s8: -0x80n, s16: -0x8000n, s32: -0x80000000n, s64: -(2n ** 63n), s128: -(2n ** 127n - 1n), 395 | }) 396 | }) 397 | 398 | }) 399 | 400 | it('ids', () => { 401 | const schema: AppSchema = { 402 | id: 'Example', 403 | root: {type: 'list', fieldType: ref('IdTest')}, 404 | types: { 405 | IdTest: { 406 | fields: { 407 | foo: 'id', 408 | bar: 'id', 409 | } 410 | } 411 | } 412 | } 413 | 414 | testRoundTrip(schema, [{foo: 'a', bar: 'a'}]) 415 | testRoundTrip(schema, [{foo: 'a', bar: 'b'}]) 416 | testRoundTrip(schema, [{foo: 'a', bar: 'b'}, {foo: 'a', bar: 'b'}]) 417 | testRoundTrip(schema, [{foo: 'a', bar: 'b'}, {foo: 'b', bar: 'a'}]) 418 | }) 419 | 420 | }) 421 | }) 422 | -------------------------------------------------------------------------------- /test/tldraw_test.ts: -------------------------------------------------------------------------------- 1 | import { Schema, AppSchema } from "../lib/schema.js" 2 | import { enumOfStrings, ref, fillSchemaDefaults } from "../lib/utils.js" 3 | import fs from 'fs' 4 | import { writeRaw, write } from "../lib/write.js" 5 | import { metaSchema } from "../lib/metaschema.js" 6 | import { readRaw } from "../lib/read.js" 7 | import { extendSchema } from '../lib/extendschema.js' 8 | import * as assert from 'assert/strict' 9 | 10 | // import {Console} from 'node:console' 11 | // const console = new Console({ 12 | // stdout: process.stdout, 13 | // stderr: process.stderr, 14 | // inspectOptions: {depth: null} 15 | // }) 16 | 17 | const tldrawTest = () => { 18 | const testSchema: AppSchema = { 19 | id: 'Shape', 20 | // root: ref('Shape'), 21 | root: {type: 'list', fieldType: ref('Shape')}, 22 | types: { 23 | Shape: { 24 | fields: { 25 | x: 'f32', 26 | y: 'f32', 27 | rotation: 'f32', 28 | id: 'id', 29 | parentId: 'id', 30 | index: 'string', 31 | typeName: 'ShapeType', 32 | props: 'Props', 33 | // {key: 'type', valType: enumOfStrings(['geo', 'arrow', 'text'])}, 34 | } 35 | }, 36 | 37 | ShapeType: enumOfStrings('shape'), 38 | Color: enumOfStrings('light-blue', 'light-red', 'black', 'light-green', 'yellow', 'light-violet'), 39 | Size: enumOfStrings('l', 'xl'), 40 | Alignment: enumOfStrings('middle', 'start', 'end'), 41 | GeoType: enumOfStrings('ellipse', 'rectangle'), 42 | Fill: enumOfStrings('pattern', 'none'), 43 | Dash: enumOfStrings('draw'), 44 | ArrowHead: enumOfStrings('arrow', 'none'), 45 | 46 | Props: { 47 | type: 'enum', 48 | numericOnly: false, 49 | exhaustive: false, 50 | typeFieldOnParent: 'type', 51 | variants: { 52 | text: { 53 | fields: { 54 | opacity: 'string', 55 | color: 'Color', 56 | size: 'Size', 57 | w: 'u32', 58 | text: 'string', 59 | font: 'string', 60 | align: 'Alignment', 61 | autoSize: 'bool', 62 | } 63 | }, 64 | 65 | geo: { 66 | fields: { 67 | w: 'f32', 68 | h: 'f32', 69 | geo: ref('GeoType'), 70 | color: ref('Color'), 71 | fill: ref('Fill'), 72 | dash: ref('Dash'), 73 | size: ref('Size'), 74 | opacity: 'string', // Why is this a string? 75 | font: 'string', // Or enumOfStrings(['draw']) 76 | text: 'string', 77 | align: ref('Alignment'), 78 | growY: 'u32', 79 | } 80 | }, 81 | 82 | arrow: { 83 | fields: { 84 | opacity: 'string', // Why is this a string? 85 | dash: ref('Dash'), 86 | size: ref('Size'), 87 | fill: ref('Fill'), 88 | color: ref('Color'), 89 | w: 'f32', 90 | h: 'f32', 91 | bend: 'f32', 92 | 93 | start: ref('ArrowEnd'), 94 | end: ref('ArrowEnd'), 95 | 96 | arrowheadStart: ref('ArrowHead'), 97 | arrowheadEnd: ref('ArrowHead'), 98 | } 99 | } 100 | } 101 | }, 102 | 103 | Vec2: { 104 | fields: { 105 | x: 'f32', 106 | y: 'f32', 107 | } 108 | }, 109 | 110 | ArrowEnd: { 111 | fields: { 112 | x: 'f32', 113 | y: 'f32', 114 | binding: 'id', 115 | anchor: 'Vec2', 116 | } 117 | } 118 | } 119 | } 120 | 121 | // console.log('\n\n') 122 | const shapes = JSON.parse(fs.readFileSync('./tldraw-example.json', 'utf8')).data.shape 123 | // console.log(shapes) 124 | const fullSchema = extendSchema(testSchema) 125 | fillSchemaDefaults(fullSchema, true) 126 | 127 | const sOut = writeRaw(metaSchema, fullSchema) 128 | console.log('Schema size', sOut.length) 129 | const mm = readRaw(metaSchema, sOut) 130 | fillSchemaDefaults(mm, true) 131 | // console.log(mm) 132 | // console.log(fullSchema) 133 | // assert.deepEqual(fullSchema, mm) 134 | 135 | 136 | console.log('schema', sOut) 137 | fs.writeFileSync('tld_schema.scb', sOut) 138 | 139 | 140 | let out = write(fullSchema, shapes) 141 | 142 | // let out = toBinary(fullSchema, shapes) 143 | // // const out = readData(testSchema, shapes) 144 | console.log('Output length', out.length) 145 | fs.writeFileSync('tld2.scb', out) 146 | 147 | } 148 | 149 | // tldrawTest() -------------------------------------------------------------------------------- /test/write.ts: -------------------------------------------------------------------------------- 1 | import 'mocha' 2 | import * as assert from 'assert/strict' 3 | import { writeRaw } from '../lib/write.js' 4 | import { Schema, AppSchema, Field, EnumSchema } from '../lib/schema.js' 5 | import { Bool, prim, ref, String, structSchema } from '../lib/utils.js' 6 | import { extendSchema } from '../lib/extendschema.js' 7 | 8 | describe('write', () => { 9 | it('simple test', () => { 10 | const schema: Schema = { 11 | id: 'Example', 12 | root: ref('Contact'), 13 | types: { 14 | Contact: structSchema([ 15 | ['name', {type: String, optional: true}], 16 | ['age', {type: prim('u32')}] 17 | // address: {type: String}, 18 | ]) 19 | } 20 | } 21 | 22 | const data = {name: 'seph', age: 21} 23 | 24 | const out = writeRaw(schema, data) 25 | // console.log('out', out) 26 | assert.deepEqual(out, new Uint8Array([1, 4, 115, 101, 112, 104, 21])) 27 | }) 28 | 29 | it('kitchen sink test', () => { 30 | const schema: AppSchema = { 31 | id: 'Example', 32 | root: ref('Contact'), 33 | types: { 34 | Contact: { 35 | fields: { 36 | name: String, 37 | age: {type: 'u32', optional: false}, 38 | supercool: {type: 'bool', defaultValue: true}, 39 | addresses: {type: 'list', fieldType: String}, 40 | // address: {type: String}, 41 | favoriteColor: {type: 'ref', key: 'Color'}, 42 | worstColor: {type: 'ref', key: 'Color'}, 43 | hairColor: {type: 'ref', key: 'Color'}, 44 | } 45 | }, 46 | 47 | Color: { 48 | type: 'enum', 49 | numericOnly: false, 50 | variants: { 51 | Blue: null, 52 | Red: null, 53 | RGB: { 54 | fields: { 55 | r: 'u32', 56 | g: 'u32', 57 | b: 'u32', 58 | } 59 | } 60 | } 61 | } 62 | } 63 | } 64 | 65 | const data = { 66 | name: 'seph', 67 | age: 21, 68 | addresses: ['123 Example St', '456 Somewhere else'], 69 | favoriteColor: 'Red', 70 | hairColor: {type: 'Blue'}, 71 | worstColor: {type: 'RGB', r: 10, g: 50, b: 100}, 72 | } 73 | 74 | // console.log('schema', extendSchema(schema)) 75 | // console.log(toBinary(extendSchema(schema), data)) 76 | 77 | const out = writeRaw(extendSchema(schema), data) 78 | // console.log(out) 79 | 80 | const expected = new Uint8Array([ 81 | 1, 4, 115, 101, 112, 104, 21, 2, 14, 49, 82 | 50, 51, 32, 69, 120, 97, 109, 112, 108, 101, 83 | 32, 83, 116, 18, 52, 53, 54, 32, 83, 111, 84 | 109, 101, 119, 104, 101, 114, 101, 32, 101, 108, 85 | 115, 101, 1, 2, 10, 50, 100, 0 86 | ]) 87 | // This will fail if the encoding system changes. 88 | assert.deepEqual(expected, out) 89 | }) 90 | 91 | it('encodes varint when asked', () => { 92 | { 93 | const schema: AppSchema = { 94 | id: 'Example', 95 | root: {type: 'u8', numericEncoding: 'le'}, 96 | types: {} 97 | } 98 | 99 | const out = writeRaw(extendSchema(schema), 205) 100 | assert.deepEqual(out, new Uint8Array([205])) 101 | } 102 | 103 | { 104 | const schema: AppSchema = { 105 | id: 'Example', 106 | root: {type: 'u8', numericEncoding: 'varint'}, 107 | types: {} 108 | } 109 | 110 | const out = writeRaw(extendSchema(schema), 205) 111 | assert.deepEqual(out, new Uint8Array([0x80, 77])) 112 | } 113 | }) 114 | }) 115 | -------------------------------------------------------------------------------- /tsconfig.json: -------------------------------------------------------------------------------- 1 | { 2 | "compilerOptions": { 3 | /* Visit https://aka.ms/tsconfig to read more about this file */ 4 | 5 | /* Projects */ 6 | // "incremental": true, /* Save .tsbuildinfo files to allow for incremental compilation of projects. */ 7 | // "composite": true, /* Enable constraints that allow a TypeScript project to be used with project references. */ 8 | // "tsBuildInfoFile": "./.tsbuildinfo", /* Specify the path to .tsbuildinfo incremental compilation file. */ 9 | // "disableSourceOfProjectReferenceRedirect": true, /* Disable preferring source files instead of declaration files when referencing composite projects. */ 10 | // "disableSolutionSearching": true, /* Opt a project out of multi-project reference checking when editing. */ 11 | // "disableReferencedProjectLoad": true, /* Reduce the number of projects loaded automatically by TypeScript. */ 12 | 13 | /* Language and Environment */ 14 | "target": "ES2020", /* Set the JavaScript language version for emitted JavaScript and include compatible library declarations. */ 15 | // "lib": [], /* Specify a set of bundled library declaration files that describe the target runtime environment. */ 16 | // "jsx": "preserve", /* Specify what JSX code is generated. */ 17 | // "experimentalDecorators": true, /* Enable experimental support for TC39 stage 2 draft decorators. */ 18 | // "emitDecoratorMetadata": true, /* Emit design-type metadata for decorated declarations in source files. */ 19 | // "jsxFactory": "", /* Specify the JSX factory function used when targeting React JSX emit, e.g. 'React.createElement' or 'h'. */ 20 | // "jsxFragmentFactory": "", /* Specify the JSX Fragment reference used for fragments when targeting React JSX emit e.g. 'React.Fragment' or 'Fragment'. */ 21 | // "jsxImportSource": "", /* Specify module specifier used to import the JSX factory functions when using 'jsx: react-jsx*'. */ 22 | // "reactNamespace": "", /* Specify the object invoked for 'createElement'. This only applies when targeting 'react' JSX emit. */ 23 | // "noLib": true, /* Disable including any library files, including the default lib.d.ts. */ 24 | // "useDefineForClassFields": true, /* Emit ECMAScript-standard-compliant class fields. */ 25 | // "moduleDetection": "auto", /* Control what method is used to detect module-format JS files. */ 26 | 27 | /* Modules */ 28 | "module": "Node16", /* Specify what module code is generated. */ 29 | // "rootDir": "./lib", /* Specify the root folder within your source files. */ 30 | "moduleResolution": "Node16", /* Specify how TypeScript looks up a file from a given module specifier. */ 31 | // "baseUrl": "./", /* Specify the base directory to resolve non-relative module names. */ 32 | // "paths": {}, /* Specify a set of entries that re-map imports to additional lookup locations. */ 33 | "rootDirs": ["./lib", "./test"], /* Allow multiple folders to be treated as one when resolving modules. */ 34 | // "typeRoots": [], /* Specify multiple folders that act like './node_modules/@types'. */ 35 | // "types": [], /* Specify type package names to be included without being referenced in a source file. */ 36 | // "allowUmdGlobalAccess": true, /* Allow accessing UMD globals from modules. */ 37 | // "moduleSuffixes": [], /* List of file name suffixes to search when resolving a module. */ 38 | // "resolveJsonModule": true, /* Enable importing .json files. */ 39 | // "noResolve": true, /* Disallow 'import's, 'require's or ''s from expanding the number of files TypeScript should add to a project. */ 40 | 41 | /* JavaScript Support */ 42 | // "allowJs": true, /* Allow JavaScript files to be a part of your program. Use the 'checkJS' option to get errors from these files. */ 43 | // "checkJs": true, /* Enable error reporting in type-checked JavaScript files. */ 44 | // "maxNodeModuleJsDepth": 1, /* Specify the maximum folder depth used for checking JavaScript files from 'node_modules'. Only applicable with 'allowJs'. */ 45 | 46 | /* Emit */ 47 | "declaration": true, /* Generate .d.ts files from TypeScript and JavaScript files in your project. */ 48 | "declarationMap": true, /* Create sourcemaps for d.ts files. */ 49 | // "emitDeclarationOnly": true, /* Only output d.ts files and not JavaScript files. */ 50 | "sourceMap": true, /* Create source map files for emitted JavaScript files. */ 51 | // "outFile": "./", /* Specify a file that bundles all outputs into one JavaScript file. If 'declaration' is true, also designates a file that bundles all .d.ts output. */ 52 | "outDir": "./dist", /* Specify an output folder for all emitted files. */ 53 | // "removeComments": true, /* Disable emitting comments. */ 54 | // "noEmit": true, /* Disable emitting files from a compilation. */ 55 | // "importHelpers": true, /* Allow importing helper functions from tslib once per project, instead of including them per-file. */ 56 | // "importsNotUsedAsValues": "remove", /* Specify emit/checking behavior for imports that are only used for types. */ 57 | // "downlevelIteration": true, /* Emit more compliant, but verbose and less performant JavaScript for iteration. */ 58 | // "sourceRoot": "", /* Specify the root path for debuggers to find the reference source code. */ 59 | // "mapRoot": "", /* Specify the location where debugger should locate map files instead of generated locations. */ 60 | // "inlineSourceMap": true, /* Include sourcemap files inside the emitted JavaScript. */ 61 | // "inlineSources": true, /* Include source code in the sourcemaps inside the emitted JavaScript. */ 62 | // "emitBOM": true, /* Emit a UTF-8 Byte Order Mark (BOM) in the beginning of output files. */ 63 | // "newLine": "crlf", /* Set the newline character for emitting files. */ 64 | // "stripInternal": true, /* Disable emitting declarations that have '@internal' in their JSDoc comments. */ 65 | // "noEmitHelpers": true, /* Disable generating custom helper functions like '__extends' in compiled output. */ 66 | // "noEmitOnError": true, /* Disable emitting files if any type checking errors are reported. */ 67 | // "preserveConstEnums": true, /* Disable erasing 'const enum' declarations in generated code. */ 68 | // "declarationDir": "./", /* Specify the output directory for generated declaration files. */ 69 | // "preserveValueImports": true, /* Preserve unused imported values in the JavaScript output that would otherwise be removed. */ 70 | 71 | /* Interop Constraints */ 72 | // "isolatedModules": true, /* Ensure that each file can be safely transpiled without relying on other imports. */ 73 | // "allowSyntheticDefaultImports": true, /* Allow 'import x from y' when a module doesn't have a default export. */ 74 | "esModuleInterop": true, /* Emit additional JavaScript to ease support for importing CommonJS modules. This enables 'allowSyntheticDefaultImports' for type compatibility. */ 75 | // "preserveSymlinks": true, /* Disable resolving symlinks to their realpath. This correlates to the same flag in node. */ 76 | "forceConsistentCasingInFileNames": true, /* Ensure that casing is correct in imports. */ 77 | 78 | /* Type Checking */ 79 | "strict": true, /* Enable all strict type-checking options. */ 80 | // "noImplicitAny": true, /* Enable error reporting for expressions and declarations with an implied 'any' type. */ 81 | // "strictNullChecks": true, /* When type checking, take into account 'null' and 'undefined'. */ 82 | // "strictFunctionTypes": true, /* When assigning functions, check to ensure parameters and the return values are subtype-compatible. */ 83 | // "strictBindCallApply": true, /* Check that the arguments for 'bind', 'call', and 'apply' methods match the original function. */ 84 | // "strictPropertyInitialization": true, /* Check for class properties that are declared but not set in the constructor. */ 85 | // "noImplicitThis": true, /* Enable error reporting when 'this' is given the type 'any'. */ 86 | // "useUnknownInCatchVariables": true, /* Default catch clause variables as 'unknown' instead of 'any'. */ 87 | // "alwaysStrict": true, /* Ensure 'use strict' is always emitted. */ 88 | // "noUnusedLocals": true, /* Enable error reporting when local variables aren't read. */ 89 | // "noUnusedParameters": true, /* Raise an error when a function parameter isn't read. */ 90 | // "exactOptionalPropertyTypes": true, /* Interpret optional property types as written, rather than adding 'undefined'. */ 91 | // "noImplicitReturns": true, /* Enable error reporting for codepaths that do not explicitly return in a function. */ 92 | // "noFallthroughCasesInSwitch": true, /* Enable error reporting for fallthrough cases in switch statements. */ 93 | // "noUncheckedIndexedAccess": true, /* Add 'undefined' to a type when accessed using an index. */ 94 | // "noImplicitOverride": true, /* Ensure overriding members in derived classes are marked with an override modifier. */ 95 | // "noPropertyAccessFromIndexSignature": true, /* Enforces using indexed accessors for keys declared using an indexed type. */ 96 | // "allowUnusedLabels": true, /* Disable error reporting for unused labels. */ 97 | // "allowUnreachableCode": true, /* Disable error reporting for unreachable code. */ 98 | 99 | /* Completeness */ 100 | // "skipDefaultLibCheck": true, /* Skip type checking .d.ts files that are included with TypeScript. */ 101 | "skipLibCheck": true /* Skip type checking all .d.ts files. */ 102 | } 103 | } 104 | --------------------------------------------------------------------------------