├── .gitignore ├── LICENSE ├── README.md ├── cpp ├── .gitignore ├── CMakeLists.txt ├── README.md └── src │ └── main.cpp ├── data ├── 0 │ ├── data.beve │ └── data.json └── 1 │ ├── data.beve │ └── data.json ├── example └── example.js ├── extensions.md ├── golang ├── go.mod └── main.go ├── javascript ├── beve.js └── beve_file.js ├── matlab ├── load_beve.m └── write_beve.m ├── python └── load_beve.py └── rust ├── Cargo.toml └── src └── main.rs /.gitignore: -------------------------------------------------------------------------------- 1 | /build 2 | *.DS_Store 3 | *.beve 4 | *.cache 5 | *.vscode 6 | *.json 7 | 8 | # Allow .beve and .json files in data 9 | !data/**/*.json 10 | !data/**/*.beve 11 | 12 | # Added by cargo 13 | 14 | /target 15 | # RUST 16 | rust/target 17 | rust/Cargo.lock -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2022 Stephen Berry 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # BEVE - Binary Efficient Versatile Encoding 2 | Version 1.0 3 | 4 | *High performance, tagged binary data specification like JSON, MessagePack, CBOR, etc. But, designed for higher performance and scientific computing.* 5 | 6 | > See [Discussions](https://github.com/stephenberry/eve/discussions) for polls and active development on the specification. 7 | 8 | - Maps to and from JSON 9 | - Schema less, fully described, like JSON (can be used in documents) 10 | - Little endian for maximum performance on modern CPUs 11 | - Blazingly fast, designed for SIMD 12 | - Future proof, supports large numerical types (such as 128 bit integers and higher) 13 | - Designed for scientific computing, supports [brain floats](https://en.wikipedia.org/wiki/Bfloat16_floating-point_format), matrices, and complex numbers 14 | - Simple, designed to be easy to integrate 15 | 16 | > BEVE is designed to be faster on modern hardware than CBOR, BSON, and MessagePack, but it is also more space efficient for typed arrays. 17 | 18 | ## Performance vs MessagePack 19 | 20 | The following table lists the performance increase between BEVE with [Glaze](https://github.com/stephenberry/glaze) versus other libraries and their binary formats. 21 | 22 | | Test | Libraries (vs [Glaze](https://github.com/stephenberry/glaze)) | Read (Times Faster) | Write (Times Faster) | 23 | | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------- | -------------------- | 24 | | [Test Object](https://github.com/stephenberry/json_performance) | [msgpack-c](https://github.com/msgpack/msgpack-c) (c++) | 1.9X | 13X | 25 | | double array | [msgpack-c](https://github.com/msgpack/msgpack-c) (c++) | 14X | 50X | 26 | | float array | [msgpack-c](https://github.com/msgpack/msgpack-c) (c++) | 29X | 81X | 27 | | uint16_t array | [msgpack-c](https://github.com/msgpack/msgpack-c) (c++) | 73X | 167X | 28 | 29 | [Performance test code](https://github.com/stephenberry/binary_performance) 30 | 31 | The table below shows binary message size versus BEVE. A positive value means the binary produced is larger than BEVE. 32 | 33 | | Test | Libraries (vs [Glaze](https://github.com/stephenberry/glaze)) | Message Size | 34 | | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------ | 35 | | [Test Object](https://github.com/stephenberry/json_performance) | [msgpack-c](https://github.com/msgpack/msgpack-c) (c++) | -3.4% | 36 | | double array | [msgpack-c](https://github.com/msgpack/msgpack-c) (c++) | +12% | 37 | | float array | [msgpack-c](https://github.com/msgpack/msgpack-c) (c++) | +25% | 38 | | uint16_t array | [msgpack-c](https://github.com/msgpack/msgpack-c) (c++) | +50% | 39 | 40 | ## Why Tagged Messages? 41 | 42 | *Flexibility and efficiency* 43 | 44 | JSON is ubiquitous because it is tagged (has keys), and therefore messages can be sent in part. Furthermore, extending specifications and adding more fields is far easier with tagged messages and unordered mapping. Tags also make the format more human friendly. However, tags are entirely optional, and structs can be serialized as generic arrays. 45 | 46 | ## Endianness 47 | 48 | The endianness must be `little endian`. 49 | 50 | ## File Extension 51 | 52 | The standard extension for BEVE files is `.beve` 53 | 54 | ## Implementations 55 | 56 | ### C++ 57 | 58 | - [Glaze](https://github.com/stephenberry/glaze) (supports JSON and BEVE through the same API) 59 | 60 | ### Matlab 61 | 62 | - [load_beve.m](https://github.com/stephenberry/eve/blob/main/matlab/load_beve.m) (this repository) 63 | - [write_beve.m](https://github.com/stephenberry/eve/blob/main/matlab/write_beve.m) (this repository) 64 | - Work in progress 65 | 66 | 67 | ### Python 68 | 69 | - [load_beve.py](https://github.com/stephenberry/eve/blob/main/python/load_beve.py) (this repository) 70 | 71 | ## Right Most Bit Ordering 72 | 73 | The right most bit is denoted as the first bit, or bit of index 0. 74 | 75 | ## Concerning Compression 76 | 77 | Note that BEVE is not a compression algorithm. It uses some bit packing to be more space efficient, but strings and numerical values see no compression. This means that BEVE binary is very compressible, like JSON, and it is encouraged to use compression algorithms like [LZ4](https://lz4.org), [Zstandard](https://github.com/facebook/zstd), [Brotli](https://github.com/google/brotli), etc. where size is critical. 78 | 79 | ## Compressed Unsigned Integer 80 | 81 | A compressed unsigned integer uses the first two bits to denote the number of bytes used to express an integer. The rest of the bits indicate the integer value. 82 | 83 | > Wherever all caps `SIZE` is used in the specification, it refers to a size indicator that uses a compressed unsigned integer. 84 | > 85 | > `SIZE` refers to the count of array members, object members, or bytes in a string. It does **not** refer to the number of raw bytes except for UTF-8 strings. 86 | 87 | | # | Number of Bytes | Integer Value (N) | 88 | | ---- | --------------- | -------------------------------- | 89 | | 0 | 1 | N < 64 `[2^6]` | 90 | | 1 | 2 | N < 16384 `[2^14]` | 91 | | 2 | 4 | N < 1073741824 `[2^30]` | 92 | | 3 | 8 | N < 4611686018427387904 `[2^62]` | 93 | 94 | ## Byte Count Indicator 95 | 96 | > Wherever all caps `BYTE COUNT` is used, it describes this mapping. 97 | 98 | ```c++ 99 | # Number of bytes 100 | 0 1 101 | 1 2 102 | 2 4 103 | 3 8 104 | 4 16 105 | 5 32 106 | 6 64 107 | 7 128 108 | ... 109 | ``` 110 | 111 | ## Header 112 | 113 | Every `VALUE` begins with a byte header. Any unspecified bits must be set to zero. 114 | 115 | > Wherever all caps `HEADER` is used, it describes this header. 116 | 117 | The first three bits denote types: 118 | 119 | ```c++ 120 | 0 -> null or boolean 0b00000'000 121 | 1 -> number 0b00000'001 122 | 2 -> string 0b00000'010 123 | 3 -> object 0b00000'011 124 | 4 -> typed array 0b00000'100 125 | 5 -> generic array 0b00000'101 126 | 6 -> extension 0b00000'110 127 | 7 -> reserved 0b00000'111 128 | ``` 129 | 130 | ## Nomenclature 131 | 132 | Wherever `DATA` is used, it denotes bytes of data without a `HEADER`. 133 | 134 | Wherever `VALUE` is used, it denotes a binary structure that begins with a `HEADER`. 135 | 136 | Wherever `SIZE` is used, it refers to a compressed unsigned integer that denotes a count of array members, object members, or bytes in a string. 137 | 138 | ## 0 - Null 139 | 140 | Null is simply `0` 141 | 142 | ## 0 - Boolean 143 | 144 | The next bit is set to indicate a boolean. The 5th bit is set to denote true or false. 145 | 146 | ```c++ 147 | false 0b000'01'000 148 | true 0b000'11'000 149 | ``` 150 | 151 | ## 1 - Number 152 | 153 | The next two bits of the HEADER indicates whether the number is floating point, signed integer, or unsigned integer. 154 | 155 | Float point types must conform to the IEEE-754 standard. 156 | 157 | ```c++ 158 | 0 -> floating point 0b000'00'001 159 | 1 -> signed integer 0b000'01'001 160 | 2 -> unsigned integer 0b000'10'001 161 | ``` 162 | 163 | The next three bits of the HEADER are used as the BYTE COUNT. 164 | 165 | > Note: brain floats use a byte count indicator of 1, even though they use 2 bytes per value. This is used because float8_t is not supported and not typically useful. 166 | 167 | > See [Fixed width integer types](https://en.cppreference.com/w/cpp/types/integer) for integer specification. 168 | 169 | ```c++ 170 | bfloat16_t 0b000'00'001 // brain float 171 | float16_t 0b001'00'001 172 | float32_t 0b010'00'001 // float 173 | float64_t 0b011'00'001 // double 174 | float128_t 0b100'00'001 175 | ``` 176 | 177 | ```c++ 178 | int8_t 0b000'01'001 179 | int16_t 0b001'01'001 180 | int32_t 0b010'01'001 181 | int64_t 0b011'01'001 182 | int128_t 0b100'01'001 183 | ``` 184 | 185 | ```c++ 186 | uint8_t 0b000'10'001 187 | uint16_t 0b001'10'001 188 | uint32_t 0b010'10'001 189 | uint64_t 0b011'10'001 190 | uint128_t 0b100'10'001 191 | ``` 192 | 193 | ## 2 - Strings 194 | 195 | Strings must be encoded with UTF-8. 196 | 197 | Layout: `HEADER | SIZE | DATA` 198 | 199 | ### Strings as Object Keys or Typed String Arrays 200 | 201 | When strings are used as keys in objects or typed string arrays the HEADER is not included. 202 | 203 | Layout: `SIZE | DATA` 204 | 205 | ## 3 - Object 206 | 207 | The next two bits of the HEADER indicates the type of key. 208 | 209 | ```c++ 210 | 0 -> string 211 | 1 -> signed integer 212 | 2 -> unsigned integer 213 | ``` 214 | 215 | For integer keys the next three bits of the HEADER indicate the BYTE COUNT. 216 | 217 | > An object `KEY` must not contain a HEADER as the type of the key has already been defined. 218 | 219 | Layout: `HEADER | SIZE | KEY[0] | VALUE[0] | ... KEY[N] | VALUE[N]` 220 | 221 | ## 4 - Typed Array 222 | 223 | The next two bits indicate the type stored in the array: 224 | 225 | ```c++ 226 | 0 -> floating point 227 | 1 -> signed integer 228 | 2 -> unsigned integer 229 | 3 -> boolean or string 230 | ``` 231 | 232 | For integral and floating point types, the next three bits of the type header are the BYTE COUNT. 233 | 234 | For boolean or string types the next bit indicates whether the type is a boolean or a string 235 | 236 | ```c++ 237 | 0 -> boolean // packed as single bits to the nearest byte 238 | 1 -> string // an array of strings (not an array of characters) 239 | ``` 240 | 241 | Layout: `HEADER | SIZE | data` 242 | 243 | ### Boolean Arrays 244 | 245 | Boolean arrays are stored using single bits for booleans and packed to the nearest byte. 246 | 247 | ### String Arrays 248 | 249 | String arrays do not include the string HEADER for each element. 250 | 251 | Layout: `HEADER | SIZE | string[0] | ... string[N]` 252 | 253 | ## 5 - Generic Array 254 | 255 | Generic arrays expect elements to have headers. 256 | 257 | Layout: `HEADER | SIZE | VALUE[0] | ... VALUE[N]` 258 | 259 | ## 6 - [Extensions](https://github.com/stephenberry/eve/blob/main/extensions.md) 260 | 261 | See [extensions.md](https://github.com/stephenberry/eve/blob/main/extensions.md) for additional extension specifications. These are considered to be a formal part of the BEVE specification, but are not expected to be as broadly implemented. 262 | -------------------------------------------------------------------------------- /cpp/.gitignore: -------------------------------------------------------------------------------- 1 | /build -------------------------------------------------------------------------------- /cpp/CMakeLists.txt: -------------------------------------------------------------------------------- 1 | cmake_minimum_required(VERSION 3.30) 2 | project(beve_cpp) 3 | 4 | set(CMAKE_CXX_STANDARD 20) 5 | 6 | file(GLOB srcs src/*.cpp include/*.hpp) 7 | 8 | include(FetchContent) 9 | 10 | FetchContent_Declare( 11 | glaze 12 | GIT_REPOSITORY https://github.com/stephenberry/glaze.git 13 | GIT_TAG main 14 | GIT_SHALLOW TRUE 15 | ) 16 | 17 | FetchContent_MakeAvailable(glaze) 18 | 19 | add_executable(${PROJECT_NAME} ${srcs}) 20 | target_link_libraries(${PROJECT_NAME} glaze::glaze) -------------------------------------------------------------------------------- /cpp/README.md: -------------------------------------------------------------------------------- 1 | ## C++ example for BEVE using [Glaze](https://github.com/stephenberry/glaze) 2 | -------------------------------------------------------------------------------- /cpp/src/main.cpp: -------------------------------------------------------------------------------- 1 | 2 | 3 | #include "glaze/glaze.hpp" 4 | #include "glaze/glaze_exceptions.hpp" 5 | #include 6 | 7 | #include 8 | 9 | static constexpr std::string_view json0 = R"( 10 | { 11 | "fixed_object": { 12 | "int_array": [0, 1, 2, 3, 4, 5, 6], 13 | "float_array": [0.1, 0.2, 0.3, 0.4, 0.5, 0.6], 14 | "double_array": [3288398.238, 233e22, 289e-1, 0.928759872, 0.22222848, 0.1, 0.2, 0.3, 0.4] 15 | }, 16 | "fixed_name_object": { 17 | "name0": "James", 18 | "name1": "Abraham", 19 | "name2": "Susan", 20 | "name3": "Frank", 21 | "name4": "Alicia" 22 | }, 23 | "another_object": { 24 | "string": "here is some text", 25 | "another_string": "Hello World", 26 | "boolean": false, 27 | "nested_object": { 28 | "v3s": [[0.12345, 0.23456, 0.001345], 29 | [0.3894675, 97.39827, 297.92387], 30 | [18.18, 87.289, 2988.298]], 31 | "id": "298728949872" 32 | } 33 | }, 34 | "string_array": ["Cat", "Dog", "Elephant", "Tiger"], 35 | "string": "Hello world", 36 | "number": 3.14, 37 | "boolean": true, 38 | "another_bool": false 39 | } 40 | )"; 41 | 42 | struct fixed_object_t 43 | { 44 | std::vector int_array; 45 | std::vector float_array; 46 | std::vector double_array; 47 | }; 48 | 49 | struct fixed_name_object_t 50 | { 51 | std::string name0{}; 52 | std::string name1{}; 53 | std::string name2{}; 54 | std::string name3{}; 55 | std::string name4{}; 56 | }; 57 | 58 | struct nested_object_t 59 | { 60 | std::vector> v3s{}; 61 | std::string id{}; 62 | }; 63 | 64 | struct another_object_t 65 | { 66 | std::string string{}; 67 | std::string another_string{}; 68 | bool boolean{}; 69 | nested_object_t nested_object{}; 70 | }; 71 | 72 | struct obj_t 73 | { 74 | fixed_object_t fixed_object{}; 75 | fixed_name_object_t fixed_name_object{}; 76 | another_object_t another_object{}; 77 | std::vector string_array{}; 78 | std::string string{}; 79 | double number{}; 80 | bool boolean{}; 81 | bool another_bool{}; 82 | }; 83 | 84 | void example() 85 | { 86 | obj_t obj{}; 87 | glz::ex::read_json(obj, json0); 88 | glz::ex::write_file_beve(obj, "output.beve", std::string{}); 89 | 90 | obj = {}; 91 | glz::ex::read_file_beve(obj, "output.beve", std::string{}); 92 | 93 | glz::ex::write_file_json(obj, "output.json", std::string{}); 94 | } 95 | 96 | struct complex_object 97 | { 98 | std::vector> complex_floats{ {1.0f, 0.5f}, {0.1f, 2.0f} }; 99 | std::vector> complex_int32_t{ {-1, 5}, {7, -9} }; 100 | }; 101 | 102 | void complex_data() 103 | { 104 | complex_object obj{}; 105 | std::string buffer{}; 106 | glz::ex::write_file_beve(obj, "complex.beve", buffer); 107 | glz::ex::write_file_json(obj, "complex.json", buffer); 108 | }; 109 | 110 | int main() { 111 | example(); 112 | complex_data(); 113 | 114 | return 0; 115 | } 116 | -------------------------------------------------------------------------------- /data/0/data.beve: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/beve-org/beve/eeeb779cb77c9f7186cff2713b1ed233c3b7a0cd/data/0/data.beve -------------------------------------------------------------------------------- /data/0/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "fixed_object": { 3 | "int_array": [ 4 | 0, 5 | 1, 6 | 2, 7 | 3, 8 | 4, 9 | 5, 10 | 6 11 | ], 12 | "float_array": [ 13 | 0.1, 14 | 0.2, 15 | 0.3, 16 | 0.4, 17 | 0.5, 18 | 0.6 19 | ], 20 | "double_array": [ 21 | 3288398.238, 22 | 2.33E24, 23 | 28.9, 24 | 0.928759872, 25 | 0.22222848, 26 | 0.1, 27 | 0.2, 28 | 0.3, 29 | 0.4 30 | ] 31 | }, 32 | "fixed_name_object": { 33 | "name0": "James", 34 | "name1": "Abraham", 35 | "name2": "Susan", 36 | "name3": "Frank", 37 | "name4": "Alicia" 38 | }, 39 | "another_object": { 40 | "string": "here is some text", 41 | "another_string": "Hello World", 42 | "boolean": false, 43 | "nested_object": { 44 | "v3s": [ 45 | [ 46 | 0.12345, 47 | 0.23456, 48 | 0.001345 49 | ], 50 | [ 51 | 0.3894675, 52 | 97.39827, 53 | 297.92387 54 | ], 55 | [ 56 | 18.18, 57 | 87.289, 58 | 2988.298 59 | ] 60 | ], 61 | "id": "298728949872" 62 | } 63 | }, 64 | "string_array": [ 65 | "Cat", 66 | "Dog", 67 | "Elephant", 68 | "Tiger" 69 | ], 70 | "string": "Hello world", 71 | "number": 3.14, 72 | "boolean": true, 73 | "another_bool": false 74 | } -------------------------------------------------------------------------------- /data/1/data.beve: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/beve-org/beve/eeeb779cb77c9f7186cff2713b1ed233c3b7a0cd/data/1/data.beve -------------------------------------------------------------------------------- /data/1/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "complex_floats": [ 3 | [ 4 | 1, 5 | 0.5 6 | ], 7 | [ 8 | 0.1, 9 | 2 10 | ] 11 | ], 12 | "complex_int32_t": [ 13 | [ 14 | -1, 15 | 5 16 | ], 17 | [ 18 | 7, 19 | -9 20 | ] 21 | ] 22 | } -------------------------------------------------------------------------------- /example/example.js: -------------------------------------------------------------------------------- 1 | // Reference: https://github.com/stephenberry/beve 2 | 3 | const fs = require('fs'); 4 | 5 | const beve = require('../javascript/beve.js'); 6 | 7 | // Function to read the .beve file into a buffer 8 | function loadFileToBuffer(filename) { 9 | if (!filename) { 10 | throw new Error('No filename provided.'); 11 | } 12 | 13 | const buffer = fs.readFileSync(filename); 14 | return new Uint8Array(buffer); 15 | } 16 | 17 | try { 18 | const filename = './example.beve'; 19 | const buffer = loadFileToBuffer(filename); 20 | var data = beve.read_beve(buffer); 21 | /*delete data['fixed_object']; 22 | delete data['fixed_name_object']; 23 | delete data['another_object']['string']; 24 | delete data['another_object']['another_string']; 25 | delete data['another_object']['boolean'];*/ 26 | console.log(data); 27 | console.log('--------------'); 28 | console.log(' '); 29 | //console.log(JSON.stringify(data)); 30 | //write_beve(data, '../example/examplejs.beve'); 31 | const binary = beve.write_beve(data); 32 | data = beve.read_beve(binary); 33 | console.log(data); 34 | } catch (error) { 35 | console.error('Error:', error); 36 | } -------------------------------------------------------------------------------- /extensions.md: -------------------------------------------------------------------------------- 1 | # BEVE Extensions 2 | 3 | Following the first three HEADER bits, the next five bits denote various extensions. These extensions are not expected to be implemented in every parser/serializer, but they provide convenient binary storage for more specialized use cases, such as variants, matrices, and complex numbers. 4 | 5 | ```c++ 6 | 0 -> data delimiter // for specs like Newline Delimited JSON 7 | 1 -> type tag // for variant like structures 8 | 2 -> matrices 9 | 3 -> complex numbers 10 | ``` 11 | 12 | ## 0 - Data Delimiter 13 | 14 | Used to separate chunks of data to match specifications like [NDJSON](http://ndjson.org). 15 | 16 | When converted to JSON this should add a new line (`'\n'`) character to the JSON. 17 | 18 | ## 1 - Type Tag (Variants) 19 | 20 | Expects a subsequent compressed unsigned integer to denote a type tag. A compressed SIZE indicator is used to efficiently store the tag. 21 | 22 | Layout : `HEADER | SIZE (i.e. type tag) | VALUE` 23 | 24 | The converted JSON format should look like: 25 | 26 | ```json 27 | { 28 | "index": 0, 29 | "value": "the JSON value" 30 | } 31 | ``` 32 | 33 | The `"index"` should refer to an array of types, from zero to one less than the count of types. 34 | 35 | The `"value"` is any JSON value. 36 | 37 | ## 2 - Matrices 38 | 39 | Matrices can be stored as object or array types. However, this tag provides a more compact mechanism to introspect matrices. 40 | 41 | Matrices add a one byte MATRIX HEADER. 42 | 43 | The first bit of the matrix header denotes the data layout of the matrix. 44 | 45 | ```c++ 46 | 0 -> layout_right // row-major 47 | 1 -> layout_left // column-major 48 | ``` 49 | 50 | Layout: `HEADER | MATRIX HEADER | EXTENTS | VALUE` 51 | 52 | EXTENTS are written out as a typed array of unsigned integers. 53 | 54 | > The VALUE in the matrix must be a typed array of numerical data. 55 | 56 | The converted JSON format should look like: 57 | 58 | ```json 59 | { 60 | "layout": "layout_right", 61 | "extents": [3, 3], 62 | "value": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] 63 | } 64 | ``` 65 | 66 | ## 3 - Complex Numbers 67 | 68 | An additional COMPLEX HEADER byte is used. 69 | 70 | - Complex numbers are stored as pairs of numerical types. 71 | 72 | The first three bits denote whether this is a single complex number or a complex array. 73 | 74 | ```c++ 75 | 0 -> complex number 76 | 1 -> complex array 77 | ``` 78 | 79 | For a single complex number the layout is: `HEADER | COMPLEX HEADER | DATA` 80 | 81 | > A complex value is a pair of numbers. 82 | 83 | For a complex array the layout is: `HEADER | COMPLEX HEADER | SIZE | DATA` 84 | 85 | > Three bits are used to align the left bits with the layouts for numbers. 86 | 87 | The next two bits denote the numerical type: 88 | 89 | ```c++ 90 | 0 -> floating point 0b000'00'000 91 | 1 -> signed integer 0b000'01'000 92 | 2 -> unsigned integer 0b000'10'000 93 | ``` 94 | 95 | The next three bits are used to indicate the BYTE COUNT. This is the same specification for BEVE numbers. 96 | 97 | The converted JSON format should look like: 98 | 99 | ```json 100 | [1, 2] // for a complex number 101 | [[1, 2], [2.0, 3]] // for a complex array 102 | ``` 103 | 104 | -------------------------------------------------------------------------------- /golang/go.mod: -------------------------------------------------------------------------------- 1 | module beve 2 | 3 | go 1.22.3 4 | -------------------------------------------------------------------------------- /golang/main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "encoding/binary" 5 | "errors" 6 | "fmt" 7 | "math" 8 | "math/big" 9 | "os" 10 | ) 11 | 12 | type Beve struct { 13 | buffer []byte 14 | cursor int 15 | } 16 | 17 | // Helper functions to simulate reading different data types from the buffer 18 | func (b *Beve) readUInt8() uint8 { 19 | val := b.buffer[b.cursor] 20 | b.cursor++ 21 | return val 22 | } 23 | 24 | func (b *Beve) readInt8() int8 { 25 | return int8(b.readUInt8()) 26 | } 27 | 28 | func (b *Beve) readUInt16() uint16 { 29 | val := binary.LittleEndian.Uint16(b.buffer[b.cursor:]) 30 | b.cursor += 2 31 | return val 32 | } 33 | 34 | func (b *Beve) readInt16() int16 { 35 | return int16(b.readUInt16()) 36 | } 37 | 38 | // ... (similar functions for readUInt32, readInt32, readFloat, readDouble, etc.) 39 | func (b *Beve) readUInt32() uint32 { 40 | val := binary.LittleEndian.Uint32(b.buffer[b.cursor:]) 41 | b.cursor += 4 42 | return val 43 | } 44 | func (b *Beve) readInt32() int32 { 45 | return int32(b.readUInt32()) 46 | } 47 | func (b *Beve) readUInt64() uint64 { 48 | val := binary.LittleEndian.Uint64(b.buffer[b.cursor:]) 49 | b.cursor += 8 50 | return val 51 | } 52 | 53 | func (b *Beve) readInt64() int64 { 54 | return int64(b.readUInt64()) 55 | } 56 | func (b *Beve) readFloat() float32 { 57 | return math.Float32frombits(b.readUInt32()) 58 | } 59 | func (b *Beve) readDouble() float64 { 60 | return math.Float64frombits(b.readUInt64()) 61 | } 62 | func (b *Beve) readBigInt64() *big.Int { 63 | // Read and construct big.Int from buffer 64 | bytes := make([]byte, 8) 65 | copy(bytes, b.buffer[b.cursor:b.cursor+8]) 66 | b.cursor += 8 67 | return new(big.Int).SetBytes(bytes) 68 | } 69 | 70 | func (b *Beve) readBigUInt64() *big.Int { 71 | // Read and construct big.Int from buffer 72 | bytes := make([]byte, 8) 73 | copy(bytes, b.buffer[b.cursor:b.cursor+8]) 74 | b.cursor += 8 75 | return new(big.Int).SetBytes(bytes) 76 | } 77 | func (b *Beve) readFloat32() float32 { 78 | bits := binary.LittleEndian.Uint32(b.buffer[b.cursor:]) 79 | b.cursor += 4 80 | return math.Float32frombits(bits) 81 | } 82 | 83 | func (b *Beve) readFloat64() float64 { 84 | bits := binary.LittleEndian.Uint64(b.buffer[b.cursor:]) 85 | b.cursor += 8 86 | return math.Float64frombits(bits) 87 | } 88 | 89 | func (b *Beve) readCompressed() int { 90 | header := b.buffer[b.cursor] 91 | b.cursor++ 92 | config := header & 0b00000011 93 | 94 | switch config { 95 | case 0: 96 | return int(header >> 2) 97 | case 1: 98 | value := binary.LittleEndian.Uint16(b.buffer[b.cursor:]) 99 | b.cursor += 2 100 | return int(value >> 2) 101 | case 2: 102 | value := binary.LittleEndian.Uint32(b.buffer[b.cursor:]) 103 | b.cursor += 4 104 | return int(value >> 2) 105 | case 3: 106 | var val big.Int 107 | for i := 0; i < 8; i++ { 108 | val.Or(&val, new(big.Int).Lsh( 109 | new(big.Int).SetUint64(uint64(b.buffer[b.cursor])), 110 | uint(8*i), 111 | )) 112 | b.cursor++ 113 | } 114 | val.Rsh(&val, 2) 115 | return int(val.Int64()) // Büyük sayıyı int'e dönüştür 116 | default: 117 | return 0 118 | } 119 | } 120 | 121 | func (b *Beve) readString() string { 122 | size := b.readCompressed() 123 | // Assuming you have a way to decode the string from the buffer 124 | str := string(b.buffer[b.cursor : b.cursor+size]) 125 | b.cursor += size 126 | return str 127 | } 128 | 129 | // TODO Implement readValue function 130 | func reshape[T int8 | int | int16 | int32 | int64 | uint8 | uint | uint16 | uint32 | uint64 | float32 | float64](data []T, rows, cols int) [][]T { 131 | // Varsayalım ki data, []float64 şeklinde bir dilim olarak geliyor 132 | 133 | if rows*cols != len(data) { 134 | panic("Invalid dimensions for matrix reshape") 135 | } 136 | 137 | result := make([][]T, rows) 138 | for i := range result { 139 | result[i] = make([]T, cols) 140 | copy(result[i], data[i*cols:(i+1)*cols]) 141 | } 142 | return result 143 | } 144 | 145 | // TODO Implement readComplex function 146 | func (b *Beve) readComplex() complex128 { 147 | real := b.readFloat64() 148 | imag := b.readFloat64() 149 | return complex(real, imag) 150 | } 151 | 152 | func (b *Beve) readValue() (interface{}, error) { 153 | header := b.buffer[b.cursor] 154 | b.cursor++ 155 | config := []uint8{1, 2, 4, 8} 156 | typ := header & 0b00000111 157 | 158 | switch typ { 159 | case 0: // null or boolean 160 | isBool := (header & 0b00001000) >> 3 161 | if isBool > 0 { 162 | return (header&0b11110000)>>4 > 0, nil 163 | } else { 164 | return nil, nil 165 | } 166 | case 1: // number 167 | numType := (header & 0b00011000) >> 3 168 | isFloat := numType == 0 169 | isSigned := numType == 1 170 | byteCountIndex := (header & 0b11100000) >> 5 171 | byteCount := config[byteCountIndex] 172 | 173 | if isFloat { 174 | switch byteCount { 175 | case 4: 176 | return b.readFloat(), nil 177 | case 8: 178 | return b.readDouble(), nil 179 | } 180 | } else { 181 | 182 | if isSigned { 183 | switch byteCount { 184 | case 1: 185 | return b.readInt8(), nil 186 | case 2: 187 | return b.readInt16(), nil 188 | case 4: 189 | return b.readInt32(), nil 190 | case 8: 191 | return b.readInt64(), nil 192 | } 193 | } else { 194 | switch byteCount { 195 | case 1: 196 | return b.readUInt8(), nil 197 | case 2: 198 | return b.readUInt16(), nil 199 | case 4: 200 | return b.readUInt32(), nil 201 | case 8: 202 | return b.readUInt64(), nil 203 | } 204 | } 205 | } 206 | case 2: // string 207 | size := b.readCompressed() 208 | // Assuming you have a way to decode the string from the buffer 209 | str := string(b.buffer[b.cursor : b.cursor+size]) 210 | b.cursor += size 211 | return str, nil 212 | 213 | case 3: // object 214 | keyType := (header & 0b00011000) >> 3 215 | isString := keyType == 0 216 | // isSigned := keyType == 1 // Kullanılmayan değişken, kaldırıldı 217 | // byteCountIndex := (header & 0b11100000) >> 5 218 | // byteCount := config[byteCountIndex] 219 | N := b.readCompressed() 220 | 221 | // objectData'yı Go'da map[string]interface{} olarak temsil edelim 222 | objectData := make(map[string]interface{}) 223 | 224 | for i := 0; i < N; i++ { 225 | if isString { 226 | size := b.readCompressed() 227 | key := string(b.buffer[b.cursor : b.cursor+size]) 228 | b.cursor += size 229 | value, err := b.readValue() 230 | if err != nil { 231 | return nil, err // Hata durumunu işleyin 232 | } 233 | objectData[key] = value 234 | } else { 235 | return nil, errors.New("TODO: support integer keys") // Hata fırlatın 236 | } 237 | } 238 | return objectData, nil 239 | 240 | case 4: // typed array 241 | numType := (header & 0b00011000) >> 3 242 | isFloat := numType == 0 243 | isSigned := numType == 1 244 | byteCountIndexArray := (header & 0b11100000) >> 5 245 | byteCountArray := config[byteCountIndexArray] 246 | 247 | if numType == 3 { 248 | isString := (header & 0b00100000) >> 5 249 | if isString != 0 { // ">" ile kontrol edebiliriz, "!= 0" ile aynı 250 | N := b.readCompressed() 251 | array := make([]string, N) 252 | for i := 0; i < N; i++ { 253 | size := b.readCompressed() 254 | // UTF-8 varsayımı ile stringe çevirme 255 | array[i] = string(b.buffer[b.cursor : b.cursor+size]) 256 | b.cursor += size 257 | } 258 | return array, nil 259 | } else { 260 | return nil, errors.New("Boolean array support not implemented") 261 | } 262 | } else if isFloat { 263 | N := b.readCompressed() 264 | var array interface{} 265 | 266 | switch byteCountArray { 267 | case 4: 268 | array = make([]float32, N) 269 | for i := 0; i < N; i++ { 270 | array.([]float32)[i] = b.readFloat32() 271 | } 272 | case 8: 273 | array = make([]float64, N) 274 | for i := 0; i < N; i++ { 275 | array.([]float64)[i] = b.readFloat64() 276 | } 277 | default: 278 | return nil, errors.New("Unsupported float size") 279 | } 280 | // array tip dönüşümü 281 | return array, nil 282 | } else { 283 | N := b.readCompressed() 284 | // array tip belirleme 285 | var array interface{} 286 | if isSigned { 287 | switch byteCountArray { 288 | case 1: 289 | array = make([]int8, N) 290 | for i := 0; i < N; i++ { 291 | array.([]int8)[i] = b.readInt8() 292 | } 293 | case 2: 294 | array = make([]int16, N) 295 | for i := 0; i < N; i++ { 296 | array.([]int16)[i] = b.readInt16() 297 | } 298 | case 4: 299 | array = make([]int32, N) 300 | for i := 0; i < N; i++ { 301 | array.([]int32)[i] = b.readInt32() 302 | } 303 | case 8: 304 | array = make([]*big.Int, N) 305 | for i := 0; i < N; i++ { 306 | array.([]*big.Int)[i] = b.readBigInt64() 307 | } 308 | default: 309 | return nil, errors.New("Unsupported signed integer size") 310 | } 311 | } else { // unsigned 312 | switch byteCountArray { 313 | case 1: 314 | array = make([]uint8, N) 315 | for i := 0; i < N; i++ { 316 | array.([]uint8)[i] = b.readUInt8() 317 | } 318 | case 2: 319 | array = make([]uint16, N) 320 | for i := 0; i < N; i++ { 321 | array.([]uint16)[i] = b.readUInt16() 322 | } 323 | case 4: 324 | array = make([]uint32, N) 325 | for i := 0; i < N; i++ { 326 | array.([]uint32)[i] = b.readUInt32() 327 | } 328 | case 8: 329 | array = make([]*big.Int, N) 330 | for i := 0; i < N; i++ { 331 | array.([]*big.Int)[i] = b.readBigUInt64() 332 | } 333 | default: 334 | return nil, errors.New("Unsupported unsigned integer size") 335 | } 336 | } 337 | return array, nil 338 | } 339 | 340 | case 5: // untyped array 341 | N := b.readCompressed() 342 | arr := make([]interface{}, N) 343 | for i := 0; i < N; i++ { 344 | val, err := b.readValue() 345 | if err != nil { 346 | return nil, err 347 | } 348 | arr[i] = val 349 | } 350 | return arr, nil 351 | 352 | case 6: // extensions 353 | extension := (header & 0b11111000) >> 3 354 | switch extension { 355 | case 1: // variants 356 | _ = b.readCompressed() // Skip variant tag 357 | return b.readValue() 358 | case 2: // matrices 359 | layout := b.buffer[b.cursor] & 0b00000001 360 | b.cursor++ 361 | switch layout { 362 | case 0: // row major 363 | return nil, errors.New("Row major matrix layout not implemented") 364 | case 1: // column major 365 | extents, err := b.readValue() 366 | if err != nil { 367 | return nil, err 368 | } 369 | extentsSlice, ok := extents.([]interface{}) 370 | if !ok || len(extentsSlice) != 2 { 371 | return nil, errors.New("Invalid extents for matrix") 372 | } 373 | rows, _ := extentsSlice[0].(int) 374 | cols, _ := extentsSlice[1].(int) 375 | 376 | matrixData, err := b.readValue() 377 | if err != nil { 378 | return nil, err 379 | } 380 | // reshape function implementation needed here 381 | switch matrixData.(type) { 382 | case []float64: 383 | return reshape[float64](matrixData.([]float64), rows, cols), nil 384 | case []int: 385 | return reshape[int](matrixData.([]int), rows, cols), nil 386 | case []int16: 387 | return reshape[int16](matrixData.([]int16), rows, cols), nil 388 | case []int32: 389 | return reshape[int32](matrixData.([]int32), rows, cols), nil 390 | case []int64: 391 | return reshape[int64](matrixData.([]int64), rows, cols), nil 392 | case []uint8: 393 | return reshape[uint8](matrixData.([]uint8), rows, cols), nil 394 | case []uint: 395 | return reshape[uint](matrixData.([]uint), rows, cols), nil 396 | case []uint16: 397 | return reshape[uint16](matrixData.([]uint16), rows, cols), nil 398 | case []uint32: 399 | return reshape[uint32](matrixData.([]uint32), rows, cols), nil 400 | case []uint64: 401 | return reshape[uint64](matrixData.([]uint64), rows, cols), nil 402 | case []float32: 403 | return reshape[float32](matrixData.([]float32), rows, cols), nil 404 | default: 405 | return nil, errors.New("Unsupported matrix data type") 406 | } 407 | default: 408 | return nil, errors.New("Unsupported matrix layout") 409 | } 410 | case 3: // complex numbers 411 | return b.readComplex(), nil // readComplex function implementation needed here 412 | default: 413 | return nil, errors.New("Unsupported extension") 414 | } 415 | 416 | default: 417 | return nil, errors.New("Unsupported type") 418 | } 419 | 420 | return nil, errors.New("Shouldn't reach here") 421 | } 422 | 423 | // Writer struct 424 | type Writer struct { 425 | buffer []byte 426 | offset int 427 | } 428 | 429 | func NewWriter(size int) *Writer { 430 | if size <= 0 { 431 | size = 256 // Default size 432 | } 433 | return &Writer{buffer: make([]byte, size), offset: 0} 434 | } 435 | 436 | func (w *Writer) ensureCapacity(size int) { 437 | if w.offset+size > len(w.buffer) { 438 | newBuffer := make([]byte, (len(w.buffer)+size)*2) 439 | copy(newBuffer, w.buffer) 440 | w.buffer = newBuffer 441 | } 442 | } 443 | 444 | // append uint8 445 | func (w *Writer) appendUint8(value uint8) error { 446 | if value > 255 { 447 | return errors.New("Value must be an integer between 0 and 255") 448 | } 449 | w.ensureCapacity(1) 450 | w.buffer[w.offset] = value 451 | w.offset++ 452 | return nil 453 | } 454 | 455 | // append uint16 456 | func (w *Writer) appendUint16(value uint16) error { 457 | if value > 65535 { 458 | return errors.New("Value must be an integer between 0 and 65535") 459 | } 460 | w.ensureCapacity(2) 461 | binary.LittleEndian.PutUint16(w.buffer[w.offset:], value) 462 | w.offset += 2 463 | return nil 464 | } 465 | 466 | // append uint32 467 | func (w *Writer) appendUint32(value uint32) error { 468 | if value > 4294967295 { 469 | return errors.New("Value must be an integer between 0 and 4294967295") 470 | } 471 | w.ensureCapacity(4) 472 | binary.LittleEndian.PutUint32(w.buffer[w.offset:], value) 473 | w.offset += 4 474 | return nil 475 | } 476 | 477 | // append uint64 478 | func (w *Writer) appendUint64(value *big.Int) error { 479 | if value.Cmp(big.NewInt(0)) == -1 || value.Cmp(new(big.Int).Lsh(big.NewInt(1), 64)) == 1 { 480 | return errors.New("Value must be an integer between 0 and 18446744073709551615") 481 | } 482 | w.ensureCapacity(8) 483 | low := new(big.Int).And(value, big.NewInt(0xffffffff)) 484 | high := new(big.Int).Rsh(value, 32) 485 | 486 | binary.LittleEndian.PutUint32(w.buffer[w.offset:], uint32(low.Uint64())) 487 | binary.LittleEndian.PutUint32(w.buffer[w.offset+4:], uint32(high.Uint64())) 488 | w.offset += 8 489 | return nil 490 | } 491 | 492 | // append 493 | func (w *Writer) Append(value interface{}) error { 494 | switch v := value.(type) { 495 | case []interface{}: 496 | for _, element := range v { 497 | err := w.Append(element) 498 | if err != nil { 499 | return err 500 | } 501 | } 502 | case string: 503 | bytes := []byte(v) 504 | w.ensureCapacity(len(bytes)) 505 | copy(w.buffer[w.offset:], bytes) 506 | w.offset += len(bytes) 507 | case int, int8, int16, int32: 508 | w.ensureCapacity(4) 509 | binary.LittleEndian.PutUint32(w.buffer[w.offset:], uint32(v.(int32))) 510 | w.offset += 4 511 | case float32, float64: 512 | w.ensureCapacity(8) 513 | binary.LittleEndian.PutUint64(w.buffer[w.offset:], math.Float64bits(v.(float64))) 514 | w.offset += 8 515 | default: 516 | return errors.New("Unsupported value type") 517 | } 518 | return nil 519 | } 520 | 521 | // writeBeve 522 | func WriteBeve(data interface{}) ([]byte, error) { 523 | writer := NewWriter(0) // Use default size 524 | err := writeValue(writer, data) 525 | return writer.buffer[:writer.offset], err // Return the actual used part of the buffer 526 | } 527 | 528 | // writeValue 529 | func writeValue(writer *Writer, value interface{}) error { 530 | switch v := value.(type) { 531 | case []float64: 532 | writer.appendUint8(0b01100000 | 4) // float64_t, 8 bytes 533 | writeCompressed(writer, len(v)) 534 | for _, f := range v { 535 | writer.Append(f) 536 | } 537 | case []int: 538 | writer.appendUint8(0b01001000 | 4) // int32_t, 4 bytes 539 | writeCompressed(writer, len(v)) 540 | for _, i := range v { 541 | writer.Append(i) 542 | } 543 | case bool: 544 | if v { 545 | writer.appendUint8(0b00011000) 546 | } else { 547 | writer.appendUint8(0b00001000) 548 | } 549 | case int, int8, int16, int32, int64, uint, uint8, uint16, uint32, uint64, float32, float64: 550 | if v.(float64)-float64(int(v.(float64))) != 0 { 551 | writer.appendUint8(0b01100000) 552 | } else { 553 | writer.appendUint8(0b01001000) 554 | } 555 | writer.Append(v) 556 | case string: 557 | writer.appendUint8(2) 558 | writeCompressed(writer, len(v)) 559 | writer.Append(v) 560 | case []interface{}: 561 | writer.appendUint8(5) 562 | writeCompressed(writer, len(v)) 563 | for _, val := range v { 564 | writeValue(writer, val) 565 | } 566 | case map[string]interface{}: 567 | writer.appendUint8(3) // Assume string keys 568 | writeCompressed(writer, len(v)) 569 | for key, val := range v { 570 | writeCompressed(writer, len(key)) 571 | writer.Append(key) 572 | writeValue(writer, val) 573 | } 574 | default: 575 | return errors.New("Unsupported data type") 576 | } 577 | 578 | return nil 579 | } 580 | 581 | // writeCompressed 582 | func writeCompressed(writer *Writer, N int) { 583 | if N < 64 { 584 | compressed := (N << 2) | 0 585 | writer.appendUint8(uint8(compressed)) 586 | } else if N < 16384 { 587 | compressed := (N << 2) | 1 588 | writer.appendUint16(uint16(compressed)) 589 | } else if N < 1073741824 { 590 | compressed := (N << 2) | 2 591 | writer.appendUint32(uint32(compressed)) 592 | } else { 593 | compressed := (N << 2) | 3 594 | writer.appendUint64(big.NewInt(int64(compressed))) 595 | } 596 | } 597 | 598 | func ReadFromBuffer(buffer []byte) (interface{}, error) { 599 | beve := Beve{buffer: buffer, cursor: 0} 600 | return beve.readValue() 601 | } 602 | 603 | func WriteToBuffer(data interface{}) ([]byte, error) { 604 | writer := NewWriter(0) 605 | err := writeValue(writer, data) 606 | if err != nil { 607 | return nil, err 608 | } 609 | return writer.buffer[:writer.offset], nil 610 | } 611 | 612 | func ReadFromFile(filename string) (interface{}, error) { 613 | buffer, err := os.ReadFile(filename) 614 | if err != nil { 615 | return nil, err 616 | } 617 | return ReadFromBuffer(buffer) 618 | } 619 | 620 | func WriteToFile(filename string, data interface{}) error { 621 | buffer, err := WriteToBuffer(data) 622 | if err != nil { 623 | return err 624 | } 625 | return os.WriteFile(filename, buffer, 0644) 626 | } 627 | 628 | // TODO toJSON,fromJSON 629 | 630 | func main() { 631 | // ... initialize the buffer 632 | buffer := []byte{0x81, 0x35, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00} 633 | beve := Beve{buffer: buffer, cursor: 0} 634 | value, err := beve.readValue() 635 | if err != nil { 636 | fmt.Println("Error reading value:", err) 637 | return 638 | } 639 | fmt.Println("Decoded value:", value) 640 | } 641 | -------------------------------------------------------------------------------- /javascript/beve.js: -------------------------------------------------------------------------------- 1 | // Reference: https://github.com/stephenberry/beve 2 | 3 | (function (root, factory) { 4 | if (typeof define === 'function' && define.amd) { 5 | // AMD. Register as an anonymous module. 6 | define([], factory); 7 | } else if (typeof module === 'object' && module.exports) { 8 | // Node. Does not work with strict CommonJS, but 9 | // only CommonJS-like environments that support module.exports, 10 | // like Node. 11 | module.exports = factory(); 12 | } else { 13 | // Browser globals (root is window) 14 | root.otherFile = factory(); 15 | } 16 | }(typeof self !== 'undefined' ? self : this, function () { 17 | // Reading BEVE 18 | function read_beve(buffer) { 19 | if (!buffer || !(buffer instanceof Uint8Array)) { 20 | throw new Error('Invalid buffer provided.'); 21 | } 22 | 23 | let cursor = 0; 24 | 25 | function read_value() { 26 | const header = buffer[cursor++]; 27 | const config = [1, 2, 4, 8]; 28 | const type = header & 0b00000111; 29 | 30 | switch (type) { 31 | case 0: // null or boolean 32 | { 33 | const is_bool = (header & 0b00001000) >> 3; 34 | if (is_bool) { 35 | return Boolean((header & 0b11110000) >> 4); 36 | } else { 37 | return null; 38 | } 39 | } 40 | case 1: // number 41 | { 42 | const num_type = (header & 0b00011000) >> 3; 43 | const is_float = num_type === 0; 44 | const is_signed = num_type === 1; 45 | const byte_count_index = (header & 0b11100000) >> 5; 46 | const byte_count = config[byte_count_index]; 47 | 48 | if (is_float) { 49 | switch (byte_count) { 50 | case 4: 51 | return readFloat(); 52 | case 8: 53 | return readDouble(); 54 | } 55 | } else { 56 | if (is_signed) { 57 | switch (byte_count) { 58 | case 1: 59 | return readInt8(); 60 | case 2: 61 | return readInt16(); 62 | case 4: 63 | return readInt32(); 64 | case 8: 65 | return readBigInt64(); 66 | } 67 | } else { 68 | switch (byte_count) { 69 | case 1: 70 | return readUInt8(); 71 | case 2: 72 | return readUInt16(); 73 | case 4: 74 | return readUInt32(); 75 | case 8: 76 | return readBigUInt64(); 77 | } 78 | } 79 | } 80 | break; 81 | } 82 | case 2: // string 83 | { 84 | const size = read_compressed(); 85 | const str = new TextDecoder().decode(buffer.subarray(cursor, cursor + size)); 86 | cursor += size; 87 | return str; 88 | } 89 | case 3: // object 90 | { 91 | const key_type = (header & 0b00011000) >> 3; 92 | const is_string = key_type === 0; 93 | const is_signed = key_type === 1; 94 | const byte_count_index = (header & 0b11100000) >> 5; 95 | const byte_count = config[byte_count_index]; 96 | const N = read_compressed(); 97 | const objectData = {}; 98 | 99 | for (let i = 0; i < N; ++i) { 100 | if (is_string) { 101 | const size = read_compressed(); 102 | const key = new TextDecoder().decode(buffer.subarray(cursor, cursor + size)); 103 | cursor += size; 104 | objectData[key] = read_value(); 105 | } else { 106 | throw new Error('TODO: support integer keys'); 107 | } 108 | } 109 | 110 | return objectData; 111 | } 112 | case 4: // typed array 113 | { 114 | const num_type = (header & 0b00011000) >> 3; 115 | const is_float = num_type === 0; 116 | const is_signed = num_type === 1; 117 | const byte_count_index_array = (header & 0b11100000) >> 5; 118 | const byte_count_array = config[byte_count_index_array]; 119 | 120 | if (num_type === 3) { 121 | const is_string = (header & 0b00100000) >> 5; 122 | if (is_string) { 123 | const N = read_compressed(); 124 | const array = new Array(N); 125 | for (let i = 0; i < N; ++i) { 126 | const size = read_compressed(); 127 | const str = new TextDecoder().decode(buffer.subarray(cursor, cursor + size)); 128 | cursor += size; 129 | array[i] = str; 130 | } 131 | return array; 132 | } else { 133 | throw new Error("Boolean array support not implemented"); 134 | } 135 | } else if (is_float) { 136 | const N = read_compressed(); 137 | const array = new Array(N); 138 | switch (byte_count_array) { 139 | case 4: 140 | for (let i = 0; i < N; ++i) { 141 | array[i] = readFloat(); 142 | } 143 | break; 144 | case 8: 145 | for (let i = 0; i < N; ++i) { 146 | array[i] = readDouble(); 147 | } 148 | break; 149 | } 150 | return array; 151 | } else { 152 | const N = read_compressed(); 153 | const array = new Array(N); 154 | 155 | if (is_signed) { 156 | switch (byte_count_array) { 157 | case 1: 158 | for (let i = 0; i < N; ++i) { 159 | array[i] = readInt8(); 160 | } 161 | break; 162 | case 2: 163 | for (let i = 0; i < N; ++i) { 164 | array[i] = readInt16(); 165 | } 166 | break; 167 | case 4: 168 | for (let i = 0; i < N; ++i) { 169 | array[i] = readInt32(); 170 | } 171 | break; 172 | case 8: 173 | for (let i = 0; i < N; ++i) { 174 | array[i] = readBigInt64(); 175 | } 176 | break; 177 | } 178 | } else { 179 | switch (byte_count_array) { 180 | case 1: 181 | for (let i = 0; i < N; ++i) { 182 | array[i] = readUInt8(); 183 | } 184 | break; 185 | case 2: 186 | for (let i = 0; i < N; ++i) { 187 | array[i] = readUInt16(); 188 | } 189 | break; 190 | case 4: 191 | for (let i = 0; i < N; ++i) { 192 | array[i] = readUInt32(); 193 | } 194 | break; 195 | case 8: 196 | for (let i = 0; i < N; ++i) { 197 | array[i] = readBigUInt64(); 198 | } 199 | break; 200 | } 201 | } 202 | return array; 203 | } 204 | } 205 | case 5: // untyped array 206 | { 207 | const N = read_compressed(); 208 | const unarray = new Array(N); 209 | 210 | for (let i = 0; i < N; ++i) { 211 | unarray[i] = read_value(); 212 | } 213 | 214 | return unarray; 215 | } 216 | case 6: // extensions 217 | { 218 | const extension = (header & 0b11111000) >> 3; 219 | switch (extension) { 220 | case 1: // variants 221 | read_compressed(); // Skipping variant tag 222 | return read_value(); 223 | case 2: // matrices 224 | const layout = buffer[cursor++] & 0b00000001; 225 | switch (layout) { 226 | case 0: // row major 227 | throw new Error('TODO: add row major support'); 228 | case 1: // column major 229 | const extents = read_value(); 230 | const matrix_data = read_value(); 231 | return reshape(matrix_data, extents[0], extents[1]); 232 | default: 233 | throw new Error('Unsupported layout'); 234 | } 235 | case 3: // complex numbers 236 | return read_complex(); 237 | default: 238 | throw new Error('Unsupported extension'); 239 | } 240 | } 241 | default: 242 | throw new Error('Unsupported type'); 243 | } 244 | } 245 | 246 | function readFloat() { 247 | const value = new DataView(buffer.buffer, cursor, 4).getFloat32(0, true); 248 | cursor += 4; 249 | return value; 250 | } 251 | 252 | function readDouble() { 253 | const value = new DataView(buffer.buffer, cursor, 8).getFloat64(0, true); 254 | cursor += 8; 255 | return value; 256 | } 257 | 258 | function readInt8() { 259 | const value = new DataView(buffer.buffer, cursor, 1).getInt8(0); 260 | cursor += 1; 261 | return value; 262 | } 263 | 264 | function readInt16() { 265 | const value = new DataView(buffer.buffer, cursor, 2).getInt16(0, true); 266 | cursor += 2; 267 | return value; 268 | } 269 | 270 | function readInt32() { 271 | const value = new DataView(buffer.buffer, cursor, 4).getInt32(0, true); 272 | cursor += 4; 273 | return value; 274 | } 275 | 276 | function readBigInt64() { 277 | const value = new DataView(buffer.buffer, cursor, 8).getBigInt64(0, true); 278 | cursor += 8; 279 | return value; 280 | } 281 | 282 | function readUInt8() { 283 | const value = new DataView(buffer.buffer, cursor, 1).getUint8(0); 284 | cursor += 1; 285 | return value; 286 | } 287 | 288 | function readUInt16() { 289 | const value = new DataView(buffer.buffer, cursor, 2).getUint16(0, true); 290 | cursor += 2; 291 | return value; 292 | } 293 | 294 | function readUInt32() { 295 | const value = new DataView(buffer.buffer, cursor, 4).getUint32(0, true); 296 | cursor += 4; 297 | return value; 298 | } 299 | 300 | function readBigUInt64() { 301 | const value = new DataView(buffer.buffer, cursor, 8).getBigUint64(0, true); 302 | cursor += 8; 303 | return value; 304 | } 305 | 306 | function read_compressed() { 307 | const header = buffer[cursor++]; 308 | const config = header & 0b00000011; 309 | 310 | switch (config) { 311 | case 0: 312 | return header >> 2; 313 | case 1: { 314 | const h = new DataView(buffer.buffer, cursor, 2); 315 | cursor += 2; 316 | return (h.getUint16(0, true)) >> 2; 317 | } 318 | case 2: { 319 | const h = new DataView(buffer.buffer, cursor, 4); 320 | cursor += 4; 321 | return (h.getUint32(0, true)) >> 2; 322 | } 323 | case 3: { 324 | let val = BigInt(0); 325 | for (let i = 0; i < 8; ++i) { 326 | val |= BigInt(buffer[cursor++]) << BigInt(8 * i); 327 | } 328 | return Number(val >> BigInt(2)); 329 | } 330 | default: 331 | return 0; 332 | } 333 | } 334 | 335 | function read_complex() { 336 | const real = read_value(); 337 | const imag = read_value(); 338 | return { real, imag }; 339 | } 340 | 341 | function reshape(array, rows, cols) { 342 | const reshaped = []; 343 | for (let i = 0; i < rows; ++i) { 344 | reshaped.push(array.slice(i * cols, (i + 1) * cols)); 345 | } 346 | return reshaped; 347 | } 348 | 349 | return read_value(); 350 | } 351 | 352 | class Writer { 353 | constructor(size = 256) { 354 | this.buffer = new Uint8Array(size); 355 | this.offset = 0; 356 | } 357 | 358 | ensureCapacity(size) { 359 | if (this.offset + size > this.buffer.length) { 360 | let newBuffer = new Uint8Array((this.buffer.length + size) * 2); 361 | newBuffer.set(this.buffer); 362 | this.buffer = newBuffer; 363 | } 364 | } 365 | 366 | append_uint8(value) { 367 | if (Number.isInteger(value) && value >= 0 && value <= 255) { 368 | this.ensureCapacity(1); 369 | this.buffer[this.offset] = value; 370 | this.offset += 1; 371 | } else { 372 | throw new Error('Value must be an integer between 0 and 255'); 373 | } 374 | } 375 | 376 | append_uint16(value) { 377 | if (Number.isInteger(value) && value >= 0 && value <= 65535) { 378 | // 16-bit unsigned integer 379 | this.ensureCapacity(2); 380 | let view = new DataView(this.buffer.buffer); 381 | view.setUint16(this.offset, value, true); // little-endian 382 | this.offset += 2; 383 | } else { 384 | throw new Error('Value must be an integer between 0 and 65535'); 385 | } 386 | } 387 | 388 | append_uint32(value) { 389 | if (Number.isInteger(value) && value >= 0 && value <= 4294967295) { 390 | // 32-bit unsigned integer 391 | this.ensureCapacity(4); 392 | let view = new DataView(this.buffer.buffer); 393 | view.setUint32(this.offset, value, true); // little-endian 394 | this.offset += 4; 395 | } else { 396 | throw new Error('Value must be an integer between 0 and 4294967295'); 397 | } 398 | } 399 | 400 | append_uint64(value) { 401 | if (Number.isInteger(value) && value >= 0 && value <= 18446744073709551615) { 402 | // 64-bit unsigned integer 403 | this.ensureCapacity(8); 404 | let high = Math.floor(value / 0x100000000); 405 | let low = value % 0x100000000; 406 | let view = new DataView(this.buffer.buffer); 407 | view.setUint32(this.offset, low, true); // little-endian 408 | view.setUint32(this.offset + 4, high, true); // little-endian 409 | this.offset += 8; 410 | } else { 411 | throw new Error('Value must be an integer between 0 and 18446744073709551615'); 412 | } 413 | } 414 | 415 | append(value) { 416 | if (Array.isArray(value)) { 417 | // Iterate over each element in the array and append 418 | for (const element of value) { 419 | this.append(element); 420 | } 421 | } else if (typeof value === 'string') { 422 | // Convert string to UTF-8 byte sequence 423 | const encoder = new TextEncoder(); 424 | const bytes = encoder.encode(value); 425 | const length = bytes.length; 426 | 427 | // Ensure capacity for the string bytes 428 | this.ensureCapacity(length); 429 | 430 | // Append bytes to the buffer 431 | this.buffer.set(bytes, this.offset); 432 | this.offset += length; 433 | 434 | // Debugging: 435 | //const stringFromUint8Array = String.fromCharCode.apply(null, this.buffer); 436 | //console.log(stringFromUint8Array); 437 | } else if (Number.isInteger(value) && value >= -0x80000000 && value <= 0x7FFFFFFF) { 438 | // 32-bit signed integer 439 | this.ensureCapacity(4); 440 | let view = new DataView(this.buffer.buffer); 441 | view.setInt32(this.offset, value, true); // little-endian 442 | this.offset += 4; 443 | } else if (typeof value === 'number') { 444 | // 64-bit floating-point number (double) 445 | this.ensureCapacity(8); 446 | let view = new DataView(this.buffer.buffer); 447 | view.setFloat64(this.offset, value, true); // little-endian 448 | this.offset += 8; 449 | } else { 450 | throw new Error('Unsupported value type'); 451 | } 452 | } 453 | } 454 | 455 | function write_beve(data) { 456 | const writer = new Writer(); 457 | write_value(writer, data); 458 | return writer.buffer; 459 | } 460 | 461 | function write_value(writer, value) { 462 | if (Array.isArray(value) && value.length > 1 && typeof value[0] === 'number') { 463 | let header = 4; 464 | if (typeof value[0] === 'number' && !Number.isInteger(value[0])) { 465 | header |= 0b01100000; // float64_t (double) 466 | } else { 467 | header |= 0b01001000; // int32_t (int) 468 | } 469 | writer.append_uint8(header); 470 | writeCompressed(writer, value.length); 471 | writer.append(value); 472 | } else if (typeof value === 'boolean') { 473 | let header = 0; 474 | if (value) { 475 | header |= 0b00011000; 476 | } else { 477 | header |= 0b00001000; 478 | } 479 | writer.append_uint8(header); 480 | } else if (typeof value === 'number') { 481 | let header = 1; 482 | if (typeof value === 'number' && !Number.isInteger(value)) { 483 | header |= 0b01100000; // float64_t (double) 484 | } else { 485 | header |= 0b01001000; // int32_t (int) 486 | } 487 | writer.append_uint8(header); 488 | writer.append(value); 489 | } else if (typeof value === 'string') { 490 | let header = 2; 491 | writer.append_uint8(header); 492 | writeCompressed(writer, value.length); 493 | writer.append(value); 494 | } else if (Array.isArray(value)) { 495 | let header = 5; 496 | writer.append_uint8(header); 497 | writeCompressed(writer, value.length); 498 | for (let i = 0; i < value.length; i++) { 499 | write_value(writer, value[i]); 500 | } 501 | } else if (typeof value === 'object' && Object.keys(value).length > 0) { 502 | let header = 3; 503 | let keyType = 0; // Assuming keys are always strings 504 | let isSigned = false; 505 | header |= keyType << 3; 506 | header |= isSigned << 5; 507 | writer.append_uint8(header); 508 | writeCompressed(writer, Object.keys(value).length); 509 | for (const key in value) { 510 | writeCompressed(writer, key.length); 511 | writer.append(key); 512 | write_value(writer, value[key]); 513 | } 514 | } else { 515 | throw new Error('Unsupported data type'); 516 | } 517 | } 518 | 519 | function writeCompressed(writer, N) { 520 | if (N < 64) { 521 | const compressed = (N << 2) | 0; 522 | writer.append_uint8(compressed); 523 | } else if (N < 16384) { 524 | const compressed = (N << 2) | 1; 525 | writer.append_uint16(compressed); 526 | } else if (N < 1073741824) { 527 | const compressed = (N << 2) | 2; 528 | writer.append_uint32(compressed); 529 | } else if (N < 4611686018427387904) { 530 | const compressed = (N << 2) | 3; 531 | writer.append_uint64(BigInt(compressed)); 532 | } 533 | } 534 | 535 | return { 536 | read_beve: read_beve, 537 | write_beve: write_beve 538 | }; 539 | })); -------------------------------------------------------------------------------- /javascript/beve_file.js: -------------------------------------------------------------------------------- 1 | // Reference: https://github.com/stephenberry/beve 2 | 3 | const fs = require('fs'); 4 | 5 | // Reading BEVE 6 | function read_beve(filename) { 7 | // Open dialog box if filename isn't provided. 8 | if (!filename) { 9 | throw new Error('No filename provided.'); 10 | } 11 | 12 | const fid = fs.openSync(filename, 'r'); 13 | if (!fid) { 14 | throw new Error('Failed to open file'); 15 | } 16 | 17 | const data = read_value(fid); 18 | 19 | fs.closeSync(fid); 20 | return data; 21 | } 22 | 23 | function read_value(fid) { 24 | const header = Buffer.alloc(1); 25 | fs.readSync(fid, header, 0, 1, null); 26 | 27 | const config = [1, 2, 4, 8]; 28 | 29 | const type = header[0] & 0b00000111; 30 | 31 | switch (type) { 32 | case 0: // null or boolean 33 | { 34 | const is_bool = (header[0] & 0b00001000) >> 3; 35 | if (is_bool) { 36 | return Boolean((header[0] & 0b11110000) >> 4); 37 | } else { 38 | return null; 39 | } 40 | } 41 | case 1: // number 42 | { 43 | const num_type = (header[0] & 0b00011000) >> 3; 44 | const is_float = num_type === 0; 45 | const is_signed = num_type === 1; 46 | 47 | const byte_count_index = (header[0] & 0b11100000) >> 5; 48 | const byte_count = config[byte_count_index]; 49 | 50 | if (is_float) { 51 | switch (byte_count) { 52 | case 4: 53 | return readFloat(fid); 54 | case 8: 55 | return readDouble(fid); 56 | } 57 | } else { 58 | if (is_signed) { 59 | switch (byte_count) { 60 | case 1: 61 | return readInt8(fid); 62 | case 2: 63 | return readInt16(fid); 64 | case 4: 65 | return readInt32(fid); 66 | case 8: 67 | return readBigInt64(fid); 68 | } 69 | } else { 70 | switch (byte_count) { 71 | case 1: 72 | return readUInt8(fid); 73 | case 2: 74 | return readUInt16(fid); 75 | case 4: 76 | return readUInt32(fid); 77 | case 8: 78 | return readBigUInt64(fid); 79 | } 80 | } 81 | } 82 | break; 83 | } 84 | case 2: // string 85 | { 86 | const size = read_compressed(fid); 87 | const buffer = Buffer.alloc(size); 88 | fs.readSync(fid, buffer, 0, size, null); 89 | return buffer.toString('utf8'); 90 | } 91 | case 3: // object 92 | { 93 | const key_type = (header[0] & 0b00011000) >> 3; 94 | const is_string = key_type === 0; 95 | const is_signed = key_type === 1; 96 | 97 | const byte_count_index = (header[0] & 0b11100000) >> 5; 98 | const byte_count = config[byte_count_index]; 99 | 100 | const N = read_compressed(fid); 101 | const objectData = {}; 102 | 103 | for (let i = 0; i < N; ++i) { 104 | if (is_string) { 105 | const size = read_compressed(fid); 106 | const buffer = Buffer.alloc(size); 107 | fs.readSync(fid, buffer, 0, size, null); 108 | const key = buffer.toString('utf8'); 109 | objectData[key] = read_value(fid); 110 | } else { 111 | throw new Error('TODO: support integer keys'); 112 | } 113 | } 114 | 115 | return objectData; 116 | } 117 | case 4: // typed array 118 | { 119 | const num_type = (header[0] & 0b00011000) >> 3; 120 | const is_float = num_type === 0; 121 | const is_signed = num_type === 1; 122 | 123 | const byte_count_index_array = (header[0] & 0b11100000) >> 5; 124 | const byte_count_array = config[byte_count_index_array]; 125 | 126 | if (num_type === 3) { 127 | const is_string = (header[0] & 0b00100000) >> 5; 128 | if (is_string) { 129 | const N = read_compressed(fid); 130 | const array = new Array(N); 131 | for (let i = 0; i < N; ++i) { 132 | const size = read_compressed(fid); 133 | const buffer = Buffer.alloc(size); 134 | fs.readSync(fid, buffer, 0, size, null); 135 | array[i] = buffer.toString('utf8'); 136 | } 137 | return array; 138 | } 139 | else { 140 | throw new Error("Boolean array support not implemented"); 141 | } 142 | } else if (is_float) { 143 | const N = read_compressed(fid); 144 | const array = new Array(N); 145 | switch (byte_count_array) { 146 | case 4: 147 | for (let i = 0; i < N; ++i) { 148 | array[i] = readFloat(fid); 149 | } 150 | break; 151 | case 8: 152 | for (let i = 0; i < N; ++i) { 153 | array[i] = readDouble(fid); 154 | } 155 | break; 156 | } 157 | return array; 158 | } else { 159 | const N = read_compressed(fid); 160 | const array = new Array(N); 161 | 162 | if (is_signed) { 163 | switch (byte_count_array) { 164 | case 1: 165 | for (let i = 0; i < N; ++i) { 166 | array[i] = readInt8(fid); 167 | } 168 | break; 169 | case 2: 170 | for (let i = 0; i < N; ++i) { 171 | array[i] = readInt16(fid); 172 | } 173 | break; 174 | case 4: 175 | for (let i = 0; i < N; ++i) { 176 | array[i] = readInt32(fid); 177 | } 178 | break; 179 | case 8: 180 | for (let i = 0; i < N; ++i) { 181 | array[i] = readBigInt64(fid); 182 | } 183 | break; 184 | } 185 | } else { 186 | switch (byte_count_array) { 187 | case 1: 188 | for (let i = 0; i < N; ++i) { 189 | array[i] = readUInt8(fid); 190 | } 191 | break; 192 | case 2: 193 | for (let i = 0; i < N; ++i) { 194 | array[i] = readUInt16(fid); 195 | } 196 | break; 197 | case 4: 198 | for (let i = 0; i < N; ++i) { 199 | array[i] = readUInt32(fid); 200 | } 201 | break; 202 | case 8: 203 | for (let i = 0; i < N; ++i) { 204 | array[i] = readBigUInt64(fid); 205 | } 206 | break; 207 | } 208 | } 209 | return array; 210 | } 211 | } 212 | case 5: // untyped array 213 | { 214 | const N = read_compressed(fid); 215 | const unarray = new Array(N); 216 | 217 | for (let i = 0; i < N; ++i) { 218 | unarray[i] = read_value(fid); 219 | } 220 | 221 | return unarray; 222 | } 223 | case 6: // extensions 224 | { 225 | const extension = (header[0] & 0b11111000) >> 3; 226 | switch (extension) { 227 | case 1: // variants 228 | read_compressed(fid); // Skipping variant tag 229 | return read_value(fid); 230 | case 2: // matrices 231 | const layout = fs.readSync(fid, Buffer.alloc(1), 0, 1, null)[0] & 0b00000001; 232 | switch (layout) { 233 | case 0: // row major 234 | throw new Error('TODO: add row major support'); 235 | case 1: // column major 236 | const extents = read_value(fid); 237 | const matrix_data = read_value(fid); 238 | return reshape(matrix_data, extents[0], extents[1]); 239 | default: 240 | throw new Error('Unsupported layout'); 241 | } 242 | case 3: // complex numbers 243 | return read_complex(fid); 244 | default: 245 | throw new Error('Unsupported extension'); 246 | } 247 | } 248 | default: 249 | throw new Error('Unsupported type'); 250 | } 251 | } 252 | 253 | function readFloat(fid) { 254 | const buffer = Buffer.alloc(4); 255 | fs.readSync(fid, buffer, 0, 4, null); 256 | return buffer.readFloatLE(); 257 | } 258 | 259 | function readDouble(fid) { 260 | const buffer = Buffer.alloc(8); 261 | fs.readSync(fid, buffer, 0, 8, null); 262 | return buffer.readDoubleLE(); 263 | } 264 | 265 | function readInt8(fid) { 266 | const buffer = Buffer.alloc(1); 267 | fs.readSync(fid, buffer, 0, 1, null); 268 | return buffer.readInt8(); 269 | } 270 | 271 | function readInt16(fid) { 272 | const buffer = Buffer.alloc(2); 273 | fs.readSync(fid, buffer, 0, 2, null); 274 | return buffer.readInt16LE(); 275 | } 276 | 277 | function readInt32(fid) { 278 | const buffer = Buffer.alloc(4); 279 | fs.readSync(fid, buffer, 0, 4, null); 280 | return buffer.readInt32LE(); 281 | } 282 | 283 | function readBigInt64(fid) { 284 | const buffer = Buffer.alloc(8); 285 | fs.readSync(fid, buffer, 0, 8, null); 286 | return buffer.readBigInt64LE(); 287 | } 288 | 289 | function readUInt8(fid) { 290 | const buffer = Buffer.alloc(1); 291 | fs.readSync(fid, buffer, 0, 1, null); 292 | return buffer.readUInt8(); 293 | } 294 | 295 | function readUInt16(fid) { 296 | const buffer = Buffer.alloc(2); 297 | fs.readSync(fid, buffer, 0, 2, null); 298 | return buffer.readUInt16LE(); 299 | } 300 | 301 | function readUInt32(fid) { 302 | const buffer = Buffer.alloc(4); 303 | fs.readSync(fid, buffer, 0, 4, null); 304 | return buffer.readUInt32LE(); 305 | } 306 | 307 | function readBigUInt64(fid) { 308 | const buffer = Buffer.alloc(8); 309 | fs.readSync(fid, buffer, 0, 8, null); 310 | return buffer.readBigUInt64LE(); 311 | } 312 | 313 | const byte_count_lookup = [1, 2, 4, 8, 16, 32, 64, 128]; 314 | 315 | function read_compressed(fid) { 316 | let header = Buffer.alloc(1); 317 | fs.readSync(fid, header, 0, 1, null); 318 | const config = header[0] & 0b00000011; 319 | 320 | switch (config) { 321 | case 0: 322 | return header[0] >> 2; 323 | case 1: { 324 | const h = Buffer.alloc(2); 325 | fs.readSync(fid, h, 0, 2, null); 326 | return ((h[0] << 8) | h[1]) >> 2; 327 | } 328 | case 2: { 329 | const h = Buffer.alloc(4); 330 | fs.readSync(fid, h, 0, 4, null); 331 | return ((h[0] << 24) | (h[1] << 16) | (h[2] << 8) | h[3]) >> 2; 332 | } 333 | case 3: { 334 | const h = Buffer.alloc(8); 335 | fs.readSync(fid, h, 0, 8, null); 336 | let val = BigInt(0); 337 | for (let i = 0; i < 8; ++i) { 338 | val |= BigInt(h[i]) << BigInt(8 * i); 339 | } 340 | return Number(val >> BigInt(2)); 341 | } 342 | default: 343 | return 0; 344 | } 345 | } 346 | 347 | // Writing BEVE 348 | function write_beve(data, filename) { 349 | const file = new BinaryFile(filename); 350 | if (!file) { 351 | throw new Error('Failed to open file for writing'); 352 | } 353 | try { 354 | write_value(file, data); 355 | } finally { 356 | file.close(); 357 | } 358 | } 359 | 360 | function write_value(file, value) { 361 | if (Array.isArray(value) && value.length > 1) { 362 | const header = 4; 363 | if (typeof value[0] === 'number' && !Number.isInteger(value[0])) { 364 | write_float(file, header, value, 1); 365 | } else { 366 | write_integer(file, header, value, 1); 367 | } 368 | } else if (typeof value === 'boolean') { 369 | let header = 0; 370 | if (value) { 371 | header |= 0b00011000; 372 | } else { 373 | header |= 0b00001000; 374 | } 375 | file.writeUint8(header); 376 | } else if (typeof value === 'number') { 377 | let header = 1; 378 | if (!Number.isInteger(value)) { 379 | write_float(file, header, value, 0); 380 | } else { 381 | write_integer(file, header, value, 0); 382 | } 383 | } else if (typeof value === 'string') { 384 | throw new Error('Unsupported data type'); 385 | } else if (typeof value === 'object' && Object.keys(value).length > 0) { 386 | let header = 3; 387 | let keyType = 0; // Assuming keys are always strings 388 | let isSigned = false; 389 | header |= keyType << 3; 390 | header |= isSigned << 5; 391 | file.writeUint8(header); 392 | writeCompressed(file, Object.keys(value).length); 393 | for (const key in value) { 394 | writeCompressed(file, key.length); 395 | file.writeString(key); 396 | write_value(file, value[key]); 397 | } 398 | } else if (Array.isArray(value)) { 399 | let header = 5; 400 | file.writeUint8(header); 401 | writeCompressed(file, value.length); 402 | for (let i = 0; i < value.length; i++) { 403 | write_value(file, value[i]); 404 | } 405 | } else { 406 | throw new Error('Unsupported data type'); 407 | } 408 | } 409 | 410 | function write_float(file, header, value, isArray) { 411 | if (Math.fround(value) === value) { // single precision float 412 | header |= 0b01000000; 413 | file.writeUint8(header); 414 | if (isArray) { 415 | writeCompressed(file, value.length); 416 | } 417 | file.write_float32LE(value); 418 | } else { // double precision float 419 | header |= 0b01100000; 420 | file.writeUint8(header); 421 | if (isArray) { 422 | writeCompressed(file, value.length); 423 | } 424 | file.write_float64LE(value); 425 | } 426 | } 427 | 428 | function write_integer(file, header, value, isArray) { 429 | if (value >= 0 && value <= 255) { // uint8 430 | header |= 0b00010001; 431 | file.writeUint8(header); 432 | if (isArray) { 433 | writeCompressed(file, value.length); 434 | } 435 | file.writeUint8(value); 436 | } else if (value >= 0 && value <= 65535) { // uint16 437 | header |= 0b00110001; 438 | file.writeUint8(header); 439 | if (isArray) { 440 | writeCompressed(file, value.length); 441 | } 442 | file.writeUint16LE(value); 443 | } else if (value >= 0 && value <= 4294967295) { // uint32 444 | header |= 0b01010001; 445 | file.writeUint8(header); 446 | if (isArray) { 447 | writeCompressed(file, value.length); 448 | } 449 | file.writeUint32LE(value); 450 | } else if (value >= 0 && value <= Number.MAX_SAFE_INTEGER) { // uint64 451 | header |= 0b01110001; 452 | file.writeUint8(header); 453 | if (isArray) { 454 | writeCompressed(file, value.length); 455 | } 456 | file.writeBigInt64LE(BigInt(value)); 457 | } else if (value >= -128 && value <= 127) { // int8 458 | header |= 0b00001001; 459 | file.writeUint8(header); 460 | if (isArray) { 461 | writeCompressed(file, value.length); 462 | } 463 | file.writeInt8(value); 464 | } else if (value >= -32768 && value <= 32767) { // int16 465 | header |= 0b00101001; 466 | file.writeUint8(header); 467 | if (isArray) { 468 | writeCompressed(file, value.length); 469 | } 470 | } else if (value >= -2147483648 && value <= 2147483647) { // int32 471 | header |= 0b01001001; 472 | file.writeUint8(header); 473 | if (isArray) { 474 | writeCompressed(file, value.length); 475 | } 476 | file.writeInt32LE(value); 477 | } else if (value >= -9223372036854775808 && value <= 9223372036854775807) { // int64 478 | header |= 0b01101001; 479 | file.writeUint8(header); 480 | if (isArray) { 481 | writeCompressed(file, value.length); 482 | } 483 | file.writeBigInt64LE(BigInt(value)); 484 | } else { 485 | throw new Error('Unsupported type'); 486 | } 487 | } 488 | 489 | function writeCompressed(file, N) { 490 | if (N < 64) { 491 | const compressed = (N << 2) | 0; 492 | file.writeUint8(compressed); 493 | } else if (N < 16384) { 494 | const compressed = (N << 2) | 1; 495 | file.writeUint16LE(compressed); 496 | } else if (N < 1073741824) { 497 | const compressed = (N << 2) | 2; 498 | file.writeUint32LE(compressed); 499 | } else if (N < 4611686018427387904) { 500 | const compressed = (N << 2) | 3; 501 | file.writeUint64LE(BigInt(compressed)); 502 | } 503 | } 504 | 505 | class BinaryFile { 506 | constructor(filename) { 507 | this.filename = filename; 508 | this.fd = require('fs').openSync(filename, 'wb'); 509 | } 510 | writeUint8(value) { 511 | require('fs').writeSync(this.fd, Buffer.from([value])); 512 | } 513 | writeUint16LE(value) { 514 | require('fs').writeSync(this.fd, Buffer.from([value & 0xFF, (value >> 8) & 0xFF])); 515 | } 516 | writeUint32LE(value) { 517 | require('fs').writeSync(this.fd, Buffer.from([(value >> 0) & 0xFF, (value >> 8) & 0xFF, (value >> 16) & 0xFF, (value >> 24) & 0xFF])); 518 | } 519 | writeUint64LE(value) { 520 | require('fs').writeSync(this.fd, Buffer.from([(value >> 0n) & BigInt(0xFF), (value >> 8n) & BigInt(0xFF), (value >> 16n) & BigInt(0xFF), (value >> 24n) & BigInt(0xFF), (value >> 32n) & BigInt(0xFF), (value >> 40n) & BigInt(0xFF), (value >> 48n) & BigInt(0xFF), (value >> 56n) & BigInt(0xFF)])); 521 | } 522 | write_float32LE(value) { 523 | const buffer = Buffer.alloc(4); 524 | buffer.write_floatLE(value, 0); 525 | require('fs').writeSync(this.fd, buffer); 526 | } 527 | write_float64LE(value) { 528 | const buffer = Buffer.alloc(8); 529 | buffer.writeDoubleLE(value, 0); 530 | require('fs').writeSync(this.fd, buffer); 531 | } 532 | writeBigInt64LE(value) { 533 | const buffer = Buffer.alloc(8); 534 | buffer.writeBigInt64LE(value, 0); 535 | require('fs').writeSync(this.fd, buffer); 536 | } 537 | writeString(str) { 538 | require('fs').writeSync(this.fd, Buffer.from(str)); 539 | } 540 | close() { 541 | require('fs').closeSync(this.fd); 542 | } 543 | } 544 | 545 | // Usage example: 546 | try { 547 | const filename = '../example/example.beve'; 548 | const data = read_beve(filename); 549 | console.log(data); 550 | //console.log(JSON.stringify(data)); 551 | //write_beve(data, '../example/examplejs.beve'); 552 | } catch (error) { 553 | console.error(error); 554 | } 555 | -------------------------------------------------------------------------------- /matlab/load_beve.m: -------------------------------------------------------------------------------- 1 | % Load a .beve file 2 | % Reference: https://github.com/stephenberry/beve 3 | % Given Path to file: Load 4 | % Given Path to folder: Open load dialog box in folder 5 | % Given no arguments: Open load dialog box in working directory 6 | function data = load_beve(path) 7 | 8 | isPathDirectory = false; 9 | defaultFolder = ''; % present working directory 10 | isPathGiven = exist('path','var') && ~isempty(path); 11 | if(isPathGiven) 12 | %isPathFile = isfile(path); 13 | isPathDirectory = isfolder(path); 14 | if(isPathDirectory) 15 | defaultFolder = path; 16 | end 17 | end 18 | 19 | % Open Dialog Box 20 | if ( ~isPathGiven || isPathDirectory ) 21 | [file,path] = uigetfile(fullfile(defaultFolder,'*.beve'),... 22 | 'Select a BEVE file to load.'); 23 | path = fullfile(path,file); 24 | fprintf("BEVE File Selected:\n'%s'\n",path); 25 | end 26 | 27 | fid = fopen(path, 'rb'); 28 | if fid == -1 29 | error('Failed to open file'); 30 | end 31 | 32 | data = read_value(fid); 33 | 34 | fclose(fid); 35 | end 36 | 37 | % 'l' denotes little endian format 38 | % (https://www.mathworks.com/help/matlab/ref/fread.html) 39 | function data = read_value(fid) 40 | % Read the header 41 | header = fread(fid, 1, '*uint8', 'l'); 42 | assert(~isempty(header), 'Unexpected end of data'); 43 | 44 | % Configuration mapping 45 | config = uint8([1, 2, 4, 8]); 46 | 47 | % Extract header components 48 | type = bitand(header, 0b00000111); 49 | switch type 50 | case 0 % null or boolean 51 | is_bool = bitshift(bitand(header, 0b00001000), -3); 52 | if is_bool 53 | data = logical(bitshift(bitand(header, 0b11110000), -4)); 54 | else 55 | data = NaN; 56 | end 57 | case 1 % number 58 | element_type = bitshift(bitand(header, 0b00011000), -3); 59 | is_float = false; 60 | is_signed = false; 61 | switch element_type 62 | case 0 63 | is_float = true; 64 | case 1 65 | is_signed = true; 66 | end 67 | 68 | byte_count_index = bitshift(bitand(header, 0b11100000), -5); 69 | byte_count = config(byte_count_index + 1); 70 | 71 | if is_float 72 | switch byte_count 73 | case 4 74 | data = fread(fid, 1, '*float32', 'l'); 75 | case 8 76 | data = fread(fid, 1, '*float64', 'l'); 77 | end 78 | else 79 | if is_signed 80 | switch byte_count 81 | case 1 82 | data = fread(fid, 1, '*int8', 'l'); 83 | case 2 84 | data = fread(fid, 1, '*int16', 'l'); 85 | case 4 86 | data = fread(fid, 1, '*int32', 'l'); 87 | case 8 88 | data = fread(fid, 1, '*int64', 'l'); 89 | end 90 | else 91 | switch byte_count 92 | case 1 93 | data = fread(fid, 1, '*uint8', 'l'); 94 | case 2 95 | data = fread(fid, 1, '*uint16', 'l'); 96 | case 4 97 | data = fread(fid, 1, '*uint32', 'l'); 98 | case 8 99 | data = fread(fid, 1, '*uint64', 'l'); 100 | end 101 | end 102 | end 103 | case 2 % string 104 | string_size = read_compressed(fid); 105 | data = fread(fid, string_size, 'char=>char', 'l')'; 106 | case 3 % object 107 | key_type = bitshift(bitand(header, 0b00011000), -3); 108 | is_string = (key_type == 0); 109 | is_signed = (key_type == 1); 110 | 111 | % Get byte count for integer keys 112 | byte_count_index = bitshift(bitand(header, 0b11100000), -5); 113 | byte_count = config(byte_count_index + 1); 114 | 115 | N = read_compressed(fid); 116 | 117 | % Initialize empty struct if there are fields 118 | if N > 0 119 | data = struct(); 120 | else 121 | data = []; % Empty object 122 | empty_key = 'object'; 123 | try 124 | empty_key = evalin('caller','legal_string'); 125 | catch 126 | end 127 | warning("Zero object keys found for %s", empty_key); 128 | return; 129 | end 130 | 131 | for ii = 1:N 132 | if is_string 133 | % Read string key 134 | string_size = read_compressed(fid); 135 | string = fread(fid, string_size, 'char=>char', 'l')'; 136 | legal_string = makeValidFieldName(string); 137 | data.(legal_string) = read_value(fid); 138 | else 139 | % Handle integer keys (signed or unsigned) 140 | if is_signed 141 | switch byte_count 142 | case 1 143 | key = fread(fid, 1, '*int8', 'l'); 144 | case 2 145 | key = fread(fid, 1, '*int16', 'l'); 146 | case 4 147 | key = fread(fid, 1, '*int32', 'l'); 148 | case 8 149 | key = fread(fid, 1, '*int64', 'l'); 150 | end 151 | else % unsigned 152 | switch byte_count 153 | case 1 154 | key = fread(fid, 1, '*uint8', 'l'); 155 | case 2 156 | key = fread(fid, 1, '*uint16', 'l'); 157 | case 4 158 | key = fread(fid, 1, '*uint32', 'l'); 159 | case 8 160 | key = fread(fid, 1, '*uint64', 'l'); 161 | end 162 | end 163 | % Convert integer key to valid MATLAB field name 164 | key_str = sprintf('int_%d', key); 165 | data.(key_str) = read_value(fid); 166 | end 167 | end 168 | 169 | case 4 % typed array 170 | element_type = bitshift(bitand(header, 0b00011000), -3); 171 | [is_float, is_signed, is_bool_or_string] = ... 172 | deal(element_type == 0, element_type == 1, element_type == 3); 173 | is_numeric = not(is_bool_or_string); 174 | string_flag = bitshift(bitand(header, 0b00100000), -5); 175 | is_string = is_bool_or_string && string_flag; 176 | is_bool = is_bool_or_string && not(string_flag); 177 | 178 | %% Only used for numeric types 179 | byte_count_index = bitshift(bitand(header, 0b11100000), -5); 180 | byte_count = config(byte_count_index + 1); 181 | 182 | % Read the N of the array 183 | N = read_compressed(fid); 184 | 185 | if is_numeric 186 | if is_float 187 | switch byte_count 188 | case 4 189 | data = fread(fid, N, '*float32', 'l'); 190 | case 8 191 | data = fread(fid, N, '*float64', 'l'); 192 | end 193 | else 194 | if is_signed 195 | switch byte_count 196 | case 1 197 | data = fread(fid, N, '*int8', 'l'); 198 | case 2 199 | data = fread(fid, N, '*int16', 'l'); 200 | case 4 201 | data = fread(fid, N, '*int32', 'l'); 202 | case 8 203 | data = fread(fid, N, '*int64', 'l'); 204 | end 205 | else 206 | switch byte_count 207 | case 1 208 | data = fread(fid, N, '*uint8', 'l'); 209 | case 2 210 | data = fread(fid, N, '*uint16', 'l'); 211 | case 4 212 | data = fread(fid, N, '*uint32', 'l'); 213 | case 8 214 | data = fread(fid, N, '*uint64', 'l'); 215 | end 216 | end 217 | end 218 | elseif is_string 219 | % Read an array of strings (or cell array of character vectors?) 220 | % For each element... (there are N) 221 | % Read the compressed size, then read that number of chars 222 | data=strings(N,1); % Initialize string array 223 | for ii=1:N 224 | string_size = read_compressed(fid); 225 | data{ii} = fread(fid, string_size, 'char=>char', 'l')'; 226 | end 227 | elseif is_bool 228 | % Read packed boolean values (8 per byte) 229 | num_bytes = ceil(N / 8); 230 | packed_bytes = fread(fid, num_bytes, '*uint8', 'l'); 231 | 232 | % Unpack the booleans from the bytes 233 | data = false(N, 1); 234 | for i = 1:N 235 | byte_index = floor((i-1) / 8) + 1; 236 | bit_position = mod(i-1, 8); 237 | data(i) = bitand(bitshift(packed_bytes(byte_index), -bit_position), 1) == 1; 238 | end 239 | end 240 | 241 | case 5 % untyped array 242 | N = read_compressed(fid); 243 | 244 | data = cell(N, 1); 245 | for ii = 1:N 246 | data{ii} = read_value(fid); 247 | end 248 | case 6 % extensions 249 | extension = bitshift(bitand(header, 0b11111000), -3); 250 | switch extension 251 | case 1 % variants 252 | read_compressed(fid); 253 | data = read_value(fid); 254 | case 2 % matrices 255 | layout = bitand(fread(fid, 1, '*uint8', 'l'), 0b00000001); 256 | extents = read_value(fid); 257 | matrix_data = read_value(fid); 258 | 259 | switch layout 260 | case 0 % row major 261 | % For row-major, we need to reshape and transpose 262 | data = reshape(matrix_data, extents(2), extents(1))'; 263 | case 1 % column major 264 | data = reshape(matrix_data, extents(1), extents(2)); 265 | otherwise 266 | error('Unsupported layout'); 267 | end 268 | case 3 % complex numbers 269 | data = read_complex(fid); 270 | otherwise 271 | error('Unsupported extension'); 272 | end 273 | otherwise 274 | error('Unsupported type'); 275 | end 276 | end 277 | 278 | function data = read_complex(fid) 279 | complex_header = fread(fid, 1, '*uint8', 'l'); 280 | type = bitand(complex_header, 0b00000111); 281 | 282 | num_type = bitshift(bitand(complex_header, 0b00011000), -3); 283 | is_float = false; 284 | is_signed = false; 285 | switch num_type 286 | case 0 287 | is_float = true; 288 | case 1 289 | is_signed = true; 290 | end 291 | 292 | byte_count_index = bitshift(bitand(complex_header, 0b11100000), -5); 293 | config = uint8([1, 2, 4, 8]); 294 | byte_count = config(byte_count_index + 1); 295 | 296 | switch type 297 | case 0 % complex number 298 | if is_float 299 | switch byte_count 300 | case 4 301 | data = complex(fread(fid, 2, '*float32', 'l')); 302 | case 8 303 | data = complex(fread(fid, 2, '*float64', 'l')); 304 | end 305 | else 306 | if is_signed 307 | switch byte_count 308 | case 1 309 | data = complex(fread(fid, 2, '*int8', 'l')); 310 | case 2 311 | data = complex(fread(fid, 2, '*int16', 'l')); 312 | case 4 313 | data = complex(fread(fid, 2, '*int32', 'l')); 314 | case 8 315 | data = complex(fread(fid, 2, '*int64', 'l')); 316 | end 317 | else 318 | switch byte_count 319 | case 1 320 | data = complex(fread(fid, 2, '*uint8', 'l')); 321 | case 2 322 | data = complex(fread(fid, 2, '*uint16', 'l')); 323 | case 4 324 | data = complex(fread(fid, 2, '*uint32', 'l')); 325 | case 8 326 | data = complex(fread(fid, 2, '*uint64', 'l')); 327 | end 328 | end 329 | end 330 | case 1 % complex array 331 | % Read the N of the array 332 | N = read_compressed(fid); 333 | 334 | if is_float 335 | switch byte_count 336 | case 4 337 | raw = fread(fid, [2, N], '*float32', 'l'); 338 | data = complex(raw(1, :), raw(2, :)); 339 | case 8 340 | raw = fread(fid, [2, N], '*float64', 'l'); 341 | data = complex(raw(1, :), raw(2, :)); 342 | end 343 | else 344 | if is_signed 345 | switch byte_count 346 | case 1 347 | raw = fread(fid, [2, N], '*int8', 'l'); 348 | case 2 349 | raw = fread(fid, [2, N], '*int16', 'l'); 350 | case 4 351 | raw = fread(fid, [2, N], '*int32', 'l'); 352 | case 8 353 | raw = fread(fid, [2, N], '*int64', 'l'); 354 | end 355 | else 356 | switch byte_count 357 | case 1 358 | raw = fread(fid, N, '*uint8', 'l'); 359 | case 2 360 | raw = fread(fid, N, '*uint16', 'l'); 361 | case 4 362 | raw = fread(fid, N, '*uint32', 'l'); 363 | case 8 364 | raw = fread(fid, N, '*uint64', 'l'); 365 | end 366 | end 367 | data = complex(raw(1, :), raw(2, :)); % remap to complex 368 | end 369 | end 370 | end 371 | 372 | function N = read_compressed(fid) 373 | config = uint8([1, 2, 4, 8]); 374 | 375 | compressed = fread(fid, 1, '*uint8', 'l'); 376 | n_size_bytes = config(bitand(compressed, 0b00000011) + 1); 377 | fseek(fid, -1, 'cof'); % 'cof' means current position 378 | switch n_size_bytes 379 | case 1 380 | N = fread(fid, 1, '*uint8', 'l'); 381 | case 2 382 | N = fread(fid, 1, '*uint16', 'l'); 383 | case 4 384 | N = fread(fid, 1, '*uint32', 'l'); 385 | case 8 386 | N = fread(fid, 1, '*uint64', 'l'); 387 | otherwise 388 | error('Unsupported N'); 389 | end 390 | N = bitshift(N, -2); 391 | end 392 | 393 | 394 | function validName = makeValidFieldName(fieldName) 395 | % Convert string to char array if necessary 396 | if isstring(fieldName) 397 | fieldName = char(fieldName); 398 | end 399 | 400 | % Transpose if is a column vector 401 | if(size(fieldName,1) > 1 && size(fieldName,2) == 1) 402 | fieldName = fieldName'; 403 | end 404 | 405 | % Replace invalid characters with underscores 406 | validName = regexprep(fieldName, '[^a-zA-Z0-9_]', ''); 407 | 408 | % Ensure the name starts with a letter 409 | if isempty(regexp(validName, '^[a-zA-Z]', 'once')) 410 | validName = ['A', validName]; % Prepend 'A' if the name does not start with a letter 411 | end 412 | 413 | % Truncate if the name is too long 414 | maxNameLength = namelengthmax(); 415 | if length(validName) > maxNameLength 416 | validName = validName(1:maxNameLength); 417 | end 418 | end -------------------------------------------------------------------------------- /matlab/write_beve.m: -------------------------------------------------------------------------------- 1 | % Write a .beve file 2 | % Reference: https://github.com/stephenberry/beve 3 | function write_beve(data, filename) 4 | % Open dialog box if filename isn't provided. 5 | if ( ~exist('filename','var') || isempty(filename) ) 6 | [file, path] = uiputfile('*.beve', 'Save BEVE file as'); 7 | if isequal(file, 0) || isequal(path, 0) 8 | error('File selection cancelled'); 9 | end 10 | filename = fullfile(path, file); 11 | fprintf("BEVE File Selected for Writing:\n'%s'\n", filename); 12 | end 13 | 14 | fid = fopen(filename, 'wb'); 15 | if fid == -1 16 | error('Failed to open file for writing'); 17 | end 18 | 19 | write_value(fid, data); 20 | 21 | fclose(fid); 22 | end 23 | 24 | function write_value(fid, value) 25 | if isnumeric(value) && isscalar(value) && isnan(value) 26 | % Handle null value (NaN in MATLAB) 27 | header = uint8(0); % Type 0 = null 28 | write_byte(fid, header); 29 | elseif ischar(value) 30 | % Handle string values 31 | header = uint8(2); % Type 2 = string 32 | write_byte(fid, header); 33 | 34 | % Write string length 35 | write_compressed(fid, length(value)); 36 | 37 | % Write string content 38 | fwrite(fid, value, 'char', 'l'); 39 | elseif iscell(value) 40 | header = uint8(5); 41 | write_byte(fid, header); 42 | write_compressed(fid, numel(value)); 43 | for ii = 1:numel(value) 44 | write_value(fid, value{ii}); 45 | end 46 | elseif isnumeric(value) && ~isreal(value) 47 | % Handle complex numbers 48 | header = uint8(6); % Type 6 = extensions 49 | header = bitor(header, bitshift(3, 3)); % Extension 3 = complex numbers 50 | write_byte(fid, header); 51 | write_complex(fid, value); 52 | elseif isstring(value) && length(value) > 1 53 | % Handle string arrays 54 | header = uint8(4); % Type 4 = typed array 55 | header = bitor(header, bitshift(3, 3)); % element_type = 3 (bool_or_string) 56 | header = bitor(header, bitshift(1, 5)); % string_flag = 1 57 | write_byte(fid, header); 58 | 59 | % Write array size 60 | write_compressed(fid, numel(value)); 61 | 62 | % Write each string with its length 63 | for i = 1:numel(value) 64 | str = char(value(i)); 65 | write_compressed(fid, length(str)); 66 | fwrite(fid, str, 'char', 'l'); 67 | end 68 | elseif islogical(value) && length(value) > 1 69 | % Handle logical arrays - packed as bits (8 per byte) 70 | header = uint8(4); % Type 4 = typed array 71 | header = bitor(header, bitshift(3, 3)); % element_type = 3 (bool_or_string) 72 | header = bitor(header, bitshift(0, 5)); % string_flag = 0 (boolean) 73 | write_byte(fid, header); 74 | 75 | % Write array size 76 | write_compressed(fid, numel(value)); 77 | 78 | % Pack booleans into bits (8 per byte) 79 | num_values = numel(value); 80 | num_bytes = ceil(num_values / 8); 81 | packed_bytes = zeros(num_bytes, 1, 'uint8'); 82 | 83 | % Pack each boolean into the appropriate bit 84 | for i = 1:num_values 85 | byte_index = floor((i-1) / 8) + 1; 86 | bit_position = mod(i-1, 8); 87 | if value(i) 88 | packed_bytes(byte_index) = bitor(packed_bytes(byte_index), bitshift(uint8(1), bit_position)); 89 | end 90 | end 91 | 92 | % Write the packed bytes 93 | fwrite(fid, packed_bytes, 'uint8', 'l'); 94 | elseif isnumeric(value) && isempty(value) 95 | % Handle empty numeric arrays - this is the fix for empty arrays 96 | header = uint8(4); % Type 4 = typed array 97 | 98 | % Determine element type 99 | if isfloat(value) 100 | header = bitor(header, bitshift(0, 3)); % element_type = 0 (float) 101 | elseif any(strcmp(class(value), {'int8', 'int16', 'int32', 'int64'})) 102 | header = bitor(header, bitshift(1, 3)); % element_type = 1 (signed) 103 | else 104 | header = bitor(header, bitshift(2, 3)); % element_type = 2 (unsigned) 105 | end 106 | 107 | % Determine byte count 108 | if isa(value, 'single') || isa(value, 'int32') || isa(value, 'uint32') 109 | header = bitor(header, bitshift(2, 5)); % byte_count_index = 2 (4 bytes) 110 | elseif isa(value, 'double') || isa(value, 'int64') || isa(value, 'uint64') 111 | header = bitor(header, bitshift(3, 5)); % byte_count_index = 3 (8 bytes) 112 | elseif isa(value, 'int16') || isa(value, 'uint16') 113 | header = bitor(header, bitshift(1, 5)); % byte_count_index = 1 (2 bytes) 114 | else % int8 or uint8 115 | header = bitor(header, bitshift(0, 5)); % byte_count_index = 0 (1 byte) 116 | end 117 | 118 | write_byte(fid, header); 119 | 120 | % Write array size (0 for empty array) 121 | write_compressed(fid, 0); 122 | elseif isvector(value) && length(value) > 1 123 | if iscell(value) && all(cellfun(@ischar, value)) 124 | % Handle cell array of strings as a string array 125 | header = uint8(4); % Type 4 = typed array 126 | header = bitor(header, bitshift(3, 3)); % element_type = 3 (bool_or_string) 127 | header = bitor(header, bitshift(1, 5)); % string_flag = 1 128 | write_byte(fid, header); 129 | 130 | % Write array size 131 | write_compressed(fid, numel(value)); 132 | 133 | % Write each string with its length 134 | for i = 1:numel(value) 135 | str = value{i}; 136 | write_compressed(fid, length(str)); 137 | fwrite(fid, str, 'char', 'l'); 138 | end 139 | else 140 | % Handle numeric vectors 141 | header = uint8(4); 142 | if isfloat(value) 143 | write_float(fid, header, value, 1); 144 | else 145 | write_integer(fid, header, value, 1); 146 | end 147 | end 148 | elseif islogical(value) 149 | header = uint8(0); 150 | if value 151 | header = bitor(header, 0b00011000); 152 | else 153 | header = bitor(header, 0b00001000); 154 | end 155 | write_byte(fid, header); 156 | elseif ismatrix(value) && ~isvector(value) && ~isstruct(value) 157 | % Handle matrices 158 | header = uint8(6); % Type 6 = extensions 159 | header = bitor(header, bitshift(2, 3)); % Extension 2 = matrices 160 | write_byte(fid, header); 161 | 162 | % Write layout (column major = 1) 163 | write_byte(fid, uint8(1)); 164 | 165 | % Write dimensions 166 | write_value(fid, uint32([size(value,1), size(value,2)])); 167 | 168 | % Write matrix data 169 | write_value(fid, value(:)); 170 | elseif isnumeric(value) 171 | header = uint8(1); 172 | if isfloat(value) 173 | write_float(fid, header, value, 0); 174 | else 175 | write_integer(fid, header, value, 0); 176 | end 177 | elseif isstring(value) && length(value) == 1 178 | % Handle single MATLAB string (convert to char) 179 | write_value(fid, char(value)); 180 | elseif isstruct(value) && isfield(value, 'tag') && isfield(value, 'value') && numel(fieldnames(value)) == 2 181 | % Handle variant (Extension 1 - Type Tag) 182 | write_variant(fid, value.tag, value.value); 183 | elseif isstruct(value) 184 | header = uint8(3); 185 | key_type = 0; % Assuming keys are always strings 186 | header = bitor(header, bitshift(key_type, 3)); 187 | write_byte(fid, header); 188 | 189 | write_compressed(fid, numel(fieldnames(value))); 190 | fields = fieldnames(value); 191 | for ii = 1:numel(fields) 192 | field_name = fields{ii}; 193 | write_compressed(fid, length(field_name)); 194 | fwrite(fid, field_name, 'char', 'l'); 195 | write_value(fid, value.(field_name)); 196 | end 197 | else 198 | error('Unsupported data type: %s', class(value)); 199 | end 200 | end 201 | 202 | function write_variant(fid, tag, value) 203 | % Write a variant (Extension 1 - Type Tag) 204 | header = uint8(6); % Type 6 = extensions 205 | header = bitor(header, bitshift(1, 3)); % Extension 1 = type tag (variant) 206 | write_byte(fid, header); 207 | 208 | % Write the type tag as a compressed unsigned integer 209 | write_compressed(fid, tag); 210 | 211 | % Write the value 212 | write_value(fid, value); 213 | end 214 | 215 | function write_complex(fid, value) 216 | is_array = ~isscalar(value); 217 | type = uint8(is_array); 218 | 219 | % Determine numeric type 220 | if isfloat(real(value)) 221 | num_type = 0; % is_float = true 222 | elseif isinteger(real(value)) && any(real(value(:)) < 0) 223 | num_type = 1; % is_signed = true 224 | else 225 | num_type = 2; % is_unsigned = true 226 | end 227 | 228 | % Determine byte count 229 | if isa(real(value), 'single') 230 | byte_count_index = 2; % 4 bytes 231 | elseif isa(real(value), 'double') 232 | byte_count_index = 3; % 8 bytes 233 | elseif isa(real(value), 'int8') || isa(real(value), 'uint8') 234 | byte_count_index = 0; % 1 byte 235 | elseif isa(real(value), 'int16') || isa(real(value), 'uint16') 236 | byte_count_index = 1; % 2 bytes 237 | elseif isa(real(value), 'int32') || isa(real(value), 'uint32') 238 | byte_count_index = 2; % 4 bytes 239 | elseif isa(real(value), 'int64') || isa(real(value), 'uint64') 240 | byte_count_index = 3; % 8 bytes 241 | else 242 | error('Unsupported complex number type: %s', class(real(value))); 243 | end 244 | 245 | % Build the header 246 | complex_header = type; 247 | complex_header = bitor(complex_header, bitshift(num_type, 3)); 248 | complex_header = bitor(complex_header, bitshift(byte_count_index, 5)); 249 | 250 | % Write the header 251 | write_byte(fid, complex_header); 252 | 253 | % If it's an array, write the size 254 | if is_array 255 | write_compressed(fid, numel(value)); 256 | end 257 | 258 | % Write the complex values 259 | % For performance: 260 | % Reshape data into a real array with alternating real/imag parts 261 | value_type = class(real(value)); 262 | 263 | % Create array of correct type for the data 264 | flat_data = zeros(2*numel(value), 1, value_type); 265 | 266 | % Populate the array with alternating real and imaginary parts 267 | flat_data(1:2:end) = real(value(:)); 268 | flat_data(2:2:end) = imag(value(:)); 269 | 270 | % Convert MATLAB type names to fwrite format specifiers where needed 271 | if strcmp(value_type, 'single') 272 | format_specifier = 'float32'; 273 | elseif strcmp(value_type, 'double') 274 | format_specifier = 'float64'; 275 | else 276 | % For integer types, the format specifier is the same as the MATLAB type name 277 | format_specifier = value_type; 278 | end 279 | 280 | fwrite(fid, flat_data, format_specifier, 'l'); 281 | end 282 | 283 | function write_float(fid, header, value, is_array) 284 | if isa(value, 'single') 285 | % Single precision (4 bytes) 286 | header = bitor(header, bitshift(0, 3)); % is_float = true 287 | header = bitor(header, bitshift(2, 5)); % byte_count_index = 2 (4 bytes) 288 | write_byte(fid, header); 289 | if is_array 290 | write_compressed(fid, length(value)); 291 | end 292 | fwrite(fid, value, 'float32', 'l'); 293 | elseif isa(value, 'double') 294 | % Double precision (8 bytes) 295 | header = bitor(header, bitshift(0, 3)); % is_float = true 296 | header = bitor(header, bitshift(3, 5)); % byte_count_index = 3 (8 bytes) 297 | write_byte(fid, header); 298 | if is_array 299 | write_compressed(fid, length(value)); 300 | end 301 | fwrite(fid, value, 'float64', 'l'); 302 | else 303 | error('Unsupported floating-point type: %s', class(value)); 304 | end 305 | end 306 | 307 | function write_integer(fid, header, value, is_array) 308 | if isa(value, 'uint8') 309 | header = bitor(header, bitshift(2, 3)); % is_unsigned = true 310 | header = bitor(header, bitshift(0, 5)); % byte_count_index = 0 (1 byte) 311 | write_byte(fid, header); 312 | if is_array 313 | write_compressed(fid, length(value)); 314 | end 315 | fwrite(fid, value, 'uint8', 'l'); 316 | elseif isa(value, 'uint16') 317 | header = bitor(header, bitshift(2, 3)); % is_unsigned = true 318 | header = bitor(header, bitshift(1, 5)); % byte_count_index = 1 (2 bytes) 319 | write_byte(fid, header); 320 | if is_array 321 | write_compressed(fid, length(value)); 322 | end 323 | fwrite(fid, value, 'uint16', 'l'); 324 | elseif isa(value, 'uint32') 325 | header = bitor(header, bitshift(2, 3)); % is_unsigned = true 326 | header = bitor(header, bitshift(2, 5)); % byte_count_index = 2 (4 bytes) 327 | write_byte(fid, header); 328 | if is_array 329 | write_compressed(fid, length(value)); 330 | end 331 | fwrite(fid, value, 'uint32', 'l'); 332 | elseif isa(value, 'uint64') 333 | header = bitor(header, bitshift(2, 3)); % is_unsigned = true 334 | header = bitor(header, bitshift(3, 5)); % byte_count_index = 3 (8 bytes) 335 | write_byte(fid, header); 336 | if is_array 337 | write_compressed(fid, length(value)); 338 | end 339 | fwrite(fid, value, 'uint64', 'l'); 340 | elseif isa(value, 'int8') 341 | header = bitor(header, bitshift(1, 3)); % is_signed = true 342 | header = bitor(header, bitshift(0, 5)); % byte_count_index = 0 (1 byte) 343 | write_byte(fid, header); 344 | if is_array 345 | write_compressed(fid, length(value)); 346 | end 347 | fwrite(fid, value, 'int8', 'l'); 348 | elseif isa(value, 'int16') 349 | header = bitor(header, bitshift(1, 3)); % is_signed = true 350 | header = bitor(header, bitshift(1, 5)); % byte_count_index = 1 (2 bytes) 351 | write_byte(fid, header); 352 | if is_array 353 | write_compressed(fid, length(value)); 354 | end 355 | fwrite(fid, value, 'int16', 'l'); 356 | elseif isa(value, 'int32') 357 | header = bitor(header, bitshift(1, 3)); % is_signed = true 358 | header = bitor(header, bitshift(2, 5)); % byte_count_index = 2 (4 bytes) 359 | write_byte(fid, header); 360 | if is_array 361 | write_compressed(fid, length(value)); 362 | end 363 | fwrite(fid, value, 'int32', 'l'); 364 | elseif isa(value, 'int64') 365 | header = bitor(header, bitshift(1, 3)); % is_signed = true 366 | header = bitor(header, bitshift(3, 5)); % byte_count_index = 3 (8 bytes) 367 | write_byte(fid, header); 368 | if is_array 369 | write_compressed(fid, length(value)); 370 | end 371 | fwrite(fid, value, 'int64', 'l'); 372 | else 373 | error('Unsupported integer type: %s', class(value)); 374 | end 375 | end 376 | 377 | function write_byte(fid, byte) 378 | fwrite(fid, byte, 'uint8', 'l'); 379 | end 380 | 381 | function write_compressed(fid, N) 382 | if N < 64 383 | compressed = bitor(bitshift(N, 2), uint8(0)); 384 | fwrite(fid, compressed, 'uint8', 'l'); 385 | elseif N < 16384 386 | compressed = bitor(bitshift(N, 2), uint16(1)); 387 | fwrite(fid, compressed, 'uint16', 'l'); 388 | elseif N < 1073741824 389 | compressed = bitor(bitshift(N, 2), uint32(2)); 390 | fwrite(fid, compressed, 'uint32', 'l'); 391 | else 392 | compressed = bitor(bitshift(N, 2), uint64(3)); 393 | fwrite(fid, compressed, 'uint64', 'l'); 394 | end 395 | end 396 | -------------------------------------------------------------------------------- /python/load_beve.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import tkinter as tk 3 | from tkinter import filedialog 4 | 5 | def load_beve(filename): 6 | if filename == None: 7 | filename = filedialog.askopenfilename() 8 | 9 | def read_value(fid): 10 | header = np.fromfile(fid, dtype=np.uint8, count=1) 11 | type_val = np.bitwise_and(header, 0b00000111)[0] 12 | 13 | if type_val == 0: # null or boolean 14 | is_bool = np.bitwise_and(header, 0b00001000) >> 3 15 | if is_bool: 16 | data = bool(np.bitwise_and(header, 0b00010000) >> 5) 17 | else: 18 | data = None 19 | elif type_val == 1: # number 20 | config = np.uint8([1, 2, 4, 8, 16, 32, 64, 128]) 21 | byte_count_index = np.bitwise_and(header, 0b11100000) >> 5 22 | byte_count = config[byte_count_index] 23 | 24 | num_type = np.bitwise_and(header, 0b00011000) >> 3 25 | is_float = num_type == 0 26 | is_signed = num_type == 1 27 | 28 | if is_float: 29 | if byte_count == 4: 30 | data = np.fromfile(fid, dtype=np.float32, count=1, sep="")[0] 31 | elif byte_count == 8: 32 | data = np.fromfile(fid, dtype=np.float64, count=1, sep="")[0] 33 | else: 34 | raise NotImplementedError("float byte count not implemented") 35 | else: 36 | if is_signed: 37 | if byte_count == 1: 38 | data = np.fromfile(fid, dtype=np.int8, count=1, sep="")[0] 39 | elif byte_count == 2: 40 | data = np.fromfile(fid, dtype=np.int16, count=1, sep="")[0] 41 | elif byte_count == 4: 42 | data = np.fromfile(fid, dtype=np.int32, count=1, sep="")[0] 43 | elif byte_count == 8: 44 | data = np.fromfile(fid, dtype=np.int64, count=1, sep="")[0] 45 | else: 46 | raise NotImplementedError("float byte count not implemented") 47 | else: 48 | if byte_count == 1: 49 | data = np.fromfile(fid, dtype=np.uint8, count=1, sep="")[0] 50 | elif byte_count == 2: 51 | data = np.fromfile(fid, dtype=np.uint16, count=1, sep="")[0] 52 | elif byte_count == 4: 53 | data = np.fromfile(fid, dtype=np.uint32, count=1, sep="")[0] 54 | elif byte_count == 8: 55 | data = np.fromfile(fid, dtype=np.uint64, count=1, sep="")[0] 56 | else: 57 | raise NotImplementedError("float byte count not implemented") 58 | elif type_val == 2: # string 59 | string_size = read_compressed(fid) 60 | data = fid.read(string_size).decode("utf-8") 61 | elif type_val == 3: # object 62 | key_type = np.bitwise_and(header, 0b00011000) >> 3 63 | is_string = key_type == 0 64 | is_signed = key_type == 1 65 | 66 | if is_string: 67 | size = read_compressed(fid) 68 | 69 | data = {} 70 | for _ in range(size): 71 | string_size = read_compressed(fid) 72 | string = fid.read(string_size).decode("utf-8") 73 | data[string] = read_value(fid) 74 | else: 75 | byte_count_index = np.bitwise_and(header, 0b11100000) >> 5 76 | byte_count = config[byte_count_index] 77 | 78 | raise NotImplementedError("Integer key support not implemented") 79 | 80 | elif type_val == 4: # typed array 81 | num_type = np.bitwise_and(header, 0b00011000) >> 3 82 | is_float = num_type == 0 83 | is_signed = num_type == 1 84 | 85 | if num_type == 3: # boolean or string 86 | is_string = np.bitwise_and(header, 0b00100000) >> 5 87 | if is_string: 88 | array_size = read_compressed(fid) 89 | data = [] 90 | for _ in range(array_size): 91 | string_size = read_compressed(fid) 92 | data.append(fid.read(string_size).decode("utf-8")) 93 | 94 | else: 95 | raise NotImplementedError("Boolean array support not implemented") 96 | else: 97 | config = np.uint8([1, 2, 4, 8, 16, 32, 64, 128]) 98 | byte_count_index = np.bitwise_and(header, 0b11100000) >> 5 99 | byte_count = config[byte_count_index] 100 | 101 | size = read_compressed(fid) 102 | 103 | if is_float: 104 | data = np.fromfile(fid, dtype=np.float32 if byte_count == 4 else np.float64, count=size, sep="") 105 | else: 106 | if is_signed: 107 | data = np.fromfile(fid, dtype=np.int8 if byte_count == 1 else np.int16 if byte_count == 2 else np.int32 if byte_count == 4 else np.int64, count=size, sep="") 108 | else: 109 | data = np.fromfile(fid, dtype=np.uint8 if byte_count == 1 else np.uint16 if byte_count == 2 else np.uint32 if byte_count == 4 else np.uint64, count=size, sep="") 110 | elif type_val == 5: # untyped array 111 | array_size = read_compressed(fid) 112 | data = [] 113 | for _ in range(array_size): 114 | data.append(read_value(fid)) 115 | elif type_val == 6: # extensions 116 | extension = np.bitwise_and(header, 0b11111000) >> 3 117 | if extension == 1: # variants 118 | read_compressed(fid) 119 | data = read_value(fid) 120 | elif extension == 2: # matrices 121 | layout = np.bitwise_and(np.fromfile(fid, dtype=np.uint8, count=1, sep=""), 0b00000001)[0] 122 | if layout == 0: # row major 123 | raise NotImplementedError("Row major support not implemented") 124 | elif layout == 1: # column major 125 | extents = read_value(fid) 126 | matrix_data = read_value(fid) 127 | data = np.reshape(matrix_data, (extents[0], extents[1]), order ='F') 128 | else: 129 | raise ValueError("Unsupported layout") 130 | elif extension == 3: # complex numbers 131 | data = read_complex(fid) 132 | else: 133 | raise ValueError("Unsupported extension") 134 | else: 135 | raise ValueError("Unsupported type") 136 | 137 | return data 138 | 139 | def read_complex(fid): 140 | complex_header = np.fromfile(fid, dtype=np.uint8, count=1)[0] 141 | type_val = np.bitwise_and(complex_header, 0b00000111) 142 | 143 | num_type = np.bitwise_and(complex_header, 0b00011000) >> 3 144 | is_float = num_type == 0 145 | is_signed = num_type == 1 146 | 147 | byte_count_index = np.bitwise_and(complex_header, 0b11100000) >> 5 148 | byte_count = config[byte_count_index] 149 | 150 | if type_val == 0: # complex number 151 | complex_data = None 152 | if is_float: 153 | complex_data = np.fromfile(fid, dtype=np.float32 if byte_count == 4 else np.float64, count=2, sep="") 154 | else: 155 | if is_signed: 156 | complex_data = np.fromfile(fid, dtype=np.int8 if byte_count == 1 else np.int16 if byte_count == 2 else np.int32 if byte_count == 4 else np.int64, count=2, sep="") 157 | else: 158 | complex_data = np.fromfile(fid, dtype=np.uint8 if byte_count == 1 else np.uint16 if byte_count == 2 else np.uint32 if byte_count == 4 else np.uint64, count=2, sep="") 159 | data = complex(complex_data[0], complex_data[1]) 160 | elif type_val == 1: # complex array 161 | size = read_compressed(fid) 162 | 163 | if is_float: 164 | data = np.fromfile(fid, dtype=np.complex64 if byte_count == 4 else np.complex128, count=size, sep="") 165 | else: 166 | raise ValueError("Unsupported complex integer type") 167 | else: 168 | raise ValueError("Unsupported complex type") 169 | 170 | return data 171 | 172 | def read_compressed(fid): 173 | config = np.uint8([1, 2, 4, 8, 16, 32, 64, 128]) 174 | 175 | compressed = np.fromfile(fid, dtype=np.uint8, count=1)[0] 176 | n_size_bytes = config[np.bitwise_and(compressed, 0b00000011)] 177 | fid.seek(-1, 1) # revert one byte (offset, whence), whence of 1 is relative to the current position 178 | if n_size_bytes == 1: 179 | size = np.fromfile(fid, dtype=np.uint8, count=1)[0] 180 | elif n_size_bytes == 2: 181 | size = np.fromfile(fid, dtype=np.uint16, count=1)[0] 182 | elif n_size_bytes == 4: 183 | size = np.fromfile(fid, dtype=np.uint32, count=1)[0] 184 | elif n_size_bytes == 8: 185 | size = np.fromfile(fid, dtype=np.uint64, count=1)[0] 186 | else: 187 | raise ValueError("Unsupported size") 188 | size = np.right_shift(size, 2) 189 | return size 190 | 191 | config = np.uint8([1, 2, 4, 8, 16, 32, 64, 128]) 192 | 193 | with open(filename, 'rb') as fid: 194 | data = read_value(fid) 195 | 196 | return data 197 | 198 | # Load .beve file 199 | data = load_beve(None) 200 | print(data) -------------------------------------------------------------------------------- /rust/Cargo.toml: -------------------------------------------------------------------------------- 1 | [package] 2 | name = "rust" 3 | version = "0.1.0" 4 | edition = "2021" 5 | 6 | # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html 7 | 8 | [dependencies] 9 | num-complex = "0.4" 10 | rand = "0.8" 11 | num-bigint = { version = "0.4", features = ["rand"] } 12 | num-traits = "0.2" 13 | byteorder = "1" 14 | anyhow = "1.0" 15 | -------------------------------------------------------------------------------- /rust/src/main.rs: -------------------------------------------------------------------------------- 1 | use anyhow::Error; 2 | use std::convert::TryInto; 3 | use std::io::Cursor; 4 | use std::io::Read; 5 | use std::io::Write; 6 | use std::mem::size_of; 7 | use std::num::FpCategory; 8 | use std::str; 9 | use std::str::FromStr; 10 | //num_complex 11 | use num_complex::Complex; 12 | 13 | use byteorder::{LittleEndian, ReadBytesExt, WriteBytesExt}; 14 | use num_bigint::BigUint; 15 | use num_traits::cast::ToPrimitive; 16 | 17 | struct Beve { 18 | buffer: Vec, 19 | cursor: usize, 20 | } 21 | 22 | impl Beve { 23 | fn read_uint8(&mut self) -> u8 { 24 | let val = self.buffer[self.cursor]; 25 | self.cursor += 1; 26 | val 27 | } 28 | 29 | fn read_int8(&mut self) -> i8 { 30 | self.read_uint8() as i8 31 | } 32 | 33 | fn read_uint16(&mut self) -> u16 { 34 | let mut rdr = Cursor::new(&self.buffer[self.cursor..]); 35 | let val = rdr.read_u16::().unwrap(); 36 | self.cursor += 2; 37 | val 38 | } 39 | 40 | fn read_int16(&mut self) -> i16 { 41 | self.read_uint16() as i16 42 | } 43 | 44 | fn read_uint32(&mut self) -> u32 { 45 | let mut rdr = Cursor::new(&self.buffer[self.cursor..]); 46 | let val = rdr.read_u32::().unwrap(); 47 | self.cursor += 4; 48 | val 49 | } 50 | 51 | fn read_int32(&mut self) -> i32 { 52 | self.read_uint32() as i32 53 | } 54 | 55 | fn read_uint64(&mut self) -> u64 { 56 | let mut rdr = Cursor::new(&self.buffer[self.cursor..]); 57 | let val = rdr.read_u64::().unwrap(); 58 | self.cursor += 8; 59 | val 60 | } 61 | 62 | fn read_int64(&mut self) -> i64 { 63 | self.read_uint64() as i64 64 | } 65 | 66 | fn read_float(&mut self) -> f32 { 67 | let bits = self.read_uint32(); 68 | f32::from_bits(bits) 69 | } 70 | 71 | fn read_double(&mut self) -> f64 { 72 | let bits = self.read_uint64(); 73 | f64::from_bits(bits) 74 | } 75 | 76 | fn read_big_int64(&mut self) -> BigUint { 77 | let mut bytes = [0u8; 8]; 78 | bytes.copy_from_slice(&self.buffer[self.cursor..self.cursor + 8]); 79 | self.cursor += 8; 80 | BigUint::from_bytes_le(&bytes) 81 | } 82 | 83 | fn read_big_uint64(&mut self) -> BigUint { 84 | self.read_big_int64() 85 | } 86 | 87 | fn read_compressed(&mut self) -> i32 { 88 | let header = self.buffer[self.cursor]; 89 | self.cursor += 1; 90 | let config = header & 0b00000011; 91 | 92 | match config { 93 | 0 => (header >> 2) as i32, 94 | 1 => { 95 | let value = self.read_uint16(); 96 | (value >> 2) as i32 97 | } 98 | 2 => { 99 | let value = self.read_uint32(); 100 | (value >> 2) as i32 101 | } 102 | 3 => { 103 | let mut val = BigUint::default(); 104 | for _ in 0..8 { 105 | let byte = BigUint::from(self.buffer[self.cursor]); 106 | val |= &byte << (8 * self.cursor); 107 | self.cursor += 1; 108 | } 109 | val >>= 2; 110 | val.to_i64().unwrap() 111 | } 112 | _ => 0, 113 | } 114 | } 115 | 116 | fn read_string(&mut self) -> String { 117 | let size = self.read_compressed(); 118 | let str_bytes = &self.buffer[self.cursor..self.cursor + size as usize]; 119 | self.cursor += size as usize; 120 | String::from_utf8_lossy(str_bytes).to_string() 121 | } 122 | 123 | fn reshape(&mut self, data: Vec, rows: usize, cols: usize) -> Vec> { 124 | assert_eq!(rows * cols, data.len()); 125 | 126 | let mut result = Vec::with_capacity(rows); 127 | for i in 0..rows { 128 | let start = i * cols; 129 | let end = (i + 1) * cols; 130 | result.push(data[start..end].to_vec()); 131 | } 132 | result 133 | } 134 | 135 | fn read_complex(&mut self) -> num_complex::Complex { 136 | let real = self.read_double(); 137 | let imag = self.read_double(); 138 | num_complex::Complex::new(real, imag) 139 | } 140 | 141 | fn read_value(&mut self) -> Result, Box> { 142 | let header = self.buffer[self.cursor]; 143 | self.cursor += 1; 144 | let typ = header & 0b00000111; 145 | 146 | match typ { 147 | 0 => { 148 | let is_bool = (header & 0b00001000) >> 3; 149 | if is_bool > 0 { 150 | let value = (header & 0b11110000) >> 4 > 0; 151 | Ok(Box::new(value)) 152 | } else { 153 | Ok(Box::new(())) 154 | } 155 | } 156 | 1 => { 157 | let num_type = (header & 0b00011000) >> 3; 158 | let is_float = num_type == 0; 159 | let is_signed = num_type == 1; 160 | let byte_count_index = (header & 0b11100000) >> 5; 161 | let byte_count = [1, 2, 4, 8][byte_count_index as usize]; 162 | 163 | if is_float { 164 | match byte_count { 165 | 4 => { 166 | let value = self.read_float(); 167 | Ok(Box::new(value)) 168 | } 169 | 8 => { 170 | let value = self.read_double(); 171 | Ok(Box::new(value)) 172 | } 173 | _ => Err(Box::new("Unsupported float size".to_string())), 174 | } 175 | } else { 176 | if is_signed { 177 | match byte_count { 178 | 1 => { 179 | let value = self.read_int8(); 180 | Ok(Box::new(value)) 181 | } 182 | 2 => { 183 | let value = self.read_int16(); 184 | Ok(Box::new(value)) 185 | } 186 | 4 => { 187 | let value = self.read_int32(); 188 | Ok(Box::new(value)) 189 | } 190 | 8 => { 191 | let value = self.read_int64(); 192 | Ok(Box::new(value)) 193 | } 194 | _ => Err(Box::new("Unsupported signed integer size".to_string())), 195 | } 196 | } else { 197 | match byte_count { 198 | 1 => { 199 | let value = self.read_uint8(); 200 | Ok(Box::new(value)) 201 | } 202 | 2 => { 203 | let value = self.read_uint16(); 204 | Ok(Box::new(value)) 205 | } 206 | 4 => { 207 | let value = self.read_uint32(); 208 | Ok(Box::new(value)) 209 | } 210 | 8 => { 211 | let value = self.read_uint64(); 212 | Ok(Box::new(value)) 213 | } 214 | _ => Err(Box::new("Unsupported unsigned integer size".to_string())), 215 | } 216 | } 217 | } 218 | } 219 | 2 => { 220 | let value = self.read_string(); 221 | Ok(Box::new(value)) 222 | } 223 | 3 => { 224 | let key_type = (header & 0b00011000) >> 3; 225 | let is_string = key_type == 0; 226 | let n = self.read_compressed(); 227 | 228 | let mut object_data = std::collections::HashMap::new(); 229 | 230 | for _ in 0..n { 231 | if is_string { 232 | let key = self.read_string(); 233 | let value = self.read_value()?; 234 | object_data.insert(key, value); 235 | } else { 236 | return Err(Box::new("TODO: support integer keys".to_string())); 237 | } 238 | } 239 | 240 | Ok(Box::new(object_data)) 241 | } 242 | 4 => { 243 | let num_type = (header & 0b00011000) >> 3; 244 | let is_float = num_type == 0; 245 | let is_signed = num_type == 1; 246 | let byte_count_index_array = (header & 0b11100000) >> 5; 247 | let byte_count_array = [1, 2, 4, 8][byte_count_index_array as usize]; 248 | 249 | if num_type == 3 { 250 | let is_string = (header & 0b00100000) >> 5; 251 | if is_string != 0 { 252 | let n = self.read_compressed(); 253 | let mut array = Vec::with_capacity(n as usize); 254 | for _ in 0..n { 255 | let size = self.read_compressed(); 256 | let value = self.read_string(); 257 | array.push(value); 258 | } 259 | Ok(Box::new(array)) 260 | } else { 261 | Err(Box::new( 262 | "Boolean array support not implemented".to_string(), 263 | )) 264 | } 265 | } else if is_float { 266 | let n = self.read_compressed(); 267 | let mut array = Vec::with_capacity(n as usize); 268 | 269 | match byte_count_array { 270 | 4 => { 271 | for _ in 0..n { 272 | let value = self.read_float(); 273 | array.push(value); 274 | } 275 | } 276 | 8 => { 277 | for _ in 0..n { 278 | let value = self.read_double(); 279 | array.push(value); 280 | } 281 | } 282 | _ => return Err(Box::new("Unsupported float size".to_string())), 283 | } 284 | 285 | Ok(Box::new(array)) 286 | } else { 287 | let n = self.read_compressed(); 288 | let mut array = Vec::with_capacity(n as usize); 289 | 290 | if is_signed { 291 | match byte_count_array { 292 | 1 => { 293 | for _ in 0..n { 294 | let value = self.read_int8(); 295 | array.push(value); 296 | } 297 | } 298 | 2 => { 299 | for _ in 0..n { 300 | let value = self.read_int16(); 301 | array.push(value); 302 | } 303 | } 304 | 4 => { 305 | for _ in 0..n { 306 | let value = self.read_int32(); 307 | array.push(value); 308 | } 309 | } 310 | 8 => { 311 | for _ in 0..n { 312 | let value = self.read_big_int64(); 313 | array.push(value); 314 | } 315 | } 316 | _ => { 317 | return Err(Box::new("Unsupported signed integer size".to_string())) 318 | } 319 | } 320 | } else { 321 | match byte_count_array { 322 | 1 => { 323 | for _ in 0..n { 324 | let value = self.read_uint8(); 325 | array.push(value); 326 | } 327 | } 328 | 2 => { 329 | for _ in 0..n { 330 | let value = self.read_uint16(); 331 | array.push(value); 332 | } 333 | } 334 | 4 => { 335 | for _ in 0..n { 336 | let value = self.read_uint32(); 337 | array.push(value); 338 | } 339 | } 340 | 8 => { 341 | for _ in 0..n { 342 | let value = self.read_big_uint64(); 343 | array.push(value); 344 | } 345 | } 346 | _ => { 347 | return Err(Box::new( 348 | "Unsupported unsigned integer size".to_string(), 349 | )) 350 | } 351 | } 352 | } 353 | 354 | Ok(Box::new(array)) 355 | } 356 | } 357 | 5 => { 358 | let n = self.read_compressed(); 359 | let mut arr = Vec::with_capacity(n as usize); 360 | for _ in 0..n { 361 | let value = self.read_value()?; 362 | arr.push(value); 363 | } 364 | Ok(Box::new(arr)) 365 | } 366 | 6 => { 367 | let extension = (header & 0b11111000) >> 3; 368 | match extension { 369 | 1 => { 370 | let _ = self.read_compressed(); // Skip variant tag 371 | self.read_value() 372 | } 373 | 2 => { 374 | let layout = self.buffer[self.cursor] & 0b00000001; 375 | self.cursor += 1; 376 | match layout { 377 | 0 => Err(Box::new( 378 | "Row major matrix layout not implemented".to_string(), 379 | )), 380 | 1 => { 381 | let extents = self.read_value()?; 382 | let extents_slice = extents 383 | .downcast_ref::>>() 384 | .unwrap(); 385 | let rows = extents_slice[0].downcast_ref::().unwrap(); 386 | let cols = extents_slice[1].downcast_ref::().unwrap(); 387 | 388 | let matrix_data = self.read_value()?; 389 | match matrix_data.downcast_ref::>() { 390 | Some(data) => { 391 | let reshaped = self.reshape(data.clone(), *rows, *cols); 392 | Ok(Box::new(reshaped)) 393 | } 394 | None => match matrix_data.downcast_ref::>() { 395 | Some(data) => { 396 | let reshaped = self.reshape(data.clone(), *rows, *cols); 397 | Ok(Box::new(reshaped)) 398 | } 399 | None => match matrix_data.downcast_ref::>() { 400 | Some(data) => { 401 | let reshaped = self.reshape(data.clone(), *rows, *cols); 402 | Ok(Box::new(reshaped)) 403 | } 404 | None => match matrix_data.downcast_ref::>() { 405 | Some(data) => { 406 | let reshaped = self.reshape(data.clone(), *rows, *cols); 407 | Ok(Box::new(reshaped)) 408 | } 409 | None => match matrix_data.downcast_ref::>() { 410 | Some(data) => { 411 | let reshaped = self.reshape(data.clone(), *rows, *cols); 412 | Ok(Box::new(reshaped)) 413 | } 414 | None => match matrix_data.downcast_ref::>() { 415 | Some(data) => { 416 | let reshaped = self.reshape(data.clone(), *rows, *cols); 417 | Ok(Box::new(reshaped)) 418 | } 419 | None => match matrix_data.downcast_ref::>() { 420 | Some(data) => { 421 | let reshaped = self.reshape(data.clone(), *rows, *cols); 422 | Ok(Box::new(reshaped)) 423 | } 424 | None => match matrix_data.downcast_ref::>() { 425 | Some(data) => { 426 | let reshaped = self.reshape(data.clone(), *rows, *cols); 427 | Ok(Box::new(reshaped)) 428 | } 429 | None => match matrix_data.downcast_ref::>() { 430 | Some(data) => { 431 | let reshaped = self.reshape(data.clone(), *rows, *cols); 432 | Ok(Box::new(reshaped)) 433 | } 434 | None => match matrix_data.downcast_ref::>() { 435 | Some(data) => { 436 | let reshaped = self.reshape(data.clone(), *rows, *cols); 437 | Ok(Box::new(reshaped)) 438 | } 439 | None => match matrix_data.downcast_ref::>() { 440 | Some(data) => { 441 | let reshaped = self.reshape(data.clone(), *rows, *cols); 442 | Ok(Box::new(reshaped)) 443 | } 444 | None => Err(Box::new("Unsupported matrix data type".to_string())), 445 | }, 446 | }, 447 | }, 448 | }, 449 | }, 450 | }, 451 | }, 452 | }, 453 | }, 454 | }, 455 | } 456 | } 457 | _ => Err(Box::new("Unsupported matrix layout".to_string())), 458 | } 459 | } 460 | 3 => { 461 | let complex = self.read_complex(); 462 | Ok(Box::new(complex)) 463 | } 464 | _ => Err(Box::new("Unsupported extension".to_string())), 465 | } 466 | } 467 | _ => Err(Box::new("Unsupported type".to_string())), 468 | } 469 | } 470 | } 471 | 472 | struct Writer { 473 | buffer: Vec, 474 | offset: usize, 475 | } 476 | 477 | impl Writer { 478 | fn new(size: usize) -> Self { 479 | let size = if size <= 0 { 256 } else { size }; 480 | Writer { 481 | buffer: vec![0; size], 482 | offset: 0, 483 | } 484 | } 485 | 486 | fn ensure_capacity(&mut self, size: usize) { 487 | if self.offset + size > self.buffer.len() { 488 | let new_size = (self.buffer.len() + size) * 2; 489 | self.buffer.resize(new_size, 0); 490 | } 491 | } 492 | 493 | fn append_uint8(&mut self, value: u8) -> Result<(), Box> { 494 | if value > 255 { 495 | return Err(Box::new( 496 | "Value must be an integer between 0 and 255".to_string(), 497 | )); 498 | } 499 | self.ensure_capacity(1); 500 | self.buffer[self.offset] = value; 501 | self.offset += 1; 502 | Ok(()) 503 | } 504 | 505 | fn append_uint16(&mut self, value: u16) -> Result<(), Box> { 506 | if value > 65535 { 507 | return Err(Box::new( 508 | "Value must be an integer between 0 and 65535".to_string(), 509 | )); 510 | } 511 | self.ensure_capacity(2); 512 | self.buffer[self.offset..].write_u16::(value)?; 513 | self.offset += 2; 514 | Ok(()) 515 | } 516 | 517 | fn append_uint32(&mut self, value: u32) -> Result<(), Box> { 518 | if value > 4294967295 { 519 | return Err(Box::new( 520 | "Value must be an integer between 0 and 4294967295".to_string(), 521 | )); 522 | } 523 | self.ensure_capacity(4); 524 | self.buffer[self.offset..].write_u32::(value)?; 525 | self.offset += 4; 526 | Ok(()) 527 | } 528 | 529 | fn append_uint64(&mut self, value: &BigUint) -> Result<(), Box> { 530 | if value < &BigUint::from(0) || value > &BigUint::from(18446744073709551615u64) { 531 | return Err(Box::new( 532 | "Value must be an integer between 0 and 18446744073709551615".to_string(), 533 | )); 534 | } 535 | self.ensure_capacity(8); 536 | let low = value & BigUint::from(0xffffffff); 537 | let high = value >> 32; 538 | 539 | self.buffer[self.offset..].write_u32::(low.to_u32().unwrap())?; 540 | self.buffer[self.offset + 4..].write_u32::(high.to_u32().unwrap())?; 541 | self.offset += 8; 542 | Ok(()) 543 | } 544 | 545 | fn append(&mut self, value: T) -> Result<(), Box> { 546 | match value.downcast_ref::>>() { 547 | Some(arr) => { 548 | for element in arr { 549 | self.append(element)?; 550 | } 551 | } 552 | None => match value.downcast_ref::() { 553 | Some(s) => { 554 | let bytes = s.as_bytes(); 555 | self.ensure_capacity(bytes.len()); 556 | self.buffer[self.offset..self.offset + bytes.len()].copy_from_slice(bytes); 557 | self.offset += bytes.len(); 558 | } 559 | None => match value.downcast_ref::() { 560 | Some(i) => { 561 | self.ensure_capacity(4); 562 | self.buffer[self.offset..].write_i32::(*i)?; 563 | self.offset += 4; 564 | } 565 | None => match value.downcast_ref::() { 566 | Some(f) => { 567 | self.ensure_capacity(8); 568 | self.buffer[self.offset..].write_f64::(*f)?; 569 | self.offset += 8; 570 | } 571 | None => return Err(Box::new("Unsupported value type".to_string())), 572 | }, 573 | }, 574 | }, 575 | } 576 | Ok(()) 577 | } 578 | 579 | fn write_beve( 580 | &mut self, 581 | data: Box, 582 | ) -> Result, Box> { 583 | self.append(data)?; 584 | Ok(self.buffer[..self.offset].to_vec()) 585 | } 586 | } 587 | 588 | fn write_value( 589 | writer: &mut Writer, 590 | value: Box, 591 | ) -> Result<(), Box> { 592 | match value.downcast_ref::>() { 593 | Some(arr) => { 594 | writer.append_uint8(0b01100000 | 4)?; // float64_t, 8 bytes 595 | writer.append_uint32(arr.len().try_into().unwrap())?; 596 | for f in arr { 597 | writer.append(f)?; 598 | } 599 | } 600 | None => { 601 | match value.downcast_ref::>() { 602 | Some(arr) => { 603 | writer.append_uint8(0b01001000 | 4)?; // int32_t, 4 bytes 604 | writer.append_uint32(arr.len().try_into().unwrap())?; 605 | for i in arr { 606 | writer.append(i)?; 607 | } 608 | } 609 | None => { 610 | match value.downcast_ref::() { 611 | Some(b) => { 612 | if *b { 613 | writer.append_uint8(0b00011000)?; 614 | } else { 615 | writer.append_uint8(0b00001000)?; 616 | } 617 | } 618 | None => match value.downcast_ref::() { 619 | Some(i) => { 620 | if f64::from(*i) - f64::from(*i as i64) != 0.0 { 621 | writer.append_uint8(0b01100000)?; 622 | } else { 623 | writer.append_uint8(0b01001000)?; 624 | } 625 | writer.append(i)?; 626 | } 627 | None => match value.downcast_ref::() { 628 | Some(s) => { 629 | writer.append_uint8(2)?; 630 | writer.append_uint32(s.len().try_into().unwrap())?; 631 | writer.append(s)?; 632 | } 633 | None => match value.downcast_ref::>>() { 634 | Some(arr) => { 635 | writer.append_uint8(5)?; 636 | writer.append_uint32(arr.len().try_into().unwrap())?; 637 | for val in arr { 638 | write_value(writer, val.clone())?; 639 | } 640 | } 641 | None => { 642 | match value.downcast_ref::, 645 | >>() { 646 | Some(map) => { 647 | writer.append_uint8(3)?; // Assume string keys 648 | writer 649 | .append_uint32(map.len().try_into().unwrap())?; 650 | for (key, val) in map { 651 | writer.append_uint32( 652 | key.len().try_into().unwrap(), 653 | )?; 654 | writer.append(key)?; 655 | write_value(writer, val.clone())?; 656 | } 657 | } 658 | None => { 659 | return Err(Box::new( 660 | "Unsupported data type".to_string(), 661 | )) 662 | } 663 | } 664 | } 665 | }, 666 | }, 667 | }, 668 | } 669 | } 670 | } 671 | } 672 | } 673 | Ok(()) 674 | } 675 | --------------------------------------------------------------------------------