├── LICENSE └── README.md /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2020 Nick Krecklow 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # FSEQ File Format 2 | A third-party reverse engineering of the *version 2* FSEQ (`.fseq`) "Falcon sequence" file format, derived from the [Falcon Player](https://github.com/FalconChristmas/fpp) ("fpp") software implementation and its usage in [xLights](https://github.com/smeighan/xLights). FSEQ is a time series file format used to describe channel output states, typically for controlling lighting equipment. 3 | 4 | First-party documentation concerning the file format, as well as the [ESEQ](https://github.com/FalconChristmas/fpp/blob/master/docs/ESEQ_Effect_Sequence_file_format.txt) file format, is available in the [fpp repository](https://github.com/FalconChristmas/fpp/blob/master/docs/FSEQ_Sequence_File_Format.txt). 5 | 6 | Given the reverse engineered nature, this documentation should be considered incomplete, incorrect and outdated. 7 | 8 | ## Encoding 9 | FSEQ files are encoded in [little-endian](https://en.wikipedia.org/wiki/Endianness) format. 10 | 11 | ## Diagram 12 | ``` 13 | ┌────────────────┬────────────────┬────────────────┬────────────────┐ 14 | │'P' 0x50│'S' 0x53│'E' 0x45│'Q' 0x51│ 15 | ├────────────────┴────────────────┼────────────────┼────────────────┤ 16 | │Channel Data Offset uint16│Minor Version │Major Version 2│ 17 | ┌────────────────────────┐ ├─────────────────────────────────┼────────────────┴────────────────┤ 18 | │Extended Compression │ │Variable Data Offset uint16│Channel Count uint32 19 | │Block Count (v2.1) uint4│ ├─────────────────────────────────┼─────────────────────────────────┤ 20 | ├────────────────────────┤ Channel Count (contd) uint32│Frame Count uint32 21 | │Compression Type uint4│ ├─────────────────────────────────┼────────────────┬────────────────┤ 22 | └────────────────────────┘ Frame Count (contd) uint32│Step Time Ms. │Flags (unused) │ 23 | │ ├────────┬───────┬────────────────┼────────────────┼────────────────┤ 24 | └─────────────▶│ECBC │CT │Compr. Block Cnt│Sparse Range Cnt│Reserved │ 25 | ├────────┴───────┴────────────────┴────────────────┴────────────────┤ 26 | │Unique ID/Creation Time Microseconds uint64 27 | ├───────────────────────────────────────────────────────────────────┤ 28 | Unique ID/Creation Time Microseconds (contd) uint64│ 29 | └───────────────────────────────────────────────────────────────────┘ 30 | ``` 31 | 32 | ## Structure 33 | ### Header 34 | Size is 32 bytes. 35 | 36 | | Byte Index | Data Type | Field Name | Notes | 37 | | --- | --- | --- | --- | 38 | | 0 | `[4]uint8` | Identifier | Always `PSEQ` (older encodings may contain `FSEQ`) | 39 | | 4 | `uint16` | Channel Data Offset | Byte index of the channel data portion of the file | 40 | | 6 | `uint8` | Minor Version | Normally `0x00`, optionally `0x01` is required to enable support for [Extended Compression Blocks](#extended-compression-blocks) (see [xLights@e33c065](https://github.com/smeighan/xLights/commit/e33c0651aa6886d2ab10c04cb83ef1d1fdd25062)) | 41 | | 7 | `uint8` | Major Version | Currently `0x02` | 42 | | 8 | `uint16` | Header Size | Address of first variable. Equal to the size of the header (32 bytes) + `Compression Block Count` * size of a `Compression Block` (8 bytes) + `Sparse Range Count` * size of a `Sparse Range` (12 bytes) | 43 | | 10 | `uint32` | Channel Count | Sum of `Sparse Range` channel counts, or a freestanding absolute value when `Sparse Range Count` = 0 | 44 | | 14 | `uint32` | Frame Count | Total number of frames in the sequence | 45 | | 18 | `uint8` | Step Time | Frame timing interval in milliseconds | 46 | | 19 | `uint8` | Flags | Unused by the [fpp](https://github.com/FalconChristmas/fpp) & [xLights](https://github.com/smeighan/xLights) implementations | 47 | | 20 | `uint4` | Ext. Compression Block Count | Upper 4 bits, likely 0 | 48 | | 20 | `uint4` | Compression Type | 0 = none, 1 = [zstd](https://github.com/facebook/zstd), 2 = [zlib](https://www.zlib.net/) | 49 | | 21 | `uint8` | Compression Block Count | Lower 8 bits, ignored if `Compression Type` = 0 | 50 | | 22 | `uint8` | Sparse Range Count | | 51 | | 23 | `uint8` | | (Reserved for future use) | 52 | | 24 | `uint64` | Unique ID | Implemented as the creation time in microseconds | 53 | 54 | ### Data Tables 55 | Variable size, determined by `Header->Channel Data Offset` - size of `Header` (32 bytes). 56 | 57 | | Data Type | Corresponding Count Field | 58 | | --- | --- | 59 | | `[]Compression Block` | `Header->Compression Block Count` | 60 | | `[]Sparse Range` | `Header->Sparse Range Count` | 61 | | `[]Variable`| None | 62 | 63 | When reading the `[]Variable` data, a software implementation should instead continue to read as long as there is at least 4 bytes free between the current reader index and `Header->Channel Data Offset`. See the [fpp](https://github.com/FalconChristmas/fpp/blob/master/src/fseq/FSEQFile.cpp#L425) source code for more context. 64 | 65 | If (after reading all fields) the current reader index is less than the `Header->Channel Data Offset` value, it should seek forward (up to 4 bytes) to `Header->Channel Data Offset`. A difference of more than 4 bytes likely denotes a decoding error in your implementation. See the [fpp](https://github.com/FalconChristmas/fpp/blob/master/src/fseq/FSEQFile.cpp#L1458-L1462) source code for more context. 66 | 67 | Both of these implementation details appear to stem from the [rounding behavior](https://github.com/FalconChristmas/fpp/blob/master/src/fseq/FSEQFile.cpp#L1433) for the `Channel Data Offset`. 68 | 69 | #### Compression Block 70 | Size is 8 bytes. 71 | 72 | | Byte Index | Data Type | Field Name | Notes | 73 | | --- | --- | --- | --- | 74 | | 0 | `uint32` | First Frame Number | Index of the first frame encoded in the block | 75 | | 4 | `uint32` | Size | Size in bytes of the compressed block | 76 | 77 | #### Sparse Range 78 | Size is 6 bytes. 79 | 80 | | Byte Index | Data Type | Field Name | 81 | | --- | --- | --- | 82 | | 0 | `uint24` | Start Channel | 83 | | 3 | `uint24` | End Channel Offset | 84 | 85 | A `Sparse Range` is a channel range, defined by its starting channel and the range count. Channel indexes start at 0. 86 | 87 | ##### Example 88 | To denote channels 16-32, `Start Channel` would have a value of 15 (16 - 1 since channel indexes start at 0) and an `End Channel Offset` of 16 (32 - 16 = 16). 89 | 90 | #### Variable 91 | Variable size, at least 4 bytes due to the header structure. 92 | 93 | | Byte Index | Data Type | Field Name | Notes | 94 | | --- | --- | --- | --- | 95 | | 0 | `uint16` | Size | Size of `Data` + this 4 byte header | 96 | | 2 | `[2]uint8` | Code | Two character unique ID | 97 | | 4 | `[Data Size - 4]uint8` | Data | Typically used as a string | 98 | 99 | While effectively useless, the [fpp](https://github.com/FalconChristmas/fpp) implementation seems to support zero length variables. However when reading it skips forward 4 bytes, ignoring the `Code` field and resulting in a variable named "NULNUL" (`[0x00, 0x00]`). 100 | 101 | The `Data` field may be null terminated depending on the encoding program. If your programming language does not null terminate its strings, your `Data` array should instead be `[Data Size - 4 - 1]uint8`. 102 | 103 | ##### Common Variable Codes 104 | | Code | Name | Description | Example | 105 | | --- | --- | --- | --- | 106 | | `mf` | Media File | File path of the audio to play | `C:\test.mp3` | 107 | | `sp` | Sequence Producer | Identifies the program used to create the sequence | `xLights Macintosh 2019.22` | 108 | 109 | These (2) variable codes are currently the only codes referenced by the [fpp](https://github.com/FalconChristmas/fpp) & [xLights](https://github.com/smeighan/xLights) implementations. However, there is no validation that prevents third-party software or users from defining and using their own codes. The caveat being that they may clash with future additions given the limited namespace availability. 110 | 111 | **Author's Note**: Use of the `mf` variable should be discouraged. Encoding the media file path directly into the sequence results in path breaks when moving files, requiring a re-export of the sequence file. As a minor security flaw, `mf` may expose the user's file structure unintentionally when distributing sequences. Software implementations should instead store this in an external configuration, only using the `mf` value if present and valid, as a configuration fallback. 112 | 113 | ##### Recommended Variable Codes 114 | | Code | Name | Description | 115 | | --- | --- | --- | 116 | | `an` | Author Name | Name of the file's author | 117 | | `ae` | Author Email | Email address of the file's author | 118 | | `aw` | Author Website | Website URL of the file's author | 119 | | `ad` | Authorship Date | ISO-8601 compliant date & timestamp of the file's authorship | 120 | 121 | These variable codes are not supported by either the [fpp](https://github.com/FalconChristmas/fpp) or the [xLights](https://github.com/smeighan/xLights) implementations, but recommended for new software implementations as a method for optionally storing authorship metadata within the existing file format. 122 | 123 | ##### Encoding Example 124 | The variable `mf` (Media File) with a value of "xy" would be encoded in 6 bytes. 125 | 126 | | Bytes | Description | 127 | | --- | --- | 128 | | `[0x00, 0x06]` | 6 byte size (2 bytes of data + 4 byte header) | 129 | | `[0x6D, 0x66]` | 2 byte code (`mf`) | 130 | | `[0x78, 0x79]` | 2 bytes of data ("xy") | 131 | 132 | ### Compressed Channel Data 133 | For compressed FSEQ files, the channel data is written normally as uncompressed channel data, and then split into fixed-size chunk allocations which are then individually compressed (with up to 4095 of these chunks per file). This enables software implementations to decompress chunks of the file without buffering the full file size. 134 | 135 | #### Extended Compression Blocks 136 | The `Compression Block Count` value is split across two seperate fields. [This change](https://github.com/smeighan/xLights/commit/9d09555728f43c863ab24118ba901f4ae45dc3c5#diff-6f85e85fd47664285c3b8811d79aff161ebce060186e33ddea44e1f38f121283R1412) was done to increase the compression block limit to 4095 (previously 255) without extreme breakage in backwards compatability. Although the current minor version is `0x00`, a minor version greater than or equal to `0x01` is required to enable support within xLights. See [xLights@e33c065](https://github.com/smeighan/xLights/commit/e33c0651aa6886d2ab10c04cb83ef1d1fdd25062). 137 | 138 | `Compression Block Count` should be treated as a `uint16` value, however only the lower 12 bits are used. The upper 4 bits (of the lower 12 used bits) are stored as the upper 4 bits of `Compression Type` (the "extended" compression block count). As such, it is important when working with the `Compression Type` field to ignore the lower 4 bits (which store the compression type enum value) and shift the upper 4 bits to index 0. The lower 8 bits are stored within the pre-existing `Compression Block Count` field. 139 | 140 | ##### Code Examples 141 | ``` 142 | // Prefix the 8-bit compressionBlockCount field with the upper 4 bits of compressionType as a 12-bit int within a uint16_t, 143 | // this 4 highest order bits will remain zero 144 | // 145 | // +-> unused highest 4 bits, 146 | // | set to zero 147 | // | 148 | // +------------------+ 149 | // |0000| | +--> original 8 bit 150 | // +------------------+ `compressionBlockCount` 151 | // | 152 | // | 153 | // +-> upper 4 bits of 154 | // `compressionType` 155 | // 156 | uint16_t extendedCompressionBlockCount = ((compressionType & 0xF0) << 4) | compressionBlockCount; 157 | ``` 158 | 159 | ``` 160 | // Strip the 4-bit prefix and re-align it with the 4-bit compressionType, encode the remaining 8 bits as the compressionBlockCount 161 | uint8_t encodedCompressionType = ((extendedCompressionBlockCount & 0x0F00) >> 4) | (compressionType & 0xF); 162 | uint8_t encodedCompressionBlockCount = extendedCompressionBlockCount & 0x00FF; 163 | ``` 164 | 165 | #### Odds & Ends 166 | - The first `Compression Block` will only contain 10 frames. Comments within the [fpp source code](https://github.com/FalconChristmas/fpp/blob/master/src/fseq/FSEQFile.cpp#L1129) indicates this is done to ensure the program can be started quicker. 167 | - When compressed using zstd, [fpp](https://github.com/FalconChristmas/fpp) & [xLights](https://github.com/smeighan/xLights/blob/master/xLights/FSEQFile.cpp) do not include the [Zstandard frame header](https://github.com/facebook/zstd/blob/master/doc/zstd_compression_format.md#zstandard-frames). In its absence, some libraries (such as the [Python zstandard library](https://pypi.org/project/zstandard/#id2)) may require an explicit content length be provided. 168 | - [Depending on the zlib version](https://github.com/FalconChristmas/fpp/blob/master/src/fseq/FSEQFile.cpp#L1086), the first `Compression Block` might not be compressed, even if the remaining data is. 169 | - [fpp source code](https://github.com/FalconChristmas/fpp/blob/master/src/fseq/FSEQFile.cpp#L781) attempts to allocate 64KB compression blocks. However instead of `64 * 1024`, it uses `64 * 2014` which allocates 125.8KB compression blocks. **Update: This bug has been fixed. This note is preserved for sequences encoded prior to the fix.** 170 | 171 | #### Encoding Example 172 | A compressed file (in this example, with `Compression Type` = 1/zstd) with a size of 50,454 bytes reports a `Compression Block Count` of 4. A software implementation should begin by reading the `Compression Block` values that follow the 32 byte `Header`. 173 | 174 | ``` 175 | Block #0 176 | First Frame Number: 0 177 | Size: 18 178 | 179 | Block #1 180 | First Frame Number: 10 181 | Size: 50268 182 | 183 | Block #2 184 | First Frame Number: 0 185 | Size: 0 186 | 187 | Block #3 188 | First Frame Number: 0 189 | Size: 0 190 | ``` 191 | 192 | The [fpp source code](https://github.com/FalconChristmas/fpp/blob/master/src/fseq/FSEQFile.cpp#L1508) immediately discards any `Compression Block` values with a size of 0 (they appear to be a product of memory alignment), leaving us with the initial 2 values. 193 | 194 | The start address (relative to `Header->Channel Data Offset`) of each `Compression Block` can be calculated by summing the `Size` value of all previous `Compression Block` values. The end address can be calculated by adding the `Size` value to the previously calculated start address. 195 | 196 | ``` 197 | Block #0 198 | First Frame Number: 0 199 | Size: 18 200 | Relative Start Address: 0 201 | Relative End Address: 18 202 | 203 | Block #1 204 | Size Frame Number: 10 205 | Size: 50268 206 | Relative Start Addressing: 18 207 | Relative End Address: 50286 208 | 209 | (Blocks #2 & #3 ignored due to Size = 0) 210 | ``` 211 | 212 | To help validate software implementations, the end address of the last `Compression Block` + `Header->Channel Data Offset` should match the size of the file. 213 | 214 | If no `Compression Block` values were read, the [fpp source code](https://github.com/FalconChristmas/fpp/blob/master/src/fseq/FSEQFile.cpp#L1522) will consider the file corrupted. It will attempt to recover the file by inserting a `Compression Block` with a `First Frame Number` value of 0 and a `Size` value of the file's size - `Header->Channel Data Offset`. 215 | 216 | ### Uncompressed Channel Data 217 | Variable size, `Header->Channel Count` * `Header->Frame Count` bytes. 218 | 219 | For each frame between [0, `Header->Frame Count`) a software implementation should read a `[Header->Channel Count]uint8` array. Each `uint8` within this array represents the channel state for its given index. If applicable, remember to apply its corresponding `Sparse Range->Start Channel` offset. 220 | 221 | Uncompressed channel data starts at `Header->Channel Data Offset` and continues to the end of the file. A software implementation may easily validate the file by ensuring that `Header->Channel Data Offset` + `Header->Channel Count` * `Header->Frame Count` matches the full size of the file. 222 | 223 | #### Seeking 224 | A software implementation can seek to a specific frame by taking the product of the frame index and the size of each frame. 225 | 226 | ``` 227 | var targetFrameIndex = 9 // Frame #10 228 | var absoluteSeek = Header->Channel Data Offset // Skip the header and data tables 229 | absoluteSeek += targetFrameIndex * Header->Channel Count 230 | ``` 231 | 232 | #### Encoding Example 233 | A controller with 4 channels (indexes 0-3) would have its data encoded as `[4]uint8` _per frame_. Given a sequence with 10 total frames, the channel data size would be 40 bytes. How that data is interpreted is controlled by the controller's corresponding [channeloutput](https://github.com/FalconChristmas/fpp/tree/master/src/channeloutput) class. For example, the [Light-O-Rama channeloutput](https://github.com/FalconChristmas/fpp/blob/master/src/channeloutput/LOR.cpp#L243), and many others, interpret the `uint8` as a brightness level (with RGB simply using a channel per color). 234 | 235 | ## Reference Implementations 236 | * [fpp](https://github.com/FalconChristmas/fpp/blob/master/src/fseq/FSEQFile.cpp) is a C++ implementation of the FSEQ file format. It is the project which also originated the file format and maintains it. 237 | * [xLights](https://github.com/smeighan/xLights/blob/master/xLights/FSEQFile.cpp) is a C++ sequencing program which uses the FSEQ file format. However, its implementation is a copy/paste of the fpp source code and provides no additional context. 238 | * [libtinyfseq](https://github.com/Cryptkeeper/libtinyfseq) is a header-only C library for decoding fseq files. 239 | * [fplayer](https://github.com/Cryptkeeper/fplayer) is a FSEQ file player for LOR hardware. 240 | --------------------------------------------------------------------------------