├── CMakeLists.txt ├── LICENSE ├── README.md ├── driver ├── STM32QSPIStorageBank.cpp ├── STM32QSPIStorageBank.h ├── STM32StorageBank.cpp ├── STM32StorageBank.h ├── StorageBank.h ├── TestStorageBank.cpp └── TestStorageBank.h ├── kvs ├── BankHeader.h ├── KVS.cpp ├── KVS.h └── LogEntry.h └── tests ├── .gitignore ├── Makefile └── main.cpp /CMakeLists.txt: -------------------------------------------------------------------------------- 1 | # CMake build script for microkvs 2 | # Intended to be integrated into a larger project, not built standalone. 3 | 4 | add_library(microkvs STATIC 5 | driver/STM32QSPIStorageBank.cpp 6 | driver/STM32StorageBank.cpp 7 | driver/TestStorageBank.cpp 8 | 9 | kvs/KVS.cpp 10 | ) 11 | 12 | # TODO: only for stm32 targets? 13 | target_include_directories(microkvs 14 | PUBLIC ${CMAKE_SOURCE_DIR} 15 | "$" 16 | ) 17 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | BSD 3-Clause License 2 | 3 | Copyright (c) 2021, Andrew Zonenberg 4 | All rights reserved. 5 | 6 | Redistribution and use in source and binary forms, with or without 7 | modification, are permitted provided that the following conditions are met: 8 | 9 | * Redistributions of source code must retain the above copyright notice, this 10 | list of conditions and the following disclaimer. 11 | 12 | * Redistributions in binary form must reproduce the above copyright notice, 13 | this list of conditions and the following disclaimer in the documentation 14 | and/or other materials provided with the distribution. 15 | 16 | * Neither the name of the copyright holder nor the names of its 17 | contributors may be used to endorse or promote products derived from 18 | this software without specific prior written permission. 19 | 20 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 21 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 22 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 23 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 24 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 25 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 26 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 27 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 28 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Introduction 2 | 3 | microkvs is a log structured key-value store for embedded devices intended for storage of configuration data, 4 | encryption keys, and other relatively small information that needs to be persisted across power cycles. An object 5 | oriented driver layer provides access to the underlying storage media, typically either unused space in code flash or 6 | an external SPI NOR flash. This enables microkvs to be easily ported to new platforms in the future. 7 | 8 | There are no dependencies on dynamic RAM allocation or any fancy libc APIs that might be unavailable in an embedded 9 | platform. 10 | 11 | This is a work in progress and subject to change. Until a formal release, there may be breaking changes to the flash 12 | data structure at any time. 13 | 14 | # General concepts 15 | 16 | From a programmer's perspective, microkvs provides access to a series of objects (arbitrary blobs of binary data), 17 | identified by keys (16-character ASCII strings, analogous to file names although no hierarchy is supported). 18 | 19 | Objects are immutable and copy-on-write due to the nature of the underlying raw flash storage, which defaults to all 1s 20 | in the erased state and can only be written to logic 0. Writes typically support fine granularity (down to byte level) 21 | however erases have very large granularity (4 to 256 kB is common). 22 | 23 | microkvs does not support extent based / fragmented object storage. All objects are stored contiguously in the media; 24 | any change to an object requires completely rewriting it. If portions of an object need to be changed frequently while 25 | others do not, consider breaking it up into several objects to reduce write amplification effects. 26 | 27 | If memory mapping is supported by the underlying storage, objects can be directly memory mapped for read-only access. 28 | Memory mapped writing is not supported due to hardware limitations. 29 | 30 | # Architecture details 31 | 32 | The backing store for microkvs consists of two equally sized "banks" of flash memory in separate erase blocks. Microkvs 33 | treats each bank as one erase block at a logical level even if it physically consists of multiple blocks which must be 34 | erased in sequence, since the typical use case is to store a small amount of configuration data which easily fits in a 35 | single flash block. 36 | 37 | microkvs prefers byte writable storage. It can function with storage that has larger write block granularity (for 38 | example STM32H7 internal flash memory, with a 256 bit / 32 byte write block size) however overhead is increased as all 39 | data structures must be padded to multiples of a write block. Byte writable storage is assumed by default; to enable 40 | padding set a global preprocessor definition MICROKVS_WRITE_BLOCK_SIZE to the desired block size. 41 | 42 | The store is divided into two regions, log and data. The split must be decided at compile time and cannot be changed 43 | later on. The optimal split is application dependent and varies based on average file size weighted by how often each 44 | file is modified. A minimum of 28 bytes of storage are required in the log area for each object stored in the data 45 | area, unless padded by minimum write block sizes. 46 | 47 | For example, with 256K byte storage and 228 byte average file size, a good split would be 28K bytes of log and 228K 48 | bytes of data. This would allow roughly 1024 objects worth of both log and payload to be stored before both areas are 49 | exhausted simultaneously and a garbage collection is required. 50 | 51 | ## Locating the active bank 52 | 53 | Each bank has a static header at the start of the log area containing a 32-bit version number. Version numbers start at 54 | 0 and increment; the special value 0xffffffff is used for marking an invalid/blank block. No provision is made for 55 | handling overflow as typical flash memory has far less than 2^32 erase cycles of endurance, so the media will wear out 56 | before the counter wraps. 57 | 58 | The bank with the highest non-0xffffffff version number is active. 59 | 60 | ## Locating an object 61 | 62 | To locate an object, the log in the active bank is scanned from start to end searching for entries with the requested 63 | key. The last matching entry in the log points to the current version of the object. 64 | 65 | The CRC-32 checksum of the object is verified before the location is returned. If the checksum fails, earlier versions 66 | of the object (if present) are scanned, and the most recent version with a valid checksum is returned. If no copy with 67 | a valid checksum can be located, an error is returned. 68 | 69 | ## Writing an object 70 | 71 | To write an object, the start address and length of the last valid log entry are used to calculate the location of the 72 | first free byte in the data area. If insufficient space is present, or no free log space is available, a garbage 73 | collection must be performed. 74 | 75 | Once the space is confirmed available, a new log entry is created and the start address and length are written. The key 76 | and CRC are left blank since the data is not yet present (the previous version of the object is current), however the 77 | space is now reserved and cannot be used by another object. 78 | 79 | After reading back the log entry to confirm a correct write, the object content is written and the CRC is calculated. 80 | The object content is immediately read back to confirm a correct write; if an error is seen the log entry is left with 81 | a blank key and CRC (space reserved but data ignored) and the write is attempted again to a new location. 82 | 83 | Once all object data has been successfully written, the CRC and name in the log entry are written to commit the object. 84 | 85 | ## Garbage collection 86 | 87 | To garbage collect, the inactive block is erased. The latest entry of each object is then written to the new block, and 88 | after verification the block header is written with a new revision number one higher than the current. 89 | 90 | # Flash storage format 91 | 92 | ## Bank header 93 | 94 | ``` 95 | uint32_t magic = 0xc0def00d 96 | uint32_t version 97 | uint32_t logSize 98 | ``` 99 | 100 | ## Log entry 101 | 102 | ``` 103 | char key[16] 104 | uint32_t start 105 | uint32_t len 106 | uint32_t crc 107 | ``` 108 | 109 | ## Data area 110 | 111 | Data objects consist of raw binary data with a CRC-32 checksum (using the common Ethernet/ZIP polynomial 0x04c11db7) 112 | appended. 113 | -------------------------------------------------------------------------------- /driver/STM32QSPIStorageBank.cpp: -------------------------------------------------------------------------------- 1 | /*********************************************************************************************************************** 2 | * * 3 | * microkvs * 4 | * * 5 | * Copyright (c) 2021-2025 Andrew D. Zonenberg and contributors * 6 | * All rights reserved. * 7 | * * 8 | * Redistribution and use in source and binary forms, with or without modification, are permitted provided that the * 9 | * following conditions are met: * 10 | * * 11 | * * Redistributions of source code must retain the above copyright notice, this list of conditions, and the * 12 | * following disclaimer. * 13 | * * 14 | * * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the * 15 | * following disclaimer in the documentation and/or other materials provided with the distribution. * 16 | * * 17 | * * Neither the name of the author nor the names of any contributors may be used to endorse or promote products * 18 | * derived from this software without specific prior written permission. * 19 | * * 20 | * THIS SOFTWARE IS PROVIDED BY THE AUTHORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * 21 | * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL * 22 | * THE AUTHORS BE HELD LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES * 23 | * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR * 24 | * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * 25 | * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * 26 | * POSSIBILITY OF SUCH DAMAGE. * 27 | * * 28 | ***********************************************************************************************************************/ 29 | 30 | /** 31 | @file 32 | @author Andrew D. Zonenberg 33 | @brief Implementation of STM32QSPIStorageBank 34 | */ 35 | #include 36 | #include 37 | #include 38 | #include 39 | #include "STM32QSPIStorageBank.h" 40 | 41 | #include 42 | extern Logger g_log; 43 | 44 | #ifdef HAVE_QUADSPI 45 | 46 | bool STM32QSPIStorageBank::Erase() 47 | { 48 | m_qspi.EraseSector(m_baseAddress); 49 | return true; 50 | } 51 | 52 | bool STM32QSPIStorageBank::Write(uint32_t offset, const uint8_t* data, uint32_t len) 53 | { 54 | m_qspi.Write(GetBase() + offset, data, len); 55 | return true; 56 | } 57 | 58 | //TODO: use CRC hard IP to speed this up!! 59 | uint32_t STM32QSPIStorageBank::CRC(const uint8_t* ptr, uint32_t size) 60 | { 61 | uint32_t poly = 0xedb88320; 62 | 63 | uint32_t crc = 0xffffffff; 64 | for(size_t n=0; n < size; n++) 65 | { 66 | uint8_t d = ptr[n]; 67 | for(int i=0; i<8; i++) 68 | { 69 | bool b = ( crc ^ (d >> i) ) & 1; 70 | crc >>= 1; 71 | if(b) 72 | crc ^= poly; 73 | } 74 | } 75 | 76 | return ~( ((crc & 0x000000ff) << 24) | 77 | ((crc & 0x0000ff00) << 8) | 78 | ((crc & 0x00ff0000) >> 8) | 79 | (crc >> 24) ); 80 | } 81 | 82 | #endif 83 | -------------------------------------------------------------------------------- /driver/STM32QSPIStorageBank.h: -------------------------------------------------------------------------------- 1 | /*********************************************************************************************************************** 2 | * * 3 | * microkvs * 4 | * * 5 | * Copyright (c) 2021-2025 Andrew D. Zonenberg and contributors * 6 | * All rights reserved. * 7 | * * 8 | * Redistribution and use in source and binary forms, with or without modification, are permitted provided that the * 9 | * following conditions are met: * 10 | * * 11 | * * Redistributions of source code must retain the above copyright notice, this list of conditions, and the * 12 | * following disclaimer. * 13 | * * 14 | * * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the * 15 | * following disclaimer in the documentation and/or other materials provided with the distribution. * 16 | * * 17 | * * Neither the name of the author nor the names of any contributors may be used to endorse or promote products * 18 | * derived from this software without specific prior written permission. * 19 | * * 20 | * THIS SOFTWARE IS PROVIDED BY THE AUTHORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * 21 | * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL * 22 | * THE AUTHORS BE HELD LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES * 23 | * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR * 24 | * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * 25 | * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * 26 | * POSSIBILITY OF SUCH DAMAGE. * 27 | * * 28 | ***********************************************************************************************************************/ 29 | 30 | /** 31 | @file 32 | @author Andrew D. Zonenberg 33 | @brief Declaration of STM32QSPIStorageBank 34 | */ 35 | 36 | #ifndef STM32QSPIStorageBank_h 37 | #define STM32QSPIStorageBank_h 38 | 39 | #include 40 | #include 41 | #include 42 | #include "StorageBank.h" 43 | 44 | #ifdef HAVE_QUADSPI 45 | 46 | /** 47 | @brief A StorageBank backed by STM32QSPI flash 48 | */ 49 | class STM32QSPIStorageBank : public StorageBank 50 | { 51 | public: 52 | STM32QSPIStorageBank(QuadSPI_SpiFlashInterface& qspi, uint8_t* ptr, uint32_t size) 53 | : StorageBank(ptr, size) 54 | , m_qspi(qspi) 55 | {} 56 | 57 | virtual bool Erase(); 58 | virtual bool Write(uint32_t offset, const uint8_t* data, uint32_t len); 59 | virtual uint32_t CRC(const uint8_t* ptr, uint32_t size); 60 | 61 | protected: 62 | QuadSPI_SpiFlashInterface& m_qspi; 63 | }; 64 | 65 | #endif 66 | 67 | #endif 68 | -------------------------------------------------------------------------------- /driver/STM32StorageBank.cpp: -------------------------------------------------------------------------------- 1 | /*********************************************************************************************************************** 2 | * * 3 | * microkvs * 4 | * * 5 | * Copyright (c) 2021-2024 Andrew D. Zonenberg and contributors * 6 | * All rights reserved. * 7 | * * 8 | * Redistribution and use in source and binary forms, with or without modification, are permitted provided that the * 9 | * following conditions are met: * 10 | * * 11 | * * Redistributions of source code must retain the above copyright notice, this list of conditions, and the * 12 | * following disclaimer. * 13 | * * 14 | * * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the * 15 | * following disclaimer in the documentation and/or other materials provided with the distribution. * 16 | * * 17 | * * Neither the name of the author nor the names of any contributors may be used to endorse or promote products * 18 | * derived from this software without specific prior written permission. * 19 | * * 20 | * THIS SOFTWARE IS PROVIDED BY THE AUTHORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * 21 | * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL * 22 | * THE AUTHORS BE HELD LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES * 23 | * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR * 24 | * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * 25 | * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * 26 | * POSSIBILITY OF SUCH DAMAGE. * 27 | * * 28 | ***********************************************************************************************************************/ 29 | 30 | /** 31 | @file 32 | @author Andrew D. Zonenberg 33 | @brief Implementation of STM32StorageBank 34 | */ 35 | #include 36 | #include 37 | #include 38 | #include 39 | #include "STM32StorageBank.h" 40 | 41 | #include 42 | extern Logger g_log; 43 | 44 | bool STM32StorageBank::Erase() 45 | { 46 | bool err = Flash::BlockErase(m_baseAddress); 47 | #ifdef HAVE_FLASH_ECC 48 | Flash::ClearECCFaults(); 49 | #endif 50 | return err; 51 | } 52 | 53 | bool STM32StorageBank::Write(uint32_t offset, const uint8_t* data, uint32_t len) 54 | { 55 | bool err = Flash::Write(GetBase() + offset, data, len); 56 | #ifdef HAVE_FLASH_ECC 57 | Flash::ClearECCFaults(); 58 | #endif 59 | return err; 60 | } 61 | 62 | //TODO: use CRC hard IP to speed this up!! 63 | uint32_t STM32StorageBank::CRC(const uint8_t* ptr, uint32_t size) 64 | { 65 | uint32_t poly = 0xedb88320; 66 | 67 | uint32_t crc = 0xffffffff; 68 | for(size_t n=0; n < size; n++) 69 | { 70 | uint8_t d = ptr[n]; 71 | for(int i=0; i<8; i++) 72 | { 73 | bool b = ( crc ^ (d >> i) ) & 1; 74 | crc >>= 1; 75 | if(b) 76 | crc ^= poly; 77 | } 78 | } 79 | 80 | return ~( ((crc & 0x000000ff) << 24) | 81 | ((crc & 0x0000ff00) << 8) | 82 | ((crc & 0x00ff0000) >> 8) | 83 | (crc >> 24) ); 84 | } 85 | -------------------------------------------------------------------------------- /driver/STM32StorageBank.h: -------------------------------------------------------------------------------- 1 | /*********************************************************************************************************************** 2 | * * 3 | * microkvs * 4 | * * 5 | * Copyright (c) 2021-2025 Andrew D. Zonenberg and contributors * 6 | * All rights reserved. * 7 | * * 8 | * Redistribution and use in source and binary forms, with or without modification, are permitted provided that the * 9 | * following conditions are met: * 10 | * * 11 | * * Redistributions of source code must retain the above copyright notice, this list of conditions, and the * 12 | * following disclaimer. * 13 | * * 14 | * * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the * 15 | * following disclaimer in the documentation and/or other materials provided with the distribution. * 16 | * * 17 | * * Neither the name of the author nor the names of any contributors may be used to endorse or promote products * 18 | * derived from this software without specific prior written permission. * 19 | * * 20 | * THIS SOFTWARE IS PROVIDED BY THE AUTHORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * 21 | * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL * 22 | * THE AUTHORS BE HELD LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES * 23 | * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR * 24 | * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * 25 | * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * 26 | * POSSIBILITY OF SUCH DAMAGE. * 27 | * * 28 | ***********************************************************************************************************************/ 29 | 30 | /** 31 | @file 32 | @author Andrew D. Zonenberg 33 | @brief Declaration of STM32StorageBank 34 | */ 35 | 36 | #ifndef STM32StorageBank_h 37 | #define STM32StorageBank_h 38 | 39 | #include 40 | #include "StorageBank.h" 41 | 42 | /** 43 | @brief A StorageBank backed by STM32 flash 44 | */ 45 | class STM32StorageBank : public StorageBank 46 | { 47 | public: 48 | STM32StorageBank(uint8_t* ptr, uint32_t size) 49 | : StorageBank(ptr, size) 50 | {} 51 | 52 | virtual bool Erase(); 53 | virtual bool Write(uint32_t offset, const uint8_t* data, uint32_t len); 54 | virtual uint32_t CRC(const uint8_t* ptr, uint32_t size); 55 | }; 56 | 57 | #endif 58 | -------------------------------------------------------------------------------- /driver/StorageBank.h: -------------------------------------------------------------------------------- 1 | /*********************************************************************************************************************** 2 | * * 3 | * microkvs v0.1 * 4 | * * 5 | * Copyright (c) 2021 Andrew D. Zonenberg and contributors * 6 | * All rights reserved. * 7 | * * 8 | * Redistribution and use in source and binary forms, with or without modification, are permitted provided that the * 9 | * following conditions are met: * 10 | * * 11 | * * Redistributions of source code must retain the above copyright notice, this list of conditions, and the * 12 | * following disclaimer. * 13 | * * 14 | * * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the * 15 | * following disclaimer in the documentation and/or other materials provided with the distribution. * 16 | * * 17 | * * Neither the name of the author nor the names of any contributors may be used to endorse or promote products * 18 | * derived from this software without specific prior written permission. * 19 | * * 20 | * THIS SOFTWARE IS PROVIDED BY THE AUTHORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * 21 | * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL * 22 | * THE AUTHORS BE HELD LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES * 23 | * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR * 24 | * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * 25 | * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * 26 | * POSSIBILITY OF SUCH DAMAGE. * 27 | * * 28 | ***********************************************************************************************************************/ 29 | 30 | /** 31 | @file 32 | @author Andrew D. Zonenberg 33 | @brief Declaration of StorageBank 34 | */ 35 | 36 | #ifndef StorageBank_h 37 | #define StorageBank_h 38 | 39 | #include "../kvs/BankHeader.h" 40 | #include "../kvs/LogEntry.h" 41 | 42 | /** 43 | @brief A single "bank" of flash storage. 44 | 45 | There is typically a 1:1 mapping from banks to erase blocks, however a bank may span multiple erase blocks. No two 46 | StorageBank objects may occupy the same flash erase block, and no other code or data may occupy an erase block 47 | claimed by a StorageBank or it runs the risk of being unexpectedly erased. 48 | 49 | Requirements for underlying storage: 50 | * Memory mapped for reads 51 | * Block level erase 52 | * Byte level writes 53 | */ 54 | class StorageBank 55 | { 56 | public: 57 | StorageBank(uint8_t* base, uint32_t size) 58 | : m_baseAddress(base) 59 | , m_bankSize(size) 60 | {} 61 | 62 | //Raw block access API (needs to be implemented by derived driver class) 63 | virtual bool Erase() =0; 64 | virtual bool Write(uint32_t offset, const uint8_t* data, uint32_t len) =0; 65 | 66 | //Checksumming of block content (may be HW accelerated) 67 | virtual uint32_t CRC(const uint8_t* ptr, uint32_t size) =0; 68 | 69 | BankHeader* GetHeader() 70 | { return reinterpret_cast(m_baseAddress); } 71 | 72 | uint32_t GetSize() 73 | { return m_bankSize; } 74 | 75 | LogEntry* GetLog() 76 | { return reinterpret_cast(m_baseAddress + sizeof(BankHeader)); } 77 | 78 | uint8_t* GetBase() 79 | { return m_baseAddress; } 80 | 81 | protected: 82 | ///@brief Address of the start of this block 83 | uint8_t* m_baseAddress; 84 | 85 | ///@brief Number of bytes of storage available 86 | uint32_t m_bankSize; 87 | }; 88 | 89 | #endif 90 | -------------------------------------------------------------------------------- /driver/TestStorageBank.cpp: -------------------------------------------------------------------------------- 1 | /*********************************************************************************************************************** 2 | * * 3 | * microkvs v0.1 * 4 | * * 5 | * Copyright (c) 2021-2023 Andrew D. Zonenberg and contributors * 6 | * All rights reserved. * 7 | * * 8 | * Redistribution and use in source and binary forms, with or without modification, are permitted provided that the * 9 | * following conditions are met: * 10 | * * 11 | * * Redistributions of source code must retain the above copyright notice, this list of conditions, and the * 12 | * following disclaimer. * 13 | * * 14 | * * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the * 15 | * following disclaimer in the documentation and/or other materials provided with the distribution. * 16 | * * 17 | * * Neither the name of the author nor the names of any contributors may be used to endorse or promote products * 18 | * derived from this software without specific prior written permission. * 19 | * * 20 | * THIS SOFTWARE IS PROVIDED BY THE AUTHORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * 21 | * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL * 22 | * THE AUTHORS BE HELD LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES * 23 | * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR * 24 | * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * 25 | * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * 26 | * POSSIBILITY OF SUCH DAMAGE. * 27 | * * 28 | ***********************************************************************************************************************/ 29 | 30 | /** 31 | @file 32 | @author Andrew D. Zonenberg 33 | @brief Implementation of TestStorageBank 34 | */ 35 | #include 36 | #include 37 | #include "TestStorageBank.h" 38 | 39 | #ifdef SIMULATION 40 | #include 41 | #include 42 | #endif 43 | 44 | bool TestStorageBank::Erase() 45 | { 46 | memset(m_data, 0xff, sizeof(m_data)); 47 | return true; 48 | } 49 | 50 | bool TestStorageBank::Write(uint32_t offset, const uint8_t* data, uint32_t len) 51 | { 52 | memcpy(m_data+offset, data, len); 53 | return true; 54 | } 55 | 56 | uint32_t TestStorageBank::CRC(const uint8_t* ptr, uint32_t size) 57 | { 58 | uint32_t poly = 0xedb88320; 59 | 60 | uint32_t crc = 0xffffffff; 61 | for(size_t n=0; n < size; n++) 62 | { 63 | uint8_t d = ptr[n]; 64 | for(int i=0; i<8; i++) 65 | { 66 | bool b = ( crc ^ (d >> i) ) & 1; 67 | crc >>= 1; 68 | if(b) 69 | crc ^= poly; 70 | } 71 | } 72 | 73 | return ~( ((crc & 0x000000ff) << 24) | 74 | ((crc & 0x0000ff00) << 8) | 75 | ((crc & 0x00ff0000) >> 8) | 76 | (crc >> 24) ); 77 | } 78 | 79 | #ifdef SIMULATION 80 | 81 | void TestStorageBank::Load(const char* path) 82 | { 83 | //Open file and silently exit if not found 84 | //(this is a normal condition on the first run) 85 | FILE* fp = fopen(path, "rb"); 86 | if(!fp) 87 | return; 88 | 89 | if(TEST_BANK_SIZE != fread(m_data, 1, TEST_BANK_SIZE, fp)) 90 | perror("fail to read KVS data"); 91 | fclose(fp); 92 | } 93 | 94 | void TestStorageBank::Serialize(const char* path) 95 | { 96 | FILE* fp = fopen(path, "wb"); 97 | if(!fp) 98 | { 99 | perror("fail to open KVS file for writing"); 100 | return; 101 | } 102 | if(TEST_BANK_SIZE != fwrite(m_data, 1, TEST_BANK_SIZE, fp)) 103 | perror("fail to write KVS data"); 104 | fclose(fp); 105 | } 106 | 107 | #endif 108 | -------------------------------------------------------------------------------- /driver/TestStorageBank.h: -------------------------------------------------------------------------------- 1 | /*********************************************************************************************************************** 2 | * * 3 | * microkvs v0.1 * 4 | * * 5 | * Copyright (c) 2021-2023 Andrew D. Zonenberg and contributors * 6 | * All rights reserved. * 7 | * * 8 | * Redistribution and use in source and binary forms, with or without modification, are permitted provided that the * 9 | * following conditions are met: * 10 | * * 11 | * * Redistributions of source code must retain the above copyright notice, this list of conditions, and the * 12 | * following disclaimer. * 13 | * * 14 | * * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the * 15 | * following disclaimer in the documentation and/or other materials provided with the distribution. * 16 | * * 17 | * * Neither the name of the author nor the names of any contributors may be used to endorse or promote products * 18 | * derived from this software without specific prior written permission. * 19 | * * 20 | * THIS SOFTWARE IS PROVIDED BY THE AUTHORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * 21 | * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL * 22 | * THE AUTHORS BE HELD LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES * 23 | * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR * 24 | * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * 25 | * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * 26 | * POSSIBILITY OF SUCH DAMAGE. * 27 | * * 28 | ***********************************************************************************************************************/ 29 | 30 | /** 31 | @file 32 | @author Andrew D. Zonenberg 33 | @brief Declaration of TestStorageBank 34 | */ 35 | 36 | #ifndef TestStorageBank_h 37 | #define TestStorageBank_h 38 | 39 | #ifndef TEST_BANK_SIZE 40 | #define TEST_BANK_SIZE 32768 41 | #endif 42 | 43 | #include 44 | #include "StorageBank.h" 45 | 46 | /** 47 | @brief A simulated StorageBank backed by RAM 48 | */ 49 | class TestStorageBank : public StorageBank 50 | { 51 | public: 52 | TestStorageBank() 53 | : StorageBank(m_data, TEST_BANK_SIZE) 54 | { 55 | memset(m_data, 0xff, sizeof(m_data)); 56 | } 57 | 58 | virtual bool Erase(); 59 | virtual bool Write(uint32_t offset, const uint8_t* data, uint32_t len); 60 | virtual uint32_t CRC(const uint8_t* ptr, uint32_t size); 61 | 62 | #ifdef SIMULATION 63 | void Load(const char* path); 64 | void Serialize(const char* path); 65 | #endif 66 | 67 | protected: 68 | uint8_t m_data[TEST_BANK_SIZE]; 69 | }; 70 | 71 | #endif 72 | -------------------------------------------------------------------------------- /kvs/BankHeader.h: -------------------------------------------------------------------------------- 1 | /*********************************************************************************************************************** 2 | * * 3 | * microkvs v0.1 * 4 | * * 5 | * Copyright (c) 2021-2022 Andrew D. Zonenberg and contributors * 6 | * All rights reserved. * 7 | * * 8 | * Redistribution and use in source and binary forms, with or without modification, are permitted provided that the * 9 | * following conditions are met: * 10 | * * 11 | * * Redistributions of source code must retain the above copyright notice, this list of conditions, and the * 12 | * following disclaimer. * 13 | * * 14 | * * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the * 15 | * following disclaimer in the documentation and/or other materials provided with the distribution. * 16 | * * 17 | * * Neither the name of the author nor the names of any contributors may be used to endorse or promote products * 18 | * derived from this software without specific prior written permission. * 19 | * * 20 | * THIS SOFTWARE IS PROVIDED BY THE AUTHORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * 21 | * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL * 22 | * THE AUTHORS BE HELD LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES * 23 | * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR * 24 | * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * 25 | * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * 26 | * POSSIBILITY OF SUCH DAMAGE. * 27 | * * 28 | ***********************************************************************************************************************/ 29 | 30 | /** 31 | @file 32 | @author Andrew D. Zonenberg 33 | @brief Declaration of BankHeader 34 | */ 35 | 36 | #ifndef BankHeader_h 37 | #define BankHeader_h 38 | 39 | /** 40 | @brief Header for the flash bank 41 | */ 42 | class BankHeader 43 | { 44 | public: 45 | uint32_t m_magic; 46 | uint32_t m_version; 47 | uint32_t m_logSize; 48 | 49 | //pad to write block size 50 | #ifdef MICROKVS_WRITE_BLOCK_SIZE 51 | uint8_t m_padding[MICROKVS_WRITE_BLOCK_SIZE - (12 % MICROKVS_WRITE_BLOCK_SIZE)]; 52 | #endif 53 | }; 54 | 55 | #endif 56 | -------------------------------------------------------------------------------- /kvs/KVS.cpp: -------------------------------------------------------------------------------- 1 | /*********************************************************************************************************************** 2 | * * 3 | * microkvs * 4 | * * 5 | * Copyright (c) 2021-2024 Andrew D. Zonenberg and contributors * 6 | * All rights reserved. * 7 | * * 8 | * Redistribution and use in source and binary forms, with or without modification, are permitted provided that the * 9 | * following conditions are met: * 10 | * * 11 | * * Redistributions of source code must retain the above copyright notice, this list of conditions, and the * 12 | * following disclaimer. * 13 | * * 14 | * * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the * 15 | * following disclaimer in the documentation and/or other materials provided with the distribution. * 16 | * * 17 | * * Neither the name of the author nor the names of any contributors may be used to endorse or promote products * 18 | * derived from this software without specific prior written permission. * 19 | * * 20 | * THIS SOFTWARE IS PROVIDED BY THE AUTHORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * 21 | * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL * 22 | * THE AUTHORS BE HELD LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES * 23 | * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR * 24 | * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * 25 | * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * 26 | * POSSIBILITY OF SUCH DAMAGE. * 27 | * * 28 | ***********************************************************************************************************************/ 29 | 30 | /** 31 | @file 32 | @author Andrew D. Zonenberg 33 | @brief Implementation of KVS 34 | */ 35 | #include "KVS.h" 36 | #include 37 | #include 38 | #include 39 | 40 | #include 41 | 42 | extern Logger g_log; 43 | 44 | #define HEADER_MAGIC 0xc0def00d 45 | 46 | char g_blankKey[KVS_NAMELEN]; 47 | 48 | //Instantiate common KVS overrides so they don't get inlined 49 | template uint8_t KVS::ReadObject(const char* name, uint8_t defaultValue); 50 | template uint16_t KVS::ReadObject(const char* name, uint16_t defaultValue); 51 | template bool KVS::ReadObject(const char* name, bool defaultValue); 52 | template uint16_t KVS::ReadObject(uint16_t defaultValue, const char* format, ...); 53 | 54 | template bool KVS::StoreObjectIfNecessary(const char* name, bool currentValue, bool defaultValue); 55 | template bool KVS::StoreObjectIfNecessary(const char* name, uint8_t currentValue, uint8_t defaultValue); 56 | template bool KVS::StoreObjectIfNecessary(const char* name, uint16_t currentValue, uint16_t defaultValue); 57 | 58 | template bool KVS::StoreObjectIfNecessary(uint16_t currentValue, uint16_t defaultValue, const char* format, ...); 59 | 60 | #ifdef STM32L031 61 | #define BLANK_FLASH_BYTE 0x00 62 | #define BLANK_FLASH_X32 0x00000000 63 | #else 64 | #define BLANK_FLASH_BYTE 0xff 65 | #define BLANK_FLASH_X32 0xffffffff 66 | #endif 67 | 68 | //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// 69 | // Construction / destruction 70 | 71 | /** 72 | @brief Creates a new KVS 73 | 74 | @param left One of two flash blocks, arbitrarily named "left" 75 | @param right One of two flash blocks, arbitrarily named "right" 76 | @param defaultLogSize Number of log entries to use when creating a new block header 77 | */ 78 | KVS::KVS(StorageBank* left, StorageBank* right, uint32_t defaultLogSize) 79 | : m_left(left) 80 | , m_right(right) 81 | , m_active(nullptr) 82 | , m_defaultLogSize(defaultLogSize) 83 | , m_firstFreeLogEntry(0) 84 | , m_firstFreeData(0) 85 | , m_eccFault(false) 86 | { 87 | memset(g_blankKey, BLANK_FLASH_BYTE, KVS_NAMELEN); 88 | 89 | FindCurrentBank(); 90 | ScanCurrentBank(); 91 | } 92 | 93 | //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// 94 | // Reading 95 | 96 | /** 97 | @brief Scan the current bank looking for free space 98 | */ 99 | void KVS::ScanCurrentBank() 100 | { 101 | //Find free space 102 | //Scan the entire log beginning to end to account for used space 103 | //(This is needed so that we can properly ignore corrupted entries) 104 | auto log = m_active->GetLog(); 105 | auto logsize = m_active->GetHeader()->m_logSize; 106 | m_firstFreeLogEntry = logsize-1; 107 | LogEntry* lastlog = nullptr; 108 | for(int64_t i = 0; i= GetBlockSize() ) 123 | continue; 124 | 125 | //If it's good, save the pointer 126 | if(!m_eccFault) 127 | lastlog = &log[i]; 128 | } 129 | 130 | //It's blank, mark it as available 131 | else 132 | { 133 | m_firstFreeLogEntry = i; 134 | break; 135 | } 136 | } 137 | 138 | //This entry is corrupted, and thus not available 139 | if(m_eccFault) 140 | { 141 | g_log(Logger::WARNING, "KVS::ScanCurrentBank: uncorrectable ECC error at address 0x%08x (pc=%08x)\n", 142 | m_eccFaultAddr, m_eccFaultPC); 143 | continue; 144 | } 145 | } 146 | 147 | //If nothing in the log, free data area starts right after the log area 148 | if(!lastlog) 149 | m_firstFreeData = sizeof(BankHeader) + logsize*sizeof(LogEntry); 150 | 151 | //If we have at least one log entry in the store, free data starts after the last log entry ends 152 | else 153 | m_firstFreeData = lastlog->m_start + lastlog->m_len; 154 | 155 | m_firstFreeData = RoundUpToWriteBlockSize(m_firstFreeData); 156 | } 157 | 158 | /** 159 | @brief Determine which bank is active, and set m_active appropriately 160 | */ 161 | void KVS::FindCurrentBank() 162 | { 163 | auto lh = m_left->GetHeader(); 164 | auto rh = m_right->GetHeader(); 165 | 166 | bool leftValid = false; 167 | bool rightValid = false; 168 | 169 | unsafe 170 | { 171 | m_eccFault = false; 172 | 173 | //See which block(s) have valid headers 174 | //Header magic number must be valid, but log size (last field written) must also be sane 175 | //(if we interrupt midway through a compact operation we might not have the full block header written) 176 | //Assume any log size >2GB is invalid since we're running on tiny MCUs 177 | leftValid = (lh->m_magic == HEADER_MAGIC); 178 | if(lh->m_logSize > 0x80000000) 179 | leftValid = false; 180 | if(m_eccFault) 181 | { 182 | leftValid = false; 183 | 184 | g_log(Logger::WARNING, "KVS::FindCurrentBank: uncorrectable ECC error at address 0x%08x (pc=%08x)\n", 185 | m_eccFaultAddr, m_eccFaultPC); 186 | m_eccFault = false; 187 | } 188 | 189 | rightValid = (rh->m_magic == HEADER_MAGIC); 190 | if(rh->m_logSize > 0x80000000) 191 | rightValid = false; 192 | if(m_eccFault) 193 | { 194 | rightValid = false; 195 | 196 | g_log(Logger::WARNING, "KVS::FindCurrentBank: uncorrectable ECC error at address 0x%08x (pc=%08x)\n", 197 | m_eccFaultAddr, m_eccFaultPC); 198 | m_eccFault = false; 199 | } 200 | } 201 | 202 | //If NEITHER bank is valid, we have a blank chip. 203 | //Initialize and declare the left one active. 204 | if(!leftValid && !rightValid) 205 | { 206 | InitializeBank(m_left); 207 | m_active = m_left; 208 | } 209 | 210 | //If only one one bank is active, mark that one as active 211 | else if(leftValid && !rightValid) 212 | m_active = m_left; 213 | else if(!leftValid && rightValid) 214 | m_active = m_right; 215 | 216 | //If BOTH banks are active, the higher version number is our active bank 217 | //(as long as that version number isn't invalid) 218 | else if( (lh->m_version > rh->m_version) && (lh->m_version != BLANK_FLASH_X32) ) 219 | m_active = m_left; 220 | else 221 | m_active = m_right; 222 | } 223 | 224 | /** 225 | @brief Find the latest version of an object in the active bank, if present. 226 | 227 | Returns NULL if no object by that name exists. 228 | */ 229 | LogEntry* KVS::FindObject(const char* name) 230 | { 231 | m_eccFault = false; 232 | 233 | //Actual lookup key: zero padded if too short, but not guaranteed to be null terminated 234 | char key[KVS_NAMELEN] = {0}; 235 | #pragma GCC diagnostic push 236 | #pragma GCC diagnostic ignored "-Wstringop-truncation" 237 | strncpy(key, name, KVS_NAMELEN); 238 | #pragma GCC diagnostic pop 239 | 240 | LogEntry* log = nullptr; 241 | 242 | //Start searching the log 243 | auto len = m_active->GetHeader()->m_logSize; 244 | auto base = m_active->GetLog(); 245 | for(uint32_t i=0; iCRC(m_active->GetBase() + base[i].m_start, base[i].m_len) == base[i].m_crc); 266 | } 267 | 268 | //If ECC fault, this entry is invalid 269 | if(m_eccFault) 270 | { 271 | m_eccFault = false; 272 | g_log(Logger::WARNING, "KVS::FindObject: uncorrectable ECC error at address 0x%08x (pc=%08x)\n", 273 | m_eccFaultAddr, m_eccFaultPC); 274 | continue; 275 | } 276 | 277 | //If CRC match, this is the latest log entry 278 | if(crcok) 279 | log = &base[i]; 280 | 281 | //If CRC mismatch, entry is corrupted - fall back to the previous entry 282 | } 283 | 284 | //If the log entry has no data, return null 285 | if(log && (log->m_len == 0)) 286 | return nullptr; 287 | 288 | return log; 289 | } 290 | 291 | /** 292 | @brief Calculates the expected CRC of a log entry 293 | */ 294 | uint32_t KVS::HeaderCRC(const LogEntry* log) 295 | { 296 | return m_active->CRC((const uint8_t*)log, KVS_NAMELEN + 2*sizeof(uint32_t)); 297 | } 298 | 299 | /** 300 | @brief Returns a pointer to the object described by a log entry 301 | */ 302 | uint8_t* KVS::MapObject(LogEntry* log) 303 | { 304 | return m_active->GetBase() + log->m_start; 305 | } 306 | 307 | /** 308 | @brief Reads an object into a provided buffer. 309 | 310 | If the object is more than len bytes in size, the readback is truncated but no error is returned. 311 | 312 | @param name Name of the object to read 313 | @param data Output buffer 314 | @param len Size of the output buffer 315 | */ 316 | bool KVS::ReadObject(const char* name, uint8_t* data, uint32_t len) 317 | { 318 | auto log = FindObject(name); 319 | if(!log) 320 | return false; 321 | 322 | uint32_t readlen = log->m_len; 323 | if(readlen > len) 324 | readlen = len; 325 | 326 | memcpy(data, MapObject(log), readlen); 327 | return true; 328 | } 329 | 330 | //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// 331 | // Writing 332 | 333 | /** 334 | @brief Initializes a blank bank with a header 335 | */ 336 | bool KVS::InitializeBank(StorageBank* bank) 337 | { 338 | //Erase the bank just to be safe 339 | if(!bank->Erase()) 340 | return false; 341 | 342 | //Write the content of the new block header 343 | BankHeader header; 344 | memset(&header, 0, sizeof(header)); 345 | header.m_magic = HEADER_MAGIC; 346 | header.m_version = 0; 347 | header.m_logSize = m_defaultLogSize; 348 | if(bank->Write(0, (uint8_t*)&header, sizeof(header))) 349 | return false; 350 | 351 | return true; 352 | } 353 | 354 | /** 355 | @brief Writes a new object to the store. 356 | 357 | Any existing object by the same name is overwritten. 358 | 359 | This function will try up to five times to store the object, returning an error only if all attempts are 360 | unsuccessful. 361 | 362 | This is a workaround for (among others) STM32L431 errata 2.2.10. 363 | 364 | @param name Name of the object. 365 | Names must be exactly KVS_NAMELEN bytes in size. 366 | Shorter names are padded to KVS_NAMELEN with 0x00 bytes. 367 | Longer names are truncated. 368 | The names 0x00..00 0xFF...FF are reserved and may not be used for an object. 369 | @param data Object content 370 | @param len Length of the object 371 | */ 372 | bool KVS::StoreObject(const char* name, const uint8_t* data, uint32_t len) 373 | { 374 | for(int i=0; i<5; i++) 375 | { 376 | if(StoreObjectInternal(name, data, len)) 377 | return true; 378 | } 379 | return false; 380 | } 381 | 382 | /** 383 | @brief Core of StoreObject 384 | */ 385 | bool KVS::StoreObjectInternal(const char* name, const uint8_t* data, uint32_t len) 386 | { 387 | m_eccFault = false; 388 | 389 | //Actual lookup key: zero padded if too short, but not guaranteed to be null terminated 390 | char key[KVS_NAMELEN] = {0}; 391 | #pragma GCC diagnostic push 392 | #pragma GCC diagnostic ignored "-Wstringop-truncation" 393 | strncpy(key, name, KVS_NAMELEN); 394 | #pragma GCC diagnostic pop 395 | 396 | //If there's not enough space for the file, compact the store to make more room 397 | if(GetFreeDataSpace() < len) 398 | { 399 | if(!Compact()) 400 | return false; 401 | } 402 | 403 | //If not enough space after compaction, we're out of flash. Give up. 404 | if(GetFreeDataSpace() < len) 405 | return false; 406 | 407 | //Same thing, but make sure there's header space 408 | if(GetFreeLogEntries() < 1) 409 | Compact(); 410 | if(GetFreeLogEntries() < 1) 411 | return false; 412 | 413 | //Calculate expected data CRC 414 | auto dataCRC = m_active->CRC(data, len); 415 | 416 | //Calculate expected header CRC 417 | LogEntry tempHeader; 418 | memset(&tempHeader, 0, sizeof(tempHeader)); 419 | memcpy(tempHeader.m_key, key, KVS_NAMELEN); 420 | tempHeader.m_start = m_firstFreeData; 421 | tempHeader.m_len = len; 422 | tempHeader.m_crc = dataCRC; 423 | tempHeader.m_headerCRC = 0; 424 | auto headerCRC = HeaderCRC(&tempHeader); 425 | 426 | unsafe 427 | { 428 | //Write header data to reserve the log entry 429 | uint32_t logoff = sizeof(BankHeader) + m_firstFreeLogEntry*sizeof(LogEntry); 430 | uint32_t header[4] = { m_firstFreeData, len, dataCRC, headerCRC}; 431 | m_firstFreeLogEntry ++; 432 | if(!m_active->Write(logoff + KVS_NAMELEN, reinterpret_cast(&header[0]), sizeof(header))) 433 | return false; 434 | 435 | //Write and verify object content 436 | //(skip this if there's no data, empty objects are allowed and treated as nonexistent) 437 | if(len != 0) 438 | { 439 | auto offset = m_firstFreeData; 440 | 441 | //Blank check the region as a sanity check 442 | auto base = m_active->GetBase(); 443 | while(true) 444 | { 445 | bool blank = true; 446 | for(uint32_t i=0; iWrite(offset, data, len)) 475 | return false; 476 | if(memcmp(data, base + offset, len) != 0) 477 | return false; 478 | } 479 | 480 | //Write and verify object name 481 | if(!m_active->Write(logoff, reinterpret_cast(key), KVS_NAMELEN)) 482 | return false; 483 | if(memcmp(key, m_active->GetBase() + logoff, KVS_NAMELEN) != 0) 484 | return false; 485 | } 486 | 487 | //All good! 488 | return true; 489 | } 490 | 491 | /** 492 | @brief Writes a value to the KVS if necessary. 493 | 494 | If the value is the same as the previous value in the KVS, or there is no previous value 495 | but the value being written is the same as the default value, no data is written. 496 | */ 497 | bool KVS::StoreStringObjectIfNecessary(const char* name, const char* currentValue, const char* defaultValue) 498 | { 499 | auto valueLen = strlen(currentValue); 500 | 501 | //Check if we already have the same string stored, early out if so 502 | auto hobject = FindObject(name); 503 | if(hobject) 504 | { 505 | auto oldval = (const char*)MapObject(hobject); 506 | if( (valueLen == hobject->m_len) && (!strncmp(currentValue, oldval, hobject->m_len)) ) 507 | return true; 508 | } 509 | 510 | //No existing object. Before we store the new one, check if it's the default and skip the store if so 511 | else if(!strcmp(currentValue, defaultValue)) 512 | return true; 513 | 514 | //Need to store the value (different from default and/or existing value 515 | return StoreObject(name, (uint8_t*)currentValue, valueLen); 516 | } 517 | 518 | //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// 519 | // Compaction 520 | 521 | /** 522 | @brief Moves all active objects to the inactive bank, reclaiming free space in the process 523 | */ 524 | bool KVS::Compact() 525 | { 526 | const uint32_t cachesize = 16; 527 | char cache[cachesize][KVS_NAMELEN]; 528 | memset(cache, BLANK_FLASH_BYTE, sizeof(cache)); 529 | uint32_t nextCache = 0; 530 | 531 | //Find the INACTIVE storage bank 532 | StorageBank* inactive = nullptr; 533 | if(m_active == m_left) 534 | inactive = m_right; 535 | else 536 | inactive = m_left; 537 | 538 | auto base = m_active->GetBase(); 539 | auto log = m_active->GetLog(); 540 | auto outlog = inactive->GetLog(); 541 | uint32_t nextLog = 0; 542 | uint32_t nextData = RoundUpToWriteBlockSize(sizeof(BankHeader) + m_defaultLogSize*sizeof(LogEntry)); 543 | 544 | //Erase the inactive bank and give it a header, but do NOT write the version number yet. 545 | //If we're interrupted during the compaction, we want the block to read as invalid. 546 | if(!inactive->Erase()) 547 | return false; 548 | 549 | //Loop over the log and copy objects one by one 550 | for(int64_t i = static_cast(m_firstFreeLogEntry)-1; i>=0; i--) 551 | { 552 | //See if this item is in the cache. 553 | //If so, it was already copied so no need to do a full search of the log 554 | bool found = false; 555 | for(uint32_t j=0; jCRC(base + log[i].m_start, log[i].m_len) != log[i].m_crc) 604 | continue; 605 | } 606 | 607 | //If ECC fault, this entry is invalid 608 | //If ECC fault, this entry is invalid 609 | if(m_eccFault) 610 | { 611 | m_eccFault = false; 612 | g_log(Logger::WARNING, "KVS::Compact: uncorrectable ECC error at address 0x%08x (pc=%08x)\n", 613 | m_eccFaultAddr, m_eccFaultPC); 614 | 615 | continue; 616 | } 617 | 618 | //Not found. This is the most up to date version. 619 | //Only write it if there's valid data (empty objects get removed during the compaction step) 620 | if(log[i].m_len != 0) 621 | { 622 | //Copy the data first, then the log 623 | if(!inactive->Write(nextData, base + log[i].m_start, log[i].m_len)) 624 | return false; 625 | 626 | LogEntry entry = log[i]; 627 | entry.m_start = nextData; 628 | entry.m_headerCRC = HeaderCRC(&entry); 629 | if(!inactive->Write(sizeof(BankHeader) + nextLog*sizeof(LogEntry), (uint8_t*)&entry, sizeof(entry))) 630 | return false; 631 | 632 | //Update pointers for next output 633 | nextData = RoundUpToWriteBlockSize(nextData + log[i].m_len); 634 | nextLog ++; 635 | } 636 | 637 | //Add this entry to the cache of recently copied stuff 638 | memcpy(cache[nextCache], log[i].m_key, KVS_NAMELEN); 639 | nextCache = (nextCache + 1) % cachesize; 640 | } 641 | 642 | //Write block header with the new version number 643 | //Need to write the entire bank header in one go, since our flash write block size may be >4 bytes! 644 | BankHeader header; 645 | memset(&header, 0, sizeof(header)); 646 | header.m_magic = HEADER_MAGIC; 647 | header.m_version = m_active->GetHeader()->m_version + 1; 648 | header.m_logSize = m_defaultLogSize; 649 | if(!inactive->Write(0, (uint8_t*)&header, sizeof(header))) 650 | return false; 651 | 652 | //Done, switch banks 653 | m_active = inactive; 654 | m_firstFreeLogEntry = nextLog; 655 | m_firstFreeData = nextData; 656 | 657 | //Round free data pointer to start of next write block 658 | #ifdef MICROKVS_WRITE_BLOCK_SIZE 659 | m_firstFreeData += (MICROKVS_WRITE_BLOCK_SIZE - (m_firstFreeData % MICROKVS_WRITE_BLOCK_SIZE)); 660 | #endif 661 | 662 | return true; 663 | } 664 | 665 | //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// 666 | // Zeroization 667 | 668 | /** 669 | @brief Destroys all data in the INACTIVE bank. 670 | 671 | Performing a compaction followed by zeroizing the inactive bank does not affect the CURRENT contents of any 672 | objects, but ensures that PREVIOUS content of all objects is destroyed. 673 | */ 674 | void KVS::WipeInactive() 675 | { 676 | if(m_active == m_left) 677 | m_right->Erase(); 678 | else 679 | m_left->Erase(); 680 | } 681 | 682 | /** 683 | @brief Destroys the entire contents of the KVS, including both the active and inactive blocks. 684 | 685 | This would typically be done as part of a factory reset or to purge sensitive data such as keys prior to 686 | decommissioning a piece of equipment. 687 | */ 688 | void KVS::WipeAll() 689 | { 690 | m_left->Erase(); 691 | m_right->Erase(); 692 | } 693 | 694 | //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// 695 | // Enumeration 696 | 697 | /** 698 | @brief Enumerates all objects in the KVS. 699 | 700 | @param list Result buffer containing at least "size" entries 701 | @param size Number of entries in "list" 702 | 703 | If the list is too small to contain all objects, the first "size" entries are written to "list" and the function 704 | returns "size". 705 | 706 | @return Number of objects written to "list" 707 | */ 708 | uint32_t KVS::EnumObjects(KVSListEntry* list, uint32_t size) 709 | { 710 | m_eccFault = false; 711 | 712 | uint32_t ret = 0; 713 | 714 | //Start searching the log 715 | auto len = m_active->GetHeader()->m_logSize; 716 | auto base = m_active->GetLog(); 717 | for(uint32_t i=0; iCRC(m_active->GetBase() + base[i].m_start, base[i].m_len) != base[i].m_crc) 732 | continue; 733 | } 734 | 735 | //If ECC fault, this entry is invalid 736 | if(m_eccFault) 737 | { 738 | m_eccFault = false; 739 | g_log(Logger::WARNING, "KVS::EnumObjects: uncorrectable ECC error at address 0x%08x (pc=%08x)\n", 740 | m_eccFaultAddr, m_eccFaultPC); 741 | continue; 742 | } 743 | 744 | //See if this object is already in the output list. 745 | //If so, increment the number of copies and update the size (latest object is current) 746 | bool found = false; 747 | for(uint32_t j=0; j(a); 778 | auto pb = reinterpret_cast(b); 779 | 780 | for(int i=0; ikey[i] > pb->key[i]) 783 | return 1; 784 | if(pa->key[i] < pb->key[i]) 785 | return -1; 786 | } 787 | return 0; 788 | } 789 | -------------------------------------------------------------------------------- /kvs/KVS.h: -------------------------------------------------------------------------------- 1 | /*********************************************************************************************************************** 2 | * * 3 | * microkvs * 4 | * * 5 | * Copyright (c) 2021-2024 Andrew D. Zonenberg and contributors * 6 | * All rights reserved. * 7 | * * 8 | * Redistribution and use in source and binary forms, with or without modification, are permitted provided that the * 9 | * following conditions are met: * 10 | * * 11 | * * Redistributions of source code must retain the above copyright notice, this list of conditions, and the * 12 | * following disclaimer. * 13 | * * 14 | * * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the * 15 | * following disclaimer in the documentation and/or other materials provided with the distribution. * 16 | * * 17 | * * Neither the name of the author nor the names of any contributors may be used to endorse or promote products * 18 | * derived from this software without specific prior written permission. * 19 | * * 20 | * THIS SOFTWARE IS PROVIDED BY THE AUTHORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * 21 | * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL * 22 | * THE AUTHORS BE HELD LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES * 23 | * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR * 24 | * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * 25 | * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * 26 | * POSSIBILITY OF SUCH DAMAGE. * 27 | * * 28 | ***********************************************************************************************************************/ 29 | 30 | /** 31 | @file 32 | @author Andrew D. Zonenberg 33 | @brief Declaration of KVS 34 | */ 35 | 36 | #ifndef KVS_h 37 | #define KVS_h 38 | 39 | #ifndef SIMULATION 40 | #include 41 | #endif 42 | 43 | #include 44 | #include "../driver/StorageBank.h" 45 | #include 46 | 47 | /** 48 | @brief A list entry used for enumerating the content of the KVS 49 | */ 50 | struct KVSListEntry 51 | { 52 | char key[KVS_NAMELEN+1]; //[KVS_NAMELEN] is always null for easy printing 53 | //even if original key is not null terminated 54 | uint32_t size; //Size of the most recent copy of the object 55 | uint32_t revs; //Number of copies (including the current one) stored in the current erase block 56 | }; 57 | 58 | /** 59 | @brief Top level KVS object 60 | */ 61 | class KVS 62 | { 63 | public: 64 | KVS(StorageBank* left, StorageBank* right, uint32_t defaultLogSize); 65 | 66 | /** 67 | @brief Exception handler 68 | 69 | If your MCU throws bus faults, NMIs, or similar when ECC faults occur, some special handling is needed 70 | for resilience to power outages during writes. 71 | 72 | You will need to catch the exception and detect if it is caused by a bad flash access within the KVS region. 73 | If so, call this function with the offending address then return to the instruction after the one which 74 | triggered the fault. 75 | */ 76 | void OnUncorrectableECCFault(uint32_t flashAddr, uint32_t insnAddr) 77 | { 78 | m_eccFault = true; 79 | m_eccFaultAddr = flashAddr; 80 | m_eccFaultPC = insnAddr; 81 | } 82 | 83 | //Main API 84 | LogEntry* FindObject(const char* name); 85 | 86 | /** 87 | @brief Wrapper around FindObject with sprintf-style formatting 88 | */ 89 | LogEntry* FindObjectF(const char* format, ...) 90 | { 91 | char objname[KVS_NAMELEN+1] = {0}; 92 | StringBuffer sbuf(objname, sizeof(objname)); 93 | 94 | __builtin_va_list list; 95 | __builtin_va_start(list, format); 96 | sbuf.Printf(format, list); 97 | __builtin_va_end(list); 98 | 99 | return FindObject(objname); 100 | } 101 | 102 | uint8_t* MapObject(LogEntry* log); 103 | bool ReadObject(const char* name, uint8_t* data, uint32_t len); 104 | 105 | bool StoreObject(const char* name, const uint8_t* data, uint32_t len); 106 | 107 | /** 108 | @brief Wrapper around StoreObject with sprintf-style formatting 109 | */ 110 | bool StoreObject(const uint8_t* data, uint32_t len, const char* format, ...) 111 | { 112 | char objname[KVS_NAMELEN+1] = {0}; 113 | StringBuffer sbuf(objname, sizeof(objname)); 114 | 115 | __builtin_va_list list; 116 | __builtin_va_start(list, format); 117 | sbuf.Printf(format, list); 118 | __builtin_va_end(list); 119 | 120 | return StoreObject(objname, data, len); 121 | } 122 | 123 | //Maintenance operations 124 | bool Compact(); 125 | void WipeInactive(); 126 | void WipeAll(); 127 | 128 | //Enumeration 129 | uint32_t EnumObjects(KVSListEntry* list, uint32_t size); 130 | 131 | /** 132 | @brief Reads a value from the KVS, returning a default value if not found 133 | */ 134 | template 135 | T ReadObject(const char* name, T defaultValue) 136 | { 137 | auto hlog = FindObject(name); 138 | if(hlog) 139 | return *reinterpret_cast(MapObject(hlog)); 140 | else 141 | return defaultValue; 142 | } 143 | 144 | /** 145 | @brief Reads a value from the KVS, returning a default value if not found 146 | */ 147 | template 148 | T ReadObject(T defaultValue, const char* format, ...) 149 | { 150 | char objname[KVS_NAMELEN+1] = {0}; 151 | StringBuffer sbuf(objname, sizeof(objname)); 152 | 153 | __builtin_va_list list; 154 | __builtin_va_start(list, format); 155 | sbuf.Printf(format, list); 156 | __builtin_va_end(list); 157 | 158 | return ReadObject(objname, defaultValue); 159 | } 160 | 161 | bool StoreStringObjectIfNecessary(const char* name, const char* currentValue, const char* defaultValue); 162 | 163 | /** 164 | @brief Writes a value to the KVS if necessary. 165 | 166 | If the value is the same as the previous value in the KVS, or there is no previous value 167 | but the value being written is the same as the default value, no data is written. 168 | */ 169 | template 170 | bool StoreObjectIfNecessary(const char* name, T currentValue, T defaultValue) 171 | { 172 | //See if the value is already there 173 | auto hlog = FindObject(name); 174 | 175 | //If not found: write if non-default 176 | if(!hlog) 177 | { 178 | if(currentValue != defaultValue) 179 | return StoreObject(name, (const uint8_t*)¤tValue, sizeof(currentValue)); 180 | } 181 | 182 | //If found: write if changed 183 | else 184 | { 185 | if(currentValue != *reinterpret_cast(MapObject(hlog))) 186 | return StoreObject(name, (const uint8_t*)¤tValue, sizeof(currentValue)); 187 | } 188 | return true; 189 | } 190 | 191 | /** 192 | @brief Writes a value to the KVS if necessary, using a sprintf'd object name 193 | 194 | If the value is the same as the previous value in the KVS, or there is no previous value 195 | but the value being written is the same as the default value, no data is written. 196 | */ 197 | template 198 | bool StoreObjectIfNecessary(T currentValue, T defaultValue, const char* format, ...) 199 | { 200 | char objname[KVS_NAMELEN+1] = {0}; 201 | StringBuffer sbuf(objname, sizeof(objname)); 202 | 203 | __builtin_va_list list; 204 | __builtin_va_start(list, format); 205 | sbuf.Printf(format, list); 206 | __builtin_va_end(list); 207 | 208 | return StoreObjectIfNecessary(objname, currentValue, defaultValue); 209 | } 210 | 211 | //Accessors 212 | public: 213 | 214 | /** 215 | @brief Returns the number of log entries in the active block available for use 216 | */ 217 | uint32_t GetFreeLogEntries() 218 | { return m_active->GetHeader()->m_logSize - m_firstFreeLogEntry; } 219 | 220 | /** 221 | @brief Returns the number of data bytes in the active block available for use 222 | */ 223 | uint32_t GetFreeDataSpace() 224 | { return m_active->GetSize() - m_firstFreeData; } 225 | 226 | /** 227 | @brief Returns the version of the bank header 228 | */ 229 | uint32_t GetBankHeaderVersion() 230 | { return m_active->GetHeader()->m_version; } 231 | 232 | /** 233 | @brief Returns true if the left bank is active 234 | */ 235 | bool IsLeftBankActive() 236 | { return (m_active == m_left); } 237 | 238 | /** 239 | @brief Returns true if the right bank is active 240 | */ 241 | bool IsRightBankActive() 242 | { return (m_active == m_right); } 243 | 244 | /** 245 | @brief Returns the number of log spaces in the active block, both used and unused 246 | */ 247 | uint32_t GetLogCapacity() 248 | { return m_active->GetHeader()->m_logSize; } 249 | 250 | /** 251 | @brief Returns the total number of bytes in the active block including header, log, and data 252 | */ 253 | uint32_t GetBlockSize() 254 | { return m_active->GetSize(); } 255 | 256 | /** 257 | @brief Returns the total space allocated to data, both used and unused 258 | */ 259 | uint32_t GetDataCapacity() 260 | { return GetBlockSize() - (sizeof(BankHeader) + GetLogCapacity()*sizeof(LogEntry)); } 261 | 262 | ///@brief Rounds a value up to the next multiple of the flash write block size 263 | uint32_t RoundUpToWriteBlockSize(uint32_t val) 264 | { 265 | #ifdef MICROKVS_WRITE_BLOCK_SIZE 266 | val += (MICROKVS_WRITE_BLOCK_SIZE - (val % MICROKVS_WRITE_BLOCK_SIZE)); 267 | #endif 268 | return val; 269 | } 270 | 271 | uint32_t HeaderCRC(const LogEntry* log); 272 | 273 | protected: 274 | bool StoreObjectInternal(const char* name, const uint8_t* data, uint32_t len); 275 | 276 | void FindCurrentBank(); 277 | void ScanCurrentBank(); 278 | 279 | static int ListCompare(const void* a, const void* b); 280 | 281 | bool InitializeBank(StorageBank* bank); 282 | 283 | ///@brief First storage bank ("left") 284 | StorageBank* m_left; 285 | 286 | ///@brief Second storage bank ("right") 287 | StorageBank* m_right; 288 | 289 | ///@brief The active bank (most recent copy). Points to either m_left or m_right. 290 | StorageBank* m_active; 291 | 292 | ///@brief Log size to use when formatting a new bank (number of entries) 293 | uint32_t m_defaultLogSize; 294 | 295 | ///@brief Index of the next log entry to write to 296 | uint32_t m_firstFreeLogEntry; 297 | 298 | ///@brief Offset (from start of block) of the first free data byte 299 | uint32_t m_firstFreeData; 300 | 301 | ///@brief Error flag thrown from NMI/fault handler 302 | volatile bool m_eccFault; 303 | 304 | ///@brief Bad flash address when m_eccFault was set 305 | volatile uint32_t m_eccFaultAddr; 306 | 307 | ///@brief Program counter value when m_eccFault was set 308 | volatile uint32_t m_eccFaultPC; 309 | }; 310 | 311 | //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// 312 | // Helper macro to disable data faults (but only if we have flash ECC) 313 | 314 | #ifdef HAVE_FLASH_ECC 315 | #define unsafe if(DataFaultDisabler df; 1) 316 | #else 317 | #define unsafe if(1) 318 | #endif 319 | 320 | #endif 321 | -------------------------------------------------------------------------------- /kvs/LogEntry.h: -------------------------------------------------------------------------------- 1 | /*********************************************************************************************************************** 2 | * * 3 | * microkvs * 4 | * * 5 | * Copyright (c) 2021-2024 Andrew D. Zonenberg and contributors * 6 | * All rights reserved. * 7 | * * 8 | * Redistribution and use in source and binary forms, with or without modification, are permitted provided that the * 9 | * following conditions are met: * 10 | * * 11 | * * Redistributions of source code must retain the above copyright notice, this list of conditions, and the * 12 | * following disclaimer. * 13 | * * 14 | * * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the * 15 | * following disclaimer in the documentation and/or other materials provided with the distribution. * 16 | * * 17 | * * Neither the name of the author nor the names of any contributors may be used to endorse or promote products * 18 | * derived from this software without specific prior written permission. * 19 | * * 20 | * THIS SOFTWARE IS PROVIDED BY THE AUTHORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * 21 | * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL * 22 | * THE AUTHORS BE HELD LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES * 23 | * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR * 24 | * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * 25 | * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * 26 | * POSSIBILITY OF SUCH DAMAGE. * 27 | * * 28 | ***********************************************************************************************************************/ 29 | 30 | /** 31 | @file 32 | @author Andrew D. Zonenberg 33 | @brief Declaration of LogEntry 34 | */ 35 | 36 | #ifndef LogEntry_h 37 | #define LogEntry_h 38 | 39 | //Block writable flash 40 | #ifdef MICROKVS_WRITE_BLOCK_SIZE 41 | 42 | //User-specified block size? Make sure it's valid 43 | #ifdef KVS_NAMELEN 44 | 45 | //Name length must be at least one write block 46 | #if ( KVS_NAMELEN < MICROKVS_WRITE_BLOCK_SIZE ) 47 | #error KVS_NAMELEN must be an integer multiple of MICROKVS_WRITE_BLOCK_SIZE 48 | #endif 49 | 50 | //Block size must evenly divide name length 51 | #if ( (KVS_NAMELEN % MICROKVS_WRITE_BLOCK_SIZE) != 0 ) 52 | #error KVS_NAMELEN must be an integer multiple of MICROKVS_WRITE_BLOCK_SIZE 53 | #endif 54 | 55 | 56 | //Use write block size as default name length 57 | #else 58 | #define KVS_NAMELEN MICROKVS_WRITE_BLOCK_SIZE 59 | #endif 60 | 61 | //Byte writable flash 62 | #else 63 | #define KVS_NAMELEN 16 64 | #endif 65 | 66 | /** 67 | @brief A single entry in the flash log 68 | */ 69 | class LogEntry 70 | { 71 | public: 72 | char m_key[KVS_NAMELEN]; 73 | uint32_t m_start; 74 | uint32_t m_len; 75 | uint32_t m_crc; //crc32 of packet content 76 | uint32_t m_headerCRC; //crc32 of {key, start, len} 77 | 78 | //pad to write block size 79 | #ifdef MICROKVS_WRITE_BLOCK_SIZE 80 | #if MICROKVS_WRITE_BLOCK_SIZE >= 16 81 | uint8_t m_padding[MICROKVS_WRITE_BLOCK_SIZE - (16 % MICROKVS_WRITE_BLOCK_SIZE)]; 82 | #endif 83 | #endif 84 | }; 85 | 86 | #endif 87 | -------------------------------------------------------------------------------- /tests/.gitignore: -------------------------------------------------------------------------------- 1 | test 2 | *.o 3 | -------------------------------------------------------------------------------- /tests/Makefile: -------------------------------------------------------------------------------- 1 | CFLAGS=-g -O2 2 | CXXFLAGS=$(CFLAGS) --std=c++17 -fno-exceptions -fno-rtti \ 3 | -I../ 4 | CC=gcc 5 | CXX=g++ 6 | 7 | all: 8 | $(CXX) -c ../kvs/*.cpp $(CXXFLAGS) 9 | $(CXX) -c ../driver/TestStorageBank.cpp $(CXXFLAGS) 10 | $(CXX) -c *.cpp $(CXXFLAGS) 11 | $(CXX) *.o -o test $(CXXFLAGS) 12 | -------------------------------------------------------------------------------- /tests/main.cpp: -------------------------------------------------------------------------------- 1 | /*********************************************************************************************************************** 2 | * * 3 | * microkvs v0.1 * 4 | * * 5 | * Copyright (c) 2021 Andrew D. Zonenberg and contributors * 6 | * All rights reserved. * 7 | * * 8 | * Redistribution and use in source and binary forms, with or without modification, are permitted provided that the * 9 | * following conditions are met: * 10 | * * 11 | * * Redistributions of source code must retain the above copyright notice, this list of conditions, and the * 12 | * following disclaimer. * 13 | * * 14 | * * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the * 15 | * following disclaimer in the documentation and/or other materials provided with the distribution. * 16 | * * 17 | * * Neither the name of the author nor the names of any contributors may be used to endorse or promote products * 18 | * derived from this software without specific prior written permission. * 19 | * * 20 | * THIS SOFTWARE IS PROVIDED BY THE AUTHORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * 21 | * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL * 22 | * THE AUTHORS BE HELD LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES * 23 | * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR * 24 | * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * 25 | * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * 26 | * POSSIBILITY OF SUCH DAMAGE. * 27 | * * 28 | ***********************************************************************************************************************/ 29 | 30 | #include 31 | #include 32 | #include 33 | 34 | void PrintState(KVS& kvs); 35 | 36 | bool WriteAndVerify(KVS& kvs, const char* name, uint8_t* data, uint32_t len); 37 | bool Verify(KVS& kvs, const char* name, uint8_t* data, uint32_t len); 38 | 39 | int main(int argc, char* argv[]) 40 | { 41 | //Create the KVS 42 | //128 log entries = 3.5 kB used by log, about 28.5 kB free for data 43 | TestStorageBank left; 44 | TestStorageBank right; 45 | KVS kvs(&left, &right, 128); 46 | 47 | //Verify sane initial config 48 | printf("INITIAL STATE\n"); 49 | PrintState(kvs); 50 | 51 | const char* data = "hello world"; 52 | if(!WriteAndVerify(kvs, "OHAI", (uint8_t*)data, strlen(data))) 53 | return 1; 54 | 55 | printf("WITH ONE OBJECT\n"); 56 | PrintState(kvs); 57 | 58 | //Write a second file 59 | const char* data2 = "lolcat"; 60 | if(!WriteAndVerify(kvs, "shibe", (uint8_t*)data2, strlen(data2))) 61 | return 1; 62 | 63 | printf("WITH TWO OBJECTS\n"); 64 | PrintState(kvs); 65 | 66 | //Write new content to the first object 67 | const char* data3 = "i herd u leik mudkipz"; 68 | if(!WriteAndVerify(kvs, "OHAI", (uint8_t*)data3, strlen(data3))) 69 | return 1; 70 | if(!Verify(kvs, "shibe", (uint8_t*)data2, strlen(data2))) 71 | return 1; 72 | 73 | printf("MODIFIED 1\n"); 74 | PrintState(kvs); 75 | 76 | //Write new content to the second object 77 | const char* data4 = "ceiling cat is watching"; 78 | if(!WriteAndVerify(kvs, "shibe", (uint8_t*)data4, strlen(data4))) 79 | return 1; 80 | if(!Verify(kvs, "OHAI", (uint8_t*)data3, strlen(data3))) 81 | return 1; 82 | 83 | printf("MODIFIED 2\n"); 84 | PrintState(kvs); 85 | 86 | //Write new content to a third object 87 | const char* data5 = "basement cat attacks!!!1!1!"; 88 | if(!WriteAndVerify(kvs, "monorail", (uint8_t*)data5, strlen(data5))) 89 | return 1; 90 | if(!Verify(kvs, "OHAI", (uint8_t*)data3, strlen(data3))) 91 | return 1; 92 | if(!Verify(kvs, "shibe", (uint8_t*)data4, strlen(data4))) 93 | return 1; 94 | 95 | printf("THREE OBJECTS\n"); 96 | PrintState(kvs); 97 | 98 | //Compact the array and verify both objects 99 | kvs.Compact(); 100 | printf("COMPACTED\n"); 101 | PrintState(kvs); 102 | if(!Verify(kvs, "OHAI", (uint8_t*)data3, strlen(data3))) 103 | return 1; 104 | if(!Verify(kvs, "shibe", (uint8_t*)data4, strlen(data4))) 105 | return 1; 106 | if(!Verify(kvs, "monorail", (uint8_t*)data5, strlen(data5))) 107 | return 1; 108 | 109 | return 0; 110 | } 111 | 112 | bool WriteAndVerify(KVS& kvs, const char* name, uint8_t* data, uint32_t len) 113 | { 114 | if(!kvs.StoreObject(name, data, len)) 115 | { 116 | printf("Failed to store object\n"); 117 | return false; 118 | } 119 | 120 | return Verify(kvs, name, data, len); 121 | } 122 | 123 | bool Verify(KVS& kvs, const char* name, uint8_t* data, uint32_t len) 124 | { 125 | //Read it back 126 | auto log = kvs.FindObject(name); 127 | if(!log) 128 | { 129 | printf("Object couldn't be found\n"); 130 | return false; 131 | } 132 | if(log->m_len != len) 133 | { 134 | printf("Log entry length is wrong\n"); 135 | return false; 136 | } 137 | if(memcmp(data, kvs.MapObject(log), log->m_len) != 0) 138 | { 139 | printf("Object content is wrong\n"); 140 | return false; 141 | } 142 | return true; 143 | } 144 | 145 | void PrintState(KVS& kvs) 146 | { 147 | if(kvs.IsLeftBankActive()) 148 | printf(" Active bank: left\n"); 149 | else 150 | printf(" Active bank: right\n"); 151 | printf(" Free log entries: %d\n", kvs.GetFreeLogEntries()); 152 | printf(" Free data space: %d\n", kvs.GetFreeDataSpace()); 153 | } 154 | --------------------------------------------------------------------------------