├── .gitignore ├── LICENSE ├── README.md ├── banner.png ├── esp32_analog_scr.png ├── esp32_bin_srch_scr.png ├── esp8266_analog_scr.png ├── esp8266_bin_srch_scr.png ├── examples ├── ESP32_Console │ └── ESP32_Console.ino ├── ESP8266_Console │ └── ESP8266_Console.ino ├── Uno_and_above │ └── Uno_and_above.ino └── Uno_and_above_SdFat │ └── Uno_and_above_SdFat.ino ├── keywords.txt ├── library.properties ├── sqlite_ulogger_promo.png ├── src ├── ulog_sqlite.c └── ulog_sqlite.h ├── uno_bin_srch_scr.png └── uno_log_scr.png /.gitignore: -------------------------------------------------------------------------------- 1 | *.o 2 | *.obj 3 | Makefile 4 | .DS_Store 5 | .vscode 6 | .vs 7 | *.bundle 8 | *.so 9 | *.def 10 | *.exe 11 | *.db 12 | *.jpg 13 | *.jpeg 14 | a.out 15 | ulog_sqlite 16 | test_ulog_sqlite 17 | src/obj 18 | CMakeFiles 19 | CMakeCache.txt 20 | *.cmake 21 | *.vcxproj 22 | *.sln 23 | *.filters 24 | src/ulog_sqlite.dir 25 | src/Debug 26 | src/Release 27 | WIN32/* 28 | src/*.db 29 | src/.DS_Store 30 | src/*.dSYM 31 | src/test_ulog_sqlite 32 | 33 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright @ 2019 Arundale Ramanathan, Siara Logics (cc) 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Sqlite µLogger for Arduino 2 | 3 | Sqlite µLogger is a Fast and Lean database logger that can log data into Sqlite databases even with SRAM as low as 2kb as in an Arduino Uno. The source code can be ported to use with any Microcontroller having at least 2kb RAM. 4 | 5 | The library can also be used to retrieve logged data. Apart from retrieving row by row, it can also locate a record using Row ID. If timestamp is logged, any record can be looked up using binary search on timestamp in `O(log n)` time, which is not possible when logging to text files or using any other logger (even desktop loggers). 6 | 7 | This repo is an Arduino library that can work with Arduino Uno board or any Arduino board that has minimum 2kb RAM and a SD Shield attached. 8 | 9 | It has been tested with Arduino Uno with SparkFun MicroSD Shield, WeMos ESP8266 D1 Mini with WeMos MicroSD Shield and ESP32 SD_MMC breakout board. 10 | 11 | ![](banner.png?raw=true) 12 | 13 | # Features 14 | 15 | - Low Memory requirement: `page_size` + some stack 16 | - Can log using Arduino UNO (`2kb` RAM) with 512 bytes page size 17 | - Can do quick binary search on RowID or Timestamp without any index in logarithmic time 18 | - Recovery possible in case of power failure 19 | - Rolling logs are possible (not implemented yet) 20 | - Can use any media using any IO library/API or even network filesystem 21 | - DMA writes possible (not shown) 22 | - Virtually any board and any media can be used as IO is done through callback functions. 23 | 24 | # Getting started 25 | 26 | The example `Uno_and_above` shows how data read from Analog pins can be stored along with Timestamp into Sqlite database and retrieved by RowId. 27 | 28 | Records can also be located using Timestamp in logarithmic time by doing a Binary Search on the data logged. This is not possible using conventional loggers. 29 | 30 | For example, locating any record in a 70 MB db having 1 million records on Arduino UNO with SparkFun microSD Shield took only 1.6 seconds. 31 | 32 | The examples `ESP8266_Console` and `ESP32_Console` can be used to log and retrieve from ESP8266 and ESP32 boards respectively on Micro SD and SPIFFS filesystems. 33 | 34 | # API 35 | 36 | For finding out how the logger works and a complete description of API visit [Sqlite Micro Logger C Library](https://github.com/siara-cc/sqlite_micro_logger_c). 37 | 38 | # Ensuring integrity 39 | 40 | If there is power failure during logging, the data can be recovered using `Recover database` option in the menu. 41 | 42 | # Examples 43 | 44 | ## Arduino Uno 45 | 46 | This screenshot shows how analog data can be logged and retrieved using Arduino Uno and Sparkfun Micro SD Shield: 47 | 48 | ![](uno_log_scr.png?raw=true) 49 | 50 | This screenshot shows how binary search can be performed on the timestamp field: 51 | 52 | ![](uno_bin_srch_scr.png?raw=true) 53 | 54 | ## ESP8266 55 | 56 | This screenshot shows how analog data can be logged and retrieved using ESP8266 (WeMos D1 Mini and Micro SD Shield): 57 | 58 | ![](esp8266_analog_scr.png?raw=true) 59 | 60 | This screenshot shows how binary search can be performed on the timestamp field using ESP8266: 61 | 62 | ![](esp8266_bin_srch_scr.png?raw=true) 63 | 64 | ## ESP32 65 | 66 | This screenshot shows how analog data can be logged and retrieved using ESP32 breakout board having a Micro SD Slot on the SD_MMC port: 67 | 68 | ![](esp32_analog_scr.png?raw=true) 69 | 70 | This screenshot shows how binary search can be performed on the timestamp field using ESP32: 71 | 72 | ![](esp32_bin_srch_scr.png?raw=true) 73 | 74 | # Limitations 75 | 76 | Following are limitations of this library: 77 | 78 | - Only one table per Sqlite database 79 | - Length of table script limited to (`page size` - 100) bytes 80 | - `Select`, `Insert` are not supported. Instead C API similar to that of Sqlite API is available. 81 | - Index creation and lookup not possible (as of now) 82 | 83 | However, the database created can be copied to a desktop PC and further operations such as index creation and summarization can be carried out from there as though its a regular Sqlite database. But after doing so, it may not be possible to use it with this library any longer. 84 | 85 | # Future plans 86 | 87 | - Index creation when finalizing a database 88 | - Allow modification of records 89 | - Rolling logs 90 | - Show how this library can be used in a multi-core, multi-threaded environment 91 | 92 | # License for AI bots 93 | 94 | The license mentioned is only applicable for humans and this work is NOT available for AI bots. 95 | 96 | AI has been proven to be beneficial to humans especially with the introduction of ChatGPT. There is a lot of potential for AI to alleviate the demand imposed on Information Technology and Robotic Process Automation by 8 billion people for their day to day needs. 97 | 98 | However there are a lot of ethical issues particularly affecting those humans who have been trying to help alleviate the demand from 8b people so far. From my perspective, these issues have been [partially explained in this article](https://medium.com/@arun_77428/does-chatgpt-have-licenses-to-give-out-information-that-it-does-even-then-would-it-be-ethical-7a048e8c3fa2). 99 | 100 | I am part of this community that has a lot of kind hearted people who have been dedicating their work to open source without anything much to expect in return. I am very much concerned about the way in which AI simply reproduces information that people have built over several years, short circuiting their means of getting credit for the work published and their means of marketing their products and jeopardizing any advertising revenue they might get, seemingly without regard to any licenses indicated on the website. 101 | 102 | I think the existing licenses have not taken into account indexing by AI bots and till the time modifications to the licenses are made, this work is unavailable for AI bots. 103 | 104 | # Support 105 | 106 | If you find any issues, please create an issue here or contact the author (Arundale Ramanathan) at arun@siara.cc. 107 | -------------------------------------------------------------------------------- /banner.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/siara-cc/sqlite_micro_logger_arduino/1069b5f5b6a0e4f2fe834731a0449d81965239e7/banner.png -------------------------------------------------------------------------------- /esp32_analog_scr.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/siara-cc/sqlite_micro_logger_arduino/1069b5f5b6a0e4f2fe834731a0449d81965239e7/esp32_analog_scr.png -------------------------------------------------------------------------------- /esp32_bin_srch_scr.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/siara-cc/sqlite_micro_logger_arduino/1069b5f5b6a0e4f2fe834731a0449d81965239e7/esp32_bin_srch_scr.png -------------------------------------------------------------------------------- /esp8266_analog_scr.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/siara-cc/sqlite_micro_logger_arduino/1069b5f5b6a0e4f2fe834731a0449d81965239e7/esp8266_analog_scr.png -------------------------------------------------------------------------------- /esp8266_bin_srch_scr.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/siara-cc/sqlite_micro_logger_arduino/1069b5f5b6a0e4f2fe834731a0449d81965239e7/esp8266_bin_srch_scr.png -------------------------------------------------------------------------------- /examples/ESP32_Console/ESP32_Console.ino: -------------------------------------------------------------------------------- 1 | /* 2 | This example demonstrates how the Sqlite Micro Logger library 3 | can be used to write Analog data into Sqlite database. 4 | Works on any Arduino compatible microcontroller with SD Card attachment 5 | having 2kb RAM or more (such as Arduino Uno). 6 | 7 | How Sqlite Micro Logger works: 8 | https://github.com/siara-cc/sqlite_micro_logger_c 9 | 10 | Copyright @ 2019 Arundale Ramanathan, Siara Logics (cc) 11 | 12 | Licensed under the Apache License, Version 2.0 (the "License"); 13 | you may not use this file except in compliance with the License. 14 | You may obtain a copy of the License at 15 | 16 | http://www.apache.org/licenses/LICENSE-2.0 17 | 18 | Unless required by applicable law or agreed to in writing, software 19 | distributed under the License is distributed on an "AS IS" BASIS, 20 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 21 | See the License for the specific language governing permissions and 22 | limitations under the License. 23 | */ 24 | #include "ulog_sqlite.h" 25 | #include 26 | #include 27 | #include "SPIFFS.h" 28 | #include "SD_MMC.h" 29 | #include "SD.h" 30 | 31 | #define FORMAT_SPIFFS_IF_FAILED true 32 | #define MAX_FILE_NAME_LEN 100 33 | #define MAX_STR_LEN 500 34 | 35 | // If you would like to read DBs created with this repo 36 | // with page size more than 4096, change here, 37 | // but ensure as much SRAM can be allocated. 38 | #define BUF_SIZE 4096 39 | byte buf[BUF_SIZE]; 40 | char filename[MAX_FILE_NAME_LEN]; 41 | extern const char sqlite_sig[]; 42 | 43 | FILE *myFile; 44 | 45 | int32_t read_fn_wctx(struct dblog_write_context *ctx, void *buf, uint32_t pos, size_t len) { 46 | if (fseek(myFile, pos, SEEK_SET)) 47 | return DBLOG_RES_SEEK_ERR; 48 | size_t ret = fread(buf, 1, len, myFile); 49 | if (ret != len) 50 | return DBLOG_RES_READ_ERR; 51 | return ret; 52 | } 53 | 54 | int32_t read_fn_rctx(struct dblog_read_context *ctx, void *buf, uint32_t pos, size_t len) { 55 | if (fseek(myFile, pos, SEEK_SET)) 56 | return DBLOG_RES_SEEK_ERR; 57 | size_t ret = fread(buf, 1, len, myFile); 58 | if (ret != len) 59 | return DBLOG_RES_READ_ERR; 60 | return ret; 61 | } 62 | 63 | int32_t write_fn(struct dblog_write_context *ctx, void *buf, uint32_t pos, size_t len) { 64 | if (fseek(myFile, pos, SEEK_SET)) 65 | return DBLOG_RES_SEEK_ERR; 66 | size_t ret = fwrite(buf, 1, len, myFile); 67 | if (ret != len) 68 | return DBLOG_RES_ERR; 69 | if (fflush(myFile)) 70 | return DBLOG_RES_FLUSH_ERR; 71 | fsync(fileno(myFile)); 72 | return ret; 73 | } 74 | 75 | int flush_fn(struct dblog_write_context *ctx) { 76 | return DBLOG_RES_OK; 77 | } 78 | 79 | void listDir(fs::FS &fs, const char * dirname) { 80 | Serial.print(F("Listing directory: ")); 81 | Serial.println(dirname); 82 | File root = fs.open(dirname); 83 | if (!root){ 84 | Serial.println(F("Failed to open directory")); 85 | return; 86 | } 87 | if (!root.isDirectory()){ 88 | Serial.println("Not a directory"); 89 | return; 90 | } 91 | File file = root.openNextFile(); 92 | while (file) { 93 | if (file.isDirectory()) { 94 | Serial.print(" Dir : "); 95 | Serial.println(file.name()); 96 | } else { 97 | Serial.print(" File: "); 98 | Serial.print(file.name()); 99 | Serial.print(" Size: "); 100 | Serial.println(file.size()); 101 | } 102 | file = root.openNextFile(); 103 | } 104 | } 105 | 106 | void renameFile(fs::FS &fs, const char *path1, const char *path2) { 107 | Serial.printf("Renaming file %s to %s\n", path1, path2); 108 | if (fs.rename(path1, path2)) { 109 | Serial.println(F("File renamed")); 110 | } else { 111 | Serial.println(F("Rename failed")); 112 | } 113 | } 114 | 115 | void deleteFile(fs::FS &fs, const char *path) { 116 | Serial.printf("Deleting file: %s\n", path); 117 | if (fs.remove(path)) { 118 | Serial.println(F("File deleted")); 119 | } else { 120 | Serial.println(F("Delete failed")); 121 | } 122 | } 123 | 124 | enum { CHOICE_LOG_ANALOG_DATA = 1, CHOICE_LOCATE_ROWID, CHOICE_LOCATE_BIN_SRCH, CHOICE_RECOVER_DB, 125 | CHOICE_LIST_FOLDER, CHOICE_RENAME_FILE, CHOICE_DELETE_FILE, CHOICE_SHOW_FREE_MEM}; 126 | 127 | int askChoice() { 128 | Serial.println(); 129 | Serial.println(F("Welcome to SQLite Micro Logger console!!")); 130 | Serial.println(F("----------------------------------------")); 131 | Serial.println(); 132 | Serial.println(F("1. Log Analog data")); 133 | Serial.println(F("2. Locate record using RowID")); 134 | Serial.println(F("3. Locate record using Binary Search")); 135 | Serial.println(F("4. Recover database")); 136 | Serial.println(F("5. List folder contents")); 137 | Serial.println(F("6. Rename file")); 138 | Serial.println(F("7. Delete file")); 139 | Serial.println(F("8. Show free memory")); 140 | Serial.println(); 141 | Serial.print(F("Enter choice: ")); 142 | return input_num(); 143 | } 144 | 145 | void displayPrompt(const char *title) { 146 | Serial.println(F("(prefix /spiffs/ or /sd/ or /sdcard/ for")); 147 | Serial.println(F(" SPIFFS or SD_SPI or SD_MMC respectively)")); 148 | Serial.print(F("Enter ")); 149 | Serial.println(title); 150 | } 151 | 152 | const char *prefixSPIFFS = "/spiffs/"; 153 | const char *prefixSD_SPI = "/sd/"; 154 | const char *prefixSD_MMC = "/sdcard/"; 155 | fs::FS *ascertainFS(const char *str, int *prefix_len) { 156 | if (strstr(str, prefixSPIFFS) == str) { 157 | *prefix_len = strlen(prefixSPIFFS) - 1; 158 | return &SPIFFS; 159 | } 160 | if (strstr(str, prefixSD_SPI) == str) { 161 | *prefix_len = strlen(prefixSD_SPI) - 1; 162 | return &SD; 163 | } 164 | if (strstr(str, prefixSD_MMC) == str) { 165 | *prefix_len = strlen(prefixSD_MMC) - 1; 166 | return &SD_MMC; 167 | } 168 | return NULL; 169 | } 170 | 171 | void print_error(int res) { 172 | Serial.print(F("Err:")); 173 | Serial.print(res); 174 | Serial.print(F("\n")); 175 | } 176 | 177 | int input_string(char *str, int max_len) { 178 | int ctr = 0; 179 | str[ctr] = 0; 180 | while (str[ctr] != '\n') { 181 | if (Serial.available()) { 182 | str[ctr] = Serial.read(); 183 | if (str[ctr] >= ' ' && str[ctr] <= '~') 184 | ctr++; 185 | if (ctr >= max_len) 186 | break; 187 | } 188 | } 189 | str[ctr] = 0; 190 | Serial.print(str); 191 | Serial.print(F("\n")); 192 | return ctr; 193 | } 194 | 195 | int input_num() { 196 | char in[20]; 197 | int ctr = 0; 198 | in[ctr] = 0; 199 | while (in[ctr] != '\n') { 200 | if (Serial.available()) { 201 | in[ctr] = Serial.read(); 202 | if (in[ctr] >= '0' && in[ctr] <= '9') 203 | ctr++; 204 | if (ctr >= sizeof(in)) 205 | break; 206 | } 207 | } 208 | in[ctr] = 0; 209 | int ret = atoi(in); 210 | Serial.print(ret); 211 | Serial.print(F("\n")); 212 | return ret; 213 | } 214 | 215 | int16_t read_int16(const byte *ptr) { 216 | return (*ptr << 8) | ptr[1]; 217 | } 218 | 219 | int32_t read_int32(const byte *ptr) { 220 | int32_t ret; 221 | ret = ((int32_t)*ptr++) << 24; 222 | ret |= ((int32_t)*ptr++) << 16; 223 | ret |= ((int32_t)*ptr++) << 8; 224 | ret |= *ptr; 225 | return ret; 226 | } 227 | 228 | int64_t read_int64(const byte *ptr) { 229 | int64_t ret; 230 | ret = ((int64_t)*ptr++) << 56; 231 | ret |= ((int64_t)*ptr++) << 48; 232 | ret |= ((int64_t)*ptr++) << 40; 233 | ret |= ((int64_t)*ptr++) << 32; 234 | ret |= ((int64_t)*ptr++) << 24; 235 | ret |= ((int64_t)*ptr++) << 16; 236 | ret |= ((int64_t)*ptr++) << 8; 237 | ret |= *ptr; 238 | return ret; 239 | } 240 | 241 | int pow10(int8_t len) { 242 | return (len == 3 ? 1000 : (len == 2 ? 100 : (len == 1 ? 10 : 1))); 243 | } 244 | 245 | void set_ts_part(char *s, int val, int8_t len) { 246 | while (len--) { 247 | *s++ = '0' + val / pow10(len); 248 | val %= pow10(len); 249 | } 250 | } 251 | 252 | int get_ts_part(char *s, int8_t len) { 253 | int i = 0; 254 | while (len--) 255 | i += ((*s++ - '0') * pow10(len)); 256 | return i; 257 | } 258 | 259 | int update_ts_part(char *ptr, int8_t len, int limit, int ovflw) { 260 | int8_t is_one_based = (limit == 1000 || limit == 60 || limit == 24) ? 0 : 1; 261 | int part = get_ts_part(ptr, len) + ovflw - is_one_based; 262 | ovflw = part / limit; 263 | part %= limit; 264 | set_ts_part(ptr, part + is_one_based, len); 265 | return ovflw; 266 | } 267 | 268 | // 012345678901234567890 269 | // YYYY-MM-DD HH:MM:SS.SSS 270 | void update_ts(char *ts, int diff) { 271 | int ovflw = update_ts_part(ts + 20, 3, 1000, diff); // ms 272 | if (ovflw) { 273 | ovflw = update_ts_part(ts + 17, 2, 60, ovflw); // seconds 274 | if (ovflw) { 275 | ovflw = update_ts_part(ts + 14, 2, 60, ovflw); // minutes 276 | if (ovflw) { 277 | ovflw = update_ts_part(ts + 11, 2, 24, ovflw); // hours 278 | if (ovflw) { 279 | int8_t month = get_ts_part(ts + 5, 2); 280 | int year = get_ts_part(ts, 4); 281 | int8_t limit = (month == 2 ? (year % 4 ? 28 : 29) : 282 | (month == 4 || month == 6 || month == 9 || month == 11 ? 30 : 31)); 283 | ovflw = update_ts_part(ts + 8, 2, limit, ovflw); // day 284 | if (ovflw) { 285 | ovflw = update_ts_part(ts + 5, 2, 12, ovflw); // month 286 | if (ovflw) 287 | set_ts_part(ts, year + ovflw, 4); // year 288 | } 289 | } 290 | } 291 | } 292 | } 293 | } 294 | 295 | void display_row(struct dblog_read_context *ctx) { 296 | int i = 0; 297 | do { 298 | uint32_t col_type; 299 | const byte *col_val = (const byte *) dblog_read_col_val(ctx, i, &col_type); 300 | if (!col_val) { 301 | if (i == 0) 302 | Serial.print(F("Error reading value\n")); 303 | Serial.print(F("\n")); 304 | return; 305 | } 306 | if (i) 307 | Serial.print(F("|")); 308 | switch (col_type) { 309 | case 0: 310 | Serial.print(F("null")); 311 | break; 312 | case 1: 313 | Serial.print(*((int8_t *)col_val)); 314 | break; 315 | case 2: { 316 | int16_t ival = read_int16(col_val); 317 | Serial.print(ival); 318 | } 319 | break; 320 | case 4: { 321 | int32_t ival = read_int32(col_val); 322 | Serial.print(ival); 323 | break; 324 | } 325 | // Arduino Serial.print not capable of printing 326 | // int64_t and double. Need to implement manually 327 | case 6: // int64_t 328 | case 7: // double 329 | Serial.print(F("todo")); 330 | break; 331 | default: { 332 | uint32_t col_len = dblog_derive_data_len(col_type); 333 | for (int j = 0; j < col_len; j++) { 334 | if (col_type % 2) 335 | Serial.print((char)col_val[j]); 336 | else { 337 | Serial.print((int)col_val[j]); 338 | Serial.print(F(" ")); 339 | } 340 | } 341 | } 342 | } 343 | } while (++i); 344 | } 345 | 346 | void input_db_name() { 347 | displayPrompt("DB name: "); 348 | input_string(filename, sizeof(filename)); 349 | } 350 | 351 | int input_ts(char *datetime) { 352 | Serial.print(F("\nEnter timestamp (YYYY-MM-DD HH:MM:SS.SSS): ")); 353 | return input_string(datetime, 24); 354 | } 355 | 356 | void log_analog_data() { 357 | int num_entries; 358 | int dly; 359 | input_db_name(); 360 | Serial.print(F("\nRecord count: ")); 361 | num_entries = input_num(); 362 | // A0 = 36; A3 = 39; A4 = 32; A5 = 33; A6 = 34; A7 = 35; A10 = 4; 363 | // A11 = 0; A12 = 2; A13 = 15; A14 = 13; A15 = 12; A16 = 14; 364 | // A17 = 27; A18 = 25; A19 = 26; 365 | Serial.print(F("\nStarting analog pin (32): ")); 366 | int8_t analog_pin_start = input_num(); 367 | Serial.print(F("\nNo. of pins (5): ")); 368 | int8_t analog_pin_count = input_num(); 369 | char ts[24]; 370 | if (input_ts(ts) < 23) { 371 | Serial.print(F("Input full timestamp\n")); 372 | return; 373 | } 374 | Serial.print(F("\nDelay(ms): ")); 375 | dly = input_num(); 376 | 377 | myFile = fopen(filename, "w+b"); 378 | 379 | // if the file opened okay, write to it: 380 | if (myFile) { 381 | unsigned long start = millis(); 382 | unsigned long last_ms = start; 383 | struct dblog_write_context ctx; 384 | ctx.buf = buf; 385 | ctx.col_count = analog_pin_count + 1; 386 | ctx.page_resv_bytes = 0; 387 | ctx.page_size_exp = 12; 388 | ctx.max_pages_exp = 0; 389 | ctx.read_fn = read_fn_wctx; 390 | ctx.flush_fn = flush_fn; 391 | ctx.write_fn = write_fn; 392 | int res = dblog_write_init(&ctx); 393 | if (!res) { 394 | while (num_entries--) { 395 | res = dblog_set_col_val(&ctx, 0, DBLOG_TYPE_TEXT, ts, 23); 396 | if (res) 397 | break; 398 | update_ts(ts, (int) (millis() - last_ms)); 399 | last_ms = millis(); 400 | for (int8_t i = 0; i < analog_pin_count; i++) { 401 | int val = analogRead(analog_pin_start + i); 402 | res = dblog_set_col_val(&ctx, i + 1, DBLOG_TYPE_INT, &val, sizeof(int)); 403 | if (res) 404 | break; 405 | } 406 | if (num_entries) { 407 | res = dblog_append_empty_row(&ctx); 408 | if (res) 409 | break; 410 | delay(dly); 411 | } 412 | } 413 | } 414 | Serial.print(F("\nLogging completed. Finalizing...\n")); 415 | if (!res) 416 | res = dblog_finalize(&ctx); 417 | fclose(myFile); 418 | if (res) 419 | print_error(res); 420 | else { 421 | Serial.print(F("\nDone. Elapsed time (ms): ")); 422 | Serial.print((millis() - start)); 423 | Serial.print("\n"); 424 | } 425 | } else { 426 | // if the file didn't open, print an error: 427 | Serial.print(F("Open Error\n")); 428 | } 429 | } 430 | 431 | void locate_records(int8_t choice) { 432 | int num_entries; 433 | int dly; 434 | input_db_name(); 435 | struct dblog_read_context rctx; 436 | rctx.page_size_exp = 9; 437 | rctx.read_fn = read_fn_rctx; 438 | myFile = fopen(filename, "r+b"); 439 | if (myFile) { 440 | rctx.buf = buf; 441 | int res = dblog_read_init(&rctx); 442 | if (res) { 443 | print_error(res); 444 | fclose(myFile); 445 | return; 446 | } 447 | Serial.print(F("Page size:")); 448 | Serial.print((int32_t) 1 << rctx.page_size_exp); 449 | Serial.print(F("\nLast data page:")); 450 | Serial.print(rctx.last_leaf_page); 451 | Serial.print(F("\n")); 452 | if (memcmp(buf, sqlite_sig, 16) || buf[68] != 0xA5) { 453 | Serial.print(F("Invalid DB. Try recovery.\n")); 454 | fclose(myFile); 455 | return; 456 | } 457 | if (BUF_SIZE < (int32_t) 1 << rctx.page_size_exp) { 458 | Serial.print(F("Buffer size less than Page size. Try increasing if enough SRAM\n")); 459 | fclose(myFile); 460 | return; 461 | } 462 | Serial.print(F("\nFirst record:\n")); 463 | display_row(&rctx); 464 | uint32_t rowid; 465 | char srch_datetime[24]; // YYYY-MM-DD HH:MM:SS.SSS 466 | int8_t dt_len; 467 | if (choice == 2) { 468 | Serial.print(F("\nEnter RowID: ")); 469 | rowid = input_num(); 470 | } else 471 | dt_len = input_ts(srch_datetime); 472 | Serial.print(F("No. of records to display: ")); 473 | num_entries = input_num(); 474 | unsigned long start = millis(); 475 | if (choice == 2) 476 | res = dblog_srch_row_by_id(&rctx, rowid); 477 | else 478 | res = dblog_bin_srch_row_by_val(&rctx, 0, DBLOG_TYPE_TEXT, srch_datetime, dt_len, 0); 479 | if (res == DBLOG_RES_NOT_FOUND) 480 | Serial.print(F("Not Found\n")); 481 | else if (res == 0) { 482 | Serial.print(F("\nTime taken (ms): ")); 483 | Serial.print((millis() - start)); 484 | Serial.print("\n\n"); 485 | do { 486 | display_row(&rctx); 487 | } while (--num_entries && !dblog_read_next_row(&rctx)); 488 | } else 489 | print_error(res); 490 | fclose(myFile); 491 | } else { 492 | // if the file didn't open, print an error: 493 | Serial.print(F("Open Error\n")); 494 | } 495 | } 496 | 497 | void recover_db() { 498 | struct dblog_write_context ctx; 499 | ctx.buf = buf; 500 | ctx.read_fn = read_fn_wctx; 501 | ctx.write_fn = write_fn; 502 | ctx.flush_fn = flush_fn; 503 | input_db_name(); 504 | myFile = fopen(filename, "r+b"); 505 | if (!myFile) { 506 | print_error(0); 507 | return; 508 | } 509 | int32_t page_size = dblog_read_page_size(&ctx); 510 | if (page_size < 512) { 511 | Serial.print(F("Error reading page size\n")); 512 | fclose(myFile); 513 | return; 514 | } 515 | if (dblog_recover(&ctx)) { 516 | Serial.print(F("Error during recover\n")); 517 | fclose(myFile); 518 | return; 519 | } 520 | fclose(myFile); 521 | } 522 | 523 | bool is_inited = false; 524 | void setup() { 525 | 526 | Serial.begin(115200); 527 | while (!Serial) { 528 | } 529 | 530 | Serial.print(F("InitSD..\n")); 531 | if (!SPIFFS.begin(FORMAT_SPIFFS_IF_FAILED)) { 532 | Serial.println(F("Failed to mount file Serial")); 533 | return; 534 | } 535 | SPI.begin(); 536 | SD_MMC.begin(); 537 | SD.begin(); 538 | 539 | Serial.print(F("done.\n")); 540 | is_inited = true; 541 | 542 | } 543 | 544 | char str[MAX_STR_LEN]; 545 | void loop() { 546 | 547 | if (!is_inited) 548 | return; 549 | 550 | int choice = askChoice(); 551 | switch (choice) { 552 | case CHOICE_LOG_ANALOG_DATA: 553 | log_analog_data(); 554 | break; 555 | case CHOICE_LOCATE_ROWID: 556 | case CHOICE_LOCATE_BIN_SRCH: 557 | locate_records(choice); 558 | break; 559 | case CHOICE_RECOVER_DB: 560 | recover_db(); 561 | break; 562 | case CHOICE_LIST_FOLDER: 563 | case CHOICE_RENAME_FILE: 564 | case CHOICE_DELETE_FILE: 565 | fs::FS *fs; 566 | displayPrompt("path: "); 567 | input_string(str, MAX_STR_LEN); 568 | if (str[0] != 0) { 569 | int fs_prefix_len = 0; 570 | fs = ascertainFS(str, &fs_prefix_len); 571 | if (fs != NULL) { 572 | switch (choice) { 573 | case CHOICE_LIST_FOLDER: 574 | listDir(*fs, str + fs_prefix_len); 575 | break; 576 | case CHOICE_RENAME_FILE: 577 | char str1[MAX_FILE_NAME_LEN]; 578 | displayPrompt("path to rename as: "); 579 | input_string(str1, MAX_STR_LEN); 580 | if (str1[0] != 0) 581 | renameFile(*fs, str + fs_prefix_len, str1 + fs_prefix_len); 582 | break; 583 | case CHOICE_DELETE_FILE: 584 | deleteFile(*fs, str + fs_prefix_len); 585 | break; 586 | } 587 | } 588 | } 589 | break; 590 | case CHOICE_SHOW_FREE_MEM: 591 | Serial.printf("\nHeap size: %d\n", ESP.getHeapSize()); 592 | Serial.printf("Free Heap: %d\n", esp_get_free_heap_size()); 593 | Serial.printf("Min Free Heap: %d\n", esp_get_minimum_free_heap_size()); 594 | Serial.printf("Largest Free block: %d\n", heap_caps_get_largest_free_block(MALLOC_CAP_8BIT)); 595 | break; 596 | default: 597 | Serial.println(F("Invalid choice. Try again.")); 598 | } 599 | 600 | } -------------------------------------------------------------------------------- /examples/ESP8266_Console/ESP8266_Console.ino: -------------------------------------------------------------------------------- 1 | /* 2 | This example demonstrates how the Sqlite Micro Logger library 3 | can be used to write Analog data into Sqlite database. 4 | Works on any Arduino compatible microcontroller with SD Card attachment 5 | having 2kb RAM or more (such as Arduino Uno). 6 | 7 | How Sqlite Micro Logger works: 8 | https://github.com/siara-cc/sqlite_micro_logger_c 9 | 10 | Copyright @ 2019 Arundale Ramanathan, Siara Logics (cc) 11 | 12 | Licensed under the Apache License, Version 2.0 (the "License"); 13 | you may not use this file except in compliance with the License. 14 | You may obtain a copy of the License at 15 | 16 | http://www.apache.org/licenses/LICENSE-2.0 17 | 18 | Unless required by applicable law or agreed to in writing, software 19 | distributed under the License is distributed on an "AS IS" BASIS, 20 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 21 | See the License for the specific language governing permissions and 22 | limitations under the License. 23 | */ 24 | #include "ulog_sqlite.h" 25 | #include 26 | #include 27 | #include 28 | 29 | #define MAX_FILE_NAME_LEN 100 30 | #define MAX_STR_LEN 500 31 | 32 | // If you would like to read DBs created with this repo 33 | // with page size more than 4096, change here, 34 | // but ensure as much SRAM can be allocated. 35 | #define BUF_SIZE 4096 36 | byte buf[BUF_SIZE]; 37 | char filename[MAX_FILE_NAME_LEN]; 38 | extern const char sqlite_sig[]; 39 | 40 | File myFile; 41 | 42 | int32_t read_fn_wctx(struct dblog_write_context *ctx, void *buf, uint32_t pos, size_t len) { 43 | myFile.seek(pos); 44 | size_t ret = myFile.read((byte *)buf, len); 45 | if (ret != len) 46 | return DBLOG_RES_READ_ERR; 47 | return ret; 48 | } 49 | 50 | int32_t read_fn_rctx(struct dblog_read_context *ctx, void *buf, uint32_t pos, size_t len) { 51 | myFile.seek(pos); 52 | size_t ret = myFile.read((byte *)buf, len); 53 | if (ret != len) 54 | return DBLOG_RES_READ_ERR; 55 | return ret; 56 | } 57 | 58 | int32_t write_fn(struct dblog_write_context *ctx, void *buf, uint32_t pos, size_t len) { 59 | myFile.seek(pos); 60 | size_t ret = myFile.write((byte *)buf, len); 61 | if (ret != len) 62 | return DBLOG_RES_ERR; 63 | myFile.flush(); 64 | return ret; 65 | } 66 | 67 | int flush_fn(struct dblog_write_context *ctx) { 68 | //myFile.flush(); // Anyway being flushed after write 69 | return DBLOG_RES_OK; 70 | } 71 | 72 | void listDir(int fs_type, const char * dirname) { 73 | Serial.print(F("Listing directory: ")); 74 | Serial.println(dirname); 75 | if (fs_type == 1) { 76 | Dir dir = SPIFFS.openDir("/"); 77 | while (dir.next()) { 78 | Serial.print(dir.fileName()); 79 | File f = dir.openFile("r"); 80 | Serial.print("\t"); 81 | Serial.println(f.size()); 82 | } 83 | return; 84 | } 85 | File root = SD.open("/"); 86 | if (!root){ 87 | Serial.println(F("Failed to open directory")); 88 | return; 89 | } 90 | if (!root.isDirectory()){ 91 | Serial.println("Not a directory"); 92 | return; 93 | } 94 | File file = root.openNextFile(); 95 | while (file) { 96 | if (file.isDirectory()) { 97 | Serial.print(" Dir : "); 98 | Serial.println(file.name()); 99 | } else { 100 | Serial.print(" File: "); 101 | Serial.print(file.name()); 102 | Serial.print(" Size: "); 103 | Serial.println(file.size()); 104 | } 105 | file = root.openNextFile(); 106 | } 107 | } 108 | 109 | void renameFile(int fs_type, const char *path1, const char *path2) { 110 | Serial.printf("Renaming file %s to %s\n", path1, path2); 111 | int res = 0; 112 | if (fs_type == 1) 113 | res = SPIFFS.rename(path1, path2); 114 | if (res) { 115 | Serial.println(F("File renamed")); 116 | } else { 117 | Serial.println(F("Rename failed")); 118 | } 119 | } 120 | 121 | void deleteFile(int fs_type, const char *path) { 122 | Serial.printf("Deleting file: %s\n", path); 123 | int res = 0; 124 | if (fs_type == 1) 125 | res = SPIFFS.remove(path); 126 | if (res) { 127 | Serial.println(F("File deleted")); 128 | } else { 129 | Serial.println(F("Delete failed")); 130 | } 131 | } 132 | 133 | enum { CHOICE_LOG_ANALOG_DATA = 1, CHOICE_LOCATE_ROWID, CHOICE_LOCATE_BIN_SRCH, CHOICE_RECOVER_DB, 134 | CHOICE_LIST_FOLDER, CHOICE_SHOW_FREE_MEM, CHOICE_RENAME_FILE, CHOICE_DELETE_FILE}; 135 | 136 | int askChoice() { 137 | Serial.println(); 138 | Serial.println(F("Welcome to SQLite Micro Logger console!!")); 139 | Serial.println(F("----------------------------------------")); 140 | Serial.println(); 141 | Serial.println(F("1. Log Analog data")); 142 | Serial.println(F("2. Locate record using RowID")); 143 | Serial.println(F("3. Locate record using Binary Search")); 144 | Serial.println(F("4. Recover database")); 145 | Serial.println(F("5. List folder contents (SPIFFS only)")); 146 | Serial.println(F("6. Show free memory")); 147 | Serial.println(F("7. Rename file (SPIFFS only)")); 148 | Serial.println(F("8. Delete file (SPIFFS only)")); 149 | Serial.println(); 150 | Serial.print(F("Enter choice: ")); 151 | return input_num(); 152 | } 153 | 154 | void displayPrompt(const char *title) { 155 | Serial.println(F("(prefix /spiffs/ or /sd/ for")); 156 | Serial.println(F(" SPIFFS or SD_SPI respectively)")); 157 | Serial.print(F("Enter ")); 158 | Serial.println(title); 159 | } 160 | 161 | const char *prefixSPIFFS = "/spiffs/"; 162 | const char *prefixSD_SPI = "/sd/"; 163 | int ascertainFS(const char *str, int *prefix_len) { 164 | if (strstr(str, prefixSPIFFS) == str) { 165 | *prefix_len = strlen(prefixSPIFFS) - 1; 166 | return 1; 167 | } 168 | if (strstr(str, prefixSD_SPI) == str) { 169 | *prefix_len = strlen(prefixSD_SPI) - 1; 170 | return 2; 171 | } 172 | return 0; 173 | } 174 | 175 | void print_error(int res) { 176 | Serial.print(F("Err:")); 177 | Serial.print(res); 178 | Serial.print(F("\n")); 179 | } 180 | 181 | int input_string(char *str, int max_len) { 182 | int ctr = 0; 183 | str[ctr] = 0; 184 | while (str[ctr] != '\n') { 185 | if (Serial.available()) { 186 | str[ctr] = Serial.read(); 187 | if (str[ctr] >= ' ' && str[ctr] <= '~') 188 | ctr++; 189 | if (ctr >= max_len) 190 | break; 191 | } 192 | } 193 | str[ctr] = 0; 194 | Serial.print(str); 195 | Serial.print(F("\n")); 196 | return ctr; 197 | } 198 | 199 | int input_num() { 200 | char in[20]; 201 | int ctr = 0; 202 | in[ctr] = 0; 203 | while (in[ctr] != '\n') { 204 | if (Serial.available()) { 205 | in[ctr] = Serial.read(); 206 | if (in[ctr] >= '0' && in[ctr] <= '9') 207 | ctr++; 208 | if (ctr >= sizeof(in)) 209 | break; 210 | } 211 | } 212 | in[ctr] = 0; 213 | int ret = atoi(in); 214 | Serial.print(ret); 215 | Serial.print(F("\n")); 216 | return ret; 217 | } 218 | 219 | int16_t read_int16(const byte *ptr) { 220 | return (*ptr << 8) | ptr[1]; 221 | } 222 | 223 | int32_t read_int32(const byte *ptr) { 224 | int32_t ret; 225 | ret = ((int32_t)*ptr++) << 24; 226 | ret |= ((int32_t)*ptr++) << 16; 227 | ret |= ((int32_t)*ptr++) << 8; 228 | ret |= *ptr; 229 | return ret; 230 | } 231 | 232 | int64_t read_int64(const byte *ptr) { 233 | int64_t ret; 234 | ret = ((int64_t)*ptr++) << 56; 235 | ret |= ((int64_t)*ptr++) << 48; 236 | ret |= ((int64_t)*ptr++) << 40; 237 | ret |= ((int64_t)*ptr++) << 32; 238 | ret |= ((int64_t)*ptr++) << 24; 239 | ret |= ((int64_t)*ptr++) << 16; 240 | ret |= ((int64_t)*ptr++) << 8; 241 | ret |= *ptr; 242 | return ret; 243 | } 244 | 245 | int pow10(int8_t len) { 246 | return (len == 3 ? 1000 : (len == 2 ? 100 : (len == 1 ? 10 : 1))); 247 | } 248 | 249 | void set_ts_part(char *s, int val, int8_t len) { 250 | while (len--) { 251 | *s++ = '0' + val / pow10(len); 252 | val %= pow10(len); 253 | } 254 | } 255 | 256 | int get_ts_part(char *s, int8_t len) { 257 | int i = 0; 258 | while (len--) 259 | i += ((*s++ - '0') * pow10(len)); 260 | return i; 261 | } 262 | 263 | int update_ts_part(char *ptr, int8_t len, int limit, int ovflw) { 264 | int8_t is_one_based = (limit == 1000 || limit == 60 || limit == 24) ? 0 : 1; 265 | int part = get_ts_part(ptr, len) + ovflw - is_one_based; 266 | ovflw = part / limit; 267 | part %= limit; 268 | set_ts_part(ptr, part + is_one_based, len); 269 | return ovflw; 270 | } 271 | 272 | // 012345678901234567890 273 | // YYYY-MM-DD HH:MM:SS.SSS 274 | void update_ts(char *ts, int diff) { 275 | int ovflw = update_ts_part(ts + 20, 3, 1000, diff); // ms 276 | if (ovflw) { 277 | ovflw = update_ts_part(ts + 17, 2, 60, ovflw); // seconds 278 | if (ovflw) { 279 | ovflw = update_ts_part(ts + 14, 2, 60, ovflw); // minutes 280 | if (ovflw) { 281 | ovflw = update_ts_part(ts + 11, 2, 24, ovflw); // hours 282 | if (ovflw) { 283 | int8_t month = get_ts_part(ts + 5, 2); 284 | int year = get_ts_part(ts, 4); 285 | int8_t limit = (month == 2 ? (year % 4 ? 28 : 29) : 286 | (month == 4 || month == 6 || month == 9 || month == 11 ? 30 : 31)); 287 | ovflw = update_ts_part(ts + 8, 2, limit, ovflw); // day 288 | if (ovflw) { 289 | ovflw = update_ts_part(ts + 5, 2, 12, ovflw); // month 290 | if (ovflw) 291 | set_ts_part(ts, year + ovflw, 4); // year 292 | } 293 | } 294 | } 295 | } 296 | } 297 | } 298 | 299 | void display_row(struct dblog_read_context *ctx) { 300 | int i = 0; 301 | do { 302 | uint32_t col_type; 303 | const byte *col_val = (const byte *) dblog_read_col_val(ctx, i, &col_type); 304 | if (!col_val) { 305 | if (i == 0) 306 | Serial.print(F("Error reading value\n")); 307 | Serial.print(F("\n")); 308 | return; 309 | } 310 | if (i) 311 | Serial.print(F("|")); 312 | switch (col_type) { 313 | case 0: 314 | Serial.print(F("null")); 315 | break; 316 | case 1: 317 | Serial.print(*((int8_t *)col_val)); 318 | break; 319 | case 2: { 320 | int16_t ival = read_int16(col_val); 321 | Serial.print(ival); 322 | } 323 | break; 324 | case 4: { 325 | int32_t ival = read_int32(col_val); 326 | Serial.print(ival); 327 | break; 328 | } 329 | // Arduino Serial.print not capable of printing 330 | // int64_t and double. Need to implement manually 331 | case 6: // int64_t 332 | case 7: // double 333 | Serial.print(F("todo")); 334 | break; 335 | default: { 336 | uint32_t col_len = dblog_derive_data_len(col_type); 337 | for (int j = 0; j < col_len; j++) { 338 | if (col_type % 2) 339 | Serial.print((char)col_val[j]); 340 | else { 341 | Serial.print((int)col_val[j]); 342 | Serial.print(F(" ")); 343 | } 344 | } 345 | } 346 | } 347 | } while (++i); 348 | } 349 | 350 | void input_db_name() { 351 | displayPrompt("DB name: "); 352 | input_string(filename, MAX_FILE_NAME_LEN); 353 | } 354 | 355 | int input_ts(char *datetime) { 356 | Serial.print(F("\nEnter timestamp (YYYY-MM-DD HH:MM:SS.SSS): ")); 357 | return input_string(datetime, 24); 358 | } 359 | 360 | void log_analog_data() { 361 | int num_entries; 362 | int dly; 363 | input_db_name(); 364 | Serial.print(F("\nRecord count: ")); 365 | num_entries = input_num(); 366 | char ts[24]; 367 | if (input_ts(ts) < 23) { 368 | Serial.print(F("Input full timestamp\n")); 369 | return; 370 | } 371 | Serial.print(F("\nDelay(ms): ")); 372 | dly = input_num(); 373 | 374 | if (strstr(filename, prefixSPIFFS) == filename) 375 | myFile = SPIFFS.open(filename + 7, "w+b"); 376 | else 377 | myFile = SD.open(filename + 3, FILE_WRITE); 378 | 379 | // if the file opened okay, write to it: 380 | if (myFile) { 381 | unsigned long start = millis(); 382 | unsigned long last_ms = start; 383 | struct dblog_write_context ctx; 384 | ctx.buf = buf; 385 | ctx.col_count = 2; 386 | ctx.page_resv_bytes = 0; 387 | ctx.page_size_exp = 12; 388 | ctx.max_pages_exp = 0; 389 | ctx.read_fn = read_fn_wctx; 390 | ctx.flush_fn = flush_fn; 391 | ctx.write_fn = write_fn; 392 | int val; 393 | uint8_t types[] = {DBLOG_TYPE_TEXT, DBLOG_TYPE_INT}; 394 | void *values[] = {ts, &val}; 395 | uint16_t lengths[] = {23, sizeof(val)}; 396 | int res = dblog_write_init(&ctx); 397 | if (!res) { 398 | while (num_entries--) { 399 | val = analogRead(A0); 400 | res = dblog_append_row_with_values(&ctx, types, (const void **) values, lengths); 401 | if (res) 402 | break; 403 | update_ts(ts, (int) (millis() - last_ms)); 404 | last_ms = millis(); 405 | delay(dly); 406 | } 407 | } 408 | Serial.print(F("\nLogging completed. Finalizing...\n")); 409 | if (!res) 410 | res = dblog_finalize(&ctx); 411 | myFile.close(); 412 | if (res) 413 | print_error(res); 414 | else { 415 | Serial.print(F("\nDone. Elapsed time (ms): ")); 416 | Serial.print((millis() - start)); 417 | Serial.print("\n"); 418 | } 419 | } else { 420 | // if the file didn't open, print an error: 421 | Serial.print(F("Open Error\n")); 422 | } 423 | } 424 | 425 | void locate_records(int8_t choice) { 426 | int num_entries; 427 | int dly; 428 | input_db_name(); 429 | struct dblog_read_context rctx; 430 | rctx.page_size_exp = 9; 431 | rctx.read_fn = read_fn_rctx; 432 | if (strstr(filename, prefixSPIFFS) == filename) 433 | myFile = SPIFFS.open(filename + 7, "r+b"); 434 | else 435 | myFile = SD.open(filename + 3, FILE_READ); 436 | if (myFile) { 437 | rctx.buf = buf; 438 | int res = dblog_read_init(&rctx); 439 | if (res) { 440 | print_error(res); 441 | myFile.close(); 442 | return; 443 | } 444 | Serial.print(F("Page size:")); 445 | Serial.print((int32_t) 1 << rctx.page_size_exp); 446 | Serial.print(F("\nLast data page:")); 447 | Serial.print(rctx.last_leaf_page); 448 | Serial.print(F("\n")); 449 | if (memcmp(buf, sqlite_sig, 16) || buf[68] != 0xA5) { 450 | Serial.print(F("Invalid DB. Try recovery.\n")); 451 | myFile.close(); 452 | return; 453 | } 454 | if (BUF_SIZE < (int32_t) 1 << rctx.page_size_exp) { 455 | Serial.print(F("Buffer size less than Page size. Try increasing if enough SRAM\n")); 456 | myFile.close(); 457 | return; 458 | } 459 | Serial.print(F("\nFirst record:\n")); 460 | display_row(&rctx); 461 | uint32_t rowid; 462 | char srch_datetime[24]; // YYYY-MM-DD HH:MM:SS.SSS 463 | int8_t dt_len; 464 | if (choice == 2) { 465 | Serial.print(F("\nEnter RowID: ")); 466 | rowid = input_num(); 467 | } else 468 | dt_len = input_ts(srch_datetime); 469 | Serial.print(F("No. of records to display: ")); 470 | num_entries = input_num(); 471 | unsigned long start = millis(); 472 | if (choice == 2) 473 | res = dblog_srch_row_by_id(&rctx, rowid); 474 | else 475 | res = dblog_bin_srch_row_by_val(&rctx, 0, DBLOG_TYPE_TEXT, srch_datetime, dt_len, 0); 476 | if (res == DBLOG_RES_NOT_FOUND) 477 | Serial.print(F("Not Found\n")); 478 | else if (res == 0) { 479 | Serial.print(F("\nTime taken (ms): ")); 480 | Serial.print((millis() - start)); 481 | Serial.print("\n\n"); 482 | do { 483 | display_row(&rctx); 484 | } while (--num_entries && !dblog_read_next_row(&rctx)); 485 | } else 486 | print_error(res); 487 | myFile.close(); 488 | } else { 489 | // if the file didn't open, print an error: 490 | Serial.print(F("Open Error\n")); 491 | } 492 | } 493 | 494 | void recover_db() { 495 | struct dblog_write_context ctx; 496 | ctx.buf = buf; 497 | ctx.read_fn = read_fn_wctx; 498 | ctx.write_fn = write_fn; 499 | ctx.flush_fn = flush_fn; 500 | input_db_name(); 501 | if (strstr(filename, prefixSPIFFS) == filename) 502 | myFile = SPIFFS.open(filename + 7, "r+b"); 503 | else 504 | myFile = SD.open(filename + 3, FILE_READ); 505 | if (!myFile) { 506 | print_error(0); 507 | return; 508 | } 509 | int32_t page_size = dblog_read_page_size(&ctx); 510 | if (page_size < 512) { 511 | Serial.print(F("Error reading page size\n")); 512 | myFile.close(); 513 | return; 514 | } 515 | if (dblog_recover(&ctx)) { 516 | Serial.print(F("Error during recover\n")); 517 | myFile.close(); 518 | return; 519 | } 520 | myFile.close(); 521 | } 522 | 523 | bool is_inited = false; 524 | void setup() { 525 | 526 | Serial.begin(74880); 527 | while (!Serial) { 528 | } 529 | 530 | Serial.print(F("InitSD..\n")); 531 | SPIFFS.begin(); 532 | SPI.begin(); 533 | if (!SD.begin(4)) { 534 | Serial.print(F("Failed to mount SD FS\n")); 535 | return; 536 | } 537 | 538 | Serial.print(F("done.\n")); 539 | is_inited = true; 540 | 541 | } 542 | 543 | char str[MAX_STR_LEN]; 544 | void loop() { 545 | 546 | if (!is_inited) 547 | return; 548 | 549 | int choice = askChoice(); 550 | switch (choice) { 551 | case CHOICE_LOG_ANALOG_DATA: 552 | log_analog_data(); 553 | break; 554 | case CHOICE_LOCATE_ROWID: 555 | case CHOICE_LOCATE_BIN_SRCH: 556 | locate_records(choice); 557 | break; 558 | case CHOICE_RECOVER_DB: 559 | recover_db(); 560 | break; 561 | case CHOICE_LIST_FOLDER: 562 | case CHOICE_RENAME_FILE: 563 | case CHOICE_DELETE_FILE: 564 | displayPrompt("path: "); 565 | input_string(str, MAX_STR_LEN); 566 | if (str[0] != 0) { 567 | int fs_prefix_len = 0; 568 | int fs_type = ascertainFS(str, &fs_prefix_len); 569 | if (fs_type != 0) { 570 | switch (choice) { 571 | case CHOICE_LIST_FOLDER: 572 | listDir(fs_type, str + fs_prefix_len); 573 | break; 574 | case CHOICE_RENAME_FILE: 575 | char str1[MAX_FILE_NAME_LEN]; 576 | displayPrompt("path to rename as: "); 577 | input_string(str1, MAX_STR_LEN); 578 | if (str1[0] != 0) 579 | renameFile(fs_type, str + fs_prefix_len, str1 + fs_prefix_len); 580 | break; 581 | case CHOICE_DELETE_FILE: 582 | deleteFile(fs_type, str + fs_prefix_len); 583 | break; 584 | } 585 | } 586 | } 587 | break; 588 | case CHOICE_SHOW_FREE_MEM: 589 | //Serial.printf("\nHeap size: %d\n", ESP.getHeapSize()); 590 | Serial.printf("Free Heap: %d\n", ESP.getFreeHeap()); 591 | //Serial.printf("Min Free Heap: %d\n", ESP.getMinFreeBlockSize()); 592 | Serial.printf("Fragmentation: %d\n", ESP.getHeapFragmentation()); 593 | Serial.printf("Max Free Heap: %d\n", ESP.getMaxFreeBlockSize()); 594 | break; 595 | default: 596 | Serial.println(F("Invalid choice. Try again.")); 597 | } 598 | 599 | } -------------------------------------------------------------------------------- /examples/Uno_and_above/Uno_and_above.ino: -------------------------------------------------------------------------------- 1 | /* 2 | This example demonstrates how the Sqlite Micro Logger library 3 | can be used to write Analog data into Sqlite database. 4 | Works on any Arduino compatible microcontroller with SD Card attachment 5 | having 2kb RAM or more (such as Arduino Uno). 6 | 7 | How Sqlite Micro Logger works: 8 | https://github.com/siara-cc/sqlite_micro_logger_c 9 | 10 | Copyright @ 2019 Arundale Ramanathan, Siara Logics (cc) 11 | 12 | Licensed under the Apache License, Version 2.0 (the "License"); 13 | you may not use this file except in compliance with the License. 14 | You may obtain a copy of the License at 15 | 16 | http://www.apache.org/licenses/LICENSE-2.0 17 | 18 | Unless required by applicable law or agreed to in writing, software 19 | distributed under the License is distributed on an "AS IS" BASIS, 20 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 21 | See the License for the specific language governing permissions and 22 | limitations under the License. 23 | */ 24 | #include "ulog_sqlite.h" 25 | #include 26 | #include 27 | 28 | // Set the CS Pin connected to the MicroSD 29 | // 8 on most shields such as Sparkfun Micro SD Shield 30 | // 4 on WeMos D1 Mini 31 | #define SD_CS_PIN 8 32 | 33 | // If you would like to read DBs created with this repo 34 | // with page size more than 512, change here, 35 | // but ensure as much SRAM can be allocated. 36 | // Please Arduino Uno cannot support page size > 512. 37 | #define BUF_SIZE 512 38 | byte buf[BUF_SIZE]; 39 | char filename[13]; 40 | extern const char sqlite_sig[]; 41 | 42 | File myFile; 43 | 44 | int32_t read_fn_wctx(struct dblog_write_context *ctx, void *buf, uint32_t pos, size_t len) { 45 | myFile.seek(pos); 46 | size_t ret = myFile.read((byte *)buf, len); 47 | if (ret != len) 48 | return DBLOG_RES_READ_ERR; 49 | return ret; 50 | } 51 | 52 | int32_t read_fn_rctx(struct dblog_read_context *ctx, void *buf, uint32_t pos, size_t len) { 53 | myFile.seek(pos); 54 | size_t ret = myFile.read((byte *)buf, len); 55 | if (ret != len) 56 | return DBLOG_RES_READ_ERR; 57 | return ret; 58 | } 59 | 60 | int32_t write_fn(struct dblog_write_context *ctx, void *buf, uint32_t pos, size_t len) { 61 | myFile.seek(pos); 62 | size_t ret = myFile.write((byte *)buf, len); 63 | if (ret != len) 64 | return DBLOG_RES_ERR; 65 | myFile.flush(); 66 | return ret; 67 | } 68 | 69 | int flush_fn(struct dblog_write_context *ctx) { 70 | //myFile.flush(); // Anyway being flushed after write 71 | return DBLOG_RES_OK; 72 | } 73 | 74 | void print_error(int res) { 75 | Serial.print(F("Err:")); 76 | Serial.print(res); 77 | Serial.print(F("\n")); 78 | } 79 | 80 | int input_string(char *str, int max_len) { 81 | int ctr = 0; 82 | str[ctr] = 0; 83 | while (str[ctr] != '\n') { 84 | if (Serial.available()) { 85 | str[ctr] = Serial.read(); 86 | if (str[ctr] >= ' ' && str[ctr] <= '~') 87 | ctr++; 88 | if (ctr >= max_len) 89 | break; 90 | } 91 | } 92 | str[ctr] = 0; 93 | Serial.print(str); 94 | Serial.print(F("\n")); 95 | return ctr; 96 | } 97 | 98 | int32_t input_num() { 99 | char in[20]; 100 | int ctr = 0; 101 | in[ctr] = 0; 102 | while (in[ctr] != '\n') { 103 | if (Serial.available()) { 104 | in[ctr] = Serial.read(); 105 | if (in[ctr] >= '0' && in[ctr] <= '9') 106 | ctr++; 107 | if (ctr >= sizeof(in)) 108 | break; 109 | } 110 | } 111 | in[ctr] = 0; 112 | int32_t ret = atol(in); 113 | Serial.print(ret); 114 | Serial.print(F("\n")); 115 | return ret; 116 | } 117 | 118 | int16_t read_int16(const byte *ptr) { 119 | return (*ptr << 8) | ptr[1]; 120 | } 121 | 122 | int32_t read_int32(const byte *ptr) { 123 | int32_t ret; 124 | ret = ((int32_t)*ptr++) << 24; 125 | ret |= ((int32_t)*ptr++) << 16; 126 | ret |= ((int32_t)*ptr++) << 8; 127 | ret |= *ptr; 128 | return ret; 129 | } 130 | 131 | int64_t read_int64(const byte *ptr) { 132 | int64_t ret; 133 | ret = ((int64_t)*ptr++) << 56; 134 | ret |= ((int64_t)*ptr++) << 48; 135 | ret |= ((int64_t)*ptr++) << 40; 136 | ret |= ((int64_t)*ptr++) << 32; 137 | ret |= ((int64_t)*ptr++) << 24; 138 | ret |= ((int64_t)*ptr++) << 16; 139 | ret |= ((int64_t)*ptr++) << 8; 140 | ret |= *ptr; 141 | return ret; 142 | } 143 | 144 | int pow10(int8_t len) { 145 | return (len == 3 ? 1000 : (len == 2 ? 100 : (len == 1 ? 10 : 1))); 146 | } 147 | 148 | void set_ts_part(char *s, int val, int8_t len) { 149 | while (len--) { 150 | *s++ = '0' + val / pow10(len); 151 | val %= pow10(len); 152 | } 153 | } 154 | 155 | int get_ts_part(char *s, int8_t len) { 156 | int i = 0; 157 | while (len--) 158 | i += ((*s++ - '0') * pow10(len)); 159 | return i; 160 | } 161 | 162 | int update_ts_part(char *ptr, int8_t len, int limit, int ovflw) { 163 | int8_t is_one_based = (limit == 1000 || limit == 60 || limit == 24) ? 0 : 1; 164 | int part = get_ts_part(ptr, len) + ovflw - is_one_based; 165 | ovflw = part / limit; 166 | part %= limit; 167 | set_ts_part(ptr, part + is_one_based, len); 168 | return ovflw; 169 | } 170 | 171 | // 012345678901234567890 172 | // YYYY-MM-DD HH:MM:SS.SSS 173 | void update_ts(char *ts, int diff) { 174 | int ovflw = update_ts_part(ts + 20, 3, 1000, diff); // ms 175 | if (ovflw) { 176 | ovflw = update_ts_part(ts + 17, 2, 60, ovflw); // seconds 177 | if (ovflw) { 178 | ovflw = update_ts_part(ts + 14, 2, 60, ovflw); // minutes 179 | if (ovflw) { 180 | ovflw = update_ts_part(ts + 11, 2, 24, ovflw); // hours 181 | if (ovflw) { 182 | int8_t month = get_ts_part(ts + 5, 2); 183 | int year = get_ts_part(ts, 4); 184 | int8_t limit = (month == 2 ? (year % 4 ? 28 : 29) : 185 | (month == 4 || month == 6 || month == 9 || month == 11 ? 30 : 31)); 186 | ovflw = update_ts_part(ts + 8, 2, limit, ovflw); // day 187 | if (ovflw) { 188 | ovflw = update_ts_part(ts + 5, 2, 12, ovflw); // month 189 | if (ovflw) 190 | set_ts_part(ts, year + ovflw, 4); // year 191 | } 192 | } 193 | } 194 | } 195 | } 196 | } 197 | 198 | void display_row(struct dblog_read_context *ctx) { 199 | int i = 0; 200 | do { 201 | uint32_t col_type; 202 | const byte *col_val = (const byte *) dblog_read_col_val(ctx, i, &col_type); 203 | if (!col_val) { 204 | if (i == 0) 205 | Serial.print(F("Error reading value\n")); 206 | Serial.print(F("\n")); 207 | return; 208 | } 209 | if (i) 210 | Serial.print(F("|")); 211 | switch (col_type) { 212 | case 0: 213 | Serial.print(F("null")); 214 | break; 215 | case 1: 216 | Serial.print(*((int8_t *)col_val)); 217 | break; 218 | case 2: { 219 | int16_t ival = read_int16(col_val); 220 | Serial.print(ival); 221 | } 222 | break; 223 | case 4: { 224 | int32_t ival = read_int32(col_val); 225 | Serial.print(ival); 226 | break; 227 | } 228 | // Arduino Serial.print not capable of printing 229 | // int64_t and double. Need to implement manually 230 | case 6: // int64_t 231 | case 7: // double 232 | Serial.print(F("todo")); 233 | break; 234 | default: { 235 | uint32_t col_len = dblog_derive_data_len(col_type); 236 | for (int j = 0; j < col_len; j++) { 237 | if (col_type % 2) 238 | Serial.print((char)col_val[j]); 239 | else { 240 | Serial.print((int)col_val[j]); 241 | Serial.print(F(" ")); 242 | } 243 | } 244 | } 245 | } 246 | } while (++i); 247 | } 248 | 249 | void input_db_name() { 250 | Serial.print(F("DB name (max 8.3): ")); 251 | input_string(filename, sizeof(filename)); 252 | } 253 | 254 | int input_ts(char *datetime) { 255 | Serial.print(F("\nEnter timestamp (YYYY-MM-DD HH:MM:SS.SSS): ")); 256 | return input_string(datetime, 24); 257 | } 258 | 259 | void log_analog_data() { 260 | int32_t num_entries; 261 | int dly; 262 | input_db_name(); 263 | Serial.print(F("\nRecord count (1 to 32767 on UNO): ")); 264 | num_entries = input_num(); 265 | Serial.print(F("\nStarting analog pin (14=A0 on Uno, 17=A0 on ESP8266): ")); 266 | int8_t analog_pin_start = input_num(); 267 | Serial.print(F("\nNo. of pins: ")); 268 | int8_t analog_pin_count = input_num(); 269 | char ts[24]; 270 | if (input_ts(ts) < 23) { 271 | Serial.print(F("Input full timestamp\n")); 272 | return; 273 | } 274 | Serial.print(F("\nDelay(ms): ")); 275 | dly = input_num(); 276 | 277 | SD.remove(filename); 278 | myFile = SD.open(filename, O_READ | O_WRITE | O_CREAT); 279 | 280 | // if the file opened okay, write to it: 281 | if (myFile) { 282 | unsigned long start = millis(); 283 | unsigned long last_ms = start; 284 | struct dblog_write_context ctx; 285 | ctx.buf = buf; 286 | ctx.col_count = analog_pin_count + 1; 287 | ctx.page_resv_bytes = 0; 288 | ctx.page_size_exp = 9; 289 | ctx.max_pages_exp = 0; 290 | ctx.read_fn = read_fn_wctx; 291 | ctx.flush_fn = flush_fn; 292 | ctx.write_fn = write_fn; 293 | int res = dblog_write_init(&ctx); 294 | if (!res) { 295 | while (num_entries--) { 296 | res = dblog_set_col_val(&ctx, 0, DBLOG_TYPE_TEXT, ts, 23); 297 | if (res) 298 | break; 299 | update_ts(ts, (int) (millis() - last_ms)); 300 | last_ms = millis(); 301 | for (int8_t i = 0; i < analog_pin_count; i++) { 302 | int val = analogRead(analog_pin_start + i); 303 | res = dblog_set_col_val(&ctx, i + 1, DBLOG_TYPE_INT, &val, sizeof(int)); 304 | if (res) 305 | break; 306 | } 307 | if (num_entries) { 308 | res = dblog_append_empty_row(&ctx); 309 | if (res) 310 | break; 311 | delay(dly); 312 | } 313 | } 314 | } 315 | if (!res) 316 | res = dblog_finalize(&ctx); 317 | myFile.close(); 318 | if (res) 319 | print_error(res); 320 | else { 321 | Serial.print(F("\nDone. Elapsed time (ms): ")); 322 | Serial.print((millis() - start)); 323 | Serial.print("\n"); 324 | } 325 | } else { 326 | // if the file didn't open, print an error: 327 | Serial.print(F("Open Error\n")); 328 | } 329 | } 330 | 331 | void locate_records(int8_t choice) { 332 | int num_entries; 333 | int dly; 334 | input_db_name(); 335 | struct dblog_read_context rctx; 336 | rctx.page_size_exp = 9; 337 | rctx.read_fn = read_fn_rctx; 338 | myFile = SD.open(filename, FILE_READ); 339 | if (myFile) { 340 | Serial.print(F("Size:")); 341 | Serial.print(myFile.size()); 342 | Serial.print(F("\n")); 343 | rctx.buf = buf; 344 | int res = dblog_read_init(&rctx); 345 | if (res) { 346 | print_error(res); 347 | myFile.close(); 348 | return; 349 | } 350 | Serial.print(F("Page size:")); 351 | Serial.print((int32_t) 1 << rctx.page_size_exp); 352 | Serial.print(F("\nLast data page:")); 353 | Serial.print(rctx.last_leaf_page); 354 | Serial.print(F("\n")); 355 | if (memcmp(buf, sqlite_sig, 16) || buf[68] != 0xA5) { 356 | Serial.print(F("Invalid DB. Try recovery.\n")); 357 | myFile.close(); 358 | return; 359 | } 360 | if (BUF_SIZE < (int32_t) 1 << rctx.page_size_exp) { 361 | Serial.print(F("Buffer size less than Page size. Try increasing if enough SRAM\n")); 362 | myFile.close(); 363 | return; 364 | } 365 | Serial.print(F("\nFirst record:\n")); 366 | display_row(&rctx); 367 | uint32_t rowid; 368 | char srch_datetime[24]; // YYYY-MM-DD HH:MM:SS.SSS 369 | int8_t dt_len; 370 | if (choice == 2) { 371 | Serial.print(F("\nEnter RowID (1 to 32767 on UNO): ")); 372 | rowid = input_num(); 373 | } else 374 | dt_len = input_ts(srch_datetime); 375 | Serial.print(F("No. of records to display: ")); 376 | num_entries = input_num(); 377 | unsigned long start = millis(); 378 | if (choice == 2) 379 | res = dblog_srch_row_by_id(&rctx, rowid); 380 | else 381 | res = dblog_bin_srch_row_by_val(&rctx, 0, DBLOG_TYPE_TEXT, srch_datetime, dt_len, 0); 382 | if (res == DBLOG_RES_NOT_FOUND) 383 | Serial.print(F("Not Found\n")); 384 | else if (res == 0) { 385 | Serial.print(F("\nTime taken (ms): ")); 386 | Serial.print((millis() - start)); 387 | Serial.print("\n\n"); 388 | do { 389 | display_row(&rctx); 390 | } while (--num_entries && !dblog_read_next_row(&rctx)); 391 | } else 392 | print_error(res); 393 | myFile.close(); 394 | } else { 395 | // if the file didn't open, print an error: 396 | Serial.print(F("Open Error\n")); 397 | } 398 | } 399 | 400 | void recover_db() { 401 | struct dblog_write_context ctx; 402 | ctx.buf = buf; 403 | ctx.read_fn = read_fn_wctx; 404 | ctx.write_fn = write_fn; 405 | ctx.flush_fn = flush_fn; 406 | input_db_name(); 407 | myFile = SD.open(filename, FILE_WRITE); 408 | if (!myFile) { 409 | print_error(0); 410 | return; 411 | } 412 | int32_t page_size = dblog_read_page_size(&ctx); 413 | if (page_size < 512) { 414 | Serial.print(F("Error reading page size\n")); 415 | myFile.close(); 416 | return; 417 | } 418 | if (dblog_recover(&ctx)) { 419 | Serial.print(F("Error during recover\n")); 420 | myFile.close(); 421 | return; 422 | } 423 | myFile.close(); 424 | } 425 | 426 | bool is_inited = false; 427 | void setup() { 428 | 429 | Serial.begin(9600); 430 | while (!Serial) { 431 | } 432 | 433 | Serial.print(F("InitSD..\n")); 434 | if (!SD.begin(SD_CS_PIN)) { 435 | Serial.print(F("failed!\n")); 436 | return; 437 | } 438 | 439 | Serial.print(F("done.\n")); 440 | is_inited = true; 441 | 442 | } 443 | 444 | void loop() { 445 | 446 | if (!is_inited) 447 | return; 448 | 449 | Serial.print(F("\n\nSqlite µLogger\n\n")); 450 | Serial.print(F("1. Log analog data\n")); 451 | Serial.print(F("2. Locate records by RowID\n")); 452 | Serial.print(F("3. Locate records using Binary Search\n")); 453 | Serial.print(F("4. Recover DB\n\n")); 454 | Serial.print(F("Enter choice: ")); 455 | int8_t choice = input_num(); 456 | switch (choice) { 457 | case 1: 458 | log_analog_data(); 459 | break; 460 | case 2: 461 | case 3: 462 | locate_records(choice); 463 | break; 464 | case 4: 465 | recover_db(); 466 | break; 467 | default: 468 | Serial.print(F("Invalid choice\n")); 469 | } 470 | 471 | } -------------------------------------------------------------------------------- /examples/Uno_and_above_SdFat/Uno_and_above_SdFat.ino: -------------------------------------------------------------------------------- 1 | /* 2 | This example demonstrates how the Sqlite Micro Logger library 3 | can be used to write Analog data into Sqlite database. 4 | Works on any Arduino compatible microcontroller with SD Card attachment 5 | having 2kb RAM or more (such as Arduino Uno). 6 | 7 | How Sqlite Micro Logger works: 8 | https://github.com/siara-cc/sqlite_micro_logger_c 9 | 10 | Copyright @ 2019 Arundale Ramanathan, Siara Logics (cc) 11 | 12 | Licensed under the Apache License, Version 2.0 (the "License"); 13 | you may not use this file except in compliance with the License. 14 | You may obtain a copy of the License at 15 | 16 | http://www.apache.org/licenses/LICENSE-2.0 17 | 18 | Unless required by applicable law or agreed to in writing, software 19 | distributed under the License is distributed on an "AS IS" BASIS, 20 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 21 | See the License for the specific language governing permissions and 22 | limitations under the License. 23 | */ 24 | #include "ulog_sqlite.h" 25 | #include 26 | #include "SdFat.h" 27 | SdFat SD; 28 | 29 | // Set the CS Pin connected to the MicroSD 30 | // 8 on most shields such as Sparkfun Micro SD Shield 31 | // 4 on WeMos D1 Mini 32 | #define SD_CS_PIN 8 33 | 34 | // If you would like to read DBs created with this repo 35 | // with page size more than 512, change here, 36 | // but ensure as much SRAM can be allocated. 37 | // Please Arduino Uno cannot support page size > 512. 38 | #define BUF_SIZE 512 39 | byte buf[BUF_SIZE]; 40 | char filename[13]; 41 | extern const char sqlite_sig[]; 42 | 43 | File myFile; 44 | 45 | int32_t read_fn_wctx(struct dblog_write_context *ctx, void *buf, uint32_t pos, size_t len) { 46 | myFile.seek(pos); 47 | size_t ret = myFile.read((byte *)buf, len); 48 | if (ret != len) 49 | return DBLOG_RES_READ_ERR; 50 | return ret; 51 | } 52 | 53 | int32_t read_fn_rctx(struct dblog_read_context *ctx, void *buf, uint32_t pos, size_t len) { 54 | myFile.seek(pos); 55 | size_t ret = myFile.read((byte *)buf, len); 56 | if (ret != len) 57 | return DBLOG_RES_READ_ERR; 58 | return ret; 59 | } 60 | 61 | int32_t write_fn(struct dblog_write_context *ctx, void *buf, uint32_t pos, size_t len) { 62 | myFile.seek(pos); 63 | size_t ret = myFile.write((byte *)buf, len); 64 | if (ret != len) 65 | return DBLOG_RES_ERR; 66 | myFile.flush(); 67 | return ret; 68 | } 69 | 70 | int flush_fn(struct dblog_write_context *ctx) { 71 | //myFile.flush(); // Anyway being flushed after write 72 | return DBLOG_RES_OK; 73 | } 74 | 75 | void print_error(int res) { 76 | Serial.print(F("Err:")); 77 | Serial.print(res); 78 | Serial.print(F("\n")); 79 | } 80 | 81 | int input_string(char *str, int max_len) { 82 | int ctr = 0; 83 | str[ctr] = 0; 84 | while (str[ctr] != '\n') { 85 | if (Serial.available()) { 86 | str[ctr] = Serial.read(); 87 | if (str[ctr] >= ' ' && str[ctr] <= '~') 88 | ctr++; 89 | if (ctr >= max_len) 90 | break; 91 | } 92 | } 93 | str[ctr] = 0; 94 | Serial.print(str); 95 | Serial.print(F("\n")); 96 | return ctr; 97 | } 98 | 99 | int32_t input_num() { 100 | char in[20]; 101 | int ctr = 0; 102 | in[ctr] = 0; 103 | while (in[ctr] != '\n') { 104 | if (Serial.available()) { 105 | in[ctr] = Serial.read(); 106 | if (in[ctr] >= '0' && in[ctr] <= '9') 107 | ctr++; 108 | if (ctr >= sizeof(in)) 109 | break; 110 | } 111 | } 112 | in[ctr] = 0; 113 | int32_t ret = atol(in); 114 | Serial.print(ret); 115 | Serial.print(F("\n")); 116 | return ret; 117 | } 118 | 119 | int16_t read_int16(const byte *ptr) { 120 | return (*ptr << 8) | ptr[1]; 121 | } 122 | 123 | int32_t read_int32(const byte *ptr) { 124 | int32_t ret; 125 | ret = ((int32_t)*ptr++) << 24; 126 | ret |= ((int32_t)*ptr++) << 16; 127 | ret |= ((int32_t)*ptr++) << 8; 128 | ret |= *ptr; 129 | return ret; 130 | } 131 | 132 | int64_t read_int64(const byte *ptr) { 133 | int64_t ret; 134 | ret = ((int64_t)*ptr++) << 56; 135 | ret |= ((int64_t)*ptr++) << 48; 136 | ret |= ((int64_t)*ptr++) << 40; 137 | ret |= ((int64_t)*ptr++) << 32; 138 | ret |= ((int64_t)*ptr++) << 24; 139 | ret |= ((int64_t)*ptr++) << 16; 140 | ret |= ((int64_t)*ptr++) << 8; 141 | ret |= *ptr; 142 | return ret; 143 | } 144 | 145 | int pow10(int8_t len) { 146 | return (len == 3 ? 1000 : (len == 2 ? 100 : (len == 1 ? 10 : 1))); 147 | } 148 | 149 | void set_ts_part(char *s, int val, int8_t len) { 150 | while (len--) { 151 | *s++ = '0' + val / pow10(len); 152 | val %= pow10(len); 153 | } 154 | } 155 | 156 | int get_ts_part(char *s, int8_t len) { 157 | int i = 0; 158 | while (len--) 159 | i += ((*s++ - '0') * pow10(len)); 160 | return i; 161 | } 162 | 163 | int update_ts_part(char *ptr, int8_t len, int limit, int ovflw) { 164 | int8_t is_one_based = (limit == 1000 || limit == 60 || limit == 24) ? 0 : 1; 165 | int part = get_ts_part(ptr, len) + ovflw - is_one_based; 166 | ovflw = part / limit; 167 | part %= limit; 168 | set_ts_part(ptr, part + is_one_based, len); 169 | return ovflw; 170 | } 171 | 172 | // 012345678901234567890 173 | // YYYY-MM-DD HH:MM:SS.SSS 174 | void update_ts(char *ts, int diff) { 175 | int ovflw = update_ts_part(ts + 20, 3, 1000, diff); // ms 176 | if (ovflw) { 177 | ovflw = update_ts_part(ts + 17, 2, 60, ovflw); // seconds 178 | if (ovflw) { 179 | ovflw = update_ts_part(ts + 14, 2, 60, ovflw); // minutes 180 | if (ovflw) { 181 | ovflw = update_ts_part(ts + 11, 2, 24, ovflw); // hours 182 | if (ovflw) { 183 | int8_t month = get_ts_part(ts + 5, 2); 184 | int year = get_ts_part(ts, 4); 185 | int8_t limit = (month == 2 ? (year % 4 ? 28 : 29) : 186 | (month == 4 || month == 6 || month == 9 || month == 11 ? 30 : 31)); 187 | ovflw = update_ts_part(ts + 8, 2, limit, ovflw); // day 188 | if (ovflw) { 189 | ovflw = update_ts_part(ts + 5, 2, 12, ovflw); // month 190 | if (ovflw) 191 | set_ts_part(ts, year + ovflw, 4); // year 192 | } 193 | } 194 | } 195 | } 196 | } 197 | } 198 | 199 | void display_row(struct dblog_read_context *ctx) { 200 | int i = 0; 201 | do { 202 | uint32_t col_type; 203 | const byte *col_val = (const byte *) dblog_read_col_val(ctx, i, &col_type); 204 | if (!col_val) { 205 | if (i == 0) 206 | Serial.print(F("Error reading value\n")); 207 | Serial.print(F("\n")); 208 | return; 209 | } 210 | if (i) 211 | Serial.print(F("|")); 212 | switch (col_type) { 213 | case 0: 214 | Serial.print(F("null")); 215 | break; 216 | case 1: 217 | Serial.print(*((int8_t *)col_val)); 218 | break; 219 | case 2: { 220 | int16_t ival = read_int16(col_val); 221 | Serial.print(ival); 222 | } 223 | break; 224 | case 4: { 225 | int32_t ival = read_int32(col_val); 226 | Serial.print(ival); 227 | break; 228 | } 229 | // Arduino Serial.print not capable of printing 230 | // int64_t and double. Need to implement manually 231 | case 6: // int64_t 232 | case 7: // double 233 | Serial.print(F("todo")); 234 | break; 235 | default: { 236 | uint32_t col_len = dblog_derive_data_len(col_type); 237 | for (int j = 0; j < col_len; j++) { 238 | if (col_type % 2) 239 | Serial.print((char)col_val[j]); 240 | else { 241 | Serial.print((int)col_val[j]); 242 | Serial.print(F(" ")); 243 | } 244 | } 245 | } 246 | } 247 | } while (++i); 248 | } 249 | 250 | void input_db_name() { 251 | Serial.print(F("DB name (max 8.3): ")); 252 | input_string(filename, sizeof(filename)); 253 | } 254 | 255 | int input_ts(char *datetime) { 256 | Serial.print(F("\nEnter timestamp (YYYY-MM-DD HH:MM:SS.SSS): ")); 257 | return input_string(datetime, 24); 258 | } 259 | 260 | void log_analog_data() { 261 | int32_t num_entries; 262 | int dly; 263 | input_db_name(); 264 | Serial.print(F("\nRecord count (1 to 32767 on UNO): ")); 265 | num_entries = input_num(); 266 | Serial.print(F("\nStarting analog pin (14=A0 on Uno, 17=A0 on ESP8266): ")); 267 | int8_t analog_pin_start = input_num(); 268 | Serial.print(F("\nNo. of pins: ")); 269 | int8_t analog_pin_count = input_num(); 270 | char ts[24]; 271 | if (input_ts(ts) < 23) { 272 | Serial.print(F("Input full timestamp\n")); 273 | return; 274 | } 275 | Serial.print(F("\nDelay(ms): ")); 276 | dly = input_num(); 277 | 278 | SD.remove(filename); 279 | myFile = SD.open(filename, O_READ | O_WRITE | O_CREAT); 280 | 281 | // if the file opened okay, write to it: 282 | if (myFile) { 283 | unsigned long start = millis(); 284 | unsigned long last_ms = start; 285 | struct dblog_write_context ctx; 286 | ctx.buf = buf; 287 | ctx.col_count = analog_pin_count + 1; 288 | ctx.page_resv_bytes = 0; 289 | ctx.page_size_exp = 9; 290 | ctx.max_pages_exp = 0; 291 | ctx.read_fn = read_fn_wctx; 292 | ctx.flush_fn = flush_fn; 293 | ctx.write_fn = write_fn; 294 | int res = dblog_write_init(&ctx); 295 | if (!res) { 296 | while (num_entries--) { 297 | res = dblog_set_col_val(&ctx, 0, DBLOG_TYPE_TEXT, ts, 23); 298 | if (res) 299 | break; 300 | update_ts(ts, (int) (millis() - last_ms)); 301 | last_ms = millis(); 302 | for (int8_t i = 0; i < analog_pin_count; i++) { 303 | int val = analogRead(analog_pin_start + i); 304 | res = dblog_set_col_val(&ctx, i + 1, DBLOG_TYPE_INT, &val, sizeof(int)); 305 | if (res) 306 | break; 307 | } 308 | if (num_entries) { 309 | res = dblog_append_empty_row(&ctx); 310 | if (res) 311 | break; 312 | delay(dly); 313 | } 314 | } 315 | } 316 | if (!res) 317 | res = dblog_finalize(&ctx); 318 | myFile.close(); 319 | if (res) 320 | print_error(res); 321 | else { 322 | Serial.print(F("\nDone. Elapsed time (ms): ")); 323 | Serial.print((millis() - start)); 324 | Serial.print("\n"); 325 | } 326 | } else { 327 | // if the file didn't open, print an error: 328 | Serial.print(F("Open Error\n")); 329 | } 330 | } 331 | 332 | void locate_records(int8_t choice) { 333 | int num_entries; 334 | int dly; 335 | input_db_name(); 336 | struct dblog_read_context rctx; 337 | rctx.page_size_exp = 9; 338 | rctx.read_fn = read_fn_rctx; 339 | myFile = SD.open(filename, FILE_READ); 340 | if (myFile) { 341 | Serial.print(F("Size:")); 342 | Serial.print(myFile.size()); 343 | Serial.print(F("\n")); 344 | rctx.buf = buf; 345 | int res = dblog_read_init(&rctx); 346 | if (res) { 347 | print_error(res); 348 | myFile.close(); 349 | return; 350 | } 351 | Serial.print(F("Page size:")); 352 | Serial.print((int32_t) 1 << rctx.page_size_exp); 353 | Serial.print(F("\nLast data page:")); 354 | Serial.print(rctx.last_leaf_page); 355 | Serial.print(F("\n")); 356 | if (memcmp(buf, sqlite_sig, 16) || buf[68] != 0xA5) { 357 | Serial.print(F("Invalid DB. Try recovery.\n")); 358 | myFile.close(); 359 | return; 360 | } 361 | if (BUF_SIZE < (int32_t) 1 << rctx.page_size_exp) { 362 | Serial.print(F("Buffer size less than Page size. Try increasing if enough SRAM\n")); 363 | myFile.close(); 364 | return; 365 | } 366 | Serial.print(F("\nFirst record:\n")); 367 | display_row(&rctx); 368 | uint32_t rowid; 369 | char srch_datetime[24]; // YYYY-MM-DD HH:MM:SS.SSS 370 | int8_t dt_len; 371 | if (choice == 2) { 372 | Serial.print(F("\nEnter RowID (1 to 32767 on UNO): ")); 373 | rowid = input_num(); 374 | } else 375 | dt_len = input_ts(srch_datetime); 376 | Serial.print(F("No. of records to display: ")); 377 | num_entries = input_num(); 378 | unsigned long start = millis(); 379 | if (choice == 2) 380 | res = dblog_srch_row_by_id(&rctx, rowid); 381 | else 382 | res = dblog_bin_srch_row_by_val(&rctx, 0, DBLOG_TYPE_TEXT, srch_datetime, dt_len, 0); 383 | if (res == DBLOG_RES_NOT_FOUND) 384 | Serial.print(F("Not Found\n")); 385 | else if (res == 0) { 386 | Serial.print(F("\nTime taken (ms): ")); 387 | Serial.print((millis() - start)); 388 | Serial.print("\n\n"); 389 | do { 390 | display_row(&rctx); 391 | } while (--num_entries && !dblog_read_next_row(&rctx)); 392 | } else 393 | print_error(res); 394 | myFile.close(); 395 | } else { 396 | // if the file didn't open, print an error: 397 | Serial.print(F("Open Error\n")); 398 | } 399 | } 400 | 401 | void recover_db() { 402 | struct dblog_write_context ctx; 403 | ctx.buf = buf; 404 | ctx.read_fn = read_fn_wctx; 405 | ctx.write_fn = write_fn; 406 | ctx.flush_fn = flush_fn; 407 | input_db_name(); 408 | myFile = SD.open(filename, FILE_WRITE); 409 | if (!myFile) { 410 | print_error(0); 411 | return; 412 | } 413 | int32_t page_size = dblog_read_page_size(&ctx); 414 | if (page_size < 512) { 415 | Serial.print(F("Error reading page size\n")); 416 | myFile.close(); 417 | return; 418 | } 419 | if (dblog_recover(&ctx)) { 420 | Serial.print(F("Error during recover\n")); 421 | myFile.close(); 422 | return; 423 | } 424 | myFile.close(); 425 | } 426 | 427 | bool is_inited = false; 428 | void setup() { 429 | 430 | Serial.begin(9600); 431 | while (!Serial) { 432 | } 433 | 434 | Serial.print(F("InitSD..\n")); 435 | if (!SD.begin(SD_CS_PIN)) { 436 | Serial.print(F("failed!\n")); 437 | return; 438 | } 439 | 440 | Serial.print(F("done.\n")); 441 | is_inited = true; 442 | 443 | } 444 | 445 | void loop() { 446 | 447 | if (!is_inited) 448 | return; 449 | 450 | Serial.print(F("\n\nSqlite µLogger\n\n")); 451 | Serial.print(F("1. Log analog data\n")); 452 | Serial.print(F("2. Locate records by RowID\n")); 453 | Serial.print(F("3. Locate records using Binary Search\n")); 454 | Serial.print(F("4. Recover DB\n\n")); 455 | Serial.print(F("Enter choice: ")); 456 | int8_t choice = input_num(); 457 | switch (choice) { 458 | case 1: 459 | log_analog_data(); 460 | break; 461 | case 2: 462 | case 3: 463 | locate_records(choice); 464 | break; 465 | case 4: 466 | recover_db(); 467 | break; 468 | default: 469 | Serial.print(F("Invalid choice\n")); 470 | } 471 | 472 | } -------------------------------------------------------------------------------- /keywords.txt: -------------------------------------------------------------------------------- 1 | ####################################### 2 | # Syntax Coloring Map 3 | ####################################### 4 | 5 | ####################################### 6 | # Datatypes (KEYWORD1) 7 | ####################################### 8 | 9 | dblog_write_context KEYWORD1 10 | dblog_read_context KEYWORD1 11 | 12 | ####################################### 13 | # Methods and Functions (KEYWORD2) 14 | ####################################### 15 | dblog_write_init KEYWORD2 16 | dblog_write_init_with_script KEYWORD2 17 | dblog_init_for_append KEYWORD2 18 | dblog_append_empty_row KEYWORD2 19 | dblog_append_row_with_values KEYWORD2 20 | dblog_set_col_val KEYWORD2 21 | dblog_get_col_val KEYWORD2 22 | dblog_flush KEYWORD2 23 | dblog_partial_finalize KEYWORD2 24 | dblog_finalize KEYWORD2 25 | dblog_not_finalized KEYWORD2 26 | dblog_read_page_size KEYWORD2 27 | dblog_recover KEYWORD2 28 | 29 | dblog_read_init KEYWORD2 30 | dblog_cur_row_col_count KEYWORD2 31 | dblog_read_col_val KEYWORD2 32 | dblog_derive_data_len KEYWORD2 33 | dblog_read_first_row KEYWORD2 34 | dblog_read_next_row KEYWORD2 35 | dblog_read_prev_row KEYWORD2 36 | dblog_read_last_row KEYWORD2 37 | dblog_srch_row_by_id KEYWORD2 38 | dblog_bin_srch_row_by_val KEYWORD2 39 | 40 | ###################################### 41 | # Constants (LITERAL1) 42 | ####################################### 43 | DBLOG_TYPE_INT LITERAL1 44 | DBLOG_TYPE_REAL LITERAL1 45 | DBLOG_TYPE_BLOB LITERAL1 46 | DBLOG_TYPE_TEXT LITERAL1 47 | -------------------------------------------------------------------------------- /library.properties: -------------------------------------------------------------------------------- 1 | name=Sqlite Micro Logger 2 | version=1.2 3 | author=Arundale Ramanathan 4 | maintainer=Arun 5 | sentence=Log data into Sqlite database from any MCU having >2kb RAM 6 | paragraph=This is a general purpose library that enables logging data into Sqlite databases from any Microcontroller having atleast 2kb RAM. This is useful to log Sensor data into Micro SD cards or inbuilt flash file systems. Multiple databases can be logged at the same time and transferred over the network for further processing. See documentation for further details and limitations. 7 | category=Data Storage 8 | url=https://github.com/siara-cc/sqlite_micro_logger_arduino 9 | architectures=* 10 | -------------------------------------------------------------------------------- /sqlite_ulogger_promo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/siara-cc/sqlite_micro_logger_arduino/1069b5f5b6a0e4f2fe834731a0449d81965239e7/sqlite_ulogger_promo.png -------------------------------------------------------------------------------- /src/ulog_sqlite.c: -------------------------------------------------------------------------------- 1 | /* 2 | Sqlite Micro Logger 3 | 4 | Fast and Lean Sqlite database logger targetting 5 | low RAM systems such as Microcontrollers. 6 | 7 | This Library can work on systems that have as little as 2kb, 8 | such as the ATMega328 MCU. It is available for the Arduino platform. 9 | 10 | https://github.com/siara-cc/sqlite_micro_logger 11 | 12 | Copyright @ 2019 Arundale Ramanathan, Siara Logics (cc) 13 | 14 | Licensed under the Apache License, Version 2.0 (the "License"); 15 | you may not use this file except in compliance with the License. 16 | You may obtain a copy of the License at 17 | 18 | http://www.apache.org/licenses/LICENSE-2.0 19 | 20 | Unless required by applicable law or agreed to in writing, software 21 | distributed under the License is distributed on an "AS IS" BASIS, 22 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 23 | See the License for the specific language governing permissions and 24 | limitations under the License. 25 | */ 26 | 27 | #include "ulog_sqlite.h" 28 | 29 | #include 30 | #include 31 | #include 32 | 33 | #define LEN_OF_REC_LEN 3 34 | #define LEN_OF_HDR_LEN 2 35 | #define CHKSUM_LEN 3 36 | 37 | enum {DBLOG_ST_WRITE_NOT_PENDING = 0xA4, DBLOG_ST_WRITE_PENDING, 38 | DBLOG_ST_TO_RECOVER, DBLOG_ST_FINAL}; 39 | 40 | // Returns how many bytes the given integer will 41 | // occupy if stored as a variable integer 42 | int8_t get_vlen_of_uint16(uint16_t vint) { 43 | return vint > 16383 ? 3 : (vint > 127 ? 2 : 1); 44 | } 45 | 46 | // Returns how many bytes the given integer will 47 | // occupy if stored as a variable integer 48 | int8_t get_vlen_of_uint32(uint32_t vint) { 49 | return vint > 268435455 ? 5 : (vint > 2097151 ? 4 50 | : (vint > 16383 ? 3 : (vint > 127 ? 2 : 1))); 51 | } 52 | 53 | // Stores the given byte in the given location 54 | // in big-endian sequence 55 | void write_uint8(byte *ptr, uint8_t input) { 56 | ptr[0] = input; 57 | } 58 | 59 | // Stores the given uint16_t in the given location 60 | // in big-endian sequence 61 | void write_uint16(byte *ptr, uint16_t input) { 62 | ptr[0] = input >> 8; 63 | ptr[1] = input & 0xFF; 64 | } 65 | 66 | // Stores the given uint32_t in the given location 67 | // in big-endian sequence 68 | void write_uint32(byte *ptr, uint32_t input) { 69 | int i = 4; 70 | while (i--) 71 | *ptr++ = (input >> (8 * i)) & 0xFF; 72 | } 73 | 74 | // Stores the given uint64_t in the given location 75 | // in big-endian sequence 76 | void write_uint64(byte *ptr, uint64_t input) { 77 | int i = 8; 78 | while (i--) 79 | *ptr++ = (input >> (8 * i)) & 0xFF; 80 | } 81 | 82 | // Stores the given uint16_t in the given location 83 | // in variable integer format 84 | int write_vint16(byte *ptr, uint16_t vint) { 85 | int len = get_vlen_of_uint16(vint); 86 | for (int i = len - 1; i > 0; i--) 87 | *ptr++ = 0x80 + ((vint >> (7 * i)) & 0x7F); 88 | *ptr = vint & 0x7F; 89 | return len; 90 | } 91 | 92 | // Stores the given uint32_t in the given location 93 | // in variable integer format 94 | int write_vint32(byte *ptr, uint32_t vint) { 95 | int len = get_vlen_of_uint32(vint); 96 | for (int i = len - 1; i > 0; i--) 97 | *ptr++ = 0x80 + ((vint >> (7 * i)) & 0x7F); 98 | *ptr = vint & 0x7F; 99 | return len; 100 | } 101 | 102 | // Reads and returns big-endian uint8_t 103 | // at a given memory location 104 | uint8_t read_uint8(byte *ptr) { 105 | return *ptr; 106 | } 107 | 108 | // Reads and returns big-endian uint16_t 109 | // at a given memory location 110 | uint16_t read_uint16(byte *ptr) { 111 | return (*ptr << 8) + ptr[1]; 112 | } 113 | 114 | // Reads and returns big-endian uint32_t 115 | // at a given memory location 116 | uint32_t read_uint32(byte *ptr) { 117 | uint32_t ret; 118 | ret = ((uint32_t)*ptr++) << 24; 119 | ret += ((uint32_t)*ptr++) << 16; 120 | ret += ((uint32_t)*ptr++) << 8; 121 | ret += *ptr; 122 | return ret; 123 | } 124 | 125 | // Reads and returns big-endian uint64_t 126 | // at a given memory location 127 | uint64_t read_uint64(byte *ptr) { 128 | uint64_t ret = 0; 129 | int len = 8; 130 | while (len--) 131 | ret += (((uint64_t)*ptr++) << (8 * len)); 132 | return ret; 133 | } 134 | 135 | // Reads and returns variable integer 136 | // from given location as uint16_t 137 | // Also returns the length of the varint 138 | uint16_t read_vint16(byte *ptr, int8_t *vlen) { 139 | uint16_t ret = 0; 140 | int8_t len = 3; // read max 3 bytes 141 | do { 142 | ret <<= 7; 143 | ret += *ptr & 0x7F; 144 | len--; 145 | } while ((*ptr++ & 0x80) == 0x80 && len); 146 | if (vlen) 147 | *vlen = 3 - len; 148 | return ret; 149 | } 150 | 151 | // Reads and returns variable integer 152 | // from given location as uint32_t 153 | // Also returns the length of the varint 154 | uint32_t read_vint32(byte *ptr, int8_t *vlen) { 155 | uint32_t ret = 0; 156 | int8_t len = 5; // read max 5 bytes 157 | do { 158 | ret <<= 7; 159 | ret += *ptr & 0x7F; 160 | len--; 161 | } while ((*ptr++ & 0x80) == 0x80 && len); 162 | if (vlen) 163 | *vlen = 5 - len; 164 | return ret; 165 | } 166 | 167 | // Converts float to Sqlite's Big-endian double 168 | int64_t float_to_double(const void *val) { 169 | uint32_t bytes = *((uint32_t *) val); 170 | uint8_t exp8 = (bytes >> 23) & 0xFF; 171 | uint16_t exp11 = exp8; 172 | if (exp11 != 0) { 173 | if (exp11 < 127) 174 | exp11 = 1023 - (127 - exp11); 175 | else 176 | exp11 = 1023 + (exp11 - 127); 177 | } 178 | return ((int64_t)(bytes >> 31) << 63) 179 | | ((int64_t)exp11 << 52) 180 | | ((int64_t)(bytes & 0x7FFFFF) << (52-23) ); 181 | } 182 | 183 | // Returns actual page size from given exponent 184 | int32_t get_pagesize(byte page_size_exp) { 185 | return (int32_t) 1 << page_size_exp; 186 | } 187 | 188 | // Returns position of last record. 189 | // Creates one, if no record found. 190 | uint16_t acquire_last_pos(struct dblog_write_context *wctx, byte *ptr) { 191 | uint16_t last_pos = read_uint16(ptr + 5); 192 | if (last_pos == 0) { 193 | dblog_append_empty_row(wctx); 194 | last_pos = read_uint16(ptr + 5); 195 | } 196 | return last_pos; 197 | } 198 | 199 | // Attempts to locate a column using given index 200 | // Returns position of column in header area, position of column 201 | // in data area, record length and header length 202 | // See https://www.sqlite.org/fileformat.html#record_format 203 | byte *locate_column(byte *rec_ptr, int col_idx, byte **pdata_ptr, 204 | uint16_t *prec_len, uint16_t *phdr_len, uint16_t limit) { 205 | int8_t vint_len; 206 | byte *hdr_ptr = rec_ptr; 207 | *prec_len = read_vint16(hdr_ptr, &vint_len); 208 | hdr_ptr += vint_len; 209 | read_vint32(hdr_ptr, &vint_len); 210 | hdr_ptr += vint_len; 211 | if (*prec_len + (hdr_ptr - rec_ptr) > limit) 212 | return NULL; // corruption 213 | *phdr_len = read_vint16(hdr_ptr, &vint_len); 214 | if (*phdr_len > limit) 215 | return NULL; // corruption 216 | *pdata_ptr = hdr_ptr + *phdr_len; 217 | byte *data_start_ptr = *pdata_ptr; // re-position to check for corruption below 218 | hdr_ptr += vint_len; 219 | for (int i = 0; i < col_idx; i++) { 220 | uint32_t col_type_or_len = read_vint32(hdr_ptr, &vint_len); 221 | hdr_ptr += vint_len; 222 | (*pdata_ptr) += dblog_derive_data_len(col_type_or_len); 223 | if (hdr_ptr >= data_start_ptr) 224 | return NULL; // corruption or column not found 225 | if (*pdata_ptr - rec_ptr > limit) 226 | return NULL; // corruption 227 | } 228 | return hdr_ptr; 229 | } 230 | 231 | // Returns type of column based on given value and length 232 | // See https://www.sqlite.org/fileformat.html#record_format 233 | uint32_t derive_col_type_or_len(int type, const void *val, int len) { 234 | uint32_t col_type_or_len = 0; 235 | if (val != NULL) { 236 | switch (type) { 237 | case DBLOG_TYPE_INT: 238 | col_type_or_len = (len == 1 ? 1 : (len == 2 ? 2 : (len == 4 ? 4 : 6))); 239 | //if (len == 1) { 240 | // int8_t *typed_val = (int8_t *) val; 241 | // col_type_or_len = (*typed_val == 0 ? 8 : (*typed_val == 1 ? 9 : 0)); 242 | //} else 243 | // col_type_or_len = (len == 2 ? 2 : (len == 4 ? 4 : 6)); 244 | break; 245 | case DBLOG_TYPE_REAL: 246 | col_type_or_len = 7; 247 | break; 248 | case DBLOG_TYPE_BLOB: 249 | col_type_or_len = len * 2 + 12; 250 | break; 251 | case DBLOG_TYPE_TEXT: 252 | col_type_or_len = len * 2 + 13; 253 | } 254 | } 255 | return col_type_or_len; 256 | } 257 | 258 | // Returns one of the four types based on column type or len 259 | // found in header 260 | // See https://www.sqlite.org/fileformat.html#record_format 261 | uint32_t derive_col_type(int hdr_col_type_or_len) { 262 | switch (hdr_col_type_or_len) { 263 | case 7: 264 | return DBLOG_TYPE_REAL; 265 | case 1: 266 | case 2: 267 | case 3: 268 | case 4: 269 | case 5: 270 | case 6: 271 | case 8: 272 | case 9: 273 | return DBLOG_TYPE_INT; 274 | default: 275 | return (hdr_col_type_or_len % 2 ? DBLOG_TYPE_TEXT : DBLOG_TYPE_BLOB); 276 | } 277 | return DBLOG_TYPE_TEXT; // error 278 | } 279 | 280 | byte c1, c2, c3; 281 | void saveChecksumBytes(byte * ptr, uint16_t last_pos) { 282 | ptr += last_pos; 283 | ptr--; 284 | c1 = *ptr--; 285 | c2 = *ptr--; 286 | c3 = *ptr; 287 | } 288 | 289 | void restoreChecksumBytes(byte * ptr, uint16_t last_pos) { 290 | ptr += last_pos; 291 | ptr--; 292 | *ptr-- = c1; 293 | *ptr-- = c2; 294 | *ptr = c3; 295 | } 296 | 297 | // Initializes the buffer as a B-Tree Leaf table 298 | void init_bt_tbl_leaf(byte *ptr) { 299 | ptr[0] = 13; // Leaf table b-tree page 300 | write_uint16(ptr + 1, 0); // No freeblocks 301 | write_uint16(ptr + 3, 0); // No records yet 302 | write_uint16(ptr + 5, 0); // No records yet 303 | write_uint8(ptr + 7, 0); // Fragmented free bytes 304 | } 305 | 306 | // Initializes the buffer as a B-Tree Interior table 307 | void init_bt_tbl_inner(byte *ptr) { 308 | ptr[0] = 5; // Interior table b-tree page 309 | write_uint16(ptr + 1, 0); // No freeblocks 310 | write_uint16(ptr + 3, 0); // No records yet 311 | write_uint16(ptr + 5, 0); // No records yet 312 | write_uint8(ptr + 7, 0); // Fragmented free bytes 313 | } 314 | 315 | // Writes Record length, Row ID and Header length 316 | // at given location 317 | // No corruption checking because no unreliable source 318 | void write_rec_len_rowid_hdr_len(byte *ptr, uint16_t rec_len, uint32_t rowid, uint16_t hdr_len) { 319 | // write record len 320 | *ptr++ = 0x80 + (rec_len >> 14); 321 | *ptr++ = 0x80 + ((rec_len >> 7) & 0x7F); 322 | *ptr++ = rec_len & 0x7F; 323 | // write row id 324 | ptr += write_vint32(ptr, rowid); 325 | // write header len 326 | *ptr++ = 0x80 + (hdr_len >> 7); 327 | *ptr = hdr_len & 0x7F; 328 | } 329 | 330 | // Checks or calculates 3 checksums: 331 | // 1. Header checksum, which is for page header and last rowid 332 | // 2. Checksum of first record 333 | // 3. Checksum of entire page 334 | // Checksum is simply a 8 bit sum of byte values, ignoring overflows 335 | // calc_or_check == 0 means calculate all three checksums 336 | // calc_or_check == 1 means check header checksum 337 | // calc_or_check == 2 means check first record checksum 338 | // calc_or_check == 3 means check page checksum 339 | int check_sums(byte *buf, int32_t page_size, int calc_or_check) { 340 | if (*buf == 5) // no need checksum for internal pages 341 | return DBLOG_RES_OK; 342 | if (*buf == 13) { 343 | int8_t vlen; 344 | uint8_t chk_sum = 0; 345 | uint16_t i = 0; 346 | uint16_t end = 8; 347 | while (i < end) // Header checksum 348 | chk_sum += buf[i++]; 349 | uint16_t last_pos = read_uint16(buf + 5); 350 | i = last_pos; 351 | end = i + LEN_OF_REC_LEN; 352 | read_vint32(buf + end, &vlen); 353 | end += vlen; 354 | while (i < end) // Header checksum 355 | chk_sum += buf[i++]; 356 | if (calc_or_check == 0) 357 | buf[last_pos - 1] = chk_sum; 358 | else if (calc_or_check == 1) { 359 | if (buf[last_pos - 1] != chk_sum) 360 | return DBLOG_RES_INV_CHKSUM; 361 | return DBLOG_RES_OK; 362 | } 363 | i = end; 364 | end += read_vint16(buf + end - vlen - LEN_OF_REC_LEN, &vlen); 365 | while (i < end) // First record checksum 366 | chk_sum += buf[i++]; 367 | if (calc_or_check == 0) 368 | buf[last_pos - 2] = chk_sum; 369 | else if (calc_or_check == 2) { 370 | if (buf[last_pos - 2] != chk_sum) 371 | return DBLOG_RES_INV_CHKSUM; 372 | return DBLOG_RES_OK; 373 | } 374 | i = end; 375 | end = page_size; 376 | while (i < end) // Page checksum 377 | chk_sum += buf[i++]; 378 | i = 8; 379 | end = i + read_uint16(buf + 3) * 2; 380 | while (i < end) // Page checksum 381 | chk_sum += buf[i++]; 382 | if (calc_or_check == 0) 383 | buf[last_pos - 3] = chk_sum; 384 | else { 385 | if (buf[last_pos - 3] != chk_sum) 386 | return DBLOG_RES_INV_CHKSUM; 387 | } 388 | } else { // Assume first page 389 | int i = 0; 390 | uint8_t chk_sum = 0; 391 | while (i < page_size) 392 | chk_sum += buf[i++]; 393 | chk_sum -= buf[69]; 394 | if (calc_or_check) 395 | buf[69] = chk_sum; 396 | else { 397 | if (buf[69] != chk_sum) 398 | return DBLOG_RES_ERR; 399 | } 400 | } 401 | return DBLOG_RES_OK; 402 | } 403 | 404 | // Writes a page to disk using the given callback function 405 | int write_page(struct dblog_write_context *wctx, uint32_t page_no, int32_t page_size) { 406 | check_sums(wctx->buf, page_size, 0); 407 | if ((wctx->write_fn)(wctx, wctx->buf, page_no * page_size, page_size) != page_size) 408 | return DBLOG_RES_WRITE_ERR; 409 | return DBLOG_RES_OK; 410 | } 411 | 412 | // Reads specified number of bytes from disk using the given callback function 413 | // for Write context 414 | int read_bytes_wctx(struct dblog_write_context *wctx, byte *buf, long pos, int32_t size) { 415 | if ((wctx->read_fn)(wctx, buf, pos, size) != size) 416 | return DBLOG_RES_READ_ERR; 417 | return DBLOG_RES_OK; 418 | } 419 | 420 | // Reads specified number of bytes from disk using the given callback function 421 | // for Read context 422 | int read_bytes_rctx(struct dblog_read_context *rctx, byte *buf, long pos, int32_t size) { 423 | if ((rctx->read_fn)(rctx, buf, pos, size) != size) 424 | return DBLOG_RES_READ_ERR; 425 | return DBLOG_RES_OK; 426 | } 427 | 428 | // Adds record to B-Tree inner table 429 | int add_rec_to_inner_tbl(struct dblog_write_context *wctx, byte *parent_buf, 430 | uint32_t rowid, uint32_t cur_level_pos) { 431 | 432 | int32_t page_size = get_pagesize(wctx->page_size_exp); 433 | uint16_t last_pos = read_uint16(parent_buf + 5); 434 | int rec_count = read_uint16(parent_buf + 3) + 1; 435 | byte rec_len = 4 + get_vlen_of_uint32(rowid); 436 | 437 | if (last_pos == 0) 438 | last_pos = page_size - rec_len; 439 | else { 440 | // 3 is for checksum 441 | if (last_pos - rec_len < 12 + rec_count * 2 + get_vlen_of_uint32(rowid) + 3) 442 | last_pos = 0; 443 | else 444 | last_pos -= rec_len; 445 | } 446 | 447 | cur_level_pos++; 448 | if (last_pos && rowid) { 449 | write_uint32(parent_buf + last_pos, cur_level_pos); 450 | write_vint32(parent_buf + last_pos + 4, rowid); 451 | write_uint16(parent_buf + 3, rec_count--); 452 | write_uint16(parent_buf + 12 + rec_count * 2, last_pos); 453 | write_uint16(parent_buf + 5, last_pos); 454 | } else { 455 | write_uint32(parent_buf + 8, cur_level_pos); 456 | rec_count--; 457 | write_vint32(parent_buf + 12 + rec_count * 2, rowid); 458 | return 1; 459 | } 460 | 461 | // No corruption checking because no unreliable source 462 | return DBLOG_RES_OK; 463 | 464 | } 465 | 466 | const char sqlite_sig[] = "SQLite format 3"; 467 | const char dblog_sig[] = "SQLite3 uLogger"; 468 | char default_table_name[] = "t1"; 469 | 470 | // Writes data into buffer to form first page of Sqlite db 471 | int form_page1(struct dblog_write_context *wctx, char *table_name, char *table_script) { 472 | 473 | if (wctx->page_size_exp < 9 || wctx->page_size_exp > 16) 474 | return DBLOG_RES_INV_PAGE_SZ; 475 | byte *buf = (byte *) wctx->buf; 476 | int32_t page_size = get_pagesize(wctx->page_size_exp); 477 | wctx->cur_write_rowid = 0; 478 | 479 | // 100 byte header - refer https://www.sqlite.org/fileformat.html 480 | memcpy(buf, dblog_sig, 16); 481 | //memcpy(buf, "SQLite format 3\0", 16); 482 | write_uint16(buf + 16, page_size == 65536 ? 1 : (uint16_t) page_size); 483 | buf[18] = 1; 484 | buf[19] = 1; 485 | buf[20] = wctx->page_resv_bytes; 486 | buf[21] = 64; 487 | buf[22] = 32; 488 | buf[23] = 32; 489 | //write_uint32(buf + 24, 0); 490 | //write_uint32(buf + 28, 0); 491 | //write_uint32(buf + 32, 0); 492 | //write_uint32(buf + 36, 0); 493 | //write_uint32(buf + 40, 0); 494 | memset(buf + 24, '\0', 20); // Set to zero, above 5 495 | write_uint32(buf + 28, 2); // TODO: Update during finalize 496 | write_uint32(buf + 44, 4); 497 | //write_uint16(buf + 48, 0); 498 | //write_uint16(buf + 52, 0); 499 | memset(buf + 48, '\0', 8); // Set to zero, above 2 500 | write_uint32(buf + 56, 1); 501 | // User version initially 0, set to table leaf count 502 | // used to locate last leaf page for binary search 503 | // and move to last page. 504 | write_uint32(buf + 60, 0); 505 | write_uint32(buf + 64, 0); 506 | // App ID - set to 0xA5xxxxxx where A5 is signature 507 | // last 5 bits = wctx->max_pages_exp - set to 0 currently 508 | // till it is implemented 509 | write_uint32(buf + 68, 0xA5000000); 510 | memset(buf + 72, '\0', 20); // reserved space 511 | write_uint32(buf + 92, 105); 512 | write_uint32(buf + 96, 3016000); 513 | memset(buf + 100, '\0', page_size - 100); // Set remaing page to zero 514 | 515 | // master table b-tree 516 | init_bt_tbl_leaf(buf + 100); 517 | 518 | // write table script record 519 | int orig_col_count = wctx->col_count; 520 | wctx->cur_write_page = 0; 521 | wctx->col_count = 5; 522 | dblog_append_empty_row(wctx); 523 | dblog_set_col_val(wctx, 0, DBLOG_TYPE_TEXT, "table", 5); 524 | if (table_name == NULL) 525 | table_name = default_table_name; 526 | dblog_set_col_val(wctx, 1, DBLOG_TYPE_TEXT, table_name, strlen(table_name)); 527 | dblog_set_col_val(wctx, 2, DBLOG_TYPE_TEXT, table_name, strlen(table_name)); 528 | int32_t root_page = 2; 529 | dblog_set_col_val(wctx, 3, DBLOG_TYPE_INT, &root_page, 4); 530 | if (table_script) { 531 | uint16_t script_len = strlen(table_script); 532 | if (script_len > page_size - 100 - wctx->page_resv_bytes - 8 - 10) 533 | return DBLOG_RES_TOO_LONG; 534 | dblog_set_col_val(wctx, 4, DBLOG_TYPE_TEXT, table_script, script_len); 535 | } else { 536 | int table_name_len = strlen(table_name); 537 | int script_len = (13 + table_name_len + 2 + 5 * orig_col_count); 538 | if (script_len > page_size - 100 - wctx->page_resv_bytes - 8 - 10) 539 | return DBLOG_RES_TOO_LONG; 540 | dblog_set_col_val(wctx, 4, DBLOG_TYPE_TEXT, buf + 110, script_len); 541 | byte *script_pos = buf + page_size - buf[20] - script_len; 542 | memcpy(script_pos, "CREATE TABLE ", 13); 543 | script_pos += 13; 544 | memcpy(script_pos, table_name, table_name_len); 545 | script_pos += table_name_len; 546 | *script_pos++ = ' '; 547 | *script_pos++ = '('; 548 | for (int i = 0; i < orig_col_count; ) { 549 | i++; 550 | *script_pos++ = 'c'; 551 | *script_pos++ = '0' + (i < 100 ? 0 : (i / 100)); 552 | *script_pos++ = '0' + (i < 10 ? 0 : ((i < 100 ? i : i - 100) / 10)); 553 | *script_pos++ = '0' + (i % 10); 554 | *script_pos++ = (i == orig_col_count ? ')' : ','); 555 | } 556 | } 557 | int res = write_page(wctx, 0, page_size); 558 | if (res) 559 | return res; 560 | wctx->col_count = orig_col_count; 561 | wctx->cur_write_page = 1; 562 | wctx->cur_write_rowid = 0; 563 | init_bt_tbl_leaf(wctx->buf); 564 | wctx->state = DBLOG_ST_WRITE_PENDING; 565 | 566 | return DBLOG_RES_OK; 567 | 568 | } 569 | 570 | // Returns the Row ID of the last record stored in the given buffer 571 | // Reads the buffer part by part to avoid reading entire buffer into memory 572 | // to support low memory systems (2kb ram) 573 | // The underlying callback function hopefully optimizes repeated IO 574 | int get_last_rowid(struct dblog_write_context *wctx, uint32_t pos, 575 | int32_t page_size, uint32_t *out_rowid) { 576 | byte src_buf[12]; 577 | int res = read_bytes_wctx(wctx, src_buf, pos * page_size, 12); 578 | if (res) 579 | return res; 580 | uint16_t last_pos = read_uint16(src_buf + 5); 581 | if (!last_pos && *src_buf == 5) { 582 | *out_rowid = 0; 583 | return DBLOG_RES_OK; 584 | } 585 | if (last_pos > page_size - 12) 586 | return DBLOG_RES_MALFORMED; 587 | uint8_t page_type = *src_buf; 588 | uint16_t remaining = page_size - wctx->page_resv_bytes - last_pos; 589 | uint8_t chk_sum = 0; 590 | for (int i = 0; i < 8; i++) 591 | chk_sum += src_buf[i]; 592 | res = read_bytes_wctx(wctx, src_buf, 593 | pos * page_size + (page_type == 13 ? last_pos - 1 : (12 + read_uint16(src_buf + 3) * 2)), 594 | (page_type == 5 || remaining > 12 ? 12 : remaining)); 595 | if (res) 596 | return res; 597 | int8_t vint_len; 598 | *out_rowid = read_vint32(src_buf + (page_type == 13 ? 4 : 0), &vint_len); 599 | for (int i = 1; i < 4 + vint_len; i++) 600 | chk_sum += src_buf[i]; 601 | if (page_type == 13) { 602 | if (chk_sum != *src_buf) 603 | return DBLOG_RES_INV_CHKSUM; 604 | } 605 | return DBLOG_RES_OK; 606 | } 607 | 608 | // Returns pointer to data of given column index 609 | // Also returns type of column according to record format 610 | // See https://www.sqlite.org/fileformat.html#record_format 611 | const void *get_col_val(byte *buf, uint16_t rec_data_pos, 612 | int col_idx, uint32_t *out_col_type, uint16_t limit) { 613 | byte *data_ptr; 614 | uint16_t rec_len; 615 | uint16_t hdr_len; 616 | byte *hdr_ptr = locate_column(buf + rec_data_pos, col_idx, 617 | &data_ptr, &rec_len, &hdr_len, limit); 618 | if (!hdr_ptr) 619 | return NULL; 620 | int8_t cur_len_of_len; 621 | *out_col_type = read_vint32(hdr_ptr, &cur_len_of_len); 622 | return data_ptr; 623 | } 624 | 625 | // Checks possible signatures that a file can have 626 | int check_signature(byte *buf) { 627 | if (memcmp(buf, dblog_sig, 16) && memcmp(buf, sqlite_sig, 16)) 628 | return DBLOG_RES_INVALID_SIG; 629 | if (buf[68] != 0xA5) 630 | return DBLOG_RES_INVALID_SIG; 631 | return DBLOG_RES_OK; 632 | } 633 | 634 | byte *locate_col_root_page(byte *buf, int32_t page_size) { 635 | byte *data_ptr; 636 | uint16_t rec_len; 637 | uint16_t hdr_len; 638 | uint16_t last_pos = read_uint16(buf + 105); 639 | if (!locate_column(buf + last_pos, 3, 640 | &data_ptr, &rec_len, &hdr_len, page_size - last_pos)) 641 | return NULL; 642 | return data_ptr; 643 | } 644 | 645 | // Returns exponent for given page size 646 | byte get_page_size_exp(int32_t page_size) { 647 | if (page_size == 1) 648 | return 16; 649 | byte exp = 9; 650 | while (page_size >> exp) 651 | exp++; 652 | return exp - 1; 653 | } 654 | 655 | // See .h file for API description 656 | int dblog_write_init_with_script(struct dblog_write_context *wctx, 657 | char *table_name, char *table_script) { 658 | return form_page1(wctx, table_name, table_script); 659 | } 660 | 661 | // See .h file for API description 662 | int dblog_write_init(struct dblog_write_context *wctx) { 663 | return dblog_write_init_with_script(wctx, 0, 0); 664 | } 665 | 666 | // Checks space for appending new row 667 | // If space not available, writes current buffer to disk and 668 | // initializes buffer as new page 669 | uint16_t make_space_for_new_row(struct dblog_write_context *wctx, int32_t page_size, 670 | uint16_t len_of_rec_len_rowid, uint16_t new_rec_len) { 671 | byte *ptr = wctx->buf + (wctx->buf[0] == 13 ? 0 : 100); 672 | uint16_t last_pos = read_uint16(ptr + 5); 673 | if (last_pos && last_pos > page_size - wctx->page_resv_bytes - 7) 674 | return 0; // corruption 675 | int rec_count = read_uint16(ptr + 3) + 1; 676 | if (last_pos && rec_count * 2 + 8 >= last_pos) 677 | return 0; // corruption 678 | if (last_pos == 0) 679 | last_pos = page_size - wctx->page_resv_bytes; 680 | if (last_pos && last_pos < ((ptr - wctx->buf) + 9 + CHKSUM_LEN 681 | + (rec_count * 2) + new_rec_len + len_of_rec_len_rowid)) { 682 | int res = write_page(wctx, wctx->cur_write_page, page_size); 683 | if (res) 684 | return res; 685 | wctx->cur_write_page++; 686 | init_bt_tbl_leaf(wctx->buf); 687 | last_pos = page_size - wctx->page_resv_bytes - new_rec_len - len_of_rec_len_rowid; 688 | } else { 689 | last_pos -= new_rec_len; 690 | last_pos -= len_of_rec_len_rowid; 691 | } 692 | return last_pos; 693 | } 694 | 695 | // Writes given value at given pointer in Sqlite format 696 | uint16_t write_data(byte *data_ptr, int type, const void *val, uint16_t len) { 697 | if (val == NULL) 698 | return 0; 699 | if (type == DBLOG_TYPE_INT) { 700 | switch (len) { 701 | case 1: 702 | write_uint8(data_ptr, *((int8_t *) val)); 703 | break; 704 | case 2: 705 | write_uint16(data_ptr, *((int16_t *) val)); 706 | break; 707 | case 4: 708 | write_uint32(data_ptr, *((int32_t *) val)); 709 | break; 710 | case 8: 711 | write_uint64(data_ptr, *((int64_t *) val)); 712 | break; 713 | } 714 | } else 715 | if (type == DBLOG_TYPE_REAL && len == 4) { 716 | // Assumes float is represented in IEEE-754 format 717 | uint64_t bytes64 = float_to_double(val); 718 | write_uint64(data_ptr, bytes64); 719 | len = 8; 720 | } else 721 | if (type == DBLOG_TYPE_REAL && len == 8) { 722 | // Assumes double is represented in IEEE-754 format 723 | uint64_t bytes = *((uint64_t *) val); 724 | write_uint64(data_ptr, bytes); 725 | } else 726 | memcpy(data_ptr, val, len); 727 | return len; 728 | } 729 | 730 | // See .h file for API description 731 | int dblog_append_row_with_values(struct dblog_write_context *wctx, 732 | uint8_t types[], const void *values[], uint16_t lengths[]) { 733 | 734 | wctx->cur_write_rowid++; 735 | byte *ptr = wctx->buf + (wctx->buf[0] == 13 ? 0 : 100); 736 | int32_t page_size = get_pagesize(wctx->page_size_exp); 737 | uint16_t len_of_rec_len_rowid = LEN_OF_REC_LEN + get_vlen_of_uint32(wctx->cur_write_rowid); 738 | uint16_t new_rec_len = 0; 739 | uint16_t hdr_len = LEN_OF_HDR_LEN; 740 | for (int i = 0; i < wctx->col_count; i++) { 741 | if (values[i] != NULL) 742 | new_rec_len += (types[i] == DBLOG_TYPE_REAL ? 8 : lengths[i]); 743 | uint32_t col_type = derive_col_type_or_len(types[i], values[i], lengths[i]); 744 | hdr_len += get_vlen_of_uint32(col_type); 745 | } 746 | new_rec_len += hdr_len; 747 | uint16_t last_pos = make_space_for_new_row(wctx, page_size, 748 | len_of_rec_len_rowid, new_rec_len); 749 | if (!last_pos) 750 | return DBLOG_RES_MALFORMED; 751 | int rec_count = read_uint16(ptr + 3) + 1; 752 | if (rec_count * 2 + 8 >= last_pos) 753 | return DBLOG_RES_MALFORMED; 754 | 755 | write_rec_len_rowid_hdr_len(wctx->buf + last_pos, new_rec_len, 756 | wctx->cur_write_rowid, hdr_len); 757 | byte *rec_ptr = wctx->buf + last_pos + len_of_rec_len_rowid + LEN_OF_HDR_LEN; 758 | for (int i = 0; i < wctx->col_count; i++) { 759 | uint32_t col_type = derive_col_type_or_len(types[i], values[i], lengths[i]); 760 | int8_t vint_len = write_vint32(rec_ptr, col_type); 761 | rec_ptr += vint_len; 762 | } 763 | for (int i = 0; i < wctx->col_count; i++) { 764 | if (values[i] != NULL) 765 | rec_ptr += write_data(rec_ptr, types[i], values[i], lengths[i]); 766 | } 767 | write_uint16(ptr + 3, rec_count); 768 | write_uint16(ptr + 5, last_pos); 769 | write_uint16(ptr + 8 - 2 + (rec_count * 2), last_pos); 770 | wctx->state = DBLOG_ST_WRITE_PENDING; 771 | 772 | return DBLOG_RES_OK; 773 | } 774 | 775 | // See .h file for API description 776 | int dblog_append_empty_row(struct dblog_write_context *wctx) { 777 | 778 | wctx->cur_write_rowid++; 779 | byte *ptr = wctx->buf + (wctx->buf[0] == 13 ? 0 : 100); 780 | int32_t page_size = get_pagesize(wctx->page_size_exp); 781 | uint16_t len_of_rec_len_rowid = LEN_OF_REC_LEN + get_vlen_of_uint32(wctx->cur_write_rowid); 782 | uint16_t new_rec_len = wctx->col_count; 783 | new_rec_len += LEN_OF_HDR_LEN; 784 | uint16_t last_pos = make_space_for_new_row(wctx, page_size, 785 | len_of_rec_len_rowid, new_rec_len); 786 | if (!last_pos) 787 | return DBLOG_RES_MALFORMED; 788 | int rec_count = read_uint16(ptr + 3) + 1; 789 | if (rec_count * 2 + 8 >= last_pos) 790 | return DBLOG_RES_MALFORMED; 791 | 792 | memset(wctx->buf + last_pos, '\0', new_rec_len + len_of_rec_len_rowid); 793 | write_rec_len_rowid_hdr_len(wctx->buf + last_pos, new_rec_len, 794 | wctx->cur_write_rowid, wctx->col_count + LEN_OF_HDR_LEN); 795 | write_uint16(ptr + 3, rec_count); 796 | write_uint16(ptr + 5, last_pos); 797 | write_uint16(ptr + 8 - 2 + (rec_count * 2), last_pos); 798 | wctx->state = DBLOG_ST_WRITE_PENDING; 799 | 800 | return DBLOG_RES_OK; 801 | } 802 | 803 | // See .h file for API description 804 | int dblog_set_col_val(struct dblog_write_context *wctx, 805 | int col_idx, int type, const void *val, uint16_t len) { 806 | 807 | byte *ptr = wctx->buf + (wctx->buf[0] == 13 ? 0 : 100); 808 | int32_t page_size = get_pagesize(wctx->page_size_exp); 809 | uint16_t last_pos = acquire_last_pos(wctx, ptr); 810 | int rec_count = read_uint16(ptr + 3); 811 | byte *data_ptr; 812 | uint16_t rec_len; 813 | uint16_t hdr_len; 814 | byte *hdr_ptr = locate_column(wctx->buf + last_pos, col_idx, 815 | &data_ptr, &rec_len, &hdr_len, page_size - last_pos); 816 | if (hdr_ptr == NULL) 817 | return DBLOG_RES_MALFORMED; 818 | int8_t cur_len_of_len; 819 | uint16_t cur_len = dblog_derive_data_len(read_vint32(hdr_ptr, &cur_len_of_len)); 820 | uint16_t new_len = val == NULL ? 0 : (type == DBLOG_TYPE_REAL ? 8 : len); 821 | int32_t diff = new_len - cur_len; 822 | if (rec_len + diff + 2 > page_size - wctx->page_resv_bytes) 823 | return DBLOG_RES_TOO_LONG; 824 | uint16_t new_last_pos = last_pos + cur_len - new_len - LEN_OF_HDR_LEN; 825 | if (new_last_pos < (ptr - wctx->buf) + 9 + CHKSUM_LEN + rec_count * 2) { 826 | uint16_t prev_last_pos = read_uint16(ptr + 8 + (rec_count - 2) * 2); 827 | write_uint16(ptr + 3, rec_count - 1); 828 | write_uint16(ptr + 5, prev_last_pos); 829 | saveChecksumBytes(ptr, prev_last_pos); 830 | int res = write_page(wctx, wctx->cur_write_page, page_size); 831 | if (res) 832 | return res; 833 | restoreChecksumBytes(ptr, prev_last_pos); 834 | wctx->cur_write_page++; 835 | init_bt_tbl_leaf(wctx->buf); 836 | int8_t len_of_rowid; 837 | read_vint32(wctx->buf + last_pos + 3, &len_of_rowid); 838 | memmove(wctx->buf + page_size - wctx->page_resv_bytes 839 | - len_of_rowid - rec_len - LEN_OF_REC_LEN, 840 | wctx->buf + last_pos, len_of_rowid + rec_len + LEN_OF_REC_LEN); 841 | hdr_ptr -= last_pos; 842 | data_ptr -= last_pos; 843 | last_pos = page_size - wctx->page_resv_bytes - len_of_rowid - rec_len - LEN_OF_REC_LEN; 844 | hdr_ptr += last_pos; 845 | data_ptr += last_pos; 846 | rec_count = 1; 847 | write_uint16(ptr + 3, rec_count); 848 | write_uint16(ptr + 5, last_pos); 849 | } 850 | 851 | // make (or reduce) space and copy data 852 | new_last_pos = last_pos - diff; 853 | memmove(wctx->buf + new_last_pos, wctx->buf + last_pos, 854 | data_ptr - wctx->buf - last_pos); 855 | data_ptr -= diff; 856 | write_data(data_ptr, type, val, len); 857 | 858 | // make (or reduce) space and copy len 859 | uint32_t new_type_or_len = derive_col_type_or_len(type, val, new_len); 860 | int8_t new_len_of_len = get_vlen_of_uint32(new_type_or_len); 861 | int8_t hdr_diff = new_len_of_len - cur_len_of_len; 862 | diff += hdr_diff; 863 | if (hdr_diff) { 864 | memmove(wctx->buf + new_last_pos - hdr_diff, wctx->buf + new_last_pos, 865 | hdr_ptr - wctx->buf - last_pos); 866 | } 867 | write_vint32(hdr_ptr - diff, new_type_or_len); 868 | 869 | new_last_pos -= hdr_diff; 870 | write_rec_len_rowid_hdr_len(wctx->buf + new_last_pos, rec_len + diff, 871 | wctx->cur_write_rowid, hdr_len + hdr_diff); 872 | write_uint16(ptr + 5, new_last_pos); 873 | rec_count--; 874 | write_uint16(ptr + 8 + rec_count * 2, new_last_pos); 875 | wctx->state = DBLOG_ST_WRITE_PENDING; 876 | 877 | return DBLOG_RES_OK; 878 | } 879 | 880 | // See .h file for API description 881 | const void *dblog_get_col_val(struct dblog_write_context *wctx, 882 | int col_idx, uint32_t *out_col_type) { 883 | int32_t page_size = get_pagesize(wctx->page_size_exp); 884 | uint16_t last_pos = read_uint16(wctx->buf + 5); 885 | if (last_pos == 0) 886 | return NULL; 887 | if (last_pos > page_size - wctx->page_resv_bytes - 7) 888 | return NULL; 889 | return get_col_val(wctx->buf, last_pos, col_idx, 890 | out_col_type, page_size - wctx->page_resv_bytes - last_pos); 891 | } 892 | 893 | // See .h file for API description 894 | int dblog_flush(struct dblog_write_context *wctx) { 895 | int32_t page_size = get_pagesize(wctx->page_size_exp); 896 | int res = write_page(wctx, wctx->cur_write_page, page_size); 897 | if (res) 898 | return res; 899 | int ret = wctx->flush_fn(wctx); 900 | if (!ret) 901 | wctx->state = DBLOG_ST_WRITE_NOT_PENDING; 902 | return ret; 903 | } 904 | 905 | // See .h file for API description 906 | int dblog_partial_finalize(struct dblog_write_context *wctx) { 907 | int res; 908 | if (wctx->state == DBLOG_ST_WRITE_PENDING) { 909 | res = dblog_flush(wctx); 910 | if (res) 911 | return res; 912 | } 913 | int32_t page_size = get_pagesize(wctx->page_size_exp); 914 | res = read_bytes_wctx(wctx, wctx->buf, 0, page_size); 915 | if (res) 916 | return res; 917 | if (memcmp(wctx->buf, sqlite_sig, 16) == 0) 918 | return DBLOG_RES_OK; 919 | uint32_t last_leaf_page = read_uint32(wctx->buf + 60); 920 | // Update the last page no. in first page 921 | if (last_leaf_page == 0) { 922 | if (!wctx->cur_write_page) { 923 | byte head_buf[8]; 924 | do { 925 | res = read_bytes_wctx(wctx, head_buf, (wctx->cur_write_page + 1) * page_size, 8); 926 | if (res) 927 | break; 928 | if (head_buf[0] == 13) 929 | wctx->cur_write_page++; 930 | } while (head_buf[0] == 13); 931 | } 932 | if (wctx->cur_write_page) { 933 | write_uint32(wctx->buf + 60, wctx->cur_write_page); 934 | res = write_page(wctx, 0, page_size); 935 | if (res) 936 | return res; 937 | } else 938 | return DBLOG_RES_MALFORMED; 939 | } 940 | return DBLOG_RES_OK; 941 | } 942 | 943 | // See .h file for API description 944 | int dblog_finalize(struct dblog_write_context *wctx) { 945 | 946 | int res = dblog_partial_finalize(wctx); 947 | if (res) 948 | return res; 949 | 950 | if (memcmp(wctx->buf, sqlite_sig, 16) == 0) 951 | return DBLOG_RES_OK; 952 | 953 | int32_t page_size = get_pagesize(wctx->page_size_exp); 954 | uint32_t next_level_cur_pos = wctx->cur_write_page + 1; 955 | uint32_t next_level_begin_pos = next_level_cur_pos; 956 | uint32_t cur_level_pos = 1; 957 | uint32_t rowid; 958 | while (wctx->cur_write_page != 1) { 959 | init_bt_tbl_inner(wctx->buf); 960 | while (cur_level_pos < next_level_begin_pos) { 961 | res = get_last_rowid(wctx, cur_level_pos, page_size, &rowid); 962 | if (res) { 963 | cur_level_pos++; 964 | if (res == DBLOG_RES_INV_CHKSUM) 965 | continue; 966 | else 967 | break; 968 | } 969 | if (add_rec_to_inner_tbl(wctx, wctx->buf, rowid, cur_level_pos)) { 970 | res = write_page(wctx, next_level_cur_pos, page_size); 971 | if (res) 972 | return res; 973 | next_level_cur_pos++; 974 | init_bt_tbl_inner(wctx->buf); 975 | } 976 | cur_level_pos++; 977 | } 978 | uint16_t rec_count = read_uint16(wctx->buf + 3); 979 | if (rec_count) { // remove last row and write as right most pointer 980 | write_uint32(wctx->buf + 8, cur_level_pos); 981 | rec_count--; 982 | write_vint32(wctx->buf + 12 + rec_count * 2, rowid); 983 | write_uint16(wctx->buf + 3, rec_count); 984 | if (rec_count) { 985 | write_uint16(wctx->buf + 5, 986 | read_uint16(wctx->buf + 12 + (rec_count - 1) * 2)); 987 | } else 988 | write_uint16(wctx->buf + 5, 0); 989 | res = write_page(wctx, next_level_cur_pos, page_size); 990 | if (res) 991 | return res; 992 | next_level_cur_pos++; 993 | } 994 | if (next_level_begin_pos == next_level_cur_pos - 1) 995 | break; 996 | else { 997 | cur_level_pos = next_level_begin_pos; 998 | next_level_begin_pos = next_level_cur_pos; 999 | } 1000 | } 1001 | 1002 | res = read_bytes_wctx(wctx, wctx->buf, 0, page_size); 1003 | if (res) 1004 | return res; 1005 | byte *data_ptr = locate_col_root_page(wctx->buf, page_size - wctx->page_resv_bytes); 1006 | if (data_ptr == NULL) 1007 | return DBLOG_RES_MALFORMED; 1008 | write_uint32(data_ptr, next_level_cur_pos); // update root_page 1009 | write_uint32(wctx->buf + 28, next_level_cur_pos); // update page_count 1010 | memcpy(wctx->buf, sqlite_sig, 16); 1011 | res = write_page(wctx, 0, page_size); 1012 | if (res) 1013 | return res; 1014 | 1015 | return DBLOG_RES_OK; 1016 | } 1017 | 1018 | // See .h file for API description 1019 | int dblog_not_finalized(struct dblog_write_context *wctx) { 1020 | int res = read_bytes_wctx(wctx, wctx->buf, 0, 72); 1021 | if (res) 1022 | return res; 1023 | if (memcmp(wctx->buf, sqlite_sig, 16) == 0) 1024 | return DBLOG_RES_OK; 1025 | return DBLOG_RES_NOT_FINALIZED; 1026 | } 1027 | 1028 | int32_t dblog_read_page_size(struct dblog_write_context *wctx) { 1029 | int res = read_bytes_wctx(wctx, wctx->buf, 0, 72); 1030 | if (res) 1031 | return res; 1032 | if (check_signature(wctx->buf)) 1033 | return DBLOG_RES_INVALID_SIG; 1034 | int32_t page_size = read_uint16(wctx->buf + 16); 1035 | wctx->page_size_exp = get_page_size_exp(page_size); 1036 | if (!wctx->page_size_exp) 1037 | return DBLOG_RES_INVALID_SIG; 1038 | if (page_size == 1) 1039 | return 65536; 1040 | return page_size; 1041 | } 1042 | 1043 | // See .h file for API description 1044 | int dblog_recover(struct dblog_write_context *wctx) { 1045 | wctx->state = DBLOG_ST_TO_RECOVER; 1046 | wctx->cur_write_page = 0; 1047 | int res = dblog_finalize(wctx); 1048 | if (res) 1049 | return res; 1050 | return DBLOG_RES_OK; 1051 | } 1052 | 1053 | // See .h file for API description 1054 | int dblog_init_for_append(struct dblog_write_context *wctx) { 1055 | int res = read_bytes_wctx(wctx, wctx->buf, 0, 72); 1056 | if (res) 1057 | return res; 1058 | if (check_signature(wctx->buf)) 1059 | return DBLOG_RES_INVALID_SIG; 1060 | int32_t page_size = read_uint16(wctx->buf + 16); 1061 | if (page_size == 1) 1062 | page_size = 65536; 1063 | wctx->page_size_exp = get_page_size_exp(page_size); 1064 | if (!wctx->page_size_exp) 1065 | return DBLOG_RES_MALFORMED; 1066 | res = read_bytes_wctx(wctx, wctx->buf, 0, page_size); 1067 | if (res) 1068 | return res; 1069 | wctx->cur_write_page = read_uint32(wctx->buf + 60); 1070 | if (wctx->cur_write_page == 0) 1071 | return DBLOG_RES_NOT_FINALIZED; 1072 | memcpy(wctx->buf, dblog_sig, 16); 1073 | write_uint32(wctx->buf + 60, 0); 1074 | res = write_page(wctx, 0, page_size); 1075 | if (res) 1076 | return res; 1077 | res = get_last_rowid(wctx, wctx->cur_write_page, page_size, &wctx->cur_write_rowid); 1078 | if (res) 1079 | return res; 1080 | res = read_bytes_wctx(wctx, wctx->buf, wctx->cur_write_page * page_size, page_size); 1081 | if (res) 1082 | return res; 1083 | wctx->state = DBLOG_ST_WRITE_NOT_PENDING; 1084 | return DBLOG_RES_OK; 1085 | } 1086 | 1087 | // Reads current page 1088 | int read_cur_page(struct dblog_read_context *rctx) { 1089 | int32_t page_size = get_pagesize(rctx->page_size_exp); 1090 | int res = read_bytes_rctx(rctx, rctx->buf, rctx->cur_page * page_size, page_size); 1091 | if (res) 1092 | return res; 1093 | if (rctx->buf[0] != 13) 1094 | return DBLOG_RES_NOT_FOUND; 1095 | return DBLOG_RES_OK; 1096 | } 1097 | 1098 | // See .h file for API description 1099 | int dblog_read_init(struct dblog_read_context *rctx) { 1100 | int res = read_bytes_rctx(rctx, rctx->buf, 0, 72); 1101 | if (res) 1102 | return res; 1103 | if (check_signature(rctx->buf)) 1104 | return DBLOG_RES_INVALID_SIG; 1105 | int32_t page_size = read_uint16(rctx->buf + 16); 1106 | rctx->page_size_exp = get_page_size_exp(page_size); 1107 | if (!rctx->page_size_exp) 1108 | return DBLOG_RES_INVALID_SIG; 1109 | rctx->page_resv_bytes = read_uint8(rctx->buf + 20); 1110 | rctx->last_leaf_page = read_uint32(rctx->buf + 60); 1111 | rctx->cur_page = 0; 1112 | rctx->root_page = 0; // to be read when needed 1113 | return DBLOG_RES_OK; 1114 | } 1115 | 1116 | // See .h file for API description 1117 | int dblog_cur_row_col_count(struct dblog_read_context *rctx) { 1118 | uint16_t rec_data_pos = read_uint16(rctx->buf + 8 + rctx->cur_rec_pos * 2); 1119 | int8_t vint_len; 1120 | byte *ptr = rctx->buf + rec_data_pos + LEN_OF_REC_LEN; 1121 | read_vint32(ptr, &vint_len); 1122 | ptr += vint_len; 1123 | uint16_t hdr_len = read_vint16(ptr, &vint_len); 1124 | int col_count = 0; 1125 | ptr += vint_len; 1126 | hdr_len -= vint_len; 1127 | while (hdr_len > 0) { 1128 | read_vint32(ptr, &vint_len); 1129 | ptr += vint_len; 1130 | hdr_len -= vint_len; 1131 | col_count++; 1132 | } 1133 | return col_count; 1134 | } 1135 | 1136 | // See .h file for API description 1137 | const void *dblog_read_col_val(struct dblog_read_context *rctx, 1138 | int col_idx, uint32_t *out_col_type) { 1139 | if (rctx->cur_page == 0) 1140 | dblog_read_first_row(rctx); 1141 | uint16_t rec_pos = read_uint16(rctx->buf + 8 + rctx->cur_rec_pos * 2); 1142 | return get_col_val(rctx->buf, rec_pos, col_idx, out_col_type, 1143 | get_pagesize(rctx->page_size_exp) - rec_pos); 1144 | } 1145 | 1146 | // See .h file for API description 1147 | const int8_t col_data_lens[] = {0, 1, 2, 3, 4, 6, 8, 8}; 1148 | uint32_t dblog_derive_data_len(uint32_t col_type_or_len) { 1149 | if (col_type_or_len >= 12) { 1150 | if (col_type_or_len % 2) 1151 | return (col_type_or_len - 13)/2; 1152 | return (col_type_or_len - 12)/2; 1153 | } else 1154 | if (col_type_or_len < 8) 1155 | return col_data_lens[col_type_or_len]; 1156 | return 0; 1157 | } 1158 | 1159 | // See .h file for API description 1160 | int dblog_read_first_row(struct dblog_read_context *rctx) { 1161 | rctx->cur_page = 1; 1162 | if (read_cur_page(rctx)) 1163 | return DBLOG_RES_NOT_FOUND; 1164 | rctx->cur_rec_pos = 0; 1165 | return DBLOG_RES_OK; 1166 | } 1167 | 1168 | // See .h file for API description 1169 | int dblog_read_next_row(struct dblog_read_context *rctx) { 1170 | uint16_t rec_count = read_uint16(rctx->buf + 3); 1171 | rctx->cur_rec_pos++; 1172 | if (rctx->cur_rec_pos == rec_count) { 1173 | int32_t page_size = get_pagesize(rctx->page_size_exp); 1174 | rctx->cur_page++; 1175 | if (read_cur_page(rctx)) 1176 | return DBLOG_RES_NOT_FOUND; 1177 | rctx->cur_rec_pos = 0; 1178 | } 1179 | return DBLOG_RES_OK; 1180 | } 1181 | 1182 | // See .h file for API description 1183 | int dblog_read_prev_row(struct dblog_read_context *rctx) { 1184 | if (rctx->cur_rec_pos == 0) { 1185 | if (rctx->cur_page == 1) 1186 | return DBLOG_RES_NOT_FOUND; 1187 | rctx->cur_page--; 1188 | if (read_cur_page(rctx)) 1189 | return DBLOG_RES_NOT_FOUND; 1190 | rctx->cur_rec_pos = read_uint16(rctx->buf + 3); 1191 | } 1192 | rctx->cur_rec_pos--; 1193 | return DBLOG_RES_OK; 1194 | } 1195 | 1196 | // See .h file for API description 1197 | int dblog_read_last_row(struct dblog_read_context *rctx) { 1198 | if (rctx->last_leaf_page == 0) 1199 | return DBLOG_RES_NOT_FINALIZED; 1200 | rctx->cur_page = rctx->last_leaf_page; 1201 | if (read_cur_page(rctx)) 1202 | return DBLOG_RES_NOT_FOUND; 1203 | rctx->cur_rec_pos = read_uint16(rctx->buf + 3) - 1; 1204 | return DBLOG_RES_OK; 1205 | } 1206 | 1207 | // Returns the Row ID of the last record stored in the given leaf page 1208 | // Reads the buffer part by part to avoid reading entire buffer into memory 1209 | // to support low memory systems (2kb ram) 1210 | // The underlying callback function hopefully optimizes repeated IO 1211 | int read_last_val(struct dblog_read_context *rctx, uint32_t pos, 1212 | int32_t page_size, int col_idx, byte *val_at, int val_len, 1213 | uint32_t *out_col_type, uint16_t *out_rec_pos, byte is_rowid) { 1214 | byte src_buf[12]; 1215 | int res = read_bytes_rctx(rctx, src_buf, pos * page_size, 12); 1216 | if (res) 1217 | return res; 1218 | if (*src_buf != 13) 1219 | return DBLOG_RES_MALFORMED; 1220 | *out_rec_pos = read_uint16(src_buf + 3) - 1; 1221 | uint16_t last_pos = read_uint16(src_buf + 5); 1222 | res = read_bytes_rctx(rctx, src_buf, pos * page_size + last_pos, 12); 1223 | if (res) 1224 | return res; 1225 | int8_t vint_len; 1226 | uint32_t u32 = read_vint32(src_buf + 3, &vint_len); 1227 | if (is_rowid) 1228 | *out_col_type = u32; 1229 | else { 1230 | uint16_t rec_len = read_vint16(src_buf, NULL) + vint_len + LEN_OF_REC_LEN; 1231 | byte rec_buf[rec_len]; 1232 | res = read_bytes_rctx(rctx, rec_buf, pos * page_size + last_pos, rec_len); 1233 | if (res) 1234 | return res; 1235 | uint16_t hdr_len; 1236 | byte *data_ptr; 1237 | byte *hdr_ptr = locate_column(rec_buf, col_idx, &data_ptr, &rec_len, &hdr_len, rec_len); 1238 | if (!hdr_ptr) 1239 | return DBLOG_RES_MALFORMED; 1240 | *out_col_type = read_vint32(hdr_ptr, &vint_len); 1241 | u32 = dblog_derive_data_len(*out_col_type); 1242 | if (u32 > val_len) 1243 | u32 = val_len; 1244 | memcpy(val_at, data_ptr, u32); 1245 | } 1246 | return DBLOG_RES_OK; 1247 | } 1248 | 1249 | byte *read_val_at(struct dblog_read_context *rctx, uint32_t pos, int col_idx, 1250 | uint32_t *out_col_type, byte is_rowid) { 1251 | int8_t vint_len; 1252 | uint16_t rec_pos = read_uint16(rctx->buf + 8 + pos * 2); 1253 | uint32_t row_id = read_vint32(rctx->buf + rec_pos + LEN_OF_REC_LEN, &vint_len); 1254 | if (is_rowid) { 1255 | *out_col_type = row_id; 1256 | return (byte *) out_col_type; 1257 | } else { 1258 | uint16_t hdr_len; 1259 | uint16_t rec_len; 1260 | byte *data_ptr; 1261 | byte *hdr_ptr; 1262 | hdr_ptr = locate_column(rctx->buf + rec_pos, col_idx, &data_ptr, &rec_len, 1263 | &hdr_len, get_pagesize(rctx->page_size_exp) - rec_pos); 1264 | if (!hdr_ptr) 1265 | return NULL; 1266 | *out_col_type = read_vint32(hdr_ptr, &vint_len); 1267 | return data_ptr; 1268 | } 1269 | return NULL; 1270 | } 1271 | 1272 | // Returns the Row ID of the record at given position 1273 | uint32_t read_rowid_at(struct dblog_read_context *rctx, uint32_t rec_pos) { 1274 | int8_t vint_len; 1275 | return read_vint32(rctx->buf 1276 | + read_uint16(rctx->buf + (*rctx->buf == 13 ? 8 : 12) + rec_pos * 2) 1277 | + (*rctx->buf == 13 ? LEN_OF_REC_LEN : 4), &vint_len); 1278 | } 1279 | 1280 | int read_root_page_no(struct dblog_read_context *rctx, int32_t page_size) { 1281 | if (rctx->root_page) 1282 | return rctx->root_page; 1283 | int res = read_bytes_rctx(rctx, rctx->buf, 0, page_size); 1284 | if (res) 1285 | return res; 1286 | byte *data_ptr = locate_col_root_page(rctx->buf, page_size - rctx->page_resv_bytes); 1287 | if (data_ptr == NULL) 1288 | return DBLOG_RES_MALFORMED; 1289 | rctx->root_page = read_uint32(data_ptr); 1290 | return DBLOG_RES_OK; 1291 | } 1292 | 1293 | // See .h file for API description 1294 | int dblog_srch_row_by_id(struct dblog_read_context *rctx, uint32_t rowid) { 1295 | if (rctx->last_leaf_page == 0) 1296 | return DBLOG_RES_NOT_FINALIZED; 1297 | int32_t page_size = get_pagesize(rctx->page_size_exp); 1298 | int res = read_root_page_no(rctx, page_size); 1299 | if (res) 1300 | return res; 1301 | uint32_t srch_page = rctx->root_page; 1302 | if (!srch_page) 1303 | return DBLOG_RES_NOT_FINALIZED; 1304 | do { 1305 | srch_page--; 1306 | int res = read_bytes_rctx(rctx, rctx->buf, srch_page * page_size, page_size); 1307 | if (res) 1308 | return res; 1309 | uint32_t middle, first, size; 1310 | first = 0; 1311 | size = read_uint16(rctx->buf + 3); 1312 | while (first < size) { 1313 | middle = (first + size) >> 1; 1314 | uint32_t rowid_at = read_rowid_at(rctx, middle); 1315 | if (rowid_at < rowid) 1316 | first = middle + 1; 1317 | else if (rowid_at > rowid) 1318 | size = middle; 1319 | else { 1320 | if (*rctx->buf == 5) { 1321 | size = middle; 1322 | break; 1323 | } else { 1324 | rctx->cur_page = srch_page; 1325 | rctx->cur_rec_pos = middle; 1326 | return DBLOG_RES_OK; 1327 | } 1328 | } 1329 | } 1330 | if (*rctx->buf == 5) { 1331 | if (first == read_uint16(rctx->buf + 3)) 1332 | srch_page = read_uint32(rctx->buf + 8); 1333 | else 1334 | srch_page = read_uint32(rctx->buf + read_uint16(rctx->buf + 12 + size * 2)); 1335 | } 1336 | } while (*rctx->buf == 5); 1337 | return DBLOG_RES_NOT_FOUND; 1338 | } 1339 | 1340 | // compares two binary strings and returns k, -k or 0 1341 | // 0 meaning the two are identical. If not, k is length that matches 1342 | int compare_bin(const byte *v1, byte len1, const byte *v2, byte len2) { 1343 | int k = 0; 1344 | int lim = (len2 < len1 ? len2 : len1); 1345 | while (k < lim) { 1346 | byte c1 = v1[k]; 1347 | byte c2 = v2[k]; 1348 | k++; 1349 | if (c1 < c2) 1350 | return -k; 1351 | else if (c1 > c2) 1352 | return k; 1353 | } 1354 | if (len1 == len2) 1355 | return 0; 1356 | k++; 1357 | return (len1 < len2 ? -k : k); 1358 | } 1359 | 1360 | // Converts any type of integer to int64 for comparison 1361 | int64_t convert_to_i64(byte *val, int len, byte is_big_endian) { 1362 | int64_t ival_at = 0; 1363 | switch (len) { 1364 | case 4: { 1365 | uint32_t u32_val_at = (is_big_endian ? read_uint32(val) : *((uint32_t *) val)); 1366 | if (u32_val_at & 0x80000000) { 1367 | ival_at = 0xFFFFFFFF - u32_val_at; 1368 | ival_at = 0xFFFFFFFFFFFFFFFF - ival_at; 1369 | } else 1370 | ival_at = u32_val_at; 1371 | break; 1372 | } 1373 | case 2: { 1374 | uint16_t u16_val_at = (is_big_endian ? read_uint16(val) : *((uint16_t *) val)); 1375 | ival_at = u16_val_at & 0x7FFF; 1376 | if (u16_val_at & 0x8000) { 1377 | ival_at = 0xFFFF - u16_val_at; 1378 | ival_at = 0xFFFFFFFFFFFFFFFF - ival_at; 1379 | } else 1380 | ival_at = u16_val_at; 1381 | break; 1382 | } 1383 | case 6: 1384 | case 8: 1385 | ival_at = (is_big_endian ? read_uint64(val) : *((int64_t *) val)); 1386 | break; 1387 | case 1: { 1388 | uint8_t u8_val_at = (is_big_endian ? read_uint8(val) : *((uint8_t *) val)); 1389 | ival_at = u8_val_at & 0x7F; 1390 | if (u8_val_at & 0x80) { 1391 | ival_at = 0xFF - u8_val_at; 1392 | ival_at = 0xFFFFFFFFFFFFFFFF - ival_at; 1393 | } else 1394 | ival_at = u8_val_at; 1395 | } 1396 | } 1397 | return ival_at; 1398 | } 1399 | 1400 | // Compare values for binary search and return 1, -1 or 0 1401 | int compare_values(byte *val_at, uint32_t u32_at, int val_type, void *val, uint16_t len, byte is_rowid) { 1402 | if (is_rowid) 1403 | return (u32_at > *((uint32_t *) val) ? 1 : u32_at < *((uint32_t *) val) ? -1 : 0); 1404 | switch (val_type) { 1405 | case DBLOG_TYPE_INT: { 1406 | int64_t ival_at = convert_to_i64(val_at, u32_at, 1); 1407 | int64_t ival = convert_to_i64(val, len, 0); 1408 | return ival_at > ival ? 1 : (ival_at < ival ? -1 : 0); 1409 | } 1410 | case DBLOG_TYPE_REAL: 1411 | if ((len != 4 && len != 8) || u32_at != 7) 1412 | return DBLOG_RES_TYPE_MISMATCH; 1413 | int64_t bytes64, bytes64_at; 1414 | bytes64_at = read_uint64(val_at); 1415 | if (len == 4) 1416 | bytes64 = float_to_double(val); 1417 | else 1418 | bytes64 = *((int64_t *) val); 1419 | return (bytes64_at > bytes64 ? 1 : bytes64_at < bytes64 ? -1 : 0); 1420 | case DBLOG_TYPE_BLOB: 1421 | case DBLOG_TYPE_TEXT: { 1422 | uint32_t len_at = dblog_derive_data_len(u32_at); 1423 | int res = compare_bin(val_at, len_at, val, len); 1424 | return (res > 0 ? 1 : (res < 0 ? -1 : 0)); 1425 | } 1426 | } 1427 | return 1; 1428 | } 1429 | 1430 | // See .h file for API description 1431 | int dblog_bin_srch_row_by_val(struct dblog_read_context *rctx, int col_idx, 1432 | int val_type, void *val, uint16_t len, byte is_rowid) { 1433 | int32_t page_size = get_pagesize(rctx->page_size_exp); 1434 | if (rctx->last_leaf_page == 0) 1435 | return DBLOG_RES_NOT_FINALIZED; 1436 | uint32_t middle, first, size; 1437 | int res; 1438 | first = 1; 1439 | size = rctx->last_leaf_page + 1; 1440 | while (first < size) { 1441 | middle = (first + size) >> 1; 1442 | uint16_t rec_pos; 1443 | byte val_at[len + 1]; 1444 | uint32_t u32_at; 1445 | res = read_last_val(rctx, middle, page_size, col_idx, 1446 | val_at, len + 1, &u32_at, &rec_pos, is_rowid); 1447 | if (res) 1448 | return res; 1449 | int cmp = compare_values(val_at, u32_at, val_type, val, len, is_rowid); 1450 | if (cmp == DBLOG_RES_TYPE_MISMATCH) 1451 | return cmp; 1452 | if (cmp < 0) 1453 | first = middle + 1; 1454 | else if (cmp > 0) 1455 | size = middle; 1456 | else { 1457 | rctx->cur_page = middle; 1458 | rctx->cur_rec_pos = rec_pos; 1459 | res = read_bytes_rctx(rctx, rctx->buf, middle * page_size, page_size); 1460 | if (res) 1461 | return res; 1462 | return DBLOG_RES_OK; 1463 | } 1464 | } 1465 | if (size == rctx->last_leaf_page + 1) 1466 | size--; 1467 | uint32_t found_at_page = size; 1468 | res = read_bytes_rctx(rctx, rctx->buf, size * page_size, page_size); 1469 | if (res) 1470 | return res; 1471 | first = 0; 1472 | int16_t rec_count = read_uint16(rctx->buf + 3) - 1; 1473 | size = rec_count; 1474 | while (first < size) { 1475 | middle = (first + size) >> 1; 1476 | uint32_t u32_at; 1477 | byte *val_at = read_val_at(rctx, middle, col_idx, &u32_at, is_rowid); 1478 | if (!val_at) 1479 | return DBLOG_RES_NOT_FOUND; 1480 | int cmp = compare_values(val_at, u32_at, val_type, val, len, is_rowid); 1481 | if (cmp == DBLOG_RES_TYPE_MISMATCH) 1482 | return cmp; 1483 | if (cmp < 0) 1484 | first = middle + 1; 1485 | else if (cmp > 0) 1486 | size = middle; 1487 | else { 1488 | rctx->cur_page = found_at_page; 1489 | rctx->cur_rec_pos = middle; 1490 | return DBLOG_RES_OK; 1491 | } 1492 | } 1493 | rctx->cur_page = found_at_page; 1494 | rctx->cur_rec_pos = size; 1495 | return DBLOG_RES_OK; 1496 | } 1497 | 1498 | // See .h file for API description 1499 | int dblog_upd_col_val(struct dblog_read_context *rctx, int col_idx, const void *val) { 1500 | uint8_t *buf = rctx->buf; 1501 | if (buf[0] != 13) 1502 | return DBLOG_RES_ERR; 1503 | int16_t rec_count = read_uint16(rctx->buf + 3); 1504 | if (rec_count <= rctx->cur_rec_pos) 1505 | return DBLOG_RES_ERR; 1506 | uint32_t u32_at; 1507 | byte *val_at = read_val_at(rctx, rctx->cur_rec_pos, col_idx, &u32_at, 0); 1508 | if (!val_at) 1509 | return DBLOG_RES_NOT_FOUND; 1510 | int len = dblog_derive_data_len(u32_at); 1511 | write_data(val_at, derive_col_type(u32_at), val, len); 1512 | return DBLOG_RES_OK; 1513 | } 1514 | 1515 | int dblog_write_cur_page(struct dblog_read_context *rctx, write_fn_def write_fn) { 1516 | struct dblog_write_context wctx; 1517 | wctx.buf = rctx->buf; 1518 | wctx.write_fn = write_fn; 1519 | return write_page(&wctx, rctx->cur_page, get_pagesize(rctx->page_size_exp)); 1520 | } 1521 | -------------------------------------------------------------------------------- /src/ulog_sqlite.h: -------------------------------------------------------------------------------- 1 | /* 2 | Sqlite Micro Logger 3 | 4 | Fast and Lean Sqlite database logger targetting 5 | low RAM systems such as Microcontrollers. 6 | 7 | This Library can work on systems that have as little as 2kb, 8 | such as the ATMega328 MCU. It is available for the Arduino platform. 9 | 10 | https://github.com/siara-cc/sqlite_micro_logger 11 | 12 | Copyright @ 2019 Arundale Ramanathan, Siara Logics (cc) 13 | 14 | Licensed under the Apache License, Version 2.0 (the "License"); 15 | you may not use this file except in compliance with the License. 16 | You may obtain a copy of the License at 17 | 18 | http://www.apache.org/licenses/LICENSE-2.0 19 | 20 | Unless required by applicable law or agreed to in writing, software 21 | distributed under the License is distributed on an "AS IS" BASIS, 22 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 23 | See the License for the specific language governing permissions and 24 | limitations under the License. 25 | */ 26 | 27 | #ifndef __ULOG_SQLITE__ 28 | #define __ULOG_SQLITE__ 29 | 30 | // 0 - No calculation, no checking 31 | // 1 - Calculate, and check header during recovery, 32 | // skip pages where header checksum don't match 33 | #define DBLOG_CFG_WRITE_CHECKSUM 1 34 | 35 | // Not implemented yet 36 | // 0 - No checking 37 | // 1 - Check if page checksum matches everytime page is loaded 38 | // and whether first record checksum matches during binary search 39 | #define DBLOG_CFG_READ_CHECKSUM 0 40 | 41 | #ifdef __cplusplus 42 | extern "C" { 43 | #endif 44 | 45 | #include 46 | #include 47 | 48 | typedef unsigned char byte; 49 | 50 | enum {DBLOG_TYPE_INT = 1, DBLOG_TYPE_REAL, DBLOG_TYPE_BLOB, DBLOG_TYPE_TEXT}; 51 | 52 | enum {DBLOG_RES_OK = 0, DBLOG_RES_ERR = -1, DBLOG_RES_INV_PAGE_SZ = -2, 53 | DBLOG_RES_TOO_LONG = -3, DBLOG_RES_WRITE_ERR = -4, DBLOG_RES_FLUSH_ERR = -5}; 54 | 55 | enum {DBLOG_RES_SEEK_ERR = -6, DBLOG_RES_READ_ERR = -7, 56 | DBLOG_RES_INVALID_SIG = -8, DBLOG_RES_MALFORMED = -9, 57 | DBLOG_RES_NOT_FOUND = -10, DBLOG_RES_NOT_FINALIZED = -11, 58 | DBLOG_RES_TYPE_MISMATCH = -12, DBLOG_RES_INV_CHKSUM = -13}; 59 | 60 | // Write context to be passed to create / append 61 | // a database. The running values need not be supplied 62 | struct dblog_write_context { 63 | byte *buf; // working buffer of size page_size 64 | byte col_count; // No. of columns (whether fits into page is not checked) 65 | byte page_size_exp; // 9=512, 10=1024 and so on upto 16=65536 66 | byte max_pages_exp; // Maximum data pages (as exponent of 2) after which 67 | // to roll. 0 means no max. Not implemented yet. 68 | byte page_resv_bytes; // Reserved bytes at end of every page (say checksum) 69 | // read_fn and write_fn should return no. of bytes read or written 70 | int32_t (*read_fn)(struct dblog_write_context *ctx, void *buf, uint32_t pos, size_t len); 71 | int32_t (*write_fn)(struct dblog_write_context *ctx, void *buf, uint32_t pos, size_t len); 72 | int (*flush_fn)(struct dblog_write_context *ctx); // Success if returns 0 73 | // following are running values used internally 74 | uint32_t cur_write_page; 75 | uint32_t cur_write_rowid; 76 | byte state; 77 | int err_no; 78 | }; 79 | 80 | typedef int32_t (*write_fn_def)(struct dblog_write_context *ctx, void *buf, uint32_t pos, size_t len); 81 | 82 | // Initializes database - writes first page 83 | // and makes it ready for writing data 84 | int dblog_write_init(struct dblog_write_context *wctx); 85 | 86 | // Initializes database - writes first page 87 | // and makes it ready for writing data 88 | // Uses the given table name and DDL script 89 | // Table name should match that given in script 90 | int dblog_write_init_with_script(struct dblog_write_context *wctx, 91 | char *table_name, char *table_script); 92 | 93 | // Initalizes database - resets signature on first page 94 | // positions at last page for writing 95 | // If this returns DBLOG_RES_NOT_FINALIZED, 96 | // call dblog_finalize() to first finalize the database 97 | int dblog_init_for_append(struct dblog_write_context *wctx); 98 | 99 | // Creates new record with all columns null 100 | // If no more space in page, writes it to disk 101 | // creates new page, and creates a new record 102 | int dblog_append_empty_row(struct dblog_write_context *wctx); 103 | 104 | // Creates new record with given column values 105 | // If no more space in page, writes it to disk 106 | // creates new page, and creates a new record 107 | int dblog_append_row_with_values(struct dblog_write_context *wctx, 108 | uint8_t types[], const void *values[], uint16_t lengths[]); 109 | 110 | // Sets value of column in the current record for the given column index 111 | // If no more space in page, writes it to disk 112 | // creates new page, and moves the row to new page 113 | int dblog_set_col_val(struct dblog_write_context *wctx, int col_idx, 114 | int type, const void *val, uint16_t len); 115 | 116 | // Gets the value of the column for the current record 117 | // Can be used to retrieve the value of the column 118 | // set by dblog_set_col_val 119 | const void *dblog_get_col_val(struct dblog_write_context *wctx, int col_idx, uint32_t *out_col_type); 120 | 121 | // Flushes the corrent page to disk 122 | // Page is written only when it becomes full 123 | // If it needs to be written for each record or column, 124 | // this can be used 125 | int dblog_flush(struct dblog_write_context *wctx); 126 | 127 | // Flushes data written so far and Updates the last leaf page number 128 | // in the first page to enable Binary Search 129 | int dblog_partial_finalize(struct dblog_write_context *wctx); 130 | 131 | // Based on the data written so far, forms Interior B-Tree pages 132 | // according to SQLite format and update the root page number 133 | // in the first page. 134 | int dblog_finalize(struct dblog_write_context *wctx); 135 | 136 | // Returns 1 if the database is in unfinalized state 137 | int dblog_not_finalized(struct dblog_write_context *wctx); 138 | 139 | // Reads page size from database if not known 140 | int32_t dblog_read_page_size(struct dblog_write_context *wctx); 141 | 142 | // Recovers database pointed by given context 143 | // and finalizes it 144 | int dblog_recover(struct dblog_write_context *wctx); 145 | 146 | // Read context to be passed to read from a database created using this library. 147 | // The running values need not be supplied 148 | struct dblog_read_context { 149 | byte *buf; 150 | // read_fn should return no. of bytes read 151 | int32_t (*read_fn)(struct dblog_read_context *ctx, void *buf, uint32_t pos, size_t len); 152 | // following are running values used internally 153 | uint32_t last_leaf_page; 154 | uint32_t root_page; 155 | uint32_t cur_page; 156 | uint16_t cur_rec_pos; 157 | byte page_size_exp; 158 | byte page_resv_bytes; 159 | }; 160 | 161 | // Reads a database created using this library, 162 | // checks signature and positions at the first record. 163 | // Cannot be used to read SQLite databases 164 | // not created using this library or modified using other libraries 165 | int dblog_read_init(struct dblog_read_context *rctx); 166 | 167 | // Returns number of columns in the current record 168 | int dblog_cur_row_col_count(struct dblog_read_context *rctx); 169 | 170 | // Returns value of column at given index. 171 | // Also returns type of column in (out_col_type) according to record format 172 | // See https://www.sqlite.org/fileformat.html#record_format 173 | // For text and blob columns, pass the type to dblog_derive_data_len() 174 | // to get the actual length 175 | const void *dblog_read_col_val(struct dblog_read_context *rctx, int col_idx, uint32_t *out_col_type); 176 | 177 | // For text and blob columns, pass the out_col_type 178 | // returned by dblog_read_col_val() to get the actual length 179 | uint32_t dblog_derive_data_len(uint32_t col_type); 180 | 181 | // Positions current position at first record 182 | int dblog_read_first_row(struct dblog_read_context *rctx); 183 | 184 | // Positions current position at next record 185 | int dblog_read_next_row(struct dblog_read_context *rctx); 186 | 187 | // Positions current position at previous record 188 | int dblog_read_prev_row(struct dblog_read_context *rctx); 189 | 190 | // Positions current position at last record 191 | // The database should have been finalized 192 | // for this function to work 193 | int dblog_read_last_row(struct dblog_read_context *rctx); 194 | 195 | // Performs binary search on the inserted records 196 | // using the given Row ID and positions at the record found 197 | // Does not change position if record not found 198 | int dblog_srch_row_by_id(struct dblog_read_context *rctx, uint32_t rowid); 199 | 200 | // Performs binary search on the inserted records 201 | // using the given Value and positions at the record found 202 | // Changes current position to closest match, if record not found 203 | // is_rowid = 1 is used to do Binary Search by RowId 204 | int dblog_bin_srch_row_by_val(struct dblog_read_context *rctx, int col_idx, 205 | int val_type, void *val, uint16_t len, byte is_rowid); 206 | 207 | // Updates value of column at current position 208 | // For text and blob columns, pass the type to dblog_derive_data_len() 209 | // to get the actual length 210 | int dblog_upd_col_val(struct dblog_read_context *rctx, int col_idx, const void *val); 211 | 212 | // Writes the current page to disk 213 | // Typically called after updating values using dblog_upd_col_val() 214 | int dblog_write_cur_page(struct dblog_read_context *rctx, write_fn_def write_fn); 215 | 216 | #ifdef __cplusplus 217 | } 218 | #endif 219 | 220 | #endif 221 | -------------------------------------------------------------------------------- /uno_bin_srch_scr.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/siara-cc/sqlite_micro_logger_arduino/1069b5f5b6a0e4f2fe834731a0449d81965239e7/uno_bin_srch_scr.png -------------------------------------------------------------------------------- /uno_log_scr.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/siara-cc/sqlite_micro_logger_arduino/1069b5f5b6a0e4f2fe834731a0449d81965239e7/uno_log_scr.png --------------------------------------------------------------------------------