├── LICENSE ├── README.md ├── changelog.md ├── device_test.py ├── micropydatabase.py └── setup.py /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2019 sungkhum 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # MicroPyDatabase 2 | A low-memory json-based database for MicroPython. 3 | Data is stored in a folder structure in json for easy inspection. 4 | 5 | Install prerequisites: 6 | `micropython -m upip install micropython-os` 7 | `micropython -m upip install micropython-os.path` 8 | 9 | or 10 | ``` 11 | >>> import upip 12 | >>> upip.install("micropython-os”) 13 | >>> upip.install("micropython-os.path”) 14 | ``` 15 | 16 | Usage instructions : 17 | 18 | ``` 19 | import micropydatabase 20 | ``` 21 | Create a new database: 22 | ``` 23 | db_object = micropydatabase.Database.create("mydb") 24 | ``` 25 | Open an existing database: 26 | ``` 27 | db_object = micropydatabase.Database.open("mydb") 28 | ``` 29 | Create a new table (specifying column names [and types if you need it]): 30 | *(Table column definition supported types are **str**, **int** and **bool**. Default is **str**.)* 31 | ``` 32 | db_object = micropydatabase.Database.open("mydb") 33 | db_object.create_table("mytable", ["name", "password"]) 34 | db_object.create_table("mytable", { 35 | "name":str, 36 | "age":int, 37 | "isMember":bool 38 | }) 39 | ``` 40 | Insert data into table: 41 | ``` 42 | db_object = micropydatabase.Database.open("mydb") 43 | db_table = db_object.open_table("mytable") 44 | db_table.insert({"name": "lucy", "password": "coolpassword"}) # as dict 45 | db_table.insert(["Rose", "MySecret"]) # as list 46 | ``` 47 | Multi-insert data into table: 48 | ``` 49 | db_object = micropydatabase.Database.open("mydb") 50 | db_table = db_object.open_table("mytable") 51 | db_table.insert([ 52 | {"name": "john", "password": "apassword"}, 53 | {"name": "john", "password": "apassword"}, 54 | {"name": "bob", "password": "thispassword"}, 55 | {"name": "sally", "password": "anotherpassword"} 56 | ]) 57 | ``` 58 | Find (returns first result): 59 | ``` 60 | db_object = micropydatabase.Database.open("mydb") 61 | db_table = db_object.open_table("mytable") 62 | db_table.find({"name": "john", "password": "apassword"}) 63 | ``` 64 | Query (returns all results): 65 | ``` 66 | db_object = micropydatabase.Database.open("mydb") 67 | db_table = db_object.open_table("mytable") 68 | db_table.query({"name": "john", "password": "apassword"}) 69 | ``` 70 | Update Row (search query, updated row data): 71 | ``` 72 | db_object = micropydatabase.Database.open("mydb") 73 | db_table = db_object.open_table("mytable") 74 | db_table.update( 75 | {"name": "bob", "password": "thispassword"}, #find what 76 | {"name": "george", "password": "somethingelse"} # change with 77 | ) 78 | ``` 79 | Delete Row: 80 | ``` 81 | db_object = micropydatabase.Database.open("mydb") 82 | db_table = db_object.open_table("mytable") 83 | db_table.delete({"name": "george", "password": "somethingelse"}) 84 | ``` 85 | Scan (iterate through each row in table): 86 | ``` 87 | db_object = micropydatabase.Database.open("mydb") 88 | db_table = db_object.open_table("mytable") 89 | f = db_table.scan() 90 | f.__next__() 91 | ``` 92 | Scan with Query (iterate through rows that match query): 93 | ``` 94 | db_object = micropydatabase.Database.open("mydb") 95 | db_table = db_object.open_table("mytable") 96 | f = db_table.scan({"name": "john", "password": "apassword"}) 97 | f.__next__() 98 | ``` 99 | Truncate Table (delete all table contents): 100 | ``` 101 | db_object = micropydatabase.Database.open("mydb") 102 | db_table = db_object.open_table("mytable") 103 | db_table.truncate() 104 | ``` 105 | 106 | Vaccum Table (reorganize all content): 107 | ``` 108 | db_object = micropydatabase.Database.open("mydb") 109 | db_table = db_object.open_table("mytable") 110 | db_table.vaccum() 111 | ``` 112 | -------------------------------------------------------------------------------- /changelog.md: -------------------------------------------------------------------------------- 1 | # Changelog# 2 | 3 | ## 2021-10-18 ## 4 | > Breaking changes! 5 | * Shaved 8 bytes from each line in database, by changing `data` and `row_id` to `d` and `r`. This is incompatible with already existing data files. 6 | * Added database method `Database.exist("database_name")` for easy check. 7 | * changed string concatenation to use `format()` function, to save RAM. 8 | * Shortened exception texts. 9 | 10 | ## 2020-12-05 ## 11 | 12 | * Fixed method `Table.__return_query` so now querying by `tbl.query({"name": "Bob"})` works correctly. Expanded capability (read bellow). Backward compatibility left. **Note**: Value strings are Case Sensitive! 13 | * Make compatible with full python (windows, linux). 14 | * Table creattion method inherit `rows_per_page` and `max_rows` from database schema settings. 15 | * Do not increase `tbl.current_row` counter if data has not passed validation yet. 16 | * New method `tbl.vacuum()` to optimize pagefiles. Worth to use after some data has been deleted. 17 | * updated method `Table.__return_query` so now is possible to search by multiple keys and values. Imagine following **persons_table** data: 18 | | fname | lname | age | 19 | |---|---|---| 20 | | John | Smith | 37 | 21 | | Nicole | Smith | *None* | 22 | | Kim | Smith| 7 | 23 | | John | Lee | *None* | 24 | | Nicole | Lee | 32 | 25 | | Bart | Lee | 3 | 26 | We want to get John and Nicole Smiths, but dont want all Smiths and no Lees. Following query `tbl.query({"fname":["John", "Nicole"], "lname": "Smith"})` and here is the result: 27 | ``` python 28 | [{'fname': 'John', 'lname': 'Smith', 'age': 34}, 29 | {'fname': 'Nicole', 'lname': 'Smith', 'age': None}] 30 | ``` 31 | You may add any number of search parameter in your query and all of them will be `AND`. In sql definition upper search query represent following SQL: 32 | ```sql 33 | select * from persons_table where fname in ("John", "Nicole") and lname = "Smith" 34 | ``` 35 | * Changed `__insert_modify_data_file` so writing to files would be more efficient and faster in exchange in reliability if something unexpected happens. 36 | * Annotated all function's parameters 37 | * Fixed `tests/test.py` file to correcttly calculatee current row in `check_current_row()` function 38 | 39 | `test/test.py` and `device_test.py` passed succesfully on 40 | > MicroPython v1.9.4-2922-gce1dfde98-dirty on 2020-11-26; ESP32 module with ESP32 41 | -------------------------------------------------------------------------------- /device_test.py: -------------------------------------------------------------------------------- 1 | """ 2 | A file that contains tests of MicroPyDatabase features to ensure 3 | nothing is broken when updates are written. 4 | 5 | Run on device with: 6 | with open("device_test.py") as f: 7 | exec(f.read(), globals()) 8 | """ 9 | 10 | import micropydatabase as mdb 11 | import gc 12 | import time 13 | import sys 14 | 15 | # Is device microcontroller 16 | uC = True if sys.platform not in ("unix", "linux", "win32") else False 17 | 18 | 19 | def test_database_open_exception(): 20 | try: 21 | mdb.Database.open("dabatase_not_exist") 22 | except Exception: 23 | return 'Success.' 24 | else: 25 | return 'Error.' 26 | 27 | 28 | def test_database_creation(): 29 | try: 30 | gc.collect() 31 | if uC: 32 | before = gc.mem_free() 33 | start_time = time.ticks_ms() 34 | mdb.Database.create("testdb") 35 | gc.collect() 36 | if uC: 37 | after = gc.mem_free() 38 | end_time = time.ticks_diff(time.ticks_ms(), start_time) 39 | print("Database creation took", end_time, "ms to run") 40 | print("Database creation took up", before - after, "bytes") 41 | except Exception: 42 | return 'Error.' 43 | else: 44 | return 'Success.' 45 | 46 | 47 | def test_database_creation_exception(): 48 | try: 49 | mdb.Database.create("testdb") 50 | except Exception: 51 | return 'Success.' 52 | else: 53 | return 'Error.' 54 | 55 | 56 | def test_table_open_exception(): 57 | try: 58 | db_object = mdb.Database.open("testdb") 59 | db_object.open_table("testblahtable") 60 | except Exception: 61 | return 'Success.' 62 | else: 63 | return 'Error.' 64 | 65 | 66 | def test_table_creation(): 67 | try: 68 | db_object = mdb.Database.open("testdb") 69 | db_object.create_table("testtable", ["name", "password"]) 70 | except Exception: 71 | return 'Error.' 72 | else: 73 | return 'Success.' 74 | 75 | 76 | def test_table_open(): 77 | try: 78 | db_object = mdb.Database.open("testdb") 79 | db_object.open_table("testtable") 80 | except Exception: 81 | return 'Error.' 82 | else: 83 | return 'Success.' 84 | 85 | 86 | def test_insert_row(): 87 | i = 1 88 | try: 89 | all_memory = [] 90 | all_time = [] 91 | db_object = mdb.Database.open("testdb") 92 | db_table = db_object.open_table("testtable") 93 | for x in range(550): 94 | if uC: 95 | gc.collect() 96 | before = gc.mem_free() 97 | start_time = time.ticks_ms() 98 | db_table.insert({"name": "bob", "password": "coolpassword"}) 99 | if uC: 100 | gc.collect() 101 | after = gc.mem_free() 102 | end_time = time.ticks_diff(time.ticks_ms(), start_time) 103 | all_memory.append(before - after) 104 | all_time.append(end_time) 105 | i += 1 106 | except Exception: 107 | return 'Error.' 108 | else: 109 | if uC: 110 | print("Average memory used per insert was", 111 | sum(all_memory) / len(all_memory), "bytes.") 112 | print("Average time per insert was", 113 | sum(all_time) / len(all_time), "ms.") 114 | all_time.sort() 115 | all_memory.sort() 116 | print("Most memory used was", all_memory[-1], "bytes.") 117 | print("Longest insert time was", all_time[-1], "ms.") 118 | return 'Success.' 119 | 120 | 121 | def test_insert_multiple_rows(): 122 | try: 123 | if uC: 124 | gc.collect() 125 | before = gc.mem_free() 126 | start_time = time.ticks_ms() 127 | db_object = mdb.Database.open("testdb") 128 | if uC: 129 | gc.collect() 130 | after = gc.mem_free() 131 | end_time = time.ticks_diff(time.ticks_ms(), start_time) 132 | print("Database open took", end_time, "ms to run.") 133 | print("Database open took up", before - after, "bytes.") 134 | gc.collect() 135 | before = gc.mem_free() 136 | start_time = time.ticks_ms() 137 | db_table = db_object.open_table("testtable") 138 | if uC: 139 | gc.collect() 140 | after = gc.mem_free() 141 | end_time = time.ticks_diff(time.ticks_ms(), start_time) 142 | print("Opening table with 550 rows took", end_time, "ms.") 143 | print("Opening table with 550 rows took", before - after, "bytes.") 144 | gc.collect() 145 | before = gc.mem_free() 146 | start_time = time.ticks_ms() 147 | db_table.insert([{"name": "whothere", "password": "ohyeah"}, 148 | {"name": "whothere", "password": "ohyeah"}, 149 | {"name": "whothere", "password": "ohyeah"}, 150 | {"name": "whothere", "password": "ohyeah"}, 151 | {"name": "whothere", "password": "ohyeah"}, 152 | {"name": "whothere", "password": "ohyeah"}, 153 | {"name": "whothere", "password": "ohyeah"}, 154 | {"name": "whothere", "password": "ohyeah"}, 155 | {"name": "whothere", "password": "ohyeah"}, 156 | {"name": "whothere", "password": "ohyeah"}, 157 | {"name": "whothere", "password": "ohyeah"}, 158 | {"name": "whothere", "password": "ohyeah"}, 159 | {"name": "whothere", "password": "ohyeah"}, 160 | {"name": "whothere", "password": "ohyeah"}, 161 | {"name": "whothere", "password": "ohyeah"}, 162 | {"name": "whothere", "password": "ohyeah"}, 163 | {"name": "whothere", "password": "ohyeah"}, 164 | {"name": "whothere", "password": "ohyeah"}, 165 | {"name": "whothere", "password": "ohyeah"}, 166 | {"name": "whothere", "password": "ohyeah"}, 167 | {"name": "whothere", "password": "ohyeah"}, 168 | {"name": "whothere", "password": "ohyeah"}, 169 | {"name": "whothere", "password": "ohyeah"}, 170 | {"name": "whothere", "password": "ohyeah"}, 171 | {"name": "whothere", "password": "ohyeah"}, 172 | {"name": "whothere", "password": "ohyeah"}, 173 | {"name": "whothere", "password": "ohyeah"}, 174 | {"name": "whothere", "password": "ohyeah"}, 175 | {"name": "whothere", "password": "ohyeah"}]) 176 | if uC: 177 | gc.collect() 178 | after = gc.mem_free() 179 | end_time = time.ticks_diff(time.ticks_ms(), start_time) 180 | print("Multi-inserting 28 rows took", end_time, "ms to run.") 181 | print("Multi-inserting 28 rows took up", before - after, "bytes.") 182 | gc.collect() 183 | before = gc.mem_free() 184 | start_time = time.ticks_ms() 185 | db_table.insert([{"name": "whothere", "password": "ohyeah"}, 186 | {"name": "whothere", "password": "ohyeah"}, 187 | {"name": "whothere", "password": "ohyeah"}, 188 | {"name": "whothere", "password": "ohyeah"}, 189 | {"name": "whothere", "password": "ohyeah"}, 190 | {"name": "whothere", "password": "ohyeah"}, 191 | {"name": "whothere", "password": "ohyeah"}, 192 | {"name": "whothere", "password": "ohyeah"}, 193 | {"name": "whothere", "password": "ohyeah"}, 194 | {"name": "whothere", "password": "ohyeah"}, 195 | {"name": "whothere", "password": "ohyeah"}, 196 | {"name": "whothere", "password": "ohyeah"}, 197 | {"name": "whothere", "password": "ohyeah"}, 198 | {"name": "whothere", "password": "ohyeah"}, 199 | {"name": "whothere", "password": "ohyeah"}, 200 | {"name": "whothere", "password": "ohyeah"}, 201 | {"name": "whothere", "password": "ohyeah"}, 202 | {"name": "whothere", "password": "ohyeah"}, 203 | {"name": "whothere", "password": "ohyeah"}, 204 | {"name": "whothere", "password": "ohyeah"}, 205 | {"name": "whothere", "password": "ohyeah"}, 206 | {"name": "whothere", "password": "ohyeah"}, 207 | {"name": "whothere", "password": "ohyeah"}, 208 | {"name": "whothere", "password": "ohyeah"}, 209 | {"name": "whothere", "password": "ohyeah"}, 210 | {"name": "whothere", "password": "ohyeah"}, 211 | {"name": "whothere", "password": "ohyeah"}, 212 | {"name": "whothere", "password": "ohyeah"}, 213 | {"name": "whothere", "password": "ohyeah"}, 214 | {"name": "whothere", "password": "ohyeah"}, 215 | {"name": "whothere", "password": "ohyeah"}]) 216 | if uC: 217 | gc.collect() 218 | after = gc.mem_free() 219 | end_time = time.ticks_diff(time.ticks_ms(), start_time) 220 | print("Multi-inserting 30 rows took", end_time, "ms to run.") 221 | print("Multi-inserting 30 rows took up", before - after, "bytes.") 222 | except Exception: 223 | return 'Error.' 224 | else: 225 | return 'Success.' 226 | 227 | 228 | def test_update_row_exception_row(): 229 | try: 230 | db_object = mdb.Database.open("testdb") 231 | db_table = db_object.open_table("testtable") 232 | db_table.update_row(1000, {"name": "john"}) 233 | except Exception: 234 | return 'Success.' 235 | else: 236 | return 'Error.' 237 | 238 | 239 | def test_update_row_exception_column(): 240 | try: 241 | db_object = mdb.Database.open("testdb") 242 | db_table = db_object.open_table("testtable") 243 | db_table.update_row(300, {"what": "john"}) 244 | except Exception: 245 | return 'Success.' 246 | else: 247 | return 'Error.' 248 | 249 | 250 | def test_update_row(): 251 | try: 252 | db_object = mdb.Database.open("testdb") 253 | db_table = db_object.open_table("testtable") 254 | if uC: 255 | gc.collect() 256 | before = gc.mem_free() 257 | start_time = time.ticks_ms() 258 | db_table.update_row(300, {"name": "george", "password": "anotherone"}) 259 | if uC: 260 | gc.collect() 261 | after = gc.mem_free() 262 | end_time = time.ticks_diff(time.ticks_ms(), start_time) 263 | print("Updating by row id took", end_time, "ms to run.") 264 | print("Updating by row id took", before - after, "bytes.") 265 | except Exception: 266 | return 'Error.' 267 | else: 268 | return 'Success.' 269 | 270 | 271 | def test_update(): 272 | try: 273 | db_object = mdb.Database.open("testdb") 274 | db_table = db_object.open_table("testtable") 275 | if uC: 276 | gc.collect() 277 | before = gc.mem_free() 278 | start_time = time.ticks_ms() 279 | db_table.update({"name": "george", "password": "anotherone"}, 280 | {"name": "sally", "password": "whatisthis"}) 281 | if uC: 282 | gc.collect() 283 | after = gc.mem_free() 284 | end_time = time.ticks_diff(time.ticks_ms(), start_time) 285 | print("Updating by query (at row 300) took", 286 | end_time, "ms to run.") 287 | print("Updating by query (at row 300) took", 288 | before - after, "bytes.") 289 | except Exception: 290 | return 'Error.' 291 | else: 292 | return 'Success.' 293 | 294 | 295 | def test_update_exception(): 296 | try: 297 | db_object = mdb.Database.open("testdb") 298 | db_table = db_object.open_table("testtable") 299 | db_table.update({"name": "whowhowho", "password": "anotherone"}, 300 | {"name": "sally", "password": "whatisthis"}) 301 | except Exception: 302 | return 'Success.' 303 | else: 304 | return 'Error.' 305 | 306 | 307 | def test_delete_row(): 308 | try: 309 | db_object = mdb.Database.open("testdb") 310 | db_table = db_object.open_table("testtable") 311 | if uC: 312 | gc.collect() 313 | before = gc.mem_free() 314 | start_time = time.ticks_ms() 315 | db_table.delete_row(200) 316 | if uC: 317 | gc.collect() 318 | after = gc.mem_free() 319 | end_time = time.ticks_diff(time.ticks_ms(), start_time) 320 | print("Delete row by row id took", end_time, "ms to run.") 321 | print("Delete row by row id took", before - after, "bytes.") 322 | except Exception: 323 | return 'Error.' 324 | else: 325 | return 'Success.' 326 | 327 | 328 | def test_delete_row_exception(): 329 | try: 330 | db_object = mdb.Database.open("testdb") 331 | db_table = db_object.open_table("testtable") 332 | db_table.delete_row(3300) 333 | except Exception: 334 | return 'Success.' 335 | else: 336 | return 'Error.' 337 | 338 | 339 | def test_delete(): 340 | try: 341 | db_object = mdb.Database.open("testdb") 342 | db_table = db_object.open_table("testtable") 343 | if uC: 344 | gc.collect() 345 | before = gc.mem_free() 346 | start_time = time.ticks_ms() 347 | db_table.delete({"name": "sally", "password": "whatisthis"}) 348 | if uC: 349 | gc.collect() 350 | after = gc.mem_free() 351 | end_time = time.ticks_diff(time.ticks_ms(), start_time) 352 | print("Delete by query (at row 300) took", end_time, "ms to run.") 353 | print("Delete by query (at row 300) took", before - after, 354 | "bytes.") 355 | except Exception: 356 | return 'Error.' 357 | else: 358 | return 'Success.' 359 | 360 | 361 | def test_delete_exception(): 362 | try: 363 | db_object = mdb.Database.open("testdb") 364 | db_table = db_object.open_table("testtable") 365 | db_table.delete({"name": "sallywho", "password": "whatisthis"}) 366 | except Exception: 367 | return 'Success.' 368 | else: 369 | return 'Error.' 370 | 371 | 372 | def test_find_row(): 373 | try: 374 | db_object = mdb.Database.open("testdb") 375 | db_table = db_object.open_table("testtable") 376 | if uC: 377 | gc.collect() 378 | before = gc.mem_free() 379 | start_time = time.ticks_ms() 380 | db_table.find_row(150) 381 | if uC: 382 | gc.collect() 383 | after = gc.mem_free() 384 | end_time = time.ticks_diff(time.ticks_ms(), start_time) 385 | print("Find by row id (row 150) took", end_time, "ms to run.") 386 | print("Find by row id (row 150) took", before - after, "bytes.") 387 | except Exception: 388 | return 'Error.' 389 | else: 390 | return 'Success.' 391 | 392 | 393 | def test_find_row_exception(): 394 | try: 395 | db_object = mdb.Database.open("testdb") 396 | db_table = db_object.open_table("testtable") 397 | db_table.find_row(4500) 398 | except Exception: 399 | return 'Success.' 400 | else: 401 | return 'Error.' 402 | 403 | 404 | def test_query(): 405 | try: 406 | db_object = mdb.Database.open("testdb") 407 | db_table = db_object.open_table("testtable") 408 | db_table.update_row(250, {"name": "blah", "password": "something"}) 409 | db_table.update_row(101, {"name": "blah", "password": "something"}) 410 | if uC: 411 | gc.collect() 412 | before = gc.mem_free() 413 | start_time = time.ticks_ms() 414 | return_list = db_table.query({"name": "blah", "password": "something"}) 415 | if uC: 416 | gc.collect() 417 | after = gc.mem_free() 418 | end_time = time.ticks_diff(time.ticks_ms(), start_time) 419 | print("Query with two results (row 101 and 250) took", 420 | end_time, "ms to run.") 421 | print("Query with two results (row 101 and 250) took", 422 | before - after, "bytes.") 423 | except Exception: 424 | return 'Error.' 425 | if len(return_list) == 2: 426 | return 'Success.' 427 | else: 428 | return 'Error.' 429 | 430 | 431 | def test_find(): 432 | try: 433 | db_object = mdb.Database.open("testdb") 434 | db_table = db_object.open_table("testtable") 435 | if uC: 436 | gc.collect() 437 | before = gc.mem_free() 438 | start_time = time.ticks_ms() 439 | db_table.find({"name": "blah", "password": "something"}) 440 | if uC: 441 | gc.collect() 442 | after = gc.mem_free() 443 | end_time = time.ticks_diff(time.ticks_ms(), start_time) 444 | print("Find by query (row 101) took", end_time, "ms to run.") 445 | print("Find by query (row 101) took", before - after, "bytes.") 446 | except Exception: 447 | return 'Error.' 448 | else: 449 | return 'Success.' 450 | 451 | 452 | def test_scan_no_query(): 453 | try: 454 | db_object = mdb.Database.open("testdb") 455 | db_table = db_object.open_table("testtable") 456 | db_table.update_row(1, {"name": "tom", "password": "alright"}) 457 | if uC: 458 | gc.collect() 459 | before = gc.mem_free() 460 | start_time = time.ticks_ms() 461 | scan_return = db_table.scan() 462 | the_scan = scan_return.__next__() 463 | if uC: 464 | gc.collect() 465 | after = gc.mem_free() 466 | end_time = time.ticks_diff(time.ticks_ms(), start_time) 467 | print("Scan returning first row took", end_time, "ms to run.") 468 | print("Scan returning first row took", before - after, "bytes.") 469 | except Exception: 470 | return 'Error.' 471 | if the_scan['name'] == 'tom': 472 | return 'Success.' 473 | else: 474 | return 'Error.' 475 | 476 | 477 | def test_scan_with_query(): 478 | try: 479 | db_object = mdb.Database.open("testdb") 480 | db_table = db_object.open_table("testtable") 481 | if uC: 482 | gc.collect() 483 | before = gc.mem_free() 484 | start_time = time.ticks_ms() 485 | scan_return = db_table.scan({"name": "blah", "password": "something"}) 486 | the_scan = scan_return.__next__() 487 | if uC: 488 | gc.collect() 489 | after = gc.mem_free() 490 | end_time = time.ticks_diff(time.ticks_ms(), start_time) 491 | print("Scan with query returning first result (row 101) took", 492 | end_time, "ms to run.") 493 | print("Scan with query returning first result (row 101) took", 494 | before - after, "bytes.") 495 | except Exception: 496 | return 'Error.' 497 | if the_scan['name'] == 'blah': 498 | return 'Success.' 499 | else: 500 | return 'Error.' 501 | 502 | 503 | # A test to be sure data row files were created correctly. 504 | def check_data_file_name(): 505 | location = mdb.os.listdir('testdb/testtable') 506 | # Remove non-data files from our list of dirs. 507 | location = [element for element in location if 'data' in element] 508 | # Check we have the correct number of data page files 509 | if len(location) == 61: 510 | pass 511 | else: 512 | raise Exception('Error.') 513 | # Sort as integers so we get them in the right order. 514 | location = sorted(location, 515 | key=lambda x: int(x.split('.')[0].split('_')[1]), 516 | reverse=True) 517 | for f in location: 518 | if f in ['data1_10.dat', 'data11_20.dat', 519 | 'data21_30.dat', 'data31_40.dat', 'data41_50.dat', 520 | 'data51_60.dat', 'data61_70.dat', 'data71_80.dat', 521 | 'data81_90.dat', 'data91_100.dat', 'data101_110.dat', 522 | 'data111_120.dat', 'data121_130.dat', 'data131_140.dat', 523 | 'data141_150.dat', 'data151_160.dat', 'data161_170.dat', 524 | 'data171_180.dat', 'data181_190.dat', 'data191_200.dat', 525 | 'data201_210.dat', 'data211_220.dat', 'data221_230.dat', 526 | 'data231_240.dat', 'data241_250.dat', 'data251_260.dat', 527 | 'data261_270.dat', 'data271_280.dat', 'data281_290.dat', 528 | 'data291_300.dat', 'data301_310.dat', 'data311_320.dat', 529 | 'data321_330.dat', 'data331_340.dat', 'data341_350.dat', 530 | 'data351_360.dat', 'data361_370.dat', 'data371_380.dat', 531 | 'data381_390.dat', 'data391_400.dat', 'data401_410.dat', 532 | 'data411_420.dat', 'data421_430.dat', 'data431_440.dat', 533 | 'data441_450.dat', 'data451_460.dat', 'data461_470.dat', 534 | 'data471_480.dat', 'data481_490.dat', 'data491_500.dat', 535 | 'data501_510.dat', 'data511_520.dat', 'data521_530.dat', 536 | 'data531_540.dat', 'data541_550.dat', 'data551_560.dat', 537 | 'data561_570.dat', 'data571_580.dat', 'data581_590.dat', 538 | 'data591_600.dat', 'data601_610.dat']: 539 | continue 540 | else: 541 | raise Exception('Error.') 542 | return 'Success.' 543 | 544 | 545 | def check_current_row(): 546 | try: 547 | db_object = mdb.Database.open("testdb") 548 | db_table = db_object.open_table("testtable") 549 | current_row = db_table.__calculate_current_row() 550 | except Exception: 551 | return 'Error.' 552 | if int(current_row) == 610: 553 | return 'Success.' 554 | else: 555 | return 'Error.' 556 | 557 | 558 | # Make sure the data files have the correct number of rows in the files 559 | def test_data_files(): 560 | location = mdb.os.listdir('testdb/testtable') 561 | # Remove non-data files from our list of dirs. 562 | location = [element for element in location if 'data' in element] 563 | # Sort as integers so we get them in the right order. 564 | location = sorted(location, 565 | key=lambda x: int(x.split('.')[0].split('_')[1]), 566 | reverse=True) 567 | for f in location: 568 | if f != 'data601_610.dat': 569 | with open('testdb/testtable/' + f, 'r') as output_file: 570 | i = 0 571 | for line in output_file: 572 | i += 1 573 | if i == 10: 574 | pass 575 | else: 576 | raise Exception('Error.') 577 | else: 578 | with open('testdb/testtable/' + f, 'r') as output_file: 579 | i = 0 580 | for line in output_file: 581 | i += 1 582 | if i == 10: 583 | pass 584 | else: 585 | raise Exception('Error.') 586 | return 'Success.' 587 | 588 | 589 | def test_truncate(): 590 | try: 591 | db_object = mdb.Database.open("testdb") 592 | db_table = db_object.open_table("testtable") 593 | db_table.truncate() 594 | except Exception: 595 | return 'Error.' 596 | for file_name in mdb.os.listdir('testdb/testtable'): 597 | if file_name[0:4] == 'data': 598 | return 'Error.' 599 | else: 600 | return 'Success.' 601 | 602 | 603 | def test_vacuum(): 604 | """ 605 | add 12 non-identical records 606 | delete first 6 rows 607 | perform vaccum 608 | check if 2th row represent inserted 8th 609 | check that we have only one data file 610 | """ 611 | try: 612 | db_object = mdb.Database.open("testdb") 613 | db_table = db_object.open_table("testtable") 614 | for i in range(12): 615 | db_table.insert({ 616 | "name": "blah_{0}".format(i+1), 617 | "password": "something_{0}".format(i+1) 618 | }) 619 | for i in range(6): 620 | db_table.delete_row(i+1) 621 | db_table.vacuum() 622 | if db_table.current_row != 6: 623 | return "Error" 624 | if db_table.find_row(2)["d"]["password"] != "something_8": 625 | return "Error" 626 | except Exception: 627 | return "Error" 628 | else: 629 | return 'Success.' 630 | 631 | 632 | # Clean up all the test data 633 | def remove_test_database_files(): 634 | try: 635 | for file_name in mdb.os.listdir('testdb/testtable'): 636 | mdb.os.remove('testdb/testtable/' + file_name) 637 | mdb.os.rmdir('testdb/testtable') 638 | for file_name in mdb.os.listdir('testdb'): 639 | mdb.os.remove('testdb/' + file_name) 640 | mdb.os.rmdir('testdb') 641 | except Exception: 642 | return 'Failed to delete test data.' 643 | 644 | 645 | print("Testing started") 646 | print("Platform: {} so Micro Controller = {}".format(sys.platform, uC)) 647 | print("------") 648 | assert test_database_open_exception() == "Success.", \ 649 | "Error: Open database that doesn't exist" 650 | assert test_database_creation() == "Success.", "Error: Create database" 651 | assert test_database_creation_exception() == "Success.", \ 652 | "Error: Create database with same name" 653 | assert test_table_open_exception() == "Success.", \ 654 | "Error: Open table that doesn't exist" 655 | assert test_table_creation() == "Success.", "Error: Create table" 656 | assert test_table_open() == "Success.", "Error: Open table" 657 | assert test_insert_row() == "Success.", "Error: Insert row" 658 | assert test_insert_multiple_rows() == "Success.", "Error: Insert multiple rows" 659 | assert test_update_row_exception_row() == "Success.", \ 660 | "Error: Update row that doesn't exist" 661 | assert test_update_row_exception_column() == "Success.", \ 662 | "Error: Update row with column that doesn't exist" 663 | assert test_update_row() == "Success.", "Error: Update row" 664 | assert test_update() == "Success.", "Error: Update" 665 | assert test_update_exception() == "Success.", \ 666 | "Error: Update with query that doesn't match" 667 | assert test_delete_row() == "Success.", "Error: Delete row" 668 | assert test_delete_row_exception() == "Success.", "Error: Delete row exception" 669 | assert test_find_row() == "Success.", "Error: Find row" 670 | assert test_find_row_exception() == "Success.", "Error: Find row exception" 671 | assert test_query() == "Success.", "Error: Query exception" 672 | assert test_find() == "Success.", "Error: Find exception" 673 | assert test_scan_no_query() == "Success.", "Error: Scan without query" 674 | assert test_scan_with_query() == "Success.", "Error: Scan with query" 675 | assert check_data_file_name() == "Success.", "Error: Data row files" 676 | assert test_truncate() == "Success.", "Error: Truncate" 677 | assert test_vacuum() == "Success.", "Error: Vacuum" 678 | remove_test_database_files() 679 | print("------") 680 | print("All tests passed.") 681 | -------------------------------------------------------------------------------- /micropydatabase.py: -------------------------------------------------------------------------------- 1 | """Low-memory json-based databse for MicroPython. 2 | Data is stored in a folder structure in json for easy inspection. 3 | Indexing multiple columns is supported, and RAM usage is optimized 4 | for embedded systems. 5 | Database examples: 6 | db_object = Database.create("mydb") 7 | db_object = Database.open("mydb") 8 | Table examples: 9 | db_object.create_table("mytable", ["name", "password"]) 10 | this case all fields will be string type. If you want to define columns 11 | precisely- pass dict of fields with types defined. Supported: str, int, float, bool 12 | db_object.create_table("mytable", {"name":str, "age":int, "height":float, "isMember":bool}) 13 | db_table = db_object.open_table("mytable") 14 | db_table.truncate() 15 | Insert examples: 16 | db_table.insert({"name": "nate", "password": "coolpassword"}) 17 | or you can use dict for fields: 18 | db_table.insert(["John", 37, True]) 19 | Query data 20 | db_table.query({"name": "nate"}) 21 | In case you need to get row_id with your data, pass second optional boolean parameter 22 | db_table.insert({"name": "nate", "password": "coolpassword"}, True) 23 | You'll get additional column '_row' with your data. Works with scan() find() and query() 24 | Low-level operations using internal row_id: 25 | db_table.find_row(5) 26 | db_table.update_row(300, {'name': 'bob'}) 27 | db_table.delete_row(445) 28 | """ 29 | import json as json 30 | import os 31 | 32 | 33 | class OutOfMemoryError(Exception): 34 | opt = '' 35 | msg = '' 36 | 37 | def __init__(self, msg, opt=''): 38 | self.msg = msg 39 | self.opt = opt 40 | Exception.__init__(self, msg, opt) 41 | 42 | def __str__(self): 43 | return self.msg 44 | 45 | 46 | def file_exists(path): 47 | try: 48 | f = open(path, 'r') 49 | f.close() 50 | return True 51 | except OSError: 52 | return False 53 | 54 | 55 | def dir_exists(path): 56 | try: 57 | return os.stat(path)[0] & 0o170000 == 0o040000 58 | except OSError: 59 | return False 60 | 61 | 62 | class Database: 63 | def __init__(self, database: str, rows_per_page: int, max_rows: int, 64 | storage_format_version: int): 65 | self.path = database 66 | self.rows_per_page = rows_per_page 67 | self.max_rows = max_rows 68 | self.storage_format_version = storage_format_version 69 | 70 | if not dir_exists(self.path): 71 | raise Exception("Database not found at {}".format(self.path)) 72 | 73 | @staticmethod 74 | def create(database: str, rows_per_page: int = 10, max_rows: int = 10000): 75 | """ 76 | store the current version of the MicroPyDB to prevent future upgrade 77 | errors. 78 | """ 79 | version = 1 80 | # Check if database already exists. 81 | if not dir_exists(database): 82 | os.mkdir(database) 83 | # If database doesn't exist, create the directory structure and the 84 | # Schema json file with default values. 85 | with open("{}/schema.json".format(database), 'w') as f: 86 | data = { 87 | 'max_rows': max_rows, 88 | 'storage_format_version': version, 89 | 'rows_per_page': rows_per_page 90 | } 91 | f.write(json.dumps(data)) 92 | return Database(database, rows_per_page, max_rows, version) 93 | else: 94 | raise Exception("Database {} is already in use".format(database)) 95 | 96 | @staticmethod 97 | def open(database: str): 98 | # Check if database exists. 99 | schema_path = '{}/schema.json'.format(database) 100 | if file_exists(schema_path): 101 | with open(schema_path) as json_file: 102 | data = json.load(json_file) 103 | rows_per_page = data['rows_per_page'] 104 | max_rows = data['max_rows'] 105 | storage_format_version = data['storage_format_version'] 106 | return Database(database, rows_per_page, max_rows, 107 | storage_format_version) 108 | else: 109 | raise Exception("Database {} does not exist".format(database)) 110 | 111 | def create_table(self, table: str, columns: any, 112 | rows_per_page: int = None, 113 | max_rows: int = None): 114 | # Convert all column names to lowercase 115 | # columns = [element.lower() for element in columns] # logic moved to Table.create_table() 116 | if rows_per_page is None: 117 | rows_per_page = self.rows_per_page 118 | max_rows = max_rows if max_rows is not None else self.max_rows 119 | Table.create_table(self, table.lower(), columns, rows_per_page, 120 | max_rows=self.max_rows) 121 | 122 | def open_table(self, table_name: str): 123 | return Table.open_table(self, table_name) 124 | 125 | def list_tables(self) -> list: 126 | """ 127 | Get the list of available tables 128 | """ 129 | tables_list = [] 130 | for item in os.listdir(self.path): 131 | if dir_exists("{}/{}".format(self.path, item)): 132 | tables_list.append(item) 133 | return tables_list 134 | 135 | @staticmethod 136 | def exist(database: str) -> bool: 137 | return True if dir_exists(database) else False 138 | 139 | 140 | class Table: 141 | def __init__(self, database: str, table: str, 142 | columns: list, rows_per_page: int, 143 | max_rows: int): 144 | self.database = database 145 | self.name = table.lower() 146 | self.columns = columns 147 | self.rows_per_page = rows_per_page 148 | self.max_rows = max_rows 149 | self.path = '{}/{}'.format(database.path, table) 150 | self.current_row = self.__calculate_current_row() 151 | 152 | # TODO: validate and self-heal to recover from data corruption 153 | 154 | @staticmethod 155 | def create_table(database, table: str, columns: any, 156 | rows_per_page: int = None, max_rows: int = None): 157 | """ 158 | Create a table in a database that already exists. 159 | Takes string input for table name and a comma seperated list 160 | for column names. 161 | """ 162 | # Inherit rows_per_page and max_rows from database metadata 163 | if rows_per_page is None: 164 | rows_per_page = int(database.rows_per_page) 165 | max_rows = int(database.max_rows) if max_rows is None else max_rows 166 | 167 | table_folder = "{}/{}".format(database.path, table) 168 | # Check if table already exists, if it doesn't, then proceed, 169 | # ortherwise throw an error. 170 | if not dir_exists(table_folder): 171 | # Add our hard-coded meta-ida to the beginning of the 172 | # column name variable. 173 | # columns.insert(0, "meta_id") 174 | # create the table json file and populate it. 175 | data = { 176 | 'settings': { 177 | 'rows_per_page': rows_per_page, 178 | 'max_rows': max_rows, 179 | }, 180 | 'columns': {} 181 | } 182 | 183 | # dictionary style columns declaration, all types default to str 184 | if(isinstance(columns, list)): 185 | for col in columns: 186 | data['columns'][col.lower()] = {'data_type': 'str', 'max_length': 10000} 187 | 188 | # dictionary style columns declaration, together with types 189 | elif(isinstance(columns, dict)): 190 | for col in columns: 191 | if(columns[col].__name__ == 'str'): 192 | data['columns'][col.lower()] = {'data_type': 'str', 'max_length': 10000} 193 | elif(columns[col].__name__ in ["int", "float", "bool"]): 194 | data['columns'][col.lower()] = {'data_type': columns[col].__name__} 195 | else: 196 | raise Exception("Data type '{}' for column '{}' is not suported".format( 197 | columns[col].__name__, col)) 198 | else: 199 | raise Exception("Columns definition is incorrect") 200 | os.mkdir(table_folder) 201 | with open('{}/definition.json'.format(table_folder), 'w') as f: 202 | f.write(json.dumps(data)) 203 | return Table(database, table, columns, rows_per_page, max_rows) 204 | else: 205 | raise Exception("Table {} already exists".format(table)) 206 | 207 | @staticmethod 208 | def open_table(database, table: str): 209 | path = '{}/{}'.format(database.path, table) 210 | if dir_exists(path): 211 | with open('{}/definition.json'.format(path)) as json_file: 212 | definition = json.load(json_file) 213 | # Check to make sure there are not any temporary files left over 214 | # from previous session. 215 | for file_name in os.listdir(path): 216 | if file_name[-4:] in ['temp', 'vacu']: 217 | raise Exception("Some temporary data page files are still" 218 | " in your table. Delete temp and vacu files manually") 219 | return Table(database, table, definition['columns'], 220 | definition['settings']['rows_per_page'], 221 | definition['settings']['max_rows']) 222 | else: 223 | raise Exception("Table {} does not exist in {}".format( 224 | table, database.path)) 225 | 226 | def stats(self) -> dict: 227 | with open("{}/definition.json".format(self.path)) as json_file: 228 | definition = json.load(json_file) 229 | table_size = 0 230 | for entry in os.ilistdir(self.path): 231 | if not entry[0] == 'definition.json': 232 | table_size +=entry[3] 233 | return { 234 | 'Settings': definition['settings'], 235 | 'Columns': definition['columns'], 236 | 'Pages_Count': len(os.listdir(self.path)) - 1, 237 | 'Current_row': self.__calculate_current_row(), 238 | 'Data_Size' : table_size 239 | } 240 | 241 | def insert(self, data: any) -> bool: 242 | """ 243 | Inserts new data in a table 244 | """ 245 | # Check for multiple row insert and prepare for each 246 | if isinstance(data, list) and isinstance(data[0], dict): 247 | total = len(data) - 1 248 | new_data = data 249 | 250 | if isinstance(data[0], dict): 251 | for x in range(len(new_data)): 252 | if self.__scrub_data(new_data[x]): 253 | pass 254 | else: 255 | raise Exception("Data element {} is not formatted correctly".format(x)) 256 | 257 | # while we still have data to insert 258 | while total > 0: 259 | current_line = self.__row_id_in_file(self.current_row)+1 260 | # calculate how many lines we will insert on the first loop 261 | insert_number = int(self.rows_per_page) - int(current_line) 262 | if insert_number == 0: 263 | insert_number = int(self.rows_per_page) 264 | # Check that we aren't at max rows: 265 | if self.current_row + insert_number > self.max_rows: 266 | raise Exception("Table {} can not fit all those" 267 | " rows".format(self.name)) 268 | # populate first_data based on how many total rows are being 269 | # inserted and how much room is on data page 270 | if insert_number < total: 271 | # grab how many we need to fill the current data page 272 | first_data = data[:insert_number] 273 | del data[:insert_number] 274 | else: 275 | first_data = data 276 | # record how many rows we are inserting this time: 277 | number_rows_to_insert = len(first_data) 278 | # prepare data 279 | first_data_string = '' 280 | first_path = self.__data_file_for_row_id( 281 | int(self.current_row) + 1) 282 | for x in range(len(first_data)): 283 | self.current_row += 1 284 | first_data_string = "{0}{{\"r\": {1}, \"d\": {2}}}\n" \ 285 | .format(first_data_string,str(self.current_row), 286 | json.dumps(first_data[x])) 287 | if not self.__multi_append_row(first_data_string, first_path): 288 | raise Exception("There was a problem inserting " 289 | "multiple rows") 290 | if not self.__is_multi_insert_success(first_data_string, 291 | first_path, 292 | number_rows_to_insert, 293 | current_line): 294 | raise Exception("There was a problem validating the " 295 | "write during multiple row insert") 296 | total -= insert_number 297 | return True 298 | # If not multi-insert 299 | else: 300 | data = self.__scrub_data(data) 301 | if data: 302 | self.current_row += 1 303 | row_id = self.current_row 304 | path = self.__data_file_for_row_id(row_id) 305 | # Check that we aren't at max rows: 306 | if self.current_row < self.max_rows: 307 | if self.__insert_modify_data_file(path, data): 308 | return True 309 | else: 310 | raise Exception("There was a problem inserting " 311 | "row at {}".format(row_id)) 312 | else: 313 | raise Exception("Table {} is full".format(self.name)) 314 | else: 315 | raise Exception("Data you tried to insert is invalid") 316 | 317 | def update(self, query_conditions: dict, data: dict): 318 | """ 319 | Update data based on matched query 320 | """ 321 | matched_queries = self.__return_query('update', query_conditions) 322 | if matched_queries is None: 323 | raise Exception("Query did not match any data") 324 | else: 325 | # Loop through and update each row where the query returned true 326 | for row_id in matched_queries: 327 | # Check to make sure all the column names given by user match 328 | # the column names in the table. 329 | self.update_row(row_id, data) 330 | 331 | def update_row(self, row_id: int, update_data: any): 332 | """ 333 | Update data based on row_id. 334 | """ 335 | # get data from database and update with user provided 336 | combined = self.find_row(row_id)["d"] 337 | combined.update(update_data) 338 | 339 | # Check to make sure all the column names given by user 340 | # match the column names in the table. 341 | data = self.__scrub_data(combined, False) 342 | path = self.__data_file_for_row_id(row_id) 343 | if data: 344 | # Create a temp data file with the updated row data. 345 | if self.__modify_data_file(path, {row_id: data}, 'update'): 346 | pass 347 | else: 348 | raise Exception("There was a problem updating " 349 | "row at {}".format(row_id)) 350 | else: 351 | raise Exception("Data you tried to insert is invalid") 352 | 353 | def delete(self, query_conditions: dict) -> bool: 354 | """ 355 | Delete row based on query search. 356 | """ 357 | matched_queries = self.__return_query('delete', query_conditions) 358 | if matched_queries is None: 359 | raise Exception("Query did not match any data") 360 | else: 361 | # Loop through and update each row where the query returned true 362 | for found_row in matched_queries: 363 | row_id = found_row['r'] 364 | self.delete_row(row_id) 365 | return True 366 | 367 | def delete_row(self, row_id: int): 368 | """ 369 | Delete data based on row_id. 370 | """ 371 | if self.__modify_data_file(self.__data_file_for_row_id(row_id), 372 | {row_id: None}, 'delete'): 373 | pass 374 | else: 375 | raise Exception("There was a problem deleting " 376 | "row at {}".format(row_id)) 377 | 378 | def truncate(self): 379 | """ 380 | Nuke all data in the table. 381 | """ 382 | for file_name in os.listdir(self.path): 383 | if file_name[0:4] == 'data': 384 | os.remove('{}/{}'.format(self.path, file_name)) 385 | self.current_row = 0 386 | 387 | def find_row(self, row_id: int): 388 | """ 389 | Find data based on row_id. 390 | """ 391 | # Calculate what line in the file the row_id will be found at 392 | looking_for_line = self.__row_id_in_file(row_id) 393 | # Prevous method of counting lines is not reliable 394 | if looking_for_line is not None: 395 | with open(self.__data_file_for_row_id(row_id), 'r') as f: 396 | for current_line, line in enumerate(f): 397 | if current_line == looking_for_line: 398 | return json.loads(line) 399 | else: 400 | raise Exception("Could not find row_id {}".format(row_id)) 401 | 402 | def query(self, queries: dict, show_row: bool = False): 403 | """ 404 | Search through the whole table and return all rows where column 405 | data matches searched for value. 406 | """ 407 | final_result = [] 408 | results = self.__return_query('query', queries, show_row) 409 | if results is None: 410 | return [] 411 | else: 412 | if len(results) > 1: 413 | for result in results: 414 | final_result.append(result) 415 | else: 416 | final_result = results 417 | return final_result 418 | 419 | def find(self, queries: dict, show_row: bool = False): 420 | """ 421 | Search through the whole table and return first row where column 422 | data matches searched for value. 423 | """ 424 | return self.__return_query('find', queries, show_row) 425 | 426 | def scan(self, queries: any = None, show_row: bool = False): 427 | """ 428 | Iterate through the whole table and return data by line 429 | """ 430 | if queries: 431 | queries = self.__scrub_data(queries, False) 432 | location = os.listdir(self.path) 433 | # Remove non-data files from our list of dirs. 434 | location = [element for element in location if 'data' in element] 435 | # Sort as integers so we get them in the right order 436 | # (not reversed because we want smallest first). 437 | location = sorted(location, key=lambda x: 438 | int(x.split('.')[0].split('_')[1])) 439 | for f in location: 440 | with open("{}/{}".format(self.path, f), 'r') as data: 441 | for line in data: 442 | if line != "\n": # empty lines fails to json.loads() 443 | current_data = json.loads(line) 444 | # If we are not searching for anything 445 | if not queries: 446 | if show_row: 447 | current_data['d']['_row'] = current_data['r'] 448 | yield current_data['d'] 449 | else: 450 | for query in queries: 451 | if current_data['d'][query] == queries[query]: 452 | if show_row: 453 | current_data['d']['_row'] = current_data['r'] 454 | yield current_data['d'] 455 | else: 456 | break 457 | 458 | def vacuum(self) -> bool: 459 | """ 460 | This will reorganize your data files- remove spaces after records has 461 | been deleted 462 | NOTE: this also change row ID of your data 463 | """ 464 | location = os.listdir(self.path) 465 | # Remove non-data files from our list of dirs. 466 | location = [element for element in location if 'data' in element] 467 | # Sort as integers so we get them in the right order. 468 | location = sorted(location, 469 | key=lambda x: int(x.split('.')[0].split('_')[1]), 470 | reverse=False) 471 | for f in location: 472 | os.rename('{}/{}'.format(self.path, f), 473 | '{}/{}.vacu'.format(self.path, f)) 474 | # Reset row id counter 475 | self.current_row = 0 476 | for f in location: 477 | with open("{}/{}.vacu".format(self.path, f), 'r') as data: 478 | for line in data: 479 | if line != '\n': 480 | current_data = json.loads(line) 481 | self.insert(current_data['d']) 482 | # delete temporary files 483 | os.remove('{}/{}.vacu'.format(self.path, f)) 484 | return True 485 | 486 | def drop(self): 487 | for filename in os.ilistdir(self.path): 488 | location = "{}/{}".format(self.path, filename[0]) 489 | os.remove(location) 490 | os.remove(self.path) 491 | 492 | def __return_query(self, search_type: str, queries: any = None, show_row: bool = False) -> list: 493 | """ 494 | Helper function to process a query and return the result. 495 | """ 496 | if queries: 497 | queries = self.__scrub_data(queries, False) 498 | 499 | for query in queries: 500 | if type(queries[query]) is not list: 501 | queries[query] = [queries[query]] 502 | 503 | # different handling logic must be performed if there are mutiple keys 504 | multiple_keys = True if len(list(queries.keys())) > 1 else False 505 | result = [] 506 | location = os.listdir(self.path) 507 | # Remove non-data files from our list of dirs. 508 | location = [element for element in location if 'data' in element] 509 | # Sort as integers so we get them in the right order. 510 | location = sorted(location, 511 | key=lambda x: int(x.split('.')[0].split('_')[1]), 512 | reverse=True) 513 | found = False 514 | for f in location: 515 | with open("{}/{}".format(self.path, f), 'r') as data: 516 | for line in data: 517 | # Make sure the line isn't blank (ex. if it was deleted). 518 | if line != '\n': 519 | found = False 520 | cur_data = json.loads(line) 521 | for query in queries: 522 | if query in cur_data['d'].keys() and \ 523 | cur_data['d'][query] in queries[query]: 524 | found = True 525 | if not multiple_keys: 526 | break 527 | else: 528 | found = False 529 | if multiple_keys: 530 | break 531 | if found: 532 | if show_row: 533 | cur_data['d']['_row'] = cur_data['r'] 534 | if search_type == 'find': 535 | return cur_data['d'] 536 | elif search_type == 'query': 537 | result.append(cur_data['d']) 538 | # for delete and update commands- only row id is needed 539 | elif search_type in ['update', 'delete']: 540 | result.append(cur_data['r']) 541 | if result: 542 | return result 543 | else: 544 | return None 545 | 546 | 547 | 548 | # def __check_write_success(self, data, page: str, method: str) -> bool: 549 | # """ 550 | # Checks to make sure the previous update or delete was successful. 551 | # """ 552 | # # Calculate what line will have the row we are looking for. 553 | # looking_for_line = self.__row_id_in_file(list(data)[0]) 554 | # row_id = list(data)[0] 555 | # # open file at path 556 | # with open(page, 'r') as f: 557 | # for current_line, line in enumerate(f): 558 | # if current_line == looking_for_line: 559 | # if method == "update": 560 | # json_line = json.loads(line) 561 | # if json_line['r'] == row_id and \ 562 | # json_line['d'] == data[row_id]: 563 | # return True 564 | # elif method == "delete": 565 | # if line == "\n": 566 | # return True 567 | # # There was a problem writing, so return false 568 | # return False 569 | 570 | # def __check_write_success_insert(self, data: dict, page: str) -> bool: 571 | # """ 572 | # Checks to make sure the previous insert was successful. 573 | # """ 574 | # last_line = None 575 | # with open(page, 'r') as f: 576 | # for line in f: 577 | # if len(line) > 1: 578 | # last_line = line 579 | # if last_line: 580 | # json_line = json.loads(last_line) 581 | # if data['r'] == json_line['r'] and data['d'] == json_line['d']: 582 | # return True 583 | # return False 584 | 585 | def __is_multi_insert_success(self, data: dict, page: str, 586 | rows_added: int, start_line: int) -> bool: 587 | """ 588 | Checks to make sure the previous insert was successful. 589 | """ 590 | line_counter = 1 591 | # if we were at the end of a data page, then start at the first 592 | # line of the file 593 | if int(start_line) == int(self.rows_per_page): 594 | start_line = 0 595 | temp_data = '' 596 | with open(page, 'r') as f: 597 | for line in f: 598 | # check if we are past the line we started the insert at in 599 | # the data page 600 | if line_counter > start_line: 601 | # If we are, then add the line to our temp_data so we 602 | # can compare it with the data insert 603 | temp_data = '{}{}'.format(temp_data, line) 604 | line_counter += 1 605 | if temp_data == data: 606 | return True 607 | return False 608 | 609 | def __multi_append_row(self, data: dict, page: str) -> bool: 610 | """ 611 | This function assumes the data has already been scrubbed! 612 | """ 613 | # Write the row to the data page file ('a' positions the stream at the 614 | # end of the file). 615 | # temp_current_row = self.current_row 616 | with open(page, 'a+') as f: 617 | f.write(data) 618 | return True 619 | 620 | def __append_row(self, data: dict, page: str) -> bool: 621 | """ 622 | This function assumes the data has already been scrubbed! 623 | """ 624 | # Write the row to the data page file ('a' positions the stream at the 625 | # end of the file). 626 | with open(page, 'a+') as f: 627 | f.write(json.dumps({'r': self.current_row, 'd': data})) 628 | f.write('\n') 629 | return True 630 | # if self.__check_write_success_insert(new_data, page): 631 | # return True 632 | # else: 633 | # raise Exception("Data was corrupted at row: {}".format( 634 | # self.current_row)) 635 | # return False 636 | 637 | def __calculate_current_row(self) -> int: 638 | """ 639 | We don't want to write table metadata to disk every insert, 640 | so just find it when we open the table and keep it in memory. 641 | """ 642 | # last_data_file = None 643 | location = os.listdir(self.path) 644 | # Remove non-data files from our list of dirs. 645 | location = [element for element in location if 'data' in element] 646 | # Sort as integers so we get them in the right order. 647 | location = sorted(location, 648 | key=lambda x: int(x.split('.')[0].split('_')[1]), 649 | reverse=True) 650 | for f in location: 651 | if f[0:4] == 'data': 652 | last_line = None 653 | with open("{}/{}".format(self.path, f), 'r') as f: 654 | for line in f: 655 | if len(line) > 1: 656 | last_line = line 657 | if last_line: 658 | return json.loads(last_line)['r'] 659 | 660 | return 0 661 | 662 | def __insert_modify_data_file(self, page: str, data: dict = None, 663 | fast: bool = True) -> bool: 664 | """ 665 | Insert data page write helper 666 | """ 667 | if fast: 668 | if self.__append_row(data, page): 669 | return True 670 | else: 671 | # not recomended, becouse its heavy on flash storage! 672 | # First we copy the data page file to a temp file if 673 | # the data page file exists 674 | temp_path = "{}.temp".format(page) 675 | piece_size = 512 # 512 bytes at a time 676 | if file_exists(page): 677 | with open(page, 'rb') as in_file,\ 678 | open(temp_path, 'wb') as out_file: 679 | while True: 680 | piece = in_file.read(piece_size) 681 | if piece == b'': 682 | break # end of file 683 | out_file.write(piece) 684 | 685 | elif not file_exists(page): 686 | open(temp_path, 'w').close 687 | 688 | if self.__append_row(data, temp_path): 689 | # make it compatible with PC and 690 | try: 691 | if os.path.exists(page): 692 | os.remove(page) 693 | except AttributeError: 694 | pass 695 | os.rename(temp_path, page) 696 | return True 697 | 698 | def __modify_data_file(self, path: str, update_data, action: str) -> bool: 699 | """ 700 | Creates a duplicate of a data file in the same path and appends 701 | ".temp". You also can specify whether or not to replace certain rows 702 | with new data. update_data = {334: {name: John}} will update the 703 | "name" field of row 334 update_data = {334: None} will delete row 334 704 | """ 705 | # Check that data page file exists 706 | if not file_exists(path): 707 | raise Exception("Cannot {} a row that " 708 | "does not exist".format(action)) 709 | 710 | temp_path = "{}.temp".format(path) 711 | current_data = '' 712 | row_id = next(iter(update_data)) 713 | # Open the master data page file 714 | with open(path, 'r') as input_file: 715 | # Create a temporary data page file 716 | with open(temp_path, 'w') as output: 717 | for line_num, line in enumerate(input_file): 718 | if line != "\n": 719 | current_data = json.loads(line) 720 | # If this is our line 721 | if current_data['r'] == row_id: 722 | # Write the modified line to the file 723 | if action == 'delete': 724 | # output.write('\n') 725 | pass 726 | # If we are updating a row: 727 | else: 728 | if current_data['r'] == row_id: 729 | current_data['d'].update(update_data[row_id]) 730 | output.write(json.dumps(current_data)) 731 | output.write('\n') 732 | else: 733 | raise Exception("Woah we thought {} was row_id" 734 | " {} and almost stomped the " 735 | "wrong row's data!".format( 736 | line, line_num)) 737 | # Otherwise, write the line to the file as-is, skipping empty lines 738 | else: 739 | output.write(line) 740 | # imposible to check as we have ommited empty lines 741 | os.remove(path) 742 | os.rename(temp_path, path) 743 | return True 744 | 745 | 746 | def __data_file_for_row_id(self, row_id: int) -> str: 747 | """ 748 | Calculate what data file we are currently in and return the path 749 | to that file. 750 | """ 751 | file_row_id = int(row_id) % int(self.rows_per_page) 752 | if file_row_id == 0: 753 | second_number = (int(row_id) // int(self.rows_per_page)) *\ 754 | int(self.rows_per_page) 755 | first_number = second_number - int(self.rows_per_page) + 1 756 | else: 757 | first_number = (int(row_id) // int(self.rows_per_page)) *\ 758 | int(self.rows_per_page) + 1 759 | second_number = first_number + int(self.rows_per_page) - 1 760 | return '{}/data{}_{}.dat'.format(self.path, first_number, 761 | second_number) 762 | 763 | def __row_id_in_file(self, row_id: int) -> int: 764 | """ 765 | Calculates the line in a data page file that row will be found at. 766 | """ 767 | # if our table doesn't have any rows yet 768 | if row_id == 0: 769 | return 0 770 | else: 771 | with open(self.__data_file_for_row_id(row_id), 'r') as f: 772 | for line_num, line in enumerate(f): 773 | if line != "\n": 774 | if int(json.loads(line)['r']) == int(row_id): 775 | return line_num 776 | return None 777 | 778 | def __scrub_data(self, data: any, fill_missing: bool = True): 779 | """ 780 | Check to see if user data input contains valid column data for 781 | current table. 782 | a) Validates column names actually exist in table 783 | b) Fills in null values for missing columns. 784 | c) Downcases column names. 785 | """ 786 | all_columns = list(self.columns.keys()) 787 | result = {} 788 | column = None 789 | 790 | # if received list of values - make a dict with fields 791 | if(isinstance(data, list)): 792 | ndata = {} 793 | for k, v in zip(self.columns, data): 794 | ndata[k]=v 795 | data = ndata 796 | del(ndata) 797 | try: 798 | for column, value in data.items(): 799 | column = column.lower() 800 | # column_definition = self.columns[column] 801 | # validate type/length 802 | if(self.columns[column]['data_type'] == "str" and isinstance(value, str)): 803 | if(len(value)>self.columns[column]["max_length"]): 804 | raise Exception("Max_length of {} exceeded for {} ".format( 805 | self.columns[column]['max_length'], column)) 806 | elif(self.columns[column]['data_type'] == "int" and isinstance(value, int)): 807 | pass 808 | elif(self.columns[column]['data_type'] == "float" and isinstance(value, float)): 809 | pass 810 | elif(self.columns[column]['data_type'] == "bool" and isinstance(value, bool)): 811 | pass 812 | else: 813 | raise Exception("Data types mismath: Expected but received {} for column '{}'".format( 814 | self.columns[column]['data_type'], type(value), column)) 815 | 816 | all_columns.remove(column) 817 | result[column] = value 818 | except KeyError: 819 | raise Exception("Column {} does not exist in {}".format( 820 | column, self.name)) 821 | 822 | # now add default values 823 | if fill_missing: 824 | for lefover_column in all_columns: 825 | result[lefover_column] = None 826 | 827 | return result 828 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | import sys 2 | # Remove current dir from sys.path, otherwise setuptools will peek up our 3 | # module instead of system's. 4 | sys.path.pop(0) 5 | from setuptools import setup 6 | sys.path.append("..") 7 | import sdist_upip 8 | 9 | setup(name='micropydatabase', 10 | version='0.1', 11 | description='Low-memory json-based databse for MicroPython.', 12 | long_description='Data is stored in a folder structure in json for easy inspection. RAM usage is optimized for embedded systems.', 13 | url='https://github.com/sungkhum/micropydatabase', 14 | author='Nathan Wells and James Hill', 15 | license='MIT', 16 | cmdclass={'sdist': sdist_upip.sdist}, 17 | py_modules=['micropython'], 18 | install_requires=['micropython-os', 'micropython-os.path']) 19 | --------------------------------------------------------------------------------