├── .gitattributes ├── .gitmodules ├── README.md ├── micropython.mk ├── modufastlz.c └── tests └── fastlz_json.py /.gitattributes: -------------------------------------------------------------------------------- 1 | # Per default everything gets normalized and gets LF line endings on checkout. 2 | * text eol=lf 3 | 4 | # These will always have CRLF line endings on checkout. 5 | *.vcxproj text eol=crlf 6 | *.props text eol=crlf 7 | *.bat text eol=crlf 8 | 9 | # These are binary so should never be modified by git. 10 | *.a binary 11 | *.png binary 12 | *.jpg binary 13 | *.dxf binary 14 | *.mpy binary 15 | -------------------------------------------------------------------------------- /.gitmodules: -------------------------------------------------------------------------------- 1 | [submodule "FastLZ"] 2 | path = FastLZ 3 | url = https://github.com/ariya/FastLZ.git 4 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ufastlz 2 | ===== 3 | 4 | Micropython wrapper for [FastLZ](https://github.com/ariya/FastLZ), a lightning-fast lossless compression library. 5 | 6 | Compiling the cmodule into MicroPython 7 | ===== 8 | 9 | To build such a module, compile MicroPython with an extra make flag named ```USER_C_MODULES``` set to the directory containing all modules you want included (not to the module itself). 10 | 11 | 12 | #### unix port 13 | 14 | ```bash 15 | ➜ ~ git clone https://github.com/micropython/micropython.git 16 | ➜ ~ cd micropython 17 | ➜ micropython (master) ✗ git clone https://github.com/dmazzella/ufastlz.git ports/unix/cmodules/ufastlz 18 | ➜ micropython (ufastlz) ✗ cd ports/unix/cmodules/ufastlz 19 | ➜ micropython (ufastlz) ✗ git submodule update --init 20 | ➜ micropython (ufastlz) ✗ cd ../../../../ 21 | ➜ micropython (master) ✗ make -j8 -C mpy-cross && make -j2 -C ports/unix/ VARIANT="dev" USER_C_MODULES="$(pwd)/ports/unix/cmodules" 22 | ``` 23 | 24 | 25 | #### stm32 port 26 | 27 | ```bash 28 | ~ git clone https://github.com/micropython/micropython.git micropython 29 | ➜ ~ cd micropython 30 | ➜ micropython (master) ✗ git submodule update --init 31 | ➜ micropython (master) ✗ git clone https://github.com/dmazzella/ufastlz.git ports/stm32/boards/PYBD_SF6/cmodules/ufastlz 32 | ➜ micropython (ufastlz) ✗ cd ports/stm32/boards/PYBD_SF6/cmodules/ufastlz 33 | ➜ micropython (ufastlz) ✗ git submodule update --init 34 | ➜ micropython (ufastlz) ✗ cd ../../../../../../ 35 | ➜ micropython (master) ✗ make -j2 -C mpy-cross && make -j2 -C ports/stm32/ BOARD="PYBD_SF6" USER_C_MODULES="$(pwd)/ports/stm32/boards/PYBD_SF6/cmodules" 36 | ``` 37 | 38 | Usage 39 | ===== 40 | 41 | ```python 42 | MicroPython v1.17-74-gd42cba0d2-dirty on 2021-10-06; PYBD-SF6W with STM32F767IIK 43 | Type "help()" for more information. 44 | >>> import _fastlz 45 | >>> d = "test " * 100 46 | >>> d 47 | 'test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test ' 48 | >>> c = _fastlz.compress(d) 49 | >>> c 50 | b'\x04test \xe0\xfd\x04\xe0\xdb\x04\x04test ' 51 | >>> _fastlz.decompress(c) 52 | b'test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test ' 53 | >>> 54 | ``` 55 | 56 | API 57 | ===== 58 | - ```_fastlz.compress(data, level=2)``` 59 | - ```_fastlz.decompress(data, maxout=2048)``` -------------------------------------------------------------------------------- /micropython.mk: -------------------------------------------------------------------------------- 1 | MOD_UFASTLZ_DIR := $(USERMOD_DIR) 2 | 3 | CFLAGS_USERMOD += -DMICROPY_PY_UFASTLZ 4 | CFLAGS_USERMOD += -I$(MOD_UFASTLZ_DIR) 5 | CFLAGS_USERMOD += -I$(MOD_UFASTLZ_DIR)/FastLZ/ 6 | 7 | SRC_USERMOD += $(MOD_UFASTLZ_DIR)/FastLZ/fastlz.c 8 | SRC_USERMOD += $(MOD_UFASTLZ_DIR)/modufastlz.c 9 | -------------------------------------------------------------------------------- /modufastlz.c: -------------------------------------------------------------------------------- 1 | /* 2 | * This file is part of the Micro Python project, http://micropython.org/ 3 | * 4 | * The MIT License (MIT) 5 | * 6 | * Copyright (c) 2021 Damiano Mazzella 7 | * 8 | * Permission is hereby granted, free of charge, to any person obtaining a copy 9 | * of this software and associated documentation files (the "Software"), to deal 10 | * in the Software without restriction, including without limitation the rights 11 | * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 12 | * copies of the Software, and to permit persons to whom the Software is 13 | * furnished to do so, subject to the following conditions: 14 | * 15 | * The above copyright notice and this permission notice shall be included in 16 | * all copies or substantial portions of the Software. 17 | * 18 | * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 19 | * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 20 | * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 21 | * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 22 | * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 23 | * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 24 | * THE SOFTWARE. 25 | */ 26 | 27 | #if defined(MICROPY_PY_UFASTLZ) 28 | 29 | #include 30 | #include 31 | #include 32 | #include 33 | #include 34 | #include 35 | 36 | #include "py/objstr.h" 37 | #include "py/objint.h" 38 | #include "py/runtime.h" 39 | 40 | #include "fastlz.h" 41 | 42 | STATIC mp_obj_t ufastlz_compress(size_t n_args, const mp_obj_t *args, mp_map_t *kw_args) 43 | { 44 | enum 45 | { 46 | ARG_level, 47 | }; 48 | static const mp_arg_t allowed_args[] = { 49 | {MP_QSTR_level, MP_ARG_KW_ONLY | MP_ARG_INT, {.u_int = 1}}, 50 | }; 51 | 52 | mp_arg_val_t vals[MP_ARRAY_SIZE(allowed_args)]; 53 | mp_arg_parse_all(n_args - 1, args + 1, kw_args, MP_ARRAY_SIZE(allowed_args), allowed_args, vals); 54 | 55 | mp_buffer_info_t bufinfo_data; 56 | mp_get_buffer_raise(args[0], &bufinfo_data, MP_BUFFER_READ); 57 | mp_int_t level = vals[ARG_level].u_int; 58 | 59 | if (level != 1 && level != 2) 60 | { 61 | mp_raise_ValueError(MP_ERROR_TEXT("level must be either 1 or 2")); 62 | } 63 | 64 | if (bufinfo_data.len < 16) 65 | { 66 | mp_raise_ValueError(MP_ERROR_TEXT("input must be >= 16 bytes len")); 67 | } 68 | 69 | vstr_t vstr_out; 70 | vstr_init(&vstr_out, MAX((mp_int_t)(1.05 * bufinfo_data.len), 66)); 71 | 72 | if (!(vstr_out.len = fastlz_compress_level(level, bufinfo_data.buf, bufinfo_data.len, vstr_out.buf))) 73 | { 74 | mp_raise_msg_varg(&mp_type_ValueError, MP_ERROR_TEXT("compress: %d"), vstr_out.len); 75 | } 76 | 77 | mp_obj_t out = mp_obj_new_bytes((const byte *)vstr_out.buf, vstr_out.len); 78 | vstr_clear(&vstr_out); 79 | return out; 80 | } 81 | 82 | STATIC MP_DEFINE_CONST_FUN_OBJ_KW(fastlz_compress_obj, 1, ufastlz_compress); 83 | 84 | STATIC mp_obj_t ufastlz_decompress(size_t n_args, const mp_obj_t *args, mp_map_t *kw_args) 85 | { 86 | enum 87 | { 88 | ARG_maxout 89 | }; 90 | static const mp_arg_t allowed_args[] = { 91 | {MP_QSTR_maxout, MP_ARG_KW_ONLY | MP_ARG_INT, {.u_int = 1024}}, 92 | }; 93 | 94 | mp_arg_val_t vals[MP_ARRAY_SIZE(allowed_args)]; 95 | mp_arg_parse_all(n_args - 1, args + 1, kw_args, MP_ARRAY_SIZE(allowed_args), allowed_args, vals); 96 | 97 | mp_buffer_info_t bufinfo_data; 98 | mp_get_buffer_raise(args[0], &bufinfo_data, MP_BUFFER_READ); 99 | 100 | mp_int_t maxout = vals[ARG_maxout].u_int; 101 | vstr_t vstr_out; 102 | vstr_init(&vstr_out, maxout); 103 | 104 | if (!(vstr_out.len = fastlz_decompress(bufinfo_data.buf, bufinfo_data.len, vstr_out.buf, maxout))) 105 | { 106 | mp_raise_msg_varg(&mp_type_ValueError, MP_ERROR_TEXT("decompress: %d"), vstr_out.len); 107 | } 108 | 109 | mp_obj_t out = mp_obj_new_bytes((const byte *)vstr_out.buf, vstr_out.len); 110 | vstr_clear(&vstr_out); 111 | return out; 112 | } 113 | 114 | STATIC MP_DEFINE_CONST_FUN_OBJ_KW(fastlz_decompress_obj, 1, ufastlz_decompress); 115 | 116 | STATIC const mp_rom_map_elem_t mp_module_ufastlz_globals_table[] = { 117 | {MP_ROM_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR__fastlz)}, 118 | {MP_ROM_QSTR(MP_QSTR_compress), MP_ROM_PTR(&fastlz_compress_obj)}, 119 | {MP_ROM_QSTR(MP_QSTR_decompress), MP_ROM_PTR(&fastlz_decompress_obj)}, 120 | }; 121 | 122 | STATIC MP_DEFINE_CONST_DICT(mp_module_ufastlz_globals, mp_module_ufastlz_globals_table); 123 | 124 | const mp_obj_module_t mp_module_ufastlz = { 125 | .base = {&mp_type_module}, 126 | .globals = (mp_obj_dict_t *)&mp_module_ufastlz_globals, 127 | }; 128 | 129 | // Register the module to make it available in Python 130 | MP_REGISTER_MODULE(MP_QSTR__fastlz, mp_module_ufastlz); 131 | 132 | #endif // MICROPY_PY_UFASTLZ 133 | -------------------------------------------------------------------------------- /tests/fastlz_json.py: -------------------------------------------------------------------------------- 1 | try: 2 | import _fastlz as fastlz 3 | except ImportError: 4 | import fastlz 5 | 6 | data = b'[{"_id":"615dc2818a8646b6895b3227","index":0,"guid":"922ac005-f725-43f5-ba00-4d0bff9e6913","isActive":false,"balance":"$2,672.04","picture":"http://placehold.it/32x32","age":39,"eyeColor":"green","name":"Luz Avery","gender":"female","company":"ENAUT","email":"luzavery@enaut.com","phone":"+1 (975) 518-3645","address":"622 Richardson Street, Terlingua, Illinois, 413","about":"Qui esse adipisicing veniam labore elit ex reprehenderit exercitation culpa magna officia veniam. Quis fugiat irure sint enim nulla est. Eu Lorem ad est ut et voluptate dolor velit nisi magna aute deserunt. Sit sit laborum ad et cillum culpa non anim esse incididunt reprehenderit ut est.","registered":"2021-01-25T12:11:55 -01:00","latitude":-35.705031,"longitude":113.270485,"tags":["in","laboris","ipsum","fugiat","incididunt","eu","exercitation"],"friends":[{"id":0,"name":"Annie Hodge"},{"id":1,"name":"Franco Peck"},{"id":2,"name":"Justice Huber"}],"greeting":"Hello, Luz Avery! You have 1 unread messages.","favoriteFruit":"apple"},{"_id":"615dc281e7027d327a147e9e","index":1,"guid":"0daca687-e2cc-4bef-98a5-337341a52cc0","isActive":true,"balance":"$3,055.46","picture":"http://placehold.it/32x32","age":31,"eyeColor":"brown","name":"Aguilar Sloan","gender":"male","company":"MAZUDA","email":"aguilarsloan@mazuda.com","phone":"+1 (943) 485-2821","address":"775 Woods Place, Ferney, Pennsylvania, 6153","about":"Sit commodo cupidatat ut culpa. Exercitation duis sunt cillum sint exercitation do nisi amet commodo est Lorem et non. Laboris laborum est ipsum cupidatat.","registered":"2017-12-01T06:04:39 -01:00","latitude":-0.738155,"longitude":-58.749487,"tags":["esse","eu","esse","ad","ipsum","laboris","excepteur"],"friends":[{"id":0,"name":"Hinton Ware"},{"id":1,"name":"Angel Cantrell"},{"id":2,"name":"Gay Merrill"}],"greeting":"Hello, Aguilar Sloan! You have 5 unread messages.","favoriteFruit":"strawberry"},{"_id":"615dc281d172f24a7d905fb1","index":2,"guid":"c642689f-61cf-479a-b43e-8eeae5b4e627","isActive":true,"balance":"$1,044.71","picture":"http://placehold.it/32x32","age":20,"eyeColor":"blue","name":"Ernestine Knox","gender":"female","company":"EMOLTRA","email":"ernestineknox@emoltra.com","phone":"+1 (889) 444-2104","address":"550 Seton Place, Hayden, Idaho, 4996","about":"Nisi laboris id in anim. Reprehenderit laborum aliqua occaecat incididunt ipsum exercitation est eiusmod adipisicing incididunt ex. Labore ex excepteur magna id nisi ut fugiat cupidatat elit. Dolor sint cupidatat amet non sint sint consectetur labore qui ipsum. Nulla voluptate proident consequat excepteur amet exercitation nisi laboris officia commodo enim cupidatat minim. Sit incididunt sit elit commodo nostrud ad qui enim culpa. Tempor proident sunt aliquip nisi nostrud incididunt est ipsum non incididunt laborum.","registered":"2014-09-04T04:20:37 -02:00","latitude":87.45108,"longitude":-171.056391,"tags":["consequat","ipsum","tempor","veniam","quis","do","qui"],"friends":[{"id":0,"name":"Juliette Tanner"},{"id":1,"name":"Herrera Levy"},{"id":2,"name":"Annette Pearson"}],"greeting":"Hello, Ernestine Knox! You have 3 unread messages.","favoriteFruit":"banana"},{"_id":"615dc281e41ebe23f6ce0830","index":3,"guid":"9c356278-beb3-4ebf-83bf-f359982f8258","isActive":true,"balance":"$1,992.22","picture":"http://placehold.it/32x32","age":29,"eyeColor":"blue","name":"Ivy Peterson","gender":"female","company":"ENORMO","email":"ivypeterson@enormo.com","phone":"+1 (983) 560-2735","address":"669 National Drive, Libertytown, American Samoa, 1019","about":"Ex cupidatat aliquip sit magna incididunt labore excepteur. In eiusmod nulla cillum sit est irure aliqua. Cillum et est in est. Dolor occaecat id commodo irure minim minim sit.","registered":"2016-08-14T04:59:39 -02:00","latitude":84.945576,"longitude":90.608004,"tags":["nostrud","nisi","minim","tempor","nulla","consequat","sunt"],"friends":[{"id":0,"name":"Florence Mcintyre"},{"id":1,"name":"Helen Walker"},{"id":2,"name":"Brittney Beasley"}],"greeting":"Hello, Ivy Peterson! You have 3 unread messages.","favoriteFruit":"banana"},{"_id":"615dc2815f0f17ea80644ea4","index":4,"guid":"e925674d-08eb-4608-bf5b-54ea9dc90e97","isActive":false,"balance":"$1,896.09","picture":"http://placehold.it/32x32","age":31,"eyeColor":"green","name":"Wade Murray","gender":"male","company":"AQUASSEUR","email":"wademurray@aquasseur.com","phone":"+1 (846) 504-2684","address":"474 Gem Street, Caroline, District Of Columbia, 6016","about":"Cupidatat sunt exercitation nisi amet sint consectetur cillum velit veniam culpa ipsum est. Culpa exercitation excepteur esse amet tempor. Lorem amet nostrud ut sunt. Quis veniam adipisicing cillum nostrud consectetur adipisicing aliquip ea nisi ea amet consectetur reprehenderit amet. Nisi ad ut et sunt sint voluptate nisi aliquip sint laboris ullamco elit reprehenderit.","registered":"2017-02-02T10:43:07 -01:00","latitude":52.133924,"longitude":147.390515,"tags":["anim","dolore","duis","eu","officia","nulla","culpa"],"friends":[{"id":0,"name":"Shaw Keller"},{"id":1,"name":"Tia Fry"},{"id":2,"name":"Flossie Wagner"}],"greeting":"Hello, Wade Murray! You have 7 unread messages.","favoriteFruit":"banana"},{"_id":"615dc28159f43f988c64940b","index":5,"guid":"5382db59-f001-4f88-8d9d-33d23fb9c693","isActive":false,"balance":"$2,210.98","picture":"http://placehold.it/32x32","age":20,"eyeColor":"green","name":"Hilary Livingston","gender":"female","company":"ACCRUEX","email":"hilarylivingston@accruex.com","phone":"+1 (985) 549-2793","address":"436 Quay Street, Nile, Virgin Islands, 3656","about":"Ut ea laborum id fugiat irure aliquip enim non. Commodo consectetur labore amet exercitation nostrud anim minim Lorem deserunt anim incididunt esse minim. Irure deserunt non reprehenderit mollit elit nostrud dolore aliquip. Fugiat aute ad irure labore laboris eu. Magna nulla minim amet laboris proident sunt proident tempor non veniam minim et amet consectetur. Minim velit in consequat velit esse.","registered":"2021-08-22T06:09:51 -02:00","latitude":-89.264829,"longitude":90.134794,"tags":["deserunt","anim","velit","qui","pariatur","elit","labore"],"friends":[{"id":0,"name":"Estelle Patel"},{"id":1,"name":"Hawkins Dorsey"},{"id":2,"name":"Della Everett"}],"greeting":"Hello, Hilary Livingston! You have 3 unread messages.","favoriteFruit":"strawberry"}]' 7 | 8 | def main(): 9 | c_json_data = fastlz.compress(data, level=1) 10 | try: 11 | d_json_data = fastlz.decompress(c_json_data, maxout=len(data)) 12 | except TypeError: 13 | d_json_data = fastlz.decompress(c_json_data) 14 | assert len(d_json_data) == len(data), (len(data), len(c_json_data)) 15 | 16 | print(f"Compressed from {len(data)} to {len(c_json_data)} bytes") 17 | print(f"Decompressed from {len(c_json_data)} to {len(data)} bytes") 18 | 19 | 20 | if __name__ == "__main__": 21 | main() 22 | --------------------------------------------------------------------------------