├── .gitignore ├── LICENSE ├── README.md ├── freezefs ├── __main__.py ├── archive.py ├── ffsextract.py └── ffsmount.py ├── pyproject.toml └── tests ├── cleantest.bat ├── runtest.bat └── test.py /.gitignore: -------------------------------------------------------------------------------- 1 | dist 2 | __pycache__ 3 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2022 bixb922 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # freezefs: Create self-extracting compressed or self-mounting archives for MicroPython 2 | 3 | ## Purpose 4 | 5 | freezefs saves a file structure with subfolders and builds an self-extractable or self-mountable archive in a .py file, optionally with compression, to be frozen as bytecode or extracted. 6 | 7 | There are several ways to use freezefs: 8 | * Freeze the archive as frozen bytecode in a MicroPython image. Import the archive and it gets mounted as a read-only file system. The files continue to reside in the frozen image. 9 | * Freeze the compressed archive as frozen bytecode in a MicroPython image. Import the archive once to extract the file structure to the flash file system. The purpose is to aid initial deployment of read/write files. 10 | 11 | These two options require RAM to import the archive: 12 | * Run a compressed .py archive with ```mpremote run```. The files get extracted to the microprocessor. This is a easy way to install many files at once, and it is quite fast. 13 | * Install a compressed .py archive with ```mpremote mip install``` and then import the file to extract all files. This aids in installing complete systems over-the-air (OTA). The file should be compiled with mpy-cross, to get all the gain from compression. 14 | 15 | Overall, it simplifies deploying text and binary files, such as MicroPython code, html pages, json data files, etc. 16 | 17 | 18 | ## Description 19 | freezefs is a utility program that runs on a PC and converts an arbitrary folder, subfolder and file structure into a Python source file. The files can be compressed. The generated Python file can then be frozen as bytecode into a MicroPython image, installed with mip on a microcontroller. 20 | 21 | The archive file can be either mounted as a Virtual File System or extracted. 22 | 23 | The files can be compressed or be left uncompressed. 24 | 25 | The drivers for mounting or extracting are included in the same generated Python file, making the output file a self-mounting or self-extracting archive file. 26 | 27 | ## Feedback 28 | Please report any problem or ask for support in the issues section. If it works for you, please star the repository. 29 | 30 | ## Installation 31 | 32 | Install the software with ```pip install freezefs``` 33 | 34 | This installs the freezefs utility to be run on PC or MAC. The necessary MicroPython code is also installed on the PC or MAC and then automatically included in the output file. 35 | 36 | ## An example: freeze a files to a MicroPython image and mount file system 37 | Suppose you have the following folders, files and subfolders on your PC and want to freeze that together with your MicroPython programs in a MicroPython image: 38 | 39 | ``` 40 | myfolder 41 | | 42 | +---index.html 43 | +---tunes.html 44 | +---favicon.ico 45 | +---css 46 | | | 47 | | +---mystyles.css 48 | | +---normalize.css 49 | | 50 | +---images 51 | | 52 | +---myimage.jpg 53 | ``` 54 | The following command will archive the complete structure to the myfolder.py file: 55 | ``` 56 | python -m freezefs myfolder frozen_myfolder.py 57 | ``` 58 | The frozen_myfolder.py will now contain all the files and folders, together with the code to mount this as a read only file system. To mount on the microcontroller, add this line to boot.py or main.py: 59 | ``` 60 | import frozen_myfolder 61 | ``` 62 | 63 | When booting up the microcontroller, and once ```import frozen_myfolder``` has been executed, the above file structure is mounted (using ```os.mount()``` internally) at /myfolder, and the files and folders will appear under ```/myfolder``` on the microcontroller as read only files. The files are not copied to ```/myfolder```, but remain in the MicroPython image on flash. They now can be accessed with MicroPython statements such as ```open( "/myfolder/index.html", "r"), read(), readline(), open in "rb" or "r" mode, os.listdir("/myfolder")``` etc. lsIf the import is in ```boot.py```, the files are also visible with ```mpremote ls```. The RAM overhead is low, and access speed is similar to regular flash files. 64 | 65 | ## Another example: create a self-extractable file archive 66 | Use: 67 | ``` 68 | python -m freezefs myfolder frozen_myfolder.py --on-import=extract --compress 69 | ``` 70 | 71 | The frozen_myfolder.py will now contain all the files and folders compressed with zlib, together with the code to extract the files to the flash file system at ```/```. Optionally compile with ```mpy-cross frozen_myfolder.py``` to reduce file size. Have your code ```import frozen_myfolder```. This will decompress and extract (copy) the complete folder and subfolders to flash memory. On the next import, the files won't be overwritten (see ```--overwrite``` option). 72 | 73 | Importing or running the file, for example ```mpremote run frozen_myfolder``` will also extract all files. Since this is quite fast, this is aids deploying software. 74 | 75 | 76 | ## freezefs utility 77 | 78 | You run this program on your PC to freeze a folder and its content (files and subfolders) to a .py file. 79 | 80 | 81 | ```python -m freezefs --help``` will show the command format and options. 82 | 83 | ``` 84 | usage: python -m freezefs [-h] [--on-import {mount,extract}] [--target TARGET] [--overwrite {never,always}] 85 | [--compress | --no-compress | -c] [--wbits WBITS] [--level LEVEL] [--silent] 86 | infolder outfile 87 | 88 | freezefs.py 89 | freezefs saves a file structure with subfolders and builds an self-extractable or self-mountable archive in a .py file, optionally with compression, to be frozen as bytecode or extracted. 90 | 91 | Examples: 92 | freezefs.py input_folder frozenfiles.py --target=/myfiles --on_import mount 93 | freezefs.py input_folder frozenfiles.py --target=/myfiles --on_import=extract --compress 94 | 95 | positional arguments: 96 | infolder Input folder path 97 | outfile Path and name of output module. Must have .py extension. 98 | 99 | options: 100 | -h, --help show this help message and exit 101 | --on-import {mount,extract}, -oi {mount,extract} 102 | Action when importing output module. Default is mount. 103 | --target TARGET, -t TARGET 104 | For --on-import=mount: mount point. For --on-import=extract: destination folder. 105 | Default: the infolder. 106 | Example: --target /myfiles. Must start with / 107 | --overwrite {never,always}, -ov {never,always} 108 | always: on extract, all files are overwritten. never: on extract, no file is 109 | overwritten, only new files are extracted. Default: never. 110 | --compress, --no-compress, -c 111 | Compress files before writing to output .py. See python zlib compression. (default: 112 | False) 113 | --wbits WBITS, -w WBITS 114 | Compression window of 2**WBITS bytes. Between 9 and 14. Default is 10 (1024 bytes) 115 | --level LEVEL, -l LEVEL 116 | Compression level. Between 0 (no compression) and 9 (best compression). Default is 9 117 | --silent, -s Supress messages printed when mounting/copying files and while running this program. 118 | ``` 119 | ### The infolder 120 | The input folder and subfolders contain the files to be archived in the output .py file. 121 | 122 | 123 | ### The output .py file 124 | The outfile is overwritten with the MicroPython code with file contents (possibly compressed), file and folder names and the code to mount the archive as file system or extract the files. 125 | 126 | 127 | ### freezefs with --on-import mount 128 | 129 | With this option, the output .py module mounts its file system on import at the mount point (virtual folder) specified by --target as read-only files. 130 | 131 | The purpose --on-import=mount option to enable mounting a file system frozen in bytecode in a MicroPython image. So the best use for this option is to put the .py output file or files into a manifest.py, generate the MicroPython image and load the image to a microcontroller. Add a import of the output .py file in the main.py or boot.py and the files get visible read only at the specified target. 132 | 133 | See section on RAM usage below for --on-import with --compress. 134 | 135 | 136 | ### freezefs with --on-import=extract 137 | This option is intended for use with --compress to makme a self extractable .py file. 138 | 139 | To obtain maximum gain from compression, compile the .py with mpy-cross to a .mpy file. 140 | 141 | When importing or running this .py file on a MicroPython system, the file system gets decompressed and extracted. 142 | 143 | Also see --overwrite option. 144 | 145 | ### --on-import=extract with --overwrite=never 146 | When extracting, each file that exists will be skipped. Only non-existing files will be extracted. Existing files will be never overwritten. 147 | 148 | 149 | 150 | ### --on-import=extract with --overwrite=always 151 | When extracting, existing files will be always overwritten. 152 | 153 | 154 | ### freezefs --target 155 | For ```--on-import=mount``` this is the mount point on the file system of the microcontroller. 156 | 157 | For ```--on-import=export```, this is the destination folder on the file system of the microcontroller. 158 | 159 | Must start with / 160 | 161 | For ```--on-import=extract```, ```--target=/``` is used to deploy files to the root folder, such as main.py. 162 | 163 | If omitted, the last subfolder of the infolder is set as target, for example if the infolder is ```/myfolder/subfolder```, target will be set to ```/subfolder```. 164 | 165 | ### freezefs --compress 166 | This option compresses the files when packing them into the output .py files and decompresses them using MicroPytnon ```deflate``` on the microcontroller. 167 | 168 | This option is best for use with --on-import=extract. It works with ```--on-import=mount```, but the RAM usage is high when opening text files with "r" mode. See section on RAM usage below. 169 | 170 | ### freezefs --compress, --wbits and --level options 171 | 172 | --wbits indicates the number of bytes used at any time for compressing (called the window size). The size of the window is 2\*\*WBITS, so --wbits=9 means windows size of 2\*\*9=512 bytes and --wbits=14 means 2\*\*14=16384 bytes. The higher the value, the better the compression. However, to decompress, up to 2\*\*WBITS bytes may needed on the microcontroller. 173 | 174 | --level goes from 0 (no compression) to 9 (highest compression). Level 9 is a bit slower to decompress. 175 | 176 | See Python docs for zlib and MicroPython docs for deflate for more details. 177 | 178 | 179 | ### freezefs --silent 180 | By default, mount and extract print the progress. If you want to suppress those messages, freeze the files with --silent. 181 | 182 | If an exception occurs during mount or extract, the exception will be raised independently of the --silent option. 183 | 184 | ## The frozen .py output file 185 | 186 | The output file of the freezefs utility is a module with the frozen file system. This generated module contains consts with all the file data. MicroPython will access the file data directly in flash, if the .py file is frozen as bytecode in a MicroPython image. 187 | 188 | The code for extract or mount is included in the file. When compiled to .mpy files, the additional code amounts to about 1800 bytes for mount and 1300 bytes for extract. 189 | 190 | 191 | ### The mounted virtual file system 192 | 193 | freezefs implements a Virtual File System (VFS), with the driver included in the output file when using --on-import=mount 194 | 195 | The VFS implements ```os.mount```, ```os.umount```, ```os.chdir```, ```os.getcwd```, ```os.ilistdir```, ```os.listdir```, ```os.stat```, ```open```, ```close```, ```read```, ```readinto```, ```readline```, the iterator for reading lines and the decoding of UTF-8 format files to MicroPython strings. 196 | 197 | ```statvfs``` returns block size of 1. The file system size is the sum of all file sizes, without overhead. Mode flag is 1 (read only). The maximum file length is set to 255. 198 | 199 | ```open``` will only accept modes "r", "rt" and "rb". As usual, "r" will decode UTF-8 to strings, and "rb" will return bytes. 200 | 201 | ```open``` with modes "w", "wb", "a", etc. raises an OSError, since the file system frozen into the MicroPython image is read only. 202 | 203 | ```remove```, ```mkdir``` and ```rename``` will raise an OSError( errno.EPERM ). 204 | 205 | ```ilistdir``` will show file type (0x4000 for folders, 0x8000 for files as usual) and file size. Unused fields returned by ilistdir are set to zero. 206 | 207 | If ```--compress``` was used, the files are decompressed on open while reading the stream. ```read()```, ```readinto()```, ```readline()```, ```readlines()``` are available. However, ```seek()``` and ```tell()``` are not available for compress. See also RAM usage below. 208 | 209 | ## RAM usage for --on-import=mount 210 | 211 | When frozen as bytecode in a MicroPython image, the RAM usage is low, about 1 kbyte. 212 | 213 | ```---on-import=mount``` with ```--compress``` is RAM intensive on the microcontroller. Files opened with "r" (text mode) have to be decompressed in RAM, and the complete file gets loaded to RAM. This does not affect files opened with "rb" mode (binary mode), RAM usage is similar to opening a regular file system file. 214 | 215 | When the .py file resides in the flash file system (as opposed to being frozen in the MicroPython image) the complete file is read to RAM, and it's now essentially a read only RAM disk. 216 | 217 | ## RAM usage for --on-import=extract 218 | 219 | When frozen as bytecode in a MicroPython image, the RAM usage is very low while extracting. The buffer size to read/write is set at 256 bytes. 220 | 221 | Compressed .py files will use up to 2\*\*WBITS bytes of RAM while decompressing. The --wbits option can be used to set this value if RAM is low. The higher the WBITS value, the better the compression. 222 | 223 | When extracting a .py archive residing in the flash file system (or on SD card), the .py file is best compiled with mpy-cross to a .mpy to have the best gain in size. The complete .py (or .mpy) file will be loaded to RAM. To get that memory back once the extract is done, use ```__import__("module-name")```, without assigning the result of ```__import__``` to a variable. The extract driver will delete the itself from the ```sys.modules[]``` list, so the next garbage collection will free the memory. 224 | 225 | ## Unit tests 226 | 227 | The /tests folder on github has unit tests. 228 | 229 | 230 | ## Dependencies 231 | 232 | These standard MicroPython libraries are needed: sys, os, io.BytesIO, io.StringIO, collections.OrderedDict and errno. If --compress is used, deflate is needed. Deflate is present in MicroPython 1.20 or later. 233 | 234 | Python 3.10 or later must be installed on the PC. Probably earlier versions of Python will work too. pahtlib is used to be platform independent. 235 | 236 | The code is MicroPython/Python only, no C/C++ code. There are no processor or board specific dependencies. 237 | 238 | # Restrictions 239 | 240 | While a freezefs filesystem is mounted, ```os.sync()``` may crash the microcontroller, raise a TypeError, or it may even appear to work. ```os.sync()``` cannot be used with freezefs. 241 | 242 | If you are using ```os.sync()``` and want to use freezefs, you may consider dropping ```os.sync()``` altogether. See MicroPython issue #11449 (https://github.com/micropython/micropython/issues/11449). On most architectures and filesystems (including ESP32, RP2040 and LittleFS2), ```os.sync()``` is a no-op (at lest for MicroPython up to version 1.23). Verify that for you case ```os.sync()``` really does something. 243 | 244 | # Changes since version 1 245 | Version number 2. If you are using version 1, please regenerate the output .py files with the new version of freezefs as they are incompatible. 246 | 247 | Added --compress and --overwrite switches. Drivers for extracting and mounting are now included in the compressed file, no need to install drivers. freezefs is now pip installable. 248 | 249 | ## Compatibility with MicroPython/Python versions 250 | Tested with MicroPython 1.20 to 1.22 and Python 3.10.7 and 3.11.4. 251 | 252 | ## Copyright and license 253 | Source code and documentation Copyright (c) 2023 Hermann Paul von Borries. 254 | 255 | This software and documentation is licensed according to the MIT license: 256 | 257 | Permission is hereby granted, free of charge, to any person obtaining a copy 258 | of this software and associated documentation files (the "Software"), to deal 259 | in the Software without restriction, including without limitation the rights 260 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 261 | copies of the Software, and to permit persons to whom the Software is 262 | furnished to do so, subject to the following conditions: 263 | 264 | The above copyright notice and this permission notice shall be included in all 265 | copies or substantial portions of the Software. 266 | 267 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 268 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 269 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 270 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 271 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 272 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 273 | SOFTWARE. 274 | 275 | 276 | 277 | -------------------------------------------------------------------------------- /freezefs/__main__.py: -------------------------------------------------------------------------------- 1 | import sys 2 | from freezefs.archive import main 3 | sys.exit( main() ) 4 | 5 | -------------------------------------------------------------------------------- /freezefs/archive.py: -------------------------------------------------------------------------------- 1 | # (c) 2023 Hermann Paul von Borries 2 | # MIT License 3 | # freeze 4 | import argparse 5 | from pathlib import Path, PurePosixPath 6 | from glob import glob 7 | import os 8 | import sys 9 | import time 10 | import zlib 11 | 12 | MAX_FILENAME_LEN = 255 13 | VERSION = 2 # Directory entries changed 14 | silent = False 15 | def _verbose_print( *args ): 16 | if silent: 17 | return 18 | print( *args ) 19 | 20 | varcounter = 0 21 | class FileObject: 22 | def __init__( self, pc_infolder, path, request_compression, level, wbits ): 23 | global varcounter 24 | self.path = path 25 | self.request_compression = request_compression 26 | self.level = level 27 | self.wbits = wbits 28 | 29 | self.mp_path = "/" + self.path.as_posix() 30 | if len( self.mp_path ) >= MAX_FILENAME_LEN: 31 | raise ValueError(f"File name too long {self.mp_path}") 32 | 33 | self.pc_path = pc_infolder / path 34 | 35 | # Assign a variable name and get the file data (raw or compressed) 36 | self.is_file = self.pc_path.is_file() 37 | if self.is_file: 38 | self.varname = f"_f{varcounter}" 39 | varcounter += 1 40 | self._get_data() 41 | 42 | def _get_data( self ): 43 | # get data, compress (if indicated) 44 | # assign self.size, self.compressed_size, self.compressed 45 | self.data = self._file_read( ) 46 | self.size = len( self.data ) 47 | self.compressed = False 48 | self.compressed_size = self.size 49 | if self.request_compression: 50 | compressed_data = self._zlib_compress( self.data ) 51 | if len( compressed_data ) < self.size: 52 | # Use compression if compression gives some gain 53 | self.data = compressed_data 54 | self.compressed = True 55 | self.compressed_size = len( self.data ) 56 | 57 | def _zlib_compress( self, original_data ): 58 | # This will create the zlib header (8 bytes) that contains level and wbits for deflate 59 | zco = zlib.compressobj( level=self.level, wbits=self.wbits ) 60 | compressed_data = zco.compress( self.data ) 61 | compressed_data += zco.flush() 62 | return compressed_data 63 | 64 | 65 | def get_pythonized( self ): 66 | pythonized = "" 67 | p = 0 68 | while True: 69 | chunk = self.data[p:p+16] 70 | pythonized += f" {str(chunk)}\\\n" 71 | if len(chunk) == 0: 72 | break 73 | p += 16 74 | 75 | # Last item without \ nor \n 76 | return pythonized[0:-2] 77 | 78 | def _file_read( self ): 79 | with open( self.pc_path, "rb" ) as file: 80 | return file.read() 81 | 82 | 83 | def to_python( pc_infolder, pc_outfile, 84 | mc_target, on_import, overwrite, 85 | silent, 86 | request_compression, wbits, level ): 87 | 88 | # Get files 89 | files = [] 90 | for path in glob( "**", root_dir=pc_infolder, recursive=True ): 91 | fo = FileObject( pc_infolder, Path( path ), request_compression, level, wbits ) 92 | files.append( fo ) 93 | 94 | # Generate output 95 | files.sort( key=lambda fo: fo.mp_path ) 96 | with open( pc_outfile, "w", encoding="utf-8") as outfile: 97 | _files_to_python( outfile, files, silent ) 98 | _generate_appended_code( outfile, 99 | mc_target, on_import, overwrite, 100 | silent ) 101 | 102 | # Print some statistics 103 | sum_size = sum( fo.size for fo in files if fo.is_file ) 104 | number_of_files = sum( 1 for fo in files if fo.is_file ) 105 | number_of_folders = len( files ) - number_of_files 106 | _verbose_print(f"Sum of file sizes {sum_size} bytes, {number_of_files} files {number_of_folders} folders" ) 107 | if request_compression and sum_size != 0: 108 | sum_compressed_size = sum( fo.compressed_size for fo in files if fo.is_file ) 109 | r = sum_compressed_size/sum_size*100 110 | _verbose_print(f"Sum of compressed sizes {sum_compressed_size}, compressed/original={r:.1f}%") 111 | 112 | return True 113 | 114 | def _files_to_python( outfile, files, silent ): 115 | 116 | # Generate one const for each file 117 | for fo in files: 118 | if fo.is_file: 119 | pythonized = fo.get_pythonized() 120 | c = "" 121 | if fo.compressed: 122 | c = f", compressed level={fo.level} wbits={fo.wbits} compressed size={fo.compressed_size}" 123 | outfile.write(f"# {fo.mp_path} size={fo.size}{c}\n" ) 124 | outfile.write(f"{fo.varname} = const(\n{pythonized})\n") 125 | 126 | # Generate the directory entries 127 | outfile.write("direntries = const((") 128 | for fo in files: 129 | if fo.is_file: 130 | direntry = f"( {fo.varname}, {fo.compressed}, {fo.size} )" 131 | ratio = "" 132 | if fo.compressed: 133 | r = fo.compressed_size / fo.size * 100 134 | ratio = f", compressed/original={r:.0f}%" 135 | _verbose_print(f"Appended file {fo.pc_path} ({fo.size} bytes) as {fo.mp_path}{ratio}" ) 136 | else: 137 | direntry = f"None" 138 | _verbose_print(f"Appended folder {fo.pc_path} as {fo.mp_path}") 139 | outfile.write(f" ( '{fo.mp_path}', {direntry} ),\n" ) 140 | outfile.write("))\n\n" ) 141 | # Generate file info 142 | outfile.write(f"version = const({VERSION})\n") 143 | t = time.localtime() 144 | outfile.write(f"date_frozen = const( '{t[0]}/{t[1]:02d}/{t[2]:02d} {t[3]:02d}:{t[4]:02d}:{t[5]:02d}' )\n\n") 145 | sum_size = sum( fo.size for fo in files if fo.is_file ) 146 | outfile.write(f"sum_size = const({sum_size})\n" ) 147 | outfile.write(f"files_folders = const({len(files)})\n" ) 148 | 149 | def _generate_appended_code( outfile, 150 | mc_target, on_import, overwrite, silent ): 151 | 152 | # Compute which drivers to include 153 | 154 | include_file = "ffs" + on_import + ".py" 155 | # Include driver 156 | 157 | # Open driver files same folder where freezefs.py is 158 | f = Path( __file__ ).parent / include_file 159 | with open( f, "r" ) as file: 160 | while True: 161 | line = file.readline() 162 | if line == "": 163 | break 164 | # Don't copy comments nor empty lnes 165 | s = line.strip(" ") 166 | if s.startswith("#") or s == "\n": 167 | continue 168 | # Make indents one by one instead of by four 169 | line = line.replace(" ", " ") 170 | outfile.write( line ) 171 | outfile.write("\n") 172 | 173 | if on_import == "mount": 174 | outfile.write(f"mount_fs( __name__, '{mc_target}', {silent} )") 175 | elif on_import == "extract": 176 | outfile.write(f"extract_fs( __name__, '{mc_target}', '{overwrite}', {silent} )") 177 | 178 | 179 | DESC = """freezefs.py 180 | freezefs saves a file structure with subfolders and builds an self-extractable or self-mountable archive in a .py file, optionally with compression, to be frozen as bytecode or extracted. 181 | 182 | 183 | Examples: 184 | freezefs.py input_folder frozenfiles.py --target=/myfiles --on_import mount 185 | freezefs.py input_folder frozenfiles.py --target=/myfiles --on_import=extract --compress 186 | 187 | """ 188 | def main(): 189 | parser = argparse.ArgumentParser( 190 | "python -m freezefs", 191 | description=DESC, 192 | formatter_class = argparse.RawDescriptionHelpFormatter) 193 | parser.add_argument('infolder', type=str, help='Input folder path') 194 | parser.add_argument('outfile', type=str, 195 | help='Path and name of output module. Must have .py extension.') 196 | 197 | parser.add_argument("--on-import", "-oi", type=str, 198 | dest="on_import", 199 | choices=["mount", "extract"], 200 | default="mount", 201 | help="Action when importing output module. Default is mount.") 202 | 203 | parser.add_argument("--target", "-t", type=str, 204 | dest="target", 205 | help="For --on-import=mount: mount point." 206 | " For --on-import=extract: destination folder." 207 | " Default: the infolder." 208 | " Example: --target /myfiles. Must start with /") 209 | 210 | parser.add_argument("--overwrite", "-ov", type=str, 211 | dest="overwrite", 212 | choices=["never", "always"], 213 | default="never", 214 | help="always: on extract, all files are overwritten. never: on extract, no file is overwritten, only new files are extracted. Default: never. ") 215 | 216 | # Compress parameters 217 | parser.add_argument("--compress", "-c", 218 | dest="compress", default=False, 219 | action=argparse.BooleanOptionalAction, 220 | help="Compress files before writing to output .py. See python zlib compression." ) 221 | 222 | parser.add_argument("--wbits", "-w", 223 | type=int, 224 | dest="wbits", 225 | default=10, 226 | help="Compression window of 2**WBITS bytes. Between 9 and 14." 227 | " Default is 10 (1024 bytes)") 228 | parser.add_argument("--level", "-l", 229 | type=int, 230 | dest="level", 231 | default=9, 232 | help="Compression level. Between 0 (no compression) and 9 (best compression)." 233 | " Default is 9" ) 234 | parser.add_argument("--silent", "-s", 235 | dest="silent", default=False, 236 | action="store_true", 237 | help="Supress messages printed when mounting/copying files" 238 | " and while running this program.") 239 | 240 | args = parser.parse_args() 241 | 242 | # Use pathlib paths to make code independent of operati 243 | pc_infolder = Path( args.infolder ) 244 | pc_outfile = Path( args.outfile ) 245 | 246 | 247 | 248 | if not pc_infolder.is_dir(): 249 | quit("Input folder does not exist, or is not a folder") 250 | 251 | if not pc_outfile.suffix.lower() == '.py': 252 | quit('Output filename must have a .py extension.') 253 | 254 | if args.target: 255 | mc_target = PurePosixPath( args.target ) 256 | if not args.target.startswith("/"): 257 | quit( "Target must start with /") 258 | 259 | if args.target.endswith("/") and args.target != "/": 260 | quit( "Target must not end with /") 261 | else: 262 | mc_target = PurePosixPath( "/" + pc_infolder.stem ) 263 | print( f"Target set to {mc_target}" ) 264 | 265 | if not ( 9 <= args.wbits <= 14 ): 266 | quit("--wbits must be between 9 and 14") 267 | 268 | if not( 0 <= args.level <= 9 ): 269 | quit("--level must be between 0 and 9" ) 270 | 271 | 272 | if mc_target.stem == pc_outfile.stem: 273 | print(f"--target must be a different name than output file: {pc_outfile.stem}") 274 | sys.exit(1) 275 | 276 | _verbose_print(f'Writing Python file {pc_outfile}.') 277 | if not to_python( pc_infolder, pc_outfile, 278 | mc_target, args.on_import, args.overwrite, 279 | args.silent, 280 | args.compress, args.wbits, args.level ): 281 | sys.exit(1) 282 | 283 | 284 | module_name = pc_outfile.stem 285 | 286 | if args.on_import == "mount": 287 | _verbose_print(f"On import the file system will be mounted at {mc_target}." ) 288 | elif args.on_import == "extract": 289 | if args.overwrite == "never": 290 | _verbose_print(f"On import the file system will be extracted to {mc_target} writing only files that don't exist." ) 291 | else: 292 | _verbose_print(f"On import the file system will be extracted to {mc_target} overwriting all files." ) 293 | 294 | if args.on_import == "mount" and args.compress: 295 | # This is the only combination that is heavy in RAM use. 296 | print("sWarning: --on-import=mount and --compress loads file in RAM when opening files in text 'r' mode. High RAM use." ) 297 | 298 | _verbose_print(pc_outfile, 'written successfully.') 299 | -------------------------------------------------------------------------------- /freezefs/ffsextract.py: -------------------------------------------------------------------------------- 1 | # (c) 2023 Hermann Paul von Borries 2 | # MIT License 3 | # freezefs file extract driver for MicroPython 4 | 5 | import os 6 | import errno 7 | import sys 8 | 9 | class _VerbosePrint: 10 | def __init__( self, module_name, function, silent ): 11 | self.silent = silent 12 | self.prefix = f"{module_name}.{function}" 13 | 14 | def print( self, *args ): 15 | if not self.silent: 16 | print( self.prefix, *args ) 17 | 18 | def _file_exists( filename ): 19 | try: 20 | open( filename ).close() 21 | return True 22 | except: 23 | return False 24 | 25 | 26 | def _extract_file( dir_entry, destination ): 27 | 28 | from io import BytesIO 29 | data = dir_entry[0] 30 | if not data: 31 | return 32 | # Process in small chunks to reduce memory use 33 | buffer = bytearray(256) 34 | stream = BytesIO( data ) 35 | if dir_entry[1]: 36 | # This is a compressed file 37 | # Import here, so this is imported only when needed 38 | from deflate import DeflateIO, AUTO 39 | stream = DeflateIO( stream, AUTO, 0, True ) 40 | with open( destination, "wb" ) as outfile: 41 | while True: 42 | n = stream.readinto( buffer ) 43 | if n == 0: 44 | break 45 | outfile.write( memoryview( buffer )[0:n] ) 46 | stream.close() 47 | 48 | def _create_folder( folder, vp ): 49 | path = "" 50 | # Create parent folders first, then the specified folder 51 | for p in folder.split("/"): 52 | path += "/" + p 53 | path = path.replace("//", "/" ) 54 | try: 55 | os.mkdir( path ) 56 | vp.print( f"folder {path} created." ) 57 | except Exception as e: 58 | if type(e) is not OSError or e.errno != errno.EEXIST: 59 | vp.print( f"folder {path} not created: ", e ) 60 | 61 | def _extract_all( direntries, target, overwrite, vp ): 62 | _create_folder( target, vp ) 63 | 64 | # Don't mount the frozen file system, 65 | # access file data through the internal file structure. 66 | for filename, dir_entry in direntries: 67 | # get destination filename 68 | dest = (target + filename).replace("//", "/") 69 | if dir_entry: 70 | # Copy file 71 | try: 72 | if overwrite == "never" and _file_exists( dest ): 73 | vp.print( f"file {dest} exists, not extracted." ) 74 | else: 75 | vp.print( f"extracting file {dest}." ) 76 | _extract_file( dir_entry, dest ) 77 | except Exception as e: 78 | vp.print( f"file {dest} not copied: {e}." ) 79 | raise 80 | else: 81 | # Create folder (and it's parent folders if not created yet) 82 | # This relies on directory entries being sorted, so 83 | # the dir_entry for the folder comes up before the files it contains. 84 | _create_folder( dest, vp ) 85 | 86 | 87 | # Called from the generated (frozen) .py module 88 | def extract_fs( module_name, target, overwrite, silent ): 89 | 90 | # Delete this module from list of loaded modules to help 91 | # free the memory after use. Calling program must use __import__ 92 | # __main__ is not in sys.modules[] 93 | module = __import__( module_name ) 94 | if module_name != "__main__": 95 | del sys.modules[ module_name ] 96 | 97 | vp = _VerbosePrint( module_name, "extract", silent ) 98 | 99 | vp.print( f"extracting files to {target}." ) 100 | _extract_all( direntries, target, overwrite, vp ) 101 | 102 | return -------------------------------------------------------------------------------- /freezefs/ffsmount.py: -------------------------------------------------------------------------------- 1 | # (c) 2023 Hermann Paul von Borries 2 | # MIT License 3 | # MicroPython VFS mount driver for freezefs 4 | import os 5 | import errno 6 | from io import BytesIO, StringIO 7 | from collections import OrderedDict 8 | import sys 9 | 10 | def _get_basename( filename ): 11 | return filename.split("/")[-1] 12 | 13 | def _get_folder( filename ): 14 | basename = _get_basename( filename ) 15 | folder = filename[0:-len(basename)-1] 16 | if folder == "": 17 | folder = "/" 18 | return folder 19 | 20 | 21 | class VfsFrozen: 22 | # File system for frozen files. Implements mount, listdir, stat, open etc 23 | # Actual read, readinto, seek, are done by BytesIO and StringIO 24 | def __init__( self, direntries, sum_size, files_folders ): 25 | # direntries is (filename, ( data, compressed, size ) for files 26 | # (filename, None) for folders 27 | self.filedict = OrderedDict( direntries ) 28 | self.sum_size = sum_size 29 | self.files_folders = files_folders 30 | self.path = "/" 31 | 32 | def _to_absolute_filename( self, filename ): 33 | if not filename.startswith( "/" ) : 34 | filename = self.path + "/" + filename 35 | if filename.endswith("/") and filename != "/": 36 | filename = filename[0:-1] 37 | filename = filename.replace("//", "/") 38 | 39 | # Solve ".." 40 | parts = filename.split("/") 41 | i = -1 42 | while ".." in parts: 43 | i = parts.index("..") 44 | if i > 1: 45 | del parts[i] 46 | del parts[i-1] 47 | else: 48 | # Can't access /.. 49 | self._raise_perm() 50 | # Solve ./file or /folder/./file 51 | while "." in parts: 52 | i = parts.index(".") 53 | del parts[i] 54 | filename = "".join( "/" + p for p in parts ).replace( "//", "/" ) 55 | return filename 56 | 57 | def _find_file( self, filename ): 58 | filename = self._to_absolute_filename( filename ) 59 | if filename == "/": 60 | # The root is a folder. filedict doesn't have 61 | # the root. Return folder direntry for the root. 62 | return None 63 | if filename in self.filedict: 64 | # Return the file directory entry (dir_entry) 65 | return self.filedict[filename] 66 | else: 67 | raise OSError( errno.ENOENT ) 68 | 69 | def _raise_perm( self ): 70 | raise OSError(errno.EPERM) # Very common here 71 | 72 | def open( self, filename, mode, buffering=None ): 73 | # Validate mode before opening file 74 | for c in mode: 75 | # Modes may be "r"/"rt"/"tr" or "rb"/"br" 76 | if c not in "rbt": 77 | raise OSError( errno.EINVAL ) 78 | 79 | dir_entry = self._find_file( filename ) 80 | if dir_entry is None: 81 | # This is a folder or the root of this file system 82 | if filename == "/": 83 | self._raise_perm() 84 | raise OSError(errno.EISDIR) 85 | 86 | data = dir_entry[0] # data 87 | if not dir_entry[1]: # compressed 88 | if "b" in mode: 89 | return BytesIO( data ) 90 | else: 91 | return StringIO( data ) 92 | else: 93 | # Compressed file - late import of deflate library 94 | from deflate import DeflateIO, AUTO 95 | uncompressed_stream = DeflateIO( BytesIO( data ), 96 | AUTO, 0, True ) 97 | if "b" in mode: 98 | return uncompressed_stream 99 | else: 100 | # This requires to buffer the entire file... 101 | # Only useful if enough RAM is available. 102 | return StringIO( uncompressed_stream.read() ) 103 | 104 | def chdir( self, path ): 105 | newdir = self._to_absolute_filename( path ) 106 | dir_entry = self._find_file( newdir ) 107 | if dir_entry is None: 108 | # ok, it's a folder 109 | self.path = newdir 110 | return 111 | raise OSError( -2 ) 112 | 113 | def getcwd( self ): 114 | if self.path != "/" and self.path.endswith( "/" ): 115 | return self.path[0:-1] 116 | return self.path 117 | 118 | def ilistdir( self, path ): 119 | abspath = self._to_absolute_filename( path ) 120 | # Test if folder exists, if not raise OSError ENOENT 121 | self._find_file( abspath ) 122 | # Find all files 123 | for filename, dir_entry in self.filedict.items(): 124 | if _get_folder( filename ) == abspath: 125 | basename = _get_basename( filename ) 126 | if dir_entry is not None: 127 | yield ( basename, 0x8000, 0, dir_entry[2] ) 128 | else: 129 | yield ( basename, 0x4000, 0, 0 ) 130 | 131 | def stat( self, filename ): 132 | dir_entry = self._find_file( filename ) 133 | if dir_entry is None: 134 | return (0x4000, 0,0,0,0,0, 0, 0,0,0) 135 | else: 136 | return (0x8000, 0,0,0,0,0, dir_entry[2], 0,0,0) 137 | 138 | 139 | def statvfs( self, *args ): 140 | # statvfs returns: 141 | # block size, fragment size, blocks, 142 | # free blocks, available blocks, files, 143 | # free inodes1, free_inodes2, mount flags=readonly 144 | # maximum filename length 145 | # Return block size of 1 byte = allocation unit 146 | # No free space. One "inode" per file or folder. 147 | # Mount flags: readonly 148 | # Max filename size 255 (checked in freezeFS) 149 | #sum_size = sum( d[2] for d in self.filedict.values() if d is not None ) 150 | return (1,1,self.sum_size, 0,0,self.files_folders, 0,0,1, 255) 151 | 152 | def mount( self, readonly, x ): 153 | self.path = "/" 154 | 155 | def remove( self, filename ): 156 | self._raise_perm() 157 | 158 | def mkdir( self, *args ): 159 | self._raise_perm() 160 | 161 | def rename( self, oldfname, newfname ): 162 | self._raise_perm() 163 | 164 | def umount( self ): 165 | # No specific cleanup necessary on umount. 166 | pass 167 | 168 | # Added rmdir for completeness 169 | def rmdir(self): 170 | self._raise_perm() 171 | 172 | def mount_fs( frozen_module_name, target, silent ): 173 | 174 | if target is None: 175 | raise ValueError("No target specified") 176 | 177 | # Check target doesn't exist 178 | file_exists = False 179 | try: 180 | os.stat( target ) 181 | file_exists = True 182 | except: 183 | pass 184 | if file_exists: 185 | raise OSError( errno.EEXIST ) 186 | 187 | if not silent: 188 | print( f"mounting {__name__} at {target}." ) 189 | 190 | os.mount( VfsFrozen( direntries, sum_size, files_folders ), target, readonly=True ) 191 | return True 192 | -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | [build-system] 2 | requires = ["hatchling"] 3 | build-backend = "hatchling.build" 4 | 5 | [project] 6 | name = "freezefs" 7 | version = "2.2" 8 | authors = [ 9 | { name="Hermann von Borries", email="bixb922@gmail.com" } 10 | ] 11 | description = "freezefs - Create self-mounting or self-extracting compressed archives for MicroPython" 12 | readme = "README.md" 13 | license = { file="LICENSE" } 14 | requires-python = ">=3.9" 15 | classifiers = [ 16 | "Programming Language :: Python :: 3", 17 | "License :: OSI Approved :: MIT License", 18 | "Operating System :: OS Independent", 19 | "Intended Audience :: Developers", 20 | "Programming Language :: Python :: Implementation :: MicroPython", 21 | "Programming Language :: Python :: 3.9", 22 | "Programming Language :: Python :: 3.10", 23 | "Programming Language :: Python :: 3.11", 24 | "Programming Language :: Python :: Implementation :: CPython", 25 | "Programming Language :: Python :: Implementation :: MicroPython", 26 | "Topic :: Desktop Environment :: File Managers" 27 | ] 28 | 29 | [project.urls] 30 | "Homepage" = "https://github.com/bixb922/freezefs" 31 | "Bug Tracker" = "https://github.com/bixb922/freezefs/issues" 32 | "Blog" = "https://solazyasueto.blogspot.com" 33 | 34 | -------------------------------------------------------------------------------- /tests/cleantest.bat: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | mpremote rm frozenfiles_mount.mpy 3 | mpremote run "import shutil;shutil.rmtree('testfiles')" 4 | del frozenfiles_mount.py 5 | del frozenfiles_mount.mpy 6 | del /S /Q testfiles 7 | rmdir /S /Q testfiles -------------------------------------------------------------------------------- /tests/runtest.bat: -------------------------------------------------------------------------------- 1 | rem create testfiles folder with testfiles locally 2 | test.py 3 | rem create freezeFS file to mount and test 4 | python ..\freezefs testfiles frozenfiles_mount.py --target /fz --on-import mount 5 | mpremote cp frozenfiles_mount.py : 6 | mpremote run test.py 7 | -------------------------------------------------------------------------------- /tests/test.py: -------------------------------------------------------------------------------- 1 | # (c) 2023 Hermann Paul von Borries 2 | # MIT License 3 | 4 | 5 | # Test path: /./folder /folder/../folder /.. 6 | 7 | import sys 8 | import os 9 | import unittest 10 | import time 11 | import errno 12 | import gc 13 | import shutil # mpremote mip install shutil (standard from micropython-lib) 14 | 15 | PYTHON = sys.implementation.name 16 | if PYTHON == "cpython": 17 | from pathlib import Path 18 | else: 19 | Path = lambda *args : "".join( args ) 20 | 21 | referencefolder = "testfiles" 22 | testfolder = "fz" 23 | 24 | test_folders = [ "sub1", "sub2", "sub1/sub2", "sub1/sub2/sub3" ] 25 | 26 | def iter_both_test_folders(): 27 | for folder in test_folders: 28 | yield Path( referencefolder + "/" + folder ), Path( testfolder + "/" + folder ) 29 | 30 | # txt files are ascii, bin are binary, uni are unicode 31 | test_files = [ 32 | ("file1.txt", 100), 33 | ("file2.txt", 0), 34 | ("file3.bin", 100), 35 | ("file4.bin", 0), 36 | ("file5.txt", 1), 37 | ("file6.uni", 10), # Size of unicode files is in characters, not bytes 38 | ("file7.uni", 100), # Size of unicode files is in characters, not bytes 39 | ("file8.uni", 1000), # Size of unicode files is in characters, not bytes 40 | ("file9.uni", 10000), # Size of unicode files is in characters, not bytes 41 | ("sub1/file1.txt", 10), 42 | ("sub1/file2.txt", 20), 43 | ("sub1/sub2/file1.txt", 13), 44 | ("sub1/sub2/file2.txt", 11 ), 45 | ("sub1/sub2/file3.txt", 21 ), 46 | ("sub1/sub2/sub3/file4.txt", 77 ) 47 | ] 48 | 49 | 50 | def iter_both_test_files( filetype=None ): 51 | for f, length in test_files: 52 | if filetype == None or f.endswith( "." + filetype ): 53 | yield Path( referencefolder + "/" + f ), Path( testfolder + "/" + f ) 54 | 55 | 56 | # Old fashioned pseudorandom generator to generate 57 | # same test data on PC and microcontroller 58 | rand = 11 59 | m = 2**31-1 60 | a = 1664525 61 | c = 1013904223 62 | def getrand(n): 63 | global rand 64 | rand += (a*rand+c)%m 65 | return rand%n 66 | 67 | 68 | # Some characters for testing 69 | unicodes = "aáéíóúÁÉÍÓÚäëïöüÄËÏÖÜñÑ" + chr(0x1f600) + chr(0x1f603) + chr(0x1f604) 70 | asciis = "".join( chr(x) for x in range(0,127) ) + "\r" + "\n" + "\t" 71 | 72 | def write_str( filename, length, alphabet ): 73 | # Don't use newline to mirror Micropython 74 | if PYTHON == "cpython": 75 | fileopen = lambda fn : open( filename, "wt", encoding="utf-8", newline="" ) 76 | else: 77 | fileopen = lambda fn : open( filename, "wt" ) 78 | with fileopen( filename ) as file: 79 | for _ in range(length): 80 | p = getrand( len(alphabet) ) 81 | s = alphabet[p:p+1] 82 | file.write( s ) 83 | 84 | def write_text( filename, length ): 85 | write_str( filename, length, asciis ) 86 | 87 | def write_bin( filename, length ): 88 | with open( filename, "wb" ) as file: 89 | for _ in range(length): 90 | b = bytearray(1) 91 | b[0] = getrand( 255 ) 92 | file.write( b ) 93 | 94 | 95 | def write_uni( filename, length ): 96 | if length <= 10: 97 | write_str( filename, length, unicodes ) 98 | else: 99 | write_str( filename, length, asciis + unicodes ) 100 | 101 | def make_testfiles( ): 102 | try: 103 | shutil.rmtree( referencefolder ) 104 | except: 105 | pass 106 | # This has to run both on PC/Python and MicroPython 107 | # Create folders 108 | for f in [""] + test_folders: 109 | if f == "": 110 | filename = referencefolder 111 | else: 112 | filename = referencefolder + "/" + f 113 | print("make_testfiles: Create folder", filename ) 114 | try: 115 | os.mkdir( Path( filename ) ) 116 | except Exception as e: 117 | print(f"make_testfiles: Could not create folder {filename}, {e}" ) 118 | # Ignore errors, sometimes delete folder fails on Windows 119 | 120 | # Write files 121 | for f, length in test_files: 122 | filename = Path( referencefolder + "/" + f ) 123 | if ".txt" in f: 124 | write_text( filename, length ) 125 | elif ".bin" in f: 126 | write_bin( filename, length ) 127 | elif ".uni" in f: 128 | write_uni( filename, length ) 129 | print("make_testfiles: Test files created in folder ", referencefolder) 130 | 131 | 132 | class TestFiles(unittest.TestCase): 133 | 134 | def compare_lists( self, list1, list2 ): 135 | # With infinite memory, this could compare tuple( list1 ) with tuple( list2 ) 136 | self.assertEqual( len(list1), len(list2) ) 137 | for i, p in enumerate( list1 ): 138 | self.assertEqual( p, list2[i] ) 139 | 140 | 141 | def check_testfile_size( self, folder ): 142 | for f, defsize in test_files: 143 | if ".uni" in f: 144 | # File size of unicode files is in characters, not bytes 145 | # can't compare here 146 | continue 147 | filename = "/" + folder + "/" + f 148 | filename = filename.replace("//", "/") 149 | try: 150 | size = os.stat( filename )[6] 151 | except OSError: 152 | break 153 | self.assertEqual( defsize, size ) 154 | 155 | def test_full_read_binary( self ): 156 | for filename1, filename2 in iter_both_test_files( "bin" ): 157 | with open( filename1, "rb" ) as file1: 158 | c1 = file1.read() 159 | with open( filename2, "rb") as file2: 160 | c2 = file2.read() 161 | self.assertEqual( c1, c2 ) 162 | 163 | def test_full_read_text( self ): 164 | for filename1, filename2 in iter_both_test_files( "txt" ): 165 | with open( filename1, "r" ) as file1: 166 | c1 = file1.read() 167 | with open( filename2, "r" ) as file2: 168 | c2 = file2.read() 169 | 170 | self.assertEqual( c1, c2 ) 171 | 172 | def test_full_try_binary_as_text( self ): 173 | for filename1, filename2 in iter_both_test_files( "bin" ): 174 | if os.stat( filename1 )[6] == 0 and os.stat( filename2 )[6] == 0: 175 | # Length 0, will not raise error 176 | continue 177 | # Reading binary file as text should raise UnicodeError 178 | with self.assertRaises( UnicodeError ): 179 | with open( filename1, "r" ) as file1: 180 | c1 = file1.read() 181 | with self.assertRaises( UnicodeError ): 182 | with open( filename2, "r") as file2: 183 | c2 = file2.read() 184 | 185 | 186 | def test_full_read_0( self ): 187 | for file1, file2 in iter_both_test_files( "txt" ): 188 | with open( file1, "r" ) as file: 189 | c1 = file.read(0) 190 | d1 = file.read(10) 191 | e1 = file.read(0) 192 | f1 = file.read(20) 193 | with open( file2, "r" ) as file: 194 | c2 = file.read(0) 195 | d2 = file.read(10) 196 | e2 = file.read(0) 197 | f2 = file.read(20) 198 | 199 | self.assertEqual( c1, c2 ) 200 | self.assertEqual( d1, d2 ) 201 | self.assertEqual( e1, e2 ) 202 | self.assertEqual( f1, f2 ) 203 | 204 | 205 | def test_read_entire_text( self ): 206 | print("") 207 | for filetype in ("txt", "uni"): 208 | for filename1, filename2 in iter_both_test_files( filetype ): 209 | 210 | with open( filename1, "r") as file1: 211 | gc.collect() 212 | data1 = file1.read() 213 | print(f"test_read_entire_text {filename1=} {len(data1)=} characters") 214 | with open( filename2, "r") as file2: 215 | gc.collect() 216 | data2 = file2.read() 217 | print(f"test_read_entire_text {filename2=} {len(data2)=} characters") 218 | self.assertEqual( data1, data2 ) 219 | data1 = None 220 | data2 = None 221 | gc.collect() 222 | 223 | def test_read_text( self ): 224 | for filetype in ("txt", "uni"): 225 | for length in ( 1, 10, 100, 1000 ): 226 | print(f"Test read text with read({length})" ) 227 | for filename1, filename2 in iter_both_test_files( filetype ): 228 | file1 = open( filename1, "r" ) 229 | file2 = open( filename2, "r" ) 230 | while True: 231 | data1 = file1.read( length ) 232 | data2 = file2.read( length ) 233 | self.assertEqual( data1, data2 ) 234 | if not data1: 235 | break 236 | file1.close() 237 | file2.close() 238 | 239 | 240 | def test_readline( self ): 241 | for filetype in ("txt", "uni"): 242 | for filename1, filename2 in iter_both_test_files( filetype ): 243 | for mode in ("r", "rb"): 244 | file1 = open( filename1, mode ) 245 | file2 = open( filename2, mode ) 246 | while True: 247 | data1 = file1.readline() 248 | data2 = file2.readline() 249 | self.assertEqual( data1, data2 ) 250 | if not data1: 251 | break 252 | file1.close() 253 | file2.close() 254 | 255 | 256 | 257 | def test_readline3( self ): 258 | for filetype in ("txt", "uni"): 259 | for file1, file2 in iter_both_test_files( filetype ): 260 | with open( file1, "r" ) as file: 261 | parts1 = [ _ for _ in file ] 262 | with open( file2, "r") as file: 263 | parts2 = [ _ for _ in file ] 264 | self.compare_lists( parts1, parts2 ) 265 | 266 | 267 | def test_read_binary( self ): 268 | for length in ( 1, 10, 100, 1000 ): 269 | print(f"Test read binary with read({length})" ) 270 | for filename1, filename2 in iter_both_test_files( "bin" ): 271 | file1 = open( filename1, "rb" ) 272 | file2 = open( filename2, "rb" ) 273 | while True: 274 | data1 = file1.read( length ) 275 | data2 = file2.read( length ) 276 | self.assertEqual( data1, data2 ) 277 | if not data2: 278 | break 279 | file1.close() 280 | file2.close() 281 | 282 | def test_listdir( self ): 283 | for rf, tf in iter_both_test_folders(): 284 | files1 = os.listdir( "/" + rf ) 285 | files2 = os.listdir( "/" + tf ) 286 | self.assertEqual( tuple(files1), tuple(files2) ) 287 | 288 | def test_chdir( self ): 289 | for fol in test_folders: 290 | os.chdir( "/" + referencefolder + "/" + fol ) 291 | files1 = os.listdir( "" ) 292 | os.chdir( "/" + testfolder + "/" + fol ) 293 | files2 = os.listdir( "" ) 294 | self.assertEqual( tuple( files1 ), tuple( files2 ) ) 295 | 296 | # chdir with ending / 297 | os.chdir( "/" + referencefolder + "/sub1/" ) 298 | self.assertEqual( "/" + referencefolder + "/sub1", os.getcwd() ) 299 | os.stat( "file9.uni" ) 300 | files1 = os.listdir( "" ) 301 | 302 | 303 | os.chdir( "/" + testfolder + "/sub1/" ) 304 | self.assertEqual( "/" + testfolder + "/sub1", os.getcwd() ) 305 | os.stat( "file9.uni" ) 306 | files2 = os.listdir( "" ) 307 | 308 | self.assertEqual( tuple( files1 ), tuple( files2 ) ) 309 | 310 | 311 | # Bad chdir: to nonexistent folder 312 | for folder in ( referencefolder, testfolder ): 313 | with self.assertRaises( OSError ): 314 | os.chdir("/" + folder + "/nonexistent") 315 | 316 | # Bad chdir: to file 317 | for folder in ( referencefolder, testfolder ): 318 | with self.assertRaises( OSError ): 319 | os.chdir("/" + folder + "/file1.txt") 320 | 321 | # chdir to relative path 322 | for folder in ( referencefolder, testfolder ): 323 | newdir = "/" + folder 324 | os.chdir( newdir ) 325 | self.assertEqual( mewdir, os.getcwd() ) 326 | newdir += "sub1" 327 | os.chdir( "sub1" ) 328 | self.assertEqual( mewdir, os.getcwd() ) 329 | newdir += "sub2" 330 | os.chdir( "sub2" ) 331 | self.assertEqual( mewdir, os.getcwd() ) 332 | 333 | os.chdir(rf + "/sub1/sub2/.." ) 334 | files1 = os.listdir() 335 | os.chdir( tf + "/sub1/sub2/.." ) 336 | files2 = os.listdir() 337 | self.assertEqual(files1, files2 ) 338 | 339 | os.chdir(rf + "/sub1/sub2/." ) 340 | files1 = os.listdir() 341 | os.chdir( tf + "/sub1/sub2/." ) 342 | files2 = os.listdir() 343 | self.assertEqual(files1, files2 ) 344 | 345 | os.chdir(rf + "/sub1/./sub2/.." ) 346 | files1 = os.listdir() 347 | os.chdir( tf + "/sub1/./sub2/.." ) 348 | files2 = os.listdir() 349 | self.assertEqual(files1, files2 ) 350 | 351 | os.chdir(rf + "/,/sub1/sub2/.." ) 352 | files1 = os.listdir() 353 | os.chdir( tf + "/./sub1/sub2/.." ) 354 | files2 = os.listdir() 355 | self.assertEqual(files1, files2 ) 356 | 357 | 358 | os.chdir("/") 359 | 360 | 361 | def test_ilistdir( self ): 362 | for rf, tf in iter_both_test_folders(): 363 | files1 = [ _ for _ in os.listdir( "/" + rf ) ] 364 | files2 = [ _ for _ in os.listdir( "/" + tf ) ] 365 | self.assertEqual( tuple(files1), tuple(files2) ) 366 | 367 | 368 | def test_stat( self ): 369 | for filename, length in test_files: 370 | file1 = "/" + testfolder + "/" + filename 371 | file2 = "/" + referencefolder + "/" + filename 372 | stat1 = os.stat( file1 )[0:6] 373 | stat2 = os.stat( file2 )[0:6] 374 | self.assertEqual( stat1, stat2 ) 375 | 376 | stat1 = os.stat( testfolder )[0:6] 377 | stat2 = os.stat( referencefolder )[0:6] 378 | self.assertEqual( stat1, stat2 ) 379 | 380 | def test_parent_dir( self ): 381 | rf = "/" + referencefolder 382 | tf = "/" + testfolder 383 | stat1 = os.stat( rf + "/sub1/sub2/../file1.txt" ) 384 | stat2 = os.stat( tf + "/sub1/sub2/../file1.txt" ) 385 | self.assertEqual( stat1[0:6], stat2[0:6] ) 386 | 387 | os.chdir( rf + "/sub1/../sub1/sub2") 388 | p1 = os.getcwd().replace( rf, "/") 389 | os.chdir( tf + "/sub1/../sub1/sub2") 390 | p2 = os.getcwd().replace( tf, "/") 391 | self.assertEqual( p1, p2 ) 392 | 393 | with self.assertRaises( OSError ): 394 | os.chdir( tf + "/.." ) 395 | 396 | 397 | os.chdir("/") 398 | 399 | def test_stat_folder( self ): 400 | for filename in test_folders: 401 | file1 = "/" + testfolder + "/" + filename 402 | file2 = "/" + referencefolder + "/" + filename 403 | stat1 = os.stat( file1 )[0:6] 404 | stat2 = os.stat( file2 )[0:6] 405 | self.assertEqual( stat1, stat2 ) 406 | 407 | def test_chdir( self ): 408 | tf = "/" + testfolder 409 | rf = "/" + referencefolder 410 | os.chdir( tf ) 411 | self.assertEqual( os.getcwd(), tf ) 412 | files1 = os.listdir("") 413 | files2 = os.listdir( rf ) 414 | self.assertEqual( tuple( files1 ), tuple( files2 ) ) 415 | 416 | os.chdir("sub1") 417 | self.assertEqual( os.getcwd(), tf +"/sub1") 418 | files1 = os.listdir("") 419 | files2 = os.listdir( rf + "/sub1" ) 420 | self.assertEqual( tuple( files1 ), tuple( files2 ) ) 421 | 422 | os.chdir( tf +"/sub1") 423 | self.assertEqual( os.getcwd(), tf + "/sub1") 424 | files1 = os.listdir("") 425 | files2 = os.listdir( rf + "/sub1" ) 426 | self.assertEqual( tuple( files1 ), tuple( files2 ) ) 427 | 428 | os.chdir("sub2") 429 | self.assertEqual( os.getcwd(), "/fz/sub1/sub2") 430 | files1 = os.listdir("") 431 | files2 = os.listdir( rf + "/sub1/sub2" ) 432 | self.assertEqual( tuple( files1 ), tuple( files2 ) ) 433 | 434 | os.chdir("sub3") 435 | self.assertEqual( os.getcwd(), "/fz/sub1/sub2/sub3") 436 | files1 = os.listdir("") 437 | files2 = os.listdir( rf + "/sub1/sub2/sub3" ) 438 | self.assertEqual( tuple( files1 ), tuple( files2 ) ) 439 | 440 | os.chdir( "/fz" ) 441 | self.assertEqual( os.getcwd(), "/fz") 442 | files1 = os.listdir("") 443 | files2 = os.listdir( "/" + referencefolder ) 444 | self.assertEqual( tuple( files1 ), tuple( files2 ) ) 445 | 446 | os.chdir("/") 447 | 448 | def test_remove( self ): 449 | with self.assertRaises( OSError ): 450 | os.remove( testfolder +"/file1.txt") 451 | 452 | def test_write_not_allowed( self ): 453 | file1 = testfolder +"/file1.txt" 454 | with self.assertRaises( OSError ): 455 | open( file1, "w") 456 | with self.assertRaises( OSError ): 457 | open( file1, "wb") 458 | with self.assertRaises( OSError ): 459 | open( file1, "r+") 460 | with self.assertRaises( OSError ): 461 | open( file1, "a") 462 | 463 | with self.assertRaises( OSError ): 464 | os.rename( file1, testfolder + "/a.a") 465 | 466 | with self.assertRaises( OSError ): 467 | os.remove( file1 ) 468 | 469 | 470 | def test_seek_binary( self ): 471 | # Test seek/tell on binary files 472 | file1 = open( referencefolder + "/file3.bin" ,"rb") 473 | file2 = open( testfolder + "/file3.bin", "rb" ) 474 | for i in [1,51,2,92]: 475 | file1.seek(i) 476 | file2.seek(i) 477 | self.assertEqual( file1.tell(), file2.tell() ) 478 | c1 = file1.read(1) 479 | c2 = file2.read(1) 480 | self.assertEqual( c1, c2 ) 481 | 482 | for i in [1,51,-29,33]: 483 | file1.seek(i,1) 484 | file2.seek(i,1) 485 | self.assertEqual( file1.tell(), file2.tell() ) 486 | c1 = file1.read(1) 487 | c2 = file2.read(1) 488 | self.assertEqual( c1, c2 ) 489 | 490 | file1.seek(i,2) 491 | file2.seek(i,2) 492 | self.assertEqual( file1.tell(), file2.tell() ) 493 | c1 = file1.read(1) 494 | c2 = file2.read(1) 495 | self.assertEqual( c1, c2 ) 496 | 497 | def test_flush( self ): 498 | file1 = open( "/" + referencefolder + "/file3.bin" ,"rb") 499 | file2 = open( "/" + testfolder + "/file3.bin", "rb" ) 500 | self.assertEqual( file1.flush(), file2.flush() ) 501 | 502 | def test_statvfs( self ): 503 | t = os.statvfs( "/" + testfolder ) 504 | self.assertEqual( (1, 1, 13578, 0, 0, 19, 0, 0, 1, 255), t ) 505 | 506 | 507 | 508 | def timing_tests(): 509 | print("\nTiming tests. /testfolder is on the standard file system, /fz is the VfsFrozen file system running in RAM.\n" ) 510 | class HowLong: 511 | def __init__( self, name ): 512 | self.name = name 513 | def __enter__( self ): 514 | self.t0 = time.ticks_ms() 515 | return self 516 | def __exit__( self, exception_type, exception_value, traceback ): 517 | print(self.name, time.ticks_diff( time.ticks_ms(), self.t0 )) 518 | 519 | size = os.stat( referencefolder + "/file9.uni" )[6] 520 | 521 | for mode in ("r", "rb"): 522 | for i in (1,10,100,1000): 523 | for folder in (referencefolder, testfolder ) : 524 | filename = folder + "/file9.uni" 525 | size = os.stat( filename )[6] 526 | with HowLong(f"Timed file.read({i:4d}) for {filename:19s}, size={size} bytes, mode={mode:2s}, msec="): 527 | with open( filename, mode ) as file: 528 | s = "" 529 | while True: 530 | r = file.read(i) 531 | if len(r) == 0: 532 | break 533 | 534 | 535 | for folder in (referencefolder, testfolder ) : 536 | filename = folder + "/file9.uni" 537 | with HowLong(f"Timed file.readline() for {filename:19s}, size={size} bytes, mode=rb, msec="): 538 | with open( filename, "rb") as file: 539 | while True: 540 | line = file.readline() 541 | if len(line) == 0: 542 | break 543 | 544 | 545 | 546 | 547 | if __name__ == "__main__": 548 | if PYTHON == "cpython": 549 | make_testfiles() 550 | sys.exit() 551 | gc.collect() 552 | # Collect frequently 553 | gc.threshold( gc.mem_free()//2) 554 | make_testfiles() 555 | 556 | import frozenfiles_mount 557 | 558 | unittest.main() 559 | 560 | # Run without unittest: 561 | #t = TestFiles() 562 | #t.test_chdir() 563 | 564 | timing_tests() 565 | 566 | --------------------------------------------------------------------------------