├── .gitignore ├── README.md ├── freezetag ├── __init__.py ├── __main__.py ├── base.py ├── commands.py ├── core.py ├── formats │ ├── flac.py │ ├── generic.py │ └── mp3.py └── freezefs.py └── setup.py /.gitignore: -------------------------------------------------------------------------------- 1 | __pycache__ 2 | .idea 3 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | freezetag 2 | ========= 3 | 4 | About 5 | ----- 6 | 7 | `freezetag` is a tool that saves, strips, and restores file paths and music metadata. This metadata is written to a 8 | freezetag file (usually just a few kB) that can transform downloaded music files between different filename/tag states. 9 | 10 | Use cases: 11 | 1. Freezetags can be generated after a torrent finishes downloading to [freeze](#freeze) the directory state. Later, 12 | even after moving, retagging, and renaming those files, the original downloaded state can be restored 13 | ([thawed](#thaw)). 14 | 15 | 2. Additionally, these freezetags can be [mounted](#mount), allowing retagged music to coexist with the original 16 | downloaded music files without taking up extra disk space. This means you can retag your music and continue to seed 17 | in your torrent client from the same set of files. _Have your cake and eat it too!_ 18 | 19 | 3. In the same vein, users can seed torrents between different trackers with just one copy of the music files, even if 20 | the music has been retagged and renamed between trackers. 21 | 22 | 4. Freezetags can also be generated for an entire music library *after* the music has been retagged to [back up](#--backup) 23 | your personal tags. These quick incremental backups will archive your tags and filenames, which can later be restored 24 | from the original torrents. 25 | 26 | Requirements 27 | ------------ 28 | 29 | `freezetag` requires Python 3.5.2 or greater (older versions may technically work, but are untested). 30 | 31 | A FUSE implementation must also be installed to use [`freezetag mount`](#mount): 32 | * Windows users can install [WinFsp](https://github.com/billziss-gh/winfsp/releases/latest). 33 | * Mac users can install FUSE for macOS via `brew cask install osxfuse` or 34 | [manually](https://github.com/osxfuse/osxfuse/releases/latest). 35 | * Linux users can install `fuse2` from their package managers or 36 | [manually](https://github.com/libfuse/libfuse/releases/tag/fuse-2.9.9). 37 | 38 | Installation 39 | ------------ 40 | 41 | $> pip install git+https://github.com/x1ppy/freezetag 42 | 43 | Usage 44 | ----- 45 | 46 | ~~~ 47 | usage: freezetag [-h] command ... 48 | 49 | Saves, strips, and restores file paths and music metadata. 50 | 51 | positional arguments: 52 | command 53 | freeze Save paths and music metadata to a freezetag file. 54 | thaw Restore paths and music metadata from a freezetag file. 55 | mount Recursively mount a directory and its freezetags. 56 | shave Strip metadata from all music files. 57 | show Display the contents of a freezetag file. 58 | 59 | optional arguments: 60 | -h, --help show this help message and exit 61 | 62 | Use "freezetag [command] --help" for more information about a command. 63 | ~~~ 64 | 65 | Supported Formats 66 | ----------------- 67 | 68 | Currently, FLAC and MP3 formats are supported. Vorbis comments are supported for FLAC files, and ID3 tags are supported 69 | for MP3 files. 70 | 71 | Note that metadata will be frozen/thawed/shaved for supported music and metadata formats only. 72 | 73 | Examples 74 | -------- 75 | 76 | ### `freeze` 77 | 78 | Create a freezetag that stores paths and metadata for all files in this directory, saving the freezetag to a file 79 | named F**a**-**b**-**c**.ftag (as described in [Freezetag ID](#freezetag-id)) to this directory: 80 | 81 | $> freezetag freeze 82 | 83 | Same as above, except `downloads/Pink Floyd - Dark Side of the Moon (1973 MSFL UDCD 517) - FLAC` is used instead of the 84 | current directory: 85 | 86 | $> freezetag freeze "downloads/Pink Floyd - Dark Side of the Moon (1973 MSFL UDCD 517) - FLAC" 87 | 88 | If `--ftag` is passed, the freezetag will be written to that directory instead of the directory being frozen: 89 | 90 | $> freezetag freeze --ftag ~/ftags 91 | 92 | Or, if the `--ftag` argument is a file, the freezetag will be explicitly named (`redacted-pink-floyd-the-wall.ftag` in 93 | this case): 94 | 95 | $> freezetag freeze --ftag ~/ftags/redacted-pink-floyd-the-wall.ftag 96 | 97 | These examples all freeze a single album's state. With the default usage, it's recommended that each album has its own 98 | freezetag, as shown here. This way, other directories under `downloads` can be added/changed/removed without affecting 99 | the freezetag corresponding to each individual download. 100 | 101 | Freezetags are unique for a given group of files with the same metadata, so album-level freezetags can be shared among 102 | users to recreate the exact torrent download state. In fact, if two users independently create freezetags for the same 103 | downloaded directory, the freezetags (and their IDs) will be identical. In theory, trackers could even provide 104 | freezetags alongside download links that match the given releases, and freezetag IDs could be an API-queryable 105 | alternative to torrent info hashes. 106 | 107 | #### `--backup` 108 | 109 | `freezetag freeze` also includes an incremental backup mode, enabled with the `--backup` flag: 110 | 111 | $> freezetag freeze ~/music --backup 112 | 113 | This mode is intended to be used on your top-level music directory, and allows you to export all of your personal tags. 114 | `freezetag thaw` can later be used on the original torrent downloads in your `~/downloads` directory to fully recreate 115 | your personal tagged library. 116 | 117 | Unlike the default mode, backup mode writes last modified dates and file sizes to the freezetag. This enables 118 | incremental backups, meaning that the original `freezetag freeze --backup` of your library can take awhile (minutes to 119 | hours depending on the size of your library), but subsequent `freezetag freeze --backup`s can complete in mere seconds. 120 | Only the most recent freezetag will be read for an incremental backup, and if the library is unchanged since the last 121 | backup, a new freezetag will not be created. 122 | 123 | Backup freezetags follow a different naming scheme, and will be named F**yyyy**-**MM**-**dd**\_**hh**-**mm**-**ss**.ftag 124 | using the date of creation. 125 | 126 | ### `thaw` 127 | 128 | Restore files in-place in the current directory to the freezetag state, using whatever freezetag is in the current 129 | directory (prompting the user if multiple `.ftag`s exist): 130 | 131 | $> freezetag thaw 132 | 133 | Same as above, except `downloads/Pink Floyd - Dark Side of the Moon (1973 MSFL UDCD 517) - FLAC` is used instead of the 134 | current directory: 135 | 136 | $> freezetag thaw "downloads/Pink Floyd - Dark Side of the Moon (1973 MSFL UDCD 517) - FLAC" 137 | 138 | For parity with `freeze`, `thaw` supports an `--ftag` flag that searches the given directory for a freezetag instead: 139 | 140 | $> freezetag thaw --ftag ~/ftags 141 | 142 | Or, if a freezetag is explicitly passed with `--ftag`, that freezetag will be used to thaw. The following thaws files 143 | in-place in the current directory using the given freezetag: 144 | 145 | $> freezetag thaw --ftag ~/ftags/redacted-pink-floyd-the-wall.ftag 146 | 147 | By default, `thaw` will thaw files in-place, meaning files and directories will be moved/modified/created to match the 148 | freezetag state. If the `--to` flag is passed, no files will be modified, but will instead be copied and thawed to the 149 | given directory: 150 | 151 | $> freezetag thaw --to ~/out 152 | 153 | The thawed files will be written to a subdirectory named according to the `root` directory from the freezetag. The 154 | `root` is the name of the top directory when the files were frozen. So if we freeze a directory and thaw it with `--to`: 155 | 156 | $> cd "downloads/Pink Floyd - Dark Side of the Moon (1973 MSFL UDCD 517) - FLAC" 157 | $> freezetag freeze 158 | $> freezetag thaw --to ~/out 159 | 160 | This will restore the freezetag state to `~/out/Pink Floyd - Dark Side of the Moon (1973 MSFL UDCD 517) - FLAC`. 161 | 162 | As another example, say we regularly make backups of our music library state: 163 | 164 | $> freezetag freeze ~/music --backup --ftag ~/ftags 165 | 166 | We could then recover this backed-up state from a directory of downloaded music: 167 | 168 | $> freezetag thaw ~/downloads --to ~ --ftag ~/ftags 169 | 170 | This will restore our library from `~/downloads` to `~/music`, keeping `~/downloads` intact. 171 | 172 | ### `mount` 173 | 174 | Recursively mount music files and freezetags in `~/music` to `~/freezefs`: 175 | 176 | $> freezetag mount ~/music ~/freezefs 177 | 178 | Music files in `~/music` will appear under `~/freezefs` in their original frozen states (paths and tags) from 179 | corresponding freezetag files found under `~/music`. A file in `~/music` will *not* be mapped to`~/freezefs` if there is 180 | no freezetag matching that file. On the other hand, a file in `~/music` can be mapped to `~/freezefs` multiple times 181 | under different paths (and possibly with different tags) if there are multiple freezetags referring to that same file. 182 | 183 | `freezetag mount` enables renamed/retagged music to be seeded without requiring copies on disk. For example, let's say 184 | we download a torrent to `~/downloads/Pink Floyd - Dark Side of the Moon (1973 MSFL UDCD 517) - FLAC`. Assuming our 185 | personal tagged library is stored under `~/music` and mounted as shown above, we can then: 186 | 1. Run `freezetag freeze `~/downloads/Pink Floyd - Dark Side of the Moon (1973 MSFL UDCD 517) - FLAC` after the torrent 187 | finishes. 188 | 2. Retag/rename/move the files (e.g., to `~/music/Pink Floyd/Dark Side of the Moon`). Assuming we also move the 189 | freezetag from step 1 along with it, this will automatically make a new `Pink Floyd - Dark Side of the Moon (1973 190 | MSFL UDCD 517) - FLAC` directory appear under our mount point (`~/freezefs`). 191 | 3. Point our torrent client to seed the files from `~/freezefs/Pink Floyd - Dark Side of the Moon (1973 MSFL UDCD 517) - 192 | FLAC`. 193 | 194 | Ideally, the above steps would be automated by your torrent client/music importer. 195 | 196 | Mounts are read-only. Mounts are "live", meaning new files added to the source directory will automatically appear under 197 | the mount point (assuming there's a matching freezetag), and deleted files will automatically disappear. Similarly, 198 | changes in tags or freezetag files will be reflected automatically. 199 | 200 | Note: The initial mount may take awhile depending on how large your library is. Mount metadata is cached on disk, so 201 | subsequent mounts should activate in just a few seconds. 202 | 203 | ### `shave` 204 | 205 | Strips all metadata from music files in the current directory: 206 | 207 | $> freezetag shave 208 | 209 | Same as above, except `downloads/Pink Floyd - Dark Side of the Moon (1973 MSFL UDCD 517) - FLAC` is used instead of the 210 | current directory: 211 | 212 | $> freezetag shave "downloads/Pink Floyd - Dark Side of the Moon (1973 MSFL UDCD 517) - FLAC" 213 | 214 | `freezetag shave` can be useful if you want to share the "bare" music files. This has the advantage of smaller overall 215 | distribution size (especially if the music contains images), and it allows tags to be shared separately. That is, users 216 | can create and distribute different freezetag files using the same bare music files, and the bare music files will 217 | remain unchanged. 218 | 219 | ### `show` 220 | 221 | Shows the contents of a freezetag file. 222 | 223 | $> freezetag show 224 | version: 1 225 | mode: default 226 | id: Fc58e43e83c0487f5-56ddb165-e8a1171b 227 | root: Pink Floyd - Dark Side of the Moon (1973 MSFL UDCD 517) - FLAC 228 | 96f5c1b465b263d5090deae5d177df47f0f36efa 01 - Speak To Me.flac 229 | dbd1dc0c8ad514d9641919bc52c99010f5cf3ad2 02 - Breathe.flac 230 | 65cbc81d99b7ed68125b1652819e5fbec7e7a845 03 - On The Run.flac 231 | 40af1c90071abc0d66dbbfad5be39fac25967b5e 04 - Time.flac 232 | a9fe89c25630794252aa3fe6b1e8078a4eb2308d 05 - The Great Gig In The Sky.flac 233 | 1a3dfb4396cd1c0ccf7804fc66c0e512b65df65d 06 - Money.flac 234 | caeaf72fcee0df11c167bea2edffe4f7acbcec72 07 - Us And Them.flac 235 | 3ed0c16c900d8430f470011ba877fbaa15a5f7ef 08 - Any Colour You Like.flac 236 | fee17fdb47c04037d57094ad14260f20cf8e7a83 09 - Brain Damage.flac 237 | f3539291afa214e7af868ff9b0f044a372cf9e9b 10 - Eclipse.flac 238 | 25041f6026518a0e0a0aa4df3c5fdfd62f8001ea Pink Floyd - The Dark Side Of The Moon.log 239 | 76e4c8325809583bca447ba2d0e9fdaa4b39fa1e Pink Floyd - The Dark Side Of The Moon.m3u 240 | 15fdc9bc1bbe2646ba3cd076789c805f811f9b10 The Dark Side Of The Moon.cue 241 | 242 | You can also use the `--json` flag to get parse-friendly output: 243 | 244 | $> freezetag show --json 245 | { 246 | "version": 1, 247 | "mode": "default", 248 | "id": "Fc58e43e83c0487f5-56ddb165-e8a1171b", 249 | "root": "Pink Floyd - Dark Side of the Moon (1973 MSFL UDCD 517) - FLAC", 250 | "files": [ 251 | { 252 | "path": "01 - Speak To Me.flac", 253 | "checksum": "96f5c1b465b263d5090deae5d177df47f0f36efa" 254 | }, 255 | { 256 | "path": "02 - Breathe.flac", 257 | "checksum": "dbd1dc0c8ad514d9641919bc52c99010f5cf3ad2" 258 | }, 259 | { 260 | "path": "03 - On The Run.flac", 261 | "checksum": "65cbc81d99b7ed68125b1652819e5fbec7e7a845" 262 | }, 263 | ... 264 | ] 265 | } 266 | 267 | Freezetag ID 268 | ------------ 269 | 270 | By default, the freezetag file will be saved to the processed directory as F**a**-**b**-**c**.ftag, where: 271 | * **a** is a 16-character hex string that uniquely identifies the music 272 | * **b** is an 8-character hex string that uniquely identifies the metadata 273 | * **c** is an 8-character hex string that uniquely identifies the freezetag 274 | 275 | While this naming scheme might seem unusual, the uniqueness property lets you quickly see whether two freezetags are 276 | identical simply by comparing their file names: 277 | 278 | * The **a** and **b** segments do not change between freezes if any paths change or if any non-music files are modified. 279 | Additionally, the **a** segment doesn't change if the music files are retagged. 280 | 281 | * This means if two different .ftag files have the same **a** segment, they represent the same set of raw music. If they 282 | have both the same **a** and **b** segments, they represent the same set of raw music *and* their metadata. 283 | 284 | * If all three segments are the same, the two freezetag files are identical. Therefore, running `freezetag freeze` twice 285 | will only result in a single freezetag file being created: since the ID will stay the same, the existing freezetag file 286 | will be replaced on the second invocation. 287 | 288 | If you use a custom naming scheme for your freezetags (by passing `--ftag` to `freeze`), the ID can still be found using 289 | `freezetag show`. 290 | 291 | Changelog 292 | --------- 293 | ### [1.2.1] - 2021-04-21 294 | * Fixed `--json` flag for `show` 295 | * Fixed file monitoring for failed stat 296 | ### [1.2.0] - 2020-04-26 297 | * Added `mount` command 298 | * Refactored modules and API 299 | ### [1.1.1] - 2020-04-16 300 | * Fixed `thaw` where multiple copies of the same file are frozen 301 | ### [1.1.0] - 2020-04-16 302 | * Added `freeze --backup` for incremental backups 303 | * Added `freeze --ftag` for custom output path 304 | * Fixed `show --json` to properly dump JSON 305 | * In-place `thaw` now deletes processed files immediately to reduce temporary disk usage 306 | * Created safety checks and prompts for `thaw` 307 | * Added `thaw --skip-checks` to allow skipping checks 308 | ### [1.0.4] - 2020-04-10 309 | * Don't delete files on thaw if they aren't listed in ftag 310 | ### [1.0.3] - 2020-04-09 311 | * Allow `show` to accept .ftag directly 312 | ### [1.0.2] - 2020-04-09 313 | * Fixed directory argument parsing 314 | ### [1.0.1] - 2020-04-06 315 | * Added selection prompt for multiple freezetags 316 | ### [1.0.0] - 2020-04-06 317 | * Initial release 318 | 319 | [1.2.1]: https://github.com/x1ppy/freezetag/compare/1.2.0...1.2.1 320 | [1.2.0]: https://github.com/x1ppy/freezetag/compare/1.1.1...1.2.0 321 | [1.1.1]: https://github.com/x1ppy/freezetag/compare/1.1.0...1.1.1 322 | [1.1.0]: https://github.com/x1ppy/freezetag/compare/1.0.4...1.1.0 323 | [1.0.4]: https://github.com/x1ppy/freezetag/compare/1.0.3...1.0.4 324 | [1.0.3]: https://github.com/x1ppy/freezetag/compare/1.0.2...1.0.3 325 | [1.0.2]: https://github.com/x1ppy/freezetag/compare/1.0.1...1.0.2 326 | [1.0.1]: https://github.com/x1ppy/freezetag/compare/1.0.0...1.0.1 327 | [1.0.0]: https://github.com/x1ppy/freezetag/releases/tag/1.0.0 328 | -------------------------------------------------------------------------------- /freezetag/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/x1ppy/freezetag/89ac8a8563183d2079a95419d82f691719c5ef32/freezetag/__init__.py -------------------------------------------------------------------------------- /freezetag/__main__.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | import argparse 3 | import sys 4 | from pathlib import Path 5 | 6 | from . import commands 7 | 8 | 9 | def parse_args(): 10 | parent = argparse.ArgumentParser(add_help=False) 11 | main = argparse.ArgumentParser(parents=[parent], 12 | epilog='Use "freezetag [command] --help" for more information about a command.', 13 | description='Saves, strips, and restores file paths and music metadata.', 14 | formatter_class=argparse.RawTextHelpFormatter) 15 | sub = main.add_subparsers(metavar='command', dest='command') 16 | sub.required = True 17 | 18 | def add_subparser(command, help, description=''): 19 | return sub.add_parser(command, help=help, description=help + description, 20 | formatter_class=argparse.RawTextHelpFormatter, parents=[parent]) 21 | 22 | freeze = add_subparser('freeze', 23 | 'Save paths and music metadata to a freezetag file.', 24 | '\n\nMusic files with supported extensions (.mp3, .flac) will have their metadata' 25 | '\nsaved in the .ftag file. All files will have their paths saved in the .ftag' 26 | '\nfile. Metadata and path state can be restored using "freezetag thaw".' 27 | '\n\nThe freezetag file will be saved in `directory`. Unless --backup is used, the' 28 | '\nfreezetag file will be named Fa-b-c.ftag, where:' 29 | '\n a is a 16-character segment that uniquely identifies the music' 30 | '\n b is an 8-character segment that uniquely identifies the metadata' 31 | '\n c is an 8-character segment that uniquely identifies the freezetag' 32 | '\n\nThe "a" and "b" segments do not change between freezes if any paths change or' 33 | '\nif any non-music files are modified. Additionally, the "a" segment doesn\'t' 34 | '\nchange if the music files are retagged.' 35 | '\n\nThis means if two different .ftag files have the same "a" segment, they' 36 | '\nrepresent the same set of raw music. If they have both the same "a" and "b"' 37 | '\nsegments, they represent the same set of raw music AND their metadata.' 38 | ) 39 | 40 | thaw = add_subparser('thaw', 'Restore paths and music metadata from a freezetag file.', 41 | '\n\nBy default, the files will be renamed and restored in-place inside `directory`.' 42 | '\nNote that `directory` itself will be renamed to the "root" saved in the' 43 | '\nfreezetag file.' 44 | '\n\nIf --to is used, files will instead be copied and restored to a subdirectory' 45 | '\n(named "root" from the freezetag file) under the `to` directory, leaving the' 46 | '\nfiles in the source `directory` untouched.') 47 | 48 | mount = add_subparser('mount', 49 | help='Recursively mount a directory and its freezetags.') 50 | mount.add_argument('directory', 51 | help='Directory to mount.' 52 | '\n\nThis directory will be scanned for freezetags and matching files. Any matches' 53 | '\nwill then be mounted as their original state under `mount_point`.' 54 | '\n\nMounts are read-only. Mounts are "live", meaning new files added to the source' 55 | '\ndirectory will automatically appear under the mount point (assuming there\'s a' 56 | '\nmatching freezetag), and deleted files will automatically disappear. Similarly,' 57 | '\nchanges in tags or freezetag files will be reflected automatically.' 58 | '\n\nNote: The initial mount may take awhile depending on how large your library is.' 59 | '\nMount metadata is cached on disk, so subsequent mounts should activate in just' 60 | '\na few seconds.') 61 | mount.add_argument('--verbose', '-v', action='store_true', help='Verbose mode.') 62 | mount.add_argument('mount_point', help='Mount destination.') 63 | 64 | shave = add_subparser('shave', 'Strip metadata from all music files.', 65 | '\n\nOnly music files with supported extensions (.mp3, .flac) will be modified,' 66 | '\nand only supported metadata (Vorbis comments, ID3) will be stripped.') 67 | 68 | show = add_subparser('show', 'Display the contents of a freezetag file.') 69 | 70 | for parser in [freeze, thaw, shave]: 71 | parser.add_argument('directory', nargs='?', default=Path.cwd(), 72 | help='Directory to process (default: current directory).') 73 | 74 | show.add_argument('path', nargs='?', metavar='path', default=Path.cwd(), 75 | help='Directory containing .ftag file, or the .ftag file itself\n' 76 | '(default: current directory)') 77 | show.add_argument('--json', action='store_true', dest='as_json', help='Prints JSON output.') 78 | 79 | thaw.add_argument('--ftag', metavar='path', 80 | help='Path to Freezetag file to use.' 81 | '\n\nIf path is a directory, the .ftag in this directory will be' 82 | '\nused. Otherwise, path must be an .ftag file that will be used' 83 | '\nto thaw.' 84 | '\n\nIf --ftag is not specified, the .ftag in `directory` will be' 85 | '\nused.') 86 | thaw.add_argument('--to', metavar='directory', 87 | help='Directory to which thawed files will be copied and restored.' 88 | '\nIf omitted, files will be renamed and restored in-place.') 89 | thaw.add_argument('--skip-checks', action='store_true', 90 | help='Skips safety checks.' 91 | '\n\nBy default, thaw verifies that (1) all music files in the' 92 | '\nfreezetag are in `directory`, and (2) `directory` doesn\'t' 93 | '\ncontain unrecognized files when the common music path doesn\'t' 94 | '\nmatch `directory`. The user will be prompted if either of' 95 | '\nthese conditions is not met.' 96 | '\n\nThese checks help prevent unintentional file/directory changes' 97 | '\nif thaw is called with the wrong `directory`; however, the' 98 | '\nthaw will take longer and requires user interaction for prompts.' 99 | '\nPass --skip-checks to disable these checks.') 100 | 101 | freeze.add_argument('--backup', action='store_true', 102 | help='Freeze in incremental backup mode.' 103 | '\n\nThis mode is optimized for repeated incremental backups of the' 104 | '\nsame directory.' 105 | '\n\nIn backup mode, the freezetag file will be saved as' 106 | '\nFyyyy-MM-dd_hh-mm-ss.ftag. File sizes and last modified times' 107 | '\nwill be written to the freezetag file. On subsequent incremental' 108 | '\nbackups, the last created freezetag in this directory will be' 109 | '\nread. Any files whose names haven\'t changed will not have their' 110 | '\nhashes recalculated, making the freeze operation significantly' 111 | '\nfaster.') 112 | freeze.add_argument('--ftag', metavar='path', 113 | help='Path to output freezetag file.' 114 | '\n\nIf path is a directory, the freezetag file will be written to' 115 | '\nthis directory, following the naming specifications outlined' 116 | '\nabove. Otherwise, the freezetag file will be named and saved' 117 | '\naccording to the given path.' 118 | '\n\nIf --ftag is not specified, the freezetag file will be written' 119 | '\nto `directory`.') 120 | 121 | return main.parse_args() 122 | 123 | 124 | def main(): 125 | try: 126 | args = parse_args() 127 | getattr(commands, args.command)(**vars(args)) 128 | except commands.CommandException as e: 129 | print(e, file=sys.stderr) 130 | sys.exit(1) 131 | 132 | 133 | if __name__ == '__main__': 134 | main() 135 | -------------------------------------------------------------------------------- /freezetag/base.py: -------------------------------------------------------------------------------- 1 | import hashlib 2 | 3 | from construct import * 4 | 5 | from . import formats 6 | 7 | 8 | class ParsedFile: 9 | @staticmethod 10 | def from_path(path): 11 | suffix = path.suffix.lower() 12 | if suffix == '.flac': 13 | return formats.flac.ParsedFile(path) 14 | if suffix == '.mp3': 15 | return formats.mp3.ParsedFile(path) 16 | return formats.generic.ParsedFile(path) 17 | 18 | def __init__(self, path, format): 19 | self.path = path 20 | self.format = format 21 | self._instance = None 22 | 23 | @property 24 | def instance(self): 25 | if not self._instance: 26 | self._instance = self.parse() 27 | return self._instance 28 | 29 | def parse(self): 30 | raise NotImplementedError() 31 | 32 | def strip(self): 33 | raise NotImplementedError() 34 | 35 | def restore_metadata(self, metadata): 36 | raise NotImplementedError() 37 | 38 | def checksum(self): 39 | raise NotImplementedError() 40 | 41 | 42 | class MusicMetadata: 43 | @staticmethod 44 | def from_state(state): 45 | if state.format == 1: 46 | return formats.flac.MusicMetadata(state.metadata) 47 | if state.format == 2: 48 | return formats.mp3.MusicMetadata(state.metadata) 49 | return None 50 | 51 | def __init__(self, format, value): 52 | self.format = format 53 | self.value = value 54 | self.size = sum(m[1] for m in self) 55 | 56 | def checksum(self): 57 | return hashlib.sha1(self.format.build(self.value)).digest() 58 | 59 | def __iter__(self): 60 | raise NotImplementedError() 61 | 62 | 63 | class MusicParsedFile(ParsedFile): 64 | def parse(self): 65 | instance = self.format.parse_file(self.path) 66 | 67 | # Workaround for https://github.com/construct/construct/issues/852 68 | instance._io.close() 69 | 70 | return instance 71 | 72 | def checksum(self): 73 | b = self.format.build(self.instance) 74 | return hashlib.sha1(b).digest() 75 | 76 | def write(self, path): 77 | self.format.build_file(self.instance, path) 78 | 79 | 80 | class FuseFile: 81 | fh = 0 82 | 83 | @staticmethod 84 | def from_info(file_path, *args): 85 | suffix = file_path.suffix.lower() 86 | if suffix == '.flac': 87 | return formats.flac.FuseFile(file_path, *args) 88 | if suffix == '.mp3': 89 | return formats.mp3.FuseFile(file_path, *args) 90 | return formats.generic.FuseFile(file_path, *args) 91 | 92 | def __init__(self, file_path, flags, metadata, file_metadata_info, file_metadata_len, frozen_metadata_len): 93 | self.file = file_path.open('rb') 94 | self.metadata = metadata 95 | self.fh = FuseFile.fh 96 | FuseFile.fh += 1 97 | 98 | def read(self, length, offset): 99 | raise NotImplementedError() 100 | 101 | def close(self): 102 | self.file.close() 103 | -------------------------------------------------------------------------------- /freezetag/commands.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | import hashlib 3 | import json 4 | import os 5 | import re 6 | import shutil 7 | import sys 8 | from datetime import datetime 9 | from pathlib import Path 10 | 11 | from .base import ParsedFile, MusicMetadata 12 | from .core import Freezetag 13 | 14 | # Version 2 is used only for "freeze --backup" freezetags. 15 | # All other freezetags are still created using version 1 so the bytes/IDs stay consistent. 16 | # These can be unified in a future version if the schema is updated. 17 | DEFAULT_VERSION = 1 18 | VERSION = 2 19 | 20 | DEFAULT_MODE = 0 21 | BACKUP_MODE = 1 22 | 23 | freeze_modes = ['default', 'backup'] 24 | 25 | 26 | class CommandException(Exception): 27 | pass 28 | 29 | 30 | def get_version(is_backup): 31 | return VERSION if is_backup else DEFAULT_VERSION 32 | 33 | 34 | def get_mode(is_backup): 35 | return BACKUP_MODE if is_backup else DEFAULT_MODE 36 | 37 | 38 | class Reprinter(): 39 | def __init__(self): 40 | self.last_width = 0 41 | 42 | def print(self, text): 43 | text = str(text) 44 | print(text.ljust(self.last_width, ' '), end='\r') 45 | self.last_width = len(text) 46 | 47 | 48 | def hash_bytes(b): 49 | return hashlib.sha1(b).digest() 50 | 51 | 52 | def find_ftag(path): 53 | if not path.exists(): 54 | raise CommandException(f'Given ftag is not a file or directory: {path}') 55 | 56 | if path.is_file(): 57 | return path 58 | 59 | freezetag_paths = [p for p in path.iterdir() if p.suffix.lower() == '.ftag'] 60 | 61 | if not len(freezetag_paths): 62 | raise CommandException(f'No freezetag file found in {path}') 63 | 64 | index = 0 65 | 66 | if len(freezetag_paths) > 1: 67 | print(f'Multiple freezetags found in directory: {path.resolve()}') 68 | for i, path in enumerate(freezetag_paths): 69 | print(f'{i}: {path.name}') 70 | choice = '' 71 | while not choice.isdecimal() or int(choice) < 0 or int(choice) >= len(freezetag_paths): 72 | choice = input(f'Select freezetag [0-{len(freezetag_paths) - 1}], or q to quit: ') 73 | if choice.lower() == 'q': 74 | sys.exit(0) 75 | index = int(choice) 76 | 77 | return freezetag_paths[index] 78 | 79 | 80 | def walk_dir(path): 81 | for dirpath, dirnames, filenames in os.walk(path): 82 | dirnames.sort() 83 | 84 | for filename in sorted(filenames): 85 | if filename.lower().endswith('.ftag'): 86 | continue 87 | 88 | p = Path(dirpath) / filename 89 | yield p, p.relative_to(path) 90 | 91 | 92 | def shave(directory, **kwargs): 93 | root = Path(directory).resolve() 94 | if not root.exists(): 95 | raise CommandException(f'Directory does not exist: {root}') 96 | 97 | for path, rel_path in walk_dir(root): 98 | file = ParsedFile.from_path(path) 99 | 100 | if not file.format: 101 | continue 102 | 103 | print(path.name) 104 | 105 | metadata = file.strip() 106 | 107 | if not next(iter(metadata), None): 108 | print(' no metadata found') 109 | continue 110 | 111 | print(' shaved {0}'.format(', '.join(f'{label} ({size})' for label, size in metadata))) 112 | 113 | file.write(path) 114 | 115 | 116 | def prepare_thaw(root, frozen, thaw_in_place, checksum_to_item): 117 | paths = {} 118 | commonpath = None 119 | unrecognized_found = False 120 | reprinter = Reprinter() 121 | 122 | for path, rel_path in walk_dir(root): 123 | reprinter.print(f'Checking...{rel_path}') 124 | 125 | file = ParsedFile.from_path(path) 126 | file.strip() 127 | checksum = file.checksum() 128 | 129 | if checksum not in checksum_to_item: 130 | unrecognized_found = True 131 | reprinter.print(f' Unrecognized file: {path}') 132 | print() 133 | continue 134 | 135 | checksum_to_item[checksum][1] = True 136 | paths[rel_path] = checksum_to_item[checksum] 137 | commonpath = os.path.commonpath(list(filter(None, [commonpath, path]))) 138 | 139 | reprinter.print('Checking...done.') 140 | print() 141 | 142 | if thaw_in_place and unrecognized_found and Path(commonpath) != root: 143 | print(f'\nCommon path ({commonpath}) does not match thaw directory ({root}).') 144 | print(f"You're thawing in-place, so the structure of {root} will be changed.") 145 | print('This directory may be renamed, and unrecognized files will be left in their paths' 146 | ' relative to this directory.') 147 | print(f'Make sure that you didn\'t intend to thaw {commonpath} instead.') 148 | while True: 149 | choice = input('Continue anyway? (y/n): ').lower() 150 | if choice == 'y': 151 | break 152 | if choice == 'n': 153 | sys.exit(0) 154 | 155 | missing_music = False 156 | for file in frozen.files: 157 | if not checksum_to_item[file.checksum][1]: 158 | print(f' Missing: {file.path} ({file.checksum.hex()})', file=sys.stderr) 159 | if file.format: 160 | missing_music = True 161 | 162 | if missing_music: 163 | print('One or more music files listed in freezetag are missing.') 164 | while True: 165 | choice = input('Continue anyway? (y/n): ').lower() 166 | if choice == 'y': 167 | break 168 | if choice == 'n': 169 | sys.exit(0) 170 | 171 | return paths 172 | 173 | 174 | def thaw(directory, to, ftag, skip_checks, **kwargs): 175 | root = Path(directory).resolve() 176 | if not root.exists(): 177 | raise CommandException(f'Directory does not exist: {root}') 178 | 179 | ftag = find_ftag(Path(ftag or root)) 180 | freezetag = Freezetag.from_path(ftag) 181 | 182 | if freezetag.data.version > VERSION: 183 | raise CommandException('Freezetag file version greater than freezetag version ({0} > {1}).\n' 184 | 'Update freezetag and try again.'.format(freezetag.data.version, VERSION)) 185 | 186 | frozen = freezetag.data.frozen 187 | to_dir = Path(to).resolve() / frozen.root if to else root 188 | tmp_dir = root / ftag.with_suffix('.ftag-tmp').name 189 | thaw_in_place = root == to_dir 190 | 191 | # Map each checksum to an array of files in case multiple files with identical checksums were frozen. 192 | checksum_to_item = {} 193 | for f in frozen.files: 194 | if f.checksum not in checksum_to_item: 195 | checksum_to_item[f.checksum] = [[], False, False] 196 | checksum_to_item[f.checksum][0].append(f) 197 | 198 | print(f'Processing {root}...') 199 | 200 | # First pass: verify directory and calculate checksums. 201 | path_to_item = None 202 | if not skip_checks: 203 | path_to_item = prepare_thaw(root, frozen, thaw_in_place, checksum_to_item) 204 | 205 | reprinter = Reprinter() 206 | import_fn = shutil.move if thaw_in_place else shutil.copy2 207 | 208 | # Second pass: move (or copy) files to tmp_dir and update their metadata. 209 | for path, rel_path in walk_dir(root): 210 | file = None 211 | 212 | if path_to_item: 213 | if rel_path not in path_to_item: 214 | continue 215 | item = path_to_item[rel_path] 216 | else: 217 | file = ParsedFile.from_path(path) 218 | file.strip() 219 | checksum = file.checksum() 220 | if checksum not in checksum_to_item: 221 | continue 222 | item = checksum_to_item[checksum] 223 | 224 | reprinter.print(f'Thawing metadata...{rel_path}') 225 | 226 | if item[2]: 227 | continue 228 | 229 | item[2] = True 230 | 231 | for state in item[0]: 232 | to_path = tmp_dir / state.path 233 | to_path.parent.mkdir(parents=True, exist_ok=True) 234 | 235 | if not state.format: 236 | try: 237 | if len(item[0]) == 1: 238 | import_fn(path, to_path) 239 | else: 240 | shutil.copy2(path, to_path) 241 | except shutil.SameFileError: 242 | pass 243 | continue 244 | 245 | if not file: 246 | file = ParsedFile.from_path(path) 247 | 248 | file.restore_metadata(state.metadata) 249 | file.write(to_path) 250 | 251 | if thaw_in_place and rel_path.parts[0] != tmp_dir.name and path.exists(): 252 | path.unlink() 253 | parent = path.parent 254 | while not len(os.listdir(parent)): 255 | parent.rmdir() 256 | parent = parent.parent 257 | 258 | reprinter.print('Thawing metadata...done.') 259 | print() 260 | 261 | # Third pass: move files from tmp_dir to their final destinations. 262 | for path, rel_path in walk_dir(tmp_dir): 263 | reprinter.print(f'Restoring files...{rel_path}') 264 | to_path = to_dir / rel_path 265 | to_path.parent.mkdir(parents=True, exist_ok=True) 266 | path.rename(to_path) 267 | 268 | reprinter.print('Restoring files...done.') 269 | print() 270 | 271 | shutil.rmtree(tmp_dir) 272 | 273 | if thaw_in_place: 274 | new_root = root.parent / frozen.root 275 | if root != new_root: 276 | print(f'Renaming {root} to {new_root}') 277 | root.rename(new_root) 278 | 279 | 280 | def freeze(directory, backup, ftag, **kwargs): 281 | root = Path(directory).resolve() 282 | if not root.exists(): 283 | raise CommandException(f'Directory does not exist: {root}') 284 | 285 | tmp_paths = [p for p in root.iterdir() if p.suffix.lower() == '.ftag-tmp'] 286 | 287 | # We're trying to create a new freeze state, but we found existing freezetag tmp 288 | # directories. We almost certainly don't want tmp directories to be frozen, so 289 | # abort and have the user fix it first. 290 | if len(tmp_paths): 291 | raise CommandException( 292 | f'Unrestored freezetag data found at {tmp_paths[0]}.\n' 293 | 'Run freezetag thaw again to finish processing.') 294 | 295 | to_path = Path(ftag or root) 296 | existing = {} 297 | files = [] 298 | music_checksums = [] 299 | metadata_checksums = [] 300 | last_ftag = (None, 0) 301 | reprinter = Reprinter() 302 | 303 | reprinter.print('Collecting metadata...') 304 | 305 | if backup: 306 | to_dir = to_path if to_path.is_dir() else to_path.parent 307 | for f in to_dir.iterdir(): 308 | if not re.match('F\d{4}-\d{2}-\d{2}_\d{2}-\d{2}-\d{2}\.ftag', f.name): 309 | continue 310 | mtime = os.stat(f).st_mtime 311 | if mtime > last_ftag[1]: 312 | last_ftag = (f, mtime) 313 | if last_ftag[0]: 314 | last_frozen = Freezetag.from_path(last_ftag[0]).data.frozen 315 | for f in last_frozen.files: 316 | existing[f.path] = f 317 | 318 | existing_path_count = 0 319 | 320 | for path, rel_path in walk_dir(root): 321 | checksum = None 322 | 323 | if str(rel_path) in existing: 324 | stat = os.stat(path) 325 | state = existing[str(rel_path)] 326 | if stat.st_size == state.stat.size and abs(stat.st_mtime - state.stat.mtime) < 1e-3: 327 | files.append(f) 328 | existing_path_count += 1 329 | metadata = MusicMetadata.from_state(state) 330 | checksum = state.checksum 331 | 332 | file = ParsedFile.from_path(path) 333 | 334 | if not checksum: 335 | metadata = file.strip() 336 | checksum = file.checksum() 337 | 338 | dict = { 339 | 'path': rel_path.as_posix(), 340 | 'format': file.format_id, 341 | 'checksum': checksum, 342 | 'metadata': metadata.value if metadata else None, 343 | } 344 | 345 | if backup: 346 | stat = os.stat(path) 347 | dict['stat'] = { 348 | 'mtime': stat.st_mtime, 349 | 'size': stat.st_size, 350 | } 351 | else: 352 | dict['stat'] = None 353 | 354 | files.append(dict) 355 | 356 | if file.format: 357 | music_checksums.append(checksum) 358 | metadata_checksum = metadata.checksum() 359 | metadata_checksums.append(metadata_checksum) 360 | 361 | reprinter.print(f'Collecting metadata...{rel_path}') 362 | 363 | reprinter.print('Collecting metadata...done.') 364 | print() 365 | 366 | if not len(music_checksums): 367 | raise CommandException('No music files found.') 368 | 369 | if existing_path_count == len(existing) and existing_path_count == len(files) \ 370 | and last_frozen.root == root.name: 371 | print(f'No changes since last freezetag ({last_ftag[0].name}).') 372 | return 373 | 374 | print('Building freezetag...') 375 | 376 | freezetag = Freezetag({ 377 | 'signature': b'freezetag', 378 | 'version': get_version(backup), 379 | 'frozen': { 380 | 'mode': get_mode(backup), 381 | 'music_checksum': hash_bytes(b''.join(sorted(music_checksums)))[0:8], 382 | 'metadata_checksum': hash_bytes(b''.join(sorted(metadata_checksums)))[0:4], 383 | 'root': root.name, 384 | 'files': files, 385 | }, 386 | }) 387 | 388 | if to_path.is_dir(): 389 | filename = 'F' + datetime.now().strftime('%Y-%m-%d_%H-%M-%S') if backup else freezetag.get_id() 390 | to_path = to_path / f'{filename}.ftag' 391 | 392 | freezetag.write(to_path) 393 | 394 | print(f'Freezetag created at {to_path}') 395 | 396 | 397 | def show(path, as_json, **kwargs): 398 | ftag = find_ftag(Path(path)) 399 | 400 | freezetag = Freezetag.from_path(ftag) 401 | version = freezetag.data.version 402 | 403 | if version > VERSION: 404 | raise CommandException('Freezetag file version greater than freezetag version ({0} > {1}).\n' 405 | 'Update freezetag and try again.'.format(version, VERSION)) 406 | 407 | frozen = freezetag.data.frozen 408 | 409 | if as_json: 410 | print(json.dumps({ 411 | 'version': version, 412 | 'mode': freeze_modes[frozen.mode], 413 | 'id': freezetag.get_id(), 414 | 'root': frozen.root, 415 | 'files': [{ 416 | 'path': f.path, 417 | 'checksum': f.checksum.hex(), 418 | } for f in frozen.files], 419 | }, indent=2)) 420 | else: 421 | print(f'version: {version}') 422 | print(f'mode: {freeze_modes[frozen.mode]}') 423 | print(f'id: {freezetag.get_id()}') 424 | print(f'root: {frozen.root}') 425 | for f in frozen.files: 426 | print(f'{f.checksum.hex()} {f.path}') 427 | 428 | 429 | def mount(directory, mount_point, verbose, **kwargs): 430 | from .freezefs import FreezeFS 431 | FreezeFS(verbose).mount(directory, mount_point) 432 | -------------------------------------------------------------------------------- /freezetag/core.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | import hashlib 3 | 4 | from construct import * 5 | 6 | from .formats import generic, flac, mp3 7 | 8 | # Used for FrozenFormat > files > format. 9 | generic.ParsedFile.format_id = 0 10 | flac.ParsedFile.format_id = 1 11 | mp3.ParsedFile.format_id = 2 12 | 13 | # Used for freezetag mount cache. 14 | DB_VERSION = 1 15 | 16 | FrozenFormatV1 = Struct( 17 | 'mode' / Computed(0), 18 | 'music_checksum' / Bytes(8), 19 | 'metadata_checksum' / Bytes(4), 20 | 'root' / CString('utf8'), 21 | 'files' / Compressed(PrefixedArray(Int16ub, Struct( 22 | 'path' / CString('utf8'), 23 | 'format' / Int8ub, 24 | 'checksum' / Bytes(20), 25 | 'metadata' / Switch(this.format, { 26 | 1: flac.FrozenMetadataFormat, 27 | 2: mp3.FrozenMetadataFormat, 28 | }), 29 | )), 'lzma'), 30 | ) 31 | 32 | FrozenFormatV2 = Struct( 33 | 'mode' / Int8ub, 34 | 'music_checksum' / Bytes(8), 35 | 'metadata_checksum' / Bytes(4), 36 | 'root' / CString('utf8'), 37 | 'files' / Compressed(PrefixedArray(Int16ub, Struct( 38 | 'path' / CString('utf8'), 39 | 'format' / Int8ub, 40 | 'checksum' / Bytes(20), 41 | 'stat' / If(this._._.mode == 1, Struct( 42 | 'mtime' / Double, 43 | 'size' / Long, 44 | )), 45 | 'metadata' / Switch(this.format, { 46 | 1: flac.FrozenMetadataFormat, 47 | 2: mp3.FrozenMetadataFormat, 48 | }), 49 | )), 'lzma'), 50 | ) 51 | 52 | FreezeFormat = Struct( 53 | 'signature' / Const(b'freezetag'), 54 | 'version' / Int8ub, 55 | 'frozen' / Switch(this.version, { 56 | 1: FrozenFormatV1, 57 | 2: FrozenFormatV2, 58 | }, GreedyBytes), 59 | Terminated, 60 | ) 61 | 62 | DBItemFormat = Struct( 63 | 'device' / Int32ub, 64 | 'inode' / Int64ub, 65 | 'mtime' / Double, 66 | 'checksum' / Bytes(20), 67 | 'metadata_len' / Int32ub, 68 | 'metadata_info' / PrefixedArray(Int8ub, Struct( 69 | 'type' / CString('ascii'), 70 | 'size' / Int32ub, 71 | )), 72 | ) 73 | 74 | DBFormat = Struct( 75 | 'version' / Const(DB_VERSION, Int8ub), 76 | 'entries' / GreedyRange(DBItemFormat), 77 | ) 78 | 79 | 80 | class ChecksumDBAdapter(Adapter): 81 | def _decode(self, obj, context, path): 82 | dict = {} 83 | for item in obj.entries: 84 | if item.device not in dict: 85 | dict[item.device] = {} 86 | metadata_info = [(m.type, m.size) for m in item.metadata_info] 87 | dict[item.device][item.inode] = (item.checksum, metadata_info, item.metadata_len, item.mtime) 88 | return dict 89 | 90 | def _encode(self, obj, context, path): 91 | entries = [] 92 | for device, inodes in obj.items(): 93 | for inode, (checksum, metadata_info, metadata_len, mtime) in inodes.items(): 94 | entries.append({ 95 | 'device': device, 96 | 'inode': inode, 97 | 'mtime': mtime, 98 | 'checksum': checksum, 99 | 'metadata_len': metadata_len, 100 | 'metadata_info': [{'type': m[0], 'size': m[1]} for m in metadata_info], 101 | }) 102 | 103 | return {'version': DB_VERSION, 'entries': entries} 104 | 105 | 106 | class ChecksumDB: 107 | def __init__(self, path): 108 | self.path = path 109 | self._flush_counter = 0 110 | self._format = ChecksumDBAdapter(DBFormat) 111 | try: 112 | self._db = self._format.parse_file(str(path)) 113 | print(f'using existing database {path}') 114 | except: 115 | print(f'creating database at {path}') 116 | self._db = {} 117 | 118 | def get(self, device, inode, mtime): 119 | dict = self._db.get(device) 120 | if not dict: 121 | return 122 | item = dict.get(inode) 123 | if not item or item[3] != mtime: 124 | return 125 | return item 126 | 127 | def add(self, device, inode, mtime, checksum, metadata_info, metadata_len): 128 | if device not in self._db: 129 | self._db[device] = {} 130 | self._db[device][inode] = (checksum, metadata_info, metadata_len, mtime) 131 | self._try_flush() 132 | 133 | def _try_flush(self): 134 | self._flush_counter += 1 135 | if self._flush_counter < 50: 136 | return 137 | self.flush() 138 | 139 | def flush(self): 140 | self._flush_counter = 0 141 | self._format.build_file(self._db, str(self.path)) 142 | 143 | 144 | class Freezetag: 145 | @staticmethod 146 | def from_bytes(b): 147 | data = FreezeFormat.parse(b) 148 | # Workaround for https://github.com/construct/construct/issues/852 149 | data._io.close() 150 | freezetag = Freezetag(data) 151 | freezetag._bytes = b 152 | return freezetag 153 | 154 | @staticmethod 155 | def from_path(path): 156 | with path.open('rb') as f: 157 | return Freezetag.from_bytes(f.read()) 158 | 159 | def __init__(self, data): 160 | self.data = data 161 | self._bytes = None 162 | 163 | # Call this manually if data is changed. 164 | def data_updated(self): 165 | self._bytes = None 166 | 167 | def bytes(self): 168 | if not self._bytes: 169 | self._bytes = FreezeFormat.build(self.data) 170 | return self._bytes 171 | 172 | def get_id(self): 173 | checksum = hashlib.sha1(self.bytes()).digest()[0:4] 174 | return 'F' + '-'.join([self.data['frozen']['music_checksum'].hex(), 175 | self.data['frozen']['metadata_checksum'].hex(), 176 | checksum.hex()]) 177 | 178 | def write(self, path): 179 | with path.open('wb') as f: 180 | f.write(self.bytes()) 181 | -------------------------------------------------------------------------------- /freezetag/formats/flac.py: -------------------------------------------------------------------------------- 1 | from freezetag import base 2 | from construct import * 3 | 4 | 5 | BLOCK_TYPES = [ 6 | 'STREAMINFO', 7 | 'PADDING', 8 | 'APPLICATION', 9 | 'SEEKTABLE', 10 | 'VORBIS_COMMENT', 11 | 'CUESHEET', 12 | 'PICTURE', 13 | ] 14 | 15 | MetadataFormat = Struct( 16 | 'info' / BitStruct( 17 | 'last' / Flag, 18 | 'block_type' / BitsInteger(7), 19 | ), 20 | 'size' / Int24ub, 21 | 'data' / Bytes(this.size), 22 | ) 23 | 24 | FrozenMetadataFormat = PrefixedArray(Int8ub, MetadataFormat) 25 | 26 | Format = Struct( 27 | 'signature' / Const(b'fLaC'), 28 | 'metadata' / RepeatUntil(lambda metadata, *_: metadata.info.last, MetadataFormat), 29 | 'audio' / GreedyBytes, 30 | ) 31 | 32 | 33 | class MusicMetadata(base.MusicMetadata): 34 | def __init__(self, value): 35 | super().__init__(FrozenMetadataFormat, value) 36 | 37 | def __iter__(self): 38 | for item in self.value: 39 | yield BLOCK_TYPES[item.info.block_type], item.size + 4 40 | 41 | 42 | class ParsedFile(base.MusicParsedFile): 43 | def __init__(self, path): 44 | super().__init__(path, Format) 45 | 46 | def strip(self): 47 | streaminfo = self.instance.metadata[0] 48 | streaminfo.info.last = True 49 | metadata = self.instance.metadata[1:] 50 | self.instance.metadata = [streaminfo] 51 | return MusicMetadata(metadata) 52 | 53 | def restore_metadata(self, metadata): 54 | self.strip() 55 | 56 | # Append the frozen metadata to the stripped media. 57 | self.instance.metadata[0].info.last = not len(metadata) 58 | self.instance.metadata += metadata 59 | 60 | 61 | class FuseFile(base.FuseFile): 62 | def __init__(self, file_path, flags, metadata, file_metadata_info, file_metadata_len, frozen_metadata_len): 63 | super().__init__(file_path, flags, metadata, file_metadata_info, file_metadata_len, frozen_metadata_len) 64 | self.file_metadata_len = file_metadata_len 65 | self.frozen_metadata_len = frozen_metadata_len 66 | self._metadata_bytes = None 67 | 68 | def metadata_bytes(self): 69 | if self._metadata_bytes == None: 70 | self._metadata_bytes = b''.join(MetadataFormat.build(m) for m in self.metadata) 71 | return self._metadata_bytes 72 | 73 | def read(self, length, offset): 74 | f = self.file 75 | buf = b'' 76 | 77 | if offset < 42: 78 | f.seek(0, 0) 79 | head = f.read(42) 80 | if head[4] >= 128: 81 | head = head[0:4] + bytes([head[4] - 128]) + head[5:] 82 | end = min(42, offset + length) 83 | buf = head[offset:end] 84 | length -= len(buf) 85 | if not length: 86 | return buf 87 | offset = 42 88 | offset -= 42 89 | 90 | metadata_len = self.frozen_metadata_len 91 | if offset < metadata_len: 92 | metadata_bytes = self.metadata_bytes() 93 | end = min(metadata_len, offset + length) 94 | tmp = metadata_bytes[offset:end] 95 | length -= len(tmp) 96 | buf += tmp 97 | if not length: 98 | return buf 99 | offset = metadata_len 100 | offset -= metadata_len 101 | 102 | offset += 42 + self.file_metadata_len 103 | f.seek(offset, 0) 104 | return buf + f.read(length) 105 | -------------------------------------------------------------------------------- /freezetag/formats/generic.py: -------------------------------------------------------------------------------- 1 | import hashlib 2 | from construct import * 3 | from freezetag import base 4 | 5 | 6 | class ParsedFile(base.ParsedFile): 7 | def __init__(self, path): 8 | super().__init__(path, None) 9 | 10 | def parse(self): 11 | with self.path.open('rb') as f: 12 | return f.read() 13 | 14 | def strip(self): 15 | return None 16 | 17 | def restore_metadata(self, metadata): 18 | pass 19 | 20 | def checksum(self): 21 | return hashlib.sha1(self.instance).digest() 22 | 23 | 24 | class FuseFile(base.FuseFile): 25 | def read(self, length, offset): 26 | self.file.seek(offset, 0) 27 | return self.file.read(length) 28 | -------------------------------------------------------------------------------- /freezetag/formats/mp3.py: -------------------------------------------------------------------------------- 1 | from freezetag import base 2 | from construct import * 3 | 4 | 5 | class Id3SyncSafeIntAdapter(Adapter): 6 | def _decode(self, obj, context, path): 7 | return obj[0]*2097152 + obj[1]*16384 + obj[2]*128 + obj[3] 8 | 9 | def _encode(self, obj, context, path): 10 | return list(obj >> 7 * i & 0x7f for i in reversed(range(4))) 11 | 12 | 13 | Id3SyncSafeInt = Id3SyncSafeIntAdapter(Int8ub[4]) 14 | 15 | Id3v1Format = Struct( 16 | 'signature' / Const(b'TAG'), 17 | 'data' / Bytes(125), 18 | ) 19 | 20 | Id3v2HeaderFormat = Struct( 21 | 'signature' / OneOf(Bytes(3), [b'ID3', b'3DI']), 22 | 'version_major' / Int8ub, 23 | 'version_rev' / Int8ub, 24 | 'flags' / BitStruct( 25 | 'unsynchronization' / Flag, 26 | 'extended' / Flag, 27 | 'experimental' / Flag, 28 | 'footer' / Flag, 29 | 'unused' / BitsInteger(4)), 30 | 'size' / Id3SyncSafeInt, 31 | ) 32 | 33 | Id3v2Format = Struct( 34 | 'header' / Id3v2HeaderFormat, 35 | 'extended_header' / If(this.header.flags.extended, Struct( 36 | 'size' / IfThenElse(this._.header.version_major == 3, Int, Id3SyncSafeInt), 37 | 'data' / Bytes(this.header.size))), 38 | 'body' / FixedSized(this.header.size, Struct( 39 | 'frames' / RepeatUntil((this.next == 0 or this.next == None), Struct( 40 | 'id' / PaddedString(4, 'ascii'), 41 | 'size' / IfThenElse(this._._.header.version_major == 3, Int, Id3SyncSafeInt), 42 | 'flags' / Int16ub, 43 | 'data' / Bytes(this.size), 44 | 'next' / Peek(Int8ub), 45 | )), 46 | 'padding' / GreedyBytes)), 47 | 'footer' / If(this.header.flags.footer, Id3v2HeaderFormat), 48 | 'size' / Computed(lambda this: this.header.size + 10 + (10 if this.footer else 0)), 49 | ) 50 | 51 | class FormatAdapter(Adapter): 52 | def _decode(self, obj, context, path): 53 | return Struct( 54 | 'id3v2_head' / Optional(Id3v2Format), 55 | 'offset' / Tell, 56 | 'try_id3v1' / Pointer(-128, Optional(RawCopy(Id3v1Format))), 57 | 'try_id3v2' / Pointer(-138 if this.try_id3v1 else -10, Optional(RawCopy(Id3v2HeaderFormat))), 58 | 'audio' / IfThenElse(this.try_id3v2, Bytes(this.try_id3v2.offset1 - this.try_id3v2.size - 10 - this.offset), 59 | IfThenElse(this.try_id3v1, Bytes(this.try_id3v1.offset1 - this.offset), GreedyBytes)), 60 | 'id3v2_tail' / If(this.try_id3v2, Id3v2Format), 61 | 'id3v1' / If(this.try_id3v1, Id3v1Format), 62 | Terminated, 63 | ).parse(obj) 64 | 65 | def _encode(self, obj, context, path): 66 | return Struct( 67 | 'id3v2_head' / Optional(Id3v2Format), 68 | 'audio' / GreedyBytes, 69 | 'id3v2_tail' / Optional(Id3v2Format), 70 | 'id3v1' / Optional(Id3v1Format), 71 | Terminated, 72 | ).build(obj) 73 | 74 | Format = FormatAdapter(GreedyBytes) 75 | 76 | FrozenMetadataFormat = Struct( 77 | 'flags' / BitStruct( 78 | 'has_id3v2_head' / Flag, 79 | 'has_id3v2_tail' / Flag, 80 | 'has_id3v1' / Flag, 81 | Padding(5), 82 | ), 83 | 'id3v2_head' / If(this.flags.has_id3v2_head, Id3v2Format), 84 | 'id3v2_tail' / If(this.flags.has_id3v2_tail, Id3v2Format), 85 | 'id3v1' / If(this.flags.has_id3v1, Id3v1Format), 86 | ) 87 | 88 | 89 | class MusicMetadata(base.MusicMetadata): 90 | def __init__(self, value): 91 | super().__init__(FrozenMetadataFormat, value) 92 | 93 | def __iter__(self): 94 | if self.value['id3v2_head']: 95 | header = self.value['id3v2_head'].header 96 | yield 'head-ID3v2.{0}'.format(header.version_major), self.value['id3v2_head'].size 97 | if self.value['id3v2_tail']: 98 | header = self.value['id3v2_tail'].header 99 | yield 'tail-ID3v2.{0}'.format(header.version_major), self.value['id3v2_tail'].size 100 | if self.value['flags']['has_id3v1']: 101 | yield 'ID3v1', 128 102 | 103 | 104 | class ParsedFile(base.MusicParsedFile): 105 | def __init__(self, path): 106 | super().__init__(path, Format) 107 | 108 | def strip(self): 109 | metadata = { 110 | 'flags': { 111 | 'has_id3v2_head': self.instance.id3v2_head != None, 112 | 'has_id3v2_tail': self.instance.id3v2_tail != None, 113 | 'has_id3v1': self.instance.id3v1 != None, 114 | }, 115 | 'id3v2_head': self.instance.id3v2_head, 116 | 'id3v2_tail': self.instance.id3v2_tail, 117 | 'id3v1': self.instance.id3v1, 118 | } 119 | self.instance.id3v2_head = None 120 | self.instance.id3v2_tail = None 121 | self.instance.id3v1 = None 122 | return MusicMetadata(metadata) 123 | 124 | def restore_metadata(self, metadata): 125 | self.instance.id3v2_head = None 126 | self.instance.id3v2_tail = None 127 | self.instance.id3v1 = None 128 | self.instance.id3v2_head = metadata.id3v2_head 129 | self.instance.id3v2_tail = metadata.id3v2_tail 130 | self.instance.id3v1 = metadata.id3v1 131 | 132 | 133 | class FuseFile(base.FuseFile): 134 | def __init__(self, file_path, flags, metadata, file_metadata_info, file_metadata_len, frozen_metadata_len): 135 | super().__init__(file_path, flags, metadata, file_metadata_info, file_metadata_len, frozen_metadata_len) 136 | self.audio_len = file_path.stat().st_size - frozen_metadata_len 137 | self._id3v2_head_bytes = None 138 | self._id3v2_tail_bytes = None 139 | self._id3v1_bytes = None 140 | 141 | self.id3v2_head_len = metadata['id3v2_head'].size if metadata and metadata['id3v2_head'] else 0 142 | self.id3v2_tail_len = metadata['id3v2_tail'].size if metadata and metadata['id3v2_tail'] else 0 143 | self.id3v1_len = 128 if metadata and metadata['id3v1'] else 0 144 | 145 | self.audio_offset = 0 146 | for type, size in file_metadata_info: 147 | if type.startswith('head-ID3v2'): 148 | self.audio_offset = size 149 | break 150 | 151 | def id3v2_head_bytes(self): 152 | if self.metadata['id3v2_head'] and self._id3v2_head_bytes == None: 153 | self._id3v2_head_bytes = Optional(Id3v2Format).build(self.metadata['id3v2_head']) 154 | return self._id3v2_head_bytes 155 | 156 | def id3v2_tail_bytes(self): 157 | if self.metadata['id3v2_tail'] and self._id3v2_tail_bytes == None: 158 | self._id3v2_tail_bytes = Optional(Id3v2Format).build(self.metadata['id3v2_tail']) 159 | return self._id3v2_tail_bytes 160 | 161 | def id3v1_bytes(self): 162 | if self.metadata['id3v1'] and self._id3v1_bytes == None: 163 | self._id3v1_bytes = Optional(Id3v1Format).build(self.metadata['id3v1']) 164 | return self._id3v1_bytes 165 | 166 | def read(self, length, offset): 167 | f = self.file 168 | buf = b'' 169 | 170 | id3v2_head_len = self.id3v2_head_len 171 | if offset < id3v2_head_len: 172 | b = self.id3v2_head_bytes() 173 | end = min(id3v2_head_len, offset + length) 174 | buf = b[offset:end] 175 | length -= len(buf) 176 | if not length: 177 | return buf 178 | offset = id3v2_head_len 179 | offset -= id3v2_head_len 180 | 181 | audio_len = self.audio_len 182 | if offset < audio_len: 183 | audio_offset = self.audio_offset + offset 184 | f.seek(audio_offset, 0) 185 | tmp = f.read(length) 186 | length -= len(tmp) 187 | buf += tmp 188 | if not length: 189 | return buf 190 | offset = audio_len 191 | offset -= audio_len 192 | 193 | id3v2_tail_len = self.id3v2_tail_len 194 | if offset < id3v2_tail_len: 195 | b = self.id3v2_tail_bytes() 196 | end = min(id3v2_tail_len, offset + length) 197 | tmp = b[offset:end] 198 | length -= len(tmp) 199 | buf += tmp 200 | if not length: 201 | return buf 202 | offset = id3v2_tail_len 203 | offset -= id3v2_tail_len 204 | 205 | b = self.id3v1_bytes() 206 | return buf + b[offset:offset+length] 207 | -------------------------------------------------------------------------------- /freezetag/freezefs.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import gc 4 | import os 5 | import platform 6 | import sys 7 | import time 8 | from collections import OrderedDict 9 | from errno import ENOENT 10 | from pathlib import Path 11 | from stat import S_IFDIR 12 | from threading import Lock, Timer 13 | 14 | from appdirs import user_cache_dir 15 | from watchdog.events import FileSystemEventHandler 16 | from watchdog.observers import Observer 17 | 18 | from .base import FuseFile, MusicMetadata, ParsedFile 19 | from .core import ChecksumDB, Freezetag 20 | 21 | try: 22 | from fuse import FUSE, FuseOSError, Operations 23 | except: 24 | fuse_urls = { 25 | 'Darwin': 'Install it via homebrew by running "brew cask install osxfuse", or manually from https://github.com/osxfuse/osxfuse/releases/latest.', 26 | 'Linux': 'Install fuse2 from your package manager, or manually from https://github.com/libfuse/libfuse/releases/tag/fuse-2.9.9.', 27 | 'Windows': 'Download and install it from https://github.com/billziss-gh/winfsp/releases/latest.', 28 | } 29 | print('Could not load FUSE.', file=sys.stderr) 30 | print(fuse_urls[platform.system()], file=sys.stderr) 31 | sys.exit(1) 32 | 33 | FREEZETAG_CACHE_LIMIT = 10 34 | FREEZETAG_KEEPALIVE_TIME = 10 35 | CACHE_DIR = Path(user_cache_dir('freezetag', 'x1ppy')) 36 | 37 | ST_ITEMS = ['st_atime', 'st_ctime', 'st_gid', 'st_mode', 'st_mtime', 'st_nlink', 'st_size', 'st_uid'] 38 | if platform.system() != 'Windows': 39 | ST_ITEMS.append('st_birthtime') 40 | 41 | 42 | # An LRU cache with mostly standard behavior besides one thing: it asks whether 43 | # an item can be purged before doing so. This prevents the cache from purging 44 | # freezetags that are still be in use by open files; that way, they won't be 45 | # needlessly recreated in memory if another file with the same freezetag is 46 | # opened. This also means the cache could technically expand past its maxsize 47 | # if all items cannot be purged. 48 | class PoliteLRUCache(OrderedDict): 49 | def __init__(self, getter, can_purge, maxsize=128, *args, **kwds): 50 | self.maxsize = maxsize 51 | self.getter = getter 52 | self.can_purge = can_purge 53 | super().__init__(*args, **kwds) 54 | 55 | def __getitem__(self, key): 56 | if key not in self: 57 | self.__setitem__(key, self.getter(key)) 58 | value = super().__getitem__(key) 59 | self.move_to_end(key) 60 | return value 61 | 62 | def __setitem__(self, key, value): 63 | super().__setitem__(key, value) 64 | if len(self) > self.maxsize: 65 | for i in range(len(self) - 1): 66 | oldest = next(iter(self)) 67 | if self.can_purge(oldest): 68 | del self[oldest] 69 | break 70 | self.move_to_end(oldest) 71 | 72 | 73 | class FrozenItemFreezetagEntry: 74 | def __init__(self, freezetag_path, path, metadata_len): 75 | self.freezetag_path = freezetag_path 76 | self.path = path 77 | self.metadata_len = metadata_len 78 | 79 | 80 | class FrozenItemFileEntry: 81 | def __init__(self, path, metadata_info, metadata_len): 82 | self.path = path 83 | self.metadata_info = metadata_info 84 | self.metadata_len = metadata_len 85 | 86 | 87 | class FrozenItem: 88 | def __init__(self, checksum): 89 | self.checksum = checksum 90 | self.freezetags = [] 91 | self.files = [] 92 | 93 | 94 | class FreezeFS(Operations, FileSystemEventHandler): 95 | def __init__(self, verbose=False): 96 | self.path_map = {Path('/').root: {}} 97 | self.checksum_map = {} 98 | self.abs_path_map = {} 99 | self.freezetag_map = {} 100 | self.inactive_freezetags = [] 101 | self.freezetag_ref_lock = Lock() 102 | self.fh_map = {} 103 | self.checksum_db = ChecksumDB(CACHE_DIR / 'freezefs.db') 104 | self.verbose = verbose 105 | 106 | # self.freezetag_ref_lock must be acquired before accessing. 107 | self.freezetag_cache = PoliteLRUCache(Freezetag.from_path, self._can_purge_ftag, FREEZETAG_CACHE_LIMIT) 108 | 109 | # self.freezetag_ref_lock must be acquired before accessing. 110 | self.freezetag_refs = {} 111 | 112 | now = time.time() 113 | 114 | try: 115 | gid = os.getgid() 116 | uid = os.getuid() 117 | except: 118 | # Windows. 119 | gid = 0 120 | uid = 0 121 | 122 | self.dir_stat = { 123 | 'st_atime': now, 124 | 'st_ctime': now, 125 | 'st_mtime': now, 126 | 'st_birthtime': now, 127 | 'st_mode': S_IFDIR | 0o755, 128 | 'st_nlink': 2, 129 | 'st_gid': gid, 130 | 'st_uid': uid, 131 | } 132 | 133 | def mount(self, directory, mount_point): 134 | CACHE_DIR.mkdir(parents=True, exist_ok=True) 135 | 136 | observer = Observer() 137 | observer.schedule(self, directory, recursive=True) 138 | observer.start() 139 | 140 | print(f'scanning {directory} for files and freezetags...') 141 | 142 | for path in walk_dir(directory): 143 | (self._add_ftag if path.suffix.lower() == '.ftag' else self._add_file)(path) 144 | self.checksum_db.flush() 145 | 146 | print(f'mounting {mount_point}') 147 | FUSE(self, mount_point, nothreads=True, foreground=True, fsname='freezefs', volname=Path(mount_point).name) 148 | 149 | # Helpers 150 | # ======= 151 | 152 | def _log_verbose(self, msg): 153 | if self.verbose: 154 | print(msg) 155 | 156 | def _add_freezetag_entry(self, checksum, entry): 157 | if checksum not in self.checksum_map: 158 | item = FrozenItem(checksum) 159 | self.checksum_map[checksum] = item 160 | else: 161 | item = self.checksum_map[checksum] 162 | 163 | item.freezetags.append(entry) 164 | 165 | assert (not self._get_item(entry.path)) 166 | 167 | map = self.path_map 168 | parts = Path(entry.path).parts 169 | for part in parts[0:-1]: 170 | if part not in map: 171 | map[part] = {} 172 | map = map[part] 173 | map[parts[-1]] = item 174 | 175 | def _add_path_entry(self, checksum, entry): 176 | if checksum not in self.checksum_map: 177 | item = FrozenItem(checksum) 178 | self.checksum_map[checksum] = item 179 | else: 180 | item = self.checksum_map[checksum] 181 | 182 | item.files.append(entry) 183 | self.abs_path_map[entry.path] = item 184 | 185 | def _get_item(self, path): 186 | item = self.path_map 187 | for part in Path(path).parts: 188 | if part not in item: 189 | return None 190 | item = item[part] 191 | return item 192 | 193 | def _add_ftag(self, path): 194 | self.freezetag_ref_lock.acquire() 195 | try: 196 | freezetag = self.freezetag_cache[path] 197 | self._schedule_purge_ftag(path) 198 | except KeyboardInterrupt: 199 | raise 200 | except: 201 | print(f'cannot parse freezetag: {path}') 202 | return 203 | finally: 204 | self.freezetag_ref_lock.release() 205 | 206 | self._log_verbose(f'adding freezetag: {path}') 207 | 208 | root = Path('/') / freezetag.data.frozen.root 209 | item = self._get_item(root) 210 | if item: 211 | print(f'cannot mount {path} to {root}: path already mounted by another freezetag') 212 | self.inactive_freezetags.append([root, path]) 213 | return 214 | 215 | self.freezetag_map[path] = freezetag_map = (root, []) 216 | 217 | for state in freezetag.data.frozen.files: 218 | fuse_path = Path('/') / freezetag.data.frozen.root / state.path 219 | metadata = MusicMetadata.from_state(state) 220 | metadata_len = sum(m[1] for m in metadata) if metadata else 0 221 | entry = FrozenItemFreezetagEntry(path, fuse_path, metadata_len) 222 | self._add_freezetag_entry(state.checksum, entry) 223 | freezetag_map[1].append(state.checksum) 224 | 225 | def _add_file(self, src): 226 | try: 227 | st = src.stat() 228 | except: 229 | print(f'cannot stat file: {src}') 230 | return 231 | 232 | cached = self.checksum_db.get(st.st_dev, st.st_ino, st.st_mtime) 233 | if cached: 234 | self._log_verbose(f'adding cached file: {src}') 235 | checksum, metadata_info, metadata_len, mtime = cached 236 | entry = FrozenItemFileEntry(src, metadata_info, metadata_len) 237 | self._add_path_entry(checksum, entry) 238 | return 239 | 240 | file = ParsedFile.from_path(src) 241 | try: 242 | file.parse() 243 | except KeyboardInterrupt: 244 | raise 245 | except: 246 | print(f'cannot parse file: {src}') 247 | return 248 | 249 | self._log_verbose(f'adding new file: {src}') 250 | 251 | metadata = file.strip() 252 | metadata_info = list(metadata) if metadata else [] 253 | metadata_len = sum(m[1] for m in metadata_info) if metadata else 0 254 | checksum = file.checksum() 255 | entry = FrozenItemFileEntry(src, metadata_info, metadata_len) 256 | self.checksum_db.add(st.st_dev, st.st_ino, st.st_mtime, checksum, metadata_info, metadata_len) 257 | self._add_path_entry(checksum, entry) 258 | 259 | def _delete_if_dangling(self, item, fuse_path, file_path): 260 | if not len(item.freezetags) and not len(item.files): 261 | del self.checksum_map[item.checksum] 262 | 263 | if fuse_path and not any(entry.path == fuse_path for entry in item.freezetags): 264 | map = self.path_map 265 | parents = [] 266 | for part in Path(fuse_path).parts: 267 | parents.append((part, map)) 268 | map = map[part] 269 | while len(parents): 270 | part, parent = parents.pop() 271 | del parent[part] 272 | if len(parent): 273 | break 274 | 275 | if file_path and not len(item.files): 276 | del self.abs_path_map[file_path] 277 | 278 | def _can_purge_ftag(self, path): 279 | assert (self.freezetag_ref_lock.locked()) 280 | 281 | if path not in self.freezetag_refs: 282 | return True 283 | return self.freezetag_refs[path][1] <= 0 284 | 285 | def _purge_ftag(self, path, force): 286 | self.freezetag_ref_lock.acquire() 287 | try: 288 | no_refs = path in self.freezetag_refs and self.freezetag_refs[path][1] <= 0 289 | if (force or no_refs) and path in self.freezetag_cache: 290 | del self.freezetag_cache[path] 291 | gc.collect() 292 | if no_refs and path in self.freezetag_refs: 293 | del self.freezetag_refs[path] 294 | finally: 295 | self.freezetag_ref_lock.release() 296 | 297 | def _schedule_purge_ftag(self, path): 298 | assert (self.freezetag_ref_lock.locked()) 299 | 300 | t = Timer(FREEZETAG_KEEPALIVE_TIME, lambda: self._purge_ftag(path, force=False)) 301 | t.start() 302 | 303 | if not path in self.freezetag_refs: 304 | self.freezetag_refs[path] = [t, 0] 305 | else: 306 | timer = self.freezetag_refs[path][0] 307 | timer and timer.cancel() 308 | self.freezetag_refs[path][0] = t 309 | 310 | # Filesystem methods 311 | # ================== 312 | 313 | def getattr(self, path, fh=None): 314 | path = Path(path) 315 | 316 | item = self._get_item(path) 317 | if item == None: 318 | raise FuseOSError(ENOENT) 319 | 320 | if isinstance(item, FrozenItem): 321 | if not len(item.freezetags) or not len(item.files): 322 | raise FuseOSError(ENOENT) 323 | 324 | file_entry = item.files[0] 325 | frozen_entry = None 326 | for entry in item.freezetags: 327 | if entry.path == path: 328 | frozen_entry = entry 329 | break 330 | 331 | if not frozen_entry: 332 | raise FuseOSError(ENOENT) 333 | 334 | st = file_entry.path.stat() 335 | d = {key: getattr(st, key) for key in ST_ITEMS} 336 | d['st_size'] += frozen_entry.metadata_len - file_entry.metadata_len 337 | return d 338 | 339 | return self.dir_stat 340 | 341 | def readdir(self, path, fh): 342 | yield '.' 343 | yield '..' 344 | for name, item in self._get_item(path).items(): 345 | if isinstance(item, FrozenItem) and (not len(item.freezetags) or not len(item.files)): 346 | continue 347 | yield name 348 | 349 | # File methods 350 | # ============ 351 | 352 | def open(self, path, flags): 353 | path = Path(path) 354 | item = self._get_item(path) 355 | if not item: 356 | raise FuseOSError(ENOENT) 357 | 358 | # As long as the raw checksum matches, any file should work, so just use the first one we have. 359 | file_entry = item.files[0] 360 | 361 | frozen_entry = None 362 | for entry in item.freezetags: 363 | if entry.path == path: 364 | frozen_entry = entry 365 | break 366 | 367 | if not frozen_entry: 368 | raise FuseOSError(ENOENT) 369 | 370 | freezetag_path = None 371 | metadata = None 372 | if frozen_entry.metadata_len: 373 | freezetag_path = frozen_entry.freezetag_path 374 | 375 | self.freezetag_ref_lock.acquire() 376 | try: 377 | if freezetag_path not in self.freezetag_refs: 378 | self.freezetag_refs[freezetag_path] = [None, 0] 379 | self.freezetag_refs[freezetag_path][1] += 1 380 | freezetag = self.freezetag_cache[freezetag_path] 381 | finally: 382 | self.freezetag_ref_lock.release() 383 | 384 | for f in freezetag.data.frozen.files: 385 | if f.checksum == item.checksum: 386 | metadata = f.metadata 387 | break 388 | 389 | file = FuseFile.from_info(file_entry.path, flags, metadata, file_entry.metadata_info, file_entry.metadata_len, 390 | frozen_entry.metadata_len) 391 | self.fh_map[file.fh] = (file, freezetag_path) 392 | return file.fh 393 | 394 | def read(self, path, length, offset, fh): 395 | return self.fh_map[fh][0].read(length, offset) 396 | 397 | def release(self, path, fh): 398 | f, freezetag_path = self.fh_map[fh] 399 | 400 | if freezetag_path: 401 | self.freezetag_ref_lock.acquire() 402 | try: 403 | self.freezetag_refs[freezetag_path][1] -= 1 404 | self._schedule_purge_ftag(freezetag_path) 405 | finally: 406 | self.freezetag_ref_lock.release() 407 | 408 | del self.fh_map[fh] 409 | return f.close() 410 | 411 | # watchdog observers 412 | # ================== 413 | 414 | def on_moved(self, event): 415 | src = Path(event.src_path) 416 | dst = Path(event.dest_path) 417 | if dst.is_dir(): 418 | return 419 | 420 | self._log_verbose(f'moved: {src} to {dst}') 421 | 422 | if src.suffix.lower() == '.ftag': 423 | self._purge_ftag(src, force=True) 424 | 425 | freezetag_map = self.freezetag_map.get(src) 426 | if not freezetag_map: 427 | for tag in self.inactive_freezetags: 428 | if tag[1] == src: 429 | tag[1] = dst 430 | break 431 | return 432 | 433 | del self.freezetag_map[src] 434 | self.freezetag_map[dst] = freezetag_map 435 | 436 | for checksum in freezetag_map[1]: 437 | item = self.checksum_map[checksum] 438 | for entry in item.freezetags: 439 | if entry.freezetag_path == src: 440 | entry.freezetag_path = dst 441 | break 442 | return 443 | 444 | item = self.abs_path_map[src] 445 | del self.abs_path_map[src] 446 | self.abs_path_map[dst] = item 447 | 448 | for entry in item.files: 449 | if entry.path == src: 450 | entry.path = dst 451 | 452 | def on_created(self, event): 453 | path = Path(event.src_path) 454 | if path.is_dir(): 455 | return 456 | 457 | if path.suffix.lower() == '.ftag': 458 | self._add_ftag(path) 459 | return 460 | 461 | self._add_file(path) 462 | 463 | def on_deleted(self, event): 464 | path = Path(event.src_path) 465 | 466 | if path.suffix.lower() != '.ftag': 467 | self._log_verbose(f'deleting file {path}') 468 | 469 | item = self.abs_path_map.get(path) 470 | if not item: 471 | return 472 | 473 | for entry in item.files: 474 | if entry.path == path: 475 | item.files.remove(entry) 476 | self._delete_if_dangling(item, fuse_path=None, file_path=path) 477 | break 478 | return 479 | 480 | self._log_verbose(f'deleting freezetag {path}') 481 | 482 | self._purge_ftag(path, force=True) 483 | 484 | freezetag_map = self.freezetag_map.get(path) 485 | if not freezetag_map: 486 | for tag in self.inactive_freezetags: 487 | if tag[1] == path: 488 | self.inactive_freezetags.remove(tag) 489 | break 490 | return 491 | 492 | for checksum in freezetag_map[1]: 493 | item = self.checksum_map[checksum] 494 | for entry in item.freezetags: 495 | if entry.freezetag_path == path: 496 | item.freezetags.remove(entry) 497 | self._delete_if_dangling(item, fuse_path=entry.path, file_path=None) 498 | break 499 | del self.freezetag_map[path] 500 | 501 | root = freezetag_map[0] 502 | for tag in self.inactive_freezetags: 503 | if tag[0] == root: 504 | self.inactive_freezetags.remove(tag) 505 | self._add_ftag(tag[1]) 506 | break 507 | 508 | def on_modified(self, event): 509 | self.on_deleted(event) 510 | self.on_created(event) 511 | 512 | 513 | def walk_dir(path): 514 | for dirpath, dirnames, filenames in os.walk(path): 515 | dirnames.sort() 516 | for filename in sorted(filenames): 517 | yield Path(dirpath) / filename 518 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | import setuptools 2 | 3 | with open("README.md", "r") as fh: 4 | long_description = fh.read() 5 | 6 | setuptools.setup( 7 | name="freezetag", 8 | version="1.2.1", 9 | author="x1ppy", 10 | author_email="", 11 | packages=[ 12 | 'freezetag', 13 | 'freezetag.formats', 14 | ], 15 | entry_points={ 16 | 'console_scripts': [ 17 | 'freezetag = freezetag.__main__:main', 18 | ], 19 | }, 20 | description="save, strip, and restore file paths and music metadata", 21 | long_description=long_description, 22 | long_description_content_type="text/markdown", 23 | url="https://github.com/x1ppy/freezetag", 24 | python_requires='>=3.5.2', 25 | install_requires=[ 26 | 'appdirs', 27 | 'construct', 28 | 'fusepy', 29 | 'watchdog', 30 | ], 31 | ) 32 | --------------------------------------------------------------------------------