├── .github ├── ISSUE_TEMPLATE │ ├── bug-report.md │ └── feature_request.md └── pull_request_template.md ├── .gitignore ├── .travis.yml ├── CONTRIBUTING.md ├── LICENCE ├── README.md ├── bin └── sync-dl ├── icon.png ├── privacy-policy.md ├── setup.py ├── sync_dl ├── __init__.py ├── cli.py ├── commands.py ├── config │ ├── __init__.py │ ├── parsing.py │ ├── tmpdir.py │ └── ytdlParams.py ├── helpers.py ├── plManagement.py ├── tests │ ├── integrationTests.py │ ├── test.mp3 │ └── unitTests.py ├── timestamps │ ├── __init__.py │ └── scraping.py ├── ytapiInterface.py └── ytdlWrappers.py └── test.py /.github/ISSUE_TEMPLATE/bug-report.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Bug Report 3 | about: Create a report to help us improve 4 | title: Bug 5 | labels: bug 6 | assignees: '' 7 | 8 | --- 9 | 10 | # Before Report 11 | Be sure you are using the current version of sync-dl, update using: 12 | ``` 13 | pip install sync-dl --upgrade 14 | ``` 15 | # Bug Report 16 | ## Version 17 | ``` 18 | Copy paste output of the following command here: 19 | sync-dl --version 20 | ``` 21 | ## Details 22 | Give information about the bug here, including: 23 | - Affected commands 24 | - What system you are using (Windows/specific Linux distro/Mac) 25 | - URL of playlist used (if you are comfortable with it) 26 | 27 | ## Verbose Output 28 | ``` 29 | Copy paste output of the following command here: 30 | sync-dl -v [affected_command] 31 | ``` 32 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/feature_request.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Feature request 3 | about: Suggest an idea for this project 4 | title: '' 5 | labels: enhancement 6 | assignees: '' 7 | 8 | --- 9 | 10 | # Feature Request 11 | What do you want added or changed? 12 | 13 | State what commands that need to be updated, or what new commands you want added. 14 | ``` 15 | sync-dl [NEW_COMMAND] 16 | ``` 17 | 18 | Give some details on the use case your new feature. 19 | -------------------------------------------------------------------------------- /.github/pull_request_template.md: -------------------------------------------------------------------------------- 1 | # Before Pull Request 2 | 3 | Make sure your build passes the unit and integration tests: 4 | ``` 5 | python3 test.py [PLAYLIST_URL] 6 | ``` 7 | 8 | # Pull Request 9 | 10 | ## This pull request is for: 11 | - [ ] bug fix 12 | - [ ] new feature 13 | - [ ] improvement 14 | - [ ] other (specify below): 15 | 16 | ## Details: 17 | Be sure to include links relevant issues if applicable. 18 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | __pycache__ 2 | sync_dl/tests/testPlaylists 3 | sync_dl/tests/testing.log 4 | notepad* 5 | debug.log 6 | sync_dl/tmp 7 | sync_dl/config.ini 8 | config.ini 9 | sync_dl.egg-info 10 | 11 | /build 12 | /dist 13 | sync_dl/yt_api/credentials 14 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: python 2 | 3 | python: 4 | - 3.6 5 | - 3.8 6 | - 3.9 7 | 8 | branches: 9 | - main 10 | 11 | before_script: 12 | - sudo apt-get install ffmpeg 13 | - pip install sync_dl_ytapi 14 | - pip install -e . 15 | 16 | script: 17 | - python3 test.py https://www.youtube.com/playlist\?list=PLbg8uA1JzGJD56Lfl7aYK4iW2vJHDo0DE 18 | 19 | after_failure: 20 | - ls 21 | - ls sync_dl 22 | - ls sync_dl/tests 23 | - echo $(cat sync_dl/tests/testing.log) 24 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # ISSUES 2 | State your issue, what platform you are on, and, if applicable, the output of running the relevent command with the verbose flag: 3 | ``` 4 | sync-dl -v [REST_OF_COMMAND] 5 | ``` 6 | If your issue involves metadata corruption, the output of the -d and -p flags would be helpful 7 | ``` 8 | sync-dl -d [NAME_OF_PLAYLIST] 9 | ``` 10 | ``` 11 | sync-dl -p [NAME_OF_PLAYLIST] 12 | ``` 13 | # Devlopment 14 | The Hiarchy of the project is as follows (with scripts only depending on scripts beneath them in the list) 15 | 16 | 17 | ### `cli.py` 18 | -> The command line interface 19 | 20 | ### `commands.py` 21 | -> The commands run by cli, commands from here can also imported and envoked by other projects (such as the android app). 22 | 23 | ### `plManagement.py` 24 | -> Large functions used by the commands 25 | 26 | ### `helpers.py` 27 | -> small functions and wrappers 28 | 29 | ### `ytdlWrappers.py` 30 | -> everything which directly interfaces with youtube-dl 31 | 32 | ### `config.py` 33 | -> configuration and global variables 34 | 35 | ### `config.ini` 36 | -> holds user editable configuration, if this does exist a default one will be created by `config.py` 37 | 38 | ## Entry points 39 | The entry point for the code is in `__init__.py` which calls the main function of `cli.py`. on windows `__init__.py` is the console script that is added to path, whereas and on linux bin/sync-dl is used, which simply calls the main() function of `__init__.py`. 40 | 41 | `__init__.py` also holds a singleton called noInturrupt, which is used as a handler for SIGINT, it can also be used to simulate a SIGINT through code (used for canceling in the android app). 42 | 43 | ## encrypted/ 44 | This holds the encrypted oauth api key along with the obfuscated code needed to decrypt it, this is only used for the --push-order command. any modifications made to newCredentials.py will cause the decryption to fail (by design) 45 | 46 | # PULL REQUESTS 47 | Before all else be sure all changes pass the unit and integration tests 48 | ``` 49 | python3 test.py [PLAYLIST_URL] 50 | ``` 51 | 52 | Travis CI will also run these on your pull request, however its best if you also run them on your own machine. 53 | 54 | Say what the pull request is for: 55 | - bug fix 56 | - new feature 57 | - improvement 58 | 59 | And link to any relevent issues that the pull request is addressing. 60 | -------------------------------------------------------------------------------- /LICENCE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2020 Joshua McPherson 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # sync-dl 4 |

5 | 6 | 7 | 8 | 9 | 10 | 11 |

12 | 13 | 14 | 15 | > A tool for downloading and syncing remote playlists to your computer 16 | - [INSTALLATION](#INSTALLATION) 17 | - [ABOUT](#ABOUT) 18 | - [USAGE](#USAGE) 19 | - [EXAMPLE](#EXAMPLE) 20 | - [DEVLOPMENT](#DEVLOPMENT) 21 | 22 | 23 | # INSTALLATION 24 | Requires ffmpeg or avconv, however one of these comes pre-installed on most machines. 25 | 26 | Install sync-dl via pypi using pip: 27 | ``` 28 | pip install sync-dl 29 | ``` 30 | 31 | # ABOUT 32 | Created to avoid having music deleted but still have the convenience of browsing, adding and reordering new music using youtube. 33 | 34 | The application does not store any of its metadata in songs, metadata is stored next to them in a .metadata file, the music files are managed through numbering, allowing them to be played alphanumerically using any playback service (such as VLC). 35 | 36 | 37 | # Usage 38 | ``` 39 | sync-dl [options] COMMAND [options] PLAYLIST 40 | ``` 41 | 42 | sync-dl has the several subcommands, run `sync-dl -h` to see them all and `sync-dl [COMMAND] -h` to get info on a particular one. 43 | As an example, here is the new command which creates new playlists from a youtube [URL]: 44 | 45 | ``` 46 | sync-dl new [URL] [PLAYLIST] 47 | ``` 48 | 49 | The playlist will be put it in directory [PLAYLIST], which is relative to the current working directory unless you specify your music directory using: 50 | 51 | ``` 52 | sync-dl config -l [PATH] 53 | ``` 54 | 55 | Where [PATH] is where you wish to store all your playlists in, ie) `~/Music`. 56 | 57 | 58 | ## Smart Sync: 59 | The main feature of sync-dl: 60 | ``` 61 | sync-dl sync -s PLAYLIST 62 | ``` 63 | 64 | Adds new music from remote playlist to local playlist, also takes ordering of remote playlist 65 | without deleting songs no longer available in remote playlist. 66 | 67 | Songs that are no longer available in remote, will remain after the song they are currently after 68 | in the local playlist to maintain playlist flow. 69 | 70 | 71 | ## Push Order: 72 | ``` 73 | sync-dl ytapi --push order [PLAYLIST] 74 | ``` 75 | sync-dl has a submodule which uses the youtube api the preform the reverse of Smart Sync called Push Order. sync-dl will prompt you to install the submodule if you use any of its options ie) --push-order. you must also sign in with google (so sync-dl can edit the order of your playlist). 76 | 77 | For more information see https://github.com/PrinceOfPuppers/sync-dl-ytapi 78 | 79 | ## Many More! 80 | Includes tools managing large playlists, For example `sync-dl edit --move-range [I1] [I2] [NI] [PLAYLIST]` which allows a user to move a block of songs From [I1] to [I2] to after song [N1]. 81 | 82 | Moving large blocks of songs on youtube requires dragging each song individually up/down a the page as it trys to dynamically load the hunders of songs you're scrolling past, which you would have to do every time you would want to add new music to somewhere other than the end of the playlist... (ask me how I know :^P) 83 | 84 | 85 | # EXAMPLE 86 | ``` 87 | sync-dl config -l my/local/music/folder 88 | ``` 89 | Will use my/local/music/folder to store and manipulate playlists in the future. 90 | ``` 91 | sync-dl new https://www.youtube.com/playlist?list=PL-lQgJ3eJ1HtLvM6xtxOY_1clTLov0qM3 sweetJams 92 | ``` 93 | Will create a new playlist at my/local/music/folder/sweetJams and 94 | download the playlist at the provided url to it. 95 | 96 | ``` 97 | sync-dl timestamps --scrape-range 0 4 sweetJams 98 | ``` 99 | Will scrape youtube comments for timestamps to add to songs 0 to 4 of sweetJams. Will ask you to review them before it adds them (can be changed with option -a). 100 | 101 | ``` 102 | sync-dl edit -m 1 5 sweetJams 103 | ``` 104 | Will move song number 1 in the playlist to position 5. 105 | 106 | ``` 107 | sync-dl sync -a sweetJams 108 | ``` 109 | Will check for any new songs in the remote playlist and append them to the end of sweetJams. 110 | 111 | ``` 112 | sync-dl sync -s sweetJams 113 | ``` 114 | Will use smart sync on sweetJams, downloading new songs from the remote playlist and reordering the playlist to match the order of the remote playlist without deleting any songs that are no longer available. 115 | 116 | ``` 117 | sync-dl edit --move-range 0 4 8 sweetJams 118 | ``` 119 | Will move all songs from 0 to 4 to after song 8. 120 | 121 | ``` 122 | sync-dl info -p sweetJams 123 | ``` 124 | Will give you all the urls for the songs in sweetJams. 125 | 126 | ``` 127 | sync-dl ytapi --push-order sweetJams 128 | ``` 129 | Will prompt you to install sync-dl-ytapi and sign in with google (if you havent already), after doing so it will push the local order of the playlist to youtube. 130 | 131 | ``` 132 | sync-dl ytapi --logout 133 | ``` 134 | Will remove invalidate and delete access and refresh token for the youtube api, requiring you to log in next time you use `sync-dl ytapi --pushorder`. 135 | 136 | 137 | # DEVLOPMENT 138 | To build for devlopment run: 139 | ``` 140 | git clone https://github.com/PrinceOfPuppers/sync-dl.git 141 | 142 | cd sync-dl 143 | 144 | pip install -e . 145 | ``` 146 | This will build and install sync-dl in place, allowing you to work on the code without having to reinstall after changes. 147 | 148 | 149 | ## Automated Testing 150 | ``` 151 | python test.py [options] TEST_PLAYLIST_URL 152 | ``` 153 | Will run all unit and integration tests, for the integration tests it will use the playlist TEST_PLAYLIST_URL, options are -u and -i to only run the unit/integration tests respectively. 154 | 155 | Testing requires sync-dl-ytapi to be installed aswell, and will test its helper functions. 156 | -------------------------------------------------------------------------------- /bin/sync-dl: -------------------------------------------------------------------------------- 1 | #!/url/bin/env python3 2 | 3 | if __name__ == "__main__": 4 | from sync_dl import main 5 | main() -------------------------------------------------------------------------------- /icon.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PrinceOfPuppers/sync-dl/bb2cbf208664ac7407e18abd5e7d30779a7407a0/icon.png -------------------------------------------------------------------------------- /privacy-policy.md: -------------------------------------------------------------------------------- 1 | # Privacy Policy 2 | For information about sync-dl's privacy policy please visit: 3 | http://sync-dl.com/privacy-policy/ -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | import setuptools 2 | from ntpath import dirname 3 | 4 | with open('README.md', 'r') as f: 5 | longDescription = f.read() 6 | 7 | # single sourcing version number to __init__.py 8 | def getVersion(pkgDir): 9 | currentPath = dirname(__file__) 10 | initPath = f"{currentPath}/{pkgDir}/__init__.py" 11 | 12 | with open(initPath) as f: 13 | for line in f.readlines(): 14 | if line.startswith("__version__"): 15 | delim = '"' if '"' in line else "'" 16 | return line.split(delim)[1] 17 | 18 | else: 19 | raise RuntimeError("Unable to find version string.") 20 | 21 | setuptools.setup( 22 | name="sync-dl", 23 | version=getVersion("sync_dl"), 24 | author="Joshua McPherson", 25 | author_email="joshuamcpherson5@gmail.com", 26 | description="A tool for downloading and syncing remote playlists to your computer", 27 | long_description = longDescription, 28 | long_description_content_type = 'text/markdown', 29 | url="https://github.com/PrinceOfPuppers/sync-dl", 30 | packages=setuptools.find_packages(), 31 | include_package_data=True, 32 | install_requires=['yt-dlp', 'sync-dl-ytapi>=1.1.2'], 33 | classifiers=[ 34 | "Programming Language :: Python :: 3", 35 | "License :: OSI Approved :: MIT License", 36 | "Operating System :: OS Independent", 37 | "Environment :: Console", 38 | "Intended Audience :: End Users/Desktop", 39 | ], 40 | python_requires='>=3.6', 41 | scripts=["bin/sync-dl"], 42 | entry_points={ 43 | 'console_scripts': ['sync-dl = sync_dl:main'], 44 | }, 45 | ) 46 | 47 | 48 | -------------------------------------------------------------------------------- /sync_dl/__init__.py: -------------------------------------------------------------------------------- 1 | 2 | from threading import current_thread 3 | from sync_dl import config as cfg 4 | from signal import signal, SIGINT, Signals#,SIGABRT,SIGTERM 5 | 6 | __version__ = "2.3.8" 7 | 8 | class InterruptTriggered(Exception): 9 | pass 10 | 11 | class _NoInterrupt: 12 | noInterruptDepth = 0 13 | signalReceived=False 14 | 15 | def __enter__(self): 16 | if self.interruptible() and self.signalReceived: 17 | self.signalReceived = False 18 | self.interrupt() 19 | 20 | self.noInterruptDepth += 1 21 | 22 | 23 | def __exit__(self, type, value, traceback): 24 | self.noInterruptDepth -= 1 25 | self.noInterruptDepth = max(0, self.noInterruptDepth) 26 | if self.interruptible() and self.signalReceived: 27 | self.signalReceived = False 28 | self.interrupt() 29 | 30 | def interrupt(self): 31 | raise InterruptTriggered 32 | 33 | def notInterruptible(self): 34 | return self.noInterruptDepth > 0 35 | 36 | def interruptible(self): 37 | return self.noInterruptDepth == 0 38 | 39 | def handler(self,sig,frame): 40 | if current_thread().__class__.__name__ != '_MainThread': 41 | return 42 | 43 | if self.interruptible(): 44 | self.interrupt() 45 | 46 | self.signalReceived = True 47 | cfg.logger.info(f'{Signals(2).name} Received, Closing after this Operation') 48 | 49 | def simulateSigint(self): 50 | '''can be used to trigger intrrupt from another thread''' 51 | self.signalReceived = True 52 | 53 | 54 | 55 | noInterrupt = _NoInterrupt() 56 | signal(SIGINT,noInterrupt.handler) 57 | 58 | def main(): 59 | from sync_dl.cli import cli 60 | cli() 61 | -------------------------------------------------------------------------------- /sync_dl/cli.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import logging 4 | 5 | import argparse 6 | import shelve 7 | 8 | 9 | from sync_dl import __version__, InterruptTriggered 10 | from sync_dl.plManagement import correctStateCorruption 11 | import sync_dl.config as cfg 12 | 13 | from sync_dl.commands import newPlaylist,smartSync,appendNew,manualAdd,swap, showPlaylist, compareMetaData, moveRange, peek, togglePrepends, addTimestampsFromComments 14 | from sync_dl.ytapiInterface import logout, pushLocalOrder, transferSongs 15 | 16 | 17 | 18 | #modified version of help formatter which only prints args once in help message 19 | class ArgsOnce(argparse.HelpFormatter): 20 | def __init__(self,prog): 21 | super().__init__(prog,max_help_position=40) 22 | 23 | def _format_action_invocation(self, action): 24 | if not action.option_strings: 25 | metavar, = self._metavar_formatter(action, action.dest)(1) 26 | return metavar 27 | else: 28 | parts = [] 29 | 30 | if action.nargs == 0: 31 | parts.extend(action.option_strings) 32 | 33 | else: 34 | default = action.dest.upper() 35 | args_string = self._format_args(action, default) 36 | for option_string in action.option_strings: 37 | parts.append('%s' % option_string) 38 | parts[-1] += ' %s'%args_string 39 | return ', '.join(parts) 40 | 41 | 42 | def setupLogger(args): 43 | '''sets cfg.logger level based on verbosity''' 44 | #verbosity 45 | stream = logging.StreamHandler(sys.stdout) 46 | if args.verbose: 47 | cfg.logger.setLevel(logging.DEBUG) 48 | stream.setFormatter(logging.Formatter("[%(levelname)s] %(message)s")) 49 | 50 | elif args.quiet: 51 | cfg.logger.setLevel(logging.ERROR) 52 | stream.setFormatter(logging.Formatter("[%(levelname)s] %(message)s")) 53 | else: 54 | cfg.logger.setLevel(logging.INFO) 55 | stream.setFormatter(logging.Formatter("%(message)s")) 56 | cfg.logger.addHandler(stream) 57 | 58 | def getCwd(): 59 | if cfg.musicDir == '': 60 | return os.getcwd() 61 | else: 62 | return cfg.musicDir 63 | 64 | 65 | def playlistExists(plPath, noError = False): 66 | '''tests if valid playlist with metadata file exists at provided path''' 67 | #everything past this point only works if the playlist exists 68 | if not os.path.exists(plPath): 69 | cfg.logger.error(f"Directory {plPath} Doesnt Exist") 70 | return False 71 | 72 | try: 73 | shelve.open(f'{plPath}/{cfg.metaDataName}', flag='r').close() 74 | except Exception as e: 75 | if noError: 76 | return False 77 | 78 | if e.args == (13, 'Permission denied') or e.__class__ == PermissionError: 79 | cfg.logger.error(f"Could Not Access Playlist at {plPath}, Permission Denined") 80 | else: 81 | cfg.logger.error(f"No Playlist Exists at {plPath}, Could not Find Metadata") 82 | return False 83 | #if not os.path.exists(f'{plPath}/{cfg.metaDataName}'): 84 | # cfg.logger.error(f"No Playlist Exists at {plPath}, Could not Find Metadata") 85 | # return False 86 | return True 87 | 88 | 89 | def getPlPath(playlist): 90 | cwd = getCwd() 91 | 92 | playlist = playlist.strip('/') 93 | if playlist == '.': 94 | return cwd 95 | 96 | return f"{cwd}/{playlist}" 97 | 98 | def getPlaylistPathsRecursive(path, plPaths): 99 | if playlistExists(path, noError = True): 100 | plPaths.append(path) 101 | return 102 | dir = os.listdir(path) 103 | for f in dir: 104 | fpath = f"{path}/{f}" 105 | if not os.path.isdir(fpath): 106 | continue 107 | getPlaylistPathsRecursive(fpath,plPaths) 108 | return 109 | 110 | def setupParsers(): 111 | description = ("A tool for downloading and syncing remote playlists to your computer. Created to avoid having\n" 112 | "music deleted but still have the convenience of browsing and adding and reordering new music using\n" 113 | "remote services such as youtube.") 114 | 115 | parser = argparse.ArgumentParser(description=description,formatter_class=ArgsOnce) 116 | parser.add_argument('-v','--verbose',action='store_true', help='runs application in verbose mode' ) 117 | parser.add_argument('-q','--quiet',action='store_true', help='runs application with no print outs' ) 118 | parser.add_argument('--version', action='version', version='%(prog)s ' + __version__) 119 | parser.set_defaults(func = lambda args: baseHandler(args, parser)) 120 | 121 | subparsers = parser.add_subparsers() 122 | timestamps = subparsers.add_parser('timestamps', help='detect, add and remove tracks from songs', formatter_class=ArgsOnce) 123 | timestamps.add_argument('-s', '--scrape', nargs=1, metavar='I', type=int, help='detect tracks in pinned/top comments for song index I') 124 | timestamps.add_argument('-r', '--scrape-range', nargs=2, metavar=('I1','I2'), type=int, help='detect tracks in pinned/top comments for song index I1 to I2') 125 | timestamps.add_argument('-a', '--auto-accept', action='store_true', help='automatically accept new timestamps') 126 | timestamps.add_argument('-o', '--overwrite', action='store_true', help='allow overwriting of existing timestamps, will be prompted') 127 | timestamps.add_argument('--auto-overwrite', action='store_true', help='allow overwriting of existing timestamps, will not be prompted to approve overwritng/accepting') 128 | 129 | timestamps.add_argument('PLAYLIST', type=str, help='the name of the directory for the playlist') 130 | timestamps.set_defaults(func = lambda args: timestampsHandler(args, timestamps)) 131 | 132 | new = subparsers.add_parser('new', help='downloads new playlist from URL with name PLAYLIST', formatter_class=ArgsOnce) 133 | new.add_argument('URL', type=str, help='playlist URL') 134 | new.add_argument('PLAYLIST', type=str, help='the name of the directory for the playlist') 135 | new.set_defaults(func = lambda args: newHandler(args, new)) 136 | 137 | sync = subparsers.add_parser("sync", help='smart sync playlist, unless options are added', formatter_class=ArgsOnce) 138 | sync.add_argument('-s','--smart-sync', action='store_true', help='smart sync local playlist with remote playlist') 139 | sync.add_argument('-a','--append-new', action='store_true', help='append new songs in remote playlist to end of local playlist') 140 | sync.add_argument('-r','--recursive', action='store_true', help='syncs all playlists in subfolders of PLAYLIST recursively (effected by config --local-dir)') 141 | sync.add_argument('PLAYLIST', type=str, help='the name of the directory for the playlist') 142 | sync.set_defaults(func = lambda args: syncHandler(args, sync)) 143 | 144 | 145 | edit = subparsers.add_parser("edit", help='change order of local playlist', formatter_class=ArgsOnce) 146 | edit.add_argument('-M','--manual-add',nargs=2, metavar=('PATH','INDEX'), type=str, help = 'manually add song at PATH to playlist in position INDEX') 147 | edit.add_argument('-m','--move',nargs=2, metavar=('I1','NI'), type = int, help='makes song index I1 come after NI (NI=-1 will move to start)') 148 | edit.add_argument('-r','--move-range',nargs=3, metavar=('I1','I2','NI'), type = int, help='makes songs in range [I1, I2] come after song index NI (NI=-1 will move to start)') 149 | edit.add_argument('-w','--swap',nargs=2, metavar=('I1','I2'), type = int, help='swaps order of songs index I1 and I2') 150 | edit.add_argument('-T','--toggle-prepends', action='store_true', help='toggles number prepends on/off') 151 | edit.add_argument('PLAYLIST', type=str, help='the name of the directory for the playlist') 152 | edit.set_defaults(func = lambda args: editHandler(args, edit)) 153 | 154 | #changing remote 155 | ytapi = subparsers.add_parser("ytapi", help='push local playlist order to youtube playlist', formatter_class=ArgsOnce) 156 | ytapi.add_argument('--logout', action='store_true', help='revokes youtube oauth access tokens and deletes them') 157 | ytapi.add_argument('--push-order', nargs=1, metavar='PLAYLIST', type=str, help='changes remote order of PLAYLIST to match local order (requires sign in with google)') 158 | ytapi.set_defaults(func = lambda args: ytapiHandler(args, ytapi)) 159 | 160 | #transfer 161 | ytapiSubParsers = ytapi.add_subparsers() 162 | transfer = ytapiSubParsers.add_parser("transfer", help="transfer songs between playlist on both local and remote", formatter_class=ArgsOnce) 163 | transfer.add_argument('-t','--transfer',nargs=2, metavar=('S1','DI'), type = int, help='makes SRC_PLAYLIST song index I1 come after DEST_PLAYLIST song index NI (NI=-1 will move to start)') 164 | transfer.add_argument('-r','--transfer-range',nargs=3, metavar=('S1','S2','DI'), type = int, help='makes SRC_PLAYLIST songs in range [I1, I2] come after DEST_PLAYLIST song index NI (NI=-1 will move to start)') 165 | transfer.add_argument('SRC_PLAYLIST', type=str, help='the name of the playlist to transfer songs from') 166 | transfer.add_argument('DEST_PLAYLIST', type=str, help='the name of the playlist to transfer songs to') 167 | transfer.set_defaults(func = lambda args: transferHandler(args, transfer)) 168 | 169 | 170 | config = subparsers.add_parser("config", help='change configuration (carries over between runs)', formatter_class=ArgsOnce) 171 | # the '\n' are used as defaults so they dont get confused with actual paths, '' will reset 172 | config.add_argument('-l','--local-dir',nargs='?',metavar='PATH',const='\n',type=str,help="sets music directory to PATH. if no PATH is provided, prints current music directory. to reset pass '' for PATH") 173 | config.add_argument('-f','--audio-format',nargs='?',metavar='FMT',const='\n',type=str,help='sets audio format to FMT (eg bestaudio, m4a, mp3, aac). if no FMT is provided, prints current FMT') 174 | config.add_argument('--list-formats',action='store_true', help='list all acceptable audio formats') 175 | config.add_argument('-t', '--toggle-timestamps', action='store_true', help='toggles automatic scraping of comments for timestamps when downloading') 176 | config.add_argument('-T', '--toggle-thumbnails', action='store_true', help='toggles embedding of thumbnails on download') 177 | config.add_argument('-s', '--show-config', action='store_true', help='shows current configuration') 178 | config.set_defaults(func= lambda args: configHandler(args, config)) 179 | 180 | #info 181 | info = subparsers.add_parser("info", help='get info about playlist', formatter_class=ArgsOnce) 182 | info.add_argument('-p','--print',action='store_true', help='shows song titles and urls for local playlist' ) 183 | info.add_argument('-d','--diff',action='store_true', help='shows differences between local and remote playlists' ) 184 | info.add_argument('--peek',nargs='?',metavar='FMT', const=cfg.defualtPeekFmt, type=str, help='prints remote PLAYLIST (url or name) without downloading, optional FMT string can contain: {id}, {url}, {title}, {duration}, {uploader}') 185 | info.add_argument('PLAYLIST',nargs='?', type=str, help='the name of the directory for the playlist') 186 | info.set_defaults(func = lambda args: infoHandler(args, info)) 187 | 188 | args = parser.parse_args() 189 | return args 190 | 191 | def baseHandler(_,parser): 192 | parser.print_help() 193 | 194 | def newHandler(args,_): 195 | plPath = getPlPath(args.PLAYLIST) 196 | if os.path.exists(plPath): 197 | cfg.logging.error(f"Cannot Make Playlist {args.PLAYLIST} Because Directory Already Exists at Path: \n{plPath}") 198 | return 199 | newPlaylist(plPath, args.URL) 200 | 201 | 202 | 203 | def syncHandler(args,parser): 204 | path = getPlPath(args.PLAYLIST) 205 | 206 | if args.recursive: 207 | plPaths = [] 208 | cfg.logger.info("Searching for Playlists...") 209 | getPlaylistPathsRecursive(path, plPaths) 210 | if len(plPaths) == 0: 211 | errMessage = f"No Playlists Found in: {path}" 212 | if cfg.musicDir: 213 | errMessage += f"\n(Note, Pathing is relative to {cfg.musicDir} use sync-dl config -l '' to change this)" 214 | cfg.logger.error(errMessage) 215 | return 216 | 217 | plPathsStr = '\n'.join(plPaths) 218 | answer = input(f"Playlists Found: \n{plPathsStr} \nContinue with Theses Playlists? (y)es/(n)o: ").lower().strip() 219 | if answer != 'y': 220 | return 221 | else: 222 | if not playlistExists(path): 223 | return 224 | plPaths = [path] 225 | 226 | 227 | for plPath in plPaths: 228 | if len(plPaths) > 1: 229 | cfg.logger.info("\n===============================================") 230 | 231 | #smart syncing 232 | if args.smart_sync: 233 | smartSync(plPath) 234 | 235 | #appending 236 | elif args.append_new: 237 | appendNew(plPath) 238 | 239 | else: 240 | parser.print_help() 241 | cfg.logger.error("Please Select an Option") 242 | break 243 | 244 | 245 | 246 | def configHandler(args,parser): 247 | if args.local_dir is not None: 248 | if args.local_dir == '\n': 249 | if cfg.musicDir=='': 250 | cfg.logger.error("Music Directory Not Set, Set With: sync-dl config -l PATH") 251 | else: 252 | cfg.logger.info(cfg.musicDir) 253 | return 254 | 255 | # reset musicDir to relative pathing 256 | if args.local_dir == '': 257 | cfg.writeToConfig('musicDir','') 258 | 259 | #set music dir to args.local_dir 260 | else: 261 | if not os.path.exists(args.local_dir): 262 | cfg.logger.error("Provided Music Directory Does not Exist") 263 | return 264 | #saves args.local_dir to config 265 | music = os.path.abspath(args.local_dir) 266 | 267 | cfg.writeToConfig('musicDir',music) 268 | 269 | if args.audio_format: 270 | if args.audio_format == '\n': 271 | cfg.logger.info(cfg.audioFormat) 272 | return 273 | fmt = args.audio_format.lower() 274 | if fmt not in cfg.knownFormats: 275 | cfg.logger.error(f"Unknown Format: {fmt}\nKnown Formats Are: {', '.join(cfg.knownFormats)}") 276 | return 277 | 278 | if fmt != 'best' and not cfg.testFfmpeg(): 279 | cfg.logger.error("ffmpeg is Required to Use Audio Format Other Than 'best'") 280 | return 281 | 282 | cfg.writeToConfig('audioFormat', fmt) 283 | cfg.setAudioFormat() 284 | cfg.logger.info(f"Audio Format Set to: {cfg.audioFormat}") 285 | 286 | if args.list_formats: 287 | cfg.logger.info(' '.join(cfg.knownFormats)) 288 | 289 | if args.toggle_thumbnails: 290 | if cfg.embedThumbnail: 291 | cfg.writeToConfig('embedThumbnail', '0') 292 | cfg.setEmbedThumbnails() 293 | else: 294 | cfg.writeToConfig('embedThumbnail', '1') 295 | cfg.setEmbedThumbnails() 296 | 297 | if cfg.embedThumbnail: 298 | cfg.logger.info("Embedding Thumbnails: ON") 299 | else: 300 | cfg.logger.info("Embedding Thumbnails: OFF") 301 | 302 | if args.toggle_timestamps: 303 | cfg.autoScrapeCommentTimestamps = not cfg.autoScrapeCommentTimestamps 304 | if cfg.autoScrapeCommentTimestamps: 305 | cfg.logger.info("Automatic Comment Timestamp Scraping: ON") 306 | else: 307 | cfg.logger.info("Automatic Comment Timestamp Scraping: OFF") 308 | 309 | cfg.writeToConfig('autoScrapeCommentTimestamps', str(int(cfg.autoScrapeCommentTimestamps))) 310 | 311 | if args.show_config: 312 | cfg.logger.info(f"(-l) (--local-dir): {cfg.musicDir}") 313 | 314 | cfg.logger.info(f"(-f) (--audio-format): {cfg.audioFormat}") 315 | 316 | if cfg.autoScrapeCommentTimestamps: 317 | cfg.logger.info("(-t) (--toggle-timestamps): ON") 318 | else: 319 | cfg.logger.info("(-t) (--toggle-timestamps): OFF") 320 | 321 | if cfg.embedThumbnail: 322 | cfg.logger.info("(-T) (--toggle-thumbnails): ON") 323 | else: 324 | cfg.logger.info("(-T) (--toggle-thumbnails): OFF") 325 | 326 | if not (args.toggle_timestamps or (args.local_dir is not None) or args.audio_format or args.list_formats or args.toggle_thumbnails or args.show_config): 327 | parser.print_help() 328 | cfg.logger.error("Please Select an Option") 329 | 330 | 331 | def editHandler(args,parser): 332 | 333 | plPath = getPlPath(args.PLAYLIST) 334 | if not playlistExists(plPath): 335 | return 336 | 337 | if args.move: 338 | moveRange(plPath,args.move[0],args.move[0],args.move[1]) 339 | 340 | elif args.move_range: 341 | moveRange(plPath,args.move_range[0],args.move_range[1],args.move_range[2]) 342 | 343 | elif args.swap: 344 | swap(plPath,args.swap[0],args.swap[1]) 345 | 346 | elif args.manual_add: 347 | if not args.manual_add[1].isdigit(): 348 | cfg.logger.error("Index must be positive Integer") 349 | else: 350 | manualAdd(plPath,args.manual_add[0],int(args.manual_add[1])) 351 | 352 | # if no options are selected, show help 353 | elif not args.toggle_prepends: 354 | parser.print_help() 355 | cfg.logger.error("Please Select an Option") 356 | 357 | if args.toggle_prepends: 358 | togglePrepends(plPath) 359 | 360 | 361 | def ytapiHandler(args,parser): 362 | if args.logout: 363 | logout() 364 | return 365 | 366 | if args.push_order: 367 | 368 | plPath = getPlPath(args.push_order[0]) 369 | if not playlistExists(plPath): 370 | return 371 | 372 | pushLocalOrder(plPath) 373 | 374 | else: 375 | parser.print_help() 376 | cfg.logger.error("Please Select an Option") 377 | 378 | def transferHandler(args, parser): 379 | if not args.SRC_PLAYLIST: 380 | cfg.logger.error("SRC_PLAYLIST required for transfer") 381 | 382 | if not args.DEST_PLAYLIST: 383 | cfg.logger.error("DEST_PLAYLIST required for transfer") 384 | 385 | srcPlPath = getPlPath(args.SRC_PLAYLIST) 386 | if not playlistExists(srcPlPath): 387 | return 388 | 389 | destPlPath = getPlPath(args.DEST_PLAYLIST) 390 | if not playlistExists(destPlPath): 391 | return 392 | 393 | if args.transfer: 394 | transferSongs(srcPlPath, destPlPath, args.transfer[0], args.transfer[0], args.transfer[1]) 395 | 396 | elif args.transfer_range: 397 | transferSongs(srcPlPath, destPlPath, args.transfer_range[0], args.transfer_range[1], args.transfer_range[2]) 398 | 399 | else: 400 | parser.print_help() 401 | cfg.logger.error("Please Select an Option") 402 | 403 | 404 | def infoHandler(args,parser): 405 | if args.peek: 406 | if not args.PLAYLIST: 407 | if args.peek != cfg.defualtPeekFmt: #removes the need to have posistional args before empty nargs='?' option 408 | url = args.peek 409 | fmt = cfg.defualtPeekFmt 410 | else: 411 | cfg.logger.error("Playlist URL Required") 412 | return 413 | else: 414 | url = args.PLAYLIST 415 | fmt = args.peek 416 | 417 | peek(url,fmt) 418 | return 419 | 420 | # if no playlist was provided all further functions cannot run 421 | if not args.PLAYLIST: 422 | cfg.logger.error("Playlist Name Required") 423 | return 424 | 425 | plPath = getPlPath(args.PLAYLIST) 426 | if not playlistExists(plPath): 427 | return 428 | 429 | #viewing playlist 430 | if args.print: 431 | showPlaylist(plPath) 432 | 433 | if args.diff: 434 | compareMetaData(plPath) 435 | 436 | if (not args.print) and (not args.diff): 437 | parser.print_help() 438 | cfg.logger.error("Please Select an Option") 439 | 440 | def timestampsHandler(args,parser): 441 | plPath = getPlPath(args.PLAYLIST) 442 | if not playlistExists(plPath): 443 | return 444 | 445 | autoAccept = args.auto_accept 446 | overwrite = args.overwrite 447 | autoOverwrite = args.auto_overwrite 448 | 449 | if autoOverwrite: 450 | autoAccept = True 451 | overwrite = True 452 | 453 | if args.scrape: 454 | addTimestampsFromComments(plPath, args.scrape[0], args.scrape[0], autoAccept=autoAccept, overwrite=overwrite, autoOverwrite=autoOverwrite) 455 | 456 | elif args.scrape_range: 457 | addTimestampsFromComments(plPath, args.scrape_range[0], args.scrape_range[1], autoAccept=autoAccept, overwrite=overwrite, autoOverwrite=autoOverwrite) 458 | 459 | else: 460 | parser.print_help() 461 | cfg.logger.error("Please Select an Option") 462 | 463 | 464 | def checkAllStateCorruption(args): 465 | 466 | plPaths = [] 467 | if 'PLAYLIST' in vars(args) and args.PLAYLIST: 468 | try: 469 | plPaths.append(getPlPath(args.PLAYLIST)) 470 | except: 471 | pass 472 | 473 | if 'SRC_PLAYLIST' in vars(args) and args.SRC_PLAYLIST: 474 | try: 475 | plPaths.append(getPlPath(args.SRC_PLAYLIST)) 476 | except: 477 | pass 478 | 479 | if 'DEST_PLAYLIST' in vars(args) and args.DEST_PLAYLIST: 480 | try: 481 | plPaths.append(getPlPath(args.DEST_PLAYLIST)) 482 | except: 483 | pass 484 | 485 | for plPath in plPaths: 486 | metaDataPath = f"{plPath}/{cfg.metaDataName}" 487 | if os.path.exists(metaDataPath): 488 | with shelve.open(metaDataPath, 'c',writeback=True) as metaData: 489 | correctStateCorruption(plPath,metaData) 490 | cfg.logger.info(f"State Recovered For Playlist: {plPath}") 491 | 492 | def cli(): 493 | ''' 494 | Runs command line application, talking in args and running commands 495 | ''' 496 | args = setupParsers() 497 | 498 | setupLogger(args) 499 | 500 | try: 501 | args.func(args) 502 | 503 | except InterruptTriggered as e: 504 | checkAllStateCorruption(args) 505 | 506 | except Exception as e: 507 | cfg.logger.exception(e) 508 | checkAllStateCorruption(args) 509 | 510 | -------------------------------------------------------------------------------- /sync_dl/commands.py: -------------------------------------------------------------------------------- 1 | import os 2 | 3 | import shelve 4 | import re 5 | import ntpath 6 | import shutil 7 | 8 | from sync_dl import noInterrupt 9 | 10 | from sync_dl.ytdlWrappers import getIDs, getIdsAndTitles,getJsonPlData 11 | from sync_dl.plManagement import editPlaylist, correctStateCorruption, removePrepend 12 | from sync_dl.helpers import createNumLabel, smartSyncNewOrder, getLocalSongs, rename, relabel,download,getNumDigets, numOccurance, getNthOccuranceIndex, getOrdinalIndicator, padZeros 13 | 14 | from sync_dl.timestamps.scraping import scrapeCommentsForTimestamps 15 | from sync_dl.timestamps import createChapterFile, addTimestampsToChapterFile, applyChapterFileToSong, wipeChapterFile, addTimestampsIfNoneExist 16 | 17 | import sync_dl.config as cfg 18 | 19 | 20 | 21 | def newPlaylist(plPath,url): 22 | dirMade = False # if sync-dl cant download any songs, it will delete the directory only if it made it 23 | if not os.path.exists(plPath): 24 | dirMade = True 25 | os.makedirs(plPath) 26 | 27 | elif len(os.listdir(path=plPath))!=0: 28 | cfg.logger.error(f"Directory Exists and is Not Empty, Cannot Make Playlist in {plPath}") 29 | return 30 | 31 | 32 | cfg.logger.info(f"Creating New Playlist Named {ntpath.basename(plPath)} from URL: {url}") 33 | 34 | ids = getIDs(url) 35 | idsLen = len(ids) 36 | if idsLen == 0: 37 | cfg.logger.error(f"No Songs Found at {url}\nMake Sure the Playlist is Public!") 38 | answer = input("Proceed with Creating Empty Playlist (y)es/(n)o: ").lower().strip() 39 | if answer != 'y': 40 | os.rmdir(plPath) 41 | return 42 | 43 | numDigits = getNumDigets(idsLen) #needed for creating starting number for auto ordering ie) 001, 0152 44 | 45 | with shelve.open(f"{plPath}/{cfg.metaDataName}", 'c',writeback=True) as metaData: 46 | metaData["url"] = url 47 | metaData["ids"] = [] 48 | 49 | invalidSongs = 0 50 | for i,songId in enumerate(ids): 51 | 52 | counter = f'{i+1}/{idsLen}' 53 | songName = download(metaData,plPath,songId,-1,numDigits,counter) 54 | if not songName: 55 | invalidSongs+=1 56 | continue 57 | if cfg.autoScrapeCommentTimestamps: 58 | addTimestampsIfNoneExist(plPath, songName, songId) 59 | 60 | 61 | cfg.logger.info(f"Downloaded {idsLen-invalidSongs}/{idsLen} Songs") 62 | 63 | if (idsLen-invalidSongs == 0) and dirMade and idsLen!=0: 64 | cfg.logger.error(f"Unable to Download any Songs from {url}") 65 | answer = input("Proceed with Creating Empty Playlist (y)es/(n)o: ").lower().strip() 66 | if answer != 'y': 67 | shutil.rmtree(plPath) 68 | 69 | 70 | def smartSync(plPath): 71 | ''' 72 | Syncs to remote playlist however will Not delete local songs (will reorder). Songs not in remote (ie ones deleted) 73 | will be after the song they are currently after in local 74 | Example 1 75 | Local order: A B C D 76 | Remote order: A 1 B C 2 77 | 78 | Local becomes: A 1 B C D 2 79 | 80 | notice D was removed from remote but is still after C in Local 81 | 82 | Example 2 83 | Local order: A B C D 84 | Remote order: A 1 C B 2 85 | 86 | Local becomes: A 1 C D B 2 87 | 88 | notice C and B where swapped and D was deleted from Remote 89 | 90 | see test_smartSyncNewOrder in tests.py for more examples 91 | ''' 92 | cfg.logger.info(f"Smart Syncing {plPath}") 93 | 94 | 95 | 96 | with shelve.open(f"{plPath}/{cfg.metaDataName}", 'c',writeback=True) as metaData: 97 | correctStateCorruption(plPath,metaData) 98 | url = metaData["url"] 99 | localIds = metaData["ids"] 100 | 101 | remoteIds = getIDs(url) 102 | 103 | 104 | newOrder = smartSyncNewOrder(localIds,remoteIds) 105 | 106 | editPlaylist(plPath,newOrder) 107 | 108 | 109 | def appendNew(plPath): 110 | '''will append new songs in remote playlist to local playlist in order that they appear''' 111 | 112 | cfg.logger.info(f"Appending New Songs to {plPath}") 113 | 114 | 115 | with shelve.open(f"{plPath}/{cfg.metaDataName}", 'c',writeback=True) as metaData: 116 | correctStateCorruption(plPath,metaData) 117 | 118 | idsLen = len(metaData["ids"]) 119 | numDigits = getNumDigets(idsLen) 120 | 121 | remoteIds = getIDs(metaData['url']) 122 | 123 | for remoteId in remoteIds: 124 | if remoteId not in metaData['ids']: 125 | songName = download(metaData,plPath,remoteId,idsLen,numDigits) 126 | if songName and cfg.autoScrapeCommentTimestamps: 127 | addTimestampsIfNoneExist(plPath, songName, remoteId) 128 | 129 | 130 | 131 | def manualAdd(plPath, songPath, posistion): 132 | '''put song in posistion in the playlist''' 133 | 134 | if not os.path.exists(songPath): 135 | cfg.logger.error(f'{songPath} Does Not Exist') 136 | return 137 | 138 | with shelve.open(f"{plPath}/{cfg.metaDataName}", 'c',writeback=True) as metaData: 139 | correctStateCorruption(plPath,metaData) 140 | 141 | currentDir = getLocalSongs(plPath) 142 | 143 | idsLen = len(metaData["ids"]) 144 | numDigits = getNumDigets(idsLen) 145 | 146 | #clamp posistion 147 | if posistion > idsLen: 148 | posistion = idsLen 149 | elif posistion < 0: 150 | posistion = 0 151 | 152 | cfg.logger.info(f"Adding {ntpath.basename(songPath)} to {ntpath.basename(plPath)} in Posistion {posistion}") 153 | 154 | #shifting elements 155 | for i in reversed(range(posistion, idsLen)): 156 | oldName = currentDir[i] 157 | 158 | newName = re.sub(cfg.filePrependRE, f"{createNumLabel(i+1,numDigits)}_" , oldName) 159 | 160 | with noInterrupt: 161 | rename(metaData,cfg.logger.debug,plPath,oldName,newName,i+1,metaData["ids"][i]) 162 | 163 | metaData["ids"][i] = '' #wiped in case of crash, this blank entries can be removed restoring state 164 | 165 | 166 | newSongName = f"{createNumLabel(posistion,numDigits)}_" + ntpath.basename(songPath) 167 | 168 | with noInterrupt: 169 | os.rename(songPath,f'{plPath}/{newSongName}') 170 | 171 | if posistion >= len(metaData["ids"]): 172 | metaData["ids"].append(cfg.manualAddId) 173 | else: 174 | metaData["ids"][posistion] = cfg.manualAddId 175 | 176 | def swap(plPath, index1, index2): 177 | '''moves song to provided posistion, shifting all below it down''' 178 | if index1 == index2: 179 | cfg.logger.info(f"Given Index are the Same") 180 | 181 | 182 | with shelve.open(f"{plPath}/{cfg.metaDataName}", 'c',writeback=True) as metaData: 183 | correctStateCorruption(plPath,metaData) 184 | 185 | currentDir = getLocalSongs(plPath) 186 | 187 | idsLen = len(metaData["ids"]) 188 | numDigits = getNumDigets(idsLen) 189 | 190 | if index1>=idsLen or index2>=idsLen: 191 | cfg.logger.error(f"Given Index is Larger than Max {idsLen-1}") 192 | return 193 | elif index1<0 or index2<0: 194 | cfg.logger.error(f"Given Index is Negative") 195 | return 196 | 197 | cfg.logger.info(f"Swapping: \n{currentDir[index1]} \nand \n{currentDir[index2]}") 198 | #shift index1 out of the way (to idsLen) 199 | 200 | oldName = currentDir[index1] 201 | tempName = relabel(metaData,cfg.logger.debug,plPath,oldName,index1,idsLen,numDigits) 202 | 203 | 204 | #move index2 to index1's old location 205 | 206 | oldName = currentDir[index2] 207 | relabel(metaData,cfg.logger.debug,plPath,oldName,index2,index1,numDigits) 208 | 209 | #move index1 (now =idsLen) to index2's old location 210 | 211 | oldName = tempName 212 | relabel(metaData,cfg.logger.debug,plPath,oldName,idsLen,index2,numDigits) 213 | 214 | del metaData["ids"][idsLen] 215 | 216 | 217 | # move is no longer used in CLI, functionality has been merged with moveRange to be more consistant 218 | # move is still used in integration tests to reorder playlists before testing smart sync 219 | def move(plPath, currentIndex, newIndex): 220 | if currentIndex==newIndex: 221 | cfg.logger.info("Indexes Are the Same") 222 | return 223 | 224 | 225 | with shelve.open(f"{plPath}/{cfg.metaDataName}", 'c',writeback=True) as metaData: 226 | correctStateCorruption(plPath,metaData) 227 | 228 | currentDir = getLocalSongs(plPath) 229 | 230 | idsLen = len(metaData["ids"]) 231 | numDigits = getNumDigets(idsLen) 232 | 233 | 234 | if currentIndex>=idsLen: 235 | cfg.logger.error(f"No song has Index {currentIndex}, Largest Index is {idsLen-1}") 236 | return 237 | elif currentIndex<0: 238 | cfg.logger.error(f"No Song has a Negative Index") 239 | return 240 | 241 | #clamp newIndex 242 | if newIndex >= idsLen: 243 | newIndex = idsLen-1 244 | elif newIndex < 0: 245 | newIndex = 0 246 | 247 | cfg.logger.info(f"Moving {currentDir[currentIndex]} to Index {newIndex}") 248 | 249 | #moves song to end of list 250 | tempName = relabel(metaData,cfg.logger.debug,plPath,currentDir[currentIndex],currentIndex,idsLen,numDigits) 251 | 252 | if currentIndex>newIndex: 253 | #shifts all songs from newIndex to currentIndex-1 by +1 254 | for i in reversed(range(newIndex,currentIndex)): 255 | 256 | oldName = currentDir[i] 257 | relabel(metaData,cfg.logger.debug,plPath,oldName,i,i+1,numDigits) 258 | 259 | 260 | else: 261 | #shifts all songs from currentIndex+1 to newIndex by -1 262 | for i in range(currentIndex+1,newIndex+1): 263 | oldName = currentDir[i] 264 | relabel(metaData,cfg.logger.debug,plPath,oldName,i,i-1,numDigits) 265 | 266 | #moves song back 267 | relabel(metaData,cfg.logger.debug,plPath,tempName,idsLen,newIndex,numDigits) 268 | del metaData['ids'][idsLen] 269 | 270 | 271 | def moveRange(plPath, start, end, newStart): 272 | ''' 273 | moves block of songs from start to end indices, to after newStart 274 | ie) start = 4, end = 6, newStart = 2 275 | 0 1 2 3 4 5 6 7 -> 0 1 2 4 5 6 3 276 | ''' 277 | 278 | if start == newStart: 279 | return 280 | 281 | 282 | with shelve.open(f"{plPath}/{cfg.metaDataName}", 'c',writeback=True) as metaData: 283 | correctStateCorruption(plPath,metaData) 284 | 285 | currentDir = getLocalSongs(plPath) 286 | 287 | idsLen = len(metaData["ids"]) 288 | numDigits = getNumDigets(idsLen) 289 | 290 | 291 | if start>=idsLen: 292 | cfg.logger.error(f"No song has Index {start}, Largest Index is {idsLen-1}") 293 | return 294 | 295 | elif start<0: 296 | cfg.logger.error(f"No Song has a Negative Index") 297 | return 298 | 299 | #clamp end index 300 | if end>=idsLen or end == -1: 301 | end = idsLen-1 302 | 303 | elif end= idsLen: 309 | newStart = idsLen - 1 310 | elif newStart < -1: 311 | newStart = -1 312 | 313 | # Sanatization over 314 | 315 | # number of elements to move 316 | blockSize = end-start+1 317 | 318 | # make room for block 319 | for i in reversed(range(newStart+1,idsLen)): 320 | oldName = currentDir[i] 321 | newIndex =i+blockSize 322 | relabel(metaData,cfg.logger.debug,plPath,oldName,i,newIndex,numDigits) 323 | 324 | 325 | #accounts for block of songs being shifted if start>newStart 326 | offset = 0 327 | if start>newStart: 328 | currentDir = getLocalSongs(plPath) 329 | offset = blockSize 330 | 331 | # shift block into gap made 332 | for i,oldIndex in enumerate(range(start,end+1)): 333 | 334 | oldName = currentDir[oldIndex] 335 | 336 | newIndex = i + newStart+1 337 | relabel(metaData,cfg.logger.debug,plPath,oldName,oldIndex+offset,newIndex,numDigits) 338 | 339 | # remove number gap in playlist and remove blanks in metadata 340 | correctStateCorruption(plPath,metaData) 341 | 342 | # logged changes 343 | startSong = re.sub(cfg.filePrependRE,"",currentDir[start]) 344 | endSong = re.sub(cfg.filePrependRE,"",currentDir[end]) 345 | leaderSong = re.sub(cfg.filePrependRE,"",currentDir[newStart]) # the name of the song the range will come after 346 | 347 | ######## Single song moved ########## 348 | if start==end: 349 | cfg.logger.info(f"Song {start}: {startSong}") 350 | if newStart == -1: 351 | cfg.logger.info(f"Is Now First in The Playlist") 352 | else: 353 | cfg.logger.info(f"Is Now After Song {newStart}: {leaderSong}") 354 | 355 | return 356 | ##################################### 357 | 358 | 359 | ####### Multiple Songs Moved ######## 360 | cfg.logger.info(f"Moved Songs in Range [{start}, {end}] to After {newStart}") 361 | 362 | cfg.logger.info(f"Start Range: {startSong}") 363 | cfg.logger.info(f"End Range: {endSong}") 364 | 365 | if newStart != -1: 366 | leaderSong = re.sub(cfg.filePrependRE,"",currentDir[newStart]) # the name of the song the range will come after 367 | cfg.logger.info(f"Are Now After: {leaderSong}") 368 | else: 369 | cfg.logger.info(f"Are Now First in the Playlist") 370 | 371 | ###################################### 372 | 373 | 374 | 375 | # TODO not yet added to CLI (doesnt seem useful) 376 | def shuffle(plPath): 377 | '''randomizes playlist order''' 378 | from random import randint 379 | 380 | cfg.logger.info("Shuffling Playlist") 381 | 382 | 383 | with shelve.open(f"{plPath}/{cfg.metaDataName}", 'c',writeback=True) as metaData: 384 | correctStateCorruption(plPath,metaData) 385 | 386 | plLen = len(metaData["ids"]) 387 | ids = metaData["ids"] 388 | 389 | avalibleNums = [i for i in range(plLen)] 390 | newOrder = [] 391 | for _ in range(plLen): 392 | oldIndex = avalibleNums.pop(randint(0,len(avalibleNums)-1)) 393 | newOrder.append( (ids[oldIndex],oldIndex) ) 394 | 395 | editPlaylist(plPath, newOrder) 396 | 397 | 398 | def showPlaylist(plPath, lineBreak='', urlWithoutId = "https://www.youtube.com/watch?v="): 399 | ''' 400 | lineBreak can be set to newline if you wish to format for small screens 401 | urlWithoutId is added if you wish to print out all full urls 402 | ''' 403 | with shelve.open(f"{plPath}/{cfg.metaDataName}", 'c',writeback=True) as metaData: 404 | cfg.logger.info(f"Playlist URL: {metaData['url']}") 405 | 406 | correctStateCorruption(plPath,metaData) 407 | 408 | currentDir = getLocalSongs(plPath) 409 | 410 | maxNum = len(currentDir) 411 | numDigits = len(str(maxNum)) 412 | 413 | if urlWithoutId != None: 414 | spacer=' '*(len(urlWithoutId)+10) 415 | 416 | cfg.logger.info(f"{' '*numDigits}: URL{spacer}{lineBreak}-> Local Title{lineBreak}") 417 | for i,songId in enumerate(metaData['ids']): 418 | url = f"{urlWithoutId}{songId}" 419 | title = re.sub(cfg.filePrependRE, '' , currentDir[i]) 420 | cfg.logger.info(f"{str(i).zfill(numDigits)}: {url}{lineBreak} -> {title}{lineBreak}") 421 | 422 | 423 | 424 | def compareMetaData(plPath): 425 | '''Tool for comparing ids held in metadata and their order compared to remote playlist ids''' 426 | with shelve.open(f"{plPath}/{cfg.metaDataName}", 'c',writeback=True) as metaData: 427 | correctStateCorruption(plPath, metaData) 428 | 429 | remoteIds, remoteTitles = getIdsAndTitles(metaData["url"]) 430 | localIds = metaData["ids"] 431 | assert isinstance(localIds, list) 432 | currentDir = getLocalSongs(plPath) 433 | 434 | maxNum = max(len(localIds), len(remoteIds)) 435 | numDigits = len(str(maxNum)) 436 | 437 | 438 | cfg.logger.info(f"\n==================[Playlist Data]==================") 439 | cfg.logger.info(f"{' '*numDigits}: Local ID -> {' '*numDigits}: Remote ID : Title") 440 | 441 | inLocalNotRemote = [] 442 | inRemoteNotLocal = [(i,numOccurance(remoteIds, i),remoteId,remoteTitles[i]) for i,remoteId in enumerate(remoteIds)] 443 | 444 | for i,localId in enumerate(localIds): 445 | localOccuranceNum = numOccurance(localIds, i) 446 | 447 | title = re.sub(cfg.filePrependRE, '' , currentDir[i]) 448 | remoteIndex = getNthOccuranceIndex(remoteIds, localId, localOccuranceNum) 449 | if remoteIndex is not None: 450 | cfg.logger.info(f"{str(i).zfill(numDigits)}: {localId} -> {str(remoteIndex).zfill(numDigits)}: {localId} : {title}") 451 | inRemoteNotLocal[remoteIndex] = None 452 | 453 | else: 454 | cfg.logger.info(f"{str(i).zfill(numDigits)}: {localId} -> {' '*numDigits}: : {title}") 455 | inLocalNotRemote.append((i,localOccuranceNum,localId,title)) 456 | 457 | 458 | j=0 459 | while j {str(x).zfill(numDigits)}: {remoteId} : {title}") 466 | j+=1 467 | 468 | # summery 469 | cfg.logger.info(f"\n=====================[Summary]=====================") 470 | 471 | if len(inLocalNotRemote)>0: 472 | cfg.logger.info(f"\n------------[In Local But Not In Remote]-----------") 473 | cfg.logger.info(f"{' '*numDigits}: Local ID : Local Title") 474 | for i, localOccuranceNum, localId, title in inLocalNotRemote: 475 | occuranceText = '' if localOccuranceNum == 0 else f' : {str(localOccuranceNum+1)}{getOrdinalIndicator(localOccuranceNum+1)} Occurance' 476 | cfg.logger.info(f"{str(i).zfill(numDigits)}: {localId} : {title}{occuranceText}") 477 | 478 | if len(inRemoteNotLocal)>0: 479 | cfg.logger.info(f"\n------------[In Remote But Not In Local]-----------") 480 | cfg.logger.info(f"{' '*numDigits}: Remote ID : Remote Title") 481 | for j, remoteOccuranceNum, remoteId, title in inRemoteNotLocal: 482 | occuranceText = '' if remoteOccuranceNum== 0 else f' : {str(remoteOccuranceNum+1)}{getOrdinalIndicator(remoteOccuranceNum+1)} Occurance' 483 | cfg.logger.info(f"{str(j).zfill(numDigits)}: {remoteId} : {title}{occuranceText}") 484 | 485 | if len(inLocalNotRemote) == 0 and len(inRemoteNotLocal) == 0: 486 | cfg.logger.info(f"Local And Remote Contain The Same Songs") 487 | 488 | 489 | 490 | def peek(urlOrPlName,fmt="{index}: {url} {title}"): 491 | ''' 492 | prints out data about the playlist without downloading it, fmt parameters include: 493 | - id 494 | - url 495 | - title 496 | - duration 497 | - view_count (currently bugged in youtube-dl, will always be none) 498 | - uploader (soon to be fixed in youtube-dl) 499 | ''' 500 | 501 | # check if urlOrPlName is playlist name 502 | if not cfg.musicDir: 503 | musicPath = os.getcwd() 504 | else: 505 | musicPath = cfg.musicDir 506 | 507 | musicDir = os.listdir(path= musicPath) 508 | 509 | if urlOrPlName in musicDir: 510 | with shelve.open(f"{musicPath}/{urlOrPlName}/{cfg.metaDataName}", 'c',writeback=True) as metaData: 511 | urlOrPlName = metaData['url'] 512 | 513 | 514 | # at this point urlOrPlName is a url 515 | plData = getJsonPlData(urlOrPlName) 516 | 517 | for i,songData in enumerate(plData): 518 | songStr = fmt.format(index = i,**songData) 519 | cfg.logger.info(songStr) 520 | 521 | 522 | def togglePrepends(plPath): 523 | with shelve.open(f"{plPath}/{cfg.metaDataName}", 'c',writeback=True) as metaData: 524 | if "removePrependOrder" in metaData: 525 | # prepends where removed, hence we must add them 526 | correctStateCorruption(plPath,metaData) # part of correcting state corruption is re-adding prepends 527 | cfg.logger.info("Prepends Added") 528 | return 529 | removePrepend(plPath,metaData) 530 | cfg.logger.info("Prepends Removed") 531 | 532 | 533 | def addTimestampsFromComments(plPath, start, end, autoAccept = False, overwrite = False, autoOverwrite = False): 534 | 535 | with shelve.open(f"{plPath}/{cfg.metaDataName}", 'c',writeback=True) as metaData: 536 | correctStateCorruption(plPath,metaData) 537 | 538 | currentDir = getLocalSongs(plPath) 539 | idsLen = len(metaData["ids"]) 540 | 541 | ### Sanitize Inputs ### 542 | if start >= idsLen: 543 | cfg.logger.error(f"No song has Index {start}, Largest Index is {idsLen-1}") 544 | return 545 | 546 | elif start < 0: 547 | cfg.logger.error(f"No Song has a Negative Index") 548 | return 549 | 550 | #clamp end index 551 | if end >= idsLen or end == -1: 552 | end = idsLen-1 553 | 554 | elif end < start: 555 | cfg.logger.error("End Index Must be Greater Than or Equal To Start Index (or -1)") 556 | return 557 | ### Sanitize Inputs Over ### 558 | 559 | 560 | multipleSongs = start!=end 561 | for i in range(start,end+1): 562 | 563 | if autoOverwrite and not overwrite: 564 | cfg.logger.error("auto-overwrite can only be enabled if overwrite is") 565 | return 566 | 567 | if autoOverwrite and not autoAccept: 568 | cfg.logger.error("auto-overwrite can only be enabled if auto-accept is") 569 | return 570 | 571 | songName = currentDir[i] 572 | songPath = f"{plPath}/{songName}" 573 | videoId = metaData['ids'][i] 574 | 575 | if not createChapterFile(songPath, songName): 576 | continue 577 | 578 | # Check for existing chapters in ffmpegMetadata file, wipe them, and prompt user to continue 579 | existingTimestamps = wipeChapterFile() 580 | if len(existingTimestamps) > 0: 581 | if not overwrite: 582 | cfg.logger.info(f"{i}: Existing Timestamps Detected on Song: {songName}\nSkipping...\n") 583 | continue 584 | 585 | # Get timestamps 586 | cfg.logger.info(f"{i}: Scraping Comments for Timestamps of Song: {songName}") 587 | timestamps = scrapeCommentsForTimestamps(videoId) 588 | 589 | if len(timestamps) == 0: 590 | cfg.logger.info(f"No Timestamps Found\n") 591 | continue 592 | 593 | if existingTimestamps == timestamps: 594 | cfg.logger.info("Timestamps Found are Idential to Existing Timestamps\n") 595 | continue 596 | 597 | if len(existingTimestamps) > 0: 598 | if not overwrite: 599 | cfg.logger.info("Comment Timestamps and Existing Timestamps Found but Overwrite is Disabled\n") 600 | continue 601 | 602 | # existing timestamps, plus timestamps found 603 | cfg.logger.info(f"\nExisting Timestamps Found:") 604 | numDigitsExistingTimestamps = len(str(len(existingTimestamps))) 605 | numDigitsCommentTimestamps = len(str(len(timestamps))) 606 | for i,timestamp in enumerate(existingTimestamps): 607 | leadChar = ' ' 608 | if timestamp not in timestamps: 609 | leadChar = '-' 610 | cfg.logger.info(f"{leadChar} {padZeros(i, numDigitsExistingTimestamps)}) {timestamp}") 611 | cfg.logger.info(f"\nComment Timestamps Found:") 612 | for i,timestamp in enumerate(timestamps): 613 | leadChar = ' ' 614 | if timestamp not in existingTimestamps: 615 | leadChar = '+' 616 | cfg.logger.info(f"{leadChar} {padZeros(i, numDigitsCommentTimestamps)}) {timestamp}") 617 | 618 | cfg.logger.info('\n') 619 | 620 | if autoOverwrite and autoAccept: 621 | cfg.logger.info("Auto Overwriting\n") 622 | 623 | else: 624 | cfg.logger.info("Existing and Comment Timestamps Found") 625 | response = (input(f"OVERWRITE Timestamps for: {songName}? \n[y]es, [n]o" + (", [a]uto-overwrite/accepts, [d]eny-overwrites" if multipleSongs else "") + ":")).lower() 626 | if response == 'a': 627 | autoAccept = True 628 | overwrite = True 629 | autoOverwrite = True 630 | elif response == 'd': 631 | overwrite = False 632 | continue 633 | elif response != 'y' and response != 'a': 634 | cfg.logger.info('\n') 635 | continue 636 | 637 | else: 638 | # comment timestamps found, no existing timestamps found 639 | cfg.logger.info(f"\nComment Timestamps Found:") 640 | for timestamp in timestamps: 641 | cfg.logger.info(timestamp) 642 | cfg.logger.info('\n') 643 | 644 | 645 | if not autoAccept: 646 | response = (input(f"\nAccept Timestamps for: {songName}? \n[y]es, [n]o" + (", [a]uto-accept" if multipleSongs else "") + ":")).lower() 647 | if response == 'a': 648 | autoAccept = True 649 | if response != 'y' and response != 'a': 650 | cfg.logger.info('\n') 651 | continue 652 | else: 653 | cfg.logger.info("Auto Accepting Timestamps") 654 | 655 | addTimestampsToChapterFile(timestamps, songPath) 656 | 657 | if not applyChapterFileToSong(songPath, songName): 658 | cfg.logger.error(f"Failed to Add Timestamps To Song {songName}\n") 659 | continue 660 | 661 | cfg.logger.info("Timestamps Applied!\n") 662 | 663 | 664 | 665 | -------------------------------------------------------------------------------- /sync_dl/config/__init__.py: -------------------------------------------------------------------------------- 1 | from re import compile 2 | from yt_dlp.postprocessor import FFmpegExtractAudioPP 3 | 4 | from sync_dl.config.tmpdir import createTmpDir, clearTmpDir, clearTmpSubPath, \ 5 | tmpDownloadPath, songSegmentsPath, thumbnailPath, songDownloadPath, ffmpegMetadataPath, songEditPath 6 | 7 | from sync_dl.config.parsing import modulePath, writeToConfig, readConfig 8 | 9 | from sync_dl.config.ytdlParams import setAudioFormat, setEmbedThumbnails, dlParams 10 | 11 | ''' 12 | contains all global variables, also parses config into global variables 13 | ''' 14 | 15 | # global config variables 16 | filePrependRE = compile(r'\d+_') 17 | plIdRe = compile(r'list=.{34}') 18 | 19 | knownFormats = (*FFmpegExtractAudioPP.SUPPORTED_EXTS, 'best') 20 | 21 | metaDataName = readConfig('metaDataName') 22 | manualAddId = readConfig('manualAddId') 23 | testPlPath = f"{modulePath}/{readConfig('testPlPath')}" 24 | testSongName = "test.mp3" 25 | testSongPath = f"{modulePath}/tests/{testSongName}" 26 | musicDir = readConfig('musicDir') 27 | autoScrapeCommentTimestamps = readConfig('autoScrapeCommentTimestamps', boolean=True) 28 | audioFormat = readConfig('audioFormat') 29 | embedThumbnail = readConfig('embedThumbnail', boolean=True) 30 | 31 | 32 | #TODO move add to ini 33 | defualtPeekFmt='{index}: {url} {title}' 34 | 35 | # logger 36 | import logging 37 | logger = logging.getLogger('sync_dl') 38 | 39 | 40 | -------------------------------------------------------------------------------- /sync_dl/config/parsing.py: -------------------------------------------------------------------------------- 1 | import configparser 2 | from ntpath import dirname 3 | import os 4 | 5 | modulePath = dirname(dirname(__file__)) 6 | 7 | _defaultConfig = { 8 | 'metaDataName' : '.metaData', 9 | 'manualAddId' : '-manualAddition', 10 | 'testPlPath' : 'tests/testPlaylists', 11 | 'musicDir' : '', 12 | 'autoScrapeCommentTimestamps': '0', 13 | 'audioFormat': 'best', 14 | 'embedThumbnail': '0' 15 | } 16 | 17 | #loading config 18 | _parser = configparser.ConfigParser(allow_no_value=True) 19 | _parser.optionxform = str 20 | 21 | 22 | def writeToConfig(key,value): 23 | global _parser 24 | global _config 25 | _parser.set('CONFIG',key,value) 26 | with open(f'{modulePath}/config.ini', 'w') as configfile: 27 | _parser.write(configfile) 28 | 29 | _config[key] = value 30 | 31 | def readConfig(key, boolean = False): 32 | global _config 33 | if boolean: 34 | return _config.getboolean(key) 35 | return _config[key] 36 | 37 | def checkConfig(): 38 | global _defaultConfig 39 | global _config 40 | cfgPath = f'{modulePath}/config.ini' 41 | 42 | if not os.path.exists(cfgPath): 43 | _parser['CONFIG'] = _defaultConfig 44 | with open(cfgPath, 'w+') as f: 45 | _parser.write(f) 46 | else: 47 | _parser.read(cfgPath) 48 | _config = _parser['CONFIG'] 49 | 50 | for key in _defaultConfig.keys(): 51 | if not key in _config.keys(): 52 | writeToConfig(key, _defaultConfig[key]) 53 | 54 | checkConfig() 55 | -------------------------------------------------------------------------------- /sync_dl/config/tmpdir.py: -------------------------------------------------------------------------------- 1 | import shutil 2 | from tempfile import TemporaryDirectory 3 | import os 4 | 5 | 6 | def _clearSubPath(path): 7 | try: 8 | if os.path.isfile(path) or os.path.islink(path): 9 | os.unlink(path) 10 | return 11 | for f in os.listdir(path=path): 12 | p = f"{path}/{f}" 13 | if os.path.isfile(p) or os.path.islink(p): 14 | os.unlink(p) 15 | elif os.path.isdir(p): 16 | shutil.rmtree(p) 17 | except OSError: 18 | pass 19 | #cfg.logger.error(f"Error Clearing Directory: {dir}") 20 | #cfg.logger.debug(f"Reason: {e}") 21 | 22 | # tmp paths 23 | _tmpDir = None 24 | tmpDownloadPath = '' #top level 25 | 26 | # dirs 27 | songSegmentsPath = '' 28 | thumbnailPath = '' 29 | songDownloadPath = '' 30 | songEditPath = '' 31 | 32 | # files 33 | ffmpegMetadataPath = '' 34 | 35 | def _createSubDirs(): 36 | os.mkdir(songSegmentsPath) 37 | os.mkdir(thumbnailPath) 38 | os.mkdir(songDownloadPath) 39 | os.mkdir(songEditPath) 40 | 41 | def createTmpDir(): 42 | global _tmpDir 43 | global tmpDownloadPath 44 | global ffmpegMetadataPath 45 | global songSegmentsPath 46 | global thumbnailPath 47 | global songDownloadPath 48 | global songEditPath 49 | _tmpDir = TemporaryDirectory() 50 | tmpDownloadPath = _tmpDir.name 51 | ffmpegMetadataPath = f'{tmpDownloadPath}/FFMETADATAFILE' 52 | songSegmentsPath = f'{tmpDownloadPath}/songSegmants' 53 | thumbnailPath = f'{tmpDownloadPath}/thumbnails' 54 | songDownloadPath = f'{tmpDownloadPath}/songDownloads' 55 | songEditPath = f'{tmpDownloadPath}/songEdit' 56 | _createSubDirs() 57 | createTmpDir() 58 | 59 | def clearTmpDir(): 60 | if not os.path.exists(tmpDownloadPath): 61 | createTmpDir() 62 | 63 | _clearSubPath(tmpDownloadPath) 64 | _createSubDirs() 65 | 66 | def clearTmpSubPath(path:str): 67 | '''clears file or directory subtree''' 68 | if not path.startswith(tmpDownloadPath): 69 | raise Exception(f"{path} is not contained in {tmpDownloadPath}") 70 | 71 | if not os.path.exists(tmpDownloadPath): 72 | createTmpDir() 73 | _clearSubPath(path) 74 | -------------------------------------------------------------------------------- /sync_dl/config/ytdlParams.py: -------------------------------------------------------------------------------- 1 | import subprocess 2 | from sync_dl.config.parsing import writeToConfig, readConfig 3 | 4 | # youtube-dl dlParams, used in downloadToTmp 5 | dlParams={"quiet": True, "noplaylist": True, 'format': 'bestaudio'} 6 | 7 | 8 | _ffmpegTested = False 9 | _hasFfmpeg = False 10 | 11 | def testFfmpeg(): 12 | global _ffmpegTested 13 | global _hasFfmpeg 14 | if _ffmpegTested: 15 | return _hasFfmpeg 16 | try: 17 | subprocess.check_output(['ffmpeg', '-version']) 18 | except: 19 | _ffmpegTested = True 20 | _hasFfmpeg = False 21 | return False 22 | 23 | _ffmpegTested = True 24 | _hasFfmpeg = True 25 | return True 26 | 27 | 28 | dlParams['postprocessors'] = [] 29 | if testFfmpeg(): 30 | dlParams['postprocessors'].append( 31 | {'key': 'FFmpegMetadata'} 32 | ) 33 | 34 | def _addIfNotExists(l, key, val): 35 | for i in range(len(l)): 36 | if l[i]['key'] == key: 37 | l[i] = val 38 | return 39 | 40 | l.append(val) 41 | 42 | def _removeIfExists(l, key): 43 | for i in range(len(l)): 44 | if l[i]['key'] == key: 45 | l.pop(i) 46 | return 47 | 48 | 49 | def setAudioFormat(): 50 | global audioFormat 51 | global dlParams 52 | audioFormat = readConfig('audioFormat') 53 | 54 | if audioFormat == 'best': 55 | dlParams['format'] = f'bestaudio' 56 | if testFfmpeg(): 57 | _addIfNotExists(dlParams['postprocessors'], 'FFmpegExtractAudio', {'key': 'FFmpegExtractAudio'}) 58 | else: 59 | if not testFfmpeg(): 60 | writeToConfig('audioFormat', 'best') 61 | raise Exception("ffmpeg is Required to Use Audio Format Other Than 'best'") 62 | dlParams['format'] = f'{audioFormat}/bestaudio' 63 | _addIfNotExists(dlParams['postprocessors'], 'FFmpegExtractAudio', { 64 | 'key': 'FFmpegExtractAudio', 65 | 'preferredcodec': audioFormat, 66 | 'preferredquality': 'bestaudio', 67 | 'nopostoverwrites': True 68 | }) 69 | 70 | setAudioFormat() 71 | 72 | def setEmbedThumbnails(): 73 | global embedThumbnail 74 | embedThumbnail = readConfig('embedThumbnail', boolean=True) 75 | if embedThumbnail: 76 | dlParams['writethumbnail'] = True 77 | _addIfNotExists(dlParams['postprocessors'], 'EmbedThumbnail', { 78 | 'key': 'EmbedThumbnail', 79 | 'already_have_thumbnail': False 80 | }) 81 | else: 82 | _removeIfExists(dlParams['postprocessors'], 'EmbedThumbnail') 83 | if 'writethumbnail' in dlParams: 84 | dlParams.pop('writethumbnail') 85 | 86 | setEmbedThumbnails() 87 | -------------------------------------------------------------------------------- /sync_dl/helpers.py: -------------------------------------------------------------------------------- 1 | import os 2 | import re 3 | import shutil 4 | from typing import Union, List 5 | 6 | from sync_dl import noInterrupt 7 | import sync_dl.config as cfg 8 | from sync_dl.ytdlWrappers import downloadToTmp,moveFromTmp 9 | 10 | _ords = ('th', 'st', 'nd', 'rd') 11 | def getOrdinalIndicator(n: int): 12 | return 'th' if ( 4 <= n%100 <= 20 ) or n%10 >= 3 else _ords[n%10] 13 | 14 | 15 | 16 | def getNumDigets(plLen): 17 | return len(str( plLen+1 )) 18 | 19 | def padZeros(s, numDigits): 20 | return str(s).zfill(numDigits) 21 | 22 | def rename(metaData, printer, plPath, oldName, newName, index, newId): 23 | with noInterrupt: 24 | printer(f"Renaming {oldName} to {newName}") 25 | os.rename(f"{plPath}/{oldName}",f"{plPath}/{newName}") 26 | 27 | if index >= len(metaData["ids"]): 28 | metaData["ids"].append(newId) 29 | else: 30 | metaData["ids"][index] = newId 31 | printer("Renaming Complete") 32 | 33 | def relabel(metaData, printer,plPath, oldName, oldIndex, newIndex, numDigets): 34 | ''' 35 | used for changing the number of an element in the playlist 36 | will blank out old posistion in the metaData 37 | 38 | note this does NOT prevent you from overwriting numbering of 39 | an existing song 40 | 41 | returns new name which is needed in some cases (ie when a song is temporarily moved) 42 | ''' 43 | 44 | newName = re.sub(cfg.filePrependRE, f"{createNumLabel(newIndex,numDigets)}_" , oldName) 45 | 46 | songId = metaData['ids'][oldIndex] 47 | with noInterrupt: 48 | printer(f"Relabeling {oldName} to {newName}") 49 | 50 | os.rename(f"{plPath}/{oldName}",f"{plPath}/{newName}") 51 | 52 | numIds = len(metaData["ids"]) 53 | if newIndex >= numIds: 54 | 55 | for _ in range(newIndex-numIds): 56 | metaData["ids"].append('') 57 | 58 | metaData["ids"].append(songId) 59 | else: 60 | metaData["ids"][newIndex] = songId 61 | 62 | metaData['ids'][oldIndex] = '' 63 | printer("Relabeling Complete") 64 | return newName 65 | 66 | def copy(srcMetaData, destMetaData, printer, srcPlPath, destPlPath, srcName, srcIndex, destIndex, numDigets): 67 | destName = re.sub(cfg.filePrependRE, f"{createNumLabel(destIndex,numDigets)}_" , srcName) 68 | songId = srcMetaData['ids'][srcIndex] 69 | numDestIds = len(destMetaData["ids"]) 70 | 71 | with noInterrupt: 72 | printer(f"Copying {srcPlPath}/{srcName} to {destPlPath}/{destName}") 73 | 74 | shutil.copy(f"{srcPlPath}/{srcName}",f"{destPlPath}/{destName}") 75 | 76 | if destIndex >= numDestIds: 77 | for _ in range(destIndex-numDestIds): 78 | destMetaData["ids"].append('') 79 | 80 | destMetaData["ids"].append(songId) 81 | else: 82 | destMetaData["ids"][destIndex] = songId 83 | 84 | printer("Copy Complete") 85 | return destName 86 | 87 | 88 | def delete(metaData, plPath, name, index): 89 | with noInterrupt: 90 | cfg.logger.debug(f"Deleting {metaData['ids'][index]} {name}") 91 | os.remove(f"{plPath}/{name}") 92 | 93 | del metaData["ids"][index] 94 | 95 | cfg.logger.debug("Deleting Complete") 96 | 97 | 98 | 99 | def download(metaData,plPath, songId, index,numDigets, counter = ''): 100 | ''' 101 | downloads song and adds it to metadata at index 102 | returns whether or not the download succeeded 103 | 104 | counter is an optional string indicating what number the download is ie) 10 or 23/149 105 | ''' 106 | if index == -1: 107 | index = len(metaData["ids"]) 108 | 109 | num = createNumLabel(index,numDigets) 110 | 111 | if counter: 112 | message = f"Dowloading song {counter}, Id {songId}" 113 | else: 114 | message = f"Dowloading song Id {songId}" 115 | 116 | cfg.logger.info(message) 117 | if downloadToTmp(songId,num): #returns true if succeeds 118 | 119 | with noInterrupt: # moving the song from tmp and editing the metadata must occur togeather 120 | songName = moveFromTmp(plPath) 121 | if index >= len(metaData["ids"]): 122 | metaData["ids"].append(songId) 123 | else: 124 | metaData["ids"][index] = songId 125 | cfg.logger.debug("Download Complete") 126 | return songName 127 | return '' 128 | 129 | 130 | def createNumLabel(n,numDigets): 131 | n = str(n) 132 | lenN = len(n) 133 | if lenN>numDigets: 134 | 135 | cfg.logger.warning(f"Number Label Too large! Expected {numDigets} but got {lenN} digets") 136 | #raise Exception(f"Number Label Too large! Expected {numDigets} but got {lenN} digets") 137 | numDigets+=1 138 | 139 | return (numDigets-lenN)*"0"+n 140 | 141 | 142 | def getSongNum(element): 143 | ''' 144 | returns the number in front of each song file 145 | ''' 146 | match = re.match(cfg.filePrependRE,element) 147 | 148 | assert match is not None 149 | return int(match.group()[:-1]) 150 | 151 | def _filterFunc(element): 152 | ''' 153 | returns false for any string not preceded by some number followed by an underscore 154 | used for filtering non song files 155 | ''' 156 | match = re.match(cfg.filePrependRE,element) 157 | if match: 158 | return True 159 | 160 | return False 161 | 162 | 163 | def getLocalSongs(plPath): 164 | ''' 165 | returns sanatized list of all songs in local playlist, in order 166 | ''' 167 | currentDir = os.listdir(path=plPath) 168 | currentDir = sorted(filter(_filterFunc,currentDir), key= getSongNum) #sorted and sanitized dir 169 | return currentDir 170 | 171 | def smartSyncNewOrder(localIds,remoteIds): 172 | ''' 173 | used by smartSync, localIds will not be mutated but remoteIds will 174 | output is newOrder, a list of tuples ( Id of song, where to find it ) 175 | the "where to find it" is the number in the old ordering (None if song is to be downloaded) 176 | ''' 177 | newOrder=[] 178 | 179 | localIdPairs = [(localIds[index],index) for index in range(len(localIds))] #contins ( Id of song, local order ) 180 | 181 | while True: 182 | if len(localIdPairs)==0: 183 | newOrder.extend( (remoteId,None) for remoteId in remoteIds ) 184 | break 185 | 186 | elif len(remoteIds)==0: 187 | newOrder.extend( localIdPairs ) 188 | break 189 | 190 | remoteId=remoteIds[0] 191 | localId,localIndex = localIdPairs[0] 192 | 193 | if localId==remoteId: 194 | #remote song is already saved locally in correct posistion 195 | newOrder.append( localIdPairs.pop(0) ) 196 | 197 | remoteId = remoteIds.pop(0) #must also remove this remote element 198 | 199 | elif localId not in remoteIds: 200 | # current local song has been removed from remote playlist, it must remain in current order 201 | newOrder.append( localIdPairs.pop(0) ) 202 | 203 | 204 | # at this point the current local song and remote song arent the same, but the current local 205 | # song still exists remotly, hence we can insert the remote song into the current posistion 206 | elif remoteId in localIds: 207 | # remote song exists in local but in wrong place 208 | 209 | index = localIds[localIndex+1:].index(remoteId)+localIndex+1 210 | j = -1 211 | for i,_ in enumerate(localIdPairs): 212 | if localIdPairs[i][1]==index: 213 | j = i 214 | break 215 | 216 | assert j != -1 217 | newOrder.append( localIdPairs.pop(j) ) 218 | remoteId = remoteIds.pop(0) #must also remove this remote element 219 | 220 | #checks if songs after the moved song are not in remote, if so they must be moved with it 221 | while j Union[int, None]: 232 | '''returns the nth occurance of item in l''' 233 | num = 0 234 | lastOccuranceIndex = None 235 | for i,item2 in enumerate(l): 236 | if item == item2: 237 | if num == n: 238 | return i 239 | lastOccuranceIndex = i 240 | 241 | num+=1 242 | 243 | return lastOccuranceIndex 244 | 245 | def numOccurance(l: list, index: int) -> int: 246 | ''' 247 | the reverse of getNthOccuranceIndex 248 | let A = l[index], and this function returns 1, that means l[index] is the 1th A in l (after the 0th) 249 | ie) l = [A, B ,A ,C, B , B] index = 2 will return 1 250 | ''' 251 | item = l[index] 252 | num = 0 253 | for i in range(0, index): 254 | if l[i] == item: 255 | num+=1 256 | 257 | return num 258 | 259 | class TransferMove: 260 | songId: str 261 | songName: str 262 | 263 | performCopy:bool = False 264 | srcCopyName: str 265 | srcCopyIndex: int 266 | destCopyIndex: int 267 | 268 | performRemoteAdd:bool = False 269 | destRemoteAddIndex: int 270 | 271 | performLocalDelete: bool = False 272 | srcLocalDeleteIndex: int 273 | srcLocalDeleteName: str 274 | 275 | performRemoteDelete:bool = False 276 | srcRemoteDeleteIndex: int 277 | 278 | def __init__(self, songId, songName): 279 | self.songId = songId 280 | self.songName = songName 281 | return 282 | 283 | def copyAction(self, srcCopyName, srcCopyIndex, destCopyIndex): 284 | self.performCopy = True 285 | self.srcCopyName = srcCopyName 286 | self.srcCopyIndex = srcCopyIndex 287 | self.destCopyIndex = destCopyIndex 288 | 289 | def remoteAddAction(self, destRemoteAddIndex): 290 | self.performRemoteAdd = True 291 | self.destRemoteAddIndex = destRemoteAddIndex 292 | 293 | assert self.performCopy 294 | 295 | def localDeleteAction(self, srcLocalDeleteIndex, srcLocalDeleteName): 296 | self.performLocalDelete = True 297 | self.srcLocalDeleteIndex = srcLocalDeleteIndex 298 | self.srcLocalDeleteName = srcLocalDeleteName 299 | 300 | assert self.performCopy 301 | 302 | assert self.srcLocalDeleteIndex == self.srcCopyIndex 303 | assert self.srcCopyName == self.srcLocalDeleteName 304 | 305 | def remoteDeleteAction(self, srcRemoteDeleteIndex): 306 | self.performRemoteDelete = True 307 | self.srcRemoteDeleteIndex = srcRemoteDeleteIndex 308 | 309 | assert self.performRemoteAdd 310 | assert self.performLocalDelete 311 | 312 | 313 | def calcuateTransferMoves(currentSrcDir: List[str], 314 | srcLocalIds: List[str], destLocalIds: List[str], 315 | srcRemoteIds: List[str], destRemoteIds: List[str], 316 | srcStart:int, srcEnd:int, destIndex:int) -> List[TransferMove]: 317 | 318 | '''calculates moves for transferSongs''' 319 | 320 | # number of elements to move 321 | blockSize = srcEnd-srcStart+1 322 | 323 | 324 | # get index to add songs in remote dest 325 | if destIndex != -1: 326 | destLocalNumOccurances = numOccurance(destLocalIds, destIndex) 327 | destRemoteIndex = getNthOccuranceIndex(destRemoteIds, destLocalIds[destIndex], destLocalNumOccurances) 328 | if destRemoteIndex == None: 329 | destRemoteIndex = 0 330 | else: 331 | destRemoteIndex += 1 332 | else: 333 | destRemoteIndex = 0 334 | 335 | 336 | # store moves 337 | songTransfers: List[TransferMove] = [] 338 | 339 | ### Calculate moves 340 | copyIndex = destIndex + blockSize 341 | for srcIndex in reversed(range(srcStart, srcEnd+1)): 342 | songId = srcLocalIds[srcIndex] 343 | srcLocalNumOccurances = numOccurance(srcLocalIds, srcIndex) 344 | srcRemoteIndex = getNthOccuranceIndex(srcRemoteIds, songId, srcLocalNumOccurances) 345 | # if srcRemoteIndex is None, or if there are more of songId in srcLocalIds, dont delete 346 | 347 | # copy song to dest local 348 | srcName = currentSrcDir[srcIndex] 349 | songName = re.sub(cfg.filePrependRE,"",srcName) 350 | data = TransferMove(songId, songName) 351 | 352 | data.copyAction(srcName, srcIndex, copyIndex) 353 | copyIndex -= 1 354 | 355 | # add song to dest remote 356 | if songId != cfg.manualAddId: 357 | data.remoteAddAction(destRemoteIndex) 358 | 359 | # delete song from src local 360 | data.localDeleteAction(srcIndex, srcName) 361 | 362 | # delete song from src remote 363 | if srcRemoteIndex is not None: 364 | data.remoteDeleteAction(srcRemoteIndex) 365 | 366 | songTransfers.append(data) 367 | 368 | return songTransfers 369 | 370 | def promptAndSanitize(promptText, *args): 371 | while True: 372 | answer = input(promptText).lower().strip() 373 | if answer not in args: 374 | cfg.logger.error(f"{answer} is not an option") 375 | continue 376 | return answer 377 | 378 | 379 | 380 | def logTransferInfo(songTransfers, srcPlName, destPlName, srcIdsLen, destIdsLen, srcRemoteIds, destRemoteIds, currentDestDir, srcStart, srcEnd, destIndex): 381 | blockSize = srcEnd-srcStart+1 382 | 383 | maxNum = max(srcIdsLen, destIdsLen, len(srcRemoteIds), len(destRemoteIds)) 384 | numIndexDigits = len(str(maxNum)) 385 | 386 | numMoveDigits = len(str(len(songTransfers))) 387 | 388 | cfg.logger.info(f"------------[Transfer Moves (applied in reverse)]-----------") 389 | cfg.logger.info(f"{'i'*numMoveDigits}: Move Parts | Song Id | Song name\n") 390 | for i,move in reversed(list(enumerate(songTransfers))): 391 | actionPrompt = f"{padZeros(i, numMoveDigits)}: " 392 | actionPrompt += f"{'LC' if move.performCopy else ' '} " 393 | actionPrompt += f"{'RA' if move.performRemoteAdd else ' '} " 394 | actionPrompt += f"{'LR' if move.performLocalDelete else ' '} " 395 | actionPrompt += f"{'RR' if move.performRemoteDelete else ' '} | " 396 | prompt = f"{move.songId} | {move.songName}\n" 397 | 398 | localPrompt = "" 399 | if move.performCopy or move.performLocalDelete: 400 | localPrompt += ' '*len(actionPrompt) + 'Local ' 401 | if move.performLocalDelete: 402 | localPrompt += f"{srcPlName}: {padZeros(move.srcLocalDeleteIndex, numIndexDigits)} " 403 | if move.performCopy: 404 | localPrompt += f"-> {destPlName}: {padZeros(move.destCopyIndex, numIndexDigits)} | " 405 | 406 | 407 | remotePrompt = "" 408 | if move.performRemoteAdd or move.performRemoteDelete: 409 | remotePrompt += "Remote " 410 | if move.performRemoteDelete: 411 | remotePrompt += f"{srcPlName}: {padZeros(move.srcRemoteDeleteIndex, numIndexDigits)} " 412 | if move.performRemoteAdd: 413 | remoteActualMoveNum = padZeros(move.destRemoteAddIndex, numIndexDigits) 414 | remoteFinalMoveNum = padZeros(blockSize + move.destRemoteAddIndex - i - 1, numIndexDigits) 415 | remotePrompt += f"-> {destPlName}: {remoteFinalMoveNum} ({remoteActualMoveNum})\n" 416 | 417 | 418 | 419 | cfg.logger.info(actionPrompt + prompt + localPrompt + remotePrompt) 420 | 421 | 422 | cfg.logger.info(f"------------[Legend]-----------") 423 | cfg.logger.info ( 424 | f"LC: Local Copy, from {srcPlName} to {destPlName}\n" \ 425 | f"RA: Remote Add to {destPlName} \n" \ 426 | f"LR: Local Remove from {srcPlName} \n" \ 427 | f"RR: Remote Remove from {srcPlName}\n" \ 428 | ) 429 | 430 | cfg.logger.info(f"------------[Summery]-----------") 431 | cfg.logger.info(f"{srcPlName}: [{srcStart}, {srcEnd}] -> {destPlName}: [{srcStart + destIndex}, {srcEnd + destIndex}]\n") 432 | 433 | cfg.logger.info(f"Transfering Songs in Range [{srcStart}, {srcEnd}] in {srcPlName}") 434 | cfg.logger.info(f" Start Range: {songTransfers[-1].songName}") 435 | cfg.logger.info(f" End Range: {songTransfers[0].songName}") 436 | 437 | if destIndex != -1: 438 | leaderSong = re.sub(cfg.filePrependRE,"", currentDestDir[destIndex]) # the name of the song the range will come after 439 | cfg.logger.info(f"\nTo After song Index {destIndex} in {destPlName}") 440 | cfg.logger.info(f" {destIndex}: {leaderSong}") 441 | else: 442 | cfg.logger.info(f"\nTo Start of {destPlName}") 443 | 444 | 445 | 446 | cfg.logger.info(f"\n------------[Note]-----------") 447 | cfg.logger.info(f"For Best Remote Playlists Results, Ensure Local and Remote Playlists are Synced") 448 | cfg.logger.info(f"(failing to do so may lead to ordering changes in remote if there are duplicate songs)") 449 | 450 | 451 | -------------------------------------------------------------------------------- /sync_dl/plManagement.py: -------------------------------------------------------------------------------- 1 | import os 2 | import re 3 | import shelve 4 | 5 | from sync_dl import noInterrupt 6 | from sync_dl.helpers import createNumLabel, getLocalSongs,download, delete, relabel, getNumDigets, getSongNum 7 | from sync_dl.timestamps import addTimestampsIfNoneExist 8 | import sync_dl.config as cfg 9 | 10 | def _checkDeletions(plPath,metaData): 11 | ''' 12 | checks if metadata has songs that are no longer in directory 13 | ''' 14 | currentDir = getLocalSongs(plPath) 15 | idsLen = len(metaData['ids']) 16 | 17 | # there have been no deletions, however there may be a gap in numberings 18 | # which would be falsely detected as a deletion if this check wheren't here 19 | if idsLen == len(currentDir): 20 | return 21 | 22 | #song numbers in currentDir 23 | currentDirNums = [int(re.match(cfg.filePrependRE,song).group()[:-1]) for song in currentDir] 24 | 25 | numRange = range(idsLen) 26 | 27 | #difference between whats in the folder and whats in metadata 28 | deleted = [i for i in numRange if i not in currentDirNums] 29 | numDeleted = len(deleted) 30 | 31 | if numDeleted > 0: 32 | cfg.logger.debug(f"songs numbered {deleted} are no longer in playlist") 33 | 34 | numDidgets = len(str( len(metaData["ids"]) - numDeleted )) 35 | newIndex = 0 36 | 37 | for newIndex, oldIndex in enumerate(currentDirNums): 38 | if newIndex != oldIndex: 39 | oldName = currentDir[newIndex] 40 | newName = re.sub(cfg.filePrependRE, f"{createNumLabel(newIndex,numDidgets)}_" , oldName) 41 | 42 | with noInterrupt: 43 | cfg.logger.debug(f"Renaming {oldName} to {newName}") 44 | os.rename(f"{plPath}/{oldName}",f"{plPath}/{newName}") 45 | 46 | if newIndex in deleted: 47 | # we only remove the deleted entry from metadata when its posistion has been filled 48 | cfg.logger.debug(f"Removing {metaData['ids'][newIndex]} from metadata") 49 | 50 | #need to adjust for number already deleted 51 | removedAlready = (numDeleted - len(deleted)) 52 | del metaData["ids"][newIndex - removedAlready] 53 | del deleted[0] 54 | 55 | cfg.logger.debug("Renaming Complete") 56 | 57 | 58 | 59 | while len(deleted)!=0: 60 | index = deleted[0] 61 | # we remove any remaining deleted entries from metadata 62 | # note even if the program crashed at this point, running this fuction 63 | # again would yeild an uncorrupted state 64 | removedAlready = (numDeleted - len(deleted)) 65 | with noInterrupt: 66 | cfg.logger.debug(f"Removing {metaData['ids'][index - removedAlready]} from metadata") 67 | del metaData["ids"][index - removedAlready] 68 | del deleted[0] 69 | 70 | def _checkBlanks(plPath,metaData): 71 | for i in reversed(range(len(metaData["ids"]))): 72 | songId = metaData['ids'][i] 73 | if songId == '': 74 | cfg.logger.debug(f'Blank MetaData id Found at Index {i}, removing') 75 | del metaData["ids"][i] 76 | 77 | def _removeGaps(plPath): 78 | currentDir = getLocalSongs(plPath) 79 | numDidgets = len(str(len(currentDir))) 80 | 81 | for i,oldName in enumerate(currentDir): 82 | newPrepend = f"{createNumLabel(i,numDidgets)}_" 83 | oldPrepend = re.search(cfg.filePrependRE, oldName).group(0) 84 | if oldPrepend!=newPrepend: 85 | newName = re.sub(cfg.filePrependRE, f"{createNumLabel(i,numDidgets)}_" , oldName) 86 | cfg.logger.debug(f"Renaming {oldName} to {newName}") 87 | os.rename(f"{plPath}/{oldName}",f"{plPath}/{newName}") 88 | cfg.logger.debug("Renaming Complete") 89 | 90 | 91 | 92 | def _restorePrepend(plPath,metaData): 93 | 94 | if "removePrependOrder" not in metaData: 95 | #prepends already restored 96 | return 97 | 98 | cfg.logger.debug("Restoring Prepends") 99 | 100 | currentDir = os.listdir(path=plPath) 101 | idsLen = len(metaData["ids"]) 102 | numDigets = getNumDigets(idsLen) 103 | 104 | for file in currentDir: 105 | try: 106 | index = metaData["removePrependOrder"][file] 107 | except KeyError: 108 | # file isn't in playlist (or its the metadata 109 | continue 110 | 111 | # at this point we add the prepend to the file 112 | label = createNumLabel(index,numDigets) 113 | 114 | cfg.logger.debug(f"Adding Prepend {label} to {file}") 115 | with noInterrupt: 116 | os.rename(f"{plPath}/{file}",f"{plPath}/{label}_{file}") 117 | 118 | # removed item from dictionary to prevent double restoring 119 | del metaData["removePrependOrder"][file] 120 | 121 | # metaData["removePrependOrder"] is removed, this is used to signal that all the prepends are there 122 | # and the playlist can be treated like normal 123 | 124 | del metaData["removePrependOrder"] 125 | 126 | 127 | def correctStateCorruption(plPath,metaData): 128 | cfg.logger.debug("Checking for playlist state Corruption") 129 | 130 | _checkBlanks(plPath,metaData) # must come first so later steps dont assume blanks are valid when checking len 131 | 132 | _restorePrepend(plPath,metaData) # can only restore if prepends where removed by remove prepends 133 | 134 | _checkDeletions(plPath,metaData) 135 | 136 | _removeGaps(plPath) # must come after check deletions (if user manually deletes, then we only have number 137 | # on song to go off of, hence removing gaps by chaning the numbers would break this) 138 | 139 | 140 | def removePrepend(plPath, metaData): 141 | #TODO this step might be unnessisary because its already done in togglePrepend 142 | if "removePrependOrder" in metaData: 143 | # prepends already removed or partially removed 144 | return 145 | 146 | currentDir = getLocalSongs(plPath) 147 | 148 | metaData["removePrependOrder"] = {} 149 | 150 | for oldName in currentDir: 151 | index = getSongNum(oldName) 152 | newName = re.sub(cfg.filePrependRE, "" , oldName) 153 | 154 | with noInterrupt: 155 | os.rename(f"{plPath}/{oldName}",f"{plPath}/{newName}") 156 | metaData["removePrependOrder"][newName] = index 157 | 158 | 159 | def editPlaylist(plPath, newOrder, deletions=False): 160 | ''' 161 | metaData is json as defined in newPlaylist 162 | newOrder is an ordered list of tuples (Id of song, where to find it ) 163 | the "where to find it" is the number in the old ordering (None if song is to be downloaded) 164 | 165 | note if song is in playlist already the id of song in newOrder will not be used 166 | ''' 167 | 168 | currentDir = getLocalSongs(plPath) 169 | numDigets = len(str(2*len(newOrder))) #needed for creating starting number for auto ordering ie) 001, 0152 170 | # len is doubled because we will be first numbering with numbers above the 171 | # so state remains recoverable in event of crash 172 | 173 | with shelve.open(f"{plPath}/{cfg.metaDataName}", 'c',writeback=True) as metaData: 174 | idsLen = len(metaData['ids']) 175 | cfg.logger.info(f"Editing Playlist...") 176 | cfg.logger.debug(f"Old Order: {metaData['ids']}") 177 | 178 | for i in range(len(newOrder)): 179 | newId,oldIndex = newOrder[i] 180 | newIndex = idsLen + i # we reorder the playlist with exclusivly new numbers in case a crash occurs 181 | 182 | if oldIndex == None: 183 | # must download new song 184 | songName = download(metaData,plPath,newId,newIndex,numDigets) 185 | if songName and cfg.autoScrapeCommentTimestamps: 186 | addTimestampsIfNoneExist(plPath, songName, newId) 187 | 188 | else: 189 | #song exists locally, but must be reordered/renamed 190 | oldName = currentDir[oldIndex] 191 | relabel(metaData,cfg.logger.debug,plPath,oldName,oldIndex,newIndex,numDigets) 192 | 193 | 194 | if deletions: 195 | oldIndices = [item[1] for item in newOrder] 196 | 197 | for i in reversed(range(len(currentDir))): 198 | if i not in oldIndices: 199 | delete(metaData,plPath,currentDir[i],i) 200 | 201 | _checkBlanks(plPath,metaData) 202 | _removeGaps(plPath) 203 | -------------------------------------------------------------------------------- /sync_dl/tests/integrationTests.py: -------------------------------------------------------------------------------- 1 | import re 2 | import unittest 3 | import os 4 | import shelve 5 | import shutil 6 | import random 7 | 8 | from sync_dl.commands import smartSync,newPlaylist,swap,shuffle,move 9 | import sync_dl.config as cfg 10 | from sync_dl.commands import compareMetaData, showPlaylist 11 | from sync_dl.helpers import getLocalSongs 12 | from sync_dl.ytdlWrappers import getTitle,getIdsAndTitles 13 | 14 | from sync_dl.timestamps import getTimestamps, extractChapters, createChapterFile, wipeChapterFile, addTimestampsToChapterFile, applyChapterFileToSong 15 | from sync_dl.timestamps.scraping import Timestamp, scrapeCommentsForTimestamps 16 | 17 | 18 | def metaDataSongsCorrect(metaData,plPath): 19 | ''' 20 | tests if metadata ids corrispond to the correct remote songs. 21 | returns boolian test result and logs details 22 | 23 | test fails if local title does not match remote title 24 | manually added songs are ignored 25 | ''' 26 | cfg.logger.info("Testing if Metadata IDs Corrispond to Correct Songs") 27 | 28 | currentDir = getLocalSongs(plPath) 29 | 30 | localIds = metaData['ids'] 31 | localTitles = map(lambda title: re.sub(cfg.filePrependRE,'',title), currentDir) 32 | 33 | for i,localTitle in enumerate(localTitles): 34 | localId = localIds[i] 35 | if localId!=cfg.manualAddId: 36 | 37 | remoteTitle = getTitle(f"https://www.youtube.com/watch?v={localId}") 38 | if localTitle != remoteTitle: 39 | message = (f"{i}th Local Title: {localTitle}\n" 40 | f"Differes from Remote Title: {remoteTitle}\n" 41 | f"With same Id: {localId}") 42 | cfg.logger.error(message) 43 | return False 44 | cfg.logger.info("Test Passed") 45 | return True 46 | 47 | def metaDataMatches(metaData,plPath): 48 | ''' 49 | metadata and local playlist must perfectly match remote playlist for this to return true 50 | ''' 51 | cfg.logger.info("Testing If Metadata Perfectly Matches Remote Playlist") 52 | 53 | currentDir = getLocalSongs(plPath) 54 | 55 | localIds = metaData['ids'] 56 | 57 | remoteIds, remoteTitles = getIdsAndTitles(metaData['url']) 58 | 59 | if len(localIds) != len(currentDir): 60 | cfg.logger.error(f"metadata ids and local playlist differ in length {len(localIds)} to {len(currentDir)}") 61 | return False 62 | if len(localIds)!= len(remoteIds): 63 | cfg.logger.error(f"local and remote playlists differ in length {len(localIds)} to {len(remoteIds)}") 64 | return False 65 | 66 | 67 | for i,localTitle in enumerate(currentDir): 68 | localTitle,_ = os.path.splitext(localTitle) 69 | localTitle = re.sub(cfg.filePrependRE,'',localTitle) 70 | 71 | localId = localIds[i] 72 | remoteId = remoteIds[i] 73 | remoteTitle = remoteTitles[i] 74 | 75 | if localId!=remoteId: 76 | message = (f"{i}th Local id: {localId}\n" 77 | f"With title: {localTitle}\n" 78 | f"Differes from Remote id: {remoteId}\n" 79 | f"With title: {remoteTitle}") 80 | cfg.logger.error(message) 81 | return False 82 | 83 | 84 | if localTitle!=remoteTitle: 85 | message = (f"{i}th Local Title: {localTitle}\n" 86 | f"With Id: {localId}\n" 87 | f"Differes from Remote Title: {remoteTitle}\n" 88 | f"With Id: {localId}") 89 | cfg.logger.error(message) 90 | return False 91 | return True 92 | 93 | 94 | 95 | class test_integration(unittest.TestCase): 96 | '''All tests are in order, failing one will fail subsequent tests. 97 | This is intentional, redownloading entire playlist for each 98 | test would be a waste of bandwidth''' 99 | 100 | PL_URL = None #must be passed 101 | plName = 'integration' 102 | plPath = f'{cfg.testPlPath}/{plName}' 103 | 104 | def test_0_creation(self): 105 | 106 | cfg.logger.info("Running test_creation") 107 | 108 | if os.path.exists(self.plPath): 109 | shutil.rmtree(self.plPath) 110 | 111 | newPlaylist(self.plPath,self.PL_URL) 112 | 113 | with shelve.open(f"{self.plPath}/{cfg.metaDataName}", 'c',writeback=True) as metaData: 114 | passed = metaDataMatches(metaData,self.plPath) 115 | 116 | 117 | self.assertTrue(passed) 118 | 119 | 120 | 121 | def test_1_smartSyncNoEdit(self): 122 | cfg.logger.info("Running test_smartSyncNoEdit") 123 | smartSync(self.plPath) 124 | with shelve.open(f"{self.plPath}/{cfg.metaDataName}", 'c',writeback=True) as metaData: 125 | passed = metaDataMatches(metaData,self.plPath) 126 | 127 | self.assertTrue(passed) 128 | 129 | 130 | def test_2_smartSyncSwap(self): 131 | '''Simulates remote reordering by reordering local''' 132 | cfg.logger.info("Running test_smartSyncSwap") 133 | swap(self.plPath,0 , 1) 134 | 135 | smartSync(self.plPath) 136 | with shelve.open(f"{self.plPath}/{cfg.metaDataName}", 'c',writeback=True) as metaData: 137 | passed = metaDataMatches(metaData,self.plPath) 138 | 139 | self.assertTrue(passed) 140 | 141 | def test_3_smartSyncMove(self): 142 | cfg.logger.info("Running test_smartSyncSwap") 143 | 144 | with shelve.open(f"{self.plPath}/{cfg.metaDataName}", 'c',writeback=True) as metaData: 145 | numIds = len(metaData['ids']) 146 | 147 | move(self.plPath,0 , int(numIds/2)) 148 | 149 | smartSync(self.plPath) 150 | 151 | with shelve.open(f"{self.plPath}/{cfg.metaDataName}", 'c',writeback=True) as metaData: 152 | passed = metaDataMatches(metaData,self.plPath) 153 | 154 | self.assertTrue(passed) 155 | 156 | 157 | def test_4_smartSyncShuffle(self): 158 | '''Simulates remote reordering by shuffling local''' 159 | cfg.logger.info("Running test_smartSyncShuffle") 160 | shuffle(self.plPath) 161 | 162 | smartSync(self.plPath) 163 | with shelve.open(f"{self.plPath}/{cfg.metaDataName}", 'c',writeback=True) as metaData: 164 | passed = metaDataMatches(metaData,self.plPath) 165 | 166 | self.assertTrue(passed) 167 | 168 | def test_5_smartSyncDelAndShuffle(self): 169 | cfg.logger.info("Running test_smartSyncDelAndShuffle") 170 | shuffle(self.plPath) 171 | 172 | currentDir = getLocalSongs(self.plPath) 173 | 174 | #deletes a third (rounded down) of the songs in the playlist 175 | toDelete = [] 176 | for _ in range(int(len(currentDir)/3)): 177 | randSong = random.choice(currentDir) 178 | 179 | while randSong in toDelete: 180 | #ensures we dont try to delete the same song twice 181 | randSong = random.choice(currentDir) 182 | 183 | toDelete.append(randSong) 184 | 185 | for song in toDelete: 186 | os.remove(f'{self.plPath}/{song}') 187 | 188 | 189 | smartSync(self.plPath) 190 | 191 | with shelve.open(f"{self.plPath}/{cfg.metaDataName}", 'c',writeback=True) as metaData: 192 | passed = metaDataMatches(metaData,self.plPath) 193 | 194 | self.assertTrue(passed) 195 | 196 | def test_7_scrapeTimeStamps(self): 197 | videoId = '9WbtgupHTPA' 198 | knownTimeStamps = [ 199 | Timestamp(time = 0, label = 'beginning'), 200 | Timestamp(time = 20, label = 'some stuff'), 201 | Timestamp(time = 90, label = 'shaking up the format'), 202 | Timestamp(time = 201, label = 'more shaking up the format'), 203 | Timestamp(time = 361, label = 'wowee whats this'), 204 | ] 205 | 206 | scrapedTimestamps = scrapeCommentsForTimestamps(videoId) 207 | 208 | 209 | self.assertEqual(len(knownTimeStamps), len(scrapedTimestamps)) 210 | 211 | for i in range(0,len(scrapedTimestamps)): 212 | self.assertEqual(scrapedTimestamps[i], knownTimeStamps[i]) 213 | 214 | def test_8_stateSummery(self): 215 | '''logs state of playlist after all tests (should be last in test chain)''' 216 | cfg.logger.info("Integration Test End Report:") 217 | 218 | compareMetaData(self.plPath) 219 | showPlaylist(self.plPath) 220 | 221 | -------------------------------------------------------------------------------- /sync_dl/tests/test.mp3: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PrinceOfPuppers/sync-dl/bb2cbf208664ac7407e18abd5e7d30779a7407a0/sync_dl/tests/test.mp3 -------------------------------------------------------------------------------- /sync_dl/tests/unitTests.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | import os 3 | import shutil 4 | import shelve 5 | import sys 6 | import inspect 7 | from string import ascii_uppercase 8 | from typing import List 9 | 10 | import sync_dl.config as cfg 11 | from sync_dl.helpers import smartSyncNewOrder,createNumLabel,getLocalSongs,getNumDigets, calcuateTransferMoves, TransferMove 12 | from sync_dl.plManagement import editPlaylist,correctStateCorruption 13 | 14 | from sync_dl.commands import move, swap, manualAdd, moveRange,togglePrepends 15 | 16 | from sync_dl.timestamps import getTimestamps, extractChapters, createChapterFile, wipeChapterFile, addTimestampsToChapterFile, applyChapterFileToSong 17 | from sync_dl.timestamps.scraping import Timestamp 18 | 19 | try: 20 | from sync_dl_ytapi.helpers import longestIncreasingSequence,oldToNewPushOrder,pushOrderMoves 21 | except: 22 | cfg.logger.error("Please Install sync_dl_ytapi to Run Unittests") 23 | sys.exit(1) 24 | 25 | def createFakePlaylist(name,songs): 26 | '''creates fake playlist with all songs being as if they where locally added''' 27 | 28 | if not os.path.exists(cfg.testPlPath): 29 | os.mkdir(cfg.testPlPath) 30 | 31 | os.mkdir(f'{cfg.testPlPath}/{name}') 32 | 33 | numDigets = getNumDigets(len(songs)) 34 | 35 | with shelve.open(f"{cfg.testPlPath}/{name}/{cfg.metaDataName}", 'c',writeback=True) as metaData: 36 | metaData["url"] = "placeholder" 37 | metaData["ids"] = [] 38 | 39 | for i,song in enumerate(songs): 40 | songName = f"{createNumLabel(i,numDigets)}_{song}" 41 | open(f"{cfg.testPlPath}/{name}/{songName}",'a').close() 42 | metaData["ids"].append(str(i)) # i is used to trace songs in metadata during testing 43 | 44 | 45 | def getPlaylistData(name): 46 | '''used to validate playlist returns list of tups (id, song name)''' 47 | result = [] 48 | songs = getLocalSongs(f"{cfg.testPlPath}/{name}") 49 | with shelve.open(f"{cfg.testPlPath}/{name}/{cfg.metaDataName}", 'c',writeback=True) as metaData: 50 | for i,songId in enumerate(metaData['ids']): 51 | result.append( (songId,songs[i]) ) 52 | 53 | return result 54 | 55 | class test_correctStateCorruption(unittest.TestCase): 56 | 57 | def test_removedSongs(self): 58 | name = inspect.currentframe().f_code.co_name 59 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 60 | 61 | songs = ['A' ,'B' ,'C' ,'D', 'E'] 62 | 63 | plPath = f'{cfg.testPlPath}/{name}' 64 | 65 | createFakePlaylist(name,songs) 66 | 67 | os.remove(f'{plPath}/0_A') 68 | os.remove(f'{plPath}/4_E') 69 | os.remove(f'{plPath}/2_C') 70 | 71 | with shelve.open(f"{plPath}/{cfg.metaDataName}", 'c',writeback=True) as metaData: 72 | correctStateCorruption(f'{cfg.testPlPath}/{name}',metaData) 73 | 74 | correct = [ ('1', '0_B'), ('3','1_D') ] 75 | 76 | 77 | result = getPlaylistData(name) 78 | 79 | shutil.rmtree(f'{cfg.testPlPath}/{name}') 80 | self.assertEqual(result,correct) 81 | 82 | def test_blankMetaData(self): 83 | name = inspect.currentframe().f_code.co_name 84 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 85 | 86 | plPath = f'{cfg.testPlPath}/{name}' 87 | 88 | songs = ['A' ,'B' ,'C' ,'D'] 89 | 90 | createFakePlaylist(name,songs) 91 | 92 | with shelve.open(f"{plPath}/{cfg.metaDataName}", 'c',writeback=True) as metaData: 93 | metaData['ids'].insert(2,'') 94 | 95 | correctStateCorruption(plPath,metaData) 96 | 97 | correct = [ ('0', '0_A'), ('1', '1_B'), ('2','2_C'), ('3','3_D') ] 98 | 99 | 100 | result = getPlaylistData(name) 101 | 102 | shutil.rmtree(f'{cfg.testPlPath}/{name}') 103 | self.assertEqual(result,correct) 104 | 105 | 106 | def test_removePrepend(self): 107 | name = inspect.currentframe().f_code.co_name 108 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 109 | 110 | songs = ['A' ,'B' ,'C' ,'D'] 111 | createFakePlaylist(name,songs) 112 | 113 | correct = [ ('0', '0_A'), ('1','1_B'), ('2','2_C'), ('3','3_D') ] 114 | 115 | plPath = f'{cfg.testPlPath}/{name}' 116 | 117 | togglePrepends(plPath) 118 | 119 | with shelve.open(f"{plPath}/{cfg.metaDataName}", 'c',writeback=True) as metaData: 120 | correctStateCorruption(plPath,metaData) 121 | 122 | result = getPlaylistData(name) 123 | shutil.rmtree(plPath) 124 | self.assertEqual(result,correct) 125 | 126 | def test_removeSongsAndPrepends(self): 127 | name = inspect.currentframe().f_code.co_name 128 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 129 | 130 | songs = ['A' ,'B' ,'C' ,'D', 'E'] 131 | 132 | createFakePlaylist(name,songs) 133 | 134 | plPath = f'{cfg.testPlPath}/{name}' 135 | 136 | os.remove(f'{plPath}/0_A') 137 | os.remove(f'{plPath}/4_E') 138 | os.remove(f'{plPath}/2_C') 139 | togglePrepends(plPath) 140 | 141 | 142 | with shelve.open(f"{plPath}/{cfg.metaDataName}", 'c',writeback=True) as metaData: 143 | correctStateCorruption(plPath,metaData) 144 | 145 | correct = [ ('1', '0_B'), ('3','1_D') ] 146 | 147 | 148 | result = getPlaylistData(name) 149 | 150 | shutil.rmtree(f'{cfg.testPlPath}/{name}') 151 | self.assertEqual(result,correct) 152 | 153 | class test_togglePrepends(unittest.TestCase): 154 | def test_togglePrepends1(self): 155 | name = inspect.currentframe().f_code.co_name 156 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 157 | 158 | songs = ['A' ,'B' ,'C' ,'D'] 159 | plPath = f'{cfg.testPlPath}/{name}' 160 | 161 | createFakePlaylist(name,songs) 162 | 163 | togglePrepends(plPath) 164 | togglePrepends(plPath) 165 | 166 | correct = [ ('0', '0_A'), ('1','1_B'), ('2','2_C'), ('3','3_D') ] 167 | 168 | result = getPlaylistData(name) 169 | shutil.rmtree(plPath) 170 | self.assertEqual(result,correct) 171 | 172 | 173 | 174 | class test_editPlaylist(unittest.TestCase): 175 | 176 | def test_editPl1(self): 177 | name = inspect.currentframe().f_code.co_name 178 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 179 | 180 | songs = ['A' ,'B' ,'C' ,'D'] 181 | 182 | createFakePlaylist(name,songs) 183 | 184 | newOrder = [ ('3', 3), ('1',1), ('2',2), ('0',0) ] 185 | 186 | correct = [ ('3', '0_D'), ('1','1_B'), ('2','2_C'), ('0','3_A') ] 187 | 188 | editPlaylist(f'{cfg.testPlPath}/{name}',newOrder) 189 | 190 | 191 | result = getPlaylistData(name) 192 | 193 | shutil.rmtree(f'{cfg.testPlPath}/{name}') 194 | self.assertEqual(result,correct) 195 | 196 | 197 | def test_editPl2(self): 198 | name = inspect.currentframe().f_code.co_name 199 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 200 | 201 | songs = ['A' ,'B' ,'C' ,'D','E','F'] 202 | 203 | createFakePlaylist(name,songs) 204 | 205 | newOrder = [ ('3', 3), ('4',4), ('2',2), ('0',0) ] 206 | 207 | correct = [ ('3', '0_D'), ('4','1_E'), ('2','2_C'), ('0','3_A') ] 208 | 209 | editPlaylist(f'{cfg.testPlPath}/{name}',newOrder,True) 210 | 211 | 212 | result = getPlaylistData(name) 213 | 214 | shutil.rmtree(f'{cfg.testPlPath}/{name}') 215 | self.assertEqual(result,correct) 216 | 217 | def test_editPl3(self): 218 | name = inspect.currentframe().f_code.co_name 219 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 220 | 221 | songs = ['A' ,'B' ,'C'] 222 | 223 | createFakePlaylist(name,songs) 224 | 225 | newOrder = [ ] 226 | 227 | correct = [ ] 228 | 229 | editPlaylist(f'{cfg.testPlPath}/{name}',newOrder,True) 230 | 231 | 232 | result = getPlaylistData(name) 233 | 234 | shutil.rmtree(f'{cfg.testPlPath}/{name}') 235 | self.assertEqual(result,correct) 236 | 237 | def test_editPl4(self): 238 | name = inspect.currentframe().f_code.co_name 239 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 240 | 241 | songs = ['A' ,'B' ,'C' ,'D', 'E', 'F', 'G'] 242 | 243 | createFakePlaylist(name,songs) 244 | 245 | newOrder = [ ('6',6), ('3', 3), ('4',4), ('5',5), ('2',2) ] 246 | 247 | correct = [ ('6', '0_G'), ('3','1_D'), ('4','2_E'),('5','3_F'), ('2','4_C') ] 248 | editPlaylist(f'{cfg.testPlPath}/{name}',newOrder,True) 249 | 250 | 251 | result = getPlaylistData(name) 252 | 253 | shutil.rmtree(f'{cfg.testPlPath}/{name}') 254 | self.assertEqual(result,correct) 255 | 256 | class test_smartSyncNewOrder(unittest.TestCase): 257 | 258 | def test_insertAndDelete(self): 259 | name = inspect.currentframe().f_code.co_name 260 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 261 | 262 | localIds = ['A' ,'B' ,'C' ,'D'] 263 | remoteIds = ['A' ,'1' ,'B' ,'C' ,'2'] 264 | 265 | correct = [('A',0) ,('1',None) ,('B',1) ,('C',2) ,('D',3) ,('2',None)] 266 | 267 | 268 | result = smartSyncNewOrder(localIds,remoteIds) 269 | self.assertEqual(result,correct) 270 | 271 | def test_insertDeleteSwap(self): 272 | name = inspect.currentframe().f_code.co_name 273 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 274 | localIds = ['A' ,'B' ,'C' ,'D'] 275 | remoteIds = ['A' ,'1' ,'C' ,'B' ,'2'] 276 | 277 | 278 | correct = [('A',0) ,('1',None) ,('C',2) ,('D',3) ,('B',1) ,('2',None)] 279 | 280 | 281 | result = smartSyncNewOrder(localIds,remoteIds) 282 | self.assertEqual(result,correct) 283 | 284 | def test_3(self): 285 | name = inspect.currentframe().f_code.co_name 286 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 287 | 288 | localIds = ['A' ,'B' ,'C' ,'D', 'E', 'F','G'] 289 | remoteIds = ['A' ,'1' ,'C' ,'B' ,'2','F'] 290 | 291 | 292 | correct = [('A',0) ,('1',None) ,('C',2),('D',3),('E',4) ,('B',1) , ('2',None), ('F',5), ('G',6)] 293 | 294 | 295 | result = smartSyncNewOrder(localIds,remoteIds) 296 | self.assertEqual(result,correct) 297 | 298 | 299 | def test_LocalDeleteAll(self): 300 | name = inspect.currentframe().f_code.co_name 301 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 302 | 303 | localIds = [] 304 | remoteIds = ['A' ,'1' ,'C' ,'B' ,'2','F'] 305 | 306 | 307 | correct = [('A',None) ,('1',None) ,('C',None),('B',None) , ('2',None), ('F',None)] 308 | 309 | 310 | result = smartSyncNewOrder(localIds,remoteIds) 311 | self.assertEqual(result,correct) 312 | 313 | def test_RemoteDeleteAll(self): 314 | name = inspect.currentframe().f_code.co_name 315 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 316 | 317 | localIds = ['A' ,'B' ,'C' ,'D', 'E', 'F','G'] 318 | remoteIds = [] 319 | 320 | 321 | correct = [('A',0) ,('B',1), ('C',2), ('D',3), ('E',4), ('F',5), ('G',6)] 322 | 323 | 324 | result = smartSyncNewOrder(localIds,remoteIds) 325 | self.assertEqual(result,correct) 326 | 327 | def test_Reversal(self): 328 | name = inspect.currentframe().f_code.co_name 329 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 330 | 331 | localIds = ['A' ,'B' ,'C' ,'D', 'E'] 332 | remoteIds = ['E','D','C','B','A'] 333 | 334 | 335 | correct = [('E',4), ('D',3), ('C',2), ('B',1), ('A',0)] 336 | 337 | 338 | result = smartSyncNewOrder(localIds,remoteIds) 339 | self.assertEqual(result,correct) 340 | 341 | 342 | def test_7(self): 343 | name = inspect.currentframe().f_code.co_name 344 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 345 | 346 | localIds = ['A' ,'B' ,'C' ,'D', 'E'] 347 | remoteIds = ['E','1','D','2','B','A'] 348 | 349 | 350 | correct = [('E',4), ('1',None),('D',3),('2',None), ('B',1), ('C',2), ('A',0)] 351 | 352 | 353 | result = smartSyncNewOrder(localIds,remoteIds) 354 | self.assertEqual(result,correct) 355 | 356 | 357 | class test_move(unittest.TestCase): 358 | 359 | def test_moveLarger(self): 360 | name = inspect.currentframe().f_code.co_name 361 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 362 | 363 | songs = ['A' ,'B' ,'C' ,'D','E'] 364 | 365 | createFakePlaylist(name,songs) 366 | 367 | 368 | correct = [ ('0', '0_A'), ('2','1_C'), ('3','2_D'), ('1','3_B'), ('4','4_E') ] 369 | 370 | plPath = f'{cfg.testPlPath}/{name}' 371 | move(plPath,1,3) 372 | 373 | result = getPlaylistData(name) 374 | 375 | shutil.rmtree(plPath) 376 | self.assertEqual(result,correct) 377 | 378 | def test_moveSmaller(self): 379 | name = inspect.currentframe().f_code.co_name 380 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 381 | 382 | songs = ['A' ,'B' ,'C' ,'D','E'] 383 | 384 | createFakePlaylist(name,songs) 385 | 386 | 387 | correct = [ ('2', '0_C'), ('0','1_A'), ('1','2_B'), ('3','3_D'), ('4','4_E') ] 388 | 389 | plPath = f'{cfg.testPlPath}/{name}' 390 | move(plPath,2,0) 391 | 392 | result = getPlaylistData(name) 393 | 394 | shutil.rmtree(plPath) 395 | self.assertEqual(result,correct) 396 | 397 | 398 | class test_swap(unittest.TestCase): 399 | 400 | def test_swap1(self): 401 | name = inspect.currentframe().f_code.co_name 402 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 403 | songs = ['A' ,'B' ,'C' ,'D','E'] 404 | 405 | createFakePlaylist(name,songs) 406 | 407 | 408 | correct = [ ('0', '0_A'), ('3','1_D'), ('2','2_C'), ('1','3_B'), ('4','4_E') ] 409 | 410 | plPath = f'{cfg.testPlPath}/{name}' 411 | swap(plPath,1,3) 412 | 413 | result = getPlaylistData(name) 414 | 415 | shutil.rmtree(plPath) 416 | self.assertEqual(result,correct) 417 | 418 | def test_swap2(self): 419 | name = inspect.currentframe().f_code.co_name 420 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 421 | songs = ['A' ,'B' ,'C' ,'D','E'] 422 | 423 | createFakePlaylist(name,songs) 424 | 425 | 426 | correct = [ ('4', '0_E'), ('1','1_B'), ('2','2_C'), ('3','3_D'), ('0','4_A') ] 427 | 428 | plPath = f'{cfg.testPlPath}/{name}' 429 | swap(plPath,0,4) 430 | 431 | result = getPlaylistData(name) 432 | 433 | shutil.rmtree(plPath) 434 | self.assertEqual(result,correct) 435 | 436 | 437 | class test_manualAdd(unittest.TestCase): 438 | def test_manualAdd1(self): 439 | name = inspect.currentframe().f_code.co_name 440 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 441 | songs = ['A' ,'B' ,'C' ,'D','E'] 442 | 443 | createFakePlaylist(name,songs) 444 | 445 | songPath = f"{cfg.testPlPath}/X" 446 | open(songPath,'a').close() 447 | 448 | 449 | correct = [ ('0', '0_A'), ('1','1_B'), ('2','2_C'), ('3','3_D'),(cfg.manualAddId,'4_X'), ('4','5_E') ] 450 | plPath = f'{cfg.testPlPath}/{name}' 451 | manualAdd(plPath,songPath,4) 452 | 453 | result = getPlaylistData(name) 454 | 455 | shutil.rmtree(plPath) 456 | self.assertEqual(result,correct) 457 | 458 | 459 | def test_manualAdd2(self): 460 | name = inspect.currentframe().f_code.co_name 461 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 462 | songs = ['A' ,'B' ,'C' ,'D','E'] 463 | 464 | createFakePlaylist(name,songs) 465 | 466 | songPath = f"{cfg.testPlPath}/X" 467 | open(songPath,'a').close() 468 | 469 | 470 | correct = [ (cfg.manualAddId,'0_X'), ('0', '1_A'), ('1','2_B'), ('2','3_C'), ('3','4_D'), ('4','5_E') ] 471 | plPath = f'{cfg.testPlPath}/{name}' 472 | manualAdd(plPath,songPath,0) 473 | 474 | result = getPlaylistData(name) 475 | 476 | shutil.rmtree(plPath) 477 | self.assertEqual(result,correct) 478 | 479 | 480 | 481 | class test_moveRange(unittest.TestCase): 482 | 483 | def test_moveRangeLarger1(self): 484 | 485 | name = inspect.currentframe().f_code.co_name 486 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 487 | songs = ['A' ,'B' ,'C' ,'D','E','F','G'] 488 | 489 | createFakePlaylist(name,songs) 490 | 491 | 492 | correct = [ ('0', '0_A'), ('4','1_E'), ('5','2_F'), ('1','3_B'), ('2','4_C'), ('3','5_D'), ('6','6_G') ] 493 | 494 | plPath = f'{cfg.testPlPath}/{name}' 495 | moveRange(plPath,1,3,5) 496 | 497 | result = getPlaylistData(name) 498 | 499 | shutil.rmtree(plPath) 500 | self.assertEqual(result,correct) 501 | 502 | def test_moveRangeLarger2(self): 503 | 504 | name = inspect.currentframe().f_code.co_name 505 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 506 | songs = ['A' ,'B' ,'C' ,'D','E','F','G'] 507 | 508 | createFakePlaylist(name,songs) 509 | 510 | 511 | correct = [ ('3','0_D'),('4','1_E'), ('5','2_F'), ('6','3_G'),('0', '4_A'), ('1','5_B'), ('2','6_C') ] 512 | 513 | plPath = f'{cfg.testPlPath}/{name}' 514 | moveRange(plPath,0,2,6) 515 | 516 | result = getPlaylistData(name) 517 | 518 | shutil.rmtree(plPath) 519 | self.assertEqual(result,correct) 520 | 521 | def test_moveRangeSmaller1(self): 522 | 523 | name = inspect.currentframe().f_code.co_name 524 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 525 | songs = ['A' ,'B' ,'C' ,'D','E','F','G'] 526 | 527 | createFakePlaylist(name,songs) 528 | 529 | 530 | correct = [ ('3','0_D'), ('4','1_E'),('5','2_F'),('0', '3_A'), ('1','4_B'), ('2','5_C') , ('6','6_G') ] 531 | 532 | plPath = f'{cfg.testPlPath}/{name}' 533 | moveRange(plPath,3,5,-1) 534 | 535 | result = getPlaylistData(name) 536 | 537 | shutil.rmtree(plPath) 538 | self.assertEqual(result,correct) 539 | 540 | 541 | def test_moveRangeSmaller2(self): 542 | 543 | name = inspect.currentframe().f_code.co_name 544 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 545 | songs = ['A' ,'B' ,'C' ,'D','E','F','G'] 546 | 547 | createFakePlaylist(name,songs) 548 | 549 | correct = [ ('0', '0_A'), ('1','1_B'),('4','2_E'),('5','3_F'),('6','4_G'), ('2','5_C'), ('3','6_D') ] 550 | 551 | plPath = f'{cfg.testPlPath}/{name}' 552 | moveRange(plPath,4,-1,1) 553 | 554 | result = getPlaylistData(name) 555 | 556 | shutil.rmtree(plPath) 557 | self.assertEqual(result,correct) 558 | 559 | def test_moveRangeSingleSmaller(self): 560 | 561 | name = inspect.currentframe().f_code.co_name 562 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 563 | songs = ['A' ,'B' ,'C' ,'D','E','F','G'] 564 | 565 | createFakePlaylist(name,songs) 566 | 567 | correct = [ ('0', '0_A'), ('1','1_B'),('4','2_E'), ('2','3_C'), ('3','4_D'),('5','5_F'),('6','6_G') ] 568 | 569 | plPath = f'{cfg.testPlPath}/{name}' 570 | moveRange(plPath,4,4,1) 571 | 572 | result = getPlaylistData(name) 573 | 574 | shutil.rmtree(plPath) 575 | self.assertEqual(result,correct) 576 | 577 | def test_moveRangeSingleLarger(self): 578 | 579 | name = inspect.currentframe().f_code.co_name 580 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 581 | songs = ['A' ,'B' ,'C' ,'D','E','F','G'] 582 | 583 | createFakePlaylist(name,songs) 584 | 585 | correct = [ ('0', '0_A'), ('1','1_B'), ('2','2_C'),('4','3_E'),('5','4_F'), ('3','5_D'),('6','6_G') ] 586 | 587 | plPath = f'{cfg.testPlPath}/{name}' 588 | moveRange(plPath,3,3,5) 589 | 590 | result = getPlaylistData(name) 591 | 592 | shutil.rmtree(plPath) 593 | self.assertEqual(result,correct) 594 | 595 | ################################# 596 | ## youtube api submodule tests ## 597 | ################################# 598 | 599 | class test_yt_api_helpers(unittest.TestCase): 600 | def test_longestIncreasingSequence(self): 601 | name = inspect.currentframe().f_code.co_name 602 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 603 | 604 | def acending(numList): 605 | 606 | for i in range(1,len(numList)): 607 | if numList[i] <= numList[i-1]: 608 | return False 609 | return True 610 | 611 | def isSubSequenceInSequence(subSequence, sequence): 612 | prevIndex = 0 613 | for _,num in enumerate(subSequence): 614 | try: 615 | prevIndex = sequence[prevIndex:].index(num) + 1 616 | except: 617 | return False 618 | return True 619 | 620 | pairs = [ 621 | ([7, 9, 1, 4, 0, 8, 3, 6, 5, 2],3), 622 | ([5, 4, 2, 1, 0, 6, 3, 9, 8, 7],3), 623 | ([4, 1, 9, 5, 3, 7, 0, 6, 2, 8],4), 624 | ([2, 0, 5, 7, 4, 9, 8, 6, 1, 3],4), 625 | ([0, 1, 7, 5, 9, 6, 4, 8, 3, 2],5), 626 | ([0, 4, 5, 7, 9, 8, 2, 3, 1, 6],5), 627 | ([5, 0, 9, 1, 2, 8, 4, 3, 7, 6],5), 628 | ([5, 4, 6, 0, 2, 8, 3, 9, 1, 7],4), 629 | ([5, 2, 1, 4, 0, 8, 9, 6, 7, 3],4), 630 | ([2, 0, 3, 8, 5, 6, 4, 7, 9, 1],6), 631 | ([1, 3, 5, 9, 4, 8, 2, 0, 6, 7],5), 632 | ([9, 2, 6, 4, 3, 5, 7, 0, 8, 1],5), 633 | ([7, 1, 2, 8, 9, 0, 5, 3, 4, 6],5), 634 | ([3, 4, 8, 1, 9, 5, 6, 2, 0, 7],5), 635 | ([4, 1, 2, 8, 7, 0, 9, 3, 5, 6],5), 636 | ([1, 4, 2, 0, 7, 9, 3, 6, 5, 8],5), 637 | ([6, 0, 2, 3, 8, 9, 1, 4, 5, 7],6), 638 | ([4, 1, 5, 2, 8, 0, 9, 3, 6, 7],5), 639 | ([4, 8, 0, 2, 9, 7, 6, 1, 3, 5],4), 640 | ] 641 | 642 | for pair in pairs: 643 | numList,correctLen = pair 644 | ans = longestIncreasingSequence(numList) 645 | 646 | if not acending(ans): 647 | self.fail(f'Answer is not acending! \nanswer: {ans} \ninput: {numList}') 648 | if len(ans)!=correctLen: 649 | self.fail(f'Answer has Length {len(ans)}, correct Length is {correctLen}. \nanswer: {ans} \ninput: {numList}') 650 | 651 | if not isSubSequenceInSequence(ans,numList): 652 | self.fail(f'Answer Subsequence is not in Input Sequence. \nanswer: {ans} \ninput: {numList}') 653 | 654 | 655 | 656 | class test_yt_api_getNewRemoteOrder(unittest.TestCase): 657 | def test_insertAndDelete(self): 658 | name = inspect.currentframe().f_code.co_name 659 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 660 | 661 | localIds = ['A','1' ,'C','2' ,'D'] 662 | remoteIds = ['A','B','C','D'] 663 | 664 | 665 | correct = [0, 1 ,2 ,3] 666 | 667 | 668 | result = oldToNewPushOrder(remoteIds,localIds) 669 | self.assertEqual(result,correct) 670 | 671 | def test_insertDeleteSwap(self): 672 | name = inspect.currentframe().f_code.co_name 673 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 674 | 675 | localIds = ['C','1','A','2' ,'D'] 676 | remoteIds = ['A','B','C','D'] 677 | 678 | correct = [1,2,0,3] 679 | 680 | 681 | result = oldToNewPushOrder(remoteIds,localIds) 682 | 683 | self.assertEqual(result,correct) 684 | 685 | def test_3(self): 686 | name = inspect.currentframe().f_code.co_name 687 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 688 | 689 | localIds = ['A', '1', 'E','B', 'D', 'C', '2'] 690 | remoteIds = ['A','B','C','D','E','F','G'] 691 | correct = [ 0, 4, 6, 5, 1, 2, 3 ] 692 | 693 | 694 | result = oldToNewPushOrder(remoteIds,localIds) 695 | self.assertEqual(result,correct) 696 | 697 | 698 | def test_4(self): 699 | name = inspect.currentframe().f_code.co_name 700 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 701 | 702 | localIds = ['1', 'E', 'D', 'C', '2'] 703 | remoteIds = ['A','B','C','D','E','F','G'] 704 | correct = [ 0, 1, 6, 5, 2, 3, 4 ] 705 | 706 | 707 | result = oldToNewPushOrder(remoteIds,localIds) 708 | self.assertEqual(result,correct) 709 | 710 | def test_5(self): 711 | name = inspect.currentframe().f_code.co_name 712 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 713 | 714 | localIds = [] 715 | remoteIds = ['A','B','C','D','E','F','G'] 716 | correct = [ 0, 1, 2, 3, 4, 5, 6 ] 717 | 718 | 719 | result = oldToNewPushOrder(remoteIds,localIds) 720 | self.assertEqual(result,correct) 721 | 722 | 723 | def test_6(self): 724 | name = inspect.currentframe().f_code.co_name 725 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 726 | 727 | localIds = ['A','B','C','D','E','F','G'] 728 | remoteIds = [] 729 | correct = [] 730 | 731 | 732 | result = oldToNewPushOrder(remoteIds,localIds) 733 | self.assertEqual(result,correct) 734 | 735 | 736 | def test_7(self): 737 | name = inspect.currentframe().f_code.co_name 738 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 739 | 740 | localIds = [ 741 | 'NigPr2mmENA',#0 742 | 'DzoYNerl4P8',#1 743 | '6WYsoeVNEDY',#2 744 | '0Wc2Og4vr2I',#3 745 | 'jhFDyDgMVUI',#4 746 | 'MfB3l3PHEFQ',#5 747 | 'I3EEhcLlKdo',#6 748 | 'bYwnf8r92G0',#7 749 | 'HiLKVdNJt_o',#8 750 | 'irIxkjXOp5M',#9 751 | '8Yuorb-rB9A',# 752 | 'lUYZJRlOHyw',#10 753 | 'p1cymwvDf0w',#11 754 | '4TAGhEVNCFc',#12 755 | 'RkMzJGAV7Vs',#13 756 | 'BhEfvGZM6jw',#14 757 | ] 758 | remoteIds = [ 759 | 'MfB3l3PHEFQ',#5 760 | 'DzoYNerl4P8',#1 761 | '6WYsoeVNEDY',#2 762 | '0Wc2Og4vr2I',#3 763 | 'jhFDyDgMVUI',#4 764 | 'I3EEhcLlKdo',#6 765 | 'bYwnf8r92G0',#7 766 | 'HiLKVdNJt_o',#8 767 | 'irIxkjXOp5M',#9 768 | 'lUYZJRlOHyw',#10 769 | 'p1cymwvDf0w',#11 770 | '4TAGhEVNCFc',#12 771 | 'RkMzJGAV7Vs',#13 772 | 'BhEfvGZM6jw',#14 773 | 'NigPr2mmENA',#0 774 | ] 775 | 776 | correct = [5,1,2,3,4,6,7,8,9,10,11,12,13,14,0] 777 | 778 | 779 | result = oldToNewPushOrder(remoteIds,localIds) 780 | self.assertEqual(result,correct) 781 | 782 | 783 | 784 | # Helpers for test_yt_api_pushOrderMoves 785 | def simulateMove(remoteIds,remoteItemIds,move): 786 | newIndex,_,remoteItemId = move 787 | oldIndex = remoteItemIds.index(remoteItemId) 788 | 789 | remoteIds.insert(newIndex,remoteIds.pop(oldIndex)) 790 | remoteItemIds.insert(newIndex,remoteItemIds.pop(oldIndex)) 791 | 792 | def compareLocalAndRemote(remoteIds, localIds): 793 | remoteIndex = 0 794 | localIndex = 0 795 | while True: 796 | if remoteIndex>=len(remoteIds) or localIndex>=len(localIds): 797 | break 798 | 799 | remoteId = remoteIds[remoteIndex] 800 | localId = localIds[localIndex] 801 | if remoteId not in localIds: 802 | remoteIndex+=1 803 | continue 804 | 805 | if localId not in remoteIds: 806 | localIndex+=1 807 | continue 808 | 809 | if remoteId!=localId: 810 | return False 811 | 812 | remoteIndex+=1 813 | localIndex+=1 814 | 815 | return True 816 | 817 | def remoteCorrectOrder(remoteIds,localIds): 818 | ''' any id not in local must remain after the id that came before it before moving ''' 819 | 820 | 821 | alphaToNum = lambda char: ord(char.lower()) - 96 822 | 823 | for i in range(1,len(remoteIds)): 824 | currentId = remoteIds[i] 825 | prevId = remoteIds[i-1] 826 | 827 | if currentId not in localIds: 828 | if alphaToNum(currentId)!= alphaToNum(prevId)+1: 829 | return False 830 | 831 | return True 832 | 833 | 834 | 835 | class test_yt_api_pushOrderMoves(unittest.TestCase): 836 | def test_1(self): 837 | name = inspect.currentframe().f_code.co_name 838 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 839 | 840 | localIds = ['A','1','B','C'] 841 | remoteIds = ['A','B','C','D','E','F','G'] # must be in alphabetical order for remoteCorrectOrder 842 | remoteItemIds = ['A','B','C','D','E','F','G'] 843 | 844 | moves = pushOrderMoves(remoteIds,remoteItemIds,localIds) 845 | for move in moves: 846 | simulateMove(remoteIds,remoteItemIds,move) 847 | 848 | # Checks if remote ids have taken on localIds order 849 | if not compareLocalAndRemote(remoteIds,localIds): 850 | self.fail(f'RemoteIds Do Not Match LocalIds After Moves\nremoteIds: {remoteIds}\nlocalIds: {localIds}') 851 | 852 | # Checks if any remoteIds not in localIds stayed after the correct Id 853 | if not remoteCorrectOrder(remoteIds,localIds): 854 | self.fail(f'RemoteIds Ids Not In Correct Order After Moves\nremoteIds: {remoteIds}\nlocalIds: {localIds}') 855 | 856 | def test_2(self): 857 | name = inspect.currentframe().f_code.co_name 858 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 859 | 860 | localIds = ['1','B','E','A','C'] 861 | remoteIds = ['A','B','C','D','E','F','G'] # must be in alphabetical order for remoteCorrectOrder 862 | remoteItemIds = ['A','B','C','D','E','F','G'] 863 | 864 | moves = pushOrderMoves(remoteIds,remoteItemIds,localIds) 865 | for move in moves: 866 | simulateMove(remoteIds,remoteItemIds,move) 867 | 868 | # Checks if remote ids have taken on localIds order 869 | if not compareLocalAndRemote(remoteIds,localIds): 870 | self.fail(f'RemoteIds Do Not Match LocalIds After Moves\nremoteIds: {remoteIds}\nlocalIds: {localIds}') 871 | 872 | # Checks if any remoteIds not in localIds stayed after the correct Id 873 | if not remoteCorrectOrder(remoteIds,localIds): 874 | self.fail(f'RemoteIds Ids Not In Correct Order After Moves\nremoteIds: {remoteIds}\nlocalIds: {localIds}') 875 | 876 | def test_3(self): 877 | name = inspect.currentframe().f_code.co_name 878 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 879 | 880 | localIds = ['1','F','B','A','C'] 881 | remoteIds = ['A','B','C','D','E','F','G'] # must be in alphabetical order for remoteCorrectOrder 882 | remoteItemIds = ['A','B','C','D','E','F','G'] 883 | 884 | moves = pushOrderMoves(remoteIds,remoteItemIds,localIds) 885 | for move in moves: 886 | simulateMove(remoteIds,remoteItemIds,move) 887 | 888 | # Checks if remote ids have taken on localIds order 889 | if not compareLocalAndRemote(remoteIds,localIds): 890 | self.fail(f'RemoteIds Do Not Match LocalIds After Moves\nremoteIds: {remoteIds}\nlocalIds: {localIds}') 891 | 892 | # Checks if any remoteIds not in localIds stayed after the correct Id 893 | if not remoteCorrectOrder(remoteIds,localIds): 894 | self.fail(f'RemoteIds Ids Not In Correct Order After Moves\nremoteIds: {remoteIds}\nlocalIds: {localIds}') 895 | 896 | def test_reversal(self): 897 | name = inspect.currentframe().f_code.co_name 898 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 899 | 900 | localIds = ['G','F','E','D','C','B','A'] 901 | remoteIds = ['A','B','C','D','E','F','G'] # must be in alphabetical order for remoteCorrectOrder 902 | remoteItemIds = ['A','B','C','D','E','F','G'] 903 | 904 | moves = pushOrderMoves(remoteIds,remoteItemIds,localIds) 905 | for move in moves: 906 | simulateMove(remoteIds,remoteItemIds,move) 907 | 908 | # Checks if remote ids have taken on localIds order 909 | if not compareLocalAndRemote(remoteIds,localIds): 910 | self.fail(f'RemoteIds Do Not Match LocalIds After Moves\nremoteIds: {remoteIds}\nlocalIds: {localIds}') 911 | 912 | # Checks if any remoteIds not in localIds stayed after the correct Id 913 | if not remoteCorrectOrder(remoteIds,localIds): 914 | self.fail(f'RemoteIds Ids Not In Correct Order After Moves\nremoteIds: {remoteIds}\nlocalIds: {localIds}') 915 | 916 | 917 | def test_partialReversal(self): 918 | name = inspect.currentframe().f_code.co_name 919 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 920 | 921 | localIds = ['F','E','C','B'] 922 | remoteIds = ['A','B','C','D','E','F','G'] # must be in alphabetical order for remoteCorrectOrder 923 | remoteItemIds = ['A','B','C','D','E','F','G'] 924 | 925 | moves = pushOrderMoves(remoteIds,remoteItemIds,localIds) 926 | for move in moves: 927 | simulateMove(remoteIds,remoteItemIds,move) 928 | 929 | # Checks if remote ids have taken on localIds order 930 | if not compareLocalAndRemote(remoteIds,localIds): 931 | self.fail(f'RemoteIds Do Not Match LocalIds After Moves\nremoteIds: {remoteIds}\nlocalIds: {localIds}') 932 | 933 | # Checks if any remoteIds not in localIds stayed after the correct Id 934 | if not remoteCorrectOrder(remoteIds,localIds): 935 | self.fail(f'RemoteIds Ids Not In Correct Order After Moves\nremoteIds: {remoteIds}\nlocalIds: {localIds}') 936 | 937 | 938 | 939 | def test_firstLastSwap(self): 940 | name = inspect.currentframe().f_code.co_name 941 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 942 | 943 | localIds = ['G','1','B','E','C','A'] 944 | remoteIds = ['A','B','C','D','E','F','G'] # must be in alphabetical order for remoteCorrectOrder 945 | remoteItemIds = ['A','B','C','D','E','F','G'] 946 | 947 | moves = pushOrderMoves(remoteIds,remoteItemIds,localIds) 948 | for move in moves: 949 | simulateMove(remoteIds,remoteItemIds,move) 950 | 951 | # Checks if remote ids have taken on localIds order 952 | if not compareLocalAndRemote(remoteIds,localIds): 953 | self.fail(f'RemoteIds Do Not Match LocalIds After Moves\nremoteIds: {remoteIds}\nlocalIds: {localIds}') 954 | 955 | # Checks if any remoteIds not in localIds stayed after the correct Id 956 | if not remoteCorrectOrder(remoteIds,localIds): 957 | self.fail(f'RemoteIds Ids Not In Correct Order After Moves\nremoteIds: {remoteIds}\nlocalIds: {localIds}') 958 | 959 | # helper for test_calculateTransferMoves 960 | def getGenericState(length: int, filePrepends: str = '', idPrepends: str = ''): 961 | if length > len(ascii_uppercase): 962 | raise Exception() 963 | files = list(ascii_uppercase[0:length]) 964 | ids = list(ascii_uppercase[0:length]) 965 | for i in range(0, length): 966 | files[i] = filePrepends + files[i] 967 | ids[i] = idPrepends + ids[i] 968 | 969 | return files, ids 970 | 971 | 972 | def _setPadded(l: list, index: int, ele): 973 | for _ in range(index - len(l) + 1): 974 | l.append('') 975 | l[index] = ele 976 | 977 | def _movePadded(l: list, srcIndex: int, destIndex: int): 978 | for _ in range(destIndex - len(l) + 1): 979 | l.append('') 980 | l[destIndex] = l[srcIndex] 981 | l[srcIndex] = '' 982 | 983 | def _removeBlanks(l: list): 984 | for i in reversed(range(len(l))): 985 | if l[i] == '': 986 | l.pop(i) 987 | 988 | 989 | def simulateTransferMoves(moves: List[TransferMove], srcStart: int, srcEnd: int, destIndex: int, 990 | srcDir:list, destDir:list, srcLocalIds:list, destLocalIds:list, srcRemoteIds:list, destRemoteIDs:list): 991 | 992 | # make room for block 993 | blockSize = srcEnd - srcStart + 1 994 | for i in reversed(range(destIndex+1,len(destLocalIds))): 995 | _movePadded(destDir, i, i+blockSize) 996 | _movePadded(destLocalIds, i, i+blockSize) 997 | 998 | 999 | for move in moves: 1000 | if move.performCopy: 1001 | _setPadded(destDir, move.destCopyIndex, srcDir[move.srcCopyIndex]) 1002 | _setPadded(destLocalIds, move.destCopyIndex, srcLocalIds[move.srcCopyIndex]) 1003 | 1004 | if move.performRemoteAdd: 1005 | destRemoteIDs.insert(move.destRemoteAddIndex, move.songId) 1006 | 1007 | if move.performLocalDelete: 1008 | srcDir.pop(move.srcLocalDeleteIndex) 1009 | srcLocalIds.pop(move.srcLocalDeleteIndex) 1010 | 1011 | # remote deletes are done using playlist item ids, whos posisitions are calculated at the start of the 1012 | # process, hence we must keep the posistion of remote list items static during testing to mimic this 1013 | if move.performRemoteDelete: 1014 | srcRemoteIds[move.srcRemoteDeleteIndex] = '' 1015 | 1016 | _removeBlanks(srcRemoteIds) 1017 | 1018 | 1019 | 1020 | 1021 | class test_calculateTransferMoves(unittest.TestCase): 1022 | def test_emptyDestIdenticalRemote(self): 1023 | name = inspect.currentframe().f_code.co_name 1024 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 1025 | 1026 | srcDir, srcLocalIds = getGenericState(5, 'f', 'i') 1027 | srcRemoteIds = srcLocalIds.copy() 1028 | 1029 | destLocalIds = [] 1030 | destRemoteIds = [] 1031 | destDir = [] 1032 | 1033 | srcStart = 1 1034 | srcEnd = 3 1035 | destIndex = -1 1036 | 1037 | moves = calcuateTransferMoves(srcDir, srcLocalIds, destLocalIds, srcRemoteIds, destRemoteIds, srcStart, srcEnd, destIndex) 1038 | simulateTransferMoves(moves, srcStart, srcEnd, destIndex, srcDir, destDir, srcLocalIds, destLocalIds, srcRemoteIds, destRemoteIds) 1039 | 1040 | self.assertListEqual(srcDir, ['fA', 'fE']) 1041 | self.assertListEqual(srcLocalIds, ['iA', 'iE']) 1042 | self.assertListEqual(srcRemoteIds, ['iA', 'iE']) 1043 | 1044 | self.assertListEqual(destDir, ['fB', 'fC', 'fD']) 1045 | self.assertListEqual(destLocalIds, ['iB', 'iC', 'iD']) 1046 | self.assertListEqual(destRemoteIds, ['iB', 'iC', 'iD']) 1047 | 1048 | def test_appendDestIdenticalRemote(self): 1049 | name = inspect.currentframe().f_code.co_name 1050 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 1051 | 1052 | srcDir, srcLocalIds = getGenericState(5, 's', 's') 1053 | srcRemoteIds = srcLocalIds.copy() 1054 | 1055 | destDir, destLocalIds = getGenericState(3, 'd', 'd') 1056 | destRemoteIds = destLocalIds.copy() 1057 | 1058 | srcStart = 1 1059 | srcEnd = 3 1060 | destIndex = len(destDir) - 1 1061 | 1062 | moves = calcuateTransferMoves(srcDir, srcLocalIds, destLocalIds, srcRemoteIds, destRemoteIds, srcStart, srcEnd, destIndex) 1063 | simulateTransferMoves(moves, srcStart, srcEnd, destIndex, srcDir, destDir, srcLocalIds, destLocalIds, srcRemoteIds, destRemoteIds) 1064 | 1065 | self.assertListEqual(srcDir, ['sA', 'sE']) 1066 | self.assertListEqual(srcLocalIds, ['sA', 'sE']) 1067 | self.assertListEqual(srcRemoteIds, ['sA', 'sE']) 1068 | 1069 | self.assertListEqual(destDir, ['dA', 'dB', 'dC', 'sB', 'sC', 'sD']) 1070 | self.assertListEqual(destLocalIds, ['dA', 'dB', 'dC', 'sB', 'sC', 'sD']) 1071 | self.assertListEqual(destRemoteIds, ['dA', 'dB', 'dC', 'sB', 'sC', 'sD']) 1072 | 1073 | def test_prependDestIdenticalRemote(self): 1074 | name = inspect.currentframe().f_code.co_name 1075 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 1076 | 1077 | srcDir, srcLocalIds = getGenericState(5, 's', 's') 1078 | srcRemoteIds = srcLocalIds.copy() 1079 | 1080 | destDir, destLocalIds = getGenericState(3, 'd', 'd') 1081 | destRemoteIds = destLocalIds.copy() 1082 | 1083 | srcStart = 1 1084 | srcEnd = 3 1085 | destIndex = -1 1086 | 1087 | moves = calcuateTransferMoves(srcDir, srcLocalIds, destLocalIds, srcRemoteIds, destRemoteIds, srcStart, srcEnd, destIndex) 1088 | simulateTransferMoves(moves, srcStart, srcEnd, destIndex, srcDir, destDir, srcLocalIds, destLocalIds, srcRemoteIds, destRemoteIds) 1089 | 1090 | self.assertListEqual(srcDir, ['sA', 'sE']) 1091 | self.assertListEqual(srcLocalIds, ['sA', 'sE']) 1092 | self.assertListEqual(srcRemoteIds, ['sA', 'sE']) 1093 | 1094 | self.assertListEqual(destDir, ['sB', 'sC', 'sD', 'dA', 'dB', 'dC']) 1095 | self.assertListEqual(destLocalIds, ['sB', 'sC', 'sD', 'dA', 'dB', 'dC']) 1096 | self.assertListEqual(destRemoteIds, ['sB', 'sC', 'sD', 'dA', 'dB', 'dC']) 1097 | 1098 | def test_middleDestIdenticalRemote(self): 1099 | name = inspect.currentframe().f_code.co_name 1100 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 1101 | 1102 | srcDir, srcLocalIds = getGenericState(5, 's', 's') 1103 | srcRemoteIds = srcLocalIds.copy() 1104 | 1105 | destDir, destLocalIds = getGenericState(3, 'd', 'd') 1106 | destRemoteIds = destLocalIds.copy() 1107 | 1108 | srcStart = 0 1109 | srcEnd = 3 1110 | destIndex = 0 1111 | 1112 | moves = calcuateTransferMoves(srcDir, srcLocalIds, destLocalIds, srcRemoteIds, destRemoteIds, srcStart, srcEnd, destIndex) 1113 | simulateTransferMoves(moves, srcStart, srcEnd, destIndex, srcDir, destDir, srcLocalIds, destLocalIds, srcRemoteIds, destRemoteIds) 1114 | 1115 | self.assertListEqual(srcDir, ['sE']) 1116 | self.assertListEqual(srcLocalIds, ['sE']) 1117 | self.assertListEqual(srcRemoteIds, ['sE']) 1118 | 1119 | self.assertListEqual(destDir, ['dA', 'sA', 'sB', 'sC', 'sD', 'dB', 'dC']) 1120 | self.assertListEqual(destLocalIds, ['dA', 'sA', 'sB', 'sC', 'sD', 'dB', 'dC']) 1121 | self.assertListEqual(destRemoteIds, ['dA', 'sA', 'sB', 'sC', 'sD', 'dB', 'dC']) 1122 | 1123 | def test_AllSrcIdenticalRemote(self): 1124 | name = inspect.currentframe().f_code.co_name 1125 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 1126 | 1127 | srcDir, srcLocalIds = getGenericState(5, 's', 's') 1128 | srcRemoteIds = srcLocalIds.copy() 1129 | 1130 | destDir, destLocalIds = getGenericState(3, 'd', 'd') 1131 | destRemoteIds = destLocalIds.copy() 1132 | 1133 | srcStart = 0 1134 | srcEnd = len(srcDir) - 1 1135 | destIndex = 0 1136 | 1137 | moves = calcuateTransferMoves(srcDir, srcLocalIds, destLocalIds, srcRemoteIds, destRemoteIds, srcStart, srcEnd, destIndex) 1138 | simulateTransferMoves(moves, srcStart, srcEnd, destIndex, srcDir, destDir, srcLocalIds, destLocalIds, srcRemoteIds, destRemoteIds) 1139 | 1140 | self.assertListEqual(srcDir, []) 1141 | self.assertListEqual(srcLocalIds, []) 1142 | self.assertListEqual(srcRemoteIds, []) 1143 | 1144 | self.assertListEqual(destDir, ['dA', 'sA', 'sB', 'sC', 'sD', 'sE', 'dB', 'dC']) 1145 | self.assertListEqual(destLocalIds, ['dA', 'sA', 'sB', 'sC', 'sD', 'sE', 'dB', 'dC']) 1146 | self.assertListEqual(destRemoteIds, ['dA', 'sA', 'sB', 'sC', 'sD', 'sE', 'dB', 'dC']) 1147 | 1148 | def test_srcRemoteIsSubset1(self): 1149 | name = inspect.currentframe().f_code.co_name 1150 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 1151 | 1152 | srcDir, srcLocalIds = getGenericState(5, 's', 's') 1153 | srcRemoteIds = srcLocalIds.copy() 1154 | srcRemoteIds.pop(1) 1155 | 1156 | destDir, destLocalIds = getGenericState(3, 'd', 'd') 1157 | destRemoteIds = destLocalIds.copy() 1158 | 1159 | srcStart = 1 1160 | srcEnd = 2 1161 | destIndex = 0 1162 | 1163 | moves = calcuateTransferMoves(srcDir, srcLocalIds, destLocalIds, srcRemoteIds, destRemoteIds, srcStart, srcEnd, destIndex) 1164 | simulateTransferMoves(moves, srcStart, srcEnd, destIndex, srcDir, destDir, srcLocalIds, destLocalIds, srcRemoteIds, destRemoteIds) 1165 | 1166 | self.assertListEqual(srcDir, ['sA', 'sD', 'sE']) 1167 | self.assertListEqual(srcLocalIds, ['sA', 'sD', 'sE']) 1168 | self.assertListEqual(srcRemoteIds, ['sA', 'sD', 'sE']) 1169 | 1170 | self.assertListEqual(destDir, ['dA', 'sB', 'sC', 'dB', 'dC']) 1171 | self.assertListEqual(destLocalIds, ['dA', 'sB', 'sC', 'dB', 'dC']) 1172 | self.assertListEqual(destRemoteIds, ['dA', 'sB', 'sC', 'dB', 'dC']) 1173 | 1174 | def test_srcRemoteIsSubset2(self): 1175 | name = inspect.currentframe().f_code.co_name 1176 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 1177 | 1178 | srcDir, srcLocalIds = getGenericState(5, 's', 's') 1179 | srcRemoteIds = srcLocalIds.copy() 1180 | srcRemoteIds.pop(2) 1181 | srcRemoteIds.pop(1) 1182 | 1183 | destDir, destLocalIds = getGenericState(3, 'd', 'd') 1184 | destRemoteIds = destLocalIds.copy() 1185 | 1186 | srcStart = 1 1187 | srcEnd = 2 1188 | destIndex = 0 1189 | 1190 | moves = calcuateTransferMoves(srcDir, srcLocalIds, destLocalIds, srcRemoteIds, destRemoteIds, srcStart, srcEnd, destIndex) 1191 | simulateTransferMoves(moves, srcStart, srcEnd, destIndex, srcDir, destDir, srcLocalIds, destLocalIds, srcRemoteIds, destRemoteIds) 1192 | 1193 | self.assertListEqual(srcDir, ['sA', 'sD', 'sE']) 1194 | self.assertListEqual(srcLocalIds, ['sA', 'sD', 'sE']) 1195 | self.assertListEqual(srcRemoteIds, ['sA', 'sD', 'sE']) 1196 | 1197 | self.assertListEqual(destDir, ['dA', 'sB', 'sC', 'dB', 'dC']) 1198 | self.assertListEqual(destLocalIds, ['dA', 'sB', 'sC', 'dB', 'dC']) 1199 | self.assertListEqual(destRemoteIds, ['dA', 'sB', 'sC', 'dB', 'dC']) 1200 | 1201 | def test_srcRemoteIsArbitrary(self): 1202 | name = inspect.currentframe().f_code.co_name 1203 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 1204 | 1205 | srcDir, srcLocalIds = getGenericState(5, 's', 's') 1206 | srcRemoteIds = ['sG', 'sC', 'sB', 'sC', 'sA', 'sE', 'sB', 'sF'] 1207 | 1208 | destDir, destLocalIds = getGenericState(4, 'd', 'd') 1209 | destRemoteIds = destLocalIds.copy() 1210 | 1211 | srcStart = 1 1212 | srcEnd = 3 1213 | destIndex = 0 1214 | 1215 | moves = calcuateTransferMoves(srcDir, srcLocalIds, destLocalIds, srcRemoteIds, destRemoteIds, srcStart, srcEnd, destIndex) 1216 | simulateTransferMoves(moves, srcStart, srcEnd, destIndex, srcDir, destDir, srcLocalIds, destLocalIds, srcRemoteIds, destRemoteIds) 1217 | 1218 | self.assertListEqual(srcDir, ['sA', 'sE']) 1219 | self.assertListEqual(srcLocalIds, ['sA', 'sE']) 1220 | self.assertListEqual(srcRemoteIds, ['sG', 'sC', 'sA', 'sE', 'sB', 'sF']) 1221 | 1222 | self.assertListEqual(destDir, ['dA', 'sB', 'sC', 'sD', 'dB', 'dC', 'dD']) 1223 | self.assertListEqual(destLocalIds, ['dA', 'sB', 'sC', 'sD', 'dB', 'dC', 'dD']) 1224 | self.assertListEqual(destRemoteIds, ['dA', 'sB', 'sC', 'sD', 'dB', 'dC', 'dD']) 1225 | 1226 | def test_repeats1(self): 1227 | name = inspect.currentframe().f_code.co_name 1228 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 1229 | 1230 | srcDir, srcLocalIds = getGenericState(3, 's', 's') 1231 | srcDir.extend(srcDir) 1232 | srcLocalIds.extend(srcLocalIds) 1233 | srcRemoteIds = srcLocalIds.copy() 1234 | 1235 | destDir, destLocalIds = getGenericState(4, 'd', 'd') 1236 | destDir.extend(destDir) 1237 | destLocalIds.extend(destLocalIds) 1238 | destRemoteIds = destLocalIds.copy() 1239 | 1240 | srcStart = 4 1241 | srcEnd = 5 1242 | destIndex = 5 1243 | 1244 | moves = calcuateTransferMoves(srcDir, srcLocalIds, destLocalIds, srcRemoteIds, destRemoteIds, srcStart, srcEnd, destIndex) 1245 | simulateTransferMoves(moves, srcStart, srcEnd, destIndex, srcDir, destDir, srcLocalIds, destLocalIds, srcRemoteIds, destRemoteIds) 1246 | 1247 | self.assertListEqual(srcDir, ['sA', 'sB', 'sC', 'sA']) 1248 | self.assertListEqual(srcLocalIds, ['sA', 'sB', 'sC', 'sA']) 1249 | self.assertListEqual(srcRemoteIds, ['sA', 'sB', 'sC', 'sA']) 1250 | 1251 | self.assertListEqual(destDir, ['dA', 'dB', 'dC', 'dD', 'dA', 'dB', 'sB', 'sC', 'dC', 'dD']) 1252 | self.assertListEqual(destLocalIds, ['dA', 'dB', 'dC', 'dD', 'dA', 'dB', 'sB', 'sC', 'dC', 'dD']) 1253 | self.assertListEqual(destRemoteIds, ['dA', 'dB', 'dC', 'dD', 'dA', 'dB', 'sB', 'sC', 'dC', 'dD']) 1254 | 1255 | def test_repeats2(self): 1256 | name = inspect.currentframe().f_code.co_name 1257 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 1258 | 1259 | srcDir, srcLocalIds = getGenericState(3, 's', 's') 1260 | srcDir.extend(srcDir) 1261 | srcLocalIds.extend(srcLocalIds) 1262 | srcRemoteIds = srcLocalIds.copy() 1263 | 1264 | destDir, destLocalIds = getGenericState(4, 'd', 'd') 1265 | destDir.extend(destDir) 1266 | destLocalIds.extend(destLocalIds) 1267 | destRemoteIds = destLocalIds.copy() 1268 | 1269 | srcStart = 1 1270 | srcEnd = 4 1271 | destIndex = 5 1272 | 1273 | moves = calcuateTransferMoves(srcDir, srcLocalIds, destLocalIds, srcRemoteIds, destRemoteIds, srcStart, srcEnd, destIndex) 1274 | simulateTransferMoves(moves, srcStart, srcEnd, destIndex, srcDir, destDir, srcLocalIds, destLocalIds, srcRemoteIds, destRemoteIds) 1275 | 1276 | self.assertListEqual(srcDir, ['sA', 'sC']) 1277 | self.assertListEqual(srcLocalIds, ['sA', 'sC']) 1278 | self.assertListEqual(srcRemoteIds, ['sA', 'sC']) 1279 | 1280 | self.assertListEqual(destDir, ['dA', 'dB', 'dC', 'dD', 'dA', 'dB', 'sB', 'sC', 'sA', 'sB', 'dC', 'dD']) 1281 | self.assertListEqual(destLocalIds, ['dA', 'dB', 'dC', 'dD', 'dA', 'dB', 'sB', 'sC', 'sA', 'sB', 'dC', 'dD']) 1282 | self.assertListEqual(destRemoteIds, ['dA', 'dB', 'dC', 'dD', 'dA', 'dB', 'sB', 'sC', 'sA', 'sB', 'dC', 'dD']) 1283 | 1284 | def test_repeats3(self): 1285 | name = inspect.currentframe().f_code.co_name 1286 | cfg.logger.info(f"Running {self.__class__.__name__}: {name}") 1287 | 1288 | srcDir, srcLocalIds = getGenericState(3, 's', 's') 1289 | srcDir.extend(srcDir) 1290 | srcLocalIds.extend(srcLocalIds) 1291 | srcRemoteIds = srcLocalIds.copy() 1292 | 1293 | destDir, destLocalIds = getGenericState(4, 'd', 'd') 1294 | destDir.extend(destDir) 1295 | destLocalIds.extend(destLocalIds) 1296 | destRemoteIds = destLocalIds.copy() 1297 | 1298 | srcStart = 0 1299 | srcEnd = 5 1300 | destIndex = 5 1301 | 1302 | moves = calcuateTransferMoves(srcDir, srcLocalIds, destLocalIds, srcRemoteIds, destRemoteIds, srcStart, srcEnd, destIndex) 1303 | simulateTransferMoves(moves, srcStart, srcEnd, destIndex, srcDir, destDir, srcLocalIds, destLocalIds, srcRemoteIds, destRemoteIds) 1304 | 1305 | self.assertListEqual(srcDir, []) 1306 | self.assertListEqual(srcLocalIds, []) 1307 | self.assertListEqual(srcRemoteIds, []) 1308 | 1309 | self.assertListEqual(destDir, ['dA', 'dB', 'dC', 'dD', 'dA', 'dB', 'sA', 'sB', 'sC', 'sA', 'sB', 'sC', 'dC', 'dD']) 1310 | self.assertListEqual(destLocalIds, ['dA', 'dB', 'dC', 'dD', 'dA', 'dB', 'sA', 'sB', 'sC', 'sA', 'sB', 'sC', 'dC', 'dD']) 1311 | self.assertListEqual(destRemoteIds, ['dA', 'dB', 'dC', 'dD', 'dA', 'dB', 'sA', 'sB', 'sC', 'sA', 'sB', 'sC', 'dC', 'dD']) 1312 | 1313 | def test_manualAdd(self): 1314 | srcDir, srcLocalIds = getGenericState(3, 's', 's') 1315 | srcRemoteIds = srcLocalIds.copy() 1316 | srcDir.insert(1, cfg.manualAddId) 1317 | srcLocalIds.insert(1, cfg.manualAddId) 1318 | 1319 | destDir, destLocalIds = getGenericState(4, 'd', 'd') 1320 | destRemoteIds = destLocalIds.copy() 1321 | 1322 | srcStart = 1 1323 | srcEnd = 2 1324 | destIndex = 0 1325 | 1326 | moves = calcuateTransferMoves(srcDir, srcLocalIds, destLocalIds, srcRemoteIds, destRemoteIds, srcStart, srcEnd, destIndex) 1327 | simulateTransferMoves(moves, srcStart, srcEnd, destIndex, srcDir, destDir, srcLocalIds, destLocalIds, srcRemoteIds, destRemoteIds) 1328 | 1329 | self.assertListEqual(srcDir, ['sA', 'sC']) 1330 | self.assertListEqual(srcLocalIds, ['sA', 'sC']) 1331 | self.assertListEqual(srcRemoteIds, ['sA', 'sC']) 1332 | 1333 | self.assertListEqual(destDir, ['dA', cfg.manualAddId, 'sB', 'dB', 'dC', 'dD']) 1334 | self.assertListEqual(destLocalIds, ['dA', cfg.manualAddId, 'sB', 'dB', 'dC', 'dD']) 1335 | self.assertListEqual(destRemoteIds, ['dA', 'sB', 'dB', 'dC', 'dD']) 1336 | 1337 | def test_addTimeStamps(self): 1338 | # Get timestamps 1339 | timestamps = [Timestamp(time=0, label="test 0"), Timestamp(time = 1, label="test1"), Timestamp(time=2, label="test2")] 1340 | 1341 | if not createChapterFile(cfg.testSongPath, cfg.testSongName): 1342 | self.fail("Chapter Creation Failed") 1343 | 1344 | wipeChapterFile() 1345 | 1346 | addTimestampsToChapterFile(timestamps, cfg.testSongPath) 1347 | 1348 | if not applyChapterFileToSong(cfg.testSongPath, cfg.testSongName): 1349 | cfg.logger.error(f"Failed to Add Timestamps To Song {cfg.testSongName}") 1350 | self.fail("Chapter Creation Failed") 1351 | 1352 | preAppliedTimestamps = getTimestamps(cfg.ffmpegMetadataPath) 1353 | appliedTimestamps = extractChapters(cfg.testSongPath) 1354 | 1355 | self.assertEqual(len(timestamps), len(appliedTimestamps)) 1356 | self.assertEqual(len(timestamps), len(preAppliedTimestamps)) 1357 | 1358 | for i in range(0,len(timestamps)): 1359 | self.assertEqual(timestamps[i], appliedTimestamps[i]) 1360 | self.assertEqual(timestamps[i], preAppliedTimestamps[i]) 1361 | -------------------------------------------------------------------------------- /sync_dl/timestamps/__init__.py: -------------------------------------------------------------------------------- 1 | import re 2 | import os 3 | import subprocess 4 | import shutil 5 | 6 | import sync_dl.config as cfg 7 | from sync_dl.timestamps.scraping import Timestamp 8 | from typing import List 9 | from sync_dl import noInterrupt 10 | 11 | from sync_dl.timestamps.scraping import scrapeCommentsForTimestamps 12 | 13 | 14 | chapterRe = re.compile(r"\[CHAPTER\]\nTIMEBASE=(.+)\nSTART=(.+)\nEND=(.+)\ntitle=(.+)\n",flags = re.M) 15 | 16 | def _getSongLengthSeconds(songPath:str) -> float: 17 | result = subprocess.run(["ffprobe", "-v", "error", "-show_entries", 18 | "format=duration", "-of", 19 | "default=noprint_wrappers=1:nokey=1", songPath], 20 | stdout=subprocess.PIPE, 21 | stderr=subprocess.STDOUT) 22 | return float(result.stdout) 23 | 24 | 25 | def getTimestamps(ffmpegMetadataPath:str) -> List[Timestamp]: 26 | with open(ffmpegMetadataPath, "r") as f: 27 | contents = f.read() 28 | timestamps = [] 29 | for (timebase, start, _, title) in chapterRe.findall(contents): 30 | timestamps.append(Timestamp.fromFfmpegChapter(timebase, start, title)) 31 | 32 | return timestamps 33 | 34 | def extractChapters(songPath:str) -> List[Timestamp]: 35 | cfg.clearTmpSubPath(cfg.ffmpegMetadataPath) 36 | createChapterFileCmd = ['ffmpeg', '-hide_banner', '-loglevel', 'error', '-i', songPath, '-an', '-vn', '-sn','-f', 'ffmetadata', cfg.ffmpegMetadataPath] 37 | 38 | try: 39 | cfg.logger.debug(f"Extracting Chapters from Song Using FFMPEG Metadata File") 40 | subprocess.run(createChapterFileCmd, check=True) 41 | except subprocess.CalledProcessError as e: 42 | cfg.logger.debug(e) 43 | cfg.logger.error(f"Failed to Extract Chapters for Song: {songName}") 44 | return [] 45 | 46 | return getTimestamps(cfg.ffmpegMetadataPath) 47 | 48 | 49 | def createChapterFile(songPath:str, songName:str) -> bool: 50 | if not os.path.exists(songPath): 51 | cfg.logger.error(f"No Song at Path {songPath}") 52 | return False 53 | 54 | cfg.clearTmpSubPath(cfg.ffmpegMetadataPath) 55 | 56 | createChapterFileCmd = ['ffmpeg', '-hide_banner', '-loglevel', 'error', '-i', songPath,'-f', 'ffmetadata', cfg.ffmpegMetadataPath] 57 | 58 | try: 59 | cfg.logger.debug(f"Creating FFMPEG Metadata File") 60 | subprocess.run(createChapterFileCmd, check=True) 61 | except subprocess.CalledProcessError as e: 62 | cfg.logger.debug(e) 63 | cfg.logger.error(f"Failed to Create ffmpeg Chapter File for Song: {songName}") 64 | return False 65 | 66 | return True 67 | 68 | def wipeChapterFile() -> List[Timestamp]: 69 | '''detects chapters in file and wipes them. returns list of existing timestamps''' 70 | 71 | existingTimestamps = [] 72 | with open(cfg.ffmpegMetadataPath, "r+") as f: 73 | contents = f.read() 74 | 75 | for (timebase, start, _, title) in chapterRe.findall(contents): 76 | existingTimestamps.append(Timestamp.fromFfmpegChapter(timebase, start, title)) 77 | 78 | if len(existingTimestamps) > 0: 79 | cfg.logger.debug(f"Wiping Chapters from FFMPEG Metadata File") 80 | newContents = chapterRe.sub("", contents) 81 | f.seek(0) 82 | f.write(newContents) 83 | f.truncate() 84 | 85 | return existingTimestamps 86 | 87 | 88 | def addTimestampsToChapterFile(timestamps:List[Timestamp], songPath:str): 89 | 90 | timestamps.sort(key = lambda ele: ele.time) 91 | 92 | if len(timestamps) > 0: 93 | with open(cfg.ffmpegMetadataPath, "a") as f: 94 | for i in range(0, len(timestamps) - 1): 95 | t1 = timestamps[i] 96 | t2 = timestamps[i+1] 97 | ch = t1.toFfmpegChapter(t2.time) 98 | cfg.logger.debug(f"Adding Chapter to FFMPEG Metadata File: \n{ch}") 99 | f.write(ch) 100 | 101 | t1 = timestamps[-1] 102 | end = _getSongLengthSeconds(songPath) 103 | ch = t1.toFfmpegChapter(end) 104 | cfg.logger.debug(f"Adding Chapter to FFMPEG Metadata File: \n{ch}") 105 | f.write(ch) 106 | 107 | 108 | def applyChapterFileToSong(songPath:str, songName:str) -> bool: 109 | cfg.clearTmpSubPath(cfg.songEditPath) 110 | 111 | applyChapterFile = ['ffmpeg', '-hide_banner', '-loglevel', 'error', '-i', songPath, '-i', cfg.ffmpegMetadataPath, '-map_metadata', '1', '-map_chapters', '1', '-codec', 'copy', f"{cfg.songEditPath}/{songName}"] 112 | 113 | try: 114 | subprocess.run(applyChapterFile,check=True) 115 | except subprocess.CalledProcessError as e: 116 | cfg.logger.debug(e) 117 | return False 118 | 119 | with noInterrupt: 120 | shutil.move(f"{cfg.songEditPath}/{songName}", songPath) 121 | return True 122 | 123 | 124 | def addTimestampsIfNoneExist(plPath, songName, videoId): 125 | songPath = f"{plPath}/{songName}" 126 | if not createChapterFile(songPath, songName): 127 | return 128 | 129 | existingTimestamps = wipeChapterFile() 130 | if len(existingTimestamps) > 0: 131 | cfg.logger.info(f"Timestamps Found\n") 132 | return 133 | 134 | # Get timestamps 135 | cfg.logger.debug(f"Scraping Comments for Timestamps of Song: {songName}") 136 | timestamps = scrapeCommentsForTimestamps(videoId) 137 | 138 | if len(timestamps) == 0: 139 | cfg.logger.info(f"No Comment Timestamps Found\n") 140 | return 141 | 142 | # comment timestamps found, no existing timestamps found 143 | cfg.logger.debug(f"\nComment Timestamps Found:") 144 | for timestamp in timestamps: 145 | cfg.logger.debug(timestamp) 146 | 147 | cfg.logger.info(f"Adding Comment Timestamps\n") 148 | addTimestampsToChapterFile(timestamps, songPath) 149 | 150 | if not applyChapterFileToSong(songPath, songName): 151 | cfg.logger.error(f"Failed to Add Timestamps To Song {songName}\n") 152 | 153 | 154 | -------------------------------------------------------------------------------- /sync_dl/timestamps/scraping.py: -------------------------------------------------------------------------------- 1 | import requests 2 | import re 3 | import json 4 | from typing import NamedTuple, List, Union 5 | 6 | import sync_dl.config as cfg 7 | 8 | 9 | def jsonRegex(*args, surroundingBrace = False): 10 | r = "" 11 | 12 | numPairs = len(args)//2 13 | for i in range(numPairs): 14 | r += r"\s*[\'\"]" + args[2*i] + r"[\'\"]\s*:\s*" 15 | r += r"[\'\"]" + args[2*i+1] + r"[\'\"]\s*.?" 16 | 17 | if surroundingBrace: 18 | r = "{" + r + "}" 19 | return r 20 | 21 | apiKeyRe = re.compile(jsonRegex("INNERTUBE_API_KEY", "(.*?)")) 22 | clientVersionRe = re.compile(jsonRegex("key", "cver", "value", "(.*?)" , surroundingBrace = True)) 23 | continuationTokenRe = re.compile(r'[\'\"]token[\'\"]:[\'\"](.*?)[\'\"]') 24 | 25 | labelSanitizeRe = re.compile(r'^(?:[:\-|\s>]*)(?:\d{1,3}[\.:|\)])?\]?(?:\s)?(.*?)[:\-|\s>]*$') 26 | 27 | class Timestamp(NamedTuple): 28 | time: int 29 | label: str 30 | 31 | chapterFmt ="[CHAPTER]\nTIMEBASE=1/1000\nSTART={start}\nEND={end}\ntitle={title}\n\n" 32 | 33 | reprFmt = "{hh}:{mm:02d}:{ss:02d} - {ll}" 34 | 35 | def __eq__(self, other): 36 | return (self.time == other.time) and (self.label == other.label) 37 | 38 | def __repr__(self): 39 | secondsRemainder = self.time%60 40 | 41 | minutes = (self.time//60) 42 | minutesRemainder = minutes % 60 43 | 44 | hours = minutes//60 45 | 46 | return self.reprFmt.format(hh=hours, mm=minutesRemainder, ss=secondsRemainder, ll=self.label) 47 | 48 | @classmethod 49 | def fromFfmpegChapter(cls, timeBase:str, start, title) -> 'Timestamp': 50 | timeBaseNum = eval(timeBase) 51 | return cls(label = title, time = round(timeBaseNum*int(start))) 52 | 53 | 54 | def toFfmpegChapter(self, nextTime): 55 | return self.chapterFmt.format(start = round(1000*self.time), end = round(1000*nextTime) - 1, title = self.label) 56 | 57 | 58 | 59 | 60 | def scrapeJson(j, desiredKey: str, results:List): 61 | if isinstance(j,List): 62 | for value in j: 63 | if isinstance(value,List) or isinstance(value,dict): 64 | scrapeJson(value, desiredKey, results) 65 | return 66 | 67 | if isinstance(j, dict): 68 | for key,value in j.items(): 69 | if key == desiredKey: 70 | results.append(value) 71 | elif isinstance(value, dict) or isinstance(value, List): 72 | scrapeJson(value, desiredKey, results) 73 | return 74 | 75 | def scrapeFirstJson(j, desiredKey: str): 76 | if isinstance(j,List): 77 | for value in j: 78 | if isinstance(value,List) or isinstance(value,dict): 79 | res = scrapeFirstJson(value, desiredKey) 80 | if res is not None: 81 | return res 82 | return None 83 | 84 | if isinstance(j, dict): 85 | for key,value in j.items(): 86 | if key == desiredKey: 87 | return value 88 | elif isinstance(value, dict) or isinstance(value, List): 89 | res = scrapeFirstJson(value, desiredKey) 90 | if res is not None: 91 | return res 92 | return None 93 | 94 | return None 95 | 96 | def _sanitizeLabel(label): 97 | match = labelSanitizeRe.match(label) 98 | if match: 99 | return match.group(1) 100 | return label 101 | 102 | 103 | 104 | def _getComments(url): 105 | r=requests.get(url) 106 | 107 | 108 | x,y,z = apiKeyRe.search(r.text), continuationTokenRe.search(r.text), clientVersionRe.search(r.text) 109 | 110 | if not x: 111 | raise Exception("Unable to Find INNERTUBE_API_KEY") 112 | 113 | if not y: 114 | raise Exception("Unable to Find Continuation Token") 115 | 116 | if not z: 117 | raise Exception("Unable to Find Youtube Client Version") 118 | 119 | key = x.group(1) 120 | continuationToken = y.group(1) 121 | clientVersion = z.group(1) 122 | 123 | requestData = '''{ 124 | "context": { 125 | "adSignalsInfo": { 126 | }, 127 | "clickTracking": { 128 | }, 129 | "client": { 130 | "clientName": "WEB", 131 | "clientVersion": "'''+clientVersion+'''", 132 | }, 133 | "request": { 134 | }, 135 | "user": { 136 | } 137 | }, 138 | "continuation": "'''+continuationToken+'''" 139 | }''' 140 | 141 | b = requests.post('https://www.youtube.com/youtubei/v1/next?key='+key,data=requestData) 142 | commentJson:dict = json.loads(b.text) 143 | comments = [] 144 | scrapeJson(commentJson,"contentText",comments) 145 | return comments 146 | 147 | 148 | 149 | def _getTime(url, timeRe): 150 | matches = timeRe.search(url) 151 | if matches and matches.group(0): 152 | if not matches.group(2): 153 | return 0 154 | return int(matches.group(2)) 155 | return None 156 | 157 | def _getTimestamp(line, timeRe) -> Union[Timestamp, None]: 158 | '''line must be of form [text] [time] or [time] [text]''' 159 | text = "" 160 | url = "" 161 | time = -1 162 | if len(line) == 0: 163 | return None 164 | 165 | urlFirst = False 166 | ele = line[0] 167 | foundUrl = scrapeFirstJson(ele, "url") 168 | if foundUrl is None: 169 | text+=ele['text'] 170 | else: 171 | urlFirst = True 172 | url = foundUrl 173 | foundTime = _getTime(url,timeRe) 174 | if foundTime is None: 175 | return None 176 | time=foundTime 177 | 178 | for i in range(1,len(line)): 179 | ele = line[i] 180 | foundUrl = scrapeFirstJson(ele, "url") 181 | 182 | # found text 183 | if foundUrl is None: 184 | if url and not urlFirst: 185 | if ele['text'].isspace(): 186 | continue 187 | return None 188 | 189 | text+=ele['text'] 190 | continue 191 | 192 | # found url 193 | if foundUrl is not None: 194 | if urlFirst: 195 | return None 196 | if url: 197 | return None 198 | 199 | url = foundUrl 200 | foundTime = _getTime(url,timeRe) 201 | if foundTime is None: 202 | return None 203 | time = foundTime 204 | 205 | if text and url and time != -1: 206 | return Timestamp(label=_sanitizeLabel(text), time=time) 207 | return None 208 | 209 | 210 | def _getTimeStamps(comments, videoId): 211 | timeRe = re.compile(r'\/watch\?v=' + videoId + r'(&t=(\d+)s)?') 212 | 213 | timeStampCandidates = [] 214 | for comment in comments: 215 | runs = scrapeFirstJson(comment,'runs') 216 | 217 | if runs is None: 218 | return [] 219 | 220 | lines = [] 221 | line = [] 222 | # group into lines 223 | for ele in runs: 224 | if ele['text'] == '\n': 225 | lines.append(line) 226 | line = list() 227 | else: 228 | line.append(ele) 229 | 230 | lines.append(line) 231 | 232 | # parse lines for timestamps 233 | timeStamps:List[Timestamp] = [] 234 | for line in lines: 235 | timeStamp = _getTimestamp(line,timeRe) 236 | if timeStamp is not None: 237 | timeStamps.append(timeStamp) 238 | 239 | if len(timeStamps) > 1: 240 | timeStamps.sort(key = lambda ele: ele.time) 241 | timeStampCandidates.append(timeStamps) 242 | timeStampCandidates.sort(key=lambda ele: len(ele), reverse=True) 243 | 244 | if len(timeStampCandidates) > 0: 245 | return timeStampCandidates[0] 246 | return [] 247 | 248 | 249 | def scrapeCommentsForTimestamps(videoId): 250 | url = 'https://www.youtube.com/watch?v=' + videoId 251 | 252 | try: 253 | comments = _getComments(url) 254 | except: 255 | cfg.logger.info("No Comments Found") 256 | return [] 257 | 258 | timeStamps = _getTimeStamps(comments, videoId) 259 | return timeStamps 260 | 261 | if __name__ == '__main__': 262 | tests = ['- 01. "Goodmorning America!"', 263 | '- 01. "Goodmorning America!"', 264 | '- 00: "Goodmorning America!"', 265 | '- 00:"Goodmorning America!"', 266 | '-00: "Goodmorning America!"', 267 | '|->: 01. "Goodmorning America!"', 268 | '- "Goodmorning America!"', 269 | '-> "Goodmorning America!"', 270 | '04.1 "Goodmorning America!"', 271 | '04.1 "Goodmorning America!"', 272 | '104. "Goodmorning America!"', 273 | "01. ＬＯＯＫＩＮＧ ＵＰ_______________ " 274 | ] 275 | -------------------------------------------------------------------------------- /sync_dl/ytapiInterface.py: -------------------------------------------------------------------------------- 1 | import os 2 | import shelve 3 | import re 4 | 5 | from sync_dl import noInterrupt 6 | from sync_dl.ytdlWrappers import getIDs 7 | from sync_dl.plManagement import correctStateCorruption 8 | from sync_dl.helpers import getLocalSongs, relabel, getNumDigets, copy, delete, padZeros, calcuateTransferMoves, logTransferInfo, promptAndSanitize 9 | import sync_dl.config as cfg 10 | 11 | 12 | def pushLocalOrder(plPath): 13 | from sync_dl_ytapi.commands import pushLocalOrder 14 | pushLocalOrder(plPath) 15 | 16 | 17 | def logout(): 18 | from sync_dl_ytapi.commands import logout 19 | logout() 20 | 21 | 22 | def transferSongs(srcPlPath: str, destPlPath: str, srcStart: int, srcEnd: int, destIndex: int): 23 | ''' 24 | transfers block of songs in srcPl from srcStart to srcEnd indices, to after destIndex in destPl 25 | 26 | ie)srcStart = 4, srcEnd = 6, destIndex = 2 27 | srcPl: s0 s1 s2 s3 s4 s5 s6 s7 destPl: d0 d1 d2 d3 d4 d5 d6 28 | 29 | becomes: 30 | srcPl: s0 s1 s2 s3 s7 destPl: d0 d1 d2 s4 s5 s6 d3 d4 d5 d6 31 | ''' 32 | 33 | from sync_dl_ytapi.commands import getPlAdder, getPlRemover 34 | 35 | 36 | srcMetaDataPath = f"{srcPlPath}/{cfg.metaDataName}" 37 | destMetaDataPath = f"{destPlPath}/{cfg.metaDataName}" 38 | with shelve.open(srcMetaDataPath, 'c',writeback=True) as srcMetaData, shelve.open(destMetaDataPath, 'c',writeback=True) as destMetaData: 39 | correctStateCorruption(srcPlPath, srcMetaData) 40 | correctStateCorruption(destPlPath, destMetaData) 41 | 42 | ### Loading 43 | srcPlUrl = srcMetaData["url"] 44 | srcLocalIds = srcMetaData["ids"] 45 | assert isinstance(srcLocalIds, list) 46 | srcIdsLen = len(srcLocalIds) 47 | currentSrcDir = getLocalSongs(srcPlPath) 48 | srcPlName = os.path.basename(srcPlPath) 49 | 50 | destPlUrl = destMetaData["url"] 51 | destLocalIds = destMetaData["ids"] 52 | assert isinstance(destLocalIds, list) 53 | destIdsLen = len(destLocalIds) 54 | currentDestDir = getLocalSongs(destPlPath) 55 | destPlName = os.path.basename(destPlPath) 56 | 57 | cfg.logger.info("Loading Youtube Api Resources...") 58 | plAdder = getPlAdder(destPlUrl) 59 | if plAdder is None: 60 | return 61 | 62 | destRemoteIds = getIDs(destPlUrl) 63 | 64 | plRemover, srcRemoteIds = getPlRemover(srcPlUrl) 65 | if plRemover is None or srcRemoteIds is None: 66 | return 67 | 68 | 69 | ### Src start/end sanitization 70 | if srcStart>=srcIdsLen: 71 | cfg.logger.error(f"No Song has Index {srcStart} in {srcPlName}, Largest Index is {srcIdsLen-1}") 72 | return 73 | 74 | elif srcStart<0: 75 | cfg.logger.error(f"No Song has a Negative Index") 76 | return 77 | 78 | if srcEnd>=srcIdsLen or srcEnd == -1: 79 | srcEnd = srcIdsLen-1 80 | 81 | elif srcEnd= destIdsLen: 87 | destIndex = destIdsLen -1 88 | elif destIndex < -1: 89 | destIndex = -1 90 | 91 | 92 | # number of elements to move 93 | blockSize = srcEnd-srcStart+1 94 | 95 | numDestDigits = getNumDigets(destIdsLen + blockSize) 96 | 97 | songTransfers = calcuateTransferMoves(currentSrcDir, srcLocalIds, destLocalIds, srcRemoteIds, destRemoteIds, srcStart, srcEnd, destIndex) 98 | 99 | ### Inform User 100 | logTransferInfo(songTransfers, srcPlName, destPlName, srcIdsLen, destIdsLen, srcRemoteIds, destRemoteIds, currentDestDir, srcStart, srcEnd, destIndex) 101 | 102 | ### Prompt User 103 | cfg.logger.info(f"\n------------[Prompt]-----------") 104 | answer = input("Prefrom Transfer? (y)es/(n)o: ").lower().strip() 105 | if answer != 'y': 106 | return 107 | 108 | cfg.logger.info("") 109 | 110 | #actual editing 111 | # make room for block 112 | for i in reversed(range(destIndex+1,destIdsLen)): 113 | oldName = currentDestDir[i] 114 | newIndex =i+blockSize 115 | relabel(destMetaData, cfg.logger.debug, destPlPath, oldName, i,newIndex, numDestDigits) 116 | 117 | 118 | endEarly = False 119 | songsCompleted = 0 120 | songsPartiallyCompleated = 0 121 | 122 | for i,move in enumerate(songTransfers): 123 | if endEarly: 124 | break 125 | 126 | cfg.logger.info(f"Transfering Song: {move.songName}") 127 | with noInterrupt: 128 | if move.performCopy: 129 | copyDestName = copy(srcMetaData, destMetaData, cfg.logger.debug, srcPlPath, destPlPath, move.srcCopyName, move.srcCopyIndex, move.destCopyIndex, numDestDigits) 130 | cfg.logger.debug(f"Locally Copied Song {move.songName} from {srcPlName} to {destPlName}") 131 | 132 | if move.performRemoteAdd: 133 | if not plAdder(move.songId, move.destRemoteAddIndex): 134 | if not move.performRemoteDelete: 135 | cfg.logger.error(f"Error When Adding {move.songName} to Remote Dest: {destPlName}, URL: {destPlUrl}") 136 | cfg.logger.error(f"This Song Does Not Occur in Remote Src: {srcPlName}, So It was Probably Removed from Youtube") 137 | cfg.logger.info(f"Continuing with Transfer") 138 | else: 139 | cfg.logger.error(f"Error When Adding {move.songName} to Remote Dest: {destPlName}, URL: {destPlUrl}") 140 | cfg.logger.error(f"Fix by Either:") 141 | cfg.logger.error(f"- (r)evert Transfer for This Song") 142 | cfg.logger.error(f"- (m)anually Adding https://www.youtube.com/watch?v={move.songId} to Remote Playlist Provided Above And (c)ontinuing or (f)inishing this song.") 143 | 144 | 145 | answer = promptAndSanitize("Would You Like to: (r)evert song, (m)anual fix, (q)uit: ", 'r', 'm', 'q') 146 | 147 | if answer == 'r': 148 | if move.performCopy: 149 | delete(destMetaData, destPlPath, copyDestName, move.destCopyIndex) 150 | answer = promptAndSanitize("Would You Like to finish the rest of the transfer (y)es/(n)o: ", 'y','n') 151 | 152 | if answer != 'y': 153 | break 154 | continue 155 | 156 | elif answer == 'q': 157 | songsPartiallyCompleated += 1 158 | break 159 | 160 | elif answer == 'm': 161 | cfg.logger.info(f"Please Add, {move.songName}: https://www.youtube.com/watch?v={move.songId} \nTo Playlist {destPlName}: {destPlUrl}") 162 | input("Hit Enter to Proceed: ") 163 | 164 | 165 | answer = promptAndSanitize("Would You Like to: (c)ontinue transfer, (f)inish this song, (q)uit: ", 'c', 'f', 'q') 166 | if answer == 'f': 167 | endEarly = True 168 | if answer == 'q': 169 | songsPartiallyCompleated += 1 170 | break 171 | 172 | 173 | if move.performLocalDelete: 174 | delete(srcMetaData, srcPlPath, move.srcLocalDeleteName, move.srcLocalDeleteIndex) 175 | cfg.logger.debug(f"Locally Deleted Song {move.songName} from {srcPlName}") 176 | 177 | if move.performRemoteDelete: 178 | if not (plRemover(move.srcRemoteDeleteIndex)): 179 | cfg.logger.error(f"Error When Removing {move.songName} from Remote Src: {srcPlName}, URL: {srcPlUrl}") 180 | cfg.logger.error(f"Fix by:") 181 | cfg.logger.error(f"- Manually Removing Song Index: {move.srcRemoteDeleteIndex} URL: https://www.youtube.com/watch?v={move.songId} from Remote Playlist Provided Above") 182 | input("Hit Enter to Proceed: ") 183 | answer = promptAndSanitize("Would You Like to: (c)ontinue transfer, (q)uit: ", 'c', 'q') 184 | if answer == 'q': 185 | songsPartiallyCompleated += 1 186 | break 187 | 188 | songsCompleted += 1 189 | 190 | #remove gaps, removeBlanks 191 | correctStateCorruption(srcPlPath, srcMetaData) 192 | correctStateCorruption(destPlPath, destMetaData) 193 | 194 | # end transfer summery: 195 | cfg.logger.info(f"{songsCompleted}/{len(songTransfers)} Songs Transfered Successfully") 196 | if songsPartiallyCompleated != 0: 197 | cfg.logger.error(f"{songsPartiallyCompleated} / {len(songTransfers)} Songs Had Issues with Transfering, logged above") 198 | 199 | cfg.logger.info("\nTransfer Complete.") 200 | return 201 | 202 | 203 | -------------------------------------------------------------------------------- /sync_dl/ytdlWrappers.py: -------------------------------------------------------------------------------- 1 | #import youtube_dl 2 | import yt_dlp as youtube_dl 3 | import os 4 | import time 5 | 6 | import shutil 7 | 8 | import sync_dl.config as cfg 9 | 10 | 11 | class MyLogger: 12 | def debug(self, msg): 13 | cfg.logger.debug(msg) 14 | 15 | def info(self, msg): 16 | cfg.logger.debug(msg) 17 | 18 | def warning(self, msg): 19 | cfg.logger.debug(msg) 20 | 21 | def error(self, msg): 22 | cfg.logger.debug(msg) 23 | 24 | 25 | #ids are the unique part of each videos url 26 | def getIDs(playlistUrl): 27 | try: 28 | params={"extract_flat": True, "quiet": True} 29 | params['logger'] = MyLogger() 30 | with youtube_dl.YoutubeDL(params) as ydl: 31 | result = ydl.extract_info(playlistUrl,download=False) 32 | ids = [] 33 | for videoData in result['entries']: 34 | ids.append(videoData["id"]) 35 | return ids 36 | except: 37 | return [] 38 | 39 | 40 | def getIdsAndTitles(url): 41 | ''' 42 | used to check for corrupted metadata in integration tests 43 | Title will differ from what is on youtube because it is sanitized for use in filenames 44 | ''' 45 | try: 46 | params={"extract_flat": True, "quiet": True, "outtmpl": f'%(title)s'} 47 | params['logger'] = MyLogger() 48 | with youtube_dl.YoutubeDL(params) as ydl: 49 | result = ydl.extract_info(url,download=False) 50 | ids = [] 51 | titles = [] 52 | for videoData in result['entries']: 53 | ids.append(videoData["id"]) 54 | titles.append(ydl.prepare_filename(videoData)) 55 | return ids, titles 56 | except: 57 | return [],[] 58 | 59 | 60 | def getTitle(url): 61 | ''' 62 | used to check for corrupted metadata in integration tests 63 | Title will differ from what is on youtube because it is sanitized for use in filenames 64 | ''' 65 | 66 | params = {} 67 | params['extract_flat'] = True 68 | params['quiet'] = True 69 | params["outtmpl"] = f'%(title)s' 70 | params['logger'] = MyLogger() 71 | 72 | with youtube_dl.YoutubeDL(params) as ydl: 73 | 74 | song = ydl.extract_info(url,download=False) 75 | 76 | title = ydl.prepare_filename(song) 77 | return title 78 | 79 | 80 | def downloadToTmp(videoId,numberStr): 81 | url = f"https://www.youtube.com/watch?v={videoId}" 82 | 83 | cfg.dlParams["outtmpl"] = f'{cfg.songDownloadPath}/{numberStr}_%(title)s.%(ext)s' 84 | cfg.dlParams['logger'] = MyLogger() 85 | 86 | with youtube_dl.YoutubeDL(cfg.dlParams) as ydl: 87 | 88 | cfg.clearTmpSubPath(cfg.songDownloadPath) 89 | 90 | attemptNumber = 1 91 | numAttempts = 2 92 | 93 | while True: 94 | try: 95 | ydl.download([url]) 96 | return True 97 | except Exception as e: 98 | cfg.logger.debug(e) 99 | cfg.logger.info(f"Unable to Download Song at {url}") 100 | if attemptNumber == numAttempts: 101 | cfg.logger.info(f"Max Download Retries Reached! If Song URL is Accessable Try Updating yt-dlp") 102 | return False 103 | cfg.logger.info("Retrying...") 104 | time.sleep(0.5) 105 | attemptNumber += 1 106 | 107 | 108 | def moveFromTmp(path): 109 | tmp = os.listdir(path=cfg.songDownloadPath) 110 | shutil.move(f"{cfg.songDownloadPath}/{tmp[0]}", path) 111 | return tmp[0] 112 | 113 | def getJsonPlData(url): 114 | '''returns list of dicts of data for each video in playlist at url (order is playlist order)''' 115 | params = {} 116 | params['extract_flat'] = True 117 | 118 | params['quiet'] = True 119 | params['logger'] = MyLogger() 120 | with youtube_dl.YoutubeDL(params) as ydl: 121 | try: 122 | entries = ydl.extract_info(url,download=False)['entries'] 123 | except: 124 | cfg.logger.error(f"No Playlist At URL: {url}") 125 | entries = [] 126 | return entries 127 | 128 | 129 | def _getBestThumbnail(thumbnails): 130 | maxWidth = 0 131 | best = None 132 | for thumbnail in thumbnails: 133 | width = thumbnail['width'] 134 | if width > maxWidth: 135 | maxWidth = width 136 | best = thumbnail['url'] 137 | 138 | if best is None: 139 | return None 140 | 141 | return best.split('?')[0] 142 | 143 | def getThumbnailUrls(url:str) -> dict: 144 | ''' 145 | used to check for corrupted metadata in integration tests 146 | Title will differ from what is on youtube because it is sanitized for use in filenames 147 | ''' 148 | 149 | params = { 150 | "extract_flat": True, 151 | "quiet": True, 152 | "simulate": True, 153 | #"list_thumbnails": True, 154 | "logger": MyLogger(), 155 | } 156 | 157 | with youtube_dl.YoutubeDL(params) as ydl: 158 | result = ydl.extract_info(url,download=False) 159 | 160 | try: 161 | entries = result['entries'] 162 | except KeyError: 163 | return {} 164 | 165 | thumbnails = {} 166 | for entry in entries: 167 | try: 168 | thumbnail = _getBestThumbnail(entry['thumbnails']) 169 | if thumbnail is None: 170 | continue 171 | thumbnails[entry['id']] = thumbnail 172 | except KeyError: 173 | continue 174 | 175 | return thumbnails 176 | -------------------------------------------------------------------------------- /test.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | import logging 3 | 4 | 5 | import argparse 6 | import sys 7 | 8 | import sync_dl.tests.unitTests as unitTests 9 | import sync_dl.tests.integrationTests as integrationTests 10 | 11 | import sync_dl.config as cfg 12 | 13 | 14 | def parseArgs(): 15 | description = ("unit and integration testing for sync-dl") 16 | 17 | 18 | parser = argparse.ArgumentParser(description=description) 19 | 20 | #posistional 21 | parser.add_argument('URL',nargs='?', type=str, help='the name of the directory for the playlist') 22 | 23 | 24 | parser.add_argument('-u','--unit', action='store_true', help='runs all unit tests') 25 | parser.add_argument('-i','--integration',action='store_true', help='tests integration using playlist at URL') 26 | 27 | parser.add_argument('-p','--print',action='store_true', help='prints to terminal in addition to tests/testing.log' ) 28 | 29 | args,other = parser.parse_known_args() 30 | 31 | sys.argv[1:] = other #additional unittest args 32 | 33 | return args 34 | 35 | def setLogging(args): 36 | formatter = logging.Formatter("%(asctime)s [%(levelname)s] %(message)s") 37 | fh = logging.FileHandler(f"{cfg.modulePath}/tests/testing.log") 38 | fh.setFormatter(formatter) 39 | 40 | cfg.logger.addHandler(fh) 41 | 42 | if args.print: 43 | sh = logging.StreamHandler() 44 | sh.setFormatter(formatter) 45 | cfg.logger.addHandler(sh) 46 | 47 | cfg.logger.setLevel(level=logging.DEBUG) 48 | 49 | if __name__ == "__main__": 50 | args = parseArgs() 51 | 52 | setLogging(args) 53 | 54 | runall = not (args.unit or args.integration) 55 | 56 | #checks if integration test has to run (whenever -i is used, or if all tests are run) 57 | if args.integration or runall: 58 | # if no playlist was provided all further functions cannot run 59 | if not args.URL: 60 | print("URL Required for Integration Testing, Use -u to Run Only Unit Tests") 61 | exit() 62 | integrationTests.test_integration.PL_URL = args.URL 63 | 64 | if runall: 65 | success = unittest.main(unitTests,exit=False).result.wasSuccessful() 66 | if not success: 67 | sys.exit(1) 68 | unittest.main(integrationTests,failfast=True) 69 | 70 | elif args.unit: 71 | unittest.main(unitTests) 72 | 73 | elif args.integration: 74 | unittest.main(integrationTests,failfast=True) 75 | --------------------------------------------------------------------------------