├── .gitattributes ├── .gitignore ├── ChangeLog.txt ├── LICENSE.md ├── README.md ├── contributors.md ├── docs ├── developer_guide.md └── user_guide.md ├── icons ├── vortexdm.ico ├── vortexdm.png └── vortexdm.svg ├── pyproject.toml ├── requirements.txt ├── scripts └── exe_build │ ├── exe-fullbuild.py │ └── exe-quickbuild.py ├── setup.py ├── vortexdm.py └── vortexdm ├── VortexDM.py ├── __init__.py ├── __main__.py ├── about.py ├── brain.py ├── cmdview.py ├── config.py ├── controller.py ├── dependency.py ├── downloaditem.py ├── iconsbase64.py ├── model.py ├── setting.py ├── systray.py ├── themes.py ├── tkview.py ├── update.py ├── utils.py ├── version.py ├── video.py ├── view.py └── worker.py /.gitattributes: -------------------------------------------------------------------------------- 1 | # all text files 2 | * text eol=lf 3 | 4 | # avoid images 5 | *.jpg -text 6 | *.png -text 7 | *.svg -text 8 | 9 | *.vcproj text eol=crlf 10 | *.txt text eol=crlf 11 | 12 | # Batch files (bat,btm,cmd) must be run with CRLF line endings. 13 | *.bat text eol=crlf 14 | 15 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # ignore all files and directories 2 | * 3 | 4 | # include only the following 5 | !docs/ 6 | !docs/* 7 | !vortexdm/ 8 | !vortexdm/*.py 9 | !ChangeLog.txt 10 | !contributors.md 11 | !LICENSE 12 | !vortexdm.py 13 | !README.md 14 | !requirements.txt 15 | !setup.py 16 | !pyproject.toml 17 | !.gitignore 18 | !.gitattributes 19 | !scripts/ 20 | !scripts/exe_build/ 21 | !scripts/exe_build/exe-fullbuild.py 22 | !scripts/exe_build/exe-quickbuild.py 23 | !icons/ 24 | !icons/* -------------------------------------------------------------------------------- /ChangeLog.txt: -------------------------------------------------------------------------------- 1 | 2023.1.0 2 | -Renamed project from VDM to VortexDM 3 | -Fixed Windows ffmpeg.exe download in controller.py 4 | -Removed rawbitrate variable in Stream class under video.py as it was causing an exception in controller.create_video_playlist. Looks unused. 5 | 6 | 2022.1.0: 7 | - Inital release of Vortex Download Manager (VortexDM) based on FireDM 2022.2.5. Original project, FireDM, by Mahmoud Elshahat. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ![Logo](https://raw.githubusercontent.com/Sixline/VortexDM/main/icons/vortexdm.png) 2 | Vortex Download Manager (VortexDM) is an open-source Python Internet download manager with a high speed multi-connection engine. It downloads general files and videos. Developed in Python, based on "PycURL" and "youtube_dl". 3 | 4 | Original project, FireDM, by Mahmoud Elshahat. 5 | 6 | Homepage: https://github.com/Sixline/VortexDM 7 | PyPI Homepage: https://pypi.org/project/vortexdm 8 | 9 | [![GitHub Issues](https://img.shields.io/github/issues-raw/Sixline/VortexDM?color=brightgreen)](https://github.com/Sixline/VortexDM/issues) - [![GitHub Closed Issues](https://img.shields.io/github/issues-closed-raw/Sixline/VortexDM?color=blueviolet)](https://github.com/Sixline/VortexDM/issues?q=is%3Aissue+is%3Aclosed) 10 | 11 | **Features**: 12 | * High download speeds - based on PycURL 13 | * Multi-connection downloading 14 | * Automatic file segmentation 15 | * Automatic refresh for dead links 16 | * Resume uncompleted downloads 17 | * Support for YouTube and a lot of other stream websites using youtube-dl to fetch info and PycURL to download media 18 | * Download entire videos, playlists, or selected videos 19 | * Download fragmented video streams and encrypted/nonencrypted HLS media streams 20 | * Watch videos while downloading *some videos will have no audio until they finish downloading* 21 | * Download video subtitles 22 | * Write video metadata to downloaded files 23 | * Built-in updater 24 | * Scheduled downloads 25 | * Re-using existing connections 26 | * Clipboard monitor 27 | * Proxy support (http, https, socks4, and socks5) 28 | * User/pass authentication, referee link, video thumbnail, and subtitles 29 | * Use custom cookie files 30 | * MD5 and SHA256 checksums 31 | * Custom GUI themes 32 | * Set download speed limit 33 | * Shell commands or computer shutdown on download completion 34 | * Control number of the concurrent downloads and maximum connections 35 | 36 | # How to use VortexDM: 37 | Running in command line: show help by typing `vortexdm -h` 38 | 39 | Running the GUI: Refer to the user guide at https://github.com/Sixline/VortexDM/blob/master/docs/user_guide.md 40 | 41 | # Portable VortexDM Versions: 42 | 43 | Run VortexDM without any installation (recommended) 44 | - **Windows Portable Version** ([Download!](https://github.com/Sixline/VortexDM/releases/latest)): 45 | Available in .zip format. Built with 64-bit Python 3.11+ and will only work on 64-bit Windows 10+. 46 | Unzip and run VortexDM-GUI.exe, no installation required. 47 | 48 | - **Linux Portable Version** 49 | Removing this section for now as I am not familiar with building AppImages. Will revisit. 50 | 51 | ## Manually installing VortexDM with pip (Linux Only - Debian/Ubuntu Based Shown): 52 | 1- Check python version (minimum version required is 3.8): `python3 --version` 53 | 54 | 2- Install required packages: 55 | ```sh 56 | sudo apt install ffmpeg libcurl4-openssl-dev libssl-dev python3-pip python3-pil python3-pil.imagetk python3-tk python3-dbus gir1.2-appindicator3-0.1 57 | sudo apt install fonts-symbola fonts-linuxlibertine fonts-inconsolata fonts-emojione 58 | ``` 59 | 60 | 3- Install Vortex Download Manager using pip: 61 | 62 | ```sh 63 | python3 -m pip install vortexdm --user --upgrade --no-cache 64 | ``` 65 | 66 | ## Running from source code inside a Python virtual environment (Linux Only - Debian/Ubuntu Based Shown): 67 | 1- Check python version (minimum version required is 3.8): `python3 --version` 68 | 69 | 2- Install required packages: 70 | ```sh 71 | sudo apt install ffmpeg libcurl4-openssl-dev libssl-dev python3-pip python3-pil python3-pil.imagetk python3-tk python3-dbus gir1.2-appindicator3-0.1 72 | sudo apt install fonts-symbola fonts-linuxlibertine fonts-inconsolata fonts-emojione 73 | ``` 74 | 75 | 3- Run below code to do the following: 76 | * Clone this repo 77 | * Create Python virtual environment 78 | * Install the requirements 79 | * Create launch script 80 | * Run VortexDM 81 | 82 | ```sh 83 | git clone https://github.com/Sixline/VortexDM 84 | python3 -m venv ./.env 85 | source ./.env/bin/activate 86 | python3 -m pip install -r ./VortexDM/requirements.txt 87 | echo "source ./.env/bin/activate 88 | python3 ./VortexDM/vortexdm.py \$@ " > vortexdm.sh 89 | chmod +x ./vortexdm.sh 90 | ./vortexdm.sh 91 | ``` 92 | 93 | > Optionally create .desktop file and add VortexDM to your applications 94 | ```sh 95 | VortexDMLSPATH=$(realpath ./vortexdm.sh) 96 | echo "[Desktop Entry] 97 | Name=VortexDM 98 | GenericName=VortexDM 99 | Comment=Vortex Download Manager 100 | Exec=$VortexDMLSPATH 101 | Icon=vortexdm 102 | Terminal=false 103 | Type=Application 104 | Categories=Network; 105 | Keywords=Internet;download 106 | " > VortexDM.desktop 107 | cp ./VortexDM.desktop ~/.local/share/applications/ 108 | mkdir -p ~/.local/share/icons/hicolor/48x48/apps/ 109 | cp ./VortexDM/icons/vortexdm.png ~/.local/share/icons/hicolor/48x48/apps/vortexdm.png 110 | ``` 111 | 112 | # Known Issues: 113 | - Linux X Server will raise an error if some fonts are missing, especially emoji fonts - See Dependencies below 114 | 115 | - Mac - Tkinter - Can have issues depending on versions. See here: https://www.python.org/download/mac/tcltk 116 | 117 | - Systray Icon: Depends on GTK 3+ and AppIndicator3 on Linux. Install these packages if you need systray to run properly. 118 | 119 | # Dependencies: 120 | - Python 3.8+: Tested with Python 3.11+ on Windows 10 and Ubuntu Linux 121 | - [Tkinter](https://docs.python.org/3/library/tkinter.html): standard Python interface to the Tcl/Tk GUI toolkit. 122 | - [FFmpeg](https://www.ffmpeg.org/): for merging audio with DASH videos. 123 | - Fonts: (Linux X Server will raise an error if some fonts are missing, especially emoji fonts. Below are the 124 | recommended fonts to be installed. 125 | 126 | ``` 127 | ttf-linux-libertine 128 | ttf-inconsolata 129 | ttf-emojione 130 | ttf-symbola 131 | noto-fonts 132 | ``` 133 | - [PycURL](http://pycurl.io/docs/latest/index.html): a Python interface to libcurl, the multiprotocol file transfer library. Used as the download engine. 134 | - [youtube_dl](https://github.com/ytdl-org/youtube-dl): Famous YouTube downloader, limited use for meta information extraction only but videos are downloaded using PycURL. 135 | - [yt_dlp](https://github.com/yt-dlp/yt-dlp): yt-dlp is a youtube-dl fork based on the now inactive youtube-dlc. 136 | - [Certifi](https://github.com/certifi/python-certifi): required by PycURL for validating the trustworthiness of SSL certificates. 137 | - [Plyer](https://github.com/kivy/plyer): for systray area notification. 138 | - [AwesomeTkinter](https://github.com/Aboghazala/AwesomeTkinter): for application GUI. 139 | - [Pillow](https://python-pillow.org): the friendly PIL fork. PIL is an acronym for Python Imaging Library. 140 | - [pystray](https://github.com/moses-palmer/pystray): for systray icon. 141 | 142 | > [!NOTE] 143 | > PycURL 7.45.3 - 2024-02-17 - Windows binary wheels are now available. 144 | 145 | ~~**Note for PycURL:** 146 | For Windows users who wants to run from source or use pip: 147 | Unfortunately, PycURL removed binary versions for Windows and it now has to be built from source. See here: http://pycurl.io/docs/latest/install.html#windows 148 | `python -m pip install pycurl` will fail on Windows, your best choice is to use the portable version.~~ 149 | 150 | # How to contribute to this project: 151 | 1- By testing the application and opening [new issues](https://github.com/Sixline/VortexDM/issues/new) for bugs, feature requests, or suggestions. 152 | 153 | 2- Check the [Developer Guidelines](https://github.com/Sixline/VortexDM/blob/master/docs/developer_guide.md). 154 | 155 | 3- Check [open issues](https://github.com/Sixline/VortexDM/issues?q=is%3Aopen+is%3Aissue) and see if you can help. 156 | 157 | 4- Fork this repo and make a pull request. 158 | 159 | # Contributors: 160 | Please check [contributors.md](https://github.com/Sixline/VortexDM/blob/master/contributors.md) for a list of contributors. 161 | -------------------------------------------------------------------------------- /contributors.md: -------------------------------------------------------------------------------- 1 | I would like to thank everyone who contributed in this project in any 2 | way, with code, bug reports, or ideas. 3 | 4 | If you think your name should be included in this list or want to modify 5 | your name please make an issue or pull request. 6 | 7 | --- 8 | 9 | Original project, FireDM, by Mahmoud Elshahat. All original contributors are listed below. 10 | 11 | --- 12 | 13 | Developers who contributed with code: 14 | - @Aboghazala 15 | - @shin-illua 16 | - @ryneeverett 17 | - @bertaga 18 | - @qontinuum-dev 19 | - @fidele000 (Fidele K.Cyisa) 20 | - @sajjadhossanshimanto 21 | - @TgSeed 22 | 23 | --- 24 | 25 | Themes creators: 26 | - @ahmed-tasaly 27 | - @Mr-Personality 28 | - @tazihad 29 | 30 | --- 31 | 32 | People who contributed with Testing, ideas, and bug reports: 33 | - @Ton1A (Toni) 34 | - @smaragdus 35 | - @hongyi-zhao 36 | - @jdaniele71 (Daniele) 37 | 38 | -------------------------------------------------------------------------------- /docs/developer_guide.md: -------------------------------------------------------------------------------- 1 | # OUTDATED - ON TO DO LIST TO UPDATE 2 | 3 | ### Developer Guide 4 | 5 | This Guide for developer who want to contribute or understand how this project work, feel free to improve this guide anytime 6 | 7 | 8 | ### Purpose of this project: 9 | This project is built upon famous youtube-dl project, with a proper use of 10 | multithreading and the use of Libcurl as a download engine, it willl reach 11 | upto 10x higher speeds in case of hls or fragmented video files, 12 | in addition it can download general files too. 13 | 14 | GUI, is based on tkinter, which is a lightweight and responsive. 15 | 16 | This project is never made to compete with other download managers, it 17 | is just a "hopefully useful" Simple enough and fast video downloader. 18 | 19 | 20 | --- 21 | 22 | 23 | ### Current project logic: 24 | Generally FireDM is using Libcurl as a download engine via threads to 25 | achieve multi-connections, for videos, youtube-dl is our player, where 26 | its sole role is to extract video information from a specific url "No 27 | other duties for youtube-dl". 28 | FFMPEG will be used for post processing e.g. mux audio and video, merge 29 | HLS video segments into one video file, and other useful media 30 | manipulation. 31 | 32 | 33 | Current application design adopts "MVC" design pattern, where "Model" in model.py, 34 | controller in controller.py and view is tkview.py for tkinter gui or cmdview.py 35 | which run interactively in terminal. 36 | 37 | also an "observer" pattern is used to notify controller when model "data object" 38 | chenage its state. 39 | 40 | Work flow using gui as follow: 41 | - user enter a url in url entry widget. 42 | - gui will ask controller to process url. 43 | - controller will make a data object e.g. ObservableDownloadItem() and call its 44 | url_update method which send http request to remote server and based on the received 45 | 'response headers' from server it will update properties like name, size, mime type, 46 | download url, etc..., and controller will be notified with changes. 47 | - in case mime type is html, then controller will pass url to youtube-dl to search for 48 | videos, and it will create ObservableVideo() object. 49 | - controller will send update messages to gui and it will show file info in main tab. 50 | - when user press download button, gui will ask controller to download current file, 51 | controller will make some pre-download checks and if all ok, it will create a thread 52 | to run 'brain function' to download the file 53 | - brain function will run both 'thread manager' and file manager 54 | - thread manager will make multiple connection to download file in "chunks" 55 | - file manager will write completed chunks into the target file 56 | - after completing all chunks, a post processing will be run if necessary, e.g. ffmpeg 57 | will mux audio and video in one final video file. 58 | 59 | 60 | --- 61 | 62 | 63 | ### Files: 64 | 65 | - **FireDM.py:** main file, it will start application in either 66 | interactive terminal mode or in gui mode. 67 | 68 | - **config.py:** Contains all shared variables and settings. 69 | 70 | - **utils.py:** all helper functions. 71 | 72 | - **tkview.py:** This module has application gui, designed by tkinter. 73 | 74 | - **settings.py:** this where we save / load settings, and download items list 75 | 76 | - **brain.py:** every download item object will be sent to brain to 77 | download it, this module has thread manager, and file manager 78 | 79 | - **cmdview.py:** an interactive user interface run in terminal. 80 | 81 | - **controller.py:** a part of "MVC" design, where it will contain the 82 | application logic and communicate to both Model and view 83 | 84 | - **model.py:** contains "ObservableDownloadItem", "ObservableVideo" which acts 85 | as Model in "MVC" design with "observer" design. 86 | 87 | - **downloaditem.py:** It has DownloadItem class which contains 88 | information for a download item, and you will find a lot of 89 | DownloadItem objects in this project named shortly as "d" or 90 | "self.d". 91 | 92 | - **video.py:** it contains Video class which is subclassed from 93 | DownloadItem, for video objects. also this file has most video related 94 | function, e.g. merge_video_audio, pre_process_hls, etc... 95 | 96 | - **worker.py:** Worker class object acts as a standalone workers, every 97 | worker responsible for downloading a chunk or file segment. 98 | 99 | - **update.py:** contains functions for updating FireDM frozen version 100 | "currently cx_freeze windows portable version", also update 101 | youtube-dl. 102 | 103 | - **version.py:** contains version number, which is date based, example 104 | content, __version__ = '2020.8.13' 105 | 106 | - **dependency.py:** contains a list of required external packages for 107 | FireDM to run and has "install_missing_pkgs" function to install the 108 | missing packages automatically. 109 | 110 | - **ChangeLog.txt:** Log changes to each new version. 111 | 112 | --- 113 | 114 | ### Documentation format: 115 | code documentation if found doesn't follow a specific format, 116 | something that should be fixed, the selected project format should 117 | follow Google Python Style Guide, resources: 118 | 119 | - [Example Google Style Python Docstrings](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html#example-google) 120 | - ["Google Python Style Guide"](http://google.github.io/styleguide/pyguide.html) 121 | 122 | 123 | --- 124 | 125 | ### How can I contribute to this project: 126 | - check open issues in this project and find something that you can fix. 127 | - It's recommended that you open an issue first to discuss what you want 128 | to do, this will create a better communication with other developers 129 | working on the project. 130 | - pull request, and add a good description of your modification. 131 | - it doesn't matter how small the change you make, it will make a 132 | difference. 133 | 134 | 135 | 136 | -------------------------------------------------------------------------------- /docs/user_guide.md: -------------------------------------------------------------------------------- 1 | # OUTDATED - ON TO DO LIST TO UPDATE 2 | 3 | # FireDM user guide: 4 | Parts of these guidelines has been taken from an 5 | [article](https://www.ghacks.net/2020/08/13/firedm-is-an-open-source-download-manager-that-can-download-videos-and-playlists/) 6 | about FireDM, thanks to Ashwin @ ghacks.net 7 | 8 | 9 | FireDM is an open source download manager that can download videos and 10 | playlists, written in Python. 11 | 12 | The interface of FireDM has four tabs, The Main tab is used to add new 13 | downloads. The program captures downloads from the clipboard, but you 14 | can manually start a download by pasting a URL in the Link box. 15 | 16 | When FireDM recognizes that the clipboard contains a link to a 17 | downloadable file, it displays its interface. To be precise, the Main 18 | tab is brought to focus. It displays the captured link, the name of the 19 | file, its size, type (ZIP, EXE, etc), and whether the download can be 20 | resumed or not. 21 | 22 | ![Main Tab](https://user-images.githubusercontent.com/58998813/92562020-939f2f80-f275-11ea-94ea-fe41c9c72abc.png) 23 | 24 | Want to download videos? You can, copy the URL or paste it manually in 25 | the link box and FireDM will pull up options to download the media. It 26 | allows you to choose the videos to download, the video format and 27 | resolution 28 | 29 | ![pl_main](https://user-images.githubusercontent.com/58998813/94432366-f3af3480-0196-11eb-8449-3e35bfb13e5c.png) 30 | 31 | Need only audio stream, you can selct a variety of available formats, 32 | like mp3, aac, ogg, beside ma, and webm. 33 | 34 | ![audio](https://user-images.githubusercontent.com/58998813/94432501-26f1c380-0197-11eb-9d25-59cbef8c2279.png) 35 | 36 | For dash videos, "which have no audio" a proper audio will be 37 | automatically selected, but if you need to select audio manually you 38 | should enable manual dash audio selection option in settings tab. 39 | ![dash audio](https://user-images.githubusercontent.com/58998813/94432513-2b1de100-0197-11eb-86d5-6cd27763fba6.png) 40 | 41 | You can change the folder where the file will be saved to, before 42 | clicking on the download button. FireDM displays the download progress in 43 | downloads tab that indicates the download's file size progress, the 44 | speed, time remaining for the process to complete. 45 | 46 | **what about subtitles?** 47 | you can press "sub" button in main tab and a window will show up with 48 | available subtitles and captions 49 | you can select any number of subtitles with your preferred format, e.g. 50 | srt, vtt, ttml, srv, etc ... 51 | 52 | ![subs](https://user-images.githubusercontent.com/58998813/94432649-5acce900-0197-11eb-98f5-d47221859dae.png) 53 | 54 | Once the download has been completed, a notification appears near os 55 | notification area e.g. the Windows Action Center. 56 | 57 | 58 | Manage your queue from the Downloads tab. You can pause, resume 59 | downloads, refresh the URL, open the download folder and delete files 60 | from the queue. 61 | 62 | ![downloads tab](https://user-images.githubusercontent.com/58998813/92564079-e4fcee00-f278-11ea-83e1-9a272bc06b0f.png) 63 | 64 | The download queue displays the filename, size, percentage of the 65 | download that's been completed, the download speed, status, etc. 66 | 67 | Right-click on an item in the download queue to view a context menu that 68 | can be used to open the file or the folder where it is saved. The "watch 69 | while downloading" option opens your default video player to play the 70 | media as it is being downloaded. The menu also lets you copy the URL of 71 | the web page, the file's direct link or the playlist's URL. 72 | 73 | ![rcm](https://user-images.githubusercontent.com/58998813/94441493-1c3d2b80-01a3-11eb-9a91-9d67e91375f7.png) 74 | 75 | 76 | --- 77 | 78 | 79 | # Settings Tab: 80 | FireDM has a lot of options you can tweak, below will list most options 81 | briefly. 82 | ![rcm](https://user-images.githubusercontent.com/58998813/94432701-6f10e600-0197-11eb-9d5a-397980d8fa57.png) 83 | 84 | ### Select theme: 85 | There is some default themes you can switch betweenthem, if you don't 86 | like the available themes you can use "New" button to create your own 87 | theme. 88 | 89 | Below is a "theme editor" window where you can name your custom theme, 90 | choose some basic colors and the rest of colors will be auto-selected, 91 | but if you need more fine tuning you can click advanced 92 | button 93 | 94 | ![rcm](https://user-images.githubusercontent.com/58998813/94432668-628c8d80-0197-11eb-9276-0f7d778c244f.png) 95 | 96 | There is also an option to edit and delete themes, but these options 97 | available only for custom themes not default themes. 98 | 99 | ### Systray: 100 | you can enable/disable systray icon with an option tosendFireDM to 101 | systray when closing main window. 102 | 103 | ### Monitor urls: 104 | FireDM will monitor clipboard for any copied urls"enabled by default", 105 | however you can disable this option anytime. 106 | 107 | ### Write metadata to video files: 108 | video files have some metadata e.g.name, chapter, series, duration, 109 | etc... which can be hardcoded into video file when it gets downloaded. 110 | 111 | ### Auto rename: 112 | if file with the same name exist in download folder,with this option 113 | enabled, firedm will rename the new file, generally will add a number 114 | suffix to avoid overwriting existing file, if you disable this option, a 115 | warning popup will show up to ask user to overwrite file or cancel 116 | download. 117 | 118 | ![overwrite](https://user-images.githubusercontent.com/58998813/94441003-86090580-01a2-11eb-8b9f-326e2f52f45d.png) 119 | 120 | 121 | ### Show MD5 and SHA256: 122 | A very handy feature that save your time to check downloaded file 123 | integrity. 124 | 125 | --- 126 | 127 | ![overwrite](https://user-images.githubusercontent.com/58998813/94432706-70421300-0197-11eb-83d3-6be27caf841a.png) 128 | 129 | ### concurrent downloads: 130 | number of maximum files downloads that can bedone in same time, set this 131 | to 1 if you need to have only one download at a time and new added files 132 | will be in pending list. 133 | 134 | ### connections per download: 135 | maximum number of connections that every file download can establish 136 | with the remote server in the same time. 137 | 138 | ### speed limit: 139 | limit download speed for each download item, e.g. 100kb, 5 MB, etc.., 140 | units can be upper or lower case. 141 | 142 | ### proxy: 143 | you can use a proxy to access a restricted website, supported protocols 144 | (http, https, socks4, and socks5) 145 | 146 | proxy general format examples: 147 | 148 | - http://proxy_address:port 149 | - 157.245.224.29:3128 150 | - or if authentication required: 151 | `http://username:password@proxyserveraddress:port` 152 | 153 | then you should select proxy type: 154 | - http 155 | - https 156 | - socks4 157 | - socks5 158 | 159 | Optionally you can choose to use proxy DNS by checking: 160 | - 'Use proxy DNS' option 161 | 162 | ### login: 163 | if website requires authentication you can add username and password, 164 | these credential will not be saved in settings file. 165 | 166 | 167 | ### Referee url: 168 | some servers will refuse downloading a file if you try to access 169 | download link directly, as a workaround you should pass a referee url, 170 | normally this is the website main webpage url. 171 | 172 | 173 | 174 | ## Using cookies with FireDM: 175 | 176 | Passing cookies to FireDM is a good way to work around CAPTCHA, some websites require you to solve in particular 177 | cases in order to get access (e.g. YouTube, CloudFlare). 178 | 179 | you need to extract cookie file from your browser save it some where (for example: cookies.txt) then goto 180 | Settings > Network > then check Use Cookies option 181 | browse to select your cookies file ... done. 182 | 183 | In order to extract cookies from browser use any conforming browser extension for exporting cookies. 184 | For example, [cookies.txt](https://chrome.google.com/webstore/detail/cookiestxt/njabckikapfpffapmjgojcnbfjonfjfg) (for Chrome) 185 | or [cookies.txt](https://addons.mozilla.org/en-US/firefox/addon/cookies-txt/) (for Firefox) 186 | 187 | Note that the cookies file must be in Mozilla/Netscape format and the first line of the cookies file must be either 188 | `# HTTP Cookie File or # Netscape HTTP Cookie File`. 189 | Make sure you have correct newline format in the cookies file and convert newlines if necessary to correspond with your OS, 190 | namely CRLF (\r\n) for Windows and LF (\n) for Unix and Unix-like systems (Linux, macOS, etc.). 191 | HTTP Error 400: Bad Request when using cookies is a good sign of invalid newline format. 192 | 193 | 194 | ![settings screenshot 3](https://user-images.githubusercontent.com/58998813/94432708-70daa980-0197-11eb-95ee-e89ff05a46b2.png) 195 | 196 | --- 197 | 198 | ## Debugging: 199 | There is some options to help developers catch problems: 200 | 201 | ### keep temp files: 202 | An option to leave temp folder when done downloading for inspecting 203 | segment files for errors. 204 | 205 | ### Re-raise exceptions: 206 | crash application and show detailed error description, useful when 207 | running FireDM from source. 208 | 209 | 210 | --- 211 | 212 | ## Update: 213 | 214 | ### check for update frequency: 215 | FireDM will try to check for new version on github every "7 days as 216 | default" it will first ask user for permission to do so, also frequency 217 | can be changed or uncheck this option to disable periodic check for update 218 | 219 | you can check for new FireDM version manually any time you press refresh 220 | button in front of FireDM version. 221 | 222 | for Youtube-dl you can check for new version and if found, FireDM will 223 | ask user to install it automatically, after finish, you should restart 224 | firedm for new youtube-dl version to be loaded, in case new youtube-dl 225 | version doesn't work as expected, you can press "Rollback update" to 226 | restore the last youtube-dl version 227 | 228 | 229 | -------------------------------------------------------------------------------- /icons/vortexdm.ico: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Sixline/VortexDM/25a86da74e1b8fddd650e5dadb6350a0de9ab118/icons/vortexdm.ico -------------------------------------------------------------------------------- /icons/vortexdm.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Sixline/VortexDM/25a86da74e1b8fddd650e5dadb6350a0de9ab118/icons/vortexdm.png -------------------------------------------------------------------------------- /icons/vortexdm.svg: -------------------------------------------------------------------------------- 1 | 2 | 21 | 41 | 43 | 46 | 50 | 54 | 55 | 66 | 77 | 88 | 99 | 110 | 121 | 132 | 133 | 137 | 144 | 149 | 154 | 159 | 164 | 169 | 174 | 175 | 176 | 178 | 179 | 180 | image/svg+xml 181 | 183 | 185 | 186 | 188 | Openclipart 189 | 190 | 191 | 2007-04-30T03:52:54 192 | artistic, artistic, clip art, clipart, flower, flower, image, media, png, public domain, svg, 193 | http://openclipart.org/detail/4221/spirale-by-ernes 194 | 195 | 196 | ernes 197 | 198 | 199 | 200 | 201 | artistic 202 | clip art 203 | clipart 204 | flower 205 | image 206 | media 207 | png 208 | public domain 209 | svg 210 | 211 | 212 | 213 | 215 | 217 | 219 | 221 | 222 | 223 | 224 | 225 | -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | [build-system] 2 | requires = ["setuptools"] 3 | build-backend = "setuptools.build_meta" -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | plyer 2 | certifi 3 | youtube_dl 4 | yt_dlp 5 | pycurl 6 | Pillow >= 6.0.0 7 | pystray 8 | awesometkinter >= 2021.6.4 9 | packaging 10 | distro; platform_system == "Linux" 11 | -------------------------------------------------------------------------------- /scripts/exe_build/exe-fullbuild.py: -------------------------------------------------------------------------------- 1 | """ 2 | Vortex Download Manager (VortexDM) 3 | 4 | A multi-connection internet download manager, based on "PycURL" and "youtube_dl". Original project, FireDM, by Mahmoud Elshahat. 5 | :copyright: (c) 2022 by Sixline 6 | :copyright: (c) 2019-2021 by Mahmoud Elshahat. 7 | :license: GNU GPLv3, see LICENSE.md for more details. 8 | 9 | Module description: 10 | build an executable (exe) for windows using cx_freeze 11 | you should execute this module from command line using: "python exe-fullbuild.py build" on windows only. 12 | 13 | *Can be used for 32-bit Windows but you will need to find or build a 32-bit version of ffmpeg. BtbN doesn't auto-build a 32-bit version.* 14 | """ 15 | 16 | import os 17 | import sys 18 | import shutil 19 | import subprocess 20 | 21 | from cx_Freeze import setup, Executable 22 | 23 | APP_NAME = 'VortexDM' 24 | 25 | # to run setup.py directly 26 | if len(sys.argv) == 1: 27 | sys.argv.append("build") 28 | 29 | # get current directory 30 | fp = os.path.realpath(os.path.abspath(__file__)) 31 | current_folder = os.path.dirname(fp) 32 | 33 | project_folder = os.path.dirname(os.path.dirname(current_folder)) 34 | build_folder = current_folder 35 | app_folder = os.path.join(build_folder, APP_NAME) 36 | icon_path = os.path.join(project_folder, 'icons', 'vortexdm.ico') # best use size 48, and must be an "ico" format 37 | version_fp = os.path.join(project_folder, 'vortexdm', 'version.py') 38 | requirements_fp = os.path.join(project_folder, 'requirements.txt') 39 | main_script_path = os.path.join(project_folder, 'vortexdm.py') 40 | 41 | sys.path.insert(0, project_folder) # for imports to work 42 | from vortexdm.utils import simpledownload, delete_folder, delete_file, create_folder, zip_extract 43 | 44 | # create build folder 45 | create_folder(build_folder) 46 | 47 | # get version 48 | version_module = {} 49 | with open(version_fp) as f: 50 | exec(f.read(), version_module) # then we can use it as: version_module['__version__'] 51 | version = version_module['__version__'] 52 | 53 | # get required packages 54 | with open(requirements_fp) as f: 55 | packages = [line.strip().split(' ')[0] for line in f.readlines() if line.strip()] + ['vortexdm'] 56 | 57 | # clean names 58 | packages = [pkg.replace(';', '') for pkg in packages] 59 | 60 | # filter some packages 61 | for pkg in ['distro', 'Pillow']: 62 | if pkg in packages: 63 | packages.remove(pkg) 64 | 65 | # add keyring to packages 66 | packages.append('keyring') 67 | 68 | print(packages) 69 | 70 | includes = [] 71 | include_files = [] 72 | excludes = ['numpy', 'test', 'setuptools', 'unittest', 'PySide2'] 73 | 74 | cmd_target_name = f'{APP_NAME.lower()}.exe' 75 | gui_target_name = f'{APP_NAME}-GUI.exe' 76 | 77 | executables = [ 78 | Executable(main_script_path, base='Console', target_name=cmd_target_name), 79 | Executable(main_script_path, base='Win32GUI', target_name=gui_target_name, icon=icon_path), 80 | ] 81 | 82 | setup( 83 | 84 | version=version, 85 | description=f"{APP_NAME}", 86 | author="Sixline ; Original project, FireDM, by Mahmoud Elshahat", 87 | name=APP_NAME, 88 | 89 | options={"build_exe": { 90 | "includes": includes, 91 | 'include_files': include_files, 92 | "excludes": excludes, 93 | "packages": packages, 94 | 'build_exe': app_folder, 95 | 'include_msvcr': True, 96 | } 97 | }, 98 | 99 | executables=executables 100 | ) 101 | 102 | # Post processing 103 | 104 | # ffmpeg 105 | ffmpeg_zip_path = os.path.join(current_folder, 'ffmpeg-master-latest-win64-gpl.zip') 106 | ffmpeg_extract_path = os.path.join(current_folder, 'ffmpeg-master-latest-win64-gpl') 107 | ffmpeg_path = os.path.join(ffmpeg_extract_path, 'ffmpeg-master-latest-win64-gpl', 'bin', 'ffmpeg.exe') 108 | if not os.path.isfile(os.path.join(app_folder, 'ffmpeg.exe')): 109 | if not os.path.isfile(ffmpeg_zip_path): 110 | # Download 64-bit BtbN auto-build. BtbN doesn't auto-build a 32-bit version. 111 | simpledownload('https://github.com/BtbN/FFmpeg-Builds/releases/download/latest/ffmpeg-master-latest-win64-gpl.zip', fp=ffmpeg_zip_path) 112 | print('Download done! Extracting and moving ffmpeg.exe...') 113 | zip_extract(z_fp=ffmpeg_zip_path, extract_folder=ffmpeg_extract_path) 114 | shutil.copy(ffmpeg_path, os.path.join(app_folder, 'ffmpeg.exe')) 115 | print('Cleaning up...') 116 | delete_folder(ffmpeg_extract_path, verbose=True) 117 | delete_file(ffmpeg_zip_path, verbose=True) 118 | 119 | 120 | # write resource fields for exe files 121 | # install pe-tools https://github.com/avast/pe_tools 122 | cmd = f'"{sys.executable}" -m pip install pe_tools' 123 | subprocess.run(cmd, shell=True) 124 | 125 | for fname in (cmd_target_name, gui_target_name): 126 | fp = os.path.join(app_folder, fname) 127 | info = { 128 | 'Comments': 'https://github.com/Sixline/VortexDM', 129 | 'CompanyName': 'Vortex Download Manager', 130 | 'FileDescription': 'Vortex Download Manager', 131 | 'FileVersion': version, 132 | 'InternalName': fname, 133 | 'LegalCopyright': 'copyright: (c) 2022 by Sixline - Original project, FireDM, by Mahmoud Elshahat', 134 | 'LegalTrademarks': 'Vortex Download Manager', 135 | 'OriginalFilename': fname, 136 | 'ProductName': 'Vortex Download Manager', 137 | 'ProductVersion': version, 138 | 'legalcopyright': 'copyright: (c) 2022 by Sixline - Original project, FireDM, by Mahmoud Elshahat' 139 | } 140 | 141 | param = ' -V '.join([f'"{k}={v}"' for k, v in info.items()]) 142 | cmd = f'"{sys.executable}" -m pe_tools.peresed -V {param} "{fp}"' 143 | subprocess.run(cmd, shell=True) 144 | 145 | # Check if 32 or 64 bit for zip file name 146 | if sys.maxsize > 2**32: 147 | win_arch = 64 148 | else: 149 | win_arch = 32 150 | 151 | # create zip file 152 | output_filename = f'{APP_NAME}-{version}-win{win_arch}' 153 | print(f'Preparing zip file: {output_filename}.zip') 154 | fname = shutil.make_archive(output_filename, 'zip', root_dir=build_folder, base_dir='VortexDM') 155 | delete_folder(app_folder, verbose=True) 156 | 157 | print(f'Done! {output_filename}.zip') 158 | -------------------------------------------------------------------------------- /scripts/exe_build/exe-quickbuild.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | """ 3 | Vortex Download Manager (VortexDM) 4 | 5 | A multi-connection internet download manager, based on "PycURL" and "youtube_dl". Original project, FireDM, by Mahmoud Elshahat. 6 | :copyright: (c) 2022 by Sixline 7 | :copyright: (c) 2019-2021 by Mahmoud Elshahat. 8 | :license: GNU GPLv3, see LICENSE.md for more details. 9 | 10 | Module description: 11 | build an executable (exe) for windows using existing template or download a template from github 12 | this module can be executed from any operating system e.g. linux, windows, etc.. 13 | to create exe version from scratch use exe_fullbuild.py on windows os 14 | """ 15 | 16 | import os 17 | import re 18 | import sys 19 | import json 20 | import shutil 21 | import subprocess 22 | 23 | fp = os.path.realpath(os.path.abspath(__file__)) 24 | current_folder = os.path.dirname(fp) 25 | project_folder = os.path.dirname(os.path.dirname(current_folder)) 26 | sys.path.insert(0, project_folder) # for imports to work 27 | 28 | from vortexdm.utils import simpledownload, zip_extract, get_pkg_version 29 | 30 | APP_NAME = 'VortexDM' 31 | 32 | build_folder = current_folder 33 | app_folder = os.path.join(build_folder, APP_NAME) 34 | 35 | # check for app folder existence, otherwise download latest version from github 36 | if not os.path.isdir(app_folder): 37 | print('downloading ', APP_NAME) 38 | data = simpledownload('https://api.github.com/repos/Sixline/VortexDM/releases/latest').decode("utf-8") 39 | # example: "browser_download_url": "https://github.com/Sixline/VortexDM/releases/download/2022.1.0/VortexDM-2022.1.0-win64.zip" 40 | data = json.loads(data) 41 | assets = data['assets'] 42 | 43 | url = None 44 | for asset in assets: 45 | filename = asset.get('name', '') 46 | if filename.lower().endswith('zip'): # e.g. VortexDM-2022.1.0-win64.zip 47 | url = asset.get('browser_download_url') 48 | break 49 | 50 | if url: 51 | # download file 52 | z_fp = os.path.join(build_folder, filename) 53 | if not os.path.isfile(z_fp): 54 | simpledownload(url, z_fp) 55 | 56 | # unzip 57 | print('extracting, please wait ...') 58 | zip_extract(z_fp, build_folder) 59 | os.remove(z_fp) 60 | 61 | else: 62 | print('Failed to download latest version, download manually ' 63 | 'from https://github.com/Sixline/VortexDM/releases/latest') 64 | exit(1) 65 | 66 | lib_folder = os.path.join(app_folder, 'lib') 67 | 68 | # update packages, ---------------------------------------------------------------------------------------------------- 69 | print('update packages') 70 | 71 | # update vortexdm pkg 72 | src_folder = os.path.join(project_folder, 'vortexdm') 73 | target_folder = os.path.join(lib_folder, 'vortexdm') 74 | shutil.rmtree(target_folder, ignore_errors=True) 75 | shutil.copytree(src_folder, target_folder, dirs_exist_ok=True) 76 | 77 | # update other packages 78 | pkgs = ['youtube_dl', 'yt_dlp', 'awesometkinter', 'certifi', 'python_bidi'] 79 | cmd = f'{sys.executable} -m pip install {" ".join(pkgs)} --upgrade --no-compile --no-deps --target "{lib_folder}" ' 80 | subprocess.run(cmd, shell=True) 81 | 82 | # get application version ---------------------------------------------------------------------------------------------- 83 | version = get_pkg_version(os.path.join(project_folder, 'vortexdm')) 84 | 85 | # edit info for exe files ---------------------------------------------------------------------------------------------- 86 | cmd = f'"{sys.executable}" -m pip install pe_tools' 87 | subprocess.run(cmd, shell=True) 88 | 89 | for fname in ('vortexdm.exe', 'VortexDM-GUI.exe'): 90 | fp = os.path.join(app_folder, fname) 91 | info = { 92 | 'Comments': 'https://github.com/Sixline/VortexDM', 93 | 'CompanyName': 'Vortex Download Manager', 94 | 'FileDescription': 'Vortex Download Manager', 95 | 'FileVersion': version, 96 | 'InternalName': fname, 97 | 'LegalCopyright': 'copyright: (c) 2022 by Sixline - Original project, FireDM, by Mahmoud Elshahat', 98 | 'LegalTrademarks': 'Vortex Download Manager', 99 | 'OriginalFilename': fname, 100 | 'ProductName': 'Vortex Download Manager', 101 | 'ProductVersion': version, 102 | 'legalcopyright': 'copyright: (c) 2022 by Sixline - Original project, FireDM, by Mahmoud Elshahat' 103 | } 104 | 105 | param = ' -V '.join([f'"{k}={v}"' for k, v in info.items()]) 106 | cmd = f'"{sys.executable}" -m pe_tools.peresed -V {param} {fp}' 107 | subprocess.run(cmd, shell=True) 108 | 109 | # create zip file 110 | output_filename = f'{APP_NAME}-{version}-win64' 111 | print(f'Preparing zip file: {output_filename}.zip') 112 | fname = shutil.make_archive(output_filename, 'zip', root_dir=build_folder, base_dir='VortexDM') 113 | delete_folder(app_folder, verbose=True) 114 | 115 | print(f'Done! {output_filename}.zip') 116 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | """ 2 | Vortex Download Manager (VortexDM) 3 | 4 | A multi-connection internet download manager, based on "PycURL" and "youtube_dl". Original project, FireDM, by Mahmoud Elshahat. 5 | :copyright: (c) 2023 by Sixline 6 | :copyright: (c) 2019-2021 by Mahmoud Elshahat. 7 | :license: GNU GPLv3, see LICENSE.md for more details. 8 | """ 9 | 10 | import os 11 | import setuptools 12 | 13 | # get current directory 14 | path = os.path.realpath(os.path.abspath(__file__)) 15 | current_directory = os.path.dirname(path) 16 | 17 | # get version 18 | version = {} 19 | with open(f"{current_directory}/vortexdm/version.py") as f: 20 | exec(f.read(), version) # then we can use it as: version['__version__'] 21 | 22 | # get long description from readme 23 | with open(f"{current_directory}/README.md", "r") as fh: 24 | long_description = fh.read() 25 | 26 | try: 27 | with open(f"{current_directory}/requirements.txt", "r") as fh: 28 | requirements = fh.readlines() 29 | except: 30 | requirements = ['plyer', 'certifi', 'youtube_dl', 'yt_dlp', 'pycurl', 'pillow >= 6.0.0', 'pystray', 31 | 'awesometkinter >= 2021.3.19'] 32 | 33 | setuptools.setup( 34 | name="vortexdm", 35 | version=version['__version__'], 36 | scripts=[], # ['VortexDM.py'], no need since added an entry_points 37 | author="Mahmoud Elshahat", 38 | maintainer="Sixline", 39 | description="Vortex Download Manager", 40 | long_description=long_description, 41 | long_description_content_type="text/markdown", 42 | url="https://github.com/Sixline/VortexDM ", 43 | packages=setuptools.find_packages(), 44 | keywords="vortexdm vdm firedm internet download manager youtube hls pycurl curl youtube-dl tkinter", 45 | project_urls={ 46 | 'Source': 'https://github.com/Sixline/VortexDM', 47 | 'Tracker': 'https://github.com/Sixline/VortexDM/issues', 48 | 'Releases': 'https://github.com/Sixline/VortexDM/releases', 49 | }, 50 | install_requires=requirements, 51 | entry_points={ 52 | # our executable: "exe file on windows for example" 53 | 'console_scripts': [ 54 | 'vortexdm = vortexdm.VortexDM:main', 55 | ]}, 56 | classifiers=[ 57 | 'Programming Language :: Python :: 3.8', 58 | 'Programming Language :: Python :: 3.9', 59 | 'Programming Language :: Python :: 3.10', 60 | 'Programming Language :: Python :: 3.11', 61 | "License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)", 62 | "Operating System :: OS Independent", 63 | ], 64 | python_requires='>=3.8', 65 | ) 66 | -------------------------------------------------------------------------------- /vortexdm.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | """ 3 | Vortex Download Manager (VortexDM) 4 | 5 | A multi-connection internet download manager, based on "PycURL" and "youtube_dl". Original project, FireDM, by Mahmoud Elshahat. 6 | :copyright: (c) 2023 by Sixline 7 | :copyright: (c) 2019-2021 by Mahmoud Elshahat. 8 | :license: GNU GPLv3, see LICENSE.md for more details. 9 | """ 10 | 11 | from vortexdm import VortexDM 12 | 13 | VortexDM.main() -------------------------------------------------------------------------------- /vortexdm/VortexDM.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | """ 3 | Vortex Download Manager (VortexDM) 4 | 5 | A multi-connection internet download manager, based on "PycURL" and "youtube_dl". Original project, FireDM, by Mahmoud Elshahat. 6 | :copyright: (c) 2023 by Sixline 7 | :copyright: (c) 2019-2021 by Mahmoud Elshahat. 8 | :license: GNU GPLv3, see LICENSE.md for more details. 9 | 10 | Module description: 11 | This is main application module 12 | """ 13 | 14 | # standard modules 15 | import os 16 | import subprocess 17 | import sys 18 | import argparse 19 | import re 20 | import signal 21 | 22 | # This code should stay on top to handle relative imports in case of direct call of VortexDM.py 23 | if __package__ is None: 24 | path = os.path.realpath(os.path.abspath(__file__)) 25 | sys.path.insert(0, os.path.dirname(path)) 26 | sys.path.insert(0, os.path.dirname(os.path.dirname(path))) 27 | 28 | __package__ = 'vortexdm' 29 | import vortexdm 30 | 31 | 32 | # local modules 33 | from . import config, setting 34 | from .controller import Controller 35 | from .tkview import MainWindow 36 | from .cmdview import CmdView 37 | from .utils import parse_urls, parse_bytes, format_bytes 38 | from .setting import load_setting 39 | from .version import __version__ 40 | 41 | 42 | def pars_args(arguments): 43 | """parse arguments vector 44 | Args: 45 | arguments(list): list contains arguments, could be sys.argv[1:] i.e. without script name 46 | """ 47 | 48 | description = """Vortex Download Manager (VortexDM) - An open-source Python 49 | Internet download manager with a high speed multi-connection engine. It 50 | downloads general files and videos. 51 | Developed in Python, based on "PycURL" and "youtube_dl". 52 | GNU GPLv3, see LICENSE.md for more details. 53 | Source: https://github.com/Sixline/VortexDM""" 54 | 55 | def iterable(txt): 56 | # process iterable in arguments, e.g. tuple or list, 57 | # example --window=(600,300) 58 | return re.findall(r'\d+', txt) 59 | 60 | def int_iterable(txt): 61 | return map(int, iterable(txt)) 62 | 63 | def speed(txt): 64 | return parse_bytes(txt) 65 | 66 | # region cmdline arguments 67 | # some args' names are taken from youtube-dl, reference: 68 | # https://github.com/ytdl-org/youtube-dl/blob/master/youtube_dl/options.py 69 | # Note: we should use "default=argparse.SUPPRESS" to discard option if not used by user, 70 | # and prevent default value overwrite in config module 71 | 72 | parser = argparse.ArgumentParser( 73 | prog='vortexdm', 74 | description=description, 75 | epilog='copyright: (c) 2022 Vortex Download Manager. license: GNU GPLv3, see LICENSE.md file for more details.' 76 | 'Original project, FireDM, by Mahmoud Elshahat' 77 | 'Isuues: https://github.com/Sixline/VortexDM/issues', 78 | usage='\n' 79 | '%(prog)s [OPTIONS] URL1 URL2 URL3 \n' 80 | 'example: %(prog)s "https://somesite.com/somevideo" "https://somesite.com/anothervideo"\n' 81 | 'Note: to run %(prog)s in GUI(Graphical User Interface) mode, use "--gui" option along with other ' 82 | 'arguments, or start %(prog)s without any arguments.', 83 | add_help=False 84 | ) 85 | 86 | parser.add_argument('url', nargs='*', # list of urls or empty list 87 | help="""url / link of the file you want to download or multiple urls, if url contains special 88 | shell characters e.g. "&", it must be quoted by a single or double quotation to avoid shell 89 | error""") 90 | 91 | # ------------------------------------------------------------------------------------General options--------------- 92 | general = parser.add_argument_group(title='General options') 93 | general.add_argument( 94 | '-h', '--help', 95 | action='help', 96 | help='show this help message and exit') 97 | general.add_argument( 98 | '-v', '--version', 99 | action='version', version='VortexDM Version: ' + __version__, 100 | help='Print program version and exit') 101 | general.add_argument( 102 | '--show-settings', 103 | action='store_true', default=argparse.SUPPRESS, 104 | help='show current application settings and their current values and exit') 105 | general.add_argument( 106 | '--edit-config', dest='edit_config', 107 | type=str, metavar='EDITOR', default=argparse.SUPPRESS, 108 | action='store', nargs='?', const='nano', # const if use argument without value 109 | help='Edit config file, you should specify your text editor executable, otherwise "nano" will be used') 110 | general.add_argument( 111 | '--ignore-config', dest='ignore_config', default=argparse.SUPPRESS, 112 | action='store_true', 113 | help='Do not load settings from config file. in ~/.config/VortexDM/ or (APPDATA/VortexDM/ on Windows)') 114 | general.add_argument( 115 | '--dlist', dest='ignore_dlist', 116 | action='store_false', default=argparse.SUPPRESS, 117 | help='load/save "download list" from/to d-list config file. ' 118 | 'default="False in cmdline mode and True in GUI mode"') 119 | general.add_argument( 120 | '--ignore-dlist', dest='ignore_dlist', 121 | action='store_true', default=argparse.SUPPRESS, 122 | help='opposite of "--dlist" option') 123 | general.add_argument( 124 | '-g', '--gui', 125 | action='store_true', default=argparse.SUPPRESS, 126 | help='use graphical user interface, same effect if you try running %(prog)s without any parameters') 127 | general.add_argument( 128 | '--interactive', 129 | action='store_true', default=argparse.SUPPRESS, 130 | help='interactive command line') 131 | general.add_argument( 132 | '--imports-only', 133 | action='store_true', default=argparse.SUPPRESS, 134 | help='import all packages and exit, useful when building AppImage or exe releases, since it ' 135 | 'will build pyc files and make application start faster') 136 | general.add_argument( 137 | '--persistent', 138 | action='store_true', default=argparse.SUPPRESS, 139 | help='save current options in global configuration file, used in cmdline mode.') 140 | 141 | # ----------------------------------------------------------------------------------------Filesystem options-------- 142 | filesystem = parser.add_argument_group(title='Filesystem options') 143 | filesystem.add_argument( 144 | '-o', '--output', 145 | type=str, metavar='', default=argparse.SUPPRESS, 146 | help='output file path, filename, or download folder: if input value is a file name without path, file will ' 147 | f'be saved in current folder, if input value is a folder path only, ' 148 | 'remote file name will be used, ' 149 | 'be careful with video extension in filename, since ffmpeg will convert video based on extension') 150 | filesystem.add_argument( 151 | '-b', '--batch-file', default=argparse.SUPPRESS, 152 | type=argparse.FileType('r', encoding='UTF-8'), metavar='', 153 | help='path to text file containing multiple urls to be downloaded, file should have ' 154 | 'every url in a separate line, empty lines and lines start with "#" will be ignored.') 155 | filesystem.add_argument( 156 | '--auto-rename', 157 | action='store_true', default=argparse.SUPPRESS, 158 | help=f'auto rename file if same name already exist on disk, default={config.auto_rename}') 159 | 160 | # ---------------------------------------------------------------------------------------Network Options------------ 161 | network = parser.add_argument_group(title='Network Options') 162 | network.add_argument( 163 | '--proxy', dest='proxy', 164 | metavar='URL', default=argparse.SUPPRESS, 165 | help='proxy url should have one of these schemes: (http, https, socks4, socks4a, socks5, or socks5h) ' 166 | 'e.g. "scheme://proxy_address:port", and if proxy server requires login ' 167 | '"scheme://usr:pass@proxy_address:port", ' 168 | f'examples: "socks5://127.0.0.1:8080", "socks4://john:pazzz@127.0.0.1:1080", default="{config.proxy}"') 169 | 170 | # ---------------------------------------------------------------------------------------Authentication Options----- 171 | authentication = parser.add_argument_group(title='Authentication Options') 172 | authentication.add_argument( 173 | '-u', '--username', 174 | dest='username', metavar='USERNAME', default=argparse.SUPPRESS, 175 | help='Login with this account ID') 176 | authentication.add_argument( 177 | '-p', '--password', 178 | dest='password', metavar='PASSWORD', default=argparse.SUPPRESS, 179 | help='Account password.') 180 | 181 | # --------------------------------------------------------------------------------------Video Options--------------- 182 | vid = parser.add_argument_group(title='Video Options') 183 | vid.add_argument( 184 | '--engine', dest='active_video_extractor', 185 | type=str, metavar='ENGINE', default=argparse.SUPPRESS, 186 | help="select video extractor engine, available choices are: ('youtube_dl', and 'yt_dlp'), " 187 | f"default={config.active_video_extractor}") 188 | vid.add_argument( 189 | '--quality', dest='quality', 190 | type=str, metavar='QUALITY', default=argparse.SUPPRESS, 191 | help="select video quality, available choices are: ('best', '1080p', '720p', '480p', '360p', " 192 | "and 'lowest'), default=best") 193 | vid.add_argument( 194 | '--prefer-mp4', dest='prefer_mp4', 195 | action='store_true', default=argparse.SUPPRESS, 196 | help='prefer mp4 streams if available, default=False') 197 | 198 | # --------------------------------------------------------------------------------------Workarounds----------------- 199 | workarounds = parser.add_argument_group(title='Workarounds') 200 | workarounds.add_argument( 201 | '--check-certificate', dest='ignore_ssl_cert', 202 | action='store_false', default=argparse.SUPPRESS, 203 | help=f'validate ssl certificate, default="{not config.ignore_ssl_cert}"') 204 | workarounds.add_argument( 205 | '--no-check-certificate', dest='ignore_ssl_cert', 206 | action='store_true', default=argparse.SUPPRESS, 207 | help=f'ignore ssl certificate validation, default="{config.ignore_ssl_cert}"') 208 | workarounds.add_argument( 209 | '--user-agent', 210 | metavar='UA', dest='custom_user_agent', default=argparse.SUPPRESS, 211 | help='Specify a custom user agent') 212 | workarounds.add_argument( 213 | '--referer', dest='referer_url', 214 | metavar='URL', default=argparse.SUPPRESS, 215 | help='Specify a custom referer, use if the video access is restricted to one domain') 216 | 217 | # --------------------------------------------------------------------------------------Post-processing Options----- 218 | postproc = parser.add_argument_group(title='Post-processing Options') 219 | postproc.add_argument( 220 | '--add-metadata', dest='write_metadata', 221 | action='store_true', default=argparse.SUPPRESS, 222 | help=f'Write metadata to the video file, default="{config.write_metadata}"') 223 | postproc.add_argument( 224 | '--no-metadata', dest='write_metadata', 225 | action='store_false', default=argparse.SUPPRESS, 226 | help=f'Don\'t Write metadata to the video file, default="{not config.write_metadata}"') 227 | postproc.add_argument( 228 | '--write-thumbnail', dest='download_thumbnail', 229 | action='store_true', default=argparse.SUPPRESS, 230 | help=f'Write thumbnail image to disk after downloading video file, default="{config.download_thumbnail}"') 231 | postproc.add_argument( 232 | '--no-thumbnail', dest='download_thumbnail', 233 | action='store_false', default=argparse.SUPPRESS, 234 | help='Don\'t Write thumbnail image to disk after downloading video file') 235 | postproc.add_argument( 236 | '--checksum', dest='checksum', 237 | action='store_true', default=argparse.SUPPRESS, 238 | help=f'calculate checksums for completed files MD5 and SHA256, default="{config.checksum}"') 239 | postproc.add_argument( 240 | '--no-checksum', dest='checksum', 241 | action='store_false', default=argparse.SUPPRESS, 242 | help='Don\'t calculate checksums') 243 | 244 | # -------------------------------------------------------------------------------------Application Update Options--- 245 | appupdate = parser.add_argument_group(title='Application Update Options') 246 | appupdate.add_argument( 247 | '--update', 248 | action='store_true', dest='update_self', default=argparse.SUPPRESS, 249 | help='Update this application and video libraries to latest version.') 250 | 251 | # -------------------------------------------------------------------------------------Downloader Options----------- 252 | downloader = parser.add_argument_group(title='Downloader Options') 253 | downloader.add_argument( 254 | '-R', '--retries', dest='refresh_url_retries', 255 | type=int, metavar='RETRIES', default=argparse.SUPPRESS, 256 | help=f'Number of retries to download a file, default="{config.refresh_url_retries}".') 257 | downloader.add_argument( 258 | '-l', '--speed-limit', dest='speed_limit', 259 | type=speed, metavar='LIMIT', default=argparse.SUPPRESS, 260 | help=f'download speed limit, in bytes per second (e.g. 100K or 5M), zero means no limit, ' 261 | f'current value={format_bytes(config.speed_limit)}.') 262 | downloader.add_argument( 263 | '--concurrent', dest='max_concurrent_downloads', 264 | type=int, metavar='NUMBER', default=argparse.SUPPRESS, 265 | help=f'max concurrent downloads, default="{config.max_concurrent_downloads}".') 266 | downloader.add_argument( 267 | '--connections', dest='max_connections', 268 | type=int, metavar='NUMBER', default=argparse.SUPPRESS, 269 | help=f'max download connections per item, default="{config.max_connections}".') 270 | 271 | # -------------------------------------------------------------------------------------Debugging options------------ 272 | debug = parser.add_argument_group(title='Debugging Options') 273 | debug.add_argument( 274 | '-V', '--verbose', dest='verbose', 275 | type=int, metavar='LEVEL', default=argparse.SUPPRESS, 276 | help=f'verbosity level 1, 2, or 3, default(cmdline mode)=1, default(gui mode)={config.log_level}.') 277 | debug.add_argument( 278 | '--keep-temp', dest='keep_temp', 279 | action='store_true', default=argparse.SUPPRESS, 280 | help=f'keep temp files for debugging, default="{config.keep_temp}".') 281 | debug.add_argument( 282 | '--remove-temp', dest='keep_temp', 283 | action='store_false', default=argparse.SUPPRESS, 284 | help='remove temp files after finish') 285 | 286 | # -------------------------------------------------------------------------------------GUI options------------------ 287 | gui = parser.add_argument_group(title='GUI Options') 288 | gui.add_argument( 289 | '--theme', dest='current_theme', 290 | type=str, metavar='THEME', default=argparse.SUPPRESS, 291 | help=f'theme name, e.g. "Dark", default="{config.current_theme}".') 292 | gui.add_argument( 293 | '--monitor-clipboard', dest='monitor_clipboard', 294 | action='store_true', default=argparse.SUPPRESS, 295 | help=f'monitor clipboard, and process any copied url, default="{config.monitor_clipboard}".') 296 | gui.add_argument( 297 | '--no-clipboard', dest='monitor_clipboard', 298 | action='store_false', default=argparse.SUPPRESS, 299 | help='Don\'t monitor clipboard, in gui mode') 300 | gui.add_argument( 301 | '--window', dest='window_size', 302 | type=int_iterable, metavar='(WIDTH,HIGHT)', default=argparse.SUPPRESS, 303 | help=f'window size, example: --window=(600,400) no space allowed, default="{config.window_size}".') 304 | # ------------------------------------------------------------------------------------------------------------------ 305 | # endregion 306 | 307 | args = parser.parse_args(arguments) 308 | sett = vars(args) 309 | 310 | return sett 311 | 312 | 313 | def main(argv=sys.argv): 314 | """ 315 | app main 316 | Args: 317 | argv(list): command line arguments vector, argv[0] is the script pathname if known 318 | """ 319 | 320 | # workaround for missing stdout/stderr for windows Win32GUI app e.g. cx_freeze gui app 321 | try: 322 | sys.stdout.write('\n') 323 | sys.stdout.flush() 324 | except AttributeError: 325 | # dummy class to export a "do nothing methods", expected methods to be called (read, write, flush, close) 326 | class Dummy: 327 | def __getattr__(*args): 328 | return lambda *args: None 329 | 330 | for x in ('stdout', 'stderr', 'stdin'): 331 | setattr(sys, x, Dummy()) 332 | 333 | guimode = True if len(argv) == 1 or '--gui' in argv else False 334 | cmdmode = not guimode 335 | 336 | # read config file 337 | config_fp = os.path.join(config.sett_folder, 'setting.cfg') 338 | if '--ignore-config' not in argv: 339 | load_setting() 340 | 341 | sett = pars_args(argv[1:]) 342 | 343 | if sett.get('referer_url'): 344 | sett['use_referer'] = True 345 | 346 | if any((sett.get('username'), sett.get('password'))): 347 | sett['use_web_auth'] = True 348 | 349 | if sett.get('proxy'): 350 | sett['enable_proxy'] = True 351 | 352 | if sett.get('output'): 353 | fp = os.path.realpath(sett.get('output')) 354 | if os.path.isdir(fp): 355 | folder = fp 356 | else: 357 | folder = os.path.dirname(fp) 358 | name = os.path.basename(fp) 359 | if name: 360 | sett['name'] = name 361 | 362 | if folder: 363 | sett['folder'] = folder 364 | elif cmdmode: 365 | sett['folder'] = os.path.realpath('.') 366 | 367 | verbose = sett.get('verbose') 368 | if verbose is None and cmdmode: 369 | sett['log_level'] = 1 370 | 371 | config.__dict__.update(sett) 372 | 373 | if sett.get('config'): 374 | for key in config.settings_keys: 375 | value = getattr(config, key) 376 | print(f'{key}: {value}') 377 | print('\nconfig file path:', config_fp) 378 | sys.exit(0) 379 | 380 | if sett.get('edit_config'): 381 | executable = sett.get('edit_config') 382 | cmd = f'{executable} {config_fp}' 383 | subprocess.run(cmd, shell=True) 384 | sys.exit(0) 385 | 386 | if sett.get('imports_only'): 387 | import importlib, time 388 | total_time = 0 389 | 390 | def getversion(mod): 391 | try: 392 | version = mod.version.__version__ 393 | except: 394 | version = '' 395 | return version 396 | 397 | for module in ['plyer', 'certifi', 'youtube_dl', 'yt_dlp', 'pycurl', 'PIL', 'pystray', 'awesometkinter', 398 | 'tkinter']: 399 | start = time.time() 400 | 401 | try: 402 | m = importlib.import_module(module) 403 | version = getversion(m) 404 | total_time += time.time() - start 405 | print(f'imported module: {module} {version}, in {round(time.time() - start, 1)} sec') 406 | except Exception as e: 407 | print(module, 'package import error:', e) 408 | 409 | print(f'Done, importing modules, total time: {round(total_time, 2)} sec ...') 410 | sys.exit(0) 411 | 412 | # set ignore_dlist argument to True in cmdline mode if not explicitly used 413 | if sett.get('ignore_dlist') is None and not guimode: 414 | sett['ignore_dlist'] = True 415 | 416 | controller = None 417 | 418 | def cleanup(): 419 | if guimode or sett.get('persistent'): 420 | setting.save_setting() 421 | controller.quit() 422 | import time 423 | time.sleep(1) # give time to other threads to quit 424 | 425 | def signal_handler(signum, frame): 426 | print('\n\nuser interrupt operation, cleanup ...') 427 | signal.signal(signum, signal.SIG_IGN) # ignore additional signals 428 | cleanup() 429 | print('\n\ndone cleanup ...') 430 | sys.exit(0) 431 | 432 | signal.signal(signal.SIGINT, signal_handler) 433 | 434 | # ------------------------------------------------------------------------------------------------------------------ 435 | # if running application without arguments will start the gui, otherwise will run application in cmdline 436 | if guimode: 437 | # GUI 438 | controller = Controller(view_class=MainWindow, custom_settings=sett) 439 | controller.run() 440 | else: 441 | controller = Controller(view_class=CmdView, custom_settings=sett) 442 | controller.run() 443 | 444 | if sett.get('update_self'): 445 | controller.check_for_update(wait=True, threaded=False) 446 | sys.exit(0) 447 | 448 | urls = sett.pop('url') # list of urls or empty list 449 | 450 | if sett.get('batch_file'): 451 | text = sett['batch_file'].read() 452 | urls += parse_urls(text) 453 | 454 | if not urls: 455 | print('No url(s) to download') 456 | 457 | elif sett.get('interactive'): 458 | for url in urls: 459 | controller.interactive_download(url, **sett) 460 | else: 461 | controller.cmdline_download(urls, **sett) 462 | 463 | cleanup() 464 | 465 | 466 | if __name__ == '__main__': 467 | main(sys.argv) 468 | 469 | -------------------------------------------------------------------------------- /vortexdm/__init__.py: -------------------------------------------------------------------------------- 1 | # not required for python 3+ 2 | -------------------------------------------------------------------------------- /vortexdm/__main__.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | """ 3 | Vortex Download Manager (VortexDM) 4 | 5 | A multi-connection internet download manager, based on "PycURL" and "youtube_dl". Original project, FireDM, by Mahmoud Elshahat. 6 | :copyright: (c) 2023 by Sixline 7 | :copyright: (c) 2019-2021 by Mahmoud Elshahat. 8 | :license: GNU GPLv3, see LICENSE.md for more details. 9 | """ 10 | # main module executed when run command 11 | # python -m vortexdm 12 | 13 | import sys 14 | 15 | if __package__ is None and not hasattr(sys, 'frozen'): 16 | # direct call of __main__.py 17 | import os.path 18 | path = os.path.realpath(os.path.abspath(__file__)) 19 | sys.path.insert(0, os.path.dirname(os.path.dirname(path))) 20 | 21 | from vortexdm import VortexDM 22 | 23 | if __name__ == '__main__': 24 | VortexDM.main() -------------------------------------------------------------------------------- /vortexdm/about.py: -------------------------------------------------------------------------------- 1 | about_notes = """Vortex Download Manager (VortexDM) - An open-source Python 2 | Internet download manager with a high speed multi-connection engine. 3 | 4 | Developed in Python, based on "PycURL" and "youtube_dl". 5 | 6 | :copyright: (c) 2023 by Sixline 7 | :copyright: (c) 2019-2021 by Mahmoud Elshahat. 8 | :license: GNU GPLv3, see LICENSE.md for more details. 9 | """ -------------------------------------------------------------------------------- /vortexdm/brain.py: -------------------------------------------------------------------------------- 1 | """ 2 | Vortex Download Manager (VortexDM) 3 | 4 | A multi-connection internet download manager, based on "PycURL" and "youtube_dl". Original project, FireDM, by Mahmoud Elshahat. 5 | :copyright: (c) 2023 by Sixline 6 | :copyright: (c) 2019-2021 by Mahmoud Elshahat. 7 | :license: GNU GPLv3, see LICENSE.md for more details. 8 | """ 9 | import os 10 | import time 11 | from threading import Thread 12 | from queue import Queue 13 | import concurrent.futures 14 | 15 | from .video import merge_video_audio, pre_process_hls, post_process_hls, \ 16 | convert_audio, download_subtitles, write_metadata 17 | from . import config 18 | from .config import Status 19 | from .utils import (log, format_bytes, delete_file, rename_file, run_command, read_in_chunks) 20 | from .worker import Worker 21 | from .downloaditem import Segment 22 | 23 | 24 | def brain(d=None): 25 | """main brain for a single download, it controls thread manger, file manager 26 | """ 27 | 28 | # set status 29 | if d.status == Status.downloading: 30 | log('another brain thread may be running') 31 | return 32 | else: 33 | d.status = Status.downloading 34 | 35 | # first we will remove temp files because file manager is appending segments blindly to temp file 36 | delete_file(d.temp_file) 37 | delete_file(d.audio_file) 38 | 39 | # reset downloaded 40 | d.downloaded = 0 41 | 42 | log('\n') 43 | log('-' * 50) 44 | log(f'start downloading file: "{d.name}", size: {format_bytes(d.total_size)}, to: {d.folder}') 45 | log(f'url: "{d.url}" \n') 46 | 47 | # hls / m3u8 protocols 48 | if 'hls' in d.subtype_list: 49 | try: 50 | success = pre_process_hls(d) 51 | if not success: 52 | d.status = Status.error 53 | return 54 | except Exception as e: 55 | d.status = Status.error 56 | log('pre_process_hls()> error: ', e, showpopup=True) 57 | if config.test_mode: 58 | raise e 59 | return 60 | else: 61 | # build segments 62 | d.build_segments() 63 | 64 | # load progress info 65 | d.load_progress_info() 66 | 67 | # create some queues to send quit flag to threads 68 | fpr_q = Queue() 69 | spr_q = Queue() 70 | tm_q = Queue() 71 | fm_q = Queue() 72 | 73 | if d.type == config.MediaType.video: 74 | # run files processing reporter in a separate thread 75 | Thread(target=fpr, daemon=True, args=(d, fpr_q)).start() 76 | 77 | Thread(target=spr, daemon=True, args=(d, spr_q)).start() 78 | 79 | # run file manager in a separate thread 80 | Thread(target=file_manager, daemon=True, args=(d, fm_q)).start() 81 | 82 | # run thread manager in a separate thread 83 | Thread(target=thread_manager, daemon=True, args=(d, tm_q)).start() 84 | 85 | while True: 86 | time.sleep(0.1) # a sleep time to make the program responsive 87 | 88 | if d.status not in Status.active_states: 89 | log(f'File {d.status}.', log_level=2) 90 | break 91 | 92 | # check file size 93 | if os.path.isfile(d.target_file): 94 | fs = os.path.getsize(d.target_file) 95 | if fs == 0: 96 | log('error, nothing downloaded, file size is zero:', d.name) 97 | d.status = Status.error 98 | os.unlink(d.target_file) 99 | 100 | # report quitting 101 | log(f'brain {d.uid}: quitting', log_level=2) 102 | for q in (spr_q, fpr_q, tm_q, fm_q): 103 | q.put('quit') 104 | 105 | log('-' * 50, '\n', log_level=2) 106 | 107 | 108 | def file_manager(d, q, keep_segments=True): 109 | """write downloaded segments to a single file, and report download completed""" 110 | 111 | # create temp folder if it doesn't exist 112 | if not os.path.isdir(d.temp_folder): 113 | os.mkdir(d.temp_folder) 114 | 115 | # create temp files, needed for future opening in 'rb+' mode otherwise it will raise file not found error 116 | temp_files = set([seg.tempfile for seg in d.segments]) 117 | for file in temp_files: 118 | open(file, 'ab').close() 119 | 120 | # report all blocks 121 | d.update_segments_progress() 122 | 123 | while True: 124 | time.sleep(0.1) 125 | 126 | job_list = [seg for seg in d.segments if not seg.completed] 127 | 128 | # sort segments based on ranges, faster in writing to target file 129 | try: 130 | if job_list and job_list[0].range: 131 | # it will raise "TypeError: 'NoneType' object is not subscriptable" if for example video is normal 132 | # and audio is fragmented, the latter will have range=None 133 | job_list = sorted(job_list, key=lambda seg: seg.range[0]) 134 | except: 135 | pass 136 | 137 | for seg in job_list: 138 | 139 | # for segments which have no range, it must be appended to temp file in order, or final file will be 140 | # corrupted, therefore if the first non completed segment is not "downloaded", will exit loop 141 | if not seg.downloaded: 142 | if not seg.range: 143 | break 144 | else: 145 | continue 146 | 147 | # append downloaded segment to temp file, mark as completed 148 | try: 149 | seg.merge_errors = getattr(seg, 'merge_errors', 0) 150 | if seg.merge_errors > 10: 151 | log('merge max errors exceeded for:', seg.name, seg.last_merge_error) 152 | d.status = Status.error 153 | break 154 | elif seg.merge_errors > 0: 155 | time.sleep(1) 156 | 157 | if seg.merge: 158 | 159 | # use 'rb+' mode if we use seek, 'ab' doesn't work, 'rb+' will raise error if file doesn't exist 160 | # open/close target file with every segment will avoid operating system buffering, which cause 161 | # almost 90 sec wait on some windows machine to be able to rename the file, after close it 162 | # fd.flush() and os.fsync(fd) didn't solve the problem 163 | if seg.range: 164 | target_file = open(seg.tempfile, 'rb+') 165 | # must seek exact position, segments are not in order for simple append 166 | target_file.seek(seg.range[0]) 167 | 168 | # read file in chunks to save memory in case of big segments 169 | # read the exact segment size, sometimes segment has extra data as a side effect of 170 | # auto segmentation 171 | chunks = read_in_chunks(seg.name, bytes_range=(0, seg.range[1] - seg.range[0]), flag='rb') 172 | else: 173 | target_file = open(seg.tempfile, 'ab') 174 | chunks = read_in_chunks(seg.name) 175 | 176 | # write data 177 | for chunk in chunks: 178 | target_file.write(chunk) 179 | 180 | # close file 181 | target_file.close() 182 | 183 | seg.completed = True 184 | log('completed segment: ', seg.basename, log_level=3) 185 | 186 | if not keep_segments and not config.keep_temp: 187 | delete_file(seg.name) 188 | 189 | except Exception as e: 190 | seg.merge_errors += 1 191 | seg.last_merge_error = e 192 | log('failed to merge segment', seg.name, ' - ', seg.range, ' - ', e) 193 | 194 | if config.test_mode: 195 | raise e 196 | 197 | # all segments already merged 198 | if not job_list: 199 | 200 | # handle HLS streams 201 | if 'hls' in d.subtype_list: 202 | log('handling hls videos') 203 | # Set status to processing 204 | d.status = Status.processing 205 | 206 | success = post_process_hls(d) 207 | if not success: 208 | if d.status == Status.processing: 209 | d.status = Status.error 210 | log('file_manager()> post_process_hls() failed, file: \n', d.name, showpopup=True) 211 | break 212 | 213 | # handle dash video 214 | if 'dash' in d.subtype_list: 215 | log('handling dash videos', log_level=2) 216 | # merge audio and video 217 | output_file = d.target_file 218 | 219 | # set status to processing 220 | d.status = Status.processing 221 | error, output = merge_video_audio(d.temp_file, d.audio_file, output_file, d) 222 | 223 | if not error: 224 | log('done merging video and audio for: ', d.target_file, log_level=2) 225 | 226 | # delete temp files 227 | d.delete_tempfiles() 228 | 229 | else: # error merging 230 | d.status = Status.error 231 | log('failed to merge audio for file: \n', d.name, showpopup=True) 232 | break 233 | 234 | # handle audio streams 235 | if d.type == 'audio': 236 | log('handling audio streams') 237 | d.status = Status.processing 238 | success = convert_audio(d) 239 | if not success: 240 | d.status = Status.error 241 | log('file_manager()> convert_audio() failed, file:', d.target_file, showpopup=True) 242 | break 243 | else: 244 | d.delete_tempfiles() 245 | 246 | else: 247 | # final / target file might be created by ffmpeg in case of dash video for example 248 | if os.path.isfile(d.target_file): 249 | # delete temp files 250 | d.delete_tempfiles() 251 | else: 252 | # report video progress before renaming temp video file 253 | d.update_media_files_progress() 254 | 255 | # rename temp file 256 | success = rename_file(d.temp_file, d.target_file) 257 | if success: 258 | # delete temp files 259 | d.delete_tempfiles() 260 | 261 | # download subtitles 262 | if d.selected_subtitles: 263 | Thread(target=download_subtitles, args=(d.selected_subtitles, d)).start() 264 | 265 | # if type is subtitle, will convert vtt to srt 266 | if d.type == 'subtitle' and 'hls' not in d.subtype_list and d.name.endswith('srt'): 267 | # ffmpeg file full location 268 | ffmpeg = config.ffmpeg_actual_path 269 | 270 | input_file = d.target_file 271 | output_file = f'{d.target_file}2.srt' # must end with srt for ffmpeg to recognize output format 272 | 273 | log('verifying "srt" subtitle:', input_file, log_level=2) 274 | cmd = f'"{ffmpeg}" -y -i "{input_file}" "{output_file}"' 275 | 276 | error, _ = run_command(cmd, verbose=True) 277 | if not error: 278 | delete_file(d.target_file) 279 | rename_file(oldname=output_file, newname=input_file) 280 | log('verified subtitle successfully:', input_file, log_level=2) 281 | else: 282 | # if failed to convert 283 | log("couldn't convert subtitle to srt, check file format might be corrupted") 284 | 285 | # write metadata 286 | if d.metadata_file_content and config.write_metadata: 287 | log('file manager()> writing metadata info to:', d.name, log_level=2) 288 | # create metadata file 289 | metadata_filename = d.target_file + '.meta' 290 | 291 | try: 292 | with open(metadata_filename, 'w', encoding="utf-8") as f: 293 | f.write(d.metadata_file_content) 294 | 295 | # let ffmpeg write metadata to file 296 | write_metadata(d.target_file, metadata_filename) 297 | 298 | except Exception as e: 299 | log('file manager()> writing metadata error:', e) 300 | 301 | finally: 302 | # delete meta file 303 | delete_file(metadata_filename) 304 | 305 | # at this point all done successfully 306 | # report all blocks 307 | d.update_segments_progress() 308 | 309 | d.status = Status.completed 310 | # print('---------file manager done merging segments---------') 311 | break 312 | 313 | # change status or get quit signal from brain 314 | try: 315 | if d.status != Status.downloading or q.get_nowait() == 'quit': 316 | # print('--------------file manager cancelled-----------------') 317 | break 318 | except: 319 | pass 320 | 321 | # save progress info for future resuming 322 | if os.path.isdir(d.temp_folder): 323 | d.save_progress_info() 324 | 325 | # Report quitting 326 | log(f'file_manager {d.uid}: quitting', log_level=2) 327 | 328 | 329 | def thread_manager(d, q): 330 | """create multiple worker threads to download file segments""" 331 | 332 | # soft start, connections will be gradually increase over time to reach max. number 333 | # set by user, this prevent impact on servers/network, and avoid "service not available" response 334 | # from server when exceeding multi-connection number set by server. 335 | limited_connections = 1 336 | 337 | # create worker/connection list 338 | all_workers = [Worker(tag=i, d=d) for i in range(config.max_connections)] 339 | free_workers = set([w for w in all_workers]) 340 | threads_to_workers = dict() 341 | 342 | num_live_threads = 0 343 | 344 | def sort_segs(segs): 345 | # sort segments based on their range in reverse to use .pop() 346 | def sort_key(seg): 347 | if seg.range: 348 | return seg.range[0] 349 | else: 350 | return seg.num 351 | 352 | def sort(_segs): 353 | return sorted(_segs, key=sort_key, reverse=True) 354 | 355 | try: 356 | video_segs = [seg for seg in segs if seg.media_type == config.MediaType.video] 357 | other_segs = [seg for seg in segs if seg not in video_segs] 358 | 359 | # put video at the end of the list to get processed first 360 | sorted_segs = sort(other_segs) + sort(video_segs) 361 | # log('sorted seg:', [f'{seg.media_type}-{seg.range}' for seg in sorted_segs]) 362 | except Exception as e: 363 | sorted_segs = segs 364 | log('sort_segs error:', e) 365 | 366 | # print('segs:', [f'{seg.media_type}-{seg.num}' for seg in segs]) 367 | # print('sorted_segs:', [f'{seg.media_type}-{seg.num}' for seg in sorted_segs]) 368 | return sorted_segs 369 | 370 | # job_list 371 | job_list = [seg for seg in d.segments if not seg.downloaded] 372 | job_list = sort_segs(job_list) 373 | 374 | d.remaining_parts = len(job_list) 375 | 376 | # error track, if receive many errors with no downloaded data, abort 377 | downloaded = 0 378 | total_errors = 0 379 | max_errors = 100 380 | errors_descriptions = set() # store unique errors 381 | error_timer = 0 382 | error_timer2 = 0 383 | conn_increase_interval = 0.5 384 | errors_check_interval = 0.2 # in seconds 385 | segmentation_timer = 0 386 | 387 | # speed limit 388 | sl_timer = time.time() 389 | 390 | def clear_error_q(): 391 | # clear error queue 392 | for _ in range(config.error_q.qsize()): 393 | errors_descriptions.add(config.error_q.get()) 394 | 395 | while True: 396 | time.sleep(0.001) # a sleep time to while loop to make the app responsive 397 | 398 | # Failed jobs returned from workers, will be used as a flag to rebuild job_list -------------------------------- 399 | if config.jobs_q.qsize() > 0: 400 | # rebuild job_list 401 | job_list = [seg for seg in d.segments if not seg.downloaded and not seg.locked] 402 | 403 | # sort segments based on its ranges smaller ranges at the end 404 | job_list = sort_segs(job_list) 405 | 406 | # empty queue 407 | for _ in range(config.jobs_q.qsize()): 408 | _ = config.jobs_q.get() 409 | 410 | # create new workers if user increases max_connections while download is running 411 | if config.max_connections > len(all_workers): 412 | extra_num = config.max_connections - len(all_workers) 413 | index = len(all_workers) 414 | for i in range(extra_num): 415 | index += i 416 | worker = Worker(tag=index, d=d) 417 | all_workers.append(worker) 418 | free_workers.add(worker) 419 | 420 | # allowable connections 421 | allowable_connections = min(config.max_connections, limited_connections) 422 | 423 | # dynamic connection manager --------------------------------------------------------------------------------- 424 | # check every n seconds for connection errors 425 | if time.time() - error_timer >= errors_check_interval: 426 | error_timer = time.time() 427 | errors_num = config.error_q.qsize() 428 | 429 | total_errors += errors_num 430 | d.errors = total_errors # update errors property of download item 431 | 432 | clear_error_q() 433 | 434 | if total_errors >= 1 and limited_connections > 1: 435 | limited_connections -= 1 436 | conn_increase_interval += 1 437 | error_timer2 = time.time() 438 | log('Errors:', errors_descriptions, 'Total:', total_errors, log_level=3) 439 | log('Thread Manager: received server errors, connections limited to:', limited_connections, log_level=3) 440 | 441 | elif limited_connections < config.max_connections and time.time() - error_timer2 >= conn_increase_interval: 442 | error_timer2 = time.time() 443 | limited_connections += 1 444 | log('Thread Manager: allowable connections:', limited_connections, log_level=3) 445 | 446 | # reset total errors if received any data 447 | if downloaded != d.downloaded: 448 | downloaded = d.downloaded 449 | # print('reset errors to zero') 450 | total_errors = 0 451 | clear_error_q() 452 | 453 | if total_errors >= max_errors: 454 | d.status = Status.error 455 | 456 | # speed limit ------------------------------------------------------------------------------------------------ 457 | # wait some time for dynamic connection manager to release all connections 458 | if time.time() - sl_timer < config.max_connections * errors_check_interval: 459 | worker_sl = (config.speed_limit // config.max_connections) if config.max_connections else 0 460 | else: 461 | worker_sl = (config.speed_limit // allowable_connections) if allowable_connections else 0 462 | 463 | # Threads ------------------------------------------------------------------------------------------------------ 464 | if d.status == Status.downloading: 465 | if free_workers and num_live_threads < allowable_connections: 466 | seg = None 467 | if job_list: 468 | seg = job_list.pop() 469 | 470 | # Auto file segmentation, share segments and help other workers 471 | elif time.time() - segmentation_timer >= 1: 472 | segmentation_timer = time.time() 473 | 474 | # calculate minimum segment size based on speed, e.g. for 3 MB/s speed, and 2 live threads, 475 | # worker speed = 1.5 MB/sec, min seg size will be 1.5 x 6 = 9 MB 476 | worker_speed = d.speed // num_live_threads if num_live_threads else 0 477 | min_seg_size = max(config.SEGMENT_SIZE, worker_speed * 6) 478 | 479 | filtered_segs = [seg for seg in d.segments if seg.range is not None 480 | and seg.remaining > min_seg_size * 2] 481 | 482 | # sort segments based on its ranges smaller ranges at the end 483 | filtered_segs = sort_segs(filtered_segs) 484 | 485 | if filtered_segs: 486 | current_seg = filtered_segs.pop() 487 | 488 | # range boundaries 489 | start = current_seg.range[0] 490 | middle = start + current_seg.current_size + current_seg.remaining // 2 491 | end = current_seg.range[1] 492 | 493 | # assign new range for current segment 494 | current_seg.range = [start, middle] 495 | 496 | # create new segment 497 | seg = Segment(name=os.path.join(d.temp_folder, f'{len(d.segments)}'), url=current_seg.url, 498 | tempfile=current_seg.tempfile, range=[middle + 1, end], 499 | media_type=current_seg.media_type) 500 | 501 | # add to segments 502 | d.segments.append(seg) 503 | log('-' * 10, f'new segment: {seg.basename} {seg.range}, updated seg {current_seg.basename} ' 504 | f'{current_seg.range}, minimum seg size:{format_bytes(min_seg_size)}', log_level=3) 505 | 506 | if seg and not seg.downloaded and not seg.locked: 507 | worker = free_workers.pop() 508 | # sometimes download chokes when remaining only one worker, will set higher minimum speed and 509 | # less timeout for last workers batch 510 | if len(job_list) + config.jobs_q.qsize() <= allowable_connections: 511 | # worker will abort if speed less than 20 KB for 10 seconds 512 | minimum_speed, timeout = 20 * 1024, 10 513 | else: 514 | minimum_speed = timeout = None # default as in utils.set_curl_option 515 | 516 | ready = worker.reuse(seg=seg, speed_limit=worker_sl, minimum_speed=minimum_speed, timeout=timeout) 517 | if ready: 518 | # check max download retries 519 | if seg.retries >= config.max_seg_retries: 520 | log('seg:', seg.basename, f'exceeded max. of ({config.max_seg_retries}) download retries,', 521 | 'try to decrease num of connections in settings and try again') 522 | d.status = Status.error 523 | else: 524 | seg.retries += 1 525 | 526 | thread = Thread(target=worker.run, daemon=True) 527 | thread.start() 528 | threads_to_workers[thread] = worker 529 | 530 | # save progress info for future resuming 531 | if os.path.isdir(d.temp_folder): 532 | d.save_progress_info() 533 | 534 | # check thread completion 535 | for thread in list(threads_to_workers.keys()): 536 | if not thread.is_alive(): 537 | worker = threads_to_workers.pop(thread) 538 | free_workers.add(worker) 539 | 540 | # update d param ----------------------------------------------------------------------------------------------- 541 | num_live_threads = len(all_workers) - len(free_workers) 542 | d.live_connections = num_live_threads 543 | d.remaining_parts = d.live_connections + len(job_list) + config.jobs_q.qsize() 544 | 545 | # Required check if things goes wrong -------------------------------------------------------------------------- 546 | if num_live_threads + len(job_list) + config.jobs_q.qsize() == 0: 547 | # rebuild job_list 548 | job_list = [seg for seg in d.segments if not seg.downloaded] 549 | if not job_list: 550 | break 551 | else: 552 | # remove an orphan locks 553 | for seg in job_list: 554 | seg.locked = False 555 | 556 | # monitor status change or get quit signal from brain ---------------------------------------------------------- 557 | try: 558 | if d.status != Status.downloading or q.get_nowait() == 'quit': 559 | break 560 | except: 561 | pass 562 | 563 | # update d param 564 | d.live_connections = 0 565 | d.remaining_parts = num_live_threads + len(job_list) + config.jobs_q.qsize() 566 | log(f'thread_manager {d.uid}: quitting', log_level=2) 567 | 568 | 569 | def fpr(d, q): 570 | """file processing progress reporter 571 | 572 | Args: 573 | d: DownloadItem object 574 | """ 575 | 576 | while True: 577 | 578 | d.update_media_files_progress() 579 | 580 | try: 581 | if d.status not in config.Status.active_states or q.get_nowait() == 'quit': 582 | break 583 | except: 584 | pass 585 | 586 | time.sleep(1) 587 | 588 | 589 | def spr(d, q): 590 | """segments progress reporter 591 | 592 | Args: 593 | d: DownloadItem object 594 | """ 595 | 596 | while True: 597 | 598 | try: 599 | if d.status not in config.Status.active_states or q.get_nowait() == 'quit': 600 | break 601 | except: 602 | pass 603 | 604 | # report active blocks only 605 | d.update_segments_progress(activeonly=True) 606 | 607 | time.sleep(1) -------------------------------------------------------------------------------- /vortexdm/cmdview.py: -------------------------------------------------------------------------------- 1 | """ 2 | Vortex Download Manager (VortexDM) 3 | 4 | A multi-connection internet download manager, based on "PycURL" and "youtube_dl". Original project, FireDM, by Mahmoud Elshahat. 5 | :copyright: (c) 2023 by Sixline 6 | :copyright: (c) 2019-2021 by Mahmoud Elshahat. 7 | :license: GNU GPLv3, see LICENSE.md for more details. 8 | 9 | Module description: 10 | This is a command line / Terminal view, as a layer between user and controller 11 | it must inherit from IView and implement all its abstract methods, see view.py 12 | currently it runs in interactive mode and not suitable for automated jobs. 13 | 14 | """ 15 | 16 | import os 17 | import sys 18 | import shutil 19 | from queue import Queue 20 | from threading import Event 21 | from collections import namedtuple 22 | 23 | if not __package__: 24 | __package__ = 'vortexdm' 25 | 26 | from vortexdm.view import IView 27 | from vortexdm import utils 28 | from vortexdm.utils import format_bytes, format_seconds 29 | from vortexdm import config 30 | 31 | 32 | def write(s, end=''): 33 | sys.stdout.write(s + end) 34 | sys.stdout.flush() 35 | 36 | 37 | terminal_size = namedtuple('terminal_size', ('width', 'height')) 38 | 39 | 40 | def get_terminal_size(): 41 | """get terminal window size, return 2-tuple (width, height)""" 42 | try: 43 | size = shutil.get_terminal_size() 44 | except: 45 | # default fallback values 46 | size = (100, 20) 47 | return terminal_size(*size) 48 | 49 | 50 | class CmdView(IView): 51 | """concrete class for terminal user interface""" 52 | 53 | def __init__(self, controller=None): 54 | self.controller = controller 55 | self.total_size = None 56 | self.progress = 0 57 | self.terminal_size = None 58 | self.printing_bar = Event() 59 | self.sdt_buffer = Queue() 60 | 61 | self.printing_bar.clear() 62 | 63 | def reserve_last_line(self): 64 | self.printing_bar.set() 65 | 66 | write("\0337") # Save cursor position 67 | write(f"\033[0;{self.terminal_size.height - 1}r") # Reserve the bottom line 68 | write("\0338") # Restore the cursor position 69 | write("\033[1A") # Move up one line 70 | 71 | self.printing_bar.clear() 72 | 73 | def release_last_line(self): 74 | self.printing_bar.set() 75 | 76 | write("\0337") # Save cursor position 77 | write(f"\033[0;{self.terminal_size.height}r") # Drop margin reservation 78 | write(f"\033[{self.terminal_size.height};0f") # Move the cursor to the bottom line 79 | write("\033[0K") # Clean that line 80 | write("\0338") 81 | 82 | self.printing_bar.clear() 83 | 84 | def run(self): 85 | """setup terminal for progress bar""" 86 | if config.operating_system == 'Windows': 87 | return 88 | 89 | utils.my_print = self.normal_print 90 | self.terminal_size = get_terminal_size() 91 | self.reserve_last_line() 92 | 93 | def quit(self): 94 | """reset terminal""" 95 | if config.operating_system == 'Windows': 96 | return 97 | 98 | self.release_last_line() 99 | utils.my_print = print 100 | 101 | def print_progress_bar(self, percent, suffix='', bar_length=20, fill='█'): 102 | """print progress bar to screen, percent is number between 0 and 100""" 103 | 104 | scale = bar_length / 100 105 | filled_length = int(percent * scale) 106 | bar = fill * filled_length + ' ' * (bar_length - filled_length) 107 | 108 | # get screen size 109 | terminal = get_terminal_size() 110 | 111 | line = f'\r {percent}% [{bar}] {suffix}' 112 | end_spaces = terminal.width - len(line) 113 | line += ' ' * end_spaces 114 | 115 | # truncate line 116 | line = line[:terminal.width] 117 | 118 | if config.operating_system == 'Windows': 119 | write(line, end='\r') 120 | return 121 | 122 | # terminal size has changed 123 | if terminal != self.terminal_size: 124 | self.release_last_line() 125 | self.terminal_size = terminal 126 | self.reserve_last_line() 127 | 128 | self.print_onlast(line) 129 | 130 | def print_onlast(self, s): 131 | self.printing_bar.set() 132 | 133 | write("\0337") # Save cursor position 134 | write(f"\033[{self.terminal_size.height};0f") # Move cursor to the bottom margin 135 | write(s, end='\r') # Write the data 136 | write("\0338") # Restore cursor position 137 | 138 | self.printing_bar.clear() 139 | 140 | def normal_print(self, s, end='\n'): 141 | self.sdt_buffer.put(s) 142 | if self.printing_bar.is_set(): 143 | return 144 | 145 | for _ in range(self.sdt_buffer.qsize()): 146 | write(self.sdt_buffer.get(), end=end) 147 | 148 | def update_view(self, total_size=None, **kwargs): 149 | """update view""" 150 | 151 | progress = kwargs.get('progress', 0) 152 | speed = kwargs.get('speed') 153 | eta = kwargs.get('eta') 154 | downloaded = kwargs.get('downloaded') 155 | if total_size: 156 | self.total_size = total_size 157 | 158 | # in terminal view, it will be only one download takes place in a time 159 | # since there is no updates coming from d_list items, it is easier to identify the currently downloading item 160 | # by checking progress 161 | if 0 < progress < 100 <= self.progress: 162 | self.progress = progress 163 | 164 | if progress > 0 and self.progress < 100: 165 | # print progress bar on screen 166 | 167 | suffix = f'{format_bytes(downloaded, sep="", percision=1)}' 168 | if self.total_size: 169 | suffix += f'/{format_bytes(self.total_size, sep="", percision=1)}' 170 | suffix += f" {format_bytes(speed, tail='/s', percision=1)}" if speed else '' 171 | suffix += f', {format_seconds(eta, percision=0, fullunit=True)}' if eta else '' 172 | 173 | try: 174 | self.print_progress_bar(progress, suffix=suffix, fill='=') 175 | except: 176 | if config.test_mode: 177 | raise 178 | 179 | # to ignore repeated updates after 100% 180 | self.progress = progress 181 | 182 | def get_user_response(self, msg, options, **kwargs): 183 | """a mimic for a popup window in terminal, to get user response, 184 | example: if msg = "File with the same name already exists\n 185 | /home/mahmoud/Downloads/7z1900.exe\n 186 | Do you want to overwrite file? " 187 | 188 | and option = ['Overwrite', 'Cancel download'] 189 | the resulting box will looks like: 190 | 191 | ******************************************* 192 | * File with the same name already exists * 193 | * /home/mahmoud/Downloads/7z1900.exe * 194 | * Do you want to overwrite file? * 195 | * --------------------------------------- * 196 | * Options: * 197 | * 1: Overwrite * 198 | * 2: Cancel download * 199 | ******************************************* 200 | Select Option Number: 201 | 202 | """ 203 | # map options to numbers starting from 1 204 | options_map = {i + 1: x for i, x in enumerate(options)} 205 | 206 | # split message to list of lines 207 | msg_lines = msg.split('\n') 208 | 209 | # format options in lines example: " 1: Overwrite", and " 2: Cancel download" 210 | options_lines = [f' {k}: {str(v)}' for k, v in options_map.items()] 211 | 212 | # get the width of longest line in msg body or options 213 | max_line_width = max(max([len(line) for line in msg_lines]), max([len(line) for line in options_lines])) 214 | 215 | # get current terminal window size (width) 216 | terminal_width = get_terminal_size()[0] 217 | 218 | # the overall width of resulting msg box including border ('*' stars in our case) 219 | box_width = min(max_line_width + 4, terminal_width) 220 | 221 | # build lines without border 222 | output_lines = [] 223 | output_lines += msg_lines 224 | separator = '-' * (box_width - 4) 225 | output_lines.append(separator) 226 | output_lines.append("Options:") 227 | output_lines += options_lines 228 | 229 | # add stars and space padding for each line 230 | for i, line in enumerate(output_lines): 231 | allowable_line_width = box_width - 4 232 | 233 | # calculate the required space to fill the line 234 | delta = allowable_line_width - len(line) if allowable_line_width > len(line) else 0 235 | 236 | # add stars 237 | line = '* ' + line + ' ' * delta + ' *' 238 | 239 | output_lines[i] = line 240 | 241 | # create message string 242 | msg = '\n'.join(output_lines) 243 | msg = '\n' + '*' * box_width + '\n' + msg + '\n' + '*' * box_width 244 | msg += '\n Select Option Number: ' 245 | 246 | while True: 247 | txt = input(msg) 248 | try: 249 | # get user selection 250 | # it will raise exception if user tries to input number not in options_map 251 | response = options_map[int(txt)] 252 | print() # print empty line 253 | break # exit while loop if user entered a valid selection 254 | except: 255 | print('\n invalid entry, try again.\n') 256 | 257 | return response 258 | -------------------------------------------------------------------------------- /vortexdm/config.py: -------------------------------------------------------------------------------- 1 | """ 2 | Vortex Download Manager (VortexDM) 3 | 4 | A multi-connection internet download manager, based on "PycURL" and "youtube_dl". Original project, FireDM, by Mahmoud Elshahat. 5 | :copyright: (c) 2023 by Sixline 6 | :copyright: (c) 2019-2021 by Mahmoud Elshahat. 7 | :license: GNU GPLv3, see LICENSE.md for more details. 8 | """ 9 | 10 | from queue import Queue 11 | import os 12 | import sys 13 | import platform 14 | 15 | from .version import __version__ 16 | 17 | # settings parameters to be saved on disk 18 | settings_keys = [ 19 | 'active_video_extractor', 'auto_rename', 'autoscroll_download_tab', 'check_for_update', 'checksum', 'current_theme', 20 | 'custom_user_agent', 'disable_log_popups', 'ditem_show_top', 'download_folder', 'download_thumbnail', 21 | 'enable_proxy', 'enable_systray', 'gui_font', 'ibus_workaround', 'ignore_ssl_cert', 22 | 'keep_temp', 'last_update_check', 'log_level', 'max_concurrent_downloads', 'remember_web_auth', 'use_web_auth', 23 | 'username', 'password', 'max_connections', 'minimize_to_systray', 'monitor_clipboard', 'on_download_notification', 24 | 'proxy', 'recent_folders', 'refresh_url_retries', 'scrollbar_width', 'speed_limit', 'update_frequency', 25 | 'playlist_autonum_options', 'use_server_timestamp', 'window_size', 'write_metadata', 'view_mode', 'temp_folder', 26 | 'window_maximized', 'force_window_maximize', 'd_preview', 'updater_version', 'media_presets', 27 | 'video_title_template', 'ffmpeg_actual_path' 28 | ] 29 | 30 | # ----------------------------------------------------------------------------------------General ---------------------- 31 | # CONSTANTS 32 | APP_NAME = 'Vortex Download Manager' 33 | APP_VERSION = __version__ 34 | APP_TITLE = f'{APP_NAME} Version {APP_VERSION} .. an open-source download manager' 35 | 36 | # minimum segment size used in auto-segmentation process, refer to brain.py>thread_manager. 37 | SEGMENT_SIZE = 1024 * 100 # 100 KB 38 | 39 | APP_URL = 'https://github.com/Sixline/VortexDM' 40 | LATEST_RELEASE_URL = 'https://github.com/Sixline/VortexDM/releases/latest' 41 | 42 | FROZEN = getattr(sys, "frozen", False) # check if app is being compiled by cx_freeze 43 | 44 | operating_system = platform.system() # current operating system ('Windows', 'Linux', 'Darwin') 45 | 46 | # Example output: Os: Linux - Platform: Linux-5.11.0-7614-generic-x86_64-with-glibc2.32 - Machine: x86_64 47 | operating_system_info = f"OS: {platform.system()} - Platform: {platform.platform()} - Machine: {platform.machine()}" 48 | 49 | try: 50 | import distro 51 | 52 | # Example output: Distribution: ('Pop!_OS', '20.10', 'groovy') 53 | operating_system_info += f"\nDistribution: {distro.linux_distribution(full_distribution_name=True)}" 54 | except: 55 | pass 56 | 57 | # release type 58 | isappimage = False # will be set to True by AppImage run script 59 | appimage_update_folder = None # will be set by AppImage run script 60 | 61 | # application exit flag 62 | shutdown = False 63 | 64 | on_download_notification = True # show notify message when done downloading 65 | 66 | # ----------------------------------------------------------------------------------------Filesystem options------------ 67 | # current folders 68 | if hasattr(sys, 'frozen'): # like if application frozen by cx_freeze 69 | current_directory = os.path.dirname(sys.executable) 70 | else: 71 | path = os.path.realpath(os.path.abspath(__file__)) 72 | current_directory = os.path.dirname(path) 73 | sys.path.insert(0, os.path.dirname(current_directory)) 74 | sys.path.insert(0, current_directory) 75 | 76 | sett_folder = None 77 | global_sett_folder = None 78 | download_folder = os.path.join(os.path.expanduser("~"), 'Downloads') 79 | recent_folders = [] 80 | 81 | auto_rename = False # auto rename file if there is an existing file with same name at download folder 82 | checksum = False # calculate checksums for completed files MD5 and SHA256 83 | playlist_autonum_options = dict( 84 | enable=True, 85 | reverse=False, 86 | zeropadding=True, 87 | ) 88 | 89 | # video file title template, ref: https://github.com/ytdl-org/youtube-dl#output-template 90 | video_title_template = '' # '%(title)s' 91 | 92 | temp_folder = '' 93 | # ---------------------------------------------------------------------------------------Network Options---------------- 94 | proxy = '' # must be string example: 127.0.0.1:8080 95 | enable_proxy = False 96 | 97 | # ---------------------------------------------------------------------------------------Authentication Options--------- 98 | use_web_auth = False 99 | remember_web_auth = False 100 | username = '' 101 | password = '' 102 | 103 | # ---------------------------------------------------------------------------------------Video Options------------------ 104 | # youtube-dl abort flag, will be used by decorated YoutubeDl.urlopen(), see video.set_interrupt_switch() 105 | ytdl_abort = False 106 | video_extractors_list = ['youtube_dl', 'yt_dlp'] 107 | active_video_extractor = 'yt_dlp' 108 | 109 | ffmpeg_actual_path = '' 110 | ffmpeg_version = '' 111 | ffmpeg_download_folder = sett_folder 112 | 113 | # media presets 114 | media_presets = dict( 115 | video_ext='mp4', 116 | video_quality='best', 117 | dash_audio='best', 118 | audio_ext='mp3', 119 | audio_quality='best' 120 | ) 121 | 122 | # video qualities 123 | vq = { 124 | 4320: '4320p-8K', 125 | 2160: '2160p-4K', 126 | 1440: '1440p-HD', 127 | 1080: '1080-HD', 128 | 720: '720p', 129 | 480: '480p', 130 | 360: '360p', 131 | 240: '240p', 132 | 144: '144p', 133 | } 134 | standard_video_qualities = list(vq.keys()) 135 | video_quality_choices = ['best'] + list(vq.values()) + ['lowest'] 136 | video_ext_choices = ('mp4', 'webm', '3gp') 137 | dash_audio_choices = ('best', 'lowest') 138 | audio_ext_choices = ('mp3', 'aac', 'wav', 'm4a', 'opus', 'flac', 'ogg', 'webm') 139 | audio_quality_choices = ('best', 'lowest') 140 | 141 | # ---------------------------------------------------------------------------------------Workarounds-------------------- 142 | ibus_workaround = False 143 | ignore_ssl_cert = False # ignore ssl certificate validation 144 | 145 | # a random user agent will be used later when importing youtube-dl, if no custom user agent 146 | DEFAULT_USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3721.3' 147 | custom_user_agent = None 148 | http_headers = { 149 | 'User-Agent': custom_user_agent or DEFAULT_USER_AGENT, 150 | 'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.7', 151 | 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 152 | 'Accept-Language': 'en-us,en;q=0.5', 153 | } 154 | 155 | use_referer = False 156 | referer_url = '' # referer website url 157 | use_cookies = False 158 | cookie_file_path = '' 159 | 160 | # ---------------------------------------------------------------------------------------Post-processing Options-------- 161 | download_thumbnail = False 162 | write_metadata = False # write metadata to video file 163 | shutdown_pc = False 164 | on_completion_command = '' 165 | on_completion_exit = False 166 | use_server_timestamp = False # write 'last modified' timestamp to downloaded file 167 | 168 | # ---------------------------------------------------------------------------------------Application Update Options----- 169 | # set this flag to True to disable update feature completely 170 | disable_update_feature = False 171 | 172 | check_for_update = not disable_update_feature 173 | update_frequency = 7 # days 174 | last_update_check = None # date format (year, month, day) 175 | updater_version = '' # application version that did last update check 176 | 177 | youtube_dl_version = None 178 | yt_dlp_version = None 179 | atk_version = None # awesometkinter 180 | 181 | # ---------------------------------------------------------------------------------------Downloader Options------------- 182 | refresh_url_retries = 1 # number of retries to refresh expired url when downloading a file, zero to disable 183 | speed_limit = 0 # in bytes, zero == no limit 184 | max_concurrent_downloads = 3 185 | max_connections = 10 186 | max_seg_retries = 10 # maximum download retries for a segment until reporting downloaded failed 187 | 188 | # ---------------------------------------------------------------------------------------Debugging options-------------- 189 | keep_temp = False # keep temp files / folders after done downloading for debugging 190 | 191 | max_log_size = 1024 * 1024 * 5 # 5 MB 192 | log_level = 2 # standard=1, verbose=2, debug=3 193 | 194 | # log callbacks that will be executed when calling log func in utils 195 | # callback and popup should accept 3 positional args e.g. log_callback(start, text, end) 196 | log_callbacks = [] 197 | log_popup_callback = None 198 | test_mode = False 199 | 200 | # ---------------------------------------------------------------------------------------GUI options-------------------- 201 | DEFAULT_THEME = 'Orange_Black' 202 | current_theme = DEFAULT_THEME 203 | gui_font = {} 204 | gui_font_size_default = 10 205 | gui_font_size_range = range(6, 26) 206 | 207 | scrollbar_width_default = 10 208 | scrollbar_width = scrollbar_width_default 209 | scrollbar_width_range = range(1, 51) 210 | monitor_clipboard = True 211 | autoscroll_download_tab = False 212 | ditem_show_top = True 213 | 214 | # systray, it will be disabled by default since it doesn't work properly on most operating systems except Windows. 215 | enable_systray = True if operating_system == 'Windows' else False 216 | minimize_to_systray = False 217 | 218 | DEFAULT_WINDOW_SIZE = (925, 500) # width, height in pixels 219 | window_size = DEFAULT_WINDOW_SIZE 220 | window_maximized = False 221 | force_window_maximize = False 222 | 223 | BULK = 'Bulk' 224 | COMPACT = 'Compact' 225 | MIX = 'Mix' 226 | DEFAULT_VIEW_MODE = MIX 227 | view_mode = DEFAULT_VIEW_MODE 228 | view_mode_choices = (COMPACT, BULK, MIX) 229 | view_filter = 'ALL' # show all 230 | d_preview = False # preview for download items 231 | 232 | # ---------------------------------------------------------------------------------------------------------------------- 233 | # ---------------------------------------------------------------------------------------------------------------------- 234 | 235 | # queues 236 | error_q = Queue() # used by workers to report server refuse connection errors 237 | jobs_q = Queue() # # required for failed worker jobs 238 | 239 | 240 | # status class as an Enum 241 | class Status: 242 | """used to identify status, work as an Enum""" 243 | downloading = 'Downloading' 244 | cancelled = 'Paused' 245 | completed = 'Completed' 246 | pending = 'Pending' 247 | processing = 'Processing' # for any ffmpeg operations 248 | error = 'Failed' 249 | scheduled = 'Scheduled' 250 | refreshing_url = 'Refreshing url' 251 | active_states = (downloading, processing, refreshing_url) 252 | all_states = (downloading, cancelled, completed, pending, processing, error, scheduled, refreshing_url) 253 | 254 | 255 | view_filter_map = { 256 | 'ALL': Status.all_states, 257 | 'Selected': (), 258 | 'Active': Status.active_states, 259 | 'Uncompleted': [s for s in Status.all_states if s != Status.completed] 260 | } 261 | 262 | for status in [x for x in Status.all_states if x not in Status.active_states]: 263 | view_filter_map[status] = (status,) 264 | 265 | 266 | # media type class 267 | class MediaType: 268 | general = 'general' 269 | video = 'video' 270 | audio = 'audio' 271 | key = 'key' 272 | 273 | 274 | # popup windows, get user responses 275 | disable_log_popups = False 276 | 277 | popups = { 278 | 1: {'tag': 'html contents', 279 | 'description': 'Show "Contents might be an html web page warning".', 280 | 'body': 'Contents might be a web page / html, Download anyway?', 281 | 'options': ['Ok', 'Cancel'], 282 | 'default': 'Ok', 283 | 'show': True 284 | }, 285 | 286 | 2: {'tag': 'ffmpeg', 287 | 'description': 'Prompt to download "FFMPEG" if not found on windows os.', 288 | 'body': 'FFMPEG is missing!', 289 | 'options': ['Download', 'Cancel'], 290 | 'default': 'Download', 291 | 'show': True 292 | }, 293 | 294 | 4: {'tag': 'overwrite file', 295 | 'description': 'Ask what to do if same file already exist on disk.', 296 | 'body': 'File with the same name already exist on disk', 297 | 'options': ['Overwrite', 'Rename', 'Cancel'], 298 | 'default': 'Rename', 299 | 'show': True 300 | }, 301 | 302 | 5: {'tag': 'non-resumable', 303 | 'description': 'Show "Non-resumable downloads warning".', 304 | 'body': ("Warning! \n" 305 | "This remote server doesn't support chunk downloading, \n" 306 | "if for any reason download stops resume won't be available and this file will be downloaded \n" 307 | "from the beginning, \n" 308 | 'Are you sure you want to continue??'), 309 | 'options': ['Yes', 'Cancel'], 310 | 'default': 'Yes', 311 | 'show': True 312 | }, 313 | 314 | 6: {'tag': 'ssl-warning', 315 | 'description': 'Show warning when Disabling SSL verification.', 316 | 'body': ('WARNING: disable SSL certificate verification could allow hackers to man-in-the-middle attack ' 317 | 'and make the communication insecure. \n\n' 318 | 'Are you sure?'), 319 | 'options': ['Yes', 'Cancel'], 320 | 'default': 'Yes', 321 | 'show': True 322 | }, 323 | 324 | 7: {'tag': 'delete-item', 325 | 'description': 'Confirm when deleting an item from download list.', 326 | 'body': 'Remove item(s) from the list?', 327 | 'options': ['Yes', 'Cancel'], 328 | 'default': 'Yes', 329 | 'show': True 330 | }, 331 | } 332 | 333 | for k in popups.keys(): 334 | var_name = f'popup_{k}' 335 | globals()[var_name] = True if k in (2, 4, 6, 7) else False 336 | settings_keys.append(var_name) 337 | 338 | 339 | def get_popup(k): 340 | item = popups[k] 341 | item['show'] = globals()[f'popup_{k}'] 342 | return item 343 | 344 | 345 | def enable_popup(k, value): 346 | globals()[f'popup_{k}'] = value # True or false 347 | 348 | # disable some popups 349 | -------------------------------------------------------------------------------- /vortexdm/dependency.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | """ 3 | Vortex Download Manager (VortexDM) 4 | 5 | A multi-connection internet download manager, based on "PycURL" and "youtube_dl". Original project, FireDM, by Mahmoud Elshahat. 6 | :copyright: (c) 2023 by Sixline 7 | :copyright: (c) 2019-2021 by Mahmoud Elshahat. 8 | :license: GNU GPLv3, see LICENSE.md for more details. 9 | """ 10 | 11 | # The purpose of this module is checking and auto installing dependencies 12 | import sys 13 | import subprocess 14 | import importlib.util 15 | 16 | # add the required packages here without any version numbers 17 | requirements = ['plyer', 'certifi', 'youtube_dl', 'yt_dlp', 'pycurl', 'PIL', 'pystray', 'awesometkinter'] 18 | 19 | 20 | def is_venv(): 21 | """check if running inside virtual environment 22 | there is no 100% working method to tell, but we can check for both real_prefix and base_prefix""" 23 | if hasattr(sys, 'real_prefix') or (hasattr(sys, 'base_prefix') and sys.base_prefix != sys.prefix): 24 | return True 25 | else: 26 | return False 27 | 28 | 29 | def install_missing_pkgs(): 30 | 31 | # list of dependency 32 | missing_pkgs = [pkg for pkg in requirements if importlib.util.find_spec(pkg) is None] 33 | 34 | if missing_pkgs: 35 | print('required pkgs: ', requirements) 36 | print('missing pkgs: ', missing_pkgs) 37 | 38 | for pkg in missing_pkgs: 39 | # because 'pillow' is installed under different name 'PIL' will use pillow with pip github issue #60 40 | if pkg == 'PIL': 41 | pkg = 'pillow' 42 | 43 | # using "--user" flag is safer also avoid the need for admin privilege , but it fails inside venv, where pip 44 | # will install packages normally to user folder but venv still can't see those packages 45 | 46 | if is_venv(): 47 | cmd = [sys.executable, "-m", "pip", "install", '--upgrade', pkg] # no --user flag 48 | else: 49 | cmd = [sys.executable, "-m", "pip", "install", '--user', '--upgrade', pkg] 50 | 51 | print('running command:', ' '.join(cmd)) 52 | subprocess.run(cmd, shell=False) 53 | 54 | 55 | 56 | 57 | -------------------------------------------------------------------------------- /vortexdm/downloaditem.py: -------------------------------------------------------------------------------- 1 | """ 2 | Vortex Download Manager (VortexDM) 3 | 4 | A multi-connection internet download manager, based on "PycURL" and "youtube_dl". Original project, FireDM, by Mahmoud Elshahat. 5 | :copyright: (c) 2023 by Sixline 6 | :copyright: (c) 2019-2021 by Mahmoud Elshahat. 7 | :license: GNU GPLv3, see LICENSE.md for more details. 8 | """ 9 | 10 | # Download Item Class 11 | 12 | import os 13 | import mimetypes 14 | import time 15 | from collections import deque 16 | from threading import Lock 17 | from urllib.parse import urljoin, unquote, urlparse 18 | 19 | from .utils import (validate_file_name, get_headers, translate_server_code, log, delete_file, delete_folder, save_json, 20 | load_json, get_range_list) 21 | from . import config 22 | from .config import MediaType 23 | 24 | 25 | class Segment: 26 | def __init__(self, name=None, num=None, range=None, size=0, url=None, tempfile=None, seg_type='', merge=True, 27 | media_type=MediaType.general, d=None): 28 | self.d = d # reference to parent download item 29 | self.name = name # full path file name 30 | # self.basename = os.path.basename(self.name) 31 | self.num = num 32 | self._range = range # a list of start and end bytes 33 | self.size = size 34 | # todo: change bool (downloaded, and completed) to (isdownloaded, and iscompleted), and down_bytes to downloaded 35 | self.downloaded = False 36 | self._down_bytes = 0 37 | self.completed = False # done downloading and merging into tempfile 38 | self.tempfile = tempfile 39 | self.headers = {} 40 | self.url = url 41 | self.seg_type = seg_type 42 | self.merge = merge 43 | self.key = None 44 | self.locked = False # set True by the worker which is currently downloading this segment 45 | self.media_type = media_type 46 | self.retries = 0 # number of download retries 47 | 48 | # override size if range available 49 | if range: 50 | self.size = range[1] - range[0] + 1 51 | 52 | @property 53 | def current_size(self): 54 | try: 55 | size = os.path.getsize(self.name) 56 | except: 57 | size = 0 58 | return size 59 | 60 | @property 61 | def down_bytes(self): 62 | return self._down_bytes if self._down_bytes > 0 else self.current_size 63 | 64 | @down_bytes.setter 65 | def down_bytes(self, value): 66 | self._down_bytes = value 67 | 68 | @property 69 | def remaining(self): 70 | return max(self.size - self.current_size, 0) 71 | 72 | @property 73 | def range(self): 74 | return self._range 75 | 76 | @range.setter 77 | def range(self, value): 78 | self._range = value 79 | if value: 80 | self.size = value[1] - value[0] + 1 81 | 82 | @property 83 | def basename(self): 84 | if self.name: 85 | return os.path.basename(self.name) 86 | else: 87 | return 'undefined' 88 | 89 | def get_size(self): 90 | http_headers = self.d.http_headers if self.d else None 91 | self.headers = get_headers(self.url, http_headers=http_headers) 92 | try: 93 | self.size = int(self.headers.get('content-length', 0)) 94 | print('Segment num:', self.num, 'getting size:', self.size) 95 | except: 96 | pass 97 | return self.size 98 | 99 | def __repr__(self): 100 | return repr(self.__dict__) 101 | 102 | 103 | class DownloadItem: 104 | """base class for download items""" 105 | 106 | def __init__(self, url='', name='', folder=''): 107 | """initialize 108 | 109 | Args: 110 | url (str): download link 111 | name (str): file name including extension e.g. myvideo.mp4 112 | folder (str): download folder 113 | """ 114 | # unique name for download item, will be calculated based on name and target folder 115 | self.uid = None 116 | 117 | self.title = '' # file name without extension 118 | self._name = name 119 | self.extension = '' # note: filename extension includes a dot, ex: '.mp4' 120 | 121 | self.folder = os.path.abspath(folder) 122 | self.alternative_temp_folder = config.temp_folder 123 | 124 | self.url = url 125 | self.eff_url = '' 126 | 127 | self.size = 0 128 | self._video_size = 0 129 | self._total_size = 0 130 | self.resumable = False 131 | 132 | # type and subtypes 133 | self.type = '' # general, video, audio 134 | self.subtype_list = [] # it might contains one or more eg "hls, dash, fragmented, normal" 135 | 136 | self._segment_size = config.SEGMENT_SIZE 137 | 138 | self.live_connections = 0 139 | self._downloaded = 0 140 | self._lock = None # Lock() to access downloaded property from different threads 141 | self._status = config.Status.cancelled 142 | 143 | self._remaining_parts = 0 144 | self.total_parts = 0 145 | 146 | # connection status 147 | self.status_code = 0 148 | self.status_code_description = '' 149 | 150 | # audio 151 | self.audio_stream = None 152 | self.audio_url = None 153 | self.audio_size = 0 154 | self.is_audio = False 155 | self.audio_quality = None 156 | 157 | # media files progress 158 | self.video_progress = 0 159 | self.audio_progress = 0 160 | self.merge_progress = 0 161 | 162 | # postprocessing callback is a string represent any function name need to be called after done downloading 163 | # this function must be available or imported in brain.py namespace 164 | self.callback = '' 165 | 166 | # schedule download 167 | self.sched = None 168 | 169 | # speed 170 | self._speed = 0 171 | self.prev_downloaded_value = 0 172 | self.speed_buffer = deque() # store some speed readings for calculating average speed afterwards 173 | self.speed_timer = 0 174 | self.speed_refresh_rate = 0.5 # calculate speed every n seconds 175 | 176 | # segments 177 | self.segments = [] 178 | 179 | # fragmented video parameters will be updated from video subclass object / update_param() 180 | self.fragment_base_url = None 181 | self.fragments = None 182 | 183 | # fragmented audio parameters will be updated from video subclass object / update_param() 184 | self.audio_fragment_base_url = None 185 | self.audio_fragments = None 186 | 187 | # protocol 188 | self.protocol = '' 189 | 190 | # format id, youtube-dl specific 191 | self.format_id = None 192 | self.audio_format_id = None 193 | 194 | # quality for video and audio 195 | self.abr = None 196 | self.tbr = None # for video equal Bandwidth/1000 197 | self.resolution = None # for videos only example for 720p: 1280x720 198 | 199 | # hls m3u8 manifest url 200 | self.manifest_url = '' 201 | 202 | # thumbnails 203 | self.thumbnail_url = None 204 | self.thumbnail = None # base64 string 205 | 206 | # playlist info 207 | self.playlist_url = '' 208 | self.playlist_title = '' 209 | 210 | # selected stream name for video objects 211 | self.selected_quality = None 212 | 213 | # subtitles 214 | # template: {language1:[sub1, sub2, ...], language2: [sub1, ...]}, where sub = {'url': 'xxx', 'ext': 'xxx'} 215 | self.subtitles = {} 216 | self.automatic_captions = {} 217 | self.selected_subtitles = {} # chosen subtitles that will be downloaded 218 | 219 | # accept html contents 220 | self.accept_html = False # if server sent html contents instead of bytes 221 | 222 | # errors 223 | self.errors = 0 # an indicator for server, network, or other errors while downloading 224 | 225 | # subprocess references 226 | self.subprocess = None 227 | 228 | # test 229 | self.seg_names = [] 230 | 231 | # http-headers 232 | self.http_headers = {} 233 | 234 | # metadata 235 | self.metadata_file_content = '' 236 | 237 | # shutdown computer after completing download 238 | self.shutdown_pc = False 239 | 240 | # custom command to run in terminal after completing download 241 | self.on_completion_command = '' 242 | 243 | self.segments_progress = [] 244 | self.segments_progress_bool = [] 245 | 246 | # properties names that will be saved on disk 247 | self.saved_properties = ['_name', 'folder', 'url', 'eff_url', 'playlist_url', 'playlist_title', 'size', 248 | 'resumable', 'selected_quality', '_segment_size', '_downloaded', '_status', 249 | '_remaining_parts', 'total_parts', 'audio_url', 'audio_size', 'type', 'subtype_list', 'fragments', 250 | 'fragment_base_url', 'audio_fragments', 'audio_fragment_base_url', 251 | '_total_size', 'protocol', 'manifest_url', 'selected_subtitles', 252 | 'abr', 'tbr', 'format_id', 'audio_format_id', 'resolution', 'audio_quality', 253 | 'http_headers', 'metadata_file_content', 'title', 'extension', 'sched', 'thumbnail_url'] 254 | 255 | # property to indicate a time consuming operation is running on download item now 256 | self.busy = False 257 | 258 | def __repr__(self): 259 | return f'DownloadItem object(name:{self.name}, url:{self.url})' 260 | 261 | @property 262 | def remaining_parts(self): 263 | return self._remaining_parts 264 | 265 | @remaining_parts.setter 266 | def remaining_parts(self, value): 267 | self._remaining_parts = value 268 | 269 | # should recalculate total size again with every completed segment, most of the time segment size won't be 270 | # available until actually downloaded this segment, "check worker.report_completed()" 271 | self.total_size = self.calculate_total_size() 272 | 273 | @property 274 | def video_size(self): 275 | return self._video_size or self.size 276 | 277 | @video_size.setter 278 | def video_size(self, value): 279 | self._video_size = value 280 | 281 | @property 282 | def total_size(self): 283 | # recalculate total size only if there is size change in segment size 284 | if not self._total_size: 285 | self._total_size = self.calculate_total_size() 286 | 287 | return self._total_size 288 | 289 | @total_size.setter 290 | def total_size(self, value): 291 | self._total_size = value 292 | 293 | @property 294 | def speed(self): 295 | """average speed""" 296 | if self.status != config.Status.downloading: # or not self.speed_buffer: 297 | self._speed = 0 298 | else: 299 | if not self.prev_downloaded_value: 300 | self.prev_downloaded_value = self.downloaded 301 | 302 | time_passed = time.time() - self.speed_timer 303 | if time_passed >= self.speed_refresh_rate: 304 | self.speed_timer = time.time() 305 | delta = self.downloaded - self.prev_downloaded_value 306 | self.prev_downloaded_value = self.downloaded 307 | _speed = delta / time_passed 308 | if _speed >= 0: 309 | # to get a stable speed reading will use an average of multiple speed readings 310 | self.speed_buffer.append(_speed) 311 | avg_speed = sum(self.speed_buffer) / len(self.speed_buffer) 312 | if len(self.speed_buffer) >= 10: 313 | self.speed_buffer.popleft() 314 | 315 | self._speed = int(avg_speed) if avg_speed > 0 else 0 316 | 317 | return self._speed 318 | 319 | @property 320 | def lock(self): 321 | # Lock() to access downloaded property from different threads 322 | if not self._lock: 323 | self._lock = Lock() 324 | return self._lock 325 | 326 | @property 327 | def downloaded(self): 328 | return self._downloaded 329 | 330 | @downloaded.setter 331 | def downloaded(self, value): 332 | """this property might be set from threads, expecting int (number of bytes)""" 333 | if not isinstance(value, int): 334 | return 335 | 336 | with self.lock: 337 | self._downloaded = value 338 | 339 | @property 340 | def progress(self): 341 | p = 0 342 | 343 | if self.status == config.Status.completed: 344 | p = 100 345 | 346 | elif self.total_size == 0 and self.segments: 347 | if len(self.segments) == 1: 348 | seg = self.segments[0] 349 | if seg.completed and seg.current_size == 0: 350 | p = 0 351 | else: 352 | # to handle fragmented files 353 | finished = len([seg for seg in self.segments if seg.completed]) 354 | p = round(finished * 100 / len(self.segments), 1) 355 | elif self.total_size: 356 | p = round(self.downloaded * 100 / self.total_size, 1) 357 | 358 | # make progress 99% if not completed 359 | if p >= 100: 360 | if not self.status == config.Status.completed: 361 | p = 99 362 | else: 363 | p = 100 364 | 365 | return p 366 | 367 | @property 368 | def eta(self): 369 | """estimated time remaining to finish download""" 370 | ret = '' 371 | try: 372 | if self.status == config.Status.downloading: 373 | ret = int((self.total_size - self.downloaded) / self.speed) 374 | except: 375 | pass 376 | 377 | return ret 378 | 379 | @property 380 | def status(self): 381 | return self._status 382 | 383 | @status.setter 384 | def status(self, value): 385 | self._status = value 386 | 387 | # kill subprocess if currently active 388 | if self.subprocess and value in (config.Status.cancelled, config.Status.error): 389 | self.kill_subprocess() 390 | 391 | @property 392 | def name(self): 393 | return self._name 394 | 395 | @name.setter 396 | def name(self, new_value): 397 | # validate new name 398 | self._name = validate_file_name(new_value) 399 | 400 | self.title, self.extension = os.path.splitext(self._name) 401 | 402 | @property 403 | def temp_folder(self): 404 | fp = self.alternative_temp_folder if os.path.isdir(self.alternative_temp_folder) else self.folder 405 | name = f'vortexdm_{self.uid}' 406 | return os.path.join(fp, name) 407 | 408 | @property 409 | def target_file(self): 410 | """return file name including path""" 411 | return os.path.join(self.folder, self.name) 412 | 413 | @property 414 | def temp_file(self): 415 | """return temp file name including path""" 416 | name = f'_temp_{self.name}'.replace(' ', '_') 417 | return os.path.join(self.temp_folder, name) 418 | 419 | @property 420 | def audio_file(self): 421 | """return temp file name including path""" 422 | name = f'audio_for_{self.name}'.replace(' ', '_') 423 | return os.path.join(self.temp_folder, name) 424 | 425 | @property 426 | def segment_size(self): 427 | return self._segment_size 428 | 429 | @segment_size.setter 430 | def segment_size(self, value): 431 | self._segment_size = value if value <= self.size else self.size 432 | # print('segment size = ', self._segment_size) 433 | 434 | @property 435 | def video_segments(self): 436 | return [seg for seg in self.segments if seg.media_type == MediaType.video] 437 | 438 | @property 439 | def audio_segments(self): 440 | return [seg for seg in self.segments if seg.media_type == MediaType.audio] 441 | 442 | def select_subs(self, subs_names=None): 443 | """ 444 | search subtitles names and build a dict of name:url for all selected subs 445 | :param subs_names: list of subs names 446 | :return: None 447 | """ 448 | if not isinstance(subs_names, list): 449 | return 450 | 451 | subs = {} 452 | # search for subs 453 | for k in subs_names: 454 | v = self.subtitles.get(k) or self.automatic_captions.get(k) 455 | if v: 456 | subs[k] = v 457 | 458 | self.selected_subtitles = subs 459 | 460 | # print('self.selected_subtitles:', self.selected_subtitles) 461 | 462 | def calculate_total_size(self): 463 | # this is heavy and should be used carefully, calculate size by getting every segment's size 464 | def guess_size(seglist): 465 | known_sizes = [seg.size for seg in seglist if seg.size] 466 | unknown_sizes_num = len(seglist) - len(known_sizes) 467 | tsize = sum(known_sizes) 468 | 469 | # guess remaining segments' sizes 470 | if known_sizes and unknown_sizes_num: 471 | avg_seg_size = sum(known_sizes) // len(known_sizes) 472 | tsize += avg_seg_size * unknown_sizes_num 473 | return tsize 474 | 475 | total_size = 0 476 | 477 | if self.segments: 478 | 479 | # get video segments 480 | video_segs = [seg for seg in self.segments if seg.media_type == MediaType.video] 481 | 482 | # get audio segments 483 | audio_segs = [seg for seg in self.segments if seg.media_type == MediaType.audio] 484 | 485 | # get other segments if any 486 | other_segs = [seg for seg in self.segments if seg not in (*video_segs, *audio_segs)] 487 | 488 | # calculate sizes 489 | video_size = guess_size(video_segs) 490 | audio_size = guess_size(audio_segs) 491 | othres_size = guess_size(other_segs) 492 | 493 | self.video_size = video_size 494 | self.audio_size = audio_size 495 | total_size = video_size + audio_size + othres_size 496 | 497 | self.total_parts = len(self.segments) 498 | 499 | total_size = total_size or self.size 500 | 501 | return total_size 502 | 503 | def kill_subprocess(self): 504 | """it will kill any subprocess running for this download item, ex: ffmpeg merge video/audio""" 505 | try: 506 | # to work subprocess should have shell=False 507 | self.subprocess.kill() 508 | log('run_command()> cancelled', self.subprocess.args) 509 | self.subprocess = None 510 | except Exception as e: 511 | log('DownloadItem.kill_subprocess()> error', e) 512 | 513 | def update(self, url): 514 | """get headers and update properties (eff_url, name, ext, size, type, resumable, status code/description)""" 515 | # log('*'*20, 'update download item') 516 | 517 | if url in ('', None): 518 | return 519 | 520 | headers = get_headers(url, http_headers=self.http_headers) 521 | # print('update d parameters:', headers) 522 | 523 | # update headers 524 | # print('update()> url, self.url', url, self.url) 525 | self.url = url 526 | self.eff_url = headers.get('eff_url') 527 | self.status_code = headers.get('status_code', '') 528 | self.status_code_description = f"{self.status_code} - {translate_server_code(self.status_code)}" 529 | 530 | # get file name 531 | name = '' 532 | if 'content-disposition' in headers: 533 | # example 'content-disposition': 'attachment; filename="proxmox-ve_6.3-1.iso"; modification-date="wed, 25 nov 2020 16:51:19 +0100"; size=852299776;' 534 | # when both "filename" and "filename*" are present in a single header field value, "filename*" should be used 535 | # more info at https://tools.ietf.org/html/rfc6266 536 | 537 | try: 538 | content = headers['content-disposition'].split(';') 539 | match = [x for x in content if 'filename*' in x.lower()] 540 | if not match: 541 | match = [x for x in content if 'filename' in x.lower()] 542 | 543 | name = match[0].split('=')[1].strip('"') 544 | except: 545 | pass 546 | 547 | if not name: 548 | name = headers.get('file-name', '') 549 | 550 | if not name: 551 | # extract name from url 552 | basename = urlparse(url).path 553 | name = basename.strip('/').split('/')[-1] 554 | 555 | # decode percent-encoded strings, example 'silver%20bullet' >> 'silver bullet' 556 | name = unquote(name) 557 | 558 | # file size 559 | size = int(headers.get('content-length', 0)) 560 | 561 | # type 562 | content_type = headers.get('content-type', '').split(';')[0] 563 | 564 | # file extension: 565 | ext = os.path.splitext(name)[1] 566 | if not ext: # if no ext in file name 567 | ext = mimetypes.guess_extension(content_type, strict=False) if content_type not in ('N/A', None) else '' 568 | 569 | if ext: 570 | name += ext 571 | 572 | self.name = name 573 | self.extension = ext 574 | self.size = size 575 | self.type = content_type 576 | self.resumable = self.is_resumable(url, headers) 577 | 578 | # build segments 579 | self.build_segments() 580 | 581 | log('headers:', headers, log_level=3) 582 | 583 | def is_resumable(self, url, headers): 584 | # check resume support / chunk downloading 585 | resumable = headers.get('accept-ranges', 'none') != 'none' 586 | size = int(headers.get('content-length', 0)) 587 | 588 | if not resumable and size > config.SEGMENT_SIZE: 589 | # 'status_code': 206, 'content-length': '401', 'content-range': 'bytes 100-500/40772008' 590 | seg_range = [100, 500] # test range 401 bytes 591 | h = get_headers(url, seg_range=seg_range, http_headers=self.http_headers) 592 | 593 | if h.get('status_code') == 206 and int(h.get('content-length', 0)) == 401: 594 | resumable = True 595 | 596 | return resumable 597 | 598 | def delete_tempfiles(self, force_delete=False): 599 | """delete temp files and folder for a given download item""" 600 | 601 | if force_delete or not config.keep_temp: 602 | delete_folder(self.temp_folder) 603 | delete_file(self.temp_file) 604 | delete_file(self.audio_file) 605 | 606 | def build_segments(self): 607 | # log('-'*20, 'build segments') 608 | # don't handle hls videos 609 | if 'hls' in self.subtype_list: 610 | return 611 | 612 | # handle fragmented video 613 | if self.fragments: 614 | # print(self.fragments) 615 | # example 'fragments': [{'path': 'range/0-640'}, {'path': 'range/2197-63702', 'duration': 9.985},] 616 | _segments = [Segment(name=os.path.join(self.temp_folder, str(i)), num=i, range=None, size=0, 617 | url=urljoin(self.fragment_base_url, x.get('path', '')), tempfile=self.temp_file, 618 | media_type=MediaType.video) 619 | for i, x in enumerate(self.fragments)] 620 | 621 | else: 622 | # general files or video files with known sizes and resumable 623 | if self.resumable and self.size: 624 | # get list of ranges i.e. [[0, 100], [101, 2000], ... ] 625 | range_list = get_range_list(self.size, config.SEGMENT_SIZE) 626 | else: 627 | range_list = [None] # add None in a list to make one segment with range=None 628 | 629 | _segments = [ 630 | Segment(name=os.path.join(self.temp_folder, str(i)), num=i, range=x, 631 | url=self.eff_url, tempfile=self.temp_file, media_type=self.type) 632 | for i, x in enumerate(range_list)] 633 | 634 | # get an audio stream to be merged with dash video 635 | if 'dash' in self.subtype_list: 636 | # handle fragmented audio 637 | if self.audio_fragments: 638 | # example 'fragments': [{'path': 'range/0-640'}, {'path': 'range/2197-63702', 'duration': 9.985},] 639 | audio_segments = [ 640 | Segment(name=os.path.join(self.temp_folder, str(i) + '_audio'), num=i, range=None, size=0, 641 | url=urljoin(self.audio_fragment_base_url, x.get('path', '')), tempfile=self.audio_file, 642 | media_type=MediaType.audio) 643 | for i, x in enumerate(self.audio_fragments)] 644 | 645 | else: 646 | range_list = get_range_list(self.audio_size, config.SEGMENT_SIZE) 647 | 648 | audio_segments = [ 649 | Segment(name=os.path.join(self.temp_folder, str(i) + '_audio'), num=i, range=x, 650 | url=self.audio_url, tempfile=self.audio_file, media_type=MediaType.audio) 651 | for i, x in enumerate(range_list)] 652 | 653 | # append to main list 654 | _segments += audio_segments 655 | 656 | seg_names = [f'{seg.basename}:{seg.range}' for seg in _segments] 657 | log(f'Segments-{self.name}, ({len(seg_names)}):', seg_names, log_level=3) 658 | 659 | # pass additional parametrs 660 | for seg in _segments: 661 | seg.d = self 662 | 663 | self.segments = _segments 664 | 665 | def save_progress_info(self): 666 | """save segments info to disk""" 667 | progress_info = [{'name': seg.name, 'downloaded': seg.downloaded, 'completed': seg.completed, 'size': seg.size, 668 | '_range': seg.range, 'media_type': seg.media_type} 669 | for seg in self.segments] 670 | file = os.path.join(self.temp_folder, 'progress_info.txt') 671 | save_json(file, progress_info) 672 | 673 | def load_progress_info(self): 674 | """ 675 | load progress info from disk, update segments' info, verify actual segments' size on disk 676 | :return: None 677 | """ 678 | 679 | # check if file already exist 680 | if self.status != config.Status.completed and os.path.isfile(self.target_file): 681 | log('file already exist .............', self.target_file) 682 | # report completed 683 | self.status = config.Status.completed 684 | self.size = os.path.getsize(self.target_file) 685 | self.downloaded = self.size 686 | self.delete_tempfiles() 687 | 688 | # log('load_progress_info()> Loading progress info') 689 | progress_info = [] 690 | 691 | # load progress info from temp folder if exist 692 | progress_fp = os.path.join(self.temp_folder, 'progress_info.txt') 693 | if os.path.isfile(progress_fp): 694 | data = load_json(progress_fp) 695 | if isinstance(data, list): 696 | progress_info = data 697 | 698 | # # delete any segment which is not in progress_info file 699 | # if os.path.isdir(self.temp_folder): 700 | # fps = [item.get('name', '') for item in progress_info] + [progress_fp] 701 | # for f_name in os.listdir(self.temp_folder): 702 | # fp = os.path.join(self.temp_folder, f_name) 703 | # if fp not in fps: 704 | # delete_file(fp, verbose=True) 705 | 706 | # update segments from progress info 707 | if progress_info: 708 | downloaded = 0 709 | # log('load_progress_info()> Found previous download on the disk') 710 | 711 | # verify segments on disk 712 | for item in progress_info: 713 | # reset flags 714 | item['downloaded'] = False 715 | item['completed'] = False 716 | 717 | try: 718 | size_on_disk = os.path.getsize(item.get('name')) 719 | downloaded += size_on_disk 720 | if size_on_disk > 0 and size_on_disk == item.get('size'): 721 | item['downloaded'] = True 722 | except: 723 | continue 724 | 725 | # for dynamic made segments will build new segments from progress info 726 | if self.size and self.resumable and not self.fragments and 'hls' not in self.subtype_list: 727 | self.segments.clear() 728 | for i, item in enumerate(progress_info): 729 | try: 730 | seg = Segment() 731 | seg.__dict__.update(item) 732 | 733 | # update tempfile and url 734 | if seg.media_type == MediaType.audio: 735 | seg.tempfile = self.audio_file 736 | seg.url = self.audio_url 737 | else: 738 | seg.tempfile = self.temp_file 739 | seg.url = self.eff_url 740 | 741 | self.segments.append(seg) 742 | except: 743 | pass 744 | log('load_progress_info()> rebuild segments from previous download for:', self.name) 745 | 746 | # for fixed segments will update segments list only 747 | elif self.segments: 748 | for seg, item in zip(self.segments, progress_info): 749 | if seg.name == item.get('name'): 750 | seg.__dict__.update(item) 751 | log('load_progress_info()> updated current segments for:', self.name) 752 | 753 | # update self.downloaded 754 | self.downloaded = downloaded 755 | 756 | # update media files progress 757 | self.update_media_files_progress() 758 | 759 | def update_media_files_progress(self): 760 | """get the percentage of media files completion, e.g. temp video file, audio file, and final target file 761 | 762 | """ 763 | 764 | if self.status == config.Status.completed: 765 | self.video_progress = 100 766 | self.audio_progress = 100 767 | self.merge_progress = 100 768 | return 769 | 770 | def _get_progress(fp, full_size): 771 | try: 772 | current_size = os.path.getsize(fp) 773 | except: 774 | current_size = 0 775 | 776 | if current_size == 0 or full_size == 0: 777 | progress = 0 778 | else: 779 | progress = round(current_size * 100 / full_size, 2) 780 | 781 | if progress > 100: 782 | progress = 100 783 | 784 | return progress 785 | 786 | self.video_progress = _get_progress(self.temp_file, self.video_size) 787 | 788 | if 'normal' in self.subtype_list: 789 | self.audio_progress = self.video_progress 790 | else: 791 | self.audio_progress = _get_progress(self.audio_file, self.audio_size) 792 | 793 | self.merge_progress = _get_progress(self.target_file, self.total_size) 794 | 795 | def update_segments_progress(self, activeonly=False): 796 | """set self.segments_progress, e.g [total size, [(starting range, length, total file size), ...]]""" 797 | segments_progress = None 798 | 799 | if self.status == config.Status.completed: 800 | segments_progress = [100, [(0, 100)]] 801 | 802 | else: 803 | spb = self.segments_progress_bool 804 | c = (lambda seg: seg not in spb) if activeonly else (lambda seg: True) # condition 805 | total_size = self.total_size 806 | 807 | try: 808 | # handle hls or fragmented videos 809 | if any([x in self.subtype_list for x in ('hls', 'fragmented')]): 810 | # one hls file might contain more than 5000 segment with unknown sizes 811 | # will use segments numbers as a starting point and segment size = 1 812 | 813 | total_size = len(self.segments) 814 | sp = [(self.segments.index(seg), 1) for seg in self.segments if seg.downloaded and c(seg)] 815 | 816 | # handle other video types 817 | elif self.type == MediaType.video: 818 | 819 | vid = [(seg.range[0], seg.down_bytes) for seg in self.video_segments if c(seg)] 820 | aud = [(seg.range[0] + self.video_size - 1, seg.down_bytes) for seg in self.audio_segments if c(seg)] 821 | 822 | sp = vid + aud 823 | 824 | # handle non video items 825 | else: 826 | sp = [(seg.range[0], seg.down_bytes) for seg in self.segments if c(seg)] 827 | 828 | sp = [item for item in sp if item[1]] 829 | spb.extend([seg for seg in self.segments if seg.downloaded]) 830 | segments_progress = [total_size, sp] 831 | 832 | except Exception as e: 833 | if config.test_mode: 834 | log('update_segments_progress()>', e) 835 | 836 | self.segments_progress = segments_progress 837 | return segments_progress 838 | -------------------------------------------------------------------------------- /vortexdm/model.py: -------------------------------------------------------------------------------- 1 | """ 2 | Vortex Download Manager (VortexDM) 3 | 4 | A multi-connection internet download manager, based on "PycURL" and "youtube_dl". Original project, FireDM, by Mahmoud Elshahat. 5 | :copyright: (c) 2023 by Sixline 6 | :copyright: (c) 2019-2021 by Mahmoud Elshahat. 7 | :license: GNU GPLv3, see LICENSE.md for more details. 8 | 9 | Module description: 10 | this module contains an observables data models 11 | """ 12 | import os 13 | 14 | from .downloaditem import DownloadItem 15 | from .video import Video 16 | from . import utils 17 | 18 | 19 | class Observable: 20 | """super class for observable download item / Video""" 21 | def __init__(self, observer_callbacks=None): 22 | """initialize 23 | 24 | Args: 25 | observer_callbacks (iterable): list or tuple of callbacks that will be called when any property in 26 | watch_list changes 27 | """ 28 | 29 | self.watch_list = ['uid', 'name', 'progress', 'speed', 'eta', 'downloaded', 'size', '_total_size', 30 | 'total_size', 'status', 'busy', 'thumbnail', 'type', 'subtype_list', 'resumable', 'title', 31 | 'extension', 'errors', 'sched', 'remaining_parts', 'live_connections', 'total_parts', 32 | 'shutdown_pc', 'on_completion_command', 'video_progress', 'audio_progress', 'merge_progress', 33 | 'segments_progress', 'duration', 'duration_string'] 34 | 35 | # list of callbacks to be executed on properties change 36 | self.observer_callbacks = observer_callbacks or [] 37 | 38 | # unique name for download item, will be calculated based on name and target folder 39 | self.uid = None 40 | 41 | def setter(self, super_class, key, value): 42 | """intended to be used in subclass' __setattr__ 43 | 44 | """ 45 | 46 | try: 47 | old_value = super_class.__getattribute__(self, key) 48 | except: 49 | old_value = None 50 | 51 | # normalize folder path 52 | if key == 'folder': 53 | value = os.path.normpath(value) 54 | 55 | # set new value 56 | super_class.__setattr__(self, key, value) 57 | # self.__dict__[key] = value # don't use this because it doesn't work well with decorated properties 58 | 59 | if value != old_value: 60 | # calculate uid if name or folder changed 61 | if key in ('name', 'folder'): 62 | self.calculate_uid() 63 | 64 | self.notify(key, value) 65 | 66 | def notify(self, key, value): 67 | """notify observer callbacks with every property change, 68 | uid will be sent with every property""" 69 | if key in self.watch_list: 70 | # include 'uid' with every property change, 71 | # in same time we should avoid repeated 'uid' key in the same dictionary, 72 | # note "repeated dictionary keys won't raise an error any way" 73 | # see https://bugs.python.org/issue16385 74 | buffer = {'uid': self.uid, key: value} if key != 'uid' else {key: value} 75 | self._notify(**buffer) 76 | 77 | def _notify(self, **kwargs): 78 | """execute registered callbacks""" 79 | 80 | try: 81 | for callback in self.observer_callbacks: 82 | callback(**kwargs) 83 | except: 84 | raise 85 | 86 | def register_callback(self, callback): 87 | if callback not in self.observer_callbacks: 88 | self.observer_callbacks.append(callback) 89 | 90 | def unregister_callback(self, callback): 91 | if callback in self.observer_callbacks: 92 | self.observer_callbacks.remove(callback) 93 | 94 | def calculate_uid(self): 95 | """calculate or update uid based on full file path""" 96 | self.uid = utils.generate_unique_name(self.folder, self.name, prefix='uid_') 97 | 98 | def add_to_saved_properties(self, key): 99 | """append to the list in original DownloadItem class, to be saved to disk""" 100 | self.saved_properties.append(key) 101 | 102 | 103 | class ObservableDownloadItem(DownloadItem, Observable): 104 | """Observable DownloadItem data model""" 105 | 106 | def __init__(self, observer_callbacks=None, **kwargs): 107 | Observable.__init__(self, observer_callbacks=observer_callbacks) 108 | DownloadItem.__init__(self, **kwargs) 109 | 110 | def __setattr__(self, key, value): 111 | """Called when an attribute assignment is attempted.""" 112 | self.setter(DownloadItem, key, value) 113 | 114 | 115 | class ObservableVideo(Video, Observable): 116 | """Observable Video data model""" 117 | 118 | def __init__(self, url, vid_info=None, observer_callbacks=None): 119 | Observable.__init__(self, observer_callbacks=observer_callbacks) 120 | Video.__init__(self, url, vid_info=vid_info) 121 | 122 | def __setattr__(self, key, value): 123 | """Called when an attribute assignment is attempted.""" 124 | self.setter(Video, key, value) 125 | 126 | def prepare_subtitles(self): 127 | """merge subtitles and captions in one list and handle duplicated names 128 | 129 | # subtitles stored in download item in a dictionary format 130 | # template: subtitles = {language1:[sub1, sub2, ...], language2: [sub1, ...]}, where sub = {'url': 'xxx', 'ext': 'xxx'} 131 | # Example: {'en': [{'url': 'http://x.com/s1', 'ext': 'srv1'}, {'url': 'http://x.com/s2', 'ext': 'vtt'}], 'ar': [{'url': 'https://www.youtub}, {},...] 132 | 133 | Returns: 134 | (dict): one dict contains all subtitles 135 | """ 136 | # build subtitles from self.d.subtitles and self.d.automatic_captions, and rename repeated keys 137 | all_subtitles = {} 138 | for k, v in self.subtitles.items(): 139 | if k in all_subtitles: 140 | k = k + '_2' 141 | k = k + '_sub' 142 | all_subtitles[k] = v 143 | 144 | for k, v in self.automatic_captions.items(): 145 | if k in all_subtitles: 146 | k = k + '_2' 147 | k = k + '_caption' 148 | all_subtitles[k] = v 149 | 150 | # sort subtitles 151 | sorted_keys = utils.natural_sort(all_subtitles.keys()) 152 | all_subtitles = {k: all_subtitles[k] for k in sorted_keys} 153 | 154 | # add 'srt' extension 155 | for lang, ext_list in all_subtitles.items(): 156 | 157 | for item in ext_list: # item example: [{'url': 'http://x.com/s1', 'ext': 'srv1'}, {'url': 'http://x.com/s2', 'ext': 'vtt'}] 158 | item.setdefault('ext', 'txt') 159 | 160 | extensions = [item.get('ext') for item in ext_list] 161 | 162 | # add 'srt' extension if 'vtt' available 163 | if 'vtt' in extensions and 'srt' not in extensions: 164 | vtt_item = [item for item in ext_list if item.get('ext') == 'vtt'][-1] 165 | srt_item = vtt_item.copy() 166 | srt_item['ext'] = 'srt' 167 | ext_list.insert(0, srt_item) 168 | 169 | return all_subtitles 170 | 171 | -------------------------------------------------------------------------------- /vortexdm/setting.py: -------------------------------------------------------------------------------- 1 | """ 2 | Vortex Download Manager (VortexDM) 3 | 4 | A multi-connection internet download manager, based on "PycURL" and "youtube_dl". Original project, FireDM, by Mahmoud Elshahat. 5 | :copyright: (c) 2023 by Sixline 6 | :copyright: (c) 2019-2021 by Mahmoud Elshahat. 7 | :license: GNU GPLv3, see LICENSE.md for more details. 8 | """ 9 | 10 | import os 11 | import json 12 | import shutil 13 | 14 | from . import config 15 | from . import downloaditem 16 | from . import model 17 | from .utils import log, update_object 18 | 19 | 20 | def get_global_sett_folder(): 21 | """return a proper global setting folder""" 22 | home_folder = os.path.expanduser('~') 23 | 24 | if config.operating_system == 'Windows': 25 | roaming = os.getenv('APPDATA') # return APPDATA\Roaming\ under windows 26 | _sett_folder = os.path.join(roaming, f'.{config.APP_NAME}') 27 | 28 | elif config.operating_system == 'Linux': 29 | _sett_folder = f'{home_folder}/.config/{config.APP_NAME}/' 30 | 31 | elif config.operating_system == 'Darwin': 32 | _sett_folder = f'{home_folder}/Library/Application Support/{config.APP_NAME}/' 33 | 34 | else: 35 | _sett_folder = config.current_directory 36 | 37 | return _sett_folder 38 | 39 | 40 | config.global_sett_folder = get_global_sett_folder() 41 | 42 | 43 | def locate_setting_folder(): 44 | """check local folder and global setting folder for setting.cfg file""" 45 | 46 | # look for previous setting file 47 | if os.path.isfile(os.path.join(config.current_directory, 'setting.cfg')): 48 | setting_folder = config.current_directory 49 | elif os.path.isfile(os.path.join(config.global_sett_folder, 'setting.cfg')): 50 | setting_folder = config.global_sett_folder 51 | else: 52 | # no setting file found will check local folder for writing permission, otherwise will return global sett folder 53 | try: 54 | folder = config.current_directory 55 | with open(os.path.join(folder, 'test'), 'w') as test_file: 56 | test_file.write('0') 57 | os.unlink(os.path.join(folder, 'test')) 58 | setting_folder = config.current_directory 59 | 60 | except (PermissionError, OSError): 61 | log("No enough permission to store setting at local folder:", folder) 62 | log('Global setting folder will be selected:', config.global_sett_folder) 63 | 64 | # create global setting folder if it doesn't exist 65 | if not os.path.isdir(config.global_sett_folder): 66 | os.mkdir(config.global_sett_folder) 67 | 68 | setting_folder = config.global_sett_folder 69 | 70 | return setting_folder 71 | 72 | 73 | config.sett_folder = locate_setting_folder() 74 | 75 | 76 | def load_d_map(): 77 | """create and return a dictionary of 'uid: DownloadItem objects' based on data extracted from 'downloads.dat' file 78 | 79 | """ 80 | d_map = {} 81 | 82 | try: 83 | 84 | log('Load previous download items from', config.sett_folder) 85 | 86 | # get data 87 | file = os.path.join(config.sett_folder, 'downloads.dat') 88 | with open(file, 'r') as f: 89 | # expecting a list of dictionaries 90 | data = json.load(f) 91 | 92 | # converting data to a map of uid: ObservableDownloadItem() objects 93 | for uid, d_dict in data.items(): # {'uid': d_dict, 'uid2': d_dict2, ...} 94 | d = update_object(model.ObservableDownloadItem(), d_dict) 95 | if d: # if update_object() returned an updated object not None 96 | d.uid = uid 97 | d_map[uid] = d 98 | 99 | # get thumbnails 100 | file = os.path.join(config.sett_folder, 'thumbnails.dat') 101 | with open(file, 'r') as f: 102 | # expecting a list of dictionaries 103 | thumbnails = json.load(f) 104 | 105 | # clean d_map and load thumbnails 106 | for d in d_map.values(): 107 | d.live_connections = 0 108 | 109 | # for compatibility, after change status value, will correct old stored status values 110 | valid_values = [x for x in vars(config.Status).values() if isinstance(x, str)] 111 | for x in valid_values: 112 | if d.status.lower() == x.lower(): 113 | d.status = x 114 | 115 | if d.status not in (config.Status.completed, config.Status.scheduled, config.Status.error): 116 | d.status = config.Status.cancelled 117 | 118 | # use encode() to convert base64 string to byte, however it does work without it, will keep it to be safe 119 | d.thumbnail = thumbnails.get(d.uid, '').encode() 120 | 121 | # update progress info 122 | d.load_progress_info() 123 | 124 | except Exception as e: 125 | log(f'load_d_map()>: {e}') 126 | raise e 127 | finally: 128 | if not isinstance(d_map, dict): 129 | d_map = {} 130 | return d_map 131 | 132 | 133 | def save_d_map(d_map): 134 | try: 135 | data = {} # dictionary, key=d.uid, value=ObservableDownloadItem 136 | thumbnails = {} # dictionary, key=d.uid, value=base64 binary string for thumbnail 137 | for uid, d in d_map.items(): 138 | d_dict = {key: d.__dict__.get(key) for key in d.saved_properties} 139 | data[uid] = d_dict 140 | 141 | # thumbnails 142 | if d.thumbnail: 143 | # convert base64 byte to string is required because json can't handle byte objects 144 | thumbnails[d.uid] = d.thumbnail.decode("utf-8") 145 | 146 | # store d_map in downloads.cfg file 147 | downloads_fp = os.path.join(config.sett_folder, 'downloads.dat') 148 | with open(downloads_fp, 'w') as f: 149 | try: 150 | json.dump(data, f, indent=4) 151 | except Exception as e: 152 | print('error save d_list:', e) 153 | 154 | # store thumbnails in thumbnails.cfg file 155 | thumbnails_fp = os.path.join(config.sett_folder, 'thumbnails.dat') 156 | with open(thumbnails_fp, 'w') as f: 157 | try: 158 | json.dump(thumbnails, f) 159 | except Exception as e: 160 | print('error save thumbnails file:', e) 161 | 162 | log('downloads items list saved in:', downloads_fp, log_level=2) 163 | except Exception as e: 164 | log('save_d_map()> ', e) 165 | 166 | 167 | def get_user_settings(): 168 | settings = {} 169 | try: 170 | # log('Load user setting from', config.sett_folder) 171 | file = os.path.join(config.sett_folder, 'setting.cfg') 172 | with open(file, 'r') as f: 173 | settings = json.load(f) 174 | 175 | except FileNotFoundError: 176 | log('setting.cfg not found') 177 | except Exception as e: 178 | log('load_setting()> ', e) 179 | finally: 180 | if not isinstance(settings, dict): 181 | settings = {} 182 | 183 | return settings 184 | 185 | 186 | def load_setting(): 187 | 188 | # log('Load Application setting from', config.sett_folder) 189 | settings = get_user_settings() 190 | 191 | # update config module 192 | config.__dict__.update(settings) 193 | 194 | 195 | def save_setting(): 196 | # web authentication 197 | if not config.remember_web_auth: 198 | config.username = '' 199 | config.password = '' 200 | 201 | settings = {key: config.__dict__.get(key) for key in config.settings_keys} 202 | 203 | try: 204 | file = os.path.join(config.sett_folder, 'setting.cfg') 205 | with open(file, 'w') as f: 206 | json.dump(settings, f, indent=4) 207 | log('settings saved in:', file) 208 | except Exception as e: 209 | log('save_setting() > error', e) 210 | 211 | 212 | 213 | 214 | -------------------------------------------------------------------------------- /vortexdm/systray.py: -------------------------------------------------------------------------------- 1 | """ 2 | Vortex Download Manager (VortexDM) 3 | 4 | A multi-connection internet download manager, based on "PycURL" and "youtube_dl". Original project, FireDM, by Mahmoud Elshahat. 5 | :copyright: (c) 2023 by Sixline 6 | :copyright: (c) 2019-2021 by Mahmoud Elshahat. 7 | :license: GNU GPLv3, see LICENSE.md for more details. 8 | 9 | Module description: 10 | system tray icon based on GTK and pystray, tested on windows and Manjaro using GTK 3.0 11 | """ 12 | 13 | import os 14 | import awesometkinter as atk 15 | 16 | from . import config 17 | from .iconsbase64 import APP_ICON 18 | from .utils import log, delete_file 19 | 20 | 21 | class SysTray: 22 | """ 23 | systray icon using pystray package 24 | """ 25 | def __init__(self, main_window): 26 | self.main_window = main_window 27 | self.tray_icon_path = os.path.join(config.sett_folder, 'systray.png') # path to icon 28 | self.icon = None 29 | self._hover_text = None 30 | self.Gtk = None 31 | self.active = False 32 | 33 | def show_main_window(self, *args): 34 | # unhide and bring on top 35 | self.main_window.focus() 36 | 37 | def minimize_to_systray(self, *args): 38 | self.main_window.hide() 39 | 40 | @property 41 | def tray_icon(self): 42 | """return pillow image""" 43 | try: 44 | img = atk.create_pil_image(b64=APP_ICON, size=48) 45 | 46 | return img 47 | except Exception as e: 48 | log('systray: tray_icon', e) 49 | if config.test_mode: 50 | raise e 51 | 52 | def run(self): 53 | # not supported on mac 54 | if config.operating_system == 'Darwin': 55 | log('Systray is not supported on mac yet') 56 | return 57 | 58 | options_map = {'Show': self.show_main_window, 59 | 'Minimize to Systray': self.minimize_to_systray, 60 | 'Quit': self.quit} 61 | 62 | # make our own Gtk statusIcon, since pystray failed to run icon properly on Gtk 3.0 from a thread 63 | if config.operating_system == 'Linux': 64 | try: 65 | import gi 66 | gi.require_version('Gtk', '3.0') 67 | from gi.repository import Gtk 68 | self.Gtk = Gtk 69 | 70 | # delete previous icon file (it might contains an icon file for old VortexDM versions) 71 | delete_file(self.tray_icon_path) 72 | 73 | # save file to settings folder 74 | self.tray_icon.save(self.tray_icon_path, format='png') 75 | 76 | # creating menu 77 | menu = Gtk.Menu() 78 | for option, callback in options_map.items(): 79 | item = Gtk.MenuItem(label=option) 80 | item.connect('activate', callback) 81 | menu.append(item) 82 | menu.show_all() 83 | 84 | APPINDICATOR_ID = config.APP_NAME 85 | 86 | # setup notify system, will be used in self.notify() 87 | gi.require_version('Notify', '0.7') 88 | from gi.repository import Notify as notify 89 | self.Gtk_notify = notify # get reference for later deinitialize when quit systray 90 | self.Gtk_notify.init(APPINDICATOR_ID) # initialize first 91 | 92 | # try appindicator 93 | try: 94 | gi.require_version('AppIndicator3', '0.1') 95 | from gi.repository import AppIndicator3 as appindicator 96 | 97 | indicator = appindicator.Indicator.new(APPINDICATOR_ID, self.tray_icon_path, appindicator.IndicatorCategory.APPLICATION_STATUS) 98 | indicator.set_status(appindicator.IndicatorStatus.ACTIVE) 99 | indicator.set_menu(menu) 100 | 101 | # use .set_name to prevent error, Gdk-CRITICAL **: gdk_window_thaw_toplevel_updates: assertion 'window->update_and_descendants_freeze_count > 0' failed 102 | indicator.set_name = APPINDICATOR_ID 103 | 104 | # can set label beside systray icon 105 | # indicator.set_label('1.2 MB/s', '') 106 | 107 | self.active = True 108 | log('Systray active backend: Gtk.AppIndicator') 109 | Gtk.main() 110 | return 111 | except: 112 | pass 113 | 114 | # try GTK StatusIcon 115 | def icon_right_click(icon, button, time): 116 | menu.popup(None, None, None, icon, button, time) 117 | 118 | icon = Gtk.StatusIcon() 119 | icon.set_from_file(self.tray_icon_path) # DeprecationWarning: Gtk.StatusIcon.set_from_file is deprecated 120 | icon.connect("popup-menu", icon_right_click) # right click 121 | icon.connect('activate', self.show_main_window) # left click 122 | icon.set_name = APPINDICATOR_ID 123 | 124 | self.active = True 125 | log('Systray active backend: Gtk.StatusIcon') 126 | Gtk.main() 127 | return 128 | except Exception as e: 129 | log('Systray Gtk 3.0:', e, log_level=2) 130 | self.active = False 131 | else: 132 | # let pystray run for other platforms, basically windows 133 | try: 134 | from pystray import Icon, Menu, MenuItem 135 | items = [] 136 | for option, callback in options_map.items(): 137 | items.append(MenuItem(option, callback, default=True if option == 'Show' else False)) 138 | 139 | menu = Menu(*items) 140 | self.icon = Icon(config.APP_NAME, self.tray_icon, menu=menu) 141 | self.active = True 142 | self.icon.run() 143 | except Exception as e: 144 | log('systray: - run() - ', e) 145 | self.active = False 146 | 147 | def shutdown(self): 148 | try: 149 | self.active = False 150 | self.icon.stop() # must be called from main thread 151 | except: 152 | pass 153 | 154 | try: 155 | # if we use Gtk notify we should deinitialize 156 | self.Gtk_notify.uninit() 157 | except: 158 | pass 159 | 160 | try: 161 | # Gtk.main_quit(), if called from a thread might raise 162 | # (Gtk-CRITICAL **:gtk_main_quit: assertion 'main_loops != NULL' failed) 163 | # should call this from main thread 164 | self.Gtk.main_quit() 165 | except: 166 | pass 167 | 168 | def notify(self, msg, title=None): 169 | """show os notifications, e.g. balloon pop up at systray icon on windows""" 170 | if getattr(self.icon, 'HAS_NOTIFICATION', False): 171 | self.icon.notify(msg, title=title) 172 | 173 | else: 174 | try: 175 | self.Gtk_notify.Notification.new(title, msg, self.tray_icon_path).show() # to show notification 176 | except: 177 | try: 178 | # fallback to plyer 179 | import plyer 180 | plyer.notification.notify(message=msg, title=title, app_name=config.APP_NAME, 181 | app_icon=self.tray_icon_path, timeout=5) 182 | except: 183 | pass 184 | 185 | def quit(self, *args): 186 | """callback when selecting quit from systray menu""" 187 | # thread safe call for main window close 188 | self.main_window.run_method(self.main_window.quit) 189 | 190 | -------------------------------------------------------------------------------- /vortexdm/themes.py: -------------------------------------------------------------------------------- 1 | """ 2 | Vortex Download Manager (VortexDM) 3 | 4 | A multi-connection internet download manager, based on "PycURL" and "youtube_dl". Original project, FireDM, by Mahmoud Elshahat. 5 | :copyright: (c) 2023 by Sixline 6 | :copyright: (c) 2019-2021 by Mahmoud Elshahat. 7 | :license: GNU GPLv3, see LICENSE.md for more details. 8 | """ 9 | 10 | import awesometkinter as atk 11 | 12 | # main colors 13 | MAIN_BG = "#1c1c21" 14 | MAIN_FG = "white" 15 | 16 | # side frame colors 17 | SF_BG = "#000300" 18 | SF_FG = "white" 19 | SF_BTN_BG = "#d9dc4b" 20 | SF_CHKMARK = "#d9dc4b" 21 | 22 | THUMBNAIL_BG = "#000300" # color of thumbnail frame in Home 23 | THUMBNAIL_FG = "#d9dc4b" # color of base thumbnail photo 24 | THUMBNAIL_BD = "#d9dc4b" # thumbnail border color 25 | 26 | # progressbar 27 | PBAR_BG = "#26262b" 28 | PBAR_FG = "#d9dc4b" 29 | PBAR_TXT = "white" 30 | 31 | ENTRY_BD_COLOR = "#000300" 32 | 33 | BTN_BG = "#d9dc4b" 34 | BTN_FG = "black" 35 | BTN_HBG = "#000300" # highlight background 36 | BTN_ABG = "#000300" # active background 37 | BTN_AFG = "white" 38 | 39 | # heading e.g. "Network:" heading in Settings tab 40 | HDG_BG = "#d9dc4b" 41 | HDG_FG = "black" 42 | 43 | # scrollbar 44 | SBAR_BG = "#1c1c21" 45 | SBAR_FG = "white" 46 | 47 | # right click menu 48 | RCM_BG = "#1c1c21" 49 | RCM_FG = "white" 50 | RCM_ABG = "#d9dc4b" 51 | RCM_AFG = "black" 52 | 53 | # titlebar 54 | TITLE_BAR_BG = "#d9dc4b" 55 | TITLE_BAR_FG = "black" 56 | 57 | # selection color for DItem 58 | SEL_BG = "#000300" 59 | SEL_FG = "white" 60 | 61 | builtin_themes = { 62 | "Black & White & Midnight": { 63 | "MAIN_BG": "white", 64 | "SF_BG": "Black", 65 | "SF_BTN_BG": "White", 66 | "SF_FG": "#f2a541", 67 | "BTN_BG": "#631d76", 68 | "BTN_FG": "white", 69 | "BTN_HBG": "white", 70 | "RCM_FG": "black", 71 | "RCM_ABG": "#631d76", 72 | "RCM_AFG": "white", 73 | "SEL_BG": "#f2a541", 74 | "SEL_FG": "black", 75 | "TITLE_BAR_BG": "#631d76", 76 | "TITLE_BAR_FG": "white", 77 | }, 78 | "Black_Grey": { 79 | "MAIN_BG": "#393939", 80 | "SF_BG": "#202020", 81 | "SF_BTN_BG": "#7A7A7A", 82 | "PBAR_FG": "#06B025", 83 | "SF_CHKMARK": "#C6C6C6", 84 | "THUMBNAIL_FG": "#797979", 85 | "THUMBNAIL_BD": "#797979", 86 | "BTN_FG": "white", 87 | "BTN_HBG": "#C6C6C6", 88 | "BTN_ABG": "#C6C6C6", 89 | "HDG_FG": "white", 90 | "SBAR_BG": "#171717", 91 | "SBAR_FG": "#4D4D4D", 92 | "RCM_ABG": "#414141", 93 | "RCM_AFG": "white", 94 | "SEL_BG": "#004D8A" 95 | }, 96 | "Black_Grey_Shade-of-Pink": { 97 | "MAIN_BG": "#444444", 98 | "SF_BG": "#171717", 99 | "SF_BTN_BG": "#EDEDED", 100 | "PBAR_FG": "#DA0037", 101 | "MAIN_FG": "#EDEDED", 102 | "SF_FG": "#EDEDED", 103 | "SF_CHKMARK": "#DA0037", 104 | "THUMBNAIL_FG": "#DA0037", 105 | "PBAR_TXT": "#DA0037", 106 | "BTN_AFG": "#DA0037", 107 | "SBAR_BG": "#171717", 108 | "SBAR_FG": "#DA0037", 109 | "RCM_FG": "#EDEDED", 110 | "RCM_AFG": "#DA0037", 111 | "SEL_FG": "#EDEDED", 112 | "THUMBNAIL_BD": "#EDEDED", 113 | }, 114 | "Dark": { 115 | "MAIN_BG": "#1c1c21", 116 | "SF_BG": "#000300", 117 | "SF_BTN_BG": "#d9dc4b", 118 | "THUMBNAIL_FG": "#d9dc4b", 119 | "PBAR_FG": "#d9dc4b", 120 | "THUMBNAIL_BD": "#d9dc4b", 121 | }, 122 | "Gainsboro-SandyBrown-Teal": { 123 | "MAIN_BG": "#DDDDDD", 124 | "SF_BG": "#F5A962", 125 | "SF_BTN_BG": "#3C8DAD", 126 | "PBAR_FG": "#125D98", 127 | "SF_FG": "#125D98", 128 | "SF_CHKMARK": "#125D98", 129 | "THUMBNAIL_FG": "#125D98", 130 | "PBAR_TXT": "#125D98", 131 | "BTN_FG": "#DDDDDD", 132 | "HDG_FG": "#DDDDDD", 133 | "SBAR_FG": "#125D98", 134 | "RCM_FG": "#125D98", 135 | "RCM_AFG": "#DDDDDD", 136 | "TITLE_BAR_FG": "#125D98", 137 | "SEL_FG": "#125D98", 138 | }, 139 | "Green-Brown": { 140 | "MAIN_BG": "#3A6351", 141 | "SF_BG": "#F2EDD7", 142 | "SF_BTN_BG": "#A0937D", 143 | "PBAR_FG": "#5F939A", 144 | "BTN_ABG": "#446d5b", 145 | }, 146 | "Orange_Black": { 147 | "SF_BTN_BG": "#e09f3e", 148 | "PBAR_FG": "#FFFFFF", 149 | "SF_CHKMARK": "white", 150 | "PBAR_BG": "#0a0a0a", 151 | "SBAR_FG": "#e09f3e", 152 | "MAIN_BG": "#1c1c21", 153 | "SF_BG": "#000300", 154 | }, 155 | "Red_Black": { 156 | "SF_BTN_BG": "#960000", 157 | "PBAR_FG": "#e09f3e", 158 | "SF_CHKMARK": "#e09f3e", 159 | "PBAR_BG": "#0a0a0a", 160 | "BTN_FG": "white", 161 | "HDG_FG": "white", 162 | "SBAR_FG": "#960000", 163 | "RCM_AFG": "white", 164 | "MAIN_BG": "#1c1c21", 165 | "SF_BG": "#000300", 166 | "TITLE_BAR_FG": "white", 167 | }, 168 | "White & Black": { 169 | "MAIN_BG": "white", 170 | "SF_BG": "white", 171 | "SF_BTN_BG": "black", 172 | "PBAR_BG": "white", 173 | "ENTRY_BD_COLOR": "black", 174 | "BTN_FG": "white", 175 | "BTN_AFG": "black", 176 | "HDG_FG": "white", 177 | "RCM_FG": "black", 178 | "RCM_AFG": "white", 179 | "TITLE_BAR_FG": "white", 180 | "SEL_BG": "#d8d8d8", 181 | }, 182 | "White_BlueCryola": { 183 | "MAIN_BG": "white", 184 | "SF_BG": "white", 185 | "SF_BTN_BG": "#2d82b7", 186 | "SF_CHKMARK": "black", 187 | "BTN_FG": "white", 188 | "BTN_HBG": "black", 189 | "BTN_AFG": "#2d82b7", 190 | "HDG_FG": "white", 191 | "SBAR_FG": "#2d82b7", 192 | "RCM_FG": "black", 193 | "RCM_AFG": "white", 194 | "TITLE_BAR_FG": "white", 195 | "SEL_BG": "#58a7d6", 196 | "SEL_FG": "white", 197 | }, 198 | "White_DimGrey_BrightYellowCrayola": { 199 | "MAIN_BG": "white", 200 | "SF_BG": "#716969", 201 | "SF_BTN_BG": "#fbb13c", 202 | "PBAR_FG": "#fbb13c", 203 | "SF_CHKMARK": "white", 204 | "THUMBNAIL_BG": "#fbb13c", 205 | "BTN_BG": "black", 206 | "BTN_FG": "white", 207 | "BTN_ABG": "#fbb13c", 208 | "HDG_BG": "white", 209 | "RCM_FG": "black", 210 | "RCM_ABG": "#716969", 211 | "RCM_AFG": "white", 212 | "BTN_AFG": "black", 213 | "TITLE_BAR_BG": "black", 214 | "TITLE_BAR_FG": "white", 215 | }, 216 | "White_RoyalePurple_GoldFusion": { 217 | "MAIN_BG": "white", 218 | "SF_BG": "#7d5ba6", 219 | "SF_BTN_BG": "white", 220 | "PBAR_FG": "#7d5ba6", 221 | "THUMBNAIL_BG": "#72705b", 222 | "BTN_BG": "#72705b", 223 | "BTN_FG": "white", 224 | "BTN_HBG": "black", 225 | "SBAR_FG": "#7d5ba6", 226 | "RCM_FG": "black", 227 | "RCM_ABG": "#7d5ba6", 228 | "RCM_AFG": "white", 229 | "SEL_BG": "black", 230 | "TITLE_BAR_BG": "#72705b", 231 | "TITLE_BAR_FG": "white", 232 | }, 233 | "White_UpsdellRed_Marigold": { 234 | "MAIN_BG": "white", 235 | "SF_BG": "white", 236 | "SF_BTN_BG": "#b10f2e", 237 | "SF_CHKMARK": "#eca72c", 238 | "BTN_FG": "white", 239 | "BTN_HBG": "black", 240 | "BTN_AFG": "#b10f2e", 241 | "HDG_FG": "white", 242 | "SBAR_FG": "#b10f2e", 243 | "RCM_FG": "black", 244 | "RCM_AFG": "white", 245 | "TITLE_BAR_FG": "white", 246 | "SEL_BG": "#eca72c", 247 | }, 248 | "Yellow-Foil-covered Sneakers": { 249 | "MAIN_BG": "#333652", 250 | "SF_BG": "#90adc6", 251 | "SF_BTN_BG": "#fad02c", 252 | "PBAR_FG": "#e9eaec", 253 | "MAIN_FG": "#e9eaec", 254 | "SF_CHKMARK": "#e9eaec", 255 | "BTN_HBG": "#e9eaec", 256 | "SBAR_FG": "#90adc6", 257 | "RCM_FG": "#e9eaec", 258 | "THUMBNAIL_FG": "#e9eaec", 259 | "THUMBNAIL_BD": "#e9eaec", 260 | }, 261 | 262 | "White-grey-blue": { 263 | "MAIN_BG": "#f6f6f6", 264 | "SF_BG": "#d6cebf", 265 | "SF_BTN_BG": "#368ee6", 266 | "PBAR_FG": "#0085ff", 267 | "BTN_FG": "white", 268 | "HDG_FG": "white", 269 | "RCM_FG": "black", 270 | "TITLE_BAR_FG": "white" 271 | }, 272 | "White-sky-blue": { 273 | "MAIN_BG": "#ffffff", 274 | "SF_BG": "#d0eaff", 275 | "SF_BTN_BG": "#009ddc", 276 | "PBAR_FG": "#009ddc", 277 | "BTN_FG": "white", 278 | "HDG_FG": "white", 279 | "RCM_FG": "black", 280 | "TITLE_BAR_FG": "white" 281 | }, 282 | 283 | "Light-Orange": { 284 | "MAIN_BG": "white", 285 | "SF_BG": "#ffad00", 286 | "SF_BTN_BG": "#006cff", 287 | "PBAR_FG": "#006cff", 288 | "THUMBNAIL_FG": "#006cff", 289 | "HDG_FG": "white", 290 | "RCM_FG": "black", 291 | "SBAR_FG": "#ffad00" 292 | } 293 | } 294 | 295 | # key:(reference key, description), reference key will be used to get the color value in case of missing key, but in 296 | # case of some font keys, reference key is refering to background color which will be used to calculate font color 297 | # if reference key is None, this means it can't be calculated if missing 298 | theme_map = { 299 | 'MAIN_BG': (None, 'Main background'), 300 | 'MAIN_FG': ('MAIN_BG', 'Main text color'), 301 | 'SF_BG': (None, 'Side frame background'), 302 | 'SF_BTN_BG': (None, 'Side frame button color'), 303 | 'SF_FG': ('SF_BG', 'Side frame text color'), 304 | 'SF_CHKMARK': ('SF_BTN_BG', 'Side Frame check mark color'), 305 | 'THUMBNAIL_BG': ('SF_BG', 'Thumbnail background'), 306 | 'THUMBNAIL_FG': ('MAIN_FG', 'Default Thumbnail image color'), 307 | 'THUMBNAIL_BD': ('MAIN_FG', 'Thumbnail border color'), 308 | 'PBAR_BG': (None, 'Progressbar inactive ring color'), 309 | 'PBAR_FG': ('MAIN_FG', 'Progressbar active ring color'), 310 | 'PBAR_TXT': ('MAIN_BG', 'Progressbar text color'), 311 | 'ENTRY_BD_COLOR': ('SF_BG', 'Entry widget border color'), 312 | 'BTN_BG': ('SF_BTN_BG', 'Button background'), 313 | 'BTN_FG': ('BTN_BG', 'Button text color'), 314 | 'BTN_HBG': ('SF_BG', 'Button highlight background'), 315 | 'BTN_ABG': ('SF_BG', 'Button active background'), 316 | 'BTN_AFG': ('BTN_ABG', 'Button active text color'), 317 | 'HDG_BG': ('SF_BTN_BG', 'Heading title background'), 318 | 'HDG_FG': ('HDG_BG', 'Heading title text color'), 319 | 'SBAR_BG': ('MAIN_BG', 'Scrollbar background'), 320 | 'SBAR_FG': ('MAIN_FG', 'scrollbar active color'), 321 | 'RCM_BG': ('MAIN_BG', 'Right click menu background'), 322 | 'RCM_FG': ('RCM_BG', 'Right click menu text color'), 323 | 'RCM_ABG': ('BTN_BG', 'Right click menu active background'), 324 | 'RCM_AFG': ('RCM_ABG', 'Right click menu active text color'), 325 | 'TITLE_BAR_BG': ('BTN_BG', 'Window custom titlebar background'), 326 | 'TITLE_BAR_FG': ('BTN_FG', 'Window custom titlebar text color'), 327 | 'SEL_BG': ('SF_BG', 'Download item selection background'), 328 | 'SEL_FG': ('SF_FG', 'Download item selection foreground')} 329 | 330 | # fonts keys in theme map 331 | theme_fonts_keys = ('MAIN_FG', 'SF_FG', 'BTN_FG', 'BTN_AFG', 'PBAR_TXT', 'HDG_FG', 'RCM_FG', 'RCM_AFG') 332 | 333 | 334 | def calculate_missing_theme_keys(theme): 335 | """calculate missing key colors 336 | Args: 337 | theme (dict): theme dictionary 338 | """ 339 | 340 | # make sure we have main keys 341 | main_keys = ('MAIN_BG', 'SF_BG', 'SF_BTN_BG') 342 | for key in main_keys: 343 | theme.setdefault(key, globals()[key]) 344 | 345 | # progressbar 346 | theme.setdefault('PBAR_BG', atk.calc_contrast_color(theme['MAIN_BG'], 10)) 347 | 348 | for k, v in theme_map.items(): 349 | if k not in theme_fonts_keys: 350 | fallback_key = v[0] 351 | if fallback_key is not None: 352 | theme.setdefault(k, theme.get(fallback_key, globals()[fallback_key])) 353 | 354 | for key in theme_fonts_keys: 355 | bg_key = theme_map[key][0] 356 | bg = theme.get(bg_key, globals()[bg_key]) 357 | theme.setdefault(key, atk.calc_font_color(bg)) 358 | 359 | 360 | def strip_theme(theme): 361 | """remove any keys that can be calculated, make user themes more compact 362 | Args: 363 | theme (dict): theme dictionary 364 | 365 | Return: 366 | (dict): new stripped theme 367 | """ 368 | 369 | main_keys = ('MAIN_BG', 'SF_BG', 'SF_BTN_BG') 370 | dummy_theme = {k: theme[k] for k in main_keys} 371 | calculate_missing_theme_keys(dummy_theme) 372 | # dummy_theme = {k: v for k, v in dummy_theme.items() if k not in main_keys} 373 | for k in main_keys: 374 | dummy_theme[k] = None 375 | 376 | theme = {k: v for k, v in theme.items() if v != dummy_theme[k]} 377 | return theme 378 | 379 | 380 | # calculate missing keys for builtin themes 381 | for t in builtin_themes.values(): 382 | calculate_missing_theme_keys(t) 383 | 384 | if __name__ == '__main__': 385 | keys = sorted(builtin_themes.keys()) 386 | for name in keys: 387 | theme = builtin_themes[name] 388 | print(f'"{name}": ', '{') 389 | for k, v in strip_theme(theme).items(): 390 | print(f' "{k}": "{v}",') 391 | print('},') 392 | -------------------------------------------------------------------------------- /vortexdm/update.py: -------------------------------------------------------------------------------- 1 | """ 2 | Vortex Download Manager (VortexDM) 3 | 4 | A multi-connection internet download manager, based on "PycURL" and "youtube_dl". Original project, FireDM, by Mahmoud Elshahat. 5 | :copyright: (c) 2023 by Sixline 6 | :copyright: (c) 2019-2021 by Mahmoud Elshahat. 7 | :license: GNU GPLv3, see LICENSE.md for more details. 8 | """ 9 | 10 | # todo: change docstring to google format and clean unused code 11 | # check and update application 12 | 13 | import hashlib 14 | import json 15 | import py_compile 16 | import re 17 | import shutil 18 | import sys 19 | import zipfile, tarfile 20 | import queue 21 | import time 22 | from threading import Thread 23 | from distutils.dir_util import copy_tree 24 | import os 25 | import webbrowser 26 | from packaging.version import parse as parse_version 27 | 28 | from . import config 29 | from .utils import log, download, run_command, delete_folder, delete_file 30 | 31 | 32 | def open_update_link(): 33 | """open browser window with latest release url on github for frozen application or source code url""" 34 | url = config.LATEST_RELEASE_URL if config.FROZEN else config.APP_URL 35 | webbrowser.open_new(url) 36 | 37 | 38 | def check_for_new_version(): 39 | """ 40 | Check for new VortexDM version 41 | 42 | Return: 43 | changelog text or None 44 | 45 | """ 46 | 47 | latest_version = '0' 48 | changelog = None 49 | 50 | try: 51 | if config.FROZEN: 52 | # use github API to get latest version 53 | url = 'https://api.github.com/repos/Sixline/VortexDM/releases/latest' 54 | contents = download(url, verbose=False) 55 | 56 | if contents: 57 | j = json.loads(contents) 58 | latest_version = j.get('tag_name', '0') 59 | 60 | else: 61 | # check pypi version 62 | latest_version, _ = get_pkg_latest_version('vortexdm') 63 | 64 | if parse_version(latest_version) > parse_version(config.APP_VERSION): 65 | log('Found new version:', str(latest_version)) 66 | 67 | # download change log file 68 | url = 'https://github.com/Sixline/VortexDM/raw/master/ChangeLog.txt' 69 | changelog = download(url, verbose=False) 70 | except Exception as e: 71 | log('check_for_new_version()> error:', e) 72 | 73 | return changelog 74 | 75 | 76 | def get_pkg_latest_version(pkg, fetch_url=True): 77 | """get latest stable package release version on https://pypi.org/ 78 | reference: https://warehouse.pypa.io/api-reference/ 79 | Available strategies: 80 | 1 - rss feed (faster and lighter), send xml info with latest release version but no info on "wheel file" url, 81 | pattern example: https://pypi.org/rss/project/youtube-dl/releases.xml 82 | example data: 83 | 84 | 2020.12.14 85 | https://pypi.org/project/youtube-dl/2020.12.14/ 86 | YouTube video downloader 87 | dstftw@gmail.com 88 | Sun, 13 Dec 2020 17:59:21 GMT 89 | 90 | 91 | 2- json, (slower and bigger file), send all info for the package 92 | url pattern: f'https://pypi.org/pypi/{pkg}/json' e.g. https://pypi.org/pypi/vortexdm/json 93 | received json will be a dict with: 94 | keys = 'info', 'last_serial', 'releases', 'urls' 95 | releases = {'release_version': [{dict for wheel file}, {dict for tar file}], ...} 96 | dict for wheel file = {"filename":"yt_dlp-2020.10.24.post6-py2.py3-none-any.whl", 'url': 'file url'} 97 | dict for tar file = {"filename":"yt_dlp-2020.10.24.post6.tar.gz", 'url': 'file url'} 98 | 99 | 100 | Args: 101 | pkg (str): package name 102 | fetch_url (bool): if true, will use json API to get download url, else it will use rss feed to get version only 103 | 104 | Return: 105 | 2-tuple(str, str): latest_version, and download url (for wheel file) if available 106 | """ 107 | 108 | # download json info 109 | url = f'https://pypi.org/pypi/{pkg}/json' if fetch_url else f'https://pypi.org/rss/project/{pkg}/releases.xml' 110 | 111 | # get BytesIO object 112 | log(f'Checking for the latest version of {pkg} on pypi.org...') 113 | contents = download(url, verbose=False) 114 | latest_version = None 115 | url = None 116 | 117 | if contents: 118 | # rss feed 119 | if not fetch_url: 120 | match = re.findall(r'(\d+.\d+.\d+.*)', contents) 121 | latest_version = max([parse_version(release) for release in match]) if match else None 122 | 123 | if latest_version: 124 | latest_version = str(latest_version) 125 | # json 126 | else: 127 | j = json.loads(contents) 128 | 129 | releases = j.get('releases', {}) 130 | if releases: 131 | 132 | latest_version = max([parse_version(release) for release in releases.keys()]) or None 133 | if latest_version: 134 | latest_version = str(latest_version) 135 | 136 | # get latest release url 137 | release_info = releases[latest_version] 138 | for _dict in release_info: 139 | file_name = _dict['filename'] 140 | url = None 141 | if file_name.endswith('.whl'): 142 | url = _dict['url'] 143 | break 144 | 145 | return latest_version, url 146 | 147 | else: 148 | log(f"get_pkg_latest_version() --> couldn't check for {pkg}, url is unreachable") 149 | return None, None 150 | 151 | 152 | def get_target_folder(pkg): 153 | # determine target folder 154 | current_directory = config.current_directory 155 | if config.FROZEN: # windows cx_freeze 156 | # current directory is the directory of exe file 157 | target_folder = os.path.join(config.current_directory, 'lib') 158 | elif config.isappimage: 159 | # keep every package in isolated folder, to add individual package path to sys.path if it has newer version 160 | # than same pkg in AppImage's site-packages folder 161 | target_folder = os.path.join(config.sett_folder, config.appimage_update_folder, f'updated-{pkg}') 162 | else: 163 | target_folder = None 164 | 165 | return target_folder 166 | 167 | 168 | def update_pkg(pkg, url): 169 | """updating a package in frozen application folder 170 | expect to download and extract a wheel file e.g. "yt_dlp-2020.10.24.post6-py2.py3-none-any.whl", which in fact 171 | is a zip file 172 | 173 | Args: 174 | pkg (str): package name 175 | url (str): download url (for a wheel file) 176 | """ 177 | 178 | log(f'start updating {pkg}') 179 | 180 | target_folder = get_target_folder(pkg) 181 | 182 | # check if the application is frozen, e.g. runs from a windows cx_freeze executable 183 | # if run from source, we will update system installed package and exit 184 | if not target_folder: 185 | cmd = f'"{sys.executable}" -m pip install {pkg} --upgrade' 186 | error, output = run_command(cmd) 187 | if not error: 188 | log(f'successfully updated {pkg}') 189 | return True 190 | 191 | # paths 192 | temp_folder = os.path.join(target_folder, f'temp_{pkg}') 193 | extract_folder = os.path.join(temp_folder, 'extracted') 194 | z_fn = f'{pkg}.zip' 195 | z_fp = os.path.join(temp_folder, z_fn) 196 | 197 | target_pkg_folder = os.path.join(target_folder, pkg) 198 | bkup_folder = os.path.join(target_folder, f'{pkg}_bkup') 199 | new_pkg_folder = None 200 | 201 | # make temp folder 202 | log('making temp folder in:', target_folder) 203 | os.makedirs(temp_folder, exist_ok=True) 204 | 205 | def bkup(): 206 | # backup current package folder 207 | log(f'delete previous backup and backup current {pkg}:') 208 | delete_folder(bkup_folder) 209 | shutil.copytree(target_pkg_folder, bkup_folder) 210 | 211 | def tar_extract(): 212 | with tarfile.open(z_fp, 'r') as tar: 213 | tar.extractall(path=extract_folder) 214 | 215 | def zip_extract(): 216 | with zipfile.ZipFile(z_fp, 'r') as z: 217 | z.extractall(path=extract_folder) 218 | 219 | extract = zip_extract 220 | 221 | def overwrite_pkg(): 222 | delete_folder(target_pkg_folder) 223 | shutil.move(new_pkg_folder, target_pkg_folder) 224 | log('new package copied to:', target_pkg_folder) 225 | 226 | # start processing ------------------------------------------------------- 227 | log(f'start updating {pkg} please wait ...') 228 | 229 | try: 230 | # use a thread to show some progress while backup 231 | t = Thread(target=bkup) 232 | t.start() 233 | while t.is_alive(): 234 | log('#', start='', end='') 235 | time.sleep(0.3) 236 | 237 | log('\n', start='') 238 | 239 | # download from pypi 240 | log(f'downloading {pkg} raw files') 241 | data = download(url, fp=z_fp) 242 | if not data: 243 | log(f'failed to download {pkg}, abort update') 244 | return 245 | 246 | # extract tar file 247 | log(f'extracting {z_fn}') 248 | 249 | # use a thread to show some progress while unzipping 250 | t = Thread(target=extract) 251 | t.start() 252 | while t.is_alive(): 253 | log('#', start='', end='') 254 | time.sleep(0.3) 255 | 256 | log('\n', start='') 257 | log(f'{z_fn} extracted to: {temp_folder}') 258 | 259 | # define new pkg folder 260 | new_pkg_folder = os.path.join(extract_folder, pkg) 261 | 262 | # delete old package and replace it with new one 263 | log(f'overwrite old {pkg} files') 264 | overwrite_pkg() 265 | 266 | # clean old files 267 | log('delete temp folder') 268 | delete_folder(temp_folder) 269 | log(f'{pkg} ..... done updating') 270 | return True 271 | except Exception as e: 272 | log(f'update_pkg()> error', e) 273 | 274 | 275 | def rollback_pkg_update(pkg): 276 | """rollback last package update 277 | 278 | Args: 279 | pkg (str): package name 280 | """ 281 | 282 | target_folder = get_target_folder(pkg) 283 | 284 | if not target_folder: 285 | log(f'rollback {pkg} update is currently working on windows exe and Linux AppImage versions', showpopup=True) 286 | return 287 | 288 | log(f'rollback last {pkg} update ................................') 289 | 290 | # paths 291 | target_pkg_folder = os.path.join(target_folder, pkg) 292 | bkup_folder = os.path.join(target_folder, f'{pkg}_bkup') 293 | 294 | try: 295 | # find a backup first 296 | if os.path.isdir(bkup_folder): 297 | log(f'delete active {pkg} module') 298 | delete_folder(target_pkg_folder) 299 | 300 | log(f'copy backup {pkg} module') 301 | shutil.copytree(bkup_folder, target_pkg_folder) 302 | 303 | log(f'Done restoring {pkg} module, please restart Application now', showpopup=True) 304 | else: 305 | log(f'No {pkg} backup found') 306 | 307 | except Exception as e: 308 | log('rollback_pkg_update()> error', e) 309 | 310 | 311 | 312 | 313 | -------------------------------------------------------------------------------- /vortexdm/version.py: -------------------------------------------------------------------------------- 1 | __version__ = '2023.1.0' 2 | -------------------------------------------------------------------------------- /vortexdm/view.py: -------------------------------------------------------------------------------- 1 | """ 2 | Vortex Download Manager (VortexDM) 3 | 4 | A multi-connection internet download manager, based on "PycURL" and "youtube_dl". Original project, FireDM, by Mahmoud Elshahat. 5 | :copyright: (c) 2023 by Sixline 6 | :copyright: (c) 2019-2021 by Mahmoud Elshahat. 7 | :license: GNU GPLv3, see LICENSE.md for more details. 8 | 9 | Module description: 10 | An interface for All views / GUIs 11 | """ 12 | 13 | from abc import ABC, abstractmethod 14 | 15 | class IView(ABC): 16 | @abstractmethod 17 | def run(self): 18 | """run view mainloop if any""" 19 | pass 20 | 21 | @abstractmethod 22 | def quit(self): 23 | """quit view mainloop if any""" 24 | pass 25 | 26 | @abstractmethod 27 | def update_view(self, **kwargs): 28 | """update view, it will be called automatically by controller, when a model changes 29 | this method shouldn't block 30 | """ 31 | pass 32 | 33 | @abstractmethod 34 | def get_user_response(self, msg, options, **kwargs): 35 | """get user choice and send it back to controller, 36 | mainly this is a popup window or input() method in terminal 37 | 38 | Args: 39 | msg(str): a message to show 40 | options (list): a list of options, example: ['yes', 'no', 'cancel'] 41 | 42 | Returns: 43 | (str): response from user as a selected item from "options" 44 | 45 | Example: 46 | msg ="File with the same name already exists\n" \ 47 | "/home/mahmoud/Downloads/7z1900.exe\n" \ 48 | "Do you want to overwrite file?" 49 | 50 | option = ['Overwrite', 'Cancel'] 51 | 52 | get_user_response(msg, options) 53 | """ 54 | pass 55 | -------------------------------------------------------------------------------- /vortexdm/worker.py: -------------------------------------------------------------------------------- 1 | """ 2 | Vortex Download Manager (VortexDM) 3 | 4 | A multi-connection internet download manager, based on "PycURL" and "youtube_dl". Original project, FireDM, by Mahmoud Elshahat. 5 | :copyright: (c) 2023 by Sixline 6 | :copyright: (c) 2019-2021 by Mahmoud Elshahat. 7 | :license: GNU GPLv3, see LICENSE.md for more details. 8 | """ 9 | 10 | # worker class 11 | 12 | # todo: change docstring to google format and clean unused code 13 | 14 | import os 15 | import time 16 | import pycurl 17 | 18 | from .config import Status, error_q, jobs_q, max_seg_retries 19 | from .utils import log, set_curl_options, format_bytes, translate_server_code 20 | 21 | 22 | class Worker: 23 | def __init__(self, tag=0, d=None): 24 | self.tag = tag 25 | self.d = d 26 | self.seg = None 27 | self.resume_range = None 28 | 29 | # writing data parameters 30 | self.file = None 31 | self.mode = 'wb' # file opening mode default to new write binary 32 | 33 | self.buffer = 0 34 | self.timer1 = 0 35 | 36 | # connection parameters 37 | self.c = pycurl.Curl() 38 | self.speed_limit = 0 39 | self.headers = {} 40 | 41 | # minimum speed and timeout, abort if download speed slower than n byte/sec during n seconds 42 | self.minimum_speed = None 43 | self.timeout = None 44 | 45 | self.print_headers = True 46 | 47 | def __repr__(self): 48 | return f"worker_{self.tag}" 49 | 50 | def reuse(self, seg=None, speed_limit=0, minimum_speed=None, timeout=None): 51 | """Recycle same object again, better for performance as recommended by curl docs""" 52 | if seg.locked: 53 | log('Seg', self.seg.basename, 'segment in use by another worker', '- worker', {self.tag}, log_level=2) 54 | return False 55 | 56 | self.reset() 57 | 58 | self.seg = seg 59 | 60 | # set lock 61 | self.seg.locked = True 62 | 63 | self.speed_limit = speed_limit 64 | 65 | # minimum speed and timeout, abort if download speed slower than n byte/sec during n seconds 66 | self.minimum_speed = minimum_speed 67 | self.timeout = timeout 68 | 69 | msg = f'Seg {self.seg.basename} start, size: {format_bytes(self.seg.size)} - range: {self.seg.range}' 70 | if self.speed_limit: 71 | msg += f'- SL= {self.speed_limit}' 72 | if self.minimum_speed: 73 | msg += f'- minimum speed= {self.minimum_speed}, timeout={self.timeout}' 74 | 75 | log(msg, ' - worker', self.tag, log_level=2) 76 | 77 | self.check_previous_download() 78 | 79 | return True 80 | 81 | def reset(self): 82 | # reset curl options "only", other info cache stay intact, https://curl.haxx.se/libcurl/c/curl_easy_reset.html 83 | self.c.reset() 84 | 85 | # reset variables 86 | self.file = None 87 | self.mode = 'wb' # file opening mode default to new write binary 88 | self.buffer = 0 89 | self.resume_range = None 90 | self.headers = {} 91 | 92 | self.print_headers = True 93 | 94 | def check_previous_download(self): 95 | def overwrite(): 96 | # reset start size and remove value from d.downloaded 97 | self.report_download(-self.seg.current_size) 98 | self.mode = 'wb' 99 | log('Seg', self.seg.basename, 'overwrite the previous part-downloaded segment', ' - worker', self.tag, 100 | log_level=3) 101 | 102 | # if file doesn't exist will start fresh 103 | if not os.path.exists(self.seg.name): 104 | self.mode = 'wb' 105 | return 106 | 107 | if self.seg.current_size == 0: 108 | self.mode = 'wb' 109 | return 110 | 111 | # if no seg.size, we will overwrite current file because resume is not possible 112 | if not self.seg.size: 113 | overwrite() 114 | return 115 | 116 | # at this point file exists and resume is possible 117 | # case-1: segment is completed before 118 | if self.seg.current_size == self.seg.size: 119 | log('Seg', self.seg.basename, 'already completed before', ' - worker', self.tag, log_level=3) 120 | self.seg.downloaded = True 121 | 122 | # Case-2: over-sized, in case the server sent extra bytes from last session by mistake, truncate file 123 | elif self.seg.current_size > self.seg.size: 124 | log('Seg', self.seg.basename, 'over-sized', self.seg.current_size, 'will be truncated to:', 125 | format_bytes(self.seg.size), ' - worker', self.tag, log_level=3) 126 | 127 | self.seg.downloaded = True 128 | self.report_download(- (self.seg.current_size - self.seg.size)) 129 | 130 | # truncate file 131 | with open(self.seg.name, 'rb+') as f: 132 | f.truncate(self.seg.size) 133 | 134 | # Case-3: Resume, with new range 135 | elif self.seg.range and self.seg.current_size < self.seg.size: 136 | # set new range and file open mode 137 | a, b = self.seg.range 138 | self.resume_range = [a + self.seg.current_size, b] 139 | self.mode = 'ab' # open file for append 140 | 141 | # report 142 | log('Seg', self.seg.basename, 'resuming, new range:', self.resume_range, 143 | 'current segment size:', format_bytes(self.seg.current_size), ' - worker', self.tag, log_level=3) 144 | 145 | # case-x: overwrite 146 | else: 147 | overwrite() 148 | 149 | def verify(self): 150 | """check if segment completed""" 151 | # unknown segment size, will report done if there is any downloaded data > 0 152 | if self.seg.size == 0 and self.seg.current_size > 0: 153 | return True 154 | 155 | # segment has a known size 156 | elif self.seg.current_size >= self.seg.size: 157 | return True 158 | 159 | else: 160 | return False 161 | 162 | def report_not_completed(self): 163 | log('Seg', self.seg.basename, 'did not complete', '- done', format_bytes(self.seg.current_size), '- target size:', 164 | format_bytes(self.seg.size), '- left:', format_bytes(self.seg.size - self.seg.current_size), '- worker', self.tag, log_level=3) 165 | 166 | def report_completed(self): 167 | # self.debug('worker', self.tag, 'completed', self.seg.name) 168 | self.seg.downloaded = True 169 | 170 | # in case couldn't fetch segment size from headers 171 | if not self.seg.size: 172 | self.seg.size = self.seg.current_size 173 | # print(self.headers) 174 | 175 | log('downloaded segment: ', self.seg.basename, self.seg.range, format_bytes(self.seg.size), '- worker', self.tag, log_level=2) 176 | 177 | def set_options(self): 178 | 179 | # set general curl options 180 | 181 | # don't accept compressed contents 182 | self.d.http_headers['Accept-Encoding'] = '*;q=0' 183 | 184 | set_curl_options(self.c, http_headers=self.d.http_headers) 185 | 186 | self.c.setopt(pycurl.URL, self.seg.url) 187 | 188 | range_ = self.resume_range or self.seg.range 189 | if range_: 190 | self.c.setopt(pycurl.RANGE, f'{range_[0]}-{range_[1]}') # download segment only not the whole file 191 | 192 | self.c.setopt(pycurl.NOPROGRESS, 0) # will use a progress function 193 | 194 | # set speed limit selected by user 195 | self.c.setopt(pycurl.MAX_RECV_SPEED_LARGE, self.speed_limit) # cap download speed to n bytes/sec, 0=disabled 196 | 197 | # verbose 198 | self.c.setopt(pycurl.VERBOSE, 0) 199 | 200 | # call back functions 201 | self.c.setopt(pycurl.HEADERFUNCTION, self.header_callback) 202 | self.c.setopt(pycurl.WRITEFUNCTION, self.write) 203 | self.c.setopt(pycurl.XFERINFOFUNCTION, self.progress) 204 | 205 | # set minimum speed and timeout, abort if download speed slower than n byte/sec during n seconds 206 | if self.minimum_speed: 207 | self.c.setopt(pycurl.LOW_SPEED_LIMIT, self.minimum_speed) 208 | 209 | if self.timeout: 210 | self.c.setopt(pycurl.LOW_SPEED_TIME, self.timeout) 211 | 212 | def header_callback(self, header_line): 213 | header_line = header_line.decode('iso-8859-1') 214 | header_line = header_line.lower() 215 | 216 | if ':' not in header_line: 217 | return 218 | 219 | name, value = header_line.split(':', 1) 220 | name = name.strip() 221 | value = value.strip() 222 | self.headers[name] = value 223 | 224 | # update segment size if not available 225 | if not self.seg.size and name == 'content-length': 226 | try: 227 | self.seg.size = int(self.headers.get('content-length', 0)) 228 | 229 | seg = self.seg 230 | if seg.size and len(self.d.segments) == 1: 231 | if all([x not in self.d.subtype_list for x in ('hls', 'fragmented')]) and not seg.range: 232 | seg.range = [0, seg.size - 1] 233 | # print('self.seg.size = ', self.seg.size) 234 | except: 235 | pass 236 | 237 | def progress(self, *args): 238 | """it receives progress from curl and can be used as a kill switch 239 | Returning a non-zero value from this callback will cause curl to abort the transfer 240 | """ 241 | 242 | # check termination by user 243 | if self.d.status != Status.downloading: 244 | return -1 # abort 245 | 246 | if self.headers and self.headers.get('content-range') and self.print_headers: 247 | range_ = self.resume_range or self.seg.range 248 | log('Seg', self.seg.basename, 'range:', range_, 'server headers, range, size', 249 | self.headers.get('content-range'), self.headers.get('content-length'), log_level=3) 250 | self.print_headers = False 251 | 252 | def report_error(self, description='unspecified error'): 253 | # report server error to thread manager, to dynamically control connections number 254 | error_q.put(description) 255 | 256 | def report_download(self, value): 257 | """report downloaded to DownloadItem""" 258 | if isinstance(value, (int, float)): 259 | self.d.downloaded += value 260 | self.seg.down_bytes += value 261 | 262 | def run(self): 263 | try: 264 | 265 | # check if file completed before and exit 266 | if self.seg.downloaded: 267 | raise Exception('completed before') 268 | 269 | if not self.seg.url: 270 | log('Seg', self.seg.basename, 'segment has no valid url', '- worker', {self.tag}, log_level=2) 271 | raise Exception('invalid url') 272 | 273 | # set options 274 | self.set_options() 275 | 276 | # make sure target directory exist 277 | target_directory = os.path.dirname(self.seg.name) 278 | if not os.path.isdir(target_directory): 279 | os.makedirs(target_directory) # it will also create any intermediate folders in the given path 280 | 281 | # open segment file 282 | self.file = open(self.seg.name, self.mode, buffering=0) 283 | 284 | # Main Libcurl operation 285 | self.c.perform() 286 | 287 | # get response code and check for connection errors 288 | response_code = self.c.getinfo(pycurl.RESPONSE_CODE) 289 | if response_code in range(400, 512): 290 | log('Seg', self.seg.basename, 'server refuse connection', response_code, translate_server_code(response_code), 291 | 'content type:', self.headers.get('content-type'), log_level=3) 292 | 293 | # send error to thread manager, it will reduce connections number to fix this error 294 | self.report_error(f'server refuse connection: {response_code}, {translate_server_code(response_code)}') 295 | 296 | except Exception as e: 297 | # this error generated when user cancel download, or write function abort 298 | if '23' in repr(e) or '42' in repr(e): # ('Failed writing body', 'Callback aborted') 299 | error = f'terminated' 300 | log('Seg', self.seg.basename, error, 'worker', self.tag, log_level=3) 301 | else: 302 | error = repr(e) 303 | log('Seg', self.seg.basename, '- worker', self.tag, 'quitting ...', error, log_level=3) 304 | 305 | # report server error to thread manager 306 | self.report_error(repr(e)) 307 | 308 | finally: 309 | # report download 310 | self.report_download(self.buffer) 311 | self.buffer = 0 312 | 313 | # close segment file handle 314 | if self.file: 315 | self.file.close() 316 | 317 | # check if download completed 318 | completed = self.verify() 319 | if completed: 320 | self.report_completed() 321 | else: 322 | # if segment not fully downloaded send it back to thread manager to try again 323 | self.report_not_completed() 324 | 325 | # put back to jobs queue to try again 326 | jobs_q.put(self.seg) 327 | 328 | # remove segment lock 329 | self.seg.locked = False 330 | 331 | def write(self, data): 332 | """write to file""" 333 | 334 | quit_flag = False 335 | 336 | content_type = self.headers.get('content-type') 337 | if content_type and 'text/html' in content_type: 338 | # some video encryption keys has content-type 'text/html' 339 | try: 340 | decoded_data = data.decode('utf-8').lower() 341 | if not self.d.accept_html and (' 0: 356 | oversize = self.seg.current_size + len(data) - self.seg.size 357 | if oversize > 0: 358 | data = data[:-oversize] 359 | quit_flag = True 360 | 361 | # write to file 362 | self.file.write(data) 363 | 364 | self.buffer += len(data) 365 | 366 | # report to download item 367 | if time.time() - self.timer1 >= 1: 368 | self.timer1 = time.time() 369 | self.report_download(self.buffer) 370 | self.buffer = 0 371 | 372 | if quit_flag: 373 | return -1 # abort 374 | 375 | 376 | 377 | --------------------------------------------------------------------------------