├── .DS_Store ├── .gitignore ├── LICENSE.txt ├── MANIFEST ├── README.md ├── setup.cfg ├── setup.py └── src ├── DSDownload.py └── __init__.py /.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/d6o/DSDownload/c2566ccbf30a605be975ecff4ff0df0d61645ca6/.DS_Store -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | env/ 12 | build/ 13 | develop-eggs/ 14 | dist/ 15 | downloads/ 16 | eggs/ 17 | .eggs/ 18 | lib/ 19 | lib64/ 20 | parts/ 21 | sdist/ 22 | var/ 23 | *.egg-info/ 24 | .installed.cfg 25 | *.egg 26 | 27 | # PyInstaller 28 | # Usually these files are written by a python script from a template 29 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 30 | *.manifest 31 | *.spec 32 | 33 | # Installer logs 34 | pip-log.txt 35 | pip-delete-this-directory.txt 36 | 37 | # Unit test / coverage reports 38 | htmlcov/ 39 | .tox/ 40 | .coverage 41 | .coverage.* 42 | .cache 43 | nosetests.xml 44 | coverage.xml 45 | *,cover 46 | .hypothesis/ 47 | 48 | # Translations 49 | *.mo 50 | *.pot 51 | 52 | # Django stuff: 53 | *.log 54 | 55 | # Sphinx documentation 56 | docs/_build/ 57 | 58 | # PyBuilder 59 | target/ 60 | 61 | .idea/ -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | 2 | The MIT License (MIT) 3 | 4 | Copyright (c) 2016 Diego Siqueira 5 | 6 | Permission is hereby granted, free of charge, to any person obtaining a copy 7 | of this software and associated documentation files (the "Software"), to deal 8 | in the Software without restriction, including without limitation the rights 9 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 10 | copies of the Software, and to permit persons to whom the Software is 11 | furnished to do so, subject to the following conditions: 12 | 13 | The above copyright notice and this permission notice shall be included in all 14 | copies or substantial portions of the Software. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 17 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 18 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 19 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 20 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 21 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 22 | SOFTWARE. 23 | -------------------------------------------------------------------------------- /MANIFEST: -------------------------------------------------------------------------------- 1 | # file GENERATED by distutils, do NOT edit 2 | setup.cfg 3 | setup.py 4 | DSDownload/DSDownload.py 5 | DSDownload/__init__.py 6 | DSDownload/config.py 7 | DSDownload/downloadthread.py 8 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # DSDownload ![Language Badge](https://img.shields.io/badge/Language-Python-red.svg) ![Dependencies Badge](https://img.shields.io/badge/Dependencies-None-brightgreen.svg) ![License Badge](https://img.shields.io/badge/License-MIT-blue.svg) ![Status Badge](https://img.shields.io/badge/Status-Stable-brightgreen.svg) 2 | 3 | Program and module for download queue optimization using multi-thread 4 | 5 | ![](https://i.imgur.com/ytEp7fG.gif) 6 | 7 | ## Features 8 | 9 | - Can be used as script and module 10 | - Written in uncomplicated Python 11 | - Easily download files in the fastest speed possible 12 | - Easy to [install](https://github.com/DiSiqueira/DSDownload#installation) 13 | - Stupidly [easy to use](https://github.com/DiSiqueira/DSDownload#usage) 14 | - Very fast start up and response time 15 | - Uses natives libs 16 | - Option to organize your files 17 | - Download 100 files in less than 40s 18 | - Download files from everywhere 19 | - Get real filename 20 | - Supports HTTPS 21 | 22 | ## Installation 23 | 24 | ### Option 1: [Pip](https://pip.pypa.io/en/stable/installing/) 25 | 26 | ```bash 27 | $ pip install DSDownload 28 | ``` 29 | 30 | ### Option 2: From source 31 | 32 | ```bash 33 | $ git clone https://github.com/DiSiqueira/DSDownload.git 34 | $ cd DSDownload/ 35 | $ python setup.py install 36 | ``` 37 | 38 | ## Usage 39 | 40 | ### Basic usage 41 | 42 | ```bash 43 | # Download a file 44 | $ dsdownload https://github.com/DiSiqueira/DSDownload/archive/1.6.0.0.tar.gz 45 | ``` 46 | 47 | ### Download using Workers 48 | 49 | ```bash 50 | # Download 3 files using 2 Workers 51 | $ dsdownload --workers 2 https://i.imgur.com/eUrbKtO.jpg https://i.imgur.com/9am20SK.jpg https://github.com/DiSiqueira/DSDownload/archive/1.6.0.0.tar.gz 52 | ``` 53 | 54 | ### Combine everything 55 | 56 | ```bash 57 | # Download 3 files using 2 Workers and put on my-images folder 58 | $ dsdownload --output my-images --workers 2 https://github.com/DiSiqueira/DSDownload/archive/1.6.0.0.tar.gz https://i.imgur.com/9am20SK.jpg https://i.imgur.com/KR06C.jpg 59 | ``` 60 | 61 | ## Module Usage 62 | The module allows you to download url lists in your own Python programs without going through the command line. Here's an example of it's usage: 63 | 64 | ### Example 65 | ```python 66 | from DSDownload import DSDownload 67 | 68 | urls = ['https://i.imgur.com/eUrbKtO.jpg', 'https://github.com/DiSiqueira/DSDownload/archive/1.6.0.0.tar.gz'] 69 | workers = 2 70 | output = 'My-Files' 71 | 72 | DSDownload.DSDownload(urls, workers, output) 73 | ``` 74 | 75 | ## Program Help 76 | 77 | ![](https://i.imgur.com/NTzkmKJ.png) 78 | 79 | ## Contributing 80 | 81 | ### Bug Reports & Feature Requests 82 | 83 | Please use the [issue tracker](https://github.com/DiSiqueira/DSDownload/issues) to report any bugs or file feature requests. 84 | 85 | ### Developing 86 | 87 | PRs are welcome. To begin developing, do this: 88 | 89 | ```bash 90 | $ git clone --recursive git@github.com:DiSiqueira/DSDownload.git 91 | $ cd DSDownload/DSDownload/ 92 | $ python DSDownload.py 93 | ``` 94 | 95 | ## Social Coding 96 | 97 | 1. Create an issue to discuss about your idea 98 | 2. [Fork it] (https://github.com/DiSiqueira/DSDownload/fork) 99 | 3. Create your feature branch (`git checkout -b my-new-feature`) 100 | 4. Commit your changes (`git commit -am 'Add some feature'`) 101 | 5. Push to the branch (`git push origin my-new-feature`) 102 | 6. Create a new Pull Request 103 | 7. Profit! :white_check_mark: 104 | 105 | ## License 106 | 107 | The MIT License (MIT) 108 | 109 | Copyright (c) 2013-2017 Diego Siqueira 110 | 111 | Permission is hereby granted, free of charge, to any person obtaining a copy 112 | of this software and associated documentation files (the "Software"), to deal 113 | in the Software without restriction, including without limitation the rights 114 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 115 | copies of the Software, and to permit persons to whom the Software is 116 | furnished to do so, subject to the following conditions: 117 | 118 | The above copyright notice and this permission notice shall be included in 119 | all copies or substantial portions of the Software. 120 | 121 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 122 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 123 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 124 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 125 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 126 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 127 | THE SOFTWARE. 128 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [metadata] 2 | description-file = README.md -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from setuptools import setup 2 | 3 | setup( 4 | name='DSDownload', 5 | version='1.6.0.2', 6 | description='Program and module for download queue optimization using multi-thread.', 7 | url='https://github.com/DiSiqueira/DSDownload', 8 | author='Diego Siqueira', 9 | author_email='dieg0@live.com', 10 | license='MIT', 11 | package_dir={'DSDownload': 'src'}, 12 | packages=['DSDownload'], 13 | zip_safe=False, 14 | keywords=['download', 'thread', 'speed', 'resume', 'multi', 'simple'], 15 | entry_points= 16 | { 17 | 'console_scripts': 18 | [ 19 | 'dsdownload = DSDownload:main', 20 | ], 21 | }, 22 | ) 23 | -------------------------------------------------------------------------------- /src/DSDownload.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | """ 4 | Diego Martins de Siqueira 5 | MIT License 6 | DSDownload - DSDownload is a fully featured download library with focus on performance 7 | """ 8 | import threading 9 | import os 10 | import logging 11 | import queue as Queue 12 | import urllib.request as urllib2 13 | 14 | 15 | class DownloadThread(threading.Thread): 16 | colors = [ 17 | '\033[91m', 18 | '\033[92m', 19 | '\033[94m', 20 | '\033[96m', 21 | '\033[97m', 22 | '\033[93m', 23 | '\033[95m', 24 | '\033[90m', 25 | '\033[90m', 26 | '\033[99m', 27 | ] 28 | 29 | color_reset = '\033[0m' 30 | 31 | def __init__(self, queue, dest_folder='downloads'): 32 | super(DownloadThread, self).__init__() 33 | 34 | self._queue = queue 35 | self._dest_folder = dest_folder 36 | self.daemon = True 37 | 38 | self._handle_folder(self._dest_folder) 39 | 40 | @staticmethod 41 | def _handle_folder(folder): 42 | logging.debug('Testing Folder: ' + folder) 43 | if not os.path.exists(folder): 44 | logging.debug('Folder not found') 45 | logging.debug('Creating Folder: ' + folder) 46 | os.makedirs(folder) 47 | 48 | def run(self): 49 | 50 | while True: 51 | logging.debug('Getting URL from Queue') 52 | url = self._queue.get() 53 | logging.debug('URL:' + url) 54 | 55 | try: 56 | logging.debug('Preparing to Download url:' + url) 57 | self.prepare_download(url) 58 | logging.debug('Download complete url:' + url) 59 | 60 | except Exception as e: 61 | logging.debug('Error downloading') 62 | print("Error: {0}".format(e)) 63 | 64 | logging.debug('Task Done!') 65 | self._queue.task_done() 66 | 67 | def get_color(self, num): 68 | logging.debug('Getting Color') 69 | color_index = int(num) % len(self.colors) 70 | 71 | logging.debug('ColorIndex to Worker' + str(num) + ': ' + str(color_index)) 72 | return self.colors[color_index] 73 | 74 | @staticmethod 75 | def get_file_name(headers, url): 76 | 77 | logging.debug('Getting Content Disposition') 78 | con_disposition = headers.get('Content-Disposition') 79 | 80 | if con_disposition: 81 | logging.debug('Content Disposition Found') 82 | logging.debug('Searching for filename') 83 | con_name = con_disposition.split('filename=')[-1].replace('"', '').replace(';', '') 84 | 85 | if con_name: 86 | logging.debug('Filename found: ' + con_name) 87 | return con_name 88 | 89 | logging.debug('Spliting URL for filename') 90 | file_name = url.split('/')[-1] 91 | logging.debug('Filename: ' + file_name) 92 | 93 | return file_name 94 | 95 | def prepare_download(self, url): 96 | 97 | wnum = threading.current_thread().name.split('-')[-1] 98 | logging.debug('Finding Worker number: ' + wnum) 99 | 100 | color = self.get_color(wnum) 101 | 102 | logging.debug('Opening connection') 103 | url_con = urllib2.urlopen(url) 104 | 105 | logging.debug('Getting metadata') 106 | meta = url_con.info() 107 | 108 | file_name = self.get_file_name(meta, url) 109 | 110 | dest = os.path.join(self._dest_folder, file_name) 111 | logging.debug('Folder: ' + self._dest_folder) 112 | logging.debug('Destination: ' + dest) 113 | 114 | logging.debug('Opening file') 115 | dest_file = open(dest, 'wb') 116 | 117 | logging.debug('Getting file size') 118 | file_size = meta.get("Content-Length") 119 | logging.debug('File size:' + str(file_size)) 120 | 121 | print(color + "Downloading: %s Bytes: %s" % (file_name, file_size) + self.color_reset) 122 | 123 | file_size = 0 124 | block_size = 8192 125 | logging.debug('Block size:' + str(block_size)) 126 | 127 | while True: 128 | buffer = url_con.read(block_size) 129 | if not buffer: 130 | break 131 | 132 | logging.debug('Total Downloaded: ' + str(file_size)) 133 | file_size += len(buffer) 134 | dest_file.write(buffer) 135 | 136 | logging.debug('Total Downloaded: ' + str(file_size)) 137 | logging.debug('Closing file') 138 | dest_file.close() 139 | 140 | 141 | def DSDownload(url_list, workers=5, folder_path='downloads'): 142 | logging.debug('Starting DSDownload') 143 | queue = Queue.Queue() 144 | logging.debug('Instantiating Queue') 145 | 146 | for url in url_list: 147 | logging.debug('Add url to Queue: ' + url) 148 | queue.put(url) 149 | 150 | logging.debug('Testing number of Workers') 151 | workers = workers if workers < len(url_list) else len(url_list) 152 | logging.debug('Workers:' + str(workers)) 153 | 154 | for worker in range(workers): 155 | worker = str(worker + 1) 156 | logging.debug('Instantiating Worker: ' + worker) 157 | t = DownloadThread(queue, folder_path) 158 | logging.debug('Starting Worker: ' + worker) 159 | t.start() 160 | 161 | logging.debug('Waiting Workers') 162 | queue.join() 163 | -------------------------------------------------------------------------------- /src/__init__.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | """ 4 | Diego Martins de Siqueira 5 | MIT License 6 | DSDownload - DSDownload is a fully featured download library with focus on performance 7 | """ 8 | 9 | import sys 10 | import argparse 11 | import logging 12 | from DSDownload import DSDownload 13 | 14 | 15 | def main(): 16 | description = "DSDownload is a fully featured download library with focus on performance" 17 | parser = argparse.ArgumentParser( 18 | description=description) 19 | parser.add_argument("--version", action="version", version='1.6.0.2', 20 | help="Version Info") 21 | parser.add_argument("--workers", type=int, default=5, 22 | help="Number of parallel downloads. The default is 5.") 23 | parser.add_argument("--output", type=str, default="downloads", 24 | help="Output folder") 25 | parser.add_argument('urls', type=str, nargs='+', 26 | help='URLs to be downloaded') 27 | parser.add_argument("-v", "--verbose", action="store_true", 28 | help="increase output verbosity") 29 | 30 | args = parser.parse_args() 31 | 32 | if args.verbose: 33 | logging.basicConfig(level=logging.DEBUG) 34 | 35 | download = DSDownload 36 | if type(sys) is type(DSDownload): 37 | download = DSDownload.DSDownload 38 | 39 | try: 40 | download(args.urls, args.workers, args.output) 41 | print('All files were downloaded.') 42 | except KeyboardInterrupt: 43 | print('Interrupt received, stopping downloads') 44 | 45 | sys.exit() 46 | 47 | 48 | if __name__ == "__main__": 49 | main() 50 | --------------------------------------------------------------------------------