├── .gitignore ├── CHANGELOG.txt ├── MANIFEST.in ├── README.rst ├── TODO.txt ├── doc └── rest.css ├── misc ├── csv_dups.py ├── epydoc.patch └── test_threadworker.py ├── requirements-dev.txt ├── setup.cfg ├── setup.py ├── src ├── release.py └── threadpool.py ├── test └── test_issue1.py ├── tools ├── make_release.sh └── upload.sh └── tox.ini /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | 5 | # C extensions 6 | *.so 7 | 8 | # Distribution / packaging 9 | .Python 10 | venv/ 11 | build/ 12 | develop-eggs/ 13 | dist/ 14 | downloads/ 15 | eggs/ 16 | .eggs/ 17 | lib/ 18 | lib64/ 19 | parts/ 20 | sdist/ 21 | var/ 22 | *.egg-info/ 23 | .installed.cfg 24 | *.egg 25 | 26 | # PyInstaller 27 | # Usually these files are written by a python script from a template 28 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 29 | *.manifest 30 | *.spec 31 | 32 | # Installer logs 33 | pip-log.txt 34 | pip-delete-this-directory.txt 35 | 36 | # Unit test / coverage reports 37 | htmlcov/ 38 | .tox/ 39 | .coverage 40 | .coverage.* 41 | .cache 42 | nosetests.xml 43 | coverage.xml 44 | *,cover 45 | 46 | # Translations 47 | *.mo 48 | *.pot 49 | 50 | # Django stuff: 51 | *.log 52 | 53 | # Generated documentation 54 | doc/*.html 55 | doc/*.css 56 | doc/api 57 | 58 | # PyBuilder 59 | target/ 60 | -------------------------------------------------------------------------------- /CHANGELOG.txt: -------------------------------------------------------------------------------- 1 | .. -*- coding: UTF-8 -*- 2 | 3 | 2015-11-29 (1.3.2) 4 | - Added missing release.py to source distribution 5 | 6 | 2015-10-14 (1.3.1) 7 | - Minor distribution procedure changes 8 | 9 | 2015-10-14 (1.3.0) 10 | - Migrated repository from SVN to Git. 11 | - Incorporated changes (with minor adjustments) from Lutz Prechelt from 12 | https://github.com/prechelt/threadpool to make Threadpool Python 3 13 | compatible. Thanks, Lutz! 14 | - Fixed some build errors. 15 | 16 | 2009-10-07 (1.2.7) 17 | - I made a stupid error and made threadpool.py import from release.py but 18 | this module is not installed by setup.py. Removed import again. 19 | 20 | 2009-10-06 (1.2.6) 21 | - Due to some mix up up the I got the bugfix for the 'timeout' parameter 22 | to ThreadPool.putRequest exactly the wrong way round (or I "fixed" it 23 | twice). It now defaults to None as its should, so putRequest blocks by 24 | default, if the requests queue is full. Thanks for Guillaume Taglang for 25 | reporting the issue. 26 | - Rename NEWS.txt to CHANGELOG.txt (this file). 27 | - Add SVN checkout instructions to README. 28 | 29 | 2008-11-19 30 | - Update reference to "Python In A Nutshell" to second edition (suggested 31 | by Alex Martelli). 32 | - Fixed typo in WorkerThread.run() (thanks to Nicholas Bollweg, Aaron 33 | Levinson, Rogério Schneider, Grégory Starck for reporting). 34 | - Fixed missing first argument in call to Queue.get() in WorkerThread.run() 35 | (thanks to Aaron Levinson for report). 36 | - added new argument 'do_join' to ThreadPool.dismissWorkers(). When True, 37 | the method will perform Thread.join() on each thread after dismissing it. 38 | - Added joinAllDismissedWorkers method to ThreadPool to join dismissed 39 | threads at a later time (thanks to Aaron Levinson for patch for these two 40 | changes). 41 | 42 | 2008-05-04 43 | - 'timeout' parameter of ThreadPool.putRequest now correctly defaults to 0 44 | instead of None (thanks to Mads Sülau Jørgensen for bug report). 45 | - Added default exception handler callback (thanks to Moshe Cohen for the 46 | patch). 47 | - Fixed locking issue that prevented worker threads from being dismissed 48 | when no work requests are in the requests queue (thanks to Guillaume 49 | Pratte for the bug report). 50 | - Add option for results queue size to ThreadPool (thanks to Krzysztof 51 | Jakubczyk for the idea). 52 | - Changed name of reuquestQueue and resultsQueue attributes in WorkerThread 53 | and ThreadPool to _requests_queue and _results_queue to be more consistent 54 | and compliant with PEP 8 and properly indicate private nature. 55 | - Moved repository to Subversion. 56 | 57 | 2008-05-03 58 | - Updated homepage and download URL 59 | - Updated README 60 | - Enable packaging as an eggs with the use of setuptools 61 | - License changes to MIT License (Python license is only for code licensed 62 | by the PSF) 63 | 64 | 2006-06-23 1.2.3 (never announced) 65 | - fixed typo in ThreadPool.putRequest() (reported by Jérôme Schneider) 66 | 67 | 2006-05-19 1.2.2 (first release as a package) 68 | - fixed wrong usage of isinstance in makeRequests() 69 | Thanks to anonymous for bug report in comment on ASPN 70 | - added setup.py and created a proper distribution package 71 | - added timeout parameter to putRequest() 72 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | recursive-include doc *.css *.html 2 | include README.rst CHANGELOG.txt src/release.py 3 | global-exclude CVS 4 | prune venv 5 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | Threadpool 2 | ########## 3 | 4 | .. -*- coding: UTF-8 -*- 5 | 6 | :Title: Easy to use object-oriented thread pool framework 7 | :Author: Christopher Arndt 8 | :Version: 1.3.2 9 | :Date: 2015-11-29 10 | :License: MIT License 11 | 12 | 13 | .. warning:: 14 | This module is **OBSOLETE** and is only provided on PyPI to support old 15 | projects that still use it. Please **DO NOT USE IT FOR NEW PROJECTS!** 16 | 17 | Use modern alternatives like the `multiprocessing `_ 18 | module in the standard library or even an asynchroneous approach with 19 | `asyncio <_asyncio: https://docs.python.org/3/library/asyncio.html>`_. 20 | 21 | 22 | Description 23 | =========== 24 | 25 | A thread pool is an object that maintains a pool of worker threads to perform 26 | time consuming operations in parallel. It assigns jobs to the threads 27 | by putting them in a work request queue, where they are picked up by the 28 | next available thread. This then performs the requested operation in the 29 | background and puts the results in another queue. 30 | 31 | The thread pool object can then collect the results from all threads from 32 | this queue as soon as they become available or after all threads have 33 | finished their work. It's then possible to define callbacks to handle 34 | each result as it comes in. 35 | 36 | .. note:: 37 | This module is regarded as an extended example, not as a finished product. 38 | Feel free to adapt it too your needs. 39 | 40 | 41 | Basic usage 42 | =========== 43 | 44 | :: 45 | 46 | >>> pool = ThreadPool(poolsize) 47 | >>> requests = makeRequests(some_callable, list_of_args, callback) 48 | >>> [pool.putRequest(req) for req in requests] 49 | >>> pool.wait() 50 | 51 | See the end of the module source code for a longer, annotated usage example. 52 | 53 | 54 | Documentation 55 | ============= 56 | 57 | You can view the API documentation, generated by epydoc, here: 58 | 59 | `API documentation`_ 60 | 61 | The documentation is also packaged in the distribution. 62 | 63 | .. _api documentation: 64 | http://chrisarndt.de/projects/threadpool/api/ 65 | 66 | 67 | Download 68 | ======== 69 | 70 | You can download the latest version of this module here: 71 | 72 | `Download directory`_ 73 | 74 | or see the colorized source code: 75 | 76 | threadpool.py_ 77 | 78 | You can also install it from the Python Package Index PyPI_ via `pip`:: 79 | 80 | [sudo] pip install threadpool 81 | 82 | Or you can check out the latest development version from the Git 83 | repository:: 84 | 85 | git clone https://github.com/SpotlightKid/threadpool.git 86 | 87 | .. _download directory: 88 | http://chrisarndt.de/projects/threadpool/download/ 89 | .. _threadpool.py: 90 | http://chrisarndt.de/projects/threadpool/threadpool.py.html 91 | .. _pypi: http://pypi.python.org/pypi/threadpool 92 | 93 | 94 | Discussion 95 | ========== 96 | 97 | The basic concept and some code was taken from the book "Python in a Nutshell" 98 | by Alex Martelli, copyright O'Reilly 2003, ISBN 0-596-00188-6, from section 99 | 14.5 "Threaded Program Architecture". I wrapped the main program logic in the 100 | ``ThreadPool`` class, added the ``WorkRequest`` class and the callback system 101 | and tweaked the code here and there. 102 | 103 | There are some other recipes in the Python Cookbook, that serve a similar 104 | purpose. This one distinguishes itself by the following characteristics: 105 | 106 | * Object-oriented, reusable design 107 | 108 | * Provides callback mechanism to process results as they are returned from the 109 | worker threads. 110 | 111 | * ``WorkRequest`` objects wrap the tasks assigned to the worker threads and 112 | allow for easy passing of arbitrary data to the callbacks. 113 | 114 | * The use of the ``Queue`` class solves most locking issues. 115 | 116 | * All worker threads are daemonic, so they exit when the main programm exits, 117 | no need for joining. 118 | 119 | * Threads start running as soon as you create them. No need to start or stop 120 | them. You can increase or decrease the pool size at any time, superfluous 121 | threads will just exit when they finish their current task. 122 | 123 | * You don't need to keep a reference to a thread after you have assigned the 124 | last task to it. You just tell it: "don't come back looking for work, when 125 | you're done!" 126 | 127 | * Threads don't eat up cycles while waiting to be assigned a task, they just 128 | block when the task queue is empty (though they wake up every few seconds to 129 | check whether they are dismissd). 130 | 131 | 132 | Notes 133 | ----- 134 | 135 | Due to the parallel nature of threads, you have to keep some things in mind: 136 | 137 | * Do not use simultaneous threads for tasks were they compete for a single, 138 | scarce resource (e.g. a harddisk or stdout). This will probably be slower 139 | than taking a serialized approach. 140 | 141 | * If you call ``ThreadPool.wait()`` the main thread will block until _all_ 142 | results have arrived. If you only want to check for results that are available 143 | immediately, use ``ThreadPool.poll()``. 144 | 145 | * The results of the work requests are not stored anywhere. You should provide 146 | an appropriate callback if you want to do so. 147 | 148 | 149 | References 150 | ========== 151 | 152 | There are several other recipes similar to this module in the Python Cookbook, 153 | for example: 154 | 155 | * http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/203871 156 | * http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/196618 157 | * http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/302746 158 | 159 | 160 | News 161 | ==== 162 | 163 | .. include:: CHANGELOG.txt 164 | -------------------------------------------------------------------------------- /TODO.txt: -------------------------------------------------------------------------------- 1 | - Standard handler for exceptions in WorkRequests - done 2 | - Fix issue with not being able to dismiss WorkerThreads when the queue is 3 | empty, due to blocking Queue.get() - done 4 | 5 | - Add unit tests! 6 | - Collect results in list or provide generator when using wait 7 | - Sort out RPM vs Egg packaging issues (inclusion of documentation) 8 | -------------------------------------------------------------------------------- /doc/rest.css: -------------------------------------------------------------------------------- 1 | /* 2 | :Authors: Ian Bicking, Michael Foord 3 | :Contact: fuzzyman@voidspace.org.uk 4 | :Date: 2005/08/26 5 | :Version: 0.1.0 6 | :Copyright: This stylesheet has been placed in the public domain. 7 | 8 | Stylesheet for Docutils. 9 | Based on ``blue_box.css`` by Ian Bicking 10 | and ``default.css`` revision 1.46 11 | */ 12 | 13 | /* 14 | @import url(default.css); 15 | @import url(pysrc.css); 16 | */ 17 | 18 | body { 19 | font-family: Arial, sans-serif; 20 | background-color: #444499; 21 | margin: 0; 22 | padding: 0; 23 | } 24 | 25 | .document, div.footer { 26 | background-color: #ffffff; 27 | margin: 0 auto; 28 | width: 50em; 29 | padding: 0.5em; 30 | } 31 | 32 | .document { 33 | border-top: 4em solid #9999ee; 34 | padding-top: 1px 35 | } 36 | 37 | .footer { 38 | color: gray; 39 | } 40 | 41 | em, i { 42 | /* Typically serif fonts have much nicer italics */ 43 | font-family: Times New Roman, Times, serif; 44 | } 45 | 46 | 47 | a.target { 48 | color: blue; 49 | } 50 | 51 | a.toc-backref { 52 | text-decoration: none; 53 | color: black; 54 | } 55 | 56 | a.toc-backref:hover { 57 | background-color: inherit; 58 | } 59 | 60 | a:hover { 61 | background-color: #cccccc; 62 | } 63 | 64 | .topic-title { 65 | font-size: x-large; 66 | font-family: 'Trebuchet MS','Lucida Grande',Verdana,Arial,Sans-Serif; 67 | font-weight: bold; 68 | text-align: left; 69 | color: #555555; 70 | } 71 | 72 | div.attention, div.caution, div.danger, div.error, div.hint, 73 | div.important, div.note, div.tip, div.warning { 74 | background-color: #cccccc; 75 | padding: 3px; 76 | width: 80%; 77 | } 78 | 79 | div.admonition p.admonition-title, div.hint p.admonition-title, 80 | div.important p.admonition-title, div.note p.admonition-title, 81 | div.tip p.admonition-title { 82 | text-align: center; 83 | background-color: #999999; 84 | display: block; 85 | margin: 0; 86 | } 87 | 88 | div.attention p.admonition-title, div.caution p.admonition-title, 89 | div.danger p.admonition-title, div.error p.admonition-title, 90 | div.warning p.admonition-title { 91 | color: #cc0000; 92 | font-family: sans-serif; 93 | text-align: center; 94 | background-color: #999999; 95 | display: block; 96 | margin: 0; 97 | } 98 | 99 | h1, h2, h3, h4, h5, h6 { 100 | font-family: 'Trebuchet MS','Lucida Grande',Verdana,Arial,Sans-Serif; 101 | font-weight: bold; 102 | text-align: left; 103 | color: #555555; 104 | } 105 | 106 | h1 { 107 | font-size: 150%; 108 | } 109 | 110 | h1 a.toc-backref, h2 a.toc-backref { 111 | color: #555555; 112 | } 113 | 114 | h2 { 115 | font-size: 130%; 116 | } 117 | 118 | h3, h4, h5, h6 { 119 | font-size: 120%; 120 | } 121 | 122 | h3 a.toc-backref, h4 a.toc-backref, h5 a.toc-backref, 123 | h6 a.toc-backref { 124 | color: #555555; 125 | } 126 | 127 | h1.title { 128 | font-size: 400%; 129 | color: #555555; 130 | position: relative; 131 | display: block; 132 | top: -0.65em; 133 | left: 1em; 134 | margin: 0; 135 | } 136 | 137 | table.footnote { 138 | padding-left: 0.5ex; 139 | } 140 | 141 | table.citation { 142 | padding-left: 0.5ex 143 | } 144 | 145 | pre, .pre { 146 | font-family: monospace; 147 | font-size: 1.1em; 148 | color: #444499; 149 | } 150 | 151 | pre.literal-block, pre.doctest-block { 152 | border: 1px solid gray; 153 | padding: 5px; 154 | background-color: #dddddd; 155 | } 156 | 157 | .image img { border-style : solid; 158 | border-width : 2px; 159 | } 160 | 161 | h1 tt, h2 tt, h3 tt, h4 tt, h5 tt, h6 tt { 162 | font-size: 100%; 163 | } 164 | 165 | code, tt { 166 | color: #000066; 167 | } 168 | -------------------------------------------------------------------------------- /misc/csv_dups.py: -------------------------------------------------------------------------------- 1 | import csv, sys 2 | 3 | raw = open(sys.argv[1]) 4 | 5 | md5dict = dict() 6 | for md5 in csv.reader(raw): 7 | if md5dict.has_key(md5[1]): 8 | md5dict[md5[1]].append(md5[0]) 9 | else: 10 | md5dict[md5[1]] = [md5[0]] 11 | 12 | for key in md5dict: 13 | if len(md5dict[key]) > 1: 14 | print("\n".join(md5dict[key]), '\n') 15 | -------------------------------------------------------------------------------- /misc/epydoc.patch: -------------------------------------------------------------------------------- 1 | --- markup/restructuredtext.py.orig 2015-10-14 01:30:54.374853970 +0200 2 | +++ markup/restructuredtext.py 2015-10-14 01:30:07.174851600 +0200 3 | @@ -304,7 +304,11 @@ 4 | # Extract the first sentence. 5 | for child in node: 6 | if isinstance(child, docutils.nodes.Text): 7 | - m = self._SUMMARY_RE.match(child.data) 8 | + try: 9 | + m = self._SUMMARY_RE.match(child.data) 10 | + except AttributeError: 11 | + m = None 12 | + 13 | if m: 14 | summary_pieces.append(docutils.nodes.Text(m.group(1))) 15 | other = child.data[m.end():] 16 | -------------------------------------------------------------------------------- /misc/test_threadworker.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | # 4 | # test_threadworker.py 5 | # 6 | 7 | try: # Python 2: 8 | import sha 9 | def sha1_new(): return sha.new() 10 | except ImportError: # Python 3: 11 | import hashlib 12 | def sha1_new(): return hashlib.sha1() 13 | 14 | import threadpool as tp 15 | 16 | 17 | def getsha(dummy, data): 18 | m = sha1_new() 19 | m.update(data) 20 | return m.hexdigest() 21 | 22 | def print_result(request, result): 23 | print('"%s" -->\t%s' % (os.path.basename(request.args[0]), result)) 24 | 25 | 26 | if __name__ == '__main__': 27 | import os, sys 28 | 29 | try: 30 | topdir = sys.argv[1] 31 | except IndexError: 32 | topdir = os.curdir 33 | 34 | files = [] 35 | for dirpath, dirname, filenames in os.walk(topdir): 36 | for filename in filenames: 37 | filename = os.path.join(dirpath, filename) 38 | if os.path.isfile(filename): 39 | files.append(filename, ) 40 | 41 | files.sort() 42 | main = tp.ThreadPool(10, q_size=50) 43 | 44 | for file in files: 45 | # Bad approach: 46 | ## main.putRequest(WorkRequest(getmd5, args=(file, None), 47 | ## callback=print_result)) 48 | 49 | # Better approach: 50 | main.putRequest(tp.WorkRequest(getsha, 51 | args=(file, open(file, 'rb').read()), 52 | callback=print_result)) 53 | try: 54 | main.poll() 55 | except tp.NoResultsPending: 56 | pass 57 | 58 | main.wait() 59 | -------------------------------------------------------------------------------- /requirements-dev.txt: -------------------------------------------------------------------------------- 1 | pytest 2 | tox 3 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [egg_info] 2 | tag_build = .dev 3 | #tag_date = true 4 | 5 | [aliases] 6 | # A handy alias to build a release (source and egg) 7 | release = build egg_info -RDb "" sdist --formats=zip,gztar,bztar bdist_wheel 8 | # A handy alias to upload a release to pypi 9 | release_upload = build egg_info -RDb "" sdist --formats=zip,gztar,bztar bdist_wheel upload 10 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | 4 | from setuptools import setup 5 | 6 | 7 | with open('src/release.py') as f: 8 | exec(compile(f.read(), 'src/release.py', 'exec')) 9 | 10 | setup( 11 | name=name, 12 | version=version, 13 | description=description, 14 | long_description=long_description, 15 | keywords=keywords, 16 | author=author, 17 | author_email=author_email, 18 | license=license, 19 | url=url, 20 | download_url=download_url, 21 | classifiers=classifiers, 22 | platforms=platforms, 23 | py_modules = ['threadpool'], 24 | package_dir = {'': 'src'}, 25 | ) 26 | -------------------------------------------------------------------------------- /src/release.py: -------------------------------------------------------------------------------- 1 | # -*- coding: UTF-8 -*- 2 | """Easy to use object-oriented thread pool framework. 3 | 4 | .. warning:: 5 | This module is **OBSOLETE** and is only provided on PyPI to support old 6 | projects that still use it. Please **DO NOT USE IT FOR NEW PROJECTS!** 7 | 8 | Use modern alternatives like the `multiprocessing `_ 9 | module in the standard library or even an asynchroneous approach with 10 | `asyncio <_asyncio: https://docs.python.org/3/library/asyncio.html>`_. 11 | 12 | A thread pool is an object that maintains a pool of worker threads to perform 13 | time consuming operations in parallel. It assigns jobs to the threads 14 | by putting them in a work request queue, where they are picked up by the 15 | next available thread. This then performs the requested operation in the 16 | background and puts the results in another queue. 17 | 18 | The thread pool object can then collect the results from all threads from 19 | this queue as soon as they become available or after all threads have 20 | finished their work. It's also possible, to define callbacks to handle 21 | each result as it comes in. 22 | 23 | .. note:: 24 | This module is regarded as an extended example, not as a finished product. 25 | Feel free to adapt it too your needs. 26 | 27 | """ 28 | # Release info for Threadpool 29 | 30 | name = 'threadpool' 31 | version = '1.3.2' 32 | description = __doc__.splitlines()[0] 33 | keywords = 'threads, design pattern, thread pool' 34 | author = 'Christopher Arndt' 35 | author_email = 'chris@chrisarndt.de' 36 | url = 'http://chrisarndt.de/projects/threadpool/' 37 | download_url = url + 'download/' 38 | license = "MIT license" 39 | long_description = "".join(__doc__.splitlines()[2:]) 40 | platforms = "POSIX, Windows, MacOS X" 41 | classifiers = [ 42 | 'Development Status :: 5 - Production/Stable', 43 | 'Intended Audience :: Developers', 44 | 'License :: OSI Approved :: Python Software Foundation License', 45 | 'Operating System :: Microsoft :: Windows', 46 | 'Operating System :: POSIX', 47 | 'Operating System :: MacOS :: MacOS X', 48 | 'Programming Language :: Python', 49 | 'Programming Language :: Python :: 2', 50 | 'Programming Language :: Python :: 3', 51 | 'Topic :: Software Development :: Libraries :: Python Modules' 52 | ] 53 | -------------------------------------------------------------------------------- /src/threadpool.py: -------------------------------------------------------------------------------- 1 | # -*- coding: UTF-8 -*- 2 | """Easy to use object-oriented thread pool framework. 3 | 4 | A thread pool is an object that maintains a pool of worker threads to perform 5 | time consuming operations in parallel. It assigns jobs to the threads 6 | by putting them in a work request queue, where they are picked up by the 7 | next available thread. This then performs the requested operation in the 8 | background and puts the results in another queue. 9 | 10 | The thread pool object can then collect the results from all threads from 11 | this queue as soon as they become available or after all threads have 12 | finished their work. It's also possible, to define callbacks to handle 13 | each result as it comes in. 14 | 15 | The basic concept and some code was taken from the book "Python in a Nutshell, 16 | 2nd edition" by Alex Martelli, O'Reilly 2006, ISBN 0-596-10046-9, from section 17 | 14.5 "Threaded Program Architecture". I wrapped the main program logic in the 18 | ThreadPool class, added the WorkRequest class and the callback system and 19 | tweaked the code here and there. Kudos also to Florent Aide for the exception 20 | handling mechanism. 21 | 22 | Basic usage:: 23 | 24 | >>> pool = ThreadPool(poolsize) 25 | >>> requests = makeRequests(some_callable, list_of_args, callback) 26 | >>> [pool.putRequest(req) for req in requests] 27 | >>> pool.wait() 28 | 29 | See the end of the module code for a brief, annotated usage example. 30 | 31 | Website : http://chrisarndt.de/projects/threadpool/ 32 | 33 | """ 34 | __docformat__ = "restructuredtext en" 35 | 36 | __all__ = [ 37 | 'makeRequests', 38 | 'NoResultsPending', 39 | 'NoWorkersAvailable', 40 | 'ThreadPool', 41 | 'WorkRequest', 42 | 'WorkerThread' 43 | ] 44 | 45 | __author__ = "Christopher Arndt" 46 | __version__ = '1.3.2' 47 | __license__ = "MIT license" 48 | 49 | 50 | # standard library modules 51 | import sys 52 | import threading 53 | import traceback 54 | 55 | try: 56 | import Queue # Python 2 57 | except ImportError: 58 | import queue as Queue # Python 3 59 | 60 | 61 | # exceptions 62 | class NoResultsPending(Exception): 63 | """All work requests have been processed.""" 64 | pass 65 | 66 | class NoWorkersAvailable(Exception): 67 | """No worker threads available to process remaining requests.""" 68 | pass 69 | 70 | 71 | # internal module helper functions 72 | def _handle_thread_exception(request, exc_info): 73 | """Default exception handler callback function. 74 | 75 | This just prints the exception info via ``traceback.print_exception``. 76 | 77 | """ 78 | traceback.print_exception(*exc_info) 79 | 80 | 81 | # utility functions 82 | def makeRequests(callable_, args_list, callback=None, 83 | exc_callback=_handle_thread_exception): 84 | """Create several work requests for same callable with different arguments. 85 | 86 | Convenience function for creating several work requests for the same 87 | callable where each invocation of the callable receives different values 88 | for its arguments. 89 | 90 | ``args_list`` contains the parameters for each invocation of callable. 91 | Each item in ``args_list`` should be either a 2-item tuple of the list of 92 | positional arguments and a dictionary of keyword arguments or a single, 93 | non-tuple argument. 94 | 95 | See docstring for ``WorkRequest`` for info on ``callback`` and 96 | ``exc_callback``. 97 | 98 | """ 99 | requests = [] 100 | for item in args_list: 101 | if isinstance(item, tuple): 102 | requests.append( 103 | WorkRequest(callable_, item[0], item[1], callback=callback, 104 | exc_callback=exc_callback) 105 | ) 106 | else: 107 | requests.append( 108 | WorkRequest(callable_, [item], None, callback=callback, 109 | exc_callback=exc_callback) 110 | ) 111 | return requests 112 | 113 | 114 | # classes 115 | class WorkerThread(threading.Thread): 116 | """Background thread connected to the requests/results queues. 117 | 118 | A worker thread sits in the background and picks up work requests from 119 | one queue and puts the results in another until it is dismissed. 120 | 121 | """ 122 | 123 | def __init__(self, requests_queue, results_queue, poll_timeout=5, **kwds): 124 | """Set up thread in daemonic mode and start it immediatedly. 125 | 126 | ``requests_queue`` and ``results_queue`` are instances of 127 | ``Queue.Queue`` passed by the ``ThreadPool`` class when it creates a 128 | new worker thread. 129 | 130 | """ 131 | threading.Thread.__init__(self, **kwds) 132 | self.setDaemon(1) 133 | self._requests_queue = requests_queue 134 | self._results_queue = results_queue 135 | self._poll_timeout = poll_timeout 136 | self._dismissed = threading.Event() 137 | self.start() 138 | 139 | def run(self): 140 | """Repeatedly process the job queue until told to exit.""" 141 | while True: 142 | if self._dismissed.isSet(): 143 | # we are dismissed, break out of loop 144 | break 145 | # get next work request. If we don't get a new request from the 146 | # queue after self._poll_timout seconds, we jump to the start of 147 | # the while loop again, to give the thread a chance to exit. 148 | try: 149 | request = self._requests_queue.get(True, self._poll_timeout) 150 | except Queue.Empty: 151 | continue 152 | else: 153 | if self._dismissed.isSet(): 154 | # we are dismissed, put back request in queue and exit loop 155 | self._requests_queue.put(request) 156 | break 157 | try: 158 | result = request.callable(*request.args, **request.kwds) 159 | self._results_queue.put((request, result)) 160 | except: 161 | request.exception = True 162 | self._results_queue.put((request, sys.exc_info())) 163 | 164 | def dismiss(self): 165 | """Sets a flag to tell the thread to exit when done with current job. 166 | """ 167 | self._dismissed.set() 168 | 169 | 170 | class WorkRequest: 171 | """A request to execute a callable for putting in the request queue later. 172 | 173 | See the module function ``makeRequests`` for the common case 174 | where you want to build several ``WorkRequest`` objects for the same 175 | callable but with different arguments for each call. 176 | 177 | """ 178 | 179 | def __init__(self, callable_, args=None, kwds=None, requestID=None, 180 | callback=None, exc_callback=_handle_thread_exception): 181 | """Create a work request for a callable and attach callbacks. 182 | 183 | A work request consists of the a callable to be executed by a 184 | worker thread, a list of positional arguments, a dictionary 185 | of keyword arguments. 186 | 187 | A ``callback`` function can be specified, that is called when the 188 | results of the request are picked up from the result queue. It must 189 | accept two anonymous arguments, the ``WorkRequest`` object and the 190 | results of the callable, in that order. If you want to pass additional 191 | information to the callback, just stick it on the request object. 192 | 193 | You can also give custom callback for when an exception occurs with 194 | the ``exc_callback`` keyword parameter. It should also accept two 195 | anonymous arguments, the ``WorkRequest`` and a tuple with the exception 196 | details as returned by ``sys.exc_info()``. The default implementation 197 | of this callback just prints the exception info via 198 | ``traceback.print_exception``. If you want no exception handler 199 | callback, just pass in ``None``. 200 | 201 | ``requestID``, if given, must be hashable since it is used by 202 | ``ThreadPool`` object to store the results of that work request in a 203 | dictionary. It defaults to the return value of ``id(self)``. 204 | 205 | """ 206 | if requestID is None: 207 | self.requestID = id(self) 208 | else: 209 | try: 210 | self.requestID = hash(requestID) 211 | except TypeError: 212 | raise TypeError("requestID must be hashable.") 213 | self.exception = False 214 | self.callback = callback 215 | self.exc_callback = exc_callback 216 | self.callable = callable_ 217 | self.args = args or [] 218 | self.kwds = kwds or {} 219 | 220 | def __str__(self): 221 | return "" % \ 222 | (self.requestID, self.args, self.kwds, self.exception) 223 | 224 | class ThreadPool: 225 | """A thread pool, distributing work requests and collecting results. 226 | 227 | See the module docstring for more information. 228 | 229 | """ 230 | 231 | def __init__(self, num_workers, q_size=0, resq_size=0, poll_timeout=5): 232 | """Set up the thread pool and start num_workers worker threads. 233 | 234 | ``num_workers`` is the number of worker threads to start initially. 235 | 236 | If ``q_size > 0`` the size of the work *request queue* is limited and 237 | the thread pool blocks when the queue is full and it tries to put 238 | more work requests in it (see ``putRequest`` method), unless you also 239 | use a positive ``timeout`` value for ``putRequest``. 240 | 241 | If ``resq_size > 0`` the size of the *results queue* is limited and the 242 | worker threads will block when the queue is full and they try to put 243 | new results in it. 244 | 245 | .. warning: 246 | If you set both ``q_size`` and ``resq_size`` to ``!= 0`` there is 247 | the possibilty of a deadlock, when the results queue is not pulled 248 | regularly and too many jobs are put in the work requests queue. 249 | To prevent this, always set ``timeout > 0`` when calling 250 | ``ThreadPool.putRequest()`` and catch ``Queue.Full`` exceptions. 251 | 252 | """ 253 | self._requests_queue = Queue.Queue(q_size) 254 | self._results_queue = Queue.Queue(resq_size) 255 | self.workers = [] 256 | self.dismissedWorkers = [] 257 | self.workRequests = {} 258 | self.createWorkers(num_workers, poll_timeout) 259 | 260 | def createWorkers(self, num_workers, poll_timeout=5): 261 | """Add num_workers worker threads to the pool. 262 | 263 | ``poll_timout`` sets the interval in seconds (int or float) for how 264 | ofte threads should check whether they are dismissed, while waiting for 265 | requests. 266 | 267 | """ 268 | for i in range(num_workers): 269 | self.workers.append(WorkerThread(self._requests_queue, 270 | self._results_queue, poll_timeout=poll_timeout)) 271 | 272 | def dismissWorkers(self, num_workers, do_join=False): 273 | """Tell num_workers worker threads to quit after their current task.""" 274 | dismiss_list = [] 275 | for i in range(min(num_workers, len(self.workers))): 276 | worker = self.workers.pop() 277 | worker.dismiss() 278 | dismiss_list.append(worker) 279 | 280 | if do_join: 281 | for worker in dismiss_list: 282 | worker.join() 283 | else: 284 | self.dismissedWorkers.extend(dismiss_list) 285 | 286 | def joinAllDismissedWorkers(self): 287 | """Perform Thread.join() on all worker threads that have been dismissed. 288 | """ 289 | for worker in self.dismissedWorkers: 290 | worker.join() 291 | self.dismissedWorkers = [] 292 | 293 | def putRequest(self, request, block=True, timeout=None): 294 | """Put work request into work queue and save its id for later.""" 295 | assert isinstance(request, WorkRequest) 296 | # don't reuse old work requests 297 | assert not getattr(request, 'exception', None) 298 | self._requests_queue.put(request, block, timeout) 299 | self.workRequests[request.requestID] = request 300 | 301 | def poll(self, block=False): 302 | """Process any new results in the queue.""" 303 | while True: 304 | # still results pending? 305 | if not self.workRequests: 306 | raise NoResultsPending 307 | # are there still workers to process remaining requests? 308 | elif block and not self.workers: 309 | raise NoWorkersAvailable 310 | try: 311 | # get back next results 312 | request, result = self._results_queue.get(block=block) 313 | # has an exception occured? 314 | if request.exception and request.exc_callback: 315 | request.exc_callback(request, result) 316 | # hand results to callback, if any 317 | if request.callback and not \ 318 | (request.exception and request.exc_callback): 319 | request.callback(request, result) 320 | del self.workRequests[request.requestID] 321 | except Queue.Empty: 322 | break 323 | 324 | def wait(self): 325 | """Wait for results, blocking until all have arrived.""" 326 | while 1: 327 | try: 328 | self.poll(True) 329 | except NoResultsPending: 330 | break 331 | 332 | 333 | ################ 334 | # USAGE EXAMPLE 335 | ################ 336 | 337 | if __name__ == '__main__': 338 | import random 339 | import time 340 | 341 | # the work the threads will have to do (rather trivial in our example) 342 | def do_something(data): 343 | time.sleep(random.randint(1,5)) 344 | result = round(random.random() * data, 5) 345 | # just to show off, we throw an exception once in a while 346 | if result > 5: 347 | raise RuntimeError("Something extraordinary happened!") 348 | return result 349 | 350 | # this will be called each time a result is available 351 | def print_result(request, result): 352 | print("**** Result from request #%s: %r" % (request.requestID, result)) 353 | 354 | # this will be called when an exception occurs within a thread 355 | # this example exception handler does little more than the default handler 356 | def handle_exception(request, exc_info): 357 | if not isinstance(exc_info, tuple): 358 | # Something is seriously wrong... 359 | print(request) 360 | print(exc_info) 361 | raise SystemExit 362 | print("**** Exception occured in request #%s: %s" % \ 363 | (request.requestID, exc_info)) 364 | 365 | # assemble the arguments for each job to a list... 366 | data = [random.randint(1,10) for i in range(20)] 367 | # ... and build a WorkRequest object for each item in data 368 | requests = makeRequests(do_something, data, print_result, handle_exception) 369 | # to use the default exception handler, uncomment next line and comment out 370 | # the preceding one. 371 | #requests = makeRequests(do_something, data, print_result) 372 | 373 | # or the other form of args_lists accepted by makeRequests: ((,), {}) 374 | data = [((random.randint(1,10),), {}) for i in range(20)] 375 | requests.extend( 376 | makeRequests(do_something, data, print_result, handle_exception) 377 | #makeRequests(do_something, data, print_result) 378 | # to use the default exception handler, uncomment next line and comment 379 | # out the preceding one. 380 | ) 381 | 382 | # we create a pool of 3 worker threads 383 | print("Creating thread pool with 3 worker threads.") 384 | main = ThreadPool(3) 385 | 386 | # then we put the work requests in the queue... 387 | for req in requests: 388 | main.putRequest(req) 389 | print("Work request #%s added." % req.requestID) 390 | # or shorter: 391 | # [main.putRequest(req) for req in requests] 392 | 393 | # ...and wait for the results to arrive in the result queue 394 | # by using ThreadPool.wait(). This would block until results for 395 | # all work requests have arrived: 396 | # main.wait() 397 | 398 | # instead we can poll for results while doing something else: 399 | i = 0 400 | while True: 401 | try: 402 | time.sleep(0.5) 403 | main.poll() 404 | print("Main thread working...") 405 | print("(active worker threads: %i)" % (threading.activeCount()-1, )) 406 | if i == 10: 407 | print("**** Adding 3 more worker threads...") 408 | main.createWorkers(3) 409 | if i == 20: 410 | print("**** Dismissing 2 worker threads...") 411 | main.dismissWorkers(2) 412 | i += 1 413 | except KeyboardInterrupt: 414 | print("**** Interrupted!") 415 | break 416 | except NoResultsPending: 417 | print("**** No pending results.") 418 | break 419 | if main.dismissedWorkers: 420 | print("Joining all dismissed worker threads...") 421 | main.joinAllDismissedWorkers() 422 | -------------------------------------------------------------------------------- /test/test_issue1.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | 4 | import threading 5 | import pytest 6 | 7 | import threadpool 8 | 9 | 10 | def run_threads(threadnum, func, args, callback=None): 11 | """Convenience wrappet to spawn multiple threads""" 12 | def exp_callback(request, exc_info): 13 | pass 14 | 15 | pool = threadpool.ThreadPool(threadnum) 16 | 17 | for req in threadpool.makeRequests(func, args, callback, exp_callback): 18 | pool.putRequest(req) 19 | 20 | while True: 21 | try: 22 | pool.poll() 23 | except KeyboardInterrupt: 24 | break 25 | except threadpool.NoResultsPending: 26 | break 27 | 28 | if pool.dismissedWorkers: 29 | pool.joinAllDismissedWorkers() 30 | 31 | 32 | @pytest.mark.xfail(raises=AttributeError) 33 | def test_run_threads(): 34 | num_threads = 3 35 | num_calls = 10000 36 | results = [] 37 | lock = threading.Lock() 38 | 39 | def cb(req, res): 40 | with lock: 41 | results.append(res) 42 | 43 | def func(i): 44 | return i 45 | 46 | run_threads(num_threads, func, list(range(num_calls)), cb) 47 | assert len(results) == num_calls 48 | 49 | 50 | def task1(arg): 51 | print(arg) 52 | 53 | 54 | def dispatcher(arg): 55 | args = list(range(50)) 56 | run_threads(3, task1, args) 57 | 58 | 59 | def test_threads_run_threads(): 60 | args = list(range(50)) 61 | run_threads(3, dispatcher, args) 62 | 63 | 64 | if __name__ == '__main__': 65 | test_run_threads() 66 | -------------------------------------------------------------------------------- /tools/make_release.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # make_release.sh - automates steps to build a threadpool release 4 | 5 | # generates documentation files and packages distribution archive 6 | 7 | if [ "x$1" = "x-f" ]; then 8 | FINAL=yes 9 | shift 10 | fi 11 | 12 | echo "Before you go ahead, check that the version numbers in README.rst, " 13 | echo "src/threadpooly.py and src/release.py are correct!" 14 | echo 15 | echo "Press ENTER to continue, Ctrl-C to abort..." 16 | read 17 | echo 18 | 19 | GIT_URL="git@github.com:SpotlightKid/threadpool.git" 20 | PROJECT_NAME=$(python2 -c 'execfile("src/release.py"); print name') 21 | VERSION=$(python2 -c 'execfile("src/release.py"); print version') 22 | HOMEPAGE=$(python2 -c 'execfile("src/release.py"); print url') 23 | 24 | VENV="./venv" 25 | RST2HTML_OPTS='--stylesheet-path=rest.css --link-stylesheet --input-encoding=UTF-8 --output-encoding=UTF-8 --language=en --no-xml-declaration --date --time' 26 | 27 | if [ ! -d "$VENV" ]; then 28 | virtualenv -p python2.7 --no-site-packages "$VENV" 29 | source "$VENV/bin/activate" 30 | pip install Pygments docutils "epydoc>3.0" wheel 31 | cwd="$(pwd)" 32 | ( cd $VENV/lib/python2.7/site-packages/epydoc ; patch -p0 -i $cwd/misc/epydoc.patch ; ) 33 | else 34 | source "$VENV/bin/activate" 35 | fi 36 | 37 | 38 | # Create HTML file with syntax highlighted source 39 | echo "Making colorized source code HTML page..." 40 | pygmentize -P full -P cssfile=hilight.css -P title=threadpool.py \ 41 | -o doc/threadpool.py.html src/threadpool.py 42 | # Create API documentation 43 | echo "Generating API documentation with epydoc..." 44 | epydoc --debug -v -n Threadpool -o doc/api \ 45 | --url "$HOMEPAGE" \ 46 | --no-private --docformat restructuredtext \ 47 | src/threadpool.py 48 | # Create HTMl version of README 49 | echo "Creating HTML version of README..." 50 | rst2html $RST2HTML_OPTS README.rst >doc/index.html 51 | 52 | # Build distribution packages 53 | if [ "x$FINAL" != "xyes" ]; then 54 | python setup.py bdist_egg bdist_wheel sdist --formats=zip,bztar 55 | if [ "x$1" = "xupload" ]; then 56 | ./tools/upload.sh 57 | fi 58 | else 59 | # Check if everything is commited 60 | GIT_STATUS=$(git status -s) 61 | if [ -n "$GIT_STATUS" ]; then 62 | echo "Git is not up to date. Please fix." 2>&1 63 | exit 1 64 | fi 65 | 66 | # and upload & register them at the Cheeseshop if "-f" option is given 67 | python setup.py egg_info -RDb "" bdist_egg bdist_wheel sdist \ 68 | --formats=zip,bztar register upload 69 | ret=$? 70 | # tag release in the Git repo 71 | if [ $ret -eq 0 ]; then 72 | git tag -m "Tagging $PROJECT_NAME release $VERSION" $VERSION 73 | fi 74 | # update web site 75 | ./tools/upload.sh -f 76 | fi 77 | -------------------------------------------------------------------------------- /tools/upload.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | BASE_DIR="/home/www/chrisarndt.de/htdocs/projects/threadpool" 4 | HOST="chrisarndt.de" 5 | USER="chris" 6 | 7 | if [ "x$1" != "x-f" ]; then 8 | RSYNC_OPTS="-n" 9 | fi 10 | 11 | # upload API docs and index.html 12 | rsync $RSYNC_OPTS -av --update --delete \ 13 | --exclude=download \ 14 | --exclude=.svn \ 15 | --exclude=.git \ 16 | --exclude=.DS_Store \ 17 | doc/ "$USER@$HOST:$BASE_DIR" 18 | 19 | # Upload distribution packages 20 | rsync $RSYNC_OPTS -av --update \ 21 | "--exclude=*.dev*" \ 22 | --exclude=.DS_Store \ 23 | dist/ "$USER@$HOST:$BASE_DIR/download" 24 | -------------------------------------------------------------------------------- /tox.ini: -------------------------------------------------------------------------------- 1 | [tox] 2 | envlist = py26,py27,py33,py34,py35 3 | 4 | [testenv] 5 | deps = pytest 6 | commands = py.test -vs test 7 | usedevelop = True 8 | setenv = 9 | VIRTUALENV_NO_WHEEL = 1 10 | 11 | --------------------------------------------------------------------------------