├── .github └── workflows │ ├── release.yml │ └── test.yml ├── .gitignore ├── AUTHORS.rst ├── LICENSE ├── MANIFEST.in ├── README.rst ├── grequests.py ├── requirements.txt ├── setup.py └── tests.py /.github/workflows/release.yml: -------------------------------------------------------------------------------- 1 | name: release 2 | on: 3 | push: 4 | tags: 5 | - 'v*.*.*' 6 | 7 | jobs: 8 | build: 9 | runs-on: ubuntu-latest 10 | steps: 11 | - name: Checkout 12 | uses: actions/checkout@v2 13 | - name: setup python 14 | uses: actions/setup-python@v2 15 | with: 16 | python-version: 3.9 17 | 18 | - name: build 19 | shell: bash 20 | run: | 21 | python -m pip install --upgrade wheel gevent 22 | python setup.py sdist bdist_wheel --universal 23 | - name: Release PyPI 24 | shell: bash 25 | env: 26 | TWINE_USERNAME: ${{ secrets.TWINE_USERNAME }} 27 | TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }} 28 | run: | 29 | pip install --upgrade twine 30 | twine upload dist/* 31 | 32 | - name: Release GitHub 33 | uses: softprops/action-gh-release@v1 34 | with: 35 | files: "dist/*" 36 | env: 37 | GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} -------------------------------------------------------------------------------- /.github/workflows/test.yml: -------------------------------------------------------------------------------- 1 | on: [ push, pull_request ] 2 | 3 | jobs: 4 | test: 5 | runs-on: ubuntu-latest 6 | steps: 7 | - name: Checkout 8 | uses: actions/checkout@v2 9 | - name: Setup Python 10 | uses: actions/setup-python@v2 11 | with: 12 | python-version: 3.9 13 | 14 | - name: Install dependencies 15 | run: | 16 | python -m pip install --upgrade pip 17 | python -m pip install gevent requests pytest 18 | - name: Test with pytest 19 | run: | 20 | pytest tests.py 21 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .coverage 2 | MANIFEST 3 | coverage.xml 4 | junit-report.xml 5 | pylint.txt 6 | toy.py 7 | violations.pyflakes.txt 8 | cover/ 9 | docs/_build 10 | grequests.egg-info/ 11 | *.py[cx] 12 | *.swp 13 | env/ 14 | -------------------------------------------------------------------------------- /AUTHORS.rst: -------------------------------------------------------------------------------- 1 | GRequests is authored by Kenneth Reitz, maintained by Spencer Young and 2 | various contributors: 3 | 4 | Development Leads 5 | ````````````````` 6 | 7 | - Spencer Phillip Young 8 | - Kenneth Reitz 9 | 10 | Patches and Suggestions 11 | ``````````````````````` 12 | 13 | Adam Tauber 14 | Akshat Mahajan 15 | Alexander Simeonov 16 | Antonio A 17 | Chris Drackett 18 | Eugene Eeo 19 | Frost Ming 20 | Ian Cordasco 21 | Joe Gordon 22 | Luke Hutscal 23 | Marc Abramowitz 24 | Mathieu Lecarme 25 | Michael Newman 26 | Mircea Ulinic 27 | Nate Lawson 28 | Nathan Hoad 29 | Roman Haritonov 30 | Ryan T. Dean 31 | Spencer Phillip Young 32 | Spencer Young 33 | Yuri Prezument 34 | koobs 35 | kracekumar 36 | 崔庆才丨静觅 37 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2012, Kenneth Reitz 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 5 | 6 | Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 7 | Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 8 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include LICENSE 2 | include README.rst 3 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | GRequests: Asynchronous Requests 2 | =============================== 3 | 4 | GRequests allows you to use Requests with Gevent to make asynchronous HTTP 5 | Requests easily. 6 | 7 | |version| |pyversions| 8 | 9 | 10 | 11 | Installation 12 | ------------ 13 | 14 | Installation is easy with pip:: 15 | 16 | $ pip install grequests 17 | ✨🍰✨ 18 | 19 | 20 | Usage 21 | ----- 22 | 23 | Usage is simple: 24 | 25 | .. code-block:: python 26 | 27 | import grequests 28 | 29 | urls = [ 30 | 'http://www.heroku.com', 31 | 'http://python-tablib.org', 32 | 'http://httpbin.org', 33 | 'http://python-requests.org', 34 | 'http://fakedomain/', 35 | 'http://kennethreitz.com' 36 | ] 37 | 38 | Create a set of unsent Requests: 39 | 40 | .. code-block:: python 41 | 42 | >>> rs = (grequests.get(u) for u in urls) 43 | 44 | Send them all at the same time using ``map``: 45 | 46 | .. code-block:: python 47 | 48 | >>> grequests.map(rs) 49 | [, , , , None, ] 50 | 51 | 52 | The HTTP verb methods in ``grequests`` (e.g., ``grequests.get``, ``grequests.post``, etc.) accept all the same keyword arguments as in the ``requests`` library. 53 | 54 | Error Handling 55 | ^^^^^^^^^^^^^^ 56 | 57 | To handle timeouts or any other exception during the connection of 58 | the request, you can add an optional exception handler that will be called with the request and 59 | exception inside the main thread. The value returned by your exception handler will be used in the result list returned by ``map``. 60 | 61 | 62 | .. code-block:: python 63 | 64 | >>> def exception_handler(request, exception): 65 | ... print("Request failed") 66 | 67 | >>> reqs = [ 68 | ... grequests.get('http://httpbin.org/delay/1', timeout=0.001), 69 | ... grequests.get('http://fakedomain/'), 70 | ... grequests.get('http://httpbin.org/status/500')] 71 | >>> grequests.map(reqs, exception_handler=exception_handler) 72 | Request failed 73 | Request failed 74 | [None, None, ] 75 | 76 | 77 | imap 78 | ^^^^ 79 | 80 | For some speed/performance gains, you may also want to use ``imap`` instead of ``map``. ``imap`` returns a generator of responses. Order of these responses does not map to the order of the requests you send out. The API for ``imap`` is equivalent to the API for ``map``. You can also adjust the ``size`` argument to ``map`` or ``imap`` to increase the gevent pool size. 81 | 82 | 83 | .. code-block:: python 84 | 85 | for resp in grequests.imap(reqs, size=10): 86 | print(resp) 87 | 88 | 89 | There is also an enumerated version of ``imap``, ``imap_enumerated`` which yields the index of the request from the original request list and its associated response. However, unlike ``imap``, failed requests and exception handler results that return ``None`` will also be yielded (whereas in ``imap`` they are ignored). Aditionally, the ``requests`` parameter for ``imap_enumerated`` must be a sequence. Like in ``imap``, the order in which requests are sent and received should still be considered arbitrary. 90 | 91 | .. code-block:: python 92 | 93 | >>> rs = [grequests.get(f'https://httpbin.org/status/{code}') for code in range(200, 206)] 94 | >>> for index, response in grequests.imap_enumerated(rs, size=5): 95 | ... print(index, response) 96 | 1 97 | 0 98 | 4 99 | 2 100 | 5 101 | 3 102 | 103 | gevent - when things go wrong 104 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 105 | 106 | Because ``grequests`` leverages ``gevent`` (which in turn uses monkeypatching for enabling concurrency), you will often need to make sure ``grequests`` is imported before other libraries, especially ``requests``, to avoid problems. See `grequests gevent issues `_ for additional information. 107 | 108 | 109 | .. code-block:: python 110 | 111 | # GOOD 112 | import grequests 113 | import requests 114 | 115 | # BAD 116 | import requests 117 | import grequests 118 | 119 | 120 | 121 | 122 | 123 | 124 | 125 | .. |version| image:: https://img.shields.io/pypi/v/grequests.svg?colorB=blue 126 | :target: https://pypi.org/project/grequests/ 127 | 128 | .. |pyversions| image:: https://img.shields.io/pypi/pyversions/grequests.svg? 129 | :target: https://pypi.org/project/grequests/ 130 | 131 | 132 | -------------------------------------------------------------------------------- /grequests.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | """ 4 | grequests 5 | ~~~~~~~~~ 6 | 7 | This module contains an asynchronous replica of ``requests.api``, powered 8 | by gevent. All API methods return a ``Request`` instance (as opposed to 9 | ``Response``). A list of requests can be sent with ``map()``. 10 | """ 11 | from functools import partial 12 | import traceback 13 | 14 | try: 15 | import gevent 16 | from gevent import monkey as curious_george 17 | from gevent.pool import Pool 18 | except ImportError: 19 | raise RuntimeError('Gevent is required for grequests.') 20 | 21 | # Monkey-patch. 22 | curious_george.patch_all(thread=False, select=False) 23 | 24 | from requests import Session 25 | 26 | __all__ = ( 27 | 'map', 'imap', 28 | 'get', 'options', 'head', 'post', 'put', 'patch', 'delete', 'request' 29 | ) 30 | 31 | 32 | class AsyncRequest(object): 33 | """ Asynchronous request. 34 | 35 | Accept same parameters as ``Session.request`` and some additional: 36 | 37 | :param session: Session which will do request 38 | :param callback: Callback called on response. 39 | Same as passing ``hooks={'response': callback}`` 40 | """ 41 | def __init__(self, method, url, **kwargs): 42 | #: Request method 43 | self.method = method 44 | #: URL to request 45 | self.url = url 46 | #: Associated ``Session`` 47 | self.session = kwargs.pop('session', None) 48 | if self.session is None: 49 | self.session = Session() 50 | self._close = True 51 | else: 52 | self._close = False # don't close adapters after each request if the user provided the session 53 | 54 | callback = kwargs.pop('callback', None) 55 | if callback: 56 | kwargs['hooks'] = {'response': callback} 57 | 58 | #: The rest arguments for ``Session.request`` 59 | self.kwargs = kwargs 60 | #: Resulting ``Response`` 61 | self.response = None 62 | 63 | def send(self, **kwargs): 64 | """ 65 | Prepares request based on parameter passed to constructor and optional ``kwargs```. 66 | Then sends request and saves response to :attr:`response` 67 | 68 | :returns: ``Response`` 69 | """ 70 | merged_kwargs = {} 71 | merged_kwargs.update(self.kwargs) 72 | merged_kwargs.update(kwargs) 73 | try: 74 | self.response = self.session.request(self.method, 75 | self.url, **merged_kwargs) 76 | except Exception as e: 77 | self.exception = e 78 | self.traceback = traceback.format_exc() 79 | finally: 80 | if self._close: 81 | # if we provided the session object, make sure we're cleaning up 82 | # because there's no sense in keeping it open at this point if it wont be reused 83 | self.session.close() 84 | return self 85 | 86 | 87 | def send(r, pool=None, stream=False): 88 | """Sends the request object using the specified pool. If a pool isn't 89 | specified this method blocks. Pools are useful because you can specify size 90 | and can hence limit concurrency.""" 91 | if pool is not None: 92 | return pool.spawn(r.send, stream=stream) 93 | 94 | return gevent.spawn(r.send, stream=stream) 95 | 96 | 97 | # Shortcuts for creating AsyncRequest with appropriate HTTP method 98 | get = partial(AsyncRequest, 'GET') 99 | options = partial(AsyncRequest, 'OPTIONS') 100 | head = partial(AsyncRequest, 'HEAD') 101 | post = partial(AsyncRequest, 'POST') 102 | put = partial(AsyncRequest, 'PUT') 103 | patch = partial(AsyncRequest, 'PATCH') 104 | delete = partial(AsyncRequest, 'DELETE') 105 | 106 | # synonym 107 | def request(method, url, **kwargs): 108 | return AsyncRequest(method, url, **kwargs) 109 | 110 | 111 | def map(requests, stream=False, size=None, exception_handler=None, gtimeout=None): 112 | """Concurrently converts a list of Requests to Responses. 113 | 114 | :param requests: a collection of Request objects. 115 | :param stream: If True, the content will not be downloaded immediately. 116 | :param size: Specifies the number of requests to make at a time. If None, no throttling occurs. 117 | :param exception_handler: Callback function, called when exception occured. Params: Request, Exception 118 | :param gtimeout: Gevent joinall timeout in seconds. (Note: unrelated to requests timeout) 119 | """ 120 | 121 | requests = list(requests) 122 | 123 | pool = Pool(size) if size else None 124 | jobs = [send(r, pool, stream=stream) for r in requests] 125 | gevent.joinall(jobs, timeout=gtimeout) 126 | 127 | ret = [] 128 | 129 | for request in requests: 130 | if request.response is not None: 131 | ret.append(request.response) 132 | elif exception_handler and hasattr(request, 'exception'): 133 | ret.append(exception_handler(request, request.exception)) 134 | elif exception_handler and not hasattr(request, 'exception'): 135 | ret.append(exception_handler(request, None)) 136 | else: 137 | ret.append(None) 138 | 139 | return ret 140 | 141 | 142 | def imap(requests, stream=False, size=2, exception_handler=None): 143 | """Concurrently converts a generator object of Requests to 144 | a generator of Responses. 145 | 146 | :param requests: a generator of Request objects. 147 | :param stream: If True, the content will not be downloaded immediately. 148 | :param size: Specifies the number of requests to make at a time. default is 2 149 | :param exception_handler: Callback function, called when exception occurred. Params: Request, Exception 150 | """ 151 | 152 | pool = Pool(size) 153 | 154 | def send(r): 155 | return r.send(stream=stream) 156 | 157 | for request in pool.imap_unordered(send, requests): 158 | if request.response is not None: 159 | yield request.response 160 | elif exception_handler: 161 | ex_result = exception_handler(request, request.exception) 162 | if ex_result is not None: 163 | yield ex_result 164 | 165 | pool.join() 166 | 167 | 168 | def imap_enumerated(requests, stream=False, size=2, exception_handler=None): 169 | """ 170 | Like imap, but yields tuple of original request index and response object 171 | 172 | Unlike imap, failed results and responses from exception handlers that return None are not ignored. Instead, a 173 | tuple of (index, None) is yielded. Additionally, the ``requests`` parameter must be a sequence of Request objects 174 | (generators or other non-sequence iterables are not allowed) 175 | 176 | The index is merely the original index of the original request in the requests list and does NOT provide any 177 | indication of the order in which requests or responses are sent or received. Responses are still in arbitrary order. 178 | 179 | :: 180 | >>> rs = [grequests.get(f'https://httpbin.org/status/{i}') for i in range(200, 206)] 181 | >>> for index, response in grequests.imap_enumerated(rs, size=5): 182 | ... print(index, response) 183 | 1 184 | 0 185 | 4 186 | 2 187 | 5 188 | 3 189 | 190 | 191 | :param requests: a sequence of Request objects. 192 | :param stream: If True, the content will not be downloaded immediately. 193 | :param size: Specifies the number of requests to make at a time. default is 2 194 | :param exception_handler: Callback function, called when exception occurred. Params: Request, Exception 195 | """ 196 | 197 | pool = Pool(size) 198 | 199 | def send(r): 200 | return r._index, r.send(stream=stream) 201 | 202 | requests = list(requests) 203 | for index, req in enumerate(requests): 204 | req._index = index 205 | 206 | for index, request in pool.imap_unordered(send, requests): 207 | if request.response is not None: 208 | yield index, request.response 209 | elif exception_handler: 210 | ex_result = exception_handler(request, request.exception) 211 | yield index, ex_result 212 | else: 213 | yield index, None 214 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | requests 2 | gevent 3 | pytest 4 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | GRequests allows you to use Requests with Gevent to make asynchronous HTTP 4 | Requests easily. 5 | 6 | Usage 7 | ----- 8 | 9 | Usage is simple:: 10 | 11 | import grequests 12 | 13 | urls = [ 14 | 'http://www.heroku.com', 15 | 'http://tablib.org', 16 | 'http://httpbin.org', 17 | 'http://python-requests.org', 18 | 'http://kennethreitz.com' 19 | ] 20 | 21 | Create a set of unsent Requests:: 22 | 23 | >>> rs = (grequests.get(u) for u in urls) 24 | 25 | Send them all at the same time:: 26 | 27 | >>> grequests.map(rs) 28 | [, , , , ] 29 | 30 | """ 31 | 32 | from setuptools import setup 33 | 34 | setup( 35 | name='grequests', 36 | version='0.7.0', 37 | url='https://github.com/spyoungtech/grequests', 38 | license='BSD', 39 | author='Kenneth Reitz', 40 | author_email='me@kennethreitz.com', 41 | description='Requests + Gevent', 42 | long_description=__doc__, 43 | install_requires=[ 44 | 'gevent', 45 | 'requests' 46 | ], 47 | tests_require = ['pytest'], 48 | py_modules=['grequests'], 49 | zip_safe=False, 50 | include_package_data=True, 51 | platforms='any', 52 | classifiers=[ 53 | 'Environment :: Web Environment', 54 | 'Intended Audience :: Developers', 55 | 'License :: OSI Approved :: BSD License', 56 | 'Operating System :: OS Independent', 57 | 'Programming Language :: Python', 58 | 'Programming Language :: Python :: 2.7', 59 | 'Programming Language :: Python :: 3', 60 | 'Topic :: Internet :: WWW/HTTP :: Dynamic Content', 61 | 'Topic :: Software Development :: Libraries :: Python Modules' 62 | ] 63 | ) 64 | -------------------------------------------------------------------------------- /tests.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | 4 | from grequests import get, map, imap 5 | 6 | ########### Constants ############ 7 | urls = [ 8 | 'http://github.com', 9 | 'http://www.google.com', 10 | 'http://www.psf.org' 11 | ] 12 | ############# tests ############## 13 | def test_get(): 14 | global urls 15 | to_fetch = (get(url) for url in urls) 16 | map(to_fetch) 17 | for fetched in to_fetch: 18 | assert fetched.ok 19 | 20 | def test_imap_with_size(): 21 | global urls 22 | to_fetch = (get(url) for url in urls) 23 | imap(to_fetch, size = len(urls) - 1) 24 | for fetching in to_fetch: 25 | assert fetching.send() 26 | 27 | import os 28 | import time 29 | import unittest 30 | 31 | import requests 32 | from requests.exceptions import Timeout 33 | import grequests 34 | 35 | HTTPBIN_URL = os.environ.get('HTTPBIN_URL', 'http://httpbin.org/') 36 | 37 | def httpbin(*suffix): 38 | """Returns url for HTTPBIN resource.""" 39 | return HTTPBIN_URL + '/'.join(suffix) 40 | 41 | 42 | N = 5 43 | URLS = [httpbin('get?p=%s' % i) for i in range(N)] 44 | 45 | 46 | class GrequestsCase(unittest.TestCase): 47 | 48 | def test_map(self): 49 | reqs = [grequests.get(url) for url in URLS] 50 | resp = grequests.map(reqs, size=N) 51 | self.assertEqual([r.url for r in resp], URLS) 52 | 53 | def test_imap(self): 54 | reqs = (grequests.get(url) for url in URLS) 55 | i = 0 56 | for i, r in enumerate(grequests.imap(reqs, size=N)): 57 | self.assertTrue(r.url in URLS) 58 | self.assertEqual(i, N - 1) 59 | 60 | def test_hooks(self): 61 | result = {} 62 | 63 | def hook(r, **kwargs): 64 | result[r.url] = True 65 | return r 66 | 67 | reqs = [grequests.get(url, hooks={'response': [hook]}) for url in URLS] 68 | grequests.map(reqs, size=N) 69 | self.assertEqual(sorted(result.keys()), sorted(URLS)) 70 | 71 | def test_callback_kwarg(self): 72 | result = {'ok': False} 73 | 74 | def callback(r, **kwargs): 75 | result['ok'] = True 76 | return r 77 | 78 | self.get(URLS[0], callback=callback) 79 | self.assertTrue(result['ok']) 80 | 81 | def test_session_and_cookies(self): 82 | c1 = {'k1': 'v1'} 83 | r = self.get(httpbin('cookies/set'), params=c1).json() 84 | self.assertEqual(r['cookies'], c1) 85 | s = requests.Session() 86 | r = self.get(httpbin('cookies/set'), session=s, params=c1).json() 87 | self.assertEqual(dict(s.cookies), c1) 88 | 89 | # ensure all cookies saved 90 | c2 = {'k2': 'v2'} 91 | c1.update(c2) 92 | r = self.get(httpbin('cookies/set'), session=s, params=c2).json() 93 | self.assertEqual(dict(s.cookies), c1) 94 | 95 | # ensure new session is created 96 | r = self.get(httpbin('cookies')).json() 97 | self.assertEqual(r['cookies'], {}) 98 | 99 | # cookies as param 100 | c3 = {'p1': '42'} 101 | r = self.get(httpbin('cookies'), cookies=c3).json() 102 | self.assertEqual(r['cookies'], c3) 103 | 104 | def test_calling_request(self): 105 | reqs = [grequests.request('POST', httpbin('post'), data={'p': i}) 106 | for i in range(N)] 107 | resp = grequests.map(reqs, size=N) 108 | self.assertEqual([int(r.json()['form']['p']) for r in resp], list(range(N))) 109 | 110 | def test_stream_enabled(self): 111 | r = grequests.map([grequests.get(httpbin('stream/10'))], 112 | size=2, stream=True)[0] 113 | self.assertFalse(r._content_consumed) 114 | 115 | def test_concurrency_with_delayed_url(self): 116 | t = time.time() 117 | n = 10 118 | reqs = [grequests.get(httpbin('delay/1')) for _ in range(n)] 119 | grequests.map(reqs, size=n) 120 | self.assertLess((time.time() - t), n) 121 | 122 | def test_map_timeout_no_exception_handler(self): 123 | """ 124 | compliance with existing 0.2.0 behaviour 125 | """ 126 | reqs = [grequests.get(httpbin('delay/1'), timeout=0.001), grequests.get(httpbin('/'))] 127 | responses = grequests.map(reqs) 128 | self.assertIsNone(responses[0]) 129 | self.assertTrue(responses[1].ok) 130 | self.assertEqual(len(responses), 2) 131 | 132 | def test_map_timeout_exception_handler_no_return(self): 133 | """ 134 | ensure default behaviour for a handler that returns None 135 | """ 136 | def exception_handler(request, exception): 137 | pass 138 | reqs = [grequests.get(httpbin('delay/1'), timeout=0.001), grequests.get(httpbin('/'))] 139 | responses = grequests.map(reqs, exception_handler=exception_handler) 140 | self.assertIsNone(responses[0]) 141 | self.assertTrue(responses[1].ok) 142 | self.assertEqual(len(responses), 2) 143 | 144 | def test_map_timeout_exception_handler_returns_exception(self): 145 | """ 146 | ensure returned value from exception handler is stuffed in the map result 147 | """ 148 | def exception_handler(request, exception): 149 | return exception 150 | reqs = [grequests.get(httpbin('delay/1'), timeout=0.001), grequests.get(httpbin('/'))] 151 | responses = grequests.map(reqs, exception_handler=exception_handler) 152 | self.assertIsInstance(responses[0], Timeout) 153 | self.assertTrue(responses[1].ok) 154 | self.assertEqual(len(responses), 2) 155 | 156 | def test_imap_timeout_no_exception_handler(self): 157 | """ 158 | compliance with existing 0.2.0 behaviour 159 | """ 160 | reqs = [grequests.get(httpbin('delay/1'), timeout=0.001)] 161 | out = [] 162 | try: 163 | for r in grequests.imap(reqs): 164 | out.append(r) 165 | except Timeout: 166 | pass 167 | self.assertEqual(out, []) 168 | 169 | def test_imap_timeout_exception_handler_no_return(self): 170 | """ 171 | ensure imap-default behaviour for a handler that returns None 172 | """ 173 | def exception_handler(request, exception): 174 | pass 175 | reqs = [grequests.get(httpbin('delay/1'), timeout=0.001)] 176 | out = [] 177 | for r in grequests.imap(reqs, exception_handler=exception_handler): 178 | out.append(r) 179 | self.assertEqual(out, []) 180 | 181 | 182 | def test_imap_timeout_exception_handler_returns_value(self): 183 | """ 184 | ensure behaviour for a handler that returns a value 185 | """ 186 | def exception_handler(request, exception): 187 | return 'a value' 188 | reqs = [grequests.get(httpbin('delay/1'), timeout=0.001)] 189 | out = [] 190 | for r in grequests.imap(reqs, exception_handler=exception_handler): 191 | out.append(r) 192 | self.assertEqual(out, ['a value']) 193 | 194 | def test_map_timeout_exception(self): 195 | class ExceptionHandler: 196 | def __init__(self): 197 | self.counter = 0 198 | 199 | def callback(self, request, exception): 200 | self.counter += 1 201 | eh = ExceptionHandler() 202 | reqs = [grequests.get(httpbin('delay/1'), timeout=0.001)] 203 | list(grequests.map(reqs, exception_handler=eh.callback)) 204 | self.assertEqual(eh.counter, 1) 205 | 206 | def test_imap_timeout_exception(self): 207 | class ExceptionHandler: 208 | def __init__(self): 209 | self.counter = 0 210 | 211 | def callback(self, request, exception): 212 | self.counter += 1 213 | eh = ExceptionHandler() 214 | reqs = [grequests.get(httpbin('delay/1'), timeout=0.001)] 215 | list(grequests.imap(reqs, exception_handler=eh.callback)) 216 | self.assertEqual(eh.counter, 1) 217 | 218 | def get(self, url, **kwargs): 219 | return grequests.map([grequests.get(url, **kwargs)])[0] 220 | 221 | 222 | if __name__ == '__main__': 223 | unittest.main() 224 | --------------------------------------------------------------------------------