├── .coveralls.yml ├── .gitignore ├── .travis.yml ├── LICENSE ├── MANIFEST.in ├── README.md ├── batch_requests ├── __init__.py ├── concurrent │ ├── __init__.py │ └── executor.py ├── exceptions.py ├── migrations │ └── __init__.py ├── settings.py ├── utils.py └── views.py ├── requirements.txt ├── runtests.py ├── setup.py ├── tests ├── __init__.py ├── conftest.py ├── test_bad_batch_request.py ├── test_base.py ├── test_compatibility.py ├── test_concurrency_base.py ├── test_process_concurrency.py ├── test_settings.py ├── test_thread_concurrency.py ├── test_views.py └── urls.py └── tox.ini /.coveralls.yml: -------------------------------------------------------------------------------- 1 | repo_token: 2Rny9Ty9dyQxSmERQGo4WlZyxSM3W57rN 2 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | 5 | # C extensions 6 | *.so 7 | 8 | # Distribution / packaging 9 | .Python 10 | env/ 11 | build/ 12 | develop-eggs/ 13 | dist/ 14 | downloads/ 15 | eggs/ 16 | lib/ 17 | lib64/ 18 | parts/ 19 | sdist/ 20 | var/ 21 | *.egg-info/ 22 | .installed.cfg 23 | *.egg 24 | 25 | # PyInstaller 26 | # Usually these files are written by a python script from a template 27 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 28 | *.manifest 29 | *.spec 30 | 31 | # Installer logs 32 | pip-log.txt 33 | pip-delete-this-directory.txt 34 | 35 | # Unit test / coverage reports 36 | htmlcov/ 37 | .tox/ 38 | .coverage 39 | .cache 40 | nosetests.xml 41 | coverage.xml 42 | 43 | # Translations 44 | *.mo 45 | *.pot 46 | 47 | # Django stuff: 48 | *.log 49 | 50 | # Sphinx documentation 51 | docs/_build/ 52 | 53 | # PyBuilder 54 | target/ 55 | 56 | db.sqlite3 57 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: python 2 | 3 | python: 4 | - "2.7" 5 | 6 | install: pip install -r requirements.txt 7 | 8 | script: python setup.py test 9 | 10 | after_success: 11 | coveralls 12 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2014 Rahul 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | 23 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include README.md LICENSE 2 | recursive-exclude * __pycache__ 3 | recursive-exclude * *.py[co] 4 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | Django Batch Requests 2 | ========================= 3 | 4 | [![build-status-image]][travis] 5 | [![pypi-version]][pypi] 6 | [![coverage]][coverage-repo] 7 | [![Downloads](https://pepy.tech/badge/django-batch-requests)](https://pepy.tech/project/django-batch-requests) 8 | 9 | Django batch requests allow developers to combine multiple http requests into a single batch request. This is essentially useful to avoid making multiple http requests. 10 | 11 | Its built on top of Django and relies on Django's URL dispatching, hence it could also be used with any web framework built on top of Django such as [Django Rest Framework]. 12 | 13 | # Requirements 14 | 15 | * Python (2.7) 16 | * Django (1.7) 17 | 18 | # Installation 19 | 20 | Install using `pip`... 21 | 22 | pip install django-batch-requests 23 | 24 | Install from source... 25 | 26 | python setup.py install 27 | 28 | 29 | # Usage 30 | 31 | Add `'batch_requests'` to your `INSTALLED_APPS` setting. 32 | 33 | INSTALLED_APPS = ( 34 | ... 35 | 'batch_requests', 36 | ) 37 | 38 | In your `urls.py` add `batch_requests.views.handle_batch_requests` view with one of the endpoints as shown below: 39 | 40 | 41 | url(r'^api/v1/batch/', 'batch_requests.views.handle_batch_requests') 42 | 43 | This is the only setup required to get it working. 44 | 45 | 46 | 47 | # Making batch requests 48 | 49 | Using `batch_requests` is simple. Once we have an endpoint for `batch_requests` working, we can make a `POST` call specifying the list of requests. Each request is a JSON object with 4 keys: 50 | 51 | * method: Specifies the http method. Valid values include GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS, CONNECT, and TRACE. 52 | 53 | * url: Relative URL for the request. 54 | 55 | * body: Serialized / Encoded HTTP body as a String. 56 | 57 | * headers: Any additional headers that you would like to pass to this request. 58 | 59 | At the minimum we need to specify url and method for a request object to be considered valid. Consider for instance we need to batch two http requests. The valid json should look like as shown: 60 | 61 | ```json 62 | [ 63 | { 64 | "url" : "/views/", 65 | "method" : "get" 66 | }, 67 | { 68 | "url" : "/views/", 69 | "method" : "post", 70 | "body" : "{\"text\": \"some text\"}", 71 | "headers" : {"Content-Type": "application/json"} 72 | } 73 | ] 74 | ``` 75 | 76 | Here, we are making 2 requests. Get on /views/ and POST to /views/. Please also note that, body of the post request is JSON encoded string (Serialized JSON). 77 | 78 | For such a request, batch api replies back with list of HTTP response objects. Each response object consist of status_code, body and response headers. For the above request, the valid response may look like: 79 | 80 | ```json 81 | [ 82 | { 83 | "headers": { 84 | "Content-Type": "text/html; charset=utf-8" 85 | }, 86 | "status_code": 200, 87 | "body": "Success!", 88 | "reason_phrase": "OK" 89 | }, 90 | { 91 | "headers": { 92 | "Content-Type": "text/html; charset=utf-8" 93 | }, 94 | "status_code": 201, 95 | "body": "{\"text\": \"some text\"}", 96 | "reason_phrase": "CREATED" 97 | } 98 | ] 99 | ``` 100 | 101 | Please note that, requests and responses in `batch_requests` are ordered, meaning the first response in the list is for the first request specified. 102 | 103 | It is also important to note that all the requests by default execute sequentially one after another. Yes, you can change this behavior by configuring the concurrency settings. 104 | 105 | 106 | # Executing requests in parallel (Concurrency) 107 | 108 | Before we jump to concurrency, lets first examine the execution time required to run all requests sequentially. Assume we have an API `/sleep/?seconds=3` which mimics the time consuming APIs by putting the thread to sleep for the specified duration. Let us now make 2 requests in batch for the above sleep API. 109 | 110 | ```json 111 | [ 112 | { 113 | "method": "get", 114 | "url": "/sleep/?seconds=3" 115 | }, 116 | { 117 | "method": "get", 118 | "url": "/sleep/?seconds=3" 119 | } 120 | ] 121 | ``` 122 | Here is the response we get. 123 | 124 | ```json 125 | [ 126 | { 127 | "headers": { 128 | "Content-Type": "text/html; charset=utf-8", 129 | "batch_requests.duration": 3 130 | }, 131 | "status_code": 200, 132 | "body": "Success!", 133 | "reason_phrase": "OK" 134 | }, 135 | { 136 | "headers": { 137 | "Content-Type": "text/html; charset=utf-8", 138 | "batch_requests.duration": 3 139 | }, 140 | "status_code": 200, 141 | "body": "Success!", 142 | "reason_phrase": "OK" 143 | } 144 | ] 145 | ``` 146 | Each request took about 3 seconds as is evident with the header `"batch_requests.duration": 3`. 147 | 148 | Batch api response header also include: 149 | ` 150 | batch_requests.duration 6 151 | ` 152 | This shows the total batch request took 6 seconds - sum of the individual requests. Let us now turn ON the concurrency. 153 | 154 | ## Configuring Parallelism / Concurrency: 155 | 156 | Parallelism / Concurrency can be turned ON by specifying the appropriate settings: 157 | 158 | `"EXECUTE_PARALLEL": True` 159 | 160 | `batch_requests` will now execute the individual requests in parallel. Let us check how it impacts the performance. We will make the same request once again after enabling the parallelism. 161 | 162 | ```json 163 | [ 164 | { 165 | "method": "get", 166 | "url": "/sleep/?seconds=3" 167 | }, 168 | { 169 | "method": "get", 170 | "url": "/sleep/?seconds=3" 171 | } 172 | ] 173 | ``` 174 | 175 | This request yields the following response: 176 | 177 | ```json 178 | [ 179 | { 180 | "headers": { 181 | "Content-Type": "text/html; charset=utf-8", 182 | "batch_requests.duration": 3 183 | }, 184 | "status_code": 200, 185 | "body": "Success!", 186 | "reason_phrase": "OK" 187 | }, 188 | { 189 | "headers": { 190 | "Content-Type": "text/html; charset=utf-8", 191 | "batch_requests.duration": 3 192 | }, 193 | "status_code": 200, 194 | "body": "Success!", 195 | "reason_phrase": "OK" 196 | } 197 | ] 198 | ``` 199 | with the batch response header: 200 | 201 | `batch_requests.duration: 3`. 202 | 203 | Though each request still took 3 seconds, total time that batch request took is only 3 seconds. This is an evident that requests were executed concurrently. 204 | 205 | `batch_requests` add duration header for all the individual requests and also the main batch request. If it is too verbose for you, you can turn this feature off by setting: 206 | 207 | `"ADD_DURATION_HEADER": True` 208 | 209 | You can also change the duration header name to whatever suites you better, by setting: 210 | 211 | `"DURATION_HEADER_NAME": "batch_requests.duration"` 212 | 213 | ## More Parallelism / Concurrency settings: 214 | 215 | There are two widely used approached to achieve concurrency. One through launching multiple threads and another through launching multiple processes. `batch_requests` support both these approaches. There are two settings you can configure in this regard: 216 | 217 | ``` 218 | "CONCURRENT_EXECUTOR": "batch_requests.concurrent.executor.ThreadBasedExecutor" 219 | "NUM_WORKERS": 10 220 | ``` 221 | 222 | `CONCURRENT_EXECUTOR` value must be one of the following two: 223 | 1. batch_requests.concurrent.executor.ThreadBasedExecutor 224 | 2. batch_requests.concurrent.executor.ProcessBasedExecutor 225 | 226 | to achive thread and process based concurrency respectively. `NUM_WORKERS` determines how may threads / processes to pool to execute the requests. Configure this number wisely based on the hardware resources you have. By default, if you turn ON the parallelism, `ThreadBasedExecutor` with `number_of_cpu * 4` workers is configured on the pool. 227 | 228 | 229 | ## Choosing between threads vs processes for concurrency: 230 | 231 | There is no abvious answer to this, and it depends on various settings - the resources you have, the amount of web workers you are running, whether the application is blocking or non blocking, if the application is cpu or io bound etc. However, the good way to start off with is: 232 | 233 | * Choose ThreadBasedExecutor if your application is doing too much IO and the code is blocking. 234 | * Choose ProcessBasedExecutor if your application is CPU bound. 235 | 236 | 237 | 238 | [build-status-image]: https://secure.travis-ci.org/tanwanirahul/django-batch-requests.svg?branch=master 239 | [travis]: http://travis-ci.org/tanwanirahul/django-batch-requests?branch=master 240 | [pypi-version]: https://badge.fury.io/py/django-batch-requests.svg 241 | [pypi]: https://pypi.python.org/pypi/django-batch-requests 242 | [Django Rest Framework]: https://github.com/tomchristie/django-rest-framework 243 | [coverage]: https://coveralls.io/repos/tanwanirahul/django-batch-requests/badge.png?branch=master 244 | [coverage-repo]: https://coveralls.io/r/tanwanirahul/django-batch-requests?branch=master 245 | -------------------------------------------------------------------------------- /batch_requests/__init__.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | __title__ = 'django-batch-requests' 4 | __version__ = '0.1.1' 5 | __author__ = 'Rahul Tanwani' 6 | __license__ = 'MIT' 7 | 8 | # Version synonym 9 | VERSION = __version__ 10 | -------------------------------------------------------------------------------- /batch_requests/concurrent/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tanwanirahul/django-batch-requests/9c5afc42f7542f466247f4ffed9c44e1c49fa20d/batch_requests/concurrent/__init__.py -------------------------------------------------------------------------------- /batch_requests/concurrent/executor.py: -------------------------------------------------------------------------------- 1 | ''' 2 | Created on Feb 20, 2016 3 | 4 | @author: Rahul Tanwani 5 | ''' 6 | from abc import ABCMeta 7 | from concurrent.futures.thread import ThreadPoolExecutor 8 | from concurrent.futures.process import ProcessPoolExecutor 9 | 10 | 11 | class Executor(object): 12 | ''' 13 | Based executor class to encapsulate the job execution. 14 | ''' 15 | __metaclass__ = ABCMeta 16 | 17 | def execute(self, requests, resp_generator, *args, **kwargs): 18 | ''' 19 | Calls the resp_generator for all the requests in parallel in an asynchronous way. 20 | ''' 21 | result_futures = [self.executor_pool.submit(resp_generator, req, *args, **kwargs) for req in requests] 22 | resp = [res_future.result() for res_future in result_futures] 23 | return resp 24 | 25 | 26 | class SequentialExecutor(Executor): 27 | ''' 28 | Executor for executing the requests sequentially. 29 | ''' 30 | 31 | def execute(self, requests, resp_generator, *args, **kwargs): 32 | ''' 33 | Calls the resp_generator for all the requests in sequential order. 34 | ''' 35 | return [resp_generator(request) for request in requests] 36 | 37 | 38 | class ThreadBasedExecutor(Executor): 39 | ''' 40 | An implementation of executor using threads for parallelism. 41 | ''' 42 | def __init__(self, num_workers): 43 | ''' 44 | Create a thread pool for concurrent execution with specified number of workers. 45 | ''' 46 | self.executor_pool = ThreadPoolExecutor(num_workers) 47 | 48 | 49 | class ProcessBasedExecutor(Executor): 50 | ''' 51 | An implementation of executor using process(es) for parallelism. 52 | ''' 53 | def __init__(self, num_workers): 54 | ''' 55 | Create a process pool for concurrent execution with specified number of workers. 56 | ''' 57 | self.executor_pool = ProcessPoolExecutor(num_workers) 58 | -------------------------------------------------------------------------------- /batch_requests/exceptions.py: -------------------------------------------------------------------------------- 1 | ''' 2 | Created on Dec 30, 2014 3 | 4 | @author: Rahul Tanwani 5 | 6 | @summary: Holds exception required for batch_requests app. 7 | ''' 8 | 9 | 10 | class BadBatchRequest(Exception): 11 | ''' 12 | Raised when client sends an invalid batch request. 13 | ''' 14 | def __init__(self, *args, **kwargs): 15 | ''' 16 | Initialize. 17 | ''' 18 | Exception.__init__(self, *args, **kwargs) 19 | -------------------------------------------------------------------------------- /batch_requests/migrations/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tanwanirahul/django-batch-requests/9c5afc42f7542f466247f4ffed9c44e1c49fa20d/batch_requests/migrations/__init__.py -------------------------------------------------------------------------------- /batch_requests/settings.py: -------------------------------------------------------------------------------- 1 | ''' 2 | @author: Rahul Tanwani 3 | 4 | @summary: Contains the default settings. 5 | ''' 6 | 7 | from django.conf import settings 8 | from django.utils.importlib import import_module 9 | import multiprocessing 10 | 11 | DEFAULTS = { 12 | "HEADERS_TO_INCLUDE": ["HTTP_USER_AGENT", "HTTP_COOKIE"], 13 | "DEFAULT_CONTENT_TYPE": "application/json", 14 | "USE_HTTPS": False, 15 | "EXECUTE_PARALLEL": False, 16 | "CONCURRENT_EXECUTOR": "batch_requests.concurrent.executor.ThreadBasedExecutor", 17 | "NUM_WORKERS": multiprocessing.cpu_count() * 4, 18 | "ADD_DURATION_HEADER": True, 19 | "DURATION_HEADER_NAME": "batch_requests.duration", 20 | "MAX_LIMIT": 20 21 | } 22 | 23 | 24 | USER_DEFINED_SETTINGS = getattr(settings, 'BATCH_REQUESTS', {}) 25 | 26 | 27 | def import_class(class_path): 28 | ''' 29 | Imports the class for the given class name. 30 | ''' 31 | module_name, class_name = class_path.rsplit(".", 1) 32 | module = import_module(module_name) 33 | claz = getattr(module, class_name) 34 | return claz 35 | 36 | 37 | class BatchRequestSettings(object): 38 | 39 | ''' 40 | Allow API settings to be accessed as properties. 41 | ''' 42 | 43 | def __init__(self, user_settings=None, defaults=None): 44 | self.user_settings = user_settings or {} 45 | self.defaults = defaults or {} 46 | self.executor = self._executor() 47 | 48 | def _executor(self): 49 | ''' 50 | Creating an ExecutorPool is a costly operation. Executor needs to be instantiated only once. 51 | ''' 52 | if self.EXECUTE_PARALLEL is False: 53 | executor_path = "batch_requests.concurrent.executor.SequentialExecutor" 54 | executor_class = import_class(executor_path) 55 | return executor_class() 56 | else: 57 | executor_path = self.CONCURRENT_EXECUTOR 58 | executor_class = import_class(executor_path) 59 | return executor_class(self.NUM_WORKERS) 60 | 61 | def __getattr__(self, attr): 62 | ''' 63 | Override the attribute access behavior. 64 | ''' 65 | 66 | if attr not in self.defaults.keys(): 67 | raise AttributeError("Invalid API setting: '%s'" % attr) 68 | 69 | try: 70 | # Check if present in user settings 71 | val = self.user_settings[attr] 72 | except KeyError: 73 | # Fall back to defaults 74 | val = self.defaults[attr] 75 | 76 | # Cache the result 77 | setattr(self, attr, val) 78 | return val 79 | 80 | 81 | br_settings = BatchRequestSettings(USER_DEFINED_SETTINGS, DEFAULTS) 82 | -------------------------------------------------------------------------------- /batch_requests/utils.py: -------------------------------------------------------------------------------- 1 | ''' 2 | @author: Rahul Tanwani 3 | 4 | @summary: Holds all the utilities functions required to support batch_requests. 5 | ''' 6 | from django.test.client import RequestFactory, FakePayload 7 | from batch_requests.settings import br_settings as _settings 8 | 9 | 10 | class BatchRequestFactory(RequestFactory): 11 | 12 | ''' 13 | Extend the RequestFactory and update the environment variables for WSGI. 14 | ''' 15 | 16 | def _base_environ(self, **request): 17 | ''' 18 | Override the default values for the wsgi environment variables. 19 | ''' 20 | # This is a minimal valid WSGI environ dictionary, plus: 21 | # - HTTP_COOKIE: for cookie support, 22 | # - REMOTE_ADDR: often useful, see #8551. 23 | # See http://www.python.org/dev/peps/pep-3333/#environ-variables 24 | 25 | environ = { 26 | 'HTTP_COOKIE': self.cookies.output(header='', sep='; '), 27 | 'PATH_INFO': str('/'), 28 | 'REMOTE_ADDR': str('127.0.0.1'), 29 | 'REQUEST_METHOD': str('GET'), 30 | 'SCRIPT_NAME': str(''), 31 | 'SERVER_NAME': str('localhost'), 32 | 'SERVER_PORT': str('8000'), 33 | 'SERVER_PROTOCOL': str('HTTP/1.1'), 34 | 'wsgi.version': (1, 0), 35 | 'wsgi.url_scheme': str('http'), 36 | 'wsgi.input': FakePayload(b''), 37 | 'wsgi.errors': self.errors, 38 | 'wsgi.multiprocess': True, 39 | 'wsgi.multithread': True, 40 | 'wsgi.run_once': False, 41 | } 42 | environ.update(self.defaults) 43 | environ.update(request) 44 | return environ 45 | 46 | 47 | def pre_process_method_headers(method, headers): 48 | ''' 49 | Returns the lowered method. 50 | Capitalize headers, prepend HTTP_ and change - to _. 51 | ''' 52 | method = method.lower() 53 | 54 | # Standard WSGI supported headers 55 | _wsgi_headers = ["content_length", "content_type", "query_string", 56 | "remote_addr", "remote_host", "remote_user", 57 | "request_method", "server_name", "server_port"] 58 | 59 | _transformed_headers = {} 60 | 61 | # For every header, replace - to _, prepend http_ if necessary and convert 62 | # to upper case. 63 | for header, value in headers.items(): 64 | 65 | header = header.replace("-", "_") 66 | header = "http_{header}".format( 67 | header=header) if header.lower() not in _wsgi_headers else header 68 | _transformed_headers.update({header.upper(): value}) 69 | 70 | return method, _transformed_headers 71 | 72 | 73 | def headers_to_include_from_request(curr_request): 74 | ''' 75 | Define headers that needs to be included from the current request. 76 | ''' 77 | return { 78 | h: v for h, v in curr_request.META.items() if h in _settings.HEADERS_TO_INCLUDE} 79 | 80 | 81 | def get_wsgi_request_object(curr_request, method, url, headers, body): 82 | ''' 83 | Based on the given request parameters, constructs and returns the WSGI request object. 84 | ''' 85 | x_headers = headers_to_include_from_request(curr_request) 86 | method, t_headers = pre_process_method_headers(method, headers) 87 | 88 | # Add default content type. 89 | if "CONTENT_TYPE" not in t_headers: 90 | t_headers.update({"CONTENT_TYPE": _settings.DEFAULT_CONTENT_TYPE}) 91 | 92 | # Override existing batch requests headers with the new headers passed for this request. 93 | x_headers.update(t_headers) 94 | 95 | content_type = x_headers.get("CONTENT_TYPE", _settings.DEFAULT_CONTENT_TYPE) 96 | 97 | # Get hold of request factory to construct the request. 98 | _request_factory = BatchRequestFactory() 99 | _request_provider = getattr(_request_factory, method) 100 | 101 | secure = _settings.USE_HTTPS 102 | 103 | request = _request_provider(url, data=body, secure=secure, 104 | content_type=content_type, **x_headers) 105 | 106 | return request 107 | -------------------------------------------------------------------------------- /batch_requests/views.py: -------------------------------------------------------------------------------- 1 | ''' 2 | @author: Rahul Tanwani 3 | 4 | @summary: A module to perform batch request processing. 5 | ''' 6 | 7 | import json 8 | 9 | from django.core.urlresolvers import resolve 10 | from django.http.response import HttpResponse, HttpResponseBadRequest,\ 11 | HttpResponseServerError 12 | from django.template.response import ContentNotRenderedError 13 | from django.views.decorators.csrf import csrf_exempt 14 | from django.views.decorators.http import require_http_methods 15 | 16 | from batch_requests.exceptions import BadBatchRequest 17 | from batch_requests.settings import br_settings as _settings 18 | from batch_requests.utils import get_wsgi_request_object 19 | from datetime import datetime 20 | 21 | 22 | def get_response(wsgi_request): 23 | ''' 24 | Given a WSGI request, makes a call to a corresponding view 25 | function and returns the response. 26 | ''' 27 | service_start_time = datetime.now() 28 | # Get the view / handler for this request 29 | view, args, kwargs = resolve(wsgi_request.path_info) 30 | 31 | kwargs.update({"request": wsgi_request}) 32 | 33 | # Let the view do his task. 34 | try: 35 | resp = view(*args, **kwargs) 36 | except Exception as exc: 37 | resp = HttpResponseServerError(content=exc.message) 38 | 39 | headers = dict(resp._headers.values()) 40 | # Convert HTTP response into simple dict type. 41 | d_resp = {"status_code": resp.status_code, "reason_phrase": resp.reason_phrase, 42 | "headers": headers} 43 | try: 44 | d_resp.update({"body": resp.content}) 45 | except ContentNotRenderedError: 46 | resp.render() 47 | d_resp.update({"body": resp.content}) 48 | 49 | # Check if we need to send across the duration header. 50 | if _settings.ADD_DURATION_HEADER: 51 | d_resp['headers'].update({_settings.DURATION_HEADER_NAME: (datetime.now() - service_start_time).seconds}) 52 | 53 | return d_resp 54 | 55 | 56 | def get_wsgi_requests(request): 57 | ''' 58 | For the given batch request, extract the individual requests and create 59 | WSGIRequest object for each. 60 | ''' 61 | valid_http_methods = ["get", "post", "put", "patch", "delete", "head", "options", "connect", "trace"] 62 | requests = json.loads(request.body) 63 | 64 | if type(requests) not in (list, tuple): 65 | raise BadBatchRequest("The body of batch request should always be list!") 66 | 67 | # Max limit check. 68 | no_requests = len(requests) 69 | 70 | if no_requests > _settings.MAX_LIMIT: 71 | raise BadBatchRequest("You can batch maximum of %d requests." % (_settings.MAX_LIMIT)) 72 | 73 | # We could mutate the current request with the respective parameters, but mutation is ghost in the dark, 74 | # so lets avoid. Construct the new WSGI request object for each request. 75 | 76 | def construct_wsgi_from_data(data): 77 | ''' 78 | Given the data in the format of url, method, body and headers, construct a new 79 | WSGIRequest object. 80 | ''' 81 | url = data.get("url", None) 82 | method = data.get("method", None) 83 | 84 | if url is None or method is None: 85 | raise BadBatchRequest("Request definition should have url, method defined.") 86 | 87 | if method.lower() not in valid_http_methods: 88 | raise BadBatchRequest("Invalid request method.") 89 | 90 | body = data.get("body", "") 91 | headers = data.get("headers", {}) 92 | return get_wsgi_request_object(request, method, url, headers, body) 93 | 94 | return [construct_wsgi_from_data(data) for data in requests] 95 | 96 | 97 | def execute_requests(wsgi_requests): 98 | ''' 99 | Execute the requests either sequentially or in parallel based on parallel 100 | execution setting. 101 | ''' 102 | executor = _settings.executor 103 | return executor.execute(wsgi_requests, get_response) 104 | 105 | 106 | @csrf_exempt 107 | @require_http_methods(["POST"]) 108 | def handle_batch_requests(request, *args, **kwargs): 109 | ''' 110 | A view function to handle the overall processing of batch requests. 111 | ''' 112 | batch_start_time = datetime.now() 113 | try: 114 | # Get the Individual WSGI requests. 115 | wsgi_requests = get_wsgi_requests(request) 116 | except BadBatchRequest as brx: 117 | return HttpResponseBadRequest(content=brx.message) 118 | 119 | # Fire these WSGI requests, and collect the response for the same. 120 | response = execute_requests(wsgi_requests) 121 | 122 | # Evrything's done, return the response. 123 | resp = HttpResponse( 124 | content=json.dumps(response), content_type="application/json") 125 | 126 | if _settings.ADD_DURATION_HEADER: 127 | resp.__setitem__(_settings.DURATION_HEADER_NAME, str((datetime.now() - batch_start_time).seconds)) 128 | return resp 129 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | Django==1.7.6 2 | flake8==2.4.0 3 | mccabe==0.3 4 | pep8==1.5.7 5 | py==1.4.26 6 | pyflakes==0.8.1 7 | pytest==2.6.4 8 | pytest-django==2.8.0 9 | coveralls==0.5 10 | pytest-cov==1.8.1 11 | futures==3.0.5 12 | -------------------------------------------------------------------------------- /runtests.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | """ 3 | From https://github.com/tomchristie/django-rest-framework/blob/bf09c32de8f9d528f83e9cb7a2773d1f4c9ab563/runtests.py 4 | """ 5 | from __future__ import print_function 6 | 7 | import os 8 | import subprocess 9 | import sys 10 | 11 | import pytest 12 | 13 | 14 | PYTEST_ARGS = { 15 | 'default': ['tests'], 16 | 'fast': ['tests', '-q'], 17 | } 18 | 19 | FLAKE8_ARGS = ['batch_requests', 'tests', '--ignore=E501'] 20 | 21 | 22 | sys.path.append(os.path.dirname(__file__)) 23 | 24 | 25 | def exit_on_failure(ret, message=None): 26 | if ret: 27 | sys.exit(ret) 28 | 29 | 30 | def flake8_main(args): 31 | print('Running flake8 code linting') 32 | ret = subprocess.call(['flake8'] + args) 33 | print('flake8 failed' if ret else 'flake8 passed') 34 | return ret 35 | 36 | 37 | def split_class_and_function(string): 38 | class_string, function_string = string.split('.', 1) 39 | return "%s and %s" % (class_string, function_string) 40 | 41 | 42 | def is_function(string): 43 | # `True` if it looks like a test function is included in the string. 44 | return string.startswith('test_') or '.test_' in string 45 | 46 | 47 | def is_class(string): 48 | # `True` if first character is uppercase - assume it's a class name. 49 | return string[0] == string[0].upper() 50 | 51 | 52 | if __name__ == "__main__": 53 | try: 54 | sys.argv.remove('--nolint') 55 | except ValueError: 56 | run_flake8 = True 57 | else: 58 | run_flake8 = False 59 | 60 | try: 61 | sys.argv.remove('--lintonly') 62 | except ValueError: 63 | run_tests = True 64 | else: 65 | run_tests = False 66 | 67 | try: 68 | sys.argv.remove('--fast') 69 | except ValueError: 70 | style = 'default' 71 | else: 72 | style = 'fast' 73 | run_flake8 = False 74 | 75 | if len(sys.argv) > 1: 76 | pytest_args = sys.argv[1:] 77 | first_arg = pytest_args[0] 78 | if first_arg.startswith('-'): 79 | # `runtests.py [flags]` 80 | pytest_args = ['tests'] + pytest_args 81 | elif is_class(first_arg) and is_function(first_arg): 82 | # `runtests.py TestCase.test_function [flags]` 83 | expression = split_class_and_function(first_arg) 84 | pytest_args = ['tests', '-k', expression] + pytest_args[1:] 85 | elif is_class(first_arg) or is_function(first_arg): 86 | # `runtests.py TestCase [flags]` 87 | # `runtests.py test_function [flags]` 88 | pytest_args = ['tests', '-k', pytest_args[0]] + pytest_args[1:] 89 | else: 90 | pytest_args = PYTEST_ARGS[style] 91 | 92 | if run_tests: 93 | print("Running tests...") 94 | exit_on_failure(pytest.main(pytest_args)) 95 | if run_flake8: 96 | exit_on_failure(flake8_main(FLAKE8_ARGS)) 97 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | 4 | import os 5 | import sys 6 | 7 | import batch_requests 8 | 9 | from setuptools import setup 10 | from setuptools import setup, Command 11 | 12 | class PyTest(Command): 13 | ''' 14 | A command handler for setup.py test. 15 | ''' 16 | user_options = [] 17 | def initialize_options(self): 18 | pass 19 | 20 | def finalize_options(self): 21 | pass 22 | 23 | def run(self): 24 | import subprocess 25 | import sys 26 | errno = subprocess.call('py.test --cov-report html --cov batch_requests tests/', shell=True) 27 | raise SystemExit(errno) 28 | 29 | name = 'django-batch-requests' 30 | version = batch_requests.__version__ 31 | package = 'batch_requests' 32 | description = 'Create batch APIs for Django.' 33 | long_description = 'Django batch requests allow developers to combine multiple http requests into a single batch request. This is essentially useful to avoid making multiple http requests to save on round trip network latency.' 34 | url = 'https://github.com/tanwanirahul/django-batch-requests' 35 | author = 'Rahul Tanwani' 36 | author_email = 'tanwanirahul@gmail.com' 37 | license = 'MIT' 38 | install_requires = [] 39 | 40 | 41 | def read(*paths): 42 | """ 43 | Build a file path from paths and return the contents. 44 | """ 45 | with open(os.path.join(*paths), 'r') as f: 46 | return f.read() 47 | 48 | 49 | def get_packages(package): 50 | """ 51 | Return root package and all sub-packages. 52 | """ 53 | return [dirpath 54 | for dirpath, dirnames, filenames in os.walk(package) 55 | if os.path.exists(os.path.join(dirpath, '__init__.py'))] 56 | 57 | 58 | def get_package_data(package): 59 | """ 60 | Return all files under the root package, that are not in a 61 | package themselves. 62 | """ 63 | walk = [(dirpath.replace(package + os.sep, '', 1), filenames) 64 | for dirpath, dirnames, filenames in os.walk(package) 65 | if not os.path.exists(os.path.join(dirpath, '__init__.py'))] 66 | 67 | filepaths = [] 68 | for base, filenames in walk: 69 | filepaths.extend([os.path.join(base, filename) 70 | for filename in filenames]) 71 | return {package: filepaths} 72 | 73 | 74 | if sys.argv[-1] == 'publish': 75 | os.system("python setup.py sdist upload") 76 | os.system("python setup.py bdist_wheel upload") 77 | print("You probably want to also tag the version now:") 78 | print(" git tag -a {0} -m 'version {0}'".format(version)) 79 | print(" git push --tags") 80 | sys.exit() 81 | 82 | 83 | setup( 84 | name=name, 85 | version=version, 86 | url=url, 87 | license=license, 88 | description=description, 89 | long_description=long_description, 90 | author=author, 91 | author_email=author_email, 92 | packages=get_packages(package), 93 | package_data=get_package_data(package), 94 | install_requires=install_requires, 95 | cmdclass = {'test': PyTest}, 96 | classifiers=[ 97 | 'Development Status :: 5 - Production/Stable', 98 | 'Environment :: Web Environment', 99 | 'Framework :: Django', 100 | 'Intended Audience :: Developers', 101 | 'License :: OSI Approved :: MIT License', 102 | 'Operating System :: OS Independent', 103 | 'Programming Language :: Python', 104 | 'Programming Language :: Python :: 2', 105 | 'Programming Language :: Python :: 2.7', 106 | 'Topic :: Internet :: WWW/HTTP' 107 | ] 108 | ) 109 | -------------------------------------------------------------------------------- /tests/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tanwanirahul/django-batch-requests/9c5afc42f7542f466247f4ffed9c44e1c49fa20d/tests/__init__.py -------------------------------------------------------------------------------- /tests/conftest.py: -------------------------------------------------------------------------------- 1 | def pytest_configure(): 2 | from django.conf import settings 3 | 4 | settings.configure( 5 | DEBUG_PROPAGATE_EXCEPTIONS=True, 6 | DATABASES={'default': {'ENGINE': 'django.db.backends.sqlite3', 7 | 'NAME': ':memory:'}}, 8 | SITE_ID=1, 9 | SECRET_KEY='not very secret in tests', 10 | USE_I18N=True, 11 | USE_L10N=True, 12 | STATIC_URL='/static/', 13 | ROOT_URLCONF='tests.urls', 14 | TEMPLATE_LOADERS=( 15 | 'django.template.loaders.filesystem.Loader', 16 | 'django.template.loaders.app_directories.Loader', 17 | ), 18 | MIDDLEWARE_CLASSES=( 19 | 'django.middleware.common.CommonMiddleware', 20 | 'django.contrib.sessions.middleware.SessionMiddleware', 21 | 'django.contrib.auth.middleware.AuthenticationMiddleware', 22 | 'django.contrib.messages.middleware.MessageMiddleware', 23 | ), 24 | INSTALLED_APPS=( 25 | 'django.contrib.auth', 26 | 'django.contrib.contenttypes', 27 | 'django.contrib.sessions', 28 | 'django.contrib.sites', 29 | 'django.contrib.staticfiles', 30 | 'tests', 31 | 'batch_requests', 32 | ), 33 | PASSWORD_HASHERS=( 34 | 'django.contrib.auth.hashers.MD5PasswordHasher', 35 | ), 36 | BATCH_REQUESTS = { 37 | "HEADERS_TO_INCLUDE": ["HTTP_USER_AGENT", "HTTP_COOKIE"], 38 | "DEFAULT_CONTENT_TYPE": "application/xml", 39 | "USE_HTTPS": True, 40 | "MAX_LIMIT": 3 41 | } 42 | ) 43 | try: 44 | import django 45 | django.setup() 46 | except AttributeError: 47 | pass 48 | -------------------------------------------------------------------------------- /tests/test_bad_batch_request.py: -------------------------------------------------------------------------------- 1 | ''' 2 | @author: Rahul Tanwani 3 | 4 | @summary: Test cases to check the behavior when the batch request is 5 | not constructed properly. 6 | ''' 7 | import json 8 | 9 | from django.test import TestCase 10 | 11 | 12 | class TestBadBatchRequest(TestCase): 13 | 14 | ''' 15 | Check the behavior of bad batch request. 16 | ''' 17 | 18 | def _batch_request(self, method, path, data, headers={}): 19 | ''' 20 | Prepares a batch request. 21 | ''' 22 | return {"url": path, "method": method, "headers": headers, "body": data} 23 | 24 | def test_invalid_http_method(self): 25 | ''' 26 | Make a batch request with invalid HTTP method. 27 | ''' 28 | resp = self.client.post("/api/v1/batch/", json.dumps([self._batch_request("select", "/views", "", {})]), 29 | content_type="application/json") 30 | 31 | self.assertEqual(resp.status_code, 400, "Method validation is broken!") 32 | self.assertEqual(resp.content.lower(), "invalid request method.", "Method validation is broken!") 33 | 34 | def test_missing_http_method(self): 35 | ''' 36 | Make a batch request without HTTP method. 37 | ''' 38 | resp = self.client.post("/api/v1/batch/", json.dumps([{"body": "/views"}]), content_type="application/json") 39 | 40 | self.assertEqual(resp.status_code, 400, "Method & URL validation is broken!") 41 | self.assertEqual(resp.content.lower(), "request definition should have url, method defined.", 42 | "Method validation is broken!") 43 | 44 | def test_missing_url(self): 45 | ''' 46 | Make a batch request without the URL. 47 | ''' 48 | resp = self.client.post("/api/v1/batch/", json.dumps([{"method": "get"}]), content_type="application/json") 49 | 50 | self.assertEqual(resp.status_code, 400, "Method & URL validation is broken!") 51 | self.assertEqual(resp.content.lower(), "request definition should have url, method defined.", 52 | "Method validation is broken!") 53 | 54 | def test_invalid_batch_request(self): 55 | ''' 56 | Make a batch request without wrapping in the list. 57 | ''' 58 | resp = self.client.post("/api/v1/batch/", json.dumps({"method": "get", "url": "/views/"}), 59 | content_type="application/json") 60 | 61 | print resp.content 62 | self.assertEqual(resp.status_code, 400, "Batch requests should always be in list.") 63 | self.assertEqual(resp.content.lower(), "the body of batch request should always be list!", 64 | "List validation is broken!") 65 | 66 | def test_view_that_raises_exception(self): 67 | ''' 68 | Make a batch request to a view that raises exception. 69 | ''' 70 | resp = self.client.post("/api/v1/batch/", json.dumps([{"method": "get", "url": "/exception/"}]), 71 | content_type="application/json") 72 | 73 | resp = json.loads(resp.content)[0] 74 | self.assertEqual(resp['status_code'], 500, "Exceptions should return 500.") 75 | self.assertEqual(resp['body'].lower(), "exception", "Exception handling is broken!") 76 | -------------------------------------------------------------------------------- /tests/test_base.py: -------------------------------------------------------------------------------- 1 | ''' 2 | @author: Rahul Tanwani 3 | 4 | @summary: Contains base test case for reusable test methods. 5 | ''' 6 | import json 7 | 8 | from django.test import TestCase 9 | from batch_requests.settings import br_settings as settings 10 | 11 | 12 | class TestBase(TestCase): 13 | 14 | ''' 15 | Base class for all reusable test methods. 16 | ''' 17 | 18 | def assert_reponse_compatible(self, ind_resp, batch_resp): 19 | ''' 20 | Assert if the response of independent request is compatible with 21 | batch response. 22 | ''' 23 | # Remove duration header to compare. 24 | if settings.ADD_DURATION_HEADER: 25 | del batch_resp['headers'][settings.DURATION_HEADER_NAME] 26 | 27 | self.assertDictEqual(ind_resp, batch_resp, "Compatibility is broken!") 28 | 29 | def headers_dict(self, headers): 30 | ''' 31 | Converts the headers from the response in to a dict. 32 | ''' 33 | return dict(headers.values()) 34 | 35 | def prepare_response(self, status_code, body, headers): 36 | ''' 37 | Returns a dict of all the parameters. 38 | ''' 39 | return {"status_code": status_code, "body": body, "headers": self.headers_dict(headers)} 40 | 41 | def _batch_request(self, method, path, data, headers={}): 42 | ''' 43 | Prepares a batch request. 44 | ''' 45 | return {"url": path, "method": method, "headers": headers, "body": data} 46 | 47 | def make_a_batch_request(self, method, url, body, headers={}): 48 | ''' 49 | Makes a batch request using django client. 50 | ''' 51 | return self.client.post("/api/v1/batch/", json.dumps([self._batch_request(method, url, body, headers)]), 52 | content_type="application/json") 53 | 54 | def make_multiple_batch_request(self, requests): 55 | ''' 56 | Makes multiple batch request using django client. 57 | ''' 58 | batch_requests = [self._batch_request(method, path, data, headers) for method, path, data, headers in requests] 59 | return self.client.post("/api/v1/batch/", json.dumps(batch_requests), 60 | content_type="application/json") 61 | -------------------------------------------------------------------------------- /tests/test_compatibility.py: -------------------------------------------------------------------------------- 1 | ''' 2 | @author: Rahul Tanwani 3 | 4 | @summary: Test cases to check compatibility between individual requests 5 | and batch requests. 6 | ''' 7 | import json 8 | 9 | from tests.test_base import TestBase 10 | 11 | 12 | class TestCompatibility(TestBase): 13 | 14 | ''' 15 | Tests compatibility. 16 | ''' 17 | 18 | def test_compatibility_of_get_request(self): 19 | ''' 20 | Make a GET request without the batch and in the batch and assert 21 | that both gives the same results. 22 | ''' 23 | # Get the response for an individual request. 24 | inv_req = self.client.get("/views/") 25 | inv_resp = self.prepare_response(inv_req.status_code, inv_req.content, inv_req._headers) 26 | 27 | # Get the response for a batch request. 28 | batch_request = self.make_a_batch_request("GET", "/views/", "") 29 | batch_resp = json.loads(batch_request.content)[0] 30 | del batch_resp["reason_phrase"] 31 | 32 | # Assert both individual request response and batch response are equal. 33 | self.assert_reponse_compatible(inv_resp, batch_resp) 34 | 35 | def test_compatibility_of_post_request(self): 36 | ''' 37 | Make a POST request without the batch and in the batch and assert 38 | that both gives the same results. 39 | ''' 40 | data = json.dumps({"text": "hello"}) 41 | 42 | # Get the response for an individual request. 43 | inv_req = self.client.post("/views/", data, content_type="text/plain") 44 | inv_resp = self.prepare_response(inv_req.status_code, inv_req.content, inv_req._headers) 45 | 46 | # Get the response for a batch request. 47 | batch_request = self.make_a_batch_request("POST", "/views/", data, {"content_type": "text/plain"}) 48 | batch_resp = json.loads(batch_request.content)[0] 49 | del batch_resp["reason_phrase"] 50 | 51 | # Assert both individual request response and batch response are equal. 52 | self.assert_reponse_compatible(inv_resp, batch_resp) 53 | 54 | def test_compatibility_of_put_request(self): 55 | ''' 56 | Make a PUT request without the batch and in the batch and assert 57 | that both gives the same results. 58 | ''' 59 | data = json.dumps({"text": "hello"}) 60 | 61 | # Get the response for an individual request. 62 | inv_req = self.client.patch("/views/", data, content_type="text/plain") 63 | inv_resp = self.prepare_response(inv_req.status_code, inv_req.content, inv_req._headers) 64 | 65 | # Get the response for a batch request. 66 | batch_request = self.make_a_batch_request("patch", "/views/", data, {"content_type": "text/plain"}) 67 | batch_resp = json.loads(batch_request.content)[0] 68 | del batch_resp["reason_phrase"] 69 | 70 | # Assert both individual request response and batch response are equal. 71 | self.assert_reponse_compatible(inv_resp, batch_resp) 72 | 73 | def test_compatibility_of_patch_request(self): 74 | ''' 75 | Make a POST request without the batch and in the batch and assert 76 | that both gives the same results. 77 | ''' 78 | data = json.dumps({"text": "hello"}) 79 | 80 | # Get the response for an individual request. 81 | inv_req = self.client.post("/views/", data, content_type="text/plain") 82 | inv_resp = self.prepare_response(inv_req.status_code, inv_req.content, inv_req._headers) 83 | 84 | # Get the response for a batch request. 85 | batch_request = self.make_a_batch_request("POST", "/views/", data, {"CONTENT_TYPE": "text/plain"}) 86 | batch_resp = json.loads(batch_request.content)[0] 87 | del batch_resp["reason_phrase"] 88 | 89 | # Assert both individual request response and batch response are equal. 90 | self.assert_reponse_compatible(inv_resp, batch_resp) 91 | 92 | def test_compatibility_of_delete_request(self): 93 | ''' 94 | Make a DELETE request without the batch and in the batch and assert 95 | that both gives the same results. 96 | ''' 97 | # Get the response for an individual request. 98 | inv_req = self.client.delete("/views/") 99 | inv_resp = self.prepare_response(inv_req.status_code, inv_req.content, inv_req._headers) 100 | 101 | # Get the response for a batch request. 102 | batch_request = self.make_a_batch_request("delete", "/views/", "") 103 | batch_resp = json.loads(batch_request.content)[0] 104 | del batch_resp["reason_phrase"] 105 | 106 | # Assert both individual request response and batch response are equal. 107 | self.assert_reponse_compatible(inv_resp, batch_resp) 108 | 109 | def test_compatibility_of_multiple_requests(self): 110 | ''' 111 | Make multiple requests without the batch and in the batch and 112 | assert that both gives the same results. 113 | ''' 114 | 115 | data = json.dumps({"text": "Batch"}) 116 | # Make GET, POST and PUT requests individually. 117 | 118 | # Get the response for an individual GET request. 119 | inv_req = self.client.get("/views/") 120 | inv_get = self.prepare_response(inv_req.status_code, inv_req.content, inv_req._headers) 121 | 122 | # Get the response for an individual POST request. 123 | inv_req = self.client.post("/views/", data, content_type="text/plain") 124 | inv_post = self.prepare_response(inv_req.status_code, inv_req.content, inv_req._headers) 125 | 126 | # Get the response for an individual PUT request. 127 | inv_req = self.client.patch("/views/", data, content_type="text/plain") 128 | inv_put = self.prepare_response(inv_req.status_code, inv_req.content, inv_req._headers) 129 | 130 | # Consolidate all the responses. 131 | indv_responses = [inv_get, inv_post, inv_put] 132 | 133 | # Make a batch call for GET, POST and PUT request. 134 | get_req = ("get", "/views/", '', {}) 135 | post_req = ("post", "/views/", data, {"content_type": "text/plain"}) 136 | put_req = ("put", "/views/", data, {"content_type": "text/plain"}) 137 | 138 | # Get the response for a batch request. 139 | batch_requests = self.make_multiple_batch_request([get_req, post_req, put_req]) 140 | batch_responses = json.loads(batch_requests.content) 141 | 142 | # Assert all the responses are compatible. 143 | for indv_resp, batch_resp in zip(indv_responses, batch_responses): 144 | del batch_resp["reason_phrase"] 145 | self.assert_reponse_compatible(indv_resp, batch_resp) 146 | -------------------------------------------------------------------------------- /tests/test_concurrency_base.py: -------------------------------------------------------------------------------- 1 | ''' 2 | @author: Rahul Tanwani 3 | 4 | @summary: Contains base test case for concurrency related tests. 5 | ''' 6 | import json 7 | 8 | from tests.test_base import TestBase 9 | from batch_requests.settings import br_settings 10 | 11 | 12 | class TestBaseConcurrency(TestBase): 13 | ''' 14 | Base class for all reusable test methods related to concurrency. 15 | ''' 16 | # FIXME: Find the better way to manage / update settings. 17 | def setUp(self): 18 | ''' 19 | Change the concurrency settings. 20 | ''' 21 | self.number_workers = 10 22 | self.orig_executor = br_settings.executor 23 | 24 | def tearDown(self): 25 | # Restore the original batch requests settings. 26 | br_settings.executor = self.orig_executor 27 | 28 | def compare_seq_and_concurrent_req(self): 29 | ''' 30 | Make a request with sequential and concurrency based executor and compare 31 | the response. 32 | ''' 33 | data = json.dumps({"text": "Batch"}) 34 | 35 | # Make a batch call for GET, POST and PUT request. 36 | get_req = ("get", "/views/", '', {}) 37 | post_req = ("post", "/views/", data, {"content_type": "text/plain"}) 38 | put_req = ("put", "/views/", data, {"content_type": "text/plain"}) 39 | 40 | # Get the response for a batch request. 41 | batch_requests = self.make_multiple_batch_request([get_req, post_req, put_req]) 42 | 43 | # FIXME: Find the better way to manage / update settings. 44 | # Update the settings. 45 | br_settings.executor = self.get_executor() 46 | threaded_batch_requests = self.make_multiple_batch_request([get_req, post_req, put_req]) 47 | 48 | seq_responses = json.loads(batch_requests.content) 49 | conc_responses = json.loads(threaded_batch_requests.content) 50 | 51 | for idx, seq_resp in enumerate(seq_responses): 52 | self.assertDictEqual(seq_resp, conc_responses[idx], "Sequential and concurrent response not same!") 53 | 54 | def compare_seq_concurrent_duration(self): 55 | ''' 56 | Makes the batch requests run sequentially and in parallel and asserts 57 | parallelism to reduce the total duration time. 58 | ''' 59 | # Make a batch call for GET, POST and PUT request. 60 | sleep_2_seconds = ("get", "/sleep/?seconds=1", '', {}) 61 | sleep_1_second = ("get", "/sleep/?seconds=1", '', {}) 62 | 63 | # Get the response for a batch request. 64 | batch_requests = self.make_multiple_batch_request([sleep_2_seconds, sleep_1_second, sleep_2_seconds]) 65 | seq_duration = int(batch_requests._headers.get(br_settings.DURATION_HEADER_NAME)[1]) 66 | 67 | # Update the executor settings. 68 | br_settings.executor = self.get_executor() 69 | concurrent_batch_requests = self.make_multiple_batch_request([sleep_2_seconds, sleep_1_second, sleep_2_seconds]) 70 | concurrency_duration = int(concurrent_batch_requests._headers.get(br_settings.DURATION_HEADER_NAME)[1]) 71 | 72 | self.assertLess(concurrency_duration, seq_duration, "Concurrent requests are slower than running them in sequence.") 73 | -------------------------------------------------------------------------------- /tests/test_process_concurrency.py: -------------------------------------------------------------------------------- 1 | ''' 2 | @author: Rahul Tanwani 3 | 4 | @summary: Test cases to make sure sequential execution and process based concurrent execution return 5 | the same response. 6 | ''' 7 | from tests.test_concurrency_base import TestBaseConcurrency 8 | from batch_requests.concurrent.executor import ProcessBasedExecutor 9 | 10 | 11 | class TestProcessConcurrency(TestBaseConcurrency): 12 | ''' 13 | Tests sequential and concurrent process based execution. 14 | ''' 15 | def get_executor(self): 16 | ''' 17 | Returns the executor to use for running tests defined in this suite. 18 | ''' 19 | return ProcessBasedExecutor(self.number_workers) 20 | 21 | def test_thread_concurrency_response(self): 22 | ''' 23 | Make a request with sequential and process based concurrent executor and compare 24 | the response. 25 | ''' 26 | self.compare_seq_and_concurrent_req() 27 | 28 | def test_duration(self): 29 | ''' 30 | Compare that running tests with ProcessBasedConcurreny return faster than running 31 | them sequentially. 32 | ''' 33 | self.compare_seq_concurrent_duration() 34 | -------------------------------------------------------------------------------- /tests/test_settings.py: -------------------------------------------------------------------------------- 1 | ''' 2 | @author: Rahul Tanwani 3 | 4 | @summary: Test cases for configured settings. 5 | ''' 6 | import json 7 | 8 | from django.conf import settings 9 | from batch_requests.settings import br_settings 10 | from django.test import TestCase 11 | 12 | 13 | class TestSettings(TestCase): 14 | 15 | ''' 16 | Tests cases to make sure settings are working as configured. 17 | ''' 18 | 19 | def test_max_limit(self): 20 | ''' 21 | Test max_limit should enforced. 22 | ''' 23 | data = json.dumps({"text": "test max limit"}) 24 | 25 | # Make a batch call for requests > MAX_LIMIT (3). 26 | get_req = ("get", "/views/", '', {}) 27 | post_req = ("post", "/views/", data, {"content_type": "text/plain"}) 28 | put_req = ("put", "/views/", data, {"content_type": "text/plain"}) 29 | get_req2 = ("get", "/views/", '', {}) 30 | 31 | # Get the response for a batch request. 32 | batch_requests = self._make_multiple_batch_request([get_req, post_req, put_req, get_req2]) 33 | 34 | # Assert we get a bad request. 35 | self.assertEqual(batch_requests.status_code, 400, "MAX_LIMIT setting not working.") 36 | self.assertTrue(batch_requests.content.lower().startswith("you can batch maximum of")) 37 | 38 | def test_custom_header(self): 39 | ''' 40 | Tests the custom header to be present in individual request. 41 | ''' 42 | # Define the header and its value 43 | header = "X_CUSTOM" 44 | value = "custom value" 45 | 46 | # Make a batch request querying for that particular header. 47 | batch_req = self._make_a_batch_request("get", "/echo/?header=HTTP_%s" % (header), "", headers={header: value}) 48 | batch_resp = json.loads(batch_req.content)[0] 49 | 50 | self.assertEqual(batch_resp['body'], value, "Custom header not working") 51 | 52 | def test_use_https_setting(self): 53 | ''' 54 | Tests the scheme for individual requests to be https as defined in settings. 55 | ''' 56 | # Define the header and its value 57 | header = "scheme" 58 | value = "https" 59 | 60 | # Make a batch request querying for that particular header. 61 | batch_req = self._make_a_batch_request("get", "/echo/?header=%s" % (header), "", headers={}) 62 | batch_resp = json.loads(batch_req.content)[0] 63 | 64 | self.assertEqual(batch_resp['body'], value, "Custom header not working") 65 | 66 | def test_http_default_content_type_header(self): 67 | ''' 68 | Tests the default content type to be as per settings.DEFAULT_CONTENT_TYPE. 69 | ''' 70 | data = json.dumps({"text": "content type"}) 71 | 72 | # Define the header and its value 73 | header = "CONTENT_TYPE" 74 | value = settings.BATCH_REQUESTS.get("DEFAULT_CONTENT_TYPE") 75 | 76 | # Make a batch request querying for that particular header. 77 | batch_req = self._make_a_batch_request("post", "/echo/?header=%s" % (header), data, headers={}) 78 | batch_resp = json.loads(batch_req.content)[0] 79 | 80 | self.assertEqual(batch_resp['body'], value, "Default content type not working.") 81 | 82 | def test_duration_header(self): 83 | ''' 84 | Tests that duration header be present in all the requests. 85 | ''' 86 | # Make a batch call to any API. 87 | get_req = ("get", "/views/", '', {}) 88 | 89 | # Get the response for a batch request. 90 | batch_requests = self._make_multiple_batch_request([get_req]) 91 | 92 | # Make sure we have the header present in enclosing batch response and all individual responses. 93 | self.assertIn(br_settings.DURATION_HEADER_NAME, batch_requests._headers, 94 | "Enclosing batch request does not contain duration header.") 95 | 96 | self.assertIn(br_settings.DURATION_HEADER_NAME, json.loads(batch_requests.content)[0]['headers'], 97 | "Individual batch request does not contain duration header.") 98 | 99 | def _batch_request(self, method, path, data, headers={}): 100 | ''' 101 | Prepares a batch request. 102 | ''' 103 | return {"url": path, "method": method, "headers": headers, "body": data} 104 | 105 | def _make_a_batch_request(self, method, url, body, headers={}): 106 | ''' 107 | Makes a batch request using django client. 108 | ''' 109 | return self.client.post("/api/v1/batch/", json.dumps([self._batch_request(method, url, body, headers)]), 110 | content_type="application/json") 111 | 112 | def _make_multiple_batch_request(self, requests): 113 | ''' 114 | Makes multiple batch request using django client. 115 | ''' 116 | batch_requests = [self._batch_request(method, path, data, headers) for method, path, data, headers in requests] 117 | return self.client.post("/api/v1/batch/", json.dumps(batch_requests), content_type="application/json") 118 | -------------------------------------------------------------------------------- /tests/test_thread_concurrency.py: -------------------------------------------------------------------------------- 1 | ''' 2 | @author: Rahul Tanwani 3 | 4 | @summary: Test cases to make sure sequential execution and concurrent execution return 5 | the same response. 6 | ''' 7 | from tests.test_concurrency_base import TestBaseConcurrency 8 | from batch_requests.concurrent.executor import ThreadBasedExecutor 9 | 10 | 11 | class TestThreadConcurrency(TestBaseConcurrency): 12 | ''' 13 | Tests sequential and concurrent execution. 14 | ''' 15 | def get_executor(self): 16 | ''' 17 | Returns the executor to use for running tests defined in this suite. 18 | ''' 19 | return ThreadBasedExecutor(self.number_workers) 20 | 21 | def test_thread_concurrency_response(self): 22 | ''' 23 | Make a request with sequential and concurrency based executor and compare 24 | the response. 25 | ''' 26 | self.compare_seq_and_concurrent_req() 27 | 28 | def test_duration(self): 29 | ''' 30 | Compare that running tests with ThreadBasedConcurreny return faster than running 31 | them sequentially. 32 | ''' 33 | self.compare_seq_concurrent_duration() 34 | -------------------------------------------------------------------------------- /tests/test_views.py: -------------------------------------------------------------------------------- 1 | ''' 2 | @author: rahul 3 | ''' 4 | import json 5 | 6 | from django.http.response import HttpResponse 7 | from django.views.decorators.csrf import csrf_exempt 8 | from django.views.generic import View 9 | from time import sleep 10 | 11 | 12 | class SimpleView(View): 13 | 14 | ''' 15 | A simple view with dummy implementation for all the methods. 16 | The view is only intended to be used for testing. 17 | ''' 18 | 19 | def get(self, request): 20 | ''' 21 | Handles GET requests. 22 | ''' 23 | return HttpResponse("Success!") 24 | 25 | def post(self, request): 26 | ''' 27 | Handles POST requests. 28 | This implementation just echos back the data coming in. 29 | ''' 30 | return HttpResponse(status=201, content=request.body) 31 | 32 | def put(self, request): 33 | ''' 34 | Handles PUT requests 35 | ''' 36 | # Imaginary current view of data. 37 | data = {"method": "PUT", "status": 202, "text": "Updated"} 38 | data.update(json.loads(request.body)) 39 | return HttpResponse(status=202, content=json.dumps(data)) 40 | 41 | def patch(self, request): 42 | ''' 43 | Handles PATCH requests 44 | ''' 45 | # Imaginary current view of data. 46 | data = {"method": "PUT", "status": 202, "text": "Updated"} 47 | data.update(json.loads(request.body)) 48 | return HttpResponse(status=202, content=json.dumps(data)) 49 | 50 | def delete(self, request): 51 | ''' 52 | Handles delete requests 53 | ''' 54 | return HttpResponse(status=202, content="No Content!") 55 | 56 | def head(self, request): 57 | ''' 58 | Handles head requests. 59 | ''' 60 | return HttpResponse() 61 | 62 | @csrf_exempt 63 | def dispatch(self, *args, **kwargs): 64 | ''' 65 | Overiding to exempt csrf. 66 | ''' 67 | return super(SimpleView, self).dispatch(*args, **kwargs) 68 | 69 | 70 | class EchoHeaderView(View): 71 | 72 | ''' 73 | Echos back the header value. 74 | ''' 75 | 76 | def get(self, request, *args, **kwargs): 77 | ''' 78 | Handles the get request. 79 | ''' 80 | # Lookup for the header client is requesting for. 81 | header = request.GET.get("header", None) 82 | 83 | # Get the value for the associated header. 84 | value = getattr(request, header, None) 85 | 86 | # If header is not an attribute of request, look into request.META 87 | if value is None: 88 | value = request.META.get(header, None) 89 | 90 | return HttpResponse(value) 91 | 92 | def post(self, request, *args, **kwargs): 93 | ''' 94 | Delegates to the get request. 95 | ''' 96 | return self.get(request, *args, **kwargs) 97 | 98 | 99 | class ExceptionView(View): 100 | 101 | ''' 102 | Views that raises exception for testing purpose. 103 | ''' 104 | 105 | def get(self, request, *args, **kwargs): 106 | ''' 107 | Handles the get request. 108 | ''' 109 | raise Exception("exception") 110 | 111 | 112 | class SleepingView(View): 113 | 114 | ''' 115 | Make the current thread sleep for the number of seconds passed. 116 | This is to mimic the long running services. 117 | ''' 118 | 119 | def get(self, request, *args, **kwargs): 120 | ''' 121 | Handles the get request. 122 | ''' 123 | # Lookup for the duration to sleep. 124 | seconds = int(request.GET.get("seconds", "5")) 125 | 126 | # Make the current thread sleep for the specified duration. 127 | sleep(seconds) 128 | # Make the current thread sleep. 129 | return HttpResponse("Success!") 130 | -------------------------------------------------------------------------------- /tests/urls.py: -------------------------------------------------------------------------------- 1 | from django.conf.urls import patterns, url 2 | 3 | from batch_requests.views import handle_batch_requests 4 | from tests.test_views import SimpleView, EchoHeaderView, ExceptionView,\ 5 | SleepingView 6 | 7 | 8 | urlpatterns = patterns('', 9 | url(r'^views/', SimpleView.as_view()), 10 | url(r'^echo/', EchoHeaderView.as_view()), 11 | url(r'^exception/', ExceptionView.as_view()), 12 | url(r'^sleep/', SleepingView.as_view()), 13 | url(r'^api/v1/batch/', handle_batch_requests), 14 | ) 15 | -------------------------------------------------------------------------------- /tox.ini: -------------------------------------------------------------------------------- 1 | [tox] 2 | envlist = py27-{tests,flake8} 3 | 4 | [testenv:py27-tests] 5 | deps = 6 | -rrequirements.txt 7 | commands = ./runtests.py --fast 8 | 9 | [testenv:py27-flake8] 10 | deps = 11 | -rrequirements.txt 12 | commands = ./runtests.py --lintonly 13 | --------------------------------------------------------------------------------