├── .gitignore ├── LICENSE ├── MANIFEST.in ├── Makefile ├── README.rst ├── django_mysqlpool ├── __init__.py └── backends │ ├── __init__.py │ └── mysqlpool │ ├── __init__.py │ └── base.py ├── setup.cfg └── setup.py /.gitignore: -------------------------------------------------------------------------------- 1 | .tox 2 | *.egg-info 3 | *.pyc 4 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2015, SmartFile 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of 4 | this software and associated documentation files (the "Software"), to deal in 5 | the Software without restriction, including without limitation the rights to 6 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies 7 | of the Software, and to permit persons to whom the Software is furnished to do 8 | so, subject to the following conditions: 9 | 10 | The above copyright notice and this permission notice shall be included in all 11 | copies or substantial portions of the Software. 12 | 13 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 14 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 15 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 16 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 17 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 18 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 19 | SOFTWARE. 20 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include README.rst LICENSE 2 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | test: 2 | python tests.py 3 | 4 | verify: 5 | pyflakes -x W django_mysqlpool 6 | pep8 --exclude=migrations --ignore=E501,E225 django_myqlpool 7 | 8 | install: 9 | python setup.py install 10 | 11 | publish: 12 | python setup.py register 13 | python setup.py sdist upload 14 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | django-mysqlpool 2 | ================ 3 | 4 | Introduction 5 | ------------ 6 | 7 | This is a simple Django database backend that pools MySQL connections. This 8 | backend is based on a blog post by Ed Menendez. 9 | 10 | http://menendez.com/blog/mysql-connection-pooling-django-and-sqlalchemy/ 11 | 12 | The main differences being: 13 | 14 | 1. The work is done for you. 15 | 2. Instead of cloning the Django mysql backend, we monkey-patch it. 16 | 17 | The second point sounds bad, but it is the best option because it does not 18 | freeze the Django MySQL backend at a specific revision. Using this method 19 | allows us to benefit from any bugs that the Django project fixes, while 20 | layering on connection pooling. 21 | 22 | The actual pooling is done using SQLAlchemy. While imperfect (this backend 23 | is per-process only) it has usefulness. The main problem it solves for us 24 | is that it restricts a process to a certain number of total connections. 25 | 26 | Usage 27 | ----- 28 | 29 | Configure this backend instead of the default Django mysql backend. 30 | 31 | :: 32 | 33 | DATABASES = { 34 | 'default': { 35 | 'ENGINE': 'django_mysqlpool.backends.mysqlpool', 36 | 'NAME': 'db_name', 37 | 'USER': 'username', 38 | 'PASSWORD': 'password', 39 | 'HOST': '', 40 | 'PORT': '', 41 | }, 42 | } 43 | 44 | Configuration 45 | ------------- 46 | 47 | You can define the pool implementation and the specific arguments passed to it. 48 | The available implementations (backends) and their arguments are defined within 49 | the SQLAlchemy documentation. 50 | 51 | http://docs.sqlalchemy.org/en/rel_0_7/core/pooling.html 52 | 53 | * MYSQLPOOL_BACKEND - The pool implementation name ('QueuePool' by default). 54 | * MYSQLPOOL_ARGUMENTS - The kwargs passed to the pool. 55 | 56 | For example, to use a QueuePool without threadlocal, you could use the following 57 | configuration:: 58 | 59 | MYSQLPOOL_BACKEND = 'QueuePool' 60 | MYSQLPOOL_ARGUMENTS = { 61 | 'use_threadlocal': False, 62 | } 63 | 64 | Connection Closing 65 | ------------------ 66 | 67 | While this has nothing to do directly with connection pooling, it is tangentially 68 | related. Once you start pooling (and limiting) the database connections it 69 | becomes important to close them. 70 | 71 | This is really only relevant when you are dealing with a threaded application. Such 72 | was the case for one of our servers. It would create many threads for handling 73 | conncurrent operations. Each thread resulted in a connection to the database being 74 | opened persistently. Once we deployed connection pooling, this service quickly 75 | exhausted the connection limit of it's pool. 76 | 77 | This sounds like a huge failure, but for us it was a great success. The reason is 78 | that we implemented pooling specifically to limit each process to a certain 79 | number of connections. This prevents any given process from impacting other 80 | services, turning a global issue into a local issue. Once we were able to identify 81 | the specific service that was abusing our MySQL server, we were able to fix it. 82 | 83 | The problem we were having with this threaded server is very well described below. 84 | 85 | http://stackoverflow.com/questions/1303654/threaded-django-task-doesnt-automatically-handle-transactions-or-db-connections 86 | 87 | Therefore, this library provides a decorator that can be used in a similar situation 88 | to help with connection management. You can use it like so:: 89 | 90 | from django_mysqlpool import auto_close_db 91 | 92 | @auto_close_db 93 | def function_that_uses_db(): 94 | MyModel.objects.all().delete() 95 | 96 | With pooling (and threads), closing the connection early and often is the key to good 97 | performance. Closing returns the connection to the pool to be reused, thus the total 98 | number of connections is decreased. We also needed to disable the `use_threadlocal` 99 | option of the QueuePool, so that multiple threads could share the same connection. 100 | Once we decorated all functions that utilized a connection, this service used less 101 | connections than it's total thread count. 102 | 103 | .. _SmartFile: http://www.smartfile.com/ 104 | .. _Read more: http://www.smartfile.com/open-source.html 105 | -------------------------------------------------------------------------------- /django_mysqlpool/__init__.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """The top-level package for ``django-mysqlpool``.""" 3 | # These imports make 2 act like 3, making it easier on us to switch to PyPy or 4 | # some other VM if we need to for performance reasons. 5 | from __future__ import (absolute_import, print_function, unicode_literals, 6 | division) 7 | 8 | # Make ``Foo()`` work the same in Python 2 as it does in Python 3. 9 | __metaclass__ = type 10 | 11 | 12 | from functools import wraps 13 | 14 | 15 | __version__ = "0.2.1" 16 | 17 | 18 | def auto_close_db(f): 19 | "Ensures the database connection is closed when the function returns." 20 | from django.db import connections 21 | @wraps(f) 22 | def wrapper(*args, **kwargs): 23 | try: 24 | return f(*args, **kwargs) 25 | finally: 26 | for connection in connections.all(): 27 | connection.close() 28 | return wrapper 29 | -------------------------------------------------------------------------------- /django_mysqlpool/backends/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/smartfile/django-mysqlpool/2b5709116e03bd9331ee304350141f210cd285de/django_mysqlpool/backends/__init__.py -------------------------------------------------------------------------------- /django_mysqlpool/backends/mysqlpool/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/smartfile/django-mysqlpool/2b5709116e03bd9331ee304350141f210cd285de/django_mysqlpool/backends/mysqlpool/__init__.py -------------------------------------------------------------------------------- /django_mysqlpool/backends/mysqlpool/base.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """The top-level package for ``django-mysqlpool``.""" 3 | # These imports make 2 act like 3, making it easier on us to switch to PyPy or 4 | # some other VM if we need to for performance reasons. 5 | from __future__ import (absolute_import, print_function, unicode_literals, 6 | division) 7 | 8 | # Make ``Foo()`` work the same in Python 2 as it does in Python 3. 9 | __metaclass__ = type 10 | 11 | 12 | import os 13 | 14 | 15 | from django.conf import settings 16 | from django.db.backends.mysql import base 17 | from django.core.exceptions import ImproperlyConfigured 18 | 19 | try: 20 | import sqlalchemy.pool as pool 21 | except ImportError as e: 22 | raise ImproperlyConfigured("Error loading SQLAlchemy module: %s" % e) 23 | 24 | 25 | # Global variable to hold the actual connection pool. 26 | MYSQLPOOL = None 27 | # Default pool type (QueuePool, SingletonThreadPool, AssertionPool, NullPool, 28 | # StaticPool). 29 | DEFAULT_BACKEND = 'QueuePool' 30 | # Needs to be less than MySQL connection timeout (server setting). The default 31 | # is 120, so default to 119. 32 | DEFAULT_POOL_TIMEOUT = 119 33 | 34 | 35 | def isiterable(value): 36 | """Determine whether ``value`` is iterable.""" 37 | try: 38 | iter(value) 39 | return True 40 | except TypeError: 41 | return False 42 | 43 | 44 | class OldDatabaseProxy(): 45 | 46 | """Saves a reference to the old connect function. 47 | 48 | Proxies calls to its own connect() method to the old function. 49 | """ 50 | 51 | def __init__(self, old_connect): 52 | """Store ``old_connect`` to be used whenever we connect.""" 53 | self.old_connect = old_connect 54 | 55 | def connect(self, **kwargs): 56 | """Delegate to the old ``connect``.""" 57 | # Bounce the call to the old function. 58 | return self.old_connect(**kwargs) 59 | 60 | 61 | class HashableDict(dict): 62 | 63 | """A dictionary that is hashable. 64 | 65 | This is not generally useful, but created specifically to hold the ``conv`` 66 | parameter that needs to be passed to MySQLdb. 67 | """ 68 | 69 | def __hash__(self): 70 | """Calculate the hash of this ``dict``. 71 | 72 | The hash is determined by converting to a sorted tuple of key-value 73 | pairs and hashing that. 74 | """ 75 | items = [(n, tuple(v)) for n, v in self.items() if isiterable(v)] 76 | return hash(tuple(items)) 77 | 78 | 79 | # Define this here so Django can import it. 80 | DatabaseWrapper = base.DatabaseWrapper 81 | 82 | 83 | # Wrap the old connect() function so our pool can call it. 84 | OldDatabase = OldDatabaseProxy(base.Database.connect) 85 | 86 | 87 | def get_pool(): 88 | """Create one and only one pool using the configured settings.""" 89 | global MYSQLPOOL 90 | if MYSQLPOOL is None: 91 | backend_name = getattr(settings, 'MYSQLPOOL_BACKEND', DEFAULT_BACKEND) 92 | backend = getattr(pool, backend_name) 93 | kwargs = getattr(settings, 'MYSQLPOOL_ARGUMENTS', {}) 94 | kwargs.setdefault('poolclass', backend) 95 | kwargs.setdefault('recycle', DEFAULT_POOL_TIMEOUT) 96 | MYSQLPOOL = pool.manage(OldDatabase, **kwargs) 97 | setattr(MYSQLPOOL, '_pid', os.getpid()) 98 | 99 | if getattr(MYSQLPOOL, '_pid', None) != os.getpid(): 100 | pool.clear_managers() 101 | return MYSQLPOOL 102 | 103 | 104 | def connect(**kwargs): 105 | """Obtain a database connection from the connection pool.""" 106 | # SQLAlchemy serializes the parameters to keep unique connection 107 | # parameter groups in their own pool. We need to store certain 108 | # values in a manner that is compatible with their serialization. 109 | conv = kwargs.pop('conv', None) 110 | ssl = kwargs.pop('ssl', None) 111 | if conv: 112 | kwargs['conv'] = HashableDict(conv) 113 | 114 | if ssl: 115 | kwargs['ssl'] = HashableDict(ssl) 116 | 117 | # Open the connection via the pool. 118 | return get_pool().connect(**kwargs) 119 | 120 | 121 | # Monkey-patch the regular mysql backend to use our hacked-up connect() 122 | # function. 123 | base.Database.connect = connect 124 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [bdist_wheel] 2 | universal = 1 3 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/bin/env python 2 | # -*- coding: utf-8 -*- 3 | """The setup script for ``django_mysqlpool``.""" 4 | # These imports make 2 act like 3, making it easier on us to switch to PyPy or 5 | # some other VM if we need to for performance reasons. 6 | from __future__ import (absolute_import, print_function, unicode_literals, 7 | division) 8 | 9 | # Make ``Foo()`` work the same in Python 2 as it does in Python 3. 10 | __metaclass__ = type 11 | 12 | 13 | import re 14 | from setuptools import setup, find_packages 15 | 16 | 17 | REQUIRES = [ 18 | "sqlalchemy >=0.7, <1.0", 19 | ] 20 | 21 | 22 | def find_version(fname): 23 | """Attempt to find the version number in the file names fname. 24 | 25 | Raises ``RuntimeError`` if not found. 26 | """ 27 | version = "" 28 | with open(fname, "r") as fp: 29 | reg = re.compile(r'__version__ = [\'"]([^\'"]*)[\'"]') 30 | for line in fp: 31 | m = reg.match(line) 32 | if m: 33 | version = m.group(1) 34 | break 35 | if not version: 36 | raise RuntimeError("Cannot find version information") 37 | return version 38 | 39 | 40 | __version__ = find_version("django_mysqlpool/__init__.py") 41 | 42 | 43 | def read(fname): 44 | """Return the contents of the file-like ``fname``.""" 45 | with open(fname) as fp: 46 | content = fp.read() 47 | return content 48 | 49 | setup( 50 | name="django-mysqlpool", 51 | version=__version__, 52 | description="Django database backend for MySQL that provides pooling ala SQLAlchemy.", # noqa 53 | long_description=read("README.rst"), 54 | author=["Ben Timby", "Hank Gay"], 55 | author_email=["btimby@gmail.com", "hank@rescuetime.com"], 56 | maintainer=["Ben Timby", "Hank Gay"], 57 | maintainer_email=["btimby@gmail.com", "hank@rescuetime.com"], 58 | url="https://github.com/gthank/django-mysqlpool", 59 | install_requires=REQUIRES, 60 | license=read("LICENSE"), 61 | zip_safe=False, 62 | classifiers=[ 63 | "License :: OSI Approved :: MIT License", 64 | "Development Status :: 4 - Beta", 65 | "Intended Audience :: Developers", 66 | "Operating System :: OS Independent", 67 | "Programming Language :: Python", 68 | "Topic :: Software Development :: Libraries :: Python Modules", 69 | "Programming Language :: Python :: 2", 70 | "Programming Language :: Python :: 2.7", 71 | "Programming Language :: Python :: 3", 72 | "Programming Language :: Python :: 3.4", 73 | "Programming Language :: Python :: Implementation :: CPython", 74 | "Programming Language :: Python :: Implementation :: PyPy" 75 | ], 76 | packages=find_packages(), 77 | ) 78 | --------------------------------------------------------------------------------