├── .hgignore ├── AUTHORS ├── LICENSE ├── MANIFEST.in ├── README.rst ├── docs ├── conf.py └── index.rst ├── fuzzycount.py ├── setup.cfg └── setup.py /.hgignore: -------------------------------------------------------------------------------- 1 | syntax: glob 2 | *.pyc 3 | *.pyo 4 | .DS_Store 5 | build 6 | dist 7 | *.egg-info 8 | -------------------------------------------------------------------------------- /AUTHORS: -------------------------------------------------------------------------------- 1 | * Stephen McDonald 2 | * Mohammad Tayseer 3 | * Mohammad Taleb 4 | * Andrew Cordery 5 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) Stephen McDonald and individual contributors. 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without modification, 5 | are permitted provided that the following conditions are met: 6 | 7 | 1. Redistributions of source code must retain the above copyright notice, 8 | this list of conditions and the following disclaimer. 9 | 10 | 2. Redistributions in binary form must reproduce the above copyright 11 | notice, this list of conditions and the following disclaimer in the 12 | documentation and/or other materials provided with the distribution. 13 | 14 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 15 | ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 16 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 17 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR 18 | ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 19 | (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 20 | LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON 21 | ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 22 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 23 | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 24 | 25 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include LICENSE 2 | include README.rst 3 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | Created by `Stephen McDonald `_ 2 | 3 | Introduction 4 | ============ 5 | 6 | Up until PostgreSQL 9.2, ``COUNT`` queries generally required scanning 7 | every row in a database table. With millions of rows, this can become 8 | quite slow. One work-around for this is to query statistics in 9 | PostgreSQL for an approximate row count, which in many cases is an 10 | acceptable trade-off. 11 | 12 | Given a table called ``bigdata``, the following query will return an 13 | approximate row count:: 14 | 15 | SELECT reltuples FROM pg_class WHERE relname = 'bigdata'; 16 | 17 | You can read more information about `slow COUNT queries in PostgreSQL`_ 18 | in the PostgreSQL wiki. 19 | 20 | What ``django-postgres-fuzzycount`` provides is a way of using this 21 | approach directly in your Django model managers. It was originally 22 | built for displaying statistics in the `kouio RSS reader`_, a popular alternative to Google Reader, that acquired over 5 million news articles 23 | in its database during the first week of its launch. 24 | 25 | Installation 26 | ============ 27 | 28 | The easiest way to install ``django-postgres-fuzzycount`` is directly 29 | from PyPi using `pip`_ by running the following command:: 30 | 31 | $ pip install -U django-postgres-fuzzycount 32 | 33 | Otherwise you can download and install it directly from source:: 34 | 35 | $ python setup.py install 36 | 37 | Usage 38 | ===== 39 | 40 | By using the ``fuzzycount.FuzzyCountManager`` on your Django models, 41 | its ``count()`` method will return an approximate value when querying 42 | PostgreSQL tables without any ``WHERE`` OR ``HAVING`` clauses:: 43 | 44 | from django.db import models 45 | from fuzzycount import FuzzyCountManager 46 | 47 | class BigData(models.Model): 48 | 49 | big = models.BooleanField(default=True) 50 | data = models.TextField() 51 | 52 | objects = FuzzyCountManager() 53 | 54 | BigData.objects.count() # Uses fuzzycount 55 | BigData.objects.filter(id__gt=9000).count() # Doesn't use fuzzycount 56 | 57 | The ``fuzzycount.FuzzyCountManager`` also checks the database engine 58 | being used, and only applies the approximate count query when using 59 | PostgreSQL, so other database backends can be used and will behave as 60 | usual (for varying definitions of `usual`, depending on the database :-). 61 | 62 | Inspiration 63 | =========== 64 | 65 | * `postgres_loose_table_counts`_, a Ruby gem providing the same 66 | approach for Rails 67 | * This `Django snippet`_, which bakes the approach into the admin 68 | 69 | .. _`slow COUNT queries in PostgreSQL`: http://wiki.postgresql.org/wiki/Slow_Counting 70 | .. _`kouio RSS reader`: https://kouio.com 71 | .. _`pip`: http://www.pip-installer.org/ 72 | .. _`postgres_loose_table_counts`: https://github.com/goodfilms/postgres_loose_table_counts 73 | .. _`Django snippet`: http://djangosnippets.org/snippets/2855/ 74 | -------------------------------------------------------------------------------- /docs/conf.py: -------------------------------------------------------------------------------- 1 | # This file is automatically generated via sphinx-me 2 | from sphinx_me import setup_conf; setup_conf(globals()) 3 | -------------------------------------------------------------------------------- /docs/index.rst: -------------------------------------------------------------------------------- 1 | .. include:: ../README.rst -------------------------------------------------------------------------------- /fuzzycount.py: -------------------------------------------------------------------------------- 1 | from distutils.version import LooseVersion 2 | 3 | from django.conf import settings 4 | from django.db import connections 5 | from django.db.models import QuerySet, Manager 6 | import django 7 | 8 | 9 | DJANGO_VERSION_GTE_19 = LooseVersion(django.get_version()) \ 10 | >= LooseVersion('1.9') 11 | 12 | 13 | class FuzzyCountQuerySet(QuerySet): 14 | def count(self): 15 | postgres_engines = ("postgis", "postgresql", "django_postgrespool") 16 | engine = settings.DATABASES[self.db]["ENGINE"].split(".")[-1] 17 | is_postgres = engine.startswith(postgres_engines) 18 | 19 | # In Django 1.9 the query.having property was removed and the 20 | # query.where property will be truthy if either where or having 21 | # clauses are present. In earlier versions these were two separate 22 | # properties query.where and query.having 23 | if DJANGO_VERSION_GTE_19: 24 | is_filtered = self.query.where 25 | else: 26 | is_filtered = self.query.where or self.query.having 27 | if not is_postgres or is_filtered: 28 | return super(FuzzyCountQuerySet, self).count() 29 | cursor = connections[self.db].cursor() 30 | cursor.execute("SELECT reltuples FROM pg_class " 31 | "WHERE relname = '%s';" % self.model._meta.db_table) 32 | return int(cursor.fetchone()[0]) 33 | 34 | 35 | FuzzyCountManager = Manager.from_queryset(FuzzyCountQuerySet) 36 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [wheel] 2 | universal = 1 3 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from setuptools import setup 2 | 3 | 4 | setup( 5 | name = "django-postgres-fuzzycount", 6 | version = "0.1.7", 7 | author = "Stephen McDonald", 8 | author_email = "stephen.mc@gmail.com", 9 | description = ("A Django model manager providing fast / fuzzy counts " 10 | "for PostgreSQL database tables."), 11 | long_description = open("README.rst").read(), 12 | url = "http://github.com/stephenmcd/django-postgres-fuzzycount", 13 | zip_safe = False, 14 | py_modules=["fuzzycount",], 15 | install_requires = [ 16 | "sphinx-me >= 0.1.2", 17 | "django >= 1.7", 18 | ], 19 | classifiers = [ 20 | "Development Status :: 5 - Production/Stable", 21 | "Environment :: Web Environment", 22 | "Intended Audience :: Developers", 23 | "Operating System :: OS Independent", 24 | "Programming Language :: Python", 25 | "Programming Language :: Python :: 2.6", 26 | "Programming Language :: Python :: 2.7", 27 | "Programming Language :: Python :: 3", 28 | "Programming Language :: Python :: 3.3", 29 | "Framework :: Django", 30 | "Topic :: Internet :: WWW/HTTP :: Dynamic Content", 31 | "Topic :: Internet :: WWW/HTTP :: Site Management", 32 | ] 33 | ) 34 | --------------------------------------------------------------------------------