├── .editorconfig ├── .gitignore ├── .travis.yml ├── AUTHORS.rst ├── CONTRIBUTING.rst ├── HISTORY.rst ├── LICENSE ├── MANIFEST.in ├── README.rst ├── django_fulltext_search.py ├── requirements-dev.txt ├── requirements.txt ├── setup.cfg ├── setup.py ├── test_django_fulltext_search.py └── tox.ini /.editorconfig: -------------------------------------------------------------------------------- 1 | # http://editorconfig.org 2 | 3 | root = true 4 | 5 | [*] 6 | indent_style = space 7 | indent_size = 4 8 | trim_trailing_whitespace = true 9 | insert_final_newline = true 10 | charset = utf-8 11 | end_of_line = lf 12 | 13 | [*.bat] 14 | indent_style = tab 15 | end_of_line = crlf 16 | 17 | [LICENSE] 18 | insert_final_newline = false 19 | 20 | [Makefile] 21 | indent_style = tab -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.py[cod] 2 | 3 | # C extensions 4 | *.so 5 | 6 | # Packages 7 | *.egg 8 | *.egg-info 9 | dist 10 | build 11 | eggs 12 | parts 13 | bin 14 | var 15 | sdist 16 | develop-eggs 17 | .installed.cfg 18 | lib 19 | lib64 20 | 21 | # Installer logs 22 | pip-log.txt 23 | 24 | # Unit test / coverage reports 25 | .coverage 26 | .tox 27 | nosetests.xml 28 | 29 | # Translations 30 | *.mo 31 | 32 | # Mr Developer 33 | .mr.developer.cfg 34 | .project 35 | .pydevproject 36 | 37 | # Complexity 38 | output/*.html 39 | output/*/index.html 40 | 41 | # Sphinx 42 | docs/_build 43 | 44 | # Cookiecutter 45 | output/ 46 | boilerplate/ 47 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | # Config file for automatic testing at travis-ci.org 2 | 3 | language: python 4 | 5 | python: 6 | - "3.4" 7 | - "3.3" 8 | - "2.7" 9 | - "2.6" 10 | - "pypy" 11 | 12 | before_install: 13 | - pip install codecov 14 | 15 | # command to run tests, e.g. python setup.py test 16 | script: coverage run setup.py test 17 | 18 | after_success: 19 | - codecov 20 | -------------------------------------------------------------------------------- /AUTHORS.rst: -------------------------------------------------------------------------------- 1 | ======= 2 | Credits 3 | ======= 4 | 5 | Development Lead 6 | ---------------- 7 | 8 | * Dominique Barton 9 | 10 | Contributors 11 | ------------ 12 | 13 | None yet. Why not be the first? -------------------------------------------------------------------------------- /CONTRIBUTING.rst: -------------------------------------------------------------------------------- 1 | ============ 2 | Contributing 3 | ============ 4 | 5 | Contributions are welcome, and they are greatly appreciated! Every 6 | little bit helps, and credit will always be given. 7 | 8 | You can contribute in many ways: 9 | 10 | Types of Contributions 11 | ---------------------- 12 | 13 | Report Bugs 14 | ~~~~~~~~~~~ 15 | 16 | Report bugs at https://github.com/confirm/django-fulltext-search/issues. 17 | 18 | If you are reporting a bug, please include: 19 | 20 | * Your operating system name and version. 21 | * Any details about your local setup that might be helpful in troubleshooting. 22 | * Detailed steps to reproduce the bug. 23 | 24 | Fix Bugs 25 | ~~~~~~~~ 26 | 27 | Look through the GitHub issues for bugs. Anything tagged with "bug" 28 | is open to whoever wants to implement it. 29 | 30 | Implement Features 31 | ~~~~~~~~~~~~~~~~~~ 32 | 33 | Look through the GitHub issues for features. Anything tagged with "feature" 34 | is open to whoever wants to implement it. 35 | 36 | Write Documentation 37 | ~~~~~~~~~~~~~~~~~~~ 38 | 39 | django_fulltext_search could always use more documentation, whether as part of the 40 | official django_fulltext_search docs, in docstrings, or even on the web in blog posts, 41 | articles, and such. 42 | 43 | Submit Feedback 44 | ~~~~~~~~~~~~~~~ 45 | 46 | The best way to send feedback is to file an issue at https://github.com/confirm/django-fulltext-search/issues. 47 | 48 | If you are proposing a feature: 49 | 50 | * Explain in detail how it would work. 51 | * Keep the scope as narrow as possible, to make it easier to implement. 52 | * Remember that this is a volunteer-driven project, and that contributions 53 | are welcome :) 54 | 55 | Get Started! 56 | ------------ 57 | 58 | Ready to contribute? Here's how to set up `django_fulltext_search` for local development. 59 | 60 | 1. Fork the `django_fulltext_search` repo on GitHub. 61 | 2. Clone your fork locally:: 62 | 63 | $ git clone git@github.com:your_name_here/django-fulltext-search.git 64 | 65 | 3. Install your local copy into a virtualenv. Assuming you have virtualenvwrapper installed, this is how you set up your fork for local development:: 66 | 67 | $ mkvirtualenv django-fulltext-search 68 | $ cd django-fulltext-search/ 69 | $ python setup.py develop 70 | 71 | 4. Create a branch for local development:: 72 | 73 | $ git checkout -b name-of-your-bugfix-or-feature 74 | 75 | Now you can make your changes locally. 76 | 77 | 5. When you're done making changes, check that your changes pass flake8 and the tests, including testing other Python versions with tox:: 78 | 79 | $ flake8 django_fulltext_search.py test_django_fulltext_search.py 80 | $ py.test 81 | $ tox 82 | 83 | To get flake8 and tox, just pip install them into your virtualenv. 84 | 85 | 6. Commit your changes and push your branch to GitHub:: 86 | 87 | $ git add . 88 | $ git commit -m "Your detailed description of your changes." 89 | $ git push origin name-of-your-bugfix-or-feature 90 | 91 | 7. Submit a pull request through the GitHub website. 92 | 93 | Pull Request Guidelines 94 | ----------------------- 95 | 96 | Before you submit a pull request, check that it meets these guidelines: 97 | 98 | 1. The pull request should include tests. 99 | 2. If the pull request adds functionality, the docs should be updated. Put 100 | your new functionality into a function with a docstring, and add the 101 | feature to the list in README.rst. 102 | 3. The pull request should work for Python 2.6, 2.7, 3.3, and 3.4, and for PyPy. Check 103 | https://travis-ci.org/confirm/django-fulltext-search/pull_requests 104 | and make sure that the tests pass for all supported Python versions. 105 | 106 | Tips 107 | ---- 108 | 109 | To run a subset of tests:: 110 | 111 | TODO 112 | -------------------------------------------------------------------------------- /HISTORY.rst: -------------------------------------------------------------------------------- 1 | History 2 | ========= 3 | 4 | 0.2.0 (2018-02-16) 5 | --------------------- 6 | 7 | * Bugfix many_to_many kwarg error 8 | 9 | 0.1.0 (2017-08-09) 10 | --------------------- 11 | 12 | * First release on PyPI. -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2017, Dominique Barton 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 5 | 6 | * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 7 | 8 | * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 9 | 10 | * Neither the name of django_fulltext_search nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. 11 | 12 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include AUTHORS.rst 2 | include CONTRIBUTING.rst 3 | include HISTORY.rst 4 | include LICENSE 5 | include README.rst 6 | 7 | recursive-exclude * __pycache__ 8 | recursive-exclude * *.py[co] -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | =============================== 2 | Django Fulltext Search 3 | =============================== 4 | 5 | .. image:: https://badge.fury.io/py/django-fulltext-search.png 6 | :target: http://badge.fury.io/py/django_fulltext_search 7 | 8 | .. image:: https://travis-ci.org/confirm/django-fulltext-search.png?branch=master 9 | :target: https://travis-ci.org/confirm/django_fulltext_search 10 | 11 | .. image:: https://pypip.in/d/django-fulltext-search/badge.png 12 | :target: https://pypi.python.org/pypi/django_fulltext_search 13 | 14 | 15 | This repository provides a Python library for `Django `_ which supports native full-text search capabilities for `MySQL `_ and `MariaDB `_ . 16 | 17 | Django already `supports boolean full-text searches `_ . However, with the integrated full-text search support you can only search one single column and IMHO there's currently no way to search over multiple columns. 18 | 19 | Add fulltext index 20 | ------------------ 21 | 22 | Before you can run any full-text search you've to create at least one `full-text index `_ . 23 | Unfortunately Djangos' `native migrations `_ doesn't support full-text indexes and the manual says: 24 | 25 | > Note this is only available in MySQL and requires direct manipulation of the database to add the full-text index. 26 | 27 | However, you can easily create your own migration by starting with an empty one:: 28 | 29 | ./manage.py makemigrations --empty customer 30 | 31 | Open the created migration file and add a `RunSQL command `_ to the `operations` list:: 32 | 33 | operations = [ 34 | migrations.RunSQL( 35 | ('CREATE FULLTEXT INDEX customer_fulltext_index ON customer_customer (first_name, last_name)',), 36 | ('DROP INDEX customer_fulltext_index on customer_customer',) 37 | ) 38 | ] 39 | 40 | Then migrate the database:: 41 | 42 | ./manage.py migrate 43 | 44 | Update your model 45 | ----------------- 46 | 47 | If you want to add full-text search capabilities to your existing model you've to create a new `SearchManager()` instance and define it as your models' `objects` attribute:: 48 | 49 | from django_fulltext_search import SearchManager 50 | 51 | class Customer(models.Model): 52 | ''' Customer model. ''' 53 | 54 | # Enable full-text search support for first_name and last_name fields. 55 | objects = SearchManager(['first_name', 'last_name']) 56 | 57 | first_name = models.CharField(max_length=32) 58 | last_name = models.CharField(max_length=32) 59 | # more fields... 60 | 61 | As you can see, you can create the `SearchManager()` with a **list of by default searchable fields**. This means you don't have to bother about the field names later when you search your model. However, if you don't want to specify default fields you can also create the ``SearchManager()`` object without any arguments:: 62 | 63 | objects = SearchManager() 64 | 65 | Search 66 | ------ 67 | 68 | The library currently supports `boolean full-text seaches `_ by default if you use one of the operators (`+ - > < ( ) * "`) in your search query. Please note that the at-operator (`@`) will not enable the boolean search mode, which means you can also search for mail addresses, as long as you don't include any other operator in your search query. 69 | 70 | To search your model use the new `search()` method of the models' queryset:: 71 | 72 | Customer.objects.search('John*') 73 | 74 | This only works if you've defined default fields in the constructor of the ``SearchManager()``. If you haven't set them or you want to search alternative fields you have to define a list of fields:: 75 | 76 | Customer.objects.search('John*', ['first_name', 'last_name']) 77 | 78 | The search method can also be called with keyword arguments: 79 | 80 | * `query`: The search query itself 81 | * `fields`: A list of fields 82 | * _optional_ 83 | * _default are the fields defined in the ``SearchManager()`` 84 | * `mode`: Sets the search mode 85 | * _optional_ 86 | * _default is `AUTO` which will enable `BOOLEAN` mode when a boolean operator is found in the ``fseaquery`` 87 | * _can be set to a valid search mode, e.g. `BOOLEAN` or `NATURAL LANGUAGE` 88 | 89 | **IMPORTANT:** Please remember you've to create a full-text index for the defined fields before you can search them. 90 | 91 | Related models 92 | -------------- 93 | 94 | If you want to search a related models' field (e.g. `models.ForeignKey`) you can also use the `__` syntax:: 95 | 96 | SearchManager(['customer__first_name', 'customer__last_name'] 97 | -------------------------------------------------------------------------------- /django_fulltext_search.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | from django.db import models, connection 3 | 4 | __author__ = 'confirm IT solutions' 5 | __email__ = 'contactus@confirm.ch' 6 | __version__ = '0.2.0' 7 | 8 | 9 | class SearchQuerySet(models.query.QuerySet): 10 | ''' 11 | QuerySet which supports MySQL and MariaDB full-text search. 12 | ''' 13 | 14 | def __init__(self, fields=None, **kwargs): 15 | super(SearchQuerySet, self).__init__(**kwargs) 16 | self.search_fields = fields 17 | 18 | def get_query_set(self, query, columns, mode): 19 | ''' 20 | Returns the query set for the columns and search mode. 21 | ''' 22 | # Create the WHERE MATCH() ... AGAINST() expression. 23 | fulltext_columns = ', '.join(columns) 24 | where_expression = ('MATCH({}) AGAINST("%s" {})'.format(fulltext_columns, mode)) 25 | 26 | # Get query set via extra() method. 27 | return self.extra(where=[where_expression], params=[query]) 28 | 29 | def search(self, query, fields=None, mode=None): 30 | ''' 31 | Runs a fulltext search against the fields defined in the method's 32 | kwargs. If no fields are defined in the method call, then the fields 33 | defined in the constructor's kwargs will be used. 34 | 35 | Just define a query (the search term) and a fulltext search will be 36 | executed. In case mode is set to None, the method will automatically 37 | switch to "BOOLEAN" in case any boolean operators were found. 38 | Of course you can set the search mode to any type you want, e.g. 39 | "NATURAL LANGUAGE". 40 | ''' 41 | # 42 | # Get all requried attributes and initialize our empty sets. 43 | # 44 | 45 | meta = self.model._meta 46 | quote_name = connection.ops.quote_name 47 | seperator = models.constants.LOOKUP_SEP 48 | 49 | columns = set() 50 | related_fields = set() 51 | 52 | # 53 | # Loop through the defined search fields to build a list of all 54 | # searchable columns. We need to differ between simple fields and 55 | # fields with a related model, because the meta data of those fields 56 | # are stored in the related model itself. 57 | # 58 | 59 | fields = self.search_fields if not fields else fields 60 | 61 | for field in fields: 62 | 63 | # Handling fields with a related model. 64 | if seperator in field: 65 | field, rfield = field.split(seperator) 66 | rmodel = meta.get_field(field).related_model 67 | rmeta = rmodel._meta 68 | table = rmeta.db_table 69 | column = rmeta.get_field(rfield).column 70 | related_fields.add(field) 71 | 72 | # Handle fields without a related model. 73 | else: 74 | table = meta.db_table 75 | column = meta.get_field(field, many_to_many=False).column 76 | 77 | # Add field with `table`.`column` style to columns set. 78 | columns.add('{}.{}'.format(quote_name(table), quote_name(column))) 79 | 80 | # 81 | # We now have all the required informations to build the query with the 82 | # fulltext "MATCH(…) AGAINST(…)" WHERE statement. However, we also need 83 | # to conside the search mode. Thus, if the mode argument is set to 84 | # None, we need to inspect the search query and enable the BOOLEAN mode 85 | # in case any boolean operators were found. This is also a workaround 86 | # for using at-signs (@) in search queries, because we don't enable the 87 | # boolean mode in case no other operator was found. 88 | # 89 | 90 | # Set boolean mode if mode argument is set to None. 91 | if mode is None and any(x in query for x in '+-><()*"'): 92 | mode = 'BOOLEAN' 93 | 94 | # Convert the mode into a valid "IN … MODE" or empty string. 95 | if mode is None: 96 | mode = '' 97 | else: 98 | mode = 'IN {} MODE'.format(mode) 99 | 100 | # Get the query set. 101 | query_set = self.get_query_set(query, columns, mode) 102 | 103 | # 104 | # If related fields were involved we've to select them as well. 105 | # 106 | 107 | if related_fields: 108 | query_set = query_set.select_related(','.join(related_fields)) 109 | 110 | # Return query_set. 111 | return query_set 112 | 113 | def count(self): 114 | ''' 115 | Returns the count database records. 116 | ''' 117 | # 118 | # We need to overwrite the default count() method. Unfortunately 119 | # Django's internal count() method will clone the query object and then 120 | # re-create the SQL query based on the default table and WHERE clause, 121 | # but without the related tables. So if related tables are included in 122 | # the query (i.e. JOINs), then Django will forget about the JOINs and 123 | # the MATCH() of the related fields will fail with an "unknown column" 124 | # error. 125 | # 126 | 127 | return self.__len__() 128 | 129 | 130 | class SearchManager(models.Manager): 131 | ''' 132 | SearchManager which supports MySQL and MariaDB full-text search. 133 | ''' 134 | 135 | query_set = SearchQuerySet 136 | 137 | def __init__(self, fields=None): 138 | super(SearchManager, self).__init__() 139 | self.search_fields = fields 140 | 141 | def get_query_set(self): 142 | ''' 143 | Returns the query set. 144 | ''' 145 | return self.query_set(model=self.model, fields=self.search_fields) 146 | 147 | def search(self, query, **kwargs): 148 | ''' 149 | Runs a fulltext search against the fields defined in the method's kwargs 150 | or in the constructor's kwargs. 151 | 152 | For more informations read the documentation string of the 153 | SearchQuerySet's search() method. 154 | ''' 155 | return self.get_query_set().search(query, **kwargs) 156 | -------------------------------------------------------------------------------- /requirements-dev.txt: -------------------------------------------------------------------------------- 1 | -r requirements.txt 2 | 3 | flake8 4 | pytest 5 | wheel>=0.23.0 -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | django 2 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [wheel] 2 | universal=1 3 | 4 | [bdist_wheel] 5 | universal=1 -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | 4 | import os 5 | import sys 6 | 7 | try: 8 | from setuptools import setup 9 | except ImportError: 10 | from distutils.core import setup 11 | 12 | version = "0.2.0" 13 | 14 | if sys.argv[-1] == 'publish': 15 | try: 16 | import wheel 17 | except ImportError: 18 | raise ImportError("Fix: pip install wheel") 19 | os.system('python setup.py sdist bdist_wheel upload') 20 | print("You probably want to also tag the version now:") 21 | print(" git tag -a %s -m 'version %s'" % (version, version)) 22 | print(" git push --tags") 23 | sys.exit() 24 | 25 | if sys.argv[-1] == 'tag': 26 | print("Tagging the version on github:") 27 | os.system("git tag -a %s -m 'version %s'" % (version, version)) 28 | os.system("git push --tags") 29 | sys.exit() 30 | 31 | readme = open('README.rst').read() 32 | history = open('HISTORY.rst').read().replace('.. :changelog:', '') 33 | 34 | 35 | def get_requirements(filename): 36 | f = open(filename).read() 37 | reqs = [ 38 | # loop through list of requirements 39 | x.strip() for x in f.splitlines() 40 | # filter out comments and empty lines 41 | if not x.strip().startswith('#') 42 | ] 43 | return reqs 44 | 45 | setup( 46 | name='django_fulltext_search', 47 | version=version, 48 | description="""Django support for MySQL / MariaDB full-text search""", 49 | long_description=readme + '\n\n' + history, 50 | author='Dominique Barton', 51 | author_email='contactus@confirm.ch', 52 | url='https://github.com/confirm/django_fulltext_search', 53 | include_package_data=True, 54 | py_modules=['django_fulltext_search'], 55 | install_requires=get_requirements('requirements.txt'), 56 | license="BSD", 57 | zip_safe=False, 58 | keywords='django_fulltext_search', 59 | classifiers=[ 60 | 'Development Status :: 2 - Pre-Alpha', 61 | 'Intended Audience :: Developers', 62 | 'License :: OSI Approved :: BSD License', 63 | 'Natural Language :: English', 64 | 'Programming Language :: Python :: 2', 65 | 'Programming Language :: Python :: 2.7', 66 | 'Programming Language :: Python :: 3', 67 | 'Programming Language :: Python :: 3.3', 68 | 'Programming Language :: Python :: 3.4', 69 | ], 70 | 71 | ) 72 | -------------------------------------------------------------------------------- /test_django_fulltext_search.py: -------------------------------------------------------------------------------- 1 | import pytest 2 | import django_fulltext_search 3 | 4 | 5 | def test_the_obvious(): 6 | assert True == True 7 | 8 | 9 | if __name__ == '__main__': 10 | pytest.main() 11 | -------------------------------------------------------------------------------- /tox.ini: -------------------------------------------------------------------------------- 1 | [tox] 2 | envlist = py26, py27, py33, py34 3 | 4 | [testenv] 5 | setenv = 6 | PYTHONPATH = {toxinidir}:{toxinidir}/django_fulltext_search 7 | commands = py.test 8 | deps = 9 | -r{toxinidir}/requirements-dev.txt 10 | --------------------------------------------------------------------------------