├── .gitignore ├── .readthedocs.yml ├── .travis.yml ├── CONTRIBUTING.rst ├── CONTRIBUTORS ├── LICENSE ├── README.rst ├── codecov.yml ├── dev ├── docker-compose.yml └── postgres_init.sh ├── docs ├── Makefile ├── readthedocs.txt └── source │ ├── _static │ ├── carbon.png │ ├── logo.png │ └── safeline.svg │ ├── _templates │ └── sidebarlogo.html │ ├── api.rst │ ├── conf.py │ ├── decorators.rst │ ├── design.rst │ ├── index.rst │ ├── installation.rst │ └── signals.rst ├── pg_partitioning ├── __init__.py ├── apps.py ├── constants.py ├── decorators.py ├── manager.py ├── migrations │ ├── 0001_initial.py │ └── __init__.py ├── models.py ├── patch │ ├── __init__.py │ └── schema.py ├── shortcuts.py └── signals.py ├── run_test.py ├── setup.cfg ├── setup.py ├── tests ├── __init__.py ├── models.py └── tests.py └── tox.ini /.gitignore: -------------------------------------------------------------------------------- 1 | *.egg-info 2 | dist/ 3 | build/ 4 | docs/_build 5 | docs/source/.doctrees/ 6 | __pycache__/ 7 | .cache 8 | *.sqlite3 9 | .tox/ 10 | *.swp 11 | *.pyc 12 | .coverage* 13 | .pytest_cache 14 | node_modules 15 | venv/ 16 | 17 | # IDE and Tooling files 18 | .idea/* 19 | *~ 20 | 21 | # macOS 22 | .DS_Store 23 | -------------------------------------------------------------------------------- /.readthedocs.yml: -------------------------------------------------------------------------------- 1 | version: 2 2 | build: 3 | image: latest 4 | sphinx: 5 | configuration: docs/source/conf.py 6 | python: 7 | version: 3.6 8 | install: 9 | - requirements: docs/readthedocs.txt 10 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | dist: xenial 2 | language: python 3 | cache: pip 4 | sudo: required 5 | matrix: 6 | fast_finish: true 7 | include: 8 | - { python: "3.6", env: DJANGO=2.0 } 9 | - { python: "3.6", env: DJANGO=2.1 } 10 | - { python: "3.6", env: DJANGO=2.2 } 11 | - { python: "3.7", env: DJANGO=2.0 } 12 | - { python: "3.7", env: DJANGO=2.1 } 13 | - { python: "3.7", env: DJANGO=2.2 } 14 | - { python: "3.7", env: DJANGO=master } 15 | - { python: "3.6", env: TOXENV=lint } 16 | - { python: "3.6", env: TOXENV=docs } 17 | allow_failures: 18 | - env: TOXENV=docs 19 | - env: DJANGO=master 20 | services: 21 | - docker 22 | before_install: 23 | - sudo /etc/init.d/postgresql stop 24 | install: 25 | - pip install '.[dev]' tox-travis 26 | before_script: 27 | - docker-compose -f dev/docker-compose.yml up -d 28 | - sleep 5 29 | script: 30 | - tox 31 | after_success: 32 | - pip install codecov 33 | - codecov -e TOXENV,DJANGO 34 | notifications: 35 | email: false 36 | -------------------------------------------------------------------------------- /CONTRIBUTING.rst: -------------------------------------------------------------------------------- 1 | Contributing Guidelines 2 | ======================= 3 | 4 | Issue tracker 5 | ------------- 6 | 您可以通过 `issue tracker `__ 提交改进建议、缺陷报告或功能需求，但 **必须** 遵守以下规范： 7 | 8 | * **请勿** 重复提交相似的主题或内容。 9 | * **请勿** 讨论任何与本项目无关的内容。 10 | * 我们非常欢迎您提交程序缺陷报告，但在此之前，请确保您已经完整阅读过相关文档，并已经做了一些必要的调查，确定错误并非您自身造成的。在您编写程序缺陷报告时， 11 | 请详细描述您所出现的问题和复现步骤，并附带详细的信息，以便我们能尽快定位问题。 12 | 13 | ---- 14 | 15 | You can submit improvement suggestions, bug reports, or feature requests through the `issue tracker `_, 16 | but you **MUST** adhere to the following specifications: 17 | 18 | * **Do not** submit similar topics or content repeatedly. 19 | * **Do not** discuss any content not related to this project. 20 | * We welcome you to submit a bug report, but before doing so, please make sure that you have read the documentation in its entirety and 21 | have done some necessary investigations to determine that the error is not yours. When you write a bug report, Please describe in detail 22 | the problem and recurring steps that you have with detailed information so that we can locate the problem as quickly as possible. 23 | 24 | Code guidelines 25 | --------------- 26 | * 本项目采用 `语义化版本 2.0.0 `_ 27 | * 本项目使用了 `flask8` `isort` `black` 等代码静态检查工具。提交的代码 **必须** 通过 `lint` 工具检查。某些特殊情况不符合规范的部分，需要按照检查工具要求的方式具体标记出来。 28 | * 公开的 API **必须** 使用 Type Hint 并编写 Docstrings，其他部分 **建议** 使用并在必要的地方为代码编写注释，增强代码的可读性。 29 | * **必须** 限定非 Development 的外部依赖的模块版本为某一个完全兼容的系列。 30 | 31 | 相关文档: 32 | 33 | | `Google Python Style Guide `_ 34 | | `PEP 8 Style Guide for Python Code `_ 35 | 36 | ---- 37 | 38 | * This project uses a `Semantic Version 2.0.0 `_ 39 | * This project uses a code static check tool such as `flask8` `isort` `black`. The submitted code **MUST** be checked by the `lint` tool. 40 | Some special cases that do not meet the specifications need to be specifically marked in the way required by the inspection tool. 41 | * The public API **MUST** use Type Hint and write Docstrings, other parts **SHOULD** use it and write comments to the code where necessary 42 | to enhance the readability of code. 43 | * External dependencies published with the project **MUST** at least define a fully compatible version family. 44 | 45 | Related documents: 46 | 47 | | `Google Python Style Guide `_ 48 | | `PEP 8 Style Guide for Python Code `_ 49 | -------------------------------------------------------------------------------- /CONTRIBUTORS: -------------------------------------------------------------------------------- 1 | virusdefender 2 | monouno 3 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2019 Chaitin Tech Co., Ltd. 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy 4 | of this software and associated documentation files (the "Software"), to deal 5 | in the Software without restriction, including without limitation the rights 6 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 7 | copies of the Software, and to permit persons to whom the Software is 8 | furnished to do so, subject to the following conditions: 9 | 10 | The above copyright notice and this permission notice shall be included in all 11 | copies or substantial portions of the Software. 12 | 13 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 14 | EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 15 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 16 | IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, 17 | DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR 18 | OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE 19 | OR OTHER DEALINGS IN THE SOFTWARE. -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | django-pg-partitioning 2 | ====================== 3 | .. image:: https://img.shields.io/badge/License-MIT-orange.svg?style=flat-square 4 | :target: https://raw.githubusercontent.com/chaitin/django-pg-partitioning/master/LICENSE 5 | .. image:: https://img.shields.io/badge/Django-2.x-green.svg?style=flat-square&logo=django 6 | :target: https://www.djangoproject.com/ 7 | .. image:: https://img.shields.io/badge/PostgreSQL-11-lightgrey.svg?style=flat-square&logo=postgresql 8 | :target: https://www.postgresql.org/ 9 | .. image:: https://readthedocs.org/projects/django-pg-partitioning/badge/?version=latest&style=flat-square 10 | :target: https://django-pg-partitioning.readthedocs.io 11 | .. image:: https://img.shields.io/pypi/v/django-pg-partitioning.svg?style=flat-square 12 | :target: https://pypi.org/project/django-pg-partitioning/ 13 | .. image:: https://api.travis-ci.org/chaitin/django-pg-partitioning.svg?branch=master 14 | :target: https://travis-ci.org/chaitin/django-pg-partitioning 15 | .. image:: https://api.codacy.com/project/badge/Grade/c872699c1b254e90b540b053343d1e81 16 | :target: https://www.codacy.com/app/xingji2163/django-pg-partitioning?utm_source=github.com&utm_medium=referral&utm_content=chaitin/django-pg-partitioning&utm_campaign=Badge_Grade 17 | .. image:: https://codecov.io/gh/chaitin/django-pg-partitioning/branch/master/graph/badge.svg 18 | :target: https://codecov.io/gh/chaitin/django-pg-partitioning 19 | 20 | ⚠️ 21 | ---- 22 | 23 | 目前已经有了更好用的 Django 插件（比如 django-postgres-extra）使得基于 Django 开发的项目能够方便地使用 PostgreSQL 数据库的高级功能，因此本项目不再维护。你依然可以 fork 本项目并进行二次开发，祝你生活愉快 :) 24 | 25 | There are already better Django plugins (such as django-postgres-extra) that make it easy for Django-based projects to use the advanced features of PostgreSQL databases, so this project is no longer maintained. You can still fork this project and do secondary development, have a nice life :) 26 | 27 | ---- 28 | 29 | 一个支持 PostgreSQL 11 原生表分区的 Django 扩展，使您可以在 Django 中创建分区表并管理它们。目前它支持两种分区类型： 30 | 31 | - 时间范围分区（Time Range Partitioning）：将时序数据分开存储到不同的时间范围分区表中，支持创建连续且不重叠的时间范围分区并进行归档管理。 32 | - 列表分区（List Partitioning）：根据分区字段的确定值将数据分开存储到不同的分区表中。 33 | 34 | A Django extension that supports PostgreSQL 11 native table partitioning, allowing you to create partitioned tables in Django 35 | and manage them. Currently it supports the following two partition types: 36 | 37 | - **Time Range Partitioning**: Separate time series data into different time range partition tables, 38 | support the creation of continuous and non-overlapping time range partitions and archival management. 39 | - **List Partitioning**: Store data separately into different partition tables based on the determined values of the partition key. 40 | 41 | Documentation 42 | https://django-pg-partitioning.readthedocs.io 43 | 44 | .. image:: https://raw.githubusercontent.com/chaitin/django-pg-partitioning/master/docs/source/_static/carbon.png 45 | :align: center 46 | 47 | TODO 48 | ---- 49 | - Improve the details of the function. 50 | - Improve documentation and testing. 51 | - Optimization implementation. 52 | 53 | maybe more... 54 | 55 | Contributing 56 | ------------ 57 | If you want to contribute to a project and make it better, you help is very welcome! 58 | Please read through `Contributing Guidelines `__. 59 | 60 | License 61 | ------- 62 | This project is licensed under the MIT. Please see `LICENSE `_. 63 | 64 | Project Practice 65 | ---------------- 66 | .. image:: https://raw.githubusercontent.com/chaitin/django-pg-timepart/master/docs/source/_static/safeline.svg?sanitize=true 67 | :target: https://www.chaitin.cn/en/safeline 68 | -------------------------------------------------------------------------------- /codecov.yml: -------------------------------------------------------------------------------- 1 | coverage: 2 | precision: 2 3 | round: down 4 | range: "80...100" 5 | 6 | status: 7 | project: yes 8 | patch: no 9 | changes: no 10 | 11 | comment: off 12 | -------------------------------------------------------------------------------- /dev/docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: "3" 2 | services: 3 | postgres: 4 | image: postgres:${POSTGRES_VERSION:-11.1-alpine} 5 | container_name: postgres 6 | restart: always 7 | volumes: 8 | - ./postgres_init.sh:/docker-entrypoint-initdb.d/postgres_init.sh 9 | environment: 10 | - POSTGRES_DB=test 11 | - POSTGRES_USER=test 12 | - POSTGRES_PASSWORD=test 13 | ports: 14 | - "127.0.0.1:5432:5432" 15 | -------------------------------------------------------------------------------- /dev/postgres_init.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | mkdir /tmp/data1 /tmp/data2 4 | psql -U test -c "CREATE TABLESPACE data1 LOCATION '/tmp/data1'" 5 | psql -U test -c "CREATE TABLESPACE data2 LOCATION '/tmp/data2'" 6 | -------------------------------------------------------------------------------- /docs/Makefile: -------------------------------------------------------------------------------- 1 | # Minimal makefile for Sphinx documentation 2 | 3 | 4 | # You can set these variables from the command line. 5 | SPHINXOPTS = 6 | SPHINXBUILD = sphinx-build 7 | SPHINXPROJ = django-partitioning 8 | SOURCEDIR = source 9 | BUILDDIR = _build 10 | 11 | 12 | .PHONY: all 13 | all: 14 | @$(SPHINXBUILD) "$(SOURCEDIR)" "$(BUILDDIR)/" 15 | 16 | 17 | .PHONY: help 18 | help: 19 | @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) 20 | 21 | 22 | .PHONY: Makefile 23 | # Catch-all target: route all unknown targets to Sphinx using the new 24 | # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). 25 | %: Makefile 26 | @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) -------------------------------------------------------------------------------- /docs/readthedocs.txt: -------------------------------------------------------------------------------- 1 | alabaster==0.7.12 2 | sphinx==1.8.3 3 | sphinxcontrib-napoleon==0.7 4 | Pygments==2.3.1 5 | psycopg2-binary==2.7.6.1 6 | Django>=2.0,<2.2 7 | python-dateutil~=2.7 8 | -------------------------------------------------------------------------------- /docs/source/_static/carbon.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaitin/django-pg-partitioning/129f5ddd11c61f855b97e558159dd067a15ca18d/docs/source/_static/carbon.png -------------------------------------------------------------------------------- /docs/source/_static/logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaitin/django-pg-partitioning/129f5ddd11c61f855b97e558159dd067a15ca18d/docs/source/_static/logo.png -------------------------------------------------------------------------------- /docs/source/_static/safeline.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | SafeLine 5 | django-pg-partitioning 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | -------------------------------------------------------------------------------- /docs/source/_templates/sidebarlogo.html: -------------------------------------------------------------------------------- 1 | 7 | -------------------------------------------------------------------------------- /docs/source/api.rst: -------------------------------------------------------------------------------- 1 | API Reference 2 | ============= 3 | 4 | .. py:currentmodule:: pg_partitioning.manager 5 | 6 | Time Range Partitioning 7 | ----------------------- 8 | 9 | .. autoclass:: TimeRangePartitionManager 10 | :members: 11 | 12 | .. autoclass:: PartitionConfig 13 | :members: period, interval, attach_tablespace, detach_tablespace, save 14 | 15 | .. autoclass:: PartitionLog 16 | :members: is_attached, detach_time, save, delete 17 | 18 | List Partitioning 19 | ----------------- 20 | 21 | .. autoclass:: ListPartitionManager 22 | :members: 23 | 24 | .. py:currentmodule:: pg_partitioning.shortcuts 25 | 26 | Shortcuts 27 | --------- 28 | 29 | .. automodule:: pg_partitioning.shortcuts 30 | :members: truncate_table, set_tablespace, drop_table 31 | -------------------------------------------------------------------------------- /docs/source/conf.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | 4 | import os 5 | import sys 6 | 7 | import django 8 | from django.conf import settings 9 | 10 | sys.path.insert(0, os.path.abspath('../..')) 11 | sys.path.insert(0, os.path.abspath('./')) 12 | 13 | settings.configure( 14 | INSTALLED_APPS=( 15 | "pg_partitioning", 16 | ), 17 | ) 18 | django.setup() 19 | 20 | extensions = [ 21 | 'sphinx.ext.autodoc', 22 | 'sphinx.ext.intersphinx', 23 | 'sphinx.ext.viewcode', 24 | 'sphinx.ext.napoleon', 25 | 'alabaster', 26 | ] 27 | 28 | templates_path = ['_templates'] 29 | 30 | source_suffix = '.rst' 31 | 32 | master_doc = 'index' 33 | 34 | project = 'django-pg-partitioning' 35 | copyright = '2019, Chaitin Tech' 36 | author = 'Boyce Li' 37 | 38 | language = 'en' 39 | 40 | exclude_patterns = ['_build'] 41 | 42 | pygments_style = 'sphinx' 43 | 44 | html_theme = 'alabaster' 45 | 46 | html_theme_options = { 47 | 'github_user': 'chaitin', 48 | 'github_repo': 'django-pg-partitioning', 49 | 'github_type': 'star', 50 | 'github_banner': 'true', 51 | 'show_powered_by': 'false', 52 | 'code_font_size': '14px', 53 | } 54 | 55 | html_static_path = ['_static'] 56 | 57 | html_sidebars = { 58 | '**': [ 59 | 'sidebarlogo.html', 60 | 'navigation.html', 61 | 'searchbox.html', 62 | ] 63 | } 64 | 65 | htmlhelp_basename = 'django-pg-partitioning-doc' 66 | 67 | man_pages = [ 68 | (master_doc, 'django-pg-partitioning', 'django-pg-partitioning Documentation', 69 | [author], 1) 70 | ] 71 | 72 | texinfo_documents = [ 73 | (master_doc, 'django-pg-partitioning', 'django-pg-partitioning Documentation', 74 | author, 'django-pg-partitioning', 'A Django extension that supports PostgreSQL 11 time ranges and list partitioning.', 75 | 'Miscellaneous'), 76 | ] 77 | 78 | intersphinx_mapping = { 79 | 'https://docs.python.org/3/': None, 80 | 'https://pika.readthedocs.io/en/0.10.0/': None, 81 | } 82 | 83 | add_module_names = False 84 | -------------------------------------------------------------------------------- /docs/source/decorators.rst: -------------------------------------------------------------------------------- 1 | Decorators 2 | ========== 3 | 4 | Now the decorator must be used on a non-abstract model class that has not yet built a table in the database. 5 | If you must use the decorator on a model class that has previously performed a migrate operation, you need 6 | to back up the model's data, then drop the table, and then import the data after you have created a 7 | partitioned table. 8 | 9 | .. py:currentmodule:: pg_partitioning.decorators 10 | 11 | .. autodata:: TimeRangePartitioning 12 | :annotation: 13 | 14 | .. autodata:: ListPartitioning 15 | :annotation: 16 | 17 | Post-Decoration 18 | --------------- 19 | 20 | You can run ``makemigrations`` and ``migrate`` commands to create and apply new migrations. 21 | Once the table has been created, it is not possible to turn a regular table into a partitioned table or vice versa. 22 | -------------------------------------------------------------------------------- /docs/source/design.rst: -------------------------------------------------------------------------------- 1 | Design 2 | ====== 3 | 4 | A brief description of the architecture of django-pg-partitioning. 5 | 6 | Partitioned Table Support 7 | ------------------------- 8 | 9 | Currently Django does not support creating partitioned tables, so django-partitioning monkey patch ``create_model`` method in 10 | ``DatabaseSchemaEditor`` to make it generate SQL statements that create partitioned tables. 11 | 12 | Some of the operations that Django applies to regular database tables may not be supported or even conflicted on the partitioned 13 | table, eventually throwing a database exception. Therefore, it is recommended that you read the section on the partition table 14 | in the official database documentation and refer to the relevant implementation inside Django. 15 | 16 | Constraint Limitations 17 | ---------------------- 18 | 19 | It is important to note that PostgreSQL table partitioning has some restrictions on field constraints. 20 | In order for the extension to work, we turned off Django's automatically generated primary key constraint, but did not do other legality checks. 21 | For example, if you mistakenly used a unique or foreign key constraint, it will throw an exception directly, which is what you are coding and 22 | it needs to be manually circumvented during use. 23 | 24 | Tablespace 25 | ---------- 26 | 27 | ``pg_partitioning`` will silently set the tablespace of all local partitioned indexes under one partition to be consistent with 28 | the partition. 29 | 30 | Partition Information 31 | --------------------- 32 | 33 | ``pg_partitioning`` saves partition configuration and state information in ``PartitionConfig`` and ``PartitionLog``. 34 | The problem with this is that once this information is inconsistent with the actual situation, ``pg_partitioning`` 35 | will not work properly, so you can only fix it manually. 36 | 37 | Management 38 | ---------- 39 | 40 | You can use ``Model.partitioning.create_partition`` and ``Model.partitioning.detach_partition`` to automatically create and 41 | archive partitions. In addition setting ``default_detach_tablespace`` and ``default_attach_tablespace``, you can also use the 42 | ``set_tablespace`` method of the PartitionLog object to move the partition. See :doc:`api` for details. 43 | -------------------------------------------------------------------------------- /docs/source/index.rst: -------------------------------------------------------------------------------- 1 | .. include:: ../../README.rst 2 | 3 | .. toctree:: 4 | :hidden: 5 | :maxdepth: 2 6 | 7 | installation 8 | design 9 | decorators 10 | api 11 | signals 12 | Contributing 13 | Project License 14 | -------------------------------------------------------------------------------- /docs/source/installation.rst: -------------------------------------------------------------------------------- 1 | Installation 2 | ============ 3 | 4 | PyPI 5 | ---- 6 | 7 | .. code-block:: bash 8 | 9 | $ pip install django-pg-partitioning 10 | 11 | Or you can install from GitHub 12 | 13 | .. code-block:: bash 14 | 15 | $ pip install git+https://github.com/chaitin/django-pg-partitioning.git@master 16 | 17 | Integrate with Django 18 | --------------------- 19 | 20 | Add ``pg_partitioning`` to ``INSTALLED_APPS`` in settings.py. 21 | 22 | Important - Please note 'pg_partitioning' should be loaded earlier than other apps that depend on it:: 23 | 24 | INSTALLED_APPS = [ 25 | 'pg_partitioning', 26 | ... 27 | ] 28 | 29 | PARTITION_TIMEZONE = "Asia/Shanghai" 30 | 31 | You can specify the time zone referenced by the time range partitioned table via ``PARTITION_TIMEZONE``, 32 | and if it is not specified, ``TIME_ZONE`` value is used. 33 | 34 | Post-Installation 35 | ----------------- 36 | 37 | In your Django root execute the command below to create 'pg_partitioning' database tables:: 38 | 39 | ./manage.py migrate pg_partitioning 40 | 41 | -------------------------------------------------------------------------------- /docs/source/signals.rst: -------------------------------------------------------------------------------- 1 | Signals 2 | ======= 3 | 4 | Note that these signals are only triggered when the save methods of ``PartitionConfig`` and ``PartitionLog`` 5 | You can hook to them for your own needs (for example to create corresponding table index). 6 | 7 | .. py:currentmodule:: pg_partitioning.signals 8 | 9 | .. autodata:: post_create_partition(sender, partition_log) 10 | :annotation: 11 | .. autodata:: post_attach_partition(sender, partition_log) 12 | :annotation: 13 | .. autodata:: post_detach_partition(sender, partition_log) 14 | :annotation: 15 | -------------------------------------------------------------------------------- /pg_partitioning/__init__.py: -------------------------------------------------------------------------------- 1 | """ 2 | A Django extension that supports PostgreSQL 11 time ranges and list partitioning. 3 | """ 4 | REQUIRED_DJANGO_VERSION = [(2, 0), (3, 0)] 5 | DJANGO_VERSION_ERROR = "django-pg-partitioning isn't available on the currently installed version of Django." 6 | 7 | try: 8 | import django 9 | except ImportError: 10 | raise ImportError(DJANGO_VERSION_ERROR) 11 | 12 | if REQUIRED_DJANGO_VERSION[0] > tuple(django.VERSION[:2]) or tuple(django.VERSION[:2]) > REQUIRED_DJANGO_VERSION[1]: 13 | raise ImportError(DJANGO_VERSION_ERROR) 14 | 15 | 16 | __version__ = "0.11" 17 | 18 | default_app_config = "pg_partitioning.apps.AppConfig" 19 | -------------------------------------------------------------------------------- /pg_partitioning/apps.py: -------------------------------------------------------------------------------- 1 | from django.apps import AppConfig as DefaultAppConfig 2 | 3 | 4 | class AppConfig(DefaultAppConfig): 5 | name = "pg_partitioning" 6 | 7 | def ready(self): 8 | from .patch import schema # noqa 9 | -------------------------------------------------------------------------------- /pg_partitioning/constants.py: -------------------------------------------------------------------------------- 1 | SQL_CREATE_TIME_RANGE_PARTITION = """\ 2 | CREATE TABLE IF NOT EXISTS %(child)s PARTITION OF %(parent)s FOR VALUES FROM (%(date_start)s) TO (%(date_end)s)""" 3 | SQL_CREATE_LIST_PARTITION = """\ 4 | CREATE TABLE IF NOT EXISTS %(child)s PARTITION OF %(parent)s FOR VALUES IN (%(value)s)""" 5 | SQL_SET_TABLE_TABLESPACE = """\ 6 | ALTER TABLE IF EXISTS %(name)s SET TABLESPACE %(tablespace)s""" 7 | SQL_APPEND_TABLESPACE = " TABLESPACE %(tablespace)s" 8 | SQL_ATTACH_TIME_RANGE_PARTITION = """\ 9 | ALTER TABLE IF EXISTS %(parent)s ATTACH PARTITION %(child)s FOR VALUES FROM (%(date_start)s) TO (%(date_end)s)""" 10 | SQL_ATTACH_LIST_PARTITION = """\ 11 | ALTER TABLE IF EXISTS %(parent)s ATTACH PARTITION %(child)s FOR VALUES IN (%(value)s)""" 12 | SQL_DETACH_PARTITION = "ALTER TABLE IF EXISTS %(parent)s DETACH PARTITION %(child)s" 13 | SQL_DROP_TABLE = "DROP TABLE IF EXISTS %(name)s" 14 | SQL_TRUNCATE_TABLE = "TRUNCATE TABLE %(name)s" 15 | SQL_DROP_INDEX = "DROP INDEX IF EXISTS %(name)s" 16 | SQL_CREATE_INDEX = "CREATE INDEX IF NOT EXISTS %(name)s ON %(table_name)s USING %(method)s (%(column_name)s)" 17 | SQL_SET_INDEX_TABLESPACE = "ALTER INDEX %(name)s SET TABLESPACE %(tablespace)s" 18 | SQL_GET_TABLE_INDEXES = "SELECT indexname FROM pg_indexes WHERE tablename = %(table_name)s" 19 | 20 | DT_FORMAT = "%Y-%m-%d" 21 | 22 | 23 | class PartitioningType: 24 | Range = "RANGE" 25 | List = "LIST" 26 | 27 | 28 | class PeriodType: 29 | Day = "Day" 30 | Week = "Week" 31 | Month = "Month" 32 | Year = "Year" 33 | -------------------------------------------------------------------------------- /pg_partitioning/decorators.py: -------------------------------------------------------------------------------- 1 | import logging 2 | from typing import Type 3 | 4 | from django.db import models 5 | 6 | from pg_partitioning.manager import ListPartitionManager, TimeRangePartitionManager 7 | 8 | logger = logging.getLogger(__name__) 9 | 10 | 11 | class _PartitioningBase: 12 | def __init__(self, partition_key: str, **options): 13 | self.partition_key = partition_key 14 | self.options = options 15 | 16 | def __call__(self, model: Type[models.Model]): 17 | if model._meta.abstract: 18 | raise NotImplementedError("Decorative abstract model classes are not supported.") 19 | 20 | 21 | class TimeRangePartitioning(_PartitioningBase): 22 | """Use this decorator to declare the database table corresponding to the model to be partitioned by time range. 23 | 24 | Parameters: 25 | partition_key(str): Partition field name of DateTimeField. 26 | options: Currently supports the following keyword parameters: 27 | 28 | - default_period(PeriodType): Default partition period. 29 | - default_interval(int): Default detach partition interval. 30 | - default_attach_tablespace(str): Default tablespace for attached tables. 31 | - default_detach_tablespace(str): Default tablespace for attached tables. 32 | 33 | Example: 34 | .. code-block:: python 35 | 36 | from django.db import models 37 | from django.utils import timezone 38 | 39 | from pg_partitioning.decorators import TimeRangePartitioning 40 | 41 | 42 | @TimeRangePartitioning(partition_key="timestamp") 43 | class MyLog(models.Model): 44 | name = models.TextField(default="Hello World!") 45 | timestamp = models.DateTimeField(default=timezone.now, primary_key=True) 46 | """ 47 | 48 | def __call__(self, model: Type[models.Model]): 49 | super().__call__(model) 50 | if model._meta.get_field(self.partition_key).get_internal_type() != models.DateTimeField().get_internal_type(): 51 | raise ValueError("The partition_key must be DateTimeField type.") 52 | model.partitioning = TimeRangePartitionManager(model, self.partition_key, self.options) 53 | return model 54 | 55 | 56 | class ListPartitioning(_PartitioningBase): 57 | """Use this decorator to declare the database table corresponding to the model to be partitioned by list. 58 | 59 | Parameters: 60 | partition_key(str): Partition key name, the type of the key must be one of boolean, text or integer. 61 | 62 | Example: 63 | .. code-block:: python 64 | 65 | from django.db import models 66 | from django.utils import timezone 67 | 68 | from pg_partitioning.decorators import ListPartitioning 69 | 70 | 71 | @ListPartitioning(partition_key="category") 72 | class MyLog(models.Model): 73 | category = models.TextField(default="A") 74 | timestamp = models.DateTimeField(default=timezone.now, primary_key=True) 75 | """ 76 | 77 | def __call__(self, model: Type[models.Model]): 78 | super().__call__(model) 79 | model.partitioning = ListPartitionManager(model, self.partition_key, self.options) 80 | return model 81 | -------------------------------------------------------------------------------- /pg_partitioning/manager.py: -------------------------------------------------------------------------------- 1 | import datetime 2 | from collections import Iterable 3 | from typing import Optional, Type, Union 4 | 5 | import pytz 6 | from dateutil.relativedelta import MO, relativedelta 7 | from django.conf import settings 8 | from django.db import IntegrityError, models 9 | from django.db.models import Q 10 | from django.utils import timezone 11 | 12 | from pg_partitioning.shortcuts import double_quote, execute_sql, generate_set_indexes_tablespace_sql, single_quote 13 | 14 | from .constants import ( 15 | DT_FORMAT, 16 | SQL_APPEND_TABLESPACE, 17 | SQL_ATTACH_LIST_PARTITION, 18 | SQL_CREATE_LIST_PARTITION, 19 | SQL_DETACH_PARTITION, 20 | SQL_SET_TABLE_TABLESPACE, 21 | PartitioningType, 22 | PeriodType, 23 | ) 24 | from .models import PartitionConfig, PartitionLog 25 | 26 | 27 | class _PartitionManagerBase: 28 | type = None 29 | 30 | def __init__(self, model: Type[models.Model], partition_key: str, options: dict): 31 | self.model = model 32 | self.partition_key = partition_key 33 | self.options = options 34 | 35 | 36 | class TimeRangePartitionManager(_PartitionManagerBase): 37 | """Manage time-based partition APIs.""" 38 | 39 | type = PartitioningType.Range 40 | 41 | @property 42 | def config(self) -> PartitionConfig: 43 | """Get the latest PartitionConfig instance of this model. 44 | In order to avoid the race condition, we used **select_for_update** when querying. 45 | 46 | Returns: 47 | PartitionConfig: The latest PartitionConfig instance of this model. 48 | """ 49 | try: 50 | return PartitionConfig.objects.select_for_update().get(model_label=self.model._meta.label_lower) 51 | except PartitionConfig.DoesNotExist: 52 | try: 53 | return PartitionConfig.objects.create( 54 | model_label=self.model._meta.label_lower, 55 | period=self.options.get("default_period", PeriodType.Month), 56 | interval=self.options.get("default_interval"), 57 | attach_tablespace=self.options.get("default_attach_tablespace"), 58 | detach_tablespace=self.options.get("default_detach_tablespace"), 59 | ) 60 | except IntegrityError: 61 | return PartitionConfig.objects.select_for_update().get(model_label=self.model._meta.label_lower) 62 | 63 | @property 64 | def latest(self) -> Optional[PartitionLog]: 65 | """Get the latest PartitionLog instance of this model. 66 | 67 | Returns: 68 | Optional[PartitionLog]: The latest PartitionLog instance of this model or none. 69 | """ 70 | return self.config.logs.order_by("-id").first() 71 | 72 | @classmethod 73 | def _get_period_bound(cls, date_start, initial, addition_zeros=None, is_week=False, **kwargs): 74 | zeros = {"hour": 0, "minute": 0, "second": 0, "microsecond": 0} 75 | if addition_zeros: 76 | zeros.update(addition_zeros) 77 | 78 | def func(): # lazy evaluation 79 | if initial: 80 | start = date_start.replace(**zeros) 81 | if is_week: 82 | start -= relativedelta(days=start.weekday()) 83 | else: 84 | start = date_start 85 | end = start + relativedelta(**kwargs, **zeros) 86 | return start, end 87 | 88 | return func 89 | 90 | def create_partition(self, max_days_to_next_partition: int = 1) -> None: 91 | """The partition of the next cycle is created according to the configuration. 92 | After modifying the period field, the new period will take effect the next time. 93 | The start time of the new partition is the end time of the previous partition table, 94 | or the start time of the current archive period when no partition exists. 95 | 96 | For example: 97 | the current time is June 5, 2018, and the archiving period is one year, 98 | then the start time of the first partition is 00:00:00 on January 1, 2018. 99 | 100 | Parameters: 101 | max_days_to_next_partition(int): 102 | If numbers of days remained in current partition is greater than ``max_days_to_next_partition``, no new partitions will be created. 103 | """ 104 | while True: 105 | if max_days_to_next_partition > 0 and self.latest and timezone.now() < (self.latest.end - relativedelta(days=max_days_to_next_partition)): 106 | return 107 | 108 | partition_timezone = getattr(settings, "PARTITION_TIMEZONE", None) 109 | if partition_timezone: 110 | partition_timezone = pytz.timezone(partition_timezone) 111 | date_start = timezone.localtime(self.latest.end if self.latest else None, timezone=partition_timezone) 112 | initial = not bool(self.latest) 113 | date_start, date_end = { 114 | PeriodType.Day: self._get_period_bound(date_start, initial, days=+1), 115 | PeriodType.Week: self._get_period_bound(date_start, initial, is_week=True, days=+1, weekday=MO), 116 | PeriodType.Month: self._get_period_bound(date_start, initial, addition_zeros=dict(day=1), months=+1), 117 | PeriodType.Year: self._get_period_bound(date_start, initial, addition_zeros=dict(month=1, day=1), years=+1), 118 | }[self.config.period]() 119 | 120 | partition_table_name = "_".join((self.model._meta.db_table, date_start.strftime(DT_FORMAT), date_end.strftime(DT_FORMAT))) 121 | PartitionLog.objects.create(config=self.config, table_name=partition_table_name, start=date_start, end=date_end) 122 | 123 | if not max_days_to_next_partition > 0: 124 | return 125 | 126 | def attach_partition(self, partition_log: Optional[Iterable] = None, detach_time: Optional[datetime.datetime] = None) -> None: 127 | """Attach partitions. 128 | 129 | Parameters: 130 | partition_log(Optional[Iterable]): 131 | All partitions are attached when you don't specify partitions to attach. 132 | detach_time(Optional[datetime.datetime]): 133 | When the partition specifies the archive time, it will **not** be automatically archived until that time. 134 | """ 135 | if not partition_log: 136 | partition_log = PartitionLog.objects.filter(config=self.config, is_attached=False) 137 | 138 | for log in partition_log: 139 | log.is_attached = True 140 | log.detach_time = detach_time 141 | log.save() 142 | 143 | def detach_partition(self, partition_log: Optional[Iterable] = None) -> None: 144 | """Detach partitions. 145 | 146 | Parameters: 147 | partition_log(Optional[Iterable]): 148 | Specify a partition to archive. When you don't specify a partition to archive, all partitions that meet the configuration rule are archived. 149 | """ 150 | if not partition_log: 151 | if self.config.interval: 152 | # fmt: off 153 | period = {PeriodType.Day: {"days": 1}, 154 | PeriodType.Week: {"weeks": 1}, 155 | PeriodType.Month: {"months": 1}, 156 | PeriodType.Year: {"years": 1}}[self.config.period] 157 | # fmt: on 158 | now = timezone.now() 159 | detach_timeline = now - self.config.interval * relativedelta(**period) 160 | partition_log = PartitionLog.objects.filter(config=self.config, end__lt=detach_timeline, is_attached=True) 161 | partition_log = partition_log.filter(Q(detach_time=None) | Q(detach_time__lt=now)) 162 | else: 163 | return 164 | 165 | for log in partition_log: 166 | log.is_attached = False 167 | log.detach_time = None 168 | log.save() 169 | 170 | def delete_partition(self, partition_log: Iterable) -> None: 171 | """Delete partitions. 172 | 173 | Parameters: 174 | partition_log(Iterable): The partitions to be deleted. 175 | """ 176 | for log in partition_log: 177 | if log.config == self.config: 178 | log.delete() 179 | 180 | 181 | def _db_value(value: Union[str, int, bool, None]) -> str: 182 | if value is None: 183 | return "null" 184 | return single_quote(value) if isinstance(value, str) else str(value) 185 | 186 | 187 | class ListPartitionManager(_PartitionManagerBase): 188 | """Manage list-based partition APIs.""" 189 | 190 | type = PartitioningType.List 191 | 192 | def create_partition(self, partition_name: str, value: Union[str, int, bool, None], tablespace: str = None) -> None: 193 | """Create partitions. 194 | 195 | Parameters: 196 | partition_name(str): Partition name. 197 | value(Union[str, int, bool, None]): Partition key value. 198 | tablespace(str): Partition tablespace name. 199 | """ 200 | 201 | create_partition_sql = SQL_CREATE_LIST_PARTITION % { 202 | "parent": double_quote(self.model._meta.db_table), 203 | "child": double_quote(partition_name), 204 | "value": _db_value(value), 205 | } 206 | if tablespace: 207 | create_partition_sql += SQL_APPEND_TABLESPACE % {"tablespace": tablespace} 208 | execute_sql(create_partition_sql) 209 | execute_sql(generate_set_indexes_tablespace_sql(partition_name, tablespace)) 210 | 211 | def attach_partition(self, partition_name: str, value: Union[str, int, bool, None], tablespace: str = None) -> None: 212 | """Attach partitions. 213 | 214 | Parameters: 215 | partition_name(str): Partition name. 216 | value(Union[str, int, bool, None]): Partition key value. 217 | tablespace(str): Partition tablespace name. 218 | """ 219 | 220 | sql_sequence = list() 221 | if tablespace: 222 | sql_sequence.append(SQL_SET_TABLE_TABLESPACE % {"name": double_quote(partition_name), "tablespace": tablespace}) 223 | sql_sequence.extend(generate_set_indexes_tablespace_sql(partition_name, tablespace)) 224 | sql_sequence.append( 225 | SQL_ATTACH_LIST_PARTITION % {"parent": double_quote(self.model._meta.db_table), "child": double_quote(partition_name), "value": _db_value(value)} 226 | ) 227 | execute_sql(sql_sequence) 228 | 229 | def detach_partition(self, partition_name: str, tablespace: str = None) -> None: 230 | """Detach partitions. 231 | 232 | Parameters: 233 | partition_name(str): Partition name. 234 | tablespace(str): Partition tablespace name. 235 | """ 236 | sql_sequence = [SQL_DETACH_PARTITION % {"parent": double_quote(self.model._meta.db_table), "child": double_quote(partition_name)}] 237 | if tablespace: 238 | sql_sequence.append(SQL_SET_TABLE_TABLESPACE % {"name": double_quote(partition_name), "tablespace": tablespace}) 239 | sql_sequence.extend(generate_set_indexes_tablespace_sql(partition_name, tablespace)) 240 | execute_sql(sql_sequence) 241 | -------------------------------------------------------------------------------- /pg_partitioning/migrations/0001_initial.py: -------------------------------------------------------------------------------- 1 | # Generated by Django 2.1.7 on 2019-02-17 12:00 2 | 3 | import django.db.models.deletion 4 | from django.db import migrations, models 5 | 6 | 7 | class Migration(migrations.Migration): 8 | 9 | initial = True 10 | 11 | dependencies = [ 12 | ] 13 | 14 | operations = [ 15 | migrations.CreateModel( 16 | name='PartitionConfig', 17 | fields=[ 18 | ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), 19 | ('model_label', models.TextField(unique=True)), 20 | ('period', models.TextField(default='Month')), 21 | ('interval', models.PositiveIntegerField(null=True)), 22 | ('attach_tablespace', models.TextField(null=True)), 23 | ('detach_tablespace', models.TextField(null=True)), 24 | ], 25 | ), 26 | migrations.CreateModel( 27 | name='PartitionLog', 28 | fields=[ 29 | ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), 30 | ('table_name', models.TextField(unique=True)), 31 | ('start', models.DateTimeField()), 32 | ('end', models.DateTimeField()), 33 | ('is_attached', models.BooleanField(default=True)), 34 | ('detach_time', models.DateTimeField(null=True)), 35 | ('config', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='logs', to='pg_partitioning.PartitionConfig')), 36 | ], 37 | options={ 38 | 'ordering': ('-id',), 39 | }, 40 | ), 41 | ] 42 | -------------------------------------------------------------------------------- /pg_partitioning/migrations/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaitin/django-pg-partitioning/129f5ddd11c61f855b97e558159dd067a15ca18d/pg_partitioning/migrations/__init__.py -------------------------------------------------------------------------------- /pg_partitioning/models.py: -------------------------------------------------------------------------------- 1 | from django.apps import apps 2 | from django.db import models, transaction 3 | 4 | from pg_partitioning.signals import post_attach_partition, post_create_partition, post_detach_partition 5 | 6 | from .constants import ( 7 | SQL_APPEND_TABLESPACE, 8 | SQL_ATTACH_TIME_RANGE_PARTITION, 9 | SQL_CREATE_TIME_RANGE_PARTITION, 10 | SQL_DETACH_PARTITION, 11 | SQL_SET_TABLE_TABLESPACE, 12 | PeriodType, 13 | ) 14 | from .shortcuts import double_quote, drop_table, execute_sql, generate_set_indexes_tablespace_sql, single_quote 15 | 16 | 17 | class PartitionConfig(models.Model): 18 | """You can get the configuration object of the partition table through ``Model.partitioning.config``, 19 | You can only edit the following fields via the object's ``save`` method: 20 | """ 21 | 22 | model_label = models.TextField(unique=True) 23 | period = models.TextField(default=PeriodType.Month) 24 | """Partition period. you can only set options in the `PeriodType`. 25 | The default value is ``PeriodType.Month``. Changing this value will trigger the ``detach_partition`` method.""" 26 | interval = models.PositiveIntegerField(null=True) 27 | """Detaching period. The ``detach_partition`` method defaults to detach partitions before the interval * period. 28 | The default is None, ie no partition will be detached. Changing this value will trigger the ``detach_partition`` method.""" 29 | attach_tablespace = models.TextField(null=True) 30 | """The name of the tablespace specified when creating or attaching a partition. Modifying this field will only affect subsequent operations. 31 | A table migration may occur at this time.""" 32 | detach_tablespace = models.TextField(null=True) 33 | """The name of the tablespace specified when detaching a partition. Modifying this field will only affect subsequent operations. 34 | A table migration may occur at this time.""" 35 | 36 | def save(self, force_insert=False, force_update=False, using=None, update_fields=None): 37 | """This setting will take effect immediately when you modify the value of 38 | ``interval`` in the configuration. 39 | """ 40 | 41 | adding = self._state.adding 42 | model = apps.get_model(self.model_label) 43 | 44 | if not adding: 45 | prev = self.__class__.objects.get(pk=self.pk) 46 | 47 | with transaction.atomic(): 48 | super().save(force_insert, force_update, using, update_fields) 49 | if adding: 50 | # Creating first partition. 51 | model.partitioning.create_partition(0) 52 | 53 | if not adding: 54 | # Period or interval changed. 55 | if prev.period != self.period or (prev.interval != self.interval): 56 | model.partitioning.detach_partition() 57 | 58 | 59 | class PartitionLog(models.Model): 60 | """You can only edit the following fields via the object's ``save`` method:""" 61 | 62 | config = models.ForeignKey(PartitionConfig, on_delete=models.CASCADE, related_name="logs") 63 | table_name = models.TextField(unique=True) 64 | # range bound: [start, end) 65 | start = models.DateTimeField() 66 | end = models.DateTimeField() 67 | is_attached = models.BooleanField(default=True) 68 | """Whether the partition is a attached partition. changing the value will trigger an attaching or detaching operation.""" 69 | detach_time = models.DateTimeField(null=True) 70 | """When the value is not `None`, the partition will not be automatically detached before this time. The default is `None`.""" 71 | 72 | def save(self, force_insert=False, force_update=False, using=None, update_fields=None): 73 | """This setting will take effect immediately when you modify the value of 74 | ``is_attached`` in the configuration. 75 | """ 76 | 77 | model = apps.get_model(self.config.model_label) 78 | if self._state.adding: 79 | create_partition_sql = SQL_CREATE_TIME_RANGE_PARTITION % { 80 | "parent": double_quote(model._meta.db_table), 81 | "child": double_quote(self.table_name), 82 | "date_start": single_quote(self.start.isoformat()), 83 | "date_end": single_quote(self.end.isoformat()), 84 | } 85 | 86 | if self.config.attach_tablespace: 87 | create_partition_sql += SQL_APPEND_TABLESPACE % {"tablespace": self.config.attach_tablespace} 88 | 89 | with transaction.atomic(): 90 | super().save(force_insert, force_update, using, update_fields) 91 | execute_sql(create_partition_sql) 92 | execute_sql(generate_set_indexes_tablespace_sql(self.table_name, self.config.attach_tablespace)) 93 | post_create_partition.send(sender=model, partition_log=self) 94 | else: 95 | with transaction.atomic(): 96 | prev = self.__class__.objects.select_for_update().get(pk=self.pk) 97 | # Detach partition. 98 | if prev.is_attached and (not self.is_attached): 99 | sql_sequence = [SQL_DETACH_PARTITION % {"parent": double_quote(model._meta.db_table), "child": double_quote(self.table_name)}] 100 | if self.config.detach_tablespace: 101 | sql_sequence.append(SQL_SET_TABLE_TABLESPACE % {"name": double_quote(self.table_name), "tablespace": self.config.detach_tablespace}) 102 | sql_sequence.extend(generate_set_indexes_tablespace_sql(self.table_name, self.config.detach_tablespace)) 103 | 104 | super().save(force_insert, force_update, using, update_fields) 105 | execute_sql(sql_sequence) 106 | post_detach_partition.send(sender=model, partition_log=self) 107 | # Attach partition. 108 | elif (not prev.is_attached) and self.is_attached: 109 | sql_sequence = list() 110 | if self.config.attach_tablespace: 111 | sql_sequence.append(SQL_SET_TABLE_TABLESPACE % {"name": double_quote(self.table_name), "tablespace": self.config.attach_tablespace}) 112 | sql_sequence.extend(generate_set_indexes_tablespace_sql(self.table_name, self.config.attach_tablespace)) 113 | 114 | sql_sequence.append( 115 | SQL_ATTACH_TIME_RANGE_PARTITION 116 | % { 117 | "parent": double_quote(model._meta.db_table), 118 | "child": double_quote(self.table_name), 119 | "date_start": single_quote(self.start.isoformat()), 120 | "date_end": single_quote(self.end.isoformat()), 121 | } 122 | ) 123 | 124 | super().save(force_insert, force_update, using, update_fields) 125 | execute_sql(sql_sequence) 126 | post_attach_partition.send(sender=model, partition_log=self) 127 | # State has not changed. 128 | else: 129 | super().save(force_insert, force_update, using, update_fields) 130 | 131 | @transaction.atomic 132 | def delete(self, using=None, keep_parents=False): 133 | """When the instance is deleted, the partition corresponding to it will also be deleted.""" 134 | 135 | drop_table(self.table_name) 136 | super().delete(using, keep_parents) 137 | 138 | class Meta: 139 | ordering = ("-id",) 140 | -------------------------------------------------------------------------------- /pg_partitioning/patch/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaitin/django-pg-partitioning/129f5ddd11c61f855b97e558159dd067a15ca18d/pg_partitioning/patch/__init__.py -------------------------------------------------------------------------------- /pg_partitioning/patch/schema.py: -------------------------------------------------------------------------------- 1 | import logging 2 | from importlib import import_module 3 | 4 | from django.apps.config import MODELS_MODULE_NAME 5 | from django.db.backends.postgresql.schema import DatabaseSchemaEditor 6 | 7 | from pg_partitioning.manager import _PartitionManagerBase 8 | 9 | logger = logging.getLogger(__name__) 10 | 11 | 12 | default_create_model_method = DatabaseSchemaEditor.create_model 13 | default_sql_create_table = DatabaseSchemaEditor.sql_create_table 14 | 15 | 16 | def create_model(self, model): 17 | meta = model._meta 18 | try: 19 | cls_module = import_module(f"{meta.app_label}.{MODELS_MODULE_NAME}") 20 | except ModuleNotFoundError: 21 | cls_module = None 22 | cls = getattr(cls_module, meta.object_name, None) 23 | partitioning = getattr(cls, "partitioning", None) 24 | if isinstance(partitioning, _PartitionManagerBase): 25 | # XXX: Monkeypatch create_model. 26 | logger.debug("Partitioned model detected: %s", meta.label) 27 | _type = partitioning.type 28 | key = partitioning.partition_key 29 | DatabaseSchemaEditor.sql_create_table = f"CREATE TABLE %(table)s (%(definition)s) PARTITION BY {_type} ({key})" 30 | if meta.pk.name != key: 31 | """The partition key must be part of the primary key, 32 | and currently Django does not support setting a composite primary key, 33 | so its properties are turned off.""" 34 | meta.pk.primary_key = False 35 | logger.info("Note that PK constraints for %s has been temporarily closed.", meta.label) 36 | else: 37 | DatabaseSchemaEditor.sql_create_table = default_sql_create_table 38 | default_create_model_method(self, model) 39 | meta.pk.primary_key = True 40 | 41 | 42 | DatabaseSchemaEditor.create_model = create_model 43 | -------------------------------------------------------------------------------- /pg_partitioning/shortcuts.py: -------------------------------------------------------------------------------- 1 | import logging 2 | from typing import List, Optional, Tuple, Union 3 | 4 | from django.db import connection 5 | 6 | from pg_partitioning.constants import SQL_DROP_TABLE, SQL_GET_TABLE_INDEXES, SQL_SET_INDEX_TABLESPACE, SQL_SET_TABLE_TABLESPACE, SQL_TRUNCATE_TABLE 7 | 8 | logger = logging.getLogger(__name__) 9 | 10 | 11 | def single_quote(name: str) -> str: 12 | """Represent a string constants in SQL.""" 13 | 14 | if name.startswith("'") and name.endswith("'"): 15 | return name 16 | return "'%s'" % name 17 | 18 | 19 | def double_quote(name: str) -> str: 20 | """Represent a identify in SQL.""" 21 | 22 | if name.startswith('"') and name.endswith('"'): 23 | return name 24 | return '"%s"' % name 25 | 26 | 27 | def execute_sql(sql_sequence: Union[str, List[str], Tuple[str]], fetch: bool = False) -> Optional[List]: 28 | """Execute SQL sequence and returning result.""" 29 | if not sql_sequence: 30 | if fetch: 31 | return [] 32 | return 33 | 34 | sql_str = "" 35 | for statement in sql_sequence if isinstance(sql_sequence, (list, tuple)) else [sql_sequence]: 36 | sql_str += ";\n" + statement if sql_str else statement 37 | logger.debug("The sequence of SQL statements to be executed:\n %s", sql_str) 38 | with connection.cursor() as cursor: 39 | cursor.execute(sql_str) 40 | if fetch: 41 | return cursor.fetchall() 42 | 43 | 44 | def generate_set_indexes_tablespace_sql(table_name: str, tablespace: str) -> List[str]: 45 | """Generate set indexes tablespace SQL sequence. 46 | 47 | Parameters: 48 | table_name(str): Table name. 49 | tablespace(str): Partition tablespace. 50 | """ 51 | 52 | sql_sequence = [] 53 | result = execute_sql(SQL_GET_TABLE_INDEXES % {"table_name": single_quote(table_name)}, fetch=True) 54 | for item in result: 55 | sql_sequence.append(SQL_SET_INDEX_TABLESPACE % {"name": double_quote(item[0]), "tablespace": tablespace}) 56 | return sql_sequence 57 | 58 | 59 | def set_tablespace(table_name: str, tablespace: str) -> None: 60 | """Set the tablespace for a table and indexes. 61 | 62 | Parameters: 63 | table_name(str): Table name. 64 | tablespace(str): Tablespace name. 65 | """ 66 | 67 | sql_sequence = [SQL_SET_TABLE_TABLESPACE % {"name": double_quote(table_name), "tablespace": tablespace}] 68 | sql_sequence.extend(generate_set_indexes_tablespace_sql(table_name, tablespace)) 69 | execute_sql(sql_sequence) 70 | 71 | 72 | def truncate_table(table_name: str) -> None: 73 | """Truncate table. 74 | 75 | Parameters: 76 | table_name(str): Table name. 77 | """ 78 | 79 | execute_sql(SQL_TRUNCATE_TABLE % {"name": double_quote(table_name)}) 80 | 81 | 82 | def drop_table(table_name: str) -> None: 83 | """Drop table. 84 | 85 | Parameters: 86 | table_name(str): Table name. 87 | """ 88 | 89 | execute_sql(SQL_DROP_TABLE % {"name": double_quote(table_name)}) 90 | -------------------------------------------------------------------------------- /pg_partitioning/signals.py: -------------------------------------------------------------------------------- 1 | from django.dispatch import Signal 2 | 3 | post_create_partition = Signal(providing_args=["partition_log"]) 4 | """Sent when a partition is created. 5 | """ 6 | post_attach_partition = Signal(providing_args=["partition_log"]) 7 | """Sent when a partition is attached. 8 | """ 9 | post_detach_partition = Signal(providing_args=["partition_log"]) 10 | """Sent when a partition is detached. 11 | """ 12 | -------------------------------------------------------------------------------- /run_test.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import sys 3 | 4 | import dj_database_url 5 | import django 6 | from django.core.management import call_command 7 | 8 | 9 | def setup_django_environment(): 10 | from django.conf import settings 11 | 12 | settings.configure( 13 | DEBUG_PROPAGATE_EXCEPTIONS=True, 14 | DATABASES={ 15 | "default": dj_database_url.config(env="DATABASE_URL", 16 | default="postgres://test:test@localhost/test", 17 | conn_max_age=20) 18 | }, 19 | SECRET_KEY="not very secret in tests", 20 | USE_I18N=True, 21 | USE_L10N=True, 22 | USE_TZ=True, 23 | TIME_ZONE="Asia/Shanghai", 24 | INSTALLED_APPS=( 25 | "pg_partitioning", 26 | "tests", 27 | ), 28 | LOGGING={ 29 | "version": 1, 30 | "disable_existing_loggers": False, 31 | "formatters": { 32 | "standard": { 33 | "format": "[%(asctime)s] %(message)s", 34 | "datefmt": "%Y-%m-%d %H:%M:%S" 35 | } 36 | }, 37 | "handlers": { 38 | "console": { 39 | "level": "DEBUG", 40 | "class": "logging.StreamHandler", 41 | "formatter": "standard" 42 | } 43 | }, 44 | "loggers": { 45 | "pg_partitioning.shortcuts": { 46 | "handlers": ["console"], 47 | "level": "DEBUG", 48 | "propagate": False, 49 | }, 50 | "pg_partitioning.patch.schema": { 51 | "handlers": ["console"], 52 | "level": "DEBUG", 53 | "propagate": False, 54 | }, 55 | }, 56 | } 57 | ) 58 | 59 | django.setup() 60 | 61 | 62 | if __name__ == "__main__": 63 | parser = argparse.ArgumentParser(description="Run the pg_partitioning test suite.") 64 | parser.add_argument("-c", "--coverage", dest="use_coverage", action="store_true", help="Run coverage to collect code coverage and generate report.") 65 | options = parser.parse_args() 66 | 67 | if options.use_coverage: 68 | try: 69 | from coverage import coverage 70 | except ImportError: 71 | options.use_coverage = False 72 | 73 | if options.use_coverage: 74 | cov = coverage() 75 | cov.start() 76 | 77 | setup_django_environment() 78 | call_command("test", verbosity=2, interactive=False, stdout=sys.stdout) 79 | 80 | if options.use_coverage: 81 | print("\nRunning Code Coverage...\n") 82 | cov.stop() 83 | cov.report() 84 | cov.xml_report() 85 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [flake8] 2 | max-complexity = 20 3 | max-line-length = 157 4 | 5 | inline-quotes = " 6 | multiline-quotes = """ 7 | 8 | [isort] 9 | not_skip = __init__.py 10 | line_length = 157 11 | include_trailing_comma = true 12 | combine_as_imports = true 13 | multi_line_output = 3 14 | order_by_type = true 15 | 16 | [bdist_wheel] 17 | universal=1 18 | 19 | [coverage:run] 20 | include = pg_partitioning/*,test/* 21 | omit = 22 | pg_partitioning/migrations/* 23 | branch = True 24 | 25 | [coverage:report] 26 | exclude_lines = 27 | pragma: no cover 28 | raise NotImplementedError 29 | ignore_errors = True 30 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | import os 2 | 3 | from setuptools import setup 4 | 5 | 6 | def rel(*xs): 7 | return os.path.join(os.path.abspath(os.path.dirname(__file__)), *xs) 8 | 9 | 10 | with open(rel("README.rst")) as f: 11 | long_description = f.read() 12 | 13 | 14 | with open(rel("pg_partitioning", "__init__.py"), "r") as f: 15 | version_marker = "__version__ = " 16 | for line in f: 17 | if line.startswith(version_marker): 18 | _, version = line.split(version_marker) 19 | version = version.strip().strip('"') 20 | break 21 | else: 22 | raise RuntimeError("Version marker not found.") 23 | 24 | 25 | dependencies = [ 26 | "python-dateutil~=2.7", 27 | ] 28 | 29 | extra_dependencies = { 30 | "django": [ 31 | "Django>=2.0,<3.0" 32 | ], 33 | } 34 | 35 | extra_dependencies["all"] = list(set(sum(extra_dependencies.values(), []))) 36 | extra_dependencies["dev"] = extra_dependencies["all"] + [ 37 | # Pinned due to https://bitbucket.org/ned/coveragepy/issues/578/incomplete-file-path-in-xml-report 38 | "coverage>=4.0,<4.4", 39 | 40 | # Docs 41 | "alabaster==0.7.12", 42 | "sphinx==1.8.3", 43 | "sphinxcontrib-napoleon==0.7", 44 | 45 | # Linting 46 | "flake8~=3.6.0", 47 | "isort~=4.3.4", 48 | "black~=18.9b0", 49 | "flake8-bugbear~=18.8.0", 50 | "flake8-quotes~=1.0.0", 51 | 52 | # Misc 53 | "dj-database-url==0.5.0", 54 | "psycopg2-binary==2.7.6.1", 55 | "twine==1.12.1", 56 | 57 | # Testing 58 | "tox==3.9.0", 59 | "tox-venv==0.4.0" 60 | ] 61 | 62 | 63 | setup( 64 | name="django-pg-partitioning", 65 | version=version, 66 | author="Boyce Li", 67 | author_email="monobiao@gmail.com", 68 | description="A Django extension that supports PostgreSQL 11 time ranges and list partitioning.", 69 | long_description=long_description, 70 | long_description_content_type="text/x-rst", 71 | url="https://github.com/chaitin/django-pg-partitioning", 72 | packages=["pg_partitioning", "pg_partitioning.migrations", "pg_partitioning.patch"], 73 | include_package_data=True, 74 | install_requires=dependencies, 75 | extras_require=extra_dependencies, 76 | python_requires=">=3.6", 77 | classifiers=[ 78 | "Development Status :: 4 - Beta", 79 | "Programming Language :: Python :: 3.6", 80 | "Programming Language :: Python :: 3.7", 81 | "Framework :: Django :: 2.0", 82 | "Framework :: Django :: 2.1", 83 | "License :: OSI Approved :: MIT License", 84 | "Operating System :: OS Independent", 85 | "Topic :: Software Development :: Libraries :: Python Modules" 86 | ], 87 | ) 88 | -------------------------------------------------------------------------------- /tests/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaitin/django-pg-partitioning/129f5ddd11c61f855b97e558159dd067a15ca18d/tests/__init__.py -------------------------------------------------------------------------------- /tests/models.py: -------------------------------------------------------------------------------- 1 | from django.contrib.postgres.indexes import BrinIndex 2 | from django.db import models 3 | from django.utils import timezone 4 | 5 | from pg_partitioning.constants import PeriodType 6 | from pg_partitioning.decorators import ListPartitioning, TimeRangePartitioning 7 | 8 | 9 | @TimeRangePartitioning(partition_key="timestamp", default_period=PeriodType.Month, default_attach_tablespace="data1", default_detach_tablespace="data2") 10 | class TimeRangeTableA(models.Model): 11 | text = models.TextField() 12 | timestamp = models.DateTimeField(default=timezone.now) 13 | 14 | class Meta: 15 | db_tablespace = "pg_default" 16 | indexes = [BrinIndex(fields=["timestamp"])] 17 | unique_together = ("text", "timestamp") 18 | ordering = ["text"] 19 | 20 | 21 | @TimeRangePartitioning(partition_key="timestamp", default_period=PeriodType.Day, default_attach_tablespace="data2", default_detach_tablespace="data1") 22 | class TimeRangeTableB(models.Model): 23 | text = models.TextField() 24 | timestamp = models.DateTimeField(default=timezone.now) 25 | 26 | class Meta: 27 | indexes = [BrinIndex(fields=["timestamp"])] 28 | unique_together = ("text", "timestamp") 29 | ordering = ["text"] 30 | 31 | 32 | @ListPartitioning(partition_key="category") 33 | class ListTableText(models.Model): 34 | category = models.TextField(default="A", null=True, blank=True) 35 | timestamp = models.DateTimeField(default=timezone.now) 36 | 37 | 38 | @ListPartitioning(partition_key="category") 39 | class ListTableInt(models.Model): 40 | category = models.IntegerField(default=0, null=True) 41 | timestamp = models.DateTimeField(default=timezone.now) 42 | 43 | 44 | @ListPartitioning(partition_key="category") 45 | class ListTableBool(models.Model): 46 | category = models.NullBooleanField(default=False, null=True) 47 | timestamp = models.DateTimeField(default=timezone.now) 48 | -------------------------------------------------------------------------------- /tests/tests.py: -------------------------------------------------------------------------------- 1 | import datetime 2 | from unittest.mock import patch 3 | 4 | from dateutil.relativedelta import MO, relativedelta 5 | from django.db import connection 6 | from django.test import TestCase 7 | from django.utils import timezone 8 | from django.utils.crypto import get_random_string 9 | 10 | from pg_partitioning.constants import SQL_GET_TABLE_INDEXES, PeriodType 11 | from pg_partitioning.models import PartitionConfig, PartitionLog 12 | from pg_partitioning.shortcuts import single_quote 13 | 14 | from .models import ListTableBool, ListTableInt, ListTableText, TimeRangeTableA, TimeRangeTableB 15 | 16 | 17 | def t(year=2018, month=8, day=25, hour=7, minute=15, second=15, millisecond=0): 18 | """A point in time.""" 19 | return timezone.get_current_timezone().localize(datetime.datetime(year, month, day, hour, minute, second, millisecond)) 20 | 21 | 22 | def tz(time): 23 | return timezone.localtime(time) 24 | 25 | 26 | class GeneralTestCase(TestCase): 27 | def assertTablespace(self, table_name, tablespace): 28 | with connection.cursor() as cursor: 29 | cursor.execute(f"SELECT tablespace FROM pg_tables WHERE tablename = {single_quote(table_name)};") 30 | rows = cursor.fetchall() 31 | self.assertEqual(tablespace, rows[0][0]) 32 | cursor.execute(SQL_GET_TABLE_INDEXES % {"table_name": single_quote(table_name)}) 33 | rows = cursor.fetchall() 34 | for row in rows: 35 | cursor.execute(f"SELECT tablespace FROM pg_indexes WHERE indexname = {single_quote(row[0])};") 36 | rows = cursor.fetchall() 37 | self.assertEqual(tablespace, rows[0][0]) 38 | 39 | 40 | class TimeRangePartitioningTestCase(GeneralTestCase): 41 | def assertTimeRangeEqual(self, model, time_start, time_end): 42 | self.assertListEqual([time_start, time_end], [tz(model.partitioning.latest.start), tz(model.partitioning.latest.end)]) 43 | 44 | # Verify that the partition has been created by inserting data. 45 | model.objects.create(text=get_random_string(length=32), timestamp=time_start) 46 | model.objects.create(text=get_random_string(length=32), timestamp=time_end - relativedelta(microseconds=1)) 47 | 48 | def _create_partition(self, period, start_date, delta): 49 | TimeRangeTableB.partitioning.options["default_period"] = period 50 | 51 | for i in range(0, 3): 52 | if i == 0: 53 | TimeRangeTableB.partitioning.config # Create first partition by side effect. 54 | else: 55 | TimeRangeTableB.partitioning.create_partition(0) 56 | end_date = start_date + delta 57 | self.assertTimeRangeEqual(TimeRangeTableB, start_date, end_date) 58 | start_date = end_date 59 | 60 | @patch("django.utils.timezone.now", new=t) 61 | def test_create_partition_week(self): 62 | self._create_partition(PeriodType.Week, t(2018, 8, 20, 0, 0, 0), relativedelta(days=1, weekday=MO)) 63 | 64 | @patch("django.utils.timezone.now", new=t) 65 | def test_create_partition_day(self): 66 | self._create_partition(PeriodType.Day, t(2018, 8, 25, 0, 0, 0), relativedelta(days=1)) 67 | 68 | @patch("django.utils.timezone.now", new=t) 69 | def test_create_partition_month(self): 70 | self._create_partition(PeriodType.Month, t(2018, 8, 1, 0, 0, 0), relativedelta(months=1)) 71 | 72 | @patch("django.utils.timezone.now", new=t) 73 | def test_create_partition_year(self): 74 | self._create_partition(PeriodType.Year, t(2018, 1, 1, 0, 0, 0), relativedelta(years=1)) 75 | 76 | @classmethod 77 | def _update_config_period(cls, config: PartitionConfig, period: str): 78 | config.period = period 79 | config.save() 80 | 81 | def test_create_partition(self): 82 | with patch("django.utils.timezone.now", new=t): 83 | config_a: PartitionConfig = TimeRangeTableA.partitioning.config 84 | 85 | self.assertEqual(1, config_a.logs.count()) 86 | self.assertTimeRangeEqual(TimeRangeTableA, t(2018, 8, 1, 0, 0, 0), t(2018, 9, 1, 0, 0, 0)) 87 | 88 | # Repeated calls will not produce wrong results (idempotence). 89 | for _ in range(3): 90 | TimeRangeTableA.partitioning.create_partition() 91 | 92 | self.assertTimeRangeEqual(TimeRangeTableA, t(2018, 8, 1, 0, 0, 0), t(2018, 9, 1, 0, 0, 0)) 93 | 94 | # Perform a series of partition creation operations. 95 | self._update_config_period(config_a, PeriodType.Week) 96 | TimeRangeTableA.partitioning.create_partition(0) 97 | self.assertTimeRangeEqual(TimeRangeTableA, t(2018, 9, 1, 0, 0, 0), t(2018, 9, 3, 0, 0, 0)) 98 | 99 | self._update_config_period(config_a, PeriodType.Day) 100 | TimeRangeTableA.partitioning.create_partition(0) 101 | self.assertTimeRangeEqual(TimeRangeTableA, t(2018, 9, 3, 0, 0, 0), t(2018, 9, 4, 0, 0, 0)) 102 | 103 | self._update_config_period(config_a, PeriodType.Month) 104 | TimeRangeTableA.partitioning.create_partition(0) 105 | self.assertTimeRangeEqual(TimeRangeTableA, t(2018, 9, 4, 0, 0, 0), t(2018, 10, 1, 0, 0, 0)) 106 | TimeRangeTableA.partitioning.create_partition(0) 107 | self.assertTimeRangeEqual(TimeRangeTableA, t(2018, 10, 1, 0, 0, 0), t(2018, 11, 1, 0, 0, 0)) 108 | 109 | self._update_config_period(config_a, PeriodType.Year) 110 | TimeRangeTableA.partitioning.create_partition(0) 111 | self.assertTimeRangeEqual(TimeRangeTableA, t(2018, 11, 1, 0, 0, 0), t(2019, 1, 1, 0, 0, 0)) 112 | TimeRangeTableA.partitioning.create_partition(0) 113 | self.assertTimeRangeEqual(TimeRangeTableA, t(2019, 1, 1, 0, 0, 0), t(2020, 1, 1, 0, 0, 0)) 114 | 115 | def test_max_days_to_next_partition(self): 116 | with patch("django.utils.timezone.now", new=t): 117 | TimeRangeTableA.partitioning.create_partition() 118 | TimeRangeTableA.partitioning.create_partition(0) 119 | self.assertTimeRangeEqual(TimeRangeTableA, t(2018, 9, 1, 0, 0, 0), t(2018, 10, 1, 0, 0, 0)) 120 | 121 | with patch("django.utils.timezone.now", return_value=t(2018, 8, 2, 0, 0, 0)): 122 | TimeRangeTableA.partitioning.create_partition(59) 123 | self.assertTimeRangeEqual(TimeRangeTableA, t(2018, 9, 1, 0, 0, 0), t(2018, 10, 1, 0, 0, 0)) 124 | TimeRangeTableA.partitioning.create_partition(60) 125 | self.assertTimeRangeEqual(TimeRangeTableA, t(2018, 10, 1, 0, 0, 0), t(2018, 11, 1, 0, 0, 0)) 126 | 127 | with patch("django.utils.timezone.now", return_value=t(2019, 3, 3, 0, 0, 0)): 128 | TimeRangeTableA.partitioning.create_partition(5) 129 | self.assertTimeRangeEqual(TimeRangeTableA, t(2019, 3, 1, 0, 0, 0), t(2019, 4, 1, 0, 0, 0)) 130 | 131 | def test_attach_or_detach_partition(self): 132 | self.test_create_partition() 133 | 134 | config_a: PartitionConfig = TimeRangeTableA.partitioning.config 135 | 136 | self.assertEqual(0, config_a.logs.filter(is_attached=False).count()) 137 | TimeRangeTableA.partitioning.detach_partition(config_a.logs.all()) 138 | self.assertEqual(0, config_a.logs.filter(is_attached=True).count()) 139 | TimeRangeTableA.partitioning.attach_partition(config_a.logs.all()) 140 | self.assertEqual(0, config_a.logs.filter(is_attached=False).count()) 141 | 142 | with patch("django.utils.timezone.now", return_value=t(2018, 10, 15, 12, 1, 4)): 143 | config_a.period = PeriodType.Day 144 | config_a.interval = 15 145 | config_a.save() 146 | self.assertEqual(t(2018, 10, 1, 0, 0, 0), tz(config_a.logs.filter(is_attached=True).order_by("start").first().end)) 147 | TimeRangeTableA.partitioning.attach_partition(config_a.logs.all()) 148 | 149 | config_a.period = PeriodType.Week 150 | config_a.interval = 2 151 | config_a.save() 152 | self.assertEqual(t(2018, 11, 1, 0, 0, 0), tz(config_a.logs.filter(is_attached=True).order_by("start").first().end)) 153 | TimeRangeTableA.partitioning.attach_partition(config_a.logs.all()) 154 | 155 | log = config_a.logs.filter(is_attached=True).order_by("start").first() 156 | self.assertEqual(t(2018, 9, 1, 0, 0, 0), log.end) 157 | 158 | log.detach_time = t(2018, 10, 15, 12, 1, 5) 159 | log.save() 160 | 161 | config_a.period = PeriodType.Month 162 | config_a.interval = 1 163 | config_a.save() 164 | 165 | self.assertEqual(t(2018, 9, 1, 0, 0, 0), tz(config_a.logs.filter(is_attached=True).order_by("start").first().end)) 166 | log.refresh_from_db() 167 | self.assertEqual(True, log.is_attached) 168 | 169 | log.detach_time = None 170 | log.save() 171 | 172 | TimeRangeTableA.partitioning.detach_partition() 173 | self.assertEqual(t(2018, 10, 1, 0, 0, 0), tz(config_a.logs.filter(is_attached=True).order_by("start").first().end)) 174 | log.refresh_from_db() 175 | self.assertEqual(False, log.is_attached) 176 | 177 | def test_delete_partition(self): 178 | for _ in range(4): 179 | TimeRangeTableA.partitioning.create_partition(0) 180 | 181 | config_a: PartitionConfig = TimeRangeTableA.partitioning.config 182 | 183 | TimeRangeTableA.partitioning.delete_partition(config_a.logs.all()) 184 | self.assertEqual(0, config_a.logs.count()) 185 | 186 | with patch("django.utils.timezone.now", new=t): 187 | self._update_config_period(config_a, PeriodType.Day) 188 | TimeRangeTableA.partitioning.create_partition() 189 | 190 | self.assertEqual(2, config_a.logs.count()) 191 | self.assertTimeRangeEqual(TimeRangeTableA, t(2018, 8, 26, 0, 0, 0), t(2018, 8, 27, 0, 0, 0)) 192 | 193 | def test_attach_detach_tablespace(self): 194 | TimeRangeTableA.partitioning.create_partition() 195 | log: PartitionLog = TimeRangeTableA.partitioning.latest 196 | self.assertEqual(True, log.is_attached) 197 | self.assertTablespace(log.table_name, log.config.attach_tablespace) 198 | 199 | TimeRangeTableA.partitioning.detach_partition([log]) 200 | log.refresh_from_db() 201 | self.assertEqual(False, log.is_attached) 202 | self.assertTablespace(log.table_name, log.config.detach_tablespace) 203 | 204 | TimeRangeTableA.partitioning.attach_partition() 205 | log.refresh_from_db() 206 | self.assertEqual(True, log.is_attached) 207 | self.assertTablespace(log.table_name, log.config.attach_tablespace) 208 | 209 | 210 | class ListPartitioningTestCase(GeneralTestCase): 211 | @classmethod 212 | def assertCreated(cls, model, category): 213 | # Verify that the partition has been created by inserting data. 214 | model.objects.create(category=category) 215 | 216 | def test_create_partition(self): 217 | ListTableText.partitioning.create_partition("list_table_text_a", "A", "data1") 218 | self.assertCreated(ListTableText, "A") 219 | 220 | ListTableText.partitioning.create_partition("list_table_text_b", "B") 221 | self.assertCreated(ListTableText, "B") 222 | 223 | ListTableText.partitioning.create_partition("list_table_text_blank", "") 224 | self.assertCreated(ListTableText, "") 225 | 226 | ListTableText.partitioning.create_partition("list_table_text_none", None, "data2") 227 | self.assertCreated(ListTableText, None) 228 | 229 | ListTableInt.partitioning.create_partition("list_table_int_1", 1, "data1") 230 | self.assertCreated(ListTableInt, 1) 231 | 232 | ListTableInt.partitioning.create_partition("list_table_int_2", 2) 233 | self.assertCreated(ListTableInt, 2) 234 | 235 | ListTableInt.partitioning.create_partition("list_table_int_none", None, "data2") 236 | self.assertCreated(ListTableInt, None) 237 | 238 | ListTableBool.partitioning.create_partition("list_table_bool_true", True, "data1") 239 | self.assertCreated(ListTableBool, True) 240 | 241 | ListTableBool.partitioning.create_partition("list_table_bool_false", False) 242 | self.assertCreated(ListTableBool, False) 243 | 244 | ListTableBool.partitioning.create_partition("list_table_bool_none", None, "data2") 245 | self.assertCreated(ListTableBool, None) 246 | 247 | def assertTablespace(self, table_name, tablespace): 248 | with connection.cursor() as cursor: 249 | cursor.execute(f"SELECT tablespace FROM pg_tables WHERE tablename = '{table_name}';") 250 | rows = cursor.fetchall() 251 | self.assertEqual(tablespace, rows[0][0]) 252 | 253 | def test_attach_or_detach_partition(self): 254 | self.test_create_partition() 255 | ListTableText.partitioning.detach_partition("list_table_text_none", "data1") 256 | self.assertTablespace("list_table_text_none", "data1") 257 | ListTableText.partitioning.attach_partition("list_table_text_none", None, "data2") 258 | self.assertTablespace("list_table_text_none", "data2") 259 | -------------------------------------------------------------------------------- /tox.ini: -------------------------------------------------------------------------------- 1 | [tox] 2 | envlist = 3 | {py36,py37}-django{20,21,22} 4 | docs 5 | lint 6 | 7 | [travis:env] 8 | DJANGO = 9 | 2.0: django20 10 | 2.1: django21 11 | 2.2: django22 12 | 13 | [testenv] 14 | setenv = 15 | PYTHONDONTWRITEBYTECODE=1 16 | PYTHONWARNINGS=once 17 | envdir = {toxworkdir}/venvs/{envname} 18 | extras = 19 | dev 20 | deps = 21 | lint: isort 22 | black 23 | flake8 24 | flake8-bugbear 25 | flake8-quotes 26 | django20: Django>=2.0,<2.1 27 | django21: Django>=2.1,<2.2 28 | django22: Django>=2.2a1,<3.0 29 | 30 | commands = 31 | python3 ./run_test.py -c {posargs} 32 | 33 | [testenv:docs] 34 | basepython = python3.6 35 | whitelist_externals = make 36 | changedir = docs 37 | commands = 38 | make html 39 | 40 | [testenv:lint] 41 | basepython = python3.6 42 | commands = 43 | isort -rc pg_partitioning tests 44 | black pg_partitioning tests -l 157 --exclude pg_partitioning/migrations/* 45 | flake8 {toxinidir}/pg_partitioning {toxinidir}/tests {toxinidir}/*.py --exclude */migrations 46 | --------------------------------------------------------------------------------