├── .gitignore ├── README.md ├── django_s3_sqlite ├── __init__.py ├── base.py ├── client.py ├── features.py ├── management │ ├── __init__.py │ └── commands │ │ ├── __init__.py │ │ ├── create_admin_user.py │ │ └── s3_sqlite_vacuum.py └── migrations │ └── __init__.py ├── setup.py └── shared-objects ├── python-3-6 └── _sqlite3.so └── python-3-8 └── _sqlite3.so /.gitignore: -------------------------------------------------------------------------------- 1 | *~ 2 | *.zip 3 | 4 | # Byte-compiled / optimized / DLL files 5 | __pycache__/ 6 | *.py[cod] 7 | *$py.class 8 | 9 | # C extensions 10 | *.so 11 | 12 | # Distribution / packaging 13 | .Python 14 | env/ 15 | build/ 16 | develop-eggs/ 17 | dist/ 18 | downloads/ 19 | eggs/ 20 | .eggs/ 21 | lib/ 22 | lib64/ 23 | parts/ 24 | sdist/ 25 | var/ 26 | *.egg-info/ 27 | .installed.cfg 28 | *.egg 29 | 30 | # PyInstaller 31 | # Usually these files are written by a python script from a template 32 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 33 | *.manifest 34 | *.spec 35 | 36 | # Installer logs 37 | pip-log.txt 38 | pip-delete-this-directory.txt 39 | 40 | # Unit test / coverage reports 41 | htmlcov/ 42 | .tox/ 43 | .coverage 44 | .coveragerc 45 | .coverage.* 46 | .cache 47 | nosetests.xml 48 | coverage.xml 49 | *,cover 50 | .hypothesis/ 51 | 52 | # Translations 53 | *.mo 54 | *.pot 55 | 56 | # Django stuff: 57 | *.log 58 | 59 | # Sphinx documentation 60 | docs/_build/ 61 | 62 | # PyBuilder 63 | target/ 64 | 65 | # pyenv 66 | .python-version 67 | 68 | # PyCharm 69 | # Project settings from the PyCharm IDE are stored in the .idea folder 70 | .idea/ 71 | 72 | # Apple OS X specific files are put into the .DS_Store file 73 | .DS_Store 74 | 75 | # Vim stuff: 76 | *.swp 77 | *.swo 78 | *~ 79 | tests/zappa_settings.json 80 | tests/example_template_outputs 81 | 82 | # Locally generated settings file 83 | zappa_settings.json 84 | 85 | # Sublime Text stuff 86 | *.sublime-project 87 | *.sublime-workspace 88 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # django-s3-sqlite 2 | 3 | [![PyPI](https://img.shields.io/pypi/v/django-s3-sqlite.svg)](https://pypi.python.org/pypi/django-s3-sqlite) 4 | 5 | This project was inspired and started with [Zappa](https://github.com/Miserlou/Zappa). Thanks to [Rich Jones](https://github.com/Miserlou) for all of his amazing work. 6 | 7 | ## Installation 8 | 9 | Install via `pip`: 10 | 11 | $ pip install django-s3-sqlite 12 | 13 | Add to your installed apps: 14 | 15 | INSTALLED_APPS += ["django_s3_sqlite"] 16 | 17 | ## Using an S3-Backed Database Engine 18 | 19 | `django-s3-sqlite` allows use of an [S3-synced SQLite database](https://blog.zappa.io/posts/s3sqlite-a-serverless-relational-database) as a Django database engine. 20 | 21 | This will cause problems for applications with concurrent writes**, but it scales very well for high-read applications that don't have concurrent writes (like a CMS for your blog), and it's orders of magnitude cheaper than AWS RDS or Aurora (pennies per month instead of many dollars per month). 22 | 23 | ** Concurrent writes will often be lost and not show up in concurrent readers. This is because the database is transferred between S3 storage and the Lambda instance for each request. 24 | 25 | #### Django Settings & Commands 26 | 27 | ```python 28 | DATABASES = { 29 | "default": { 30 | "ENGINE": "django_s3_sqlite", 31 | "NAME": "sqlite.db", 32 | "BUCKET": "your-db-bucket", 33 | "AWS_S3_ACCESS_KEY": "AKIA0000000000000000", # optional, to lock down your S3 bucket to an IAM user 34 | "AWS_S3_ACCESS_SECRET": "9tIZfakefakefakefakeT9Q6LD6jB5UyofakeISN", # optional, to lock down your S3 bucket to an IAM user 35 | } 36 | } 37 | ``` 38 | 39 | Newer versions of Django (v2.1+) require a newer version of SQLite (3.8.3+) than is available on AWS Lambda instances (3.7.17). [Use the pysqlite3 package](https://github.com/coleifer/pysqlite3), and add these lines to your Django settings to override the built-in `sqlite3` module: 40 | 41 | ```python 42 | __import__('pysqlite3') 43 | import sys 44 | sys.modules['sqlite3'] = sys.modules.pop('pysqlite3') 45 | ``` 46 | 47 | There was support for custom `_sqlite.so` files for different Python versions, but the above method is more flexible and doesn't require a new compilation with every new runtime. You may also need to add this line to your Zappa JSON settings file in each environment: 48 | 49 | ``` 50 | "use_precompiled_packages": false, 51 | ``` 52 | 53 | Since SQLite keeps the database in a single file, you will want to keep it as small and defragmented as possible. It is good to occasionally perform a database vacuum, especially after deleting or updating data. There's a command to vacuum your database: 54 | 55 | ```bash 56 | zappa manage [instance] s3_sqlite_vacuum 57 | ``` 58 | 59 | ## Creating a Default Admin User 60 | 61 | You'll probably need a default user to manage your application with, so you can now: 62 | 63 | $ zappa manage create_admin_user 64 | 65 | Or you can pass some arguments: 66 | 67 | $ zappa manage create_admin_user one two three 68 | 69 | This will internally make this call: 70 | 71 | ```python 72 | User.objects.create_superuser('one', 'two', 'three') 73 | ``` 74 | 75 | # Release Notes & Contributors 76 | 77 | * Thank you to our [wonderful contributors](https://github.com/FlipperPA/django-s3-sqlite/graphs/contributors)! 78 | * Release notes are [available on GitHub](https://github.com/FlipperPA/django-s3-sqlite/releases). 79 | 80 | # Maintainers and Creator 81 | 82 | * Maintainer: Tim Allen (https://github.com/FlipperPA/) 83 | * Maintainer: Peter Baumgartner (https://github.com/ipmb/) 84 | * Original Creator: Rich Jones (https://github.com/Miserlou/) 85 | 86 | This package is largely maintained by the staff of [Wharton Research Data Services](https://wrds.wharton.upenn.edu/). We are thrilled that [The Wharton School](https://www.wharton.upenn.edu/) allows us a certain amount of time to contribute to open-source projects. We add features as they are necessary for our projects, and try to keep up with Issues and Pull Requests as best we can. Due to time constraints (our full time jobs!), Feature Requests without a Pull Request may not be implemented, but we are always open to new ideas and grateful for contributions and our package users. 87 | 88 | # Build Instructions for _sqlite3.so 89 | 90 | If you'd like to use a different version of Python or SQLite than what is provided in this repo, you will need to build the static binary yourself. These instructions show you how to build the file: https://charlesleifer.com/blog/compiling-sqlite-for-use-with-python-applications/ 91 | 92 | After you've downloaded SQLite, follow the instructions to install it into a virtual environment. You must perform the installation on Amazon Linux or CentOS 7 (which Amazon Linux is based on). 93 | -------------------------------------------------------------------------------- /django_s3_sqlite/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/FlipperPA/django-s3-sqlite/a90d050d74a934231bc5219a02786dcfb5db88ca/django_s3_sqlite/__init__.py -------------------------------------------------------------------------------- /django_s3_sqlite/base.py: -------------------------------------------------------------------------------- 1 | import base64 2 | import binascii 3 | import logging 4 | from hashlib import md5 5 | from os import path 6 | 7 | import boto3 8 | import botocore 9 | from django.db.backends.sqlite3.base import DatabaseWrapper 10 | 11 | log = logging.getLogger(__name__) 12 | 13 | 14 | def _get_md5(file_bytes: bytes) -> str: 15 | """Given bytes, calculate their md5 hash""" 16 | m = md5() 17 | m.update(file_bytes) 18 | return m.hexdigest() 19 | 20 | 21 | def _get_bytes(filename: str) -> bytes: 22 | """Read file as bytes""" 23 | with open(filename, "rb") as f: 24 | return f.read() 25 | 26 | 27 | class DatabaseWrapper(DatabaseWrapper): 28 | """ 29 | Wraps the normal Django SQLite DB engine in one that shuttles the SQLite database 30 | back and forth from an S3 bucket. 31 | """ 32 | 33 | def load_remote_db(self): 34 | """ 35 | Load the database from the S3 storage bucket into the current AWS Lambda 36 | instance. 37 | """ 38 | 39 | if "/tmp/" not in self.settings_dict["NAME"]: 40 | local_file_path = "/tmp/" + self.settings_dict["NAME"] 41 | if path.isfile(local_file_path): 42 | current_md5 = _get_md5(_get_bytes(local_file_path)) 43 | else: 44 | current_md5 = "" 45 | try: 46 | # In general the ETag is the md5 of the file, in some cases it's 47 | # not, and in that case we will just need to reload the file, 48 | # I don't see any other way 49 | obj_bytes = self.s3.Object( 50 | self.settings_dict["BUCKET"], self.settings_dict["NAME"], 51 | ).get(IfNoneMatch=current_md5,)[ 52 | "Body" 53 | ] # Will throw E on 304 or 404 54 | 55 | # Remote does not match local. Replace local copy. 56 | with open(local_file_path, "wb") as f: 57 | file_bytes = obj_bytes.read() 58 | self.db_hash = _get_md5(file_bytes) 59 | f.write(file_bytes) 60 | log.debug("Database downloaded from S3.") 61 | 62 | except botocore.exceptions.ClientError as e: 63 | if e.response["Error"]["Code"] == "304": 64 | log.debug( 65 | "ETag matches md5 of local copy, using local copy of DB!", 66 | ) 67 | self.db_hash = current_md5 68 | else: 69 | log.exception("Couldn't load remote DB object.") 70 | except Exception as e: 71 | # Weird one 72 | log.exception("An unexpected error occurred.") 73 | 74 | # SQLite DatabaseWrapper will treat our tmp as normal now 75 | # Check because Django likes to call this function a lot more than it should 76 | if "/tmp/" not in self.settings_dict["NAME"]: 77 | self.settings_dict["REMOTE_NAME"] = self.settings_dict["NAME"] 78 | self.settings_dict["NAME"] = "/tmp/" + self.settings_dict["NAME"] 79 | 80 | # Make sure it exists if it doesn't yet 81 | if not path.isfile(self.settings_dict["NAME"]): 82 | open(self.settings_dict["NAME"], "a").close() 83 | 84 | if self.db_hash is None: 85 | self.db_hash = _get_md5(_get_bytes(self.settings_dict["NAME"])) 86 | log.debug("Local database is ready. md5:%s", self.db_hash) 87 | 88 | def __init__(self, *args, **kwargs): 89 | super(DatabaseWrapper, self).__init__(*args, **kwargs) 90 | signature_version = self.settings_dict.get("SIGNATURE_VERSION", "s3v4") 91 | aws_s3_access_key = self.settings_dict.get("AWS_S3_ACCESS_KEY", None) 92 | aws_s3_access_secret = self.settings_dict.get("AWS_S3_ACCESS_SECRET", None) 93 | if aws_s3_access_key and aws_s3_access_secret: 94 | session = boto3.Session( 95 | aws_access_key_id=aws_s3_access_key, 96 | aws_secret_access_key=aws_s3_access_secret, 97 | ) 98 | self.s3 = session.resource( 99 | "s3", config=botocore.client.Config(signature_version=signature_version), 100 | ) 101 | else: 102 | self.s3 = boto3.resource( 103 | "s3", config=botocore.client.Config(signature_version=signature_version), 104 | ) 105 | self.db_hash = None 106 | self.load_remote_db() 107 | 108 | def close(self, *args, **kwargs): 109 | """ 110 | Engine closed, copy file to DB if it has changed 111 | """ 112 | super(DatabaseWrapper, self).close(*args, **kwargs) 113 | 114 | file_bytes = _get_bytes(self.settings_dict["NAME"]) 115 | current_md5 = _get_md5(file_bytes) 116 | if self.db_hash == current_md5: 117 | log.debug("Database unchanged, not saving to remote DB!") 118 | return 119 | log.debug( 120 | "Current md5:%s, Expected md5:%s. Database changed, pushing to S3.", 121 | current_md5, 122 | self.db_hash, 123 | ) 124 | try: 125 | self.s3.Object( 126 | self.settings_dict["BUCKET"], self.settings_dict["REMOTE_NAME"], 127 | ).put(Body=file_bytes, ContentMD5=base64.b64encode(binascii.unhexlify(current_md5)).decode("utf-8")) 128 | self.db_hash = current_md5 129 | log.debug("Saved to remote DB!") 130 | except Exception as e: 131 | log.exception("An error occurred pushing the database to S3.") 132 | -------------------------------------------------------------------------------- /django_s3_sqlite/client.py: -------------------------------------------------------------------------------- 1 | import subprocess 2 | 3 | from django.db.backends.base.client import BaseDatabaseClient 4 | 5 | 6 | class DatabaseClient(BaseDatabaseClient): 7 | executable_name = "sqlite3" 8 | 9 | def runshell(self): 10 | args = [self.executable_name, self.connection.settings_dict["NAME"]] 11 | 12 | subprocess.check_call(args) 13 | -------------------------------------------------------------------------------- /django_s3_sqlite/features.py: -------------------------------------------------------------------------------- 1 | from django.db.backends.base.features import BaseDatabaseFeatures 2 | -------------------------------------------------------------------------------- /django_s3_sqlite/management/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/FlipperPA/django-s3-sqlite/a90d050d74a934231bc5219a02786dcfb5db88ca/django_s3_sqlite/management/__init__.py -------------------------------------------------------------------------------- /django_s3_sqlite/management/commands/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/FlipperPA/django-s3-sqlite/a90d050d74a934231bc5219a02786dcfb5db88ca/django_s3_sqlite/management/commands/__init__.py -------------------------------------------------------------------------------- /django_s3_sqlite/management/commands/create_admin_user.py: -------------------------------------------------------------------------------- 1 | from django.core.management.base import BaseCommand 2 | from django.contrib.auth import get_user_model 3 | 4 | import random 5 | import string 6 | 7 | 8 | class Command(BaseCommand): 9 | """ 10 | This command will create a default Django admin superuser. 11 | """ 12 | 13 | help = "Creates a Django admin superuser." 14 | 15 | def add_arguments(self, parser): 16 | parser.add_argument("arguments", nargs="*") 17 | 18 | def handle(self, *args, **options): 19 | # Gets the model for the current Django project's user. 20 | # This handles custom user models as well as Django's default. 21 | User = get_user_model() 22 | 23 | self.stdout.write(self.style.SUCCESS("Creating a new admin superuser...")) 24 | 25 | # If the command args are given -> try to create user with given args 26 | if options["arguments"]: 27 | try: 28 | user = User.objects.create_superuser(*options["arguments"]) 29 | self.stdout.write( 30 | self.style.SUCCESS( 31 | 'Created the admin superuser "{user}" with the given parameters.'.format( 32 | user=user 33 | ) 34 | ) 35 | ) 36 | except Exception as e: 37 | self.stdout.write( 38 | "ERROR: Django returned an error when creating the admin superuser:" 39 | ) 40 | self.stdout.write(str(e)) 41 | self.stdout.write("") 42 | self.stdout.write( 43 | "The arguments expected by the command are in this order:" 44 | ) 45 | self.stdout.write( 46 | str(User.objects.create_superuser.__code__.co_varnames[1:-1]) 47 | ) 48 | 49 | # or create default admin user 50 | else: 51 | pw = "".join( 52 | random.choice(string.ascii_uppercase + string.digits) for _ in range(10) 53 | ) 54 | User.objects.create_superuser("admin", "admin@admin.com", pw) 55 | self.stdout.write( 56 | self.style.SUCCESS( 57 | 'Created user "admin", email: "admin@admin.com", password: ' + pw 58 | ) 59 | ) 60 | self.stdout.write( 61 | self.style.SUCCESS("Log in and change this password immediately!") 62 | ) 63 | -------------------------------------------------------------------------------- /django_s3_sqlite/management/commands/s3_sqlite_vacuum.py: -------------------------------------------------------------------------------- 1 | from django.core.management.base import BaseCommand 2 | from django.db import connection, DatabaseError 3 | from django.conf import settings 4 | 5 | 6 | class Command(BaseCommand): 7 | """ 8 | This command performs a vacuum on the S3_Sqlite database. 9 | It is good to do this occasionally to keep the SQLite database stored 10 | on S3 as small and unfragmented as possible. It is recommended to be 11 | run after deleting data. 12 | """ 13 | 14 | help = "Performs a vacuum command on a S3 stored SQLite database to minimize size and fragmentation." 15 | 16 | def handle(self, *args, **options): 17 | if settings.DATABASES["default"]["ENGINE"] != "django_s3_sqlite": 18 | raise DatabaseError( 19 | "This command is only for the 'django_s3_sqlite' Django DB engine." 20 | ) 21 | else: 22 | self.stdout.write(self.style.SUCCESS("Starting database VACUUM...")) 23 | cursor = connection.cursor() 24 | cursor.execute("VACUUM;") 25 | cursor.close() 26 | self.stdout.write(self.style.SUCCESS("VACUUM complete.")) 27 | -------------------------------------------------------------------------------- /django_s3_sqlite/migrations/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/FlipperPA/django-s3-sqlite/a90d050d74a934231bc5219a02786dcfb5db88ca/django_s3_sqlite/migrations/__init__.py -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from setuptools import setup, find_packages 2 | from io import open 3 | 4 | with open("README.md") as f: 5 | README = f.read() 6 | 7 | setup( 8 | name="django-s3-sqlite", 9 | packages=find_packages(), 10 | install_requires=["Django>=2"], 11 | setup_requires=["setuptools_scm"], 12 | use_scm_version=True, 13 | include_package_data=True, 14 | license="BSD License", 15 | description="An AWS S3-hosted SQLite database backend for Django.", 16 | long_description=README, 17 | long_description_content_type="text/markdown", 18 | url="https://github.com/FlipperPA/django-s3-sqlite/", 19 | author="Timothy Allen", 20 | author_email="flipper@peregrinesalon.com", 21 | classifiers=[ 22 | "Development Status :: 4 - Beta", 23 | "Environment :: Web Environment", 24 | "Intended Audience :: Developers", 25 | "License :: OSI Approved :: BSD License", 26 | "Operating System :: OS Independent", 27 | "Programming Language :: Python", 28 | "Programming Language :: Python :: 3", 29 | "Programming Language :: Python :: 3.6", 30 | "Programming Language :: Python :: 3.8", 31 | "Programming Language :: Python :: 3 :: Only", 32 | "Framework :: Django", 33 | "Framework :: Django :: 2.2", 34 | "Framework :: Django :: 3.0", 35 | "Framework :: Django :: 3.1", 36 | "Framework :: Django :: 3.2", 37 | "Framework :: Django :: 4.0", 38 | "Topic :: Internet :: WWW/HTTP", 39 | "Topic :: Internet :: WWW/HTTP :: Dynamic Content", 40 | ], 41 | ) 42 | -------------------------------------------------------------------------------- /shared-objects/python-3-6/_sqlite3.so: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/FlipperPA/django-s3-sqlite/a90d050d74a934231bc5219a02786dcfb5db88ca/shared-objects/python-3-6/_sqlite3.so -------------------------------------------------------------------------------- /shared-objects/python-3-8/_sqlite3.so: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/FlipperPA/django-s3-sqlite/a90d050d74a934231bc5219a02786dcfb5db88ca/shared-objects/python-3-8/_sqlite3.so --------------------------------------------------------------------------------