30 |
31 |
--------------------------------------------------------------------------------
/.idea/vcs.xml:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |
6 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 |
2 | # Django + Serverless framework a match made in heaven
3 |
4 | I started using Django seriously 2 years ago, an exceptional framework that in addition to its core strength has also a vast list of add-ons and supporting libraries, one of those gems is called Django Rest Framework or DRF for short, a library which gives an easy to use/out of the box REST functionality that plugs seamlessly with Django’s ORM functionality.
5 |
6 | Up until now I’ve been using [Zappa](https://github.com/Miserlou/Zappa) which is an extremely powerful Serverless framework that is tailored made to WSGI web apps, for me the selling point was the capability to take my Django knowledge and transfer it in a flick of a switch to the serverless world. Zappa has many features that enable an easy Django (and Flask BTW) development, I’m using it in production and all is well, so how the Serverless framework (SF for short) fits into the picture ?
7 |
8 | Curiosity, I want to try it out.
9 |
10 | ## Django — a SQL beast
11 |
12 | Django is very powerful, however it’s heavily dependent on SQL database, MySql or Postgresql for example, no matter how hard I tried I couldn’t find any Django DB engine that is able to work on top of AWS DynamoDB. Currently the suggested solution is built from 2 components:
13 |
14 | 1. Using RDS, which is a managed SQL service, but not completely Serverless in that you are paying even if you are not using and it does not scale automatically.
15 |
16 | 1. Using VPC, for security reasons you do not want to put your DB on the public internet, therefore VPC is the suggested solution, when adding VPC into the mix it requires your Lambda to run inside the VPC → slow start and [complicated configuration](https://gist.github.com/efi-mk/d6586669a472be8ea16b6cf8e9c6ba7f).
17 |
18 | It’s too complicated for my demo, I wanted something quick (and dirty?)
19 | 
20 | ###### Quick and Dirty, Photo by Quino Al on Unsplash
21 | SQLite here I come. [SQLite](https://www.sqlite.org/index.html) actually is not that dirty, in the right scenarios its the the right tool, scenarios like constrained environments (mobile) or when you do not need to save a lot of data and you want to keep everything in memory. Global shared configuration might be a good idea. Have a look at the following diagram:
22 |
23 | 
24 |
25 | * You have a lambda function that requires configuration in order to function, the configuration is saved in a SQLite DB located in S3 bucket.
26 |
27 | * The Lambda pulls the SQLite on startup and does its magic.
28 |
29 | * On the other end, you have a management console that does something similar, it pulls the SQLite DB, changes it and puts it back
30 |
31 | * Pay attention that only **a single writer is allowed here**, otherwise things will get out of sync.
32 |
33 | Mmmmm, interesting, I wonder how much time is going to take us to develop it… Nothing, nada, zilch, [we already have one](https://blog.zappa.io/posts/s3sqlite-a-serverless-relational-database), the good folks of Zappa developed one for us, we shell call it Serverless SQLite or SSQL for short.
34 |
35 | ## Let’s start with the action
36 |
37 | Let’s define what the app is going to do:
38 |
39 | * A Django app with appropriate Django admin for our models.
40 |
41 | * You can log into the admin and add or change configuration.
42 |
43 | * The user is able to call a REST api created by DRF to read configuration details, something very similar to to [this](https://serverless.com/blog/flask-python-rest-api-serverless-lambda-dynamodb/).
44 |
45 | You can find the latest code [here](https://github.com/efi-mk/serverless-django-demo)
46 |
47 | You already know how to create a [Django app](https://docs.djangoproject.com/en/2.0/intro/tutorial01/) so we’ll skip the boring stuff and concentrate on the extra steps required to setup this app.
48 |
49 | ### WSGI configuration
50 |
51 | It’s something small, but that’s what’s doing the magic, in `serverless.yml` the wsgi configuration points to the `wsgi` app that Django exposes.
52 |
53 | ### SSQL configuration
54 |
55 | Under `settings.py` a configuration was added which loads the SSQL DB driver:
56 | ```
57 | DATABASES = {
58 | 'default': {
59 | 'ENGINE': 'zappa_django_utils.db.backends.s3sqlite',
60 | 'NAME': 'sqlite.db',
61 | 'BUCKET': SQLITE_BUCKET
62 | }
63 | }
64 | ```
65 | but when testing locally I do not want to connect to any S3 bucket, it slows the operation, therefore a check is made in order to verify whether we are running a Lambda environment or not if not then load the regular SQLite driver —
66 |
67 | `IS_OFFLINE = os.environ.get(‘LAMBDA_TASK_ROOT’) is None`
68 |
69 | I prefer not to run `sls wsgi serve` because Django already has wonderful management CLI support, I prefer to run `manage.py runserver`.
70 |
71 | SSQL as part of its configuration requires a bucket name, you can create it manually and set the name in `local_settings.py` , pay attention that under `serverless.yml` the lambda function has `Get` and `Put` permissions on all S3 buckets, you should use your S3 bucket ARN instead.
72 |
73 | ### WhiteNoise configuration
74 |
75 | [WhiteNoise](http://whitenoise.evans.io/en/stable/) allows our web app to serve its own static files, without relying on nginx, Amazon S3 or any other external service. We’ll use this library to serve our static admin files. I’m not going to go over all the configuration details, [you can follow them on your own](https://github.com/evansd/whitenoise/issues/164). Pay attention that the static files should be part of the Lambda package.
76 |
77 | ### A tale of a missing SO
78 |
79 | While trying to make it work, I encountered a strange error — * Unable to import module ‘app’: No module named ‘_sqlite3’. *After some digging I found out that the Lambda environment does not contain the shared library which is required by SQLite 😲. Again the good folks of Zappa provided a compiled SO which is packaged as part of the deployment script
80 |
81 | ### Deployment script
82 |
83 | So, what do we have ?
84 |
85 | * Collect all static files ✔️
86 |
87 | * Migrate our remote DB before code deployment✔️
88 |
89 | * Create a default admin `root` user with password `MyPassword` ✔️
90 |
91 | * Add _sqlite3.so to the mix ✔️
92 |
93 | * `sls deploy` ✔️
94 |
95 | You have a deploy script located under `scripts` folder
96 |
97 | ### So how do I prepare my environment locally?
98 |
99 | 1. `npm install — save-dev serverless-wsgi serverless-python-requirements`
100 |
101 | 1. Create a virtual env for your python project.
102 |
103 | 1. `pip install -r requirements.txt`
104 |
105 | 1. Run DB migration `./manage.py migrate`
106 |
107 | 1. Create super user for the management console `./manage.py createsuperuser`
108 |
109 | 1. Run the server locally `./manage.py runserver`
110 |
111 | 1. Go to [http://127.0.0.1:8000/admin](http://127.0.0.1:8000/admin) and login onto the management console. Add a configuration
112 |
113 | 1. Try `curl -H “Content-Type: application/json” -X GET [http://127.0.0.1:8000/configuration/](http://127.0.0.1:8000/configuration/)` and see if you get the configuration back.
114 |
115 | ## Fin
116 |
117 | I hope you enjoyed the journey, showing you how to use Django with SF, we used SQLite as our SQL database which was served from a S3 bucket.
118 |
119 | You are more than welcomed to ask question and [fork](https://github.com/efi-mk/serverless-django-demo) the repository.
120 |
--------------------------------------------------------------------------------
/manage.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python
2 | import os
3 | import sys
4 |
5 | if __name__ == "__main__":
6 | os.environ.setdefault("DJANGO_SETTINGS_MODULE", "serverless_django.settings")
7 | try:
8 | from django.core.management import execute_from_command_line
9 | except ImportError as exc:
10 | raise ImportError(
11 | "Couldn't import Django. Are you sure it's installed and "
12 | "available on your PYTHONPATH environment variable? Did you "
13 | "forget to activate a virtual environment?"
14 | ) from exc
15 | execute_from_command_line(sys.argv)
16 |
--------------------------------------------------------------------------------
/package.json:
--------------------------------------------------------------------------------
1 | {
2 | "name": "serverless_django",
3 | "version": "1.0.0",
4 | "description": "",
5 | "main": "index.js",
6 | "scripts": {
7 | "test": "echo \"Error: no test specified\" && exit 1"
8 | },
9 | "keywords": [],
10 | "author": "",
11 | "license": "ISC",
12 | "devDependencies": {
13 | "serverless-python-requirements": "^4.1.0",
14 | "serverless-wsgi": "^1.4.9"
15 | }
16 | }
17 |
--------------------------------------------------------------------------------
/requirements.txt:
--------------------------------------------------------------------------------
1 | boto3==1.7.58
2 | botocore==1.10.58
3 | Django==2.0.7
4 | djangorestframework==3.8.2
5 | docutils==0.14
6 | jmespath==0.9.3
7 | python-dateutil==2.7.3
8 | pytz==2018.5
9 | s3transfer==0.1.13
10 | six==1.11.0
11 | Werkzeug==0.14.1
12 | whitenoise==3.3.1
13 | zappa-django-utils==0.4.0
14 |
--------------------------------------------------------------------------------
/scripts/deploy.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 | set -e
3 |
4 | ./manage.py collectstatic --noinput
5 | # Cheating, making sure that migrate is working
6 | export LAMBDA_TASK_ROOT=/home
7 | ./manage.py migrate
8 | # Create an admin when deploying for the first time
9 | echo "from django.contrib.auth.models import User; import os; User.objects.create_superuser('root', 'my@email.com', os.environ.get('DJANGO_ADMIN_PASSWORD','MyPassword')) if len(User.objects.filter(email='my@email.com')) == 0 else print('Admin exists')"|./manage.py shell
10 | # We need the so, Lambda env does not contaon it.
11 | cp ./scripts/_sqlite3.so .
12 | sls deploy
13 | rm _sqlite3.so
14 |
--------------------------------------------------------------------------------
/serverless.yml:
--------------------------------------------------------------------------------
1 | service: serverless-django
2 |
3 | plugins:
4 | - serverless-python-requirements
5 | - serverless-wsgi
6 |
7 | custom:
8 | wsgi:
9 | app: serverless_django.wsgi.application
10 | packRequirements: false
11 | pythonRequirements:
12 | dockerizePip: non-linux
13 |
14 | provider:
15 | name: aws
16 | runtime: python3.6
17 | stage: dev
18 | region: us-east-1
19 | iamRoleStatements:
20 | - Effect: "Allow"
21 | Action:
22 | - s3:GetObject
23 | - s3:PutObject
24 | Resource: "arn:aws:s3:::*"
25 |
26 | functions:
27 | app:
28 | handler: wsgi.handler
29 | events:
30 | - http: ANY /
31 | - http: 'ANY {proxy+}'
--------------------------------------------------------------------------------
/serverless_django/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/efi-mk/serverless-django-demo/64a7ae1cd3aea141107e18d6df65955ab8c2d1a8/serverless_django/__init__.py
--------------------------------------------------------------------------------
/serverless_django/settings.py:
--------------------------------------------------------------------------------
1 | """
2 | Django settings for serverless_django project.
3 |
4 | Generated by 'django-admin startproject' using Django 2.0.7.
5 |
6 | For more information on this file, see
7 | https://docs.djangoproject.com/en/2.0/topics/settings/
8 |
9 | For the full list of settings and their values, see
10 | https://docs.djangoproject.com/en/2.0/ref/settings/
11 | """
12 | import logging
13 | import os
14 | # Build paths inside the project like this: os.path.join(BASE_DIR, ...)
15 | from logging import getLogger
16 | from sys import stdout
17 |
18 | BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
19 |
20 | # Quick-start development settings - unsuitable for production
21 | # See https://docs.djangoproject.com/en/2.0/howto/deployment/checklist/
22 |
23 | # SECURITY WARNING: keep the secret key used in production secret!
24 | SECRET_KEY = 'c7i)i3b29)el++h0)2(-d690xz!y(&*h)vv=1vy66@9^*1m4a#'
25 |
26 | # SECURITY WARNING: don't run with debug turned on in production!
27 | DEBUG = True
28 |
29 | ALLOWED_HOSTS = ['127.0.0.1', '.execute-api.us-east-1.amazonaws.com']
30 |
31 | # Application definition
32 |
33 | INSTALLED_APPS = [
34 | 'django.contrib.admin',
35 | 'django.contrib.auth',
36 | 'django.contrib.contenttypes',
37 | 'django.contrib.sessions',
38 | 'django.contrib.messages',
39 | 'django.contrib.staticfiles',
40 | 'zappa_django_utils',
41 | 'rest_framework',
42 | 'users'
43 | ]
44 |
45 | MIDDLEWARE = [
46 | 'django.middleware.security.SecurityMiddleware',
47 | 'whitenoise.middleware.WhiteNoiseMiddleware',
48 | 'django.contrib.sessions.middleware.SessionMiddleware',
49 | 'django.middleware.common.CommonMiddleware',
50 | 'django.middleware.csrf.CsrfViewMiddleware',
51 | 'django.contrib.auth.middleware.AuthenticationMiddleware',
52 | 'django.contrib.messages.middleware.MessageMiddleware',
53 | 'django.middleware.clickjacking.XFrameOptionsMiddleware',
54 | ]
55 |
56 | ROOT_URLCONF = 'serverless_django.urls'
57 |
58 | TEMPLATES = [
59 | {
60 | 'BACKEND': 'django.template.backends.django.DjangoTemplates',
61 | 'DIRS': [os.path.join(BASE_DIR, 'templates')]
62 | ,
63 | 'APP_DIRS': True,
64 | 'OPTIONS': {
65 | 'context_processors': [
66 | 'django.template.context_processors.debug',
67 | 'django.template.context_processors.request',
68 | 'django.contrib.auth.context_processors.auth',
69 | 'django.contrib.messages.context_processors.messages',
70 | ],
71 | },
72 | },
73 | ]
74 |
75 | WSGI_APPLICATION = 'serverless_django.wsgi.application'
76 |
77 | # Password validation
78 | # https://docs.djangoproject.com/en/2.0/ref/settings/#auth-password-validators
79 |
80 | AUTH_PASSWORD_VALIDATORS = [
81 | {
82 | 'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
83 | },
84 | {
85 | 'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
86 | },
87 | {
88 | 'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
89 | },
90 | {
91 | 'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
92 | },
93 | ]
94 |
95 | LOGGING = {
96 | 'version': 1,
97 | 'disable_existing_loggers': False,
98 | 'handlers': {
99 | 'console': {
100 | 'class': 'logging.StreamHandler',
101 | },
102 | },
103 | 'loggers': {
104 | '': {
105 | 'handlers': ['console'],
106 | 'level': 'INFO',
107 | },
108 | },
109 | }
110 |
111 | # Internationalization
112 | # https://docs.djangoproject.com/en/2.0/topics/i18n/
113 |
114 | LANGUAGE_CODE = 'en-us'
115 |
116 | TIME_ZONE = 'UTC'
117 |
118 | USE_I18N = True
119 |
120 | USE_L10N = True
121 |
122 | USE_TZ = True
123 |
124 | REST_FRAMEWORK = {
125 | # Use Django's standard `django.contrib.auth` permissions,
126 | # or allow read-only access for unauthenticated users.
127 | 'DEFAULT_PERMISSION_CLASSES': [
128 | 'rest_framework.permissions.AllowAny'
129 | ]
130 | }
131 |
132 | # Static files (CSS, JavaScript, Images)
133 | # https://docs.djangoproject.com/en/2.0/howto/static-files/
134 |
135 | STATIC_URL = '/dev/static/'
136 | STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
137 | WHITENOISE_STATIC_PREFIX = '/static/'
138 | STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'
139 |
140 | SQLITE_BUCKET = os.environ.get('SQLITE_BUCKET', "serverless-django")
141 | try:
142 | from .local_settings import *
143 | except ImportError:
144 | logging.error("Unable to find local_settings.py")
145 |
146 | # Are we running in Lambda environment ?
147 | # See https://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html#lambda-environment-variables
148 | IS_OFFLINE = os.environ.get('LAMBDA_TASK_ROOT') is None
149 |
150 | # I hate different configuration for local and cloud, but this is what we have now.
151 | if IS_OFFLINE:
152 | DATABASES = {
153 | 'default': {
154 | 'ENGINE': 'django.db.backends.sqlite3',
155 | 'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
156 | }
157 | }
158 | else:
159 | DATABASES = {
160 | 'default': {
161 | 'ENGINE': 'zappa_django_utils.db.backends.s3sqlite',
162 | 'NAME': 'sqlite.db',
163 | 'BUCKET': SQLITE_BUCKET
164 | }
165 | }
166 |
--------------------------------------------------------------------------------
/serverless_django/urls.py:
--------------------------------------------------------------------------------
1 | from django.conf.urls import url
2 | from django.contrib import admin
3 | from django.urls import path, include
4 | from rest_framework import routers
5 |
6 | from users.views import UserViewSet
7 |
8 | router = routers.DefaultRouter()
9 | router.register(r'configuration', UserViewSet)
10 |
11 | urlpatterns = [
12 | url(r'^', include(router.urls)),
13 | path('admin/', admin.site.urls),
14 | ]
15 |
--------------------------------------------------------------------------------
/serverless_django/wsgi.py:
--------------------------------------------------------------------------------
1 | """
2 | WSGI config for serverless_django project.
3 |
4 | It exposes the WSGI callable as a module-level variable named ``application``.
5 |
6 | For more information on this file, see
7 | https://docs.djangoproject.com/en/2.0/howto/deployment/wsgi/
8 | """
9 |
10 | import os
11 |
12 | from django.core.wsgi import get_wsgi_application
13 |
14 | os.environ.setdefault("DJANGO_SETTINGS_MODULE", "serverless_django.settings")
15 |
16 | application = get_wsgi_application()
17 |
--------------------------------------------------------------------------------
/users/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/efi-mk/serverless-django-demo/64a7ae1cd3aea141107e18d6df65955ab8c2d1a8/users/__init__.py
--------------------------------------------------------------------------------
/users/admin.py:
--------------------------------------------------------------------------------
1 | from django.contrib import admin
2 |
3 | from users.models import Configuration
4 |
5 |
6 | class BaseAdmin(admin.ModelAdmin):
7 | empty_value_display = "-empty-"
8 | date_hierarchy = "updated"
9 |
10 |
11 | @admin.register(Configuration)
12 | class UserAdmin(BaseAdmin):
13 | list_display = ["key", "value"]
14 | search_fields = ["key"]
15 |
--------------------------------------------------------------------------------
/users/apps.py:
--------------------------------------------------------------------------------
1 | from django.apps import AppConfig
2 |
3 |
4 | class UsersConfig(AppConfig):
5 | name = 'users'
6 |
--------------------------------------------------------------------------------
/users/migrations/0001_initial.py:
--------------------------------------------------------------------------------
1 | # Generated by Django 2.0.7 on 2018-07-18 10:28
2 |
3 | from django.db import migrations, models
4 |
5 |
6 | class Migration(migrations.Migration):
7 |
8 | initial = True
9 |
10 | dependencies = [
11 | ]
12 |
13 | operations = [
14 | migrations.CreateModel(
15 | name='Configuration',
16 | fields=[
17 | ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
18 | ('added', models.DateTimeField(auto_now_add=True)),
19 | ('updated', models.DateTimeField(auto_now=True)),
20 | ('key', models.TextField()),
21 | ('value', models.TextField()),
22 | ],
23 | options={
24 | 'abstract': False,
25 | },
26 | ),
27 | ]
28 |
--------------------------------------------------------------------------------
/users/migrations/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/efi-mk/serverless-django-demo/64a7ae1cd3aea141107e18d6df65955ab8c2d1a8/users/migrations/__init__.py
--------------------------------------------------------------------------------
/users/models.py:
--------------------------------------------------------------------------------
1 | from django.db import models
2 |
3 |
4 | class BaseModel(models.Model):
5 | """
6 | All models should inherit from this model which includes basic fields.
7 | """
8 |
9 | added = models.DateTimeField(auto_now_add=True)
10 | updated = models.DateTimeField(auto_now=True)
11 |
12 | class Meta:
13 | abstract = True
14 |
15 |
16 | class Configuration(BaseModel):
17 | key = models.TextField()
18 | value = models.TextField()
19 |
20 | def __str__(self):
21 | return f"{self.key} = {self.value}"
22 |
--------------------------------------------------------------------------------
/users/serializers.py:
--------------------------------------------------------------------------------
1 | from rest_framework import serializers
2 |
3 | from users.models import Configuration
4 |
5 |
6 | class ConfigurationSerializer(serializers.HyperlinkedModelSerializer):
7 | class Meta:
8 | model = Configuration
9 | fields = ('key', 'value')
10 |
--------------------------------------------------------------------------------
/users/tests.py:
--------------------------------------------------------------------------------
1 | from django.test import TestCase
2 |
3 | # Create your tests here.
4 |
--------------------------------------------------------------------------------
/users/views.py:
--------------------------------------------------------------------------------
1 | from rest_framework import viewsets
2 |
3 | from users.models import Configuration
4 | from users.serializers import ConfigurationSerializer
5 |
6 |
7 | class UserViewSet(viewsets.ReadOnlyModelViewSet):
8 | queryset = Configuration.objects.all()
9 | serializer_class = ConfigurationSerializer
10 |
--------------------------------------------------------------------------------