├── .gitignore ├── api.http ├── assets ├── api_container_memory_peak.png ├── api_response_times.png └── locust_api_response_times.png ├── django_drf ├── Dockerfile ├── apis │ ├── __init__.py │ ├── car_listing_api.py │ ├── car_listing_api_2.py │ ├── car_listing_api_3.py │ ├── car_listing_api_4_paginated.py │ └── car_listing_api_drf_serializer_with_orjson.py ├── car_registry │ ├── __init__.py │ ├── admin.py │ ├── apps.py │ ├── management │ │ ├── __init__.py │ │ └── commands │ │ │ ├── __init__.py │ │ │ └── populate.py │ ├── migrations │ │ ├── 0001_initial.py │ │ └── __init__.py │ ├── models.py │ ├── serializers.py │ ├── tests.py │ └── views.py ├── config │ ├── __init__.py │ ├── asgi.py │ ├── settings.py │ ├── urls.py │ └── wsgi.py ├── custom_response.py ├── manage.py ├── requirements.txt ├── services │ ├── __init__.py │ └── car_services.py └── wait_for_db.sh ├── django_ninja ├── Dockerfile ├── apis │ ├── __init__.py │ └── car_listing_api.py ├── car_registry │ ├── __init__.py │ ├── admin.py │ ├── apps.py │ ├── management │ │ ├── __init__.py │ │ └── commands │ │ │ ├── __init__.py │ │ │ └── populate.py │ ├── migrations │ │ ├── 0001_initial.py │ │ └── __init__.py │ ├── models.py │ └── tests.py ├── config │ ├── __init__.py │ ├── asgi.py │ ├── settings.py │ ├── urls.py │ └── wsgi.py ├── custom_renderer.py ├── manage.py ├── requirements.txt ├── services │ ├── __init__.py │ └── car_services.py └── wait_for_db.sh ├── docker-compose.yml ├── docs └── readme.md ├── fast_api ├── Dockerfile ├── apis │ ├── __init__.py │ └── cars.py ├── database.py ├── database_test.py ├── main.py ├── models │ ├── __init__.py │ └── cars.py ├── requirements.txt ├── schemas.py └── services │ ├── __init__.py │ └── cars.py ├── go_sqlc_mux ├── Dockerfile ├── bin │ └── server ├── cmd │ └── server │ │ └── main.go ├── go.mod ├── go.sum ├── internal │ ├── db │ │ └── db.go │ ├── repository │ │ ├── car_registry.sql.go │ │ ├── db.go │ │ └── models.go │ ├── service │ │ └── car_registry.go │ └── transport │ │ └── http │ │ ├── car_registry.go │ │ └── handler.go ├── schemas │ ├── 0001_initial.down.sql │ └── 0001_initial.up.sql ├── sqlc.yaml └── sqlc │ └── queries │ └── car_registry.sql ├── load_testing ├── Dockerfile ├── __init__.py ├── api_response_times.conf └── locustfile.py ├── makefile └── readme.md /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | build/ 12 | develop-eggs/ 13 | dist/ 14 | downloads/ 15 | eggs/ 16 | .eggs/ 17 | lib/ 18 | lib64/ 19 | parts/ 20 | sdist/ 21 | var/ 22 | wheels/ 23 | share/python-wheels/ 24 | *.egg-info/ 25 | .installed.cfg 26 | *.egg 27 | MANIFEST 28 | 29 | # PyInstaller 30 | # Usually these files are written by a python script from a template 31 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 32 | *.manifest 33 | *.spec 34 | 35 | # Installer logs 36 | pip-log.txt 37 | pip-delete-this-directory.txt 38 | 39 | # Unit test / coverage reports 40 | htmlcov/ 41 | .tox/ 42 | .nox/ 43 | .coverage 44 | .coverage.* 45 | .cache 46 | nosetests.xml 47 | coverage.xml 48 | *.cover 49 | *.py,cover 50 | .hypothesis/ 51 | .pytest_cache/ 52 | cover/ 53 | 54 | # Translations 55 | *.mo 56 | *.pot 57 | 58 | # Django stuff: 59 | *.log 60 | local_settings.py 61 | db.sqlite3 62 | db.sqlite3-journal 63 | 64 | # Flask stuff: 65 | instance/ 66 | .webassets-cache 67 | 68 | # Scrapy stuff: 69 | .scrapy 70 | 71 | # Sphinx documentation 72 | docs/_build/ 73 | 74 | # PyBuilder 75 | .pybuilder/ 76 | target/ 77 | 78 | # Jupyter Notebook 79 | .ipynb_checkpoints 80 | 81 | # IPython 82 | profile_default/ 83 | ipython_config.py 84 | 85 | # pyenv 86 | # For a library or package, you might want to ignore these files since the code is 87 | # intended to run in multiple environments; otherwise, check them in: 88 | # .python-version 89 | 90 | # pipenv 91 | # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. 92 | # However, in case of collaboration, if having platform-specific dependencies or dependencies 93 | # having no cross-platform support, pipenv may install dependencies that don't work, or not 94 | # install all needed dependencies. 95 | #Pipfile.lock 96 | 97 | # poetry 98 | # Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control. 99 | # This is especially recommended for binary packages to ensure reproducibility, and is more 100 | # commonly ignored for libraries. 101 | # https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control 102 | #poetry.lock 103 | 104 | # pdm 105 | # Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control. 106 | #pdm.lock 107 | # pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it 108 | # in version control. 109 | # https://pdm.fming.dev/latest/usage/project/#working-with-version-control 110 | .pdm.toml 111 | .pdm-python 112 | .pdm-build/ 113 | 114 | # PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm 115 | __pypackages__/ 116 | 117 | # Celery stuff 118 | celerybeat-schedule 119 | celerybeat.pid 120 | 121 | # SageMath parsed files 122 | *.sage.py 123 | 124 | # Environments 125 | .env 126 | .venv 127 | env/ 128 | venv/ 129 | ENV/ 130 | env.bak/ 131 | venv.bak/ 132 | 133 | # Spyder project settings 134 | .spyderproject 135 | .spyproject 136 | 137 | # Rope project settings 138 | .ropeproject 139 | 140 | # mkdocs documentation 141 | /site 142 | 143 | # mypy 144 | .mypy_cache/ 145 | .dmypy.json 146 | dmypy.json 147 | 148 | # Pyre type checker 149 | .pyre/ 150 | 151 | # pytype static type analyzer 152 | .pytype/ 153 | 154 | # Cython debug symbols 155 | cython_debug/ 156 | 157 | # PyCharm 158 | # JetBrains specific template is maintained in a separate JetBrains.gitignore that can 159 | # be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore 160 | # and can be added to the global gitignore or merged into this file. For a more nuclear 161 | # option (not recommended) you can uncomment the following to ignore the entire idea folder. 162 | #.idea/ 163 | 164 | environment 165 | 166 | .idea 167 | .DS_Store -------------------------------------------------------------------------------- /api.http: -------------------------------------------------------------------------------- 1 | ### Django Rest Framework Chapter 1: Basic setup 2 | GET http://localhost:8000/cars/ 3 | 4 | 5 | ### Django Rest Framework Chapter 2: Retrieve cars with model info 6 | GET http://localhost:8000/retrieve-cars-with-model/ 7 | 8 | 9 | ### Django Rest Framework Chapter 3: Adding prefetching to the query 10 | GET http://localhost:8000/retrieve-cars-with-prefetch-related/ 11 | 12 | 13 | ### Django Rest Framework Chapter 5: Replacing ModelSerializer with Serializer 14 | GET http://localhost:8000/api/cars/ 15 | 16 | 17 | ### Django Rest Framework Chapter 6: Skipping serializers 18 | GET http://localhost:8000/api/cars-2/ 19 | 20 | 21 | ### Django Rest Framework Chapter 7: Using Orjson 22 | GET http://localhost:8000/api/cars-3/ 23 | 24 | 25 | ### Django Rest Framework Chapter 9: Paginated 26 | GET http://localhost:8000/api/cars-4-paginated/?page=4 27 | 28 | 29 | ### Django Ninja Chapter 1: Using Schema 30 | GET http://localhost:8001/ninja/with-schema/ 31 | 32 | 33 | ### Django Ninja Chapter 2/3: Skipping schema, implement Orjson 34 | GET http://localhost:8001/ninja/without-schema/ 35 | 36 | 37 | ### Django Rest Framework Serializer with Orjson 38 | GET http://localhost:8000/api/cars-serializer-orjson/ 39 | 40 | 41 | ### FastAPI 42 | GET http://localhost:8002/fastapi/ 43 | 44 | 45 | ### Go 46 | GET http://localhost:8003/go/ -------------------------------------------------------------------------------- /assets/api_container_memory_peak.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/oscarychen/building-efficient-api/552a263c8eb4bfc5a5c44e21758fae95352c0e70/assets/api_container_memory_peak.png -------------------------------------------------------------------------------- /assets/api_response_times.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/oscarychen/building-efficient-api/552a263c8eb4bfc5a5c44e21758fae95352c0e70/assets/api_response_times.png -------------------------------------------------------------------------------- /assets/locust_api_response_times.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/oscarychen/building-efficient-api/552a263c8eb4bfc5a5c44e21758fae95352c0e70/assets/locust_api_response_times.png -------------------------------------------------------------------------------- /django_drf/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM python:3.12.8-slim 2 | 3 | ENV PYTHONDONTWRITEBYTECODE 1 4 | ENV PYTHONUNBUFFERED 1 5 | 6 | WORKDIR /app 7 | 8 | # Install system dependencies 9 | RUN apt-get update && apt-get install -y \ 10 | postgresql-client \ 11 | && rm -rf /var/lib/apt/lists/* 12 | 13 | 14 | COPY ./django_drf/requirements.txt /app/requirements.txt 15 | 16 | RUN pip install --upgrade pip 17 | RUN pip install --no-cache-dir -r requirements.txt 18 | 19 | COPY ./django_drf/ . 20 | 21 | CMD ["daphne", "--b", "0.0.0.0", "-p", "8000", "config.asgi:application"] 22 | -------------------------------------------------------------------------------- /django_drf/apis/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/oscarychen/building-efficient-api/552a263c8eb4bfc5a5c44e21758fae95352c0e70/django_drf/apis/__init__.py -------------------------------------------------------------------------------- /django_drf/apis/car_listing_api.py: -------------------------------------------------------------------------------- 1 | from rest_framework.views import APIView 2 | from rest_framework.response import Response 3 | from rest_framework.request import Request 4 | from rest_framework import status 5 | from rest_framework import serializers 6 | 7 | from services.car_services import CarService 8 | 9 | 10 | class CarListingResponseSerializer(serializers.Serializer): 11 | id = serializers.IntegerField() 12 | vin = serializers.CharField() 13 | owner = serializers.CharField() 14 | created_at = serializers.DateTimeField() 15 | updated_at = serializers.DateTimeField() 16 | car_model_id = serializers.IntegerField(source="model_id") 17 | car_model_name = serializers.CharField(source='model.name') 18 | car_model_year = serializers.IntegerField(source='model.year') 19 | color = serializers.CharField(source='model.color') 20 | 21 | 22 | class CarListingAPI(APIView): 23 | def get(self, request: Request, *args, **kwargs) -> Response: 24 | car_instances = CarService().retrieve_all_cars() 25 | response_serializer = CarListingResponseSerializer(car_instances, many=True) 26 | return Response(response_serializer.data, status=status.HTTP_200_OK) -------------------------------------------------------------------------------- /django_drf/apis/car_listing_api_2.py: -------------------------------------------------------------------------------- 1 | from rest_framework import status 2 | from rest_framework.request import Request 3 | from rest_framework.response import Response 4 | from rest_framework.views import APIView 5 | 6 | from services.car_services import CarService 7 | 8 | 9 | class CarListingAPI(APIView): 10 | def get(self, request: Request, *args, **kwargs) -> Response: 11 | car_dicts = CarService().retrieve_all_cars_as_dicts() 12 | return Response(car_dicts, status=status.HTTP_200_OK) -------------------------------------------------------------------------------- /django_drf/apis/car_listing_api_3.py: -------------------------------------------------------------------------------- 1 | from rest_framework import status 2 | from rest_framework.request import Request 3 | from rest_framework.views import APIView 4 | 5 | from custom_response import OrJsonResponse 6 | from services.car_services import CarService 7 | 8 | 9 | 10 | class CarListingAPI(APIView): 11 | def get(self, request: Request, *args, **kwargs) -> OrJsonResponse: 12 | car_dicts = CarService().retrieve_all_cars_as_dicts() 13 | return OrJsonResponse(car_dicts, status=status.HTTP_200_OK) 14 | -------------------------------------------------------------------------------- /django_drf/apis/car_listing_api_4_paginated.py: -------------------------------------------------------------------------------- 1 | from rest_framework import status 2 | from rest_framework.pagination import PageNumberPagination 3 | from rest_framework.request import Request 4 | from rest_framework.views import APIView 5 | 6 | from custom_response import OrJsonResponse 7 | from services.car_services import CarService 8 | 9 | 10 | class CarListingAPI(APIView): 11 | def get(self, request: Request, *args, **kwargs) -> OrJsonResponse: 12 | car_dicts = CarService().retrieve_all_cars_as_dicts() 13 | paginator = PageNumberPagination() 14 | paginated_car_dicts = paginator.paginate_queryset(car_dicts, request) 15 | return OrJsonResponse(paginated_car_dicts, status=status.HTTP_200_OK) 16 | -------------------------------------------------------------------------------- /django_drf/apis/car_listing_api_drf_serializer_with_orjson.py: -------------------------------------------------------------------------------- 1 | from rest_framework.views import APIView 2 | from rest_framework.response import Response 3 | from rest_framework.request import Request 4 | from rest_framework import status 5 | from rest_framework import serializers 6 | 7 | from custom_response import OrJsonResponse 8 | from services.car_services import CarService 9 | 10 | 11 | class CarListingResponseSerializer(serializers.Serializer): 12 | id = serializers.IntegerField() 13 | vin = serializers.CharField() 14 | owner = serializers.CharField() 15 | created_at = serializers.DateTimeField() 16 | updated_at = serializers.DateTimeField() 17 | car_model_id = serializers.IntegerField(source="model_id") 18 | car_model_name = serializers.CharField(source='model.name') 19 | car_model_year = serializers.IntegerField(source='model.year') 20 | color = serializers.CharField(source='model.color') 21 | 22 | 23 | class CarListingAPI(APIView): 24 | def get(self, request: Request, *args, **kwargs) -> Response: 25 | car_instances = CarService().retrieve_all_cars() 26 | response_serializer = CarListingResponseSerializer(car_instances, many=True) 27 | return OrJsonResponse(response_serializer.data, status=status.HTTP_200_OK) -------------------------------------------------------------------------------- /django_drf/car_registry/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/oscarychen/building-efficient-api/552a263c8eb4bfc5a5c44e21758fae95352c0e70/django_drf/car_registry/__init__.py -------------------------------------------------------------------------------- /django_drf/car_registry/admin.py: -------------------------------------------------------------------------------- 1 | from django.contrib import admin 2 | 3 | # Register your models here. 4 | -------------------------------------------------------------------------------- /django_drf/car_registry/apps.py: -------------------------------------------------------------------------------- 1 | from django.apps import AppConfig 2 | 3 | 4 | class CarRegistryConfig(AppConfig): 5 | default_auto_field = 'django.db.models.BigAutoField' 6 | name = 'car_registry' 7 | -------------------------------------------------------------------------------- /django_drf/car_registry/management/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/oscarychen/building-efficient-api/552a263c8eb4bfc5a5c44e21758fae95352c0e70/django_drf/car_registry/management/__init__.py -------------------------------------------------------------------------------- /django_drf/car_registry/management/commands/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/oscarychen/building-efficient-api/552a263c8eb4bfc5a5c44e21758fae95352c0e70/django_drf/car_registry/management/commands/__init__.py -------------------------------------------------------------------------------- /django_drf/car_registry/management/commands/populate.py: -------------------------------------------------------------------------------- 1 | from typing import Any 2 | from django.core.management.base import BaseCommand 3 | import random 4 | from car_registry.models import Car, CarModel 5 | from faker import Faker 6 | 7 | 8 | class Command(BaseCommand): 9 | def handle(self, *args: Any, **options: Any) -> str | None: 10 | 11 | self.bulk_create_car_models(200) 12 | 13 | for i in range(50): 14 | print(f'Creating cars batch {i+1}...') 15 | self.bulk_create_cars(2000) 16 | 17 | 18 | def bulk_create_car_models(self, quantity=100): 19 | colors = ['Red', 'Blue', 'Green', 'Yellow', 'Black', 'White', 'Silver', 'Gold', 'Purple', 'Orange'] 20 | years = [2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024] 21 | prices = [10000, 20000, 30000, 40000, 50000, 60000, 70000, 80000, 90000, 100000] 22 | 23 | car_models = [] 24 | 25 | for i in range(quantity): 26 | car_models.append( 27 | CarModel( 28 | name=f'Car Model {i}', 29 | make=f'Car Make {i}', 30 | year=random.choice(years), 31 | color=random.choice(colors), 32 | price=random.choice(prices) 33 | ) 34 | ) 35 | 36 | CarModel.objects.bulk_create(car_models) 37 | 38 | def bulk_create_cars(self, quantity=1000): 39 | car_models = CarModel.objects.all().values_list('id', flat=True) 40 | 41 | cars = [] 42 | 43 | for i in range(quantity): 44 | cars.append( 45 | Car( 46 | vin=f'VIN-{i}', 47 | model_id=random.choice(car_models), 48 | owner=Faker().name() 49 | ) 50 | ) 51 | 52 | Car.objects.bulk_create(cars) 53 | -------------------------------------------------------------------------------- /django_drf/car_registry/migrations/0001_initial.py: -------------------------------------------------------------------------------- 1 | # Generated by Django 5.1.2 on 2024-11-05 03:50 2 | 3 | import django.db.models.deletion 4 | from django.db import migrations, models 5 | 6 | 7 | class Migration(migrations.Migration): 8 | 9 | initial = True 10 | 11 | dependencies = [ 12 | ] 13 | 14 | operations = [ 15 | migrations.CreateModel( 16 | name='CarModel', 17 | fields=[ 18 | ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), 19 | ('name', models.CharField(max_length=100)), 20 | ('make', models.CharField(max_length=100)), 21 | ('year', models.IntegerField()), 22 | ('color', models.CharField(max_length=100)), 23 | ('price', models.DecimalField(decimal_places=2, max_digits=10)), 24 | ('created_at', models.DateTimeField(auto_now_add=True)), 25 | ('updated_at', models.DateTimeField(auto_now=True)), 26 | ], 27 | ), 28 | migrations.CreateModel( 29 | name='Car', 30 | fields=[ 31 | ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), 32 | ('vin', models.CharField(max_length=17)), 33 | ('owner', models.CharField(max_length=100)), 34 | ('created_at', models.DateTimeField(auto_now_add=True)), 35 | ('updated_at', models.DateTimeField(auto_now=True)), 36 | ('model', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='car_registry.carmodel')), 37 | ], 38 | ), 39 | ] 40 | -------------------------------------------------------------------------------- /django_drf/car_registry/migrations/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/oscarychen/building-efficient-api/552a263c8eb4bfc5a5c44e21758fae95352c0e70/django_drf/car_registry/migrations/__init__.py -------------------------------------------------------------------------------- /django_drf/car_registry/models.py: -------------------------------------------------------------------------------- 1 | from django.db import models 2 | 3 | 4 | class CarModel(models.Model): 5 | name = models.CharField(max_length=100) 6 | make = models.CharField(max_length=100) 7 | year = models.IntegerField() 8 | color = models.CharField(max_length=100) 9 | price = models.DecimalField(max_digits=10, decimal_places=2) 10 | created_at = models.DateTimeField(auto_now_add=True) 11 | updated_at = models.DateTimeField(auto_now=True) 12 | 13 | 14 | class Car(models.Model): 15 | vin = models.CharField(max_length=17) 16 | model = models.ForeignKey(CarModel, on_delete=models.CASCADE) 17 | owner = models.CharField(max_length=100) 18 | created_at = models.DateTimeField(auto_now_add=True) 19 | updated_at = models.DateTimeField(auto_now=True) 20 | 21 | -------------------------------------------------------------------------------- /django_drf/car_registry/serializers.py: -------------------------------------------------------------------------------- 1 | from rest_framework import serializers 2 | from car_registry.models import Car 3 | 4 | 5 | class CarSerializer(serializers.ModelSerializer): 6 | class Meta: 7 | model = Car 8 | fields = '__all__' 9 | 10 | 11 | class CarSerializerWithRelatedModel(serializers.ModelSerializer): 12 | car_model_id = serializers.IntegerField(source='model_id', read_only=True) 13 | car_model_name = serializers.CharField(source='model.name', read_only=True) 14 | car_model_year = serializers.IntegerField(source='model.year', read_only=True) 15 | color = serializers.CharField(source='model.color', read_only=True) 16 | class Meta: 17 | model = Car 18 | fields = [ 19 | "id", 20 | "vin", 21 | "owner", 22 | "created_at", 23 | "updated_at", 24 | "car_model_id", 25 | "car_model_name", 26 | "car_model_year", 27 | "color", 28 | ] -------------------------------------------------------------------------------- /django_drf/car_registry/tests.py: -------------------------------------------------------------------------------- 1 | from django.test import TestCase 2 | 3 | # Create your tests here. 4 | -------------------------------------------------------------------------------- /django_drf/car_registry/views.py: -------------------------------------------------------------------------------- 1 | from rest_framework.generics import ListAPIView 2 | from car_registry.serializers import CarSerializer, CarSerializerWithRelatedModel 3 | from car_registry.models import Car 4 | 5 | 6 | class CarListView(ListAPIView): 7 | serializer_class = CarSerializer 8 | queryset = Car.objects.all() 9 | pagination_class = None 10 | 11 | 12 | class CarListViewWithModel(ListAPIView): 13 | serializer_class = CarSerializerWithRelatedModel 14 | queryset = Car.objects.all() 15 | pagination_class = None 16 | 17 | 18 | class CarListViewWithModelPrefetched(ListAPIView): 19 | serializer_class = CarSerializerWithRelatedModel 20 | queryset = Car.objects.all().select_related('model') 21 | pagination_class = None 22 | -------------------------------------------------------------------------------- /django_drf/config/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/oscarychen/building-efficient-api/552a263c8eb4bfc5a5c44e21758fae95352c0e70/django_drf/config/__init__.py -------------------------------------------------------------------------------- /django_drf/config/asgi.py: -------------------------------------------------------------------------------- 1 | """ 2 | ASGI config for demo project. 3 | 4 | It exposes the ASGI callable as a module-level variable named ``application``. 5 | 6 | For more information on this file, see 7 | https://docs.djangoproject.com/en/5.1/howto/deployment/asgi/ 8 | """ 9 | 10 | import os 11 | 12 | from django.core.asgi import get_asgi_application 13 | 14 | os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'config.settings') 15 | 16 | application = get_asgi_application() 17 | -------------------------------------------------------------------------------- /django_drf/config/settings.py: -------------------------------------------------------------------------------- 1 | """ 2 | Django settings for demo project. 3 | 4 | Generated by 'django-admin startproject' using Django 5.1.2. 5 | 6 | For more information on this file, see 7 | https://docs.djangoproject.com/en/5.1/topics/settings/ 8 | 9 | For the full list of settings and their values, see 10 | https://docs.djangoproject.com/en/5.1/ref/settings/ 11 | """ 12 | 13 | from pathlib import Path 14 | 15 | # Build paths inside the project like this: BASE_DIR / 'subdir'. 16 | BASE_DIR = Path(__file__).resolve().parent.parent 17 | 18 | 19 | # Quick-start development settings - unsuitable for production 20 | # See https://docs.djangoproject.com/en/5.1/howto/deployment/checklist/ 21 | 22 | # SECURITY WARNING: keep the secret key used in production secret! 23 | SECRET_KEY = 'django-insecure-eo-w8#)7r4n%p86a5tb(yfumoett_28^n6r&++1u5xzbu19fzf' 24 | 25 | # SECURITY WARNING: don't run with debug turned on in production! 26 | DEBUG = True 27 | 28 | ALLOWED_HOSTS = ["localhost", "django-drf"] 29 | 30 | 31 | # Application definition 32 | 33 | INSTALLED_APPS = [ 34 | 'daphne', 35 | 'django.contrib.admin', 36 | 'django.contrib.auth', 37 | 'django.contrib.contenttypes', 38 | 'django.contrib.sessions', 39 | 'django.contrib.messages', 40 | 'django.contrib.staticfiles', 41 | 'car_registry', 42 | ] 43 | 44 | MIDDLEWARE = [ 45 | 'django.middleware.security.SecurityMiddleware', 46 | 'django.contrib.sessions.middleware.SessionMiddleware', 47 | 'django.middleware.common.CommonMiddleware', 48 | 'django.middleware.csrf.CsrfViewMiddleware', 49 | 'django.contrib.auth.middleware.AuthenticationMiddleware', 50 | 'django.contrib.messages.middleware.MessageMiddleware', 51 | 'django.middleware.clickjacking.XFrameOptionsMiddleware', 52 | ] 53 | 54 | ROOT_URLCONF = 'config.urls' 55 | 56 | TEMPLATES = [ 57 | { 58 | 'BACKEND': 'django.template.backends.django.DjangoTemplates', 59 | 'DIRS': [], 60 | 'APP_DIRS': True, 61 | 'OPTIONS': { 62 | 'context_processors': [ 63 | 'django.template.context_processors.debug', 64 | 'django.template.context_processors.request', 65 | 'django.contrib.auth.context_processors.auth', 66 | 'django.contrib.messages.context_processors.messages', 67 | ], 68 | }, 69 | }, 70 | ] 71 | 72 | # WSGI_APPLICATION = 'config.wsgi.application' 73 | ASGI_APPLICATION = 'config.asgi.application' 74 | 75 | # Database 76 | # https://docs.djangoproject.com/en/5.1/ref/settings/#databases 77 | 78 | DATABASES = { 79 | "default": { 80 | "ENGINE": "django.db.backends.postgresql", 81 | "NAME": "api_demo_db", 82 | "USER": "postgres", 83 | "PASSWORD": "postgres", 84 | # "HOST": "127.0.0.1", 85 | "HOST": "db", 86 | "PORT": "5432", 87 | } 88 | } 89 | 90 | 91 | # Password validation 92 | # https://docs.djangoproject.com/en/5.1/ref/settings/#auth-password-validators 93 | 94 | AUTH_PASSWORD_VALIDATORS = [ 95 | { 96 | 'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator', 97 | }, 98 | { 99 | 'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator', 100 | }, 101 | { 102 | 'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator', 103 | }, 104 | { 105 | 'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator', 106 | }, 107 | ] 108 | 109 | 110 | # Internationalization 111 | # https://docs.djangoproject.com/en/5.1/topics/i18n/ 112 | 113 | LANGUAGE_CODE = 'en-us' 114 | 115 | TIME_ZONE = 'UTC' 116 | 117 | USE_I18N = True 118 | 119 | USE_TZ = True 120 | 121 | 122 | # Static files (CSS, JavaScript, Images) 123 | # https://docs.djangoproject.com/en/5.1/howto/static-files/ 124 | 125 | STATIC_URL = 'static/' 126 | 127 | # Default primary key field type 128 | # https://docs.djangoproject.com/en/5.1/ref/settings/#default-auto-field 129 | 130 | DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField' 131 | 132 | 133 | REST_FRAMEWORK = { 134 | 'DEFAULT_PAGINATION_CLASS': 'rest_framework.pagination.PageNumberPagination', 135 | 'PAGE_SIZE': 1000, 136 | } 137 | -------------------------------------------------------------------------------- /django_drf/config/urls.py: -------------------------------------------------------------------------------- 1 | 2 | from django.urls import path 3 | from car_registry.views import CarListView, CarListViewWithModel, CarListViewWithModelPrefetched 4 | from apis.car_listing_api import CarListingAPI 5 | from apis.car_listing_api_2 import CarListingAPI as CarListingAPI2 6 | from apis.car_listing_api_3 import CarListingAPI as CarListingAPI3 7 | from apis.car_listing_api_4_paginated import CarListingAPI as CarListingAPI4 8 | from apis.car_listing_api_drf_serializer_with_orjson import CarListingAPI as CarListingAPI5 9 | 10 | 11 | urlpatterns = [ 12 | path('cars/', CarListView.as_view()), 13 | path('retrieve-cars-with-model/', CarListViewWithModel.as_view()), 14 | path('retrieve-cars-with-prefetch-related/', CarListViewWithModelPrefetched.as_view()), 15 | 16 | path('api/cars/', CarListingAPI.as_view()), 17 | path('api/cars-2/', CarListingAPI2.as_view()), 18 | path('api/cars-3/', CarListingAPI3.as_view()), 19 | path('api/cars-4-paginated/', CarListingAPI4.as_view()), 20 | path('api/cars-serializer-orjson/', CarListingAPI5.as_view()), 21 | 22 | path('drf/with-serializer/', CarListingAPI5.as_view()), 23 | path('drf/without-serializer/', CarListingAPI3.as_view()), 24 | ] 25 | -------------------------------------------------------------------------------- /django_drf/config/wsgi.py: -------------------------------------------------------------------------------- 1 | """ 2 | WSGI config for demo project. 3 | 4 | It exposes the WSGI callable as a module-level variable named ``application``. 5 | 6 | For more information on this file, see 7 | https://docs.djangoproject.com/en/5.1/howto/deployment/wsgi/ 8 | """ 9 | 10 | import os 11 | 12 | from django.core.wsgi import get_wsgi_application 13 | 14 | os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'config.settings') 15 | 16 | application = get_wsgi_application() 17 | -------------------------------------------------------------------------------- /django_drf/custom_response.py: -------------------------------------------------------------------------------- 1 | import orjson 2 | from django.db.models import QuerySet 3 | from django.http import HttpResponse 4 | 5 | 6 | class OrJsonResponse(HttpResponse): 7 | def __init__(self, data, json_dumps_params=None, *args, **kwargs): 8 | if json_dumps_params is None: 9 | json_dumps_params = {} 10 | kwargs.setdefault("content_type", "application/json") 11 | if isinstance(data, QuerySet): 12 | data = list(data) 13 | data = orjson.dumps(data, **json_dumps_params) 14 | super().__init__(content=data, **kwargs) 15 | -------------------------------------------------------------------------------- /django_drf/manage.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | """Django's command-line utility for administrative tasks.""" 3 | import os 4 | import sys 5 | 6 | 7 | def main(): 8 | """Run administrative tasks.""" 9 | os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'config.settings') 10 | try: 11 | from django.core.management import execute_from_command_line 12 | except ImportError as exc: 13 | raise ImportError( 14 | "Couldn't import Django. Are you sure it's installed and " 15 | "available on your PYTHONPATH environment variable? Did you " 16 | "forget to activate a virtual environment?" 17 | ) from exc 18 | execute_from_command_line(sys.argv) 19 | 20 | 21 | if __name__ == '__main__': 22 | main() 23 | -------------------------------------------------------------------------------- /django_drf/requirements.txt: -------------------------------------------------------------------------------- 1 | faker==30.8.2 2 | daphne==4.1.2 3 | Django==5.1.5 4 | djangorestframework==3.15.2 5 | orjson==3.10.15 6 | psycopg[binary]==3.2.4 7 | -------------------------------------------------------------------------------- /django_drf/services/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/oscarychen/building-efficient-api/552a263c8eb4bfc5a5c44e21758fae95352c0e70/django_drf/services/__init__.py -------------------------------------------------------------------------------- /django_drf/services/car_services.py: -------------------------------------------------------------------------------- 1 | from django.db.models import F 2 | 3 | from car_registry.models import Car 4 | 5 | 6 | class CarService: 7 | def retrieve_all_cars(self): 8 | return Car.objects.all().select_related('model') 9 | 10 | def retrieve_all_cars_annotated(self): 11 | qs = self.retrieve_all_cars() 12 | qs = qs.annotate( 13 | car_model_id=F('model_id'), 14 | car_model_name=F('model__name'), 15 | car_model_year=F('model__year'), 16 | color=F('model__color') 17 | ) 18 | return qs 19 | 20 | def retrieve_all_cars_as_dicts(self): 21 | cars = self.retrieve_all_cars_annotated() 22 | return cars.values( 23 | 'id', 24 | 'vin', 25 | 'owner', 26 | 'created_at', 27 | 'updated_at', 28 | 'car_model_id', 29 | 'car_model_name', 30 | 'car_model_year', 31 | 'color' 32 | ) -------------------------------------------------------------------------------- /django_drf/wait_for_db.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | # wait_for_db.sh 3 | 4 | set -e 5 | 6 | host="$1" 7 | shift 8 | cmd="$@" 9 | 10 | until pg_isready -h db -p 5432; do 11 | >&2 echo "Postgres is unavailable - sleeping" 12 | sleep 1 13 | done 14 | 15 | >&2 echo "Postgres is up - executing command" 16 | exec $cmd -------------------------------------------------------------------------------- /django_ninja/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM python:3.12.8-slim 2 | 3 | ENV PYTHONDONTWRITEBYTECODE 1 4 | ENV PYTHONUNBUFFERED 1 5 | 6 | WORKDIR /app 7 | 8 | # Install system dependencies 9 | RUN apt-get update && apt-get install -y \ 10 | postgresql-client \ 11 | && rm -rf /var/lib/apt/lists/* 12 | 13 | COPY ./django_ninja/requirements.txt /app/requirements.txt 14 | 15 | RUN pip install --upgrade pip 16 | RUN pip install --no-cache-dir -r requirements.txt 17 | 18 | COPY ./django_ninja/ . 19 | 20 | CMD ["daphne", "--b", "0.0.0.0", "-p", "8001", "config.asgi:application"] 21 | -------------------------------------------------------------------------------- /django_ninja/apis/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/oscarychen/building-efficient-api/552a263c8eb4bfc5a5c44e21758fae95352c0e70/django_ninja/apis/__init__.py -------------------------------------------------------------------------------- /django_ninja/apis/car_listing_api.py: -------------------------------------------------------------------------------- 1 | from datetime import datetime 2 | 3 | from django.http import HttpRequest 4 | from ninja import Router, Schema 5 | 6 | from services.car_services import CarService 7 | 8 | router = Router() 9 | 10 | class ListCarResponseItem(Schema): 11 | id: int 12 | vin: str 13 | owner: str 14 | created_at: datetime 15 | updated_at: datetime 16 | car_model_id: int 17 | car_model_name: str 18 | car_model_year: int 19 | color: str 20 | 21 | @router.get("/ninja/with-schema/", response=list[ListCarResponseItem]) 22 | def list_cars(request: HttpRequest): 23 | return CarService().retrieve_all_cars_annotated() 24 | 25 | 26 | @router.get("/ninja/without-schema/") 27 | def list_cars_2(request: HttpRequest): 28 | return list(CarService().retrieve_all_cars_as_dicts()) 29 | -------------------------------------------------------------------------------- /django_ninja/car_registry/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/oscarychen/building-efficient-api/552a263c8eb4bfc5a5c44e21758fae95352c0e70/django_ninja/car_registry/__init__.py -------------------------------------------------------------------------------- /django_ninja/car_registry/admin.py: -------------------------------------------------------------------------------- 1 | from django.contrib import admin 2 | 3 | # Register your models here. 4 | -------------------------------------------------------------------------------- /django_ninja/car_registry/apps.py: -------------------------------------------------------------------------------- 1 | from django.apps import AppConfig 2 | 3 | 4 | class CarRegistryConfig(AppConfig): 5 | default_auto_field = 'django.db.models.BigAutoField' 6 | name = 'car_registry' 7 | -------------------------------------------------------------------------------- /django_ninja/car_registry/management/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/oscarychen/building-efficient-api/552a263c8eb4bfc5a5c44e21758fae95352c0e70/django_ninja/car_registry/management/__init__.py -------------------------------------------------------------------------------- /django_ninja/car_registry/management/commands/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/oscarychen/building-efficient-api/552a263c8eb4bfc5a5c44e21758fae95352c0e70/django_ninja/car_registry/management/commands/__init__.py -------------------------------------------------------------------------------- /django_ninja/car_registry/management/commands/populate.py: -------------------------------------------------------------------------------- 1 | from typing import Any 2 | from django.core.management.base import BaseCommand 3 | import random 4 | from car_registry.models import Car, CarModel 5 | from faker import Faker 6 | 7 | 8 | class Command(BaseCommand): 9 | def handle(self, *args: Any, **options: Any) -> str | None: 10 | 11 | self.bulk_create_car_models(200) 12 | 13 | for i in range(50): 14 | print(f'Creating cars batch {i+1}...') 15 | self.bulk_create_cars(2000) 16 | 17 | 18 | def bulk_create_car_models(self, quantity=100): 19 | colors = ['Red', 'Blue', 'Green', 'Yellow', 'Black', 'White', 'Silver', 'Gold', 'Purple', 'Orange'] 20 | years = [2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024] 21 | prices = [10000, 20000, 30000, 40000, 50000, 60000, 70000, 80000, 90000, 100000] 22 | 23 | car_models = [] 24 | 25 | for i in range(quantity): 26 | car_models.append( 27 | CarModel( 28 | name=f'Car Model {i}', 29 | make=f'Car Make {i}', 30 | year=random.choice(years), 31 | color=random.choice(colors), 32 | price=random.choice(prices) 33 | ) 34 | ) 35 | 36 | CarModel.objects.bulk_create(car_models) 37 | 38 | def bulk_create_cars(self, quantity=1000): 39 | car_models = CarModel.objects.all().values_list('id', flat=True) 40 | 41 | cars = [] 42 | 43 | for i in range(quantity): 44 | cars.append( 45 | Car( 46 | vin=f'VIN-{i}', 47 | model_id=random.choice(car_models), 48 | owner=Faker().name() 49 | ) 50 | ) 51 | 52 | Car.objects.bulk_create(cars) 53 | -------------------------------------------------------------------------------- /django_ninja/car_registry/migrations/0001_initial.py: -------------------------------------------------------------------------------- 1 | # Generated by Django 5.1.2 on 2024-11-05 03:50 2 | 3 | import django.db.models.deletion 4 | from django.db import migrations, models 5 | 6 | 7 | class Migration(migrations.Migration): 8 | 9 | initial = True 10 | 11 | dependencies = [ 12 | ] 13 | 14 | operations = [ 15 | migrations.CreateModel( 16 | name='CarModel', 17 | fields=[ 18 | ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), 19 | ('name', models.CharField(max_length=100)), 20 | ('make', models.CharField(max_length=100)), 21 | ('year', models.IntegerField()), 22 | ('color', models.CharField(max_length=100)), 23 | ('price', models.DecimalField(decimal_places=2, max_digits=10)), 24 | ('created_at', models.DateTimeField(auto_now_add=True)), 25 | ('updated_at', models.DateTimeField(auto_now=True)), 26 | ], 27 | ), 28 | migrations.CreateModel( 29 | name='Car', 30 | fields=[ 31 | ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), 32 | ('vin', models.CharField(max_length=17)), 33 | ('owner', models.CharField(max_length=100)), 34 | ('created_at', models.DateTimeField(auto_now_add=True)), 35 | ('updated_at', models.DateTimeField(auto_now=True)), 36 | ('model', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='car_registry.carmodel')), 37 | ], 38 | ), 39 | ] 40 | -------------------------------------------------------------------------------- /django_ninja/car_registry/migrations/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/oscarychen/building-efficient-api/552a263c8eb4bfc5a5c44e21758fae95352c0e70/django_ninja/car_registry/migrations/__init__.py -------------------------------------------------------------------------------- /django_ninja/car_registry/models.py: -------------------------------------------------------------------------------- 1 | from django.db import models 2 | 3 | 4 | class CarModel(models.Model): 5 | name = models.CharField(max_length=100) 6 | make = models.CharField(max_length=100) 7 | year = models.IntegerField() 8 | color = models.CharField(max_length=100) 9 | price = models.DecimalField(max_digits=10, decimal_places=2) 10 | created_at = models.DateTimeField(auto_now_add=True) 11 | updated_at = models.DateTimeField(auto_now=True) 12 | 13 | 14 | class Car(models.Model): 15 | vin = models.CharField(max_length=17) 16 | model = models.ForeignKey(CarModel, on_delete=models.CASCADE) 17 | owner = models.CharField(max_length=100) 18 | created_at = models.DateTimeField(auto_now_add=True) 19 | updated_at = models.DateTimeField(auto_now=True) 20 | 21 | -------------------------------------------------------------------------------- /django_ninja/car_registry/tests.py: -------------------------------------------------------------------------------- 1 | from django.test import TestCase 2 | 3 | # Create your tests here. 4 | -------------------------------------------------------------------------------- /django_ninja/config/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/oscarychen/building-efficient-api/552a263c8eb4bfc5a5c44e21758fae95352c0e70/django_ninja/config/__init__.py -------------------------------------------------------------------------------- /django_ninja/config/asgi.py: -------------------------------------------------------------------------------- 1 | """ 2 | ASGI config for demo project. 3 | 4 | It exposes the ASGI callable as a module-level variable named ``application``. 5 | 6 | For more information on this file, see 7 | https://docs.djangoproject.com/en/5.1/howto/deployment/asgi/ 8 | """ 9 | 10 | import os 11 | 12 | from django.core.asgi import get_asgi_application 13 | 14 | os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'config.settings') 15 | 16 | application = get_asgi_application() 17 | -------------------------------------------------------------------------------- /django_ninja/config/settings.py: -------------------------------------------------------------------------------- 1 | """ 2 | Django settings for demo project. 3 | 4 | Generated by 'django-admin startproject' using Django 5.1.2. 5 | 6 | For more information on this file, see 7 | https://docs.djangoproject.com/en/5.1/topics/settings/ 8 | 9 | For the full list of settings and their values, see 10 | https://docs.djangoproject.com/en/5.1/ref/settings/ 11 | """ 12 | 13 | from pathlib import Path 14 | 15 | # Build paths inside the project like this: BASE_DIR / 'subdir'. 16 | BASE_DIR = Path(__file__).resolve().parent.parent 17 | 18 | 19 | # Quick-start development settings - unsuitable for production 20 | # See https://docs.djangoproject.com/en/5.1/howto/deployment/checklist/ 21 | 22 | # SECURITY WARNING: keep the secret key used in production secret! 23 | SECRET_KEY = 'django-insecure-eo-w8#)7r4n%p86a5tb(yfumoett_28^n6r&++1u5xzbu19fzf' 24 | 25 | # SECURITY WARNING: don't run with debug turned on in production! 26 | DEBUG = True 27 | 28 | ALLOWED_HOSTS = ["localhost", "django-ninja"] 29 | 30 | 31 | # Application definition 32 | 33 | INSTALLED_APPS = [ 34 | 'daphne', 35 | 'django.contrib.admin', 36 | 'django.contrib.auth', 37 | 'django.contrib.contenttypes', 38 | 'django.contrib.sessions', 39 | 'django.contrib.messages', 40 | 'django.contrib.staticfiles', 41 | 'car_registry', 42 | ] 43 | 44 | MIDDLEWARE = [ 45 | 'django.middleware.security.SecurityMiddleware', 46 | 'django.contrib.sessions.middleware.SessionMiddleware', 47 | 'django.middleware.common.CommonMiddleware', 48 | 'django.middleware.csrf.CsrfViewMiddleware', 49 | 'django.contrib.auth.middleware.AuthenticationMiddleware', 50 | 'django.contrib.messages.middleware.MessageMiddleware', 51 | 'django.middleware.clickjacking.XFrameOptionsMiddleware', 52 | ] 53 | 54 | ROOT_URLCONF = 'config.urls' 55 | 56 | TEMPLATES = [ 57 | { 58 | 'BACKEND': 'django.template.backends.django.DjangoTemplates', 59 | 'DIRS': [], 60 | 'APP_DIRS': True, 61 | 'OPTIONS': { 62 | 'context_processors': [ 63 | 'django.template.context_processors.debug', 64 | 'django.template.context_processors.request', 65 | 'django.contrib.auth.context_processors.auth', 66 | 'django.contrib.messages.context_processors.messages', 67 | ], 68 | }, 69 | }, 70 | ] 71 | 72 | WSGI_APPLICATION = 'config.wsgi.application' 73 | 74 | 75 | # Database 76 | # https://docs.djangoproject.com/en/5.1/ref/settings/#databases 77 | 78 | DATABASES = { 79 | "default": { 80 | "ENGINE": "django.db.backends.postgresql", 81 | "NAME": "api_demo_db", 82 | "USER": "postgres", 83 | "PASSWORD": "postgres", 84 | # "HOST": "127.0.0.1", 85 | "HOST": "db", 86 | "PORT": "5432", 87 | } 88 | } 89 | 90 | 91 | # Password validation 92 | # https://docs.djangoproject.com/en/5.1/ref/settings/#auth-password-validators 93 | 94 | AUTH_PASSWORD_VALIDATORS = [ 95 | { 96 | 'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator', 97 | }, 98 | { 99 | 'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator', 100 | }, 101 | { 102 | 'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator', 103 | }, 104 | { 105 | 'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator', 106 | }, 107 | ] 108 | 109 | 110 | # Internationalization 111 | # https://docs.djangoproject.com/en/5.1/topics/i18n/ 112 | 113 | LANGUAGE_CODE = 'en-us' 114 | 115 | TIME_ZONE = 'UTC' 116 | 117 | USE_I18N = True 118 | 119 | USE_TZ = True 120 | 121 | 122 | # Static files (CSS, JavaScript, Images) 123 | # https://docs.djangoproject.com/en/5.1/howto/static-files/ 124 | 125 | STATIC_URL = 'static/' 126 | 127 | # Default primary key field type 128 | # https://docs.djangoproject.com/en/5.1/ref/settings/#default-auto-field 129 | 130 | DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField' 131 | 132 | -------------------------------------------------------------------------------- /django_ninja/config/urls.py: -------------------------------------------------------------------------------- 1 | 2 | from django.urls import path 3 | from ninja import NinjaAPI 4 | from apis.car_listing_api import router 5 | from custom_renderer import ORJSONRenderer 6 | 7 | # api = NinjaAPI() # Chapter 2 8 | api = NinjaAPI(renderer=ORJSONRenderer()) # Chapter 3 9 | 10 | api.add_router("", router) 11 | 12 | 13 | urlpatterns = [ 14 | path("", api.urls), 15 | ] 16 | -------------------------------------------------------------------------------- /django_ninja/config/wsgi.py: -------------------------------------------------------------------------------- 1 | """ 2 | WSGI config for demo project. 3 | 4 | It exposes the WSGI callable as a module-level variable named ``application``. 5 | 6 | For more information on this file, see 7 | https://docs.djangoproject.com/en/5.1/howto/deployment/wsgi/ 8 | """ 9 | 10 | import os 11 | 12 | from django.core.wsgi import get_wsgi_application 13 | 14 | os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'config.settings') 15 | 16 | application = get_wsgi_application() 17 | -------------------------------------------------------------------------------- /django_ninja/custom_renderer.py: -------------------------------------------------------------------------------- 1 | import orjson 2 | from ninja.renderers import BaseRenderer 3 | 4 | 5 | class ORJSONRenderer(BaseRenderer): 6 | media_type = "application/json" 7 | 8 | def render(self, request, data, *, response_status): 9 | return orjson.dumps(data) 10 | -------------------------------------------------------------------------------- /django_ninja/manage.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | """Django's command-line utility for administrative tasks.""" 3 | import os 4 | import sys 5 | 6 | 7 | def main(): 8 | """Run administrative tasks.""" 9 | os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'config.settings') 10 | try: 11 | from django.core.management import execute_from_command_line 12 | except ImportError as exc: 13 | raise ImportError( 14 | "Couldn't import Django. Are you sure it's installed and " 15 | "available on your PYTHONPATH environment variable? Did you " 16 | "forget to activate a virtual environment?" 17 | ) from exc 18 | execute_from_command_line(sys.argv) 19 | 20 | 21 | if __name__ == '__main__': 22 | main() 23 | -------------------------------------------------------------------------------- /django_ninja/requirements.txt: -------------------------------------------------------------------------------- 1 | faker==30.8.2 2 | daphne==4.1.2 3 | Django==5.1.5 4 | django-ninja==1.3.0 5 | orjson==3.10.15 6 | psycopg[binary]==3.2.4 7 | -------------------------------------------------------------------------------- /django_ninja/services/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/oscarychen/building-efficient-api/552a263c8eb4bfc5a5c44e21758fae95352c0e70/django_ninja/services/__init__.py -------------------------------------------------------------------------------- /django_ninja/services/car_services.py: -------------------------------------------------------------------------------- 1 | from django.db.models import F 2 | 3 | from car_registry.models import Car 4 | 5 | 6 | class CarService: 7 | def retrieve_all_cars(self): 8 | return Car.objects.all().select_related('model') 9 | 10 | def retrieve_all_cars_annotated(self): 11 | qs = self.retrieve_all_cars() 12 | qs = qs.annotate( 13 | car_model_id=F('model_id'), 14 | car_model_name=F('model__name'), 15 | car_model_year=F('model__year'), 16 | color=F('model__color') 17 | ) 18 | return qs 19 | 20 | def retrieve_all_cars_as_dicts(self): 21 | cars = self.retrieve_all_cars_annotated() 22 | return cars.values( 23 | 'id', 24 | 'vin', 25 | 'owner', 26 | 'created_at', 27 | 'updated_at', 28 | 'car_model_id', 29 | 'car_model_name', 30 | 'car_model_year', 31 | 'color' 32 | ) -------------------------------------------------------------------------------- /django_ninja/wait_for_db.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | # wait_for_db.sh 3 | 4 | set -e 5 | 6 | host="$1" 7 | shift 8 | cmd="$@" 9 | 10 | until pg_isready -h db -p 5432; do 11 | >&2 echo "Postgres is unavailable - sleeping" 12 | sleep 1 13 | done 14 | 15 | >&2 echo "Postgres is up - executing command" 16 | exec $cmd -------------------------------------------------------------------------------- /docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: "3.8" 2 | 3 | services: 4 | db: 5 | image: postgres:16.1-alpine 6 | container_name: "api-demo-db" 7 | ports: 8 | - "5432:5432" 9 | environment: 10 | - POSTGRES_DB=api_demo_db 11 | - POSTGRES_USER=postgres 12 | - POSTGRES_PASSWORD=postgres 13 | networks: 14 | - api_demo_app 15 | volumes: 16 | - database_postgres:/var/lib/postgresql/data 17 | healthcheck: 18 | test: [ "CMD-SHELL", "pg_isready -U postgres" ] 19 | interval: 5s 20 | timeout: 5s 21 | retries: 5 22 | 23 | django-drf: 24 | build: 25 | context: . 26 | dockerfile: ./django_drf/Dockerfile 27 | container_name: "api-demo-django-drf" 28 | ports: 29 | - "8000:8000" 30 | depends_on: 31 | - db 32 | networks: 33 | - api_demo_app 34 | 35 | django-ninja: 36 | build: 37 | context: . 38 | dockerfile: ./django_ninja/Dockerfile 39 | container_name: "api-demo-django-ninja" 40 | ports: 41 | - "8001:8001" 42 | depends_on: 43 | - db 44 | networks: 45 | - api_demo_app 46 | 47 | fastapi: 48 | build: 49 | context: . 50 | dockerfile: ./fast_api/Dockerfile 51 | container_name: "api-demo-fastapi" 52 | ports: 53 | - "8002:8002" 54 | depends_on: 55 | - db 56 | networks: 57 | - api_demo_app 58 | 59 | go-sqlc-mux: 60 | build: 61 | context: . 62 | dockerfile: ./go_sqlc_mux/Dockerfile 63 | container_name: "api-demo-go-sqlc-mux" 64 | ports: 65 | - "8003:8003" 66 | depends_on: 67 | - db 68 | networks: 69 | - api_demo_app 70 | 71 | locust: 72 | build: 73 | context: . 74 | dockerfile: ./load_testing/Dockerfile 75 | container_name: "api-demo-locust" 76 | ports: 77 | - "8089:8089" 78 | networks: 79 | - api_demo_app 80 | 81 | volumes: 82 | database_postgres: 83 | 84 | networks: 85 | api_demo_app: 86 | driver: bridge 87 | -------------------------------------------------------------------------------- /docs/readme.md: -------------------------------------------------------------------------------- 1 | # Building efficient maintainable REST APIs for data retrieval 2 | _comparing methodologies in Django REST Framework, Django-Ninja, and Golang_ 3 | 4 | 5 | In this repo, I have set up multiple mini projects to compare how different API code designs can affect performance. 6 | 7 | We start with a simple toy example of a **Django REST Framework API**, we will add a bit requirement to the API and see that it struggles to return 100k rows in over 30 seconds, and then step-by-step optimize it to return in under 1 second. 8 | 9 | I've also set up an identical API using **Django-Ninja**, as well as using **Golang** with **sqlc** and **mux**, to compare their performances. 10 | 11 | Jump to TLDR [Conclusions](#conclusions-tldr) 12 | 13 | 14 | # Data model 15 | 16 | For this experiment, we will set up small data models in each tech stack: 17 | 18 | ``` 19 | +------------+ +------------+ 20 | | Car | | CarModel | 21 | +------------+ +------------+ 22 | | model_id | -----------> | id | 23 | +------------+ +------------+ 24 | ``` 25 | 26 | The Car and CarModel models have 1-to-many relationship between them, each with some of their own attributes not shown in the diagram. 27 | 28 | ## Part A: Django REST Framework (DRF) 29 | 30 | Django is a popular choice for startups and hobby projects. It offers a feature-rich ORM and migration system. Django REST Framework is widely used for building web application backends with Django. 31 | To get started, I've included some basic docker configuration for Django development server and postgres server, as well as a simple management command to populate 100k rows of Car records. 32 | - build: `make docker-build` 33 | - run Django REST Framework: `make docker-up-drf` 34 | - run Django-Ninja: `make docker-up-ninja` 35 | - run Go: `make docker-up-go` 36 | - populate database with 100k Car records: `make django-drf-populate` 37 | 38 | ### Chapter 1: Retrieving Car records Using DRF ModelSerializer 39 | 40 | In typical Django REST framework toy example fashion, we set up a ModelSerializer and API View as such: 41 | 42 | ```python 43 | # django_drf/car_registry/serializers.py 44 | class CarSerializer(serializers.ModelSerializer): 45 | class Meta: 46 | model = Car 47 | fields = '__all__' 48 | 49 | # django_drf/car_registry/views.py 50 | class CarListView(ListAPIView): 51 | serializer_class = CarSerializer 52 | queryset = Car.objects.all() 53 | ``` 54 | 55 | and expose this API View as on the follow route: 56 | 57 | ```python 58 | # django_drf/config/urls.py 59 | urlpatterns = [ 60 | path('cars/', CarListView.as_view()), 61 | ... 62 | ] 63 | ``` 64 | 65 | Now, we can hit this endpoint (you can use the api.http file provided in the repository root) 66 | 67 | > GET http://localhost:8000/cars/ 68 | 69 | > Response code: 200 (OK); Time: 2564ms (2 s 564 ms); Content length: 15107465 bytes (15.11 MB) 70 | 71 | ### Chapter 2: Retrieving Car records with related model attributes using DRF ModelSerializer 72 | 73 | Next, we are going to want to return some additional data for each of the Car record retrieved while maintaining the response data structure, particular the related CarModel's attributes `name`, `year`, and `color`. Most Django developers would simply make the following changes to the serializer: 74 | 75 | ```python 76 | # django_drf/car_registry/serializers.py 77 | class CarSerializerWithRelatedModel(serializers.ModelSerializer): 78 | car_model_id = serializers.IntegerField(source='model_id', read_only=True) 79 | car_model_name = serializers.CharField(source='model.name', read_only=True) 80 | car_model_year = serializers.IntegerField(source='model.year', read_only=True) 81 | color = serializers.CharField(source='model.color', read_only=True) 82 | class Meta: 83 | model = Car 84 | fields = ... 85 | ``` 86 | 87 | For comparison purpose, we keep the original API as-is, and added the modified API as a new API. Now we hit this new endpoint 88 | 89 | > GET http://localhost:8000/retrieve-cars-with-model/ 90 | 91 | > Response code: 200 (OK); Time: 30779ms (30 s 779 ms); Content length: 22856443 bytes (22.86 MB) 92 | 93 | That took a lot longer. 94 | The query in the API view `Car.objects.all()` did not contain data about each `Car` record's related `CarModel` record, by the time the Serializer is trying to serialize each record, Django has to fire off an additional query to fetch the related `CarModel` in order to get the `model_name`, `model_year` and `color` that the serializer is asking for. This is called N+1 query problem, and it's very common problem in stacks involving some sort of ORM. 95 | 96 | In Django REST Framework, this can be a very easy mistake to make (simply by modifying a serializer) and sometimes not easy to spot; luckily it's usually pretty simple to fix. 97 | 98 | ### Chapter 3: Optimizing retrieving Car records with related model attributes using query Select_related / Prefetch_related 99 | 100 | Based on the previous example, we are going to modify the query by "prefetching" the related records: 101 | 102 | ```python 103 | # django_drf/car_registry/views.py 104 | class CarListViewWithModelPrefetched(ListAPIView): 105 | serializer_class = CarSerializerWithRelatedModel 106 | queryset = Car.objects.all().select_related('model') # or .prefetch_related('model') 107 | ``` 108 | 109 | Again, for comparison purpose, we keep the previous API as-is, and added the modified API as a new API. Now we hit this new endpoint 110 | 111 | > GET http://localhost:8000/retrieve-cars-with-prefetch-related/ 112 | 113 | > Response code: 200 (OK); Time: 3968ms (3 s 968 ms); Content length: 22856443 bytes (22.86 MB) 114 | 115 | Now we are back down to around 3 seconds. 116 | 117 | The `prefetch_related('model')` tells to the ORM to perform a single query to fetch all the related `CarModel` model instances, and the returned data from database contains all the attributes of `CarModel`, so when the serializer tries to access those attributes, the ORM does not need to query the database again, thus the N+1 query problem is eliminated. 118 | The `select_related('model')` performs a SQL join on the original SQL query as long as the relation is not a many-to-many relation, and the SQL query result contains the related data in a single query, and is technically preferred in our example here. 119 | 120 | This is typically where most Django REST Framework application would stop at in terms of API query optimization. However, sometimes we are requesting so much data, this is still not enough. But first, let's re-organize our project structure a little bit. 121 | 122 | ### Chapter 4: Digress - Changing up the project structure a bit 123 | 124 | We are going to leave the previous implemented serializers.py and views.py where they are for now even though we will not be using them anymore. 125 | 126 | We want a project with 3 tiers: Models, Services, and APIs. Service and API code will be outside the Django app directory, while Django app directories will store models. This setup allows us to develop services that can encapsulate complex business logic using multiple models and across different Django apps if needed. 127 | 128 | ``` 129 | /django_project_root 130 | |-/project_level_config 131 | |-/apis 132 | | |-/some_api.py 133 | | |-/some_other_api.py 134 | |-/services 135 | | |-/feature_a_services.py 136 | | |-/feature_b_services.py 137 | | |-... 138 | |-/app_1 139 | | |--/app_1_models.py 140 | |-/app_2 141 | | |--/app_2_models.py 142 | |-... 143 | ``` 144 | 145 | I also do not like separating Serializer implementation from the API View implementation. Saving a bit of code duplication but create coupling between serializers and therefore APIs can be a source of problem in larger teams. As you have seen previously, it's very easy for a developer to make a little change in 1 serializer and somehow causes performance problems in serializers and views elsewhere. I would put the serializer and the view implementation together in a `some_api.py`, in fact I would put request param serializer, request body data serializer, and response serializer implementation all in the same file with the API View implementation that they are responsible for. 146 | 147 | You could probably also put all the Django app modules in a repository directory. In a way we are opinionated to use Django mostly only as a Model Repository, and we want to manage rest of the Django project structure ourselves more or less in a different pattern. But we won't go that far. 148 | 149 | With these in mind, we are going to implement the following: 150 | 151 | ```python 152 | # django_drf/services/car_services.py 153 | class CarService: 154 | def retrieve_all_cars(self): 155 | pass 156 | 157 | # django_drf/apis/car_listing_api.py 158 | class CarListingAPI(APIView): 159 | def get(self, *args, **kwargs) -> Response: 160 | return Response({'detail': 'to be implemented'}, status=status.HTTP_501_NOT_IMPLEMENTED) 161 | ``` 162 | 163 | Now we are ready to flesh out some of the implementation details next. 164 | 165 | ### Chapter 5: Replacing ModelSerializer with Serializer 166 | 167 | The first thing we are going to try is to use a DRF Serializer instead of ModelSerializer. Serializers are lighter weight than Model Serializer, while ModelSerializer automatically creates field-level validators from their corresponding Model's field validators and simplifies write operations, it's an unnecessary overhead for read operations like listing and retrieving records from the database. 168 | 169 | We are going to fill out the service and API from the previous section as the following: 170 | 171 | ```python 172 | # django_drf/services/car_services.py 173 | class CarService: 174 | def retrieve_all_cars(self): 175 | return Car.objects.all().select_related('model') 176 | 177 | # django_drf/apis/car_listing_api.py 178 | class CarListingResponseSerializer(serializers.Serializer): 179 | vin = serializers.CharField() 180 | owner = serializers.CharField() 181 | created_at = serializers.DateTimeField() 182 | updated_at = serializers.DateTimeField() 183 | model = serializers.IntegerField(source="model_id") 184 | model_name = serializers.CharField(source='model.name') 185 | model_year = serializers.IntegerField(source='model.year') 186 | color = serializers.CharField(source='model.color') 187 | 188 | class CarListingAPI(APIView): 189 | def get(self, *args, **kwargs) -> Response: 190 | car_instances = CarService().retrieve_all_cars() 191 | response_serializer = CarListingResponseSerializer(car_instances, many=True) 192 | return Response(response_serializer.data, status=status.HTTP_200_OK) 193 | ``` 194 | Now we hit this new endpoint 195 | > GET http://localhost:8000/api/cars/ 196 | 197 | > Response code: 200 (OK); Time: 3996ms (3 s 996 ms); Content length: 22856443 bytes (22.86 MB) 198 | 199 | This is not really faster than previous implementation. The DRF Serializer is still iterating through each of the Car instances to turn it into a Python dictionary. There are few other things you can try here in Python to speed up the process, but we are going to move onto something else. 200 | 201 | ### Chapter 6: Using Django ORM more and DRF Serializer less 202 | Next we are going to write the query in a way that we can get the data we want directly from the database, so that we can skip turning the data into Python dictionaries before serializing them into json. 203 | 204 | Building on the existing CarService class: 205 | ```python 206 | # django_drf/services/car_services.py 207 | class CarService: 208 | def retrieve_all_cars(self): 209 | return Car.objects.all().select_related('model') 210 | 211 | def retrieve_all_cars_annotated(self): 212 | qs = self.retrieve_all_cars() 213 | qs = qs.annotate( 214 | car_model_id=F('model_id'), 215 | car_model_name=F('model__name'), 216 | car_model_year=F('model__year'), 217 | color=F('model__color') 218 | ) 219 | return qs 220 | 221 | def retrieve_all_cars_as_dicts(self): 222 | cars = self.retrieve_all_cars_annotated() 223 | return cars.values( 224 | 'id', 225 | 'vin', 226 | 'owner', 227 | 'created_at', 228 | 'updated_at', 229 | 'car_model_id', 230 | 'car_model_name', 231 | 'car_model_year', 232 | 'color' 233 | ) 234 | ``` 235 | Here we try to leverage the database as much as possible to produce the final data as closely as we would want to return through the API. 236 | We will implement the API as follows: 237 | ```python 238 | # django_drf/apis/car_listing_api_2.py 239 | class CarListingAPI(APIView): 240 | def get(self, *args, **kwargs) -> Response: 241 | car_dicts = CarService().retrieve_all_cars_as_dicts() 242 | return Response(car_dicts, status=status.HTTP_200_OK) 243 | ``` 244 | For comparison purpose, again we keep the original API as-is, and added the modified API as a new API. 245 | Now we hit this new endpoint 246 | 247 | > GET http://localhost:8000/api/cars-2/ 248 | 249 | > Response code: 200 (OK); Time: 1373ms (1 s 373 ms); Content length: 22856443 bytes (22.86 MB) 250 | 251 | There we cut the time in half again, mostly because we are not iterating the N number of Car instances in Python from the queryset as it is already serializeable. 252 | This is not always possible and sometimes may require you to write some pretty complicated Django queries. But as long as there is a way to write the query and make the database do the work, it's probably going to be faster than doing it in Python. 253 | 254 | ### Chapter 7: Faster serialization using OrJson 255 | The last thing we are going to try to use a faster JSON serializer. Django REST Framework uses Python's built-in JSON serializer, which is not the fastest JSON serializer out there. We are going to use orjson, which is a fast JSON serializer written in Rust. 256 | 257 | First, we implement a custom response class that uses orjson to serialize the response data: 258 | 259 | ```python 260 | # django_drf/custom_response.py 261 | class OrJsonResponse(HttpResponse): 262 | def __init__(self, data, json_dumps_params=None, *args, **kwargs): 263 | if json_dumps_params is None: 264 | json_dumps_params = {} 265 | kwargs.setdefault("content_type", "application/json") 266 | if isinstance(data, QuerySet): 267 | data = list(data) 268 | data = orjson.dumps(data, **json_dumps_params) 269 | super().__init__(content=data, **kwargs) 270 | ``` 271 | 272 | With that in place, lets implement the previous API but returns the response using OrJsonResponse: 273 | 274 | ```python 275 | # django_drf/apis/car_listing_api_3.py 276 | class CarListingAPI(APIView): 277 | def get(self, request: Request, *args, **kwargs) -> Response: 278 | car_dicts = CarService().retrieve_all_cars_as_dicts() 279 | return OrJsonResponse(car_dicts, status=status.HTTP_200_OK) 280 | ``` 281 | Finally, we hit the new endpoint that uses OrJsonResponse 282 | 283 | > GET http://localhost:8000/api/cars-3/ 284 | 285 | > Response code: 200 (OK); Time: 1035ms (1 s 35 ms); Content length: 23856443 bytes (23.86 MB) 286 | 287 | And now we are able to get this API to return in about 1 second with 100k records with data from two tables. 288 | 289 | ### Chapter 8: Other performance considerations 290 | At this point we have optimized the API from database query to response serialization. There are still some other areas that would affect API performance: 291 | - database indexing: It is likely that you would have some sort of filtering in the query, which means you need to consider database indexing. You would want to design your services in a way that they can take the parsed query parameters and use them for filtering as part of the Django query, and avoid doing any sort of filtering in Python. 292 | - query caching: generally a good idea especially if the data is relatively static 293 | - web server compression: such as gzip, would compress the response data to be magnitudes smaller especially for large responses like the example used here 294 | 295 | ### Chapter 9: Pagination 296 | By this point, you probably are wondering why we are not using pagination. I don't think pagination is strictly a performance optimization as it requires a different design in the API consumer (frontend app) which sometimes result in a different user experience requirement. Nevertheless, it is easy enough to add the Pagination class from DRF: 297 | ```python 298 | # django_drf/apis/car_listing_api_4_paginated.py 299 | class CarListingAPI(APIView): 300 | def get(self, request: Request, *args, **kwargs) -> OrJsonResponse: 301 | car_dicts = CarService().retrieve_all_cars_as_dicts() 302 | paginator = PageNumberPagination() 303 | paginated_car_dicts = paginator.paginate_queryset(car_dicts, request) 304 | return OrJsonResponse(paginated_car_dicts, status=status.HTTP_200_OK) 305 | ``` 306 | In the django settings, I set the page size to be 1000, and hit the new endpoint 307 | > GET http://localhost:8000/api/cars-4-paginated/?page=4 308 | 309 | > Response code: 200 (OK); Time: 65ms (65 ms); Content length: 238164 bytes (238.16 kB) 310 | 311 | As expected, the response is much smaller and faster to return. 312 | 313 | ## Part B: Django-Ninja 314 | 315 | Django-Ninja is a newer API framework that is built on top of Django and Pydantic. It is designed to be faster than Django REST Framework, and has similar syntax to FastAPI. 316 | I'm going to copy the django_drf project and modify it to use Django-Ninja instead of Django REST Framework. 317 | The new ninja-powered Django project is set up in its own docker container and exposed on port 8001. 318 | 319 | ### Chapter 1: Retrieving Car records Using Ninja API 320 | Similar to how we have previously set up the Django REST Framework API, we are going to set up a Ninja API in django_ninja/apis/: 321 | ```python 322 | # django_ninja/apis/car_listing_api.py 323 | class ListCarResponseItem(Schema): 324 | vin: str 325 | owner: str 326 | created_at: datetime 327 | updated_at: datetime 328 | car_model_id: int 329 | car_model_name: str 330 | car_model_year: int 331 | color: str 332 | 333 | @router.get("/cars/", response=list[ListCarResponseItem]) 334 | def list_cars(request: HttpRequest): 335 | return CarService().retrieve_all_cars_annotated() 336 | ``` 337 | The Ninja Schemas are analogous to Django REST Framework Serializers, and defines the API's request and response data structure. 338 | Note that this API is making use of the previously implemented CarService method `retrieve_all_cars_annotated`, so this example is comparable to Part A Chapter 5 example. 339 | 340 | Now we hit this new endpoint 341 | > GET http://localhost:8001/ninja/with-schema/ 342 | 343 | > Response code: 200 (OK); Time: 3602ms (3 s 602 ms); Content length: 23856443 bytes (23.86 MB) 344 | 345 | This is comparable performance to the Django REST Framework implementation using Serializer with CarService's annotated queryset (Part A Chapter 5), I suspect that there is no significant performance advantage using the Ninja Schema over DRF Serializer since both are converting ORM objects in Python. 346 | 347 | 348 | ### Chapter 2: Returning data directly from the database 349 | Similar to Part A Chapter 6, we are going to try to return the data directly from the database without converting the ORM objects into Python dictionaries: 350 | ```python 351 | @router.get("/cars-2/") 352 | # django_ninja/apis/car_listing_api.py 353 | def list_cars_2(request: HttpRequest): 354 | return list(CarService().retrieve_all_cars_as_dicts()) 355 | ``` 356 | Now we hit this new endpoint 357 | > GET http://localhost:8001/ninja/without-schema/ 358 | 359 | > Response code: 200 (OK); Time: 1367ms (1 s 367 ms); Content length: 24056442 bytes (24.06 MB) 360 | 361 | Again, we cut down the time in half by not converting the ORM objects into Python dictionaries before serializing them into json. This is also comparable performance to the Django REST Framework implementation using CarService (Part A Chapter 6). 362 | 363 | ### Chapter 3: Using OrJson 364 | Similar to Part A Chapter 7 of DRF, we are going to try to use OrJson to serialize the response data. Based on the Django-nina documentation, we implement an OrJSONRenderer: 365 | 366 | ```python 367 | # django_ninja/custom_renderer.py 368 | class ORJSONRenderer(BaseRenderer): 369 | media_type = "application/json" 370 | 371 | def render(self, request, data, *, response_status): 372 | return orjson.dumps(data) 373 | ``` 374 | 375 | and we add the renderer argument to the NinjaAPI instance: 376 | ```python 377 | # django_ninja/config/urls.py 378 | api = NinjaAPI(renderer=ORJSONRenderer()) 379 | ``` 380 | We hit the endpoint implemented in previous chapter 381 | > GET http://localhost:8001/orjson-api/cars/ 382 | 383 | > Response code: 200 (OK); Time: 1061ms (1 s 61 ms); Content length: 23856443 bytes (23.86 MB) 384 | 385 | The performance is similar to the Django REST Framework implementation using OrJsonResponse (Part A Chapter 7). 386 | 387 | ## Part C: Golang with sqlc and mux 388 | 389 | Finally, we are going to implement the same API using Golang. We are going to use sqlc to generate the database query code, and mux as the web server router. This is one of the more "bare metal" sort of approach to writing Go web server, and it does not involve any ORM. 390 | 391 | The way sqlc works is that you write a SQL query in a .sql file, and sqlc generates Go code that can execute the query and scan the result into a struct. This is similar to how Django's ORM works, but sqlc is much more lightweight and does not have the same level of abstraction as Django's ORM. 392 | 393 | To make things easier to set up, I'm going to "re-engineer" the Django models and put them into Go. Django has a command that allows you to see the SQL query generated for a migration: 394 | ```bash 395 | docker exec -it api-demo-django-drf python manage.py sqlmigrate car_registry 0001 396 | ``` 397 | And we copy the SQL query into `go_sqlc_mux/schemas/0001_initial.up.sql`. 398 | 399 | This file is essentially a migration step that would be used to make changes to the database. 400 | 401 | Similarly, we can steal the SQL query Django used by our previous CarService method `retrieve_all_cars_annotated`, you can simply put a breakpoint or print statement 402 | after each query to see what SQL Django generated, ie: `print(str(qs.query))`. 403 | 404 | We would put such query into `go_sqlc_mux/sqlc/queries/car_registry.sql`. 405 | 406 | Next, we run `sqlc generate` to generate the go Repository code. These code will be in the `go_sqlc_mux/internal/repository/` folder. 407 | Inside of this folder you will see that sqlc has created database models `CarRegistryCar` and `CarRegistryCarmodel` as Go structs, that are akin to models we had set up in Django previously. 408 | 409 | After implementing the Service and Transport layers, and a bunch of other rather verbose almost boilerplate setups, the API is complete. I have the Go server exposed on port 8080. 410 | 411 | > GET http://localhost:8003/go/ 412 | 413 | > Response code: 200 (OK); Time: 486ms (486 ms); Content length: 22834258 bytes (22.83 MB) 414 | 415 | Unsurprisingly, the Go implementation is the fastest yet. 416 | 417 | 418 | # Part D: Comparison 419 | 420 | Previously in Part A, we populated the database with 100k rows of records using the Django management command `make django-drf-populate`. Each time this command is run it adds another 100k records. Let add more data and see how Django REST Framework, Django-Ninja, and Golang compare. 421 | 422 | For comparison purpose, I'm going to add an API that uses Django REST Framework's Serializer (Part A Chapter 5) and with returning the response using ORJson (Part A Chapter 7), so it can be a fair comparison against using Ninja's Schema and ORJson renderer (Part B Chapter 1/3). 423 | 424 | For each data size, each API is tested 10 times, and the response time is recorded, the highest container memory usage during these 10 tests is also recorded. 5 of APIs implemented previously are part of this comparison: 425 | - API implemented using Django REST Framework Serializer and OrJsonResponse 426 | - API implemented using Django REST Framework without Serializer and OrJsonResponse 427 | - API implemented using Django-Ninja Schema and ORJSONRenderer 428 | - API implemented using Django-Ninja without Schema and ORJSONRenderer 429 | - Go API implemented using mux and sqlc 430 | 431 | The data sizes used for testing are as the following: 432 | - 100k records, 23 MB uncompressed response data size 433 | - 200k records, 46 MB uncompressed response data size 434 | - 300k records, 69 MB uncompressed response data size 435 | - 400k records, 92 MB uncompressed response data size 436 | 437 | ### Average response time in milliseconds 438 | 439 | | | 100k records | 200k records | 300k records | 400k records | 440 | |------------------------|--------------|--------------|--------------|--------------| 441 | | DRF with Serializer | 3750 | 7832 | 11235 | 14939 | 442 | | Ninja with Schema | 3455 | 6804 | 10148 | 13309 | 443 | | DRF without Serializer | 1034 | 1929 | 2947 | 3529 | 444 | | Ninja without Schema | 947 | 1850 | 2739 | 3526 | 445 | | Go with mux | 504 | 904 | 1231 | 1716 | 446 | 447 | ![API response times charted](assets/api_response_times.png) 448 | 449 | While Django REST Framework's Serializer and Django-Ninja's Schema are very expensive, Django-Ninja is overall faster and more memory efficient than Django REST Framework. Go is the fastest by a good margin. See appendix for test detail data. 450 | 451 | 452 | ### Docker container peak memory usage in MB during API request 453 | 454 | The memory usage below are observed during any of the test of a particular API, and average memory usage may be lower. This these numbers may be safer for sizing server requirements. 455 | One thing that stood out to me was that Django REST Framework don't tend to release memory after the request is done, ie: the resting state memory usage appear to be often closer to the peak memory usage even when there is not request being processed. 456 | This could have server sizing implications, because in real world a server would not be handling just one request at a time, and the memory usage would be cumulative. 457 | 458 | | | 100k records | 200k records | 300k records | 400k records | 459 | |------------------------|--------------|--------------|--------------|--------------| 460 | | DRF with Serializer | 430 | 750 | 1060 | 1430 | 461 | | Ninja with Schema | 400 | 780 | 1145 | 1380 | 462 | | DRF without Serializer | 310 | 430 | 515 | 655 | 463 | | Ninja without Schema | 160 | 340 | 380 | 580 | 464 | | Go with mux | 140 | 300 | 500 | 620 | 465 | 466 | ![API memory usage charted](assets/api_container_memory_peak.png) 467 | 468 | Surprisingly, when Ninja is used without the Schema, it is as memory efficient and even more so than Go in some cases. 469 | 470 | 471 | # Conclusions (TLDR) 472 | 473 | ### DRF Serializer and Ninja Schema are very expensive 474 | Serializer from Django REST Framework and Schema from Ninja are Python implementation that helps to convert data and object on a per-record basis, for this reason when dealing with large dataset, they can be very slow. 475 | Personally I think they are great for validating and sanitizing user input data during WRITE operations, but for READ operations, they are not always necessary and can be a performance bottleneck. 476 | 477 | Using them as a way of accessing relations can also introduce N+1 query problem very easily as seen in Part A Chapter 2 example. 478 | 479 | ### Ninja is overall faster and more memory efficient than Django REST Framework 480 | When comparing the APIs that implements DRF Serializer and Ninja Schema, Ninja appears to provide 8-13% performance improvement, while using about the same amount of memory. 481 | When comparing the APIs that do not implement Serializer and Schema, Ninja appears to provide up to 8% performance improvement, while using 10-50% less memory; and is even comparable to API set up in Go in some cases. 482 | 483 | ### Go is the fastest 484 | Being lightweight (no ORM), compiled, and statically typed, Go is the fastest and in most cases the most memory efficient implementation tested. 485 | 486 | However, without many of the tools Django provides out of the box, it can be more difficult to get a project set up and running. 487 | In particular, I miss Django's ORM for writing simple queries, migrations, centralized settings, and ready-to-use authentication systems and various standard middlewares. 488 | If you are interested, I have set up a [template Go web project](https://github.com/oscarychen/eau-de-go) structure that I think is maintainable and scalable, with some of the typical web backend features you would expect from Django. 489 | 490 | # Appendix: test data 491 | 492 | ## 100k records retrieval test 493 | API response times in milliseconds 494 | 495 | | DRF with Serializer | Ninja with Schema | DRF without Serializer | Ninja without Schema | Go with mux | 496 | |---------------------|-------------------|------------------------|----------------------|-------------| 497 | | 3727 | 3531 | 1226 | 972 | 623 | 498 | | 3667 | 3336 | 1205 | 1014 | 442 | 499 | | 3987 | 3697 | 947 | 948 | 549 | 500 | | 3761 | 3326 | 1012 | 897 | 512 | 501 | | 3939 | 3434 | 929 | 913 | 468 | 502 | | 3597 | 3304 | 981 | 961 | 458 | 503 | | 3702 | 3592 | 1134 | 935 | 455 | 504 | | 3872 | 3528 | 950 | 928 | 463 | 505 | | 3606 | 3390 | 1002 | 945 | 639 | 506 | | 3644 | 3409 | 951 | 952 | 429 | 507 | 508 | 509 | ## 200k records retrieval test 510 | API response times in milliseconds 511 | 512 | | DRF with Serializer | Ninja with Schema | DRF without Serializer | Ninja without Schema | Go with mux | 513 | |---------------------|-------------------|------------------------|----------------------|-------------| 514 | | 7913 | 6837 | 1876 | 1802 | 922 | 515 | | 7679 | 6920 | 1916 | 1898 | 941 | 516 | | 8085 | 6753 | 1955 | 1816 | 1058 | 517 | | 7954 | 6679 | 2279 | 1841 | 790 | 518 | | 7418 | 6833 | 1881 | 1821 | 959 | 519 | | 7846 | 6728 | 1916 | 1844 | 811 | 520 | | 7387 | 6588 | 1881 | 1828 | 904 | 521 | | 7921 | 7224 | 1879 | 1821 | 941 | 522 | | 7871 | 6636 | 1821 | 1983 | 872 | 523 | | 8247 | 6840 | 1887 | 1848 | 840 | 524 | 525 | 526 | ## 300k records retrieval test 527 | API response times in milliseconds 528 | 529 | | DRF with Serializer | Ninja with Schema | DRF without Serializer | Ninja without Schema | Go with mux | 530 | |---------------------|-------------------|------------------------|----------------------|-------------| 531 | | 11417 | 10634 | 2984 | 2741 | 1277 | 532 | | 11129 | 9907 | 2942 | 2858 | 1241 | 533 | | 11184 | 10113 | 2957 | 2750 | 1119 | 534 | | 11306 | 10201 | 2924 | 2733 | 1274 | 535 | | 11333 | 9831 | 2954 | 2725 | 1222 | 536 | | 11229 | 10274 | 2973 | 2755 | 1309 | 537 | | 11197 | 10022 | 2915 | 2689 | 1141 | 538 | | 11157 | 10056 | 2924 | 2729 | 1210 | 539 | | 11200 | 10188 | 2996 | 2701 | 1161 | 540 | | 11196 | 10254 | 2896 | 2709 | 1358 | 541 | 542 | 543 | ## 400k records retrieval test 544 | API response times in milliseconds 545 | 546 | | DRF with Serializer | Ninja with Schema | DRF without Serializer | Ninja without Schema | Go with mux | 547 | |---------------------|-------------------|------------------------|----------------------|-------------| 548 | | 15668 | 13421 | 3508 | 3475 | 1824 | 549 | | 15051 | 13210 | 3563 | 3505 | 1808 | 550 | | 14754 | 13586 | 3449 | 3542 | 1624 | 551 | | 14836 | 13280 | 3602 | 3483 | 1643 | 552 | | 14690 | 13352 | 3508 | 3502 | 1787 | 553 | | 14834 | 13382 | 3534 | 3493 | 1654 | 554 | | 15168 | 13356 | 3480 | 3687 | 1714 | 555 | | 15127 | 13142 | 3523 | 3512 | 1784 | 556 | | 14451 | 13232 | 3583 | 3541 | 1644 | 557 | | 14814 | 13132 | 3536 | 3517 | 1676 | 558 | -------------------------------------------------------------------------------- /fast_api/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM python:3.12.8-slim 2 | 3 | ENV PYTHONDONTWRITEBYTECODE 1 4 | ENV PYTHONUNBUFFERED 1 5 | 6 | WORKDIR /app 7 | 8 | # Install system dependencies 9 | RUN apt-get update && apt-get install -y \ 10 | postgresql-client \ 11 | && rm -rf /var/lib/apt/lists/* 12 | 13 | COPY ./fast_api/requirements.txt /app/requirements.txt 14 | 15 | RUN pip install --upgrade pip 16 | RUN pip install --no-cache-dir -r requirements.txt 17 | 18 | COPY ./fast_api/ . 19 | 20 | CMD ["fastapi", "run", "main.py", "--port", "8002"] 21 | -------------------------------------------------------------------------------- /fast_api/apis/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/oscarychen/building-efficient-api/552a263c8eb4bfc5a5c44e21758fae95352c0e70/fast_api/apis/__init__.py -------------------------------------------------------------------------------- /fast_api/apis/cars.py: -------------------------------------------------------------------------------- 1 | from fastapi import APIRouter, Depends 2 | from services.cars import CarService 3 | from typing import Annotated, List 4 | 5 | from database import get_db 6 | from sqlalchemy.ext.asyncio import AsyncSession 7 | 8 | router = APIRouter() 9 | 10 | from datetime import datetime 11 | from pydantic import BaseModel 12 | 13 | 14 | class CarSchema(BaseModel): 15 | id: int 16 | vin: str 17 | owner: str 18 | created_at: datetime 19 | updated_at: datetime 20 | car_model_id: int 21 | car_model_name: str 22 | car_model_year: int 23 | color: str 24 | 25 | 26 | @router.get("/fastapi/", response_model=list[CarSchema]) 27 | async def list_cars(db: Annotated[AsyncSession, Depends(get_db)]): 28 | return await CarService(db).retrieve_all_cars() 29 | -------------------------------------------------------------------------------- /fast_api/database.py: -------------------------------------------------------------------------------- 1 | from contextlib import asynccontextmanager 2 | 3 | from fastapi import FastAPI 4 | from sqlalchemy.ext.asyncio import AsyncSession 5 | from sqlalchemy.ext.asyncio import create_async_engine 6 | from sqlalchemy.ext.declarative import declarative_base 7 | from sqlalchemy.orm import sessionmaker 8 | 9 | Base = declarative_base() 10 | # DATABASE_URL = "postgresql+asyncpg://postgres:postgres@localhost:5432/api_demo_db" 11 | DATABASE_URL = "postgresql+asyncpg://postgres:postgres@db:5432/api_demo_db" 12 | 13 | engine = create_async_engine( 14 | DATABASE_URL, 15 | echo=True, 16 | pool_size=5, 17 | max_overflow=3, 18 | pool_timeout=30, 19 | pool_recycle=1800, 20 | ) 21 | 22 | AsyncSessionLocal = sessionmaker(engine, class_=AsyncSession, expire_on_commit=False) 23 | 24 | 25 | @asynccontextmanager 26 | async def lifespan(app: FastAPI): 27 | # async with engine.begin() as conn: # startup, create tables 28 | # await conn.run_sync(Base.metadata.create_all) 29 | 30 | yield # server is running and handling requests 31 | 32 | await engine.dispose() # server shut down 33 | 34 | 35 | async def get_db() -> AsyncSession: 36 | async with AsyncSessionLocal() as db: 37 | try: 38 | yield db 39 | finally: 40 | await db.close() 41 | -------------------------------------------------------------------------------- /fast_api/database_test.py: -------------------------------------------------------------------------------- 1 | import asyncio 2 | 3 | from sqlalchemy import text 4 | from sqlalchemy.ext.asyncio import create_async_engine 5 | 6 | DATABASE_URL = "postgresql+asyncpg://postgres:postgres@localhost/api_demo_db" 7 | 8 | 9 | async def test_connection(): 10 | engine = create_async_engine(DATABASE_URL, echo=True) 11 | try: 12 | async with engine.connect() as conn: 13 | await conn.execute(text("SELECT 1")) 14 | print("Connection successful") 15 | except Exception as e: 16 | print(f"Connection failed: {e}") 17 | finally: 18 | await engine.dispose() 19 | 20 | 21 | asyncio.run(test_connection()) 22 | -------------------------------------------------------------------------------- /fast_api/main.py: -------------------------------------------------------------------------------- 1 | from fastapi import FastAPI 2 | from fastapi.middleware.cors import CORSMiddleware 3 | from apis.cars import router 4 | from database import lifespan 5 | 6 | app = FastAPI(lifespan=lifespan) 7 | app.add_middleware( 8 | CORSMiddleware, 9 | allow_origins=["localhost", "fastapi"], 10 | allow_methods=["*"], 11 | allow_headers=["*"], 12 | ) 13 | app.include_router(router) 14 | -------------------------------------------------------------------------------- /fast_api/models/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/oscarychen/building-efficient-api/552a263c8eb4bfc5a5c44e21758fae95352c0e70/fast_api/models/__init__.py -------------------------------------------------------------------------------- /fast_api/models/cars.py: -------------------------------------------------------------------------------- 1 | from sqlalchemy import Column, Integer, String, ForeignKey, DateTime 2 | from sqlalchemy.types import DECIMAL 3 | from database import Base 4 | from sqlalchemy.sql import func 5 | 6 | 7 | class CarModel(Base): 8 | __tablename__ = "car_registry_carmodel" 9 | 10 | id = Column(Integer, primary_key=True, index=True) 11 | name = Column(String(length=100)) 12 | make = Column(String(length=100)) 13 | year = Column(Integer) 14 | color = Column(String(length=100)) 15 | price = Column(DECIMAL(precision=10, scale=2)) 16 | created_at = Column(DateTime, default=func.now()) 17 | updated_at = Column(DateTime, default=func.now(), onupdate=func.now()) 18 | 19 | 20 | class Car(Base): 21 | __tablename__ = "car_registry_car" 22 | 23 | id = Column(Integer, primary_key=True, index=True) 24 | vin = Column(String(length=17)) 25 | model_id = Column(Integer, ForeignKey("car_registry_carmodel.id")) 26 | owner = Column(String(length=100)) 27 | created_at = Column(DateTime, default=func.now()) 28 | updated_at = Column(DateTime, default=func.now(), onupdate=func.now()) 29 | -------------------------------------------------------------------------------- /fast_api/requirements.txt: -------------------------------------------------------------------------------- 1 | fastapi[standard]==0.115.6 2 | greenlet==3.1.1 3 | sqlalchemy==2.0.37 4 | asyncpg==0.30.0 5 | psycopg_pool==3.2.4 -------------------------------------------------------------------------------- /fast_api/schemas.py: -------------------------------------------------------------------------------- 1 | from datetime import datetime 2 | from pydantic import BaseModel 3 | 4 | 5 | class CarSchema(BaseModel): 6 | id: int 7 | vin: str 8 | owner: str 9 | created_at: datetime 10 | updated_at: datetime 11 | car_model_id: int 12 | car_model_name: str 13 | car_model_year: int 14 | color: str 15 | -------------------------------------------------------------------------------- /fast_api/services/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/oscarychen/building-efficient-api/552a263c8eb4bfc5a5c44e21758fae95352c0e70/fast_api/services/__init__.py -------------------------------------------------------------------------------- /fast_api/services/cars.py: -------------------------------------------------------------------------------- 1 | from sqlalchemy import select 2 | from sqlalchemy.ext.asyncio import AsyncSession 3 | 4 | from models.cars import Car, CarModel 5 | 6 | 7 | class CarService: 8 | def __init__(self, db: AsyncSession): 9 | self.db = db 10 | 11 | async def retrieve_all_cars(self): 12 | async with self.db.begin(): 13 | statement = select( 14 | Car.id, 15 | Car.vin, 16 | Car.owner, 17 | Car.created_at, 18 | Car.updated_at, 19 | Car.model_id.label('car_model_id'), 20 | CarModel.name.label('car_model_name'), 21 | CarModel.year.label('car_model_year'), 22 | CarModel.color.label('color') 23 | ).join(CarModel, Car.model_id == CarModel.id) 24 | result = await self.db.execute(statement) 25 | return result.all() 26 | -------------------------------------------------------------------------------- /go_sqlc_mux/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM golang:1.23-alpine 2 | 3 | WORKDIR /usr/src/app 4 | 5 | COPY ./go_sqlc_mux /usr/src/app 6 | 7 | RUN go mod download 8 | RUN go build -o main ./cmd/server/main.go 9 | 10 | EXPOSE 8080 11 | CMD ["./main"] -------------------------------------------------------------------------------- /go_sqlc_mux/bin/server: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/oscarychen/building-efficient-api/552a263c8eb4bfc5a5c44e21758fae95352c0e70/go_sqlc_mux/bin/server -------------------------------------------------------------------------------- /go_sqlc_mux/cmd/server/main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "go_sqlc_mux/internal/db" 5 | "go_sqlc_mux/internal/repository" 6 | "go_sqlc_mux/internal/service" 7 | "go_sqlc_mux/internal/transport/http" 8 | "log" 9 | ) 10 | 11 | func Run() error { 12 | log.Println("Setting Up Our APP") 13 | 14 | database, err := db.NewDatabase() 15 | if err != nil { 16 | log.Println("failed to setup connection to the database") 17 | return err 18 | } 19 | log.Println(database) 20 | 21 | // database.Ping(context.Background()) 22 | 23 | queries := repository.New(database.Client) 24 | carRegistryService := service.NewCarRegistryService(queries) 25 | handler := http.NewHandler(carRegistryService) 26 | 27 | if err := handler.Serve(); err != nil { 28 | log.Println("failed to gracefully serve our application") 29 | return err 30 | } 31 | 32 | return nil 33 | } 34 | 35 | func main() { 36 | if err := Run(); err != nil { 37 | log.Println(err) 38 | log.Fatal("Error starting up our REST API") 39 | } 40 | } 41 | -------------------------------------------------------------------------------- /go_sqlc_mux/go.mod: -------------------------------------------------------------------------------- 1 | module go_sqlc_mux 2 | 3 | go 1.23.3 4 | 5 | require ( 6 | github.com/jmoiron/sqlx v1.4.0 7 | github.com/lib/pq v1.10.9 8 | ) 9 | 10 | require github.com/gorilla/mux v1.8.1 // indirect 11 | -------------------------------------------------------------------------------- /go_sqlc_mux/go.sum: -------------------------------------------------------------------------------- 1 | filippo.io/edwards25519 v1.1.0/go.mod h1:BxyFTGdWcka3PhytdK4V28tE5sGfRvvvRV7EaN4VDT4= 2 | github.com/go-sql-driver/mysql v1.8.1/go.mod h1:wEBSXgmK//2ZFJyE+qWnIsVGmvmEKlqwuVSjsCm7DZg= 3 | github.com/gorilla/mux v1.8.1 h1:TuBL49tXwgrFYWhqrNgrUNEY92u81SPhu7sTdzQEiWY= 4 | github.com/gorilla/mux v1.8.1/go.mod h1:AKf9I4AEqPTmMytcMc0KkNouC66V3BtZ4qD5fmWSiMQ= 5 | github.com/jmoiron/sqlx v1.4.0 h1:1PLqN7S1UYp5t4SrVVnt4nUVNemrDAtxlulVe+Qgm3o= 6 | github.com/jmoiron/sqlx v1.4.0/go.mod h1:ZrZ7UsYB/weZdl2Bxg6jCRO9c3YHl8r3ahlKmRT4JLY= 7 | github.com/lib/pq v1.10.9 h1:YXG7RB+JIjhP29X+OtkiDnYaXQwpS4JEWq7dtCCRUEw= 8 | github.com/lib/pq v1.10.9/go.mod h1:AlVN5x4E4T544tWzH6hKfbfQvm3HdbOxrmggDNAPY9o= 9 | github.com/mattn/go-sqlite3 v1.14.22/go.mod h1:Uh1q+B4BYcTPb+yiD3kU8Ct7aC0hY9fxUwlHK0RXw+Y= 10 | -------------------------------------------------------------------------------- /go_sqlc_mux/internal/db/db.go: -------------------------------------------------------------------------------- 1 | package db 2 | 3 | import ( 4 | "context" 5 | "fmt" 6 | "github.com/jmoiron/sqlx" 7 | _ "github.com/lib/pq" 8 | "log" 9 | ) 10 | 11 | type Database struct { 12 | Client *sqlx.DB 13 | } 14 | 15 | // NewDatabase - returns a pointer to a database object 16 | func NewDatabase() (*Database, error) { 17 | log.Println("Setting up new database connection") 18 | 19 | connectionString := "host=db port=5432 sslmode=disable user=postgres dbname=api_demo_db password=postgres" 20 | //connectionString := "host=127.0.0.1 port=5432 sslmode=disable user=postgres dbname=api_demo_db password=postgres" 21 | 22 | db, err := sqlx.Connect("postgres", connectionString) 23 | if err != nil { 24 | return &Database{}, fmt.Errorf("could not connect to database: %w", err) 25 | } 26 | 27 | return &Database{ 28 | Client: db, 29 | }, nil 30 | } 31 | 32 | func (d *Database) Ping(ctx context.Context) error { 33 | return d.Client.DB.PingContext(ctx) 34 | } 35 | -------------------------------------------------------------------------------- /go_sqlc_mux/internal/repository/car_registry.sql.go: -------------------------------------------------------------------------------- 1 | // Code generated by sqlc. DO NOT EDIT. 2 | // versions: 3 | // sqlc v1.25.0 4 | // source: car_registry.sql 5 | 6 | package repository 7 | 8 | import ( 9 | "context" 10 | "time" 11 | ) 12 | 13 | const getAllCars = `-- name: GetAllCars :many 14 | SELECT "car_registry_car"."id", "car_registry_car"."vin", "car_registry_car"."owner", "car_registry_car"."created_at", "car_registry_car"."updated_at", "car_registry_car"."model_id" AS "car_model_id", "car_registry_carmodel"."name" AS "car_model_name", "car_registry_carmodel"."year" AS "car_model_year", "car_registry_carmodel"."color" AS "color" FROM "car_registry_car" INNER JOIN "car_registry_carmodel" ON ("car_registry_car"."model_id" = "car_registry_carmodel"."id") 15 | ` 16 | 17 | type GetAllCarsRow struct { 18 | ID int64 `json:"id"` 19 | Vin string `json:"vin"` 20 | Owner string `json:"owner"` 21 | CreatedAt time.Time `json:"created_at"` 22 | UpdatedAt time.Time `json:"updated_at"` 23 | CarModelID int64 `json:"car_model_id"` 24 | CarModelName string `json:"car_model_name"` 25 | CarModelYear int32 `json:"car_model_year"` 26 | Color string `json:"color"` 27 | } 28 | 29 | func (q *Queries) GetAllCars(ctx context.Context) ([]GetAllCarsRow, error) { 30 | rows, err := q.db.QueryContext(ctx, getAllCars) 31 | if err != nil { 32 | return nil, err 33 | } 34 | defer rows.Close() 35 | var items []GetAllCarsRow 36 | for rows.Next() { 37 | var i GetAllCarsRow 38 | if err := rows.Scan( 39 | &i.ID, 40 | &i.Vin, 41 | &i.Owner, 42 | &i.CreatedAt, 43 | &i.UpdatedAt, 44 | &i.CarModelID, 45 | &i.CarModelName, 46 | &i.CarModelYear, 47 | &i.Color, 48 | ); err != nil { 49 | return nil, err 50 | } 51 | items = append(items, i) 52 | } 53 | if err := rows.Close(); err != nil { 54 | return nil, err 55 | } 56 | if err := rows.Err(); err != nil { 57 | return nil, err 58 | } 59 | return items, nil 60 | } 61 | -------------------------------------------------------------------------------- /go_sqlc_mux/internal/repository/db.go: -------------------------------------------------------------------------------- 1 | // Code generated by sqlc. DO NOT EDIT. 2 | // versions: 3 | // sqlc v1.25.0 4 | 5 | package repository 6 | 7 | import ( 8 | "context" 9 | "database/sql" 10 | ) 11 | 12 | type DBTX interface { 13 | ExecContext(context.Context, string, ...interface{}) (sql.Result, error) 14 | PrepareContext(context.Context, string) (*sql.Stmt, error) 15 | QueryContext(context.Context, string, ...interface{}) (*sql.Rows, error) 16 | QueryRowContext(context.Context, string, ...interface{}) *sql.Row 17 | } 18 | 19 | func New(db DBTX) *Queries { 20 | return &Queries{db: db} 21 | } 22 | 23 | type Queries struct { 24 | db DBTX 25 | } 26 | 27 | func (q *Queries) WithTx(tx *sql.Tx) *Queries { 28 | return &Queries{ 29 | db: tx, 30 | } 31 | } 32 | -------------------------------------------------------------------------------- /go_sqlc_mux/internal/repository/models.go: -------------------------------------------------------------------------------- 1 | // Code generated by sqlc. DO NOT EDIT. 2 | // versions: 3 | // sqlc v1.25.0 4 | 5 | package repository 6 | 7 | import ( 8 | "time" 9 | ) 10 | 11 | type CarRegistryCar struct { 12 | ID int64 `json:"id"` 13 | Vin string `json:"vin"` 14 | Owner string `json:"owner"` 15 | CreatedAt time.Time `json:"created_at"` 16 | UpdatedAt time.Time `json:"updated_at"` 17 | ModelID int64 `json:"model_id"` 18 | } 19 | 20 | type CarRegistryCarmodel struct { 21 | ID int64 `json:"id"` 22 | Name string `json:"name"` 23 | Make string `json:"make"` 24 | Year int32 `json:"year"` 25 | Color string `json:"color"` 26 | Price string `json:"price"` 27 | CreatedAt time.Time `json:"created_at"` 28 | UpdatedAt time.Time `json:"updated_at"` 29 | } 30 | -------------------------------------------------------------------------------- /go_sqlc_mux/internal/service/car_registry.go: -------------------------------------------------------------------------------- 1 | package service 2 | 3 | import ( 4 | "context" 5 | "go_sqlc_mux/internal/repository" 6 | ) 7 | 8 | type CarRegistryStore interface { 9 | GetAllCars(ctx context.Context) ([]repository.GetAllCarsRow, error) 10 | } 11 | 12 | type CarRegistryService struct { 13 | CarRegistryStore CarRegistryStore 14 | } 15 | 16 | func NewCarRegistryService(carRegistryStore CarRegistryStore) *CarRegistryService { 17 | return &CarRegistryService{ 18 | CarRegistryStore: carRegistryStore, 19 | } 20 | } 21 | 22 | func (service *CarRegistryService) GetAllCars(ctx context.Context) ([]repository.GetAllCarsRow, error) { 23 | cars, err := service.CarRegistryStore.GetAllCars(ctx) 24 | if err != nil { 25 | return []repository.GetAllCarsRow{}, err 26 | } 27 | return cars, nil 28 | } 29 | -------------------------------------------------------------------------------- /go_sqlc_mux/internal/transport/http/car_registry.go: -------------------------------------------------------------------------------- 1 | package http 2 | 3 | import ( 4 | "context" 5 | "encoding/json" 6 | "go_sqlc_mux/internal/repository" 7 | "net/http" 8 | ) 9 | 10 | type CarRegistryService interface { 11 | GetAllCars(ctx context.Context) ([]repository.GetAllCarsRow, error) 12 | } 13 | 14 | func (h *Handler) GetAllCars(w http.ResponseWriter, r *http.Request) { 15 | cars, err := h.CarRegistryService.GetAllCars(r.Context()) 16 | if err != nil { 17 | http.Error(w, err.Error(), http.StatusBadRequest) 18 | return 19 | } 20 | 21 | jsonData, err := json.Marshal(cars) 22 | if err != nil { 23 | http.Error(w, err.Error(), http.StatusBadRequest) 24 | return 25 | } 26 | 27 | _, err = w.Write(jsonData) 28 | if err != nil { 29 | http.Error(w, err.Error(), http.StatusBadRequest) 30 | return 31 | } 32 | } 33 | -------------------------------------------------------------------------------- /go_sqlc_mux/internal/transport/http/handler.go: -------------------------------------------------------------------------------- 1 | package http 2 | 3 | import ( 4 | "context" 5 | "github.com/gorilla/mux" 6 | "log" 7 | "net/http" 8 | "os" 9 | "os/signal" 10 | "time" 11 | ) 12 | 13 | type Handler struct { 14 | Router *mux.Router 15 | ProtectedRouter *mux.Router 16 | CarRegistryService CarRegistryService 17 | Server *http.Server 18 | } 19 | 20 | func NewHandler(carRegistryService CarRegistryService) *Handler { 21 | h := &Handler{ 22 | CarRegistryService: carRegistryService, 23 | } 24 | h.Router = mux.NewRouter() 25 | 26 | h.mapRoutes() 27 | //h.Router.Use(middleware.JSONMiddleware) 28 | 29 | h.Server = &http.Server{ 30 | Addr: "0.0.0.0:8003", 31 | Handler: h.Router, 32 | } 33 | return h 34 | } 35 | 36 | func (h *Handler) mapRoutes() { 37 | 38 | h.Router.HandleFunc("/go/", h.GetAllCars).Methods("GET") 39 | } 40 | 41 | func (h *Handler) Serve() error { 42 | go func() { 43 | if err := h.Server.ListenAndServe(); err != nil { 44 | log.Println(err.Error()) 45 | } 46 | }() 47 | 48 | c := make(chan os.Signal, 1) 49 | signal.Notify(c, os.Interrupt) 50 | <-c 51 | 52 | ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second) 53 | defer cancel() 54 | h.Server.Shutdown(ctx) 55 | 56 | log.Println("shut down gracefully") 57 | return nil 58 | } 59 | -------------------------------------------------------------------------------- /go_sqlc_mux/schemas/0001_initial.down.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/oscarychen/building-efficient-api/552a263c8eb4bfc5a5c44e21758fae95352c0e70/go_sqlc_mux/schemas/0001_initial.down.sql -------------------------------------------------------------------------------- /go_sqlc_mux/schemas/0001_initial.up.sql: -------------------------------------------------------------------------------- 1 | BEGIN; 2 | -- 3 | -- Create model CarModel 4 | -- 5 | CREATE TABLE "car_registry_carmodel" ("id" bigint NOT NULL PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY, "name" varchar(100) NOT NULL, "make" varchar(100) NOT NULL, "year" integer NOT NULL, "color" varchar(100) NOT NULL, "price" numeric(10, 2) NOT NULL, "created_at" timestamp with time zone NOT NULL, "updated_at" timestamp with time zone NOT NULL); 6 | -- 7 | -- Create model Car 8 | -- 9 | CREATE TABLE "car_registry_car" ("id" bigint NOT NULL PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY, "vin" varchar(17) NOT NULL, "owner" varchar(100) NOT NULL, "created_at" timestamp with time zone NOT NULL, "updated_at" timestamp with time zone NOT NULL, "model_id" bigint NOT NULL); 10 | ALTER TABLE "car_registry_car" ADD CONSTRAINT "car_registry_car_model_id_e7a8fb5f_fk_car_registry_carmodel_id" FOREIGN KEY ("model_id") REFERENCES "car_registry_carmodel" ("id") DEFERRABLE INITIALLY DEFERRED; 11 | CREATE INDEX "car_registry_car_model_id_e7a8fb5f" ON "car_registry_car" ("model_id"); 12 | COMMIT; 13 | -------------------------------------------------------------------------------- /go_sqlc_mux/sqlc.yaml: -------------------------------------------------------------------------------- 1 | version: "2" 2 | sql: 3 | - engine: "postgresql" 4 | queries: "sqlc/queries/" 5 | schema: "schemas/" 6 | gen: 7 | go: 8 | out: "internal/repository/" 9 | emit_json_tags: true 10 | 11 | -------------------------------------------------------------------------------- /go_sqlc_mux/sqlc/queries/car_registry.sql: -------------------------------------------------------------------------------- 1 | -- name: GetAllCars :many 2 | SELECT "car_registry_car"."id", "car_registry_car"."vin", "car_registry_car"."owner", "car_registry_car"."created_at", "car_registry_car"."updated_at", "car_registry_car"."model_id" AS "car_model_id", "car_registry_carmodel"."name" AS "car_model_name", "car_registry_carmodel"."year" AS "car_model_year", "car_registry_carmodel"."color" AS "color" FROM "car_registry_car" INNER JOIN "car_registry_carmodel" ON ("car_registry_car"."model_id" = "car_registry_carmodel"."id"); -------------------------------------------------------------------------------- /load_testing/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM python:3.12.8-slim 2 | 3 | WORKDIR /usr/src/app 4 | 5 | RUN pip install --upgrade pip 6 | RUN pip install locust --no-cache-dir 7 | 8 | COPY ./load_testing/ . 9 | 10 | EXPOSE 8089 11 | 12 | CMD ["locust", "--config", "api_response_times.conf"] -------------------------------------------------------------------------------- /load_testing/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/oscarychen/building-efficient-api/552a263c8eb4bfc5a5c44e21758fae95352c0e70/load_testing/__init__.py -------------------------------------------------------------------------------- /load_testing/api_response_times.conf: -------------------------------------------------------------------------------- 1 | locustfile = locustfile.py 2 | host = '' 3 | users = 100 4 | spawn-rate = 10 5 | run-time = 300s -------------------------------------------------------------------------------- /load_testing/locustfile.py: -------------------------------------------------------------------------------- 1 | 2 | import threading 3 | from locust import HttpUser, task, between, FastHttpUser, tag 4 | 5 | # Create a global lock object 6 | task_lock = threading.Lock() 7 | 8 | class ApiLoadTest(FastHttpUser): 9 | wait_time = between(1, 3) # Time between tasks (1 to 3 seconds) 10 | host = "http://localhost" 11 | 12 | @tag("drf_with_serializer") 13 | @task(1) 14 | def test_drf_with_serializer(self): 15 | with task_lock: # Only one task can run at a time due to this lock 16 | self.client.get("http://django-drf:8000/drf/with-serializer/") 17 | 18 | @tag("drf_without_serializer") 19 | @task(1) 20 | def test_drf_without_serializer(self): 21 | with task_lock: 22 | self.client.get("http://django-drf:8000/drf/without-serializer/") 23 | 24 | @tag("ninja_with_schema") 25 | @task(1) 26 | def test_ninja_with_schema(self): 27 | with task_lock: 28 | self.client.get("http://django-ninja:8001/ninja/with-schema/") 29 | 30 | @tag("ninja_without_schema") 31 | @task(1) 32 | def test_ninja_without_schema(self): 33 | with task_lock: 34 | self.client.get("http://django-ninja:8001/ninja/without-schema/") 35 | 36 | @tag("fastapi_with_pydantic") 37 | @task(1) 38 | def test_fastapi_with_pydantic(self): 39 | with task_lock: 40 | self.client.get("http://fastapi:8002/fastapi/") 41 | 42 | @tag("go") 43 | @task(1) 44 | def test_go(self): 45 | with task_lock: 46 | self.client.get("http://go-sqlc-mux:8003/go/") -------------------------------------------------------------------------------- /makefile: -------------------------------------------------------------------------------- 1 | SHELL := /bin/bash 2 | 3 | #!make 4 | ifneq (,$(wildcard ./.env)) 5 | include .env 6 | export 7 | endif 8 | 9 | docker-build: 10 | docker-compose build --no-cache 11 | 12 | docker-up: 13 | docker-compose up -d 14 | 15 | docker-up-drf: 16 | docker-compose up -d django-drf 17 | 18 | docker-up-ninja: 19 | docker-compose up -d django-ninja 20 | 21 | docker-up-fastapi: 22 | docker-compose up -d fastapi 23 | 24 | docker-up-go: 25 | docker-compose up -d go-sqlc-mux 26 | 27 | docker-down: 28 | docker-compose down 29 | 30 | docker-clean: 31 | @docker stop $$(docker ps -aq) || true 32 | @docker rm $$(docker ps -aq) || true 33 | @docker volume rm $$(docker volume ls -q) || true 34 | @docker rmi $$(docker images -q) || true 35 | @docker network rm $$(docker network ls -q) || true 36 | @docker system prune -a --volumes || true 37 | 38 | 39 | django-drf-migrate: 40 | docker exec -it api-demo-django-drf python manage.py migrate 41 | 42 | django-drf-populate: 43 | docker exec -it api-demo-django-drf python manage.py populate -------------------------------------------------------------------------------- /readme.md: -------------------------------------------------------------------------------- 1 | # API framework serialization performance 2 | 3 | ## Introduction 4 | This project is to compare the performance of serialization between different API frameworks. 5 | The frameworks that are compared are: 6 | - Django REST Framework 7 | - Django Ninja 8 | - FastAPI with Pydantic 9 | - Golang with sqlc 10 | 11 | Most of these are Python framworks, except Golang with sqlc. 12 | Using Golang with sqlc is a relatively barebones approach to building an API, it does not involve an ORM, 13 | and database records are serialized into native Go structs. 14 | It is expected to be fast, 15 | but I want to see just how close we can get to it from any of the Python frameworks listed above. 16 | 17 | Django REST Framework is a heavy weight full-featured framework, there is a ton of tuning that could be done to it, 18 | I wrote a step-by-step changes I have done to the Django REST Framework and Django Ninja examples to make them faster 19 | in the [related documentation](docs/readme.md). 20 | 21 | ## Methodology 22 | A postgres database is used to store the data and shared between all the frameworks. 23 | The database is dockerized, as well as each of the frameworks. 24 | A dockerized Locust API load testing tool is also included to run the tests. 25 | To ensure similar resource is available to each docker container during the tests, each Locust test is run sequentially, 26 | so when one framework is handling a request, the others are idle. 27 | 28 | ## Quick Start 29 | A set of makefile commands are provided: 30 | ```bash 31 | make docker-build // build the docker images 32 | make docker-up // start the docker containers 33 | make django-drf-migrate // using the Django REST Framework container to run set up database schema 34 | make django-drf-populate // using the Django REST Framework container to populate the database with 100k records 35 | make docker-down // stop the docker containers 36 | ``` 37 | Locust testing interface will be available on http://localhost:8089 38 | 39 | I also included an api.http file so you can manually test the API client that is part of the JetBrains IDEs, 40 | or if you are using VS Code, the HttpYAC extension can be used. 41 | 42 | ## Results 43 | ![](assets/locust_api_response_times.png) 44 | --------------------------------------------------------------------------------