├── .gitignore ├── .pre-commit-config.yaml ├── README.rst ├── setup.cfg ├── setup.py └── yamdl ├── __init__.py ├── apps.py ├── loader.py └── router.py /.gitignore: -------------------------------------------------------------------------------- 1 | *.egg-info 2 | *.pyc 3 | /build/ 4 | /dist/ 5 | __pycache__ 6 | -------------------------------------------------------------------------------- /.pre-commit-config.yaml: -------------------------------------------------------------------------------- 1 | repos: 2 | - repo: https://github.com/pre-commit/pre-commit-hooks 3 | rev: v3.3.0 4 | hooks: 5 | - id: check-case-conflict 6 | - id: check-merge-conflict 7 | - id: check-yaml 8 | - id: end-of-file-fixer 9 | - id: file-contents-sorter 10 | args: ["--ignore-case"] 11 | files: "^.gitignore$" 12 | - id: mixed-line-ending 13 | args: ["--fix=lf"] 14 | - id: trailing-whitespace 15 | - id: pretty-format-json 16 | 17 | - repo: https://github.com/psf/black 18 | rev: 22.10.0 19 | hooks: 20 | - id: black 21 | args: ["--target-version=py37"] 22 | 23 | - repo: https://github.com/pycqa/isort 24 | rev: 5.10.1 25 | hooks: 26 | - id: isort 27 | args: ["--profile=black"] 28 | 29 | - repo: https://gitlab.com/pycqa/flake8 30 | rev: 5.0.4 31 | hooks: 32 | - id: flake8 33 | 34 | - repo: https://github.com/pre-commit/mirrors-mypy 35 | rev: v0.982 36 | hooks: 37 | - id: mypy 38 | additional_dependencies: 39 | - "types-pyyaml" 40 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | yamdl 2 | ===== 3 | 4 | .. image:: https://img.shields.io/pypi/v/yamdl.svg 5 | :target: https://pypi.python.org/pypi/yamdl 6 | 7 | .. image:: https://img.shields.io/pypi/l/yamdl.svg 8 | :target: https://pypi.python.org/pypi/yamdl 9 | 10 | Lets you store instances of Django models as flat files (simplified fixtures). 11 | For when you want to store content in a Git repo, but still want to be able to 12 | use the normal Django ORM methods and shortcut functions. 13 | 14 | It works by loading the data into an in-memory SQLite database on startup, and 15 | then serving queries from there. This means it adds a little time to your app's 16 | boot, but versus static files, it still lets you write queries and have dynamic 17 | views, while all being incredibly fast as queries return in microseconds. 18 | 19 | It does not persist changes to the models back into files - this is purely for 20 | authoring content in a text editor and using it via Django. 21 | 22 | 23 | Why not use normal fixtures? 24 | ---------------------------- 25 | 26 | They're not only a little verbose, but they need to be loaded into a non-memory 27 | database (slower) and you need lots of logic to work out if you should update 28 | or delete existing entries. 29 | 30 | Installation 31 | ------------ 32 | 33 | First, install the package:: 34 | 35 | pip install yamdl 36 | 37 | Then, add it to ``INSTALLED_APPS``:: 38 | 39 | INSTALLED_APPS = [ 40 | ... 41 | 'yamdl', 42 | ... 43 | ] 44 | 45 | Then, add the in-memory database to ``DATABASES``:: 46 | 47 | DATABASES = { 48 | ... 49 | 'yamdl': { 50 | 'ENGINE': 'django.db.backends.sqlite3', 51 | 'NAME': 'file:yamdl-db?mode=memory&cache=shared', 52 | } 53 | } 54 | 55 | Then, add a ``YAMDL_DIRECTORIES`` setting which defines where your directories 56 | of YAML files can be found (it's a list):: 57 | 58 | YAMDL_DIRECTORIES = [ 59 | os.path.join(PROJECT_DIR, "content"), 60 | ] 61 | 62 | Finally, add the database router:: 63 | 64 | DATABASE_ROUTERS = [ 65 | "yamdl.router.YamdlRouter", 66 | ] 67 | 68 | 69 | Usage 70 | ----- 71 | 72 | First, add the ``__yamdl__`` attribute to the models you want to use static 73 | content. A model can only be static or dynamic, not both:: 74 | 75 | class MyModel(models.Model): 76 | ... 77 | __yamdl__ = True 78 | 79 | Then, start making static files under one of the directories you listed in the 80 | ``YAMDL_DIRECTORIES`` setting above. Within one of these, make a directory with 81 | the format ``appname.modelname``, and then YAML files ending in ``.yaml``:: 82 | 83 | andrew-site/ 84 | content/ 85 | speaking.Talk/ 86 | 2017.yaml 87 | 2016.yaml 88 | 89 | You can override the expected directory name by setting to the expected 90 | directory name instead:: 91 | 92 | class MyModel(models.Model): 93 | ... 94 | __yamdl__ = True 95 | __yamdl_directory__ = "talks" 96 | 97 | Within those YAML files, you can define either a list of model instances, like 98 | this:: 99 | 100 | - title: 'Alabama' 101 | section: us-states 102 | 103 | - title: 'Alaska' 104 | section: us-states 105 | done: 2016-11-18 106 | place_name: Fairbanks 107 | 108 | - title: 'Arizona' 109 | section: us-states 110 | done: 2016-05-20 111 | place_name: Flagstaff 112 | 113 | Or a single model instance at the top level, like this:: 114 | 115 | conference: DjangoCon AU 116 | title: Horrors of Distributed Systems 117 | when: 2017-08-04 118 | description: Stepping through some of the myriad ways in which systems can fail that programmers don't expect, and how this hostile environment affects the design of distributed systems. 119 | city: Melbourne 120 | country: AU 121 | slides_url: https://speakerdeck.com/andrewgodwin/horrors-of-distributed-systems 122 | video_url: https://www.youtube.com/watch?v=jx1Hkxe64Xs 123 | 124 | You can also define a Markdown document (ending in ``.md``) below a document 125 | separator, and it will be loaded into the column called ``content``:: 126 | 127 | date: 2022-01-18 21:00:00+00:00 128 | image: blog/2022/241.jpg 129 | image_expand: true 130 | section: van-build 131 | slug: planning-a-van 132 | title: Planning A Van 133 | 134 | --- 135 | 136 | What's In A Van? 137 | ---------------- 138 | 139 | So, I have decided to embark on my biggest project to date (and probably for a while, unless I finally get somewhere to build a cabin) - building myself a camper van, from scratch. Well, from an empty cargo van, anyway. 140 | 141 | Files can be nested at any level under their model directory, so you can group 142 | the files together in directories (for example, blog posts by year) if you 143 | want. 144 | 145 | The files are also added to the Django autoreloader, so if you are using the 146 | development server, it will reload as you edit the files (so you can see 147 | changes reflected live - they are only read on server start). 148 | 149 | To customize how content files are loaded, you can set ``YAMDL_LOADER`` to a subclass of ``yamdl.loader.ModelLoader``, which will be imported and used instead. 150 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [metadata] 2 | name = yamdl 3 | version = attr: yamdl.__version__ 4 | url = https://github.com/andrewgodwin/yamdl/ 5 | author = Andrew Godwin 6 | description = Flat-file model instances for Django 7 | long_description = file: README.rst 8 | license = BSD 9 | classifiers = 10 | Development Status :: 5 - Production/Stable 11 | Environment :: Web Environment 12 | Framework :: Django 13 | Intended Audience :: Developers 14 | License :: OSI Approved :: BSD License 15 | Operating System :: OS Independent 16 | Programming Language :: Python :: 3 17 | Topic :: Internet :: WWW/HTTP 18 | 19 | [options] 20 | include_package_data = True 21 | packages = find: 22 | install_requires = 23 | Django>=3.2 24 | pyyaml~=6.0 25 | python_requires = >= 3.7 26 | 27 | [flake8] 28 | exclude = venv/*,tox/*,specs/* 29 | ignore = E123,E128,E266,E402,F405,E501,W503,E731,W601 30 | max-line-length = 119 31 | 32 | [isort] 33 | profile = black 34 | multi_line_output = 3 35 | 36 | [mypy] 37 | warn_unused_ignores = True 38 | 39 | [mypy-airflow.*] 40 | ignore_missing_imports = True 41 | 42 | [mypy-dateutil.tz] 43 | ignore_missing_imports = True 44 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from setuptools import setup 2 | 3 | setup() 4 | -------------------------------------------------------------------------------- /yamdl/__init__.py: -------------------------------------------------------------------------------- 1 | __version__ = "1.0.0" 2 | default_app_config = "yamdl.apps.YamdlConfig" 3 | -------------------------------------------------------------------------------- /yamdl/apps.py: -------------------------------------------------------------------------------- 1 | import os 2 | import warnings 3 | 4 | from django.apps import AppConfig 5 | from django.conf import settings 6 | from django.core.exceptions import ImproperlyConfigured 7 | from django.db import connections 8 | from django.utils.autoreload import autoreload_started 9 | from django.utils.module_loading import import_string 10 | 11 | from .loader import ModelLoader 12 | 13 | 14 | class YamdlConfig(AppConfig): 15 | 16 | name = "yamdl" 17 | loaded = False 18 | 19 | def ready(self): 20 | """ 21 | Startup and signal handling code 22 | """ 23 | if not self.loaded: 24 | # Verify we have a db alias to write to. 25 | self.db_alias = getattr(settings, "YAMDL_DATABASE_ALIAS", "yamdl") 26 | if self.db_alias not in connections: 27 | raise ImproperlyConfigured( 28 | "No database is set up for Yamdl (expecting alias %s).\n" 29 | "You should either set YAMDL_DATABASE_ALIAS or add a DATABASES entry." 30 | % self.db_alias 31 | ) 32 | self.connection = connections[self.db_alias] 33 | # Make sure it's shared in-memory SQLite 34 | if self.connection.vendor != "sqlite": 35 | raise ImproperlyConfigured( 36 | "The Yamdl database must be shared in-memory SQLite" 37 | ) 38 | if not self.connection.is_in_memory_db(): 39 | raise ImproperlyConfigured( 40 | "The Yamdl database must be shared in-memory SQLite" 41 | ) 42 | if "mode=memory" not in self.connection.settings_dict.get("NAME", ""): 43 | raise ImproperlyConfigured( 44 | "The Yamdl database must be shared in-memory SQLite" 45 | ) 46 | # Check the directory settings 47 | if not hasattr(settings, "YAMDL_DIRECTORIES"): 48 | raise ImproperlyConfigured( 49 | "You have not set YAMDL_DIRECTORIES to a list of directories to scan" 50 | ) 51 | for directory in settings.YAMDL_DIRECTORIES: 52 | if not os.path.isdir(directory): 53 | raise ImproperlyConfigured( 54 | "Yamdl directory %s does not exist" % directory 55 | ) 56 | 57 | self.loader_class = import_string(getattr(settings, "YAMDL_LOADER", "yamdl.loader.ModelLoader")) 58 | if not issubclass(self.loader_class, ModelLoader): 59 | raise ImproperlyConfigured( 60 | "YAMDL_LOADER must be an instance of ModelLoader" 61 | ) 62 | 63 | self.loader = self.loader_class(self.connection, settings.YAMDL_DIRECTORIES) 64 | with warnings.catch_warnings(): 65 | # Django doesn't like running DB queries during app initialization 66 | warnings.filterwarnings("ignore", module="django.db", category=RuntimeWarning) 67 | 68 | # Read the fixtures into memory 69 | self.loader.load() 70 | 71 | self.loaded = True 72 | 73 | # Add autoreload watches for our files 74 | autoreload_started.connect(self.autoreload_ready) 75 | 76 | def autoreload_ready(self, sender, **kwargs): 77 | for directory in settings.YAMDL_DIRECTORIES: 78 | for ext in self.loader.EXT_MARKDOWN: 79 | sender.watch_dir(directory, f"**/*{ext}") 80 | for ext in self.loader.EXT_YAML: 81 | sender.watch_dir(directory, f"**/*{ext}") 82 | -------------------------------------------------------------------------------- /yamdl/loader.py: -------------------------------------------------------------------------------- 1 | import logging 2 | from pathlib import Path 3 | 4 | import yaml 5 | from django.apps import apps 6 | from django.db import transaction 7 | 8 | 9 | logger = logging.getLogger(__name__) 10 | 11 | 12 | class ModelLoader(object): 13 | """ 14 | Reads model instances from disk as .yaml files and loads them into 15 | a database (designed for an in-memory one due to PK removal etc.) 16 | """ 17 | 18 | EXT_MARKDOWN = [".md", ".markdown"] 19 | EXT_YAML = [".yml", ".yaml"] 20 | 21 | def __init__(self, connection, directories): 22 | self.connection = connection 23 | self.directories = directories 24 | 25 | def get_content_field(self, model_class): 26 | """ 27 | Determine which field is used to store file content 28 | """ 29 | return "content" 30 | 31 | def load(self): 32 | """ 33 | Loads everything 34 | """ 35 | self.loaded = 0 36 | # Discover models we manage and load their schema into memory 37 | self.load_schema() 38 | 39 | with transaction.atomic(using=self.connection.alias): 40 | # Scan content directories 41 | for directory in self.directories: 42 | # First level should be folders named after models 43 | for model_folder in Path(directory).iterdir(): 44 | if ( 45 | model_folder.is_dir() 46 | and model_folder.name in self.managed_directories 47 | ): 48 | model = self.managed_directories[model_folder.name] 49 | # Second level should be files, or more folders 50 | self.load_folder_files(model._meta.label_lower, model_folder) 51 | logger.info("Loaded %d yamdl fixtures.", self.loaded) 52 | 53 | def load_folder_files(self, model_name, folder_path: Path): 54 | """ 55 | Loads a folder full of either fixtures or other files 56 | """ 57 | model_class = self.get_model_class(model_name) 58 | 59 | for filename in folder_path.iterdir(): 60 | if filename.is_dir(): 61 | self.load_folder_files(model_name, filename) 62 | elif filename.suffix in self.EXT_YAML and filename.is_file(): 63 | self.load_yaml_file(model_class, filename) 64 | elif filename.suffix in self.EXT_MARKDOWN and filename.is_file(): 65 | self.load_markdown_file(model_class, filename) 66 | 67 | def get_model_class(self, model_name): 68 | # Make sure it's for a valid model 69 | try: 70 | return self.managed_models[model_name] 71 | except KeyError: 72 | raise ValueError( 73 | "Cannot load yamdl fixture - the model name %s is not managed." 74 | % (model_name,) 75 | ) 76 | 77 | def load_yaml_file(self, model_class, file_path: Path): 78 | """ 79 | Loads a single file of fixtures. 80 | """ 81 | # Read it into memory 82 | with file_path.open(mode="r", encoding="utf8") as fh: 83 | fixture_data = yaml.safe_load(fh) 84 | # Write it into our fixtures storage 85 | if isinstance(fixture_data, list): 86 | for fixture in fixture_data: 87 | self.load_fixture(model_class, fixture, file_path) 88 | elif isinstance(fixture_data, dict): 89 | self.load_fixture(model_class, fixture_data, file_path) 90 | else: 91 | raise ValueError( 92 | "Cannot load yamdl fixture %s - not a dict or list." % file_path 93 | ) 94 | 95 | def load_markdown_file(self, model_class, file_path: Path): 96 | """ 97 | Loads a markdown-hybrid file (yaml, then ---, then markdown). 98 | """ 99 | with file_path.open(mode="r", encoding="utf8") as fh: 100 | # Read line by line until we hit the document separator 101 | yaml_data = "" 102 | for line in fh: 103 | if line.strip() == "---": 104 | if not yaml_data: 105 | # File started with "---" 106 | continue 107 | break 108 | else: 109 | yaml_data += line 110 | else: 111 | if not yaml_data: 112 | # Empty file. 113 | return 114 | raise ValueError( 115 | f"Markdown hybrid file {file_path} contains no document separator (---)" 116 | ) 117 | fixture_data = yaml.safe_load(yaml_data) 118 | if not isinstance(fixture_data, dict): 119 | _type = type(fixture_data).__name__ 120 | raise ValueError(f"Markdown hybrid header is not a YAML dict, but {_type}") 121 | # The rest goes into "content" 122 | fixture_data[self.get_content_field(model_class)] = fh.read() 123 | self.load_fixture(model_class, fixture_data, file_path) 124 | 125 | def load_fixture(self, model_class, data, file_path: Path): 126 | """ 127 | Loads a single fixture from a dict object. 128 | """ 129 | # Make an instance of the model, then save it 130 | if hasattr(model_class, "from_yaml"): 131 | model_class.from_yaml(**data) 132 | else: 133 | model_class.objects.create(**data) 134 | self.loaded += 1 135 | 136 | def load_schema(self): 137 | """ 138 | Works out which models are marked as managed by us, and writes a schema 139 | for them into the database. 140 | """ 141 | # Go through and collect the models 142 | self.managed_models = {} 143 | self.managed_directories = {} 144 | for app in apps.get_app_configs(): 145 | for model in app.get_models(): 146 | if getattr(model, "__yamdl__", False): 147 | directory = getattr(model, "__yamdl_directory__", model._meta.label) 148 | self.managed_models[model._meta.label_lower] = model 149 | self.managed_directories[directory] = model 150 | # Make the tables in the database 151 | with self.connection.schema_editor() as editor: 152 | for model in self.managed_models.values(): 153 | editor.create_model(model) 154 | logger.info("Created yamdl schema for %s" % (", ".join(self.managed_models.keys()),)) 155 | -------------------------------------------------------------------------------- /yamdl/router.py: -------------------------------------------------------------------------------- 1 | from django.apps import apps 2 | from django.conf import settings 3 | 4 | 5 | class YamdlRouter(object): 6 | """ 7 | Database router that intercepts models marked as being managed by us. 8 | """ 9 | 10 | DEFAULT_DB_ALIAS = "yamdl" 11 | 12 | def __init__(self): 13 | self.yamdl_app = apps.get_app_config("yamdl") 14 | 15 | def _is_yamdl(self, obj): 16 | return getattr(obj, "__yamdl__", False) 17 | 18 | def db_for_read(self, model, **hints): 19 | if self._is_yamdl(model): 20 | return getattr(settings, "YAMDL_DATABASE_ALIAS", self.DEFAULT_DB_ALIAS) 21 | 22 | def db_for_write(self, model, **hints): 23 | if self._is_yamdl(model): 24 | if self.yamdl_app.loaded: 25 | raise RuntimeError("You cannot write to Yamdl-backed models") 26 | else: 27 | return getattr(settings, "YAMDL_DATABASE_ALIAS", self.DEFAULT_DB_ALIAS) 28 | --------------------------------------------------------------------------------