├── docs ├── changelog.rst ├── favicon.ico ├── contributing.rst ├── favicon-16x16.png ├── favicon-32x32.png ├── piny_logo_border.png ├── piny_logo_noborder.png ├── requirements.txt ├── code │ ├── strict_matcher.py │ ├── config.yaml │ ├── simple_yaml_loader.py │ ├── trafaret_validation.py │ ├── ma_validation.py │ ├── pydantic_validation.py │ └── flask_integration.py ├── misc.rst ├── Makefile ├── install.rst ├── best.rst ├── integration.rst ├── index.rst ├── conf.py └── usage.rst ├── tests ├── configs │ ├── db.yaml │ ├── mail.yaml │ └── defaults.yaml ├── conftest.py ├── __init__.py ├── test_errors.py ├── test_cli.py ├── test_loaders.py └── test_validators.py ├── MANIFEST.in ├── requirements.txt ├── .gitignore ├── .github ├── dependabot.yml ├── ISSUE_TEMPLATE │ ├── issue.md │ └── feature_request.md ├── PULL_REQUEST_TEMPLATE.md └── workflows │ └── ci.yml ├── src └── piny │ ├── __init__.py │ ├── errors.py │ ├── cli.py │ ├── matchers.py │ ├── validators.py │ └── loaders.py ├── .readthedocs.yml ├── LICENSE ├── pyproject.toml ├── Makefile ├── CONTRIBUTING.rst ├── CHANGELOG.rst └── README.rst /docs/changelog.rst: -------------------------------------------------------------------------------- 1 | .. include:: ../CHANGELOG.rst -------------------------------------------------------------------------------- /docs/favicon.ico: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pilosus/piny/HEAD/docs/favicon.ico -------------------------------------------------------------------------------- /docs/contributing.rst: -------------------------------------------------------------------------------- 1 | .. _contributing-docs: 2 | 3 | .. include:: ../CONTRIBUTING.rst 4 | -------------------------------------------------------------------------------- /docs/favicon-16x16.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pilosus/piny/HEAD/docs/favicon-16x16.png -------------------------------------------------------------------------------- /docs/favicon-32x32.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pilosus/piny/HEAD/docs/favicon-32x32.png -------------------------------------------------------------------------------- /docs/piny_logo_border.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pilosus/piny/HEAD/docs/piny_logo_border.png -------------------------------------------------------------------------------- /docs/piny_logo_noborder.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pilosus/piny/HEAD/docs/piny_logo_noborder.png -------------------------------------------------------------------------------- /tests/configs/db.yaml: -------------------------------------------------------------------------------- 1 | db: 2 | host: db.example.com 3 | login: user 4 | password: ${DB_PASSWORD} 5 | -------------------------------------------------------------------------------- /tests/configs/mail.yaml: -------------------------------------------------------------------------------- 1 | mail: 2 | host: smtp.example.com 3 | login: user 4 | password: ${MAIL_PASSWORD} 5 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | global-exclude .gitignore 2 | exclude .readthedocs.yml 3 | prune tests 4 | prune docs 5 | prune .github 6 | -------------------------------------------------------------------------------- /docs/requirements.txt: -------------------------------------------------------------------------------- 1 | Sphinx<8 2 | sphinx-rtd-theme==1.3.0 3 | sphinx-click==5.1.0 4 | 5 | # Integration examples 6 | Flask==2.3.3 7 | -------------------------------------------------------------------------------- /docs/code/strict_matcher.py: -------------------------------------------------------------------------------- 1 | from piny import YamlLoader, StrictMatcher 2 | 3 | config = YamlLoader(path="config.yaml", matcher=StrictMatcher).load() 4 | -------------------------------------------------------------------------------- /docs/code/config.yaml: -------------------------------------------------------------------------------- 1 | db: 2 | login: user 3 | password: ${DB_PASSWORD} 4 | mail: 5 | login: user 6 | password: ${MAIL_PASSWORD:-my_default_password} 7 | sentry: 8 | dsn: ${VAR_NOT_SET} 9 | -------------------------------------------------------------------------------- /docs/misc.rst: -------------------------------------------------------------------------------- 1 | Fun facts 2 | --------- 3 | 4 | *Piny* is a recursive acronym for *Piny Is Not YAML*. 5 | Not only it's a library name, but also a name for YAML marked up 6 | with environment variables. 7 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | # Linters, formatters, security checkers 2 | black==25.1.0 3 | isort==5.12.0 4 | mypy==1.15.0 5 | 6 | # Testing 7 | pytest==7.4.3 8 | pytest-cov==4.1.0 9 | 10 | # Build tools 11 | twine 12 | build 13 | -------------------------------------------------------------------------------- /docs/code/simple_yaml_loader.py: -------------------------------------------------------------------------------- 1 | from piny import YamlLoader 2 | 3 | config = YamlLoader(path="config.yaml").load() 4 | print(config) 5 | # {'db': {'login': 'user', 'password': 'my_db_password'}, 6 | # 'mail': {'login': 'user', 'password': 'my_default_password'}, 7 | # 'sentry': {'dsn': None}} 8 | -------------------------------------------------------------------------------- /tests/conftest.py: -------------------------------------------------------------------------------- 1 | def pytest_make_parametrize_id(config, val, argname): 2 | """ 3 | Prettify output for parametrized tests 4 | """ 5 | if isinstance(val, dict): 6 | return "{}({})".format( 7 | argname, ", ".join("{}={}".format(k, v) for k, v in val.items()) 8 | ) 9 | -------------------------------------------------------------------------------- /tests/__init__.py: -------------------------------------------------------------------------------- 1 | from pathlib import Path 2 | 3 | config_directory = Path(__file__).resolve().parent.joinpath("configs") 4 | 5 | config_map = { 6 | "db": "my_db_password", 7 | "mail": "my_mail_password", 8 | "sentry": "my_sentry_password", 9 | "logging": "my_logging_password", 10 | } 11 | -------------------------------------------------------------------------------- /tests/test_errors.py: -------------------------------------------------------------------------------- 1 | import pytest 2 | 3 | from piny import LoadingError, YamlLoader 4 | 5 | 6 | def test_loading_error(): 7 | with pytest.raises( 8 | LoadingError, match=r"Loading YAML file failed.+no-such-config.yaml" 9 | ): 10 | YamlLoader(path="no-such-config.yaml").load() 11 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyo 2 | *.pyc 3 | *.swp 4 | .DS_Store 5 | *~ 6 | build/ 7 | dist/ 8 | forgery-py.sublime-project 9 | forgery-py.sublime-workspace 10 | docs/_build/ 11 | .idea/ 12 | tmp/ 13 | .coverage 14 | TODO.rst 15 | *.egg-info/ 16 | .mypy_cache/ 17 | .pytest_cache/ 18 | htmlcov/ 19 | __pycache__/ 20 | _version.py -------------------------------------------------------------------------------- /.github/dependabot.yml: -------------------------------------------------------------------------------- 1 | version: 2 2 | updates: 3 | - package-ecosystem: "pip" 4 | directory: "/" 5 | schedule: 6 | interval: "weekly" 7 | open-pull-requests-limit: 5 8 | 9 | - package-ecosystem: "github-actions" 10 | directory: "/" 11 | schedule: 12 | interval: "weekly" 13 | open-pull-requests-limit: 5 14 | -------------------------------------------------------------------------------- /src/piny/__init__.py: -------------------------------------------------------------------------------- 1 | from .errors import ConfigError, LoadingError, ValidationError 2 | from .loaders import YamlLoader, YamlStreamLoader 3 | from .matchers import Matcher, MatcherWithDefaults, StrictMatcher 4 | from .validators import ( 5 | MarshmallowValidator, 6 | PydanticV2Validator, 7 | PydanticValidator, 8 | TrafaretValidator, 9 | ) 10 | -------------------------------------------------------------------------------- /docs/code/trafaret_validation.py: -------------------------------------------------------------------------------- 1 | import trafaret 2 | from piny import TrafaretValidator, StrictMatcher, YamlLoader 3 | 4 | 5 | DBSchema = trafaret.Dict(login=trafaret.String, password=trafaret.String) 6 | ConfigSchema = trafaret.Dict(db=DBSchema) 7 | 8 | config = YamlLoader( 9 | path="database.yaml", 10 | matcher=StrictMatcher, 11 | validator=TrafaretValidator, 12 | schema=ConfigSchema, 13 | ).load() 14 | -------------------------------------------------------------------------------- /tests/configs/defaults.yaml: -------------------------------------------------------------------------------- 1 | db: 2 | host: db.example.com 3 | login: user 4 | password: ${DB_PASSWORD} 5 | mail: 6 | host: smtp.example.org 7 | login: user 8 | password: ${MAIL_PASSWORD:-My123~!@#$%^&*())_+Password!} 9 | logging: 10 | host: logging.example.org 11 | login: user 12 | password: ${LOGGING_PASSWORD:-:-test:-} 13 | sentry: 14 | host: sentry.example.com 15 | login: user 16 | password: ${SENTRY_PASSWORD:-} 17 | -------------------------------------------------------------------------------- /docs/code/ma_validation.py: -------------------------------------------------------------------------------- 1 | import marshmallow as ma 2 | from piny import MarshmallowValidator, StrictMatcher, YamlLoader 3 | 4 | 5 | class DBSchema(ma.Schema): 6 | login = ma.fields.String(required=True) 7 | password = ma.fields.String() 8 | 9 | 10 | class ConfigSchema(ma.Schema): 11 | db = ma.fields.Nested(DBSchema) 12 | 13 | 14 | config = YamlLoader( 15 | path="database.yaml", 16 | matcher=StrictMatcher, 17 | validator=MarshmallowValidator, 18 | schema=ConfigSchema, 19 | ).load(many=False) 20 | -------------------------------------------------------------------------------- /docs/code/pydantic_validation.py: -------------------------------------------------------------------------------- 1 | from pydantic import BaseModel 2 | from piny import PydanticV2Validator, StrictMatcher, YamlLoader 3 | 4 | # Watch out! 5 | # Pydantic V2 deprecated some model's methods: 6 | # https://docs.pydantic.dev/2.0/migration/ 7 | # 8 | # For Pydantic v2 use `PydanticV2Validator` 9 | # For Pydantic v1 use `PydanticValidator` 10 | 11 | class DBModel(BaseModel): 12 | login: str 13 | password: str 14 | 15 | 16 | class ConfigModel(BaseModel): 17 | db: DBModel 18 | 19 | 20 | config = YamlLoader( 21 | path="database.yaml", 22 | matcher=StrictMatcher, 23 | validator=PydanticV2Validator, 24 | schema=ConfigModel, 25 | ).load() 26 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/issue.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Issue 3 | about: Create a report to help us improve Piny 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | ### Describe the bug ### 11 | 12 | A clear and concise description of what the bug is. 13 | 14 | ### To Reproduce ### 15 | 16 | Steps to reproduce the behavior 17 | 18 | ### Expected behavior ### 19 | 20 | A clear and concise description of what you expected to happen. 21 | 22 | ### Context ### 23 | 24 | - Piny version [e.g. 0.4.0] 25 | - Python version [e.g. CPython 3.7.2] 26 | - OS version (e.g. Linux distribution, version, kernel version) 27 | 28 | ### Additional context ### 29 | 30 | Add any other context about the problem here. 31 | -------------------------------------------------------------------------------- /docs/Makefile: -------------------------------------------------------------------------------- 1 | # Minimal makefile for Sphinx documentation 2 | # 3 | 4 | # You can set these variables from the command line, and also 5 | # from the environment for the first two. 6 | SPHINXOPTS ?= 7 | SPHINXBUILD ?= sphinx-build 8 | SOURCEDIR = . 9 | BUILDDIR = _build 10 | 11 | # Put it first so that "make" without argument is like "make help". 12 | help: 13 | @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) 14 | 15 | .PHONY: help Makefile 16 | 17 | # Catch-all target: route all unknown targets to Sphinx using the new 18 | # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). 19 | %: Makefile 20 | @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) 21 | -------------------------------------------------------------------------------- /.readthedocs.yml: -------------------------------------------------------------------------------- 1 | # .readthedocs.yml 2 | # Read the Docs configuration file 3 | # See https://docs.readthedocs.io/en/stable/config-file/v2.html for details 4 | 5 | # Required 6 | version: 2 7 | 8 | # Set the version of Python and other tools you might need 9 | build: 10 | os: ubuntu-22.04 11 | tools: 12 | python: "3.11" 13 | 14 | # Build documentation in the docs/ directory with Sphinx 15 | sphinx: 16 | configuration: docs/conf.py 17 | 18 | # Optionally build your docs in additional formats such as PDF and ePub 19 | formats: all 20 | 21 | # Optionally set the version of Python and requirements required to build your docs 22 | python: 23 | install: 24 | - method: pip 25 | path: . 26 | - requirements: docs/requirements.txt 27 | -------------------------------------------------------------------------------- /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | ## Change Summary 2 | 3 | A clear and concise description of what has been changed by your Pull Request. 4 | 5 | ## Related Issue Number 6 | 7 | Issue number that will be resolved by the Pull Request. 8 | 9 | ## Checklist 10 | 11 | * [ ] Unit tests for the changes added 12 | * [ ] Test coverage remains at 100% 13 | * [ ] All checks pass on CI 14 | * [ ] Documentation has been updated to reflect the changes made 15 | * [ ] ``CHANGELOG.rst`` has been updated 16 | * add a new section if this is the first change since a release 17 | * add a concise (< 70 characters) change summary 18 | * add an issue number or this pull request number ``(#)`` 19 | * add your GitHub username ``@`` 20 | -------------------------------------------------------------------------------- /docs/install.rst: -------------------------------------------------------------------------------- 1 | Installation 2 | ============ 3 | 4 | pip 5 | --- 6 | 7 | Just use:: 8 | 9 | pip install -U piny 10 | 11 | 12 | *Piny* supports a few third-party validation libraries (see :ref:`usage-validators-docs`). 13 | You may install *Piny* with one of them as en extra requirement:: 14 | 15 | pip install -U 'piny[pydantic]' 16 | 17 | The full list of extra validation libraries is the following: 18 | 19 | - ``marshmallow`` 20 | - ``pydantic`` 21 | - ``trafaret`` 22 | 23 | 24 | GitHub 25 | ------ 26 | 27 | You can also clone *Piny* from `GitHub`_ and install it using ``make install`` 28 | (see :ref:`contributing-docs`):: 29 | 30 | git clone https://github.com/pilosus/piny 31 | cd piny 32 | make install 33 | 34 | .. _GitHub: https://github.com/pilosus/piny 35 | -------------------------------------------------------------------------------- /docs/best.rst: -------------------------------------------------------------------------------- 1 | Best practices 2 | ============== 3 | 4 | - Maintain a healthy security/convenience balance for your config 5 | 6 | - Mark up entity as an environment variable in your YAML if and only if 7 | it really is a *secret* (login/passwords, private API keys, crypto keys, 8 | certificates, or maybe DB hostname too? You decide) 9 | 10 | - When loading config file, validate your data. 11 | Piny supports a few popular data validation tools. 12 | 13 | - Store your config files in the version control system along with your app’s code. 14 | 15 | - Environment variables are set by whoever is responsible for the deployment. 16 | Modern orchestration systems like `Kubernetes`_ make it easier to keep envs secure 17 | (see `Kubernetes Secrets`_). 18 | 19 | .. _Kubernetes: https://kubernetes.io/ 20 | .. _Kubernetes Secrets: https://kubernetes.io/docs/concepts/configuration/secret/ 21 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/feature_request.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Feature request 3 | about: Suggest an idea for this project 4 | title: '' 5 | labels: enhancement 6 | assignees: pilosus 7 | 8 | --- 9 | 10 | ### Is your feature request related to a problem? Please describe ### 11 | 12 | A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] 13 | 14 | ### Describe the solution you'd like ### 15 | 16 | A clear and concise description of what you want to happen. 17 | 18 | Does your solution follow a Unix-way rule *Do One Thing and Do It Well*? 19 | If it doesn't, can you make it to follow the rule? 20 | 21 | ### Describe alternatives you've considered ### 22 | 23 | A clear and concise description of any alternative solutions or features you've considered. 24 | 25 | ### Additional context ### 26 | 27 | Add any other context or screenshots about the feature request here. 28 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2016 Vitaly R. Samigullin 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy of 6 | this software and associated documentation files (the "Software"), to deal in 7 | the Software without restriction, including without limitation the rights to 8 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 9 | the Software, and to permit persons to whom the Software is furnished to do so, 10 | subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS 17 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 18 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 19 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 20 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 21 | -------------------------------------------------------------------------------- /src/piny/errors.py: -------------------------------------------------------------------------------- 1 | from typing import Any, Optional 2 | 3 | 4 | class PinyErrorMixin: 5 | """ 6 | Mixin class to wrap and format original exception 7 | """ 8 | 9 | msg_template: str 10 | 11 | def __init__(self, origin: Optional[Exception] = None, **context: Any) -> None: 12 | """ 13 | Mixing for wrapping original exception 14 | 15 | :param origin: original exception, 16 | may be helpful when special method invocation needed 17 | :param context: mapping used for exception message formatting 18 | """ 19 | self.origin = origin 20 | self.context = context or None 21 | super().__init__() 22 | 23 | def __str__(self) -> str: 24 | return self.msg_template.format(**self.context or {}) 25 | 26 | 27 | class ConfigError(PinyErrorMixin, Exception): 28 | """ 29 | Base class for Piny exceptions 30 | """ 31 | 32 | pass 33 | 34 | 35 | class LoadingError(ConfigError): 36 | """ 37 | Exception for reading or parsing configuration file errors 38 | """ 39 | 40 | msg_template = "Loading YAML file failed: {reason}" 41 | 42 | 43 | class ValidationError(ConfigError): 44 | """ 45 | Exception for data validation errors 46 | """ 47 | 48 | msg_template = "Validation failed: {reason}" 49 | -------------------------------------------------------------------------------- /src/piny/cli.py: -------------------------------------------------------------------------------- 1 | from typing import Any 2 | 3 | import click 4 | import yaml 5 | 6 | from .loaders import YamlStreamLoader 7 | from .matchers import MatcherWithDefaults, StrictMatcher 8 | 9 | CONTEXT_SETTINGS = dict(help_option_names=["-h", "--help"]) 10 | 11 | 12 | @click.command(context_settings=CONTEXT_SETTINGS) 13 | @click.argument("input", required=False, type=click.File("r")) 14 | @click.argument("output", required=False, type=click.File("w")) 15 | @click.option( 16 | "--strict/--no-strict", default=True, help="Enable or disable strict matcher" 17 | ) 18 | def cli(input, output, strict) -> Any: 19 | """ 20 | Substitute environment variables with their values. 21 | 22 | Read INPUT, find environment variables in it, 23 | substitute them with their values and write to OUTPUT. 24 | 25 | INPUT and OUTPUT can be files or standard input and output respectively. 26 | With no INPUT, or when INPUT is -, read standard input. 27 | With no OUTPUT, or when OUTPUT is -, write to standard output. 28 | 29 | Examples: 30 | 31 | \b 32 | piny input.yaml output.yaml 33 | piny - output.yaml 34 | piny input.yaml - 35 | tail -n 12 input.yaml | piny > output.yaml 36 | """ 37 | stdin = click.get_text_stream("stdin") 38 | stdout = click.get_text_stream("stdout") 39 | 40 | config = YamlStreamLoader( 41 | stream=input or stdin, matcher=StrictMatcher if strict else MatcherWithDefaults 42 | ).load() 43 | 44 | yaml.dump(config, output or stdout) 45 | -------------------------------------------------------------------------------- /src/piny/matchers.py: -------------------------------------------------------------------------------- 1 | import os 2 | import re 3 | from typing import Pattern 4 | 5 | import yaml 6 | 7 | 8 | class Matcher(yaml.SafeLoader): 9 | """ 10 | Base class for matchers 11 | 12 | Use this class only to derive new child classes 13 | """ 14 | 15 | matcher: Pattern[str] = re.compile("") 16 | 17 | @staticmethod 18 | def constructor(loader, node): 19 | raise NotImplementedError 20 | 21 | 22 | class StrictMatcher(Matcher): 23 | """ 24 | Expand an environment variable of form ${VAR} with its value 25 | 26 | If value is not set return None. 27 | """ 28 | 29 | matcher = re.compile(r"\$\{([^}^{^:]+)\}") 30 | 31 | @staticmethod 32 | def constructor(loader, node): 33 | match = StrictMatcher.matcher.match(node.value) 34 | return os.environ.get(match.groups()[0]) # type: ignore 35 | 36 | 37 | class MatcherWithDefaults(Matcher): 38 | """ 39 | Expand an environment variable with its value 40 | 41 | Forms supported: ${VAR}, ${VAR:-default} 42 | If value is not set and no default value given return None. 43 | """ 44 | 45 | matcher = re.compile(r"\$\{([a-zA-Z_$0-9]+)(:-.*)?\}") 46 | 47 | @staticmethod 48 | def constructor(loader, node): 49 | match = MatcherWithDefaults.matcher.match(node.value) 50 | variable, default = match.groups() # type: ignore 51 | 52 | if default: 53 | # lstrip() is dangerous! 54 | # It can remove legitimate first two letters in a value starting with `:-` 55 | default = default[2:] 56 | 57 | return os.environ.get(variable, default) 58 | -------------------------------------------------------------------------------- /tests/test_cli.py: -------------------------------------------------------------------------------- 1 | from unittest import mock 2 | 3 | import yaml 4 | from click.testing import CliRunner 5 | 6 | from piny import LoadingError 7 | from piny.cli import cli 8 | 9 | from . import config_directory 10 | 11 | 12 | def test_cli_input_stdin_output_stdout(): 13 | runner = CliRunner() 14 | with mock.patch("piny.matchers.StrictMatcher.constructor") as expand_mock: 15 | expand_mock.return_value = "MySecretPassword" 16 | result = runner.invoke(cli, input="password: ${DB_PASSWORD}") 17 | 18 | assert result.exit_code == 0 19 | assert result.output == "password: MySecretPassword\n" 20 | 21 | 22 | def test_cli_input_file_output_file(): 23 | runner = CliRunner() 24 | with open(config_directory.joinpath("db.yaml"), "r") as f: 25 | input_lines = f.readlines() 26 | 27 | with runner.isolated_filesystem(): 28 | with open("input.yaml", "w") as input_fd: 29 | input_fd.writelines(input_lines) 30 | 31 | with mock.patch("piny.matchers.StrictMatcher.constructor") as expand_mock: 32 | expand_mock.return_value = "MySecretPassword" 33 | result = runner.invoke(cli, ["input.yaml", "output.yaml"]) 34 | 35 | with open("output.yaml", "r") as output_fd: 36 | output_lines = output_fd.readlines() 37 | 38 | assert result.exit_code == 0 39 | assert "password: MySecretPassword" in map( 40 | lambda x: x.strip(), output_lines 41 | ) 42 | 43 | 44 | def test_cli_fail(): 45 | runner = CliRunner() 46 | with mock.patch("piny.loaders.yaml.load") as loader_mock: 47 | loader_mock.side_effect = yaml.YAMLError("Oops!") 48 | result = runner.invoke(cli, input="password: ${DB_PASSWORD}") 49 | assert result.exit_code == 1 50 | assert type(result.exception) == LoadingError 51 | -------------------------------------------------------------------------------- /src/piny/validators.py: -------------------------------------------------------------------------------- 1 | from abc import ABC, abstractmethod 2 | from typing import Any, Dict, List, Union 3 | 4 | from .errors import ValidationError 5 | 6 | LoadedData = Union[Dict[str, Any], List[Any]] 7 | 8 | 9 | class Validator(ABC): 10 | """ 11 | Abstract base class for optional validator classes 12 | 13 | Use only to derive new child classes, implement all abstract methods 14 | """ 15 | 16 | def __init__(self, schema: Any, **params): 17 | self.schema = schema 18 | self.schema_params = params 19 | 20 | @abstractmethod 21 | def load(self, data: LoadedData, **params): 22 | """ 23 | Load data, return validated data or raise en error 24 | """ 25 | pass # pragma: no cover 26 | 27 | 28 | class PydanticValidator(Validator): # pragma: no cover 29 | """ 30 | Validator class for Pydantic Version 1 31 | """ 32 | 33 | def load(self, data: LoadedData, **params): 34 | try: 35 | return self.schema(**data).dict() 36 | except Exception as e: 37 | raise ValidationError(origin=e, reason=str(e)) 38 | 39 | 40 | class PydanticV2Validator(Validator): 41 | """ 42 | Validator class for Pydantic Version 2 43 | """ 44 | 45 | def load(self, data: LoadedData, **params): 46 | try: 47 | return self.schema(**data).model_dump() 48 | except Exception as e: 49 | raise ValidationError(origin=e, reason=str(e)) 50 | 51 | 52 | class MarshmallowValidator(Validator): 53 | """ 54 | Validator class for Marshmallow library 55 | """ 56 | 57 | def load(self, data: LoadedData, **params): 58 | try: 59 | return self.schema(**self.schema_params).load(data, **params) 60 | except Exception as e: 61 | raise ValidationError(origin=e, reason=str(e)) 62 | 63 | 64 | class TrafaretValidator(Validator): 65 | """ 66 | Validator class for Trafaret library 67 | """ 68 | 69 | def load(self, data: LoadedData, **params): 70 | try: 71 | return self.schema.check(data) 72 | except Exception as e: 73 | raise ValidationError(origin=e, reason=str(e)) 74 | -------------------------------------------------------------------------------- /docs/integration.rst: -------------------------------------------------------------------------------- 1 | Integration Examples 2 | ==================== 3 | 4 | Flask 5 | ----- 6 | 7 | `Flask`_ is a microframework for Python web applications. It's flexible and extensible. 8 | Although there are best practices and traditions, Flask doesn't really enforce 9 | the only one way to do it. 10 | 11 | If you are working on a small project the chances are that you are using some Flask 12 | extensions like `Flask-Mail`_ or `Flask-WTF`_. The extensions of the past are often 13 | got configured through environment variables only. It makes the use of *Piny* cumbersome. 14 | In mid-sized and large Flask projects though, you usually avoid using extra dependencies 15 | whenever possible. In such a case you can fit your code to use *Piny* pretty easy. 16 | 17 | Here is an example of a simple Flask application. Configuration file is loaded with *Piny* 18 | and validated with *Pydantic*. 19 | 20 | .. literalinclude:: code/flask_integration.py 21 | 22 | You can use the same pattern with application factory in other frameworks, 23 | like `aiohttp`_ or `sanic`_. 24 | 25 | .. _Flask: http://flask.pocoo.org/docs/1.0/ 26 | .. _Flask-Mail: https://pythonhosted.org/Flask-Mail/ 27 | .. _Flask-WTF: https://flask-wtf.readthedocs.io/en/stable/ 28 | .. _aiohttp: https://aiohttp.readthedocs.io/en/stable/ 29 | .. _sanic: https://sanic.readthedocs.io/en/latest/ 30 | 31 | 32 | Command line 33 | ------------ 34 | 35 | There are many possible applications for *Piny* CLI utility. 36 | For example, you can use it for `Kubernetes deployment automation`_ 37 | in your CI/CD pipeline. 38 | 39 | Piny command line tool works both with standard input/output and files. 40 | 41 | 42 | Standard input and output 43 | ......................... 44 | 45 | .. code-block:: bash 46 | 47 | $ export PASSWORD=mySecretPassword 48 | $ echo "db: \${PASSWORD}" | piny 49 | db: mySecretPassword 50 | 51 | 52 | Files 53 | ..... 54 | 55 | 56 | .. code-block:: bash 57 | 58 | $ piny config.template config.yaml 59 | 60 | 61 | Or you can substitute environment variables in place: 62 | 63 | .. code-block:: bash 64 | 65 | $ piny production.yaml production.yaml 66 | 67 | 68 | .. _Kubernetes deployment automation: https://www.digitalocean.com/community/tutorials/how-to-automate-deployments-to-digitalocean-kubernetes-with-circleci -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | # https://packaging.python.org/en/latest/tutorials/packaging-projects/ 2 | # https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html 3 | 4 | [build-system] 5 | requires = ["setuptools>=61.0", "setuptools_scm[toml]>=6.3.1"] 6 | build-backend = "setuptools.build_meta" 7 | 8 | [project] 9 | name = "piny" 10 | description = "Load YAML configs with environment variables interpolation" 11 | readme = "README.rst" 12 | authors = [ 13 | {name = "Vitaly Samigullin", email = "vrs@pilosus.org"}, 14 | ] 15 | dynamic = ["version"] 16 | license = {text = "MIT"} 17 | classifiers = [ 18 | "Programming Language :: Python :: 3", 19 | "Intended Audience :: Developers", 20 | "Intended Audience :: Information Technology", 21 | "Intended Audience :: System Administrators", 22 | "License :: OSI Approved :: MIT License", 23 | "Operating System :: Unix", 24 | "Operating System :: POSIX :: Linux", 25 | "Environment :: Console", 26 | "Environment :: MacOS X", 27 | "Topic :: Software Development :: Libraries :: Python Modules", 28 | "Topic :: Internet", 29 | ] 30 | requires-python = ">=3.7" 31 | dependencies = [ 32 | "PyYAML>=6,<7", 33 | "Click>=8,<9", 34 | ] 35 | 36 | [project.optional-dependencies] 37 | pydantic = ["pydantic>=0.28"] 38 | marshmallow = ["marshmallow>=3"] 39 | trafaret = ["trafaret>=1.2.0"] 40 | 41 | [project.urls] 42 | "Source code" = "https://github.com/pilosus/piny/" 43 | "Issue tracker" = "https://github.com/pilosus/piny/issues" 44 | "Documentation" = "https://piny.readthedocs.io/en/latest/" 45 | 46 | [project.scripts] 47 | piny = "piny.cli:cli" 48 | 49 | [tool.setuptools_scm] 50 | write_to = "_version.py" 51 | 52 | [tool.isort] 53 | atomic = true 54 | line_length = 88 55 | skip_gitignore = true 56 | known_first_party = ["piny"] 57 | multi_line_output = 3 58 | include_trailing_comma = true 59 | force_grid_wrap = 0 60 | 61 | [tool.mypy] 62 | ignore_missing_imports = true 63 | follow_imports = "silent" 64 | strict_optional = true 65 | warn_redundant_casts = true 66 | warn_unused_ignores = true 67 | disallow_any_generics = true 68 | check_untyped_defs = true 69 | 70 | [tool.black] 71 | target-version = ["py37", "py38", "py39", "py310", "py311"] 72 | line-length = 88 73 | 74 | [tool.coverage.report] 75 | fail_under = 95 76 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | .DEFAULT_GOAL := check 2 | isort = isort src tests 3 | black = black src tests 4 | mypy = mypy --install-types --non-interactive src 5 | 6 | .PHONY: install-pip 7 | install-pip: 8 | @echo "Install pip" 9 | pip install -U pip 10 | 11 | 12 | .PHONY: install-deps 13 | install-deps: 14 | @echo "Install dependencies" 15 | pip install -U -r requirements.txt 16 | 17 | .PHONY: install-package 18 | install-package: 19 | @echo "Install package" 20 | pip install -e . 21 | 22 | .PHONY: install-package-with-extra 23 | install-package-with-extra: 24 | @echo "Install package with extra dependencies" 25 | pip install -e .[pydantic,marshmallow,trafaret] 26 | 27 | .PHONY: install 28 | install: install-pip install-deps install-package-with-extra 29 | 30 | .PHONY: format 31 | format: 32 | @echo "Run code formatters" 33 | $(isort) 34 | $(black) 35 | 36 | 37 | .PHONY: lint 38 | lint: 39 | @echo "Run linters" 40 | $(isort) --check-only 41 | $(black) --check 42 | 43 | 44 | .PHONY: test 45 | test: 46 | @echo "Run tests" 47 | pytest -vvs --cov=piny tests 48 | 49 | .PHONY: testcov 50 | testcov: test 51 | @echo "Build coverage html" 52 | @coverage html 53 | 54 | .PHONY: mypy 55 | mypy: 56 | @echo "Run mypy static analysis" 57 | $(mypy) 58 | 59 | 60 | .PHONY: check 61 | check: lint test mypy 62 | 63 | 64 | .PHONY: build 65 | build: 66 | @echo "Build Python package" 67 | python -m build --sdist --wheel 68 | python -m twine check dist/* 69 | 70 | .PHONY: push-test 71 | push-test: 72 | @echo "Push package to test.pypi.org" 73 | python -m twine upload --verbose --repository testpypi dist/* 74 | 75 | 76 | .PHONY: push 77 | push: 78 | @echo "Run package to PyPI" 79 | python -m twine upload --verbose dist/* 80 | 81 | 82 | .PHONY: clean 83 | clean: 84 | @echo "Clean up files" 85 | rm -rf `find . -name __pycache__` 86 | rm -f `find . -type f -name '*.py[co]' ` 87 | rm -f `find . -type f -name '*~' ` 88 | rm -f `find . -type f -name '.*~' ` 89 | rm -rf .cache 90 | rm -rf .pytest_cache 91 | rm -rf .mypy_cache 92 | rm -rf htmlcov 93 | rm -rf *.egg-info 94 | rm -f .coverage 95 | rm -f .coverage.* 96 | rm -rf build 97 | rm -rf dist 98 | 99 | .PHONY: docs 100 | docs: 101 | pip install -U -r docs/requirements.txt 102 | make -C docs html 103 | -------------------------------------------------------------------------------- /.github/workflows/ci.yml: -------------------------------------------------------------------------------- 1 | name: CI 2 | 3 | on: 4 | push: 5 | branches: 6 | - master 7 | tags: 8 | - 'v[0-9]+.[0-9]+.[0-9]+*' 9 | pull_request: 10 | branches: 11 | - master 12 | 13 | permissions: 14 | contents: read 15 | 16 | jobs: 17 | test: 18 | name: Tests and Linters 19 | runs-on: ubuntu-22.04 20 | steps: 21 | - name: Checkout the code 22 | uses: actions/checkout@v4 23 | with: 24 | fetch-depth: 0 25 | - name: Set up Python 26 | uses: actions/setup-python@v4 27 | with: 28 | python-version: '3.11' 29 | - name: Install dependencies 30 | run: make install 31 | - name: Run linters 32 | run: make lint 33 | - name: Run mypy 34 | run: make mypy 35 | - name: Run tests 36 | run: make test 37 | 38 | license: 39 | name: License compliance check 40 | runs-on: ubuntu-22.04 41 | steps: 42 | - name: Checkout the code 43 | uses: actions/checkout@v4 44 | with: 45 | fetch-depth: 0 46 | - name: Set up Python 47 | uses: actions/setup-python@v4 48 | with: 49 | python-version: '3.11' 50 | - name: Dump all package's dependencies 51 | run: | 52 | make install-pip 53 | make install-package 54 | pip freeze > requirements.dump.txt 55 | - name: Check Python deps licenses 56 | id: license_check_report 57 | uses: pilosus/action-pip-license-checker@v2 58 | with: 59 | requirements: 'requirements.dump.txt' 60 | fail: 'StrongCopyleft,NetworkCopyleft,Other,Error' 61 | totals: true 62 | headers: true 63 | - name: Print report 64 | if: ${{ always() }} 65 | run: echo "${{ steps.license_check_report.outputs.report }}" 66 | 67 | publish: 68 | name: Publish a package on PyPI 69 | runs-on: ubuntu-22.04 70 | needs: [test, license] 71 | if: github.event_name == 'push' && contains(github.ref, 'refs/tags/') 72 | steps: 73 | - name: Checkout the code 74 | uses: actions/checkout@v4 75 | with: 76 | fetch-depth: 0 77 | - name: Set up Python 78 | uses: actions/setup-python@v4 79 | with: 80 | python-version: '3.11' 81 | - name: Install dependencies 82 | run: | 83 | make install-pip 84 | make install-deps 85 | - name: Build the package 86 | run: make build 87 | - name: Publish on PyPI 88 | env: 89 | TWINE_USERNAME: ${{ secrets.TWINE_USERNAME }} 90 | TWINE_PASSWORD: ${{ secrets.TWINE_PASSWORD }} 91 | run: make push 92 | -------------------------------------------------------------------------------- /CONTRIBUTING.rst: -------------------------------------------------------------------------------- 1 | Contributing to Piny 2 | ==================== 3 | 4 | Piny is a `proof-of-concept`_. It's developed specifically (but not 5 | limited to!) for the containerized Python applications deployed with 6 | orchestration systems like ``docker compose`` or ``Kubernetes``. 7 | 8 | Piny is still in its early stage of development. The API may change, 9 | backward compatibility between `minor versions`_ is not guaranteed until 10 | version 1.0.0 is reached. 11 | 12 | Piny sticks to the Unix-way's rule *Do One Thing and Do It Well*. 13 | Piny is all about interpolating environment variables in configuration files. 14 | Other features like YAML-parsing or data validation are implemented 15 | using third-party libraries whenever possible. 16 | 17 | You are welcome to contribute to *Piny* as long as you follow the rules. 18 | 19 | 20 | General rules 21 | ------------- 22 | 23 | 1. Before writing any *code* take a look at the existing `open issues`_. 24 | If none of them is about the changes you want to contribute, 25 | open up a new issue. Fixing a typo requires no issue though, 26 | just submit a Pull Request. 27 | 28 | 2. If you're looking for an open issue to fix, check out 29 | labels `help wanted`_ and `good first issue`_ on GitHub. 30 | 31 | 3. If you plan to work on an issue open not by you, write about your 32 | intention in the comments *before* you start working. 33 | 34 | 4. Follow an Issue/Pull Request template. 35 | 36 | 37 | Development rules 38 | ----------------- 39 | 40 | 1. Fork `Piny`_ on GitHub. 41 | 42 | 2. Clone your fork with ``git clone``. 43 | 44 | 3. Use ``Python 3.6+``, ``git``, ``make`` and ``virtualenv``. 45 | 46 | 4. Create and activate ``virtualenv``. 47 | 48 | 5. Install *Piny* and its dependencies with ``make install``. 49 | 50 | 6. Follow `GitHub Flow`_: create a new branch from ``master`` with 51 | ``git checkout -b ``. Make your changes. 52 | 53 | 7. Fix your code's formatting and imports with ``make format``. 54 | 55 | 8. Run unit-tests and linters with ``make check``. 56 | 57 | 9. Build documentation with ``make docs``. 58 | 59 | 10. Commit, push, open new Pull Request. 60 | 61 | 11. Make sure Travis CI/CD pipeline succeeds. 62 | 63 | .. _proof-of-concept: https://blog.pilosus.org/posts/2019/06/07/application-configs-files-or-environment-variables-actually-both/ 64 | .. _minor versions: https://semver.org/ 65 | .. _open issues: https://github.com/pilosus/piny/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen 66 | .. _help wanted: https://github.com/pilosus/piny/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22 67 | .. _good first issue: https://github.com/pilosus/piny/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22 68 | .. _GitHub Flow: https://guides.github.com/introduction/flow/ 69 | .. _Piny: https://github.com/pilosus/piny/fork 70 | -------------------------------------------------------------------------------- /docs/index.rst: -------------------------------------------------------------------------------- 1 | .. Piny documentation master file, created by 2 | sphinx-quickstart on Mon Jun 17 11:38:51 2019. 3 | You can adapt this file completely to your liking, but it should at least 4 | contain the root `toctree` directive. 5 | 6 | Piny: envs interpolation for config files 7 | ========================================= 8 | 9 | |PyPI| |Coverage| |License| 10 | 11 | *Piny* is YAML config loader with environment variables interpolation for Python. 12 | 13 | - Keep your app's configuration in a YAML file. 14 | - Mark up sensitive data in config as *environment variables*. 15 | - Set environment variables on application deployment. 16 | - Let *Piny* load your configuration file and substitute environment variables with their values. 17 | 18 | Piny is developed with Docker and Kubernetes in mind, 19 | though it's not limited to any deployment system. 20 | 21 | 22 | Simple example 23 | -------------- 24 | 25 | Set your environment variables, mark up your configuration file with them: 26 | 27 | .. literalinclude:: code/config.yaml 28 | :language: yaml 29 | 30 | Then load your config with *Piny*: 31 | 32 | .. literalinclude:: code/simple_yaml_loader.py 33 | 34 | 35 | CLI utility 36 | ----------- 37 | 38 | Piny's also got a command line tool working both with files and standard input and output: 39 | 40 | .. code-block:: bash 41 | 42 | $ export PASSWORD=mySecretPassword 43 | $ echo "db: \${PASSWORD}" | piny 44 | db: mySecretPassword 45 | 46 | 47 | Rationale 48 | --------- 49 | 50 | Piny allows you to maintain healthy security/convenience balance 51 | when it comes to application's configuration. Piny combines readability 52 | and versioning you get when using config files, and security that 53 | environment variables provide. 54 | 55 | Read more about this approach in the `blog post`_. 56 | 57 | .. _blog post: https://blog.pilosus.org/posts/2019/06/07/application-configs-files-or-environment-variables-actually-both/?utm_source=docs&utm_medium=link&utm_campaign=rationale 58 | 59 | 60 | .. _user-docs: 61 | 62 | .. toctree:: 63 | :maxdepth: 2 64 | :caption: User Documentation 65 | 66 | install 67 | usage 68 | integration 69 | best 70 | 71 | .. _dev-docs: 72 | 73 | .. toctree:: 74 | :maxdepth: 2 75 | :caption: Developer Documentation 76 | 77 | contributing 78 | changelog 79 | 80 | .. _misc-docs: 81 | 82 | .. toctree:: 83 | :maxdepth: 2 84 | :caption: Misc 85 | 86 | misc 87 | 88 | .. |PyPI| image:: https://img.shields.io/pypi/v/piny 89 | :alt: PyPI 90 | :target: https://pypi.org/project/piny/ 91 | .. |Coverage| image:: https://img.shields.io/codecov/c/github/pilosus/piny.svg 92 | :alt: Codecov 93 | :target: https://codecov.io/gh/pilosus/piny 94 | .. |License| image:: https://img.shields.io/github/license/pilosus/piny.svg 95 | :alt: MIT License 96 | :target: https://github.com/pilosus/piny/blob/master/LICENSE 97 | -------------------------------------------------------------------------------- /CHANGELOG.rst: -------------------------------------------------------------------------------- 1 | Changelog 2 | --------- 3 | 4 | v1.1.0 (2023-09-22) 5 | ................... 6 | 7 | * Added: new validator `PydanticV2Validator` to support Pydantic v2 8 | 9 | 10 | v1.0.2 (2023-02-03) 11 | ................... 12 | 13 | * Update GitHub workflow for CI: run tests & license checks for PRs, pushes to master and tags (#202) by @pilosus 14 | * Make dependabot update GitHub Actions (#202) by @pilosus 15 | 16 | 17 | v1.0.1 (2023-02-03) 18 | ................... 19 | 20 | * Run tests against locally installed package instead of using ugly imports (#200) by @pilosus 21 | 22 | v1.0.0 (2023-01-02) 23 | ...................... 24 | 25 | See release notes to `v1.0.0rc1` 26 | 27 | v1.0.0rc1 (2023-01-01) 28 | ...................... 29 | 30 | **Release breaks backward compatibility!** 31 | 32 | * Bump major dependencies: `PyYAML>=6,<7` `Click>=8,<9` (#192) by @pilosus 33 | * `Marshmallow` integration supports only v3.0.0 and later (#192) by @pilosus 34 | * Move to `pyproject.toml` for packaging (#193) by @pilosus 35 | * Raise Python requirement to `>=3.7` (#193) by @pilosus 36 | 37 | v0.6.0 (2019-06-27) 38 | ................... 39 | * Add CLI utility (#35) by @pilosus 40 | * Update documentation, add integration examples (#34) by @pilosus 41 | 42 | v0.5.2 (2019-06-17) 43 | ................... 44 | * Fix ``Help`` section in ``README.rst`` (#31) by @pilosus 45 | * Fix Sphinx release variable (#30) by @pilosus 46 | 47 | v0.5.1 (2019-06-17) 48 | ................... 49 | * Fix Sphinx config, fix README.rst image markup (#28) by @pilosus 50 | 51 | v0.5.0 (2019-06-17) 52 | ................... 53 | * Sphinx documentation added (#12) by @pilosus 54 | * Piny artwork added (#6) by Daria Runenkova and @pilosus 55 | 56 | v0.4.2 (2019-06-17) 57 | ................... 58 | * Rename parent exception ``PinyError`` to ``ConfigError`` (#18) by @pilosus 59 | * Add feature request template for GitHub Issues (#20) by @pilosus 60 | 61 | v0.4.1 (2019-06-17) 62 | ................... 63 | * Issue and PR templates added, minor docs fixes (#16) by @pilosus 64 | 65 | v0.4.0 (2019-06-16) 66 | ................... 67 | * Data validators support added for ``Pydantic``, ``Marshmallow`` (#2) by @pilosus 68 | * ``CONTRIBUTING.rst`` added (#4) by @pilosus 69 | 70 | v0.3.1 (2019-06-09) 71 | ................... 72 | * Minor RST syntax fix in README.rst (#9) by @pilosus 73 | 74 | v0.3.0 (2019-06-09) 75 | ................... 76 | * README.rst extended with ``Rationale`` and ``Best practices`` sections (#5) by @pilosus 77 | 78 | v0.2.0 (2019-06-09) 79 | ................... 80 | * StrictMatcher added (#3) by @pilosus 81 | 82 | v0.1.1 (2019-06-07) 83 | ................... 84 | * CI/CD config minor tweaks 85 | * README updated 86 | 87 | v0.1.0 (2019-06-07) 88 | ................... 89 | * YamlLoader added 90 | * Makefile added 91 | * CI/CD minimal pipeline added 92 | 93 | v0.0.1 (2019-06-07) 94 | ................... 95 | * Start the project 96 | -------------------------------------------------------------------------------- /docs/code/flask_integration.py: -------------------------------------------------------------------------------- 1 | from flask import Flask 2 | from flask.logging import default_handler 3 | from piny import YamlLoader, StrictMatcher, PydanticV2Validator 4 | from pydantic import BaseModel, validator 5 | from typing import Any, Dict, Optional 6 | from werkzeug.serving import run_simple 7 | import logging 8 | import sys 9 | 10 | # Watch out! 11 | # Pydantic V2 deprecated some model's methods: 12 | # https://docs.pydantic.dev/2.0/migration/ 13 | # 14 | # For Pydantic v2 use `PydanticV2Validator` 15 | # For Pydantic v1 use `PydanticValidator` 16 | 17 | 18 | # 19 | # Validation 20 | # 21 | 22 | 23 | class AppSettings(BaseModel): 24 | company: str 25 | secret: str 26 | max_content_len: Optional[int] = None 27 | debug: bool = False 28 | testing: bool = False 29 | 30 | 31 | class LoggingSettings(BaseModel): 32 | fmt: str 33 | date_fmt: str 34 | level: str 35 | 36 | @validator("level") 37 | def validate_name(cls, value): 38 | upper = value.upper() 39 | if upper not in logging._nameToLevel: 40 | raise ValueError("Invalid logging level") 41 | return upper 42 | 43 | 44 | class Configuration(BaseModel): 45 | app: AppSettings 46 | logging: LoggingSettings 47 | 48 | 49 | # 50 | # Helpers 51 | # 52 | 53 | 54 | def configure_app(app: Flask, configuration: Dict[str, Any]) -> None: 55 | """ 56 | Apply configs to application 57 | """ 58 | app.settings = configuration 59 | app.secret_key = app.settings["app"]["secret"].encode("utf-8") 60 | 61 | 62 | def configure_logging(app: Flask) -> None: 63 | """ 64 | Configure app's logging 65 | """ 66 | app.logger.removeHandler(default_handler) 67 | log_formatter = logging.Formatter( 68 | fmt=app.settings["logging"]["fmt"], datefmt=app.settings["logging"]["date_fmt"] 69 | ) 70 | log_handler = logging.StreamHandler() 71 | log_handler.setFormatter(log_formatter) 72 | log_handler.setLevel(app.settings["logging"]["level"]) 73 | app.logger.addHandler(log_handler) 74 | 75 | 76 | # 77 | # Factory 78 | # 79 | 80 | 81 | def create_app(path: str) -> Flask: 82 | """ 83 | Application factory 84 | """ 85 | # Get and validate config 86 | config = YamlLoader( 87 | path=path, 88 | matcher=StrictMatcher, 89 | validator=PydanticV2Validator, 90 | schema=Configuration, 91 | ).load() 92 | 93 | # Initialize app 94 | app = Flask(__name__) 95 | 96 | # Configure app 97 | configure_app(app, config) 98 | configure_logging(app) 99 | 100 | return app 101 | 102 | 103 | if __name__ == "__main__": 104 | app = create_app(sys.argv[1]) 105 | 106 | @app.route("/") 107 | def hello(): 108 | return "Hello World!" 109 | 110 | # Run application: 111 | # $ python flask_integration.py your-config.yaml 112 | run_simple(hostname="localhost", port=5000, application=app) 113 | -------------------------------------------------------------------------------- /docs/conf.py: -------------------------------------------------------------------------------- 1 | # Configuration file for the Sphinx documentation builder. 2 | # 3 | # This file only contains a selection of the most common options. For a full 4 | # list see the documentation: 5 | # http://www.sphinx-doc.org/en/master/config 6 | 7 | # -- Path setup -------------------------------------------------------------- 8 | 9 | # If extensions (or modules to document with autodoc) are in another directory, 10 | # add these directories to sys.path here. If the directory is relative to the 11 | # documentation root, use os.path.abspath to make it absolute, like shown here. 12 | # 13 | import os 14 | import sys 15 | 16 | sys.path.append(os.path.abspath("..")) 17 | 18 | # -- Project information ----------------------------------------------------- 19 | 20 | project = "Piny" 21 | copyright = "2019-2023, Vitaly Samigullin" 22 | author = "Vitaly Samigullin" 23 | 24 | # https://github.com/pypa/setuptools_scm#usage-from-sphinx 25 | from importlib.metadata import version 26 | 27 | release = version("piny") 28 | version = ".".join(release.split(".")[:2]) 29 | 30 | # -- General configuration --------------------------------------------------- 31 | 32 | # Add any Sphinx extension module names here, as strings. They can be 33 | # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom 34 | # ones. 35 | extensions = [ 36 | "sphinx.ext.autodoc", 37 | "sphinx.ext.viewcode", 38 | "sphinx_click.ext", 39 | ] 40 | 41 | # Add any paths that contain templates here, relative to this directory. 42 | templates_path = ["_templates"] 43 | 44 | # List of patterns, relative to source directory, that match files and 45 | # directories to ignore when looking for source files. 46 | # This pattern also affects html_static_path and html_extra_path. 47 | exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"] 48 | 49 | 50 | # -- Options for HTML output ------------------------------------------------- 51 | 52 | # The theme to use for HTML and HTML Help pages. See the documentation for 53 | # a list of builtin themes. 54 | # 55 | # html_theme = 'alabaster' 56 | 57 | html_theme = "sphinx_rtd_theme" 58 | html_theme_options = { 59 | # 'canonical_url': '', 60 | # 'analytics_id': 'UA-XXXXXXX-1', 61 | # 'vcs_pageview_mode': '', 62 | # 'style_nav_header_background': 'white', 63 | "logo_only": False, 64 | "display_version": True, 65 | "prev_next_buttons_location": "bottom", 66 | "style_external_links": False, 67 | # Toc options 68 | "collapse_navigation": False, 69 | "sticky_navigation": True, 70 | "navigation_depth": 4, 71 | "includehidden": True, 72 | "titles_only": False, 73 | } 74 | 75 | # html_logo = "piny_logo.png" 76 | html_logo = "piny_logo_noborder.png" 77 | html_favicon = "favicon-32x32.png" 78 | 79 | # Add any paths that contain custom static files (such as style sheets) here, 80 | # relative to this directory. They are copied after the builtin static files, 81 | # so a file named "default.css" will overwrite the builtin "default.css". 82 | html_static_path = ["_static"] 83 | -------------------------------------------------------------------------------- /tests/test_loaders.py: -------------------------------------------------------------------------------- 1 | import re 2 | from unittest import mock 3 | 4 | import pytest 5 | 6 | from piny import Matcher, MatcherWithDefaults, StrictMatcher, YamlLoader 7 | 8 | from . import config_directory, config_map 9 | 10 | 11 | @pytest.mark.parametrize("name", ["db", "mail"]) 12 | def test_strict_matcher_values_undefined(name): 13 | config = YamlLoader( 14 | path=config_directory.joinpath("{}.yaml".format(name)), matcher=StrictMatcher 15 | ).load() 16 | assert config[name]["password"] is None 17 | 18 | 19 | @pytest.mark.parametrize("name", ["db", "mail"]) 20 | def test_strict_matcher_values_set(name): 21 | with mock.patch("piny.matchers.StrictMatcher.constructor") as expand_mock: 22 | expand_mock.return_value = config_map[name] 23 | config = YamlLoader( 24 | path=config_directory.joinpath("{}.yaml".format(name)), 25 | matcher=StrictMatcher, 26 | ).load() 27 | assert config[name]["password"] == config_map[name] 28 | 29 | 30 | def test_strict_matcher_default_do_not_matched(): 31 | config = YamlLoader( 32 | path=config_directory.joinpath("defaults.yaml"), matcher=StrictMatcher 33 | ).load() 34 | assert config["db"]["password"] is None 35 | assert ( 36 | config["mail"]["password"] == "${MAIL_PASSWORD:-My123~!@#$%^&*())_+Password!}" 37 | ) 38 | assert config["logging"]["password"] == "${LOGGING_PASSWORD:-:-test:-}" 39 | assert config["sentry"]["password"] == "${SENTRY_PASSWORD:-}" 40 | 41 | 42 | def test_matcher_with_defaults_values_undefined(): 43 | config = YamlLoader( 44 | path=config_directory.joinpath("defaults.yaml"), matcher=MatcherWithDefaults 45 | ).load() 46 | assert config["db"]["password"] is None 47 | assert config["mail"]["password"] == "My123~!@#$%^&*())_+Password!" 48 | assert config["logging"]["password"] == ":-test:-" 49 | assert config["sentry"]["password"] == "" 50 | 51 | 52 | def test_matcher_with_defaults_values_set(): 53 | with mock.patch("piny.matchers.os.environ.get") as expand_mock: 54 | expand_mock.side_effect = lambda v, _: config_map[v.split("_")[0].lower()] 55 | config = YamlLoader( 56 | path=config_directory.joinpath("defaults.yaml"), matcher=MatcherWithDefaults 57 | ).load() 58 | assert config["db"]["password"] == config_map["db"] 59 | assert config["mail"]["password"] == config_map["mail"] 60 | assert config["sentry"]["password"] == config_map["sentry"] 61 | assert config["logging"]["password"] == config_map["logging"] 62 | 63 | 64 | def test_base_matcher(): 65 | """ 66 | WATCH OUT! Black magic of Pytest in action! 67 | 68 | When placed at the beginning of test module this test make all other tests fail. 69 | Make sure this test is at the bottom! 70 | """ 71 | with pytest.raises(NotImplementedError): 72 | with mock.patch( 73 | "piny.matchers.Matcher.matcher", new_callable=mock.PropertyMock 74 | ) as matcher_mock: 75 | matcher_mock.return_value = re.compile("") 76 | YamlLoader( 77 | path=config_directory.joinpath("defaults.yaml"), matcher=Matcher 78 | ).load() 79 | -------------------------------------------------------------------------------- /src/piny/loaders.py: -------------------------------------------------------------------------------- 1 | from typing import IO, Any, Type, Union 2 | 3 | import yaml 4 | 5 | from .errors import LoadingError 6 | from .matchers import Matcher, MatcherWithDefaults 7 | from .validators import Validator 8 | 9 | # 10 | # Loader 11 | # 12 | 13 | 14 | class YamlLoader: 15 | """ 16 | YAML configuration file loader 17 | """ 18 | 19 | def __init__( 20 | self, 21 | path: str, 22 | *, 23 | matcher: Type[Matcher] = MatcherWithDefaults, 24 | validator: Union[Type[Validator], None] = None, 25 | schema: Any = None, 26 | **schema_params, 27 | ) -> None: 28 | """ 29 | Initialize YAML loader 30 | 31 | :param path: string with path to YAML-file 32 | :param matcher: matcher class 33 | :param validator: validator class for one of the supported validation libraries 34 | :param schema: validation schema for the validator of choice 35 | :param schema_params: named arguments used as optional validation schema params 36 | """ 37 | self.path = path 38 | self.matcher = matcher 39 | self.validator = validator 40 | self.schema = schema 41 | self.schema_params = schema_params 42 | 43 | def _init_resolvers(self): 44 | self.matcher.add_implicit_resolver("!env", self.matcher.matcher, None) 45 | self.matcher.add_constructor("!env", self.matcher.constructor) 46 | 47 | def load(self, **params) -> Any: 48 | """ 49 | Return Python object loaded (optionally validated) from the YAML-file 50 | 51 | :param params: named arguments used as optional loading params in validation 52 | """ 53 | self._init_resolvers() 54 | try: 55 | with open(self.path) as fh: 56 | load = yaml.load(fh, Loader=self.matcher) 57 | except (yaml.YAMLError, FileNotFoundError) as e: 58 | raise LoadingError(origin=e, reason=str(e)) 59 | 60 | if (self.validator is not None) and (self.schema is not None): 61 | return self.validator(self.schema, **self.schema_params).load( 62 | data=load, **params 63 | ) 64 | return load 65 | 66 | 67 | class YamlStreamLoader(YamlLoader): 68 | """ 69 | YAML configuration loader for IO streams, e.g. file objects or stdin 70 | """ 71 | 72 | def __init__( 73 | self, 74 | stream: Union[str, IO[str]], 75 | *, 76 | matcher: Type[Matcher] = MatcherWithDefaults, 77 | validator: Union[Type[Validator], None] = None, 78 | schema: Any = None, 79 | **schema_params, 80 | ) -> None: 81 | self.stream = stream 82 | self.matcher = matcher 83 | self.validator = validator 84 | self.schema = schema 85 | self.schema_params = schema_params 86 | 87 | def load(self, **params) -> Any: 88 | self._init_resolvers() 89 | try: 90 | load = yaml.load(self.stream, Loader=self.matcher) 91 | except yaml.YAMLError as e: 92 | raise LoadingError(origin=e, reason=str(e)) 93 | 94 | if (self.validator is not None) and (self.schema is not None): 95 | return self.validator(self.schema, **self.schema_params).load( 96 | data=load, **params 97 | ) 98 | return load 99 | -------------------------------------------------------------------------------- /docs/usage.rst: -------------------------------------------------------------------------------- 1 | Usage 2 | ===== 3 | 4 | *Piny* loads your YAML configuration file. It optionally validates 5 | data loaded from config file. *Piny* main logic is in a loader class. 6 | You can pass arguments in the loader class to change the way YAML file 7 | is parsed and validated. 8 | 9 | .. _usage-loaders-docs: 10 | 11 | Loaders 12 | ------- 13 | 14 | ``YamlLoader`` loader class is dedicated for use in Python applications. 15 | Based on `PyYAML`_, it parses YAML files, (arguably) the most beautiful 16 | file format for configuration files! 17 | 18 | Basic loader usage is the following. 19 | 20 | 1. Set your environment variables 21 | 22 | 2. Mark up your YAML configuration file with these env names: 23 | 24 | .. literalinclude:: code/config.yaml 25 | :language: yaml 26 | 27 | 3. In your app load config with *Piny*: 28 | 29 | .. literalinclude:: code/simple_yaml_loader.py 30 | 31 | ``YamlStreamLoader`` class primary use is Piny CLI tool (see :ref:`usage-cli-docs`). 32 | But it also can be used interchargably with ``YamlLoader`` whenever IO streams 33 | are used instead of file paths. 34 | 35 | .. automodule:: piny.loaders 36 | :members: 37 | :undoc-members: 38 | :show-inheritance: 39 | 40 | .. _PyYAML: https://pypi.org/project/PyYAML/ 41 | 42 | .. _usage-matchers-docs: 43 | 44 | Matchers 45 | -------- 46 | 47 | In the :ref:`usage-loaders-docs` section we used Bash-style environment 48 | variables with defaults. You may want to discourage such envs in 49 | your project. This is where *matchers* come in handy. They apply a 50 | regular expression when parsing your YAML file that matches environment 51 | variables we want to interpolate. 52 | 53 | By default ``MatcherWithDefaults`` is used. ``StrictMatcher`` is another 54 | matcher class used for plain vanilla envs with no default values support. 55 | 56 | Both strict and default matchers return ``None`` value if environment variable 57 | matched is not set in the system. 58 | 59 | Basic usage example is the following: 60 | 61 | .. literalinclude:: code/strict_matcher.py 62 | 63 | .. automodule:: piny.matchers 64 | :members: 65 | :undoc-members: 66 | :show-inheritance: 67 | 68 | 69 | .. _usage-validators-docs: 70 | 71 | Validators 72 | ---------- 73 | 74 | Piny supports *optional* data validation using third-party libraries: 75 | `Marshmallow`_, `Pydantic`_, `Trafaret`_. 76 | 77 | In order to use data validation pass ``validator`` and ``schema`` arguments 78 | in the :ref:`usage-loaders-docs` class. You may also initialize loader class 79 | with optional named arguments that will be passed to the validator's schema. 80 | Additional loading arguments may be passed in ``load`` method invocation. 81 | 82 | .. automodule:: piny.validators 83 | :members: 84 | :undoc-members: 85 | :show-inheritance: 86 | 87 | 88 | Marshmallow validation example 89 | .............................. 90 | 91 | .. literalinclude:: code/ma_validation.py 92 | 93 | 94 | Pydantic validation example 95 | ........................... 96 | 97 | .. literalinclude:: code/pydantic_validation.py 98 | 99 | 100 | Trafaret validation example 101 | ........................... 102 | 103 | .. literalinclude:: code/trafaret_validation.py 104 | 105 | .. _Pydantic: https://pydantic-docs.helpmanual.io/ 106 | .. _Marshmallow: https://marshmallow.readthedocs.io/ 107 | .. _Trafaret: https://trafaret.readthedocs.io/ 108 | 109 | .. _usage-exceptions-docs: 110 | 111 | Exceptions 112 | ---------- 113 | 114 | ``LoadingError`` is thrown when something goes wrong with reading or 115 | parsing a YAML file. ``ValidationError`` is a wrapper for exceptions 116 | raised by the libraries for optional data validation. Original exception 117 | can be accessed by ``origin`` attribute. It comes in handy when you need 118 | more than just an original exception message (e.g. a dictionary of validation 119 | errors). 120 | 121 | Both exceptions inherit from the ``ConfigError``. 122 | 123 | .. automodule:: piny.errors 124 | :members: 125 | :undoc-members: 126 | :show-inheritance: 127 | 128 | 129 | .. _usage-cli-docs: 130 | 131 | Command line utility 132 | -------------------- 133 | 134 | Piny comes with CLI tool that substitutes the values of environment variables 135 | in input file or ``stdin`` and write result to an output file or ``stdout``. 136 | Piny CLI utility is somewhat similar to ``GNU/gettext`` `envsubst`_ but works 137 | with files too. 138 | 139 | .. click:: piny.cli:cli 140 | :prog: piny 141 | :show-nested: 142 | 143 | .. _envsubst: https://www.gnu.org/software/gettext/manual/html_node/envsubst-Invocation.html 144 | -------------------------------------------------------------------------------- /tests/test_validators.py: -------------------------------------------------------------------------------- 1 | from unittest import mock 2 | 3 | import pydantic 4 | import pytest 5 | import trafaret 6 | from marshmallow import Schema, fields 7 | from packaging import version 8 | 9 | from piny import ( 10 | MarshmallowValidator, 11 | PydanticV2Validator, 12 | PydanticValidator, 13 | StrictMatcher, 14 | TrafaretValidator, 15 | ValidationError, 16 | YamlLoader, 17 | YamlStreamLoader, 18 | ) 19 | 20 | from . import config_directory, config_map 21 | 22 | if version.parse(pydantic.__version__) >= version.parse("2.0"): 23 | PydanticValidator = PydanticV2Validator 24 | 25 | # 26 | # Const 27 | # 28 | 29 | 30 | class PydanticDB(pydantic.BaseModel): 31 | host: str 32 | login: str 33 | password: str 34 | 35 | 36 | class PydanticConfig(pydantic.BaseModel): 37 | db: PydanticDB 38 | 39 | 40 | class MarshmallowDB(Schema): 41 | host = fields.String(required=True) 42 | login = fields.String(required=True) 43 | password = fields.String(required=True) 44 | 45 | 46 | class MarshmallowConfig(Schema): 47 | db = fields.Nested(MarshmallowDB, required=True) 48 | 49 | 50 | TrafaretDB = trafaret.Dict( 51 | host=trafaret.String, login=trafaret.String, password=trafaret.String 52 | ) 53 | TrafaretConfig = trafaret.Dict(db=TrafaretDB) 54 | 55 | 56 | # 57 | # Tests 58 | # 59 | 60 | 61 | @pytest.mark.parametrize("name", ["db"]) 62 | def test_pydantic_validator_success(name): 63 | with mock.patch("piny.matchers.StrictMatcher.constructor") as expand_mock: 64 | expand_mock.return_value = config_map[name] 65 | config = YamlLoader( 66 | path=config_directory.joinpath("{}.yaml".format(name)), 67 | matcher=StrictMatcher, 68 | validator=PydanticValidator, 69 | schema=PydanticConfig, 70 | ).load() 71 | 72 | assert config[name]["host"] == "db.example.com" 73 | assert config[name]["login"] == "user" 74 | assert config[name]["password"] == config_map[name] 75 | 76 | 77 | @pytest.mark.parametrize("name", ["db"]) 78 | def test_pydantic_validator_fail(name): 79 | with pytest.raises(ValidationError, match=r"db.*password"): 80 | YamlLoader( 81 | path=config_directory.joinpath("{}.yaml".format(name)), 82 | matcher=StrictMatcher, 83 | validator=PydanticValidator, 84 | schema=PydanticConfig, 85 | ).load() 86 | 87 | 88 | @pytest.mark.parametrize("name", ["db"]) 89 | def test_marshmallow_validator_success(name): 90 | with mock.patch("piny.matchers.StrictMatcher.constructor") as expand_mock: 91 | expand_mock.return_value = config_map[name] 92 | config = YamlLoader( 93 | path=config_directory.joinpath("{}.yaml".format(name)), 94 | matcher=StrictMatcher, 95 | validator=MarshmallowValidator, 96 | schema=MarshmallowConfig, 97 | ).load() 98 | 99 | assert config[name]["host"] == "db.example.com" 100 | assert config[name]["login"] == "user" 101 | assert config[name]["password"] == config_map[name] 102 | 103 | 104 | @pytest.mark.parametrize("name", ["db"]) 105 | def test_marshmallow_validator_fail(name): 106 | with pytest.raises( 107 | ValidationError, match=r"\{'db': \{'password': \['Field may not be null.'\]\}\}" 108 | ): 109 | YamlLoader( 110 | path=config_directory.joinpath("{}.yaml".format(name)), 111 | matcher=StrictMatcher, 112 | validator=MarshmallowValidator, 113 | schema=MarshmallowConfig, 114 | ).load(many=False) 115 | 116 | 117 | @pytest.mark.parametrize("name", ["db"]) 118 | def test_marshmallow_validator_stream_success(name): 119 | with mock.patch("piny.matchers.StrictMatcher.constructor") as expand_mock: 120 | expand_mock.return_value = config_map[name] 121 | 122 | with open(config_directory.joinpath("{}.yaml".format(name)), "r") as fd: 123 | config = YamlStreamLoader( 124 | stream=fd, 125 | matcher=StrictMatcher, 126 | validator=MarshmallowValidator, 127 | schema=MarshmallowConfig, 128 | ).load() 129 | 130 | assert config[name]["host"] == "db.example.com" 131 | assert config[name]["login"] == "user" 132 | assert config[name]["password"] == config_map[name] 133 | 134 | 135 | @pytest.mark.parametrize("name", ["db"]) 136 | def test_trafaret_validator_success(name): 137 | with mock.patch("piny.matchers.StrictMatcher.constructor") as expand_mock: 138 | expand_mock.return_value = config_map[name] 139 | config = YamlLoader( 140 | path=config_directory.joinpath("{}.yaml".format(name)), 141 | matcher=StrictMatcher, 142 | validator=TrafaretValidator, 143 | schema=TrafaretConfig, 144 | ).load() 145 | 146 | assert config[name]["host"] == "db.example.com" 147 | assert config[name]["login"] == "user" 148 | assert config[name]["password"] == config_map[name] 149 | 150 | 151 | @pytest.mark.parametrize("name", ["db"]) 152 | def test_trafaret_validator_fail(name): 153 | with pytest.raises( 154 | ValidationError, 155 | match=r"Validation failed: \{'db': DataError\(\"\{'password': DataError\('value is not a string'\)\}\"\)\}", 156 | ): 157 | YamlLoader( 158 | path=config_directory.joinpath("{}.yaml".format(name)), 159 | matcher=StrictMatcher, 160 | validator=TrafaretValidator, 161 | schema=TrafaretConfig, 162 | ).load() 163 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | Piny 2 | ==== 3 | 4 | |Logo| 5 | 6 | |PyPI| |Coverage| |Downloads| |License| 7 | 8 | **Piny** is YAML config loader with environment variables interpolation for Python. 9 | 10 | Keep your app's configuration in a YAML file. 11 | Mark up sensitive data in the config as *environment variables*. 12 | Set environment variables on application deployment. 13 | Now let the *piny* load your config and substitute environment variables 14 | in it with their values. 15 | 16 | Piny is developed with Docker and Kubernetes in mind, 17 | though it's not limited to any deployment system. 18 | 19 | 20 | Rationale 21 | --------- 22 | 23 | Piny combines *readability and versioning* you get when using config files, 24 | and *security* that environment variables provide. Read more about this approach 25 | in the `blog post`_. 26 | 27 | 28 | Help 29 | ---- 30 | 31 | See `documentation`_ for more details. 32 | 33 | 34 | Installation 35 | ------------ 36 | 37 | Just run:: 38 | 39 | pip install -U piny 40 | 41 | 42 | Usage 43 | ----- 44 | 45 | Set your environment variables, add them to your YAML configuration file: 46 | 47 | .. code-block:: yaml 48 | 49 | db: 50 | login: user 51 | password: ${DB_PASSWORD} 52 | mail: 53 | login: user 54 | password: ${MAIL_PASSWORD:-my_default_password} 55 | sentry: 56 | dsn: ${VAR_NOT_SET} 57 | 58 | Then load your config: 59 | 60 | .. code-block:: python 61 | 62 | from piny import YamlLoader 63 | 64 | config = YamlLoader(path="config.yaml").load() 65 | print(config) 66 | # {'db': {'login': 'user', 'password': 'my_db_password'}, 67 | # 'mail': {'login': 'user', 'password': 'my_default_password'}, 68 | # 'sentry': {'dsn': None}} 69 | 70 | You may want to discourage Bash-style envs with defaults in your configs. 71 | In such case, use a ``StrictMatcher``: 72 | 73 | .. code-block:: python 74 | 75 | from piny import YamlLoader, StrictMatcher 76 | 77 | config = YamlLoader(path="config.yaml", matcher=StrictMatcher).load() 78 | 79 | Both strict and default matchers produce ``None`` value if environment variable 80 | matched is not set in the system (and no default syntax used in the case of 81 | default matcher). 82 | 83 | Piny also comes with *command line utility* that works both with files and standard 84 | input and output: 85 | 86 | .. code-block:: bash 87 | 88 | $ export PASSWORD=mySecretPassword 89 | $ echo "db: \${PASSWORD}" | piny 90 | db: mySecretPassword 91 | 92 | 93 | Validation 94 | ---------- 95 | 96 | Piny supports *optional* data validation using third-party libraries: 97 | `Marshmallow`_, `Pydantic`_, `Trafaret`_. 98 | 99 | .. code-block:: python 100 | 101 | import marshmallow as ma 102 | from piny import MarshmallowValidator, StrictMatcher, YamlLoader 103 | 104 | class DBSchema(ma.Schema): 105 | login = ma.fields.String(required=True) 106 | password = ma.fields.String() 107 | 108 | class ConfigSchema(ma.Schema): 109 | db = ma.fields.Nested(DBSchema) 110 | 111 | config = YamlLoader( 112 | path="database.yaml", 113 | matcher=StrictMatcher, 114 | validator=MarshmallowValidator, 115 | schema=ConfigSchema, 116 | ).load(many=False) 117 | 118 | 119 | Exceptions 120 | ---------- 121 | 122 | ``LoadingError`` is thrown when something goes wrong with reading or parsing YAML-file. 123 | ``ValidationError`` is a wrapper for exceptions raised by the libraries for optional data validation. 124 | Original exception can be accessed by ``origin`` attribute. It comes in handy when you need more than 125 | just an original exception message (e.g. a dictionary of validation errors). 126 | 127 | Both exceptions inherit from the ``ConfigError``. 128 | 129 | 130 | Best practices 131 | -------------- 132 | 133 | - Maintain a healthy security/convenience balance for your config 134 | 135 | - Mark up entity as an environment variable in your YAML if and only if 136 | it really is a *secret* (login/passwords, private API keys, crypto keys, 137 | certificates, or maybe DB hostname too? You decide) 138 | 139 | - When loading config file, validate your data. 140 | Piny supports a few popular data validation tools. 141 | 142 | - Store your config files in the version control system along with your app’s code. 143 | 144 | - Environment variables are set by whoever is responsible for the deployment. 145 | Modern orchestration systems like `Kubernetes`_ make it easier to keep envs secure 146 | (see `Kubernetes Secrets`_). 147 | 148 | 149 | Fun facts 150 | --------- 151 | 152 | *Piny* is a recursive acronym for *Piny Is Not YAML*. 153 | Not only it's a library name, but also a name for YAML marked up 154 | with environment variables. 155 | 156 | 157 | Changelog 158 | --------- 159 | 160 | See `CHANGELOG.rst`_. 161 | 162 | 163 | Contributing 164 | ------------ 165 | 166 | See `CONTRIBUTING.rst`_. 167 | 168 | .. |PyPI| image:: https://img.shields.io/pypi/v/piny 169 | :alt: PyPI 170 | :target: https://pypi.org/project/piny/ 171 | .. |Coverage| image:: https://img.shields.io/codecov/c/github/pilosus/piny.svg 172 | :alt: Codecov 173 | :target: https://codecov.io/gh/pilosus/piny 174 | .. |License| image:: https://img.shields.io/github/license/pilosus/piny.svg 175 | :alt: MIT License 176 | :target: https://github.com/pilosus/piny/blob/master/LICENSE 177 | .. |Logo| image:: https://piny.readthedocs.io/en/latest/_static/piny_logo_noborder.png 178 | :alt: Piny logo 179 | :target: https://pypi.org/project/piny/ 180 | .. |Downloads| image:: https://img.shields.io/pypi/dm/piny 181 | :alt: PyPI - Downloads 182 | :target: https://pypistats.org/packages/piny 183 | 184 | .. _blog post: https://blog.pilosus.org/blog/application-configs-files-or-environment-variables-actually-both 185 | .. _future releases: https://github.com/pilosus/piny/issues/2 186 | .. _Kubernetes: https://kubernetes.io/ 187 | .. _Kubernetes Secrets: https://kubernetes.io/docs/concepts/configuration/secret/ 188 | .. _Pydantic: https://pydantic-docs.helpmanual.io/ 189 | .. _Marshmallow: https://marshmallow.readthedocs.io/ 190 | .. _Trafaret: https://trafaret.readthedocs.io/ 191 | .. _tests: https://github.com/pilosus/piny/tree/master/tests 192 | .. _source code: https://github.com/pilosus/piny/tree/master/piny 193 | .. _coming soon: https://github.com/pilosus/piny/issues/12 194 | .. _CONTRIBUTING.rst: https://github.com/pilosus/piny/tree/master/CONTRIBUTING.rst 195 | .. _CHANGELOG.rst: https://github.com/pilosus/piny/tree/master/CHANGELOG.rst 196 | .. _documentation: https://piny.readthedocs.io/ 197 | --------------------------------------------------------------------------------