├── .envrc ├── data ├── raw │ └── .gitkeep ├── external │ └── .gitkeep ├── interim │ └── .gitkeep ├── processed │ └── .gitkeep └── README.md ├── requirements.txt ├── .python-version ├── .venv ├── .changelog ├── .gitignore └── template.md ├── .env.template ├── .github ├── workflows │ ├── constraints.txt │ ├── codeql-analysis.yml │ └── main.yaml └── dependabot.yml ├── tests ├── __init__.py └── test_folk.py ├── folk └── __init__.py ├── .vscode ├── extensions.json └── settings.json ├── CHANGELOG.md ├── docs ├── project_specific_setup.md ├── updating_requirements.md ├── quickstart.md ├── using_poetry.md ├── release_process.md ├── detect_secrets.md └── getting_started.md ├── .gitignore ├── LICENSE ├── README.md ├── pyproject.toml ├── .secrets.baseline └── .pre-commit-config.yaml /.envrc: -------------------------------------------------------------------------------- 1 | dotenv 2 | -------------------------------------------------------------------------------- /data/raw/.gitkeep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /data/external/.gitkeep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /data/interim/.gitkeep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /data/processed/.gitkeep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /.python-version: -------------------------------------------------------------------------------- 1 | 3.12.4 2 | -------------------------------------------------------------------------------- /.venv: -------------------------------------------------------------------------------- 1 | folk-music-recommender 2 | -------------------------------------------------------------------------------- /.changelog/.gitignore: -------------------------------------------------------------------------------- 1 | !.gitignore 2 | -------------------------------------------------------------------------------- /.env.template: -------------------------------------------------------------------------------- 1 | export SECRET='value' 2 | export PYTHONBREAKPOINT=ipdb.set_trace 3 | -------------------------------------------------------------------------------- /.github/workflows/constraints.txt: -------------------------------------------------------------------------------- 1 | pip==24.0 2 | poetry==1.8.3 3 | virtualenv==20.26.2 4 | -------------------------------------------------------------------------------- /tests/__init__.py: -------------------------------------------------------------------------------- 1 | """Tests should go here.""" 2 | 3 | from __future__ import annotations 4 | -------------------------------------------------------------------------------- /folk/__init__.py: -------------------------------------------------------------------------------- 1 | """Generated using modern-data-cookiecutter.""" 2 | 3 | __version__ = "0.1.0" 4 | -------------------------------------------------------------------------------- /tests/test_folk.py: -------------------------------------------------------------------------------- 1 | """Example test.""" 2 | 3 | from __future__ import annotations 4 | 5 | from folk import __version__ 6 | 7 | 8 | def test_version(): 9 | """Test example docstring.""" 10 | assert __version__ == "0.1.0" 11 | -------------------------------------------------------------------------------- /.vscode/extensions.json: -------------------------------------------------------------------------------- 1 | { 2 | "recommendations": [ 3 | "ms-python.python", 4 | "ms-python.vscode-pylance", 5 | "esbenp.prettier-vscode", 6 | "wayou.vscode-todo-highlight", 7 | "bungcip.better-toml" 8 | ] 9 | } 10 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | # Changelog 2 | 3 | All notable changes to this project will be documented in this file. 4 | 5 | The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) 6 | and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html). 7 | 8 | 9 | -------------------------------------------------------------------------------- /.github/dependabot.yml: -------------------------------------------------------------------------------- 1 | # Basic dependabot.yml file with minimum configuration 2 | 3 | version: 2 4 | updates: 5 | - package-ecosystem: github-actions 6 | directory: "/" 7 | schedule: 8 | interval: daily 9 | - package-ecosystem: pip 10 | directory: "/.github/workflows" 11 | schedule: 12 | interval: daily 13 | - package-ecosystem: pip 14 | directory: "/" 15 | schedule: 16 | interval: daily 17 | versioning-strategy: lockfile-only 18 | allow: 19 | - dependency-type: "all" 20 | -------------------------------------------------------------------------------- /docs/project_specific_setup.md: -------------------------------------------------------------------------------- 1 | # Project-specific setup 2 | 3 | This document contains setup instructions specifically for this project only. This design enables 4 | us to keep other docs easily aligned with future upstream changes to 5 | [coefficient-cookiecutter](https://github.com/CoefficientSystems/coefficient-cookiecutter/). 6 | 7 | 8 | ## Install system-level dependencies with [homebrew](https://brew.sh/) 9 | 10 | ```sh 11 | # brew install wkhtmltopdf 12 | ``` 13 | 14 | 15 | ## Jupyter kernel 16 | 17 | ```sh 18 | python -m ipykernel install --user --name folk-music-recommender --display-name "Python (folk-music-recommender)" 19 | ``` 20 | -------------------------------------------------------------------------------- /data/README.md: -------------------------------------------------------------------------------- 1 | # `data` folder overview 2 | 3 | Any data that needs to be stored locally should be saved in this location. This folder, 4 | and its sub-folders, are not version-controlled. 5 | 6 | The sub-folders should be used as follows: 7 | 8 | - `external`: any data that will not be processed at all, such as reference data; 9 | - `raw`: any raw data before any processing; 10 | - `interim`: any raw data that has been partially processed and, for whatever reason, 11 | needs to be stored before further processing is completed; and 12 | - `processed`: any raw or interim data that has been fully processed into its final 13 | state. 14 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Caches 2 | **/.ipynb_checkpoints/* 3 | **/__pycache__/* 4 | *.pyc 5 | 6 | # Data 7 | data/ 8 | data/raw/TheSession-data/ 9 | !**/data/**/README.md 10 | !**/data/**/*.gitkeep 11 | 12 | # Secrets 13 | .env 14 | 15 | # Personal TODO files 16 | TODO.txt 17 | 18 | # VisualStudioCode 19 | .vscode/* 20 | !.vscode/settings.json 21 | !.vscode/tasks.json 22 | !.vscode/launch.json 23 | !.vscode/extensions.json 24 | .RData 25 | 26 | # Environment 27 | venv/ 28 | 29 | # coverage 30 | .coverage 31 | .coverage.* 32 | htmlcov 33 | 34 | # IDE config 35 | .idea/ 36 | ipython_config.py 37 | profile_default/ 38 | 39 | # Other 40 | .DS_Store 41 | ignore/ 42 | *.log 43 | *.pem 44 | 45 | # Editor 46 | pyrightconfig.json 47 | 48 | # https://www.gitignore.io/ 49 | -------------------------------------------------------------------------------- /.changelog/template.md: -------------------------------------------------------------------------------- 1 | {% for section, _ in sections.items() %} 2 | {% if section %}## {{section}}{% endif %} 3 | 4 | {% if sections[section] %} 5 | {% for category, val in definitions.items() if category in sections[section]%} 6 | ### {{ definitions[category]['name'] }} 7 | 8 | {% if definitions[category]['showcontent'] %} 9 | {% for text, values in sections[section][category].items() %} 10 | - {{ text }} ({{ values|join(', ') }}) 11 | {% endfor %} 12 | 13 | {% else %} 14 | - {{ sections[section][category]['']|join(', ') }} 15 | 16 | {% endif %} 17 | {% if sections[section][category]|length == 0 %} 18 | No significant changes. 19 | 20 | {% else %} 21 | {% endif %} 22 | 23 | {% endfor %} 24 | {% else %} 25 | No significant changes. 26 | 27 | {% endif %} 28 | {% endfor %} 29 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2024 John Sandall 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /docs/updating_requirements.md: -------------------------------------------------------------------------------- 1 | # Updating requirements 2 | 3 | ```sh 4 | # Create feature branch 5 | poetry-sync 6 | poetryup --latest 7 | # If any issues, likely due to two "latest" packages conflicting. See example below. 8 | poetry-regenerate 9 | poetry-sync 10 | pre-commit run --all-files --hook-stage=manual 11 | poetry-sync 12 | pytest 13 | # Commit & push 14 | ``` 15 | 16 | If the latest version of `green` requires `blue (>=1.2, <1.3)` and the latest version of `blue` is 17 | `1.4` then you will encounter a `SolverProblemError`, for example: 18 | 19 | ```sh 20 | SolverProblemError 21 | 22 | Because green (0.8) depends on blue (>=1.2,<1.3) 23 | and no versions of green match (>=0.8,<1.0) requires blue (>=1.2,<1.3). 24 | So, because src depends on both blue (^1.4) and green (^0.8), version solving failed. 25 | ``` 26 | 27 | In this situation, do the following: 28 | - Comment out `blue` 29 | - Re-run `poetryup --latest` 30 | - Handle any other new package conflicts the same way until poetryup resolves 31 | - Uncomment out `blue` with package version that works with `green`, e.g. `blue = "^1.2"` 32 | - Run `poetry-regenerate` onwards 33 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Folk Music Recommender 2 | 3 | [![CI](https://github.com/john-sandall/folk-music-recommender/actions/workflows/main.yaml/badge.svg)](https://github.com/john-sandall/folk-music-recommender/actions/workflows/main.yaml) 4 | 5 | Fine Tuning: Building A Folk Music Recommendation System with LLMs 6 | 7 | ## Project cheatsheet 8 | 9 | - **pre-commit:** `pre-commit run --all-files` 10 | - **pytest:** `pytest` or `pytest -s` 11 | - **coverage:** `coverage run -m pytest` or `coverage html` 12 | - **poetry sync:** `poetry install --no-root --sync` 13 | - **updating requirements:** see [docs/updating_requirements.md](docs/updating_requirements.md) 14 | 15 | 16 | 17 | ## Initial project setup 18 | 19 | 1. See [docs/getting_started.md](docs/getting_started.md) or [docs/quickstart.md](docs/quickstart.md) 20 | for how to get up & running. 21 | 2. Check [docs/project_specific_setup.md](docs/project_specific_setup.md) for project specific setup. 22 | 3. See [docs/using_poetry.md](docs/using_poetry.md) for how to update Python requirements using 23 | [Poetry](https://python-poetry.org/). 24 | 4. See [docs/detect_secrets.md](docs/detect_secrets.md) for more on creating a `.secrets.baseline` 25 | file using [detect-secrets](https://github.com/Yelp/detect-secrets). 26 | -------------------------------------------------------------------------------- /docs/quickstart.md: -------------------------------------------------------------------------------- 1 | # Quickstart 2 | 3 | This document contains instructions _only_ to get a fully working development environment for 4 | running this repo. For pre-requisites (e.g. `pyenv` install instructions) plus details on what's 5 | being installed and why, please see [docs/getting_started.md](docs/getting_started.md). 6 | 7 | We assume the following are installed and configured: 8 | - [pyenv](https://github.com/pyenv/pyenv) 9 | - [pyenv-virtualenvwrapper](https://github.com/pyenv/pyenv-virtualenvwrapper) 10 | - [Poetry](https://python-poetry.org/docs/) 11 | - [zsh-autoswitch-virtualenv](https://github.com/MichaelAquilina/zsh-autoswitch-virtualenv) 12 | - [direnv](https://direnv.net/) 13 | - [poetry up](https://github.com/MousaZeidBaker/poetry-plugin-up) 14 | 15 | 16 | ## Part 1: Generic Python setup 17 | 18 | ```sh 19 | # Get the repo 20 | git clone ${REPO_GIT_URL} 21 | 22 | # Install Python 23 | pyenv install $(cat .python-version) 24 | pyenv shell $(cat .python-version) 25 | python -m pip install --upgrade pip 26 | python -m pip install virtualenvwrapper 27 | pyenv virtualenvwrapper 28 | 29 | # Setup the virtualenv 30 | mkvirtualenv -p python$(cat .python-version) $(cat .venv) 31 | python -V 32 | python -m pip install --upgrade pip 33 | 34 | # Install dependencies with Poetry 35 | poetry self update 36 | poetry install --no-root --sync 37 | 38 | # Create templated .env for storing secrets 39 | cp .env.template .env 40 | direnv allow 41 | 42 | # Create and audit secrets baseline 43 | # N.B. Adjust the exclusions here depending on your needs (check .pre-commit-config.yaml) 44 | detect-secrets --verbose scan \ 45 | --exclude-files 'poetry\.lock' \ 46 | --exclude-files '\.secrets\.baseline' \ 47 | --exclude-files '\.env\.template' \ 48 | --exclude-secrets 'password|ENTER_PASSWORD_HERE|INSERT_API_KEY_HERE' \ 49 | --exclude-lines 'integrity=*sha' \ 50 | > .secrets.baseline 51 | 52 | detect-secrets audit .secrets.baseline 53 | ``` 54 | 55 | 56 | ## Part 2: Project-specific setup 57 | 58 | Please check [docs/project_specific_setup.md](project_specific_setup.md) for further instructions. 59 | -------------------------------------------------------------------------------- /.vscode/settings.json: -------------------------------------------------------------------------------- 1 | { 2 | // editor & autoformatter settings 3 | "editor.bracketPairColorization.enabled": true, 4 | "editor.formatOnSave": true, 5 | "editor.guides.bracketPairs": "active", 6 | "editor.trimAutoWhitespace": true, 7 | "files.trimTrailingWhitespace": true, 8 | "prettier.enable": true, 9 | 10 | // python - other 11 | "python.languageServer": "Pylance", 12 | 13 | // python - linting & static analysis 14 | "python.analysis.extraPaths": ["folk"], 15 | "python.analysis.typeCheckingMode": "off", 16 | 17 | // Python files only 18 | "[python]": { 19 | // isort on save 20 | "editor.codeActionsOnSave": {"source.organizeImports": "explicit"}, 21 | // Stop the 'Black does not support "Format Selection"' message every 22 | // time you paste something (https://stackoverflow.com/a/63612401/3279076) 23 | "editor.formatOnPaste": false, 24 | "editor.defaultFormatter": "charliermarsh.ruff" 25 | }, 26 | 27 | // python - pytest (https://code.visualstudio.com/docs/python/testing) 28 | "python.testing.unittestEnabled": false, 29 | "python.testing.pytestEnabled": true, 30 | "python.testing.pytestArgs": ["tests"], 31 | 32 | // git 33 | "git.enabled": true, 34 | 35 | // file associations 36 | "files.associations": { 37 | "**/*.html": "html", 38 | "**/*.js": "javascript", 39 | "**/requirements{/**,*}.{txt,in}": "pip-requirements" 40 | }, 41 | 42 | // markdownlint 43 | "markdownlint.run": "onSave", 44 | "markdownlint.ignore": [], 45 | "markdownlint.config": { 46 | // MD003 - Heading style 47 | "MD003": { 48 | "style": "atx" 49 | }, 50 | // MD007 - Unordered list indentation 51 | "MD007": { 52 | "start_indented": true, 53 | "indent": 2 54 | }, 55 | // MD012 - Multiple consecutive blank lines 56 | "MD012": false, 57 | // MD022 - Headings should be surrounded by blank lines 58 | "MD022": false, 59 | // MD024 - Multiple headings with the same content 60 | "MD024": false, 61 | // MD032 - Lists should be surrounded by blank lines 62 | "MD032": false, 63 | // MD046 - Code block style 64 | "MD046": { 65 | "style": "fenced" 66 | }, 67 | "no-hard-tabs": true 68 | }, 69 | } 70 | -------------------------------------------------------------------------------- /.github/workflows/codeql-analysis.yml: -------------------------------------------------------------------------------- 1 | # For most projects, this workflow file will not need changing; you simply need 2 | # to commit it to your repository. 3 | # 4 | # You may wish to alter this file to override the set of languages analyzed, 5 | # or to provide custom queries or build logic. 6 | # 7 | # ******** NOTE ******** 8 | # We have attempted to detect the languages in your repository. Please check 9 | # the `language` matrix defined below to confirm you have the correct set of 10 | # supported CodeQL languages. 11 | # 12 | name: "CodeQL" 13 | 14 | on: 15 | push: 16 | branches: [main] 17 | pull_request: 18 | # The branches below must be a subset of the branches above 19 | branches: [main] 20 | schedule: 21 | - cron: "33 18 * * 2" 22 | 23 | jobs: 24 | analyze: 25 | name: Analyze 26 | runs-on: ubuntu-latest 27 | permissions: 28 | actions: read 29 | contents: read 30 | security-events: write 31 | 32 | strategy: 33 | fail-fast: false 34 | matrix: 35 | language: ["python"] 36 | # CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python', 'ruby' ] 37 | # Learn more about CodeQL language support at https://git.io/codeql-language-support 38 | 39 | steps: 40 | - name: Checkout repository 41 | uses: actions/checkout@v3 42 | 43 | # Initializes the CodeQL tools for scanning. 44 | - name: Initialize CodeQL 45 | uses: github/codeql-action/init@v3 46 | with: 47 | languages: ${{ matrix.language }} 48 | # If you wish to specify custom queries, you can do so here or in a config file. 49 | # By default, queries listed here will override any specified in a config file. 50 | # Prefix the list here with "+" to use these queries and those in the config file. 51 | # queries: ./path/to/local/query, your-org/your-repo/queries@main 52 | 53 | # Autobuild attempts to build any compiled languages (C/C++, C#, or Java). 54 | # If this step fails, then you should remove it and run the build manually (see below) 55 | - name: Autobuild 56 | uses: github/codeql-action/autobuild@v3 57 | 58 | # ℹ️ Command-line programs to run using the OS shell. 59 | # 📚 https://git.io/JvXDl 60 | 61 | # ✏️ If the Autobuild fails above, remove it and uncomment the following three lines 62 | # and modify them (or add more) to build your code if your project 63 | # uses a compiled language 64 | 65 | #- run: | 66 | # make bootstrap 67 | # make release 68 | 69 | - name: Perform CodeQL Analysis 70 | uses: github/codeql-action/analyze@v3 71 | -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | ########### 2 | # 📜 Poetry 3 | ########### 4 | [tool.poetry] 5 | name = "folk" 6 | version = "0.1.0" 7 | description = "Fine Tuning: Building A Folk Music Recommendation System with LLMs" 8 | authors = ["John Sandall <2884159+john-sandall@users.noreply.github.com>"] 9 | license = "UNLICENSED" 10 | classifiers = ["Private :: Do Not Upload"] 11 | packages = [ 12 | { include = "folk", from = "folk" }, 13 | # Add more packages here, e.g. 14 | # { include = "module_name", from = "folk" }, 15 | ] 16 | 17 | [tool.poetry.dependencies] 18 | python = "^3.12.4" 19 | # Everything below here is alphabetically sorted 20 | dask = "^2024.6.0" 21 | dask-ml = "^2024.4.4" 22 | distributed = "^2024.6.0" 23 | fsspec = "^2024.6.0" 24 | ipython = "^8.25.0" 25 | joblib = "^1.4.2" 26 | jupyter = "^1.0.0" 27 | jupyterlab = "^4.2.2" 28 | numpy = "^1.26.4" 29 | openai = "^1.34.0" 30 | pandas = "^2.2.2" 31 | plotly = "^5.22.0" 32 | scikit-learn = "^1.5.0" 33 | torch = "^2.3.1" 34 | torchaudio = "^2.3.1" 35 | torchvision = "^0.18.1" 36 | tpot = "^0.12.2" 37 | 38 | [tool.poetry.dev-dependencies] 39 | # Everything below here is alphabetically sorted 40 | detect-secrets = "1.2.0" 41 | ipdb = "^0.13.13" 42 | pip-audit = "^2.7.3" 43 | pre-commit = "^3.7.1" 44 | pytest = "^7.4.4" 45 | ruff = "^0.1.14" 46 | 47 | 48 | [build-system] 49 | requires = ["poetry-core>=1.0.0"] 50 | build-backend = "poetry.core.masonry.api" 51 | 52 | ############ 53 | # ✅ Linters 54 | ############ 55 | [tool.ruff] 56 | line-length = 100 57 | indent-width = 4 58 | target-version = "py312" 59 | extend-include = ["*.ipynb"] 60 | 61 | [tool.ruff.lint] 62 | select = [ 63 | "A", 64 | "AIR", 65 | "ARG", 66 | "B", 67 | "C4", 68 | "C90", 69 | "COM", 70 | # "CPY", 71 | "D", 72 | "DJ", 73 | "DTZ", 74 | "E", 75 | # "E4", "E7", "E9", 76 | "ERA", 77 | "EXE", 78 | "F", 79 | "FA", 80 | "FIX", 81 | "FLY", 82 | # "FURB", 83 | "G", 84 | "I", 85 | "ICN", 86 | "INP", 87 | "INT", 88 | "ISC", 89 | # "LOG", 90 | "N", 91 | "NPY", 92 | "PD", 93 | "PERF", 94 | "PGH", 95 | "PIE", 96 | "PL", 97 | "PT", 98 | "PTH", 99 | "PYI", 100 | "Q", 101 | "RET", 102 | "RSE", 103 | "RUF", 104 | "SIM", 105 | "SLF", 106 | "SLOT", 107 | "T10", 108 | "T20", 109 | "TCH", 110 | "TD", 111 | "TID", 112 | "TRY", 113 | "UP", 114 | "W", 115 | ] 116 | fixable = ["ALL"] 117 | 118 | [tool.ruff.lint.pydocstyle] 119 | convention = "pep257" 120 | 121 | [tool.ruff.format] 122 | quote-style = "double" 123 | indent-style = "space" 124 | 125 | ############## 126 | # 📣 Towncrier 127 | ############## 128 | -------------------------------------------------------------------------------- /docs/using_poetry.md: -------------------------------------------------------------------------------- 1 | # Updating Python requirements using Poetry 2 | 3 | This project uses Poetry to manage dependencies: 4 | [https://python-poetry.org/docs/](https://python-poetry.org/docs/) 5 | 6 | The Poetry virtualenv should activate automatically if you are using 7 | [zsh-autoswitch-virtualenv](https://github.com/MichaelAquilina/zsh-autoswitch-virtualenv). You can 8 | also activate it manually by running `workon folk-music-recommender` (preferred) or `poetry shell` (Poetry 9 | created/managed virtualenv). 10 | 11 | 12 | ## Poetry tl;dr 13 | 14 | ```bash 15 | # Sync your environment with poetry.lock 16 | poetry install --no-root --sync 17 | 18 | # Add and install a new package into your environment (or update an existing one) 19 | # 1. Add a package to pyproject.toml 20 | # 2. Run this 21 | poetry add package@version 22 | 23 | # N.B. We lock & export to requirements when you run `pre-commit run --all-files` so you 24 | # shouldn't need to run the commands below yourself. 25 | 26 | # Compile all poetry.lock file 27 | poetry update 28 | 29 | # Export to requirements.txt (this is for compatibility with team members who may not 30 | # have Poetry; Poetry itself uses poetry.lock when you run `poetry install`) 31 | poetry export -f requirements.txt --output requirements.txt 32 | ``` 33 | 34 | 35 | ## Poetry FAQ 36 | 37 | **How do I sync my environment with the latest `poetry.lock`?** 38 | To install dependencies, simply `cd backend-security` and run `poetry install --no-root --sync`. 39 | This will resolve & install all deps in `pyproject.toml` via `poetry.lock`. Options: 40 | - `--no-root` skips installing the repo itself as a Python package 41 | - `--sync` removes old dependencies no longer present in the lock file 42 | 43 | You may wish to set an alias e.g. `alias poetry-sync='poetry install --no-root --sync'` 44 | 45 | **How do I add/change packages?** 46 | In order to install a new package or remove an old one, please edit `pyproject.toml` 47 | or use either `poetry add package` or `pipx run poetry add package@version` as desired. 48 | 49 | **How do I update `poetry.lock` to match new changes in `pyproject.toml`?** 50 | You can run `poetry update`, although this will also get run when you run pre-commit. This fetches 51 | the latest matching versions (according to `pyproject.toml`) and updates `poetry.lock`, and is 52 | equivalent to deleting `poetry.lock` and running `poetry install --no-root` again from scratch. 53 | 54 | **What do `~`, `^` and `*` mean?** 55 | Check out the Poetry docs page for [specifying version constraints](https://python-poetry.org/docs/dependency-specification/#version-constraints). 56 | Environment markers are great, e.g. this means "for Python 2.7.* OR Windows, install pathlib2 >=2.2, <3.0" 57 | `pathlib2 = { version = "^2.2", markers = "python_version ~= '2.7' or sys_platform == 'win32'" }` 58 | -------------------------------------------------------------------------------- /.github/workflows/main.yaml: -------------------------------------------------------------------------------- 1 | name: CI 2 | 3 | on: 4 | push: 5 | branches: [main] 6 | pull_request: 7 | workflow_dispatch: 8 | 9 | concurrency: 10 | group: ${{ github.head_ref || github.run_id }} 11 | cancel-in-progress: true 12 | 13 | jobs: 14 | build: 15 | runs-on: ubuntu-latest 16 | timeout-minutes: 30 17 | strategy: 18 | matrix: 19 | python-version: ["3.12"] 20 | 21 | steps: 22 | - uses: actions/checkout@v4 23 | 24 | - name: Set up Python ${{ matrix.python-version }} 25 | uses: actions/setup-python@v5 26 | with: 27 | python-version: ${{ matrix.python-version }} 28 | 29 | - name: Create virtualenv 30 | run: | 31 | which python 32 | python -m venv venv 33 | source venv/bin/activate 34 | python -m pip install --constraint=.github/workflows/constraints.txt --upgrade pip 35 | which python 36 | 37 | - name: Set up Poetry cache 38 | uses: actions/cache@v3.2.6 39 | with: 40 | path: venv 41 | key: venv-${{ matrix.python-version }}-${{ hashFiles('poetry.lock') }} 42 | 43 | - name: Install Poetry and Python dependencies 44 | run: | 45 | curl -sSL https://install.python-poetry.org | python3 - 46 | poetry --version 47 | poetry config virtualenvs.in-project true 48 | poetry config virtualenvs.create false 49 | poetry config virtualenvs.path venv 50 | source venv/bin/activate 51 | which python 52 | poetry install --no-root 53 | 54 | - name: Compute pre-commit cache key 55 | id: pre-commit-cache 56 | shell: python 57 | run: | 58 | import hashlib 59 | import sys 60 | python = "py{}.{}".format(*sys.version_info[:2]) 61 | payload = sys.version.encode() + sys.executable.encode() 62 | digest = hashlib.sha256(payload).hexdigest() 63 | result = "${{ runner.os }}-{}-{}-pre-commit".format(python, digest[:8]) 64 | print("::set-output name=result::{}".format(result)) 65 | 66 | - name: Restore pre-commit cache 67 | uses: actions/cache@v3.2.6 68 | with: 69 | path: ~/.cache/pre-commit 70 | key: ${{ steps.pre-commit-cache.outputs.result }}-${{ hashFiles('.pre-commit-config.yaml') }} 71 | restore-keys: | 72 | ${{ steps.pre-commit-cache.outputs.result }}- 73 | 74 | - name: pre-commit 75 | run: | 76 | source venv/bin/activate 77 | pre-commit run --hook-stage=manual --show-diff-on-failure --all-files 78 | 79 | # - name: mypy 80 | # run: mypy . 81 | 82 | - name: pytest 83 | # run: pytest --junitxml=junit.xml --cov --cov-report=term-missing:skip-covered --cov-report=xml 84 | run: | 85 | source venv/bin/activate 86 | pytest 87 | -------------------------------------------------------------------------------- /.secrets.baseline: -------------------------------------------------------------------------------- 1 | { 2 | "version": "1.2.0", 3 | "plugins_used": [ 4 | { 5 | "name": "ArtifactoryDetector" 6 | }, 7 | { 8 | "name": "AWSKeyDetector" 9 | }, 10 | { 11 | "name": "AzureStorageKeyDetector" 12 | }, 13 | { 14 | "name": "Base64HighEntropyString", 15 | "limit": 4.5 16 | }, 17 | { 18 | "name": "BasicAuthDetector" 19 | }, 20 | { 21 | "name": "CloudantDetector" 22 | }, 23 | { 24 | "name": "GitHubTokenDetector" 25 | }, 26 | { 27 | "name": "HexHighEntropyString", 28 | "limit": 3.0 29 | }, 30 | { 31 | "name": "IbmCloudIamDetector" 32 | }, 33 | { 34 | "name": "IbmCosHmacDetector" 35 | }, 36 | { 37 | "name": "JwtTokenDetector" 38 | }, 39 | { 40 | "name": "KeywordDetector", 41 | "keyword_exclude": "" 42 | }, 43 | { 44 | "name": "MailchimpDetector" 45 | }, 46 | { 47 | "name": "NpmDetector" 48 | }, 49 | { 50 | "name": "PrivateKeyDetector" 51 | }, 52 | { 53 | "name": "SendGridDetector" 54 | }, 55 | { 56 | "name": "SlackDetector" 57 | }, 58 | { 59 | "name": "SoftlayerDetector" 60 | }, 61 | { 62 | "name": "SquareOAuthDetector" 63 | }, 64 | { 65 | "name": "StripeDetector" 66 | }, 67 | { 68 | "name": "TwilioKeyDetector" 69 | } 70 | ], 71 | "filters_used": [ 72 | { 73 | "path": "detect_secrets.filters.allowlist.is_line_allowlisted" 74 | }, 75 | { 76 | "path": "detect_secrets.filters.common.is_ignored_due_to_verification_policies", 77 | "min_level": 2 78 | }, 79 | { 80 | "path": "detect_secrets.filters.heuristic.is_indirect_reference" 81 | }, 82 | { 83 | "path": "detect_secrets.filters.heuristic.is_likely_id_string" 84 | }, 85 | { 86 | "path": "detect_secrets.filters.heuristic.is_lock_file" 87 | }, 88 | { 89 | "path": "detect_secrets.filters.heuristic.is_not_alphanumeric_string" 90 | }, 91 | { 92 | "path": "detect_secrets.filters.heuristic.is_potential_uuid" 93 | }, 94 | { 95 | "path": "detect_secrets.filters.heuristic.is_prefixed_with_dollar_sign" 96 | }, 97 | { 98 | "path": "detect_secrets.filters.heuristic.is_sequential_string" 99 | }, 100 | { 101 | "path": "detect_secrets.filters.heuristic.is_swagger_file" 102 | }, 103 | { 104 | "path": "detect_secrets.filters.heuristic.is_templated_secret" 105 | }, 106 | { 107 | "path": "detect_secrets.filters.regex.should_exclude_file", 108 | "pattern": [ 109 | "poetry\\.lock", 110 | "\\.secrets\\.baseline", 111 | "\\.env\\.template" 112 | ] 113 | }, 114 | { 115 | "path": "detect_secrets.filters.regex.should_exclude_line", 116 | "pattern": [ 117 | "integrity=*sha" 118 | ] 119 | }, 120 | { 121 | "path": "detect_secrets.filters.regex.should_exclude_secret", 122 | "pattern": [ 123 | "password|ENTER_PASSWORD_HERE|INSERT_API_KEY_HERE" 124 | ] 125 | } 126 | ], 127 | "results": {}, 128 | "generated_at": "2024-06-14T22:50:58Z" 129 | } 130 | -------------------------------------------------------------------------------- /docs/release_process.md: -------------------------------------------------------------------------------- 1 | # Release Process 2 | 3 | > This process follows [this Git Flow release model](https://nvie.com/posts/a-successful-git-branching-model/). 4 | > There is an excellent [Git Flow cheatsheet here](https://danielkummer.github.io/git-flow-cheatsheet/). 5 | > You will need to install the Git Flow helper extension: `brew install git-flow-avh` and 6 | > initialise your repo (see the cheatsheet above, or this can also be done from within Sourcetree). 7 | 8 | **Replace VERSION with the relevant milestone, e.g. `0.3`** 9 | 10 | 11 | ## 1. CLI Steps 12 | 13 | Sourcetree has great built-in support for Git Flow from the `Repository > Git flow` menu. We list 14 | the commands below, but this can also be done via the GUI if you prefer. 15 | 16 | ```sh 17 | # Ensure your working copy is clean before starting (e.g. stash any WIP). 18 | # Fetch & pull latest origin/develop 19 | git fetch && git checkout develop && git pull 20 | 21 | # Locally, start a new release 22 | git flow release start VERSION 23 | 24 | # Summary of actions: 25 | # - A new branch 'release/VERSION' was created, based on 'develop' 26 | # - You are now on branch 'release/VERSION' 27 | 28 | # Follow-up actions: 29 | # - Bump the version number now! 30 | # - Start committing last-minute fixes in preparing your release (e.g. towncrier) 31 | # - Use amend commits here if possible to keep commits to a minimum 32 | # git commit --amend -m "updated commit message" 33 | # - You don't have to push the release branch unless a) you'd like it reviewed, b) to run CI, c) others may wish to add commits to this release. 34 | 35 | # Towncrier update 36 | # - Review all towncrier entries. 37 | # - Any missing vs issues closed within the milestone (don't forget bugs & maintenance)? Do the entries look good? 38 | # - (Optional) towncrier preview: `towncrier build --version=VERSION --draft` 39 | # - Publish towncrier update: `towncrier build --version=VERSION` 40 | # - Add any additional notes to the CHANGELOG.md as required 41 | 42 | # Update project version number 43 | # - This is in [poetry] section at the top of pyproject.toml 44 | 45 | # (Optional) pre-commit 46 | # - You shouldn't need to run pre-commit unless you've changed things manually 47 | # - If you're seeing changes to poetry.lock, try clearing your Poetry cache & run again 48 | poetry cache clear pypi --all 49 | pre-commit run --all-files --hook-stage=manual 50 | 51 | # Commit & amend commit as required 52 | 53 | # (Optional) If others have commits to add to this release, you can push as follows 54 | git flow release publish VERSION 55 | 56 | # Complete the release by merging back into `main` and `develop` 57 | # - Add -k if you do not want to auto-delete the release branch 58 | # - Add -p if you want to auto-push to origin 59 | # - Just use "Release VERSION" as commit messages 60 | git flow release finish -n VERSION 61 | 62 | # Summary of actions: 63 | # - Release branch 'release/VERSION' has been merged into 'main' 64 | # - Master branch 'main' has been back-merged into 'develop' 65 | # - Release branch 'release/VERSION' is still locally available 66 | # - You are now on branch 'develop' 67 | 68 | # Tag the release 69 | git checkout main 70 | git tag VERSION 71 | git push origin --tags 72 | 73 | # Check everything over, if you're happy, push `develop`, push `main` and delete your release branch. 74 | git checkout develop && git push 75 | git checkout main && git push 76 | git branch -D release/VERSION 77 | ``` 78 | 79 | ## 2. GitHub Steps 80 | 81 | - Copy the **raw markdown** for the release notes in CHANGELOG: [https://github.com/CoefficientSystems/coefficient-cookiecutter/blob/main/CHANGELOG.md] 82 | - Once you've pushed the tag, you will see it on this page: [https://github.com/CoefficientSystems/coefficient-cookiecutter/tags] 83 | - Edit the tag and add the release notes 84 | - You will then see the release appear here: [https://github.com/CoefficientSystems/coefficient-cookiecutter/releases] 85 | - This also sends an email update to anyone on the team who has subscribed containing formatted release notes. 86 | - Once the release is created, edit the release and assign the milestone to the release. Save changes. 87 | 88 | To finish, copy the release notes and post in any relevant Slack channel or email lists to inform members about the release. 89 | -------------------------------------------------------------------------------- /.pre-commit-config.yaml: -------------------------------------------------------------------------------- 1 | # Update all versions in this file by running: 2 | # $ pre-commit autoupdate 3 | minimum_pre_commit_version: 3.3.2 4 | default_language_version: 5 | python: python3.12 6 | repos: 7 | - repo: https://github.com/pre-commit/pre-commit-hooks 8 | rev: v4.5.0 9 | hooks: 10 | - id: end-of-file-fixer 11 | name: Check for a blank line at the end of scripts (auto-fixes) 12 | - id: trailing-whitespace 13 | - id: check-builtin-literals 14 | - id: check-byte-order-marker 15 | - id: check-case-conflict 16 | - id: check-merge-conflict 17 | - id: check-symlinks 18 | - id: check-toml 19 | - id: check-vcs-permalinks 20 | - id: check-xml 21 | - id: debug-statements 22 | - id: detect-private-key 23 | - id: mixed-line-ending 24 | - id: fix-encoding-pragma 25 | args: ["--remove"] 26 | - id: check-yaml 27 | - id: check-added-large-files 28 | name: Check for files larger than 5 MB 29 | args: ["--maxkb=5120"] 30 | - id: check-ast 31 | - id: check-docstring-first 32 | - id: name-tests-test 33 | args: ["--django"] 34 | 35 | - repo: https://github.com/astral-sh/ruff-pre-commit 36 | rev: v0.1.14 37 | hooks: 38 | # Run the formatter. 39 | - id: ruff-format 40 | types_or: [python, pyi, jupyter] 41 | 42 | # Run the linter. 43 | - id: ruff 44 | types_or: [python, pyi, jupyter] 45 | args: [--fix] 46 | 47 | - repo: https://github.com/Yelp/detect-secrets 48 | rev: v1.4.0 49 | hooks: 50 | - id: detect-secrets 51 | name: detect-secrets - Detect secrets in staged code 52 | args: [ 53 | "--baseline", 54 | ".secrets.baseline", 55 | # https://regex101.com/ 56 | "--exclude-files 'poetry\\.lock'", 57 | "--exclude-files '\\.secrets\\.baseline'", 58 | "--exclude-files '\\.env\\.template'", 59 | # "--exclude-files '.*\\.ipynb$'", 60 | # "--exclude-files '.*build/'", 61 | "--exclude-secrets 'password|ENTER_PASSWORD_HERE|INSERT_API_KEY_HERE'", 62 | "--exclude-lines 'integrity=*sha'", 63 | ] 64 | # https://pre-commit.com/#regular-expressions 65 | exclude: | 66 | (?x)^( 67 | poetry\.lock| 68 | \.secrets\.baseline| 69 | \.env\.template 70 | )$ 71 | # - repo: https://github.com/python-poetry/poetry 72 | # rev: 1.4.0 73 | # hooks: 74 | # - id: poetry-check 75 | # - id: poetry-lock 76 | # args: ["--no-update"] 77 | # - id: poetry-export 78 | # args: 79 | # [ 80 | # "-f", 81 | # "requirements.txt", 82 | # "-o", 83 | # "requirements.txt", 84 | # "--without-hashes", 85 | # ] 86 | - repo: https://github.com/pre-commit/mirrors-prettier 87 | rev: v4.0.0-alpha.8 88 | hooks: 89 | - id: prettier 90 | types_or: [yaml] 91 | additional_dependencies: 92 | - "prettier@2.8.4" 93 | - repo: local 94 | hooks: 95 | - id: pip-audit 96 | name: pip-audit 97 | entry: pip-audit 98 | # Exclusions example 99 | # entry: pip-audit --ignore-vuln PYSEC-2022-42974 --ignore-vuln GHSA-w596-4wvx-j9j6 100 | language: system 101 | always_run: true 102 | pass_filenames: false 103 | stages: [manual] 104 | # TODO: Remove once this is resolved 105 | # https://github.com/python-poetry/poetry/issues/2765 106 | # https://github.com/python-poetry/poetry/issues/7075 107 | - id: poetry-lock 108 | name: poetry-lock 109 | # entry: bash -c 'poetry lock && python add_markers.py' 110 | entry: bash -c 'poetry lock --no-update' 111 | language: system 112 | always_run: true 113 | pass_filenames: false 114 | - id: poetry-export 115 | name: poetry-export 116 | entry: poetry export -f requirements.txt --output requirements.txt --without-hashes 117 | language: system 118 | always_run: true 119 | pass_filenames: false 120 | -------------------------------------------------------------------------------- /docs/detect_secrets.md: -------------------------------------------------------------------------------- 1 | # Using the `detect-secrets` pre-commit hook 2 | 3 | [We use `detect-secrets` to check that no secrets are accidentally committed](detect-secrets). The 4 | `detect-secrets` package does its best to prevent accidental committing of secrets, but it may miss 5 | things. Instead, focus on good software development practices! See the [definition of a secret for 6 | further information](#definition-of-a-secret-according-to-detect-secrets). 7 | 8 | 9 | ## Initial setup 10 | 11 | This pre-commit hook requires you to generate a baseline file if one is not already present within 12 | the root directory. To create the baseline file, run the following at the root of the repository: 13 | 14 | ```shell 15 | detect-secrets scan > .secrets.baseline 16 | ``` 17 | 18 | Next, audit the baseline that has been generated by running: 19 | 20 | ```shell 21 | detect-secrets audit .secrets.baseline 22 | ``` 23 | 24 | When you run this command, you'll enter an interactive console. This will present you with a list 25 | of high-entropy string and/or anything which could be a secret. It will then ask you to verify 26 | whether this is the case. This allows the hook to remember false positives in the future, and alert 27 | you to new secrets. 28 | 29 | 30 | ## Definition of a "secret" according to `detect-secrets` 31 | 32 | The `detect-secrets` documentation, as of January 2021, says it works: 33 | 34 | > ...by running periodic diff outputs against heuristically crafted \[regular 35 | > expression\] statements, to identify whether any new secret has been committed. 36 | 37 | This means it uses regular expression patterns to scan your code changes for anything that looks 38 | like a secret according to the patterns. By definition, there are only a limited number of 39 | patterns, so the `detect-secrets` package cannot detect every conceivable type of secret. 40 | 41 | To understand what types of secrets will be detected, read the `detect-secrets` documentation on 42 | caveats, and the list of supported plugins. Also, you should use secret variable names with words 43 | that will trip the KeywordDetector plugin; see the 44 | [`DENYLIST` variable for the full list of words][detect-secrets-keyword-detector]. 45 | 46 | 47 | ## If `pre-commit` detects secrets during commit 48 | 49 | If `pre-commit` detects any secrets when you try to create a commit, it will detail what it found 50 | and where to go to check the secret. 51 | 52 | If the detected secret is a false positive, there are two options to resolve this, and prevent your 53 | commit from being blocked: 54 | 55 | - [inline allowlisting of false positives (recommended)](#inline-allowlisting-recommended); or 56 | - [updating the `.secrets.baseline` to include the false positives](#updating-secretsbaseline). 57 | 58 | In either case, if an actual secret is detected (or a combination of actual secrets and false 59 | positives), first remove the actual secret. Then following either of these processes. 60 | 61 | ### Inline allowlisting (recommended) 62 | 63 | To exclude a false positive, add a `pragma` comment such as: 64 | 65 | ```python 66 | secret = "Password123" # pragma: allowlist secret 67 | ``` 68 | 69 | or 70 | 71 | ```python 72 | # pragma: allowlist nextline secret 73 | secret = "Password123" 74 | ``` 75 | 76 | If the detected secret is actually a secret (or other sensitive information), remove the secret and 77 | re-commit; there is no need to add any `pragma` comments. 78 | 79 | If your commit contains a mixture of false positives and actual secrets, remove the actual secrets 80 | first before adding `pragma` comments to the false positives. 81 | 82 | 83 | ### Updating `.secrets.baseline` 84 | 85 | To exclude a false positive, you can also [update the `.secrets.baseline` by repeating the same two 86 | commands as in the initial setup](#using-the-detect-secrets-pre-commit-hook). 87 | 88 | During auditing, if the detected secret is actually a secret (or other sensitive information), 89 | remove the secret and re-commit. There is no need to update the `.secrets.baseline` file in this 90 | case. 91 | 92 | If your commit contains a mixture of false positives and actual secrets, remove the actual secrets 93 | first before updating and auditing the `.secrets.baseline` file. 94 | 95 | 96 | [detect-secrets]: https://github.com/Yelp/detect-secrets 97 | [detect-secrets-plugins]: https://github.com/Yelp/detect-secrets#currently-supported-plugins 98 | 99 | --- 100 | 101 | Credit: This document has been adapted from the 102 | [govukcookiecutter](https://github.com/best-practice-and-impact/govcookiecutter/). 103 | -------------------------------------------------------------------------------- /docs/getting_started.md: -------------------------------------------------------------------------------- 1 | # Getting Started 2 | 3 | This document contains instructions to get a fully working development environment for running this repo. 4 | 5 | 6 | ## 1. pyenv 7 | 8 | Install here: [https://github.com/pyenv/pyenv#homebrew-on-macos] 9 | 10 | Configure by adding the following to your `~/.zshrc` or equivalent: 11 | 12 | ```sh 13 | # Pyenv environment variables 14 | export PYENV_ROOT="$HOME/.pyenv" 15 | export PATH="$PYENV_ROOT/bin:$PATH" 16 | 17 | # Pyenv initialization 18 | eval "$(pyenv init --path)" 19 | eval "$(pyenv init -)" 20 | ``` 21 | 22 | Basic usage: 23 | 24 | ```sh 25 | # Check Python versions 26 | pyenv install --list 27 | 28 | # Install the Python version defined in this repo 29 | pyenv install $(cat .python-version) 30 | 31 | # See installed Python versions 32 | pyenv versions 33 | ``` 34 | 35 | 36 | ## 2. [pyenv-virtualenvwrapper](https://github.com/pyenv/pyenv-virtualenvwrapper) 37 | 38 | ```sh 39 | # Install with homebrew (recommended if you installed pyenv with homebrew) 40 | brew install pyenv-virtualenvwrapper 41 | ``` 42 | 43 | Configure by adding the following to your `~/.zshrc` or equivalent: 44 | 45 | ```sh 46 | # pyenv-virtualenvwrapper 47 | export PYENV_VIRTUALENVWRAPPER_PREFER_PYVENV="true" 48 | export WORKON_HOME=$HOME/.virtualenvs 49 | export PROJECT_HOME=$HOME/code # <- change this to wherever you store your repos 50 | export VIRTUALENVWRAPPER_PYTHON=$HOME/.pyenv/shims/python 51 | pyenv virtualenvwrapper_lazy 52 | ``` 53 | 54 | Test everything is working by opening a new shell (e.g. new Terminal window): 55 | 56 | ```sh 57 | # Change to the Python version you just installed 58 | pyenv shell $(cat .python-version) 59 | # This only needs to be run once after installing a new Python version through pyenv 60 | # in order to initialise virtualenvwrapper for this Python version 61 | python -m pip install --upgrade pip 62 | python -m pip install virtualenvwrapper 63 | pyenv virtualenvwrapper_lazy 64 | 65 | # Create test virtualenv (if this doesn't work, try sourcing ~/.zshrc or opening new shell) 66 | mkvirtualenv venv_test 67 | which python 68 | python -V 69 | 70 | # Deactivate & remove test virtualenv 71 | deactivate 72 | rmvirtualenv venv_test 73 | ``` 74 | 75 | 76 | ## 3. Get the repo & initialise the repo environment 77 | 78 | ⚠️ N.B. You should replace `REPO_GIT_URL` here with your actual URL to your GitHub repo. 79 | 80 | ```sh 81 | git clone ${REPO_GIT_URL} 82 | pyenv shell $(cat .python-version) 83 | 84 | # Make a new virtual environment using the Python version & environment name specified in the repo 85 | mkvirtualenv -p python$(cat .python-version) $(cat .venv) 86 | python -V # check this is the correct version of Python 87 | python -m pip install --upgrade pip 88 | ``` 89 | 90 | 91 | ## 4. Install Python requirements into the virtual environment using [Poetry](https://python-poetry.org/docs/) 92 | 93 | Install Poetry onto your system by following the instructions here: [https://python-poetry.org/docs/] 94 | 95 | Note that Poetry "lives" outside of project/environment, and if you follow the recommended install 96 | process it will be installed isolated from the rest of your system. 97 | 98 | ```sh 99 | # Update Poetry regularly as you would any other system-level tool. Poetry is environment agnostic, 100 | # it doesn't matter if you run this command inside/outside the virtualenv. 101 | poetry self update 102 | 103 | # This command should be run inside the virtualenv. 104 | poetry install --no-root --sync 105 | ``` 106 | 107 | 108 | ## 5. [zsh-autoswitch-virtualenv](https://github.com/MichaelAquilina/zsh-autoswitch-virtualenv) 109 | 110 | Download with `git clone "https://github.com/MichaelAquilina/zsh-autoswitch-virtualenv.git" "$ZSH_CUSTOM/plugins/autoswitch_virtualenv"` 111 | 112 | Configure by adding the following to your `~/.zshrc` or equivalent: 113 | 114 | ```sh 115 | # Find line containing plugins=(git) and replace with below 116 | plugins=(git autoswitch_virtualenv) 117 | ``` 118 | 119 | Check it's working by cd-ing into & out of the repo. The environment should load & unload respectively. 120 | 121 | 122 | ## 6. Install [Poetry Up](https://github.com/MousaZeidBaker/poetry-plugin-up) 123 | 124 | This is a useful Poetry plugin that updates dependencies and bumps their versions in the 125 | pyproject.toml file. The version constraints are respected, unless the `--latest` flag is passed, in 126 | which case dependencies are updated to latest available compatible versions. 127 | 128 | ```sh 129 | # Installation 130 | poetry self add poetry-plugin-up 131 | 132 | # Usage 133 | poetry up 134 | poetry up --latest 135 | ``` 136 | 137 | 138 | ## 7. Add secrets into .env 139 | 140 | - Run `cp .env.template .env` and update the secrets. 141 | - [Install direnv](https://direnv.net/) to autoload environment variables specified in `.env` 142 | - Run `direnv allow` to authorise direnv to load the secrets from `.env` into the environment 143 | (these will unload when you `cd` out of the repo; note that you will need to re-run this 144 | command whenever you change `.env`) 145 | 146 | 147 | ## 8. Initialise the `detect-secrets` pre-commit hook 148 | 149 | We use [`detect-secrets`](https://github.com/Yelp/detect-secrets) to check that no secrets are 150 | accidentally committed. Please read [docs/detect_secrets.md](docs/detect_secrets.md) for more information. 151 | 152 | 153 | ```shell 154 | # Generate a baseline 155 | detect-secrets scan > .secrets.baseline 156 | 157 | # You may want to check/amend the exclusions in `.pre-commit-config.yaml` e.g. 158 | detect-secrets --verbose scan \ 159 | --exclude-files 'poetry\.lock' \ 160 | --exclude-files '\.secrets\.baseline' \ 161 | --exclude-files '\.env\.template' \ 162 | --exclude-secrets 'password|ENTER_PASSWORD_HERE|INSERT_API_KEY_HERE' \ 163 | --exclude-lines 'integrity=*sha' \ 164 | > .secrets.baseline 165 | 166 | # Audit the generated baseline 167 | detect-secrets audit .secrets.baseline 168 | ``` 169 | 170 | When you run this command, you'll enter an interactive console. This will present you with a list 171 | of high-entropy string and/or anything which could be a secret. It will then ask you to verify 172 | whether this is the case. This allows the hook to remember false positives in the future, and alert 173 | you to new secrets. 174 | 175 | 176 | ## 9. Project-specific setup 177 | 178 | Please check [docs/project_specific_setup.md](docs/project_specific_setup.md) for further instructions. 179 | --------------------------------------------------------------------------------