├── tests ├── __init__.py ├── e2e │ └── __init__.py ├── unit │ ├── __init__.py │ ├── services │ │ ├── __init__.py │ │ ├── test_nav.py │ │ └── test_semantic_changes.py │ ├── test_version.py │ └── test_model.py ├── integration │ └── __init__.py ├── assets │ └── test_data │ │ ├── requirements.txt │ │ ├── docs │ │ ├── img │ │ │ └── helm_logo.png │ │ ├── coding │ │ │ ├── python │ │ │ │ └── gitpython.md │ │ │ └── tdd.md │ │ ├── devops │ │ │ ├── helm │ │ │ │ └── helm.md │ │ │ └── devops.md │ │ ├── index.md │ │ ├── emojis.md │ │ └── botany │ │ │ └── trees.md │ │ └── mkdocs.yml └── conftest.py ├── src └── mkdocs_newsletter │ ├── py.typed │ ├── services │ ├── __init__.py │ ├── git.py │ ├── rss.py │ └── nav.py │ ├── adapters │ └── __init__.py │ ├── entrypoints │ ├── __init__.py │ └── mkdocs_plugin.py │ ├── __init__.py │ ├── version.py │ ├── templates │ ├── newsletter_landing_page.j2 │ ├── marcos.j2 │ ├── newsletter_article.j2 │ └── rss.xml.j2 │ └── model.py ├── docs ├── examples │ └── simple-example.py ├── reference.md ├── img │ └── screencast.gif ├── theme │ └── assets │ │ └── images │ │ ├── bluebook.bmp │ │ ├── misc.svg │ │ ├── stackexchange.svg │ │ ├── dropbox.svg │ │ ├── pydo.svg │ │ ├── patreon.svg │ │ ├── mega.svg │ │ ├── wired.svg │ │ ├── hn.svg │ │ ├── image.svg │ │ ├── txt.svg │ │ ├── video.svg │ │ ├── archive.svg │ │ ├── google-scholar.svg │ │ ├── chi-dna.svg │ │ ├── audio.svg │ │ ├── spreadsheet.svg │ │ ├── amazon.svg │ │ ├── github.svg │ │ ├── worddoc.svg │ │ ├── twitter.svg │ │ ├── theguardian.svg │ │ ├── newyorktimes.svg │ │ ├── openai.svg │ │ ├── code.svg │ │ ├── erowid.svg │ │ ├── internetarchive.svg │ │ ├── csv.svg │ │ ├── gitea.svg │ │ ├── uptontea.svg │ │ ├── reddit.svg │ │ ├── washingtonpost.svg │ │ ├── miri.svg │ │ ├── plos.svg │ │ ├── bluebook.svg │ │ ├── thenewyorker.svg │ │ ├── deepmind.svg │ │ ├── kubernetes.svg │ │ └── nlm-ncbi.svg ├── stylesheets │ └── extra.css ├── adr │ ├── 00X-changelog_support.md │ ├── 002-initial_plugin_design.md │ ├── adr.md │ ├── 006-rss_feed_creation.md │ ├── 005-create_the_newsletter_articles.md │ ├── 004-article_newsletter_structure.md │ ├── 003-select_the_changes_to_record.md │ └── 001-initial_approach.md ├── usage.md ├── index.md ├── contributing.md └── install.md ├── .github ├── PULL_REQUEST_TEMPLATE.md ├── ISSUE_TEMPLATE │ ├── question.md │ ├── feature_request.md │ └── bug.md ├── SECURITY.md └── workflows │ ├── install.yml │ ├── update.yml │ ├── build.yml │ └── tests.yml ├── .markdownlint.json ├── .pre-commit-config.yaml ├── .cruft.json ├── CHANGELOG.md ├── mkdocs.yml ├── .gitignore ├── README.md └── Makefile /tests/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/e2e/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/unit/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /src/mkdocs_newsletter/py.typed: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/integration/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/unit/services/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /docs/examples/simple-example.py: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /docs/reference.md: -------------------------------------------------------------------------------- 1 | ::: mkdocs_newsletter 2 | -------------------------------------------------------------------------------- /src/mkdocs_newsletter/services/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/assets/test_data/requirements.txt: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /docs/img/screencast.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lyz-code/mkdocs-newsletter/HEAD/docs/img/screencast.gif -------------------------------------------------------------------------------- /src/mkdocs_newsletter/adapters/__init__.py: -------------------------------------------------------------------------------- 1 | """Module to store the functions shared by the different adapters.""" 2 | -------------------------------------------------------------------------------- /src/mkdocs_newsletter/entrypoints/__init__.py: -------------------------------------------------------------------------------- 1 | """Define the different ways to expose the program functionality.""" 2 | -------------------------------------------------------------------------------- /docs/theme/assets/images/bluebook.bmp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lyz-code/mkdocs-newsletter/HEAD/docs/theme/assets/images/bluebook.bmp -------------------------------------------------------------------------------- /tests/assets/test_data/docs/img/helm_logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lyz-code/mkdocs-newsletter/HEAD/tests/assets/test_data/docs/img/helm_logo.png -------------------------------------------------------------------------------- /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | 6 | 7 | ## Checklist 8 | 9 | * [ ] Add test cases to all the changes you introduce 10 | * [ ] Update the documentation for the changes 11 | -------------------------------------------------------------------------------- /docs/theme/assets/images/misc.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | -------------------------------------------------------------------------------- /docs/theme/assets/images/stackexchange.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | -------------------------------------------------------------------------------- /docs/theme/assets/images/dropbox.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | -------------------------------------------------------------------------------- /docs/theme/assets/images/pydo.svg: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /.markdownlint.json: -------------------------------------------------------------------------------- 1 | { 2 | "default": true, 3 | "MD003": { 4 | "style": "atx" 5 | }, 6 | "MD013": { 7 | "line_length": 180 8 | }, 9 | "MD004": { 10 | "style": "asterisk" 11 | }, 12 | "MD007": { 13 | "indent": 4 14 | }, 15 | "MD025": false, 16 | "MD030": false, 17 | "MD035": { 18 | "style": "---" 19 | }, 20 | "MD041": false, 21 | "MD046": false 22 | } 23 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/question.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Question 3 | about: Ask a question about how to use mkdocs-newsletter 4 | labels: question 5 | --- 6 | 7 | 14 | 15 | ## Question 16 | -------------------------------------------------------------------------------- /docs/theme/assets/images/patreon.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | -------------------------------------------------------------------------------- /tests/unit/test_version.py: -------------------------------------------------------------------------------- 1 | """Test the version message""" 2 | 3 | import platform 4 | import sys 5 | 6 | from mkdocs_newsletter.version import __version__, version_info 7 | 8 | 9 | def test_version() -> None: 10 | """ 11 | Given: Nothing 12 | When: version_info is called 13 | Then: the expected output is given 14 | """ 15 | result = version_info() 16 | 17 | assert sys.version.split(" ", maxsplit=1)[0] in result 18 | assert platform.platform() in result 19 | assert __version__ in result 20 | -------------------------------------------------------------------------------- /src/mkdocs_newsletter/__init__.py: -------------------------------------------------------------------------------- 1 | """Automatically create newsletters from the changes in a mkdocs repository.""" 2 | 3 | from typing import List 4 | 5 | from .entrypoints.mkdocs_plugin import Newsletter 6 | from .model import Change 7 | from .services.git import semantic_changes 8 | from .services.newsletter import digital_garden_changes, last_newsletter_changes 9 | 10 | __all__: List[str] = [ 11 | "Change", 12 | "semantic_changes", 13 | "digital_garden_changes", 14 | "last_newsletter_changes", 15 | "Newsletter", 16 | ] 17 | -------------------------------------------------------------------------------- /docs/theme/assets/images/mega.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | -------------------------------------------------------------------------------- /docs/theme/assets/images/wired.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/feature_request.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Feature Request 3 | about: Suggest a new feature or change to mkdocs-newsletter 4 | labels: feature request 5 | --- 6 | 7 | ## Description 8 | 9 | 10 | ## Possible Solution 11 | 12 | 13 | ## Additional context 14 | 15 | 16 | ## Related Issue 17 | 18 | -------------------------------------------------------------------------------- /docs/theme/assets/images/hn.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | -------------------------------------------------------------------------------- /docs/theme/assets/images/image.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | -------------------------------------------------------------------------------- /docs/theme/assets/images/txt.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | -------------------------------------------------------------------------------- /docs/theme/assets/images/video.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | -------------------------------------------------------------------------------- /docs/theme/assets/images/archive.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | -------------------------------------------------------------------------------- /docs/theme/assets/images/google-scholar.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | -------------------------------------------------------------------------------- /docs/theme/assets/images/chi-dna.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | -------------------------------------------------------------------------------- /docs/theme/assets/images/audio.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | -------------------------------------------------------------------------------- /docs/theme/assets/images/spreadsheet.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | -------------------------------------------------------------------------------- /src/mkdocs_newsletter/version.py: -------------------------------------------------------------------------------- 1 | """Utilities to retrieve the information of the program version.""" 2 | 3 | import platform 4 | import sys 5 | from textwrap import dedent 6 | 7 | # Do not edit this line manually, let `make bump` do it. 8 | __version__ = "1.1.0" 9 | 10 | 11 | def version_info() -> str: 12 | """Display the version of the program, python and the platform.""" 13 | return dedent( 14 | f"""\ 15 | ------------------------------------------------------------------ 16 | mkdocs_newsletter: {__version__} 17 | Python: {sys.version.split(" ", maxsplit=1)[0]} 18 | Platform: {platform.platform()} 19 | ------------------------------------------------------------------""" 20 | ) 21 | -------------------------------------------------------------------------------- /docs/theme/assets/images/amazon.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | -------------------------------------------------------------------------------- /docs/theme/assets/images/github.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | -------------------------------------------------------------------------------- /docs/theme/assets/images/worddoc.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | -------------------------------------------------------------------------------- /.github/SECURITY.md: -------------------------------------------------------------------------------- 1 | # Security Policy 2 | 3 | ## Supported Versions 4 | 5 | We will endeavour to support: 6 | 7 | * The most recent minor release with bug fixes. 8 | * The latest minor release from the last major version for 6 months after a new 9 | major version is released with critical bug fixes. 10 | * All versions if a security vulnerability is found provided: 11 | * Upgrading to a later version is non-trivial. 12 | * Sufficient people are using that version to make support worthwhile. 13 | 14 | ## Reporting a Vulnerability 15 | 16 | If you find what you think might be a security vulnerability with 17 | mkdocs-newsletter, please do not create an issue on github. Instead please 18 | email lyz-code-security-advisories@riseup.net I'll reply to your email promptly 19 | and try to get a patch out ASAP. 20 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/bug.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Bug 3 | about: Create a bug report to help us improve mkdocs-newsletter 4 | labels: bug 5 | --- 6 | 7 | ## Description 8 | 9 | 10 | ## Steps to reproduce 11 | 15 | 16 | ## Current behavior 17 | 18 | 19 | ## Desired behavior 20 | 26 | 27 | ## Environment 28 | 35 | -------------------------------------------------------------------------------- /.github/workflows/install.yml: -------------------------------------------------------------------------------- 1 | --- 2 | name: Install 3 | on: # yamllint disable-line rule:truthy 4 | schedule: 5 | - cron: 21 08 * * * 6 | workflow_dispatch: 7 | 8 | jobs: 9 | Install: 10 | runs-on: ubuntu-latest 11 | strategy: 12 | max-parallel: 4 13 | matrix: 14 | python-version: [3.8, 3.9, '3.10', '3.11'] 15 | steps: 16 | - uses: actions/checkout@v1 17 | - name: Set up Python ${{ matrix.python-version }} 18 | uses: actions/setup-python@v1 19 | with: 20 | python-version: ${{ matrix.python-version }} 21 | - name: Install the program 22 | run: pip install mkdocs-newsletter 23 | - name: Test the program works 24 | run: |- 25 | python -c "import mkdocs_newsletter.version; 26 | print(mkdocs_newsletter.version.version_info())" 27 | -------------------------------------------------------------------------------- /docs/theme/assets/images/twitter.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | -------------------------------------------------------------------------------- /src/mkdocs_newsletter/templates/newsletter_landing_page.j2: -------------------------------------------------------------------------------- 1 | If you want to follow the meaningful changes of this site, you can either: 2 | 3 | * [Subscribe to the RSS feed](#rss_feed) 4 | * [Browse the Newsletter section](#newsletter_section) 5 | 6 | # RSS feed 7 | 8 | You can choose how often you want to see the site updates: 9 | 10 | * [Daily]({{ site_url }}/daily.xml) 11 | * [Weekly]({{ site_url }}/weekly.xml) 12 | * [Monthly]({{ site_url }}/monthly.xml) 13 | * [Yearly]({{ site_url }}/yearly.xml) 14 | 15 | # Newsletter section 16 | 17 | We aggregate the changes by year, month, week and day. You can navigate this section to 18 | see the latest changes. 19 | 20 | # Credits 21 | 22 | The newsletters and the RSS feeds are automatically created from the message commits of 23 | the repository thanks to the 24 | [mkdocs-newsletter](https://lyz-code.github.io/mkdocs-newsletter/) plugin. 25 | -------------------------------------------------------------------------------- /docs/theme/assets/images/theguardian.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | -------------------------------------------------------------------------------- /docs/stylesheets/extra.css: -------------------------------------------------------------------------------- 1 | .md-content a:link { 2 | text-decoration:underline; 3 | } 4 | 5 | .md-typeset a:hover { 6 | color: #abb9c1; 7 | text-decoration:underline; 8 | } 9 | 10 | .md-typeset h1 { 11 | font-size: 32px; 12 | font-family: Content-font, Roboto, sans-serif; 13 | font-weight: 500; 14 | line-height: 1.5; 15 | color: #28292d; 16 | } 17 | 18 | .md-typeset h1::after { 19 | width:93%; 20 | height:2px; 21 | background: #283551; 22 | content:""; 23 | display: block; 24 | margin-top: 1px; 25 | opacity: 0.3; 26 | } 27 | 28 | .md-typeset h2 { 29 | font-size: 24px; 30 | font-family: Content-font, Roboto, sans-serif; 31 | font-weight: 700; 32 | line-height: 1.5; 33 | color: #28292d; 34 | } 35 | 36 | .md-typeset h2::after { 37 | width:100%; 38 | height:1px; 39 | background: #283551; 40 | content:""; 41 | display: block; 42 | margin-top: -5px; 43 | opacity: 0.2; 44 | } 45 | -------------------------------------------------------------------------------- /src/mkdocs_newsletter/templates/marcos.j2: -------------------------------------------------------------------------------- 1 | {% macro title(section, level) -%} 2 | {% set header = level * '#' %} 3 | {% if section.url is not none %} 4 | {{ header }} [{{ section.title }}]({{section.url}}) 5 | {% else %} 6 | {{ header }} {{ section.title }} 7 | {% endif %} 8 | {%- endmacro %} 9 | 10 | {% macro changes(changes) -%} 11 | {% for change in changes %} 12 | {% if change.file_section is not none %} 13 | * [{{ change.summary }}]({{ change.file_}}{{ change.file_section}}) 14 | {% else %} 15 | * {{ change.summary }} 16 | {% endif %} 17 | {% if change.message is not none %} 18 | {{ change.message }} 19 | {% endif %} 20 | {% endfor %} 21 | {%- endmacro %} 22 | 23 | {% macro print_section(section, level=1) -%} 24 | {{ title(section, level) }} 25 | {{ changes(section.changes) }} 26 | {% if section.subsections is not none %} 27 | {% set level = level + 1 %} 28 | {{ section(section.subsections, level) }} 29 | {% endif %} 30 | {%- endmacro %} 31 | -------------------------------------------------------------------------------- /.pre-commit-config.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | repos: 3 | - repo: https://github.com/pre-commit/pre-commit-hooks 4 | rev: v3.1.0 5 | hooks: 6 | - id: trailing-whitespace 7 | - id: check-added-large-files 8 | - id: check-docstring-first 9 | - id: check-merge-conflict 10 | - id: end-of-file-fixer 11 | - repo: https://github.com/ambv/black 12 | rev: 21.12b0 13 | hooks: 14 | - id: black 15 | language_version: python3.7 16 | - repo: https://github.com/pre-commit/mirrors-mypy 17 | rev: v0.910-1 18 | hooks: 19 | - name: Run mypy static analysis tool 20 | id: mypy 21 | - repo: https://github.com/Lucas-C/pre-commit-hooks-safety 22 | rev: v1.2.2 23 | hooks: 24 | - id: python-safety-dependencies-check 25 | - repo: https://github.com/flakehell/flakehell/ 26 | rev: v.0.9.0 27 | hooks: 28 | - name: Run flakehell static analysis tool 29 | id: flakehell 30 | -------------------------------------------------------------------------------- /docs/theme/assets/images/newyorktimes.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | -------------------------------------------------------------------------------- /docs/theme/assets/images/openai.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | -------------------------------------------------------------------------------- /docs/theme/assets/images/code.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | -------------------------------------------------------------------------------- /.github/workflows/update.yml: -------------------------------------------------------------------------------- 1 | --- 2 | name: Update 3 | on: # yamllint disable-line rule:truthy 4 | schedule: 5 | - cron: 11 08 * * * 6 | workflow_dispatch: 7 | 8 | jobs: 9 | Update: 10 | runs-on: ubuntu-latest 11 | steps: 12 | - uses: actions/checkout@master 13 | with: 14 | persist-credentials: false 15 | fetch-depth: 0 16 | - name: Set up Python 17 | uses: actions/setup-python@v1 18 | with: 19 | python-version: 3.9 20 | - name: Install program 21 | run: | 22 | # All these steps are required so that mypy behaves equally than the 23 | # local environment, once mypy supports __pypackages__ try to use the 24 | # github action 25 | pip install virtualenv pdm 26 | virtualenv .venv 27 | source .venv/bin/activate 28 | pdm config python.use_venv True 29 | - name: Update requirements 30 | run: make update-production 31 | - name: Run tests 32 | run: make all 33 | -------------------------------------------------------------------------------- /tests/assets/test_data/mkdocs.yml: -------------------------------------------------------------------------------- 1 | --- 2 | site_name: The Blue Book 3 | site_description: My second brain 4 | site_author: Lyz 5 | site_url: https://lyz-code.github.io/blue-book 6 | site_dir: site 7 | nav: 8 | - Introduction: index.md 9 | - DevOps: 10 | - devops/devops.md 11 | - Infrastructure as Code: 12 | - Helm: 13 | - devops/helm/helm.md 14 | - Coding: 15 | - TDD: coding/tdd.md 16 | - Python: 17 | - Libraries: 18 | - GitPython: coding/python/gitpython.md 19 | - Botany: 20 | - Trees: botany/trees.md 21 | - Emojis: emojis.md 22 | 23 | plugins: 24 | - git-revision-date-localized: 25 | type: timeago 26 | - autolinks 27 | - section-index 28 | - mkdocs-newsletter 29 | 30 | markdown_extensions: 31 | - toc: 32 | baselevel: 2 33 | theme: 34 | name: material 35 | logo: img/logo.bmp 36 | features: 37 | - navigation.footer 38 | - navigation.instant 39 | - navigation.top 40 | - content.code.annotate 41 | - search.suggest 42 | - search.highlight 43 | -------------------------------------------------------------------------------- /docs/theme/assets/images/erowid.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | -------------------------------------------------------------------------------- /.cruft.json: -------------------------------------------------------------------------------- 1 | { 2 | "template": "https://github.com/lyz-code/cookiecutter-python-project", 3 | "commit": "e2247dc1d557217a1c6f8b89f9355cb9be1c8238", 4 | "context": { 5 | "cookiecutter": { 6 | "project_name": "MkDocs Newsletter", 7 | "project_slug": "mkdocs-newsletter", 8 | "project_description": "Automatically create newsletters from the changes in a mkdocs repository", 9 | "requirements": "", 10 | "configure_command_line": true, 11 | "read_configuration_from_yaml": true, 12 | "github_user": "lyz-code", 13 | "github_token_pass_path": "internet/github.lyz-code.api_token", 14 | "pypi_token_pass_path": "internet/pypi.token", 15 | "test_pypi_token_pass_path": "internet/test.pypi.token", 16 | "author": "Lyz", 17 | "author_email": "lyz-code-security-advisories@riseup.net", 18 | "security_advisories_email": "lyz-code-security-advisories@riseup.net", 19 | "project_underscore_slug": "mkdocs_newsletter", 20 | "_template": "https://github.com/lyz-code/cookiecutter-python-project" 21 | } 22 | }, 23 | "directory": null, 24 | "checkout": null 25 | } 26 | -------------------------------------------------------------------------------- /docs/theme/assets/images/internetarchive.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | -------------------------------------------------------------------------------- /docs/theme/assets/images/csv.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | -------------------------------------------------------------------------------- /docs/theme/assets/images/gitea.svg: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /docs/adr/00X-changelog_support.md: -------------------------------------------------------------------------------- 1 | Date: 2021-02-10 2 | 3 | # Status 4 | 6 | Draft 7 | 8 | # Context 9 | 10 | Software documentation could benefit from the local newsletter, rss and email 11 | newsletter system. 12 | 13 | There are some changes that need to be done though. Instead of creating items 14 | per week, month or year, they should be created each time a release is made. 15 | Also, the template of the newsletter should be different. 16 | 17 | # Proposals 18 | 19 | * Change the template to ... 20 | * Select which changes are added to the changelog 21 | * How do we configure the plugin so it knows it's in changelog mode 22 | * The CI reacts to the tag event to rebuild the documentation and create the 23 | newsletter. 24 | 25 | # Decision 26 | 27 | 28 | # Consequences 29 | 30 | -------------------------------------------------------------------------------- /docs/theme/assets/images/uptontea.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | -------------------------------------------------------------------------------- /src/mkdocs_newsletter/templates/newsletter_article.j2: -------------------------------------------------------------------------------- 1 | {%- macro title(section, level) -%} 2 | {% set header = level * '#' %} 3 | {%- if section.url is not none -%} 4 | {{ header }} [{{ section.title }}]({{section.url}}) 5 | {%- else -%} 6 | {{ header }} {{ section.title }} 7 | {%- endif %} 8 | {%- endmacro %} 9 | 10 | {%- macro changes(changes) -%} 11 | {%- for change in changes %} 12 | {% if change.file_subsection is not none -%} 13 | * {{ change_type_text[change.type_] }}: [{{ change.summary }}]({{ change.file_}}{{ change.file_subsection}}) 14 | {%- else -%} 15 | * {{ change_type_text[change.type_] }}: {{ change.summary }} 16 | {%- endif %} 17 | {%- if change.message is not none %} 18 | 19 | {{ (' ' ~ change.message) | replace('\n', '\n ') }} 20 | {% endif -%} 21 | {% endfor -%} 22 | {% endmacro -%} 23 | 24 | {%- macro print_section(section, level=1) -%} 25 | {{ title(section, level) }} 26 | {{ changes(section.changes) }} 27 | 28 | {% set level = level + 1 -%} 29 | {%- for subsection in section.subsections -%} 30 | {{ print_section(subsection, level) }} 31 | {%- endfor -%} 32 | {%- endmacro -%} 33 | 34 | {%- for section in sections -%} 35 | {{ print_section(section) }} 36 | {%- endfor -%} 37 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | ## 1.1.0 (2023-11-23) 2 | 3 | ### Feat 4 | 5 | - support Python 3.11 6 | 7 | ### Fix 8 | 9 | - ignore feedparser deprecation warning 10 | - drop support for Python 3.7 11 | - switch from pdm-pep517 to pdm-backend 12 | - pin pydantic < 2.0.0 13 | - mkdocs incompatibilities with latest version 14 | - correct mkdocs signature changes 15 | - remove dependency with py 16 | 17 | ## 1.0.1 (2022-03-10) 18 | 19 | ### Fix 20 | 21 | - add the missing requirements 22 | - add the missing requirements 23 | 24 | ## 1.0.0 (2022-03-10) 25 | 26 | ### Feat 27 | 28 | - use pdm package manager 29 | - use pdm package manager 30 | 31 | ## 0.3.1 (2021-06-08) 32 | 33 | ### Fix 34 | 35 | - adapt to mkdocs 1.2 36 | 37 | ## 0.3.0 (2021-04-23) 38 | 39 | ## 0.2.1 (2021-03-29) 40 | 41 | ### Fix 42 | 43 | - don't wrap commit messages when building articles 44 | 45 | ## 0.2.0 (2021-03-28) 46 | 47 | ### Feat 48 | 49 | - add rss support 50 | 51 | ### feat 52 | 53 | - create newsletter landing page 54 | 55 | ## 0.1.0 (2021-03-19) 56 | 57 | ### Fix 58 | 59 | - add dateutil to required dependencies 60 | - Remove links from the Other section 61 | 62 | ### Feat 63 | 64 | - create newsletters from changes 65 | - create initial project structure 66 | -------------------------------------------------------------------------------- /tests/assets/test_data/docs/coding/python/gitpython.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: GitPython 3 | date: 20210210 4 | author: Lyz 5 | --- 6 | 7 | [GitPython](https://gitpython.readthedocs.io) is a python library used to 8 | interact with git repositories, high-level like git-porcelain, or low-level like 9 | git-plumbing. 10 | 11 | It provides abstractions of git objects for easy access of repository data, and 12 | additionally allows you to access the git repository more directly using either 13 | a pure python implementation, or the faster, but more resource intensive git 14 | command implementation. 15 | 16 | The object database implementation is optimized for handling large quantities of 17 | objects and large datasets, which is achieved by using low-level structures and 18 | data streaming. 19 | 20 | # [Installation](https://gitpython.readthedocs.io/en/stable/intro.html#installing-gitpython) 21 | 22 | ```bash 23 | pip install GitPython 24 | ``` 25 | 26 | # Usage 27 | 28 | # Testing 29 | 30 | There is no testing functionality, you need to either Mock or build fake 31 | adapters. 32 | 33 | # References 34 | 35 | * [Docs](https://gitpython.readthedocs.io) 36 | * [Git](https://github.com/gitpython-developers/GitPython) 37 | * [Tutorial](https://gitpython.readthedocs.io/en/stable/tutorial.html#tutorial-label) 38 | -------------------------------------------------------------------------------- /docs/theme/assets/images/reddit.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | -------------------------------------------------------------------------------- /tests/assets/test_data/docs/devops/helm/helm.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Helm 3 | date: 20200210 4 | author: Lyz 5 | --- 6 | 7 | [Helm](https://helm.sh/) is the package manager for Kubernetes. Through charts 8 | it helps you define, install and upgrade even the most complex Kubernetes 9 | applications. 10 | 11 |

12 | 13 |

14 | 15 | The advantages of using helm over `kubectl apply` are the easiness of: 16 | 17 | * Repeatable application installation. 18 | * CI integration. 19 | * Versioning and sharing. 20 | 21 | Charts are a group of Go templates of kubernetes yaml resource manifests, they 22 | are easy to create, version, share, and publish. 23 | 24 | Helm alone lacks some features, that are satisfied through some external 25 | programs: 26 | 27 | * Helmfile is used to declaratively configure your charts, so 28 | they can be versioned through git. 29 | * Helm-secrets is used to remove hardcoded credentials from `values.yaml` 30 | files. Helm has an [open issue](https://github.com/helm/helm/issues/2196) to 31 | integrate it into it's codebase. 32 | * Helm-git is used to install helm charts directly from Git 33 | repositories. 34 | 35 | # Links 36 | 37 | * [Homepage](http://www.helm.sh/) 38 | * [Docs](https://docs.helm.sh) 39 | * [Git](https://github.com/kubernetes/helm) 40 | * [Chart hub](https://hub.helm.sh) 41 | * [Git charts repositories](https://github.com/kubernetes/charts) 42 | -------------------------------------------------------------------------------- /docs/adr/002-initial_plugin_design.md: -------------------------------------------------------------------------------- 1 | Date: 2021-02-09 2 | 3 | # Status 4 | 6 | Accepted 7 | 8 | Based on: [001](001-initial_approach.md) 9 | 10 | Extended by: [003](003-select_the_changes_to_record.md), 11 | [004](004-article_newsletter_structure.md), and 12 | [005](005-create_the_newsletter_articles.md). 13 | 14 | # Context 15 | 16 | Taking [001](001-initial_approach.md) as a starting point, we want to define the 17 | processes that the mkdocs plugin need to have to fulfill the desired requirements. 18 | 19 | # Proposals 20 | 21 | The plugin will consist on the following phases: 22 | 23 | * [Select the changes that need to be recorded](003-select_the_changes_to_record.md). 24 | * [Create the newsletter articles from those changes](005-create_the_newsletter_articles.md). 25 | * [Build the MkDocs nav with the newsletter articles](004-article_newsletter_structure.md) 26 | * Decide which changes need to be notified 27 | * Send the notifications: 28 | * [Update the RSS](006-rss_feed_creation.md) 29 | * Send the email. 30 | 31 | # Decision 32 | 33 | Implement the only proposal. 34 | 35 | # Consequences 36 | 37 | -------------------------------------------------------------------------------- /.github/workflows/build.yml: -------------------------------------------------------------------------------- 1 | --- 2 | name: Build 3 | 4 | on: # yamllint disable-line rule:truthy 5 | push: 6 | branches: 7 | - main 8 | workflow_dispatch: 9 | 10 | jobs: 11 | PyPI: 12 | name: Build and publish Python distributions to TestPyPI 13 | runs-on: ubuntu-latest 14 | steps: 15 | - uses: actions/checkout@master 16 | - uses: pdm-project/setup-pdm@main 17 | name: Setup Python and PDM 18 | with: 19 | python-version: 3.9 20 | - name: Build package 21 | run: make build-package 22 | - name: Publish distribution to Test PyPI 23 | uses: pypa/gh-action-pypi-publish@master 24 | with: 25 | password: ${{ secrets.test_pypi_password }} 26 | repository_url: https://test.pypi.org/legacy/ 27 | skip_existing: true 28 | Documentation: 29 | runs-on: ubuntu-latest 30 | steps: 31 | - name: Checkout 32 | uses: actions/checkout@v2 33 | with: 34 | # Number of commits to fetch. 0 indicates all history. 35 | # Default: 1 36 | fetch-depth: 0 37 | - uses: pdm-project/setup-pdm@main 38 | name: Setup Python and PDM 39 | with: 40 | python-version: 3.9 41 | - name: Install dependencies 42 | run: make install 43 | - name: Build the Documentation 44 | run: make build-docs 45 | - name: Deploy 46 | uses: peaceiris/actions-gh-pages@v3 47 | with: 48 | deploy_key: ${{ secrets.ACTIONS_DEPLOY_KEY }} 49 | publish_dir: ./site 50 | -------------------------------------------------------------------------------- /docs/adr/adr.md: -------------------------------------------------------------------------------- 1 | [ADR](https://lyz-code.github.io/blue-book/adr/) are short text documents that 2 | captures an important architectural decision made along with its context and 3 | consequences. 4 | 5 | ```mermaid 6 | graph TD 7 | 001[001: High level analysis] 8 | 002[002: Initial MkDocs plugin design] 9 | 003[003: Selected changes to record] 10 | 004[004: Article newsletter structure] 11 | 005[005: Article newsletter creation] 12 | 006[006: RSS feed creation] 13 | 14 | 001 -- Extended --> 002 15 | 002 -- Extended --> 003 16 | 002 -- Extended --> 004 17 | 002 -- Extended --> 005 18 | 002 -- Extended --> 006 19 | 003 -- Extended --> 004 20 | 004 -- Extended --> 005 21 | 22 | click 001 "https://lyz-code.github.io/mkdocs-newsletter/adr/001-initial_approach" _blank 23 | click 002 "https://lyz-code.github.io/mkdocs-newsletter/adr/002-initial_plugin_design" _blank 24 | click 003 "https://lyz-code.github.io/mkdocs-newsletter/adr/003-select_the_changes_to_record" _blank 25 | click 004 "https://lyz-code.github.io/mkdocs-newsletter/adr/004-article_newsletter_structure" _blank 26 | click 005 "https://lyz-code.github.io/mkdocs-newsletter/adr/005-create_the_newsletter_articles" _blank 27 | click 006 "https://lyz-code.github.io/mkdocs-newsletter/adr/006-rss_feed_creation" _blank 28 | 29 | 001:::accepted 30 | 002:::accepted 31 | 003:::accepted 32 | 004:::accepted 33 | 005:::accepted 34 | 006:::accepted 35 | 36 | classDef draft fill:#CDBFEA; 37 | classDef proposed fill:#B1CCE8; 38 | classDef accepted fill:#B1E8BA; 39 | classDef rejected fill:#E8B1B1; 40 | classDef deprecated fill:#E8B1B1; 41 | classDef superseeded fill:#E8E5B1; 42 | ``` 43 | -------------------------------------------------------------------------------- /docs/theme/assets/images/washingtonpost.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | -------------------------------------------------------------------------------- /docs/theme/assets/images/miri.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | -------------------------------------------------------------------------------- /tests/assets/test_data/docs/index.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Introduction 3 | author: Lyz 4 | --- 5 | 6 | This is my personal digital garden where I share everything 7 | I know about this world in form of an online [MkDocs](https://www.mkdocs.org/) 8 | book hosted on [GitHub](https://github.com/lyz-code/blue-book). 9 | 10 | If this is your first visit, welcome!, you may be overwhelmed by the amount of 11 | content and don't know where to start reading. A good entrypoint is meta, 12 | where I describe this wiki, its structure and goals in more detail. 13 | 14 | # Visiting the garden 15 | 16 | You can search the contents of this wiki with the top search field or you can 17 | explore the tree view to the left. 18 | 19 | Start with the first article that grabs your attention and be ready to 20 | incrementally read the rest. 21 | 22 | Or you can use it as a reference, cloning the [git 23 | repository](https://github.com/lyz-code/blue-book) and using grep like tools. 24 | 25 | # Make your own digital garden 26 | 27 | Don't be afraid to create one of your own and share what you know with the 28 | world. If you don't want to build your own, I invite you to use a fork of 29 | mine and make [contributions](#contributing). I would love to see the blue-book 30 | maintained by multiple people. 31 | 32 | You can view other similar gardens to get inspiration. 33 | 34 | # Contributing 35 | 36 | If you find a mistake or want to add new content, please make the changes. You 37 | can use the edit button on the top right of any article to add them in a pull 38 | request, if you don't know what that means, you cal always open an 39 | [issue](https://github.com/lyz-code/blue-book/issues/new) or send me an 40 | email.md). 41 | 42 | # Thank you 43 | 44 | If you liked my book and want to show your support, please see if you know how 45 | can I fulfill any item of my wish list or contribute to my other projects. 46 | 47 | [![CC0](https://img.shields.io/badge/license-CC0-0a0a0a.svg?style=flat&colorA=0a0a0a)](https://creativecommons.org/publicdomain/zero/1.0/) 48 | -------------------------------------------------------------------------------- /docs/theme/assets/images/plos.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | -------------------------------------------------------------------------------- /src/mkdocs_newsletter/templates/rss.xml.j2: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | {# Mandatory elements #} 5 | {% if feed.title is not none %}{{ feed.title|e }}{% endif %} 6 | {% if feed.description is not none %}{{ feed.description|e }}{% endif %} 7 | {% if feed.link is not none %}{{ feed.link }}{% endif %} 8 | {% if feed.rss_link is not none %}{% endif %} 9 | 10 | {# Optional elements #} 11 | {% if feed.author is not none %}{{ feed.author }}{% endif %} 12 | 13 | {# Timestamps and frequency #} 14 | {{ feed.published }} 15 | {{ feed.ttl }} 16 | 17 | {# Credits #} 18 | {{ feed.generator }} 19 | 20 | {# Feed illustration #} 21 | {% if feed.logo is defined %} 22 | {{ feed.logo }}{{ feed.title }} 23 | {% if feed.link is not none %}{{ feed.link }}{% endif %} 24 | 25 | {% endif %} 26 | 27 | {# Entries #} 28 | {% for item in feed.entries %} 29 | {{ item.title|e }} 30 | {% if item.author is not none %}{{ item.author }}{% endif %} 31 | {{ item.description|e }} 32 | {% if item.link is not none %}{{ item.link }}{% endif %} 33 | {{ item.published }} 34 | {% if item.link is not none %}{{ feed.title }}{% endif %} 35 | {% if item.link is not none %}{{ item.link }}{% endif %} 36 | {% if item.image is not none %} 37 | 38 | {% endif %} 39 | 40 | {% endfor %} 41 | 42 | 43 | -------------------------------------------------------------------------------- /.github/workflows/tests.yml: -------------------------------------------------------------------------------- 1 | --- 2 | name: Tests 3 | 4 | on: # yamllint disable-line rule:truthy 5 | push: 6 | branches: 7 | - main 8 | pull_request: 9 | types: [opened, synchronize] 10 | workflow_dispatch: 11 | 12 | jobs: 13 | Tests: 14 | runs-on: ubuntu-latest 15 | strategy: 16 | max-parallel: 4 17 | matrix: 18 | python-version: [3.8, 3.9, '3.10', '3.11'] 19 | steps: 20 | - uses: actions/checkout@v1 21 | - name: Set up Python ${{ matrix.python-version }} 22 | uses: actions/setup-python@v1 23 | with: 24 | python-version: ${{ matrix.python-version }} 25 | - name: Install program 26 | run: | 27 | # All these steps are required so that mypy behaves equally than the 28 | # local environment, once mypy supports __pypackages__ try to use the 29 | # github action 30 | pip install virtualenv pdm 31 | virtualenv .venv 32 | source .venv/bin/activate 33 | pdm config python.use_venv True 34 | make install 35 | - name: Test linters 36 | run: make lint 37 | - name: Test type checkers 38 | run: make mypy 39 | - name: Test security 40 | run: make security 41 | - name: Test with pytest 42 | run: make test 43 | # - name: Upload Coverage 44 | # run: | 45 | # pip3 install 'coveralls[toml]' 46 | # coveralls --service=github 47 | # env: 48 | # GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} 49 | # COVERALLS_FLAG_NAME: ${{ matrix.test-name }} 50 | # COVERALLS_PARALLEL: true 51 | - name: Test documentation 52 | run: make build-docs 53 | - name: Build the package 54 | run: make build-package 55 | # Coveralls: 56 | # name: Finish Coveralls 57 | # needs: Tests 58 | # runs-on: ubuntu-latest 59 | # container: python:3-slim 60 | # steps: 61 | # - name: Finished 62 | # run: | 63 | # pip3 install 'coveralls[toml]' 64 | # coveralls --service=github --finish 65 | # env: 66 | # GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} 67 | -------------------------------------------------------------------------------- /docs/theme/assets/images/bluebook.svg: -------------------------------------------------------------------------------- 1 | 2 | 4 | 7 | 8 | 10 | 32 | 33 | 34 | -------------------------------------------------------------------------------- /tests/assets/test_data/docs/emojis.md: -------------------------------------------------------------------------------- 1 | --- 2 | Title: Emojis 3 | Author: Lyz 4 | Date: 20170302 5 | --- 6 | 7 | Curated list of emojis to copy paste. 8 | 9 | # Angry 10 | 11 | ``` 12 | (҂⌣̀_⌣́) 13 | 14 | ( >д<) 15 | 16 | ʕ•̀o•́ʔ 17 | 18 | ヽ(≧Д≦)ノ 19 | 20 | ᕙ(⇀‸↼‶)ᕗ 21 | 22 | ٩(╬ʘ益ʘ╬)۶ 23 | ``` 24 | 25 | # Annoyed 26 | 27 | ``` 28 | (>_<) 29 | ``` 30 | 31 | # Awesome 32 | 33 | ``` 34 | ( ·_·) 35 | ( ·_·) --■-■ 36 | ( ·_·)--■-■ 37 | (-■_■) YEAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHH 38 | ``` 39 | 40 | # Conforting 41 | 42 | ``` 43 | (ｏ・_・)ノ”(ᴗ_ ᴗ。) 44 | ``` 45 | 46 | # Congratulations 47 | 48 | ``` 49 | ( ᐛ )و 50 | 51 | ＼\ ٩( ᐛ )و /／ 52 | ``` 53 | 54 | # Crying 55 | 56 | ``` 57 | (╥﹏╥) 58 | ``` 59 | 60 | # Excited 61 | 62 | ``` 63 | (((o(*ﾟ▽ﾟ*)o))) 64 | 65 | o(≧∇≦o) 66 | ``` 67 | 68 | # Dance 69 | 70 | ``` 71 | (~‾▿‾)~ ~(‾▿‾)~ ~(‾▿‾~) 72 | 73 | ┌(・。・)┘ ♪ └(・。・)┐ ♪ ┌(・。・)┘ 74 | 75 | ƪ(˘⌣˘)┐ ƪ(˘⌣˘)ʃ ┌(˘⌣˘)ʃ 76 | 77 | (>'-')> 78 | <('-'<) 79 | ^('-')^ 80 | v('-')v 81 | (>'-')> 82 | (^-^) 83 | ``` 84 | 85 | # Happy 86 | 87 | ``` 88 | ᕕ( ᐛ )ᕗ 89 | 90 | ʕ•ᴥ•ʔ 91 | 92 | (•‿•) 93 | 94 | (◡‿◡✿) 95 | 96 | (✿◠‿◠) 97 | 98 | ♪(๑ᴖ◡ᴖ๑)♪ 99 | ``` 100 | 101 | # Kisses 102 | 103 | ``` 104 | (づ￣ ³￣)づ 105 | 106 | ( ˘ ³˘)♥ 107 | ``` 108 | 109 | # Love 110 | 111 | ``` 112 | ❤ 113 | ``` 114 | 115 | # Pride 116 | 117 | ``` 118 | <(￣＾￣)> 119 | ``` 120 | 121 | # Relax 122 | 123 | ``` 124 | _へ__(‾◡◝ )> 125 | ``` 126 | 127 | # Sad 128 | 129 | ``` 130 | ｡ﾟ(*´□`)ﾟ｡ 131 | 132 | (◞‸◟；) 133 | ``` 134 | 135 | # Scared 136 | 137 | ``` 138 | ヽ(ﾟДﾟ)ﾉ 139 | 140 | ヽ〔ﾟДﾟ〕丿 141 | ``` 142 | 143 | # Sleepy 144 | 145 | ``` 146 | (ᴗ˳ᴗ) 147 | ``` 148 | 149 | # Smug 150 | 151 | ``` 152 | （￣～￣） 153 | ``` 154 | 155 | # Whyyyy? 156 | 157 | ``` 158 | (/ﾟДﾟ)/ 159 | ``` 160 | 161 | # Surprised 162 | 163 | ``` 164 | (\_/) 165 | (O.o) 166 | (> <) 167 | 168 | (⊙_☉) 169 | 170 | (¬º-°)¬ 171 | 172 | (☉_☉) 173 | 174 | (• ̥̆ •) 175 | 176 | ¯\(°_o)/¯ 177 | 178 | (・0・。(・-・。(・0・。(・-・。) 179 | 180 | (*ﾟ◯ﾟ*) 181 | ``` 182 | 183 | # Who cares 184 | 185 | ``` 186 | ¯\_(ツ)_/¯ 187 | ``` 188 | 189 | # WTF 190 | 191 | ``` 192 | (╯°□°)╯ ┻━┻ 193 | 194 | ヘ（。□°）ヘ 195 | ``` 196 | 197 | # Links 198 | 199 | * [Japanese Emoticons](http://japaneseemoticons.me) 200 | -------------------------------------------------------------------------------- /docs/adr/006-rss_feed_creation.md: -------------------------------------------------------------------------------- 1 | Date: 2021-03-26 2 | 3 | # Status 4 | 6 | Accepted 7 | 8 | # Context 9 | 10 | We want the readers to be notified by RSS of the changes of the documentation 11 | site. They should be able to choose the frequency of the updates. 12 | 13 | # Proposals 14 | 15 | 16 | Once the newsletter mkdocs articles are created, we need to expose them through 17 | RSS feeds for each of the periodicities (daily, weekly, monthly and yearly). 18 | 19 | We have the newsletter articles both in markdown and in html if we use the 20 | [`on_post_build`](https://www.mkdocs.org/user-guide/plugins/#on_post_build) 21 | event. 22 | 23 | We need to create both the RSS site documentation and the entries content. The 24 | first one can be created with the contents of the mkdocs `config` object, the 25 | second ones using the html might be more interesting as it will have already the 26 | internal links resolved to working urls. 27 | 28 | We'll refactor the code from 29 | [mkdocs-rss-plugin](https://github.com/Guts/mkdocs-rss-plugin), as they've 30 | already solved the problem of creating an RSS for a MkDocs site. 31 | 32 | We need to decide: 33 | 34 | * [How to select the newsletters to be published in each 35 | feed.](#newsletter-selection) 36 | * [How to create each feed.](#how-to-create-each-feed) 37 | 38 | ## Newsletter selection 39 | 40 | We're going to expose 15 entries in each feed. 41 | 42 | To get the 15 last newsletters for each feed we need to analyze the existent 43 | elements in the newsletter directory and then see the latest modification date 44 | of each of them. 45 | 46 | The `published_date` of the channel must be the published date of the last entry 47 | of the feed. 48 | 49 | ## How to create each feed 50 | 51 | We'll use the mkdocs-rss-plugin jinja2 template. I tried to use feedparser, but 52 | it's only for parsing and not for building RSSs. 53 | 54 | # Decision 55 | 56 | Implement the only proposal. 57 | 58 | # Consequences 59 | 60 | -------------------------------------------------------------------------------- /docs/theme/assets/images/thenewyorker.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | -------------------------------------------------------------------------------- /docs/adr/005-create_the_newsletter_articles.md: -------------------------------------------------------------------------------- 1 | Date: 2021-02-18 2 | 3 | # Status 4 | 6 | Accepted 7 | 8 | Based on: [004](004-article_newsletter_structure.md) 9 | 10 | # Context 11 | 12 | We need to present the user the changes in the cleanest way: 13 | 14 | * Have the minimum number of title levels, maybe a maximum of 3 levels. 15 | * Group the related changes. 16 | 17 | # Proposals 18 | 19 | 20 | To present the changes we want to: 21 | 22 | * Group the articles by category and subcategory following the nav order. 23 | * Group the changes by article ordered chronologically. 24 | 25 | A TOC doesn't make sense for the article stored in the mkdocs repository as it 26 | is already created by MkDocs. 27 | 28 | The skeleton of each article will be: 29 | 30 | * Categories as title one, respecting the nav order, with a link to the category 31 | file if it exists. 32 | * Subcategories as title two, respecting the nav order, with a link to the 33 | subcategory file if it exists. 34 | * File title as title three, respecting the nav order, with a link to the 35 | subcategory file if it exists. 36 | * Each change of the file will be added as bullet points ordered by date. 37 | Where the first line will be the first line of the commit and the body will 38 | be added below. 39 | 40 | If the scope of the change contains the information of the section of the 41 | file that it's changing, the link should point to that section instead. 42 | 43 | We need to differentiate the different types of changes: 44 | 45 | * feat: New content additions 46 | * perf: Improvements on existent articles 47 | * fix: Corrections on existent articles 48 | * refactor: Reorganization of articles 49 | 50 | We'll then create articles for each feed following the structure defined in 51 | [004](004-article_newsletter_structure.md). 52 | 53 | Inspiration: 54 | 55 | * [mkdocs-tags](https://github.com/jldiaz/mkdocs-plugin-tags) 56 | * [mkdocs_blog](https://github.com/andyoakley/mkdocs-blog) 57 | * [mkdocs-rss-plugin](https://github.com/Guts/mkdocs-rss-plugin) 58 | 59 | # Decision 60 | 61 | Implement the only proposal. 62 | 63 | # Consequences 64 | 65 | -------------------------------------------------------------------------------- /mkdocs.yml: -------------------------------------------------------------------------------- 1 | --- 2 | site_name: MkDocs Newsletter 3 | site_author: Lyz 4 | site_url: https://lyz-code.github.io/mkdocs-newsletter 5 | nav: 6 | - MkDocs Newsletter: index.md 7 | - Installation: install.md 8 | - Usage: usage.md 9 | - Reference: reference.md 10 | - Contributing: 11 | - contributing.md 12 | - Architecture Decision Records: 13 | - adr/adr.md 14 | - '001: High level problem analysis': adr/001-initial_approach.md 15 | - '002: Initial mkdocs plugin design': adr/002-initial_plugin_design.md 16 | - '003: Select the changes to record': adr/003-select_the_changes_to_record.md 17 | - '004: Article newsletter structure': adr/004-article_newsletter_structure.md 18 | - '005: Article newsletter creation': adr/005-create_the_newsletter_articles.md 19 | - '006: RSS feed creation': adr/006-rss_feed_creation.md 20 | - '00X: Changelog support': adr/00X-changelog_support.md 21 | 22 | plugins: 23 | - search 24 | - mkdocstrings: 25 | handlers: 26 | python: 27 | options: 28 | show_root_heading: true 29 | heading_level: 1 30 | - autolinks 31 | - git-revision-date-localized: 32 | type: timeago 33 | fallback_to_build_date: true 34 | # Not compatible with mermaid2 35 | # - minify: 36 | # minify_html: true 37 | - section-index 38 | 39 | markdown_extensions: 40 | - abbr 41 | - def_list 42 | - admonition 43 | # We need the markdown-include to inject files into other files 44 | - markdown_include.include: 45 | base_path: docs 46 | - meta 47 | - toc: 48 | permalink: true 49 | baselevel: 2 50 | - pymdownx.arithmatex 51 | - pymdownx.betterem: 52 | smart_enable: all 53 | - pymdownx.caret 54 | - pymdownx.critic 55 | - pymdownx.details 56 | - pymdownx.emoji: 57 | emoji_generator: '!!python/name:pymdownx.emoji.to_svg' 58 | - pymdownx.inlinehilite 59 | - pymdownx.magiclink 60 | - pymdownx.mark 61 | - pymdownx.smartsymbols 62 | - pymdownx.superfences: 63 | custom_fences: 64 | - name: mermaid 65 | class: mermaid 66 | format: !!python/name:pymdownx.superfences.fence_code_format 67 | - pymdownx.tabbed: 68 | alternate_style: true 69 | - pymdownx.tasklist: 70 | custom_checkbox: true 71 | - pymdownx.tilde 72 | 73 | theme: 74 | name: material 75 | custom_dir: docs/theme 76 | # logo: 'images/logo.bmp' 77 | features: 78 | - navigation.instant 79 | - navigation.footer 80 | - navigation.top 81 | - content.code.annotate 82 | 83 | palette: 84 | primary: blue grey 85 | accent: light blue 86 | 87 | extra_css: 88 | - stylesheets/extra.css 89 | - stylesheets/links.css 90 | 91 | repo_name: lyz-code/mkdocs-newsletter 92 | repo_url: https://github.com/lyz-code/mkdocs-newsletter 93 | watch: 94 | - src 95 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .pdm.toml 2 | 3 | # Created by https://www.toptal.com/developers/gitignore/api/python 4 | # Edit at https://www.toptal.com/developers/gitignore?templates=python 5 | 6 | ### Python ### 7 | # Byte-compiled / optimized / DLL files 8 | __pycache__/ 9 | *.py[cod] 10 | *$py.class 11 | 12 | # C extensions 13 | *.so 14 | 15 | # Distribution / packaging 16 | .Python 17 | build/ 18 | develop-eggs/ 19 | dist/ 20 | downloads/ 21 | eggs/ 22 | .eggs/ 23 | lib/ 24 | lib64/ 25 | parts/ 26 | sdist/ 27 | var/ 28 | wheels/ 29 | pip-wheel-metadata/ 30 | share/python-wheels/ 31 | *.egg-info/ 32 | .installed.cfg 33 | *.egg 34 | MANIFEST 35 | 36 | # PyInstaller 37 | # Usually these files are written by a python script from a template 38 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 39 | *.manifest 40 | *.spec 41 | 42 | # Installer logs 43 | pip-log.txt 44 | pip-delete-this-directory.txt 45 | 46 | # Unit test / coverage reports 47 | htmlcov/ 48 | .tox/ 49 | .nox/ 50 | .coverage 51 | .coverage.* 52 | .cache 53 | nosetests.xml 54 | coverage.xml 55 | *.cover 56 | *.py,cover 57 | .hypothesis/ 58 | .pytest_cache/ 59 | pytestdebug.log 60 | 61 | # Translations 62 | *.mo 63 | *.pot 64 | 65 | # Django stuff: 66 | *.log 67 | local_settings.py 68 | db.sqlite3 69 | db.sqlite3-journal 70 | 71 | # Flask stuff: 72 | instance/ 73 | .webassets-cache 74 | 75 | # Scrapy stuff: 76 | .scrapy 77 | 78 | # Sphinx documentation 79 | docs/_build/ 80 | doc/_build/ 81 | 82 | # PyBuilder 83 | target/ 84 | 85 | # Jupyter Notebook 86 | .ipynb_checkpoints 87 | 88 | # IPython 89 | profile_default/ 90 | ipython_config.py 91 | 92 | # pyenv 93 | .python-version 94 | 95 | # pipenv 96 | # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. 97 | # However, in case of collaboration, if having platform-specific dependencies or dependencies 98 | # having no cross-platform support, pipenv may install dependencies that don't work, or not 99 | # install all needed dependencies. 100 | #Pipfile.lock 101 | 102 | # PEP 582; used by e.g. github.com/David-OConnor/pyflow 103 | __pypackages__/ 104 | 105 | # Celery stuff 106 | celerybeat-schedule 107 | celerybeat.pid 108 | 109 | # SageMath parsed files 110 | *.sage.py 111 | 112 | # Environments 113 | .env 114 | .venv 115 | env/ 116 | venv/ 117 | ENV/ 118 | env.bak/ 119 | venv.bak/ 120 | 121 | # Spyder project settings 122 | .spyderproject 123 | .spyproject 124 | 125 | # Rope project settings 126 | .ropeproject 127 | 128 | # mkdocs documentation 129 | /site 130 | 131 | # mypy 132 | .mypy_cache/ 133 | .dmypy.json 134 | dmypy.json 135 | 136 | # Pyre type checker 137 | .pyre/ 138 | 139 | # pytype static type analyzer 140 | .pytype/ 141 | 142 | # End of https://www.toptal.com/developers/gitignore/api/python 143 | .flakeheaven_cache 144 | -------------------------------------------------------------------------------- /src/mkdocs_newsletter/entrypoints/mkdocs_plugin.py: -------------------------------------------------------------------------------- 1 | """Define the mkdocs plugin.""" 2 | 3 | import os 4 | from typing import Optional 5 | 6 | from git import Repo 7 | from mkdocs.config.defaults import MkDocsConfig 8 | from mkdocs.plugins import BasePlugin 9 | 10 | from ..services.git import semantic_changes 11 | from ..services.nav import build_nav 12 | from ..services.newsletter import ( 13 | add_change_categories, 14 | create_newsletter_landing_page, 15 | create_newsletters, 16 | digital_garden_changes, 17 | last_newsletter_changes, 18 | ) 19 | from ..services.rss import create_rss 20 | 21 | 22 | # Class cannot subclass 'BasePlugin' (has type 'Any'). It's how the docs say you need 23 | # to subclass it. 24 | class Newsletter(BasePlugin): # type: ignore 25 | """Define the MkDocs plugin to create newsletters.""" 26 | 27 | def __init__(self) -> None: 28 | """Initialize the basic attributes. 29 | 30 | Attributes: 31 | repo: Git repository to analyze. 32 | """ 33 | self.working_dir = os.getenv("NEWSLETTER_WORKING_DIR", default=os.getcwd()) 34 | self.repo = Repo(self.working_dir) 35 | 36 | def on_config(self, config: Optional[MkDocsConfig]) -> MkDocsConfig: 37 | """Create the new newsletters and load them in the navigation. 38 | 39 | Through the following steps: 40 | 41 | * Detect which were the last changes for each of the feeds. 42 | * Parse the changes from the git history that were done before the last 43 | changes. 44 | * Create the newsletter articles. 45 | * Update the navigation. 46 | 47 | Args: 48 | config: MkDocs global configuration object. 49 | 50 | Returns: 51 | config: MkDocs config object with the new newsletters in the Newsletter 52 | section. 53 | """ 54 | if config is None: 55 | config = MkDocsConfig() 56 | newsletter_dir = f"{self.working_dir}/docs/newsletter" 57 | if not os.path.exists(newsletter_dir): 58 | os.makedirs(newsletter_dir) 59 | last_published_changes = last_newsletter_changes(newsletter_dir) 60 | changes_to_publish = add_change_categories( 61 | semantic_changes(self.repo, last_published_changes.min()), config 62 | ) 63 | changes_per_feed = digital_garden_changes( 64 | changes_to_publish, 65 | last_published_changes, 66 | ) 67 | 68 | create_newsletters(changes_per_feed, self.repo) 69 | create_newsletter_landing_page(config, self.repo) 70 | 71 | config = build_nav(config, newsletter_dir) 72 | 73 | return config 74 | 75 | # The * in the signature is to mimic the parent class signature 76 | def on_post_build(self, *, config: MkDocsConfig) -> None: 77 | """Create the RSS feeds.""" 78 | create_rss(config, self.working_dir) 79 | -------------------------------------------------------------------------------- /docs/theme/assets/images/deepmind.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | image/svg+xml 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | -------------------------------------------------------------------------------- /tests/assets/test_data/docs/botany/trees.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Trees 3 | date: 20200910 4 | author: Lyz 5 | --- 6 | 7 | # [Beech](https://en.wikipedia.org/wiki/Beech) 8 | 9 | Beech (Fagus) is a genus of deciduous trees in the family Fagaceae, native to 10 | temperate Europe, Asia, and North America. The better known Fagus subgenus 11 | beeches are high-branching with tall, stout trunks and smooth silver-grey bark. 12 | 13 | Beeches are monoecious, bearing both male and female flowers on the same plant. 14 | The small flowers are unisexual, the female flowers borne in pairs, the male 15 | flowers wind-pollinating catkins. They are produced in spring shortly after the 16 | new leaves appear. The fruit of the beech tree, known as beechnuts or mast, is 17 | found in small burrs that drop from the tree in autumn. They are small, roughly 18 | triangular and edible, with a bitter, astringent, or in some cases, mild and 19 | nut-like taste. They have a high enough fat content that they can be pressed for 20 | edible oil. 21 | 22 | The leaves of beech trees are entire or sparsely toothed, from 5–15 cm (2–6 in) 23 | long and 4–10 cm (2–4 in) broad. Beeches are monoecious, bearing both male and 24 | female flowers on the same plant. The small flowers are unisexual, the female 25 | flowers borne in pairs, the male flowers wind-pollinating catkins. They are 26 | produced in spring shortly after the new leaves appear. 27 | 28 | The bark is smooth and light grey. The fruit is a small, sharply three-angled 29 | nut 10–15 mm (3⁄8–5⁄8 in) long, borne singly or in pairs in soft-spined husks 30 | 1.5–2.5 cm (5⁄8–1 in) long, known as cupules. The husk can have a variety of 31 | spine- to scale-like appendages, the character of which is, in addition to leaf 32 | shape, one of the primary ways beeches are differentiated.[3] The nuts are 33 | edible, though bitter (though not nearly as bitter as acorns) with a high tannin 34 | content, and are called beechnuts or beechmast. 35 | 36 | # [Birch](https://en.wikipedia.org/wiki/Birch) 37 | 38 | A birch is a thin-leaved deciduous hardwood tree of the genus Betula, in the 39 | family Betulaceae, which also includes alders, hazels, and hornbeams. It is 40 | closely related to the beech-oak family Fagaceae. They are a typically rather 41 | short-lived pioneer species. 42 | 43 | Birch species are generally small to medium-sized trees or shrubs, mostly of 44 | northern temperate and boreal climates. The simple leaves are alternate, singly 45 | or doubly serrate, feather-veined, petiolate and stipulate. They often appear in 46 | pairs, but these pairs are really borne on spur-like, two-leaved, lateral 47 | branchlets. The fruit is a small samara, although the wings may be obscure in 48 | some species. They differ from the alders in that the female catkins are not 49 | woody and disintegrate at maturity, falling apart to release the seeds, unlike 50 | the woody, cone-like female alder catkins. 51 | 52 | The bark of all birches is characteristically marked with long, horizontal 53 | lenticels, and often separates into thin, papery plates, especially upon the 54 | paper birch. Distinctive colors give the common names gray, white, black, silver 55 | and yellow birch to different species. 56 | 57 | The buds form early and are full grown by midsummer, all are lateral, no 58 | terminal bud is formed; the branch is prolonged by the upper lateral bud. The 59 | wood of all the species is close-grained with a satiny texture and capable of 60 | taking a fine polish; its fuel value is fair. 61 | -------------------------------------------------------------------------------- /docs/theme/assets/images/kubernetes.svg: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # MkDocs Newsletter 2 | 3 | [![Actions Status](https://github.com/lyz-code/mkdocs-newsletter/workflows/Tests/badge.svg)](https://github.com/lyz-code/mkdocs-newsletter/actions) 4 | [![Actions Status](https://github.com/lyz-code/mkdocs-newsletter/workflows/Build/badge.svg)](https://github.com/lyz-code/mkdocs-newsletter/actions) 5 | [![Coverage Status](https://coveralls.io/repos/github/lyz-code/mkdocs-newsletter/badge.svg?branch=main)](https://coveralls.io/github/lyz-code/mkdocs-newsletter?branch=main) 6 | 7 | MkDocs plugin to show the changes of documentation repositories in a user 8 | friendly format, at the same time that it's easy for the authors to maintain. 9 | 10 | It creates daily, weekly, monthly and yearly newsletter articles with the 11 | changes of each period. Those pages, stored under the `Newsletters` section, are 12 | filled with the changes extracted from the commit messages of the git history. 13 | The changes are grouped by categories, subcategories and then by file using the 14 | order of the site's navigation structure. [RSS feeds](rss_feeds.md) are also 15 | created for each newsletter type, so it's easy for people to keep updated with 16 | the evolution of the site. 17 | 18 | It assumes that you're using [semantic versioning](https://semver.org/) or our 19 | [enhanced version](docs/usage.md#commit-message-guidelines) to create your commits. Only 20 | those changes that are interesting to the reader will be added to the 21 | newsletter. You can use 22 | [mkdocs-rss-plugin](https://github.com/Guts/mkdocs-rss-plugin) instead if this 23 | workflow doesn't feel good. 24 | 25 | ![ ](screencast.gif) 26 | 27 | Check [a live 28 | version](https://lyz-code.github.io/blue-book/newsletter/0_newsletter_index/). 29 | 30 | ## Help 31 | 32 | See [documentation](https://lyz-code.github.io/mkdocs-newsletter) for more details. 33 | 34 | ## [Installing](https://lyz-code.github.io/mkdocs-newsletter/install/) 35 | 36 | You should check the [install 37 | docs](https://lyz-code.github.io/mkdocs-newsletter/install/), but in short, 38 | you'll need to: 39 | 40 | ```bash 41 | pip install mkdocs-newsletter 42 | ``` 43 | 44 | And enable this plugin, by changing your `mkdocs.yml`. 45 | 46 | ```yaml 47 | plugins: 48 | - git-revision-date-localized: 49 | type: timeago 50 | - autolinks 51 | - section-index 52 | - mkdocs-newsletter 53 | ``` 54 | 55 | ## Contributing 56 | 57 | For guidance on setting up a development environment, and how to make 58 | a contribution to *mkdocs-newsletter*, see [Contributing to 59 | mkdocs-newsletter](https://lyz-code.github.io/mkdocs-newsletter/contributing). 60 | 61 | ## Donations 62 | 63 | 65 | or 66 | [![ko-fi](https://ko-fi.com/img/githubbutton_sm.svg)](https://ko-fi.com/T6T3GP0V8) 67 | 68 | If you are using some of my open-source tools, have enjoyed them, and want to 69 | say "thanks", this is a very strong way to do it. 70 | 71 | If your product/company depends on these tools, you can sponsor me to ensure I 72 | keep happily maintaining them. 73 | 74 | If these tools are helping you save money, time, effort, or frustrations; or 75 | they are helping you make money, be more productive, efficient, secure, enjoy a 76 | bit more your work, or get your product ready faster, this is a great way to 77 | show your appreciation. Thanks for that! 78 | 79 | And by sponsoring me, you are helping make these tools, that already help you, 80 | sustainable and healthy. 81 | 82 | ## License 83 | 84 | GPLv3 85 | -------------------------------------------------------------------------------- /tests/unit/test_model.py: -------------------------------------------------------------------------------- 1 | """Test the models.""" 2 | 3 | from datetime import datetime 4 | from pathlib import Path 5 | 6 | import pytest 7 | from dateutil import tz 8 | 9 | from mkdocs_newsletter.model import ( 10 | FeedEntry, 11 | LastNewsletter, 12 | Newsletter, 13 | NewsletterSection, 14 | ) 15 | 16 | 17 | def test_last_newsletter_min_returns_the_smallest_date() -> None: 18 | """ 19 | Given: A LastNewsletter object with different dates. 20 | When: min is called 21 | Then: The smallest one is returned 22 | """ 23 | last_changes = LastNewsletter( 24 | yearly=datetime(2020, 1, 1, tzinfo=tz.tzlocal()), 25 | weekly=datetime(2020, 2, 2, tzinfo=tz.tzlocal()), 26 | ) 27 | 28 | result = last_changes.min() 29 | 30 | assert result == datetime(2020, 1, 1, tzinfo=tz.tzlocal()) 31 | 32 | 33 | def test_last_newsletter_min_returns_none_if_all_empty() -> None: 34 | """ 35 | Given: A LastNewsletter object with no last dates. 36 | When: min is called 37 | Then: None is returned 38 | """ 39 | last_changes = LastNewsletter() 40 | 41 | result = last_changes.min() 42 | 43 | assert result is None 44 | 45 | 46 | # AAA01: No act block 47 | def test_newslettersection_can_order_objects() -> None: # noqa: AAA01 48 | """ 49 | Given: Two NewsletterSection objects with different orders. 50 | When: They are compared 51 | Then: The return the expected behavior 52 | """ 53 | important = NewsletterSection(title="Important section", order=0) 54 | unimportant = NewsletterSection(title="Unimportant section", order=1) 55 | 56 | assert important > unimportant 57 | # SIM204, C0113, C0117: We don't want to simplify to important > unimportant 58 | # because we want to test the __lt__ method 59 | assert not important < unimportant # noqa: C0113, SIM204, C0117 60 | 61 | 62 | # AAA01: No act block 63 | def test_newsletter_can_order_objects() -> None: # noqa: AAA01 64 | """ 65 | Given: Two Newsletter objects with different orders. 66 | When: They are compared 67 | Then: The return the expected behavior 68 | """ 69 | greater = Newsletter(file_=Path("2021_01.md")) 70 | smaller = Newsletter(file_=Path("2020_01.md")) 71 | 72 | assert greater > smaller 73 | # SIM204, C0113, C0117: We don't want to simplify to important > unimportant 74 | # because we want to test the __lt__ method 75 | assert not greater < smaller # noqa: C0113, SIM204, C0117 76 | 77 | 78 | # AAA01: No act block 79 | def test_feedentry_can_order_objects() -> None: # noqa: AAA01 80 | """ 81 | Given: Two FeedEntry objects with different orders. 82 | When: They are compared 83 | Then: The return the expected behavior 84 | """ 85 | greater = FeedEntry( 86 | published=datetime(2021, 1, 1), 87 | title="Greater", 88 | link="https://test.com", # type: ignore 89 | description="", 90 | ) 91 | smaller = FeedEntry( 92 | published=datetime(2020, 1, 1), 93 | title="Smaller", 94 | link="https://test.com", # type: ignore 95 | description="", 96 | ) 97 | 98 | assert greater > smaller 99 | # SIM204, C0113, C0117: We don't want to simplify to important > unimportant 100 | # because we want to test the __lt__ method 101 | assert not greater < smaller # noqa: C0113, SIM204, C0117 102 | 103 | 104 | @pytest.mark.parametrize( 105 | ("property_", "message"), 106 | [ 107 | ("type_", "Can't extract type from file path"), 108 | ("date", "Can't extract date from file path"), 109 | ], 110 | ) 111 | def test_newsletter_handles_wrong_path(property_: str, message: str) -> None: 112 | """ 113 | Given: A Newsletter with a path that doesn't have a date. 114 | When: calling the type_ and date methods 115 | Then: Errors are raised 116 | """ 117 | newsletter = Newsletter(file_=Path("wrong_path.md")) 118 | 119 | with pytest.raises(ValueError, match=message): 120 | getattr(newsletter, property_) 121 | -------------------------------------------------------------------------------- /docs/usage.md: -------------------------------------------------------------------------------- 1 | Every time you build the site, the plugin will inspect the git history and 2 | create the new newsletter articles under the `docs/newsletter` directory and 3 | configure the `Newsletter` section. 4 | 5 | The entrypoints for the authors are: 6 | 7 | * [Writing the commit messages](#commit-message-guidelines). 8 | * [Manually changing the created newsletter 9 | articles](#manual-newsletter-changes): to fix errors. 10 | 11 | # Commit message guidelines 12 | 13 | The plugin assumes that you're using the [Angular semantic versioning 14 | format](https://github.com/angular/angular/blob/22b96b9/CONTRIBUTING.md#-commit-message-guidelines) 15 | to create the commits. Adapted to a documentation repository such as the [digital 16 | gardens](https://lyz-code.github.io/blue-book/digital_garden/), the structure 17 | would be: 18 | 19 | ``` 20 | {type_of_change}({file_changed}): {short_description} 21 | 22 | {full_description} 23 | ``` 24 | 25 | Where: 26 | 27 | * `type_of_change` is one of: 28 | 29 | * `feat`: Add new content to the repository, it can be a new file or new content on an 30 | existent file. 31 | * `fix`: Correct existing content. 32 | * `perf`: Improve existing content. 33 | * `refactor`: Reorder the articles content. 34 | * `style`: Correct grammar, orthography or broken links. 35 | * `ci`: Change the continuous integration pipelines. 36 | * `chore`: Update the dependencies required to build the site. 37 | 38 | * `file_changed`: Name of the changed file (without the `.md` extension). 39 | * `short_description`: A succinct description of the change. It doesn't need to 40 | start with a capitalize letter nor end with a dot. 41 | * `full_description`: A summary of the added changes. 42 | 43 | For example: 44 | 45 | ``` 46 | feat(adr): introduce the Architecture Decision Records 47 | 48 | [ADR](https://github.com/joelparkerhenderson/architecture_decision_record) are 49 | short text documents that captures an important architectural decision made 50 | along with its context and consequences. 51 | ``` 52 | 53 | Only changes of type `feat`, `fix`, `perf` or `refactor` will be added to the 54 | newsletter. The reader is not interested in the others. 55 | 56 | ## Multiple changes in the same commit 57 | 58 | When growing [digital 59 | gardens](https://lyz-code.github.io/blue-book/digital_garden/), it's normal to 60 | do many small changes on different files. Making a commit for each of them is 61 | cumbersome and can break your writing flow. That's why the plugin is able to 62 | parse different changes from the same commit. For example: 63 | 64 | ``` 65 | feat(pexpect): introduce the pexpect python library 66 | 67 | A pure Python module for spawning 68 | child applications; controlling them; and responding to expected patterns in 69 | their output. Pexpect works like Don Libes’ Expect. Pexpect allows your script 70 | to spawn a child application and control it as if a human were typing commands. 71 | 72 | style(prompt_toolkit): correct links and indentations 73 | 74 | fix(python_snippets): explain how to show the message in custom exceptions 75 | 76 | feat(python_snippets): explain how to import a module or object from within a python program 77 | ``` 78 | 79 | ## Link specific parts of the articles 80 | 81 | You can specify the section of the article where the change has been made by 82 | appending the anchor to the file changed. For example: 83 | 84 | ``` 85 | perf(prometheus_installation#upgrading-notes): Add upgrading notes from 10.x -> 11.1.7 86 | ``` 87 | 88 | The format of the anchor can be Markdown's default or you can use the user 89 | friendly one with caps and spaces `perf(prometheus_installation#Upgrading 90 | notes)`. 91 | 92 | ## Rich full description content 93 | 94 | The `full_description` content will be processed by MkDocs, that means that all 95 | it's features applies, such as autolinking or admonitions. 96 | 97 | ~~~markdown 98 | perf(prometheus_installation): Add upgrading notes from 10.x -> 11.1.7 99 | 100 | !!! warning "Don't upgrade to 12.x if you're still using Helm 2." 101 | 102 | Helm is deprecated and you should migrate to v3. 103 | ~~~ 104 | 105 | # Manual newsletter changes 106 | 107 | To change the contents of the newsletters directly edit the files under 108 | `docs/newsletters`. 109 | -------------------------------------------------------------------------------- /src/mkdocs_newsletter/services/git.py: -------------------------------------------------------------------------------- 1 | """Gather all the orchestration functionality required by the program to work. 2 | 3 | Classes and functions that connect the different domain model objects with the adapters 4 | and handlers to achieve the program's purpose. 5 | """ 6 | 7 | import datetime 8 | import re 9 | from contextlib import suppress 10 | from typing import List, Optional, Tuple 11 | 12 | from dateutil import tz 13 | from git import Commit, Repo 14 | 15 | from ..model import Change 16 | 17 | TYPES = { 18 | "feat": "feature", 19 | "fix": "fix", 20 | "test": "test", 21 | "docs": "documentation", 22 | "style": "style", 23 | "refactor": "refactor", 24 | "build": "build", 25 | "ci": "ci", 26 | "perf": "performance", 27 | "chore": "chore", 28 | } 29 | 30 | 31 | def semantic_changes( 32 | repo: Repo, min_date: Optional[datetime.datetime] = None 33 | ) -> List[Change]: 34 | """Extract meaningful changes from a git repository. 35 | 36 | Args: 37 | repo: Git repository to analyze. 38 | 39 | Returns: 40 | changes: List of Change objects. 41 | """ 42 | now = datetime.datetime.now(tz=tz.tzlocal()) 43 | if min_date is None: 44 | min_date = datetime.datetime(1800, 1, 1, tzinfo=tz.tzlocal()) 45 | 46 | commits = [ 47 | commit 48 | for commit in repo.iter_commits(rev=repo.head.reference) 49 | if commit.authored_datetime < now and commit.authored_datetime > min_date 50 | ] 51 | 52 | return commits_to_changes(commits) 53 | 54 | 55 | def commits_to_changes(commits: List[Commit]) -> List[Change]: 56 | """Extract the semantic changes from a list of commits. 57 | 58 | Args: 59 | commits: List of commits to parse. 60 | 61 | Returns: 62 | changes: List of semantic changes. 63 | """ 64 | changes = [] 65 | 66 | for commit in commits: 67 | with suppress(ValueError): 68 | changes += commit_to_changes(commit) 69 | 70 | return changes 71 | 72 | 73 | def commit_to_changes(commit: Commit) -> List[Change]: 74 | """Extract the semantic changes from a commit. 75 | 76 | Args: 77 | commit: Commit to parse. 78 | 79 | Returns: 80 | changes: List of semantic changes. 81 | """ 82 | changes: List[Change] = [] 83 | remaining = str(commit.message) 84 | 85 | while True: 86 | try: 87 | change, next_remaining = _parse_change(remaining, commit.authored_datetime) 88 | except ValueError: 89 | return changes 90 | changes.append(change) 91 | if next_remaining is None: 92 | break 93 | remaining = next_remaining 94 | return changes 95 | 96 | 97 | def _parse_change( 98 | message: str, date: datetime.datetime 99 | ) -> Tuple[Change, Optional[str]]: 100 | """Extract a semantic change from a commit message. 101 | 102 | Args: 103 | message: Commit message to parse. 104 | 105 | Returns: 106 | changes: List of semantic changes. 107 | remaining: The rest of the commit message. 108 | 109 | Raises: 110 | ValueError: when the commit message doesn't follow the commit guidelines. 111 | """ 112 | commit_regexp = re.compile( 113 | rf"(?P{'|'.join(TYPES.keys())})" 114 | r"(?:\((?P[^\)]+)\))?" 115 | r": (?P[^\n\n]+)" 116 | r"(:?\n\n(?P.+))?", 117 | re.DOTALL, 118 | ) 119 | 120 | commit_match = commit_regexp.match(message) 121 | if not commit_match: 122 | raise ValueError(f"Unable to parse the given commit message: {message}") 123 | 124 | change = Change( 125 | date=date, 126 | summary=_clean_summary(commit_match.group("summary")), 127 | type_=TYPES[commit_match.group("type")], 128 | scope=commit_match.group("scope"), 129 | ) 130 | 131 | remaining = commit_match.group("text") 132 | 133 | description_lines: List[str] = [] 134 | while remaining not in [None, ""] and not commit_regexp.match(remaining): 135 | remaining_lines = remaining.split("\n\n") 136 | description_lines.append(remaining_lines.pop(0)) 137 | remaining = "\n\n".join(remaining_lines) 138 | 139 | if len(description_lines) > 0: 140 | change.message = "\n\n".join(description_lines) 141 | 142 | return change, remaining 143 | 144 | 145 | def _clean_summary(summary: str) -> str: 146 | """Clean the commit summary line. 147 | 148 | Ensure that: 149 | 150 | * The first character of the first word is in upper caps. 151 | * The line ends with a dot. 152 | """ 153 | summary = summary[0].upper() + summary[1:] 154 | if summary[-1] != ".": 155 | summary += "." 156 | 157 | return summary 158 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | .DEFAULT_GOAL := test 2 | isort = pdm run isort src tests 3 | black = pdm run black --target-version py37 src tests 4 | 5 | .PHONY: install 6 | install: 7 | pdm install --dev 8 | 9 | .PHONY: update 10 | update: 11 | @echo "-------------------------" 12 | @echo "- Updating dependencies -" 13 | @echo "-------------------------" 14 | 15 | pdm update --no-sync --update-eager 16 | pdm sync --clean 17 | 18 | @echo "\a" 19 | 20 | .PHONY: update-production 21 | update-production: 22 | @echo "------------------------------------" 23 | @echo "- Updating production dependencies -" 24 | @echo "------------------------------------" 25 | 26 | pdm update --production --no-sync --update-eager 27 | pdm sync --clean 28 | 29 | @echo "\a" 30 | 31 | .PHONY: outdated 32 | outdated: 33 | @echo "-------------------------" 34 | @echo "- Outdated dependencies -" 35 | @echo "-------------------------" 36 | 37 | pdm update --dry-run --unconstrained 38 | 39 | @echo "\a" 40 | 41 | .PHONY: format 42 | format: 43 | @echo "----------------------" 44 | @echo "- Formating the code -" 45 | @echo "----------------------" 46 | 47 | $(black) 48 | autoimport . 49 | $(isort) 50 | 51 | @echo "" 52 | 53 | .PHONY: lint 54 | lint: 55 | @echo "--------------------" 56 | @echo "- Testing the lint -" 57 | @echo "--------------------" 58 | 59 | pdm run flakeheaven lint src/ tests/ 60 | $(isort) --check-only --df 61 | $(black) --check --diff 62 | 63 | @echo "" 64 | 65 | .PHONY: mypy 66 | mypy: 67 | @echo "----------------" 68 | @echo "- Testing mypy -" 69 | @echo "----------------" 70 | 71 | pdm run mypy src tests 72 | 73 | @echo "" 74 | 75 | .PHONY: test 76 | test: test-code test-examples 77 | 78 | @echo "\a" 79 | 80 | .PHONY: test-code 81 | test-code: 82 | @echo "----------------" 83 | @echo "- Testing code -" 84 | @echo "----------------" 85 | 86 | pdm run pytest --cov-report term-missing --cov src tests ${ARGS} 87 | 88 | @echo "" 89 | 90 | .PHONY: test-examples 91 | test-examples: 92 | @echo "--------------------" 93 | @echo "- Testing examples -" 94 | @echo "--------------------" 95 | 96 | @find docs/examples -type f -name '*.py' | xargs -I'{}' sh -c 'echo {}; pdm run python {} >/dev/null 2>&1 || (echo "{} failed" ; exit 1)' 97 | @echo "" 98 | 99 | # pdm run pytest docs/examples/* 100 | 101 | @echo "" 102 | 103 | .PHONY: all 104 | all: lint mypy test security build-docs 105 | 106 | @echo "\a" 107 | 108 | .PHONY: clean 109 | clean: 110 | @echo "---------------------------" 111 | @echo "- Cleaning unwanted files -" 112 | @echo "---------------------------" 113 | 114 | rm -rf `find . -name __pycache__` 115 | rm -f `find . -type f -name '*.py[co]' ` 116 | rm -f `find . -type f -name '*.rej' ` 117 | rm -rf `find . -type d -name '*.egg-info' ` 118 | rm -rf `find . -type d -name '.mypy_cache' ` 119 | rm -f `find . -type f -name '*~' ` 120 | rm -f `find . -type f -name '.*~' ` 121 | rm -rf .cache 122 | rm -rf .pytest_cache 123 | rm -rf .mypy_cache 124 | rm -rf htmlcov 125 | rm -f .coverage 126 | rm -f .coverage.* 127 | rm -rf build 128 | rm -rf dist 129 | rm -f src/*.c pydantic/*.so 130 | rm -rf site 131 | rm -rf docs/_build 132 | rm -rf docs/.changelog.md docs/.version.md docs/.tmp_schema_mappings.html 133 | rm -rf codecov.sh 134 | rm -rf coverage.xml 135 | 136 | @echo "" 137 | 138 | .PHONY: docs 139 | docs: test-examples 140 | @echo "-------------------------" 141 | @echo "- Serving documentation -" 142 | @echo "-------------------------" 143 | 144 | pdm run mkdocs serve 145 | 146 | @echo "" 147 | 148 | .PHONY: bump 149 | bump: pull-main bump-version build-package upload-pypi clean 150 | 151 | @echo "\a" 152 | 153 | .PHONY: pull-main 154 | pull-main: 155 | @echo "------------------------" 156 | @echo "- Updating repository -" 157 | @echo "------------------------" 158 | 159 | git checkout main 160 | git pull 161 | 162 | @echo "" 163 | 164 | .PHONY: build-package 165 | build-package: clean 166 | @echo "------------------------" 167 | @echo "- Building the package -" 168 | @echo "------------------------" 169 | 170 | pdm build 171 | 172 | @echo "" 173 | 174 | .PHONY: build-docs 175 | build-docs: 176 | @echo "--------------------------" 177 | @echo "- Building documentation -" 178 | @echo "--------------------------" 179 | 180 | pdm run mkdocs build --strict 181 | 182 | @echo "" 183 | 184 | .PHONY: upload-pypi 185 | upload-pypi: 186 | @echo "-----------------------------" 187 | @echo "- Uploading package to pypi -" 188 | @echo "-----------------------------" 189 | 190 | twine upload -r pypi dist/* 191 | 192 | @echo "" 193 | 194 | .PHONY: bump-version 195 | bump-version: 196 | @echo "---------------------------" 197 | @echo "- Bumping program version -" 198 | @echo "---------------------------" 199 | 200 | pdm run cz bump --changelog --no-verify 201 | git push 202 | git push --tags 203 | 204 | @echo "" 205 | 206 | .PHONY: security 207 | security: 208 | @echo "--------------------" 209 | @echo "- Testing security -" 210 | @echo "--------------------" 211 | 212 | pdm run pip-audit 213 | @echo "" 214 | pdm run bandit -r src 215 | 216 | @echo "" 217 | 218 | .PHONY: version 219 | version: 220 | @python -c "import mkdocs_newsletter.version; print(mkdocs_newsletter.version.version_info())" 221 | -------------------------------------------------------------------------------- /src/mkdocs_newsletter/services/rss.py: -------------------------------------------------------------------------------- 1 | """Define the RSS management services.""" 2 | 3 | import datetime 4 | import os 5 | import re 6 | from contextlib import suppress 7 | from typing import List, Optional 8 | 9 | from bs4 import BeautifulSoup 10 | from jinja2 import Environment, PackageLoader, select_autoescape 11 | from mkdocs.config.base import Config 12 | 13 | from ..model import Feed, FeedEntry, NewsletterType 14 | from ..version import __version__ 15 | from .newsletter import _list_newsletters 16 | 17 | TTL = { 18 | "daily": 1440, 19 | "weekly": 10080, 20 | "monthly": 43200, 21 | "yearly": 525600, 22 | } 23 | 24 | 25 | def create_rss(config: Config, working_dir: str) -> None: 26 | """Create RSS feed with the newsletters of a period.""" 27 | feed_types = [type_.value for type_ in NewsletterType] 28 | for feed_type in feed_types: 29 | feed = build_rss_feed(config, working_dir, feed_type) 30 | 31 | env = Environment( 32 | loader=PackageLoader("mkdocs_newsletter", "templates"), 33 | autoescape=select_autoescape(["html", "xml"]), 34 | ) 35 | template = env.get_template("rss.xml.j2") 36 | 37 | feed_path = os.path.join(config["site_dir"], f"{feed_type}.xml") 38 | feed_content = template.render(feed=feed) 39 | with open(feed_path, "+w", encoding="utf-8") as feed_file: 40 | feed_file.write(feed_content) 41 | 42 | 43 | def build_rss_feed(config: Config, working_dir: str, type_: str) -> Feed: 44 | """Create the RSS feed data from the content. 45 | 46 | Args: 47 | config: MkDocs config object. 48 | type_: type of feed, one of: daily, weekly, monthly or yearly. 49 | 50 | Returns: 51 | Feed object with the data 52 | """ 53 | site_url = re.sub("/$", "", config["site_url"]) 54 | 55 | try: 56 | logo_url: Optional[str] = f"{site_url}/{config['theme']['logo']}" 57 | except KeyError: 58 | logo_url = None 59 | 60 | author = config.get("site_author") 61 | 62 | entries = _build_rss_entries(config, working_dir, type_, author) 63 | 64 | try: 65 | published = max(entries).published 66 | except ValueError: 67 | published = datetime.datetime.now() 68 | 69 | return Feed( 70 | ttl=TTL[type_], 71 | generator=f"mkdocs-newsletter - v{__version__}", 72 | title=config.get("site_name"), 73 | link=site_url, # type: ignore 74 | rss_link=f"{site_url}/{type_}.xml", # type: ignore 75 | logo=logo_url, # type: ignore 76 | description=config.get("site_description"), 77 | author=author, 78 | published=published, 79 | entries=entries, 80 | ) 81 | 82 | 83 | def _build_rss_entries( 84 | config: Config, 85 | working_dir: str, 86 | type_: str, 87 | author: Optional[str], 88 | ) -> List[FeedEntry]: 89 | """Create the RSS feed entries for a feed type. 90 | 91 | Args: 92 | config: MkDocs config object. 93 | type_: type of feed, one of: daily, weekly, monthly or yearly. 94 | working_dir: Mkdocs root directory. 95 | author: author name. 96 | 97 | Returns: 98 | List of FeedEntry objects with the data. 99 | """ 100 | entries = [] 101 | site_url = re.sub("/$", "", config["site_url"]) 102 | 103 | newsletter_dir = os.path.join( 104 | working_dir, f'{config.get("site_dir", "site")}/newsletter' 105 | ) 106 | 107 | for newsletter in getattr( 108 | _list_newsletters(os.path.join(working_dir, "docs/newsletter")), type_ 109 | ): 110 | with open( 111 | f"{newsletter_dir}/{newsletter.basename}/index.html", "r", encoding="utf-8" 112 | ) as newsletter_file: 113 | html = BeautifulSoup(newsletter_file, "html.parser") 114 | 115 | try: 116 | if html.find("span", {"class": "timeago"}) is None: 117 | raise ValueError("Could not find timeago") 118 | # ignore: The object doesn't have __getitem__ defined but it still works. 119 | # It's probably a typing error 120 | published = html.find("span", {"class": "timeago"})[ 121 | "datetime" 122 | ] # type: ignore 123 | except IndexError: 124 | published = newsletter.date.isoformat() 125 | 126 | # Clean the source code 127 | 128 | # Remove the h1 as it's already in the title 129 | if html.article is None: 130 | raise ValueError("Could not find the article") 131 | if html.article.h1 is None: 132 | raise ValueError("Could not find h1 title") 133 | title = html.article.h1.text 134 | html.article.h1.extract() 135 | 136 | # Remove the Last updated: line 137 | with suppress(AttributeError): 138 | if html.article.div is None: 139 | raise AttributeError 140 | html.article.div.extract() 141 | 142 | # Remove the permalinks 143 | for permalink in html.article.find_all("a", {"class": "headerlink"}): 144 | permalink.extract() 145 | 146 | description = re.sub( 147 | r' 15: 160 | break 161 | 162 | return entries 163 | -------------------------------------------------------------------------------- /docs/theme/assets/images/nlm-ncbi.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | -------------------------------------------------------------------------------- /docs/index.md: -------------------------------------------------------------------------------- 1 | [![Actions Status](https://github.com/lyz-code/mkdocs-newsletter/workflows/Tests/badge.svg)](https://github.com/lyz-code/mkdocs-newsletter/actions) 2 | [![Actions Status](https://github.com/lyz-code/mkdocs-newsletter/workflows/Build/badge.svg)](https://github.com/lyz-code/mkdocs-newsletter/actions) 3 | [![Coverage Status](https://coveralls.io/repos/github/lyz-code/mkdocs-newsletter/badge.svg?branch=main)](https://coveralls.io/github/lyz-code/mkdocs-newsletter?branch=main) 4 | 5 | MkDocs plugin to show the changes of documentation repositories in a user 6 | friendly format, at the same time that it's easy for the authors to maintain. 7 | 8 | It creates daily, weekly, monthly and yearly newsletter articles with the 9 | changes of each period. Those pages, stored under the `Newsletters` section, are 10 | filled with the changes extracted from the commit messages of the git history. 11 | The changes are grouped by categories, subcategories and then by file using the 12 | order of the site's navigation structure. RSS feeds are also created for each 13 | newsletter type, so it's easy for people to keep updated with the evolution of 14 | the site. 15 | 16 | It assumes that you're using [semantic versioning](https://semver.org/) or our 17 | [enhanced version](#commit-message-guidelines) to create your commits. Only 18 | those changes that are interesting to the reader will be added to the 19 | newsletter. You can use 20 | [mkdocs-rss-plugin](https://github.com/Guts/mkdocs-rss-plugin) instead if this 21 | workflow doesn't feel good. 22 | 23 | ![ ](screencast.gif) 24 | 25 | Check [a live 26 | version](https://lyz-code.github.io/blue-book/newsletter/0_newsletter_index/). 27 | 28 | # Alternatives 29 | 30 | ## [mkdocs-rss-plugin](https://github.com/Guts/mkdocs-rss-plugin) 31 | 32 | This cool plugin creates two RSS feeds for the changes of the git history, one 33 | for new files and another for updated ones. 34 | 35 | Creating an RSS entry for each change, it's not the ideal solution for digital 36 | gardens because: 37 | 38 | * *The user will receive too many updates*: In a normal day, you can edit up to 39 | 10 files, which will create 10 RSS entries. That can annoy the user so it 40 | will stop reading your feed. 41 | * *The user will receive updates on irrelevant content*: As an entry is created 42 | for each change, styling and grammar corrections are sent as a new full 43 | entry. 44 | * *The user receives no context of the change*: The RSS entry links to the 45 | article but not it's sections, so if you frequently edit a big file, the, 46 | the user will see no point on the entry and skip it and in the end drop the 47 | RSS. 48 | 49 | If you feel that your use case wont suffer from those conditions, I suggest you 50 | use their plugin instead, as it's much easier to use. 51 | 52 | # Future plans 53 | 54 | If there is enough people interested, we may develop: 55 | 56 | * Support more notification channels such as [email 57 | newsletters](https://github.com/lyz-code/mkdocs-newsletter/issues/16). 58 | * Adding support for software repositories, to give updates on the releases. 59 | 60 | # References 61 | 62 | As most open sourced programs, `mkdocs-newsletter` is standing on the shoulders of 63 | giants, namely: 64 | 65 | [Pytest](https://docs.pytest.org/en/latest) 66 | : Testing framework, enhanced by the awesome 67 | [pytest-cases](https://smarie.github.io/python-pytest-cases/) library that made 68 | the parametrization of the tests a lovely experience. 69 | 70 | [Mypy](https://mypy.readthedocs.io/en/stable/) 71 | : Python static type checker. 72 | 73 | [Flakeheaven](https://github.com/flakeheaven/flakeheaven) 74 | : Python linter with [lots of 75 | checks](https://lyz-code.github.io/blue-book/devops/flakeheaven#plugins). 76 | 77 | [Black](https://black.readthedocs.io/en/stable/) 78 | : Python formatter to keep a nice style without effort. 79 | 80 | [Autoimport](https://lyz-code.github.io/autoimport) 81 | : Python formatter to automatically fix wrong import statements. 82 | 83 | [isort](https://github.com/timothycrosley/isort) 84 | : Python formatter to order the import statements. 85 | 86 | [PDM](https://pdm.fming.dev/) 87 | : Command line tool to manage the dependencies. 88 | 89 | [Mkdocs](https://www.mkdocs.org/) 90 | : To build this documentation site, with the 91 | [Material theme](https://squidfunk.github.io/mkdocs-material). 92 | 93 | [Safety](https://github.com/pyupio/safety) 94 | : To check the installed dependencies for known security vulnerabilities. 95 | 96 | [Bandit](https://bandit.readthedocs.io/en/latest/) 97 | : To finds common security issues in Python code. 98 | 99 | [Yamlfix](https://github.com/lyz-code/yamlfix) 100 | : YAML fixer. 101 | 102 | # Contributing 103 | 104 | For guidance on setting up a development environment, and how to make 105 | a contribution to *mkdocs-newsletter*, see [Contributing to 106 | mkdocs-newsletter](https://lyz-code.github.io/mkdocs-newsletter/contributing). 107 | 108 | ## Donations 109 | 110 | Donate using 111 | Liberapay 112 | or 113 | [![ko-fi](https://ko-fi.com/img/githubbutton_sm.svg)](https://ko-fi.com/T6T3GP0V8) 114 | 115 | If you are using some of my open-source tools, have enjoyed them, and want to 116 | say "thanks", this is a very strong way to do it. 117 | 118 | If your product/company depends on these tools, you can sponsor me to ensure I 119 | keep happily maintaining them. 120 | 121 | If these tools are helping you save money, time, effort, or frustrations; or 122 | they are helping you make money, be more productive, efficient, secure, enjoy a 123 | bit more your work, or get your product ready faster, this is a great way to 124 | show your appreciation. Thanks for that! 125 | 126 | And by sponsoring me, you are helping make these tools, that already help you, 127 | sustainable and healthy. 128 | 129 | -------------------------------------------------------------------------------- /src/mkdocs_newsletter/services/nav.py: -------------------------------------------------------------------------------- 1 | """Gather services to create the newsletters MkDocs nav section.""" 2 | 3 | import calendar 4 | import datetime 5 | import os 6 | import re 7 | from typing import Any, Dict, List, Tuple, Union 8 | 9 | from mkdocs.config.defaults import MkDocsConfig 10 | 11 | NavData = Dict[Union[int, str], Any] 12 | Sections = List[Union[str, Dict[str, Any]]] 13 | 14 | 15 | def build_nav(config: MkDocsConfig, newsletter_dir: str) -> MkDocsConfig: 16 | """Build the navigation section of the newsletters. 17 | 18 | Args: 19 | config: MkDocs configuration object. 20 | newsletter_dir: Directory containing the newsletter articles. 21 | 22 | Returns: 23 | The config object with the newsletters. 24 | """ 25 | nav_data: Dict[Union[int, str], Any] = {} 26 | 27 | newsletter_regex = ( 28 | r"(?P\d{4})" 29 | r"(_w(?P\d{2}))?" 30 | r"(_(?P\d{2}))?" 31 | r"(_(?P\d{2}))?.md" 32 | ) 33 | for file_ in os.scandir(newsletter_dir): 34 | file_path = f"newsletter/{file_.name}" 35 | match = re.match(newsletter_regex, file_.name) 36 | if match is None: 37 | if file_.name == "0_newsletter_index.md": 38 | nav_data["index"] = file_path 39 | continue 40 | 41 | year_match = match.group("year") 42 | week_match = match.group("week_number") 43 | month_match = match.group("month") 44 | day_match = match.group("day") 45 | 46 | if year_match is not None: 47 | year = int(year_match) 48 | nav_data.setdefault(year, {}) 49 | if month_match is not None: 50 | month = int(month_match) 51 | nav_data[year].setdefault(month, {}) 52 | if day_match is not None: 53 | day = int(day_match) 54 | week = datetime.datetime(year, month, day).isocalendar()[1] 55 | nav_data[year][month].setdefault(week, {}) 56 | nav_data[year][month][week][day] = file_path 57 | else: 58 | nav_data[year][month]["index"] = file_path 59 | elif week_match is not None: 60 | week = int(week_match) 61 | month = datetime.datetime.strptime(f"{year}{week}-1", "%Y%W-%w").month 62 | nav_data[year].setdefault(month, {}) 63 | nav_data[year][month].setdefault(week, {}) 64 | nav_data[year][month][week]["index"] = file_path 65 | else: 66 | nav_data[year]["index"] = file_path 67 | 68 | return _nav_data_to_nav(nav_data, config) 69 | 70 | 71 | def _nav_data_to_nav(nav_data: NavData, config: MkDocsConfig) -> MkDocsConfig: 72 | """Convert the nav_data dictionary to the Mkdocs nav section. 73 | 74 | Args: 75 | nav_data: dictionary with the newsletter file data with the following structure. 76 | { 77 | 'index': 0_newsletter_index.md 78 | year: { 79 | 'index': year.md 80 | month_number: { 81 | 'index': year_month.md 82 | week_number: { 83 | 'index': year_wweek_number.md 84 | day: year_month_day.md 85 | } 86 | } 87 | } 88 | } 89 | config: MkDocs configuration object. 90 | 91 | Returns: 92 | MkDocs config object with the list of newsletters under the Newsletters section. 93 | """ 94 | newsletter_nav, nav_data = _initialize_section(nav_data) 95 | 96 | for year, year_data in sorted(nav_data.items(), reverse=True): 97 | year_nav, year_data = _initialize_section(year_data) 98 | 99 | for month, month_data in sorted(year_data.items(), reverse=True): 100 | month_nav, month_data = _initialize_section(month_data) 101 | 102 | for week, week_data in sorted(month_data.items(), reverse=True): 103 | week_nav, week_data = _initialize_section(week_data) 104 | 105 | for day, day_data in sorted(week_data.items(), reverse=True): 106 | day_title = ( 107 | f"{_int_to_ordinal(day)} {calendar.month_name[month]} {year}" 108 | ) 109 | week_nav.append({day_title: day_data}) 110 | month_nav.append({f"{_int_to_ordinal(week)} Week of {year}": week_nav}) 111 | year_nav.append({f"{calendar.month_name[month]} of {year}": month_nav}) 112 | newsletter_nav.append({str(year): year_nav}) 113 | config["nav"].append({"Newsletters": newsletter_nav}) 114 | 115 | return config 116 | 117 | 118 | def _initialize_section(section_data: NavData) -> Tuple[Sections, NavData]: 119 | """Create the section object with the section data. 120 | 121 | If the section_data contains an 'index' key it will index the section page, 122 | otherwise it will create an empty list. 123 | 124 | Args: 125 | section_data: Dictionary with the section data 126 | config: MkDocs config object. 127 | 128 | Returns: 129 | List of sections. 130 | Updated section_data without the 'index' key. 131 | """ 132 | try: 133 | section_nav = [section_data["index"]] 134 | section_data.pop("index") 135 | except KeyError: 136 | section_nav = [] 137 | 138 | return section_nav, section_data 139 | 140 | 141 | def _int_to_ordinal(number: int) -> str: 142 | """Convert an integer into its ordinal representation. 143 | 144 | Args: 145 | number: Number to convert 146 | 147 | Returns: 148 | ordinal representation of the number 149 | """ 150 | suffix = ["th", "st", "nd", "rd", "th"][min(number % 10, 4)] 151 | if 11 <= (number % 100) <= 13: 152 | suffix = "th" 153 | return f"{number}{suffix}" 154 | -------------------------------------------------------------------------------- /docs/contributing.md: -------------------------------------------------------------------------------- 1 | So you've started using `mkdocs-newsletter` and want to show your gratitude to the project, 2 | depending on your programming skills there are different ways to do so. 3 | 4 | # I don't know how to program 5 | 6 | There are several ways you can contribute: 7 | 8 | * [Open an issue](https://github.com/lyz-code/mkdocs-newsletter/issues/new) if you encounter 9 | any bug or to let us know if you want a new feature to be implemented. 10 | * Spread the word about the program. 11 | * Review the [documentation](https://lyz-code.github.io/mkdocs-newsletter) and try to improve 12 | it. 13 | 14 | # I know how to program in Python 15 | 16 | If you have some python knowledge there are some additional ways to contribute. 17 | We've ordered the [issues](https://github.com/lyz-code/mkdocs-newsletter/issues) in 18 | [milestones](https://github.com/lyz-code/mkdocs-newsletter/milestones), check the issues in 19 | the smaller one, as it's where we'll be spending most of our efforts. Try the 20 | [good first 21 | issues](https://github.com/lyz-code/mkdocs-newsletter/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22), 22 | as they are expected to be easier to get into the project. 23 | 24 | We develop the program with 25 | [TDD](https://en.wikipedia.org/wiki/Test-driven_development), so we expect any 26 | contribution to have it's associated tests. We also try to maintain an updated 27 | [documentation](https://lyz-code.github.io/mkdocs-newsletter) of the project, so think if 28 | your contribution needs to update it. 29 | 30 | We know that the expected code quality is above average. Therefore it might 31 | be changeling to get the initial grasp of the project structure, know how to make the 32 | tests, update the documentation or use all the project technology stack. but please 33 | don't let this fact discourage you from contributing: 34 | 35 | * If you want to develop a new feature, explain how you'd like to do it in the related issue. 36 | * If you don't know how to test your code, do the pull request without the tests 37 | and we'll try to do them for you. 38 | 39 | # Issues 40 | 41 | Questions, feature requests and bug reports are all welcome as issues. 42 | **To report a security vulnerability, please see our [security 43 | policy](https://github.com/lyz-code/mkdocs-newsletter/security/policy) instead.** 44 | 45 | To make it as simple as possible for us to help you, please include the output 46 | of the following call in your issue: 47 | 48 | ```bash 49 | python -c "import mkdocs_newsletter.version; print(mkdocs_newsletter.version.version_info())" 50 | ``` 51 | 52 | or if you have `make` installed, you can use `make version`. 53 | 54 | Please try to always include the above unless you're unable to install `mkdocs-newsletter` or know it's not relevant to your question or 55 | feature request. 56 | 57 | # Pull Requests 58 | 59 | *mkdocs-newsletter* is released regularly so you should see your 60 | improvements release in a matter of days or weeks. 61 | 62 | !!! note 63 | Unless your change is trivial (typo, docs tweak etc.), please create an 64 | issue to discuss the change before creating a pull request. 65 | 66 | If you're looking for something to get your teeth into, check out the ["help 67 | wanted"](https://github.com/lyz-code/mkdocs-newsletter/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22) 68 | label on github. 69 | 70 | # Development facilities 71 | 72 | To make contributing as easy and fast as possible, you'll want to run tests and 73 | linting locally. 74 | 75 | !!! note "" 76 | **tl;dr**: use `make format` to fix formatting, `make` to run tests and linting & `make docs` 77 | to build the docs. 78 | 79 | You'll need to have python 3.8 or greater, virtualenv, git, and make installed. 80 | 81 | * Clone your fork and go into the repository directory: 82 | 83 | ```bash 84 | git clone git@github.com:/mkdocs-newsletter.git 85 | cd mkdocs-newsletter 86 | ``` 87 | 88 | * Set up the virtualenv for running tests: 89 | 90 | ```bash 91 | virtualenv -p `which python3.8` env 92 | source env/bin/activate 93 | ``` 94 | 95 | * Install `pdm` our package manager with the next command or using any of the 96 | other [recommended 97 | methods](https://pdm.fming.dev/latest/#recommended-installation-method). 98 | 99 | ```bash 100 | curl -sSL https://raw.githubusercontent.com/pdm-project/pdm/main/install-pdm.py | python3 - 101 | ``` 102 | 103 | * Install mkdocs-newsletter, dependencies and configure the 104 | pre-commits: 105 | 106 | ```bash 107 | make install 108 | ``` 109 | 110 | * Checkout a new branch and make your changes: 111 | 112 | ```bash 113 | git checkout -b my-new-feature-branch 114 | ``` 115 | 116 | * Fix formatting and imports: mkdocs-newsletter uses 117 | [black](https://github.com/ambv/black) to enforce formatting and 118 | [isort](https://github.com/timothycrosley/isort) to fix imports. 119 | 120 | ```bash 121 | make format 122 | ``` 123 | 124 | * Run tests and linting: 125 | 126 | ```bash 127 | make 128 | ``` 129 | 130 | There are more sub-commands in Makefile like `test-code`, `test-examples`, 131 | `mypy` or `security` which you might want to use, but generally `make` 132 | should be all you need. 133 | 134 | If you need to pass specific arguments to pytest use the `ARGS` variable, 135 | for example `make test ARGs='-k test_markdownlint_passes'`. 136 | 137 | * Build documentation: If you have changed the documentation, make sure it 138 | builds the static site. Once built it will serve the documentation at 139 | `localhost:8000`: 140 | 141 | ```bash 142 | make docs 143 | ``` 144 | 145 | * Commit, push, and create your pull request. 146 | 147 | * Make a new release: To generate the changelog of the new changes, build the 148 | package, upload to pypi and clean the build files use `make bump`. 149 | 150 | We'd love you to contribute to *mkdocs-newsletter*! 151 | -------------------------------------------------------------------------------- /tests/assets/test_data/docs/coding/tdd.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: TDD 3 | date: 20200702 4 | author: Lyz 5 | --- 6 | 7 | [Test-driven development](https://en.wikipedia.org/wiki/Test-driven_development) 8 | (TDD) is a software development process that relies on the repetition of a very 9 | short development cycle: requirements are turned into very specific test cases, 10 | then the code is improved so that the tests pass. This is opposed to software 11 | development that allows code to be added that is not proven to meet 12 | requirements. 13 | 14 | # [Abstractions in testing](https://www.cosmicpython.com/book/chapter_03_abstractions.html) 15 | 16 | Writing tests that couple our high-level code with low-level details will make 17 | your life hard, because as the scenarios we consider get more complex, our tests 18 | will get more unwieldy. 19 | 20 | To avoid it, abstract the low-level code from the high-level one, unit test it 21 | and edge-to-edge test the high-level code. 22 | 23 | Edge-to-edge testing involves writing end-to-end tests, substituting the low 24 | level code for fakes that behave in the same way. The advantage of this approach 25 | is that our tests act on the exact same function that's used by our production 26 | code. The disadvantage is that we have to make our stateful components explicit 27 | and pass them around. 28 | 29 | ## Fakes vs Mocks 30 | 31 | *Mocks* are used to verify *how* something gets used. *Fakes* are working 32 | implementations of the things they're replacing, but they're designed for use 33 | only in tests. They wouldn't work in the real life but they can be used to make 34 | assertions about the end state of a system rather than the behaviours along the 35 | way. 36 | 37 | Using fakes instead of mocks have these advantages: 38 | 39 | * Overuse of mocks leads to complicated test suites that fail to explain the 40 | code 41 | * Patching out the dependency you're using makes it possible to unit test the 42 | code, but it does nothing to improve the design. Faking makes you identify 43 | the responsibilities of your codebase, and to separate those 44 | responsibilities into small, focused objects that are easy to replace. 45 | * Tests that use mocks tend to be more coupled to the implementation details 46 | of the codebase. That's because mock tests verify the interactions between 47 | things. This coupling between code and test tends to make tests more 48 | brittle. 49 | 50 | Using the right abstractions is tricky, but here are a few questions that may 51 | help you: 52 | 53 | * Can I choose a familiar Python data structure to represent the state of the 54 | messy system and then try to imagine a single function that can return that 55 | state? 56 | * Where can I draw a line between my systems, where can I carve out a seam to 57 | stick that abstraction in? 58 | * What is a sensible way of dividing things into components with different 59 | responsibilities? What implicit concepts can I make explicit? 60 | * What are the dependencies, and what is the core business logic? 61 | 62 | 63 | # [TDD in High Gear and Low Gear](https://www.cosmicpython.com/book/chapter_05_high_gear_low_gear.html) 64 | 65 | Tests are supposed to help us change our system fearlessly, but often we write 66 | too many tests against the domain model. This causes problems when we want to 67 | change our codebase and find that we need to update tens or even hundreds of 68 | unit tests. 69 | 70 | Every line of code that we put in a test is like a blob of glue, holding the 71 | system in a particular shape. The more low-level tests we have, the harder it 72 | will be to change things. 73 | 74 | Tests can be written at the different levels of abstraction, high level tests 75 | gives us low feedback, low barrier to change and a high system coverage, while 76 | low level tests gives us high feedback, high barrier to change and focused 77 | coverage. 78 | 79 | A test for an HTTP API tells us nothing about the fine grained design of our 80 | objects, because it sits at a much higher level of abstraction. On the other 81 | hand, we can rewrite our entire application and, so long as we don't change the 82 | URLs or request formats, our HTTP tests will continue to pass. This gives us 83 | confidence that large-scale changes, like changing the database schema, haven't 84 | broken our code. 85 | 86 | At the other end of the spectrum, tests in the domain model help us to 87 | understand the objects we need. These tests guide us to a design that makes 88 | sense and reads in the domain language. When our tests read in the domain 89 | language, we feel comfortable that our code matches our intuition about the 90 | problem we're trying to solve. 91 | 92 | We often sketch new behaviours by writing tests at this level to see how the 93 | code might look. When we want to improve the design of the code, though, we will 94 | need to replace or delete these tests, because they are tightly coupled to 95 | a particular implementation. 96 | 97 | Most of the time, when we are adding a new feature or fixing a bug, we don't 98 | need to make extensive changes to the domain model. In these cases, we prefer to 99 | write tests against services because of the lower coupling and higher coverage. 100 | 101 | When starting a new project or when hitting a particularly difficult problem, we 102 | will drop back down to writing tests against the domain model so we get better 103 | feedback and executable documentation of our intent. 104 | 105 | !!! note Test writing level metaphor 106 | When starting a journey, the bicycle needs to be in a low gear so that it 107 | can overcome inertia. Once we're off and running, we can go faster and more 108 | efficiently by changing into a high gear; but if we suddenly encounter 109 | a steep hill or are forced to slow down by a hazard, we again drop down to 110 | a low gear until we can pick up speed again. 111 | 112 | 113 | # References 114 | 115 | * [Architecture Patterns with 116 | Python](https://www.cosmicpython.com/book/preface.html) by 117 | Harry J.W. Percival and Bob Gregory. 118 | 119 | # Further reading 120 | 121 | * [Martin Fowler o Mocks aren't stubs](https://martinfowler.com/articles/mocksArentStubs.html) 122 | -------------------------------------------------------------------------------- /tests/conftest.py: -------------------------------------------------------------------------------- 1 | """Store the classes and fixtures used throughout the tests.""" 2 | 3 | import datetime 4 | import os 5 | import shutil 6 | import textwrap 7 | from pathlib import Path 8 | 9 | import pytest 10 | from dateutil import tz 11 | from git import Actor, Repo 12 | from mkdocs.config.base import load_config 13 | from mkdocs.config.defaults import MkDocsConfig 14 | 15 | 16 | @pytest.fixture(name="repo") 17 | def repo_(tmp_path: Path) -> Repo: 18 | """Initialize an empty git repository with fake unstaged data. 19 | 20 | Args: 21 | tmp_path: Pytest fixture that creates a temporal directory 22 | """ 23 | # Copy the content from `tests/assets/test_data`. 24 | repo_path = tmp_path / "test_data" 25 | os.environ["NEWSLETTER_WORKING_DIR"] = str(repo_path) 26 | shutil.copytree("tests/assets/test_data", repo_path) 27 | 28 | return Repo.init(repo_path) 29 | 30 | 31 | @pytest.fixture(name="full_repo") 32 | def full_repo_(repo: Repo) -> Repo: 33 | """Create a git repository with fake data and history. 34 | 35 | Args: 36 | repo: an initialized Repo 37 | """ 38 | index = repo.index 39 | author = Actor("An author", "author@example.com") 40 | committer = Actor("A committer", "committer@example.com") 41 | 42 | # --------------------------- 43 | # Creates a fake git history 44 | # --------------------------- 45 | 46 | # February 2021 47 | # Mo Tu We Th Fr Sa Su 48 | # 1 2 3 4 5 6 7 49 | # 8 9 10 11 12 13 14 50 | # 15 16 17 18 19 20 21 51 | # 22 23 24 25 26 27 28 52 | 53 | # Commit that doesn't follow the semantic versioning syntax (will be ignored). 54 | commit_date = datetime.datetime(2021, 2, 1, 12, tzinfo=tz.tzlocal()) 55 | index.add(["mkdocs.yml"]) 56 | index.commit( 57 | "Initial skeleton", 58 | author=author, 59 | committer=committer, 60 | author_date=commit_date, 61 | commit_date=commit_date, 62 | ) 63 | 64 | # Single change commit that follows the correct syntax without specifying the file 65 | # it affects 66 | commit_date = datetime.datetime(2021, 2, 2, 12, tzinfo=tz.tzlocal()) 67 | index.add(["docs/emojis.md"]) 68 | index.commit( 69 | "feat: add funny emojis", 70 | author=author, 71 | committer=committer, 72 | author_date=commit_date, 73 | commit_date=commit_date, 74 | ) 75 | 76 | # Multi change commit that follows the correct syntax specifying the files 77 | # it affects and a short description of each change. 78 | commit_date = datetime.datetime(2021, 2, 5, 12, tzinfo=tz.tzlocal()) 79 | index.add(["docs/devops/helm/helm.md", "docs/devops/devops.md"]) 80 | index.commit( 81 | textwrap.dedent( 82 | """\ 83 | feat(helm): introduce Helm the Kubernetes package manager 84 | 85 | [Helm](https://helm.sh/) is the package manager for Kubernetes. Through 86 | charts it helps you define, install and upgrade even the most complex 87 | Kubernetes applications. 88 | 89 | feat(devops): define DevOps 90 | 91 | [DevOps](https://en.wikipedia.org/wiki/DevOps) is a set of practices that 92 | combines software development (Dev) and information-technology operations 93 | (Ops) which aims to shorten the systems development life cycle and provide 94 | continuous delivery with high software quality. 95 | 96 | One of the most important goals of the DevOps initiative is to break the 97 | silos between the developers and the sysadmins, that lead to ill feelings 98 | and unproductivity.""" 99 | ), 100 | author=author, 101 | committer=committer, 102 | author_date=commit_date, 103 | commit_date=commit_date, 104 | ) 105 | 106 | # Single change commit that corrects the style of a file. 107 | with open( 108 | os.path.join(str(repo.working_dir), "docs/emojis.md"), "a", encoding="utf-8" 109 | ) as file_object: 110 | # Simulate the change by appending a string at the end of the file. 111 | file_object.write("correct link") 112 | commit_date = datetime.datetime(2021, 2, 6, 12, tzinfo=tz.tzlocal()) 113 | index.add(["docs/emojis.md"]) 114 | index.commit( 115 | "style(devops): correct link", 116 | author=author, 117 | committer=committer, 118 | author_date=commit_date, 119 | commit_date=commit_date, 120 | ) 121 | 122 | # Single chore change. 123 | commit_date = datetime.datetime(2021, 2, 7, 12, tzinfo=tz.tzlocal()) 124 | index.add(["requirements.txt"]) 125 | index.commit( 126 | "chore: update requirements", 127 | author=author, 128 | committer=committer, 129 | author_date=commit_date, 130 | commit_date=commit_date, 131 | ) 132 | 133 | # Another single change commit done the next week. 134 | commit_date = datetime.datetime(2021, 2, 8, 12, tzinfo=tz.tzlocal()) 135 | index.add(["docs/botany/trees.md"]) 136 | index.commit( 137 | "feat(botany): add ash, birch and beech information", 138 | author=author, 139 | committer=committer, 140 | author_date=commit_date, 141 | commit_date=commit_date, 142 | ) 143 | 144 | # Another multi change commit done the next month. 145 | commit_date = datetime.datetime(2021, 3, 2, 12, tzinfo=tz.tzlocal()) 146 | index.add(["docs/coding/tdd.md", "docs/coding/python/gitpython.md"]) 147 | index.commit( 148 | textwrap.dedent( 149 | """\ 150 | feat(tdd): define test driven development 151 | 152 | feat(gitpython): present the python library""" 153 | ), 154 | author=author, 155 | committer=committer, 156 | author_date=commit_date, 157 | commit_date=commit_date, 158 | ) 159 | 160 | return repo 161 | 162 | 163 | @pytest.fixture(name="config") 164 | def config_(full_repo: Repo) -> MkDocsConfig: 165 | """Load the mkdocs configuration.""" 166 | mkdocs_config = load_config(os.path.join(str(full_repo.working_dir), "mkdocs.yml")) 167 | mkdocs_config["site_dir"] = os.path.join(str(full_repo.working_dir), "site") 168 | return mkdocs_config 169 | -------------------------------------------------------------------------------- /docs/adr/004-article_newsletter_structure.md: -------------------------------------------------------------------------------- 1 | Date: 2021-02-12 2 | 3 | # Status 4 | 6 | Accepted 7 | 8 | # Context 9 | 10 | We want to define how to organize the articles that contain the updates as 11 | a section of the MkDocs site in a way that: 12 | 13 | * It's easy and pleasant for the user to navigate. 14 | * It's easy for us to parse programmatically. 15 | 16 | # Proposals 17 | 18 | 19 | The frequency of updates can be weekly or monthly, structured in a mkdocs nav 20 | similar to: 21 | 22 | ```yaml 23 | Newsletters: 24 | - newsletter/0_newsletter_index.md 25 | - 2020: 26 | - newsletter/2020.md 27 | - February of 2020: 28 | - newsletter/2020_01.md 29 | - 6th Week of 2020: 30 | - newsletter/2020_w06.md 31 | - 11st February 2020: newsletter/2020_01_01.md 32 | - 5th Week of 2020: 33 | - newsletter/2020_w05.md 34 | - 1st February 2020: newsletter/2020_01_01.md 35 | - January of 2020: 36 | - newsletter/2020_01.md 37 | - 1st Week of 2020: 38 | - newsletter/2020_w01.md 39 | - 3rd January 2020: newsletter/2020_01_03.md 40 | - 1st January 2020: newsletter/2020_01_01.md 41 | ``` 42 | 43 | Where: 44 | 45 | * `0_newsletter_index.md`: Is the landing page of the newsletters. It's 46 | prepended with `0_` so it shows the first item when you do `ls` in the 47 | directory. It will be created from a template the first time you run it, 48 | then you can change the file to fit your liking. 49 | * `2020.md`: Is an automatic year summary done at the end of the year. 50 | * `2020_01.md`: Is an automatic month summary for the monthly rss done at the end 51 | of the month joining the changes of the month weeks. 52 | * `2020_w01.md`: Is an automatic week summary for the weekly rss done at the 53 | end of the week joining the changes of the week days. 54 | * `2020_01_01.md`: Is an automatic day summary for the daily rss. 55 | 56 | My first idea as a MkDocs user, and newborn plugin developer was to add the 57 | navigation items to the `nav` key in the `config` object, as it's more easy to 58 | add items to a dictionary I'm used to work with than to dive into the code and 59 | understand how MkDocs creates the navigation. As I understood from the 60 | docs, the files should be created in the `on_files` event. the problem with this 61 | approach is that the only event that allows you to change the `config` is the 62 | `on_config` event, which is before the `on_files` one, so you can't build the 63 | navigation this way after you've created the files. 64 | 65 | Next idea was to add the items in the `on_nav` event, that means creating 66 | yourself the [`Section`](#section), [`Pages`](#page), 67 | [`SectionPages`](#sectionpage) or `Link` objects and append them to the 68 | `nav.items`. [The problem](https://github.com/mkdocs/mkdocs/issues/2324) is 69 | that MkDocs initializes and processes the `Navigation` object in the 70 | [`get_navigation`](https://github.com/mkdocs/mkdocs/blob/master/mkdocs/structure/nav.py#L99) 71 | function. If you want to add items with a plugin in the `on_nav` event, you need 72 | to manually run all the post processing functions such as building the `pages` 73 | attribute, by running the `_get_by_type`, ` _add_previous_and_next_links` or 74 | ` _add_parent_links` yourself. Additionally, when building the site you'll get 75 | the `The following pages exist in the docs directory, but are not included in 76 | the "nav" configuration` error, because that check is done *before* all plugins 77 | change the navigation in the `on_nav` object. 78 | 79 | The last approach is to build the files and tweak the navigation in the 80 | `on_config` event. This approach has the next advantages: 81 | 82 | * You need less knowledge of how MkDocs works. 83 | * You don't need to create the `File` or `Files` objects. 84 | * You don't need to create the `Page`, `Section`, `SectionPage` objects. 85 | * More robust as you rely on existent MkDocs functionality. 86 | 87 | We need to define: 88 | 89 | * How to translate from a list of newsletter file names to the nav structure. 90 | * Whether to build the nav from scratch on each build or reuse the done job. 91 | 92 | ## How to translate from a list of newsletter file names to the nav structure 93 | 94 | The complex part here is how to get the ordering of the elements in the nav 95 | right. We could: 96 | 97 | Create a `nav_data` dictionary with the following structure: 98 | ```yaml 99 | { 100 | year: { 101 | 'index': year.md, 102 | month_number: { 103 | 'index': year_month.md, 104 | week_number: { 105 | 'index': year_wweek_number.md, 106 | day: year_month_day.md 107 | } 108 | } 109 | } 110 | } 111 | ``` 112 | 113 | And then translate the `nav_data` to the actual `nav` contents. 114 | 115 | ## Build the nav from scratch or reuse previous run's nav 116 | 117 | We can either build the whole newsletter nav each time we build the site or we 118 | can store the nav somewhere and only append the new articles. The second option 119 | is more efficient in terms of energy, I don't want to store in the `mkdocs.yml` 120 | file as the newsletter nav can grow fast, making the file dirty. Another 121 | possibility is to save the nav in `docs/newsletter/.newsletter_nav.yaml`. 122 | 123 | If we store the newsletter nav dictionary, it would be difficult to deduce where 124 | does the new entries fit in so that the nav is still ordered. It would make more 125 | sense to store the `nav_data` object, but building that object is relatively 126 | cheap, so it may not be worth even storing it. 127 | 128 | # Decision 129 | 130 | Follow the only proposal regarding the structure, and we'll build the nav 131 | from scratch each time we build the site. 132 | 133 | # Consequences 134 | 135 | -------------------------------------------------------------------------------- /tests/unit/services/test_nav.py: -------------------------------------------------------------------------------- 1 | """Test the building of the newsletter nav. 2 | 3 | The sections and pages need to be defined in the `items` attribute and an ordered 4 | list of Pages and SectionPages need to be stored in the `pages` attribute. 5 | """ 6 | 7 | from git import Repo 8 | from mkdocs.config.defaults import MkDocsConfig 9 | 10 | from mkdocs_newsletter.services.nav import build_nav 11 | 12 | from .test_last_newsletters import create_files 13 | 14 | 15 | def test_build_nav_skips_files_whose_filename_doesnt_match_regexp( 16 | repo: Repo, 17 | config: MkDocsConfig, 18 | ) -> None: 19 | """ 20 | Given: A file that doesn't match the newsletter file format. 21 | When: build_nav is called 22 | Then: The nav doesn't show the file 23 | """ 24 | newsletter_dir = create_files(["invalid_file.md"], repo) 25 | 26 | result = build_nav(config, newsletter_dir) 27 | 28 | assert result["nav"][-1] == {"Newsletters": []} 29 | 30 | 31 | def test_build_nav_accepts_newsletter_landing_page( 32 | repo: Repo, 33 | config: MkDocsConfig, 34 | ) -> None: 35 | """ 36 | Given: A newsletter landing page file. 37 | When: build_nav is called 38 | Then: The nav Newsletter section uses the page. 39 | """ 40 | newsletter_dir = create_files(["0_newsletter_index.md"], repo) 41 | 42 | result = build_nav(config, newsletter_dir) 43 | 44 | assert result["nav"][-1] == {"Newsletters": ["newsletter/0_newsletter_index.md"]} 45 | 46 | 47 | def test_build_nav_creates_year_entry( 48 | repo: Repo, 49 | config: MkDocsConfig, 50 | ) -> None: 51 | """ 52 | Given: A new year file. 53 | When: build_nav is called 54 | Then: the year nav SectionPage entry is created 55 | """ 56 | newsletter_dir = create_files(["2021.md"], repo) 57 | 58 | result = build_nav(config, newsletter_dir) 59 | 60 | assert result["nav"][-1] == {"Newsletters": [{"2021": ["newsletter/2021.md"]}]} 61 | 62 | 63 | def test_build_nav_sorts_year_entries( 64 | repo: Repo, 65 | config: MkDocsConfig, 66 | ) -> None: 67 | """ 68 | Given: two year files. 69 | When: build_nav is called 70 | Then: the year nav entries are sorted descending. 71 | """ 72 | newsletter_dir = create_files(["2020.md", "2021.md"], repo) 73 | 74 | result = build_nav(config, newsletter_dir) 75 | 76 | assert result["nav"][-1] == { 77 | "Newsletters": [ 78 | {"2021": ["newsletter/2021.md"]}, 79 | {"2020": ["newsletter/2020.md"]}, 80 | ] 81 | } 82 | 83 | 84 | def test_build_nav_creates_month_entry( 85 | repo: Repo, 86 | config: MkDocsConfig, 87 | ) -> None: 88 | """ 89 | Given: A new month file. 90 | When: build_nav is called 91 | Then: the month nav entry is created 92 | """ 93 | newsletter_dir = create_files(["2021_01.md"], repo) 94 | 95 | result = build_nav(config, newsletter_dir) 96 | 97 | assert result["nav"][-1]["Newsletters"] == [ 98 | {"2021": [{"January of 2021": ["newsletter/2021_01.md"]}]}, 99 | ] 100 | 101 | 102 | def test_build_nav_sorts_month_entries( 103 | repo: Repo, 104 | config: MkDocsConfig, 105 | ) -> None: 106 | """ 107 | Given: Two month files. 108 | When: build_nav is called 109 | Then: the month nav entries are sorted. 110 | """ 111 | newsletter_dir = create_files(["2021_01.md", "2021_02.md"], repo) 112 | 113 | result = build_nav(config, newsletter_dir) 114 | 115 | assert result["nav"][-1]["Newsletters"] == [ 116 | { 117 | "2021": [ 118 | {"February of 2021": ["newsletter/2021_02.md"]}, 119 | {"January of 2021": ["newsletter/2021_01.md"]}, 120 | ] 121 | }, 122 | ] 123 | 124 | 125 | def test_build_nav_creates_week_entry( 126 | repo: Repo, 127 | config: MkDocsConfig, 128 | ) -> None: 129 | """ 130 | Given: A new week file. 131 | When: build_nav is called 132 | Then: the week nav entry is created 133 | """ 134 | newsletter_dir = create_files(["2021_w01.md"], repo) 135 | 136 | result = build_nav(config, newsletter_dir) 137 | 138 | assert result["nav"][-1]["Newsletters"][0]["2021"] == [ 139 | {"January of 2021": [{"1st Week of 2021": ["newsletter/2021_w01.md"]}]}, 140 | ] 141 | 142 | 143 | def test_build_nav_sorts_week_entries( 144 | repo: Repo, 145 | config: MkDocsConfig, 146 | ) -> None: 147 | """ 148 | Given: Two week files. 149 | When: build_nav is called 150 | Then: the week nav entries are sorted. 151 | """ 152 | newsletter_dir = create_files(["2021_w01.md", "2021_w02.md"], repo) 153 | 154 | result = build_nav(config, newsletter_dir) 155 | 156 | assert result["nav"][-1]["Newsletters"][0]["2021"] == [ 157 | { 158 | "January of 2021": [ 159 | {"2nd Week of 2021": ["newsletter/2021_w02.md"]}, 160 | {"1st Week of 2021": ["newsletter/2021_w01.md"]}, 161 | ] 162 | }, 163 | ] 164 | 165 | 166 | def test_build_nav_creates_day_entry( 167 | repo: Repo, 168 | config: MkDocsConfig, 169 | ) -> None: 170 | """ 171 | Given: A new day file. 172 | When: build_nav is called 173 | Then: the day nav entry is created 174 | """ 175 | newsletter_dir = create_files(["2021_01_05.md"], repo) 176 | 177 | result = build_nav(config, newsletter_dir) 178 | 179 | year_nav = result["nav"][-1]["Newsletters"][0]["2021"] 180 | assert year_nav[0]["January of 2021"] == [ 181 | {"1st Week of 2021": [{"5th January 2021": "newsletter/2021_01_05.md"}]}, 182 | ] 183 | 184 | 185 | def test_build_nav_sorts_day_entries( 186 | repo: Repo, 187 | config: MkDocsConfig, 188 | ) -> None: 189 | """ 190 | Given: Two day files. 191 | When: build_nav is called 192 | Then: the day nav entries are sorted. 193 | """ 194 | newsletter_dir = create_files(["2021_01_11.md", "2021_01_12.md"], repo) 195 | 196 | result = build_nav(config, newsletter_dir) 197 | 198 | year_nav = result["nav"][-1]["Newsletters"][0]["2021"] 199 | assert year_nav[0]["January of 2021"] == [ 200 | { 201 | "2nd Week of 2021": [ 202 | {"12th January 2021": "newsletter/2021_01_12.md"}, 203 | {"11th January 2021": "newsletter/2021_01_11.md"}, 204 | ] 205 | }, 206 | ] 207 | -------------------------------------------------------------------------------- /tests/unit/services/test_semantic_changes.py: -------------------------------------------------------------------------------- 1 | """Tests the extraction of semantic changes from the git history. 2 | 3 | The time freezes are meant to simulate the special state of the fake git repository 4 | defined in the Given: statement of the test docstring. You can check it in the repo 5 | fixture definition at conftest.py. 6 | """ 7 | 8 | import datetime 9 | import textwrap 10 | from textwrap import dedent 11 | 12 | import pytest 13 | from dateutil import tz 14 | from git import Actor, Repo 15 | 16 | from mkdocs_newsletter import Change, semantic_changes 17 | 18 | author = Actor("An author", "author@example.com") 19 | committer = Actor("A committer", "committer@example.com") 20 | 21 | 22 | @pytest.mark.freeze_time("2021-02-01T12:00:00") 23 | def test_changes_dont_extract_commits_that_dont_comply_with_syntax(repo: Repo) -> None: 24 | """ 25 | Given: A mkdocs git repo with a change whose message is not compliant with the 26 | commit message guidelines. 27 | When: changes is called 28 | Then: No Change is returned 29 | """ 30 | commit_date = datetime.datetime(2021, 2, 1, tzinfo=tz.tzlocal()) 31 | repo.index.add(["mkdocs.yml"]) 32 | repo.index.commit( 33 | "Initial skeleton", 34 | author=author, 35 | committer=committer, 36 | author_date=commit_date, 37 | commit_date=commit_date, 38 | ) 39 | 40 | result = semantic_changes(repo) 41 | 42 | assert len(result) == 0 43 | 44 | 45 | @pytest.mark.freeze_time("2021-02-02T12:00:00") 46 | def test_changes_extracts_commits_that_dont_specify_scope(repo: Repo) -> None: 47 | """ 48 | Given: A mkdocs git repo with a change whose message follows the correct syntax 49 | but doesn't specify the scope of the change. 50 | When: changes is called 51 | Then: The expected Change is returned 52 | """ 53 | commit_date = datetime.datetime(2021, 2, 2, tzinfo=tz.tzlocal()) 54 | repo.index.add(["docs/emojis.md"]) 55 | repo.index.commit( 56 | "feat: add funny emojis", 57 | author=author, 58 | committer=committer, 59 | author_date=commit_date, 60 | commit_date=commit_date, 61 | ) 62 | expected_change = Change( 63 | date=commit_date, 64 | summary="Add funny emojis.", 65 | type_="feature", 66 | scope=None, 67 | ) 68 | 69 | result = semantic_changes(repo) 70 | 71 | assert result == [expected_change] 72 | 73 | 74 | @pytest.mark.freeze_time("2021-02-05T12:00:00") 75 | def test_changes_extracts_commits_with_multiple_changes(repo: Repo) -> None: 76 | """ 77 | Given: A mkdocs git repo with a change whose message follows the correct syntax, 78 | contains two semantic changes with scope and description. 79 | When: changes is called. 80 | Then: The two expected Changes are returned, where the message respects the 81 | line breaks. 82 | """ 83 | commit_date = datetime.datetime(2021, 2, 5, tzinfo=tz.tzlocal()) 84 | repo.index.add(["docs/devops/helm/helm.md", "docs/devops/devops.md"]) 85 | repo.index.commit( 86 | textwrap.dedent( 87 | """\ 88 | feat(helm): introduce Helm the Kubernetes package manager 89 | 90 | [Helm](https://helm.sh/) is the package manager for Kubernetes. 91 | Through charts it helps you define, install and upgrade even the most 92 | complex Kubernetes applications. 93 | 94 | feat(devops): define DevOps 95 | 96 | [DevOps](https://en.wikipedia.org/wiki/DevOps) is a set of practices 97 | that combines software development (Dev) and information-technology 98 | operations (Ops) which aims to shorten the systems development life 99 | cycle and provide continuous delivery with high software quality. 100 | 101 | One of the most important goals of the DevOps initiative is to break 102 | the silos between the developers and the sysadmins, that lead to ill 103 | feelings and unproductivity.""" 104 | ), 105 | author=author, 106 | committer=committer, 107 | author_date=commit_date, 108 | commit_date=commit_date, 109 | ) 110 | expected_changes = [ 111 | Change( 112 | date=commit_date, 113 | summary="Introduce Helm the Kubernetes package manager.", 114 | message=dedent( 115 | """\ 116 | [Helm](https://helm.sh/) is the package manager for Kubernetes. 117 | Through charts it helps you define, install and upgrade even the most 118 | complex Kubernetes applications.""" 119 | ), 120 | type_="feature", 121 | scope="helm", 122 | ), 123 | Change( 124 | date=commit_date, 125 | summary="Define DevOps.", 126 | message=dedent( 127 | """\ 128 | [DevOps](https://en.wikipedia.org/wiki/DevOps) is a set of practices 129 | that combines software development (Dev) and information-technology 130 | operations (Ops) which aims to shorten the systems development life 131 | cycle and provide continuous delivery with high software quality. 132 | 133 | One of the most important goals of the DevOps initiative is to break 134 | the silos between the developers and the sysadmins, that lead to ill 135 | feelings and unproductivity.""" 136 | ), 137 | type_="feature", 138 | scope="devops", 139 | ), 140 | ] 141 | 142 | result = semantic_changes(repo) 143 | 144 | assert result == expected_changes 145 | 146 | 147 | @pytest.mark.freeze_time("2021-02-05T12:00:00") 148 | def test_changes_dont_extract_commits_older_than_min_date(repo: Repo) -> None: 149 | """ 150 | Given: A mkdocs git repo with a change older and other younger than the min_date. 151 | When: changes is called 152 | Then: Only the younger Change is returned 153 | """ 154 | repo.index.add(["mkdocs.yml"]) 155 | commit_date = datetime.datetime(2021, 2, 1, tzinfo=tz.tzlocal()) 156 | repo.index.commit( 157 | "feat: Old commit", 158 | author=author, 159 | committer=committer, 160 | author_date=commit_date, 161 | commit_date=commit_date, 162 | ) 163 | commit_date = datetime.datetime(2021, 2, 3, tzinfo=tz.tzlocal()) 164 | repo.index.commit( 165 | "feat: New commit", 166 | author=author, 167 | committer=committer, 168 | author_date=commit_date, 169 | commit_date=commit_date, 170 | ) 171 | 172 | result = semantic_changes(repo, datetime.datetime(2021, 2, 2, tzinfo=tz.tzlocal())) 173 | 174 | assert len(result) == 1 175 | assert result[0].summary == "New commit." 176 | 177 | 178 | @pytest.mark.freeze_time("2021-02-02T12:00:00") 179 | def test_changes_extracts_commits_with_scope_with_spaced_subsection(repo: Repo) -> None: 180 | """ 181 | Given: A mkdocs git repo with a change whose message follows the correct syntax 182 | and specifies the scope of the change with subsection with spaces in it. 183 | When: changes is called 184 | Then: The expected Change is returned 185 | """ 186 | commit_date = datetime.datetime(2021, 2, 2, tzinfo=tz.tzlocal()) 187 | repo.index.add(["docs/emojis.md"]) 188 | repo.index.commit( 189 | "feat(emojis#Spaced subsection): add funny emojis", 190 | author=author, 191 | committer=committer, 192 | author_date=commit_date, 193 | commit_date=commit_date, 194 | ) 195 | expected_change = Change( 196 | date=commit_date, 197 | summary="Add funny emojis.", 198 | type_="feature", 199 | scope="emojis#Spaced subsection", 200 | ) 201 | 202 | result = semantic_changes(repo) 203 | 204 | assert result == [expected_change] 205 | -------------------------------------------------------------------------------- /docs/adr/003-select_the_changes_to_record.md: -------------------------------------------------------------------------------- 1 | Date: 2021-02-12 2 | 3 | # Status 4 | 6 | Accepted. 7 | 8 | Based on: [002](002-initial_plugin_design.md), 9 | [004](004-article_newsletter_structure.md) 10 | 11 | # Context 12 | 13 | We need to create some logic that reads from the git log to: 14 | 15 | * [Extract the last published changes](#extract-the-last-published-changes). To 16 | update the articles and feeds, we need to know which changes have been 17 | already published, so subsequent mkdocs build processes don't repeat work 18 | already done. 19 | 20 | * [Parse the commit messages](#parse-the-commit-messages) to extract the 21 | semantic versioning information from them. 22 | 23 | The message must follow the [angular commit 24 | guidelines](https://github.com/angular/angular.js/blob/master/DEVELOPERS.md#commits) 25 | with the exception that we'll allow many changes in the same commit message. 26 | 27 | * [Extract file context from the MkDocs nav](#extract-the-file-context-from-the-mkdocs-nav) 28 | * [Get the changes that need to be added to the newsletter 29 | articles](#get-the-commits-that-need-to-be-added-to-the-newsletter-articles). 30 | We've defined different levels of aggregation for the user to choose how 31 | often they want to be notified: daily, weekly, monthly or yearly. 32 | 33 | # Proposals 34 | 35 | 36 | ## Extract the last published changes 37 | 38 | We can: 39 | 40 | * *Add meaningful tags to the repository*: If we create a tag `last_weekly` on the 41 | commit of the last published changed for the weekly feed, then we can 42 | add to the changes to publish only those posterior to that commit. 43 | * *Save the last commit ids in a file in the repo*: If we create a hidden 44 | `.last_published.json` with the last commit id of each feed type, 45 | we can add to the feeds only the changes that are posterior to those 46 | commits. 47 | * *Save the last commit ids in the mkdocs.yaml file*: We can save the last 48 | commit ids in the configuration section of the plugin in the mkdocs.yaml 49 | file. 50 | * *Deduce the last published date from existent articles*: We already have this 51 | information in the existent articles. All we would need to do is analyze the 52 | files in the `on_files` and deduce the last content publication. 53 | 54 | ### Add meaningful tags to the repository 55 | 56 | Using tags has the advantage that we could reuse the `Repo` object to extract 57 | the information, which we already use in the services that process the changes. 58 | The disadvantage is that we'll pollute the git repository with tags that may not 59 | be interesting to the user. In the digital garden case, it's not a problem, but 60 | it can be in the changelog one. 61 | 62 | ### Save the last commit ids in a file in the repo 63 | 64 | Storing the commit ids in a hidden json file has the advantage that we don't 65 | pollute the git repository, we can use the 66 | [`json`](https://pydantic-docs.helpmanual.io/usage/exporting_models/#modeljson) 67 | export and import functionality of pydantic. The disadvantage is that we'll 68 | create an additional file in the repository. 69 | [cruft](https://cruft.github.io/cruft/) uses this method. 70 | 71 | ### Save the last commit ids in the mkdocs.yaml file 72 | 73 | Storing the commit ids in the mkdocs.yaml file has the advantage that it doesn't 74 | pollute either the git repository nor the project directory. The disadvantage is 75 | that it pollutes the mkdocs configuration file. 76 | 77 | ### Deduce the last published date from existent articles 78 | 79 | Using the existent articles has these advantages: 80 | 81 | * It doesn't pollute either the git repository, nor the project directory nor 82 | the mkdocs configuration file. 83 | * We can fetch the last published date per feed before parsing the commits, so 84 | instead of analyzing the whole history, we can go from the newest till the 85 | desired dates, making the algorithm more efficient.. 86 | * We don't overwrite the existent articles, so if the user makes changes on 87 | them, they are respected. 88 | 89 | The disadvantage is that we need to code the extraction of that dates from the 90 | files collection, but it should not be difficult. 91 | 92 | Assuming that we have the existent articles organized following 93 | [004](004-article_newsletter_structure.md#how-to-organize-the-articles-in-the-mkdocs-nav-and-in-the-repository), 94 | we can create a function that reads the `files` attribute from the `on_files` 95 | event and deduces the date of the last change of the last newsletter article for 96 | each feed. 97 | 98 | ## Parse the commit messages 99 | 100 | We can: 101 | 102 | * Use the 103 | [python-semantic-release](https://python-semantic-release.readthedocs.io/) 104 | library. 105 | * Use the [commitizen](https://github.com/commitizen-tools/commitizen) library. 106 | * Write our own parser. 107 | 108 | After reviewing the existent libraries, I've found that python-semantic-release 109 | exposes the parsed data in a friendlier way. The only drawback is that it uses 110 | NamedTuples for the objects that hold the parsed data, so if we want to change 111 | one, we need to create a new one from the old. 112 | 113 | Using an existent library, if it's well maintained, is always better than 114 | writing your own. 115 | 116 | ## Extract file context from the MkDocs nav 117 | 118 | For each file that contains changes, we need to deduce the relative position in 119 | the nav against the rest of articles. This is necessary so we're able to group 120 | changes together by category, subcategory and file later in the [newsletter 121 | creation](005-create_the_newsletter_articles.md). 122 | 123 | We can use the 124 | [Deepdiff](https://lyz-code.github.io/blue-book/coding/python/deepdiff/) to 125 | locate the file in the nav tree, and then extract the context information from 126 | the path. 127 | 128 | ## Get the changes that need to be added to the newsletter articles 129 | 130 | For a change to be added to each feed it needs to: 131 | 132 | * year: Be made before the first day of the year and after the last published 133 | change in the year feed. 134 | * month: Be made before the first day of the month and after the last published 135 | change in the month feed. 136 | * week: Be made before the last Monday and after the last published change in 137 | the week feed. 138 | * day: Be made before today and after the last published change in the day 139 | feed. 140 | 141 | # Decision 142 | 143 | We will: 144 | 145 | * Run this phase in the 146 | [`on_files`](https://www.mkdocs.org/user-guide/plugins/#on_files) MkDocs 147 | event to be able to have the list of articles to process. 148 | * [Deduce the last published date from existent 149 | articles](#deduce-the-last-published-date-from-existent-articles) as it's 150 | the cleanest solution in terms of repository pollution. 151 | * Extract the last published dates *before* we parse the commit messages. 152 | * Process only the commits that are posterior to those dates. 153 | * Use python-semantic-release library to parse the commit messages. 154 | * Extract file context from the MkDocs nav 155 | * Get the changes that need to be added to the newsletter articles with the 156 | different feeds. 157 | 158 | # Consequences 159 | 160 | We don't have to write the parsing logic, it's maintained by the community and 161 | it needs few changes to fulfill our needs. 162 | 163 | Advantages: 164 | 165 | * We'll know between builds which changes have been already published. 166 | * We'll reduce the amount of commits that we parse, thus making the process more 167 | efficient. 168 | * We wont overwrite manual user changes in the published articles. 169 | 170 | Disadvantages: 171 | 172 | * We need to code and maintain the extraction of last published date from the 173 | list of repository files. 174 | -------------------------------------------------------------------------------- /docs/install.md: -------------------------------------------------------------------------------- 1 | We've envisioned the plugin with zero configuration as it's quicker to develop 2 | and easier to use. If you'd like to change the default behaviour, please open 3 | [an issue](https://github.com/lyz-code/mkdocs-newsletter/issues/new). 4 | 5 | ```bash 6 | pip install mkdocs-newsletter 7 | ``` 8 | 9 | To enable this plugin, you need to declare it in your config file `mkdocs.yml`. 10 | 11 | ```yaml 12 | plugins: 13 | - git-revision-date-localized: 14 | type: timeago 15 | - autolinks 16 | - section-index 17 | - mkdocs-newsletter 18 | ``` 19 | 20 | We rely on: 21 | 22 | * [git-revision-date-localized](https://github.com/timvink/mkdocs-git-revision-date-localized-plugin) 23 | to extract the latest modification date for each newsletter for the RSS. 24 | * [mkdocs-autolink-plugin](https://github.com/midnightprioriem/mkdocs-autolinks-plugin) 25 | to create the links between the articles. 26 | * [mkdocs-section-index](https://github.com/oprypin/mkdocs-section-index/) to 27 | make the sections clickable. 28 | 29 | To create the RSS feeds well we also need the next configuration: 30 | 31 | ```yaml 32 | site_url: https://substitute.with.your.site.url 33 | markdown_extensions: 34 | - toc: 35 | baselevel: 2 36 | ``` 37 | 38 | Finally make sure that you have a `nav` section in your `mkdocs.yml` file with 39 | at least one entry, otherwise [the plugin won't 40 | work](https://github.com/lyz-code/mkdocs-newsletter/issues/67). 41 | 42 | # MkDocs configuration enhancements 43 | 44 | There are some MkDocs tweaks that can make the plugin work better: 45 | 46 | * [Add newsletter information in the footer and the 47 | header](#add-newsletter-information-in-the-footer-and-the-header). 48 | * [Update the site automatically](#update-the-site-automatically). 49 | * [Exclude the newsletters from the 50 | search](#exclude-the-newsletters-from-the-search). 51 | 52 | ## Add newsletter information in the footer and the header 53 | 54 | It can be useful to let know the readers that you publish the changes through 55 | newsletters and RSS feeds, we can do that in the footer and the header. 56 | 57 | This section is assuming you're using the Material theme. It may work on others 58 | but I haven't tested it 59 | 60 | ### Footer 61 | 62 | You can add the RSS icon to the [Social 63 | links](https://squidfunk.github.io/mkdocs-material/setup/setting-up-the-footer/#social-links) 64 | footer section that points to the newsletter landing page with the next snippet: 65 | 66 | ```yaml 67 | extra: 68 | social: 69 | - icon: fontawesome/solid/rss 70 | link: https://your.site/newsletter/0_newsletter_index/ 71 | ``` 72 | 73 | You can select from 74 | [`fontawesome/solid/rss-square`](https://github.com/squidfunk/mkdocs-material/blob/master/material/.icons/fontawesome/solid/rss-square.svg) 75 | or 76 | [`fontawesome/solid/rss`](https://github.com/squidfunk/mkdocs-material/blob/master/material/.icons/fontawesome/solid/rss.svg). 77 | 78 | ### Header 79 | 80 | You can add an [announce 81 | banner](https://squidfunk.github.io/mkdocs-material/setup/setting-up-the-header/#announcement-bar) 82 | at the top telling your readers to subscribe to the RSS feed by [extending the 83 | theme](https://squidfunk.github.io/mkdocs-material/customization/#extending-the-theme) 84 | and [overriding the `announce` 85 | block](https://squidfunk.github.io/mkdocs-material/customization/#overriding-blocks) 86 | with something like: 87 | 88 | Assuming that the override theme directory is `theme`: 89 | 90 | !!! note "File: `mkdocs.yml`" 91 | ```yaml 92 | theme: 93 | name: material 94 | custom_dir: theme 95 | ``` 96 | 97 | !!! note "File: `theme/main.html`" 98 | 99 | ```jinja2 100 | {% extends "base.html" %} 101 | 102 | {% block announce %} 103 | 104 | For updates subscribe to the RSS feed 105 | 106 | {% include ".icons/fontawesome/solid/rss.svg" %} 107 | 108 | 109 | {% endblock %} 110 | ``` 111 | 112 | ## Update the site automatically 113 | 114 | It's useful to create a continuous integration pipeline to keep the site updated 115 | and automatically build the newsletters. 116 | 117 | If you're using github, you can use the next configuration: 118 | 119 | ??? note "File: `.github/workflows/update.yml`" 120 | ```yaml 121 | --- 122 | name: github pages 123 | 124 | on: 125 | push: 126 | branches: 127 | - master 128 | schedule: 129 | - cron: 11 08 * * * 130 | 131 | jobs: 132 | deploy: 133 | runs-on: ubuntu-latest 134 | steps: 135 | - uses: actions/checkout@v2 136 | with: 137 | persist-credentials: false 138 | # Number of commits to fetch. 0 indicates all history. 139 | # Default: 1 140 | fetch-depth: 0 141 | 142 | - name: Setup Python 143 | # https://github.com/actions/virtual-environments/issues/1202 144 | # uses: actions/setup-python@v1 145 | # with: 146 | # python-version: '3.8' 147 | uses: actions/setup-python@v2 148 | with: 149 | python-version: 3.8 150 | architecture: x64 151 | 152 | - name: Cache dependencies 153 | uses: actions/cache@v1 154 | with: 155 | path: ~/.cache/pip 156 | key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }} 157 | restore-keys: | 158 | ${{ runner.os }}-pip- 159 | 160 | - name: Install dependencies 161 | run: pip install pip-tools 162 | 163 | - name: Update requirements 164 | run: make update 165 | 166 | - name: Make the site 167 | run: make build-docs 168 | 169 | - name: Commit files 170 | run: | 171 | git config --local user.email "action@github.com" 172 | git config --local user.name "GitHub Action" 173 | git add requirements.txt 174 | git add docs/newsletter 175 | git diff-index --quiet HEAD \ 176 | || git commit -m "chore: update dependency and publish newsletters" 177 | 178 | - name: Push changes 179 | uses: ad-m/github-push-action@master 180 | with: 181 | github_token: ${{ secrets.GITHUB_TOKEN }} 182 | branch: master 183 | 184 | - name: Deploy 185 | uses: peaceiris/actions-gh-pages@v3 186 | with: 187 | deploy_key: ${{ secrets.ACTIONS_DEPLOY_KEY }} 188 | publish_dir: ./site 189 | ``` 190 | 191 | It assumes that you have the next `Makefile`: 192 | 193 | ??? note "File: `Makefile`" 194 | ```makefile 195 | .PHONY: update 196 | update: 197 | @echo "-------------------------" 198 | @echo "- Updating dependencies -" 199 | @echo "-------------------------" 200 | 201 | rm requirements.txt 202 | touch requirements.txt 203 | pip-compile -Ur requirements.in --allow-unsafe 204 | 205 | pip install -r requirements.txt 206 | 207 | @echo "" 208 | 209 | .PHONY: build-docs 210 | build-docs: 211 | @echo "--------------------------" 212 | @echo "- Building documentation -" 213 | @echo "--------------------------" 214 | 215 | mkdocs build 216 | 217 | @echo "" 218 | ``` 219 | 220 | ## Exclude the newsletters from the search 221 | 222 | If you don't want to see the newsletters in the result of the search, use the 223 | [mkdocs-exclude-search](https://pypi.org/project/mkdocs-exclude-search/) plugin 224 | to exclude all articles under `newsletter/`. 225 | 226 | To do so: 227 | 228 | 1. Install the plugin with 229 | ```bash 230 | pip install mkdocs-exclude-search 231 | ``` 232 | 1. Add the following configuration to your `mkdocs.yml` 233 | ```yaml 234 | plugins: 235 | - search 236 | - exclude-search: 237 | exclude: 238 | - newsletter/* 239 | ``` 240 | 241 | Note that `newsletter/*` excludes all markdown files within a directory and its children. The paths must be relative to the `docs/` directory. 242 | -------------------------------------------------------------------------------- /src/mkdocs_newsletter/model.py: -------------------------------------------------------------------------------- 1 | """Module to store the common business model of all entities.""" 2 | 3 | import os 4 | import re 5 | from datetime import datetime, timedelta 6 | from enum import Enum 7 | from pathlib import Path 8 | from typing import List, Optional 9 | 10 | from dateutil import tz 11 | from pydantic import BaseModel, Field, HttpUrl 12 | 13 | 14 | class Change(BaseModel): 15 | """Represent a single semantic change in a git repository. 16 | 17 | Attributes: 18 | date: When the change was done. 19 | summary: short description of the change. 20 | type_: semantic type of change, such as feature or fix. 21 | message: long description of the change. 22 | breaking: if the change breaks previous functionality. 23 | category: name of the group of files that share meaning. 24 | category_order: order of the category against all categories. 25 | subcategory: name of the subgroup of files that share meaning. 26 | category_order: order of the subcategory against all subcategories. 27 | file_: markdown file name. 28 | file_section: title of the file containing the change. 29 | file_section_order: order of the file in the subcategory or category that holds 30 | the file. 31 | file_subsection: title of the section of the file the change belongs to. 32 | """ 33 | 34 | date: datetime 35 | summary: str 36 | scope: Optional[str] 37 | type_: Optional[str] 38 | message: Optional[str] = None 39 | breaking: bool = False 40 | publish: Optional[bool] = None 41 | category: Optional[str] = None 42 | category_order: Optional[int] = None 43 | subcategory: Optional[str] = None 44 | subcategory_order: Optional[int] = None 45 | file_: Optional[str] = None 46 | file_section: Optional[str] = None 47 | file_section_order: Optional[int] = None 48 | file_subsection: Optional[str] = None 49 | 50 | 51 | class DigitalGardenChanges(BaseModel): 52 | """Represents all changes that need to be published for each feed type.""" 53 | 54 | daily: List[Change] = Field(default_factory=list) 55 | weekly: List[Change] = Field(default_factory=list) 56 | monthly: List[Change] = Field(default_factory=list) 57 | yearly: List[Change] = Field(default_factory=list) 58 | 59 | 60 | class NewsletterType(str, Enum): 61 | """Defines the possible newsletter types.""" 62 | 63 | YEARLY = "yearly" 64 | MONTHLY = "monthly" 65 | WEEKLY = "weekly" 66 | DAILY = "daily" 67 | 68 | 69 | class Newsletter(BaseModel): 70 | """Represents a newsletter.""" 71 | 72 | file_: Path 73 | 74 | @property 75 | def basename(self) -> str: 76 | """Return the basename of the Newsletter.""" 77 | return os.path.splitext(self.file_.name)[0] 78 | 79 | @property 80 | def type_(self) -> str: 81 | """Return the type of the Newsletter.""" 82 | if re.match(r"\d{4}$", self.basename): 83 | return "yearly" 84 | if re.match(r"\d{4}_\d{2}$", self.basename): 85 | return "monthly" 86 | if re.match(r"\d{4}_w\d{2}$", self.basename): 87 | return "weekly" 88 | if re.match(r"\d{4}_\d{2}_\d{2}$", self.basename): 89 | return "daily" 90 | raise ValueError("Can't extract type from file path") 91 | 92 | @property 93 | def date(self) -> datetime: 94 | """Return the date of the Newsletter.""" 95 | if re.match(r"\d{4}$", self.basename): 96 | return datetime(int(self.basename), 1, 1, tzinfo=tz.tzlocal()) 97 | if re.match(r"\d{4}_\d{2}$", self.basename): 98 | year = int(self.basename.split("_")[0]) 99 | month = int(self.basename.split("_")[1]) 100 | return datetime(year, month, 1, tzinfo=tz.tzlocal()) 101 | if re.match(r"\d{4}_w\d{2}$", self.basename): 102 | year = int(self.basename.split("_")[0]) 103 | week = int(self.basename.split("w")[1]) 104 | first_day = datetime(year, 1, 1, tzinfo=tz.tzlocal()) 105 | return first_day + timedelta(days=7 * (week - 1) - first_day.weekday()) 106 | if re.match(r"\d{4}_\d{2}_\d{2}$", self.basename): 107 | return datetime.strptime(self.basename, "%Y_%m_%d").replace( 108 | tzinfo=tz.tzlocal() 109 | ) 110 | raise ValueError("Can't extract date from file path") 111 | 112 | def __lt__(self, other: "Newsletter") -> bool: 113 | """Assert if an object is smaller than us. 114 | 115 | Args: 116 | other: Newsletter to compare. 117 | 118 | Raises: 119 | TypeError: If the id type of the objects is not compatible. 120 | """ 121 | return self.basename < other.basename 122 | 123 | def __gt__(self, other: "Newsletter") -> bool: 124 | """Assert if an object is greater than us. 125 | 126 | Args: 127 | other: Newsletter to compare. 128 | 129 | Raises: 130 | TypeError: If the id type of the objects is not compatible. 131 | """ 132 | return self.basename > other.basename 133 | 134 | 135 | class Newsletters(BaseModel): 136 | """Represents the newsletters for each feed type.""" 137 | 138 | yearly: List[Newsletter] = Field(default_factory=list) 139 | monthly: List[Newsletter] = Field(default_factory=list) 140 | weekly: List[Newsletter] = Field(default_factory=list) 141 | daily: List[Newsletter] = Field(default_factory=list) 142 | 143 | def sort(self) -> None: 144 | """Sort the newsletters.""" 145 | self.yearly = sorted(self.yearly, reverse=True) 146 | self.monthly = sorted(self.monthly, reverse=True) 147 | self.weekly = sorted(self.weekly, reverse=True) 148 | self.daily = sorted(self.daily, reverse=True) 149 | 150 | 151 | class LastNewsletter(BaseModel): 152 | """Represents the last newsletter for each feed type.""" 153 | 154 | yearly: Optional[datetime] = None 155 | monthly: Optional[datetime] = None 156 | weekly: Optional[datetime] = None 157 | daily: Optional[datetime] = None 158 | 159 | def min(self) -> Optional[datetime]: 160 | """Return the smallest date of all the feeds.""" 161 | try: 162 | return min(value for key, value in self.dict().items() if value is not None) 163 | except ValueError: 164 | return None 165 | 166 | 167 | class NewsletterSection(BaseModel): 168 | """Represent the section of a newsletter article. 169 | 170 | Attributes: 171 | title: Category title 172 | order: The order in comparison with the other categories 173 | changes: Changes to be printed in the section 174 | subsections: A list of subsections. 175 | """ 176 | 177 | title: str 178 | order: int 179 | url: Optional[str] = None 180 | changes: List[Change] = Field(default_factory=list) 181 | subsections: List["NewsletterSection"] = Field(default_factory=list) 182 | 183 | def __lt__(self, other: "NewsletterSection") -> bool: 184 | """Assert if an object is smaller than us. 185 | 186 | Args: 187 | other: NewsletterSection to compare. 188 | 189 | Raises: 190 | TypeError: If the id type of the objects is not compatible. 191 | """ 192 | return self.order > other.order 193 | 194 | def __gt__(self, other: "NewsletterSection") -> bool: 195 | """Assert if an object is greater than us. 196 | 197 | Args: 198 | other: NewsletterSection to compare. 199 | 200 | Raises: 201 | TypeError: If the id type of the objects is not compatible. 202 | """ 203 | return self.order < other.order 204 | 205 | 206 | class FeedEntry(BaseModel): 207 | """Model an RSS feed entry.""" 208 | 209 | title: str 210 | link: HttpUrl 211 | published: datetime 212 | description: str 213 | author: Optional[str] 214 | image: None = None 215 | 216 | def __lt__(self, other: "FeedEntry") -> bool: 217 | """Assert if an object is smaller than us. 218 | 219 | Args: 220 | other: FeedEntry to compare. 221 | 222 | Raises: 223 | TypeError: If the id type of the objects is not compatible. 224 | """ 225 | return self.published < other.published 226 | 227 | def __gt__(self, other: "FeedEntry") -> bool: 228 | """Assert if an object is greater than us. 229 | 230 | Args: 231 | other: FeedEntry to compare. 232 | 233 | Raises: 234 | TypeError: If the id type of the objects is not compatible. 235 | """ 236 | return self.published > other.published 237 | 238 | 239 | class Feed(BaseModel): 240 | """Model an RSS feed.""" 241 | 242 | ttl: int 243 | generator: str 244 | title: Optional[str] 245 | link: Optional[HttpUrl] 246 | rss_link: Optional[HttpUrl] 247 | published: Optional[datetime] 248 | logo: Optional[HttpUrl] 249 | description: Optional[str] 250 | author: Optional[str] 251 | entries: List[FeedEntry] = Field(default_factory=list) 252 | 253 | 254 | Newsletter.update_forward_refs() 255 | NewsletterSection.update_forward_refs() 256 | FeedEntry.update_forward_refs() 257 | -------------------------------------------------------------------------------- /tests/assets/test_data/docs/devops/devops.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Devops 3 | date: 20200212 4 | author: Lyz 5 | tags: 6 | --- 7 | 8 | [DevOps](https://en.wikipedia.org/wiki/DevOps) is a set of practices that 9 | combines software development (Dev) and information-technology operations (Ops) 10 | which aims to shorten the systems development life cycle and provide continuous 11 | delivery with high software quality. 12 | 13 | One of the most important goals of the DevOps initiative is to break the silos 14 | between the developers and the sysadmins, that lead to ill feelings and 15 | unproductivity 16 | 17 | It's a [relatively new concept](https://en.wikipedia.org/wiki/DevOps#History), 18 | the main ideas emerged in the 1990s and the first conference was in 2009. That 19 | means that as of 2021 there is still a lot of debate of what people understand 20 | as DevOps. 21 | 22 | # DevOps pitfalls 23 | 24 | I've found that the DevOps word leads to some pitfalls that we should try to 25 | avoid. 26 | 27 | ## Getting lost in the label 28 | 29 | [Labels](https://en.wikipedia.org/wiki/Labelling) are a language tool used to 30 | speed up communication by describing someone or something in a word or short 31 | phrase. 32 | 33 | However, there are times when labels achieve the complete oposite, as it's the 34 | case with DevOps, where there are different views on what the label represents 35 | and usually one of the communication parties strongly feels they belong to the 36 | label while the other doesn't agree. These discussions can fall into an 37 | unproductive, agitated semantic debate where each part tries to convince each 38 | other. 39 | 40 | So instead of starting a [twitter 41 | thread](https://twitter.com/christianhujer/status/1356481078626639873) telling 42 | people why they aren't a DevOps team, we could invest those energies in creating 43 | resources that close the gap between both parties. Similarly, instead of 44 | starting an internal discussion of *what do we understand as DevOps?*, we could 45 | discuss how to improve our existent processes so that the team members feel more 46 | comfortable contributing to the application or infrastructure code. 47 | 48 | ## You need to do it all to be awarded the DevOps pin 49 | 50 | I find specially harmful the idea that to be qualified as DevOps you need to 51 | develop and maintain the application at the same time as the infrastructure that 52 | holds it. 53 | 54 | To be able to do that in a typical company product you'll need to know (between 55 | another thousand more things): 56 | 57 | * How to operate the cloud infrastructure where the project lives, which can be 58 | AWS, Google Cloud, Azure or/and baremetal servers. 59 | * Deploy new resources in that infrastructure, which probably would mean knowing 60 | Terraform, Ansible, Docker or/and Kubernetes. 61 | * How to integrate the new resources with the operations processes, for example: 62 | * The monitoring system, so you'll need to know how to use 63 | Prometheus, Nagios, Zabbix or the existent solution. 64 | * The continuous integration or delivery system, that you'll need to know 65 | how to maintain, so you have to know how it works and how is it built. 66 | * The backup system. 67 | * The log centralizer system. 68 | * Infrastructure architecture to know what you need to deploy, how and where. 69 | * To code efficiently in the language that the application is developed in, for 70 | example Java, Python, Rust, Go, PHP or Javascript, in a way that meets the 71 | quality requirements (code style, linters, coverage and documentation). 72 | * Knowing how to test your code in that language. 73 | * Software architecture to structure complex code projects in a maintainable 74 | way. 75 | * The product you're developing to be able to suggest features and fixtures when 76 | the product owner or the stakeholders show their needs. 77 | * How to make the application user friendly so anyone wants to use it. 78 | * And don't forget that you also need to do that in a secure way, so you should 79 | also have to know about pentesting, static and dynamic security tools, 80 | common security vulnerabilities... 81 | 82 | And I could keep on going forever. Even if you could get there (and you won't), it 83 | wouldn't matter, because when you did, the technologies will have changed so much 84 | that you will already be outdated and would need to start over. 85 | 86 | It's the sickness of the fullstack developer. If you make job openings with this 87 | mindset you're going to end up with a team of cis white males in their thirties 88 | or forties that are used to earn 3 or 4 times the minimum salary. No other kind 89 | of people can reach that point and hold it in time. 90 | 91 | But bare with me a little longer, even if you make there. What happens when the 92 | project changes so that you need to: 93 | 94 | * Change the programming language of your application. 95 | * Change the cloud provider. 96 | * Change the deployment system. 97 | * Change the program architecture. 98 | * Change the language framework. 99 | 100 | Or any other thousand of possible changes? You would need to be able to keep up with 101 | them. Noooo way. 102 | 103 | Luckily you are not alone. You are a cog in a team, that as a whole is able to 104 | overcome these changes. That is why we have developers, sysadmins, security, 105 | user experience, quality, scrum master and product owners. So each profile can 106 | design, create, investigate and learn it's best in their area of expertise. In 107 | the merge of all that personal knowledge is where the team thrives. 108 | 109 | DevOps then, as I understand it, is the philosophy where the developers and 110 | sysadmins try their best to break the barriers that separate them. That idea can 111 | be brought to earth by for example: 112 | 113 | * Open discussions on how to: 114 | * Improve the development workflow. 115 | * Make developers or sysadmins life easier. 116 | * Make it easier to use the sysadmin tools. 117 | * Make it easier to understand the developers code. 118 | * Formations on the technologies or architecture used by either side. 119 | * A clear documentation that allows either side to catch up with new changes. 120 | * Periodic meetings to update each other with the changes. 121 | * Periodic meetings to release the tension that have appeared between 122 | them. 123 | * Joined design sessions to decide how to solve problems. 124 | 125 | # Learn path 126 | 127 | DevOps has become a juicy work, if you want to become one, I think you first 128 | need to get the basic knowledge of each of them (developing and operating) 129 | before being able to unlock the benefits of the combination of both. You can try 130 | to learn both at the same time, but I think it can be a daunting task. 131 | 132 | To get the basic knowledge of the Ops side I would: 133 | 134 | * Learn basic Linux administration, otherwise you'll be lost. 135 | 136 | * Learn how to be comfortable searching for anything you don't know, most of 137 | your questions are already answered, and even the most senior people spent 138 | a great amount of time searching for solutions in the project's 139 | documentation, Github issues or Stackoverflow. 140 | 141 | When you start, navigating this knowledge sources is hard and consumes a lot 142 | of your life, but it will get easier with the time. 143 | 144 | * Learn how to use Git. If you can, host your own Gitea, if not, use an existing 145 | service such as Gitlab or Github. 146 | 147 | * Learn how to install and maintain services, (that is why I suggested hosting 148 | your own Gitea). If you don't know what to install, take a look at the 149 | [awesome 150 | self-hosted](https://github.com/awesome-selfhosted/awesome-selfhosted) list. 151 | 152 | * Learn how to use Ansible, from now on try to deploy every machine with it. 153 | 154 | * Build a small project inside AWS so you can get used to the most 155 | common services (VPC, EC2, S3, Route53, RDS), most of them have free-tier 156 | resources so you don't need to pay anything. You can try also with Google 157 | Cloud or Azure, but I recommend against it. 158 | 159 | * Once you are comfortable with AWS, learn how to use Terraform. You could for 160 | example deploy the previous project. From now on only use Terraform to 161 | provision AWS resources. 162 | 163 | * Get into the CI/CD world hosting your own Drone, if not, use [Gitlab 164 | runners](https://docs.gitlab.com/runner/) or [Github 165 | Actions](https://github.com/features/actions). 166 | 167 | To get the basic knowledge of the Dev side I would: 168 | 169 | * Learn the basics of a programming language, for example Python. There are 170 | thousand sources there on how to do it, books, articles, videos, forums or 171 | courses, choose the one that suits you best. 172 | 173 | * As with the Ops path, get comfortable with git and searching for things you 174 | don't know. 175 | 176 | * As soon as you can, start doing small programming projects that make your life 177 | easier. Coding your stuff is what's going to make you internalize the 178 | learned concepts, by finding solutions to the blocks you encounter. 179 | 180 | * Publish those projects into a public git server, don't be afraid if you code 181 | is good enough, it works for you, you did your best and you should be happy 182 | about it. That's all that matters. By doing so, you'll start collaborating 183 | to the open source world and it will probably force yourself to make your 184 | code better. 185 | 186 | * Step into the TDD world, learn why, how and when to test your code. 187 | 188 | * For those projects that you want to maintain, create CI/CD pipelines 189 | that enhance the quality of your code, by for example running your tests or 190 | some linters. 191 | 192 | * Once you're comfortable, try to collaborate with existent projects (right now 193 | you may not now where to look for projects to collaborate, but when you reach 194 | this point, I promise you will). 195 | -------------------------------------------------------------------------------- /docs/adr/001-initial_approach.md: -------------------------------------------------------------------------------- 1 | Date: 2021-02-08 2 | 3 | # Status 4 | 6 | Accepted 7 | 8 | Extended by: [002](002-initial_plugin_design.md) 9 | 10 | # Context 11 | 12 | 13 | Gathering knowledge in my [blue book](https://lyz-code.github.io/blue-book) has 14 | been a pleasant experience in the last year. The only drawback of the system is 15 | that I don't know a user friendly way to inform the readers of content updates 16 | or fixes, such as an RSS feed of/and a periodic newsletter. 17 | 18 | That means that when the user stumbles upon the site, they spend a limited 19 | amount of time reading it, and then forgets of it's existence. 20 | 21 | If they see themselves back to the site, they don't know what have they've 22 | already seen, which can make them feel lost or loosing their time. 23 | 24 | Even if they clone the repository, they need to [go through the 25 | log](https://github.com/nikitavoloboev/knowledge/issues/19) to see the changes, 26 | which is awful. 27 | 28 | And that means that all the efforts we put into growing our [digital 29 | gardens](https://lyz-code.github.io/blue-book/digital_garden/) or our software 30 | applications is not appreciated all that it should be. 31 | 32 | If we created a way to show them what has changed in a friendly way we will: 33 | 34 | * Make their experience more pleasant. 35 | * Make them more prone to visit the site again. 36 | 37 | We must be careful though, we don't want to worsen the bombardment of 38 | information we are suffering. We need to notify the user in a respectful way by 39 | sending them only relevant content at the pace they choose to. 40 | 41 | For the idea to work, we'd first need that the authors want to use the solution. 42 | So we need to build a system that doesn't increase the mental load or effort to 43 | a point that makes it unappealing. That can be done by: 44 | 45 | * Automating all the processes that we can. 46 | * Reducing to the minimum the author entrypoints. 47 | * Making it easy for them to introduce the manual data that we need. 48 | * Reduce the deviation from their current workflow. 49 | 50 | # Proposals 51 | 52 | 53 | To solve the issue we need to decide: 54 | 55 | * Which updates are relevant. 56 | * How to reduce the number of notifications the user receives. 57 | * Which notification channels we want to support. 58 | * Which programming solution to use. 59 | 60 | ## How to extract relevant changes 61 | 62 | ### Manual processing 63 | 64 | At the time of building the update, we could manually review all the changes done since 65 | the last update and store that information in a programmatically processable 66 | way. 67 | 68 | The downside is that we'll need to spend a relative big amount of time to review 69 | content written in the past. I found myself skipping these batch processing 70 | tasks in the past, as my brain sees them as dreadful, long and heavy. 71 | 72 | ### Use semantic versioning in git commit messages 73 | 74 | To give semantic meaning to the changes introduced in the repository, we can use the [semantic 75 | versioning](https://semver.org/) principles to classify and describe each change 76 | in a way that can be programmatically decide if it's worth publishing. 77 | 78 | The idea is to create commits following [a specific 79 | format](https://github.com/angular/angular/blob/22b96b9/CONTRIBUTING.md#-commit-message-guidelines) 80 | that describes what kind of change you're making. I've [been following this 81 | practice](https://github.com/lyz-code/blue-book/commits/master) for a while with 82 | the following personal commit convention: 83 | 84 | ``` 85 | {type_of_change}({file_changed}): {short_description} 86 | {full_description} 87 | ``` 88 | 89 | Where: 90 | 91 | * `type_of_change` is one of: 92 | 93 | * `feat`: Add new content to the repository, it can be a new file or new content on an 94 | existent file. 95 | * `fix`: Correct existing content. 96 | * `style`: Correct grammar, orthography or broken links. 97 | * `ci`: Change the continuous integration pipelines. 98 | * `chore`: Update the python dependencies required to build the site. 99 | 100 | * `file_changed`: name of the file changed (without the `.md` extension). 101 | * `short_description`: A succinct description of the change. It doesn't need to 102 | start with a capitalize letter nor end with a dot. 103 | * `full_description`: A summary of the added changes. 104 | 105 | For example: [a single feat 106 | commit](https://github.com/lyz-code/blue-book/commit/5eb3f57da4de99e58bf25ab4b5e24fbb007f7319) or 107 | [a multientry commit](https://github.com/lyz-code/blue-book/commit/53f7b1f67bc7aa2c654e8bfe286d8175e099747b) 108 | 109 | With this method the responsibility of giving meaning to the changes is shared 110 | in time, which makes the task more appealing and less heavy. Additionally, with 111 | the IDE support, creating the commit messages is easier as you can copy and 112 | paste or autocomplete from the changes themselves. 113 | 114 | ## How to reduce the number of notifications 115 | 116 | There are cases where the user is still spammed even if we select the relevant 117 | changes. 118 | 119 | We can copy the solution adopted by the system's monitorization solutions, which 120 | is to group the changes to reduce the notifications. 121 | 122 | We should present different levels of aggregation, so they can receive daily, 123 | weekly or monthly notifications. 124 | 125 | ## Which notification channels we want to support 126 | 127 | Different users use different tools to get updates on what's going in the world. 128 | We can expose the updates through: 129 | 130 | * An article in the documentation site. 131 | * RSS. 132 | * Email newsletter. 133 | * API. 134 | * Reddit. 135 | * Hackernews. 136 | * GNUsocial. 137 | * Twitter. 138 | * XMPP/Jabber 139 | * Telegram 140 | 141 | The article in the documentation makes sense because it can be the landing page 142 | for the rest of channels, which can trigger further page views. 143 | 144 | Most of the people around me, whom I want to share my updates use RSS, which is 145 | an awesome protocol that is sadly being replaced by Twitter like webs, and we 146 | don't want that to happen. I feel that only "technical" users use this 147 | technology though. 148 | 149 | Everyone has an email, and sending mails through Python is easy, so it can be 150 | the easiest path to reach the global public. 151 | 152 | For the more technical ones, extracting the data from an API can help gluing 153 | together other functionalities. It could help if people want to extend the 154 | functionality that we give, for example by creating more specific RSS feeds 155 | based on categories. As we have a static site, it will only support GET methods, 156 | and we will already publish all the information in RSS format, which is 157 | parseable by programs, so adding another enpoint that exposes the same data in 158 | json format doesn't make much sense. 159 | 160 | I haven't used Reddit or Hackernews enough to know if it makes sense either this 161 | channel. 162 | 163 | I think GNUsocial and Twitter are wrongly being used to replace RSS, so it won't 164 | be a priority. 165 | 166 | A similar argument applies instant message solutions like XMPP or Telegram. 167 | 168 | ## Which programming solution to use 169 | 170 | To create the functionality, we'll need to: 171 | 172 | * Semantic versioning information given by the user in the commit messages since 173 | the last change. 174 | * Link the files mentioned in the commit messages to actual working links. 175 | 176 | The processing can be done through: 177 | 178 | * A [mkdocs plugin](https://www.mkdocs.org/user-guide/plugins/) so it creates it at 179 | build time. It can be inspired by: 180 | 181 | * [mkdocs-new-features-notifier](https://pypi.org/project/mkdocs-new-features-notifier/#history) 182 | * [mkdocs_blog](https://github.com/andyoakley/mkdocs-blog) 183 | * [mkdocs_latest_release_plugin](https://github.com/agarthetiger/mkdocs_latest_release_plugin) 184 | 185 | * An existent external command line tool such as 186 | [python-semantic-release](https://python-semantic-release.readthedocs.io/en/latest/commit-log-parsing.html) 187 | and [commitizen bump](https://github.com/commitizen-tools/commitizen). 188 | 189 | * A complete new external command line tool. 190 | 191 | * An hybrid between both solutions, like [mike](https://github.com/jimporter/mike). 192 | 193 | ### Developing a mkdocs plugin 194 | 195 | The advantages are: 196 | 197 | * It will be easier for people to use, as it already exists a widely used plugin 198 | system in MkDocs. 199 | * It will be easier for us too, as the plugin system gives a lot of information 200 | on the articles, so making working links should be easy. 201 | 202 | The disadvantages are that we'll need to: 203 | 204 | * Assume that the user is not meant to manually trigger the updates creation. 205 | They will need to use a periodic automatic process similar to the one that 206 | generates the website. 207 | 208 | # Decision 209 | 210 | 211 | We will develop a [Mkdocs plugin](#developing-a-mkdocs-plugin) that creates 212 | MkDocs articles whose content: 213 | 214 | * Contains only relevant updates, deduced by the [commit 215 | messages](#use-semantic-versioning-in-git-commit-messages) of the 216 | repository. 217 | * Updates [are grouped to reduce the user 218 | notifications](#how-to-reduce-the-number-of-notifications). 219 | * Is exposed to the user with different levels of aggregation, such as: real 220 | time, daily, weekly, monthly or yearly. 221 | * Is exposed to the user through: 222 | * Articles in the same MkDocs site. 223 | * RSS feeds. 224 | * Email newsletters. 225 | 226 | # Consequences 227 | 228 | The user will be able to be kept updated of the relevant MkDocs site changes at 229 | the pace they desire. 230 | 231 | That will: 232 | 233 | * Make their browsing experience more pleasant. 234 | * Make them more prone to visit the site again. 235 | 236 | For the authors it will mean that they need to: 237 | 238 | * Spend more time and thought writing the commit messages. 239 | * Be tied to the solution we can give them, or contribute the desired changes. 240 | * Debug possible errors produced by the increase of complexity of the system. 241 | 242 | For us it will mean that we'll need to: 243 | 244 | * Build and maintain the logic and code for: 245 | * Extracting and grouping the relevant updates. 246 | * Sending the notifications. 247 | * Keep the system updated. 248 | * React to security vulnerabilities. 249 | * React to issues and pull requests. 250 | --------------------------------------------------------------------------------