├── doc ├── dfetch-action.md ├── images │ ├── dfetch_logo.png │ ├── dfetch_header.png │ ├── local-change-github.png │ ├── out-of-date-jenkins.png │ ├── out-of-date-jenkins2.png │ ├── github-actions-result.png │ ├── gitlab-highlighted-manifest.png │ ├── local-change-github-details.png │ └── gitlab-check-pipeline-result.png ├── landing-page │ ├── robots.txt │ ├── Makefile │ ├── make.bat │ └── static │ │ └── css │ │ └── custom.css ├── changelog.rst ├── robots.txt ├── manifest.rst ├── generate-casts │ ├── environment-demo.sh │ ├── README.md │ ├── import-demo.sh │ ├── validate-demo.sh │ ├── report-demo.sh │ ├── check-demo.sh │ ├── init-demo.sh │ ├── update-demo.sh │ ├── report-sbom-demo.sh │ ├── freeze-demo.sh │ ├── demo-magic │ │ ├── .dfetch_data.yaml │ │ └── license.txt │ ├── basic-demo.sh │ ├── check-ci-demo.sh │ ├── diff-demo.sh │ ├── strip-setup-from-cast.sh │ └── generate-casts.sh ├── _ext │ ├── sphinxcontrib_asciinema │ │ ├── .dfetch_data.yaml │ │ ├── __init__.py │ │ └── LICENSE │ └── scenario_directive.py ├── static │ ├── uml │ │ ├── styles │ │ │ └── plantuml-c4 │ │ │ │ ├── .dfetch_data.yaml │ │ │ │ └── LICENSE │ │ ├── c1_dfetch_context.puml │ │ ├── check.puml │ │ ├── update.puml │ │ ├── c2_dfetch_containers.puml │ │ ├── c3_dfetch_components_manifest.puml │ │ ├── c3_dfetch_components_project.puml │ │ └── c3_dfetch_components_commands.puml │ └── css │ │ └── custom.css ├── Makefile ├── internal.rst ├── asciicasts │ ├── validate.cast │ ├── environment.cast │ ├── init.cast │ ├── report.cast │ ├── import.cast │ └── freeze.cast ├── make.bat ├── troubleshooting.rst ├── index.rst └── manual.rst ├── features ├── steps │ ├── __init__.py │ └── manifest_steps.py ├── environment.py ├── guard-against-overwriting.feature ├── check-specific-projects.feature ├── handle-invalid-metadata.feature ├── fetch-specific-project.feature ├── fetch-single-file-svn.feature ├── fetch-file-pattern-svn.feature ├── import-from-git.feature ├── fetch-file-pattern-git.feature ├── journey-basic-usage.feature ├── validate-manifest.feature ├── import-from-svn.feature ├── suggest-project-name.feature ├── fetch-checks-destination.feature ├── freeze-projects.feature ├── diff-in-git.feature ├── diff-in-svn.feature ├── fetch-single-file-git.feature ├── patch-after-fetch-git.feature ├── patch-after-fetch-svn.feature ├── fetch-git-repo.feature ├── fetch-with-ignore-svn.feature ├── list-projects.feature └── fetch-with-ignore-git.feature ├── tests ├── __init__.py ├── run_tests.bat ├── test_resources.py ├── manifest_mock.py ├── test_report.py ├── test_project_entry.py ├── test_check.py └── test_cmdline.py ├── .korbitignore ├── dfetch ├── manifest │ ├── __init__.py │ ├── validate.py │ ├── version.py │ └── remote.py ├── util │ ├── __init__.py │ ├── license.py │ ├── cmdline.py │ └── versions.py ├── vcs │ └── __init__.py ├── commands │ ├── __init__.py │ ├── environment.py │ ├── init.py │ ├── validate.py │ ├── common.py │ └── command.py ├── reporting │ ├── check │ │ └── __init__.py │ ├── __init__.py │ ├── reporter.py │ └── stdout_reporter.py ├── __init__.py ├── resources │ ├── __init__.py │ ├── template.yaml │ └── schema.yaml ├── project │ ├── __init__.py │ └── abstract_check_reporter.py ├── log.py └── __main__.py ├── .gitignore ├── script ├── create_docs.bat ├── check_quality.bat ├── release.py ├── build.py ├── dependabot_hook.py └── create_venv.py ├── dfetch.yaml ├── .readthedocs.yml ├── .github ├── dependabot.yml └── workflows │ ├── docs.yml │ ├── dependency-review.yml │ ├── landing-page.yml │ ├── devcontainer.yml │ ├── test.yml │ ├── python-publish.yml │ ├── codeql-analysis.yml │ ├── run.yml │ ├── build.yml │ └── scorecard.yml ├── .devcontainer ├── Dockerfile └── devcontainer.json ├── LICENSE ├── SECURITY.md ├── example └── dfetch.yaml └── action.yml /doc/dfetch-action.md: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /features/steps/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/__init__.py: -------------------------------------------------------------------------------- 1 | """Tests for this project.""" 2 | -------------------------------------------------------------------------------- /.korbitignore: -------------------------------------------------------------------------------- 1 | doc/_ext/sphinxcontrib_asciinema/_static 2 | -------------------------------------------------------------------------------- /dfetch/manifest/__init__.py: -------------------------------------------------------------------------------- 1 | """Manifest related items.""" 2 | -------------------------------------------------------------------------------- /dfetch/util/__init__.py: -------------------------------------------------------------------------------- 1 | """Non domain specific utilities.""" 2 | -------------------------------------------------------------------------------- /dfetch/vcs/__init__.py: -------------------------------------------------------------------------------- 1 | """Version control system wrappers in python.""" 2 | -------------------------------------------------------------------------------- /dfetch/commands/__init__.py: -------------------------------------------------------------------------------- 1 | """Contains all commandline-related commands.""" 2 | -------------------------------------------------------------------------------- /dfetch/reporting/check/__init__.py: -------------------------------------------------------------------------------- 1 | """Reporters used for checking a project's state.""" 2 | -------------------------------------------------------------------------------- /doc/images/dfetch_logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dfetch-org/dfetch/HEAD/doc/images/dfetch_logo.png -------------------------------------------------------------------------------- /doc/landing-page/robots.txt: -------------------------------------------------------------------------------- 1 | User-agent: * 2 | 3 | Sitemap: https://dfetch-org.github.io/sitemap.xml 4 | -------------------------------------------------------------------------------- /doc/images/dfetch_header.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dfetch-org/dfetch/HEAD/doc/images/dfetch_header.png -------------------------------------------------------------------------------- /dfetch/__init__.py: -------------------------------------------------------------------------------- 1 | """Dfetch.""" 2 | 3 | __version__ = "0.10.0" 4 | 5 | DEFAULT_MANIFEST_NAME: str = "dfetch.yaml" 6 | -------------------------------------------------------------------------------- /doc/images/local-change-github.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dfetch-org/dfetch/HEAD/doc/images/local-change-github.png -------------------------------------------------------------------------------- /doc/images/out-of-date-jenkins.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dfetch-org/dfetch/HEAD/doc/images/out-of-date-jenkins.png -------------------------------------------------------------------------------- /doc/images/out-of-date-jenkins2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dfetch-org/dfetch/HEAD/doc/images/out-of-date-jenkins2.png -------------------------------------------------------------------------------- /doc/images/github-actions-result.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dfetch-org/dfetch/HEAD/doc/images/github-actions-result.png -------------------------------------------------------------------------------- /doc/images/gitlab-highlighted-manifest.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dfetch-org/dfetch/HEAD/doc/images/gitlab-highlighted-manifest.png -------------------------------------------------------------------------------- /doc/images/local-change-github-details.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dfetch-org/dfetch/HEAD/doc/images/local-change-github-details.png -------------------------------------------------------------------------------- /doc/images/gitlab-check-pipeline-result.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dfetch-org/dfetch/HEAD/doc/images/gitlab-check-pipeline-result.png -------------------------------------------------------------------------------- /doc/changelog.rst: -------------------------------------------------------------------------------- 1 | :tocdepth: 1 2 | 3 | .. _changes: 4 | 5 | ========= 6 | Changelog 7 | ========= 8 | 9 | .. include:: ../CHANGELOG.rst 10 | -------------------------------------------------------------------------------- /doc/robots.txt: -------------------------------------------------------------------------------- 1 | User-agent: * 2 | 3 | Disallow: / 4 | 5 | Allow: /en/stable 6 | 7 | Allow: /en/latest 8 | 9 | Sitemap: https://dfetch.readthedocs.io/en/latest/sitemap-custom.xml 10 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | __pycache__ 2 | .coverage 3 | .mypy_cache 4 | .pytest_cache 5 | .ruff_cache 6 | .vscode 7 | build 8 | coverage.xml 9 | dfetch.egg-info 10 | dist 11 | doc/_build 12 | doc/landing-page/_build 13 | example/Tests/ 14 | venv* 15 | -------------------------------------------------------------------------------- /doc/manifest.rst: -------------------------------------------------------------------------------- 1 | .. Dfetch documentation master file 2 | 3 | Manifest 4 | ======== 5 | .. automodule:: dfetch.manifest.manifest 6 | 7 | Remotes 8 | ------- 9 | .. automodule:: dfetch.manifest.remote 10 | 11 | Projects 12 | -------- 13 | .. automodule:: dfetch.manifest.project 14 | -------------------------------------------------------------------------------- /doc/generate-casts/environment-demo.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | source ./demo-magic/demo-magic.sh 4 | 5 | PROMPT_TIMEOUT=1 6 | 7 | # Remove any existing manifest 8 | clear 9 | 10 | # Run the command 11 | pe "dfetch environment" 12 | 13 | PROMPT_TIMEOUT=3 14 | wait 15 | 16 | pei "" 17 | -------------------------------------------------------------------------------- /doc/generate-casts/README.md: -------------------------------------------------------------------------------- 1 | # Generate casts 2 | 3 | This folder makes it possible to generate asciinema casts. 4 | The solution was completely inspired by https://stackoverflow.com/a/63080929/1149326 5 | It can only run on linux. 6 | 7 | ## Usage 8 | ```console 9 | ./generate-casts.sh 10 | ``` 11 | -------------------------------------------------------------------------------- /doc/generate-casts/import-demo.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | source ../demo-magic/demo-magic.sh 4 | 5 | PROMPT_TIMEOUT=1 6 | 7 | clear 8 | 9 | # Run the command 10 | pe "ls -l" 11 | pe "cat .gitmodules" 12 | pe "dfetch import" 13 | pe "cat dfetch.yaml" 14 | 15 | PROMPT_TIMEOUT=3 16 | wait 17 | 18 | pei "" 19 | -------------------------------------------------------------------------------- /script/create_docs.bat: -------------------------------------------------------------------------------- 1 | @echo off 2 | setlocal enabledelayedexpansion 3 | 4 | cd %~dp0 5 | 6 | py create_venv.py --extra_requirements "docs" 7 | if not !ERRORLEVEL! == 0 echo "Something went wrong creating the venv." && exit /b !ERRORLEVEL! 8 | 9 | call ..\venv\Scripts\activate.bat 10 | ..\doc\make.bat html 11 | -------------------------------------------------------------------------------- /script/check_quality.bat: -------------------------------------------------------------------------------- 1 | @echo off 2 | setlocal enabledelayedexpansion 3 | 4 | cd %~dp0 5 | 6 | py create_venv.py --extra_requirements "development" 7 | if not !ERRORLEVEL! == 0 echo "Something went wrong creating the venv." && exit /b !ERRORLEVEL! 8 | 9 | call ..\venv\Scripts\activate.bat 10 | pre-commit run 11 | 12 | pause 13 | 14 | exit /b 0 15 | -------------------------------------------------------------------------------- /tests/run_tests.bat: -------------------------------------------------------------------------------- 1 | @echo off 2 | setlocal enabledelayedexpansion 3 | 4 | cd %~dp0.. 5 | 6 | py script/create_venv.py --extra_requirements "test" 7 | if not !ERRORLEVEL! == 0 echo "Something went wrong creating the venv." && exit /b !ERRORLEVEL! 8 | 9 | call .\venv\Scripts\activate.bat 10 | 11 | echo Running tests.... 12 | python -m pytest tests 13 | -------------------------------------------------------------------------------- /doc/generate-casts/validate-demo.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | source ./demo-magic/demo-magic.sh 4 | 5 | PROMPT_TIMEOUT=1 6 | 7 | # Copy example manifest 8 | mkdir validate 9 | pushd validate 10 | 11 | dfetch init 12 | clear 13 | 14 | # Run the command 15 | pe "dfetch validate" 16 | 17 | PROMPT_TIMEOUT=3 18 | wait 19 | 20 | popd 21 | rm -rf validate 22 | -------------------------------------------------------------------------------- /doc/generate-casts/report-demo.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | source ./demo-magic/demo-magic.sh 4 | 5 | PROMPT_TIMEOUT=1 6 | 7 | # Copy example manifest 8 | mkdir report 9 | pushd report 10 | 11 | cp -r ../update/* . 12 | clear 13 | 14 | # Run the command 15 | pe "ls -l" 16 | pe "dfetch report" 17 | 18 | PROMPT_TIMEOUT=3 19 | wait 20 | 21 | popd 22 | rm -rf report 23 | -------------------------------------------------------------------------------- /doc/generate-casts/check-demo.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | source ./demo-magic/demo-magic.sh 4 | 5 | PROMPT_TIMEOUT=1 6 | 7 | # Copy example manifest 8 | mkdir check 9 | pushd check 10 | 11 | dfetch init 12 | clear 13 | 14 | # Run the command 15 | pe "cat dfetch.yaml" 16 | pe "dfetch check" 17 | 18 | PROMPT_TIMEOUT=3 19 | wait 20 | 21 | pei "" 22 | 23 | popd 24 | rm -rf check 25 | -------------------------------------------------------------------------------- /doc/generate-casts/init-demo.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | source ./demo-magic/demo-magic.sh 4 | 5 | PROMPT_TIMEOUT=1 6 | 7 | # Copy example manifest 8 | mkdir init 9 | pushd init 10 | 11 | clear 12 | 13 | # Run the command 14 | pe "ls -l" 15 | pe "dfetch init" 16 | pe "ls -l" 17 | pe "cat dfetch.yaml" 18 | 19 | PROMPT_TIMEOUT=3 20 | wait 21 | 22 | pei "" 23 | 24 | popd 25 | rm -rf init 26 | -------------------------------------------------------------------------------- /doc/generate-casts/update-demo.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | source ./demo-magic/demo-magic.sh 4 | 5 | PROMPT_TIMEOUT=1 6 | 7 | # Copy example manifest 8 | mkdir update 9 | pushd update 10 | 11 | dfetch init 12 | clear 13 | 14 | # Run the command 15 | pe "ls -l" 16 | pe "cat dfetch.yaml" 17 | pe "dfetch update" 18 | pe "ls -l" 19 | pe "dfetch update" 20 | 21 | PROMPT_TIMEOUT=3 22 | wait 23 | 24 | popd 25 | -------------------------------------------------------------------------------- /doc/generate-casts/report-sbom-demo.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | source ./demo-magic/demo-magic.sh 4 | 5 | PROMPT_TIMEOUT=1 6 | 7 | # Copy example manifest 8 | mkdir report_sbom 9 | pushd report_sbom 10 | 11 | cp -r ../update/* . 12 | clear 13 | 14 | # Run the command 15 | pe "ls -l" 16 | pe "dfetch report -t sbom" 17 | pe "cat report.json" 18 | 19 | 20 | PROMPT_TIMEOUT=3 21 | wait 22 | 23 | popd 24 | rm -rf report_sbom 25 | -------------------------------------------------------------------------------- /doc/generate-casts/freeze-demo.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | source ./demo-magic/demo-magic.sh 4 | 5 | PROMPT_TIMEOUT=1 6 | 7 | # Copy example manifest 8 | mkdir freeze 9 | pushd freeze 10 | 11 | cp -r ../update/* . 12 | clear 13 | 14 | # Run the command 15 | pe "cat dfetch.yaml" 16 | pe "dfetch freeze" 17 | pe "cat dfetch.yaml" 18 | pe "ls -l ." 19 | 20 | 21 | PROMPT_TIMEOUT=3 22 | wait 23 | 24 | pei "" 25 | 26 | popd 27 | rm -rf freeze 28 | -------------------------------------------------------------------------------- /doc/generate-casts/demo-magic/.dfetch_data.yaml: -------------------------------------------------------------------------------- 1 | # This is a generated file by dfetch. Don't edit this, but edit the manifest. 2 | # For more info see https://dfetch.rtfd.io/en/latest/getting_started.html 3 | dfetch: 4 | branch: master 5 | hash: 476a29a874df3840ac2bd916e7097b92 6 | last_fetch: 14/10/2025, 19:16:12 7 | patch: '' 8 | remote_url: https://github.com/paxtonhare/demo-magic.git 9 | revision: 2a2f439c26a93286dc2adc6ef2a81755af83f36e 10 | tag: '' 11 | -------------------------------------------------------------------------------- /doc/_ext/sphinxcontrib_asciinema/.dfetch_data.yaml: -------------------------------------------------------------------------------- 1 | # This is a generated file by dfetch. Don't edit this, but edit the manifest. 2 | # For more info see https://dfetch.rtfd.io/en/latest/getting_started.html 3 | dfetch: 4 | branch: master 5 | hash: 7ff48178ffcab6713155878a34566aea 6 | last_fetch: 14/10/2025, 21:25:52 7 | patch: '' 8 | remote_url: https://github.com/divi255/sphinxcontrib.asciinema.git 9 | revision: 28fddf798f8d9e387919e17c3d91d3f4f41b9bd6 10 | tag: '' 11 | -------------------------------------------------------------------------------- /doc/static/uml/styles/plantuml-c4/.dfetch_data.yaml: -------------------------------------------------------------------------------- 1 | # This is a generated file by dfetch. Don't edit this, but edit the manifest. 2 | # For more info see https://dfetch.rtfd.io/en/latest/getting_started.html 3 | dfetch: 4 | branch: master 5 | hash: e9e4271bd4138bae014e8417d1798da2 6 | last_fetch: 14/10/2025, 19:25:14 7 | patch: '' 8 | remote_url: https://github.com/plantuml-stdlib/C4-PlantUML.git 9 | revision: 3b05cdddd3503c193f90e3f70a3a87d972d4897f 10 | tag: '' 11 | -------------------------------------------------------------------------------- /doc/generate-casts/basic-demo.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | source ./demo-magic/demo-magic.sh 4 | 5 | PROMPT_TIMEOUT=1 6 | 7 | # Copy example manifest 8 | mkdir basic 9 | pushd basic 10 | 11 | dfetch init 12 | clear 13 | 14 | # Run the command 15 | pe "ls -l" 16 | pe "cat dfetch.yaml" 17 | pe "dfetch check" 18 | pe "sed -i 's/v3.4/v4.0/g' dfetch.yaml" 19 | pe "cat dfetch.yaml" 20 | pe "dfetch update" 21 | pe "ls -l" 22 | 23 | PROMPT_TIMEOUT=3 24 | wait 25 | 26 | pei "" 27 | 28 | popd 29 | rm -rf basic 30 | -------------------------------------------------------------------------------- /doc/generate-casts/check-ci-demo.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | source ./demo-magic/demo-magic.sh 4 | 5 | PROMPT_TIMEOUT=1 6 | 7 | # Copy example manifest 8 | mkdir check_ci 9 | pushd check_ci 10 | 11 | dfetch init 12 | clear 13 | 14 | # Run the command 15 | pe "cat dfetch.yaml" 16 | pe "dfetch check --jenkins-json jenkins.json --sarif sarif.json" 17 | pe "ls -l ." 18 | pe "cat jenkins.json" 19 | pe "cat sarif.json" 20 | 21 | PROMPT_TIMEOUT=3 22 | wait 23 | 24 | pei "" 25 | 26 | popd 27 | rm -rf check_ci 28 | -------------------------------------------------------------------------------- /doc/generate-casts/diff-demo.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | source ./demo-magic/demo-magic.sh 4 | 5 | PROMPT_TIMEOUT=1 6 | 7 | # Copy example manifest 8 | mkdir diff 9 | pushd diff 10 | 11 | git init 12 | cp -r ../update/* . 13 | git add . 14 | git commit -m "Initial commit" 15 | clear 16 | 17 | # Run the command 18 | pe "ls -l ." 19 | pe "ls -l cpputest/src/README.md" 20 | pe "sed -i 's/github/gitlab/g' cpputest/src/README.md" 21 | pe "dfetch diff cpputest" 22 | pe "cat cpputest.patch" 23 | 24 | 25 | PROMPT_TIMEOUT=3 26 | wait 27 | 28 | pei "" 29 | 30 | popd 31 | rm -rf diff 32 | -------------------------------------------------------------------------------- /dfetch.yaml: -------------------------------------------------------------------------------- 1 | manifest: 2 | version: 0.0 3 | 4 | remotes: 5 | - name: github 6 | url-base: https://github.com/ 7 | 8 | projects: 9 | - name: demo-magic 10 | repo-path: paxtonhare/demo-magic.git 11 | dst: doc/generate-casts/demo-magic 12 | src: '*.sh' 13 | 14 | - name: plantuml-c4 15 | repo-path: plantuml-stdlib/C4-PlantUML.git 16 | dst: doc/static/uml/styles/plantuml-c4 17 | src: '*.puml' 18 | 19 | - name: sphinxcontrib.asciinema 20 | repo-path: divi255/sphinxcontrib.asciinema.git 21 | dst: doc/_ext/sphinxcontrib_asciinema 22 | src: sphinxcontrib/asciinema 23 | -------------------------------------------------------------------------------- /.readthedocs.yml: -------------------------------------------------------------------------------- 1 | # .readthedocs.yml 2 | # Read the Docs configuration file 3 | # See https://docs.readthedocs.io/en/stable/config-file/v2.html for details 4 | 5 | # Required 6 | version: 2 7 | 8 | # Build documentation in the docs/ directory with Sphinx 9 | sphinx: 10 | configuration: doc/conf.py 11 | 12 | formats: 13 | - epub 14 | - htmlzip 15 | 16 | build: 17 | os: ubuntu-22.04 18 | tools: 19 | python: "3.13" 20 | 21 | # Optionally set the version of Python and requirements required to build your docs 22 | python: 23 | install: 24 | - method: pip 25 | path: . 26 | extra_requirements: 27 | - docs 28 | -------------------------------------------------------------------------------- /dfetch/resources/__init__.py: -------------------------------------------------------------------------------- 1 | """Resources needed when dfetch is distributed.""" 2 | 3 | import importlib.resources as importlib_resources 4 | from pathlib import Path 5 | from typing import ContextManager 6 | 7 | from dfetch import resources # pylint: disable=import-self 8 | 9 | 10 | def _resource_path(filename: str) -> ContextManager[Path]: 11 | """Get the path to the resource.""" 12 | return importlib_resources.as_file(importlib_resources.files(resources) / filename) 13 | 14 | 15 | def schema_path() -> ContextManager[Path]: 16 | """Get path to schema.""" 17 | return _resource_path("schema.yaml") 18 | 19 | 20 | TEMPLATE_PATH = _resource_path("template.yaml") 21 | -------------------------------------------------------------------------------- /dfetch/reporting/__init__.py: -------------------------------------------------------------------------------- 1 | """Various reporters for generating reports.""" 2 | 3 | from enum import Enum 4 | 5 | from dfetch.reporting.reporter import Reporter 6 | from dfetch.reporting.sbom_reporter import SbomReporter 7 | from dfetch.reporting.stdout_reporter import StdoutReporter 8 | 9 | 10 | class ReportTypes(Enum): 11 | """Enum giving a name to a type of reporter.""" 12 | 13 | SBOM = "sbom" 14 | STDOUT = "list" 15 | 16 | def __str__(self) -> str: 17 | """Get the string.""" 18 | return self.value 19 | 20 | 21 | REPORTERS: dict[ReportTypes, type[Reporter]] = { 22 | ReportTypes.STDOUT: StdoutReporter, 23 | ReportTypes.SBOM: SbomReporter, 24 | } 25 | -------------------------------------------------------------------------------- /doc/Makefile: -------------------------------------------------------------------------------- 1 | # Minimal makefile for Sphinx documentation 2 | # 3 | 4 | # You can set these variables from the command line. 5 | SPHINXOPTS = 6 | SPHINXBUILD = sphinx-build 7 | SPHINXPROJ = DFetch 8 | SOURCEDIR = . 9 | BUILDDIR = _build 10 | 11 | # Put it first so that "make" without argument is like "make help". 12 | help: 13 | @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) 14 | 15 | .PHONY: help Makefile 16 | 17 | # Catch-all target: route all unknown targets to Sphinx using the new 18 | # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). 19 | %: Makefile 20 | @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) 21 | -------------------------------------------------------------------------------- /.github/dependabot.yml: -------------------------------------------------------------------------------- 1 | version: 2 2 | updates: 3 | - package-ecosystem: "pip" 4 | directory: "/" 5 | schedule: 6 | interval: "daily" 7 | # Raise pull requests for version updates 8 | # to pip against the `develop` branch 9 | target-branch: "main" 10 | # Labels on pull requests for version updates only 11 | labels: 12 | - "dependencies" 13 | - package-ecosystem: "github-actions" 14 | directory: "/" 15 | schedule: 16 | interval: "daily" 17 | - package-ecosystem: "devcontainers" 18 | directory: "/" 19 | schedule: 20 | interval: "daily" 21 | - package-ecosystem: "docker" 22 | directory: "/.devcontainer" 23 | schedule: 24 | interval: "daily" 25 | -------------------------------------------------------------------------------- /doc/static/uml/c1_dfetch_context.puml: -------------------------------------------------------------------------------- 1 | @startuml 2 | 3 | !include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Context.puml 4 | 5 | Person(user, "Developer") 6 | 7 | System(DFetch, "Dfetch") 8 | 9 | Rel(user, DFetch, "Uses") 10 | 11 | System_Boundary(Local, "Local") { 12 | System_Ext(git, "Git") 13 | System_Ext(svn, "Svn") 14 | } 15 | 16 | System_Boundary(Remote, "Remote") { 17 | System_Ext(github, "GitHub") 18 | System_Ext(gitlab, "GitLab") 19 | System_Ext(jenkins, "Jenkins") 20 | } 21 | 22 | Rel(DFetch, git, "Uses") 23 | Rel(DFetch, svn, "Uses") 24 | Rel(DFetch, github, "Reports to") 25 | Rel(DFetch, gitlab, "Reports to") 26 | Rel(DFetch, jenkins, "Reports to") 27 | 28 | 29 | @enduml 30 | -------------------------------------------------------------------------------- /tests/test_resources.py: -------------------------------------------------------------------------------- 1 | """Test the resources.""" 2 | 3 | # mypy: ignore-errors 4 | # flake8: noqa 5 | 6 | import os 7 | 8 | import dfetch.resources 9 | 10 | 11 | def test_schema_path() -> None: 12 | """Test that schema path can be used as context manager.""" 13 | 14 | with dfetch.resources.schema_path() as schema_path: 15 | assert os.path.isfile(schema_path) 16 | 17 | 18 | def test_call_schema_path_twice() -> None: 19 | """Had a lot of problems with calling contextmanager twice.""" 20 | 21 | with dfetch.resources.schema_path() as schema_path: 22 | assert os.path.isfile(schema_path) 23 | 24 | with dfetch.resources.schema_path() as schema_path: 25 | assert os.path.isfile(schema_path) 26 | -------------------------------------------------------------------------------- /doc/landing-page/Makefile: -------------------------------------------------------------------------------- 1 | # Minimal makefile for Sphinx documentation 2 | # 3 | 4 | # You can set these variables from the command line. 5 | SPHINXOPTS = 6 | SPHINXBUILD = sphinx-build 7 | SPHINXPROJ = DFetch 8 | SOURCEDIR = . 9 | BUILDDIR = _build 10 | 11 | # Put it first so that "make" without argument is like "make help". 12 | help: 13 | @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) 14 | 15 | .PHONY: help Makefile 16 | 17 | # Catch-all target: route all unknown targets to Sphinx using the new 18 | # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). 19 | %: Makefile 20 | @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) 21 | touch "$(BUILDDIR)/html/.nojekyll" 22 | -------------------------------------------------------------------------------- /tests/manifest_mock.py: -------------------------------------------------------------------------------- 1 | """Mock for Manifest class.""" 2 | 3 | from unittest.mock import MagicMock, Mock 4 | 5 | from dfetch.manifest.manifest import Manifest 6 | from dfetch.manifest.project import ProjectEntry 7 | 8 | 9 | def mock_manifest(projects, path: str = "/some/path") -> MagicMock: 10 | """Create a manifest mock.""" 11 | 12 | project_mocks = [] 13 | 14 | for project in projects: 15 | mock_project = Mock(spec=ProjectEntry) 16 | mock_project.name = project["name"] 17 | mock_project.destination = "some_dest" 18 | project_mocks += [mock_project] 19 | 20 | mocked_manifest = MagicMock(spec=Manifest, projects=project_mocks, path=path) 21 | mocked_manifest.selected_projects.return_value = project_mocks 22 | return mocked_manifest 23 | -------------------------------------------------------------------------------- /doc/_ext/sphinxcontrib_asciinema/__init__.py: -------------------------------------------------------------------------------- 1 | __copyright__ = 'Copyright (C) 2023' 2 | __license__ = 'MIT' 3 | __version__ = "0.4.2" 4 | 5 | 6 | def setup(app): 7 | from .asciinema import Asciinema, ASCIINemaDirective 8 | from .asciinema import copy_asset_files, _NODE_VISITORS 9 | 10 | app.add_config_value('sphinxcontrib_asciinema_defaults', {}, 'html') 11 | 12 | app.connect('build-finished', copy_asset_files) 13 | app.add_js_file("asciinema-player_3.12.1.js") 14 | app.add_css_file("asciinema-player_3.12.1.css") 15 | 16 | app.add_node(Asciinema, **_NODE_VISITORS) 17 | app.add_directive('asciinema', ASCIINemaDirective) 18 | 19 | return { 20 | 'version': __version__, 21 | 'parallel_read_safe': True, 22 | 'parallel_write_safe': True, 23 | } 24 | -------------------------------------------------------------------------------- /dfetch/resources/template.yaml: -------------------------------------------------------------------------------- 1 | manifest: 2 | version: 0.0 # DFetch Module syntax version 3 | 4 | remotes: # declare common sources in one place 5 | - name: github 6 | url-base: https://github.com/ 7 | 8 | projects: 9 | - name: cpputest 10 | dst: cpputest/src/ # Destination of this project (relative to this file) 11 | repo-path: cpputest/cpputest.git # Use default github remote 12 | tag: v3.4 # tag 13 | 14 | - name: jsmn # without destination, defaults to project name 15 | repo-path: zserge/jsmn.git # only repo-path is enough 16 | -------------------------------------------------------------------------------- /dfetch/project/__init__.py: -------------------------------------------------------------------------------- 1 | """All Project related items.""" 2 | 3 | import dfetch.manifest.project 4 | from dfetch.project.git import GitRepo 5 | from dfetch.project.svn import SvnRepo 6 | from dfetch.project.vcs import VCS 7 | 8 | SUPPORTED_PROJECT_TYPES = [GitRepo, SvnRepo] 9 | 10 | 11 | def make(project_entry: dfetch.manifest.project.ProjectEntry) -> VCS: 12 | """Create a new VCS based on a project from the manifest.""" 13 | for project_type in SUPPORTED_PROJECT_TYPES: 14 | if project_type.NAME == project_entry.vcs: 15 | return project_type(project_entry) 16 | 17 | for project_type in SUPPORTED_PROJECT_TYPES: 18 | project = project_type(project_entry) 19 | 20 | if project.check(): 21 | return project 22 | raise RuntimeError("vcs type unsupported") 23 | -------------------------------------------------------------------------------- /doc/internal.rst: -------------------------------------------------------------------------------- 1 | .. Dfetch documentation internal 2 | 3 | Internal 4 | ======== 5 | *DFetch* is becoming larger everyday. To give it some structure below a description of the internals 6 | 7 | Architecture 8 | ------------ 9 | These diagrams are based on `Simon Brown's C4-model`_. 10 | 11 | .. _`Simon Brown's C4-model` : https://c4model.com/#CoreDiagrams 12 | 13 | C1 - Context 14 | '''''''''''' 15 | .. uml:: /static/uml/c1_dfetch_context.puml 16 | 17 | C2 - Containers 18 | ''''''''''''''' 19 | .. uml:: /static/uml/c2_dfetch_containers.puml 20 | 21 | C3 - Components 22 | ''''''''''''''' 23 | 24 | Commands 25 | ~~~~~~~~ 26 | .. uml:: /static/uml/c3_dfetch_components_commands.puml 27 | 28 | Manifest 29 | ~~~~~~~~ 30 | .. uml:: /static/uml/c3_dfetch_components_manifest.puml 31 | 32 | Project 33 | ~~~~~~~ 34 | .. uml:: /static/uml/c3_dfetch_components_project.puml 35 | -------------------------------------------------------------------------------- /doc/asciicasts/validate.cast: -------------------------------------------------------------------------------- 1 | {"version": 2, "width": 128, "height": 31, "timestamp": 1760470362, "env": {"SHELL": "/bin/sh", "TERM": "xterm-256color"}} 2 | [0.508194, "o", "\u001b[H\u001b[2J\u001b[3J"] 3 | [0.51095, "o", "$ "] 4 | [1.512628, "o", "\u001b"] 5 | [1.692769, "o", "[1"] 6 | [1.782922, "o", "md"] 7 | [1.873157, "o", "fe"] 8 | [1.963282, "o", "t"] 9 | [2.053443, "o", "ch"] 10 | [2.143562, "o", " v"] 11 | [2.233754, "o", "al"] 12 | [2.324059, "o", "id"] 13 | [2.414189, "o", "a"] 14 | [2.594487, "o", "te"] 15 | [2.684623, "o", "\u001b["] 16 | [2.774932, "o", "0m"] 17 | [3.775516, "o", "\r\n"] 18 | [4.206025, "o", "\u001b[1;38;5;4m\u001b[34mDfetch (0.10.0)\u001b[0m\r\n\u001b[0m"] 19 | [4.223288, "o", "\u001b[1;38;5;4m \u001b[32mdfetch.yaml :\u001b[34m valid\u001b[0m\r\n\u001b[0m"] 20 | [4.223567, "o", "\u001b[0m"] 21 | [7.278054, "o", "/workspaces/dfetch/doc/generate-casts\r\n"] 22 | -------------------------------------------------------------------------------- /doc/generate-casts/strip-setup-from-cast.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Check if a file path was provided as the first argument 4 | if [ -z "$1" ]; then 5 | echo "Error: No file path provided" 6 | exit 1 7 | fi 8 | 9 | # Read the asciicast file into a variable 10 | asciicast=$(<"$1") 11 | 12 | # Find the line containing the escape sequence for clearing the screen 13 | line=$(echo "$asciicast" | grep -n '\\u001b\[H\\u001b\[2J\\u001b\[3J' | cut -d: -f1) 14 | 15 | # Check if a matching line was found 16 | if [ -z "$line" ]; then 17 | echo "Warning: No line containing the escape sequence for clearing the screen was found" 18 | else 19 | echo "Will remove lines 2 --> $line" 20 | # Remove all the lines before the line containing the escape sequence 21 | asciicast=$(echo "$asciicast" | sed -n "2,$((line-1))!p") 22 | 23 | # Write the updated asciicast to the file 24 | echo "$asciicast" > "$1" 25 | 26 | fi 27 | -------------------------------------------------------------------------------- /.github/workflows/docs.yml: -------------------------------------------------------------------------------- 1 | name: "Docs" 2 | on: 3 | - pull_request 4 | 5 | permissions: 6 | contents: read 7 | 8 | jobs: 9 | docs: 10 | runs-on: ubuntu-latest 11 | steps: 12 | - name: Harden the runner (Audit all outbound calls) 13 | uses: step-security/harden-runner@20cf305ff2072d973412fa9b1e3a4f227bda3c76 # v2.14.0 14 | with: 15 | egress-policy: audit 16 | 17 | - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v5.0.0 18 | 19 | - name: Install Python 20 | uses: actions/setup-python@83679a892e2d95755f2dac6acb0bfd1e9ac5d548 # v6.1.0 21 | with: 22 | python-version: '3.x' 23 | 24 | - name: Install documentation requirements 25 | run: "pip install .[docs] && pip install sphinx_design" 26 | 27 | - name: Build docs 28 | run: "make -C doc html" 29 | 30 | - name: Build landing-page 31 | run: "make -C doc/landing-page html" 32 | -------------------------------------------------------------------------------- /doc/make.bat: -------------------------------------------------------------------------------- 1 | @ECHO OFF 2 | 3 | pushd %~dp0 4 | 5 | REM Command file for Sphinx documentation 6 | 7 | if "%SPHINXBUILD%" == "" ( 8 | set SPHINXBUILD=sphinx-build 9 | ) 10 | set SOURCEDIR=. 11 | set BUILDDIR=_build 12 | set SPHINXPROJ=DFetch 13 | 14 | if "%1" == "" goto help 15 | 16 | %SPHINXBUILD% >NUL 2>NUL 17 | if errorlevel 9009 ( 18 | echo. 19 | echo.The 'sphinx-build' command was not found. Make sure you have Sphinx 20 | echo.installed, then set the SPHINXBUILD environment variable to point 21 | echo.to the full path of the 'sphinx-build' executable. Alternatively you 22 | echo.may add the Sphinx directory to PATH. 23 | echo. 24 | echo.If you don't have Sphinx installed, grab it from 25 | echo.http://sphinx-doc.org/ 26 | exit /b 1 27 | ) 28 | 29 | %SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% 30 | goto end 31 | 32 | :help 33 | %SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% 34 | 35 | :end 36 | popd 37 | -------------------------------------------------------------------------------- /dfetch/manifest/validate.py: -------------------------------------------------------------------------------- 1 | """Validate manifests.""" 2 | 3 | import logging 4 | 5 | import pykwalify 6 | from pykwalify.core import Core, SchemaError 7 | from yaml.scanner import ScannerError 8 | 9 | import dfetch.resources 10 | 11 | 12 | def validate(path: str) -> None: 13 | """Validate the given manifest.""" 14 | logging.getLogger(pykwalify.__name__).setLevel(logging.CRITICAL) 15 | 16 | with dfetch.resources.schema_path() as schema_path: 17 | try: 18 | validator = Core(source_file=path, schema_files=[str(schema_path)]) 19 | except ScannerError as err: 20 | raise RuntimeError(f"{schema_path} is not a valid YAML file!") from err 21 | 22 | try: 23 | validator.validate(raise_exception=True) 24 | except SchemaError as err: 25 | raise RuntimeError( 26 | str(err.msg) # pyright: ignore[reportAttributeAccessIssue, reportCallIssue] 27 | ) from err 28 | -------------------------------------------------------------------------------- /doc/landing-page/make.bat: -------------------------------------------------------------------------------- 1 | @ECHO OFF 2 | 3 | pushd %~dp0 4 | 5 | REM Command file for Sphinx documentation 6 | 7 | if "%SPHINXBUILD%" == "" ( 8 | set SPHINXBUILD=sphinx-build 9 | ) 10 | set SOURCEDIR=. 11 | set BUILDDIR=_build 12 | set SPHINXPROJ=DFetch 13 | 14 | if "%1" == "" goto help 15 | 16 | %SPHINXBUILD% >NUL 2>NUL 17 | if errorlevel 9009 ( 18 | echo. 19 | echo.The 'sphinx-build' command was not found. Make sure you have Sphinx 20 | echo.installed, then set the SPHINXBUILD environment variable to point 21 | echo.to the full path of the 'sphinx-build' executable. Alternatively you 22 | echo.may add the Sphinx directory to PATH. 23 | echo. 24 | echo.If you don't have Sphinx installed, grab it from 25 | echo.http://sphinx-doc.org/ 26 | exit /b 1 27 | ) 28 | 29 | %SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% 30 | goto end 31 | 32 | :help 33 | %SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% 34 | 35 | :end 36 | popd 37 | -------------------------------------------------------------------------------- /dfetch/manifest/version.py: -------------------------------------------------------------------------------- 1 | """Version of a project.""" 2 | 3 | from typing import Any, NamedTuple 4 | 5 | 6 | class Version(NamedTuple): 7 | """Version of a project. 8 | 9 | In DFetch a version consists of a tag, branch or revision. 10 | A tag has precedence over branches/revisions. 11 | """ 12 | 13 | tag: str = "" 14 | branch: str = "" 15 | revision: str = "" 16 | 17 | def __eq__(self, other: Any) -> bool: 18 | """Check if two versions can be considered as equal.""" 19 | if not other: 20 | return False 21 | 22 | if self.tag or other.tag: 23 | return bool(self.tag == other.tag) 24 | 25 | return bool(self.branch == other.branch and self.revision == other.revision) 26 | 27 | def __repr__(self) -> str: 28 | """Get the string representing this version.""" 29 | if self.tag: 30 | return self.tag 31 | 32 | return " - ".join(filter(None, [self.branch.strip(), self.revision])) 33 | -------------------------------------------------------------------------------- /doc/static/uml/check.puml: -------------------------------------------------------------------------------- 1 | @startuml 2 | start 3 | 4 | skinparam monochrome true 5 | skinparam defaultFontName Frutiger 6 | 7 | :Get version on disk; 8 | 9 | if (Tag given?) then (yes) 10 | :Get all tags; 11 | if (Tag exists?) then (no) 12 | stop 13 | endif 14 | :Parse tags; 15 | :Show versions: 16 | - wanted tag 17 | - on disk tag 18 | - available tag; 19 | else (no) 20 | 21 | if (Branch given?) then (no) 22 | if (Revision enough?) then (yes) 23 | if (Does revision exist?) then (no) 24 | stop 25 | else (yes) 26 | :Show versions: 27 | - wanted revision 28 | - on disk revision; 29 | stop 30 | endif 31 | else (no) 32 | :Use default branch; 33 | endif 34 | else (yes) 35 | endif 36 | 37 | :Get latest revision of branch; 38 | :Show versions: 39 | - wanted revision / branch 40 | - on disk revision / branch 41 | - available revision / branch; 42 | endif 43 | 44 | stop 45 | @enduml 46 | -------------------------------------------------------------------------------- /.devcontainer/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM mcr.microsoft.com/devcontainers/python:3.13-bullseye@sha256:735ecc489de65d36b6ac4e8e5e37287cb30711bac0dd292770bf5347228c438e 2 | 3 | # Install dependencies 4 | # pv is required for asciicasts 5 | RUN apt-get update && apt-get install --no-install-recommends -y \ 6 | ccache=4.2-1 \ 7 | pv=1.6.6-1+b1 \ 8 | patchelf=0.12-1 \ 9 | subversion=1.14.1-3+deb11u2 && \ 10 | rm -rf /var/lib/apt/lists/* 11 | 12 | WORKDIR /workspaces/dfetch 13 | 14 | # Add a non-root user (dev) 15 | RUN useradd -m dev && chown -R dev:dev /workspaces/dfetch 16 | 17 | USER dev 18 | 19 | ENV PATH="/home/dev/.local/bin:${PATH}" 20 | ENV PYTHONPATH="/home/dev/.local/lib/python3.13" 21 | ENV PYTHONUSERBASE="/home/dev/.local" 22 | 23 | COPY --chown=dev:dev . . 24 | 25 | RUN pip install --no-cache-dir --root-user-action=ignore --upgrade pip==25.2 \ 26 | && pip install --no-cache-dir --root-user-action=ignore -e .[development,docs,test,casts,build] \ 27 | && pre-commit install --install-hooks 28 | 29 | # Set bash as the default shell 30 | SHELL ["/bin/bash", "-ec"] 31 | -------------------------------------------------------------------------------- /dfetch/commands/environment.py: -------------------------------------------------------------------------------- 1 | """*Dfetch* can generate output about its working environment.""" 2 | 3 | import argparse 4 | import platform 5 | 6 | import dfetch.commands.command 7 | from dfetch.log import get_logger 8 | from dfetch.project import SUPPORTED_PROJECT_TYPES 9 | 10 | logger = get_logger(__name__) 11 | 12 | 13 | class Environment(dfetch.commands.command.Command): 14 | """Get information about the environment dfetch is working in. 15 | 16 | Generate output that can be used by dfetch developers to investigate issues. 17 | """ 18 | 19 | @staticmethod 20 | def create_menu(subparsers: dfetch.commands.command.SubparserActionType) -> None: 21 | """Add the parser menu for this action.""" 22 | dfetch.commands.command.Command.parser(subparsers, Environment) 23 | 24 | def __call__(self, _: argparse.Namespace) -> None: 25 | """Perform listing the environment.""" 26 | logger.print_info_line("platform", f"{platform.system()} {platform.release()}") 27 | for vcs in SUPPORTED_PROJECT_TYPES: 28 | vcs.list_tool_info() 29 | -------------------------------------------------------------------------------- /dfetch/reporting/reporter.py: -------------------------------------------------------------------------------- 1 | """Abstract reporting interface.""" 2 | 3 | from abc import ABC, abstractmethod 4 | 5 | from dfetch.manifest.manifest import Manifest 6 | from dfetch.manifest.project import ProjectEntry 7 | from dfetch.util.license import License 8 | 9 | 10 | class Reporter(ABC): 11 | """Reporter for generating report.""" 12 | 13 | name: str = "abstract" 14 | 15 | def __init__(self, manifest: Manifest) -> None: 16 | """Create the reporter. 17 | 18 | Args: 19 | manifest (Manifest): The manifest to report on 20 | """ 21 | self._manifest = manifest 22 | 23 | @property 24 | def manifest(self) -> Manifest: 25 | """Get the manifest.""" 26 | return self._manifest 27 | 28 | @abstractmethod 29 | def add_project( 30 | self, 31 | project: ProjectEntry, 32 | licenses: list[License], 33 | version: str, 34 | ) -> None: 35 | """Add a project to the report.""" 36 | 37 | @abstractmethod 38 | def dump_to_file(self, outfile: str) -> bool: 39 | """Do nothing.""" 40 | -------------------------------------------------------------------------------- /features/environment.py: -------------------------------------------------------------------------------- 1 | """General hooks for behave tests.""" 2 | 3 | import os 4 | import tempfile 5 | 6 | from behave import fixture, use_fixture 7 | 8 | from dfetch.util.util import safe_rmtree 9 | 10 | 11 | @fixture 12 | def tmpdir(context): 13 | """Create tempdir during test""" 14 | # -- HINT: @behave.fixture is similar to @contextlib.contextmanager 15 | context.orig_cwd = os.getcwd() 16 | context.tmpdir = tempfile.mkdtemp() 17 | os.chdir(context.tmpdir) 18 | context.remotes_dir_path = os.path.abspath( 19 | os.path.join(os.getcwd(), "some-remote-server") 20 | ) 21 | yield context.tmpdir 22 | # -- CLEANUP-FIXTURE PART: 23 | os.chdir(context.orig_cwd) 24 | safe_rmtree(context.tmpdir) 25 | 26 | 27 | def before_scenario(context, _): 28 | """Hook called before scenario is executed.""" 29 | use_fixture(tmpdir, context) 30 | 31 | 32 | def before_all(context): 33 | """Hook called before first test is run.""" 34 | context.config.log_capture = True 35 | context.config.logging_format = "%(message)s" 36 | 37 | context.remotes_dir = "some-remote-server" 38 | -------------------------------------------------------------------------------- /.github/workflows/dependency-review.yml: -------------------------------------------------------------------------------- 1 | # Dependency Review Action 2 | # 3 | # This Action will scan dependency manifest files that change as part of a Pull Request, 4 | # surfacing known-vulnerable versions of the packages declared or updated in the PR. 5 | # Once installed, if the workflow run is marked as required, 6 | # PRs introducing known-vulnerable packages will be blocked from merging. 7 | # 8 | # Source repository: https://github.com/actions/dependency-review-action 9 | name: 'Dependency Review' 10 | on: [pull_request] 11 | 12 | permissions: 13 | contents: read 14 | 15 | jobs: 16 | dependency-review: 17 | runs-on: ubuntu-latest 18 | steps: 19 | - name: Harden the runner (Audit all outbound calls) 20 | uses: step-security/harden-runner@20cf305ff2072d973412fa9b1e3a4f227bda3c76 # v2.14.0 21 | with: 22 | egress-policy: audit 23 | 24 | - name: 'Checkout Repository' 25 | uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v4.3.0 26 | - name: 'Dependency Review' 27 | uses: actions/dependency-review-action@3c4e3dcb1aa7874d2c16be7d79418e9b7efd6261 # v4.8.2 28 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2020 dfetch-org 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /doc/_ext/sphinxcontrib_asciinema/LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2019 Serhij S. 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /doc/generate-casts/demo-magic/license.txt: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2015-2022 Paxton Hare 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /features/guard-against-overwriting.feature: -------------------------------------------------------------------------------- 1 | Feature: Guard against overwriting 2 | 3 | Accidentally overwriting local changes could lead to introducing regressions. 4 | To make developers aware of overwriting, store hash after an update and compare 5 | this with hashes just before an update. 6 | 7 | Scenario: No update is done when hash is different 8 | Given MyProject with dependency "SomeProject.git" that must be updated 9 | When "SomeProject/README.md" in MyProject is changed locally 10 | And I run "dfetch update" 11 | Then the output shows 12 | """ 13 | Dfetch (0.10.0) 14 | SomeProject : skipped - local changes after last update (use --force to overwrite) 15 | """ 16 | 17 | Scenario: Force flag overrides local changes check 18 | Given MyProject with dependency "SomeProject.git" that must be updated 19 | When "SomeProject/README.md" in MyProject is changed locally 20 | And I run "dfetch update --force" 21 | Then the output shows 22 | """ 23 | Dfetch (0.10.0) 24 | SomeProject : Fetched v2 25 | """ 26 | -------------------------------------------------------------------------------- /doc/static/uml/styles/plantuml-c4/LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | (c) 2018-2020 Ricardo Niepel, 2021-2025 Ricardo Niepel, kirchsth and contributors 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /doc/asciicasts/environment.cast: -------------------------------------------------------------------------------- 1 | {"version": 2, "width": 128, "height": 31, "timestamp": 1760470353, "env": {"SHELL": "/bin/sh", "TERM": "xterm-256color"}} 2 | [0.008844, "o", "$ "] 3 | [1.010498, "o", "\u001b"] 4 | [1.190849, "o", "[1"] 5 | [1.28095, "o", "md"] 6 | [1.371157, "o", "fe"] 7 | [1.461276, "o", "t"] 8 | [1.551428, "o", "ch"] 9 | [1.641627, "o", " e"] 10 | [1.731818, "o", "nv"] 11 | [1.821982, "o", "ir"] 12 | [1.912129, "o", "o"] 13 | [2.092631, "o", "nm"] 14 | [2.182759, "o", "en"] 15 | [2.272891, "o", "t\u001b"] 16 | [2.363053, "o", "[0"] 17 | [2.453221, "o", "m"] 18 | [3.453816, "o", "\r\n"] 19 | [3.907812, "o", "\u001b[1;38;5;4m\u001b[34mDfetch (0.10.0)\u001b[0m\r\n\u001b[0m"] 20 | [3.910405, "o", "\u001b[1;38;5;4m \u001b[32mplatform :\u001b[34m Linux 6.8.0-1030-azure\u001b[0m\r\n\u001b[0m"] 21 | [3.912009, "o", "\u001b[1;38;5;4m \u001b[32mgit :\u001b[34m 2.51.0\u001b[0m\r\n\u001b[0m"] 22 | [4.796021, "o", "\u001b[1;38;5;4m \u001b[32msvn :\u001b[34m 1.14.1 (r1886195)\u001b[0m\r\n"] 23 | [4.796168, "o", "\u001b[0m\u001b[0m"] 24 | [7.867355, "o", "$ "] 25 | [7.868533, "o", "\u001b["] 26 | [8.048866, "o", "1m"] 27 | [8.139019, "o", "\u001b["] 28 | [8.229291, "o", "0m"] 29 | [8.229772, "o", "\r\n"] 30 | -------------------------------------------------------------------------------- /.devcontainer/devcontainer.json: -------------------------------------------------------------------------------- 1 | // For format details, see https://aka.ms/devcontainer.json. For config options, see the 2 | // README at: https://github.com/devcontainers/templates/tree/main/src/python 3 | { 4 | "name": "Python 3", 5 | "build": { 6 | "dockerfile": "Dockerfile", 7 | "context": ".." 8 | }, 9 | "postCreateCommand": "pip install -e .[development,docs,casts]", 10 | "customizations": { 11 | "vscode": { 12 | "extensions": [ 13 | "lextudio.restructuredtext", 14 | "alexkrechik.cucumberautocomplete", 15 | "ms-python.python", 16 | "ms-python.pylint", 17 | "ms-python.isort", 18 | "ms-python.black-formatter", 19 | "ms-python.debugpy", 20 | "mhutchie.git-graph", 21 | "tamasfe.even-better-toml", 22 | "trond-snekvik.simple-rst", 23 | "jebbs.plantuml", 24 | "jimasp.behave-vsc" 25 | ], 26 | "settings": { 27 | "terminal.integrated.profiles.linux": { 28 | "bash": { 29 | "path": "bash", 30 | "icon": "terminal-bash" 31 | } 32 | }, 33 | "terminal.integrated.defaultProfile.linux": "bash" 34 | } 35 | } 36 | }, 37 | "workspaceFolder": "/workspaces/dfetch", 38 | "remoteUser": "dev" 39 | } 40 | -------------------------------------------------------------------------------- /dfetch/commands/init.py: -------------------------------------------------------------------------------- 1 | """*Dfetch* can generate a starting manifest. 2 | 3 | It will be created in the current folder. 4 | """ 5 | 6 | import argparse 7 | import os 8 | import shutil 9 | 10 | import dfetch.commands.command 11 | from dfetch import DEFAULT_MANIFEST_NAME 12 | from dfetch.log import get_logger 13 | from dfetch.resources import TEMPLATE_PATH 14 | 15 | logger = get_logger(__name__) 16 | 17 | 18 | class Init(dfetch.commands.command.Command): 19 | """Initialize a manifest. 20 | 21 | Generate a manifest that can be used as basis for a project. 22 | """ 23 | 24 | @staticmethod 25 | def create_menu(subparsers: dfetch.commands.command.SubparserActionType) -> None: 26 | """Add the parser menu for this action.""" 27 | dfetch.commands.command.Command.parser(subparsers, Init) 28 | 29 | def __call__(self, args: argparse.Namespace) -> None: 30 | """Perform the init.""" 31 | del args # unused 32 | 33 | if os.path.isfile(DEFAULT_MANIFEST_NAME): 34 | logger.warning(f"{DEFAULT_MANIFEST_NAME} already exists!") 35 | return 36 | 37 | with TEMPLATE_PATH as template_path: 38 | dest = shutil.copyfile(template_path, DEFAULT_MANIFEST_NAME) 39 | 40 | logger.info(f"Created {dest}") 41 | -------------------------------------------------------------------------------- /features/check-specific-projects.feature: -------------------------------------------------------------------------------- 1 | Feature: Checking specific projects 2 | 3 | *DFetch* can check specific projects, this is useful when you have a lot 4 | of projects in your manifest. 5 | 6 | Scenario: Single project is checked 7 | Given the manifest 'dfetch.yaml' 8 | """ 9 | manifest: 10 | version: '0.0' 11 | 12 | remotes: 13 | - name: github-com-dfetch-org 14 | url-base: https://github.com/dfetch-org/test-repo 15 | 16 | projects: 17 | - name: ext/test-repo-rev-only 18 | revision: e1fda19a57b873eb8e6ae37780594cbb77b70f1a 19 | dst: ext/test-repo-rev-only 20 | 21 | - name: ext/test-rev-and-branch 22 | revision: 8df389d0524863b85f484f15a91c5f2c40aefda1 23 | branch: main 24 | dst: ext/test-rev-and-branch 25 | 26 | """ 27 | When I run "dfetch check ext/test-rev-and-branch" 28 | Then the output shows 29 | """ 30 | Dfetch (0.10.0) 31 | ext/test-rev-and-branch: wanted (main - 8df389d0524863b85f484f15a91c5f2c40aefda1), available (main - e1fda19a57b873eb8e6ae37780594cbb77b70f1a) 32 | """ 33 | -------------------------------------------------------------------------------- /tests/test_report.py: -------------------------------------------------------------------------------- 1 | """Test the report command.""" 2 | 3 | # mypy: ignore-errors 4 | # flake8: noqa 5 | 6 | import argparse 7 | from unittest.mock import patch 8 | 9 | import pytest 10 | 11 | from dfetch.commands.report import Report, ReportTypes 12 | from tests.manifest_mock import mock_manifest 13 | 14 | DEFAULT_ARGS = argparse.Namespace() 15 | DEFAULT_ARGS.projects = [] 16 | DEFAULT_ARGS.type = ReportTypes.STDOUT 17 | DEFAULT_ARGS.outfile = "" 18 | 19 | 20 | @pytest.mark.parametrize( 21 | "name, projects", 22 | [ 23 | ("empty", []), 24 | ("single_project", [{"name": "my_project"}]), 25 | ("two_projects", [{"name": "first"}, {"name": "second"}]), 26 | ], 27 | ) 28 | def test_report(name, projects): 29 | report = Report() 30 | 31 | with patch("dfetch.manifest.manifest.get_manifest") as mocked_get_manifest: 32 | with patch("dfetch.log.DLogger.print_info_line") as mocked_print_info_line: 33 | mocked_get_manifest.return_value = mock_manifest(projects) 34 | 35 | report(DEFAULT_ARGS) 36 | 37 | if projects: 38 | for project in projects: 39 | mocked_print_info_line.assert_any_call("project", project["name"]) 40 | else: 41 | mocked_print_info_line.assert_not_called() 42 | -------------------------------------------------------------------------------- /features/handle-invalid-metadata.feature: -------------------------------------------------------------------------------- 1 | Feature: Handle invalid metadata files 2 | 3 | *Dfetch* will keep metadata about the fetched project locally to prevent re-fetching unchanged projects 4 | or replacing locally changed projects. Sometimes the metadata file will become incorrect and this should not lead 5 | to unpredictable behavior in *DFetch*. 6 | For instance, the metadata may be invalid 7 | 8 | Scenario: Invalid metadata is ignored 9 | Given the manifest 'dfetch.yaml' 10 | """ 11 | manifest: 12 | version: '0.0' 13 | 14 | projects: 15 | - name: ext/test-repo-tag 16 | url: https://github.com/dfetch-org/test-repo 17 | tag: v1 18 | 19 | """ 20 | And all projects are updated 21 | And the metadata file ".dfetch_data.yaml" of "ext/test-repo-tag" is corrupt 22 | When I run "dfetch update" 23 | Then the output shows 24 | """ 25 | Dfetch (0.10.0) 26 | ext/test-repo-tag/.dfetch_data.yaml is an invalid metadata file, not checking on disk version! 27 | ext/test-repo-tag/.dfetch_data.yaml is an invalid metadata file, not checking local hash! 28 | ext/test-repo-tag : Fetched v1 29 | """ 30 | -------------------------------------------------------------------------------- /.github/workflows/landing-page.yml: -------------------------------------------------------------------------------- 1 | name: Landing-page 2 | 3 | on: 4 | push: 5 | branches: 6 | - main 7 | - feature/simplify-landing-page 8 | 9 | permissions: 10 | contents: read 11 | 12 | jobs: 13 | publish: 14 | runs-on: ubuntu-latest 15 | steps: 16 | - name: Harden the runner (Audit all outbound calls) 17 | uses: step-security/harden-runner@20cf305ff2072d973412fa9b1e3a4f227bda3c76 # v2.14.0 18 | with: 19 | egress-policy: audit 20 | 21 | - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v5.0.0 22 | 23 | - name: Setup Python 24 | uses: actions/setup-python@83679a892e2d95755f2dac6acb0bfd1e9ac5d548 # v6.1.0 25 | with: 26 | python-version: "3.13" 27 | 28 | - name: Install dependencies 29 | run: | 30 | pip install .[docs] 31 | pip install sphinx_design 32 | 33 | - name: Build landing-page 34 | run: | 35 | cd doc/landing-page 36 | make html 37 | - name: Publish 38 | uses: tsunematsu21/actions-publish-gh-pages@c04b531c52b8f9d25c596bc6e6a7ddc116b2f3f8 # v1.0.2 39 | with: 40 | dir: doc/landing-page/_build/html 41 | repo: dfetch-org/dfetch-org.github.io 42 | branch: main 43 | token: ${{ secrets.GH_DFETCH_ORG_DEPLOY }} 44 | -------------------------------------------------------------------------------- /SECURITY.md: -------------------------------------------------------------------------------- 1 | # Security Policy 2 | 3 | ## Reporting Security Issues 4 | 5 | If you discover a security vulnerability in Dfetch, please let us know right away and don't post a public issue. 6 | You can report issues by opening a confidential issue via [GitHub Security Advisories](https://github.com/dfetch/dfetch/security/advisories). See [GitHub's private vulnerability reporting](https://docs.github.com/code-security/security-advisories/guidance-on-reporting-and-writing-information-about-vulnerabilities/privately-reporting-a-security-vulnerability) for more info. 7 | If you have no contact please contact us through the mail listed in the pyproject.toml. 8 | 9 | We appreciate your help in keeping Dfetch safe for everyone. 10 | We aim to respond to security reports within 5 business days. 11 | 12 | ## Supported Versions 13 | 14 | We actively maintain and patch the latest release of Dfetch. 15 | Older versions may not receive security updates. 16 | 17 | ## Disclosure Policy 18 | 19 | We ask that you give us a reasonable amount of time to address security issues before public disclosure. 20 | We will keep you updated on our progress and let you know when the issue has been resolved. 21 | 22 | ## Acknowledgements 23 | 24 | Thank you to everyone who helps keep Dfetch secure! 25 | We’re grateful for responsible disclosures and contributions from the community. 26 | -------------------------------------------------------------------------------- /features/fetch-specific-project.feature: -------------------------------------------------------------------------------- 1 | Feature: Fetching specific dependencies 2 | 3 | *DFetch* can update specific projects, this is useful when you have a lot 4 | of projects in your manifest. 5 | 6 | Scenario: Two Specific projects are fetched 7 | Given the manifest 'dfetch.yaml' 8 | """ 9 | manifest: 10 | version: '0.0' 11 | 12 | remotes: 13 | - name: github-com-dfetch-org 14 | url-base: https://github.com/dfetch-org/test-repo 15 | 16 | projects: 17 | - name: ext/test-repo-rev-only 18 | revision: e1fda19a57b873eb8e6ae37780594cbb77b70f1a 19 | dst: ext/test-repo-rev-only 20 | 21 | - name: ext/test-rev-and-branch 22 | revision: 8df389d0524863b85f484f15a91c5f2c40aefda1 23 | branch: main 24 | dst: ext/test-rev-and-branch 25 | 26 | - name: ext/test-repo-tag-v1 27 | tag: v1 28 | dst: ext/test-repo-tag-v1 29 | 30 | """ 31 | When I run "dfetch update ext/test-repo-tag-v1 ext/test-rev-and-branch" 32 | Then the following projects are fetched 33 | | path | 34 | | ext/test-rev-and-branch | 35 | | ext/test-repo-tag-v1 | 36 | -------------------------------------------------------------------------------- /features/fetch-single-file-svn.feature: -------------------------------------------------------------------------------- 1 | Feature: Fetch single file from svn repo 2 | 3 | Sometimes only one file is enough. *DFetch* makes it possible to specify 4 | only one file from a repository. 5 | 6 | Scenario: A single file is fetched from svn repo 7 | Given the manifest 'dfetch.yaml' in MyProject 8 | """ 9 | manifest: 10 | version: 0.0 11 | projects: 12 | - name: SomeProjectWithAnInterestingFile 13 | dst: MyOwnFileName.txt 14 | url: some-remote-server/SomeProjectWithAnInterestingFile 15 | src: SomeFolder/SomeFile.txt 16 | """ 17 | And a svn-server "SomeProjectWithAnInterestingFile" with the files 18 | | path | 19 | | SomeFolder/SomeFile.txt | 20 | | SomeOtherFolder/SomeOtherFile.txt | 21 | When I run "dfetch update" 22 | Then the output shows 23 | """ 24 | Dfetch (0.10.0) 25 | SomeProjectWithAnInterestingFile: Fetched trunk - 1 26 | """ 27 | And 'MyProject' looks like: 28 | """ 29 | MyProject/ 30 | .dfetch_data-MyOwnFileName.txt.yaml 31 | MyOwnFileName.txt 32 | dfetch.yaml 33 | """ 34 | -------------------------------------------------------------------------------- /tests/test_project_entry.py: -------------------------------------------------------------------------------- 1 | """Test the Version object command.""" 2 | 3 | # mypy: ignore-errors 4 | # flake8: noqa 5 | 6 | import pytest 7 | 8 | from dfetch.manifest.project import ProjectEntry 9 | 10 | 11 | def test_projectentry_name(): 12 | assert ProjectEntry({"name": "SomeProject"}).name == "SomeProject" 13 | 14 | 15 | def test_projectentry_revision(): 16 | assert ProjectEntry({"name": "SomeProject", "revision": "123"}).revision == "123" 17 | 18 | 19 | def test_projectentry_remote(): 20 | assert ( 21 | ProjectEntry({"name": "SomeProject", "remote": "SomeRemote"}).remote 22 | == "SomeRemote" 23 | ) 24 | 25 | 26 | def test_projectentry_source(): 27 | assert ProjectEntry({"name": "SomeProject", "src": "SomePath"}).source == "SomePath" 28 | 29 | 30 | def test_projectentry_vcs(): 31 | assert ProjectEntry({"name": "SomeProject", "vcs": "git"}).vcs == "git" 32 | 33 | 34 | def test_projectentry_patch(): 35 | assert ( 36 | ProjectEntry({"name": "SomeProject", "patch": "diff.patch"}).patch 37 | == "diff.patch" 38 | ) 39 | 40 | 41 | def test_projectentry_as_yaml(): 42 | assert ProjectEntry({"name": "SomeProject"}).as_yaml() == {"name": "SomeProject"} 43 | 44 | 45 | def test_projectentry_as_str(): 46 | assert ( 47 | str(ProjectEntry({"name": "SomeProject"})) 48 | == "SomeProject latest SomeProject" 49 | ) 50 | -------------------------------------------------------------------------------- /dfetch/commands/validate.py: -------------------------------------------------------------------------------- 1 | """Note that you can validate your manifest using :ref:`validate`. 2 | 3 | This will parse your :ref:`Manifest` and check if all fields can be parsed. 4 | 5 | .. scenario-include:: ../features/validate-manifest.feature 6 | 7 | """ 8 | 9 | import argparse 10 | import os 11 | 12 | import dfetch.commands.command 13 | from dfetch.log import get_logger 14 | from dfetch.manifest.manifest import find_manifest 15 | from dfetch.manifest.validate import validate 16 | 17 | logger = get_logger(__name__) 18 | 19 | 20 | class Validate(dfetch.commands.command.Command): 21 | """Validate a manifest. 22 | 23 | The Manifest is validated against a schema. See manifest for requirements. 24 | Note that each time either ``update`` or ``check`` is run, the manifest is also validated. 25 | """ 26 | 27 | @staticmethod 28 | def create_menu(subparsers: dfetch.commands.command.SubparserActionType) -> None: 29 | """Add the parser menu for this action.""" 30 | dfetch.commands.command.Command.parser(subparsers, Validate) 31 | 32 | def __call__(self, args: argparse.Namespace) -> None: 33 | """Perform the validation.""" 34 | del args # unused 35 | 36 | manifest_path = find_manifest() 37 | validate(manifest_path) 38 | manifest_path = os.path.relpath(manifest_path, os.getcwd()) 39 | logger.print_info_line(manifest_path, "valid") 40 | -------------------------------------------------------------------------------- /dfetch/resources/schema.yaml: -------------------------------------------------------------------------------- 1 | # Schema file (schema.yaml) 2 | type: map 3 | mapping: 4 | "manifest": 5 | type: map 6 | required: True 7 | mapping: 8 | "version": { type: number, required: True} 9 | "remotes": 10 | required: False 11 | type: seq 12 | sequence: 13 | - type: map 14 | mapping: 15 | "name": { type: str, required: True, unique: True} 16 | "url-base": { type: str, required: True} 17 | "default": { type: bool } 18 | "projects": 19 | required: True 20 | type: seq 21 | sequence: 22 | - type: map 23 | mapping: 24 | "name": { type: str, required: True, unique: True} 25 | "dst": { type: str, unique: True } 26 | "branch": { type: str } 27 | "tag": { type: str } 28 | "revision": { type: str } 29 | "url": { type: str } 30 | "repo-path": { type: str } 31 | "remote": { type: str } 32 | "patch": { type: str } 33 | "vcs": { type: str, enum: ['git', 'svn'] } 34 | "src": 35 | type: any 36 | "ignore": 37 | required: False 38 | type: seq 39 | sequence: 40 | - type: str 41 | -------------------------------------------------------------------------------- /doc/static/uml/update.puml: -------------------------------------------------------------------------------- 1 | @startuml 2 | start 3 | 4 | skinparam monochrome true 5 | skinparam defaultFontName Frutiger 6 | 7 | partition "Update required" { 8 | 9 | if (Project +\nMetadata on disk?) then (no) 10 | elseif (Force flag given?) then (yes) 11 | elseif (Version in metadata,\nsame as in manifest?) then (yes) 12 | stop 13 | elseif (Hash on disk same\nas in metadata?) then (yes) 14 | else (no) 15 | stop 16 | endif 17 | 18 | ' if (Hash on disk same\nas in metadata?) then (yes) 19 | ' elseif (Force flag given?) then (yes) 20 | ' else 21 | ' stop 22 | ' endif 23 | } 24 | 25 | partition "Prepare for update" { 26 | :Clear target folder; 27 | } 28 | 29 | partition "Perform Update" { 30 | 31 | if (Tag given?) then (yes) 32 | :Use tag; 33 | elseif (Revision only given\nand enough?) then (yes) 34 | :Use exact revision; 35 | elseif (Branch given?) then (yes) 36 | :Use branch 37 | and optionally 38 | revision; 39 | else 40 | :Use default branch 41 | and optionally 42 | revision; 43 | endif 44 | 45 | :Fetch target: 46 | revision/tag/branch; 47 | 48 | if (Patch given?) then (yes) 49 | :Apply patch; 50 | endif 51 | } 52 | 53 | if (Successful?) then (no) 54 | stop 55 | else (yes) 56 | endif 57 | 58 | partition "Update administration" { 59 | :Hash directory; 60 | :Update Metadata; 61 | } 62 | 63 | stop 64 | @enduml 65 | -------------------------------------------------------------------------------- /features/fetch-file-pattern-svn.feature: -------------------------------------------------------------------------------- 1 | Feature: Fetch file pattern from svn repo 2 | 3 | Sometimes all files matching a pattern can be useful. 4 | *DFetch* makes it possible to specify a file pattern from a repository. 5 | 6 | Scenario: A file pattern is fetched from a repo 7 | Given the manifest 'dfetch.yaml' in MyProject 8 | """ 9 | manifest: 10 | version: 0.0 11 | projects: 12 | - name: SomeProjectWithAnInterestingFile 13 | url: some-remote-server/SomeProjectWithAnInterestingFile 14 | src: SomeFolder/SomeSubFolder/*.txt 15 | """ 16 | And a svn-server "SomeProjectWithAnInterestingFile" with the files 17 | | path | 18 | | SomeFolder/SomeSubFolder/SomeFile.txt | 19 | | SomeFolder/SomeSubFolder/OtherFile.txt | 20 | | SomeFolder/SomeSubFolder/SomeFile.md | 21 | When I run "dfetch update" 22 | Then the output shows 23 | """ 24 | Dfetch (0.10.0) 25 | SomeProjectWithAnInterestingFile: Fetched trunk - 1 26 | """ 27 | Then 'MyProject' looks like: 28 | """ 29 | MyProject/ 30 | SomeProjectWithAnInterestingFile/ 31 | .dfetch_data.yaml 32 | OtherFile.txt 33 | SomeFile.txt 34 | dfetch.yaml 35 | """ 36 | -------------------------------------------------------------------------------- /features/import-from-git.feature: -------------------------------------------------------------------------------- 1 | Feature: Importing submodules from an existing git repository 2 | 3 | One alternative to *Dfetch* is git submodules. To make the transition 4 | as easy as possible, a user should be able to generate a manifest that 5 | is filled with the submodules and their pinned versions. 6 | 7 | Scenario: Multiple submodules are imported 8 | Given a git repo with the following submodules 9 | | path | url | revision | 10 | | ext/test-repo1 | https://github.com/dfetch-org/test-repo | e1fda19a57b873eb8e6ae37780594cbb77b70f1a | 11 | | ext/test-repo2 | https://github.com/dfetch-org/test-repo | 8df389d0524863b85f484f15a91c5f2c40aefda1 | 12 | When I run "dfetch import" 13 | Then it should generate the manifest 'dfetch.yaml' 14 | """ 15 | manifest: 16 | version: '0.0' 17 | 18 | remotes: 19 | - name: github-com-dfetch-org 20 | url-base: https://github.com/dfetch-org 21 | 22 | projects: 23 | - name: ext/test-repo1 24 | revision: e1fda19a57b873eb8e6ae37780594cbb77b70f1a 25 | branch: main 26 | repo-path: test-repo 27 | 28 | - name: ext/test-repo2 29 | revision: 8df389d0524863b85f484f15a91c5f2c40aefda1 30 | tag: v1 31 | repo-path: test-repo 32 | 33 | """ 34 | -------------------------------------------------------------------------------- /features/fetch-file-pattern-git.feature: -------------------------------------------------------------------------------- 1 | Feature: Fetch file pattern from git repo 2 | 3 | Sometimes all files matching a pattern can be useful. 4 | *DFetch* makes it possible to specify a file pattern from a repository. 5 | 6 | Scenario: A file pattern is fetched from a repo 7 | Given the manifest 'dfetch.yaml' in MyProject 8 | """ 9 | manifest: 10 | version: 0.0 11 | projects: 12 | - name: SomeProjectWithAnInterestingFile 13 | url: some-remote-server/SomeProjectWithAnInterestingFile.git 14 | src: SomeFolder/SomeSubFolder/*.txt 15 | tag: v1 16 | """ 17 | And a git-repository "SomeProjectWithAnInterestingFile.git" with the files 18 | | path | 19 | | SomeFolder/SomeSubFolder/SomeFile.txt | 20 | | SomeFolder/SomeSubFolder/OtherFile.txt | 21 | | SomeFolder/SomeSubFolder/SomeFile.md | 22 | When I run "dfetch update" 23 | Then the output shows 24 | """ 25 | Dfetch (0.10.0) 26 | SomeProjectWithAnInterestingFile: Fetched v1 27 | """ 28 | Then 'MyProject' looks like: 29 | """ 30 | MyProject/ 31 | SomeProjectWithAnInterestingFile/ 32 | .dfetch_data.yaml 33 | OtherFile.txt 34 | SomeFile.txt 35 | dfetch.yaml 36 | """ 37 | -------------------------------------------------------------------------------- /doc/generate-casts/generate-casts.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Uses relative paths 4 | cd "$(dirname "$0")" 5 | 6 | rm -rf ../asciicasts/* 7 | 8 | asciinema rec --overwrite -c "./basic-demo.sh" ../asciicasts/basic.cast 9 | asciinema rec --overwrite -c "./init-demo.sh" ../asciicasts/init.cast 10 | asciinema rec --overwrite -c "./environment-demo.sh" ../asciicasts/environment.cast 11 | asciinema rec --overwrite -c "./validate-demo.sh" ../asciicasts/validate.cast 12 | asciinema rec --overwrite -c "./check-demo.sh" ../asciicasts/check.cast 13 | asciinema rec --overwrite -c "./check-ci-demo.sh" ../asciicasts/check-ci.cast 14 | asciinema rec --overwrite -c "./update-demo.sh" ../asciicasts/update.cast 15 | 16 | # Depends on artifacts from update 17 | asciinema rec --overwrite -c "./report-demo.sh" ../asciicasts/report.cast 18 | asciinema rec --overwrite -c "./report-sbom-demo.sh" ../asciicasts/sbom.cast 19 | asciinema rec --overwrite -c "./freeze-demo.sh" ../asciicasts/freeze.cast 20 | asciinema rec --overwrite -c "./diff-demo.sh" ../asciicasts/diff.cast 21 | 22 | rm -rf update 23 | 24 | git clone --quiet --depth=1 --branch 3.0.0 https://github.com/jgeudens/ModbusScope.git 2> /dev/null 25 | pushd ModbusScope 26 | git submodule update --quiet --init > /dev/null 27 | asciinema rec --overwrite -c "../import-demo.sh" ../../asciicasts/import.cast 28 | popd 29 | rm -rf ModbusScope 30 | 31 | # Find all files with the .cast extension in the specified directory 32 | files=$(find "../asciicasts" -type f -name '*.cast') 33 | 34 | # Process each file 35 | for file in $files; do 36 | ./strip-setup-from-cast.sh "${file}" 37 | done 38 | -------------------------------------------------------------------------------- /tests/test_check.py: -------------------------------------------------------------------------------- 1 | """Test the check command.""" 2 | 3 | # mypy: ignore-errors 4 | # flake8: noqa 5 | 6 | import argparse 7 | from unittest.mock import patch 8 | 9 | import pytest 10 | 11 | from dfetch.commands.check import Check 12 | from tests.manifest_mock import mock_manifest 13 | 14 | DEFAULT_ARGS = argparse.Namespace( 15 | no_recommendations=False, jenkins_json=None, sarif=None, code_climate=None 16 | ) 17 | DEFAULT_ARGS.projects = [] 18 | 19 | 20 | @pytest.mark.parametrize( 21 | "name, projects", 22 | [ 23 | ("empty", []), 24 | ("single_project", [{"name": "my_project"}]), 25 | ("two_projects", [{"name": "first"}, {"name": "second"}]), 26 | ], 27 | ) 28 | def test_check(name, projects): 29 | check = Check() 30 | 31 | with patch("dfetch.manifest.manifest.get_manifest") as mocked_get_manifest: 32 | with patch( 33 | "dfetch.manifest.manifest.get_childmanifests" 34 | ) as mocked_get_childmanifests: 35 | with patch("dfetch.project.make") as mocked_make: 36 | with patch("os.path.exists"): 37 | with patch("dfetch.commands.check.in_directory"): 38 | with patch("dfetch.commands.check.CheckStdoutReporter"): 39 | mocked_get_manifest.return_value = mock_manifest(projects) 40 | mocked_get_childmanifests.return_value = [] 41 | 42 | check(DEFAULT_ARGS) 43 | 44 | for _ in projects: 45 | mocked_make.return_value.check_for_update.assert_called() 46 | -------------------------------------------------------------------------------- /example/dfetch.yaml: -------------------------------------------------------------------------------- 1 | manifest: 2 | version: 0.0 # DFetch Module syntax 3 | 4 | remotes: # declare common sources in one place 5 | - name: github 6 | url-base: https://github.com/ # Allow git modules 7 | default: true # Set it as default 8 | 9 | - name: sourceforge 10 | url-base: svn://svn.code.sf.net/p/ 11 | 12 | projects: 13 | 14 | - name: cpputest-git-tag 15 | dst: Tests/cpputest-git-tag 16 | url: https://github.com/cpputest/cpputest.git # Use external git directly 17 | tag: v3.4 # revision can also be a tag 18 | 19 | - name: tortoise-svn-branch-rev 20 | dst: Tests/tortoise-svn-branch-rev/ 21 | remote: sourceforge 22 | branch: 1.10.x 23 | revision: '28553' 24 | src: src 25 | vcs: svn 26 | repo-path: tortoisesvn/code 27 | 28 | - name: tortoise-svn-tag 29 | dst: Tests/tortoise-svn-tag/ 30 | remote: sourceforge 31 | tag: version-1.13.1 32 | src: src 33 | vcs: svn 34 | repo-path: tortoisesvn/code 35 | 36 | - name: cpputest-git-src 37 | dst: Tests/cpputest-git-src 38 | repo-path: cpputest/cpputest.git # Use external git directly 39 | src: src 40 | 41 | - name: cpputest-git-rev-only 42 | dst: Tests/cpputest-git-rev-only 43 | revision: d14505cc9191fcf17ccbd92af1c3409eb3969890 44 | repo-path: cpputest/cpputest.git # Use external git directly 45 | -------------------------------------------------------------------------------- /doc/troubleshooting.rst: -------------------------------------------------------------------------------- 1 | .. Dfetch documentation master file 2 | 3 | Troubleshooting 4 | =============== 5 | 6 | Although we do our best, *Dfetch* can always do something unexpected. 7 | A great deal of *Dfetch* functionality is dependent on plain-old command line commands. 8 | 9 | First of all, it is important to see what tools the system has. 10 | This can be seen with :ref:`dfetch environment`. 11 | 12 | Each command *Dfetch* performs and its result can be shown with increasing the verbosity 13 | with the `-v` flag. For example, if an :ref:`dfetch import` is giving strange results, re-run it with:: 14 | 15 | dfetch -v import 16 | 17 | Reporting issues 18 | ---------------- 19 | We are glad to help, if you you are stuck, either create an issue_ on github or contact us through gitter_! 20 | 21 | .. _issue: https://github.com/dfetch-org/dfetch/issues 22 | .. _gitter: https://gitter.im/dfetch-org/community 23 | 24 | Security issues 25 | ---------------- 26 | 27 | If you discover a security vulnerability in *Dfetch*, please let us know right away and don't post a public issue. 28 | You can report issues by opening a confidential issue via `GitHub Security Advisories`_. See 29 | `GitHub's private vulnerability reporting`_ for more info. If you have no contact please contact us through 30 | the mail listed in the pyproject.toml. 31 | 32 | .. _`GitHub Security Advisories`: https://github.com/dfetch/dfetch/security/advisories 33 | .. _`GitHub's private vulnerability reporting`: https://docs.github.com/code-security/security-advisories/guidance-on-reporting-and-writing-information-about-vulnerabilities/privately-reporting-a-security-vulnerability) 34 | -------------------------------------------------------------------------------- /features/journey-basic-usage.feature: -------------------------------------------------------------------------------- 1 | Feature: Basic usage journey 2 | 3 | The main user journey is: 4 | - Creating a manifest. 5 | - Fetching all projects in manifest. 6 | - Checking for new updates 7 | - Updating manifest due to changes. 8 | - Fetching new projects. 9 | 10 | Below scenario is described in the getting started and should at least work. 11 | 12 | Scenario: Basic user journey 13 | 14 | Given the manifest 'dfetch.yaml' 15 | """ 16 | manifest: 17 | version: '0.0' 18 | 19 | projects: 20 | - name: ext/test-repo-tag 21 | tag: v1 22 | url: https://github.com/dfetch-org/test-repo 23 | """ 24 | When I run "dfetch update" 25 | Then the following projects are fetched 26 | | path | 27 | | ext/test-repo-tag | 28 | When I run "dfetch check" 29 | Then the output shows 30 | """ 31 | Dfetch (0.10.0) 32 | ext/test-repo-tag : wanted & current (v1), available (v2.0) 33 | """ 34 | When the manifest 'dfetch.yaml' is changed to 35 | """ 36 | manifest: 37 | version: '0.0' 38 | 39 | projects: 40 | - name: ext/test-repo-tag 41 | url: https://github.com/dfetch-org/test-repo 42 | tag: v2.0 43 | """ 44 | And I run "dfetch update" 45 | Then the following projects are fetched 46 | | path | 47 | | ext/test-repo-tag | 48 | -------------------------------------------------------------------------------- /features/validate-manifest.feature: -------------------------------------------------------------------------------- 1 | Feature: Validate a manifest 2 | 3 | *DFetch* can check if the manifest is valid. 4 | 5 | Scenario: A valid manifest is provided 6 | Given the manifest 'dfetch.yaml' 7 | """ 8 | manifest: 9 | version: '0.0' 10 | 11 | remotes: 12 | - name: github-com-dfetch-org 13 | url-base: https://github.com/dfetch-org/test-repo 14 | 15 | projects: 16 | - name: ext/test-repo-rev-only 17 | revision: e1fda19a57b873eb8e6ae37780594cbb77b70f1a 18 | 19 | """ 20 | When I run "dfetch validate" 21 | Then the output shows 22 | """ 23 | Dfetch (0.10.0) 24 | dfetch.yaml : valid 25 | """ 26 | 27 | Scenario: An invalid manifest is provided 28 | Given the manifest 'dfetch.yaml' 29 | """ 30 | manifest-wrong: 31 | version: '0.0' 32 | 33 | remotes: 34 | - name: github-com-dfetch-org 35 | url-base: https://github.com/dfetch-org/test-repo 36 | 37 | projects: 38 | - name: ext/test-repo-rev-only 39 | revision: e1fda19a57b873eb8e6ae37780594cbb77b70f1a 40 | 41 | """ 42 | When I run "dfetch validate" 43 | Then the output shows 44 | """ 45 | Dfetch (0.10.0) 46 | Schema validation failed: 47 | - Cannot find required key 'manifest'. Path: ''. 48 | - Key 'manifest-wrong' was not defined. Path: ''. 49 | """ 50 | -------------------------------------------------------------------------------- /doc/static/uml/c2_dfetch_containers.puml: -------------------------------------------------------------------------------- 1 | @startuml 2 | 3 | !include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Container.puml 4 | 5 | Person(user, "Developer") 6 | 7 | System_Boundary(DFetch, "Dfetch") { 8 | 9 | Container(contCommands, "Commands", "python", "Single user command to start interacting with dfetch.") 10 | Container(contManifest, "Manifest", "python", "Parsing, editing and finding of manifests.") 11 | Container(contProject, "Project", "python", "Main project that has a manifest.") 12 | Container(contVcs, "Vcs", "python", "Abstraction of various Version Control Systems.") 13 | Container(contReporting, "Reporting", "python", "Output formatters for various reporting formats.") 14 | 15 | Rel(contCommands, contManifest, "Uses") 16 | Rel(contCommands, contReporting, "Uses") 17 | Rel(contCommands, contProject, "Uses") 18 | Rel(contCommands, contVcs, "Uses") 19 | Rel_U(contReporting, contProject, "Implements") 20 | Rel_R(contProject, contManifest, "Has") 21 | Rel_U(contReporting, contManifest, "Uses") 22 | Rel(contProject, contVcs, "Uses") 23 | } 24 | 25 | System_Boundary(Local, "Local") { 26 | System_Ext(git, "Git") 27 | System_Ext(svn, "Svn") 28 | } 29 | 30 | System_Boundary(Remote, "Remote") { 31 | System_Ext(github, "GitHub") 32 | System_Ext(gitlab, "GitLab") 33 | System_Ext(jenkins, "Jenkins") 34 | } 35 | 36 | Rel(contVcs, git, "Uses") 37 | Rel(contVcs, svn, "Uses") 38 | 39 | Rel(contReporting, github, "Reports to") 40 | Rel(contReporting, gitlab, "Reports to") 41 | Rel(contReporting, jenkins, "Reports to") 42 | 43 | 44 | Rel(user, contCommands, "Uses") 45 | 46 | @enduml 47 | -------------------------------------------------------------------------------- /action.yml: -------------------------------------------------------------------------------- 1 | name: 'Dfetch Check' 2 | description: 'Run dfetch check and upload SARIF results.' 3 | author: 'dfetch-org' 4 | branding: 5 | icon: 'check-circle' 6 | color: 'blue' 7 | 8 | inputs: 9 | working-directory: 10 | description: 'Directory to run dfetch in (default: project root)' 11 | required: false 12 | default: '.' 13 | 14 | outputs: 15 | sarif-path: 16 | description: 'Path to the generated SARIF file.' 17 | value: sarif.json 18 | 19 | runs: 20 | using: 'composite' 21 | steps: 22 | - name: Checkout repository 23 | uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v5.0.0 24 | - name: Setup Python 25 | uses: actions/setup-python@83679a892e2d95755f2dac6acb0bfd1e9ac5d548 # v6.1.0 26 | with: 27 | python-version: '3.13' 28 | 29 | # Install dfetch from main if NOT running on a branch in the dfetch repo 30 | - name: Install dfetch from GitHub 31 | if: ${{ github.repository != 'dfetch-org/dfetch' || github.ref_name == 'main' }} 32 | run: pip install git+https://github.com/dfetch-org/dfetch.git@main#egg=dfetch 33 | shell: bash 34 | 35 | # Install dfetch locally if running inside the dfetch repo 36 | - name: Install dfetch locally 37 | if: ${{ github.repository == 'dfetch-org/dfetch' }} 38 | run: pip install . 39 | shell: bash 40 | 41 | - name: Run dfetch check (SARIF) 42 | run: dfetch check --sarif sarif.json 43 | shell: bash 44 | working-directory: ${{ inputs.working-directory }} 45 | - name: Upload SARIF file 46 | uses: github/codeql-action/upload-sarif@17783bfb99b07f70fae080b654aed0c514057477 # v3.30.7 47 | with: 48 | sarif_file: sarif.json 49 | -------------------------------------------------------------------------------- /tests/test_cmdline.py: -------------------------------------------------------------------------------- 1 | """Test the cmdline.""" 2 | 3 | # mypy: ignore-errors 4 | # flake8: noqa 5 | 6 | import os 7 | import subprocess 8 | from subprocess import CalledProcessError, CompletedProcess 9 | from unittest.mock import MagicMock, Mock, patch 10 | 11 | import pytest 12 | 13 | from dfetch.util.cmdline import SubprocessCommandError, run_on_cmdline 14 | 15 | LS_CMD = "ls ." 16 | LS_OK_RESULT = CompletedProcess( 17 | returncode=0, stdout="myfile".encode(), stderr="".encode(), args=LS_CMD 18 | ) 19 | LS_NON_ZERO_RESULT = CalledProcessError( 20 | returncode=1, output="myfile".encode(), stderr="".encode(), cmd=LS_CMD 21 | ) 22 | LS_NOK_RESULT = CalledProcessError( 23 | returncode=0, output="myfile".encode(), stderr="".encode(), cmd=LS_CMD 24 | ) 25 | MISSING_CMD_RESULT = FileNotFoundError() 26 | 27 | 28 | @pytest.mark.parametrize( 29 | "name, cmd, cmd_result, expectation", 30 | [ 31 | ("cmd succeeds", LS_CMD, [LS_OK_RESULT], LS_OK_RESULT), 32 | ("cmd non-zero return", LS_CMD, [LS_NON_ZERO_RESULT], SubprocessCommandError), 33 | ("cmd raises", LS_CMD, [LS_NOK_RESULT], SubprocessCommandError), 34 | ("cmd missing", LS_CMD, [MISSING_CMD_RESULT], RuntimeError), 35 | ], 36 | ) 37 | def test_run_on_cmdline(name, cmd, cmd_result, expectation): 38 | with patch("dfetch.util.cmdline.subprocess.run") as subprocess_mock: 39 | subprocess_mock.side_effect = cmd_result 40 | logger_mock = MagicMock() 41 | 42 | if isinstance(expectation, CompletedProcess): 43 | assert expectation == run_on_cmdline(logger_mock, cmd) 44 | else: 45 | with pytest.raises(expectation): 46 | run_on_cmdline(logger_mock, cmd) 47 | -------------------------------------------------------------------------------- /features/import-from-svn.feature: -------------------------------------------------------------------------------- 1 | Feature: Importing externals from an existing svn repository 2 | 3 | One alternative to *Dfetch* is svn externals. To make the transition 4 | as easy as possible, a user should be able to generate a manifest that 5 | is filled with the externals and their pinned versions. 6 | 7 | Scenario: Multiple externals are imported 8 | Given a svn repo with the following externals 9 | | path | url | revision | 10 | | ext/test-repo1 | https://github.com/dfetch-org/test-repo/trunk | 1 | 11 | | ext/test-repo2 | https://github.com/dfetch-org/test-repo/tags/v2.0 | | 12 | | ext/test-repo3 | https://github.com/dfetch-org/test-repo | | 13 | When I run "dfetch import" 14 | Then it should generate the manifest 'dfetch.yaml' 15 | """ 16 | manifest: 17 | version: '0.0' 18 | 19 | remotes: 20 | - name: github-com-dfetch-org 21 | url-base: https://github.com/dfetch-org 22 | 23 | projects: 24 | - name: ext/test-repo1 25 | revision: '1' 26 | dst: ./ext/test-repo1 27 | repo-path: test-repo 28 | 29 | - name: ext/test-repo2 30 | dst: ./ext/test-repo2 31 | tag: v2.0 32 | repo-path: test-repo 33 | 34 | - name: ext/test-repo3 35 | dst: ./ext/test-repo3 36 | branch: ' ' 37 | repo-path: test-repo 38 | 39 | """ 40 | -------------------------------------------------------------------------------- /.github/workflows/devcontainer.yml: -------------------------------------------------------------------------------- 1 | name: DevContainer 2 | 3 | on: 4 | push: 5 | branches: [main, dev] 6 | pull_request: 7 | branches: [main, dev] 8 | 9 | permissions: 10 | contents: read 11 | 12 | jobs: 13 | devcontainer: 14 | name: DevContainer Build & Test 15 | runs-on: ubuntu-latest 16 | 17 | steps: 18 | - name: Harden the runner (Audit all outbound calls) 19 | uses: step-security/harden-runner@20cf305ff2072d973412fa9b1e3a4f227bda3c76 # v2.14.0 20 | with: 21 | egress-policy: audit 22 | 23 | - name: Checkout repository 24 | uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v5.0.0 25 | 26 | - name: Cache Docker layers 27 | uses: actions/cache@9255dc7a253b0ccc959486e2bca901246202afeb # v5.0.1 28 | with: 29 | path: /tmp/.buildx-cache 30 | key: devcontainer-${{ runner.os }}-${{ github.sha }} 31 | restore-keys: | 32 | devcontainer-${{ runner.os }}- 33 | 34 | - name: Set up Docker Buildx 35 | uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v3.11.1 36 | 37 | - name: Build DevContainer image 38 | uses: devcontainers/ci@8bf61b26e9c3a98f69cb6ce2f88d24ff59b785c6 # v0.3.1900000417 39 | with: 40 | runCmd: | 41 | echo "Installing test dependencies..." 42 | pip install -e .[development,docs,casts] 43 | 44 | echo "Running pre-commit checks..." 45 | pre-commit run --all-files 46 | 47 | echo "Running unit tests..." 48 | python -m pytest tests 49 | 50 | echo "Building documentation..." 51 | make -C doc html 52 | make -C doc/landing-page html 53 | -------------------------------------------------------------------------------- /doc/static/uml/c3_dfetch_components_manifest.puml: -------------------------------------------------------------------------------- 1 | @startuml 2 | 3 | !include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Component.puml 4 | 5 | Person(user, "Developer") 6 | 7 | System_Boundary(DFetch, "Dfetch") { 8 | 9 | Container(contCommands, "Commands", "python", "Parsing, editing and finding of manifests.") 10 | 11 | Boundary(DFetchManifest, "Manifest") { 12 | Component(compManifest, "Manifest", "python", "Main configuration file describing all projects.") 13 | Component(compProject, "Project", "python", "A single project requirement with optionally specific version") 14 | Component(compRemote, "Remote", "python", "A remote source that contains one or more projects.") 15 | Component(compValidate, "Validate", "python", "Validate a manifest.") 16 | Component(compVersion, "Version", "python", "Check and compare versions.") 17 | 18 | Rel(compManifest, compProject, "Uses") 19 | Rel(compProject, compVersion, "Uses") 20 | Rel_L(compProject, compRemote, "Uses") 21 | Rel(compManifest, compValidate, "Uses") 22 | Rel(compManifest, compRemote, "Uses") 23 | } 24 | 25 | Container(contProject, "Project", "python", "Main project that has a manifest.") 26 | Container(contVcs, "Vcs", "python", "Abstraction of various Version Control Systems.") 27 | Container(contReporting, "Reporting", "python", "Output formatters for various reporting formats.") 28 | 29 | Rel(contCommands, compManifest, "Uses") 30 | Rel(contReporting, contProject, "Extends") 31 | Rel_R(contProject, compManifest, "Has") 32 | Rel(contReporting, compManifest, "Uses") 33 | Rel(contProject, contVcs, "Uses") 34 | } 35 | 36 | Rel(user, contCommands, "Uses") 37 | 38 | @enduml 39 | -------------------------------------------------------------------------------- /features/suggest-project-name.feature: -------------------------------------------------------------------------------- 1 | Feature: Suggest a project name 2 | 3 | Users sometimes mistype project names, this lead to blaming *DFetch* of wrong behavior. 4 | To help the user of *DFetch*, if a user specifies a project name, that can't be found, suggest 5 | the closest. 6 | 7 | Background: 8 | Given the manifest 'dfetch.yaml' with the projects: 9 | | name | 10 | | project with space | 11 | | project-with-l | 12 | | Project-With-Capital | 13 | 14 | Scenario: Name with space 15 | When I run "dfetch check project with space" 16 | Then the output shows 17 | """ 18 | Dfetch (0.10.0) 19 | Not all projects found! "project", "with", "space" 20 | This manifest contains: "project with space", "project-with-l", "Project-With-Capital" 21 | Did you mean: "project with space"? 22 | """ 23 | 24 | Scenario: Name with one token different 25 | When I run "dfetch check project-with-1" 26 | Then the output shows 27 | """ 28 | Dfetch (0.10.0) 29 | Not all projects found! "project-with-1" 30 | This manifest contains: "project with space", "project-with-l", "Project-With-Capital" 31 | Did you mean: "project-with-l"? 32 | """ 33 | 34 | Scenario: Multiple wrong 35 | When I run "dfetch check project-with-1 project-with-space Project-With-Capital" 36 | Then the output shows 37 | """ 38 | Dfetch (0.10.0) 39 | Not all projects found! "project-with-1", "project-with-space" 40 | This manifest contains: "project with space", "project-with-l", "Project-With-Capital" 41 | Did you mean: "project with space" and "project-with-l"? 42 | """ 43 | -------------------------------------------------------------------------------- /features/fetch-checks-destination.feature: -------------------------------------------------------------------------------- 1 | Feature: Fetch checks destinations 2 | 3 | Destinations marked with the 'dst:' attribute can be misused and lead to 4 | issues in the existing project. For instance path traversal or overwriting 5 | of other fetched projects. *DFetch* should do some sanity checks on the destinations. 6 | 7 | Scenario: No fetch is done directly into manifest folder 8 | Given the manifest 'dfetch.yaml' 9 | """ 10 | manifest: 11 | version: '0.0' 12 | 13 | projects: 14 | - name: ext/test-repo-tag 15 | url: https://github.com/dfetch-org/test-repo 16 | tag: v1 17 | dst: . 18 | 19 | """ 20 | When I run "dfetch update" 21 | Then the output shows 22 | """ 23 | Dfetch (0.10.0) 24 | ext/test-repo-tag : Skipping, path "." is not allowed as destination. 25 | Destination must be in a valid subfolder. "." is not valid! 26 | """ 27 | 28 | Scenario: Path traversal is not allowed 29 | Given the manifest 'dfetch.yaml' 30 | """ 31 | manifest: 32 | version: '0.0' 33 | 34 | projects: 35 | - name: ext/test-repo-tag 36 | url: https://github.com/dfetch-org/test-repo 37 | tag: v1 38 | dst: ../../some-higher-folder 39 | 40 | """ 41 | When I run "dfetch update" 42 | Then the output shows 43 | """ 44 | Dfetch (0.10.0) 45 | ext/test-repo-tag : Skipping, path "../../some-higher-folder" is outside manifest directory tree. 46 | Destination must be in the manifests folder or a subfolder. "../../some-higher-folder" is outside this tree! 47 | """ 48 | -------------------------------------------------------------------------------- /dfetch/reporting/stdout_reporter.py: -------------------------------------------------------------------------------- 1 | """*Dfetch* can generate an report on stdout. 2 | 3 | Depending on the state of the projects it will show as much information 4 | from the manifest or the metadata (``.dfetch_data.yaml``). 5 | """ 6 | 7 | from dfetch.log import get_logger 8 | from dfetch.manifest.project import ProjectEntry 9 | from dfetch.project.metadata import Metadata 10 | from dfetch.reporting.reporter import Reporter 11 | from dfetch.util.license import License 12 | 13 | logger = get_logger(__name__) 14 | 15 | 16 | class StdoutReporter(Reporter): 17 | """Reporter for generating report on stdout.""" 18 | 19 | name = "stdout" 20 | 21 | def add_project( 22 | self, 23 | project: ProjectEntry, 24 | licenses: list[License], 25 | version: str, 26 | ) -> None: 27 | """Add a project to the report.""" 28 | del version 29 | logger.print_info_field("project", project.name) 30 | logger.print_info_field(" remote", project.remote) 31 | try: 32 | metadata = Metadata.from_file(Metadata.from_project_entry(project).path) 33 | logger.print_info_field(" remote url", metadata.remote_url) 34 | logger.print_info_field(" branch", metadata.branch) 35 | logger.print_info_field(" tag", metadata.tag) 36 | logger.print_info_field(" last fetch", str(metadata.last_fetch)) 37 | logger.print_info_field(" revision", metadata.revision) 38 | logger.print_info_field(" patch", metadata.patch) 39 | logger.print_info_field( 40 | " licenses", ",".join(license.name for license in licenses) 41 | ) 42 | 43 | except FileNotFoundError: 44 | logger.print_info_field(" last fetch", "never") 45 | 46 | def dump_to_file(self, outfile: str) -> bool: 47 | """Do nothing.""" 48 | del outfile 49 | return False 50 | -------------------------------------------------------------------------------- /features/steps/manifest_steps.py: -------------------------------------------------------------------------------- 1 | """Steps for features tests.""" 2 | 3 | # pylint: disable=function-redefined, missing-function-docstring, import-error, not-callable 4 | # pyright: reportRedeclaration=false, reportAttributeAccessIssue=false, reportCallIssue=false 5 | 6 | import os 7 | import pathlib 8 | from typing import Optional 9 | 10 | from behave import given, then, when # pylint: disable=no-name-in-module 11 | 12 | from features.steps.generic_steps import check_file, generate_file, remote_server_path 13 | 14 | 15 | def generate_manifest( 16 | context, name="dfetch.yaml", contents: Optional[str] = None, path=None 17 | ): 18 | contents = contents or context.text 19 | manifest = contents.replace( 20 | "url: some-remote-server", f"url: file:///{remote_server_path(context)}" 21 | ) 22 | generate_file(os.path.join(path or os.getcwd(), name), manifest) 23 | 24 | 25 | @given("the manifest '{name}' in {path}") 26 | @given("the manifest '{name}'") 27 | @when("the manifest '{name}' is changed to") 28 | @when("the manifest '{name}' in {path} is changed to") 29 | def step_impl(context, name, path=None): 30 | if path: 31 | pathlib.Path(path).mkdir(parents=True, exist_ok=True) 32 | 33 | generate_manifest(context, name, contents=context.text, path=path) 34 | 35 | 36 | @then("the manifest '{name}' is replaced with") 37 | @then("it should generate the manifest '{name}'") 38 | def step_impl(context, name): 39 | """Check a manifest.""" 40 | check_file(name, context.text) 41 | 42 | 43 | @given("the manifest '{name}' with the projects:") 44 | def step_impl(context, name): 45 | projects = "\n".join(f" - name: {row['name']}" for row in context.table) 46 | manifest = f"""manifest: 47 | version: '0.0' 48 | remotes: 49 | - name: github-com-dfetch-org 50 | url-base: https://github.com/dfetch-org/test-repo 51 | 52 | projects: 53 | {projects} 54 | """ 55 | generate_file(os.path.join(os.getcwd(), name), manifest) 56 | -------------------------------------------------------------------------------- /doc/static/uml/c3_dfetch_components_project.puml: -------------------------------------------------------------------------------- 1 | @startuml 2 | 3 | !include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Component.puml 4 | 5 | Person(user, "Developer") 6 | 7 | System_Boundary(DFetch, "Dfetch") { 8 | 9 | Container(contCommands, "Commands", "python", "Parsing, editing and finding of manifests.") 10 | 11 | Container(contManifest, "Manifest", "python", "Parsing, editing and finding of manifests.") 12 | Boundary(DfetchProject, "Project", "python", "Main project that has a manifest.") { 13 | 14 | Component(compAbstractCheckReporter, "AbstractCheckReporter", "python", "Abstract interface for generating a check report.") 15 | Component(compGit, "Git", "python", "A remote source project based on git.") 16 | Component(compMetadata, "Metadata", "python", "A file containing metadata about a project.") 17 | Component(compSvn, "Svn", "python", "A remote source project based on svn.") 18 | Component(compVcs, "Vcs", "python", "An abstract remote version control system.") 19 | 20 | Rel_U(compGit, compVcs, "Implements") 21 | Rel_U(compSvn, compVcs, "Implements") 22 | Rel(compVcs, compAbstractCheckReporter, "Uses") 23 | Rel_L(compVcs, compMetadata, "Uses") 24 | } 25 | 26 | 27 | Container(contVcs, "Vcs", "python", "Abstraction of various Version Control Systems.") 28 | Container(contReporting, "Reporting", "python", "Output formatters for various reporting formats.") 29 | 30 | Rel(contCommands, compVcs, "Uses") 31 | Rel(contCommands, compGit, "Uses") 32 | Rel(contCommands, compSvn, "Uses") 33 | Rel(contReporting, compAbstractCheckReporter, "Implements") 34 | Rel_R(contReporting, compMetadata, "Uses") 35 | Rel_R(compMetadata, contManifest, "Has") 36 | Rel(compVcs, contManifest, "Has") 37 | Rel(contReporting, contManifest, "Uses") 38 | Rel(compGit, contVcs, "Uses") 39 | Rel(compSvn, contVcs, "Uses") 40 | } 41 | 42 | Rel(user, contCommands, "Uses") 43 | 44 | @enduml 45 | -------------------------------------------------------------------------------- /doc/static/css/custom.css: -------------------------------------------------------------------------------- 1 | @import url("https://fonts.googleapis.com/css?family=Roboto:100,300,300i,400,500,700,900"); 2 | 3 | .sphinxsidebar .caption-text { 4 | font-size: 130%; 5 | } 6 | 7 | .logo { 8 | width: 100%; 9 | } 10 | 11 | .sphinxsidebarwrapper .internal, 12 | .sphinxsidebarwrapper .external { 13 | font-weight: 300; 14 | } 15 | 16 | body { 17 | font-family: Roboto; 18 | } 19 | 20 | h1, 21 | h2, 22 | h3, 23 | h4, 24 | h5, 25 | h6 { 26 | font-family: Roboto !important; 27 | font-weight: 300 !important; 28 | } 29 | 30 | .toctree-l1 { 31 | padding-bottom: 0.5em; 32 | } 33 | 34 | .body p, 35 | .body dd, 36 | .body li { 37 | font-family: Roboto; 38 | font-size: 1em; 39 | font-weight: 300; 40 | line-height: 2; 41 | text-align: justify; 42 | } 43 | 44 | .search { 45 | margin-bottom: 1em; 46 | } 47 | 48 | .sphinxsidebarwrapper ul { 49 | margin-bottom: 1em; 50 | } 51 | 52 | .sphinxsidebar input[type="submit"] { 53 | font-family: "Roboto", serif; 54 | } 55 | 56 | .caption { 57 | margin-top: 1em; 58 | } 59 | 60 | .caption-text { 61 | font-weight: 500; 62 | } 63 | 64 | .sphinxsidebarwrapper .logo-name { 65 | display: none; 66 | } 67 | 68 | .sphinxsidebarwrapper .logo { 69 | margin-bottom: 2em; 70 | } 71 | 72 | img { 73 | height: auto; 74 | max-width: 100%; 75 | } 76 | 77 | div.admonition { 78 | border: none; 79 | border-radius: 0; 80 | box-shadow: 0 2px 2px 0 rgba(0, 0, 0, 0.14), 0 1px 5px 0 rgba(0, 0, 0, 0.12), 81 | 0 3px 1px -2px rgba(0, 0, 0, 0.2); 82 | font-family: Roboto; 83 | padding: 2em; 84 | position: relative; 85 | transition: box-shadow 0.25s; 86 | } 87 | 88 | div.admonition p.admonition-title { 89 | font-family: "Roboto", serif; 90 | font-weight: 300; 91 | margin: 0 0 5px 0; 92 | } 93 | 94 | div.admonition p { 95 | font-weight: 300; 96 | margin-top: 1em; 97 | } 98 | 99 | .admonition a { 100 | font-weight: 300; 101 | } 102 | 103 | strong { 104 | font-weight: 400; 105 | } 106 | 107 | .pre { 108 | padding-left: 0.2em; 109 | padding-right: 0.2em; 110 | } 111 | -------------------------------------------------------------------------------- /doc/landing-page/static/css/custom.css: -------------------------------------------------------------------------------- 1 | @import url("https://fonts.googleapis.com/css?family=Roboto:100,300,300i,400,500,700,900"); 2 | 3 | .sphinxsidebar .caption-text { 4 | font-size: 130%; 5 | } 6 | 7 | .logo { 8 | width: 100%; 9 | } 10 | 11 | .sphinxsidebarwrapper .internal, 12 | .sphinxsidebarwrapper .external { 13 | font-weight: 300; 14 | } 15 | 16 | body { 17 | font-family: Roboto; 18 | } 19 | 20 | h1, 21 | h2, 22 | h3, 23 | h4, 24 | h5, 25 | h6 { 26 | font-family: Roboto !important; 27 | font-weight: 300 !important; 28 | } 29 | 30 | .toctree-l1 { 31 | padding-bottom: 0.5em; 32 | } 33 | 34 | .body p, 35 | .body dd, 36 | .body li { 37 | font-family: Roboto; 38 | font-size: 1em; 39 | font-weight: 300; 40 | line-height: 2; 41 | text-align: justify; 42 | } 43 | 44 | .search { 45 | margin-bottom: 1em; 46 | } 47 | 48 | .sphinxsidebarwrapper ul { 49 | margin-bottom: 1em; 50 | } 51 | 52 | .sphinxsidebar input[type="submit"] { 53 | font-family: "Roboto", serif; 54 | } 55 | 56 | .caption { 57 | margin-top: 1em; 58 | } 59 | 60 | .caption-text { 61 | font-weight: 500; 62 | } 63 | 64 | .sphinxsidebarwrapper .logo-name { 65 | display: none; 66 | } 67 | 68 | .sphinxsidebarwrapper .logo { 69 | margin-bottom: 2em; 70 | } 71 | 72 | img { 73 | height: auto; 74 | max-width: 100%; 75 | } 76 | 77 | div.admonition { 78 | border: none; 79 | border-radius: 0; 80 | box-shadow: 0 2px 2px 0 rgba(0, 0, 0, 0.14), 0 1px 5px 0 rgba(0, 0, 0, 0.12), 81 | 0 3px 1px -2px rgba(0, 0, 0, 0.2); 82 | font-family: Roboto; 83 | padding: 2em; 84 | position: relative; 85 | transition: box-shadow 0.25s; 86 | } 87 | 88 | div.admonition p.admonition-title { 89 | font-family: "Roboto", serif; 90 | font-weight: 300; 91 | margin: 0 0 5px 0; 92 | } 93 | 94 | div.admonition p { 95 | font-weight: 300; 96 | margin-top: 1em; 97 | } 98 | 99 | .admonition a { 100 | font-weight: 300; 101 | } 102 | 103 | strong { 104 | font-weight: 400; 105 | } 106 | 107 | .pre { 108 | padding-left: 0.2em; 109 | padding-right: 0.2em; 110 | } 111 | -------------------------------------------------------------------------------- /features/freeze-projects.feature: -------------------------------------------------------------------------------- 1 | Feature: Freeze dependencies 2 | 3 | If a user didn't use a revision or branch in his manifest, he can add these automatically with `dfetch freeze`. 4 | During development it shouldn't be a problem to track a branch and always fetch the latest changes, 5 | but when a project becomes more mature, you typically want to freeze the dependencies. 6 | 7 | The same rules apply as for fetching, a tag has precedence over a branch and revision. 8 | 9 | Scenario: Git projects are frozen 10 | Given the manifest 'dfetch.yaml' 11 | """ 12 | manifest: 13 | version: '0.0' 14 | 15 | projects: 16 | - name: ext/test-repo-tag 17 | url: https://github.com/dfetch-org/test-repo 18 | branch: main 19 | 20 | """ 21 | And all projects are updated 22 | When I run "dfetch freeze" 23 | Then the manifest 'dfetch.yaml' is replaced with 24 | """ 25 | manifest: 26 | version: '0.0' 27 | 28 | projects: 29 | - name: ext/test-repo-tag 30 | revision: e1fda19a57b873eb8e6ae37780594cbb77b70f1a 31 | url: https://github.com/dfetch-org/test-repo 32 | branch: main 33 | 34 | """ 35 | 36 | @remote-svn 37 | Scenario: SVN projects are specified in the manifest 38 | Given the manifest 'dfetch.yaml' 39 | """ 40 | manifest: 41 | version: '0.0' 42 | 43 | projects: 44 | - name: cunit-svn 45 | vcs: svn 46 | url: svn://svn.code.sf.net/p/cunit/code 47 | 48 | """ 49 | And all projects are updated 50 | When I run "dfetch freeze" 51 | Then the manifest 'dfetch.yaml' is replaced with 52 | """ 53 | manifest: 54 | version: '0.0' 55 | 56 | projects: 57 | - name: cunit-svn 58 | revision: '170' 59 | url: svn://svn.code.sf.net/p/cunit/code 60 | branch: trunk 61 | vcs: svn 62 | 63 | """ 64 | -------------------------------------------------------------------------------- /dfetch/log.py: -------------------------------------------------------------------------------- 1 | """Logging related items.""" 2 | 3 | import logging 4 | from typing import cast 5 | 6 | import coloredlogs 7 | from colorama import Fore 8 | 9 | from dfetch import __version__ 10 | 11 | 12 | class DLogger(logging.Logger): 13 | """Logging class extended with specific log items for dfetch.""" 14 | 15 | def print_info_line(self, name: str, info: str) -> None: 16 | """Print a line of info.""" 17 | self.info(f" {Fore.GREEN}{name:20s}:{Fore.BLUE} {info}") 18 | 19 | def print_warning_line(self, name: str, info: str) -> None: 20 | """Print a line of info.""" 21 | self.info(f" {Fore.GREEN}{name:20s}:{Fore.YELLOW} {info}") 22 | 23 | def print_title(self) -> None: 24 | """Print the DFetch tool title and version.""" 25 | self.info(f"{Fore.BLUE}Dfetch ({__version__})") 26 | 27 | def print_info_field(self, field_name: str, field: str) -> None: 28 | """Print a field with corresponding value.""" 29 | self.print_info_line(field_name, field if field else "") 30 | 31 | 32 | def setup_root(name: str) -> DLogger: 33 | """Create the root logger.""" 34 | logger = get_logger(name) 35 | 36 | msg_format = "%(message)s" 37 | 38 | level_style = { 39 | "critical": {"color": "magenta", "bright": True, "bold": True}, 40 | "debug": {"color": "green", "bright": True, "bold": True}, 41 | "error": {"color": "red", "bright": True, "bold": True}, 42 | "info": {"color": 4, "bright": True, "bold": True}, 43 | "notice": {"color": "magenta", "bright": True, "bold": True}, 44 | "spam": {"color": "green", "faint": True}, 45 | "success": {"color": "green", "bright": True, "bold": True}, 46 | "verbose": {"color": "blue", "bright": True, "bold": True}, 47 | "warning": {"color": "yellow", "bright": True, "bold": True}, 48 | } 49 | 50 | coloredlogs.install(fmt=msg_format, level_styles=level_style, level="INFO") 51 | 52 | return logger 53 | 54 | 55 | def increase_verbosity() -> None: 56 | """Increase the verbosity of the logger.""" 57 | coloredlogs.increase_verbosity() 58 | 59 | 60 | def get_logger(name: str) -> DLogger: 61 | """Get logger for a module.""" 62 | logging.setLoggerClass(DLogger) 63 | return cast(DLogger, logging.getLogger(name)) 64 | -------------------------------------------------------------------------------- /script/release.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | """Script for simplifying the release.""" 4 | 5 | import glob 6 | import os 7 | import re 8 | from datetime import datetime 9 | 10 | from dfetch import __version__ 11 | 12 | 13 | def replace_pattern_in_files(file_path_pattern, search_pattern, replacement, flags=0): 14 | """ 15 | Searches for a given pattern in all matching files and replaces it with the specified replacement. 16 | 17 | Args: 18 | file_path_pattern (str): The glob pattern for files to search (e.g., "./**/*.feature"). 19 | search_pattern (str): The regex pattern to search for in files. 20 | replacement (str): The replacement string. 21 | flags (int): Optional regex flags (e.g., re.DOTALL for multiline matching). 22 | """ 23 | pattern = re.compile(search_pattern, flags) 24 | 25 | files_changed = [] 26 | 27 | for file_path in glob.glob(file_path_pattern, recursive=True): 28 | with open(file_path, "r", encoding="utf-8") as f: 29 | content = f.read() 30 | 31 | new_content = pattern.sub(replacement, content) 32 | 33 | if content != new_content: 34 | with open(file_path, "w", encoding="utf-8") as f: 35 | f.write(new_content) 36 | files_changed.append(file_path) 37 | print( 38 | f"Replaced '{search_pattern}' with '{replacement}' in {len(files_changed)} files" 39 | ) 40 | 41 | 42 | if __name__ == "__main__": 43 | base_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..") 44 | 45 | replace_pattern_in_files( 46 | file_path_pattern=f"{base_dir}/CHANGELOG.rst", 47 | search_pattern=r"(Release \d+\.\d+\.\d+) \(unreleased\)", 48 | replacement=r"\1 (released " + datetime.now().strftime("%Y-%m-%d") + ")", 49 | flags=re.DOTALL, 50 | ) 51 | 52 | replace_pattern_in_files( 53 | file_path_pattern=f"{base_dir}/**/*.feature", 54 | search_pattern=r"Dfetch \((\d+)\.(\d+)\.(\d+)\)", 55 | replacement=f"Dfetch ({__version__})", 56 | ) 57 | 58 | # replace_pattern_in_files( 59 | # file_path_pattern=f"{base_dir}/features/report-sbom.feature", 60 | # search_pattern=r'("name":\s*"dfetch",\s*"version":\s*")\d+\.\d+\.\d+(")', 61 | # replacement=r"\1" + __version__ + r"\2", 62 | # flags=re.DOTALL, 63 | # ) 64 | -------------------------------------------------------------------------------- /features/diff-in-git.feature: -------------------------------------------------------------------------------- 1 | Feature: Diff in git 2 | 3 | If a project contains issues that need to be fixed, the user can work with the *Dfetch'ed* project as 4 | any other piece of code within the project. To upstream the changes back to the original project, *Dfetch* 5 | should allow to generate a patch file. 6 | 7 | Background: 8 | Given a git repository "SomeProject.git" 9 | And a fetched and committed MyProject with the manifest 10 | """ 11 | manifest: 12 | version: 0.0 13 | projects: 14 | - name: SomeProject 15 | url: some-remote-server/SomeProject.git 16 | """ 17 | 18 | Scenario: A patch file is generated 19 | Given "SomeProject/README.md" in MyProject is changed and committed with 20 | """ 21 | An important sentence for the README! 22 | """ 23 | When I run "dfetch diff SomeProject" 24 | Then the patch file 'MyProject/SomeProject.patch' is generated 25 | """ 26 | diff --git a/README.md b/README.md 27 | index 1e65bd6..faa3b21 100644 28 | --- a/README.md 29 | +++ b/README.md 30 | @@ -1 +1,2 @@ 31 | Generated file for SomeProject.git 32 | +An important sentence for the README! 33 | """ 34 | 35 | Scenario: No change is present 36 | When I run "dfetch diff SomeProject" 37 | Then the output shows 38 | """ 39 | Dfetch (0.10.0) 40 | SomeProject : No diffs found since 59efb91396fd369eb113b43382783294dc8ed6d2 41 | """ 42 | 43 | Scenario: Diff is generated on uncommitted changes 44 | Given "SomeProject/README.md" in MyProject is changed with 45 | """ 46 | An important sentence for the README! 47 | """ 48 | When I run "dfetch diff SomeProject" 49 | Then the patch file 'MyProject/SomeProject.patch' is generated 50 | """ 51 | diff --git a/README.md b/README.md 52 | index 1e65bd6..faa3b21 100644 53 | --- a/README.md 54 | +++ b/README.md 55 | @@ -1 +1,2 @@ 56 | Generated file for SomeProject.git 57 | +An important sentence for the README! 58 | """ 59 | -------------------------------------------------------------------------------- /dfetch/commands/common.py: -------------------------------------------------------------------------------- 1 | """Module for common command operations.""" 2 | 3 | import os 4 | 5 | import yaml 6 | 7 | from dfetch.log import get_logger 8 | from dfetch.manifest.manifest import Manifest, get_childmanifests 9 | from dfetch.manifest.project import ProjectEntry 10 | 11 | logger = get_logger(__name__) 12 | 13 | 14 | def check_child_manifests(manifest: Manifest, project: ProjectEntry) -> None: 15 | """Check for child manifests within a project. 16 | 17 | Args: 18 | manifest (dfetch.manifest.manifest.Manifest): The parent manifest with projects. 19 | project (ProjectEntry): The parent project. 20 | """ 21 | for childmanifest in get_childmanifests(skip=[manifest.path]): 22 | recommendations: list[ProjectEntry] = [] 23 | for childproject in childmanifest.projects: 24 | if childproject.remote_url not in [ 25 | project.remote_url for project in manifest.projects 26 | ]: 27 | recommendations.append(childproject.as_recommendation()) 28 | 29 | if recommendations: 30 | childmanifest_relpath = os.path.relpath( 31 | childmanifest.path, start=os.path.dirname(manifest.path) 32 | ).replace("\\", "/") 33 | _make_recommendation(project, recommendations, childmanifest_relpath) 34 | 35 | 36 | def _make_recommendation( 37 | project: ProjectEntry, recommendations: list[ProjectEntry], childmanifest_path: str 38 | ) -> None: 39 | """Make recommendations to the user. 40 | 41 | Args: 42 | project (ProjectEntry): The parent project. 43 | recommendations (List[ProjectEntry]): List of recommendations 44 | childmanifest_path (str): Path to the source of recommendations 45 | """ 46 | logger.warning( 47 | "\n".join( 48 | [ 49 | "", 50 | f'"{project.name}" depends on the following project(s) ' 51 | "which are not part of your manifest:", 52 | f"(found in {childmanifest_path})", 53 | ] 54 | ) 55 | ) 56 | 57 | recommendation_json = yaml.dump( 58 | [proj.as_yaml() for proj in recommendations], 59 | indent=4, 60 | sort_keys=False, 61 | ) 62 | logger.warning("") 63 | for line in recommendation_json.splitlines(): 64 | logger.warning(line) 65 | logger.warning("") 66 | -------------------------------------------------------------------------------- /script/build.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | """This script builds the dfetch executable using Nuitka.""" 3 | import subprocess # nosec 4 | import sys 5 | import tomllib as toml 6 | from typing import Union 7 | 8 | from dfetch import __version__ 9 | 10 | 11 | def parse_option( 12 | option_name: str, option_value: Union[bool, str, list, dict] 13 | ) -> list[str]: 14 | """ 15 | Convert a config value to Nuitka CLI arguments. 16 | 17 | Handles booleans (--flag), strings (--flag=value), lists (multiple --flag=value), 18 | and nested dicts (--flag=key1=val1=key2=val2). 19 | 20 | Returns: 21 | list[str]: Nuitka CLI arguments in the format ['--flag', '--key=value'] 22 | """ 23 | args = [] 24 | cli_key = f"--{option_name.replace('_','-')}" 25 | 26 | if isinstance(option_value, bool): 27 | if option_value: 28 | args.append(cli_key) 29 | elif isinstance(option_value, str): 30 | args.append(f"{cli_key}={option_value}".replace("{VERSION}", __version__)) 31 | elif isinstance(option_value, list): 32 | for v in option_value: 33 | if isinstance(v, dict): 34 | parts = [f"{v[k]}" for k in v] 35 | args.append(f"{cli_key}={'='.join(parts)}") 36 | else: 37 | args.append(f"{cli_key}={v}") 38 | else: 39 | args.append(f"{cli_key}={option_value}") 40 | 41 | return args 42 | 43 | 44 | # Load pyproject.toml 45 | with open("pyproject.toml", "rb") as pyproject_file: 46 | pyproject = toml.load(pyproject_file) 47 | nuitka_opts = pyproject.get("tool", {}).get("nuitka", {}) 48 | 49 | 50 | if sys.platform.startswith("win"): 51 | nuitka_opts["output-filename"] = nuitka_opts["output-filename-win"] 52 | elif sys.platform.startswith("linux"): 53 | nuitka_opts["output-filename"] = nuitka_opts["output-filename-linux"] 54 | elif sys.platform.startswith("darwin"): 55 | nuitka_opts["output-filename"] = nuitka_opts["output-filename-macos"] 56 | 57 | 58 | nuitka_opts = { 59 | k: v 60 | for k, v in nuitka_opts.items() 61 | if k 62 | not in {"output-filename-win", "output-filename-linux", "output-filename-macos"} 63 | } 64 | 65 | command = [sys.executable, "-m", "nuitka"] 66 | for key, value in nuitka_opts.items(): 67 | command.extend(parse_option(key, value)) 68 | 69 | command.append("dfetch") 70 | 71 | print(command) 72 | subprocess.check_call(command) # nosec 73 | -------------------------------------------------------------------------------- /script/dependabot_hook.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | """This file performs any updates needed after a Dependabot PR.""" 3 | 4 | import re 5 | import sys 6 | from pathlib import Path 7 | from typing import Optional 8 | 9 | # Config 10 | SBOM_FILE = "sbom.json" # path to your CycloneDX SBOM 11 | PYPROJECT_FILE = "pyproject.toml" 12 | PACKAGE_NAME = "cyclonedx-python-lib" 13 | FEATURE_FILE = "features/report-sbom.feature" 14 | 15 | 16 | def get_new_version_from_pyproject(name: str) -> str: 17 | """Extract the updated version of a package from pyproject.toml assuming '==' pin.""" 18 | content = Path(PYPROJECT_FILE).read_text(encoding="UTF-8") 19 | 20 | # Match lines like: cyclonedx-python-lib=="11.5.0" 21 | pattern = re.compile(rf'{re.escape(name)}==["\']?([\d\.]+)["\']?', re.MULTILINE) 22 | 23 | match = pattern.search(content) 24 | if match: 25 | return match.group(1) 26 | 27 | raise ValueError(f"{name} not found in {PYPROJECT_FILE}") 28 | 29 | 30 | def replace_cyclonedx_version_if_outdated( 31 | new_version: str, 32 | ) -> Optional[str]: 33 | """Update the SBOM JSON file with the new version""" 34 | feature_file_path = Path(FEATURE_FILE) 35 | content = feature_file_path.read_text(encoding="UTF-8") 36 | 37 | pattern = re.compile( 38 | r'("name":\s*"cyclonedx-python-lib",\s*' 39 | r'"type":\s*"library",\s*' 40 | r'"version":\s*")(?P[^"]+)(")', 41 | re.MULTILINE, 42 | ) 43 | 44 | match = pattern.search(content) 45 | if not match: 46 | print("Error: cyclonedx-python-lib block not found in feature file") 47 | sys.exit(1) 48 | 49 | old_version = match.group("version") 50 | if old_version == new_version: 51 | print(f'No update needed: "{PACKAGE_NAME}" is already {new_version}') 52 | return None 53 | 54 | def replacer(m): 55 | return m.group(1) + new_version + m.group(3) 56 | 57 | new_content = pattern.sub(replacer, content) 58 | feature_file_path.write_text(new_content, encoding="UTF-8") 59 | print( 60 | f'Updated "{PACKAGE_NAME}" version: {old_version} → {new_version} in "{FEATURE_FILE}"' 61 | ) 62 | return old_version 63 | 64 | 65 | def main(): 66 | """Main entry point.""" 67 | new_version = get_new_version_from_pyproject("cyclonedx-python-lib") 68 | replace_cyclonedx_version_if_outdated(new_version) 69 | 70 | 71 | if __name__ == "__main__": 72 | main() 73 | -------------------------------------------------------------------------------- /features/diff-in-svn.feature: -------------------------------------------------------------------------------- 1 | Feature: Diff in svn 2 | 3 | If a project contains issues that need to be fixed, the user can work with the *Dfetch'ed* project as 4 | any other piece of code within the project. To upstream the changes back to the original project, *Dfetch* 5 | should allow to generate a patch file. 6 | 7 | Background: 8 | Given a svn-server "SomeProject" 9 | And a fetched and committed MySvnProject with the manifest 10 | """ 11 | manifest: 12 | version: 0.0 13 | projects: 14 | - name: SomeProject 15 | url: some-remote-server/SomeProject 16 | vcs: svn 17 | """ 18 | 19 | Scenario: A patch file is generated 20 | Given "SomeProject/README.md" in MySvnProject is changed, added and committed with 21 | """ 22 | An important sentence for the README! 23 | """ 24 | When I run "dfetch diff SomeProject" in MySvnProject 25 | Then the patch file 'MySvnProject/SomeProject.patch' is generated 26 | """ 27 | Index: SomeProject/README.md 28 | =================================================================== 29 | --- SomeProject/README.md (revision 1) 30 | +++ SomeProject/README.md (working copy) 31 | @@ -1 +1,2 @@ 32 | some content 33 | +An important sentence for the README! 34 | """ 35 | 36 | Scenario: No change is present 37 | When I run "dfetch diff SomeProject" in MySvnProject 38 | Then the output shows 39 | """ 40 | Dfetch (0.10.0) 41 | SomeProject : No diffs found since 1 42 | """ 43 | 44 | Scenario: A patch file is generated on uncommitted changes 45 | Given "SomeProject/README.md" in MySvnProject is changed with 46 | """ 47 | An important sentence for the README! 48 | """ 49 | When I run "dfetch diff SomeProject" in MySvnProject 50 | Then the patch file 'MySvnProject/SomeProject.patch' is generated 51 | """ 52 | Index: SomeProject/README.md 53 | =================================================================== 54 | --- SomeProject/README.md (revision 1) 55 | +++ SomeProject/README.md (working copy) 56 | @@ -1 +1,2 @@ 57 | some content 58 | +An important sentence for the README! 59 | """ 60 | -------------------------------------------------------------------------------- /features/fetch-single-file-git.feature: -------------------------------------------------------------------------------- 1 | Feature: Fetch single file from git repo 2 | 3 | Sometimes only one file is enough. *DFetch* makes it possible to specify 4 | only one file from a repository. 5 | 6 | Scenario: A single file is fetched from a repo 7 | Given the manifest 'dfetch.yaml' in MyProject 8 | """ 9 | manifest: 10 | version: 0.0 11 | projects: 12 | - name: SomeProjectWithAnInterestingFile 13 | url: some-remote-server/SomeProjectWithAnInterestingFile.git 14 | src: SomeFolder/SomeSubFolder 15 | tag: v1 16 | """ 17 | And a git-repository "SomeProjectWithAnInterestingFile.git" with the files 18 | | path | 19 | | SomeFolder/SomeSubFolder/SomeFile.txt | 20 | | SomeOtherFolder/SomeOtherFile.txt | 21 | When I run "dfetch update" 22 | Then the output shows 23 | """ 24 | Dfetch (0.10.0) 25 | SomeProjectWithAnInterestingFile: Fetched v1 26 | """ 27 | Then 'MyProject' looks like: 28 | """ 29 | MyProject/ 30 | SomeProjectWithAnInterestingFile/ 31 | .dfetch_data.yaml 32 | SomeFile.txt 33 | dfetch.yaml 34 | """ 35 | 36 | Scenario: A single file is fetched from a repo (dst) 37 | Given the manifest 'dfetch.yaml' in MyProject 38 | """ 39 | manifest: 40 | version: 0.0 41 | projects: 42 | - name: SomeProjectWithAnInterestingFile 43 | url: some-remote-server/SomeProjectWithAnInterestingFile.git 44 | dst: ext 45 | src: SomeFolder/SomeFile.txt 46 | tag: v1 47 | """ 48 | And a git-repository "SomeProjectWithAnInterestingFile.git" with the files 49 | | path | 50 | | SomeFolder/SomeFile.txt | 51 | | SomeOtherFolder/SomeOtherFile.txt | 52 | When I run "dfetch update" 53 | Then the output shows 54 | """ 55 | Dfetch (0.10.0) 56 | SomeProjectWithAnInterestingFile: Fetched v1 57 | """ 58 | Then 'MyProject' looks like: 59 | """ 60 | MyProject/ 61 | dfetch.yaml 62 | ext/ 63 | .dfetch_data.yaml 64 | SomeFile.txt 65 | """ 66 | -------------------------------------------------------------------------------- /features/patch-after-fetch-git.feature: -------------------------------------------------------------------------------- 1 | Feature: Patch after fetching from git repo 2 | 3 | Sometimes a patch needs to be applied after fetching. *DFetch* makes it 4 | possible to specify a patch file. 5 | 6 | Scenario: A patch file is applied after fetching 7 | Given the manifest 'dfetch.yaml' 8 | """ 9 | manifest: 10 | version: '0.0' 11 | 12 | remotes: 13 | - name: github-com-dfetch-org 14 | url-base: https://github.com/dfetch-org/test-repo 15 | 16 | projects: 17 | - name: ext/test-repo-tag 18 | tag: v2.0 19 | dst: ext/test-repo-tag 20 | patch: diff.patch 21 | """ 22 | And the patch file 'diff.patch' 23 | """ 24 | diff --git a/README.md b/README.md 25 | index 32d9fad..62248b7 100644 26 | --- a/README.md 27 | +++ b/README.md 28 | @@ -1,2 +1,2 @@ 29 | # Test-repo 30 | -A test repo for testing dfetch. 31 | +A test repo for testing patch. 32 | """ 33 | When I run "dfetch update" 34 | Then the patched 'ext/test-repo-tag/README.md' is 35 | """ 36 | # Test-repo 37 | A test repo for testing patch. 38 | """ 39 | 40 | Scenario: Applying patch file fails 41 | Given the manifest 'dfetch.yaml' 42 | """ 43 | manifest: 44 | version: '0.0' 45 | 46 | remotes: 47 | - name: github-com-dfetch-org 48 | url-base: https://github.com/dfetch-org/test-repo 49 | 50 | projects: 51 | - name: ext/test-repo-tag 52 | tag: v2.0 53 | dst: ext/test-repo-tag 54 | patch: diff.patch 55 | """ 56 | And the patch file 'diff.patch' 57 | """ 58 | diff --git a/README.md b/README1.md 59 | index 32d9fad..62248b7 100644 60 | --- a/README1.md 61 | +++ b/README1.md 62 | @@ -1,2 +1,2 @@ 63 | # Test-repo 64 | -A test repo for testing dfetch. 65 | +A test repo for testing patch. 66 | """ 67 | When I run "dfetch update" 68 | Then the output shows 69 | """ 70 | Dfetch (0.10.0) 71 | ext/test-repo-tag : Fetched v2.0 72 | source/target file does not exist: 73 | --- b'README1.md' 74 | +++ b'README1.md' 75 | Applying patch "diff.patch" failed 76 | """ 77 | -------------------------------------------------------------------------------- /features/patch-after-fetch-svn.feature: -------------------------------------------------------------------------------- 1 | @remote-svn 2 | Feature: Patch after fetching from svn repo 3 | 4 | Sometimes a patch needs to be applied after fetching. *DFetch* makes it 5 | possible to specify a patch file. 6 | 7 | Scenario: A patch file is applied after fetching 8 | Given the manifest 'dfetch.yaml' 9 | """ 10 | manifest: 11 | version: '0.0' 12 | 13 | remotes: 14 | - name: cutter 15 | url-base: svn://svn.code.sf.net/p/cutter/svn/cutter 16 | 17 | projects: 18 | - name: cutter 19 | vcs: svn 20 | tag: 1.1.7 21 | dst: ext/cutter 22 | patch: diff.patch 23 | src: apt 24 | """ 25 | And the patch file 'diff.patch' 26 | """ 27 | Index: build-deb.sh 28 | =================================================================== 29 | --- build-deb.sh (revision 4007) 30 | +++ build-deb.sh (working copy) 31 | @@ -1,1 +1,1 @@ 32 | -#!/bin/sh 33 | +#!/bin/bash 34 | """ 35 | When I run "dfetch update" 36 | Then the first line of 'ext/cutter/build-deb.sh' is changed to 37 | """ 38 | #!/bin/bash 39 | """ 40 | 41 | Scenario: Applying patch file fails 42 | Given the manifest 'dfetch.yaml' 43 | """ 44 | manifest: 45 | version: '0.0' 46 | 47 | remotes: 48 | - name: cutter 49 | url-base: svn://svn.code.sf.net/p/cutter/svn/cutter 50 | 51 | projects: 52 | - name: cutter 53 | vcs: svn 54 | tag: 1.1.7 55 | dst: ext/cutter 56 | patch: diff.patch 57 | src: apt 58 | """ 59 | And the patch file 'diff.patch' 60 | """ 61 | Index: build-deb.sh 62 | =================================================================== 63 | --- build-deb2.sh (revision 4007) 64 | +++ build-deb2.sh (working copy) 65 | @@ -1,1 +1,1 @@ 66 | -#!/bin/sh 67 | +#!/bin/bash 68 | """ 69 | When I run "dfetch update" 70 | Then the output shows 71 | """ 72 | Dfetch (0.10.0) 73 | cutter : Fetched 1.1.7 74 | source/target file does not exist: 75 | --- b'build-deb2.sh' 76 | +++ b'build-deb2.sh' 77 | Applying patch "diff.patch" failed 78 | """ 79 | -------------------------------------------------------------------------------- /dfetch/__main__.py: -------------------------------------------------------------------------------- 1 | """Find the complete documentation online. 2 | 3 | https://dfetch.rtfd.org 4 | """ 5 | 6 | import argparse 7 | import sys 8 | from collections.abc import Sequence 9 | 10 | import dfetch.commands.check 11 | import dfetch.commands.diff 12 | import dfetch.commands.environment 13 | import dfetch.commands.freeze 14 | import dfetch.commands.import_ 15 | import dfetch.commands.init 16 | import dfetch.commands.report 17 | import dfetch.commands.update 18 | import dfetch.commands.validate 19 | import dfetch.log 20 | import dfetch.util.cmdline 21 | 22 | logger = dfetch.log.setup_root(__name__) 23 | 24 | 25 | class DfetchFatalException(Exception): 26 | """Exception thrown when dfetch did not run successfully.""" 27 | 28 | 29 | def create_parser() -> argparse.ArgumentParser: 30 | """Create the main argument parser.""" 31 | parser = argparse.ArgumentParser( 32 | formatter_class=argparse.RawTextHelpFormatter, epilog=__doc__ 33 | ) 34 | parser.add_argument( 35 | "--verbose", "-v", action="store_true", help="Increase verbosity" 36 | ) 37 | parser.set_defaults(func=_help) 38 | subparsers = parser.add_subparsers(help="commands") 39 | 40 | dfetch.commands.check.Check.create_menu(subparsers) 41 | dfetch.commands.diff.Diff.create_menu(subparsers) 42 | dfetch.commands.environment.Environment.create_menu(subparsers) 43 | dfetch.commands.freeze.Freeze.create_menu(subparsers) 44 | dfetch.commands.import_.Import.create_menu(subparsers) 45 | dfetch.commands.init.Init.create_menu(subparsers) 46 | dfetch.commands.report.Report.create_menu(subparsers) 47 | dfetch.commands.update.Update.create_menu(subparsers) 48 | dfetch.commands.validate.Validate.create_menu(subparsers) 49 | 50 | return parser 51 | 52 | 53 | def _help(args: argparse.Namespace) -> None: 54 | """Show the help.""" 55 | raise RuntimeError("Select a function") 56 | 57 | 58 | def run(argv: Sequence[str]) -> None: 59 | """Start dfetch.""" 60 | logger.print_title() 61 | args = create_parser().parse_args(argv) 62 | 63 | if args.verbose: 64 | dfetch.log.increase_verbosity() 65 | 66 | try: 67 | args.func(args) 68 | except RuntimeError as exc: 69 | for msg in exc.args: 70 | logger.error(msg, stack_info=False) 71 | raise DfetchFatalException from exc 72 | except dfetch.util.cmdline.SubprocessCommandError as exc: 73 | logger.error(exc.message) 74 | raise DfetchFatalException from exc 75 | 76 | 77 | def main() -> None: 78 | """Start dfetch and let it collect arguments from the command-line.""" 79 | try: 80 | run(sys.argv[1:]) 81 | except DfetchFatalException: 82 | sys.exit(1) 83 | 84 | 85 | if __name__ == "__main__": 86 | main() 87 | -------------------------------------------------------------------------------- /doc/asciicasts/init.cast: -------------------------------------------------------------------------------- 1 | {"version": 2, "width": 128, "height": 31, "timestamp": 1760470338, "env": {"SHELL": "/bin/sh", "TERM": "xterm-256color"}} 2 | [0.008845, "o", "\u001b[H\u001b[2J\u001b[3J"] 3 | [0.011609, "o", "$ "] 4 | [1.013226, "o", "\u001b"] 5 | [1.193494, "o", "[1"] 6 | [1.283657, "o", "ml"] 7 | [1.373829, "o", "s "] 8 | [1.464009, "o", "-"] 9 | [1.554057, "o", "l\u001b"] 10 | [1.644186, "o", "[0"] 11 | [1.734335, "o", "m"] 12 | [2.734948, "o", "\r\n"] 13 | [2.737275, "o", "total 0\r\n"] 14 | [2.742561, "o", "$ "] 15 | [3.744047, "o", "\u001b["] 16 | [3.924301, "o", "1m"] 17 | [4.01443, "o", "df"] 18 | [4.104622, "o", "et"] 19 | [4.194768, "o", "ch"] 20 | [4.284999, "o", " i"] 21 | [4.375129, "o", "ni"] 22 | [4.465299, "o", "t\u001b"] 23 | [4.555448, "o", "[0"] 24 | [4.645861, "o", "m"] 25 | [5.645998, "o", "\r\n"] 26 | [6.0752, "o", "\u001b[1;38;5;4m\u001b[34mDfetch (0.10.0)\u001b[0m\r\n\u001b[0m"] 27 | [6.077876, "o", "\u001b[1;38;5;4mCreated dfetch.yaml\u001b[0m\r\n"] 28 | [6.077936, "o", "\u001b[0m"] 29 | [6.078025, "o", "\u001b[0m"] 30 | [6.133437, "o", "$ "] 31 | [7.134952, "o", "\u001b["] 32 | [7.315199, "o", "1m"] 33 | [7.405338, "o", "ls"] 34 | [7.495511, "o", " -"] 35 | [7.585626, "o", "l\u001b"] 36 | [7.675976, "o", "[0"] 37 | [7.766134, "o", "m"] 38 | [8.766606, "o", "\r\n"] 39 | [8.769051, "o", "total 4\r\n"] 40 | [8.769155, "o", "-rw-rw-rw- 1 dev dev 733 Oct 14 19:32 dfetch.yaml\r\n"] 41 | [8.772774, "o", "$ "] 42 | [9.774398, "o", "\u001b["] 43 | [9.954656, "o", "1m"] 44 | [10.044814, "o", "ca"] 45 | [10.13496, "o", "t "] 46 | [10.225115, "o", "df"] 47 | [10.315327, "o", "et"] 48 | [10.405404, "o", "ch"] 49 | [10.495577, "o", ".y"] 50 | [10.585679, "o", "am"] 51 | [10.675946, "o", "l\u001b"] 52 | [10.856176, "o", "[0m"] 53 | [11.856806, "o", "\r\n"] 54 | [11.858855, "o", "manifest:\r\n version: 0.0 # DFetch Module syntax version\r\n\r\n remotes: # declare common sources in one place\r\n - name: github\r\n url-base: https://github.com/\r\n\r\n projects:\r\n - name: cpputest\r\n dst: cpputest/src/ # Destination of this project (relative to this file)\r\n repo-path: cpputest/cpputest.git # Use default github remote\r\n tag: v3.4 # tag\r\n\r\n - name: jsmn # without destination, defaults to project name\r\n repo-path: zserge/jsmn.git # only repo-path is enough\r\n"] 55 | [14.864551, "o", "$ "] 56 | [14.865884, "o", "\u001b["] 57 | [15.046183, "o", "1m"] 58 | [15.13633, "o", "\u001b["] 59 | [15.226714, "o", "0m"] 60 | [15.227241, "o", "\r\n"] 61 | [15.229048, "o", "/workspaces/dfetch/doc/generate-casts\r\n"] 62 | -------------------------------------------------------------------------------- /dfetch/util/license.py: -------------------------------------------------------------------------------- 1 | """*Dfetch* uses *Infer-License* to guess licenses from files.""" 2 | 3 | from dataclasses import dataclass 4 | from os import PathLike 5 | from typing import Optional, Union 6 | 7 | import infer_license 8 | from infer_license.types import License as InferredLicense 9 | 10 | # Limit license file size to below number of bytes to prevent memory issues with large files 11 | MAX_LICENSE_FILE_SIZE = 1024 * 1024 # 1 MB 12 | 13 | 14 | @dataclass 15 | class License: 16 | """Represents a software license with its SPDX identifiers and detection confidence. 17 | 18 | This class encapsulates license information detected by the infer-license library, 19 | providing standardized identifiers and confidence level of the detection. 20 | """ 21 | 22 | name: str #: SPDX Full name 23 | spdx_id: str #: SPDX Identifier 24 | trove_classifier: Optional[str] #: Python package classifier 25 | probability: float #: Confidence level of the license inference 26 | 27 | @staticmethod 28 | def from_inferred( 29 | inferred_license: InferredLicense, probability: float 30 | ) -> "License": 31 | """Convert an infer-license License object to our internal License representation. 32 | 33 | Args: 34 | inferred_license: The license object from infer-license library 35 | probability: The confidence score (0-1) of the license detection 36 | 37 | Returns: 38 | License: A new License instance with the inferred information 39 | """ 40 | return License( 41 | name=inferred_license.name, 42 | spdx_id=inferred_license.shortname, 43 | trove_classifier=inferred_license.trove_classifier, 44 | probability=probability, 45 | ) 46 | 47 | 48 | def guess_license_in_file( 49 | filename: Union[str, PathLike[str]], 50 | ) -> Optional[License]: 51 | """Attempt to identify the license of a given file. 52 | 53 | Tries UTF-8 encoding first, falling back to Latin-1 for legacy license files. 54 | If the file cannot be read or no license is detected, returns None. 55 | 56 | Args: 57 | filename (Union[str, os.PathLike[str]]): Path to the file to analyze 58 | 59 | Returns: 60 | Optional[License]: The most probable license if found, None if no license could be detected 61 | """ 62 | try: 63 | with open(filename, "rb") as f: 64 | file_bytes = f.read(MAX_LICENSE_FILE_SIZE) 65 | try: 66 | license_text = file_bytes.decode("utf-8") 67 | except UnicodeDecodeError: 68 | license_text = file_bytes.decode("latin-1") 69 | except (FileNotFoundError, PermissionError, IsADirectoryError, OSError): 70 | return None 71 | 72 | probable_licenses = infer_license.api.probabilities(license_text) 73 | 74 | return ( 75 | None if not probable_licenses else License.from_inferred(*probable_licenses[0]) 76 | ) 77 | -------------------------------------------------------------------------------- /dfetch/util/cmdline.py: -------------------------------------------------------------------------------- 1 | """Module for performing cmd line arguments.""" 2 | 3 | import logging 4 | import os 5 | import subprocess # nosec 6 | from typing import Any, Optional, Union # pylint: disable=unused-import 7 | 8 | 9 | class SubprocessCommandError(Exception): 10 | """Error raised when a subprocess fails. 11 | 12 | Whenever a subprocess is executed something can happen. This exception 13 | contains all the results for easier usage later on. 14 | """ 15 | 16 | def __init__( 17 | self, 18 | cmd: Optional[list[str]] = None, 19 | stdout: str = "", 20 | stderr: str = "", 21 | returncode: int = 0, 22 | ): 23 | """Error.""" 24 | cmd_str: str = " ".join(cmd or []) 25 | self._message = f">>>{cmd_str}<<< returned {returncode}:{os.linesep}{stderr}" 26 | self.cmd = cmd_str 27 | self.stderr = stdout 28 | self.stdout = stderr 29 | self.returncode = returncode 30 | super().__init__(self._message) 31 | 32 | @property 33 | def message(self) -> str: 34 | """Return the message of this SubprocessCommandError.""" 35 | return self._message 36 | 37 | 38 | def run_on_cmdline( 39 | logger: logging.Logger, cmd: Union[str, list[str]] 40 | ) -> "subprocess.CompletedProcess[Any]": 41 | """Run a command and log the output, and raise if something goes wrong.""" 42 | logger.debug(f"Running {cmd}") 43 | 44 | if not isinstance(cmd, list): 45 | cmd = cmd.split(" ") 46 | 47 | try: 48 | proc = subprocess.run(cmd, capture_output=True, check=True) # nosec 49 | except subprocess.CalledProcessError as exc: 50 | raise SubprocessCommandError( 51 | exc.cmd, 52 | exc.output.decode().strip(), 53 | exc.stderr.decode().strip(), 54 | exc.returncode, 55 | ) from exc 56 | except FileNotFoundError as exc: 57 | cmd = cmd[0] 58 | raise RuntimeError(f"{cmd} not available on system, please install") from exc 59 | 60 | stdout, stderr = proc.stdout, proc.stderr 61 | 62 | _log_output(proc, logger) 63 | 64 | if proc.returncode: 65 | raise SubprocessCommandError( 66 | cmd, stdout.decode(), stderr.decode().strip(), proc.returncode 67 | ) 68 | 69 | return proc 70 | 71 | 72 | def _log_output(proc: subprocess.CompletedProcess, logger: logging.Logger) -> None: # type: ignore 73 | logger.debug(f"Return code: {proc.returncode}") 74 | 75 | _log_output_stream("stdout", proc.stdout, logger) 76 | _log_output_stream("stderr", proc.stderr, logger) 77 | 78 | 79 | def _log_output_stream(name: str, stream: Any, logger: logging.Logger) -> None: 80 | logger.debug(f"{name}:") 81 | try: 82 | for line in stream.decode().split("\n\n"): 83 | logger.debug(line) 84 | except UnicodeDecodeError: 85 | for line in stream.decode(encoding="cp1252").split("\n\n"): 86 | logger.debug(line) 87 | -------------------------------------------------------------------------------- /doc/index.rst: -------------------------------------------------------------------------------- 1 | .. Dfetch documentation master file 2 | 3 | .. meta:: 4 | :description: Dfetch is a VCS-agnostic tool that simplifies dependency management by retrieving 5 | source-only dependencies from various repositories, promoting upstream changes and 6 | allowing local customizations. 7 | :keywords: dfetch, dependency management, embedded development, fetch tool, vendoring, multi-repo, dependencies, git, svn, package manager, multi-project, monorepo 8 | :author: Dfetch Contributors 9 | :google-site-verification: yCnoTogJMh7Nm5gxlREDuONIXT4ijHcj972Y5k9p-sU 10 | 11 | .. raw:: html 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | .. image:: images/dfetch_header.png 25 | :width: 100% 26 | :align: center 27 | 28 | .. toctree:: 29 | :maxdepth: 2 30 | 31 | getting_started 32 | manifest 33 | manual 34 | troubleshooting 35 | contributing 36 | changelog 37 | alternatives 38 | legal 39 | internal 40 | 41 | Dfetch - *a source-only no-hassle project-dependency aggregator* 42 | ================================================================ 43 | 44 | What is Dfetch? 45 | --------------- 46 | 47 | We needed a dependency manager that was flexible enough to retrieve dependencies as plain text 48 | from various sources. `svn externals`, `git submodules` and `git subtrees` solve a similar 49 | problem, but not in a vcs agnostic way or completely user friendly way. 50 | We want self-contained code repositories without any hassle for end-users. 51 | Dfetch must promote upstreaming changes, but allow for local customizations. 52 | 53 | Other tools that do similar things are `Zephyr's West`, `CMake ExternalProject` and other meta tools. 54 | See :ref:`alternatives` for a complete list. 55 | 56 | Installation 57 | ------------ 58 | `Dfetch` is a python based cross-platform cli tool. 59 | 60 | Install the latest release with: 61 | 62 | .. code-block:: 63 | 64 | pip install dfetch 65 | 66 | Or install the latest version from the main branch: 67 | 68 | .. code-block:: 69 | 70 | pip install https://github.com/dfetch-org/dfetch/archive/main.zip 71 | 72 | Once installed dfetch output can be seen. 73 | 74 | .. code-block:: 75 | 76 | dfetch --version 77 | 78 | Basic usage 79 | ----------- 80 | 81 | .. asciinema:: asciicasts/basic.cast 82 | -------------------------------------------------------------------------------- /doc/static/uml/c3_dfetch_components_commands.puml: -------------------------------------------------------------------------------- 1 | @startuml 2 | 3 | !include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Component.puml 4 | 5 | Person(user, "Developer") 6 | 7 | System_Boundary(DFetch, "Dfetch") { 8 | 9 | Boundary(DFetchCommands, "Commands") { 10 | Component(compCommon, "Common", "python", "Does stuff") 11 | Component(compCommand, "Command", "python", "Does stuff") 12 | 13 | Component(compCheck, "Check", "python", "Does stuff") 14 | Component(compDiff, "Diff", "python", "Does stuff") 15 | Component(compEnv, "Environment", "python", "Does stuff") 16 | Component(compFreeze, "Freeze", "python", "Does stuff") 17 | Component(compImport, "Import", "python", "Does stuff") 18 | Component(compInit, "Init", "python", "Does stuff") 19 | Component(compReport, "Report", "python", "Does stuff") 20 | Component(compUpdate, "Update", "python", "Does stuff") 21 | Component(compValidate, "Validate", "python", "Does stuff") 22 | 23 | Rel_U(compValidate, compCommand, "Extends") 24 | Rel_U(compCheck, compCommand, "Extends") 25 | Rel_U(compDiff, compCommand, "Extends") 26 | Rel_U(compEnv, compCommand, "Extends") 27 | Rel_U(compFreeze, compCommand, "Extends") 28 | Rel_U(compImport, compCommand, "Extends") 29 | Rel_U(compInit, compCommand, "Extends") 30 | Rel_U(compReport, compCommand, "Extends") 31 | Rel_U(compUpdate, compCommand, "Extends") 32 | 33 | Rel_U(compUpdate, compCommon, "Uses") 34 | Rel_U(compCheck, compCommon, "Uses") 35 | } 36 | 37 | Container(contManifest, "Manifest", "python", "Parsing, editing and finding of manifests.") 38 | Container(contProject, "Project", "python", "Main project that has a manifest.") 39 | Container(contVcs, "Vcs", "python", "Abstraction of various Version Control Systems.") 40 | Container(contReporting, "Reporting", "python", "Output formatters for various reporting formats.") 41 | 42 | Rel(compCheck, contManifest, "Uses") 43 | Rel(compCheck, contProject, "Uses") 44 | Rel(compCheck, contReporting, "Uses") 45 | 46 | Rel(compDiff, contManifest, "Uses") 47 | Rel(compDiff, contProject, "Uses") 48 | 49 | Rel(compEnv, contProject, "Uses") 50 | 51 | Rel(compFreeze, contManifest, "Uses") 52 | Rel(compFreeze, contProject, "Uses") 53 | 54 | Rel(compImport, contManifest, "Uses") 55 | Rel(compImport, contProject, "Uses") 56 | Rel(compImport, contVcs, "Uses") 57 | 58 | Rel(compReport, contManifest, "Uses") 59 | Rel(compReport, contProject, "Uses") 60 | Rel(compReport, contReporting, "Uses") 61 | 62 | Rel(compUpdate, contManifest, "Uses") 63 | Rel(compUpdate, contProject, "Uses") 64 | 65 | Rel(compValidate, contManifest, "Uses") 66 | 67 | Rel(contProject, contReporting, "Uses") 68 | Rel_R(contProject, contManifest, "Has") 69 | Rel_U(contReporting, contManifest, "Uses") 70 | Rel(contProject, contVcs, "Uses") 71 | } 72 | 73 | Rel(user, compCommand, "Uses") 74 | 75 | @enduml 76 | -------------------------------------------------------------------------------- /script/create_venv.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | """Script to setup a venv.""" 3 | 4 | import argparse 5 | import pathlib 6 | import subprocess # nosec 7 | import sys 8 | import venv 9 | from typing import Any 10 | 11 | PROJECT_ROOT = pathlib.Path(__file__).resolve().parent.parent 12 | 13 | MIN_VERSION = (3, 9) # minimum supported; change if needed 14 | RECOMMENDED_VERSION = (3, 13) # preferred for development 15 | 16 | 17 | class MyEnvBuilder(venv.EnvBuilder): 18 | """Create a virtual environment. 19 | 20 | Optionally install extra requirements from pyproject.toml. 21 | """ 22 | 23 | def __init__( 24 | self, 25 | *args: Any, 26 | extra_requirements: str = "", 27 | **kwargs: Any, 28 | ) -> None: # pylint: disable=line-too-long 29 | """:param extra_requirements: Install any additional parts as mentioned in pyproject.toml.""" 30 | super().__init__(*args, **kwargs) 31 | self.extra_requirements = ( 32 | f"[{extra_requirements}]" if extra_requirements else "" 33 | ) 34 | 35 | def post_setup(self, context: Any) -> None: 36 | """Set up proper environment for testing.""" 37 | super().post_setup(context) 38 | 39 | print("Upgrading pip") 40 | self.pip_install(context, "--upgrade", "pip") 41 | print("Installing package and any extra requirements") 42 | self.pip_install( 43 | context, "--use-pep517", "-e", f"{PROJECT_ROOT!s}{self.extra_requirements}" 44 | ) 45 | 46 | @staticmethod 47 | def pip_install(context: Any, *args: Any) -> None: 48 | """Install something using pip. 49 | 50 | We run pip in isolated mode to avoid side effects from 51 | environment vars, the current directory and anything else 52 | intended for the global Python environment 53 | (same as EnvBuilder's _setup_pip) 54 | """ 55 | subprocess.check_call( # nosec 56 | (context.env_exe, "-Im", "pip", "install") + args, 57 | stderr=subprocess.STDOUT, 58 | ) 59 | 60 | 61 | if __name__ == "__main__": 62 | PARSER = argparse.ArgumentParser() 63 | PARSER.add_argument( 64 | "-e", "--extra_requirements", type=str, default="development,test,docs" 65 | ) 66 | ARGS = PARSER.parse_args() 67 | 68 | CURRENT_VERSION = sys.version_info[:2] 69 | 70 | if CURRENT_VERSION < MIN_VERSION: 71 | raise RuntimeError( 72 | f"⚠ Unsupported Python version {sys.version_info.major}.{sys.version_info.minor}. " 73 | f"Please use Python {MIN_VERSION[0]}.{MIN_VERSION[1]} or newer." 74 | ) 75 | if CURRENT_VERSION != RECOMMENDED_VERSION: 76 | print( 77 | f"⚠ Warning: Running with Python {sys.version_info.major}.{sys.version_info.minor}, " 78 | f"dfetch is primarily developed with Python {RECOMMENDED_VERSION[0]}.{RECOMMENDED_VERSION[1]}." 79 | ) 80 | 81 | MyEnvBuilder( 82 | clear=False, 83 | with_pip=True, 84 | extra_requirements=ARGS.extra_requirements, 85 | ).create(str(PROJECT_ROOT / "venv")) 86 | -------------------------------------------------------------------------------- /features/fetch-git-repo.feature: -------------------------------------------------------------------------------- 1 | Feature: Fetching dependencies from a git repository 2 | 3 | The main functionality of *DFetch* is fetching remote dependencies. 4 | A key VCS that is used in the world is git. *DFetch* makes it possible to 5 | fetch git repositories, using the revision, branch, tag or a combination. 6 | 7 | Scenario: Git projects are specified in the manifest 8 | Given the manifest 'dfetch.yaml' 9 | """ 10 | manifest: 11 | version: '0.0' 12 | 13 | remotes: 14 | - name: github-com-dfetch-org 15 | url-base: https://github.com/dfetch-org/test-repo 16 | 17 | projects: 18 | - name: ext/test-repo-rev-only 19 | revision: e1fda19a57b873eb8e6ae37780594cbb77b70f1a 20 | dst: ext/test-repo-rev-only 21 | 22 | - name: ext/test-rev-and-branch 23 | revision: 8df389d0524863b85f484f15a91c5f2c40aefda1 24 | branch: main 25 | dst: ext/test-rev-and-branch 26 | 27 | - name: ext/test-repo-tag-v1 28 | tag: v1 29 | dst: ext/test-repo-tag-v1 30 | 31 | """ 32 | When I run "dfetch update" 33 | Then the following projects are fetched 34 | | path | 35 | | ext/test-repo-rev-only | 36 | | ext/test-rev-and-branch | 37 | | ext/test-repo-tag-v1 | 38 | 39 | Scenario: Tag is updated in manifest 40 | Given the manifest 'dfetch.yaml' 41 | """ 42 | manifest: 43 | version: '0.0' 44 | 45 | projects: 46 | - name: ext/test-repo-tag 47 | url: https://github.com/dfetch-org/test-repo 48 | tag: v1 49 | 50 | """ 51 | And all projects are updated 52 | When the manifest 'dfetch.yaml' is changed to 53 | """ 54 | manifest: 55 | version: '0.0' 56 | 57 | projects: 58 | - name: ext/test-repo-tag 59 | url: https://github.com/dfetch-org/test-repo 60 | tag: v2.0 61 | 62 | """ 63 | And I run "dfetch update" 64 | Then the output shows 65 | """ 66 | Dfetch (0.10.0) 67 | ext/test-repo-tag : Fetched v2.0 68 | """ 69 | 70 | Scenario: Version check ignored when force flag is given 71 | Given the manifest 'dfetch.yaml' 72 | """ 73 | manifest: 74 | version: '0.0' 75 | 76 | projects: 77 | - name: ext/test-repo-tag 78 | url: https://github.com/dfetch-org/test-repo 79 | tag: v1 80 | 81 | """ 82 | And all projects are updated 83 | When I run "dfetch update --force" 84 | Then the output shows 85 | """ 86 | Dfetch (0.10.0) 87 | ext/test-repo-tag : Fetched v1 88 | """ 89 | -------------------------------------------------------------------------------- /dfetch/project/abstract_check_reporter.py: -------------------------------------------------------------------------------- 1 | """Interface for reporting check results.""" 2 | 3 | from abc import ABC, abstractmethod 4 | 5 | from dfetch.manifest.project import ProjectEntry 6 | from dfetch.manifest.version import Version 7 | 8 | 9 | class AbstractCheckReporter(ABC): 10 | """Reporter for generating report.""" 11 | 12 | @abstractmethod 13 | def __init__(self, manifest_path: str) -> None: 14 | """Create the reporter. 15 | 16 | Args: 17 | manifest_path (str): The path to the manifest. 18 | """ 19 | 20 | @abstractmethod 21 | def unfetched_project( 22 | self, project: ProjectEntry, wanted_version: Version, latest: Version 23 | ) -> None: 24 | """Report an unfetched project. 25 | 26 | Args: 27 | project (ProjectEntry): The unfetched project. 28 | wanted_version (Version): The wanted version. 29 | latest (Version): The latest available version. 30 | """ 31 | 32 | @abstractmethod 33 | def up_to_date_project(self, project: ProjectEntry, latest: Version) -> None: 34 | """Report an up-to-date project. 35 | 36 | Args: 37 | project (ProjectEntry): The up-to-date project 38 | latest (Version): The last version. 39 | """ 40 | 41 | @abstractmethod 42 | def pinned_but_out_of_date_project( 43 | self, project: ProjectEntry, wanted_version: Version, latest: Version 44 | ) -> None: 45 | """Report a pinned but out-of-date project. 46 | 47 | Args: 48 | project (ProjectEntry): Project that is pinned but out-of-date 49 | wanted_version (Version): Version that is wanted by manifest 50 | latest (Version): Available version 51 | """ 52 | 53 | @abstractmethod 54 | def unavailable_project_version( 55 | self, project: ProjectEntry, wanted_version: Version 56 | ) -> None: 57 | """Report a pinned but unavailable project version. 58 | 59 | Args: 60 | project (ProjectEntry): Project that does not have the wanted_version available. 61 | wanted_version (Version): Version that is wanted by manifest 62 | """ 63 | 64 | @abstractmethod 65 | def out_of_date_project( 66 | self, 67 | project: ProjectEntry, 68 | wanted_version: Version, 69 | current: Version, 70 | latest: Version, 71 | ) -> None: 72 | """Report an out-of-date project. 73 | 74 | Args: 75 | project (ProjectEntry): Project that is out-of-date 76 | wanted_version (Version): Version that is wanted by manifest 77 | current (Version): Current version on disk 78 | latest (Version): Available version 79 | """ 80 | 81 | @abstractmethod 82 | def local_changes(self, project: ProjectEntry) -> None: 83 | """Report an project with local changes. 84 | 85 | Args: 86 | project (ProjectEntry): The project with local changes. 87 | """ 88 | 89 | @abstractmethod 90 | def dump_to_file(self) -> None: 91 | """Do nothing.""" 92 | -------------------------------------------------------------------------------- /.github/workflows/test.yml: -------------------------------------------------------------------------------- 1 | name: Test 2 | 3 | on: 4 | push: 5 | branches: 6 | - main 7 | pull_request: 8 | types: [opened, synchronize, reopened] 9 | 10 | permissions: 11 | contents: read 12 | 13 | jobs: 14 | test: 15 | runs-on: ubuntu-latest 16 | steps: 17 | - name: Harden the runner (Audit all outbound calls) 18 | uses: step-security/harden-runner@20cf305ff2072d973412fa9b1e3a4f227bda3c76 # v2.14.0 19 | with: 20 | egress-policy: audit 21 | 22 | - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v5.0.0 23 | 24 | - name: Setup Python 25 | uses: actions/setup-python@83679a892e2d95755f2dac6acb0bfd1e9ac5d548 # v6.1.0 26 | with: 27 | python-version: '3.13' 28 | 29 | - name: Install Subversion (SVN) 30 | run: | 31 | sudo apt-get update 32 | sudo apt-get install -y subversion 33 | svn --version # Verify installation 34 | svnadmin --version # Verify installation 35 | 36 | - name: Install dependencies 37 | run: | 38 | pip install .[development,test] 39 | 40 | - run: codespell # Check for typo's 41 | - run: isort --diff dfetch # Checks import order 42 | - run: black --check dfetch # Checks code style 43 | # - run: flake8 dfetch # Checks pep8 conformance 44 | - run: pylint dfetch # Checks pep8 conformance 45 | - run: ruff check dfetch # Check using ruff 46 | - run: mypy dfetch # Check types 47 | - run: pyright . # Check types 48 | - run: doc8 doc # Checks documentation 49 | - run: pydocstyle dfetch # Checks doc strings 50 | - run: bandit -r dfetch # Checks security issues 51 | - run: xenon -b B -m A -a A dfetch # Check code quality 52 | - run: pytest --cov=dfetch tests # Run tests 53 | - run: coverage run --source=dfetch --append -m behave features # Run features tests 54 | - run: coverage xml -o coverage.xml # Create XML report 55 | - run: pyroma --directory --min=10 . # Check pyproject 56 | - run: find dfetch -name "*.py" | xargs pyupgrade --py39-plus # Check syntax 57 | 58 | - name: Run codacy-coverage-reporter 59 | uses: codacy/codacy-coverage-reporter-action@a38818475bb21847788496e9f0fddaa4e84955ba # master 60 | with: 61 | project-token: ${{ secrets.CODACY_PROJECT_TOKEN }} 62 | coverage-reports: coverage.xml 63 | env: 64 | CODACY_PROJECT_TOKEN: ${{ secrets.CODACY_PROJECT_TOKEN }} 65 | if: "${{ (!!env.CODACY_PROJECT_TOKEN) }}" 66 | -------------------------------------------------------------------------------- /doc/manual.rst: -------------------------------------------------------------------------------- 1 | .. Dfetch documentation master file 2 | 3 | Manual 4 | ====== 5 | 6 | Init 7 | ----- 8 | .. argparse:: 9 | :module: dfetch.__main__ 10 | :func: create_parser 11 | :prog: dfetch 12 | :path: init 13 | 14 | .. asciinema:: asciicasts/init.cast 15 | 16 | .. automodule:: dfetch.commands.init 17 | 18 | Check 19 | ----- 20 | .. argparse:: 21 | :module: dfetch.__main__ 22 | :func: create_parser 23 | :prog: dfetch 24 | :path: check 25 | 26 | .. asciinema:: asciicasts/check.cast 27 | 28 | .. automodule:: dfetch.commands.check 29 | 30 | Reporting 31 | ````````` 32 | .. automodule:: dfetch.reporting.check.reporter 33 | 34 | Jenkins reporter 35 | '''''''''''''''' 36 | .. automodule:: dfetch.reporting.check.jenkins_reporter 37 | 38 | .. asciinema:: asciicasts/check-ci.cast 39 | 40 | Sarif reporter 41 | '''''''''''''' 42 | .. automodule:: dfetch.reporting.check.sarif_reporter 43 | 44 | Code-climate reporter 45 | ''''''''''''''''''''' 46 | .. automodule:: dfetch.reporting.check.code_climate_reporter 47 | 48 | Report 49 | ------ 50 | .. argparse:: 51 | :module: dfetch.__main__ 52 | :func: create_parser 53 | :prog: dfetch 54 | :path: report 55 | 56 | .. asciinema:: asciicasts/report.cast 57 | 58 | .. automodule:: dfetch.commands.report 59 | 60 | List (default) 61 | `````````````` 62 | .. automodule:: dfetch.reporting.stdout_reporter 63 | 64 | Software Bill-of-Materials 65 | `````````````````````````` 66 | .. automodule:: dfetch.reporting.sbom_reporter 67 | 68 | .. asciinema:: asciicasts/sbom.cast 69 | 70 | Update 71 | ------ 72 | .. argparse:: 73 | :module: dfetch.__main__ 74 | :func: create_parser 75 | :prog: dfetch 76 | :path: update 77 | 78 | .. asciinema:: asciicasts/update.cast 79 | 80 | .. automodule:: dfetch.commands.update 81 | 82 | Validate 83 | -------- 84 | .. argparse:: 85 | :module: dfetch.__main__ 86 | :func: create_parser 87 | :prog: dfetch 88 | :path: validate 89 | 90 | .. asciinema:: asciicasts/validate.cast 91 | 92 | .. automodule:: dfetch.commands.validate 93 | 94 | Diff 95 | ----- 96 | .. argparse:: 97 | :module: dfetch.__main__ 98 | :func: create_parser 99 | :prog: dfetch 100 | :path: diff 101 | 102 | .. asciinema:: asciicasts/diff.cast 103 | 104 | .. automodule:: dfetch.commands.diff 105 | 106 | Freeze 107 | ------ 108 | .. argparse:: 109 | :module: dfetch.__main__ 110 | :func: create_parser 111 | :prog: dfetch 112 | :path: freeze 113 | 114 | .. asciinema:: asciicasts/freeze.cast 115 | 116 | .. automodule:: dfetch.commands.freeze 117 | 118 | Environment 119 | ----------- 120 | .. argparse:: 121 | :module: dfetch.__main__ 122 | :func: create_parser 123 | :prog: dfetch 124 | :path: environment 125 | 126 | .. asciinema:: asciicasts/environment.cast 127 | 128 | .. automodule:: dfetch.commands.environment 129 | 130 | 131 | Import 132 | ------ 133 | .. argparse:: 134 | :module: dfetch.__main__ 135 | :func: create_parser 136 | :prog: dfetch 137 | :path: import 138 | 139 | .. asciinema:: asciicasts/import.cast 140 | 141 | .. automodule:: dfetch.commands.import_ 142 | -------------------------------------------------------------------------------- /doc/asciicasts/report.cast: -------------------------------------------------------------------------------- 1 | {"version": 2, "width": 128, "height": 31, "timestamp": 1760470426, "env": {"SHELL": "/bin/sh", "TERM": "xterm-256color"}} 2 | [0.100571, "o", "\u001b[H\u001b[2J\u001b[3J"] 3 | [0.103378, "o", "$ "] 4 | [1.10493, "o", "\u001b["] 5 | [1.285291, "o", "1m"] 6 | [1.37574, "o", "ls"] 7 | [1.465857, "o", " -"] 8 | [1.556061, "o", "l\u001b"] 9 | [1.64622, "o", "[0"] 10 | [1.736415, "o", "m"] 11 | [2.736969, "o", "\r\n"] 12 | [2.739698, "o", "total 12\r\n"] 13 | [2.739871, "o", "drwxr-xr-x+ 3 dev dev 4096 Oct 14 19:33 cpputest\r\n-rw-rw-rw- 1 dev dev 733 Oct 14 19:33 dfetch.yaml\r\ndrwxr-xr-x+ 4 dev dev 4096 Oct 14 19:33 jsmn\r\n"] 14 | [2.743518, "o", "$ "] 15 | [3.74509, "o", "\u001b"] 16 | [3.925364, "o", "[1"] 17 | [4.015494, "o", "md"] 18 | [4.105664, "o", "fe"] 19 | [4.195784, "o", "t"] 20 | [4.285923, "o", "ch"] 21 | [4.376085, "o", " r"] 22 | [4.466324, "o", "ep"] 23 | [4.556514, "o", "or"] 24 | [4.646622, "o", "t"] 25 | [4.826889, "o", "\u001b["] 26 | [4.917045, "o", "0m"] 27 | [5.917681, "o", "\r\n"] 28 | [6.345753, "o", "\u001b[1;38;5;4m\u001b[34mDfetch (0.10.0)\u001b[0m\r\n\u001b[0m"] 29 | [6.380575, "o", "\u001b[1;38;5;4m \u001b[32mproject :\u001b[34m cpputest\u001b[0m\r\n"] 30 | [6.380628, "o", "\u001b[0m\u001b[1;38;5;4m \u001b[32m remote :\u001b[34m github\u001b[0m\r\n\u001b[0m"] 31 | [6.381683, "o", "\u001b[1;38;5;4m \u001b[32m remote url :\u001b[34m https://github.com/cpputest/cpputest.git\u001b[0m\r\n\u001b[0m"] 32 | [6.381735, "o", "\u001b[1;38;5;4m \u001b[32m branch :\u001b[34m master\u001b[0m\r\n\u001b[0m"] 33 | [6.381846, "o", "\u001b[1;38;5;4m \u001b[32m tag :\u001b[34m v3.4\u001b[0m\r\n\u001b[0m\u001b[1;38;5;4m \u001b[32m last fetch :\u001b[34m 14/10/2025, 19:33:35\u001b[0m\r\n\u001b[0m"] 34 | [6.382171, "o", "\u001b[1;38;5;4m \u001b[32m revision :\u001b[34m \u001b[0m\r\n\u001b[0m\u001b[1;38;5;4m \u001b[32m patch :\u001b[34m \u001b[0m\r\n\u001b[0m\u001b[1;38;5;4m \u001b[32m licenses :\u001b[34m BSD 3-Clause \"New\" or \"Revised\" License\u001b[0m\r\n\u001b[0m"] 35 | [6.384749, "o", "\u001b[1;38;5;4m \u001b[32mproject :\u001b[34m jsmn\u001b[0m\r\n\u001b[0m"] 36 | [6.384825, "o", "\u001b[1;38;5;4m \u001b[32m remote :\u001b[34m github\u001b[0m\r\n\u001b[0m"] 37 | [6.385695, "o", "\u001b[1;38;5;4m \u001b[32m remote url :\u001b[34m https://github.com/zserge/jsmn.git\u001b[0m\r\n\u001b[0m"] 38 | [6.385782, "o", "\u001b[1;38;5;4m \u001b[32m branch :\u001b[34m master\u001b[0m\r\n\u001b[0m"] 39 | [6.385928, "o", "\u001b[1;38;5;4m \u001b[32m tag :\u001b[34m \u001b[0m\r\n\u001b[0m"] 40 | [6.386147, "o", "\u001b[1;38;5;4m \u001b[32m last fetch :\u001b[34m 14/10/2025, 19:33:36\u001b[0m\r\n\u001b[0m\u001b[1;38;5;4m \u001b[32m revision :\u001b[34m 25647e692c7906b96ffd2b05ca54c097948e879c\u001b[0m\r\n\u001b[0m\u001b[1;38;5;4m \u001b[32m patch :\u001b[34m \u001b[0m\r\n\u001b[0m"] 41 | [6.38617, "o", "\u001b[1;38;5;4m \u001b[32m licenses :\u001b[34m MIT License\u001b[0m\r\n\u001b[0m\u001b[0m"] 42 | [9.442338, "o", "/workspaces/dfetch/doc/generate-casts\r\n"] 43 | -------------------------------------------------------------------------------- /dfetch/commands/command.py: -------------------------------------------------------------------------------- 1 | """A generic command.""" 2 | 3 | import argparse 4 | import sys 5 | from abc import ABC, abstractmethod 6 | from argparse import ArgumentParser # pylint: disable=unused-import 7 | from typing import TYPE_CHECKING, TypeVar 8 | 9 | if TYPE_CHECKING and sys.version_info >= (3, 10): 10 | from typing import TypeAlias 11 | 12 | SubparserActionType: TypeAlias = ( 13 | "argparse._SubParsersAction[ArgumentParser]" # pyright: ignore[reportPrivateUsage] #pylint: disable=protected-access 14 | ) 15 | else: 16 | SubparserActionType = ( 17 | argparse._SubParsersAction # pyright: ignore[reportPrivateUsage] #pylint: disable=protected-access 18 | ) 19 | 20 | 21 | class Command(ABC): 22 | """An abstract command that dfetch can perform. 23 | 24 | When adding a new command to dfetch this class should be sub-classed. 25 | That subclass should implement: 26 | 27 | - ``create_menu`` which should add an appropriate subparser. 28 | Likely calling parser is enough. 29 | - ``__call__`` which will be called when the user selects the command. 30 | """ 31 | 32 | CHILD_TYPE = TypeVar("CHILD_TYPE", bound="Command") # noqa 33 | 34 | @staticmethod 35 | @abstractmethod 36 | def create_menu(subparsers: SubparserActionType) -> None: 37 | """Add a sub-parser to the given parser. 38 | 39 | Args: 40 | subparsers (argparse._SubParsersAction): subparser that the parser should be added to. 41 | 42 | This method must be implemented by a subclass. It is called when the menu structure is built. 43 | """ 44 | 45 | @abstractmethod 46 | def __call__(self, args: argparse.Namespace) -> None: 47 | """Perform the command. 48 | 49 | Args: 50 | args (argparse.Namespace): arguments as provided by the user. 51 | 52 | Raises: 53 | NotImplementedError: This is an abstract method that should be implemented by a subclass. 54 | """ 55 | 56 | @staticmethod 57 | def parser( 58 | subparsers: SubparserActionType, 59 | command: type["Command.CHILD_TYPE"], 60 | ) -> "argparse.ArgumentParser": 61 | """Generate the parser. 62 | 63 | The name of the class will be used as command. The class docstring will be split into 64 | the help text, description and epilog. 65 | 66 | Args: 67 | subparsers: The subparser to add the command to. 68 | command: The command class that should be instantiated and called when this command is called. 69 | 70 | Raises: 71 | NotImplementedError: If the child class doesn't have a docstring. 72 | 73 | Returns: 74 | Command: A argparse.ArgumentParser that can be used to add arguments. 75 | """ 76 | if not command.__doc__: 77 | raise NotImplementedError("Must add docstring to class") 78 | help_str, epilog = command.__doc__.split("\n", 1) 79 | 80 | parser = subparsers.add_parser( 81 | command.__name__.lower(), 82 | description=help_str, 83 | help=help_str, 84 | epilog=epilog, 85 | ) 86 | 87 | parser.set_defaults(func=command()) 88 | return parser 89 | -------------------------------------------------------------------------------- /dfetch/manifest/remote.py: -------------------------------------------------------------------------------- 1 | """Remotes are the external repository where the code should be retrieved from. 2 | 3 | The ``remotes:`` section is not mandatory. 4 | If only one remote is added this is assumed to be the default. 5 | If multiple remotes are listed ``default:`` can be explicitly specified. 6 | If multiple remotes are marked as default, the first marked as default is chosen. 7 | 8 | .. code-block:: yaml 9 | 10 | manifest: 11 | version: 0.0 12 | 13 | remotes: 14 | - name: mycompany-git-modules 15 | url-base: http://git.mycompany.local/mycompany/ 16 | default: true 17 | - name: github 18 | url-base: https://github.com/ 19 | """ 20 | 21 | from typing import Optional, Union 22 | 23 | from typing_extensions import TypedDict 24 | 25 | _MandatoryRemoteDict = TypedDict("_MandatoryRemoteDict", {"name": str, "url-base": str}) 26 | 27 | 28 | class RemoteDict(_MandatoryRemoteDict, total=False): 29 | """Class representing data types of Remote class construction.""" 30 | 31 | default: Optional[bool] 32 | 33 | 34 | class Remote: 35 | """A single remote entry in the manifest file.""" 36 | 37 | def __init__(self, kwargs: RemoteDict) -> None: 38 | """Create the remote entry.""" 39 | self._name: str = kwargs["name"] 40 | self._url_base: str = kwargs["url-base"] 41 | self._default: bool = bool(kwargs.get("default", False)) 42 | 43 | @classmethod 44 | def from_yaml(cls, yamldata: Union[dict[str, str], RemoteDict]) -> "Remote": 45 | """Create a remote entry in the manifest from yaml data. 46 | 47 | Returns: 48 | Remote: Entry containing the immutable remote entry 49 | """ 50 | return cls( 51 | { 52 | "name": yamldata["name"], 53 | "url-base": yamldata["url-base"], 54 | "default": bool(yamldata.get("default", False)), 55 | } 56 | ) 57 | 58 | @classmethod 59 | def copy(cls, other: "Remote") -> "Remote": 60 | """Generate a new remote entry in the manifest from another. 61 | 62 | Args: 63 | other (Remote): Other Remote to copy the values from 64 | 65 | Returns: 66 | Remote: Entry containing the immutable remote entry 67 | """ 68 | return cls( 69 | {"name": other.name, "url-base": other.url, "default": other.is_default} 70 | ) 71 | 72 | @property 73 | def name(self) -> str: 74 | """Get the name of the remote.""" 75 | return self._name 76 | 77 | @property 78 | def url(self) -> str: 79 | """Get the url of the remote.""" 80 | return self._url_base 81 | 82 | @property 83 | def is_default(self) -> bool: 84 | """Check if this is a default remote.""" 85 | return self._default 86 | 87 | def __repr__(self) -> str: 88 | """Get a string representation of this remote.""" 89 | return str(self.as_yaml()) 90 | 91 | def as_yaml(self) -> RemoteDict: 92 | """Get this remote as yaml data.""" 93 | yamldata: RemoteDict = {"name": self._name, "url-base": self._url_base} 94 | 95 | if self.is_default: 96 | yamldata["default"] = True 97 | 98 | return yamldata 99 | -------------------------------------------------------------------------------- /.github/workflows/python-publish.yml: -------------------------------------------------------------------------------- 1 | # This workflows will upload a Python Package using Twine when a release is created 2 | # For more information see: https://help.github.com/en/actions/language-and-framework-guides/using-python-with-github-actions#publishing-to-package-registries 3 | 4 | name: Upload Python Package 5 | 6 | on: 7 | release: 8 | types: [created] 9 | pull_request: 10 | types: [opened, synchronize, reopened] 11 | 12 | # Allows to run this workflow manually 13 | workflow_dispatch: 14 | 15 | permissions: 16 | contents: read 17 | 18 | jobs: 19 | build: 20 | name: Build distribution 📦 21 | runs-on: ubuntu-latest 22 | 23 | steps: 24 | - name: Harden the runner (Audit all outbound calls) 25 | uses: step-security/harden-runner@20cf305ff2072d973412fa9b1e3a4f227bda3c76 # v2.14.0 26 | with: 27 | egress-policy: audit 28 | 29 | - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v5.0.0 30 | with: 31 | persist-credentials: false 32 | fetch-depth: 0 # Fetches all history and tags 33 | - name: Set up Python 34 | uses: actions/setup-python@83679a892e2d95755f2dac6acb0bfd1e9ac5d548 # v6.1.0 35 | with: 36 | python-version: '3.x' 37 | - name: Install dependencies 38 | run: python -m pip install --upgrade pip build --user 39 | - name: Build a binary wheel and a source tarball 40 | run: python3 -m build 41 | - name: Store the distribution packages 42 | uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0 43 | with: 44 | name: python-package-distributions 45 | path: dist/ 46 | 47 | publish-to-testpypi: 48 | name: Publish Python distribution 📦 to TestPyPI 49 | needs: 50 | - build 51 | runs-on: ubuntu-latest 52 | 53 | environment: 54 | name: testpypi 55 | url: https://test.pypi.org/p/dfetch 56 | 57 | permissions: 58 | id-token: write 59 | 60 | steps: 61 | - name: Download all the dists 62 | uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v5 63 | with: 64 | name: python-package-distributions 65 | path: dist/ 66 | - name: Publish distribution 📦 to TestPyPI 67 | uses: pypa/gh-action-pypi-publish@3317ede93a4981d0fc490510c6fcf8bf0e92ed05 # v1 68 | with: 69 | repository-url: https://test.pypi.org/legacy/ 70 | skip-existing: true 71 | 72 | - name: Test install from TestPyPI 73 | run: | 74 | pip install --pre --index-url https://test.pypi.org/simple/ dfetch --extra-index-url https://pypi.org/simple --user 75 | dfetch --help 76 | 77 | deploy: 78 | if: github.event_name == 'release' 79 | runs-on: ubuntu-latest 80 | needs: 81 | - build 82 | environment: 83 | name: pypi 84 | url: https://pypi.org/p/dfetch 85 | permissions: 86 | id-token: write 87 | 88 | steps: 89 | - name: Download all the dists 90 | uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v5 91 | with: 92 | name: python-package-distributions 93 | path: dist/ 94 | - name: Publish distribution 📦 to PyPI 95 | uses: pypa/gh-action-pypi-publish@3317ede93a4981d0fc490510c6fcf8bf0e92ed05 # v1 96 | -------------------------------------------------------------------------------- /.github/workflows/codeql-analysis.yml: -------------------------------------------------------------------------------- 1 | # For most projects, this workflow file will not need changing; you simply need 2 | # to commit it to your repository. 3 | # 4 | # You may wish to alter this file to override the set of languages analyzed, 5 | # or to provide custom queries or build logic. 6 | # 7 | # ******** NOTE ******** 8 | # We have attempted to detect the languages in your repository. Please check 9 | # the `language` matrix defined below to confirm you have the correct set of 10 | # supported CodeQL languages. 11 | # ******** NOTE ******** 12 | 13 | name: "CodeQL" 14 | 15 | on: 16 | push: 17 | branches: [ main ] 18 | pull_request: 19 | # The branches below must be a subset of the branches above 20 | branches: [ main ] 21 | schedule: 22 | - cron: '24 22 * * 4' 23 | 24 | permissions: 25 | contents: read 26 | 27 | jobs: 28 | analyze: 29 | permissions: 30 | actions: read # for github/codeql-action/init to get workflow details 31 | contents: read # for actions/checkout to fetch code 32 | security-events: write # for github/codeql-action/autobuild to send a status report 33 | name: Analyze 34 | runs-on: ubuntu-latest 35 | 36 | strategy: 37 | fail-fast: false 38 | matrix: 39 | language: [ 'python' ] 40 | # CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python' ] 41 | # Learn more... 42 | # https://docs.github.com/en/github/finding-security-vulnerabilities-and-errors-in-your-code/configuring-code-scanning#overriding-automatic-language-detection 43 | 44 | steps: 45 | - name: Harden the runner (Audit all outbound calls) 46 | uses: step-security/harden-runner@20cf305ff2072d973412fa9b1e3a4f227bda3c76 # v2.14.0 47 | with: 48 | egress-policy: audit 49 | 50 | - name: Checkout repository 51 | uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v5.0.0 52 | 53 | # Initializes the CodeQL tools for scanning. 54 | - name: Initialize CodeQL 55 | uses: github/codeql-action/init@17783bfb99b07f70fae080b654aed0c514057477 # v3.30.7 56 | with: 57 | languages: ${{ matrix.language }} 58 | # If you wish to specify custom queries, you can do so here or in a config file. 59 | # By default, queries listed here will override any specified in a config file. 60 | # Prefix the list here with "+" to use these queries and those in the config file. 61 | # queries: ./path/to/local/query, your-org/your-repo/queries@main 62 | 63 | # Autobuild attempts to build any compiled languages (C/C++, C#, or Java). 64 | # If this step fails, then you should remove it and run the build manually (see below) 65 | - name: Autobuild 66 | uses: github/codeql-action/autobuild@17783bfb99b07f70fae080b654aed0c514057477 # v3.30.7 67 | 68 | # ℹ️ Command-line programs to run using the OS shell. 69 | # 📚 https://git.io/JvXDl 70 | 71 | # ✏️ If the Autobuild fails above, remove it and uncomment the following three lines 72 | # and modify them (or add more) to build your code if your project 73 | # uses a compiled language 74 | 75 | #- run: | 76 | # make bootstrap 77 | # make release 78 | 79 | - name: Perform CodeQL Analysis 80 | uses: github/codeql-action/analyze@17783bfb99b07f70fae080b654aed0c514057477 # v3.30.7 81 | -------------------------------------------------------------------------------- /dfetch/util/versions.py: -------------------------------------------------------------------------------- 1 | """Module for handling version information from strings.""" 2 | 3 | import re 4 | from collections import defaultdict 5 | from typing import Optional 6 | 7 | from semver.version import Version 8 | 9 | BASEVERSION = re.compile( 10 | r"""[vV]? 11 | (?P0|[1-9]\d*) 12 | (\. 13 | (?P0|[1-9]\d*) 14 | (\. 15 | (?P0|[1-9]\d*) 16 | )? 17 | )? 18 | """, 19 | re.VERBOSE, 20 | ) 21 | 22 | 23 | def coerce(version: str) -> tuple[str, Optional[Version], str]: 24 | """Convert an incomplete version string into a semver-compatible Version object. 25 | 26 | * Tries to detect a "basic" version string (``major.minor.patch``). 27 | * If not enough components can be found, missing components are 28 | set to zero to obtain a valid semver version. 29 | 30 | :param str version: the version string to convert 31 | :return: a tuple with a prefix string, a :class:`Version` instance (or ``None`` 32 | if it's not a version) and the rest of the string which doesn't 33 | belong to a basic version. 34 | :rtype: tuple(None, str | :class:`Version` | None, str) 35 | """ 36 | match = None if not version else BASEVERSION.search(version) 37 | if not match: 38 | return ("", None, version) 39 | 40 | ver = { 41 | key: 0 if value is None else int(value) 42 | for key, value in match.groupdict().items() 43 | } 44 | 45 | return ( 46 | match.string[: match.start()], 47 | Version(**ver), 48 | match.string[match.end() :], # noqa:E203 49 | ) 50 | 51 | 52 | def latest_tag_from_list(current_tag: str, available_tags: list[str]) -> str: 53 | """Based on the given tag string and list of tags, get the latest available.""" 54 | parsed_tags = _create_available_version_dict(available_tags) 55 | 56 | prefix, current_version, _ = coerce(current_tag) 57 | 58 | latest_string: str = current_tag 59 | 60 | if current_version: 61 | latest_tag: Version = current_version 62 | for tag in parsed_tags[prefix]: 63 | if tag[0] > latest_tag: 64 | latest_tag, latest_string = tag 65 | 66 | return latest_string 67 | 68 | 69 | def _create_available_version_dict( 70 | available_tags: list[str], 71 | ) -> dict[str, list[tuple[Version, str]]]: 72 | """Create a dictionary where each key is a prefix with a list of versions. 73 | 74 | Args: 75 | available_tags (List[str]): A list of available tags. 76 | 77 | Returns: 78 | Dict[str, List[Tuple[Version, str]]]: A dictionary mapping prefixes to lists of versions. 79 | 80 | Example: 81 | >>> available_tags = [ 82 | ... 'release/v1.2.3', 83 | ... 'release/v2.0.0' 84 | ... ] 85 | >>> dict(_create_available_version_dict(available_tags)) 86 | {'release/': [(Version(major=1, minor=2, patch=3, prerelease=None, build=None), 'release/v1.2.3'), 87 | (Version(major=2, minor=0, patch=0, prerelease=None, build=None), 'release/v2.0.0')]} 88 | """ 89 | parsed_tags: dict[str, list[tuple[Version, str]]] = defaultdict(list) 90 | for available_tag in available_tags: 91 | prefix, version, _ = coerce(available_tag) 92 | if version: 93 | parsed_tags[prefix] += [(version, available_tag)] 94 | return parsed_tags 95 | 96 | 97 | if __name__ == "__main__": 98 | import doctest 99 | 100 | doctest.testmod(optionflags=doctest.NORMALIZE_WHITESPACE) 101 | -------------------------------------------------------------------------------- /doc/_ext/scenario_directive.py: -------------------------------------------------------------------------------- 1 | """ 2 | This custom Sphinx directive dynamically includes scenarios from a Gherkin feature file. 3 | 4 | 1. Enable the directive in your Sphinx `conf.py`: 5 | 6 | ```python 7 | extensions = ["your_extension_folder.scenario_directive"] 8 | ``` 9 | 10 | Use it in an .rst file: 11 | 12 | ```rst 13 | .. scenario-include:: path/to/feature_file.feature 14 | :scenario: 15 | Scenario Title 1 16 | Scenario Title 2 17 | ``` 18 | 19 | If `:scenario:` is omitted, all scenarios in the feature file will be included. 20 | The directive automatically detects Scenario: and Scenario Outline: titles. 21 | 22 | """ 23 | 24 | import os 25 | import re 26 | from typing import Tuple 27 | 28 | from docutils import nodes 29 | from docutils.parsers.rst import Directive 30 | from docutils.statemachine import StringList 31 | 32 | 33 | class ScenarioIncludeDirective(Directive): 34 | """Custom directive to dynamically include scenarios from a Gherkin feature file.""" 35 | 36 | required_arguments = 1 # Only the feature file is required 37 | optional_arguments = 0 38 | final_argument_whitespace = False 39 | option_spec = { 40 | "scenario": str, 41 | } 42 | 43 | def list_of_scenarios(self, feature_file_path: str) -> Tuple[str]: 44 | """Parse the list of scenarios from the feature file""" 45 | env = self.state.document.settings.env 46 | feature_path = os.path.abspath(os.path.join(env.app.srcdir, feature_file_path)) 47 | if not os.path.exists(feature_path): 48 | raise self.error(f"Feature file not found: {feature_path}") 49 | 50 | with open(feature_path, encoding="utf-8") as f: 51 | scenarios = tuple( 52 | m[1] 53 | for m in re.findall( 54 | r"^\s*(Scenario(?: Outline)?):\s*(.+)$", f.read(), re.MULTILINE 55 | ) 56 | ) 57 | return scenarios 58 | 59 | def run(self): 60 | """Generate the same literalinclude block for every scenario.""" 61 | feature_file = self.arguments[0].strip() 62 | 63 | scenarios_available = self.list_of_scenarios(feature_file) 64 | 65 | scenario_titles = [ 66 | title.strip() 67 | for title in self.options.get("scenario", "").splitlines() 68 | if title.strip() 69 | ] or scenarios_available 70 | 71 | container = nodes.section() 72 | 73 | for scenario_title in scenario_titles: 74 | end_before = ( 75 | ":end-before: Scenario:" 76 | if scenario_title != scenarios_available[-1] 77 | else "" 78 | ) 79 | 80 | directive_rst = f""" 81 | .. details:: **Example**: {scenario_title} 82 | 83 | .. literalinclude:: {feature_file} 84 | :language: gherkin 85 | :caption: {feature_file} 86 | :force: 87 | :dedent: 88 | :start-after: Scenario: {scenario_title} 89 | {end_before} 90 | """ 91 | viewlist = StringList() 92 | for i, line in enumerate(directive_rst.splitlines()): 93 | viewlist.append(line, source=f"<{self.name} directive>", offset=i) 94 | 95 | self.state.nested_parse( 96 | viewlist, 97 | self.content_offset, 98 | container, 99 | ) 100 | 101 | return container.children 102 | 103 | 104 | def setup(app): 105 | """Setup the directive.""" 106 | app.add_directive("scenario-include", ScenarioIncludeDirective) 107 | -------------------------------------------------------------------------------- /doc/asciicasts/import.cast: -------------------------------------------------------------------------------- 1 | {"version": 2, "width": 128, "height": 31, "timestamp": 1760470492, "env": {"SHELL": "/bin/sh", "TERM": "xterm-256color"}} 2 | [0.009059, "o", "$ "] 3 | [1.010918, "o", "\u001b["] 4 | [1.194556, "o", "1m"] 5 | [1.284575, "o", "ls"] 6 | [1.380283, "o", " -"] 7 | [1.469452, "o", "l\u001b"] 8 | [1.559604, "o", "[0"] 9 | [1.649915, "o", "m"] 10 | [2.650294, "o", "\r\n"] 11 | [2.653428, "o", "total 580\r\n"] 12 | [2.653964, "o", "-rw-rw-rw- 1 dev dev 1381 Oct 14 19:34 appveyor.yml\r\n-rw-rw-rw- 1 dev dev 1137 Oct 14 19:34 CMakeLists.txt\r\n-rwxrwxrwx 1 dev dev 229 Oct 14 19:34 create_doc.sh\r\ndrwxrwxrwx+ 2 dev dev 4096 Oct 14 19:34 data\r\ndrwxrwxrwx+ 4 dev dev 4096 Oct 14 19:34 doc\r\ndrwxrwxrwx+ 4 dev dev 4096 Oct 14 19:34 docs\r\ndrwxrwxrwx+ 2 dev dev 4096 Oct 14 19:34 installer\r\ndrwxrwxrwx+ 4 dev dev 4096 Oct 14 19:34 libraries\r\n-rw-rw-rw- 1 dev dev 35147 Oct 14 19:34 LICENSE\r\n-rw-rw-rw- 1 dev dev 505101 Oct 14 19:34 modbusscope_demo.gif\r\n-rw-rw-rw- 1 dev dev 1796 Oct 14 19:34 README.md\r\ndrwxrwxrwx+ 5 dev dev 4096 Oct 14 19:34 resources\r\ndrwxrwxrwx+ 9 dev dev 4096 Oct 14 19:34 src\r\ndrwxrwxrwx+ 9 dev dev 4096 Oct 14 19:34 tests\r\n"] 13 | [2.657287, "o", "$ "] 14 | [3.658848, "o", "\u001b["] 15 | [3.839154, "o", "1m"] 16 | [3.929294, "o", "ca"] 17 | [4.019495, "o", "t "] 18 | [4.10961, "o", ".g"] 19 | [4.199838, "o", "it"] 20 | [4.29017, "o", "mo"] 21 | [4.380629, "o", "du"] 22 | [4.470736, "o", "le"] 23 | [4.560884, "o", "s\u001b"] 24 | [4.741184, "o", "[0m"] 25 | [5.741658, "o", "\r\n"] 26 | [5.743669, "o", "[submodule \"tests/googletest\"]\r\n\tpath = tests/googletest\r\n\turl = https://github.com/google/googletest.git\r\n[submodule \"libraries/muparser\"]\r\n\tpath = libraries/muparser\r\n\turl = https://github.com/beltoforion/muparser.git\r\n"] 27 | [5.747321, "o", "$ "] 28 | [6.74882, "o", "\u001b["] 29 | [6.929064, "o", "1m"] 30 | [7.019233, "o", "df"] 31 | [7.109347, "o", "et"] 32 | [7.199497, "o", "ch"] 33 | [7.28972, "o", " i"] 34 | [7.379971, "o", "mp"] 35 | [7.470086, "o", "or"] 36 | [7.56023, "o", "t\u001b"] 37 | [7.650392, "o", "[0"] 38 | [7.830715, "o", "m"] 39 | [8.831479, "o", "\r\n"] 40 | [9.267313, "o", "\u001b[1;38;5;4m\u001b[34mDfetch (0.10.0)\u001b[0m\r\n\u001b[0m"] 41 | [9.916438, "o", "\u001b[1;38;5;4mFound libraries/muparser\u001b[0m\r\n"] 42 | [9.916494, "o", "\u001b[0m\u001b[1;38;5;4mFound tests/googletest\u001b[0m\r\n\u001b[0m"] 43 | [9.918412, "o", "\u001b[1;38;5;4mCreated manifest (dfetch.yaml) in /workspaces/dfetch/doc/generate-casts/ModbusScope\u001b[0m\r\n"] 44 | [9.918549, "o", "\u001b[0m\u001b[0m"] 45 | [9.974899, "o", "$ "] 46 | [10.976406, "o", "\u001b["] 47 | [11.156678, "o", "1m"] 48 | [11.246826, "o", "ca"] 49 | [11.336953, "o", "t "] 50 | [11.427113, "o", "dfe"] 51 | [11.517408, "o", "tc"] 52 | [11.607605, "o", "h."] 53 | [11.697727, "o", "ya"] 54 | [11.787878, "o", "ml"] 55 | [11.878017, "o", "\u001b[0"] 56 | [12.058299, "o", "m"] 57 | [13.059017, "o", "\r\n"] 58 | [13.060949, "o", "manifest:\r\n version: '0.0'\r\n\r\n remotes:\r\n - name: github-com-google\r\n url-base: https://github.com/google\r\n\r\n - name: github-com-beltoforion\r\n url-base: https://github.com/beltoforion\r\n\r\n projects:\r\n - name: libraries/muparser\r\n revision: 207d5b77c05c9111ff51ab91082701221220c477\r\n remote: github-com-beltoforion\r\n tag: v2.3.2\r\n repo-path: muparser.git\r\n\r\n - name: tests/googletest\r\n revision: dcc92d0ab6c4ce022162a23566d44f673251eee4\r\n remote: github-com-google\r\n repo-path: googletest.git\r\n"] 59 | [16.065266, "o", "$ "] 60 | [16.066571, "o", "\u001b["] 61 | [16.246911, "o", "1m"] 62 | [16.337053, "o", "\u001b["] 63 | [16.427234, "o", "0m"] 64 | [16.427562, "o", "\r\n"] 65 | -------------------------------------------------------------------------------- /doc/asciicasts/freeze.cast: -------------------------------------------------------------------------------- 1 | {"version": 2, "width": 128, "height": 31, "timestamp": 1760470449, "env": {"SHELL": "/bin/sh", "TERM": "xterm-256color"}} 2 | [0.048565, "o", "\u001b[H\u001b[2J\u001b[3J"] 3 | [0.054345, "o", "$ "] 4 | [1.055812, "o", "\u001b["] 5 | [1.23617, "o", "1m"] 6 | [1.326312, "o", "ca"] 7 | [1.416504, "o", "t "] 8 | [1.50662, "o", "df"] 9 | [1.596759, "o", "et"] 10 | [1.686882, "o", "ch"] 11 | [1.777029, "o", ".y"] 12 | [1.867184, "o", "am"] 13 | [1.95734, "o", "l\u001b"] 14 | [2.13757, "o", "[0m"] 15 | [3.13821, "o", "\r\n"] 16 | [3.140136, "o", "manifest:\r\n version: 0.0 # DFetch Module syntax version\r\n\r\n remotes: # declare common sources in one place\r\n - name: github\r\n url-base: https://github.com/\r\n\r\n projects:\r\n - name: cpputest\r\n dst: cpputest/src/ # Destination of this project (relative to this file)\r\n repo-path: cpputest/cpputest.git # Use default github remote\r\n tag: v3.4 # tag\r\n\r\n - name: jsmn # without destination, defaults to project name\r\n repo-path: zserge/jsmn.git # only repo-path is enough\r\n"] 17 | [3.143679, "o", "$ "] 18 | [4.14551, "o", "\u001b"] 19 | [4.325823, "o", "[1"] 20 | [4.416486, "o", "md"] 21 | [4.506109, "o", "fe"] 22 | [4.596233, "o", "tc"] 23 | [4.686391, "o", "h "] 24 | [4.776516, "o", "fr"] 25 | [4.866655, "o", "ee"] 26 | [4.95692, "o", "ze"] 27 | [5.047131, "o", "\u001b["] 28 | [5.227393, "o", "0"] 29 | [5.317533, "o", "m"] 30 | [6.317998, "o", "\r\n"] 31 | [6.783419, "o", "\u001b[1;38;5;4m\u001b[34mDfetch (0.10.0)\u001b[0m\r\n\u001b[0m"] 32 | [6.800304, "o", "\u001b[1;38;5;4m \u001b[32mcpputest :\u001b[34m Already pinned in manifest on version v3.4\u001b[0m\r\n"] 33 | [6.800675, "o", "\u001b[0m"] 34 | [6.801477, "o", "\u001b[1;38;5;4m \u001b[32mjsmn :\u001b[34m Freezing on version master - 25647e692c7906b96ffd2b05ca54c097948e879c\u001b[0m\r\n\u001b[0m"] 35 | [6.802757, "o", "\u001b[1;38;5;4mUpdated manifest (dfetch.yaml) in /workspaces/dfetch/doc/generate-casts/freeze\u001b[0m\r\n\u001b[0m"] 36 | [6.802907, "o", "\u001b[0m"] 37 | [6.857288, "o", "$ "] 38 | [7.858834, "o", "\u001b["] 39 | [8.039093, "o", "1m"] 40 | [8.129242, "o", "ca"] 41 | [8.219358, "o", "t "] 42 | [8.309506, "o", "df"] 43 | [8.399701, "o", "et"] 44 | [8.489821, "o", "ch"] 45 | [8.57995, "o", ".y"] 46 | [8.670149, "o", "am"] 47 | [8.760325, "o", "l\u001b"] 48 | [8.940613, "o", "[0m"] 49 | [9.941196, "o", "\r\n"] 50 | [9.943202, "o", "manifest:\r\n version: '0.0'\r\n\r\n remotes:\r\n - name: github\r\n url-base: https://github.com/\r\n\r\n projects:\r\n - name: cpputest\r\n dst: cpputest/src/\r\n tag: v3.4\r\n repo-path: cpputest/cpputest.git\r\n\r\n - name: jsmn\r\n revision: 25647e692c7906b96ffd2b05ca54c097948e879c\r\n branch: master\r\n repo-path: zserge/jsmn.git\r\n"] 51 | [9.946811, "o", "$ "] 52 | [10.948401, "o", "\u001b["] 53 | [11.128722, "o", "1m"] 54 | [11.21886, "o", "ls"] 55 | [11.309008, "o", " -"] 56 | [11.399165, "o", "l ."] 57 | [11.489311, "o", "\u001b["] 58 | [11.579506, "o", "0m"] 59 | [12.580033, "o", "\r\n"] 60 | [12.63339, "o", "total 16\r\n"] 61 | [12.633528, "o", "drwxr-xr-x+ 3 dev dev 4096 Oct 14 19:34 cpputest\r\n-rw-rw-rw- 1 dev dev 317 Oct 14 19:34 dfetch.yaml\r\n-rw-rw-rw- 1 dev dev 733 Oct 14 19:34 dfetch.yaml.backup\r\ndrwxr-xr-x+ 4 dev dev 4096 Oct 14 19:34 jsmn\r\n"] 62 | [15.638096, "o", "$ "] 63 | [15.639459, "o", "\u001b["] 64 | [15.819851, "o", "1m"] 65 | [15.909964, "o", "\u001b["] 66 | [16.00011, "o", "0m"] 67 | [16.000573, "o", "\r\n"] 68 | [16.002565, "o", "/workspaces/dfetch/doc/generate-casts\r\n"] 69 | -------------------------------------------------------------------------------- /.github/workflows/run.yml: -------------------------------------------------------------------------------- 1 | name: Run 2 | 3 | on: 4 | push: 5 | branches: 6 | - main 7 | pull_request: 8 | types: [opened, synchronize, reopened] 9 | 10 | permissions: 11 | contents: read 12 | 13 | jobs: 14 | 15 | test-cygwin: 16 | runs-on: windows-latest 17 | permissions: 18 | contents: read 19 | security-events: write 20 | 21 | steps: 22 | - name: Harden the runner (Audit all outbound calls) 23 | uses: step-security/harden-runner@20cf305ff2072d973412fa9b1e3a4f227bda3c76 # v2.14.0 24 | with: 25 | egress-policy: audit 26 | 27 | - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v5.0.0 28 | 29 | - uses: cygwin/cygwin-install-action@b9bf9147075ee9811ac11beee9351eeb93e2f2fb # master 30 | 31 | - name: Install Subversion (SVN) on Windows 32 | run: | 33 | choco install svn -y 34 | $env:PATH = "C:\Program Files (x86)\Subversion\bin;$env:PATH" 35 | echo "C:\Program Files (x86)\Subversion\bin" >> $env:GITHUB_PATH 36 | svn --version # Verify installation 37 | 38 | - name: Install dfetch 39 | run: pip install . 40 | 41 | - run: dfetch environment 42 | - run: dfetch validate 43 | - run: dfetch check 44 | - run: dfetch update 45 | - run: dfetch update 46 | - run: dfetch report -t sbom 47 | - name: Dfetch SARIF Check 48 | uses: ./ 49 | with: 50 | working-directory: '.' 51 | 52 | - name: Run example 53 | working-directory: ./example 54 | env: 55 | CI: 'false' 56 | run: | 57 | dfetch update 58 | dfetch update 59 | dfetch report 60 | 61 | test: 62 | strategy: 63 | matrix: 64 | platform: [ubuntu-latest, macos-latest, windows-latest] 65 | python-version: ['3.9', '3.10', '3.11', '3.12', '3.13', '3.14'] 66 | runs-on: ${{ matrix.platform }} 67 | permissions: 68 | contents: read 69 | security-events: write 70 | 71 | steps: 72 | - name: Harden the runner (Audit all outbound calls) 73 | uses: step-security/harden-runner@20cf305ff2072d973412fa9b1e3a4f227bda3c76 # v2.14.0 74 | with: 75 | egress-policy: audit 76 | 77 | - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v5.0.0 78 | 79 | - name: Setup Python 80 | uses: actions/setup-python@83679a892e2d95755f2dac6acb0bfd1e9ac5d548 # v6.1.0 81 | with: 82 | python-version: ${{ matrix.python-version }} 83 | 84 | - name: Install Subversion (SVN) 85 | if: matrix.platform == 'ubuntu-latest' 86 | run: | 87 | sudo apt-get update 88 | sudo apt-get install -y subversion 89 | svn --version # Verify installation 90 | 91 | - name: Install Subversion (SVN) 92 | if: matrix.platform == 'macos-latest' 93 | run: | 94 | brew install svn 95 | svn --version # Verify installation 96 | 97 | - name: Install Subversion (SVN) 98 | if: matrix.platform == 'windows-latest' 99 | run: | 100 | choco install svn -y 101 | $env:PATH = "C:\Program Files (x86)\Subversion\bin;$env:PATH" 102 | echo "C:\Program Files (x86)\Subversion\bin" >> $env:GITHUB_PATH 103 | svn --version # Verify installation 104 | 105 | - name: Install dfetch 106 | run: pip install . 107 | 108 | - run: dfetch environment 109 | - run: dfetch validate 110 | - run: dfetch check 111 | - run: dfetch update 112 | - run: dfetch update 113 | - run: dfetch report -t sbom 114 | 115 | - name: Dfetch SARIF Check 116 | uses: ./ 117 | with: 118 | working-directory: '.' 119 | -------------------------------------------------------------------------------- /.github/workflows/build.yml: -------------------------------------------------------------------------------- 1 | name: Build 2 | 3 | on: 4 | push: 5 | branches: 6 | - main 7 | pull_request: 8 | types: [opened, synchronize, reopened] 9 | 10 | permissions: 11 | contents: read 12 | 13 | jobs: 14 | 15 | build: 16 | strategy: 17 | matrix: 18 | platform: [ubuntu-latest, macos-latest, windows-latest] 19 | runs-on: ${{ matrix.platform }} 20 | permissions: 21 | contents: read 22 | security-events: write 23 | 24 | steps: 25 | - name: Harden the runner (Audit all outbound calls) 26 | uses: step-security/harden-runner@20cf305ff2072d973412fa9b1e3a4f227bda3c76 # v2.14.0 27 | with: 28 | egress-policy: audit 29 | 30 | - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v5.0.0 31 | 32 | - name: Setup Python 33 | uses: actions/setup-python@83679a892e2d95755f2dac6acb0bfd1e9ac5d548 # v6.1.0 34 | with: 35 | python-version: '3.13' 36 | 37 | - name: ccache 38 | uses: hendrikmuhs/ccache-action@5ebbd400eff9e74630f759d94ddd7b6c26299639 # v1.2 39 | if: ${{ matrix.platform != 'windows-latest' }} 40 | with: 41 | key: ${{ github.job }}-${{ matrix.platform }} 42 | verbose: 1 43 | create-symlink: true 44 | 45 | - name: Setup cache for clcache 46 | if: ${{ matrix.platform == 'windows-latest' }} 47 | uses: actions/cache@9255dc7a253b0ccc959486e2bca901246202afeb # v5.0.1 48 | with: 49 | path: ${{ github.workspace }}\.clcache 50 | key: ${{ github.job }}-${{ matrix.platform }} 51 | 52 | - name: Create binary 53 | env: 54 | CCACHE_BASEDIR: ${{ github.workspace }} 55 | CCACHE_NOHASHDIR: true 56 | NUITKA_CACHE_DIR_CCACHE: ${{ github.workspace }}/.ccache 57 | NUITKA_CACHE_DIR_CLCACHE: ${{ github.workspace }}\.clcache 58 | NUITKA_CCACHE_BINARY: /usr/bin/ccache 59 | run: | 60 | pip install .[build] 61 | python script/build.py 62 | 63 | - name: Store the distribution packages 64 | uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0 65 | with: 66 | name: binary-distribution-${{ matrix.platform }} 67 | path: build/dfetch-* 68 | 69 | test-binary: 70 | name: test binary 71 | needs: 72 | - build 73 | strategy: 74 | matrix: 75 | platform: [ubuntu-latest, macos-latest, windows-latest] 76 | runs-on: ${{ matrix.platform }} 77 | 78 | steps: 79 | - name: Download the binary artifact 80 | uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v5 81 | with: 82 | name: binary-distribution-${{ matrix.platform }} 83 | path: . 84 | 85 | - name: Prepare binary 86 | if: matrix.platform == 'ubuntu-latest' 87 | run: | 88 | binary=$(ls dfetch-*-x86_64) 89 | ln -sf "$binary" dfetch 90 | chmod +x dfetch 91 | ls -la . 92 | shell: bash 93 | 94 | - name: Prepare binary 95 | if: matrix.platform == 'macos-latest' 96 | run: | 97 | binary=$(ls dfetch-*-osx) 98 | ln -sf "$binary" dfetch 99 | chmod +x dfetch 100 | ls -la . 101 | shell: bash 102 | 103 | - name: Prepare binary on Windows 104 | if: matrix.platform == 'windows-latest' 105 | run: | 106 | $binary = Get-ChildItem dfetch-*.exe | Select-Object -First 1 107 | Copy-Item $binary -Destination dfetch.exe -Force 108 | Get-ChildItem 109 | shell: pwsh 110 | 111 | - run: ./dfetch init 112 | - run: ./dfetch environment 113 | - run: ./dfetch validate 114 | - run: ./dfetch check 115 | - run: ./dfetch update 116 | - run: ./dfetch update 117 | - run: ./dfetch report -t sbom 118 | -------------------------------------------------------------------------------- /features/fetch-with-ignore-svn.feature: -------------------------------------------------------------------------------- 1 | Feature: Fetch with ignore in svn 2 | 3 | Sometimes you want to ignore files from a project 4 | These can be specified using the `ignore:` tag 5 | 6 | Background: 7 | Given a svn-server "SomeInterestingProject" with the files 8 | | path | 9 | | SomeFolder/SomeSubFolder/SomeFile.txt | 10 | | SomeFolder/SomeSubFolder/OtherFile.txt | 11 | | SomeFolder/SomeSubFolder/SomeFile.md | 12 | | SomeFolder/SomeOtherSubFolder/SomeFile.txt | 13 | | SomeFolder/SomeOtherSubFolder/OtherFile.txt | 14 | 15 | Scenario: A file pattern is fetched from a repo 16 | Given the manifest 'dfetch.yaml' in MyProject 17 | """ 18 | manifest: 19 | version: 0.0 20 | projects: 21 | - name: SomeInterestingProject 22 | url: some-remote-server/SomeInterestingProject 23 | src: SomeFolder/SomeSubFolder 24 | ignore: 25 | - OtherFile.txt 26 | """ 27 | When I run "dfetch update" 28 | Then the output shows 29 | """ 30 | Dfetch (0.10.0) 31 | SomeInterestingProject: Fetched trunk - 1 32 | """ 33 | Then 'MyProject' looks like: 34 | """ 35 | MyProject/ 36 | SomeInterestingProject/ 37 | .dfetch_data.yaml 38 | SomeFile.md 39 | SomeFile.txt 40 | dfetch.yaml 41 | """ 42 | 43 | Scenario: Combination of directories and a single file can be ignored 44 | Given the manifest 'dfetch.yaml' in MyProject 45 | """ 46 | manifest: 47 | version: 0.0 48 | projects: 49 | - name: SomeInterestingProject 50 | url: some-remote-server/SomeInterestingProject 51 | ignore: 52 | - SomeFolder/SomeOtherSubFolder 53 | - SomeFolder/SomeSubFolder/SomeFile.md 54 | """ 55 | When I run "dfetch update" 56 | Then the output shows 57 | """ 58 | Dfetch (0.10.0) 59 | SomeInterestingProject: Fetched trunk - 1 60 | """ 61 | Then 'MyProject' looks like: 62 | """ 63 | MyProject/ 64 | SomeInterestingProject/ 65 | .dfetch_data.yaml 66 | SomeFolder/ 67 | SomeSubFolder/ 68 | OtherFile.txt 69 | SomeFile.txt 70 | dfetch.yaml 71 | """ 72 | 73 | Scenario: Ignore overrides the file pattern match in src attribute 74 | Given the manifest 'dfetch.yaml' in MyProject 75 | """ 76 | manifest: 77 | version: 0.0 78 | projects: 79 | - name: SomeInterestingProject 80 | url: some-remote-server/SomeInterestingProject 81 | src: SomeFolder/SomeSubFolder/*.txt 82 | ignore: 83 | - /SomeNonExistingPath 84 | - SomeFile.* 85 | """ 86 | When I run "dfetch update" 87 | Then the output shows 88 | """ 89 | Dfetch (0.10.0) 90 | SomeInterestingProject: Fetched trunk - 1 91 | """ 92 | Then 'MyProject' looks like: 93 | """ 94 | MyProject/ 95 | SomeInterestingProject/ 96 | .dfetch_data.yaml 97 | OtherFile.txt 98 | dfetch.yaml 99 | """ 100 | -------------------------------------------------------------------------------- /.github/workflows/scorecard.yml: -------------------------------------------------------------------------------- 1 | # This workflow uses actions that are not certified by GitHub. They are provided 2 | # by a third-party and are governed by separate terms of service, privacy 3 | # policy, and support documentation. 4 | 5 | name: Scorecard supply-chain security 6 | on: 7 | # For Branch-Protection check. Only the default branch is supported. See 8 | # https://github.com/ossf/scorecard/blob/main/docs/checks.md#branch-protection 9 | branch_protection_rule: 10 | # To guarantee Maintained check is occasionally updated. See 11 | # https://github.com/ossf/scorecard/blob/main/docs/checks.md#maintained 12 | schedule: 13 | - cron: '27 17 * * 0' 14 | push: 15 | branches: [ "main" ] 16 | 17 | # Declare default permissions as read only. 18 | permissions: read-all 19 | 20 | jobs: 21 | analysis: 22 | name: Scorecard analysis 23 | runs-on: ubuntu-latest 24 | # `publish_results: true` only works when run from the default branch. conditional can be removed if disabled. 25 | if: github.event.repository.default_branch == github.ref_name || github.event_name == 'pull_request' 26 | permissions: 27 | # Needed to upload the results to code-scanning dashboard. 28 | security-events: write 29 | # Needed to publish results and get a badge (see publish_results below). 30 | id-token: write 31 | # Uncomment the permissions below if installing in a private repository. 32 | # contents: read 33 | # actions: read 34 | 35 | steps: 36 | - name: Harden the runner (Audit all outbound calls) 37 | uses: step-security/harden-runner@20cf305ff2072d973412fa9b1e3a4f227bda3c76 # v2.14.0 38 | with: 39 | egress-policy: audit 40 | 41 | - name: "Checkout code" 42 | uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v4.2.2 43 | with: 44 | persist-credentials: false 45 | 46 | - name: "Run analysis" 47 | uses: ossf/scorecard-action@4eaacf0543bb3f2c246792bd56e8cdeffafb205a # v2.4.3 48 | with: 49 | results_file: results.sarif 50 | results_format: sarif 51 | # (Optional) "write" PAT token. Uncomment the `repo_token` line below if: 52 | # - you want to enable the Branch-Protection check on a *public* repository, or 53 | # - you are installing Scorecard on a *private* repository 54 | # To create the PAT, follow the steps in https://github.com/ossf/scorecard-action?tab=readme-ov-file#authentication-with-fine-grained-pat-optional. 55 | # repo_token: ${{ secrets.SCORECARD_TOKEN }} 56 | 57 | # Public repositories: 58 | # - Publish results to OpenSSF REST API for easy access by consumers 59 | # - Allows the repository to include the Scorecard badge. 60 | # - See https://github.com/ossf/scorecard-action#publishing-results. 61 | # For private repositories: 62 | # - `publish_results` will always be set to `false`, regardless 63 | # of the value entered here. 64 | publish_results: true 65 | 66 | # (Optional) Uncomment file_mode if you have a .gitattributes with files marked export-ignore 67 | # file_mode: git 68 | 69 | # Upload the results as artifacts (optional). Commenting out will disable uploads of run results in SARIF 70 | # format to the repository Actions tab. 71 | - name: "Upload artifact" 72 | uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0 73 | with: 74 | name: SARIF file 75 | path: results.sarif 76 | retention-days: 5 77 | 78 | # Upload the results to GitHub's code scanning dashboard (optional). 79 | # Commenting out will disable upload of results to your repo's Code Scanning dashboard 80 | - name: "Upload to code-scanning" 81 | uses: github/codeql-action/upload-sarif@17783bfb99b07f70fae080b654aed0c514057477 # v3.30.7 82 | with: 83 | sarif_file: results.sarif 84 | -------------------------------------------------------------------------------- /features/list-projects.feature: -------------------------------------------------------------------------------- 1 | Feature: List dependencies 2 | 3 | The report command lists the current state of the projects. It will aggregate the metadata for each project 4 | and list it. This includes metadata from the manifest file and the metadata file (.dfetch_data.yaml) 5 | 6 | Scenario: Git projects are specified in the manifest 7 | Given the manifest 'dfetch.yaml' 8 | """ 9 | manifest: 10 | version: '0.0' 11 | 12 | remotes: 13 | - name: github-com-dfetch-org 14 | url-base: https://github.com/dfetch-org/test-repo 15 | 16 | projects: 17 | - name: ext/test-repo-tag 18 | url: https://github.com/dfetch-org/test-repo 19 | branch: main 20 | 21 | - name: ext/test-rev-and-branch 22 | tag: v1 23 | dst: ext/test-rev-and-branch 24 | 25 | """ 26 | And all projects are updated 27 | When I run "dfetch report" 28 | Then the output shows 29 | """ 30 | Dfetch (0.10.0) 31 | project : ext/test-repo-tag 32 | remote : 33 | remote url : https://github.com/dfetch-org/test-repo 34 | branch : main 35 | tag : 36 | last fetch : 02/07/2021, 20:25:56 37 | revision : e1fda19a57b873eb8e6ae37780594cbb77b70f1a 38 | patch : 39 | licenses : MIT License 40 | project : ext/test-rev-and-branch 41 | remote : github-com-dfetch-org 42 | remote url : https://github.com/dfetch-org/test-repo 43 | branch : main 44 | tag : v1 45 | last fetch : 02/07/2021, 20:25:56 46 | revision : 47 | patch : 48 | licenses : MIT License 49 | """ 50 | 51 | @remote-svn 52 | Scenario: SVN projects are specified in the manifest 53 | Given the manifest 'dfetch.yaml' 54 | """ 55 | manifest: 56 | version: '0.0' 57 | 58 | projects: 59 | - name: cutter-svn-tag 60 | url: svn://svn.code.sf.net/p/cutter/svn/cutter 61 | tag: 1.1.7 62 | vcs: svn 63 | src: acmacros 64 | 65 | """ 66 | And all projects are updated 67 | When I run "dfetch report" 68 | Then the output shows 69 | """ 70 | Dfetch (0.10.0) 71 | project : cutter-svn-tag 72 | remote : 73 | remote url : svn://svn.code.sf.net/p/cutter/svn/cutter 74 | branch : 75 | tag : 1.1.7 76 | last fetch : 29/12/2024, 20:09:21 77 | revision : 4007 78 | patch : 79 | licenses : 80 | """ 81 | 82 | Scenario: Git repo with applied patch 83 | Given MyProject with applied patch 'diff.patch' 84 | When I run "dfetch report" 85 | Then the output shows 86 | """ 87 | Dfetch (0.10.0) 88 | project : ext/test-repo-tag 89 | remote : github-com-dfetch-org 90 | remote url : https://github.com/dfetch-org/test-repo 91 | branch : main 92 | tag : v2.0 93 | last fetch : 02/07/2021, 20:25:56 94 | revision : 95 | patch : diff.patch 96 | licenses : MIT License 97 | """ 98 | -------------------------------------------------------------------------------- /features/fetch-with-ignore-git.feature: -------------------------------------------------------------------------------- 1 | Feature: Fetch with ignore in git 2 | 3 | Sometimes you want to ignore files from a project 4 | These can be specified using the `ignore:` tag 5 | 6 | Background: 7 | Given a git-repository "SomeInterestingProject.git" with the files 8 | | path | 9 | | SomeFolder/SomeSubFolder/SomeFile.txt | 10 | | SomeFolder/SomeSubFolder/OtherFile.txt | 11 | | SomeFolder/SomeSubFolder/SomeFile.md | 12 | | SomeFolder/SomeOtherSubFolder/SomeFile.txt | 13 | | SomeFolder/SomeOtherSubFolder/OtherFile.txt | 14 | 15 | Scenario: A file pattern is fetched from a repo 16 | Given the manifest 'dfetch.yaml' in MyProject 17 | """ 18 | manifest: 19 | version: 0.0 20 | projects: 21 | - name: SomeInterestingProject 22 | url: some-remote-server/SomeInterestingProject.git 23 | src: SomeFolder/SomeSubFolder 24 | ignore: 25 | - OtherFile.txt 26 | tag: v1 27 | """ 28 | When I run "dfetch update" 29 | Then the output shows 30 | """ 31 | Dfetch (0.10.0) 32 | SomeInterestingProject: Fetched v1 33 | """ 34 | Then 'MyProject' looks like: 35 | """ 36 | MyProject/ 37 | SomeInterestingProject/ 38 | .dfetch_data.yaml 39 | SomeFile.md 40 | SomeFile.txt 41 | dfetch.yaml 42 | """ 43 | 44 | Scenario: Combination of directories and a single file can be ignored 45 | Given the manifest 'dfetch.yaml' in MyProject 46 | """ 47 | manifest: 48 | version: 0.0 49 | projects: 50 | - name: SomeInterestingProject 51 | url: some-remote-server/SomeInterestingProject.git 52 | ignore: 53 | - SomeFolder/SomeOtherSubFolder 54 | - SomeFolder/SomeSubFolder/SomeFile.md 55 | tag: v1 56 | """ 57 | When I run "dfetch update" 58 | Then the output shows 59 | """ 60 | Dfetch (0.10.0) 61 | SomeInterestingProject: Fetched v1 62 | """ 63 | Then 'MyProject' looks like: 64 | """ 65 | MyProject/ 66 | SomeInterestingProject/ 67 | .dfetch_data.yaml 68 | SomeFolder/ 69 | SomeSubFolder/ 70 | OtherFile.txt 71 | SomeFile.txt 72 | dfetch.yaml 73 | """ 74 | 75 | Scenario: Ignore overrides the file pattern match in src attribute 76 | Given the manifest 'dfetch.yaml' in MyProject 77 | """ 78 | manifest: 79 | version: 0.0 80 | projects: 81 | - name: SomeInterestingProject 82 | url: some-remote-server/SomeInterestingProject.git 83 | src: SomeFolder/SomeSubFolder/*.txt 84 | ignore: 85 | - /SomeNonExistingPath 86 | - SomeFile.* 87 | tag: v1 88 | """ 89 | When I run "dfetch update" 90 | Then the output shows 91 | """ 92 | Dfetch (0.10.0) 93 | SomeInterestingProject: Fetched v1 94 | """ 95 | Then 'MyProject' looks like: 96 | """ 97 | MyProject/ 98 | SomeInterestingProject/ 99 | .dfetch_data.yaml 100 | OtherFile.txt 101 | dfetch.yaml 102 | """ 103 | --------------------------------------------------------------------------------