├── .flake8 ├── .github ├── CODEOWNERS ├── ISSUE_TEMPLATE │ ├── new-collaborator.md │ └── proposed-change.md ├── PULL_REQUEST_TEMPLATE.md └── workflows │ ├── documentation.yml │ ├── python-publish.yml │ └── python-test.yml ├── .gitignore ├── .pre-commit-config.yaml ├── CHANGES.md ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── CONTRIBUTORS.md ├── DCO1.1.txt ├── LICENSE ├── MANIFEST.in ├── Makefile ├── README.md ├── compliance ├── __init__.py ├── agent.py ├── check.py ├── config.py ├── controls.py ├── evidence.py ├── fetch.py ├── fix.py ├── locker.py ├── notify.py ├── report.py ├── runners.py ├── scripts │ ├── __init__.py │ └── compliance_cli.py ├── templates │ └── readme_toc.md.tmpl └── utils │ ├── __init__.py │ ├── credentials.py │ ├── data_parse.py │ ├── exceptions.py │ ├── fetch_local_commands │ ├── http.py │ ├── path.py │ ├── services │ ├── __init__.py │ ├── github.py │ └── pagerduty.py │ └── test.py ├── demo ├── README.md ├── at-logo.png ├── auditree_demo.json ├── controls.json ├── demo_examples │ ├── __init__.py │ ├── checks │ │ ├── __init__.py │ │ ├── test_common.py │ │ ├── test_github.py │ │ └── test_image_content.py │ ├── evidence │ │ ├── __init__.py │ │ └── utils.py │ ├── fetchers │ │ ├── __init__.py │ │ ├── fetch_auditree_logo.py │ │ ├── fetch_common.py │ │ └── fetch_github.py │ ├── requirements.txt │ └── templates │ │ ├── default.md.tmpl │ │ └── reports │ │ └── time │ │ └── api_versions.md.tmpl └── requirements.txt ├── doc-source ├── coding-standards.rst ├── conf.py ├── credentials-example.cfg ├── design-principles.rst ├── evidence-partitioning.rst ├── fixers.rst ├── index.rst ├── notifiers.rst ├── oscal.rst ├── quick-start.rst ├── report-builder.rst ├── running-on-tekton.rst ├── running-on-travis.rst └── verifying-signed-evidence.rst ├── setup.cfg ├── setup.py └── test ├── __init__.py ├── fixtures └── controls │ ├── original │ └── controls.json │ └── simplified │ └── controls.json └── t_compliance ├── __init__.py ├── t_agent ├── __init__.py └── test_agent.py ├── t_check ├── __init__.py └── test_base_check.py ├── t_controls ├── __init__.py └── test_controls.py ├── t_evidence ├── __init__.py ├── test_evidence.py ├── test_partitioning.py └── test_signing.py ├── t_fetch ├── __init__.py └── test_base_fetcher.py ├── t_fix ├── __init__.py └── test_fixer.py ├── t_locker ├── __init__.py └── test_locker.py ├── t_notify ├── __init__.py ├── test_base_notify.py ├── test_fd_notifier.py ├── test_gh_notifier.py ├── test_locker_notifier.py ├── test_md_notify.py ├── test_pd_notify.py └── test_slack_notifier.py ├── t_report ├── __init__.py └── test_report.py ├── t_utils ├── __init__.py └── test_data_parse.py └── t_workflow ├── __init__.py └── demo_case.xml /.flake8: -------------------------------------------------------------------------------- 1 | [flake8] 2 | max-line-length = 120 3 | exclude = 4 | .venv 5 | .build 6 | __pycache__ 7 | .idea 8 | build 9 | -------------------------------------------------------------------------------- /.github/CODEOWNERS: -------------------------------------------------------------------------------- 1 | # These owners will be the default owners for everything in 2 | # the repo. 3 | * @alfinkel @drsm79 @cletomartin @mattwhite @smithsz 4 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/new-collaborator.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: New Collaborator 3 | about: Request collaborator access 4 | --- 5 | 6 | I would like collaborator (write) access to this repository. 7 | 8 | - [ ] I have read the [contributing guidelines][CONTRIBUTING] 9 | - [ ] I understand the responsibilities of a collaborator are to: 10 | - help review contributions to the Auditree Framework 11 | - help make & test releases 12 | - help promote the project 13 | 14 | [CONTRIBUTING]: https://github.com/ComplianceAsCode/auditree-framework/blob/master/CONTRIBUTING.md 15 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/proposed-change.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Proposed Change 3 | about: Use this template to file defects, propose enhancements or new features. 4 | 5 | --- 6 | 7 | ## Overview 8 | 9 | _Provide a high level summary of the issue to be resolved. This is mandatory 10 | for issue creation._ 11 | 12 | ## Requirements 13 | 14 | _Provide a bulleted list of requirements that when implemented will define the 15 | closure of the issue. This level of detail may not be available at the time of 16 | issue creation and can be completed at a later time._ 17 | 18 | ## Approach 19 | 20 | _Provide a detailed approach to satisfy all of the requirements listed in the 21 | previous section. This level of detail may not be available at the time of 22 | issue creation and can be completed at a later time._ 23 | 24 | ## Security and Privacy 25 | 26 | _Provide the impact on security and privacy as it relates to the completion of 27 | this issue. This level of detail may not be available at the time of 28 | issue creation and can be completed at a later time. N/A if not applicable._ 29 | 30 | ## Test Plan 31 | 32 | _Provide the test process that will be followed to adequately verify that the 33 | approach above satisfies the requirements provided. This level of detail may 34 | not be available at the time of issue creation and can be completed at a later 35 | time._ 36 | -------------------------------------------------------------------------------- /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | - [ ] Tick to sign-off your agreement to the [Developer Certificate of Origin (DCO) 1.1](../blob/master/DCO1.1.txt) 2 | 3 | 4 | ## What 5 | 6 | _Provide an overview of the scope of the pull request._ 7 | 8 | ## Why 9 | 10 | _Provide the business justification for the work included in the pull request._ 11 | 12 | ## How 13 | 14 | _Provide a bulleted list of the changes included in the pull request._ 15 | 16 | ## Test 17 | 18 | _Provide a bulleted list of tests included in the pull request and/or tests 19 | performed to validate the work included in the pull request._ 20 | 21 | ## Context 22 | 23 | _Provide a bulleted list of GitHub issues, or any other references (mailing list discussion, etc...) that reviewers can reference 24 | for additional information regarding scope of the pull request._ 25 | -------------------------------------------------------------------------------- /.github/workflows/documentation.yml: -------------------------------------------------------------------------------- 1 | name: docs 2 | on: 3 | release: 4 | types: [created] 5 | jobs: 6 | build_documentation: 7 | name: Generate documentation 8 | 9 | runs-on: ubuntu-latest 10 | 11 | steps: 12 | - name: Check out 13 | uses: actions/checkout@v2 14 | with: 15 | path: source 16 | - name: Set up Python 17 | uses: actions/setup-python@v2 18 | with: 19 | python-version: '3.8' 20 | - name: Install dependencies 21 | working-directory: ./source 22 | run: | 23 | make install 24 | make develop 25 | - name: Prep pages branch 26 | working-directory: ./source 27 | run: | 28 | git worktree prune 29 | git fetch origin 30 | git branch -l 31 | git worktree add ../build gh-pages 32 | git worktree list 33 | - name: Build documentation 34 | working-directory: ./source 35 | run: | 36 | make docs DOC_TARGET=../build 37 | - name: Commit docs 38 | working-directory: build 39 | run: | 40 | git config user.name github-actions 41 | git config user.email github-actions@github.com 42 | git add * 43 | git commit -a -m 'Documentation update for release' --no-verify 44 | git push origin gh-pages 45 | -------------------------------------------------------------------------------- /.github/workflows/python-publish.yml: -------------------------------------------------------------------------------- 1 | # This workflows will upload a Python Package using Twine when a release is created 2 | # For more information see: https://help.github.com/en/actions/language-and-framework-guides/using-python-with-github-actions#publishing-to-package-registries 3 | 4 | name: PyPI upload 5 | 6 | on: 7 | release: 8 | types: [created] 9 | 10 | jobs: 11 | deploy: 12 | 13 | runs-on: ubuntu-latest 14 | 15 | steps: 16 | - uses: actions/checkout@v2 17 | - name: Set up Python 18 | uses: actions/setup-python@v2 19 | with: 20 | python-version: '3.8' 21 | - name: Install dependencies 22 | run: | 23 | make develop 24 | - name: Test & lint 25 | run: | 26 | git config --global user.email "you@example.com" 27 | git config --global user.name "A. Name" 28 | make code-lint 29 | make test 30 | 31 | - name: Build and publish 32 | env: 33 | TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }} 34 | TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }} 35 | run: | 36 | python setup.py sdist bdist_wheel 37 | twine upload dist/* 38 | -------------------------------------------------------------------------------- /.github/workflows/python-test.yml: -------------------------------------------------------------------------------- 1 | name: format | lint | security | test 2 | on: [push, pull_request] 3 | jobs: 4 | lint_unit_tests_coverage: 5 | name: Run code validation steps 6 | 7 | runs-on: ubuntu-latest 8 | 9 | steps: 10 | - uses: actions/checkout@v2 11 | - name: Set up Python 12 | uses: actions/setup-python@v2 13 | with: 14 | python-version: '3.8' 15 | - name: Install dependencies 16 | run: | 17 | make develop 18 | - name: Run formatter 19 | run: | 20 | make code-format 21 | - name: Run linter 22 | run: | 23 | make code-lint 24 | - name: Run security check 25 | run: | 26 | make code-security 27 | - name: Run unit tests with coverage 28 | run: | 29 | git config --global user.email "you@example.com" 30 | git config --global user.name "A. Name" 31 | make test 32 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # -*- mode: gitignore; -*- 2 | 3 | # Sphinx documentation 4 | doc 5 | doc-source/compliance.rst 6 | doc-source/compliance.*.rst 7 | doc-source/modules.rst 8 | 9 | ### Emacs 10 | 11 | *~ 12 | \#*\# 13 | /.emacs.desktop 14 | /.emacs.desktop.lock 15 | *.elc 16 | auto-save-list 17 | tramp 18 | .\#* 19 | 20 | # Org-mode 21 | .org-id-locations 22 | *_archive 23 | 24 | # flymake-mode 25 | *_flymake.* 26 | 27 | # eshell files 28 | /eshell/history 29 | /eshell/lastdir 30 | 31 | # elpa packages 32 | /elpa/ 33 | 34 | # reftex files 35 | *.rel 36 | 37 | # AUCTeX auto folder 38 | /auto/ 39 | 40 | # cask packages 41 | .cask/ 42 | dist/ 43 | 44 | # Flycheck 45 | flycheck_*.el 46 | 47 | # server auth directory 48 | /server/ 49 | 50 | # projectiles files 51 | .projectile 52 | 53 | # directory configuration 54 | .dir-locals.el 55 | 56 | 57 | ### Vim 58 | 59 | # Swap 60 | [._]*.s[a-v][a-z] 61 | [._]*.sw[a-p] 62 | [._]s[a-v][a-z] 63 | [._]sw[a-p] 64 | 65 | # Session 66 | Session.vim 67 | 68 | # Temporary 69 | .netrwhist 70 | # Auto-generated tag files 71 | tags 72 | # Byte-compiled / optimized / DLL files 73 | __pycache__/ 74 | *.py[cod] 75 | *$py.class 76 | 77 | # C extensions 78 | *.so 79 | 80 | # Distribution / packaging 81 | .Python 82 | build/ 83 | develop-eggs/ 84 | dist/ 85 | downloads/ 86 | eggs/ 87 | .eggs/ 88 | lib/ 89 | lib64/ 90 | parts/ 91 | sdist/ 92 | var/ 93 | wheels/ 94 | *.egg-info/ 95 | .installed.cfg 96 | *.egg 97 | MANIFEST 98 | 99 | # PyInstaller 100 | # Usually these files are written by a python script from a template 101 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 102 | *.manifest 103 | *.spec 104 | 105 | # Installer logs 106 | pip-log.txt 107 | pip-delete-this-directory.txt 108 | 109 | # Unit test / coverage reports 110 | htmlcov/ 111 | .tox/ 112 | .coverage 113 | .coverage.* 114 | .cache 115 | nosetests.xml 116 | coverage.xml 117 | *.cover 118 | .hypothesis/ 119 | 120 | # Translations 121 | *.mo 122 | *.pot 123 | 124 | # Django stuff: 125 | *.log 126 | .static_storage/ 127 | .media/ 128 | local_settings.py 129 | 130 | # Flask stuff: 131 | instance/ 132 | .webassets-cache 133 | 134 | # Scrapy stuff: 135 | .scrapy 136 | 137 | 138 | # PyBuilder 139 | target/ 140 | 141 | # Jupyter Notebook 142 | .ipynb_checkpoints 143 | 144 | # pyenv 145 | .python-version 146 | 147 | # celery beat schedule file 148 | celerybeat-schedule 149 | 150 | # SageMath parsed files 151 | *.sage.py 152 | 153 | # Environments 154 | .env 155 | .venv 156 | env/ 157 | venv/ 158 | ENV/ 159 | env.bak/ 160 | venv.bak/ 161 | 162 | # Spyder project settings 163 | .spyderproject 164 | .spyproject 165 | 166 | # Rope project settings 167 | .ropeproject 168 | 169 | # mkdocs documentation 170 | /site 171 | 172 | # mypy 173 | .mypy_cache/ 174 | 175 | # creds 176 | .pip.conf 177 | .pypirc 178 | 179 | # results 180 | results.json 181 | -------------------------------------------------------------------------------- /.pre-commit-config.yaml: -------------------------------------------------------------------------------- 1 | repos: 2 | - repo: https://github.com/pre-commit/pre-commit-hooks 3 | rev: v4.6.0 4 | hooks: 5 | - id: trailing-whitespace 6 | - id: check-yaml 7 | - id: fix-encoding-pragma 8 | args: ["--remove"] # Not needed on python3 9 | - repo: https://github.com/ambv/black 10 | rev: 24.4.2 11 | hooks: 12 | - id: black 13 | - repo: https://github.com/PyCQA/flake8 14 | rev: 7.1.0 15 | hooks: 16 | - id: flake8 17 | files: "^(compliance|test|demo)" 18 | - repo: https://github.com/PyCQA/bandit 19 | rev: 1.7.9 20 | hooks: 21 | - id: bandit 22 | args: [--recursive] 23 | files: "^(compliance|test)" 24 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Contributor Covenant Code of Conduct 2 | 3 | ## Our Pledge 4 | 5 | In the interest of fostering an open and welcoming environment, we as 6 | contributors and maintainers pledge to making participation in our project and 7 | our community a harassment-free experience for everyone, regardless of age, body 8 | size, disability, ethnicity, sex characteristics, gender identity and expression, 9 | level of experience, education, socio-economic status, nationality, personal 10 | appearance, race, religion, or sexual identity and orientation. 11 | 12 | ## Our Standards 13 | 14 | Examples of behavior that contributes to creating a positive environment 15 | include: 16 | 17 | * Using welcoming and inclusive language 18 | * Being respectful of differing viewpoints and experiences 19 | * Gracefully accepting constructive criticism 20 | * Focusing on what is best for the community 21 | * Showing empathy towards other community members 22 | 23 | Examples of unacceptable behavior by participants include: 24 | 25 | * The use of sexualized language or imagery and unwelcome sexual attention or 26 | advances 27 | * Trolling, insulting/derogatory comments, and personal or political attacks 28 | * Public or private harassment 29 | * Publishing others' private information, such as a physical or electronic 30 | address, without explicit permission 31 | * Other conduct which could reasonably be considered inappropriate in a 32 | professional setting 33 | 34 | ## Our Responsibilities 35 | 36 | Project maintainers are responsible for clarifying the standards of acceptable 37 | behavior and are expected to take appropriate and fair corrective action in 38 | response to any instances of unacceptable behavior. 39 | 40 | Project maintainers have the right and responsibility to remove, edit, or 41 | reject comments, commits, code, wiki edits, issues, and other contributions 42 | that are not aligned to this Code of Conduct, or to ban temporarily or 43 | permanently any contributor for other behaviors that they deem inappropriate, 44 | threatening, offensive, or harmful. 45 | 46 | ## Scope 47 | 48 | This Code of Conduct applies both within project spaces and in public spaces 49 | when an individual is representing the project or its community. Examples of 50 | representing a project or community include using an official project e-mail 51 | address, posting via an official social media account, or acting as an appointed 52 | representative at an online or offline event. Representation of a project may be 53 | further defined and clarified by project maintainers. 54 | 55 | ## Enforcement 56 | 57 | Instances of abusive, harassing, or otherwise unacceptable behavior may be 58 | reported by contacting the project team ([Al Finkelstein](https://github.com/alfinkel) 59 | or [Simon Metson](https://github.com/drsm79)). All 60 | complaints will be reviewed and investigated and will result in a response that 61 | is deemed necessary and appropriate to the circumstances. The project team is 62 | obligated to maintain confidentiality with regard to the reporter of an incident. 63 | Further details of specific enforcement policies may be posted separately. 64 | 65 | Project maintainers who do not follow or enforce the Code of Conduct in good 66 | faith may face temporary or permanent repercussions as determined by other 67 | members of the project's leadership. 68 | 69 | ## Attribution 70 | 71 | This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, 72 | available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html 73 | 74 | [homepage]: https://www.contributor-covenant.org 75 | 76 | For answers to common questions about this code of conduct, see 77 | https://www.contributor-covenant.org/faq 78 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing 2 | 3 | If you want to add to the framework, please familiarize yourself with the code & our [Coding Standards][]. Make a fork of the repository & file a Pull Request from your fork with the changes. You will need to click the checkbox in the template to show you agree to the [Developer Certificate of Origin](https://github.com/ComplianceAsCode/auditree-framework/blob/main/DCO1.1.txt). 4 | 5 | If you make **regular & substantial contributions** to Auditree, you may want to become a collaborator. This means you can approve pull requests (though not your own) & create releases of the tool. Please [file an issue][new collab] to request collaborator access. A collaborator supports the project, ensuring coding standards are met & best practices are followed in contributed code, cutting & documenting releases, promoting the project etc. 6 | 7 | ## Fetchers & checks 8 | 9 | If you would like to contribute checks, either add them via PR to [Arboretum][] or push to your own repository & let us know of its existence. 10 | 11 | There are some guidelines to follow when making a common fetcher or check: 12 | 13 | - Be sure to leverage the configuration JSON file used at runtime by the compliance-tool. 14 | 15 | - The sub-document structure to adhere to for a common module is one where the root of the sub-document is the org. Under the org there should be a name field which would refer to the organization’s name. Each common module configuration sub-document should also be under the org sub-document. For example: 16 | 17 | ``` 18 | { 19 | ... 20 | "org": { 21 | "name": "my-org-name", 22 | "check-and-fetch-baz": { ... }, 23 | "check-foo": { ... }, 24 | "fetch-bar": { ... }, 25 | ... 26 | }, 27 | ... 28 | } 29 | ``` 30 | 31 | - If you need to sub-class a basic evidence type (raw, derived, report) in order to provide helper methods that handle the evidence content, be sure not to use a decorator to reference that evidence. Instead use the compliance tool evidence module get_evidence_by_path function. 32 | 33 | - Additionally when dealing with any evidence in a common fetcher or check it would be best to use the compliance tool evidence module get_evidence_by_path function rather than adding the evidence to the evidence cache directly. 34 | 35 | - Be sure to provide a report template as this will be included in the common-compliance package and used for your common check(s). 36 | 37 | - Be sure to provide notifier methods that correspond to your checks when appropriate. 38 | 39 | - Happy coding. 40 | 41 | 42 | ## Code formatting and style 43 | 44 | Please ensure all code contributions are formatted by `black` and pass all `flake8` linter requirements. 45 | CI/CD will run `black` and `flake8` on all new commits and reject changes if there are failures. If you 46 | run `make develop` to setup and maintain your virtual environment then `black` and `flake8` will be executed 47 | automatically as part of all git commits. If you'd like to run things manually you can do so locally by using: 48 | 49 | ```shell 50 | make code-format 51 | make code-lint 52 | ``` 53 | 54 | ## Testing 55 | 56 | Please ensure all code contributions are covered by appropriate unit tests and that all tests run cleanly. 57 | CI/CD will run tests on all new commits and reject changes if there are failures. You should run the test 58 | suite locally by using: 59 | 60 | ```shell 61 | make test 62 | ``` 63 | 64 | ## Releases and change logs 65 | 66 | We follow [semantic versioning][semver] and [changelog standards][changelog] with 67 | the following addendum: 68 | 69 | - We set the main package `__init__.py` `version` for version tracking. 70 | - Our change log is CHANGES.md. 71 | - In addition to the _types of changes_ outlined in the 72 | [changelog standards][changelog] we also include a BREAKING _type of change_ to 73 | call out any change that may cause downstream execution disruption. 74 | - Change types are always capitalized and enclosed in square brackets. For 75 | example `[ADDED]`, `[CHANGED]`, etc. 76 | - Changes are in the form of complete sentences with appropriate punctuation. 77 | 78 | [semver]: https://semver.org/ 79 | [changelog]: https://keepachangelog.com/en/1.0.0/#how 80 | [Arboretum]: https://github.com/ComplianceAsCode/auditree-arboretum 81 | [Coding Standards]: https://complianceascode.github.io/auditree-framework/coding-standards.html 82 | [flake8]: https://gitlab.com/pycqa/flake8 83 | [new collab]: https://github.com/ComplianceAsCode/auditree-framework/issues/new?template=new-collaborator.md 84 | [black]: https://github.com/psf/black 85 | -------------------------------------------------------------------------------- /CONTRIBUTORS.md: -------------------------------------------------------------------------------- 1 | # Pre-OSS Contributors 2 | 3 | The following IBMers have all contributed to this code base: 4 | 5 | - Cleto Martin 6 | - Al Finkelstein 7 | - Harshdeep Harshdeep 8 | - Matt White 9 | - Simon Metson 10 | - Doug Chivers 11 | - Ignacio Diez 12 | - Timothy Kelsey 13 | - Zhong Shi Wang 14 | - Gino Cubeddu 15 | - Sef Mutari 16 | - Huan Wu 17 | - Emre Yildirim 18 | - Jagdish Kumar 19 | - Louis R Degenaro 20 | - Oliver Koeth 21 | - Vasileios Gkoumplias 22 | - Andrew Toolan 23 | 24 | Graham Thackrah was intrinsically involved in its design. 25 | -------------------------------------------------------------------------------- /DCO1.1.txt: -------------------------------------------------------------------------------- 1 | Developer Certificate of Origin 2 | Version 1.1 3 | 4 | Copyright (C) 2004, 2006 The Linux Foundation and its contributors. 5 | 1 Letterman Drive 6 | Suite D4700 7 | San Francisco, CA, 94129 8 | 9 | Everyone is permitted to copy and distribute verbatim copies of this 10 | license document, but changing it is not allowed. 11 | 12 | 13 | Developer's Certificate of Origin 1.1 14 | 15 | By making a contribution to this project, I certify that: 16 | 17 | (a) The contribution was created in whole or in part by me and I 18 | have the right to submit it under the open source license 19 | indicated in the file; or 20 | 21 | (b) The contribution is based upon previous work that, to the best 22 | of my knowledge, is covered under an appropriate open source 23 | license and I have the right under that license to submit that 24 | work with modifications, whether created in whole or in part 25 | by me, under the same open source license (unless I am 26 | permitted to submit under a different license), as indicated 27 | in the file; or 28 | 29 | (c) The contribution was provided directly to me by some other 30 | person who certified (a), (b) or (c) and I have not modified 31 | it. 32 | 33 | (d) I understand and agree that this project and the contribution 34 | are public and that a record of the contribution (including all 35 | personal information I submit with it, including my sign-off) is 36 | maintained indefinitely and may be redistributed consistent with 37 | this project or the open source license(s) involved. 38 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | graft compliance/templates 2 | include compliance/utils/fetch_local_commands 3 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | # -*- mode:makefile; coding:utf-8 -*- 2 | 3 | # Copyright (c) 2020 IBM Corp. All rights reserved. 4 | # 5 | # Licensed under the Apache License, Version 2.0 (the "License"); 6 | # you may not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | DOC_TARGET?=doc 18 | 19 | develop: 20 | pip install -q -e .[dev] --upgrade --upgrade-strategy eager 21 | pre-commit install 22 | 23 | update-pre-commit: 24 | pre-commit autoupdate 25 | 26 | install: 27 | pip install -q --upgrade pip setuptools 28 | pip install -q . --upgrade --upgrade-strategy eager 29 | 30 | uninstall: 31 | pip uninstall auditree-framework 32 | 33 | code-format: 34 | pre-commit run black --all-files 35 | 36 | code-lint: 37 | pre-commit run flake8 --all-files 38 | 39 | code-security: 40 | pre-commit run bandit --all-files 41 | 42 | test:: 43 | pytest --cov compliance test -v 44 | 45 | docs: 46 | # Build the API docs from the source code - overwrites those files, which are ignored by git 47 | sphinx-apidoc -o doc-source compliance 48 | sphinx-build doc-source $(DOC_TARGET) 49 | 50 | clean-docs: 51 | $(RM) -r $(DOC_TARGET) 52 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | [![OS Compatibility][platform-badge]](#prerequisites) 2 | [![Python Compatibility][python-badge]][python] 3 | [![pre-commit][pre-commit-badge]][pre-commit] 4 | [![Code validation](https://github.com/ComplianceAsCode/auditree-framework/workflows/format%20%7C%20lint%20%7C%20security%20%7C%20test/badge.svg)][lint-test] 5 | [![Upload Python Package](https://github.com/ComplianceAsCode/auditree-framework/workflows/PyPI%20upload/badge.svg)][pypi-upload] 6 | 7 | # auditree-framework 8 | 9 | Tool to run compliance control checks as unit tests and build up a body of evidence. 10 | 11 | This framework gives you the tools you need to create an auditable body of evidence, and is designed to be "DevSecOps" friendly. Collection & validation of evidence is modelled as python unit tests, evidence is stored & versioned in a git repository, notifications can be configured to send to Slack, create issues, contact PagerDuty, or just write files into git. The goal is to enable the digital transformation of compliance activities, and make these everyday operational tasks for the team managing the system. 12 | 13 | ## Installation 14 | 15 | ### Prerequisites 16 | 17 | - Supported for execution on OSX and LINUX. 18 | - Supported for execution with Python 3.6 and above. 19 | 20 | If you haven't already you need to generate a new ssh key for your Github account as per [this guide](https://help.github.com/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent/) 21 | 22 | ### Check out the code 23 | 24 | ```shell 25 | git clone git@github.com:ComplianceAsCode/auditree-framework 26 | cd auditree-framework 27 | ``` 28 | 29 | ### For users 30 | 31 | ```shell 32 | python3 -m venv venv 33 | . venv/bin/activate 34 | make install 35 | ``` 36 | 37 | ### For developers 38 | 39 | ```shell 40 | python3 -m venv venv 41 | . venv/bin/activate 42 | make develop 43 | ``` 44 | 45 | #### Code style and formatting 46 | 47 | This repository uses [black][black] for code formatting and [flake8][flake8] for code styling. It also 48 | uses [pre-commit][pre-commit] hooks that are integrated into the development process and the CI. When 49 | you run `make develop` you are ensuring that the pre-commit hooks are installed and updated to their 50 | latest versions for this repository. This ensures that all delivered code has been properly formatted 51 | and passes the linter rules. See the [pre-commit configuration file][pre-commit-config] for details on 52 | `black` and `flake8` configurations. 53 | 54 | Since `black` and `flake8` are installed as part of the `pre-commit` hooks, running `black` and `flake8` 55 | manually must be done through `pre-commit`. See examples below: 56 | 57 | ```shell 58 | make code-format 59 | make code-lint 60 | ``` 61 | 62 | ...will run `black` and `flake8` on the entire repo and is equivalent to: 63 | 64 | ```shell 65 | pre-commit run black --all-files 66 | pre-commit run flake8 --all-files 67 | ``` 68 | 69 | ...and when looking to limit execution to a subset of files do similar to: 70 | 71 | ```shell 72 | pre-commit run black --files compliance/* 73 | pre-commit run flake8 --files compliance/* 74 | ``` 75 | 76 | #### Unit tests 77 | 78 | To run the frameworks test suite, use: 79 | 80 | ```shell 81 | make test 82 | ``` 83 | 84 | #### Build Documentation 85 | 86 | Documentation sources live in `doc-source`, and are also auto-generated from the source codes doc strings. The auto-generated documentation (`compliance*rst, modules.rst`) is ignored by git & should not be modified directly - make changes in the python code. 87 | 88 | To build the documentation locally run: 89 | 90 | ```shell 91 | make docs 92 | ``` 93 | 94 | This will update the files in `doc` with the latest documentation. These files should not be modified by hand. 95 | 96 | ## Try it 97 | 98 | Successfully complete the steps below and you should be able to find your local 99 | evidence locker in your `$TMPDIR/compliance` folder. There you will find a `raw` 100 | folder that contains all of the raw evidence fetched by the fetchers found in the 101 | `demo/demo_examples/fetchers` folder along with a `reports` folder that contains 102 | the reports generated by the checks found in the `demo/demo_examples/checks` folder. 103 | 104 | - Create an empty [credentials][] file: 105 | 106 | ```shell 107 | $ touch ~/.credentials 108 | ``` 109 | 110 | - Set up your environment: 111 | 112 | ```shell 113 | cd demo 114 | python -m venv 115 | . ./venv/bin/activate 116 | pip install -r requirements.txt 117 | ``` 118 | 119 | - Run the fetchers: 120 | 121 | ```shell 122 | compliance --fetch --evidence local -C auditree_demo.json -v 123 | ``` 124 | 125 | - Run the checks: 126 | 127 | ```shell 128 | compliance --check demo.arboretum.accred,demo.custom.accred --evidence local -C auditree_demo.json -v 129 | ``` 130 | 131 | ## Contribute 132 | 133 | Help us to improve the Auditree framework. See [CONTRIBUTING][]. 134 | 135 | ## Ecosystem 136 | 137 | We are building a set of common fetchers/checks in [Arboretum](https://github.com/ComplianceAsCode/auditree-arboretum). If you have a library of checks, please let us know & we'll link here. 138 | 139 | We have a data gathering and reporting tool called [Harvest](https://github.com/ComplianceAsCode/auditree-harvest) which lets you process your evidence locker and generate reports over the data held. 140 | 141 | We have a tool called [Prune](https://github.com/ComplianceAsCode/auditree-prune) which lets you mark evidence as no longer being collected, in a suitably tracked manner. 142 | 143 | We have a tool called [Plant](https://github.com/ComplianceAsCode/auditree-plant) which lets you add evidence to evidence lockers without the use of fetchers or checks. 144 | 145 | [CONTRIBUTING]: https://github.com/ComplianceAsCode/auditree-framework/blob/master/CONTRIBUTING.md 146 | [credentials file]: https://github.com/ComplianceAsCode/auditree-framework/blob/master/doc/design-principles.rst#credentials 147 | [flake8]: https://gitlab.com/pycqa/flake8 148 | [platform-badge]: https://img.shields.io/badge/platform-osx%20|%20linux-orange.svg 149 | [pre-commit-badge]: https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white 150 | [pre-commit]: https://github.com/pre-commit/pre-commit 151 | [pre-commit-config]: https://github.com/ComplianceAsCode/auditree-framework/blob/master/.pre-commit-config.yaml 152 | [python-badge]: https://img.shields.io/badge/python-v3.6+-blue.svg 153 | [python]: https://www.python.org/downloads/ 154 | [quick start guide]: https://github.com/ComplianceAsCode/auditree-framework/blob/master/doc-source/quick-start.rst 155 | [black]: https://github.com/psf/black 156 | [lint-test]: https://github.com/ComplianceAsCode/auditree-framework/actions?query=workflow%3A%22format+%7C+lint+%7C+test%22 157 | [pypi-upload]: https://github.com/ComplianceAsCode/auditree-framework/actions?query=workflow%3A%22PyPI+upload%22 158 | [credentials]: https://complianceascode.github.io/auditree-framework/design-principles.html#credentials 159 | -------------------------------------------------------------------------------- /compliance/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020, 2022 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance automation package.""" 15 | 16 | __version__ = "3.0.1" 17 | -------------------------------------------------------------------------------- /compliance/agent.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2022 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance check automation module.""" 15 | 16 | import base64 17 | import hashlib 18 | from pathlib import PurePath 19 | 20 | from compliance.config import get_config 21 | 22 | from cryptography.exceptions import InvalidSignature 23 | from cryptography.hazmat.backends import default_backend 24 | from cryptography.hazmat.primitives import hashes, serialization 25 | from cryptography.hazmat.primitives.asymmetric import padding 26 | from cryptography.hazmat.primitives.asymmetric.utils import Prehashed 27 | 28 | 29 | class ComplianceAgent: 30 | """Compliance agent class.""" 31 | 32 | AGENTS_DIR = "agents" 33 | PUBLIC_KEYS_EVIDENCE_PATH = "raw/auditree/agent_public_keys.json" 34 | 35 | def __init__(self, name=None, use_agent_dir=True): 36 | """Construct and initialize the agent object.""" 37 | self._name = name 38 | self._private_key = self._public_key = None 39 | self._use_agent_dir = use_agent_dir 40 | 41 | @property 42 | def name(self): 43 | """Get agent name.""" 44 | return self._name 45 | 46 | @property 47 | def private_key(self): 48 | """Get agent private key.""" 49 | return self._private_key 50 | 51 | @private_key.setter 52 | def private_key(self, data_bytes): 53 | """ 54 | Set agent private key. 55 | 56 | :param data_bytes: The PEM encoded key data as `bytes`. 57 | """ 58 | self._private_key = serialization.load_pem_private_key( 59 | data_bytes, None, default_backend() 60 | ) 61 | 62 | @property 63 | def public_key(self): 64 | """Get agent public key.""" 65 | return self._public_key 66 | 67 | @public_key.setter 68 | def public_key(self, data_bytes): 69 | """ 70 | Set agent public key. 71 | 72 | :param data_bytes: The PEM encoded key data as `bytes`. 73 | """ 74 | if self.name: 75 | self._public_key = serialization.load_pem_public_key(data_bytes) 76 | 77 | def get_path(self, path): 78 | """ 79 | Get the full evidence path. 80 | 81 | :param path: The relative evidence path as a string. 82 | 83 | :returns: The full evidence path as a string. 84 | """ 85 | if self.name and self._use_agent_dir: 86 | if PurePath(path).parts[0] != self.AGENTS_DIR: 87 | return str(PurePath(self.AGENTS_DIR, self.name, path)) 88 | return path 89 | 90 | def signable(self): 91 | """Determine if the agent can sign evidence.""" 92 | return all([self.name, self.private_key]) 93 | 94 | def verifiable(self): 95 | """Determine if the agent can verify evidence.""" 96 | return all([self.name, self.public_key]) 97 | 98 | def load_public_key_from_locker(self, locker): 99 | """ 100 | Load agent public key from locker. 101 | 102 | :param locker: A locker of type :class:`compliance.locker.Locker`. 103 | """ 104 | if not self.name: 105 | return 106 | try: 107 | public_keys = locker.get_evidence(self.PUBLIC_KEYS_EVIDENCE_PATH) 108 | public_key_str = public_keys.content_as_json[self.name] 109 | self.public_key = public_key_str.encode() 110 | except Exception: 111 | self._public_key = None # Missing public key evidence. 112 | 113 | def hash_and_sign(self, data_bytes): 114 | """ 115 | Hash and sign evidence using the agent private key. 116 | 117 | :param data_bytes: The data to sign as `bytes`. 118 | 119 | :returns: A `tuple` containing the hexadecimal digest string and the 120 | base64 encoded signature string. Returns tuple `(None, None)` if the 121 | agent is not configured to sign evidence. 122 | """ 123 | if not self.signable(): 124 | return None, None 125 | hashed = hashlib.sha256(data_bytes) 126 | signature = self.private_key.sign( 127 | hashed.digest(), 128 | padding.PSS( 129 | mgf=padding.MGF1(hashes.SHA256()), salt_length=padding.PSS.MAX_LENGTH 130 | ), 131 | Prehashed(hashes.SHA256()), 132 | ) 133 | return hashed.hexdigest(), base64.b64encode(signature).decode() 134 | 135 | def verify(self, data_bytes, signature_b64): 136 | """ 137 | Verify evidence using the agent public key. 138 | 139 | :param data_bytes: The data to verify as `bytes`. 140 | :param signature_b64: The base64 encoded signature string. 141 | 142 | :returns: `True` if data can be verified, else `False`. 143 | """ 144 | if not self.verifiable(): 145 | return False 146 | try: 147 | self.public_key.verify( 148 | base64.b64decode(signature_b64), 149 | data_bytes, 150 | padding.PSS( 151 | mgf=padding.MGF1(hashes.SHA256()), 152 | salt_length=padding.PSS.MAX_LENGTH, 153 | ), 154 | hashes.SHA256(), 155 | ) 156 | return True 157 | except InvalidSignature: 158 | return False 159 | 160 | @classmethod 161 | def from_config(cls): 162 | """Load agent from configuration.""" 163 | config = get_config() 164 | agent = cls( 165 | name=config.get("agent_name"), 166 | use_agent_dir=config.get("use_agent_dir", True), 167 | ) 168 | private_key_path = config.get("agent_private_key") 169 | public_key_path = config.get("agent_public_key") 170 | if private_key_path: 171 | with open(private_key_path, "rb") as key_file: 172 | agent.private_key = key_file.read() 173 | agent.public_key = agent.private_key.public_key().public_bytes( 174 | encoding=serialization.Encoding.PEM, 175 | format=serialization.PublicFormat.SubjectPublicKeyInfo, 176 | ) 177 | elif public_key_path: 178 | with open(public_key_path, "rb") as key_file: 179 | agent.public_key = key_file.read() 180 | return agent 181 | -------------------------------------------------------------------------------- /compliance/config.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance configuration automation module.""" 15 | 16 | import inspect 17 | import json 18 | from collections import OrderedDict 19 | from copy import deepcopy 20 | from pathlib import Path 21 | 22 | from compliance.utils.credentials import Config 23 | 24 | 25 | class ComplianceConfig(object): 26 | """ 27 | The configuration for a compliance run. 28 | 29 | Credentials and a cache of known evidences are included. 30 | """ 31 | 32 | DEFAULTS = { 33 | "locker": { 34 | "dirname": "compliance", 35 | "repo_url": "https://github.com/YOUR_ORG/YOUR_PROJECT", 36 | }, 37 | "runbooks": {"enabled": False, "base_url": "https://example.com/runbooks"}, 38 | "notify": { 39 | # Slack channel to use for an accreditation 40 | # E.g. {"mycompany.soc2": ["#compliance"]} 41 | "slack": {}, 42 | # GH repo to use for an accreditation 43 | # E.g. {"mycompany.soc2": {"repo": ["my-org/accr1-repo"]}} 44 | "gh_issues": {}, 45 | # GHE repo to use for an accreditation 46 | # Deprecated (use gh_issues), included for backward compatibility. 47 | # E.g. {"mycompany.soc2": {"repo": ["my-org/accr1-repo"]}} 48 | "ghe_issues": {}, 49 | # Pagerduty service id to use for an accreditation 50 | # E.g. {"mycompany.soc2": "ABCDEFG"} 51 | "pagerduty": {}, 52 | }, 53 | "org": {"name": "YOUR_ORG", "settings": {}}, 54 | } 55 | 56 | def __init__(self): 57 | """Construct and initialize the configuration object.""" 58 | self._config = {} 59 | self._creds = None 60 | self._evidence_cache = OrderedDict() 61 | self._org = None 62 | self.creds_path = None 63 | self.dependency_rerun = False 64 | 65 | @property 66 | def creds(self): 67 | """Credentials used for locker management and running fetchers.""" 68 | if not self._creds: 69 | path = None if self.creds_path is None else str(self.creds_path) 70 | self._creds = Config(path) 71 | return self._creds 72 | 73 | @property 74 | def evidences(self): 75 | """All evidence objects currently in the evidence cache.""" 76 | return self._evidence_cache.values() 77 | 78 | @property 79 | def raw_config(self): 80 | """Raw configuration settings as a dictionary.""" 81 | return self._config 82 | 83 | def load(self, config_file=None): 84 | """ 85 | Load configuration from a JSON file. 86 | 87 | :param config_file: the path to the JSON config file. 88 | If ``None``, the ``DEFAULT`` configuration is used. 89 | """ 90 | if config_file is None: 91 | self._config = self.DEFAULTS.copy() 92 | return 93 | try: 94 | self._config = json.loads(Path(config_file).read_text()) 95 | except ValueError as err: 96 | err.args += (config_file,) 97 | raise 98 | 99 | def get(self, config_path, default=None): 100 | """ 101 | Provide the configuration value for the supplied ``config_path``. 102 | 103 | Returns the default if ``config_path`` cannot be retrieved. 104 | 105 | :param config_path: dot notation path with the following format 106 | ``'key[.subkey]``. For instance, ``locker.dirname``. 107 | """ 108 | chunks = config_path.split(".") 109 | value = self._config 110 | for c in chunks: 111 | if value is None: 112 | break 113 | value = value.get(c) 114 | if value is None: 115 | value = self.DEFAULTS 116 | for c in chunks: 117 | if value is None: 118 | break 119 | value = value.get(c) 120 | return deepcopy(value) if value is not None else deepcopy(default) 121 | 122 | def get_evidence(self, evidence_path): 123 | """ 124 | Provide an evidence object from the evidence cache. 125 | 126 | :param path: the path to the evidence within the Locker. 127 | For example, ``raw/source1/evidence.json`` 128 | """ 129 | return self._evidence_cache.get(evidence_path) 130 | 131 | def add_evidences(self, evidence_list): 132 | """ 133 | Add a list of evidence objects to the evidence cache. 134 | 135 | :param evidence_list: a list of evidence objects. 136 | """ 137 | for e in evidence_list: 138 | if not self.dependency_rerun and e.path in self._evidence_cache.keys(): 139 | raise ValueError(f"Evidence {e.path} duplicated") 140 | self._evidence_cache[e.path] = e 141 | 142 | def get_template_dir(self, test_obj=None): 143 | """ 144 | Provide absolute path to the template directory for the test object. 145 | 146 | The associated path will be the first directory found named 147 | ``templates`` in the test object absolute path traversed in reverse. 148 | If ``test_obj`` is ``None``, then current directory ``'.'`` is 149 | assumed as initial path. 150 | 151 | :param test_obj: a :class:`compliance.ComplianceTest` object from 152 | where the template directory search will start from. 153 | """ 154 | paths = [Path().resolve()] + list(Path().resolve().parents) 155 | if test_obj is not None: 156 | paths = list(Path(inspect.getfile(test_obj.__class__)).parents) 157 | for path in paths[:-1]: 158 | templates = Path(path, "templates") 159 | if templates.is_dir(): 160 | return str(templates) 161 | 162 | 163 | __config = None 164 | 165 | 166 | def get_config(): 167 | """Provide the global configuration object.""" 168 | global __config 169 | if __config is None: 170 | __config = ComplianceConfig() 171 | return __config 172 | -------------------------------------------------------------------------------- /compliance/controls.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance check controls automation module.""" 15 | 16 | import copy 17 | import itertools 18 | import json 19 | from collections import defaultdict 20 | from pathlib import Path 21 | 22 | 23 | class ControlDescriptor(object): 24 | """ 25 | Class abstraction for controls.json content. 26 | 27 | Used when processing controls.json content. 28 | """ 29 | 30 | def __init__(self, dirs=None): 31 | """Construct and initialize the ControlDescriptor object.""" 32 | self._controls = {} 33 | self._paths = [] 34 | for d in dirs or ["."]: 35 | json_file = Path(d, "controls.json").resolve() 36 | if not json_file.is_file(): 37 | continue 38 | self._controls.update(json.loads(json_file.read_text())) 39 | self._paths.append(str(json_file)) 40 | 41 | @property 42 | def paths(self): 43 | """All absolute paths to ``controls.json`` file(s).""" 44 | return self._paths 45 | 46 | @property 47 | def as_dict(self): 48 | """Provide control descriptor content as a modifiable dictionary.""" 49 | return copy.deepcopy(self._controls) 50 | 51 | @property 52 | def accred_checks(self): 53 | """Provide all checks by accreditation (key) as a dictionary.""" 54 | if not hasattr(self, "_accred_checks"): 55 | self._accred_checks = defaultdict(set) 56 | for check in self._controls.keys(): 57 | accreds = self.get_accreditations(check) 58 | for accred in accreds: 59 | self._accred_checks[accred].add(check) 60 | return self._accred_checks 61 | 62 | def get_accreditations(self, test_path): 63 | """ 64 | Provide the accreditation list for a given test_path. 65 | 66 | This works for original and simplified controls formats. 67 | 68 | Original: {'test_path': {'key': {'sub-key: ['accred1', 'accred2']}}} 69 | Simplified: {'test_path': ['accred1', 'accred2']} 70 | 71 | :param test_path: the Python path to the test. 72 | For example: ``package.category.checks.module.TestClass`` 73 | """ 74 | check_details = self._controls.get(test_path, {}) 75 | if isinstance(check_details, list): 76 | return set(check_details) 77 | accreditations = [itertools.chain(*c.values()) for c in check_details.values()] 78 | return set(itertools.chain(*accreditations)) 79 | 80 | def is_test_included(self, test_path, accreditations): 81 | """ 82 | Provide boolean result of whether a check is part of accreditations. 83 | 84 | :param test_path: the Python path to the test. For instance: 85 | ``package.accr1.TestClass`` or ``package.accr2.test_function`` 86 | :param accreditations: list of accreditations names where ``test_path`` 87 | may be included. 88 | """ 89 | current_accreditations = self.get_accreditations(test_path) 90 | matched = set(accreditations).intersection(set(current_accreditations)) 91 | return len(matched) != 0 92 | -------------------------------------------------------------------------------- /compliance/fetch.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2021, 2022 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance fetcher automation module.""" 15 | 16 | import os 17 | import shutil 18 | import tempfile 19 | import unittest 20 | from pathlib import Path 21 | from subprocess import check_output # nosec 22 | 23 | from compliance.config import get_config 24 | from compliance.utils.http import BaseSession 25 | 26 | import requests 27 | 28 | ROOT = os.path.dirname(os.path.abspath(__file__)) 29 | FETCH_TERMINAL_SCRIPT = f"{ROOT}/utils/fetch_local_commands" 30 | 31 | 32 | class ComplianceFetcher(unittest.TestCase): 33 | """Compliance fetcher automation TestCase class.""" 34 | 35 | @classmethod 36 | def tearDownClass(cls): 37 | """Perform clean up.""" 38 | if hasattr(cls, "_session"): 39 | cls._session.close() 40 | 41 | @classmethod 42 | def session(cls, url=None, creds=None, **headers): 43 | """ 44 | Provide a requests session object with User-Agent header. 45 | 46 | :param url: optional base URL for the session requests to use. A url 47 | argument triggers a new session object to be created whereas no url 48 | argument will return the current session object if one exists. 49 | :param creds: optional authentication credentials. 50 | :param headers: optional kwargs to add to session headers. 51 | 52 | :returns: a requests Session object. 53 | """ 54 | if url is not None and hasattr(cls, "_session"): 55 | cls._session.close() 56 | delattr(cls, "_session") 57 | if not hasattr(cls, "_session"): 58 | if url: 59 | cls._session = BaseSession(url) 60 | else: 61 | cls._session = requests.Session() 62 | if creds: 63 | cls._session.auth = creds 64 | cls._session.headers.update(headers) 65 | org = cls.config.raw_config.get("org", {}).get("name", "") 66 | ua = f'{org.lower().replace(" ", "-")}-compliance-checks' 67 | cls._session.headers.update({"User-Agent": ua}) 68 | return cls._session 69 | 70 | def __init__(self, *args, **kwargs): 71 | """Construct and initialize the fetcher test object.""" 72 | super(ComplianceFetcher, self).__init__(*args, **kwargs) 73 | if hasattr(ComplianceFetcher, "config"): 74 | self.config = ComplianceFetcher.config 75 | else: 76 | self.config = get_config() 77 | 78 | def fetchURL(self, url, params=None, creds=None): # noqa: N802 79 | """ 80 | Retrieve remote content through an HTTP GET request. 81 | 82 | Helper/Convenience method. 83 | 84 | :param url: the URL of the file. 85 | :param params: optional parameters to include in the GET request. 86 | :param creds: optional tuple with (user, password) to be used. 87 | 88 | :returns: the remote content. 89 | """ 90 | org = self.config.raw_config.get("org", {}).get("name", "") 91 | ua = f'{org.lower().replace(" ", "-")}-compliance-checks' 92 | response = requests.get( 93 | url, params=params, auth=creds, headers={"User-Agent": ua}, timeout=3600 94 | ) 95 | response.raise_for_status() 96 | return response.content 97 | 98 | def fetchCloudantDoc(self, db_url, params=None): # noqa: N802 99 | """ 100 | Retrieve a Cloudant document. 101 | 102 | Helper/Convenience method. 103 | 104 | :param db_url: the URL to the Cloudant doc to retrieve. 105 | :param params: optional parameters to include in the GET request. 106 | 107 | :returns: the Cloudant document content. 108 | """ 109 | creds = self.config.creds 110 | return self.fetchURL( 111 | db_url, 112 | params=params, 113 | creds=(creds["cloudant"].user, creds["cloudant"].password), 114 | ) 115 | 116 | def fetchLocalCommands( # noqa: N802 117 | self, 118 | commands, 119 | cwd=None, 120 | env=None, 121 | show_exit_status=False, 122 | show_timestamp=False, 123 | timeout=20, 124 | user=None, 125 | ): 126 | """ 127 | Retrieve the output from locally executed commands. 128 | 129 | Helper/Convenience method. 130 | 131 | :param commands: A `list` of string commands to be executed. 132 | :param cwd: Override the current working directory. Defaults to `None`. 133 | :param env: Override environment variables. Defaults to `None`. 134 | :param show_exit_status: Show the exit status for each command. 135 | Defaults to `False`. 136 | :param show_timestamp: Show a timestamp denoting when each command was 137 | executed. Defaults to `False`. 138 | :param timeout: Terminate after timeout seconds. Defaults to 20. 139 | :param user: Execute commands as specified user. Defaults to current 140 | user. 141 | 142 | :returns: Command output as a string. 143 | """ 144 | cmd = [FETCH_TERMINAL_SCRIPT] 145 | if user: 146 | cmd = ["sudo", "-u", user] + cmd 147 | if show_exit_status: 148 | cmd += ["-s"] 149 | if show_timestamp: 150 | cmd += ["-t"] 151 | if not cwd: 152 | cwd = os.path.expanduser("~") 153 | stdin = "\n".join(commands) + "\n" 154 | return check_output( # nosec: B603 The input command can be anything. 155 | cmd, cwd=cwd, env=env, input=stdin, timeout=timeout, universal_newlines=True 156 | ).rstrip() 157 | 158 | 159 | def fetch(url, name): 160 | """ 161 | Write content retrieved from provided url to a file. 162 | 163 | :param url: the URL to GET content from. 164 | :param name: the name of the file to write to in the TMPDIR. 165 | 166 | :returns: the path to the file. 167 | """ 168 | r = requests.get(url, timeout=3600) 169 | r.raise_for_status() 170 | path = Path(tempfile.gettempdir(), name) 171 | with path.open("wb") as f: 172 | r.raw.decode_content = True 173 | shutil.copyfileobj(r.raw, f) 174 | return path 175 | -------------------------------------------------------------------------------- /compliance/fix.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance fixer automation module.""" 15 | 16 | import sys 17 | 18 | from compliance.config import get_config 19 | from compliance.utils.test import parse_test_id 20 | 21 | 22 | class Fixer(object): 23 | """This class attempts to resolve check failures.""" 24 | 25 | def __init__(self, results, dry_run=True, out=sys.stdout): 26 | """ 27 | Construct and initialize the fixer object. 28 | 29 | :param results: dictionary of check results. 30 | :param dry_run: dictate whether to inform or perform actions to fix. 31 | :param out: where to write the output for dry-run messages. 32 | """ 33 | self._results = results 34 | self._dry_run = dry_run 35 | self._out = out 36 | self._creds = get_config().creds 37 | 38 | def fix(self): 39 | """ 40 | Perform all fix routines found in executed checks. 41 | 42 | Iterate through all compliance checks and looks for `fix_*` methods 43 | with corresponding `test_*` methods. These are executed, and results 44 | are recorded if fixes are made. 45 | 46 | Note: Instead of individual `fix_*` methods, you can instead define 47 | a single `fix_failures` method which handles all fixes for the class. 48 | """ 49 | if not self._results: 50 | return 51 | 52 | for test_id, test_desc in self._results.items(): 53 | if test_desc["status"] != "fail": 54 | continue 55 | 56 | test_obj = test_desc["test"].test 57 | method_name = parse_test_id(test_id)["method"] 58 | candidate = method_name.replace("test_", "fix_") 59 | 60 | if len(test_obj.tests) > 1 and hasattr(test_obj, candidate): 61 | getattr(test_obj, candidate)(self) 62 | elif hasattr(test_obj, "fix_failures"): 63 | test_obj.fix_failures(self) 64 | 65 | def execute_fix(self, test_obj, fct, args=None): 66 | """ 67 | Execute the fix routine for a given check test object. 68 | 69 | This method gets called by fix_* methods in checks. 70 | It gets passed a function (method) that must be called in order 71 | to fix one specific issue. The method's fix will either get 72 | executed, or if in dry-run mode, will print out the method's 73 | docstring, injecting any arguments if needed. The check's 74 | fixed_failure_count is incremented if the fix was successful. 75 | 76 | :param test_obj: instance of subclass of 77 | :py:class:`compliance.utils.check.ComplianceCheck` on which 78 | to execute the fix 79 | :param fct: a callback function that will actually perform 80 | the fix. this function will be passed a reference to this 81 | fixer, along with a reference to a 82 | compliance.utils.credentials.Config object. this 83 | callback will also need to have a docstring, which is 84 | what will be displayed in dry-run mode. the docstring 85 | will get formatted with the arguments passed to the fix 86 | function. 87 | :param args: dictionary of named arguments to pass to the 88 | fix function fct. 89 | """ 90 | args = args or {} 91 | if self._dry_run: 92 | self._out.write(f"DRY-RUN: {fct.__doc__.format(**args)}\n") 93 | else: 94 | success = fct(**dict(list(args.items()) + [("creds", self._creds)])) 95 | if success: 96 | test_obj.fixed_failure_count += 1 97 | -------------------------------------------------------------------------------- /compliance/scripts/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance automation command line interface package.""" 15 | -------------------------------------------------------------------------------- /compliance/scripts/compliance_cli.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance automation command line interface.""" 15 | 16 | from compliance import __version__ 17 | from compliance.notify import get_notifiers 18 | from compliance.runners import CheckMode, FetchMode 19 | 20 | from ilcli import Command 21 | 22 | 23 | class ComplianceCLI(Command): 24 | """The Compliance Framework CLI.""" 25 | 26 | name = "compliance" 27 | 28 | def _init_arguments(self): 29 | self.add_argument( 30 | "-V", 31 | "--version", 32 | help="Displays the auditree framework version.", 33 | action="version", 34 | version=f"Auditree Framework version v{__version__}", 35 | ) 36 | self.add_argument( 37 | "-v", 38 | "--verbose", 39 | help="Displays verbose output.", 40 | action="store_const", 41 | const=2, 42 | default=1, 43 | ) 44 | self.add_argument( 45 | "--fetch", 46 | help="Enables the fetch process.", 47 | action="store_true", 48 | default=False, 49 | ) 50 | self.add_argument( 51 | "--check", 52 | help=( 53 | "Enables the check process. " 54 | "Check groupings can be a comma separated list without spaces." 55 | ), 56 | metavar="chk.grp1,chk.grp2,...", 57 | nargs="?", 58 | const="", 59 | ) 60 | self.add_argument( 61 | "--evidence", 62 | help=("Defines the evidence storage mode. Defaults to %(default)s."), 63 | choices=["local", "no-push", "full-remote"], 64 | default="no-push", 65 | ) 66 | self.add_argument( 67 | "--fix", 68 | help="Attempts to fix check failures. Defaults to %(default)s.", 69 | choices=["off", "on", "dry-run"], 70 | default="off", 71 | ) 72 | self.add_argument( 73 | "-C", 74 | "--compliance-config", 75 | help="Specifies the path/name of the compliance config JSON file.", 76 | metavar="auditree.json", 77 | default=None, 78 | ) 79 | self.add_argument( 80 | "--creds-path", 81 | help=( 82 | "Specifies the path/name of the credentials ini file. " 83 | "Defaults to %(default)s." 84 | ), 85 | metavar="/path/to/creds.ini", 86 | default=None, 87 | ) 88 | notify_options = [k for k in get_notifiers().keys() if k != "stdout"] 89 | self.add_argument( 90 | "--notify", 91 | help=( 92 | "Specifies a list of notifiers for sending notifications. " 93 | "Valid values (can be a comma separated list - no spaces): " 94 | f'{", ".join(notify_options)}. NOTE: In addition to those ' 95 | "specified, the %(default)s notifier will always execute." 96 | ), 97 | metavar="[slack,gh_issues,...]", 98 | default="stdout", 99 | ) 100 | self.add_argument( 101 | "--force", 102 | help="Forces an evidence to be fetched, ignoring TTL.", 103 | metavar="raw/category/evidence.ext", 104 | action="append", 105 | default=[], 106 | ) 107 | self.add_argument( 108 | "--include", 109 | help="Specifies the path/name of the fetcher include JSON file.", 110 | metavar="fetchers.json", 111 | default=None, 112 | ) 113 | self.add_argument( 114 | "--exclude", 115 | help="Specifies the path/name of the fetcher exclude JSON file.", 116 | metavar="fetchers.json", 117 | default=None, 118 | ) 119 | 120 | def _validate_arguments(self, args): 121 | if "stdout" not in args.notify: 122 | args.notify += ",stdout" 123 | if args.check == "": 124 | self.parser.error("--check option requires accreditation grouping(s).") 125 | if not args.fetch and not args.check: 126 | self.parser.error("--fetch or --check option is expected.") 127 | if not args.fetch and (args.include or args.exclude): 128 | self.parser.error("--include/--exclude options only valid with --fetch.") 129 | if not args.check and args.fix != "off": 130 | self.parser.error("--fix option only valid with --check.") 131 | 132 | def _validate_extra_arguments(self, extra_args): 133 | self.extra_args = list(set(extra_args) - {"--no-nose", "-s"}) 134 | unrecognized = [ea for ea in self.extra_args if ea.startswith("-")] 135 | if unrecognized: 136 | self.parser.error(f'unrecognized arguments: {", ".join(unrecognized)}') 137 | if "-s" in extra_args: 138 | self.out("WARNING: The -s option is deprecated/no longer used.") 139 | 140 | def _run(self, args): 141 | success = True 142 | if args.fetch: 143 | with FetchMode(args, self.extra_args) as fetch: 144 | # Handle fetcher primary run. 145 | self.out("\nFetcher Primary Run\n") 146 | success = fetch.run_fetchers() 147 | # Handle fetcher dependency reruns. 148 | previous = set() 149 | reruns = fetch.locker.get_dependency_reruns() 150 | rerun_count = 1 151 | while reruns and reruns != previous and rerun_count <= 100: 152 | # Upper bound for reruns set to 100 153 | # to guard against endless executions. 154 | self.out(f"\nFetcher Dependency Re-Run #{rerun_count}\n") 155 | success = fetch.run_fetchers(reruns) and success 156 | rerun_count += 1 157 | previous = reruns 158 | reruns = fetch.locker.get_dependency_reruns() 159 | if reruns: 160 | success = False 161 | self.err( 162 | "\nUnable to resolve dependency issues with %s.\n", 163 | ", ".join(reruns), 164 | ) 165 | for fetch_load_error in fetch.load_errors: 166 | self.err(f"\nERROR: {fetch_load_error}\n") 167 | if args.check: 168 | with CheckMode(args, self.extra_args) as check: 169 | accreds = ", ".join(check.accreds) 170 | self.out(f"\nCheck Run - Accreditations: {accreds}\n") 171 | success = check.run_checks() and success 172 | for check_load_error in check.load_errors: 173 | self.err(f"\nERROR: {check_load_error}\n") 174 | return 0 if success else 1 175 | 176 | 177 | def run(): 178 | """Execute the Compliance CLI.""" 179 | exit(ComplianceCLI().run()) 180 | 181 | 182 | if __name__ == "__main__": 183 | run() 184 | -------------------------------------------------------------------------------- /compliance/templates/readme_toc.md.tmpl: -------------------------------------------------------------------------------- 1 | {#- -*- mode:jinja2; coding: utf-8 -*- -#} 2 | {# 3 | Copyright (c) 2020 IBM Corp. All rights reserved. 4 | 5 | Licensed under the Apache License, Version 2.0 (the "License"); 6 | you may not use this file except in compliance with the License. 7 | You may obtain a copy of the License at 8 | 9 | http://www.apache.org/licenses/LICENSE-2.0 10 | 11 | Unless required by applicable law or agreed to in writing, software 12 | distributed under the License is distributed on an "AS IS" BASIS, 13 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | See the License for the specific language governing permissions and 15 | limitations under the License. 16 | #} 17 | {%- if original|length == 0 -%} 18 | # Evidence Locker 19 | {%- else -%} 20 | {%- set ns = namespace(found=false) -%} 21 | {%- for line in original if ns.found == false -%} 22 | {%- if line == '## Table of Contents - Check Reports' -%} 23 | {%- set ns.found = true -%} 24 | {%- else -%} 25 | {{ line }} 26 | {% endif %} 27 | {%- endfor -%} 28 | {%- endif %} 29 | 30 | ## Table of Contents - Check Reports 31 | 32 | {% for rpt in reports -%} 33 |
34 | {{ rpt['descr'] }} 35 | 36 | - Accreditations: **{{ rpt['accreditations']|upper }}** 37 | - Check:`{{ rpt['check'] }}` 38 | - From: {{ rpt['from'] }} 39 | {%- if rpt['evidences']|length == 0 %} 40 | - Evidences used: N/A 41 | {% else %} 42 | - Evidences used: 43 | {%- for ev in rpt['evidences'] %} 44 | - {{ ev['descr'] }} from {{ ev['from'] }} 45 | {%- endfor -%} 46 | {%- endif %} 47 |
48 | 49 | {% endfor -%} 50 | -------------------------------------------------------------------------------- /compliance/utils/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance automation utilities package.""" 15 | -------------------------------------------------------------------------------- /compliance/utils/credentials.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance credentials configuration.""" 15 | 16 | import logging 17 | from collections import OrderedDict, namedtuple 18 | from configparser import RawConfigParser 19 | from os import environ 20 | from pathlib import Path 21 | 22 | # used to differentiate a user passing None vs 23 | # not passing a value for an optional argument 24 | _sentinel = object() 25 | 26 | logger = logging.getLogger("compliance.utils.credentials") 27 | 28 | 29 | class Config: 30 | """Handle credentials configuration.""" 31 | 32 | def __init__(self, cfg_file="~/.credentials"): 33 | """ 34 | Create an instance of a dictionary-like configuration object. 35 | 36 | :param cfg_file: The path to the RawConfigParser compatible config file 37 | """ 38 | self._cfg = RawConfigParser() 39 | self._cfg_file = cfg_file 40 | if cfg_file is not None: 41 | self._cfg.read(str(Path(cfg_file).expanduser())) 42 | 43 | def __getitem__(self, section): 44 | """ 45 | Get the named tuple representing the configuration held at `section`. 46 | 47 | Build a named tuple representing the configuration at `section`. If a 48 | config file does not have an option for the section ignore it. 49 | Resulting in an AttributeError if accessed later in the code. 50 | 51 | :param section: the section to retrieve 52 | """ 53 | 54 | def _getattr_wrapper(t, attr): 55 | """ 56 | Replace the standard __getattr__ functionality. 57 | 58 | In the case when a section and/or attribute is not set in the 59 | config file, the error shown will be more helpful. 60 | """ 61 | try: 62 | return t.__getattribute__(attr) 63 | except AttributeError as exc: 64 | if self._cfg_file: 65 | msg = ( 66 | f'Unable to locate attribute "{attr}" ' 67 | f'in section "{type(t).__name__}" ' 68 | f'at config file "{self._cfg_file}"' 69 | ) 70 | else: 71 | env_var_name = f"{type(t).__name__}_{attr}".upper() 72 | msg = f"Unable to find the env var: {env_var_name}" 73 | exc.args = (msg,) 74 | raise exc 75 | 76 | env_vars = [k for k in environ.keys() if k.startswith(f"{section.upper()}_")] 77 | env_keys = [k.split(section.upper())[1].lstrip("_").lower() for k in env_vars] 78 | env_values = [environ[e] for e in env_vars] 79 | if env_vars: 80 | logger.debug(f'Loading credentials from ENV vars: {", ".join(env_vars)}') 81 | params = [] 82 | if self._cfg.has_section(section): 83 | params = self._cfg.options(section) 84 | values = [self._cfg.get(section, x) for x in params] 85 | 86 | d = OrderedDict(zip(params, values)) 87 | 88 | if env_vars: 89 | d.update(zip(env_keys, env_values)) 90 | t = namedtuple(section, " ".join(list(d.keys()))) 91 | t.__getattr__ = _getattr_wrapper 92 | return t(*list(d.values())) 93 | 94 | def get(self, section, key=None, account=None, default=_sentinel): 95 | """ 96 | Retrieve sections and keys by account. 97 | 98 | :param section: the section from which to retrieve keys. 99 | :parm key: the key in the section whose value you want to retrieve. if 100 | not specified, returns the whole section as a dictionary. 101 | :param account: if provided, fetches the value for the specific account. 102 | assumes the account is prefixed to the key and separated by _. 103 | :param default: if provided, returns this value if a value cannot be 104 | found; otherwise raises an exception. 105 | """ 106 | if key is None: 107 | return self[section] 108 | if account: 109 | key = "_".join([account, key]) 110 | if default == _sentinel: 111 | return getattr(self[section], key) 112 | else: 113 | return getattr(self[section], key, default) 114 | -------------------------------------------------------------------------------- /compliance/utils/data_parse.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance automation data and formatting utilities module.""" 15 | 16 | import hashlib 17 | import json 18 | 19 | 20 | def parse_dot_key(data, key): 21 | """ 22 | Provide the element from the ``data`` dictionary defined by the ``key``. 23 | 24 | The key may be a key path depicted by dot notation. 25 | 26 | :param data: A dictionary 27 | :param key: A dictionary key string that may be a key path depicted by dot 28 | notation. For example "foo.bar". 29 | 30 | :returns: The dictionary value from ``data`` associated to the ``key``. 31 | """ 32 | for key_part in key.split("."): 33 | data = data.get(key_part) 34 | if data is None: 35 | break 36 | return data 37 | 38 | 39 | def get_sha256_hash(key, size=None): 40 | """ 41 | Provide a SHA256 hash based on the supplied key values. 42 | 43 | :param key: An iterable of key values. 44 | :param size: The size of the returned hash. Defaults to full hash. If 45 | size provided is greater than the hash size the full hash is returned. 46 | 47 | :returns: a SHA256 hash for the key values supplied. 48 | """ 49 | partition_hash = hashlib.sha256() 50 | for part in key: 51 | partition_hash.update(str(part).encode("utf-8")) 52 | sha256_hash = partition_hash.hexdigest() 53 | if not size or size > len(sha256_hash): 54 | size = len(sha256_hash) 55 | return sha256_hash[:size] 56 | 57 | 58 | def format_json(data, **addl_kwargs): 59 | """ 60 | Provide a JSON string formatted to the standards of the library. 61 | 62 | This function ensures that the JSON is sorted, indented, and uses 63 | the appropriate separators for all instances of JSON generated 64 | by this library. 65 | 66 | :param data: The data structure to be formatted. 67 | :param add_kwargs: Additional json.dumps options 68 | 69 | :returns: A formatted JSON string. 70 | """ 71 | return json.dumps( 72 | data, indent=2, sort_keys=True, separators=(",", ": "), **addl_kwargs 73 | ) 74 | 75 | 76 | def deep_merge(a, b, path=None, append=False): 77 | r""" 78 | Merge two dicts, taking into account any sub (or sub-sub-\*) dicts. 79 | 80 | If ``append`` is ``True`` then list values from ``b`` will be appended to 81 | ``a``'s. Modified from: https://stackoverflow.com/a/7205107/566346 82 | """ 83 | for key in b: 84 | if key in a: 85 | if isinstance(a[key], dict) and isinstance(b[key], dict): 86 | deep_merge(a[key], b[key], (path or []) + [str(key)], append) 87 | continue 88 | is_lists = isinstance(a[key], list) and isinstance(b[key], list) 89 | if is_lists and append: 90 | a[key] += b[key] 91 | else: 92 | a[key] = b[key] 93 | else: 94 | a[key] = b[key] 95 | return a 96 | -------------------------------------------------------------------------------- /compliance/utils/exceptions.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2021, 2022 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance automation custom exceptions module.""" 15 | 16 | 17 | class StaleEvidenceError(Exception): 18 | """Stale evidence exception class.""" 19 | 20 | pass 21 | 22 | 23 | class EvidenceNotFoundError(ValueError): 24 | """Missing evidence exception class.""" 25 | 26 | pass 27 | 28 | 29 | class HistoricalEvidenceNotFoundError(ValueError): 30 | """Missing historical evidence exception class.""" 31 | 32 | pass 33 | 34 | 35 | class DependencyUnavailableError(ValueError): 36 | """Missing dependent evidence exception class.""" 37 | 38 | pass 39 | 40 | 41 | class DependencyFetcherNotFoundError(ValueError): 42 | """Dependency fetcher not found exception class.""" 43 | 44 | pass 45 | 46 | 47 | class UnverifiedEvidenceError(Exception): 48 | """Unverified evidence exception class.""" 49 | 50 | pass 51 | 52 | 53 | class LockerPushError(Exception): 54 | """Locker push exception class.""" 55 | 56 | def __init__(self, push_info=None): 57 | """ 58 | Construct the locker push exception. 59 | 60 | :param push_info: a GitPython PushInfo object containing Git remote 61 | push information 62 | """ 63 | self.push_info = push_info 64 | 65 | def __str__(self): 66 | """Display the error as a string.""" 67 | msg = "Push to remote locker failed.\n" 68 | if self.push_info: 69 | msg += ( 70 | f" Summary: {self.push_info.summary}" 71 | f" Error Flags: {self.push_info.flags}" 72 | ) 73 | return msg 74 | -------------------------------------------------------------------------------- /compliance/utils/fetch_local_commands: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Copyright (c) 2022 IBM Corp. All rights reserved. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | 16 | usage() { 17 | echo "Usage: $0 [-h] [-u ]" 18 | echo " -h Print usage." 19 | echo " -s Show exit status." 20 | echo " -t Show timestamp." 21 | return 1 22 | } 1>&2 23 | 24 | show_exit_status=0 25 | show_timestamp=0 26 | while getopts :hst option 27 | do 28 | case $option in 29 | h) 30 | usage 31 | exit 0 32 | ;; 33 | s) 34 | show_exit_status=1 35 | ;; 36 | t) 37 | show_timestamp=1 38 | ;; 39 | *) 40 | echo "Error: Illegal option -$OPTARG specified." 1>&2 41 | exit "$(usage)" 42 | ;; 43 | esac 44 | done 45 | shift $((OPTIND - 1)) 46 | 47 | ps1() { 48 | if [[ "$show_timestamp" -eq 1 ]]; then 49 | printf '[%s] %s@%s:%s$' "$(date -u +%FT%TZ)" "$(whoami)" "$(hostname -f)" "${PWD/#$HOME/~}" 50 | else 51 | printf '%s@%s:%s$' "$(whoami)" "$(hostname -f)" "${PWD/#$HOME/~}" 52 | fi 53 | } 54 | 55 | run() { 56 | printf '%s ' "$(ps1)" "$@" 57 | printf '\n' 58 | eval "$@"; rc="$?" 59 | if [[ "$show_exit_status" -eq 1 ]]; then 60 | printf '%s %s\n%s\n' "$(ps1)" 'echo $?' "$rc" 61 | fi 62 | } 63 | 64 | while IFS=$'\n' read -r cmd; do 65 | run "$cmd" 2>&1 66 | done 67 | -------------------------------------------------------------------------------- /compliance/utils/http.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance HTTP helpers.""" 15 | 16 | import requests 17 | 18 | 19 | class BaseSession(requests.Session): 20 | """Subclass of requests.Session to support a base URL.""" 21 | 22 | def __init__(self, baseurl): 23 | """ 24 | Set the baseurl for the session. 25 | 26 | All requests using this session will be run against this URL. 27 | """ 28 | super(BaseSession, self).__init__() 29 | self.baseurl = baseurl.strip("/") 30 | 31 | def prepare_request(self, request): 32 | """Prefix with self.baseurl if request.url does not include it.""" 33 | if not request.url.startswith(self.baseurl): 34 | # the behavior of urljoin isn't what we want here 35 | request.url = f'{self.baseurl}/{request.url.lstrip("/")}' 36 | return super(BaseSession, self).prepare_request(request) 37 | -------------------------------------------------------------------------------- /compliance/utils/path.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance automation path formatting utilities module.""" 15 | 16 | import importlib 17 | import sys 18 | from pathlib import Path, PurePath 19 | 20 | from compliance.config import get_config 21 | 22 | FETCH_PREFIX = "fetch_" 23 | CHECK_PREFIX = "test_" 24 | 25 | 26 | def get_toplevel_dirpath(path): 27 | """ 28 | Provide the top level directory for the given path. 29 | 30 | The top level directory contains the ``controls.json`` file. 31 | This function returns ``None`` if a top level path can not be found. 32 | 33 | :param path: absolute or relative path to a file or directory. 34 | 35 | :returns: the absolute path to the top level directory. 36 | """ 37 | if path is None: 38 | return 39 | paths = list(Path(path).resolve().parents) 40 | if Path(path).resolve().is_dir(): 41 | paths = [Path(path).resolve()] + paths 42 | for path in paths[:-1]: 43 | if Path(path, "controls.json").is_file(): 44 | return str(path) 45 | 46 | 47 | def get_module_path(path): 48 | """ 49 | Provide the full module dot notation path based on the file path provided. 50 | 51 | :param path: absolute or relative path to a file or directory. 52 | 53 | :returns: the module path in dot notation. 54 | """ 55 | if path is None: 56 | return 57 | for parent in PurePath(path).parents: 58 | if parent.name == "site-packages": 59 | return _path_to_dot(str(PurePath(path).relative_to(parent))) 60 | return _path_to_dot(str(PurePath(path).relative_to(get_toplevel_dirpath(path)))) 61 | 62 | 63 | def load_evidences_modules(path): 64 | """ 65 | Load all evidences modules found within the ``path`` directory structure. 66 | 67 | :param path: absolute path to a top level directory. 68 | """ 69 | for ev_mod in [p for p in Path(path).rglob("evidences") if p.is_dir()]: 70 | module_name = get_module_path(str(ev_mod)) 71 | spec = None 72 | try: 73 | spec = importlib.util.spec_from_file_location( 74 | module_name, str(Path(ev_mod, "__init__.py")) 75 | ) 76 | except ImportError: 77 | continue 78 | if spec is None or module_name in sys.modules: 79 | continue 80 | module = importlib.util.module_from_spec(spec) 81 | sys.modules[module_name] = module 82 | spec.loader.exec_module(module) 83 | 84 | 85 | def substitute_config(path_tmpl): 86 | """ 87 | Substitute the config values on the given path template. 88 | 89 | :param path_tmpl: a string template of a path 90 | """ 91 | return path_tmpl.format(**(get_config().raw_config)) 92 | 93 | 94 | def _path_to_dot(path): 95 | return path.rstrip(".py").replace("/", ".").replace("\\", ".") 96 | -------------------------------------------------------------------------------- /compliance/utils/services/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance automation utilities services package.""" 15 | -------------------------------------------------------------------------------- /compliance/utils/services/pagerduty.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance Pagerduty service helper.""" 15 | 16 | import json 17 | from urllib.parse import urljoin 18 | 19 | from compliance.utils.credentials import Config 20 | 21 | import requests 22 | 23 | PAGERDUTY_API_URL = "https://api.pagerduty.com" 24 | PD_EVENTS_V2_URL = "https://events.pagerduty.com/v2/enqueue" 25 | # 100 is the maximum page size 26 | PAGES_LIMIT = 100 27 | 28 | 29 | def _init_request(path, params, headers, creds): 30 | credentials = creds or Config() 31 | hdrs = { 32 | "Accept": "application/vnd.pagerduty+json;version=2", 33 | "Authorization": f'Token token={credentials["pagerduty"].api_key}', 34 | } 35 | if headers: 36 | hdrs.update(headers) 37 | params = params or {} 38 | url = urljoin(PAGERDUTY_API_URL, path) 39 | return url, params, hdrs 40 | 41 | 42 | def get(path, params=None, headers=None, creds=None): 43 | """ 44 | Perform a GET operation. 45 | 46 | Returns the pages wrapped as :py:class:`requests.Response` object. 47 | This uses the PD API v2. 48 | 49 | :param path: API endpoint to call (e.g. 'users') 50 | :param params: a dictionary with parameters 51 | :param headers: a dictionary with headers to include 52 | :param creds: a Config object with PagerDuty credentials 53 | """ 54 | url, params, hdrs = _init_request(path, params, headers, creds) 55 | offset = 0 56 | params.update({"limit": PAGES_LIMIT, "offset": offset}) 57 | more = True 58 | while more: 59 | r = requests.get(url, headers=hdrs, params=params, timeout=180) 60 | yield r 61 | more = r.json().get("more", False) 62 | if more: 63 | offset = offset + PAGES_LIMIT 64 | params.update({"offset": offset}) 65 | 66 | 67 | def delete(path, params=None, headers=None, creds=None): 68 | """ 69 | Perform a DELETE operation. 70 | 71 | Returns the result as :py:class:`requests.Response` object. 72 | This uses the PD API v2. 73 | 74 | :param path: API endpoint to call (e.g. 'users') 75 | :param params: a dictionary with parameters 76 | :param headers: a dictionary with headers to include 77 | :param creds: a Config object with PagerDuty credentials 78 | """ 79 | url, params, hdrs = _init_request(path, params, headers, creds) 80 | return requests.delete(url, headers=hdrs, params=params, timeout=180) 81 | 82 | 83 | def put(path, params=None, headers=None, creds=None): 84 | """ 85 | Perform a PUT operation. 86 | 87 | Return the result as :py:class:`requests.Response` object. 88 | This uses the PD API v2. 89 | 90 | :param path: API endpoint to call (e.g. 'users') 91 | :param params: a dictionary with parameters 92 | :param headers: a dictionary with headers to include 93 | :param creds: a Config object with PagerDuty credentials 94 | """ 95 | url, params, hdrs = _init_request(path, params, headers, creds) 96 | return requests.put(url, headers=hdrs, params=params, timeout=180) 97 | 98 | 99 | def post(path, params=None, headers=None, creds=None): 100 | """ 101 | Perform a POST operation. 102 | 103 | Returns the result as :py:class:`requests.Response` objects. 104 | This uses the PD API v2. 105 | 106 | :param path: API endpoint to call (e.g. 'users') 107 | :param params: a dictionary with parameters 108 | :param headers: a dictionary with headers to include 109 | :param creds: a Config object with PagerDuty credentials 110 | """ 111 | url, params, hdrs = _init_request(path, params, headers, creds) 112 | return requests.post(url, headers=hdrs, params=params, timeout=180) 113 | 114 | 115 | def send_event( 116 | action, check, title, source, severity="error", details="", links=None, creds=None 117 | ): 118 | """Send an event to PD using the Events API.""" 119 | credentials = creds or Config() 120 | 121 | msg = { 122 | "event_action": action, 123 | "routing_key": credentials["pagerduty"].events_integration_key, 124 | "dedup_key": check, 125 | "payload": { 126 | "summary": title, 127 | "source": source, 128 | "severity": severity, 129 | "custom_details": details, 130 | }, 131 | "links": links or [], 132 | } 133 | headers = {"Content-Type": "application/json"} 134 | 135 | response = requests.post( 136 | PD_EVENTS_V2_URL, headers=headers, data=json.dumps(msg), timeout=180 137 | ) 138 | response.raise_for_status() 139 | if response.json().get("status") != "success": 140 | raise RuntimeError("PagerDuty Error: " + response.json()) 141 | -------------------------------------------------------------------------------- /compliance/utils/test.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance automation underlying test framework utilities module.""" 15 | 16 | 17 | def parse_test_id(test_id): 18 | """ 19 | Parse a test ID into useful parts. 20 | 21 | Takes a test ID and parses out useful parts of it:: 22 | 23 | > parse_test_id('foo.bar.baz.test_mod.MyTestClass.test_method') 24 | { 25 | 'scope': 'foo', 26 | 'type': 'bar', 27 | 'accreditation': 'baz', 28 | 'file': 'test_mod', 29 | 'class': 'MyTestClass', 30 | 'method': 'test_method', 31 | 'class_path': 'foo.bar.baz.test_mod.MyTestClass.test_method' 32 | } 33 | 34 | Note: scope/type/accreditation might not be returned if your 35 | path structure is different from the suggested one. 36 | """ 37 | parts = test_id.split(".") 38 | full_path = len(parts) == 6 39 | return { 40 | "scope": full_path and parts[0], 41 | "type": full_path and parts[1], 42 | "accreditation": full_path and parts[2], 43 | "file": parts[-3], 44 | "class": parts[-2], 45 | "method": parts[-1], 46 | "class_path": ".".join(parts[0:-1]), 47 | } 48 | -------------------------------------------------------------------------------- /demo/README.md: -------------------------------------------------------------------------------- 1 | # Execution Configuration Example 2 | -------------------------------------------------------------------------------- /demo/at-logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ComplianceAsCode/auditree-framework/7ffa114d475f337a75b214e3f3b4811f50dd324d/demo/at-logo.png -------------------------------------------------------------------------------- /demo/auditree_demo.json: -------------------------------------------------------------------------------- 1 | { 2 | "locker": { 3 | "default_branch": "main", 4 | "repo_url": "https://github.com/MY_ORG/MY_EVIDENCE_REPO" 5 | }, 6 | "notify": { 7 | "slack": { 8 | "demo.arboretum.accred": ["#some-slack-channel", "#some-other-slack-channel"], 9 | "demo.custom.accred": ["#some-slack-channel"] 10 | }, 11 | "gh_issues": { 12 | "demo.arboretum.accred": { 13 | "repo": ["MY_ORG/MY_GH_ISSUES_REPO", "MY_ORG/MY_OTHER_GH_ISSUES_REPO"], 14 | "title": "Check results for demo.arboretum.accred accreditation" 15 | } 16 | } 17 | }, 18 | "org": { 19 | "gh": { 20 | "orgs": ["nasa", "esa"] 21 | } 22 | } 23 | } 24 | -------------------------------------------------------------------------------- /demo/controls.json: -------------------------------------------------------------------------------- 1 | { 2 | "arboretum.auditree.checks.test_python_packages.PythonPackageCheck": ["demo.arboretum.accred"], 3 | "demo_examples.checks.test_github.GitHubAPIVersionsCheck": ["demo.custom.accred"], 4 | "demo_examples.checks.test_github.GitHubOrgs": ["demo.custom.accred"], 5 | "demo_examples.checks.test_image_content.ImageCheck": ["demo.custom.accred"] 6 | } 7 | -------------------------------------------------------------------------------- /demo/demo_examples/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Demo module.""" 15 | -------------------------------------------------------------------------------- /demo/demo_examples/checks/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Demo checks module.""" 15 | -------------------------------------------------------------------------------- /demo/demo_examples/checks/test_common.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Common checks module.""" 15 | 16 | from arboretum.auditree.checks.test_python_packages import PythonPackageCheck # noqa 17 | -------------------------------------------------------------------------------- /demo/demo_examples/checks/test_github.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2023 EnterpriseDB. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | """GitHub demo checks.""" 16 | 17 | import json 18 | 19 | from compliance.check import ComplianceCheck 20 | from compliance.evidence import with_raw_evidences 21 | 22 | from demo_examples.evidence import utils 23 | 24 | from parameterized import parameterized 25 | 26 | 27 | class GitHubAPIVersionsCheck(ComplianceCheck): 28 | """Perform analysis on GitHub supported versions API response evidence.""" 29 | 30 | @property 31 | def title(self): 32 | """ 33 | Return the title of the checks. 34 | 35 | :returns: the title of the checks 36 | """ 37 | return "GitHub API Versions" 38 | 39 | @with_raw_evidences("github/api_versions.json") 40 | def test_supported_versions(self, evidence): 41 | """ 42 | Check whether there are any supported versions. 43 | 44 | Always warn about something, for demo purposes. 45 | """ 46 | version_list = json.loads(evidence.content) 47 | versions_str = ", ".join(version_list) 48 | if not version_list: 49 | self.add_failures( 50 | "Supported GitHub API Versions Violation", 51 | "No API versions were indicated as supported by GitHub.", 52 | ) 53 | elif len(version_list) == 1: 54 | self.add_warnings( 55 | "Supported GitHub API Versions Warning", 56 | "There is only one supported version. " 57 | f"Get with the program: {versions_str}", 58 | ) 59 | elif len(version_list) > 1: 60 | self.add_warnings( 61 | "Supported GitHub API Versions Warning", 62 | "There are more than one supported versions. " 63 | f"Check the docs for the latest changes: {versions_str}", 64 | ) 65 | 66 | def get_reports(self): 67 | """ 68 | Provide the check report name. 69 | 70 | :returns: the report(s) generated for this check 71 | """ 72 | return ["github/api_versions.md"] 73 | 74 | def msg_supported_versions(self): 75 | """ 76 | Supported GitHub API versions check notifier. 77 | 78 | :returns: notification dictionary. 79 | """ 80 | return {"subtitle": "Supported GitHub API Versions Violation", "body": None} 81 | 82 | 83 | class GitHubOrgs(ComplianceCheck): 84 | """Perform analysis on GitHub some orgs.""" 85 | 86 | @property 87 | def title(self): 88 | """ 89 | Return the title of the checks. 90 | 91 | :returns: the title of the checks 92 | """ 93 | return "GitHub Org checks" 94 | 95 | @parameterized.expand(utils.get_gh_orgs) 96 | def test_members_is_not_empty(self, org): 97 | """Check whether the GitHub org is not empty.""" 98 | evidence = self.locker.get_evidence(f"raw/github/{org}_members.json") 99 | members = json.loads(evidence.content) 100 | if not members: 101 | self.add_failures(org, "There is nobody!") 102 | elif len(members) < 5: 103 | self.add_warnings(org, "There are people in there, but less than 5!") 104 | 105 | def get_reports(self): 106 | """Return GitHub report name.""" 107 | return ["github/members.md"] 108 | -------------------------------------------------------------------------------- /demo/demo_examples/checks/test_image_content.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | """Image content demo.""" 16 | 17 | from compliance.check import ComplianceCheck 18 | from compliance.evidence import evidences, with_raw_evidences 19 | 20 | 21 | class ImageCheck(ComplianceCheck): 22 | """Perform analysis on image evidence.""" 23 | 24 | @property 25 | def title(self): 26 | """ 27 | Return the title of the checks. 28 | 29 | :returns: the title of the checks 30 | """ 31 | return "Auditree Image" 32 | 33 | @with_raw_evidences("images/auditree_logo.png") 34 | def test_image_content_with_decorator(self, evidence): 35 | """Check that the evidence content exists using decorator.""" 36 | if not evidence.content: 37 | self.add_failures("Using decorator", "auditree_logo.png evidence is empty") 38 | 39 | def test_image_content_with_ctx_mgr(self): 40 | """Check that the evidence content exists using context manager.""" 41 | with evidences(self, "raw/images/auditree_logo.png") as evidence: 42 | if not evidence.content: 43 | self.add_failures( 44 | "Using context manager", "auditree_logo.png evidence is empty" 45 | ) 46 | 47 | def get_reports(self): 48 | """ 49 | Provide the check report name. 50 | 51 | :returns: the report(s) generated for this check 52 | """ 53 | return ["images/image_check.md"] 54 | 55 | def msg_image_content_with_decorator(self): 56 | """ 57 | Image exists using decorator check notifier. 58 | 59 | :returns: notification dictionary. 60 | """ 61 | return {"body": None} 62 | 63 | def msg_image_content_with_ctx_mgr(self): 64 | """ 65 | Image exists using context manager check notifier. 66 | 67 | :returns: notification dictionary. 68 | """ 69 | return {"body": None} 70 | -------------------------------------------------------------------------------- /demo/demo_examples/evidence/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | """Evidence module.""" 16 | 17 | from compliance.config import get_config 18 | from compliance.evidence import DAY, RawEvidence, ReportEvidence 19 | 20 | get_config().add_evidences( 21 | [ 22 | RawEvidence( 23 | "api_versions.json", "github", DAY, "Supported GitHub API versions" 24 | ), 25 | RawEvidence( 26 | "auditree_logo.png", 27 | "images", 28 | DAY, 29 | "The Auditree logo image", 30 | binary_content=True, 31 | ), 32 | ReportEvidence( 33 | "api_versions.md", "github", DAY, "Supported GitHub versions report." 34 | ), 35 | ReportEvidence("image_check.md", "images", DAY, "Image Check Analysis report."), 36 | ] 37 | ) 38 | -------------------------------------------------------------------------------- /demo/demo_examples/evidence/utils.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2023 EnterpriseDB. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | """Utils for evidence.""" 16 | 17 | from compliance.config import get_config 18 | 19 | 20 | def get_gh_orgs(): 21 | """Return the GitHub organization names.""" 22 | return get_config().get("org.gh.orgs") 23 | -------------------------------------------------------------------------------- /demo/demo_examples/fetchers/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Demo fetcher module.""" 15 | -------------------------------------------------------------------------------- /demo/demo_examples/fetchers/fetch_auditree_logo.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """A demo fetcher that gets a logo image.""" 15 | 16 | from compliance.evidence import store_raw_evidence 17 | from compliance.fetch import ComplianceFetcher 18 | 19 | 20 | class ImageFetcher(ComplianceFetcher): 21 | """Fetch the Auditree logo image and store as evidence.""" 22 | 23 | @store_raw_evidence("images/auditree_logo.png") 24 | def fetch_auditree_logo(self): 25 | """Fetch the Auditree logo.""" 26 | return open("at-logo.png", "rb").read() 27 | -------------------------------------------------------------------------------- /demo/demo_examples/fetchers/fetch_common.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """A demo common fetcher module.""" 15 | 16 | from arboretum.auditree.fetchers.fetch_python_packages import ( # noqa 17 | PythonPackageFetcher, 18 | ) 19 | -------------------------------------------------------------------------------- /demo/demo_examples/fetchers/fetch_github.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2023 EnterpriseDB. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | """Demo fetchers for GitHub.""" 16 | 17 | import json 18 | 19 | from compliance.evidence import DAY, RawEvidence, store_raw_evidence 20 | from compliance.fetch import ComplianceFetcher 21 | 22 | from demo_examples.evidence import utils 23 | 24 | from parameterized import parameterized 25 | 26 | 27 | class GitHubFetcher(ComplianceFetcher): 28 | """Fetch the current supported GitHub API versions.""" 29 | 30 | @classmethod 31 | def setUpClass(cls): 32 | """Initialise the fetcher class with common functionality.""" 33 | cls.client = cls.session( 34 | "https://api.github.com/", **{"Accept": "application/json"} 35 | ) 36 | 37 | @store_raw_evidence("github/api_versions.json") 38 | def fetch_api_versions(self): 39 | """Fetch the current supported GitHub API versions.""" 40 | # This is where you might e.g. fetch your evidence 41 | # from a remote API 42 | versions = self.client.get("versions") 43 | versions.raise_for_status() 44 | return versions.text 45 | 46 | @parameterized.expand(utils.get_gh_orgs) 47 | def fetch_github_members(self, org): 48 | """Fetch GitHub members from the organization.""" 49 | # We don't use the helper decorator in this case, so we have to manage 50 | # the envidence life-cycle: creation, fetch and store in the locker. 51 | evidence = RawEvidence( 52 | f"{org}_members.json", "github", DAY, f"GH members of org {org}" 53 | ) 54 | 55 | if self.locker.validate(evidence): 56 | return 57 | resp = self.client.get(f"/orgs/{org}/members", params={"page": 1}) 58 | resp.raise_for_status() 59 | evidence.set_content(json.dumps(resp.json(), indent=2)) 60 | self.locker.add_evidence(evidence) 61 | -------------------------------------------------------------------------------- /demo/demo_examples/requirements.txt: -------------------------------------------------------------------------------- 1 | parameterized 2 | auditree-arboretum 3 | -------------------------------------------------------------------------------- /demo/demo_examples/templates/default.md.tmpl: -------------------------------------------------------------------------------- 1 | {#- -*- mode:jinja2; coding: utf-8 -*- -#} 2 | {# 3 | Copyright (c) 2020 IBM Corp. All rights reserved. 4 | 5 | Licensed under the Apache License, Version 2.0 (the "License"); 6 | you may not use this file except in compliance with the License. 7 | You may obtain a copy of the License at 8 | 9 | http://www.apache.org/licenses/LICENSE-2.0 10 | 11 | Unless required by applicable law or agreed to in writing, software 12 | distributed under the License is distributed on an "AS IS" BASIS, 13 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | See the License for the specific language governing permissions and 15 | limitations under the License. 16 | #} 17 | 18 | # {{ test.title }} Report {{ now.strftime('%Y-%m-%d') }} 19 | {% if test.total_issues_count(results) == 0 %} 20 | No issues found! 21 | {% else %} 22 | ## Results 23 | 24 | {% if test.warnings_for_check_count(results) > 0 -%} 25 | * [Warnings](#warnings): {{ test.warnings_for_check_count(results) }} 26 | {% for k in all_warnings.keys() -%} 27 | {% if all_warnings[k]|length > 0 -%} 28 | {% set anchor = k.lower()|replace(' ', '-') %} 29 | * [{{ k|capitalize }}](#{{ anchor }}): {{ all_warnings[k]|length }} 30 | {%- endif %} 31 | {%- endfor -%} 32 | {% endif %} 33 | {% if test.failures_for_check_count(results) > 0 %} 34 | * [Failures](#failures): {{ test.failures_for_check_count(results) }} 35 | {% for k in all_failures.keys() -%} 36 | {% if all_failures[k]|length > 0 -%} 37 | {% set anchor = k.lower()|replace(' ', '-') %} 38 | * [{{ k|capitalize }}](#{{ anchor }}): {{ all_failures[k]|length }} 39 | {%- endif %} 40 | {%- endfor -%} 41 | {% endif -%} 42 | {% endif %} 43 | 44 | 45 | {% if test.warnings_for_check_count(results) > 0 -%} 46 | ## Warnings 47 | 48 | {% for type in all_warnings.keys()|sort -%} 49 | {% if all_warnings[type]|length > 0 %} 50 | #### {{ type|capitalize }} #### 51 | {% for at in all_warnings[type]| sort %} 52 | {% if not link -%} 53 | * {{ at -}} 54 | {%- else -%} 55 | * [{{ at }}]({{ link }}/{{ at }}) 56 | {%- endif %} 57 | {%- endfor %} 58 | {% endif -%} 59 | {% endfor %} 60 | {% endif %} 61 | 62 | {% if test.failures_for_check_count(results) > 0 -%} 63 | ## Failures 64 | 65 | {% for type in all_failures.keys()|sort -%} 66 | {% if all_failures[type]|length > 0 %} 67 | #### {{ type|capitalize }} #### 68 | {% for at in all_failures[type]| sort %} 69 | {% if not link -%} 70 | * {{ at -}} 71 | {%- else -%} 72 | * [{{ at -}}]({{ link }}/{{ at }}) 73 | {%- endif %} 74 | {%- endfor %} 75 | {% endif -%} 76 | {% endfor %} 77 | {% endif %} 78 | -------------------------------------------------------------------------------- /demo/demo_examples/templates/reports/time/api_versions.md.tmpl: -------------------------------------------------------------------------------- 1 | {# 2 | Copyright (c) 2020 IBM Corp. All rights reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"); 5 | you may not use this file except in compliance with the License. 6 | You may obtain a copy of the License at 7 | 8 | http://www.apache.org/licenses/LICENSE-2.0 9 | 10 | Unless required by applicable law or agreed to in writing, software 11 | distributed under the License is distributed on an "AS IS" BASIS, 12 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | See the License for the specific language governing permissions and 14 | limitations under the License. 15 | #} 16 | # {{ test.title }} Report: {{ now.strftime('%Y-%m-%d') }} 17 | 18 | This report contains all GitHub API versions violations 19 | 20 | {% if test.total_issues_count(results) == 0 %} 21 | **No violations found!** 22 | {% else %} 23 | {% if test.failures_for_check_count(results) > 0 %} 24 | {% for section in all_failures.keys()|sort -%} 25 | ## {{ section }} 26 | {% for failure in all_failures[section]|sort -%} 27 | - {{ failure }} 28 | {% endfor %} 29 | {% endfor %} 30 | {% endif %} 31 | {% if test.warnings_for_check_count(results) > 0 %} 32 | {% for section in all_warnings.keys()|sort -%} 33 | ## {{ section }} 34 | {% for warning in all_warnings[section]|sort -%} 35 | - {{ warning }} 36 | {% endfor %} 37 | {% endfor %} 38 | {% endif %} 39 | {% endif %} 40 | -------------------------------------------------------------------------------- /demo/requirements.txt: -------------------------------------------------------------------------------- 1 | auditree-arboretum>=0.6.1 2 | auditree-framework>=1.6.3 3 | -------------------------------------------------------------------------------- /doc-source/coding-standards.rst: -------------------------------------------------------------------------------- 1 | .. -*- mode:rst; coding:utf-8 -*- 2 | 3 | .. _coding-standards: 4 | 5 | Coding Standards 6 | ---------------- 7 | 8 | In this project, we use Python as programming language so please 9 | follow these rules: 10 | 11 | * Keep the code tidy using `black `_ 12 | and `flake8 `_. Don't introduce new 13 | violations and remove them if you spot any. This is enforced by 14 | Github Actions builds. To check your code locally, use:: 15 | 16 | make code-format 17 | make code-lint 18 | 19 | * Please provide good unit tests for your code changes. It is 20 | important to keep a good level of test coverage. 21 | 22 | * Document everything you create using docstring within the code. We 23 | use `sphinx `_ for documentation. 24 | 25 | * Test your code locally and make sure it works before creating a PR:: 26 | 27 | make test 28 | 29 | 30 | Avoid code smells 31 | ~~~~~~~~~~~~~~~~~ 32 | 33 | * `An interested reading 34 | `_ 35 | 36 | * Your change makes the tool to run slower. Ask yourself if actually 37 | that's the only way to do it (or somebody for advice). 38 | 39 | * Unit tests must be fast (really fast) and never use network 40 | resources. 41 | 42 | * Do not use shell commands (`os.system`, `subprocess`, etc.) unless 43 | the usage would save us from days/tons of work. Python libraries are 44 | preferred in most cases and if there is not a Python library for it, 45 | then explain your case and try to convince people why it is better 46 | using a shell command. 47 | 48 | Avoid third-party stuff 49 | ~~~~~~~~~~~~~~~~~~~~~~~ 50 | 51 | * Third-party libraries: using third-party libraries is not always a 52 | good idea. While they can be beneficial, using third-party libraries 53 | means we must maintain the code to be compatible with them and the 54 | installation process gets more complicated. So, by default, try to use 55 | Python built-ins. If you actually need to introduce a new third-party 56 | library, please explain your reasoning and get a +1 from somebody else 57 | for it. 58 | 59 | * Third-party programs: due the reasons above, avoid using external programs as 60 | much as possible and get consensus if you need to introduce one. 61 | 62 | Always Test 63 | ~~~~~~~~~~~ 64 | 65 | * Always test your code, please don't assume it works. To do this, add 66 | unit tests and pay attention to the coverage results that come back:: 67 | 68 | make test 69 | 70 | * Always make sure that the entire test suite runs cleanly 71 | -------------------------------------------------------------------------------- /doc-source/credentials-example.cfg: -------------------------------------------------------------------------------- 1 | # -*- mode:conf -*- 2 | 3 | # Token for github.com to be used by the Locker 4 | [github] 5 | token=XXX 6 | 7 | # Webhook for Slack notifications 8 | [slack] 9 | webhook=XXX 10 | # token=XXX # can be used instead of webhook 11 | 12 | # Token for PagerDuty notifications 13 | [pagerduty] 14 | api_key=XXX 15 | events_integration_key=XXX 16 | -------------------------------------------------------------------------------- /doc-source/evidence-partitioning.rst: -------------------------------------------------------------------------------- 1 | .. -*- mode:rst; coding:utf-8 -*- 2 | 3 | .. _evidence-partitioning: 4 | 5 | Partitioned Evidence 6 | ==================== 7 | 8 | Depending on the repository hosting service being used, it may be necessary to 9 | manage evidence more effectively. For example Github has limits on individual 10 | file sizes. In cases like this it is possible to partition evidence into 11 | smaller, more manageable chunks if that evidence satisfies the following 12 | conditions: 13 | 14 | * Evidence is of the type :py:class:`~compliance.evidence.RawEvidence` or a 15 | sub-class of :py:class:`~compliance.evidence.RawEvidence`. 16 | 17 | * The evidence content is JSON and the evidence file has a ``.json`` file 18 | extension. 19 | 20 | * A partition key exists, that can logically separate the evidence into parts. 21 | 22 | * A key can be a single field or multiple fields in the evidence JSON 23 | document structure. If multiple fields are used, the partitioning of the 24 | evidence will be handled in the order of the fields provided. 25 | 26 | * The path to a key can be represented using ``dot`` notation if necessary. 27 | 28 | * A single partition root exists where partitioning can be applied. 29 | 30 | * The partition root must be a list field that serves as the origin of 31 | evidence partitioning based on the partition key provided. 32 | 33 | * The partition root can be the entire evidence JSON document or a nested 34 | field with the evidence JSON document. 35 | 36 | Partition By Configuration 37 | -------------------------- 38 | 39 | Turning partitioning on for a given evidence can be accomplished by setting the 40 | appropriate ``locker.partitions`` parameters in your configuration file. 41 | Parameters include the relative repository path to the raw evidence (sans the 42 | *raw*), the partition key(s) as a list, and optionally the partition root 43 | where the evidence will be partitioned. For example:: 44 | 45 | { 46 | "locker": { 47 | "repo_url": "https://github.com/my-org/my-evidence-repo", 48 | "partitions": { 49 | "foo/evidence_bar.json": { 50 | "fields": ["location.country", "location.region"], 51 | "root": "path.to.list" 52 | } 53 | } 54 | } 55 | } 56 | 57 | or:: 58 | 59 | { 60 | "locker": { 61 | "repo_url": "https://github.com/my-org/my-evidence-repo", 62 | "partitions": { 63 | "foo/evidence_bar.json": { 64 | "fields": ["location.country", "location.region"] 65 | } 66 | } 67 | } 68 | } 69 | 70 | Partition By Evidence 71 | --------------------- 72 | 73 | Turning partitioning on for a given evidence can also be accomplished by 74 | directly constructing your raw evidence object in your fetcher with the 75 | appropriate partitioning parameters. Parameters include the partition key(s) 76 | as a list, and optionally the partition root where the evidence will be 77 | partitioned. For example:: 78 | 79 | RawEvidence( 80 | 'evidence_bar.json', 81 | 'foo', 82 | DAY, 83 | 'My partitioned evidence', 84 | partition={ 85 | 'fields': ['location.country', 'location.region'], 86 | 'root': 'path.to.list' 87 | } 88 | ) 89 | 90 | or:: 91 | 92 | RawEvidence( 93 | 'evidence_bar.json', 94 | 'foo', 95 | DAY, 96 | 'My partitioned evidence', 97 | partition={'fields': ['location.country', 'location.region']} 98 | ) 99 | 100 | **NOTE:** Constructing a partitioned :py:class:`~compliance.evidence.RawEvidence` 101 | object overrides partitioning by configuration. 102 | 103 | Examples 104 | -------- 105 | 106 | Default Partition Root 107 | ~~~~~~~~~~~~~~~~~~~~~~ 108 | 109 | * Unpartitioned evidence content:: 110 | 111 | [ 112 | {"foo": "...", "bar": "...", "country": "US", "region": "IL"}, 113 | {"foo": "...", "bar": "...", "country": "US", "region": "IL"}, 114 | {"foo": "...", "bar": "...", "country": "US", "region": "NY"}, 115 | {"foo": "...", "bar": "...", "country": "UK", "region": "Essex"}, 116 | {"foo": "...", "bar": "...", "country": "UK", "region": "Essex"} 117 | ] 118 | 119 | * Partitioned configuration:: 120 | 121 | { 122 | "locker": { 123 | "repo_url": "https://github.com/my-org/my-evidence-repo", 124 | "partitions": { 125 | "foo/evidence_bar.json": { 126 | "fields": ["country", "region"] 127 | } 128 | } 129 | } 130 | } 131 | 132 | * Partitions yielded 133 | 134 | * US/IL partition - ``foo/_evidence_bar.json``:: 135 | 136 | [ 137 | {"foo": "...", "bar": "...", "country": "US", "region": "IL"}, 138 | {"foo": "...", "bar": "...", "country": "US", "region": "IL"} 139 | ] 140 | 141 | * US/NY partition - ``foo/_evidence_bar.json``:: 142 | 143 | [ 144 | {"foo": "...", "bar": "...", "country": "US", "region": "NY"} 145 | ] 146 | 147 | * UK/Essex partition - ``foo/_evidence_bar.json``:: 148 | 149 | [ 150 | {"foo": "...", "bar": "...", "country": "UK", "region": "Essex"}, 151 | {"foo": "...", "bar": "...", "country": "UK", "region": "Essex"} 152 | ] 153 | 154 | Explicit Partition Root 155 | ~~~~~~~~~~~~~~~~~~~~~~~ 156 | 157 | * Unpartitioned evidence content:: 158 | 159 | { 160 | "nested": { 161 | "good_stuff": [ 162 | {"foo": "...", "bar": "...", "country": "US", "region": "IL"}, 163 | {"foo": "...", "bar": "...", "country": "US", "region": "IL"}, 164 | {"foo": "...", "bar": "...", "country": "US", "region": "NY"}, 165 | {"foo": "...", "bar": "...", "country": "UK", "region": "Essex"}, 166 | {"foo": "...", "bar": "...", "country": "UK", "region": "Essex"} 167 | ], 168 | "nested_other_stuff": "nested meh" 169 | }, 170 | "other_stuff": "other meh" 171 | } 172 | 173 | * Partitioned configuration:: 174 | 175 | { 176 | "locker": { 177 | "repo_url": "https://github.com/my-org/my-evidence-repo", 178 | "partitions": { 179 | "foo/evidence_bar.json": { 180 | "fields": ["country", "region"], 181 | "root": "nested.good_stuff" 182 | } 183 | } 184 | } 185 | } 186 | 187 | * Partitions yielded 188 | 189 | * US/IL partition - ``foo/_evidence.bar.json``:: 190 | 191 | { 192 | "nested": { 193 | "good_stuff": [ 194 | {"foo": "...", "bar": "...", "country": "US", "region": "IL"}, 195 | {"foo": "...", "bar": "...", "country": "US", "region": "IL"} 196 | ], 197 | "nested_other_stuff": "nested meh" 198 | }, 199 | "other_stuff": "other meh" 200 | } 201 | 202 | * US/NY partition - ``foo/_evidence.bar.json``:: 203 | 204 | { 205 | "nested": { 206 | "good_stuff": [ 207 | {"foo": "...", "bar": "...", "country": "US", "region": "NY"} 208 | ], 209 | "nested_other_stuff": "nested meh" 210 | }, 211 | "other_stuff": "other meh" 212 | } 213 | 214 | * UK/Essex partition - ``foo/_evidence.bar.json``:: 215 | 216 | { 217 | "nested": { 218 | "good_stuff": [ 219 | {"foo": "...", "bar": "...", "country": "UK", "region": "Essex"}, 220 | {"foo": "...", "bar": "...", "country": "UK", "region": "Essex"} 221 | ], 222 | "nested_other_stuff": "nested meh" 223 | }, 224 | "other_stuff": "other meh" 225 | } 226 | -------------------------------------------------------------------------------- /doc-source/fixers.rst: -------------------------------------------------------------------------------- 1 | .. -*- mode:rst; coding:utf-8 -*- 2 | 3 | .. _fixers: 4 | 5 | Fixers 6 | ====== 7 | 8 | TBD 9 | -------------------------------------------------------------------------------- /doc-source/index.rst: -------------------------------------------------------------------------------- 1 | .. -*- mode:rst; coding:utf-8 -*- 2 | 3 | Auditree framework documentation 4 | ================================ 5 | 6 | Tool to run compliance control checks as unit tests. 7 | 8 | Installation 9 | ------------ 10 | 11 | For users 12 | ~~~~~~~~~ 13 | 14 | The framework is uploaded to `pypi `_. 15 | You can install it via: 16 | 17 | .. code-block:: bash 18 | 19 | $ pip install auditree-framework 20 | 21 | See the :ref:`quick-start` section for a brief introduction to the 22 | tool usage. Also, see :ref:`running-on-travis` section for getting 23 | information about how to automate the execution in Travis. 24 | 25 | For developers 26 | ~~~~~~~~~~~~~~ 27 | 28 | .. code-block:: bash 29 | 30 | $ git clone git@github.com:ComplianceAsCode/auditree-framework.git 31 | $ cd auditree-framework 32 | $ python3 -m venv venv 33 | $ . venv/bin/activate 34 | $ make install && make develop 35 | 36 | Guides 37 | ------ 38 | 39 | .. toctree:: 40 | :maxdepth: 1 41 | 42 | quick-start 43 | design-principles 44 | oscal 45 | evidence-partitioning 46 | fixers 47 | report-builder 48 | notifiers 49 | coding-standards 50 | running-on-travis 51 | running-on-tekton 52 | verifying-signed-evidence 53 | 54 | Source code 55 | ----------- 56 | 57 | .. toctree:: 58 | :maxdepth: 2 59 | 60 | modules 61 | -------------------------------------------------------------------------------- /doc-source/oscal.rst: -------------------------------------------------------------------------------- 1 | .. -*- mode:rst; coding:utf-8 -*- 2 | 3 | .. _oscal: 4 | 5 | OSCAL Support 6 | ================= 7 | 8 | Coming soon... -------------------------------------------------------------------------------- /doc-source/report-builder.rst: -------------------------------------------------------------------------------- 1 | .. -*- mode:rst; coding:utf-8 -*- 2 | 3 | .. _report-builder: 4 | 5 | Report Builder 6 | ============== 7 | 8 | After tests have been run by the Auditree framework, the 9 | :py:class:`~compliance.report.ReportBuilder` is executed in order to 10 | render all the required reports. This is how it works: 11 | 12 | 1) Execution tree is passed to the ReportBuilder and it iterates over 13 | each ``ComplianceCheck`` object. 14 | 15 | 2) If the test object implements ``get_reports()``, then the method is 16 | called. If it is not implemented, it is assumed that the check is 17 | not required to create any reports. 18 | 19 | 3) If it is implemented, a list is expected with the following types 20 | of elements: 21 | 22 | * Evidence string paths, e.g. ``'reports/category1/evidence1.md'`` 23 | or just ``'category1/evidence1.md'`` (``reports/`` will be 24 | appended automatically). In this case, ``ReportBuilder`` will 25 | look for a Jinja2 template at 26 | ``templates/category1/evidence1.md.tmpl`` and it will render and 27 | store into the locker. See :ref:`templating` section for more 28 | information. 29 | 30 | This is the recommended way of generating reports. 31 | 32 | * :py:class:`~compliance.evidence.ReportEvidence` object: the test 33 | creates them and should populate the content because 34 | ``ReportBuilder`` will expect the ``.content`` property to contain the 35 | content of the report to render. 36 | 37 | This method is not recommended as you will need to generate the 38 | content of the report within the check code, so there will be too 39 | many things there (format rendering, structure, etc.). 40 | 41 | 4) Report evidences are rendered and stored in the locker. If an error occurs 42 | during report generation, ``ReportBuilder`` will skip that repot and 43 | notify that the error occurred, in standard output. Note that report 44 | generation will not halt on an error, and will attempt to generate all other 45 | reports. 46 | 47 | 48 | .. _templating: 49 | 50 | Report templates 51 | ---------------- 52 | 53 | As above, reports can be generated using Jinja2_ 54 | report templates. When a path to report evidence is provided in a 55 | ComplianceCheck object's ``get_reports`` method then the ``ReportBuilder`` will 56 | attempt to locate a template matching the report evidence path provided. For 57 | example if the following is the path provided: 58 | 59 | .. code-block:: python 60 | 61 | def get_reports(): 62 | return ['reports/users/ssh.md'] 63 | 64 | The ``ReportBuilder`` will search for ``reports/users/ssh.md.tmpl`` in the 65 | first ``templates`` directory that it can find. Typically the ``templates`` 66 | directory would be found at the same level as the ``checks`` and ``fetchers`` 67 | directories. For example:: 68 | 69 | fetchers/ 70 | checks/ 71 | templates/ 72 | reports/ 73 | users/ 74 | ssh.md.tmpl 75 | 76 | The content of ``ssh.md.tmpl`` should be follow Jinja2 template syntax. The 77 | following variables are available for use within the template: 78 | 79 | * ``test``: the ComplianceCheck object. 80 | 81 | * ``results``: a dictionary containing the statuses of the checks from the 82 | ``test`` object. 83 | 84 | * ``all_failures``: a dictionary containing failures for each section/type. 85 | 86 | * ``all_warnings``: a dictionary containing warnings for each section/type. 87 | 88 | * ``evidence``: the ReportEvidence object. 89 | 90 | * ``builder``: a reference to the ``ReportBuilder``. 91 | 92 | * ``now``: a ``datetime`` object with the date at render time. 93 | 94 | It is also possible to have a default template and if no template matching 95 | the report evidence path/name provided by ``get_reports`` can be found, 96 | ``ReportBuilder`` will use the default template to generate the report 97 | identified by the report evidence path/name. Typically the default template 98 | would live in the ``template`` directory as ``reports/default.md.tmpl``. 99 | 100 | 101 | .. _Jinja2: http://jinja.pocoo.org/docs/latest/templates/ 102 | -------------------------------------------------------------------------------- /doc-source/running-on-tekton.rst: -------------------------------------------------------------------------------- 1 | .. -*- mode:rst; coding:utf-8 -*- 2 | 3 | .. _running-on-tekton: 4 | 5 | Running on Tekton 6 | ================= 7 | 8 | Coming soon... -------------------------------------------------------------------------------- /doc-source/running-on-travis.rst: -------------------------------------------------------------------------------- 1 | .. -*- mode:rst; coding:utf-8 -*- 2 | 3 | .. _running-on-travis: 4 | 5 | Running on Travis 6 | ================= 7 | 8 | Running the Auditree framework from a CI like Travis can be really useful for 9 | executing your compliance checks periodically. Thus you can track the 10 | current level of compliance for different standards and also notify 11 | people whenever there is a failure, so it can be fixed in some way. 12 | 13 | This can be done in many different ways, so you don't have to follow 14 | this guide if it does not fit your requirements. However, it always 15 | useful to know what it is required so you can adapt this guide to your 16 | needs. 17 | 18 | 19 | Basically, this will what you will need: 20 | 21 | * A ``.travis.yml``: this will define the Travis run which will run 22 | ``travis/run.sh`` script. 23 | 24 | * A git repository for storing generated evidence. You should create a 25 | private project/org for this. 26 | 27 | * Credentials generator: for that, a python script can be used for 28 | generating the credentials files from environment variables defined 29 | in Travis. 30 | 31 | * Results storage: check results are stored to the evidence locker as 32 | ``check_results.json``. 33 | 34 | Bare in mind that a compliance check project is a bit tricky to 35 | configure since you will be pushing new code there and also running 36 | **official** compliance executions. You can resolve this issue by 37 | having 2 Git repositories: one for check development with a 38 | development Travis configuration and another one just for cloning it 39 | and run ``compliance`` with official parameters. 40 | 41 | However, this can be done in the same repository noting that there are 42 | `development` runs (they will not notify nor push any evidence to the 43 | evidence collector repository) and `official` runs (which will send 44 | notifications and push evidences to Git). 45 | 46 | 47 | 48 | Travis artifacts 49 | ---------------- 50 | 51 | This is a typical `.travis.yml` file: 52 | 53 | .. code-block:: yaml 54 | 55 | language: python 56 | python: 57 | - "3.7" 58 | install: 59 | - pip install -r requirements.txt 60 | script: 61 | - make clean 62 | - ./travis/run.sh 63 | 64 | Basically, this will firstly install the dependencies through 65 | ``pip install -r requirements.txt`` and then generate the credentials file from 66 | using Travis environment variables. 67 | 68 | Credentials 69 | ~~~~~~~~~~~ 70 | 71 | The recommended way to use credentials in a CI job is to export them as environment variables. 72 | Auditree will automatically parsed the environment variables available to the process and make them available to the fetchers if they follow a specific structure. 73 | 74 | For more information on how to do this, have a look to the :ref:`credentials` section. 75 | 76 | 77 | ``travis/run.sh`` 78 | ~~~~~~~~~~~~~~~~~ 79 | 80 | Travis will call this script in two different ways: 81 | 82 | * As part of a change in the repo, so it would be considered a 83 | development run. 84 | 85 | * A call through Travis API, made by a cron job (or a robot) 86 | periodically. This will be considered the `official` run. 87 | 88 | This is an example of a ``travis/run.sh`` file: 89 | 90 | .. code-block:: bash 91 | 92 | #!/bin/bash 93 | 94 | NON_OFFICIAL="--evidence no-push --notify stdout" 95 | OFFICIAL="--evidence full-remote --notify slack" 96 | 97 | # is this an official run or not? 98 | if [ "$TRAVIS_BRANCH" == "master" ] && [ -z $TRAVIS_COMMIT_RANGE ]; then 99 | # this is official as it has been run by an external call 100 | OPTIONS="$OFFICIAL" 101 | else 102 | OPTIONS="$NON_OFFICIAL" 103 | fi 104 | 105 | # run fetchers 106 | compliance --fetch $OPTIONS -C official.json 107 | 108 | # run checks 109 | compliance --check $ACCREDITATIONS $OPTIONS -C official.json 110 | retval=$? 111 | 112 | exit $retval 113 | 114 | Note that the arguments used in the ``compliance`` invocation depend 115 | on whether this is an official run or not. This script assumes you 116 | have stored the official configuration into ``official.json`` file: 117 | 118 | .. code-block:: json 119 | 120 | { 121 | "locker": { 122 | "repo_url": "https://github.com/my-org/my-evidence-repo" 123 | }, 124 | "notify": { 125 | "slack": { 126 | "demo.hipaa": ["#security-team", "#hipaa-compliance"], 127 | "demo.soc2": ["#soc2-compliance", "#operations"] 128 | } 129 | } 130 | } 131 | -------------------------------------------------------------------------------- /doc-source/verifying-signed-evidence.rst: -------------------------------------------------------------------------------- 1 | .. -*- mode:rst; coding:utf-8 -*- 2 | 3 | .. _verifying-signed-evidence: 4 | 5 | Verifying Signed Evidence 6 | ========================= 7 | 8 | Follow the instructions to manually verify the sample evidence below:: 9 | 10 | -----BEGIN AGENT----- 11 | auditree.local 12 | -----END AGENT----- 13 | -----BEGIN CONTENT----- 14 | This is my evidence. 15 | -----END CONTENT----- 16 | -----BEGIN DIGEST----- 17 | 81ddd37cb8aba90077a717b7d6c067815add58e658bb2be0dea4d4d9301c762d 18 | -----END DIGEST----- 19 | -----BEGIN SIGNATURE----- 20 | xRIu2dey1WSCSRpBWHlar5XUv13vZtm1n/KEDckA85UoQjEqEo7xlmnpzBtkNcieME6frhBMmBOYPW4uFYS1EUtLxkixYkYjt3wKlHl8CkvKDFoqAMqG8AC/cCdqwP7D7SlO5RH1pJ1kp2yX2XB2MTMHkd/9tguNZBpaCnscYCmpBvng6okB7HbToOlVUfKY1tWDDIm3JefFMEoJqXgIEZMmVnF+nLniF/PvPTL+q38e6Wd1xeJpZYiLk12imarzkf9MweA5D22xkv51pI2ils3jovxymzio26cSkL7iHBsbiNOWWXoETo0aYm2g9CzhxnRGku9OEkW97JGNASkjSw== 21 | -----END SIGNATURE----- 22 | 23 | 1. Fetch the public key for the ``auditree.local`` agent. This can be collected 24 | directly from the agent or the public keys evidence (available in the locker 25 | under ``raw/auditree/agent_public_keys.json``):: 26 | 27 | $ cat > key.pub 28 | -----BEGIN PUBLIC KEY----- 29 | MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxYosRYnahnSuH3SmNupn 30 | zQhxJsDEhqChKjrcyN19L8+vcjUUiMSaKRoAHuUKp5Pfwkoylryd4AyXIU9UnXZg 31 | dIOl2+r5xzXqfdLwi+PAU/eEWPLAQfCpIodqKqBLCyzpMoJHv9GDqg8XJkY/2i8j 32 | 7oiqLR7vibIgRAJXqF95KdNvbW7Gvu8JHigN4aoGdbQSPp/jJ30wBvy7hHOSrMWF 33 | iQUt7H25YbvOZGWQeC8HZ2EXruzG+FV2rkW52FaTn31lX1EEc2Yz8AI7/yF/8C5j 34 | SSL/pmzxBzh/P4zGDNlm2habpwAIQpHnJJ8XeXYS//RXuOYNObeRwfhm82TB9+nS 35 | lQIDAQAB 36 | -----END PUBLIC KEY----- 37 | 38 | 2. Save the evidence content to your local filesystem and verify the digest:: 39 | 40 | $ cat > evidence.txt 41 | This is my evidence. 42 | 43 | $ openssl dgst -sha256 evidence.txt 44 | SHA256(evidence.txt)= 81ddd37cb8aba90077a717b7d6c067815add58e658bb2be0dea4d4d9301c762d 45 | 46 | *Be sure not to add any additional whitespace when saving evidence locally.* 47 | 48 | 3. Save the signature to your local filesystem:: 49 | 50 | $ cat > signature.txt 51 | xRIu2dey1WSCSRpBWHlar5XUv13vZtm1n/KEDckA85UoQjEqEo7xlmnpzBtkNcieME6frhBMmBOYPW4uFYS1EUtLxkixYkYjt3wKlHl8CkvKDFoqAMqG8AC/cCdqwP7D7SlO5RH1pJ1kp2yX2XB2MTMHkd/9tguNZBpaCnscYCmpBvng6okB7HbToOlVUfKY1tWDDIm3JefFMEoJqXgIEZMmVnF+nLniF/PvPTL+q38e6Wd1xeJpZYiLk12imarzkf9MweA5D22xkv51pI2ils3jovxymzio26cSkL7iHBsbiNOWWXoETo0aYm2g9CzhxnRGku9OEkW97JGNASkjSw== 52 | 53 | 4. Convert the Base64 signature to binary:: 54 | 55 | $ openssl base64 -d -in signature.txt -out evidence.sig 56 | 57 | 5. Verify the signature:: 58 | 59 | $ openssl dgst -sha256 -verify key.pub -signature evidence.sig -sigopt rsa_padding_mode:pss evidence.txt 60 | Verified OK 61 | 62 | If the verification is successful, the OpenSSL command will print the 63 | ``Verified OK`` message, otherwise it will print ``Verification Failure``. 64 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [metadata] 2 | name = auditree-framework 3 | version = attr: compliance.__version__ 4 | description = Tool to run compliance control checks as unit tests 5 | author = Auditree Security and Compliance 6 | author_email = al.finkelstein@ibm.com 7 | url = https://auditree.github.io/ 8 | license = Apache License 2.0 9 | classifiers = 10 | Programming Language :: Python :: 3.8 11 | Programming Language :: Python :: 3.9 12 | Programming Language :: Python :: 3.10 13 | Programming Language :: Python :: 3.11 14 | Programming Language :: Python :: 3.12 15 | License :: OSI Approved :: Apache Software License 16 | Operating System :: MacOS :: MacOS X 17 | Operating System :: POSIX :: Linux 18 | long_description_content_type = text/markdown 19 | long_description = file: README.md 20 | 21 | [options] 22 | include_package_data = True 23 | packages = find: 24 | install_requires = 25 | inflection>=0.3.1 26 | GitPython>=2.1.3 27 | jinja2>=2.10 28 | ilcli>=0.3.1 29 | cryptography>=35.0.0 30 | requests>=2.30.0 31 | 32 | [options.packages.find] 33 | exclude = 34 | test.* 35 | test 36 | demo 37 | 38 | [bdist_wheel] 39 | universal = 1 40 | 41 | [options.entry_points] 42 | console_scripts = 43 | compliance = compliance.scripts.compliance_cli:run 44 | 45 | [options.extras_require] 46 | dev = 47 | pytest>=5.4.3 48 | pytest-cov>=2.10.0 49 | pre-commit>=2.4.0 50 | Sphinx>=1.7.2 51 | setuptools 52 | wheel 53 | twine 54 | 55 | [flake8] 56 | max-line-length = 88 57 | extend-ignore = E203 58 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | from setuptools import setup 16 | 17 | setup() 18 | -------------------------------------------------------------------------------- /test/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance automation tests package.""" 15 | -------------------------------------------------------------------------------- /test/fixtures/controls/original/controls.json: -------------------------------------------------------------------------------- 1 | { 2 | "foo_pkg.checks.test_foo_module.FooCheck": { 3 | "foo_evidence": { 4 | "foo_control": ["accred.foo"] 5 | } 6 | } 7 | } 8 | -------------------------------------------------------------------------------- /test/fixtures/controls/simplified/controls.json: -------------------------------------------------------------------------------- 1 | {"bar_pkg.checks.test_bar_module.BarCheck": ["accred.bar"]} 2 | -------------------------------------------------------------------------------- /test/t_compliance/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance automation compliance tests package.""" 15 | 16 | from unittest.mock import MagicMock, Mock, PropertyMock 17 | 18 | from compliance.check import ComplianceCheck 19 | 20 | 21 | def build_test_mock(name="one", baseurl="http://mockedrunbooks", fails=0, warns=0): 22 | """Build a mock of a ComplianceCheck, with minimal attributes set.""" 23 | mock_test = MagicMock() 24 | if baseurl: 25 | type(mock_test.test).runbook_url = PropertyMock( 26 | return_value=f"{baseurl}/path/to/runbook_{name}" 27 | ) 28 | type(mock_test.test).enabled = PropertyMock(return_value=True) 29 | else: 30 | type(mock_test.test).runbook_url = PropertyMock(return_value="") 31 | type(mock_test.test).enabled = PropertyMock(return_value=False) 32 | 33 | type(mock_test.test).title = PropertyMock(return_value=f"mock check title {name}") 34 | 35 | mock_test.test.failures_count = Mock(return_value=fails) 36 | mock_test.test.warnings_count = Mock(return_value=warns) 37 | 38 | mock_test.test.__str__.return_value = f"test_{name}" 39 | mock_test.test.fixed_failure_count = 0 40 | return mock_test 41 | 42 | 43 | def build_compliance_check(class_name, title, report, tests=None, fix_return=True): 44 | """ 45 | Build an actual :py:class:`compliance.check.ComplianceCheck` subclass. 46 | 47 | Class is based on the specific attributes given. 48 | 49 | If multiple tests are provided, then corresponding fix functions 50 | will be created; otherwise, just a single fix_failures() function 51 | will be created. 52 | 53 | All fix functions will be set up to call a single function called 54 | fix_one(), passing the name of the test as the parameter param. 55 | 56 | :param class_name: the name of the class to build 57 | :param title: the title property of the class 58 | :param report: the string that the get_reports() function 59 | should return 60 | :param tests: list of strings of test functions to add to 61 | the class. each one will simply assert a true statement 62 | and return. 63 | :param fix_return: boolean specifying whether the fix function 64 | should return True (indicating it fixed the issue) or False 65 | (indicating it tried but failed) 66 | """ 67 | tests = tests or [] 68 | fcts = { 69 | "title": property(lambda self: title), 70 | "get_reports": lambda self: [report], 71 | "fix_one": MagicMock(return_value=fix_return), 72 | "tests": property(lambda self: tests), 73 | } 74 | 75 | for test in tests: 76 | fcts[test] = lambda self: self.assertEqual(0, 0) 77 | fix_fct = "fix_failures" 78 | if len(tests) > 1: 79 | fix_fct = test.replace("test_", "fix_") 80 | fcts[fix_fct] = lambda self, fixer, t=test: ( 81 | fixer.execute_fix(self, self.fix_one, {"param": t}) 82 | ) 83 | 84 | cls = type(class_name, (ComplianceCheck,), fcts) 85 | vars(cls)["fix_one"].__doc__ = "Fixing {param}" 86 | 87 | return cls 88 | 89 | 90 | def build_compliance_check_obj(*args, **kwargs): 91 | """ 92 | Build a :py:class:`compliance.check.ComplianceCheck` class. 93 | 94 | Class built using build_compliance_check(), and then create a 95 | single object from this class and returns it. 96 | """ 97 | cls = build_compliance_check(*args, **kwargs) 98 | obj = cls("__doc__") 99 | return obj 100 | -------------------------------------------------------------------------------- /test/t_compliance/t_agent/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2022 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance automation agent tests package.""" 15 | -------------------------------------------------------------------------------- /test/t_compliance/t_check/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance automation check tests package.""" 15 | -------------------------------------------------------------------------------- /test/t_compliance/t_check/test_base_check.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance automation check tests module.""" 15 | 16 | import unittest 17 | from datetime import datetime 18 | from unittest.mock import MagicMock, call, create_autospec 19 | 20 | from compliance.check import ComplianceCheck 21 | from compliance.config import ComplianceConfig 22 | from compliance.locker import Locker 23 | 24 | from git import Commit 25 | 26 | 27 | class ComplianceCheckTest(unittest.TestCase): 28 | """ComplianceCheck test class.""" 29 | 30 | def setUp(self): 31 | """Initialize each test.""" 32 | # Since unittest.TestCase needs a method for running the test 33 | # (runTest, by default) and ComplianceCheck is a child of 34 | # unittest.TestCase, we must pass a method in the 35 | # constructor (otherwise, we will get a ValueError). Since we 36 | # don't need this method, passing ``__doc__`` is enough for 37 | # building a ComplianceCheck object successfully. 38 | self.check = ComplianceCheck("__doc__") 39 | 40 | # Ensures that the check object has a (mocked) locker attribute/object 41 | # on it as expected. 42 | self.check.locker = create_autospec(Locker) 43 | self.check.locker.repo_url = "https://my.locker.url" 44 | 45 | def test_title(self): 46 | """Check title raises an exception in the base class.""" 47 | with self.assertRaises(NotImplementedError) as cm: 48 | self.check.title 49 | self.assertEqual( 50 | str(cm.exception), "Property title not implemented on ComplianceCheck" 51 | ) 52 | 53 | def test_config(self): 54 | """Check that the config property returns a ComplianceConfig object.""" 55 | self.assertIsInstance(self.check.config, ComplianceConfig) 56 | 57 | def test_reports(self): 58 | """Check reports property.""" 59 | self.assertEqual(self.check.reports, []) 60 | self.check.reports.append("dummy") 61 | self.assertEqual(self.check.reports, ["dummy"]) 62 | 63 | def test_disabled_runbook_url(self): 64 | """Check runbook URL is none - disabled.""" 65 | self.check.config._config.update( 66 | {"runbooks": {"enabled": False, "base_url": "http://configuredrunbooks"}} 67 | ) 68 | self.assertEqual(self.check.runbook_url, None) 69 | 70 | def test_unconfigured_runbook_url(self): 71 | """Check runbook URL is none - not configured.""" 72 | self.check.config._config.update( 73 | {"runbooks": {"enabled": True, "base_url": ""}} 74 | ) 75 | self.assertEqual(self.check.runbook_url, None) 76 | 77 | def test_configured_runbook_url(self): 78 | """Check runbook URL is set.""" 79 | self.check.config._config.update( 80 | {"runbooks": {"enabled": True, "base_url": "http://configuredrunbooks"}} 81 | ) 82 | self.assertEqual( 83 | self.check.runbook_url, "http://configuredrunbooks/compliance_check.html" 84 | ) 85 | 86 | def test_evidence_metadata(self): 87 | """Check evidence_metadata property.""" 88 | self.assertEqual(self.check.evidence_metadata, {}) 89 | 90 | def test_fixed_failure_count(self): 91 | """Check fixed_failure_count property.""" 92 | self.assertEqual(self.check.fixed_failure_count, 0) 93 | self.check.fixed_failure_count = 100 94 | self.assertEqual(self.check.fixed_failure_count, 100) 95 | 96 | def test_failures(self): 97 | """Test failures property, and the length of dict and of type.""" 98 | self.assertEqual(self.check.failures, {}) 99 | self.check.add_failures("fail_type", "fail_for") 100 | self.check.add_failures("fail_type_2", "fail_for_2") 101 | expected_failure = {"fail_type": ["fail_for"], "fail_type_2": ["fail_for_2"]} 102 | self.assertEqual(expected_failure, self.check.failures) 103 | self.assertEqual(self.check.failures_count(), 2) 104 | 105 | def test_warnings(self): 106 | """Test warning property and if key does not exist, throws KeyError.""" 107 | self.check._failures = {} 108 | self.assertEqual(self.check.warnings, {}) 109 | self.check.add_warnings("warn_type", "warn_for") 110 | expected_warning = {"warn_type": ["warn_for"]} 111 | self.assertEqual(expected_warning, self.check.warnings) 112 | 113 | def test_add_issue_if_diff_failure(self): 114 | """Test add_issue_if_diff adds a failure as expected.""" 115 | # Throw a fail and make sure it did not warn 116 | self.check.add_issue_if_diff({1, 2, 3, 5}, {1, 2, 3, 4}, "Extra users found") 117 | self.assertEqual(self.check.failures_count(), 1) 118 | self.assertEqual(self.check.warnings_count(), 0) 119 | self.assertEqual(self.check._failures, {"Extra users found": [5]}) 120 | 121 | def test_add_issue_if_diff_warning(self): 122 | """Test add_issue_if_diff adds a warning as expected.""" 123 | # Throw a fail and make sure it did not warn 124 | self.check.add_issue_if_diff( 125 | {1, 2, 3, 4}, {1, 2, 3, 5}, "Users not found", True 126 | ) 127 | self.assertEqual(self.check.failures_count(), 0) 128 | self.assertEqual(self.check.warnings_count(), 1) 129 | self.assertEqual(self.check._warnings, {"Users not found": [4]}) 130 | 131 | def test_add_issue_if_diff_no_diff(self): 132 | """Test add_issue_if_diff does not add a fail/warning when no diff.""" 133 | # Ensure no issues are raised when there is no diff 134 | self.check.add_issue_if_diff([], [], "FAILED") 135 | self.assertEqual(self.check.failures_count(), 0) 136 | self.assertEqual(self.check.warnings_count(), 0) 137 | 138 | def test_add_evidence_metadata(self): 139 | """Test evidence_metadata is populated correctly.""" 140 | commit_mock = create_autospec(Commit) 141 | commit_mock.hexsha = "mycommitsha" 142 | self.check.locker.get_latest_commit = MagicMock() 143 | self.check.locker.get_latest_commit.return_value = commit_mock 144 | self.check.locker.get_evidence_metadata = MagicMock() 145 | self.check.locker.get_evidence_metadata.return_value = { 146 | "foo": "bar", 147 | "last_update": "2019-11-15", 148 | } 149 | ev_date = datetime(2019, 11, 15) 150 | 151 | self.check.add_evidence_metadata("raw/foo/foo.json", ev_date) 152 | 153 | self.check.locker.get_latest_commit.assert_called_once_with( 154 | "raw/foo/foo.json", ev_date 155 | ) 156 | self.check.locker.get_evidence_metadata.assert_called_once_with( 157 | "raw/foo/foo.json", ev_date 158 | ) 159 | self.assertEqual( 160 | self.check.evidence_metadata, 161 | { 162 | ("raw/foo/foo.json", "2019-11-15"): { 163 | "path": "raw/foo/foo.json", 164 | "commit_sha": "mycommitsha", 165 | "foo": "bar", 166 | "last_update": "2019-11-15", 167 | "locker_url": "https://my.locker.url", 168 | } 169 | }, 170 | ) 171 | 172 | def test_add_partitioned_evidence_metadata(self): 173 | """Test evidence_metadata is populated correctly for partitions.""" 174 | commit_mock = create_autospec(Commit) 175 | commit_mock.hexsha = "mycommitsha" 176 | self.check.locker.get_latest_commit = MagicMock() 177 | self.check.locker.get_latest_commit.return_value = commit_mock 178 | self.check.locker.get_evidence_metadata = MagicMock() 179 | self.check.locker.get_evidence_metadata.return_value = { 180 | "foo": "bar", 181 | "last_update": "2019-11-15", 182 | "partitions": {"123": ["foo"], "456": ["bar"]}, 183 | "tombstones": "zombie", 184 | } 185 | ev_date = datetime(2019, 11, 15) 186 | 187 | self.check.add_evidence_metadata("raw/foo/foo.json", ev_date) 188 | 189 | self.assertEqual(self.check.locker.get_latest_commit.call_count, 2) 190 | self.check.locker.get_latest_commit.assert_has_calls( 191 | [ 192 | call("raw/foo/123_foo.json", ev_date), 193 | call("raw/foo/456_foo.json", ev_date), 194 | ], 195 | any_order=True, 196 | ) 197 | self.check.locker.get_evidence_metadata.assert_called_once_with( 198 | "raw/foo/foo.json", ev_date 199 | ) 200 | self.assertEqual( 201 | self.check.evidence_metadata, 202 | { 203 | ("raw/foo/foo.json", "2019-11-15"): { 204 | "path": "raw/foo/foo.json", 205 | "partitions": { 206 | "123": {"key": ["foo"], "commit_sha": "mycommitsha"}, 207 | "456": {"key": ["bar"], "commit_sha": "mycommitsha"}, 208 | }, 209 | "foo": "bar", 210 | "last_update": "2019-11-15", 211 | "locker_url": "https://my.locker.url", 212 | } 213 | }, 214 | ) 215 | 216 | def test_has_assertEquals(self): 217 | """Test assertEquals is still present""" 218 | self.check.assertEquals(1, 1) 219 | -------------------------------------------------------------------------------- /test/t_compliance/t_controls/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance automation control descriptor tests module.""" 15 | -------------------------------------------------------------------------------- /test/t_compliance/t_controls/test_controls.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance automation control descriptor tests module.""" 15 | 16 | import unittest 17 | import pytest 18 | 19 | from compliance.controls import ControlDescriptor 20 | 21 | 22 | class ControlDescriptorTest(unittest.TestCase): 23 | """ControlDescriptor test class.""" 24 | 25 | def setUp(self): 26 | """Initialize each test.""" 27 | cd_paths = [ 28 | "./test/fixtures/controls/original", 29 | "./test/fixtures/controls/simplified", 30 | "./faker", 31 | ] 32 | self.cd = ControlDescriptor(cd_paths) 33 | self.expected_foo_check = "foo_pkg.checks.test_foo_module.FooCheck" 34 | self.expected_bar_check = "bar_pkg.checks.test_bar_module.BarCheck" 35 | 36 | def test_contructor_and_base_properties(self): 37 | """Check ControlDescriptor constructed as expected.""" 38 | self.assertEqual(len(self.cd.paths), 2) 39 | expected_ends_with = [ 40 | "/test/fixtures/controls/original/controls.json", 41 | "/test/fixtures/controls/simplified/controls.json", 42 | ] 43 | self.assertNotEqual(self.cd.paths[0], expected_ends_with[0]) 44 | self.assertTrue(self.cd.paths[0].endswith(expected_ends_with[0])) 45 | self.assertNotEqual(self.cd.paths[1], expected_ends_with[1]) 46 | self.assertTrue(self.cd.paths[1].endswith(expected_ends_with[1])) 47 | self.assertEqual( 48 | self.cd.as_dict, 49 | { 50 | self.expected_foo_check: { 51 | "foo_evidence": {"foo_control": ["accred.foo"]} 52 | }, 53 | self.expected_bar_check: ["accred.bar"], 54 | }, 55 | ) 56 | 57 | def test_as_dict_immutability(self): 58 | """Ensure that control content cannot be changed through as_dict.""" 59 | with pytest.raises(AttributeError): 60 | self.cd.as_dict = {"foo": "bar"} 61 | controls_copy = self.cd.as_dict 62 | self.assertEqual(controls_copy, self.cd.as_dict) 63 | controls_copy.update({"foo": "bar"}) 64 | self.assertNotEqual(controls_copy, self.cd.as_dict) 65 | 66 | def test_accred_checks(self): 67 | """Check that checks are organized by accreditations correctly.""" 68 | self.assertEqual( 69 | self.cd.accred_checks, 70 | { 71 | "accred.foo": {self.expected_foo_check}, 72 | "accred.bar": {self.expected_bar_check}, 73 | }, 74 | ) 75 | 76 | def test_get_accreditations(self): 77 | """Ensure the correct accreditations are returned based on check.""" 78 | self.assertEqual( 79 | self.cd.get_accreditations(self.expected_foo_check), {"accred.foo"} 80 | ) 81 | self.assertEqual( 82 | self.cd.get_accreditations(self.expected_bar_check), {"accred.bar"} 83 | ) 84 | 85 | def test_is_test_included(self): 86 | """Test check is included in accreditations functionality.""" 87 | self.assertTrue( 88 | self.cd.is_test_included(self.expected_foo_check, ["accred.foo"]) 89 | ) 90 | self.assertFalse( 91 | self.cd.is_test_included(self.expected_foo_check, ["accred.bar"]) 92 | ) 93 | self.assertTrue( 94 | self.cd.is_test_included(self.expected_bar_check, ["accred.bar"]) 95 | ) 96 | self.assertFalse( 97 | self.cd.is_test_included(self.expected_bar_check, ["accred.foo"]) 98 | ) 99 | self.assertTrue( 100 | self.cd.is_test_included( 101 | self.expected_foo_check, ["accred.foo", "accred.bar"] 102 | ) 103 | ) 104 | self.assertTrue( 105 | self.cd.is_test_included( 106 | self.expected_bar_check, ["accred.foo", "accred.bar"] 107 | ) 108 | ) 109 | self.assertFalse( 110 | self.cd.is_test_included(self.expected_foo_check, ["accred.baz"]) 111 | ) 112 | -------------------------------------------------------------------------------- /test/t_compliance/t_evidence/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance automation evidence tests package.""" 15 | -------------------------------------------------------------------------------- /test/t_compliance/t_evidence/test_evidence.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | from unittest.mock import MagicMock, create_autospec 3 | 4 | from compliance.evidence import store_derived_evidence 5 | from compliance.locker import Locker 6 | from compliance.utils.exceptions import DependencyUnavailableError, StaleEvidenceError 7 | 8 | 9 | class TestEvidence(unittest.TestCase): 10 | 11 | def test_store_derived_evidence_adds_to_rerun(self): 12 | """ 13 | Ensure that when running a fetcher that stores derived evidence 14 | that it is re-run if one of the dependant evidence is not available. 15 | """ 16 | self.locker = create_autospec(Locker) 17 | self.locker.dependency_rerun = [] 18 | self.locker.validate.side_effect = [False, StaleEvidenceError] 19 | self.locker.repo_url = "https://my.locker.url" 20 | self.locker.get_evidence_metadata = MagicMock() 21 | self.locker.get_evidence_metadata.return_value = None 22 | self.locker.get_evidence.side_effect = StaleEvidenceError 23 | 24 | with self.assertRaises(DependencyUnavailableError): 25 | self.fetch_some_derived_evidence() 26 | 27 | self.assertEquals(1, len(self.locker.dependency_rerun)) 28 | f = self.fetch_some_derived_evidence 29 | self.assertDictEqual( 30 | self.locker.dependency_rerun[0], 31 | { 32 | "module": f.__module__, 33 | "class": self.__class__.__name__, 34 | "method": f.__name__, 35 | }, 36 | ) 37 | 38 | @store_derived_evidence(["raw/cos/cos_bucket_metadata.json"], target="cos/bar.json") 39 | def fetch_some_derived_evidence(self, cos_metadata): 40 | return "{}" 41 | -------------------------------------------------------------------------------- /test/t_compliance/t_fetch/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance automation fetcher tests package.""" 15 | -------------------------------------------------------------------------------- /test/t_compliance/t_fetch/test_base_fetcher.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance automation fetcher base tests module.""" 15 | 16 | import unittest 17 | 18 | from compliance.config import ComplianceConfig 19 | from compliance.fetch import ComplianceFetcher 20 | from compliance.utils.http import BaseSession 21 | 22 | import requests 23 | 24 | 25 | class ComplianceFetchTest(unittest.TestCase): 26 | """ComplianceFetcher base test class.""" 27 | 28 | def setUp(self): 29 | """Initialize each test.""" 30 | # Since unittest.TestCase needs a method for running the test 31 | # (runTest, by default) and ComplianceFetcher is a child of 32 | # unittest.TestCase, we must pass a method in the 33 | # constructor (otherwise, we will get a ValueError). Since we 34 | # don't need this method, passing ``__doc__`` is enough for 35 | # building a ComplianceFetcher object successfully. 36 | ComplianceFetcher.config = ComplianceConfig() 37 | self.fetcher = ComplianceFetcher("__doc__") 38 | self.fetcher.config.load() 39 | 40 | def test_config(self): 41 | """Check that the config property returns a ComplianceConfig object.""" 42 | self.assertIsInstance(self.fetcher.config, ComplianceConfig) 43 | 44 | def test_session(self): 45 | """Ensure that a session is constructed correctly.""" 46 | # Create a requests.Session 47 | self.assertIsInstance(self.fetcher.session(), requests.Session) 48 | # Recycle session, create a BaseSession and persist it 49 | self.assertIsInstance( 50 | self.fetcher.session( 51 | "https://foo.bar.com", ("foo", "bar"), foo="FOO", bar="BAR" 52 | ), 53 | BaseSession, 54 | ) 55 | self.assertEqual(self.fetcher.session().baseurl, "https://foo.bar.com") 56 | self.assertEqual(self.fetcher.session().auth, ("foo", "bar")) 57 | self.assertEqual( 58 | self.fetcher.session().headers["User-Agent"], "your_org-compliance-checks" 59 | ) 60 | self.assertEqual(self.fetcher.session().headers["foo"], "FOO") 61 | self.assertEqual(self.fetcher.session().headers["bar"], "BAR") 62 | -------------------------------------------------------------------------------- /test/t_compliance/t_fix/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance automation fixer tests package.""" 15 | -------------------------------------------------------------------------------- /test/t_compliance/t_fix/test_fixer.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance automation fixer tests module.""" 15 | 16 | import time 17 | import unittest 18 | from io import StringIO 19 | from unittest.mock import patch 20 | 21 | from compliance.check import ComplianceCheck 22 | from compliance.fix import Fixer 23 | from compliance.runners import ComplianceTestWrapper 24 | 25 | from .. import build_compliance_check_obj 26 | 27 | 28 | class ComplianceFixerTest(unittest.TestCase): 29 | """Compliance Fixer test class.""" 30 | 31 | @patch("compliance.fix.get_config") 32 | def setUp(self, config_mock): 33 | """Initialize each test.""" 34 | compliance_check1 = build_compliance_check_obj( 35 | "Dummy1Check", "dummy1", "Report for dummy1", ["test_dummy1_test1"], True 36 | ) 37 | compliance_check2 = build_compliance_check_obj( 38 | "Dummy2Check", "dummy2", "Report for dummy2", ["test_dummy2_test1"], True 39 | ) 40 | compliance_check3 = build_compliance_check_obj( 41 | "Dummy3Check", 42 | "dummy3", 43 | "Report for dummy3", 44 | ["test_dummy3_test1", "test_dummy3_test2"], 45 | True, 46 | ) 47 | compliance_check4 = build_compliance_check_obj( 48 | "Dummy4Check", "dummy4", "Report for dummy4", ["test_dummy4_test1"], False 49 | ) 50 | compliance_check5 = build_compliance_check_obj( 51 | "Dummy5Check", "dummy5", "Report for dummy5", [], False 52 | ) 53 | 54 | test1_obj = ComplianceTestWrapper(compliance_check1) 55 | test2_obj = ComplianceTestWrapper(compliance_check2) 56 | test3_obj = ComplianceTestWrapper(compliance_check3) 57 | test4_obj = ComplianceTestWrapper(compliance_check4) 58 | test5_obj = ComplianceTestWrapper(compliance_check5) 59 | 60 | results_empty = {} 61 | results_full = { 62 | "compliance.dummy_accred.test_dummy1_test1": { 63 | "status": "pass", 64 | "timestamp": time.time(), 65 | "test": test1_obj, 66 | }, 67 | "compliance.dummy_accred.test_dummy2_test1": { 68 | "status": "error", 69 | "timestamp": time.time(), 70 | "test": test2_obj, 71 | }, 72 | "compliance.dummy_accred.test_dummy3_test1": { 73 | "status": "fail", 74 | "timestamp": time.time(), 75 | "test": test3_obj, 76 | }, 77 | "compliance.dummy_accred.test_dummy3_test2": { 78 | "status": "fail", 79 | "timestamp": time.time(), 80 | "test": test3_obj, 81 | }, 82 | "compliance.dummy_accred.test_dummy4_test1": { 83 | "status": "fail", 84 | "timestamp": time.time(), 85 | "test": test4_obj, 86 | }, 87 | # this test doesn't actually exist (check when 88 | # no tests defined in a class) 89 | "compliance.dummy_accred.test_dummy5_test0": { 90 | "status": "fail", 91 | "timestamp": time.time(), 92 | "test": test5_obj, 93 | }, 94 | } 95 | 96 | self.empty_results_out = StringIO() 97 | self.empty_results_fixer = Fixer( 98 | results_empty, dry_run=False, out=self.empty_results_out 99 | ) 100 | 101 | self.real_out = StringIO() 102 | self.real_fixer = Fixer(results_full, dry_run=False, out=self.real_out) 103 | 104 | self.dry_run_out = StringIO() 105 | self.dry_run_fixer = Fixer(results_full, dry_run=True, out=self.dry_run_out) 106 | 107 | def test_empty_results_fix(self): 108 | """Check that the fixer does nothing if there are no results.""" 109 | self.empty_results_fixer.fix() 110 | 111 | self.assertEqual(self.empty_results_out.getvalue(), "") 112 | 113 | def test_real_fix(self): 114 | """Check that the fixer works when not in dry-run mode.""" 115 | self.real_fixer.fix() 116 | 117 | self.assertEqual(self.real_out.getvalue(), "") 118 | self.assertEqual( 119 | sorted(self.dry_run_fixer._results.keys()), 120 | [ 121 | "compliance.dummy_accred.test_dummy1_test1", 122 | "compliance.dummy_accred.test_dummy2_test1", 123 | "compliance.dummy_accred.test_dummy3_test1", 124 | "compliance.dummy_accred.test_dummy3_test2", 125 | "compliance.dummy_accred.test_dummy4_test1", 126 | "compliance.dummy_accred.test_dummy5_test0", 127 | ], 128 | ) 129 | 130 | for k in sorted(self.real_fixer._results.keys()): 131 | method_name = k.split(".")[-1] 132 | result = self.real_fixer._results[k] 133 | status = result["status"] 134 | test = result["test"].test 135 | 136 | self.assertTrue(issubclass(test.__class__, ComplianceCheck)) 137 | if status == "fail": 138 | # dummy5 doesn't actually have any tests defined 139 | if test.title == "dummy5": 140 | test.fix_one.assert_not_called() 141 | else: 142 | test.fix_one.assert_any_call( 143 | creds=self.real_fixer._creds, param=method_name 144 | ) 145 | 146 | # dummy4's fix function has been set to return False 147 | if test.title == "dummy4": 148 | self.assertEqual(test.fixed_failure_count, 0) 149 | else: 150 | self.assertEqual(test.fixed_failure_count, len(test.tests)) 151 | else: 152 | test.fix_one.assert_not_called 153 | self.assertEqual(test.fixed_failure_count, 0) 154 | 155 | def test_dry_run_fix(self): 156 | """Check that the fixer works in dry-run mode.""" 157 | self.dry_run_fixer.fix() 158 | 159 | # only check things not already checked in the real fixer test above 160 | 161 | out_msgs = sorted(self.dry_run_out.getvalue().strip("\n").split("\n")) 162 | self.assertEqual( 163 | out_msgs, 164 | [ 165 | "DRY-RUN: Fixing test_dummy3_test1", 166 | "DRY-RUN: Fixing test_dummy3_test2", 167 | "DRY-RUN: Fixing test_dummy4_test1", 168 | ], 169 | ) 170 | 171 | for k in sorted(self.dry_run_fixer._results.keys()): 172 | result = self.dry_run_fixer._results[k] 173 | status = result["status"] 174 | test = result["test"].test 175 | 176 | if status == "fail": 177 | test.fix_one.assert_not_called 178 | self.assertEqual(test.fixed_failure_count, 0) 179 | -------------------------------------------------------------------------------- /test/t_compliance/t_locker/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance automation locker tests package.""" 15 | -------------------------------------------------------------------------------- /test/t_compliance/t_notify/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance automation notifier tests package.""" 15 | -------------------------------------------------------------------------------- /test/t_compliance/t_notify/test_base_notify.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance automation notifier base tests module.""" 15 | 16 | import unittest 17 | from unittest.mock import create_autospec, patch 18 | 19 | from compliance.config import ComplianceConfig 20 | from compliance.controls import ControlDescriptor 21 | from compliance.notify import _BaseNotifier 22 | 23 | from .. import build_test_mock 24 | 25 | 26 | class BaseNotifierTest(unittest.TestCase): 27 | """Base notifier test class.""" 28 | 29 | def _test_url(self, test_desc, msg, notifier, expected=True): 30 | test_name = str(test_desc["test"].test).split("_", 1)[1] 31 | summary, addl_content = notifier._get_summary_and_body(test_desc, msg) 32 | test_url = f"http://mockedrunbooks/path/to/runbook_{test_name}" 33 | if expected: 34 | self.assertIn(f" | <{test_url}|Run Book>", summary) 35 | else: 36 | self.assertNotIn(f" | <{test_url}|Run Book>", summary) 37 | 38 | @patch("compliance.notify.get_config") 39 | def test_notify_with_runbooks(self, get_config_mock): 40 | """Check that _BaseNotifier notifications have runbook links.""" 41 | config_mock = create_autospec(ComplianceConfig) 42 | config_mock.get.return_value = {"infra": ["#channel"]} 43 | 44 | get_config_mock.return_value = config_mock 45 | 46 | results = { 47 | "compliance.test.runbook": {"status": "error", "test": build_test_mock()}, 48 | "compliance.test.other_runbook": { 49 | "status": "error", 50 | "test": build_test_mock("two"), 51 | }, 52 | } 53 | controls = create_autospec(ControlDescriptor) 54 | controls.get_accreditations.return_value = ["infra"] 55 | notifier = _BaseNotifier(results, controls, push_error=False) 56 | 57 | (_, _, _, errored_tests) = notifier._split_by_status(notifier.messages) 58 | 59 | for _, test_desc, msg in errored_tests: 60 | self._test_url(test_desc, msg, notifier, expected=True) 61 | 62 | @patch("compliance.notify.get_config") 63 | def test_notify_without_runbooks(self, get_config_mock): 64 | """Check that _BaseNotifier notifications have no runbook links.""" 65 | config_mock = create_autospec(ComplianceConfig) 66 | config_mock.get.return_value = { 67 | "infra": ["#channel"], 68 | "runbooks": {"base_url": "http://myrunbooks.io"}, 69 | } 70 | 71 | get_config_mock.return_value = config_mock 72 | 73 | results = { 74 | "compliance.test.runbook": { 75 | "status": "error", 76 | "test": build_test_mock(baseurl=""), 77 | }, 78 | "compliance.test.other_runbook": { 79 | "status": "error", 80 | "test": build_test_mock("two", baseurl=""), 81 | }, 82 | } 83 | controls = create_autospec(ControlDescriptor) 84 | controls.get_accreditations.return_value = ["infra"] 85 | notifier = _BaseNotifier(results, controls, push_error=False) 86 | 87 | (_, _, _, errored_tests) = notifier._split_by_status(notifier.messages) 88 | 89 | for _, test_desc, msg in errored_tests: 90 | self._test_url(test_desc, msg, notifier, expected=False) 91 | -------------------------------------------------------------------------------- /test/t_compliance/t_notify/test_fd_notifier.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance automation file descriptor notifier tests module.""" 15 | 16 | import unittest 17 | from io import StringIO 18 | from unittest.mock import create_autospec 19 | 20 | from compliance.controls import ControlDescriptor 21 | from compliance.notify import FDNotifier 22 | 23 | from .. import build_test_mock 24 | 25 | 26 | class FDNotifierTest(unittest.TestCase): 27 | """FDNotifier test class.""" 28 | 29 | def setUp(self): 30 | """Initialize each test.""" 31 | self.fd = StringIO() 32 | 33 | def test_notify_with_no_results(self): 34 | """Check that FDNotifier notifies that there are no results.""" 35 | notifier = FDNotifier({}, {}, self.fd) 36 | notifier.notify() 37 | self.assertEqual(self.fd.getvalue(), "\n-- NOTIFICATIONS --\n\nNo results\n") 38 | 39 | def test_notify_normal_run(self): 40 | """Check that FDNotifier notifies a test with Error.""" 41 | results = { 42 | "compliance.test.one": {"status": "error", "test": build_test_mock()}, 43 | "compliance.test.two": { 44 | "status": "warn", 45 | "test": build_test_mock("two", warns=1), 46 | }, 47 | "compliance.test.three": { 48 | "status": "fail", 49 | "test": build_test_mock("three", fails=1), 50 | }, 51 | "compliance.test.four": {"status": "pass", "test": build_test_mock("four")}, 52 | } 53 | controls = create_autospec(ControlDescriptor) 54 | controls.get_accreditations.return_value = ["infra-internal"] 55 | notifier = FDNotifier(results, controls, self.fd) 56 | notifier.notify() 57 | self.assertIn( 58 | ( 59 | "\n-- NOTIFICATIONS --\n\n" 60 | "Notifications for INFRA-INTERNAL accreditation" 61 | ), 62 | self.fd.getvalue(), 63 | ) 64 | self.assertIn( 65 | ( 66 | "mock check title one - ERROR () Reports: (none) " 67 | "| \n" 68 | "Check compliance.test.one failed to execute" 69 | ), 70 | self.fd.getvalue(), 71 | ) 72 | self.assertIn( 73 | ( 74 | "mock check title two - WARN (1 warnings) Reports: (none) " 75 | "| " 76 | ), 77 | self.fd.getvalue(), 78 | ) 79 | self.assertIn( 80 | ( 81 | "mock check title three - FAIL (1 failures) Reports: (none) " 82 | "| " 83 | ), 84 | self.fd.getvalue(), 85 | ) 86 | self.assertIn("PASSED checks: mock check title four", self.fd.getvalue()) 87 | 88 | def test_notify_push_error(self): 89 | """Check that FDNotifier notifies a test with Error.""" 90 | results = { 91 | "compliance.test.one": {"status": "error", "test": build_test_mock()}, 92 | "compliance.test.two": { 93 | "status": "warn", 94 | "test": build_test_mock("two", warns=1), 95 | }, 96 | "compliance.test.three": { 97 | "status": "fail", 98 | "test": build_test_mock("three", fails=1), 99 | }, 100 | "compliance.test.four": {"status": "pass", "test": build_test_mock("four")}, 101 | } 102 | controls = create_autospec(ControlDescriptor) 103 | controls.get_accreditations.return_value = ["infra-internal"] 104 | notifier = FDNotifier(results, controls, self.fd, push_error=True) 105 | notifier.notify() 106 | self.assertEqual( 107 | ( 108 | "\n-- NOTIFICATIONS --\n\n" 109 | "All accreditation checks: " 110 | "Evidence/Results failed to push to remote locker.\n" 111 | ), 112 | self.fd.getvalue(), 113 | ) 114 | -------------------------------------------------------------------------------- /test/t_compliance/t_notify/test_locker_notifier.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance automation Locker notifier tests module.""" 15 | 16 | import time 17 | import unittest 18 | from datetime import datetime 19 | from unittest.mock import MagicMock, create_autospec, patch 20 | 21 | from compliance.controls import ControlDescriptor 22 | from compliance.locker import Locker 23 | from compliance.notify import LockerNotifier 24 | 25 | from .. import build_compliance_check_obj 26 | 27 | 28 | class LockerNotifierTest(unittest.TestCase): 29 | """LockerNotifier test class.""" 30 | 31 | def setUp(self): 32 | """Initialize each test.""" 33 | self.results = {} 34 | for status in ["pass", "fail", "error", "warn"]: 35 | name, result = self._generate_result(status) 36 | self.results[name] = result 37 | 38 | @patch("compliance.notify.datetime") 39 | def test_notify_sends_content_to_locker(self, datetime_mock): 40 | """Test locker notifier sends content to locker as expected.""" 41 | controls_mock = create_autospec(ControlDescriptor) 42 | controls_mock.get_accreditations.return_value = ["foo"] 43 | locker_mock = create_autospec(Locker) 44 | datetime_mock.utcnow.return_value = datetime(2019, 10, 14) 45 | 46 | notifier = LockerNotifier(self.results, controls_mock, locker_mock) 47 | notifier.notify() 48 | 49 | self.assertEqual(locker_mock.add_content_to_locker.call_count, 1) 50 | args, kwargs = locker_mock.add_content_to_locker.call_args 51 | self.assertEqual(len(args), 3) 52 | self.assertEqual(kwargs, {}) 53 | self.assertTrue(args[0].startswith("# CHECK RESULTS: 2019-10-14 00:00:00")) 54 | self.assertTrue("## Notification for FOO accreditation" in args[0]) 55 | self.assertTrue("### Passed Checks" in args[0]) 56 | self.assertTrue("### Errored Checks" in args[0]) 57 | self.assertTrue("### Failures/Warnings" in args[0]) 58 | self.assertEqual(args[1], "notifications") 59 | self.assertEqual(args[2], "alerts_summary.md") 60 | 61 | def _build_check_mock(self, name): 62 | check_mock = MagicMock() 63 | check_mock.test = build_compliance_check_obj(name, name, name, [name]) 64 | return check_mock 65 | 66 | def _generate_result(self, status): 67 | return ( 68 | f"compliance.test.{status}_example", 69 | { 70 | "status": status, 71 | "timestamp": time.time(), 72 | "test": self._build_check_mock(f"{status}_example"), 73 | }, 74 | ) 75 | -------------------------------------------------------------------------------- /test/t_compliance/t_notify/test_md_notify.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance automation base markdown notifier tests module.""" 15 | 16 | import unittest 17 | from unittest.mock import create_autospec, patch 18 | 19 | from compliance.config import ComplianceConfig 20 | from compliance.controls import ControlDescriptor 21 | from compliance.notify import _BaseMDNotifier 22 | 23 | from .. import build_test_mock 24 | 25 | 26 | class BaseNotifierTest(unittest.TestCase): 27 | """Base notifier test class.""" 28 | 29 | @patch("compliance.notify.get_config") 30 | def test_notify_with_runbooks(self, get_config_mock): 31 | """Test that _BaseMDNotifier notifications have runbook links.""" 32 | config_mock = create_autospec(ComplianceConfig) 33 | config_mock.get.return_value = {"infra": ["#channel"]} 34 | 35 | get_config_mock.return_value = config_mock 36 | 37 | results = { 38 | "compliance.test.runbook": { 39 | "status": "warn", 40 | "test": build_test_mock(fails=1), 41 | }, 42 | "compliance.test.other_runbook": { 43 | "status": "fail", 44 | "test": build_test_mock("two", warns=1), 45 | }, 46 | } 47 | 48 | controls = create_autospec(ControlDescriptor) 49 | controls.get_accreditations.return_value = ["infra"] 50 | notifier = _BaseMDNotifier(results, controls, push_error=False) 51 | 52 | split_tests = notifier._split_by_status(notifier.messages) 53 | 54 | results_by_status = { 55 | "pass": split_tests[0], 56 | "fail": split_tests[1], 57 | "warn": split_tests[2], 58 | "error": split_tests[3], 59 | } 60 | 61 | markdown = "\n".join( 62 | notifier._generate_accred_content("unittests", results_by_status) 63 | ) 64 | 65 | self.assertIn( 66 | " | [Run Book](http://mockedrunbooks/path/to/runbook_one)", markdown 67 | ) 68 | self.assertIn( 69 | " | [Run Book](http://mockedrunbooks/path/to/runbook_two)", markdown 70 | ) 71 | -------------------------------------------------------------------------------- /test/t_compliance/t_notify/test_pd_notify.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance automation PagerDuty notifier tests module.""" 15 | 16 | import unittest 17 | from unittest.mock import create_autospec, patch 18 | 19 | from compliance.config import ComplianceConfig 20 | from compliance.controls import ControlDescriptor 21 | from compliance.notify import PagerDutyNotifier 22 | 23 | from .. import build_test_mock 24 | 25 | 26 | class PagerDutyNotifierTest(unittest.TestCase): 27 | """PagerDutyNotifier test class.""" 28 | 29 | @patch("compliance.notify.pagerduty.get") 30 | @patch("compliance.notify.pagerduty.send_event") 31 | @patch("compliance.notify.get_config") 32 | def test_notify_with_runbooks(self, get_config_mock, pd_send_mock, pd_get_mock): 33 | """Test that PagerDutyNotifier notifications have runbook links.""" 34 | config_mock = create_autospec(ComplianceConfig) 35 | config_mock.get.return_value = {"infra": ["#channel"]} 36 | 37 | get_config_mock.return_value = config_mock 38 | 39 | results = { 40 | "compliance.test.runbook": {"status": "error", "test": build_test_mock()}, 41 | "compliance.test.other_runbook": { 42 | "status": "error", 43 | "test": build_test_mock("two"), 44 | }, 45 | } 46 | controls = create_autospec(ControlDescriptor) 47 | controls.get_accreditations.return_value = ["infra"] 48 | notifier = PagerDutyNotifier(results, controls) 49 | notifier.notify() 50 | 51 | self.assertTrue(pd_get_mock.called) 52 | self.assertTrue(pd_send_mock.called) 53 | 54 | args, kwargs = pd_send_mock.call_args 55 | 56 | for link in kwargs["links"]: 57 | if link["text"] == "Runbook": 58 | self.assertIn("http://mockedrunbooks/path/to", link["href"]) 59 | 60 | @patch("compliance.notify.pagerduty.get") 61 | @patch("compliance.notify.pagerduty.send_event") 62 | @patch("compliance.notify.get_config") 63 | def test_notify_without_runbooks(self, get_config_mock, pd_send_mock, pd_get_mock): 64 | """Test that PagerDutyNotifier notifications have no runbook links.""" 65 | config_mock = create_autospec(ComplianceConfig) 66 | config_mock.get.return_value = {"infra": ["#channel"]} 67 | 68 | get_config_mock.return_value = config_mock 69 | 70 | controls = create_autospec(ControlDescriptor) 71 | controls.get_accreditations.return_value = ["infra"] 72 | 73 | results = { 74 | "compliance.test.runbook": { 75 | "status": "error", 76 | "test": build_test_mock(baseurl=""), 77 | }, 78 | "compliance.test.other_runbook": { 79 | "status": "error", 80 | "test": build_test_mock("two", baseurl=""), 81 | }, 82 | } 83 | 84 | notifier = PagerDutyNotifier(results, controls) 85 | notifier.notify() 86 | 87 | self.assertTrue(pd_get_mock.called) 88 | self.assertTrue(pd_send_mock.called) 89 | 90 | args, kwargs = pd_send_mock.call_args 91 | 92 | self.assertEqual(0, len(kwargs["links"])) 93 | -------------------------------------------------------------------------------- /test/t_compliance/t_notify/test_slack_notifier.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance automation Slack notifier tests module.""" 15 | 16 | import json 17 | import unittest 18 | 19 | try: 20 | from mock import patch, create_autospec 21 | except ImportError: 22 | from unittest.mock import patch, create_autospec 23 | 24 | from compliance.config import ComplianceConfig 25 | from compliance.controls import ControlDescriptor 26 | from compliance.notify import SlackNotifier 27 | 28 | from .. import build_test_mock 29 | 30 | 31 | class SlackNotifierTest(unittest.TestCase): 32 | """SlackNotifier test class.""" 33 | 34 | @patch("requests.post") 35 | @patch("compliance.notify.get_config") 36 | def test_notify_error(self, get_config_mock, post_mock): 37 | """Test that SlackNotifier notifies a test with Error.""" 38 | config_mock = create_autospec(ComplianceConfig) 39 | config_mock.get.return_value = {"infra": ["#channel"]} 40 | get_config_mock.return_value = config_mock 41 | 42 | results = { 43 | "compliance.test.example": {"status": "error", "test": build_test_mock()} 44 | } 45 | controls = create_autospec(ControlDescriptor) 46 | controls.get_accreditations.return_value = ["infra"] 47 | notifier = SlackNotifier(results, controls) 48 | notifier.notify() 49 | self.assertTrue(post_mock.called) 50 | 51 | args, kwargs = post_mock.call_args 52 | self.assertTrue(args[0].startswith("https://hooks.slack.com")) 53 | msg = json.loads(kwargs["data"]) 54 | self.assertIn("failed to execute", msg["attachments"][0]["text"]) 55 | 56 | @patch("requests.post") 57 | @patch("compliance.notify.get_config") 58 | def test_notify_two_errors(self, get_config_mock, post_mock): 59 | """Test that SlackNotifier notifies two tests with Error.""" 60 | config_mock = create_autospec(ComplianceConfig) 61 | config_mock.get.return_value = {"infra": ["#channel"]} 62 | get_config_mock.return_value = config_mock 63 | 64 | results = { 65 | "compliance.test.example": {"status": "error", "test": build_test_mock()}, 66 | "compliance.test.other_example": { 67 | "status": "error", 68 | "test": build_test_mock("two"), 69 | }, 70 | } 71 | controls = create_autospec(ControlDescriptor) 72 | controls.get_accreditations.return_value = ["infra"] 73 | notifier = SlackNotifier(results, controls) 74 | notifier.notify() 75 | 76 | self.assertTrue(post_mock.called) 77 | args, kwargs = post_mock.call_args 78 | self.assertTrue(args[0].startswith("https://hooks.slack.com")) 79 | msg = json.loads(kwargs["data"]) 80 | self.assertEqual(len(msg["attachments"]), 3) 81 | self.assertEqual("PASSED checks", msg["attachments"][2]["title"]) 82 | for att in msg["attachments"][:-1]: 83 | self.assertIn("failed to execute", att["text"]) 84 | 85 | @patch("requests.post") 86 | @patch("compliance.notify.get_config") 87 | def test_do_not_notify_if_unknown_acc(self, get_config_mock, post_mock): 88 | """Test that SlackNotifier does not notify if accred is unknown.""" 89 | results = { 90 | "compliance.test.example": {"status": "error", "test": build_test_mock()} 91 | } 92 | controls = create_autospec(ControlDescriptor) 93 | controls.get_accreditations.return_value = ["SOMETHING-WRONG"] 94 | notifier = SlackNotifier(results, controls) 95 | notifier.notify() 96 | post_mock.assert_not_called() 97 | 98 | @patch("requests.post") 99 | @patch("compliance.notify.get_config") 100 | def test_notify_with_runbooks(self, get_config_mock, post_mock): 101 | """Test that SlackNotifier notifications have runbook links.""" 102 | config_mock = create_autospec(ComplianceConfig) 103 | config_mock.get.return_value = {"infra": ["#channel"]} 104 | 105 | get_config_mock.return_value = config_mock 106 | 107 | results = { 108 | "compliance.test.runbook": {"status": "error", "test": build_test_mock()}, 109 | "compliance.test.other_runbook": { 110 | "status": "error", 111 | "test": build_test_mock("two"), 112 | }, 113 | } 114 | controls = create_autospec(ControlDescriptor) 115 | controls.get_accreditations.return_value = ["infra"] 116 | notifier = SlackNotifier(results, controls) 117 | notifier.notify() 118 | 119 | self.assertTrue(post_mock.called) 120 | args, kwargs = post_mock.call_args 121 | self.assertTrue(args[0].startswith("https://hooks.slack.com")) 122 | msg = json.loads(kwargs["data"]) 123 | self.assertEqual(len(msg["attachments"]), 3) 124 | self.assertEqual("PASSED checks", msg["attachments"][2]["title"]) 125 | for att in msg["attachments"][:-1]: 126 | testname = att["title"].rsplit(" ", 1)[1] 127 | url = f"http://mockedrunbooks/path/to/runbook_{testname}" 128 | notifier.logger.warning(testname) 129 | notifier.logger.warning(url) 130 | self.assertIn(f" | <{url}|Run Book>", att["text"]) 131 | -------------------------------------------------------------------------------- /test/t_compliance/t_report/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance automation report builder tests package.""" 15 | -------------------------------------------------------------------------------- /test/t_compliance/t_report/test_report.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance automation report builder tests module.""" 15 | 16 | import unittest 17 | from unittest.mock import MagicMock, create_autospec, patch 18 | 19 | from compliance.config import ComplianceConfig 20 | from compliance.controls import ControlDescriptor 21 | from compliance.evidence import ReportEvidence 22 | from compliance.locker import Locker 23 | from compliance.report import ReportBuilder 24 | 25 | from .. import build_test_mock 26 | 27 | 28 | class ReportTest(unittest.TestCase): 29 | """ReportBuilder test class.""" 30 | 31 | @patch("compliance.report.ReportBuilder.generate_toc") 32 | @patch("compliance.report.ReportBuilder.generate_check_results") 33 | @patch("compliance.report.get_config") 34 | @patch("compliance.report.get_evidence_by_path") 35 | def test_report_fail_to_generate( 36 | self, evidence_path_mock, get_cfg_mock, gen_chk_mock, gen_toc_mock 37 | ): 38 | """Test report generation failure affects on general execution.""" 39 | config_mock = create_autospec(ComplianceConfig) 40 | config_mock.get_template_dir.return_value = "/rpt/templates" 41 | get_cfg_mock.return_value = config_mock 42 | 43 | report = ReportEvidence("test", "test", 12345) 44 | report.set_content(None) 45 | evidence_path_mock.return_value = report 46 | 47 | locker = create_autospec(Locker()) 48 | locker.local_path = "/my/fake/locker" 49 | locker.get_reports_metadata = MagicMock(return_value={"foo": "bar"}) 50 | 51 | test_obj = build_test_mock() 52 | test_obj.test.get_reports.return_value = ["test/example.md"] 53 | 54 | results = {"mock.test.test_one": {"status": "pass", "test": test_obj}} 55 | controls = create_autospec(ControlDescriptor) 56 | builder = ReportBuilder(locker, results, controls) 57 | builder.build() 58 | self.assertEqual(results["mock.test.test_one"]["status"], "error") 59 | gen_chk_mock.assert_called_once_with({"foo": "bar"}) 60 | gen_toc_mock.assert_called_once_with({"foo": "bar"}) 61 | -------------------------------------------------------------------------------- /test/t_compliance/t_utils/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance automation utilities tests package.""" 15 | -------------------------------------------------------------------------------- /test/t_compliance/t_utils/test_data_parse.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance automation utilities tests module.""" 15 | 16 | import hashlib 17 | import unittest 18 | 19 | from compliance.utils.data_parse import get_sha256_hash 20 | 21 | 22 | class TestUtilityFunctions(unittest.TestCase): 23 | """Test data_parse utility functions.""" 24 | 25 | def test_sha256_hash_no_size(self): 26 | """Test when no size is provided, the full hash is returned.""" 27 | self.assertEqual(get_sha256_hash(["foo"]), hashlib.sha256(b"foo").hexdigest()) 28 | 29 | def test_sha256_hash_oversized(self): 30 | """Test when size is too big, the full hash is returned.""" 31 | expected = hashlib.sha256(b"foo").hexdigest() 32 | self.assertEqual(len(expected), 64) 33 | self.assertEqual(get_sha256_hash(["foo"], 1000), expected) 34 | 35 | def test_sha256_hash_sized(self): 36 | """Test the first 'size' number of characters are returned.""" 37 | actual = get_sha256_hash(["foo"], 10) 38 | self.assertEqual(len(actual), 10) 39 | self.assertTrue(hashlib.sha256(b"foo").hexdigest().startswith(actual)) 40 | 41 | def test_sha256_hash_multiple_keys(self): 42 | """Test hash is correct when a list of keys is provided.""" 43 | self.assertEqual( 44 | get_sha256_hash(["foo", "bar", "baz", 1234]), 45 | hashlib.sha256(b"foobarbaz1234").hexdigest(), 46 | ) 47 | -------------------------------------------------------------------------------- /test/t_compliance/t_workflow/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 IBM Corp. All rights reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Compliance automation workflow tests package.""" 15 | -------------------------------------------------------------------------------- /test/t_compliance/t_workflow/demo_case.xml: -------------------------------------------------------------------------------- 1 | 16 | 17 | 18 | 19 | 20 | 75714 21 | 22 | 23 | 24 | 25 | 26 | true 27 | 28 | 29 | 30 | 31 | 692872 32 | 33 | 34 | 1 35 | 36 | 37 | 38 | 39 | 40 | 363 41 | 42 | 43 | 0 44 | 45 |
46 | 2016-10-05T13:43:49Z 47 |
48 | 49 | 50 | 51 | false 52 | 53 | 54 | false 55 | 56 | 57 | false 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | false 70 | 71 | 72 | false 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | --------------------------------------------------------------------------------