├── .trivyignore ├── _config.yml ├── docs ├── user-interface-engineering │ ├── stability.md │ ├── usability.md │ └── maintainability.md ├── resources │ ├── ms_icon.png │ └── templates │ │ ├── README.md │ │ ├── CONTRIBUTING.md │ │ └── LICENSE ├── observability │ ├── images │ │ ├── flame.png │ │ ├── signals.png │ │ ├── collectors.png │ │ ├── pprof-dot.png │ │ ├── newarchitecture.png │ │ ├── oldarchitecture.png │ │ ├── automaticamanual.png │ │ └── contextpropagation.png │ ├── microservices.png │ ├── README.md │ ├── pillars │ │ └── tracing.md │ ├── observability-pipelines.md │ ├── pitfalls.md │ ├── best-practices.md │ ├── tools │ │ ├── README.md │ │ ├── loki.md │ │ └── KubernetesDashboards.md │ ├── profiling.md │ ├── alerting.md │ ├── observability-as-code.md │ └── log-vs-metric-vs-trace.md ├── security │ ├── images │ │ ├── data_flow.png │ │ ├── arch_diagram.png │ │ ├── threat_list.png │ │ └── threat_model.png │ ├── rules-of-engagement.md │ └── threat-modelling.md ├── machine-learning │ ├── images │ │ ├── flow.png │ │ └── ml-tpm-skills.png │ ├── README.md │ ├── responsible-ai.md │ └── ml-fundamentals-checklist.md ├── developer-experience │ ├── images │ │ ├── VSCode.png │ │ └── github-copilot.png │ ├── code-completion.md │ └── recipes │ │ ├── onboarding-guide-template.md │ │ └── cross-platform-tasks.md ├── continuous-delivery │ ├── images │ │ ├── blue_green.png │ │ ├── environments.png │ │ ├── canary_release.png │ │ └── example_release_flow.png │ ├── deploying-with-gitops │ │ ├── images │ │ │ └── GitopsWorflowVsTraditionalPush.jpg │ │ └── README.md │ ├── recipes │ │ └── github-workflows │ │ │ ├── images │ │ │ ├── Workflow-Designs-Dependent-Workflows.png │ │ │ └── Workflow-Designs-Independent-Workflows.png │ │ │ └── workflow-per-environment.md │ └── devops-provider-recipes │ │ └── github-actions │ │ └── runtime-variables │ │ ├── images │ │ ├── PRTrue.PNG │ │ ├── CommitTrue.PNG │ │ ├── PRExample.PNG │ │ ├── AzDoExample.PNG │ │ ├── CommitFalse.PNG │ │ ├── DevAWorkflow.png │ │ ├── DevBWorkflow.png │ │ └── workflow-diagram.png │ │ └── examples │ │ ├── pr-example.yaml │ │ └── commit-example.yaml ├── documentation │ ├── tools │ │ ├── images │ │ │ ├── docs-checks.png │ │ │ ├── example-mermaid-diagram.png │ │ │ └── example-mermaid-diagram.mmd │ │ ├── integrations.md │ │ ├── wikis.md │ │ ├── automation.md │ │ └── languages.md │ ├── recipes │ │ ├── images │ │ │ ├── docfx-toc-yaml.png │ │ │ ├── docfx-order-file.png │ │ │ └── docfx-sample-website.png │ │ ├── sync-wiki-between-repos.md │ │ └── static-website-with-mkdocs.md │ ├── guidance │ │ ├── images │ │ │ ├── agile-artifacts.png │ │ │ └── scrum-artifacts.png │ │ ├── engineering-feedback.md │ │ ├── pull-requests.md │ │ └── rest-apis.md │ └── best-practices │ │ ├── good-documentation.md │ │ ├── establish-and-manage.md │ │ └── automation.md ├── code-reviews │ ├── recipes │ │ ├── images │ │ │ ├── vscode-roslyn.png │ │ │ ├── key-concepts-overview.png │ │ │ └── authorization_issue_due_to_initialize_time.png │ │ ├── README.md │ │ └── java.md │ ├── README.md │ ├── process-guidance │ │ ├── author-guidance.md │ │ └── README.md │ ├── pull-request-template │ │ └── pull-request-template.md │ └── evidence-and-measures │ │ └── README.md ├── automated-testing │ ├── cdc-testing │ │ └── images │ │ │ ├── images.pptx │ │ │ ├── cdc-testing.png │ │ │ ├── cdc-vs-e2e.png │ │ │ └── testing-pyramid.png │ ├── e2e-testing │ │ ├── images │ │ │ ├── gauge.jpg │ │ │ ├── images.pptx │ │ │ ├── robot.jpg │ │ │ ├── e2e-blocks.png │ │ │ ├── e2e-testing.png │ │ │ ├── TestCraft-logo.png │ │ │ ├── ranorex-studio2.png │ │ │ ├── testing-pyramid.png │ │ │ ├── bugbug-logo-208x65.png │ │ │ ├── postman-e2e-script.png │ │ │ ├── horizontal-e2e-testing.png │ │ │ ├── vertical-e2e-testing.png │ │ │ └── New-Logo-Katalon-Studio.png │ │ ├── testing-methods.md │ │ └── testing-comparison.md │ ├── integration-testing │ │ └── images │ │ │ ├── bigBang.jpg │ │ │ ├── bottomUp.jpg │ │ │ └── topDown.png │ ├── shadow-testing │ │ └── images │ │ │ ├── shadow-testing.png │ │ │ └── diffy-shadow-testing.png │ ├── synthetic-monitoring-tests │ │ └── images │ │ │ └── syntheticMonitoring.png │ ├── tech-specific-samples │ │ └── blobstorage-unit-tests │ │ │ └── images │ │ │ ├── azure_pipeline.png │ │ │ ├── blob_storage_connection.png │ │ │ ├── http_local_blob_storage.png │ │ │ └── blob_storage_connection_attach.png │ ├── unit-testing │ │ ├── custom-connector.md │ │ └── why-unit-tests.md │ ├── templates │ │ ├── case-study-template.md │ │ └── test_type_template.md │ ├── performance-testing │ │ └── iterative-perf-test-template.md │ └── smoke-testing │ │ └── README.md ├── design │ ├── design-reviews │ │ ├── recipes │ │ │ ├── images │ │ │ │ ├── high-level-box-diagram.png │ │ │ │ ├── high-level-sequence-diagram.png │ │ │ │ ├── high-level-system-diagram.png │ │ │ │ ├── async-design-reviews-sequence.png │ │ │ │ └── engineering-feasibility-spike-feedback-loop.png │ │ │ ├── sprint-spike-template.md │ │ │ ├── README.md │ │ │ ├── preferred-diagram-tooling.md │ │ │ ├── milestone-epic-design-review-recipe.md │ │ │ ├── high-level-design-recipe.md │ │ │ ├── task-design-review-template.md │ │ │ └── technical-spike.md │ │ └── decision-log │ │ │ ├── examples │ │ │ └── memory │ │ │ │ └── README.md │ │ │ └── doc │ │ │ ├── adr │ │ │ ├── 0001-record-architecture-decisions.md │ │ │ └── 0002-app-level-logging.md │ │ │ └── decision-log.md │ ├── design-patterns │ │ ├── data-heavy-design-guidance │ │ │ └── images │ │ │ │ └── CI_CD_process.png │ │ ├── distributed-system-design-reference │ │ │ └── README.md │ │ ├── object-oriented-design-reference │ │ │ └── README.md │ │ ├── README.md │ │ └── cloud-resource-design-guidance │ │ │ └── README.md │ └── README.md ├── agile-development │ ├── advanced-topics │ │ ├── backlog-management │ │ │ ├── README.md │ │ │ ├── external-feedback.md │ │ │ └── minimal-slices.md │ │ ├── collaboration │ │ │ ├── images │ │ │ │ ├── PTTdiagram.PNG │ │ │ │ ├── azure-devops-boards.png │ │ │ │ ├── azure-devops-new-field.png │ │ │ │ ├── azure-devops-pairing-field.png │ │ │ │ ├── azure-devops-board-settings.png │ │ │ │ ├── azure-devops-change-process.png │ │ │ │ ├── azure-devops-pairing-process.png │ │ │ │ ├── azure-devops-project-settings.png │ │ │ │ ├── azure-devops-add-field-to-card.png │ │ │ │ ├── azure-devops-user-story-process.png │ │ │ │ ├── azure-devops-organization-settings.png │ │ │ │ ├── azure-devops-process-customization.png │ │ │ │ ├── azure-devops-project-configuration.png │ │ │ │ ├── azure-devops-add-field-to-user-story.png │ │ │ │ ├── azure-devops-change-project-process.png │ │ │ │ └── azure-devops-create-inherited-process.png │ │ │ └── README.md │ │ ├── effective-organization │ │ │ ├── README.md │ │ │ └── delivery-plan.md │ │ ├── README.md │ │ └── team-agreements │ │ │ ├── README.md │ │ │ └── definition-of-done.md │ ├── README.md │ └── basics │ │ ├── Roles │ │ └── README.md │ │ ├── README.md │ │ ├── Ceremonies │ │ └── README.md │ │ ├── Sprints │ │ └── README.md │ │ └── Backlog Management │ │ └── README.md ├── continuous-integration │ ├── ci-in-datascience │ │ └── working-with-notebooks │ │ │ └── assets │ │ │ └── repository-properties.png │ ├── dev-sec-ops │ │ ├── static-code-analysis │ │ │ └── static_code_analysis.md │ │ ├── README.md │ │ ├── penetration-testing │ │ │ └── penetration_testing.md │ │ └── secret-management │ │ │ ├── secrets_rotation.md │ │ │ └── recipes │ │ │ └── detect-secrets.md │ └── markdown-linting │ │ └── README.md ├── privacy │ └── README.md ├── source-control │ ├── secrets-management.md │ ├── naming-branches.md │ ├── merge-strategies.md │ └── component-versioning.md └── CSE.md ├── favicon.ico ├── package.json ├── github_conf └── branch_protection_rules.json ├── .markdownlintrc ├── requirements-docs.txt ├── .jscpd.json ├── .pre-commit-config.yaml ├── mkdocs-overrides └── main.html ├── .github ├── ISSUE_TEMPLATE │ ├── bug_report.md │ └── feature_request.md ├── workflows │ ├── auto_request_review.yml │ ├── stale_issues_flow │ └── mkdocs.yml ├── reviewers.yml └── pull_request_template.md ├── mkdocs.yml ├── .mega-linter.yml ├── LICENSE-CODE ├── .devcontainer ├── Dockerfile └── devcontainer.json ├── .markdown-link-check.json └── SECURITY.md /.trivyignore: -------------------------------------------------------------------------------- 1 | DS002 -------------------------------------------------------------------------------- /_config.yml: -------------------------------------------------------------------------------- 1 | theme: jekyll-theme-slate 2 | include: ["CONTRIBUTING.md", "LICENSE"] 3 | -------------------------------------------------------------------------------- /docs/user-interface-engineering/stability.md: -------------------------------------------------------------------------------- 1 | # Stability 2 | 3 | > Coming soon! 4 | -------------------------------------------------------------------------------- /docs/user-interface-engineering/usability.md: -------------------------------------------------------------------------------- 1 | # Usability 2 | 3 | > Coming soon! 4 | -------------------------------------------------------------------------------- /docs/user-interface-engineering/maintainability.md: -------------------------------------------------------------------------------- 1 | # Maintainability 2 | 3 | > Coming soon! 4 | -------------------------------------------------------------------------------- /favicon.ico: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/favicon.ico -------------------------------------------------------------------------------- /package.json: -------------------------------------------------------------------------------- 1 | { 2 | "scripts": { 3 | }, 4 | "dependencies": { 5 | "run-script-os": "1.1.5" 6 | } 7 | } -------------------------------------------------------------------------------- /docs/resources/ms_icon.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/resources/ms_icon.png -------------------------------------------------------------------------------- /github_conf/branch_protection_rules.json: -------------------------------------------------------------------------------- 1 | { 2 | "message": "Not Found", 3 | "documentation_url": "https://docs.github.com/rest" 4 | } -------------------------------------------------------------------------------- /.markdownlintrc: -------------------------------------------------------------------------------- 1 | { 2 | "no-trailing-punctuation": false, 3 | "line-length": false, 4 | "MD024": { 5 | "siblings_only": true 6 | } 7 | } -------------------------------------------------------------------------------- /docs/observability/images/flame.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/observability/images/flame.png -------------------------------------------------------------------------------- /docs/security/images/data_flow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/security/images/data_flow.png -------------------------------------------------------------------------------- /docs/machine-learning/images/flow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/machine-learning/images/flow.png -------------------------------------------------------------------------------- /docs/observability/images/signals.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/observability/images/signals.png -------------------------------------------------------------------------------- /docs/observability/microservices.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/observability/microservices.png -------------------------------------------------------------------------------- /docs/security/images/arch_diagram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/security/images/arch_diagram.png -------------------------------------------------------------------------------- /docs/security/images/threat_list.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/security/images/threat_list.png -------------------------------------------------------------------------------- /docs/security/images/threat_model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/security/images/threat_model.png -------------------------------------------------------------------------------- /docs/observability/images/collectors.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/observability/images/collectors.png -------------------------------------------------------------------------------- /docs/observability/images/pprof-dot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/observability/images/pprof-dot.png -------------------------------------------------------------------------------- /docs/developer-experience/images/VSCode.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/developer-experience/images/VSCode.png -------------------------------------------------------------------------------- /docs/observability/images/newarchitecture.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/observability/images/newarchitecture.png -------------------------------------------------------------------------------- /docs/observability/images/oldarchitecture.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/observability/images/oldarchitecture.png -------------------------------------------------------------------------------- /docs/continuous-delivery/images/blue_green.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/continuous-delivery/images/blue_green.png -------------------------------------------------------------------------------- /docs/continuous-delivery/images/environments.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/continuous-delivery/images/environments.png -------------------------------------------------------------------------------- /docs/documentation/tools/images/docs-checks.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/documentation/tools/images/docs-checks.png -------------------------------------------------------------------------------- /docs/machine-learning/images/ml-tpm-skills.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/machine-learning/images/ml-tpm-skills.png -------------------------------------------------------------------------------- /docs/observability/images/automaticamanual.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/observability/images/automaticamanual.png -------------------------------------------------------------------------------- /docs/observability/images/contextpropagation.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/observability/images/contextpropagation.png -------------------------------------------------------------------------------- /docs/code-reviews/recipes/images/vscode-roslyn.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/code-reviews/recipes/images/vscode-roslyn.png -------------------------------------------------------------------------------- /docs/continuous-delivery/images/canary_release.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/continuous-delivery/images/canary_release.png -------------------------------------------------------------------------------- /docs/automated-testing/cdc-testing/images/images.pptx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/automated-testing/cdc-testing/images/images.pptx -------------------------------------------------------------------------------- /docs/automated-testing/e2e-testing/images/gauge.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/automated-testing/e2e-testing/images/gauge.jpg -------------------------------------------------------------------------------- /docs/automated-testing/e2e-testing/images/images.pptx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/automated-testing/e2e-testing/images/images.pptx -------------------------------------------------------------------------------- /docs/automated-testing/e2e-testing/images/robot.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/automated-testing/e2e-testing/images/robot.jpg -------------------------------------------------------------------------------- /docs/developer-experience/images/github-copilot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/developer-experience/images/github-copilot.png -------------------------------------------------------------------------------- /docs/documentation/recipes/images/docfx-toc-yaml.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/documentation/recipes/images/docfx-toc-yaml.png -------------------------------------------------------------------------------- /docs/documentation/guidance/images/agile-artifacts.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/documentation/guidance/images/agile-artifacts.png -------------------------------------------------------------------------------- /docs/documentation/guidance/images/scrum-artifacts.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/documentation/guidance/images/scrum-artifacts.png -------------------------------------------------------------------------------- /docs/documentation/recipes/images/docfx-order-file.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/documentation/recipes/images/docfx-order-file.png -------------------------------------------------------------------------------- /docs/automated-testing/cdc-testing/images/cdc-testing.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/automated-testing/cdc-testing/images/cdc-testing.png -------------------------------------------------------------------------------- /docs/automated-testing/cdc-testing/images/cdc-vs-e2e.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/automated-testing/cdc-testing/images/cdc-vs-e2e.png -------------------------------------------------------------------------------- /docs/automated-testing/e2e-testing/images/e2e-blocks.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/automated-testing/e2e-testing/images/e2e-blocks.png -------------------------------------------------------------------------------- /docs/automated-testing/e2e-testing/images/e2e-testing.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/automated-testing/e2e-testing/images/e2e-testing.png -------------------------------------------------------------------------------- /docs/code-reviews/recipes/images/key-concepts-overview.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/code-reviews/recipes/images/key-concepts-overview.png -------------------------------------------------------------------------------- /docs/continuous-delivery/images/example_release_flow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/continuous-delivery/images/example_release_flow.png -------------------------------------------------------------------------------- /docs/documentation/recipes/images/docfx-sample-website.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/documentation/recipes/images/docfx-sample-website.png -------------------------------------------------------------------------------- /docs/automated-testing/e2e-testing/images/TestCraft-logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/automated-testing/e2e-testing/images/TestCraft-logo.png -------------------------------------------------------------------------------- /docs/documentation/tools/images/example-mermaid-diagram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/documentation/tools/images/example-mermaid-diagram.png -------------------------------------------------------------------------------- /docs/automated-testing/cdc-testing/images/testing-pyramid.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/automated-testing/cdc-testing/images/testing-pyramid.png -------------------------------------------------------------------------------- /docs/automated-testing/e2e-testing/images/ranorex-studio2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/automated-testing/e2e-testing/images/ranorex-studio2.png -------------------------------------------------------------------------------- /docs/automated-testing/e2e-testing/images/testing-pyramid.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/automated-testing/e2e-testing/images/testing-pyramid.png -------------------------------------------------------------------------------- /docs/automated-testing/integration-testing/images/bigBang.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/automated-testing/integration-testing/images/bigBang.jpg -------------------------------------------------------------------------------- /docs/automated-testing/integration-testing/images/bottomUp.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/automated-testing/integration-testing/images/bottomUp.jpg -------------------------------------------------------------------------------- /docs/automated-testing/integration-testing/images/topDown.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/automated-testing/integration-testing/images/topDown.png -------------------------------------------------------------------------------- /docs/automated-testing/shadow-testing/images/shadow-testing.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/automated-testing/shadow-testing/images/shadow-testing.png -------------------------------------------------------------------------------- /requirements-docs.txt: -------------------------------------------------------------------------------- 1 | mkdocs==1.3.* 2 | mkdocs-material==8.2.* 3 | pymdown-extensions==9.4.* 4 | markdown==3.3.* 5 | mdx_truly_sane_lists==1.2 6 | mkdocs-git-revision-date-localized-plugin==1.0.* 7 | -------------------------------------------------------------------------------- /docs/automated-testing/e2e-testing/images/bugbug-logo-208x65.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/automated-testing/e2e-testing/images/bugbug-logo-208x65.png -------------------------------------------------------------------------------- /docs/automated-testing/e2e-testing/images/postman-e2e-script.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/automated-testing/e2e-testing/images/postman-e2e-script.png -------------------------------------------------------------------------------- /docs/automated-testing/e2e-testing/images/horizontal-e2e-testing.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/automated-testing/e2e-testing/images/horizontal-e2e-testing.png -------------------------------------------------------------------------------- /docs/automated-testing/e2e-testing/images/vertical-e2e-testing.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/automated-testing/e2e-testing/images/vertical-e2e-testing.png -------------------------------------------------------------------------------- /docs/design/design-reviews/recipes/images/high-level-box-diagram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/design/design-reviews/recipes/images/high-level-box-diagram.png -------------------------------------------------------------------------------- /docs/automated-testing/e2e-testing/images/New-Logo-Katalon-Studio.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/automated-testing/e2e-testing/images/New-Logo-Katalon-Studio.png -------------------------------------------------------------------------------- /docs/automated-testing/shadow-testing/images/diffy-shadow-testing.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/automated-testing/shadow-testing/images/diffy-shadow-testing.png -------------------------------------------------------------------------------- /docs/agile-development/advanced-topics/backlog-management/README.md: -------------------------------------------------------------------------------- 1 | # Advanced recommendations for Backlog Management 2 | 3 | - [External Feedback](./external-feedback.md) 4 | - [Minimal slices](./minimal-slices.md) 5 | -------------------------------------------------------------------------------- /docs/design/design-reviews/recipes/images/high-level-sequence-diagram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/design/design-reviews/recipes/images/high-level-sequence-diagram.png -------------------------------------------------------------------------------- /docs/design/design-reviews/recipes/images/high-level-system-diagram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/design/design-reviews/recipes/images/high-level-system-diagram.png -------------------------------------------------------------------------------- /docs/agile-development/advanced-topics/collaboration/images/PTTdiagram.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/agile-development/advanced-topics/collaboration/images/PTTdiagram.PNG -------------------------------------------------------------------------------- /docs/design/design-reviews/recipes/images/async-design-reviews-sequence.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/design/design-reviews/recipes/images/async-design-reviews-sequence.png -------------------------------------------------------------------------------- /docs/agile-development/advanced-topics/effective-organization/README.md: -------------------------------------------------------------------------------- 1 | # Advanced recommendations for a more effective organization 2 | 3 | - [Delivery/Release plan](./delivery-plan.md) 4 | - [Scrum of Scrum](./scrum-of-scrums.md) 5 | -------------------------------------------------------------------------------- /docs/automated-testing/synthetic-monitoring-tests/images/syntheticMonitoring.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/automated-testing/synthetic-monitoring-tests/images/syntheticMonitoring.png -------------------------------------------------------------------------------- /docs/code-reviews/recipes/images/authorization_issue_due_to_initialize_time.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/code-reviews/recipes/images/authorization_issue_due_to_initialize_time.png -------------------------------------------------------------------------------- /docs/design/design-patterns/data-heavy-design-guidance/images/CI_CD_process.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/design/design-patterns/data-heavy-design-guidance/images/CI_CD_process.png -------------------------------------------------------------------------------- /docs/agile-development/advanced-topics/collaboration/images/azure-devops-boards.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/agile-development/advanced-topics/collaboration/images/azure-devops-boards.png -------------------------------------------------------------------------------- /docs/agile-development/advanced-topics/collaboration/images/azure-devops-new-field.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/agile-development/advanced-topics/collaboration/images/azure-devops-new-field.png -------------------------------------------------------------------------------- /docs/continuous-delivery/deploying-with-gitops/images/GitopsWorflowVsTraditionalPush.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/continuous-delivery/deploying-with-gitops/images/GitopsWorflowVsTraditionalPush.jpg -------------------------------------------------------------------------------- /docs/agile-development/advanced-topics/collaboration/images/azure-devops-pairing-field.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/agile-development/advanced-topics/collaboration/images/azure-devops-pairing-field.png -------------------------------------------------------------------------------- /docs/design/design-reviews/recipes/images/engineering-feasibility-spike-feedback-loop.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/design/design-reviews/recipes/images/engineering-feasibility-spike-feedback-loop.png -------------------------------------------------------------------------------- /docs/agile-development/advanced-topics/collaboration/images/azure-devops-board-settings.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/agile-development/advanced-topics/collaboration/images/azure-devops-board-settings.png -------------------------------------------------------------------------------- /docs/agile-development/advanced-topics/collaboration/images/azure-devops-change-process.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/agile-development/advanced-topics/collaboration/images/azure-devops-change-process.png -------------------------------------------------------------------------------- /docs/agile-development/advanced-topics/collaboration/images/azure-devops-pairing-process.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/agile-development/advanced-topics/collaboration/images/azure-devops-pairing-process.png -------------------------------------------------------------------------------- /docs/agile-development/advanced-topics/collaboration/images/azure-devops-project-settings.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/agile-development/advanced-topics/collaboration/images/azure-devops-project-settings.png -------------------------------------------------------------------------------- /docs/automated-testing/tech-specific-samples/blobstorage-unit-tests/images/azure_pipeline.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/automated-testing/tech-specific-samples/blobstorage-unit-tests/images/azure_pipeline.png -------------------------------------------------------------------------------- /docs/agile-development/advanced-topics/collaboration/images/azure-devops-add-field-to-card.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/agile-development/advanced-topics/collaboration/images/azure-devops-add-field-to-card.png -------------------------------------------------------------------------------- /docs/agile-development/advanced-topics/collaboration/images/azure-devops-user-story-process.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/agile-development/advanced-topics/collaboration/images/azure-devops-user-story-process.png -------------------------------------------------------------------------------- /docs/agile-development/advanced-topics/collaboration/images/azure-devops-organization-settings.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/agile-development/advanced-topics/collaboration/images/azure-devops-organization-settings.png -------------------------------------------------------------------------------- /docs/agile-development/advanced-topics/collaboration/images/azure-devops-process-customization.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/agile-development/advanced-topics/collaboration/images/azure-devops-process-customization.png -------------------------------------------------------------------------------- /docs/agile-development/advanced-topics/collaboration/images/azure-devops-project-configuration.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/agile-development/advanced-topics/collaboration/images/azure-devops-project-configuration.png -------------------------------------------------------------------------------- /docs/continuous-delivery/recipes/github-workflows/images/Workflow-Designs-Dependent-Workflows.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/continuous-delivery/recipes/github-workflows/images/Workflow-Designs-Dependent-Workflows.png -------------------------------------------------------------------------------- /docs/agile-development/advanced-topics/collaboration/images/azure-devops-add-field-to-user-story.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/agile-development/advanced-topics/collaboration/images/azure-devops-add-field-to-user-story.png -------------------------------------------------------------------------------- /docs/agile-development/advanced-topics/collaboration/images/azure-devops-change-project-process.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/agile-development/advanced-topics/collaboration/images/azure-devops-change-project-process.png -------------------------------------------------------------------------------- /docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/images/PRTrue.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/images/PRTrue.PNG -------------------------------------------------------------------------------- /docs/continuous-delivery/recipes/github-workflows/images/Workflow-Designs-Independent-Workflows.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/continuous-delivery/recipes/github-workflows/images/Workflow-Designs-Independent-Workflows.png -------------------------------------------------------------------------------- /docs/documentation/tools/images/example-mermaid-diagram.mmd: -------------------------------------------------------------------------------- 1 | graph LR 2 | A[Diagram Idea] -->|Write mermaid code| B(mermaid.mmd file) 3 | B -->|Add to source control| C{Code repo} 4 | B -->|Export as .png| G(.png file of diagram) 5 | G -->|Add to source control| C -------------------------------------------------------------------------------- /docs/agile-development/advanced-topics/collaboration/images/azure-devops-create-inherited-process.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/agile-development/advanced-topics/collaboration/images/azure-devops-create-inherited-process.png -------------------------------------------------------------------------------- /docs/automated-testing/tech-specific-samples/blobstorage-unit-tests/images/blob_storage_connection.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/automated-testing/tech-specific-samples/blobstorage-unit-tests/images/blob_storage_connection.png -------------------------------------------------------------------------------- /docs/automated-testing/tech-specific-samples/blobstorage-unit-tests/images/http_local_blob_storage.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/automated-testing/tech-specific-samples/blobstorage-unit-tests/images/http_local_blob_storage.png -------------------------------------------------------------------------------- /docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/images/CommitTrue.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/images/CommitTrue.PNG -------------------------------------------------------------------------------- /docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/images/PRExample.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/images/PRExample.PNG -------------------------------------------------------------------------------- /docs/continuous-integration/ci-in-datascience/working-with-notebooks/assets/repository-properties.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/continuous-integration/ci-in-datascience/working-with-notebooks/assets/repository-properties.png -------------------------------------------------------------------------------- /docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/images/AzDoExample.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/images/AzDoExample.PNG -------------------------------------------------------------------------------- /docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/images/CommitFalse.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/images/CommitFalse.PNG -------------------------------------------------------------------------------- /docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/images/DevAWorkflow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/images/DevAWorkflow.png -------------------------------------------------------------------------------- /docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/images/DevBWorkflow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/images/DevBWorkflow.png -------------------------------------------------------------------------------- /docs/automated-testing/tech-specific-samples/blobstorage-unit-tests/images/blob_storage_connection_attach.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/automated-testing/tech-specific-samples/blobstorage-unit-tests/images/blob_storage_connection_attach.png -------------------------------------------------------------------------------- /docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/images/workflow-diagram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ilhan-mstf/code-with-engineering-playbook/main/docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/images/workflow-diagram.png -------------------------------------------------------------------------------- /docs/agile-development/README.md: -------------------------------------------------------------------------------- 1 | # Agile documentation 2 | 3 | - [Agile Basics](./basics/): Learn or refresh your basic agile knowledge. 4 | - [Agile Core Expectations](./core-expectations/): What are our core expectations from an Agile team. 5 | - [Agile Advanced Topics](./advanced-topics/): Go beyond the basics. 6 | -------------------------------------------------------------------------------- /.jscpd.json: -------------------------------------------------------------------------------- 1 | { 2 | "threshold": 0, 3 | "reporters": ["html", "markdown"], 4 | "ignore": [ 5 | "**/node_modules/**", 6 | "**/.git/**", 7 | "**/.rbenv/**", 8 | "**/.venv/**", 9 | "**/*cache*/**", 10 | "**/.github/**", 11 | "**/.idea/**", 12 | "**/report/**", 13 | "**/*.svg" 14 | ] 15 | } 16 | -------------------------------------------------------------------------------- /.pre-commit-config.yaml: -------------------------------------------------------------------------------- 1 | repos: 2 | - repo: https://github.com/markdownlint/markdownlint 3 | rev: v0.11.0 4 | hooks: 5 | - id: markdownlint 6 | name: Markdownlint 7 | description: Run markdownlint on your Markdown files 8 | entry: mdl 9 | language: ruby 10 | files: \.(md|mdown|markdown)$ 11 | -------------------------------------------------------------------------------- /docs/agile-development/advanced-topics/README.md: -------------------------------------------------------------------------------- 1 | # Agile Development advanced topics 2 | 3 | Documentation that help you going beyond the basics and core expectations. 4 | 5 | - [Backlog Management](./backlog-management/) 6 | - [Collaboration](./collaboration/) 7 | - [Effective Organization](./effective-organization/) 8 | - [Team Agreements](./team-agreements/) 9 | -------------------------------------------------------------------------------- /docs/code-reviews/recipes/README.md: -------------------------------------------------------------------------------- 1 | # Language Specific Guidance 2 | 3 | - [Bash](bash.md) 4 | - [C#](csharp.md) 5 | - [Go](go.md) 6 | - [Java](java.md) 7 | - [JavaScript and TypeScript](./javascript-and-typescript.md) 8 | - [Markdown](markdown.md) 9 | - [Python](python.md) 10 | - [Terraform](terraform.md) 11 | - [YAML (Azure Pipelines)](./azure-pipelines-yaml.md) 12 | -------------------------------------------------------------------------------- /docs/resources/templates/README.md: -------------------------------------------------------------------------------- 1 | 2 | # project-xyz 3 | 4 | Description of the project 5 | 6 | # Deploying to Azure 7 | 8 | # Getting started 9 | 10 | # Dependencies 11 | 12 | # Run it locally 13 | 14 | # 15 | 16 | By participating in this project, you 17 | agree to abide by the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/) 18 | -------------------------------------------------------------------------------- /docs/documentation/best-practices/good-documentation.md: -------------------------------------------------------------------------------- 1 | # Creating Good Documentation 2 | 3 | Review the [Documentation Review Checklist](../../code-reviews/recipes/markdown.md#code-review-checklist) for advice on how to write good documentation. 4 | 5 | Good documentation should follow good writing guidelines: [Writing Style Guidelines](../../code-reviews/recipes/markdown.md#writing-style-guidelines). 6 | -------------------------------------------------------------------------------- /docs/documentation/best-practices/establish-and-manage.md: -------------------------------------------------------------------------------- 1 | # Establishing and Managing Documentation 2 | 3 | Documentation should be source-controlled. [Pull Requests](../guidance/pull-requests.md) can be used to tell others about the changes, so they can be reviewed and discussed. E.g., [Async Design Reviews](../../design/design-reviews/recipes/async-design-reviews.md). 4 | 5 | Tools: 6 | 7 | - [Wikis](../tools/wikis.md). 8 | -------------------------------------------------------------------------------- /mkdocs-overrides/main.html: -------------------------------------------------------------------------------- 1 | {% extends "base.html" %} 2 | 3 | {% block analytics %} 4 | 11 | {% endblock %} 12 | -------------------------------------------------------------------------------- /docs/agile-development/advanced-topics/collaboration/README.md: -------------------------------------------------------------------------------- 1 | # Advanced recommendations for collaboration 2 | 3 | - [Why Collaboration](./why-collaboration.md) 4 | - [How to add a Pairing Custom Field in Azure DevOps User Stories](./add-pairing-field-azure-devops-cards.md) 5 | - [How to use the "Social Question of the Day"](./social-question.md) 6 | - [Engagement Team Development](./teaming-up.md) 7 | - [Virtual Collaboration and Pair Programming](./virtual-collaboration.md) 8 | -------------------------------------------------------------------------------- /docs/design/design-reviews/decision-log/examples/memory/README.md: -------------------------------------------------------------------------------- 1 | # Memory 2 | 3 | These examples were taken from the Memory project, an internal tool for tracking an individual's accomplishments. 4 | 5 | The main example here is the [Decision Log](Decision-Log.md). 6 | Since this log was used from the start, the decisions are mostly based on technology choices made in the start of the project. 7 | All line items have a link out to the trade studies done for each technology choice. 8 | -------------------------------------------------------------------------------- /docs/agile-development/advanced-topics/team-agreements/README.md: -------------------------------------------------------------------------------- 1 | # Team Agreements 2 | 3 | * [Definition of Done](./definition-of-done.md) 4 | * [Definition of Ready](./definition-of-ready.md) 5 | * [Working Agreements](./working-agreements.md) 6 | * [Team Manifesto](./team-manifesto.md) 7 | 8 | ## Goals 9 | 10 | Team agreements help clarify expectations for all team members, whether they are expectations around how the team works together (Working Agreements) or how to judge if a story is complete (Definition of Done). 11 | -------------------------------------------------------------------------------- /docs/documentation/tools/integrations.md: -------------------------------------------------------------------------------- 1 | # Integration with Teams/Slack 2 | 3 | Monitor your Azure repositories and receive notifications in your channel whenever code is pushed/checked in and whenever a pull request (PR) is created, updated, or a merge is attempted. 4 | 5 | - [Azure Repos with Microsoft Teams](https://docs.microsoft.com/en-us/azure/devops/repos/integrations/repos-teams?view=azure-devops) 6 | 7 | - [Azure Repos with Slack](https://docs.microsoft.com/en-us/azure/devops/repos/integrations/repos-slack?view=azure-devops) 8 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/bug_report.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Bug report 3 | about: Create a report to help us improve 4 | title: "[BUG]" 5 | labels: 'bug' 6 | assignees: '' 7 | --- 8 | Remember to label the issue with the appropriate area, and also assign it to yourself if you plan to work in the issue in the near future. 9 | 10 | **Describe the bug** 11 | A clear and concise description of what the bug is. 12 | 13 | **To Reproduce** 14 | Steps to reproduce the behavior: 15 | 16 | 1. Go to '...' 17 | 1. Click on '....' 18 | 1. Scroll down to '....' 19 | 1. See error 20 | -------------------------------------------------------------------------------- /.github/workflows/auto_request_review.yml: -------------------------------------------------------------------------------- 1 | name: Auto Request Review 2 | 3 | on: 4 | pull_request: 5 | types: [opened, ready_for_review, reopened] 6 | 7 | jobs: 8 | auto-request-review: 9 | name: Auto Request Review 10 | runs-on: ubuntu-latest 11 | steps: 12 | - name: Request review based on files changes and/or groups the author belongs to 13 | uses: necojackarc/auto-request-review@v0.7.0 14 | with: 15 | token: ${{ secrets.GITHUB_TOKEN }} 16 | config: .github/reviewers.yml # Config file location override 17 | -------------------------------------------------------------------------------- /docs/agile-development/basics/Roles/README.md: -------------------------------------------------------------------------------- 1 | # Agile/Scrum Roles 2 | 3 | This section has links directing you to definitions for the traditional roles within Agile/Scrum. After reading through the best practices you should have a basic understanding of the key Agile roles in terms of what they are and the expectations for the role. 4 | 5 | - [Product Owner](https://scrumguides.org/scrum-guide.html#product-owner 'Product Owner') 6 | - [Scrum Master](https://scrumguides.org/scrum-guide.html#scrum-master 'Scrum Master') 7 | - [Development Team](https://scrumguides.org/scrum-guide.html#developers 'Developers') 8 | -------------------------------------------------------------------------------- /docs/agile-development/advanced-topics/backlog-management/external-feedback.md: -------------------------------------------------------------------------------- 1 | # External Feedback 2 | 3 | Various stakeholders can provide feedback to the working product during a project, beyond any formal 4 | review and feedback sessions required by the organization. The frequency and method of collecting 5 | feedback through reviews varies depending on the case, but a couple of good practices are: 6 | 7 | - Capture each review in the backlog as a separate user story. 8 | - Standardize the tasks that implement this user story. 9 | - Plan for a review user story per Epic / Feature in your backlog proactively. 10 | -------------------------------------------------------------------------------- /docs/design/design-reviews/decision-log/doc/adr/0001-record-architecture-decisions.md: -------------------------------------------------------------------------------- 1 | # 1. Record architecture decisions 2 | 3 | Date: 2020-03-20 4 | 5 | ## Status 6 | 7 | Accepted 8 | 9 | ## Context 10 | 11 | We need to record the architectural decisions made on this project. 12 | 13 | ## Decision 14 | 15 | We will use Architecture Decision Records, as [described by Michael Nygard](http://thinkrelevance.com/blog/2011/11/15/documenting-architecture-decisions). 16 | 17 | ## Consequences 18 | 19 | See Michael Nygard's article, linked above. For a lightweight ADR toolset, see Nat Pryce's [adr-tools](https://github.com/npryce/adr-tools). 20 | -------------------------------------------------------------------------------- /docs/documentation/guidance/engineering-feedback.md: -------------------------------------------------------------------------------- 1 | # Engineering Feedback 2 | 3 | Good engineering feedback is: 4 | 5 | - Actionable 6 | - Specific 7 | - Detailed 8 | - Includes assets (script, data, code, etc.) to reproduce scenario and validate solution 9 | - Includes details about the customer scenario / what the customer was trying to achieve 10 | 11 | Refer to [Microsoft Engineering Feedback](../../engineering-feedback/README.md) for more details, including [guidance](../../engineering-feedback/feedback-guidance.md), [FAQ](../../engineering-feedback/feedback-faq.md) and [examples](../../engineering-feedback/feedback-examples.md). 12 | -------------------------------------------------------------------------------- /docs/agile-development/basics/README.md: -------------------------------------------------------------------------------- 1 | # Agile Development Basics 2 | 3 | If you are new to Agile development or if you are looking for a refresher, this section will provides links to information that provide best pracices for Backlog Management, Agile Ceremonies, Roles within Agile and Agile Sprints. 4 | 5 | - [What is Agile](https://docs.microsoft.com/en-us/devops/plan/what-is-agile 'What is Agile') 6 | - [What is Agile Development](https://docs.microsoft.com/en-us/devops/plan/what-is-agile-development 'What is Agile Development') 7 | - [Backlog Management](./Backlog%20Management/) 8 | - [Ceremonies](./Ceremonies/) 9 | - [Roles](./Roles/) 10 | - [Sprints](./Sprints/) 11 | -------------------------------------------------------------------------------- /docs/documentation/guidance/pull-requests.md: -------------------------------------------------------------------------------- 1 | # Pull Requests 2 | 3 | When we create [Pull Requests](../../code-reviews/pull-requests.md), we must ensure they are properly documented: 4 | 5 | - Title and Description 6 | - [Pull Request Description](../../code-reviews/pull-requests.md#pull-request-description) 7 | - [Pull Request Template](../../code-reviews/pull-request-template/pull-request-template.md) 8 | - Linked worked items 9 | - Comments 10 | - As an author, [address all comments](../../code-reviews/process-guidance/author-guidance.md#be-open-to-receive-feedback) 11 | - As a reviewer, [make comments clear](../../code-reviews/process-guidance/reviewer-guidance.md#make-comments-clear) 12 | -------------------------------------------------------------------------------- /docs/design/design-patterns/distributed-system-design-reference/README.md: -------------------------------------------------------------------------------- 1 | # Distributed System Design Reference 2 | 3 | Distributed systems introduce new and interesting problems that need addressing. 4 | Software engineering as a field has dealt with these problems for years, and there are phenomenal resources available for reference when creating a new distributed system. 5 | Some that we recommend are as follows: 6 | 7 | * [Martin Fowler's Patterns of Distributed Systems](https://martinfowler.com/articles/patterns-of-distributed-systems/) 8 | * [microservices.io](https://microservices.io/index.html) 9 | * [Azure's Cloud Design Patterns](https://docs.microsoft.com/en-us/azure/architecture/patterns/) 10 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/feature_request.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Feature request 3 | about: Suggest an idea for this project 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Is your feature request related to a problem? Please describe.** 11 | A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] 12 | 13 | **Describe the solution you'd like** 14 | A clear and concise description of what you want to happen. 15 | 16 | **Describe alternatives you've considered** 17 | A clear and concise description of any alternative solutions or features you've considered. 18 | 19 | **Additional context** 20 | Add any other context or screenshots about the feature request here. 21 | -------------------------------------------------------------------------------- /.github/reviewers.yml: -------------------------------------------------------------------------------- 1 | reviewers: 2 | # The default reviewers 3 | defaults: 4 | - repository-owners # group 5 | 6 | # Reviewer groups each of which has a list of GitHub usernames 7 | groups: 8 | repository-owners: 9 | - tessferrandez 10 | - fnocera 11 | - nyouens 12 | general: 13 | - tessferrandez 14 | - shiranr 15 | - tompaana 16 | 17 | files: 18 | # Keys are glob expressions. 19 | # You can assign groups defined above as well as GitHub usernames. 20 | "docs/**": 21 | - general # group 22 | 23 | options: 24 | ignore_draft: true 25 | ignored_keywords: 26 | - DO NOT REVIEW 27 | enable_group_assignment: false 28 | number_of_reviewers: 4 29 | -------------------------------------------------------------------------------- /docs/design/design-patterns/object-oriented-design-reference/README.md: -------------------------------------------------------------------------------- 1 | # Object-Oriented Design Reference 2 | 3 | When writing software for large projects, the hardest part is often communication and maintenance. 4 | Following proven design patterns can optimize for maintenance, readability, and ease of extension. 5 | In particular, object-oriented design patterns are well-established in the industry. 6 | 7 | Please refer to the following resources to create strong object-oriented designs: 8 | 9 | * [Class Diagram Overview](http://www.agilemodeling.com/artifacts/classDiagram.htm) 10 | * [Design Patterns Wikipedia](https://en.wikipedia.org/wiki/Design_Patterns) 11 | * [Object Oriented Design Website](https://www.oodesign.com/) 12 | -------------------------------------------------------------------------------- /docs/privacy/README.md: -------------------------------------------------------------------------------- 1 | # Privacy fundamentals 2 | 3 | This part of the engineering playbook focuses on privacy design guidelines and principles. 4 | Private data handling and protection requires both the proper design of software, 5 | systems and databases, as well as the implementation of organizational processes and procedures. 6 | 7 | In general, developers working on [CSE](../CSE.md) projects should adhere to 8 | Microsoft's recommended standard practices and regulations on Privacy and Data Handling. 9 | 10 | The playbook currently contains two main parts: 11 | 12 | 1. [Privacy and Data](data-handling.md): Best practices for properly handling sensitive and private data. 13 | 2. [Privacy frameworks](privacy-frameworks.md): A list of frameworks which could be applied in private data scenarios. 14 | -------------------------------------------------------------------------------- /docs/design/design-patterns/README.md: -------------------------------------------------------------------------------- 1 | # Design Patterns 2 | 3 | The design patterns section recommends patterns of software and architecture design. 4 | This section provides a curated list of commonly used patterns from trusted sources. 5 | Rather than duplicate or replace the cited sources, this section aims to compliment them with suggestions, guidance, and learnings based on firsthand experiences. 6 | 7 | ## Subsections 8 | 9 | * [Data Heavy Design Guidance](data-heavy-design-guidance/README.md) 10 | * [Object Oriented Design Reference](object-oriented-design-reference/README.md) 11 | * [Distributed System Design Reference](distributed-system-design-reference/README.md) 12 | * [REST API Design Guidance](rest-api-design-guidance/README.md) 13 | * [Cloud Resource Design Guidance](cloud-resource-design-guidance/README.md) 14 | -------------------------------------------------------------------------------- /docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/examples/pr-example.yaml: -------------------------------------------------------------------------------- 1 | on: 2 | pull_request: 3 | branches: 4 | - master 5 | 6 | jobs: 7 | Echo-On-PR: 8 | runs-on: ubuntu-latest 9 | steps: 10 | - name: "Checkout Repository" 11 | uses: actions/checkout@v3 12 | 13 | - name: "Set flag from PR" 14 | env: 15 | PR_VAR: ${{ contains(github.event.pull_request.body, '[pr var]') }} 16 | run: | 17 | if ${PR_VAR} == true; then 18 | echo "flag=true" >> $GITHUB_ENV 19 | echo "flag set to true" 20 | else 21 | echo "flag=false" >> $GITHUB_ENV 22 | echo "flag set to false" 23 | fi 24 | 25 | - name: "Use flag if true" 26 | if: env.flag 27 | run: echo "Flag is available and true" 28 | -------------------------------------------------------------------------------- /docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/examples/commit-example.yaml: -------------------------------------------------------------------------------- 1 | on: 2 | push: 3 | branches: 4 | - master 5 | 6 | jobs: 7 | Echo-On-Commit: 8 | runs-on: ubuntu-latest 9 | steps: 10 | - name: "Checkout Repository" 11 | uses: actions/checkout@v3 12 | 13 | - name: "Set flag from Commit" 14 | env: 15 | COMMIT_VAR: ${{ contains(github.event.head_commit.message, '[commit var]') }} 16 | run: | 17 | if ${COMMIT_VAR} == true; then 18 | echo "flag=true" >> $GITHUB_ENV 19 | echo "flag set to true" 20 | else 21 | echo "flag=false" >> $GITHUB_ENV 22 | echo "flag set to false" 23 | fi 24 | 25 | - name: "Use flag if true" 26 | if: env.flag 27 | run: echo "Flag is available and true" 28 | -------------------------------------------------------------------------------- /docs/agile-development/basics/Ceremonies/README.md: -------------------------------------------------------------------------------- 1 | # Agile Ceremonies basics 2 | 3 | This section has links directing you to best practices for conducting the Agile ceremonies. After reading through the best practices you should have a basic understanding of the key Agile ceremonies in terms of purpose, value and best practices around conducting and facilitating these ceremonies. 4 | 5 | - [Planning](https://scrumguides.org/scrum-guide.html#sprint-planning 'Sprint Planning') 6 | - [Refinement](https://docs.microsoft.com/devops/plan/what-is-agile-development#diligent-backlog-refinement 'Refinement') 7 | - [Retrospective](https://scrumguides.org/scrum-guide.html#sprint-retrospective 'Retrospective') 8 | - [Sprint Review/Demo](https://scrumguides.org/scrum-guide.html#sprint-review 'Sprint Review') 9 | - [Stand-Up/Daily Scrum](https://scrumguides.org/scrum-guide.html#daily-scrum 'Stand-up/Daily Scrum') 10 | -------------------------------------------------------------------------------- /.github/pull_request_template.md: -------------------------------------------------------------------------------- 1 | # Pull Request Template 2 | 3 | ## What are you trying to address 4 | 5 | - Describe the current behavior that you are modifying, and link to issue number 6 | - If you don't have an issue, browse through existing issues to see if this is already tracked as an issue, to assign yourself to the issue and also verify that no one else is already working on the issue. 7 | 8 | ## Description of new changes 9 | 10 | - Write a detailed description of all changes - and if appropriate, why they are needed 11 | 12 | ## For all pull requests 13 | 14 | - [ ] Link to the issue you are solving (so it gets closed) 15 | - [ ] Label the pull request with the appropriate area(s) 16 | - [ ] Assign potential reviewers (you may also want to contact them on Teams to ensure timely reviews) 17 | - [ ] Assign the project - Engineering Playbook Backlog 18 | - [ ] Assign the pull request to yourself 19 | -------------------------------------------------------------------------------- /mkdocs.yml: -------------------------------------------------------------------------------- 1 | site_name: Code With Engineering Playbook 2 | site_url: https://microsoft.github.io/code-with-engineering-playbook 3 | site_description: CSE Code With Engineering Playbook 4 | site_author: Microsoft CSE 5 | 6 | repo_url: https://github.com/microsoft/code-with-engineering-playbook/ 7 | edit_uri: "" 8 | 9 | theme: 10 | name: material 11 | custom_dir: mkdocs-overrides 12 | font: 13 | text: Roboto 14 | palette: 15 | primary: black 16 | logo: resources/ms_icon.png 17 | favicon: resources/ms_icon.png 18 | features: 19 | - navigation.instant 20 | - navigation.indexes 21 | 22 | plugins: 23 | - search 24 | - git-revision-date-localized 25 | 26 | markdown_extensions: 27 | - meta 28 | - admonition 29 | - pymdownx.highlight 30 | - pymdownx.superfences 31 | - pymdownx.pathconverter 32 | - pymdownx.tabbed 33 | - mdx_truly_sane_lists 34 | - pymdownx.tasklist 35 | -------------------------------------------------------------------------------- /docs/source-control/secrets-management.md: -------------------------------------------------------------------------------- 1 | # Working with Secrets in Source Control 2 | 3 | The best way to avoid leaking secrets is to store them in local/private files and exclude these from git tracking with a [.gitignore](https://git-scm.com/docs/gitignore) file. 4 | E.g. the following pattern will exclude all files with the extension `.private.config`: 5 | 6 | ```bash 7 | # remove private configuration 8 | *.private.config 9 | ``` 10 | 11 | For more details on proper management of credentials and secrets in source control, and handling an accidental commit of secrets to source control, please refer to the [Secrets Management](../continuous-delivery/secrets-management/README.md) document which has further information, split by language as well. 12 | 13 | As an extra security measure, apply [credential scanning](../continuous-integration/dev-sec-ops/secret-management/recipes/detect-secrets.md) in your CI/CD pipeline. 14 | -------------------------------------------------------------------------------- /.mega-linter.yml: -------------------------------------------------------------------------------- 1 | # Configuration file for MegaLinter 2 | # See all available variables at https://oxsecurity.github.io/megalinter/configuration/ and in linters documentation 3 | 4 | APPLY_FIXES: all # all, none, or list of linter keys 5 | # ENABLE: # If you use ENABLE variable, all other languages/formats/tooling-formats will be disabled by default 6 | # ENABLE_LINTERS: # If you use ENABLE_LINTERS variable, all other linters will be disabled by default 7 | DISABLE: 8 | - COPYPASTE # Comment to enable checks of excessive copy-pastes 9 | - HTML 10 | - DOCKERFILE 11 | - JSON 12 | SPELL_CSPELL_DISABLE_ERRORS: true 13 | SHOW_ELAPSED_TIME: true 14 | FILEIO_REPORTER: false 15 | PARALLEL: true 16 | GITHUB_STATUS_REPORTER: true 17 | VALIDATE_ALL_CODEBASE: false 18 | LOG_LEVEL: WARNING 19 | MARKDOWN_MARKDOWN_LINK_CHECK_ARGUMENTS: "-q" 20 | # DISABLE_ERRORS: true # Uncomment if you want MegaLinter to detect errors but not block CI to pass 21 | -------------------------------------------------------------------------------- /.github/workflows/stale_issues_flow: -------------------------------------------------------------------------------- 1 | name: Close inactive issues 2 | # At 00:00 on every day-of-week from Monday through Friday. 3 | on: 4 | schedule: 5 | - cron: "0 0 * * 1-5" 6 | 7 | jobs: 8 | close-issues: 9 | runs-on: ubuntu-latest 10 | permissions: 11 | issues: write 12 | pull-requests: write 13 | steps: 14 | - uses: actions/stale@v4 15 | with: 16 | days-before-issue-stale: 90 17 | days-before-issue-close: -1 18 | stale-issue-label: "stale" 19 | stale-issue-message: "This issue is stale because it has been open for over 90 days with no activity. Please comment on the issue to re-activate it, if it is still relevant." 20 | close-issue-message: "This issue was closed because it has been inactive for 14 days since being marked as stale." 21 | days-before-pr-stale: -1 22 | days-before-pr-close: -1 23 | repo-token: ${{ secrets.GITHUB_TOKEN }} 24 | -------------------------------------------------------------------------------- /docs/code-reviews/README.md: -------------------------------------------------------------------------------- 1 | # Code Reviews 2 | 3 | Developers working on [CSE](../CSE.md) projects should conduct peer code reviews on every pull request (or check-in to a shared branch). 4 | 5 | ## Goals 6 | 7 | Code review is a way to have a conversation about the code where participants will: 8 | 9 | - **Improve code quality** by identifying and removing defects before they can be introduced into shared code branches. 10 | - **Learn and grow** by having others review the code, we get exposed to unfamiliar design patterns or languages among other topics, and even break some bad habits. 11 | - **Shared understanding** between the developers over the project's code. 12 | 13 | ## Resources 14 | 15 | - [Code review tools](tools.md) 16 | - [Google's Engineering Practices documentation: How to do a code review](https://google.github.io/eng-practices/review/reviewer/) 17 | - [Best Kept Secrets of Peer Code Review](https://static1.smartbear.co/smartbear/media/pdfs/best-kept-secrets-of-peer-code-review_redirected.pdf) 18 | -------------------------------------------------------------------------------- /.github/workflows/mkdocs.yml: -------------------------------------------------------------------------------- 1 | name: Build and deploy mkdocs to https://microsoft.github.io/code-with-engineering-playbook 2 | on: 3 | push: 4 | branches: 5 | - main 6 | jobs: 7 | deploy: 8 | runs-on: ubuntu-latest 9 | steps: 10 | - uses: actions/checkout@v3 11 | with: 12 | submodules: "recursive" 13 | fetch-depth: 0 # Fetch all history for .GitInfo and .Lastmod 14 | - name: Setup Python 15 | uses: actions/setup-python@v1 16 | with: 17 | python-version: "3.7" 18 | architecture: "x64" 19 | - name: Install dependencies 20 | run: | 21 | python3 -m pip install --upgrade pip 22 | python3 -m pip install wheel 23 | python3 -m pip install -r requirements-docs.txt 24 | - name: Build site 25 | run: mkdocs build 26 | - name: Deploy 27 | uses: peaceiris/actions-gh-pages@v3 28 | with: 29 | github_token: ${{ secrets.GITHUB_TOKEN }} 30 | publish_dir: ./site 31 | -------------------------------------------------------------------------------- /docs/design/README.md: -------------------------------------------------------------------------------- 1 | # Design 2 | 3 | Designing software well is hard. 4 | 5 | CSE has collected a number of practices which we find help in the design process. 6 | This covers not only technical design of software, but also architecture design and non-functional requirements gathering for new projects. 7 | 8 | ## Goals 9 | 10 | - Provide recommendations for how to design software for maintainability, ease of extension, adherence to best practices, and sustainability. 11 | - Reference or define process or checklists to help ensure well-designed software. 12 | - Collate and point to reference sources (guides, repos, articles) that can help shortcut the learning process. 13 | 14 | ## Sections 15 | 16 | - [Design Patterns](design-patterns/README.md) 17 | - [Design Reviews](design-reviews/README.md) 18 | - [Non-Functional Requirements Guidance](design-patterns/non-functional-requirements-capture-guide.md) 19 | - [Sustainable Software Engineering](sustainability/README.md) 20 | 21 | ## Recipes 22 | 23 | - [Design Recipes](design-reviews/recipes/README.md) 24 | -------------------------------------------------------------------------------- /docs/agile-development/basics/Sprints/README.md: -------------------------------------------------------------------------------- 1 | # The Sprint 2 | 3 | This section has links directing you to best practices in regards to what a sprint is within agile and the practices around the sprint. After reading through the best practices you should have a basic understanding of Sprint Planning and the Sprint Backlog, Sprint Execution and the Daily Standup, Sprint Review and Sprint Retrospective and the key output of the sprint which is called the Increment. 4 | 5 | - [Sprint Planning and the Sprint Backlog](https://docs.microsoft.com/en-us/devops/plan/what-is-scrum#sprint-planning-and-the-sprint-backlog 'Sprint Planning and Sprint Backlog') 6 | - [Sprint Execution and the Daily Standup](https://docs.microsoft.com/en-us/devops/plan/what-is-scrum#sprint-execution-and-daily-scrum 'Sprint Execution and the Daily Standup') 7 | - [Sprint Review and Sprint Retrospective](https://docs.microsoft.com/en-us/devops/plan/what-is-scrum#sprint-review-and-sprint-retrospective 'Sprint Review and Sprint Retrospective') 8 | - [Increment](https://docs.microsoft.com/en-us/devops/plan/what-is-scrum#increment 'Increment') 9 | -------------------------------------------------------------------------------- /LICENSE-CODE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | Copyright (c) Microsoft Corporation 3 | 4 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and 5 | associated documentation files (the "Software"), to deal in the Software without restriction, 6 | including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, 7 | and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, 8 | subject to the following conditions: 9 | 10 | The above copyright notice and this permission notice shall be included in all copies or substantial 11 | portions of the Software. 12 | 13 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT 14 | NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 15 | IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, 16 | WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 17 | SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. -------------------------------------------------------------------------------- /docs/resources/templates/CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing 2 | 3 | We love pull requests from everyone. By participating in this project, you 4 | agree to abide by the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/) 5 | 6 | Fork, then clone the repo 7 | 8 | Make sure the tests pass 9 | 10 | Make your change. Add tests for your change. Make the tests pass 11 | 12 | Push to your fork and [submit a pull request][pr]. 13 | 14 | [pr]: https://github.com/xyz 15 | 16 | At this point you're waiting on us. We like to at least comment on pull requests 17 | within three business days (and, typically, one business day). We may suggest 18 | some changes or improvements or alternatives. 19 | 20 | Some things that will increase the chance that your pull request is accepted: 21 | 22 | * Write tests. 23 | * Follow our [engineering playbook][playbook] and the [style guide][style] for this project. 24 | * Write a [good commit message][commit]. 25 | 26 | [playbook]: https://github.com/cloudbeatsch/code-with-engineering-playbook 27 | [style]: https://github.com/xyz 28 | [commit]: http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html 29 | -------------------------------------------------------------------------------- /docs/design/design-reviews/decision-log/doc/decision-log.md: -------------------------------------------------------------------------------- 1 | # Decision Log 2 | 3 | This document is used to track key decisions that are made during the course of the project. 4 | This can be used at a later stage to understand why decisions were made and by whom. 5 | 6 | | **Decision** | **Date** | **Alternatives Considered** | **Reasoning** | **Detailed doc** | **Made By** | **Work Required** | 7 | |----------------------------------------------|-----------------------------|--------------------------------------------|---------------------------------------------------------------|-----------------------------------------------|-------------------------|---------------------------------------------| 8 | | A one-sentence summary of the decision made. | Date the decision was made. | A list of the other approaches considered. | A two to three sentence summary of why the decision was made. | A link to the ADR with the format [Title] DR. | Who made this decision? | A link to the work item for the linked ADR. | 9 | -------------------------------------------------------------------------------- /docs/resources/templates/LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2017 Microsoft Corporation 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. -------------------------------------------------------------------------------- /docs/source-control/naming-branches.md: -------------------------------------------------------------------------------- 1 | # Naming branches 2 | 3 | When contributing to existing projects, look for and stick with the agreed branch naming convention. In open source projects this information is typically found in the contributing instructions, often in a file named `CONTRIBUTING.md`. 4 | 5 | In the beginning of a new project the team agrees on the project conventions including the branch naming strategy. 6 | 7 | Here's an example of a branch naming convention: 8 | 9 | ```plaintext 10 | /[feature/bug/hotfix]/_ 11 | ``` 12 | 13 | Which could translate to something as follows: 14 | 15 | ```plaintext 16 | dickinson/feature/271_add_more_cowbell 17 | ``` 18 | 19 | The example above is just that - an example. The team can choose to omit or add parts. Choosing a branch convention can depend on the development model (e.g. [trunk-based development](https://trunkbaseddevelopment.com/)), [versioning](component-versioning.md) model, tools used in managing source control, matter of taste etc. Focus on simplicity and reducing ambiguity; a good branch naming strategy allows the team to understand the purpose and ownership of each branch in the repository. 20 | -------------------------------------------------------------------------------- /docs/agile-development/basics/Backlog Management/README.md: -------------------------------------------------------------------------------- 1 | # Backlog Management basics for the Product and Sprint backlog 2 | 3 | This section has links directing you to best practices for managing Product and Sprint backlogs. After reading through the best practices you should have a basic understanding for managing both product and sprint backlogs, how to create acceptance criteria for user stories, creating a definition of done and definition of ready for user stories and the basics around estimating user stories. 4 | 5 | - [Product Backlog](https://scrumguides.org/scrum-guide.html#product-backlog 'Product Backlog') 6 | - [Sprint Backlog](https://scrumguides.org/scrum-guide.html#sprint-backlog 'Sprint Backlog') 7 | - [Acceptance Criteria](https://docs.microsoft.com/en-us/azure/devops/boards/backlogs/best-practices-product-backlog?view=azure-devops#acceptance-criteria 'Acceptance Criteria') 8 | - [Definition of Done](https://scrumguides.org/scrum-guide.html#increment 'Definition of Done') 9 | - [Definition of Ready](https://www.scrum.org/resources/blog/walking-through-definition-ready 'Definition of Ready') 10 | - [Estimation Basics in Agile](https://www.scrum.org/resources/blog/what-scrum-says-about-estimates 'Estimation Basics in Agile') 11 | -------------------------------------------------------------------------------- /docs/automated-testing/unit-testing/custom-connector.md: -------------------------------------------------------------------------------- 1 | # Custom Connector Testing 2 | 3 | When developing Custom Connectors to put data into the Power Platform there are some strategies you can follow: 4 | 5 | ## Unit Testing 6 | 7 | There are several verifications one can do while developing custom connectors in order to be sure the code is working properly. 8 | 9 | There are two main ones: 10 | 11 | - Validating the OpenAPI schema which the connector is defined. 12 | - Validating if the schema also have all the information necessary for the certified connector process. 13 | 14 | (the later one is optional, but necessary in case you want to publish it as a certified connector). 15 | 16 | There are several tool to help validate the OpenAPI schema, a list of them are available in this [link](https://openapi.tools/#description-validators). A suggested tool would be [swagger-cli](https://github.com/APIDevTools/swagger-cli). 17 | 18 | On the other hand, to validate if the custom connector you are building is correct to become a certified connector, use the [paconn-cli](https://github.com/microsoft/PowerPlatformConnectors/tree/dev/tools/paconn-cli), since it has a validate command that shows missing information from the custom connector definition. 19 | -------------------------------------------------------------------------------- /docs/design/design-reviews/decision-log/doc/adr/0002-app-level-logging.md: -------------------------------------------------------------------------------- 1 | # 2. App-level Logging with Serilog and Application Insights 2 | 3 | Date: 2020-04-08 4 | 5 | ## Status 6 | 7 | Accepted 8 | 9 | ## Context 10 | 11 | Due to the microservices design of the platform, we need to ensure consistency of logging throughout each service so tracking of usage, performance, errors etc. can be performed end-to-end. A single logging/monitoring framework should be used where possible to achieve this, whilst allowing the flexibility for integration/export into other tools at a later stage. The developers should be equipped with a simple interface to log messages and metrics. 12 | 13 | ## Decision 14 | 15 | We have agreed to utilize Serilog as the Dotnet Logging framework of choice at the application level, with integration into Log Analytics and Application Insights for analysis. 16 | 17 | ## Consequences 18 | 19 | Sampling will need to be configured in Application Insights so that it does not become overly-expensive when ingesting millions of messages, but also does not prevent capture of essential information. 20 | The team will need to only log what is agreed to be essential for monitoring as part of design reviews, to reduce noise and unnecessary levels of sampling. 21 | -------------------------------------------------------------------------------- /docs/continuous-integration/dev-sec-ops/static-code-analysis/static_code_analysis.md: -------------------------------------------------------------------------------- 1 | # Static Code Analysis 2 | 3 | Static code analysis is a method of detecting security issues by examining the source code of the application. 4 | 5 | ## Why Static Code Analysis 6 | 7 | Compared to code reviews, Static code analysis tools are more fast, accurate and through. 8 | As it operates on the source code itself, it is a very early indicator for issues, and coding errors found earlier are less costly to fix. 9 | 10 | ## Applying Static Code Analysis 11 | 12 | Static Code Analysis should be integrated in your build process. 13 | There are [many tools](https://owasp.org/www-community/Source_Code_Analysis_Tools) available for Static Code Analysis, choose the ones that meet your programming language and development techniques. 14 | 15 | ## Static Code Analysis Frameworks and Tools 16 | 17 | [SonarCloud](https://sonarcloud.io) - static code analysis with cloud-based software as a service product. 18 | [OWASP Source code Analysis](https://owasp.org/www-community/Source_Code_Analysis_Tools) - OWASP recommendations for source code analysis tools 19 | 20 | ## Conclusion 21 | 22 | Static code analysis is essential to identify potential problems and security issues in the code. It allows you to detect bugs and security issues at an early stage. 23 | -------------------------------------------------------------------------------- /docs/design/design-reviews/recipes/sprint-spike-template.md: -------------------------------------------------------------------------------- 1 | # Spike: {Name} 2 | 3 | - **Conducted by:** {Names and at least one email address for follow-up questions} 4 | - **Backlog Work Item:** {Link to the work item to provide more context} 5 | - **Sprint**: {Which sprint did the study take place. Include sprint start date} 6 | 7 | ## Goal 8 | 9 | Describe what question(s) the spike intends to answer and why. 10 | 11 | ## Method 12 | 13 | Describe how the team will uncover the answer to the question(s) the spike intends to answer. For example: 14 | 15 | 1. Build prototype to test. 16 | 1. Research existing documents and samples. 17 | 1. Discuss with subject matter experts. 18 | 19 | ## Evidence 20 | 21 | Document the evidence collected that informed the conclusions below. Examples may include: 22 | 23 | - Recorded or live demos of a prototype providing the desired capabilities 24 | - Metrics collected while testing the prototype 25 | - Documentation that indicates the solution can provided the desired capabilities 26 | 27 | ## Conclusions 28 | 29 | What was the answer to the question(s) outlined at the start of the spike? Capture what was learned that will inform future work. 30 | 31 | ## Next Steps 32 | 33 | What work is expected as an outcome of the learning within this spike. Was there work that was blocked or dependent on the learning within this spike? 34 | -------------------------------------------------------------------------------- /docs/design/design-reviews/recipes/README.md: -------------------------------------------------------------------------------- 1 | # Design Review Recipes 2 | 3 | Design reviews come in all shapes and sizes. There are also different items to consider when creating a design at different stages during an engagement 4 | 5 | ## Design Review Process 6 | 7 | - [Incorporate design reviews throughout the lifetime of an engagement](./engagement-process.md) 8 | 9 | ## Design Review Templates 10 | 11 | ### [Game Plan](./high-level-design-recipe.md) 12 | 13 | - The same template already in use today 14 | - High level architecture and design 15 | - Includes technologies, languages & products to complete engagement objective 16 | 17 | ### [Milestone / Epic Design Review](./milestone-epic-design-review-recipe.md) 18 | 19 | - Should be considered when an engagement contains multiple milestones or epics 20 | - Design should be more detailed than game plan 21 | - May require unique deployment, security and/or privacy characteristics from other milestones 22 | 23 | ### [Feature/story design review](./feature-story-design-review-template.md) 24 | 25 | - Design for complex features or stories 26 | - Will reuse deployment, security and other characteristics defined within game plan or milestone 27 | - May require new libraries, OSS or patterns to accomplish goals 28 | 29 | ### [Task design review](./task-design-review-template.md) 30 | 31 | - Highly detailed design for a complex tasks with many unknowns 32 | - Will integrate into higher level feature/component designs 33 | -------------------------------------------------------------------------------- /docs/agile-development/advanced-topics/team-agreements/definition-of-done.md: -------------------------------------------------------------------------------- 1 | # Definition of Done 2 | 3 | To close a user story, a sprint, or a milestone it is important to verify that the tasks are complete. 4 | 5 | The development team should decide together what their Definition of Done is and document this in the project. Below are some examples of checks to verify that the user story, sprint, task is completed. 6 | 7 | ## Feature/User Story 8 | 9 | - [ ] Acceptance criteria are met 10 | - [ ] Refactoring is complete 11 | - [ ] Code builds with no error 12 | - [ ] Unit tests are written and pass 13 | - [ ] Existing Unit Tests pass 14 | - [ ] Sufficient diagnostics/telemetry are logged 15 | - [ ] Code review is complete 16 | - [ ] UX review is complete (if applicable) 17 | - [ ] Documentation is updated 18 | - [ ] The feature is merged into the develop branch 19 | - [ ] The feature is signed off by the product owner 20 | 21 | ## Sprint Goal 22 | 23 | - [ ] Definition of Done for all user stories included in the sprint are met 24 | - [ ] Product backlog is updated 25 | - [ ] Functional and Integration tests pass 26 | - [ ] Performance tests pass 27 | - [ ] End 2 End tests pass 28 | - [ ] All bugs are fixed 29 | - [ ] The sprint is signed off from developers, software architects, project manager, product owner etc. 30 | 31 | ## Release/Milestone 32 | 33 | - [ ] Code Complete (goals of sprints are met) 34 | - [ ] Release is marked as ready for production deployment by product owner 35 | -------------------------------------------------------------------------------- /docs/documentation/best-practices/automation.md: -------------------------------------------------------------------------------- 1 | # Replacing Documentation with Automation 2 | 3 | You can document how to set up your dev machine with the right version of the framework required to run the code, which extensions are useful to develop the application with your editor, or how to configure your editor to launch and debug the application. If it is possible, a better solution is to provide the means to automate tool installs, application startup, etc., instead. 4 | 5 | Some examples are provided below: 6 | 7 | ## Dev containers in Visual Studio Code 8 | 9 | The [Visual Studio Code Remote - Containers extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) lets you use a Docker container as a full-featured development environment. It allows you to open any folder inside (or mounted into) a container and take advantage of Visual Studio Code's full feature set. 10 | 11 | Additional information: [Developing inside a Container](https://code.visualstudio.com/docs/remote/containers). 12 | 13 | ## Launch configurations and Tasks in Visual Studio Code 14 | 15 | [Launch configurations](https://code.visualstudio.com/Docs/editor/debugging#_launch-configurations) allows you to configure and save debugging setup details. 16 | 17 | [Tasks](https://code.visualstudio.com/Docs/editor/tasks) can be configured to run scripts and start processes so that many of these existing tools can be used from within VS Code without having to enter a command line or write new code. 18 | -------------------------------------------------------------------------------- /docs/observability/README.md: -------------------------------------------------------------------------------- 1 | # Observability 2 | 3 | Building observable systems enables development teams at CSE to measure how well the application is behaving. Observability serves the following goals: 4 | 5 | - Provide holistic view of the _application health_. 6 | - Help measure _business performance_ for the customer. 7 | - Measure _operational performance_ of the system. 8 | - Identify and _diagnose failures_ to get to the problem fast. 9 | 10 | ## Pillars of Observability 11 | 12 | - [Logs](pillars/logging.md) 13 | - [Metrics](pillars/metrics.md) 14 | - [Tracing](pillars/tracing.md) 15 | - [Logs vs Metrics vs Traces](log-vs-metric-vs-trace.md) 16 | 17 | ## Insights 18 | 19 | - [Dashboards and Reporting](pillars/dashboard.md) 20 | 21 | ## Tools, Patterns and Recommended Practices 22 | 23 | - [Tooling and Patterns](tools/README.md) 24 | - [Observability As Code](observability-as-code.md) 25 | - [Recommended Practices](best-practices.md) 26 | - [Diagnostics tools](diagnostic-tools.md) 27 | - [OpenTelemetry](tools/OpenTelemetry.md) 28 | 29 | ## Facets of Observability 30 | 31 | - [Observability for Microservices](microservices.md) 32 | - [Observability in Machine Learning](ml-observability.md) 33 | - [Observability of CI/CD Pipelines](observability-pipelines.md) 34 | - [Observability in Azure Databricks](observability-databricks.md) 35 | - [Recipes](recipes-observability.md) 36 | 37 | ## Useful links 38 | 39 | - [Non-Functional Requirements Guidance](../design/design-patterns/non-functional-requirements-capture-guide.md) 40 | -------------------------------------------------------------------------------- /.devcontainer/Dockerfile: -------------------------------------------------------------------------------- 1 | 2 | # See here for image contents: https://github.com/microsoft/vscode-dev-containers/blob/v0.202.3/containers/python-3/.devcontainer/base.Dockerfile 3 | # [Choice] Python version (use -bullseye variants on local arm64/Apple Silicon): 3, 3.9, 3.8, 3.7, 3.6, 3-bullseye, 3.9-bullseye, 3.8-bullseye, 3.7-bullseye, 3.6-bullseye, 3-buster, 3.9-buster, 3.8-buster, 3.7-buster, 3.6-buster 4 | ARG VARIANT=3-bullseye 5 | FROM mcr.microsoft.com/vscode/devcontainers/python:0-${VARIANT} 6 | 7 | # [Optional] Allow the vscode user to pip install globally w/o sudo 8 | ENV PIP_TARGET=/usr/local/pip-global 9 | ENV PYTHONPATH=${PIP_TARGET}:${PYTHONPATH} 10 | ENV PATH=${PIP_TARGET}/bin:${PATH} 11 | RUN if ! cat /etc/group | grep -e "^pip-global:" > /dev/null 2>&1; then groupadd -r pip-global; fi \ 12 | && usermod -a -G pip-global vscode \ 13 | && umask 0002 && mkdir -p ${PIP_TARGET} \ 14 | && chown :pip-global ${PIP_TARGET} \ 15 | && ( [ ! -f "/etc/profile.d/00-restore-env.sh" ] || sed -i -e "s/export PATH=/export PATH=\/usr\/local\/pip-global:/" /etc/profile.d/00-restore-env.sh ) 16 | 17 | # [Choice] Node.js version: none, lts/*, 16, 14, 12, 10 18 | ARG NODE_VERSION="lts/*" 19 | RUN if [ "${NODE_VERSION}" != "none" ]; then su vscode -c "umask 0002 && . /usr/local/share/nvm/nvm.sh && nvm install ${NODE_VERSION} 2>&1"; fi 20 | 21 | # [Optional] If your pip requirements rarely change, uncomment this section to add them to the image. 22 | COPY requirements-docs.txt /tmp/pip-tmp/ 23 | RUN pip3 --disable-pip-version-check --no-cache-dir install -r /tmp/pip-tmp/requirements-docs.txt \ 24 | && rm -rf /tmp/pip-tmp -------------------------------------------------------------------------------- /docs/design/design-reviews/recipes/preferred-diagram-tooling.md: -------------------------------------------------------------------------------- 1 | # Preferred Diagram Tooling 2 | 3 | At each stage in the engagement process, diagrams are a key part of the design review. 4 | The preferred tooling for creating and maintaining diagrams is the `.drawio` format from [diagrams.net](http://diagrams.net) (formerly [draw.io](http://draw.io)) 5 | 6 | The `.drawio` format uses the metadata layer within the PNG file-format to hide SVG vector graphics representation, then renders the .png when saving. 7 | This clever use of both the meta layer and image layer allows anyone to further edit the PNG file. 8 | It also renders like a normal PNG in browsers and other viewers, making it easy to transfer and embed. 9 | Furthermore, it can be edited within VSCode very easily using the [Draw.io Integration VSCode Extension](https://marketplace.visualstudio.com/items?itemName=hediet.vscode-drawio). 10 | 11 | This tooling can be used like Visio or LucidChart, without the licensing/remote storage concerns. 12 | Furthermore, Diagrams.net has a collection of Azure/Office/Microsoft icons, as well as other well-known tech, so it is not only useful for swimlanes and flow diagrams, but also for architecture diagrams. 13 | 14 | Furthermore, we recommend storing the exported PNG images from these diagrams in the repo along with the .drawio files so they can easily be referenced in documentation and more easily reviewed during PRs. 15 | Lastly, while diagrams.net is generally preferred, if any other diagraming tooling is used that can generate diagram images from code, both the diagram file and the image file should be stored together in the repo for the above reasons. 16 | -------------------------------------------------------------------------------- /docs/automated-testing/templates/case-study-template.md: -------------------------------------------------------------------------------- 1 | # ~Customer Project~ Case Study 2 | 3 | ## Background 4 | 5 | Describe the customer and business requirements with the explicit problem statement. 6 | 7 | ## System Under Test (SUT) 8 | 9 | Include the system's conceptual architecture and highlight the architecture components that were included in the E2E testing. 10 | 11 | ## Problems and Limitations 12 | 13 | Describe about the problems of the overall SUT solution that prevented from testing specific (or any) part of the solution. 14 | Describe limitation of the testing tools and framework(s) used in this implementation 15 | 16 | ## E2E Testing Framework and Tools 17 | 18 | Describe what testing framework and/or tools were used to implement E2E testing in the SUT. 19 | 20 | ## Test Cases 21 | 22 | Describe the E2E test cases were created to E2E test the SUT 23 | 24 | ## Test Metrics 25 | 26 | Describe any architecture solution were used to monitor, observe and track the various service states that were used as the E2E testing metrics. Also, include the list of test cases were build to measure the progress of E2E testing. 27 | 28 | ## E2E Testing Architecture 29 | 30 | Describe any testing architecture were built to run E2E testing. 31 | 32 | ## E2E Testing Implementation (Code samples) 33 | 34 | Include sample test cases and their implementation in the programming language of choice. 35 | Include any common reusable code implementation blocks that could be leveraged in the future project's E2E testing implementation. 36 | 37 | ## E2E Testing Reporting and Results 38 | 39 | Include sample of E2E testing reports and results obtained from the E2E testing runs in this project. 40 | -------------------------------------------------------------------------------- /docs/observability/pillars/tracing.md: -------------------------------------------------------------------------------- 1 | # Tracing 2 | 3 | ## Overview 4 | 5 | Produces the information required to observe series of correlated operations in a distributed system. Once collected they show the path, measurements and faults in an end-to-end transaction. 6 | 7 | ## Best Practices 8 | 9 | - Ensure that at least key business transactions are traced. 10 | - Include in each trace necessary information to identify software releases (i.e. service name, version). This is important to correlate deployments and system degradation. 11 | - Ensure dependencies are included in trace (databases, I/O). 12 | - If costs are a concern use sampling, avoiding throwing away errors, unexpected behavior and critical information. 13 | - Don't reinvent the wheel, use existing tools to collect and analyze the data. 14 | - Ensure personal identifiable information policies and restrictions are followed. 15 | 16 | ## Recommended Tools 17 | 18 | - [Azure Monitor](https://docs.microsoft.com/en-us/azure/azure-monitor/overview) - Umbrella of services including system metrics, log analytics and more. 19 | - [Jaeger Tracing](https://www.jaegertracing.io) - Open source, end-to-end distributed tracing. 20 | - [Grafana](https://grafana.com) - Open source dashboard & visualization tool. Supports Log, Metrics and Distributed tracing data sources. 21 | 22 | >Consider using [OpenTelemetry](../tools/OpenTelemetry.md) as it implements open-source cross-platform context propagation for end-to-end distributed transactions over heterogeneous components out-of-the-box. It takes care of automatically creating and managing the Trace Context object among a full stack of microservices implemented across different technical stacks. 23 | -------------------------------------------------------------------------------- /docs/design/design-patterns/cloud-resource-design-guidance/README.md: -------------------------------------------------------------------------------- 1 | # Cloud Resource Design Guidance 2 | 3 | As cloud usage scales, considerations for subscription design, management groups, and resource naming/tagging conventions have an impact on governance, operations management, and adoption patterns. 4 | 5 | > **NOTE:** Always work with the relevant stakeholders to ensure that introducing new patterns provides the intended value. 6 | > 7 | > When working in an existing cloud environment, it is important to understand any current patterns and how they are used before making a change to them. 8 | 9 | ## References 10 | 11 | The following references can be used to understand the lastest best practices in organizing cloud resources: 12 | 13 | * [Organizing Subscriptions](https://docs.microsoft.com/en-us/azure/cloud-adoption-framework/ready/azure-best-practices/organize-subscriptions) 14 | * [Resource Tagging Decision Guide](https://docs.microsoft.com/en-us/azure/cloud-adoption-framework/decision-guides/resource-tagging/) 15 | * [Resource Naming Conventions](https://docs.microsoft.com/en-us/azure/cloud-adoption-framework/ready/azure-best-practices/resource-naming) 16 | * [Recommended Azure Resource Abbreviations](https://docs.microsoft.com/en-us/azure/cloud-adoption-framework/ready/azure-best-practices/resource-abbreviations) 17 | * [Organizing Dev/Test/Production Workloads](https://docs.microsoft.com/en-us/azure/cloud-adoption-framework/ready/enterprise-scale/faq#how-do-we-handle-devtestproduction-workload-landing-zones-in-enterprise-scale-architecture) 18 | 19 | ## Tooling 20 | 21 | * [Azure Resource Naming Tool](https://github.com/microsoft/CloudAdoptionFramework/tree/master/ready/AzNamingTool) 22 | -------------------------------------------------------------------------------- /docs/observability/observability-pipelines.md: -------------------------------------------------------------------------------- 1 | # Observability of CI/CD Pipelines 2 | 3 | With increasing complexity to delivery pipelines, it is very important 4 | to consider Observability in the context of build and release of 5 | applications. 6 | 7 | ## Benefits 8 | 9 | - Having proper instrumentation during build time helps gain insights into the various stages of the build and release process. 10 | 11 | - Helps developers understand where the pipeline performance bottlenecks are, based on the data collected. This 12 | helps in having data-driven conversations around identifying latency between jobs, performance issues, 13 | artifact upload/download times providing valuable insights into agents availability and capacity. 14 | 15 | - Helps to identify trends in failures, thus allowing developers to quickly do root cause analysis. 16 | 17 | - Helps to provide an organization-wide view of pipeline health to easily identify trends. 18 | 19 | ## Points to Consider 20 | 21 | - It is important to identify the Key Performance Indicators (KPIs) for evaluating a successful CI/CD pipeline. Where needed, additional tracing can be added to better record KPI metrics. For example, adding pipeline build tags to identify a 'Release Candidate' vs. 'Non-Release Candidate' helps in evaluating the end-to-end release process timeline. 22 | 23 | - Depending on the tooling used (Azure DevOps, Jenkins etc.,), basic reporting on the pipelines is 24 | available out-of-the-box. It is important to evaluate these reports against the KPIs to understand if 25 | a custom reporting solution for their pipelines is needed. If required, custom dashboards can be built using 26 | third-party tools like Grafana or Power BI Dashboards. 27 | -------------------------------------------------------------------------------- /docs/CSE.md: -------------------------------------------------------------------------------- 1 | # Who We Are 2 | 3 | Our team, CSE (Commercial Software Engineering), works side by side with customers to help them tackle their toughest technical problems both in the cloud and on the edge. We meet customers where they are, work in the languages they use, with the open source frameworks they use, on the operating systems they use. We work with enterprises and start-ups across many industries from financial services to manufacturing. Our work covers a broad spectrum of domains including IoT, machine learning, and high scale compute. Our "superpower" is that we work closely with both our customers’ engineering teams and Microsoft’s product engineering teams, developing real-world expertise that we can use to help our customers grow their business and help Microsoft improve our products and services. 4 | 5 | We are very community focused in our work, with one foot in Microsoft and one foot in the open source communities that we help. We make pull requests on open source projects to add support for Microsoft platforms and/or improve existing implementations. We build frameworks and other tools to make it easier for developers to use Microsoft platforms. We source all the ideas for this work by maintaining very deep connections with these communities and the customers and partners that use them. 6 | 7 | If you like variety, coding in many languages, using any available tech across our industry, digging in with our customers, hack fests, occasional travel, and telling the story of what you’ve done in [blog posts](https://www.microsoft.com/developerblog/) and at conferences, then come talk to us. 8 | 9 | > You can check out some of our work on our [Developer Blog](https://www.microsoft.com/developerblog/) 10 | -------------------------------------------------------------------------------- /docs/continuous-integration/dev-sec-ops/README.md: -------------------------------------------------------------------------------- 1 | # DevSecOps 2 | 3 | ## The concept of DevSecOps 4 | 5 | DevSecOps or DevOps security is about introducing security earlier in the life cycle of application development (a.k.a shift-left), thus minimizing the impact of vulnerabilities and bringing security closer to development team. 6 | 7 | ## Why 8 | 9 | By embracing shift-left mentality, DevSecOps encourages organizations to bridge the gap that often exists between development and security teams to the point where many of the security processes are automated and are effectively handled by the development team. 10 | 11 | ## DevSecOps Practices 12 | 13 | This section covers different tools, frameworks and resources allowing introduction of DevSecOps best practices to your project at early stages of development. 14 | Topics covered: 15 | 16 | 1. [Credential Scanning](./secret-management/credential_scanning.md) - automatically inspecting a project to ensure that no secrets are included in the project's source code. 17 | 1. [Secrets Rotation](./secret-management/secrets_rotation.md) - automated process by which the secret, used by the application, is refreshed and replaced by a new secret. 18 | 1. [Static Code Analysis](./static-code-analysis/static_code_analysis.md) - analyze source code or compiled versions of code to help find security flaws. 19 | 1. [Penetration Testing](./penetration-testing/penetration_testing.md) - a simulated attack against your application to check for exploitable vulnerabilities. 20 | 1. [Container Dependencies Scanning](./dependency-container-scanning/dependency_container_scanning.md) - search for vulnerabilities in container operating systems, language packages and application dependencies. 21 | -------------------------------------------------------------------------------- /docs/continuous-integration/dev-sec-ops/penetration-testing/penetration_testing.md: -------------------------------------------------------------------------------- 1 | # Penetration Testing 2 | 3 | A penetration test is a simulated attack against your application to check for exploitable security issues. 4 | 5 | ## Why Penetration Testing 6 | 7 | Penetration testing performed on a running application. As such, it tests the application E2E with all of its layers. It's output is a real simulated attack on the application that succeeded, therefore it is a critical issue in your application and should be addressed as soon as possible. 8 | 9 | ## Applying Penetration Testing 10 | 11 | Many organizations perform manual penetration testing. But new vulnerabilities found every day. Therefore, it is a good practice to have an automated penetration testing performed. 12 | To achieve this automation use penetration testing tools to uncover vulnerabilities, such as unsanitized inputs that are susceptible to code injection attacks. 13 | Insights provided by the penetration test can then be used to fine-tune your WAF security policies and patch detected vulnerabilities. 14 | 15 | ## Penetration Testing Frameworks and Tools 16 | 17 | [OWASP Zed Attack Proxy (ZAP)](https://www.zaproxy.org/) - OWASP penetration testing tool for web applications. 18 | 19 | ## Conclusion 20 | 21 | Penetration testing is essential to check for vulnerabilities in your application and protect it from simulated attacks. Insights provided by Penetration testing can identify weak spots in an organization's security posture, as well as measure the compliance of its security policy, test the staff's awareness of security issues and determine whether -- and how -- the organization would be subject to security disasters. 22 | -------------------------------------------------------------------------------- /docs/documentation/guidance/rest-apis.md: -------------------------------------------------------------------------------- 1 | # REST APIs 2 | 3 | When creating [REST APIs](https://en.wikipedia.org/wiki/Representational_state_transfer), you can leverage the [OpenAPI-Specification (OAI)](https://github.com/OAI/OpenAPI-Specification/) (originally known as the Swagger Specification) to describe them: 4 | 5 | > The OpenAPI Specification (OAS) defines a standard, programming language-agnostic interface description for HTTP APIs, which allows both humans and computers to discover and understand the capabilities of a service without requiring access to source code, additional documentation, or inspection of network traffic. When properly defined via OpenAPI, a consumer can understand and interact with the remote service with a minimal amount of implementation logic. 6 | > 7 | >Use cases for machine-readable API definition documents include, but are not limited to: interactive documentation; code generation for documentation, clients, and servers; and automation of test cases. OpenAPI documents describe an APIs services and are represented in either YAML or JSON formats. These documents may either be produced and served statically or be generated dynamically from an application. 8 | 9 | There are implementations available for many languages like C#, including low-level tooling, editors, user interfaces, code generators, etc. Here you can find a list of known tooling for the different languages: [OpenAPI-Specification/IMPLEMENTATIONS.md](https://github.com/OAI/OpenAPI-Specification/blob/main/IMPLEMENTATIONS.md). 10 | 11 | More information: 12 | 13 | - [ASP.NET Core web API documentation with Swagger / OpenAPI](https://docs.microsoft.com/en-us/aspnet/core/tutorials/web-api-help-pages-using-swagger?view=aspnetcore-5.0). 14 | -------------------------------------------------------------------------------- /docs/security/rules-of-engagement.md: -------------------------------------------------------------------------------- 1 | # Rules of Engagement 2 | 3 | When performing application security analysis, it is expected that the tester follow the *Rules of Engagement* as laid out below. This is to standardize the scope of application testing and provide a concrete awareness of what is considered "out of scope" for security analysis. 4 | 5 | ## Rules of Engagement - For those requesting review 6 | 7 | * Web Application Firewalls can be up and configured, but do not enable any automatic blocking. This can greatly slow down the person performing the test. 8 | * Similarly, if a service is running on a virtual machine, ensure services such as `fail2ban` are disabled. 9 | * You cannot make changes to the running application until the test is complete. This is to prevent accidentally breaking an otherwise valid attack in progress. 10 | * Any review results are not considered as "final". A security review should always be performed by a security team orchestrated by the customer prior to moving an application into production. If a customer requires further assistance, they can engage Premier Support. 11 | 12 | ## Rules of Engagement - For those performing tests 13 | 14 | * Do not attempt to perform Denial-of-Service attacks or otherwise crash services. Heavy active scanning is tolerated (and is assumed to be somewhat of a load test) but deliberate takedowns are not permitted. 15 | * Do not interact with human beings. Phishing credentials or other such client-side attacks are off-limits. Detailing XSS and similar attacks is encouraged as a part of the test, but do not leverage these against internal users or customers. 16 | * Attack from a single point. Especially if the application is currently in the customer's hands, provide the IP address or hostname of the attacking host to avoid setting off alarms. 17 | -------------------------------------------------------------------------------- /docs/observability/pitfalls.md: -------------------------------------------------------------------------------- 1 | # Things to Watch for when Building Observable Systems 2 | 3 | ## Observability as an afterthought 4 | 5 | One of the design goals when building a system should be to enable monitoring of the system. This helps planning and thinking application availability, logging and metrics at the time of design and development. Observability also acts as a great debugging tool providing developers a bird's eye view of the system. By leaving instrumentation and logging of metrics towards the end, the development teams lose valuable insights during development. 6 | 7 | ## Metric Fatigue 8 | 9 | 1. It is recommended to collect and measure *what you need* **and** *not what you can*. Don't attempt to monitor everything. 10 | 2. If the data is not actionable, it is useless and becomes noise. On the contrary, it is sometimes very difficult to forecast every possible scenario that could go wrong. 11 | 3. There must be a balance between collecting what is needed vs. logging every single activity in the system. A general rule of thumb is to follow these principles 12 | 13 | - rules that catch incidents must be simple, relevant and reliable 14 | - any data that is collected but not aggregated or alerted on must be reviewed if it is still required. 15 | 16 | ## Context 17 | 18 | All data logged must contain rich context, which is useful for getting an overall view of the system and easy to trace back errors/failures during troubleshooting. While logging data, care must also be taken to avoid data silos. 19 | 20 | ## Personally Identifiable Information 21 | 22 | As a general rule, do not log any customer sensitive and Personal Identifiable Information (PII). Ensure any pertinent privacy regulations are followed regarding PII (Ex: GDPR etc.) 23 | Read more [here](logs-privacy.md) on how to keep sensitive data out of logs. 24 | -------------------------------------------------------------------------------- /docs/source-control/merge-strategies.md: -------------------------------------------------------------------------------- 1 | # Merge strategies 2 | 3 | Agree if you want a linear or non-linear commit history. There are pros and cons to both approaches: 4 | 5 | * Pro linear: [Avoid messy git history, use linear history](https://dev.to/bladesensei/avoid-messy-git-history-3g26) 6 | * Con linear: [Why you should stop using Git rebase](https://medium.com/@fredrikmorken/why-you-should-stop-using-git-rebase-5552bee4fed1) 7 | 8 | ## Approach for non-linear commit history 9 | 10 | Merging `topic` into `main` 11 | 12 | ```md 13 | A---B---C topic 14 | / \ 15 | D---E---F---G---H main 16 | 17 | git fetch origin 18 | git checkout main 19 | git merge topic 20 | ``` 21 | 22 | ## Two approaches to achieve a linear commit history 23 | 24 | ### Rebase topic branch before merging into main 25 | 26 | Before merging `topic` into `main`, we rebase `topic` with the `main` branch: 27 | 28 | ```bash 29 | A---B---C topic 30 | / \ 31 | D---E---F-----------G---H main 32 | 33 | git checkout main 34 | git pull 35 | git checkout topic 36 | git rebase origin/main 37 | ``` 38 | 39 | Create a PR topic --> main in Azure DevOps and approve using the squash merge option 40 | 41 | ### Rebase topic branch before squash merge into main 42 | 43 | [Squash merging](https://docs.microsoft.com/en-us/azure/devops/repos/git/merging-with-squash?view=azure-devops) is a merge option that allows you to condense the Git history of topic branches when you complete a pull request. Instead of adding each commit on `topic` to the history of `main`, a squash merge takes all the file changes and adds them to a single new commit on `main`. 44 | 45 | ```bash 46 | A---B---C topic 47 | / 48 | D---E---F-----------G---H main 49 | ``` 50 | 51 | Create a PR topic --> main in Azure DevOps and approve using the squash merge option 52 | -------------------------------------------------------------------------------- /.markdown-link-check.json: -------------------------------------------------------------------------------- 1 | { 2 | "ignorePatterns": [ 3 | {"pattern": "^./INSERT_URL_TO_ISSUE"}, 4 | {"pattern": "^./link-to-the-work-item"}, 5 | {"pattern": "^https://portal.azure.com"}, 6 | {"pattern": "^http://link-to-feature-or-story-work-item"}, 7 | {"pattern": "^http://link-to-task-work-item"}, 8 | {"pattern": "^http://link-to-story-work-item"}, 9 | {"pattern": "^http://link-to-work-item"}, 10 | {"pattern": "^https://tanzu.vmware.com/developer/guides/kubernetes/observability-prometheus-grafana-p1"}, 11 | {"pattern": "^https://www.w3.org/blog/2019/12/trace-context-enters-proposed-recommendation/"}, 12 | {"pattern": "^https://blog.cloudflare.com/cloudflare-outage/"}, 13 | {"pattern": "^https://thanos.io"}, 14 | {"pattern": "^https://gitlab.com/palisade/palisade-release"}, 15 | {"pattern": "(.*\\.)?.opentelemetry.io"}, 16 | {"pattern": "(.*\\.)?.pluralsight.com"}, 17 | {"pattern": "(.*\\.)?.github.com"}, 18 | {"pattern": "^https://marketplace.visualstudio.com"}, 19 | {"pattern": "^https://opensource.org/licenses/MIT"}, 20 | {"pattern": "^https://www.researchgate.net/publication/301839557_The_landscape_of_software_failure_cause_models"}, 21 | {"pattern": "^https://www.cmu.edu/iso/governance/guidelines/data-classification.html"}, 22 | {"pattern": "^https://machinelearningmastery.com/how-to-get-baseline-results-and-why-they-matter/"}, 23 | {"pattern": "^https://www.perfecto.io/"}, 24 | {"pattern": "^https://www.ranorex.com/free-trial/"} 25 | ], 26 | "httpHeaders": [ 27 | { 28 | "urls": ["https://docs.github.com/", 29 | "https://help.github.com/", 30 | "https://github.com/", 31 | "https://blog.cloudflare.com/"], 32 | "headers": { 33 | "Accept-Encoding": "zstd, br, gzip, deflate" 34 | } 35 | } 36 | ], 37 | "retryOn429": true, 38 | "aliveStatusCodes": [200, 206, 302, 0] 39 | } 40 | -------------------------------------------------------------------------------- /docs/continuous-integration/markdown-linting/README.md: -------------------------------------------------------------------------------- 1 | # CI Pipeline for better documentation 2 | 3 | ## Introduction 4 | 5 | Most projects start with spikes, where developers and analysts produce lots of documentation. 6 | 7 | Sometimes, these documents don't have a standard and each team member writes them accordingly with their preference. Add to that 8 | the time a reviewer will spend confirming grammar, searching for typos or non-inclusive language. 9 | 10 | This pipeline helps address that! 11 | 12 | ## The Pipeline 13 | 14 | The pipeline uses the following `npm` modules: 15 | 16 | - [markdownlint](https://github.com/DavidAnson/markdownlint): add standardization using [rules](https://github.com/DavidAnson/markdownlint#rules--aliases) 17 | - [markdown-link-check](https://github.com/tcort/markdown-link-check): check the links in the documentation and report broken 18 | ones 19 | - [write-good](https://github.com/btford/write-good): linter for English prose 20 | 21 | We have been using this pipeline for more than one year in different engagements and always received great feedback from the 22 | customers! 23 | 24 | ## How does it work 25 | 26 | To start using this pipeline: 27 | 28 | 1. Download the files from [this repository](https://github.com/squassina/doc-pipeline/tree/main/repo-root) 29 | 1. Unzip the folders and files to your repository root if the repository is empty 30 | - if it's not brand new, copy the files and make the required adjustments: 31 | - check `.azdo` so it matches your repository standard 32 | - check `package.json` so you don't overwrite one you already have in the process. Also update the file if you changed 33 | the name of the `.azdo` folder. 34 | 1. Create the pipeline in Azure DevOps or GitHub 35 | 36 | ## References 37 | 38 | [Markdown Code Reviews in the Code With Engineering Playbook](https://microsoft.github.io/code-with-engineering-playbook/code-reviews/recipes/markdown/#code-review-checklist) 39 | -------------------------------------------------------------------------------- /docs/documentation/tools/wikis.md: -------------------------------------------------------------------------------- 1 | # Wikis 2 | 3 | Use a **team project wiki** to share information with other team members. When you provision a wiki from scratch, a new **Git repository** stores your [Markdown](./languages.md#markdown) files, images, attachments, and sequence of pages. This wiki supports **collaborative editing** of its content and structure. 4 | 5 | In Azure DevOps, you have the following **options for maintaining wiki content**: 6 | 7 | - Provision a wiki for your team project. This option supports only one wiki for the team project. 8 | - Publish Markdown files defined in a Git repository to a wiki. With this option, you can maintain several versioned wikis to support your content needs. 9 | 10 | More information: 11 | 12 | - [About Wikis, READMEs, and Markdown](https://docs.microsoft.com/en-us/azure/devops/project/wiki/about-readme-wiki?view=azure-devops). 13 | - [Provisioned wikis vs. published code as a wiki](https://docs.microsoft.com/en-us/azure/devops/project/wiki/provisioned-vs-published-wiki?view=azure-devops). 14 | - [Create a Wiki for your project](https://docs.microsoft.com/en-us/azure/devops/project/wiki/wiki-create-repo?view=azure-devops&tabs=browser). 15 | - [Manage wikis](https://docs.microsoft.com/en-us/azure/devops/project/wiki/manage-wikis?view=azure-devops). 16 | 17 | ## Wikis vs. digital notebooks (e.g., OneNote) 18 | 19 | When you work on a project, you may decide to document relevant details or record important decisions about the project in a digital notebook. Tools like [OneNote](https://www.microsoft.com/en-us/microsoft-365/onenote/digital-note-taking-app) allows you to easily organize, navigate and search your notes. You can provide type, highlighting, or ink annotations to your notes. These notes can easily be shared and created together with others. Still, Wikis greatly facilitate the process of [establishing and managing documentation](../best-practices/establish-and-manage.md) by allowing us to source control the documentation. 20 | -------------------------------------------------------------------------------- /docs/observability/best-practices.md: -------------------------------------------------------------------------------- 1 | # Recommended Practices 2 | 3 | 1. **Correlation Id**: Include unique identifier at the start of the interaction to tie down aggregated data from various system components and provide a holistic view. Read more guidelines about using [correlation id](correlation-id.md). 4 | 1. Ensure health of the services are **monitored** and provide insights into system's performance and behavior. 5 | 1. Ensure **dependent services** are monitored properly. Errors and exceptions in dependent services like Redis cache, Service bus, etc. should be logged and alerted. Also, metrics related to dependent services should be captured and logged. 6 | 7 | - Additionally, failures in **dependent services** should be propagated up each level of the stack by the health check. 8 | 9 | 1. **Faults, crashes, and failures** are logged as discrete events. This helps engineers identify problem area(s) during failures. 10 | 1. Ensure logging configuration (eg: setting logging to "verbose") can be controlled without code changes. 11 | 1. Ensure that **metrics** around latency and duration are collected and can be aggregated. 12 | 1. Start small and add where there is customer impact. [Avoiding metric fatigue](pitfalls.md#metric-fatigue) is very crucial to collecting actionable data. 13 | 1. It is important that every data that is collected contains relevant and rich context. 14 | 1. Personally Identifiable Information or any other customer sensitive information should never be logged. Special attention should be paid to any local privacy data regulations and collected data must adhere to those. (ex: GDPR) 15 | 1. **Health checks** : Appropriate health checks should added to determine if service is healthy and ready to serve traffic. On a kubernetes platform different types of probes e.g. Liveness, Readiness, Startup etc. can be used to determine health and readiness of the deployed service. 16 | 17 | Read more [here](pitfalls.md) to understand what to watch out for while designing and building an observable system. 18 | -------------------------------------------------------------------------------- /docs/developer-experience/code-completion.md: -------------------------------------------------------------------------------- 1 | # Code Completion 2 | 3 | Code completion is a feature used in live programming that actively reads the user's current code in context to present code suggestions in real-time. The code suggestions may be intended to complete the code for efficiency or improve the code quality. 4 | 5 | ## Code Completion Tools 6 | 7 | ### 1. GitHub Co-Pilot: A Code Completion Tool 8 | 9 | ![GitHub Code Pilot](./images/github-copilot.png) 10 | 11 | Benefits: 12 | 13 | - Autocompletion of verbose boilerplate code 14 | - For example, constructors, getters, setters, and other common code 15 | - Writing mocks for unit tests 16 | - Auto-generated code suggestions based on: 17 | - function names 18 | - descriptive comments 19 | - existing code 20 | - Easy to get started, and works automatically 21 | - Works with many common languages including Python, JavaScript, TypeScript, Ruby, Go, C#, or C++. 22 | 23 | Downsides: 24 | 25 | - Experience is limited in offline scenarios. 26 | 27 | Usage: 28 | 29 | Install the VS Code Extension, and log in to your Microsoft GitHub Account. 30 | 31 | Sign Up: 32 | 33 | [GitHub-CoPilot - Home Page](https://aka.ms/github/copilot) 34 | 35 | Docs: 36 | 37 | [GitHub-CoPilot - GitHub - Docs](https://docs.github.com/en/copilot) 38 | 39 | ### 2. VS Code's Intellisense 40 | 41 | ![Visual Studio Code - Intellisense](./images/VSCode.png) 42 | 43 | Benefits: 44 | 45 | - Native feature of VS Code 46 | - Works with many common languages out of the box (JavaScript, TypeScript, JSON, HTML, CSS, SCSS, and Less) 47 | - Has a marketplace for additional code completion extensions needed for other languages like Python, C/C++, C#, Java, Go, plus more. 48 | - Works offline 49 | 50 | Downsides: 51 | 52 | - Can conflict with GitHub CoPilot if both are enabled. 53 | 54 | Usage: 55 | 56 | Install the VS Code. Install additional extensions for additional language support. 57 | 58 | Docs: 59 | 60 | [VSCode Intellisense - Docs](https://code.visualstudio.com/docs/editor/intellisense) 61 | -------------------------------------------------------------------------------- /docs/automated-testing/templates/test_type_template.md: -------------------------------------------------------------------------------- 1 | # Insert Test Technique Name Here 2 | 3 | Put a 2-3 sentence overview about the test technique here. 4 | 5 | ## When To Use 6 | 7 | ### Problem Addressed 8 | 9 | Describing the problem that this test type addresses, this should focus on the motivation behind the test type/technique to help the reader correlate this technique to their problem. 10 | 11 | ### When to Avoid 12 | 13 | Describe when NOT to use, if applicable. 14 | 15 | ### ROI Tipping Point 16 | 17 | How much is enough? For example, some opine that unit test ROI drops significantly at 80% block coverage and when the codebase is well-exercised by real traffic in production. 18 | 19 | ### Applicable to 20 | 21 | - [ ] Local dev 'desktop' 22 | - [ ] Build pipelines 23 | - [ ] Non-production deployments 24 | - [ ] Production deployments 25 | 26 | ## NOTE: If there is great (clear, succinct) documentation for the technique on the web, supply a pointer and skip the rest of this template. No need to re-type content 27 | 28 | ## How to Use 29 | 30 | ### Architecture 31 | 32 | Describe the components of the technique and how they interact with each other and the subject of the test technique. Add a simple diagram of how the technique's parts are organized, if helpful to illustrate. 33 | 34 | ### Pre-requisites 35 | 36 | Anything required in advance? 37 | 38 | ### High-level Step-by-step 39 | 40 | 1. 41 | 1. 42 | 1. 43 | 44 | ## Best Practices and Advice 45 | 46 | Describe what good testing looks like for this technique, best practices, pitfalls. 47 | 48 | ### Anti patterns 49 | 50 | e.g. unit tests should never require off-box or even out-of-process dependencies. Are there similar things to avoid when applying this technique? 51 | 52 | ## Frameworks, Tools, Templates 53 | 54 | Describe known good (i.e. actually used and known to provide good results) frameworks, tools, templates, their pros and cons, with links. 55 | 56 | ## Resources 57 | 58 | Provide links to further readings about this technique to dive deeper. 59 | -------------------------------------------------------------------------------- /docs/security/threat-modelling.md: -------------------------------------------------------------------------------- 1 | # Threat Modeling 2 | 3 | Threat modeling is an effective way to help secure your systems, applications, networks, and services. It's a systematic approach that identifies potential threats and recommendations to help reduce risk and meet security objectives earlier in the development lifecycle. 4 | 5 | ## Threat Modeling Phases 6 | 7 | 1. *Diagram* 8 | Capture all requirements for your system and create a data-flow diagram 9 | 2. *Identify* 10 | Apply a threat-modeling framework to the data-flow diagram and find potential security issues. Here we can use [STRIDE framework](https://learn.microsoft.com/en-us/training/modules/tm-use-a-framework-to-identify-threats-and-find-ways-to-reduce-or-eliminate-risk/1b-threat-modeling-framework) to identify the threats. 11 | 3. *Mitigate* 12 | Decide how to approach each issue with the appropriate combination of security controls. 13 | 4. *Validate* 14 | Verify requirements are met, issues are found, and security controls are implemented. 15 | 16 | Example of these phases is covered in the [threat modelling example.](./threat-modelling-example.md) 17 | More details about these phases can be found at [Threat Modeling Security Fundamentals.](https://learn.microsoft.com/en-us/training/paths/tm-threat-modeling-fundamentals/) 18 | 19 | ## Threat Modeling Example 20 | 21 | [Here is an example](./threat-modelling-example.md) of a threat modeling document which talks about the architecture and different phases involved in the threat modeling. This document can be used as reference template for creating threat modeling documents. 22 | 23 | ## References 24 | 25 | * [Threat Modeling](https://www.microsoft.com/en-us/securityengineering/sdl/threatmodeling) 26 | * [Microsoft Threat Modeling Tool](https://learn.microsoft.com/en-us/azure/security/develop/threat-modeling-tool) 27 | * [STRIDE (Threat modeling framework)](https://learn.microsoft.com/en-us/training/modules/tm-use-a-framework-to-identify-threats-and-find-ways-to-reduce-or-eliminate-risk/1b-threat-modeling-framework) 28 | -------------------------------------------------------------------------------- /docs/automated-testing/e2e-testing/testing-methods.md: -------------------------------------------------------------------------------- 1 | # E2E Testing Methods 2 | 3 | ## Horizontal Test 4 | 5 | This method is used very commonly. It occurs horizontally across the context of multiple applications. Take an example of a data ingest management system. 6 | 7 | ![Horizontal Test](./images/horizontal-e2e-testing.png) 8 | 9 | The inbound data may be injected from various sources, but it then "flatten" into a horizontal processing pipeline that may include various components, such as a gateway API, data transformation, data validation, storage, etc... Throughout the entire Extract-Transform-Load (ETL) processing, the data flow can be tracked and monitored under the horizontal spectrum with little sprinkles of optional, and thus not important for the overall E2E test case, services, like logging, auditing, authentication. 10 | 11 | ## Vertical Test 12 | 13 | In this method, all most critical transactions of any application are verified and evaluated right from the start to finish. Each individual layer of the application is tested starting from top to bottom. Take an example of a web-based application that uses middleware services for reaching back-end resources. 14 | 15 | ![Vertical Test](./images/vertical-e2e-testing.png) 16 | 17 | In such case, each layer (tier) is required to be fully tested in conjunction with the "connected" layers above and beneath, in which services "talk" to each other during the end to end data flow. All these complex testing scenarios will require proper validation and dedicated automated testing. Thus, this method is much more difficult. 18 | 19 | ## E2E Test Cases Design Guidelines 20 | 21 | Below enlisted are few **guidelines** that should be kept in mind while designing the test cases for performing E2E testing: 22 | 23 | - Test cases should be designed from the end user’s perspective. 24 | - Should focus on testing some existing features of the system. 25 | - Multiple scenarios should be considered for creating multiple test cases. 26 | - Different sets of test cases should be created to focus on multiple scenarios of the system. 27 | -------------------------------------------------------------------------------- /docs/agile-development/advanced-topics/backlog-management/minimal-slices.md: -------------------------------------------------------------------------------- 1 | # Minimalism Slices 2 | 3 | ## Always deliver your work using minimal valuable slices 4 | 5 | - Split your work item into small chunks that are contributed in incremental commits. 6 | 7 | - Contribute your chunks frequently. Follow an iterative approach by regularly providing updates and changes to the team. This allows for instant feedback and early issue discovery and ensures you are developing in the right direction, both technically and functionally. 8 | 9 | - Do NOT work independently on your task without providing any updates to your team. 10 | 11 | ### Example 12 | 13 | Imagine you are working on adding UWP (Universal Windows Platform) application building functionality for existing continuous integration service which already has Android/iOS support. 14 | 15 | #### Bad approach 16 | 17 | After six weeks of work you created PR with all required functionality, including portal UI (build settings), backend REST API (UWP build functionality), telemetry, unit and integration tests, etc. 18 | 19 | #### Good approach 20 | 21 | You divided your feature into smaller user stories (which in turn were divided into multiple tasks) and started working on them one by one: 22 | 23 | - As a user I can successfully build UWP apps using current service 24 | - As a user I can see telemetry when building the apps 25 | - As a user I have the ability to select build configuration (debug, release) 26 | - As a user I have the ability to select target platform (arm, x86, x64) 27 | - ... 28 | 29 | You also divided your stories into smaller tasks and sent PRs based on those tasks. 30 | E.g. you have the following tasks for the first user story above: 31 | 32 | - Enable UWP platform on backend 33 | - Add `build` button to the UI (build first solution file found) 34 | - Add `select solution file` dropdown to the UI 35 | - Implement unit tests 36 | - Implement integration tests to verify build succeeded 37 | - Update documentation 38 | - ... 39 | 40 | ### Resources 41 | 42 | - [Minimalism Rules](http://minifesto.org/) 43 | -------------------------------------------------------------------------------- /docs/observability/tools/README.md: -------------------------------------------------------------------------------- 1 | # Tools and Patterns 2 | 3 | There are a number of modern tools to make systems observable. While identifying and/or creating tools that work for your system, here are a few things to consider to help guide the choices. 4 | 5 | - Must be simple to integrate and easy to use. 6 | - It must be possible to aggregate and visualize data. 7 | - Tools must provide real-time data. 8 | - Must be able to guide users to the problem area with suitable, adequate end-to-end context. 9 | 10 | ## Choices 11 | 12 | - [Loki](./loki.md) 13 | - [OpenTelemetry](./OpenTelemetry.md) 14 | - [Kubernetes Dashboards](./KubernetesDashboards.md) 15 | - [Prometheus](./Prometheus.md) 16 | 17 | ## Service Mesh 18 | 19 | Leveraging a Service Mesh that follows the [Sidecar Pattern](https://www.oreilly.com/library/view/designing-distributed-systems/9781491983638/ch02.html#:~:text=The%20sidecar%20pattern%20is%20a,first%20is%20the%20application%20container.&text=In%20addition%20to%20the%20application,without%20the%20application%20container's%20knowledge.) quickly sets up a go-to set of metrics, and traces (although traces need to be propagated from incoming requests to outgoing requests manually). 20 | 21 | A sidecar works by intercepting all incoming and outgoing traffic to your image. It then adds trace headers to each request and emits a standard set of logs and metrics. These metrics are extremely powerful for observability, allowing every service, whether client-side or server-side, to leverage a unified set of metrics, including: 22 | 23 | - Latency 24 | - Bytes 25 | - Request Rate 26 | - Error Rate 27 | 28 | In a microservice architecture, pinpointing the root cause of a spike in 500's can be non-trivial, but with the added observability from a sidecar you can quickly determine which service in your service mesh resulted in the spike in errors. 29 | 30 | Service Mesh's have a large surface area for configuration, and can seem like a daunting undertaking to deploy. However, most services (including Linkerd) offer a sane set of defaults, and can be deployed via the happy path to quickly land these observability wins. 31 | -------------------------------------------------------------------------------- /docs/observability/tools/loki.md: -------------------------------------------------------------------------------- 1 | # Loki 2 | 3 | Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system, created by Grafana 4 | Labs inspired by the learnings from Prometheus. Loki is commonly referred as 'Prometheus, but for logs', which 5 | makes total sense. Both tools follow the same architecture, which is an agent collecting metrics in each 6 | of the components of the software system, a server which stores the logs and also the Grafana dashboard, which 7 | access the loki server to build its visualizations and queries. That being said, Loki has three main 8 | components: 9 | 10 | ## Promtail 11 | 12 | It is the agent portion of Loki. It can be used to grab logs from several places, like var/log/ for 13 | example. The configuration of the Promtail is a yaml file called ```config-promtail.yml```. In this file, its described all the paths and log sources that will be 14 | aggregated on Loki Server. 15 | 16 | ## Loki Server 17 | 18 | Loki Server is responsible for receiving and storing all the logs received from all the different systems. The Loki Server is also 19 | responsible for the queries done on Grafana, for example. 20 | 21 | ## Grafana Dashboards 22 | 23 | Grafana Dashboards are responsible for creating the visualizations and performing queries. After all, it will 24 | be a web page that people with the right access can log into to see, query and create alerts for the aggregated 25 | logs. 26 | 27 | ## Why use Loki 28 | 29 | The main reason to use Loki instead of other log aggregation tools, is that Loki optimizes the necessary 30 | storage. It does that by following the same pattern as prometheus, which index the labels and make chunks 31 | of the log itself, using less space than just storing the raw logs. 32 | 33 | ## References 34 | 35 | - [Loki Official Site](https://grafana.com/oss/loki/) 36 | - [Inserting logs into Loki](https://grafana.com/docs/loki/latest/getting-started/get-logs-into-loki/) 37 | - [Adding Loki Source to Grafana](https://grafana.com/docs/grafana/latest/datasources/loki/#adding-the-data-source) 38 | - [Loki Best Practices](https://grafana.com/docs/loki/latest/best-practices/) 39 | -------------------------------------------------------------------------------- /docs/documentation/tools/automation.md: -------------------------------------------------------------------------------- 1 | # How to Automate Simple Checks 2 | 3 | If you want to automate some checks on your Markdown documents, there are several tools that you could leverage. For example: 4 | 5 | - [Code Analysis / Linting](../../code-reviews/recipes/markdown.md#code-analysis-linting) 6 | - [markdownlint](../../code-reviews/recipes/markdown.md#markdownlint) to verify Markdown syntax and enforce rules that make the text more readable. 7 | - [markdown-link-check](https://github.com/tcort/markdown-link-check) to extract links from markdown texts and check whether each link is alive (200 OK) or dead. 8 | - [proselint](../../code-reviews/recipes/markdown.md#proselint) to check for jargon, spelling errors, redundancy, corporate speak and other language related issues. 9 | - [write-good](../../code-reviews/recipes/markdown.md#write-good) to check English prose. 10 | - [Docker image for node-markdown-spellcheck](https://github.com/tmaier/docker-markdown-spellcheck), a lightweight docker image to spellcheck markdown files. 11 | 12 | - [VS Code Extensions](../../code-reviews/recipes/markdown.md#vs-code-extensions) 13 | - [Write Good Linter](../../code-reviews/recipes/markdown.md#write-good-linter) to get grammar and language advice while editing a document. 14 | - [markdownlint](../../code-reviews/recipes/markdown.md#markdownlint-extension) to examine Markdown documents and get warnings for rule violations while editing. 15 | 16 | - Automation 17 | - [pre-commit](https://pre-commit.com/) to use Git hook scripts to identify simple issues before submitting our code or documentation for review. 18 | - Check [Build validation](../../code-reviews/recipes/markdown.md#build-validation) to automate linting for PRs. 19 | - Check [CI Pipeline for better documentation](../../continuous-integration/markdown-linting/README.md) for a sample pipeline with `markdownlint`, `markdown-link-check` and `write-good`. 20 | 21 | Sample output: 22 | 23 | ![docs-checks](./images/docs-checks.png) 24 | 25 | ## On linting rules 26 | 27 | The team needs to be clear what linting rules are required and shouldn't be overridden with tooling or comments. The team should have consensus on when to override tooling rules. 28 | -------------------------------------------------------------------------------- /docs/developer-experience/recipes/onboarding-guide-template.md: -------------------------------------------------------------------------------- 1 | # Onboarding Guide Template 2 | 3 | When developing an onboarding document for a team, it should contain details of engagement scope, team processes, codebase, coding standards, team agreements, software requirements and setup details. The onboarding guide can be used as an index to project specific content if it already exists elsewhere. Allowing this guide to be utilized as a foundation with the links will help keep the guide concise and effective. 4 | 5 | ## Overview and Goals 6 | 7 | * List a few sentences explaining the high-level summary and the scope of the engagement. 8 | * Consider adding any additional background and context as needed. 9 | * Include the value proposition of the project, goals, what success looks like, and what the team is trying to achieve and why. 10 | 11 | ## Contacts 12 | 13 | * List a few of the main contacts for the team and project overall such as the Dev Lead and Product Owner. 14 | * Consider including the roles of these main contacts so that the team knows who to reach out to depending on the situation. 15 | 16 | ## Team Agreement and Code of Conduct 17 | 18 | * Include the team's code of conduct or agreement that defines a set of expectation from each team member and how the team has agreed to operate. 19 | * Working Agreement Template - [working agreement](../../agile-development/advanced-topics/team-agreements/working-agreements.md) 20 | 21 | ## Dev Environment Setup 22 | 23 | * Consider adding steps to run the project end-to-end. This could be in form of a separate wiki page or document that can be linked here. 24 | * Include any software that needs to be downloaded and specify if a specific version of the software is needed. 25 | 26 | ## Project Building Blocks 27 | 28 | * This can include a more in depth description with different areas of the project to help increase the project understanding. 29 | * It can include different sections on the various components of the project including deployment, e2e testing, repositories. 30 | 31 | ## Helpful Resources and Links 32 | 33 | * This can include any additional links to documents related to the project 34 | * It may include links to backlog items, work items, wiki pages or project history. 35 | -------------------------------------------------------------------------------- /docs/observability/profiling.md: -------------------------------------------------------------------------------- 1 | # Profiling 2 | 3 | ## Overview 4 | 5 | Profiling is a form of runtime analysis that measures various components of the runtime such as, memory allocation, garbage collection, threads and locks, call stacks, or frequency and duration of specific functions. It can be used to see which functions are the most costly in your binary, allowing you to focus your effort on removing the largest inefficiencies as quickly as possible. It can help you find deadlocks, memory leaks, or inefficient memory allocation, and help inform decisions around resource allocation (ie: CPU or RAM). 6 | 7 | ## How to Profile your Applications 8 | 9 | Profiling is somewhat language dependent, so start off by searching for "profile $language" (some common tools are listed below). Additionally, Linux Perf is a good fallback, since a lot of languages have bindings in C/C++. 10 | 11 | Profiling does incur some cost, as it requires inspecting the call stack, and sometimes pausing the application all together (ie: to trigger a full GC in Java). It is recommended to continuously profile your services, say for 10s every 10 minutes. Consider the cost when deciding on tuning these parameters. 12 | 13 | Different tools visualize profiles differently. Common CPU profiles might use a directed graph ![graph](images/pprof-dot.png) or a flame graph. ![flame](images/flame.png) 14 | 15 | Unfortunately, each profiler tool typically uses its own format for storing profiles, and comes with its own visualization. 16 | 17 | ## Specific tools 18 | 19 | - (Java, Go, Python, Ruby, eBPF) [Pyroscope](https://github.com/pyroscope-io/pyroscope) continuous profiling out of the box. 20 | - (Java and Go) [Flame](https://github.com/VerizonMedia/kubectl-flame) - profiling containers in Kubernetes 21 | - (Java, Python, Go) [Datadog Continuous profiler](https://www.datadoghq.com/product/code-profiling/) 22 | - (Java, Python, Go, Node.js) [Instana](https://www.instana.com/blog/instana-announces-the-industrys-first-commercial-continuous-production-profiler/) 23 | - (Go) [profefe](https://github.com/profefe/profefe), which builds `pprof` to provide continuous profiling 24 | - (Java) [Opsian](https://opsian.com/) 25 | - (Java) [Eclipse Memory Analyzer](https://www.eclipse.org/mat/) 26 | -------------------------------------------------------------------------------- /docs/continuous-delivery/deploying-with-gitops/README.md: -------------------------------------------------------------------------------- 1 | # Deploying with GitOps 2 | 3 | ## What is GitOps? 4 | 5 | GitOps is a way of managing your infrastructure and applications so that the whole system is described declaratively and version controlled (most likely in a Git repository), and having an automated process that ensures that the deployed environment matches the state specified in a repository."- [WeaveWorks](https://www.weave.works/technologies/gitops/#what-is-gitops) 6 | 7 | ## Why should I use GitOps? 8 | 9 | GitOps simply allows faster deployments by having git repositories in the center offering a clear audit trail via git commits and no direct environment access. Read more on [Why should I use GitOps?](https://www.gitops.tech/#why-should-i-use-gitops) 10 | 11 | The below diagram compares traditional CI/CD vs GitOps workflow: 12 | ![push based vs pull based deployments](images/GitopsWorflowVsTraditionalPush.jpg) 13 | 14 | ## Tools for GitOps 15 | 16 | Some popular GitOps frameworks for Kubernetes backed by [CNCF](https://landscape.cncf.io/card-mode?category=continuous-integration-delivery) community: 17 | 18 | - [Flux V2](https://fluxcd.io/docs/get-started/) 19 | - [Argo CD](https://argo-cd.readthedocs.io/en/stable/) 20 | - [Rancher Fleet](https://fleet.rancher.io/) 21 | 22 | ## Deploying using GitOps 23 | 24 | GitOps with Flux v2 can be enabled in Azure Kubernetes Service (AKS) managed clusters or Azure Arc-enabled Kubernetes connected clusters as a cluster extension. After the microsoft.flux cluster extension is installed, you can create one or more fluxConfigurations resources that sync your Git repository sources to the cluster and reconcile the cluster to the desired state. With GitOps, you can use your Git repository as the source of truth for cluster configuration and application deployment. 25 | 26 | - [Tutorial: Deploy configurations using GitOps on an Azure Arc-enabled Kubernetes cluster](https://docs.microsoft.com/en-us/azure/azure-arc/kubernetes/tutorial-use-gitops-connected-cluster) 27 | - [Tutorial: Implement CI/CD with GitOps](https://docs.microsoft.com/en-us/azure/azure-arc/kubernetes/tutorial-gitops-flux2-ci-cd) 28 | - [Multi-cluster and multi-tenant environment with Flux v2](https://github.com/microsoft/multicluster-gitops) 29 | -------------------------------------------------------------------------------- /docs/documentation/recipes/sync-wiki-between-repos.md: -------------------------------------------------------------------------------- 1 | # How to Sync a Wiki between Repositories 2 | 3 | This is a quick guide to mirroring a Project Wiki to another repository. 4 | 5 | ```bash 6 | # Clone the wiki 7 | git clone <source wiki repo url> 8 | 9 | # Add mirror repository as a remote 10 | cd <source wiki repo working folder> 11 | git remote add mirror <mirror repo that must already exist> 12 | ``` 13 | 14 | Now each time you wish to sync run the following to get latest from the source wiki repo: 15 | 16 | ```bash 17 | # Get everything 18 | git pull -v 19 | ``` 20 | 21 | > **Warning**: Check that the output of the pull shows "From source repo URL". If this shows the mirror repo url then you've forgotten to reset the tracking. Run `git branch -u origin/wikiMaster` then continue. 22 | 23 | Then run this to push it to the mirror repo and reset the branch to track the source repo again: 24 | 25 | ```bash 26 | # Push all branches up to mirror remote 27 | git push -u mirror 28 | 29 | # Reset local to track source remote 30 | git branch -u origin/wikiMaster 31 | 32 | ``` 33 | 34 | Your output should look like this when run: 35 | 36 | ```powershell 37 | PS C:\Git\MyProject.wiki> git pull -v 38 | POST git-upload-pack (909 bytes) 39 | remote: Azure Repos 40 | remote: Found 5 objects to send. (0 ms) 41 | Unpacking objects: 100% (5/5), done. 42 | From https://..... wikiMaster -> origin/wikiMaster 43 | Updating 7412b94..a0f543b 44 | Fast-forward 45 | .../dffffds.md | 4 ++++ 46 | 1 file changed, 4 insertions(+) 47 | 48 | 49 | PS C:\Git\MyProject.wiki> git push -u mirror 50 | Enumerating objects: 9, done. 51 | Counting objects: 100% (9/9), done. 52 | Delta compression using up to 8 threads 53 | Compressing objects: 100% (5/5), done. 54 | Writing objects: 100% (5/5), 2.08 KiB | 2.08 MiB/s, done. 55 | Total 5 (delta 4), reused 0 (delta 0) 56 | remote: Analyzing objects... (5/5) (6 ms) 57 | remote: Storing packfile... done (48 ms) 58 | remote: Storing index... done (59 ms) 59 | To https://...... 60 | 7412b94..a0f543b wikiMaster -> wikiMaster 61 | Branch 'wikiMaster' set up to track remote branch 'wikiMaster' from 'mirror'. 62 | 63 | 64 | PS C:\Git\MyProject.wiki> git branch -u origin/wikiMaster 65 | Branch 'wikiMaster' set up to track remote branch 'wikiMaster' from 'origin'. 66 | ``` 67 | -------------------------------------------------------------------------------- /docs/documentation/tools/languages.md: -------------------------------------------------------------------------------- 1 | # Languages 2 | 3 | ## Markdown 4 | 5 | Markdown is one of the most popular markup languages to add rich formatting, tables and images to your documentation using plain text documents. 6 | 7 | Markdown files (.md) can be source-controlled along with your code. 8 | 9 | More information: 10 | 11 | - [Getting Started](https://www.markdownguide.org/getting-started/) 12 | - [Cheat Sheet](https://www.markdownguide.org/cheat-sheet/) 13 | - [Basic Syntax](https://www.markdownguide.org/basic-syntax/) 14 | - [Extended Syntax](https://www.markdownguide.org/extended-syntax/) 15 | - [Wiki Markdown Syntax](https://docs.microsoft.com/en-us/azure/devops/project/wiki/wiki-markdown-guidance?view=azure-devops) 16 | 17 | Tools: 18 | 19 | - [Markdown and Visual Studio Code](https://code.visualstudio.com/docs/languages/markdown) 20 | - [How to automate simple checks](./automation.md) 21 | 22 | ## Mermaid 23 | 24 | Mermaid lets you create diagrams using text definitions that can later be rendered with a diagramming and charting tool. 25 | 26 | Mermaid files (.mmd) can be source-controlled along with your code. It's also recommended to include image files (.png) with the rendered diagrams under source control. Your markdown files should link the image files, so they can be read without the need of a Mermaid rendering tool (e.g., during Pull Request review). 27 | 28 | ### Example Mermaid diagram 29 | 30 | This is an example of a Mermaid flowchart diagram written as code. 31 | 32 | ```mermaid 33 | graph LR 34 | A[Diagram Idea] -->|Write mermaid code| B(mermaid.mmd file) 35 | B -->|Add to source control| C{Code repo} 36 | B -->|Export as .png| G(.png file of diagram) 37 | G -->|Add to source control| C 38 | ``` 39 | 40 | This is an example of how it can be rendered as an image. 41 | 42 | ![Example mermaid diagram](images/example-mermaid-diagram.png "Example mermaid diagram") 43 | 44 | More information: 45 | 46 | - [About Mermaid](https://mermaid-js.github.io/mermaid/#/) 47 | - [Diagram syntax](https://mermaid-js.github.io/mermaid/#/./n00b-syntaxReference) 48 | 49 | Tools: 50 | 51 | - [Mermaid Live Editor](https://mermaid.live) 52 | - [Markdown Preview Mermaid Support for Visual Studio Code](https://marketplace.visualstudio.com/items?itemName=bierner.markdown-mermaid) 53 | -------------------------------------------------------------------------------- /docs/design/design-reviews/recipes/milestone-epic-design-review-recipe.md: -------------------------------------------------------------------------------- 1 | # Milestone / Epic Design Review Recipe 2 | 3 | ## Why is this valuable? 4 | 5 | Design at epic/milestone level can help the team make better decisions about prioritization by summarizing the value, effort, complexity, risks, and dependencies. This brief document can help the team align on the selected approach and briefly explain the rationale for other teams, subject-matter experts, project advisors, and new team members. 6 | 7 | ## Things to keep in mind 8 | 9 | * As with all other aspects of the project, design reviews must provide a friendly and safe environment so that any team member feels comfortable proposing a design for review and can use the opportunity to grow and learn from the constructive / non-judgemental feedback from peers and subject-matter experts (see [Team Agreements](../../../agile-development/advanced-topics/team-agreements/README.md)). 10 | * Design reviews should be lightweight and should not feel like an additional process overhead. 11 | * Dev Lead can usually provide guidance on whether a given epic/milestone needs a design review and can help other team members in preparation. 12 | * This is *not* a strict template that must be followed and teams should not be bogged down with polished "design presentations". 13 | * Think of the recipe below as a "menu of options" for potential questions to think through in designing this epic. Not all sections are required for every epic. Focus on sections and questions that are most relevant for making the decision and rationalizing the trade-offs. 14 | * Milestone/epic design is considered high-level design but is usually more detailed than the design included in the Game Plan, but will likely re-use some technologies, non-functional requirements, and constraints mentioned in the Game Plan. 15 | * As the team learned more about the project and further refined the scope of the epic, they may specifically call out notable changes to the overall approach and, in particular, highlight any unique deployment, security, private, scalability, etc. characteristics of this milestone. 16 | 17 | ## Template 18 | 19 | You can download the **[Milestone/Epic Design Review Template](./milestone-epic-design-review-template.md)**, copy it into your project, and use it as described in the [async design review recipe](./async-design-reviews.md). 20 | -------------------------------------------------------------------------------- /docs/automated-testing/e2e-testing/testing-comparison.md: -------------------------------------------------------------------------------- 1 | # Unit vs Integration vs System vs E2E Testing 2 | 3 | The table below illustrates the most critical characteristics and differences among Unit, Integration, System, and End-to-End Testing, and when to apply each methodology in a project. 4 | 5 | | | Unit Test | Integration Test | System Testing | E2E Test | 6 | |-----------------------|------------------------|----------------------------------------------|-----------------------------------------------------------|---------------------------------------------------------------| 7 | | **Scope** | Modules, APIs | Modules, interfaces | Application, system | All sub-systems, network dependencies, services and databases | 8 | | **Size** | Tiny | Small to medium | Large | X-Large | 9 | | **Environment** | Development | Integration test | QA test | Production like | 10 | | **Data** | Mock data | Test data | Test data | Copy of real production data | 11 | | **System Under Test** | Isolated unit test | Interfaces and flow data between the modules | Particular system as a whole | Application flow from start to end | 12 | | **Scenarios** | Developer perspectives | Developers and IT Pro tester perspectives | Developer and QA tester perspectives | End-user perspectives | 13 | | **When** | After each build | After Unit testing | Before E2E testing and after Unit and Integration testing | After System testing | 14 | **Automated or Manual** | Automated | Manual or automated | Manual or automated | Manual | 15 | -------------------------------------------------------------------------------- /docs/design/design-reviews/recipes/high-level-design-recipe.md: -------------------------------------------------------------------------------- 1 | # High Level / Game Plan Design Recipe 2 | 3 | ## Why is this valuable? 4 | 5 | Design at macroscopic level shows the interactions between systems and services that will be used to accomplish the project. It is intended to ensure there is high level understanding of the plan for what to build, which off-the-shelf components will be used, and which external components will need to interact with the deliverable. 6 | 7 | ## Things to keep in mind 8 | 9 | * As with all other aspects of the project, design reviews must provide a friendly and safe environment so that any team member feels comfortable proposing a design for review and can use the opportunity to grow and learn from the constructive / non-judgemental feedback from peers and subject-matter experts (see [Team Agreements](../../../agile-development/advanced-topics/team-agreements/README.md)). 10 | * Attempt to illustrate different personas involved in the use cases and how/which boxes are their entry points. 11 | * Prefer pictures over paragraphs. The diagrams aren't intended to generate code, so they should be fairly high level. 12 | * Artifacts should indicate the direction of calls (are they outbound, inbound, or bidirectional?) and call out system boundaries where ports might need to be opened or additional infrastructure work may be needed to allow calls to be made. 13 | * Sequence diagrams are helpful to show the flow of calls among components + systems. 14 | * Generic box diagrams depicting data flow or call origination/destination are useful. However, the title should clearly define what the arrows show indicate. In most cases, a diagram will show either data flow or call directions but not both. 15 | * Visualize the contrasting aspects of the system/diagram for ease of communication. e.g. differing technologies employed, modified vs. untouched components, or internet vs. local cloud components. Colors, grouping boxes, and iconography can be used for differentiating. 16 | * Prefer ease-of-understanding for communicating ideas over strict UML correctness. 17 | * Design reviews should be lightweight and should not feel like an additional process overhead. 18 | 19 | ## Examples 20 | 21 | ![Sequence Diagram](images/high-level-sequence-diagram.png) 22 | 23 | ![Call Flow Diagram](images/high-level-box-diagram.png) 24 | 25 | ![System Diagram](images/high-level-system-diagram.png) 26 | -------------------------------------------------------------------------------- /docs/design/design-reviews/recipes/task-design-review-template.md: -------------------------------------------------------------------------------- 1 | # Your Task Design Title Here (prefix with DRAFT/WIP to indicate level of completeness) 2 | 3 | > When developing a design document for a new task, it should contain a detailed design proposal demonstrating how it will solve the goals outlined below. 4 | > Not all tasks require a design review, but when they do it is likely that there many unknowns, or the solution may be more complex. 5 | > The design should include diagrams, pseudocode, interface contracts as needed to provide a detailed understanding of the proposal. 6 | 7 | * [Task Name](http://link-to-task-work-item) 8 | * [Story Name](http://link-to-story-work-item) 9 | * Engagement: [Engagement] 10 | * Customer: [Customer] 11 | * Authors: [Author1, Author2, etc.] 12 | 13 | ## Overview/Problem Statement 14 | 15 | * It can also be a link to the [work item](http://link-to-task-work-item). 16 | * Describe the task with a high-level summary. 17 | * Consider additional background and justification, for posterity and historical context. 18 | 19 | ## Goals/In-Scope 20 | 21 | * List a few bullet points of what this task will achieve and that are most relevant for the design review discussion. 22 | * This should include acceptance criteria required to meet the [definition of done](../../../agile-development/advanced-topics/team-agreements/definition-of-done.md). 23 | 24 | ## Non-goals / Out-of-Scope 25 | 26 | * List a few bullet points of non-goals to clarify the work that is beyond the scope of the design review for this task. 27 | 28 | ## Proposed Options 29 | 30 | * Describe the detailed design to accomplish the proposed task. 31 | * What patterns & practices will be used and why were they chosen. 32 | * Were any alternate proposals considered? 33 | * What new components are required to be developed? 34 | * Are there any existing components that require updates? 35 | * Relevant diagrams (e.g. sequence, component, context, deployment) should be included here. 36 | 37 | ## Technology Choices 38 | 39 | * Describe any libraries and OSS components that will be used to complete the task. 40 | * Briefly list the languages(s) and platform(s) that comprise the stack. 41 | 42 | ## Open Questions 43 | 44 | > List any open questions/concerns here. 45 | 46 | ## Additional References 47 | 48 | > List any additional references here including links to backlog items, work items or other documents. 49 | -------------------------------------------------------------------------------- /docs/code-reviews/process-guidance/author-guidance.md: -------------------------------------------------------------------------------- 1 | # Author Guidance 2 | 3 | ## Properly describe your pull request (PR) 4 | 5 | - Give the PR a descriptive title, so that other members can easily (in one short sentence) understand what a PR is about. 6 | - Every PR should have a proper description, that shows the reviewer what has been changed and why. 7 | 8 | ## Add relevant reviewers 9 | 10 | - Add one or more reviewers (depending on your project's guidelines) to the PR. Ideally, you would add at least someone who has expertise and is familiar with the project, or the language used 11 | - Adding someone less familiar with the project or the language can aid in verifying the changes are understandable, easy to read, and increases the expertise within the team 12 | - In CSE code-with projects with a customer team, it is important to include reviewers from both organizations for knowledge transfer - [Customize Reviewers Policy](../tools.md#reviewer-policies) 13 | 14 | ## Be open to receive feedback 15 | 16 | Discuss design/code logic and address all comments as follows: 17 | 18 | - Resolve a comment, if the requested change has been made. 19 | - Mark the comment as "won't fix", if you are not going to make the requested changes and provide a clear reasoning 20 | - If the requested change is within the scope of the task, "I'll do it later" is not an acceptable reason! 21 | - If the requested change is out of scope, create a new work item (task or bug) for it 22 | - If you don't understand a comment, ask questions in the review itself as opposed to a private chat 23 | - If a thread gets bloated without a conclusion, have a meeting with the reviewer (call them or knock on door) 24 | 25 | ## Use checklists 26 | 27 | When creating a PR, it is a good idea to add a checklist of objectives of the PR in the description. This helps the reviewers to focus on the key areas of the code changes. 28 | 29 | ## Link a task to your PR 30 | 31 | Link the corresponding work items/tasks to the PR. There is no need to duplicate information between the work item and the PR, but if some details are missing in either one, together they provide more context to the reviewer. 32 | 33 | ## Code should have annotations before the review 34 | 35 | If you can't avoid large PRs, include explanations of the changes in order to make it easier for the reviewer to review the code, with clear comments the reviewer can identify the goal of every code block. 36 | -------------------------------------------------------------------------------- /docs/automated-testing/performance-testing/iterative-perf-test-template.md: -------------------------------------------------------------------------------- 1 | # Performance Test Iteration Template 2 | 3 | > This document provides template for capturing results of performance tests. Performance tests are done in iterations and each iteration should have a clear goal. The results of any iteration is immutable regardless whether the goal was achieved or not. If the iteration failed or the goal is not achieved then a new iteration of testing is carried out with appropriate fixes. It is recommended to keep track of the recorded iterations to maintain a timeline of how system evolved and which changes affected the performance in what way. Feel free to modify this template as needed. 4 | 5 | ## Iteration Template 6 | 7 | ### Goal 8 | 9 | > Mention in bullet points the goal for this iteration of test. The goal should be small and measurable within this iteration. 10 | 11 | ### Test Details 12 | 13 | - **Date**: *Date and time when this iteration started and ended* 14 | - **Duration**: *Time it took to complete this iteration.* 15 | - **Application Code**: *Commit id and link to the commit for the code(s) which are being tested in this iteration* 16 | - **Benchmarking Configuration:** 17 | - **Application Configuration:** *In bullet points mention the configuration for application that should be recorded* 18 | - **System Configuration:** *In bullet points mention the configuration of the infrastructure* 19 | 20 | > Record different types of configurations. Usually application specific configuration changes between iterations whereas system or infrastructure configurations rarely change 21 | 22 | ### Work Items 23 | 24 | > List of links to relevant work items (task, story, bug) being tested in this iteration. 25 | 26 | ### Results 27 | 28 | ```md 29 | In bullet points document the results from the test. 30 | - Attach any documents supporting the test results. 31 | - Add links to the dashboard for metrics and logs such as Application Insights. 32 | - Capture screenshots for metrics and include it in the results. Good candidate for this is CPU/Memory/Disk usage. 33 | ``` 34 | 35 | ### Observations 36 | 37 | > Observations are insights derived from test results. Keep the observations brief and as bullet points. Mention outcomes supporting the goal of the iteration. If any of the observation results in a work item (task, story, bug) then add the link to the work item together with the observation. 38 | -------------------------------------------------------------------------------- /docs/continuous-delivery/recipes/github-workflows/workflow-per-environment.md: -------------------------------------------------------------------------------- 1 | # GitHub Workflows 2 | 3 | A workflow is a configurable automated process made up of one or more jobs where each of these jobs can be an action in GitHub. Currently, a YAML file format is supported for defining a workflow in GitHub. 4 | 5 | Additional information on GitHub actions and GitHub Workflows in the links posted in the [references](#references) section below. 6 | 7 | ## Workflow Per Environment 8 | 9 | The general approach is to have one pipeline, where the code is built, tested and deployed, and the artifact is then promoted to the next environment, eventually to be deployed into production. 10 | 11 | There are multiple ways in GitHub that an environment setup can be achieved. One way it can be done is to have one workflow for multiple environments, but the complexity increases as additional processes and jobs are added to a workflow, which does not mean it cannot be done for small pipelines. The plus point of having one workflow is that, when an artifact flows from one environment to another the state and environment values between the deployment environments can be passed easily. 12 | 13 | ![Workflow-Designs-Dependent-Workflows](images/Workflow-Designs-Dependent-Workflows.png) 14 | 15 | One way to get around the complexity of a single workflow is to have separate workflows for different environments, making sure that only the artifacts created and validated are promoted from one environment to another, as well as, the workflow is small enough, to debug any issues seen in any of the workflows. In this case, the state and environment values need to be passed from one deployment environment to another. Multiple workflows also helps to keep the deployments to the environments independent thus reducing the time to deploy and find issues earlier than later in the process. Also, since the environments are independent of each other, any failures in deploying to one environment does not block deployments to other environments. One tradeoff in this method, is that with different workflows for each environment, the maintenance increases as the complexity of workflows increase over time. 16 | 17 | ![Workflow-Designs-Independent-Workflows](images/Workflow-Designs-Independent-Workflows.png) 18 | 19 | ## References 20 | 21 | - [GitHub Actions](https://docs.github.com/en/actions) 22 | - [GitHub Workflows](https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions) 23 | -------------------------------------------------------------------------------- /docs/continuous-integration/dev-sec-ops/secret-management/secrets_rotation.md: -------------------------------------------------------------------------------- 1 | # Secrets Rotation 2 | 3 | Secret rotation is the process of refreshing the secrets that are used by the application. 4 | The best way to authenticate to Azure services is by using a managed identity, but there are some scenarios where that isn't an option. In those cases, access keys or secrets are used. You should periodically rotate access keys or secrets. 5 | 6 | ## Why Secrets Rotation 7 | 8 | Secrets are an asset and as such have a potential to be leaked or stolen. By rotating the secrets, we are revoking any secrets that may have been compromised. Therefore, secrets should be rotated frequently. 9 | 10 | ## Managed Identity 11 | 12 | Azure Managed identities are automatically issues by Azure in order to indentify individual resources, and can be used for authentication in place of secrets and passwords. The appeal in using Managed Identities is the elimination of management of secrets and credentials. They are not required on developers machines or checked into source control, and they don't need to be rotated. Managed identities are considered safer than the alternatives and is the recommended choice. 13 | 14 | ## Applying Secrets Rotation 15 | 16 | If Azure Managed Identity can't be used. This and the following sections will explain how rotation of secrets can be achieved: 17 | 18 | To promote frequent rotation of a secret - define an automated periodic secret rotation process. 19 | The secret rotation process might result in a downtime when the application is restarted to introduce the new secret. A common solution for that is to have two versions of secret available, also referred to as Blue/Green Secret rotation. By having a second secret at hand, we can start a second instance of the application with that secret before the previous secret is revoked, thus avoiding any downtime. 20 | 21 | ## Secrets Rotation Frameworks and Tools 22 | 23 | 1. For rotation of a secret for resources that use one set of authentication credentials [click here](https://docs.microsoft.com/en-us/azure/key-vault/secrets/tutorial-rotation) 24 | 1. For rotation of a secret for resources that have two sets of authentication credentials [click here](https://docs.microsoft.com/en-us/azure/key-vault/secrets/tutorial-rotation-dual?tabs=azure-cli) 25 | 26 | ## Conclusion 27 | 28 | Refreshing secrets is important to ensure that your secret stays a secret without causing downtime to your application. 29 | -------------------------------------------------------------------------------- /docs/code-reviews/pull-request-template/pull-request-template.md: -------------------------------------------------------------------------------- 1 | # [Work Item ID](./link-to-the-work-item) 2 | 3 | For more information about how to contribute to this repo, visit this [page](https://github.com/microsoft/code-with-engineering-playbook/blob/main/CONTRIBUTING.md) 4 | 5 | ## Description 6 | 7 | --- 8 | 9 | > Should include a concise description of the changes (bug or feature), it's impact, along with a summary of the solution 10 | 11 | ## Steps to Reproduce Bug and Validate Solution 12 | 13 | --- 14 | > Only applicable if the work is to address a bug. Please remove this section if the work is for a feature or story 15 | > Provide details on the environment the bug is found, and detailed steps to recreate the bug. 16 | > This should be detailed enough for a team member to confirm that the bug no longer occurs 17 | 18 | ## PR Checklist 19 | 20 | --- 21 | 22 | > Use the check-list below to ensure your branch is ready for PR. If the item is not applicable, leave it blank. 23 | 24 | - [ ] I have updated the documentation accordingly. 25 | - [ ] I have added tests to cover my changes. 26 | - [ ] All new and existing tests passed. 27 | - [ ] My code follows the code style of this project. 28 | - [ ] I ran the lint checks which produced no new errors nor warnings for my changes. 29 | - [ ] I have checked to ensure there aren't other open Pull Requests for the same update/change. 30 | 31 | ## Does this introduce a breaking change? 32 | 33 | --- 34 | 35 | - [ ] Yes 36 | - [ ] No 37 | 38 | > If this introduces a breaking change, please describe the impact and migration path for existing applications below. 39 | 40 | ## Testing 41 | 42 | --- 43 | 44 | > - Instructions for testing and validation of your code: 45 | > - What OS was used for testing. 46 | > - Which test sets were used. 47 | > - Description of test scenarios that you have tried. 48 | 49 | ## Any relevant logs or outputs 50 | 51 | --- 52 | 53 | > - Use this section to attach pictures that demonstrates your changes working / healthy 54 | > - If you are printing something show a screenshot 55 | > - When you want to share long logs upload to: 56 | > `(StorageAccount)/pr-support/attachments/(PR Number)/(yourFiles) using [Azure Storage Explorer](https://azure.microsoft.com/en-us/features/storage-explorer/)` or [portal.azure.com](https://portal.azure.com) and insert the link here. 57 | 58 | ## Other information or known dependencies 59 | 60 | --- 61 | 62 | > - Any other information or known dependencies that is important to this PR. 63 | > - TODO that are to be done after this PR. 64 | -------------------------------------------------------------------------------- /docs/continuous-integration/dev-sec-ops/secret-management/recipes/detect-secrets.md: -------------------------------------------------------------------------------- 1 | # Credential Scanning Tool: detect-secrets 2 | 3 | ## Background 4 | 5 | The [`detect-secrets`](https://github.com/Yelp/detect-secrets) tool is an open source project that uses heuristics and rules to scan for [a wide range](https://github.com/Yelp/detect-secrets#currently-supported-plugins) of secrets. We can extend the tool with custom rules and heuristics via a simple [Python plugin API](https://github.com/Yelp/detect-secrets/blob/a9dff60/detect_secrets/plugins/base.py#L27-L49). 6 | 7 | Unlike other credential scanning tools, `detect-secrets` does not attempt to check a project's entire git history when invoked, but instead scans the project's current state. This means that the tool runs quickly which makes it ideal for use in continuous integration pipelines. 8 | 9 | `detect-secrets` employs the concept of a "baseline file", i.e. a list of known secrets already present in the repository, and we can configure it to ignore any of these pre-existing secrets when running. This makes it easy to gradually introduce the tool into a pre-existing project. 10 | 11 | The baseline file also provides a simple and convenient way of handling false positives. We can white-list the false positive in the baseline file to ignore it on future invocations of the tool. 12 | 13 | ## Setup 14 | 15 | ```sh 16 | # install system dependencies: diff, jq, python3 (if on Linux-based OS) 17 | apt-get install -y diffutils jq python3 python3-pip 18 | 19 | # install system dependencies: diff, jq, python3 (if on Windows) 20 | winget install Python.Python.3 21 | choco install diffutils jq -y 22 | 23 | # install the detect-secrets tool 24 | python3 -m pip install detect-secrets 25 | 26 | # run the tool to establish a list of known secrets 27 | # review this file thoroughly and check it into the repository 28 | detect-secrets scan > .secrets.baseline 29 | ``` 30 | 31 | ## Usage 32 | 33 | ```sh 34 | # backup the list of known secrets 35 | cp .secrets.baseline .secrets.new 36 | 37 | # find all the secrets in the repository 38 | detect-secrets scan --baseline .secrets.new $(find . -type f ! -name '.secrets.*' ! -path '*/.git*') 39 | 40 | # if there is any difference between the known and newly detected secrets, break the build 41 | list_secrets() { jq -r '.results | keys[] as $key | "\($key),\(.[$key] | .[] | .hashed_secret)"' "$1" | sort; } 42 | 43 | if ! diff <(list_secrets .secrets.baseline) <(list_secrets .secrets.new) >&2; then 44 | echo "Detected new secrets in the repo" >&2 45 | exit 1 46 | fi 47 | ``` 48 | -------------------------------------------------------------------------------- /docs/source-control/component-versioning.md: -------------------------------------------------------------------------------- 1 | # Component Versioning 2 | 3 | ## Goal 4 | 5 | Larger applications consist of multiple components that reference each other and rely on compatibility of the interfaces/contracts of the components. 6 | 7 | To achieve the goal of loosely coupled applications, each component should be versioned independently hence allowing developers to detect breaking changes or seamless updates just by looking at the version number. 8 | 9 | ## Version Numbers and Versioning schemes 10 | 11 | For developers or other components to detect breaking changes the version number of a component is important. 12 | 13 | There is different versioning number schemes, e.g. 14 | 15 | `major.minor[.build[.revision]]` 16 | 17 | or 18 | 19 | `major.minor[.maintenance[.build]]`. 20 | 21 | Upon build / CI these version numbers are being generated. During CD / release components are pushed to a *component repository* such as Nuget, NPM, Docker Hub where a history of different versions is being kept. 22 | 23 | Each build the version number is incremented at the last digit. 24 | 25 | Updating the major / minor version indicates changes of the API / interfaces / contracts: 26 | 27 | * Major Version: A breaking change 28 | * Minor Version: A backwards-compatible minor change 29 | * Build / Revision: No API change, just a different build. 30 | 31 | ## Semantic Versioning 32 | 33 | Semantic Versioning is a concept of calculating the version number automatically based on a certain source code repository. 34 | 35 | The `semver` tool looks at a GIT source control branch and comes up with a *repeatable* and *unique* version number based on 36 | 37 | * number of commits since last major or minor release 38 | * commit messages 39 | * tags 40 | * branch names 41 | 42 | Examples of semver version numbers: 43 | 44 | * **1.0.0-alpha.1**: +1 commit *after* the alpha release of 1.0.0 45 | * **2.1.0-beta**: 2.1.0 in beta branch 46 | * **2.4.2**: 2.4.2 release 47 | 48 | Version Updates happen through: 49 | 50 | * Commit messages or tags for Major / Minor / Revision updates. 51 | * Branch names (e.g. develop, release/..) for Alpha / Beta / RC 52 | * Otherwise: Number of commits (+12, ...) 53 | 54 | Recommendation is to run semver during your CI process to make each build uniquely identifiable. 55 | 56 | ## Resources 57 | 58 | * [Semantic Versioning](https://semver.org/) 59 | * [Versioning in C#](https://docs.microsoft.com/en-us/dotnet/csharp/versioning) 60 | * [SemVer Task for VSTS](https://marketplace.visualstudio.com/items?itemName=geeklearningio.gl-vsts-tasks-semver) 61 | -------------------------------------------------------------------------------- /docs/machine-learning/README.md: -------------------------------------------------------------------------------- 1 | # Machine Learning Fundamentals at CSE 2 | 3 | This guideline documents the Machine Learning (ML) practices in CSE. CSE works with customers on developing ML models and putting them in production, with an emphasis on engineering and research best practices throughout the project's life cycle. 4 | 5 | ## Goals 6 | 7 | * Provide a set of ML practices to follow in an ML project. 8 | * Provide clarity on ML process and how it fits within a software engineering project. 9 | * Provide best practices for the different stages of an ML project. 10 | 11 | ## How to use these fundamentals 12 | 13 | * If you are starting a new ML project, consider reading through the [general guidance documents](#general-guidance). 14 | * For specific aspects of an ML project, refer to the guidelines for different [project phases](#ml-project-phases). 15 | 16 | ## ML Project phases 17 | 18 | The diagram below shows different phases in an ideal ML project. Due to practical constraints and requirements, it might not always be possible to have a project structured in such a manner, however best practices should be followed for each individual phase. 19 | 20 | ![Project flow](images/flow.png) 21 | 22 | * **[Envisioning](ml-problem-formulation-envisioning.md)**: Initial problem understanding, customer goals and objectives. 23 | * **[Feasibility Study](ml-feasibility-study.md)**: Assess whether the problem in question is feasible to solve satisfactorily using ML with the available data. 24 | * **Model Milestone**: There is a basic model that is achieving the minimum required performance, both in terms of ML performance and system performance. Using the knowledge gathered to this milestone, define the scope, objectives, high-level architecture, definition of done and plan for the entire project. 25 | * **[Model(s) experimentation](ml-experimentation.md)**: Tools and best practices for conducting successful model experimentation. 26 | * **Model(s) Operationalization**: [Model readiness for production](ml-model-checklist.md) checklist. 27 | 28 | ## General guidance 29 | 30 | * [ML Process Guidance](ml-proposed-process.md) 31 | * [ML Fundamentals checklist](ml-fundamentals-checklist.md) 32 | * [Data Exploration](ml-data-exploration.md) 33 | * [Agile ML development](ml-project-management.md) 34 | * [Testing Data Science and ML Ops code](ml-testing.md) 35 | * [Profiling Machine Learning and ML Ops code](ml-profiling.md) 36 | * [Responsible AI](responsible-ai.md) 37 | * [Program Management for ML projects](ml-tpm-guidance.md) 38 | 39 | ## References 40 | 41 | * [Model Operationalization](https://github.com/Microsoft/MLOps) 42 | -------------------------------------------------------------------------------- /docs/observability/alerting.md: -------------------------------------------------------------------------------- 1 | # Guidance for Alerting 2 | 3 | One of the goals of building highly observable systems is to provide valuable insight into the behavior of the application. Observable systems allow problems to be identified and surfaced through alerts before end users are impacted. 4 | 5 | ## Best Practices 6 | 7 | - The foremost thing to do before creating alerts is to implement observability. Without monitoring systems in place, it becomes next to impossible to know what activities need to be monitored and when to alert the teams. 8 | - Identify what the application's minimum viable service quality needs to be. It is not what you intend to deliver, but is acceptable for the customer. These [Service Level Objectives](https://landing.google.com/sre/sre-book/chapters/service-level-objectives/)(SLOs) are a metric for measurement of the application's performance. 9 | - SLOs are defined with respect to the end users. The alerts must watch for visible impact to the user. For example, alerting on request rate, latency and errors. 10 | - Use automated, scriptable tools to mimic end-to-end important code paths relatable to activities in the application. Create alert polices on user impacting events or metric rate of change. 11 | - Alert fatigue is real. Engineers are recommended to pay attention to their monitoring system so that accurate alerts and thresholds can be defined. 12 | - Establish a primary channel for alerts that needs immediate attention and tag the right team/person(s) based on the nature of the incident. Not every single alert needs to be sent to the primary on-call channel. 13 | - Establish a secondary channel for items that need to be looked into and does not affect the users, yet. For example, storage that nearing capacity threshold. These items will be what the engineering services will look to regularly to monitor the health of the system. 14 | - Ensure to set up proper alerting for failures in dependent services like Redis cache, Service Bus etc. For example, if Redis cache is throwing 10 exceptions in last 60 secs, proper alerts are recommended to be created so that these failures are surfaced and action be taken. 15 | - It is important to learn from each incident and continually improve the process. After every incident has been triaged, conduct a [post mortem of the scenario](https://landing.google.com/sre/workbook/chapters/postmortem-culture/). Scenarios and situations that were not initially considered will occur, and the post-mortem workflow is a great way to highlight that to improve the monitoring/alerting of the system. Configuring an alert to detect that incident scenario is a good idea to see if the event occurs again. 16 | -------------------------------------------------------------------------------- /docs/automated-testing/smoke-testing/README.md: -------------------------------------------------------------------------------- 1 | # Smoke Testing 2 | 3 | Smoke tests, sometimes named ***Sanity***, ***Acceptance***, or ***Build/Release Verification*** tests, are a sub-type of system/functional tests that are usually used as gates that verify the application's readiness as a preliminary step. If an application passes the smoke tests, it is acceptable, or in a stable-enough state, for the next stages of testing or deployment. 4 | 5 | ## When To Use 6 | 7 | ### Problem Addressed 8 | 9 | Smoke tests are meant to find, as early as possible, if an application is working or not. The goal of smoke tests is to save time; if the current version of the application does not pass smoke tests, then the rest of the integration or deployment chain for it can be abandoned. Smoke tests do not aim to provide full functionality coverage but instead focus on a few quick acceptance invocations for which the application should, at all times, respond correctly to. 10 | 11 | ### ROI Tipping Point 12 | 13 | Smoke tests cover only the most critical application path, and should not be used to actually test the application's behavior, keeping execution time and complexity to minimum. The tests can be formed of a subset of the application's integration or e2e tests, and they cover as much of the functionality with as little depth as required. 14 | 15 | The golden rule of a good smoke test is that it saves time on validating that the application is acceptable to a stage where better, more thorough testing will begin. 16 | 17 | ### Applicable to 18 | 19 | - [x] **Local dev desktop** - *Example:* Applying manual smoke testing to verify that the application is OK. 20 | - [x] **Build pipelines** - *Example:* Running a small set of the integration test suite before running the full coverage of tests, which may take a long time. 21 | - [x] **Non-production and Production deployments** - *Example:* Running a curl command to the product's API and asserting the response is 200 before running load test which consume resources. 22 | - [x] **PR Validation** - *Example:* - Deploying the application chart to a test namespace and validating the release is successful and no immediate regressions are merged. 23 | 24 | ## Conclusion 25 | 26 | Smoke testing is a low-effort, high-impact step to ship more reliable software. It should be considered amongst the first stages to implement when planning continuously integrated and delivered systems. 27 | 28 | ## Resources 29 | 30 | - [Wikipedia - Smoke Testing](https://en.wikipedia.org/wiki/Smoke_testing_(software)) 31 | - [Google SRE Book - System Tests](https://landing.google.com/sre/sre-book/chapters/testing-reliability/#system-tests) 32 | -------------------------------------------------------------------------------- /docs/code-reviews/evidence-and-measures/README.md: -------------------------------------------------------------------------------- 1 | # Evidence and Measures 2 | 3 | ## Evidence 4 | 5 | Many of the code quality assurance items can be automated or enforced by policies in modern version control and work item tracking systems. Verification of the policies on the main branch in [Azure DevOps](https://azure.microsoft.com/en-us/services/devops/) (AzDO) or [GitHub](https://github.com/), for example, may be sufficient evidence that a project team is conducting code reviews. 6 | 7 | * [ ] The main branches in all repositories have branch policies. - [Configure branch policies](../tools.md#Configuring Branch Policies) 8 | * [ ] All builds produced out of project repositories include appropriate linters, run unit tests. 9 | * [ ] Every bug work item should include a link to the pull request that introduced it, once the error has been diagnosed. This helps with learning. 10 | * [ ] Each bug work item should include a note on how the bug might (or might not have) been caught in a code review. 11 | * [ ] The project team regularly updates their code review checklists to reflect common issues they have encountered. 12 | * [ ] Dev Leads should review a sample of pull requests and/or be co-reviewers with other developers to help everyone improve their skills as code reviewers. 13 | 14 | ## Measures 15 | 16 | The team can collect metrics of code reviews to measure their efficiency. Some useful metrics include: 17 | 18 | * Defect Removal Efficiency (DRE) - a measure of the development team's ability to remove defects prior to release 19 | * Time metrics: 20 | * Time used preparing for code inspection sessions 21 | * Time used in review sessions 22 | * Lines of code (LOC) inspected per time unit/meeting 23 | 24 | It is a perfectly reasonable solution to track these metrics manually e.g. in an Excel sheet. It is also possible to utilize the features of project management platforms - for example, AzDO enables dashboards for metrics including [tracking bugs](https://docs.microsoft.com/en-us/azure/devops/boards/backlogs/manage-bugs?view=azure-devops&tabs=new-web-form). You may find ready-made plugins for various platforms - see [GitHub Marketplace](https://github.com/marketplace) for instance - or you can choose to implement these features yourself. 25 | 26 | Remember that since defects removed thanks to reviews is far less costly compared to finding them in production, the cost of doing code reviews is actually negative! 27 | 28 | For more information, see links under [resources](#resources). 29 | 30 | ## Resources 31 | 32 | * [A Guide to Code Inspections](http://www.ganssle.com/inspections.pdf) 33 | * [Defect Removal Effectiveness](https://www.westfallteam.com/sites/default/files/papers/defect_removal_effectiveness.pdf) 34 | -------------------------------------------------------------------------------- /docs/observability/tools/KubernetesDashboards.md: -------------------------------------------------------------------------------- 1 | # Kubernetes UI Dashboards 2 | 3 | This document covers the options and benefits of various Kubernetes UI Dashboards which are useful tools for monitoring and debugging your application on Kubernetes Clusters. It allows the management of applications running in the cluster, debug them and manage the cluster all through these dashboards. 4 | 5 | ## Overview and Background 6 | 7 | There are times when not all solutions can be run locally. This limitation could be due to a cloud service which does not offer a robust or efficient way to locally debug the environment. In these cases, it is necessary to use other tools which provide the capabilities to monitor your application with Kubernetes. 8 | 9 | ## Advantages and Use Cases 10 | 11 | - Allows the ability to view, manage and monitor the operational aspects of the Kubernetes Cluster. 12 | 13 | - Benefits of using a UI dashboard includes the following: 14 | - see an overview of the cluster 15 | - deploy applications onto the cluster 16 | - troubleshoot applications running on the cluster 17 | - view, create, modify, and delete Kubernetes resources 18 | - view basic resource metrics including resource usage for Kubernetes objects 19 | - view and access logs 20 | - live view of the pods state (e.g. started, terminating, etc) 21 | 22 | - Different dashboards may provide different functionalities, and the use case to choose a particular dashboard will depend on the requirements. For example, many dashboards provide a way to only monitor your applications on Kubernetes but do not provide a way to manage them. 23 | 24 | ## Open Source Dashboards 25 | 26 | There are currently several UI dashboards available to monitor your applications or manage them with Kubernetes. For example: 27 | 28 | - [Octant](https://github.com/vmware-tanzu/octant) 29 | - [Prometheus and Grafana](https://prometheus.io/docs/visualization/grafana/) 30 | - [Kube Prometheus Stack Chart](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack): provides an easy way to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. 31 | - [K8Dash](https://github.com/indeedeng/k8dash) 32 | - [kube-ops-view](https://github.com/hjacobs/kube-ops-view): a tool to visualize node occupancy & utilization 33 | - [Lens](https://k8slens.dev/): Client side desktop tool 34 | - [Thanos](https://github.com/thanos-io/thanos) and [Cortex](https://cortexmetrics.io/docs/): Multi-cluster implementations 35 | 36 | ## References 37 | 38 | - [Alternatives to Kubernetes Dashboard](https://octopus.com/blog/alternative-kubernetes-dashboards) 39 | - [Prometheus and Grafana with Kubernetes](https://tanzu.vmware.com/developer/guides/kubernetes/observability-prometheus-grafana-p1/) 40 | -------------------------------------------------------------------------------- /SECURITY.md: -------------------------------------------------------------------------------- 1 | <!-- BEGIN MICROSOFT SECURITY.MD V0.0.7 BLOCK --> 2 | 3 | # Security 4 | 5 | Microsoft takes the security of our software products and services seriously, which includes all source code repositories managed through our GitHub organizations, which include [Microsoft](https://github.com/Microsoft), [Azure](https://github.com/Azure), [DotNet](https://github.com/dotnet), [AspNet](https://github.com/aspnet), [Xamarin](https://github.com/xamarin), and [our GitHub organizations](https://opensource.microsoft.com/). 6 | 7 | If you believe you have found a security vulnerability in any Microsoft-owned repository that meets [Microsoft's definition of a security vulnerability](https://aka.ms/opensource/security/definition), please report it to us as described below. 8 | 9 | ## Reporting Security Issues 10 | 11 | **Please do not report security vulnerabilities through public GitHub issues.** 12 | 13 | Instead, please report them to the Microsoft Security Response Center (MSRC) at [https://msrc.microsoft.com/create-report](https://aka.ms/opensource/security/create-report). 14 | 15 | If you prefer to submit without logging in, send email to [secure@microsoft.com](mailto:secure@microsoft.com). If possible, encrypt your message with our PGP key; please download it from the [Microsoft Security Response Center PGP Key page](https://aka.ms/opensource/security/pgpkey). 16 | 17 | You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Additional information can be found at [microsoft.com/msrc](https://aka.ms/opensource/security/msrc). 18 | 19 | Please include the requested information listed below (as much as you can provide) to help us better understand the nature and scope of the possible issue: 20 | 21 | * Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.) 22 | * Full paths of source file(s) related to the manifestation of the issue 23 | * The location of the affected source code (tag/branch/commit or direct URL) 24 | * Any special configuration required to reproduce the issue 25 | * Step-by-step instructions to reproduce the issue 26 | * Proof-of-concept or exploit code (if possible) 27 | * Impact of the issue, including how an attacker might exploit the issue 28 | 29 | This information will help us triage your report more quickly. 30 | 31 | If you are reporting for a bug bounty, more complete reports can contribute to a higher bounty award. Please visit our [Microsoft Bug Bounty Program](https://aka.ms/opensource/security/bounty) page for more details about our active programs. 32 | 33 | ## Preferred Languages 34 | 35 | We prefer all communications to be in English. 36 | 37 | ## Policy 38 | 39 | Microsoft follows the principle of [Coordinated Vulnerability Disclosure](https://aka.ms/opensource/security/cvd). 40 | 41 | <!-- END MICROSOFT SECURITY.MD BLOCK --> 42 | -------------------------------------------------------------------------------- /.devcontainer/devcontainer.json: -------------------------------------------------------------------------------- 1 | // For format details, see https://aka.ms/vscode-remote/devcontainer.json or this file's README at: 2 | // https://github.com/microsoft/vscode-dev-containers/tree/v0.202.3/containers/python-3 3 | { 4 | "name": "Python 3", 5 | "build": { 6 | "dockerfile": "Dockerfile", 7 | "context": "..", 8 | "args": { 9 | // Update 'VARIANT' to pick a Python version: 3, 3.9, 3.8, 3.7, 3.6. 10 | // Append -bullseye or -buster to pin to an OS version. 11 | // Use -bullseye variants on local on arm64/Apple Silicon. 12 | "VARIANT": "3.9-bullseye", 13 | // Options 14 | "NODE_VERSION": "lts/*" 15 | } 16 | }, 17 | 18 | // Set *default* container specific settings.json values on container create. 19 | "settings": { 20 | "terminal.integrated.profiles.linux": { 21 | "bash": { 22 | "path": "/bin/bash" 23 | } 24 | }, 25 | "python.defaultInterpreterPath": "/usr/local/bin/python", 26 | "python.languageServer": "Default", 27 | "python.linting.enabled": true, 28 | "python.linting.pylintEnabled": true, 29 | "python.formatting.autopep8Path": "/usr/local/py-utils/bin/autopep8", 30 | "python.formatting.blackPath": "/usr/local/py-utils/bin/black", 31 | "python.formatting.yapfPath": "/usr/local/py-utils/bin/yapf", 32 | "python.linting.banditPath": "/usr/local/py-utils/bin/bandit", 33 | "python.linting.flake8Path": "/usr/local/py-utils/bin/flake8", 34 | "python.linting.mypyPath": "/usr/local/py-utils/bin/mypy", 35 | "python.linting.pycodestylePath": "/usr/local/py-utils/bin/pycodestyle", 36 | "python.linting.pydocstylePath": "/usr/local/py-utils/bin/pydocstyle", 37 | "python.linting.pylintPath": "/usr/local/py-utils/bin/pylint" 38 | }, 39 | 40 | // Add the IDs of extensions you want installed when the container is created. 41 | "extensions": [ 42 | "ms-python.python", 43 | "ms-python.vscode-pylance" 44 | ], 45 | 46 | // Use 'forwardPorts' to make a list of ports inside the container available locally. 47 | "forwardPorts": [8000], 48 | 49 | // Use 'portsAttributes' to set default properties for specific forwarded ports. More info: https://code.visualstudio.com/docs/remote/devcontainerjson-reference. 50 | "portsAttributes": { 51 | "8000": { 52 | "label": "mkdocs serve", 53 | "onAutoForward": "notify" 54 | } 55 | }, 56 | 57 | // Use 'otherPortsAttributes' to configure any ports that aren't configured using 'portsAttributes'. 58 | // "otherPortsAttributes": { 59 | // "onAutoForward": "silent" 60 | // }, 61 | 62 | // Use 'postCreateCommand' to run commands after the container is created. 63 | "postCreateCommand": "pip3 install -r requirements-docs.txt", 64 | 65 | // Comment out to connect as root instead. More info: https://aka.ms/vscode-remote/containers/non-root. 66 | "remoteUser": "vscode", 67 | 68 | "mounts": [ 69 | "source=${localEnv:HOME}${localEnv:USERPROFILE}/.ssh,target=/home/vscode/.ssh,type=bind" 70 | ], 71 | } -------------------------------------------------------------------------------- /docs/machine-learning/responsible-ai.md: -------------------------------------------------------------------------------- 1 | # Responsible AI in CSE 2 | 3 | ## Microsoft's Responsible AI principles 4 | 5 | Every ML project in CSE goes through a Responsible AI (RAI) assessment to ensure that it upholds Microsoft's [6 Responsible AI principles](https://www.microsoft.com/en-us/ai/responsible-ai): 6 | 7 | - Fairness 8 | - Reliability & Safety 9 | - Privacy & Security 10 | - Inclusiveness 11 | - Transparency 12 | - Accountability 13 | 14 | Every project goes through the RAI process, whether we are building a new ML model from scratch, or putting an existing model in production. 15 | 16 | ## CSE's Responsible AI process 17 | 18 | The process begins as soon as we start a prospective project. We start to complete a Responsible AI review document, and an impact assessment, which provides a structured way to explore topics such as: 19 | 20 | - Can the problem be addressed with a non-technical (e.g. social) solution? 21 | - Can the problem be solved without AI? Would simpler technology suffice? 22 | - Will the team have access to domain experts (e.g. doctors, refugees) in the field where the AI is applicable? 23 | - Who are the stakeholders in this project? Who does the AI impact? Are there any vulnerable groups affected? 24 | - What are the possible benefits and harms to each stakeholder? 25 | - How can the technology be misused, and what can go wrong? 26 | - Has the team analyzed the input data properly to make sure that the training data is suitable for machine learning? 27 | - Is the training data an accurate representation of data that will be used as input in production? 28 | - Is there a good representation of all users? 29 | - Is there a fall-back mechanism (a human in the loop, or a way to revert decisions based on the model)? 30 | - Does data used by the model for training or scoring contain PII? What measures have been taken to remove sensitive data? 31 | - Does the model impact consequential decisions, like blocking people from getting jobs, loans, health care etc. or in the cases where it may, have appropriate ethical considerations been discussed? 32 | - Have measures for re-training been considered? 33 | - How can we address any concerns that arise, and how can we mitigate risk? 34 | 35 | At this point we research available [tools and resources](https://www.microsoft.com/en-us/ai/responsible-ai-resources), such as [InterpretML](https://interpret.ml/) or [Fairlearn](https://github.com/fairlearn/fairlearn), that we may use on the project. We may change the project scope or re-define the [ML problem definition](ml-problem-formulation-envisioning.md) if necessary. 36 | 37 | The Responsible AI review documents remain living documents that we re-visit and update throughout project development, through the [feasibility study](ml-feasibility-study.md), as the model is developed and prepared for production, and new information unfolds. The documents can be used and expanded once the model is deployed, and monitored in production. 38 | -------------------------------------------------------------------------------- /docs/observability/observability-as-code.md: -------------------------------------------------------------------------------- 1 | # Observability as Code 2 | 3 | As much as possible, configuration and management of observability assets such as cloud resource provisioning, monitoring alerts and dashboards must be managed as code. Observability as Code is achieved using any one of Terraform / Ansible / ARM Templates 4 | 5 | ## Examples of Observability as Code 6 | 7 | 1. Dashboards as Code - Monitoring Dashboards can be created as JSON or XML templates. This template is source control maintained and any changes to the dashboards can be reviewed. Automation can be built for enabling the dashboard. [More about how to do this in Azure](https://docs.microsoft.com/en-us/azure/azure-portal/azure-portal-dashboards-create-programmatically). Grafana dashboard can also be [configured as code](https://grafana.com/blog/2020/02/26/how-to-configure-grafana-as-code/) which eventually can be source-controlled to be used in automation and pipelines. 8 | 9 | 2. Alerts as Code - Alerts can be created within Azure by using Terraform or ARM templates. Such alerts can be source-controlled and be deployed as part of pipelines (Azure DevOps pipelines, Jenkins, GitHub Actions etc.). Few references of how to do this are: [Terraform Monitor Metric Alert](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/monitor_metric_alert). Alerts can also be created based on log analytics query and can be defined as code using [Terraform Monitor Scheduled Query Rules Alert](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/monitor_scheduled_query_rules_alert#example-usage). 10 | 11 | 3. Automating Log Analytics Queries - There are several use cases where automation of log analytics queries may be needed. Example, Automatic Report Generation, Running custom queries programmatically for analysis, debugging etc. For these use cases to work, log queries should be source-controlled and automation can be built using [log analytics REST](https://dev.loganalytics.io/documentation/Using-the-API) or [azure cli](https://docs.microsoft.com/en-us/cli/azure/ext/log-analytics/monitor/log-analytics?view=azure-cli-latest). 12 | 13 | ## Why 14 | 15 | - It makes configuration repeatable and automatable. It also avoids manual configuration of monitoring alerts and dashboards from scratch across environments. 16 | 17 | - Configured dashboards help troubleshoot errors during integration and deployment (CI/CD) 18 | 19 | - We can audit changes and roll them back if there are any issues. 20 | 21 | - Identify actionable insights from the generated metrics data across all environments, not just production. 22 | 23 | - Configuration and management of observability assets like alert threshold, duration, configuration 24 | values using IAC help us in avoiding configuration mistakes, errors or overlooks during deployment. 25 | 26 | - When practicing observability as code, the changes can be reviewed by the team similar to other code 27 | contributions. 28 | -------------------------------------------------------------------------------- /docs/code-reviews/recipes/java.md: -------------------------------------------------------------------------------- 1 | # Java Code Reviews 2 | 3 | ## Java Style Guide 4 | 5 | [CSE](../../CSE.md) developers generally follow the [Google Java Style Guide](https://google.github.io/styleguide/javaguide.html). 6 | 7 | ## Code Analysis / Linting 8 | 9 | We strongly believe that consistent style increases readability and maintainability of a code base. Hence, we are recommending analyzers to enforce consistency and style rules. 10 | 11 | We make use of [Checkstyle](https://github.com/checkstyle/checkstyle) using the [same configuration used in the Azure Java SDK](https://github.com/Azure/azure-sdk-for-java/blob/master/eng/code-quality-reports/src/main/resources/checkstyle/checkstyle.xml). 12 | 13 | [FindBugs](http://findbugs.sourceforge.net/) and [PMD](https://pmd.github.io/) are also commonly used. 14 | 15 | ## Automatic Code Formatting 16 | 17 | Eclipse, and other Java IDEs, support automatic code formatting. If using Maven, some developers also make use of the [formatter-maven-plugin](https://github.com/revelc/formatter-maven-plugin). 18 | 19 | ## Build Validation 20 | 21 | It's important to enforce your code style and rules in the CI to avoid any team members merging code that does not comply with standards into your git repo. If building using Azure DevOps, Azure DevOps support [Maven](https://docs.microsoft.com/azure/devops/pipelines/tasks/build/maven?view=azure-devops) and [Gradle](https://docs.microsoft.com/azure/devops/pipelines/tasks/build/gradle?view=azure-devops) build tasks using [PMD](https://pmd.github.io/), [Checkstyle](https://checkstyle.sourceforge.io/), and [FindBugs](http://findbugs.sourceforge.net/) code analysis tools as part of every build. 22 | 23 | Here is an example yaml for a Maven build task with all three analysis tools enabled: 24 | 25 | ```yaml 26 | - task: Maven@3 27 | displayName: 'Maven pom.xml' 28 | inputs: 29 | mavenPomFile: '$(Parameters.mavenPOMFile)' 30 | checkStyleRunAnalysis: true 31 | pmdRunAnalysis: true 32 | findBugsRunAnalysis: true 33 | ``` 34 | 35 | Here is an example yaml for a Gradle build task with all three analysis tools enabled: 36 | 37 | ```yaml 38 | - task: Gradle@2 39 | displayName: 'gradlew build' 40 | inputs: 41 | checkStyleRunAnalysis: true 42 | findBugsRunAnalysis: true 43 | pmdRunAnalysis: true 44 | ``` 45 | 46 | ## Code Review Checklist 47 | 48 | In addition to the [Code Review Checklist](../process-guidance/reviewer-guidance.md) you should also look for these Java specific code review items 49 | 50 | * [ ] Does the project use Lambda to make code cleaner? 51 | * [ ] Is dependency injection (DI) used? Is it setup correctly? 52 | * [ ] If the code uses Spring Boot, are you using @Inject instead of @Autowire? 53 | * [ ] Does the code handle exceptions correctly? 54 | * [ ] Is the [Azul Zulu OpenJDK](https://docs.microsoft.com/en-us/java/azure/jdk/java-jdk-install?view=azure-java-stable) being used? 55 | * [ ] Is a build automation and package management tool (Gradle or Maven) being used? 56 | -------------------------------------------------------------------------------- /docs/automated-testing/unit-testing/why-unit-tests.md: -------------------------------------------------------------------------------- 1 | # Why Unit Tests 2 | 3 | It is no secret that writing unit tests is hard, and even harder to write well. Writing unit tests also increases the 4 | development time for every feature. So why should we bother writing them? 5 | 6 | ## Reduce costs 7 | 8 | There is no question that the later a bug is found, the more expensive it is to fix; especially so if the bug makes it 9 | into production. A [2008 research study by IBM](ftp://ftp.software.ibm.com/software/rational/info/do-more/RAW14109USEN.pdf) 10 | estimates that a bug caught in production could cost 6 times as much as if it was caught during implementation. 11 | 12 | ## Increase Developer Confidence 13 | 14 | Many changes that developers make are not big features or something that requires an entire testing suite. A strong unit 15 | test suite helps increase the confidence of the developer that their change is not going to cause any downstream bugs. 16 | Having unit tests also helps with making safe, mechanical refactors that are provably safe; using things like 17 | refactoring tools to do mechanical refactoring and running unit tests that cover the refactored code should be enough to 18 | increase confidence in the commit. 19 | 20 | ## Speed up development 21 | 22 | Unit tests take time to write, but they also speed up development? While this may seem like an oxymoron, it is one of 23 | the strengths of a unit testing suite - over time it continues to grow and evolve until the tests become an essential 24 | part of the developer workflow. 25 | 26 | If the only testing available to a developer is a long-running system test, integration tests that require a deployment, 27 | or manual testing, it will increase the amount of time taken to write a feature. These types of tests should be a part of 28 | the "Outer loop"; tests that may take some time to run and validate more than just the code you are writing. Usually 29 | these types of outer loop tests get run at the PR stage or even later during merges into branches. 30 | 31 | The Developer Inner Loop is the process that developers go through as they are authoring code. This varies from 32 | developer to developer and language to language but typically is something like code -> build -> run -> repeat. When 33 | unit tests are inserted into the inner loop, developers can get early feedback and results from the code they are 34 | writing. Since unit tests execute really quickly, running tests shouldn't be seen as a barrier to entry for this loop. 35 | Tooling such as [Visual Studio Live Unit Testing](https://docs.microsoft.com/en-us/visualstudio/test/live-unit-testing-start?view=vs-2019) 36 | also help to shorten the inner loop even more. 37 | 38 | ## Documentation as code 39 | 40 | Writing unit tests is a great way to show how the units of code you are writing are supposed to be used. In some ways, 41 | unit tests are better than any documentation or samples because they are (or at least should be) executed with every 42 | build so there is confidence that they are not out of date. Unit tests also should be so simple that they are easy to follow. 43 | -------------------------------------------------------------------------------- /docs/documentation/recipes/static-website-with-mkdocs.md: -------------------------------------------------------------------------------- 1 | # How to create a static website for your documentation based on mkdocs and mkdocs-material 2 | 3 | [MkDocs](https://www.mkdocs.org/) is a tool built to create static websites from raw markdown files. Other alternatives include [Sphinx](https://www.sphinx-doc.org/en/master/), and [Jekyll](https://jekyllrb.com/). 4 | 5 | We used MkDocs to create [CSE Code-With Engineering Playbook](https://microsoft.github.io/code-with-engineering-playbook/) static website from the contents in [the GitHub repository](https://github.com/microsoft/code-with-engineering-playbook). Then we deployed it to [GitHub Pages](https://pages.github.com/). 6 | 7 | We found MkDocs to be a good choice since: 8 | 9 | 1. It's easy to set up and looks great even with the vanilla version. 10 | 2. It works well with markdown, which is what we already have in the Playbook. 11 | 3. It uses a Python stack which is friendly to many contributors of this Playbook. 12 | 13 | For comparison, Sphinx mainly generates docs from restructured-text (rst) format, and Jekyll is written in Ruby. 14 | 15 | To setup an MkDocs website, the main assets needed are: 16 | 17 | 1. An ```mkdocs.yaml``` file, similar to the one we have [in the Playbook](https://github.com/microsoft/code-with-engineering-playbook/blob/main/mkdocs.yml). This is the configuration file that defines the appearance of the website, the navigation, the plugins used and more. 18 | 2. A folder named ```docs``` (the default value for the directory) that contains the documentation source files. 19 | 3. A [GitHub Action](https://docs.github.com/actions/learn-github-actions/understanding-github-actions) for automatically generating the website (e.g. on every commit to main), similar to [this one from the Playbook](https://github.com/microsoft/code-with-engineering-playbook/blob/main/.github/workflows/mkdocs.yml). 20 | 4. A list of plugins used during the build phase of the website. We specified ours [here](https://github.com/microsoft/code-with-engineering-playbook/blob/main/requirements-docs.txt). And these are the plugins we've used: 21 | 22 | - [Material for MkDocs](https://squidfunk.github.io/mkdocs-material/): Material design appearance and user experience. 23 | - [pymdown-extensions](https://facelessuser.github.io/pymdown-extensions/): Improves the appearance of markdown based content. 24 | - [mdx_truly_sane_lists](https://github.com/radude/mdx_truly_sane_lists): For defining the indent level for lists without having to refactor the entire documentation we already had in the Playbook. 25 | 26 | Setting up locally is very easy. See [Getting Started with MkDocs](https://www.mkdocs.org/getting-started/) for details. 27 | 28 | For publishing the website, there's a [good integration with GitHub for storing the website as a GitHub Page](https://www.mkdocs.org/user-guide/deploying-your-docs/). 29 | 30 | ## Additional links 31 | 32 | - [MkDocs Plugins](https://github.com/mkdocs/mkdocs/wiki/MkDocs-Plugins) 33 | - [The best MkDocs plugins and customizations](https://chrieke.medium.com/the-best-mkdocs-plugins-and-customizations-fc820eb19759) 34 | -------------------------------------------------------------------------------- /docs/observability/log-vs-metric-vs-trace.md: -------------------------------------------------------------------------------- 1 | # Logs vs Metrics vs Traces 2 | 3 | ![Signals](./images/signals.png) 4 | 5 | ## Overview 6 | 7 | ### Metrics 8 | 9 | The purpose of metrics is to inform observers about the health & operations regarding a component or system. A metric represents a point in time measure of a particular source, and data-wise tends to be very small. The compact size allows for efficient collection even at scale in large systems. Metrics also lend themselves very well to pre-aggregation within the component before collection, reducing computation cost for processing & storing large numbers of metric time series in a central system. Due to how efficiently metrics are processed & stored, it lends itself very well for use in automated alerting, as metrics are an excellent source for the health data for all components in the system. 10 | 11 | ### Logs 12 | 13 | Log data inform observers about the discrete events that occurred within a component or a set of components. Just about every software component log information about its activities over time. This rich data tends to be much larger than metric data and can cause processing issues, especially if components are logging too verbosely. Therefore, using log data to understand the health of an extensive system tends to be avoided and depends on metrics for that data. Once metric telemetry highlights potential problem sources, filtered log data for those sources can be used to understand what occurred. 14 | 15 | ### Traces 16 | 17 | Where logging provides an overview to a discrete, event-triggered log, tracing encompasses a much wider, continuous view of an application. The goal of tracing is to following a program’s flow and data progression. 18 | 19 | In many instances, tracing represents a single user’s journey through an entire app stack. Its purpose isn’t reactive, but instead focused on optimization. By tracing through a stack, developers can identify bottlenecks and focus on improving performance. 20 | 21 | A distributed trace is defined as a collection of spans. A span is the smallest unit in a trace and represents a piece of the workflow in a distributed landscape. It can be an HTTP request, call to a database, or execution of a message from a queue. 22 | 23 | When a problem does occur, tracing allows you to see how you got there: 24 | 25 | * Which function. 26 | * The function’s duration. 27 | * Parameters passed. 28 | * How deep into the function the user could get. 29 | 30 | ## Usage Guidance 31 | 32 | When to use metric or log data to track a particular piece of telemetry can be summarized with the following points: 33 | 34 | * Use metrics to track the occurrence of an event, counting of items, the time taken to perform an action or to report the current value of a resource (CPU, memory, etc.) 35 | * Use logs to track detailed information about an event also monitored by a metric, particularly errors, warnings or other exceptional situations. 36 | * A trace provides visibility into how a request is processed across multiple services in a microservices environment. Every trace needs to have a unique identifier associated with it. 37 | -------------------------------------------------------------------------------- /docs/machine-learning/ml-fundamentals-checklist.md: -------------------------------------------------------------------------------- 1 | # ML Fundamentals Checklist 2 | 3 | This checklist helps ensure that our ML projects meet our ML Fundamentals. The items below are not sequential, but rather organized by different parts of an ML project. 4 | 5 | ## Data Quality and Governance 6 | 7 | - [ ] There is access to data. 8 | - [ ] Labels exist for dataset of interest. 9 | - [ ] Data quality evaluation. 10 | - [ ] Able to track data lineage. 11 | - [ ] Understanding of where the data is coming from and any policies related to data access. 12 | - [ ] Gather Security and Compliance requirements. 13 | 14 | ## Feasibility Study 15 | 16 | - [ ] A feasibility study was performed to assess if the data supports the proposed tasks. 17 | - [ ] Rigorous Exploratory data analysis was performed (including analysis of data distribution). 18 | - [ ] Hypotheses were tested producing sufficient evidence to either support or reject that an ML approach is feasible to solve the problem. 19 | - [ ] ROI estimation and risk analysis was performed for the project. 20 | - [ ] ML outputs/assets can be integrated within the production system. 21 | - [ ] Recommendations on how to proceed have been documented. 22 | 23 | ## Evaluation and Metrics 24 | 25 | - [ ] Clear definition of how performance will be measured. 26 | - [ ] The evaluation metrics are somewhat connected to the success criteria. 27 | - [ ] The metrics can be calculated with the datasets available. 28 | - [ ] Evaluation flow can be applied to all versions of the model. 29 | - [ ] Evaluation code is unit-tested and reviewed by all team members. 30 | - [ ] Evaluation flow facilitates further results and error analysis. 31 | 32 | ## Model Baseline 33 | 34 | - [ ] Well-defined baseline model exists and its performance is calculated. ([More details on well defined baselines](ml-model-checklist.md#is-there-a-well-defined-baseline-is-the-model-performing-better-than-the-baseline)) 35 | - [ ] The performance of other ML models can be compared with the model baseline. 36 | 37 | ## Experimentation setup 38 | 39 | - [ ] Well-defined train/test dataset with labels. 40 | - [ ] Reproducible and logged experiments in an environment accessible by all data scientists to quickly iterate. 41 | - [ ] Defined experiments/hypothesis to test. 42 | - [ ] Results of experiments are documented. 43 | - [ ] Model hyper parameters are tuned systematically. 44 | - [ ] Same performance evaluation metrics and consistent datasets are used when comparing candidate models. 45 | 46 | ## Production 47 | 48 | - [ ] [Model readiness checklist](ml-model-checklist.md) reviewed. 49 | - [ ] Model reviews were performed (covering model debugging, reviews of training and evaluation approaches, model performance). 50 | - [ ] Data pipeline for inferencing, including an end-to-end tests. 51 | - [ ] SLAs requirements for models are gathered and documented. 52 | - [ ] Monitoring of data feeds and model output. 53 | - [ ] Ensure consistent schema is used across the system with expected input/output defined for each component of the pipelines (data processing as well as models). 54 | - [ ] [Responsible AI](responsible-ai.md) reviewed. 55 | -------------------------------------------------------------------------------- /docs/agile-development/advanced-topics/effective-organization/delivery-plan.md: -------------------------------------------------------------------------------- 1 | # Delivery Plan 2 | 3 | ## Goals 4 | 5 | While Scrum does not require and discourages planning more than one sprint at a time. Most of us work in enterprises where we are dependent outside teams (for example: marketing, sales, support). 6 | 7 | A rough assessment of the planned project scope is achievable within a reasonable time frame and resources. The goal is to have a rough plan and estimate as a starting point, not to implement "Agilefall." 8 | 9 | Note that this is just a starting point to enable planning discussions. We expect the actual schedule to evolve and shift over time and that you will update the scope and timeline as you progress. 10 | 11 | Delivery Plans ensure your teams are aligning with your organizational goals. 12 | 13 | ## Benefits 14 | 15 | - As you complete the assessment, you can push back on the scope, time frame or ask for more resources. 16 | - As you progress in your project/product delivery, you can highlight risks to the scope, time frame, and resources. 17 | 18 | ## Approach 19 | 20 | One approach you can take to accomplish is with stickies and a spreadsheet. 21 | 22 | Step 1: Stack rank the features for everything in your backlog 23 | 24 | - Functional Features 25 | - [Non-functional Features] (docs/TECH-LEADS-CHECKLIST.md) 26 | - User Research and Design 27 | - Testing 28 | - Documentation 29 | - Knowledge Transfer/Support Processes 30 | 31 | Step 2: T-Shirt Features in terms of working weeks per person. In some scenarios, you have no idea how complex the work. In this situation, you can ask for time to conduct a spike (timebox the effort so you can get back on time). 32 | 33 | Step 3: Calculate the capacity for the team based on the number of weeks person with his/her start and end date and minus holidays, vacation, conferences, training, and onboarding days. Also, minus time if the person is also working on defects and support. 34 | 35 | Step 4: Based on your capacity, you know have the options 36 | 37 | - Ask for more resources. Caution: onboarding new resources take time. 38 | - Reduce the scope to the most MVP. Caution: as you trim more of the scope, it might be valuable anymore to the customer. Consider a cupcake which is everything you need. You don't want to skim on the frosting. 39 | - Ask for more time. Usually, this is the most flexible, but if there is a marketing date that you need to hit, this might be as flexible. 40 | 41 | ## Tools 42 | 43 | You can also leverage one of these tools by creating your epics and features and add the weeks estimates. 44 | 45 | The [Plans (Preview) feature on Azure DevOps](https://docs.microsoft.com/en-us/azure/devops/project/navigation/preview-features?view=azure-devops) will help you make a plan. Delivery Plans provide a schedule of stories or features your team plan to deliver. Delivery Plans show the scheduled work items by a sprint (iteration path) of selected teams against a calendar view. 46 | 47 | Confluence JIRA, Trello, Rally, Asana, Basecamp, and GitHub Issues are other similar tools in the market (some are free, others you pay a monthly fee, or you can install on-prem) that you can leverage. 48 | -------------------------------------------------------------------------------- /docs/design/design-reviews/recipes/technical-spike.md: -------------------------------------------------------------------------------- 1 | # Technical Spike 2 | 3 | From [Wikipedia](<https://en.wikipedia.org/wiki/Spike_(software_development)>)... 4 | 5 | A spike in a sprint can be used in a number of ways: 6 | 7 | - As a way to familiarize the team with new hardware or software 8 | - To analyze a problem thoroughly and assist in properly dividing work among separate team members. 9 | - Spike tests can also be used to mitigate future risk, and may uncover additional issues that have escaped notice. 10 | 11 | A distinction can be made between technical spikes and functional spikes. The technical spike is used more often for evaluating the impact new technology has on the current implementation. A functional spike is used to determine the interaction with a new feature or implementation. 12 | 13 | [Engineering feasibility spikes](./engineering-feasibility-spikes.md) can also be conducted to de-risk an engagement and increase the team's understanding. 14 | 15 | ## Deliverable 16 | 17 | Generally the deliverable from a Technical Spike should be a document detailing what was evaluated and the outcome of that evaluation. The specifics contained in the document will vary, but there are some general principles that might be helpful. 18 | 19 | - **Problem Statement/Goals:** Be sure to include a section that clearly details why an evaluation is being done and what the outcome of this evaluation should be. This is helpful to ensure that the technical spike was productive and advanced the overall project in some way. 20 | 21 | - **Make sure it is repeatable:** Detail the components used, installation instructions, configuration, etc. required to build the environment that was used for evaluation and testing. If any testing is performed, make sure to include the scripts, links to the applications, configuration options, etc. so that testing could be performed again. 22 | 23 | There are many reasons that the evaluation environment may need to be rebuilt. For example: 24 | 25 | - Another scenario needs to be tested. 26 | - A new version of the technology has been released. 27 | - The technology needs to be tested on a new platform. 28 | 29 | - **Fact-Finding:** The goal of a spike should be fact-finding, not decision-making or recommendation. Ideally, the technology spike digs into a number of technical questions and gets answers so that the _broader project team_ can then come back together and agree on an appropriate course forward. 30 | 31 | - **Evidence:** Generally you will use sections to summarize the results of testing which do not include the potentially hundreds of detailed results, however, you should include all detailed testing results in an appendix or an attachment. Having full results detailed somewhere will help the team trust the results. In addition, data can be interpreted lots of different ways, and it may be necessary to go back to the original data for a new interpretation. 32 | 33 | - **Organization:** The technical documentation can be lengthy. It is generally a good idea to organize sections with headers and include a table of contents. Generally sections towards the beginning of the document should summarize data and use one or more appendices for more details. 34 | -------------------------------------------------------------------------------- /docs/code-reviews/process-guidance/README.md: -------------------------------------------------------------------------------- 1 | # Process Guidance 2 | 3 | ## General Guidance 4 | 5 | Code reviews should be part of the software engineering team process regardless of the development model. Furthermore, the team should learn to execute reviews in a timely manner. [Pull requests (PRs)](../pull-requests.md) left hanging can cause additional merge problems and go stale resulting in lost work. Qualified PRs are expected to reflect well-defined, concise tasks, and thus be compact in content. Reviewing a single task should then take relatively little time to complete. 6 | 7 | To ensure that the code review process is healthy, inclusive and meets the goals stated above, consider following these guidelines: 8 | 9 | - Establish a [service-level agreement (SLA)](https://en.wikipedia.org/wiki/Service-level_agreement) for code reviews and add it to your teams working agreement. 10 | - Although modern DevOps environments incorporate tools for managing PRs, it can be useful to label tasks pending for review or to have a dedicated place for them on the task board - [Customize AzDO task boards](../tools.md#task-boards) 11 | - In the daily standup meeting check tasks pending for review and make sure they have reviewers assigned. 12 | - Junior teams and teams new to the process can consider creating separate tasks for reviews together with the tasks themselves. 13 | - Utilize tools to streamline the review process - [Code review tools](../tools.md) 14 | - Foster inclusive code reviews - [Inclusion in Code Review](../inclusion-in-code-review.md) 15 | 16 | ## Measuring code review process 17 | 18 | If the team is finding that code reviews are taking a significant time to merge, and it is becoming a blocker, consider the following additional recommendations: 19 | 20 | 1. Measure the average time it takes to merge a PR per sprint cycle. 21 | 1. Review during retrospective how the time to merge can be improved and prioritized. 22 | 1. Assess the time to merge across sprints to see if the process is improving. 23 | 1. Ping required approvers directly as a reminder. 24 | 25 | ## Code reviews shouldn't include too many lines of code 26 | 27 | It's easy to say a developer can review few hundred lines of code, but when the code surpasses certain amount of lines, the effectiveness of defects discovery will decrease and there is a lesser chance of doing a good review. It's not a matter of setting a code line limit, but rather using common sense. More code there is to review, the higher chances there are letting a bug sneak through. See [PR size guidance](../pull-requests.md#size-guidance). 28 | 29 | ## Automate whenever reasonable 30 | 31 | Use automation (linting, code analysis etc.) to avoid the need for "[nits](https://en.wikipedia.org/wiki/Nitpicking)" and allow the reviewer to focus more on the functional aspects of the PR. By configuring automated builds, tests and checks (something achievable in the [CI process](../../continuous-integration/README.md)), teams can save human reviewers some time and let them focus in areas like design and functionality for proper evaluation. This will ensure higher chances of success as the team is focusing on the things that matter. 32 | 33 | ## Role specific guidance 34 | 35 | - [Author Guidance](author-guidance.md) 36 | - [Reviewer Guidance](reviewer-guidance.md) 37 | -------------------------------------------------------------------------------- /docs/developer-experience/recipes/cross-platform-tasks.md: -------------------------------------------------------------------------------- 1 | # Cross Platform Tasks 2 | 3 | There are several options to alleviate cross-platform compatibility issues. 4 | 5 | - Running tasks in a container 6 | - Using the tasks-system in VS Code which provides options to allow commands to be executed specific to an operating system. 7 | 8 | ## Docker or Container based 9 | 10 | Using containers as development machines allows developers to get started with minimal setup and abstracts the development environment from the host OS by having it run in a container. 11 | DevContainers can also help in standardizing the local developer experience across the team. 12 | 13 | The following are some good resources to get started with running tasks in DevContainers 14 | 15 | - [Developing inside a container](https://code.visualstudio.com/docs/remote/containers). 16 | - [Tutorial on Development in Containers](https://code.visualstudio.com/docs/remote/containers-tutorial) 17 | - For samples projects and dev container templates see [VS Code Dev Containers Recipe](https://github.com/microsoft/vscode-dev-containers) 18 | - [Dev Containers Library](../devcontainers.md) 19 | 20 | ## Tasks in VS Code 21 | 22 | ### Running Node.js 23 | 24 | The example below offers insight into running Node.js executable as a command with tasks.json and how it can be treated differently on Windows and Linux. 25 | 26 | ```json 27 | { 28 | "label": "Run Node", 29 | "type": "process", 30 | "windows": { 31 | "command": "C:\\Program Files\\nodejs\\node.exe" 32 | }, 33 | "linux": { 34 | "command": "/usr/bin/node" 35 | } 36 | } 37 | ``` 38 | 39 | In this example, to run Node.js, there is a specific windows command, and a specific linux command. This allows for platform specific properties. When these are defined, they will be used instead of the default properties when the command is executed on the Windows operating system or on Linux. 40 | 41 | ### Custom Tasks 42 | 43 | Not all scripts or tasks can be auto-detected in the workspace. It may be necessary at times to defined your own custom tasks. In this example, we have a script to run in order to set up some environment correctly. The script is stored in a folder inside your workspace and named test.sh for Linux & macOS and test.cmd for Windows. With the tasks.json file, the execution of this script can be made possible with a custom task that defines what to do on different operating systems. 44 | 45 | ```json 46 | { 47 | "version": "2.0.0", 48 | "tasks": [ 49 | { 50 | "label": "Run tests", 51 | "type": "shell", 52 | "command": "./scripts/test.sh", 53 | "windows": { 54 | "command": ".\\scripts\\test.cmd" 55 | }, 56 | "group": "test", 57 | "presentation": { 58 | "reveal": "always", 59 | "panel": "new" 60 | } 61 | } 62 | ] 63 | } 64 | 65 | ``` 66 | 67 | The command here is a shell command and tells the system to run either the test.sh or test.cmd. By default, it will run test.sh with that given path. This example here also defines Windows specific properties and tells it execute test.cmd instead of the default. 68 | 69 | ### References 70 | 71 | VS Code Docs - [operating system specific properties](https://vscode-docs.readthedocs.io/en/stable/editor/tasks/#operating-system-specific-properties) 72 | --------------------------------------------------------------------------------