├── CITATION.cff ├── CODE_OF_CONDUCT.md ├── README.md ├── bazel-blaze-for-dependencies └── index.md ├── git-submodules └── index.md ├── git-vs-hg-vs-svn-vs-perforce-et-al └── index.md ├── partial-checkout └── index.md ├── scm-at-facebook └── index.md ├── scm-at-google └── index.md ├── scm-comparisons-across-organizations └── index.md └── security-auditing-compliance └── index.md /CITATION.cff: -------------------------------------------------------------------------------- 1 | cff-version: 1.2.0 2 | title: Source code management - notes and ideas 3 | message: >- 4 | If you use this work and you want to cite it, 5 | then you can use the metadata from this file. 6 | type: software 7 | authors: 8 | - given-names: Joel Parker 9 | family-names: Henderson 10 | email: joel@joelparkerhenderson.com 11 | affiliation: joelparkerhenderson.com 12 | orcid: 'https://orcid.org/0009-0000-4681-282X' 13 | identifiers: 14 | - type: url 15 | value: 'https://github.com/joelparkerhenderson/source-code-management/' 16 | description: Source code management - notes and ideas 17 | repository-code: 'https://github.com/joelparkerhenderson/source-code-management/' 18 | abstract: >- 19 | Source code management - notes and ideas 20 | license: See license file 21 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | 2 | # Contributor Covenant Code of Conduct 3 | 4 | ## Our Pledge 5 | 6 | We as members, contributors, and leaders pledge to make participation in our 7 | community a harassment-free experience for everyone, regardless of age, body 8 | size, visible or invisible disability, ethnicity, sex characteristics, gender 9 | identity and expression, level of experience, education, socio-economic status, 10 | nationality, personal appearance, race, caste, color, religion, or sexual 11 | identity and orientation. 12 | 13 | We pledge to act and interact in ways that contribute to an open, welcoming, 14 | diverse, inclusive, and healthy community. 15 | 16 | ## Our Standards 17 | 18 | Examples of behavior that contributes to a positive environment for our 19 | community include: 20 | 21 | * Demonstrating empathy and kindness toward other people 22 | * Being respectful of differing opinions, viewpoints, and experiences 23 | * Giving and gracefully accepting constructive feedback 24 | * Accepting responsibility and apologizing to those affected by our mistakes, 25 | and learning from the experience 26 | * Focusing on what is best not just for us as individuals, but for the overall 27 | community 28 | 29 | Examples of unacceptable behavior include: 30 | 31 | * The use of sexualized language or imagery, and sexual attention or advances of 32 | any kind 33 | * Trolling, insulting or derogatory comments, and personal or political attacks 34 | * Public or private harassment 35 | * Publishing others' private information, such as a physical or email address, 36 | without their explicit permission 37 | * Other conduct which could reasonably be considered inappropriate in a 38 | professional setting 39 | 40 | ## Enforcement Responsibilities 41 | 42 | Community leaders are responsible for clarifying and enforcing our standards of 43 | acceptable behavior and will take appropriate and fair corrective action in 44 | response to any behavior that they deem inappropriate, threatening, offensive, 45 | or harmful. 46 | 47 | Community leaders have the right and responsibility to remove, edit, or reject 48 | comments, commits, code, wiki edits, issues, and other contributions that are 49 | not aligned to this Code of Conduct, and will communicate reasons for moderation 50 | decisions when appropriate. 51 | 52 | ## Scope 53 | 54 | This Code of Conduct applies within all community spaces, and also applies when 55 | an individual is officially representing the community in public spaces. 56 | Examples of representing our community include using an official e-mail address, 57 | posting via an official social media account, or acting as an appointed 58 | representative at an online or offline event. 59 | 60 | ## Enforcement 61 | 62 | Instances of abusive, harassing, or otherwise unacceptable behavior may be 63 | reported to the community leaders responsible for enforcement at 64 | [INSERT CONTACT METHOD]. 65 | All complaints will be reviewed and investigated promptly and fairly. 66 | 67 | All community leaders are obligated to respect the privacy and security of the 68 | reporter of any incident. 69 | 70 | ## Enforcement Guidelines 71 | 72 | Community leaders will follow these Community Impact Guidelines in determining 73 | the consequences for any action they deem in violation of this Code of Conduct: 74 | 75 | ### 1. Correction 76 | 77 | **Community Impact**: Use of inappropriate language or other behavior deemed 78 | unprofessional or unwelcome in the community. 79 | 80 | **Consequence**: A private, written warning from community leaders, providing 81 | clarity around the nature of the violation and an explanation of why the 82 | behavior was inappropriate. A public apology may be requested. 83 | 84 | ### 2. Warning 85 | 86 | **Community Impact**: A violation through a single incident or series of 87 | actions. 88 | 89 | **Consequence**: A warning with consequences for continued behavior. No 90 | interaction with the people involved, including unsolicited interaction with 91 | those enforcing the Code of Conduct, for a specified period of time. This 92 | includes avoiding interactions in community spaces as well as external channels 93 | like social media. Violating these terms may lead to a temporary or permanent 94 | ban. 95 | 96 | ### 3. Temporary Ban 97 | 98 | **Community Impact**: A serious violation of community standards, including 99 | sustained inappropriate behavior. 100 | 101 | **Consequence**: A temporary ban from any sort of interaction or public 102 | communication with the community for a specified period of time. No public or 103 | private interaction with the people involved, including unsolicited interaction 104 | with those enforcing the Code of Conduct, is allowed during this period. 105 | Violating these terms may lead to a permanent ban. 106 | 107 | ### 4. Permanent Ban 108 | 109 | **Community Impact**: Demonstrating a pattern of violation of community 110 | standards, including sustained inappropriate behavior, harassment of an 111 | individual, or aggression toward or disparagement of classes of individuals. 112 | 113 | **Consequence**: A permanent ban from any sort of public interaction within the 114 | community. 115 | 116 | ## Attribution 117 | 118 | This Code of Conduct is adapted from the [Contributor Covenant][homepage], 119 | version 2.1, available at 120 | [https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1]. 121 | 122 | Community Impact Guidelines were inspired by 123 | [Mozilla's code of conduct enforcement ladder][Mozilla CoC]. 124 | 125 | For answers to common questions about this code of conduct, see the FAQ at 126 | [https://www.contributor-covenant.org/faq][FAQ]. Translations are available at 127 | [https://www.contributor-covenant.org/translations][translations]. 128 | 129 | [homepage]: https://www.contributor-covenant.org 130 | [v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html 131 | [Mozilla CoC]: https://github.com/mozilla/diversity 132 | [FAQ]: https://www.contributor-covenant.org/faq 133 | [translations]: https://www.contributor-covenant.org/translations 134 | 135 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Source code management - notes and ideas 2 | 3 | Notes on source code management such as Git, repository architecture such as monorepo vs. polyrepo, topic flows such as trunk-based-development, and more. Work in progress. Improvement ideas welcome. 4 | 5 | Files: 6 | 7 | * [Security, auditing, compliance](security-auditing-compliance/index.md) 8 | * [Bazel/Blaze for dependencies](bazel-blaze-for-dependencies/index.md) 9 | * [SCM comparisons across organizations](scm-comparisons-across-organizations/index.md) 10 | * [SCM at Facebook](scm-at-facebook/index.md) 11 | * [SCM at Google](scm-at-google/index.md) 12 | * [Git vs. Mercurial vs. Subversion vs. Perforce et al.](git-vs-mercurial-vs-subversion-vs-perforce-et-al/index.md) 13 | * [Partial checkout](partial-checkout/index.md) 14 | 15 | See also: 16 | 17 | * [Monorepo vs. polyrepo](https://github.com/joelparkerhenderson/monorepo_vs_polyrepo) 18 | * [Trunk based development](https://trunkbaseddevelopment.com/) 19 | * [Why Google stores billions of lines of code in a single repository (2016) (acm.org)](https://dl.acm.org/citation.cfm?id=2854146) 20 | * [Scaling Mercurial at Facebook](https://code.facebook.com/posts/218678814984400/scaling-mercurial-at-facebook/) 21 | * [Monorepos: Please don’t! by Matt Klein](https://medium.com/@mattklein123/monorepos-please-dont-e9a279be011b) 22 | * [Hacker News discussion](https://news.ycombinator.com/item?id=18808909) 23 | -------------------------------------------------------------------------------- /bazel-blaze-for-dependencies/index.md: -------------------------------------------------------------------------------- 1 | # Bazel/Blaze for dependencies 2 | 3 | Bazel solves the CI/CD dependencies for mono-repos. 4 | 5 | * All build targets (buildable units) explicitly define their direct dependencies. 6 | 7 | * If you modify something, you can then run all tests for all units that transitively depend on your change. 8 | 9 | * Note: At Google, Bazel is known internally as Blaze. 10 | 11 | Bazel lets you do reverse dependency querying with the query language. 12 | 13 | * I.e. "Bazel, give me the list of targets that depend on this target that I've just modified". 14 | 15 | $ bazel query 'rdeps(//foo/my:target, //...)' 16 | 17 | * For surface level changes the target list is often quite small. For changes to core libraries, well, you run a lot of tests. 18 | 19 | * Of course, the query '//...' in the monorepo will take a long time or not work, because the target universe of "//..." is far too large. This is where other systems come in. 20 | 21 | Google CI/CD works really well. 22 | 23 | * One benefit is all tests affected by your target is run as you presubmit. 24 | 25 | * The benefit is everything is at head so things like library bugs, security bugs are all handled naturally as part of the new releases. 26 | 27 | * This usually happens twice a week for most server binaries. 28 | 29 | * Changes that break lots of projects are rolled back quite fast. Usually. 30 | 31 | See https://docs.bazel.build/versions/master/query.html#rdeps 32 | 33 | -------------------------------------------------------------------------------- /git-submodules/index.md: -------------------------------------------------------------------------------- 1 | # Git submodules 2 | 3 | IIRC the Git people seemed to reject the idea of large code bases. Or, rather, their solution was to use Git submodules. There was (and maybe is?) parts of the Git codebase that didn't scale because they were O(n). Apologies if I'm misspeaking here but I peripherally followed these discussions on HN and elsewhere years ago as someone from the outside looking in so I'm no authority on this. 4 | 5 | The problem of course is that Git submodules don't give you the benefits of a single repo and I've honestly not heard anyone say anything good about Git submodules. 6 | 7 | 8 | -------------------------------------------------------------------------------- /git-vs-hg-vs-svn-vs-perforce-et-al/index.md: -------------------------------------------------------------------------------- 1 | # Git vs. Mercurial vs. Subversion vs. Perforce et al. 2 | 3 | Many open source projects have migrated to git. 4 | 5 | A lot of companies don't use git. 6 | 7 | People switched to Git because Subversion merging continues to suck, and branching and tagging are implemented on the most inane way possible. 8 | 9 | Why have many projects switch from CVS, Subversion, etc. to Git? 10 | 11 | * Git is fundamentally built on more powerful ideas than what came before it. 12 | 13 | * Git makes it easier to branch, merge, tag, and do distributed source code. 14 | 15 | * GitHub makes it easier/cheaper to host open source projects, fork, and comment. 16 | 17 | * Standardizing on one VCS makes it easier for more people to contribute. 18 | 19 | -------------------------------------------------------------------------------- /partial-checkout/index.md: -------------------------------------------------------------------------------- 1 | # Partial check out 2 | 3 | Can you check out an individual directory without having to check out the entire repo? 4 | 5 | * Git: no 6 | 7 | * Svn: yes, and this is the default way to work 8 | 9 | * Google monorepo: yes, and this is the default way to work 10 | 11 | For example in `svn`, checking out only some subdirectory instead of entire repo is pretty much the default way how you should use it. 12 | 13 | This is a common feature in most non-DCVS. 14 | 15 | This is helpful for projects where you want to skip downloading some directories that massive, such as a game project with an art directory that is 500 GB. 16 | 17 | Notes: 18 | 19 | * While technically true due to some features of tooling, that is really only masking off part of the repo under a READ-ONLY directory. 20 | 21 | * Builds can (and usually do) depend on things that aren't part of your local checkout. 22 | 23 | * I'd say CitC is a much more accurate representation of the way Piper and blaze "expect" things to work. 24 | -------------------------------------------------------------------------------- /scm-at-facebook/index.md: -------------------------------------------------------------------------------- 1 | # SCM at Facebook 2 | 3 | Facebook engineers historically see four repos: 4 | 5 | * www 6 | * fbobjc 7 | * fbandroid 8 | * fbcode (C++, Java, Thrift services, etc) 9 | 10 | As of 2017 Facebook uses one `hg` repo (www as a possible exception, I'm unsure). The "sparse checkout" machinery disguises it, but for engineers working cross platform (e.g. React Native) it's routine to make commits that span platforms. 11 | 12 | At one point these repos were Git but for various reasons ended up being migrated to Mercurial some years ago. 13 | 14 | FB uses GraphQL at a client level and Thrift at the service level. 15 | 16 | So one pain point is that, for example, you can modify a GraphQL endpoint in one repo but its used by clients in others (ie mobile clients). There are lots of warnings about making backward-incompatible changes, some of them excessively pessimistic because deterministically showing something will break some mobile build in another repo is hard. 17 | -------------------------------------------------------------------------------- /scm-at-google/index.md: -------------------------------------------------------------------------------- 1 | # SCM at Google 2 | 3 | Google has separate repos: 4 | 5 | * Android 6 | * Chrome 7 | * ChromeOS 8 | * Google3 9 | 10 | Each repo has its own build system: Gradle, gyp/ninja, Portage, Blaze. 11 | 12 | When people talk about Google's monolithic repo they're talking about Google3. 13 | 14 | * It's important to stress that the Google mono-repo uses Perforce, not git. 15 | 16 | * Google uses git/gerrit for Android. 17 | 18 | Google3 here consists of several parts: 19 | 20 | * The source code itself, which is essentially Perforce. This includes code in C++, Java, Python, Javascript, Objective-C, Go and a handful of other minor languages. 21 | 22 | * SrcFS. This allows you to check out only part of the repo and depend on the rest via read-only links to what you need from the rest. 23 | 24 | * Blaze. Much like Bazel. This is the system that defines how to build various artifacts. All dependencies are explicit, meaning you can create a true dependency graph for any piece of code. This is super-important because of... 25 | 26 | * Forge. Caching of built artifacts. The hit-rate on this is very good and it consumes a huge amount of resources given the number of artifacts produced. Forge turns build times for some binaries from hours (even days) into minutes or even seconds. 27 | 28 | * ObjFS. SrcFS is for source files. ObjFS is for built artifacts. 29 | 30 | * Google has shown the monolithic model of source code management can scale to a repository of one billion files, 35 million commits, and tens of thousands of developers. 31 | 32 | The workflow is pretty good: 33 | 34 | * You can check out directories if you want to modify them, and just use the read only version if you don't. 35 | 36 | * You can still step through the read only code with a debugger however. 37 | 38 | Google uses protobufs for platform independence. 39 | 40 | Engineers who are working on Google3 tend to compile very little to nothing locally. 41 | 42 | * Google has spent a vast amount of effort making it so the same build and caching systems can handle C++ server code as well as Objective-C iOS app code. 43 | 44 | * There are definite issues with Google3, like the dependency graph getting so large that even reading it in and figuring out what to build is a significant performance cost and optimization issue. 45 | 46 | Engineers working on Android, Chrome, and ChromeOS, tend to compile a lot of things locally, and thus get far beefier workstations. 47 | 48 | Presubmit checks, code review processes, and rollbacks, keep things sane. 49 | 50 | A monorepo this size would simply not scale on git, at least not without huge amounts of hacks (and to be fair, Google built an entire infrastructure on top of Perforce to make their monorepo work). 51 | 52 | This used to be done manually via "gcheckout" but that's long since been replaced. Users now don't do anything but create quick throwaway clients that have the entire repo in view. 53 | 54 | Until very recently there was a versioning system for core libraries so those wouldn't typically be at HEAD (minimizing global breakage). Even that has been eliminated now and it's truly just the presubmit checks and code review process that keeps things sane. 55 | 56 | Plenty of opensource projects are in google3 (that frequently gets merged into external). 57 | 58 | * Many OSS projects live in google3: e.g. gRPC, protobufs, etc. 59 | 60 | * Projects use Copybara to import/export OSS code. https://github.com/google/copybara 61 | 62 | 63 | ### Google capabilties 64 | 65 | By [malkia](https://news.ycombinator.com/user?id=malkia) 66 | 67 | I worked at google for 2-3 years, under google3 depot. 68 | 69 | You can actually compile, and run in dev/staging certain things, inspect code, click on function, and see callsites, even "debug" from the browser. Debugging, is pretty much, if your binary have stepped through a bookmark, then it'll tell you, and you may print out locals from the scope - like vars, etc. - hard to explain in short. 70 | 71 | Then "checking out", to put in p4/svn - is not really like that - but you can find pretty much videos, docs explaining it. It's more like - you create a "workspace", "client" (piper has history from p4, so some terms are similar to perforce), and in this client - you "view" the entire depot, and changes you've made are "overlayed". Then you can submit these in a CL (much like perforce). 72 | 73 | There is also git (git5) and hg mode, but I've never used them. I've used the "CitC" one (client in the cloud), and was great as I was able to edit, and later edit, even build,... even deploy all from the browser (though prefer real IDE there - like Eclipse/IntelliJ/CLion). 74 | 75 | AFAIK, only few hundreth files are not visible to employees, and certain folks (contractors) may be limited there too. 76 | 77 | 78 | ### Google does monorepo and polyrepo 79 | 80 | By [malkia](https://news.ycombinator.com/user?id=malkia) 81 | 82 | Google's significant new project, fuchsia, is set-up as multi-git repo. For fuchsia, they use a tool called "jiri" to update the repos, previously (and maybe still in use) is the "gclient" sync tool way from depot_tools. 83 | 84 | * jiri: https://fuchsia.googlesource.com/jiri/ 85 | 86 | * gclient: https://chromium.googlesource.com/chromium/tools/depot_tools.git/+/master/gclient 87 | 88 | * depot_tools: https://chromium.googlesource.com/chromium/tools/depot_tools.git 89 | 90 | * This reflects a bit to the build system of choice, GN (used in the above), previously gyp, feels similar on the surface (script) to Bazel, but has some significant differences (gn has some more imperative parts, and it's a ninja-build generator, while bazel, like pants/bucks/please.build is a build system on it's own). 91 | 92 | * Bazel is getting there to support monorepos (through WORKSPACEs), but there are some hard problems there. 93 | -------------------------------------------------------------------------------- /scm-comparisons-across-organizations/index.md: -------------------------------------------------------------------------------- 1 | # SCM comparisons across organizations 2 | 3 | 4 | ## Facebook vs. Google 5 | 6 | By [cletus](https://news.ycombinator.com/user?id=cletus) 7 | 8 | Related: 9 | 10 | * [SCM at Facebook](scm_at_facebook.md) 11 | * [SCM at Google](scm_at_google.md) 12 | 13 | At Facebook, you can modify a GraphQL endpoint in one repo, but its used by clients in others (ie mobile clients). There are lots of warnings about making backward-incompatible changes, some of them excessively pessimistic because deterministically showing something will break some mobile build in another repo is hard. 14 | 15 | At Google, Google3 has less of these problems because the code is in the same repo. On top of that, Google has spent a vast amount of effort making it so the same build and caching systems can handle C++ server code as well as Objective-C iOS app code. Basically if you're working on Google3 you basically compile very little to nothing locally. 16 | 17 | Engineers on Android, Chrome and ChromeOS however compile a lot of things locally and thus get far beefier workstations. 18 | 19 | At FB the mobile build system doesn't seem to be as advanced in that there is a far higher proportion of local building. 20 | 21 | Just to stress, the above is just my personal experience and I hope it's taken as intended: general observations rather than complaints and definitely not arguing that one is objectively better than the other. There are simply tradeoffs. 22 | 23 | 24 | ## How long does a commit with tests and checks take? 25 | 26 | Related: 27 | 28 | * [SCM at Facebook](scm_at_facebook.md) 29 | * [SCM at Google](scm_at_google.md) 30 | 31 | Google: 32 | 33 | * Unit tests finish relatively fast. 34 | 35 | * Integration tests that bring up server tend to be slow (30-40 mins best case almost for these for projects im working on). 36 | 37 | * Mobile tests that run on emulators are the worst. 38 | 39 | * The cost of testing gets amortized two ways: you can run immediate unit tests manually on command line as the fastest signal. Then when you send for code review, presubmit runs. During code reviews you may choose to run them as you go. Eventually when you submit they run again. If there has not been any changes to your commit/cl and there is an already passing submit, it will just skip and submit. 40 | 41 | Twitter: 42 | 43 | * Generally submit queue takes 10-20 minutes, longer if you're changing a core library. 44 | 45 | * The larger factor is what kind of test has to run. Feature tests can take a while on CI if you're spinning up lots of embedded services (dependent services, MySQL, storage layers, etc). 46 | 47 | -------------------------------------------------------------------------------- /security-auditing-compliance/index.md: -------------------------------------------------------------------------------- 1 | # Security, auditing, compliance 2 | 3 | Security for source code management is especially important for auditing and compliance, such as for industry best practices, goverment regulations, sensitive codebases, high-security applications, and legal needs for [SOX](https://en.wikipedia.org/wiki/Sarbanes%E2%80%93Oxley_Act), [PCI](https://en.wikipedia.org/wiki/Payment_Card_Industry_Data_Security_Standard), [HIPAA](https://en.wikipedia.org/wiki/Health_Insurance_Portability_and_Accountability_Act), [FERPA](https://en.wikipedia.org/wiki/Family_Educational_Rights_and_Privacy_Act), [GDPR](https://en.wikipedia.org/wiki/General_Data_Protection_Regulation), etc. 4 | 5 | * [Key concepts](#key-concepts) 6 | * [Best practices](#best-practices) 7 | * [Least Privilege](#least-privilege) 8 | * [Branch Protection](#branch-protection) 9 | * [Sign commits ](#sign-commits-) 10 | * [Automate builds](#automate-builds) 11 | * [Automate versioning](#automate-versioning) 12 | * [Automate deployments](#automate-deployments) 13 | * [DevSecOps](#devsecops) 14 | * [Force code reviews](#force-code-reviews) 15 | * [Create immutable release artifacts](#create-immutable-release-artifacts) 16 | * [Use annotated tags ](#use-annotated-tags-) 17 | * [Rotate user credentials](#rotate-user-credentials) 18 | * [Protect secrets ](#protect-secrets-) 19 | * [One-way logging/monitoring](#one-way-logging-monitoring) 20 | * [Build system security ](#build-system-security-) 21 | * [Binary Reproducible Builds ](#binary-reproducible-builds-) 22 | * [Audit](#audit) 23 | * [Regulator controls](#regulator-controls) 24 | * [Regulatory controls vs. DevOps](#regulatory-controls-vs-devops) 25 | * [Implementations](#implementations) 26 | * [By zieziegabor](#by-zieziegabor) 27 | * [Encryption](#encryption) 28 | * [Wordbook](#wordbook) 29 | * [Tools](#tools) 30 | * [Security laws](#security-laws) 31 | * [Related](#related) 32 | * [Books](#books) 33 | * [Blogs](#blogs) 34 | * [Links](#links) 35 | * [Organizations](#organizations) 36 | * [Thanks](#thanks) 37 | 38 | 39 | ## Key concepts 40 | 41 | Key concepts begin with these three steps: 42 | 43 | * Establish a clear policy and also a clear process. 44 | 45 | * Use auditing, automation, encryption, monitoring, tracking, and the like. 46 | 47 | * Ensure all areas are covered by multiple responsible people. 48 | 49 | Comment: To be honest there's a whole rabbit hole you can go down when you start thinking about compliance for source code/build system/testing/infrastructure-as-code/configuration-management/orchestration code/secret management/monitoring/disaster recovery/HA system architecture. 50 | 51 | 52 | ## Best practices 53 | 54 | 55 | ### Least Privilege 56 | 57 | All repositories are restricted by default, only users who need access have access, and read/write permissions are set depending on their role. Just because they are an employee, doesn't necessarily mean they need access to the product's source code. 58 | 59 | 60 | ### Branch Protection 61 | 62 | The master branch and all release branches are restricted: no one can push directly to them. 63 | 64 | Disable all ways to change history, such as force push. 65 | 66 | 67 | ### Sign commits 68 | 69 | Use auditing & second factor trust. Can be a bit complicated/annoying to setup for a whole org though. 70 | 71 | 72 | ### Automate builds 73 | 74 | Use continuous integration (CI) automation that automatically triggers on relevant events, such as when a developer creates a pull request. 75 | 76 | Run test suites & static analysis to validate code. Only code successfully passing both can be merged. 77 | 78 | Comment: Jenkins works great when used with an automation user for accessing protected resources. 79 | 80 | Comment: If you want to use Jenkins and want to separate out production builds/deploys from dev builds/deploys, then you have a couple options, and oth of these solutions can be paired with separate credentials for each environment: 1) Separate Jenkins servers per environment, 2) Single Jenkins master (use the folder plugin for restricting user access per environment, or use dedicated build slaves for each environment e.g. prod jobs run only on prod slave. 81 | 82 | 83 | ### Automate versioning 84 | 85 | Automated versioning can be a bit complicated. 86 | 87 | We use something like CapsuleCD for versioning our libraries (pip/chef cookbooks/etc) automatically, but our core applications are automatically versioned using SNAPSHOT/SHA versions and relative tags (ie. git describe --long). Version files are embedded inside the artifacts and the version string is also embedded in the filename. 88 | 89 | Comment: We use [Artifactory](https://jfrog.com/artifactory/) as our central artifact storage, though I've heard good things about [Nexus](https://www.sonatype.com/nexus-repository-sonatype) as well. 90 | 91 | 92 | ### Automate deployments 93 | 94 | Production deployments are done in a completely automated fashion. 95 | 96 | Continuous Deployment or Continuous Delivery is your friend. CD helps for auditing, and means there's less chance for a developer/operations member to fat finger something. 97 | 98 | 99 | ### DevSecOps 100 | 101 | Automated security and vulnerability scanning is done against every PR. 102 | 103 | This includes source code analysis as well as dependency vulnerability analysis. 104 | 105 | The awesome-devsecops page has a good list of various tools that you can integrate with your existing test suites. 106 | 107 | 108 | 109 | ### Force code reviews 110 | 111 | In addition to doing code reviews, if you use Github/Enterprise Github, a `CODEOWNERS` file can be used to ensure that relevant developers are notified whenever certain areas of the codebase are touched. 112 | 113 | 114 | ### Create immutable release artifacts 115 | 116 | All builds/releases/etc that generate deployable artifacts are versioned and uploaded to a centralized artifact storage system. 117 | 118 | These versioned artifacts are concrete and cannot be change/updated once created. To update your artifact a new version must be created. 119 | 120 | 121 | ### Use annotated tags 122 | 123 | Use annotated tags rather than lightweight tags if you use Git. 124 | 125 | Annotated tags create a separate commit in the repo history, and can include user information. Also can be signed, and should never be moved. 126 | 127 | 128 | ### Rotate user credentials 129 | 130 | All user credentials used to access systems are required to be rotated on a schedule, including automation users & ssh keys. 131 | 132 | 133 | ### Protect secrets 134 | 135 | Production Secrets are not accessible to developers. They are created/rotated by Ops, and developers can point to a dev vault/secret storage system during development/testing. Only automation has access to production secrets (which are also rotated). 136 | 137 | 138 | ### One-way logging/monitoring 139 | 140 | In addition to enabling monitoring & logging for your system, make sure that your logs and metrics are immediately exported into a append-only system. Basically a bad actor should never be able to compromise your system and be able to delete any trace of their actions. 141 | 142 | 143 | ### Build system security 144 | 145 | If you're following Devops best practices, you'll probably be running a system like Concourse or Jenkins. You'll want to ensure that you keep those systems up-to-date with 0-day security patches and completely lock down access to the actual machine. A compromised Build system means all your code security is moot. 146 | 147 | 148 | ### Binary Reproducible Builds 149 | 150 | This one isn't necessarily a best practice that makes sense for all organizations, however if you're in a heavily audited industry(finance/meditech/etc), binary reproducible builds can be useful. Google's [Bazel](https://bazel.build/) might be worth looking into. 151 | 152 | 153 | ### Audit 154 | 155 | Have owners for each project, such as each GitHub project, BitBucket project, etc. 156 | 157 | Ensure there are at least two project admins for each project. 158 | 159 | Inform participants where to find the admins, in order to get a repository created, or a permission update, etc. 160 | 161 | 162 | ### Regulator controls 163 | 164 | Regulatory controls are a bit tricky to tie directly back to actual best practices - they're generally written with language such as "there must be a traceable log of all changes done to this system". That of course can be accomplished by a plain 'ol ticketing system, or by a git log on a repo, provided there is documentation on our end explaining how our process implements that required control. 165 | 166 | 167 | ### Regulatory controls vs. DevOps 168 | 169 | You may have to take a slightly anti-devops stance if your services are sensitive. In my work, one cast-iron rule is that there is no overlap between the people writing the code and the people deploying the code. 170 | 171 | This essentially means Ops/Systems have no write access to code repositories or ability to build code, and developers have no access to the systems or data in production. It sounds like some sort of historical horror story in the light of modern practices, but it may unfortunately be necessary to gain regulatory approval (which you don't mention; are you subject to any?). It also mandates the existence of development and staging environments which are fit for purpose. 172 | 173 | There is also separation of privileges, which means people who grant access are never the same ones who use that access. This is easily done in large corporate entities with central IT departments who can have that delegated to them, but it may be a challenge in a smaller organization. 174 | 175 | Use auditing so if people try to abuse the system, then there is a record of it and someone (or some system) is analyzing it to alert for unexpected behaviour. 176 | 177 | 178 | 179 | ## Implementations 180 | 181 | 182 | ### By zieziegabor 183 | 184 | https://www.reddit.com/user/zieziegabor 185 | 186 | We basically do what the Linux kernel does, with Signed-Off by: <>, except we require those to also be GPG signed(along with every regular commit also gpg signed). Jenkins(really a Makefile ran by jenkins) verifies at least 2 developer GPG signatures are used for every commit into trunk/MASTER. We white-list the GPG keys allowed to be used for signatures. 187 | 188 | Every commit user gets their own branch repo. They do whatever they want there, but we force every commit to be GPG signed. When they are ready for it to be merged, they poke another person with commit access to review and commit to the main branch repo. This also is GPG signed. 189 | 190 | This gives our developers the ability to use whatever tool(s) they want to do code-reviews, and we are not forcing anyone to use any particular tool (mostly we us kiln's code review tool, however). You just have to GPG sign your commits, and 2 diff. committers have to gpg sign commits to get it into the built product. 191 | 192 | In HG we do this by using both GPG modules (the built in one for regular commits and the 3rd party one for sign offs). 193 | 194 | 195 | ## Encryption 196 | 197 | Encryption specifics: 198 | 199 | * Use encryption on transport. 200 | 201 | * Use nfs v4 for BitBucket Data Center with kerberos auth, where nfs is the only supported protocol 202 | 203 | * Use TLS 1.2 on db connections to/from BitBucket, postgres is my favorite for this 204 | 205 | * BitBucket has Personal Tokens for automation as of 5.5+. This goes a long way towards build user passwords not getting passed around 206 | 207 | * Users should use a password protected private SSH key - using keychain and jenkins credential stores to store the password for faceless automation users 208 | 209 | * A lot of content handling rules (such as HIPAA) require encryption on disk (where possible). Comment: This rule protects the data in case someone would walk into a server room and lift the hard drive, but if you are running Server (not Data Center), you can usually rely on LUKS. 210 | 211 | * Do not make automation users with passwords shared between 8 different build systems administrators on projects... this is bad automation, but if one must, separate projects used by people from the robots 212 | 213 | * Make sure to put SSL/TLS on the Elasticsearch connection (using the buckler plugin) for BitBucket Data Center 214 | 215 | * As long as you are on a newer version of BitBucket Server, you can verify that the embedded for standalone binds to localhost only, but make sure the index is not exposed, or you are excluding anything that should not go over the wire. 216 | 217 | 218 | ## Wordbook 219 | 220 | 221 | ### Tools 222 | 223 | * [Artifactory by JFrog](https://jfrog.com/artifactory/): Enterprise Universal Artifact Manager 224 | * [Awesome DevSecOps](https://github.com/devsecops/awesome-devsecops) 225 | * [Bazel by Google](https://bazel.build/): Build and test software of any size, quickly and reliably. 226 | * [CapsuleCD](https://github.com/AnalogJ/capsulecd): a generic Continuous Delivery pipeline for versioned artifacts and libraries written in any language. 227 | * [Nexus Repository by Sonatype](https://www.sonatype.com/nexus-repository-sonatype): Expert flow control for binaries, build artifacts, and release candidates. 228 | 229 | 230 | ### Security laws 231 | 232 | * [FERPA: Family Educational Rights and Privacy Act](https://en.wikipedia.org/wiki/Family_Educational_Rights_and_Privacy_Act) 233 | * [GDPR: General Data Protection Regulation](https://en.wikipedia.org/wiki/General_Data_Protection_Regulation) 234 | * [HIPAA: Health Insurance Portability and Accountability Act](https://en.wikipedia.org/wiki/Health_Insurance_Portability_and_Accountability_Act) 235 | * [PCI: Payment Card Industry Data Security Standard](https://en.wikipedia.org/wiki/Payment_Card_Industry_Data_Security_Standard) 236 | * [SOX: Sarbanes Oxley Act](https://en.wikipedia.org/wiki/Sarbanes%E2%80%93Oxley_Act) 237 | 238 | 239 | ## Related 240 | 241 | 242 | ### Books 243 | 244 | * [The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations](https://smile.amazon.com/DevOps-Handbook-World-Class-Reliability-Organizations/dp/1942788002/ref=sr_1_1) 245 | 246 | 247 | ### Blogs 248 | 249 | * [Devops for Startups & Small Teams by Jason Kulatunga at Sparktree](https://blog.thesparktree.com/devops-for-startups) 250 | 251 | 252 | ### Links 253 | 254 | * [GitHub special files and paths](https://github.com/joelparkerhenderson/github_special_files_and_paths) 255 | 256 | * [When it comes to managing repositories for highly sensitive/regulated code bases, what are some best practices that can be implemented with regards to code reviews, approvals, and deployment to production?](https://www.reddit.com/r/devops/comments/8gmf5m/when_it_comes_to_managing_repositories_for_highly/) 257 | 258 | 259 | ### Organizations 260 | 261 | * [Defense Information Systems Agency (DISA)](https://disa.mil/) 262 | * [Information Assurance Support Environment (IASE)](https://iase.disa.mil/) 263 | * [Security Technical Implementation Guides (STIGs)](https://iase.disa.mil/stigs) 264 | * [DISA IASE STIGs Master List](https://iase.disa.mil/stigs/Pages/a-z.aspx) 265 | 266 | 267 | ### Thanks 268 | 269 | * Special thanks to https://www.reddit.com/user/analogj 270 | 271 | 272 | --------------------------------------------------------------------------------