├── README.md ├── folders.md └── tools.md /README.md: -------------------------------------------------------------------------------- 1 | 2 | ## Introduction 3 | 4 | A work in progress on documenting some standards for GitOps and Kubernetes that I am using. Despite the repo being called "standards" This is not meant to be an authoritative document, I firmly believe that different organizations will need to define their own standards based on their organization structure, processes, etc. 5 | 6 | This document is reflective of my personal preferences and the standards I use in my various gitops repos. I reserve the right to make changes as I continue my journey :) 7 | 8 | ## Documents 9 | 10 | * [Tools](tools.md) - Thoughts on various manifest and gitops tools 11 | 12 | * [Folders](folders.md) - Folder structure 13 | -------------------------------------------------------------------------------- /folders.md: -------------------------------------------------------------------------------- 1 | # GitOps Folder Structure 2 | 3 | ## Introduction 4 | 5 | Many moons ago I began my GitOps journey with kustomize followed shortly thereafter with adding ArgoCD after being introduced to it by my colleague Andrew Pitt. When I started working with kustomize one thing I looked for was a set of standards around practices and folder layout in order to apply some consistency to the work I was doing. Unfortunately I didn't find much beyond overly simplistic suggestions that didn't really work, at least for me, once I scaled them out to more real world situations. 6 | 7 | As a result I ended up crafting my own set of standards which I'm still using to this day and I thought I'd share it more broadly. Now I am absolutely not saying this is the end all, be all of standards. I think standards are very dependent on the nature of the applications, the organizational structure, the development methodology being used, etc. I don't think there is a standard that is right for everyone, so in this case this is something that works for me and could maybe be useful to others as a data point when crafting there own standards. 8 | 9 | As an aside, my wife knows I have a love of laptop bags because I always feel like each bag is a little better then one before in terms of structure, layout, pockets, etc. My standards here are pretty much the same, I reserve the right to change it if I find something better. I would also love to get input from others in terms of what works for them, so feel free to comment below. 10 | 11 | ## Principles 12 | 13 | When formulating GitOps standards, it is very important to establish a set of principles as a baseline that all other standards must follow and adhere too. The set of principles being followed here include: 14 | 15 | * __Do__ separate code (i.e. java, python, etc) from manifests (i.e. yaml) into different repos. 16 | * __Do__ minimize yaml duplication, no copy paste 17 | * __Do__ support two axis of configuration: clusters and environments (prod, test, dev, etc) 18 | * __Do__ be able to split individual cluster and environment configurations into separate repos as needed to support existing organizational practices (i.e. separate production configuration in a different repo from test and dev being the most common example) 19 | * __Do__ prefer a multi-folder and/OR multi-repo structure over multi-branch, i.e do not use branching to hold different sets of files (i.e. dev in one branch, test in another). This does not preclude the use of branches for features, this is stating that there should not be permanent branches for clusters or environments. 20 | * __Do__ minimize specific gitops tool (ArgoCD, ACM, etc) dependencies as much as possible 21 | * __Do__ put dependent applications manifests in the same manifests repo when managed by the same team. A microservice or 3 tier app that is composed of multiple deployments and managed by the same team would likely be in the same repo. __Do not__ put independent applications or applications managed by different teams in the same repo. 22 | 23 | ## More On Why Not Environment Branches? 24 | 25 | So in gitops you sometimes see organizations using *permanent* branches to represent different environments. In these cases you have a dev branch for the dev environment, a test branch for the test environment, etc. 26 | 27 | This often seems like an ideal way to do things, promoting between environments simply comes a matter of merging from lower environment branches to higher environment branches. However in practice it can be quite challenging for the following reasons: 28 | 29 | * There are often many files that are environment specific and should either not be merged between environments or need to be named uniquely to avoid collisions 30 | * Typically the 1:1 branch to environment works best when the manifests are identical across all branches, tools like kustomize do not fit into this pattern 31 | * In a microservices world, a one branch per environment will quickly lead to an explosion of branches which again becomes difficult and cumbersome to maintain 32 | * Difficult to have a unified view of cluster state across all environments since the state is stored in separate branches. 33 | 34 | So in short I personally much prefer a single branch style with multiple folders to represent environments and clusters as we will see below. 35 | 36 | Obviously this does not preclude using branches for updates, PRs, etc but these branches should be short lived, temporary artifacts to support development and not permanent fixtures. 37 | 38 | ## Assumptions 39 | 40 | * As per the [tools standards](https://github.com/gnunn-gitops/standards/blob/master/tools.md), this folder structure is heavily dependent on [kustomize](https://kustomize.io). No thought is given to Helm or other alternatives at this time. 41 | * A lesser used feature of kustomize is its ability to leverage remote repos, i.e. specify a base or overlay from a separate repo. This can be used to isolate environmental configurations in separate repos without duplicating yaml. You can read more about this feature [here](https://github.com/kubernetes-sigs/kustomize/blob/master/examples/remoteBuild.md). 42 | * While the initial structure was focused on Application use cases, I've found it to work well for cluster configuration use cases as well. 43 | * My focus is on [OpenShift](https://www.openshift.com/), I have not vetted anything in this document for other kubernetes distributions however I expect this would work similarly across any distribution. 44 | 45 | ## Repository Organization 46 | 47 | I am a fan of having one or or more catalog repositories to hold common elements that will be re-used across teams and other repositories. You can see this in action with the [Red Hat Canada Catalog](https://github.com/redhat-canada-gitops/catalog) repository where colleagues and I maintain a common set of components that we re-use across our individual repositories. 48 | 49 | This is made possible by a great but under-utilized feature of kustomize that enables it to reference remote repositories as a base or resource and then patch it as needed to meet your specific requirements. The key to making this work successfully is to ensure that when you reference the common repository you do so via a tag or commit ID. Not doing this means any time there is an update to the common repo you will automatically get that change deployed by your gitops tool. Using a tag or commit ID means you control when newer versions are brought in for application by updating it in your git repo. 50 | 51 | Note: As an FYI realize in my repos I'm very bad at following this practice of using a tag/commit ID when referencing remote repos, don't be Gerald and do the right thing :) 52 | 53 | While in Red Hat Canada we have one repository that covers everything, in many organizations it will be typical to have a few different common repositories maintained by different teams. For example, the operations team may have a common repository for cluster configuration whereas the application architects may maintain a common repository for application components (Nexus, Sonarqube, frameworks, etc). 54 | 55 | For Application repositories. in general you should align your repositories along application and team boundaries. For example, if I have an application that consists of a set of microservices where team A manages one microservice and team B manages a different one then this is best done as two different repositories in my opinion. If a team is maintaining multiple applications then again this is likely different repositories, one for each application. 56 | 57 | For cluster configuration repositories, I would lean towards having different repositories for each cluster with a common repo for shared components and configuration. Having said that, if you look at my cluster-config repo referenced below I am only using a single repo however my use case is somewhat different then most organizations. 58 | 59 | ## Folder Layout 60 | 61 | Below is the folder standard I use in my own repos, it is relatively opinionated and I am definitely not saying it is the one true way. However given the paucity of documentation in this area I felt it would be useful to put it out there for others as a reference. 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 83 | 84 | 85 | 86 | 87 | 94 | 95 | 96 | 97 | 98 | 104 | 105 | 106 | 107 | 108 | 110 | 111 | 112 | 113 | 114 | 116 | 117 | 118 | 119 | 120 | 124 | 125 | 126 | 127 | 128 | 130 | 131 | 132 | 133 | 134 | 136 | 137 | 138 | 139 | 140 | 142 | 143 | 144 | 145 | 146 | 148 | 149 | 150 | 151 | 152 | 158 | 159 | 160 | 161 | 162 | 164 | 165 | 166 | 167 | 168 | 170 | 171 | 172 | 173 | 174 | 176 | 177 | 178 | 179 | 180 | 182 | 183 | 184 | 185 | 186 | 191 | 192 | 193 | 194 | 195 | 197 | 198 | 199 | 200 | 201 | 206 | 207 | 208 | 209 | 210 | 212 | 213 | 214 | 215 | 216 | 222 | 223 | 224 | 225 | 226 | 234 | 235 | 236 | 237 | 238 | 242 | 243 | 244 | 245 | 246 | 247 | 248 | 249 | 250 | 251 | 252 | 253 | 254 | 255 | 256 | 257 | 258 | 259 | 260 | 261 | 262 | 263 | 264 | 265 | 266 | 267 | 268 | 269 | 270 | 271 | 277 | 278 | 279 | 280 | 281 | 282 | 283 | 284 | 285 | 286 | 287 | 288 | 289 | 290 | 291 | 292 | 293 | 294 | 295 | 296 | 297 | 298 | 299 | 300 | 301 | 302 | 303 | 304 | 305 | 306 | 307 | 308 | 309 | 310 | 311 | 312 | 313 | 314 | 315 | 316 | 317 | 318 | 319 | 320 | 321 | 327 | 328 | 329 | 330 |
FolderComments
0├bootstrap 78 |
    79 |
  • The minimal yaml required to bootstrap the entity into a cluster. This entity could be an application, cluster configuration or something else.
    • 80 |
  • Generally this will be an Argo CD App of Apps, an ApplicationSet or something of that nature that will in turn load the rest of the components.
    • 81 |
82 |
0├components 88 |
    89 |
  • Provides yaml for all required apps, pipelines, and gitops tools.
    • 90 |
  • Generally I prefer to have no buildconfigs (i.e. for s2i) in app folders, those should be in tekton or jenkins pipelines folder
    • 91 |
  • Avoid creating namespaces in components, this should be done in environments or clusters folders. 92 |
93 |
1├──apps 99 |
    100 |
  • No distinction between apps (i.e. code we write) versus services (databases, messaging systems, etc). It’s an artificial distinction IMHO
    • 101 |
  • Having said that, if your organization feels strongly about services, have peer folder to apps called services
    • 102 |
103 |
2├────{appname1} 109 |
3├──────base 115 |
3├──────overlays 121 |
    122 |
  • Overlays related to variations (i.e. HA versus non-HA) as well as environment oriented overlays. Cluster specific overlays should not be here, instead they will be in the `/clusters` folder.
    • 123 |
2├────{appname2} 129 |
3├──────base 135 |
1├──tekton 141 |
2├────pipelines 147 |
3├──────{pipeline1}/base 153 |
    154 |
  • Includes pipeline plus all artifacts needed for pipeline not called out in separate folders. Items like PVCs for workspaces, buildconfigs used by the pipeline, etc go here
    • 155 |
  • Pipelines may but do not need to map 1:1 to apps
    • 156 |
157 |
3├──────{pipeline2}/base 163 |
2├────pipelineruns 169 |
3├──────{pipelinerun1}/base 175 |
3├──────{pipelinerun2}/base 181 |
2├────tasks 187 |
    188 |
  • All custom tasks go here, no distinction between shared tasks and pipeline specific tasks. Today’s pipeline specific task is tomorrow’s shared task
    • 189 |
190 |
3├──────base 196 |
1├──jenkins 202 |
    203 |
  • If you use Jenkins, I haven't spent much time on Jenkins layout since I'm mostly using tekton these days
    • 204 |
205 |
2├────pipelines 211 |
1├──argocd 217 |
    218 |
  • An optional folder for argocd applicationsthat may be required over and above what is in the bootstrap folder. Most commonly needed when using the App of App pattern with individually defined applications rather then an applicationset.
    • 219 |
  • In general, put things here if you find you are duplicating argo cd manifests in the bootstrap folder
    • 220 |
221 |
0├environments 227 |
    228 |
  • Optional folder that provides additional environment specific kustomization that is separate and distinct from cluster configuration. While components can have environment specific overlays sometimes you need to aggregate different components for a complete application. I like doing this here versus coming up with something artificial in components.
    • 229 |
  • I also find this useful to support demos which is not a typical use case. As I tend to deploy the same environments to multiple clusters to support these demos having an environments folder makes sense.
    • 230 |
  • Must Inherit from components only, absolutely not permitted to inherit from clusters
    • 231 |
  • When aggregating multiple components namespaces should be created here not in component overlays.
    • 232 |
233 |
1├──overlays 239 |
    240 |
  • Note that names of overlays are arbitrary, use what works for you.
    • 241 |
2├────dev
2├────test
2├────prod
2├────cicd
2├────tools
0├clusters 272 |
    273 |
  • Cluster specific overlays go here
    • 274 |
  • Clusters can inherit from environments and components
    • 275 |
276 |
1├──overlays
2├────{cluster1}
3├──────dev
3├──────test
3├──────cicd
3├──────tools
2├────{cluster2}
3├──────prod
0├tenants/{team} 322 |
    323 |
  • Only applicable to cluster configuration situations, tenants represent the different teams/applications/etc in a multi-tenant cluster.
    • 324 |
  • This is where you define cluster level resources (namespaces, quotas, limitranges, operators to install, etc) that are needed by the teams to do their work but typically require cluster-admin rights to provision
    • 325 |
326 |
331 | 332 | ## Promoting Manifest Changes 333 | 334 | A key question in any gitops scenario is how to manage promotion of changes in manifests between different environments and clusters. This process is heavily dependent on the structure and processes of the organization, however it is possible to define some basic characteristics that we are looking for as follows: 335 | 336 | * Since every overlay depends on the base manifests, every change in the manifests needs to flow through the environments in hierarchical order, i.e. (dev > test > prod). We do not want a change to a base flowing to all environments simultaneously. 337 | * To prevent changes in manifests flowing directly to environments, the state of environments and clusters needs to be pinned in git (i.e. commit revision or tag). 338 | * Changes to environments/clusters can flow directly to the target environment. i.e. a direct change to the prod overlay can flow directly to prod without a promotion process. However given the structure of our repo these direct changes should be rare (i.e. prod specific secrets, etc) and limited to emergencies. 339 | 340 | So as stated above, we need to tie specific environments to specific revisions so that changes in the repo can be promoted in a controlled manner following a proper SDLC process. Both the various GitOps tools (ArgoCD, Flux, ACM, etc) and Kustomize support referencing specific commits in a repo, as a result there are various options we have for managing environment promotions. 341 | 342 | Note that using branches is implicit in these options but not discussed directly. As per the Why Not Branches section above, the intent is for short-lived branches to be created for revisions and merged back to trunk. So managing revisions is really about tracking the appropriate revision in trunk. 343 | 344 | #### Option 1 - Manage revisions in the GitOps Tool 345 | 346 | In this option we deploy each environment as an independent entity in the GitOps tool and tie each environment to a specific revision. So in ArgoCD, we would tie the application object for the Dev environment to one revision, the Test environment to another revision, etc. 347 | 348 | When we are ready to promote a change, we simply update the revision in the GitOps tool to reference the appropriate revision or tag in the repo. 349 | 350 | This is simple to do however it does require active management of the GitOps tools entities. 351 | 352 | #### Option 2 - Manage revisions in kustomize 353 | 354 | In this option each environment we deploy uses kustomize to manage the git revision. By default you tie kustomize to the local directory, i.e: 355 | 356 | ``` 357 | apiVersion: kustomize.config.k8s.io/v1beta1 358 | kind: Kustomization 359 | 360 | namespace: product-catalog-dev 361 | 362 | bases: 363 | - ../../../manifests/app/database/base 364 | - ../../../manifests/app/server/base 365 | - ../../../manifests/app/client/base 366 | ``` 367 | 368 | However kustomize supports remote references so you can also reference the bases remotely with specific git revisions: 369 | 370 | ``` 371 | apiVersion: kustomize.config.k8s.io/v1beta1 372 | kind: Kustomization 373 | 374 | namespace: product-catalog-dev 375 | 376 | bases: 377 | - https://github.com/gnunn-gitops/product-catalog/manifests/app/database/base?ref=9769ad7 378 | - https://github.com/gnunn-gitops/product-catalog/manifests/app/server/base?ref=9769ad7 379 | - https://github.com/gnunn-gitops/product-catalog/manifests/app/client/base?ref=9769ad7 380 | ``` 381 | 382 | In this way we can promote changes across multiple environments by simply updating the reference accordingly. 383 | 384 | This approach provides very explicit control over the bases at the slight cost of having kustomize perform additional clones of the repo. 385 | 386 | #### Option 3 - Hybrid (Do Both) 387 | 388 | In this option we combine Options #1 and #2 for maximum control. 389 | 390 | #### Recommendation 391 | 392 | I don't have strong feelings at this point but my personal leaning is towards Hybrid. 393 | 394 | ## Examples 395 | 396 | Here are a couple of repositories where you can see this standard in action: 397 | 398 | * [Product Catalog](https://github.com/gnunn-gitops/product-catalog). This is a three tier application (front-end, back-end and database) deployed using GitOps with ArgoCD (or ACM) and kustomize. It deploys three separate environments (dev, test and prod) along wth Tekton pipelines to build the front-end and back-end applications. It also deploys a grafana instance for application monitoring that ties into OpenShift's [user defined monitoring](https://docs.openshift.com/container-platform/4.6/monitoring/enabling-monitoring-for-user-defined-projects.html). 399 | * [Cluster Configuration](https://github.com/gnunn-gitops/cluster-config). This repo shows how I configure my OpenShift clusters using GitOps with ArgoCD. It configures a number of things including certificates, authentication, default operators, console customizations, storage and more. 400 | 401 | I also highly recommend checking out the [Red Hat Canada GitOps](https://github.com/redhat-canada-gitops) organization as well. These repos include a default installation of the excellent [ArgoCD](https://github.com/redhat-canada-gitops/argocd) operator as well as a [catalog](https://github.com/redhat-canada-gitops/catalog) of tools and applications deployed with kustomize. 402 | 403 | ## Acknowledgements 404 | 405 | I'd like to thank Andrew Pitt who has led the way on lot of the GitOps stuff in our group, he built the ArgoCD installation above as well as a substantial portion of the catalog items. 406 | -------------------------------------------------------------------------------- /tools.md: -------------------------------------------------------------------------------- 1 | # GitOps Tools 2 | 3 | ## Introduction 4 | 5 | There are a variety of tools that can be used as part of a GitOps process. In addition to the GitOps tool itself (ArgoCD, Flux, RHACM, etc) there are a variety of ways to manage the manifest yaml for our applications including Ansible, OpenShift Templates, Helm and Kustomize. 6 | 7 | In this document we will discuss them briefly and then provide some recommendations at the end of the document. 8 | 9 | ## Manifest Tools 10 | 11 | There are a variety of tools to manage the yaml that will get deployed into your cluster. At a high level, here are the tools I typically see being used at my customers: 12 | 13 | __Ansible__. Great tool for managing infrastructure as code across a wide variety of infrastructure. Includes a *k8s* module that enables Ansible to easily integrate with kubernetes and uses jinja2 templates for manifests. 14 | 15 | While Ansible is great automation tool, I personally prefer to use native *k8s* tools where possible. Also like Helm it relies on templating which means you cannot directly apply yaml in your git repos when doing iterative development. 16 | 17 | Finally none of the common GitOps tools support Ansible so it is really only an option if you using Ansible for your kubernetes GitOps which is generally not recommended. Absolutely use Ansible for infrastructure gitops (i.e. VMs, cluster provisioning, etc) but when dealing with kubernetes manifests there are better options. 18 | 19 | __OpenShift Templates__. OpenShift templates pre-dates a lot of the tools on this list, it was released in OpenShift 3.0 when very few tools for managing managing manifests existed. OpenShift Templates are very easy to develop and work with, however like other templating solutions it's challenging to use in an iterative development process. 20 | 21 | Similar to Ansible, none of the existing DevOps as far as I know support OpenShift Templates making it a non-starter. 22 | 23 | __Helm__. Helm is a package manager for Kubernetes and OpenShift 4 now supports it natively. Helm provides charts which can be deployed into OpenShift using the Helm CLI or via the OpenShift web console. These charts contain templates along with values that can be substituted when the chart is deployed to control the yaml that is applied. A neat feature of the charts is that you can have a chart that is uses other charts as dependencies. 24 | 25 | Personally I think Helm's sweet spot is as a package manager, i.e. similar to RPMs, and is overkill for most enterprise teams. Additionally charts can quickly become very complex and difficult to maintain for anyone other then the original author. Lastly it uses templates which again is challenging for iterative development but easier then other templating solutions with the capabilities that Helm provides. 26 | 27 | Finally from an application/package manager perspective I feel like I'm getting that functionality out of my gitops tool and Helm is not providing a lot of value add here. 28 | 29 | All GitOps tools support working with Helm charts. 30 | 31 | __jsonnet/ksonnet__. No experience with this but I do know some folks quite like this approach, however personally I've never seen it used at any of my customers. 32 | 33 | Most GitOps tools support jsonnet/ksonnet. 34 | 35 | __kustomize__. Kustomize is built into Kubernetes as of 1.14 which means there is full support for it in your kubernetes or OpenShift environment out of the box. Kustomize is not a templating framework, it is a patching framework that works through the concept of inheritance. _Bases_ define the base level yaml for an application and _overlays_ inherit from those bases or other overlays with patches to differentiate as needed. 36 | 37 | Since kustomize doesn't use templating it's very easy to follow and understand what it is doing keeping initial complexity low. However because it relies on inheritance getting a structure that works for your team can be an involved process. 38 | 39 | All GitOps tools support kustomize since it is built into kubernetes. 40 | 41 | __Recommendation__. My personal recommendation is kustomize, it has worked extremely well for me so far and does what I need. All documents in this repo assume that kustomize is being used. 42 | 43 | ## GitOps Tools 44 | 45 | There are a variety of GitOps tools available and I will not provide information or make a recommendation here simply because I do not have sufficient experience with them. I personally use ArgoCD and Red Hat's Advanced Cluster Management (ACM) and am satisfied with both. Disclaimer I am a Red Hat employee. 46 | 47 | I do not have experience with Flux but it is a popular tool as well. 48 | 49 | ## Secrets 50 | 51 | One of the challenges of GitOps is managing secrets since you typically do not want to have your unprotected secrets in a git repo. There are typically two different broad approaches for dealing with this: 52 | 53 | 1. Externalize the secrets from git using a product like Hashicorp's Vault 54 | 2. Encrypt your secrets in git (Sealed Secrets, Sops, etc) 55 | 56 | I have opted for using #2 and Sealed Secrets as it is a simple approach that meets my basic needs. If you have Vault or some other tool available in your enterprise to externalize secrets I would recommend using that. 57 | --------------------------------------------------------------------------------