├── .gitignore ├── LICENSE ├── README.md ├── consul └── config │ └── server.json ├── dc.sh ├── dlogs.sh ├── docker-compose.primary.yml ├── docker-compose.yml ├── dr ├── consul │ └── config │ │ └── server.json ├── docker-compose.dr.yml ├── grafana │ ├── config.monitoring │ └── provisioning │ │ ├── dashboards │ │ ├── Docker Prometheus Monitoring-1571332751387.json │ │ └── dashboard.yml │ │ └── datasources │ │ └── datasource.yml ├── prometheus │ ├── alert.rules │ ├── prometheus.yml │ └── vault-metrics.hcl └── vault │ ├── api │ └── .keep │ └── config │ ├── vault01.hcl │ ├── vault02.hcl │ └── vault03.hcl ├── grafana ├── config.monitoring └── provisioning │ ├── dashboards │ ├── Docker Prometheus Monitoring-1571332751387.json │ └── dashboard.yml │ └── datasources │ └── datasource.yml ├── pki └── config │ └── vault-pki.hcl ├── prem └── Dockerfile ├── prometheus ├── alert.rules ├── prometheus.yml └── vault-metrics.hcl ├── proxy ├── docker-compose.yml └── haproxy.cfg ├── secondary ├── consul │ └── config │ │ └── server.json ├── docker-compose.secondary.yml ├── grafana │ ├── config.monitoring │ └── provisioning │ │ ├── dashboards │ │ ├── Docker Prometheus Monitoring-1571332751387.json │ │ └── dashboard.yml │ │ └── datasources │ │ └── datasource.yml ├── prometheus │ ├── alert.rules │ ├── prometheus.yml │ └── vault-metrics.hcl └── vault │ ├── api │ └── .keep │ └── config │ ├── vault01.hcl │ ├── vault02.hcl │ └── vault03.hcl ├── vault ├── api │ └── .keep └── config │ ├── vault01.hcl │ ├── vault02.hcl │ └── vault03.hcl └── yapi ├── consul ├── agent-token-policy.json ├── test_01-acl-token.tavern.yaml └── vault-token-policy.json ├── pki ├── 01-enable-pki.yaml ├── 02-vault-certificates.yaml └── test_pki_consul.tavern.yaml ├── tickets ├── test_control_group_20201.tavern.yaml ├── test_ldap_20163.tavern.yaml └── test_ldap_anon.tavern.yaml └── vault ├── 01-init.yaml ├── 02-unseal.yaml ├── 03-replication_enable_primary.yaml ├── 04-replication_secondary_token.yaml ├── 05-replication_activate_secondary.yaml ├── 06-replication_generate_root_secondary.yaml ├── 07-dr-enable.yaml └── optional-revoke-secondary.yaml /.gitignore: -------------------------------------------------------------------------------- 1 | .vagrant 2 | *~ 3 | prem/vault 4 | data/ 5 | .pytest_cache 6 | yapi/__pycache__ 7 | __pycache__ 8 | .vscode 9 | secondary/consul/data 10 | secondary/vault/logs 11 | secondary/vault/api/*.json 12 | dr/consul/data 13 | dr/vault/logs 14 | dr/vault/api/*.json 15 | vault/logs 16 | vault/api/*.json 17 | consul/data 18 | pki/ca/*.pem 19 | pki/ca/*.csr -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2019 Rodrigo D. L. 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | - [What](#what) 2 | - [Why](#why) 3 | - [How](#how) 4 | - [Requirements](#requirements) 5 | - [Networks or Regions](#networks-or-regions) 6 | - [Communication between them](#communication-between-them) 7 | - [Initial configuration](#initial-configuration) 8 | - [Install yapi](#install-yapi) 9 | - [Start the clusters](#start-the-clusters) 10 | - [Start replication](#start-replication) 11 | - [Check that replication is up](#check-that-replication-is-up) 12 | - [Commands supported](#commands-supported) 13 | - [How `dc.sh` works](#how-dcsh-works) 14 | - [How to manually configure performance replication](#how-to-manually-configure-performance-replication) 15 | - [View the full compose template for a given cluster](#view-the-full-compose-template-for-a-given-cluster) 16 | - [Initialization of Vault](#initialization-of-vault) 17 | - [Unsealing](#unsealing) 18 | - [Troubleshooting](#troubleshooting) 19 | - [Useful commands](#useful-commands) 20 | - [Exposed ports: local -> container](#exposed-ports-local---container) 21 | - [Primary](#primary) 22 | - [Secondary (DR primary)](#secondary-dr-primary) 23 | - [DR Secondary](#dr-secondary) 24 | - [Proxy](#proxy) 25 | - [TODO](#todo) 26 | - [Done](#done) 27 | # What 28 | A way to create multiple Vault clusters and setup different types of Replication between them as close as possible to the "Vault reference architecture" https://learn.hashicorp.com/vault/operations/ops-reference-architecture 29 | 30 | Using the 3 Region setup architecture: 31 | 32 | ![3 regions](https://d33wubrfki0l68.cloudfront.net/f4320f807477cdda5df25f904eaf3d7c9cfd761d/e6047/static/img/vault-ra-full-replication_no_region.png) 33 | 34 | Each `Region` is composed of a docker network and has a set of Vault and Consul clusters. 35 | 36 | They can only communicate with each other using a `proxy` (sometimes incorrectly called Load Balancer) in this case `HAProxy`. 37 | - Region 1 contains the `Primary` Vault cluster, configured following the [Deployment Guide](https://learn.hashicorp.com/vault/day-one/ops-deployment-guide) 38 | - Region 2 contains the `Secondary` Vault cluster, configured as [Performance Replication](https://learn.hashicorp.com/vault/operations/ops-replication) 39 | - Region 3 will contain a `DR` Vault cluster configured as [Disaster Recovery](https://learn.hashicorp.com/vault/operations/ops-disaster-recovery) 40 | 41 | All regions have a Consul cluster for storage and every Vault node has a `Consul Agent` in a different container. 42 | 43 | # Why 44 | To be able to easily setup and test different configuration and features of a full fledge Vault and Consul cluster setup. 45 | 46 | # How 47 | 48 | ## Requirements 49 | * Docker 50 | * Docker-compose: 51 | * Scheduling of containers 52 | * Overlaying of configuration to avoid duplication 53 | * [Yapi-ci](https://github.com/bruj0/yapi) 54 | * Initialization 55 | * API management 56 | * Access to Premium or Pro version of Vault 57 | * `vault` and `jq` binaries installed in the $PATH 58 | 59 | 60 | To talk to Vault we will use `Yapi-ci`, this is a yaml file where we define how the API call will look like. 61 | 62 | The benefits over a bash script using `curl` is that we can manipulate the response and its very clear what the call does. 63 | 64 | You can take a look at them under `yapi/vault`, for example: 65 | ```yaml 66 | --- 67 | test_name: 01-Enable primary 68 | stages: 69 | - name: 01-Enable primary 70 | request: 71 | url: "{env_vars.VAULT_ADDR}/v1/sys/replication/performance/primary/enable" 72 | method: POST 73 | headers: 74 | X-Vault-Token: "{env_vars.VAULT_TOKEN}" 75 | json: 76 | primary_cluster_addr: "{env_vars.PRIMARY_HAPROXY_ADDR}" 77 | response: 78 | status_code: 200 79 | ``` 80 | 81 | 82 | ## Networks or Regions 83 | It uses 3 networks: 84 | * vault_primary 85 | * vault_secondary 86 | * vault_dr 87 | 88 | Each network hosts this configuration of servers (currently only 3 Consul nodes): 89 | 90 | ![Single availability zone](https://d33wubrfki0l68.cloudfront.net/177ba67519ffb1c802f5cb699c0b70cb40533ab6/63838/static/img/vault-ra-1-az.png) 91 | 92 | ## Communication between them 93 | 94 | It uses an HAProxy instance in TCP mode by accessing the IP trough consul SRV DNS record but this can be changed to any type of service discovery supported by HAProxy. 95 | 96 | ![proxy](https://d33wubrfki0l68.cloudfront.net/b2d787641bf2dda0a8a1abf691cd9723a9c0ed8c/7b419/static/img/vault-ref-arch-9.png) 97 | 98 | 99 | ## Initial configuration 100 | 101 | - Create the docker networks 102 | ```bash 103 | $ docker network create {vault_primary,vault_secondary,vault_dr} 104 | ``` 105 | ## Install yapi 106 | 107 | ```bash 108 | $ pip install -U yapi-ci 109 | $ yapi --version 110 | 0.1.6 111 | ``` 112 | 113 | ## Start the clusters 114 | ```bash 115 | $ CLUSTER=primary ./dc.sh up 116 | $ ./dc.sh proxy start 117 | $ CLUSTER=secondary ./dc.sh up 118 | ``` 119 | 120 | ## Start replication 121 | If you want to do it manually go [here](#how-to-manually-configure-performance-replication): 122 | 123 | ```bash 124 | $ CLUSTER=primary ./dc.sh enable_secondary 125 | ``` 126 | ## Check that replication is up 127 | ```bash 128 | $ env CLUSTER=primary ./dc.sh cli vault read sys/replication/performance/status 129 | Key Value 130 | --- ----- 131 | cluster_id 2cc7aad6-026a-9620-6f0d-1e8fa939a11e 132 | known_secondaries [secondary] 133 | last_reindex_epoch 0 134 | last_wal 247 135 | merkle_root d85e48c2ec44b1e6ba6671773ea26d836b64ed09 136 | mode primary 137 | primary_cluster_addr https://172.25.0.2:8201 138 | state running 139 | 140 | $ env CLUSTER=secondary ./dc.sh cli vault read sys/replication/performance/status 141 | Key Value 142 | --- ----- 143 | cluster_id 2cc7aad6-026a-9620-6f0d-1e8fa939a11e 144 | known_primary_cluster_addrs [https://172.24.0.8:8201 https://172.24.0.9:8201 https://172.24.0.10:8201] 145 | last_reindex_epoch 1574351423 146 | last_remote_wal 0 147 | merkle_root d85e48c2ec44b1e6ba6671773ea26d836b64ed09 148 | mode secondary 149 | primary_cluster_addr https://172.25.0.2:8201 150 | secondary_id secondary 151 | state stream-wals 152 | ``` 153 | 154 | ### Commands supported 155 | All the commands read the `CLUSTER` variable to determine where is the operation going to run on. 156 | 157 | Example: 158 | ```bash 159 | $ env CLUSTER=primary ./dc.sh cli vars Exporting variables for primary 160 | export VAULT_ADDR="http://127.0.0.1:9201" 161 | export VAULT_DATA="./vault/api" 162 | export VAULT_TOKEN="s.YFfiUgyPCZAtJIQ55NtvVa2K" 163 | ``` 164 | - `help` : This help 165 | - `config`: Will execute `docker-compose config` with the proper templates 166 | - `up`: This will start the Vault and Consul cluster up for the specified type of cluster by doing a `docker-compose up -d` 167 | 168 | - `down`: It will do a `docker-compose down` with the correct template 169 | 170 | - `wipe`: Will wipe ALL the consul data files, make sure to do it after `down` 171 | 172 | - `restart` 173 | - vault 174 | - consul 175 | - proxy 176 | 177 | - `cli`: This will set the variables `VAULT_TOKEN` from `vault/api/init.json` and `VAULT_ADDR` to the port of the first node of the selected cluster. 178 | - `vars`: Prints variables for the given cluster 179 | - `vault ` 180 | - `yapi