├── .circleci └── config.yml ├── .github └── pull_request_template.md ├── .gitignore ├── .tool-versions ├── CODEOWNERS ├── LICENSE.md ├── README.md ├── bin ├── migrate_resources-11.py ├── migrate_resources-12.py └── migrate_resources.py ├── doc ├── architecture.md ├── bootstrap.md ├── bootstrap.mmd ├── checkov.md ├── dns-pipeline.mmd └── email_architectures.md ├── requirements.txt ├── terraform ├── .terraform.lock.hcl ├── 18f.gov.tf ├── analytics.usa.gov.tf ├── backend.tfvars ├── benefits.gov.tf ├── bootstrap │ ├── README.md │ ├── init.tf │ ├── provider.tf │ └── terraform.tfvars ├── businessusa.gov.tf ├── challenge.gov.tf ├── citizenscience.gov.tf ├── cld.epa.gov.tf ├── cloud.gov.tf ├── code.gov.tf ├── collegescorecard.ed.gov.tf ├── consumeraction.gov.tf ├── data.gov.tf ├── digital.gov.tf ├── digitalgov.gov.tf ├── email_security │ ├── main.tf │ └── vars.tf ├── everykidinapark.gov.tf ├── fac.gov.tf ├── fedinfo.gov.tf ├── fedramp.gov.tf ├── firstgov.gov.tf ├── forms.gov.tf ├── gobiernousa.gov.tf ├── govloans.gov.tf ├── https.cio.gov.tf ├── identitysandbox.gov.tf ├── info.gov.tf ├── init.tf ├── innovation.gov.tf ├── kids.gov.tf ├── login.gov.tf ├── notify.gov.tf ├── open.foia.gov.tf ├── pif.gov.tf ├── plainlanguage.gov.tf ├── presidentialinnovationfellows.gov.tf ├── pulse.cio.gov.tf ├── sbst.gov.tf ├── search.gov.tf ├── tock.18f.gov.tf ├── tts.gsa.gov.tf ├── us.gov.tf ├── usa.gov.tf ├── usability.gov.tf ├── usagov.gov.tf └── vote.gov.tf └── tests ├── test_email_security.py └── test_missing_aaaa.py /.circleci/config.yml: -------------------------------------------------------------------------------- 1 | version: 2.1 2 | orbs: 3 | tfsec: mycodeself/tfsec@1.1.0 4 | 5 | jobs: 6 | plan: 7 | docker: 8 | - image: hashicorp/terraform:1.1.5 9 | steps: 10 | - checkout 11 | - run: 12 | name: Set up Terraform 13 | # https://discuss.circleci.com/t/create-separate-steps-jobs-for-pr-forks-versus-branches/13419/2 14 | command: | 15 | cd terraform 16 | if [[ $(echo "$CIRCLE_BRANCH" | grep -c "pull") -gt 0 ]]; then 17 | echo "This is from a fork." 18 | terraform init -backend=false -input=false 19 | else 20 | terraform init -backend-config=backend.tfvars -input=false 21 | fi 22 | - run: 23 | name: Validate Terraform config 24 | command: | 25 | cd terraform 26 | if [[ $(echo "$CIRCLE_BRANCH" | grep -c "pull") -gt 0 ]]; then 27 | echo "This is from a fork." 28 | terraform validate 29 | else 30 | terraform plan -input=false 31 | fi 32 | 33 | - persist_to_workspace: 34 | root: . 35 | paths: 36 | - ./terraform/* 37 | 38 | tfsec: 39 | executor: tfsec/default 40 | steps: 41 | - checkout 42 | - tfsec/scan: 43 | directory: . 44 | #exclude-checks: '' 45 | 46 | checkov: 47 | docker: 48 | - image: circleci/python:latest 49 | steps: 50 | - checkout 51 | - run: 52 | name: Install checkov 53 | command: pip install checkov 54 | - run: 55 | name: Run checkov 56 | command: checkov -d ./terraform/ --skip-check CKV_AWS_18,CKV_AWS_40,CKV_AWS_144,CKV_AWS_273,CKV_AWS_289,CKV_AWS_290,CKV_AWS_355,CKV2_AWS_23,CKV2_AWS_39,CKV2_AWS_38,CKV2_AWS_62 57 | # Removed CKV2_AWS_61, CKV_AWS_145, CKV_AWS_18, CKV2_AWS_6 as those are now fixed 58 | 59 | apply: 60 | docker: 61 | - image: hashicorp/terraform:1.1.5 62 | steps: 63 | - attach_workspace: 64 | at: . 65 | 66 | - run: 67 | name: Deploy the full environment 68 | command: cd terraform && terraform apply -input=false -auto-approve 69 | 70 | validate: 71 | docker: 72 | - image: python:3.9-alpine 73 | steps: 74 | - checkout 75 | - run: 76 | name: Install dependencies 77 | command: pip install -r requirements.txt 78 | - run: 79 | name: Run validations 80 | command: pytest tests 81 | 82 | workflows: 83 | version: 2 84 | 85 | validate_and_deploy: 86 | jobs: 87 | - plan 88 | - tfsec 89 | - checkov 90 | - apply: 91 | filters: 92 | branches: 93 | only: main 94 | requires: 95 | - plan 96 | - validate: 97 | filters: 98 | branches: 99 | only: main 100 | requires: 101 | - apply 102 | 103 | experimental: 104 | notify: 105 | branches: 106 | only: 107 | - main 108 | -------------------------------------------------------------------------------- /.github/pull_request_template.md: -------------------------------------------------------------------------------- 1 | - [ ] This is a new public-facing site _(if so, please follow the additional instructions below)_ 2 | - [ ] Provide context 3 | - [ ] Assign to `@GSA-TTS/tts-tech-operations` for review 4 | - [ ] Review [GSA Pages Requirements](https://handbook.tts.gsa.gov/gsa-pages) 5 | - [ ] Review [TTS Digital Council's new site review process](https://docs.google.com/document/d/1j6eieL3oop0rxCAldVVh7uGdOCG-ajafrog_BZ-u470/edit) completed 6 | - [ ] Update [the inventory](https://docs.google.com/spreadsheets/d/1OBO6g7_OsVBv0vG8WSCI6L2FD_iRh3A7a_6eQWj2zLE/edit?ts=6025575d#gid=2013137748) with new site information 7 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Terraform artifacts 2 | .terraform/ 3 | *.tfstate* 4 | crash.log 5 | 6 | # Python bytecode and caches 7 | __pycache__/ 8 | *.pyc 9 | 10 | # macOS system files 11 | .DS_Store 12 | ._.DS_Store 13 | 14 | # Sensitive files 15 | credentials.yml 16 | credentials.yml.example 17 | credentials.* 18 | -------------------------------------------------------------------------------- /.tool-versions: -------------------------------------------------------------------------------- 1 | terraform 0.11.14 2 | -------------------------------------------------------------------------------- /CODEOWNERS: -------------------------------------------------------------------------------- 1 | * @GSA-TTS/tts-tech-operations 2 | 3 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | # Public Domain 2 | 3 | As a work of the United States Government, this project is in the 4 | public domain within the United States. 5 | 6 | Additionally, we waive copyright and related rights in the work 7 | worldwide through the CC0 1.0 Universal public domain dedication. 8 | 9 | ## CC0 1.0 Universal Summary 10 | 11 | This is a human-readable summary of the [Legal Code (read the full text)](https://creativecommons.org/publicdomain/zero/1.0/legalcode). 12 | 13 | ### No Copyright 14 | 15 | The person who associated a work with this deed has dedicated the work to 16 | the public domain by waiving all of his or her rights to the work worldwide 17 | under copyright law, including all related and neighboring rights, to the 18 | extent allowed by law. 19 | 20 | You can copy, modify, distribute and perform the work, even for commercial 21 | purposes, all without asking permission. 22 | 23 | ### Other Information 24 | 25 | In no way are the patent or trademark rights of any person affected by CC0, 26 | nor are the rights that other persons may have in the work or in how the 27 | work is used, such as publicity or privacy rights. 28 | 29 | Unless expressly stated otherwise, the person who associated a work with 30 | this deed makes no warranties about the work, and disclaims liability for 31 | all uses of the work, to the fullest extent permitted by applicable law. 32 | When using or citing the work, you should not imply endorsement by the 33 | author or the affirmer. 34 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # TTS DNS configuration 2 | 3 | [![CircleCI](https://circleci.com/gh/GSA-TTS/dns.svg?branch=main&style-svg)](https://circleci.com/gh/GSA-TTS/dns) 4 | 5 | This repository holds the source code for configuring DNS for domains managed by GSA TTS. 6 | 7 | - [Technical details about this repository](doc/architecture.md) 8 | 9 | ## Making changes 10 | 11 | Assuming you're TTS staff, it's recommended that you **make the change in a branch on this repository itself, rather than on a fork**, because the credentials aren't shared with forks. (The `main` branch is [protected](https://help.github.com/articles/about-protected-branches/) to limit write access only to certain staff, and to ensure history doesn't get overwritten, etc.) For major changes, it is recommended to keep the TTL value low just before and during the change period in order to make it easier to verify the changes went through as expected. 12 | 13 | 1. Is the domain pointing to the right nameservers? In other words, is there a file for the domain under [`terraform/`](terraform) already? 14 | - **Yes:** Continue to next step. 15 | - **No:** 16 | 1. Add a file for the domain (or subdomain, if the second-level domain isn't being added), to create the [public hosted zone](http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/CreatingHostedZone.html). 17 | - [`analytics.usa.gov`](terraform/analytics.usa.gov.tf) is a good example to copy from. 18 | - You'll be using Terraform's [`aws_route53_zone`](https://www.terraform.io/docs/providers/aws/d/route53_zone.html). 19 | - If it's an existing domain, you'll want to make sure all existing records are copied over, so that they don't break when the cutover happens. You can ask the existing DNS managers for a list of records or a [zone file](https://en.wikipedia.org/wiki/Zone_file) for the domain and all subdomains. 20 | 1. After the pull request is merged, to get the name servers for your domain check the output for your build in [CircleCI](https://circleci.com/gh/GSA-TTS/dns). If you need further assistance, check with [#admins-dns](https://gsa-tts.slack.com/messages/C4L58EQ5T). 21 | 1. Change the nameservers for the domain to point to AWS. 22 | - For second-level domains, this will be done by the "domain manager" in [dotgov.gov](https://www.dotgov.gov/). [More information about .gov domain management.](https://home.dotgov.gov/management/) 23 | 1. Add the relevant additional [record sets](http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/rrsets-working-with.html). In Terraform, these are known as [`aws_route53_record`](https://www.terraform.io/docs/providers/aws/r/route53_record.html)s. 24 | 25 | It's worth noting that if you are pointing to a CloudFront distro, you should use Route 53's own `alias` and not a CNAME record. In fact, CNAMEing a top-level domain (or the top level of a delegated subdomain) is not allowed in DNS. See the various examples in the repo, such as [this one](https://github.com/GSA-TTS/dns/blob/deploy/terraform/usa.gov.tf#L8-L17). 26 | 27 | On merge, changes are deployed to an AWS account hosting the Route53 records automatically by a [CircleCI](https://circleci.com/gh/GSA-TTS/dns) job. 28 | 29 | ### Redirects 30 | 31 | [Redirects are managed here](https://github.com/cloud-gov/pages-redirects) 32 | 33 | If the redirecting domain is assigned to any CloudFront distribution in any AWS account, it will need to be unassociated before the module above can be successfully deployed. 34 | 35 | ## Stubbing files 36 | 37 | We keep a Terraform file for every TTS domain, even if it's just a comment referencing DNS managed elsewhere. 38 | 39 | ## Public domain 40 | 41 | This project is in the worldwide [public domain](LICENSE.md). As stated in the [license](LICENSE.md): 42 | 43 | > This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the [CC0 1.0 Universal public domain dedication](https://creativecommons.org/publicdomain/zero/1.0/). 44 | 45 | > All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest. 46 | -------------------------------------------------------------------------------- /bin/migrate_resources-11.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import json 3 | import hcl 4 | import os 5 | import glob 6 | import re 7 | import terrascript 8 | import terrascript.provider as provider 9 | import terrascript.resource as resource 10 | 11 | # Initialize some variables 12 | initial_cwd = os.getcwd() 13 | 14 | # Define some constants 15 | LOG_DEBUG=3 16 | LOG_INFO=2 17 | LOG_WARN=1 18 | LOG_ERROR=0 19 | 20 | LOG_NAMES = ['ERROR', 'WARN', 'INFO', 'DEBUG'] 21 | 22 | # Define the logger 23 | def logger(level, message): 24 | if(level == LOG_DEBUG and args.debug): 25 | print ("%s: %s" % (LOG_NAMES[level], message)) 26 | elif (level < LOG_DEBUG): 27 | print ("%s: %s" % (LOG_NAMES[level], message)) 28 | 29 | # Initialize the argument parser 30 | parser = argparse.ArgumentParser(description="Migrate t11 resources to t14") 31 | parser.add_argument('--statefile', type=str) 32 | parser.add_argument('--basepath', help="Use the specified base_path instead of the current working directory", default=initial_cwd) 33 | parser.add_argument('--debug', help='Enable debugging output', type=bool, default=False) 34 | args = parser.parse_args() 35 | 36 | # Slurp up the provided statefile 37 | with open(args.statefile) as f: 38 | existing_state_raw = json.load(f) 39 | 40 | # Load the statefile 41 | resources = {} 42 | resources['root'] = {} 43 | resource_list = [] 44 | for module in existing_state_raw['modules']: 45 | # We need to determine the module path 46 | # If the path contains only one element (root), then it is not actually a module and gets added to the root structure 47 | target = "" 48 | 49 | if (len(module['path']) > 1): 50 | for path_element in module['path']: 51 | # Skip over the root element 52 | if (path_element == 'root'): 53 | continue 54 | target += ".module.%s" % path_element 55 | target = target.lstrip('.') 56 | 57 | for resource in module['resources']: 58 | final_target = ("%s.%s" % (target, resource)).lstrip('.') 59 | logger(LOG_DEBUG, 'Resource "%s" found in state' % final_target) 60 | resource_list.append(final_target) 61 | 62 | for output in module['outputs']: 63 | final_target = ("%s.%s" % (target, output)).lstrip('.') 64 | logger(LOG_DEBUG, 'Output "%s" found in state' % final_target) 65 | resource_list.append(final_target) 66 | 67 | # Initialize some variables 68 | tf_files = [] 69 | tf_data = {} 70 | 71 | # Now let's get the layout of the directory structure, and get the .tf files 72 | for filename in glob.iglob(args.basepath + '**/**', recursive=True): 73 | # We skip files that are not .tf files 74 | if not re.search(r'\.tf$', filename): 75 | continue 76 | 77 | if re.search(r'terraform-11', filename): 78 | continue 79 | 80 | if re.search(r'terraform-12', filename): 81 | continue 82 | 83 | if re.search(r'terraform-14', filename): 84 | continue 85 | 86 | if re.search(r'tmp', filename): 87 | continue 88 | 89 | tf_files.append(filename) 90 | 91 | # Now we get to the fun part of renaming resources in files 92 | counter = 0 93 | tf_data = {} 94 | tf_target_data = {} 95 | state_move_filename="state-moves.sh" 96 | 97 | state_moves_fh = open(state_move_filename, 'w') 98 | 99 | # Enumerate each file 100 | for filename in tf_files: 101 | variable_remappings = {} 102 | if (re.search("init.tf", filename)): 103 | continue 104 | 105 | logger(LOG_DEBUG, 'Processing file %s' % filename) 106 | 107 | # Replace the string 'terraform' in the file path to 'terraform-11' for the target file 108 | target_filename = filename.replace('terraform', 'terraform-11') 109 | 110 | # Load the existing hcl for the file 111 | with open(filename, 'r') as tf: 112 | tf_data[target_filename] = tf.readlines() 113 | 114 | # And create the target file 115 | tf_target = open(target_filename, 'w') 116 | logger(LOG_INFO, "Generating file %s" % target_filename) 117 | 118 | for line in tf_data[target_filename]: 119 | target_line = line 120 | 121 | # Search for quoted type strings 122 | # m = re.search('(.*type.*=.*)"(string)"(.*)',target_line) 123 | # if (m): 124 | # logger(LOG_INFO, "Removing quoted type string") 125 | # target_line = target_line.replace("\"","") 126 | 127 | # Tag definitions have changed in latest AWS provider, so fix that 128 | # m = re.search('(.*tags.*) {', target_line) 129 | # if (m): 130 | # logger(LOG_INFO, "Modifying tag definition for provider compatibility") 131 | # target_line = "%s = {" % m[1] 132 | 133 | # Search for "version = 1.2.1" - this is for mediapop version pull 134 | # m = re.search('version.*=.*1.2.1', target_line) 135 | # if (m): 136 | # logger(LOG_INFO, "Upping version of mediapop to 1.3.0 from 1.2.1") 137 | # target_line = target_line.replace('1.2.1','1.3.0') 138 | 139 | # Search for interpolation-only expressions 140 | # m = re.search('(.*)"\${(.*)}"(.*)',target_line) 141 | # if (m): 142 | # logger(LOG_INFO, "Removing static interpolation syntax from %s" % (m[2])) 143 | # target_line = "%s%s%s\n"%(m[1],m[2],m[3]) 144 | 145 | # Search for remapped resource references 146 | for remapped in variable_remappings: 147 | m = re.search('([A-Za-z0-9_]*\.%s\.[A-Za-z0-9]*)' % remapped, target_line) 148 | if (m): 149 | logger(LOG_INFO, "Remapping use of %s as %s in variable %s" % (remapped, variable_remappings[remapped], m[1])) 150 | target_line = target_line.replace(remapped, variable_remappings[remapped]) 151 | 152 | # Search for resource names beginning with an invalid character 153 | m = re.search('(^(resource|module|output) .*)("([0-9].*)"(.*$))',target_line) 154 | if (m): 155 | o_resource_name = m[4] 156 | n_resource_name = "d_%s" % m[4] 157 | target_line = target_line.replace(o_resource_name, n_resource_name) 158 | logger(LOG_INFO, "Remapping resource %s as %s" % (o_resource_name, n_resource_name)) 159 | variable_remappings[o_resource_name] = n_resource_name 160 | 161 | # Now generate the state rename 162 | if not (m[2] == 'output'): 163 | print(m[2]) 164 | matches = [match for match in resource_list if o_resource_name in match] 165 | match_mapping = matches[0].replace(o_resource_name, n_resource_name) 166 | logger(LOG_DEBUG, "State will move %s to %s" % (matches[0], match_mapping)) 167 | state_moves_fh.write("t11 state mv %s %s\n" % (matches[0], match_mapping)) 168 | 169 | if not ("\n" in target_line): 170 | target_line = target_line + "\n" 171 | 172 | tf_target.write(target_line) 173 | 174 | # Finally close out the target file 175 | tf_target.close() 176 | 177 | state_moves_fh.close() 178 | -------------------------------------------------------------------------------- /bin/migrate_resources-12.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import json 3 | import hcl 4 | import os 5 | import glob 6 | import re 7 | import terrascript 8 | import terrascript.provider as provider 9 | import terrascript.resource as resource 10 | 11 | # Initialize some variables 12 | initial_cwd = os.getcwd() 13 | 14 | # Define some constants 15 | LOG_DEBUG=3 16 | LOG_INFO=2 17 | LOG_WARN=1 18 | LOG_ERROR=0 19 | 20 | LOG_NAMES = ['ERROR', 'WARN', 'INFO', 'DEBUG'] 21 | 22 | # Define the logger 23 | def logger(level, message): 24 | if(level == LOG_DEBUG and args.debug): 25 | print ("%s: %s" % (LOG_NAMES[level], message)) 26 | elif (level < LOG_DEBUG): 27 | print ("%s: %s" % (LOG_NAMES[level], message)) 28 | 29 | # Initialize the argument parser 30 | parser = argparse.ArgumentParser(description="Migrate t11 resources to t14") 31 | parser.add_argument('--statefile', type=str) 32 | parser.add_argument('--basepath', help="Use the specified base_path instead of the current working directory", default=initial_cwd) 33 | parser.add_argument('--debug', help='Enable debugging output', type=bool, default=False) 34 | args = parser.parse_args() 35 | 36 | # Slurp up the provided statefile 37 | with open(args.statefile) as f: 38 | existing_state_raw = json.load(f) 39 | 40 | # Load the statefile 41 | resources = {} 42 | resources['root'] = {} 43 | resource_list = [] 44 | for module in existing_state_raw['modules']: 45 | # We need to determine the module path 46 | # If the path contains only one element (root), then it is not actually a module and gets added to the root structure 47 | target = "" 48 | 49 | if (len(module['path']) > 1): 50 | for path_element in module['path']: 51 | # Skip over the root element 52 | if (path_element == 'root'): 53 | continue 54 | target += ".module.%s" % path_element 55 | target = target.lstrip('.') 56 | 57 | for resource in module['resources']: 58 | final_target = ("%s.%s" % (target, resource)).lstrip('.') 59 | logger(LOG_DEBUG, 'Resource "%s" found in state' % final_target) 60 | resource_list.append(final_target) 61 | 62 | for output in module['outputs']: 63 | final_target = ("%s.%s" % (target, output)).lstrip('.') 64 | logger(LOG_DEBUG, 'Output "%s" found in state' % final_target) 65 | resource_list.append(final_target) 66 | 67 | # Initialize some variables 68 | tf_files = [] 69 | tf_data = {} 70 | 71 | # Now let's get the layout of the directory structure, and get the .tf files 72 | for filename in glob.iglob(args.basepath + '**/**', recursive=True): 73 | # We skip files that are not .tf files 74 | if not re.search(r'\.tf$', filename): 75 | continue 76 | 77 | if re.search(r'terraform-orig', filename): 78 | continue 79 | 80 | if re.search(r'terraform-11', filename): 81 | continue 82 | 83 | if re.search(r'terraform-12', filename): 84 | continue 85 | 86 | if re.search(r'terraform-13', filename): 87 | continue 88 | 89 | if re.search(r'terraform-14', filename): 90 | continue 91 | 92 | tf_files.append(filename) 93 | 94 | # Now we get to the fun part of renaming resources in files 95 | counter = 0 96 | tf_data = {} 97 | tf_target_data = {} 98 | state_move_filename="state-moves.sh" 99 | 100 | state_moves_fh = open(state_move_filename, 'w') 101 | 102 | # Enumerate each file 103 | for filename in tf_files: 104 | variable_remappings = {} 105 | if (re.search("init.tf", filename)): 106 | continue 107 | 108 | logger(LOG_DEBUG, 'Processing file %s' % filename) 109 | 110 | # Replace the string 'terraform' in the file path to 'terraform-11' for the target file 111 | target_filename = filename.replace('terraform', 'terraform-12') 112 | 113 | # Load the existing hcl for the file 114 | with open(filename, 'r') as tf: 115 | tf_data[target_filename] = tf.readlines() 116 | 117 | # And create the target file 118 | tf_target = open(target_filename, 'w') 119 | logger(LOG_INFO, "Generating file %s" % target_filename) 120 | 121 | for line in tf_data[target_filename]: 122 | target_line = line 123 | 124 | # Search for quoted type strings 125 | m = re.search('(.*type.*=.*)"(string)"(.*)',target_line) 126 | if (m): 127 | logger(LOG_INFO, "Removing quoted type string") 128 | target_line = target_line.replace("\"","") 129 | 130 | # Tag definitions have changed in latest AWS provider, so fix that 131 | m = re.search('(.*tags.*) {', target_line) 132 | if (m): 133 | logger(LOG_INFO, "Modifying tag definition for provider compatibility") 134 | target_line = "%s = {" % m[1] 135 | 136 | # Search for "version = 1.2.1" - this is for mediapop version pull 137 | m = re.search('version.*=.*1.2.1', target_line) 138 | if (m): 139 | logger(LOG_INFO, "Upping version of mediapop to 1.3.0 from 1.2.1") 140 | target_line = target_line.replace('1.2.1','1.3.0') 141 | 142 | # Search for interpolation-only expressions 143 | m = re.search('(.*)"\${(.*)}"(.*)',target_line) 144 | if (m): 145 | logger(LOG_INFO, "Removing static interpolation syntax from %s" % (m[2])) 146 | target_line = "%s%s%s\n"%(m[1],m[2],m[3]) 147 | 148 | # Search for remapped resource references 149 | for remapped in variable_remappings: 150 | m = re.search('([A-Za-z0-9_]*\.%s\.[A-Za-z0-9]*)' % remapped, target_line) 151 | if (m): 152 | logger(LOG_INFO, "Remapping use of %s as %s in variable %s" % (remapped, variable_remappings[remapped], m[1])) 153 | target_line = target_line.replace(remapped, variable_remappings[remapped]) 154 | 155 | # Search for resource names beginning with an invalid character 156 | m = re.search('(^(resource|module|output) .*)("([0-9].*)"(.*$))',target_line) 157 | if (m): 158 | o_resource_name = m[4] 159 | n_resource_name = "d_%s" % m[4] 160 | target_line = target_line.replace(o_resource_name, n_resource_name) 161 | logger(LOG_INFO, "Remapping resource %s as %s" % (o_resource_name, n_resource_name)) 162 | variable_remappings[o_resource_name] = n_resource_name 163 | 164 | # Now generate the state rename 165 | if not (m[2] == 'output'): 166 | matches = [match for match in resource_list if o_resource_name in match] 167 | match_mapping = matches[0].replace(o_resource_name, n_resource_name) 168 | logger(LOG_DEBUG, "State will move %s to %s" % (matches[0], match_mapping)) 169 | state_moves_fh.write("t11 state mv %s %s\n" % (matches[0], match_mapping)) 170 | 171 | if not ("\n" in target_line): 172 | target_line = target_line + "\n" 173 | 174 | tf_target.write(target_line) 175 | 176 | # Finally close out the target file 177 | tf_target.close() 178 | 179 | state_moves_fh.close() 180 | -------------------------------------------------------------------------------- /bin/migrate_resources.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import json 3 | import hcl 4 | import os 5 | import glob 6 | import re 7 | import terrascript 8 | import terrascript.provider as provider 9 | import terrascript.resource as resource 10 | 11 | # Initialize some variables 12 | initial_cwd = os.getcwd() 13 | 14 | # Define some constants 15 | LOG_DEBUG=3 16 | LOG_INFO=2 17 | LOG_WARN=1 18 | LOG_ERROR=0 19 | 20 | LOG_NAMES = ['ERROR', 'WARN', 'INFO', 'DEBUG'] 21 | 22 | # Define the logger 23 | def logger(level, message): 24 | if(level == LOG_DEBUG and args.debug): 25 | print ("%s: %s" % (LOG_NAMES[level], message)) 26 | elif (level < LOG_DEBUG): 27 | print ("%s: %s" % (LOG_NAMES[level], message)) 28 | 29 | # Initialize the argument parser 30 | parser = argparse.ArgumentParser(description="Migrate t11 resources to t14") 31 | parser.add_argument('--statefile', type=str) 32 | parser.add_argument('--basepath', help="Use the specified base_path instead of the current working directory", default=initial_cwd) 33 | parser.add_argument('--debug', help='Enable debugging output', type=bool, default=False) 34 | args = parser.parse_args() 35 | 36 | # Slurp up the provided statefile 37 | with open(args.statefile) as f: 38 | existing_state_raw = json.load(f) 39 | 40 | # Load the statefile 41 | resources = {} 42 | resources['root'] = {} 43 | for module in existing_state_raw['modules']: 44 | # We need to determine the module path 45 | # If the path contains only one element (root), then it is not actually a module and gets added to the root structure 46 | target = "" 47 | 48 | if (len(module['path']) > 1): 49 | for path_element in module['path']: 50 | # Skip over the root element 51 | if (path_element == 'root'): 52 | continue 53 | target += ".module.%s" % path_element 54 | target = target.lstrip('.') 55 | 56 | for resource in module['resources']: 57 | final_target = ("%s.%s" % (target, resource)).lstrip('.') 58 | logger(LOG_DEBUG, 'Resource "%s" found in state' % final_target) 59 | 60 | for output in module['outputs']: 61 | logger(LOG_DEBUG, 'Output "%s" found in state' % output) 62 | 63 | # Initialize some variables 64 | tf_files = [] 65 | tf_data = {} 66 | 67 | # Now let's get the layout of the directory structure, and get the .tf files 68 | for filename in glob.iglob(args.basepath + '**/**', recursive=True): 69 | # We skip files that are not .tf files 70 | if not re.search(r'\.tf$', filename): 71 | continue 72 | 73 | if re.search(r'terraform-14', filename): 74 | continue 75 | 76 | tf_files.append(filename) 77 | 78 | # Now we get to the fun part of renaming resources in files 79 | counter = 0 80 | tf_data = {} 81 | tf_target_data = {} 82 | # Enumerate each file 83 | for filename in tf_files: 84 | variable_remappings = {} 85 | if (re.search("init.tf", filename)): 86 | continue 87 | 88 | logger(LOG_DEBUG, 'Processing file %s' % filename) 89 | 90 | # Replace the string 'terraform' in the file path to 'terraform-14' for the target file 91 | target_filename = filename.replace('terraform', 'terraform-14') 92 | 93 | # Load the existing hcl for the file 94 | with open(filename, 'r') as tf: 95 | #tf_data[target_filename] = [x.strip() for x in tf.readlines()] 96 | tf_data[target_filename] = tf.readlines() 97 | 98 | # Now let's start having fun 99 | # Initialize a new terrascript object 100 | #config = terrascript.Terrascript() 101 | # And create the target file 102 | tf_target = open(target_filename, 'w') 103 | logger(LOG_INFO, "Generating file %s" % target_filename) 104 | 105 | for line in tf_data[target_filename]: 106 | target_line = line 107 | 108 | # Search for quoted type strings 109 | m = re.search('(.*type.*=.*)"(string)"(.*)',target_line) 110 | if (m): 111 | logger(LOG_INFO, "Removing quoted type string") 112 | target_line = target_line.replace("\"","") 113 | 114 | # Tag definitions have changed in latest AWS provider, so fix that 115 | m = re.search('(.*tags.*) {', target_line) 116 | if (m): 117 | logger(LOG_INFO, "Modifying tag definition for provider compatibility") 118 | target_line = "%s = {" % m[1] 119 | 120 | # Search for "version = 1.2.1" - this is for mediapop version pull 121 | m = re.search('version.*=.*1.2.1', target_line) 122 | if (m): 123 | logger(LOG_INFO, "Upping version of mediapop to 1.3.0 from 1.2.1") 124 | target_line = target_line.replace('1.2.1','1.3.0') 125 | 126 | # Search for interpolation-only expressions 127 | m = re.search('(.*)"\${(.*)}"(.*)',target_line) 128 | if (m): 129 | logger(LOG_INFO, "Removing static interpolation syntax from %s" % (m[2])) 130 | target_line = "%s%s%s\n"%(m[1],m[2],m[3]) 131 | 132 | # Search for remapped resource references 133 | for remapped in variable_remappings: 134 | m = re.search('([A-Za-z0-9_]*\.%s\.[A-Za-z0-9]*)' % remapped, target_line) 135 | if (m): 136 | logger(LOG_INFO, "Remapping use of %s as %s in variable %s" % (remapped, variable_remappings[remapped], m[1])) 137 | target_line = target_line.replace(remapped, variable_remappings[remapped]) 138 | 139 | # Search for resource names beginning with an invalid character 140 | m = re.search('(^(resource|module|output) .*)("([0-9].*)"(.*$))',target_line) 141 | if (m): 142 | o_resource_name = m[4] 143 | n_resource_name = "d_%s" % m[4] 144 | target_line = target_line.replace(o_resource_name, n_resource_name) 145 | logger(LOG_INFO, "Remapping resource %s as %s" % (o_resource_name, n_resource_name)) 146 | variable_remappings[o_resource_name] = n_resource_name 147 | 148 | if not ("\n" in target_line): 149 | target_line = target_line + "\n" 150 | 151 | tf_target.write(target_line) 152 | 153 | # Let's enumerate the tf classes 154 | #for tfclass in tf_data[target_filename].keys(): 155 | # Classes we care about are 'resource' and 'module' (but probably only resource) 156 | # if not ((tfclass == 'resource') or (tfclass == 'module') or (tfclass == 'output')): 157 | # continue 158 | 159 | # Now we enumerate each object in the class 160 | # for tfobject in tf_data[target_filename][tfclass]: 161 | # logger(LOG_DEBUG, 'Found %s/%s in filename %s' % (tfclass, tfobject, filename)) 162 | 163 | # for tfobject_element in tf_data[target_filename][tfclass][tfobject]: 164 | # logger(LOG_DEBUG, 'Adding element %s to %s %s with data: %s' % (tfobject_element, tfclass, tfobject, tf_data[target_filename][tfclass][tfobject][tfobject_element])) 165 | # tf_target.write('%s "%s" "%s"' % (tfclass, tfobject, tf_data[target_filename][tfclass][tfobject][tfobject_element])) 166 | 167 | # Finally close out the target file 168 | tf_target.close() 169 | -------------------------------------------------------------------------------- /doc/architecture.md: -------------------------------------------------------------------------------- 1 | # DNS architecture 2 | 3 | This page attempts to describe the overall architecture of how 18F manages DNS and the associated components that make up the architecture. If you are looking for a refresher on DNS you can check [here](https://docs.google.com/presentation/d/11_bu_a1W2jw57jRT2mteo16TZmSWLOeoVtdf-Flskcg/). The DNS zones that 18F hosts at AWS Route 53 are managed by a set of Terraform configurations stored in a GitHub repository, tested with continuous integration, and deployed and changed in a deployment pipeline, with Slack notifications. 4 | 5 | ## Diagram 6 | 7 | ![dns-pipeline](https://user-images.githubusercontent.com/20934414/34623560-7dd34d3c-f217-11e7-95fd-1cc8236d4b5b.png) 8 | 9 | ## GitHub 10 | 11 | The DNS terraform configs and testing and deployment configuration live in this repository. The main and active branch is the `main` branch, which is what is deployed to the live Route53 host. This repo is managed by 18F Infrastructure, and both the cloud.gov and Federalist teams have write access to the repo. The 18F org as a whole has read access. 12 | 13 | To make [changes](https://github.com/18F/Infrastructure/wiki/Making-DNS-changes) to the repository, one files a [pull request](https://github.com/18F/dns/pulls) against the `main` branch. The [README](../README.md#making-changes) also provides details on making changes. 14 | 15 | ## Terraform 16 | 17 | The hosted zones are created and managed by Terraform, an infrastructure-as-code management system. The DNS repository makes the assumption that we are using Route 53 and therefore employs the syntax of [aws_route53_zone](https://www.terraform.io/docs/providers/aws/d/route53_zone.html). 18 | 19 | Terraform keeps a state file whose integrity is vital to its proper functioning; that state file and its automatic last backup are stored in a S3 bucket called `tts-dns-terraform-state` (available in configuration file [`backend.tfvars`](../terraform/backend.tfvars)). 20 | 21 | This document is not meant to replace the Terraform docs themselves; however, there are two things worthy of note: 22 | 23 | 1. To associate a DNS name with an AWS resource such as a CloudFront distribution, Elastic Load Balancer, S3 bucket, etc. one should use AWS Route 53's own [Alias](http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resource-record-sets-choosing-alias-non-alias.html) feature, which is an A record (**not** a CNAME) with an `alias` block rather than with `name` and `records` directives. 24 | 2. An Alias record requires a zone ID, a name and a Boolean value for `evaluate_target_health`. A `TTL` is not allowed in an Alias record. (Conversely, `evaluate_target_health` is not allowed in a regular record entry with the `records` keyword. 25 | 26 | See below for concrete examples of the foregoing. 27 | 28 | ### Example 29 | 30 | Here's a sample zone definition with a couple of likely records and some output, for **example.com**: 31 | 32 | ```hcl 33 | # definition of zone -- note that the "example_com_zone" is the zone's name 34 | # reference, free-form but conforming to the general naming conventions seen 35 | # below, for readability 36 | 37 | resource "aws_route53_zone" "example_com_zone" { 38 | name = "example.com" # may also be a subdomain, "subdomain.example.com" 39 | tags { 40 | Project = "dns" 41 | } 42 | } 43 | 44 | # alias of root domain to CloudFront distribution. note that 45 | # a TTL is not allowed in the "alias" stanza. 46 | # "example_com_zone" refers to the zone name reference above 47 | 48 | resource "aws_route53_record" "example_com_example_com_cloudfront_a_alias" { 49 | zone_id = "${aws_route53_zone.example_com_zone.zone_id}" 50 | name = "example.com" 51 | type = "A" 52 | alias { 53 | name = ".cloudfront.net" 54 | zone_id = "Z2FDTNDATAQYW2" 55 | # n.b.: Z2FDTNDATAQYW2 is the zone ID for all CloudFront distributions 56 | evaluate_target_health = false 57 | } 58 | } 59 | 60 | # CNAME of hostname in zone to hostname elsewhere 61 | 62 | resource "aws_route53_record" "example_com_host_domain_com_cname" { 63 | zone_id = "${aws_route53_zone.example_com_zone.zone_id}" 64 | name = "host.example.com." 65 | type = "CNAME" 66 | ttl = 5 67 | records = ["host.domain.com."] 68 | } 69 | 70 | # note that this is a standard A record, not an A-alias, and therefore 71 | # has a "records" stanza. 72 | 73 | resource "aws_route53_record" "example_com_host_domain_com_a" { 74 | zone_id = "${aws_route53_zone.example_com_zone.zone_id}" 75 | name = "host2.example.com." 76 | type = "A" 77 | ttl = 5 78 | records = ["172.16.11.23"] 79 | } 80 | 81 | 82 | # output name servers for use with registrar of domain, to point 83 | # public DNS at this AWS Route 53 zone 84 | 85 | output "example_com_ns" { 86 | value="${aws_route53_zone.example_com_zone.name_servers}" 87 | } 88 | ``` 89 | 90 | ## Validation 91 | 92 | Validation is handled by a continuous integration (CI) process implemented as a [CircleCI job](https://circleci.com/gh/18F/dns). The convention for the job name is to base it off of the appropriate description of the actions being performed in that phase. For example, a job that is named validate would correspond to the command _terraform [validate](https://www.terraform.io/docs/commands/validate.html)_ which performs a linting step. Note that the scope of this validate command is very limited at the time of this writing -- this command does not do deep syntax checking, nor does it discover any problems running against existing resources. 93 | 94 | As a result, we require a peer review of the changes. GitHub won't merge the change to the `main` branch until a reviewer approves. (As mentioned above, other groups such as federalist reviews its own DNS changes.) 95 | 96 | Note that deeper linting of terraform files, along the lines of a dry run, is on [Hashicorp's radar](https://github.com/hashicorp/terraform/issues/11427). 97 | 98 | ## Deployment 99 | 100 | Deployment is handled as a workflow in [CircleCI](https://circleci.com/gh/18F/dns) that happens after a successful validation job. The job is specified in the [circle config](../.circleci/config.yml) which follows an appropriate naming scheme (see above). The deployment consists of a syntax validation check and the Terraform command [_terraform apply_](https://www.terraform.io/docs/commands/apply.html). Terraform accesses AWS from the CircleCI configuration [settings](https://circleci.com/gh/18F/dns/edit). 101 | 102 | Output from the job, typically just the nameservers configured for each zone, is visible in CircleCI builds view (see the "output" section in the sample config above). Notification regarding the success or failure of the job is also sent to Slack (see below). 103 | 104 | DNS zones that are delegated to us live in the 18F Enterprise AWS account. 105 | 106 | ## Slack 107 | 108 | Infrastructure has its own Slack channel for DNS requests, questions and automated job notifications, **#admins-dns**. When CircleCI runs a DNS job, the channel receives a notification regarding the success or failure of jobs. The Slack integration can be found in CircleCI's configuration [hooks](https://circleci.com/gh/18F/dns/edit#hooks). 109 | 110 | ## Authentication and secrets 111 | 112 | To administer every aspect of the automated DNS system, you will need the following: 113 | 114 | - Membership in one of the following GitHub teams: `infrastructure-staff`, `federalist-admins`, `cloud-gov-ops` 115 | - An AWS `ACCESS_KEY_ID` and `SECRET_ACCESS_KEY` with access to the [production Route 53 service](https://18f.signin.aws.amazon.com/console) 116 | - The Slack webhook secret set in CircleCI notifications (check in #admins-slack) 117 | -------------------------------------------------------------------------------- /doc/bootstrap.md: -------------------------------------------------------------------------------- 1 | # Bootstrap Infrastructure Architecture 2 | 3 | This document provides a visual overview of the bootstrap infrastructure created by the Terraform configuration in `bootstrap/init.tf`. 4 | 5 | ## Overview Diagram 6 | 7 | ```mermaid 8 | graph TD 9 | %% Define main resources with simpler styling 10 | User(["CircleCI User"]) 11 | MainS3["S3 Bucket
Main State"] 12 | LogsS3["S3 Bucket
Logs"] 13 | LogsLogsS3["S3 Bucket
Logs of Logs"] 14 | ReplicaS3["S3 Bucket
DR Replica"] 15 | ReplicaLogsS3["S3 Bucket
DR Logs"] 16 | 17 | %% Security & IAM 18 | IAMUser["IAM User
circleci-deployer"] 19 | SNS["SNS Topics"] 20 | Email["Email Alerts"] 21 | 22 | %% Define regions 23 | subgraph "US-EAST-1 (Primary)" 24 | MainS3 25 | LogsS3 26 | LogsLogsS3 27 | IAMUser 28 | end 29 | 30 | subgraph "US-WEST-2 (DR)" 31 | ReplicaS3 32 | ReplicaLogsS3 33 | end 34 | 35 | %% Define relationships 36 | User --> IAMUser 37 | IAMUser --> MainS3 38 | 39 | MainS3 --> ReplicaS3 40 | MainS3 --> LogsS3 41 | LogsS3 --> LogsLogsS3 42 | ReplicaS3 --> ReplicaLogsS3 43 | 44 | MainS3 --> SNS 45 | LogsS3 --> SNS 46 | ReplicaS3 --> SNS 47 | SNS --> Email 48 | ``` 49 | 50 | ## Key Components 51 | 52 | ### Storage 53 | - **Main State Bucket**: Primary S3 bucket for storing Terraform state 54 | - **Replica State Bucket**: Cross-region replica of the main state bucket for disaster recovery 55 | - **Logs Bucket**: Stores access logs from the main state bucket 56 | - **Logs-of-Logs Bucket**: Stores access logs from the logs bucket to prevent circular dependencies 57 | - **Replica Logs Bucket**: Stores access logs from the replica state bucket in the DR region 58 | 59 | ### Encryption 60 | - **KMS Keys**: Two KMS keys (one in each region) to encrypt bucket contents 61 | - **Server-Side Encryption**: All buckets use KMS-based server-side encryption 62 | 63 | ### Access Control 64 | - **Public Access Blocks**: All buckets have public access blocks to prevent exposure 65 | - **IAM User**: CircleCI deployer user with limited permissions 66 | - **IAM Policy**: Grants specific permissions for Route53 management and state bucket access 67 | - **IAM Role**: Replication role for cross-region bucket replication 68 | 69 | ### Notifications 70 | - **SNS Topics**: Three separate topics for notifications from different buckets 71 | - **Email Notifications**: All bucket events are sent to the DevOps team email 72 | 73 | ### Security Measures 74 | - **Lifecycle Rules**: Automatic transitions and expirations for old object versions 75 | - **Versioning**: All buckets have versioning enabled for audit trail and recovery 76 | - **Access Logging**: Comprehensive logging hierarchy for all operations 77 | 78 | ## Design Considerations 79 | 80 | The infrastructure follows a defense-in-depth approach with multiple security layers: 81 | 82 | 1. **Data Protection**: KMS encryption, versioning, and access controls 83 | 2. **Monitoring**: Event notifications for all significant operations 84 | 3. **Disaster Recovery**: Cross-region replication of critical state data 85 | 4. **Least Privilege**: Specific IAM permissions for the CircleCI deployer user 86 | 5. **Security Posture**: Public access blocks and proper encryption for all storage 87 | 88 | Security exceptions (skip checks) have been added where stricter compliance would create circular dependencies or excessive complexity, with clear documentation of the reasoning behind each exception. 89 | -------------------------------------------------------------------------------- /doc/bootstrap.mmd: -------------------------------------------------------------------------------- 1 | graph TD 2 | %% Define main resources 3 | User("CircleCI User") 4 | MainS3["S3 Bucket
tts-dns-terraform-state
(Main)"] 5 | LogsS3["S3 Bucket
tts-dns-terraform-state-logs
(Logs)"] 6 | LogsLogsS3["S3 Bucket
tts-dns-terraform-state-logs-logs
(Logs of Logs)"] 7 | ReplicaS3["S3 Bucket
tts-dns-terraform-state-replica
(DR Replica)"] 8 | ReplicaLogsS3["S3 Bucket
tts-dns-terraform-state-replica-logs
(DR Logs)"] 9 | 10 | %% KMS Keys 11 | KMSKey["KMS Key
terraform-state-key
(East)"] 12 | KMSKeyWest["KMS Key
terraform-state-key-west
(West)"] 13 | 14 | %% SNS Topics 15 | SNSMain["SNS Topic
terraform-state-bucket-events"] 16 | SNSLogs["SNS Topic
terraform-logs-bucket-events"] 17 | SNSReplica["SNS Topic
terraform-replica-bucket-events"] 18 | 19 | %% IAM Resources 20 | IAMUser["IAM User
circleci-deployer"] 21 | IAMPolicy["IAM Policy
route53-deployment"] 22 | IAMRole["IAM Role
terraform-state-replication-role"] 23 | IAMReplication["IAM Policy
terraform-state-replication-policy"] 24 | 25 | %% Email Notifications 26 | Email["Email Notifications
notification_email"] 27 | 28 | %% Security configurations 29 | PABMain["Public Access Block
(Main Bucket)"] 30 | PABLogs["Public Access Block
(Logs Bucket)"] 31 | PABLogsLogs["Public Access Block
(Logs of Logs Bucket)"] 32 | PABReplica["Public Access Block
(Replica Bucket)"] 33 | PABReplicaLogs["Public Access Block
(Replica Logs Bucket)"] 34 | 35 | %% Define regions 36 | subgraph "US-EAST-1 (Primary Region)" 37 | MainS3 38 | LogsS3 39 | LogsLogsS3 40 | KMSKey 41 | SNSMain 42 | SNSLogs 43 | IAMUser 44 | IAMPolicy 45 | IAMRole 46 | IAMReplication 47 | PABMain 48 | PABLogs 49 | PABLogsLogs 50 | end 51 | 52 | subgraph "US-WEST-2 (DR Region)" 53 | ReplicaS3 54 | ReplicaLogsS3 55 | KMSKeyWest 56 | SNSReplica 57 | PABReplica 58 | PABReplicaLogs 59 | end 60 | 61 | %% Define relationships 62 | User -->|Uses| IAMUser 63 | IAMUser -->|Has| IAMPolicy 64 | IAMPolicy -->|Manages| MainS3 65 | IAMPolicy -->|Uses| KMSKey 66 | 67 | MainS3 -->|Replicated to| ReplicaS3 68 | MainS3 -->|Logs to| LogsS3 69 | LogsS3 -->|Logs to| LogsLogsS3 70 | ReplicaS3 -->|Logs to| ReplicaLogsS3 71 | 72 | MainS3 -->|Encrypted by| KMSKey 73 | LogsS3 -->|Encrypted by| KMSKey 74 | LogsLogsS3 -->|Encrypted by| KMSKey 75 | ReplicaS3 -->|Encrypted by| KMSKeyWest 76 | ReplicaLogsS3 -->|Encrypted by| KMSKeyWest 77 | 78 | MainS3 -->|Notifies| SNSMain 79 | LogsS3 -->|Notifies| SNSLogs 80 | ReplicaS3 -->|Notifies| SNSReplica 81 | 82 | SNSMain -->|Sends to| Email 83 | SNSLogs -->|Sends to| Email 84 | SNSReplica -->|Sends to| Email 85 | 86 | IAMRole -->|Assumes| IAMReplication 87 | IAMReplication -->|Allows| MainS3 88 | IAMReplication -->|To| ReplicaS3 89 | 90 | MainS3 --- PABMain 91 | LogsS3 --- PABLogs 92 | LogsLogsS3 --- PABLogsLogs 93 | ReplicaS3 --- PABReplica 94 | ReplicaLogsS3 --- PABReplicaLogs 95 | -------------------------------------------------------------------------------- /doc/checkov.md: -------------------------------------------------------------------------------- 1 | # Checkov Compliance Documentation 2 | 3 | This document describes the Checkov compliance status of our Terraform code, including intentional exceptions and mitigations. 4 | 5 | ## Overview 6 | 7 | We use [Checkov](https://github.com/bridgecrewio/checkov) as part of our infrastructure-as-code security scanning. While we strive for full compliance with security best practices, some exceptions are necessary due to technical requirements or practical limitations. 8 | 9 | ## Bootstrap Infrastructure 10 | 11 | The bootstrap infrastructure in the `/terraform/bootstrap` directory has several intentional exceptions to Checkov rules: 12 | 13 | ### S3 Bucket Exceptions 14 | 15 | | Check ID | Resource | Justification | 16 | |----------|----------|--------------| 17 | | CKV_AWS_18 | `aws_s3_bucket.logs_for_logs` | This is a top-level log bucket for logging other log buckets. Adding access logging here would create a circular dependency. | 18 | | CKV_AWS_144 | `aws_s3_bucket.logs_for_logs` | Cross-region replication not needed for tertiary log data; focus is on replicating primary state data. | 19 | | CKV2_AWS_62 | `aws_s3_bucket.logs_for_logs` | Event notifications not deemed essential for logs-of-logs bucket as it's tertiary data. | 20 | | CKV_AWS_144 | `aws_s3_bucket.backend_replica` | This bucket is already a replica itself and doesn't need further replication. | 21 | | CKV2_AWS_62 | `aws_s3_bucket.backend_replica` | Event notifications are configured via separate resource. | 22 | | CKV_AWS_18 | `aws_s3_bucket.replica_logs` | This is a log bucket for the replica and doesn't need its own access logging. | 23 | | CKV_AWS_144 | `aws_s3_bucket.replica_logs` | This bucket is already in the disaster recovery region and doesn't need further replication. | 24 | | CKV2_AWS_62 | `aws_s3_bucket.replica_logs` | Event notifications not essential for log data in disaster recovery region. | 25 | 26 | ### IAM Policy Exceptions 27 | 28 | | Check ID | Resource | Justification | 29 | |----------|----------|--------------| 30 | | CKV_AWS_290 | `aws_iam_user_policy.circleci_deployer_policy` | The CircleCI deployment user needs write access to manage DNS records and state files. | 31 | | CKV_AWS_289 | `aws_iam_user_policy.circleci_deployer_policy` | The policy needs specific Route53 permissions for DNS management. | 32 | | CKV_AWS_355 | `aws_iam_user_policy.circleci_deployer_policy` | Route53 management requires resource wildcards for DNS records. | 33 | | CKV_AWS_40 | `aws_iam_user_policy.circleci_deployer_policy` | Using direct user attachment for CircleCI pipeline integration. | 34 | 35 | ### DNS/DNSSEC Exceptions 36 | 37 | | Check ID | Resource | Justification | 38 | |----------|----------|--------------| 39 | | CKV_AWS_33 | `aws_kms_key.datagov_zone` | DNSSEC requires specific KMS key policy configuration with service principals. | 40 | 41 | ## Mitigations 42 | 43 | While some security checks are bypassed, we have implemented the following mitigations: 44 | 45 | 1. **Enhanced Encryption**: 46 | - All S3 buckets use KMS encryption instead of AWS default encryption 47 | - KMS key retention periods increased to 30 days 48 | - KMS key rotation is enabled 49 | 50 | 2. **Comprehensive Logging**: 51 | - Multiple layers of access logging for S3 buckets 52 | - All state changes generate notifications 53 | - Event-driven architecture to track changes 54 | 55 | 3. **Cross-Region Redundancy**: 56 | - Critical state data is replicated across regions 57 | - Disaster recovery configured for main state buckets 58 | 59 | 4. **SNS Topic Security**: 60 | - All SNS topics are encrypted with KMS 61 | - SNS topic subscriptions use a variable for email address 62 | 63 | ## CI/CD Integration 64 | 65 | Checkov is integrated into our CI/CD pipeline and runs on all pull requests. Security findings are addressed unless specifically exempted in this document. The scan results are part of the PR review process. 66 | 67 | ## Recent Improvements 68 | 69 | Recent security improvements include: 70 | 71 | 1. Added KMS encryption to all S3 buckets and SNS topics 72 | 2. Implemented notification system for all bucket operations 73 | 3. Hardened bucket security with public access blocks 74 | 4. Added cross-region replication for disaster recovery 75 | 5. Implemented proper lifecycle policies for all buckets 76 | 6. Added logging hierarchy to capture all access events 77 | 7. Added DNSSEC to critical DNS zones 78 | 8. Made notification email address configurable via variables 79 | 9. Extended KMS key deletion window to 30 days 80 | 81 | ## Future Work 82 | 83 | Future security improvements under consideration: 84 | 85 | 1. Transition CircleCI authentication from IAM user to IAM role with OIDC 86 | 2. Implement more granular Route53 permissions 87 | 3. Add automatic key rotation for longer-term KMS keys 88 | 4. Implement AWS CloudTrail integration for enhanced auditing -------------------------------------------------------------------------------- /doc/dns-pipeline.mmd: -------------------------------------------------------------------------------- 1 | %% Sequence Diagram for dns deployment 2 | sequenceDiagram 3 | participant Github 4 | participant CI 5 | participant Deployer 6 | participant AWS 7 | 8 | note over Github: dns repo 9 | loop Github dns repo change request 10 | Github ->>+ CI: Opened pull request 11 | 12 | Note over CI: Runs terraform validation tests 13 | Note over CI: Merges pull request on success 14 | CI -->>- Github: Delete pull request branch 15 | end 16 | 17 | Note over Github: dns repo 18 | Note over Github: deploy pipeline 19 | Github ->> Deployer: Deploys updated deploy branch 20 | 21 | Note over Deployer: terraform apply 22 | Deployer ->> AWS: update dns, save terraform state 23 | 24 | Note over Deployer: Report dns nameserver information -------------------------------------------------------------------------------- /doc/email_architectures.md: -------------------------------------------------------------------------------- 1 | # Email DNS Architecture Patterns 2 | 3 | This guide explains the different approaches to email-related DNS configurations across our domains. 4 | 5 | ## Overview 6 | 7 | We have three main patterns for email DNS configuration: 8 | 9 | 1. Direct Domain Pattern (search.gov) 10 | 2. Subdomain Pattern (digital.gov) 11 | 3. Mixed Pattern (fac.gov) 12 | 13 | ## Architecture Diagrams 14 | 15 | ### Direct Domain Pattern (search.gov) 16 | 17 | ```mermaid 18 | C4Context 19 | title Email Configuration - Direct Domain Pattern 20 | 21 | Enterprise_Boundary(aws, "AWS") { 22 | System(ses, "Amazon SES") 23 | System(r53, "Route 53") 24 | } 25 | 26 | Enterprise_Boundary(domain, "search.gov") { 27 | Component(mx, "MX Record") 28 | Component(dkim, "DKIM Records") 29 | Component(spf, "SPF Record") 30 | } 31 | 32 | Rel(ses, mx, "Uses") 33 | Rel(ses, dkim, "Authenticates via") 34 | Rel(mx, spf, "Validates with") 35 | 36 | UpdateRelStyle(ses, mx, "red", "normal") 37 | ``` 38 | 39 | ### Subdomain Pattern (digital.gov) 40 | 41 | ```mermaid 42 | C4Context 43 | title Email Configuration - Subdomain Pattern 44 | 45 | Enterprise_Boundary(aws, "AWS") { 46 | System(ses, "Amazon SES") 47 | System(r53, "Route 53") 48 | } 49 | 50 | Enterprise_Boundary(domain, "digital.gov") { 51 | Component(mail, "mail.digital.gov") 52 | Component(app, "app.digital.gov") 53 | Component(dkim, "DKIM Records") 54 | } 55 | 56 | Component(spf_main, "SPF (mail subdomain)") 57 | 58 | Rel(ses, mail, "Primary email through") 59 | Rel(ses, app, "App emails through") 60 | Rel(mail, spf_main, "Validates via") 61 | Rel(app, dkim, "Authenticates via") 62 | 63 | UpdateRelStyle(ses, mail, "blue", "normal") 64 | ``` 65 | 66 | ### Mixed Pattern (fac.gov) 67 | 68 | ```mermaid 69 | C4Context 70 | title Email Configuration - Mixed Pattern 71 | 72 | Enterprise_Boundary(aws, "AWS") { 73 | System(ses, "Amazon SES") 74 | System(r53, "Route 53") 75 | } 76 | 77 | Enterprise_Boundary(domain, "fac.gov") { 78 | Component(root_mx, "Root MX") 79 | Component(sub_mx, "Subdomain MX") 80 | Component(dkim, "DKIM Records") 81 | } 82 | 83 | Component(spf_root, "Root SPF") 84 | Component(spf_sub, "Subdomain SPF") 85 | 86 | Rel(ses, root_mx, "Primary email") 87 | Rel(ses, sub_mx, "Subdomain email") 88 | Rel(root_mx, spf_root, "Validates via") 89 | Rel(sub_mx, spf_sub, "Validates via") 90 | 91 | UpdateRelStyle(ses, root_mx, "green", "normal") 92 | ``` 93 | 94 | ## Pattern Comparison 95 | 96 | ### Direct Domain Pattern (search.gov) 97 | - MX records at root domain 98 | - DKIM records directly under domain 99 | - Single SPF record at root 100 | - Pros: Simple, straightforward 101 | - Cons: Less flexibility, all mail config at root 102 | 103 | ### Subdomain Pattern (digital.gov) 104 | - MX records on subdomains 105 | - DKIM per subdomain 106 | - Separate SPF records per subdomain 107 | - Pros: Better isolation, flexible routing 108 | - Cons: More complex management 109 | 110 | ### Mixed Pattern (fac.gov) 111 | - MX records at both root and subdomains 112 | - Shared DKIM records 113 | - Multiple SPF records 114 | - Pros: Maximum flexibility 115 | - Cons: Most complex to manage 116 | 117 | ## Implementation Details 118 | 119 | ### Direct Domain Pattern 120 | ```hcl 121 | resource "aws_route53_record" "mx" { 122 | zone_id = aws_route53_zone.zone.zone_id 123 | name = "" # Root domain 124 | type = "MX" 125 | records = ["10 inbound-smtp.amazonaws.com"] 126 | } 127 | ``` 128 | 129 | ### Subdomain Pattern 130 | ```hcl 131 | resource "aws_route53_record" "mx_subdomain" { 132 | zone_id = aws_route53_zone.zone.zone_id 133 | name = "mail.domain.gov" 134 | type = "MX" 135 | records = ["10 inbound-smtp.amazonaws.com"] 136 | } 137 | ``` 138 | 139 | ### Mixed Pattern 140 | ```hcl 141 | resource "aws_route53_record" "mx_mixed" { 142 | zone_id = aws_route53_zone.zone.zone_id 143 | name = each.key 144 | type = "MX" 145 | for_each = { 146 | "" = ["10 primary-smtp.amazonaws.com"] 147 | "department1" = ["10 dept1-smtp.amazonaws.com"] 148 | "department2" = ["10 dept2-smtp.amazonaws.com"] 149 | } 150 | records = each.value 151 | } 152 | ``` 153 | 154 | ## Best Practices 155 | 156 | 1. **Choose Based on Requirements**: 157 | - Simple email: Direct Pattern 158 | - Multiple services: Subdomain Pattern 159 | - Complex routing: Mixed Pattern 160 | 161 | 2. **Security Considerations**: 162 | - Always implement both SPF and DKIM 163 | - Consider DMARC for all patterns 164 | - Use separate DKIM keys per subdomain 165 | 166 | 3. **Maintainability**: 167 | - Document pattern choice 168 | - Use consistent naming 169 | - Implement using Terraform modules 170 | 171 | 4. **Monitoring**: 172 | - Set up alerts for record expiration 173 | - Monitor email delivery statistics 174 | - Track bounce rates per pattern 175 | 176 | ## Migration Considerations 177 | 178 | When moving between patterns: 179 | 180 | 1. Plan for DNS propagation time 181 | 2. Maintain old records during transition 182 | 3. Update SPF records gradually 183 | 4. Monitor email delivery during change 184 | 185 | ## Related Documentation 186 | - [AWS SES DNS Configuration](https://docs.aws.amazon.com/ses/latest/dg/dns-verification.html) 187 | - [DKIM Best Practices](https://docs.aws.amazon.com/ses/latest/dg/send-email-authentication-dkim.html) 188 | - [SPF Record Syntax](https://docs.aws.amazon.com/ses/latest/dg/send-email-authentication-spf.html) 189 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | boto3~=1.26 2 | checkdmarc~=4.6 3 | pytest~=7.3 4 | dnspython~=2.6.0rc1 5 | urllib3~=1.26 6 | -------------------------------------------------------------------------------- /terraform/.terraform.lock.hcl: -------------------------------------------------------------------------------- 1 | # This file is maintained automatically by "terraform init". 2 | # Manual edits may be lost in future updates. 3 | 4 | provider "registry.terraform.io/hashicorp/aws" { 5 | version = "3.31.0" 6 | constraints = "3.31.0" 7 | hashes = [ 8 | "h1:AQHoeEquKeICqW0fjUvp/sYNyiGOK9SIQO93y6LZOr0=", 9 | "h1:QN/kgOC46kY001+vM/nnmJshzAOnWN+kLLpFG9ALcMU=", 10 | "h1:Wou3ZnO10ZvN+n1iwyuaxn3zyGMFj9KYL+9IFb0gGkw=", 11 | "zh:07f5b2f4cfaa25e26a4062ac675e3e5aaf65bb21b94b8fd7f30d576398e7410f", 12 | "zh:08a2154ad29ae130ea9e46948b7b332ec4b45321b4852b45ba60adcfd049f8d6", 13 | "zh:35ed643c2b999021ad56b49f7d9d3a77c98d152477fe54b5c8a68f696bb1a0b7", 14 | "zh:3a8dc51b4be1c04130fd76cda4280019020b276336d307e7074ad52f35d4fdda", 15 | "zh:3c910c4f25e3ffd6d84f051c32161f03d1843753cd545e769757d7b42d654003", 16 | "zh:5d23f316f89937cbda36207271bbe150f633298f96d4644fd02063fc6bf0c28f", 17 | "zh:61fedb2915c5188c6550677a10acb955f32834bbe99ba0cafb2a118be282827b", 18 | "zh:65076a6899c0781ce95064d47d587ad07f80becd1510e4c475e4554131caec09", 19 | "zh:acca833c2d9985e46298323222285b370ea7cf5299b131dbdfc7c3e66fa32401", 20 | "zh:c212cf8ba7fdf64e75accf7e745f76d2349b00553ebd928cc6cafbfda99d97b7", 21 | "zh:cd3f5e89ac5f5cf3f8fed3aca4cc50261d537b60a3490feaddf9ba2f06e5e7aa", 22 | ] 23 | } 24 | -------------------------------------------------------------------------------- /terraform/18f.gov.tf: -------------------------------------------------------------------------------- 1 | // Keep only the zone definition that tock.18f.gov depends on 2 | resource "aws_route53_zone" "d_18f_gov_zone" { 3 | name = "18f.gov." 4 | 5 | tags = { 6 | Project = "dns" 7 | } 8 | } 9 | 10 | // All other records have been removed 11 | -------------------------------------------------------------------------------- /terraform/analytics.usa.gov.tf: -------------------------------------------------------------------------------- 1 | resource "aws_route53_zone" "analytics_usa_gov_zone" { 2 | name = "analytics.usa.gov." 3 | 4 | tags = { 5 | Project = "dns" 6 | } 7 | } 8 | 9 | resource "aws_route53_record" "analytics_usa_gov_analytics_usa_gov_a" { 10 | zone_id = aws_route53_zone.analytics_usa_gov_zone.zone_id 11 | name = "analytics.usa.gov." 12 | type = "A" 13 | 14 | alias { 15 | name = "d2rprfiomwib2l.cloudfront.net." 16 | zone_id = local.cloud_gov_cloudfront_zone_id 17 | evaluate_target_health = false 18 | } 19 | } 20 | 21 | output "analytics_usa_gov_ns" { 22 | value = aws_route53_zone.analytics_usa_gov_zone.name_servers 23 | } 24 | -------------------------------------------------------------------------------- /terraform/backend.tfvars: -------------------------------------------------------------------------------- 1 | bucket = "tts-dns-terraform-state" 2 | key = "terraform.tfstate" 3 | -------------------------------------------------------------------------------- /terraform/benefits.gov.tf: -------------------------------------------------------------------------------- 1 | resource "aws_route53_zone" "benefits_gov_zone" { 2 | name = "benefits.gov" 3 | 4 | tags = { 5 | Project = "dns" 6 | } 7 | } 8 | 9 | resource "aws_route53_record" "benefits_gov_apex" { 10 | zone_id = aws_route53_zone.benefits_gov_zone.zone_id 11 | name = "benefits.gov." 12 | type = "A" 13 | alias { 14 | name = "d3bi0ia5r11uox.cloudfront.net." 15 | zone_id = local.cloud_gov_cloudfront_zone_id 16 | evaluate_target_health = false 17 | } 18 | } 19 | 20 | resource "aws_route53_record" "benefits_gov_apex_aaaa" { 21 | zone_id = aws_route53_zone.benefits_gov_zone.zone_id 22 | name = "benefits.gov." 23 | type = "AAAA" 24 | alias { 25 | name = "d3bi0ia5r11uox.cloudfront.net." 26 | zone_id = local.cloud_gov_cloudfront_zone_id 27 | evaluate_target_health = false 28 | } 29 | } 30 | 31 | resource "aws_route53_record" "benefits_gov_acmechallenge" { 32 | zone_id = aws_route53_zone.benefits_gov_zone.zone_id 33 | name = "_acme-challenge.benefits.gov." 34 | type = "CNAME" 35 | ttl = 300 36 | records = ["_acme-challenge.benefits.gov.external-domains-production.cloud.gov."] 37 | } 38 | 39 | resource "aws_route53_record" "benefits_gov_www_acmechallenge" { 40 | zone_id = aws_route53_zone.benefits_gov_zone.zone_id 41 | name = "_acme-challenge.www.benefits.gov." 42 | type = "CNAME" 43 | ttl = 300 44 | records = ["_acme-challenge.www.benefits.gov.external-domains-production.cloud.gov."] 45 | } 46 | 47 | resource "aws_route53_record" "benefits_gov_ssabest_acmechallenge" { 48 | zone_id = aws_route53_zone.benefits_gov_zone.zone_id 49 | name = "_acme-challenge.ssabest.benefits.gov." 50 | type = "CNAME" 51 | ttl = 300 52 | records = ["_acme-challenge.ssabest.benefits.gov.external-domains-production.cloud.gov."] 53 | } 54 | 55 | resource "aws_route53_record" "benefits_gov_ssabest" { 56 | zone_id = aws_route53_zone.benefits_gov_zone.zone_id 57 | name = "ssabest.benefits.gov." 58 | type = "CNAME" 59 | ttl = 300 60 | records = ["ssabest.benefits.gov.external-domains-production.cloud.gov."] 61 | } 62 | 63 | resource "aws_route53_record" "benefits_gov_www" { 64 | zone_id = aws_route53_zone.benefits_gov_zone.zone_id 65 | name = "www.benefits.gov." 66 | type = "CNAME" 67 | ttl = 300 68 | records = ["www.benefits.gov.external-domains-production.cloud.gov."] 69 | } 70 | 71 | module "benefits_gov_emailsecurity" { 72 | source = "./email_security" 73 | 74 | zone_id = aws_route53_zone.benefits_gov_zone.zone_id 75 | txt_records = ["v=spf1 -all"] 76 | } 77 | 78 | output "benefits_gov_ns" { 79 | value = aws_route53_zone.benefits_gov_zone.name_servers 80 | } -------------------------------------------------------------------------------- /terraform/bootstrap/README.md: -------------------------------------------------------------------------------- 1 | # Bootstrap Configuration 2 | 3 | This needs to be run as a privileged user in the account on first run to create the infrastructure needed by the circleci-deployer user, as this user does not have IAM permissions to create these resources itself. 4 | 5 | ## Security Improvements 6 | 7 | The bootstrap configuration has been updated to address several security findings: 8 | 9 | 1. **SNS Topic Encryption** 10 | - All SNS topics are now encrypted with AWS KMS 11 | 12 | 2. **S3 Bucket Event Notifications** 13 | - Added event notifications to all buckets including log buckets 14 | - Created separate SNS topics for main, logs, and replica buckets 15 | 16 | 3. **S3 Bucket Cross-Region Replication** 17 | - Main bucket already had cross-region replication 18 | - Added skip comments for log buckets as they don't need replication 19 | - Clarified replica bucket status with skip comments 20 | 21 | 4. **S3 Bucket Lifecycle Configurations** 22 | - Added lifecycle configuration to replica bucket 23 | - All buckets now have proper lifecycle rules 24 | 25 | 5. **S3 Bucket Access Logging** 26 | - Access logs bucket now logs to a logs-of-logs bucket 27 | - Replica bucket logs to a dedicated logs bucket in the west region 28 | 29 | 6. **KMS Encryption** 30 | - All buckets now use KMS encryption instead of AES256 31 | - Log buckets now use the same KMS key as the main bucket 32 | - KMS key deletion window extended to 30 days for better safety 33 | 34 | ## Resources Created 35 | 36 | - S3 bucket for Terraform state with: 37 | - Server-side encryption using KMS 38 | - Public access blocks 39 | - Versioning enabled 40 | - Lifecycle policies 41 | - Cross-region replication to disaster recovery bucket 42 | - Access logging 43 | - Event notifications 44 | 45 | - KMS keys for encryption 46 | - State bucket encryption key 47 | - Replica bucket encryption key 48 | 49 | - IAM user and policy for CircleCI deployments with permissions for: 50 | - Route53 management 51 | - S3 bucket access for Terraform state 52 | - KMS key usage for decryption/encryption 53 | 54 | ## Security Considerations 55 | 56 | Some security checks are bypassed with checkov skip comments because: 57 | 58 | 1. The Route53 permissions need broad access to manage DNS records 59 | 2. The IAM policy is attached directly to a user rather than a role as this fits the CircleCI deployment pattern 60 | 3. Some wildcard resources are necessary for the CI/CD pipeline functionality 61 | 4. Log buckets don't need cross-region replication as they're not critical 62 | 63 | ### Log Bucket Considerations 64 | 65 | We've made specific security decisions regarding log buckets: 66 | 67 | - **Log-of-logs bucket**: 68 | - Doesn't have its own access logging to avoid circular dependencies 69 | - Doesn't need event notifications as these are tertiary logs 70 | - Doesn't need cross-region replication as log data is less critical than state files 71 | 72 | - **Replica buckets in disaster recovery region**: 73 | - Don't need cross-region replication as they are already replicas 74 | - Log buckets for replicas don't need further access logging 75 | 76 | These decisions balance security needs with practical implementation considerations while maintaining a strong security posture for the primary state files. 77 | 78 | ## Example IAM Policy 79 | 80 | ```hcl 81 | # Create IAM user for CircleCI deployment permissions 82 | resource "aws_iam_user_policy" "circleci_deployer_policy" { 83 | name = "route53-deployment" 84 | user = "${aws_iam_user.deployer.name}" 85 | 86 | policy = <