├── .circleci └── config.yml ├── .gitignore ├── CONTRIBUTING.md ├── LICENSE.md ├── README.md ├── assume_role_policy.json ├── log_policy.json ├── main.tf ├── output.tf ├── test ├── Pipfile ├── Pipfile.lock ├── ec2.tf ├── network.tf ├── output.tf ├── test.py └── vars.tf └── variables.tf /.circleci/config.yml: -------------------------------------------------------------------------------- 1 | version: 2 2 | jobs: 3 | build: 4 | docker: 5 | - image: hashicorp/terraform 6 | steps: 7 | - checkout 8 | - run: 9 | name: Set up Terraform 10 | command: terraform init -backend=false test 11 | - run: 12 | name: Validate Terraform 13 | command: terraform validate -check-variables=false test 14 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.tfvars 2 | # Compiled files 3 | *.tfstate* 4 | # Module directory 5 | .terraform/ 6 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | ## Welcome! 2 | 3 | We're so glad you're thinking about contributing to an 18F open source project! If you're unsure about anything, just ask -- or submit the issue or pull request anyway. The worst that can happen is you'll be politely asked to change something. We love all friendly contributions. 4 | 5 | We want to ensure a welcoming environment for all of our projects. Our staff follow the [18F Code of Conduct](https://github.com/18F/code-of-conduct/blob/master/code-of-conduct.md) and all contributors should do the same. 6 | 7 | We encourage you to read this project's CONTRIBUTING policy (you are here), its [LICENSE](LICENSE.md), and its [README](README.md). 8 | 9 | If you have any questions or want to read more, check out the [18F Open Source Policy GitHub repository](https://github.com/18f/open-source-policy), or just [shoot us an email](mailto:18f@gsa.gov). 10 | 11 | ## Public domain 12 | 13 | This project is in the public domain within the United States, and 14 | copyright and related rights in the work worldwide are waived through 15 | the [CC0 1.0 Universal public domain dedication](https://creativecommons.org/publicdomain/zero/1.0/). 16 | 17 | All contributions to this project will be released under the CC0 18 | dedication. By submitting a pull request, you are agreeing to comply 19 | with this waiver of copyright interest. 20 | 21 | ## Testing 22 | 23 | Requires Python 3, pip, and Terraform. Note you may need to [configure AWS environment variables](https://www.terraform.io/docs/providers/aws/#environment-variables) first. 24 | 25 | ```sh 26 | cd test 27 | 28 | pip install pipenv 29 | pipenv install 30 | pipenv shell 31 | 32 | terraform apply 33 | python test.py 34 | ``` 35 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | As a work of the United States government, this project is in the 2 | public domain within the United States. 3 | 4 | Additionally, we waive copyright and related rights in the work 5 | worldwide through the CC0 1.0 Universal public domain dedication. 6 | 7 | ## CC0 1.0 Universal summary 8 | 9 | This is a human-readable summary of the [Legal Code (read the full text)](https://creativecommons.org/publicdomain/zero/1.0/legalcode). 10 | 11 | ### No copyright 12 | 13 | The person who associated a work with this deed has dedicated the work to 14 | the public domain by waiving all rights to the work worldwide 15 | under copyright law, including all related and neighboring rights, to the 16 | extent allowed by law. 17 | 18 | You can copy, modify, distribute and perform the work, even for commercial 19 | purposes, all without asking permission. 20 | 21 | ### Other information 22 | 23 | In no way are the patent or trademark rights of any person affected by CC0, 24 | nor are the rights that other persons may have in the work or in how the 25 | work is used, such as publicity or privacy rights. 26 | 27 | Unless expressly stated otherwise, the person who associated a work with 28 | this deed makes no warranties about the work, and disclaims liability for 29 | all uses of the work, to the fullest extent permitted by applicable law. 30 | When using or citing the work, you should not imply endorsement by the 31 | author or the affirmer. 32 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # AWS VPC Flow Log module for Terraform [![CircleCI](https://circleci.com/gh/GSA/terraform-vpc-flow-log.svg?style=svg)](https://circleci.com/gh/GSA/terraform-vpc-flow-log) 2 | 3 | This is a reusable Terraform module for setting up [VPC flow logs](https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/flow-logs.html) in Amazon Web Services. 4 | 5 | ## Usage 6 | 7 | ```hcl 8 | module "flow_logs" { 9 | source = "github.com/GSA/terraform-vpc-flow-log" 10 | vpc_id = "${aws_vpc.main.id}" 11 | } 12 | ``` 13 | 14 | See the [optional variables](variables.tf). 15 | -------------------------------------------------------------------------------- /assume_role_policy.json: -------------------------------------------------------------------------------- 1 | { 2 | "Version": "2012-10-17", 3 | "Statement": [ 4 | { 5 | "Sid": "", 6 | "Effect": "Allow", 7 | "Principal": { 8 | "Service": "vpc-flow-logs.amazonaws.com" 9 | }, 10 | "Action": "sts:AssumeRole" 11 | } 12 | ] 13 | } 14 | -------------------------------------------------------------------------------- /log_policy.json: -------------------------------------------------------------------------------- 1 | { 2 | "Version": "2012-10-17", 3 | "Statement": [ 4 | { 5 | "Action": [ 6 | "logs:CreateLogGroup", 7 | "logs:CreateLogStream", 8 | "logs:PutLogEvents", 9 | "logs:DescribeLogGroups", 10 | "logs:DescribeLogStreams" 11 | ], 12 | "Effect": "Allow", 13 | "Resource": "*" 14 | } 15 | ] 16 | } 17 | -------------------------------------------------------------------------------- /main.tf: -------------------------------------------------------------------------------- 1 | // https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/flow-logs.html#flow-logs-iam 2 | 3 | data "template_file" "assume_role_policy" { 4 | template = "${file("${path.module}/assume_role_policy.json")}" 5 | } 6 | 7 | data "template_file" "log_policy" { 8 | template = "${file("${path.module}/log_policy.json")}" 9 | } 10 | 11 | resource "aws_iam_role" "iam_log_role" { 12 | name = "${var.prefix}-flow-log-role" 13 | assume_role_policy = "${data.template_file.assume_role_policy.rendered}" 14 | } 15 | 16 | resource "aws_iam_role_policy" "log_policy" { 17 | name = "${var.prefix}-flow-log-policy" 18 | role = "${aws_iam_role.iam_log_role.id}" 19 | policy = "${data.template_file.log_policy.rendered}" 20 | } 21 | 22 | 23 | resource "aws_cloudwatch_log_group" "flow_log_group" { 24 | name = "${var.log_group_name == "" ? local.default_log_group_name : var.log_group_name}" 25 | } 26 | 27 | resource "aws_flow_log" "vpc_flow_log" { 28 | log_group_name = "${aws_cloudwatch_log_group.flow_log_group.name}" 29 | iam_role_arn = "${aws_iam_role.iam_log_role.arn}" 30 | vpc_id = "${var.vpc_id}" 31 | traffic_type = "${var.traffic_type}" 32 | } 33 | -------------------------------------------------------------------------------- /output.tf: -------------------------------------------------------------------------------- 1 | output "log_group_name" { 2 | value = "${aws_flow_log.vpc_flow_log.log_group_name}" 3 | } 4 | -------------------------------------------------------------------------------- /test/Pipfile: -------------------------------------------------------------------------------- 1 | [[source]] 2 | 3 | url = "https://pypi.python.org/simple" 4 | verify_ssl = true 5 | name = "pypi" 6 | 7 | 8 | [dev-packages] 9 | 10 | 11 | 12 | [packages] 13 | 14 | "boto3" = "*" 15 | retrying = "~=1.3" 16 | 17 | 18 | [requires] 19 | 20 | python_version = "3.5" 21 | -------------------------------------------------------------------------------- /test/Pipfile.lock: -------------------------------------------------------------------------------- 1 | { 2 | "_meta": { 3 | "hash": { 4 | "sha256": "20c1b79876d90b89effe671beb5406e0af744f2b9aacef8bcfbb2e44efa0ee80" 5 | }, 6 | "host-environment-markers": { 7 | "implementation_name": "cpython", 8 | "implementation_version": "3.5.1", 9 | "os_name": "posix", 10 | "platform_machine": "x86_64", 11 | "platform_python_implementation": "CPython", 12 | "platform_release": "16.7.0", 13 | "platform_system": "Darwin", 14 | "platform_version": "Darwin Kernel Version 16.7.0: Wed Oct 4 00:17:00 PDT 2017; root:xnu-3789.71.6~1/RELEASE_X86_64", 15 | "python_full_version": "3.5.1", 16 | "python_version": "3.5", 17 | "sys_platform": "darwin" 18 | }, 19 | "pipfile-spec": 6, 20 | "requires": { 21 | "python_version": "3.5" 22 | }, 23 | "sources": [ 24 | { 25 | "name": "pypi", 26 | "url": "https://pypi.python.org/simple", 27 | "verify_ssl": true 28 | } 29 | ] 30 | }, 31 | "default": { 32 | "boto3": { 33 | "hashes": [ 34 | "sha256:38057b066990172ce6ebbf2a5e046a545503793581fcf14cab0e3821c6112eb0", 35 | "sha256:f79f77dca2280f7780f39d72a5088f4cf2b626c0921e7185ed6ac17abfdd7e6c" 36 | ], 37 | "version": "==1.4.7" 38 | }, 39 | "botocore": { 40 | "hashes": [ 41 | "sha256:4f8b432463299c5b7718d1e32bcb48201d8e58761e3b7c9d7d17d62e42246c5d", 42 | "sha256:73895086a6f42d8ee0a139b068f0b6f2a8d7da3bb06e65083f37a4c274c2c458" 43 | ], 44 | "version": "==1.7.45" 45 | }, 46 | "docutils": { 47 | "hashes": [ 48 | "sha256:7a4bd47eaf6596e1295ecb11361139febe29b084a87bf005bf899f9a42edc3c6", 49 | "sha256:02aec4bd92ab067f6ff27a38a38a41173bf01bed8f89157768c1573f53e474a6", 50 | "sha256:51e64ef2ebfb29cae1faa133b3710143496eca21c530f3f71424d77687764274" 51 | ], 52 | "version": "==0.14" 53 | }, 54 | "futures": { 55 | "hashes": [ 56 | "sha256:c4884a65654a7c45435063e14ae85280eb1f111d94e542396717ba9828c4337f", 57 | "sha256:51ecb45f0add83c806c68e4b06106f90db260585b25ef2abfcda0bd95c0132fd" 58 | ], 59 | "markers": "python_version == '2.6' or python_version == '2.7'", 60 | "version": "==3.1.1" 61 | }, 62 | "jmespath": { 63 | "hashes": [ 64 | "sha256:f11b4461f425740a1d908e9a3f7365c3d2e569f6ca68a2ff8bc5bcd9676edd63", 65 | "sha256:6a81d4c9aa62caf061cb517b4d9ad1dd300374cd4706997aff9cd6aedd61fc64" 66 | ], 67 | "version": "==0.9.3" 68 | }, 69 | "python-dateutil": { 70 | "hashes": [ 71 | "sha256:95511bae634d69bc7329ba55e646499a842bc4ec342ad54a8cdb65645a0aad3c", 72 | "sha256:891c38b2a02f5bb1be3e4793866c8df49c7d19baabf9c1bad62547e0b4866aca" 73 | ], 74 | "version": "==2.6.1" 75 | }, 76 | "retrying": { 77 | "hashes": [ 78 | "sha256:08c039560a6da2fe4f2c426d0766e284d3b736e355f8dd24b37367b0bb41973b" 79 | ], 80 | "version": "==1.3.3" 81 | }, 82 | "s3transfer": { 83 | "hashes": [ 84 | "sha256:c7b16f4cca5acd2bd57ac9623bfba3fece047247392893506d0d2e6f25620eb3", 85 | "sha256:76f1f58f4a47e2c8afa135e2c76958806a3abbc42b721d87fd9d11409c75d979" 86 | ], 87 | "version": "==0.1.11" 88 | }, 89 | "six": { 90 | "hashes": [ 91 | "sha256:832dc0e10feb1aa2c68dcc57dbb658f1c7e65b9b61af69048abc87a2db00a0eb", 92 | "sha256:70e8a77beed4562e7f14fe23a786b54f6296e34344c23bc42f07b15018ff98e9" 93 | ], 94 | "version": "==1.11.0" 95 | } 96 | }, 97 | "develop": {} 98 | } 99 | -------------------------------------------------------------------------------- /test/ec2.tf: -------------------------------------------------------------------------------- 1 | resource "aws_key_pair" "deployer" { 2 | key_name_prefix = "${var.prefix}-deployer-key" 3 | public_key = "${file("~/.ssh/id_rsa.pub")}" 4 | } 5 | 6 | data "aws_ami" "ubuntu" { 7 | most_recent = true 8 | 9 | filter { 10 | name = "name" 11 | values = ["ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-*"] 12 | } 13 | 14 | filter { 15 | name = "virtualization-type" 16 | values = ["hvm"] 17 | } 18 | 19 | owners = ["099720109477"] # Canonical 20 | } 21 | 22 | resource "aws_security_group" "main" { 23 | vpc_id = "${module.network.vpc_id}" 24 | 25 | # SSH access from anywhere 26 | ingress { 27 | from_port = 22 28 | to_port = 22 29 | protocol = "tcp" 30 | cidr_blocks = ["0.0.0.0/0"] 31 | } 32 | } 33 | 34 | resource "aws_instance" "main" { 35 | ami = "${data.aws_ami.ubuntu.id}" 36 | instance_type = "t2.micro" 37 | 38 | subnet_id = "${module.network.public_subnets[0]}" 39 | vpc_security_group_ids = ["${aws_security_group.main.id}"] 40 | 41 | key_name = "${aws_key_pair.deployer.key_name}" 42 | 43 | provisioner "remote-exec" { 44 | inline = ["echo Successfully connected"] 45 | 46 | connection { 47 | user = "ubuntu" 48 | } 49 | } 50 | } 51 | -------------------------------------------------------------------------------- /test/network.tf: -------------------------------------------------------------------------------- 1 | data "aws_region" "current" { 2 | current = true 3 | } 4 | 5 | module "network" { 6 | source = "terraform-aws-modules/vpc/aws" 7 | version = "~> 1.0" 8 | 9 | name = "terraform-vpc-flow-log-test" 10 | azs = ["${data.aws_region.current.name}d"] 11 | cidr = "10.0.0.0/16" 12 | public_subnets = ["10.0.1.0/24"] 13 | enable_nat_gateway = true 14 | } 15 | 16 | module "flow_logs" { 17 | source = "../" 18 | vpc_id = "${module.network.vpc_id}" 19 | prefix = "${var.prefix}" 20 | } 21 | -------------------------------------------------------------------------------- /test/output.tf: -------------------------------------------------------------------------------- 1 | output "log_group_name" { 2 | value = "${module.flow_logs.log_group_name}" 3 | } 4 | 5 | output "ip" { 6 | value = "${aws_instance.main.public_ip}" 7 | } 8 | -------------------------------------------------------------------------------- /test/test.py: -------------------------------------------------------------------------------- 1 | import boto3 2 | from retrying import retry 3 | import socket 4 | import subprocess 5 | import unittest 6 | import urllib 7 | import warnings 8 | 9 | 10 | class TestVpcFlowLog(unittest.TestCase): 11 | 12 | def can_connect_to_port(self, host, port): 13 | # https://stackoverflow.com/a/20541919 14 | s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 15 | result = s.connect_ex((host, port)) 16 | s.close() 17 | return result == 0 18 | 19 | def get_terraform_output(self, name): 20 | cmd = ['terraform', 'output', name] 21 | result = subprocess.run(cmd, check=True, stdout=subprocess.PIPE) 22 | return result.stdout.decode('utf-8').strip() 23 | 24 | def get_log_group(self): 25 | return self.get_terraform_output('log_group_name') 26 | 27 | def is_result_empty(result): 28 | return not result 29 | 30 | # VPC flow logs can take a while to show up (over 22 minutes on 11/14/17), so retry until we get something 31 | @retry( 32 | retry_on_result=is_result_empty, 33 | wait_fixed=(5 * 1000), 34 | stop_max_delay=(25 * 60 * 1000) 35 | ) 36 | def get_flow_logs(self): 37 | """Get flow logs pertaining to the test machine's IP address.""" 38 | 39 | # indicate each try 40 | print('.', end='', flush=True) 41 | 42 | client = boto3.client('logs') 43 | 44 | log_group = self.get_log_group() 45 | ip = self.get_my_ip() 46 | 47 | response = client.filter_log_events( 48 | logGroupName=log_group, 49 | # only show events pertaining to the test machine 50 | filterPattern=ip, 51 | limit=1 52 | ) 53 | return response['events'] 54 | 55 | def get_my_ip(self): 56 | url = 'http://checkip.amazonaws.com/' 57 | with urllib.request.urlopen(url) as response: 58 | content = response.read() 59 | return content.decode('utf-8').strip() 60 | 61 | # Terraform will connect to the instance through its `provisioner`, but trigger here just in case that wasn't run from the same IP or something 62 | def test_connect_to_test_instance(self): 63 | instance_ip = self.get_terraform_output('ip') 64 | self.assertTrue(self.can_connect_to_port(instance_ip, 22)) 65 | 66 | def test_events_present(self): 67 | print("\nFetching events", end='', flush=True) 68 | 69 | # workaround for https://github.com/boto/boto3/issues/454 70 | with warnings.catch_warnings(): 71 | warnings.simplefilter('ignore', ResourceWarning) 72 | 73 | events = self.get_flow_logs() 74 | 75 | print('') 76 | self.assertGreater(len(events), 0) 77 | 78 | if __name__ == '__main__': 79 | unittest.main() 80 | -------------------------------------------------------------------------------- /test/vars.tf: -------------------------------------------------------------------------------- 1 | variable "prefix" { 2 | default = "test-flow-log" 3 | } 4 | -------------------------------------------------------------------------------- /variables.tf: -------------------------------------------------------------------------------- 1 | variable "vpc_id" {} 2 | 3 | variable "prefix" { 4 | description = "The prefix for the resource names. You will probably want to set this to the name of your VPC, if you have multiple." 5 | default = "vpc" 6 | } 7 | 8 | variable "traffic_type" { 9 | default = "ALL" 10 | description = "https://www.terraform.io/docs/providers/aws/r/flow_log.html#traffic_type" 11 | } 12 | 13 | // workaround for not being able to do interpolation in variable defaults 14 | // https://github.com/hashicorp/terraform/issues/4084 15 | locals { 16 | default_log_group_name = "${var.prefix}-flow-log" 17 | } 18 | variable "log_group_name" { 19 | default = "" 20 | description = "Defaults to `$${default_log_group_name}`" 21 | } 22 | --------------------------------------------------------------------------------