├── .gitignore ├── README.md ├── aws_sg_auditor └── requirements.txt /.gitignore: -------------------------------------------------------------------------------- 1 | .idea/* 2 | venv 3 | rules -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # UPDATE 2 | 3 | Terraform has now much better support for this. You can even import existing security groups etc into your existing terraform. I'd suggest using terraform instead of my tool. 4 | 5 | ## AWS EC2 Security Group Auditor 6 | 7 | The purpose of the script is to compare the currently running AWS EC2 security groups with your local, audited and reviewed security groups. 8 | If the scrip finds any missing or extra rules, it will note the missing ore extra rules. 9 | 10 | ### Options 11 | 12 | * `--dump-ec2` - Rather than performing the check, dump the JSON representation of our security groups 13 | * `--json-diff` - Print the changed rules as JSON instead of English 14 | * `--nagios-output` - Return nagios compatible check result outputs 15 | * `--aws-conf`/`-a` - Path to the AWS config file. 16 | * `--rule_root`/`-r` - Path to the local rules root. 17 | * `--rule_path`/`-d` - Path to the local rules. 18 | * `--region` - AWS region to query 19 | 20 | ### AWS credentials 21 | 22 | For now the script works from AWS CLI compatible ini style config. By default it looks for `~/.aws/credentials`. I'm plan to add `STS`-`AssumedRole` support too. Stay tuned. 23 | 24 | The credentials file should look like below: 25 | 26 | ```bash 27 | [profile-name] 28 | aws_access_key_id = AKIA... 29 | aws_secret_access_key = Ze7A... 30 | ``` 31 | 32 | ### Directory structure 33 | 34 | Each security group is a separate `JSON` file under `RULE_ROOT`/`AWS_OWNER_ID`/ `AWS_REGION` directory. 35 | 36 | ### Security Group JSON files 37 | 38 | The naming convention of a rule file (if it's in a VPC) looks like: 39 | 40 | `__.json` 41 | 42 | If it's not inside of the VPC: 43 | 44 | `_.json` 45 | 46 | Once you dumped the rules, you can edit them and add comments. 47 | 48 | Lines started with `#` are considered as comments. 49 | JSON elemnts called `comments` are considered as commenst too, so feel free to use them. 50 | 51 | Note that if you dump the rules again, you'll overwrite them, so try to dump only once and maintain your rules on a regular basis. 52 | 53 | Example for structure and comments: 54 | 55 | ```json 56 | { 57 | # Our shiny web server 58 | "123456789012:web-server:sg-8b8aa991:None": { 59 | "rules:-1--1-icmp": { 60 | "comment": "allow our nagios to ping this server", 61 | "from_port": "-1", "to_port": "-1", "ip_protocol": "icmp", 62 | "grants": [ 63 | "123456789012:nagios:sg-9cede390" 64 | ] 65 | }, 66 | "rules:22-22-tcp": { 67 | "comment": "Allow SSH from the the VPN only", 68 | "from_port": "22", "to_port": "22", "ip_protocol": "tcp", 69 | "grants": [ 70 | "123456789012:openvpn:sg-44417aad" 71 | ] 72 | }, 73 | "rules:443-443-tcp": { 74 | "comment": "Allow HTTPS from everywhere", 75 | "from_port": "443", "to_port": "443", "ip_protocol": "tcp", 76 | "grants": [ "0.0.0.0/0" ] 77 | }, 78 | "rules:80-80-tcp": { 79 | "comment": "Allow HTTP from everywhere", 80 | "from_port": "80", "to_port": "80", "ip_protocol": "tcp", 81 | "grants": [ "0.0.0.0/0" ] 82 | } 83 | } 84 | } 85 | ``` 86 | 87 | ### Installation 88 | 89 | ```bash 90 | virtualenv virtualenv 91 | . virtualenv/bin/activate 92 | pip install -r requirements.txt 93 | ``` 94 | 95 | ### Examples 96 | 97 | Dump current Amazon EC2 us-east-1 security groups to `company_rules` directory using the `company-test` AWS credentials profile. 98 | 99 | ```bash 100 | $ ./aws_sg_auditor --dump-ec2 --region us-east-1 --aws-profile company-test --rule_root company_rules 101 | 102 | ``` 103 | 104 | Compare the same local rules with the current EC2 rules 105 | 106 | ```bash 107 | ./aws_sg_auditor --region us-east-1 --aws-profile cloudbees-test \ 108 | --rule_path company_rules/123456789012/us-east-1 109 | 110 | ``` 111 | 112 | 113 | ### Credits 114 | 115 | Special thanks to [Parse, Inc](http://parse.com) for the initial idea. I forked and rewrote their script. The original script - which is a nagios executable - can be found [here](https://github.com/ParsePlatform/Ops/tree/master/tools). 116 | 117 | 118 | -------------------------------------------------------------------------------- /aws_sg_auditor: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | # Copyright Adam Papai 2016 4 | # Copyright Parse, Inc. 2013 5 | # This code is made available under the Apache License 6 | # For more info, please see http://www.apache.org/licenses/LICENSE-2.0.html 7 | 8 | import argparse 9 | import ConfigParser 10 | import glob 11 | import json 12 | import os 13 | 14 | from boto.ec2 import get_region 15 | from boto.ec2.connection import EC2Connection 16 | from boto.exception import BotoServerError 17 | from sys import exit 18 | 19 | 20 | def get_ec2_rules(security_groups): 21 | groupset = {} 22 | for secgroup in security_groups: 23 | # create a unique identifier of the name, id, and vpc_id 24 | secgroup_id = "%s:%s:%s:%s" % (secgroup.owner_id, secgroup.name, secgroup.id, secgroup.vpc_id) 25 | groupset[secgroup_id] = {} 26 | for rulesetname in ["rules", "rules_egress"]: 27 | for rule in secgroup.__dict__[rulesetname]: 28 | ruleset = {} 29 | for key in ["from_port", "ip_protocol", "to_port"]: 30 | ruleset[key] = rule.__dict__[key] if rule.__dict__[key] else "_None" 31 | rule_id = rulesetname + ":" + "-".join(sorted(ruleset.values())) 32 | ruleset["grants"] = [] 33 | for grant in rule.grants: 34 | # cidr_ip will exist (ip rule) or be None (group rule) 35 | if grant.cidr_ip: 36 | ruleset["grants"].append(grant.cidr_ip) 37 | else: 38 | #it's a security group instead of a cidr; 39 | # store a string of owner:name:id (names for readability, ids and owners for uniqueness) 40 | ruleset["grants"].append("%s:%s:%s" % (grant.owner_id, grant.name, grant.groupId)) 41 | # if we're appending to an existing rule_id (it's not an empty hash), do so here. 42 | try: 43 | groupset[secgroup_id][rule_id]["grants"] += ruleset["grants"] 44 | except KeyError: 45 | groupset[secgroup_id][rule_id] = ruleset 46 | groupset[secgroup_id][rule_id]["grants"].sort() # list comes back unruly 47 | return groupset 48 | 49 | 50 | def get_local_rules(rule_file): 51 | fh = open(rule_file) 52 | comments_removed = "" 53 | for line in fh: 54 | if line.strip().startswith("#"): 55 | continue 56 | comments_removed += line 57 | groupset = json.loads(comments_removed) 58 | fh.close() 59 | # strip json-formatted comment fields 60 | for secgroup in groupset: 61 | for (rule_id, rule) in groupset[secgroup].iteritems(): 62 | rule.pop('comment', None) 63 | rule["grants"].sort() # list is unordered 64 | return groupset 65 | 66 | 67 | def get_all_local_rules(directory): 68 | all_rule = {} 69 | for rule_file in glob.glob("%s/*.json" % directory): 70 | all_rule.update(get_local_rules(rule_file)) 71 | 72 | return all_rule 73 | 74 | 75 | # give detail about where the rulesets differ 76 | def compare_rulesets(ec2_security_groups, local_security_groups): 77 | diffdetails = "" 78 | diffhash = {} 79 | # are there the same number of security groups? 80 | ec2groups = set(ec2_security_groups.keys()) 81 | localgroups = set(local_security_groups.keys()) 82 | groupdiff = ec2groups.symmetric_difference(localgroups) 83 | if groupdiff: 84 | for eachdiff in groupdiff: 85 | if eachdiff not in ec2groups: 86 | diffdetails += "security group %s is missing from ec2\n" % eachdiff 87 | diffhash[eachdiff] = local_security_groups[eachdiff] 88 | if eachdiff not in localgroups: 89 | diffdetails += "security group %s is missing from git\n" % eachdiff 90 | diffhash[eachdiff] = ec2_security_groups[eachdiff] 91 | #ok, the for each of the groups that are the same, compare the rules 92 | for group in ec2groups.intersection(localgroups): 93 | ec2rules = ec2_security_groups[group].keys() 94 | ec2rules.sort() 95 | localrules = local_security_groups[group].keys() 96 | localrules.sort() 97 | # go through the ec2 rules looking for differences and extra rules 98 | for ec2rulekey in ec2rules: 99 | ec2rule = ec2_security_groups[group].get(ec2rulekey, {}) 100 | localrule = local_security_groups[group].get(ec2rulekey, {}) 101 | if ec2rule == localrule: 102 | continue 103 | # ok, we have a rule that's different; let's try to refine the difference 104 | try: 105 | ec2ports = {ec2rule['to_port'], ec2rule['from_port'], ec2rule['ip_protocol']} 106 | except KeyError: 107 | ec2ports = None 108 | try: 109 | localports = {localrule['to_port'], localrule['from_port'], localrule['ip_protocol']} 110 | except KeyError: 111 | localports = None 112 | if ec2ports == localports: 113 | # grants mismatch 114 | ec2grants = set(ec2rule['grants']) 115 | localgrants = set(localrule['grants']) 116 | grantdiff = ec2grants.symmetric_difference(localgrants) 117 | if not grantdiff: 118 | diffdetails += "The order of grants for the %s security group, %s/%s-%s rule are different." % ( 119 | group, localrule['ip_protocol'], localrule['from_port'], localrule['to_port']) 120 | for eachgrant in grantdiff: 121 | if eachgrant not in ec2grants: 122 | diffdetails += "git has a rule that for the %s security group, %s can reach %s/%s-%s. This is missing from ec2.\n" % ( 123 | group, eachgrant, localrule['ip_protocol'], localrule['from_port'], localrule['to_port']) 124 | grouphash = diffhash.get(group, {}) 125 | rulehash = grouphash.get("%s-%s-%s" % (localrule['from_port'], localrule['to_port'], localrule['ip_protocol']), {}) 126 | rulehash['from_port'] = localrule['from_port'] 127 | rulehash['to_port'] = localrule['to_port'] 128 | rulehash['ip_protocol'] = localrule['ip_protocol'] 129 | grants = rulehash.get('grants', []) 130 | grants.append(eachgrant) 131 | rulehash['grants'] = grants 132 | grouphash["%s-%s-%s" % (localrule['from_port'], localrule['to_port'], localrule['ip_protocol'])] = rulehash 133 | diffhash[group] = grouphash 134 | 135 | if eachgrant not in localgrants: 136 | diffdetails += "ec2 has a rule that for the %s security group, %s can reach %s/%s-%s. This is missing from git.\n" % ( 137 | group, eachgrant, ec2rule['ip_protocol'], ec2rule['from_port'], ec2rule['to_port']) 138 | grouphash = diffhash.get(group, {}) 139 | rulehash = grouphash.get("%s-%s-%s" % (ec2rule['from_port'], ec2rule['to_port'], ec2rule['ip_protocol']), {}) 140 | rulehash['from_port'] = ec2rule['from_port'] 141 | rulehash['to_port'] = ec2rule['to_port'] 142 | rulehash['ip_protocol'] = ec2rule['ip_protocol'] 143 | grants = rulehash.get('grants', []) 144 | grants.append(eachgrant) 145 | rulehash['grants'] = grants 146 | rulehash['grants'] = grants 147 | grouphash["%s-%s-%s" % (ec2rule['from_port'], ec2rule['to_port'], ec2rule['ip_protocol'])] = rulehash 148 | diffhash[group] = grouphash 149 | else: 150 | # the ports didn't match. Identify the entire rule as missing. 151 | diffdetails += "ec2 has an extra rule in the %s security group: %s/%s-%s is allowed from %s\n" % ( 152 | group, ec2rule['ip_protocol'], ec2rule['from_port'], ec2rule['to_port'], ec2rule['grants']) 153 | grouphash = diffhash.get(group, {}) 154 | grouphash["%s-%s-%s" % (ec2rule['from_port'], ec2rule['to_port'], ec2rule['ip_protocol'])] = ec2rule 155 | diffhash[group] = grouphash 156 | # go through the local rules looking for extra rules (we've already found differences) 157 | for localrulekey in localrules: 158 | ec2rule = ec2_security_groups[group].get(localrulekey, {}) 159 | localrule = local_security_groups[group].get(localrulekey, {}) 160 | if localrule == ec2rule: 161 | continue 162 | try: 163 | ec2ports = {ec2rule['to_port'], ec2rule['from_port'], ec2rule['ip_protocol']} 164 | except KeyError: 165 | ec2ports = None 166 | try: 167 | localports = {localrule['to_port'], localrule['from_port'], localrule['ip_protocol']} 168 | except KeyError: 169 | localports = None 170 | if ec2ports == localports: 171 | # grants mismatch, but we've already printed this out, so skip it 172 | continue 173 | else: 174 | # the ports didn't match. Identify the entire rule as missing. 175 | diffdetails += "ec2 is missing a rule in the %s security group that is present in git: %s/%s-%s is allowed from %s\n" % ( 176 | group, localrule['ip_protocol'], localrule['from_port'], localrule['to_port'], localrule['grants']) 177 | grouphash = diffhash.get(group, {}) 178 | grouphash["%s-%s-%s" % (localrule['from_port'], localrule['to_port'], localrule['ip_protocol'])] = localrule 179 | diffhash[group] = grouphash 180 | #diffdetails += "rule exists in git but not in ec2:" 181 | return diffdetails, diffhash 182 | 183 | 184 | def dump_ruleset(rules_hash, args): 185 | for secgroup in sorted(rules_hash.keys()): 186 | account_id, name, sg_name, vpc = secgroup.split(":") 187 | group = "{\n %s\n }\n" % generate_secgroup_string(secgroup, rules_hash[secgroup]) 188 | if not os.path.exists("%s/%s/%s" % (args.rule_root, account_id, args.region)): 189 | os.makedirs("%s/%s/%s" % (args.rule_root, account_id, args.region)) 190 | if vpc != "None": 191 | filename = "%s/%s/%s/%s_%s_%s.json" % (args.rule_root, account_id, args.region, name, sg_name, vpc) 192 | else: 193 | filename = "%s/%s/%s/%s_%s.json" % (args.rule_root, account_id, args.region, name, sg_name) 194 | with open(filename, "w+") as f: 195 | f.write(group) 196 | 197 | 198 | def generate_secgroup_string(secgroup, secgroup_hash): 199 | # TODO: refactor this, it's too uggly 200 | indent = 1 201 | spaces = 4 202 | secgroup_str = "" 203 | secgroup_str += '%s"%s": {\n' % ((" "*indent*spaces), secgroup) 204 | rules = [] 205 | for rule in sorted(secgroup_hash.keys()): 206 | rules.append(generate_rule_string(rule, secgroup_hash[rule])) 207 | sep = ",\n" 208 | secgroup_str += sep.join(rules) + "\n" 209 | secgroup_str += "%s}" % (" "*indent*spaces) 210 | return secgroup_str 211 | 212 | 213 | def generate_rule_string(rule, rule_hash): 214 | # TODO: refactor this, it's too uggly 215 | indent = 2 216 | spaces = 4 217 | rule_str = "" 218 | rule_str += '%s"%s": {\n' % ((" "*indent*spaces), rule) 219 | indent = 3 220 | rule_str += '%s"from_port": "%s", "to_port": "%s", "ip_protocol": "%s",\n' % ( 221 | (" "*indent*spaces), 222 | rule_hash['from_port'], 223 | rule_hash['to_port'], 224 | rule_hash['ip_protocol']) 225 | grants = rule_hash['grants'] 226 | if len(grants) == 1: 227 | rule_str += '%s"grants": [ "%s" ]\n' % ((" "*indent*spaces), grants[0]) 228 | else: 229 | rule_str += '%s"grants": [\n%s' % ((" "*indent*spaces), (" "*(indent+1)*spaces)) 230 | indent = 4 231 | sep = ',\n%s' % (" "*indent*spaces) 232 | rule_str += sep.join(sorted(map(lambda x: '"%s"'%x, grants))) + "\n" 233 | rule_str += '%s]\n' % (" "*(indent-1)*spaces) 234 | indent = 2 235 | rule_str += '%s}' % (" "*indent*spaces) 236 | return rule_str 237 | 238 | 239 | def load_aws_config(args): 240 | try: 241 | config = ConfigParser.ConfigParser() 242 | config.read(os.path.expanduser(args.aws_conf)) 243 | aws_access_key_id = config.get(args.profile, "aws_access_key_id") 244 | aws_secret_access_key = config.get(args.profile, "aws_secret_access_key") 245 | return aws_access_key_id, aws_secret_access_key 246 | except IOError, e: 247 | print "CRITICAL: %s" % e 248 | exit(2) 249 | 250 | 251 | def main(): 252 | parser = argparse.ArgumentParser(description="Check our ec2 security groups against the local rules" ) 253 | parser.add_argument('--dump-ec2', dest='dump_ec2', action="store_true", default=False, 254 | help="Rather than performing the check, dump the JSON representation of our security groups") 255 | parser.add_argument('--json-diff', dest='json_diff', action="store_true", default=False, 256 | help="Print the changed rules as JSON instead of English") 257 | parser.add_argument('--nagios-output', dest='nagios_format', action="store_true", default=False, 258 | help="Return nagios compatible check result outputs") 259 | parser.add_argument('-a', '--aws-conf', dest='aws_conf', action="store", default='~/.aws/credentials', 260 | help="Path to the AWS config file") 261 | parser.add_argument('-p', '--aws-profile', dest='profile', action='store', default="default") 262 | parser.add_argument('-r', '--rule_root', dest='rule_root', action='store', default='rules', 263 | help="Path to the local rules root") 264 | parser.add_argument('-d', '--rule_path', dest='rule_path', action='store', 265 | help="Path to the local rules directory") 266 | parser.add_argument('--region', dest='region', action="store", default='us-east-1', help="AWS region to query") 267 | args = parser.parse_args() 268 | 269 | aws_access_key_id, aws_secret_access_key = load_aws_config(args) 270 | 271 | try: 272 | selected_region = get_region(args.region, 273 | aws_access_key_id=aws_access_key_id, 274 | aws_secret_access_key=aws_secret_access_key) 275 | 276 | conn = EC2Connection(aws_access_key_id=aws_access_key_id, 277 | aws_secret_access_key=aws_secret_access_key, 278 | region=selected_region) 279 | 280 | security_groups = conn.get_all_security_groups() 281 | 282 | ec2set = get_ec2_rules(security_groups) 283 | 284 | if args.dump_ec2: 285 | dump_ruleset(ec2set, args) 286 | exit(0) 287 | 288 | localset = get_all_local_rules(args.rule_path) 289 | 290 | if ec2set != localset: 291 | if args.nagios_format: 292 | print "CRITICAL: ec2 running security groups are different from the local rules." 293 | (diffstring, diffhash) = compare_rulesets(ec2set, localset) 294 | if len(diffstring) == 0: 295 | print "The rulesets are not the same but I can't find the difference. " \ 296 | "This might be a false positive. Dump the rulesets and compare by hand." 297 | print diffstring 298 | 299 | if args.json_diff: 300 | print "This output is the union of all differences; " \ 301 | "missing and added rules from either location are all present here." 302 | dump_ruleset(diffhash) 303 | exit(2) 304 | else: 305 | if args.nagios_format: 306 | print "OK - security groups match with local rules." 307 | exit(0) 308 | except BotoServerError, e: 309 | print "CRITICAL: %s" % e 310 | exit(2) 311 | except ValueError, e: 312 | print "CRITICAL: %s" % e 313 | print "Check your JSON for wayward commas and things. The line number printed ignores comments (so is too low)." 314 | exit(2) 315 | 316 | 317 | main() -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | boto --------------------------------------------------------------------------------