├── .github └── PULL_REQUEST_TEMPLATE.md ├── .gitignore ├── LICENSE ├── MANIFEST.in ├── NOTICE ├── README.md ├── README.rst ├── TESTS.txt ├── TODO.txt ├── aws_with ├── __init__.py ├── cli.py ├── commands.py ├── main.py ├── monkey.py ├── organizations.py ├── output.py ├── regions.py ├── utils.py └── workplan.py ├── buildspec.yml ├── examples ├── enable_guardduty_with_sns_email.py └── show_spot_prices_globally.py ├── pipeline.cfg ├── run-test-pipeline.sh ├── setup.cfg ├── setup.py ├── tests └── test_one.py └── testspec.yml /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | *Issue #, if available:* 2 | 3 | *Description of changes:* 4 | 5 | 6 | By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. 7 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | dist/ 2 | build/ 3 | .cache/ 4 | .tox/ 5 | aws_with.egg-info/ 6 | *.pyc 7 | __pycache__ 8 | pipeline.cfg 9 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | 177 | END OF TERMS AND CONDITIONS 178 | 179 | APPENDIX: How to apply the Apache License to your work. 180 | 181 | To apply the Apache License to your work, attach the following 182 | boilerplate notice, with the fields enclosed by brackets "[]" 183 | replaced with your own identifying information. (Don't include 184 | the brackets!) The text should be enclosed in the appropriate 185 | comment syntax for the file format. We also recommend that a 186 | file or class name and description of purpose be included on the 187 | same "printed page" as the copyright notice for easier 188 | identification within third-party archives. 189 | 190 | Copyright [yyyy] [name of copyright owner] 191 | 192 | Licensed under the Apache License, Version 2.0 (the "License"); 193 | you may not use this file except in compliance with the License. 194 | You may obtain a copy of the License at 195 | 196 | http://www.apache.org/licenses/LICENSE-2.0 197 | 198 | Unless required by applicable law or agreed to in writing, software 199 | distributed under the License is distributed on an "AS IS" BASIS, 200 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 201 | See the License for the specific language governing permissions and 202 | limitations under the License. 203 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include LICENSE.txt 2 | -------------------------------------------------------------------------------- /NOTICE: -------------------------------------------------------------------------------- 1 | aws-with 2 | Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | aws-with 2 | 3 | ## License 4 | 5 | This library is licensed under the Apache 2.0 License. 6 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | aws_with 2 | ======== 3 | 4 | aws_with is a command line utility to help manage large and complex AWS environments. 5 | 6 | It can run the same command across a number of AWS regions, accounts and can traverse AWS Organizations. 7 | 8 | To get started:: 9 | 10 | sudo pip install aws_with 11 | aws_with --help 12 | 13 | 14 | -------------------------------------------------------------------------------- /TESTS.txt: -------------------------------------------------------------------------------- 1 | - tests: 2 | - aws_with -R us-east-1 uptime 3 | - aws_with -R 'ap*' uptime 4 | - aws_with -R 'us-*,*south*' uptime 5 | - aws_with -R 'us-*' aws ec2 describe-instances 6 | - aws_with -q -R 'us-*' aws ec2 describe-instances 7 | - aws_with -q -R 'ap-*' aws ec2 describe-instances 8 | - aws_with -a 833567632276 9 | - aws_with -a 912793868281 10 | - aws_with -a 833567632276 -r OrganizationAccountAccessRole 11 | - aws_with -a 137507561461,200616150143,912793868281 uptime 12 | - aws_with -r role-bastion-server 13 | - aws_with -r admin 14 | - aws_with -o / uptime 15 | - aws_with -o /Technology uptime 16 | - aws_with -o '/Technology/Administrative Accounts' uptime 17 | - aws_with -o /Marketing uptime 18 | - aws_with -o '/Technology/Digital Bank Products/eCommerce Sites' uptime 19 | - aws_with -x -o /Marketing,/Technology uptime 20 | - aws_with -x -o '/Technology/Digital Bank Products/eCommerce Sites' uptime 21 | - aws_with -m -x -o / uptime 22 | - aws_with -x -o / uptime 23 | - aws_with -t 0 -R '*' uptime 24 | - time aws_with -t1 -R '*' sleep 2 25 | - time aws_with -t10 -R '*' sleep 2 26 | - aws_with -f json -R 'us-east-*' uptime 27 | - aws_with -f yaml -R 'us-east-*' uptime 28 | - aws_with -f text -R 'us-east-*' uptime 29 | - aws_with -f yaml -R 'ap-south*' aws ec2 describe-instances 30 | - aws_with -f yaml -R 'ap-south*' ec2 describe-instances 31 | - aws_with -g -f yaml -R 'ap-south*' ec2 describe-instances 32 | - aws_with -g -e -t1 -f yaml -R 'ap-south*' ec2 describe-instances 33 | - aws_with -p read_only_fred -R 'ap-south*' aws sts get-caller-identity 34 | - aws_with -p read_only_fred -R 'ap-south*' aws s3 ls 35 | -------------------------------------------------------------------------------- /TODO.txt: -------------------------------------------------------------------------------- 1 | - move ideas and TODO tasks into GitHub issues and then remove this file 2 | 3 | - use codebuild and transient docker images 4 | - use this: aws/codebuild/eb-python-3.4-amazonlinux-64:2.1.6 5 | - see: http://docs.aws.amazon.com/codebuild/latest/userguide/build-env-ref.html 6 | - need to: 7 | - code build to take amazonlinux container 8 | - then add various build / testing tools and python versions 9 | - then pipeline can use that docker container (how do I get an amazonlinux in there in the first place) 10 | 11 | - think about version number tracking for "candidate" vs. actual PIP releases 12 | - probably easiest to query "pip" to get latest version and abort if current is not newer 13 | - can make the pipeline publish for manual testing and manual approval and version update to PIP publish? 14 | - inject build information into __init__.py 15 | 16 | - read up on python "coverage" tool 17 | 18 | - environment setup 19 | - install versions of python 20 | - sudo yum -y install python26 python26-pip 21 | - sudo yum -y install python27 python27-pip 22 | - sudo yum -y install python34 python34-pip 23 | - sudo yum -y install python35 python35-pip 24 | - python 3.6 will need manual installing 25 | - sudo yum -y install gcc zlib-devel 26 | - mkdir /tmp/py36 ; cd /tmp/py36 27 | - wget https://www.python.org/ftp/python/3.6.2/Python-3.6.2.tgz 28 | - tar xzf Python-3.6.2.tgz 29 | - cd Python-3.6.2 30 | - ./configure --prefix=/usr 31 | - sudo make altinstall 32 | - cd ; sudo rm -rf /tmp/py36 33 | - sudo ln -s /usr/bin/pip3.6 /usr/bin/pip-3.6 34 | - # sudo ln -s /usr/local/bin/python3.6 /usr/bin/python36 35 | - # sudo ln -s /usr/local/bin/pip3.6 /usr/bin/pip-3.6 36 | - sudo pip-3.5 install twine 37 | - sudo pip-3.5 install tox 38 | - tox -e py26,py27,py34,py35,py36 39 | - pip install twine 40 | - pip install tox 41 | 42 | - build 43 | - inject build time and git commit hash into the source code: __init__.py 44 | - CODEBUILD_RESOLVED_SOURCE_VERSION has the git commit hash 45 | - TZ=UTC date '+%Y-%m-%d %H:%M:%S' 46 | - check version is GIT is newer than latest published pip version 47 | - import aws_with 48 | - import requests 49 | - import pkg_resources 50 | - aws_with.VERSION 51 | - pypi_version=requests.get("https://pypi.org/pypi/aws-with/json") 52 | - print(pypi_version.status_code) # make sure == 200 53 | - versions=pypi_version.json()["releases"].keys() 54 | - sorted(version, key=pkg_resources.parse_version, reverse=True)[0] 55 | 56 | - unit test 57 | - INPUT: dist/aws_with-[0-9.]*.tar.gz 58 | - python setup.py install 59 | - read up on testing with tox 60 | - info: https://pypi.org/project/twine/ 61 | - http://tox.readthedocs.io/en/latest/example/basic.html 62 | - https://tox.readthedocs.io/en/latest/examples.html 63 | 64 | - integration test 65 | - read up on testing with tox 66 | 67 | - release 68 | - twine upload dist/aws_with-0.9.linux-x86_64.tar.gz 69 | - tag the git repo hash with the released version number 70 | 71 | -------------------------------------------------------------------------------- /aws_with/__init__.py: -------------------------------------------------------------------------------- 1 | """ aws_with module definition """ 2 | 3 | #from . import utils 4 | #from . import regions 5 | #from . import organizations 6 | #from . import commands 7 | #from . import workplan 8 | #from . import output 9 | #from . import cli 10 | #from . import main 11 | 12 | VERSION = "0.9.9" 13 | BUILD_TIME = "2018-01-08 06:12:42" 14 | BUILD_COMMIT_HASH = "4542b1251b20d908dc706c05c98b70ed8dfc2710" 15 | 16 | __title__ = "aws_with" 17 | __version__ = VERSION 18 | __summary__ = "aws_with command line utility" 19 | __license__ = "APACHE2" 20 | __uri__ = "https://aws.amazon.com/" 21 | __author__ = "Mike Evans" 22 | __email__ = "miev@amazon.com" 23 | -------------------------------------------------------------------------------- /aws_with/cli.py: -------------------------------------------------------------------------------- 1 | """ 2 | Licensed to the Apache Software Foundation (ASF) under one 3 | or more contributor license agreements. See the NOTICE file 4 | distributed with this work for additional information 5 | regarding copyright ownership. The ASF licenses this file 6 | to you under the Apache License, Version 2.0 (the 7 | "License"); you may not use this file except in compliance 8 | with the License. You may obtain a copy of the License at 9 | 10 | http://www.apache.org/licenses/LICENSE-2.0 11 | 12 | Unless required by applicable law or agreed to in writing, 13 | software distributed under the License is distributed on an 14 | "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY 15 | KIND, either express or implied. See the License for the 16 | specific language governing permissions and limitations 17 | under the License. 18 | """ 19 | 20 | import sys 21 | import argparse 22 | from . import utils 23 | 24 | 25 | def create_args_parser(): 26 | """ process command line arguments and perform sanity checks """ 27 | parser = argparse.ArgumentParser(description="Run the same command across a number of " 28 | "AWS accounts and/or regions.", 29 | usage="%(prog)s [options] [COMMAND]", 30 | formatter_class=argparse.RawDescriptionHelpFormatter, 31 | epilog=""" 32 | NOTE: -R, -o and -a can take a comma-separated list or the option may be given multiple times. 33 | 34 | EXAMPLES: 35 | 36 | aws_with --role=admins-role 37 | Assume the IAM role 'admins-role' within the current AWS account and then run a shell. 38 | 39 | aws_with --role=admins-role --accounts=123456789012 40 | Assume the IAM role 'admins-role' in the account 123456789012 and then run a shell. 41 | 42 | aws_with --regions='us-*' ec2 describe-instances 43 | List all EC2 instances across all US regions. 44 | 45 | aws_with --ous=/Technology/Development s3 ls 46 | List all S3 buckets owned by any account under the Organizations OU 'Development'. 47 | The 'Development' OU is itself under the 'Technology' OU off the Root OU. 48 | 49 | aws_with -o /Production,/Staging/Final -R '*' \\ 50 | cloudformation update-stack --stack-name 'security-checks' ... 51 | Update a stack across all accounts under the 'Production' Organizations OU 52 | or under the /Staging/Final OU and across all regions. 53 | """) 54 | 55 | parser.add_argument("-V", "--version", 56 | dest="show_version", action="store_true", 57 | help="Show the version number and exit") 58 | 59 | parser.add_argument("-R", "--regions", 60 | dest="regions", action="append", 61 | help="Run the command for all regions that match PATTERNS") 62 | 63 | parser.add_argument("-r", "--role", 64 | dest="role", action="store", 65 | help="Use STS assumeRole to take on a different IAM role. If not " 66 | "specified then ROLE defaults to OrganizationAccountAccessRole") 67 | 68 | parser.add_argument("-o", "--ous", 69 | dest="ous", action="append", 70 | help="Run the command for all child accounts under " 71 | "the Organizations OU PATHS") 72 | 73 | parser.add_argument("-a", "--accounts", 74 | dest="accounts", action="append", 75 | help="Run the command for all the listed ACCOUNTS") 76 | 77 | parser.add_argument("-x", "--no-recursive", 78 | dest="no_recursive", action="store_true", 79 | help="When scanning Organizations for accounts, don't look recursively") 80 | 81 | parser.add_argument("-t", "--threads", 82 | dest="threads", action="store", default=2, type=int, 83 | help="Set the number of threads to use when running commands (default: 2)") 84 | 85 | parser.add_argument("-f", "--output", 86 | dest="format", action="store", default="json", 87 | type=str, choices=["json", "yaml", "text"], 88 | help="Set the output format to use") 89 | 90 | parser.add_argument("-q", "--quiet", 91 | dest="quiet", action="store_true", 92 | help="Suppress output if a command is successful but has no output") 93 | 94 | parser.add_argument("-e", "--stop-on-error", 95 | dest="stop_on_error", action="store_true", 96 | help="Stop running commands if one throws an error") 97 | 98 | parser.add_argument("-v", "--verbose", 99 | dest="verbosity", action="count", 100 | help="Output debug messages, increase messages with -v -v") 101 | 102 | parser.add_argument("-g", "--no-cli-guess", 103 | dest="no_cli_guess", action="store_true", 104 | help="Do not attempt to guess if the command is an AWS CLI command") 105 | 106 | parser.add_argument("-m", "--no-master", 107 | dest="no_master", action="store_true", 108 | help="Do not include the Organizations master account in searches") 109 | 110 | parser.add_argument("-p", "--profile", 111 | dest="profile", action="store", 112 | help="Use the saved AWS credentials/profile called PROFILE") 113 | 114 | parser.add_argument("command", nargs=argparse.REMAINDER, metavar="COMMAND", 115 | help="This is the command that should be run across regions/accounts/OUs." 116 | " COMMAND is mandatory when the -R or -o options are used, " 117 | "otherwise it is optional and a shell will be run if it is omitted.") 118 | 119 | return parser 120 | 121 | def show_version(): 122 | """ display version information and then exit """ 123 | import os 124 | import inspect 125 | import awscli 126 | import boto3 127 | import botocore 128 | import subprocess 129 | print("aws_with version: {}".format(sys.modules["aws_with"].VERSION)) 130 | print("aws_with key libraries:") 131 | print(" aws {} from {}".format( 132 | subprocess.check_output(['aws','--version'], stderr=subprocess.STDOUT).replace('\n',''), 133 | subprocess.check_output(['which','aws']).replace('\n','') 134 | )) 135 | print(" awscli {} from {}".format(awscli.__version__, os.path.dirname(inspect.getfile(awscli)))) 136 | print(" boto3 {} from {}".format(boto3.__version__, os.path.dirname(inspect.getfile(boto3)))) 137 | print(" botocore {} from {}".format(botocore.__version__, os.path.dirname(inspect.getfile(botocore)))) 138 | print(" python {} from {}".format(sys.version.replace('\n',''), sys.executable)) 139 | sys.exit(0) 140 | 141 | def error(message): 142 | """ display an error related to command line arguments and quit """ 143 | print(message + "\nhint: try -h for help") 144 | sys.exit(1) 145 | 146 | def args_basic_checks(parsed_options): 147 | """ perform basic sanity checks on the command line arguments """ 148 | if parsed_options.show_version: 149 | show_version() 150 | 151 | if len(sys.argv) == 2 and parsed_options.verbosity > 0: 152 | show_version() 153 | 154 | if parsed_options.threads < 1: 155 | parsed_options.threads = 1 156 | 157 | check_none = [parsed_options.role, parsed_options.regions, 158 | parsed_options.ous, parsed_options.accounts] 159 | 160 | if check_none == [None]*len(check_none): 161 | error("error: you must specify at least one of -r, -R, -o or -a") 162 | 163 | if parsed_options.accounts and parsed_options.ous: 164 | error("error: you cannot specify both -a and -o") 165 | 166 | if parsed_options.regions and parsed_options.command == 0: 167 | error("error: if you specify -R or -o then you must supply a command to run") 168 | 169 | if parsed_options.ous and parsed_options.command == 0: 170 | error("error: if you specify -R or -o then you must supply a command to run") 171 | 172 | if parsed_options.accounts and len(parsed_options.accounts) > 1 and parsed_options.command: 173 | error("error: if you specify multiple accounts with -a then " 174 | "you must supply a command to run") 175 | 176 | def check_args(): 177 | """ process command line arguments """ 178 | parser = create_args_parser() 179 | parsed_options = parser.parse_args() 180 | args_basic_checks(parsed_options) 181 | 182 | # if -a or -o are specified the -r has a default value so set it... 183 | if parsed_options.accounts is not None or parsed_options.ous is not None: 184 | if parsed_options.role is None: 185 | parsed_options.role = "OrganizationAccountAccessRole" 186 | 187 | # expand out parsed_options that can take a list of values... 188 | parsed_options.ous = sorted(set(utils.split_list(parsed_options.ous, ","))) 189 | parsed_options.accounts = sorted(set(utils.split_list(parsed_options.accounts, ","))) 190 | parsed_options.regions = sorted(set(utils.split_list(parsed_options.regions, ","))) 191 | return parsed_options 192 | -------------------------------------------------------------------------------- /aws_with/commands.py: -------------------------------------------------------------------------------- 1 | """ 2 | Licensed to the Apache Software Foundation (ASF) under one 3 | or more contributor license agreements. See the NOTICE file 4 | distributed with this work for additional information 5 | regarding copyright ownership. The ASF licenses this file 6 | to you under the Apache License, Version 2.0 (the 7 | "License"); you may not use this file except in compliance 8 | with the License. You may obtain a copy of the License at 9 | 10 | http://www.apache.org/licenses/LICENSE-2.0 11 | 12 | Unless required by applicable law or agreed to in writing, 13 | software distributed under the License is distributed on an 14 | "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY 15 | KIND, either express or implied. See the License for the 16 | specific language governing permissions and limitations 17 | under the License. 18 | """ 19 | 20 | import os 21 | import subprocess 22 | import json 23 | import copy 24 | from . import utils, monkey 25 | 26 | 27 | monkey.apply_patches() 28 | 29 | 30 | def run_command(logger, options, command_list): 31 | """ safe version for run_command_unsafe that takes care of locks """ 32 | try: 33 | logger.debug("calling run_command_unsafe...") 34 | run_command_unsafe(logger, options, command_list) 35 | finally: 36 | utils.GLOBALS["thread_count"] = utils.GLOBALS["thread_count"] - 1 37 | logger.debug("commands still remaining: %s", utils.GLOBALS["thread_count"]) 38 | if utils.GLOBALS["thread_count"] == 0: 39 | utils.GLOBALS["main_thread_lock"].release() 40 | logger.debug("thread is finished, releasing lock") 41 | utils.GLOBALS["thread_pool_lock"].release() 42 | 43 | 44 | def run_command_unsafe(logger, options, command_list): 45 | """ run a command """ 46 | 47 | if utils.GLOBALS["stop_because_of_error"]: 48 | return 49 | 50 | env = copy.deepcopy(os.environ) 51 | env.update(command_list["environment"]) 52 | output = {} 53 | 54 | # check if this is a single command to run a SHELL... 55 | if not command_list["command"]: 56 | logger.debug("launching shell: %s", os.environ["SHELL"]) 57 | command_list["command"] = os.environ["SHELL"] 58 | prompt = "[\\u({}:{})@\\h \\W]\\$ " 59 | env["PS1"] = prompt.format(command_list["account"], command_list["role"]) 60 | subprocess.call(command_list["command"], env=env) 61 | return 62 | 63 | # copy some details from the command request to the command output... 64 | if isinstance(command_list["account"], dict): 65 | output["account"] = command_list["account"]["Id"] 66 | output["path"] = command_list["account"]["Path"] 67 | else: 68 | output["account"] = command_list["account"] 69 | 70 | output["role"] = command_list["role"] 71 | output["region"] = command_list["region"] 72 | output["command"] = " ".join(command_list["command"]) 73 | 74 | # run the command and capture the output... 75 | try: 76 | output["output"] = subprocess.check_output(command_list["command"], 77 | env=env, stderr=subprocess.STDOUT, 78 | shell=False, universal_newlines=True) 79 | # try and parse the command output as JSON... 80 | try: 81 | output["output"] = json.loads(output["output"]) 82 | except (ValueError, SyntaxError): 83 | pass 84 | 85 | command_list["output"] = output 86 | 87 | except subprocess.CalledProcessError as cpe: 88 | logger.info("Command returned non-zero exit code") 89 | output["error"] = {} 90 | output["error"]["message"] = format(cpe) 91 | output["output"] = cpe.output 92 | output["error"]["returncode"] = cpe.returncode 93 | command_list["output"] = output 94 | if options.stop_on_error: 95 | utils.GLOBALS["stop_because_of_error"] = True 96 | 97 | except OSError as ose: 98 | logger.info("Command failed to start") 99 | output["output"] = "" 100 | output["error"] = {} 101 | output["error"]["message"] = format(ose) 102 | command_list["output"] = output 103 | if options.stop_on_error: 104 | utils.GLOBALS["stop_because_of_error"] = True 105 | -------------------------------------------------------------------------------- /aws_with/main.py: -------------------------------------------------------------------------------- 1 | """ 2 | Licensed to the Apache Software Foundation (ASF) under one 3 | or more contributor license agreements. See the NOTICE file 4 | distributed with this work for additional information 5 | regarding copyright ownership. The ASF licenses this file 6 | to you under the Apache License, Version 2.0 (the 7 | "License"); you may not use this file except in compliance 8 | with the License. You may obtain a copy of the License at 9 | 10 | http://www.apache.org/licenses/LICENSE-2.0 11 | 12 | Unless required by applicable law or agreed to in writing, 13 | software distributed under the License is distributed on an 14 | "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY 15 | KIND, either express or implied. See the License for the 16 | specific language governing permissions and limitations 17 | under the License. 18 | """ 19 | 20 | import sys 21 | import json 22 | import yaml 23 | import boto3 24 | import botocore 25 | 26 | from . import cli, utils, workplan, commands, output 27 | 28 | 29 | def main(): 30 | """ main function """ 31 | 32 | # create a dict of globals which are accessed by different threads 33 | utils.GLOBALS["stop_because_of_error"] = False 34 | utils.GLOBALS["thread_count"] = 0 35 | 36 | # process command line arguments... 37 | options = cli.check_args() 38 | 39 | # setup logging 40 | logger = utils.setup_logging(options) 41 | logger.debug("Got optoins: %s", options) 42 | 43 | # if a profile option was specified then set it up... 44 | if options.profile: 45 | try: 46 | logger.info("loading aws configuration profile: %s", options.profile) 47 | boto3.setup_default_session(profile_name=options.profile) 48 | except botocore.exceptions.BotoCoreError as bce: 49 | print("error: " + format(bce)) 50 | sys.exit(1) 51 | 52 | # create boto3 clients... 53 | logger.debug("Creating AWS clients") 54 | try: 55 | org = boto3.client("organizations") 56 | sts = boto3.client("sts") 57 | except botocore.exceptions.BotoCoreError as bce: 58 | print("error: " + format(bce)) 59 | sys.exit(1) 60 | 61 | # main program logic... 62 | workplan.examine_regions(logger, options) 63 | workplan.examine_accounts(logger, options, org) 64 | workplan.examine_command(logger, options) 65 | commands_list = workplan.build_work_plan(logger, options, sts) 66 | 67 | # make sure we have at least one command to run... 68 | if not commands_list: 69 | print("warning: no matching accounts/regions - nothing to do...") 70 | sys.exit(1) 71 | 72 | # don't bother using a thread pool if there is only one command... 73 | if not options.command: 74 | logger.debug("Calling run_command_unsafe() without threadpool as shell command") 75 | commands.run_command_unsafe(logger, options, commands_list[0]) 76 | sys.exit(0) 77 | 78 | workplan.execute_work_plan(logger, options, commands_list) 79 | outputs = output.gather_command_outputs(logger, options, commands_list) 80 | 81 | 82 | logger.info("Nearly done, collating thread outputs") 83 | if options.format == "json": 84 | print(json.dumps(outputs, indent=4, sort_keys=True)) 85 | elif options.format == "yaml": 86 | print(yaml.safe_dump(outputs, default_flow_style=False, 87 | encoding="utf-8", allow_unicode=True, indent=4)) 88 | else: 89 | for out in outputs: 90 | header = "{}@{} in {}:".format(out["role"], out["account"], out["region"]) 91 | print("-" * len(header)) 92 | print(header) 93 | print("-" * len(header)) 94 | print("") 95 | print(out["output"]) 96 | print("") 97 | print("") 98 | 99 | if __name__ == '__main__': 100 | main() 101 | -------------------------------------------------------------------------------- /aws_with/monkey.py: -------------------------------------------------------------------------------- 1 | """ 2 | monkey patches to cover python 2.6 gaps 3 | see: https://stackoverflow.com/questions/... 4 | .../4814970/subprocess-check-output-doesnt-seem-to-exist-python-2-6-5 5 | """ 6 | 7 | import subprocess 8 | 9 | def apply_patches(): 10 | """ apply monkey patches if required """ 11 | patch_subprocess() 12 | 13 | def patch_subprocess(): 14 | """ patch in subprocess.check_output() function if it is missing """ 15 | 16 | if hasattr(subprocess, "check_output"): 17 | return 18 | else: 19 | subprocess.check_output = ___subprocess_check_output 20 | subprocess.CalledProcessError = CalledProcessError 21 | 22 | def ___subprocess_check_output(*popenargs, **kwargs): 23 | """ backwards compatible check_output function for old versions of Python """ 24 | if 'stdout' in kwargs: # pragma: no cover 25 | raise ValueError('stdout argument not allowed, ' 26 | 'it will be overridden.') 27 | process = subprocess.Popen(stdout=subprocess.PIPE, 28 | *popenargs, **kwargs) 29 | output, _ = process.communicate() 30 | retcode = process.poll() 31 | if retcode: 32 | cmd = kwargs.get("args") 33 | if cmd is None: 34 | cmd = popenargs[0] 35 | raise subprocess.CalledProcessError(retcode, cmd, 36 | output=output) 37 | return output 38 | 39 | class CalledProcessError(Exception): 40 | """ exception class to carry exec error details """ 41 | 42 | def __init__(self, returncode, cmd, output=None): 43 | Exception.__init__(self) 44 | self.returncode = returncode 45 | self.cmd = cmd 46 | self.output = output 47 | 48 | def __str__(self): 49 | return "Command '%s' returned non-zero exit status %d" % ( 50 | self.cmd, self.returncode) 51 | -------------------------------------------------------------------------------- /aws_with/organizations.py: -------------------------------------------------------------------------------- 1 | """ 2 | Licensed to the Apache Software Foundation (ASF) under one 3 | or more contributor license agreements. See the NOTICE file 4 | distributed with this work for additional information 5 | regarding copyright ownership. The ASF licenses this file 6 | to you under the Apache License, Version 2.0 (the 7 | "License"); you may not use this file except in compliance 8 | with the License. You may obtain a copy of the License at 9 | 10 | http://www.apache.org/licenses/LICENSE-2.0 11 | 12 | Unless required by applicable law or agreed to in writing, 13 | software distributed under the License is distributed on an 14 | "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY 15 | KIND, either express or implied. See the License for the 16 | specific language governing permissions and limitations 17 | under the License. 18 | """ 19 | 20 | from . import utils 21 | 22 | 23 | def get_child_ous(logger, org_client, org_unit): 24 | """ given an OU, find all the OUs within that OU... """ 25 | logger.debug("Getting OUs for: %s", org_unit) 26 | result = [org_unit] 27 | 28 | # for this OU, get all the children... 29 | args = dict(ParentId=org_unit["Id"]) 30 | children = utils.generic_paginator(logger, org_client.list_organizational_units_for_parent, 31 | "OrganizationalUnits", **args) 32 | 33 | # update child paths and then call ourselves recursively to find all children 34 | for child in children: 35 | child["Path"] = "{}/{}".format(org_unit["Path"], child["Name"]).replace("//", "/") 36 | result.extend(get_child_ous(logger, org_client, child)) 37 | 38 | return result 39 | 40 | 41 | def get_ou_from_path(logger, org_client, path): 42 | """ given a path, traverse Organizations OUs to locate the required OU... """ 43 | logger.debug("Getting OU from path: %s", path) 44 | 45 | current_ou = org_client.list_roots()["Roots"][0]["Id"] 46 | if path == "/": 47 | return {"Id":current_ou, "Path":path} 48 | 49 | for dir_name in path.split("/")[1:]: 50 | logger.debug("Getting OU from path: %s, looking for: %s", path, dir_name) 51 | found = False 52 | args = dict(ParentId=current_ou) 53 | children = utils.generic_paginator(logger, org_client.list_organizational_units_for_parent, 54 | "OrganizationalUnits", **args) 55 | 56 | for org_unit in children: 57 | if org_unit["Name"] == dir_name: 58 | current_ou = org_unit["Id"] 59 | found = True 60 | break 61 | 62 | if not found: 63 | raise ValueError("OU path not found") 64 | 65 | return {"Id":current_ou, "Path":path} 66 | 67 | 68 | def get_accounts_for_ou(logger, options, org_client, path): 69 | """ given a path, get all the AWS accounts within that part of an Organization... """ 70 | logger.debug("Getting accounts for OU: %s", path) 71 | org_unit = get_ou_from_path(logger, org_client, path) 72 | ous = [] 73 | if options.no_recursive: 74 | ous.append(org_unit) 75 | else: 76 | ous.extend(get_child_ous(logger, org_client, org_unit)) 77 | 78 | result = [] 79 | for org_unit in ous: 80 | args = {"ParentId":org_unit["Id"]} 81 | accounts = utils.generic_paginator(logger, org_client.list_accounts_for_parent, 82 | "Accounts", **args) 83 | for acc in accounts: 84 | acc["Path"] = org_unit["Path"] 85 | if 'Status' in acc: 86 | if acc['Status'] != 'SUSPENDED': 87 | result.append(acc) 88 | else: 89 | logger.info("found suspended account %s, ignoring it." % acc) 90 | return result 91 | -------------------------------------------------------------------------------- /aws_with/output.py: -------------------------------------------------------------------------------- 1 | """ 2 | Licensed to the Apache Software Foundation (ASF) under one 3 | or more contributor license agreements. See the NOTICE file 4 | distributed with this work for additional information 5 | regarding copyright ownership. The ASF licenses this file 6 | to you under the Apache License, Version 2.0 (the 7 | "License"); you may not use this file except in compliance 8 | with the License. You may obtain a copy of the License at 9 | 10 | http://www.apache.org/licenses/LICENSE-2.0 11 | 12 | Unless required by applicable law or agreed to in writing, 13 | software distributed under the License is distributed on an 14 | "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY 15 | KIND, either express or implied. See the License for the 16 | specific language governing permissions and limitations 17 | under the License. 18 | """ 19 | 20 | 21 | def gather_command_outputs(logger, options, commands_list): 22 | """ gather the command outputs together """ 23 | outputs = [] 24 | for cmd in commands_list: 25 | 26 | # check if we have an output object that shouldn't be suppressed from --quiet... 27 | if "output" in cmd.keys(): 28 | command_output = cmd["output"]["output"] 29 | logger.debug("Command output: %s", command_output) 30 | oo_single_key = False 31 | oo_single_empty_key = False 32 | if isinstance(command_output, dict): 33 | oo_key_count = len(command_output.keys()) 34 | oo_single_key = oo_key_count == 1 35 | if oo_single_key: 36 | oo_key_values = len(command_output[list(command_output.keys())[0]]) 37 | oo_single_empty_key = oo_key_values == 0 38 | 39 | # check for literally no output... 40 | if options.quiet and (command_output is None or command_output == ""): 41 | logger.debug("Command output is actually empty, skipping: %s", cmd["output"]) 42 | 43 | # check if output contains a single empty object... 44 | elif options.quiet and oo_single_empty_key: 45 | logger.debug("Command output is effectively empty, skipping: %s", cmd["output"]) 46 | 47 | else: 48 | outputs.append(cmd["output"]) 49 | return outputs 50 | -------------------------------------------------------------------------------- /aws_with/regions.py: -------------------------------------------------------------------------------- 1 | """ 2 | Licensed to the Apache Software Foundation (ASF) under one 3 | or more contributor license agreements. See the NOTICE file 4 | distributed with this work for additional information 5 | regarding copyright ownership. The ASF licenses this file 6 | to you under the Apache License, Version 2.0 (the 7 | "License"); you may not use this file except in compliance 8 | with the License. You may obtain a copy of the License at 9 | 10 | http://www.apache.org/licenses/LICENSE-2.0 11 | 12 | Unless required by applicable law or agreed to in writing, 13 | software distributed under the License is distributed on an 14 | "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY 15 | KIND, either express or implied. See the License for the 16 | specific language governing permissions and limitations 17 | under the License. 18 | """ 19 | 20 | import re 21 | import boto3 22 | from . import utils 23 | 24 | 25 | def get_regions_list(logger): 26 | """ get a list of AWS regions """ 27 | logger.debug("getting a list of AWS regions...") 28 | ec2 = boto3.client("ec2", region_name="us-east-1") 29 | return utils.generic_paginator(logger, ec2.describe_regions, "Regions") 30 | 31 | def get_regions_from_regex(logger, regex, region_list): 32 | """ search the regions list using regular expressions """ 33 | logger.debug("getting regions that match: %s", regex) 34 | regex_pattern = re.compile("^" + regex.replace("*", ".*") + "$") 35 | filtered_regions = filter(lambda x: regex_pattern.match(x["RegionName"]), region_list) 36 | return map(lambda x: x["RegionName"], filtered_regions) 37 | -------------------------------------------------------------------------------- /aws_with/utils.py: -------------------------------------------------------------------------------- 1 | """ 2 | Licensed to the Apache Software Foundation (ASF) under one 3 | or more contributor license agreements. See the NOTICE file 4 | distributed with this work for additional information 5 | regarding copyright ownership. The ASF licenses this file 6 | to you under the Apache License, Version 2.0 (the 7 | "License"); you may not use this file except in compliance 8 | with the License. You may obtain a copy of the License at 9 | 10 | http://www.apache.org/licenses/LICENSE-2.0 11 | 12 | Unless required by applicable law or agreed to in writing, 13 | software distributed under the License is distributed on an 14 | "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY 15 | KIND, either express or implied. See the License for the 16 | specific language governing permissions and limitations 17 | under the License. 18 | """ 19 | 20 | import itertools 21 | import logging 22 | 23 | GLOBALS = {} 24 | 25 | def flatten_list(the_list): 26 | """ take a list of lists and flatten it to just a list """ 27 | return [] if the_list is None else list(itertools.chain.from_iterable(the_list)) 28 | 29 | 30 | def split_list(the_list, splitter): 31 | """ take a list and split each item and put the split items back into the main list""" 32 | return [] if the_list is None else flatten_list(map(lambda x: x.split(splitter), the_list)) 33 | 34 | 35 | def generic_paginator(logger, paged_function, result_object, **kwargs): 36 | """ call an API until there are no more results """ 37 | logger.debug("in generic_paginator, for: %s", paged_function) 38 | for key, value in kwargs.items(): 39 | logger.debug(" params: %s=%s", key, value) 40 | full_results = [] 41 | next_token = "" 42 | while next_token is not None: 43 | if next_token != "": 44 | kwargs.update({"NextToken":next_token}) 45 | result = paged_function(**kwargs) 46 | full_results.extend(result[result_object]) 47 | next_token = result.get("NextToken", None) 48 | logger.debug(" next_token: %s", next_token) 49 | return full_results 50 | 51 | 52 | def setup_logging(options): 53 | """ set up logging... """ 54 | logger = logging.getLogger("aws_with") 55 | logger.setLevel(logging.ERROR) 56 | stream_handler = logging.StreamHandler() 57 | formatter = logging.Formatter('%(asctime)s %(levelname)s(%(threadName)s): %(message)s') 58 | stream_handler.setFormatter(formatter) 59 | logger.addHandler(stream_handler) 60 | if options.verbosity is None: 61 | options.verbosity = 0 62 | if options.verbosity >= 1: 63 | logger.setLevel(logging.INFO) 64 | if options.verbosity >= 2: 65 | logger.setLevel(logging.DEBUG) 66 | logger.debug("Logger set up") 67 | return logger 68 | -------------------------------------------------------------------------------- /aws_with/workplan.py: -------------------------------------------------------------------------------- 1 | """ 2 | Licensed to the Apache Software Foundation (ASF) under one 3 | or more contributor license agreements. See the NOTICE file 4 | distributed with this work for additional information 5 | regarding copyright ownership. The ASF licenses this file 6 | to you under the Apache License, Version 2.0 (the 7 | "License"); you may not use this file except in compliance 8 | with the License. You may obtain a copy of the License at 9 | 10 | http://www.apache.org/licenses/LICENSE-2.0 11 | 12 | Unless required by applicable law or agreed to in writing, 13 | software distributed under the License is distributed on an 14 | "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY 15 | KIND, either express or implied. See the License for the 16 | specific language governing permissions and limitations 17 | under the License. 18 | """ 19 | 20 | import sys 21 | import os 22 | import threading 23 | import boto3 24 | import botocore 25 | import socket 26 | 27 | from . import regions, utils, organizations, commands 28 | 29 | def examine_regions(logger, options): 30 | """ for each region provided, use it as a regex to search for regions... """ 31 | logger.debug("Getting list of regions") 32 | regions_list = regions.get_regions_list(logger) 33 | if options.regions: 34 | matched_regions = map(lambda x: regions.get_regions_from_regex(logger, x, regions_list), 35 | options.regions) 36 | options.regions = sorted(set(utils.flatten_list(matched_regions))) 37 | if not options.regions: 38 | print("error: no matching regions found") 39 | sys.exit(1) 40 | logger.info("Set regions: %s", ", ".join(options.regions)) 41 | 42 | # if we don't have any regions set, then try and get it from the current session... 43 | if not options.regions: 44 | logger.debug("No regions specified on command line, guessing...") 45 | options.regions = [boto3.session.Session().region_name] 46 | if options.regions == [None]: 47 | options.regions = [""] 48 | logger.info("No region set, using default") 49 | else: 50 | logger.info("Set regions: %s", ", ".join(options.regions)) 51 | 52 | 53 | def examine_accounts(logger, options, org_client): 54 | """ if we don't have any accounts set, then try and get guess our current account_id... """ 55 | logger.debug("Getting list of accounts") 56 | if not options.accounts: 57 | logger.debug("No accounts specified on command line, guessing...") 58 | try: 59 | # if we have any kind of AWS credentials set then this should work... 60 | account_id = boto3.client("sts").get_caller_identity()["Account"] 61 | options.accounts = [account_id] 62 | except (botocore.exceptions.ClientError, botocore.exceptions.BotoCoreError): 63 | print("error: unable to work out existing account ID, " 64 | "please check your AWS credentials, or specify it with the '-a' option") 65 | sys.exit(1) 66 | 67 | logger.info("Set accounts: %s", ", ".join(options.accounts)) 68 | 69 | # if we were given OUs then convert them into a list of accounts 70 | logger.debug("Checking if we need to traverse an Organization") 71 | if options.ous: 72 | logger.debug("Getting list of accounts from OUs") 73 | mapped_list = map(lambda path: organizations.get_accounts_for_ou(logger, options, 74 | org_client, path), 75 | options.ous) 76 | options.accounts = utils.flatten_list(mapped_list) 77 | if options.no_master: 78 | master = org_client.describe_organization()["Organization"]["MasterAccountId"] 79 | logger.debug("Removing the master account (%s) from the list of accounts", master) 80 | options.accounts = filter(lambda x: x["Id"] != master, options.accounts) 81 | 82 | logger.info("Set accounts: %s", options.accounts) 83 | 84 | 85 | def examine_command(logger, options): 86 | """ look at the command and guess if it is an AWS CLI built in command """ 87 | if not options.no_cli_guess and options.command: 88 | try: 89 | import awscli.clidriver 90 | logger.debug("Guessing if the supplied command is an AWS CLI command..") 91 | cli = awscli.clidriver.CLIDriver() 92 | cli_help = cli.create_help_command() 93 | aws_cli_commands = map(lambda x: x, cli_help.command_table) 94 | 95 | if options.command[0] in aws_cli_commands: 96 | options.command.insert(0, "aws") 97 | logger.debug("Assuming command is an AWS CLI, new command is: %s", options.command) 98 | except ImportError: 99 | logger.debug("awscli module not found or failed to load") 100 | 101 | 102 | def build_work_plan(logger, options, sts_client): 103 | """ create a big list of commands we need to run... """ 104 | logger.info("Starting analysis on work plan") 105 | commands_list = [] 106 | 107 | # iterate over accounts and regions... 108 | for account in options.accounts: 109 | 110 | logger.debug("Looking at account: %s", account) 111 | account_id = account["Id"] if isinstance(account, dict) else account 112 | 113 | # work out if we need to call STS to assume a new role... 114 | if options.role: 115 | 116 | # call STS to get credentials for the account and role... 117 | try: 118 | arn = "arn:aws:iam::{}:role/{}".format(account_id, options.role) 119 | logger.debug("Calling STS to get temporary credentials for: %s", arn) 120 | assumed_role = sts_client.assume_role( 121 | RoleArn=arn, 122 | RoleSessionName="{}@{}".format(os.environ["USER"], socket.gethostname()), 123 | ExternalId="{}@{}".format(os.environ["USER"], socket.gethostname()) 124 | ) 125 | 126 | except botocore.exceptions.BotoCoreError as be_bce: 127 | print("error switching role ({}@{}): {}".format(options.role, 128 | account_id, be_bce.args)) 129 | sys.exit(1) 130 | 131 | except botocore.exceptions.ClientError as be_ce: 132 | print("error switching role ({}@{}): {}".format(options.role, 133 | account_id, be_ce.args)) 134 | sys.exit(1) 135 | 136 | for region in options.regions: 137 | logger.debug("Looking at region: %s", region) 138 | cmd = {} 139 | cmd["command"] = options.command 140 | cmd["environment"] = {} 141 | cmd["role"] = options.role 142 | cmd["account_id"] = account_id 143 | cmd["account"] = account 144 | cmd["region"] = region 145 | env = cmd["environment"] 146 | 147 | if region != "": 148 | env["AWS_DEFAULT_REGION"] = region 149 | if options.profile: 150 | session = boto3.session.Session(profile_name=options.profile) 151 | env["AWS_ACCESS_KEY_ID"] = session.get_credentials().access_key 152 | if options.role: 153 | env["AWS_ACCESS_KEY_ID"] = assumed_role["Credentials"]["AccessKeyId"] 154 | 155 | logger.debug("Adding command to work plan: %s", cmd) 156 | cmd["options"] = options 157 | 158 | # add credentials and sensitive information after we have output debug info... 159 | if options.profile: 160 | env["AWS_SECRET_ACCESS_KEY"] = session.get_credentials().secret_key 161 | if options.role: 162 | env["AWS_SECRET_ACCESS_KEY"] = assumed_role["Credentials"]["SecretAccessKey"] 163 | env["AWS_SESSION_TOKEN"] = assumed_role["Credentials"]["SessionToken"] 164 | 165 | commands_list.append(cmd) 166 | 167 | return commands_list 168 | 169 | 170 | def execute_work_plan(logger, options, commands_list): 171 | """ run through commands_list and run various commands in the thread pool """ 172 | logger.info("Executing work plan across a thread pool of size: %s", options.threads) 173 | utils.GLOBALS["main_thread_lock"] = threading.Lock() 174 | utils.GLOBALS["thread_pool_lock"] = threading.BoundedSemaphore(options.threads) 175 | utils.GLOBALS["thread_count"] = len(commands_list) 176 | logger.debug("Locks created, task list size = %s", utils.GLOBALS["thread_count"]) 177 | 178 | # obtain the main thread lock... 179 | logger.debug("Acquiring main thread lock") 180 | utils.GLOBALS["main_thread_lock"].acquire() 181 | 182 | for cmd in commands_list: 183 | logger.debug("waiting for next thread to be available") 184 | utils.GLOBALS["thread_pool_lock"].acquire() 185 | logger.debug("thread is available, starting thread") 186 | threading.Thread(target=commands.run_command, args=(logger, options, cmd, )).start() 187 | 188 | # block on the main thread lock being released... 189 | logger.debug("Blocking main thread, waiting on commands to finish") 190 | utils.GLOBALS["main_thread_lock"].acquire() 191 | logger.debug("Main thread lock released, working on output") 192 | -------------------------------------------------------------------------------- /buildspec.yml: -------------------------------------------------------------------------------- 1 | version: 0.2 2 | 3 | phases: 4 | install: 5 | commands: 6 | - echo install dependencies 7 | pre_build: 8 | commands: 9 | - set 10 | - find . 11 | build: 12 | commands: 13 | - python setup.py sdist 14 | post_build: 15 | commands: 16 | artifacts: 17 | files: 18 | - dist/aws_with-*.tar.gz 19 | - testspec.yml 20 | discard-paths: yes 21 | -------------------------------------------------------------------------------- /examples/enable_guardduty_with_sns_email.py: -------------------------------------------------------------------------------- 1 | import boto3 2 | import json 3 | 4 | ########################################################################## 5 | 6 | # Change the email address where you want GuardDuty alerts sent to 7 | # Then run with: aws_with -R '*' python enable_guardduty_with_sns_email.py 8 | 9 | SOC_EMAIL_ADDRESS="PUT_YOUR_EMAIL_ADDRESS_HERE" 10 | 11 | ########################################################################## 12 | 13 | 14 | gd=boto3.client("guardduty") 15 | sns=boto3.client("sns") 16 | cwe=boto3.client("events") 17 | 18 | # enable GuardDuty 19 | gd.create_detector(Enable=True) 20 | 21 | # create SNS topic and subscription 22 | topic = sns.create_topic(Name="email-guardduty-alerts")["TopicArn"] 23 | subscription=sns.subscribe(TopicArn=topic, Protocol="email", Endpoint=SOC_EMAIL_ADDRESS) 24 | 25 | # Add CloudWatch Events Rule to trigger email 26 | rule=cwe.put_rule(Name="guardduty-alerts", EventPattern='{"source":["aws.guardduty"]}')["RuleArn"] 27 | target=[{"Id":"1","Arn":topic}] 28 | cwe.put_targets(Rule="guardduty-alerts", Targets=target) 29 | 30 | # Grant CloudWatch Events permission to publish to the topic 31 | policy = { 32 | "Version": "2012-10-17", 33 | "Id": "__default_policy_ID", 34 | "Statement": [ 35 | { 36 | "Sid": "__default_statement_ID", 37 | "Effect": "Allow", 38 | "Principal": { 39 | "AWS": "*" 40 | }, 41 | "Action": [ 42 | "SNS:GetTopicAttributes", 43 | "SNS:SetTopicAttributes", 44 | "SNS:AddPermission", 45 | "SNS:RemovePermission", 46 | "SNS:DeleteTopic", 47 | "SNS:Subscribe", 48 | "SNS:ListSubscriptionsByTopic", 49 | "SNS:Publish", 50 | "SNS:Receive" 51 | ], 52 | "Resource": topic, 53 | "Condition": { 54 | "StringEquals": { 55 | "AWS:SourceOwner": topic.split(":")[4] 56 | } 57 | } 58 | }, 59 | { 60 | "Sid": "AWSEvents_guardduty-alerts_1", 61 | "Effect": "Allow", 62 | "Principal": { 63 | "Service": "events.amazonaws.com" 64 | }, 65 | "Action": "sns:Publish", 66 | "Resource": topic 67 | } 68 | ] 69 | } 70 | 71 | policy=json.dumps(policy) 72 | sns.set_topic_attributes(TopicArn=topic, AttributeName="Policy", AttributeValue=policy) 73 | -------------------------------------------------------------------------------- /examples/show_spot_prices_globally.py: -------------------------------------------------------------------------------- 1 | ## aws_with --output text -R '*' python show_spot_prices_globally.py 2 | 3 | import datetime 4 | import pytz 5 | import boto3 6 | 7 | now=datetime.datetime.now(pytz.UTC) 8 | ec2=boto3.client("ec2") 9 | 10 | prices=ec2.describe_spot_price_history( 11 | InstanceTypes=["m4.4xlarge"], 12 | ProductDescriptions=["Linux/UNIX"], 13 | StartTime=now)["SpotPriceHistory"] 14 | 15 | for p in prices: 16 | print ("{},{}".format(p["AvailabilityZone"], p["SpotPrice"])) 17 | -------------------------------------------------------------------------------- /pipeline.cfg: -------------------------------------------------------------------------------- 1 | # use "git update-index --skip-worktree pipeline.cfg" to prevent your pipeline details from being tracked by GIT 2 | 3 | TEST_S3_PATH=s3://INSERT_YOUR_S3_BUCKET_NAME_HERE/source-bundle.zip 4 | TEST_PIPELINE=INSERT_YOUR_TEST_PIPELINE_NAME_HERE 5 | -------------------------------------------------------------------------------- /run-test-pipeline.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # read pipeline configuration options 4 | source pipeline.cfg 5 | 6 | # zip it up... 7 | ZIP=`mktemp -u --suffix=.zip` 8 | git ls-tree -r --name-only master | zip -@ $ZIP 9 | 10 | # copy the zip to S3... 11 | aws s3 cp $ZIP $TEST_S3_PATH 12 | 13 | # clean up... 14 | rm $ZIP 15 | rm -rf $ZIPDIR 16 | 17 | # kick the pipeline... 18 | aws codepipeline start-pipeline-execution --name $TEST_PIPELINE 19 | 20 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [wheel] 2 | universal=1 3 | 4 | [metadata] 5 | requires-dist = 6 | boto3 >= 1.4.6 7 | 8 | [egg_info] 9 | tag_build = 10 | tag_date = 0 11 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from setuptools import setup 2 | import sys 3 | import aws_with 4 | 5 | install_requires = [ 6 | "boto3 >= 1.4.6", 7 | "pyyaml", 8 | ] 9 | 10 | if sys.version_info[:2] < (2, 7): 11 | install_requires += [ 12 | "argparse", 13 | ] 14 | 15 | setup( 16 | name=aws_with.__title__, 17 | version=aws_with.__version__, 18 | description=aws_with.__summary__, 19 | long_description=open("README.rst").read(), 20 | license=aws_with.__license__, 21 | url=aws_with.__uri__, 22 | author=aws_with.__author__, 23 | author_email=aws_with.__email__, 24 | packages=["aws_with"], 25 | install_requires=install_requires, 26 | extras_require={}, 27 | data_files = [("", ["LICENSE"])], 28 | entry_points={'console_scripts': ['aws_with = aws_with.main:main']}, 29 | classifiers=[ 30 | 'Development Status :: 4 - Beta', 31 | 'Environment :: Console', 32 | 'Intended Audience :: Developers', 33 | 'Intended Audience :: End Users/Desktop', 34 | 'Intended Audience :: Information Technology', 35 | 'Intended Audience :: System Administrators', 36 | 'License :: OSI Approved :: Apache Software License', 37 | 'Operating System :: POSIX', 38 | 'Operating System :: Microsoft :: Windows', 39 | 'Operating System :: MacOS :: MacOS X', 40 | 'Topic :: System :: Systems Administration', 41 | 'Topic :: Utilities', 42 | 'Programming Language :: Python', 43 | 'Programming Language :: Python :: 2.6', 44 | 'Programming Language :: Python :: 2.7', 45 | 'Programming Language :: Python :: 3.4', 46 | 'Programming Language :: Python :: 3.5', 47 | 'Programming Language :: Python :: 3.6', 48 | 'Programming Language :: Python :: 3.7' 49 | ], 50 | 51 | ) 52 | -------------------------------------------------------------------------------- /tests/test_one.py: -------------------------------------------------------------------------------- 1 | def test_one(): 2 | print("running random test 1") 3 | a = 100 4 | a = a/2-a+51 5 | assert 1 == a 6 | 7 | def test_two(): 8 | print("running random test 2") 9 | assert 2 == 2 10 | -------------------------------------------------------------------------------- /testspec.yml: -------------------------------------------------------------------------------- 1 | version: 0.2 2 | 3 | phases: 4 | install: 5 | commands: 6 | - pip install pip --upgrade 7 | - pip install lizard --upgrade 8 | - pip install pylint --upgrade 9 | - echo TODO install Python3.6 10 | - echo TODO install/upgrade tox 11 | pre_build: 12 | commands: 13 | - tar zxvf aws_with-*.tar.gz 14 | - cd aws_with-[0-9.]* 15 | build: 16 | commands: 17 | - cd aws_with 18 | - lizard --version 19 | - lizard 20 | - pylint --version 21 | - pylint . 22 | - echo TODO run tox -e py26 23 | - echo TODO run tox -e py27 24 | - echo TODO run tox -e py34 25 | - echo TODO run tox -e py35 26 | - echo TODO run tox -e py36 27 | --------------------------------------------------------------------------------