├── tests ├── __init__.py ├── unit │ ├── __init__.py │ └── altimeter │ │ ├── __init__.py │ │ ├── aws │ │ ├── __init__.py │ │ ├── auth │ │ │ └── __init__.py │ │ └── resource │ │ │ ├── __init__.py │ │ │ ├── s3 │ │ │ └── __init__.py │ │ │ ├── ec2 │ │ │ ├── __init__.py │ │ │ └── test_volume.py │ │ │ ├── elbv1 │ │ │ ├── __init__.py │ │ │ └── test_load_balancer.py │ │ │ ├── elbv2 │ │ │ ├── __init__.py │ │ │ ├── test_target_group.py │ │ │ └── test_load_balancer.py │ │ │ ├── events │ │ │ ├── __init__.py │ │ │ └── test_cloudwatchevents_rule.py │ │ │ ├── iam │ │ │ ├── __init__.py │ │ │ ├── test_group.py │ │ │ ├── test_saml_provider.py │ │ │ ├── test_policy.py │ │ │ └── test_account_password_policy.py │ │ │ ├── rds │ │ │ ├── __init__.py │ │ │ └── test_instance.py │ │ │ ├── awslambda │ │ │ └── __init__.py │ │ │ ├── dynamodb │ │ │ └── __init__.py │ │ │ ├── test_unscanned_account.py │ │ │ ├── test_resource_spec.py │ │ │ └── test_account.py │ │ ├── core │ │ ├── __init__.py │ │ ├── graph │ │ │ ├── __init__.py │ │ │ ├── field │ │ │ │ ├── __init__.py │ │ │ │ ├── test_util.py │ │ │ │ ├── test_base.py │ │ │ │ └── test_tags_field.py │ │ │ ├── test_node_cache.py │ │ │ ├── test_schema.py │ │ │ └── test_graph_spec.py │ │ ├── neptune │ │ │ └── __init__.py │ │ ├── artifact_io │ │ │ ├── __init__.py │ │ │ └── test_writer.py │ │ ├── test_json_encoder.py │ │ └── test_config.py │ │ └── qj │ │ ├── __init__.py │ │ ├── lambdas │ │ └── __init__.py │ │ ├── schemas │ │ ├── __init__.py │ │ └── test_job.py │ │ └── test_security.py ├── integration │ ├── __init__.py │ └── altimeter │ │ ├── aws │ │ └── __init__.py │ │ └── qj │ │ ├── __init__.py │ │ └── crud │ │ └── __init__.py ├── requirements.in └── dbutil.py ├── altimeter ├── __init__.py ├── aws │ ├── __init__.py │ ├── auth │ │ ├── __init__.py │ │ └── exceptions.py │ ├── scan │ │ ├── __init__.py │ │ ├── account_scan_manifest.py │ │ ├── scan_manifest.py │ │ └── scan_plan.py │ ├── resource │ │ ├── __init__.py │ │ ├── s3 │ │ │ └── __init__.py │ │ ├── kms │ │ │ ├── __init__.py │ │ │ └── key.py │ │ ├── acm │ │ │ └── __init__.py │ │ ├── rds │ │ │ └── __init__.py │ │ ├── awslambda │ │ │ ├── __init__.py │ │ │ └── function.py │ │ ├── dynamodb │ │ │ └── __init__.py │ │ ├── ec2 │ │ │ ├── __init__.py │ │ │ ├── vpc.py │ │ │ ├── image.py │ │ │ ├── region.py │ │ │ ├── subnet.py │ │ │ ├── vpc_endpoint.py │ │ │ ├── transit_gateway_vpc_attachment.py │ │ │ ├── snapshot.py │ │ │ ├── vpc_endpoint_service.py │ │ │ ├── internet_gateway.py │ │ │ └── flow_log.py │ │ ├── eks │ │ │ ├── __init__.py │ │ │ └── cluster.py │ │ ├── elbv2 │ │ │ └── __init__.py │ │ ├── guardduty │ │ │ └── __init__.py │ │ ├── cloudtrail │ │ │ ├── __init__.py │ │ │ └── trail.py │ │ ├── events │ │ │ ├── __init__.py │ │ │ └── event_bus.py │ │ ├── elbv1 │ │ │ └── __init__.py │ │ ├── route53 │ │ │ ├── __init__.py │ │ │ └── hosted_zone.py │ │ ├── support │ │ │ ├── __init__.py │ │ │ └── severity_level.py │ │ ├── iam │ │ │ ├── __init__.py │ │ │ ├── instance_profile.py │ │ │ ├── account_password_policy.py │ │ │ └── iam_oidc_provider.py │ │ ├── organizations │ │ │ ├── org.py │ │ │ ├── __init__.py │ │ │ └── ou.py │ │ ├── account.py │ │ └── unscanned_account.py │ ├── settings.py │ └── log_events.py ├── core │ ├── __init__.py │ ├── graph │ │ ├── field │ │ │ ├── __init__.py │ │ │ ├── util.py │ │ │ ├── exceptions.py │ │ │ ├── base.py │ │ │ └── tags_field.py │ │ ├── __init__.py │ │ ├── node_cache.py │ │ ├── exceptions.py │ │ ├── schema.py │ │ └── graph_spec.py │ ├── neptune │ │ ├── __init__.py │ │ ├── exceptions.py │ │ └── sparql.py │ ├── resource │ │ ├── __init__.py │ │ ├── exceptions.py │ │ └── resource.py │ ├── exceptions.py │ ├── artifact_io │ │ ├── exceptions.py │ │ └── __init__.py │ ├── json_encoder.py │ ├── base_model.py │ └── log_events.py └── qj │ ├── __init__.py │ ├── api │ ├── __init__.py │ ├── base │ │ ├── __init__.py │ │ ├── endpoints │ │ │ ├── __init__.py │ │ │ ├── status.py │ │ │ └── auth.py │ │ └── api.py │ └── v1 │ │ ├── __init__.py │ │ ├── endpoints │ │ └── __init__.py │ │ └── api.py │ ├── db │ ├── __init__.py │ ├── base_class.py │ └── base.py │ ├── crud │ └── __init__.py │ ├── lambdas │ ├── __init__.py │ ├── executor.py │ └── pruner.py │ ├── models │ ├── __init__.py │ ├── job.py │ └── result_set.py │ ├── schemas │ ├── status.py │ ├── remediation.py │ ├── __init__.py │ └── result_set_notification.py │ ├── settings.py │ ├── security.py │ ├── notifier.py │ ├── exceptions.py │ ├── middleware.py │ ├── log.py │ └── remediator.py ├── MANIFEST.in ├── git └── pre-commit.sh ├── services └── qj │ ├── run_local.sh │ ├── requirements.in │ ├── alembic │ ├── script.py.mako │ ├── versions │ │ ├── 94f36533d115_remove_result_synthetic_pk.py │ │ ├── e6e2a6bf2a39_adding_query_job_raw_query_column.py │ │ ├── 9d956e753055_added_remediate_sqs_queue_column_to_qj_.py │ │ └── 60990e9bc347_added_notify_if_results_bool_on_job.py │ ├── alembic.ini │ └── env.py │ ├── main.py │ └── requirements.txt ├── .coveragerc ├── doc ├── requirements.in ├── source │ ├── user │ │ ├── local_blazegraph.rst │ │ └── quickstart.rst │ ├── dev │ │ ├── example_ec2_instance_resource_spec_1.py │ │ ├── sample_scan_resource_1.json │ │ ├── example_ec2_instance_resource_spec_2.py │ │ └── devguide.rst │ └── index.rst └── requirements.txt ├── ci ├── db_stop.sh └── db_start.sh ├── requirements.in ├── pytest.ini ├── qj-lambda.Dockerfile ├── scanner.Dockerfile ├── sfn-init-lambda.Dockerfile ├── sfn-load-rdf-lambda.Dockerfile ├── altimeter-lambda.Dockerfile ├── qj.Dockerfile ├── sfn-prune-graphs-lambda.Dockerfile ├── sfn-scan-account-lambda.Dockerfile ├── sfn-compile-graphs-lambda.Dockerfile ├── bin ├── altimeter ├── sfn_prune_graphs.py ├── queryjob_lambda.py ├── aws2n.py ├── rdf2blaze ├── sfn_init.py ├── runquery.py └── scan_resource.py ├── .gitignore ├── LICENSE ├── conf ├── current_single_account.toml ├── current_single_account_skip_support.toml └── current_master_multi_account.toml ├── .github └── workflows │ └── ci.yml ├── tox.ini ├── requirements.txt └── setup.py /tests/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /altimeter/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/unit/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /altimeter/aws/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /altimeter/core/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /altimeter/qj/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /altimeter/qj/api/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /altimeter/qj/db/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /altimeter/aws/auth/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /altimeter/aws/scan/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /altimeter/qj/api/base/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /altimeter/qj/api/v1/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /altimeter/qj/crud/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /altimeter/qj/lambdas/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /altimeter/qj/models/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/integration/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/unit/altimeter/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /altimeter/aws/resource/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /altimeter/core/graph/field/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /altimeter/core/neptune/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /altimeter/core/resource/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/unit/altimeter/aws/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/unit/altimeter/core/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/unit/altimeter/qj/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /altimeter/qj/api/base/endpoints/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /altimeter/qj/api/v1/endpoints/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/integration/altimeter/aws/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/integration/altimeter/qj/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/unit/altimeter/aws/auth/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/unit/altimeter/core/graph/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/unit/altimeter/qj/lambdas/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/unit/altimeter/qj/schemas/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/integration/altimeter/qj/crud/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/unit/altimeter/aws/resource/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/unit/altimeter/aws/resource/s3/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/unit/altimeter/core/neptune/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/unit/altimeter/aws/resource/ec2/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/unit/altimeter/aws/resource/elbv1/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/unit/altimeter/aws/resource/elbv2/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/unit/altimeter/aws/resource/events/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/unit/altimeter/aws/resource/iam/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/unit/altimeter/aws/resource/rds/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/unit/altimeter/core/artifact_io/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/unit/altimeter/core/graph/field/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include README.md LICENSE 2 | prune tests 3 | -------------------------------------------------------------------------------- /tests/unit/altimeter/aws/resource/awslambda/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/unit/altimeter/aws/resource/dynamodb/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /git/pre-commit.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -euf -o pipefail 4 | 5 | tox 6 | -------------------------------------------------------------------------------- /services/qj/run_local.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | export PYTHONPATH=../.. 4 | uvicorn main:app --reload 5 | -------------------------------------------------------------------------------- /.coveragerc: -------------------------------------------------------------------------------- 1 | [report] 2 | # show specific missing line number ranges in coverage report 3 | show_missing = True 4 | -------------------------------------------------------------------------------- /altimeter/core/graph/__init__.py: -------------------------------------------------------------------------------- 1 | import datetime 2 | 3 | SCALAR_TYPES = (bool, int, float, str, datetime.datetime) 4 | -------------------------------------------------------------------------------- /altimeter/core/exceptions.py: -------------------------------------------------------------------------------- 1 | """Base Exceptions.""" 2 | 3 | 4 | class AltimeterException(Exception): 5 | """An error occurred.""" 6 | -------------------------------------------------------------------------------- /doc/requirements.in: -------------------------------------------------------------------------------- 1 | MarkupSafe==2.1.1 2 | jinja2==3.0.3 3 | sphinx==3.5.4 4 | sphinx-autodoc-typehints==1.12.0 5 | requests==2.31.0 6 | certifi==2023.7.22 7 | pygments==2.15.0 8 | urllib3==1.26.18 9 | -------------------------------------------------------------------------------- /altimeter/qj/db/base_class.py: -------------------------------------------------------------------------------- 1 | """Base SQLAlchemy table class - all declarative tables should inherit from this.""" 2 | from sqlalchemy.ext.declarative import declarative_base 3 | 4 | BASE = declarative_base() 5 | -------------------------------------------------------------------------------- /ci/db_stop.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | set -ef -o pipefail 4 | 5 | if [[ -f postgres_docker_container.id ]]; then 6 | docker kill $(cat postgres_docker_container.id) 7 | rm -f postgres_docker_container.id 8 | fi 9 | -------------------------------------------------------------------------------- /altimeter/core/artifact_io/exceptions.py: -------------------------------------------------------------------------------- 1 | """Exceptions for artifact I/O""" 2 | from altimeter.core.exceptions import AltimeterException 3 | 4 | 5 | class InvalidS3URIException(AltimeterException): 6 | """An S3 uri could not be parsed.""" 7 | -------------------------------------------------------------------------------- /altimeter/aws/resource/s3/__init__.py: -------------------------------------------------------------------------------- 1 | """Base class for S3 resources.""" 2 | from altimeter.aws.resource.resource_spec import AWSResourceSpec 3 | 4 | 5 | class S3ResourceSpec(AWSResourceSpec): 6 | """Base class for S3 resources.""" 7 | 8 | service_name = "s3" 9 | -------------------------------------------------------------------------------- /altimeter/aws/resource/kms/__init__.py: -------------------------------------------------------------------------------- 1 | """Base class for KMS resources.""" 2 | from altimeter.aws.resource.resource_spec import AWSResourceSpec 3 | 4 | 5 | class KMSResourceSpec(AWSResourceSpec): 6 | """Base class for KMS resources.""" 7 | 8 | service_name = "kms" 9 | -------------------------------------------------------------------------------- /altimeter/aws/settings.py: -------------------------------------------------------------------------------- 1 | """AWS scan settings 2 | 3 | Attributes: 4 | GRAPH_NAME: name for the AWS graph. 5 | GRAPH_VERSION: AWS graph version. Generally this should change very rarely. 6 | """ 7 | 8 | GRAPH_NAME: str = "alti" 9 | GRAPH_VERSION: str = "2" 10 | -------------------------------------------------------------------------------- /altimeter/qj/schemas/status.py: -------------------------------------------------------------------------------- 1 | # pylint: disable=too-few-public-methods 2 | """Pydantic Status schemas""" 3 | from pydantic import BaseModel # pylint: disable=no-name-in-module 4 | 5 | 6 | class Status(BaseModel): 7 | """Status schema""" 8 | 9 | status: str 10 | -------------------------------------------------------------------------------- /altimeter/aws/auth/exceptions.py: -------------------------------------------------------------------------------- 1 | """Exceptions for access related errors.""" 2 | from altimeter.core.exceptions import AltimeterException 3 | 4 | 5 | class AccountAuthException(AltimeterException): 6 | """Exception indicating auth was unable to be obtained to an account.""" 7 | -------------------------------------------------------------------------------- /requirements.in: -------------------------------------------------------------------------------- 1 | MarkupSafe==2.1.1 2 | aws-requests-auth==0.4.3 3 | rdflib==6.0.2 4 | structlog==20.2.0 5 | boto3==1.28.80 6 | jinja2==3.0.3 7 | pydantic==1.9.0 8 | toml==0.10.2 9 | gremlinpython==3.4.12 10 | requests==2.31.0 11 | certifi==2023.7.22 12 | urllib3==1.26.18 13 | -------------------------------------------------------------------------------- /altimeter/aws/resource/acm/__init__.py: -------------------------------------------------------------------------------- 1 | """Base class for acm resources.""" 2 | 3 | from altimeter.aws.resource.resource_spec import AWSResourceSpec 4 | 5 | 6 | class ACMResourceSpec(AWSResourceSpec): 7 | """Base class for acm resources.""" 8 | 9 | service_name = "acm" 10 | -------------------------------------------------------------------------------- /altimeter/aws/resource/rds/__init__.py: -------------------------------------------------------------------------------- 1 | """Base class for rds resources.""" 2 | 3 | from altimeter.aws.resource.resource_spec import AWSResourceSpec 4 | 5 | 6 | class RDSResourceSpec(AWSResourceSpec): 7 | """Base class for rds resources.""" 8 | 9 | service_name = "rds" 10 | -------------------------------------------------------------------------------- /services/qj/requirements.in: -------------------------------------------------------------------------------- 1 | MarkupSafe==2.1.1 2 | alembic==1.4.2 3 | boto3==1.28.80 4 | fastapi==0.96.0 5 | psycopg2-binary==2.9.2 6 | requests==2.31.0 7 | sqlalchemy==1.3.24 8 | tableauhyperapi==0.0.18161 9 | tableauserverclient==0.17.0 10 | uvicorn==0.16.0 11 | urllib3==1.26.18 12 | -------------------------------------------------------------------------------- /altimeter/aws/resource/awslambda/__init__.py: -------------------------------------------------------------------------------- 1 | """Base class for lambda resources.""" 2 | 3 | from altimeter.aws.resource.resource_spec import AWSResourceSpec 4 | 5 | 6 | class LambdaResourceSpec(AWSResourceSpec): 7 | """Base class for lambda resources.""" 8 | 9 | service_name = "lambda" 10 | -------------------------------------------------------------------------------- /altimeter/aws/resource/dynamodb/__init__.py: -------------------------------------------------------------------------------- 1 | """Base class for DynamoDB resources.""" 2 | 3 | from altimeter.aws.resource.resource_spec import AWSResourceSpec 4 | 5 | 6 | class DynamoDBResourceSpec(AWSResourceSpec): 7 | """Base class for DynamoDB resources.""" 8 | 9 | service_name = "dynamodb" 10 | -------------------------------------------------------------------------------- /altimeter/aws/resource/ec2/__init__.py: -------------------------------------------------------------------------------- 1 | """AWSResourceSpec subclass for ec2 resources.""" 2 | 3 | from altimeter.aws.resource.resource_spec import AWSResourceSpec 4 | 5 | 6 | class EC2ResourceSpec(AWSResourceSpec): 7 | """AWSResourceSpec subclass for ec2 resources.""" 8 | 9 | service_name = "ec2" 10 | -------------------------------------------------------------------------------- /altimeter/aws/resource/eks/__init__.py: -------------------------------------------------------------------------------- 1 | """AWSResourceSpec subclass for eks resources.""" 2 | 3 | from altimeter.aws.resource.resource_spec import AWSResourceSpec 4 | 5 | 6 | class EKSResourceSpec(AWSResourceSpec): 7 | """AWSResourceSpec subclass for eks resources.""" 8 | 9 | service_name = "eks" 10 | -------------------------------------------------------------------------------- /altimeter/aws/resource/elbv2/__init__.py: -------------------------------------------------------------------------------- 1 | """ResourceSpec classes for elbv2 resources.""" 2 | from altimeter.aws.resource.resource_spec import AWSResourceSpec 3 | 4 | 5 | class ELBV2ResourceSpec(AWSResourceSpec): 6 | """Abstract base for ResourceSpec classes for elbv2 resources.""" 7 | 8 | service_name = "elbv2" 9 | -------------------------------------------------------------------------------- /altimeter/aws/resource/guardduty/__init__.py: -------------------------------------------------------------------------------- 1 | """Base class for GuardDuty resources.""" 2 | 3 | from altimeter.aws.resource.resource_spec import AWSResourceSpec 4 | 5 | 6 | class GuardDutyResourceSpec(AWSResourceSpec): 7 | """Base class for GuardDuty resources.""" 8 | 9 | service_name = "guardduty" 10 | -------------------------------------------------------------------------------- /pytest.ini: -------------------------------------------------------------------------------- 1 | [pytest] 2 | filterwarnings = 3 | ignore::DeprecationWarning:botocore.vendored.requests*: 4 | ignore:the imp module is deprecated in favour of importlib:DeprecationWarning:boto.plugin*: 5 | ignore:Using or importing the ABCs from 'collections' instead of from 'collections.abc':DeprecationWarning:jose.jws*: 6 | -------------------------------------------------------------------------------- /altimeter/aws/resource/cloudtrail/__init__.py: -------------------------------------------------------------------------------- 1 | """Base class for CloudTrail resources.""" 2 | 3 | from altimeter.aws.resource.resource_spec import AWSResourceSpec 4 | 5 | 6 | class CloudTrailResourceSpec(AWSResourceSpec): 7 | """Base class for CloudTrail resources.""" 8 | 9 | service_name = "cloudtrail" 10 | -------------------------------------------------------------------------------- /altimeter/aws/resource/events/__init__.py: -------------------------------------------------------------------------------- 1 | """Base class for CloudWatch Events resources.""" 2 | 3 | from altimeter.aws.resource.resource_spec import AWSResourceSpec 4 | 5 | 6 | class EventsResourceSpec(AWSResourceSpec): 7 | """Base class for CloudWatch Events resources.""" 8 | 9 | service_name = "events" 10 | -------------------------------------------------------------------------------- /altimeter/aws/resource/elbv1/__init__.py: -------------------------------------------------------------------------------- 1 | """ResourceSpec classes for classic elb resources.""" 2 | from altimeter.aws.resource.resource_spec import AWSResourceSpec 3 | 4 | 5 | class ELBV1ResourceSpec(AWSResourceSpec): 6 | """Abstract base for ResourceSpec classes for classic elb resources.""" 7 | 8 | service_name = "elb" 9 | -------------------------------------------------------------------------------- /qj-lambda.Dockerfile: -------------------------------------------------------------------------------- 1 | FROM amazon/aws-lambda-python:3.8 2 | 3 | COPY . /tmp/src 4 | COPY bin/queryjob_lambda.py "${LAMBDA_TASK_ROOT}" 5 | 6 | RUN pip install -r /tmp/src/requirements.txt 7 | RUN cd /tmp/src && pip install .[qj] && rm -rf /tmp/src 8 | 9 | STOPSIGNAL SIGTERM 10 | 11 | CMD ["queryjob_lambda.lambda_handler"] 12 | -------------------------------------------------------------------------------- /altimeter/qj/api/base/api.py: -------------------------------------------------------------------------------- 1 | """Base API router""" 2 | from fastapi import APIRouter 3 | 4 | from altimeter.qj.api.base.endpoints import auth, status 5 | 6 | BASE_ROUTER = APIRouter() 7 | BASE_ROUTER.include_router(auth.ROUTER, prefix="/auth", tags=["auth"]) 8 | BASE_ROUTER.include_router(status.ROUTER, prefix="/status", tags=["status"]) 9 | -------------------------------------------------------------------------------- /altimeter/qj/db/base.py: -------------------------------------------------------------------------------- 1 | """All modes are imported here such that Base has them before being used. 2 | In general Base should be imported from here.""" 3 | # noqa # pylint: disable=unused-import 4 | from altimeter.qj.db.base_class import BASE 5 | from altimeter.qj.models.job import Job 6 | from altimeter.qj.models.result_set import ResultSet, Result 7 | -------------------------------------------------------------------------------- /scanner.Dockerfile: -------------------------------------------------------------------------------- 1 | FROM python:3.8-slim 2 | 3 | COPY . /tmp/src 4 | 5 | RUN groupadd -r altimeter && useradd -r -s /bin/false -g altimeter altimeter 6 | RUN pip install -r /tmp/src/requirements.txt 7 | RUN cd /tmp/src && python setup.py install && rm -rf /tmp/src 8 | 9 | STOPSIGNAL SIGTERM 10 | 11 | USER altimeter 12 | 13 | CMD aws2n.py 14 | -------------------------------------------------------------------------------- /tests/requirements.in: -------------------------------------------------------------------------------- 1 | MarkupSafe==2.1.1 2 | sqlalchemy==1.3.24 3 | alembic==1.4.2 4 | colorama==0.4.1 5 | coverage==7.2.7 6 | moto==4.1.11 7 | pytest==7.3.1 8 | pytest-cov==4.1.0 9 | boto3==1.28.80 10 | werkzeug==2.2.3 11 | cryptography==41.0.5 12 | docker==6.1.3 13 | requests==2.31.0 14 | certifi==2023.7.22 15 | urllib3==1.26.18 16 | cffi==1.16.0 17 | -------------------------------------------------------------------------------- /altimeter/aws/resource/route53/__init__.py: -------------------------------------------------------------------------------- 1 | """Base class for Route53 resources.""" 2 | from altimeter.aws.resource.resource_spec import AWSResourceSpec, ScanGranularity 3 | 4 | 5 | class Route53ResourceSpec(AWSResourceSpec): 6 | """Base class for Route53 resources.""" 7 | 8 | scan_granularity = ScanGranularity.ACCOUNT 9 | service_name = "route53" 10 | -------------------------------------------------------------------------------- /altimeter/qj/schemas/remediation.py: -------------------------------------------------------------------------------- 1 | """Remediation schema""" 2 | # pylint: disable=too-few-public-methods 3 | 4 | from pydantic import BaseModel 5 | 6 | 7 | class Remediation(BaseModel): 8 | """Remediation schema""" 9 | 10 | job_name: str 11 | result_set_id: str 12 | 13 | class Config: 14 | """Pydantic config overrides""" 15 | 16 | orm_mode = True 17 | -------------------------------------------------------------------------------- /altimeter/qj/api/base/endpoints/status.py: -------------------------------------------------------------------------------- 1 | """Endpoints for service status""" 2 | from typing import Any 3 | 4 | from fastapi import APIRouter 5 | 6 | from altimeter.qj import schemas 7 | 8 | ROUTER = APIRouter() 9 | 10 | 11 | @ROUTER.get("", response_model=schemas.Status) 12 | def read_status() -> Any: 13 | """Get application status""" 14 | return schemas.Status(status="ok") 15 | -------------------------------------------------------------------------------- /altimeter/qj/api/base/endpoints/auth.py: -------------------------------------------------------------------------------- 1 | """Endpoints for service auth""" 2 | from typing import Any 3 | 4 | from fastapi import APIRouter, Security 5 | 6 | from altimeter.qj.api import deps 7 | 8 | ROUTER = APIRouter() 9 | 10 | 11 | @ROUTER.get("") 12 | def get_auth( 13 | api_key: str = Security(deps.api_key), 14 | ) -> Any: 15 | """Get the current auth token""" 16 | return api_key 17 | -------------------------------------------------------------------------------- /sfn-init-lambda.Dockerfile: -------------------------------------------------------------------------------- 1 | FROM amazon/aws-lambda-python:3.8 2 | 3 | COPY . /tmp/src 4 | COPY bin/sfn_init.py "${LAMBDA_TASK_ROOT}" 5 | 6 | RUN rm -rf "${LAMBDA_RUNTIME_DIR}"/boto3* "${LAMBDA_RUNTIME_DIR}"/botocore* 7 | RUN pip install -r /tmp/src/requirements.txt 8 | RUN cd /tmp/src && python setup.py install && rm -rf /tmp/src 9 | 10 | STOPSIGNAL SIGTERM 11 | 12 | CMD ["sfn_init.lambda_handler"] 13 | -------------------------------------------------------------------------------- /altimeter/aws/resource/support/__init__.py: -------------------------------------------------------------------------------- 1 | """Base class for Support resources.""" 2 | from altimeter.aws.resource.resource_spec import ScanGranularity, AWSResourceSpec 3 | 4 | 5 | class SupportResourceSpec(AWSResourceSpec): 6 | """Base class for Support resources.""" 7 | 8 | service_name = "support" 9 | scan_granularity = ScanGranularity.ACCOUNT 10 | region_whitelist = ("us-east-1",) 11 | -------------------------------------------------------------------------------- /sfn-load-rdf-lambda.Dockerfile: -------------------------------------------------------------------------------- 1 | FROM amazon/aws-lambda-python:3.8 2 | 3 | COPY . /tmp/src 4 | COPY bin/sfn_load_rdf.py "${LAMBDA_TASK_ROOT}" 5 | 6 | RUN rm -rf "${LAMBDA_RUNTIME_DIR}"/boto3* "${LAMBDA_RUNTIME_DIR}"/botocore* 7 | RUN pip install -r /tmp/src/requirements.txt 8 | RUN cd /tmp/src && python setup.py install && rm -rf /tmp/src 9 | 10 | STOPSIGNAL SIGTERM 11 | 12 | CMD ["sfn_load_rdf.lambda_handler"] 13 | -------------------------------------------------------------------------------- /altimeter-lambda.Dockerfile: -------------------------------------------------------------------------------- 1 | FROM amazon/aws-lambda-python:3.8 2 | 3 | COPY . /tmp/src 4 | COPY bin/altimeter_lambda.py "${LAMBDA_TASK_ROOT}" 5 | 6 | RUN rm -rf "${LAMBDA_RUNTIME_DIR}"/boto3* "${LAMBDA_RUNTIME_DIR}"/botocore* 7 | RUN pip install -r /tmp/src/requirements.txt 8 | RUN cd /tmp/src && python setup.py install && rm -rf /tmp/src 9 | 10 | STOPSIGNAL SIGTERM 11 | 12 | CMD ["altimeter_lambda.lambda_handler"] 13 | -------------------------------------------------------------------------------- /altimeter/qj/api/v1/api.py: -------------------------------------------------------------------------------- 1 | """V1 API router""" 2 | from fastapi import APIRouter 3 | 4 | from altimeter.qj.api.v1.endpoints.jobs import JOBS_ROUTER 5 | from altimeter.qj.api.v1.endpoints.result_sets import RESULT_SETS_ROUTER 6 | 7 | V1_ROUTER = APIRouter() 8 | V1_ROUTER.include_router(JOBS_ROUTER, prefix="/jobs", tags=["jobs"]) 9 | V1_ROUTER.include_router(RESULT_SETS_ROUTER, prefix="/result_sets", tags=["result_sets"]) 10 | -------------------------------------------------------------------------------- /qj.Dockerfile: -------------------------------------------------------------------------------- 1 | FROM tiangolo/uvicorn-gunicorn-fastapi:python3.8 2 | 3 | COPY ./requirements.txt /tmp 4 | RUN pip install -r /tmp/requirements.txt 5 | RUN rm /tmp/requirements.txt 6 | COPY ./services/qj/requirements.txt /tmp 7 | RUN pip install -r /tmp/requirements.txt 8 | RUN rm /tmp/requirements.txt 9 | 10 | COPY ./services/qj/gunicorn_conf.py /app 11 | COPY ./services/qj/main.py /app 12 | COPY ./altimeter /app/altimeter 13 | -------------------------------------------------------------------------------- /sfn-prune-graphs-lambda.Dockerfile: -------------------------------------------------------------------------------- 1 | FROM amazon/aws-lambda-python:3.8 2 | 3 | COPY . /tmp/src 4 | COPY bin/sfn_prune_graphs.py "${LAMBDA_TASK_ROOT}" 5 | 6 | RUN rm -rf "${LAMBDA_RUNTIME_DIR}"/boto3* "${LAMBDA_RUNTIME_DIR}"/botocore* 7 | RUN pip install -r /tmp/src/requirements.txt 8 | RUN cd /tmp/src && python setup.py install && rm -rf /tmp/src 9 | 10 | STOPSIGNAL SIGTERM 11 | 12 | CMD ["sfn_prune_graphs.lambda_handler"] 13 | -------------------------------------------------------------------------------- /sfn-scan-account-lambda.Dockerfile: -------------------------------------------------------------------------------- 1 | FROM amazon/aws-lambda-python:3.8 2 | 3 | COPY . /tmp/src 4 | COPY bin/sfn_scan_account.py "${LAMBDA_TASK_ROOT}" 5 | 6 | RUN rm -rf "${LAMBDA_RUNTIME_DIR}"/boto3* "${LAMBDA_RUNTIME_DIR}"/botocore* 7 | RUN pip install -r /tmp/src/requirements.txt 8 | RUN cd /tmp/src && python setup.py install && rm -rf /tmp/src 9 | 10 | STOPSIGNAL SIGTERM 11 | 12 | CMD ["sfn_scan_account.lambda_handler"] 13 | -------------------------------------------------------------------------------- /sfn-compile-graphs-lambda.Dockerfile: -------------------------------------------------------------------------------- 1 | FROM amazon/aws-lambda-python:3.8 2 | 3 | COPY . /tmp/src 4 | COPY bin/sfn_compile_graphs.py "${LAMBDA_TASK_ROOT}" 5 | 6 | RUN rm -rf "${LAMBDA_RUNTIME_DIR}"/boto3* "${LAMBDA_RUNTIME_DIR}"/botocore* 7 | RUN pip install -r /tmp/src/requirements.txt 8 | RUN cd /tmp/src && python setup.py install && rm -rf /tmp/src 9 | 10 | STOPSIGNAL SIGTERM 11 | 12 | CMD ["sfn_compile_graphs.lambda_handler"] 13 | -------------------------------------------------------------------------------- /altimeter/core/resource/exceptions.py: -------------------------------------------------------------------------------- 1 | """Resource related Exceptions.""" 2 | from altimeter.core.exceptions import AltimeterException 3 | 4 | 5 | class ResourceSpecClassNotFoundException(AltimeterException): 6 | """A specified ResourceSpecClass can not be found.""" 7 | 8 | 9 | class MultipleResourceSpecClassesFoundException(AltimeterException): 10 | """More than one ResourceSpec class exist for a given specification.""" 11 | -------------------------------------------------------------------------------- /altimeter/qj/settings.py: -------------------------------------------------------------------------------- 1 | """Settings""" 2 | 3 | API_KEY_HEADER_NAME = "X-API-KEY" 4 | DEFAULT_RESULT_EXPIRATION_SEC_DEFAULT = 7 * 24 * 60 * 60 # 7 days 5 | DEFAULT_RESULT_EXPIRATION_SEC_LIMIT = 14 * 24 * 60 * 60 # 14 days 6 | DEFAULT_MAX_GRAPH_AGE_SEC_DEFAULT = 150 * 60 # 150min 7 | DEFAULT_MAX_GRAPH_AGE_SEC_LIMIT = 7 * 24 * 60 * 60 # 1wk 8 | DEFAULT_MAX_RESULT_AGE_SEC_DEFAULT = 60 * 60 * 4 # 4 hours 9 | DEFAULT_MAX_RESULT_AGE_SEC_LIMIT = 60 * 60 * 24 # 1day 10 | -------------------------------------------------------------------------------- /bin/altimeter: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -ef -o pipefail 4 | 5 | if [[ -z "$AWS_DEFAULT_REGION" ]]; then 6 | echo "The env var AWS_DEFAULT_REGION must be defined" 7 | exit 1 8 | fi 9 | 10 | set -u 11 | 12 | # trick to use fd5 and tee to output the full output to console while 13 | # also saving the last line (which is the json filepaths used by the next 14 | # process) in a var. 15 | exec 5>&1 16 | rdf_path=$("aws2n.py" $@ | tee /dev/fd/5 | tail -n1) 17 | echo "Created RDF $rdf_path" 18 | -------------------------------------------------------------------------------- /altimeter/core/json_encoder.py: -------------------------------------------------------------------------------- 1 | """Function for encoding JSON with datetimes.""" 2 | from datetime import datetime 3 | from typing import Any 4 | 5 | 6 | def json_encoder(obj: Any) -> Any: 7 | """json encoder function supporting datetime serialization. 8 | 9 | Args: 10 | obj: object to encode to JSON 11 | 12 | Returns: 13 | json encoded data 14 | """ 15 | if isinstance(obj, datetime): 16 | return obj.isoformat() 17 | raise TypeError("Type {} not serializable".format(type(obj))) 18 | -------------------------------------------------------------------------------- /altimeter/qj/schemas/__init__.py: -------------------------------------------------------------------------------- 1 | """All Pydantic 'schema' classes should be imported here""" 2 | from altimeter.qj.schemas.job import Job, JobCreate, JobGraphSpec, JobUpdate, Category, Severity 3 | from altimeter.qj.schemas.result_set import ( 4 | Result, 5 | ResultSet, 6 | ResultSetCreate, 7 | ResultSetGraphSpec, 8 | ResultSetsPruneResult, 9 | ResultSetFormat, 10 | ) 11 | from altimeter.qj.schemas.result_set_notification import ResultSetNotification 12 | from altimeter.qj.schemas.status import Status 13 | -------------------------------------------------------------------------------- /altimeter/core/graph/field/util.py: -------------------------------------------------------------------------------- 1 | """Grab-bag functions for Field parsing""" 2 | import re 3 | 4 | 5 | FIRST_CAP_RE = re.compile("(.)([A-Z][a-z]+)") 6 | ALL_CAP_RE = re.compile("([a-z0-9])([A-Z])") 7 | 8 | 9 | def camel_case_to_snake_case(name: str) -> str: 10 | """Convert a string from CamelCase into snake_case. 11 | Args: 12 | name: string to convert 13 | 14 | Returns: 15 | snake cased string 16 | """ 17 | first_capped_str = FIRST_CAP_RE.sub(r"\1_\2", name) 18 | return ALL_CAP_RE.sub(r"\1_\2", first_capped_str).lower() 19 | -------------------------------------------------------------------------------- /tests/unit/altimeter/core/graph/test_node_cache.py: -------------------------------------------------------------------------------- 1 | from unittest import TestCase 2 | 3 | from altimeter.core.graph.node_cache import NodeCache 4 | 5 | 6 | class TestNodeCache(TestCase): 7 | def test_setitem_new_key(self): 8 | node_cache = NodeCache() 9 | node_cache["foo"] = "boo" 10 | self.assertEqual(node_cache["foo"], "boo") 11 | 12 | def test_setitem_existing_key(self): 13 | node_cache = NodeCache() 14 | node_cache["foo"] = "boo" 15 | with self.assertRaises(KeyError): 16 | node_cache["foo"] = "goo" 17 | -------------------------------------------------------------------------------- /tests/unit/altimeter/core/graph/field/test_util.py: -------------------------------------------------------------------------------- 1 | from unittest import TestCase 2 | 3 | from altimeter.core.graph.field.util import camel_case_to_snake_case 4 | 5 | 6 | class TestCamelCaseToSnakeCase(TestCase): 7 | def test_snake_case_input(self): 8 | test_str = "snake_case_input" 9 | self.assertEqual(test_str, camel_case_to_snake_case(test_str)) 10 | 11 | def test_camel_case_input(self): 12 | test_str = "CamelCaseInput" 13 | expected_out = "camel_case_input" 14 | self.assertEqual(expected_out, camel_case_to_snake_case(test_str)) 15 | -------------------------------------------------------------------------------- /altimeter/qj/schemas/result_set_notification.py: -------------------------------------------------------------------------------- 1 | # pylint: disable=too-few-public-methods 2 | """Pydantic Notification schemas""" 3 | from datetime import datetime 4 | 5 | from pydantic import BaseModel # pylint: disable=no-name-in-module 6 | 7 | from altimeter.qj.schemas.job import Job 8 | from altimeter.qj.schemas.result_set import ResultSetGraphSpec 9 | 10 | 11 | class ResultSetNotification(BaseModel): 12 | """ResultSetNotification schema""" 13 | 14 | job: Job 15 | graph_spec: ResultSetGraphSpec 16 | created: datetime 17 | num_results: int 18 | result_set_id: str 19 | -------------------------------------------------------------------------------- /altimeter/qj/security.py: -------------------------------------------------------------------------------- 1 | """Security related functions""" 2 | import boto3 3 | 4 | from altimeter.qj.config import SecurityConfig 5 | 6 | 7 | def get_api_key(region_name: str, version_stage: str = "AWSCURRENT") -> str: 8 | """Get the current API key from SecretsManager""" 9 | security_config = SecurityConfig() 10 | sm_client = boto3.client("secretsmanager", region_name=region_name) 11 | resp = sm_client.get_secret_value( 12 | SecretId=security_config.api_key_secret_name, VersionStage=version_stage 13 | ) 14 | api_key_secret = resp["SecretString"] 15 | return api_key_secret 16 | -------------------------------------------------------------------------------- /altimeter/core/base_model.py: -------------------------------------------------------------------------------- 1 | """Base pydantic altimeter model classes""" 2 | from pydantic import BaseModel 3 | 4 | 5 | class BaseImmutableModel(BaseModel): 6 | """Base immutable pydantic altimeter model""" 7 | 8 | class Config: 9 | """Pydantic config""" 10 | 11 | allow_mutation = False 12 | extra = "forbid" 13 | arbitrary_types_allowed = True 14 | 15 | 16 | class BaseMutableModel(BaseModel): 17 | """Base mutable pydantic altimeter model""" 18 | 19 | class Config: 20 | """Pydantic config""" 21 | 22 | extra = "forbid" 23 | arbitrary_types_allowed = True 24 | -------------------------------------------------------------------------------- /services/qj/alembic/script.py.mako: -------------------------------------------------------------------------------- 1 | """${message} 2 | 3 | Revision ID: ${up_revision} 4 | Revises: ${down_revision | comma,n} 5 | Create Date: ${create_date} 6 | 7 | """ 8 | from alembic import op 9 | import sqlalchemy as sa 10 | ${imports if imports else ""} 11 | 12 | # revision identifiers, used by Alembic. 13 | revision = ${repr(up_revision)} 14 | down_revision = ${repr(down_revision)} 15 | branch_labels = ${repr(branch_labels)} 16 | depends_on = ${repr(depends_on)} 17 | 18 | 19 | def upgrade(): 20 | ${upgrades if upgrades else "pass"} 21 | 22 | 23 | def downgrade(): 24 | ${downgrades if downgrades else "pass"} 25 | -------------------------------------------------------------------------------- /altimeter/aws/scan/account_scan_manifest.py: -------------------------------------------------------------------------------- 1 | """An AccountScanManifest defines the output of an account scan.""" 2 | 3 | from dataclasses import dataclass 4 | from typing import List 5 | 6 | 7 | @dataclass(frozen=True) 8 | class AccountScanManifest: 9 | """An AccountScanManifest defines the output of an account scan. It contains pointers to the 10 | scan result artifacts and summaries of what was scanned and errors which occurred. 11 | 12 | Args: 13 | account_id: account id 14 | artifacts: list of scan artifacts 15 | errors: list of error strings 16 | """ 17 | 18 | account_id: str 19 | artifacts: List[str] 20 | errors: List[str] 21 | -------------------------------------------------------------------------------- /tests/unit/altimeter/qj/test_security.py: -------------------------------------------------------------------------------- 1 | import os 2 | from unittest import mock 3 | 4 | import boto3 5 | import moto 6 | 7 | from altimeter.qj.config import SecurityConfig 8 | from altimeter.qj.security import get_api_key 9 | 10 | 11 | @moto.mock_secretsmanager 12 | def test_get_api_key(): 13 | with mock.patch.dict(os.environ, {"API_KEY_SECRET_NAME": "test-api-key"}): 14 | client = boto3.client("secretsmanager", region_name="us-west-2") 15 | client.create_secret( 16 | Name=SecurityConfig().api_key_secret_name, 17 | SecretString="testvalue123", 18 | ) 19 | api_key = get_api_key(region_name="us-west-2") 20 | assert api_key == "testvalue123" 21 | -------------------------------------------------------------------------------- /doc/source/user/local_blazegraph.rst: -------------------------------------------------------------------------------- 1 | Local Querying with Blazegraph 2 | ============================== 3 | 4 | Once you've generated an RDF (:doc:`Quickstart `) you 5 | can load these results into a local Blazegraph instance for querying: 6 | 7 | :: 8 | 9 | rdf2blaze 10 | 11 | This command will start a Blazegraph docker container, load the rdf and print 12 | details on accessing it: 13 | 14 | :: 15 | 16 | Query UI is available at http://localhost:8889/bigdata/#query 17 | 18 | Hit CTRL-C to exit. 19 | 20 | Loading the url above in the browser will open a UI where SPARQL queries can 21 | be run against the graph. 22 | 23 | Some sample SPARQL queries are included in :doc:`Sample Queries `. 24 | -------------------------------------------------------------------------------- /tests/unit/altimeter/core/test_json_encoder.py: -------------------------------------------------------------------------------- 1 | import datetime 2 | import json 3 | 4 | from unittest import TestCase 5 | 6 | from altimeter.core.json_encoder import json_encoder 7 | 8 | 9 | class TestJsonEncoder(TestCase): 10 | def test_with_timestamp(self): 11 | now = datetime.datetime.now() 12 | data = {"foo": now} 13 | json_str = json.dumps(data, default=json_encoder) 14 | expected_str = f'{{"foo": "{now.isoformat()}"}}' 15 | self.assertEqual(json_str, expected_str) 16 | 17 | def test_with_unserializable(self): 18 | class Foo: 19 | boo = 1 20 | 21 | data = {"foo": Foo} 22 | with self.assertRaises(TypeError): 23 | json.dumps(data, default=json_encoder) 24 | -------------------------------------------------------------------------------- /altimeter/core/graph/node_cache.py: -------------------------------------------------------------------------------- 1 | """A NodeCache is a simple cache for graph nodes.""" 2 | from typing import Union 3 | 4 | from rdflib import BNode, URIRef 5 | 6 | 7 | class NodeCache(dict): 8 | """A NodeCache is a simple cache for graph nodes. Unlike a standard 9 | dict it does not allow overwriting entries.""" 10 | 11 | def __setitem__(self, key: str, value: Union[BNode, URIRef]) -> None: 12 | """Set an item in this NodeCache 13 | key: cache key 14 | value: BNode value 15 | 16 | Raises: 17 | KeyError if this key is already present. 18 | """ 19 | if key in self: 20 | raise KeyError(f"Key already present for {key}") 21 | super().__setitem__(key, value) 22 | -------------------------------------------------------------------------------- /altimeter/qj/lambdas/executor.py: -------------------------------------------------------------------------------- 1 | """Execute all known QJs""" 2 | import json 3 | from typing import Any, Dict, List 4 | 5 | from altimeter.core.log import Logger 6 | from altimeter.qj.client import QJAPIClient 7 | from altimeter.qj.config import ExecutorConfig 8 | from altimeter.qj.log import QJLogEvents 9 | 10 | 11 | def executor(_: Dict[str, Any]) -> List[str]: 12 | """Return the name of active QJ names""" 13 | exec_config = ExecutorConfig() 14 | logger = Logger() 15 | logger.info(event=QJLogEvents.InitConfig) 16 | qj_client = QJAPIClient(host=exec_config.api_host, port=exec_config.api_port) 17 | jobs = qj_client.get_jobs(active_only=True) 18 | logger.info(event=QJLogEvents.GetJobs, num_jobs=len(jobs)) 19 | return [job.name for job in jobs] 20 | -------------------------------------------------------------------------------- /bin/sfn_prune_graphs.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | """Prune graphs from Neptune""" 3 | import logging 4 | from typing import Any, Dict 5 | 6 | from altimeter.core.base_model import BaseImmutableModel 7 | from altimeter.core.config import Config 8 | from altimeter.core.pruner import prune_graph_from_config 9 | 10 | 11 | class PruneGraphsInput(BaseImmutableModel): 12 | config: Config 13 | 14 | 15 | def lambda_handler(event: Dict[str, Any], _: Any) -> Dict[str, Any]: 16 | """Lambda entrypoint""" 17 | root = logging.getLogger() 18 | if root.handlers: 19 | for handler in root.handlers: 20 | root.removeHandler(handler) 21 | prune_graphs_input = PruneGraphsInput(**event) 22 | prune_results = prune_graph_from_config(prune_graphs_input.config) 23 | return prune_results.dict() 24 | -------------------------------------------------------------------------------- /tests/unit/altimeter/core/graph/test_schema.py: -------------------------------------------------------------------------------- 1 | from unittest import TestCase 2 | 3 | from altimeter.core.graph.field.scalar_field import ScalarField 4 | from altimeter.core.graph.links import LinkCollection, SimpleLink 5 | from altimeter.core.graph.schema import Schema 6 | 7 | 8 | class TestSchema(TestCase): 9 | def test_parse(self): 10 | schema = Schema(ScalarField("Key1"), ScalarField("Key2")) 11 | data = {"Key1": "Value1", "Key2": "Value2"} 12 | link_collection = schema.parse(data, {}) 13 | expected_link_collection = LinkCollection( 14 | simple_links=( 15 | SimpleLink(pred="key1", obj="Value1"), 16 | SimpleLink(pred="key2", obj="Value2"), 17 | ) 18 | ) 19 | self.assertEqual(link_collection, expected_link_collection) 20 | -------------------------------------------------------------------------------- /services/qj/alembic/versions/94f36533d115_remove_result_synthetic_pk.py: -------------------------------------------------------------------------------- 1 | """Remove result synthetic pk 2 | 3 | Revision ID: 94f36533d115 4 | Revises: e6e2a6bf2a39 5 | Create Date: 2022-08-16 09:58:18.309009 6 | 7 | """ 8 | from alembic import op 9 | import sqlalchemy as sa 10 | 11 | 12 | # revision identifiers, used by Alembic. 13 | revision = '94f36533d115' 14 | down_revision = 'e6e2a6bf2a39' 15 | branch_labels = None 16 | depends_on = None 17 | 18 | 19 | def upgrade(): 20 | # ### commands auto generated by Alembic - please adjust! ### 21 | op.drop_column('result', 'id') 22 | # ### end Alembic commands ### 23 | 24 | 25 | def downgrade(): 26 | # ### commands auto generated by Alembic - please adjust! ### 27 | op.add_column('result', sa.Column('id', sa.INTEGER(), autoincrement=True, nullable=False)) 28 | # ### end Alembic commands ### 29 | -------------------------------------------------------------------------------- /services/qj/alembic/versions/e6e2a6bf2a39_adding_query_job_raw_query_column.py: -------------------------------------------------------------------------------- 1 | """Adding query job 'raw_query' column 2 | 3 | Revision ID: e6e2a6bf2a39 4 | Revises: 9d956e753055 5 | Create Date: 2022-03-03 10:00:38.059957 6 | 7 | """ 8 | from alembic import op 9 | import sqlalchemy as sa 10 | 11 | 12 | # revision identifiers, used by Alembic. 13 | revision = 'e6e2a6bf2a39' 14 | down_revision = '9d956e753055' 15 | branch_labels = None 16 | depends_on = None 17 | 18 | 19 | def upgrade(): 20 | # ### commands auto generated by Alembic - please adjust! ### 21 | op.add_column('job', sa.Column('raw_query', sa.Boolean(), server_default='false', nullable=False)) 22 | # ### end Alembic commands ### 23 | 24 | 25 | def downgrade(): 26 | # ### commands auto generated by Alembic - please adjust! ### 27 | op.drop_column('job', 'raw_query') 28 | # ### end Alembic commands ### 29 | -------------------------------------------------------------------------------- /services/qj/alembic/versions/9d956e753055_added_remediate_sqs_queue_column_to_qj_.py: -------------------------------------------------------------------------------- 1 | """Added remediate_sqs_queue column to qj job table 2 | 3 | Revision ID: 9d956e753055 4 | Revises: 60990e9bc347 5 | Create Date: 2021-09-22 10:11:51.243002 6 | 7 | """ 8 | from alembic import op 9 | import sqlalchemy as sa 10 | 11 | 12 | # revision identifiers, used by Alembic. 13 | revision = '9d956e753055' 14 | down_revision = '60990e9bc347' 15 | branch_labels = None 16 | depends_on = None 17 | 18 | 19 | def upgrade(): 20 | # ### commands auto generated by Alembic - please adjust! ### 21 | op.add_column('job', sa.Column('remediate_sqs_queue', sa.Text(), nullable=True)) 22 | # ### end Alembic commands ### 23 | 24 | 25 | def downgrade(): 26 | # ### commands auto generated by Alembic - please adjust! ### 27 | op.drop_column('job', 'remediate_sqs_queue') 28 | # ### end Alembic commands ### 29 | -------------------------------------------------------------------------------- /services/qj/alembic/versions/60990e9bc347_added_notify_if_results_bool_on_job.py: -------------------------------------------------------------------------------- 1 | """added notify_if_results bool on job 2 | 3 | Revision ID: 60990e9bc347 4 | Revises: dc8f1df07766 5 | Create Date: 2021-05-06 07:51:08.854424 6 | 7 | """ 8 | from alembic import op 9 | import sqlalchemy as sa 10 | 11 | 12 | # revision identifiers, used by Alembic. 13 | revision = '60990e9bc347' 14 | down_revision = 'dc8f1df07766' 15 | branch_labels = None 16 | depends_on = None 17 | 18 | 19 | def upgrade(): 20 | # ### commands auto generated by Alembic - please adjust! ### 21 | op.add_column('job', sa.Column('notify_if_results', sa.Boolean(), server_default='false', nullable=False)) 22 | # ### end Alembic commands ### 23 | 24 | 25 | def downgrade(): 26 | # ### commands auto generated by Alembic - please adjust! ### 27 | op.drop_column('job', 'notify_if_results') 28 | # ### end Alembic commands ### 29 | -------------------------------------------------------------------------------- /altimeter/qj/lambdas/pruner.py: -------------------------------------------------------------------------------- 1 | """Prune results according to Job config settings""" 2 | from altimeter.core.log import Logger 3 | from altimeter.qj.client import QJAPIClient 4 | from altimeter.qj.config import PrunerConfig 5 | from altimeter.qj.log import QJLogEvents 6 | from altimeter.qj.security import get_api_key 7 | 8 | 9 | def pruner() -> None: 10 | """Prune results according to Job config settings""" 11 | logger = Logger() 12 | pruner_config = PrunerConfig() 13 | logger.info(event=QJLogEvents.InitConfig, config=pruner_config) 14 | api_key = get_api_key(region_name=pruner_config.region) 15 | qj_client = QJAPIClient( 16 | host=pruner_config.api_host, port=pruner_config.api_port, api_key=api_key 17 | ) 18 | logger.info(event=QJLogEvents.DeleteStart) 19 | result = qj_client.delete_expired_result_sets() 20 | logger.info(event=QJLogEvents.DeleteEnd, result=result) 21 | -------------------------------------------------------------------------------- /tests/dbutil.py: -------------------------------------------------------------------------------- 1 | """DB utility functions for testing""" 2 | from contextlib import contextmanager 3 | from typing import Generator 4 | from unittest.mock import patch 5 | 6 | from sqlalchemy import create_engine 7 | from sqlalchemy.orm import Session 8 | 9 | from altimeter.qj.config import DBConfig 10 | 11 | 12 | @contextmanager 13 | def temp_db_session() -> Generator[Session, None, None]: 14 | """Return a db session which will be rolled back after the context is exited""" 15 | db_config = DBConfig() 16 | engine = create_engine(db_config.get_db_uri(), pool_pre_ping=True, pool_recycle=3600) 17 | connection = engine.connect() 18 | txn = connection.begin() 19 | session = Session(bind=connection) 20 | with patch("altimeter.qj.api.deps.SessionGenerator.get_session") as mock_get_session: 21 | mock_get_session.return_value = session 22 | try: 23 | yield session 24 | finally: 25 | session.close() 26 | txn.rollback() 27 | connection.close() 28 | -------------------------------------------------------------------------------- /tests/unit/altimeter/core/graph/field/test_base.py: -------------------------------------------------------------------------------- 1 | from typing import Any, Dict, List 2 | from unittest import TestCase 3 | 4 | from altimeter.core.graph.field.base import SubField 5 | from altimeter.core.graph.field.exceptions import ( 6 | ParentKeyMissingException, 7 | InvalidParentKeyException, 8 | ) 9 | from altimeter.core.graph.links import BaseLink 10 | 11 | 12 | class TestField(SubField): 13 | def parse(self, data: Any, context: Dict[str, Any]) -> List[BaseLink]: 14 | raise NotImplementedError() 15 | 16 | 17 | class TestSubField(TestCase): 18 | def test_missing_parent_alti_key(self): 19 | test_field = TestField() 20 | with self.assertRaises(ParentKeyMissingException): 21 | test_field.get_parent_alti_key(data={}, context={}) 22 | 23 | def test_nonstr_parent_alti_key(self): 24 | test_field = TestField() 25 | with self.assertRaises(InvalidParentKeyException): 26 | test_field.get_parent_alti_key(data={}, context={"parent_alti_key": TestField()}) 27 | -------------------------------------------------------------------------------- /altimeter/aws/resource/iam/__init__.py: -------------------------------------------------------------------------------- 1 | """Base class for IAM resources.""" 2 | from typing import Type 3 | 4 | from altimeter.aws.resource.resource_spec import ScanGranularity, AWSResourceSpec 5 | 6 | 7 | class IAMResourceSpec(AWSResourceSpec): 8 | """Base class for IAM resources.""" 9 | 10 | service_name = "iam" 11 | scan_granularity = ScanGranularity.ACCOUNT 12 | 13 | @classmethod 14 | def generate_arn( 15 | cls: Type[AWSResourceSpec], 16 | resource_id: str, 17 | account_id: str = "", 18 | region: str = "", 19 | ) -> str: 20 | """Generate an ARN for this resource 21 | 22 | Args: 23 | account_id: resource account id 24 | region: resource region 25 | resource_id: resource id 26 | 27 | Returns: 28 | string containing resource arn. 29 | """ 30 | return ( 31 | ":".join(("arn", cls.provider_name, cls.service_name, "", account_id, cls.type_name)) 32 | + f"/{resource_id}" 33 | ) 34 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # MacOS desktop services store files 2 | .DS_Store* 3 | 4 | # JetBrains IDE project files (e.g. PyCharm) 5 | .idea 6 | 7 | # VSCode IDE project files (e.g. PyCharm) 8 | .vscode 9 | 10 | # Environments 11 | myvenv/ 12 | .env 13 | .venv 14 | env/ 15 | venv/ 16 | ENV/ 17 | env.bak/ 18 | venv.bak/ 19 | 20 | # Python binary artifacts we generally never want to check in. 21 | *.pyc 22 | __pycache__/ 23 | 24 | # Artifacts from coverage 25 | cover/ 26 | .coverage 27 | htmlcov/ 28 | 29 | # mypy cache 30 | .mypy_cache 31 | 32 | #pytest cache 33 | .pytest_cache 34 | 35 | # Build artifacts 36 | dist/ 37 | build/ 38 | .eggs/ 39 | *.egg-info/ 40 | *.whl 41 | 42 | # Typical root venv dir 43 | venv/ 44 | .env/ 45 | 46 | # Terraform temporary files 47 | .terraform/ 48 | *.tfstate* 49 | 50 | # vim, emacs swap files 51 | .*.s?? 52 | *~ 53 | 54 | # sphinx sources we generally don't want to add 55 | altimeter*.rst 56 | modules.rst 57 | doc/html/**/* 58 | public/** 59 | 60 | # tox 61 | .tox/ 62 | 63 | # misc 64 | 65 | *.hyper 66 | *.log 67 | postgres_docker_container.id 68 | *.prof 69 | -------------------------------------------------------------------------------- /altimeter/qj/notifier.py: -------------------------------------------------------------------------------- 1 | """ResultSetNotifier - sends SNS messages when a result set is created""" 2 | from dataclasses import dataclass 3 | import json 4 | 5 | import boto3 6 | 7 | from altimeter.core.log import Logger 8 | from altimeter.qj import schemas 9 | from altimeter.qj.log import QJLogEvents 10 | 11 | 12 | @dataclass(frozen=True) 13 | class ResultSetNotifier: 14 | sns_topic_arn: str 15 | region_name: str 16 | 17 | def notify(self, notification: schemas.ResultSetNotification) -> None: 18 | logger = Logger() 19 | with logger.bind(notification=notification): 20 | logger.info(event=QJLogEvents.NotifyNewResultsStart) 21 | session = boto3.Session(region_name=self.region_name) 22 | sns_client = session.client("sns", region_name=self.region_name) 23 | sns_client.publish( 24 | TopicArn=self.sns_topic_arn, 25 | Message=json.dumps({"default": notification.json()}), 26 | MessageStructure="json", 27 | ) 28 | logger.info(event=QJLogEvents.NotifyNewResultsEnd) 29 | -------------------------------------------------------------------------------- /tests/unit/altimeter/qj/schemas/test_job.py: -------------------------------------------------------------------------------- 1 | import os 2 | from unittest import mock 3 | 4 | from altimeter.qj.schemas.job import Category, JobCreate, JobGraphSpec, JobUpdate, Severity 5 | 6 | 7 | @mock.patch.dict(os.environ, {"REGION": "us-west-2"}) 8 | def test_job_update_from_create(): 9 | job_create = JobCreate( 10 | name="test", 11 | description="test", 12 | graph_spec=JobGraphSpec(graph_names=["test"]), 13 | category=Category.gov, 14 | severity=Severity.debug, 15 | query="select ?account_id ?account_name where { ?account a ; ?account_id ; ?account_name } order by ?account_name", 16 | notify_if_results=False, 17 | ) 18 | job_update = JobUpdate.from_job_create(job_create) 19 | expected_job_update = JobUpdate( 20 | description="test", 21 | category=Category.gov, 22 | severity=Severity.debug, 23 | notify_if_results=False, 24 | ) 25 | print(job_update) # False 26 | print(expected_job_update) # None 27 | 28 | assert job_update == expected_job_update 29 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2019 Tableau 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /doc/source/dev/example_ec2_instance_resource_spec_1.py: -------------------------------------------------------------------------------- 1 | class EC2InstanceResourceSpec(EC2ResourceSpec): 2 | type_name = "instance" 3 | schema = Schema( 4 | TransientResourceLinkField("ImageId", EC2ImageResourceSpec), 5 | ScalarField("InstanceType"), 6 | AnonymousDictField("State", ScalarField("Name", "state")), 7 | ScalarField("PublicIpAddress", optional=True), 8 | ResourceLinkField("VpcId", VPCResourceSpec, optional=True), 9 | TagsField(), 10 | ) 11 | 12 | @classmethod 13 | def list_from_aws( 14 | cls: Type[T], client: BaseClient, account_id: str, region: str 15 | ) -> ListFromAWSResult: 16 | paginator = client.get_paginator("describe_instances") 17 | instances = {} 18 | for resp in paginator.paginate(): 19 | for reservation in resp.get("Reservations", []): 20 | for instance in reservation.get("Instances", []): 21 | resource_arn = cls.generate_arn(account_id, region, instance["InstanceId"]) 22 | instances[resource_arn] = instance 23 | return ListFromAWSResult(resources=instances) 24 | -------------------------------------------------------------------------------- /doc/source/dev/sample_scan_resource_1.json: -------------------------------------------------------------------------------- 1 | { 2 | "resources": [ 3 | { 4 | "type": "aws:ec2:instance", 5 | "link_collection": { 6 | "simple_links": [ 7 | { 8 | "pred": "instance_type", 9 | "obj": "t2.2xlarge" 10 | }, 11 | { 12 | "pred": "state", 13 | "obj": "stopped" 14 | } 15 | ], 16 | "multi_links": null, 17 | "tag_links": [ 18 | { 19 | "pred": "Name", 20 | "obj": "my-instance" 21 | } 22 | ], 23 | "resource_links": [ 24 | { 25 | "pred": "vpc", 26 | "obj": "arn:aws:ec2:us-west-2:012345678901:vpc/vpc-01234567890abcdef", 27 | }, 28 | { 29 | "pred": "subnet", 30 | "obj": "arn:aws:ec2:us-west-2:012345678901:subnet/subnet-abcdef01234567890", 31 | } 32 | ], 33 | "transient_resource_links": [ 34 | { 35 | "pred": "image", 36 | "obj": "arn:aws:ec2:us-east-1:012345678901:image/ami-abcd1234", 37 | } 38 | ] 39 | } 40 | } 41 | ] 42 | } -------------------------------------------------------------------------------- /altimeter/core/neptune/exceptions.py: -------------------------------------------------------------------------------- 1 | """Neptune errors""" 2 | from altimeter.core.exceptions import AltimeterException 3 | 4 | 5 | class NeptuneClientException(AltimeterException): 6 | """Base exception class for Neptune client exceptions.""" 7 | 8 | 9 | class NeptuneQueryException(NeptuneClientException): 10 | """A server-side error occurred during a Neptune query execution.""" 11 | 12 | 13 | class NeptuneNoGraphsFoundException(NeptuneClientException): 14 | """No graphs were found in Neptune.""" 15 | 16 | 17 | class NeptuneNoFreshGraphFoundException(NeptuneClientException): 18 | """No acceptably recent graph could be found in Neptune.""" 19 | 20 | 21 | class NeptuneClearGraphException(NeptuneClientException): 22 | """An error occurred while clearing a graph.""" 23 | 24 | 25 | class NeptuneUpdateGraphException(NeptuneClientException): 26 | """An error occurred while updating a graph.""" 27 | 28 | 29 | class NeptuneLoadGraphException(NeptuneClientException): 30 | """An error occurred while loading a graph.""" 31 | 32 | 33 | class InvalidQueryException(NeptuneClientException): 34 | """A statically detected error with a query was found.""" 35 | -------------------------------------------------------------------------------- /altimeter/core/graph/exceptions.py: -------------------------------------------------------------------------------- 1 | """Exceptions related ton Graphing""" 2 | from altimeter.core.exceptions import AltimeterException 3 | 4 | 5 | class LinkParseException(AltimeterException): 6 | """Error parsing Links from JSON.""" 7 | 8 | 9 | class SchemaParseException(AltimeterException): 10 | """Schema.parse error""" 11 | 12 | 13 | class UnmergableDuplicateResourceIdsFoundException(AltimeterException): 14 | """Duplicate unmergable resource ids were found""" 15 | 16 | 17 | class DuplicateResourceIdsFoundException(AltimeterException): 18 | """Duplicate resource ids were found""" 19 | 20 | 21 | class ListFieldSourceKeyNotFoundException(AltimeterException): 22 | """The source_key of a ListField was not found""" 23 | 24 | 25 | class ListFieldValueNotAListException(AltimeterException): 26 | """A ListField does not contain a list""" 27 | 28 | 29 | class GraphSetOrphanedReferencesException(AltimeterException): 30 | """A GraphSet contained orphaned references, for instance a ResourceLink referring to 31 | an id not present in the GraphSet.""" 32 | 33 | 34 | class UnmergableGraphSetsException(AltimeterException): 35 | """GraphSets are unable to be merged.""" 36 | -------------------------------------------------------------------------------- /ci/db_start.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | set -ef -o pipefail 4 | 5 | rm -f postgres_docker_container.id 6 | 7 | echo "Running in local mode" 8 | 9 | cleanup() 10 | { 11 | echo "Cleaning up..." 12 | echo "Killing $docker_container_id" 13 | docker kill $docker_container_id >/dev/null 14 | rm -f postgres_docker_container.id 15 | exit 1 16 | } 17 | 18 | trap cleanup INT 19 | 20 | docker_container_id=$(docker run \ 21 | -d \ 22 | -p 5432:5432 \ 23 | -e POSTGRES_USER=$DB_USER \ 24 | -e POSTGRES_PASSWORD=$DB_PASSWORD \ 25 | -e POSTGRES_DB=$DB_NAME \ 26 | postgres:10.7) 27 | 28 | echo "Started postgres @ $docker_container_id" 29 | echo $docker_container_id > postgres_docker_container.id 30 | 31 | echo "Waiting for postgres local" 32 | set +e 33 | while [ 1 ]; do 34 | timeout 1 bash -c "cat < /dev/null > /dev/tcp/127.0.0.1/5432" 35 | if [[ $? -eq 0 ]]; then 36 | break 37 | fi 38 | sleep 5 39 | done 40 | set -e 41 | sleep 30 42 | echo "postgres local up" 43 | 44 | echo "Creating tables" 45 | export SQLALCHEMY_URL="postgres://${DB_USER}:${DB_PASSWORD}@${DB_HOST}/${DB_NAME}" 46 | alembic -c services/qj/alembic/alembic.ini upgrade head 47 | -------------------------------------------------------------------------------- /doc/source/dev/example_ec2_instance_resource_spec_2.py: -------------------------------------------------------------------------------- 1 | class EC2InstanceResourceSpec(EC2ResourceSpec): 2 | type_name = "instance" 3 | schema = Schema( 4 | TransientResourceLinkField("ImageId", EC2ImageResourceSpec), 5 | ScalarField("InstanceType"), 6 | AnonymousDictField("State", ScalarField("Name", "state")), 7 | ScalarField("PublicIpAddress", optional=True), 8 | ResourceLinkField("VpcId", VPCResourceSpec, optional=True), 9 | ResourceLinkField("SubnetId", SubnetResourceSpec, optional=True), 10 | TagsField(), 11 | ) 12 | 13 | @classmethod 14 | def list_from_aws( 15 | cls: Type[T], client: BaseClient, account_id: str, region: str 16 | ) -> ListFromAWSResult: 17 | paginator = client.get_paginator("describe_instances") 18 | instances = {} 19 | for resp in paginator.paginate(): 20 | for reservation in resp.get("Reservations", []): 21 | for instance in reservation.get("Instances", []): 22 | resource_arn = cls.generate_arn(account_id, region, instance["InstanceId"]) 23 | instances[resource_arn] = instance 24 | return ListFromAWSResult(resources=instances) 25 | -------------------------------------------------------------------------------- /altimeter/aws/scan/scan_manifest.py: -------------------------------------------------------------------------------- 1 | """A ScanManifest defines the output of a complete scan.""" 2 | 3 | from typing import Dict, List, Optional 4 | 5 | from altimeter.core.base_model import BaseImmutableModel 6 | 7 | 8 | class ScanManifest(BaseImmutableModel): 9 | """A ScanManifest defines the output of a complete scan. It contains pointers to the 10 | per-account scan result artifacts and summaries of what was scanned, errors which occurred, 11 | scan datetime and api call statistics. 12 | 13 | Args: 14 | scanned_accounts: List of account ids which were scanned 15 | master_artifact: artifact containing complete graph json 16 | artifacts: list of artifacts, one per account 17 | errors: Dict of account_ids to list of errors encountered during scan 18 | unscanned_accounts: List of account ids which were not scanned 19 | start_time: epoch timestamp of scan start time 20 | end_time: epoch timestamp of scan end time 21 | """ 22 | 23 | scanned_accounts: List[str] 24 | master_artifact: Optional[str] = None 25 | artifacts: List[str] 26 | errors: Dict[str, List[str]] 27 | unscanned_accounts: List[str] 28 | start_time: int 29 | end_time: int 30 | -------------------------------------------------------------------------------- /conf/current_single_account.toml: -------------------------------------------------------------------------------- 1 | # This configuration will cause altimeter to scan the currently logged in account. 2 | 3 | artifact_path = "/tmp/altimeter_single_account" 4 | graph_name = "alti" 5 | pruner_max_age_min = 4320 # prune graphs over 3 days old 6 | 7 | [accessor] 8 | cache_creds = true 9 | 10 | [scan] 11 | # accounts to scan 12 | accounts = [] 13 | # regions to scan. If empty, scan all available regions 14 | regions = [] 15 | # if true, discover and scan subaccounts of the above accounts 16 | scan_sub_accounts = false 17 | # preferred regions to use when scanning non-regional resources (e.g. IAM policies) 18 | preferred_account_scan_regions = [ 19 | "us-west-1", 20 | "us-west-2", 21 | "us-east-1", 22 | "us-east-2", 23 | ] 24 | 25 | [concurrency] 26 | # The following settings control scan concurrency. 27 | # 28 | # In general, the maximum number of concurrent scan operations is 29 | # max_account_scan_threads * max_svc_scan_threads 30 | max_account_scan_threads = 1 # number of account scan threads to spawn 31 | max_svc_scan_threads = 64 # the number of scan threads to spawn in each account scan thread 32 | -------------------------------------------------------------------------------- /altimeter/core/graph/schema.py: -------------------------------------------------------------------------------- 1 | """A Schema consists of a list of Fields which define how to parse an arbitrary dictionary 2 | into a list of Links.""" 3 | from typing import Any, Dict, Tuple, Type 4 | 5 | from altimeter.core.graph.field.base import Field 6 | from altimeter.core.graph.links import LinkCollection 7 | 8 | 9 | class Schema: 10 | """A Schema consists of a list of Fields which define how to parse an arbitrary dictionary 11 | into a :class:`altimeter.core.graph.links.LinkCollection`. 12 | 13 | Args: 14 | fields: fields for this Schema. 15 | """ 16 | 17 | def __init__(self, *fields: Field) -> None: 18 | self.fields = fields 19 | 20 | def parse( 21 | self, 22 | data: Dict[str, Any], 23 | context: Dict[str, Any], 24 | ) -> LinkCollection: 25 | """Parse this schema into a list of Links 26 | 27 | Args: 28 | data: raw data to parse 29 | context: contains auxiliary information which can be passed through the parse process. 30 | 31 | Returns: 32 | LinkCollection 33 | """ 34 | link_collection = LinkCollection() 35 | for field in self.fields: 36 | link_collection += field.parse(data, context) 37 | return link_collection 38 | -------------------------------------------------------------------------------- /altimeter/qj/exceptions.py: -------------------------------------------------------------------------------- 1 | """QJ Exception Classes""" 2 | 3 | 4 | class QJException(Exception): 5 | """Base exception class for all QJ thrown exceptions""" 6 | 7 | 8 | class JobInvalid(QJException): 9 | """A specified Job is invalid""" 10 | 11 | 12 | class JobNotFound(QJException): 13 | """A specified Job could not be found""" 14 | 15 | 16 | class JobVersionNotFound(QJException): 17 | """A specified JobVersion could not be found""" 18 | 19 | 20 | class ActiveJobVersionNotFound(QJException): 21 | """An active JobVersion for a specified Job could not be found""" 22 | 23 | 24 | class JobQueryMissingAccountId(QJException): 25 | """A Job's query is missing the required account_id field""" 26 | 27 | 28 | class JobQueryInvalid(QJException): 29 | """A Job's query is invalid SPARQL""" 30 | 31 | 32 | class ResultSetNotFound(QJException): 33 | """A specified ResultSet could not be found""" 34 | 35 | 36 | class ResultSetResultsLimitExceeded(QJException): 37 | """The number of Results in a ResultSet exceeds the configured maximum""" 38 | 39 | 40 | class ResultSizeExceeded(QJException): 41 | """The size of an individual result exceeds the configured maximum""" 42 | 43 | 44 | class RemediationError(Exception): 45 | """An error during Remediation""" 46 | -------------------------------------------------------------------------------- /doc/source/dev/devguide.rst: -------------------------------------------------------------------------------- 1 | Developer's Guide 2 | ================= 3 | 4 | This document contains information on setting up a development environment. 5 | 6 | Requirements 7 | ------------ 8 | 9 | Altimeter requires Python 3.8 or greater. 10 | 11 | To install project requirements, from the base repo dir: 12 | 13 | :: 14 | 15 | find . -name requirements.txt -exec pip install -r {} \; 16 | 17 | Pre-Commit Check 18 | ---------------- 19 | 20 | A pre-commit script is included (`git/pre-commit.sh`) which performs static analysis 21 | using mypy_ and pylint_, code autoformat checking using black_ and runs tests all via 22 | tox. 23 | 24 | This script is run as a part of Altimeter's CI and must pass for contributions 25 | to be merged. 26 | 27 | To configure this as a pre-commit hook, from the base repository directory: 28 | 29 | :: 30 | 31 | ln -s ../../git/pre-commit.sh .git/hooks/pre-commit 32 | 33 | This can be run by hand by running tox: 34 | 35 | :: 36 | 37 | tox 38 | 39 | *Next Steps* 40 | 41 | See :doc:`Extending Altimeter ` for a guide to extending Altimeter's 42 | capabilities to collect and graph more data. 43 | 44 | .. _black: https://github.com/psf/black 45 | .. _mypy: https://github.com/python/mypy 46 | .. _pylint: https://github.com/PyCQA/pylint 47 | -------------------------------------------------------------------------------- /altimeter/aws/resource/kms/key.py: -------------------------------------------------------------------------------- 1 | """Resource for KMSKeys""" 2 | from typing import Type 3 | 4 | from botocore.client import BaseClient 5 | 6 | from altimeter.aws.resource.resource_spec import ListFromAWSResult 7 | from altimeter.aws.resource.kms import KMSResourceSpec 8 | from altimeter.core.graph.field.scalar_field import ScalarField 9 | from altimeter.core.graph.schema import Schema 10 | 11 | 12 | class KMSKeyResourceSpec(KMSResourceSpec): 13 | """Resource for KMS Keys""" 14 | 15 | type_name = "key" 16 | schema = Schema(ScalarField("KeyId")) 17 | 18 | @classmethod 19 | def list_from_aws( 20 | cls: Type["KMSKeyResourceSpec"], client: BaseClient, account_id: str, region: str 21 | ) -> ListFromAWSResult: 22 | """Return a dict of dicts of the format: 23 | 24 | {'key_1_arn': {key_1_dict}, 25 | 'key_2_arn': {key_2_dict}, 26 | ...} 27 | 28 | Where the dicts represent results from list_keys.""" 29 | keys = {} 30 | paginator = client.get_paginator("list_keys") 31 | for resp in paginator.paginate(): 32 | for key in resp.get("Keys", []): 33 | resource_arn = key["KeyArn"] 34 | keys[resource_arn] = key 35 | return ListFromAWSResult(resources=keys) 36 | -------------------------------------------------------------------------------- /services/qj/main.py: -------------------------------------------------------------------------------- 1 | """Entrypoint""" 2 | from fastapi import FastAPI, Request 3 | from fastapi.responses import JSONResponse 4 | from starlette.exceptions import HTTPException 5 | import uvicorn 6 | 7 | from altimeter.core.log import Logger 8 | from altimeter.qj.api.base.api import BASE_ROUTER 9 | from altimeter.qj.api.v1.api import V1_ROUTER 10 | from altimeter.qj.config import APIServiceConfig 11 | from altimeter.qj.log import QJLogEvents 12 | from altimeter.qj.middleware import HTTPRequestLoggingMiddleware 13 | 14 | API_SVC_CONFIG = APIServiceConfig() 15 | 16 | LOGGER = Logger() 17 | 18 | app = FastAPI(title=API_SVC_CONFIG.app_name,) 19 | 20 | app.include_router(BASE_ROUTER) 21 | app.include_router(V1_ROUTER, prefix="/v1") 22 | app.add_middleware(HTTPRequestLoggingMiddleware) 23 | 24 | 25 | @app.exception_handler(HTTPException) 26 | async def http_exception_handler(request: Request, ex: HTTPException): 27 | LOGGER.info( 28 | event=QJLogEvents.APIError, 29 | detail=ex.detail, 30 | url=str(request.url), 31 | requestor=request.client.host, 32 | ) 33 | return JSONResponse(status_code=ex.status_code, content={"detail": ex.detail}) 34 | 35 | 36 | if __name__ == "__main__": 37 | uvicorn.run(app, host=API_SVC_CONFIG.api_host, port=API_SVC_CONFIG.api_port) 38 | -------------------------------------------------------------------------------- /bin/queryjob_lambda.py: -------------------------------------------------------------------------------- 1 | """Execute all known QJs, run the query portion of a QJ, remediate a QJ and prune results according to Job config 2 | settings""" 3 | import os.path 4 | 5 | from typing import Any, Dict 6 | 7 | from altimeter.qj.config import QJHandlerConfig 8 | from altimeter.qj.lambdas.executor import executor 9 | from altimeter.qj.lambdas.pruner import pruner 10 | from altimeter.qj.lambdas.publish import publish 11 | from altimeter.qj.lambdas.query import query 12 | from altimeter.qj.lambdas.remediator import remediator 13 | 14 | 15 | class InvalidLambdaModeException(Exception): 16 | """Indicates the mode associated with the queryjob lambda is invalid""" 17 | 18 | 19 | def lambda_handler(event: Dict[str, Any], _: Any) -> Any: 20 | """Lambda entrypoint""" 21 | handler = QJHandlerConfig() 22 | if handler.mode == "executor": 23 | return executor(event) 24 | elif handler.mode == "query": 25 | return query(event) 26 | elif handler.mode == "pruner": 27 | return pruner() 28 | elif handler.mode == "remediator": 29 | return remediator(event) 30 | elif handler.mode == "publish": 31 | return publish(event) 32 | else: 33 | raise InvalidLambdaModeException( 34 | f"Invalid lambda MODE value.\nENV: {os.environ}\nEvent: {event}" 35 | ) 36 | -------------------------------------------------------------------------------- /tests/unit/altimeter/aws/resource/test_unscanned_account.py: -------------------------------------------------------------------------------- 1 | from unittest import TestCase 2 | 3 | from altimeter.aws.resource.unscanned_account import UnscannedAccountResourceSpec 4 | from altimeter.core.graph.links import LinkCollection, SimpleLink 5 | from altimeter.core.resource.resource import Resource 6 | from altimeter.core.resource.resource_spec import ResourceSpec 7 | 8 | 9 | class TestUnscannedAccountMultipleErrors(TestCase): 10 | def test(self): 11 | account_id = "012345678901" 12 | errors = ["foo", "boo"] 13 | unscanned_account_resource = UnscannedAccountResourceSpec.create_resource( 14 | account_id=account_id, errors=errors 15 | ) 16 | resource = ResourceSpec.merge_resources("foo", [unscanned_account_resource]) 17 | 18 | self.assertEqual(resource.resource_id, "foo") 19 | self.assertEqual(resource.type, "aws:unscanned-account") 20 | self.assertEqual(len(resource.link_collection.simple_links), 2) 21 | self.assertEqual( 22 | resource.link_collection.simple_links[0], 23 | SimpleLink(pred="account_id", obj="012345678901"), 24 | ) 25 | self.assertEqual(resource.link_collection.simple_links[1].pred, "error") 26 | self.assertTrue(resource.link_collection.simple_links[1].obj.startswith("foo\nboo - ")) 27 | -------------------------------------------------------------------------------- /altimeter/aws/resource/events/event_bus.py: -------------------------------------------------------------------------------- 1 | """Resource for CloudWatchEvents EventBusses""" 2 | from typing import Type 3 | 4 | from botocore.client import BaseClient 5 | 6 | from altimeter.aws.resource.resource_spec import ListFromAWSResult 7 | from altimeter.aws.resource.events import EventsResourceSpec 8 | from altimeter.core.graph.field.scalar_field import ScalarField 9 | from altimeter.core.graph.schema import Schema 10 | 11 | 12 | class EventBusResourceSpec(EventsResourceSpec): 13 | """Resource for CloudWatchEvents EventBus""" 14 | 15 | type_name = "event-bus" 16 | schema = Schema( 17 | ScalarField("Name"), 18 | ScalarField("Arn"), 19 | ScalarField("Policy", optional=True), 20 | ) 21 | 22 | @classmethod 23 | def list_from_aws( 24 | cls: Type["EventBusResourceSpec"], client: BaseClient, account_id: str, region: str 25 | ) -> ListFromAWSResult: 26 | """Return a dict of dicts of the format: 27 | 28 | {'event_bus_1_arn': {event_bus_1_dict}, 29 | 'event_bus_2_arn': {event_bus_2_dict}, 30 | ...} 31 | 32 | Where the dicts represent results from describe_event_bus.""" 33 | resp = client.describe_event_bus() 34 | arn = resp["Arn"] 35 | event_busses = {arn: resp} 36 | return ListFromAWSResult(resources=event_busses) 37 | -------------------------------------------------------------------------------- /conf/current_single_account_skip_support.toml: -------------------------------------------------------------------------------- 1 | # This configuration will cause altimeter to scan the currently logged in account. 2 | 3 | artifact_path = "/tmp/altimeter_single_account" 4 | graph_name = "alti" 5 | pruner_max_age_min = 4320 # prune graphs over 3 days old 6 | 7 | [accessor] 8 | cache_creds = true 9 | 10 | [scan] 11 | # accounts to scan 12 | accounts = [] 13 | # regions to scan. If empty, scan all available regions 14 | regions = [] 15 | # if true, discover and scan subaccounts of the above accounts 16 | scan_sub_accounts = false 17 | # preferred regions to use when scanning non-regional resources (e.g. IAM policies) 18 | preferred_account_scan_regions = [ 19 | "us-west-1", 20 | "us-west-2", 21 | "us-east-1", 22 | "us-east-2", 23 | ] 24 | # ignore iam policies 25 | ignored_resources = [ 26 | "aws:support:severity-level", 27 | ] 28 | 29 | [concurrency] 30 | # The following settings control scan concurrency. 31 | # 32 | # In general, the maximum number of concurrent scan operations is 33 | # max_account_scan_threads * max_svc_scan_threads 34 | max_account_scan_threads = 1 # number of account scan threads to spawn 35 | max_svc_scan_threads = 64 # the number of scan threads to spawn in each account scan thread 36 | -------------------------------------------------------------------------------- /altimeter/aws/log_events.py: -------------------------------------------------------------------------------- 1 | """LogEvent for AWS related events.""" 2 | from dataclasses import dataclass 3 | 4 | from altimeter.core.log import BaseLogEvent, EventName 5 | 6 | 7 | @dataclass(frozen=True) 8 | class AWSLogEvents(BaseLogEvent): 9 | """AWS specific Log event names""" 10 | 11 | AuthToAccountStart: EventName 12 | AuthToAccountEnd: EventName 13 | AuthToAccountFailure: EventName 14 | 15 | GetSubAccountsStart: EventName 16 | GetSubAccountsEnd: EventName 17 | 18 | GetServiceResourceRegionMappingStart: EventName 19 | GetServiceResourceRegionMappingEnd: EventName 20 | GetServiceResourceRegionMappingWarning: EventName 21 | GetServiceResourceRegionMappingDiscrepancy: EventName 22 | 23 | RunAccountScanLambdaStart: EventName 24 | RunAccountScanLambdaEnd: EventName 25 | RunAccountScanLambdaError: EventName 26 | 27 | MuxerQueueScan: EventName 28 | MuxerStart: EventName 29 | MuxerEnd: EventName 30 | MuxerStat: EventName 31 | 32 | ScanAWSAccountsStart: EventName 33 | ScanAWSAccountsEnd: EventName 34 | 35 | ScanAWSAccountStart: EventName 36 | ScanAWSAccountEnd: EventName 37 | ScanAWSAccountError: EventName 38 | 39 | ScanAWSAccountServiceStart: EventName 40 | ScanAWSAccountServiceEnd: EventName 41 | 42 | ScanAWSResourcesNonFatalError: EventName 43 | 44 | ScanConfigured: EventName 45 | -------------------------------------------------------------------------------- /.github/workflows/ci.yml: -------------------------------------------------------------------------------- 1 | name: CI 2 | 3 | on: [push, pull_request] 4 | 5 | jobs: 6 | static_analysis_and_tests: 7 | runs-on: ubuntu-latest 8 | strategy: 9 | matrix: 10 | python-version: [3.8] 11 | steps: 12 | - uses: actions/checkout@v1 13 | - name: Set up Python ${{ matrix.python-version }} 14 | uses: actions/setup-python@v1 15 | with: 16 | python-version: ${{ matrix.python-version }} 17 | architecture: x64 18 | - name: Install dependencies ${{ matrix.python-version }} 19 | run: | 20 | python -m pip install --upgrade pip 21 | pip install tox==4.11.3 22 | - name: Run tox 23 | run: tox 24 | build_wheel: 25 | needs: static_analysis_and_tests 26 | runs-on: ubuntu-latest 27 | strategy: 28 | matrix: 29 | python-version: [3.8] 30 | steps: 31 | - uses: actions/checkout@v1 32 | - name: Set up Python ${{ matrix.python-version }} 33 | uses: actions/setup-python@v1 34 | with: 35 | python-version: ${{ matrix.python-version }} 36 | architecture: x64 37 | - name: Install pypa/build 38 | run: >- 39 | python -m 40 | pip install 41 | build 42 | --user 43 | - name: Build a binary wheel and a source tarball 44 | run: >- 45 | python -m 46 | build 47 | --sdist 48 | --wheel 49 | --outdir dist/ 50 | . 51 | -------------------------------------------------------------------------------- /altimeter/aws/resource/organizations/org.py: -------------------------------------------------------------------------------- 1 | """Resource representing an AWS Organization.""" 2 | from typing import Type 3 | 4 | from botocore.client import BaseClient 5 | 6 | from altimeter.core.graph.field.scalar_field import ScalarField 7 | from altimeter.core.graph.schema import Schema 8 | from altimeter.aws.resource.resource_spec import ListFromAWSResult 9 | from altimeter.aws.resource.organizations import OrganizationsResourceSpec 10 | 11 | 12 | class OrgResourceSpec(OrganizationsResourceSpec): 13 | """Resource representing an AWS Org.""" 14 | 15 | type_name = "organization" 16 | schema = Schema(ScalarField("MasterAccountId"), ScalarField("MasterAccountEmail")) 17 | 18 | @classmethod 19 | def get_full_type_name(cls: Type["OrgResourceSpec"]) -> str: 20 | return f"{cls.provider_name}:{cls.type_name}" 21 | 22 | @classmethod 23 | def list_from_aws( 24 | cls: Type["OrgResourceSpec"], client: BaseClient, account_id: str, region: str 25 | ) -> ListFromAWSResult: 26 | """Return a dict of dicts of the format: 27 | 28 | {'org_1_arn': {org_1_dict}, 29 | 'org_2_arn': {org_2_dict}, 30 | ...} 31 | 32 | Where the dicts represent results from describe_organization.""" 33 | resp = client.describe_organization() 34 | org = resp["Organization"] 35 | orgs = {org["Arn"]: org} 36 | return ListFromAWSResult(resources=orgs) 37 | -------------------------------------------------------------------------------- /altimeter/core/artifact_io/__init__.py: -------------------------------------------------------------------------------- 1 | from typing import Optional, Tuple 2 | 3 | from altimeter.core.artifact_io.exceptions import InvalidS3URIException 4 | 5 | S3_URI_PREFIX = "s3://" 6 | 7 | 8 | def is_s3_uri(path: str) -> bool: 9 | if path.startswith(S3_URI_PREFIX): 10 | return True 11 | return False 12 | 13 | 14 | def parse_s3_uri(uri: str) -> Tuple[str, Optional[str]]: 15 | """Parse an s3 uri (s3://bucket/key/path) into bucket and key parts 16 | 17 | Args: 18 | uri: s3 uri (s3://bucket/key/path) 19 | 20 | Returns: 21 | Tuple of (bucket, key) 22 | 23 | Raises: 24 | :class:`InvalidS3URIException` if the uri 25 | argument is not a valid S3 URI. 26 | """ 27 | if not is_s3_uri(uri): 28 | raise InvalidS3URIException(f"S3 URIs should begin with '{S3_URI_PREFIX}'") 29 | uri = uri.rstrip("/") 30 | parts = uri[len(S3_URI_PREFIX) :].split("/") 31 | if len(parts) < 1: 32 | raise InvalidS3URIException(f"{uri} missing bucket portion") 33 | bucket = parts[0] 34 | if not bucket: 35 | raise InvalidS3URIException(f"Bad bucket portion in uri {uri}") 36 | key = None 37 | if len(parts) > 1: 38 | key_parts = [part.rstrip("/ ") for part in parts[1:]] 39 | if not all(key_parts): 40 | raise InvalidS3URIException(f"Bad key portion in uri {uri}") 41 | key = "/".join(key_parts) 42 | return bucket, key 43 | -------------------------------------------------------------------------------- /altimeter/core/graph/field/exceptions.py: -------------------------------------------------------------------------------- 1 | """Field parsing related exceptions.""" 2 | from altimeter.core.exceptions import AltimeterException 3 | 4 | 5 | class ScalarFieldSourceKeyNotFoundException(AltimeterException): 6 | """The source_key of a ScalarField is not found""" 7 | 8 | 9 | class ScalarFieldValueNotAScalarException(AltimeterException): 10 | """The value of a ScalarField is not a string, bool, int or float.""" 11 | 12 | 13 | class ResourceLinkFieldSourceKeyNotFoundException(AltimeterException): 14 | """The source_key of a ResourceLinkField is not found""" 15 | 16 | 17 | class ResourceLinkFieldValueNotAStringException(AltimeterException): 18 | """The value of a ResourceLinkField is not a string.""" 19 | 20 | 21 | class TagsFieldMissingTagsKeyException(AltimeterException): 22 | """A TagsField data is missing key 'Tags'""" 23 | 24 | 25 | class ParentKeyMissingException(AltimeterException): 26 | """A required ParentKey is missing.""" 27 | 28 | 29 | class InvalidParentKeyException(AltimeterException): 30 | """A ParentKey is invalid.""" 31 | 32 | 33 | class DictFieldValueNotADictException(AltimeterException): 34 | """A DictField does not contain a dict.""" 35 | 36 | 37 | class DictFieldSourceKeyNotFoundException(AltimeterException): 38 | """The source_key of a DictField was not found""" 39 | 40 | 41 | class DictFieldValueIsNullException(AltimeterException): 42 | """The value of a non-nullable DictField was null""" 43 | -------------------------------------------------------------------------------- /conf/current_master_multi_account.toml: -------------------------------------------------------------------------------- 1 | # This configuration will cause altimeter to scan the currently logged in account 2 | # and any of its subaccounts, using the OrganizationAccountAccessRole 3 | 4 | artifact_path = "/tmp/altimeter_multi_account" 5 | graph_name = "alti" 6 | pruner_max_age_min = 4320 # prune graphs over 3 days old 7 | 8 | [accessor] 9 | cache_creds = true 10 | [[accessor.multi_hop_accessors]] 11 | role_session_name = "altimeter" 12 | [[accessor.multi_hop_accessors.access_steps]] 13 | role_name = "OrganizationAccountAccessRole" 14 | 15 | [scan] 16 | # accounts to scan 17 | accounts = [] 18 | # regions to scan. If empty, scan all available regions 19 | regions = [] 20 | # if true, discover and scan subaccounts of the above accounts 21 | scan_sub_accounts = true 22 | # preferred regions to use when scanning non-regional resources (e.g. IAM policies) 23 | preferred_account_scan_regions = [ 24 | "us-west-1", 25 | "us-west-2", 26 | "us-east-1", 27 | "us-east-2", 28 | ] 29 | 30 | [concurrency] 31 | # The following settings control scan concurrency. 32 | # 33 | # In general, the maximum number of concurrent scan operations is 34 | # max_account_scan_threads * max_svc_scan_threads 35 | max_account_scan_threads = 16 # number of account scan threads to spawn 36 | max_svc_scan_threads = 32 # the number of scan threads to spawn in each account scan thread 37 | -------------------------------------------------------------------------------- /altimeter/aws/resource/support/severity_level.py: -------------------------------------------------------------------------------- 1 | """Resource representing AWS Support.""" 2 | from typing import Type 3 | 4 | from botocore.client import BaseClient 5 | 6 | from altimeter.core.graph.field.scalar_field import ScalarField 7 | from altimeter.core.graph.schema import Schema 8 | from altimeter.aws.resource.resource_spec import ListFromAWSResult 9 | from altimeter.aws.resource.support import SupportResourceSpec 10 | 11 | 12 | class SeverityLevelResourceSpec(SupportResourceSpec): 13 | """Resource representing an AWS Support severity level.""" 14 | 15 | type_name = "severity-level" 16 | schema = Schema(ScalarField("code")) 17 | 18 | @classmethod 19 | def list_from_aws( 20 | cls: Type["SeverityLevelResourceSpec"], client: BaseClient, account_id: str, region: str 21 | ) -> ListFromAWSResult: 22 | """Return a dict of dicts of the format: 23 | 24 | {'severity_level_arn': {severity_level_dict}, 25 | 'severity_level_arn': {severity_level_dict}, 26 | ...} 27 | 28 | Where the dicts represent results from describe_organization.""" 29 | resp = client.describe_severity_levels() 30 | severity_levels_resp = resp["severityLevels"] 31 | severity_levels = {} 32 | for s_l in severity_levels_resp: 33 | code = s_l["code"] 34 | code_arn = cls.generate_arn(resource_id=code, account_id=account_id) 35 | severity_levels[code_arn] = {"code": code} 36 | return ListFromAWSResult(resources=severity_levels) 37 | -------------------------------------------------------------------------------- /altimeter/aws/resource/ec2/vpc.py: -------------------------------------------------------------------------------- 1 | """Resource for VPCs""" 2 | from typing import Type 3 | 4 | from botocore.client import BaseClient 5 | 6 | from altimeter.aws.resource.resource_spec import ListFromAWSResult 7 | from altimeter.aws.resource.ec2 import EC2ResourceSpec 8 | from altimeter.core.graph.field.scalar_field import ScalarField 9 | from altimeter.core.graph.field.tags_field import TagsField 10 | from altimeter.core.graph.schema import Schema 11 | 12 | 13 | class VPCResourceSpec(EC2ResourceSpec): 14 | """Resource for VPCs""" 15 | 16 | type_name = "vpc" 17 | schema = Schema( 18 | ScalarField("IsDefault"), ScalarField("CidrBlock"), ScalarField("State"), TagsField() 19 | ) 20 | 21 | @classmethod 22 | def list_from_aws( 23 | cls: Type["VPCResourceSpec"], client: BaseClient, account_id: str, region: str 24 | ) -> ListFromAWSResult: 25 | """Return a dict of dicts of the format: 26 | 27 | {'vpc_1_arn': {vpc_1_dict}, 28 | 'vpc_2_arn': {vpc_2_dict}, 29 | ...} 30 | 31 | Where the dicts represent results from describe_vpcs.""" 32 | vpcs = {} 33 | paginator = client.get_paginator("describe_vpcs") 34 | for resp in paginator.paginate(): 35 | for vpc in resp.get("Vpcs", []): 36 | resource_arn = cls.generate_arn( 37 | account_id=account_id, region=region, resource_id=vpc["VpcId"] 38 | ) 39 | vpcs[resource_arn] = vpc 40 | return ListFromAWSResult(resources=vpcs) 41 | -------------------------------------------------------------------------------- /altimeter/aws/resource/cloudtrail/trail.py: -------------------------------------------------------------------------------- 1 | """Resource for AWS CloudTrail Trails""" 2 | from typing import Type 3 | 4 | from botocore.client import BaseClient 5 | 6 | from altimeter.aws.resource.resource_spec import ListFromAWSResult 7 | from altimeter.aws.resource.cloudtrail import CloudTrailResourceSpec 8 | from altimeter.core.graph.field.scalar_field import ScalarField 9 | from altimeter.core.graph.schema import Schema 10 | 11 | 12 | class CloudTrailTrailResourceSpec(CloudTrailResourceSpec): 13 | """Resource representing a CloudTrail Trail""" 14 | 15 | type_name = "trail" 16 | schema = Schema( 17 | ScalarField("Name"), 18 | ScalarField("S3BucketName"), 19 | ScalarField("IncludeGlobalServiceEvents"), 20 | ScalarField("IsMultiRegionTrail"), 21 | ) 22 | 23 | @classmethod 24 | def list_from_aws( 25 | cls: Type["CloudTrailTrailResourceSpec"], 26 | client: BaseClient, 27 | account_id: str, 28 | region: str, 29 | ) -> ListFromAWSResult: 30 | """Return a dict of dicts of the format: 31 | 32 | {'trail_1_arn': {trail_1_dict}, 33 | 'trail_2_arn': {trail_2_dict}, 34 | ...} 35 | 36 | Where the dicts represent results from describe_trails.""" 37 | trails = {} 38 | resp = client.describe_trails(includeShadowTrails=False) 39 | for trail in resp.get("trailList", []): 40 | resource_arn = trail["TrailARN"] 41 | trails[resource_arn] = trail 42 | return ListFromAWSResult(resources=trails) 43 | -------------------------------------------------------------------------------- /altimeter/aws/resource/organizations/__init__.py: -------------------------------------------------------------------------------- 1 | """Base class for AWS organizations resources.""" 2 | from typing import Type, List, Dict, Any 3 | 4 | from botocore.client import BaseClient 5 | 6 | from altimeter.aws.resource.resource_spec import ScanGranularity, AWSResourceSpec 7 | 8 | 9 | class OrganizationsResourceSpec(AWSResourceSpec): 10 | """Base class for AWS organizations resources.""" 11 | 12 | service_name = "organizations" 13 | scan_granularity = ScanGranularity.ACCOUNT 14 | 15 | @classmethod 16 | def skip_resource_scan( 17 | cls: Type["OrganizationsResourceSpec"], client: BaseClient, account_id: str, region: str 18 | ) -> bool: 19 | """Return a bool indicating whether this resource class scan should be skipped, 20 | in this case skip if the current account is not an org master.""" 21 | resp = client.describe_organization() 22 | return resp["Organization"]["MasterAccountId"] != account_id 23 | 24 | 25 | def recursively_get_ou_details_for_parent( 26 | client: BaseClient, parent_id: str, parent_path: str 27 | ) -> List[Dict[str, Any]]: 28 | ous = [] 29 | paginator = client.get_paginator("list_organizational_units_for_parent") 30 | for resp in paginator.paginate(ParentId=parent_id): 31 | for ou in resp["OrganizationalUnits"]: 32 | ou_id = ou["Id"] 33 | path = f"{parent_path}/{ou['Name']}" 34 | ou["Path"] = path 35 | ous.append(ou) 36 | ous += recursively_get_ou_details_for_parent( 37 | client=client, parent_id=ou_id, parent_path=path 38 | ) 39 | return ous 40 | -------------------------------------------------------------------------------- /altimeter/aws/resource/ec2/image.py: -------------------------------------------------------------------------------- 1 | """Resource for EC2Images (AMIs)""" 2 | from typing import Type 3 | 4 | from botocore.client import BaseClient 5 | 6 | from altimeter.aws.resource.resource_spec import ListFromAWSResult 7 | from altimeter.aws.resource.ec2 import EC2ResourceSpec 8 | from altimeter.core.graph.field.scalar_field import ScalarField 9 | from altimeter.core.graph.field.tags_field import TagsField 10 | from altimeter.core.graph.schema import Schema 11 | 12 | 13 | class EC2ImageResourceSpec(EC2ResourceSpec): 14 | """Resource for EC2Images (AMIs)""" 15 | 16 | type_name = "image" 17 | schema = Schema( 18 | ScalarField("Name", optional=True), 19 | ScalarField("Description", optional=True), 20 | ScalarField("Public"), 21 | TagsField(), 22 | ) 23 | 24 | @classmethod 25 | def list_from_aws( 26 | cls: Type["EC2ImageResourceSpec"], client: BaseClient, account_id: str, region: str 27 | ) -> ListFromAWSResult: 28 | """Return a dict of dicts of the format: 29 | 30 | {'image_1_arn': {image_1_dict}, 31 | 'image_2_arn': {image_2_dict}, 32 | ...} 33 | 34 | Where the dicts represent results from describe_images.""" 35 | images = {} 36 | resp = client.describe_images(Owners=["self"]) 37 | for image in resp["Images"]: 38 | image_id = image["ImageId"] 39 | resource_arn = cls.generate_arn( 40 | account_id=account_id, region=region, resource_id=image_id 41 | ) 42 | images[resource_arn] = image 43 | return ListFromAWSResult(resources=images) 44 | -------------------------------------------------------------------------------- /altimeter/aws/resource/ec2/region.py: -------------------------------------------------------------------------------- 1 | """Resource representing an AWS Region""" 2 | from typing import Type 3 | 4 | from botocore.client import BaseClient 5 | 6 | from altimeter.aws.resource.resource_spec import ScanGranularity, ListFromAWSResult 7 | from altimeter.aws.resource.ec2 import EC2ResourceSpec 8 | from altimeter.core.graph.field.scalar_field import ScalarField 9 | from altimeter.core.graph.schema import Schema 10 | 11 | 12 | class RegionResourceSpec(EC2ResourceSpec): 13 | """Resource representing an AWS Region""" 14 | 15 | type_name = "region" 16 | scan_granularity = ScanGranularity.ACCOUNT 17 | schema = Schema(ScalarField("RegionName", "name"), ScalarField("OptInStatus")) 18 | 19 | @classmethod 20 | def get_full_type_name(cls: Type["RegionResourceSpec"]) -> str: 21 | return f"{cls.provider_name}:{cls.type_name}" 22 | 23 | @classmethod 24 | def list_from_aws( 25 | cls: Type["RegionResourceSpec"], client: BaseClient, account_id: str, region: str 26 | ) -> ListFromAWSResult: 27 | """Return a dict of dicts of the format: 28 | 29 | {'region_1_arn': {region_1_dict}, 30 | 'region_2_arn': {region_2_dict}, 31 | ...} 32 | 33 | Where the dicts represent results from describe_regions.""" 34 | regions = {} 35 | resp = client.describe_regions(AllRegions=True) 36 | for region_resp in resp["Regions"]: 37 | region_name = region_resp["RegionName"] 38 | region_arn = f"arn:aws:::{account_id}:region/{region_name}" 39 | regions[region_arn] = region_resp 40 | return ListFromAWSResult(resources=regions) 41 | -------------------------------------------------------------------------------- /tox.ini: -------------------------------------------------------------------------------- 1 | [tox] 2 | envlist = formatting,typing,lint,test,docs 3 | basepython = python3.8 4 | 5 | [testenv] 6 | allowlist_externals = bash 7 | 8 | [testenv:format] 9 | deps = 10 | black==23.11.0 11 | 12 | commands = 13 | pip check 14 | black -l 100 altimeter bin tests 15 | 16 | [testenv:formatting] 17 | deps = 18 | black==23.11.0 19 | 20 | commands = 21 | pip check 22 | black -l 100 --check altimeter bin tests 23 | 24 | [testenv:typing] 25 | deps = 26 | mypy==1.6.1 27 | types-requests==2.31.0 28 | types-toml==0.10.2 29 | 30 | commands = 31 | pip check 32 | mypy --incremental --ignore-missing-imports --disallow-untyped-defs altimeter bin 33 | 34 | [testenv:lint] 35 | deps = 36 | -r services/qj/requirements.txt 37 | pylint==3.0.2 38 | 39 | commands = 40 | pip check 41 | pylint -j 0 --fail-under=9 altimeter bin 42 | 43 | [testenv:test] 44 | deps = 45 | -r services/qj/requirements.txt 46 | -r tests/requirements.txt 47 | 48 | setenv = 49 | DB_USER=postgres 50 | DB_PASSWORD= 51 | DB_NAME=qj 52 | DB_HOST=127.0.0.1 53 | 54 | commands_pre = 55 | bash ci/db_start.sh 56 | 57 | commands = 58 | pip check 59 | pytest --ignore=altimeter/qj/alembic/env.py --cov="altimeter" --cov-report=term-missing --cov-fail-under=60 --cov-branch --doctest-modules "altimeter" "tests" 60 | 61 | commands_post = 62 | bash ci/db_stop.sh 63 | 64 | [testenv:docs] 65 | deps = 66 | -r doc/requirements.txt 67 | -r services/qj/requirements.txt 68 | 69 | commands = 70 | pip check 71 | sphinx-apidoc -f -o doc/source altimeter altimeter/qj/alembic 72 | sphinx-build doc/source doc/html -E -W 73 | -------------------------------------------------------------------------------- /tests/unit/altimeter/aws/resource/iam/test_group.py: -------------------------------------------------------------------------------- 1 | import boto3 2 | from botocore.exceptions import ClientError 3 | from unittest import TestCase 4 | from moto import mock_iam 5 | from unittest.mock import patch 6 | from altimeter.aws.resource.iam.group import IAMGroupResourceSpec 7 | from altimeter.aws.scan.aws_accessor import AWSAccessor 8 | from altimeter.aws.scan.settings import ALL_RESOURCE_SPEC_CLASSES 9 | 10 | 11 | class TestIAMGroup(TestCase): 12 | @mock_iam 13 | def test_disappearing_group_race_condition(self): 14 | account_id = "123456789012" 15 | group_name = "foo" 16 | region_name = "us-east-1" 17 | 18 | session = boto3.Session() 19 | 20 | client = session.client("iam") 21 | 22 | client.create_group(GroupName=group_name) 23 | 24 | scan_accessor = AWSAccessor(session=session, account_id=account_id, region_name=region_name) 25 | with patch( 26 | "altimeter.aws.resource.iam.group.IAMGroupResourceSpec.get_group_users" 27 | ) as mock_get_group_users: 28 | mock_get_group_users.side_effect = ClientError( 29 | operation_name="GetGroup", 30 | error_response={ 31 | "Error": { 32 | "Code": "NoSuchEntity", 33 | "Message": f"The group with name {group_name} cannot be found.", 34 | } 35 | }, 36 | ) 37 | resources = IAMGroupResourceSpec.scan( 38 | scan_accessor=scan_accessor, 39 | all_resource_spec_classes=[ALL_RESOURCE_SPEC_CLASSES], 40 | ) 41 | self.assertEqual(resources, []) 42 | -------------------------------------------------------------------------------- /altimeter/core/graph/field/base.py: -------------------------------------------------------------------------------- 1 | """Base classes for Fields. Fields define how individual elements of input JSON are parsed 2 | into a LinkCollection.""" 3 | import abc 4 | from typing import Any, Dict 5 | 6 | from altimeter.core.graph.field.exceptions import ( 7 | ParentKeyMissingException, 8 | InvalidParentKeyException, 9 | ) 10 | from altimeter.core.graph.links import LinkCollection 11 | 12 | 13 | class Field(abc.ABC): 14 | """Abstract base class for all fields""" 15 | 16 | @abc.abstractmethod 17 | def parse(self, data: Any, context: Dict[str, Any]) -> LinkCollection: 18 | """Parse data into a LinkCollection using this field's definition.""" 19 | 20 | 21 | class SubField(Field): 22 | """SubFields are fields which must have a non-anonymous parent Field.""" 23 | 24 | def get_parent_alti_key(self, data: Any, context: Dict[str, Any]) -> str: 25 | """Get the alti_key of the parent of this SubField. 26 | 27 | Args: 28 | data: field data 29 | context: contains auxiliary information which can be passed through the parse process. 30 | 31 | Returns: 32 | alti_key of parent of this SubField 33 | """ 34 | parent_alti_key = context.get("parent_alti_key") 35 | if parent_alti_key is None: 36 | raise ParentKeyMissingException( 37 | ( 38 | f"Missing parent_alti_key in context for " 39 | f"{self.__class__.__name__} , data: {data}" 40 | ) 41 | ) 42 | if not isinstance(parent_alti_key, str): 43 | raise InvalidParentKeyException(f"ParentKey {parent_alti_key} is not a str.") 44 | return parent_alti_key 45 | -------------------------------------------------------------------------------- /altimeter/qj/models/job.py: -------------------------------------------------------------------------------- 1 | """Job related table definition""" 2 | # pylint: disable=too-few-public-methods 3 | 4 | from sqlalchemy import Boolean, Column, DateTime, Enum, Index, Integer, Text, UniqueConstraint 5 | from sqlalchemy.dialects.postgresql import JSONB 6 | from sqlalchemy.orm import relationship 7 | 8 | from altimeter.qj.db.base_class import BASE 9 | from altimeter.qj.schemas.job import Category, Severity 10 | 11 | 12 | class Job(BASE): 13 | """Job table definition""" 14 | 15 | __tablename__ = "job" 16 | 17 | id = Column(Integer, primary_key=True) 18 | name = Column(Text, nullable=False) 19 | description = Column(Text, nullable=False) 20 | graph_spec = Column(JSONB, nullable=False) 21 | query_fields = Column(JSONB, nullable=False) 22 | category = Column(Enum(Category), nullable=False) 23 | severity = Column(Enum(Severity), nullable=False) 24 | query = Column(Text, nullable=False) 25 | max_graph_age_sec = Column(Integer, nullable=False) 26 | created = Column(DateTime, nullable=False) 27 | active = Column(Boolean, nullable=False, server_default="false") 28 | result_expiration_sec = Column(Integer, nullable=False) 29 | max_result_age_sec = Column(Integer, nullable=False) 30 | result_sets = relationship("ResultSet", passive_deletes=True) 31 | notify_if_results = Column(Boolean, nullable=False, server_default="false") 32 | remediate_sqs_queue = Column(Text, nullable=True) 33 | raw_query = Column(Boolean, nullable=False, server_default="false") 34 | 35 | __table_args__ = ( 36 | Index("job_name_active_key", name, active, unique=True, postgresql_where=(active)), 37 | UniqueConstraint(name, created, name="job_name_created_key"), 38 | ) 39 | -------------------------------------------------------------------------------- /doc/requirements.txt: -------------------------------------------------------------------------------- 1 | # 2 | # This file is autogenerated by pip-compile with Python 3.8 3 | # by the following command: 4 | # 5 | # pip-compile requirements.in 6 | # 7 | alabaster==0.7.12 8 | # via sphinx 9 | babel==2.9.1 10 | # via sphinx 11 | certifi==2023.7.22 12 | # via 13 | # -r requirements.in 14 | # requests 15 | charset-normalizer==2.0.12 16 | # via requests 17 | docutils==0.16 18 | # via sphinx 19 | idna==3.3 20 | # via requests 21 | imagesize==1.3.0 22 | # via sphinx 23 | jinja2==3.0.3 24 | # via 25 | # -r requirements.in 26 | # sphinx 27 | markupsafe==2.1.1 28 | # via 29 | # -r requirements.in 30 | # jinja2 31 | packaging==21.3 32 | # via sphinx 33 | pygments==2.15.0 34 | # via 35 | # -r requirements.in 36 | # sphinx 37 | pyparsing==3.0.7 38 | # via packaging 39 | pytz==2021.3 40 | # via babel 41 | requests==2.31.0 42 | # via 43 | # -r requirements.in 44 | # sphinx 45 | snowballstemmer==2.2.0 46 | # via sphinx 47 | sphinx==3.5.4 48 | # via 49 | # -r requirements.in 50 | # sphinx-autodoc-typehints 51 | sphinx-autodoc-typehints==1.12.0 52 | # via -r requirements.in 53 | sphinxcontrib-applehelp==1.0.2 54 | # via sphinx 55 | sphinxcontrib-devhelp==1.0.2 56 | # via sphinx 57 | sphinxcontrib-htmlhelp==2.0.0 58 | # via sphinx 59 | sphinxcontrib-jsmath==1.0.1 60 | # via sphinx 61 | sphinxcontrib-qthelp==1.0.3 62 | # via sphinx 63 | sphinxcontrib-serializinghtml==1.1.5 64 | # via sphinx 65 | urllib3==1.26.18 66 | # via 67 | # -r requirements.in 68 | # requests 69 | 70 | # The following packages are considered to be unsafe in a requirements file: 71 | # setuptools 72 | -------------------------------------------------------------------------------- /altimeter/core/neptune/sparql.py: -------------------------------------------------------------------------------- 1 | """SPARQL related functions.""" 2 | 3 | import re 4 | from typing import List 5 | 6 | from altimeter.core.neptune.exceptions import InvalidQueryException 7 | 8 | 9 | def finalize_query(query: str, graph_uris: List[str]) -> str: 10 | """Finalize a generic sparql query - specifically add a FROM clause 11 | containing graph uris for this query. 12 | 13 | Args: 14 | query: query string 15 | graph_uris: list of graph uris 16 | 17 | Returns: 18 | finalized query string 19 | """ 20 | # find the where clause. once found, insert a from clause before it with the 21 | # graph_uris. 22 | found_where = False 23 | output_lines = [] 24 | for line in query.split("\n"): 25 | if found_where: 26 | output_lines.append(line) 27 | else: 28 | where_regex = r"^([^#]|)+(\s|^)(where)(\s+{|{|\s*$|\s*#).*$" 29 | where_match = re.search(where_regex, line, re.IGNORECASE) 30 | if where_match: 31 | found_where = True 32 | before = line[: where_match.start(2)].rstrip() 33 | after = line[where_match.start(2) :].lstrip() 34 | final_query_lines = [] 35 | if before: 36 | final_query_lines.append(before) 37 | for graph_uri in graph_uris: 38 | final_query_lines.append(f"FROM <{graph_uri}>") 39 | final_query_lines.append(after) 40 | output_lines.append(" ".join(final_query_lines)) 41 | else: 42 | output_lines.append(line) 43 | if found_where: 44 | return "\n".join(output_lines) 45 | raise InvalidQueryException(f"Unable to find where clause in {query}") 46 | -------------------------------------------------------------------------------- /doc/source/user/quickstart.rst: -------------------------------------------------------------------------------- 1 | Quickstart 2 | ========== 3 | 4 | This quickstart guide demonstrates how to generate a graph for a single account. 5 | 6 | Installation 7 | ------------ 8 | 9 | :: 10 | 11 | pip install altimeter 12 | 13 | Configuration 14 | ------------- 15 | 16 | Altimeter's behavior is driven by a toml configuration file. A few sample 17 | configuration files are included in the `conf/` directory: 18 | 19 | * `current_single_account.toml` - scans the current account - this is the account 20 | for which the environment's currently configured AWS CLI credentials are. 21 | 22 | * `current_master_multi_account.toml` - scans the current account and attempts to 23 | scan all organizational subaccounts - this configuration should be used if you 24 | are scanning all accounts in an organization. To do this the currently 25 | configured AWS CLI credentials should be pointing to an AWS Organizations 26 | master account. 27 | 28 | To scan a subset of regions, set the region list parameter `regions` in the `scan` 29 | section to a list of region names. 30 | 31 | Generating the Graph 32 | -------------------- 33 | 34 | Assuming you have configured AWS CLI credentials 35 | (see ), 36 | run: 37 | 38 | altimeter 39 | 40 | This will scan all resources in regions specified in the config file. 41 | 42 | The full path to the generated RDF file will printed, for example: 43 | 44 | Created /tmp/altimeter/20191018/1571425383/graph.rdf 45 | 46 | This RDF file can then be loaded into a triplestore such as Neptune or Blazegraph for querying. 47 | 48 | Tooling is included for loading into a local Blazegraph instance, see 49 | :doc:`Local Querying with Blazegraph ` 50 | 51 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | # 2 | # This file is autogenerated by pip-compile with Python 3.8 3 | # by the following command: 4 | # 5 | # pip-compile requirements.in 6 | # 7 | aenum==2.2.6 8 | # via gremlinpython 9 | aws-requests-auth==0.4.3 10 | # via -r requirements.in 11 | boto3==1.28.80 12 | # via -r requirements.in 13 | botocore==1.31.80 14 | # via 15 | # boto3 16 | # s3transfer 17 | certifi==2023.7.22 18 | # via 19 | # -r requirements.in 20 | # requests 21 | charset-normalizer==2.0.12 22 | # via requests 23 | gremlinpython==3.4.12 24 | # via -r requirements.in 25 | idna==3.3 26 | # via requests 27 | isodate==0.6.1 28 | # via 29 | # gremlinpython 30 | # rdflib 31 | jinja2==3.0.3 32 | # via -r requirements.in 33 | jmespath==0.10.0 34 | # via 35 | # boto3 36 | # botocore 37 | markupsafe==2.1.1 38 | # via 39 | # -r requirements.in 40 | # jinja2 41 | pydantic==1.9.0 42 | # via -r requirements.in 43 | pyparsing==3.0.7 44 | # via rdflib 45 | python-dateutil==2.8.2 46 | # via botocore 47 | rdflib==6.0.2 48 | # via -r requirements.in 49 | requests==2.31.0 50 | # via 51 | # -r requirements.in 52 | # aws-requests-auth 53 | s3transfer==0.7.0 54 | # via boto3 55 | six==1.16.0 56 | # via 57 | # gremlinpython 58 | # isodate 59 | # python-dateutil 60 | structlog==20.2.0 61 | # via -r requirements.in 62 | toml==0.10.2 63 | # via -r requirements.in 64 | tornado==5.1.1 65 | # via gremlinpython 66 | typing-extensions==4.1.1 67 | # via pydantic 68 | urllib3==1.26.18 69 | # via 70 | # -r requirements.in 71 | # botocore 72 | # requests 73 | 74 | # The following packages are considered to be unsafe in a requirements file: 75 | # setuptools 76 | -------------------------------------------------------------------------------- /altimeter/aws/resource/ec2/subnet.py: -------------------------------------------------------------------------------- 1 | """Resource for Subnets""" 2 | import ipaddress 3 | from typing import Type 4 | 5 | from botocore.client import BaseClient 6 | 7 | from altimeter.aws.resource.resource_spec import ListFromAWSResult 8 | from altimeter.aws.resource.ec2 import EC2ResourceSpec 9 | from altimeter.aws.resource.ec2.vpc import VPCResourceSpec 10 | from altimeter.core.graph.field.resource_link_field import ResourceLinkField 11 | from altimeter.core.graph.field.scalar_field import ScalarField 12 | from altimeter.core.graph.field.tags_field import TagsField 13 | from altimeter.core.graph.schema import Schema 14 | 15 | 16 | class SubnetResourceSpec(EC2ResourceSpec): 17 | """Resource for Subnets""" 18 | 19 | type_name = "subnet" 20 | schema = Schema( 21 | ScalarField("CidrBlock"), 22 | ScalarField("FirstIp"), 23 | ScalarField("LastIp"), 24 | ScalarField("State"), 25 | TagsField(), 26 | ResourceLinkField("VpcId", VPCResourceSpec), 27 | ) 28 | 29 | @classmethod 30 | def list_from_aws( 31 | cls: Type["SubnetResourceSpec"], client: BaseClient, account_id: str, region: str 32 | ) -> ListFromAWSResult: 33 | subnets = {} 34 | resp = client.describe_subnets() 35 | for subnet in resp.get("Subnets", []): 36 | resource_arn = cls.generate_arn( 37 | account_id=account_id, region=region, resource_id=subnet["SubnetId"] 38 | ) 39 | cidr = subnet["CidrBlock"] 40 | ipv4_network = ipaddress.IPv4Network(cidr, strict=False) 41 | first_ip, last_ip = int(ipv4_network[0]), int(ipv4_network[-1]) 42 | subnet["FirstIp"] = first_ip 43 | subnet["LastIp"] = last_ip 44 | subnets[resource_arn] = subnet 45 | return ListFromAWSResult(resources=subnets) 46 | -------------------------------------------------------------------------------- /tests/unit/altimeter/core/graph/field/test_tags_field.py: -------------------------------------------------------------------------------- 1 | import json 2 | from unittest import TestCase 3 | 4 | from altimeter.core.graph.links import LinkCollection, TagLink 5 | from altimeter.core.graph.field.tags_field import TagsField 6 | from altimeter.core.graph.field.exceptions import TagsFieldMissingTagsKeyException 7 | 8 | 9 | class TestTagsField(TestCase): 10 | def test_valid_input(self): 11 | input_str = ( 12 | '{"Tags": [{"Key": "tag1", "Value": "value1"}, {"Key": "tag2", "Value": "value2"}]}' 13 | ) 14 | field = TagsField() 15 | expected_output_data = [ 16 | {"pred": "tag1", "obj": "value1", "type": "tag"}, 17 | {"pred": "tag2", "obj": "value2", "type": "tag"}, 18 | ] 19 | 20 | input_data = json.loads(input_str) 21 | link_collection = field.parse(data=input_data, context={}) 22 | 23 | expected_link_collection = LinkCollection( 24 | tag_links=(TagLink(pred="tag1", obj="value1"), TagLink(pred="tag2", obj="value2")), 25 | ) 26 | self.assertEqual(link_collection, expected_link_collection) 27 | 28 | def test_invalid_input(self): 29 | input_str = ( 30 | '{"Mags": [{"Key": "tag1", "Value": "value1"}, {"Key": "tag2", "Value": "value2"}]}' 31 | ) 32 | field = TagsField(optional=False) 33 | 34 | input_data = json.loads(input_str) 35 | with self.assertRaises(TagsFieldMissingTagsKeyException): 36 | field.parse(data=input_data, context={}) 37 | 38 | def test_optional(self): 39 | input_str = '{"NoTagsHere": []}' 40 | field = TagsField(optional=True) 41 | 42 | input_data = json.loads(input_str) 43 | link_collection = field.parse(data=input_data, context={}) 44 | 45 | self.assertCountEqual(link_collection, LinkCollection()) 46 | -------------------------------------------------------------------------------- /tests/unit/altimeter/aws/resource/elbv2/test_target_group.py: -------------------------------------------------------------------------------- 1 | import boto3 2 | from botocore.exceptions import ClientError 3 | from unittest import TestCase 4 | from moto import mock_ec2, mock_elbv2 5 | from unittest.mock import patch 6 | from altimeter.aws.resource.elbv2.target_group import TargetGroupResourceSpec 7 | from altimeter.aws.scan.aws_accessor import AWSAccessor 8 | from altimeter.aws.scan.settings import ALL_RESOURCE_SPEC_CLASSES 9 | 10 | 11 | class TestTargetGroup(TestCase): 12 | @mock_elbv2 13 | @mock_ec2 14 | def test_disappearing_target_group_race_condition(self): 15 | account_id = "123456789012" 16 | region_name = "us-east-1" 17 | tg_name = "foo" 18 | 19 | session = boto3.Session() 20 | 21 | client = session.client("elbv2", region_name=region_name) 22 | 23 | resp = client.create_target_group(Name=tg_name, Port=443) 24 | tg_arn = resp["TargetGroups"][0]["TargetGroupArn"] 25 | 26 | scan_accessor = AWSAccessor(session=session, account_id=account_id, region_name=region_name) 27 | with patch( 28 | "altimeter.aws.resource.elbv2.target_group.get_target_group_health" 29 | ) as mock_get_target_group_health: 30 | mock_get_target_group_health.side_effect = ClientError( 31 | operation_name="DescribeTargetHealth", 32 | error_response={ 33 | "Error": { 34 | "Code": "TargetGroupNotFound", 35 | "Message": f"Target groups '{tg_arn}' not found", 36 | } 37 | }, 38 | ) 39 | resources = TargetGroupResourceSpec.scan( 40 | scan_accessor=scan_accessor, 41 | all_resource_spec_classes=ALL_RESOURCE_SPEC_CLASSES, 42 | ) 43 | self.assertEqual(resources, []) 44 | -------------------------------------------------------------------------------- /altimeter/core/log_events.py: -------------------------------------------------------------------------------- 1 | from dataclasses import dataclass 2 | 3 | from altimeter.core.log import BaseLogEvent, EventName 4 | 5 | 6 | @dataclass(frozen=True) 7 | class LogEvent(BaseLogEvent): 8 | """Contains EventNames for logging.""" 9 | 10 | AuthToAccountStart: EventName 11 | AuthToAccountEnd: EventName 12 | AuthToAccountFailure: EventName 13 | 14 | CompileGraphsStart: EventName 15 | CompileGraphsEnd: EventName 16 | 17 | GraphLoadedSNSNotificationStart: EventName 18 | GraphLoadedSNSNotificationEnd: EventName 19 | 20 | GraphSetToRDFStart: EventName 21 | GraphSetToRDFEnd: EventName 22 | 23 | MetadataGraphUpdateStart: EventName 24 | MetadataGraphUpdateEnd: EventName 25 | 26 | NeptuneLoadStart: EventName 27 | NeptuneLoadEnd: EventName 28 | NeptuneLoadPolling: EventName 29 | NeptuneLoadError: EventName 30 | 31 | NeptuneRDFWriteStart: EventName 32 | NeptuneRDFWriteEnd: EventName 33 | NeptuneGremlinWriteStart: EventName 34 | NeptuneGremlinWriteEnd: EventName 35 | NeptunePeriodicWrite: EventName 36 | 37 | PruneNeptuneGraphStart: EventName 38 | PruneNeptuneGraphEnd: EventName 39 | PruneNeptuneGraphError: EventName 40 | PruneNeptuneGraphSkip: EventName 41 | PruneOrphanedNeptuneGraphStart: EventName 42 | PruneOrphanedNeptuneGraphEnd: EventName 43 | 44 | PruneNeptuneGraphsStart: EventName 45 | PruneNeptuneGraphsEnd: EventName 46 | PruneNeptuneGraphsError: EventName 47 | 48 | ReadFromFSStart: EventName 49 | ReadFromFSEnd: EventName 50 | 51 | ReadFromS3Start: EventName 52 | ReadFromS3End: EventName 53 | 54 | ScanResourceTypeStart: EventName 55 | ScanResourceTypeEnd: EventName 56 | 57 | ValidateGraphStart: EventName 58 | ValidateGraphEnd: EventName 59 | 60 | WriteToFSStart: EventName 61 | WriteToFSEnd: EventName 62 | 63 | WriteToS3Start: EventName 64 | WriteToS3End: EventName 65 | -------------------------------------------------------------------------------- /altimeter/aws/resource/ec2/vpc_endpoint.py: -------------------------------------------------------------------------------- 1 | """Resource for VPC Endpoints""" 2 | from typing import Type 3 | 4 | from botocore.client import BaseClient 5 | 6 | from altimeter.aws.resource.resource_spec import ListFromAWSResult 7 | from altimeter.aws.resource.ec2 import EC2ResourceSpec 8 | from altimeter.aws.resource.ec2.vpc import VPCResourceSpec 9 | from altimeter.core.graph.field.scalar_field import ScalarField 10 | from altimeter.core.graph.field.resource_link_field import ResourceLinkField 11 | from altimeter.core.graph.field.tags_field import TagsField 12 | from altimeter.core.graph.schema import Schema 13 | 14 | 15 | class VpcEndpointResourceSpec(EC2ResourceSpec): 16 | """Resource for VPC Endpoints""" 17 | 18 | type_name = "vpc-endpoint" 19 | schema = Schema( 20 | ScalarField("VpcEndpointType"), 21 | ScalarField("ServiceName"), 22 | ScalarField("State"), 23 | ResourceLinkField("VpcId", VPCResourceSpec), 24 | TagsField(), 25 | ) 26 | 27 | @classmethod 28 | def list_from_aws( 29 | cls: Type["VpcEndpointResourceSpec"], client: BaseClient, account_id: str, region: str 30 | ) -> ListFromAWSResult: 31 | """Return a dict of dicts of the format: 32 | 33 | {'vpc_endpoint_1_arn': {vpc_endpoint_1_dict}, 34 | 'vpc_endpoint_1_arn': {vpc_endpoint_2_dict}, 35 | ...} 36 | 37 | Where the dicts represent results from describe_vpc_endpoints.""" 38 | endpoints = {} 39 | paginator = client.get_paginator("describe_vpc_endpoints") 40 | for resp in paginator.paginate(): 41 | for endpoint in resp.get("VpcEndpoints", []): 42 | resource_arn = cls.generate_arn( 43 | account_id=account_id, region=region, resource_id=endpoint["VpcEndpointId"] 44 | ) 45 | endpoints[resource_arn] = endpoint 46 | return ListFromAWSResult(resources=endpoints) 47 | -------------------------------------------------------------------------------- /tests/unit/altimeter/aws/resource/elbv1/test_load_balancer.py: -------------------------------------------------------------------------------- 1 | import boto3 2 | from botocore.exceptions import ClientError 3 | from unittest import TestCase 4 | from moto import mock_elb 5 | from unittest.mock import patch 6 | from altimeter.aws.resource.elbv1.load_balancer import ClassicLoadBalancerResourceSpec 7 | from altimeter.aws.scan.aws_accessor import AWSAccessor 8 | from altimeter.aws.scan.settings import ALL_RESOURCE_SPEC_CLASSES 9 | 10 | 11 | class TestLB(TestCase): 12 | @mock_elb 13 | def test_disappearing_elb_race_condition(self): 14 | account_id = "123456789012" 15 | region_name = "us-east-1" 16 | lb_name = "foo" 17 | 18 | session = boto3.Session() 19 | client = session.client("elb", region_name=region_name) 20 | 21 | client.create_load_balancer( 22 | LoadBalancerName=lb_name, 23 | Listeners=[{"Protocol": "HTTP", "LoadBalancerPort": 80, "InstancePort": 80}], 24 | Tags=[{"Key": "Name", "Value": lb_name}], 25 | ) 26 | 27 | scan_accessor = AWSAccessor(session=session, account_id=account_id, region_name=region_name) 28 | with patch( 29 | "altimeter.aws.resource.elbv1.load_balancer.ClassicLoadBalancerResourceSpec.get_lb_attrs" 30 | ) as mock_get_lb_attrs: 31 | mock_get_lb_attrs.side_effect = ClientError( 32 | operation_name="DescribeLoadBalancerAttributes", 33 | error_response={ 34 | "Error": { 35 | "Code": "LoadBalancerNotFound", 36 | "Message": f"There is no ACTIVE Load Balancer named '{lb_name}'", 37 | } 38 | }, 39 | ) 40 | resources = ClassicLoadBalancerResourceSpec.scan( 41 | scan_accessor=scan_accessor, 42 | all_resource_spec_classes=ALL_RESOURCE_SPEC_CLASSES, 43 | ) 44 | self.assertEqual(resources, []) 45 | -------------------------------------------------------------------------------- /bin/aws2n.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | """Graph AWS resource data in Neptune""" 3 | import argparse 4 | import os 5 | import sys 6 | from typing import List, Optional, Type 7 | 8 | from altimeter.aws.aws2n import generate_scan_id, aws2n 9 | from altimeter.aws.resource.resource_spec import AWSResourceSpec 10 | from altimeter.aws.scan.muxer.local_muxer import LocalAWSScanMuxer 11 | from altimeter.aws.scan.settings import DEFAULT_RESOURCE_SPEC_CLASSES 12 | from altimeter.core.config import AWSConfig 13 | 14 | 15 | def main(argv: Optional[List[str]] = None) -> int: 16 | if argv is None: 17 | argv = sys.argv[1:] 18 | parser = argparse.ArgumentParser() 19 | parser.add_argument("config", type=str, nargs="?") 20 | args_ns = parser.parse_args(argv) 21 | 22 | config = args_ns.config 23 | if config is None: 24 | config = os.environ.get("CONFIG_PATH") 25 | if config is None: 26 | print("config must be provided as a positional arg or env var 'CONFIG_PATH'") 27 | return 1 28 | 29 | config = AWSConfig.from_path(config) 30 | if config.scan.ignored_resources: 31 | resource_spec_classes_list: List[Type[AWSResourceSpec]] = [] 32 | for resource_spec_class in DEFAULT_RESOURCE_SPEC_CLASSES: 33 | if resource_spec_class.get_full_type_name() not in config.scan.ignored_resources: 34 | resource_spec_classes_list.append(resource_spec_class) 35 | resource_spec_classes = tuple(resource_spec_classes_list) 36 | else: 37 | resource_spec_classes = DEFAULT_RESOURCE_SPEC_CLASSES 38 | scan_id = generate_scan_id() 39 | muxer = LocalAWSScanMuxer( 40 | scan_id=scan_id, 41 | config=config, 42 | resource_spec_classes=resource_spec_classes, 43 | ) 44 | result = aws2n(scan_id=scan_id, config=config, muxer=muxer, load_neptune=False) 45 | print(result.rdf_path) 46 | return 0 47 | 48 | 49 | if __name__ == "__main__": 50 | sys.exit(main()) 51 | -------------------------------------------------------------------------------- /bin/rdf2blaze: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -euf -o pipefail 4 | 5 | if [ $# -ne 1 ]; then 6 | >&2 echo "Usage: $0 " 7 | >&2 echo "Where path-to-rdf is a path to an RDF file." 8 | exit 1 9 | fi 10 | 11 | rdf_path=$1 12 | 13 | if [ ! -f "$rdf_path" ]; then 14 | >&2 echo "Path $rdf_path does not appear to be a file." 15 | exit 1 16 | fi 17 | 18 | blazegraph_host=localhost 19 | blazegraph_port=8889 20 | blazegraph_base_url="http://$blazegraph_host:$blazegraph_port/bigdata" 21 | blazegraph_load_url="$blazegraph_base_url/sparql" 22 | blazegraph_query_url="$blazegraph_base_url/#query" 23 | blaze_status_url="$blazegraph_base_url/status" 24 | 25 | echo "Starting Blazegraph docker container." 26 | blazegraph_container_id=$(docker run -d -p $blazegraph_port:8080 lyrasis/blazegraph:2.1.5) 27 | trap 'echo "Stopping Blazegraph docker container $blazegraph_container_id" && docker kill $blazegraph_container_id' EXIT 28 | echo "Blazegraph container running: $blazegraph_container_id" 29 | echo "Waiting for service to be available..." 30 | n=0 31 | until [ $n -ge 10 ]; do 32 | echo "Waiting for $blaze_status_url..." 33 | sleep 5 34 | set +e 35 | curl "$blaze_status_url" > /dev/null 2>&1 36 | curl_ret=$? 37 | set -e 38 | if [[ $curl_ret == 0 ]]; then 39 | echo "$blaze_status_url is up." 40 | break 41 | fi 42 | n=$[$n+1] 43 | done 44 | if [ $n -ge 10 ]; then 45 | >&2 echo "$blaze_status_url did not come up!" 46 | exit 1 47 | fi 48 | 49 | echo "Loading $rdf_path using $blazegraph_load_url" 50 | load_stats=$(curl -X POST -H "Content-Type: application/rdf+xml" -d @"$rdf_path" "$blazegraph_load_url" 2>/dev/null) 51 | echo "Load stats: $load_stats" 52 | echo "Data available at $blazegraph_query_url" 53 | 54 | echo "Blazegraph is running with data from $rdf_path." 55 | echo 56 | echo "Query UI is available at $blazegraph_query_url" 57 | echo 58 | echo "Hit CTRL-C to exit." 59 | read -r -d '' _ >> input = {"Tags": [{"Key": "Name", "Value": "Jerry"}, \ 16 | {"Key": "DOB", "Value": "1942-08-01"}]} 17 | >>> field = TagsField() 18 | >>> link_collection = field.parse(data=input, context={}) 19 | >>> print(link_collection.dict(exclude_unset=True)) 20 | {'tag_links': ({'pred': 'Name', 'obj': 'Jerry'}, {'pred': 'DOB', 'obj': '1942-08-01'})} 21 | 22 | Args: 23 | optional: Whether this key is optional. Defaults to False. 24 | """ 25 | 26 | def __init__(self, optional: bool = True): 27 | self.optional = optional 28 | 29 | def parse(self, data: Dict[str, Any], context: Dict[str, Any]) -> LinkCollection: 30 | """Parse this field and return a list of Links. 31 | 32 | Args: 33 | data: dictionary of data to parse 34 | context: context dict containing data from higher level parsing code. 35 | 36 | Returns: 37 | List of TagLink objects, one for each tag. 38 | """ 39 | links: List[TagLink] = [] 40 | tag_dicts = data.get("Tags") 41 | if tag_dicts: 42 | for tag_dict in tag_dicts: 43 | links.append(TagLink(pred=tag_dict["Key"], obj=tag_dict["Value"])) 44 | return LinkCollection(tag_links=links) 45 | if self.optional: 46 | return LinkCollection() 47 | raise TagsFieldMissingTagsKeyException(f"Expected key 'Tags' in {data}") 48 | -------------------------------------------------------------------------------- /tests/unit/altimeter/aws/resource/events/test_cloudwatchevents_rule.py: -------------------------------------------------------------------------------- 1 | import boto3 2 | from botocore.exceptions import ClientError 3 | from unittest import TestCase 4 | from moto import mock_events 5 | from unittest.mock import patch 6 | from altimeter.aws.resource.events.cloudwatchevents_rule import EventsRuleResourceSpec 7 | from altimeter.aws.scan.aws_accessor import AWSAccessor 8 | from altimeter.aws.scan.settings import ALL_RESOURCE_SPEC_CLASSES 9 | 10 | 11 | class TestEventsRule(TestCase): 12 | @mock_events 13 | def test_disappearing_rule_race_condition(self): 14 | account_id = "123456789012" 15 | region_name = "us-east-1" 16 | rule_name = "test_rule" 17 | 18 | session = boto3.Session() 19 | client = session.client("events", region_name=region_name) 20 | client.put_rule( 21 | Name=rule_name, 22 | Description="Capture all events and forward them to 012345678901", 23 | EventPattern=f"""{{"account":["012345678901"]}}""", 24 | State="ENABLED", 25 | ) 26 | 27 | scan_accessor = AWSAccessor(session=session, account_id=account_id, region_name=region_name) 28 | with patch( 29 | "altimeter.aws.resource.events.cloudwatchevents_rule.list_targets_by_rule" 30 | ) as mock_list_targets_by_rule: 31 | mock_list_targets_by_rule.side_effect = ClientError( 32 | operation_name="ListTargetsByRule", 33 | error_response={ 34 | "Error": { 35 | "Code": "ResourceNotFoundException", 36 | "Message": f"Rule {rule_name} does not exist on EventBus default.", 37 | } 38 | }, 39 | ) 40 | resources = EventsRuleResourceSpec.scan( 41 | scan_accessor=scan_accessor, 42 | all_resource_spec_classes=ALL_RESOURCE_SPEC_CLASSES, 43 | ) 44 | self.assertEqual(resources, []) 45 | -------------------------------------------------------------------------------- /tests/unit/altimeter/aws/resource/iam/test_saml_provider.py: -------------------------------------------------------------------------------- 1 | import boto3 2 | from botocore.exceptions import ClientError 3 | from unittest import TestCase 4 | from moto import mock_iam 5 | from unittest.mock import patch 6 | from altimeter.aws.resource.iam.iam_saml_provider import IAMSAMLProviderResourceSpec 7 | from altimeter.aws.scan.aws_accessor import AWSAccessor 8 | from altimeter.aws.scan.settings import ALL_RESOURCE_SPEC_CLASSES 9 | 10 | 11 | class TestIAMSAMLProvider(TestCase): 12 | @mock_iam 13 | def test_disappearing_saml_provider_race_condition(self): 14 | account_id = "123456789012" 15 | saml_provider_name = "foo" 16 | region_name = "us-east-1" 17 | 18 | session = boto3.Session() 19 | 20 | client = session.client("iam") 21 | 22 | saml_provider_resp = client.create_saml_provider( 23 | Name=saml_provider_name, SAMLMetadataDocument="a" * 1024 24 | ) 25 | saml_provider_arn = saml_provider_resp["SAMLProviderArn"] 26 | 27 | scan_accessor = AWSAccessor(session=session, account_id=account_id, region_name=region_name) 28 | with patch( 29 | "altimeter.aws.resource.iam.iam_saml_provider.IAMSAMLProviderResourceSpec.get_saml_provider_metadata_doc" 30 | ) as mock_get_saml_provider_metadata_doc: 31 | mock_get_saml_provider_metadata_doc.side_effect = ClientError( 32 | operation_name="GetSAMLProvider", 33 | error_response={ 34 | "Error": { 35 | "Code": "NoSuchEntity", 36 | "Message": f"GetSAMLProvider operation: Manifest not found for arn {saml_provider_arn}", 37 | } 38 | }, 39 | ) 40 | resources = IAMSAMLProviderResourceSpec.scan( 41 | scan_accessor=scan_accessor, 42 | all_resource_spec_classes=ALL_RESOURCE_SPEC_CLASSES, 43 | ) 44 | self.assertEqual(resources, []) 45 | -------------------------------------------------------------------------------- /services/qj/requirements.txt: -------------------------------------------------------------------------------- 1 | # 2 | # This file is autogenerated by pip-compile with Python 3.8 3 | # by the following command: 4 | # 5 | # pip-compile requirements.in 6 | # 7 | alembic==1.4.2 8 | # via -r requirements.in 9 | anyio==3.5.0 10 | # via starlette 11 | asgiref==3.5.0 12 | # via uvicorn 13 | boto3==1.28.80 14 | # via -r requirements.in 15 | botocore==1.31.80 16 | # via 17 | # boto3 18 | # s3transfer 19 | certifi==2023.7.22 20 | # via requests 21 | cffi==1.16.0 22 | # via tableauhyperapi 23 | charset-normalizer==2.0.12 24 | # via requests 25 | click==8.0.4 26 | # via uvicorn 27 | fastapi==0.96.0 28 | # via -r requirements.in 29 | h11==0.13.0 30 | # via uvicorn 31 | idna==3.3 32 | # via 33 | # anyio 34 | # requests 35 | jmespath==0.10.0 36 | # via 37 | # boto3 38 | # botocore 39 | mako==1.2.2 40 | # via alembic 41 | markupsafe==2.1.1 42 | # via 43 | # -r requirements.in 44 | # mako 45 | psycopg2-binary==2.9.2 46 | # via -r requirements.in 47 | pycparser==2.21 48 | # via cffi 49 | pydantic==1.9.0 50 | # via fastapi 51 | python-dateutil==2.8.2 52 | # via 53 | # alembic 54 | # botocore 55 | python-editor==1.0.4 56 | # via alembic 57 | requests==2.31.0 58 | # via 59 | # -r requirements.in 60 | # tableauserverclient 61 | s3transfer==0.7.0 62 | # via boto3 63 | six==1.16.0 64 | # via python-dateutil 65 | sniffio==1.2.0 66 | # via anyio 67 | sqlalchemy==1.3.24 68 | # via 69 | # -r requirements.in 70 | # alembic 71 | starlette==0.27.0 72 | # via fastapi 73 | tableauhyperapi==0.0.18161 74 | # via -r requirements.in 75 | tableauserverclient==0.17.0 76 | # via -r requirements.in 77 | typing-extensions==4.1.1 78 | # via 79 | # pydantic 80 | # starlette 81 | urllib3==1.26.18 82 | # via 83 | # -r requirements.in 84 | # botocore 85 | # requests 86 | uvicorn==0.16.0 87 | # via -r requirements.in 88 | -------------------------------------------------------------------------------- /altimeter/aws/resource/account.py: -------------------------------------------------------------------------------- 1 | """Resource representing an AWS Account""" 2 | from typing import List, Type 3 | 4 | from botocore.client import BaseClient 5 | 6 | from altimeter.aws.resource.resource_spec import ScanGranularity, ListFromAWSResult, AWSResourceSpec 7 | from altimeter.aws.resource.unscanned_account import UnscannedAccountResourceSpec 8 | from altimeter.core.resource.resource_spec import ResourceSpec 9 | from altimeter.core.graph.field.scalar_field import ScalarField 10 | from altimeter.core.graph.schema import Schema 11 | 12 | 13 | class AccountResourceSpec(AWSResourceSpec): 14 | """Resource representing an AWS Account""" 15 | 16 | type_name = "account" 17 | service_name = "sts" 18 | scan_granularity = ScanGranularity.ACCOUNT 19 | schema = Schema(ScalarField("account_id")) 20 | allow_clobber: List[Type[ResourceSpec]] = [UnscannedAccountResourceSpec] 21 | 22 | @classmethod 23 | def get_full_type_name(cls: Type["AccountResourceSpec"]) -> str: 24 | return f"{cls.provider_name}:{cls.type_name}" 25 | 26 | @classmethod 27 | def list_from_aws( 28 | cls: Type["AccountResourceSpec"], client: BaseClient, account_id: str, region: str 29 | ) -> ListFromAWSResult: 30 | """This resource is somewhat synthetic, this method simply returns a dict of form 31 | {'account_arn': {account_dict}""" 32 | sts_account_id = client.get_caller_identity()["Account"] 33 | if sts_account_id != account_id: 34 | raise ValueError(f"BUG: sts detected account_id {sts_account_id} != {account_id}") 35 | accounts = {f"arn:aws::::account/{sts_account_id}": {"account_id": sts_account_id}} 36 | return ListFromAWSResult(resources=accounts) 37 | 38 | @classmethod 39 | def generate_arn( 40 | cls: Type["AccountResourceSpec"], 41 | resource_id: str, 42 | account_id: str = "", 43 | region: str = "", 44 | ) -> str: 45 | """Generate an ARN for this resource""" 46 | return f"arn:aws::::account/{resource_id}" 47 | -------------------------------------------------------------------------------- /tests/unit/altimeter/core/test_config.py: -------------------------------------------------------------------------------- 1 | from unittest import TestCase 2 | 3 | from altimeter.core.config import ( 4 | AWSConfig, 5 | InvalidConfigException, 6 | ScanConfig, 7 | ) 8 | 9 | 10 | class TestScanConfig(TestCase): 11 | def test_from_dict(self): 12 | scan_config_dict = { 13 | "accounts": ["123", "456"], 14 | "regions": ["us-west-2", "us-west-1"], 15 | "scan_sub_accounts": False, 16 | "preferred_account_scan_regions": ["us-east-1", "us-west-2"], 17 | } 18 | scan_config = ScanConfig(**scan_config_dict) 19 | self.assertTupleEqual(scan_config.accounts, ("123", "456")) 20 | self.assertTupleEqual(scan_config.regions, ("us-west-2", "us-west-1")) 21 | self.assertEqual(scan_config.scan_sub_accounts, False) 22 | self.assertTupleEqual( 23 | scan_config.preferred_account_scan_regions, ("us-east-1", "us-west-2") 24 | ) 25 | 26 | 27 | class TestConfig(TestCase): 28 | def test_from_dict(self): 29 | config_dict = { 30 | "artifact_path": "/tmp/altimeter_single_account", 31 | "pruner_max_age_min": 4320, 32 | "graph_name": "alti", 33 | "accessor": {"multi_hop_accessors": [], "credentials_cache": {"cache": {}}}, 34 | "concurrency": { 35 | "max_account_scan_threads": 1, 36 | "max_svc_scan_threads": 64, 37 | }, 38 | "scan": { 39 | "accounts": ("1234",), 40 | "regions": (), 41 | "scan_sub_accounts": False, 42 | "preferred_account_scan_regions": ( 43 | "us-west-1", 44 | "us-west-2", 45 | "us-east-1", 46 | "us-east-2", 47 | ), 48 | }, 49 | "neptune": None, 50 | } 51 | config = AWSConfig(**config_dict) 52 | self.assertIsNone(config.neptune) 53 | self.assertEqual(config.pruner_max_age_min, 4320) 54 | -------------------------------------------------------------------------------- /altimeter/aws/resource/eks/cluster.py: -------------------------------------------------------------------------------- 1 | """Resource for Clusters""" 2 | from typing import Type 3 | 4 | from botocore.client import BaseClient 5 | from botocore.exceptions import ClientError 6 | 7 | from altimeter.aws.resource.resource_spec import ListFromAWSResult 8 | from altimeter.aws.resource.eks import EKSResourceSpec 9 | from altimeter.core.graph.field.scalar_field import ScalarField 10 | from altimeter.core.graph.schema import Schema 11 | 12 | 13 | class EKSClusterResourceSpec(EKSResourceSpec): 14 | """Resource for Clusters""" 15 | 16 | type_name = "cluster" 17 | schema = Schema( 18 | ScalarField("Name"), 19 | ) 20 | 21 | @classmethod 22 | def list_from_aws( 23 | cls: Type["EKSClusterResourceSpec"], client: BaseClient, account_id: str, region: str 24 | ) -> ListFromAWSResult: 25 | """Return a dict of dicts of the format: 26 | 27 | {'cluster_1_arn': {cluster_1_dict}, 28 | 'cluster_2_arn': {cluster_2_dict}, 29 | ...} 30 | 31 | Where the dicts represent results from list_clusters.""" 32 | clusters = {} 33 | try: 34 | paginator = client.get_paginator("list_clusters") 35 | for resp in paginator.paginate(): 36 | for cluster_name in resp.get("clusters", []): 37 | resource_arn = cls.generate_arn( 38 | account_id=account_id, region=region, resource_id=cluster_name 39 | ) 40 | clusters[resource_arn] = {"Name": cluster_name} 41 | except ClientError as c_e: 42 | response_error = getattr(c_e, "response", {}).get("Error", {}) 43 | error_code = response_error.get("Code", "") 44 | if error_code != "AccessDeniedException": 45 | raise c_e 46 | error_msg = response_error.get("Message", "") 47 | if error_msg != f"Account {account_id} is not authorized to use this service": 48 | raise c_e 49 | return ListFromAWSResult(resources=clusters) 50 | -------------------------------------------------------------------------------- /altimeter/aws/resource/awslambda/function.py: -------------------------------------------------------------------------------- 1 | """Resource for LambdaFunctions""" 2 | from typing import Type 3 | 4 | from botocore.client import BaseClient 5 | 6 | from altimeter.aws.resource.resource_spec import ListFromAWSResult 7 | from altimeter.aws.resource.ec2.vpc import VPCResourceSpec 8 | from altimeter.aws.resource.awslambda import LambdaResourceSpec 9 | from altimeter.aws.resource.iam.role import IAMRoleResourceSpec 10 | from altimeter.core.graph.field.dict_field import AnonymousDictField 11 | from altimeter.core.graph.field.resource_link_field import TransientResourceLinkField 12 | from altimeter.core.graph.field.scalar_field import ScalarField 13 | from altimeter.core.graph.schema import Schema 14 | 15 | 16 | class LambdaFunctionResourceSpec(LambdaResourceSpec): 17 | """Resource for Lambda Functions""" 18 | 19 | type_name = "function" 20 | schema = Schema( 21 | ScalarField("FunctionName"), 22 | ScalarField("Runtime", optional=True), 23 | AnonymousDictField( 24 | "VpcConfig", 25 | TransientResourceLinkField("VpcId", VPCResourceSpec, optional=True), 26 | optional=True, 27 | ), 28 | TransientResourceLinkField("Role", IAMRoleResourceSpec, value_is_id=True), 29 | ) 30 | 31 | @classmethod 32 | def list_from_aws( 33 | cls: Type["LambdaFunctionResourceSpec"], client: BaseClient, account_id: str, region: str 34 | ) -> ListFromAWSResult: 35 | """Return a dict of dicts of the format: 36 | 37 | {'function_1_arn': {function_1_dict}, 38 | 'function_2_arn': {function_2_dict}, 39 | ...} 40 | 41 | Where the dicts represent results from list_functions.""" 42 | functions = {} 43 | paginator = client.get_paginator("list_functions") 44 | for resp in paginator.paginate(): 45 | for function in resp.get("Functions", []): 46 | resource_arn = function["FunctionArn"] 47 | functions[resource_arn] = function 48 | return ListFromAWSResult(resources=functions) 49 | -------------------------------------------------------------------------------- /altimeter/aws/resource/ec2/transit_gateway_vpc_attachment.py: -------------------------------------------------------------------------------- 1 | """Resource for Transit Gateway VPC Attachments""" 2 | from typing import Type 3 | 4 | from botocore.client import BaseClient 5 | 6 | from altimeter.aws.resource.resource_spec import ListFromAWSResult 7 | from altimeter.aws.resource.ec2 import EC2ResourceSpec 8 | from altimeter.core.graph.schema import Schema 9 | from altimeter.core.graph.field.dict_field import AnonymousDictField 10 | from altimeter.core.graph.field.list_field import ListField 11 | from altimeter.core.graph.field.scalar_field import EmbeddedScalarField, ScalarField 12 | 13 | 14 | class TransitGatewayVpcAttachmentResourceSpec(EC2ResourceSpec): 15 | """Resource for Transit Gateway VPC Attachments""" 16 | 17 | type_name = "transit-gateway-vpc-attachment" 18 | schema = Schema( 19 | ScalarField("TransitGatewayAttachmentId"), 20 | ScalarField("TransitGatewayId"), 21 | ScalarField("VpcId"), 22 | ScalarField("VpcOwnerId"), 23 | ScalarField("State"), 24 | ScalarField("CreationTime"), 25 | ListField("SubnetIds", EmbeddedScalarField(), alti_key="subnet_id"), 26 | AnonymousDictField("Options", ScalarField("DnsSupport"), ScalarField("Ipv6Support")), 27 | ) 28 | 29 | @classmethod 30 | def list_from_aws( 31 | cls: Type["TransitGatewayVpcAttachmentResourceSpec"], 32 | client: BaseClient, 33 | account_id: str, 34 | region: str, 35 | ) -> ListFromAWSResult: 36 | paginator = client.get_paginator("describe_transit_gateway_vpc_attachments") 37 | attachments = {} 38 | for resp in paginator.paginate(): 39 | for attachment in resp.get("TransitGatewayVpcAttachments", []): 40 | resource_arn = cls.generate_arn( 41 | account_id=account_id, 42 | region=region, 43 | resource_id=attachment["TransitGatewayAttachmentId"], 44 | ) 45 | attachments[resource_arn] = attachment 46 | return ListFromAWSResult(resources=attachments) 47 | -------------------------------------------------------------------------------- /altimeter/aws/resource/ec2/snapshot.py: -------------------------------------------------------------------------------- 1 | """Resource for EBSSnapshots""" 2 | from typing import Type 3 | 4 | from botocore.client import BaseClient 5 | 6 | from altimeter.aws.resource.resource_spec import ListFromAWSResult 7 | from altimeter.aws.resource.ec2 import EC2ResourceSpec 8 | from altimeter.aws.resource.ec2.volume import EBSVolumeResourceSpec 9 | from altimeter.aws.resource.kms.key import KMSKeyResourceSpec 10 | from altimeter.core.graph.field.resource_link_field import TransientResourceLinkField 11 | from altimeter.core.graph.field.scalar_field import ScalarField 12 | from altimeter.core.graph.field.tags_field import TagsField 13 | from altimeter.core.graph.schema import Schema 14 | 15 | 16 | class EBSSnapshotResourceSpec(EC2ResourceSpec): 17 | """Resource for EBSSnapshots""" 18 | 19 | type_name = "snapshot" 20 | schema = Schema( 21 | ScalarField("VolumeSize"), 22 | ScalarField("Encrypted"), 23 | TransientResourceLinkField("KmsKeyId", KMSKeyResourceSpec, optional=True, value_is_id=True), 24 | TransientResourceLinkField("VolumeId", EBSVolumeResourceSpec), 25 | TagsField(), 26 | ) 27 | 28 | @classmethod 29 | def list_from_aws( 30 | cls: Type["EBSSnapshotResourceSpec"], client: BaseClient, account_id: str, region: str 31 | ) -> ListFromAWSResult: 32 | """Return a dict of dicts of the format: 33 | 34 | {'snapshot_1_arn': {snapshot_1_dict}, 35 | 'snapshot_2_arn': {snapshot_2_dict}, 36 | ...} 37 | 38 | Where the dicts represent results from describe_snapshots.""" 39 | snapshots = {} 40 | paginator = client.get_paginator("describe_snapshots") 41 | for resp in paginator.paginate(OwnerIds=["self"]): 42 | for snapshot in resp.get("Snapshots", []): 43 | resource_arn = cls.generate_arn( 44 | account_id=account_id, region=region, resource_id=snapshot["SnapshotId"] 45 | ) 46 | snapshots[resource_arn] = snapshot 47 | return ListFromAWSResult(resources=snapshots) 48 | -------------------------------------------------------------------------------- /doc/source/index.rst: -------------------------------------------------------------------------------- 1 | Altimeter 2 | ========= 3 | 4 | Altimeter is a system to scan AWS resources across multiple AWS 5 | Organizations and Accounts and link resources together in a graph 6 | which can be queried to discover security and compliance issues. 7 | 8 | Altimeter outputs RDF files which can be loaded into a triplestore 9 | such as AWS Neptune and Blazegraph. 10 | 11 | This documentation is divided into 3 parts: 12 | 13 | `User Documentation`_ - How to use Altimeter to scan your AWS accounts and query relationships. 14 | 15 | `Developer Documentation`_ - Documentation for developers who are interested in extending Altimeter's capabilities, 16 | specifically graphing additional AWS resource types or additonal fields on existing resource types. 17 | 18 | `API Documentation`_ - auto-generated API documentation. 19 | 20 | User Documentation 21 | ------------------ 22 | 23 | .. toctree:: 24 | :hidden: 25 | :caption: User Documentation 26 | 27 | user/quickstart 28 | user/local_blazegraph 29 | user/sample_queries 30 | 31 | :doc:`Quickstart ` - Introductory guide to graphing a single AWS account. 32 | 33 | Developer Documentation 34 | ----------------------- 35 | 36 | .. toctree:: 37 | :hidden: 38 | :caption: Developer Documentation 39 | 40 | dev/devguide 41 | dev/extending 42 | 43 | :doc:`Developer's Guide ` - Start here. 44 | 45 | :doc:`Extending Altimeter ` - How to extend Altimeter's capabilities to collect and graph more data. 46 | 47 | API Documentation 48 | ----------------- 49 | 50 | .. toctree:: 51 | :hidden: 52 | :caption: API Documentation 53 | 54 | modules 55 | altimeter.aws 56 | altimeter.core 57 | 58 | Altimeter is divided into two packages - `aws` and `core`. 59 | 60 | The :doc:`aws ` package contains AWS-specific access and scanning code for AWS-specific 61 | resources. 62 | 63 | The :doc:`core ` package contains generic graphing code which is used by the `aws` package but could be 64 | used by other graph-generating systems. 65 | 66 | 67 | -------------------------------------------------------------------------------- /tests/unit/altimeter/aws/resource/elbv2/test_load_balancer.py: -------------------------------------------------------------------------------- 1 | import boto3 2 | from botocore.exceptions import ClientError 3 | from unittest import TestCase 4 | from moto import mock_ec2, mock_elbv2 5 | from unittest.mock import patch 6 | from altimeter.aws.resource.elbv2.load_balancer import LoadBalancerResourceSpec 7 | from altimeter.aws.scan.aws_accessor import AWSAccessor 8 | from altimeter.aws.scan.settings import ALL_RESOURCE_SPEC_CLASSES 9 | 10 | 11 | class TestLB(TestCase): 12 | @mock_elbv2 13 | @mock_ec2 14 | def test_disappearing_elb_race_condition(self): 15 | account_id = "123456789012" 16 | region_name = "us-east-1" 17 | lb_name = "foo" 18 | 19 | session = boto3.Session() 20 | ec2_client = session.client("ec2", region_name=region_name) 21 | moto_subnets = [subnet["SubnetId"] for subnet in ec2_client.describe_subnets()["Subnets"]] 22 | 23 | client = session.client("elbv2", region_name=region_name) 24 | 25 | resp = client.create_load_balancer( 26 | Name=lb_name, 27 | Subnets=moto_subnets[:2], 28 | ) 29 | lb_arn = resp["LoadBalancers"][0]["LoadBalancerArn"] 30 | 31 | scan_accessor = AWSAccessor(session=session, account_id=account_id, region_name=region_name) 32 | with patch( 33 | "altimeter.aws.resource.elbv2.load_balancer.LoadBalancerResourceSpec.get_lb_attrs" 34 | ) as mock_get_lb_attrs: 35 | mock_get_lb_attrs.side_effect = ClientError( 36 | operation_name="DescribeLoadBalancerAttributes", 37 | error_response={ 38 | "Error": { 39 | "Code": "LoadBalancerNotFound", 40 | "Message": f"Load balancer '{lb_arn}' not found", 41 | } 42 | }, 43 | ) 44 | resources = LoadBalancerResourceSpec.scan( 45 | scan_accessor=scan_accessor, 46 | all_resource_spec_classes=ALL_RESOURCE_SPEC_CLASSES, 47 | ) 48 | self.assertEqual(resources, []) 49 | -------------------------------------------------------------------------------- /tests/unit/altimeter/aws/resource/iam/test_policy.py: -------------------------------------------------------------------------------- 1 | import boto3 2 | from botocore.exceptions import ClientError 3 | import json 4 | from unittest import TestCase 5 | from moto import mock_iam 6 | from unittest.mock import patch 7 | from altimeter.aws.resource.iam.policy import IAMPolicyResourceSpec 8 | from altimeter.aws.scan.aws_accessor import AWSAccessor 9 | from altimeter.aws.scan.settings import ALL_RESOURCE_SPEC_CLASSES 10 | 11 | 12 | class TestIAMPolicy(TestCase): 13 | @mock_iam 14 | def test_disappearing_policy_race_condition(self): 15 | account_id = "123456789012" 16 | policy_name = "foo" 17 | region_name = "us-east-1" 18 | 19 | session = boto3.Session() 20 | 21 | client = session.client("iam") 22 | 23 | policy_json = { 24 | "Version": "2012-10-17", 25 | "Statement": [{"Effect": "Allow", "Action": "logs:CreateLogGroup", "Resource": "*"}], 26 | } 27 | policy_resp = client.create_policy( 28 | PolicyName=policy_name, 29 | PolicyDocument=json.dumps(policy_json), 30 | ) 31 | policy_arn = policy_resp["Policy"]["Arn"] 32 | 33 | scan_accessor = AWSAccessor(session=session, account_id=account_id, region_name=region_name) 34 | with patch( 35 | "altimeter.aws.resource.iam.policy.IAMPolicyResourceSpec.get_policy_version_document_text" 36 | ) as mock_get_policy_version_document_text: 37 | mock_get_policy_version_document_text.side_effect = ClientError( 38 | operation_name="GetPolicyVersion", 39 | error_response={ 40 | "Error": { 41 | "Code": "NoSuchEntity", 42 | "Message": f"Policy {policy_arn} version v1 does not exist or is not attachable.", 43 | } 44 | }, 45 | ) 46 | resources = IAMPolicyResourceSpec.scan( 47 | scan_accessor=scan_accessor, 48 | all_resource_spec_classes=ALL_RESOURCE_SPEC_CLASSES, 49 | ) 50 | self.assertEqual(resources, []) 51 | -------------------------------------------------------------------------------- /altimeter/aws/resource/ec2/vpc_endpoint_service.py: -------------------------------------------------------------------------------- 1 | """Resource for VPC Endpoint Services""" 2 | from typing import Type 3 | 4 | from botocore.client import BaseClient 5 | 6 | from altimeter.aws.resource.resource_spec import ListFromAWSResult 7 | from altimeter.aws.resource.ec2 import EC2ResourceSpec 8 | from altimeter.core.graph.field.scalar_field import ScalarField, EmbeddedScalarField 9 | from altimeter.core.graph.field.list_field import ListField, AnonymousListField 10 | from altimeter.core.graph.field.tags_field import TagsField 11 | from altimeter.core.graph.schema import Schema 12 | 13 | 14 | class VpcEndpointServiceResourceSpec(EC2ResourceSpec): 15 | """Resource for VPC Endpoint Services""" 16 | 17 | type_name = "vpc-endpoint-service" 18 | schema = Schema( 19 | AnonymousListField("ServiceType", ScalarField("ServiceType")), 20 | ScalarField("ServiceName"), 21 | ScalarField("ServiceState"), 22 | ScalarField("AcceptanceRequired"), 23 | ListField("AvailabilityZones", EmbeddedScalarField()), 24 | TagsField(), 25 | ) 26 | 27 | @classmethod 28 | def list_from_aws( 29 | cls: Type["VpcEndpointServiceResourceSpec"], 30 | client: BaseClient, 31 | account_id: str, 32 | region: str, 33 | ) -> ListFromAWSResult: 34 | """Return a dict of dicts of the format: 35 | 36 | {'vpc_endpoint_svc_1_arn': {vpc_endpoint_svc_1_dict}, 37 | 'vpc_endpoint_svc_2_arn': {vpc_endpoint_svc_2_dict}, 38 | ...} 39 | 40 | Where the dicts represent results from describe_vpc_endpoint_service_configurations.""" 41 | services = {} 42 | paginator = client.get_paginator("describe_vpc_endpoint_service_configurations") 43 | for resp in paginator.paginate(): 44 | for service in resp.get("ServiceConfigurations", []): 45 | resource_arn = cls.generate_arn( 46 | account_id=account_id, region=region, resource_id=service["ServiceId"] 47 | ) 48 | services[resource_arn] = service 49 | return ListFromAWSResult(resources=services) 50 | -------------------------------------------------------------------------------- /altimeter/aws/scan/scan_plan.py: -------------------------------------------------------------------------------- 1 | """A ScanPlan defines how to scan a set of accounts.""" 2 | from typing import Optional, Tuple 3 | 4 | from altimeter.aws.auth.accessor import Accessor 5 | from altimeter.aws.resource_service_region_mapping import AWSResourceRegionMappingRepository 6 | from altimeter.core.base_model import BaseImmutableModel 7 | 8 | 9 | class AccountScanPlan(BaseImmutableModel): 10 | """An AccountScanPlan defines how to scan an account. 11 | 12 | Arguments: 13 | account_id: account id to scan 14 | regions: regions to scan 15 | aws_resource_region_mapping_repo: resource/region mapping 16 | accessor: Accessor to use to access the accounts 17 | """ 18 | 19 | account_id: str 20 | regions: Tuple[str, ...] 21 | aws_resource_region_mapping_repo: AWSResourceRegionMappingRepository 22 | accessor: Accessor 23 | 24 | 25 | class ScanPlan(BaseImmutableModel): 26 | """A ScanPlan defines how to scan a set of accounts. 27 | 28 | Arguments: 29 | account_ids: account ids to scan 30 | regions: regions to scan 31 | aws_resource_region_mapping_repo: resource/region mapping 32 | accessor: Accessor to use to access the accounts 33 | """ 34 | 35 | account_ids: Tuple[str, ...] 36 | regions: Tuple[str, ...] 37 | aws_resource_region_mapping_repo: AWSResourceRegionMappingRepository 38 | accessor: Accessor 39 | 40 | def build_account_scan_plans( 41 | self, account_id_blacklist: Optional[Tuple[str, ...]] = None 42 | ) -> Tuple[AccountScanPlan, ...]: 43 | if account_id_blacklist is None: 44 | account_id_blacklist = tuple() 45 | return tuple( 46 | [ 47 | AccountScanPlan( 48 | account_id=account_id, 49 | regions=self.regions, 50 | aws_resource_region_mapping_repo=self.aws_resource_region_mapping_repo, 51 | accessor=self.accessor, 52 | ) 53 | for account_id in self.account_ids 54 | if account_id not in account_id_blacklist 55 | ] 56 | ) 57 | -------------------------------------------------------------------------------- /altimeter/qj/middleware.py: -------------------------------------------------------------------------------- 1 | """FastAPI middlewares""" 2 | import time 3 | 4 | from starlette.exceptions import HTTPException 5 | from starlette.middleware.base import BaseHTTPMiddleware, RequestResponseEndpoint 6 | from starlette.requests import Request 7 | from starlette.responses import JSONResponse, Response 8 | from starlette.status import HTTP_500_INTERNAL_SERVER_ERROR 9 | 10 | from altimeter.core.log import Logger 11 | from altimeter.qj.log import QJLogEvents 12 | 13 | 14 | class HTTPRequestLoggingMiddleware(BaseHTTPMiddleware): 15 | """Middleware which performs HTTP request logging""" 16 | 17 | async def dispatch(self, request: Request, call_next: RequestResponseEndpoint) -> Response: 18 | """Middleware dispatch func which logs requests""" 19 | start_time = time.time() 20 | error_detail = None 21 | try: 22 | response = await call_next(request) 23 | status_code = response.status_code 24 | return response 25 | except HTTPException as ex: 26 | status_code = ex.status_code 27 | error_detail = str(ex.detail) 28 | except Exception as ex: 29 | status_code = HTTP_500_INTERNAL_SERVER_ERROR 30 | error_detail = str(ex) 31 | finally: 32 | end_time = time.time() 33 | user_agent = request.headers.get("user-agent", "unknown") 34 | xff = request.headers.get("x-forwarded_for", None) 35 | log_fields = { 36 | "event": QJLogEvents.HTTPRequest, 37 | "requestor": request.client.host, 38 | "user-agent": user_agent, 39 | "url": str(request.url), 40 | "request_time": end_time - start_time, 41 | "status_code": status_code, 42 | } 43 | if error_detail is not None: 44 | log_fields["error"] = error_detail 45 | if xff is not None: 46 | log_fields["x-forwarded-for"] = xff 47 | logger = Logger() 48 | logger.info(**log_fields) 49 | return JSONResponse({"detail": error_detail}, status_code=status_code) 50 | -------------------------------------------------------------------------------- /altimeter/aws/resource/iam/instance_profile.py: -------------------------------------------------------------------------------- 1 | """Resource for Instance Profiles""" 2 | from typing import Type 3 | 4 | from botocore.client import BaseClient 5 | 6 | from altimeter.aws.resource.resource_spec import ListFromAWSResult 7 | from altimeter.aws.resource.iam import IAMResourceSpec 8 | from altimeter.aws.resource.iam.role import IAMRoleResourceSpec 9 | from altimeter.core.graph.field.dict_field import AnonymousEmbeddedDictField 10 | from altimeter.core.graph.field.list_field import AnonymousListField 11 | from altimeter.core.graph.field.resource_link_field import ResourceLinkField 12 | from altimeter.core.graph.field.scalar_field import ScalarField 13 | from altimeter.core.graph.schema import Schema 14 | 15 | 16 | class InstanceProfileResourceSpec(IAMResourceSpec): 17 | """Resource for Instance Profiles""" 18 | 19 | type_name = "instance-profile" 20 | schema = Schema( 21 | ScalarField("InstanceProfileName", alti_key="name"), 22 | AnonymousListField( 23 | "Roles", 24 | AnonymousEmbeddedDictField( 25 | ResourceLinkField( 26 | "Arn", IAMRoleResourceSpec, value_is_id=True, alti_key="attached_role" 27 | ) 28 | ), 29 | ), 30 | ) 31 | 32 | @classmethod 33 | def list_from_aws( 34 | cls: Type["InstanceProfileResourceSpec"], client: BaseClient, account_id: str, region: str 35 | ) -> ListFromAWSResult: 36 | """Return a dict of dicts of the format: 37 | 38 | {'instance_profile_1_arn': {instance_profile_1_dict}, 39 | 'instance_profile_2_arn': {instance_profile_2_dict}, 40 | ...} 41 | 42 | Where the dicts represent results from list_instance_profiles.""" 43 | paginator = client.get_paginator("list_instance_profiles") 44 | instance_profiles = {} 45 | for resp in paginator.paginate(): 46 | for instance_profile in resp.get("InstanceProfiles", []): 47 | resource_arn = instance_profile["Arn"] 48 | instance_profiles[resource_arn] = instance_profile 49 | return ListFromAWSResult(resources=instance_profiles) 50 | -------------------------------------------------------------------------------- /altimeter/aws/resource/ec2/internet_gateway.py: -------------------------------------------------------------------------------- 1 | """Resource for Internet Gateways""" 2 | from typing import Type 3 | 4 | from botocore.client import BaseClient 5 | 6 | from altimeter.aws.resource.resource_spec import ListFromAWSResult 7 | from altimeter.aws.resource.ec2 import EC2ResourceSpec 8 | from altimeter.aws.resource.ec2.vpc import VPCResourceSpec 9 | from altimeter.core.graph.field.dict_field import EmbeddedDictField 10 | from altimeter.core.graph.field.list_field import ListField 11 | from altimeter.core.graph.field.resource_link_field import ResourceLinkField 12 | from altimeter.core.graph.field.scalar_field import ScalarField 13 | from altimeter.core.graph.field.tags_field import TagsField 14 | from altimeter.core.graph.schema import Schema 15 | 16 | 17 | class InternetGatewayResourceSpec(EC2ResourceSpec): 18 | """Resource for InternetGateways""" 19 | 20 | type_name = "internet-gateway" 21 | schema = Schema( 22 | ScalarField("OwnerId"), 23 | ListField( 24 | "Attachments", 25 | EmbeddedDictField( 26 | ScalarField("State"), 27 | ResourceLinkField("VpcId", VPCResourceSpec), 28 | ), 29 | optional=True, 30 | alti_key="attachment", 31 | ), 32 | TagsField(), 33 | ) 34 | 35 | @classmethod 36 | def list_from_aws( 37 | cls: Type["InternetGatewayResourceSpec"], client: BaseClient, account_id: str, region: str 38 | ) -> ListFromAWSResult: 39 | """Return a dict of dicts of the format: 40 | 41 | {'igw_1_arn': {igw_1_dict}, 42 | 'igw_2_arn': {igw_2_dict}, 43 | ...} 44 | 45 | Where the dicts represent results from describe_internet_gateways.""" 46 | igws = {} 47 | paginator = client.get_paginator("describe_internet_gateways") 48 | for resp in paginator.paginate(): 49 | for igw in resp["InternetGateways"]: 50 | resource_arn = cls.generate_arn( 51 | resource_id=igw["InternetGatewayId"], account_id=account_id, region=region 52 | ) 53 | igws[resource_arn] = igw 54 | return ListFromAWSResult(resources=igws) 55 | -------------------------------------------------------------------------------- /altimeter/aws/resource/ec2/flow_log.py: -------------------------------------------------------------------------------- 1 | """Resource for VPC Flow Logs""" 2 | from typing import Type 3 | 4 | from botocore.client import BaseClient 5 | 6 | from altimeter.aws.resource.resource_spec import ListFromAWSResult 7 | from altimeter.aws.resource.ec2 import EC2ResourceSpec 8 | from altimeter.aws.resource.ec2.vpc import VPCResourceSpec 9 | from altimeter.core.graph.field.resource_link_field import TransientResourceLinkField 10 | from altimeter.core.graph.field.scalar_field import ScalarField 11 | from altimeter.core.graph.schema import Schema 12 | 13 | 14 | class FlowLogResourceSpec(EC2ResourceSpec): 15 | """Resource for VPC Flow Logs""" 16 | 17 | type_name = "flow-log" 18 | schema = Schema( 19 | ScalarField("CreationTime"), 20 | ScalarField("DeliverLogsErrorMessage", optional=True), 21 | ScalarField("DeliverLogsPermissionArn", optional=True), 22 | ScalarField("DeliverLogsStatus", optional=True), 23 | ScalarField("FlowLogStatus"), 24 | ScalarField("LogGroupName", optional=True), 25 | TransientResourceLinkField("ResourceId", VPCResourceSpec, optional=True), 26 | ScalarField("TrafficType"), 27 | ScalarField("LogDestinationType"), 28 | ScalarField("LogDestination", optional=True), 29 | ScalarField("LogFormat"), 30 | ) 31 | 32 | @classmethod 33 | def list_from_aws( 34 | cls: Type["FlowLogResourceSpec"], client: BaseClient, account_id: str, region: str 35 | ) -> ListFromAWSResult: 36 | """Return a dict of dicts of the format: 37 | 38 | {'fl_1_arn': {fl_1_dict}, 39 | 'fl_2_arn': {fl_2_dict}, 40 | ...} 41 | 42 | Where the dicts represent results from describe_flow_logs.""" 43 | flow_logs = {} 44 | paginator = client.get_paginator("describe_flow_logs") 45 | for resp in paginator.paginate(): 46 | for flow_log in resp.get("FlowLogs", []): 47 | resource_arn = cls.generate_arn( 48 | account_id=account_id, region=region, resource_id=flow_log["FlowLogId"] 49 | ) 50 | flow_logs[resource_arn] = flow_log 51 | return ListFromAWSResult(resources=flow_logs) 52 | -------------------------------------------------------------------------------- /altimeter/core/resource/resource.py: -------------------------------------------------------------------------------- 1 | from typing import Dict, List, Union 2 | 3 | from rdflib import Graph, Literal, Namespace, RDF, URIRef 4 | 5 | from altimeter.core.base_model import BaseImmutableModel 6 | from altimeter.core.graph.links import LinkCollection 7 | from altimeter.core.graph.node_cache import NodeCache 8 | 9 | 10 | class Resource(BaseImmutableModel): 11 | """A Resource defines a single scanned resource which is directly translatable to a graph 12 | node. It contains an id, type name and list of Links. 13 | 14 | Args: 15 | resource_id: id of this resource 16 | type: type name of this resource 17 | link_collection: a LinkCollection representing links from this resource 18 | """ 19 | 20 | resource_id: str 21 | type: str 22 | link_collection: LinkCollection 23 | 24 | def to_rdf(self, namespace: Namespace, graph: Graph, node_cache: NodeCache) -> None: 25 | """Graph this Resource as a URIRef on a Graph. 26 | 27 | Args: 28 | namespace: RDF namespace to use for predicates and objects when graphing 29 | this resource's links 30 | graph: RDF graph 31 | node_cache: NodeCache to use for any cached URIRef lookups 32 | """ 33 | node = node_cache.setdefault(self.resource_id, URIRef(self.resource_id)) 34 | graph.add((node, RDF.type, getattr(namespace, self.type))) 35 | graph.add((node, getattr(namespace, "id"), Literal(self.resource_id))) 36 | self.link_collection.to_rdf( 37 | subj=node, namespace=namespace, graph=graph, node_cache=node_cache 38 | ) 39 | 40 | def to_lpg(self, vertices: List[Dict], edges: List[Dict]) -> None: 41 | """Graph this Resource as a dictionary into the vertices and edges lists. 42 | 43 | Args: 44 | vertices: List containing dictionaries representing a vertex 45 | edges: List containing dictionaries representing an edge 46 | """ 47 | vertex = { 48 | "~id": self.resource_id, 49 | "~label": self.type, 50 | "arn": self.resource_id, 51 | } 52 | self.link_collection.to_lpg(vertex, vertices, edges) 53 | vertices.append(vertex) 54 | -------------------------------------------------------------------------------- /altimeter/aws/resource/iam/account_password_policy.py: -------------------------------------------------------------------------------- 1 | """Resource for Account Password Policy""" 2 | from typing import Type 3 | 4 | from botocore.client import BaseClient 5 | 6 | from altimeter.aws.resource.resource_spec import ListFromAWSResult 7 | from altimeter.aws.resource.iam import IAMResourceSpec 8 | from altimeter.core.graph.field.scalar_field import ScalarField 9 | from altimeter.core.graph.schema import Schema 10 | 11 | 12 | class IAMAccountPasswordPolicyResourceSpec(IAMResourceSpec): 13 | """Resource for Account Password Policy""" 14 | 15 | DEFAULT_PASSWORD_POLICY_NAME = "default" 16 | 17 | type_name = "account-password-policy" 18 | schema = Schema( 19 | ScalarField("MinimumPasswordLength"), 20 | ScalarField("RequireSymbols"), 21 | ScalarField("RequireNumbers"), 22 | ScalarField("RequireUppercaseCharacters"), 23 | ScalarField("RequireLowercaseCharacters"), 24 | ScalarField("AllowUsersToChangePassword"), 25 | ScalarField("ExpirePasswords"), 26 | ScalarField("MaxPasswordAge", optional=True), 27 | ScalarField("PasswordReusePrevention", optional=True), 28 | ScalarField("HardExpiry", optional=True), 29 | ) 30 | 31 | @classmethod 32 | def list_from_aws( 33 | cls: Type["IAMAccountPasswordPolicyResourceSpec"], 34 | client: BaseClient, 35 | account_id: str, 36 | region: str, 37 | ) -> ListFromAWSResult: 38 | """Return a dict of dicts of the format: 39 | 40 | {'account_password_policy_1_arn': {account_password_policy_1_dict}} 41 | 42 | Where the dicts represent results from get_account_password_policy.""" 43 | password_policies = {} 44 | try: 45 | resp = client.get_account_password_policy() 46 | except client.exceptions.NoSuchEntityException: 47 | resp = {} # Indicates no policy is set for the account 48 | 49 | policy = resp.get("PasswordPolicy", {}) 50 | 51 | if policy: 52 | resource_arn = cls.generate_arn( 53 | account_id=account_id, resource_id=cls.DEFAULT_PASSWORD_POLICY_NAME 54 | ) 55 | password_policies[resource_arn] = policy 56 | return ListFromAWSResult(resources=password_policies) 57 | -------------------------------------------------------------------------------- /services/qj/alembic/alembic.ini: -------------------------------------------------------------------------------- 1 | # A generic, single database configuration. 2 | 3 | [alembic] 4 | # path to migration scripts 5 | script_location = %(here)s 6 | 7 | # template used to generate migration files 8 | # file_template = %%(rev)s_%%(slug)s 9 | 10 | # timezone to use when rendering the date 11 | # within the migration file as well as the filename. 12 | # string value is passed to dateutil.tz.gettz() 13 | # leave blank for localtime 14 | # timezone = 15 | 16 | # max length of characters to apply to the 17 | # "slug" field 18 | # truncate_slug_length = 40 19 | 20 | # set to 'true' to run the environment during 21 | # the 'revision' command, regardless of autogenerate 22 | # revision_environment = false 23 | 24 | # set to 'true' to allow .pyc and .pyo files without 25 | # a source .py file to be detected as revisions in the 26 | # versions/ directory 27 | # sourceless = false 28 | 29 | # version location specification; this defaults 30 | # to alembic/versions. When using multiple version 31 | # directories, initial revisions must be specified with --version-path 32 | version_locations = %(here)s/versions 33 | # the output encoding used when revision files 34 | # are written from script.py.mako 35 | # output_encoding = utf-8 36 | 37 | [post_write_hooks] 38 | # post_write_hooks defines scripts or Python functions that are run 39 | # on newly generated revision scripts. See the documentation for further 40 | # detail and examples 41 | 42 | # format using "black" - use the console_scripts runner, against the "black" entrypoint 43 | # hooks=black 44 | # black.type=console_scripts 45 | # black.entrypoint=black 46 | # black.options=-l 79 47 | 48 | # Logging configuration 49 | [loggers] 50 | keys = root,sqlalchemy,alembic 51 | 52 | [handlers] 53 | keys = console 54 | 55 | [formatters] 56 | keys = generic 57 | 58 | [logger_root] 59 | level = WARN 60 | handlers = console 61 | qualname = 62 | 63 | [logger_sqlalchemy] 64 | level = WARN 65 | handlers = 66 | qualname = sqlalchemy.engine 67 | 68 | [logger_alembic] 69 | level = INFO 70 | handlers = 71 | qualname = alembic 72 | 73 | [handler_console] 74 | class = StreamHandler 75 | args = (sys.stderr,) 76 | level = NOTSET 77 | formatter = generic 78 | 79 | [formatter_generic] 80 | format = %(levelname)-5.5s [%(name)s] %(message)s 81 | datefmt = %H:%M:%S 82 | -------------------------------------------------------------------------------- /tests/unit/altimeter/aws/resource/rds/test_instance.py: -------------------------------------------------------------------------------- 1 | from unittest import TestCase 2 | from unittest.mock import patch 3 | 4 | import boto3 5 | from botocore.exceptions import ClientError 6 | from moto import mock_rds 7 | 8 | from altimeter.aws.resource.rds.instance import RDSInstanceResourceSpec 9 | from altimeter.aws.scan.aws_accessor import AWSAccessor 10 | from altimeter.aws.scan.settings import ALL_RESOURCE_SPEC_CLASSES 11 | 12 | 13 | class TestRDSInstanceResourceSpec(TestCase): 14 | @mock_rds 15 | def test_disappearing_instance_race_condition(self): 16 | account_id = "123456789012" 17 | region_name = "us-east-1" 18 | instance_name = "foo" 19 | 20 | session = boto3.Session() 21 | client = session.client("rds", region_name=region_name) 22 | 23 | client.create_db_instance( 24 | DBInstanceIdentifier=instance_name, 25 | Engine="postgres", 26 | DBName=instance_name, 27 | DBInstanceClass="db.m1.small", 28 | MasterUsername="root", 29 | MasterUserPassword="hunter2", 30 | ) 31 | 32 | scan_accessor = AWSAccessor(session=session, account_id=account_id, region_name=region_name) 33 | with patch( 34 | "altimeter.aws.resource.rds.instance.RDSInstanceResourceSpec.get_instance_tags" 35 | ) as mock_get_instance_tags: 36 | with patch( 37 | "altimeter.aws.resource.rds.instance.RDSInstanceResourceSpec.set_automated_backups" 38 | ) as mock_set_automated_backups: 39 | mock_set_automated_backups.return_value = None 40 | mock_get_instance_tags.side_effect = ClientError( 41 | operation_name="ListTagsForResource", 42 | error_response={ 43 | "Error": { 44 | "Code": "DBInstanceNotFound", 45 | "Message": f"Could not find a DB Instance matching the resource name: '{instance_name}'", 46 | } 47 | }, 48 | ) 49 | resources = RDSInstanceResourceSpec.scan( 50 | scan_accessor=scan_accessor, 51 | all_resource_spec_classes=ALL_RESOURCE_SPEC_CLASSES, 52 | ) 53 | self.assertEqual(resources, []) 54 | -------------------------------------------------------------------------------- /tests/unit/altimeter/aws/resource/test_resource_spec.py: -------------------------------------------------------------------------------- 1 | import inspect 2 | from typing import Any, Type 3 | from unittest import TestCase 4 | 5 | from altimeter.aws.resource.resource_spec import ListFromAWSResult, AWSResourceSpec 6 | from altimeter.aws.scan.aws_accessor import AWSAccessor 7 | 8 | 9 | class TestAWSResourceSpecSubClassing(TestCase): 10 | def test_valid_concrete(self): 11 | class C(AWSResourceSpec): 12 | type_name = "t" 13 | service_name = "c" 14 | 15 | def list_from_aws( 16 | cls: Type["C"], client, account_id: str, region: str 17 | ) -> ListFromAWSResult: 18 | pass 19 | 20 | self.assertFalse(inspect.isabstract(C)) 21 | 22 | def test_invalid_concrete(self): 23 | with self.assertRaises(TypeError): 24 | 25 | class C(AWSResourceSpec): 26 | def list_from_aws( 27 | cls: Type["C"], client, account_id: str, region: str 28 | ) -> ListFromAWSResult: 29 | pass 30 | 31 | def test_valid_abstract(self): 32 | class C(AWSResourceSpec): 33 | pass 34 | 35 | self.assertTrue(inspect.isabstract(C)) 36 | 37 | 38 | class TestSkipResourceScanFlag(TestCase): 39 | class TestAWSAccessor(AWSAccessor): 40 | def client(self, service_name: str) -> Any: 41 | return None 42 | 43 | def test_true(self): 44 | class TestResource(AWSResourceSpec): 45 | type_name = "t" 46 | service_name = "fakesvc" 47 | 48 | @classmethod 49 | def skip_resource_scan( 50 | cls: Type["TestResource"], client, account_id: str, region: str 51 | ) -> bool: 52 | return True 53 | 54 | accessor = TestSkipResourceScanFlag.TestAWSAccessor(None, None, None) 55 | TestResource.scan(scan_accessor=accessor, all_resource_spec_classes=[TestResource]) 56 | 57 | def test_false(self): 58 | class TestResource(AWSResourceSpec): 59 | type_name = "t" 60 | service_name = "fakesvc" 61 | 62 | accessor = TestSkipResourceScanFlag.TestAWSAccessor(None, None, None) 63 | with self.assertRaises(AttributeError): 64 | TestResource.scan(scan_accessor=accessor, all_resource_spec_classes=[TestResource]) 65 | -------------------------------------------------------------------------------- /tests/unit/altimeter/aws/resource/test_account.py: -------------------------------------------------------------------------------- 1 | from unittest import TestCase 2 | 3 | import boto3 4 | from moto import mock_sts 5 | 6 | from altimeter.aws.resource.account import AccountResourceSpec 7 | from altimeter.aws.scan.aws_accessor import AWSAccessor 8 | from altimeter.aws.scan.settings import ALL_RESOURCE_SPEC_CLASSES 9 | from altimeter.core.graph.links import ( 10 | LinkCollection, 11 | SimpleLink, 12 | ) 13 | from altimeter.core.resource.resource import Resource 14 | 15 | 16 | class TestEBSVolumeResourceSpec(TestCase): 17 | @mock_sts 18 | def test_scan(self): 19 | account_id = "123456789012" 20 | region_name = "us-east-1" 21 | 22 | session = boto3.Session() 23 | scan_accessor = AWSAccessor(session=session, account_id=account_id, region_name=region_name) 24 | resources = AccountResourceSpec.scan( 25 | scan_accessor=scan_accessor, 26 | all_resource_spec_classes=ALL_RESOURCE_SPEC_CLASSES, 27 | ) 28 | 29 | expected_resources = [ 30 | Resource( 31 | resource_id="arn:aws::::account/123456789012", 32 | type="aws:account", 33 | link_collection=LinkCollection( 34 | simple_links=(SimpleLink(pred="account_id", obj="123456789012"),), 35 | ), 36 | ) 37 | ] 38 | 39 | self.assertEqual(resources, expected_resources) 40 | 41 | @mock_sts 42 | def test_detect_account_id_session_mismatch(self): 43 | account_id = "234567890121" 44 | region_name = "us-east-1" 45 | 46 | session = boto3.Session() 47 | scan_accessor = AWSAccessor(session=session, account_id=account_id, region_name=region_name) 48 | with self.assertRaises(ValueError): 49 | AccountResourceSpec.scan( 50 | scan_accessor=scan_accessor, 51 | all_resource_spec_classes=ALL_RESOURCE_SPEC_CLASSES, 52 | ) 53 | 54 | def test_generate_arn(self): 55 | account_id = "234567890121" 56 | region_name = "us-east-1" 57 | resource_id = "123456789012" 58 | 59 | expected_arn = "arn:aws::::account/123456789012" 60 | arn = AccountResourceSpec.generate_arn( 61 | account_id=account_id, region=region_name, resource_id=resource_id 62 | ) 63 | self.assertEqual(arn, expected_arn) 64 | -------------------------------------------------------------------------------- /altimeter/qj/models/result_set.py: -------------------------------------------------------------------------------- 1 | """Table definitions for Result and ResultSet""" 2 | import uuid 3 | 4 | from sqlalchemy import Column, Index, Integer, ForeignKey, DateTime, Text 5 | from sqlalchemy.dialects.postgresql import UUID, JSONB 6 | from sqlalchemy.orm import relationship 7 | 8 | from altimeter.qj import schemas 9 | from altimeter.qj.db.base_class import BASE 10 | from altimeter.qj.models.job import Job 11 | 12 | 13 | # pylint: disable=too-few-public-methods 14 | class ResultSet(BASE): 15 | """ResultSet table definition. A ResultSet represents all results for a run of a given 16 | JobVersion.""" 17 | 18 | __tablename__ = "result_set" 19 | 20 | id = Column(Integer, primary_key=True) 21 | result_set_id = Column(UUID(as_uuid=True), default=uuid.uuid4, nullable=False, unique=True) 22 | job_id = Column("job_id", ForeignKey(Job.id, ondelete="CASCADE"), nullable=False) 23 | job = relationship(Job) 24 | created = Column(DateTime, nullable=False) 25 | graph_spec = Column(JSONB, nullable=False) 26 | results = relationship("Result", passive_deletes=True) 27 | 28 | __table_args__ = ( 29 | Index( 30 | "result_set_job_id_idx", 31 | job_id, 32 | ), 33 | Index( 34 | "result_set_created_idx", 35 | created, 36 | ), 37 | ) 38 | 39 | def to_api_schema(self) -> schemas.ResultSet: 40 | """Build a ResultSet pydantic model from this result set""" 41 | base_rs = schemas.ResultSet.from_orm(self) 42 | return schemas.ResultSet.from_orm(base_rs) 43 | 44 | 45 | class Result(BASE): 46 | """Result table definition. A Result represents the single result of a Job run""" 47 | 48 | __tablename__ = "result" 49 | 50 | result_set_id = Column( 51 | "result_set_id", 52 | ForeignKey(ResultSet.id, ondelete="CASCADE"), 53 | nullable=False, 54 | ) 55 | result_set = relationship(ResultSet) 56 | account_id = Column(Text, nullable=False) 57 | result_id = Column( 58 | UUID(as_uuid=True), default=uuid.uuid4, nullable=False, unique=True, primary_key=True 59 | ) 60 | result = Column(JSONB, nullable=False) 61 | 62 | __table_args__ = ( 63 | Index( 64 | "result_result_set_id_idx", 65 | result_set_id, 66 | ), 67 | Index("result_account_id_idx", account_id), 68 | ) 69 | -------------------------------------------------------------------------------- /altimeter/qj/log.py: -------------------------------------------------------------------------------- 1 | """LogEvent for QJ events.""" 2 | from dataclasses import dataclass 3 | import logging 4 | 5 | from altimeter.core.log import BaseLogEvent, EventName 6 | 7 | # Clear handlers 8 | _ROOT = logging.getLogger() 9 | for handler in _ROOT.handlers: 10 | _ROOT.removeHandler(handler) 11 | 12 | 13 | @dataclass(frozen=True) 14 | class QJLogEvents(BaseLogEvent): 15 | """QJ Log event names""" 16 | 17 | # pylint: disable=invalid-name 18 | 19 | # General 20 | InitConfig: EventName 21 | 22 | # Executor 23 | GetJobs: EventName 24 | ScheduleJob: EventName 25 | 26 | # Pruner 27 | DeleteStart: EventName 28 | DeleteEnd: EventName 29 | 30 | # Query 31 | InitJob: EventName 32 | RunQueryStart: EventName 33 | RunQueryEnd: EventName 34 | RunQueryError: EventName 35 | PublishResultSetToQJAPIStart: EventName 36 | PublishResultSetToQJAPIEnd: EventName 37 | 38 | # Publish 39 | GetLatestResultSetStart: EventName 40 | GetLatestResultSetEnd: EventName 41 | PublishResultSetToTableauStart: EventName 42 | PublishResultSetToTableauEnd: EventName 43 | 44 | # Notifications 45 | NotifyNewResultsStart: EventName 46 | NotifyNewResultsEnd: EventName 47 | 48 | # CRUD Jobs 49 | CreateJob: EventName 50 | DeleteJob: EventName 51 | GetActiveJob: EventName 52 | GetJobVersion: EventName 53 | GetJobVersions: EventName 54 | UpdateJob: EventName 55 | CreateView: EventName 56 | DropView: EventName 57 | 58 | # CRUD Results 59 | CreateResultSet: EventName 60 | DeleteExpiredResultSets: EventName 61 | GetExpiredResultSets: EventName 62 | GetResultSet: EventName 63 | GetLatestResultSetForActiveJob: EventName 64 | 65 | # Remediations 66 | ProcessResult: EventName 67 | RemediationInit: EventName 68 | StaleResultSet: EventName 69 | JobHasNoRemediator: EventName 70 | InvokeResultRemediationLambdaStart: EventName 71 | InvokeResultRemediationLambdaEnd: EventName 72 | InvokeResultRemediationLambdaError: EventName 73 | ResultRemediationLambdaRunError: EventName 74 | ResultRemediationStart: EventName 75 | ResultRemediationFailed: EventName 76 | ResultRemediationSuccessful: EventName 77 | ResultSetRemediationFailed: EventName 78 | ResultSetRemediationSuccessful: EventName 79 | 80 | # HTTP 81 | APIError: EventName 82 | HTTPRequest: EventName 83 | -------------------------------------------------------------------------------- /services/qj/alembic/env.py: -------------------------------------------------------------------------------- 1 | # pylint: skip-file 2 | # type: ignore 3 | from logging.config import fileConfig 4 | import os 5 | 6 | from alembic import context 7 | from sqlalchemy import create_engine 8 | 9 | from altimeter.qj.db.base import BASE 10 | 11 | # this is the Alembic Config object, which provides 12 | # access to the values within the .ini file in use. 13 | config = context.config 14 | 15 | # Interpret the config file for Python logging. 16 | # This line sets up loggers basically. 17 | fileConfig(config.config_file_name) 18 | 19 | # add your model's MetaData object here 20 | # for 'autogenerate' support 21 | # from myapp import mymodel 22 | # target_metadata = mymodel.Base.metadata 23 | target_metadata = BASE.metadata 24 | 25 | # other values from the config, defined by the needs of env.py, 26 | # can be acquired: 27 | # my_important_option = config.get_main_option("my_important_option") 28 | # ... etc. 29 | 30 | 31 | def get_db_url(): 32 | url = os.getenv("SQLALCHEMY_URL") 33 | if url: 34 | return url 35 | raise Exception("Missing required env var 'SQLALCHEMY_URL'") 36 | 37 | 38 | def run_migrations_offline(): 39 | """Run migrations in 'offline' mode. 40 | 41 | This configures the context with just a URL 42 | and not an Engine, though an Engine is acceptable 43 | here as well. By skipping the Engine creation 44 | we don't even need a DBAPI to be available. 45 | 46 | Calls to context.execute() here emit the given string to the 47 | script output. 48 | 49 | """ 50 | context.configure( 51 | url=get_db_url(), 52 | target_metadata=target_metadata, 53 | literal_binds=True, 54 | dialect_opts={"paramstyle": "named"}, 55 | ) 56 | 57 | with context.begin_transaction(): 58 | context.run_migrations() 59 | 60 | 61 | def run_migrations_online(): 62 | """Run migrations in 'online' mode. 63 | 64 | In this scenario we need to create an Engine 65 | and associate a connection with the context. 66 | 67 | """ 68 | connectable = create_engine(get_db_url()) 69 | 70 | with connectable.connect() as connection: 71 | context.configure(connection=connection, target_metadata=target_metadata) 72 | 73 | with context.begin_transaction(): 74 | context.run_migrations() 75 | 76 | 77 | if context.is_offline_mode(): 78 | run_migrations_offline() 79 | else: 80 | run_migrations_online() 81 | -------------------------------------------------------------------------------- /bin/sfn_init.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | """Initialization StepFunction Lambda 3 | This is the first step of Altimeter as a step function. This reads the Altimeter config and outputs an InitOutput 4 | which contains the data required for subsequent AccountScans via sfn_scan_account.""" 5 | import logging 6 | from typing import Any, Dict, Tuple 7 | 8 | import boto3 9 | from pydantic import BaseSettings 10 | 11 | from altimeter.aws.aws2n import generate_scan_id 12 | from altimeter.aws.resource_service_region_mapping import ( 13 | build_aws_resource_region_mapping_repo, 14 | AWSResourceRegionMappingRepository, 15 | ) 16 | from altimeter.aws.scan.scan import get_sub_account_ids 17 | from altimeter.core.base_model import BaseImmutableModel 18 | from altimeter.core.config import AWSConfig 19 | 20 | 21 | class Settings(BaseSettings): 22 | config_path: str 23 | 24 | 25 | class InitOutput(BaseImmutableModel): 26 | config: AWSConfig 27 | scan_id: str 28 | account_ids: Tuple[str, ...] 29 | aws_resource_region_mapping_repo: AWSResourceRegionMappingRepository 30 | 31 | 32 | def lambda_handler(_: Dict[str, Any], __: Any) -> Dict[str, Any]: 33 | """Lambda entrypoint""" 34 | root = logging.getLogger() 35 | if root.handlers: 36 | for handler in root.handlers: 37 | root.removeHandler(handler) 38 | settings = Settings() 39 | config = AWSConfig.from_path(path=settings.config_path) 40 | scan_id = generate_scan_id() 41 | aws_resource_region_mapping_repo = build_aws_resource_region_mapping_repo( 42 | global_region_whitelist=config.scan.regions, 43 | preferred_account_scan_regions=config.scan.preferred_account_scan_regions, 44 | services_regions_json_url=config.services_regions_json_url, 45 | ) 46 | if config.scan.accounts: 47 | scan_account_ids = config.scan.accounts 48 | else: 49 | sts_client = boto3.client("sts") 50 | scan_account_id = sts_client.get_caller_identity()["Account"] 51 | scan_account_ids = (scan_account_id,) 52 | if config.scan.scan_sub_accounts: 53 | account_ids = get_sub_account_ids(scan_account_ids, config.accessor) 54 | else: 55 | account_ids = scan_account_ids 56 | return InitOutput( 57 | config=config, 58 | scan_id=scan_id, 59 | account_ids=account_ids, 60 | aws_resource_region_mapping_repo=aws_resource_region_mapping_repo, 61 | ).dict() 62 | -------------------------------------------------------------------------------- /tests/unit/altimeter/aws/resource/ec2/test_volume.py: -------------------------------------------------------------------------------- 1 | from unittest import TestCase 2 | 3 | import boto3 4 | from moto import mock_ec2 5 | 6 | from altimeter.aws.resource.ec2.volume import EBSVolumeResourceSpec 7 | from altimeter.aws.scan.aws_accessor import AWSAccessor 8 | from altimeter.aws.scan.settings import ALL_RESOURCE_SPEC_CLASSES 9 | from altimeter.core.graph.links import LinkCollection, ResourceLink, SimpleLink 10 | from altimeter.core.resource.resource import Resource 11 | 12 | 13 | class TestEBSVolumeResourceSpec(TestCase): 14 | @mock_ec2 15 | def test_scan(self): 16 | account_id = "123456789012" 17 | region_name = "us-east-1" 18 | 19 | session = boto3.Session() 20 | 21 | ec2_client = session.client("ec2", region_name=region_name) 22 | resp = ec2_client.create_volume(Size=1, AvailabilityZone="us-east-1a") 23 | create_time = resp["CreateTime"] 24 | created_volume_id = resp["VolumeId"] 25 | created_volume_arn = f"arn:aws:ec2:us-east-1:123456789012:volume/{created_volume_id}" 26 | 27 | scan_accessor = AWSAccessor(session=session, account_id=account_id, region_name=region_name) 28 | resources = EBSVolumeResourceSpec.scan( 29 | scan_accessor=scan_accessor, 30 | all_resource_spec_classes=ALL_RESOURCE_SPEC_CLASSES, 31 | ) 32 | 33 | expected_resources = [ 34 | Resource( 35 | resource_id=created_volume_arn, 36 | type="aws:ec2:volume", 37 | link_collection=LinkCollection( 38 | simple_links=( 39 | SimpleLink(pred="availability_zone", obj="us-east-1a"), 40 | SimpleLink(pred="create_time", obj=create_time), 41 | SimpleLink(pred="size", obj=True), 42 | SimpleLink(pred="state", obj="available"), 43 | SimpleLink(pred="volume_type", obj="gp2"), 44 | SimpleLink(pred="encrypted", obj=False), 45 | ), 46 | resource_links=( 47 | ResourceLink(pred="account", obj="arn:aws::::account/123456789012"), 48 | ResourceLink(pred="region", obj="arn:aws:::123456789012:region/us-east-1"), 49 | ), 50 | ), 51 | ) 52 | ] 53 | self.assertEqual(resources, expected_resources) 54 | -------------------------------------------------------------------------------- /altimeter/aws/resource/route53/hosted_zone.py: -------------------------------------------------------------------------------- 1 | """Resource for HostedZones""" 2 | from typing import Type 3 | 4 | from botocore.client import BaseClient 5 | 6 | from altimeter.aws.resource.resource_spec import ListFromAWSResult 7 | from altimeter.aws.resource.route53 import Route53ResourceSpec 8 | from altimeter.core.graph.field.dict_field import EmbeddedDictField 9 | from altimeter.core.graph.field.list_field import ListField 10 | from altimeter.core.graph.field.scalar_field import ScalarField 11 | from altimeter.core.graph.schema import Schema 12 | 13 | 14 | class HostedZoneResourceSpec(Route53ResourceSpec): 15 | """Resource for S3 Buckets""" 16 | 17 | type_name = "hostedzone" 18 | schema = Schema( 19 | ScalarField("Name"), 20 | ListField( 21 | "ResourceRecordSets", 22 | EmbeddedDictField( 23 | ScalarField("Name"), ScalarField("Type"), ScalarField("TTL", optional=True) 24 | ), 25 | optional=True, 26 | alti_key="resource_record_set", 27 | ), 28 | ) 29 | 30 | @classmethod 31 | def list_from_aws( 32 | cls: Type["HostedZoneResourceSpec"], client: BaseClient, account_id: str, region: str 33 | ) -> ListFromAWSResult: 34 | """Return a dict of dicts of the format: 35 | 36 | {'hosted_zone_1_arn': {hosted_zone_1_dict}, 37 | 'hosted_zone_2_arn': {hosted_zone_2_dict}, 38 | ...} 39 | 40 | Where the dicts represent results from list_hosted_zones.""" 41 | hosted_zones = {} 42 | paginator = client.get_paginator("list_hosted_zones") 43 | for resp in paginator.paginate(): 44 | for hosted_zone in resp.get("HostedZones", []): 45 | hosted_zone_id = hosted_zone["Id"].split("/")[-1] 46 | resource_arn = cls.generate_arn(resource_id=hosted_zone_id, account_id=account_id) 47 | record_sets_paginator = client.get_paginator("list_resource_record_sets") 48 | zone_resource_record_sets = [] 49 | for record_sets_resp in record_sets_paginator.paginate(HostedZoneId=hosted_zone_id): 50 | zone_resource_record_sets += record_sets_resp.get("ResourceRecordSets", []) 51 | hosted_zone["ResourceRecordSets"] = zone_resource_record_sets 52 | hosted_zones[resource_arn] = hosted_zone 53 | return ListFromAWSResult(resources=hosted_zones) 54 | -------------------------------------------------------------------------------- /bin/runquery.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | """Run a query against the latest graph for a given name and optional version. 3 | This finds the Neptune instance details based on naming conventions - instance identifier 4 | should begin with 'alti-'. Discover behavior can be overridden by specifying the endpoint 5 | information in argument --neptune_endpoint 6 | """ 7 | import argparse 8 | import sys 9 | from typing import List, Optional 10 | 11 | from altimeter.core.neptune.client import ( 12 | AltimeterNeptuneClient, 13 | discover_neptune_endpoint, 14 | NeptuneEndpoint, 15 | ) 16 | 17 | 18 | def main(argv: Optional[List[str]] = None) -> int: 19 | if argv is None: 20 | argv = sys.argv[1:] 21 | parser = argparse.ArgumentParser() 22 | parser.add_argument("query_file", type=str) 23 | parser.add_argument("--graph_names", type=str, default=["alti"], nargs="+") 24 | parser.add_argument("--historic_graph_names", type=str, nargs="+") 25 | parser.add_argument("--max_age_min", type=int, default=1440) 26 | parser.add_argument("--raw", default=False, action="store_true") 27 | parser.add_argument("--neptune_endpoint", help="Neptune endpoint specified as host:port:region") 28 | args_ns = parser.parse_args(argv) 29 | 30 | with open(args_ns.query_file, "r") as query_fp: 31 | query = query_fp.read() 32 | 33 | if args_ns.neptune_endpoint is not None: 34 | try: 35 | host, port_str, region = args_ns.neptune_endpoint.split(":") 36 | port: int = int(port_str) 37 | except ValueError: 38 | print(f"neptune_endpoint should be a string formatted as host:port:region") 39 | return 1 40 | endpoint = NeptuneEndpoint(host=host, port=port, region=region) 41 | else: 42 | endpoint = discover_neptune_endpoint() 43 | client = AltimeterNeptuneClient(max_age_min=args_ns.max_age_min, neptune_endpoint=endpoint) 44 | 45 | if args_ns.historic_graph_names: 46 | results = client.run_historic_query(graph_names=args_ns.historic_graph_names, query=query) 47 | print(results.to_csv(), end="") 48 | elif args_ns.raw: 49 | raw_results = client.run_raw_query(query=query) 50 | print(raw_results.to_csv(), end="") 51 | else: 52 | results = client.run_query(graph_names=args_ns.graph_names, query=query) 53 | print(results.to_csv(), end="") 54 | return 0 55 | 56 | 57 | if __name__ == "__main__": 58 | sys.exit(main()) 59 | -------------------------------------------------------------------------------- /tests/unit/altimeter/core/artifact_io/test_writer.py: -------------------------------------------------------------------------------- 1 | import json 2 | import os 3 | from pathlib import Path 4 | import tempfile 5 | import unittest 6 | 7 | import boto3 8 | import moto 9 | from pydantic import BaseModel 10 | 11 | from altimeter.core.artifact_io.writer import ArtifactWriter, FileArtifactWriter, S3ArtifactWriter 12 | 13 | 14 | class TestArtifactWriter(unittest.TestCase): 15 | def test_from_artifact_path_s3(self): 16 | writer = ArtifactWriter.from_artifact_path( 17 | artifact_path="s3://bucket", scan_id="test-scan-id" 18 | ) 19 | self.assertIsInstance(writer, S3ArtifactWriter) 20 | 21 | def test_from_artifact_path_filepath(self): 22 | writer = ArtifactWriter.from_artifact_path( 23 | artifact_path="/file/path", scan_id="test-scan-id" 24 | ) 25 | self.assertIsInstance(writer, FileArtifactWriter) 26 | 27 | 28 | class TestFileArtifactWriter(unittest.TestCase): 29 | def test_with_valid_data(self): 30 | scan_id = "test-scan-id" 31 | 32 | class TestModel(BaseModel): 33 | n: int 34 | s: str 35 | 36 | t_m = TestModel(n=123, s="abc") 37 | with tempfile.TemporaryDirectory() as temp_dir: 38 | artifact_writer = FileArtifactWriter(scan_id=scan_id, output_dir=Path(temp_dir)) 39 | artifact_writer.write_json("test_name", t_m) 40 | path = os.path.join(temp_dir, "test-scan-id", "test_name.json") 41 | with open(path, "r") as fp: 42 | written_data = json.load(fp) 43 | expected_data = {"n": 123, "s": "abc"} 44 | self.assertDictEqual(written_data, expected_data) 45 | 46 | 47 | class TestS3ArtifactWriter(unittest.TestCase): 48 | @moto.mock_s3 49 | def test_with_valid_object(self): 50 | scan_id = "test-scan-id" 51 | 52 | class TestModel(BaseModel): 53 | n: int 54 | s: str 55 | 56 | t_m = TestModel(n=123, s="abc") 57 | s3_client = boto3.Session().client("s3") 58 | s3_client.create_bucket(Bucket="test_bucket") 59 | artifact_writer = S3ArtifactWriter(bucket="test_bucket", key_prefix=scan_id) 60 | artifact_writer.write_json("test_name", t_m) 61 | resp = s3_client.get_object(Bucket="test_bucket", Key="test-scan-id/test_name.json") 62 | written_data = json.load(resp["Body"]) 63 | expected_data = {"n": 123, "s": "abc"} 64 | self.assertDictEqual(written_data, expected_data) 65 | -------------------------------------------------------------------------------- /altimeter/core/graph/graph_spec.py: -------------------------------------------------------------------------------- 1 | """A GraphSpec contains a specification to scan and create a graph.""" 2 | from typing import Any, List, Tuple, Type 3 | 4 | from altimeter.core.log import Logger 5 | from altimeter.core.log_events import LogEvent 6 | 7 | from altimeter.core.resource.resource import Resource 8 | from altimeter.core.resource.resource_spec import ResourceSpec 9 | 10 | 11 | class GraphSpec: 12 | """A GraphSpec contains a specification to scan and create a graph. It contains a set of 13 | ResourceSpec classes defining what to scan and a scan_accessor object defining how to scan. 14 | 15 | Args: 16 | name: graph name 17 | version: graph version 18 | resource_spec_classes: tuple of ResourceSpec classes 19 | scan_accessor: object which is passed to ResourceSpec.scan for each ResourceSpec in 20 | resource_spec_classes. It should provide methods which ResourceSpec.scan 21 | can use to access whatever API the ResourceSpec needs to access. 22 | """ 23 | 24 | def __init__( 25 | self, 26 | name: str, 27 | version: str, 28 | resource_spec_classes: Tuple[Type[ResourceSpec], ...], 29 | all_resource_spec_classes: Tuple[Type[ResourceSpec], ...], 30 | scan_accessor: Any, 31 | ): 32 | self.name = name 33 | self.version = version 34 | self.resource_spec_classes = resource_spec_classes 35 | self.all_resource_spec_classes = all_resource_spec_classes 36 | self.scan_accessor = scan_accessor 37 | 38 | def scan(self) -> List[Resource]: 39 | """Perform a scan on all of the resource classes in this GraphSpec and return 40 | a list of Resource objects. 41 | 42 | Returns: 43 | List of Resource objects 44 | """ 45 | resources: List[Resource] = [] 46 | logger = Logger() 47 | for resource_spec_class in self.resource_spec_classes: 48 | with logger.bind(resource_type=str(resource_spec_class.type_name)): 49 | logger.debug(event=LogEvent.ScanResourceTypeStart) 50 | scanned_resources = resource_spec_class.scan( 51 | scan_accessor=self.scan_accessor, 52 | all_resource_spec_classes=self.all_resource_spec_classes, 53 | ) 54 | resources += scanned_resources 55 | logger.debug(event=LogEvent.ScanResourceTypeEnd) 56 | return resources 57 | -------------------------------------------------------------------------------- /bin/scan_resource.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | """Tool to run a single Resource class scan. Useful during developing Resource classes 3 | and their Schemas. Run without usage for details.""" 4 | import json 5 | import sys 6 | from typing import List, Optional, Type 7 | 8 | import boto3 9 | 10 | from altimeter.aws.resource.resource_spec import AWSResourceSpec 11 | from altimeter.core.json_encoder import json_encoder 12 | from altimeter.aws.scan.aws_accessor import AWSAccessor 13 | from altimeter.aws.scan.settings import DEFAULT_RESOURCE_SPEC_CLASSES 14 | 15 | 16 | def main(argv: Optional[List[str]] = None) -> int: 17 | import argparse 18 | 19 | if argv is None: 20 | argv = sys.argv[1:] 21 | parser = argparse.ArgumentParser() 22 | parser.add_argument( 23 | "resource_spec_class", 24 | type=str, 25 | help="Name of class in altimeter.aws.scan.settings.RESOURCE_SPEC_CLASSES to scan", 26 | ) 27 | parser.add_argument("region", type=str, help="AWS region name to scan") 28 | 29 | args_ns = parser.parse_args(argv) 30 | resource_spec_class_name = args_ns.resource_spec_class 31 | region = args_ns.region 32 | 33 | resource_spec_class: Optional[Type[AWSResourceSpec]] = None 34 | for cls in DEFAULT_RESOURCE_SPEC_CLASSES: 35 | if cls.__name__ == resource_spec_class_name: 36 | resource_spec_class = cls 37 | break 38 | if resource_spec_class is None: 39 | print( 40 | ( 41 | f"Unable to find a class named {resource_spec_class_name} in " 42 | f"altimeter.aws.scan.settings.RESOURCE_SPEC_CLASSES: {DEFAULT_RESOURCE_SPEC_CLASSES}." 43 | ) 44 | ) 45 | return 1 46 | 47 | session = boto3.Session(region_name=region) 48 | sts_client = session.client("sts") 49 | account_id = sts_client.get_caller_identity()["Account"] 50 | aws_accessor = AWSAccessor(session=session, account_id=account_id, region_name=region) 51 | resource_scan_result = resource_spec_class.scan( 52 | scan_accessor=aws_accessor, 53 | all_resource_spec_classes=DEFAULT_RESOURCE_SPEC_CLASSES, 54 | ) 55 | resource_dicts = [] 56 | for resource in resource_scan_result: 57 | resource_dicts.append(resource.dict()) 58 | resource_scan_result_json = json.dumps(resource_dicts, indent=2, default=json_encoder) 59 | print(resource_scan_result_json) 60 | return 0 61 | 62 | 63 | if __name__ == "__main__": 64 | sys.exit(main()) 65 | -------------------------------------------------------------------------------- /tests/unit/altimeter/aws/resource/iam/test_account_password_policy.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | 3 | from altimeter.core.graph.links import LinkCollection, SimpleLink 4 | from altimeter.core.resource.resource import Resource 5 | from altimeter.aws.resource.iam.account_password_policy import IAMAccountPasswordPolicyResourceSpec 6 | 7 | 8 | class TestAccountPasswordPolicyResourceSpec(unittest.TestCase): 9 | def test_schema_parse(self): 10 | resource_arn = "arn:aws:iam:us-west-2:111122223333:account-password-policy/default" 11 | aws_resource_dict = { 12 | "MinimumPasswordLength": 12, 13 | "RequireSymbols": True, 14 | "RequireNumbers": True, 15 | "RequireUppercaseCharacters": True, 16 | "RequireLowercaseCharacters": True, 17 | "AllowUsersToChangePassword": True, 18 | "ExpirePasswords": True, 19 | "MaxPasswordAge": 90, 20 | "PasswordReusePrevention": 5, 21 | "HardExpiry": True, 22 | } 23 | 24 | link_collection = IAMAccountPasswordPolicyResourceSpec.schema.parse( 25 | data=aws_resource_dict, context={"account_id": "111122223333", "region": "us-west-2"} 26 | ) 27 | resource = Resource( 28 | resource_id=resource_arn, 29 | type=IAMAccountPasswordPolicyResourceSpec.type_name, 30 | link_collection=link_collection, 31 | ) 32 | 33 | expected_resource = Resource( 34 | resource_id="arn:aws:iam:us-west-2:111122223333:account-password-policy/default", 35 | type="account-password-policy", 36 | link_collection=LinkCollection( 37 | simple_links=( 38 | SimpleLink(pred="minimum_password_length", obj=12), 39 | SimpleLink(pred="require_symbols", obj=True), 40 | SimpleLink(pred="require_numbers", obj=True), 41 | SimpleLink(pred="require_uppercase_characters", obj=True), 42 | SimpleLink(pred="require_lowercase_characters", obj=True), 43 | SimpleLink(pred="allow_users_to_change_password", obj=True), 44 | SimpleLink(pred="expire_passwords", obj=True), 45 | SimpleLink(pred="max_password_age", obj=90), 46 | SimpleLink(pred="password_reuse_prevention", obj=5), 47 | SimpleLink(pred="hard_expiry", obj=True), 48 | ) 49 | ), 50 | ) 51 | self.assertEqual(resource, expected_resource) 52 | -------------------------------------------------------------------------------- /altimeter/aws/resource/organizations/ou.py: -------------------------------------------------------------------------------- 1 | """Resource representing an AWS Organizational Unit.""" 2 | from typing import Type 3 | 4 | from botocore.client import BaseClient 5 | 6 | from altimeter.aws.resource.resource_spec import ListFromAWSResult 7 | from altimeter.aws.resource.organizations import ( 8 | OrganizationsResourceSpec, 9 | recursively_get_ou_details_for_parent, 10 | ) 11 | from altimeter.aws.resource.organizations.org import OrgResourceSpec 12 | from altimeter.core.graph.field.scalar_field import ScalarField 13 | from altimeter.core.graph.field.resource_link_field import ResourceLinkField 14 | from altimeter.core.graph.schema import Schema 15 | 16 | 17 | class OUResourceSpec(OrganizationsResourceSpec): 18 | """Resource representing an AWS OU.""" 19 | 20 | type_name = "ou" 21 | schema = Schema( 22 | ScalarField("Path"), ResourceLinkField("OrganizationArn", OrgResourceSpec, value_is_id=True) 23 | ) 24 | 25 | @classmethod 26 | def get_full_type_name(cls: Type["OUResourceSpec"]) -> str: 27 | return f"{cls.provider_name}:{cls.type_name}" 28 | 29 | @classmethod 30 | def list_from_aws( 31 | cls: Type["OUResourceSpec"], client: BaseClient, account_id: str, region: str 32 | ) -> ListFromAWSResult: 33 | """Return a dict of dicts of the format: 34 | 35 | {'ou_1_arn': {ou_1_dict}, 36 | 'ou_2_arn': {ou_2_dict}, 37 | ...} 38 | 39 | Where the dicts represent results from list_organizational_units_for_parent 40 | with some additional info 'Path') tagged on.""" 41 | org_resp = client.describe_organization() 42 | org_arn = org_resp["Organization"]["Arn"] 43 | ous = {} 44 | paginator = client.get_paginator("list_roots") 45 | for resp in paginator.paginate(): 46 | for root in resp["Roots"]: 47 | root_id, root_arn = root["Id"], root["Arn"] 48 | root_path = f"/{root['Name']}" 49 | ous[root_arn] = root 50 | ous[root_arn]["OrganizationArn"] = org_arn 51 | ous[root_arn]["Path"] = root_path 52 | ou_details = recursively_get_ou_details_for_parent( 53 | client=client, parent_id=root_id, parent_path=root_path 54 | ) 55 | for ou_detail in ou_details: 56 | arn = ou_detail["Arn"] 57 | ou_detail["OrganizationArn"] = org_arn 58 | ous[arn] = ou_detail 59 | return ListFromAWSResult(resources=ous) 60 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | """setup.py""" 2 | from setuptools import setup, find_packages 3 | 4 | with open("README.md", "r") as fh: 5 | long_description = fh.read() 6 | 7 | setup( 8 | name="altimeter", 9 | version="0.0.1", 10 | packages=find_packages(exclude=["tests"]), 11 | author="Tableau", 12 | description="Graph AWS resources in Neptune", 13 | long_description=long_description, 14 | long_description_content_type="text/markdown", 15 | url="https://github.com/tableau/altimeter", 16 | python_requires=">=3.8,<3.10", 17 | install_requires=[ 18 | "MarkupSafe==2.1.1", 19 | "aws-requests-auth==0.4.3", 20 | "rdflib==6.0.2", 21 | "structlog==20.2.0", 22 | "boto3==1.28.80", 23 | "jinja2==3.0.3", 24 | "pydantic==1.9.0", 25 | "toml==0.10.2", 26 | "gremlinpython==3.4.12", 27 | "requests==2.31.0", 28 | "certifi==2023.7.22", 29 | "urllib3==1.26.18", 30 | ], 31 | extras_require={ 32 | "qj": [ 33 | "MarkupSafe==2.1.1", 34 | "alembic==1.4.2", 35 | "boto3==1.28.80", 36 | "fastapi==0.96.0", 37 | "psycopg2-binary==2.9.2", 38 | "sqlalchemy==1.3.24", 39 | "tableauhyperapi==0.0.18161", 40 | "tableauserverclient==0.17.0", 41 | "uvicorn==0.16.0", 42 | "urllib3==1.26.18", 43 | ], 44 | }, 45 | data_files=[ 46 | ( 47 | "altimeter/services/qj/alembic", 48 | ["services/qj/alembic/env.py", "services/qj/alembic/alembic.ini",], 49 | ), 50 | ( 51 | "altimeter/services/qj/alembic/versions", 52 | [ 53 | "services/qj/alembic/versions/dc8f1df07766_init.py", 54 | "services/qj/alembic/versions/60990e9bc347_added_notify_if_results_bool_on_job.py", 55 | "services/qj/alembic/versions/9d956e753055_added_remediate_sqs_queue_column_to_qj_.py", 56 | "services/qj/alembic/versions/e6e2a6bf2a39_adding_query_job_raw_query_column.py", 57 | "services/qj/alembic/versions/94f36533d115_remove_result_synthetic_pk.py", 58 | ], 59 | ), 60 | ], 61 | scripts=[ 62 | "bin/altimeter", 63 | "bin/aws2n.py", 64 | "bin/aws2neptune.py", 65 | "bin/rdf2blaze", 66 | "bin/runquery.py", 67 | "bin/scan_resource.py", 68 | ], 69 | classifiers=[ 70 | "Programming Language :: Python :: 3", 71 | "License :: OSI Approved :: MIT License", 72 | "Operating System :: OS Independent", 73 | ], 74 | ) 75 | -------------------------------------------------------------------------------- /altimeter/aws/resource/iam/iam_oidc_provider.py: -------------------------------------------------------------------------------- 1 | """Resource for IAM OIDC Providers""" 2 | from typing import Type 3 | 4 | from botocore.client import BaseClient 5 | from botocore.exceptions import ClientError 6 | 7 | from altimeter.aws.resource.resource_spec import ListFromAWSResult 8 | from altimeter.aws.resource.iam import IAMResourceSpec 9 | from altimeter.core.graph.field.scalar_field import ScalarField, EmbeddedScalarField 10 | from altimeter.core.graph.field.list_field import ListField 11 | from altimeter.core.graph.schema import Schema 12 | 13 | 14 | class IAMOIDCProviderResourceSpec(IAMResourceSpec): 15 | """Resource for IAM OIDC Providers""" 16 | 17 | type_name = "oidc-provider" 18 | schema = Schema( 19 | ScalarField("Url"), 20 | ScalarField("CreateDate"), 21 | ListField("ClientIDList", EmbeddedScalarField(), alti_key="client_id"), 22 | ListField("ThumbprintList", EmbeddedScalarField(), alti_key="thumbprint"), 23 | ) 24 | 25 | @classmethod 26 | def list_from_aws( 27 | cls: Type["IAMOIDCProviderResourceSpec"], client: BaseClient, account_id: str, region: str 28 | ) -> ListFromAWSResult: 29 | """Return a dict of dicts of the format: 30 | 31 | {'oidc_provider_1_arn': {oidc_provider_1_dict}, 32 | 'oidc_provider_2_arn': {oidc_provider_2_dict}, 33 | ...} 34 | 35 | Where the dicts represent results from list_oidc_providers and additional info per 36 | oidc_provider list_oidc_providers. An additional 'Name' key is added.""" 37 | oidc_providers = {} 38 | resp = client.list_open_id_connect_providers() 39 | for oidc_provider in resp.get("OpenIDConnectProviderList", []): 40 | resource_arn = oidc_provider["Arn"] 41 | try: 42 | oidc_provider_details = cls.get_oidc_provider_details( 43 | client=client, arn=resource_arn 44 | ) 45 | oidc_provider.update(oidc_provider_details) 46 | oidc_providers[resource_arn] = oidc_provider 47 | except ClientError as c_e: 48 | error_code = getattr(c_e, "response", {}).get("Error", {}).get("Code", {}) 49 | if error_code != "NoSuchEntity": 50 | raise c_e 51 | return ListFromAWSResult(resources=oidc_providers) 52 | 53 | @classmethod 54 | def get_oidc_provider_details( 55 | cls: Type["IAMOIDCProviderResourceSpec"], client: BaseClient, arn: str 56 | ) -> str: 57 | oidc_provider_resp = client.get_open_id_connect_provider(OpenIDConnectProviderArn=arn) 58 | return oidc_provider_resp 59 | -------------------------------------------------------------------------------- /altimeter/qj/remediator.py: -------------------------------------------------------------------------------- 1 | """Base remediator class""" 2 | from typing import Any, Dict, List, Optional 3 | 4 | import boto3 5 | from pydantic import BaseSettings 6 | 7 | from altimeter.core.log import Logger 8 | from altimeter.qj.exceptions import RemediationError 9 | from altimeter.qj.log import QJLogEvents 10 | from altimeter.qj.schemas.result_set import Result 11 | 12 | 13 | class Config(BaseSettings): 14 | dry_run: bool 15 | remediator_target_role_name: str 16 | remediator_target_role_external_id: Optional[str] = None 17 | 18 | 19 | class RemediatorLambda: 20 | @classmethod 21 | def remediate(cls, session: boto3.Session, result: Dict[str, Any], dry_run: bool) -> None: 22 | raise NotImplementedError("RemediatorLambda.remediate must be implemented in subclasses") 23 | 24 | @classmethod 25 | def lambda_handler(cls, event: Dict[str, Any], _: Any) -> None: 26 | """lambda entrypoint""" 27 | config = Config() 28 | result = Result(**event) 29 | logger = Logger() 30 | errors: List[str] = [] 31 | with logger.bind(result=result): 32 | logger.info(event=QJLogEvents.ResultRemediationStart) 33 | try: 34 | session = get_assumed_session( 35 | account_id=result.account_id, 36 | role_name=config.remediator_target_role_name, 37 | external_id=config.remediator_target_role_external_id, 38 | ) 39 | cls.remediate(session=session, result=result.result, dry_run=config.dry_run) 40 | logger.info(event=QJLogEvents.ResultRemediationSuccessful) 41 | except Exception as ex: 42 | logger.error(event=QJLogEvents.ResultRemediationFailed, error=str(ex)) 43 | errors.append(str(ex)) 44 | if errors: 45 | raise RemediationError(f"Errors found during remediation: {errors}") 46 | 47 | 48 | def get_assumed_session( 49 | account_id: str, role_name: str, external_id: Optional[str] = None 50 | ) -> boto3.Session: 51 | cws = boto3.Session() 52 | sts_client = cws.client("sts") 53 | role_arn = f"arn:aws:iam::{account_id}:role/{role_name}" 54 | assume_args = {"RoleArn": role_arn, "RoleSessionName": "qj-remediator"} 55 | if external_id is not None: 56 | assume_args["ExternalId"] = external_id 57 | assume_resp = sts_client.assume_role(**assume_args) 58 | creds = assume_resp["Credentials"] 59 | return boto3.Session( 60 | aws_access_key_id=creds["AccessKeyId"], 61 | aws_secret_access_key=creds["SecretAccessKey"], 62 | aws_session_token=creds["SessionToken"], 63 | ) 64 | -------------------------------------------------------------------------------- /altimeter/aws/resource/unscanned_account.py: -------------------------------------------------------------------------------- 1 | """Resource representing an unscanned AWS Account""" 2 | from typing import List, Tuple, Type 3 | import uuid 4 | 5 | from botocore.client import BaseClient 6 | 7 | from altimeter.core.graph.links import LinkCollection, SimpleLink 8 | from altimeter.core.graph.schema import Schema 9 | from altimeter.aws.resource.resource_spec import ScanGranularity, ListFromAWSResult, AWSResourceSpec 10 | from altimeter.core.resource.resource import Resource 11 | from altimeter.core.resource.resource_spec import ResourceSpec 12 | from altimeter.aws.scan.aws_accessor import AWSAccessor 13 | 14 | 15 | class UnscannedAccountResourceSpec(AWSResourceSpec): 16 | """Resource representing an unscanned AWS Account""" 17 | 18 | type_name = "unscanned-account" 19 | service_name = "null" 20 | scan_granularity = ScanGranularity.ACCOUNT 21 | schema = Schema() 22 | 23 | @classmethod 24 | def create_resource( 25 | cls: Type["UnscannedAccountResourceSpec"], account_id: str, errors: List[str] 26 | ) -> Resource: 27 | simple_links: List[SimpleLink] = [] 28 | simple_links.append(SimpleLink(pred="account_id", obj=account_id)) 29 | if errors: 30 | error = "\n".join(errors) 31 | simple_links.append(SimpleLink(pred="error", obj=f"{error} - {uuid.uuid4()}")) 32 | return Resource( 33 | resource_id=cls.generate_arn(resource_id=account_id), 34 | type=cls.get_full_type_name(), 35 | link_collection=LinkCollection(simple_links=simple_links), 36 | ) 37 | 38 | @classmethod 39 | def get_full_type_name(cls: Type["UnscannedAccountResourceSpec"]) -> str: 40 | return f"{cls.provider_name}:{cls.type_name}" 41 | 42 | @classmethod 43 | def list_from_aws( 44 | cls: Type["UnscannedAccountResourceSpec"], client: BaseClient, account_id: str, region: str 45 | ) -> ListFromAWSResult: 46 | """List resources from AWS using client.""" 47 | return ListFromAWSResult(resources={}) 48 | 49 | @classmethod 50 | def generate_arn( 51 | cls: Type["UnscannedAccountResourceSpec"], 52 | resource_id: str, 53 | account_id: str = "", 54 | region: str = "", 55 | ) -> str: 56 | """Generate an ARN for this resource""" 57 | return f"arn:aws::::account/{resource_id}" 58 | 59 | @classmethod 60 | def scan( 61 | cls: Type[AWSResourceSpec], 62 | scan_accessor: AWSAccessor, 63 | all_resource_spec_classes: Tuple[Type[ResourceSpec], ...], 64 | ) -> List[Resource]: 65 | raise NotImplementedError(f"{cls.__name__} is not a scannable ResourceSpec class.") 66 | -------------------------------------------------------------------------------- /tests/unit/altimeter/core/graph/test_graph_spec.py: -------------------------------------------------------------------------------- 1 | from typing import Any, List, Tuple, Type 2 | from unittest import TestCase 3 | 4 | from altimeter.core.graph.graph_spec import GraphSpec 5 | from altimeter.core.graph.links import LinkCollection 6 | from altimeter.core.resource.resource import Resource 7 | from altimeter.core.resource.resource_spec import ResourceSpec 8 | 9 | 10 | class TestResourceSpecA(ResourceSpec): 11 | type_name = "a" 12 | 13 | @classmethod 14 | def get_full_type_name(self): 15 | return "test:a" 16 | 17 | @classmethod 18 | def scan( 19 | cls: Type["TestResourceSpecA"], 20 | scan_accessor: Any, 21 | all_resource_spec_classes: Tuple[Type["ResourceSpec"], ...], 22 | ) -> List[Resource]: 23 | resources = [ 24 | Resource(resource_id="123", type=cls.type_name, link_collection=LinkCollection()), 25 | Resource(resource_id="456", type=cls.type_name, link_collection=LinkCollection()), 26 | ] 27 | return resources 28 | 29 | 30 | class TestResourceSpecB(ResourceSpec): 31 | type_name = "b" 32 | 33 | @classmethod 34 | def get_full_type_name(self): 35 | return "test:b" 36 | 37 | @classmethod 38 | def scan( 39 | cls: Type["TestResourceSpecB"], 40 | scan_accessor: Any, 41 | all_resource_spec_classes: Tuple[Type["ResourceSpec"], ...], 42 | ) -> List[Resource]: 43 | resources = [ 44 | Resource(resource_id="abc", type=cls.type_name, link_collection=LinkCollection()), 45 | Resource(resource_id="def", type=cls.type_name, link_collection=LinkCollection()), 46 | ] 47 | return resources 48 | 49 | 50 | class TestScanAccessor: 51 | pass 52 | 53 | 54 | class TestGraphSpec(TestCase): 55 | def test_scan(self): 56 | scan_accessor = TestScanAccessor() 57 | graph_spec = GraphSpec( 58 | name="test-name", 59 | version="1", 60 | resource_spec_classes=(TestResourceSpecA, TestResourceSpecB), 61 | all_resource_spec_classes=(TestResourceSpecA, TestResourceSpecB), 62 | scan_accessor=scan_accessor, 63 | ) 64 | resources = graph_spec.scan() 65 | 66 | expected_resources = [ 67 | Resource(resource_id="123", type="a", link_collection=LinkCollection()), 68 | Resource(resource_id="456", type="a", link_collection=LinkCollection()), 69 | Resource(resource_id="abc", type="b", link_collection=LinkCollection()), 70 | Resource(resource_id="def", type="b", link_collection=LinkCollection()), 71 | ] 72 | self.assertEqual(resources, expected_resources) 73 | --------------------------------------------------------------------------------