├── src ├── ide │ └── vscode.md ├── app │ └── chrome │ │ └── stuck_resolving_host.md ├── cloud │ ├── aws │ │ ├── services │ │ │ ├── cloudformation_shenanigans.md │ │ │ ├── iam_managed_policy_escalation.md │ │ │ ├── lambda_metric_alarm.md │ │ │ ├── s3_bucket_policy_500_errors.md │ │ │ ├── sns_cross_account.md │ │ │ ├── s3_public_access_block.md │ │ │ └── s3_bucket_policy_weird_behaviors.md │ │ ├── permissions_wtf.md │ │ ├── sdk │ │ │ └── boto3 │ │ │ │ └── boto3_private_beta_services.md │ │ └── event_examples │ │ │ └── cloudtrail_management_events.md │ └── infrastructure_as_code │ │ └── terraform │ │ └── for_each_multiple_values.md ├── programming │ └── python │ │ └── snippets.md └── os │ └── windows │ └── 10 │ └── power_problems.md ├── README.md ├── assets ├── images │ └── aws_s3_put_policy_500_internal_error.png └── text │ └── github_permissions_snafu_email.txt └── abbrev └── s3.md /src/ide/vscode.md: -------------------------------------------------------------------------------- 1 | Python unresolved import errors: the root directory of your vscode project matters. 2 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Moved to [elfakyn.com/knowledge](https://elfakyn.com/knowledge) 2 | 3 | This kept here for posterity. 4 | -------------------------------------------------------------------------------- /assets/images/aws_s3_put_policy_500_internal_error.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/elfakyn/knowledge/master/assets/images/aws_s3_put_policy_500_internal_error.png -------------------------------------------------------------------------------- /src/app/chrome/stuck_resolving_host.md: -------------------------------------------------------------------------------- 1 | # Stuck resolving host 2 | 3 | Try the following: 4 | 5 | **Option 1:** Go to **[chrome://net-internals/#dns](chrome://net-internals/#dns)** and click "Clear host cache". 6 | 7 | **Option 2:** It may be related to MalwareBytes. But don't know how to fix it short of uninstalling MB. Good luck and PR a change if you found a fix! 8 | 9 | That's all I got so far. 10 | -------------------------------------------------------------------------------- /src/cloud/aws/services/cloudformation_shenanigans.md: -------------------------------------------------------------------------------- 1 | # Weird cloudformation shit 2 | 3 | You're trying to do something with CFN that should work, but doesn't. What the hell? 4 | 5 | what you're trying to do | what you think you should use | what you should ACTUALLY use | ref 6 | --- | --- | --- | --- 7 | get the ARN of an SQS queue | `{"Ref" : "MyQueue"}` | `{"Fn::GetAtt" : ["MyQueue", "Arn"]}` | [1](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-sqs-queues.html#aws-properties-sqs-queues-return-values) 8 | -------------------------------------------------------------------------------- /src/cloud/aws/services/iam_managed_policy_escalation.md: -------------------------------------------------------------------------------- 1 | # AWS managed policies that give problematic permissions 2 | 3 | Some AWS managed policies are obviously dangerous such as `IAMFullAccess`. But did you know `AWSCloudTrailReadOnlyAccess` gives you read access to every object in s3? 4 | 5 | Below is an incomplete list of AWS policies and the bonkers access they give. 6 | 7 | policy | what it gives | on | why it's dangerous 8 | --- | --- | --- | --- 9 | AWSCloudTrailReadOnlyAccess | s3:GetObject | * | if you store PII in s3 you've just given someone access to all of it 10 | -------------------------------------------------------------------------------- /src/cloud/aws/permissions_wtf.md: -------------------------------------------------------------------------------- 1 | # Misleading permission names 2 | 3 | Some permissions don't do what their name suggests. 4 | 5 | Permission name | What it actually does | Access level | Link 6 | --- | --- | --- | --- 7 | `glue:GetMapping` | Creates a mapping | WRITE | [aws docs](https://docs.aws.amazon.com/en_us/IAM/latest/UserGuide/list_awsglue.html) 8 | 9 | # Phantom actions 10 | 11 | There isn't a 1:1 mapping between actions and permissions in AWS. Some actions require more permissions than you'd expect. 12 | 13 | To do this action... | you need this permission 14 | --- | --- 15 | `s3:HeadObject` | `s3:GetObject` 16 | -------------------------------------------------------------------------------- /src/programming/python/snippets.md: -------------------------------------------------------------------------------- 1 | ## Timezone madness 2 | 3 | ```python 4 | import datetime 5 | ``` 6 | 7 | ### Timestamp milliseconds to zulu time 8 | 9 | ```python 10 | datetime.datetime.fromtimestamp(unix_time_milliseconds // 1000, tz=datetime.timezone.utc).isoformat().replace('+00:00', 'Z') 11 | ``` 12 | 13 | Input: `1582154420000` 14 | Output: `'2020-02-19T23:20:20Z'` 15 | 16 | ### Zulu time to timestamp milliseconds 17 | 18 | ```python 19 | int(datetime.datetime.strptime(formatted_timestamp, '%Y-%m-%dT%H:%M:%SZ').replace(tzinfo=datetime.timezone.utc).timestamp() * 1000) 20 | ``` 21 | 22 | Input: `'2020-02-19T23:20:20Z'` 23 | Output: `1582154420000` 24 | -------------------------------------------------------------------------------- /src/cloud/infrastructure_as_code/terraform/for_each_multiple_values.md: -------------------------------------------------------------------------------- 1 | Basically every terraform `for_each` example out there has the following structure: 2 | 3 | ``` 4 | variable "rg_locations" { 5 | default = { 6 | a_group = "eastus" 7 | another_group = "westus2" 8 | } 9 | 10 | resource "azurerm_resource_group" "rg" { 11 | for_each = var.rg_locations 12 | name = each.key 13 | location = each.value 14 | } 15 | ``` 16 | 17 | But they never tell you what to do if you want to access more than one value. It's easy: 18 | 19 | ``` 20 | variable "rg_locations" { 21 | default = { 22 | a_group = { 23 | location = "eastus" 24 | description = "our main location" 25 | tags = "primary" 26 | } 27 | another_group = { 28 | location = "westus2" 29 | description = "our fallback location" 30 | tags = "backup" 31 | } 32 | } 33 | } 34 | 35 | resource "azurerm_resource_group" "rg" { 36 | for_each = var.rg_locations 37 | name = each.key 38 | location = each.value["location"] 39 | description = each.value["description"] 40 | tags = each.value["tags"] 41 | } 42 | ``` 43 | 44 | # FAQ 45 | 46 | ## Why a map and not a list? 47 | 48 | for_each doesn't accept a list, only maps and sets. 49 | 50 | ## Can you offload multiple variables without having to say `location = ... "location"` for each of them? 51 | 52 | You can do that if it's a [dynamic nested block](https://www.hashicorp.com/blog/hashicorp-terraform-0-12-preview-for-and-for-each/) but couldn't find anything for the toplevel resource block. Submit a ticket or PR if you do. 53 | -------------------------------------------------------------------------------- /src/cloud/aws/services/lambda_metric_alarm.md: -------------------------------------------------------------------------------- 1 | # Lambda metric alarm 2 | 3 | When you create a lambda metric alarm you really don't want to have to think about it. Here's the values (in terraform, but you can use the same ones in cloudformation): 4 | 5 | ```terraform 6 | resource "aws_cloudwatch_metric_alarm" "mylambda_error_alarm" { 7 | alarm_name = "mylambda_error_alarm" 8 | alarm_description = "MyLambda erorred out !!" 9 | comparison_operator = "GreaterThanOrEqualToThreshold" 10 | evaluation_periods = "1" 11 | metric_name = "Errors" 12 | namespace = "AWS/Lambda" 13 | dimensions = { 14 | FunctionName = aws_lambda_function.mylambda.function_name 15 | } 16 | threshold = "1" 17 | period = "60" 18 | statistic = "Sum" 19 | treat_missing_data = "missing" 20 | insufficient_data_actions = [] 21 | alarm_actions = [var.destination_sns_topic] 22 | } 23 | ``` 24 | 25 | ## IMPORTANT! You want to treat missing data as `missing`! 26 | 27 | Why? Even though it's no fun to have a bunch of INSUFFICIENTs in CloudWatch, the rationale for keeping it as INSUFFICIENT instead of OK is that if a lambda runs every 6 hours and fails every time, in between executions there is insufficient information to know whether it's running correctly or not; you don't want it to show OK. **You never want something to show up as OK when you can't make that determination for sure!** 28 | 29 | You also don't want to use `ignore` since in that case the alarm will only fire once for multiple consecutive failures: `ignore` needs a successful execution before it can alarm again. `missing` alarms every time. 30 | 31 | ## Frequent execution caveat 32 | 33 | If you have a lambda that executes more often than the alarm period, then the alarm will only fire once and never have time to reset to `INSUFFICIENT`. If that's important you need to either reduce the period (extra cost) or figure some other way to not drop the ball on the alarm. 34 | -------------------------------------------------------------------------------- /src/cloud/aws/sdk/boto3/boto3_private_beta_services.md: -------------------------------------------------------------------------------- 1 | # Using private beta services with boto3 2 | 3 | You've been invited to the private beta of a new AWS service, 'darkspeed'. Boto3 doesn't have that service yet, but AWS has given you a json file with the service definition (`darkspeed-2018-07-06.normal.json`). How do you access it in boto3? 4 | 5 | 1. Rename the service definition file to `service-2.json` and put it in the following directory structure: 6 | 7 | ```plaintext 8 | my_project / 9 | - __init__.py 10 | - my_program.py 11 | - darkspeed / 12 | - 2018-07-06 / 13 | - service-2.json 14 | ``` 15 | 16 | 2. Set the environment variable `AWS_DATA_PATH` to point to `my_project` (do this before you call `boto3.client`). You can do this in `my_program.py`: 17 | 18 | ```python 19 | os.environ['AWS_DATA_PATH'] = os.path.dirname(os.path.realpath(__file__)) 20 | ``` 21 | 22 | 3. Use the following boto3 call to create the client: 23 | 24 | ```python 25 | darkspeed = boto3.client( 26 | service_name = 'darkspeed', 27 | region = 'us-east-9000' 28 | ) 29 | ``` 30 | 31 | Now you can call the service as usual: 32 | 33 | ```python 34 | velocity = darkspeed.get_maximum_velocity(CharacterName = 'Sonic', Location = 'Green Hill Zone') 35 | print(velocity) 36 | ``` 37 | 38 | ## FAQ 39 | 40 | ### Does it have to be called `service-2.json`? 41 | 42 | Yes. `service-1` is the old version, all closed betas I'm aware of are `service-2`. 43 | 44 | ### Does it have to have that exact directory structure? 45 | 46 | Yes. `service_name/YYYY-MM-DD/service-2.json`. Check inside the json file. The service name has to match the `signingName` and the date has to match the `apiVersion` date. 47 | 48 | ### What if I need to provide a specific endpoint URL? 49 | 50 | You shouldn't need to, but if you do, use this: 51 | 52 | ```python 53 | darkspeed = boto3.client( 54 | service_name = 'darkspeed', 55 | region_name = 'us-east-9000', 56 | endpoint_url = 'https://darkspeed-beta.us-east-9000.amazonaws.com' 57 | ) 58 | ``` 59 | 60 | ### What if AWS gave me more than 1 file? 61 | 62 | They had better given them to you named: `service-2.json`, `paginators-1.json`, `api-2.json`. If they didn't, check the documentation they gave you or ask your rep. 63 | 64 | #### References 65 | 66 | * [Loaders reference -- found this AFTER I wrote this whole thing](https://botocore.amazonaws.com/v1/documentation/api/latest/reference/loaders.html) 67 | * [Boto3 session reference](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html) 68 | -------------------------------------------------------------------------------- /assets/text/github_permissions_snafu_email.txt: -------------------------------------------------------------------------------- 1 | 2 | GitHub Support 3 | Wed, Apr 22, 6:04 PM 4 | to me 5 | 6 | Hi elfakyn, 7 | 8 | We’re writing to let you know that on April 15, non-admin repository members were briefly able to override branch protection settings and merge pull requests when branch protection rules weren't met. Specifically, these non-admin users were already allowed to push to protected branches. 9 | 10 | If you are receiving this email, you administer or own a repository that allowed a non-admin user to merge a pull request to a protected branch during the timeframe, and you may want to review merged pull requests. 11 | 12 | GitHub itself did not experience a compromise, data breach, or data exposure as a result of this event, nor did unauthorized users gain access to repositories. User privacy and security are essential, and we want to remain as transparent as possible about events like these. Read on for more information. 13 | 14 | * What Happened? * 15 | 16 | GitHub Security learned that non-admin users who were granted the ability to push to protected branches were able to override branch protection features and merge pull requests to protected branches on April 15, between approximately 09:39 AM and 03:01 PM PDT. 17 | 18 | GitHub immediately fixed the problem and began investigating the events to identify affected repositories. 19 | 20 | * What Information Was Involved? * 21 | 22 | The following repositories had at least one unauthorized pull request merged by a non-admin user when branch protection rules weren’t met: 23 | 24 | repository,merging_user 25 | [REDACTED] 26 | 27 | Again, no repository contents were exposed to unauthorized parties on GitHub.com; any users affected by this event already had valid permissions on the repositories noted above. 28 | 29 | * What GitHub Is Doing * 30 | 31 | GitHub immediately fixed the bug and notified all affected users. We are also performing an internal assessment to determine how we can better prevent this sort of bug in the future. 32 | 33 | * What You Can Do * 34 | 35 | Review and confirm pull requests merged on April 15th for the repositories listed above. While anyone merging content would already have some level of access to the repositories, the merged content may have been unexpected. Although these settings would not have changed as a result of this event, you may also wish to review branch protection settings: 36 | 37 | https://help.github.com/en/github/administering-a-repository/about-protected-branches 38 | 39 | Please feel free to reach out to us with any additional questions or concerns through this contact form: https://github.com/contact?form%5Bsubject%5D=Re:Reference+[REDACTED]. 40 | 41 | Thanks, 42 | GitHub Support 43 | 44 | 45 | -------------------------------------------------------------------------------- /src/cloud/aws/services/s3_bucket_policy_500_errors.md: -------------------------------------------------------------------------------- 1 | # I'm getting 500 errors on PutBucketPolicy, help! 2 | 3 | ## What the error looks like 4 | 5 | Sometimes s3 spits out a cryptic error: 6 | 7 | ![Big red error message in the AWS Console. The title of the error is "Error" and the text of the error is "We encountered an internal error. Please try again."](/assets/images/aws_s3_put_policy_500_internal_error.png) 8 | 9 | or like this: 10 | 11 | ```json 12 | { 13 | "Error": { 14 | "Code": "InternalError", 15 | "Message": "We encountered an internal error. Please try again." 16 | }, 17 | "ResponseMetadata": { 18 | "HTTPHeaders": { 19 | "connection": "close", 20 | "content-type": "application/xml", 21 | "date": "Fri, 05 Jun 2020 16:43:12 GMT", 22 | "server": "AmazonS3", 23 | "transfer-encoding": "chunked", 24 | "x-amz-id-2": "00000000000000/1111111111111111/22222222222222222222222222222222=", 25 | "x-amz-request-id": "AAAAAAAAAAAAAAA" 26 | }, 27 | "HTTPStatusCode": 500, 28 | "HostId": "00000000000000/1111111111111111/22222222222222222222222222222222=", 29 | "MaxAttemptsReached": true, 30 | "RequestId": "AAAAAAAAAAAAAAA", 31 | "RetryAttempts": 4 32 | } 33 | } 34 | ``` 35 | 36 | Don't try again; it won't work. The error is likely due to one of the following reasons: 37 | 38 | ## You are trying to put a cross-account policy with `RestrictPublicBuckets` enabled 39 | 40 | ### Background 41 | 42 | What the issue is: 43 | * You have a public access block on your bucket with `RestrictPublicBuckets` enabled (also known as "Block public and cross-account access to buckets and objects through any public bucket policies") 44 | * You are trying to put a bucket policy that grants cross-account access to the bucket 45 | 46 | Why this happens: 47 | * It's likely a bug, and may have been fixed since. 48 | 49 | ### How to fix 50 | 51 | Disable `RestrictPublicBuckets` ("Block public and cross-account access to buckets and objects through any public bucket policies") in your public access block. 52 | 53 | ## You are trying to put a malformed policy 54 | 55 | ### Background 56 | 57 | Not all malformed policies will result in a `MalformedPolicy` error. 58 | 59 | Here's an example of a policy that will give you a 500 error: 60 | 61 | ```json 62 | { 63 | "Version": "2012-10-17", 64 | "Id": "whatever", 65 | "Statement": [ 66 | { 67 | "Sid": "ReadOnly", 68 | "Effect": "Allow", 69 | "Principal": { 70 | "AWS": [] 71 | }, 72 | "Action": "s3:GetObject", 73 | "Resource": "arn:aws:s3:::my-bucket/*" 74 | } 75 | ] 76 | } 77 | ``` 78 | 79 | ### How to fix 80 | 81 | Fix your policy. 82 | 83 | Potential causes: 84 | * Policy statements can't have an empty array as a principal. Add a principal or remove the statement entirely. 85 | -------------------------------------------------------------------------------- /src/cloud/aws/services/sns_cross_account.md: -------------------------------------------------------------------------------- 1 | # Posting to SNS cross account 2 | 3 | You have a CloudWatch alarm or lambda in account 111111111111 and you need it to post to an SNS topic in account 222222222222. How to do this? 4 | 5 | There's no special configuration needed in account 111111111111. All you need is the following policy on the SNS topic in account 222222222222. 6 | 7 | If you only use CloudWatch to post to this SNS, you only need the `CrossAccountWithCloudWatch` statement. If you use something other than CloudWatch (such as a lambda publishing messages), you only need the `CrossAccount` one. If you use both you need both. 8 | 9 | ```json 10 | { 11 | "Version": "2012-10-17", 12 | "Statement": [ 13 | { 14 | "Sid": "CrossAccount", 15 | "Effect": "Allow", 16 | "Principal": { 17 | "AWS": [ 18 | "arn:aws:iam::111111111111:root" 19 | ] 20 | }, 21 | "Action": "SNS:Publish", 22 | "Resource": "arn:aws:sns:REGION:222222222222:my-sns-topic" 23 | }, 24 | { 25 | "Sid": "CrossAccountWithCloudWatch", 26 | "Effect": "Allow", 27 | "Principal": { 28 | "AWS": "*" 29 | }, 30 | "Action": "SNS:Publish", 31 | "Resource": "arn:aws:sns:REGION:222222222222:my-sns-topic", 32 | "Condition": { 33 | "StringEquals": { 34 | "AWS:SourceOwner": [ 35 | "111111111111" 36 | ] 37 | } 38 | } 39 | } 40 | ] 41 | } 42 | ``` 43 | 44 | That's all, pop it and ship it! 45 | 46 | **Note:** Since this will replace the default SNS access policy ([which is the thing that grants access inside the same account to publish SNS](https://docs.aws.amazon.com/sns/latest/dg/sns-access-policy-use-cases.html)), you're going to have to explicitly grant the original account (222222222222) access to the SNS topic again if you want to publish inside the account as well. 47 | 48 | ## Explanation 49 | 50 | The `CrossAccount` statement allows entities in 111111111111 to push to the SNS topic like any normal cross-account policy, with an account principal. But for some reason that doesn't work with CloudWatch. Instead, you need to use `SourceOwner` as a condition key. I don't like having `*` in the principal, it's a spaghetti factory, but I couldn't find another way. Submit a PR if you do. 51 | 52 | ## FAQ 53 | 54 | ### Can I restrict these further? I don't want to give a whole account access. 55 | 56 | YES! Use specific principal ARNs in the first one and add further conditions in the second one. 57 | 58 | ### What if I need more than one account to publish to this SNS? 59 | 60 | You can add multiple `Principal`s and/or multiple `SourceOwner`s in the list, you don't need a whole separate `Statement` for each account. 61 | 62 | ### Can I add all accounts in my org without having to list them out by hand? 63 | 64 | I don't know, but you might want to look into the `AWS:PrincipalOrgID` condition key, maybe it'll work. 65 | -------------------------------------------------------------------------------- /abbrev/s3.md: -------------------------------------------------------------------------------- 1 | # IAM Policies are janky! 2 | 3 | 4 | 5 | 6 | 7 | 8 | # Example resource policy 9 | 10 | 11 | ```json 12 | { 13 | "Version": "2012-10-17", 14 | "Statement": [ 15 | { 16 | "Effect": "Allow", 17 | "Principal": { 18 | "AWS": "arn:aws:iam::AccountB:user/AccountBUserName" 19 | }, 20 | "Action": [ 21 | "s3:GetObject", 22 | "s3:PutObject", 23 | "s3:PutObjectAcl" 24 | ], 25 | "Resource": [ 26 | "arn:aws:s3:::AccountABucketName/*" 27 | ], 28 | "Condition": { 29 | "NotIpAddress": {"aws:SourceIp": "54.240.143.0/24"} 30 | } 31 | } 32 | ] 33 | } 34 | ``` 35 | 36 | # So what's the problem? 37 | 38 | 39 | 40 | # You can break AWS's policy validator. 41 | 42 | This policy gives a 500 error: 43 | 44 | ```json 45 | { 46 | "Version": "2012-10-17", 47 | "Id": "whatever", 48 | "Statement": [ 49 | { 50 | "Sid": "ReadOnly", 51 | "Effect": "Allow", 52 | "Principal": { 53 | "AWS": [] 54 | }, 55 | "Action": "s3:GetObject", 56 | "Resource": "arn:aws:s3:::my-bucket/*" 57 | } 58 | ] 59 | } 60 | ``` 61 | 62 | 63 | # Incident response: deleting user. 64 | 65 | Before: 66 | 67 | ```json 68 | { 69 | "Version": "2012-10-17", 70 | "Statement": [ 71 | { 72 | "Effect": "Allow", 73 | "Principal": { 74 | "AWS": "arn:aws:iam::123456789012:user/Test" 75 | }, 76 | "Action": [ 77 | "s3:ListBucket", 78 | "s3:GetBucketLocation" 79 | ], 80 | "Resource": "arn:aws:s3:::TestyMcTestFace" 81 | } 82 | ] 83 | } 84 | ``` 85 | 86 | After: 87 | 88 | ```json 89 | { 90 | "Version": "2012-10-17", 91 | "Statement": [ 92 | { 93 | "Effect": "Allow", 94 | "Principal": { 95 | "AWS": "AIDAxxxxxxxxxxxxxxxxx" 96 | }, 97 | "Action": [ 98 | "s3:ListBucket", 99 | "s3:GetBucketLocation" 100 | ], 101 | "Resource": "arn:aws:s3:::elfakyn-testytest" 102 | } 103 | ] 104 | } 105 | ``` 106 | 107 | ## Denies will stop working! 108 | 109 | 110 | # You can brute force valid user names and roles in an AWS account 111 | 112 | ```json 113 | { 114 | "Version": "2012-10-17", 115 | "Statement": [ 116 | { 117 | "Effect": "Allow", 118 | "Principal": { 119 | "AWS": "arn:aws:iam::123456789012:user/Johnny" 120 | }, 121 | "Action": [ 122 | "s3:ListBucket", 123 | ], 124 | "Resource": "arn:aws:s3:::MyEvilBucket" 125 | } 126 | ] 127 | } 128 | ``` 129 | -------------------------------------------------------------------------------- /src/cloud/aws/services/s3_public_access_block.md: -------------------------------------------------------------------------------- 1 | # s3 bucket public access block 2 | 3 | There are 4 public access block settings. They have different names in the console and in the CLI. 4 | 5 | Setting in console | Setting in CLI | What it does 6 | --- | --- | --- 7 | Block public access to buckets and objects granted through new access control lists | BlockPublicAcls | Can't create new public bucket or object ACLs (it will error out if you try), existing ones work. 8 | Block public access to buckets and objects granted through any access control lists | IgnorePublicAcls | Ignores all public bucket or object ACLs. You can create new ones but it'll ignore them. 9 | Block public access to buckets and objects granted through new public bucket policies | BlockPublicPolicy | Can't create new public bucket policies (it will error out if you try). 10 | Block public and cross-account access to buckets and objects through any public bucket policies | RestrictPublicBuckets | This is weird. If there is any public access in the bucket policy, it will ignore both public and cross-account access. Details below. 11 | 12 | If you don't want public access, check all 4 boxes. If you are encountering issues with cross-account access, verify that there isn't a public statement on the bucket. 13 | 14 | I would **strongly discourage** putting private objects and public objects in the same bucket; it's a disaster waiting to happen. Use different buckets, even if it will take some effort to make that possible. AWS makes it extremely easy to mess up access control if all your objects are in the same bucket, and public buckets are a frighteningly common source of data breaches. Don't take the risk! 15 | 16 | ## What does RestrictPublicBuckets do? 17 | 18 | If there is at least one public statement in the bucket policy, both public statements and cross-account statements are ignored. 19 | 20 | What this means is that if you have RestrictPublicBuckets enabled in a bucket with only cross-account access, the cross-account access will work. But if you then add public access to the bucket, the cross account access will stop working (and, of course, the public access will not work either). What the fuck? 21 | 22 | ### Response from AWS Support 23 | 24 | **RestrictPublicBuckets: Block public and cross-account access to buckets and objects through any public bucket or access point policies** 25 | 26 | Setting this option to TRUE restricts access to an access point or bucket with a public policy to only AWS service principals and authorized users within the bucket owner's account. This setting blocks all cross-account access to the access point or bucket (except by AWS service principals), while still allowing users within the account to manage the access point or bucket.Enabling this setting doesn't affect existing access point or bucket policies, except that Amazon S3 blocks public and cross-account access derived from any public access point or bucket policy, including non-public delegation to specific accounts. 27 | 28 | Suppose that a bucket has a policy that grants access to a set of fixed principals(`"Principal": {"AWS": "arn:aws:iam::AccountB:user/AccountBUserName"}`). Under the previously described rules, this policy isn't public. Thus, if you enable the RestrictPublicBuckets setting, the policy remains in effect as written, because RestrictPublicBuckets only applies to buckets that have public policies. 29 | 30 | **However, if you add a public statement to the policy(`"Principal":"*"`), RestrictPublicBuckets takes effect on the bucket. It allows only AWS service principals and authorized users of the bucket owner's account to access the bucket.** 31 | 32 | As an example, suppose that a bucket owned by "Account-1" has a policy that contains the following: 33 | 34 | * A statement that grants access to AWS CloudTrail (which is an AWS service principal) 35 | * A statement that grants access to account "Account-2" 36 | * A statement that grants access to the public, for example by specifying `"Principal": "*"` with no limiting Condition 37 | 38 | This policy qualifies as public because of the third statement. With this policy in place and RestrictPublicBuckets enabled, Amazon S3 allows access only by CloudTrail. Even though statement 2 isn't public, Amazon S3 disables access by "Account-2." This is because statement 3 renders the entire policy public, so RestrictPublicBuckets applies. As a result, Amazon S3 disables cross-account access, even though the policy delegates access to a specific account, "Account-2." 39 | 40 | But if you remove statement 3 from the policy, then the policy doesn't qualify as public, and RestrictPublicBuckets no longer applies. Thus, "Account-2" regains access to the bucket, even if you leave RestrictPublicBuckets enabled[2]. 41 | 42 | References: 43 | [1] Blocking public access to your Amazon S3 storage - https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-control-block-public-access.html 44 | [2] The meaning of "public" - https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-control-block-public-access.html#access-control-block-public-access-policy-status 45 | 46 | ## Possible bugs 47 | 48 | I have encountered situations in which RestirctPublicBuckets prevented me from creating new public or cross-account bucket policies _(this appears to be a bug; [it gives a 500 error if you try](https://github.com/elfakyn/knowledge/blob/master/src/cloud/aws/services/s3_bucket_policy_500_errors.md))_ I haven't been able to reliably replicate it. It may have been fixed since. 49 | -------------------------------------------------------------------------------- /src/os/windows/10/power_problems.md: -------------------------------------------------------------------------------- 1 | Applies to: Windows 10, probably also 7/8/8.1. Check the last edited date on this document. 2 | 3 | This guide assumes you know your way around Windows and have already tried the usual suspects. 4 | 5 | **Last updated:** Jan 2020 6 | 7 | # Windows sleep problems 8 | 9 | ## Rant 10 | 11 | Windows power management is a janky piece of shit that's made worse by inconsistent vendor implementation. Every fucking hardware manufacturer has its own ideas of what "sleep" and "low power" and "wake" mean resulting in a fucking patchwork of non-standards that all abuse the already crappy power management system in all sorts of horrible fucking ways (how about TURN OFF THE FANS BUT KEEP THE CPU RUNNING IN "SLEEP" MODE). Goddess forbid you have a fucking OEM license preinstalled, fuck knows what settings they set to ridiculous values and hid from you. The wake timers for Windows Update are absolutely bonkers. The system won't fucking go to sleep, or goes to sleep when it shouldn't, or doesn't sleep properly, or doesn't wake properly, what the fuck. Even fucking Linux has better power management today, when 10 years ago closing your laptop lid was a one way ticket to kernel panic land. 12 | 13 | ## Problems 14 | 15 | Problem | Solution 16 | --- | --- 17 | Classic Start doesn't show the "sleep" option | Low power idle 18 | The computer overheats during sleep | Low power idle 19 | The computer randomly wakes up from sleep | Wake timers 20 | The computer goes to sleep soon after you lock it | System unattended sleep timeout / Console lock display off timeout 21 | Windows Update is still a pain in the ass and still wakes your PC | 22 | The computer wakes up exactly X minutes after you put it to sleep | Hibernation timers 23 | 24 | 25 | # Solutions 26 | 27 | ## Low power idle 28 | 29 | ### Check 30 | 31 | To check if you have this problem: From an admin powershell run `powercfg -a`. If sleep state `Standby (S0 Low Power Idle)` is available, then you have this problem. 32 | 33 | ### Fix 34 | 35 | To fix it, open regedit and set `HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Power\CsEnabled` to `0`. Reboot. 36 | 37 | Now `powercfg -a` should show one of the other standby states: S1, S2, or S3. If it doesn't, you're SOL and will have to choose between no sleep and S0 sleep. 38 | 39 | ### Explanation 40 | 41 | In this configuration, the computer executes a "connected standby" when it's asked to sleep. Processes still run but on certain laptops fans may not spin. Windows still draws an obscene amount of power. Other standby modes are disabled by Windows when this is enabled, which means that even if the system is capable of a more reasonable sleep, then it still won't do it. Applications such as Classic Start specifically look for S1, S2, or S3 and won't display the sleep option if it isn't one of these. 42 | 43 | ## Wake timers 44 | 45 | ### Check 46 | 47 | If your windows machine randomly wakes up from sleep, you have this problem. It's either windows update, or something else. 48 | 49 | From admin shell run `powercfg -waketimers`. It should show what's the problem. 50 | 51 | ### Fix 52 | 53 | If you want to only ban windows update from doing this, check here (COMING SOON). If you want to ban everything from waking your computer up, read on. 54 | 55 | Go to Control Panel -> Power Options -> Change plan settings -> Change advanced power settings -> Sleep -> Allow wake timers. Set to Disable. 56 | 57 | If the option isn't available, set the registry key `HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Power\PowerSettings\238C9FA8-0AAD-41ED-83F4-97BE242C8F20\BD3B718A-0680-4D9D-8AB2-E1D2B4AC806D\Attributes` to `2`. Close and open the advanced power settings and you should see the option. 58 | 59 | ### Explanation 60 | 61 | Windows mostly uses wake timers with Windows Update, which is usually the primary culprit, however other systems may rely on wake timers themselves. The above registry key controls whether the option is visible or not. You can find a list of other interesting registry keys by running `powercfg -q` (query). That list may not be exhaustive, [here's an exhaustive one](https://bitsum.com/known-windows-power-guids/). 62 | 63 | ## System unattended sleep timeout / Console lock display off timeout 64 | 65 | ### Check 66 | 67 | You have this problem if your computer goes to sleep when you issue a lock command and wait a few minutes, even with sleep timeout set to Never. 68 | 69 | ### Fix 70 | 71 | Go to Control Panel -> Power Options -> Change plan settings -> Change advanced power settings -> Sleep -> System unattended sleep timeout. Set to 0. 72 | 73 | Note that this will also cause the system to not go back to sleep anymore after a wake timer triggers. If it doesn't work make sure to revert this one. 74 | 75 | If the option isn't available, set the registry key `HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Power\PowerSettings\238C9FA8-0AAD-41ED-83F4-97BE242C8F20\7bc4a2f9-d8fc-4469-b07b-33eb785aaca0\Attributes` to `2`. Close and open the advanced power settings and you should see the option. 76 | 77 | #### Alternative fix: Console lock display off timeout 78 | 79 | If that doesn't work (such as on a Zenbook Pro Duo), set `HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Power\PowerSettings\7516b95f-f776-4464-8c53-06167f40cc99\8EC4B3A5-6868-48c2-BE75-4F3044BE88A7\Attributes` to `2` then in advanced power settings set Display -> Console lock display off timeout to 0. To get the screen to still blank out after a while and avoid burn-in, set a blank screensaver with a timeout of however many minutes you want. This seems a bit finicky so test that the screensaver triggers both while unlocked and while locked before relying on it. 80 | 81 | ### Explanation 82 | 83 | Windows has this hidden config where the system sleeps when it's "unattended" as an independent timer from the usual sleep timer. The primary purpose of this key is to set the go-back-to-sleep timeout for when wake timers wake the computer, but it has the side effect of affecting the sleep timeout on the lockscreen on some computers. If that doesn't work, it may be the case that a screen off triggers sleep, in which case we disable the "display off" timeout and replace it with a blank screensaver which doesn't turn off the display, but simply sets it to black. 84 | 85 | ## Hibernation timers 86 | 87 | ### Check 88 | 89 | If your computer: 90 | 1) Has no wake timers when you run `powercfg -waketimers` and/or `Get-ScheduledTask | ? {$_.Settings.WakeToRun}` (elevated prompt), and 91 | 2) Wakes up precisely X minutes after sleeping (e.g. the computer always wakes up exactly 3 hours after it was put to sleep) 92 | 93 | Immediately when you notice that your computer has woken up, open eventvwr and go to Windows Logs -> System and check the approximate time when the device woke up. If you find an event with source "Power-Troubleshooter" and in the XML view you see a WakeSourceText of "ACPI Wake Alarm" then there's your problem. 94 | 95 | ### Fix 96 | 97 | Go to Control Panel -> Power Options -> Change plan settings -> Change advanced power settings -> Sleep -> Hibernate after and set both to 0 (Never). 98 | 99 | ### Explanation 100 | 101 | ACPI wake timers are lower level and are not your usual wake timers. If I were to guess (and this may be completely wrong), Windows sets one of these to fire before the computer goes to sleep when it's configured to hibernate after a certain amount of time (end speculation). The computer will wake but fail to correctly go into hibernation, which results in you being hung out to dry. If you're lucky, it goes back to sleep and you never notice. 102 | -------------------------------------------------------------------------------- /src/cloud/aws/event_examples/cloudtrail_management_events.md: -------------------------------------------------------------------------------- 1 | # Cloudtrail Management events in CloudWatch logs 2 | 3 | This is how cloudtrail management events appear in cloudwatch logs. The below examples are NOT Insights events. 4 | 5 | These are sampled from real events. The values are consistent, so if you see the same value (like `AROA00000000000`) in multiple places in the same event, it was the same value everywhere. 6 | 7 | [**AWS DOCUMENTATION ON WHAT EACH FIELD IS AND WHAT EVENT VERSIONS CONTAIN WHAT FIELDS.**](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-event-reference-record-contents.html) 8 | 9 | ## GetPublicAccessBlock, IAM user, failure 10 | 11 | ```json 12 | { 13 | "eventVersion": "1.05", 14 | "userIdentity": { 15 | "type": "IAMUser", 16 | "principalId": "AIDA000000000000000", 17 | "arn": "arn:aws:iam::555555555555:user/asdf", 18 | "accountId": "555555555555", 19 | "accessKeyId": "ASIA1111111111111", 20 | "userName": "asdf", 21 | "sessionContext": { 22 | "attributes": { 23 | "mfaAuthenticated": "true", 24 | "creationDate": "2019-09-17T16:04:02Z" 25 | } 26 | }, 27 | "invokedBy": "signin.amazonaws.com" 28 | }, 29 | "eventTime": "2019-09-17T16:04:44Z", 30 | "eventSource": "s3.amazonaws.com", 31 | "eventName": "GetBucketPublicAccessBlock", 32 | "awsRegion": "ap-southeast-1", 33 | "sourceIPAddress": "1.2.3.4", 34 | "userAgent": "signin.amazonaws.com", 35 | "errorCode": "NoSuchPublicAccessBlockConfiguration", 36 | "errorMessage": "The public access block configuration was not found", 37 | "requestParameters": { 38 | "publicAccessBlock": [ 39 | "" 40 | ], 41 | "host": [ 42 | "bucket.s3.ap-southeast-1.amazonaws.com" 43 | ], 44 | "bucketName": "bucket" 45 | }, 46 | "responseElements": null, 47 | "additionalEventData": { 48 | "SignatureVersion": "SigV4", 49 | "CipherSuite": "ECDHE-RSA-AES128-SHA", 50 | "AuthenticationMethod": "AuthHeader", 51 | "vpcEndpointId": "vpce-abcdefab" 52 | }, 53 | "requestID": "FFFFFFFFFFFFFFF", 54 | "eventID": "00000000-1111-2222-3333-444444444444", 55 | "eventType": "AwsApiCall", 56 | "recipientAccountId": "555555555555", 57 | "vpcEndpointId": "vpce-abcdefab" 58 | } 59 | ``` 60 | 61 | ## PutBucketPolicy, IAM user, success 62 | 63 | ```json 64 | { 65 | "eventVersion": "1.05", 66 | "userIdentity": { 67 | "type": "IAMUser", 68 | "principalId": "AIDA0000000000000000", 69 | "arn": "arn:aws:iam::555555555555:user/testymctestface", 70 | "accountId": "555555555555", 71 | "accessKeyId": "ASIA111111111111111", 72 | "userName": "testymctestface", 73 | "sessionContext": { 74 | "attributes": { 75 | "mfaAuthenticated": "true", 76 | "creationDate": "2019-09-17T15:38:13Z" 77 | } 78 | }, 79 | "invokedBy": "signin.amazonaws.com" 80 | }, 81 | "eventTime": "2019-09-17T16:17:04Z", 82 | "eventSource": "s3.amazonaws.com", 83 | "eventName": "PutBucketPolicy", 84 | "awsRegion": "us-east-1", 85 | "sourceIPAddress": "1.2.3.4", 86 | "userAgent": "signin.amazonaws.com", 87 | "requestParameters": { 88 | "bucketName": "my-bucket", 89 | "bucketPolicy": { 90 | "Version": "2012-10-17", 91 | "Statement": [ 92 | { 93 | "Sid": "Test2", 94 | "Effect": "Allow", 95 | "Principal": { 96 | "AWS": "arn:aws:iam::666666666666:root" 97 | }, 98 | "Action": "s3:ListBucket", 99 | "Resource": "arn:aws:s3:::my-bucket" 100 | } 101 | ] 102 | }, 103 | "host": [ 104 | "s3.amazonaws.com" 105 | ], 106 | "policy": [ 107 | "" 108 | ] 109 | }, 110 | "responseElements": null, 111 | "additionalEventData": { 112 | "SignatureVersion": "SigV4", 113 | "CipherSuite": "ECDHE-RSA-AES128-SHA", 114 | "AuthenticationMethod": "AuthHeader", 115 | "vpcEndpointId": "vpce-abcdefab" 116 | }, 117 | "requestID": "FFFFFFFFFFFFFF", 118 | "eventID": "00000000-1111-2222-3333-444444444444", 119 | "eventType": "AwsApiCall", 120 | "recipientAccountId": "555555555555", 121 | "vpcEndpointId": "vpce-abcdefab" 122 | } 123 | ``` 124 | 125 | ## AssumeRole, AWSService, both request and response elements 126 | 127 | ```json 128 | { 129 | "eventVersion": "1.05", 130 | "userIdentity": { 131 | "type": "AWSService", 132 | "invokedBy": "trustedadvisor.amazonaws.com" 133 | }, 134 | "eventTime": "2020-06-16T17:21:36Z", 135 | "eventSource": "sts.amazonaws.com", 136 | "eventName": "AssumeRole", 137 | "awsRegion": "eu-central-1", 138 | "sourceIPAddress": "trustedadvisor.amazonaws.com", 139 | "userAgent": "trustedadvisor.amazonaws.com", 140 | "requestParameters": { 141 | "roleArn": "arn:aws:iam::555555555555:role/aws-service-role/trustedadvisor.amazonaws.com/AWSServiceRoleForTrustedAdvisor", 142 | "roleSessionName": "TrustedAdvisor_555555555555_aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee", 143 | "durationSeconds": 3600 144 | }, 145 | "responseElements": { 146 | "credentials": { 147 | "accessKeyId": "ASIA11111111111111", 148 | "expiration": "Jun 16, 2020 6:21:36 PM", 149 | "sessionToken": "loooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooong/base64/encoded/token=" 150 | }, 151 | "assumedRoleUser": { 152 | "assumedRoleId": "AROA0000000000000000:TrustedAdvisor_555555555555_aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee", 153 | "arn": "arn:aws:sts::555555555555:assumed-role/AWSServiceRoleForTrustedAdvisor/TrustedAdvisor_555555555555_aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee" 154 | } 155 | }, 156 | "requestID": "00000000-1111-2222-3333-444444444444", 157 | "eventID": "01010101-1212-2323-3434-454545454545", 158 | "resources": [ 159 | { 160 | "accountId": "555555555555", 161 | "type": "AWS::IAM::Role", 162 | "ARN": "arn:aws:iam::555555555555:role/aws-service-role/trustedadvisor.amazonaws.com/AWSServiceRoleForTrustedAdvisor" 163 | } 164 | ], 165 | "eventType": "AwsApiCall", 166 | "recipientAccountId": "555555555555", 167 | "sharedEventID": "10101010-2121-3232-4343-545454545454" 168 | } 169 | ``` 170 | 171 | ## event version 1.06 172 | 173 | ```json 174 | { 175 | "eventVersion": "1.06", 176 | "userIdentity": { 177 | "type": "AssumedRole", 178 | "principalId": "AROA000000000000000:QuickSight-RoleSession-9876543210", 179 | "arn": "arn:aws:sts::555555555555:assumed-role/aws-quicksight-service-role-v0/QuickSight-RoleSession-9876543210", 180 | "accountId": "555555555555", 181 | "accessKeyId": "ASIA111111111111", 182 | "sessionContext": { 183 | "sessionIssuer": { 184 | "type": "Role", 185 | "principalId": "AROA000000000000000", 186 | "arn": "arn:aws:iam::555555555555:role/service-role/aws-quicksight-service-role-v0", 187 | "accountId": "555555555555", 188 | "userName": "aws-quicksight-service-role-v0" 189 | }, 190 | "webIdFederationData": {}, 191 | "attributes": { 192 | "creationDate": "2019-11-19T16:00:13Z", 193 | "mfaAuthenticated": "false" 194 | } 195 | } 196 | }, 197 | "eventTime": "2019-11-19T16:00:23Z", 198 | "eventSource": "athena.amazonaws.com", 199 | "eventName": "GetQueryExecution", 200 | "awsRegion": "us-east-1", 201 | "sourceIPAddress": "1.2.3.4", 202 | "userAgent": "sbAthenaJDBCDriver/2.0, aws-internal/3", 203 | "requestParameters": { 204 | "queryExecutionId": "12121212-3434-5656-7878-909090909090" 205 | }, 206 | "responseElements": null, 207 | "requestID": "00000000-1111-2222-3333-444444444444", 208 | "eventID": "99999999-8888-7777-6666-555555555555", 209 | "readOnly": true, 210 | "eventType": "AwsApiCall", 211 | "managementEvent": true, 212 | "recipientAccountId": "555555555555" 213 | } 214 | ``` 215 | -------------------------------------------------------------------------------- /src/cloud/aws/services/s3_bucket_policy_weird_behaviors.md: -------------------------------------------------------------------------------- 1 | # What You See is Not What You Get: Weird Behaviors in S3 Bucket Policies 2 | 3 | Have you ever seen an S3 bucket policy change itself with no human intervention? 4 | 5 | Have you ever encountered a 500 "Please try again later" error when setting an s3 bucket policy? 6 | 7 | Ever wondered why you got "Invalid Principal" when writing a bucket policy, even though there's nothing obviously wrong? 8 | 9 | Why is that one old bucket showing access to `AIDAXYZTABCDEFGHIJK` when you clearly didn't do that? 10 | 11 | And what the hell is a Canonical ID? 12 | 13 | AWS's documentation is confusing and contradictory on this matter. Let's take a deeper dive. 14 | 15 | ## Table of Contents 16 | 17 | 18 | * [Table of Contents](#table-of-contents) 19 | * [A necessary refresher: how access works in AWS, and why cross-account access is different](#a-necessary-refresher-how-access-works-in-aws-and-why-cross-account-access-is-different) 20 | + [IAM policies: principals and resources](#iam-policies-principals-and-resources) 21 | + [Examples](#examples) 22 | + [How policies interact](#how-policies-interact) 23 | * [What AWS tells you...](#what-aws-tells-you) 24 | + [How to grant cross-account access](#how-to-grant-cross-account-access) 25 | + [Example](#example) 26 | * [What AWS doesn't tell you: how bucket policies ACTUALLY work](#what-aws-doesnt-tell-you-how-bucket-policies-actually-work) 27 | + [Saving and retrieving bucket policies](#saving-and-retrieving-bucket-policies) 28 | + [AWS's internal representation](#awss-internal-representation) 29 | * [So what's the problem?](#so-whats-the-problem) 30 | + [If a user is deleted, all bucket policies with that user will appear to have changed](#if-a-user-is-deleted-all-bucket-policies-with-that-user-will-appear-to-have-changed) 31 | + [You cannot edit a bucket policy that references a deleted principal](#you-cannot-edit-a-bucket-policy-that-references-a-deleted-principal) 32 | + [If you delete a user/role and create one with the same ARN, all resource-based access will break](#if-you-delete-a-userrole-and-create-one-with-the-same-arn-all-resource-based-access-will-break) 33 | - [The AWS documentation is plain wrong](#the-aws-documentation-is-plain-wrong) 34 | + [You can't create a bucket policy before you've created the principal that is granted access](#you-cant-create-a-bucket-policy-before-youve-created-the-principal-that-is-granted-access) 35 | + [Certain bucket policies will result in a cryptic 500 error](#certain-bucket-policies-will-result-in-a-cryptic-500-error) 36 | * [Security Implications](#security-implications) 37 | + [Brute-forcing valid principal names is possible](#brute-forcing-valid-principal-names-is-possible) 38 | + [User compromise will break cross-account access](#user-compromise-will-break-cross-account-access) 39 | + [Explicit denies will stop working if the principal is deleted and recreated](#explicit-denies-will-stop-working-if-the-principal-is-deleted-and-recreated) 40 | + [Canonical IDs offer no extra security](#canonical-ids-offer-no-extra-security) 41 | * [Conclusion](#conclusion) 42 | 43 | ## A necessary refresher: how access works in AWS, and why cross-account access is different 44 | 45 | Feel free to skip this if you have a really good understanding of AWS access. 46 | 47 | Humans or automations do stuff in AWS. For that, they need permissions. 48 | 49 | These permissions are encoded in "policies", which are JSON documents. [Policies are really complicated](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html), but in a simplified way they have a list of statements. Each statement has: 50 | * A list of actions 51 | * If the actions are `Allow`ed or `Deny`ed 52 | * Who is allowed (or denied) to perform those actions 53 | * What the targets of the actions can (or cannot) be 54 | * Optionally, any additional conditions that need to apply 55 | 56 | See "Examples" below and [a whole bunch of examples from AWS here](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_examples.html). 57 | 58 | ### IAM policies: principals and resources 59 | 60 | For the purpose of access control, there are entities which perform actions, known as "principals", and entities that are the targets or actions, known as "resources". Some entities (such as IAM roles) can act as either principals or resources depending on context. 61 | 62 | For the most part, you use "IAM policies" to control access (the following is a bit simplified): 63 | 64 | * Identity-based policies: an identity-based policy (also called a "principal-based" policy) is "attached" to a principal and grants or denies the principal access to one or more resources. 65 | * Resource-based policies: a resource-based policy is "attached" to a resource and grants or denies one or more principals access to the resource 66 | 67 | When you have both an identity policy and a resource policy, the resulting access depends on whether the principal and resource are both in the same AWS account or are in different AWS accounts. 68 | 69 | ### Examples 70 | 71 | Here's an identity-based policy that is attached to an EC2 instance and allows it to assume a certain role. This identity-based policy only grants access to the EC2 instance that it is attached to. If you want other EC2 instances to have this permission, you need to attach it to those as well. 72 | 73 | ```json 74 | { 75 | "Version": "2012-10-17", 76 | "Statement": [ 77 | { 78 | "Effect": "Allow", 79 | "Action": "sts:AssumeRole", 80 | "Resource": "arn:aws:iam::111111111111:role/ROLENAME" 81 | } 82 | ] 83 | } 84 | ``` 85 | 86 | Here's a resource-based policy that is attached to a SecretsManager secret. This grants access to the `anaya` user to that secret. **I know it says `"Resource": "*"` but because this is attached to a single SecretsManager secret, the policy only grants access to that secret, not all secrets.** To grant access to more secrets, you need to attach the policy to those as well. 87 | 88 | ```json 89 | { 90 | "Version": "2012-10-17", 91 | "Statement": [ 92 | { 93 | "Effect": "Allow", 94 | "Action": "secretsmanager:*", 95 | "Principal": {"AWS": "arn:aws:iam::123456789012:user/anaya"}, 96 | "Resource": "*" 97 | } 98 | ] 99 | } 100 | ``` 101 | 102 | ### How policies interact 103 | 104 | Here's a summary of how identity and resource policies interact where the access is within the same AWS account: 105 | 106 | Situation | Outcome (Same AWS account) 107 | ---|--- 108 | No identity policy exists, no resource policy exists | No access 109 | Identity policy only | Access according to identity policy 110 | Resource policy only | Access according to resource policy 111 | Both identity and resource policies | UNION of access of identity and resource policies (except if there's an explicit deny or other edge cases as per the policy evaluation logic) 112 | 113 | And for access into a different account (known as "cross-account" access): 114 | 115 | Situation | Outcome (Cross-account access) 116 | ---|--- 117 | No identity policy exists, no resource policy exists | No access 118 | Identity policy only | No access 119 | Resource policy only | No access 120 | Both identity and resource policies | INTERSECTION of access of identity and resource policies 121 | 122 | N.B. Specifically for cross-account access, you can specify an entire AWS account (e.g. `arn:aws:iam::123456789012:root`) in a resource policy to grant access to *all* principals in that account. You still need a corresponding identity policy for the access to take effects, as it is still the intersection of access. An AWS account can be referenced by the account ARN, which contains the account number (`arn:aws:iam::123456789012:root`) or by a "Canonical ID" (`1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef`). The Canonical ID is an obfuscated form of an AWS account ARN and otherwise acts the same. 123 | 124 | Further reading on policies: 125 | * [Identity-based policies and resource-based policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_identity-vs-resource.html) 126 | * [Policy evaluation logic](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_evaluation-logic.html) 127 | * [Cross-account policy evaluation logic](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_evaluation-logic-cross-account.html) 128 | 129 | ## What AWS tells you... 130 | 131 | ### How to grant cross-account access 132 | 133 | [This is what AWS tells you you should do to grant cross-account access to an s3 bucket](https://aws.amazon.com/premiumsupport/knowledge-center/cross-account-access-s3/): 134 | 135 | 1. Create a Bucket in Account A and a User in Account B 136 | 1. Create an identity policy that allows the User in Account B access to the Bucket in Account A. Attach it to the User in Account B. 137 | 1. Create a resource policy ("bucket policy") that allows the User in Account B access to the Bucket in Account A. Attach it to the Bucket in Account A. 138 | 1. Done! 139 | 140 | ### Example 141 | 142 | This is what AWS tells you to attach to User B: 143 | 144 | ```json 145 | { 146 | "Version": "2012-10-17", 147 | "Statement": [ 148 | { 149 | "Effect": "Allow", 150 | "Action": [ 151 | "s3:GetObject", 152 | "s3:PutObject", 153 | "s3:PutObjectAcl" 154 | ], 155 | "Resource": "arn:aws:s3:::AccountABucketName/*" 156 | 157 | } 158 | ] 159 | } 160 | ``` 161 | 162 | And this is what AWS tells you to attach to bucket A: 163 | 164 | ```json 165 | { 166 | "Version": "2012-10-17", 167 | "Statement": [ 168 | { 169 | "Effect": "Allow", 170 | "Principal": { 171 | "AWS": "arn:aws:iam::AccountB:user/AccountBUserName" 172 | }, 173 | "Action": [ 174 | "s3:GetObject", 175 | "s3:PutObject", 176 | "s3:PutObjectAcl" 177 | ], 178 | "Resource": [ 179 | "arn:aws:s3:::AccountABucketName/*" 180 | ] 181 | } 182 | ] 183 | } 184 | ``` 185 | 186 | Further discussion on identity-based policies work is outside the scope of this document. Bucket policies are where things can go very, very wrong, in non-obvious ways. 187 | 188 | ## What AWS doesn't tell you: how bucket policies ACTUALLY work 189 | 190 | **The JSON that you've seen so far for bucket policies is not the actual internal representation of a bucket policy.** In a bucket policy, what you see is not what you actually get, and this is the source of many problems. 191 | 192 | There is some amount of guesswork in what follows since I don't have internal AWS access, but to my knowledge this is how things work in effect (it may be implemented differently). 193 | 194 | ### Saving and retrieving bucket policies 195 | 196 | When you click "save" on a bucket policy: 197 | 198 | 1. AWS performs basic validation on the policy (JSON is valid, the bucket ARN is correct etc.) 199 | 1. AWS reads the bucket policy and validates if all the principals in the policy exist. If any of the principals fail to validate, you get the error "Invalid Principal in Policy" 200 | 1. AWS converts all the principal ARNs into a special backend representation: 201 | 1. For users and roles, it looks up the user/role arn and converts them into a "Principal ID". 202 | 1. For AWS accounts, it looks up the AWS account number (or Canonical ID) and converts it into some internal representation. 203 | 1. This conversion *does not happen inside Condition Keys*, only in the Principal section. 204 | 1. AWS stores the converted bucket policy. 205 | 206 | When you want to see a bucket policy: 207 | 1. AWS reads the bucket policy and converts all the special backend representation IDs in the Principal section back into ARNs. If any conversion fails (e.g. a user no longer exists), it skips it. 208 | 1. AWS displays to you the resulting bucket policy. 209 | 210 | ~~These behaviors are specific for bucket policies. Other types of resource-based policies may or may not behave completely differently (like SecretsManager resource policies or assume role policies).~~ 211 | 212 | These behaviors "should" apply for *all resource policies*. 213 | 214 | ### AWS's internal representation 215 | 216 | All users have an internal unique ID, called a "Principal ID" (of the form `AIDAxxxxxxxxxxxxxxxxx`), that is distinct from the ARN of the user. 217 | 218 | All accounts have a unique ID called a "Canonical ID" that is distinct from the account number (but behaves the exact same). The account number is preferred by AWS over the Canonical ID. In fact, if you put in a Canonical ID in a bucket policy, AWS will display the account number next time you open the policy. 219 | 220 | What you see | What AWS stores internally 221 | --- | --- 222 | `arn:aws:iam::123456789012:user/SomeUser` | `AIDAxxxxxxxxxxxxxxxxx` (a user Principal ID) 223 | `arn:aws:iam::123456789012:role/SomeRole` | `AROAxxxxxxxxxxxxxxxxx` (a role Principal ID) 224 | `arn:aws:iam::123456789012:root` | A unique identifier per account (that is not the Canonical ID) 225 | 226 | Further reading: 227 | 228 | * [Finding your account canonical ID](https://docs.aws.amazon.com/general/latest/gr/acct-identifiers.html#findingcanonicalid) 229 | * [Finding the IAM identifiers (Principal IDs)](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_identifiers.html#identifiers-unique-ids) 230 | * [Principal IDs, and AWS warnings on how cross-account access will break](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_principal.html). This is a fairly obscure piece of documentation. 231 | 232 | ## So what's the problem? 233 | 234 | ### If a user is deleted, all bucket policies with that user as aprincipal will appear to have changed 235 | 236 | Let's say you have this bucket policy: 237 | 238 | ```json 239 | { 240 | "Version": "2012-10-17", 241 | "Statement": [ 242 | { 243 | "Effect": "Allow", 244 | "Principal": { 245 | "AWS": "arn:aws:iam::123456789012:user/Test" 246 | }, 247 | "Action": [ 248 | "s3:ListBucket", 249 | "s3:GetBucketLocation" 250 | ], 251 | "Resource": "arn:aws:s3:::TestyMcTestFace" 252 | } 253 | ] 254 | } 255 | ``` 256 | 257 | And now you delete the user `Test`. The bucket policy will now show: 258 | 259 | ```json 260 | { 261 | "Version": "2012-10-17", 262 | "Statement": [ 263 | { 264 | "Effect": "Allow", 265 | "Principal": { 266 | "AWS": "AIDAxxxxxxxxxxxxxxxxx" 267 | }, 268 | "Action": [ 269 | "s3:ListBucket", 270 | "s3:GetBucketLocation" 271 | ], 272 | "Resource": "arn:aws:s3:::elfakyn-testytest" 273 | } 274 | ] 275 | } 276 | ``` 277 | 278 | *The bucket policy never actually changed.* It is now displayed differently to you because the internally-stored ID can no longer be converted back into an ARN for display purposes. 279 | 280 | ### You cannot edit a bucket policy that references a deleted principal 281 | 282 | Because there is validation every time you save a bucket policy, if the bucket policy references a deleted principal, it will error with "Invalid Principal" every time you try to edit it. You actually have to fix the invalid principal before you can save the policy. 283 | 284 | But as long as you don't edit it, it will remain with an invalid principal without any issues. 285 | 286 | ### If you delete a user/role and create one with the same ARN, all resource-based access will break 287 | 288 | When you delete a user and create one with the same ARN, its Principal ID (`AIDAxxxxxxxxxxxxxxxxx`) changes. 289 | 290 | Any cross-account bucket policies use the Principal ID and not the ARN to grant access. When the user is deleted and recreated: 291 | 292 | 1. Cross-account access will break 293 | 1. Cross-account bucket policies will now display `AIDAxxxxxxxxxxxxxxxxx` instead of the user ARN 294 | 295 | To fix the problem, you have to manually go to all bucket policies and replace them with the user ARN all over again. 296 | 297 | The same happens with roles, except it displays `AROAxxxxxxxxxxxxxxxxx` 298 | 299 | Access within the same account will break if it is granted using the bucket policy (and not an identity-based policy). But typically, same-account s3 access is granted using identity-based policies and not bucket policies, so same-account access will not break in most cases. 300 | 301 | #### The AWS documentation is plain wrong 302 | 303 | [The AWS documentation claims](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_identifiers.html#identifiers-unique-ids): 304 | 305 | > Suppose that the employee named David leaves your company and you delete the corresponding IAM user. But later another employee named David starts and you create a new IAM user named David. If the bucket policy specifies the David IAM user, the policy allows the new David to access information that was left by the former David. 306 | 307 | I have tested this, and it is incorrect. If you view the bucket policy, it will show the old Principal ID and will not retain the user ARN, even within the same account. 308 | 309 | ### You can't create a bucket policy before you've created the principal that is granted access 310 | 311 | This one is pretty self-explanatory. If a principal does not exist when you create the policy, it will return an Invalid Principal error. That means that, if you grant access to a third party via a bucket policy, you'll have to coordinate such that the principal is created first. 312 | 313 | ### Certain bucket policies will result in a cryptic 500 error 314 | 315 | If a bucket policy is invalid in such a way that it breaks the internal validator, it will return a 500 error instead of anything useful. Here's one such example: 316 | 317 | ```json 318 | { 319 | "Version": "2012-10-17", 320 | "Id": "whatever", 321 | "Statement": [ 322 | { 323 | "Sid": "ReadOnly", 324 | "Effect": "Allow", 325 | "Principal": { 326 | "AWS": [] 327 | }, 328 | "Action": "s3:GetObject", 329 | "Resource": "arn:aws:s3:::my-bucket/*" 330 | } 331 | ] 332 | } 333 | ``` 334 | 335 | [Read more here.](./s3_bucket_policy_500_errors.md) 336 | 337 | If you have any automation that handles bucket policies, you need to be able to handle such errors. 338 | 339 | ## Workaround by using condition keys 340 | 341 | You can choose to instead use [a PrincipalArn inside a Condition Key](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-principalarn). These do not get converted internally, so will remain valid even if the principal is deleted and recreated. This means, however, that the above David scenario may happen. If the principal is deleted and a new principal with the same ARN is created, you might not actually want access to persist. It depends on your use case. 342 | 343 | ## Security Implications 344 | 345 | ### Brute-forcing valid principal names is possible 346 | 347 | You can use this mechanism to brute force valid AWS accounts as well as valid principals within an AWS account (such as user names). To do this, simply attempt to apply a bucket policy. If the principal is valid, it will apply successfully. If the principal doesn't exist, it will fail. AWS knows about this issue and it is intended behavior, so don't go submitting a bug report on this one; I already did, and they WontFix. 348 | 349 | Principal names may reveal potentially sensitive information such as: 350 | 351 | * Who works for your company 352 | * What clients you do business with 353 | * What technologies you use 354 | * If you do anything sketchy security-wise (such as all-powerful machine users) 355 | * Potential vulnerabilities 356 | 357 | ### User compromise will break cross-account access 358 | 359 | If the credentials for an AWS user are compromised, AWS will automatically quarantine the user (they monitor places like GitHub for leaks) and the AWS Abuse Team will ABSOLUTELY INSIST that you delete the user and create a new one (in the strongest possible terms: "You're in violation of our TOS, you must absolutely delete this user, you have no choice"). 360 | 361 | You are allowed to create a new user with the same name and ARN, but cross-account access will still break. If this cross-account access involves third parties, you're in for an awkward conversation... 362 | 363 | ### Explicit denies will stop working if the principal is deleted and recreated 364 | 365 | If you rely on explicit denies on a bucket policy to deny access to specific principals (such as users or roles), those explicit denies will stop working if the principals are deleted and recreated, since they refer to the old Principal ID. 366 | 367 | ### Canonical IDs offer no extra security 368 | 369 | AWS advertises Canonical IDs (`1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef`) as obfuscated versions of an AWS account ARN (`arn:aws:iam::123456789012:root`). If you want to give a third party your account ARN (for instance, so they can create a cross-account bucket policy granting it access), but don't want to divulge your account number, you can give them the Canonical ID, or so the story goes. 370 | 371 | But this doesn't actually work. All you have to do is save the Canonical ID in a bucket policy, and next time you view the policy, [AWS will helpfully convert it back into an account ARN with an account number.](https://docs.amazonaws.cn/en_us/AmazonS3/latest/dev/s3-bucket-user-policy-specifying-principal-intro.html) Nifty! 372 | 373 | ## Conclusion 374 | 375 | AWS works in arcane ways. It is almost certain that you will find every idiosyncracy of the system documented somewhere, but it is just as certain that it *will not be in the first place you're looking*. You may find confusing and contradictory documentation. 376 | 377 | If you're about to make a mission-critical change, open an AWS support ticket and get the correct team, in this case, s3, to explicitly tell you what will happen (they might still be wrong!). Because whatever happens, it will be a surprise to everyone. 378 | --------------------------------------------------------------------------------