├── .gitignore ├── README.md ├── ecs-instance-logs-role.json ├── ecs-logs-init-aml-1.sh └── ecs-logs-init-aml-2.sh /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_store -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | A User Data Launch script to ESSENTIAL logging in CloudWatch for ECS container instances launched either one-off or via Launch Configuration + Auto Scaling Group. Based on this [log file](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_cloudwatch_logs.html#cwlogs_user_data) from AWS. 2 | 3 | To use - 4 | 5 | 1. Ensure that your instances / launch configuration have the following IAM permissions: 6 | 7 | ``` 8 | "logs:CreateLogGroup", 9 | "logs:CreateLogStream", 10 | "logs:PutLogEvents", 11 | "logs:DescribeLogStreams" 12 | ``` 13 | 14 | 2. Grab either `ecs-logs-init-aml-1.sh`for Amazon Linux 1 based AMIs or `ecs-logs-init-aml-2.sh` for Amazon Linux 2 AMIs. If you're keeping up to date with the ECS Optimized Image, you'll need the Amazon Linux 2 Script. 15 | 16 | 3. paste in the contents to your instance or Launch Configuration's User Data under `Advanced Details`. 17 | 18 | --- 19 | 20 | The Shell script `ecs-logs-init.sh` can be used as `User Data` for you EC2 instances and Launch Configurations for `ECS` clusters. It provides the following functionality: 21 | 22 | **a)** Tell the instance what cluster to join 23 | 24 | **b)** Install the `awslogs` and `jq` 25 | 26 | **c)** Setup the default points to send our Logs to 27 | 28 | **d)** Set the region to send our logs to (which by default will be the one our instance resides in) 29 | 30 | **e)** Setup logging meta data so that we can tell which logs belong to which clusters, services and instances. Logs the following: 31 | 32 | - Global Messages 33 | - Kernal Message 34 | - SSH Logins 35 | - Cloud Init Logs 36 | - Docker Logs 37 | - ECS Logs 38 | - IAM Role Audit Logs 39 | 40 | All named and organized in cloudwatch via log type -> container instance. For example, Docker logs will be organized in this manner: 41 | 42 | Log Group - `CLUSTER_global_messages` 43 | Log Stream - `CONTAINER_INSTANCE_ID/var/log/messages` 44 | 45 | You can modify the log group and log stream name to be whatever you'd like. 46 | 47 | **f)** Install the SSM Agent to allow for `Run Command` (not present by default on ECS Optimized AMIs) 48 | 49 | The Policy just allows for putting logs to Cloudwatch for these instances. You're ECS Container instances will generally have a role for the instances. Attach this policy to that role as well. 50 | 51 | --- 52 | 53 | ## Amazon Linux 2 Update 54 | 55 | Amazon's newest Amazon Linux 2 AMI changes up how longs work. First up, they're no longer referenced to as `awslogs`, but instead of `awslogsd`. Second, to start and top logs, you'll be using the `systemctl` instead of `service`. For example, the new start command to run logs is: 56 | 57 | `systemctl start awslogsd` 58 | 59 | And to start the service on each boot: 60 | 61 | `systemctl enable awslogsd.service` 62 | 63 | If you try to to use the old script in Amazon Linux 2, you'll get all sorts of nasty errors like... 64 | 65 | ``` 66 | cwlogs.push.stream - WARNING - somenumber - Thread-1 - No file is found with given path '/var/log/docker'. 67 | ``` 68 | 69 | ... and so on. 70 | 71 | --- 72 | 73 | More details can be found on this post: 74 | 75 | [How to Unify AWS ECS Logs in CloudWatch (and SSM Run Command)](http://start.jcolemorrison.com/how-to-setup-aws-ecs-logs-in-cloudwatch-and-ssm) -------------------------------------------------------------------------------- /ecs-instance-logs-role.json: -------------------------------------------------------------------------------- 1 | { 2 | "Version": "2012-10-17", 3 | "Statement": [ 4 | { 5 | "Effect": "Allow", 6 | "Action": [ 7 | "logs:CreateLogGroup", 8 | "logs:CreateLogStream", 9 | "logs:PutLogEvents", 10 | "logs:DescribeLogStreams" 11 | ], 12 | "Resource": [ 13 | "arn:aws:logs:*:*:*" 14 | ] 15 | } 16 | ] 17 | } -------------------------------------------------------------------------------- /ecs-logs-init-aml-1.sh: -------------------------------------------------------------------------------- 1 | Content-Type: multipart/mixed; boundary="==BOUNDARY==" 2 | MIME-Version: 1.0 3 | 4 | --==BOUNDARY== 5 | Content-Type: text/x-shellscript; charset="us-ascii" 6 | #!/bin/bash 7 | 8 | # cluster to join REPLACE `YOURCLUSTERNAMEHERE` OBVIOUSLY 9 | echo ECS_CLUSTER=YOURCLUSTERNAMEHERE >> /etc/ecs/ecs.config 10 | 11 | # Install awslogs and the jq JSON parser 12 | yum install -y awslogs jq 13 | 14 | # Inject the CloudWatch Logs configuration file contents 15 | cat > /etc/awslogs/awslogs.conf <<- EOF 16 | [general] 17 | state_file = /var/lib/awslogs/agent-state 18 | 19 | # Kernel Messages 20 | [/var/log/dmesg] 21 | file = /var/log/dmesg 22 | log_group_name = {cluster}_kernal_messages 23 | log_stream_name = {container_instance_id}/var/log/dmesg 24 | 25 | # Global Messages 26 | [/var/log/messages] 27 | file = /var/log/messages 28 | log_group_name = {cluster}_global_messages 29 | log_stream_name = {container_instance_id}/var/log/messages 30 | datetime_format = %b %d %H:%M:%S 31 | 32 | # SSH logs 33 | [/var/log/secure] 34 | file = /var/log/secure 35 | log_group_name = {cluster}_ssh_logs 36 | log_stream_name = {container_instance_id}/var/log/secure 37 | datetime_format = %b %d %H:%M:%S 38 | 39 | # Cloud Init Logs (results of User Data Scripts) 40 | [/var/log/cloud-init.log] 41 | file = /var/log/cloud-init.log 42 | log_group_name = {cluster}_cloud_init 43 | log_stream_name = {container_instance_id}/var/log/cloud-init.log 44 | datetime_format = %b %d %H:%M:%S 45 | 46 | [/var/log/cloud-init-output.log] 47 | file = /var/log/cloud-init-output.log 48 | log_group_name = {cluster}_cloud_init_output 49 | log_stream_name = {container_instance_id}/var/log/cloud-init-output.log 50 | datetime_format = %b %d %H:%M:%S 51 | 52 | # Docker Logs 53 | [/var/log/docker] 54 | file = /var/log/docker 55 | log_group_name = {cluster}_docker 56 | log_stream_name = {container_instance_id}/var/log/docker 57 | datetime_format = %Y-%m-%dT%H:%M:%S.%f 58 | 59 | # ECS Initialization Logs 60 | [/var/log/ecs/ecs-init.log] 61 | file = /var/log/ecs/ecs-init.log 62 | log_group_name = {cluster}_ecs_init 63 | log_stream_name = {container_instance_id}/var/log/ecs/ecs-init.log 64 | datetime_format = %Y-%m-%dT%H:%M:%SZ 65 | 66 | # ECS Agent Logs 67 | [/var/log/ecs/ecs-agent.log] 68 | file = /var/log/ecs/ecs-agent.log.* 69 | log_group_name = {cluster}_ecs_agent 70 | log_stream_name = {container_instance_id}/var/log/ecs/ecs-agent.log 71 | datetime_format = %Y-%m-%dT%H:%M:%SZ 72 | 73 | # IAM Role Audit Logs Logs 74 | [/var/log/ecs/audit.log] 75 | file = /var/log/ecs/audit.log.* 76 | log_group_name = {cluster}_ecs_iam_audit 77 | log_stream_name = {container_instance_id}/var/log/ecs/audit.log 78 | datetime_format = %Y-%m-%dT%H:%M:%SZ 79 | 80 | EOF 81 | 82 | --==BOUNDARY== 83 | Content-Type: text/x-shellscript; charset="us-ascii" 84 | #!/bin/bash 85 | 86 | # Set the region to send CloudWatch Logs data to (the region where the container instance is located) 87 | region=$(curl -s 169.254.169.254/latest/dynamic/instance-identity/document | jq -r .region) 88 | sed -i -e "s/region = us-east-1/region = $region/g" /etc/awslogs/awscli.conf 89 | 90 | # Install AWS SSM agent RPM for later mass commands 91 | # Not present by default on ECS Optimized AMI: 92 | # https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-manual-agent-install.html#agent-install-al 93 | 94 | # For x86_64 instances, so what you're probably using 95 | yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm 96 | 97 | # For arm64 instances, you'd run this: 98 | # yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_arm64/amazon-ssm-agent.rpm 99 | 100 | # For 32-bit: 101 | # yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_386/amazon-ssm-agent.rpm 102 | 103 | --==BOUNDARY== 104 | Content-Type: text/upstart-job; charset="us-ascii" 105 | 106 | #upstart-job 107 | description "Configure and start CloudWatch Logs agent on Amazon ECS container instance" 108 | author "Amazon Web Services" 109 | start on started ecs 110 | 111 | script 112 | exec 2>>/var/log/ecs/cloudwatch-logs-start.log 113 | set -x 114 | 115 | until curl -s http://localhost:51678/v1/metadata 116 | do 117 | sleep 1 118 | done 119 | 120 | # Grab the cluster and container instance ARN from instance metadata 121 | cluster=$(curl -s http://localhost:51678/v1/metadata | jq -r '. | .Cluster') 122 | container_instance_id=$(curl -s http://localhost:51678/v1/metadata | jq -r '. | .ContainerInstanceArn' | awk -F/ '{print $2}' ) 123 | 124 | # Replace the cluster name and container instance ID placeholders with the actual values 125 | sed -i -e "s/{cluster}/$cluster/g" /etc/awslogs/awslogs.conf 126 | sed -i -e "s/{container_instance_id}/$container_instance_id/g" /etc/awslogs/awslogs.conf 127 | 128 | # start aws logs 129 | service awslogs start 130 | chkconfig awslogs on 131 | 132 | # start ssm 133 | start amazon-ssm-agent 134 | end script 135 | --==BOUNDARY==-- -------------------------------------------------------------------------------- /ecs-logs-init-aml-2.sh: -------------------------------------------------------------------------------- 1 | Content-Type: multipart/mixed; boundary="==BOUNDARY==" 2 | MIME-Version: 1.0 3 | 4 | --==BOUNDARY== 5 | Content-Type: text/x-shellscript; charset="us-ascii" 6 | #!/usr/bin/env bash 7 | 8 | # cluster to join REPLACE `YOURCLUSTERNAMEHERE` OBVIOUSLY 9 | echo ECS_CLUSTER=YOURCLUSTERNAMEHERE >> /etc/ecs/ecs.config 10 | 11 | # Install awslogs and the jq JSON parser 12 | yum install -y awslogs jq 13 | 14 | # Install AWS SSM agent RPM for later mass commands 15 | # Not present by default on ECS Optimized AMI: 16 | # https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-manual-agent-install.html#agent-install-al 17 | 18 | # For x86_64 instances, so what you're probably using 19 | yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm 20 | 21 | # For arm64 instances, you'd run this: 22 | # yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_arm64/amazon-ssm-agent.rpm 23 | 24 | # For 32-bit: 25 | # yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_386/amazon-ssm-agent.rpm 26 | 27 | # Inject the CloudWatch Logs configuration file contents 28 | cat > /etc/awslogs/awslogs.conf <<- EOF 29 | [general] 30 | state_file = /var/lib/awslogs/agent-state 31 | 32 | # Kernel Messages 33 | [/var/log/dmesg] 34 | file = /var/log/dmesg 35 | log_group_name = {cluster}_kernal_messages 36 | log_stream_name = {container_instance_id}/var/log/dmesg 37 | 38 | # Global Messages 39 | [/var/log/messages] 40 | file = /var/log/messages 41 | log_group_name = {cluster}_global_messages 42 | log_stream_name = {container_instance_id}/var/log/messages 43 | datetime_format = %b %d %H:%M:%S 44 | 45 | # SSH logs 46 | [/var/log/secure] 47 | file = /var/log/secure 48 | log_group_name = {cluster}_ssh_logs 49 | log_stream_name = {container_instance_id}/var/log/secure 50 | datetime_format = %b %d %H:%M:%S 51 | 52 | # Cloud Init Logs (results of User Data Scripts) 53 | [/var/log/cloud-init.log] 54 | file = /var/log/cloud-init.log 55 | log_group_name = {cluster}_cloud_init 56 | log_stream_name = {container_instance_id}/var/log/cloud-init.log 57 | datetime_format = %b %d %H:%M:%S 58 | 59 | [/var/log/cloud-init-output.log] 60 | file = /var/log/cloud-init-output.log 61 | log_group_name = {cluster}_cloud_init_output 62 | log_stream_name = {container_instance_id}/var/log/cloud-init-output.log 63 | datetime_format = %b %d %H:%M:%S 64 | 65 | # Docker Logs 66 | [/var/log/docker] 67 | file = /var/log/docker 68 | log_group_name = {cluster}_docker 69 | log_stream_name = {container_instance_id}/var/log/docker 70 | datetime_format = %Y-%m-%dT%H:%M:%S.%f 71 | 72 | # ECS Initialization Logs 73 | [/var/log/ecs/ecs-init.log] 74 | file = /var/log/ecs/ecs-init.log 75 | log_group_name = {cluster}_ecs_init 76 | log_stream_name = {container_instance_id}/var/log/ecs/ecs-init.log 77 | datetime_format = %Y-%m-%dT%H:%M:%SZ 78 | 79 | # ECS Agent Logs 80 | [/var/log/ecs/ecs-agent.log] 81 | file = /var/log/ecs/ecs-agent.log.* 82 | log_group_name = {cluster}_ecs_agent 83 | log_stream_name = {container_instance_id}/var/log/ecs/ecs-agent.log 84 | datetime_format = %Y-%m-%dT%H:%M:%SZ 85 | 86 | # IAM Role Audit Logs Logs 87 | [/var/log/ecs/audit.log] 88 | file = /var/log/ecs/audit.log.* 89 | log_group_name = {cluster}_ecs_iam_audit 90 | log_stream_name = {container_instance_id}/var/log/ecs/audit.log 91 | datetime_format = %Y-%m-%dT%H:%M:%SZ 92 | 93 | EOF 94 | 95 | --==BOUNDARY== 96 | Content-Type: text/x-shellscript; charset="us-ascii" 97 | #!/usr/bin/env bash 98 | # Write the awslogs bootstrap script to /usr/local/bin/bootstrap-awslogs.sh 99 | cat > /usr/local/bin/bootstrap-awslogs.sh <<- 'EOF' 100 | #!/usr/bin/env bash 101 | exec 2>>/var/log/ecs/cloudwatch-logs-start.log 102 | set -x 103 | 104 | until curl -s http://localhost:51678/v1/metadata 105 | do 106 | sleep 1 107 | done 108 | 109 | # Set the region to send CloudWatch Logs data to (the region where the container instance is located) 110 | cp /etc/awslogs/awscli.conf /etc/awslogs/awscli.conf.bak 111 | region=$(curl -s 169.254.169.254/latest/dynamic/instance-identity/document | jq -r .region) 112 | sed -i -e "s/region = .*/region = $region/g" /etc/awslogs/awscli.conf 113 | 114 | # Grab the cluster and container instance ARN from instance metadata 115 | cluster=$(curl -s http://localhost:51678/v1/metadata | jq -r '. | .Cluster') 116 | container_instance_id=$(curl -s http://localhost:51678/v1/metadata | jq -r '. | .ContainerInstanceArn' | awk -F/ '{print $2}' ) 117 | 118 | # Replace the cluster name and container instance ID placeholders with the actual values 119 | cp /etc/awslogs/awslogs.conf /etc/awslogs/awslogs.conf.bak 120 | sed -i -e "s/{cluster}/$cluster/g" /etc/awslogs/awslogs.conf 121 | sed -i -e "s/{container_instance_id}/$container_instance_id/g" /etc/awslogs/awslogs.conf 122 | EOF 123 | 124 | --==BOUNDARY== 125 | Content-Type: text/x-shellscript; charset="us-ascii" 126 | #!/usr/bin/env bash 127 | # Write the bootstrap-awslogs systemd unit file to /etc/systemd/system/bootstrap-awslogs.service 128 | cat > /etc/systemd/system/bootstrap-awslogs.service <<- EOF 129 | [Unit] 130 | Description=Bootstrap awslogs agent 131 | Requires=ecs.service 132 | After=ecs.service 133 | Before=awslogsd.service 134 | 135 | [Service] 136 | Type=oneshot 137 | RemainAfterExit=yes 138 | ExecStart=/usr/local/bin/bootstrap-awslogs.sh 139 | 140 | [Install] 141 | WantedBy=awslogsd.service 142 | EOF 143 | 144 | --==BOUNDARY== 145 | Content-Type: text/x-shellscript; charset="us-ascii" 146 | #!/bin/sh 147 | 148 | # Start logs 149 | chmod +x /usr/local/bin/bootstrap-awslogs.sh 150 | systemctl daemon-reload 151 | systemctl enable bootstrap-awslogs.service 152 | systemctl enable awslogsd.service 153 | systemctl start awslogsd.service --no-block 154 | 155 | # Start SSM 156 | systemctl enable amazon-ssm-agent 157 | systemctl start amazon-ssm-agent 158 | 159 | --==BOUNDARY==-- --------------------------------------------------------------------------------