├── LICENSE ├── README.md ├── eks-log-collector.sh └── eks-ssm-content.json /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright [yyyy] [name of copyright owner] 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## This repo is now archieved. Please redirect to https://github.com/awslabs/amazon-eks-ami/tree/master/log-collector-script 2 | 3 | ### EKS Logs Collector 4 | 5 | This project was created to collect Amazon EKS log files and OS logs for troubleshooting Amazon EKS customer support cases. 6 | 7 | #### Usage 8 | * Collect EKS logs using SSM agent, jump to below [section](#collect-eks-logs-using-ssm-agent) _(or)_ 9 | 10 | * Run this project as the root user: 11 | ``` 12 | curl -O https://raw.githubusercontent.com/nithu0115/eks-logs-collector/master/eks-log-collector.sh 13 | sudo bash eks-log-collector.sh 14 | ``` 15 | 16 | Confirm if the tarball file was successfully created (it can be .tgz or .tar.gz) 17 | 18 | #### Retrieving the logs 19 | Download the tarball using your favourite Secure Copy tool. 20 | 21 | #### Example output 22 | The project can be used in normal or enable_debug(**Caution: enable_debug will prompt to confirm if we can restart Docker daemon which would kill running containers**). 23 | 24 | ``` 25 | # sudo bash eks-log-collector.sh --help 26 | USAGE: eks-log-collector --help [ --mode=collect|enable_debug --ignore_introspection=true|false --ignore_metrics=true|false ] 27 | 28 | OPTIONS: 29 | --mode Has two parameters 1) collect or 2) enable_debug,: 30 | collect Gathers basic operating system, Docker daemon, and 31 | Amazon EKS related config files and logs. This is the default mode. 32 | enable_debug Enables debug mode for the Docker daemon(Not for production use) 33 | 34 | --ignore_introspection To ignore introspection of IPAMD; Pass this flag if DISABLE_INTROSPECTION is enabled on CNI 35 | 36 | --ignore_metrics To ignore prometheus metrics collection; Pass this flag if DISABLE_METRICS enabled on CNI 37 | 38 | --help Show this help message. 39 | 40 | Example to Ignore IPAMD introspection: 41 | sudo bash eks-log-collector.sh --ignore_introspection=true 42 | 43 | Example to Ignore IPAMD Prometheus metrics collection: 44 | sudo bash eks-log-collector.sh --ignore_metrics=true 45 | 46 | Example to Ignore IPAMD introspection and Prometheus metrics collection: 47 | sudo bash eks-log-collector.sh --ignore_introspection=true --ignore_metrics=true 48 | ``` 49 | #### Example output in normal mode 50 | The following output shows this project running in normal mode. 51 | 52 | ``` 53 | sudo bash eks-log-collector.sh 54 | 55 | This is version 0.5.0. New versions can be found at https://github.com/awslabs/amazon-eks-ami 56 | 57 | Trying to collect common operating system logs... 58 | Trying to collect kernel logs... 59 | Trying to collect mount points and volume information... 60 | Trying to collect SELinux status... 61 | Trying to collect iptables information... 62 | Trying to collect installed packages... 63 | Trying to collect active system services... 64 | Trying to collect Docker daemon information... 65 | Trying to collect kubelet information... 66 | Trying to collect L-IPAMD information... 67 | Trying to collect sysctls information... 68 | Trying to collect networking infomation... 69 | Trying to collect CNI configuration information... 70 | Trying to collect running Docker containers and gather container data... 71 | Trying to collect Docker daemon logs... 72 | Trying to archive gathered information... 73 | 74 | Done... your bundled logs are located in /opt/log-collector/eks_i-0717c9d54b6cfaa19_2019-02-02_0103-UTC_0.0.4.tar.gz 75 | ``` 76 | 77 | 78 | ### Collect EKS logs using SSM agent 79 | #### To run EKS log collector script on Worker Node(s) and upload the bundle(tar) to a S3 Bucket using SSM agent, please follow below steps 80 | 81 | ##### *Prerequisites*: 82 | 83 | * Configure AWS CLI on the system where you will run the below commands. The IAM entity (User/Role) should have permissions to run/invoke `aws ssm send-command` and `get-command-invocation` commands. 84 | 85 | * SSM agent should be installed and running on Worker Node(s). [How to Install SSM Agent link](https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-manual-agent-install.html) 86 | 87 | * Worker Node(s) should have required permissions to communicate with SSM service. IAM managed role `AmazonEC2RoleforSSM` will have all the required permission for SSM agent to run on EC2 instances. The IAM managed role `AmazonEC2RoleforSSM` has `S3:PutObject` permission to all S3 resources. 88 | 89 |         *Note:* For more granular control of the IAM permission check [AWS Systems Manager Permissions link ](https://docs.aws.amazon.com/systems-manager/latest/userguide/auth-and-access-control-permissions-reference.html) 90 | 91 | * A S3 bucket location is required which is taken as an input parameter to `aws ssm send-command` command, to which the logs should be pushed. 92 | 93 | 94 | #### *To invoke SSM agent to run EKS log collector script and push bundle to S3 from Worker Node(s):* 95 | 96 | 1. Create the SSM document named "EKSLogCollector" using the following command:
97 | ``` 98 | aws ssm create-document --name "EKSLogCollector" --document-type "Command" --content https://raw.githubusercontent.com/nithu0115/eks-logs-collector/master/eks-ssm-content.json 99 | ``` 100 | 2. To execute the bash script in the SSM document and to collect the logs from worker, run the following command:
101 | ``` 102 | aws ssm send-command --instance-ids --document-name "EKSLogCollector" --parameters "bucketName=" --output json 103 | ``` 104 | 3. To check the status of SSM command submitted in previous step use the command
105 | ``` 106 | aws ssm get-command-invocation --command-id "" --instance-id "" --output text 107 | ``` 108 |     `SSM command ID`One of the response parameters after running `aws ssm send-command` in step2
109 |     `EC2 Instance ID`The EC2 Instance ID provided in the `aws ssm send-command` in step2 110 | 111 | 4. Once the above command is executed successfully, the logs should be present in the S3 bucket specified in the previous step. 112 | 113 | -------------------------------------------------------------------------------- /eks-log-collector.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # This script generates a file in go with the license contents as a constant 4 | 5 | # Set language to C to make sorting consistent among different environments. 6 | 7 | export LANG="C" 8 | export LC_ALL="C" 9 | 10 | # Global options 11 | readonly PROGRAM_VERSION="0.6.2" 12 | readonly PROGRAM_SOURCE="https://github.com/awslabs/amazon-eks-ami/blob/master/log-collector-script/" 13 | readonly PROGRAM_NAME="$(basename "$0" .sh)" 14 | readonly PROGRAM_DIR="/opt/log-collector" 15 | readonly LOG_DIR="/var/log" 16 | readonly COLLECT_DIR="/tmp/${PROGRAM_NAME}" 17 | readonly CURRENT_TIME=$(date --utc +%Y-%m-%d_%H%M-%Z) 18 | readonly DAYS_10=$(date -d "-10 days" '+%Y-%m-%d %H:%M') 19 | INSTANCE_ID="" 20 | INIT_TYPE="" 21 | PACKAGE_TYPE="" 22 | 23 | # Script run defaults 24 | ignore_introspection='false' 25 | ignore_metrics='false' 26 | 27 | REQUIRED_UTILS=( 28 | timeout 29 | curl 30 | tar 31 | date 32 | mkdir 33 | iptables 34 | iptables-save 35 | grep 36 | awk 37 | df 38 | sysctl 39 | ) 40 | 41 | COMMON_DIRECTORIES=( 42 | kernel 43 | system 44 | docker 45 | storage 46 | var_log 47 | networking 48 | ipamd # eks 49 | sysctls # eks 50 | kubelet # eks 51 | cni # eks 52 | ) 53 | 54 | COMMON_LOGS=( 55 | syslog 56 | messages 57 | aws-routed-eni # eks 58 | containers # eks 59 | pods # eks 60 | cloud-init.log 61 | cloud-init-output.log 62 | kube-proxy.log 63 | ) 64 | 65 | # L-IPAMD introspection data points 66 | IPAMD_DATA=( 67 | enis 68 | pods 69 | networkutils-env-settings 70 | ipamd-env-settings 71 | eni-configs 72 | ) 73 | 74 | help() { 75 | echo "" 76 | echo "USAGE: ${PROGRAM_NAME} --help [ --ignore_introspection=true|false --ignore_metrics=true|false ]" 77 | echo "" 78 | echo "OPTIONS:" 79 | echo "" 80 | echo " --ignore_introspection To ignore introspection of IPAMD; Pass this flag if DISABLE_INTROSPECTION is enabled on CNI" 81 | echo "" 82 | echo " --ignore_metrics Variable To ignore prometheus metrics collection; Pass this flag if DISABLE_METRICS enabled on CNI" 83 | echo "" 84 | echo " --help Show this help message." 85 | echo "" 86 | } 87 | 88 | parse_options() { 89 | local count="$#" 90 | 91 | for i in $(seq "${count}"); do 92 | eval arg="\$$i" 93 | param="$(echo "${arg}" | awk -F '=' '{print $1}' | sed -e 's|--||')" 94 | val="$(echo "${arg}" | awk -F '=' '{print $2}')" 95 | 96 | case "${param}" in 97 | ignore_introspection) 98 | eval "${param}"="${val}" 99 | ;; 100 | ignore_metrics) 101 | eval "${param}"="${val}" 102 | ;; 103 | help) 104 | help && exit 0 105 | ;; 106 | *) 107 | echo "Parameter not found: '$param'" 108 | help && exit 1 109 | ;; 110 | esac 111 | done 112 | } 113 | 114 | ok() { 115 | echo 116 | } 117 | 118 | try() { 119 | local action=$* 120 | echo -n "Trying to $action... " 121 | } 122 | 123 | warning() { 124 | local reason=$* 125 | echo -e "\n\n\tWarning: $reason " 126 | } 127 | 128 | die() { 129 | echo -e "\n\tFatal Error! $* Exiting!\n" 130 | exit 1 131 | } 132 | 133 | is_root() { 134 | if [[ "$(id -u)" -ne 0 ]]; then 135 | die "This script must be run as root!" 136 | fi 137 | } 138 | 139 | check_required_utils() { 140 | for utils in ${REQUIRED_UTILS[*]}; do 141 | # If exit code of "command -v" not equal to 0, fail 142 | if ! command -v "${utils}" >/dev/null 2>&1; then 143 | die "Application \"${utils}\" is missing, please install \"${utils}\" as this script requires it, and will not function without it." 144 | fi 145 | done 146 | } 147 | 148 | version_output() { 149 | echo -e "\n\tThis is version ${PROGRAM_VERSION}. New versions can be found at ${PROGRAM_SOURCE}\n" 150 | } 151 | 152 | log_parameters() { 153 | echo ignore_introspection: "${ignore_introspection}" >> "${COLLECT_DIR}"/system/script-params.txt 154 | echo ignore_metrics: "${ignore_metrics}" >> "${COLLECT_DIR}"/system/script-params.txt 155 | } 156 | 157 | systemd_check() { 158 | if command -v systemctl >/dev/null 2>&1; then 159 | INIT_TYPE="systemd" 160 | if command -v snap >/dev/null 2>&1; then 161 | INIT_TYPE="snap" 162 | fi 163 | else 164 | INIT_TYPE="other" 165 | fi 166 | } 167 | 168 | create_directories() { 169 | # Make sure the directory the script lives in is there. Not an issue if 170 | # the EKS AMI is used, as it will have it. 171 | mkdir -p "${PROGRAM_DIR}" 172 | 173 | # Common directories creation 174 | for directory in ${COMMON_DIRECTORIES[*]}; do 175 | mkdir -p "${COLLECT_DIR}"/"${directory}" 176 | done 177 | } 178 | 179 | get_instance_metadata() { 180 | readonly INSTANCE_ID=$(curl --max-time 3 --silent http://169.254.169.254/latest/meta-data/instance-id 2>/dev/null) 181 | echo "${INSTANCE_ID}" > "${COLLECT_DIR}"/system/instance-id.txt 182 | } 183 | 184 | is_diskfull() { 185 | local threshold 186 | local result 187 | 188 | # 1.5GB in KB 189 | threshold=1500000 190 | result=$(df / | grep --invert-match "Filesystem" | awk '{ print $4 }') 191 | 192 | # If "result" is less than or equal to "threshold", fail. 193 | if [[ "${result}" -le "${threshold}" ]]; then 194 | die "Free space on root volume is less than or equal to $((threshold>>10))MB, please ensure adequate disk space to collect and store the log files." 195 | fi 196 | } 197 | 198 | cleanup() { 199 | rm --recursive --force "${COLLECT_DIR}" >/dev/null 2>&1 200 | } 201 | 202 | init() { 203 | check_required_utils 204 | version_output 205 | create_directories 206 | # Log parameters passed when this script is invoked 207 | log_parameters 208 | is_root 209 | systemd_check 210 | get_pkgtype 211 | } 212 | 213 | collect() { 214 | init 215 | is_diskfull 216 | get_instance_metadata 217 | get_common_logs 218 | get_kernel_info 219 | get_mounts_info 220 | get_selinux_info 221 | get_iptables_info 222 | get_pkglist 223 | get_system_services 224 | get_docker_info 225 | get_k8s_info 226 | get_ipamd_info 227 | get_sysctls_info 228 | get_networking_info 229 | get_cni_config 230 | get_docker_logs 231 | } 232 | 233 | pack() { 234 | try "archive gathered information" 235 | 236 | tar --create --verbose --gzip --file "${LOG_DIR}"/eks_"${INSTANCE_ID}"_"${CURRENT_TIME}"_"${PROGRAM_VERSION}".tar.gz --directory="${COLLECT_DIR}" . > /dev/null 2>&1 237 | 238 | ok 239 | } 240 | 241 | finished() { 242 | cleanup 243 | echo -e "\n\tDone... your bundled logs are located in ${LOG_DIR}/eks_${INSTANCE_ID}_${CURRENT_TIME}_${PROGRAM_VERSION}.tar.gz\n" 244 | } 245 | 246 | get_mounts_info() { 247 | try "collect mount points and volume information" 248 | mount > "${COLLECT_DIR}"/storage/mounts.txt 249 | echo >> "${COLLECT_DIR}"/storage/mounts.txt 250 | df --human-readable >> "${COLLECT_DIR}"/storage/mounts.txt 251 | lsblk > "${COLLECT_DIR}"/storage/lsblk.txt 252 | lvs > "${COLLECT_DIR}"/storage/lvs.txt 253 | pvs > "${COLLECT_DIR}"/storage/pvs.txt 254 | vgs > "${COLLECT_DIR}"/storage/vgs.txt 255 | 256 | ok 257 | } 258 | 259 | get_selinux_info() { 260 | try "collect SELinux status" 261 | 262 | if ! command -v getenforce >/dev/null 2>&1; then 263 | echo -e "SELinux mode:\n\t Not installed" > "${COLLECT_DIR}"/system/selinux.txt 264 | else 265 | echo -e "SELinux mode:\n\t $(getenforce)" > "${COLLECT_DIR}"/system/selinux.txt 266 | fi 267 | 268 | ok 269 | } 270 | 271 | get_iptables_info() { 272 | try "collect iptables information" 273 | 274 | iptables --wait 1 --numeric --verbose --list --table mangle > "${COLLECT_DIR}"/networking/iptables-mangle.txt 275 | iptables --wait 1 --numeric --verbose --list --table filter > "${COLLECT_DIR}"/networking/iptables-filter.txt 276 | iptables --wait 1 --numeric --verbose --list --table nat > "${COLLECT_DIR}"/networking/iptables-nat.txt 277 | iptables --wait 1 --numeric --verbose --list > "${COLLECT_DIR}"/networking/iptables.txt 278 | iptables-save > "${COLLECT_DIR}"/networking/iptables-save.txt 279 | 280 | ok 281 | } 282 | 283 | get_common_logs() { 284 | try "collect common operating system logs" 285 | 286 | for entry in ${COMMON_LOGS[*]}; do 287 | if [[ -e "/var/log/${entry}" ]]; then 288 | if [[ "${entry}" == "messages" ]]; then 289 | tail -c 10M /var/log/messages > "${COLLECT_DIR}"/var_log/messages 290 | continue 291 | fi 292 | if [[ "${entry}" == "containers" ]]; then 293 | cp --force --dereference --recursive /var/log/containers/aws-node* "${COLLECT_DIR}"/var_log/ 2>/dev/null 294 | cp --force --dereference --recursive /var/log/containers/kube-system_cni-metrics-helper* "${COLLECT_DIR}"/var_log/ 2>/dev/null 295 | cp --force --dereference --recursive /var/log/containers/coredns-* "${COLLECT_DIR}"/var_log/ 2>/dev/null 296 | cp --force --dereference --recursive /var/log/containers/kube-proxy* "${COLLECT_DIR}"/var_log/ 2>/dev/null 297 | continue 298 | fi 299 | if [[ "${entry}" == "pods" ]]; then 300 | cp --force --dereference --recursive /var/log/pods/kube-system_aws-node* "${COLLECT_DIR}"/var_log/ 2>/dev/null 301 | cp --force --dereference --recursive /var/log/pods/kube-system_cni-metrics-helper* "${COLLECT_DIR}"/var_log/ 2>/dev/null 302 | cp --force --dereference --recursive /var/log/pods/kube-system_coredns* "${COLLECT_DIR}"/var_log/ 2>/dev/null 303 | cp --force --dereference --recursive /var/log/pods/kube-system_kube-proxy* "${COLLECT_DIR}"/var_log/ 2>/dev/null 304 | continue 305 | fi 306 | cp --force --recursive --dereference /var/log/"${entry}" "${COLLECT_DIR}"/var_log/ 2>/dev/null 307 | fi 308 | done 309 | 310 | ok 311 | } 312 | 313 | get_kernel_info() { 314 | try "collect kernel logs" 315 | 316 | if [[ -e "/var/log/dmesg" ]]; then 317 | cp --force /var/log/dmesg "${COLLECT_DIR}/kernel/dmesg.boot" 318 | fi 319 | dmesg > "${COLLECT_DIR}/kernel/dmesg.current" 320 | dmesg --ctime > "${COLLECT_DIR}/kernel/dmesg.human.current" 321 | uname -a > "${COLLECT_DIR}/kernel/uname.txt" 322 | 323 | ok 324 | } 325 | 326 | get_docker_logs() { 327 | try "collect Docker daemon logs" 328 | 329 | case "${INIT_TYPE}" in 330 | systemd|snap) 331 | journalctl --unit=docker --since "${DAYS_10}" > "${COLLECT_DIR}"/docker/docker.log 332 | ;; 333 | other) 334 | for entry in docker upstart/docker; do 335 | if [[ -e "/var/log/${entry}" ]]; then 336 | cp --force --recursive --dereference /var/log/"${entry}" "${COLLECT_DIR}"/docker/ 337 | fi 338 | done 339 | ;; 340 | *) 341 | warning "The current operating system is not supported." 342 | ;; 343 | esac 344 | 345 | ok 346 | } 347 | 348 | get_k8s_info() { 349 | try "collect kubelet information" 350 | 351 | if [[ -n "${KUBECONFIG:-}" ]]; then 352 | command -v kubectl > /dev/null && kubectl get --kubeconfig="${KUBECONFIG}" svc > "${COLLECT_DIR}"/kubelet/svc.log 353 | command -v kubectl > /dev/null && kubectl --kubeconfig="${KUBECONFIG}" config view --output yaml > "${COLLECT_DIR}"/kubelet/kubeconfig.yaml 354 | 355 | elif [[ -f /etc/eksctl/kubeconfig.yaml ]]; then 356 | KUBECONFIG="/etc/eksctl/kubeconfig.yaml" 357 | command -v kubectl > /dev/null && kubectl get --kubeconfig="${KUBECONFIG}" svc > "${COLLECT_DIR}"/kubelet/svc.log 358 | command -v kubectl > /dev/null && kubectl --kubeconfig="${KUBECONFIG}" config view --output yaml > "${COLLECT_DIR}"/kubelet/kubeconfig.yaml 359 | 360 | elif [[ -f /etc/systemd/system/kubelet.service ]]; then 361 | KUBECONFIG=$(grep kubeconfig /etc/systemd/system/kubelet.service | awk '{print $2}') 362 | command -v kubectl > /dev/null && kubectl get --kubeconfig="${KUBECONFIG}" svc > "${COLLECT_DIR}"/kubelet/svc.log 363 | command -v kubectl > /dev/null && kubectl --kubeconfig="${KUBECONFIG}" config view --output yaml > "${COLLECT_DIR}"/kubelet/kubeconfig.yaml 364 | 365 | elif [[ -f /var/lib/kubelet/kubeconfig ]]; then 366 | KUBECONFIG="/var/lib/kubelet/kubeconfig" 367 | command -v kubectl > /dev/null && kubectl get --kubeconfig=${KUBECONFIG} svc > "${COLLECT_DIR}"/kubelet/svc.log 368 | command -v kubectl > /dev/null && kubectl --kubeconfig=${KUBECONFIG} config view --output yaml > "${COLLECT_DIR}"/kubelet/kubeconfig.yaml 369 | 370 | else 371 | echo "======== Unable to find KUBECONFIG, IGNORING POD DATA =========" >> "${COLLECT_DIR}"/kubelet/svc.log 372 | fi 373 | 374 | # Try to copy the kubeconfig file if kubectl command doesn't exist 375 | [[ (! -f "${COLLECT_DIR}/kubelet/kubeconfig.yaml") && ( -n ${KUBECONFIG}) ]] && cp ${KUBECONFIG} "${COLLECT_DIR}"/kubelet/kubeconfig.yaml 376 | 377 | case "${INIT_TYPE}" in 378 | systemd) 379 | timeout 75 journalctl --unit=kubelet --since "${DAYS_10}" > "${COLLECT_DIR}"/kubelet/kubelet.log 380 | 381 | systemctl cat kubelet > "${COLLECT_DIR}"/kubelet/kubelet_service.txt 2>&1 382 | ;; 383 | snap) 384 | timeout 75 snap logs kubelet-eks -n all > "${COLLECT_DIR}"/kubelet/kubelet.log 385 | 386 | timeout 75 snap get kubelet-eks > "${COLLECT_DIR}"/kubelet/kubelet-eks_service.txt 2>&1 387 | ;; 388 | *) 389 | warning "The current operating system is not supported." 390 | ;; 391 | esac 392 | 393 | ok 394 | } 395 | 396 | get_ipamd_info() { 397 | if [[ "${ignore_introspection}" == "false" ]]; then 398 | try "collect L-IPAMD introspection information" 399 | for entry in ${IPAMD_DATA[*]}; do 400 | curl --max-time 3 --silent http://localhost:61679/v1/"${entry}" >> "${COLLECT_DIR}"/ipamd/"${entry}".json 401 | done 402 | else 403 | echo "Ignoring IPAM introspection stats as mentioned"| tee -a "${COLLECT_DIR}"/ipamd/ipam_introspection_ignore.txt 404 | fi 405 | 406 | if [[ "${ignore_metrics}" == "false" ]]; then 407 | try "collect L-IPAMD prometheus metrics" 408 | curl --max-time 3 --silent http://localhost:61678/metrics > "${COLLECT_DIR}"/ipamd/metrics.json 2>&1 409 | else 410 | echo "Ignoring Prometheus Metrics collection as mentioned"| tee -a "${COLLECT_DIR}"/ipamd/ipam_metrics_ignore.txt 411 | fi 412 | 413 | try "collect L-IPAMD checkpoint" 414 | cp /var/run/aws-node/ipam.json "${COLLECT_DIR}"/ipamd/ipam.json 415 | 416 | ok 417 | } 418 | 419 | get_sysctls_info() { 420 | try "collect sysctls information" 421 | # dump all sysctls 422 | sysctl --all >> "${COLLECT_DIR}"/sysctls/sysctl_all.txt 2>/dev/null 423 | 424 | ok 425 | } 426 | 427 | get_networking_info() { 428 | try "collect networking infomation" 429 | 430 | # conntrack info 431 | echo "*** Output of conntrack -S *** " >> "${COLLECT_DIR}"/networking/conntrack.txt 432 | timeout 75 conntrack -S >> "${COLLECT_DIR}"/networking/conntrack.txt 433 | echo "*** Output of conntrack -L ***" >> "${COLLECT_DIR}"/networking/conntrack.txt 434 | timeout 75 conntrack -L >> "${COLLECT_DIR}"/networking/conntrack.txt 435 | 436 | # ifconfig 437 | timeout 75 ifconfig > "${COLLECT_DIR}"/networking/ifconfig.txt 438 | 439 | # ip rule show 440 | timeout 75 ip rule show > "${COLLECT_DIR}"/networking/iprule.txt 441 | timeout 75 ip route show table all >> "${COLLECT_DIR}"/networking/iproute.txt 442 | 443 | ok 444 | } 445 | 446 | get_cni_config() { 447 | try "collect CNI configuration information" 448 | 449 | if [[ -e "/etc/cni/net.d/" ]]; then 450 | cp --force --recursive --dereference /etc/cni/net.d/* "${COLLECT_DIR}"/cni/ 451 | fi 452 | 453 | ok 454 | } 455 | 456 | get_pkgtype() { 457 | if [[ "$(command -v rpm )" ]]; then 458 | PACKAGE_TYPE=rpm 459 | elif [[ "$(command -v dpkg )" ]]; then 460 | PACKAGE_TYPE=deb 461 | else 462 | PACKAGE_TYPE='unknown' 463 | fi 464 | } 465 | 466 | get_pkglist() { 467 | try "collect installed packages" 468 | 469 | case "${PACKAGE_TYPE}" in 470 | rpm) 471 | rpm -qa > "${COLLECT_DIR}"/system/pkglist.txt 2>&1 472 | ;; 473 | deb) 474 | dpkg --list > "${COLLECT_DIR}"/system/pkglist.txt 2>&1 475 | ;; 476 | *) 477 | warning "Unknown package type." 478 | ;; 479 | esac 480 | 481 | ok 482 | } 483 | 484 | get_system_services() { 485 | try "collect active system services" 486 | 487 | case "${INIT_TYPE}" in 488 | systemd|snap) 489 | systemctl list-units > "${COLLECT_DIR}"/system/services.txt 2>&1 490 | ;; 491 | other) 492 | initctl list | awk '{ print $1 }' | xargs -n1 initctl show-config > "${COLLECT_DIR}"/system/services.txt 2>&1 493 | printf "\n\n\n\n" >> "${COLLECT_DIR}"/system/services.txt 2>&1 494 | service --status-all >> "${COLLECT_DIR}"/system/services.txt 2>&1 495 | ;; 496 | *) 497 | warning "Unable to determine active services." 498 | ;; 499 | esac 500 | 501 | timeout 75 top -b -n 1 > "${COLLECT_DIR}"/system/top.txt 2>&1 502 | timeout 75 ps fauxwww > "${COLLECT_DIR}"/system/ps.txt 2>&1 503 | timeout 75 netstat -plant > "${COLLECT_DIR}"/system/netstat.txt 2>&1 504 | 505 | ok 506 | } 507 | 508 | get_docker_info() { 509 | try "collect Docker daemon information" 510 | 511 | if [[ "$(pgrep -o dockerd)" -ne 0 ]]; then 512 | timeout 75 docker info > "${COLLECT_DIR}"/docker/docker-info.txt 2>&1 || echo -e "\tTimed out, ignoring \"docker info output \" " 513 | timeout 75 docker ps --all --no-trunc > "${COLLECT_DIR}"/docker/docker-ps.txt 2>&1 || echo -e "\tTimed out, ignoring \"docker ps --all --no-truc output \" " 514 | timeout 75 docker images > "${COLLECT_DIR}"/docker/docker-images.txt 2>&1 || echo -e "\tTimed out, ignoring \"docker images output \" " 515 | timeout 75 docker version > "${COLLECT_DIR}"/docker/docker-version.txt 2>&1 || echo -e "\tTimed out, ignoring \"docker version output \" " 516 | else 517 | warning "The Docker daemon is not running." 518 | fi 519 | 520 | ok 521 | } 522 | 523 | # ----------------------------------------------------------------------------- 524 | # Entrypoint 525 | parse_options "$@" 526 | 527 | collect 528 | pack 529 | finished 530 | -------------------------------------------------------------------------------- /eks-ssm-content.json: -------------------------------------------------------------------------------- 1 | { 2 | "schemaVersion":"2.2", 3 | "description":"EKS Log Collector", 4 | "parameters":{ 5 | "bucketName":{ 6 | "type": "String", 7 | "default": "Enabled" 8 | } 9 | }, 10 | "mainSteps":[ 11 | { 12 | "action":"aws:runShellScript", 13 | "name":"PatchLinux", 14 | "precondition":{ 15 | "StringEquals":[ 16 | "platformType", 17 | "Linux" 18 | ] 19 | }, 20 | "inputs":{ 21 | "runCommand":[ 22 | "curl -O https://raw.githubusercontent.com/nithu0115/eks-logs-collector/master/eks-log-collector.sh", 23 | "bash ./eks-log-collector.sh >/dev/null 2>&1", 24 | "echo \"EKS logs collected\"", 25 | "if [ -f /usr/local/bin/aws ]; then", 26 | "echo \"AWS_already_installed\"", 27 | "else", 28 | "echo \"Installing AWSCLI\"", 29 | "curl \"https://s3.amazonaws.com/aws-cli/awscli-bundle.zip\" -o \"awscli-bundle.zip\" >/dev/null 2>&1", 30 | "yum install unzip -y >/dev/null 2>&1", 31 | "unzip awscli-bundle.zip >/dev/null 2>&1", 32 | "./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws", 33 | "echo \"AWSCLI version is\"", 34 | "/usr/local/bin/aws --version", 35 | "fi", 36 | "echo \"Pushing to S3\"", 37 | "/usr/local/bin/aws s3 cp --recursive /opt/log-collector/ s3://{{bucketName}}", 38 | "echo \"Logs uploaded to S3\"" 39 | ] 40 | } 41 | } 42 | ] 43 | } 44 | 45 | --------------------------------------------------------------------------------