├── LICENSE ├── README.md ├── cacher.py └── images ├── CacherServerAlert.png ├── CacherSlack_Large.png └── CacherSlack_Small.png /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "{}" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright {2017} {Erik Nicolas Gomez} 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Cacher 2 | Cacher is a python script that will parse the OS X Caching Server debug logs and present you (to the best of its abilities) serving statistics. 3 | 4 | Some of the things Cacher can display: 5 | - Total bandwidth served to clients 6 | - Total bandwidth requested from Apple 7 | - Total bandwidth requested from other Caching servers 8 | - Total IP Addresses 9 | - Total Unique IP Addresses. 10 | - Total iOS download requests including model type 11 | - Total OS download requests specified by type (iOS and macOS) 12 | - Total applications downloaded from Apple Configurator 2 devices 13 | - Total downloaded files 14 | - Total eBook (.epub) files 15 | - Total personal iCloud files 16 | - Total package (.pkg) files 17 | - Total iOS application (.ipad) files 18 | - Total Zip (.zip) files 19 | - Total unique downloaded files 20 | - Total unique eBook (.epub) files 21 | - Total unique personal iCloud files 22 | - Total unique package (.pkg) files 23 | - Total unique iOS application (.ipad) files 24 | - Total unique Zip (.zip) files 25 | 26 | ## Server support 27 | Cacher currently supports Server 5.2 and higher. 28 | 29 | - For Server 4 please see [this commit](https://github.com/erikng/Cacher/commit/17903d2dd29886c0dfc16054ae39b89f25581f79) 30 | - For Server 5-5.1 please see [this commit](https://github.com/erikng/Cacher/commit/57ea9c3c80c17bb29d4deb89cb07a2ae841613d9) 31 | 32 | ## Usage 33 | ``` 34 | Usage: cacher.py [options] 35 | 36 | Options: 37 | -h, --help show this help message and exit 38 | --targetdate=TARGETDATE 39 | Optional: Date to parse. Example: 2017-01-15. 40 | --logpath=LOGPATH Optional: Caching Log Path. Defaults to: 41 | /Library/Server/Caching/Logs 42 | --deviceids Optional: Use Device IDs (Ex: iPhone7,2). Defaults to: 43 | False 44 | --nostdout Optional: Do not print to standard out 45 | --configureserver Optional: Configure Server to log Client Data 46 | --serveralert Optional: Send Server Alert 47 | --slackalert Optional: Use Slack 48 | --slackwebhook=SLACKWEBHOOK 49 | Optional: Slack Webhook URL. Requires Slack Option. 50 | --slackusername=SLACKUSERNAME 51 | Optional: Slack username. Defaults to Cacher. Requires 52 | Slack Option. 53 | --slackchannel=SLACKCHANNEL 54 | Optional: Slack channel. Can be username or channel. 55 | Ex. #channel or @username. Requires Slack Option. 56 | ``` 57 | 58 | ## Optional features 59 | The following are optional features: 60 | 61 | ### Configure Caching service logging 62 | By default, the Caching service will not log the model and iOS/OS X version. In order to get true results from this script, run the following command (as root): 63 | 64 | `sudo cacher.py --configureserver` 65 | 66 | If successful, you will see the following output: 67 | 68 | `Caching Server settings are now: caching:LogClientIdentity = yes` 69 | 70 | ### Target date 71 | By default, Cacher will use look for logs from the previous date. To target logs from a custom date, use the `--targetdate` option. 72 | 73 | `cacher.py --targetdate "2016-11-28"` 74 | 75 | ### Log path 76 | By default, Cacher will use look for logs from in /Library/Server/Caching/Logs. To target logs in a custom path, use the `--logpath` option. 77 | 78 | `cacher.py --logpath "/path/to/logs"` 79 | 80 | ### DeviceIDs 81 | By default, Cacher will use the "Friendly Names" for iOS devices. To use the model Device ID, use the `--deviceids` option. 82 | 83 | `cacher.py --deviceids` 84 | 85 | Device IDs Example: 86 | ``` bash 87 | A total of 3513 iOS downloads were requested from the Caching Server yesterday consisting of: 88 | A total of 4 Apple TV downloads 89 | A total of 417 iPad downloads 90 | A total of 3075 iPhone downloads 91 | A total of 17 iPod downloads 92 | 4 AppleTV5,3 93 | 4 iPad2,1 94 | 7 iPad2,2 95 | 2 iPad2,3 96 | 5 iPad2,4 97 | ``` 98 | 99 | Friendly Names Example: 100 | ``` bash 101 | A total of 3513 iOS downloads were requested from the Caching Server yesterday consisting of: 102 | A total of 4 Apple TV downloads 103 | A total of 417 iPad downloads 104 | A total of 3075 iPhone downloads 105 | A total of 17 iPod downloads 106 | 4 5th Generation Apple TVs 107 | 5 iPad 2nd Generation [M2012 Wifi Revision] 108 | 2 iPad 2nd Generation [Wifi + CDMA] 109 | 7 iPad 2nd Generation [Wifi + GSM] 110 | 4 iPad 2nd Generation [Wifi] 111 | ``` 112 | 113 | ### No Standard output 114 | By default, Cacher will print the results to standard out. To skip this use the `--nostdout` option. 115 | 116 | `cacher.py --nostdout` 117 | 118 | ### Server alert 119 | By default, Cacher will __no longer__ send a server alert. To send a server alert, use the `--serveralert` option. 120 | 121 | Please note that this option requires root/sudo level permissions. 122 | 123 | `sudo cacher.py --serveralert` 124 | 125 | If you attempt to use this option without elevated permissions, Cacher will write the following note to standard out. 126 | 127 | `Did not send serverAlert - requires root` 128 | 129 | ### Slack alert 130 | By default, Cacher will not send a server alert. To send a server alert, use the `--slackalert` option. 131 | 132 | The slack alert __requires__ two other options to be passed: 133 | - `--slackchannel` 134 | - `--slackwebhook` 135 | 136 | #### Username or Channel 137 | You can pass both an username or channel via the `--slackchannel` option: 138 | Examples: 139 | - `@erik` 140 | - `#cacher` 141 | 142 | #### Slack webhook 143 | A slack webhook must be created. To create a webhook, please go [here](https://my.slack.com/services/new/incoming-webhook/) 144 | 145 | ``` bash 146 | cacher.py --slackalert \ 147 | --slackchannel "@egomez" \ 148 | --slackwebhook "https://hooks.slack.com/services/YOURURL"`` 149 | ``` 150 | 151 | ## Screenshots 152 | 153 | ### Slack Small 154 | ![Cacher Slack Example Small](/images/CacherSlack_Small.png?raw=true) 155 | 156 | ### Slack Large 157 | ![Cacher Slack Example Large](/images/CacherSlack_Large.png?raw=true) 158 | 159 | ### Original Server Alert 160 | ![Cacher Server Alert Example](/images/CacherServerAlert.png?raw=true) 161 | -------------------------------------------------------------------------------- /cacher.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | 3 | from datetime import date, timedelta 4 | from distutils.version import LooseVersion 5 | import glob 6 | import json 7 | import logging 8 | import optparse 9 | import os 10 | import plistlib 11 | import re 12 | import shutil 13 | import subprocess 14 | import sys 15 | import tempfile 16 | import urllib2 17 | 18 | """Cacher rewritten in Python. 19 | Inspired by Michael Lynn https://gist.github.com/pudquick/ffdbdb52ae6960ca8e55 20 | 21 | This script will process Caching Server Debug Logs. 22 | You can output this data to stdout, send it to the Apple email alert mechanism, 23 | or send to a slack channel. 24 | 25 | Slack section adapted from another one of my tools (APInfo). 26 | https://github.com/erikng/scripts/tree/master/APInfo 27 | 28 | Author: Erik Gomez 29 | Last Updated: 06-08-2017 30 | """ 31 | version = '3.0.4' 32 | 33 | 34 | def cacher(lines, targetDate, friendlyNames): 35 | # Basically run through all the lines a single time and collect all the 36 | # relevant data to slice, do stats with, etc. 37 | noClientIdentityLog = [] 38 | sizeLog = [] 39 | AC2Log = [] 40 | IPLog = [] 41 | OSLog = [] 42 | osVersionLog = [] 43 | iOSModelLog = [] 44 | iOSModelOnlyLog = [] 45 | fileTypeLog = [] 46 | fileTypeUniqueLog = [] 47 | urlLog = [] 48 | urlUniqueLog = [] 49 | deviceNumberLog = [] 50 | finalOutput = [] 51 | FriendlyLog = [] 52 | macOSFamilyLog = [] 53 | macOSDeviceNumber = [] 54 | iOSFamilyLog = [] 55 | iOSDeviceNumber = [] 56 | AppleTVNumberLog = [] 57 | iPadNumberLog = [] 58 | iPhoneNumberLog = [] 59 | iPodNumberLog = [] 60 | # Friendly Darwin versions for macOS. This allows us to dynamically add 61 | # the macOS version (for the alert), while dynamically looping through the 62 | # logs. 63 | friendlyDarwin = { 64 | '17.0.0': '10.13.0', 65 | '16.7.0': '10.12.6', 66 | '16.6.0': '10.12.5', 67 | '16.5.0': '10.12.4', 68 | '16.4.0': '10.12.3', 69 | '16.3.0': '10.12.2', 70 | '16.1.0': '10.12.1', 71 | '16.0.0': '10.12.0', 72 | '10.12': '10.12.0', # match 10.12 to 10.12.0 for consistency 73 | '15.6.0': '10.11.6', 74 | '15.5.0': '10.11.5', 75 | '15.4.0': '10.11.4', 76 | '15.3.0': '10.11.3', 77 | '15.2.0': '10.11.2', 78 | '15.0.0': '10.11.0/1', 79 | '14.5.0': '10.10.5', 80 | '14.4.0': '10.10.4', 81 | '14.3.0': '10.10.3', 82 | '14.1.1': '10.12.2', 83 | '14.1.0': '10.10.2', 84 | '14.0.0': '10.10.0/1', 85 | } 86 | # Friendly Models of known models. This allows us to dynamically add the 87 | # names to each model (for the alert), while dynamically looping through 88 | # the logs. 89 | friendlyModels = { 90 | 'AppleTV3,1': '3rd Generation Apple TVs', 91 | 'AppleTV3,2': '4th Generation Apple TVs', 92 | 'AppleTV5,3': '5th Generation Apple TVs', 93 | 'iPhone3,1': 'iPhone 4 [GSM]', 94 | 'iPhone3,2': 'iPhone 4 [GSM 2012]', 95 | 'iPhone3,3': 'iPhone 4 [CDMA]', 96 | 'iPhone4,1': 'iPhone 4S', 97 | 'iPhone5,1': 'iPhone 5 [GSM]', 98 | 'iPhone5,2': 'iPhone 5 [CDMA]', 99 | 'iPhone5,3': 'iPhone 5C', 100 | 'iPhone5,4': 'iPhone 5C [Global]', 101 | 'iPhone6,1': 'iPhone 5S', 102 | 'iPhone6,2': 'iPhone 5S [China Model]', 103 | 'iPhone7,1': 'iPhone 6 Plus', 104 | 'iPhone7,2': 'iPhone 6', 105 | 'iPhone8,1': 'iPhone 6S', 106 | 'iPhone8,2': 'iPhone 6S Plus', 107 | 'iPhone8,4': 'iPhone SE', 108 | 'iPhone9,1': 'iPhone 7 [Global]', 109 | 'iPhone9,2': 'iPhone 7 Plus [Global]', 110 | 'iPhone9,3': 'iPhone 7 [GSM]', 111 | 'iPhone9,4': 'iPhone 7 Plus [GSM]', 112 | 'iPhone10,1': 'iPhone 8 [Global]', 113 | 'iPhone10,2': 'iPhone 8 Plus [Global]', 114 | 'iPhone10,3': 'iPhone X [Global]', 115 | 'iPhone10,4': 'iPhone 8 [GSM]', 116 | 'iPhone10,5': 'iPhone 8 Plus [GSM]', 117 | 'iPhone10,6': 'iPhone X [GSM]', 118 | 'iPad2,1': 'iPad 2nd Generation [Wifi]', 119 | 'iPad2,2': 'iPad 2nd Generation [Wifi + GSM]', 120 | 'iPad2,3': 'iPad 2nd Generation [Wifi + CDMA]', 121 | 'iPad2,4': 'iPad 2nd Generation [M2012 Wifi Revision]', 122 | 'iPad2,5': 'iPad Mini 1st Generation [Wifi]', 123 | 'iPad2,6': 'iPad Mini 1st Generation [Wifi + GSM]', 124 | 'iPad2,7': 'iPad Mini 1st Generation [Wifi + CDMA]', 125 | 'iPad3,1': 'iPad 3rd Generation [Wifi]', 126 | 'iPad3,2': 'iPad 3rd Generation [Wifi + GSM]', 127 | 'iPad3,3': 'iPad 3rd Generation [Wifi + CDMA]', 128 | 'iPad3,4': 'iPad 4th Generation [Wifi]', 129 | 'iPad3,5': 'iPad 4th Generation [Wifi + GSM]', 130 | 'iPad3,6': 'iPad 4th Generation [Wifi + CDMA]', 131 | 'iPad4,1': 'iPad Air 1st Generation [Wifi]', 132 | 'iPad4,2': 'iPad Air 1st Generation [Wifi + Cellular]', 133 | 'iPad4,3': 'iPad Air 1st Generation [China Model]', 134 | 'iPad4,4': 'iPad Mini 2nd Generation [Wifi]', 135 | 'iPad4,5': 'iPad Mini 2nd Generation [Wifi + Cellular]', 136 | 'iPad4,6': 'iPad Mini 2nd Generation [China Model]', 137 | 'iPad4,7': 'iPad Mini 3rd Generation [Wifi]', 138 | 'iPad4,8': 'iPad Mini 3rd Generation [Wifi + Cellular]', 139 | 'iPad4,9': 'iPad Mini 3rd Generation [China Model]', 140 | 'iPad5,1': 'iPad Mini 4th Generation [Wifi]', 141 | 'iPad5,2': 'iPad Mini 4th Generation [Wifi + Cellular]', 142 | 'iPad5,3': 'iPad Air 2nd Generation [Wifi]', 143 | 'iPad5,4': 'iPad Air 2nd Generation [Wifi + Cellular]', 144 | 'iPad6,3': 'iPad Pro 9.7 Inch 1st Generation [Wifi]', 145 | 'iPad6,4': 'iPad Pro 9.7 Inch 1st Generation [Wifi + Cellular]', 146 | 'iPad6,7': 'iPad Pro 12.9 Inch 1st Generation [Wifi]', 147 | 'iPad6,8': 'iPad Pro 12.9 Inch 1st Generation [Wifi + Cellular]', 148 | 'iPad6,11': 'iPad 5th Generation [Wifi]', 149 | 'iPad6,12': 'iPad 5th Generation [Wifi + Cellular]', 150 | 'iPad7,1': 'iPad Pro 12.9 Inch 2nd Generation [Wifi]', 151 | 'iPad7,2': 'iPad Pro 12.9 Inch 2nd Generation [Wifi + Cellular]', 152 | 'iPad7,3': 'iPad Pro 10.5 Inch 1st Generation [Wifi]', 153 | 'iPad7,4': 'iPad Pro 10.5 Inch 1st Generation [Wifi + Cellular]', 154 | 'iPod5,1': 'iPod Touch 5th Generation', 155 | 'iPod7,1': 'iPod Touch 6th Generation' 156 | } 157 | totalbytesserved = [] 158 | totalbytesfromorigin = [] 159 | totalbytesfrompeers = [] 160 | for x in lines: 161 | # If there aren't at least 3 pieces somehow, they'll get filled in 162 | # with blanks 163 | datestr, timestr, logmsg = (x.split(' ', 2) + ['', '', ''])[:3] 164 | if datestr == targetDate: 165 | # Only do work if the string is on the date we care about 166 | # try: 167 | linesplit = str.split(logmsg) 168 | # split the logmsg line (by spaces) so I can hardcode some 169 | # calls. Fragile (could break with a Server update) but it meh. 170 | 171 | # Beginning of Server bandwidth section 172 | # 173 | # This is a slightly less fragile method to calculate the 174 | # amount of data the caching server has served. 175 | # Eg: 176 | # Served all 39.2 MB of 39.2 MB; 3 KB from cache, 177 | # 39.2 MB stored from Internet, 0 bytes from peers 178 | if 'Served all' in logmsg: 179 | total_served_size = linesplit[3] 180 | total_served_bwtype = linesplit[4] 181 | fromorigin_size = linesplit[12] 182 | fromoriginbwtype = linesplit[13] 183 | frompeers_size = linesplit[17] 184 | frompeersbwtype = linesplit[18] 185 | # Convert size of served to client to bytes 186 | if total_served_bwtype == 'KB': 187 | bytes_served = "%.0f" % ( 188 | float(total_served_size) * 1024) 189 | elif total_served_bwtype == 'MB': 190 | bytes_served = "%.0f" % ( 191 | float(total_served_size) * 1048576) 192 | elif total_served_bwtype == 'GB': 193 | bytes_served = "%.0f" % ( 194 | float(total_served_size) * 1073741824) 195 | elif total_served_bwtype == 'TB': 196 | bytes_served = "%.0f" % ( 197 | float(total_served_size) * 1099511627776) 198 | elif total_served_bwtype == 'bytes': 199 | bytes_served = total_served_size 200 | # Convert size of from internet(origin) to bytes 201 | if fromoriginbwtype == 'KB': 202 | bytesfromorigin = "%.0f" % ( 203 | float(fromorigin_size) * 1024) 204 | elif fromoriginbwtype == 'MB': 205 | bytesfromorigin = "%.0f" % ( 206 | float(fromorigin_size) * 1048576) 207 | elif fromoriginbwtype == 'GB': 208 | bytesfromorigin = "%.0f" % ( 209 | float(fromorigin_size) * 1073741824) 210 | elif fromoriginbwtype == 'TB': 211 | bytesfromorigin = "%.0f" % ( 212 | float(fromorigin_size) * 1099511627776) 213 | elif fromoriginbwtype == 'bytes': 214 | bytesfromorigin = fromorigin_size 215 | # Convert size of from peers to bytes 216 | if frompeersbwtype == 'KB': 217 | bytesfrompeers = "%.0f" % ( 218 | float(frompeers_size) * 1024) 219 | elif frompeersbwtype == 'MB': 220 | bytesfrompeers = "%.0f" % ( 221 | float(frompeers_size) * 1048576) 222 | elif frompeersbwtype == 'GB': 223 | bytesfrompeers = "%.0f" % ( 224 | float(frompeers_size) * 1073741824) 225 | elif frompeersbwtype == 'TB': 226 | bytesfrompeers = "%.0f" % ( 227 | float(frompeers_size) * 1099511627776) 228 | elif frompeersbwtype == 'bytes': 229 | bytesfrompeers = frompeers_size 230 | # Append each bw size to the total count 231 | totalbytesserved.append(bytes_served) 232 | totalbytesfromorigin.append(bytesfromorigin) 233 | totalbytesfrompeers.append(bytesfrompeers) 234 | # Search through the logs for incomplete transactions (served) 235 | if 'Served all' not in logmsg and 'Served' in logmsg: 236 | total_served_size = linesplit[2] 237 | total_served_bwtype = linesplit[3] 238 | fromorigin_size = linesplit[11] 239 | fromoriginbwtype = linesplit[12] 240 | frompeers_size = linesplit[16] 241 | frompeersbwtype = linesplit[17] 242 | # Convert size of from cache to bytes 243 | if total_served_bwtype == 'KB': 244 | bytes_served = "%.0f" % ( 245 | float(total_served_size) * 1024) 246 | elif total_served_bwtype == 'MB': 247 | bytes_served = "%.0f" % ( 248 | float(total_served_size) * 1048576) 249 | elif total_served_bwtype == 'GB': 250 | bytes_served = "%.0f" % ( 251 | float(total_served_size) * 1073741824) 252 | elif total_served_bwtype == 'TB': 253 | bytes_served = "%.0f" % ( 254 | float(total_served_size) * 1099511627776) 255 | elif total_served_bwtype == 'bytes': 256 | bytes_served = total_served_size 257 | # Convert size of from internet(origin) to bytes 258 | if fromoriginbwtype == 'KB': 259 | bytesfromorigin = "%.0f" % ( 260 | float(fromorigin_size) * 1024) 261 | elif fromoriginbwtype == 'MB': 262 | bytesfromorigin = "%.0f" % ( 263 | float(fromorigin_size) * 1048576) 264 | elif fromoriginbwtype == 'GB': 265 | bytesfromorigin = "%.0f" % ( 266 | float(fromorigin_size) * 1073741824) 267 | elif fromoriginbwtype == 'TB': 268 | bytesfromorigin = "%.0f" % ( 269 | float(fromorigin_size) * 1099511627776) 270 | elif fromoriginbwtype == 'bytes': 271 | bytesfromorigin = fromorigin_size 272 | # Convert size of from peers to bytes 273 | if frompeersbwtype == 'KB': 274 | bytesfrompeers = "%.0f" % ( 275 | float(frompeers_size) * 1024) 276 | elif frompeersbwtype == 'MB': 277 | bytesfrompeers = "%.0f" % ( 278 | float(frompeers_size) * 1048576) 279 | elif frompeersbwtype == 'GB': 280 | bytesfrompeers = "%.0f" % ( 281 | float(frompeers_size) * 1073741824) 282 | elif frompeersbwtype == 'TB': 283 | bytesfrompeers = "%.0f" % ( 284 | float(frompeers_size) * 1099511627776) 285 | elif frompeersbwtype == 'bytes': 286 | bytesfrompeers = frompeers_size 287 | # Append each bw size to the total count 288 | totalbytesserved.append(bytes_served) 289 | totalbytesfromorigin.append(bytesfromorigin) 290 | totalbytesfrompeers.append(bytesfrompeers) 291 | # Beginning of Server downloads section 292 | # 293 | # 294 | if 'Received GET request by' in logmsg: 295 | noClientIdentityLog.append(logmsg) 296 | elif 'Received GET request from' in logmsg: 297 | # Beginning of IP section 298 | # 299 | # 300 | # Ex: '149.166.73.137:56833'. Split 6th string at ':' and 301 | # pull only pull first value. 302 | ip = linesplit[5].split(":")[0] 303 | IPLog.append(ip) 304 | # 305 | # 306 | # End of IP section 307 | 308 | # Beginning of URL section 309 | # 310 | # 311 | # The URL is always at the end so take the split line and 312 | # pull its value. 313 | URL = linesplit[-1] 314 | urlLog.append(URL) 315 | # 316 | # 317 | # End of URL section 318 | 319 | # Beginning of OS Family, OS Version and Device section 320 | # 321 | # 322 | # Example: 'Darwin/15.0.0', 'iOS/10.0.2' or 'OS X 10.12.0' 323 | # Replace Look for iOS, Darwin or OS X. If OS X is found, 324 | # Add 'macOS/' to force the consistency and split the 325 | # string at '/'. This allows us to use 'macOS/10.12.2' for 326 | # both osFamily (Ex: macOS) and osVersion (Ex: 10.12.2). 327 | osFamily = re.match( 328 | r'.+? ((iOS|Darwin|OS X)[/ ](([0-9]+\.?){1,}))', 329 | x) 330 | if osFamily is not None: 331 | osFamily = osFamily.group(1).replace( 332 | 'OS X ', 'macOS/').split('/')[0] 333 | 334 | osVersion = re.match( 335 | r'.+? ((iOS|Darwin|OS X)[/ ](([0-9]+\.?){1,}))', 336 | x) 337 | if osVersion is not None: 338 | osVersion = osVersion.group(1).replace( 339 | 'OS X ', 'macOS/').split('/')[1] 340 | 341 | configurator = re.match( 342 | r'.+?((Configurator)[/ ](([0-9]+\.?){1,}))', 343 | x) 344 | if configurator is not None: 345 | configurator = configurator.group(1).split('/')[0] 346 | # If 'Darwin' in the name, replace to 'macOS' so our future 347 | # counts will be accurate. 348 | if osFamily == 'Darwin': 349 | osFamily = 'macOS' 350 | # Loop through the friendlyDarwin key/value pairs and if 351 | # osVersion is equal to the key (Ex: 16.3.0) replace it 352 | # with its value (Ex: 10.12.2). This is also a fix for the 353 | # count. Yay for coding in a bubble! 354 | if osVersion is not None: 355 | for k, v in friendlyDarwin.items(): 356 | if k == osVersion: 357 | osVersion = v 358 | 359 | # The iOS family is more fun, in that Caching Server logs 360 | # the model identifier. 361 | if osFamily == 'iOS': 362 | # Ex: 'model/iPhone7,2'. 363 | iOSModel = re.match( 364 | r'.+? model/([^ ]+?[0-9]+,?[0-9])?', x) 365 | # Since the regular expression is now two goups, only 366 | # take the date from the 2nd group. 367 | # Write the osVersion/osFamily data to iOSModelLog, 368 | # iOSModelOnlyLog and OSLog. 369 | iOSModelLog.append((osVersion, iOSModel.group(1))) 370 | iOSModelOnlyLog.append(iOSModel.group(1)) 371 | OSLog.append((osVersion, osFamily)) 372 | elif osFamily == 'macOS': 373 | # Write the osVersion/osFamily data to OSLog. 374 | OSLog.append((osVersion, osFamily)) 375 | elif configurator == 'Configurator': 376 | AC2Log.append(configurator) 377 | 378 | # if 'model/AppleTV' in logmsg: 379 | # I think I still need to do this section but I can't 380 | # remember. 381 | # 382 | # 383 | # End of OS Family, OS Version and Device section 384 | 385 | # Beginning of File Type section 386 | # 387 | # 388 | # Regular Expression Part. Using the URL (split early), 389 | # Look for the recognized filetypes (.pkg, .ipa, .ipsw, 390 | # .zip and .epub). Ex: 391 | # 1. '/a-09f98d6971/pre-thinned756.thinned.signed.dpkg.ipa' 392 | # 2. '/031-8/com_apple_MobileAsset_CoreSuggestion/6c93.zip' 393 | # 3. '[icloud:hvRq3yMBV7JO9hUBRo2p]' 394 | if re.match(r'.+(\.pkg|\.ipa|\.ipsw|\.zip|\.epub)', URL): 395 | fileType = re.match( 396 | r'.+(\.pkg|\.ipa|\.ipsw|\.zip|\.epub)', URL) 397 | fileTypeLog.append(fileType.group(1)) 398 | # Notice Example 3 posted above. Those are the odd URLs for 399 | # Personal iCloud data. Since it has no discernable suffix, 400 | # log a value of 'personal iCloud'. :shrug: 401 | elif re.match(r'.+(\icloud)', URL): 402 | fileType = re.match(r'.+(\icloud)', URL) 403 | fileTypeLog.append('personal iCloud') 404 | # 405 | # 406 | # End of File Type section 407 | # 408 | # 409 | # End of Server downloads section 410 | 411 | # except: 412 | # print x 413 | # raise Exception("Funky line - check it out") 414 | # Beginning of the final output. 415 | # 416 | # 417 | # Append to a new list. This then allows us to call it whenever we need. 418 | # We can then put this into the Server Alert, stdout, Slack, etc. 419 | finalOutput.append( 420 | 'Cacher has retrieved the following stats for %s:' % targetDate) 421 | finalOutput.append('') 422 | # Add up our bytes from each store from our list to get a total 423 | totalbytesserved = sum(map(int, totalbytesserved)) 424 | totalbytesfromorigin = sum(map(int, totalbytesfromorigin)) 425 | totalbytesfrompeers = sum(map(int, totalbytesfrompeers)) 426 | # Bail here since there aren't any bandwidth stats. 427 | if not totalbytesserved: 428 | print 'Cacher did not retrieve any stats for %s' % targetDate 429 | sys.exit(1) 430 | 431 | finalOutput.append( 432 | '%s of bandwith served to client devices.' % ( 433 | convert_bytes_to_human_readable(totalbytesserved))) 434 | finalOutput.append( 435 | ' %s of bandwith requested from Apple' % ( 436 | convert_bytes_to_human_readable(totalbytesfromorigin))) 437 | finalOutput.append( 438 | ' %s of bandwith requested from other Caching Servers' % ( 439 | convert_bytes_to_human_readable(totalbytesfrompeers))) 440 | finalOutput.append('') 441 | 442 | # Total Numbers of IP addresses 443 | finalOutput.append( 444 | '%s IP Addresses hit the Caching Server yesterday consisting' 445 | ' of:' % len(IPLog)) 446 | finalOutput.append(' %s Unique IP Addresses.' % len(set(IPLog))) 447 | finalOutput.append('') 448 | 449 | # Total Number of iOS devices 450 | # Don't display if 0 downloads 451 | if len(iOSModelOnlyLog) > 0: 452 | finalOutput.append( 453 | 'A total of %s iOS downloads were requested ' 454 | 'from the Caching Server yesterday consisting of:' 455 | % len(iOSModelOnlyLog)) 456 | 457 | # Sort the list by device type (AppleTV, iPad, iPhone, iPod). If we aren't 458 | # using the friendly names, we use the standard sorting, but if we use the 459 | # friendly names, we will sort the list at the very end. 460 | if friendlyNames: 461 | # Friendly Name Sorting: 462 | # In order to sort the friendly names properly, we create a new list, 463 | # counting the amount of devices and swapping the key/value pairs from 464 | # the friendly names. Since we have to sort by friendly name, we create 465 | # a new list based off the following: modeltype/numberofdevices. We 466 | # then split this output on the "/" which gives us the number of the 467 | # number of devices and the modeltype in proper order. 468 | # Example: 469 | # iPhone3,1 becomes iPhone 4 [GSM]/numberofDevices which is then sorted 470 | # and finally split. 471 | for x in set(iOSModelOnlyLog): 472 | numberofDevices = iOSModelOnlyLog.count(x) 473 | modeltype = x 474 | for k, v in friendlyModels.items(): 475 | if k == modeltype: 476 | modeltype = v 477 | FriendlyLog.append('%s/%s' % (modeltype, numberofDevices)) 478 | if 'Apple TV' in modeltype: 479 | AppleTVNumberLog.append('%s' % numberofDevices) 480 | elif 'iPad' in modeltype: 481 | iPadNumberLog.append('%s' % numberofDevices) 482 | elif 'iPhone' in modeltype: 483 | iPhoneNumberLog.append('%s' % numberofDevices) 484 | elif 'iPod' in modeltype: 485 | iPodNumberLog.append('%s' % numberofDevices) 486 | # Force conversion of lists to int 487 | AppleTVNumberLog = [int(i) for i in AppleTVNumberLog] 488 | iPadNumberLog = [int(i) for i in iPadNumberLog] 489 | iPhoneNumberLog = [int(i) for i in iPhoneNumberLog] 490 | iPodNumberLog = [int(i) for i in iPodNumberLog] 491 | # Output 492 | if sum(AppleTVNumberLog) > 0: 493 | finalOutput.append( 494 | ' A total of %s Apple TV downloads' % sum(AppleTVNumberLog)) 495 | if sum(iPadNumberLog) > 0: 496 | finalOutput.append( 497 | ' A total of %s iPad downloads' % sum(iPadNumberLog)) 498 | if sum(iPhoneNumberLog) > 0: 499 | finalOutput.append( 500 | ' A total of %s iPhone downloads' % sum(iPhoneNumberLog)) 501 | if sum(iPodNumberLog) > 0: 502 | finalOutput.append( 503 | ' A total of %s iPod downloads' % sum(iPodNumberLog)) 504 | for x in sorted(set(FriendlyLog)): 505 | numberofDevices = x.split('/')[1] 506 | modeltype = x.split('/')[0] 507 | finalOutput.append(' %s %s' % (numberofDevices, modeltype)) 508 | else: 509 | # Non Friendly Name Sorting: 510 | # This one is easier than friendly names as it's alphabetized by 511 | # sorted(). Count the devices and prefix it on the output. 512 | for x in sorted(set(iOSModelOnlyLog)): 513 | numberofDevices = iOSModelOnlyLog.count(x) 514 | modeltype = x 515 | if 'AppleTV' in modeltype: 516 | AppleTVNumberLog.append('%s' % numberofDevices) 517 | elif 'iPad' in modeltype: 518 | iPadNumberLog.append('%s' % numberofDevices) 519 | elif 'iPhone' in modeltype: 520 | iPhoneNumberLog.append('%s' % numberofDevices) 521 | elif 'iPod' in modeltype: 522 | iPodNumberLog.append('%s' % numberofDevices) 523 | # Force conversion of lists to int 524 | AppleTVNumberLog = [int(i) for i in AppleTVNumberLog] 525 | iPadNumberLog = [int(i) for i in iPadNumberLog] 526 | iPhoneNumberLog = [int(i) for i in iPhoneNumberLog] 527 | iPodNumberLog = [int(i) for i in iPodNumberLog] 528 | # Output 529 | if sum(AppleTVNumberLog) > 0: 530 | finalOutput.append( 531 | ' A total of %s Apple TV downloads' % sum(AppleTVNumberLog)) 532 | if sum(iPadNumberLog) > 0: 533 | finalOutput.append( 534 | ' A total of %s iPad downloads' % sum(iPadNumberLog)) 535 | if sum(iPhoneNumberLog) > 0: 536 | finalOutput.append( 537 | ' A total of %s iPhone downloads' % sum(iPhoneNumberLog)) 538 | if sum(iPodNumberLog) > 0: 539 | finalOutput.append( 540 | ' A total of %s iPod downloads' % sum(iPodNumberLog)) 541 | for x in sorted(set(iOSModelOnlyLog)): 542 | numberofDevices = iOSModelOnlyLog.count(x) 543 | modeltype = x 544 | finalOutput.append(' %s %s' % (numberofDevices, modeltype)) 545 | 546 | finalOutput.append('') 547 | 548 | # Total Number of OS Versions 549 | if len(OSLog) > 0: 550 | finalOutput.append( 551 | 'A total of %s OS downloads were requested from the Caching Server' 552 | ' yesterday consisting of:' % len(OSLog)) 553 | for x in sorted(set(OSLog)): 554 | numberofVersions = OSLog.count(x) 555 | osversion = x[0] 556 | osfamily = x[1] 557 | if osfamily == 'macOS': 558 | macOSFamilyLog.append( 559 | '%s/%s' % (osfamily + ' ' + osversion, numberofVersions)) 560 | macOSDeviceNumber.append(numberofVersions) 561 | elif osfamily == 'iOS': 562 | iOSFamilyLog.append( 563 | '%s/%s' % (osfamily + ' ' + osversion, numberofVersions)) 564 | iOSDeviceNumber.append(numberofVersions) 565 | 566 | # Sort the iOS versions with LooseVersion. StrictVersion fails since I am 567 | # cheating and adding /devicecount to the version. (Ex. iOS 10.2/2000) 568 | if sum(iOSDeviceNumber) > 0: 569 | finalOutput.append(' %s iOS downloads:' % sum(iOSDeviceNumber)) 570 | for x in sorted(set(iOSFamilyLog), key=LooseVersion): 571 | numberofVersions = x.split('/')[1] 572 | modeltype = x.split('/')[0] 573 | finalOutput.append(' %s %s' % (numberofVersions, modeltype)) 574 | 575 | # Sort the macOS versions normally, since they all start with 10. 576 | if sum(macOSDeviceNumber) > 0: 577 | finalOutput.append(' %s macOS downloads:' % sum(macOSDeviceNumber)) 578 | for x in sorted(set(macOSFamilyLog)): 579 | numberofVersions = x.split('/')[1] 580 | modeltype = x.split('/')[0] 581 | finalOutput.append(' %s %s' % (numberofVersions, modeltype)) 582 | finalOutput.append('') 583 | 584 | # Total Number of Apple Configurator 2 files. 585 | # I need logs with Apple Configurator 2 references so I can rewrite this. 586 | # Since you can't disintinguish between the version of AC2, I'm removing 587 | # the secondary line I had in the shell version. 588 | if len(AC2Log) > 0: 589 | finalOutput.append( 590 | 'A total of %s files were requested from Apple' 591 | ' Configurator 2 devices' % len(AC2Log)) 592 | finalOutput.append('') 593 | 594 | # Total Number of filetypes downloaded and their respect numbers 595 | if len(fileTypeLog) > 0: 596 | finalOutput.append( 597 | 'A total of %s files were downloaded from the Caching' 598 | ' Server yesterday consisting of:' % len(fileTypeLog)) 599 | for x in set(fileTypeLog): 600 | numberofFiles = fileTypeLog.count(x) 601 | finalOutput.append(' %s %s files' % (numberofFiles, x)) 602 | finalOutput.append('') 603 | 604 | # Total Number of unique filetypes downloaded and their respect numbers 605 | urlUniqueLog = set(urlLog) 606 | if len(urlUniqueLog) > 0: 607 | finalOutput.append( 608 | 'A total of %s unique files were downloaded from the' 609 | ' Caching Server yesterday consisting' 610 | ' of:' % len(urlUniqueLog)) 611 | # Same logic taken from "File Type Section" so I'm not documenting it. 612 | for x in urlUniqueLog: 613 | if re.match(r'.+(\.pkg|\.ipa|\.ipsw|\.zip|\.epub)', x): 614 | fileType = re.match( 615 | r'.+(\.pkg|\.ipa|\.ipsw|\.zip|\.epub)', x) 616 | fileTypeUniqueLog.append(fileType.group(1)) 617 | elif re.match(r'.+(\icloud)', URL): 618 | fileType = re.match(r'.+(\icloud)', x) 619 | fileTypeUniqueLog.append('personal iCloud') 620 | for x in set(sorted(fileTypeUniqueLog)): 621 | numberofFiles = fileTypeUniqueLog.count(x) 622 | finalOutput.append(' %s %s files' % (numberofFiles, x)) 623 | finalOutput.append('') 624 | # Add Cacher version 625 | finalOutput.append('Cacher version: %s' % version) 626 | finalOutput.append('Uptime: %s' % get_uptime()) 627 | # Check to see if there are entries in the noClientLog. If there are, 628 | # print to final message to warn the user. 629 | if noClientIdentityLog: 630 | finalOutput.append('') 631 | finalOutput.append( 632 | "WARNING: Found %s logs that did not contain " 633 | "the client identity. These logs have been dropped and are not " 634 | "counted in the statistics. More than likely LogClientIdentity " 635 | "was incorrectly set or not configured on this date." 636 | % len(noClientIdentityLog)) 637 | # 638 | # 639 | # End of the final output. 640 | return finalOutput 641 | # print("\n".join(finalOutput)) 642 | 643 | 644 | def convert_bytes_to_human_readable(number_of_bytes): 645 | if number_of_bytes < 0: 646 | raise ValueError("ERROR: number of bytes can not be less than 0") 647 | 648 | step_to_greater_unit = 1024. 649 | number_of_bytes = float(number_of_bytes) 650 | unit = 'bytes' 651 | if (number_of_bytes / step_to_greater_unit) >= 1: 652 | number_of_bytes /= step_to_greater_unit 653 | unit = 'KB' 654 | 655 | if (number_of_bytes / step_to_greater_unit) >= 1: 656 | number_of_bytes /= step_to_greater_unit 657 | unit = 'MB' 658 | 659 | if (number_of_bytes / step_to_greater_unit) >= 1: 660 | number_of_bytes /= step_to_greater_unit 661 | unit = 'GB' 662 | 663 | if (number_of_bytes / step_to_greater_unit) >= 1: 664 | number_of_bytes /= step_to_greater_unit 665 | unit = 'TB' 666 | 667 | precision = 1 668 | number_of_bytes = round(number_of_bytes, precision) 669 | return str(number_of_bytes) + ' ' + unit 670 | 671 | 672 | def check_serverconfig(): 673 | try: 674 | config = '/Library/Server/Caching/Config/Config.plist' 675 | plist = plistlib.readPlist(config) 676 | return plist['LogClientIdentity'] 677 | except Exception: 678 | return None 679 | 680 | 681 | def get_serverversion(): 682 | try: 683 | serverversion = '/Applications/Server.app/Contents/version.plist' 684 | plist = plistlib.readPlist(serverversion) 685 | return plist['CFBundleShortVersionString'] 686 | except Exception: 687 | return None 688 | 689 | 690 | def get_uptime(): 691 | try: 692 | cmd = ['/usr/bin/uptime'] 693 | proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, 694 | stderr=subprocess.PIPE) 695 | output, err = proc.communicate() 696 | splitout = str.split(output) 697 | uptimeamount = splitout[2] 698 | # Uptime Type to use if units are "less than days" 699 | uptimetype = splitout[3].replace(',', '') 700 | if uptimeamount[-1:]==',': 701 | # Last char is a comma; this likely indicates that the 702 | # `uptimetype` is in hours and not in a "greater" unit 703 | uptimeamount = uptimeamount[:-1] # get rid of the comma at the end 704 | uptimeamount = uptimeamount.split(':') 705 | hourtype = ' hour, ' if uptimeamount[0]==1 else ' hours, ' 706 | uptimeamount = uptimeamount[0] + hourtype + uptimeamount[1] 707 | # `uptimetype` to use if main units are in hours and minutes, 708 | # which is not the best way to handle this... but it works. 709 | uptimetype = 'minutes' 710 | return '%s %s' % (uptimeamount, uptimetype) 711 | except Exception: 712 | return None 713 | 714 | 715 | def send_serveralert(targetDate, cacherdata): 716 | try: 717 | # Change to a directory to remove shell error 718 | os.chdir('/private/tmp') 719 | # Mehhhhhhhhhhhhhh 720 | cmd = ['/Applications/Server.app/Contents/ServerRoot/usr/sbin/server ' 721 | 'postAlert CustomAlert Common subject ' + '"' 722 | 'Caching Server Data: ' + targetDate + '"' + ' message ' 723 | '"' + cacherdata + '"<<<""'] 724 | subprocess.check_call(cmd, shell=True) 725 | except Exception: 726 | return None 727 | 728 | 729 | def configureserver(): 730 | try: 731 | cmd = [ 732 | '/Applications/Server.app/Contents/ServerRoot/usr/sbin/server' 733 | 'admin', 'settings', 'caching:LogClientIdentity = yes'] 734 | proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, 735 | stderr=subprocess.PIPE) 736 | output, err = proc.communicate() 737 | return output.rstrip() 738 | except Exception: 739 | return None 740 | 741 | 742 | def serveradmin(action, service): 743 | try: 744 | cmd = [ 745 | '/Applications/Server.app/Contents/ServerRoot/usr/sbin/server' 746 | 'admin', action, service] 747 | proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, 748 | stderr=subprocess.PIPE) 749 | output, err = proc.communicate() 750 | return output.rstrip() 751 | except Exception: 752 | return None 753 | 754 | 755 | def post_to_slack(targetDate, cacherdata, slackchannel, slackusername, 756 | slackwebhook): 757 | # Server App Icon DL 758 | url = 'https://itunes.apple.com/lookup?id=883878097' 759 | try: 760 | request = urllib2.urlopen(url) 761 | jsondata = json.loads(request.read()) 762 | iconurl = jsondata['results'][0]['artworkUrl100'] 763 | except (urllib2.URLError, ValueError, KeyError) as e: 764 | # hardcode icon url in case it fails. 765 | iconurl = 'http://is5.mzstatic.com/image/thumb/Purple122/v4/b9/e8/c4' \ 766 | '/b9e8c4b9-ce9c-174a-c1a8-d0ad0fc21da9/source/100x100bb.png' 767 | # Slack payload 768 | payload = { 769 | "channel": slackchannel, 770 | "username": slackusername, 771 | "icon_url": iconurl, 772 | "attachments": [ 773 | { 774 | 'pretext': 'Caching Server Data ' + targetDate, 775 | 'text': cacherdata 776 | } 777 | ] 778 | } 779 | try: 780 | cmd = ['/usr/bin/curl', '-X', 'POST', '--data-urlencode', 781 | 'payload=' + json.dumps(payload), slackwebhook] 782 | proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, 783 | stderr=subprocess.PIPE) 784 | output, err = proc.communicate() 785 | except Exception: 786 | print 'Failed to send message to Slack' 787 | 788 | 789 | def main(): 790 | # Check for macOS Server 5.2 or higher. Use LooseVersion just in case. 791 | if LooseVersion(get_serverversion()) >= LooseVersion('5.2'): 792 | pass 793 | else: 794 | print "Server version is %s and not compatible" % get_serverversion() 795 | sys.exit(1) 796 | 797 | # Options 798 | usage = '%prog [options]' 799 | o = optparse.OptionParser(usage=usage) 800 | o.add_option('--targetdate', 801 | help=('Optional: Date to parse. Example: 2017-01-15.')) 802 | o.add_option('--logpath', 803 | help=('Optional: Caching Log Path. Defaults to: ' 804 | '/Library/Server/Caching/Logs')) 805 | o.add_option('--deviceids', 806 | help='Optional: Use Device IDs (Ex: iPhone7,2). Defaults' 807 | ' to: False', 808 | action='store_true') 809 | o.add_option('--nostdout', 810 | help='Optional: Do not print to standard out', 811 | action='store_true') 812 | o.add_option('--configureserver', 813 | help='Optional: Configure Server to log Client Data', 814 | action='store_true') 815 | o.add_option('--serveralert', 816 | help='Optional: Send Server Alert', 817 | action='store_true') 818 | o.add_option("--slackalert", action="store_true", default=False, 819 | help=("Optional: Use Slack")) 820 | o.add_option("--slackwebhook", default=None, 821 | help=("Optional: Slack Webhook URL. Requires Slack Option.")) 822 | o.add_option("--slackusername", default=None, 823 | help=("Optional: Slack username. Defaults to Cacher." 824 | "Requires Slack Option.")) 825 | o.add_option("--slackchannel", default=None, 826 | help=("Optional: Slack channel. Can be username or channel." 827 | "Ex. #channel or @username. Requires Slack Option.")) 828 | 829 | opts, args = o.parse_args() 830 | 831 | # Configure Server 832 | if opts.configureserver: 833 | configureServer = True 834 | else: 835 | configureServer = False 836 | if configureServer: 837 | if os.getuid() != 0: 838 | print 'Did not configure Caching Server - requires root' 839 | sys.exit(1) 840 | else: 841 | print 'Caching Server settings are now: ' + configureserver() 842 | print '\nRestarting Caching Service...' 843 | print '\n' + serveradmin('stop', 'caching') 844 | print '\n' + serveradmin('start', 'caching') 845 | sys.exit(1) 846 | 847 | # Check if LogClientIdentity is configured correctly. If it isn't - bail. 848 | serverconfig = check_serverconfig() 849 | if serverconfig is True: 850 | pass 851 | elif type(serverconfig) is str or type(serverconfig) is int: 852 | print "LogClientIdentity is incorrectly set to: %s - Type: %s" \ 853 | % (str(serverconfig), type(serverconfig).__name__) 854 | print "Please run sudo Cacher --configureserver and delete your " \ 855 | "log files." 856 | sys.exit(1) 857 | elif not serverconfig: 858 | print "LogClientIdentity is not set" 859 | print "Please run sudo Cacher --configureserver and delete your " \ 860 | "log files." 861 | sys.exit(1) 862 | else: 863 | print "LogClientIdentity is set to: %s" % str(serverconfig) 864 | print "Please run sudo Cacher --configureserver and delete your " \ 865 | "log files." 866 | sys.exit(1) 867 | 868 | # Grab other options 869 | if opts.targetdate: 870 | targetDate = opts.targetdate 871 | else: 872 | targetDate = str(date.today() - timedelta(1)) 873 | if opts.logpath: 874 | logPath = opts.logpath 875 | else: 876 | logPath = '/Library/Server/Caching/Logs' 877 | if opts.deviceids: 878 | friendlyNames = False 879 | else: 880 | friendlyNames = True 881 | if opts.nostdout: 882 | stdOut = False 883 | else: 884 | stdOut = True 885 | if opts.serveralert: 886 | serverAlert = True 887 | else: 888 | serverAlert = False 889 | if opts.slackalert: 890 | slackAlert = True 891 | else: 892 | slackAlert = False 893 | slackalert = opts.slackalert 894 | slackwebhook = opts.slackwebhook 895 | if opts.slackusername: 896 | slackusername = opts.slackusername 897 | else: 898 | slackusername = 'Cacher' 899 | slackchannel = opts.slackchannel 900 | 901 | # Check if log files exist and if not, bail. Try to delete .DS_Store files 902 | # just in case they exist from the GUI. Chances are we can delete this 903 | # because we are either running as root or the same user that created it. 904 | try: 905 | os.remove(os.path.join(logPath, '.DS_Store')) 906 | except OSError: 907 | pass 908 | if not os.listdir(logPath): 909 | print 'Cacher did not detect log files in %s' % logPath 910 | sys.exit(1) 911 | 912 | # Make temporary directory 913 | tmpDir = tempfile.mkdtemp() 914 | 915 | # Clone the contents of serverlogs over into the 'cachinglogs' subdirectory 916 | tmpLogs = os.path.join(tmpDir, 'cachinglogs') 917 | shutil.copytree(logPath, tmpLogs) 918 | 919 | # Expand any .bz files in the directory (Server 4.1+) 920 | os.chdir(tmpLogs) 921 | for bzLog in glob.glob(os.path.join(tmpLogs, '*.bz2')): 922 | result = subprocess.check_call(["bunzip2", bzLog]) 923 | 924 | # Now combine all .log files in the destination into a temp file that's 925 | # removed when python exits 926 | rawLog = tempfile.TemporaryFile() 927 | # We only care about Debug logs, not service logs 928 | for anyLog in glob.glob(os.path.join(tmpLogs, 'Debug*')): 929 | with open(anyLog, 'rb') as f: 930 | shutil.copyfileobj(f, rawLog) 931 | 932 | # Skip back to the beginning of our newly concatenated log 933 | rawLog.seek(0) 934 | 935 | # Purge temporary directory since it's now in memory. 936 | shutil.rmtree(tmpDir) 937 | 938 | # Run the function that does most of the work. 939 | cacherdata = cacher(rawLog.readlines(), targetDate, friendlyNames) 940 | # Output conditionals 941 | if stdOut: 942 | print("\n".join(cacherdata)) 943 | if slackAlert: 944 | print '' 945 | if serverAlert: 946 | if os.getuid() != 0: 947 | print 'Did not send serverAlert - requires root' 948 | else: 949 | send_serveralert(targetDate, "\n".join(cacherdata)) 950 | if slackalert is True: 951 | post_to_slack(targetDate, "\n".join(cacherdata), slackchannel, 952 | slackusername, slackwebhook) 953 | 954 | 955 | if __name__ == '__main__': 956 | main() 957 | -------------------------------------------------------------------------------- /images/CacherServerAlert.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/erikng/Cacher/bc816e6c0683bd20daddba9869deef9583ac5549/images/CacherServerAlert.png -------------------------------------------------------------------------------- /images/CacherSlack_Large.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/erikng/Cacher/bc816e6c0683bd20daddba9869deef9583ac5549/images/CacherSlack_Large.png -------------------------------------------------------------------------------- /images/CacherSlack_Small.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/erikng/Cacher/bc816e6c0683bd20daddba9869deef9583ac5549/images/CacherSlack_Small.png --------------------------------------------------------------------------------