├── .gitignore ├── LICENSE ├── README.md ├── banana ├── garbage_collection.json └── overview.json ├── cassandra_ingest ├── cassandra_store.py ├── clography.py ├── log_to_json ├── rules.py ├── schema ├── add-schema.sh ├── schema.xml ├── solrconfig.xml └── systemlog.cql └── systemlog.py /.gitignore: -------------------------------------------------------------------------------- 1 | config.py 2 | *.pyc 3 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | 177 | END OF TERMS AND CONDITIONS 178 | 179 | APPENDIX: How to apply the Apache License to your work. 180 | 181 | To apply the Apache License to your work, attach the following 182 | boilerplate notice, with the fields enclosed by brackets "[]" 183 | replaced with your own identifying information. (Don't include 184 | the brackets!) The text should be enclosed in the appropriate 185 | comment syntax for the file format. We also recommend that a 186 | file or class name and description of purpose be included on the 187 | same "printed page" as the copyright notice for easier 188 | identification within third-party archives. 189 | 190 | Copyright [yyyy] [name of copyright owner] 191 | 192 | Licensed under the Apache License, Version 2.0 (the "License"); 193 | you may not use this file except in compliance with the License. 194 | You may obtain a copy of the License at 195 | 196 | http://www.apache.org/licenses/LICENSE-2.0 197 | 198 | Unless required by applicable law or agreed to in writing, software 199 | distributed under the License is distributed on an "AS IS" BASIS, 200 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 201 | See the License for the specific language governing permissions and 202 | limitations under the License. 203 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Cassandra system.log parser 2 | 3 | This rule-based log parser uses regular expressions to match various messages logged 4 | by Cassandra and extract any useful information they contain into separate fields. 5 | Additional transformations can be applied to each of the captured values, and then 6 | a dictionary containing the resulting values on each line is returned. The dictionaries 7 | can be output in json format or inserted into a storage backend. 8 | 9 | ## log_to_json 10 | 11 | The [log_to_json](log_to_json) script parses system.log and outputs events in JSON format with one 12 | event per line. It takes a list of log files on the command line and parses them. 13 | If no arguments are supplied it will attempt to parse stdin. This can be used to parse 14 | a live log file by piping from tail: `tail -f /var/log/cassandra/system.log | log_to_json`. 15 | 16 | ## cassandra_ingest 17 | 18 | The [cassandra_ingest](cassandra_ingest) script parses system.log and inserts each event into the 19 | logparse.systemlog table defined in [systemlog.cql](systemlog.cql). It takes a list of log files 20 | on the command line and parses them. If no arguments are supplied it will attempt 21 | to parse stdin. This can be used to parse a live log file by piping from tail: 22 | `tail -f /var/log/cassandra/system.log | cassandra_ingest`. 23 | 24 | ## Rule-Based Message Parser 25 | To reduce the tedium of defining parsers for many different messages, I created a simple 26 | DSL using Python function objects. The function objects can be called like normal 27 | functions, but they are created by a constructor which allows you to define the specific 28 | behavior of the resulting function when it is called. 29 | 30 | The function objects themselves are defined in `rules.py`, and the rules specific to the 31 | Cassandra system.log are defined in `systemlog.py`. In the future, I may add additional 32 | sets of rules for Spark executor logs, and OpsCenter daemon and agent logs. These rules 33 | can be used to create parsers for your own application logs as well. 34 | 35 | A minimal set of two rules is defined as follows: 36 | 37 | ``` 38 | capture_message = switch(( 39 | case('CassandraDaemon'), 40 | rule( 41 | capture(r'Heap size: (?P[0-9]*)/(?P[0-9]*)'), 42 | convert(int, 'heap_used', 'total_heap'), 43 | update(event_product='cassandra', event_category='startup', event_type='heap_size')), 44 | 45 | rule( 46 | capture(r'Classpath: (?P.*)'), 47 | convert(split(':'), 'classpath'), 48 | update(event_product='cassandra', event_category='startup', event_type='classpath')))) 49 | ``` 50 | 51 | The `switch(cases)` constructor takes a tuple of cases and rules. It was necessary to use 52 | an actual tuple instead of argument unpacking because the number of rules exceeds the 53 | maximum number of parameters supported by a Python function call. The constructor returns 54 | a function that we assign the name `capture_message`. This function accepts two parameters: 55 | the first determines which group of rules will be applied, and the second is the string 56 | that the selected group of rules will be applied to until a match is found. The function 57 | returns the value returned by the first matching rule. If no rules match, None is returned. 58 | 59 | Rules are grouped using the `case(*keys)` constructor. The keys specified in the case 60 | constructor will be used by the switch function to determine which group of rules to 61 | execute. The keys in a case constructor apply to all of the rules that follow until the 62 | next case constructor is encountered. 63 | 64 | Rules are defined using `rule(source, *transformations)` where source is a function that 65 | is expected to take a string and return a dictionary of fields extracted from the string 66 | if it matches, or None if it doesn't. Unless None is returned, the rule will then pass 67 | the resulting dictionary into the transformation functions in the specified order, each 68 | of which is expected to manipulate the dictionary in some way by adding, removing, or 69 | overwriting fields. 70 | 71 | Currently the only source defined is `capture(*regexes)`, which takes a list of regular expressions 72 | to apply against the input string. Each of the regular expressions will be applied 73 | until the first match is found, and then the match's groupdict will be returned. If no 74 | matches are found, None is returned. 75 | 76 | Several transformations are provided: 77 | 78 | - `convert(function, *field_names)` will iterate over the k/v pairs in a dictionary and 79 | apply the specified function to convert the specified fields to a different type or 80 | perform some some other transformation on the string value. The function can be a simple 81 | type conversion such as int or float, or it can be a user-defined function or the 82 | constructor for a function object. The field names are just one or more strings 83 | specifying the dictionary field that the conversion should be applied to. The convert 84 | function will iterate over the fields specified and apply the conversion function to 85 | each, replacing the value of the field with the result. 86 | 87 | - `update(**fields)` simply adds the specified key-value pairs to the dictionary. This can 88 | be used to tag the event with a category or type based on the regular expression that has 89 | matched it. 90 | 91 | - `default(**fields)` is the same as update, but it will only set the key/value pairs for 92 | fields that do not already exist within the dictionary. 93 | 94 | `systemlog.py` defines a capture_line rule to match the overall log line of the format: 95 | 96 | ``` 97 | level [thread] date sourcefile:sourceline - message 98 | ``` 99 | 100 | This rule then passes the sourcefile and message fields to the capture_message function 101 | defined above, which chooses a group of rules based on the sourcefile, then applies them 102 | to the message until a match is found. 103 | 104 | These rules are wrapped by a `parse_log` generator that iterates over a sequence of log lines 105 | and yields a dictionary for each event within the log. This has special handling for 106 | exceptions which can follow on separate lines after the main line of an error message. 107 | 108 | In order to test the rules, I created a simple front-end called `log_to_json`, which reads 109 | one or more system.log files (or stdin) and converts each event into a json representation 110 | with one event per line. 111 | 112 | ## Cassandra Storage Backend 113 | 114 | The Cassandra storage backend is designed to store the data generated by the log parser in a 115 | flexible schema. Any provided key/value pairs that match the name of a field in the table 116 | will be inserted into the corresponding field. Any pairs that do not match a field in the table 117 | will instead be inserted into a set of generic map fields based on the type of the value. 118 | The table has a map for each common data type, including boolean (b_), date (d_), integer (i_), 119 | float (f_), string (s_), and list (l_). 120 | 121 | The required fields on the table are shown below, and additional fields can be added as desired. 122 | 123 | ``` 124 | create table generic ( 125 | id timeuuid primary key, 126 | b_ map, 127 | d_ map, 128 | i_ map, 129 | f_ map, 130 | s_ map, 131 | l_ map 132 | ); 133 | ``` 134 | 135 | The `genericize` function in `cassandra_store.py` handles the transformation of arbitrary 136 | dictionaries into the format of the parameters expected by the Cassandra Python Driver. 137 | Prior to insertion, any nested dictionaries are flattened by combining the key paths using 138 | underscores. For example `{'a': {'b': 'c'}, 'd': {'e': 'f', 'g': {'h': 'i'}}}` becomes 139 | `{'a_b': 'c', 'd_e': 'f', 'd_g_h': 'i'}`. Since lists can't be nested within a map in 140 | Cassandra, lists are actually expressed as a string where each element of the list 141 | separated by a newline. Anything else will be coerced to JSON and inserted into the string map. 142 | Any fields that are present in the table but not provided in the dictionary will be set to None. 143 | 144 | The `CassandraStore` class connects to the cassandra cluster using the DataStax Python Driver 145 | and handles automatic preparation and caching of insert statements. It provides an `insert` method 146 | to insert a single record into a specified table, either synchronously or asynchronously. 147 | To maximize throughput without overloading the cluster, it provides a `slurp` method that will 148 | concurrently insert records provided by a generator while maintaining an optimal number of inflight 149 | queries. 150 | 151 | ## Solr indexing 152 | 153 | The Cassandra table containing parsed log entries can be indexed using the Solr implementation from 154 | [DataStax Enterprise](http://docs.datastax.com/en/datastax_enterprise/4.7//datastax_enterprise/newFeatures.html). 155 | A [schema.xml](solr/schema.xml) and [solrconfig.xml](solr/solrconfig.xml) are provided 156 | in the [solr](solr) directory along with the [add-schema.sh](solr/add-schema.sh) script 157 | which will upload the Solr schema to DSE. 158 | 159 | Once indexed in Solr, the log events can be subsequently analyzed and visualized using 160 | [Banana](https://github.com/LucidWorks/banana). Banana is a port of Kibana 3.0 to Solr. 161 | Several pre-made dashboards are saved in json format in the [banana](banana) subdirectory. 162 | These can be loaded using the Banana web UI. 163 | 164 | Setup Instructions: 165 | 166 | 1. Clone https://github.com/LucidWorks/banana to $DSE_HOME/resources/banana. 167 | Make sure you've checked out the release branch (should be the default). 168 | If you want, you can `rm -rf .git` at this point to save space. 169 | 170 | 2. Edit resources/banana/src/config.js and: 171 | - change `solr_core` to the core you're most frequently going to work with (only a 172 | convenience, you can pick a different one later on the settings for each dashboard. 173 | - change `banana_index` to `banana.dashboards` (can be anything you want, but modify step 174 | 3 accordingly). Not strictly necessary if you don't want to save dashboards to solr. 175 | 176 | 3. Post the banana schema from `resources/banana/resources/banana-int-solr-4.5/banana-int/conf` 177 | - Use the `solrconfig.xml` from this project instead of the one provided by banana 178 | - Name the core the same name specified above in step 2. 179 | - Not strictly necessary if you don't want to save dashboards to solr. 180 | 181 | ``` 182 | curl --data-binary @solrconfig.xml -H 'Content-type:text/xml; charset=utf-8' "http://localhost:8983/solr/resource/banana.dashboards/solrconfig.xml" 183 | curl --data-binary @schema.xml -H 'Content-type:text/xml; charset=utf-8' "http://localhost:8983/solr/resource/banana.dashboards/schema.xml" 184 | curl -X POST -H 'Content-type:text/xml; charset=utf-8' "http://localhost:8983/solr/admin/cores?action=CREATE&name=banana.dashboards" 185 | ``` 186 | 187 | 4. Edit resources/tomcat/conf/server.xml and add the following inside the `` tags: 188 | 189 | ``` 190 | 191 | ``` 192 | 193 | 5. If you've previously started DSE, remove `resources/tomcat/work` and restart. 194 | 195 | 6. Start DSE and go to http://localhost:8983/banana 196 | -------------------------------------------------------------------------------- /banana/garbage_collection.json: -------------------------------------------------------------------------------- 1 | { 2 | "title": "Garbage Collection", 3 | "services": { 4 | "query": { 5 | "idQueue": [ 6 | 2, 7 | 3, 8 | 4 9 | ], 10 | "list": { 11 | "0": { 12 | "query": "s_gc_type:ParNew", 13 | "alias": "", 14 | "color": "#7EB26D", 15 | "id": 0, 16 | "pin": false, 17 | "type": "lucene" 18 | }, 19 | "1": { 20 | "id": 1, 21 | "color": "#EAB839", 22 | "query": "s_gc_type:ConcurrentMarkSweep", 23 | "alias": "", 24 | "pin": false 25 | } 26 | }, 27 | "ids": [ 28 | 0, 29 | 1 30 | ] 31 | }, 32 | "filter": { 33 | "idQueue": [ 34 | 2, 35 | 3, 36 | 4, 37 | 5, 38 | 6, 39 | 7 40 | ], 41 | "list": { 42 | "0": { 43 | "from": "NOW/YEAR-5YEAR", 44 | "to": "NOW/YEAR%2B1YEAR", 45 | "field": "date", 46 | "type": "time", 47 | "fromDateObj": "2010-07-27T20:01:37.809Z", 48 | "toDateObj": "2015-07-27T20:01:37.810Z", 49 | "mandate": "must", 50 | "active": true, 51 | "alias": "", 52 | "id": 0 53 | }, 54 | "1": { 55 | "type": "range", 56 | "from": 654, 57 | "to": 7508, 58 | "field": "i_duration", 59 | "mandate": "must", 60 | "active": true, 61 | "alias": "", 62 | "id": 1 63 | } 64 | }, 65 | "ids": [ 66 | 0, 67 | 1 68 | ] 69 | } 70 | }, 71 | "rows": [ 72 | { 73 | "title": "Help", 74 | "height": "150px", 75 | "editable": true, 76 | "collapse": true, 77 | "collapsable": true, 78 | "panels": [ 79 | { 80 | "error": false, 81 | "span": 12, 82 | "editable": true, 83 | "type": "text", 84 | "loadingEditor": false, 85 | "status": "Stable", 86 | "mode": "markdown", 87 | "content": "### Garbage Collection\n\n*Garbage collection* reclaims objects from the Java heap once they are no longer needed. Unfortunately, while some phases of garbage collection are running, the application is frozen. For Cassandra, long pauses can lead to high latency, node flapping, hint buildup, high CPU utilization, and other problems. Cassandra logs any garbage collection that takes longer than 200 ms.\n\nThe Java heap is divided into several *generations*, where objects are placed depending on how long they have survived: eden space, survivor spaces, and tenured generation. There are two different types of collections that run depending on the generation: *ParNew* and *ConcurrentMarkSweep*. \n\nParNew cleans up the eden space, and it always stops the application. Luckily, it usually runs quickly because the young generation takes up a fraction of the heap. However, if there is not enough space to promote an object to the old generation, this causes a *promotion failure*, which can cause the ParNew to take much longer.\n\nConcurrentMarkSweep cleans up the tenured generation. As the name suggests, it is normally concurrent, so it does not stop the application. However, when the concurrent collector is unable to keep up with the rate at which garbage is generated, it will perform a full collection that stops the application. This is called a *concurrent mode failure*. When a concurrent mode failure occurs, the resulting pause is usually very long because the entire old generation must be collected, which usually represents a significant majority of the heap.\n\nThis dashboard allows you to analyze the garbage collections in your system.log according to occurrence time, type, and duration. \n\n#### Search Parameters\nIn the first row, you can select the time window and the search terms. Two terms for ParNew and ConcurrentMarkSweep GC types have already been entered and color coded according to type. The third panel shows how many results have been found.\n\n#### Filtering\nThe second row allows you to control the filters applied to the dashboard. The left panel shows the filters that have already been applied to the dashboard. To remove a previously applied filter, click the x on the box representing the filter. The right panel allows you to choose a specific log file or host. The number of GC events in each log and for each host are shown, allowing you to zero in on trouble spots.\n\n#### Event Counts\n\nThe third row is a time-series histogram that shows the number of GC events that occurred within a specific time window. You can use this to determine when trouble occurred. This view also allows you to drag over a specific time period to zoom in on that period. All the other graphs will adjust to show only events within that time period.\n\n#### Breakdown\n\nThe left panel on the fourth row shows a pie chart with the break down of garbage collection types (ConcurrentMarkSweep and ParNew). You can click on a slice in the pie chart to filter for only that type of garbage collection. \n\nThe right panel is a histogram of the GC duration. This divides the duration into ranges and shows how many collections fell within that range. You can use this to focus on collections that fell within a certain duration. Like the event count window, you can drag over a specific range to show only events with a duration in that range. You can use this to focus on the worst garbage collection problems. Keep in mind that only those events within the selected range will be shown, so you may need to zoom out to see all the collections.\n\n#### Data Table\nFinally, the table at the bottom shows the date, garbage collection type, duration, and the sizes of the eden and old generation before and after. You can click on a row to view full details about the collection. You can also click on a heading to sort by a specific field. By default the table is by duration in descending order.", 88 | "style": {}, 89 | "title": "Help" 90 | } 91 | ] 92 | }, 93 | { 94 | "title": "Query and Time Window", 95 | "height": "50px", 96 | "editable": true, 97 | "collapse": false, 98 | "collapsable": true, 99 | "panels": [ 100 | { 101 | "error": "", 102 | "span": 6, 103 | "editable": true, 104 | "type": "timepicker", 105 | "loadingEditor": false, 106 | "status": "Stable", 107 | "mode": "absolute", 108 | "time_options": [ 109 | "5m", 110 | "15m", 111 | "1h", 112 | "6h", 113 | "12h", 114 | "24h", 115 | "7d", 116 | "30d", 117 | "90d", 118 | "1y", 119 | "5y" 120 | ], 121 | "timespan": "5y", 122 | "timefield": "date", 123 | "timeformat": "", 124 | "refresh": { 125 | "enable": false, 126 | "interval": 30, 127 | "min": 3 128 | }, 129 | "filter_id": 0, 130 | "spyable": true, 131 | "title": "Time Window", 132 | "time": { 133 | "from": "07/27/2010 16:01:37", 134 | "to": "07/27/2015 16:01:37" 135 | } 136 | }, 137 | { 138 | "error": false, 139 | "span": 4, 140 | "editable": true, 141 | "group": [ 142 | "default" 143 | ], 144 | "type": "query", 145 | "label": "Search", 146 | "history": [ 147 | "s_gc_type:ConcurrentMarkSweep", 148 | "s_gc_type:ParNew", 149 | "!s_gc_type:(ParNew ConcurrentMarkSweep)", 150 | "!s_gc_type:ParNew", 151 | "-s_gc_type:(ParNew ConcurrentMarkSweep)", 152 | "s_gc_type:(ParNew ConcurrentMarkSweep)", 153 | "s_gc_type:(\"ParNew\" \"ConcurrentMarkSweep\"", 154 | "-s_gc_type:(\"ParNew\" \"ConcurrentMarkSweep\"", 155 | "-s_gc_type:(\"ParNew\" \"ConcurrentMarkSweep\")", 156 | "s_gc_type:!(ParNew ConcurrentMarkSweep)" 157 | ], 158 | "remember": 10, 159 | "pinned": true, 160 | "query": "*:*", 161 | "title": "Search Terms", 162 | "spyable": true, 163 | "def_type": "" 164 | }, 165 | { 166 | "span": 2, 167 | "editable": true, 168 | "type": "hits", 169 | "loadingEditor": false, 170 | "queries": { 171 | "mode": "all", 172 | "ids": [ 173 | 0, 174 | 1 175 | ], 176 | "query": "q=s_gc_type%3AParNew&fq=event_category:garbage_collection&fq=event_type:pause&fq=date:[NOW/YEAR-5YEAR%20TO%20NOW/YEAR%2B1YEAR]&fq=i_duration:[654 TO 7508]&wt=json&rows=0\nq=s_gc_type%3AConcurrentMarkSweep&fq=event_category:garbage_collection&fq=event_type:pause&fq=date:[NOW/YEAR-5YEAR%20TO%20NOW/YEAR%2B1YEAR]&fq=i_duration:[654 TO 7508]&wt=json&rows=0\n", 177 | "basic_query": "", 178 | "custom": "" 179 | }, 180 | "style": { 181 | "font-size": "14pt" 182 | }, 183 | "arrangement": "horizontal", 184 | "chart": "total", 185 | "counter_pos": "above", 186 | "donut": false, 187 | "tilt": false, 188 | "labels": true, 189 | "spyable": true, 190 | "title": "Total Hits", 191 | "show_queries": true 192 | } 193 | ] 194 | }, 195 | { 196 | "title": "Filters", 197 | "height": "50px", 198 | "editable": true, 199 | "collapse": false, 200 | "collapsable": true, 201 | "panels": [ 202 | { 203 | "error": false, 204 | "span": 8, 205 | "editable": true, 206 | "spyable": true, 207 | "group": [ 208 | "default" 209 | ], 210 | "type": "filtering", 211 | "title": "Filters" 212 | }, 213 | { 214 | "span": 4, 215 | "editable": true, 216 | "type": "facet", 217 | "loadingEditor": false, 218 | "status": "Stable", 219 | "queries": { 220 | "mode": "all", 221 | "ids": [ 222 | 0, 223 | 1 224 | ], 225 | "query": "q=s_gc_type%3AParNew OR s_gc_type%3AConcurrentMarkSweep&fq=event_category:garbage_collection&fq=event_type:pause&fq=date:[NOW/YEAR-5YEAR%20TO%20NOW/YEAR%2B1YEAR]&fq=i_duration:[654 TO 7508]&facet=true&facet.field=log_file&facet.field=host&wt=json", 226 | "basic_query": "q=s_gc_type%3AParNew OR s_gc_type%3AConcurrentMarkSweep&fq=event_category:garbage_collection&fq=event_type:pause&fq=date:[NOW/YEAR-5YEAR%20TO%20NOW/YEAR%2B1YEAR]&fq=i_duration:[654 TO 7508]&facet=true&facet.field=log_file&facet.field=host", 227 | "custom": "" 228 | }, 229 | "group": "default", 230 | "style": { 231 | "font-size": "9pt" 232 | }, 233 | "overflow": "min-height", 234 | "fields": [ 235 | "log_file", 236 | "host" 237 | ], 238 | "field_list": true, 239 | "spyable": true, 240 | "facet_limit": 50, 241 | "foundResults": true, 242 | "header_title": "Limit Your Search", 243 | "toggle_element": null, 244 | "show_queries": true, 245 | "title": "Log File", 246 | "offset": 0, 247 | "exportSize": null 248 | } 249 | ] 250 | }, 251 | { 252 | "title": "Graph", 253 | "height": "250px", 254 | "editable": true, 255 | "collapse": false, 256 | "collapsable": true, 257 | "panels": [ 258 | { 259 | "span": 12, 260 | "editable": true, 261 | "type": "histogram", 262 | "loadingEditor": false, 263 | "mode": "count", 264 | "time_field": "event_timestamp", 265 | "queries": { 266 | "mode": "all", 267 | "ids": [ 268 | 0, 269 | 1 270 | ], 271 | "query": "q=s_gc_type%3AParNew&fq=event_category:garbage_collection&fq=event_type:pause&wt=json&rows=0&fq=date:[NOW/YEAR-5YEAR%20TO%20NOW/YEAR%2B1YEAR]&fq=i_duration:[654 TO 7508]&facet=true&facet.range=date&facet.range.start=NOW/YEAR-5YEAR&facet.range.end=NOW/YEAR%2B1YEAR&facet.range.gap=%2B7DAY\nq=s_gc_type%3AConcurrentMarkSweep&fq=event_category:garbage_collection&fq=event_type:pause&wt=json&rows=0&fq=date:[NOW/YEAR-5YEAR%20TO%20NOW/YEAR%2B1YEAR]&fq=i_duration:[654 TO 7508]&facet=true&facet.range=date&facet.range.start=NOW/YEAR-5YEAR&facet.range.end=NOW/YEAR%2B1YEAR&facet.range.gap=%2B7DAY\n", 272 | "custom": "" 273 | }, 274 | "max_rows": 100000, 275 | "value_field": null, 276 | "group_field": null, 277 | "auto_int": true, 278 | "resolution": 100, 279 | "interval": "7d", 280 | "intervals": [ 281 | "auto", 282 | "1s", 283 | "1m", 284 | "5m", 285 | "10m", 286 | "30m", 287 | "1h", 288 | "3h", 289 | "12h", 290 | "1d", 291 | "1w", 292 | "1M", 293 | "1y" 294 | ], 295 | "fill": 0, 296 | "linewidth": 3, 297 | "timezone": "browser", 298 | "spyable": true, 299 | "zoomlinks": true, 300 | "bars": true, 301 | "stack": true, 302 | "points": false, 303 | "lines": false, 304 | "legend": true, 305 | "x-axis": true, 306 | "y-axis": true, 307 | "percentage": false, 308 | "interactive": true, 309 | "options": true, 310 | "tooltip": { 311 | "value_type": "cumulative", 312 | "query_as_alias": false 313 | }, 314 | "title": "Event Counts", 315 | "lines_smooth": false, 316 | "show_queries": true 317 | } 318 | ] 319 | }, 320 | { 321 | "title": "Breakdown", 322 | "height": "300", 323 | "editable": true, 324 | "collapse": false, 325 | "collapsable": true, 326 | "panels": [ 327 | { 328 | "span": 4, 329 | "editable": true, 330 | "type": "terms", 331 | "loadingEditor": false, 332 | "queries": { 333 | "mode": "all", 334 | "ids": [ 335 | 0, 336 | 1 337 | ], 338 | "query": "q=s_gc_type%3AParNew OR s_gc_type%3AConcurrentMarkSweep&fq=event_category:garbage_collection&fq=event_type:pause&wt=json&rows=0&fq=date:[NOW/YEAR-5YEAR%20TO%20NOW/YEAR%2B1YEAR]&fq=i_duration:[654 TO 7508]&facet=true&facet.field=s_gc_type&facet.limit=10&facet.missing=true", 339 | "custom": "" 340 | }, 341 | "mode": "count", 342 | "field": "s_gc_type", 343 | "stats_field": "", 344 | "decimal_points": 0, 345 | "exclude": [], 346 | "missing": false, 347 | "other": false, 348 | "size": 10, 349 | "order": "descending", 350 | "style": { 351 | "font-size": "10pt" 352 | }, 353 | "donut": false, 354 | "tilt": false, 355 | "labels": true, 356 | "logAxis": false, 357 | "arrangement": "horizontal", 358 | "chart": "pie", 359 | "counter_pos": "below", 360 | "lastColor": "rgb(234,184,57)", 361 | "spyable": true, 362 | "show_queries": true, 363 | "chartColors": [ 364 | "#7EB26D", 365 | "#EAB839", 366 | "#6ED0E0", 367 | "#EF843C", 368 | "#E24D42", 369 | "#1F78C1", 370 | "#BA43A9", 371 | "#705DA0", 372 | "#508642", 373 | "#CCA300", 374 | "#447EBC", 375 | "#C15C17", 376 | "#890F02", 377 | "#0A437C", 378 | "#6D1F62", 379 | "#584477", 380 | "#B7DBAB", 381 | "#F4D598", 382 | "#70DBED", 383 | "#F9BA8F", 384 | "#F29191", 385 | "#82B5D8", 386 | "#E5A8E2", 387 | "#AEA2E0", 388 | "#629E51", 389 | "#E5AC0E", 390 | "#64B0C8", 391 | "#E0752D", 392 | "#BF1B00", 393 | "#0A50A1", 394 | "#962D82", 395 | "#614D93", 396 | "#9AC48A", 397 | "#F2C96D", 398 | "#65C5DB", 399 | "#F9934E", 400 | "#EA6460", 401 | "#5195CE", 402 | "#D683CE", 403 | "#806EB7", 404 | "#3F6833", 405 | "#967302", 406 | "#2F575E", 407 | "#99440A", 408 | "#58140C", 409 | "#052B51", 410 | "#511749", 411 | "#3F2B5B", 412 | "#E0F9D7", 413 | "#FCEACA", 414 | "#CFFAFF", 415 | "#F9E2D2", 416 | "#FCE2DE", 417 | "#BADFF4", 418 | "#F9D9F9", 419 | "#DEDAF7" 420 | ], 421 | "title": "GC Type" 422 | }, 423 | { 424 | "span": 8, 425 | "editable": true, 426 | "type": "rangeFacet", 427 | "loadingEditor": false, 428 | "mode": "count", 429 | "time_field": "timestamp", 430 | "queries": { 431 | "mode": "all", 432 | "ids": [ 433 | 0, 434 | 1 435 | ], 436 | "query": "q=s_gc_type%3AParNew&fq=event_category:garbage_collection&fq=event_type:pause&wt=json&rows=0&fq=date:[NOW/YEAR-5YEAR%20TO%20NOW/YEAR%2B1YEAR]&fq=i_duration:[654 TO 7508]&facet=true&facet.range=i_duration&facet.range.start=654&facet.range.end=7509&facet.range.gap=92\nq=s_gc_type%3AConcurrentMarkSweep&fq=event_category:garbage_collection&fq=event_type:pause&wt=json&rows=0&fq=date:[NOW/YEAR-5YEAR%20TO%20NOW/YEAR%2B1YEAR]&fq=i_duration:[654 TO 7508]&facet=true&facet.range=i_duration&facet.range.start=654&facet.range.end=7509&facet.range.gap=92\n", 437 | "custom": "" 438 | }, 439 | "max_rows": 100000, 440 | "value_field": null, 441 | "group_field": null, 442 | "fill": 0, 443 | "linewidth": 3, 444 | "auto_int": true, 445 | "resolution": 75, 446 | "interval": 92, 447 | "resolutions": [ 448 | 5, 449 | 10, 450 | 25, 451 | 50, 452 | 75, 453 | 100 454 | ], 455 | "spyable": true, 456 | "zoomlinks": true, 457 | "bars": true, 458 | "stack": true, 459 | "points": false, 460 | "lines": false, 461 | "lines_smooth": false, 462 | "legend": true, 463 | "x-axis": true, 464 | "y-axis": true, 465 | "percentage": false, 466 | "interactive": true, 467 | "options": true, 468 | "tooltip": { 469 | "value_type": "cumulative", 470 | "query_as_alias": false 471 | }, 472 | "showChart": true, 473 | "show_queries": true, 474 | "title": "Duration Histogram", 475 | "minimum": 654, 476 | "maximum": 7508, 477 | "range_field": "i_duration" 478 | } 479 | ] 480 | }, 481 | { 482 | "title": "Table", 483 | "height": "150px", 484 | "editable": true, 485 | "collapse": false, 486 | "collapsable": true, 487 | "panels": [ 488 | { 489 | "span": 12, 490 | "editable": true, 491 | "type": "table", 492 | "loadingEditor": false, 493 | "status": "Stable", 494 | "queries": { 495 | "mode": "all", 496 | "ids": [ 497 | 0, 498 | 1 499 | ], 500 | "query": "q=s_gc_type%3AParNew OR s_gc_type%3AConcurrentMarkSweep&fq=event_category:garbage_collection&fq=event_type:pause&fq=date:[NOW/YEAR-5YEAR%20TO%20NOW/YEAR%2B1YEAR]&fq=i_duration:[654 TO 7508]&sort=i_duration desc&wt=json&rows=500", 501 | "basic_query": "q=s_gc_type%3AParNew OR s_gc_type%3AConcurrentMarkSweep&fq=event_category:garbage_collection&fq=event_type:pause&fq=date:[NOW/YEAR-5YEAR%20TO%20NOW/YEAR%2B1YEAR]&fq=i_duration:[654 TO 7508]&sort=i_duration desc", 502 | "custom": "" 503 | }, 504 | "size": 100, 505 | "pages": 5, 506 | "offset": 0, 507 | "sort": [ 508 | "i_duration", 509 | "desc" 510 | ], 511 | "group": "default", 512 | "style": { 513 | "font-size": "9pt" 514 | }, 515 | "overflow": "min-height", 516 | "fields": [ 517 | "date", 518 | "s_gc_type", 519 | "i_duration", 520 | "i_eden_before", 521 | "i_eden_after", 522 | "i_oldgen_before", 523 | "i_oldgen_after" 524 | ], 525 | "highlight": [], 526 | "sortable": true, 527 | "header": true, 528 | "paging": true, 529 | "field_list": false, 530 | "trimFactor": 300, 531 | "normTimes": true, 532 | "spyable": true, 533 | "saveOption": "json", 534 | "exportSize": 500, 535 | "exportAll": true, 536 | "displayLinkIcon": true, 537 | "imageFields": [], 538 | "imgFieldWidth": "auto", 539 | "imgFieldHeight": "85px", 540 | "title": "Table panel", 541 | "important_fields": [ 542 | "_dynFld", 543 | "_partitionKey", 544 | "_token_long", 545 | "b_auto_determined", 546 | "b_enabled", 547 | "date", 548 | "event_category", 549 | "event_type", 550 | "exception", 551 | "f_file_size_kb", 552 | "f_just_counted", 553 | "f_live_ratio", 554 | "f_max_sane_ratio", 555 | "f_percent_full", 556 | "f_rate", 557 | "f_serialized_kb", 558 | "host", 559 | "i_active", 560 | "i_all_time_blocked", 561 | "i_async_writers", 562 | "i_batch_size", 563 | "i_batch_warn_threshold", 564 | "i_blocked", 565 | "i_bytes", 566 | "i_bytes_complete", 567 | "i_bytes_total", 568 | "i_cache_items", 569 | "i_cells", 570 | "i_client_port", 571 | "i_codecache_after", 572 | "i_codecache_before", 573 | "i_collections", 574 | "i_command", 575 | "i_completed", 576 | "i_compressed_class_after", 577 | "i_compressed_class_before", 578 | "i_data", 579 | "i_duration", 580 | "i_eden_after", 581 | "i_eden_before", 582 | "i_estimated_bytes", 583 | "i_file_size_bytes", 584 | "i_flushing_off_heap", 585 | "i_flushing_on_heap", 586 | "i_frame_size", 587 | "i_hash_code", 588 | "i_heap_used", 589 | "i_hints_delivered", 590 | "i_hints_dropped", 591 | "i_input_bytes", 592 | "i_instance", 593 | "i_listeners_notified", 594 | "i_live_bytes", 595 | "i_live_cells", 596 | "i_live_off_heap", 597 | "i_live_on_heap", 598 | "i_master_port", 599 | "i_max", 600 | "i_max_frame_size", 601 | "i_max_time", 602 | "i_memory_committed", 603 | "i_memory_init", 604 | "i_memory_max", 605 | "i_memory_used", 606 | "i_messages_dropped", 607 | "i_metaspace_after", 608 | "i_metaspace_before", 609 | "i_off_heap_bytes", 610 | "i_off_heap_limit", 611 | "i_oldgen_after", 612 | "i_oldgen_before", 613 | "i_on_heap_bytes", 614 | "i_on_heap_limit", 615 | "i_ops", 616 | "i_output_bytes", 617 | "i_partition_size", 618 | "i_pending", 619 | "i_pending_responses", 620 | "i_percent_of_original", 621 | "i_permgen_after", 622 | "i_permgen_before", 623 | "i_port", 624 | "i_position", 625 | "i_processing_time", 626 | "i_qtime", 627 | "i_queue_size", 628 | "i_range_begin", 629 | "i_range_end", 630 | "i_ranges", 631 | "i_receiving_bytes", 632 | "i_receiving_files", 633 | "i_ref_count", 634 | "i_replayed_mutations", 635 | "i_requested_columns", 636 | "i_rows", 637 | "i_save_duration", 638 | "i_save_interval", 639 | "i_segment_id", 640 | "i_sending_bytes", 641 | "i_sending_files", 642 | "i_serialized_bytes", 643 | "i_server_port", 644 | "i_sstable_count", 645 | "i_status", 646 | "i_survivor_after", 647 | "i_survivor_before", 648 | "i_target_queue_size", 649 | "i_this_off_heap", 650 | "i_this_on_heap", 651 | "i_thrift_port", 652 | "i_tombstoned_cells", 653 | "i_total_heap", 654 | "i_total_partitions", 655 | "i_ttl", 656 | "i_unique_partitions", 657 | "i_used", 658 | "i_used_off_heap", 659 | "i_used_on_heap", 660 | "i_work_requests", 661 | "id", 662 | "l_classpath", 663 | "l_column_definition", 664 | "l_commitlog_file", 665 | "l_data_file_directories", 666 | "l_deletion_info", 667 | "l_input_sstables", 668 | "l_metadata", 669 | "l_netty_version", 670 | "l_nodes", 671 | "l_params", 672 | "l_partition_merge_counts", 673 | "l_sstables", 674 | "l_supported_versions", 675 | "l_tables", 676 | "l_tokens", 677 | "level", 678 | "log_file", 679 | "log_line", 680 | "message", 681 | "s_action", 682 | "s_cache_capacity_mb", 683 | "s_cache_file", 684 | "s_cache_provider", 685 | "s_cache_type", 686 | "s_channel_id", 687 | "s_client_host", 688 | "s_commit_log_directory", 689 | "s_commitlog_file", 690 | "s_compaction_id", 691 | "s_compaction_type", 692 | "s_component", 693 | "s_consistency_level", 694 | "s_count", 695 | "s_dc_army", 696 | "s_default_field", 697 | "s_default_version", 698 | "s_directory", 699 | "s_directory_count", 700 | "s_directory_factory", 701 | "s_disk_access_mode", 702 | "s_done", 703 | "s_endpoint", 704 | "s_endpoints", 705 | "s_error", 706 | "s_event_loop_type", 707 | "s_exception_thread", 708 | "s_feature", 709 | "s_field", 710 | "s_file_size_mb", 711 | "s_filename", 712 | "s_forwarded_endpoint", 713 | "s_gc_type", 714 | "s_hash", 715 | "s_host_id", 716 | "s_hostname", 717 | "s_ignored_class", 718 | "s_index_access_mode", 719 | "s_index_class", 720 | "s_index_directory", 721 | "s_index_hash", 722 | "s_index_metadata", 723 | "s_index_type", 724 | "s_jar", 725 | "s_jetty_version", 726 | "s_jvm", 727 | "s_keys_loaded", 728 | "s_keys_to_save", 729 | "s_keyspace", 730 | "s_leader_manager", 731 | "s_lifetime_seconds", 732 | "s_listener", 733 | "s_live_ratio_estimate", 734 | "s_loader_type", 735 | "s_master_host", 736 | "s_matchversion", 737 | "s_mbean_class", 738 | "s_mbean_hash", 739 | "s_mbean_operation", 740 | "s_memory_type", 741 | "s_memtable_location", 742 | "s_memtable_threshold", 743 | "s_message_type", 744 | "s_metadata", 745 | "s_metric", 746 | "s_native_host", 747 | "s_native_port", 748 | "s_new_hash", 749 | "s_new_leader", 750 | "s_new_metadata", 751 | "s_node1", 752 | "s_node2", 753 | "s_old_hash", 754 | "s_old_leader", 755 | "s_old_metadata", 756 | "s_output_sstable", 757 | "s_partition_key", 758 | "s_path", 759 | "s_plugin_class", 760 | "s_policy", 761 | "s_pool_name", 762 | "s_process", 763 | "s_properties", 764 | "s_range_begin", 765 | "s_range_end", 766 | "s_reason", 767 | "s_resource", 768 | "s_schema_file", 769 | "s_schema_name", 770 | "s_serialized_mb", 771 | "s_server_host", 772 | "s_session_id", 773 | "s_settings_authenticator", 774 | "s_settings_authorizer", 775 | "s_settings_auto_bootstrap", 776 | "s_settings_auto_snapshot", 777 | "s_settings_batch_size_warn_threshold_in_kb", 778 | "s_settings_batchlog_replay_throttle_in_kb", 779 | "s_settings_broadcast_address", 780 | "s_settings_broadcast_rpc_address", 781 | "s_settings_cas_contention_timeout_in_ms", 782 | "s_settings_client_encryption_options", 783 | "s_settings_cluster_name", 784 | "s_settings_column_index_size_in_kb", 785 | "s_settings_commit_failure_policy", 786 | "s_settings_commitlog_directory", 787 | "s_settings_commitlog_segment_size_in_mb", 788 | "s_settings_commitlog_sync", 789 | "s_settings_commitlog_sync_period_in_ms", 790 | "s_settings_compaction_preheat_key_cache", 791 | "s_settings_compaction_throughput_mb_per_sec", 792 | "s_settings_concurrent_compactors", 793 | "s_settings_concurrent_counter_writes", 794 | "s_settings_concurrent_reads", 795 | "s_settings_concurrent_writes", 796 | "s_settings_counter_cache_save_period", 797 | "s_settings_counter_cache_size_in_mb", 798 | "s_settings_counter_write_request_timeout_in_ms", 799 | "s_settings_cross_node_timeout", 800 | "s_settings_data_file_directories", 801 | "s_settings_disk_failure_policy", 802 | "s_settings_dynamic_snitch_badness_threshold", 803 | "s_settings_dynamic_snitch_reset_interval_in_ms", 804 | "s_settings_dynamic_snitch_update_interval_in_ms", 805 | "s_settings_endpoint_snitch", 806 | "s_settings_hinted_handoff_enabled", 807 | "s_settings_hinted_handoff_throttle_in_kb", 808 | "s_settings_in_memory_compaction_limit_in_mb", 809 | "s_settings_incremental_backups", 810 | "s_settings_index_summary_capacity_in_mb", 811 | "s_settings_index_summary_resize_interval_in_minutes", 812 | "s_settings_initial_token", 813 | "s_settings_inter_dc_stream_throughput_outbound_megabits_per_sec", 814 | "s_settings_inter_dc_tcp_nodelay", 815 | "s_settings_internode_authenticator", 816 | "s_settings_internode_compression", 817 | "s_settings_key_cache_save_period", 818 | "s_settings_key_cache_size_in_mb", 819 | "s_settings_listen_address", 820 | "s_settings_max_hint_window_in_ms", 821 | "s_settings_max_hints_delivery_threads", 822 | "s_settings_memory_allocator", 823 | "s_settings_memtable_allocation_type", 824 | "s_settings_memtable_flush_queue_size", 825 | "s_settings_multithreaded_compaction", 826 | "s_settings_native_transport_max_frame_size_in_mb", 827 | "s_settings_native_transport_max_threads", 828 | "s_settings_native_transport_port", 829 | "s_settings_num_tokens", 830 | "s_settings_partitioner", 831 | "s_settings_permissions_validity_in_ms", 832 | "s_settings_preheat_kernel_page_cache", 833 | "s_settings_range_request_timeout_in_ms", 834 | "s_settings_read_request_timeout_in_ms", 835 | "s_settings_request_scheduler", 836 | "s_settings_request_timeout_in_ms", 837 | "s_settings_row_cache_save_period", 838 | "s_settings_row_cache_size_in_mb", 839 | "s_settings_rpc_address", 840 | "s_settings_rpc_keepalive", 841 | "s_settings_rpc_port", 842 | "s_settings_rpc_server_type", 843 | "s_settings_saved_caches_directory", 844 | "s_settings_seed_provider", 845 | "s_settings_server_encryption_options", 846 | "s_settings_snapshot_before_compaction", 847 | "s_settings_ssl_storage_port", 848 | "s_settings_sstable_preemptive_open_interval_in_mb", 849 | "s_settings_start_native_transport", 850 | "s_settings_start_rpc", 851 | "s_settings_storage_port", 852 | "s_settings_stream_throughput_outbound_megabits_per_sec", 853 | "s_settings_thrift_framed_transport_size_in_mb", 854 | "s_settings_tombstone_failure_threshold", 855 | "s_settings_tombstone_warn_threshold", 856 | "s_settings_trickle_fsync", 857 | "s_settings_trickle_fsync_interval_in_kb", 858 | "s_settings_truncate_request_timeout_in_ms", 859 | "s_settings_write_request_timeout_in_ms", 860 | "s_signal", 861 | "s_slice_end", 862 | "s_slice_start", 863 | "s_solr_home", 864 | "s_solrconfig", 865 | "s_source_endpoint", 866 | "s_sstable_name", 867 | "s_state", 868 | "s_table", 869 | "s_target_endpoint", 870 | "s_threshold_exceeded_by", 871 | "s_thrift_host", 872 | "s_type", 873 | "s_unique_key_field", 874 | "s_user_dir", 875 | "s_version", 876 | "s_webapp", 877 | "s_work_pool", 878 | "s_yaml_file", 879 | "source_file", 880 | "source_line", 881 | "thread_id", 882 | "thread_name" 883 | ], 884 | "show_queries": true 885 | } 886 | ] 887 | } 888 | ], 889 | "editable": true, 890 | "index": { 891 | "interval": "none", 892 | "pattern": "[logstash-]YYYY.MM.DD", 893 | "default": "_all" 894 | }, 895 | "style": "light", 896 | "failover": false, 897 | "panel_hints": true, 898 | "loader": { 899 | "save_gist": false, 900 | "save_elasticsearch": true, 901 | "save_local": true, 902 | "save_default": true, 903 | "save_temp": true, 904 | "save_temp_ttl_enable": true, 905 | "save_temp_ttl": "30d", 906 | "load_gist": true, 907 | "load_elasticsearch": true, 908 | "load_elasticsearch_size": 20, 909 | "load_local": true, 910 | "hide": false, 911 | "dropdown_collections": false 912 | }, 913 | "solr": { 914 | "server": "/solr/", 915 | "core_name": "logparse.systemlog", 916 | "core_list": [ 917 | "banana.int", 918 | "logparse.systemlog", 919 | "zendesk.organizations", 920 | "zendesk.ticket_comments", 921 | "zendesk.ticket_events", 922 | "zendesk.tickets", 923 | "zendesk.users" 924 | ], 925 | "global_params": "&fq=event_category:garbage_collection&fq=event_type:pause" 926 | } 927 | } -------------------------------------------------------------------------------- /banana/overview.json: -------------------------------------------------------------------------------- 1 | { 2 | "title": "Cassandra System.log Overview", 3 | "services": { 4 | "query": { 5 | "idQueue": [ 6 | 4 7 | ], 8 | "list": { 9 | "0": { 10 | "query": "level:ERROR", 11 | "alias": "", 12 | "color": "#E24D42", 13 | "id": 0, 14 | "pin": false, 15 | "type": "lucene" 16 | }, 17 | "1": { 18 | "id": 1, 19 | "color": "#EAB839", 20 | "query": "level:WARN", 21 | "alias": "", 22 | "pin": false 23 | }, 24 | "2": { 25 | "id": 2, 26 | "color": "#7EB26D", 27 | "query": "level:INFO", 28 | "alias": "", 29 | "pin": false 30 | }, 31 | "3": { 32 | "id": 3, 33 | "color": "#6ED0E0", 34 | "query": "level:DEBUG", 35 | "alias": "", 36 | "pin": false 37 | } 38 | }, 39 | "ids": [ 40 | 0, 41 | 1, 42 | 2, 43 | 3 44 | ] 45 | }, 46 | "filter": { 47 | "idQueue": [ 48 | 2, 49 | 3, 50 | 4, 51 | 5, 52 | 6 53 | ], 54 | "list": { 55 | "0": { 56 | "from": "NOW/YEAR-5YEAR", 57 | "to": "NOW/YEAR%2B1YEAR", 58 | "field": "date", 59 | "type": "time", 60 | "fromDateObj": "2010-07-27T21:34:48.566Z", 61 | "toDateObj": "2015-07-27T21:34:48.567Z", 62 | "mandate": "must", 63 | "active": true, 64 | "alias": "", 65 | "id": 0 66 | }, 67 | "1": { 68 | "type": "time", 69 | "from": "2015-05-20T11:39:57.789Z", 70 | "to": "2015-07-27T21:34:48.567Z", 71 | "field": "date", 72 | "mandate": "must", 73 | "active": true, 74 | "alias": "", 75 | "id": 1 76 | } 77 | }, 78 | "ids": [ 79 | 0, 80 | 1 81 | ] 82 | } 83 | }, 84 | "rows": [ 85 | { 86 | "title": "Help", 87 | "height": "150px", 88 | "editable": true, 89 | "collapse": true, 90 | "collapsable": true, 91 | "panels": [ 92 | { 93 | "error": false, 94 | "span": 12, 95 | "editable": true, 96 | "type": "text", 97 | "loadingEditor": false, 98 | "status": "Stable", 99 | "mode": "markdown", 100 | "content": "### Cassandra System.log Overview\n\nThis dashboard gives an overview of the events in your system.log, broken down in various ways and plotted over time.\n\n#### Search Parameters\n\nIn the first row, you can select the time window and the search terms. Terms have already been entered to color code the events based on severity level. NB, the colors assigned here only apply to the event count histogram. Colors in the bar charts are assigned separately and similar colors may be used. The third panel shows how many results have been found.\n\n#### Filtering\n\nThe second row allows you to control the filters applied to the dashboard. The left panel shows the filters that have already been applied to the dashboard. To remove a previously applied filter, click the x on the box representing the filter. The right panel allows you to choose a specific log file or host. The number of events in each log and for each host are shown, allowing you to zero in on trouble spots.\n\n#### Event Counts\n\nThe third row is a time-series histogram that shows the number of events that occurred within a specific time window. You can use this to identify active periods in the log. This view also allows you to drag over a specific time period to zoom in on that period. All the other graphs will adjust to show only events within that time period.\n\n#### Breakdown Charts\n\nThe next two rows show bar charts breaking down the severity level, source file, thread, source file, event type, and event category of each event. You can click on any bar to filter for that value. \n\n#### Data Table\n\nFinally, the table at the bottom shows the level, date, thread, and message for each event. You can click on a row to view full details about the event. \n\nEach event has the following fields:\n\n- id: a time-based UUID uniquely identifying the event\n- log_file: the log file that the events are from\n- log_line: the line number that the event occurs on in the log\n- host: the host that the events are from\n- level: the severity of the event (From most to least severe: ERROR, WARN, INFO, DEBUG)\n- date: the date and time the event occurred\n- thread: the thread that generated the event\n- source_file: the source file that logged the event\n- source_line: the line of code where the event was logged\n- message: the detailed message logged by the event\n- exception: an optional exception that triggered the event\n\nDynamic fields specific to a particular type of event are prefixed with a letter indicating their type:\n\n- s_ for string\n- i_ for integer (64-bit)\n- f_ for floating point (64-bit)\n- d_ for date\n- b_ for boolean\n- l_ for a newline-delimited list of strings\n\nYou can also click on a heading to sort by a specific field. By default the table is sorted by date in descending order.", 101 | "style": {}, 102 | "title": "Help" 103 | } 104 | ] 105 | }, 106 | { 107 | "title": "Query and Time Window", 108 | "height": "50px", 109 | "editable": true, 110 | "collapse": false, 111 | "collapsable": true, 112 | "panels": [ 113 | { 114 | "error": "", 115 | "span": 6, 116 | "editable": true, 117 | "type": "timepicker", 118 | "loadingEditor": false, 119 | "status": "Stable", 120 | "mode": "absolute", 121 | "time_options": [ 122 | "5m", 123 | "15m", 124 | "1h", 125 | "6h", 126 | "12h", 127 | "24h", 128 | "7d", 129 | "30d", 130 | "90d", 131 | "1y", 132 | "5y" 133 | ], 134 | "timespan": "5y", 135 | "timefield": "date", 136 | "timeformat": "", 137 | "refresh": { 138 | "enable": false, 139 | "interval": 30, 140 | "min": 3 141 | }, 142 | "filter_id": 0, 143 | "spyable": true, 144 | "title": "Time Window", 145 | "time": { 146 | "from": "05/20/2015 07:39:57", 147 | "to": "07/27/2015 17:34:48" 148 | } 149 | }, 150 | { 151 | "error": false, 152 | "span": 4, 153 | "editable": true, 154 | "group": [ 155 | "default" 156 | ], 157 | "type": "query", 158 | "label": "Search", 159 | "history": [ 160 | "level:DEBUG", 161 | "level:INFO", 162 | "level:WARN", 163 | "level:ERROR", 164 | "*:*", 165 | "event_type:unknown", 166 | "message:\"Load of settings\"", 167 | "event_type:repair*", 168 | "message:SparkWorker*", 169 | "message:SparkMaster*" 170 | ], 171 | "remember": 10, 172 | "pinned": true, 173 | "query": "*:*", 174 | "title": "Search", 175 | "spyable": true, 176 | "def_type": "" 177 | }, 178 | { 179 | "span": 2, 180 | "editable": true, 181 | "type": "hits", 182 | "loadingEditor": false, 183 | "queries": { 184 | "mode": "all", 185 | "ids": [ 186 | 0, 187 | 1, 188 | 2, 189 | 3 190 | ], 191 | "query": "q=level%3AERROR&fq=date:[2015-05-20T11:39:57.789Z%20TO%202015-07-27T21:34:48.567Z]&wt=json&rows=0\nq=level%3AWARN&fq=date:[2015-05-20T11:39:57.789Z%20TO%202015-07-27T21:34:48.567Z]&wt=json&rows=0\nq=level%3AINFO&fq=date:[2015-05-20T11:39:57.789Z%20TO%202015-07-27T21:34:48.567Z]&wt=json&rows=0\nq=level%3ADEBUG&fq=date:[2015-05-20T11:39:57.789Z%20TO%202015-07-27T21:34:48.567Z]&wt=json&rows=0\n", 192 | "basic_query": "", 193 | "custom": "" 194 | }, 195 | "style": { 196 | "font-size": "14pt" 197 | }, 198 | "arrangement": "horizontal", 199 | "chart": "total", 200 | "counter_pos": "above", 201 | "donut": false, 202 | "tilt": false, 203 | "labels": true, 204 | "spyable": true, 205 | "title": "Total Hits", 206 | "show_queries": true 207 | } 208 | ] 209 | }, 210 | { 211 | "title": "Filters", 212 | "height": "50px", 213 | "editable": true, 214 | "collapse": true, 215 | "collapsable": true, 216 | "panels": [ 217 | { 218 | "error": false, 219 | "span": 8, 220 | "editable": true, 221 | "spyable": true, 222 | "group": [ 223 | "default" 224 | ], 225 | "type": "filtering" 226 | }, 227 | { 228 | "span": 4, 229 | "editable": true, 230 | "type": "facet", 231 | "loadingEditor": false, 232 | "status": "Stable", 233 | "queries": { 234 | "mode": "all", 235 | "ids": [ 236 | 0, 237 | 1, 238 | 2, 239 | 3 240 | ], 241 | "query": "q=level%3AERROR OR level%3AWARN OR level%3AINFO OR level%3ADEBUG&fq=date:[2015-05-20T11:39:57.789Z%20TO%202015-07-27T21:34:48.567Z]&facet=true&facet.field=log_file&facet.field=host&wt=json", 242 | "basic_query": "q=level%3AERROR OR level%3AWARN OR level%3AINFO OR level%3ADEBUG&fq=date:[2015-05-20T11:39:57.789Z%20TO%202015-07-27T21:34:48.567Z]&facet=true&facet.field=log_file&facet.field=host", 243 | "custom": "" 244 | }, 245 | "group": "default", 246 | "style": { 247 | "font-size": "9pt" 248 | }, 249 | "overflow": "min-height", 250 | "fields": [ 251 | "log_file", 252 | "host" 253 | ], 254 | "field_list": true, 255 | "spyable": true, 256 | "facet_limit": 10, 257 | "foundResults": true, 258 | "header_title": "Limit Your Search", 259 | "toggle_element": null, 260 | "show_queries": true, 261 | "title": "Facets", 262 | "offset": 0, 263 | "exportSize": null 264 | } 265 | ] 266 | }, 267 | { 268 | "title": "Graph", 269 | "height": "250px", 270 | "editable": true, 271 | "collapse": false, 272 | "collapsable": true, 273 | "panels": [ 274 | { 275 | "span": 12, 276 | "editable": true, 277 | "type": "histogram", 278 | "loadingEditor": false, 279 | "mode": "count", 280 | "time_field": "event_timestamp", 281 | "queries": { 282 | "mode": "all", 283 | "ids": [ 284 | 0, 285 | 1, 286 | 2, 287 | 3 288 | ], 289 | "query": "q=level%3AERROR&wt=json&rows=0&fq=date:[2015-05-20T11:39:57.789Z%20TO%202015-07-27T21:34:48.567Z]&facet=true&facet.range=date&facet.range.start=2015-05-20T11:39:57.789Z&facet.range.end=2015-07-27T21:34:48.567Z&facet.range.gap=%2B12HOUR\nq=level%3AWARN&wt=json&rows=0&fq=date:[2015-05-20T11:39:57.789Z%20TO%202015-07-27T21:34:48.567Z]&facet=true&facet.range=date&facet.range.start=2015-05-20T11:39:57.789Z&facet.range.end=2015-07-27T21:34:48.567Z&facet.range.gap=%2B12HOUR\nq=level%3AINFO&wt=json&rows=0&fq=date:[2015-05-20T11:39:57.789Z%20TO%202015-07-27T21:34:48.567Z]&facet=true&facet.range=date&facet.range.start=2015-05-20T11:39:57.789Z&facet.range.end=2015-07-27T21:34:48.567Z&facet.range.gap=%2B12HOUR\nq=level%3ADEBUG&wt=json&rows=0&fq=date:[2015-05-20T11:39:57.789Z%20TO%202015-07-27T21:34:48.567Z]&facet=true&facet.range=date&facet.range.start=2015-05-20T11:39:57.789Z&facet.range.end=2015-07-27T21:34:48.567Z&facet.range.gap=%2B12HOUR\n", 290 | "custom": "" 291 | }, 292 | "max_rows": 100000, 293 | "value_field": null, 294 | "group_field": null, 295 | "auto_int": true, 296 | "resolution": 100, 297 | "interval": "12h", 298 | "intervals": [ 299 | "auto", 300 | "1s", 301 | "1m", 302 | "5m", 303 | "10m", 304 | "30m", 305 | "1h", 306 | "3h", 307 | "12h", 308 | "1d", 309 | "1w", 310 | "1M", 311 | "1y" 312 | ], 313 | "fill": 0, 314 | "linewidth": 3, 315 | "timezone": "browser", 316 | "spyable": true, 317 | "zoomlinks": true, 318 | "bars": true, 319 | "stack": true, 320 | "points": false, 321 | "lines": false, 322 | "legend": true, 323 | "x-axis": true, 324 | "y-axis": true, 325 | "percentage": false, 326 | "interactive": true, 327 | "options": true, 328 | "tooltip": { 329 | "value_type": "cumulative", 330 | "query_as_alias": false 331 | }, 332 | "title": "Event Counts", 333 | "lines_smooth": false, 334 | "show_queries": true 335 | } 336 | ] 337 | }, 338 | { 339 | "title": "Breakdown", 340 | "height": "300px", 341 | "editable": true, 342 | "collapse": false, 343 | "collapsable": true, 344 | "panels": [ 345 | { 346 | "span": 4, 347 | "editable": true, 348 | "type": "terms", 349 | "loadingEditor": false, 350 | "queries": { 351 | "mode": "all", 352 | "ids": [ 353 | 0, 354 | 1, 355 | 2, 356 | 3 357 | ], 358 | "query": "q=level%3AERROR OR level%3AWARN OR level%3AINFO OR level%3ADEBUG&wt=json&rows=0&fq=date:[2015-05-20T11:39:57.789Z%20TO%202015-07-27T21:34:48.567Z]&facet=true&facet.field=level&facet.limit=10&facet.missing=true", 359 | "custom": "" 360 | }, 361 | "mode": "count", 362 | "field": "level", 363 | "stats_field": "", 364 | "decimal_points": 0, 365 | "exclude": [], 366 | "missing": false, 367 | "other": false, 368 | "size": 10, 369 | "order": "descending", 370 | "style": { 371 | "font-size": "10pt" 372 | }, 373 | "donut": false, 374 | "tilt": false, 375 | "labels": true, 376 | "logAxis": false, 377 | "arrangement": "horizontal", 378 | "chart": "bar", 379 | "counter_pos": "below", 380 | "lastColor": "rgb(234,184,57)", 381 | "spyable": true, 382 | "show_queries": true, 383 | "chartColors": [ 384 | "#7EB26D", 385 | "#EAB839", 386 | "#6ED0E0", 387 | "#EF843C", 388 | "#E24D42", 389 | "#1F78C1", 390 | "#BA43A9", 391 | "#705DA0", 392 | "#508642", 393 | "#CCA300", 394 | "#447EBC", 395 | "#C15C17", 396 | "#890F02", 397 | "#0A437C", 398 | "#6D1F62", 399 | "#584477", 400 | "#B7DBAB", 401 | "#F4D598", 402 | "#70DBED", 403 | "#F9BA8F", 404 | "#F29191", 405 | "#82B5D8", 406 | "#E5A8E2", 407 | "#AEA2E0", 408 | "#629E51", 409 | "#E5AC0E", 410 | "#64B0C8", 411 | "#E0752D", 412 | "#BF1B00", 413 | "#0A50A1", 414 | "#962D82", 415 | "#614D93", 416 | "#9AC48A", 417 | "#F2C96D", 418 | "#65C5DB", 419 | "#F9934E", 420 | "#EA6460", 421 | "#5195CE", 422 | "#D683CE", 423 | "#806EB7", 424 | "#3F6833", 425 | "#967302", 426 | "#2F575E", 427 | "#99440A", 428 | "#58140C", 429 | "#052B51", 430 | "#511749", 431 | "#3F2B5B", 432 | "#E0F9D7", 433 | "#FCEACA", 434 | "#CFFAFF", 435 | "#F9E2D2", 436 | "#FCE2DE", 437 | "#BADFF4", 438 | "#F9D9F9", 439 | "#DEDAF7" 440 | ], 441 | "title": "Level" 442 | }, 443 | { 444 | "span": 4, 445 | "editable": true, 446 | "type": "terms", 447 | "loadingEditor": false, 448 | "queries": { 449 | "mode": "all", 450 | "ids": [ 451 | 0, 452 | 1, 453 | 2, 454 | 3 455 | ], 456 | "query": "q=level%3AERROR OR level%3AWARN OR level%3AINFO OR level%3ADEBUG&wt=json&rows=0&fq=date:[2015-05-20T11:39:57.789Z%20TO%202015-07-27T21:34:48.567Z]&facet=true&facet.field=source_file&facet.limit=20&facet.missing=true", 457 | "custom": "" 458 | }, 459 | "mode": "count", 460 | "field": "source_file", 461 | "stats_field": "", 462 | "decimal_points": 0, 463 | "exclude": [], 464 | "missing": false, 465 | "other": false, 466 | "size": 20, 467 | "order": "descending", 468 | "style": { 469 | "font-size": "10pt" 470 | }, 471 | "donut": false, 472 | "tilt": false, 473 | "labels": true, 474 | "logAxis": false, 475 | "arrangement": "horizontal", 476 | "chart": "bar", 477 | "counter_pos": "below", 478 | "lastColor": "rgb(112,93,160)", 479 | "spyable": true, 480 | "show_queries": true, 481 | "chartColors": [ 482 | "#7EB26D", 483 | "#EAB839", 484 | "#6ED0E0", 485 | "#EF843C", 486 | "#E24D42", 487 | "#1F78C1", 488 | "#BA43A9", 489 | "#705DA0", 490 | "#508642", 491 | "#CCA300", 492 | "#447EBC", 493 | "#C15C17", 494 | "#890F02", 495 | "#0A437C", 496 | "#6D1F62", 497 | "#584477", 498 | "#B7DBAB", 499 | "#F4D598", 500 | "#70DBED", 501 | "#F9BA8F", 502 | "#F29191", 503 | "#82B5D8", 504 | "#E5A8E2", 505 | "#AEA2E0", 506 | "#629E51", 507 | "#E5AC0E", 508 | "#64B0C8", 509 | "#E0752D", 510 | "#BF1B00", 511 | "#0A50A1", 512 | "#962D82", 513 | "#614D93", 514 | "#9AC48A", 515 | "#F2C96D", 516 | "#65C5DB", 517 | "#F9934E", 518 | "#EA6460", 519 | "#5195CE", 520 | "#D683CE", 521 | "#806EB7", 522 | "#3F6833", 523 | "#967302", 524 | "#2F575E", 525 | "#99440A", 526 | "#58140C", 527 | "#052B51", 528 | "#511749", 529 | "#3F2B5B", 530 | "#E0F9D7", 531 | "#FCEACA", 532 | "#CFFAFF", 533 | "#F9E2D2", 534 | "#FCE2DE", 535 | "#BADFF4", 536 | "#F9D9F9", 537 | "#DEDAF7" 538 | ], 539 | "title": "Source File" 540 | }, 541 | { 542 | "span": 4, 543 | "editable": true, 544 | "type": "terms", 545 | "loadingEditor": false, 546 | "queries": { 547 | "mode": "all", 548 | "ids": [ 549 | 0, 550 | 1, 551 | 2, 552 | 3 553 | ], 554 | "query": "q=level%3AERROR OR level%3AWARN OR level%3AINFO OR level%3ADEBUG&wt=json&rows=0&fq=date:[2015-05-20T11:39:57.789Z%20TO%202015-07-27T21:34:48.567Z]&facet=true&facet.field=thread_name&facet.limit=20&facet.missing=true", 555 | "custom": "" 556 | }, 557 | "mode": "count", 558 | "field": "thread_name", 559 | "stats_field": "", 560 | "decimal_points": 0, 561 | "exclude": [], 562 | "missing": false, 563 | "other": false, 564 | "size": 20, 565 | "order": "descending", 566 | "style": { 567 | "font-size": "10pt" 568 | }, 569 | "donut": false, 570 | "tilt": false, 571 | "labels": true, 572 | "logAxis": false, 573 | "arrangement": "horizontal", 574 | "chart": "bar", 575 | "counter_pos": "below", 576 | "lastColor": "rgb(234,184,57)", 577 | "spyable": true, 578 | "show_queries": true, 579 | "chartColors": [ 580 | "#7EB26D", 581 | "#EAB839", 582 | "#6ED0E0", 583 | "#EF843C", 584 | "#E24D42", 585 | "#1F78C1", 586 | "#BA43A9", 587 | "#705DA0", 588 | "#508642", 589 | "#CCA300", 590 | "#447EBC", 591 | "#C15C17", 592 | "#890F02", 593 | "#0A437C", 594 | "#6D1F62", 595 | "#584477", 596 | "#B7DBAB", 597 | "#F4D598", 598 | "#70DBED", 599 | "#F9BA8F", 600 | "#F29191", 601 | "#82B5D8", 602 | "#E5A8E2", 603 | "#AEA2E0", 604 | "#629E51", 605 | "#E5AC0E", 606 | "#64B0C8", 607 | "#E0752D", 608 | "#BF1B00", 609 | "#0A50A1", 610 | "#962D82", 611 | "#614D93", 612 | "#9AC48A", 613 | "#F2C96D", 614 | "#65C5DB", 615 | "#F9934E", 616 | "#EA6460", 617 | "#5195CE", 618 | "#D683CE", 619 | "#806EB7", 620 | "#3F6833", 621 | "#967302", 622 | "#2F575E", 623 | "#99440A", 624 | "#58140C", 625 | "#052B51", 626 | "#511749", 627 | "#3F2B5B", 628 | "#E0F9D7", 629 | "#FCEACA", 630 | "#CFFAFF", 631 | "#F9E2D2", 632 | "#FCE2DE", 633 | "#BADFF4", 634 | "#F9D9F9", 635 | "#DEDAF7" 636 | ], 637 | "title": "Thread" 638 | } 639 | ] 640 | }, 641 | { 642 | "title": "Even", 643 | "height": "300px", 644 | "editable": true, 645 | "collapse": false, 646 | "collapsable": true, 647 | "panels": [ 648 | { 649 | "span": 4, 650 | "editable": true, 651 | "type": "terms", 652 | "loadingEditor": false, 653 | "queries": { 654 | "mode": "all", 655 | "ids": [ 656 | 0, 657 | 1, 658 | 2, 659 | 3 660 | ], 661 | "query": "q=level%3AERROR OR level%3AWARN OR level%3AINFO OR level%3ADEBUG&wt=json&rows=0&fq=date:[2015-05-20T11:39:57.789Z%20TO%202015-07-27T21:34:48.567Z]&facet=true&facet.field=log_file&facet.limit=10&facet.missing=true", 662 | "custom": "" 663 | }, 664 | "mode": "count", 665 | "field": "log_file", 666 | "stats_field": "", 667 | "decimal_points": 0, 668 | "exclude": [], 669 | "missing": false, 670 | "other": false, 671 | "size": 10, 672 | "order": "descending", 673 | "style": { 674 | "font-size": "10pt" 675 | }, 676 | "donut": false, 677 | "tilt": false, 678 | "labels": true, 679 | "logAxis": false, 680 | "arrangement": "horizontal", 681 | "chart": "bar", 682 | "counter_pos": "below", 683 | "lastColor": "", 684 | "spyable": true, 685 | "show_queries": true, 686 | "chartColors": [ 687 | "#7EB26D", 688 | "#EAB839", 689 | "#6ED0E0", 690 | "#EF843C", 691 | "#E24D42", 692 | "#1F78C1", 693 | "#BA43A9", 694 | "#705DA0", 695 | "#508642", 696 | "#CCA300", 697 | "#447EBC", 698 | "#C15C17", 699 | "#890F02", 700 | "#0A437C", 701 | "#6D1F62", 702 | "#584477", 703 | "#B7DBAB", 704 | "#F4D598", 705 | "#70DBED", 706 | "#F9BA8F", 707 | "#F29191", 708 | "#82B5D8", 709 | "#E5A8E2", 710 | "#AEA2E0", 711 | "#629E51", 712 | "#E5AC0E", 713 | "#64B0C8", 714 | "#E0752D", 715 | "#BF1B00", 716 | "#0A50A1", 717 | "#962D82", 718 | "#614D93", 719 | "#9AC48A", 720 | "#F2C96D", 721 | "#65C5DB", 722 | "#F9934E", 723 | "#EA6460", 724 | "#5195CE", 725 | "#D683CE", 726 | "#806EB7", 727 | "#3F6833", 728 | "#967302", 729 | "#2F575E", 730 | "#99440A", 731 | "#58140C", 732 | "#052B51", 733 | "#511749", 734 | "#3F2B5B", 735 | "#E0F9D7", 736 | "#FCEACA", 737 | "#CFFAFF", 738 | "#F9E2D2", 739 | "#FCE2DE", 740 | "#BADFF4", 741 | "#F9D9F9", 742 | "#DEDAF7" 743 | ], 744 | "title": "File" 745 | }, 746 | { 747 | "span": 4, 748 | "editable": true, 749 | "type": "terms", 750 | "loadingEditor": false, 751 | "queries": { 752 | "mode": "all", 753 | "ids": [ 754 | 0, 755 | 1, 756 | 2, 757 | 3 758 | ], 759 | "query": "q=level%3AERROR OR level%3AWARN OR level%3AINFO OR level%3ADEBUG&wt=json&rows=0&fq=date:[2015-05-20T11:39:57.789Z%20TO%202015-07-27T21:34:48.567Z]&facet=true&facet.field=event_type&facet.limit=20&facet.missing=true", 760 | "custom": "" 761 | }, 762 | "mode": "count", 763 | "field": "event_type", 764 | "stats_field": "", 765 | "decimal_points": 0, 766 | "exclude": [], 767 | "missing": false, 768 | "other": false, 769 | "size": 20, 770 | "order": "descending", 771 | "style": { 772 | "font-size": "10pt" 773 | }, 774 | "donut": false, 775 | "tilt": false, 776 | "labels": true, 777 | "logAxis": false, 778 | "arrangement": "horizontal", 779 | "chart": "bar", 780 | "counter_pos": "below", 781 | "lastColor": "rgb(239,132,60)", 782 | "spyable": true, 783 | "show_queries": true, 784 | "chartColors": [ 785 | "#7EB26D", 786 | "#EAB839", 787 | "#6ED0E0", 788 | "#EF843C", 789 | "#E24D42", 790 | "#1F78C1", 791 | "#BA43A9", 792 | "#705DA0", 793 | "#508642", 794 | "#CCA300", 795 | "#447EBC", 796 | "#C15C17", 797 | "#890F02", 798 | "#0A437C", 799 | "#6D1F62", 800 | "#584477", 801 | "#B7DBAB", 802 | "#F4D598", 803 | "#70DBED", 804 | "#F9BA8F", 805 | "#F29191", 806 | "#82B5D8", 807 | "#E5A8E2", 808 | "#AEA2E0", 809 | "#629E51", 810 | "#E5AC0E", 811 | "#64B0C8", 812 | "#E0752D", 813 | "#BF1B00", 814 | "#0A50A1", 815 | "#962D82", 816 | "#614D93", 817 | "#9AC48A", 818 | "#F2C96D", 819 | "#65C5DB", 820 | "#F9934E", 821 | "#EA6460", 822 | "#5195CE", 823 | "#D683CE", 824 | "#806EB7", 825 | "#3F6833", 826 | "#967302", 827 | "#2F575E", 828 | "#99440A", 829 | "#58140C", 830 | "#052B51", 831 | "#511749", 832 | "#3F2B5B", 833 | "#E0F9D7", 834 | "#FCEACA", 835 | "#CFFAFF", 836 | "#F9E2D2", 837 | "#FCE2DE", 838 | "#BADFF4", 839 | "#F9D9F9", 840 | "#DEDAF7" 841 | ], 842 | "title": "Event Type" 843 | }, 844 | { 845 | "span": 4, 846 | "editable": true, 847 | "type": "terms", 848 | "loadingEditor": false, 849 | "queries": { 850 | "mode": "all", 851 | "ids": [ 852 | 0, 853 | 1, 854 | 2, 855 | 3 856 | ], 857 | "query": "q=level%3AERROR OR level%3AWARN OR level%3AINFO OR level%3ADEBUG&wt=json&rows=0&fq=date:[2015-05-20T11:39:57.789Z%20TO%202015-07-27T21:34:48.567Z]&facet=true&facet.field=event_category&facet.limit=10&facet.missing=true", 858 | "custom": "" 859 | }, 860 | "mode": "count", 861 | "field": "event_category", 862 | "stats_field": "", 863 | "decimal_points": 0, 864 | "exclude": [], 865 | "missing": false, 866 | "other": false, 867 | "size": 10, 868 | "order": "descending", 869 | "style": { 870 | "font-size": "10pt" 871 | }, 872 | "donut": false, 873 | "tilt": false, 874 | "labels": true, 875 | "logAxis": false, 876 | "arrangement": "horizontal", 877 | "chart": "bar", 878 | "counter_pos": "below", 879 | "lastColor": "", 880 | "spyable": true, 881 | "show_queries": true, 882 | "chartColors": [ 883 | "#7EB26D", 884 | "#EAB839", 885 | "#6ED0E0", 886 | "#EF843C", 887 | "#E24D42", 888 | "#1F78C1", 889 | "#BA43A9", 890 | "#705DA0", 891 | "#508642", 892 | "#CCA300", 893 | "#447EBC", 894 | "#C15C17", 895 | "#890F02", 896 | "#0A437C", 897 | "#6D1F62", 898 | "#584477", 899 | "#B7DBAB", 900 | "#F4D598", 901 | "#70DBED", 902 | "#F9BA8F", 903 | "#F29191", 904 | "#82B5D8", 905 | "#E5A8E2", 906 | "#AEA2E0", 907 | "#629E51", 908 | "#E5AC0E", 909 | "#64B0C8", 910 | "#E0752D", 911 | "#BF1B00", 912 | "#0A50A1", 913 | "#962D82", 914 | "#614D93", 915 | "#9AC48A", 916 | "#F2C96D", 917 | "#65C5DB", 918 | "#F9934E", 919 | "#EA6460", 920 | "#5195CE", 921 | "#D683CE", 922 | "#806EB7", 923 | "#3F6833", 924 | "#967302", 925 | "#2F575E", 926 | "#99440A", 927 | "#58140C", 928 | "#052B51", 929 | "#511749", 930 | "#3F2B5B", 931 | "#E0F9D7", 932 | "#FCEACA", 933 | "#CFFAFF", 934 | "#F9E2D2", 935 | "#FCE2DE", 936 | "#BADFF4", 937 | "#F9D9F9", 938 | "#DEDAF7" 939 | ], 940 | "title": "Event Category" 941 | } 942 | ] 943 | }, 944 | { 945 | "title": "Table", 946 | "height": "150px", 947 | "editable": true, 948 | "collapse": false, 949 | "collapsable": true, 950 | "panels": [ 951 | { 952 | "span": 12, 953 | "editable": true, 954 | "type": "table", 955 | "loadingEditor": false, 956 | "status": "Stable", 957 | "queries": { 958 | "mode": "all", 959 | "ids": [ 960 | 0, 961 | 1, 962 | 2, 963 | 3 964 | ], 965 | "query": "q=level%3AERROR OR level%3AWARN OR level%3AINFO OR level%3ADEBUG&fq=date:[2015-05-20T11:39:57.789Z%20TO%202015-07-27T21:34:48.567Z]&sort=date desc&wt=json&rows=500", 966 | "basic_query": "q=level%3AERROR OR level%3AWARN OR level%3AINFO OR level%3ADEBUG&fq=date:[2015-05-20T11:39:57.789Z%20TO%202015-07-27T21:34:48.567Z]&sort=date desc", 967 | "custom": "" 968 | }, 969 | "size": 100, 970 | "pages": 5, 971 | "offset": 0, 972 | "sort": [ 973 | "date", 974 | "desc" 975 | ], 976 | "group": "default", 977 | "style": { 978 | "font-size": "9pt" 979 | }, 980 | "overflow": "min-height", 981 | "fields": [ 982 | "level", 983 | "thread", 984 | "date", 985 | "source_file", 986 | "source_line", 987 | "message" 988 | ], 989 | "highlight": [], 990 | "sortable": true, 991 | "header": true, 992 | "paging": true, 993 | "field_list": false, 994 | "trimFactor": 300, 995 | "normTimes": true, 996 | "spyable": true, 997 | "saveOption": "json", 998 | "exportSize": 500, 999 | "exportAll": true, 1000 | "displayLinkIcon": true, 1001 | "imageFields": [], 1002 | "imgFieldWidth": "auto", 1003 | "imgFieldHeight": "85px", 1004 | "title": "Table panel", 1005 | "important_fields": [ 1006 | "_dynFld", 1007 | "_partitionKey", 1008 | "_token_long", 1009 | "date", 1010 | "exception", 1011 | "f_percent_full", 1012 | "host", 1013 | "i_active", 1014 | "i_all_time_blocked", 1015 | "i_blocked", 1016 | "i_bytes", 1017 | "i_capacity", 1018 | "i_collections", 1019 | "i_completed", 1020 | "i_data", 1021 | "i_duration", 1022 | "i_hash_code", 1023 | "i_heap_used", 1024 | "i_input_bytes", 1025 | "i_live_bytes", 1026 | "i_max", 1027 | "i_ops", 1028 | "i_output_bytes", 1029 | "i_pending", 1030 | "i_percent_of_original", 1031 | "i_ranges", 1032 | "i_row_size", 1033 | "i_serialized_bytes", 1034 | "i_size", 1035 | "i_sstable_count", 1036 | "i_tables_remaining", 1037 | "i_total_heap", 1038 | "i_total_rows", 1039 | "i_unique_rows", 1040 | "i_used", 1041 | "id", 1042 | "level", 1043 | "log_file", 1044 | "log_line", 1045 | "message", 1046 | "message_type", 1047 | "s_cache_type", 1048 | "s_classpath", 1049 | "s_column_families", 1050 | "s_column_family", 1051 | "s_component", 1052 | "s_gc_type", 1053 | "s_input_sstables", 1054 | "s_jvm", 1055 | "s_keys_to_save", 1056 | "s_keyspace", 1057 | "s_node", 1058 | "s_node1", 1059 | "s_node2", 1060 | "s_nodes", 1061 | "s_output_sstable", 1062 | "s_pool_name", 1063 | "s_provider", 1064 | "s_range_begin", 1065 | "s_range_end", 1066 | "s_rate", 1067 | "s_row_key", 1068 | "s_row_merge_counts", 1069 | "s_session_id", 1070 | "s_sstable_name", 1071 | "s_table", 1072 | "s_version", 1073 | "source_file", 1074 | "source_line", 1075 | "thread" 1076 | ], 1077 | "show_queries": true 1078 | } 1079 | ] 1080 | } 1081 | ], 1082 | "editable": true, 1083 | "index": { 1084 | "interval": "none", 1085 | "pattern": "[logstash-]YYYY.MM.DD", 1086 | "default": "_all" 1087 | }, 1088 | "style": "light", 1089 | "failover": false, 1090 | "panel_hints": true, 1091 | "loader": { 1092 | "save_gist": false, 1093 | "save_elasticsearch": true, 1094 | "save_local": true, 1095 | "save_default": true, 1096 | "save_temp": true, 1097 | "save_temp_ttl_enable": true, 1098 | "save_temp_ttl": "30d", 1099 | "load_gist": true, 1100 | "load_elasticsearch": true, 1101 | "load_elasticsearch_size": 20, 1102 | "load_local": true, 1103 | "hide": false, 1104 | "dropdown_collections": false 1105 | }, 1106 | "solr": { 1107 | "server": "/solr/", 1108 | "core_name": "logparse.systemlog", 1109 | "core_list": [ 1110 | "banana.int", 1111 | "logparse.systemlog", 1112 | "zendesk.organizations", 1113 | "zendesk.ticket_comments", 1114 | "zendesk.ticket_events", 1115 | "zendesk.tickets", 1116 | "zendesk.users" 1117 | ], 1118 | "global_params": "" 1119 | } 1120 | } -------------------------------------------------------------------------------- /cassandra_ingest: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | import socket 3 | import fileinput 4 | 5 | import systemlog 6 | from cassandra_store import CassandraStore 7 | 8 | cassandra = CassandraStore() 9 | 10 | hostname = socket.gethostname() 11 | log = fileinput.input() 12 | #for event in systemlog.parse_log(log): 13 | # event['host'] = hostname 14 | # event['log_file'] = fileinput.filename() 15 | # event['log_line'] = fileinput.lineno() 16 | # cassandra.insert('systemlog', event) 17 | 18 | for error in cassandra.slurp('systemlog', systemlog.parse_log(log)): 19 | print error 20 | -------------------------------------------------------------------------------- /cassandra_store.py: -------------------------------------------------------------------------------- 1 | import json 2 | import uuid 3 | import collections 4 | 5 | from copy import deepcopy 6 | from datetime import datetime 7 | from cassandra.cluster import Cluster 8 | from cassandra.query import dict_factory 9 | from cassandra.util import uuid_from_time 10 | from cassandra.concurrent import execute_concurrent_with_args 11 | 12 | import collections 13 | 14 | def flatten(d, parent_key='', sep='_'): 15 | items = [] 16 | for k, v in d.items(): 17 | new_key = parent_key + sep + k if parent_key else k 18 | if isinstance(v, collections.MutableMapping): 19 | items.extend(flatten(v, new_key, sep=sep).items()) 20 | else: 21 | items.append((new_key, v)) 22 | return dict(items) 23 | 24 | def genericize(columns, parameters): 25 | def date_handler(obj): 26 | return obj.isoformat() if hasattr(obj, 'isoformat') else obj 27 | 28 | parameters = flatten(deepcopy(parameters)) 29 | 30 | if 'id' not in parameters: 31 | if 'date' in parameters: 32 | parameters['id'] = uuid_from_time(parameters['date']) 33 | else: 34 | parameters['id'] = uuid.uuid4() 35 | 36 | generic = {'b_': {}, 'd_': {}, 'i_': {}, 'f_': {}, 's_': {}, 'l_': {}} 37 | for key, value in parameters.items(): 38 | if key not in columns: 39 | if type(value) is bool: 40 | prefix = 'b_' 41 | elif type(value) is datetime: 42 | prefix = 'd_' 43 | elif type(value) is int: 44 | prefix = 'i_' 45 | elif type(value) is float: 46 | prefix = 'f_' 47 | elif isinstance(value, (str, unicode)): 48 | prefix = 's_' 49 | elif type(value) is list: 50 | prefix = 'l_' 51 | value = '\n'.join([str(x) for x in value]) 52 | elif value is not None: 53 | prefix = 's_' 54 | value = json.dumps(value, default=date_handler) 55 | if value is not None: 56 | generic[prefix][prefix + key] = value 57 | del parameters[key] 58 | for key, value in generic.items(): 59 | if key in columns: 60 | parameters[key] = value 61 | for name in columns: 62 | if name not in parameters: 63 | parameters[name] = None 64 | return parameters 65 | 66 | class CassandraStore: 67 | def __init__(self, contacts=None, keyspace='logparse'): 68 | self.cluster = Cluster(contacts) 69 | self.session = self.cluster.connect() 70 | self.session.row_factory = dict_factory 71 | self.session.set_keyspace(keyspace) 72 | self.keyspace = keyspace 73 | self.prepare_statements() 74 | 75 | def prepare_statements(self): 76 | self.insert_statements = {} 77 | self.columns = {} 78 | for table in self.cluster.metadata.keyspaces[self.keyspace].tables: 79 | self.columns[table] = [column for column in self.cluster.metadata.keyspaces[self.keyspace].tables[table].columns] 80 | statement = 'insert into {} ({}) values ({});'.format( 81 | table, 82 | ', '.join(self.columns[table]), 83 | ', '.join([':' + column for column in self.columns[table]]) 84 | ) 85 | self.insert_statements[table] = self.session.prepare(statement) 86 | 87 | def insert(self, table, parameters, async=False): 88 | def log_error(ex): 89 | print 'ERROR: %s when inserting %s: %s' % (ex, table, parameters) 90 | 91 | parameters = genericize(self.columns[table], parameters) 92 | try: 93 | if async: 94 | future = self.session.execute_async(self.insert_statements[table], parameters) 95 | future.add_errback(log_error) 96 | else: 97 | self.session.execute(self.insert_statements[table], parameters) 98 | except Exception, ex: 99 | log_error(ex) 100 | 101 | def slurp(self, table, stream, concurrency=1000): 102 | generic_stream = (genericize(self.columns[table], parameters) for parameters in stream) 103 | for success, result in execute_concurrent_with_args(self.session, self.insert_statements[table], generic_stream, concurrency=concurrency, results_generator=True): 104 | if not success: 105 | yield result 106 | -------------------------------------------------------------------------------- /clography.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | import sys 3 | import fileinput 4 | from collections import defaultdict 5 | from getopt import getopt, GetoptError 6 | 7 | import matplotlib.pyplot as plt 8 | import matplotlib.cm as cm 9 | import numpy as np 10 | import systemlog 11 | 12 | try: 13 | # I am bad at picking single-letter switches, and feel bad 14 | (options, arguments) = getopt(sys.argv[1:], 'i:s:eturm:', 15 | ['interval', 'scale', 'events', 'events-only', 'show-unknown', 'sort', 'min-size']) 16 | except GetoptError, error: 17 | sys.stderr.write('%s\n' % str(error)) 18 | sys.exit(1) 19 | 20 | # dirty, but I'm not rewriting fileinput.input 21 | sys.argv = [sys.argv[0]] 22 | 23 | interval = 3600 24 | scale = 10 25 | allevents = False 26 | eventsonly = False 27 | unknowns = False 28 | sort = False 29 | minsize = 0 30 | 31 | for opt, arg in options: 32 | if opt in ('-i', '--interval'): 33 | interval = int(arg) 34 | if opt in ('-s', '--scale'): 35 | scale = float(arg) 36 | if opt in ('-e', '--events'): 37 | allevents = True 38 | if opt in ('-t', '--events-only'): 39 | eventsonly = True 40 | if opt in ('-u', '--show-unknown'): 41 | unknowns = True 42 | if opt in ('-r', '--sort'): 43 | sort = True 44 | if opt in ('-m', '--min-size'): 45 | minsize = int(arg) 46 | 47 | if arguments: 48 | log = fileinput.input(arguments[0]) 49 | else: 50 | log = fileinput.input() 51 | 52 | stages = defaultdict(int) 53 | enum = 0 54 | data = defaultdict(lambda: defaultdict(int)) 55 | for event in systemlog.parse_log(log): 56 | stage = event['thread_name'] + ' ' + event['source_file'] 57 | if stage[0:3].isupper() or event['thread_name'] == 'main': # skip rmi, handshaking, streams, etc 58 | continue 59 | if event['event_type'] == 'unknown' and not unknowns: 60 | continue 61 | if event['event_type'] == 'messages_dropped': 62 | stage = event['message_type'] + ' dropped' 63 | elif event['event_type'] == 'begin_flush': 64 | stage = 'flushed bytes (serialized)' 65 | elif allevents: 66 | if not eventsonly: 67 | stage = ' '.join((event['event_category'], event['event_type'], stage)) 68 | else: 69 | stage = event['event_category'] + ' ' +event['event_type'] 70 | 71 | if not stage in stages and not sort: 72 | stages[stage] = enum 73 | enum += 1 74 | 75 | ts = int(event['date'].strftime('%s')) / interval 76 | if event['event_type'] == 'pause': 77 | data[ts][stage] += event['duration'] / 1000 78 | elif event['event_type'] == 'messages_dropped': 79 | if 'internal_timeout' in event and event['internal_timeout'] != None: 80 | data[ts][stage] += event['internal_timeout'] + event['cross_node_timeout'] 81 | else: 82 | data[ts][stage] += event['messages_dropped'] 83 | elif event['event_type'] == 'begin_flush' and event['serialized_bytes'] != None: 84 | data[ts][stage] += (event['serialized_bytes'] / 1024**2 / scale) 85 | elif event['event_type'] in ('incremental_compaction', 'large_partition'): 86 | data[ts][stage] += event['partition_size'] / 1024**2 / scale 87 | elif event['event_type'] == 'begin_compaction': 88 | data[ts][stage] += len(event['input_sstables']) 89 | elif event['event_type'] == 'end_compaction': 90 | data[ts][stage] += event['output_bytes'] / 1024**2 / scale 91 | else: 92 | data[ts][stage] += 1 93 | 94 | if sort: 95 | enum = 0 96 | for stage in reversed(sorted(stages.keys())): 97 | stages[stage] = enum 98 | enum += 1 99 | size=max(stages[stage]*scale, minsize) 100 | fig, ax = plt.subplots() 101 | colors = cm.rainbow(np.linspace(0, 1, len(stages))) 102 | for ts, info in data.iteritems(): 103 | for stage in info.keys(): 104 | size=info[stage]*scale 105 | ax.scatter(ts, stages[stage], s=size, c=colors[stages[stage]], alpha=0.5) 106 | if size > 700: 107 | ax.scatter(ts, stages[stage], s=size*0.01, c='black', alpha=1, marker='_', lw=1) 108 | 109 | plt.yticks(stages.values(), stages.keys()) 110 | 111 | ax.set_xlabel('Time (%s second buckets)' % interval, fontsize=20) 112 | if not eventsonly: 113 | ax.set_ylabel('Stage', fontsize=20) 114 | else: 115 | ax.set_ylabel('Event', fontsize=20) 116 | 117 | ax.grid(True) 118 | 119 | plt.plot() 120 | try: 121 | fig.tight_layout() 122 | except ValueError: # too many things on the Y axis 123 | print "Warning, tight layout not possible, too many items on the Y axis" 124 | fig.subplots_adjust(bottom = 0.2) 125 | fig.subplots_adjust(top = 1) 126 | fig.subplots_adjust(right = 1) 127 | fig.subplots_adjust(left = 0) 128 | plt.subplots_adjust(left=0.21) 129 | plt.show() 130 | -------------------------------------------------------------------------------- /log_to_json: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | import sys 3 | import socket 4 | import json 5 | import fileinput 6 | 7 | import systemlog 8 | 9 | def date_handler(obj): 10 | return obj.isoformat() if hasattr(obj, 'isoformat') else obj 11 | 12 | hostname = socket.gethostname() 13 | log = fileinput.input() 14 | for event in systemlog.parse_log(log): 15 | event['host'] = hostname 16 | event['log_file'] = fileinput.filename() 17 | event['log_line'] = fileinput.lineno() 18 | print json.dumps(event, default=date_handler) 19 | -------------------------------------------------------------------------------- /rules.py: -------------------------------------------------------------------------------- 1 | ''' 2 | A set of functions defining a domain-specific language that specifies a set of rules for 3 | parsing the lines in a log file. 4 | ''' 5 | 6 | import re 7 | from datetime import datetime 8 | from collections import defaultdict 9 | 10 | class switch: 11 | ''' 12 | Tries multiple rules in the specified order until one returns a value other than None. 13 | Returns the result of the first successful rule. Can be configured to run only a 14 | subset of the rules using an optional case value. 15 | ''' 16 | 17 | def __init__(self, children): 18 | ''' 19 | Constructor expects to be passed one or more case and rule objects. The case objects are 20 | used to group the rules. 21 | ''' 22 | 23 | self.rules = defaultdict(list) 24 | keys = None 25 | for child in children: 26 | if isinstance(child, case): 27 | keys = child.keys 28 | else: 29 | for key in keys: 30 | self.rules[key].append(child) 31 | 32 | def __call__(self, key, data): 33 | if key in self.rules: 34 | for rule in self.rules[key]: 35 | result = rule(data) 36 | if result is not None: 37 | return result 38 | return None 39 | 40 | class case: 41 | "Specifies an alternative for a switch rule." 42 | 43 | def __init__(self, *keys): 44 | ''' 45 | Constructor expects to be passed one or more strings. At least one of the strings in 46 | the case must match the case value passed to the switch for the case to be selected. 47 | ''' 48 | self.keys = keys 49 | 50 | class rule: 51 | ''' 52 | Executes the condition, and optionally one or more actions. If the condition returns None, 53 | the rule returns None immediately. If the condition returns something other than None, the 54 | rule executes each action in order, passing the result of the condition into each. 55 | ''' 56 | 57 | 58 | def __init__(self, source, *transforms): 59 | ''' 60 | Constructor expects the first parameter to be a function that takes a string as input. The 61 | condition should return None to indicate failure or something else to indicate success. 62 | The remaining parameters should be functions that act on the result of the condition. 63 | ''' 64 | self.source = source 65 | self.transforms = transforms 66 | 67 | def __call__(self, string): 68 | fields = self.source(string) 69 | if fields is not None: 70 | for transform in self.transforms: 71 | transform(fields) 72 | return fields 73 | 74 | class capture: 75 | ''' 76 | Matches the input string against one or more regular expressions and returns a dictionary of 77 | values captured by regex's named capture groups. Returns None if none of the regular expressions 78 | match the input string. Returns an empty dict if the regular expression doesn't contains any 79 | named capture groups. 80 | ''' 81 | 82 | def __init__(self, *regex_strings): 83 | "Constructor expects a list of one or more regular expression strings." 84 | self.regexes = [] 85 | for regex in regex_strings: 86 | self.regexes.append(re.compile(regex)) 87 | 88 | def __call__(self, string): 89 | for regex in self.regexes: 90 | capture = regex.match(string) 91 | if capture: 92 | return capture.groupdict() 93 | return None 94 | 95 | class convert: 96 | ''' 97 | Applies the specified conversion function against each of the named fields in the input dictionary. 98 | The value of each field is passed to the conversion function and is replaced by the value returned 99 | by the conversion function. 100 | ''' 101 | 102 | def __init__(self, func, *field_names): 103 | ''' 104 | Constructor expects a conversion function followed by fields specifying one or more field names. 105 | The conversion function should take a string as input and return a converted value. 106 | ''' 107 | self.func = func 108 | self.field_names = field_names 109 | 110 | def __call__(self, fields): 111 | for field_name in self.field_names: 112 | if field_name in fields and fields[field_name] is not None: 113 | fields[field_name] = self.func(fields[field_name]) 114 | 115 | class update: 116 | "Updates the specified fields in the input dictionary with the specified values." 117 | 118 | def __init__(self, **extras): 119 | ''' 120 | Constructor expects a set of named parameters specifying key value pairs to be set in the input 121 | dictionary. 122 | ''' 123 | self.extras = extras 124 | 125 | def __call__(self, fields): 126 | fields.update(self.extras) 127 | 128 | class default: 129 | ''' 130 | Updates the specified fields in the input dictionary with the specified values only if the field 131 | does not already exist. 132 | ''' 133 | 134 | def __init__(self, **defaults): 135 | ''' 136 | Constructor expects a set of named parameters specifying key value pairs to be set in the input 137 | dictionary. 138 | ''' 139 | self.defaults = defaults 140 | 141 | def __call__(self, fields): 142 | for key, value in self.defaults.iteritems(): 143 | if key not in fields: 144 | fields[key] = value 145 | 146 | def strip(string): 147 | "Strips the leading and trailing whitespace from the supplied string and returns the result." 148 | return string.strip() 149 | 150 | class date: 151 | "Parses the supplied date and returns the resulting datetime value." 152 | 153 | def __init__(self, format): 154 | "Constructor expects a date string supported by datetime.strptime." 155 | self.format = format 156 | 157 | def __call__(self, date): 158 | return datetime.strptime(date, self.format) 159 | 160 | class split: 161 | "Splits the supplied string and returns the resulting list." 162 | 163 | def __init__(self, delimiter): 164 | "Constructor expects a string to use as the delimiter." 165 | self.delimiter = delimiter 166 | 167 | def __call__(self, string): 168 | return string.split(self.delimiter) 169 | 170 | def percent(value): 171 | "Converts the supplied string to a floating point and multiplies it by 100." 172 | return float(value) * 100 173 | 174 | def int_with_commas(value): 175 | "Removes any commas from the input string and converts the result to an int." 176 | return int(value.replace(',', '')) 177 | -------------------------------------------------------------------------------- /schema/add-schema.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | SCHEMA_PATH=`dirname $0` 3 | cqlsh -f $SCHEMA_PATH/systemlog.cql 4 | dsetool create_core logparse.systemlog schema=$SCHEMA_PATH/schema.xml solrconfig=$SCHEMA_PATH/solrconfig.xml 5 | -------------------------------------------------------------------------------- /schema/schema.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | id 54 | message 55 | 56 | -------------------------------------------------------------------------------- /schema/solrconfig.xml: -------------------------------------------------------------------------------- 1 | 2 | 12 | 19 | 20 | 30 | ${solr.abortOnConfigurationError:true} 31 | 32 | 38 | LUCENE_46 39 | 40 | 41 | 2 42 | 43 | 57 | 61 | 64 | 67 | 68 | 77 | 78 | 79 | 90 | 91 | 92 | 93 | true 94 | false 95 | 512 96 | 10 97 | 98 | 107 | true 108 | 109 | 112 | true 113 | 114 | 128 | 129 | 130 | 1 131 | 132 | 0 133 | 137 | 141 | 142 | 143 | 151 | false 152 | 153 | 154 | 155 | 164 | 165 | 168 | 169 | 170 | 172 | 173 | 174 | 175 | 176 | 177 | 192 | 193 | 10000 194 | 195 | 196 | 204 | 214 | 218 | 227 | 228 | 229 | 230 | 243 | 1024 244 | 245 | 256 | 257 | 274 | 275 | 276 | 281 | 282 | 283 | 284 | 288 | 289 | 295 | 301 | 302 | 310 | true 311 | 312 | 325 | 328 | 329 | 339 | 340 | 344 | 345 | 361 | 364 | 379 | 380 | 387 | true 388 | 389 | 398 | 16 399 | 400 | 401 | 402 | 403 | 418 | 419 | 437 | 438 | 439 | 446 | 447 | 456 | 461 | 486 | 492 | 493 | 494 | 495 | 514 | 515 | 516 | 525 | 526 | 529 | 530 | 10 531 | 532 | 536 | 545 | 550 | 566 | 574 | 578 | 584 | 585 | 586 | 587 | 590 | 591 | 592 | 593 | 10 594 | 595 | 596 | 597 | 598 | 609 | 610 | 614 | 619 | 620 | 621 | 622 | 625 | 626 | 627 | 628 | 631 | 632 | 633 | 634 | 652 | 653 | 654 | 672 | 673 | 674 | 675 | 705 | 706 | 707 | 708 | 713 | 714 | 715 | 723 | 726 | 735 | 736 | 737 | 738 | 739 | 740 | search 741 | solrpingquery 742 | 743 | 744 | all 745 | 746 | 747 | 748 | 749 | 750 | 751 | 752 | explicit 753 | true 754 | 755 | 756 | 757 | 758 | 800 | 801 | 824 | 825 | 849 | 850 | 856 | 857 | 858 | 874 | 875 | 884 | 885 | 896 | 897 | 898 | 906 | 913 | 914 | 924 | 925 | 1005 | 1006 | 1007 | 1018 | 1021 | 1033 | 1036 | 1039 | 1040 | 1048 | 1049 | 1057 | 1058 | 1061 | 1062 | 1069 | 1070 | 1074 | 1075 | 1076 | 1077 | *:* 1078 | 1079 | 1082 | 1085 | 1086 | 1087 | 1088 | 1089 | -------------------------------------------------------------------------------- /schema/systemlog.cql: -------------------------------------------------------------------------------- 1 | drop keyspace if exists logparse; 2 | create keyspace logparse WITH replication = {'class': 'SimpleStrategy', 'replication_factor' : 1}; 3 | 4 | create table logparse.systemlog ( 5 | id timeuuid primary key, 6 | log_file text, 7 | log_line bigint, 8 | host text, 9 | level text, 10 | thread text, 11 | thread_name text, 12 | thread_id text, 13 | date timestamp, 14 | source_file text, 15 | source_line bigint, 16 | message text, 17 | event_product text, 18 | event_category text, 19 | event_type text, 20 | exception text, 21 | b_ map, 22 | d_ map, 23 | i_ map, 24 | f_ map, 25 | s_ map, 26 | l_ map 27 | ) ; 28 | --------------------------------------------------------------------------------