├── .travis.yml ├── README.md ├── LICENSE.txt └── elasticsearch_collectd.py /.travis.yml: -------------------------------------------------------------------------------- 1 | language: python 2 | python: 3 | - "2.6" 4 | - "2.7" 5 | install: 6 | pip install flake8 7 | before_script: 8 | flake8 *.py 9 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Elasticsearch CollectD plugin 2 | 3 | A [CollectD](http://collectd.org) plugin to collect [Elasticsearch](http://elasticsearch.org) stats and metrics. Uses CollectD's [Python plugin](http://collectd.org/documentation/manpages/collectd-python.5.shtml). 4 | 5 | ## Installation 6 | 7 | 1. Place `elasticsearch_collectd.py` in `/opt/collectd/lib/collectd/plugins/python` (assuming you have collectd installed to `/opt/collectd`). 8 | 1. Configure the plugin (see below). 9 | 1. Restart collectd. 10 | 11 | ### Requirements 12 | 13 | * collectd 4.9+ 14 | * Elasticsearch 1.x or newer. 15 | 16 | ## Configuration 17 | 18 | * See [`elasticsearch.conf`](https://github.com/signalfx/integrations/blob/master/collectd-elasticsearch/20-elasticsearch.conf) 19 | * The plugin will automatically determine the version of Elasticsearch you are running. 20 | * Per-index and cluster stats can be disabled if needed; they are enabled by default. 21 | * If you are running the Elasticsearch plugin via a collectd deployment within a container, please configure the Host and Port values inside of the 20-elasticsearch.conf file that correspond to the desired Elasticsearch instance. 22 | 23 | ex: 24 | ``` 25 | 26 | Host "XXX.XXX.XXX.XXX" 27 | Port "XXXX" 28 | 29 | ``` 30 | 31 | ## Metrics 32 | 33 | ### Node stats 34 | 35 | * Documents (total docs & deleted docs) 36 | * Store size 37 | * Indexing (total, time, total delete, delete time) 38 | * Get (total, time, exists total, exists time, missing total, missing time) 39 | * Search (total query, total time, total fetch, total fetch time) 40 | * JVM uptime 41 | * JVM memory (heap commited, heap Used, non heap commited, non heap used) 42 | * JVM threads (count & peak) 43 | * JVM GC (time & count) 44 | * Transport stats (server open, RX count, RX size, TX count, TX size) 45 | * HTTP stats (current open & total open) 46 | * OS stats (CPU percent, file descriptors) 47 | * Thread pool stats (generic, index, get, snapshot, merge, optimize, bulk, warmer, flush, search, refresh) 48 | * Cache (field eviction, field size, filter evictions, filter size) 49 | * JVM collectors 50 | * FLush (total count, total time) 51 | * Merges (current count, current docs, current size, merge total size, docs a time) 52 | * Refresh (Total & Time) 53 | 54 | ### Index stats 55 | 56 | * Transaction log (size, number of operations) 57 | * Most of the common stats per index and per primary vs. total. 58 | 59 | ### Cluster stats 60 | 61 | * Shard stats (active, initializing, relocating, unassigned, primaries) 62 | * Nodes (total, data nodes) 63 | -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | 177 | END OF TERMS AND CONDITIONS 178 | 179 | APPENDIX: How to apply the Apache License to your work. 180 | 181 | To apply the Apache License to your work, attach the following 182 | boilerplate notice, with the fields enclosed by brackets "[]" 183 | replaced with your own identifying information. (Don't include 184 | the brackets!) The text should be enclosed in the appropriate 185 | comment syntax for the file format. We also recommend that a 186 | file or class name and description of purpose be included on the 187 | same "printed page" as the copyright notice for easier 188 | identification within third-party archives. 189 | 190 | Copyright 2012-2013 Aurelius LLC 191 | 192 | Licensed under the Apache License, Version 2.0 (the "License"); 193 | you may not use this file except in compliance with the License. 194 | You may obtain a copy of the License at 195 | 196 | http://www.apache.org/licenses/LICENSE-2.0 197 | 198 | Unless required by applicable law or agreed to in writing, software 199 | distributed under the License is distributed on an "AS IS" BASIS, 200 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 201 | See the License for the specific language governing permissions and 202 | limitations under the License. -------------------------------------------------------------------------------- /elasticsearch_collectd.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/python 2 | # Copyright 2014 Jeremy Carroll 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | 16 | 17 | import collections 18 | import json 19 | import urllib2 20 | import socket 21 | import httplib 22 | 23 | PREFIX = "elasticsearch" 24 | ES_CLUSTER = "elasticsearch" 25 | ES_HOST = "localhost" 26 | ES_AUTH_HOST = socket.getfqdn() 27 | ES_PORT = 9200 28 | ES_URL_SCHEME = "http" 29 | ES_AUTH_URL_SCHEME = "https" 30 | ES_HTTP_TLS_ENABLED = False 31 | ES_TLS_CERT_PATH = "" 32 | ES_TLS_KEY_PATH = "" 33 | ES_VERSION = None 34 | 35 | ENABLE_INDEX_STATS = True 36 | ENABLE_CLUSTER_STATS = True 37 | 38 | ES_NODE_URL = "" 39 | ES_CLUSTER_URL = "" 40 | ES_INDEX_URL = "" 41 | ES_INDEX = [] 42 | 43 | VERBOSE_LOGGING = False 44 | 45 | Stat = collections.namedtuple('Stat', ('type', 'path')) 46 | 47 | NODE_STATS_CUR = {} 48 | INDEX_STATS_CUR = {} 49 | CLUSTER_STATS_CUR = {} 50 | 51 | COLLECTION_INTERVAL = 10 52 | 53 | CLUSTER_STATUS = {'green': 0, 'yellow': 1, 'red': 2} 54 | 55 | 56 | # Class to store an es url 57 | # allows generating between https auth and http non-auth urls easily 58 | # url_suffix is after the port in the url, for example in 'localhost:9200/_cat/nodes' would be '/_cat/nodes' 59 | class ElasticsearchRequestUrl(object): 60 | def __init__(self, url_suffix): 61 | self.url_suffix = url_suffix 62 | 63 | def get_auth_url(self): 64 | return ES_AUTH_URL_SCHEME + "://" + ES_AUTH_HOST + ":" + str(ES_PORT) + self.url_suffix 65 | 66 | def get_non_auth_url(self): 67 | return ES_URL_SCHEME + "://" + ES_HOST + ":" + str(ES_PORT) + self.url_suffix 68 | 69 | 70 | # DICT: ElasticSearch 1.0.0 71 | NODE_STATS = { 72 | # STORE 73 | 'indices.store.throttle-time': 74 | Stat("counter", "nodes.%s.indices.store.throttle_time_in_millis"), 75 | 76 | # SEARCH 77 | 'indices.search.open-contexts': 78 | Stat("gauge", "nodes.%s.indices.search.open_contexts"), 79 | 80 | # CACHE 81 | 'indices.cache.field.eviction': 82 | Stat("counter", "nodes.%s.indices.fielddata.evictions"), 83 | 'indices.cache.field.size': 84 | Stat("gauge", "nodes.%s.indices.fielddata.memory_size_in_bytes"), 85 | 'indices.cache.filter.evictions': 86 | Stat("counter", "nodes.%s.indices.filter_cache.evictions"), 87 | 'indices.cache.filter.size': 88 | Stat("gauge", "nodes.%s.indices.filter_cache.memory_size_in_bytes"), 89 | 90 | # GC 91 | 'jvm.gc.time': 92 | Stat("counter", 93 | "nodes.%s.jvm.gc.collectors.young.collection_time_in_millis"), 94 | 'jvm.gc.count': 95 | Stat("counter", "nodes.%s.jvm.gc.collectors.young.collection_count"), 96 | 'jvm.gc.old-time': 97 | Stat("counter", 98 | "nodes.%s.jvm.gc.collectors.old.collection_time_in_millis"), 99 | 'jvm.gc.old-count': 100 | Stat("counter", "nodes.%s.jvm.gc.collectors.old.collection_count"), 101 | 102 | # FLUSH 103 | 'indices.flush.total': Stat("counter", "nodes.%s.indices.flush.total"), 104 | 'indices.flush.time': 105 | Stat("counter", "nodes.%s.indices.flush.total_time_in_millis"), 106 | 107 | # MERGES 108 | 'indices.merges.current': Stat("gauge", "nodes.%s.indices.merges.current"), 109 | 'indices.merges.current-docs': 110 | Stat("gauge", "nodes.%s.indices.merges.current_docs"), 111 | 'indices.merges.current-size': 112 | Stat("gauge", "nodes.%s.indices.merges.current_size_in_bytes"), 113 | 'indices.merges.total': Stat("counter", "nodes.%s.indices.merges.total"), 114 | 'indices.merges.total-docs': Stat("gauge", 115 | "nodes.%s.indices.merges.total_docs"), 116 | 'indices.merges.total-size': 117 | Stat("counter", "nodes.%s.indices.merges.total_size_in_bytes"), 118 | 'indices.merges.time': 119 | Stat("counter", "nodes.%s.indices.merges.total_time_in_millis"), 120 | 121 | # REFRESH 122 | 'indices.refresh.total': Stat("counter", "nodes.%s.indices.refresh.total"), 123 | 'indices.refresh.time': 124 | Stat("counter", "nodes.%s.indices.refresh.total_time_in_millis"), 125 | 126 | # SEGMENTS 127 | 'indices.segments.count': Stat("gauge", "nodes.%s.indices.segments.count"), 128 | 'indices.segments.size': Stat("gauge", 129 | "nodes.%s.indices.segments.memory_in_bytes"), 130 | 'indices.segments.index-writer-max-size': Stat("gauge", 131 | "nodes.%s.indices.segments.index_writer_max_memory_in_bytes"), 132 | 'indices.segments.index-writer-size': Stat("gauge", 133 | "nodes.%s.indices.segments.index_writer_memory_in_bytes"), 134 | 135 | # DOCS 136 | 'indices.docs.count': Stat("gauge", "nodes.%s.indices.docs.count"), 137 | 'indices.docs.deleted': Stat("gauge", "nodes.%s.indices.docs.deleted"), 138 | 139 | # STORE 140 | 'indices.store.size': 141 | Stat("gauge", "nodes.%s.indices.store.size_in_bytes"), 142 | 143 | # INDEXING 144 | 'indices.indexing.index-total': 145 | Stat("counter", "nodes.%s.indices.indexing.index_total"), 146 | 'indices.indexing.index-time': 147 | Stat("counter", "nodes.%s.indices.indexing.index_time_in_millis"), 148 | 'indices.indexing.delete-total': 149 | Stat("counter", "nodes.%s.indices.indexing.delete_total"), 150 | 'indices.indexing.delete-time': 151 | Stat("counter", "nodes.%s.indices.indexing.delete_time_in_millis"), 152 | 'indices.indexing.index-current': 153 | Stat("gauge", "nodes.%s.indices.indexing.index_current"), 154 | 'indices.indexing.delete-current': 155 | Stat("gauge", "nodes.%s.indices.indexing.delete_current"), 156 | 'indices.indexing.throttle-time': 157 | Stat("counter", "nodes.%s.indices.indexing.throttle_time_in_millis"), 158 | 159 | # GET 160 | 'indices.get.total': Stat("counter", "nodes.%s.indices.get.total"), 161 | 'indices.get.time': Stat("counter", "nodes.%s.indices.get.time_in_millis"), 162 | 'indices.get.exists-total': Stat("counter", 163 | "nodes.%s.indices.get.exists_total"), 164 | 'indices.get.exists-time': 165 | Stat("counter", "nodes.%s.indices.get.exists_time_in_millis"), 166 | 'indices.get.missing-total': Stat("counter", 167 | "nodes.%s.indices.get.missing_total"), 168 | 'indices.get.missing-time': 169 | Stat("counter", "nodes.%s.indices.get.missing_time_in_millis"), 170 | 'indices.get.current': Stat("gauge", "nodes.%s.indices.get.current"), 171 | 172 | # SEARCH 173 | 'indices.search.query-current': 174 | Stat("gauge", "nodes.%s.indices.search.query_current"), 175 | 'indices.search.query-total': Stat("counter", 176 | "nodes.%s.indices.search.query_total"), 177 | 'indices.search.query-time': 178 | Stat("counter", "nodes.%s.indices.search.query_time_in_millis"), 179 | 'indices.search.fetch-current': 180 | Stat("gauge", "nodes.%s.indices.search.fetch_current"), 181 | 'indices.search.fetch-total': Stat("counter", 182 | "nodes.%s.indices.search.fetch_total"), 183 | 'indices.search.fetch-time': 184 | Stat("counter", "nodes.%s.indices.search.fetch_time_in_millis"), 185 | 186 | # JVM METRICS # 187 | # MEM 188 | 'jvm.mem.heap-committed': Stat("gauge", 189 | "nodes.%s.jvm.mem.heap_committed_in_bytes"), 190 | 'jvm.mem.heap-used': Stat("gauge", "nodes.%s.jvm.mem.heap_used_in_bytes"), 191 | 'jvm.mem.heap-used-percent': Stat("percent", 192 | "nodes.%s.jvm.mem.heap_used_percent"), 193 | 'jvm.mem.non-heap-committed': 194 | Stat("gauge", "nodes.%s.jvm.mem.non_heap_committed_in_bytes"), 195 | 'jvm.mem.non-heap-used': Stat("gauge", 196 | "nodes.%s.jvm.mem.non_heap_used_in_bytes"), 197 | 'jvm.mem.pools.young.max_in_bytes': Stat("gauge", 198 | "nodes.%s.jvm.mem.pools.young.max_in_bytes"), 199 | 'jvm.mem.pools.young.used_in_bytes': Stat("gauge", 200 | "nodes.%s.jvm.mem.pools.young.used_in_bytes"), 201 | 'jvm.mem.pools.old.max_in_bytes': Stat("gauge", 202 | "nodes.%s.jvm.mem.pools.old.max_in_bytes"), 203 | 'jvm.mem.pools.old.used_in_bytes': Stat("gauge", 204 | "nodes.%s.jvm.mem.pools.old.used_in_bytes"), 205 | 206 | # UPTIME 207 | 'jvm.uptime': Stat("counter", "nodes.%s.jvm.uptime_in_millis"), 208 | 209 | # THREADS 210 | 'jvm.threads.count': Stat("gauge", "nodes.%s.jvm.threads.count"), 211 | 'jvm.threads.peak': Stat("gauge", "nodes.%s.jvm.threads.peak_count"), 212 | 213 | # TRANSPORT METRICS # 214 | 'transport.server_open': Stat("gauge", "nodes.%s.transport.server_open"), 215 | 'transport.rx.count': Stat("counter", "nodes.%s.transport.rx_count"), 216 | 'transport.rx.size': 217 | Stat("counter", "nodes.%s.transport.rx_size_in_bytes"), 218 | 'transport.tx.count': Stat("counter", "nodes.%s.transport.tx_count"), 219 | 'transport.tx.size': 220 | Stat("counter", "nodes.%s.transport.tx_size_in_bytes"), 221 | 222 | # HTTP METRICS # 223 | 'http.current_open': Stat("gauge", "nodes.%s.http.current_open"), 224 | 'http.total_open': Stat("counter", "nodes.%s.http.total_opened"), 225 | 226 | # PROCESS METRICS # 227 | 'process.open_file_descriptors': 228 | Stat("gauge", "nodes.%s.process.open_file_descriptors"), 229 | 'process.cpu.percent': Stat("gauge", "nodes.%s.process.cpu.percent"), 230 | 'process.mem.share_in_bytes': Stat("gauge", "nodes.%s.process.mem.share_in_bytes"), 231 | } 232 | 233 | NODE_STATS_ES_2 = { 234 | 'indices.cache.filter.evictions': 235 | Stat("counter", "nodes.%s.indices.query_cache.evictions"), 236 | 'indices.cache.filter.size': 237 | Stat("gauge", "nodes.%s.indices.query_cache.cache_size"), 238 | 'indices.cache.filter.hit-count': 239 | Stat("counter", "nodes.%s.indices.query_cache.hit_count"), 240 | 'indices.cache.filter.miss-count': 241 | Stat("counter", "nodes.%s.indices.query_cache.miss_count"), 242 | 'indices.cache.filter.cache-count': 243 | Stat("counter", "nodes.%s.indices.query_cache.cache_count"), 244 | 'indices.cache.filter.total-count': 245 | Stat("counter", "nodes.%s.indices.query_cache.total_count"), 246 | } 247 | 248 | # ElasticSearch 1.3.0 249 | INDEX_STATS_ES_1_3 = { 250 | # SEGMENTS 251 | "indices[index={index_name}].primaries.segments.index-writer-memory": 252 | Stat("gauge", "primaries.segments.index_writer_memory_in_bytes"), 253 | "indices[index={index_name}].primaries.segments.version-map-memory": 254 | Stat("gauge", "primaries.segments.version_map_memory_in_bytes"), 255 | } 256 | 257 | # ElasticSearch 1.1.0 258 | INDEX_STATS_ES_1_1 = { 259 | # SUGGEST 260 | "indices[index={index_name}].primaries.suggest.total": 261 | Stat("counter", "primaries.suggest.total"), 262 | "indices[index={index_name}].primaries.suggest.time": 263 | Stat("counter", "primaries.suggest.time_in_millis"), 264 | "indices[index={index_name}].primaries.suggest.current": 265 | Stat("gauge", "primaries.suggest.current"), 266 | } 267 | 268 | # ElasticSearch 1.0.0 269 | INDEX_STATS = { 270 | # PRIMARIES 271 | # TRANSLOG 272 | "indices[index={index_name}].primaries.translog.size": 273 | Stat("gauge", "primaries.translog.size_in_bytes"), 274 | "indices[index={index_name}].primaries.translog.operations": 275 | Stat("counter", "primaries.translog.operations"), 276 | 277 | # SEGMENTS 278 | "indices[index={index_name}].primaries.segments.memory": 279 | Stat("gauge", "primaries.segments.memory_in_bytes"), 280 | "indices[index={index_name}].primaries.segments.count": 281 | Stat("counter", "primaries.segments.count"), 282 | 283 | # ID_CACHE 284 | "indices[index={index_name}].primaries.id-cache.memory-size": 285 | Stat("gauge", "primaries.id_cache.memory_size_in_bytes"), 286 | 287 | # FLUSH 288 | "indices[index={index_name}].primaries.flush.total": 289 | Stat("counter", "primaries.flush.total"), 290 | "indices[index={index_name}].primaries.flush.total-time": 291 | Stat("counter", "primaries.flush.total_time_in_millis"), 292 | 293 | # WARMER 294 | "indices[index={index_name}].primaries.warmer.total.primaries.warmer" 295 | ".total-time": Stat( 296 | "counter", "primaries.warmer.total_time_in_millis"), 297 | "indices[index={index_name}].primaries.warmer.total": 298 | Stat("counter", "primaries.warmer.total"), 299 | "indices[index={index_name}].primaries.warmer.current": 300 | Stat("gauge", "primaries.warmer.current"), 301 | 302 | # FIELDDATA 303 | "indices[index={index_name}].primaries.fielddata.memory-size": Stat( 304 | "gauge", 305 | "primaries.fielddata.memory_size_in_bytes"), 306 | "indices[index={index_name}].primaries.fielddata.evictions": Stat( 307 | "counter", 308 | "primaries.fielddata.evictions"), 309 | 310 | # REFRESH 311 | "indices[index={index_name}].primaries.refresh.total-time": 312 | Stat("counter", "primaries.refresh.total_time_in_millis"), 313 | "indices[index={index_name}].primaries.refresh.total": 314 | Stat("counter", "primaries.refresh.total"), 315 | 316 | # MERGES 317 | "indices[index={index_name}].primaries.merges.total-docs": 318 | Stat("counter", "primaries.merges.total_docs"), 319 | "indices[index={index_name}].primaries.merges.total-size": 320 | Stat("bytes", "primaries.merges.total_size_in_bytes"), 321 | "indices[index={index_name}].primaries.merges.current": 322 | Stat("gauge", "primaries.merges.current"), 323 | "indices[index={index_name}].primaries.merges.total": 324 | Stat("counter", "primaries.merges.total"), 325 | "indices[index={index_name}].primaries.merges.current-docs": 326 | Stat("gauge", "primaries.merges.current_docs"), 327 | "indices[index={index_name}].primaries.merges.total-time": 328 | Stat("counter", "primaries.merges.total_time_in_millis"), 329 | "indices[index={index_name}].primaries.merges.current-size": 330 | Stat("gauge", "primaries.merges.current_size_in_bytes"), 331 | 332 | # COMPELTION 333 | "indices[index={index_name}].primaries.completion.size": 334 | Stat("gauge", "primaries.completion.size_in_bytes"), 335 | 336 | # PERCOLATE 337 | "indices[index={index_name}].primaries.percolate.total": 338 | Stat("counter", "primaries.percolate.total"), 339 | "indices[index={index_name}].primaries.percolate.memory-size": 340 | Stat("gauge", "primaries.percolate.memory_size_in_bytes"), 341 | "indices[index={index_name}].primaries.percolate.queries": 342 | Stat("counter", "primaries.percolate.queries"), 343 | "indices[index={index_name}].primaries.percolate.time": 344 | Stat("counter", "primaries.percolate.time_in_millis"), 345 | "indices[index={index_name}].primaries.percolate.current": 346 | Stat("gauge", "primaries.percolate.current"), 347 | 348 | # FILTER_CACHE 349 | "indices[index={index_name}].primaries.filter-cache.evictions": Stat( 350 | "counter", "primaries.filter_cache.evictions"), 351 | "indices[index={index_name}].primaries.filter-cache.memory-size": Stat( 352 | "gauge", "primaries.filter_cache.memory_size_in_bytes"), 353 | 354 | # DOCS 355 | "indices[index={index_name}].primaries.docs.count": 356 | Stat("gauge", "primaries.docs.count"), 357 | "indices[index={index_name}].primaries.docs.deleted": 358 | Stat("gauge", "primaries.docs.deleted"), 359 | 360 | # STORE 361 | "indices[index={index_name}].primaries.store.size": 362 | Stat("gauge", "primaries.store.size_in_bytes"), 363 | "indices[index={index_name}].primaries.store.throttle-time": Stat( 364 | "counter", 365 | "primaries.store.throttle_time_in_millis"), 366 | 367 | # INDEXING 368 | "indices[index={index_name}].primaries.indexing.index-total": Stat( 369 | "counter", "primaries.indexing.index_total"), 370 | "indices[index={index_name}].primaries.indexing.index-time": Stat( 371 | "counter", 372 | "primaries.indexing.index_time_in_millis"), 373 | "indices[index={index_name}].primaries.indexing.index-current": Stat( 374 | "gauge", "primaries.indexing.index_current"), 375 | "indices[index={index_name}].primaries.indexing.delete-total": Stat( 376 | "counter", "primaries.indexing.delete_total"), 377 | "indices[index={index_name}].primaries.indexing.delete-time": Stat( 378 | "counter", "primaries.indexing.delete_time_in_millis"), 379 | "indices[index={index_name}].primaries.indexing.delete-current": Stat( 380 | "gauge", "primaries.indexing.delete_current"), 381 | 382 | # GET 383 | "indices[index={index_name}].primaries.get.time": 384 | Stat("counter", "primaries.get.time_in_millis"), 385 | "indices[index={index_name}].primaries.get.exists-total": 386 | Stat("counter", "primaries.get.exists_total"), 387 | "indices[index={index_name}].primaries.get.exists-time": 388 | Stat("counter", "primaries.get.exists_time_in_millis"), 389 | "indices[index={index_name}].primaries.get.missing-total": 390 | Stat("counter", "primaries.get.missing_total"), 391 | "indices[index={index_name}].primaries.get.missing-time": 392 | Stat("counter", "primaries.get.missing_time_in_millis"), 393 | "indices[index={index_name}].primaries.get.current": 394 | Stat("gauge", "primaries.get.current"), 395 | 396 | # SEARCH 397 | "indices[index={index_name}].primaries.search.open-contexts": 398 | Stat("gauge", "primaries.search.open_contexts"), 399 | "indices[index={index_name}].primaries.search.query-total": 400 | Stat("counter", "primaries.search.query_total"), 401 | "indices[index={index_name}].primaries.search.query-time": 402 | Stat("counter", "primaries.search.query_time_in_millis"), 403 | "indices[index={index_name}].primaries.search.query-current": 404 | Stat("gauge", "primaries.search.query_current"), 405 | "indices[index={index_name}].primaries.search.fetch-total": 406 | Stat("counter", "primaries.search.fetch_total"), 407 | "indices[index={index_name}].primaries.search.fetch-time": 408 | Stat("counter", "primaries.search.fetch_time_in_millis"), 409 | "indices[index={index_name}].primaries.search.fetch-current": 410 | Stat("gauge", "primaries.search.fetch_current"), 411 | 412 | # TOTAL # 413 | # DOCS 414 | "indices[index={index_name}].total.docs.count": Stat("gauge", 415 | "total.docs.count"), 416 | "indices[index={index_name}].total.docs.deleted": 417 | Stat("gauge", "total.docs.deleted"), 418 | 419 | # STORE 420 | "indices[index={index_name}].total.store.size": 421 | Stat("gauge", "total.store.size_in_bytes"), 422 | "indices[index={index_name}].total.store.throttle-time": 423 | Stat("counter", "total.store.throttle_time_in_millis"), 424 | 425 | # INDEXING 426 | "indices[index={index_name}].total.indexing.index-total": 427 | Stat("counter", "total.indexing.index_total"), 428 | "indices[index={index_name}].total.indexing.index-time": 429 | Stat("counter", "total.indexing.index_time_in_millis"), 430 | "indices[index={index_name}].total.indexing.index-current": 431 | Stat("gauge", "total.indexing.index_current"), 432 | "indices[index={index_name}].total.indexing.delete-total": 433 | Stat("counter", "total.indexing.delete_total"), 434 | "indices[index={index_name}].total.indexing.delete-time": 435 | Stat("counter", "total.indexing.delete_time_in_millis"), 436 | "indices[index={index_name}].total.indexing.delete-current": 437 | Stat("gauge", "total.indexing.delete_current"), 438 | 439 | # GET 440 | "indices[index={index_name}].total.get.total": Stat("counter", 441 | "total.get.total"), 442 | "indices[index={index_name}].total.get.time": 443 | Stat("counter", "total.get.time_in_millis"), 444 | "indices[index={index_name}].total.get.exists-total": 445 | Stat("counter", "total.get.exists_total"), 446 | "indices[index={index_name}].total.get.exists-time": 447 | Stat("counter", "total.get.exists_time_in_millis"), 448 | "indices[index={index_name}].total.get.missing-total": 449 | Stat("counter", "total.get.missing_total"), 450 | "indices[index={index_name}].total.get.missing-time": 451 | Stat("counter", "total.get.missing_time_in_millis"), 452 | "indices[index={index_name}].total.get.current": Stat("gauge", 453 | "total.get.current"), 454 | 455 | # SEARCH 456 | "indices[index={index_name}].total.search.open-contexts": 457 | Stat("gauge", "total.search.open_contexts"), 458 | "indices[index={index_name}].total.search.query-total": 459 | Stat("counter", "total.search.query_total"), 460 | "indices[index={index_name}].total.search.query-time": 461 | Stat("counter", "total.search.query_time_in_millis"), 462 | "indices[index={index_name}].total.search.query-current": 463 | Stat("gauge", "total.search.query_current"), 464 | "indices[index={index_name}].total.search.fetch-total": 465 | Stat("counter", "total.search.fetch_total"), 466 | 467 | # MERGES 468 | "indices[index={index_name}].total.merges.total-docs": 469 | Stat("counter", "total.merges.total_docs"), 470 | "indices[index={index_name}].total.merges.total-size": 471 | Stat("bytes", "total.merges.total_size_in_bytes"), 472 | "indices[index={index_name}].total.merges.current": 473 | Stat("gauge", "total.merges.current"), 474 | "indices[index={index_name}].total.merges.total": 475 | Stat("counter", "total.merges.total"), 476 | "indices[index={index_name}].total.merges.current-docs": 477 | Stat("gauge", "total.merges.current_docs"), 478 | "indices[index={index_name}].total.merges.total-time": 479 | Stat("counter", "total.merges.total_time_in_millis"), 480 | "indices[index={index_name}].total.merges.current-size": 481 | Stat("gauge", "total.merges.current_size_in_bytes"), 482 | 483 | # FILTER_CACHE 484 | "indices[index={index_name}].total.filter-cache.evictions": 485 | Stat("counter", "total.filter_cache.evictions"), 486 | "indices[index={index_name}].total.filter-cache.memory-size": 487 | Stat("gauge", "total.filter_cache.memory_size_in_bytes"), 488 | 489 | # FIELDDATA 490 | "indices[index={index_name}].total.fielddata.memory-size": 491 | Stat("gauge", "total.fielddata.memory_size_in_bytes"), 492 | "indices[index={index_name}].total.fielddata.evictions": 493 | Stat("counter", "total.fielddata.evictions"), 494 | 495 | } 496 | 497 | # ElasticSearch cluster stats (1.0.0 and later) 498 | CLUSTER_STATS = { 499 | 'cluster.active-primary-shards': Stat("gauge", "active_primary_shards"), 500 | 'cluster.active-shards': Stat("gauge", "active_shards"), 501 | 'cluster.initializing-shards': Stat("gauge", "initializing_shards"), 502 | 'cluster.number-of-data_nodes': Stat("gauge", "number_of_data_nodes"), 503 | 'cluster.number-of-nodes': Stat("gauge", "number_of_nodes"), 504 | 'cluster.relocating-shards': Stat("gauge", "relocating_shards"), 505 | 'cluster.unassigned-shards': Stat("gauge", "unassigned_shards"), 506 | 'cluster.status': Stat("gauge", "status"), 507 | } 508 | 509 | 510 | # collectd callbacks 511 | def read_callback(): 512 | """called by collectd to gather stats. It is called per collection 513 | interval. 514 | If this method throws, the plugin will be skipped for an increasing amount 515 | of time until it returns normally again""" 516 | log_verbose('Read callback called') 517 | fetch_stats() 518 | 519 | 520 | def configure_callback(conf): 521 | """called by collectd to configure the plugin. This is called only once""" 522 | global ES_HOST, ES_PORT, ES_NODE_URL, ES_HTTP_TLS_ENABLED, ES_TLS_CERT_PATH, \ 523 | ES_TLS_KEY_PATH, ES_VERSION, VERBOSE_LOGGING, ES_CLUSTER, \ 524 | ES_INDEX, ENABLE_INDEX_STATS, ENABLE_CLUSTER_STATS, COLLECTION_INTERVAL 525 | for node in conf.children: 526 | if node.key == 'Host': 527 | ES_HOST = node.values[0] 528 | elif node.key == 'Port': 529 | ES_PORT = int(node.values[0]) 530 | elif node.key == 'Verbose': 531 | VERBOSE_LOGGING = bool(node.values[0]) 532 | elif node.key == 'Cluster': 533 | ES_CLUSTER = node.values[0] 534 | elif node.key == 'Version': 535 | ES_VERSION = node.values[0] 536 | collectd.info( 537 | "overriding elasticsearch version number to %s" % ES_VERSION) 538 | elif node.key == 'Indexes': 539 | ES_INDEX = node.values 540 | elif node.key == 'EnableIndexStats': 541 | ENABLE_INDEX_STATS = bool(node.values[0]) 542 | elif node.key == 'EnableClusterHealth': 543 | ENABLE_CLUSTER_STATS = bool(node.values[0]) 544 | elif node.key == 'Interval': 545 | COLLECTION_INTERVAL = int(node.values[0]) 546 | elif node.key == 'HttpTlsEnabled': 547 | if bool(node.values[0]): 548 | ES_HTTP_TLS_ENABLED = True 549 | elif node.key == 'TlsCertFile': 550 | ES_TLS_CERT_PATH = node.values[0] 551 | elif node.key == 'TlsKeyFile': 552 | ES_TLS_KEY_PATH = node.values[0] 553 | else: 554 | collectd.warning('elasticsearch plugin: Unknown config key: %s.' 555 | % node.key) 556 | 557 | # determine current ES version 558 | load_es_version() 559 | 560 | # intialize stats map based on ES version 561 | init_stats() 562 | 563 | # register the read callback now that we have the complete config 564 | collectd.register_read(read_callback, interval=COLLECTION_INTERVAL) 565 | collectd.info( 566 | "started elasticsearch plugin with interval = %d seconds" % 567 | COLLECTION_INTERVAL) 568 | 569 | 570 | # helper methods 571 | def init_stats(): 572 | global ES_HOST, ES_PORT, ES_NODE_URL, ES_URL_SCHEME, ES_CLUSTER_URL, ES_INDEX_URL, \ 573 | ES_VERSION, VERBOSE_LOGGING, NODE_STATS_CUR, INDEX_STATS_CUR, \ 574 | CLUSTER_STATS_CUR, ENABLE_INDEX_STATS, ENABLE_CLUSTER_STATS 575 | ES_NODE_URL = ElasticsearchRequestUrl("/_nodes/_local/stats/transport,http,process,jvm,indices,thread_pool") 576 | NODE_STATS_CUR = dict(NODE_STATS.items()) 577 | INDEX_STATS_CUR = dict(INDEX_STATS.items()) 578 | if ES_VERSION.startswith("2."): 579 | NODE_STATS_CUR.update(NODE_STATS_ES_2) 580 | 581 | if ES_VERSION.startswith("1.1") or ES_VERSION.startswith("1.2"): 582 | INDEX_STATS_CUR.update(INDEX_STATS_ES_1_1) 583 | else: 584 | # 1.3 and higher 585 | INDEX_STATS_CUR.update(INDEX_STATS_ES_1_1) 586 | INDEX_STATS_CUR.update(INDEX_STATS_ES_1_3) 587 | 588 | # version agnostic settings 589 | if not ES_INDEX: 590 | # get all index stats 591 | ES_INDEX_URL = ElasticsearchRequestUrl("/_all/_stats") 592 | else: 593 | url_suffix = "/" + ",".join(ES_INDEX) + "/_stats" 594 | ES_INDEX_URL = ElasticsearchRequestUrl(url_suffix) 595 | 596 | #common thread pools for all ES versions 597 | thread_pools = ['generic', 'index', 'get', 'snapshot', 'bulk', 'warmer', 'flush', 'search', 'refresh'] 598 | 599 | if ES_VERSION.startswith("2."): 600 | thread_pools.extend(['suggest', 'percolate', 'management', 'listener', 'force_merge', 601 | 'fetch_shard_store', 'fetch_shard_started']) 602 | else: 603 | thread_pools.extend(['merge', 'optimize']) 604 | 605 | # add info on thread pools (applicable for all versions) 606 | for pool in thread_pools: 607 | for attr in ['threads', 'queue', 'active', 'largest']: 608 | path = 'thread_pool.{0}.{1}'.format(pool, attr) 609 | NODE_STATS_CUR[path] = Stat("gauge", 'nodes.%s.{0}'.format(path)) 610 | for attr in ['completed', 'rejected']: 611 | path = 'thread_pool.{0}.{1}'.format(pool, attr) 612 | NODE_STATS_CUR[path] = Stat("counter", 'nodes.%s.{0}'.format(path)) 613 | 614 | ES_CLUSTER_URL = ElasticsearchRequestUrl("/_cluster/health") 615 | 616 | log_verbose('Configured with version=%s, host=%s, port=%s, url=%s' % 617 | (ES_VERSION, ES_HOST, ES_PORT, ES_NODE_URL)) 618 | 619 | 620 | # FUNCTION: Collect node stats from JSON result 621 | def lookup_node_stat(stat, json): 622 | node = json['nodes'].keys()[0] 623 | val = dig_it_up(json, NODE_STATS_CUR[stat].path % node) 624 | 625 | # Check to make sure we have a valid result 626 | # dig_it_up returns False if no match found 627 | if not isinstance(val, bool): 628 | return int(val) 629 | else: 630 | return None 631 | 632 | 633 | def fetch_stats(): 634 | """ 635 | fetches all required stats from ElasticSearch. This method also sets 636 | ES_CLUSTER 637 | """ 638 | global ES_CLUSTER 639 | 640 | dispatch_is_http_tls_enabled() 641 | 642 | node_json_stats = fetch_url(ES_NODE_URL) 643 | if node_json_stats: 644 | ES_CLUSTER = node_json_stats['cluster_name'] 645 | log_verbose('Configured with cluster_json_stats=%s' % ES_CLUSTER) 646 | parse_node_stats(node_json_stats, NODE_STATS_CUR) 647 | 648 | if ENABLE_CLUSTER_STATS: 649 | cluster_json_stats = fetch_url(ES_CLUSTER_URL) 650 | parse_cluster_stats(cluster_json_stats, CLUSTER_STATS) 651 | 652 | if ENABLE_INDEX_STATS: 653 | indicies = fetch_url(ES_INDEX_URL) 654 | if indicies: 655 | indexes_json_stats = indicies['indices'] 656 | for index_name in indexes_json_stats.keys(): 657 | parse_index_stats(indexes_json_stats[index_name], index_name) 658 | 659 | 660 | # SSL requests wrapper 661 | class HTTPSClientAuthHandler(urllib2.HTTPSHandler): 662 | def __init__(self, key_file, cert_file): 663 | urllib2.HTTPSHandler.__init__(self) 664 | self.key_file = key_file 665 | self.cert_file = cert_file 666 | 667 | def https_open(self, req): 668 | return self.do_open(self.get_connection, req) 669 | 670 | def get_connection(self, host, timeout=10): 671 | return httplib.HTTPSConnection(host, key_file=self.key_file, cert_file=self.cert_file, timeout=timeout) 672 | 673 | 674 | def fetch_url(es_url): 675 | # Attempts to fetch both encrypted endpoint and unencrypted, regardless of flag 676 | # The config flag for http auth is not always indicative of the actual state of the auth plugin on the node 677 | 678 | tls_response = None 679 | non_tls_response = None 680 | 681 | try: 682 | opener = urllib2.build_opener(HTTPSClientAuthHandler(ES_TLS_KEY_PATH, ES_TLS_CERT_PATH)) 683 | tls_response = opener.open(es_url.get_auth_url(), timeout=10) 684 | return json.load(tls_response) 685 | except urllib2.URLError, tls_request_error: 686 | # If the https request failed, catch the error but store for future logging 687 | try: 688 | non_tls_response = urllib2.urlopen(es_url.get_non_auth_url(), timeout=10) 689 | return json.load(non_tls_response) 690 | except urllib2.URLError, e: 691 | if tls_request_error is not None: 692 | collectd.error( 693 | 'elasticsearch plugin, error connecting to tls url: %s - %r, error connecting to non-tls url: %s - %r' 694 | % (es_url.get_auth_url(), tls_request_error, es_url.get_non_auth_url(), e)) 695 | collectd.error('elasticsearch plugin: Error connecting to %s - %r' % (es_url.get_non_auth_url(), e)) 696 | return None 697 | finally: 698 | if non_tls_response is not None: 699 | non_tls_response.close() 700 | finally: 701 | if tls_response is not None: 702 | tls_response.close() 703 | 704 | 705 | def load_es_version(): 706 | global ES_VERSION 707 | if ES_VERSION is None: 708 | json = fetch_url(ElasticsearchRequestUrl('')) 709 | if json is None: 710 | ES_VERSION = "1.0.0" 711 | collectd.warning("elasticsearch plugin: unable to determine " + 712 | "version, defaulting to %s" % ES_VERSION) 713 | else: 714 | ES_VERSION = json['version']['number'] 715 | collectd.info( 716 | "elasticsearch plugin: elasticsearch version is %s" % 717 | ES_VERSION) 718 | 719 | return ES_VERSION 720 | 721 | 722 | def dispatch_is_http_tls_enabled(): 723 | tls_response = None 724 | is_tls_enabled = 0 725 | es_url = ElasticsearchRequestUrl('/_cluster/health') 726 | 727 | try: 728 | opener = urllib2.build_opener(HTTPSClientAuthHandler(ES_TLS_KEY_PATH, ES_TLS_CERT_PATH)) 729 | tls_response = opener.open(es_url.get_auth_url(), timeout=10) 730 | is_tls_enabled = 1 731 | except urllib2.URLError, e: 732 | # If the https request failed, it means non-tls is enabled so we can ignore 733 | pass 734 | finally: 735 | if tls_response is not None: 736 | tls_response.close() 737 | 738 | # Custom stat to measure whether HTTP auth is enabled. Will be set based on success of http request 739 | http_tls_stat = Stat("gauge", "nodes.%s.http.auth.enabled") 740 | dispatch_stat(is_tls_enabled, 'node.http.auth.enabled', http_tls_stat) 741 | 742 | 743 | def parse_node_stats(json, stats): 744 | """Parse node stats response from ElasticSearch""" 745 | for name, key in stats.iteritems(): 746 | result = lookup_node_stat(name, json) 747 | dispatch_stat(result, name, key) 748 | 749 | 750 | def parse_cluster_stats(json, stats): 751 | """Parse cluster stats response from ElasticSearch""" 752 | # convert the status color into a number 753 | json['status'] = CLUSTER_STATUS[json['status']] 754 | for name, key in stats.iteritems(): 755 | result = dig_it_up(json, key.path) 756 | dispatch_stat(result, name, key) 757 | 758 | 759 | def parse_index_stats(json, index_name): 760 | """Parse index stats response from ElasticSearch""" 761 | for name, key in INDEX_STATS_CUR.iteritems(): 762 | result = dig_it_up(json, key.path) 763 | # update the index name in the type_instance to include 764 | # the index as a dimensions 765 | name = name.format(index_name=sanitize_type_instance(index_name)) 766 | dispatch_stat(result, name, key) 767 | 768 | 769 | def sanitize_type_instance(index_name): 770 | """ 771 | collectd limit the character set in type_instance to ascii and forbids 772 | the '/' character. This method does a lossy conversion to ascii and 773 | replaces the reserved character with '_' 774 | """ 775 | ascii_index_name = index_name.encode('ascii', 'ignore') 776 | # '/' is reserved, so we substitute it with '_' instead 777 | return ascii_index_name.replace('/', '_') 778 | 779 | 780 | def dispatch_stat(result, name, key): 781 | """Read a key from info response data and dispatch a value""" 782 | if result is None: 783 | collectd.warning('elasticsearch plugin: Value not found for %s' % name) 784 | return 785 | estype = key.type 786 | value = int(result) 787 | log_verbose('Sending value[%s]: %s=%s' % (estype, name, value)) 788 | 789 | val = collectd.Values(plugin='elasticsearch') 790 | val.plugin_instance = ES_CLUSTER 791 | val.type = estype 792 | val.type_instance = name 793 | val.values = [value] 794 | val.meta = {'0': True} 795 | val.dispatch() 796 | 797 | 798 | def dig_it_up(obj, path): 799 | try: 800 | if type(path) in (str, unicode): 801 | path = path.split('.') 802 | return reduce(lambda x, y: x[y], path, obj) 803 | except: 804 | return False 805 | 806 | 807 | def log_verbose(msg): 808 | if not VERBOSE_LOGGING: 809 | return 810 | collectd.info('elasticsearch plugin [verbose]: %s' % msg) 811 | 812 | 813 | # The following classes are there to launch the plugin manually 814 | # with something like ./elasticsearch_collectd.py for development 815 | # purposes. They basically mock the calls on the "collectd" symbol 816 | # so everything prints to stdout. 817 | class CollectdMock(object): 818 | def __init__(self): 819 | self.value_mock = CollectdValuesMock 820 | 821 | def info(self, msg): 822 | print 'INFO: {}'.format(msg) 823 | 824 | def warning(self, msg): 825 | print 'WARN: {}'.format(msg) 826 | 827 | def error(self, msg): 828 | print 'ERROR: {}'.format(msg) 829 | sys.exit(1) 830 | 831 | def Values(self, plugin='elasticsearch'): 832 | return (self.value_mock)() 833 | 834 | 835 | class CollectdValuesMock(object): 836 | def dispatch(self): 837 | print self 838 | 839 | def __str__(self): 840 | attrs = [] 841 | for name in dir(self): 842 | if not name.startswith('_') and name is not 'dispatch': 843 | attrs.append("{}={}".format(name, getattr(self, name))) 844 | return "".format(' '.join(attrs)) 845 | 846 | 847 | if __name__ == '__main__': 848 | import sys 849 | 850 | collectd = CollectdMock() 851 | load_es_version() 852 | init_stats() 853 | fetch_stats() 854 | else: 855 | import collectd 856 | 857 | collectd.register_config(configure_callback) 858 | --------------------------------------------------------------------------------