├── LICENSE ├── README.md └── configuration ├── cdh-hdfs-balancer.yaml ├── cdh-hdfs-datanode.yaml ├── cdh-hdfs-failover-controller.yaml ├── cdh-hdfs-httpfs.yaml ├── cdh-hdfs-journalnode.yaml ├── cdh-hdfs-namenode.yaml ├── cdh-hdfs-nfs-gateway.yaml ├── cdh-hdfs-secondarynamenode.yaml ├── cdh-hive-hiveserver2.yaml ├── cdh-hive-metastore-server.yaml ├── cdh-hive-webhcat-server.yaml ├── cdh-hue-kerberos-ticket-renewer.yaml ├── cdh-hue-loadbalancer.yaml ├── cdh-hue-server.yaml ├── cdh-kafka-broker.yaml ├── cdh-kafka-mirrormaker.yaml ├── cdh-oozie-server.yaml ├── cdh-sentry-server.yaml ├── cdh-solr-server.yaml ├── cdh-spark-history-server.yaml ├── cdh-spark2-history-server.yaml ├── cdh-yarn-jobhistory-server.yaml ├── cdh-yarn-nodemanager.yaml ├── cdh-yarn-resourcemanager.yaml └── cdh-zookeeper-server.yaml /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 Alexander Fahlke 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # jmx_exporter-cloudera-hadoop 2 | 3 | Prometheus jmx_exporter configurations for Cloudera Hadoop. 4 | 5 | ## Cloudera Hadoop Metrics 6 | 7 | For Cloudera Hadoop I enabled jmxremote for each service via the extra java options and used jmxterm to get all beans I needed. Don't forget to remove the jmxremote settings after you are finished. 8 | 9 | ## Exporter ports and configuration files 10 | 11 | The table is sorted alphabetically by service name. 12 | 13 | | Service | Role | Impl. progress | JMX port | Exporter port | Configuration file | 14 | |:----------|:------------------------|:---------------|:----------|:--------------|:--------------------------------------------| 15 | | HDFS | NameNode | ✓ | 18000 | 19000 | [cdh-hdfs-namenode.yaml][1] | 16 | | HDFS | SecondaryNameNode | ✓ | 18001 | 19001 | [cdh-hdfs-secondarynamenode.yaml][2] | 17 | | HDFS | JournalNode | 0% | 18002 | 19002 | [cdh-hdfs-journalnode.yaml][3] | 18 | | HDFS | Failover Controller | 0% | 18003 | 19003 | [cdh-hdfs-failover-controller.yaml][4] | 19 | | HDFS | HttpFS | 0% | 18004 | 19004 | [cdh-hdfs-httpfs.yaml][5] | 20 | | HDFS | Balancer | 0% | 18005 | 19005 | [cdh-hdfs-balancer.yaml][6] | 21 | | HDFS | NFS Gateway | 0% | 18006 | 19006 | [cdh-hdfs-nfs-gateway.yaml][7] | 22 | | HDFS | DataNode | ✓ | 18007 | 19007 | [cdh-hdfs-datanode.yaml][8] | 23 | | Hive | HiveServer2 | 0% | 18020 | 19020 | [cdh-hive-hiveserver2.yaml][9] | 24 | | Hive | Hive Metastore Server | 0% | 18021 | 19021 | [cdh-hive-metastore-server.yaml][10] | 25 | | Hive | WebHCat Server | 0% | 18022 | 19022 | [cdh-hive-webhcat-server.yaml][11] | 26 | | Hue | Hue Server | 0% | 18040 | 19040 | [cdh-hue-server.yaml][12] | 27 | | Hue | Kerberos Ticket Renewer | 0% | 18041 | 19041 | [cdh-hue-kerberos-ticket-renewer.yaml][13] | 28 | | Hue | Loadbalancer | 0% | 18042 | 19042 | [cdh-hue-loadbalancer.yaml][14] | 29 | | Kafka | Kafka Broker | 0% | 18090 | 19090 | [cdh-kafka-broker.yaml][15] | 30 | | Kafka | Kafka MirrorMaker | 0% | 18091 | 19091 | [cdh-kafka-mirrormaker.yaml][16] | 31 | | Oozie | Oozie Server | 0% | 18050 | 19050 | [cdh-oozie-server.yaml][17] | 32 | | Sentry | Sentry Server | 0% | 18060 | 19060 | [cdh-sentry-server.yaml][18] | 33 | | Solr | Solr Server | 0% | 18100 | 19100 | [cdh-solr-server.yaml][19] | 34 | | Spark | History Server | 0% | 18070 | 19070 | [cdh-spark-history-server.yaml][20] | 35 | | Spark 2 | History Server | 0% | 18080 | 19080 | [cdh-spark2-history-server.yaml][21] | 36 | | YARN | ResourceManager | 0% | 18010 | 19010 | [cdh-yarn-resourcemanager.yaml][22] | 37 | | YARN | JobHistory Server | 0% | 18011 | 19011 | [cdh-yarn-jobhistory-server.yaml][23] | 38 | | YARN | NodeManager | 0% | 18012 | 19012 | [cdh-yarn-nodemanager.yaml][24] | 39 | | ZooKeeper | Server | 0% | 18030 | 19030 | [cdh-zookeeper-server.yaml][25] | 40 | 41 | ## JMX configuration for getting MBeans 42 | 43 | ### Enable jmxremote 44 | 45 | Example for configuring the DataNode: 46 | 47 | HADOOP_DATANODE_OPTS= \ 48 | -Dcom.sun.management.jmxremote \ 49 | -Dcom.sun.management.jmxremote.port=18007 \ 50 | -Dcom.sun.management.jmxremote.local.only=true \ 51 | -Dcom.sun.management.jmxremote.authenticate=false \ 52 | -Dcom.sun.management.jmxremote.ssl=false 53 | 54 | ### Get MBeans with jmxterm 55 | 56 | $ java -jar jmxterm.jar --url localhost:18007 57 | $>beans 58 | #domain = Hadoop: 59 | Hadoop:name=DataNodeInfo,service=DataNode 60 | Hadoop:name=UgiMetrics,service=DataNode 61 | Hadoop:name=JvmMetrics,service=DataNode 62 | Hadoop:name=MetricsSystem,service=DataNode,sub=Control 63 | Hadoop:name=MetricsSystem,service=DataNode,sub=Stats 64 | Hadoop:name=RpcActivityForPort50020,service=DataNode 65 | Hadoop:name=RpcDetailedActivityForPort50020,service=DataNode 66 | .... 67 | 68 | [1]: configuration/cdh-hdfs-namenode.yaml 69 | [2]: configuration/cdh-hdfs-secondarynamenode.yaml 70 | [3]: configuration/cdh-hdfs-journalnode.yaml 71 | [4]: configuration/cdh-hdfs-failover-controller.yaml 72 | [5]: configuration/cdh-hdfs-httpfs.yaml 73 | [6]: configuration/cdh-hdfs-balancer.yaml 74 | [7]: configuration/cdh-hdfs-nfs-gateway.yaml 75 | [8]: configuration/cdh-hdfs-datanode.yaml 76 | [9]: configuration/cdh-hive-hiveserver2.yaml 77 | [10]: configuration/cdh-hive-metastore-server.yaml 78 | [11]: configuration/cdh-hive-webhcat-server.yaml 79 | [12]: configuration/cdh-hue-server.yaml 80 | [13]: configuration/cdh-hue-kerberos-ticket-renewer.yaml 81 | [14]: configuration/cdh-hue-loadbalancer.yaml 82 | [15]: configuration/cdh-kafka-broker.yaml 83 | [16]: configuration/cdh-kafka-mirrormaker.yaml 84 | [17]: configuration/cdh-oozie-server.yaml 85 | [18]: configuration/cdh-sentry-server.yaml 86 | [19]: configuration/cdh-solr-server.yaml 87 | [20]: configuration/cdh-spark-history-server.yaml 88 | [21]: configuration/cdh-spark2-history-server.yaml 89 | [22]: configuration/cdh-yarn-resourcemanager.yaml 90 | [23]: configuration/cdh-yarn-jobhistory-server.yaml 91 | [24]: configuration/cdh-yarn-nodemanager.yaml 92 | [25]: configuration/cdh-zookeeper-server.yaml -------------------------------------------------------------------------------- /configuration/cdh-hdfs-balancer.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | startDelaySeconds: 0 3 | ssl: false 4 | lowercaseOutputName: true 5 | lowercaseOutputLabelNames: true 6 | attrNameSnakeCase: true 7 | -------------------------------------------------------------------------------- /configuration/cdh-hdfs-datanode.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | startDelaySeconds: 0 3 | ssl: false 4 | lowercaseOutputName: true 5 | lowercaseOutputLabelNames: true 6 | whitelistObjectNames: 7 | - 'Hadoop:service=DataNode,name=*' 8 | - 'Hadoop:service=DataNode,name=MetricsSystem,sub=*' 9 | blacklistObjectNames: 10 | - 'Hadoop:service=DataNode,name=RpcActivity*' 11 | - 'Hadoop:service=DataNode,name=RpcDetailedActivity*' 12 | - 'Hadoop:service=DataNode,name=UgiMetrics' 13 | rules: 14 | # MetricsSystem 15 | - pattern: 'Hadoop<>(.*): (\d+)' 16 | attrNameSnakeCase: true 17 | name: hadoop_$1_$3 18 | value: $4 19 | labels: 20 | service: HDFS 21 | role: $1 22 | kind: 'MetricsSystem' 23 | sub: $2 24 | type: GAUGE 25 | # FSDatasetState (also extracts the FSDataset ID) 26 | - pattern: 'Hadoop<>(.*): (\d+)' 27 | attrNameSnakeCase: true 28 | name: hadoop_$1_$3 29 | value: $4 30 | labels: 31 | service: HDFS 32 | role: $1 33 | fsdatasetid: $2 34 | kind: 'FSDatasetState' 35 | type: GAUGE 36 | # DataNodeActivity (also extracts hostname and port) 37 | - pattern: 'Hadoop<>(.*): (\d+)' 38 | attrNameSnakeCase: true 39 | name: hadoop_$1_$4 40 | value: $5 41 | labels: 42 | service: HDFS 43 | role: $1 44 | host: $2 45 | port: $3 46 | kind: 'DataNodeActivity' 47 | type: GAUGE 48 | # All other services 49 | - pattern: 'Hadoop<>(.*): (\d+)' 50 | attrNameSnakeCase: true 51 | name: hadoop_$1_$3 52 | value: $4 53 | labels: 54 | service: HDFS 55 | role: $1 56 | kind: $2 57 | type: GAUGE 58 | -------------------------------------------------------------------------------- /configuration/cdh-hdfs-failover-controller.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | startDelaySeconds: 0 3 | ssl: false 4 | lowercaseOutputName: true 5 | lowercaseOutputLabelNames: true 6 | attrNameSnakeCase: true 7 | -------------------------------------------------------------------------------- /configuration/cdh-hdfs-httpfs.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | startDelaySeconds: 0 3 | ssl: false 4 | lowercaseOutputName: true 5 | lowercaseOutputLabelNames: true 6 | attrNameSnakeCase: true 7 | whitelistObjectNames: 8 | - 'Catalina:host=localhost,path=/webhdfs*' 9 | - 'Catalina:host=localhost,type=Deployer*' 10 | - 'Catalina:host=localhost,type=Host*' 11 | blacklistObjectNames: 12 | - 'Catalina:host=localhost,path=/webhdfs,type=Cache' 13 | - 'Catalina:host=localhost,path=/webhdfs,type=Loader' 14 | - 'Catalina:host=localhost,path=/webhdfs,type=Manager' 15 | - 'Catalina:host=localhost,path=/webhdfs,type=WebappClassLoader' 16 | -------------------------------------------------------------------------------- /configuration/cdh-hdfs-journalnode.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | startDelaySeconds: 0 3 | ssl: false 4 | lowercaseOutputName: true 5 | lowercaseOutputLabelNames: true 6 | attrNameSnakeCase: true 7 | -------------------------------------------------------------------------------- /configuration/cdh-hdfs-namenode.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | startDelaySeconds: 0 3 | ssl: false 4 | lowercaseOutputName: true 5 | lowercaseOutputLabelNames: true 6 | whitelistObjectNames: 7 | - 'Hadoop:service=NameNode,name=*' 8 | - 'Hadoop:service=NameNode,name=MetricsSystem,sub=*' 9 | blacklistObjectNames: 10 | - 'Hadoop:service=NameNode,name=RetryCache.NameNodeRetryCache' 11 | - 'Hadoop:service=NameNode,name=RpcActivity*' 12 | - 'Hadoop:service=NameNode,name=RpcDetailedActivity*' 13 | - 'Hadoop:service=NameNode,name=UgiMetrics' 14 | rules: 15 | # MetricsSystem 16 | - pattern: 'Hadoop<>(.*): (\d+)' 17 | attrNameSnakeCase: true 18 | name: hadoop_$1_$3 19 | value: $4 20 | labels: 21 | service: HDFS 22 | role: $1 23 | kind: 'MetricsSystem' 24 | sub: $2 25 | type: GAUGE 26 | # All NameNode infos 27 | - pattern: 'Hadoop<>(.*): (\d+)' 28 | attrNameSnakeCase: true 29 | name: hadoop_$1_$3 30 | value: $4 31 | labels: 32 | service: HDFS 33 | role: $1 34 | kind: $2 35 | type: GAUGE 36 | -------------------------------------------------------------------------------- /configuration/cdh-hdfs-nfs-gateway.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | startDelaySeconds: 0 3 | ssl: false 4 | lowercaseOutputName: true 5 | lowercaseOutputLabelNames: true 6 | attrNameSnakeCase: true 7 | -------------------------------------------------------------------------------- /configuration/cdh-hdfs-secondarynamenode.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | startDelaySeconds: 0 3 | ssl: false 4 | lowercaseOutputName: true 5 | lowercaseOutputLabelNames: true 6 | whitelistObjectNames: 7 | - 'Hadoop:service=SecondaryNameNode,name=*' 8 | - 'Hadoop:service=SecondaryNameNode,name=MetricsSystem,sub=*' 9 | blacklistObjectNames: 10 | - 'Hadoop:service=SecondaryNameNode,name=UgiMetrics' 11 | rules: 12 | # MetricsSystem 13 | - pattern: 'Hadoop<>(.*): (\d+)' 14 | attrNameSnakeCase: true 15 | name: hadoop_$1_$3 16 | value: $4 17 | labels: 18 | service: HDFS 19 | role: $1 20 | kind: 'MetricsSystem' 21 | sub: $2 22 | type: GAUGE 23 | # SecondaryNameNodeInfo 24 | - pattern: 'Hadoop<>(.*): (\d+)' 25 | attrNameSnakeCase: true 26 | name: hadoop_$1_$3 27 | value: $4 28 | labels: 29 | service: HDFS 30 | role: $1 31 | kind: $2 32 | type: GAUGE 33 | -------------------------------------------------------------------------------- /configuration/cdh-hive-hiveserver2.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | startDelaySeconds: 0 3 | ssl: false 4 | lowercaseOutputName: true 5 | lowercaseOutputLabelNames: true 6 | attrNameSnakeCase: true 7 | -------------------------------------------------------------------------------- /configuration/cdh-hive-metastore-server.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | startDelaySeconds: 0 3 | ssl: false 4 | lowercaseOutputName: true 5 | lowercaseOutputLabelNames: true 6 | attrNameSnakeCase: true 7 | -------------------------------------------------------------------------------- /configuration/cdh-hive-webhcat-server.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | startDelaySeconds: 0 3 | ssl: false 4 | lowercaseOutputName: true 5 | lowercaseOutputLabelNames: true 6 | attrNameSnakeCase: true 7 | -------------------------------------------------------------------------------- /configuration/cdh-hue-kerberos-ticket-renewer.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | startDelaySeconds: 0 3 | ssl: false 4 | lowercaseOutputName: true 5 | lowercaseOutputLabelNames: true 6 | attrNameSnakeCase: true 7 | -------------------------------------------------------------------------------- /configuration/cdh-hue-loadbalancer.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | startDelaySeconds: 0 3 | ssl: false 4 | lowercaseOutputName: true 5 | lowercaseOutputLabelNames: true 6 | attrNameSnakeCase: true 7 | -------------------------------------------------------------------------------- /configuration/cdh-hue-server.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | startDelaySeconds: 0 3 | ssl: false 4 | lowercaseOutputName: true 5 | lowercaseOutputLabelNames: true 6 | attrNameSnakeCase: true 7 | -------------------------------------------------------------------------------- /configuration/cdh-kafka-broker.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | startDelaySeconds: 0 3 | ssl: false 4 | lowercaseOutputName: true 5 | lowercaseOutputLabelNames: true 6 | attrNameSnakeCase: true 7 | -------------------------------------------------------------------------------- /configuration/cdh-kafka-mirrormaker.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | startDelaySeconds: 0 3 | ssl: false 4 | lowercaseOutputName: true 5 | lowercaseOutputLabelNames: true 6 | attrNameSnakeCase: true 7 | -------------------------------------------------------------------------------- /configuration/cdh-oozie-server.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | startDelaySeconds: 0 3 | ssl: false 4 | lowercaseOutputName: true 5 | lowercaseOutputLabelNames: true 6 | attrNameSnakeCase: true 7 | -------------------------------------------------------------------------------- /configuration/cdh-sentry-server.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | startDelaySeconds: 0 3 | ssl: false 4 | lowercaseOutputName: true 5 | lowercaseOutputLabelNames: true 6 | attrNameSnakeCase: true 7 | -------------------------------------------------------------------------------- /configuration/cdh-solr-server.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | startDelaySeconds: 0 3 | ssl: false 4 | lowercaseOutputName: true 5 | lowercaseOutputLabelNames: true 6 | attrNameSnakeCase: true 7 | -------------------------------------------------------------------------------- /configuration/cdh-spark-history-server.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | startDelaySeconds: 0 3 | ssl: false 4 | lowercaseOutputName: true 5 | lowercaseOutputLabelNames: true 6 | attrNameSnakeCase: true 7 | -------------------------------------------------------------------------------- /configuration/cdh-spark2-history-server.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | startDelaySeconds: 0 3 | ssl: false 4 | lowercaseOutputName: true 5 | lowercaseOutputLabelNames: true 6 | attrNameSnakeCase: true 7 | -------------------------------------------------------------------------------- /configuration/cdh-yarn-jobhistory-server.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | startDelaySeconds: 0 3 | ssl: false 4 | lowercaseOutputName: true 5 | lowercaseOutputLabelNames: true 6 | attrNameSnakeCase: true 7 | -------------------------------------------------------------------------------- /configuration/cdh-yarn-nodemanager.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | startDelaySeconds: 0 3 | ssl: false 4 | lowercaseOutputName: true 5 | lowercaseOutputLabelNames: true 6 | attrNameSnakeCase: true 7 | -------------------------------------------------------------------------------- /configuration/cdh-yarn-resourcemanager.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | startDelaySeconds: 0 3 | ssl: false 4 | lowercaseOutputName: true 5 | lowercaseOutputLabelNames: true 6 | attrNameSnakeCase: true 7 | -------------------------------------------------------------------------------- /configuration/cdh-zookeeper-server.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | startDelaySeconds: 0 3 | ssl: false 4 | lowercaseOutputName: true 5 | lowercaseOutputLabelNames: true 6 | attrNameSnakeCase: true 7 | --------------------------------------------------------------------------------