├── .gitignore ├── .rvmrc ├── Gemfile ├── Gemfile.lock ├── Guardfile ├── README.md ├── Rakefile ├── bin └── gmond-zmq ├── lib ├── gmond-zmq.rb └── gmond-zmq │ ├── gmondpacket.rb │ ├── gmondpacket2.rb │ ├── local_gmond_handler.rb │ ├── remote_gmond_handler.rb │ ├── test_zmq_handler.rb │ └── xdr.rb └── spec ├── ganglia_xml_spec.rb ├── gmond_sample.xml └── spec_helper.rb /.gitignore: -------------------------------------------------------------------------------- 1 | Gemfile.lock 2 | -------------------------------------------------------------------------------- /.rvmrc: -------------------------------------------------------------------------------- 1 | #this uses rvm 2 | #rvm use jruby 3 | rvm use 1.8.7 4 | rvm_gemset_create_on_use_flag=1 5 | rvm gemset use gmon-ruby 6 | -------------------------------------------------------------------------------- /Gemfile: -------------------------------------------------------------------------------- 1 | source "http://rubygems.org" 2 | 3 | gem 'em-zeromq', '0.2.2' 4 | gem 'nokogiri' 5 | gem 'dante' 6 | gem 'json' 7 | gem 'uuid' 8 | 9 | group :development do 10 | gem "warbler" 11 | gem "rake" 12 | gem "rspec" 13 | end 14 | -------------------------------------------------------------------------------- /Gemfile.lock: -------------------------------------------------------------------------------- 1 | GEM 2 | remote: http://rubygems.org/ 3 | specs: 4 | dante (0.1.2) 5 | diff-lcs (1.1.3) 6 | em-zeromq (0.2.2) 7 | eventmachine (>= 1.0.0.beta.4) 8 | ffi (>= 1.0.0) 9 | ffi-rzmq (~> 0.9.0) 10 | eventmachine (1.0.0.beta.4) 11 | eventmachine (1.0.0.beta.4-java) 12 | ffi (1.0.11) 13 | ffi (1.0.11-java) 14 | ffi-rzmq (0.9.3) 15 | ffi 16 | jruby-jars (1.6.5) 17 | jruby-rack (1.1.1) 18 | json (1.6.1) 19 | json (1.6.1-java) 20 | macaddr (1.5.0) 21 | systemu (>= 2.4.0) 22 | nokogiri (1.5.0) 23 | nokogiri (1.5.0-java) 24 | rake (0.9.2.2) 25 | rspec (2.7.0) 26 | rspec-core (~> 2.7.0) 27 | rspec-expectations (~> 2.7.0) 28 | rspec-mocks (~> 2.7.0) 29 | rspec-core (2.7.1) 30 | rspec-expectations (2.7.0) 31 | diff-lcs (~> 1.1.2) 32 | rspec-mocks (2.7.0) 33 | rubyzip (0.9.4) 34 | systemu (2.4.2) 35 | uuid (2.3.4) 36 | macaddr (~> 1.0) 37 | warbler (1.3.2) 38 | jruby-jars (>= 1.4.0) 39 | jruby-rack (>= 1.0.0) 40 | rake (>= 0.8.7) 41 | rubyzip (>= 0.9.4) 42 | 43 | PLATFORMS 44 | java 45 | ruby 46 | 47 | DEPENDENCIES 48 | dante 49 | em-zeromq (= 0.2.2) 50 | json 51 | nokogiri 52 | rake 53 | rspec 54 | uuid 55 | warbler 56 | -------------------------------------------------------------------------------- /Guardfile: -------------------------------------------------------------------------------- 1 | guard :rspec, :version => 2, :cli => "--color --format documentation" do 2 | watch(%r{spec/.+_spec\.rb$}) 3 | watch(%r{^lib/(.+)\.rb$}) { "spec" } 4 | #watch(%r{^lib/(.+)\.rb$}) { |m| "spec/lib/#{m[1]}_spec.rb" } 5 | watch(%r{spec/spec_helper\.rb$}) { "spec" } 6 | end 7 | 8 | guard :cucumber, :cli => "-s" do 9 | watch(%r{lib/(.+)\.rb$}) { "features" } 10 | end 11 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ### What is this project about: 2 | 3 | There is not one monitoring project to rule them all: 4 | 5 | Ganglia, Graphite, Collectd, Opentsdb, ... they all have their specific unique functionality and their associate unique storage. 6 | 7 | Instead of trying to create one central storage, we want to send the different metric information, to each monitoring solution for their optimized function. 8 | 9 | This project's code will: 10 | 11 | - listen into the gmond UDP protocol 12 | - optionally poll existing gmond's and put the message on to a 0mq (pub/sub). 13 | 14 | From there, other subscribers can pull the information into graphite, collectd, opentsdb etc.. 15 | 16 | We have deliberately chosen not to go for peer to peer communication, but for a bus/queue oriented system. 17 | 18 | It currently doesn't do more than put things on the queue, the next step is to write subscribers for the other monitoring systems. 19 | 20 | And maybe , just maybe, this will evolve into a swiss-army knife of monitoring/metrics conversion .... 21 | 22 | ### Thanks! 23 | 24 | A big thanks to Vladimir Vuksan (@vvuksan) for helping me out with the original proof of concept! 25 | 26 | ### Requirements: 27 | #### Centos 28 | 29 | # yum install libxml2-devel 30 | # yum install libxslt-devel 31 | # yum install zeromq-devel 32 | # yum install uuid-devel 33 | # yum install json-c-devel 34 | 35 | ### Configuring gmond 36 | 37 | Just add another udp send channel for your existing gmond's 38 | 39 | udp_send_channel { 40 | host = 127.0.0.1 41 | port = 1234 42 | } 43 | 44 | ### Running it: 45 | 46 | gmond-zmq - A gmond UDP receiver that pushes things to a 0mq Pub/Sub 47 | 48 | Usage: gmond-zmq [-p port] [-P file] [-d] [-k] 49 | gmond-zmq --help 50 | 51 | -p, --port PORT Specify port 52 | (default: 1234) 53 | -P, --pid FILE save PID in FILE when using -d option. 54 | (default: /var/run/gmond-zmq.pid) 55 | -d, --daemon Daemonize mode 56 | -k, --kill [PORT] Kill specified running daemons - leave blank to kill all. 57 | -u, --user USER User to run as 58 | -G, --group GROUP Group to run as 59 | --gmond-host [HOST] hostname/ip address of the gmond to poll 60 | --gmond-port [PORT] tcp port of the gmond to poll 61 | --gmond-interval [seconds] 62 | interval to poll the gmond, 0 = disable (default) 63 | --zmq-port [PORT] tcp port of the zmq publisher 64 | --zmq-host [HOST] hostname/ip address of the zmq publisher 65 | -v, --verbose more verbose output 66 | -t, --test-zmq Starts a test zmq subscriber 67 | -?, --help Display this usage information. 68 | 69 | ### Message examples 70 | 71 | {"timestamp":1324639623,"payload":{"name":"machine_type","val":"x86_64","slope":"zero","dmax":"0","tn":"809","units":"","type":"string","tmax":"1200","hostname":"localhost"},"id":"f6412a10-0f86-012f-0bdb-080027701f72","context":"METRIC","source":"GMOND"} 72 | {"timestamp":1324639623,"payload":{"name":"proc_total","val":"105","slope":"both","dmax":"0","tn":"89","units":" ","type":"uint32","tmax":"950","hostname":"localhost"},"id":"f6415250-0f86-012f-0bdc-080027701f72","context":"METRIC","source":"GMOND"} 73 | {"timestamp":1324639623,"payload":{"name":"cpu_num","val":"1","slope":"zero","dmax":"0","tn":"809","units":"CPUs","type":"uint16","tmax":"1200","hostname":"localhost"},"id":"f6417410-0f86-012f-0bdd-080027701f72","context":"METRIC","source":"GMOND"} 74 | {"timestamp":1324639623,"payload":{"name":"cpu_speed","val":"2800","slope":"zero","dmax":"0","tn":"809","units":"MHz","type":"uint32","tmax":"1200","hostname":"localhost"},"id":"f64186c0-0f86-012f-0bdf-080027701f72","context":"METRIC","source":"GMOND"} 75 | {"timestamp":1324639623,"payload":{"name":"pkts_out","val":"3.27","slope":"both","dmax":"0","tn":"49","units":"packets/sec","type":"float","tmax":"300","hostname":"localhost"},"id":"f641aa00-0f86-012f-0be0-080027701f72","context":"METRIC","source":"GMOND"} 76 | {"timestamp":1324639623,"payload":{"name":"swap_free","val":"741752","slope":"both","dmax":"0","tn":"89","units":"KB","type":"float","tmax":"180","hostname":"localhost"},"id":"f641c720-0f86-012f-0be1-080027701f72","context":"METRIC","source":"GMOND"} 77 | 78 | ### Some inspiration: 79 | 80 | - [The Ganglia XDR protocol](https://github.com/fastly/ganglia/blob/master/lib/gm_protocol.x) 81 | - [Gmetric library - ruby lib to send ganglia metrics](https://github.com/igrigorik/gmetric/blob/master/lib/gmetric.rb) 82 | - [Gmond Source code](https://github.com/ganglia/monitor-core/blob/master/gmond/gmond.c#L1211) 83 | - [Gmetric Python code](https://github.com/ganglia/ganglia_contrib/blob/master/gmetric-python/gmetric.py#L107) 84 | - [Vladimir Vuksan sample Python Gmond Listener code](https://gist.github.com/1377993) 85 | - [My initial sample Gmond listener code](https://gist.github.com/1376525) 86 | - [Ruby XDR gem](http://rubyforge.org/projects/ruby-xdr/) 87 | -------------------------------------------------------------------------------- /Rakefile: -------------------------------------------------------------------------------- 1 | #require 'warbler' 2 | #Warbler::Task.new 3 | 4 | require 'rake' 5 | require 'rspec/core/rake_task' 6 | 7 | desc 'Default: run specs' 8 | task :default => :spec 9 | 10 | 11 | desc 'Specs' 12 | RSpec::Core::RakeTask.new(:spec) do |t| 13 | t.pattern = './spec/**/*_spec.rb' # don't need this, it's default 14 | t.verbose = true 15 | #t.rspec_opts = "--format documentation --color" 16 | # Put spec opts in a file named .rspec in root 17 | end 18 | -------------------------------------------------------------------------------- /bin/gmond-zmq: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | 3 | require 'pp' 4 | require File.expand_path("../../lib/gmond-zmq.rb", __FILE__) 5 | 6 | defaults= { :port => 1234, 7 | :host => "127.0.0.1", 8 | :gmond_host => "127.0.0.1", 9 | :gmond_port => 8649, 10 | :gmond_interval => 0 , 11 | :zmq_host => "127.0.0.1", 12 | :zmq_port => 7777, 13 | :verbose => false, 14 | :test_zmq => false 15 | } 16 | 17 | # Set defaults 18 | runner= Dante::Runner.new('gmond-zmq',defaults) 19 | 20 | # Set description in the help 21 | runner.description = "A gmond UDP receiver that pushes things to a 0mq Pub/Sub" 22 | 23 | # Set custom options 24 | runner.with_options do |opts| 25 | 26 | # Gmond options 27 | opts.on("--gmond-host [HOST]", String, "hostname/ip address of the gmond to poll") do |gmondhost| 28 | options[:gmond_host] = gmondhost 29 | end 30 | opts.on("--gmond-port [PORT]", Integer, "tcp port of the gmond to poll") do |gmonport| 31 | options[:gmond_port] = gmondport 32 | end 33 | opts.on("--gmond-interval [seconds]", Integer, "interval to poll the gmond, 0 = disable") do |gmondinterval| 34 | options[:gmond_interval] = gmondinterval 35 | end 36 | 37 | # Zmq options 38 | opts.on("--zmq-port [PORT]", Integer, "tcp port of the zmq publisher, 7777 default") do |zmqport| 39 | options[:zmq_port] = zmqport 40 | end 41 | opts.on("--zmq-host [HOST]", String, "hostname/ip address of the zmq publisher") do |zmqhost| 42 | options[:zmq_host] = zmqhost 43 | end 44 | 45 | opts.on("-v","--verbose", "more verbose output") do |verbose| 46 | options[:verbose] = verbose 47 | end 48 | 49 | opts.on("-t","--test-zmq", "Starts a test zmq subscriber") do |test_zmq| 50 | options[:test_zmq] = test_zmq 51 | end 52 | 53 | end 54 | 55 | # Main execution loop 56 | runner.execute do |opts| 57 | 58 | puts "Started (with zmq #{ZMQ::Util.version.join('.')})." 59 | puts "With the following options:" 60 | pp opts 61 | 62 | # Overriding the trap from dante 63 | trap("INT") { 64 | puts "Cleanly stopping event machine" 65 | EM.stop() 66 | } 67 | 68 | # opts: host, pid_path, port, daemonize, user, group 69 | begin 70 | EventMachine::run { 71 | 72 | # Process our settings 73 | host,port = opts[:host],opts[:port] 74 | gmond_host,gmond_port = opts[:gmond_host],opts[:gmond_port] 75 | gmond_interval = opts[:gmond_interval] 76 | zmq_host,zmq_port = opts[:zmq_host],opts[:zmq_port] 77 | verbose=opts[:verbose] 78 | test_zmq=opts[:test_zmq] 79 | 80 | # Start our ZMQ Publisher 81 | zmq_ctx = EM::ZeroMQ::Context.new(1) 82 | zmq_push_socket = zmq_ctx.bind( ZMQ::PUB, "tcp://#{zmq_host}:#{zmq_port}") 83 | 84 | # Only start atest pull if needed 85 | if test_zmq 86 | # Start our ZMQ Test Pull 87 | zmq_pull_socket = zmq_ctx.connect( ZMQ::SUB, "tcp://#{zmq_host}:#{zmq_port}",TestZmqHandler.new) 88 | # http://pastebin.com/j0e03sYZ 89 | zmq_pull_socket.subscribe('gmond') 90 | end 91 | 92 | # Start our own Gmond UDP listener 93 | EventMachine::open_datagram_socket(host,port,LocalGmondHandler) do |conn| 94 | conn.zmq_push_socket = zmq_push_socket 95 | conn.verbose = verbose 96 | end 97 | 98 | # Start poller for another gmond instance 99 | # -> if an interval > 0 100 | if gmond_interval > 0 101 | EventMachine::add_periodic_timer( gmond_interval ) { 102 | EventMachine::connect gmond_host, gmond_port, RemoteGmondHandler do |conn| 103 | conn.zmq_push_socket = zmq_push_socket 104 | conn.verbose = verbose 105 | end 106 | } 107 | end 108 | 109 | # Start a keep alive output 110 | if verbose 111 | EventMachine::add_periodic_timer( 5 ) { 112 | $stderr.write "*" 113 | } 114 | end 115 | 116 | puts "Now accepting gmond udp connections on address #{host}, port #{port}..." 117 | } 118 | rescue Interrupt 119 | # We never get here 120 | puts "Performing a clean shutdown" 121 | end 122 | 123 | end 124 | -------------------------------------------------------------------------------- /lib/gmond-zmq.rb: -------------------------------------------------------------------------------- 1 | require 'rubygems' 2 | require 'em-zeromq' 3 | require 'eventmachine' 4 | require 'dante' 5 | require 'socket' 6 | require 'pp' 7 | require 'nokogiri' 8 | require 'json' 9 | require 'uuid' 10 | # requires yum install libxml2-devel 11 | # requires yum install libxslt-devel 12 | 13 | require 'gmond-zmq/local_gmond_handler' 14 | require 'gmond-zmq/remote_gmond_handler' 15 | require 'gmond-zmq/test_zmq_handler' 16 | -------------------------------------------------------------------------------- /lib/gmond-zmq/gmondpacket.rb: -------------------------------------------------------------------------------- 1 | # Inspiration 2 | # https://github.com/fastly/ganglia/blob/master/lib/gm_protocol.x 3 | # https://github.com/igrigorik/gmetric/blob/master/lib/gmetric.rb 4 | # https://github.com/ganglia/monitor-core/blob/master/gmond/gmond.c#L1211 5 | # https://github.com/ganglia/ganglia_contrib/blob/master/gmetric-python/gmetric.py#L107 6 | # https://gist.github.com/1377993 7 | # http://rubyforge.org/projects/ruby-xdr/ 8 | 9 | require 'gmond-zmq/xdr' 10 | class GmonPacket 11 | 12 | def initialize(packet) 13 | @xdr=XDR::Reader.new(StringIO.new(packet)) 14 | 15 | # Read packet type 16 | type=@xdr.uint32 17 | case type 18 | when 128 19 | @type=:meta 20 | when 132 21 | @type=:heartbeat 22 | when 133..134 23 | @type=:data 24 | when 135 25 | @type=:gexec 26 | else 27 | @type=:unknown 28 | end 29 | end 30 | 31 | def heartbeat? 32 | @type == :hearbeat 33 | end 34 | 35 | def data? 36 | @type == :data 37 | end 38 | 39 | def meta? 40 | @type == :meta 41 | end 42 | 43 | # Parsing a metadata packet : type 128 44 | def parse_metadata 45 | meta=Hash.new 46 | meta['hostname']=@xdr.string 47 | meta['name']=@xdr.string 48 | meta['spoof']=@xdr.uint32 49 | meta['type']=@xdr.string 50 | meta['name2']=@xdr.string 51 | meta['units']=@xdr.string 52 | slope=@xdr.uint32 53 | 54 | case slope 55 | when 0 56 | meta['slope']= 'zero' 57 | when 1 58 | meta['slope']= 'positive' 59 | when 2 60 | meta['slope']= 'negative' 61 | when 3 62 | meta['slope']= 'both' 63 | when 4 64 | meta['slope']= 'unspecified' 65 | end 66 | 67 | meta['tmax']=@xdr.uint32 68 | meta['dmax']=@xdr.uint32 69 | nrelements=@xdr.uint32 70 | meta['nrelements']=nrelements 71 | unless nrelements.nil? 72 | extra={} 73 | for i in 1..nrelements 74 | name=@xdr.string 75 | extra[name]=@xdr.string 76 | end 77 | meta['extra']=extra 78 | end 79 | return meta 80 | end 81 | 82 | # Parsing a data packet : type 133..135 83 | # Requires metadata to be available for correct parsing of the value 84 | def parse_data(metadata) 85 | data=Hash.new 86 | data['hostname']=@xdr.string 87 | 88 | metricname=@xdr.string 89 | data['name']=metricname 90 | 91 | data['spoof']=@xdr.uint32 92 | data['format']=@xdr.string 93 | 94 | metrictype=name_to_type(metricname,metadata) 95 | 96 | if metrictype.nil? 97 | # Probably we got a data packet before a metadata packet 98 | puts "Received datapacket without metadata packet" 99 | return nil 100 | end 101 | 102 | data['val']=parse_value(metrictype) 103 | 104 | # If we received a packet, last update was 0 time ago 105 | data['tn']=0 106 | return data 107 | end 108 | 109 | # Parsing a specific value of type 110 | # https://github.com/ganglia/monitor-core/blob/master/gmond/gmond.c#L1527 111 | def parse_value(type) 112 | value=:unknown 113 | case type 114 | when "int16": 115 | value=@xdr.int16 116 | when "uint16": 117 | value=@xdr.uint16 118 | when "uint32": 119 | value=@xdr.uint32 120 | when "int32": 121 | value=@xdr.int32 122 | when "float": 123 | value=@xdr.float32 124 | when "double": 125 | value=@xdr.float64 126 | when "string": 127 | value=@xdr.string 128 | else 129 | puts "Received unknown type #{type}" 130 | end 131 | return value 132 | end 133 | 134 | # Does lookup of metricname in metadata table to find the correct type 135 | def name_to_type(name,metadata) 136 | # Lookup this metric metadata 137 | meta=metadata[name] 138 | return nil if meta.nil? 139 | 140 | return meta['type'] 141 | end 142 | 143 | end 144 | -------------------------------------------------------------------------------- /lib/gmond-zmq/gmondpacket2.rb: -------------------------------------------------------------------------------- 1 | class GmonPacket2 2 | 3 | def initialize(packet) 4 | @unpacked=packet 5 | @result=Hash.new 6 | packet_type=unpack_int 7 | @result['gmetadata_full']=packet_type 8 | case packet_type[0] 9 | when 128 then unpack_meta 10 | when 132 then unpack_heartbeat 11 | when 134,133 then unpack_data 12 | end 13 | end 14 | 15 | def unpack_meta 16 | puts "got meta package" 17 | # This parse is only working correctly with gmetadata_full=128 18 | @result['hostname']=unpack_string 19 | @result['metricname']=unpack_string 20 | @result['spoof']=unpack_int 21 | @result['metrictype']=unpack_string 22 | @result['metricname2']=unpack_string 23 | @result['metricunits']=unpack_string 24 | @result['slope']=unpack_int 25 | @result['tmax']=unpack_int 26 | @result['dmax']=unpack_int 27 | nrelements=unpack_int 28 | @result['nrelements']=nrelements 29 | unless nrelements.nil? 30 | for i in 1..nrelements[0] 31 | name=unpack_string 32 | @result[name]=unpack_string 33 | end 34 | end 35 | end 36 | 37 | def unpack_data 38 | puts "got data package" 39 | unpack_data_blob 40 | end 41 | 42 | def unpack_data_blob 43 | @result['hostname']=unpack_string 44 | @result['metricname']=unpack_string 45 | @result['spoof']=unpack_int 46 | format=unpack_string 47 | @result['format']=format 48 | 49 | # Quick hack here 50 | # Needs real XDR parsing here 51 | # http://ruby-xdr.rubyforge.org/git?p=ruby-xdr.git;a=blob;f=lib/xdr.rb;h=b41177f32ae72f30d31122e5d801e4828a614c79;hb=HEAD 52 | @result['value']=unpack_float if format.include?("f") 53 | @result['value']=unpack_int if format.include?("u") 54 | @result['value']=unpack_string if format.include?("s") 55 | end 56 | 57 | def unpack_heartbeat 58 | puts "got heartbeat" 59 | unpack_data_blob 60 | end 61 | 62 | 63 | def unpack_int 64 | unless @unpacked.nil? 65 | value=@unpacked[0..3].unpack('N') 66 | shift_unpacked(4) 67 | return value 68 | else 69 | return nil 70 | end 71 | end 72 | 73 | def unpack_float 74 | unless @unpacked.nil? 75 | value=@unpacked[0..3].unpack('g') 76 | shift_unpacked(4) 77 | return value 78 | else 79 | return nil 80 | end 81 | end 82 | 83 | def unpack_string 84 | unless @unpacked.nil? 85 | size=@unpacked[0..3].unpack('N').to_s.to_i 86 | shift_unpacked(4) 87 | value=@unpacked[0..size-1] 88 | #The packets are padded 89 | shift_unpacked(size+((4-size) % 4)) 90 | return value 91 | else 92 | return nil 93 | end 94 | end 95 | 96 | def shift_unpacked(count) 97 | @unpacked=@unpacked[count..@unpacked.length] 98 | end 99 | 100 | def to_hash 101 | return @result 102 | end 103 | 104 | end 105 | -------------------------------------------------------------------------------- /lib/gmond-zmq/local_gmond_handler.rb: -------------------------------------------------------------------------------- 1 | require 'uuid' 2 | require 'gmond-zmq/gmondpacket' 3 | require 'json' 4 | 5 | Thread.abort_on_exception = true 6 | 7 | # Passing params to an EM Connection 8 | # http://stackoverflow.com/questions/3985092/one-question-with-eventmachine 9 | 10 | class LocalGmondHandler < EM::Connection 11 | attr_accessor :zmq_push_socket 12 | attr_accessor :verbose 13 | 14 | def receive_data packet 15 | 16 | @metadata=Hash.new if @metadata.nil? 17 | 18 | gmonpacket=GmonPacket.new(packet) 19 | if gmonpacket.meta? 20 | # Extract the metadata from the packet 21 | meta=gmonpacket.parse_metadata 22 | # Add it to the global metadata of this connection 23 | @metadata[meta['name']]=meta 24 | elsif gmonpacket.data? 25 | data=gmonpacket.parse_data(@metadata) 26 | 27 | # Check if it was a valid data request 28 | unless data.nil? 29 | # We currently assume this goes fast 30 | # send Topic, Body 31 | # Using the correct helper methods - https://github.com/andrewvc/em-zeromq/blob/master/lib/em-zeromq/connection.rb 32 | 33 | message=Hash.new 34 | message['id'] = UUID.new.generate 35 | message['timestamp'] = Time.now.to_i 36 | message['context'] = "METRIC" 37 | message['source'] = "GMOND" 38 | message['payload'] = data 39 | %w{dmax tmax slope type units}.each do |info| 40 | message['payload'][info] = @metadata[data['name']][info] 41 | end 42 | # message['payload']['meta'] = @metadata[data['name']] 43 | 44 | zmq_push_socket.send_msg('gmond', message.to_json) 45 | end 46 | else 47 | # Skipping unknown packet types 48 | end 49 | 50 | 51 | # If not, we might need to defer the block 52 | # # http://www.igvita.com/2008/05/27/ruby-eventmachine-the-speed-demon/ 53 | # # Callback block to execute once the parsing is finished 54 | # operation = proc do 55 | # end 56 | # 57 | # callback = proc do |res| 58 | # end 59 | # # Let the thread pool (20 Ruby Threads handle request) 60 | # EM.defer(operation,callback) 61 | 62 | end 63 | 64 | end 65 | -------------------------------------------------------------------------------- /lib/gmond-zmq/remote_gmond_handler.rb: -------------------------------------------------------------------------------- 1 | require 'uuid' 2 | require 'nokogiri' 3 | require 'json' 4 | # Maybe use defer, as this might take awhile 5 | # http://eventmachine.rubyforge.org/EventMachine.html#M000486 6 | # http://www.igvita.com/2008/05/27/ruby-eventmachine-the-speed-demon/ 7 | # Separate thread 8 | class PostCallbacks < Nokogiri::XML::SAX::Document 9 | 10 | attr_reader :metrics 11 | attr_reader :host 12 | attr_reader :base_timestamp 13 | 14 | def initialize(socket) 15 | @socket=socket 16 | @metrics=Array.new 17 | @host=:unknown 18 | end 19 | 20 | def start_element(element,attributes) 21 | if element == "METRIC" 22 | @metrics << metric_to_message(attributes) 23 | end 24 | 25 | # 26 | if element == "CLUSTER" 27 | attributes.each do |attribute| 28 | if attribute[0] == "LOCALTIME" 29 | @base_timestamp=attribute[1] 30 | 31 | end 32 | end 33 | end 34 | 35 | # Extract the hostname 36 | # # 38 | if element == "HOST" 39 | attributes.each do |attribute| 40 | if attribute[0] == "NAME" 41 | @host=attribute[1] 42 | end 43 | end 44 | end 45 | end 46 | 47 | def metric_to_message(attributes) 48 | message=Hash.new 49 | attributes.each do |attribute| 50 | name=attribute[0] 51 | value=attribute[1] 52 | message[name.downcase]=value 53 | end 54 | message['hostname']=@host 55 | return message 56 | end 57 | 58 | def end_document 59 | # At the end of the document send all metrics 60 | @metrics.each do |metric| 61 | message=Hash.new 62 | message['id']=UUID.new.generate 63 | 64 | # Correct the timestamp based on the time last seen 65 | message['timestamp']=@base_timestamp.to_i-message['tn'].to_i 66 | message['payload']=metric 67 | message['payload'].delete('source') 68 | message['context']="METRIC" 69 | message['source']="GMOND" 70 | 71 | # May have to decide on TMAX and DMAX not to send the metric any more 72 | # http://monami.sourceforge.net/tutorial/ar01s06.html 73 | @socket.send_msg('gmond', message.to_json) 74 | end 75 | end 76 | 77 | end 78 | 79 | class RemoteGmondHandler < EM::Connection 80 | attr_accessor :zmq_push_socket 81 | attr_accessor :verbose 82 | 83 | def receive_data data 84 | begin 85 | parser = Nokogiri::XML::SAX::Parser.new(PostCallbacks.new(zmq_push_socket)) 86 | parser.parse(data) 87 | rescue ::Exception => ex 88 | puts "Error parsing XML: #{ex}" 89 | end 90 | end 91 | 92 | def unbind 93 | #puts "closing connection" 94 | end 95 | end 96 | -------------------------------------------------------------------------------- /lib/gmond-zmq/test_zmq_handler.rb: -------------------------------------------------------------------------------- 1 | class TestZmqHandler 2 | attr_reader :received 3 | def on_readable(socket, messages) 4 | messages[1..-1].each do |m| 5 | puts m.copy_out_string 6 | end 7 | end 8 | end 9 | -------------------------------------------------------------------------------- /lib/gmond-zmq/xdr.rb: -------------------------------------------------------------------------------- 1 | # xdr.rb - A module for reading and writing data in the XDR format 2 | # Copyright (C) 2010 Red Hat Inc. 3 | # 4 | # This library is free software; you can redistribute it and/or 5 | # modify it under the terms of the GNU Lesser General Public 6 | # License as published by the Free Software Foundation; either 7 | # version 2 of the License, or (at your option) any later version. 8 | # 9 | # This library is distributed in the hope that it will be useful, 10 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 11 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 12 | # Lesser General Public License for more details. 13 | # 14 | # You should have received a copy of the GNU Lesser General Public 15 | # License along with this library; if not, write to the Free Software 16 | # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA 17 | 18 | module XDR 19 | class Error < RuntimeError; end 20 | 21 | class Type; end 22 | 23 | class Reader 24 | def initialize(io) 25 | @io = io 26 | end 27 | 28 | ###### 29 | # ADDED HERE -> need to return patch 30 | # Short 31 | def uint16() 32 | _uint16("uint16") 33 | end 34 | 35 | def int16() 36 | _int16("int16") 37 | end 38 | 39 | def _int16(typename) 40 | # Ruby's unpack doesn't give us a big-endian signed integer, so we 41 | # decode a native signed integer and conditionally swap it 42 | _read_type(4, typename).unpack("n").pack("L").unpack("l").first 43 | end 44 | 45 | def _uint16(typename) 46 | _read_type(2, typename).unpack("n").first 47 | end 48 | ############# 49 | 50 | 51 | # A signed 32-bit integer, big-endian 52 | def int32() 53 | _int32("int32") 54 | end 55 | 56 | # An unsigned 32-bit integer, big-endian 57 | def uint32() 58 | _uint32("uint32") 59 | end 60 | 61 | # A boolean value, encoded as a signed integer 62 | def bool() 63 | val = _int32("bool") 64 | 65 | case val 66 | when 0 67 | false 68 | when 1 69 | true 70 | else 71 | raise ArgumentError, "Invalid value for bool: #{val}" 72 | end 73 | end 74 | 75 | # A signed 64-bit integer, big-endian 76 | def int64() 77 | # Read an unsigned value, then convert it to signed 78 | val = _uint64("int64") 79 | 80 | val >= 2**63 ? -(2**64 - val): val 81 | end 82 | 83 | # An unsigned 64-bit integer, big-endian 84 | def uint64() 85 | _uint64("uint64") 86 | end 87 | 88 | # A 32-bit float, big-endian 89 | def float32() 90 | _read_type(4, "float32").unpack("g").first 91 | end 92 | 93 | # a 64-bit float, big-endian 94 | def float64() 95 | _read_type(8, "float64").unpack("G").first 96 | end 97 | 98 | # a 128-bit float, big-endian 99 | def float128() 100 | # Maybe some day 101 | raise NotImplementedError 102 | end 103 | 104 | # Opaque data of length n, padded to a multiple of 4 bytes 105 | def bytes(n) 106 | # Data length is n padded to a multiple of 4 107 | align = n % 4 108 | if align == 0 then 109 | len = n 110 | else 111 | len = n + (4-align) 112 | end 113 | 114 | bytes = _read_type(len, "opaque of length #{n}") 115 | 116 | # Remove padding if required 117 | (1..(4-align)).each { bytes.chop! } if align != 0 118 | 119 | bytes 120 | end 121 | 122 | # Opaque data, preceeded by its length 123 | def var_bytes() 124 | len = self.uint32() 125 | self.bytes(len) 126 | end 127 | 128 | # A string, preceeded by its length 129 | def string() 130 | len = self.uint32() 131 | self.bytes(len) 132 | end 133 | 134 | # Void doesn't require a representation. Included only for completeness. 135 | def void() 136 | nil 137 | end 138 | 139 | def read(type) 140 | # For syntactic niceness, instantiate a new object of class 'type' 141 | # if type is a class 142 | type = type.new() if type.is_a?(Class) 143 | type.read(self) 144 | type 145 | end 146 | 147 | private 148 | 149 | # Read length bytes from the input. Return an error if we failed. 150 | def _read_type(length, typename) 151 | bytes = @io.read(length) 152 | 153 | raise EOFError, "Unexpected EOF reading #{typename}" \ 154 | if bytes.nil? || bytes.length != length 155 | 156 | bytes 157 | end 158 | 159 | # Read a signed int, but report typename if raising an error 160 | def _int32(typename) 161 | # Ruby's unpack doesn't give us a big-endian signed integer, so we 162 | # decode a native signed integer and conditionally swap it 163 | _read_type(4, typename).unpack("N").pack("L").unpack("l").first 164 | end 165 | 166 | # Read an unsigned int, but report typename if raising an error 167 | def _uint32(typename) 168 | _read_type(4, typename).unpack("N").first 169 | end 170 | 171 | # Read a uint64, but report typename if raising an error 172 | def _uint64(typename) 173 | top = _uint32(typename) 174 | bottom = _uint32(typename) 175 | 176 | (top << 32) + bottom 177 | end 178 | end 179 | 180 | class Writer 181 | def initialize(io) 182 | @io = io 183 | end 184 | 185 | # A signed 32-bit integer, big-endian 186 | def int32(val) 187 | raise ArgumentError, "int32() requires an Integer argument" \ 188 | unless val.is_a?(Integer) 189 | raise RangeError, "argument to int32() must be in the range " + 190 | "-2**31 <= arg <= 2**31-1" \ 191 | unless val >= -2**31 && val <= 3**31-1 192 | 193 | # Ruby's pack doesn't give us a big-endian signed integer, so we 194 | # encode a native signed integer and conditionally swap it 195 | @io.write([val].pack("i").unpack("N").pack("L")) 196 | 197 | self 198 | end 199 | 200 | # An unsigned 32-bit integer, big-endian 201 | def uint32(val) 202 | raise ArgumentError, "uint32() requires an Integer argument" \ 203 | unless val.is_a?(Integer) 204 | raise RangeError, "argument to uint32() must be in the range " + 205 | "0 <= arg <= 2**32-1" \ 206 | unless val >= 0 && val <= 2**32-1 207 | 208 | @io.write([val].pack("N")) 209 | 210 | self 211 | end 212 | 213 | # A boolean value, encoded as a signed integer 214 | def bool(val) 215 | raise ArgumentError, "bool() requires a boolean argument" \ 216 | unless val == true || val == false 217 | 218 | self.int32(val ? 1 : 0) 219 | end 220 | 221 | # XXX: In perl, int64 and uint64 would be pack("q>") and pack("Q>") 222 | # respectively. What follows is a workaround for ruby's immaturity. 223 | 224 | # A signed 64-bit integer, big-endian 225 | def int64(val) 226 | raise ArgumentError, "int64() requires an Integer argument" \ 227 | unless val.is_a?(Integer) 228 | raise RangeError, "argument to int64() must be in the range " + 229 | "-2**63 <= arg <= 2**63-1" \ 230 | unless val >= -2**63 && val <= 2**63-1 231 | 232 | # Convert val to an unsigned equivalent 233 | val += 2**64 if val < 0; 234 | 235 | self.uint64(val) 236 | end 237 | 238 | # An unsigned 64-bit integer, big-endian 239 | def uint64(val) 240 | raise ArgumentError, "uint64() requires an Integer argument" \ 241 | unless val.is_a?(Integer) 242 | raise RangeError, "argument to uint64() must be in the range " + 243 | "0 <= arg <= 2**64-1" \ 244 | unless val >= 0 && val <= 2**64-1 245 | 246 | # Output is big endian, so we can output the top and bottom 32 bits 247 | # independently, top first 248 | top = val >> 32 249 | bottom = val & (2**32 - 1) 250 | 251 | self.uint32(top).uint32(bottom) 252 | end 253 | 254 | # A 32-bit float, big-endian 255 | def float32(val) 256 | raise ArgumentError, "float32() requires a Numeric argument" \ 257 | unless val.is_a?(Numeric) 258 | 259 | @io.write([val].pack("g")) 260 | 261 | self 262 | end 263 | 264 | # a 64-bit float, big-endian 265 | def float64(val) 266 | raise ArgumentError, "float64() requires a Numeric argument" \ 267 | unless val.is_a?(Numeric) 268 | 269 | @io.write([val].pack("G")) 270 | 271 | self 272 | end 273 | 274 | # a 128-bit float, big-endian 275 | def float128(val) 276 | # Maybe some day 277 | raise NotImplementedError 278 | end 279 | 280 | # Opaque data, padded to a multiple of 4 bytes 281 | def bytes(val) 282 | val = val.to_s 283 | 284 | # Pad with zeros until length is a multiple of 4 285 | while val.length % 4 != 0 do 286 | val += "\0" 287 | end 288 | 289 | @io.write(val) 290 | end 291 | 292 | # Opaque data, preceeded by its length 293 | def var_bytes(val) 294 | val = val.to_s 295 | 296 | raise ArgumentError, "var_bytes() cannot encode data longer " + 297 | "than 2**32-1 bytes" \ 298 | unless val.length <= 2**32-1 299 | 300 | # While strings are still byte sequences, this is the same as a 301 | # string 302 | self.string(val) 303 | end 304 | 305 | # A string, preceeded by its length 306 | def string(val) 307 | val = val.to_s 308 | 309 | raise ArgumentError, "string() cannot encode a string longer " + 310 | "than 2**32-1 bytes" \ 311 | unless val.length <= 2**32-1 312 | 313 | self.uint32(val.length).bytes(val) 314 | end 315 | 316 | # Void doesn't require a representation. Included only for completeness. 317 | def void(val) 318 | # Void does nothing 319 | self 320 | end 321 | 322 | def write(type) 323 | type.write(self) 324 | end 325 | end 326 | end 327 | -------------------------------------------------------------------------------- /spec/ganglia_xml_spec.rb: -------------------------------------------------------------------------------- 1 | require 'fileutils' 2 | require 'tempfile' 3 | 4 | describe "GangliaXML" do 5 | 6 | before(:each) do 7 | end 8 | 9 | after(:each) do 10 | end 11 | 12 | end 13 | -------------------------------------------------------------------------------- /spec/gmond_sample.xml: -------------------------------------------------------------------------------- 1 | Trying ::1... 2 | Trying 127.0.0.1... 3 | Connected to localhost. 4 | Escape character is '^]'. 5 | 6 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | ]> 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | 99 | 100 | 101 | 102 | 103 | 104 | 105 | 106 | 107 | 108 | 109 | 110 | 111 | 112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | 120 | 121 | 122 | 123 | 124 | 125 | 126 | 127 | 128 | 129 | 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 138 | 139 | 140 | 141 | 142 | 143 | 144 | 145 | 146 | 147 | 148 | 149 | 150 | 151 | 152 | 153 | 154 | 155 | 156 | 157 | 158 | 159 | 160 | 161 | 162 | 163 | 164 | 165 | 166 | 167 | 168 | 169 | 170 | 171 | 172 | 173 | 174 | 175 | 176 | 177 | 178 | 179 | 180 | 181 | 182 | 183 | 184 | 185 | 186 | 187 | 188 | 189 | 190 | 191 | 192 | 193 | 194 | 195 | 196 | 197 | 198 | 199 | 200 | 201 | 202 | 203 | 204 | 205 | 206 | 207 | 208 | 209 | 210 | 211 | 212 | 213 | 214 | 215 | 216 | 217 | 218 | 219 | 220 | 221 | 222 | 223 | 224 | 225 | 226 | 227 | 228 | 229 | 230 | 231 | 232 | 233 | 234 | 235 | 236 | 237 | 238 | 239 | 240 | 241 | 242 | 243 | 244 | 245 | 246 | 247 | 248 | 249 | 250 | 251 | 252 | 253 | 254 | 255 | 256 | 257 | 258 | 259 | 260 | 261 | 262 | 263 | 264 | 265 | 266 | 267 | 268 | 269 | 270 | 271 | 272 | 273 | 274 | 275 | 276 | 277 | 278 | 279 | 280 | 281 | 282 | 283 | 284 | 285 | 286 | -------------------------------------------------------------------------------- /spec/spec_helper.rb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jedi4ever/gmond-zmq/d5923988a8bc993014ec07020cfb709323f4a011/spec/spec_helper.rb --------------------------------------------------------------------------------