├── .gitignore ├── DOCS ├── README.mkd ├── app.wsgi ├── config-sample.yml ├── fullerene ├── __init__.py ├── config.py ├── graph.py ├── graphite.py ├── metric.py ├── templates │ ├── _graph.html │ ├── _metric_groups.html │ ├── base.html │ ├── collection_group.html │ ├── domain.html │ ├── group.html │ ├── host.html │ └── index.html └── utils.py ├── requirements.txt ├── runner.py ├── setup.cfg ├── setup.py └── tests ├── support ├── basic.yml ├── exclusions.yml └── no_url.yml └── test_main.py /.gitignore: -------------------------------------------------------------------------------- 1 | config.yml 2 | -------------------------------------------------------------------------------- /DOCS: -------------------------------------------------------------------------------- 1 | Docstring for expand_metric (now roughly describes Metric.normalize()) 2 | """ 3 | Take metric string or one-key dict and normalize to an iterable as needed. 4 | 5 | Each item in the iterable will be a Graphite metric string mapping to a 6 | single graph. These strings themselves may contain wildcards, etc, as 7 | defined in the Graphite URL API. 8 | 9 | This function's purpose is to read YAML config metadata and optionally 10 | break down a config chunk into >1 Graphite metric string. This allows us to 11 | exclude, filter etc without requiring support for new syntax in Graphite. 12 | 13 | String inputs just come out as [metric] (i.e. normalization to list.) 14 | 15 | Dict inputs should have a single string key; basically, mixed strings and 16 | dicts should come from YAML that looks like this:: 17 | 18 | metrics: 19 | - metric1.foo.bar 20 | - metric2.biz.baz 21 | - metric3.blah.blah: 22 | option: value 23 | option2: value2 24 | - metric4.whatever 25 | 26 | All "items" under ``metrics`` are metric paths; the difference is that they 27 | may optionally have configuration options, which turns that entry into a 28 | dict. 29 | 30 | Dict inputs will also come out as at least [metric] (in this case, the 31 | single dict key) but config options will often cause multiple metrics to be 32 | output, e.g. [metric1, metric2], due to filtering/excluding requiring us to 33 | ask Graphite for the full expansion up-front, and then manipulating that 34 | result. 35 | 36 | Options currently implemented: 37 | 38 | * ``exclude``: a list of explicit matches to exclude from the first 39 | wildcard. E.g. a metric ``df.*.free`` which expands, in Graphite, to 40 | [df.root.free, df.mnt.free, df.dev.free, df.dev-shm.free], may be 41 | filtered to remove some specific matches like so:: 42 | 43 | metrics: 44 | - df.*.free: [dev, dev-shm] 45 | 46 | Such a setup would result in a return value from this function of 47 | [df.root.free, df.mnt.free] given the expansion example above. 48 | 49 | Note that partial wildcards work the same way; the logic operates based 50 | on metric sections (i.e. separated by periods) containing wildcards 51 | (meaning asterisks; curly-brace expansion is not considered a wildcard 52 | here.) 53 | 54 | If multiple wildcards are given, and the value is still just one list, it 55 | will only apply to the first wildcard. To pair specific exclusion lists 56 | with specific wildcard positions, use a dict value instead, with numeric 57 | keys matching the wildcard positions (0-indexed.) E.g.:: 58 | 59 | metrics: 60 | - foo.*.bar.*: 61 | 0: [these, are, excluded, from, 1st, wildcard] 62 | 1: [these, from, the, 2nd] 63 | """ 64 | -------------------------------------------------------------------------------- /README.mkd: -------------------------------------------------------------------------------- 1 | ## What? 2 | 3 | Fullerene is a "meta-dashboard" tool for Graphite, allowing for both automatic 4 | and manually-specified groupings of graphs, with limited host/domain 5 | organization based on an assumed metric path (e.g. those generated by 6 | [collectd-carbon](https://github.com/indygreg/collectd-carbon).) 7 | 8 | ## How? 9 | 10 | It is currently implemented as a simple [Flask](http://flask.pocoo.org) 11 | application designed to run on the same server as one's Graphite webapp and 12 | hook into that app's URL API. 13 | 14 | ## When? 15 | 16 | Fullerene is in very, very early stages of development and is not intended for 17 | public consumption. No, not even for those who routinely nab "pre-alpha" 18 | software. This is only public for now so its code can be referenced in 19 | discussions and because I'm too cheap to upgrade to a Github plan that offers 20 | more private collaborators :) 21 | 22 | ## Why? 23 | 24 | Similar tools at time of writing (October 2011) were 25 | [Pencil](https://github.com/fetep/pencil) and 26 | [GDash](http://www.devco.net/archives/2011/10/08/gdash-graphite-dashboard.php). 27 | Neither of those tools quite fit the bill for our particular use case(s), nor 28 | were they a good fit for forking. 29 | 30 | Pencil had a good feature set but diverged from how I wanted/needed such a tool 31 | to work, to such a degree that a greenfield project felt like less/equal 32 | effort. 33 | 34 | GDash was *too* simple for my needs, and did not have enough existing 35 | functionality to merit forking vs writing my own tool; plus I work in a team 36 | which is primarily used to Python, so using a small amount of Ruby code as a 37 | base did not make sense. 38 | -------------------------------------------------------------------------------- /app.wsgi: -------------------------------------------------------------------------------- 1 | from fullerene import app as application 2 | -------------------------------------------------------------------------------- /config-sample.yml: -------------------------------------------------------------------------------- 1 | graphite_uris: 2 | internal: http://localhost 3 | # external: https://externally.accessible.url.here/ 4 | hosts: 5 | exclude: 6 | - carbon 7 | defaults: 8 | height: 250 9 | width: 400 10 | from: -4hours 11 | periods: 12 | recent: -4hours 13 | day: -24hours 14 | week: -7days 15 | metrics: 16 | free_disk_space: 17 | title: Free Disk Space 18 | path: df.*.df_complex.free.value 19 | exclude: [dev, dev-shm, var-lock, var-run] 20 | expand: all 21 | cpu: 22 | title: CPU usage 23 | # Paths with %s in them skip all expand/exclude mechanisms and instead 24 | # replace %s with the hostname at render time. 25 | path: "group(sumSeries(%s.cpu.*.cpu.system.value),sumSeries(%s.cpu.*.cpu.user.value))" 26 | # Arbitrary graphite render args may be specified as well and are passed 27 | # through 28 | areaMode: stacked 29 | groups: 30 | baseline: 31 | - free_disk_space 32 | - disk.xvda1.disk_octets.{read,write} 33 | - disk.xvdb.disk_octets.{read,write} 34 | - interface.if_octets.eth0.{rx,tx} 35 | - load.load.* 36 | - memory.memory.free.value 37 | disk: 38 | - free_disk_space 39 | - disk.{xvda1,xvdb,xvdf}.disk_octets.{read,write} 40 | cpu: 41 | - load.load.* 42 | network: 43 | - interface.if_octets.eth0.{rx,tx} 44 | collections: 45 | by_project: 46 | title: "By project" 47 | groups: 48 | project1: 49 | - cache10.shared.com 50 | - cache11.shared.com 51 | - chat1.shared.com 52 | - chat2.shared.com 53 | - chat3.shared.com 54 | - db.project1.com 55 | - haproxy1.shared.com 56 | - haproxy2.shared.com 57 | - lb2.shared.com 58 | - lb3.shared.com 59 | - redis10.shared.com 60 | - redis11.shared.com 61 | - search1.shared.com 62 | - search2.shared.com 63 | - web5.shared.com 64 | - web6.shared.com 65 | - web7.shared.com 66 | - web8.shared.com 67 | project2: 68 | - celery2.project2.com 69 | - chat1.shared.com 70 | - chat2.shared.com 71 | - chat3.shared.com 72 | - db.project2.com 73 | - lb2.shared.com 74 | - lb3.shared.com 75 | - redis10.shared.com 76 | - redis11.shared.com 77 | - search1.shared.com 78 | - search2.shared.com 79 | - web1.project2.com 80 | - web10.project2.com 81 | - web2.project2.com 82 | - web3.project2.com 83 | - web4.project2.com 84 | - web8.project2.com 85 | - web9.project2.com 86 | -------------------------------------------------------------------------------- /fullerene/__init__.py: -------------------------------------------------------------------------------- 1 | from collections import defaultdict 2 | import operator 3 | import os 4 | 5 | import requests 6 | import flask 7 | import yaml 8 | 9 | from metric import Metric 10 | from graphite import Graphite 11 | from config import Config 12 | from utils import dots, sliced 13 | 14 | 15 | # 16 | # Set up globals 17 | # 18 | 19 | ROOT = os.path.join(os.path.dirname(__file__), "..") 20 | CONFIG = os.path.join(ROOT, "config.yml") 21 | with open(CONFIG) as fd: 22 | config = Config(fd.read()) 23 | 24 | app = flask.Flask(__name__) 25 | 26 | 27 | # 28 | # Template filters 29 | # 30 | 31 | @app.template_filter('dots') 32 | def dots_(string): 33 | return dots(string) 34 | 35 | @app.template_filter('dot') 36 | def dot_(string, *args): 37 | return sliced(string, *args) or string 38 | 39 | @app.template_filter('render') 40 | def _render(graph, **overrides): 41 | """ 42 | Takes a Graph as input, prints out full render URL. 43 | """ 44 | params = dict(graph.kwargs, **overrides) 45 | return flask.url_for("render", **params) 46 | 47 | @app.template_filter('composer') 48 | def composer(graph): 49 | """ 50 | We don't typically want this to accept overrides -- if somebody wants to go 51 | to the composer view, many e.g. thumbnail/presentational options should 52 | probably get turned off so they get a more normal view. Useful things like 53 | timeperiod/from will typically be preserved. 54 | """ 55 | if config.external_graphite: 56 | return config.external_graphite + "/composer/" + graph.querystring 57 | 58 | 59 | # 60 | # Routes 61 | # 62 | 63 | @app.route('/') 64 | def index(): 65 | collections = [ 66 | ('by_domain', { 67 | 'title': 'Per-host pages, by domain', 68 | 'groups': config.graphite.hosts_by_domain(), 69 | }) 70 | ] 71 | collections.extend( 72 | sorted( 73 | config.collections.items(), 74 | key=lambda x: x[0] 75 | ) 76 | ) 77 | return flask.render_template( 78 | 'index.html', 79 | collections=collections 80 | ) 81 | 82 | @app.route('/by_domain//') 83 | def domain(domain): 84 | return flask.render_template( 85 | 'domain.html', 86 | domain=domain, 87 | hosts=config.graphite.hosts_for_domain(domain), 88 | metric_groups=config.metric_groups, 89 | ) 90 | 91 | @app.route('///') 92 | def group(collection, group): 93 | # Setup 94 | cname = collection 95 | gname = group 96 | collection = config.collections[collection] 97 | group = collection['groups'][group] 98 | return flask.render_template( 99 | 'collection_group.html', 100 | cname=cname, 101 | group=group, 102 | gname=gname, 103 | metrics=group['metrics'] 104 | ) 105 | 106 | @app.route('////') 107 | def group_metric(collection, group, metric): 108 | # Basic setup 109 | period = '-4hours' 110 | cname = collection 111 | collection = config.collections[collection] 112 | gname = group 113 | group = collection['groups'][group] 114 | # Slug => metric object 115 | mobj = None 116 | for m in group['metrics']: 117 | if m.name == metric: 118 | mobj = m 119 | break 120 | if mobj is None: 121 | flask.abort(404) 122 | # Metric-based nav 123 | metric_groups = map( 124 | lambda x: (x, flask.url_for('group_metric', collection=cname, 125 | group=gname, metric=x)), 126 | [x.name for x in group['metrics']] 127 | ) 128 | parent = flask.url_for('group', collection=cname, group=gname) 129 | # Grid setup 130 | per_row = 5 131 | col_size = (16 / per_row) 132 | # Thumbnails 133 | thumbnail_opts = { 134 | 'height': 100, 135 | 'width': 200, 136 | 'hideLegend': True, 137 | 'hideGrid': True, 138 | 'yBoundsOnly': True, 139 | 'hideXAxis': True, 140 | 'from': period, 141 | } 142 | return flask.render_template( 143 | 'group.html', 144 | collection=collection, 145 | group=group, 146 | metric=mobj, 147 | metric_groups=metric_groups, 148 | current_mgroup=metric, 149 | per_row=per_row, 150 | col_size=col_size, 151 | thumbnail_opts=thumbnail_opts, 152 | period=period, 153 | parent=parent, 154 | gname=gname 155 | ) 156 | 157 | @app.route('/by_domain/////') 158 | def host(domain, host, metric_group, period): 159 | # Get metric objects for this group 160 | raw_metrics = config.groups[metric_group].values() 161 | # Filter period value through defined aliases 162 | kwargs = {'from': config.periods.get(period, period)} 163 | # Generate graph objects from each metric, based on hostname context 164 | graphite_host = host + '_' + domain.replace('.', '_') 165 | graphs = map(lambda m: m.graphs(graphite_host, **kwargs), raw_metrics) 166 | merged = reduce(operator.add, graphs, []) 167 | # Set up metric group nav 168 | metric_groups = map( 169 | lambda x: (x, flask.url_for('host', domain=domain, metric_group=x, 170 | host=host, period=period)), 171 | config.metric_groups 172 | ) 173 | return flask.render_template( 174 | 'host.html', 175 | domain=domain, 176 | host=host, 177 | metrics=merged, 178 | metric_groups=metric_groups, 179 | periods=config.periods.keys(), 180 | current_mgroup=metric_group, 181 | current_period=period, 182 | ) 183 | 184 | @app.route('/render/') 185 | def render(): 186 | uri = config.graphite.uri + "/render/" 187 | response = requests.get(uri, params=flask.request.args) 188 | return flask.Response(response=response.raw, headers=response.headers) 189 | -------------------------------------------------------------------------------- /fullerene/config.py: -------------------------------------------------------------------------------- 1 | import yaml 2 | 3 | from graphite import Graphite 4 | from metric import Metric 5 | 6 | 7 | class Config(object): 8 | def __init__(self, text): 9 | # Load up 10 | config = yaml.load(text) 11 | # Required items 12 | try: 13 | try: 14 | exclude_hosts = config['hosts']['exclude'] 15 | except KeyError: 16 | exclude_hosts = [] 17 | self.graphite = Graphite( 18 | uri=config['graphite_uris']['internal'], 19 | exclude_hosts=exclude_hosts 20 | ) 21 | except KeyError: 22 | raise ValueError, "Configuration must specify graphite_uris: internal" 23 | # Optional external URL (for links) 24 | self.external_graphite = config['graphite_uris'].get('external', None) 25 | # 'metrics' section 26 | self.metrics = {} 27 | for name, options in config.get('metrics', {}).iteritems(): 28 | self.metrics[name] = Metric( 29 | options=options, 30 | config=self, 31 | name=name 32 | ) 33 | # Metric groups 34 | self.groups = {} 35 | for name, metrics in config.get('metric_groups', {}).iteritems(): 36 | if name not in self.groups: 37 | self.groups[name] = {} 38 | for item in metrics: 39 | self.groups[name][item] = self.parse_metric(item) 40 | # 'collections' 41 | self.collections = config.get('collections', {}) 42 | for collection in self.collections.values(): 43 | # Instantiate metrics where needed 44 | for group in collection['groups'].values(): 45 | group['metrics'] = map(self.parse_metric, group['metrics']) 46 | if 'overview' in group: 47 | group['overview'] = map( 48 | self.parse_metric, 49 | group['overview'][:] 50 | ) 51 | # Default graph args 52 | self.defaults = config.get('defaults', {}) 53 | # Timeperiod aliases 54 | self.periods = config.get('periods', {}) 55 | 56 | def parse_metric(self, item): 57 | exists = False 58 | try: 59 | exists = item in self.metrics 60 | except TypeError: 61 | pass 62 | # Name + name already exists as a metric alias == use that 63 | if exists: 64 | metric = self.metrics[item] 65 | else: 66 | # String == metric path == make new metric from it 67 | if isinstance(item, basestring): 68 | metric = Metric({'path': item}, config=self, name=item) 69 | # Non-string == assume hash/dict == make metric from that (assumes 70 | # one-item dict, name => metric) 71 | else: 72 | name, value = item.items()[0] 73 | metric = Metric(name=name, config=self, options=value) 74 | return metric 75 | 76 | @property 77 | def metric_groups(self): 78 | return sorted(self.groups) 79 | -------------------------------------------------------------------------------- /fullerene/graph.py: -------------------------------------------------------------------------------- 1 | class Graph(object): 2 | def __init__( 3 | self, 4 | path, 5 | config=None, 6 | title=None, 7 | title_param=None, 8 | **kwargs 9 | ): 10 | """ 11 | path: metric path to query Graphite for when rendering (includes host) 12 | config: config object for querying 13 | kwargs: graph drawing options such as 'from' 14 | """ 15 | # Basic init 16 | self.path = path 17 | self.title = title 18 | self.title_param = title_param 19 | 20 | # Add default title 21 | if 'title' not in kwargs: 22 | period = " (%s)" % kwargs['from'] if 'from' in kwargs else "" 23 | param = "" 24 | if self.title_param: 25 | param = " (%s)" % self.path.split('.')[self.title_param] 26 | kwargs['title'] = (self.title + param) if self.title else self.path 27 | 28 | # Construct final target path 29 | path = self.path 30 | function = kwargs.pop('function', None) 31 | if function: 32 | path = "%s(%s)" % (function, path) 33 | kwargs['target'] = path 34 | 35 | # Set kwargs for drawing 36 | self.kwargs = kwargs 37 | 38 | # Store data about what our graph shows (e.g. min/max) 39 | if config: 40 | self.stats = config.graphite.stats(kwargs) 41 | 42 | def __str__(self): 43 | return self.path 44 | 45 | def __repr__(self): 46 | return "" % ( 47 | str(self), self.children, self.kwargs 48 | ) 49 | 50 | @property 51 | def querystring(self): 52 | """ 53 | Prints out query string for easy appending to URLs. 54 | 55 | E.g. "?target=foo&from=blah&height=xxx" 56 | """ 57 | pairs = map(lambda x: "%s=%s" % (x[0], x[1]), self.kwargs.items()) 58 | return "?" + "&".join(pairs) 59 | -------------------------------------------------------------------------------- /fullerene/graphite.py: -------------------------------------------------------------------------------- 1 | from collections import defaultdict 2 | import json 3 | import requests 4 | 5 | from utils import dots, sliced 6 | 7 | 8 | class Graphite(object): 9 | """ 10 | Stand-in for the backend Graphite server/service. 11 | 12 | Mostly used for querying API endpoints under /metrics/. 13 | """ 14 | def __init__(self, uri, exclude_hosts): 15 | self.uri = uri 16 | self.exclude_hosts = exclude_hosts 17 | 18 | def query(self, *paths, **kwargs): 19 | """ 20 | Return list of metric paths based on one or more search queries. 21 | 22 | Basically just a wrapper around Graphite's /metrics/expand/ endpoint. 23 | 24 | Specify ``leaves_only=True`` to filter out any non-leaf results. 25 | """ 26 | query = "?" + "&".join(map(lambda x: "query=%s" % x, paths)) 27 | uri = self.uri + "/metrics/expand/%s" % query 28 | if kwargs.get('leaves_only', False): 29 | uri += "&leavesOnly=1" 30 | response = requests.get(uri) 31 | struct = json.loads(response.content)['results'] 32 | filtered = filter( 33 | lambda x: x not in self.exclude_hosts, 34 | struct 35 | ) 36 | return filtered 37 | 38 | def stats(self, kwargs): 39 | kwargs = dict(kwargs) # lest we screw it up for rendering later 40 | kwargs['format'] = 'json' 41 | uri = "%s/render/" % self.uri 42 | return json.loads(requests.get(uri, params=kwargs).content) 43 | 44 | def query_all(base, max_depth=7): 45 | """ 46 | Return *all* metrics starting with the given ``base`` pattern/string. 47 | 48 | Assumes maximum realistic depth of ``max_depth``, due to the method 49 | required to get multiple levels of metric paths out of Graphite. 50 | 51 | If run with ``base="*"`` be prepared to wait a very long time for any 52 | nontrivial Graphite installation to come back with the answer... 53 | """ 54 | queries = [] 55 | for num in range(1, max_depth + 1): 56 | query = "%s.%s" % (base, ".".join(['*'] * num)) 57 | queries.append(query) 58 | return self.query(queries, leaves_only=True) 59 | 60 | def hosts_by_domain(self): 61 | hosts = self.query("*") 62 | domains = defaultdict(list) 63 | for host in hosts: 64 | name, _, domain = host.partition('_') 65 | domains[dots(domain)].append(dots(host)) 66 | return domains 67 | 68 | def hosts_for_domain(self, domain): 69 | return map( 70 | lambda x: sliced(x, 1), 71 | self.hosts_by_domain()[dots(domain)] 72 | ) 73 | -------------------------------------------------------------------------------- /fullerene/metric.py: -------------------------------------------------------------------------------- 1 | from collections import defaultdict 2 | from types import StringTypes 3 | 4 | from graph import Graph 5 | 6 | 7 | def combine(paths, expansions=[], include_raw=False): 8 | """ 9 | Take a list of paths and combine into fewer using brace-expressions. 10 | 11 | E.g. ["foo.bar", "foo.biz"] => "foo.{bar,biz}" 12 | 13 | When ``include_raw`` is ``True``, returns a mapping instead of a list, 14 | where the key the brace-expression string and the value is the list of 15 | paths making up that particular brace-expression. (Which, with no 16 | expansion, will always be the same as the input list; with expansion it's 17 | usually a subset.) 18 | 19 | With expansions & include_raw, you'd get e.g. {"foo.bar": ["foo.bar"], 20 | "foo.biz": ["foo.biz"]} for the same input as above and an expansion list 21 | of [1]. Pretty tautological. 22 | 23 | A more complex example would be partial expansion. Calling 24 | combine(["a.1.b.1", "a.1.b.2", "a.2.b.1", "a.2.b.2"], expansions=[1], 25 | include_raw=True) would result in: 26 | 27 | { 28 | "a.{1,2}.b.1": ["a.1.b.1", "a.2.b.1"], 29 | "a.{1,2}.b.2": ["a.1.b.2", "a.2.b.2"] 30 | } 31 | 32 | because the first "overlapping" segment (a.1 vs a.2) is expanded, but the 33 | second (b.1 vs b.2) is not, and thus we get two keys whose values split the 34 | incoming 4-item list in half. 35 | """ 36 | # Preserve input 37 | original_paths = paths[:] 38 | buckets = defaultdict(list) 39 | # Divvy up paths into per-segment buckets 40 | for path in paths: 41 | for i, part in enumerate(path.split('.')): 42 | if part not in buckets[i]: 43 | buckets[i].append(part) 44 | # "Zip" up buckets as needed depending on expansions 45 | ret = [[]] 46 | for key in sorted(buckets.keys()): 47 | value = list(buckets[key]) 48 | # Only one value for this index: everybody gets a copy 49 | if len(value) == 1: 50 | for x in ret: 51 | x.append(value[0]) 52 | else: 53 | # If this index is to be expanded, branch out: all existing results 54 | # up to this point get cloned, one per matching item 55 | if key in expansions: 56 | previous = ret[:] 57 | ret = [] 58 | for x in value: 59 | for y in previous: 60 | ret.append(y + [x]) 61 | # No expansion = just drop in the iterable, no conversion to string 62 | else: 63 | for x in ret: 64 | x.append(value) 65 | # Now that we're done, merge the chains into strings 66 | mapping = {} 67 | # TODO: This is so dumb. Must be a way to merge with the nearly-identical 68 | # shit above. I suck at algorithms. 69 | for expr in ret: 70 | key_parts = [] 71 | paths = [[]] 72 | for part in expr: 73 | if isinstance(part, list): 74 | # Update paths 75 | previous = paths[:] 76 | paths = [] 77 | for subpart in part: 78 | for x in previous: 79 | paths.append(x + [subpart]) 80 | # Update key parts 81 | part = "{" + ",".join(part) + "}" 82 | else: 83 | for path in paths: 84 | path.append(part) 85 | key_parts.append(part) 86 | # New final key/value pair 87 | # (Strip out any incorrectly expanded paths not present in the input.) 88 | # (TODO: figure out how not to expand combinatorically when that's not 89 | # correct. sigh) 90 | raw_paths = filter( 91 | lambda x: x in original_paths, 92 | map(lambda x: ".".join(x), paths) 93 | ) 94 | # Do the same for keys, when expanding; have to search substrings. 95 | merged_path = ".".join(key_parts) 96 | merged_path_good = False 97 | lcd, _, rest = merged_path.partition('{') 98 | for original in original_paths: 99 | if original.startswith(lcd): 100 | merged_path_good = True 101 | break 102 | if merged_path_good: 103 | mapping[merged_path] = raw_paths 104 | return mapping if include_raw else mapping.keys() 105 | 106 | 107 | class Metric(object): 108 | """ 109 | Beefed-up metric object capable of substituting wildcards and more! 110 | """ 111 | def __init__(self, options, config, name=""): 112 | # Handle just-a-string options 113 | if not hasattr(options, 'pop'): 114 | options = {'path': options} 115 | self.name = name 116 | self.path = options.pop('path') 117 | self.title = options.pop('title', self.path) 118 | self.title_param = options.pop('title_param', None) 119 | self.config = config 120 | self.raw = options.pop('raw', False) 121 | # Generate split version of our path, and note any wildcards 122 | self.parts = self.path.split('.') 123 | self.wildcards = self.find_wildcards() 124 | # Normalize/clean up options 125 | self.excludes = self.set_excludes(options.pop('exclude', ())) 126 | self.to_expand = self.set_expansions(options.pop('expand', ())) 127 | # Everything else given in the YAML config is a graphite override 128 | self.extra_options = options 129 | 130 | def __repr__(self): 131 | return "" % ( 132 | self.path, self.excludes, self.to_expand 133 | ) 134 | 135 | def __eq__(self, other): 136 | return ( 137 | self.path == other.path 138 | and self.excludes == other.excludes 139 | and self.to_expand == other.to_expand 140 | ) 141 | 142 | def find_wildcards(self): 143 | """ 144 | Fill in self.wildcards from self.parts 145 | """ 146 | # Discover wildcard locations (so we can tell, while walking a split 147 | # string, "which" wildcard we may be looking at (the 0th, 1st, Nth) 148 | wildcards = [] 149 | for index, part in enumerate(self.parts): 150 | if '*' in part: 151 | wildcards.append(index) 152 | return wildcards 153 | 154 | def set_excludes(self, excludes): 155 | if not hasattr(excludes, "keys"): 156 | excludes = {0: excludes} 157 | # Stringify for any YAML ints 158 | for key in excludes: 159 | excludes[key] = map(str, excludes[key]) 160 | return excludes 161 | 162 | def set_expansions(self, expansions): 163 | if expansions == "all": 164 | expansions = self.wildcards 165 | return expansions 166 | 167 | def expand(self, hostname=""): 168 | """ 169 | Return expanded metric list from our path and the given ``hostname``. 170 | 171 | E.g. if self.path == foo.*.bar, this might return [foo.1.bar, 172 | foo.2.bar]. 173 | 174 | This method does not take into account any filtering or exclusions; it 175 | returns the largest possible expansion (basically what Graphite's 176 | /metrics/expand/ endpoint gives you.) 177 | 178 | ``hostname`` is used solely for giving context to the expansion; the 179 | returned metric paths will still be host-agnostic (in order to blend in 180 | with non-expanded metric paths.) 181 | """ 182 | sep = '.' 183 | if hostname: 184 | path = sep.join([hostname, self.path]) 185 | func = lambda x: sep.join(x.split(sep)[1:]) 186 | else: 187 | path = self.path 188 | func = lambda x: x 189 | return map(func, self.config.graphite.query(path)) 190 | 191 | def graphs(self, hostname="", **kwargs): 192 | """ 193 | Return 1+ Graph objects, optionally using ``hostname`` for context. 194 | 195 | Uses the initial ``excludes`` and ``expands`` options to determine 196 | which graphs to return. See ``metric.Metric.expand`` and 197 | ``metric.combine`` for details. 198 | 199 | Any kwargs passed in are passed into the Graph objects, so e.g. 200 | ``.graphs('foo.hostname', **{'from': '-24hours'})`` is a convenient way 201 | to get a collection of graphs for this metric all set to draw a 24 hour 202 | period. 203 | 204 | The kwargs will be used to override any defaults from the config 205 | object. 206 | """ 207 | hostname = hostname.replace('.', '_') 208 | # If %-expressions in path, or raw=True, just insert hostname and skip 209 | # parsing 210 | group = kwargs.pop('group', "") 211 | parameterized = "%s" in self.path or "%g" in self.path 212 | if parameterized or self.raw: 213 | if parameterized: 214 | path = (self.path 215 | .replace("%s", "%(hostname)s") 216 | .replace("%g", "%(group)s") 217 | ) 218 | results = [path % {'hostname': hostname, 'group': group}] 219 | else: 220 | results = [self.path] 221 | return self._graphs(results, kwargs) 222 | # Expand out to full potential list of paths, apply filters 223 | matches = [] 224 | expanded = self.expand(hostname) 225 | for item in expanded: 226 | parts = item.split('.') 227 | good = True 228 | # Exclude any exclusions 229 | for location, part in enumerate(parts): 230 | # We only care about wildcard slots 231 | if location not in self.wildcards: 232 | continue 233 | # Which wildcard slot is this? 234 | wildcard_index = self.wildcards.index(location) 235 | # Is this substring listed for exclusion in this slot? 236 | if part in self.excludes.get(wildcard_index, []): 237 | good = False 238 | break # move on to next metric/item 239 | if good: 240 | matches.append(item) 241 | # Perform any necessary combining into brace-expressions & return 242 | result = combine(matches, self.to_expand) 243 | result = map(lambda x: "%s.%s" % (hostname, x), result) 244 | return self._graphs(result, kwargs) 245 | 246 | def _graphs(self, paths, kwargs): 247 | # Precedence: defaults => overridden by extra_options => kwargs 248 | first_merge = dict(self.extra_options, **kwargs) 249 | merged_kwargs = dict(self.config.defaults, **first_merge) 250 | return [ 251 | Graph(path, self.config, self.title, self.title_param, **merged_kwargs) 252 | for path in paths 253 | ] 254 | -------------------------------------------------------------------------------- /fullerene/templates/_graph.html: -------------------------------------------------------------------------------- 1 | {% if graph|composer %} 2 | 3 | 4 | 5 | {% else %} 6 | 7 | {% endif %} 8 | 9 | 10 | 11 | 12 | 13 | {% for metric in graph.stats %} 14 | 15 | 16 | 17 | 18 | 19 | 20 | {% endfor %} 21 | 22 |
MetricMinMaxMean
{{ metric.target|dot(1, -1) }}{{ metric.stats.formatted.min }}{{ metric.stats.formatted.max }}{{ metric.stats.formatted.mean }}
23 | -------------------------------------------------------------------------------- /fullerene/templates/_metric_groups.html: -------------------------------------------------------------------------------- 1 | {% for mgroup, url in metric_groups %} 2 | {% if mgroup == current_mgroup %} 3 | {{ mgroup }} 4 | {% else %} 5 | {{ mgroup }} 6 | {% endif %} 7 | {% if not loop.last %}|{% endif %} 8 | {% endfor %} 9 | -------------------------------------------------------------------------------- /fullerene/templates/base.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 10 | Fullerene 11 | 12 | 13 |
14 | {% block body %}{% endblock %} 15 |
16 | 17 | 18 | -------------------------------------------------------------------------------- /fullerene/templates/collection_group.html: -------------------------------------------------------------------------------- 1 | {% extends "base.html" %} 2 | {% block body %} 3 |
4 |
5 |

{{ group.title }}

6 |

« Index

7 |
8 |
9 |
10 |
11 |

Overview

12 |
13 |
14 |
15 | {% for tuple in group.overview|batch(2) %} 16 |
17 | {% for metric in tuple %} 18 | {% for graph in metric.graphs(group=gname) %} 19 | {% include "_graph.html" %} 20 | {% endfor %} 21 | {% endfor %} 22 |
23 | {% endfor %} 24 |
25 |
26 |
27 |

Metric thumbnails

28 |
    29 | {% for metric in metrics %} 30 |
  • 31 | {{ metric.name }} 32 |
    • 33 | {% endfor %} 34 |
35 |
36 |
37 | {% endblock %} 38 | -------------------------------------------------------------------------------- /fullerene/templates/domain.html: -------------------------------------------------------------------------------- 1 | {% extends "base.html" %} 2 | {% block body %} 3 |
4 |
5 |

{{ domain|dots }}

6 |

« Index

7 |
8 |
9 |
10 |
11 |
    12 | {% for host in hosts|sort %} 13 |
  • 14 | {{ host|dot(1) }}: 15 | {% for mgroup in metric_groups %} 16 | {{ mgroup }} 17 | {% if not loop.last %}|{% endif %} 18 | {% endfor %} 19 |
    • 20 | {% endfor %} 21 |
22 |
23 |
24 | {% endblock %} 25 | -------------------------------------------------------------------------------- /fullerene/templates/group.html: -------------------------------------------------------------------------------- 1 | {% extends "base.html" %} 2 | {% block body %} 3 |
4 |
5 |

{{ group.title }}

6 |
7 |
8 | 9 |
10 |
11 |

Overview

12 |
13 |
14 |
15 | {% for tuple in group.overview|batch(2) %} 16 |
17 | {% for ometric in tuple %} 18 | {% for graph in ometric.graphs(group=gname) %} 19 | {% include "_graph.html" %} 20 | {% endfor %} 21 | {% endfor %} 22 |
23 | {% endfor %} 24 |
25 | 26 |
27 |
28 |

{{ metric.title }}

29 | ( 30 | {% if "%s" not in metric.path %} 31 | {{ metric.path|truncate(30, True) }} 32 | {% else %} 33 | it's complicated 34 | {% endif %} 35 | ) 36 |
37 |
38 |
39 |
40 |

{% include "_metric_groups.html" %}

41 |
42 |
43 |
44 |
45 |

Time period: {{ period }}

46 |
47 |
48 | {% for hosts in group.hosts|sort|batch(per_row) %} 49 |
50 |
51 | {% for host in hosts %} 52 | {% for graph in metric.graphs(host) %} 53 | {% if graph|composer %}{% endif %} 54 | 55 | {% if graph|composer %}{% endif %} 56 | {% if not loop.last %}
{% endif %} 57 | {% endfor %} 58 | {% endfor %} 59 |
60 |
61 | {% endfor %} 62 | {% endblock %} 63 | -------------------------------------------------------------------------------- /fullerene/templates/host.html: -------------------------------------------------------------------------------- 1 | {% extends "base.html" %} 2 | {% block body %} 3 |
4 |
5 |

{{ host }}.{{ domain }}

6 |

{% include "_metric_groups.html" %}

7 |

8 | timeperiod: 9 | {% for period in periods %} 10 | {% if period == current_period %} 11 | {{ period }} 12 | {% else %} 13 | {{ period }} 14 | {% endif %} 15 | {% if not loop.last %}|{% endif %} 16 | {% endfor %} 17 |

18 |
19 |
20 | {% for tuple in metrics|batch(2) %} 21 |
22 | {% for graph in tuple %} 23 |
24 | {% include "_graph.html" %} 25 |
26 | {% endfor %} 27 |
28 | {% endfor %} 29 | {% endblock %} 30 | -------------------------------------------------------------------------------- /fullerene/templates/index.html: -------------------------------------------------------------------------------- 1 | {% extends "base.html" %} 2 | {% block body %} 3 |
4 |
5 |

Collections

6 |
7 |
8 | {% for tuple in collections|batch(4) %} 9 |
10 | {% for slug, collection in tuple %} 11 |
12 |

{{ collection.title }}

13 |
    14 | {% for group in collection.groups|sort %} 15 |
  • {{ group }}
    • 16 | {% endfor %} 17 |
18 |
19 | {% endfor %} 20 |
21 | {% endfor %} 22 | {% endblock %} 23 | -------------------------------------------------------------------------------- /fullerene/utils.py: -------------------------------------------------------------------------------- 1 | def dots(string): 2 | return string.replace('_', '.') 3 | 4 | def sliced(string, *args): 5 | return '.'.join(string.split('.')[slice(*args)]) 6 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | mock 2 | nose 3 | -e git+https://github.com/bitprophet/rudolf#egg=rudolf 4 | -------------------------------------------------------------------------------- /runner.py: -------------------------------------------------------------------------------- 1 | from fullerene import app, CONFIG 2 | 3 | 4 | app.run(host='0.0.0.0', port=8080, debug=True, extra_files=(CONFIG,)) 5 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [nosetests] 2 | verbosity=2 3 | with-color=1 4 | nocapture=1 5 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | from setuptools import setup, find_packages 4 | 5 | setup( 6 | name='Fullerene', 7 | version='0.1', 8 | description='A Graphite dashboarding tool', 9 | author='Jeff Forcier', 10 | author_email='jeff@bitprophet.org', 11 | packages=find_packages(), 12 | #test_suite='nose.collector', 13 | tests_require=['nose', 'mock', 'rudolf'], 14 | install_requires=['requests >=0.7.3', 'flask', 'pyyaml'], 15 | ) 16 | -------------------------------------------------------------------------------- /tests/support/basic.yml: -------------------------------------------------------------------------------- 1 | graphite_uris: 2 | internal: whatever 3 | hosts: 4 | exclude: 5 | - a 6 | - b 7 | periods: 8 | day: -24hours 9 | week: -7days 10 | defaults: 11 | height: 250 12 | width: 400 13 | from: -2hours 14 | metrics: 15 | metric1: 16 | path: foo.bar 17 | metric2: 18 | path: biz.baz 19 | metric_groups: 20 | group1: 21 | - raw.path 22 | - metric1 23 | group2: 24 | - metric1 25 | -------------------------------------------------------------------------------- /tests/support/exclusions.yml: -------------------------------------------------------------------------------- 1 | graphite_uris: 2 | internal: x 3 | metrics: 4 | implicit: 5 | path: foo.*.bar 6 | exclude: [1, 2] 7 | implicit_1st: 8 | path: foo.*.* 9 | exclude: [1] 10 | explicit: 11 | path: foo.*.bar 12 | exclude: 13 | 0: [1, 2] 14 | explicit_multiple: 15 | path: foo.*.bar.*.* 16 | exclude: 17 | 0: [1] 18 | 2: [bar] 19 | -------------------------------------------------------------------------------- /tests/support/no_url.yml: -------------------------------------------------------------------------------- 1 | nope: "no graphite_uri here" 2 | -------------------------------------------------------------------------------- /tests/test_main.py: -------------------------------------------------------------------------------- 1 | import sys 2 | 3 | import os.path 4 | 5 | import mock 6 | from nose.tools import eq_, raises 7 | from nose.plugins.skip import SkipTest 8 | 9 | from fullerene.metric import Metric, combine 10 | from fullerene.config import Config 11 | 12 | 13 | def conf(name): 14 | here = os.path.dirname(__file__) 15 | with open(os.path.join(here, "support", name + ".yml")) as fd: 16 | return Config(fd.read()) 17 | 18 | 19 | class TestMetrics(object): 20 | def test_exclusions(self): 21 | config = conf("exclusions") 22 | for desc, name, expansions, result in ( 23 | ("Implicit exclude list", 24 | "implicit", 25 | ("foo.1.bar", "foo.2.bar", "foo.3.bar"), 26 | "foo.3.bar" 27 | ), 28 | ("Implicit exclude list applied only to 1st wildcard", 29 | "implicit_1st", 30 | ("foo.1.2", "foo.1.1", "foo.3.1"), 31 | "foo.3.1" 32 | ), 33 | ("Explicit exclude list", 34 | "explicit", 35 | ("foo.1.bar", "foo.2.bar", "foo.3.bar"), 36 | "foo.3.bar" 37 | ), 38 | ("Explicit exclude list, multiple wildcards", 39 | "explicit_multiple", 40 | ( 41 | # Doesn't match any excludes 42 | "foo.2.bar.biz.baz", 43 | # Matches exclude in 1st wildcard slot 44 | "foo.1.2.3.4", 45 | # Matches exclude in 3rd wildcard slot 46 | "foo.bar.biz.2.bar" 47 | ), 48 | "foo.2.bar.biz.baz" 49 | ), 50 | ): 51 | graphite = mock.Mock() 52 | graphite.query.return_value = expansions 53 | with mock.patch.object(config, 'graphite', graphite): 54 | eq_.description = desc 55 | yield eq_, map(str, config.metrics[name].graphs()), [result] 56 | del eq_.description 57 | 58 | def test_combinations(self): 59 | for desc, inputs, results in ( 60 | ("Single metric, no combinations", 61 | ("foo.bar",), "foo.bar"), 62 | ("Single combination at end", 63 | ("foo.bar", "foo.biz"), "foo.{bar,biz}"), 64 | ("Combinations in both of two positions", 65 | ("foo.bar", "biz.baz"), "{foo,biz}.{bar,baz}"), 66 | ("One combination in the 2nd of 3 positions", 67 | ("foo.1.bar", "foo.2.bar"), "foo.{1,2}.bar"), 68 | ("Two combinations surrounding one normal part", 69 | ("foo.name.bar", "biz.name.baz"), "{foo,biz}.name.{bar,baz}"), 70 | ): 71 | eq_.description = desc 72 | yield eq_, map(str, combine(inputs)), [results] 73 | del eq_.description 74 | 75 | def test_expansions(self): 76 | # Remember that expansion indexes apply only to wildcard slots, 77 | # which here are slots which differ from path to path and would thus 78 | # get combined by default. 79 | for desc, inputs, expansions, results in ( 80 | ("Expand second part", 81 | ["foo.bar", "foo.biz"], [1], ["foo.bar", "foo.biz"]), 82 | ("Expand both parts", 83 | ["1.2", "3.4"], [0, 1], ["1.2", "3.4"]), 84 | ): 85 | eq_.description = desc 86 | yield eq_, set(map(str, combine(inputs, expansions))), set(results) 87 | del eq_.description 88 | 89 | def test_include_raw_expansions(self): 90 | """ 91 | combine(include_raw=True) with expansions 92 | """ 93 | result = combine(["foo.bar", "foo.biz"], [1], True) 94 | eq_(result, {"foo.bar": ["foo.bar"], "foo.biz": ["foo.biz"]}) 95 | 96 | def test_include_raw_no_expansions(self): 97 | """ 98 | combine(include_raw=True) without expansions 99 | """ 100 | result = combine(["foo.bar", "foo.biz"], [], True) 101 | eq_(result, {"foo.{bar,biz}": ["foo.bar", "foo.biz"]}) 102 | 103 | def test_include_raw_complex(self): 104 | """ 105 | combine(include_raw=True) with complex input 106 | """ 107 | paths = ["a.1.b.1", "a.1.b.2", "a.2.b.1", "a.2.b.2"] 108 | result = combine(paths, [3], True) 109 | eq_(result, 110 | { 111 | "a.{1,2}.b.1": ["a.1.b.1", "a.2.b.1"], 112 | "a.{1,2}.b.2": ["a.1.b.2", "a.2.b.2"] 113 | } 114 | ) 115 | 116 | 117 | def cmp_metrics(dict1, dict2): 118 | for metricname, metric in dict1.items(): 119 | eq_(dict2[metricname], metric) 120 | 121 | def recursive_getattr(obj, attr_list=()): 122 | value = getattr(obj, attr_list[0]) 123 | try: 124 | return recursive_getattr(value, attr_list[1:]) 125 | except IndexError: 126 | return value 127 | 128 | class TestConfig(object): 129 | @raises(ValueError) 130 | def test_required_options(self): 131 | """ 132 | Config files must specify graphite_uri 133 | """ 134 | conf("no_url") 135 | 136 | def test_basic_attributes(self): 137 | """ 138 | Attributes which are straight-up imported from the YAML 139 | """ 140 | for attr, expected in ( 141 | ('defaults', {'height': 250, 'width': 400, 'from': '-2hours'}), 142 | ('periods', {'day': '-24hours', 'week': '-7days'}), 143 | ('graphite.uri', 'whatever'), 144 | ('graphite.exclude_hosts', ['a', 'b']), 145 | ): 146 | eq_.description = "Config.%s = YAML '%s' value" % (attr, attr) 147 | result = recursive_getattr(conf("basic"), attr.split('.')) 148 | yield eq_, result, expected 149 | del eq_.description 150 | 151 | def test_metrics(self): 152 | """ 153 | A metrics struct should turn into a dict of Metrics 154 | """ 155 | metric1 = Metric("foo.bar", mock.Mock()) 156 | metric2 = Metric("biz.baz", mock.Mock()) 157 | metrics = { 158 | "metric1": metric1, 159 | "metric2": metric2 160 | } 161 | cmp_metrics(conf("basic").metrics, metrics) 162 | eq_(conf("basic").metrics, metrics) 163 | 164 | def test_groups(self): 165 | """ 166 | A groups struct should turn into a dict of lists of Metrics 167 | """ 168 | groups = { 169 | 'group1': { 170 | "raw.path": Metric("raw.path", mock.Mock()), 171 | "metric1": Metric("foo.bar", mock.Mock()) 172 | }, 173 | 'group2': { 174 | "metric1": Metric("foo.bar", mock.Mock()) 175 | } 176 | } 177 | config = conf("basic") 178 | for name, metrics in config.groups.items(): 179 | cmp_metrics(metrics, groups[name]) 180 | 181 | def test_metric_aliases(self): 182 | """ 183 | List items in groups collections should honor custom metric names 184 | """ 185 | config = conf("basic") 186 | aliased_metric = config.groups['group1']['metric1'] 187 | eq_(aliased_metric.path, "foo.bar") 188 | 189 | 190 | if __name__ == '__main__': 191 | main() 192 | --------------------------------------------------------------------------------