├── .gitignore
├── .travis.yml
├── README.md
├── README.txt
├── esbench
    ├── __init__.py
    ├── analyze.py
    ├── api.py
    ├── bench.py
    ├── client.py
    ├── config.json
    ├── data.py
    └── test
    │   ├── __init__.py
    │   ├── test_analyze.py
    │   ├── test_api.py
    │   ├── test_bench.py
    │   ├── test_client.py
    │   ├── test_data.py
    │   └── units.py
└── setup.py


/.gitignore:
--------------------------------------------------------------------------------
 1 | .DS_Store
 2 | *.pyc
 3 | *.pyo
 4 | dist/
 5 | build/
 6 | *.egg-info
 7 | 
 8 | *.log
 9 | *.tab
10 | *.sql
11 | *.txt
12 | *.json
13 | *.csv
14 | *.zip
15 | *.xml
16 | *.old
17 | *.gz
18 | *.tgz
19 | 
20 | MANIFEST
21 | !README.txt
22 | 


--------------------------------------------------------------------------------
/.travis.yml:
--------------------------------------------------------------------------------
1 | language: python
2 | python:
3 |   - "2.7"
4 | install:
5 |   - pip install . --use-mirrors
6 | script: 
7 |   - python -m esbench.test.units
8 | 
9 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
  1 | OBSOLETE / deprecated / decrepit?
  2 | --------
  3 | Hi, I am no longer supporting this. Leaving the code for reference, but will be removing the supporting s3 data, as I don't want to pay the transfer fees any more. **This code is old and I have no idea if it even runs on modern ES!** I suggest that for benchmarking you put a load on your system while monitoring with Marvel. I consult / optimize ES clusters for a living, so if you have a problem, get in touch (via issues). Also, come over to [our ES meetup](http://www.meetup.com/Elasticsearch-San-Francisco/)! Have a great day, Mik. 
  4 | 
  5 | Elasticsearch benchmarking tools
  6 | --------------------------------
  7 | 'esbench' is a set of python 2.7 scripts for benchmarking Elasticsearch. The
  8 | two primary uses are for capacity planning (guessing how much oomph you need
  9 | to do what what you think you need to do), and for performance tuning (trying
 10 | out various index, mapping, and query settings in a consistent and
 11 | reproducible manner). 
 12 | 
 13 | Installation
 14 | ------------
 15 | 
 16 | For now, since the project is under active development, I recommend that you
 17 | install from the 'dev' branch, with: 
 18 | 
 19 |     pip install https://github.com/mkocikowski/esbench/archive/dev.zip
 20 | 
 21 | This should give you a reasonable stable and reasonably current version. There
 22 | is always the 'master' (replace 'dev.zip' with 'master.zip' in the command
 23 | above), but I really think for now you are best off with 'dev'. 
 24 | 
 25 | Quick start
 26 | -----------
 27 | With an instance of elasticsearch running on localhost:9200, do: 
 28 | 
 29 | 	esbench run
 30 | 	
 31 | When the bench run has finished, you can review the results with: 
 32 | 
 33 | 	esbench show
 34 | 
 35 | To get help: 
 36 | 	
 37 | 	esbench --help
 38 | 	esbench <command> --help
 39 | 
 40 | Overview
 41 | --------
 42 | An Elasticsearch index is composed of a set of 1 or more Lucene indexes
 43 | (designated as primary and replica 'shards' by ES). The Elasticsearch cluster
 44 | routes incoming documents and search requests to individual Lucene indexes /
 45 | shards as needed. A single Lucene index is the basic unit on which indexing
 46 | and search operations are executed, and so the performance of individual
 47 | Lucene indexes largely determines the performance of a cluster. Lucene indexes
 48 | store data on disk in append-only 'segments', which can be periodically
 49 | cleaned up and merged. The number and size (bytes) of these segments, and the
 50 | disk IO, ultimately determine the speed at which data can be retrieved. Read
 51 | the
 52 | [merge](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/
 53 | index-modules-merge.html) documentation on the ES site. 
 54 | 
 55 | The basic approach to benchmarking is to create an Elasticsearch index with 1
 56 | primary and no replica shards (a single Lucene index), load it with data, and
 57 | run representative use patterns against it, adding more data until the
 58 | performance drops below acceptable levels. That will determine the maximum
 59 | viable size of a single shard given the data, use patterns, and hardware.
 60 | Total expected data size divided by the viable single index size gives then
 61 | the number of required shards. With a rule of thumb of no more than 2 shards
 62 | per core, this allows for simple capacity planning. 
 63 | 
 64 | The 'run' command
 65 | -----------------
 66 | When the 'run' command is executed, a 'esbench_test' index is created on the
 67 | target ES host, populated with provided data, and queried at specified
 68 | intervals. The queries which will be executed against the index are provided
 69 | in the 'config.json' file. Observations taken while the benchmark is running
 70 | are recorded in the 'esbench_stats' index on the ES host. Each benchmark
 71 | results in one document of type 'bench' and one or more documents of type
 72 | 'obs' (observation) added to the 'esbench_stats' index. 
 73 | 
 74 | When executed with default arguments, the 'run' command will result in a quick
 75 | (too small to be meaningful) benchmark run against the local ES instance,
 76 | using US Patent Application data downloaded from S3 (into /tmp, size 99MB).
 77 | The intention is to provide sample data to verify that the 'rig is working'
 78 | and to familiarize the user with observation data.
 79 | 
 80 | The first argument to play with is 'maxsize', the only non-keyword parameter,
 81 | which specifies the total number of documents to be inserted into the test
 82 | index, or the total byte size of data to be inserted. Default value is 1mb
 83 | allows the benchmark to run in just few seconds. Running: 
 84 | 
 85 | 	esbench run 5gb
 86 | 
 87 | will result in an index of about 4.8GB, which starts to yield meaningful
 88 | performance information. Experimenting with queries, and the various
 89 | parameters (try adjusting the number of segments or optimize calls) will yield
 90 | interesting changes. Depending on the specified logging level, you will see
 91 | information somewhat like that when running the benchmark: 
 92 | 
 93 | 	[...]
 94 |     INFO:esbench.bench:beginning observation no: 10, 2013-12-31T19:07:37Z
 95 |     INFO:esbench.bench:ran query 'match_description_facet_date_histogram' 100 times in 0.34s
 96 |     INFO:esbench.bench:ran query 'match_description_sorted_abstract' 100 times in 0.55s
 97 |     INFO:esbench.bench:ran query 'match_description' 100 times in 0.33s
 98 |     INFO:esbench.bench:finished observation no: 10, id: fc1596c0, time: 1.221
 99 |     INFO:esbench.bench:recorded observation into: http://localhost:9200/esbench_stats/obs/fc1596c0
100 |     INFO:esbench.bench:load complete; loaded total 36 lines into index 'esbench_test', total size: 1396489 (1.00mb)
101 |     INFO:esbench.bench:recorded benchmark into: http://localhost:9200/esbench_stats/bench/2a4fb87d
102 |     [...]
103 | 
104 | As data is stored into the 'esbench_stats' index, you can access it raw (see
105 | the last log line for the URL). This is the raw data, see the 'show' command
106 | for more user-friendly way of looking at the results. 
107 | 
108 | The config file
109 | ---------------
110 | The 'run' command uses a json config file for its index and query settings.
111 | You can see the path to the default config file ('config.json') by running
112 | 'esbench run --help' and looking for the value of '--config-file-path'
113 | argument. There are 3 sections to the config file: 
114 | 
115 | 1. 'queries': here you define the queries which will be run against the test data. Each key is a human-readable name, and the value is an ES query. This is the section which you want to customize to match your use patterns; if you are using your own data source with structure different than the default data source, then you definitely need to change the queries. 
116 | 2. 'index': settings used for creating test 'esbench_test' index into which test data is loaded. Default shards 1/0, basic mapping. You can change this, specifically the mapping, if you want to experiment with different data sources. 
117 | 3. 'config': basic configuration, command line arguments override these settings.
118 | 
119 | You can specify the config file to use with the '--config-file-path' flag to
120 | the 'run' command. 
121 | 
122 | Alternative data sources
123 | ------------------------
124 | To use data other than the default USPTO Patent Application set, you need 2
125 | things: a source of json documents (formatted in a way to be acceptable to
126 | Elasticsearch), one document per line, and a config file with the 'queries'
127 | section modified to match the data you are feeding in. Assuming that your data
128 | is in a file 'mydata.json', and your config is in 'myconf.json', then this is
129 | what you want to do (the two lines below are equivalent, but the second is
130 | piping data into 'esbench run' on stdin, which you can use with tools like
131 | 'esdump' to feed data from a running index into a test rig): 
132 | 
133 |     esbench run --config-file-path myjson.json --data mydata.json
134 |     cat mydata.json | esbench run --config-file-path myjson.json --data /dev/stdin 
135 | 
136 | So, let's say you have an index 'myindex' running on 'http://myhost:9200/',
137 | and you want to take your 'real' data from that server, and feed it into your
138 | benchmarking rig (you will need to install the
139 | [esdump](https://github.com/mkocikowski/estools)):
140 | 
141 |     esdump --host myhost --port 9200 myindex | esbench run --config-file-path myjson.json --data /dev/stdin
142 | 
143 | Note that the 'size' control still works, so if you want to only get 10mb of data from the source, you do: 
144 |     
145 |     esdump --host myhost --port 9200 myindex | esbench run --config-file-path myjson.json --data /dev/stdin 10mb
146 |     
147 | 
148 | The 'show' command
149 | ------------------
150 | The 'show' command will retrieve previously recorded benchmark information
151 | from an ES instance, and format it and display it as requested. Running it
152 | without parameters will output data in csv format, with only most important
153 | columns present. There are many options available, bun in general, if you are
154 | not satisfied with what you get 'out of the box', the approach is to dump all
155 | the data into a csv, and use whatever tools you prefer to graph / analyze the
156 | data. When you run 'esbench show --help' you will get info on options
157 | available. You can run: 
158 | 
159 | 	esbench show --format tab
160 | 
161 | to get the data formatted in a table, but honestly, you will need a really big
162 | screen (& tiny font) for that to be any use at this time. Since the data,
163 | columns, orders of magnitude, will all change if you go with data sets / query
164 | sets other than default, I'm not putting much time into 'canned' data
165 | analysis. Put it out to csv files, analyze with outside tools.
166 | 
167 | Graphing
168 | --------
169 | 
170 | 	esbench show --help
171 | 
172 | The basic idea is that you dump the fields in which you are interested into a
173 | csv file, and then use whatever tools you prefer to analyze and graph that
174 | data. However, 'esbench show --help' will show you the command line commands
175 | needed to graph the standard data (you will need to have gnuplot on your
176 | system). Since you can access data on remote systems ('--host' and '--port'
177 | arguments), you only need gnuplot on your workstation. 
178 | 
179 | The 'dump' command
180 | ------------------
181 | Dump previously recorded benchmark data as a series of curl calls. These can
182 | be replayed to populate a new index. The idea with this is that this way you
183 | can easily share the raw benchmark data, say by dumping it into gist. Or you
184 | can just copy and paste right in the terminal - it is just a series of curl
185 | calls. To dump into a file:
186 | 
187 |     esbench dump > foo.curl
188 | 
189 | To load these into an elasticsearch index on localhost:
190 | 
191 |     . foo.curl
192 | 
193 | Data
194 | ----
195 | The default data set is US Patent Applications fror years 2005-2012. These
196 | have been pre-parsed and are available for download from S3. The downloading,
197 | unzipping, etc all happens automagically courtesy of the 'esbench.data'
198 | module. To get a sample document: 
199 | 
200 |     python -m esbench.data | head -1 | python -m json.tool
201 | 
202 | document counts / byte sizes are as follows: 
203 | 
204 | - 2005: 16gb, 289617 docs
205 | - 2006: 13gb, 294540 docs
206 | - 2007: 14gb, 300200 docs
207 | - 2008: 15gb, 320525 docs
208 | - 2009: 16gb, 328237 docs
209 | - 2010: 16gb, 333211 docs
210 | - 2011: 16gb, 321182 docs
211 | - 2012: 17gb, 331583 docs
212 | 
213 | for a total of 123gb of raw data. These files are pre
214 | parsed and stored in S3 so that there is a solid baseline immutable data set
215 | which can be easily shared. 
216 | 
217 | Tests
218 | -----
219 | Once 'esbench' has been installed (no need for local Elasticsearch instance):
220 | 
221 | 	python -m esbench.test.units
222 | 
223 | This may take few minutes as part of the test involves downloading a sample
224 | data file from s3. 
225 | 
226 | 
227 | License
228 | -------
229 | 
230 | The project uses [the MIT license](http://opensource.org/licenses/MIT):
231 | 
232 |     Copyright (c) 2013 Mikolaj Kocikowski
233 |     
234 |     Permission is hereby granted, free of charge, to any person obtaining a
235 |     copy of this software and associated documentation files (the "Software"),
236 |     to deal in the Software without restriction, including without limitation
237 |     the rights to use, copy, modify, merge, publish, distribute, sublicense,
238 |     and/or sell copies of the Software, and to permit persons to whom the
239 |     Software is furnished to do so, subject to the following conditions:
240 |     
241 |     The above copyright notice and this permission notice shall be included in
242 |     all copies or substantial portions of the Software.
243 |     
244 |     THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
245 |     IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
246 |     FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
247 |     THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
248 |     LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
249 |     FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
250 |     DEALINGS IN THE SOFTWARE.
251 |     
252 | 


--------------------------------------------------------------------------------
/README.txt:
--------------------------------------------------------------------------------
1 | README.md


--------------------------------------------------------------------------------
/esbench/__init__.py:
--------------------------------------------------------------------------------
 1 | # -*- coding: UTF-8 -*-
 2 | # (c)2013 Mik Kocikowski, MIT License (http://opensource.org/licenses/MIT)
 3 | # https://github.com/mkocikowski/esbench
 4 | 
 5 | __version__ = "0.2.3"
 6 | 
 7 | STATS_INDEX_NAME = 'esbench_stats'
 8 | TEST_INDEX_NAME = 'esbench_test'
 9 | TEST_DOCTYPE_NAME = 'doc'
10 | 


--------------------------------------------------------------------------------
/esbench/analyze.py:
--------------------------------------------------------------------------------
  1 | # -*- coding: UTF-8 -*-
  2 | # (c)2013 Mik Kocikowski, MIT License (http://opensource.org/licenses/MIT)
  3 | # https://github.com/mkocikowski/esbench
  4 | 
  5 | """Code for retrieving, analyzing, and displaying recorded benchmark data. """
  6 | 
  7 | from __future__ import print_function
  8 | 
  9 | import itertools
 10 | import logging
 11 | import json
 12 | import collections
 13 | import re
 14 | import types
 15 | import sys
 16 | import csv
 17 | 
 18 | import tabulate
 19 | 
 20 | import esbench
 21 | 
 22 | 
 23 | logger = logging.getLogger(__name__)
 24 | 
 25 | 
 26 | def _get_benchmarks(conn=None, stats_index_name=esbench.STATS_INDEX_NAME):
 27 |     """Call the ES server for raw benchmark records."""
 28 | 
 29 |     path = "%s/bench/_search?sort=benchmark_start:asc&size=100" % (stats_index_name, )
 30 |     resp = conn.get(path)
 31 |     return resp
 32 | 
 33 | 
 34 | def _benchmarks(resp, benchmark_ids=None):
 35 |     """Process response from _get_benchmarks(), yielding benchmark dicts."""
 36 | 
 37 |     if not benchmark_ids:
 38 |         benchmark_ids = ['all']
 39 | 
 40 |     data = json.loads(resp.data)
 41 | 
 42 |     try:
 43 | 
 44 |         benchmarks = data['hits']['hits']
 45 |         for benchmark_id in benchmark_ids:
 46 | 
 47 |             if benchmark_id == 'first':
 48 |                 yield benchmarks[0]
 49 |                 continue
 50 | 
 51 |             if benchmark_id == 'last':
 52 |                 yield benchmarks[-1]
 53 |                 continue
 54 | 
 55 |             try:
 56 |                 yield benchmarks[int(benchmark_id)]
 57 |                 continue
 58 |             except (ValueError, IndexError) as exc:
 59 | #                 logger.info(exc)
 60 |                 pass
 61 | 
 62 |             for benchmark in benchmarks:
 63 |                 if (benchmark_id == 'all') or (benchmark_id == benchmark['_id']):
 64 |                     yield benchmark
 65 | 
 66 |     except KeyError:
 67 |         logger.warning("no benchmarks found", exc_info=False)
 68 | 
 69 |     return
 70 | 
 71 | 
 72 | def _get_observations(conn=None, benchmark_id=None, stats_index_name=esbench.STATS_INDEX_NAME):
 73 |     """Call the ES server for raw observation records."""
 74 | 
 75 |     if not benchmark_id:
 76 |         raise ValueError("invalid 'benchmark_id'")
 77 | 
 78 |     path = "%s/obs/_search?q=meta.benchmark_id:%s&sort=meta.observation_start:asc&size=10000" % (stats_index_name, benchmark_id, )
 79 |     resp = conn.get(path)
 80 |     return resp
 81 | 
 82 | 
 83 | def _observations(resp):
 84 |     """Process response from _get_observations(), yielding observation dicts."""
 85 | 
 86 |     data = json.loads(resp.data)
 87 |     for observation in data['hits']['hits']:
 88 |         yield observation
 89 | 
 90 | 
 91 | def get_data(conn=None, benchmark_ids=None):
 92 |     """Get benchmark and observation data.
 93 | 
 94 |     Args:
 95 |         conn: instance of esbench.api.Conn
 96 |         benchmark_ids: list of str, ids of benchmarks to be included, default
 97 |             None. When None, include all benchmarks.
 98 | 
 99 |     Yields:
100 |         For each observation associated with included benchmark a dict where
101 |         top level keys are 'benchmark' and 'observation', containing pertinent
102 |         data. Effectively, data is denormalized, with the 'benchmark' element
103 |         being the same for all observations for that benchmark.
104 |     """
105 | 
106 |     for benchmark in _benchmarks(_get_benchmarks(conn=conn), benchmark_ids=benchmark_ids):
107 |         for observation in _observations(_get_observations(conn=conn, benchmark_id=benchmark['_id'])):
108 |             data = {
109 |                 "benchmark": benchmark['_source'],
110 |                 "observation": observation['_source'],
111 |             }
112 | #             data[u'benchmark'][u'_id'] = benchmark['_id']
113 | #             data[u'observation'][u'_id'] = observation['_id']
114 |             yield data
115 | 
116 | 
117 | def filter_tuples(tuples=None, pattern=".*", key_f=lambda x: x[0]):
118 |     """Return tuples matching provided regex.
119 | 
120 |     Given a list of tuples and a regex pattern string, apply the regex to
121 |     the specified tuple element, returning only tuples which match.
122 | 
123 |     Args:
124 |         tuples: list of tuples
125 |         pattern: string, regex pattern defauls to '.*'
126 |         key_f: function for selecting tuple element against which the regex
127 |             will be matched, defaults to the first element of each tuple
128 |             (lambda x: x[0])
129 | 
130 |     Returns:
131 |         sorted list of tuples where regex matched. The key for sorting is the
132 |         same key used for matching.
133 | 
134 |     Raises:
135 |         TypeError on invalid input
136 | 
137 |     """
138 | 
139 |     if type(tuples) is not list:
140 |         raise TypeError("'tuples' must be a list of tuples")
141 | 
142 |     compiled = re.compile(pattern, re.IGNORECASE)
143 |     filtered = [t for t in tuples if compiled.match(key_f(t))]
144 |     return sorted(filtered, key=key_f)
145 | 
146 | 
147 | def flatten_container(container=None):
148 |     """Flattens a container (dict, list, set).
149 | 
150 |     Args:
151 |         container: a dict, list, set, tuple, any nested combination of.
152 | 
153 |     Returns:
154 |         list of tuples (name, value) where name is a flattened, period-joined
155 |         list of dict keys and list indexes identifying the value's original
156 |         location in the source container, for example: "foo.bar.2.baz". The
157 |         'value' is an int, float, str, unicode, bool, or None.
158 | 
159 |     """
160 | 
161 |     def _flatten(container=container, prefix=None):
162 | 
163 |         if type(container) in [str, unicode, int, long, float, bool, types.NoneType]:
164 |             flat.append((prefix, container))
165 | 
166 |         elif type(container) is dict:
167 |             for key in container:
168 |                 _flatten(container=container[key], prefix=("%s.%s" % (prefix, key)) if prefix else key)
169 | 
170 |         elif type(container) in [list, set, tuple]:
171 |             for n, v in enumerate(container):
172 |                 _flatten(container=v, prefix=("%s.%i" % (prefix, n)) if prefix else str(n))
173 | 
174 |         else:
175 |             raise ValueError("cannot process element: %s" % container)
176 | 
177 |     flat = list()
178 |     _flatten()
179 | 
180 |     return flat
181 | 
182 | 
183 | def group_observations(data=None, fields=None):
184 | 
185 |     # returns a list of observations, where each observation is a list of
186 |     # (fieldname, value) tuples
187 |     data_flattened = [flatten_container(d) for d in data]
188 | 
189 |     # filter out tuples in each observation according to pattern.
190 |     data_filtered = [filter_tuples(tuples=t, pattern=fields) for t in data_flattened]
191 | 
192 |     # sort observations based on their benchmark_id and observation_sequence_no
193 |     def sort_f(d):
194 |         _d = dict(d)
195 |         return (_d['observation.meta.benchmark_id'], _d['observation.meta.observation_sequence_no'])
196 |     data_sorted = sorted(data_filtered, key=sort_f)
197 | 
198 |     # group observations into a list of benchmarks, where each benchmark is a
199 |     # list of observations, where each observation is [see comments above]
200 |     groups = [list(benchmark_obs) for benchmark_id, benchmark_obs in itertools.groupby(data_sorted, lambda x: dict(x)['observation.meta.benchmark_id'])]
201 |     # sort benchmark groups on timestamp benchmark started
202 |     groups_sorted = sorted(groups, key=lambda x: dict(x[0])['benchmark.meta.benchmark_start'])
203 | 
204 |     return groups_sorted
205 | 
206 | 
207 | FIELDS = (
208 |     "(?!observation.segments.segments)"
209 |         "("
210 |             "(benchmark.meta.benchmark_start)|"
211 |             "(observation.meta.benchmark_id)|"
212 |             "(observation.meta.observation_id)|"
213 |             "(observation.meta.observation_sequence_no)|"
214 |             "(observation.segments.num_committed_segments)|"
215 |             "(observation.segments.num_search_segments)|"
216 |             "(observation.segments.t_optimize_in_millis)|"
217 |             "(observation.stats.docs.count)|"
218 |             "(observation.stats.store.size_in_bytes)|"
219 |             "(observation.stats.fielddata.memory_size_in_bytes)|"
220 | #             "(observation.stats.search.groups.*query_time_in_millis$)"
221 |             "(observation.stats.search.groups.*query_time_in_millis_per_query$)"
222 |         ")"
223 | 
224 | )
225 | 
226 | 
227 | def output_benchmark(fh=None, fmt=None, observations=None):
228 | 
229 |     keys = [t[0] for t in observations[0]]
230 |     values = [[t[1] for t in o] for o in observations]
231 | 
232 |     if fmt == 'tab':
233 |         # shorten the keys a bit
234 |         def _shorten(s):
235 |             s = "".join([c for c in s if c not in "aeiou"])
236 |             return s
237 |         keys = [".".join(k.split(".")[-2:]) for k in keys]
238 |         print(tabulate.tabulate(values, headers=keys), file=fh)
239 | 
240 |     elif fmt == 'csv':
241 |         writer = csv.writer(fh, delimiter="\t")
242 |         writer.writerow(keys)
243 |         writer.writerows(values)
244 | 
245 |     else:
246 |         raise ValueError("unknown output format: %s" % fmt)
247 | 
248 | 
249 | def show_benchmarks(conn=None, benchmark_ids=None, fields=None, fmt=None, fh=None):
250 | 
251 |     data = list(get_data(conn=conn, benchmark_ids=benchmark_ids))
252 |     benchmarks = group_observations(data=data, fields=fields)
253 | 
254 |     for b in benchmarks:
255 |         output_benchmark(fh=fh, fmt=fmt, observations=b)
256 | 
257 | 
258 | def dump_benchmarks(conn=None, benchmark_ids=None, stats_index_name=esbench.STATS_INDEX_NAME):
259 |     """Dump benchmark data as a sequence of curl calls.
260 | 
261 |     You can save these calls to a file, and then replay them somewhere else.
262 |     """
263 | 
264 |     for benchmark in _benchmarks(_get_benchmarks(conn=conn, stats_index_name=stats_index_name), benchmark_ids):
265 |         curl = """curl -XPUT 'http://localhost:9200/%s/bench/%s' -d '%s'""" % (stats_index_name, benchmark['_id'], json.dumps(benchmark['_source']))
266 |         print(curl)
267 |         for o in _observations(_get_observations(conn, benchmark['_id'], stats_index_name=stats_index_name)):
268 |             curl = """curl -XPUT 'http://localhost:9200/%s/obs/%s' -d '%s'""" % (stats_index_name, o['_id'], json.dumps(o['_source']))
269 |             print(curl)
270 |     return
271 | 
272 | 
273 | def delete_benchmarks(conn=None, benchmark_ids=None, stats_index_name=esbench.STATS_INDEX_NAME):
274 | 
275 |     if not benchmark_ids:
276 |         resp = conn.delete(stats_index_name)
277 |         logger.info(resp.curl)
278 | 
279 |     else:
280 |         for benchmark in _benchmarks(_get_benchmarks(conn, stats_index_name=stats_index_name), benchmark_ids=benchmark_ids):
281 |             for observation in _observations(_get_observations(conn, benchmark_id=benchmark['_id'], stats_index_name=stats_index_name)):
282 |                 path = "%s/obs/%s" % (stats_index_name, observation['_id'], )
283 |                 resp = conn.delete(path)
284 |                 logger.info(resp.curl)
285 |             path = "%s/bench/%s" % (stats_index_name, benchmark['_id'], )
286 |             resp = conn.delete(path)
287 |             logger.info(resp.curl)
288 | 
289 |     return
290 | 
291 | 


--------------------------------------------------------------------------------
/esbench/api.py:
--------------------------------------------------------------------------------
  1 | # -*- coding: UTF-8 -*-
  2 | # (c)2013 Mik Kocikowski, MIT License (http://opensource.org/licenses/MIT)
  3 | # https://github.com/mkocikowski/esbench
  4 | 
  5 | """Use esbench.api.connect() to get a context manager for use with api calls.
  6 | """
  7 | 
  8 | 
  9 | import httplib
 10 | import contextlib
 11 | import logging
 12 | import json
 13 | import collections
 14 | import re
 15 | 
 16 | logger = logging.getLogger(__name__)
 17 | 
 18 | DEFAULT_TIMEOUT = 10
 19 | 
 20 | 
 21 | ApiResponse = collections.namedtuple(
 22 |     'ApiResponse', ['status', 'reason', 'data', 'curl']
 23 | )
 24 | 
 25 | 
 26 | def retry_and_reconnect_on_IOError(method):
 27 |     def wrapper(self, *args, **kwargs):
 28 |         for i in [1, 2, 5, 10, 25, 50, 75, 100]:
 29 |             try:
 30 |                 if not self.conn:
 31 |                     logger.debug("opening %s with timeout: %.2fs...", self.conn_cls, self.timeout*i)
 32 |                     self.connect(timeout=self.timeout*i)
 33 |                 res = method(self, *args, **kwargs)
 34 |                 # connections with really long timeouts should not be kept
 35 |                 # around, they are an exceptional thing
 36 |                 if i >= 25:
 37 |                     self.close()
 38 |                 return res
 39 |             except IOError as (exc):
 40 |                 logger.debug("%s (%s) in retry_and_reconnect_on_IOError (timeout: %.2fs) try: %i, pause: %is", type(exc), exc, self.timeout*i, i, 1, exc_info=False)
 41 |                 self.close()
 42 |         raise # re-raises the last exception, so most likely IOError
 43 |     return wrapper
 44 | 
 45 | 
 46 | def _massage_request_path(path):
 47 |     if not path:
 48 |         return "/"
 49 |     return "/%s" % re.search(r"^/*(.*)$", path).group(1)
 50 | 
 51 | class Conn(object):
 52 | 
 53 |     def __init__(self, host='localhost', port=9200, timeout=DEFAULT_TIMEOUT, conn_cls=httplib.HTTPConnection):
 54 |         self.host = host
 55 |         self.port = port
 56 |         self.timeout = timeout
 57 |         self.conn_cls = conn_cls
 58 |         self.conn = None
 59 | 
 60 |     def connect(self, timeout=DEFAULT_TIMEOUT):
 61 |         """Use context manager from api.connect() instead."""
 62 |         self.conn = self.conn_cls(host=self.host, port=self.port, timeout=timeout)
 63 |         self.conn.connect()
 64 | 
 65 |     def close(self):
 66 |         if self.conn: self.conn.close()
 67 |         self.conn = None
 68 | 
 69 | 
 70 |     @retry_and_reconnect_on_IOError
 71 |     def get(self, path, data=None):
 72 |         path = _massage_request_path(path)
 73 |         method = 'GET'
 74 |         if data:
 75 |             curl = "curl -X%s http://%s:%i%s -d '%s'" % (method, self.host, self.port, path, data)
 76 |             head = {'Content-type': 'application/json'}
 77 |         else:
 78 |             curl = "curl -X%s http://%s:%i%s" % (method, self.host, self.port, path)
 79 |             head = {}
 80 |         self.conn.request(method, path, data, head)
 81 |         resp = self.conn.getresponse()
 82 |         r_data = resp.read()
 83 | 
 84 |         if resp.status not in [200, 201]:
 85 |             logger.debug((resp.status, path, curl[:50]))
 86 | 
 87 |         return ApiResponse(resp.status, resp.reason, r_data, curl)
 88 | 
 89 | 
 90 |     @retry_and_reconnect_on_IOError
 91 |     def put(self, path, data):
 92 |         if not data:
 93 |             raise ValueError('data must not evaluate to false')
 94 |         path = _massage_request_path(path)
 95 |         method = 'PUT'
 96 |         curl = "curl -X%s http://%s:%i%s -d '%s'" % (method, self.host, self.port, path, data)
 97 |         head = {'Content-type': 'application/json'}
 98 |         self.conn.request(method, path, data, head)
 99 |         resp = self.conn.getresponse()
100 |         r_data = resp.read()
101 | 
102 |         if resp.status == 413:
103 |             logger.debug((resp.status, path, len(data)))
104 |         elif resp.status >= 400:
105 |             logger.debug((resp.status, path, r_data))
106 |         elif resp.status not in [200, 201]:
107 |             logger.debug((resp.status, path, curl[:50]))
108 | 
109 |         return ApiResponse(resp.status, resp.reason, r_data, curl)
110 | 
111 | 
112 |     @retry_and_reconnect_on_IOError
113 |     def post(self, path, data):
114 |         path = _massage_request_path(path)
115 |         method = 'POST'
116 |         if data:
117 |             curl = "curl -X%s http://%s:%i%s -d '%s'" % (method, self.host, self.port, path, data)
118 |             head = {'Content-type': 'application/json'}
119 |         else:
120 |             curl = "curl -X%s http://%s:%i%s" % (method, self.host, self.port, path)
121 |             head = {}
122 |         self.conn.request(method, path, data, head)
123 |         resp = self.conn.getresponse()
124 |         r_data = resp.read()
125 | 
126 |         if resp.status == 413:
127 |             logger.debug((resp.status, path, len(data)))
128 |         elif resp.status >= 400:
129 |             logger.debug((resp.status, path, r_data))
130 |         elif resp.status not in [200, 201]:
131 |             logger.debug((resp.status, path, curl[:50]))
132 | 
133 |         return ApiResponse(resp.status, resp.reason, r_data, curl)
134 | 
135 | 
136 |     @retry_and_reconnect_on_IOError
137 |     def delete(self, path):
138 |         path = _massage_request_path(path)
139 |         method = 'DELETE'
140 |         curl = "curl -X%s http://%s:%i%s" % (method, self.host, self.port, path)
141 |         self.conn.request(method, path)
142 |         resp = self.conn.getresponse()
143 |         data = resp.read()
144 | 
145 |         if resp.status not in [200, 201]:
146 |             logger.debug((resp.status, path, curl[:50]))
147 | 
148 |         return ApiResponse(resp.status, resp.reason, data, curl)
149 | 
150 | 
151 | @contextlib.contextmanager
152 | def connect(host='localhost', port=9200, timeout=DEFAULT_TIMEOUT, conn_cls=httplib.HTTPConnection):
153 |     conn = Conn(host=host, port=port, timeout=timeout, conn_cls=conn_cls)
154 |     yield conn
155 |     conn.close()
156 | 
157 | 
158 | 
159 | # def document_put(conn, index, doctype, docid, data):
160 | #     path = '%s/%s/%s' % (index, doctype, docid)
161 | #     resp = conn.put(path, data)
162 | #     return resp
163 | 
164 | 
165 | def document_post(conn, index, doctype, data):
166 |     path = '%s/%s' % (index, doctype)
167 |     resp = conn.post(path, data)
168 |     return resp
169 | 
170 | 
171 | def index_create(conn, index, config=None):
172 |     data = json.dumps(config)
173 |     resp = conn.put(index, data)
174 |     return resp
175 | 
176 | 
177 | def index_delete(conn, index):
178 |     resp = conn.delete(index)
179 |     return resp
180 | 
181 | 
182 | def index_get_stats(conn, index, groups):
183 |     path = "%s/_stats?clear=true&docs=true&store=true&search=true&merge=true&indexing=true&fielddata=true&fields=*&groups=%s" % (index, groups)
184 |     resp = conn.get(path)
185 |     return resp
186 | 
187 | 
188 | def index_set_refresh_interval(conn, index, ri):
189 |     path = "%s/_settings" % (index, )
190 |     data = '{"index": {"refresh_interval": "%s"}}' % ri
191 |     resp = conn.put(path, data)
192 |     return resp
193 | 
194 | 
195 | def index_optimize(conn, index, nseg=0):
196 |     if nseg:
197 |         path = "%s/_optimize?max_num_segments=%i&refresh=true&flush=true&wait_for_merge=true" % (index, nseg)
198 |     else:
199 |         path = "%s/_optimize?refresh=true&flush=true&wait_for_merge=true" % (index, )
200 |     resp = conn.post(path, None)
201 |     return resp
202 | 
203 | 
204 | def index_get_segments(conn, index):
205 |     path = "%s/_segments" % (index, )
206 |     resp = conn.get(path)
207 |     return resp
208 | 
209 | 
210 | def cluster_get_info(conn):
211 |     path = "_cluster/nodes?settings=true&os=true&process=true&jvm=true&thread_pool=true&network=true&transport=true&http=true&plugin=true"
212 |     resp = conn.get(path)
213 |     return resp
214 | 
215 | 
216 | def cluster_get_stats(conn):
217 |     path = "_cluster/nodes/stats?indices=true&os=true&process=true&jvm=true&network=true&transport=true&http=true&fs=true&thread_pool=true"
218 |     resp = conn.get(path)
219 |     return resp
220 | 
221 | 
222 | def cluster_get_fielddata_stats(conn):
223 |     path = "_nodes/stats/indices/fielddata/*"
224 |     resp = conn.get(path)
225 |     return resp
226 | 


--------------------------------------------------------------------------------
/esbench/bench.py:
--------------------------------------------------------------------------------
  1 | # -*- coding: UTF-8 -*-
  2 | # (c)2013 Mik Kocikowski, MIT License (http://opensource.org/licenses/MIT)
  3 | # https://github.com/mkocikowski/esbench
  4 | 
  5 | 
  6 | """A benchmark alternatively loads data and runs observations.
  7 | 
  8 | A benchmark consists of a set number of observations. Each observation
  9 | executes a set number of queries, and records execution stats. In between
 10 | observations, set amounts of data are inserted into the index. Load some data,
 11 | run some queries, record stats, repeat. All stats are stored in a separate ES
 12 | index ('esbench_stats') as doctypes 'obs' and 'bench'.
 13 | 
 14 | Classes:
 15 | 
 16 |     - SearchQuery: per-observation query wrapper
 17 |     - Observation: repeated n-times constitutes a benchmark
 18 |     - Benchmark: orchestrates data loading and observations
 19 | 
 20 | """
 21 | 
 22 | 
 23 | import logging
 24 | import json
 25 | import time
 26 | import random
 27 | import datetime
 28 | import hashlib
 29 | import string
 30 | 
 31 | import esbench.api
 32 | import esbench.data
 33 | 
 34 | 
 35 | logger = logging.getLogger(__name__)
 36 | 
 37 | 
 38 | def uuid():
 39 |     return hashlib.md5("%s%f" % (str(time.time()), random.random())).hexdigest()[:8]
 40 | 
 41 | 
 42 | # timeit.timeit('rands(6)', setup='from __main__ import rands', number=1000)
 43 | def rands(length=6):
 44 |     l = len(string.ascii_letters)-1
 45 |     s = "".join((string.ascii_letters[random.randint(0, l)] for _ in range(length)))
 46 |     return s
 47 | 
 48 | 
 49 | def timestamp(microseconds=False):
 50 |     DEFAULT_DATETIME_FORMAT = r'%Y-%m-%dT%H:%M:%SZ'
 51 |     DEFAULT_DATETIME_FORMAT_WITH_MICROSECONDS = r'%Y-%m-%dT%H:%M:%S.%fZ'
 52 |     dt = datetime.datetime.utcnow()
 53 |     if microseconds:
 54 |         s = dt.strftime(DEFAULT_DATETIME_FORMAT_WITH_MICROSECONDS)
 55 |     else:
 56 |         s = dt.strftime(DEFAULT_DATETIME_FORMAT)
 57 |     return s
 58 | 
 59 | 
 60 | class SearchQuery(object):
 61 |     """Each observation has a SearchQuery object for each bench query.
 62 | 
 63 |     The 'magic' is that each SearchQuery has a 'stats' field which is unique
 64 |     to this query / observation combination. This is what allows for tracking
 65 |     query execution stats with the 'stats group' unique to this particular
 66 |     observation.
 67 | 
 68 |     In addition, the execute() method will do basic templating, replacing the
 69 |     'variable' element in query template with a random string.
 70 | 
 71 |     """
 72 | 
 73 |     def __init__(self, name, query, observation_id, index, doctype):
 74 | 
 75 |         self.observation_id = observation_id
 76 |         self.name = name
 77 |         self.query = dict(query)
 78 |         self.execution_count = 0 # how many times it has been executed
 79 |         self.stats_group_name = "%s_%s" % (self.observation_id, self.name)
 80 |         self.query['stats'] = [self.stats_group_name]
 81 | 
 82 |         self.query_path = '%s/%s/_search' % (index, doctype)
 83 |         self.query_string = json.dumps(self.query, sort_keys=True)
 84 | 
 85 |         self.t_client = None
 86 | 
 87 | 
 88 |     def execute(self, conn):
 89 | 
 90 |         qs = self.query_string % {'variable': rands(6)}
 91 |         resp = conn.post(self.query_path, qs)
 92 |         self.execution_count += 1
 93 |         return resp
 94 | 
 95 | 
 96 | 
 97 | class Observation(object):
 98 |     """Runs specified queries and records the results.
 99 | 
100 |     Even though the queries are the same for each observation in a benchmark,
101 |     each observation creates unique stats groups for each of its queries -
102 |     this is what allows to pull stats specific to an observation from ES stats
103 |     api.
104 | 
105 |     Methods:
106 | 
107 |         _segments(): segment stats
108 |         _stats(): 'stats group' (query exec time), memory, and fielddata stats
109 |         _cluster_stats(): cluster stats snapshot
110 | 
111 |         run(): run an observation comprising of a set of queries
112 |         record(): pull in stats data and record it
113 | 
114 |     """
115 | 
116 |     _count = 0
117 | 
118 |     def __init__(
119 |             self,
120 |             conn=None,
121 |             benchmark_id=None,
122 |             queries=None,
123 |             reps=None, ):
124 | 
125 |         self.conn = conn
126 |         self.benchmark_id = benchmark_id
127 |         self.reps = reps # how many times each query will be executed
128 | 
129 |         Observation._count += 1
130 |         self.observation_sequence_no = Observation._count
131 |         self.observation_id = uuid()
132 | 
133 |         self.queries = []
134 |         for name, body in queries.items():
135 |             self.queries.append(
136 |                 SearchQuery(name, body, self.observation_id, esbench.TEST_INDEX_NAME, esbench.TEST_DOCTYPE_NAME)
137 |             )
138 | 
139 |         self.ts_start = None
140 |         self.ts_stop = None
141 |         self.t1 = time.time()
142 |         self.t_optimize = 0
143 | 
144 | 
145 |     def run(self):
146 | 
147 |         self.ts_start = timestamp()
148 |         logger.info("beginning observation no: %i, %s", self.observation_sequence_no, self.ts_start)
149 |         t1 = time.time()
150 | 
151 |         for query in self.queries:
152 |             tA = time.time()
153 |             for _ in range(self.reps):
154 |                 query.execute(self.conn)
155 |             query.t_client = time.time() - tA
156 |             logger.info("ran query '%s' %i times in %.2fs", query.name, self.reps, query.t_client)
157 | 
158 |         self.ts_stop = timestamp()
159 |         logger.info("finished observation no: %i, id: %s, time: %.3f",
160 |             self.observation_sequence_no, self.observation_id, time.time()-t1)
161 | 
162 | 
163 |     def _segments(self, segments_f=esbench.api.index_get_segments):
164 |         """Get and massage segment stats data.
165 | 
166 |         By default, the "/[index]/_segments" api end point is used. This
167 |         returns a lot of per-shard data, which gets aggregated, returning a
168 |         dictionary with following keys:
169 | 
170 |             - "num_search_segments": sum for all primaries and replicas
171 |             - "num_committed_segments": sum for all primaries and replicas
172 |             - "t_optimize": time spent on the explicit optimize call (0 if call was not made)
173 |             - "t_optimize_in_millis"
174 |             - "shards": total number of primaries and replicas
175 | 
176 |         """
177 | 
178 |         resp = segments_f(self.conn, esbench.TEST_INDEX_NAME)
179 |         _s = json.loads(resp.data)
180 | 
181 |         segments = {
182 |             "num_search_segments": sum([s['num_search_segments'] for shard in _s['indices'][esbench.TEST_INDEX_NAME]['shards'].values() for s in shard]),
183 |             "num_committed_segments": sum([s['num_committed_segments'] for shard in _s['indices'][esbench.TEST_INDEX_NAME]['shards'].values() for s in shard]),
184 |             "t_optimize": "%.2fs" % (self.t_optimize, ),
185 |             "t_optimize_in_millis": int(self.t_optimize * 1000),
186 |             "shards": sum([len(shard) for shard in _s['indices'][esbench.TEST_INDEX_NAME]['shards'].values()]),
187 |         }
188 | 
189 |         return segments
190 | 
191 | 
192 |     def _stats(self, stats_f=esbench.api.index_get_stats):
193 |         """Pull in stats group data.
194 | 
195 |         ES keeps track of stats groups (exec time etc) defined in the 'stats'
196 |         field of each query. Each observation will create unique 'stats'
197 |         values for its queries (see SearchQuery class) and so be able to see
198 |         stats just for the queries run as part of this specific observation.
199 |         This mathod retrieves the stats and parses them.
200 | 
201 |         """
202 | 
203 |         # we need to specifically ask for the stats groups we want, by name.
204 |         stats_group_names = [q.stats_group_name for q in self.queries]
205 |         resp = stats_f(self.conn, esbench.TEST_INDEX_NAME, ",".join(stats_group_names))
206 |         logger.debug("stats call: %s", resp.curl)
207 |         try:
208 |             stats = json.loads(resp.data)['indices'][esbench.TEST_INDEX_NAME]['primaries']
209 |         except KeyError: # compatibility with 19.9
210 |             stats = json.loads(resp.data)['_all']['indices'][esbench.TEST_INDEX_NAME]['primaries']
211 | 
212 |         def _remove_obs_id(s):
213 |             return "_".join(s.split("_")[1:])
214 | 
215 |         stats['search']['groups'] = {
216 |             _remove_obs_id(k): v for
217 |             k, v in stats['search']['groups'].items()
218 |         }
219 | 
220 |         for query in self.queries:
221 |             logger.debug("query %s execution count: %i", query.name, query.execution_count)
222 |             stats['search']['groups'][query.name]['client_total'] = query.execution_count
223 |             stats['search']['groups'][query.name]['client_time'] = "%.2fs" % (query.t_client, ) if query.t_client else None
224 |             stats['search']['groups'][query.name]['client_time_in_millis'] = int(query.t_client * 1000.0) if query.t_client else None
225 |             stats['search']['groups'][query.name]['client_time_in_millis_per_query'] = float(stats['search']['groups'][query.name]['client_time_in_millis']) / query.execution_count if query.execution_count else None
226 |             stats['search']['groups'][query.name]['fetch_time_in_millis_per_query'] = float(stats['search']['groups'][query.name]['fetch_time_in_millis']) / query.execution_count if query.execution_count else None
227 |             stats['search']['groups'][query.name]['query_time_in_millis_per_query'] = float(stats['search']['groups'][query.name]['query_time_in_millis']) / query.execution_count if query.execution_count else None
228 | 
229 |         return stats
230 | 
231 | 
232 |     def _cluster_stats(self, cluster_f=esbench.api.cluster_get_stats, fielddata_f=esbench.api.cluster_get_fielddata_stats):
233 | 
234 |         try:
235 |             resp = cluster_f(self.conn)
236 |             cluster_stats = json.loads(resp.data)
237 |             cluster_stats['node_count'] = len(cluster_stats['nodes'].keys())
238 | 
239 |             # the reason why getting fielddata here is not redundant with the
240 |             # fielddata gathered in _stats is that here information is
241 |             # gathered on per-node basis, which, in a multi-node setup, may be
242 |             # interesting.
243 |             #
244 |             try:
245 |                 resp = fielddata_f(self.conn)
246 |                 fielddata_stats = json.loads(resp.data)
247 | 
248 |                 for node in cluster_stats['nodes']:
249 |                     try:
250 |                         cluster_stats['nodes'][node]['indices']['fielddata']['fields'] = fielddata_stats['nodes'][node]['indices']['fielddata']['fields']
251 |                     except KeyError:
252 |                         logger.warning("couldn't get fielddata stats for node: %s", node)
253 | 
254 |             except (TypeError, IOError) as exc:
255 |                 logger.warning("couldn't get cluster fielddata stats: %s", exc)
256 | 
257 |         except (TypeError, IOError) as exc:
258 |             logger.warning("couldn't get cluster stats: %s", exc)
259 |             cluster_stats = None
260 | 
261 |         return cluster_stats
262 | 
263 | 
264 |     def record(self):
265 | 
266 |         t_total = time.time() - self.t1
267 |         obs = {
268 |             'meta': {
269 |                 'benchmark_id': self.benchmark_id,
270 |                 'observation_id': self.observation_id,
271 |                 'observation_sequence_no': self.observation_sequence_no,
272 |                 'observation_start': self.ts_start,
273 |                 'observation_stop': self.ts_stop,
274 |                 't_total': "%.2fm" % (t_total / 60.0),
275 |                 't_total_in_millis': int(t_total * 1000),
276 |             },
277 |             'segments': self._segments(),
278 |             'stats': self._stats(),
279 |             'cluster': self._cluster_stats(),
280 |         }
281 | 
282 |         data = json.dumps(obs, sort_keys=True)
283 |         path = '%s/obs/%s' % (esbench.STATS_INDEX_NAME, self.observation_id, )
284 |         resp = self.conn.put(path, data)
285 |         if resp.status not in [200, 201]:
286 |             logger.error(resp)
287 |         logger.info("recorded observation into: http://%s:%i/%s", self.conn.host, self.conn.port, path)
288 |         return resp
289 | 
290 | 
291 | 
292 | class Benchmark(object):
293 |     """Orchestrates the loading of data and running of observations. """
294 | 
295 |     def __init__(self, config=None, conn=None):
296 | 
297 |         self.benchmark_id = uuid()
298 | 
299 |         self.config = config
300 |         self.conn = conn
301 | 
302 |         self.ts_start = None
303 |         self.ts_stop = None
304 |         self.t_total = None
305 |         self.t1 = None
306 | 
307 | 
308 |     def __str__(self):
309 |         return str(self.benchmark_id)
310 | 
311 | 
312 |     def observe(self, obs_cls=Observation):
313 | 
314 |         observation = obs_cls(
315 |                         conn=self.conn,
316 |                         benchmark_id=self.benchmark_id,
317 |                         queries=self.config['queries'],
318 |                         reps=self.config['config']['reps'],
319 |         )
320 | 
321 |         if self.config['config']['segments']:
322 |             t1 = time.time()
323 |             logger.info("starting optimize call...")
324 |             resp = esbench.api.index_optimize(self.conn, esbench.TEST_INDEX_NAME, self.config['config']['segments'])
325 |             observation.t_optimize = time.time() - t1
326 |             logger.info("optimize call: %.2fs", observation.t_optimize)
327 | 
328 |         observation.run()
329 |         observation.record()
330 | 
331 |         return observation
332 | 
333 | 
334 |     def prepare(self):
335 | 
336 |         self.ts_start = timestamp()
337 |         self.t1 = time.time()
338 | 
339 | 
340 |     def load(self, lines):
341 | 
342 |         count = 0
343 |         size_b = 0
344 |         logger.debug("begining data load...")
345 |         for line in lines:
346 |             size_b += len(line)
347 |             resp = esbench.api.document_post(self.conn, esbench.TEST_INDEX_NAME, esbench.TEST_DOCTYPE_NAME, line)
348 |             count += 1
349 |         logger.info("loaded %i lines into index '%s', size: %i (%.2fMB)", count, esbench.TEST_INDEX_NAME, size_b, size_b/(1<<20))
350 |         return (count, size_b)
351 | 
352 | 
353 |     def run(self, batches):
354 | 
355 |         index_settings = {"settings" : {"index" : {"number_of_shards" : 1, "number_of_replicas" : 0}}}
356 |         esbench.api.index_create(self.conn, esbench.STATS_INDEX_NAME, index_settings)
357 | 
358 |         if not self.config['config']['append']:
359 |             esbench.api.index_delete(self.conn, esbench.TEST_INDEX_NAME)
360 |             esbench.api.index_create(self.conn, esbench.TEST_INDEX_NAME, self.config['index'])
361 | 
362 |         total_count = 0
363 |         total_size_b = 0
364 |         for batch in batches:
365 |             count, size_b = self.load(batch)
366 |             if not count:
367 |                 break
368 |             total_count += count
369 |             total_size_b += size_b
370 |             self.observe()
371 | 
372 |         logger.info("load complete; loaded total %i lines into index '%s', total size: %i (%.2fmb)", total_count, esbench.TEST_INDEX_NAME, total_size_b, total_size_b/(1<<20))
373 | 
374 | 
375 |     def _get_cluster_info(self, cluster_f=esbench.api.cluster_get_info):
376 | 
377 |         try:
378 |             resp = cluster_f(self.conn)
379 |             cluster_info = json.loads(resp.data)
380 |             cluster_info['node_count'] = len(cluster_info['nodes'].keys())
381 | 
382 |         except (TypeError, IOError) as exc:
383 |             logger.warning("couldn't get cluster info: %s", exc)
384 |             cluster_info = None
385 | 
386 |         return cluster_info
387 | 
388 | 
389 |     def record(self):
390 | 
391 |         self.ts_stop = timestamp()
392 |         self.t_total = time.time() - self.t1
393 | 
394 |         stat = {
395 |             'meta': {
396 |                 'benchmark_id': self.benchmark_id,
397 |                 'benchmark_name': self.config['config']['name'],
398 |                 'benchmark_start': self.ts_start,
399 |                 'benchmark_stop': self.ts_stop,
400 |                 't_total': "%.2fm" % (self.t_total / 60.0),
401 |                 't_total_in_millis': int(self.t_total * 1000),
402 |                 'config': json.dumps(self.config, sort_keys=True),
403 |             },
404 | 
405 |             'cluster': self._get_cluster_info(),
406 |         }
407 | 
408 |         data = json.dumps(stat, sort_keys=True)
409 |         path = '%s/bench/%s' % (esbench.STATS_INDEX_NAME, self,)
410 |         resp = self.conn.put(path, data)
411 |         if resp.status not in [200, 201]:
412 |             raise IOError("failed to record benchmark")
413 |         logger.info("recorded benchmark into: http://%s:%i/%s", self.conn.host, self.conn.port, path)
414 |         return resp
415 | 
416 | 


--------------------------------------------------------------------------------
/esbench/client.py:
--------------------------------------------------------------------------------
  1 | # -*- coding: UTF-8 -*-
  2 | # (c)2013 Mik Kocikowski, MIT License (http://opensource.org/licenses/MIT)
  3 | # https://github.com/mkocikowski/esbench
  4 | 
  5 | 
  6 | import argparse
  7 | import itertools
  8 | import logging
  9 | import contextlib
 10 | import sys
 11 | import os.path
 12 | import socket
 13 | import json
 14 | 
 15 | import esbench.api
 16 | import esbench.analyze
 17 | import esbench.bench
 18 | 
 19 | 
 20 | logger = logging.getLogger(__name__)
 21 | 
 22 | def args_parser():
 23 | 
 24 |     epilog = """
 25 | To get help for a command, do 'esbench command -h'
 26 | 
 27 | Other commands:
 28 | curl -XDELETE localhost:9200/esbench_* # delete existing benchmarks
 29 | \t
 30 | """
 31 | 
 32 |     parser = argparse.ArgumentParser(description="Elasticsearch benchmark runner (%s)" % (esbench.__version__, ), epilog=epilog, formatter_class=argparse.RawDescriptionHelpFormatter)
 33 |     parser.add_argument('-v', '--version', action='version', version=esbench.__version__)
 34 |     subparsers = parser.add_subparsers(dest='command', title='commands')
 35 | 
 36 |     epilog_run = """
 37 | """
 38 | 
 39 |     parser_run = subparsers.add_parser('run', help='run a benchmark', epilog=epilog_run, formatter_class=argparse.RawDescriptionHelpFormatter)
 40 |     parser_run.add_argument('-v', '--verbose', action='store_true')
 41 |     parser_run.add_argument('--host', type=str, default='localhost', help='elasticsearch host; (%(default)s)')
 42 |     parser_run.add_argument('--port', type=int, default=9200, help='elasticsearch port; (%(default)s)')
 43 | 
 44 |     parser_run.add_argument('--segments', type=int, metavar='N', default=None, help='if set, run optimize before each observation')
 45 |     parser_run.add_argument('--shards', metavar='N', action='store', type=int, help="create test index with N primaries")
 46 |     parser_run.add_argument('--observations', metavar='N', type=int, default=None, help='run n observations')
 47 |     parser_run.add_argument('--reps', metavar='N', type=int, default=None, help='run each query n times per observation')
 48 | 
 49 |     parser_run.add_argument('--no-load', action='store_true', help="if set, do not load data, just run observations")
 50 |     parser_run.add_argument('--append', action='store_true', help="if set, append data to the index; (%(default)s)")
 51 |     parser_run.add_argument('--data', metavar='PATH', type=str, action='store', default=None, help="read data from PATH; set to /dev/stdin to read from stdin. Set this only if you want to provide your own data, by default US Patent Application data will be used; (%(default)s)")
 52 | 
 53 |     parser_run.add_argument('--config-file-path', metavar='', type=str, default='%s/config.json' % (os.path.dirname(os.path.abspath(__file__)), ), help="path to json config file; (%(default)s)")
 54 |     parser_run.add_argument('--name', type=str, action='store', default="%s::%s" % (socket.gethostname(), esbench.bench.timestamp()), help="human readable name of the benchmark; (%(default)s)")
 55 |     parser_run.add_argument('maxsize', nargs="?", type=str, default='1mb', help="max size of the index, as either the number of documents or byte size. To index 100 documents, set it to 100; to index 1gb of documents, set it to 1gb. When setting the byte size of data, best effort will be made to run observations at even intervals, and the index bytesize will be ballpark, not the exact figure you specified. The default USPTO Patent Application data set has 123GB of data / 2.5m documents, so if you want more, you'll need to look elsewhere (or feed the same data in more than once); (%(default)s)")
 56 | 
 57 |     epilog_show = """
 58 | Sample use:
 59 | 
 60 | # output tabulated, all benchmarks, default fields, to stdout:
 61 | esbench show
 62 | 
 63 | # write csv-formatted data for benchmark bd97da35 to file foo.csv:
 64 | esbench show bd97da35 > foo.csv
 65 | 
 66 | # see fieldnames for default setting of '--fields':
 67 | esbench show | head -1 | tr '\\t' '\\n'
 68 | 
 69 | # see all possible fieldnames:
 70 | esbench show --fields '.*' | head -1 | tr '\\t' '\\n'
 71 | 
 72 | # plot optimize time vs data size in gnuplot and open resulting graph in google chrome (on osx):
 73 | esbench show > /tmp/esbench.csv && gnuplot -e "set terminal svg size 1000, 1000; set xlabel 'observation number'; plot '/tmp/esbench.csv' using 4:5 with fsteps title columnheader, '' using 4:(column(7)/(1000)) with fsteps title 'observation.segments.t_optimize_in_millis (SECONDS)'" > /tmp/esbench.svg && open -a 'Google Chrome' '/tmp/esbench.svg'
 74 | 
 75 | # plot basic data in gnuplot and open resulting graph in google chrome (on osx):
 76 | esbench show > /tmp/esbench.csv && gnuplot -e "set terminal svg size 1000, 1000; set xlabel 'observation number'; plot for [col=10:12] '/tmp/esbench.csv' using 4:col with lines lw 3 title columnheader, '' using 4:5 with fsteps title columnheader, '' using 4:6 with fsteps title columnheader, '' using 4:(column(9)/(2**20)) with fsteps title 'observation.stats.fielddata.memory_size_in_bytes (MB)', '' using 4:(column(13)/(2**30)) with fsteps title 'observation.stats.store.size_in_bytes (GB)'" > /tmp/esbench.svg && open -a 'Google Chrome' '/tmp/esbench.svg'
 77 | 
 78 | In general, if the 'canned' fields / graph do not meet your needs, dump all
 79 | the fields into a csv and graph /analyze it with whatever you have.
 80 | \t
 81 | """
 82 | 
 83 | 
 84 |     parser_show = subparsers.add_parser('show', help='show data from recorded benchmarks', epilog=epilog_show, formatter_class=argparse.RawDescriptionHelpFormatter)
 85 |     parser_show.add_argument('-v', '--verbose', action='store_true')
 86 |     parser_show.add_argument('--host', type=str, default='localhost', help='elasticsearch host; (%(default)s)')
 87 |     parser_show.add_argument('--port', type=int, default=9200, help='elasticsearch port; (%(default)s)')
 88 |     parser_show.add_argument('--format', choices=['csv', 'tab'], default='csv', help="output format; (%(default)s)")
 89 |     parser_show.add_argument('--fields', metavar='REGEX', type=str, action='store', default=esbench.analyze.FIELDS, help='default: %(default)s')
 90 |     parser_show.add_argument('ids', nargs='*', default=['all'], help='benchmark ids; (default: show all benchmarks)')
 91 | 
 92 |     parser_dump = subparsers.add_parser('dump', help='curl dump recorded benchmarks')
 93 |     parser_dump.add_argument('-v', '--verbose', action='store_true')
 94 |     parser_dump.add_argument('--host', type=str, default='localhost', help='elasticsearch host; (%(default)s)')
 95 |     parser_dump.add_argument('--port', type=int, default=9200, help='elasticsearch port; (%(default)s)')
 96 |     parser_dump.add_argument('ids', nargs='*', default=['all'], help='benchmark ids; (default: show all benchmarks)')
 97 | 
 98 |     return parser
 99 | 
100 | 
101 | def parse_maxsize(value):
102 | 
103 |     max_n = 0
104 |     max_byte_size = 1 << 20
105 |     try:
106 |         max_n = int(value.strip())
107 |         max_byte_size = 0
108 |     except ValueError:
109 |         max_n = 0
110 |         orders = {'kb': 10, 'mb': 20, 'gb': 30, 'tb': 40}
111 |         max_byte_size = int(value[:-2]) << orders[value[-2:].lower()]
112 |     logger.debug("Parsed maxsize; max_n: %i, max_byte_size: %i", max_n, max_byte_size)
113 |     return max_n, max_byte_size
114 | 
115 | 
116 | def load_config(path):
117 | 
118 |     c = None
119 |     with open(path, 'rU') as f:
120 |         c = json.loads(f.read())
121 | 
122 |     return c
123 | 
124 | 
125 | def merge_config(argv, config):
126 |     """Merge the config file with the command line arguments.
127 | 
128 |     Command line arguments override the config file parameters, unless they
129 |     are None. If a command line argument exists which isn't defined in the
130 |     'config' element of the config, it gets set regradless of its value (this
131 |     includes None).
132 | 
133 |     """
134 | 
135 |     c = config['config']
136 |     for k, v in argv.__dict__.items():
137 |         if k in c and v is not None:
138 |             c[k] = v
139 |         elif k not in c:
140 |             c[k] = v
141 | 
142 |     c['max_n'], c['max_byte_size'] = parse_maxsize(c['maxsize'])
143 | 
144 |     if argv.shards:
145 |         config['index']['settings']['index']['number_of_shards'] = argv.shards
146 | 
147 |     return config
148 | 
149 | 
150 | def main():
151 | 
152 |     args = args_parser().parse_args()
153 | 
154 |     if args.verbose: logging.basicConfig(
155 |         level=logging.DEBUG,
156 |         format='%(asctime)s %(process)d %(name)s.%(funcName)s:%(lineno)d %(levelname)s %(message)s')
157 |     else: logging.basicConfig(level=logging.INFO)
158 | 
159 |     with esbench.api.connect(host=args.host, port=args.port) as conn:
160 | 
161 |         try:
162 | 
163 |             if args.command == 'run':
164 | 
165 |                 config = merge_config(args, load_config(args.config_file_path))
166 |                 benchmark = esbench.bench.Benchmark(config=config, conn=conn)
167 |                 benchmark.prepare()
168 |                 if config['config']['no_load']:
169 |                     for _ in range(config['config']['observations']):
170 |                         benchmark.observe()
171 |                 else:
172 |                     with esbench.data.feed(path=config['config']['data']) as feed:
173 |                         batches = esbench.data.batches_iterator(lines=feed, batch_count=config['config']['observations'], max_n=config['config']['max_n'], max_byte_size=config['config']['max_byte_size'])
174 |                         benchmark.run(batches)
175 | 
176 |                 benchmark.record()
177 | 
178 |             elif args.command == 'show':
179 |                 esbench.analyze.show_benchmarks(conn=conn, benchmark_ids=args.ids, fields=args.fields, fmt=args.format, fh=sys.stdout)
180 | 
181 |             elif args.command == 'dump':
182 |                 esbench.analyze.dump_benchmarks(conn=conn, benchmark_ids=args.ids)
183 | 
184 |         except IOError as exc:
185 |             logger.debug(exc, exc_info=False)
186 |         except Exception as exc:
187 |             logger.error(exc, exc_info=True)
188 | 
189 | 
190 | if __name__ == "__main__":
191 |     main()
192 | 
193 | 


--------------------------------------------------------------------------------
/esbench/config.json:
--------------------------------------------------------------------------------
 1 | {
 2 | 
 3 |     "queries": {
 4 | 
 5 |         "match_description": {
 6 |             "fields": [
 7 |                 "description"
 8 |             ], 
 9 |             "from": 0, 
10 |             "query": {
11 |                 "match": {
12 |                     "description": "computing V%(variable)s device portable"
13 |                 }
14 |             }, 
15 |             "size": 10
16 |         }, 
17 | 
18 |         "match_description_facet_date_histogram": {
19 |             "facets": {
20 |                 "date_published": {
21 |                     "date_histogram": {
22 |                         "field" : "dates.date_published",
23 |                         "interval" : "month"
24 |                     }
25 |                 }
26 |             }, 
27 |             "fields": [
28 |                 "description"
29 |             ], 
30 |             "from": 0, 
31 |             "query": {
32 |                 "match": {
33 |                     "description": "computing V%(variable)s device portable"
34 |                 }
35 |             }, 
36 |             "size": 10
37 |         }, 
38 |         
39 |         "match_description_sorted_abstract": {
40 |             "fields": [
41 |                 "description"
42 |             ], 
43 |             "from": 0, 
44 |             "query": {
45 |                 "match": {
46 |                     "description": "computing V%(variable)s device portable"
47 |                 }
48 |             }, 
49 |             "size": 10, 
50 |             "sort": [
51 |                 {
52 |                     "abstract": "desc"
53 |                 }
54 |             ]
55 |         } 
56 |     }, 
57 | 
58 |     "index": {
59 |         "mappings": {
60 |             "doc": {
61 |                 "_size": {
62 |                     "enabled": true, 
63 |                     "store": "yes"
64 |                 }, 
65 |                 "_source": {
66 |                     "enabled": true
67 |                 }, 
68 |                 "properties": {
69 |                     "abstract": {
70 |                         "store": "yes", 
71 |                         "type": "string"
72 |                     }
73 |                 }
74 |             }
75 |         }, 
76 |         "settings": {
77 |             "index": {
78 |                 "number_of_replicas": 0, 
79 |                 "number_of_shards": 1
80 |             }
81 |         }
82 |     }, 
83 | 
84 |     "config": {
85 |         "observations": 10, 
86 |         "segments": null, 
87 |         "reps": 100, 
88 |         "append": false
89 |     } 
90 |     
91 | }
92 | 
93 | 


--------------------------------------------------------------------------------
/esbench/data.py:
--------------------------------------------------------------------------------
  1 | # -*- coding: UTF-8 -*-
  2 | # (c)2013 Mik Kocikowski, MIT License (http://opensource.org/licenses/MIT)
  3 | # https://github.com/mkocikowski/esbench
  4 | 
  5 | """Functions for downloading sample data, and for iterating over input to
  6 | create batches of documents, based on counts or on byte sizes. """
  7 | 
  8 | import os.path
  9 | import logging
 10 | import argparse
 11 | import sys
 12 | # import urllib2
 13 | import httplib
 14 | # import urlparse
 15 | import gzip
 16 | import itertools
 17 | import string
 18 | import contextlib
 19 | import collections
 20 | import time
 21 | 
 22 | import esbench
 23 | import esbench.api
 24 | 
 25 | logger = logging.getLogger(__name__)
 26 | 
 27 | # URL = "https://s3-us-west-1.amazonaws.com/esbench/assn_%s.gz"
 28 | URL_TEMPLATE = "https://s3-us-west-1.amazonaws.com/esbench/appl_%i_%s.gz"
 29 | 
 30 | def _aa(count=None):
 31 |     i = ("".join(i) for i in itertools.product(string.lowercase, repeat=2))
 32 |     if count:
 33 |         i = itertools.islice(i, count)
 34 |     return i
 35 | 
 36 | 
 37 | def urls(url_template=None, count=75):
 38 |     # default count=75 because that is the max number of yearly appl files
 39 |     for year in range(2005, 2013):
 40 |         for postfix in _aa(count):
 41 |             yield (url_template % (year, postfix))
 42 | 
 43 | 
 44 | def download(url, tmpd="/tmp", timeout=1):
 45 | 
 46 |     fn = os.path.basename(url)
 47 |     fn = os.path.abspath(os.path.join(tmpd, fn))
 48 | 
 49 |     logger.info("downloading '%s' to '%s'...", url, fn)
 50 |     t1 = time.time()
 51 | 
 52 |     # if the file already exists, don't download it again
 53 |     if os.path.exists(fn):
 54 |         logger.info("using cached file '%s'", fn)
 55 |         return fn
 56 | 
 57 |     try: 
 58 | 
 59 |         with esbench.api.connect(
 60 |                 host='s3-us-west-1.amazonaws.com', 
 61 |                 port=443, 
 62 |                 timeout=timeout, 
 63 |                 conn_cls=httplib.HTTPSConnection) as conn: 
 64 |     
 65 |             path = "/".join(url.split("/")[3:])
 66 |             resp = conn.get(path)
 67 |             logger.info("done downloading %s, time: %.2fs", url, time.time()-t1)
 68 |     
 69 |         if resp.status != 200: 
 70 |             raise IOError("resonse code %i, reason: %s" % (resp.status, resp.reason))
 71 |     
 72 |         fn = os.path.basename(url)
 73 |         fn = os.path.abspath(os.path.join(tmpd, fn))
 74 |         with open(fn, 'w') as f:
 75 |             f.write(resp.data)
 76 |     
 77 |         return fn
 78 |     
 79 |     except (IOError,) as exc: 
 80 |         logger.warning(exc, exc_info=True)
 81 |         return None
 82 | 
 83 | 
 84 | # def download(url, tmpd="/tmp"):
 85 | # 
 86 | #     fn = os.path.basename(url)
 87 | #     fn = os.path.abspath(os.path.join(tmpd, fn))
 88 | # 
 89 | #     logger.info("Downloading '%s' to '%s'", url, fn)
 90 | # 
 91 | #     # if the file already exists, don't download it again
 92 | #     if os.path.exists(fn):
 93 | #         logger.info("Using cached file '%s'", fn)
 94 | #         return fn
 95 | # 
 96 | #     try:
 97 | #         resp = urllib2.urlopen(url)
 98 | #         with open(fn, 'w') as f:
 99 | #             chunk = resp.read(2**16)
100 | #             while chunk:
101 | #                 f.write(chunk)
102 | #                 chunk = resp.read(2**16)
103 | #                 sys.stderr.write(".")
104 | #         logger.info("finished downloading '%s'", fn)
105 | #         resp.close()
106 | # 
107 | #     except (IOError,) as exc:
108 | #         logger.debug("error %s opening url: %s", exc, url)
109 | #         fn = None
110 | # 
111 | #     return fn
112 | # 
113 | 
114 | 
115 | def unzip(fn):
116 | 
117 |     with gzip.open(fn, 'rb') as f:
118 |         for line in f:
119 |             yield(line.strip())
120 | 
121 | 
122 | def get_data(nocache=False, urls_f=urls):
123 |     """Get default data provided with the benchmark (US Patent Applications).
124 | 
125 |     Returns an iterator, where each item is a json line with a complete US
126 |     Patent Application document which can be indexed into Elasticsearch. In
127 |     the background it deals with chunked downloads from S3, providing what in
128 |     essence is an 'unlimited' data source. See 'feed()' function below.
129 | 
130 |     """
131 | 
132 |     for url in urls_f(URL_TEMPLATE):
133 |         fn = download(url)
134 |         if not fn:
135 |             # download() will return None if data can't be downloaded, in that
136 |             # case just go to the next url
137 |             logger.debug("failed to download '%s', moving on", url)
138 |             continue
139 |         try:
140 |             for line in unzip(fn):
141 |                 yield line
142 |         except IOError:
143 |             logger.error("IOError reading file: '%s'. Looks like the cached data file is corrupted, it will now be removed, and downloaded again on the next test run. Moving on to the next data file - this error will not affect the test run.", fn)
144 |             nocache = True # this will remove the file in finally clause
145 |         finally:
146 |             if nocache:
147 |                 os.remove(fn)
148 |                 logger.info("removed file '%s'", fn)
149 | 
150 | 
151 | @contextlib.contextmanager
152 | def feed(path=None, lines_i=None, data_f=get_data):
153 |     """Return an iterator with data to be fed into the index.
154 | 
155 |     Given a source of data, return a safe iterator over that data. Data can
156 |     come from one of 3 sources, provided in the parameters. The idea is that
157 |     you use this function to provide the 'lines' iterator for the
158 |     esbench.data.batches_iterator function below. This function is wrapped in
159 |     a context manager, so you would use it like so:
160 | 
161 |         with feed() as f:
162 |             for line in f:
163 |                 print(line)
164 | 
165 |         Args:
166 |             path: path to a file (can be '/dev/stdin'). When provided, lines
167 |                 will be read from the file. The context manager ensures that
168 |                 the file is closed properly when done.
169 |             lines_i: iterator, yielding lines
170 |             data_f: generator function, when called yields lines
171 | 
172 |     """
173 | 
174 |     if lines_i:
175 |         if not isinstance(lines_i, collections.Iterable):
176 |             raise TypeError("'lines_i' must be iterable")
177 |         yield lines_i
178 |     elif path:
179 |         with open(path, 'rU') as lines_i:
180 |             yield lines_i
181 |     else:
182 |         yield data_f()
183 | 
184 |     # no cleanup needed
185 |     logger.debug("exit feed context manager")
186 |     pass
187 | 
188 | 
189 | 
190 | def batch_iterator(lines=None, max_batch_n=0, max_batch_byte_size=0):
191 |     """Yields up to n lines, or up to x byte size of data.
192 | 
193 |     Given an iterator, yields an iterator which will pass through the data up
194 |     to n lines, or until specified byte size of data has been passed.
195 | 
196 |     Args:
197 |         lines: iterator
198 |         max_batch_n: max number of lines to yield from the iterator
199 |         max_batch_byte_size: max byte size of data to yield. The actual amount
200 |             of data yielded will be greater than the amount specified - the
201 |             iterator doesn't have a look ahead capacity, so by the time it
202 |             knows the size of the next item it is too late to put it back in
203 |             if it it too large. Sorry.
204 | 
205 |     Yields:
206 |         items from the provided iterator
207 | 
208 |     """
209 | 
210 |     curr_n = 0
211 |     curr_byte_size = 0
212 | 
213 |     while ((max_batch_n and (curr_n < max_batch_n)) or
214 |            (max_batch_byte_size and (curr_byte_size < max_batch_byte_size)) ):
215 | 
216 |         line = next(lines)
217 |         curr_n += 1
218 |         curr_byte_size += len(line)
219 | 
220 |         yield line
221 | 
222 | 
223 | def batches_iterator(lines=None, batch_count=0, max_n=0, max_byte_size=0):
224 |     """Yields n batches of lines.
225 | 
226 |     Each batch is an iterator containing either n lines, or lines of certain
227 |     combined byte size. You must provide batch_count, and either max_n or
228 |     max_byte_size. If you provide max_n, then each batch will contain
229 |     max_n//batch_count lines. If you provide max_byte_size, then each batch
230 |     will contain lines whose total byte size is approximately
231 |     max_byte_size//batch_count.
232 | 
233 |         Args:
234 |             lines: iterator of lines, get it from esbench.data.feed()
235 |             batch_count: int, number of batches
236 |             max_n: total number of documents in all batches
237 |             max_byte_size: total byte size of all documents in all batches
238 | 
239 |         Yields:
240 |             batches, which themselves are iterators of lines.
241 | 
242 |         Raises:
243 |             ValueError: neither max_n not max_byte_size specified
244 | 
245 |     """
246 | 
247 |     if not (max_n or max_byte_size):
248 |         raise ValueError("must specify either max_n or max_byte_size")
249 | 
250 |     for _ in range(batch_count):
251 |         yield batch_iterator(
252 |                 lines=lines,
253 |                 max_batch_n=max_n//batch_count,
254 |                 max_batch_byte_size=max_byte_size//batch_count,
255 |         )
256 | 
257 | 
258 | 
259 | def args_parser():
260 |     parser = argparse.ArgumentParser(description="esbench USPTO patent assignment downloader.")
261 |     parser.add_argument('-v', '--version', action='version', version=esbench.__version__)
262 |     parser.add_argument('--nocache', action='store_true', help="if set, delete downloaded data (default: %(default)s)")
263 |     return parser
264 | 
265 | 
266 | def main():
267 | 
268 |     logging.basicConfig(level=logging.DEBUG)
269 |     args = args_parser().parse_args()
270 | 
271 |     try:
272 |         with feed() as f:
273 |             for line in f:
274 |                 print(line)
275 | 
276 |     except IOError as exc:
277 |         logger.warning(exc)
278 |         pass
279 | 
280 | 
281 | if __name__ == "__main__":
282 |     main()
283 | 
284 | 


--------------------------------------------------------------------------------
/esbench/test/__init__.py:
--------------------------------------------------------------------------------
1 | # -*- coding: UTF-8 -*-
2 | # (c)2013 Mik Kocikowski, MIT License (http://opensource.org/licenses/MIT)
3 | # https://github.com/mkocikowski/esbench
4 | 


--------------------------------------------------------------------------------
/esbench/test/test_analyze.py:
--------------------------------------------------------------------------------
  1 | # -*- coding: UTF-8 -*-
  2 | # (c)2013 Mik Kocikowski, MIT License (http://opensource.org/licenses/MIT)
  3 | # https://github.com/mkocikowski/esbench
  4 | 
  5 | import datetime
  6 | import os.path
  7 | import unittest
  8 | import json
  9 | import logging
 10 | 
 11 | import esbench.analyze
 12 | import esbench.api
 13 | import esbench.test.test_api
 14 | 
 15 | 
 16 | class ApiCallsTest(unittest.TestCase):
 17 | 
 18 |     def test_get_benchmarks(self):
 19 |         conn = esbench.api.Conn(conn_cls=esbench.test.test_api.MockHTTPConnection)
 20 |         resp = esbench.analyze._get_benchmarks(conn)
 21 |         self.assertEqual(resp.curl, 'curl -XGET http://localhost:9200/%s/bench/_search?sort=benchmark_start:asc&size=100' % esbench.STATS_INDEX_NAME)
 22 | 
 23 | 
 24 |     def test_get_observations(self):
 25 |         conn = esbench.api.Conn(conn_cls=esbench.test.test_api.MockHTTPConnection)
 26 |         resp = esbench.analyze._get_observations(conn=conn, benchmark_id='foo')
 27 |         self.assertEqual(resp.curl, 'curl -XGET http://localhost:9200/%s/obs/_search?q=meta.benchmark_id:foo&sort=meta.observation_start:asc&size=10000' % (esbench.STATS_INDEX_NAME, ))
 28 | 
 29 | 
 30 | class DataTest(unittest.TestCase):
 31 | 
 32 |     def setUp(self):
 33 |         self.tmp_get_benchmarks = esbench.analyze._get_benchmarks
 34 |         esbench.analyze._get_benchmarks = lambda **kwargs: esbench.api.ApiResponse(
 35 |             status=200,
 36 |             reason='OK',
 37 |             data = r"""{"took":0,"timed_out":false,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":1,"max_score":null,"hits":[{"_index":"esbench_stats","_type":"bench","_id":"3e878942","_score":null, "_source" : {"cluster": {"cluster_name": "elasticsearch", "nodes": {"TqybOUCJRPOvLdshFYQqtg": {"hostname": "MK.local", "http": {"bound_address": "inet[/0:0:0:0:0:0:0:0%0:9200]", "max_content_length": "100mb", "max_content_length_in_bytes": 104857600, "publish_address": "inet[/192.168.1.100:9200]"}, "http_address": "inet[/192.168.1.100:9200]", "jvm": {"mem": {"direct_max": "1019.8mb", "direct_max_in_bytes": 1069416448, "heap_init": "256mb", "heap_init_in_bytes": 268435456, "heap_max": "1019.8mb", "heap_max_in_bytes": 1069416448, "non_heap_init": "23.1mb", "non_heap_init_in_bytes": 24317952, "non_heap_max": "130mb", "non_heap_max_in_bytes": 136314880}, "pid": 15348, "start_time": 1388943758533, "version": "1.6.0_65", "vm_name": "Java HotSpot(TM) 64-Bit Server VM", "vm_vendor": "Apple Inc.", "vm_version": "20.65-b04-462"}, "name": "Space Turnip", "network": {"primary_interface": {"address": "192.168.1.100", "mac_address": "F8:1E:DF:D7:B4:4A", "name": "en1"}, "refresh_interval": 5000}, "os": {"available_processors": 2, "cpu": {"cache_size": "3kb", "cache_size_in_bytes": 3072, "cores_per_socket": 2, "mhz": 2260, "model": "MacBookPro5,5", "total_cores": 2, "total_sockets": 1, "vendor": "Intel"}, "mem": {"total": "4gb", "total_in_bytes": 4294967296}, "refresh_interval": 1000, "swap": {"total": "1gb", "total_in_bytes": 1073741824}}, "plugins": [{"description": "No description found for head.", "jvm": false, "name": "head", "site": true, "url": "/_plugin/head/"}], "process": {"id": 15348, "max_file_descriptors": 10240, "refresh_interval": 1000}, "settings": {"cluster.name": "elasticsearch", "foreground": "yes", "name": "Space Turnip", "path.home": "/Users/mkocikowski/dev/elasticsearch-0.90.6", "path.logs": "/Users/mkocikowski/dev/elasticsearch-0.90.6/logs"}, "thread_pool": {"bulk": {"max": 2, "min": 2, "queue_size": "50", "type": "fixed"}, "flush": {"keep_alive": "5m", "max": 1, "min": 1, "type": "scaling"}, "generic": {"keep_alive": "30s", "type": "cached"}, "get": {"max": 2, "min": 2, "queue_size": "1k", "type": "fixed"}, "index": {"max": 2, "min": 2, "queue_size": "200", "type": "fixed"}, "management": {"keep_alive": "5m", "max": 5, "min": 1, "type": "scaling"}, "merge": {"keep_alive": "5m", "max": 1, "min": 1, "type": "scaling"}, "optimize": {"max": 1, "min": 1, "type": "fixed"}, "percolate": {"max": 2, "min": 2, "queue_size": "1k", "type": "fixed"}, "refresh": {"keep_alive": "5m", "max": 1, "min": 1, "type": "scaling"}, "search": {"max": 6, "min": 6, "queue_size": "1k", "type": "fixed"}, "snapshot": {"keep_alive": "5m", "max": 1, "min": 1, "type": "scaling"}, "suggest": {"max": 2, "min": 2, "queue_size": "1k", "type": "fixed"}, "warmer": {"keep_alive": "5m", "max": 1, "min": 1, "type": "scaling"}}, "transport": {"bound_address": "inet[/0:0:0:0:0:0:0:0%0:9300]", "publish_address": "inet[/192.168.1.100:9300]"}, "transport_address": "inet[/192.168.1.100:9300]", "version": "0.90.6"}}, "ok": true}, "meta": {"argv": {"append": false, "command": "run", "config_file_path": "/Users/mkocikowski/dev/esbench/esbench/config.json", "data": null, "maxsize": "1mb", "name": "MK.local::2014-01-05T18:06:06Z", "no_load": false, "no_optimize_calls": false, "observations": 10, "record_segments": false, "repetitions": 100, "segments": null, "verbose": false}, "benchmark_id": "3e878942", "benchmark_name": "MK.local::2014-01-05T18:06:06Z", "benchmark_start": "2014-01-05T18:06:06Z", "benchmark_stop": "2014-01-05T18:06:30Z", "config": "{\"config\": {\"name_doctype\": \"doc\", \"name_index\": \"esbench_test\"}, \"index\": {\"mappings\": {\"doc\": {\"_size\": {\"enabled\": true, \"store\": \"yes\"}, \"_source\": {\"enabled\": true}, \"properties\": {\"abstract\": {\"store\": \"yes\", \"type\": \"string\"}}}}, \"settings\": {\"index\": {\"number_of_replicas\": 0, \"number_of_shards\": 1}}}, \"queries\": {\"match_description\": {\"fields\": [\"description\"], \"from\": 0, \"query\": {\"match\": {\"description\": \"computing V%(variable)s device portable\"}}, \"size\": 10}, \"match_description_facet_date_histogram\": {\"facets\": {\"date_published\": {\"date_histogram\": {\"field\": \"dates.date_published\", \"interval\": \"month\"}}}, \"fields\": [\"description\"], \"from\": 0, \"query\": {\"match\": {\"description\": \"computing V%(variable)s device portable\"}}, \"size\": 10}, \"match_description_sorted_abstract\": {\"fields\": [\"description\"], \"from\": 0, \"query\": {\"match\": {\"description\": \"computing V%(variable)s device portable\"}}, \"size\": 10, \"sort\": [{\"abstract\": \"desc\"}]}}}", "t_total": "0.41m", "t_total_in_millis": 24757}},"sort":[1388945166000]}]}}""",
 38 |             curl='curl "http://localhost:9200/esbench_stats/bench/_search?sort=benchmark_start:asc&size=100"',
 39 |         )
 40 |         self.tmp_get_observations = esbench.analyze._get_observations
 41 |         esbench.analyze._get_observations = lambda **kwargs: esbench.api.ApiResponse(
 42 |             status=200,
 43 |             reason='OK',
 44 | #             data='{"took":2,"timed_out":false,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":10,"max_score":null,"hits":[{"_index":"esbench_stats","_type":"obs","_id":"6ae5af7f","_score":null, "_source" : {"cluster": {"cluster_name": "elasticsearch", "nodes": {"0rNB3lMgQ3elZO7IZMROZg": {"fs": {"data": [{"available": "172.1gb", "available_in_bytes": 184896188416, "dev": "/dev/disk0s2", "disk_io_op": 8134147, "disk_io_size": "1913.4gb", "disk_io_size_in_bytes": 2054519834624, "disk_read_size": "796.4gb", "disk_read_size_in_bytes": 855137120256, "disk_reads": 3031481, "disk_write_size": "1117gb", "disk_write_size_in_bytes": 1199382714368, "disk_writes": 5102666, "free": "172.4gb", "free_in_bytes": 185158332416, "mount": "/", "path": "/Users/z013rqg/dev/elasticsearch-0.90.8/data/elasticsearch/nodes/0", "total": "233.4gb", "total_in_bytes": 250656219136}], "timestamp": 1388615489634, "total": {"available": "172.1gb", "available_in_bytes": 184896188416, "disk_io_op": 8134147, "disk_io_size": "1913.4gb", "disk_io_size_in_bytes": 2054519834624, "disk_read_size": "796.4gb", "disk_read_size_in_bytes": 855137120256, "disk_reads": 3031481, "disk_write_size": "1117gb", "disk_write_size_in_bytes": 1199382714368, "disk_writes": 5102666, "free": "172.4gb", "free_in_bytes": 185158332416, "total": "233.4gb", "total_in_bytes": 250656219136}}, "hostname": "14109fd2699b.local", "http": {"current_open": 1, "total_opened": 98}, "indices": {"completion": {"size": "0b", "size_in_bytes": 0}, "docs": {"count": 239829, "deleted": 0}, "fielddata": {"evictions": 0, "fields": {"abstract": {"memory_size": "52.6kb", "memory_size_in_bytes": 53952}, "dates.date_published": {"memory_size": "96b", "memory_size_in_bytes": 96}}, "memory_size": "52.7kb", "memory_size_in_bytes": 54048}, "filter_cache": {"evictions": 0, "memory_size": "48b", "memory_size_in_bytes": 48}, "flush": {"total": 416, "total_time": "1.9s", "total_time_in_millis": 1993}, "get": {"current": 0, "exists_time": "2ms", "exists_time_in_millis": 2, "exists_total": 8, "get_time": "2ms", "missing_time": "0s", "missing_time_in_millis": 0, "missing_total": 1, "time_in_millis": 2, "total": 9}, "id_cache": {"memory_size": "0b", "memory_size_in_bytes": 0}, "indexing": {"delete_current": 0, "delete_time": "0s", "delete_time_in_millis": 0, "delete_total": 0, "index_current": 0, "index_time": "2.3m", "index_time_in_millis": 139420, "index_total": 281312}, "merges": {"current": 0, "current_docs": 0, "current_size": "0b", "current_size_in_bytes": 0, "total": 37, "total_docs": 103136, "total_size": "138.9mb", "total_size_in_bytes": 145682835, "total_time": "18.3s", "total_time_in_millis": 18349}, "refresh": {"total": 3618, "total_time": "1.4m", "total_time_in_millis": 84817}, "search": {"fetch_current": 0, "fetch_time": "47.6s", "fetch_time_in_millis": 47699, "fetch_total": 39737, "open_contexts": 0, "query_current": 0, "query_time": "14.2s", "query_time_in_millis": 14219, "query_total": 39737}, "segments": {"count": 26}, "store": {"size": "287.3mb", "size_in_bytes": 301270141, "throttle_time": "5.2m", "throttle_time_in_millis": 317287}, "warmer": {"current": 0, "total": 3758, "total_time": "1.1s", "total_time_in_millis": 1103}}, "jvm": {"gc": {"collection_count": 23540, "collection_time": "47.3s", "collection_time_in_millis": 47306, "collectors": {"ConcurrentMarkSweep": {"collection_count": 50, "collection_time": "236ms", "collection_time_in_millis": 236}, "ParNew": {"collection_count": 23490, "collection_time": "47s", "collection_time_in_millis": 47070}}}, "mem": {"heap_committed": "254.6mb", "heap_committed_in_bytes": 267059200, "heap_max": "1011.2mb", "heap_max_in_bytes": 1060372480, "heap_used": "159.1mb", "heap_used_in_bytes": 166835000, "heap_used_percent": 15, "non_heap_committed": "82.4mb", "non_heap_committed_in_bytes": 86437888, "non_heap_used": "53mb", "non_heap_used_in_bytes": 55606888, "pools": {"CMS Old Gen": {"max": "896mb", "max_in_bytes": 939524096, "peak_max": "896mb", "peak_max_in_bytes": 939524096, "peak_used": "195.7mb", "peak_used_in_bytes": 205260360, "used": "152.5mb", "used_in_bytes": 160008408}, "CMS Perm Gen": {"max": "82mb", "max_in_bytes": 85983232, "peak_max": "82mb", "peak_max_in_bytes": 85983232, "peak_used": "43.8mb", "peak_used_in_bytes": 45966120, "used": "43.8mb", "used_in_bytes": 45966120}, "Code Cache": {"max": "48mb", "max_in_bytes": 50331648, "peak_max": "48mb", "peak_max_in_bytes": 50331648, "peak_used": "9.2mb", "peak_used_in_bytes": 9676288, "used": "9.1mb", "used_in_bytes": 9640768}, "Par Eden Space": {"max": "102.5mb", "max_in_bytes": 107479040, "peak_max": "102.5mb", "peak_max_in_bytes": 107479040, "peak_used": "16.6mb", "peak_used_in_bytes": 17432576, "used": "6.4mb", "used_in_bytes": 6721952}, "Par Survivor Space": {"max": "12.7mb", "max_in_bytes": 13369344, "peak_max": "12.7mb", "peak_max_in_bytes": 13369344, "peak_used": "2mb", "peak_used_in_bytes": 2162688, "used": "102.1kb", "used_in_bytes": 104640}}}, "threads": {"count": 113, "peak_count": 128}, "timestamp": 1388615489634, "uptime": "1.1d", "uptime_in_millis": 101172863}, "name": "Mace, Gideon", "network": {"tcp": {"active_opens": 95394, "attempt_fails": 442, "curr_estab": 92, "estab_resets": 1227, "in_errs": 282, "in_segs": 74566315, "out_rsts": -1, "out_segs": 73220868, "passive_opens": 1136, "retrans_segs": 12816}}, "os": {"cpu": {"idle": 94, "stolen": 0, "sys": 2, "usage": 4, "user": 2}, "load_average": [0.5927734375, 0.68505859375, 0.66064453125], "mem": {"actual_free": "8gb", "actual_free_in_bytes": 8644182016, "actual_used": "7.9gb", "actual_used_in_bytes": 8535687168, "free": "6.6gb", "free_in_bytes": 7093882880, "free_percent": 50, "used": "9.3gb", "used_in_bytes": 10085986304, "used_percent": 49}, "swap": {"free": "1gb", "free_in_bytes": 1157390336, "used": "944.2mb", "used_in_bytes": 990093312}, "timestamp": 1388615489633, "uptime": "17.3m", "uptime_in_millis": 1039002}, "process": {"cpu": {"percent": 3, "sys": "6.7m", "sys_in_millis": 406567, "total": "25.4m", "total_in_millis": 1527520, "user": "18.6m", "user_in_millis": 1120953}, "mem": {"resident": "459.1mb", "resident_in_bytes": 481472512, "share": "-1b", "share_in_bytes": -1, "total_virtual": "3.7gb", "total_virtual_in_bytes": 4048965632}, "open_file_descriptors": 343, "timestamp": 1388615489633}, "thread_pool": {"bulk": {"active": 0, "completed": 0, "largest": 0, "queue": 0, "rejected": 0, "threads": 0}, "flush": {"active": 0, "completed": 416, "largest": 3, "queue": 0, "rejected": 0, "threads": 0}, "generic": {"active": 0, "completed": 10497, "largest": 4, "queue": 0, "rejected": 0, "threads": 3}, "get": {"active": 0, "completed": 9, "largest": 8, "queue": 0, "rejected": 0, "threads": 8}, "index": {"active": 0, "completed": 341852, "largest": 8, "queue": 0, "rejected": 0, "threads": 8}, "management": {"active": 1, "completed": 7536, "largest": 2, "queue": 0, "rejected": 0, "threads": 2}, "merge": {"active": 0, "completed": 3659, "largest": 4, "queue": 0, "rejected": 0, "threads": 1}, "optimize": {"active": 0, "completed": 193, "largest": 1, "queue": 0, "rejected": 0, "threads": 1}, "percolate": {"active": 0, "completed": 0, "largest": 0, "queue": 0, "rejected": 0, "threads": 0}, "refresh": {"active": 0, "completed": 3582, "largest": 4, "queue": 0, "rejected": 0, "threads": 1}, "search": {"active": 0, "completed": 40422, "largest": 24, "queue": 0, "rejected": 0, "threads": 24}, "snapshot": {"active": 0, "completed": 768, "largest": 4, "queue": 0, "rejected": 0, "threads": 1}, "suggest": {"active": 0, "completed": 0, "largest": 0, "queue": 0, "rejected": 0, "threads": 0}, "warmer": {"active": 0, "completed": 4250, "largest": 2, "queue": 0, "rejected": 0, "threads": 1}}, "timestamp": 1388615490536, "transport": {"rx_count": 0, "rx_size": "0b", "rx_size_in_bytes": 0, "server_open": 13, "tx_count": 0, "tx_size": "0b", "tx_size_in_bytes": 0}, "transport_address": "inet[/192.168.1.100:9300]"}}}, "meta": {"benchmark_id": "9eed3105", "observation_id": "6ae5af7f", "observation_sequence_no": 2, "observation_start": "2014-01-01T22:31:29Z", "observation_stop": "2014-01-01T22:31:30Z"}, "segments": {"num_committed_segments": 2, "num_search_segments": 2, "segments": null, "t_optimize": "0.02s", "t_optimize_in_millis": 18}, "stats": {"docs": {"count": 11, "deleted": 0}, "fielddata": {"evictions": 0, "fields": {"abstract": {"memory_size": "52.6kb", "memory_size_in_bytes": 53952}, "dates.date_published": {"memory_size": "96b", "memory_size_in_bytes": 96}}, "memory_size": "52.7kb", "memory_size_in_bytes": 54048}, "indexing": {"delete_current": 0, "delete_time": "0s", "delete_time_in_millis": 0, "delete_total": 0, "index_current": 0, "index_time": "32ms", "index_time_in_millis": 32, "index_total": 11}, "merges": {"current": 0, "current_docs": 0, "current_size": "0b", "current_size_in_bytes": 0, "total": 0, "total_docs": 0, "total_size": "0b", "total_size_in_bytes": 0, "total_time": "0s", "total_time_in_millis": 0}, "search": {"fetch_current": 0, "fetch_time": "446ms", "fetch_time_in_millis": 446, "fetch_total": 600, "groups": {"match_description": {"client_time": "0.27s", "client_time_in_millis": 269, "client_total": 100, "fetch_current": 0, "fetch_time": "100ms", "fetch_time_in_millis": 100, "fetch_total": 100, "query_current": 0, "query_time": "22ms", "query_time_in_millis": 22, "query_total": 100}, "match_description_facet_date_histogram": {"client_time": "0.28s", "client_time_in_millis": 277, "client_total": 100, "fetch_current": 0, "fetch_time": "103ms", "fetch_time_in_millis": 103, "fetch_total": 100, "query_current": 0, "query_time": "21ms", "query_time_in_millis": 21, "query_total": 100}, "match_description_sorted_abstract": {"client_time": "0.30s", "client_time_in_millis": 297, "client_total": 100, "fetch_current": 0, "fetch_time": "107ms", "fetch_time_in_millis": 107, "fetch_total": 100, "query_current": 0, "query_time": "26ms", "query_time_in_millis": 26, "query_total": 100}}, "open_contexts": 0, "query_current": 0, "query_time": "114ms", "query_time_in_millis": 114, "query_total": 600}, "store": {"size": "294.1kb", "size_in_bytes": 301249, "throttle_time": "0s", "throttle_time_in_millis": 0}}},"sort":[1388615489000]},{"_index":"esbench_stats","_type":"obs","_id":"4391531c","_score":null, "_source" : {"cluster": {"cluster_name": "elasticsearch", "nodes": {"0rNB3lMgQ3elZO7IZMROZg": {"fs": {"data": [{"available": "172.1gb", "available_in_bytes": 184896188416, "dev": "/dev/disk0s2", "disk_io_op": 8134147, "disk_io_size": "1913.4gb", "disk_io_size_in_bytes": 2054519834624, "disk_read_size": "796.4gb", "disk_read_size_in_bytes": 855137120256, "disk_reads": 3031481, "disk_write_size": "1117gb", "disk_write_size_in_bytes": 1199382714368, "disk_writes": 5102666, "free": "172.4gb", "free_in_bytes": 185158332416, "mount": "/", "path": "/Users/z013rqg/dev/elasticsearch-0.90.8/data/elasticsearch/nodes/0", "total": "233.4gb", "total_in_bytes": 250656219136}], "timestamp": 1388615489634, "total": {"available": "172.1gb", "available_in_bytes": 184896188416, "disk_io_op": 8134147, "disk_io_size": "1913.4gb", "disk_io_size_in_bytes": 2054519834624, "disk_read_size": "796.4gb", "disk_read_size_in_bytes": 855137120256, "disk_reads": 3031481, "disk_write_size": "1117gb", "disk_write_size_in_bytes": 1199382714368, "disk_writes": 5102666, "free": "172.4gb", "free_in_bytes": 185158332416, "total": "233.4gb", "total_in_bytes": 250656219136}}, "hostname": "14109fd2699b.local", "http": {"current_open": 1, "total_opened": 98}, "indices": {"completion": {"size": "0b", "size_in_bytes": 0}, "docs": {"count": 239823, "deleted": 0}, "fielddata": {"evictions": 0, "fields": {"abstract": {"memory_size": "26.4kb", "memory_size_in_bytes": 27086}, "dates.date_published": {"memory_size": "48b", "memory_size_in_bytes": 48}}, "memory_size": "26.4kb", "memory_size_in_bytes": 27134}, "filter_cache": {"evictions": 0, "memory_size": "24b", "memory_size_in_bytes": 24}, "flush": {"total": 416, "total_time": "1.9s", "total_time_in_millis": 1993}, "get": {"current": 0, "exists_time": "2ms", "exists_time_in_millis": 2, "exists_total": 8, "get_time": "2ms", "missing_time": "0s", "missing_time_in_millis": 0, "missing_total": 1, "time_in_millis": 2, "total": 9}, "id_cache": {"memory_size": "0b", "memory_size_in_bytes": 0}, "indexing": {"delete_current": 0, "delete_time": "0s", "delete_time_in_millis": 0, "delete_total": 0, "index_current": 0, "index_time": "2.3m", "index_time_in_millis": 139399, "index_total": 281306}, "merges": {"current": 0, "current_docs": 0, "current_size": "0b", "current_size_in_bytes": 0, "total": 37, "total_docs": 103136, "total_size": "138.9mb", "total_size_in_bytes": 145682835, "total_time": "18.3s", "total_time_in_millis": 18349}, "refresh": {"total": 3617, "total_time": "1.4m", "total_time_in_millis": 84785}, "search": {"fetch_current": 0, "fetch_time": "47.3s", "fetch_time_in_millis": 47386, "fetch_total": 39437, "open_contexts": 0, "query_current": 0, "query_time": "14.1s", "query_time_in_millis": 14148, "query_total": 39437}, "segments": {"count": 24}, "store": {"size": "287mb", "size_in_bytes": 300994510, "throttle_time": "5.2m", "throttle_time_in_millis": 317287}, "warmer": {"current": 0, "total": 3756, "total_time": "1.1s", "total_time_in_millis": 1103}}, "jvm": {"gc": {"collection_count": 23540, "collection_time": "47.3s", "collection_time_in_millis": 47306, "collectors": {"ConcurrentMarkSweep": {"collection_count": 50, "collection_time": "236ms", "collection_time_in_millis": 236}, "ParNew": {"collection_count": 23490, "collection_time": "47s", "collection_time_in_millis": 47070}}}, "mem": {"heap_committed": "254.6mb", "heap_committed_in_bytes": 267059200, "heap_max": "1011.2mb", "heap_max_in_bytes": 1060372480, "heap_used": "159.1mb", "heap_used_in_bytes": 166835000, "heap_used_percent": 15, "non_heap_committed": "82.4mb", "non_heap_committed_in_bytes": 86437888, "non_heap_used": "53mb", "non_heap_used_in_bytes": 55606888, "pools": {"CMS Old Gen": {"max": "896mb", "max_in_bytes": 939524096, "peak_max": "896mb", "peak_max_in_bytes": 939524096, "peak_used": "195.7mb", "peak_used_in_bytes": 205260360, "used": "152.5mb", "used_in_bytes": 160008408}, "CMS Perm Gen": {"max": "82mb", "max_in_bytes": 85983232, "peak_max": "82mb", "peak_max_in_bytes": 85983232, "peak_used": "43.8mb", "peak_used_in_bytes": 45966120, "used": "43.8mb", "used_in_bytes": 45966120}, "Code Cache": {"max": "48mb", "max_in_bytes": 50331648, "peak_max": "48mb", "peak_max_in_bytes": 50331648, "peak_used": "9.2mb", "peak_used_in_bytes": 9676288, "used": "9.1mb", "used_in_bytes": 9640768}, "Par Eden Space": {"max": "102.5mb", "max_in_bytes": 107479040, "peak_max": "102.5mb", "peak_max_in_bytes": 107479040, "peak_used": "16.6mb", "peak_used_in_bytes": 17432576, "used": "6.4mb", "used_in_bytes": 6721952}, "Par Survivor Space": {"max": "12.7mb", "max_in_bytes": 13369344, "peak_max": "12.7mb", "peak_max_in_bytes": 13369344, "peak_used": "2mb", "peak_used_in_bytes": 2162688, "used": "102.1kb", "used_in_bytes": 104640}}}, "threads": {"count": 113, "peak_count": 128}, "timestamp": 1388615489634, "uptime": "1.1d", "uptime_in_millis": 101172863}, "name": "Mace, Gideon", "network": {"tcp": {"active_opens": 95394, "attempt_fails": 442, "curr_estab": 92, "estab_resets": 1227, "in_errs": 282, "in_segs": 74566315, "out_rsts": -1, "out_segs": 73220868, "passive_opens": 1136, "retrans_segs": 12816}}, "os": {"cpu": {"idle": 94, "stolen": 0, "sys": 2, "usage": 4, "user": 2}, "load_average": [0.5927734375, 0.68505859375, 0.66064453125], "mem": {"actual_free": "8gb", "actual_free_in_bytes": 8644182016, "actual_used": "7.9gb", "actual_used_in_bytes": 8535687168, "free": "6.6gb", "free_in_bytes": 7093882880, "free_percent": 50, "used": "9.3gb", "used_in_bytes": 10085986304, "used_percent": 49}, "swap": {"free": "1gb", "free_in_bytes": 1157390336, "used": "944.2mb", "used_in_bytes": 990093312}, "timestamp": 1388615489633, "uptime": "17.3m", "uptime_in_millis": 1039002}, "process": {"cpu": {"percent": 3, "sys": "6.7m", "sys_in_millis": 406567, "total": "25.4m", "total_in_millis": 1527520, "user": "18.6m", "user_in_millis": 1120953}, "mem": {"resident": "459.1mb", "resident_in_bytes": 481472512, "share": "-1b", "share_in_bytes": -1, "total_virtual": "3.7gb", "total_virtual_in_bytes": 4048965632}, "open_file_descriptors": 343, "timestamp": 1388615489633}, "thread_pool": {"bulk": {"active": 0, "completed": 0, "largest": 0, "queue": 0, "rejected": 0, "threads": 0}, "flush": {"active": 0, "completed": 416, "largest": 3, "queue": 0, "rejected": 0, "threads": 0}, "generic": {"active": 0, "completed": 10495, "largest": 4, "queue": 0, "rejected": 0, "threads": 3}, "get": {"active": 0, "completed": 9, "largest": 8, "queue": 0, "rejected": 0, "threads": 8}, "index": {"active": 0, "completed": 341846, "largest": 8, "queue": 0, "rejected": 0, "threads": 8}, "management": {"active": 1, "completed": 7532, "largest": 2, "queue": 0, "rejected": 0, "threads": 2}, "merge": {"active": 0, "completed": 3658, "largest": 4, "queue": 0, "rejected": 0, "threads": 1}, "optimize": {"active": 0, "completed": 192, "largest": 1, "queue": 0, "rejected": 0, "threads": 1}, "percolate": {"active": 0, "completed": 0, "largest": 0, "queue": 0, "rejected": 0, "threads": 0}, "refresh": {"active": 0, "completed": 3581, "largest": 4, "queue": 0, "rejected": 0, "threads": 1}, "search": {"active": 0, "completed": 40122, "largest": 24, "queue": 0, "rejected": 0, "threads": 24}, "snapshot": {"active": 0, "completed": 768, "largest": 4, "queue": 0, "rejected": 0, "threads": 1}, "suggest": {"active": 0, "completed": 0, "largest": 0, "queue": 0, "rejected": 0, "threads": 0}, "warmer": {"active": 0, "completed": 4248, "largest": 2, "queue": 0, "rejected": 0, "threads": 1}}, "timestamp": 1388615489633, "transport": {"rx_count": 0, "rx_size": "0b", "rx_size_in_bytes": 0, "server_open": 13, "tx_count": 0, "tx_size": "0b", "tx_size_in_bytes": 0}, "transport_address": "inet[/192.168.1.100:9300]"}}}, "meta": {"benchmark_id": "9eed3105", "observation_id": "4391531c", "observation_sequence_no": 1, "observation_start": "2014-01-01T22:31:29Z", "observation_stop": "2014-01-01T22:31:29Z"}, "segments": {"num_committed_segments": 1, "num_search_segments": 1, "segments": null, "t_optimize": "0.02s", "t_optimize_in_millis": 18}, "stats": {"docs": {"count": 6, "deleted": 0}, "fielddata": {"evictions": 0, "fields": {"abstract": {"memory_size": "26.4kb", "memory_size_in_bytes": 27086}, "dates.date_published": {"memory_size": "48b", "memory_size_in_bytes": 48}}, "memory_size": "26.4kb", "memory_size_in_bytes": 27134}, "indexing": {"delete_current": 0, "delete_time": "0s", "delete_time_in_millis": 0, "delete_total": 0, "index_current": 0, "index_time": "18ms", "index_time_in_millis": 18, "index_total": 6}, "merges": {"current": 0, "current_docs": 0, "current_size": "0b", "current_size_in_bytes": 0, "total": 0, "total_docs": 0, "total_size": "0b", "total_size_in_bytes": 0, "total_time": "0s", "total_time_in_millis": 0}, "search": {"fetch_current": 0, "fetch_time": "133ms", "fetch_time_in_millis": 133, "fetch_total": 300, "groups": {"match_description": {"client_time": "0.17s", "client_time_in_millis": 172, "client_total": 100, "fetch_current": 0, "fetch_time": "42ms", "fetch_time_in_millis": 42, "fetch_total": 100, "query_current": 0, "query_time": "15ms", "query_time_in_millis": 15, "query_total": 100}, "match_description_facet_date_histogram": {"client_time": "0.18s", "client_time_in_millis": 176, "client_total": 100, "fetch_current": 0, "fetch_time": "48ms", "fetch_time_in_millis": 48, "fetch_total": 100, "query_current": 0, "query_time": "12ms", "query_time_in_millis": 12, "query_total": 100}, "match_description_sorted_abstract": {"client_time": "0.18s", "client_time_in_millis": 178, "client_total": 100, "fetch_current": 0, "fetch_time": "43ms", "fetch_time_in_millis": 43, "fetch_total": 100, "query_current": 0, "query_time": "15ms", "query_time_in_millis": 15, "query_total": 100}}, "open_contexts": 0, "query_current": 0, "query_time": "43ms", "query_time_in_millis": 43, "query_total": 300}, "store": {"size": "149.7kb", "size_in_bytes": 153296, "throttle_time": "0s", "throttle_time_in_millis": 0}}},"sort":[1388615489000]}]}}',
 45 | #             data='{"took":2,"timed_out":false,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":10,"max_score":null,"hits":[{"_index":"esbench_stats","_type":"obs","_id":"e8cd3f18","_score":null, "_source" : {"cluster": {"cluster_name": "elasticsearch", "nodes": {"TqybOUCJRPOvLdshFYQqtg": {"fs": {"data": [{"available": "58gb", "available_in_bytes": 62311968768, "dev": "/dev/disk1s2", "disk_read_size": "4.3gb", "disk_read_size_in_bytes": 4703711744, "disk_reads": 248828, "disk_write_size": "15gb", "disk_write_size_in_bytes": 16135558656, "disk_writes": 468533, "free": "58gb", "free_in_bytes": 62311968768, "mount": "/Users/mkocikowski", "path": "/Users/mkocikowski/dev/elasticsearch-0.90.6/data/elasticsearch/nodes/0", "total": "465.1gb", "total_in_bytes": 499430719488}], "timestamp": 1388945167521}, "hostname": "MK.local", "http": {"current_open": 2, "total_opened": 10}, "indices": {"completion": {"size": "0b", "size_in_bytes": 0}, "docs": {"count": 6, "deleted": 0}, "fielddata": {"evictions": 0, "fields": {"abstract": {"memory_size": "26.4kb", "memory_size_in_bytes": 27086}, "dates.date_published": {"memory_size": "48b", "memory_size_in_bytes": 48}}, "memory_size": "26.4kb", "memory_size_in_bytes": 27134}, "filter_cache": {"evictions": 0, "memory_size": "24b", "memory_size_in_bytes": 24}, "flush": {"total": 0, "total_time": "0s", "total_time_in_millis": 0}, "get": {"current": 0, "exists_time": "1ms", "exists_time_in_millis": 1, "exists_total": 6, "get_time": "1ms", "missing_time": "0s", "missing_time_in_millis": 0, "missing_total": 0, "time_in_millis": 1, "total": 6}, "id_cache": {"memory_size": "0b", "memory_size_in_bytes": 0}, "indexing": {"delete_current": 0, "delete_time": "0s", "delete_time_in_millis": 0, "delete_total": 0, "index_current": 0, "index_time": "2.5s", "index_time_in_millis": 2558, "index_total": 147}, "merges": {"current": 0, "current_docs": 0, "current_size": "0b", "current_size_in_bytes": 0, "total": 6, "total_docs": 149, "total_size": "8.6mb", "total_size_in_bytes": 9066343, "total_time": "3s", "total_time_in_millis": 3053}, "refresh": {"total": 43, "total_time": "7.8s", "total_time_in_millis": 7826}, "search": {"fetch_current": 0, "fetch_time": "28.5s", "fetch_time_in_millis": 28550, "fetch_total": 9300, "open_contexts": 0, "query_current": 0, "query_time": "8.6s", "query_time_in_millis": 8629, "query_total": 9300}, "store": {"size": "149.6kb", "size_in_bytes": 153268, "throttle_time": "0s", "throttle_time_in_millis": 0}, "warmer": {"current": 0, "total": 3, "total_time": "0s", "total_time_in_millis": 0}}, "jvm": {"gc": {"collection_count": 1179, "collection_time": "3.9s", "collection_time_in_millis": 3911, "collectors": {"ConcurrentMarkSweep": {"collection_count": 0, "collection_time": "0s", "collection_time_in_millis": 0}, "ParNew": {"collection_count": 1179, "collection_time": "3.9s", "collection_time_in_millis": 3911}}}, "mem": {"heap_committed": "253.9mb", "heap_committed_in_bytes": 266272768, "heap_used": "137.8mb", "heap_used_in_bytes": 144510592, "non_heap_committed": "46.2mb", "non_heap_committed_in_bytes": 48500736, "non_heap_used": "46mb", "non_heap_used_in_bytes": 48261344, "pools": {"CMS Old Gen": {"max": "982.4mb", "max_in_bytes": 1030160384, "peak_max": "982.4mb", "peak_max_in_bytes": 1030160384, "peak_used": "128.7mb", "peak_used_in_bytes": 135035472, "used": "128.7mb", "used_in_bytes": 135035472}, "CMS Perm Gen": {"max": "82mb", "max_in_bytes": 85983232, "peak_max": "82mb", "peak_max_in_bytes": 85983232, "peak_used": "41.2mb", "peak_used_in_bytes": 43263328, "used": "41.2mb", "used_in_bytes": 43263328}, "Code Cache": {"max": "48mb", "max_in_bytes": 50331648, "peak_max": "48mb", "peak_max_in_bytes": 50331648, "peak_used": "4.7mb", "peak_used_in_bytes": 5008832, "used": "4.7mb", "used_in_bytes": 4998016}, "Par Eden Space": {"max": "33.3mb", "max_in_bytes": 34930688, "peak_max": "33.3mb", "peak_max_in_bytes": 34930688, "peak_used": "16.6mb", "peak_used_in_bytes": 17432576, "used": "9mb", "used_in_bytes": 9455216}, "Par Survivor Space": {"max": "4.1mb", "max_in_bytes": 4325376, "peak_max": "4.1mb", "peak_max_in_bytes": 4325376, "peak_used": "2mb", "peak_used_in_bytes": 2162688, "used": "19.4kb", "used_in_bytes": 19904}}}, "threads": {"count": 47, "peak_count": 48}, "timestamp": 1388945167520, "uptime": "23.4m", "uptime_in_millis": 1408987}, "name": "Space Turnip", "network": {"tcp": {"active_opens": 14551, "attempt_fails": 46, "curr_estab": 42, "estab_resets": 124, "in_errs": 82, "in_segs": 3108679, "out_rsts": -1, "out_segs": 2858839, "passive_opens": 289, "retrans_segs": 1272}}, "os": {"cpu": {"idle": 89, "stolen": 0, "sys": 2, "user": 7}, "load_average": [1.7255859375, 1.45556640625, 1.43994140625], "mem": {"actual_free": "1.6gb", "actual_free_in_bytes": 1718706176, "actual_used": "2.3gb", "actual_used_in_bytes": 2576261120, "free": "262.6mb", "free_in_bytes": 275419136, "free_percent": 40, "used": "3.7gb", "used_in_bytes": 4019548160, "used_percent": 59}, "swap": {"free": "1gb", "free_in_bytes": 1073741824, "used": "0b", "used_in_bytes": 0}, "timestamp": 1388945167520, "uptime": "3.8m", "uptime_in_millis": 228110}, "process": {"cpu": {"percent": 0, "sys": "13s", "sys_in_millis": 13058, "total": "1.7m", "total_in_millis": 102484, "user": "1.4m", "user_in_millis": 89426}, "mem": {"resident": "283.2mb", "resident_in_bytes": 296976384, "share": "-1b", "share_in_bytes": -1, "total_virtual": "3.6gb", "total_virtual_in_bytes": 3886198784}, "open_file_descriptors": 134, "timestamp": 1388945167520}, "thread_pool": {"bulk": {"active": 0, "completed": 0, "largest": 0, "queue": 0, "rejected": 0, "threads": 0}, "flush": {"active": 0, "completed": 0, "largest": 0, "queue": 0, "rejected": 0, "threads": 0}, "generic": {"active": 0, "completed": 252, "largest": 3, "queue": 0, "rejected": 0, "threads": 3}, "get": {"active": 0, "completed": 6, "largest": 2, "queue": 0, "rejected": 0, "threads": 2}, "index": {"active": 0, "completed": 147, "largest": 2, "queue": 0, "rejected": 0, "threads": 2}, "management": {"active": 1, "completed": 218, "largest": 2, "queue": 0, "rejected": 0, "threads": 2}, "merge": {"active": 0, "completed": 36, "largest": 1, "queue": 0, "rejected": 0, "threads": 1}, "optimize": {"active": 0, "completed": 31, "largest": 1, "queue": 0, "rejected": 0, "threads": 1}, "percolate": {"active": 0, "completed": 0, "largest": 0, "queue": 0, "rejected": 0, "threads": 0}, "refresh": {"active": 0, "completed": 36, "largest": 1, "queue": 0, "rejected": 0, "threads": 1}, "search": {"active": 0, "completed": 9300, "largest": 6, "queue": 0, "rejected": 0, "threads": 6}, "snapshot": {"active": 0, "completed": 20, "largest": 1, "queue": 0, "rejected": 0, "threads": 1}, "suggest": {"active": 0, "completed": 0, "largest": 0, "queue": 0, "rejected": 0, "threads": 0}, "warmer": {"active": 0, "completed": 74, "largest": 1, "queue": 0, "rejected": 0, "threads": 1}}, "timestamp": 1388945167520, "transport": {"rx_count": 0, "rx_size": "0b", "rx_size_in_bytes": 0, "server_open": 13, "tx_count": 0, "tx_size": "0b", "tx_size_in_bytes": 0}, "transport_address": "inet[/192.168.1.100:9300]"}}}, "meta": {"benchmark_id": "3e878942", "observation_id": "e8cd3f18", "observation_sequence_no": 1, "observation_start": "2014-01-05T18:06:06Z", "observation_stop": "2014-01-05T18:06:07Z", "t_total": "0.02m", "t_total_in_millis": 1178}, "segments": {"num_committed_segments": 1, "num_search_segments": 1, "segments": null, "t_optimize": "0.11s", "t_optimize_in_millis": 109}, "stats": {"docs": {"count": 6, "deleted": 0}, "fielddata": {"evictions": 0, "fields": {"abstract": {"memory_size": "26.4kb", "memory_size_in_bytes": 27086}, "dates.date_published": {"memory_size": "48b", "memory_size_in_bytes": 48}}, "memory_size": "26.4kb", "memory_size_in_bytes": 27134}, "indexing": {"delete_current": 0, "delete_time": "0s", "delete_time_in_millis": 0, "delete_total": 0, "index_current": 0, "index_time": "32ms", "index_time_in_millis": 32, "index_total": 6}, "merges": {"current": 0, "current_docs": 0, "current_size": "0b", "current_size_in_bytes": 0, "total": 0, "total_docs": 0, "total_size": "0b", "total_size_in_bytes": 0, "total_time": "0s", "total_time_in_millis": 0}, "search": {"fetch_current": 0, "fetch_time": "276ms", "fetch_time_in_millis": 276, "fetch_total": 300, "groups": {"match_description": {"client_time": "0.34s", "client_time_in_millis": 341, "client_total": 100, "fetch_current": 0, "fetch_time": "84ms", "fetch_time_in_millis": 84, "fetch_total": 100, "query_current": 0, "query_time": "18ms", "query_time_in_millis": 18, "query_total": 100}, "match_description_facet_date_histogram": {"client_time": "0.37s", "client_time_in_millis": 374, "client_total": 100, "fetch_current": 0, "fetch_time": "102ms", "fetch_time_in_millis": 102, "fetch_total": 100, "query_current": 0, "query_time": "22ms", "query_time_in_millis": 22, "query_total": 100}, "match_description_sorted_abstract": {"client_time": "0.35s", "client_time_in_millis": 352, "client_total": 100, "fetch_current": 0, "fetch_time": "89ms", "fetch_time_in_millis": 89, "fetch_total": 100, "query_current": 0, "query_time": "22ms", "query_time_in_millis": 22, "query_total": 100}}, "open_contexts": 0, "query_current": 0, "query_time": "62ms", "query_time_in_millis": 62, "query_total": 300}, "store": {"size": "149.5kb", "size_in_bytes": 153169, "throttle_time": "0s", "throttle_time_in_millis": 0}}},"sort":[1388945166000]},{"_index":"esbench_stats","_type":"obs","_id":"317405e7","_score":null, "_source" : {"cluster": {"cluster_name": "elasticsearch", "nodes": {"TqybOUCJRPOvLdshFYQqtg": {"fs": {"data": [{"available": "58gb", "available_in_bytes": 62311964672, "dev": "/dev/disk1s2", "disk_read_size": "4.3gb", "disk_read_size_in_bytes": 4703711744, "disk_reads": 248828, "disk_write_size": "15gb", "disk_write_size_in_bytes": 16137129984, "disk_writes": 468640, "free": "58gb", "free_in_bytes": 62311964672, "mount": "/Users/mkocikowski", "path": "/Users/mkocikowski/dev/elasticsearch-0.90.6/data/elasticsearch/nodes/0", "total": "465.1gb", "total_in_bytes": 499430719488}], "timestamp": 1388945169526}, "hostname": "MK.local", "http": {"current_open": 2, "total_opened": 10}, "indices": {"completion": {"size": "0b", "size_in_bytes": 0}, "docs": {"count": 12, "deleted": 0}, "fielddata": {"evictions": 0, "fields": {"abstract": {"memory_size": "52.6kb", "memory_size_in_bytes": 53952}, "dates.date_published": {"memory_size": "96b", "memory_size_in_bytes": 96}}, "memory_size": "52.7kb", "memory_size_in_bytes": 54048}, "filter_cache": {"evictions": 0, "memory_size": "48b", "memory_size_in_bytes": 48}, "flush": {"total": 0, "total_time": "0s", "total_time_in_millis": 0}, "get": {"current": 0, "exists_time": "1ms", "exists_time_in_millis": 1, "exists_total": 6, "get_time": "1ms", "missing_time": "0s", "missing_time_in_millis": 0, "missing_total": 0, "time_in_millis": 1, "total": 6}, "id_cache": {"memory_size": "0b", "memory_size_in_bytes": 0}, "indexing": {"delete_current": 0, "delete_time": "0s", "delete_time_in_millis": 0, "delete_total": 0, "index_current": 0, "index_time": "2.6s", "index_time_in_millis": 2639, "index_total": 153}, "merges": {"current": 0, "current_docs": 0, "current_size": "0b", "current_size_in_bytes": 0, "total": 6, "total_docs": 149, "total_size": "8.6mb", "total_size_in_bytes": 9066343, "total_time": "3s", "total_time_in_millis": 3053}, "refresh": {"total": 44, "total_time": "8s", "total_time_in_millis": 8045}, "search": {"fetch_current": 0, "fetch_time": "29.1s", "fetch_time_in_millis": 29168, "fetch_total": 9600, "open_contexts": 0, "query_current": 0, "query_time": "8.7s", "query_time_in_millis": 8741, "query_total": 9600}, "store": {"size": "523.3kb", "size_in_bytes": 535948, "throttle_time": "0s", "throttle_time_in_millis": 0}, "warmer": {"current": 0, "total": 5, "total_time": "2ms", "total_time_in_millis": 2}}, "jvm": {"gc": {"collection_count": 1206, "collection_time": "4s", "collection_time_in_millis": 4014, "collectors": {"ConcurrentMarkSweep": {"collection_count": 0, "collection_time": "0s", "collection_time_in_millis": 0}, "ParNew": {"collection_count": 1206, "collection_time": "4s", "collection_time_in_millis": 4014}}}, "mem": {"heap_committed": "253.9mb", "heap_committed_in_bytes": 266272768, "heap_used": "137.2mb", "heap_used_in_bytes": 143911208, "non_heap_committed": "46.2mb", "non_heap_committed_in_bytes": 48500736, "non_heap_used": "46mb", "non_heap_used_in_bytes": 48268352, "pools": {"CMS Old Gen": {"max": "982.4mb", "max_in_bytes": 1030160384, "peak_max": "982.4mb", "peak_max_in_bytes": 1030160384, "peak_used": "133.5mb", "peak_used_in_bytes": 140014968, "used": "133.5mb", "used_in_bytes": 140014968}, "CMS Perm Gen": {"max": "82mb", "max_in_bytes": 85983232, "peak_max": "82mb", "peak_max_in_bytes": 85983232, "peak_used": "41.2mb", "peak_used_in_bytes": 43265664, "used": "41.2mb", "used_in_bytes": 43265664}, "Code Cache": {"max": "48mb", "max_in_bytes": 50331648, "peak_max": "48mb", "peak_max_in_bytes": 50331648, "peak_used": "4.7mb", "peak_used_in_bytes": 5011328, "used": "4.7mb", "used_in_bytes": 5002688}, "Par Eden Space": {"max": "33.3mb", "max_in_bytes": 34930688, "peak_max": "33.3mb", "peak_max_in_bytes": 34930688, "peak_used": "16.6mb", "peak_used_in_bytes": 17432576, "used": "3.5mb", "used_in_bytes": 3768768}, "Par Survivor Space": {"max": "4.1mb", "max_in_bytes": 4325376, "peak_max": "4.1mb", "peak_max_in_bytes": 4325376, "peak_used": "2mb", "peak_used_in_bytes": 2162688, "used": "124.4kb", "used_in_bytes": 127472}}}, "threads": {"count": 47, "peak_count": 48}, "timestamp": 1388945169525, "uptime": "23.5m", "uptime_in_millis": 1410992}, "name": "Space Turnip", "network": {"tcp": {"active_opens": 14551, "attempt_fails": 46, "curr_estab": 42, "estab_resets": 124, "in_errs": 82, "in_segs": 3108679, "out_rsts": -1, "out_segs": 2858839, "passive_opens": 289, "retrans_segs": 1272}}, "os": {"cpu": {"idle": 33, "stolen": 0, "sys": 20, "user": 45}, "load_average": [1.7255859375, 1.45556640625, 1.43994140625], "mem": {"actual_free": "1.5gb", "actual_free_in_bytes": 1712320512, "actual_used": "2.4gb", "actual_used_in_bytes": 2582646784, "free": "256.2mb", "free_in_bytes": 268648448, "free_percent": 39, "used": "3.7gb", "used_in_bytes": 4026318848, "used_percent": 60}, "swap": {"free": "1gb", "free_in_bytes": 1073741824, "used": "0b", "used_in_bytes": 0}, "timestamp": 1388945169525, "uptime": "3.8m", "uptime_in_millis": 228112}, "process": {"cpu": {"percent": 89, "sys": "13.4s", "sys_in_millis": 13476, "total": "1.7m", "total_in_millis": 104279, "user": "1.5m", "user_in_millis": 90803}, "mem": {"resident": "287.8mb", "resident_in_bytes": 301838336, "share": "-1b", "share_in_bytes": -1, "total_virtual": "3.6gb", "total_virtual_in_bytes": 3886198784}, "open_file_descriptors": 136, "timestamp": 1388945169525}, "thread_pool": {"bulk": {"active": 0, "completed": 0, "largest": 0, "queue": 0, "rejected": 0, "threads": 0}, "flush": {"active": 0, "completed": 0, "largest": 0, "queue": 0, "rejected": 0, "threads": 0}, "generic": {"active": 0, "completed": 253, "largest": 3, "queue": 0, "rejected": 0, "threads": 3}, "get": {"active": 0, "completed": 6, "largest": 2, "queue": 0, "rejected": 0, "threads": 2}, "index": {"active": 0, "completed": 153, "largest": 2, "queue": 0, "rejected": 0, "threads": 2}, "management": {"active": 1, "completed": 222, "largest": 2, "queue": 0, "rejected": 0, "threads": 2}, "merge": {"active": 0, "completed": 37, "largest": 1, "queue": 0, "rejected": 0, "threads": 1}, "optimize": {"active": 0, "completed": 32, "largest": 1, "queue": 0, "rejected": 0, "threads": 1}, "percolate": {"active": 0, "completed": 0, "largest": 0, "queue": 0, "rejected": 0, "threads": 0}, "refresh": {"active": 0, "completed": 37, "largest": 1, "queue": 0, "rejected": 0, "threads": 1}, "search": {"active": 0, "completed": 9600, "largest": 6, "queue": 0, "rejected": 0, "threads": 6}, "snapshot": {"active": 0, "completed": 20, "largest": 1, "queue": 0, "rejected": 0, "threads": 1}, "suggest": {"active": 0, "completed": 0, "largest": 0, "queue": 0, "rejected": 0, "threads": 0}, "warmer": {"active": 0, "completed": 76, "largest": 1, "queue": 0, "rejected": 0, "threads": 1}}, "timestamp": 1388945169524, "transport": {"rx_count": 0, "rx_size": "0b", "rx_size_in_bytes": 0, "server_open": 13, "tx_count": 0, "tx_size": "0b", "tx_size_in_bytes": 0}, "transport_address": "inet[/192.168.1.100:9300]"}}}, "meta": {"benchmark_id": "3e878942", "observation_id": "317405e7", "observation_sequence_no": 2, "observation_start": "2014-01-05T18:06:07Z", "observation_stop": "2014-01-05T18:06:09Z", "t_total": "0.03m", "t_total_in_millis": 1890}, "segments": {"num_committed_segments": 2, "num_search_segments": 2, "segments": null, "t_optimize": "0.15s", "t_optimize_in_millis": 145}, "stats": {"docs": {"count": 11, "deleted": 0}, "fielddata": {"evictions": 0, "fields": {"abstract": {"memory_size": "52.6kb", "memory_size_in_bytes": 53952}, "dates.date_published": {"memory_size": "96b", "memory_size_in_bytes": 96}}, "memory_size": "52.7kb", "memory_size_in_bytes": 54048}, "indexing": {"delete_current": 0, "delete_time": "0s", "delete_time_in_millis": 0, "delete_total": 0, "index_current": 0, "index_time": "70ms", "index_time_in_millis": 70, "index_total": 11}, "merges": {"current": 0, "current_docs": 0, "current_size": "0b", "current_size_in_bytes": 0, "total": 0, "total_docs": 0, "total_size": "0b", "total_size_in_bytes": 0, "total_time": "0s", "total_time_in_millis": 0}, "search": {"fetch_current": 0, "fetch_time": "894ms", "fetch_time_in_millis": 894, "fetch_total": 600, "groups": {"match_description": {"client_time": "0.56s", "client_time_in_millis": 555, "client_total": 100, "fetch_current": 0, "fetch_time": "199ms", "fetch_time_in_millis": 199, "fetch_total": 100, "query_current": 0, "query_time": "34ms", "query_time_in_millis": 34, "query_total": 100}, "match_description_facet_date_histogram": {"client_time": "0.61s", "client_time_in_millis": 612, "client_total": 100, "fetch_current": 0, "fetch_time": "222ms", "fetch_time_in_millis": 222, "fetch_total": 100, "query_current": 0, "query_time": "37ms", "query_time_in_millis": 37, "query_total": 100}, "match_description_sorted_abstract": {"client_time": "0.58s", "client_time_in_millis": 575, "client_total": 100, "fetch_current": 0, "fetch_time": "196ms", "fetch_time_in_millis": 196, "fetch_total": 100, "query_current": 0, "query_time": "40ms", "query_time_in_millis": 40, "query_total": 100}}, "open_contexts": 0, "query_current": 0, "query_time": "174ms", "query_time_in_millis": 174, "query_total": 600}, "store": {"size": "293.9kb", "size_in_bytes": 300998, "throttle_time": "0s", "throttle_time_in_millis": 0}}},"sort":[1388945167000]}]}}',
 46 |             data=r"""{"took":1,"timed_out":false,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":10,"max_score":null,"hits":[{"_index":"esbench_stats","_type":"obs","_id":"e8cd3f18","_score":null, "_source" : {"cluster": {"cluster_name": "elasticsearch", "nodes": {"TqybOUCJRPOvLdshFYQqtg": {"fs": {"data": [{"available": "58gb", "available_in_bytes": 62311968768, "dev": "/dev/disk1s2", "disk_read_size": "4.3gb", "disk_read_size_in_bytes": 4703711744, "disk_reads": 248828, "disk_write_size": "15gb", "disk_write_size_in_bytes": 16135558656, "disk_writes": 468533, "free": "58gb", "free_in_bytes": 62311968768, "mount": "/Users/mkocikowski", "path": "/Users/mkocikowski/dev/elasticsearch-0.90.6/data/elasticsearch/nodes/0", "total": "465.1gb", "total_in_bytes": 499430719488}], "timestamp": 1388945167521}, "hostname": "MK.local", "http": {"current_open": 2, "total_opened": 10}, "indices": {"completion": {"size": "0b", "size_in_bytes": 0}, "docs": {"count": 6, "deleted": 0}, "fielddata": {"evictions": 0, "fields": {"abstract": {"memory_size": "26.4kb", "memory_size_in_bytes": 27086}, "dates.date_published": {"memory_size": "48b", "memory_size_in_bytes": 48}}, "memory_size": "26.4kb", "memory_size_in_bytes": 27134}, "filter_cache": {"evictions": 0, "memory_size": "24b", "memory_size_in_bytes": 24}, "flush": {"total": 0, "total_time": "0s", "total_time_in_millis": 0}, "get": {"current": 0, "exists_time": "1ms", "exists_time_in_millis": 1, "exists_total": 6, "get_time": "1ms", "missing_time": "0s", "missing_time_in_millis": 0, "missing_total": 0, "time_in_millis": 1, "total": 6}, "id_cache": {"memory_size": "0b", "memory_size_in_bytes": 0}, "indexing": {"delete_current": 0, "delete_time": "0s", "delete_time_in_millis": 0, "delete_total": 0, "index_current": 0, "index_time": "2.5s", "index_time_in_millis": 2558, "index_total": 147}, "merges": {"current": 0, "current_docs": 0, "current_size": "0b", "current_size_in_bytes": 0, "total": 6, "total_docs": 149, "total_size": "8.6mb", "total_size_in_bytes": 9066343, "total_time": "3s", "total_time_in_millis": 3053}, "refresh": {"total": 43, "total_time": "7.8s", "total_time_in_millis": 7826}, "search": {"fetch_current": 0, "fetch_time": "28.5s", "fetch_time_in_millis": 28550, "fetch_total": 9300, "open_contexts": 0, "query_current": 0, "query_time": "8.6s", "query_time_in_millis": 8629, "query_total": 9300}, "store": {"size": "149.6kb", "size_in_bytes": 153268, "throttle_time": "0s", "throttle_time_in_millis": 0}, "warmer": {"current": 0, "total": 3, "total_time": "0s", "total_time_in_millis": 0}}, "jvm": {"gc": {"collection_count": 1179, "collection_time": "3.9s", "collection_time_in_millis": 3911, "collectors": {"ConcurrentMarkSweep": {"collection_count": 0, "collection_time": "0s", "collection_time_in_millis": 0}, "ParNew": {"collection_count": 1179, "collection_time": "3.9s", "collection_time_in_millis": 3911}}}, "mem": {"heap_committed": "253.9mb", "heap_committed_in_bytes": 266272768, "heap_used": "137.8mb", "heap_used_in_bytes": 144510592, "non_heap_committed": "46.2mb", "non_heap_committed_in_bytes": 48500736, "non_heap_used": "46mb", "non_heap_used_in_bytes": 48261344, "pools": {"CMS Old Gen": {"max": "982.4mb", "max_in_bytes": 1030160384, "peak_max": "982.4mb", "peak_max_in_bytes": 1030160384, "peak_used": "128.7mb", "peak_used_in_bytes": 135035472, "used": "128.7mb", "used_in_bytes": 135035472}, "CMS Perm Gen": {"max": "82mb", "max_in_bytes": 85983232, "peak_max": "82mb", "peak_max_in_bytes": 85983232, "peak_used": "41.2mb", "peak_used_in_bytes": 43263328, "used": "41.2mb", "used_in_bytes": 43263328}, "Code Cache": {"max": "48mb", "max_in_bytes": 50331648, "peak_max": "48mb", "peak_max_in_bytes": 50331648, "peak_used": "4.7mb", "peak_used_in_bytes": 5008832, "used": "4.7mb", "used_in_bytes": 4998016}, "Par Eden Space": {"max": "33.3mb", "max_in_bytes": 34930688, "peak_max": "33.3mb", "peak_max_in_bytes": 34930688, "peak_used": "16.6mb", "peak_used_in_bytes": 17432576, "used": "9mb", "used_in_bytes": 9455216}, "Par Survivor Space": {"max": "4.1mb", "max_in_bytes": 4325376, "peak_max": "4.1mb", "peak_max_in_bytes": 4325376, "peak_used": "2mb", "peak_used_in_bytes": 2162688, "used": "19.4kb", "used_in_bytes": 19904}}}, "threads": {"count": 47, "peak_count": 48}, "timestamp": 1388945167520, "uptime": "23.4m", "uptime_in_millis": 1408987}, "name": "Space Turnip", "network": {"tcp": {"active_opens": 14551, "attempt_fails": 46, "curr_estab": 42, "estab_resets": 124, "in_errs": 82, "in_segs": 3108679, "out_rsts": -1, "out_segs": 2858839, "passive_opens": 289, "retrans_segs": 1272}}, "os": {"cpu": {"idle": 89, "stolen": 0, "sys": 2, "user": 7}, "load_average": [1.7255859375, 1.45556640625, 1.43994140625], "mem": {"actual_free": "1.6gb", "actual_free_in_bytes": 1718706176, "actual_used": "2.3gb", "actual_used_in_bytes": 2576261120, "free": "262.6mb", "free_in_bytes": 275419136, "free_percent": 40, "used": "3.7gb", "used_in_bytes": 4019548160, "used_percent": 59}, "swap": {"free": "1gb", "free_in_bytes": 1073741824, "used": "0b", "used_in_bytes": 0}, "timestamp": 1388945167520, "uptime": "3.8m", "uptime_in_millis": 228110}, "process": {"cpu": {"percent": 0, "sys": "13s", "sys_in_millis": 13058, "total": "1.7m", "total_in_millis": 102484, "user": "1.4m", "user_in_millis": 89426}, "mem": {"resident": "283.2mb", "resident_in_bytes": 296976384, "share": "-1b", "share_in_bytes": -1, "total_virtual": "3.6gb", "total_virtual_in_bytes": 3886198784}, "open_file_descriptors": 134, "timestamp": 1388945167520}, "thread_pool": {"bulk": {"active": 0, "completed": 0, "largest": 0, "queue": 0, "rejected": 0, "threads": 0}, "flush": {"active": 0, "completed": 0, "largest": 0, "queue": 0, "rejected": 0, "threads": 0}, "generic": {"active": 0, "completed": 252, "largest": 3, "queue": 0, "rejected": 0, "threads": 3}, "get": {"active": 0, "completed": 6, "largest": 2, "queue": 0, "rejected": 0, "threads": 2}, "index": {"active": 0, "completed": 147, "largest": 2, "queue": 0, "rejected": 0, "threads": 2}, "management": {"active": 1, "completed": 218, "largest": 2, "queue": 0, "rejected": 0, "threads": 2}, "merge": {"active": 0, "completed": 36, "largest": 1, "queue": 0, "rejected": 0, "threads": 1}, "optimize": {"active": 0, "completed": 31, "largest": 1, "queue": 0, "rejected": 0, "threads": 1}, "percolate": {"active": 0, "completed": 0, "largest": 0, "queue": 0, "rejected": 0, "threads": 0}, "refresh": {"active": 0, "completed": 36, "largest": 1, "queue": 0, "rejected": 0, "threads": 1}, "search": {"active": 0, "completed": 9300, "largest": 6, "queue": 0, "rejected": 0, "threads": 6}, "snapshot": {"active": 0, "completed": 20, "largest": 1, "queue": 0, "rejected": 0, "threads": 1}, "suggest": {"active": 0, "completed": 0, "largest": 0, "queue": 0, "rejected": 0, "threads": 0}, "warmer": {"active": 0, "completed": 74, "largest": 1, "queue": 0, "rejected": 0, "threads": 1}}, "timestamp": 1388945167520, "transport": {"rx_count": 0, "rx_size": "0b", "rx_size_in_bytes": 0, "server_open": 13, "tx_count": 0, "tx_size": "0b", "tx_size_in_bytes": 0}, "transport_address": "inet[/192.168.1.100:9300]"}}}, "meta": {"benchmark_id": "3e878942", "observation_id": "e8cd3f18", "observation_sequence_no": 1, "observation_start": "2014-01-05T18:06:06Z", "observation_stop": "2014-01-05T18:06:07Z", "t_total": "0.02m", "t_total_in_millis": 1178}, "segments": {"num_committed_segments": 1, "num_search_segments": 1, "segments": null, "t_optimize": "0.11s", "t_optimize_in_millis": 109}, "stats": {"docs": {"count": 6, "deleted": 0}, "fielddata": {"evictions": 0, "fields": {"abstract": {"memory_size": "26.4kb", "memory_size_in_bytes": 27086}, "dates.date_published": {"memory_size": "48b", "memory_size_in_bytes": 48}}, "memory_size": "26.4kb", "memory_size_in_bytes": 27134}, "indexing": {"delete_current": 0, "delete_time": "0s", "delete_time_in_millis": 0, "delete_total": 0, "index_current": 0, "index_time": "32ms", "index_time_in_millis": 32, "index_total": 6}, "merges": {"current": 0, "current_docs": 0, "current_size": "0b", "current_size_in_bytes": 0, "total": 0, "total_docs": 0, "total_size": "0b", "total_size_in_bytes": 0, "total_time": "0s", "total_time_in_millis": 0}, "search": {"fetch_current": 0, "fetch_time": "276ms", "fetch_time_in_millis": 276, "fetch_total": 300, "groups": {"match_description": {"client_time": "0.34s", "client_time_in_millis": 341, "client_total": 100, "fetch_current": 0, "fetch_time": "84ms", "fetch_time_in_millis": 84, "fetch_total": 100, "query_current": 0, "query_time": "18ms", "query_time_in_millis": 18, "query_total": 100}, "match_description_facet_date_histogram": {"client_time": "0.37s", "client_time_in_millis": 374, "client_total": 100, "fetch_current": 0, "fetch_time": "102ms", "fetch_time_in_millis": 102, "fetch_total": 100, "query_current": 0, "query_time": "22ms", "query_time_in_millis": 22, "query_total": 100}, "match_description_sorted_abstract": {"client_time": "0.35s", "client_time_in_millis": 352, "client_total": 100, "fetch_current": 0, "fetch_time": "89ms", "fetch_time_in_millis": 89, "fetch_total": 100, "query_current": 0, "query_time": "22ms", "query_time_in_millis": 22, "query_total": 100}}, "open_contexts": 0, "query_current": 0, "query_time": "62ms", "query_time_in_millis": 62, "query_total": 300}, "store": {"size": "149.5kb", "size_in_bytes": 153169, "throttle_time": "0s", "throttle_time_in_millis": 0}}},"sort":[1388945166000]},{"_index":"esbench_stats","_type":"obs","_id":"317405e7","_score":null, "_source" : {"cluster": {"cluster_name": "elasticsearch", "nodes": {"TqybOUCJRPOvLdshFYQqtg": {"fs": {"data": [{"available": "58gb", "available_in_bytes": 62311964672, "dev": "/dev/disk1s2", "disk_read_size": "4.3gb", "disk_read_size_in_bytes": 4703711744, "disk_reads": 248828, "disk_write_size": "15gb", "disk_write_size_in_bytes": 16137129984, "disk_writes": 468640, "free": "58gb", "free_in_bytes": 62311964672, "mount": "/Users/mkocikowski", "path": "/Users/mkocikowski/dev/elasticsearch-0.90.6/data/elasticsearch/nodes/0", "total": "465.1gb", "total_in_bytes": 499430719488}], "timestamp": 1388945169526}, "hostname": "MK.local", "http": {"current_open": 2, "total_opened": 10}, "indices": {"completion": {"size": "0b", "size_in_bytes": 0}, "docs": {"count": 12, "deleted": 0}, "fielddata": {"evictions": 0, "fields": {"abstract": {"memory_size": "52.6kb", "memory_size_in_bytes": 53952}, "dates.date_published": {"memory_size": "96b", "memory_size_in_bytes": 96}}, "memory_size": "52.7kb", "memory_size_in_bytes": 54048}, "filter_cache": {"evictions": 0, "memory_size": "48b", "memory_size_in_bytes": 48}, "flush": {"total": 0, "total_time": "0s", "total_time_in_millis": 0}, "get": {"current": 0, "exists_time": "1ms", "exists_time_in_millis": 1, "exists_total": 6, "get_time": "1ms", "missing_time": "0s", "missing_time_in_millis": 0, "missing_total": 0, "time_in_millis": 1, "total": 6}, "id_cache": {"memory_size": "0b", "memory_size_in_bytes": 0}, "indexing": {"delete_current": 0, "delete_time": "0s", "delete_time_in_millis": 0, "delete_total": 0, "index_current": 0, "index_time": "2.6s", "index_time_in_millis": 2639, "index_total": 153}, "merges": {"current": 0, "current_docs": 0, "current_size": "0b", "current_size_in_bytes": 0, "total": 6, "total_docs": 149, "total_size": "8.6mb", "total_size_in_bytes": 9066343, "total_time": "3s", "total_time_in_millis": 3053}, "refresh": {"total": 44, "total_time": "8s", "total_time_in_millis": 8045}, "search": {"fetch_current": 0, "fetch_time": "29.1s", "fetch_time_in_millis": 29168, "fetch_total": 9600, "open_contexts": 0, "query_current": 0, "query_time": "8.7s", "query_time_in_millis": 8741, "query_total": 9600}, "store": {"size": "523.3kb", "size_in_bytes": 535948, "throttle_time": "0s", "throttle_time_in_millis": 0}, "warmer": {"current": 0, "total": 5, "total_time": "2ms", "total_time_in_millis": 2}}, "jvm": {"gc": {"collection_count": 1206, "collection_time": "4s", "collection_time_in_millis": 4014, "collectors": {"ConcurrentMarkSweep": {"collection_count": 0, "collection_time": "0s", "collection_time_in_millis": 0}, "ParNew": {"collection_count": 1206, "collection_time": "4s", "collection_time_in_millis": 4014}}}, "mem": {"heap_committed": "253.9mb", "heap_committed_in_bytes": 266272768, "heap_used": "137.2mb", "heap_used_in_bytes": 143911208, "non_heap_committed": "46.2mb", "non_heap_committed_in_bytes": 48500736, "non_heap_used": "46mb", "non_heap_used_in_bytes": 48268352, "pools": {"CMS Old Gen": {"max": "982.4mb", "max_in_bytes": 1030160384, "peak_max": "982.4mb", "peak_max_in_bytes": 1030160384, "peak_used": "133.5mb", "peak_used_in_bytes": 140014968, "used": "133.5mb", "used_in_bytes": 140014968}, "CMS Perm Gen": {"max": "82mb", "max_in_bytes": 85983232, "peak_max": "82mb", "peak_max_in_bytes": 85983232, "peak_used": "41.2mb", "peak_used_in_bytes": 43265664, "used": "41.2mb", "used_in_bytes": 43265664}, "Code Cache": {"max": "48mb", "max_in_bytes": 50331648, "peak_max": "48mb", "peak_max_in_bytes": 50331648, "peak_used": "4.7mb", "peak_used_in_bytes": 5011328, "used": "4.7mb", "used_in_bytes": 5002688}, "Par Eden Space": {"max": "33.3mb", "max_in_bytes": 34930688, "peak_max": "33.3mb", "peak_max_in_bytes": 34930688, "peak_used": "16.6mb", "peak_used_in_bytes": 17432576, "used": "3.5mb", "used_in_bytes": 3768768}, "Par Survivor Space": {"max": "4.1mb", "max_in_bytes": 4325376, "peak_max": "4.1mb", "peak_max_in_bytes": 4325376, "peak_used": "2mb", "peak_used_in_bytes": 2162688, "used": "124.4kb", "used_in_bytes": 127472}}}, "threads": {"count": 47, "peak_count": 48}, "timestamp": 1388945169525, "uptime": "23.5m", "uptime_in_millis": 1410992}, "name": "Space Turnip", "network": {"tcp": {"active_opens": 14551, "attempt_fails": 46, "curr_estab": 42, "estab_resets": 124, "in_errs": 82, "in_segs": 3108679, "out_rsts": -1, "out_segs": 2858839, "passive_opens": 289, "retrans_segs": 1272}}, "os": {"cpu": {"idle": 33, "stolen": 0, "sys": 20, "user": 45}, "load_average": [1.7255859375, 1.45556640625, 1.43994140625], "mem": {"actual_free": "1.5gb", "actual_free_in_bytes": 1712320512, "actual_used": "2.4gb", "actual_used_in_bytes": 2582646784, "free": "256.2mb", "free_in_bytes": 268648448, "free_percent": 39, "used": "3.7gb", "used_in_bytes": 4026318848, "used_percent": 60}, "swap": {"free": "1gb", "free_in_bytes": 1073741824, "used": "0b", "used_in_bytes": 0}, "timestamp": 1388945169525, "uptime": "3.8m", "uptime_in_millis": 228112}, "process": {"cpu": {"percent": 89, "sys": "13.4s", "sys_in_millis": 13476, "total": "1.7m", "total_in_millis": 104279, "user": "1.5m", "user_in_millis": 90803}, "mem": {"resident": "287.8mb", "resident_in_bytes": 301838336, "share": "-1b", "share_in_bytes": -1, "total_virtual": "3.6gb", "total_virtual_in_bytes": 3886198784}, "open_file_descriptors": 136, "timestamp": 1388945169525}, "thread_pool": {"bulk": {"active": 0, "completed": 0, "largest": 0, "queue": 0, "rejected": 0, "threads": 0}, "flush": {"active": 0, "completed": 0, "largest": 0, "queue": 0, "rejected": 0, "threads": 0}, "generic": {"active": 0, "completed": 253, "largest": 3, "queue": 0, "rejected": 0, "threads": 3}, "get": {"active": 0, "completed": 6, "largest": 2, "queue": 0, "rejected": 0, "threads": 2}, "index": {"active": 0, "completed": 153, "largest": 2, "queue": 0, "rejected": 0, "threads": 2}, "management": {"active": 1, "completed": 222, "largest": 2, "queue": 0, "rejected": 0, "threads": 2}, "merge": {"active": 0, "completed": 37, "largest": 1, "queue": 0, "rejected": 0, "threads": 1}, "optimize": {"active": 0, "completed": 32, "largest": 1, "queue": 0, "rejected": 0, "threads": 1}, "percolate": {"active": 0, "completed": 0, "largest": 0, "queue": 0, "rejected": 0, "threads": 0}, "refresh": {"active": 0, "completed": 37, "largest": 1, "queue": 0, "rejected": 0, "threads": 1}, "search": {"active": 0, "completed": 9600, "largest": 6, "queue": 0, "rejected": 0, "threads": 6}, "snapshot": {"active": 0, "completed": 20, "largest": 1, "queue": 0, "rejected": 0, "threads": 1}, "suggest": {"active": 0, "completed": 0, "largest": 0, "queue": 0, "rejected": 0, "threads": 0}, "warmer": {"active": 0, "completed": 76, "largest": 1, "queue": 0, "rejected": 0, "threads": 1}}, "timestamp": 1388945169524, "transport": {"rx_count": 0, "rx_size": "0b", "rx_size_in_bytes": 0, "server_open": 13, "tx_count": 0, "tx_size": "0b", "tx_size_in_bytes": 0}, "transport_address": "inet[/192.168.1.100:9300]"}}}, "meta": {"benchmark_id": "3e878942", "observation_id": "317405e7", "observation_sequence_no": 2, "observation_start": "2014-01-05T18:06:07Z", "observation_stop": "2014-01-05T18:06:09Z", "t_total": "0.03m", "t_total_in_millis": 1890}, "segments": {"num_committed_segments": 2, "num_search_segments": 2, "segments": null, "t_optimize": "0.15s", "t_optimize_in_millis": 145}, "stats": {"docs": {"count": 11, "deleted": 0}, "fielddata": {"evictions": 0, "fields": {"abstract": {"memory_size": "52.6kb", "memory_size_in_bytes": 53952}, "dates.date_published": {"memory_size": "96b", "memory_size_in_bytes": 96}}, "memory_size": "52.7kb", "memory_size_in_bytes": 54048}, "indexing": {"delete_current": 0, "delete_time": "0s", "delete_time_in_millis": 0, "delete_total": 0, "index_current": 0, "index_time": "70ms", "index_time_in_millis": 70, "index_total": 11}, "merges": {"current": 0, "current_docs": 0, "current_size": "0b", "current_size_in_bytes": 0, "total": 0, "total_docs": 0, "total_size": "0b", "total_size_in_bytes": 0, "total_time": "0s", "total_time_in_millis": 0}, "search": {"fetch_current": 0, "fetch_time": "894ms", "fetch_time_in_millis": 894, "fetch_total": 600, "groups": {"match_description": {"client_time": "0.56s", "client_time_in_millis": 555, "client_total": 100, "fetch_current": 0, "fetch_time": "199ms", "fetch_time_in_millis": 199, "fetch_total": 100, "query_current": 0, "query_time": "34ms", "query_time_in_millis": 34, "query_total": 100}, "match_description_facet_date_histogram": {"client_time": "0.61s", "client_time_in_millis": 612, "client_total": 100, "fetch_current": 0, "fetch_time": "222ms", "fetch_time_in_millis": 222, "fetch_total": 100, "query_current": 0, "query_time": "37ms", "query_time_in_millis": 37, "query_total": 100}, "match_description_sorted_abstract": {"client_time": "0.58s", "client_time_in_millis": 575, "client_total": 100, "fetch_current": 0, "fetch_time": "196ms", "fetch_time_in_millis": 196, "fetch_total": 100, "query_current": 0, "query_time": "40ms", "query_time_in_millis": 40, "query_total": 100}}, "open_contexts": 0, "query_current": 0, "query_time": "174ms", "query_time_in_millis": 174, "query_total": 600}, "store": {"size": "293.9kb", "size_in_bytes": 300998, "throttle_time": "0s", "throttle_time_in_millis": 0}}},"sort":[1388945167000]}]}}""",
 47 |             curl='curl "http://localhost:9200/esbench_stats/obs/_search?q=meta.benchmark_id:3e878942&sort=meta.observation_start:asc&size=2"'
 48 |         )
 49 | 
 50 |     def tearDown(self):
 51 |         esbench.analyze._get_benchmarks = self.tmp_get_benchmarks
 52 |         esbench.analyze._get_observations = self.tmp_get_observations
 53 | 
 54 | 
 55 |     def test_benchmarks(self):
 56 |         self.assertEqual(1, len(list(esbench.analyze._benchmarks(resp=esbench.analyze._get_benchmarks()))))
 57 |         self.assertEqual(1, len(list(esbench.analyze._benchmarks(resp=esbench.analyze._get_benchmarks(), benchmark_ids=['3e878942', 'foo']))))
 58 |         self.assertEqual(0, len(list(esbench.analyze._benchmarks(resp=esbench.analyze._get_benchmarks(), benchmark_ids=['foo']))))
 59 | 
 60 |         # this is the response you get when there are no benchmarks
 61 |         resp = esbench.api.ApiResponse(status=200, reason='OK', data='{"took":1,"timed_out":false}', curl='curl -XGET http://localhost:9200/stats/bench/_search?sort=benchmark_start:asc&size=100')
 62 |         self.assertEqual(0, len(list(esbench.analyze._benchmarks(resp=resp))))
 63 | 
 64 |         r = esbench.api.ApiResponse(
 65 |             status=200,
 66 |             reason='OK',
 67 |             data = '{"hits": {"hits": [{"_id": "foo"}, {"_id": "bar"}, {"_id": "baz"}]}}',
 68 |             curl='curl "http://localhost:9200/esbench_stats/bench/_search?sort=benchmark_start:asc&size=100"',
 69 |         )
 70 |         self.assertEqual([{u'_id': u'foo'}, {u'_id': u'bar'}, {u'_id': u'baz'}], list(esbench.analyze._benchmarks(resp=r)))
 71 |         self.assertEqual([{u'_id': u'foo'}, {u'_id': u'bar'}, {u'_id': u'baz'}], list(esbench.analyze._benchmarks(resp=r, benchmark_ids=['all'])))
 72 |         self.assertEqual([{u'_id': u'foo'}, {u'_id': u'baz'}], list(esbench.analyze._benchmarks(resp=r, benchmark_ids=['first', 'last'])))
 73 |         self.assertEqual([{u'_id': u'bar'}, {u'_id': u'baz'}], list(esbench.analyze._benchmarks(resp=r, benchmark_ids=['1', 'last'])))
 74 |         self.assertEqual([{u'_id': u'baz'}, {u'_id': u'foo'}, {u'_id': u'bar'}, {u'_id': u'baz'}], list(esbench.analyze._benchmarks(resp=r, benchmark_ids=['-1', 'all'])))
 75 | 
 76 | 
 77 |     def test_observations(self):
 78 |         resp = esbench.api.ApiResponse(status=200, reason='OK', data='{"took":5,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":10,"max_score":null,"hits":[{"_index":"stats","_type":"obs","_id":"c5ac7b40","_score":null, "_source" : {"meta": {"benchmark_id": "315fd1aa", "observation_id": "c5ac7b40", "observation_sequence_no": 1, "observation_start": "2013-11-26T17:58:12Z", "observation_stop": "2013-11-26T17:58:13Z"}, "segments": {"num_committed_segments": 1, "num_search_segments": 1, "segments": {"_0": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 0, "num_docs": 10, "search": true, "size": "237.3kb", "size_in_bytes": 243076, "version": "4.5.1"}}, "t_optimize": "0.17s", "t_optimize_in_millis": 165}, "stats": {"docs": {"count": 10, "deleted": 0}, "indexing": {"delete_current": 0, "delete_time": "0s", "delete_time_in_millis": 0, "delete_total": 0, "index_current": 0, "index_time": "234ms", "index_time_in_millis": 234, "index_total": 10}, "merges": {"current": 0, "current_docs": 0, "current_size": "0b", "current_size_in_bytes": 0, "total": 0, "total_docs": 0, "total_size": "0b", "total_size_in_bytes": 0, "total_time": "0s", "total_time_in_millis": 0}, "search": {"fetch_current": 0, "fetch_time": "461ms", "fetch_time_in_millis": 461, "fetch_total": 500, "groups": {"match": {"client_time": "0.34s", "client_time_in_millis": 342, "client_total": 100, "fetch_current": 0, "fetch_time": "113ms", "fetch_time_in_millis": 113, "fetch_total": 100, "query_current": 0, "query_time": "39ms", "query_time_in_millis": 39, "query_total": 100}, "match_abs": {"client_time": "0.29s", "client_time_in_millis": 294, "client_total": 100, "fetch_current": 0, "fetch_time": "86ms", "fetch_time_in_millis": 86, "fetch_total": 100, "query_current": 0, "query_time": "45ms", "query_time_in_millis": 45, "query_total": 100}, "match_no_rnd": {"client_time": "0.45s", "client_time_in_millis": 448, "client_total": 100, "fetch_current": 0, "fetch_time": "156ms", "fetch_time_in_millis": 156, "fetch_total": 100, "query_current": 0, "query_time": "58ms", "query_time_in_millis": 58, "query_total": 100}, "match_srt": {"client_time": "0.32s", "client_time_in_millis": 315, "client_total": 100, "fetch_current": 0, "fetch_time": "102ms", "fetch_time_in_millis": 102, "fetch_total": 100, "query_current": 0, "query_time": "38ms", "query_time_in_millis": 38, "query_total": 100}, "mlt": {"client_time": "0.14s", "client_time_in_millis": 136, "client_total": 100, "fetch_current": 0, "fetch_time": "1ms", "fetch_time_in_millis": 1, "fetch_total": 100, "query_current": 0, "query_time": "11ms", "query_time_in_millis": 11, "query_total": 100}}, "open_contexts": 0, "query_current": 0, "query_time": "193ms", "query_time_in_millis": 193, "query_total": 500}, "store": {"size": "237.5kb", "size_in_bytes": 243242, "throttle_time": "0s", "throttle_time_in_millis": 0}}},"sort":[1385488692000]},{"_index":"stats","_type":"obs","_id":"2f74c0e9","_score":null, "_source" : {"meta": {"benchmark_id": "315fd1aa", "observation_id": "2f74c0e9", "observation_sequence_no": 2, "observation_start": "2013-11-26T17:58:13Z", "observation_stop": "2013-11-26T17:58:16Z"}, "segments": {"num_committed_segments": 2, "num_search_segments": 2, "segments": {"_0": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 0, "num_docs": 10, "search": true, "size": "237.3kb", "size_in_bytes": 243076, "version": "4.5.1"}, "_1": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 1, "num_docs": 10, "search": true, "size": "244.2kb", "size_in_bytes": 250129, "version": "4.5.1"}}, "t_optimize": "0.07s", "t_optimize_in_millis": 66}, "stats": {"docs": {"count": 20, "deleted": 0}, "indexing": {"delete_current": 0, "delete_time": "0s", "delete_time_in_millis": 0, "delete_total": 0, "index_current": 0, "index_time": "271ms", "index_time_in_millis": 271, "index_total": 20}, "merges": {"current": 0, "current_docs": 0, "current_size": "0b", "current_size_in_bytes": 0, "total": 0, "total_docs": 0, "total_size": "0b", "total_size_in_bytes": 0, "total_time": "0s", "total_time_in_millis": 0}, "search": {"fetch_current": 0, "fetch_time": "1.2s", "fetch_time_in_millis": 1206, "fetch_total": 1000, "groups": {"match": {"client_time": "0.84s", "client_time_in_millis": 844, "client_total": 100, "fetch_current": 0, "fetch_time": "189ms", "fetch_time_in_millis": 189, "fetch_total": 100, "query_current": 0, "query_time": "52ms", "query_time_in_millis": 52, "query_total": 100}, "match_abs": {"client_time": "0.35s", "client_time_in_millis": 352, "client_total": 100, "fetch_current": 0, "fetch_time": "135ms", "fetch_time_in_millis": 135, "fetch_total": 100, "query_current": 0, "query_time": "60ms", "query_time_in_millis": 60, "query_total": 100}, "match_no_rnd": {"client_time": "0.57s", "client_time_in_millis": 567, "client_total": 100, "fetch_current": 0, "fetch_time": "230ms", "fetch_time_in_millis": 230, "fetch_total": 100, "query_current": 0, "query_time": "70ms", "query_time_in_millis": 70, "query_total": 100}, "match_srt": {"client_time": "0.46s", "client_time_in_millis": 460, "client_total": 100, "fetch_current": 0, "fetch_time": "188ms", "fetch_time_in_millis": 188, "fetch_total": 100, "query_current": 0, "query_time": "49ms", "query_time_in_millis": 49, "query_total": 100}, "mlt": {"client_time": "0.13s", "client_time_in_millis": 130, "client_total": 100, "fetch_current": 0, "fetch_time": "1ms", "fetch_time_in_millis": 1, "fetch_total": 100, "query_current": 0, "query_time": "13ms", "query_time_in_millis": 13, "query_total": 100}}, "open_contexts": 0, "query_current": 0, "query_time": "439ms", "query_time_in_millis": 439, "query_total": 1000}, "store": {"size": "481.8kb", "size_in_bytes": 493437, "throttle_time": "0s", "throttle_time_in_millis": 0}}},"sort":[1385488693000]},{"_index":"stats","_type":"obs","_id":"a00c16c7","_score":null, "_source" : {"meta": {"benchmark_id": "315fd1aa", "observation_id": "a00c16c7", "observation_sequence_no": 3, "observation_start": "2013-11-26T17:58:16Z", "observation_stop": "2013-11-26T17:58:18Z"}, "segments": {"num_committed_segments": 3, "num_search_segments": 3, "segments": {"_0": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 0, "num_docs": 10, "search": true, "size": "237.3kb", "size_in_bytes": 243076, "version": "4.5.1"}, "_1": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 1, "num_docs": 10, "search": true, "size": "244.2kb", "size_in_bytes": 250129, "version": "4.5.1"}, "_2": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 2, "num_docs": 10, "search": true, "size": "374.5kb", "size_in_bytes": 383503, "version": "4.5.1"}}, "t_optimize": "0.05s", "t_optimize_in_millis": 51}, "stats": {"docs": {"count": 30, "deleted": 0}, "indexing": {"delete_current": 0, "delete_time": "0s", "delete_time_in_millis": 0, "delete_total": 0, "index_current": 0, "index_time": "328ms", "index_time_in_millis": 328, "index_total": 30}, "merges": {"current": 0, "current_docs": 0, "current_size": "0b", "current_size_in_bytes": 0, "total": 0, "total_docs": 0, "total_size": "0b", "total_size_in_bytes": 0, "total_time": "0s", "total_time_in_millis": 0}, "search": {"fetch_current": 0, "fetch_time": "2.1s", "fetch_time_in_millis": 2171, "fetch_total": 1500, "groups": {"match": {"client_time": "0.59s", "client_time_in_millis": 588, "client_total": 100, "fetch_current": 0, "fetch_time": "270ms", "fetch_time_in_millis": 270, "fetch_total": 100, "query_current": 0, "query_time": "51ms", "query_time_in_millis": 51, "query_total": 100}, "match_abs": {"client_time": "0.40s", "client_time_in_millis": 399, "client_total": 100, "fetch_current": 0, "fetch_time": "196ms", "fetch_time_in_millis": 196, "fetch_total": 100, "query_current": 0, "query_time": "54ms", "query_time_in_millis": 54, "query_total": 100}, "match_no_rnd": {"client_time": "0.61s", "client_time_in_millis": 610, "client_total": 100, "fetch_current": 0, "fetch_time": "281ms", "fetch_time_in_millis": 281, "fetch_total": 100, "query_current": 0, "query_time": "60ms", "query_time_in_millis": 60, "query_total": 100}, "match_srt": {"client_time": "0.50s", "client_time_in_millis": 496, "client_total": 100, "fetch_current": 0, "fetch_time": "214ms", "fetch_time_in_millis": 214, "fetch_total": 100, "query_current": 0, "query_time": "50ms", "query_time_in_millis": 50, "query_total": 100}, "mlt": {"client_time": "0.13s", "client_time_in_millis": 133, "client_total": 100, "fetch_current": 0, "fetch_time": "1ms", "fetch_time_in_millis": 1, "fetch_total": 100, "query_current": 0, "query_time": "16ms", "query_time_in_millis": 16, "query_total": 100}}, "open_contexts": 0, "query_current": 0, "query_time": "672ms", "query_time_in_millis": 672, "query_total": 1500}, "store": {"size": "856.4kb", "size_in_bytes": 877007, "throttle_time": "0s", "throttle_time_in_millis": 0}}},"sort":[1385488696000]},{"_index":"stats","_type":"obs","_id":"244e5a42","_score":null, "_source" : {"meta": {"benchmark_id": "315fd1aa", "observation_id": "244e5a42", "observation_sequence_no": 4, "observation_start": "2013-11-26T17:58:18Z", "observation_stop": "2013-11-26T17:58:21Z"}, "segments": {"num_committed_segments": 5, "num_search_segments": 5, "segments": {"_0": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 0, "num_docs": 10, "search": true, "size": "237.3kb", "size_in_bytes": 243076, "version": "4.5.1"}, "_1": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 1, "num_docs": 10, "search": true, "size": "244.2kb", "size_in_bytes": 250129, "version": "4.5.1"}, "_2": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 2, "num_docs": 10, "search": true, "size": "374.5kb", "size_in_bytes": 383503, "version": "4.5.1"}, "_3": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 3, "num_docs": 8, "search": true, "size": "305.1kb", "size_in_bytes": 312523, "version": "4.5.1"}, "_4": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 4, "num_docs": 2, "search": true, "size": "132.3kb", "size_in_bytes": 135496, "version": "4.5.1"}}, "t_optimize": "0.02s", "t_optimize_in_millis": 21}, "stats": {"docs": {"count": 40, "deleted": 0}, "indexing": {"delete_current": 0, "delete_time": "0s", "delete_time_in_millis": 0, "delete_total": 0, "index_current": 0, "index_time": "439ms", "index_time_in_millis": 439, "index_total": 40}, "merges": {"current": 0, "current_docs": 0, "current_size": "0b", "current_size_in_bytes": 0, "total": 0, "total_docs": 0, "total_size": "0b", "total_size_in_bytes": 0, "total_time": "0s", "total_time_in_millis": 0}, "search": {"fetch_current": 0, "fetch_time": "3.1s", "fetch_time_in_millis": 3151, "fetch_total": 2000, "groups": {"match": {"client_time": "0.63s", "client_time_in_millis": 628, "client_total": 100, "fetch_current": 0, "fetch_time": "290ms", "fetch_time_in_millis": 290, "fetch_total": 100, "query_current": 0, "query_time": "56ms", "query_time_in_millis": 56, "query_total": 100}, "match_abs": {"client_time": "0.36s", "client_time_in_millis": 363, "client_total": 100, "fetch_current": 0, "fetch_time": "184ms", "fetch_time_in_millis": 184, "fetch_total": 100, "query_current": 0, "query_time": "49ms", "query_time_in_millis": 49, "query_total": 100}, "match_no_rnd": {"client_time": "0.66s", "client_time_in_millis": 656, "client_total": 100, "fetch_current": 0, "fetch_time": "303ms", "fetch_time_in_millis": 303, "fetch_total": 100, "query_current": 0, "query_time": "57ms", "query_time_in_millis": 57, "query_total": 100}, "match_srt": {"client_time": "0.48s", "client_time_in_millis": 482, "client_total": 100, "fetch_current": 0, "fetch_time": "198ms", "fetch_time_in_millis": 198, "fetch_total": 100, "query_current": 0, "query_time": "61ms", "query_time_in_millis": 61, "query_total": 100}, "mlt": {"client_time": "0.14s", "client_time_in_millis": 137, "client_total": 100, "fetch_current": 0, "fetch_time": "1ms", "fetch_time_in_millis": 1, "fetch_total": 100, "query_current": 0, "query_time": "21ms", "query_time_in_millis": 21, "query_total": 100}}, "open_contexts": 0, "query_current": 0, "query_time": "918ms", "query_time_in_millis": 918, "query_total": 2000}, "store": {"size": "1.2mb", "size_in_bytes": 1325157, "throttle_time": "0s", "throttle_time_in_millis": 0}}},"sort":[1385488698000]},{"_index":"stats","_type":"obs","_id":"c0e74e4d","_score":null, "_source" : {"meta": {"benchmark_id": "315fd1aa", "observation_id": "c0e74e4d", "observation_sequence_no": 5, "observation_start": "2013-11-26T17:58:21Z", "observation_stop": "2013-11-26T17:58:23Z"}, "segments": {"num_committed_segments": 6, "num_search_segments": 6, "segments": {"_0": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 0, "num_docs": 10, "search": true, "size": "237.3kb", "size_in_bytes": 243076, "version": "4.5.1"}, "_1": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 1, "num_docs": 10, "search": true, "size": "244.2kb", "size_in_bytes": 250129, "version": "4.5.1"}, "_2": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 2, "num_docs": 10, "search": true, "size": "374.5kb", "size_in_bytes": 383503, "version": "4.5.1"}, "_3": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 3, "num_docs": 8, "search": true, "size": "305.1kb", "size_in_bytes": 312523, "version": "4.5.1"}, "_4": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 4, "num_docs": 2, "search": true, "size": "132.3kb", "size_in_bytes": 135496, "version": "4.5.1"}, "_5": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 5, "num_docs": 10, "search": true, "size": "350kb", "size_in_bytes": 358422, "version": "4.5.1"}}, "t_optimize": "0.04s", "t_optimize_in_millis": 39}, "stats": {"docs": {"count": 50, "deleted": 0}, "indexing": {"delete_current": 0, "delete_time": "0s", "delete_time_in_millis": 0, "delete_total": 0, "index_current": 0, "index_time": "504ms", "index_time_in_millis": 504, "index_total": 50}, "merges": {"current": 0, "current_docs": 0, "current_size": "0b", "current_size_in_bytes": 0, "total": 0, "total_docs": 0, "total_size": "0b", "total_size_in_bytes": 0, "total_time": "0s", "total_time_in_millis": 0}, "search": {"fetch_current": 0, "fetch_time": "4s", "fetch_time_in_millis": 4067, "fetch_total": 2500, "groups": {"match": {"client_time": "0.60s", "client_time_in_millis": 602, "client_total": 100, "fetch_current": 0, "fetch_time": "266ms", "fetch_time_in_millis": 266, "fetch_total": 100, "query_current": 0, "query_time": "60ms", "query_time_in_millis": 60, "query_total": 100}, "match_abs": {"client_time": "0.35s", "client_time_in_millis": 352, "client_total": 100, "fetch_current": 0, "fetch_time": "175ms", "fetch_time_in_millis": 175, "fetch_total": 100, "query_current": 0, "query_time": "51ms", "query_time_in_millis": 51, "query_total": 100}, "match_no_rnd": {"client_time": "0.60s", "client_time_in_millis": 601, "client_total": 100, "fetch_current": 0, "fetch_time": "267ms", "fetch_time_in_millis": 267, "fetch_total": 100, "query_current": 0, "query_time": "58ms", "query_time_in_millis": 58, "query_total": 100}, "match_srt": {"client_time": "0.49s", "client_time_in_millis": 492, "client_total": 100, "fetch_current": 0, "fetch_time": "205ms", "fetch_time_in_millis": 205, "fetch_total": 100, "query_current": 0, "query_time": "57ms", "query_time_in_millis": 57, "query_total": 100}, "mlt": {"client_time": "0.14s", "client_time_in_millis": 139, "client_total": 100, "fetch_current": 0, "fetch_time": "1ms", "fetch_time_in_millis": 1, "fetch_total": 100, "query_current": 0, "query_time": "23ms", "query_time_in_millis": 23, "query_total": 100}}, "open_contexts": 0, "query_current": 0, "query_time": "1.1s", "query_time_in_millis": 1169, "query_total": 2500}, "store": {"size": "1.6mb", "size_in_bytes": 1683644, "throttle_time": "0s", "throttle_time_in_millis": 0}}},"sort":[1385488701000]},{"_index":"stats","_type":"obs","_id":"6ea3adfe","_score":null, "_source" : {"meta": {"benchmark_id": "315fd1aa", "observation_id": "6ea3adfe", "observation_sequence_no": 6, "observation_start": "2013-11-26T17:58:23Z", "observation_stop": "2013-11-26T17:58:26Z"}, "segments": {"num_committed_segments": 7, "num_search_segments": 7, "segments": {"_0": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 0, "num_docs": 10, "search": true, "size": "237.3kb", "size_in_bytes": 243076, "version": "4.5.1"}, "_1": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 1, "num_docs": 10, "search": true, "size": "244.2kb", "size_in_bytes": 250129, "version": "4.5.1"}, "_2": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 2, "num_docs": 10, "search": true, "size": "374.5kb", "size_in_bytes": 383503, "version": "4.5.1"}, "_3": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 3, "num_docs": 8, "search": true, "size": "305.1kb", "size_in_bytes": 312523, "version": "4.5.1"}, "_4": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 4, "num_docs": 2, "search": true, "size": "132.3kb", "size_in_bytes": 135496, "version": "4.5.1"}, "_5": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 5, "num_docs": 10, "search": true, "size": "350kb", "size_in_bytes": 358422, "version": "4.5.1"}, "_6": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 6, "num_docs": 10, "search": true, "size": "388.6kb", "size_in_bytes": 397956, "version": "4.5.1"}}, "t_optimize": "0.03s", "t_optimize_in_millis": 30}, "stats": {"docs": {"count": 60, "deleted": 0}, "indexing": {"delete_current": 0, "delete_time": "0s", "delete_time_in_millis": 0, "delete_total": 0, "index_current": 0, "index_time": "558ms", "index_time_in_millis": 558, "index_total": 60}, "merges": {"current": 0, "current_docs": 0, "current_size": "0b", "current_size_in_bytes": 0, "total": 0, "total_docs": 0, "total_size": "0b", "total_size_in_bytes": 0, "total_time": "0s", "total_time_in_millis": 0}, "search": {"fetch_current": 0, "fetch_time": "5s", "fetch_time_in_millis": 5092, "fetch_total": 3000, "groups": {"match": {"client_time": "0.68s", "client_time_in_millis": 678, "client_total": 100, "fetch_current": 0, "fetch_time": "307ms", "fetch_time_in_millis": 307, "fetch_total": 100, "query_current": 0, "query_time": "53ms", "query_time_in_millis": 53, "query_total": 100}, "match_abs": {"client_time": "0.38s", "client_time_in_millis": 378, "client_total": 100, "fetch_current": 0, "fetch_time": "205ms", "fetch_time_in_millis": 205, "fetch_total": 100, "query_current": 0, "query_time": "45ms", "query_time_in_millis": 45, "query_total": 100}, "match_no_rnd": {"client_time": "0.68s", "client_time_in_millis": 684, "client_total": 100, "fetch_current": 0, "fetch_time": "311ms", "fetch_time_in_millis": 311, "fetch_total": 100, "query_current": 0, "query_time": "48ms", "query_time_in_millis": 48, "query_total": 100}, "match_srt": {"client_time": "0.48s", "client_time_in_millis": 480, "client_total": 100, "fetch_current": 0, "fetch_time": "198ms", "fetch_time_in_millis": 198, "fetch_total": 100, "query_current": 0, "query_time": "56ms", "query_time_in_millis": 56, "query_total": 100}, "mlt": {"client_time": "0.13s", "client_time_in_millis": 133, "client_total": 100, "fetch_current": 0, "fetch_time": "1ms", "fetch_time_in_millis": 1, "fetch_total": 100, "query_current": 0, "query_time": "19ms", "query_time_in_millis": 19, "query_total": 100}}, "open_contexts": 0, "query_current": 0, "query_time": "1.3s", "query_time_in_millis": 1393, "query_total": 3000}, "store": {"size": "1.9mb", "size_in_bytes": 2081666, "throttle_time": "0s", "throttle_time_in_millis": 0}}},"sort":[1385488703000]},{"_index":"stats","_type":"obs","_id":"d7030c18","_score":null, "_source" : {"meta": {"benchmark_id": "315fd1aa", "observation_id": "d7030c18", "observation_sequence_no": 7, "observation_start": "2013-11-26T17:58:26Z", "observation_stop": "2013-11-26T17:58:28Z"}, "segments": {"num_committed_segments": 8, "num_search_segments": 8, "segments": {"_0": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 0, "num_docs": 10, "search": true, "size": "237.3kb", "size_in_bytes": 243076, "version": "4.5.1"}, "_1": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 1, "num_docs": 10, "search": true, "size": "244.2kb", "size_in_bytes": 250129, "version": "4.5.1"}, "_2": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 2, "num_docs": 10, "search": true, "size": "374.5kb", "size_in_bytes": 383503, "version": "4.5.1"}, "_3": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 3, "num_docs": 8, "search": true, "size": "305.1kb", "size_in_bytes": 312523, "version": "4.5.1"}, "_4": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 4, "num_docs": 2, "search": true, "size": "132.3kb", "size_in_bytes": 135496, "version": "4.5.1"}, "_5": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 5, "num_docs": 10, "search": true, "size": "350kb", "size_in_bytes": 358422, "version": "4.5.1"}, "_6": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 6, "num_docs": 10, "search": true, "size": "388.6kb", "size_in_bytes": 397956, "version": "4.5.1"}, "_7": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 7, "num_docs": 10, "search": true, "size": "395.3kb", "size_in_bytes": 404797, "version": "4.5.1"}}, "t_optimize": "0.03s", "t_optimize_in_millis": 34}, "stats": {"docs": {"count": 70, "deleted": 0}, "indexing": {"delete_current": 0, "delete_time": "0s", "delete_time_in_millis": 0, "delete_total": 0, "index_current": 0, "index_time": "634ms", "index_time_in_millis": 634, "index_total": 70}, "merges": {"current": 0, "current_docs": 0, "current_size": "0b", "current_size_in_bytes": 0, "total": 0, "total_docs": 0, "total_size": "0b", "total_size_in_bytes": 0, "total_time": "0s", "total_time_in_millis": 0}, "search": {"fetch_current": 0, "fetch_time": "6.1s", "fetch_time_in_millis": 6122, "fetch_total": 3500, "groups": {"match": {"client_time": "0.68s", "client_time_in_millis": 683, "client_total": 100, "fetch_current": 0, "fetch_time": "298ms", "fetch_time_in_millis": 298, "fetch_total": 100, "query_current": 0, "query_time": "63ms", "query_time_in_millis": 63, "query_total": 100}, "match_abs": {"client_time": "0.38s", "client_time_in_millis": 378, "client_total": 100, "fetch_current": 0, "fetch_time": "203ms", "fetch_time_in_millis": 203, "fetch_total": 100, "query_current": 0, "query_time": "49ms", "query_time_in_millis": 49, "query_total": 100}, "match_no_rnd": {"client_time": "0.67s", "client_time_in_millis": 673, "client_total": 100, "fetch_current": 0, "fetch_time": "312ms", "fetch_time_in_millis": 312, "fetch_total": 100, "query_current": 0, "query_time": "48ms", "query_time_in_millis": 48, "query_total": 100}, "match_srt": {"client_time": "0.48s", "client_time_in_millis": 476, "client_total": 100, "fetch_current": 0, "fetch_time": "214ms", "fetch_time_in_millis": 214, "fetch_total": 100, "query_current": 0, "query_time": "52ms", "query_time_in_millis": 52, "query_total": 100}, "mlt": {"client_time": "0.13s", "client_time_in_millis": 134, "client_total": 100, "fetch_current": 0, "fetch_time": "1ms", "fetch_time_in_millis": 1, "fetch_total": 100, "query_current": 0, "query_time": "20ms", "query_time_in_millis": 20, "query_total": 100}}, "open_contexts": 0, "query_current": 0, "query_time": "1.6s", "query_time_in_millis": 1627, "query_total": 3500}, "store": {"size": "2.3mb", "size_in_bytes": 2486530, "throttle_time": "0s", "throttle_time_in_millis": 0}}},"sort":[1385488706000]},{"_index":"stats","_type":"obs","_id":"8b615e3f","_score":null, "_source" : {"meta": {"benchmark_id": "315fd1aa", "observation_id": "8b615e3f", "observation_sequence_no": 8, "observation_start": "2013-11-26T17:58:28Z", "observation_stop": "2013-11-26T17:58:31Z"}, "segments": {"num_committed_segments": 9, "num_search_segments": 9, "segments": {"_0": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 0, "num_docs": 10, "search": true, "size": "237.3kb", "size_in_bytes": 243076, "version": "4.5.1"}, "_1": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 1, "num_docs": 10, "search": true, "size": "244.2kb", "size_in_bytes": 250129, "version": "4.5.1"}, "_2": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 2, "num_docs": 10, "search": true, "size": "374.5kb", "size_in_bytes": 383503, "version": "4.5.1"}, "_3": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 3, "num_docs": 8, "search": true, "size": "305.1kb", "size_in_bytes": 312523, "version": "4.5.1"}, "_4": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 4, "num_docs": 2, "search": true, "size": "132.3kb", "size_in_bytes": 135496, "version": "4.5.1"}, "_5": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 5, "num_docs": 10, "search": true, "size": "350kb", "size_in_bytes": 358422, "version": "4.5.1"}, "_6": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 6, "num_docs": 10, "search": true, "size": "388.6kb", "size_in_bytes": 397956, "version": "4.5.1"}, "_7": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 7, "num_docs": 10, "search": true, "size": "395.3kb", "size_in_bytes": 404797, "version": "4.5.1"}, "_8": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 8, "num_docs": 10, "search": true, "size": "465.1kb", "size_in_bytes": 476342, "version": "4.5.1"}}, "t_optimize": "0.04s", "t_optimize_in_millis": 35}, "stats": {"docs": {"count": 80, "deleted": 0}, "indexing": {"delete_current": 0, "delete_time": "0s", "delete_time_in_millis": 0, "delete_total": 0, "index_current": 0, "index_time": "693ms", "index_time_in_millis": 693, "index_total": 80}, "merges": {"current": 0, "current_docs": 0, "current_size": "0b", "current_size_in_bytes": 0, "total": 0, "total_docs": 0, "total_size": "0b", "total_size_in_bytes": 0, "total_time": "0s", "total_time_in_millis": 0}, "search": {"fetch_current": 0, "fetch_time": "7.1s", "fetch_time_in_millis": 7157, "fetch_total": 4000, "groups": {"match": {"client_time": "0.69s", "client_time_in_millis": 691, "client_total": 100, "fetch_current": 0, "fetch_time": "313ms", "fetch_time_in_millis": 313, "fetch_total": 100, "query_current": 0, "query_time": "62ms", "query_time_in_millis": 62, "query_total": 100}, "match_abs": {"client_time": "0.38s", "client_time_in_millis": 379, "client_total": 100, "fetch_current": 0, "fetch_time": "205ms", "fetch_time_in_millis": 205, "fetch_total": 100, "query_current": 0, "query_time": "51ms", "query_time_in_millis": 51, "query_total": 100}, "match_no_rnd": {"client_time": "0.69s", "client_time_in_millis": 687, "client_total": 100, "fetch_current": 0, "fetch_time": "318ms", "fetch_time_in_millis": 318, "fetch_total": 100, "query_current": 0, "query_time": "55ms", "query_time_in_millis": 55, "query_total": 100}, "match_srt": {"client_time": "0.48s", "client_time_in_millis": 483, "client_total": 100, "fetch_current": 0, "fetch_time": "195ms", "fetch_time_in_millis": 195, "fetch_total": 100, "query_current": 0, "query_time": "69ms", "query_time_in_millis": 69, "query_total": 100}, "mlt": {"client_time": "0.13s", "client_time_in_millis": 131, "client_total": 100, "fetch_current": 0, "fetch_time": "1ms", "fetch_time_in_millis": 1, "fetch_total": 100, "query_current": 0, "query_time": "22ms", "query_time_in_millis": 22, "query_total": 100}}, "open_contexts": 0, "query_current": 0, "query_time": "1.8s", "query_time_in_millis": 1889, "query_total": 4000}, "store": {"size": "2.8mb", "size_in_bytes": 2962937, "throttle_time": "0s", "throttle_time_in_millis": 0}}},"sort":[1385488708000]},{"_index":"stats","_type":"obs","_id":"a0c28e25","_score":null, "_source" : {"meta": {"benchmark_id": "315fd1aa", "observation_id": "a0c28e25", "observation_sequence_no": 9, "observation_start": "2013-11-26T17:58:31Z", "observation_stop": "2013-11-26T17:58:34Z"}, "segments": {"num_committed_segments": 1, "num_search_segments": 1, "segments": {"_a": {"committed": true, "compound": false, "deleted_docs": 0, "generation": 10, "num_docs": 90, "search": true, "size": "2.7mb", "size_in_bytes": 2853865, "version": "4.5.1"}}, "t_optimize": "0.46s", "t_optimize_in_millis": 457}, "stats": {"docs": {"count": 90, "deleted": 0}, "indexing": {"delete_current": 0, "delete_time": "0s", "delete_time_in_millis": 0, "delete_total": 0, "index_current": 0, "index_time": "730ms", "index_time_in_millis": 730, "index_total": 90}, "merges": {"current": 0, "current_docs": 0, "current_size": "0b", "current_size_in_bytes": 0, "total": 1, "total_docs": 90, "total_size": "3mb", "total_size_in_bytes": 3248531, "total_time": "423ms", "total_time_in_millis": 423}, "search": {"fetch_current": 0, "fetch_time": "8.3s", "fetch_time_in_millis": 8334, "fetch_total": 4500, "groups": {"match": {"client_time": "0.65s", "client_time_in_millis": 654, "client_total": 100, "fetch_current": 0, "fetch_time": "326ms", "fetch_time_in_millis": 326, "fetch_total": 100, "query_current": 0, "query_time": "25ms", "query_time_in_millis": 25, "query_total": 100}, "match_abs": {"client_time": "0.35s", "client_time_in_millis": 351, "client_total": 100, "fetch_current": 0, "fetch_time": "210ms", "fetch_time_in_millis": 210, "fetch_total": 100, "query_current": 0, "query_time": "18ms", "query_time_in_millis": 18, "query_total": 100}, "match_no_rnd": {"client_time": "0.67s", "client_time_in_millis": 668, "client_total": 100, "fetch_current": 0, "fetch_time": "334ms", "fetch_time_in_millis": 334, "fetch_total": 100, "query_current": 0, "query_time": "22ms", "query_time_in_millis": 22, "query_total": 100}, "match_srt": {"client_time": "0.65s", "client_time_in_millis": 646, "client_total": 100, "fetch_current": 0, "fetch_time": "304ms", "fetch_time_in_millis": 304, "fetch_total": 100, "query_current": 0, "query_time": "28ms", "query_time_in_millis": 28, "query_total": 100}, "mlt": {"client_time": "0.11s", "client_time_in_millis": 108, "client_total": 100, "fetch_current": 0, "fetch_time": "1ms", "fetch_time_in_millis": 1, "fetch_total": 100, "query_current": 0, "query_time": "7ms", "query_time_in_millis": 7, "query_total": 100}}, "open_contexts": 0, "query_current": 0, "query_time": "1.9s", "query_time_in_millis": 1993, "query_total": 4500}, "store": {"size": "2.7mb", "size_in_bytes": 2854630, "throttle_time": "120.3ms", "throttle_time_in_millis": 120}}},"sort":[1385488711000]},{"_index":"stats","_type":"obs","_id":"6d183b4c","_score":null, "_source" : {"meta": {"benchmark_id": "315fd1aa", "observation_id": "6d183b4c", "observation_sequence_no": 10, "observation_start": "2013-11-26T17:58:34Z", "observation_stop": "2013-11-26T17:58:36Z"}, "segments": {"num_committed_segments": 2, "num_search_segments": 2, "segments": {"_a": {"committed": true, "compound": false, "deleted_docs": 0, "generation": 10, "num_docs": 90, "search": true, "size": "2.7mb", "size_in_bytes": 2853865, "version": "4.5.1"}, "_b": {"committed": true, "compound": true, "deleted_docs": 0, "generation": 11, "num_docs": 10, "search": true, "size": "231.2kb", "size_in_bytes": 236834, "version": "4.5.1"}}, "t_optimize": "0.03s", "t_optimize_in_millis": 26}, "stats": {"docs": {"count": 100, "deleted": 0}, "indexing": {"delete_current": 0, "delete_time": "0s", "delete_time_in_millis": 0, "delete_total": 0, "index_current": 0, "index_time": "765ms", "index_time_in_millis": 765, "index_total": 100}, "merges": {"current": 0, "current_docs": 0, "current_size": "0b", "current_size_in_bytes": 0, "total": 1, "total_docs": 90, "total_size": "3mb", "total_size_in_bytes": 3248531, "total_time": "423ms", "total_time_in_millis": 423}, "search": {"fetch_current": 0, "fetch_time": "9.4s", "fetch_time_in_millis": 9449, "fetch_total": 5000, "groups": {"match": {"client_time": "0.63s", "client_time_in_millis": 627, "client_total": 100, "fetch_current": 0, "fetch_time": "302ms", "fetch_time_in_millis": 302, "fetch_total": 100, "query_current": 0, "query_time": "26ms", "query_time_in_millis": 26, "query_total": 100}, "match_abs": {"client_time": "0.34s", "client_time_in_millis": 342, "client_total": 100, "fetch_current": 0, "fetch_time": "195ms", "fetch_time_in_millis": 195, "fetch_total": 100, "query_current": 0, "query_time": "26ms", "query_time_in_millis": 26, "query_total": 100}, "match_no_rnd": {"client_time": "0.64s", "client_time_in_millis": 637, "client_total": 100, "fetch_current": 0, "fetch_time": "300ms", "fetch_time_in_millis": 300, "fetch_total": 100, "query_current": 0, "query_time": "35ms", "query_time_in_millis": 35, "query_total": 100}, "match_srt": {"client_time": "0.66s", "client_time_in_millis": 662, "client_total": 100, "fetch_current": 0, "fetch_time": "315ms", "fetch_time_in_millis": 315, "fetch_total": 100, "query_current": 0, "query_time": "29ms", "query_time_in_millis": 29, "query_total": 100}, "mlt": {"client_time": "0.12s", "client_time_in_millis": 122, "client_total": 100, "fetch_current": 0, "fetch_time": "1ms", "fetch_time_in_millis": 1, "fetch_total": 100, "query_current": 0, "query_time": "10ms", "query_time_in_millis": 10, "query_total": 100}}, "open_contexts": 0, "query_current": 0, "query_time": "2.1s", "query_time_in_millis": 2122, "query_total": 5000}, "store": {"size": "2.9mb", "size_in_bytes": 3091110, "throttle_time": "120.3ms", "throttle_time_in_millis": 120}}},"sort":[1385488714000]}]}}', curl='curl -XGET http://localhost:9200/stats/obs/_search?q=meta.benchmark_id:315fd1aa&sort=meta.observation_start:asc&size=10000')
 79 |         obs = list(esbench.analyze._observations(resp=esbench.analyze._get_observations()))
 80 |         self.assertEqual(2, len(obs))
 81 | 
 82 | 
 83 |     def test_get(self):
 84 |         for data in esbench.analyze.get_data(None):
 85 |             self.assertEqual(data.keys(), ['benchmark', 'observation'])
 86 | 
 87 | 
 88 |     def test_flatten_container(self):
 89 | 
 90 |         obs = [esbench.analyze.flatten_container(container=o) for o in esbench.analyze.get_data(conn=None)]
 91 |         self.assertEqual(len(obs), 2)
 92 |         self.assertEqual(len(obs[0]), len(obs[1]))
 93 | 
 94 |         obs = [[t for t in esbench.analyze.flatten_container(container=o) if t[0] == u'observation.stats.store.size'] for o in esbench.analyze.get_data(conn=None)]
 95 |         self.assertEqual(obs, [[(u'observation.stats.store.size', u'149.5kb')], [(u'observation.stats.store.size', u'293.9kb')]])
 96 | 
 97 | 
 98 |     def test_filter(self):
 99 |         for data in esbench.analyze.get_data(None):
100 |             f = esbench.analyze.flatten_container(container=data)
101 |             d = esbench.analyze.filter_tuples(f)
102 |             self.assertEqual(sorted(f), d)
103 |             self.assertEqual(esbench.analyze.filter_tuples(f, pattern='benchmark'), esbench.analyze.filter_tuples(f, pattern='benchmark.*'))
104 | 
105 | 
106 |     def test_group_observations(self):
107 |         data = list(esbench.analyze.get_data(None))
108 |         # just check if the thing basically works
109 |         benchmarks = esbench.analyze.group_observations(data=data, fields=esbench.analyze.FIELDS)
110 | #         print(json.dumps(benchmarks, indent=4))
111 |         self.assertEqual('observation.meta.observation_id',benchmarks[0][0][2][0])
112 |         self.assertEqual('e8cd3f18',benchmarks[0][0][2][1])
113 | 
114 | 
115 | class FlattenContainerTest(unittest.TestCase):
116 | 
117 |     def setUp(self):
118 |         self.d = {'foo': [{'bar': 'baz'}, 2, 'monkey', ['foo', {'a': 1, 'b': 2, 'c': 3}], ]}
119 |         self.a = ['foo', 'bar', {'monkey': True}, set(['a1', 'b1']), ('a2', 'b2')]
120 | 
121 |     def test_flatten(self):
122 |         self.assertRaises(ValueError, esbench.analyze.flatten_container, container=iter(range(10)))
123 |         f = esbench.analyze.flatten_container(container=self.d)
124 |         self.assertEqual(sorted(f), [('foo.0.bar', 'baz'), ('foo.1', 2), ('foo.2', 'monkey'), ('foo.3.0', 'foo'), ('foo.3.1.a', 1), ('foo.3.1.b', 2), ('foo.3.1.c', 3)])
125 |         f = esbench.analyze.flatten_container(container=self.a)
126 |         self.assertEqual(sorted(f), [('0', 'foo'), ('1', 'bar'), ('2.monkey', True), ('3.0', 'a1'), ('3.1', 'b1'),  ('4.0', 'a2'), ('4.1', 'b2')])
127 | 
128 | 
129 | if __name__ == "__main__":
130 |     fmt = '%(levelname)s %(name)s.%(funcName)s:%(lineno)s %(message)s'
131 |     logging.basicConfig(level=logging.DEBUG, format=fmt)
132 |     logging.getLogger('esbench.analyze').setLevel(logging.ERROR)
133 |     unittest.main()
134 | 
135 | 


--------------------------------------------------------------------------------
/esbench/test/test_api.py:
--------------------------------------------------------------------------------
  1 | # -*- coding: UTF-8 -*-
  2 | # (c)2013 Mik Kocikowski, MIT License (http://opensource.org/licenses/MIT)
  3 | # https://github.com/mkocikowski/esbench
  4 | 
  5 | import datetime
  6 | import os.path
  7 | import unittest
  8 | import json
  9 | import httplib
 10 | import logging
 11 | 
 12 | import esbench.api
 13 | 
 14 | class MockHTTPResponse(object):
 15 | 
 16 |     def __init__(self, req=None):
 17 |         self.method, self.url, self.body = req if req else (None, None, None)
 18 |         self.status = 200
 19 |         self.reason = 'no reason'
 20 |         try:
 21 |             d = json.loads(self.body)
 22 |             self.status = d.get('status', 200)
 23 |             self.reason = d.get('reason', 'no reason')
 24 |         except (TypeError, ValueError):
 25 |             pass
 26 | 
 27 |     def read(self):
 28 |         return self.body
 29 | 
 30 | 
 31 | class MockHTTPConnection(object):
 32 |     """Mock httplib.HTTPConnection"""
 33 | 
 34 |     def __init__(self, host='localhost', port=9200, timeout=10):
 35 |         self.host = host
 36 |         self.port = port
 37 |         self.timeout = timeout
 38 |         self.sock = None
 39 |         self.req = None
 40 |         self.requests = []
 41 |         self.responses = []
 42 | 
 43 |     def connect(self):
 44 |         self.sock = True
 45 | 
 46 |     def close(self):
 47 |         self.sock = None
 48 | 
 49 |     def request(self, method, url, body=None, headers=None):
 50 |         self.req = (method, url, body)
 51 |         self.requests.append(self.req)
 52 |         return
 53 | 
 54 |     def getresponse(self):
 55 |         resp = MockHTTPResponse(self.req)
 56 |         self.responses.append(resp)
 57 |         return resp
 58 | 
 59 | 
 60 | class ApiConnTest(unittest.TestCase):
 61 | 
 62 |     def test_conn(self):
 63 |         self.assertEqual(esbench.api.Conn().conn_cls, httplib.HTTPConnection)
 64 |         c = esbench.api.Conn(conn_cls=MockHTTPConnection)
 65 |         self.assertEqual(c.__dict__, {'conn': None, 'host': 'localhost', 'port': 9200, 'timeout': 10, 'conn_cls': MockHTTPConnection})
 66 |         c.connect()
 67 |         self.assertIs(c.conn.sock, True)
 68 |         c.close()
 69 |         self.assertIsNone(c.conn)
 70 | 
 71 |     def test_connect(self):
 72 |         with esbench.api.connect(conn_cls=MockHTTPConnection) as c:
 73 |             resp = c.get("foo/bar")
 74 |             self.assertEqual(resp.status, 200)
 75 |             self.assertIs(c.conn.sock, True)
 76 |         self.assertIsNone(c.conn)
 77 | 
 78 |     def test_req_resp(self):
 79 |         # test request and response recording
 80 |         c = esbench.api.Conn(conn_cls=MockHTTPConnection)
 81 |         resp = c.get("foo/bar")
 82 |         self.assertEqual(resp.status, 200)
 83 |         resp = c.get("foo/bar", json.dumps({'status': 404, 'reason': 'not found'}))
 84 |         self.assertEqual(resp.status, 404)
 85 |         self.assertEqual(resp.reason, 'not found')
 86 |         resp = c.delete("foo/bar")
 87 |         self.assertEqual(c.conn.requests, [('GET', '/foo/bar', None), ('GET', '/foo/bar', '{"status": 404, "reason": "not found"}'), ('DELETE', '/foo/bar', None)])
 88 |         self.assertEqual([r.status for r in c.conn.responses], [200, 404, 200])
 89 | 
 90 |     def test_conn_get(self):
 91 |         c = esbench.api.Conn(conn_cls=MockHTTPConnection)
 92 |         resp = c.get("test/_stats")
 93 |         self.assertIsInstance(resp, esbench.api.ApiResponse)
 94 |         self.assertEqual(resp.curl, "curl -XGET http://localhost:9200/test/_stats")
 95 |         resp = c.get("foo/bar", 'baz')
 96 |         self.assertEqual(resp.curl, "curl -XGET http://localhost:9200/foo/bar -d 'baz'")
 97 | 
 98 |     def test_conn_put(self):
 99 |         c = esbench.api.Conn(conn_cls=MockHTTPConnection)
100 |         self.assertRaises(TypeError, c.put, "foo/bar")
101 |         self.assertRaises(ValueError, c.put, "foo/bar", "")
102 |         resp = c.put("foo/bar", "baz")
103 |         self.assertEqual(resp.curl, "curl -XPUT http://localhost:9200/foo/bar -d 'baz'")
104 | 
105 |     def test_conn_post(self):
106 |         c = esbench.api.Conn(conn_cls=MockHTTPConnection)
107 |         # if 'data' is none, it must be explicitly set so
108 |         self.assertRaises(TypeError, c.post, "foo/bar")
109 |         resp = c.post("foo/bar", None)
110 |         self.assertEqual(resp.curl, "curl -XPOST http://localhost:9200/foo/bar")
111 |         resp = c.post("foo/bar", 'baz')
112 |         self.assertEqual(resp.curl, "curl -XPOST http://localhost:9200/foo/bar -d 'baz'")
113 | 
114 |     def test_conn_delete(self):
115 |         c = esbench.api.Conn(conn_cls=MockHTTPConnection)
116 |         resp = c.delete("foo/bar")
117 |         self.assertEqual(resp.curl, "curl -XDELETE http://localhost:9200/foo/bar")
118 | 
119 |     def test_massage_request_path(self):
120 |         self.assertEqual("/", esbench.api._massage_request_path(None))        
121 |         self.assertEqual("/foo", esbench.api._massage_request_path("foo"))
122 |         self.assertEqual("/foo", esbench.api._massage_request_path("//foo"))
123 |         self.assertEqual("/foo?bar", esbench.api._massage_request_path("foo?bar"))
124 | 
125 | class ApiFuncTest(unittest.TestCase):
126 | 
127 |     def setUp(self):
128 |         self.c = esbench.api.Conn(conn_cls=MockHTTPConnection)
129 | 
130 |     def test_document_post(self):
131 |         resp = esbench.api.document_post(self.c, 'i1', 'd1', 'foo')
132 |         self.assertEqual(resp.curl, "curl -XPOST http://localhost:9200/i1/d1 -d 'foo'")
133 | 
134 |     def test_index_create(self):
135 |         resp = esbench.api.index_create(self.c, 'i1', config={'mapping': 'foo'})
136 |         self.assertEqual(resp.curl, """curl -XPUT http://localhost:9200/i1 -d \'{"mapping": "foo"}\'""")
137 | 
138 |     def test_index_delete(self):
139 |         resp = esbench.api.index_delete(self.c, 'i1')
140 |         self.assertEqual(resp.curl, """curl -XDELETE http://localhost:9200/i1""")
141 | 
142 |     def test_index_get_stats(self):
143 |         resp = esbench.api.index_get_stats(self.c, 'i1', '123_mlt,123_match')
144 |         self.assertEqual(resp.curl, """curl -XGET http://localhost:9200/i1/_stats?clear=true&docs=true&store=true&search=true&merge=true&indexing=true&fielddata=true&fields=*&groups=123_mlt,123_match""")
145 | 
146 |     def test_index_set_refresh_interval(self):
147 |         resp = esbench.api.index_set_refresh_interval(self.c, 'i1', '5s')
148 |         self.assertEqual(resp.curl, """curl -XPUT http://localhost:9200/i1/_settings -d \'{"index": {"refresh_interval": "5s"}}\'""")
149 | 
150 |     def test_index_optimize(self):
151 |         resp = esbench.api.index_optimize(self.c, 'i1')
152 |         self.assertEqual(resp.curl, """curl -XPOST http://localhost:9200/i1/_optimize?refresh=true&flush=true&wait_for_merge=true""")
153 |         resp = esbench.api.index_optimize(self.c, 'i1', nseg=10)
154 |         self.assertEqual(resp.curl, """curl -XPOST http://localhost:9200/i1/_optimize?max_num_segments=10&refresh=true&flush=true&wait_for_merge=true""")
155 | 
156 |     def test_index_get_segments(self):
157 |         resp = esbench.api.index_get_segments(self.c, 'i1')
158 |         self.assertEqual(resp.curl, "curl -XGET http://localhost:9200/i1/_segments")
159 | 
160 |     def test_cluster_get_info(self):
161 |         resp = esbench.api.cluster_get_info(self.c)
162 |         self.assertEqual(resp.curl, "curl -XGET http://localhost:9200/_cluster/nodes?settings=true&os=true&process=true&jvm=true&thread_pool=true&network=true&transport=true&http=true&plugin=true")
163 | 
164 |     def test_cluster_get_stats(self):
165 |         resp = esbench.api.cluster_get_stats(self.c)
166 |         self.assertEqual(resp.curl, "curl -XGET http://localhost:9200/_cluster/nodes/stats?indices=true&os=true&process=true&jvm=true&network=true&transport=true&http=true&fs=true&thread_pool=true")
167 | 
168 | 
169 | if __name__ == "__main__":
170 |     logging.basicConfig(level=logging.DEBUG)
171 |     unittest.main()
172 | 
173 | 


--------------------------------------------------------------------------------
/esbench/test/test_bench.py:
--------------------------------------------------------------------------------
  1 | # -*- coding: UTF-8 -*-
  2 | # (c)2013 Mik Kocikowski, MIT License (http://opensource.org/licenses/MIT)
  3 | # https://github.com/mkocikowski/esbench
  4 | 
  5 | import datetime
  6 | import os.path
  7 | import unittest
  8 | import json
  9 | import itertools
 10 | import logging
 11 | 
 12 | import esbench.bench
 13 | import esbench.api
 14 | import esbench.client
 15 | import esbench.test.test_api
 16 | 
 17 | 
 18 | logger = logging.getLogger(__name__)
 19 | 
 20 | class BenchTest(unittest.TestCase):
 21 | 
 22 |     def test_rands(self):
 23 |         s = esbench.bench.rands()
 24 |         self.assertIsInstance(s, str)
 25 |         self.assertEqual(len(s), 6)
 26 |         self.assertNotEqual(esbench.bench.rands(), esbench.bench.rands())
 27 | 
 28 | 
 29 | 
 30 | class SearchQueryTest(unittest.TestCase):
 31 | 
 32 |     def test_execute(self):
 33 | 
 34 |         with self.assertRaises(ValueError):
 35 |             q = esbench.bench.SearchQuery(
 36 |                     name='match',
 37 |                     query=json.dumps({'match': {'foo': 'bar'}}),
 38 |                     observation_id='ABCDEFGH',
 39 |                     index='test',
 40 |                     doctype='doc'
 41 |             )
 42 | 
 43 |         q = esbench.bench.SearchQuery(
 44 |                 name='match',
 45 |                 query={'match': {'foo': 'bar'}},
 46 |                 observation_id='ABCDEFGH',
 47 |                 index='test',
 48 |                 doctype='doc'
 49 |         )
 50 |         c = esbench.api.Conn(conn_cls=esbench.test.test_api.MockHTTPConnection)
 51 |         resp = q.execute(c)
 52 |         self.assertEqual(resp.curl, """curl -XPOST http://localhost:9200/test/doc/_search -d \'{"match": {"foo": "bar"}, "stats": ["ABCDEFGH_match"]}\'""")
 53 |         self.assertEqual(1, q.execution_count)
 54 | 
 55 | 
 56 | class ObservationTest(unittest.TestCase):
 57 | 
 58 |     @classmethod
 59 |     def setUpClass(cls):
 60 |         cls.queries = json.loads("""{"mlt": {"fields": ["description.txt"], "from": 0, "query": {"more_like_this": {"fields": ["description.txt"], "like_text": "USING V%(variable)s PROTECTING THE CARTRIDGE FROM WELDING HEAT", "max_query_terms": 25, "min_term_freq": 2, "percent_terms_to_match": 0.3}}, "size": 10}, "match": {"fields": ["description.txt"], "from": 0, "query": {"match": {"description.txt": "computing V%(variable)s device portable"}}, "size": 10}}""")
 61 | 
 62 | 
 63 |     def setUp(self):
 64 |         self.conn = esbench.api.Conn(conn_cls=esbench.test.test_api.MockHTTPConnection)
 65 |         self.observation = esbench.bench.Observation(
 66 |                         conn = self.conn,
 67 |                         benchmark_id = 'bench1',
 68 |                         queries = self.queries,
 69 |                         reps = 10,
 70 |         )
 71 | 
 72 | 
 73 |     def test_init(self):
 74 |         self.assertIsNone(self.observation.ts_start)
 75 |         self.assertIsNone(self.observation.ts_stop)
 76 |         self.assertEqual(0, self.observation.t_optimize)
 77 |         self.assertIsInstance(self.observation.queries[1], esbench.bench.SearchQuery)
 78 | 
 79 | 
 80 |     def test_segments(self):
 81 | 
 82 |         def _f(conn, index):
 83 |             return esbench.api.ApiResponse(200, 'ok', """{"ok":true,"_shards":{"total":1,"successful":1,"failed":0},"indices":{"esbench_test":{"shards":{"0":[{"routing":{"state":"STARTED","primary":true,"node":"YFJaFqa6Q-m-FPY_IRQ5nw"},"num_committed_segments":3,"num_search_segments":3,"segments":{"_a":{"generation":10,"num_docs":80,"deleted_docs":0,"size":"2.4mb","size_in_bytes":2524210,"committed":true,"search":true,"version":"4.4","compound":false},"_b":{"generation":11,"num_docs":10,"deleted_docs":0,"size":"271.7kb","size_in_bytes":278301,"committed":true,"search":true,"version":"4.4","compound":true},"_c":{"generation":12,"num_docs":10,"deleted_docs":0,"size":"225.3kb","size_in_bytes":230761,"committed":true,"search":true,"version":"4.4","compound":true}}}]}}}}""", "")
 84 |         s = self.observation._segments(segments_f=_f)
 85 |         self.assertEqual(s, {'num_search_segments': 3, 't_optimize': '0.00s', 't_optimize_in_millis': 0, 'num_committed_segments': 3, 'shards': 1})
 86 | 
 87 |         # test aggregation for multiple shards
 88 |         def _f(conn, index):
 89 |             return esbench.api.ApiResponse(200, 'ok', """{"ok":true,"_shards":{"total":6,"successful":6,"failed":0},"indices":{"esbench_test":{"shards":{"0":[{"routing":{"state":"STARTED","primary":false,"node":"HwNlNZuISY6xSkwJN3njdA"},"num_committed_segments":5,"num_search_segments":5,"segments":{"_0":{"generation":0,"num_docs":3,"deleted_docs":0,"size":"83.8kb","size_in_bytes":85904,"committed":true,"search":true,"version":"4.6","compound":true},"_1":{"generation":1,"num_docs":1,"deleted_docs":0,"size":"53.5kb","size_in_bytes":54833,"committed":true,"search":true,"version":"4.6","compound":true},"_2":{"generation":2,"num_docs":2,"deleted_docs":0,"size":"69.5kb","size_in_bytes":71186,"committed":true,"search":true,"version":"4.6","compound":true},"_3":{"generation":3,"num_docs":3,"deleted_docs":0,"size":"108.5kb","size_in_bytes":111161,"committed":true,"search":true,"version":"4.6","compound":true},"_4":{"generation":4,"num_docs":2,"deleted_docs":0,"size":"73kb","size_in_bytes":74845,"committed":true,"search":true,"version":"4.6","compound":true}}},{"routing":{"state":"STARTED","primary":true,"node":"paqTP4jgTtKtBt4vN2kgeQ"},"num_committed_segments":5,"num_search_segments":5,"segments":{"_0":{"generation":0,"num_docs":3,"deleted_docs":0,"size":"83.8kb","size_in_bytes":85904,"committed":true,"search":true,"version":"4.6","compound":true},"_1":{"generation":1,"num_docs":1,"deleted_docs":0,"size":"53.5kb","size_in_bytes":54833,"committed":true,"search":true,"version":"4.6","compound":true},"_2":{"generation":2,"num_docs":2,"deleted_docs":0,"size":"69.5kb","size_in_bytes":71186,"committed":true,"search":true,"version":"4.6","compound":true},"_3":{"generation":3,"num_docs":3,"deleted_docs":0,"size":"108.5kb","size_in_bytes":111161,"committed":true,"search":true,"version":"4.6","compound":true},"_4":{"generation":4,"num_docs":2,"deleted_docs":0,"size":"73kb","size_in_bytes":74845,"committed":true,"search":true,"version":"4.6","compound":true}}},{"routing":{"state":"STARTED","primary":false,"node":"nqvaE38LTESFVxwxu-4s6A"},"num_committed_segments":5,"num_search_segments":5,"segments":{"_0":{"generation":0,"num_docs":3,"deleted_docs":0,"size":"83.8kb","size_in_bytes":85904,"committed":true,"search":true,"version":"4.6","compound":true},"_1":{"generation":1,"num_docs":1,"deleted_docs":0,"size":"53.5kb","size_in_bytes":54833,"committed":true,"search":true,"version":"4.6","compound":true},"_2":{"generation":2,"num_docs":2,"deleted_docs":0,"size":"69.5kb","size_in_bytes":71186,"committed":true,"search":true,"version":"4.6","compound":true},"_3":{"generation":3,"num_docs":3,"deleted_docs":0,"size":"108.5kb","size_in_bytes":111161,"committed":true,"search":true,"version":"4.6","compound":true},"_4":{"generation":4,"num_docs":2,"deleted_docs":0,"size":"73kb","size_in_bytes":74845,"committed":true,"search":true,"version":"4.6","compound":true}}}],"1":[{"routing":{"state":"STARTED","primary":false,"node":"HwNlNZuISY6xSkwJN3njdA"},"num_committed_segments":5,"num_search_segments":5,"segments":{"_0":{"generation":0,"num_docs":3,"deleted_docs":0,"size":"78.7kb","size_in_bytes":80655,"committed":true,"search":true,"version":"4.6","compound":true},"_1":{"generation":1,"num_docs":4,"deleted_docs":0,"size":"101.8kb","size_in_bytes":104339,"committed":true,"search":true,"version":"4.6","compound":true},"_2":{"generation":2,"num_docs":3,"deleted_docs":0,"size":"76.6kb","size_in_bytes":78520,"committed":true,"search":true,"version":"4.6","compound":true},"_3":{"generation":3,"num_docs":2,"deleted_docs":0,"size":"51.5kb","size_in_bytes":52782,"committed":true,"search":true,"version":"4.6","compound":true},"_4":{"generation":4,"num_docs":3,"deleted_docs":0,"size":"112.9kb","size_in_bytes":115657,"committed":true,"search":true,"version":"4.6","compound":true}}},{"routing":{"state":"STARTED","primary":false,"node":"paqTP4jgTtKtBt4vN2kgeQ"},"num_committed_segments":5,"num_search_segments":5,"segments":{"_0":{"generation":0,"num_docs":3,"deleted_docs":0,"size":"78.7kb","size_in_bytes":80655,"committed":true,"search":true,"version":"4.6","compound":true},"_1":{"generation":1,"num_docs":4,"deleted_docs":0,"size":"101.8kb","size_in_bytes":104339,"committed":true,"search":true,"version":"4.6","compound":true},"_2":{"generation":2,"num_docs":3,"deleted_docs":0,"size":"76.6kb","size_in_bytes":78520,"committed":true,"search":true,"version":"4.6","compound":true},"_3":{"generation":3,"num_docs":2,"deleted_docs":0,"size":"51.5kb","size_in_bytes":52782,"committed":true,"search":true,"version":"4.6","compound":true},"_4":{"generation":4,"num_docs":3,"deleted_docs":0,"size":"112.9kb","size_in_bytes":115657,"committed":true,"search":true,"version":"4.6","compound":true}}},{"routing":{"state":"STARTED","primary":true,"node":"nqvaE38LTESFVxwxu-4s6A"},"num_committed_segments":5,"num_search_segments":5,"segments":{"_0":{"generation":0,"num_docs":3,"deleted_docs":0,"size":"78.7kb","size_in_bytes":80655,"committed":true,"search":true,"version":"4.6","compound":true},"_1":{"generation":1,"num_docs":4,"deleted_docs":0,"size":"101.8kb","size_in_bytes":104339,"committed":true,"search":true,"version":"4.6","compound":true},"_2":{"generation":2,"num_docs":3,"deleted_docs":0,"size":"76.6kb","size_in_bytes":78520,"committed":true,"search":true,"version":"4.6","compound":true},"_3":{"generation":3,"num_docs":2,"deleted_docs":0,"size":"51.5kb","size_in_bytes":52782,"committed":true,"search":true,"version":"4.6","compound":true},"_4":{"generation":4,"num_docs":3,"deleted_docs":0,"size":"112.9kb","size_in_bytes":115657,"committed":true,"search":true,"version":"4.6","compound":true}}}]}}}}""", '')
 90 |         s = self.observation._segments(segments_f=_f)
 91 |         self.assertEqual(s, {'num_search_segments': 30, 't_optimize': '0.00s', 't_optimize_in_millis': 0, 'num_committed_segments': 30, 'shards': 6})
 92 | 
 93 | 
 94 |     def test_stats(self):
 95 | 
 96 |         def _f(conn, index, stats_group_names):
 97 |             self.assertEqual(stats_group_names, ",".join([q.stats_group_name for q in self.observation.queries]))
 98 |             return esbench.api.ApiResponse(200, 'ok', """{"ok":true,"_shards":{"total":1,"successful":1,"failed":0},"_all":{"primaries":{"docs":{"count":100,"deleted":0},"store":{"size":"2.8mb","size_in_bytes":3024230,"throttle_time":"99.9ms","throttle_time_in_millis":99},"indexing":{"index_total":100,"index_time":"781ms","index_time_in_millis":781,"index_current":0,"delete_total":0,"delete_time":"0s","delete_time_in_millis":0,"delete_current":0},"search":{"open_contexts":0,"query_total":4000,"query_time":"2.2s","query_time_in_millis":2218,"query_current":0,"fetch_total":4000,"fetch_time":"11.1s","fetch_time_in_millis":11108,"fetch_current":0,"groups":{"7140aefb_match_abs":{"query_total":100,"query_time":"28ms","query_time_in_millis":28,"query_current":0,"fetch_total":100,"fetch_time":"91ms","fetch_time_in_millis":91,"fetch_current":0},"7140aefb_mlt":{"query_total":100,"query_time":"11ms","query_time_in_millis":11,"query_current":0,"fetch_total":100,"fetch_time":"1ms","fetch_time_in_millis":1,"fetch_current":0},"7140aefb_match":{"query_total":100,"query_time":"19ms","query_time_in_millis":19,"query_current":0,"fetch_total":100,"fetch_time":"142ms","fetch_time_in_millis":142,"fetch_current":0},"7140aefb_match_srt":{"query_total":100,"query_time":"22ms","query_time_in_millis":22,"query_current":0,"fetch_total":100,"fetch_time":"137ms","fetch_time_in_millis":137,"fetch_current":0}}},"merges":{"current":0,"current_docs":0,"current_size":"0b","current_size_in_bytes":0,"total":1,"total_time":"310ms","total_time_in_millis":310,"total_docs":84,"total_size":"2.8mb","total_size_in_bytes":2946979}},"total":{"docs":{"count":100,"deleted":0},"store":{"size":"2.8mb","size_in_bytes":3024230,"throttle_time":"99.9ms","throttle_time_in_millis":99},"indexing":{"index_total":100,"index_time":"781ms","index_time_in_millis":781,"index_current":0,"delete_total":0,"delete_time":"0s","delete_time_in_millis":0,"delete_current":0},"search":{"open_contexts":0,"query_total":4000,"query_time":"2.2s","query_time_in_millis":2218,"query_current":0,"fetch_total":4000,"fetch_time":"11.1s","fetch_time_in_millis":11108,"fetch_current":0,"groups":{"7140aefb_match_abs":{"query_total":100,"query_time":"28ms","query_time_in_millis":28,"query_current":0,"fetch_total":100,"fetch_time":"91ms","fetch_time_in_millis":91,"fetch_current":0},"7140aefb_mlt":{"query_total":100,"query_time":"11ms","query_time_in_millis":11,"query_current":0,"fetch_total":100,"fetch_time":"1ms","fetch_time_in_millis":1,"fetch_current":0},"7140aefb_match":{"query_total":100,"query_time":"19ms","query_time_in_millis":19,"query_current":0,"fetch_total":100,"fetch_time":"142ms","fetch_time_in_millis":142,"fetch_current":0},"7140aefb_match_srt":{"query_total":100,"query_time":"22ms","query_time_in_millis":22,"query_current":0,"fetch_total":100,"fetch_time":"137ms","fetch_time_in_millis":137,"fetch_current":0}}},"merges":{"current":0,"current_docs":0,"current_size":"0b","current_size_in_bytes":0,"total":1,"total_time":"310ms","total_time_in_millis":310,"total_docs":84,"total_size":"2.8mb","total_size_in_bytes":2946979}}},"indices":{"esbench_test":{"primaries":{"docs":{"count":100,"deleted":0},"store":{"size":"2.8mb","size_in_bytes":3024230,"throttle_time":"99.9ms","throttle_time_in_millis":99},"indexing":{"index_total":100,"index_time":"781ms","index_time_in_millis":781,"index_current":0,"delete_total":0,"delete_time":"0s","delete_time_in_millis":0,"delete_current":0},"search":{"open_contexts":0,"query_total":4000,"query_time":"2.2s","query_time_in_millis":2218,"query_current":0,"fetch_total":4000,"fetch_time":"11.1s","fetch_time_in_millis":11108,"fetch_current":0,"groups":{"7140aefb_match_abs":{"query_total":100,"query_time":"28ms","query_time_in_millis":28,"query_current":0,"fetch_total":100,"fetch_time":"91ms","fetch_time_in_millis":91,"fetch_current":0},"7140aefb_mlt":{"query_total":100,"query_time":"11ms","query_time_in_millis":11,"query_current":0,"fetch_total":100,"fetch_time":"1ms","fetch_time_in_millis":1,"fetch_current":0},"7140aefb_match":{"query_total":100,"query_time":"19ms","query_time_in_millis":19,"query_current":0,"fetch_total":100,"fetch_time":"142ms","fetch_time_in_millis":142,"fetch_current":0},"7140aefb_match_srt":{"query_total":100,"query_time":"22ms","query_time_in_millis":22,"query_current":0,"fetch_total":100,"fetch_time":"137ms","fetch_time_in_millis":137,"fetch_current":0}}},"merges":{"current":0,"current_docs":0,"current_size":"0b","current_size_in_bytes":0,"total":1,"total_time":"310ms","total_time_in_millis":310,"total_docs":84,"total_size":"2.8mb","total_size_in_bytes":2946979}},"total":{"docs":{"count":100,"deleted":0},"store":{"size":"2.8mb","size_in_bytes":3024230,"throttle_time":"99.9ms","throttle_time_in_millis":99},"indexing":{"index_total":100,"index_time":"781ms","index_time_in_millis":781,"index_current":0,"delete_total":0,"delete_time":"0s","delete_time_in_millis":0,"delete_current":0},"search":{"open_contexts":0,"query_total":4000,"query_time":"2.2s","query_time_in_millis":2218,"query_current":0,"fetch_total":4000,"fetch_time":"11.1s","fetch_time_in_millis":11108,"fetch_current":0,"groups":{"7140aefb_match_abs":{"query_total":100,"query_time":"28ms","query_time_in_millis":28,"query_current":0,"fetch_total":100,"fetch_time":"91ms","fetch_time_in_millis":91,"fetch_current":0},"7140aefb_mlt":{"query_total":100,"query_time":"11ms","query_time_in_millis":11,"query_current":0,"fetch_total":100,"fetch_time":"1ms","fetch_time_in_millis":1,"fetch_current":0},"7140aefb_match":{"query_total":100,"query_time":"19ms","query_time_in_millis":19,"query_current":0,"fetch_total":100,"fetch_time":"142ms","fetch_time_in_millis":142,"fetch_current":0},"7140aefb_match_srt":{"query_total":100,"query_time":"22ms","query_time_in_millis":22,"query_current":0,"fetch_total":100,"fetch_time":"137ms","fetch_time_in_millis":137,"fetch_current":0}}},"merges":{"current":0,"current_docs":0,"current_size":"0b","current_size_in_bytes":0,"total":1,"total_time":"310ms","total_time_in_millis":310,"total_docs":84,"total_size":"2.8mb","total_size_in_bytes":2946979}}}}}""", "")
 99 | 
100 |         s = self.observation._stats(stats_f=_f)
101 |         self.assertEqual(s['docs']['count'], 100)
102 |         self.assertIsNone(s['search']['groups']['mlt']['client_time'])
103 |         self.assertEqual(0, s['search']['groups']['mlt']['client_total'])
104 |         self.assertEqual(s['store']['size_in_bytes'], 3024230)
105 | 
106 | 
107 |     def test_run(self):
108 |         self.observation.run()
109 |         self.assertIsNotNone(self.observation.ts_start)
110 |         self.assertIsNotNone(self.observation.ts_stop)
111 |         self.assertEqual(20, len(self.conn.conn.requests))
112 |         self.assertEqual(10, self.observation.queries[0].execution_count)
113 |         q = json.loads(self.conn.conn.req[2])
114 |         self.assertEqual(
115 |             self.observation.queries[1].stats_group_name,
116 |             json.loads(self.conn.conn.req[2])['stats'][0])
117 | 
118 | 
119 |     def test_record(self):
120 |         self.observation.run()
121 |         self.observation._stats = lambda: {}
122 |         self.observation._segments = lambda: {}
123 |         resp = self.observation.record()
124 |         data = json.loads(resp.data)
125 |         self.assertEqual(set(['cluster', 'segments', 'meta', 'stats']), set(data.keys()))
126 |         self.assertEqual(data['meta']['benchmark_id'], self.observation.benchmark_id)
127 | 
128 | 
129 | class MockObservation(object):
130 | 
131 |     def __init__(self, *args, **kwargs):
132 |         self.did_run = False
133 |         self.did_record = False
134 |         self.__dict__.update(kwargs)
135 | #         logger.debug(self.__dict__)
136 | 
137 |     def run(self):
138 |         self.did_run = True
139 | 
140 |     def record(self):
141 |         self.did_record = True
142 | 
143 | 
144 | class BenchmarkTest(unittest.TestCase):
145 | 
146 |     def setUp(self):
147 |         self.conn = esbench.api.Conn(conn_cls=esbench.test.test_api.MockHTTPConnection)
148 |         self.argv = esbench.client.args_parser().parse_args("run --observations 20".split())
149 |         self.config = esbench.client.merge_config(self.argv, esbench.client.load_config(self.argv.config_file_path))
150 |         self.bench = esbench.bench.Benchmark(self.config, self.conn)
151 | 
152 | 
153 |     def test_init(self):
154 |         pass
155 | 
156 | 
157 |     def test_prepare(self):
158 |         self.assertIsNone(self.bench.ts_start)
159 |         self.assertIsNone(self.bench.t1)
160 |         self.bench.prepare()
161 |         self.assertIsNotNone(self.bench.ts_start)
162 |         self.assertIsNotNone(self.bench.t1)
163 | 
164 | 
165 |     def test_load(self):
166 |         lines = ("line_%02i" % i for i in range(12))
167 |         counts = [self.bench.load(itertools.islice(lines, 10)) for _ in range(3)]
168 |         self.assertEqual(counts, [(10, 70), (2, 14), (0, 0)])
169 | 
170 | 
171 |     def test_run(self):
172 | 
173 |         self.obs_count = 0
174 | 
175 |         def _obs():
176 |             self.obs_count += 1
177 | 
178 |         batches = esbench.data.batches_iterator(("line_%02i" % i for i in range(100)), batch_count=10, max_n=100, max_byte_size=0)
179 |         self.bench.observe = _obs
180 |         self.bench.run(batches)
181 |         self.assertEqual(len(self.conn.conn.requests), 103)
182 |         self.assertEqual(self.conn.conn.requests[:4], [('PUT', "/esbench_stats", '{"settings": {"index": {"number_of_replicas": 0, "number_of_shards": 1}}}'), ('DELETE', '/esbench_test', None), ('PUT', '/esbench_test', '{"mappings": {"doc": {"_size": {"enabled": true, "store": "yes"}, "properties": {"abstract": {"type": "string", "store": "yes"}}, "_source": {"enabled": true}}}, "settings": {"index": {"number_of_replicas": 0, "number_of_shards": 1}}}'), ('POST', '/esbench_test/doc', 'line_00')])
183 |         self.assertEqual(self.obs_count, 10)
184 | 
185 |         self.obs_count = 0
186 |         self.conn = esbench.api.Conn(conn_cls=esbench.test.test_api.MockHTTPConnection)
187 |         self.cmnd = "run --append"
188 |         self.argv = esbench.client.args_parser().parse_args(self.cmnd.split())
189 |         self.config = esbench.client.merge_config(self.argv, esbench.client.load_config(self.argv.config_file_path))
190 |         self.bench = esbench.bench.Benchmark(config=self.config, conn=self.conn)
191 |         batches = esbench.data.batches_iterator(("line_%02i" % i for i in range(100)), batch_count=10, max_n=100, max_byte_size=0)
192 |         self.bench.observe = _obs
193 |         self.bench.run(batches)
194 |         self.assertEqual(len(self.conn.conn.requests), 101)
195 |         self.assertEqual(self.conn.conn.requests[:4], [('PUT', "/esbench_stats", '{"settings": {"index": {"number_of_replicas": 0, "number_of_shards": 1}}}'), ('POST', '/esbench_test/doc', 'line_00'), ('POST', '/esbench_test/doc', 'line_01'), ('POST', '/esbench_test/doc', 'line_02')])
196 |         self.assertEqual(self.obs_count, 10)
197 | 
198 |         self.obs_count = 0
199 |         self.conn = esbench.api.Conn(conn_cls=esbench.test.test_api.MockHTTPConnection)
200 |         self.cmnd = "run --append --observations 5"
201 |         self.argv = esbench.client.args_parser().parse_args(self.cmnd.split())
202 |         self.config = esbench.client.merge_config(self.argv, esbench.client.load_config(self.argv.config_file_path))
203 |         self.bench = esbench.bench.Benchmark(config=self.config, conn=self.conn)
204 |         batches = esbench.data.batches_iterator(("line_%02i" % i for i in range(100)), batch_count=5, max_n=100, max_byte_size=0)
205 |         self.bench.observe = _obs
206 |         self.bench.run(batches)
207 |         self.assertEqual(len(self.conn.conn.requests), 101)
208 |         self.assertEqual(self.conn.conn.requests[:4], [('PUT', "/esbench_stats", '{"settings": {"index": {"number_of_replicas": 0, "number_of_shards": 1}}}'), ('POST', '/esbench_test/doc', 'line_00'), ('POST', '/esbench_test/doc', 'line_01'), ('POST', '/esbench_test/doc', 'line_02')])
209 |         self.assertEqual(self.obs_count, 5)
210 | 
211 | 
212 |     def test_observe(self):
213 |         self.bench.config['config']['segments'] = 10
214 |         obs = self.bench.observe(obs_cls=MockObservation)
215 |         self.assertEqual(self.conn.conn.requests, [('POST', '/esbench_test/_optimize?max_num_segments=10&refresh=true&flush=true&wait_for_merge=true', None)])
216 |         self.assertTrue(obs.did_run)
217 |         self.assertTrue(obs.did_record)
218 | 
219 | 
220 |     def test_record(self):
221 |         # must call .prepare() first
222 |         self.assertRaises(TypeError, self.bench.record)
223 |         self.bench.prepare()
224 |         self.bench._get_cluster_info = lambda: {'foo': 'bar'}
225 |         resp = self.bench.record()
226 |         data = json.loads(resp.data)
227 |         self.assertEqual(set(['meta', 'cluster']), set(data.keys()))
228 | #         self.assertEqual(data['meta']['argv']['maxsize'], '1mb')
229 |         self.assertEqual(data['cluster']['foo'], 'bar')
230 |         # TODO: more tests?
231 | 
232 | 
233 | if __name__ == "__main__":
234 |     logging.basicConfig(level=logging.DEBUG)
235 |     unittest.main()
236 | 
237 | 


--------------------------------------------------------------------------------
/esbench/test/test_client.py:
--------------------------------------------------------------------------------
  1 | # -*- coding: UTF-8 -*-
  2 | # (c)2013 Mik Kocikowski, MIT License (http://opensource.org/licenses/MIT)
  3 | # https://github.com/mkocikowski/esbench
  4 | 
  5 | import datetime
  6 | import os.path
  7 | import unittest
  8 | import json
  9 | import contextlib
 10 | import types
 11 | import collections
 12 | import sys
 13 | import StringIO
 14 | import tempfile
 15 | import time
 16 | import copy
 17 | 
 18 | import esbench.client
 19 | 
 20 | 
 21 | class ClientTest(unittest.TestCase):
 22 | 
 23 |     def setUp(self):
 24 |         # this is needed to supress output from argparse '-h'
 25 |         self.tmp = sys.stdout
 26 |         sys.stdout = StringIO.StringIO()
 27 | 
 28 |     def tearDown(self):
 29 |         sys.stdout = self.tmp
 30 | 
 31 | 
 32 |     def test_args_run(self):
 33 | 
 34 |         parser = esbench.client.args_parser()
 35 |         args = parser.parse_args("run".split())
 36 |         self.assertEqual(args.__dict__,
 37 |             {
 38 |                 'verbose': False,
 39 |                 'segments': None,
 40 |                 'reps': None,
 41 |                 'maxsize': '1mb',
 42 |                 'name': args.name, # cheating, but no clean way around it as it contains timestamp
 43 |                 'no_load': False,
 44 |                 'command': 'run',
 45 |                 'observations': None,
 46 |                 'data': None,
 47 |                 'append': False,
 48 |                 'config_file_path': os.path.normpath(os.path.join(os.path.dirname(os.path.abspath(__file__)), "../", "config.json")),
 49 |                 'host': 'localhost',
 50 |                 'port': 9200,
 51 |                 'shards': None,
 52 |             }
 53 |         )
 54 | 
 55 |         # running with -h flag will catch some errors
 56 |         self.assertRaises(SystemExit, parser.parse_args, "run -h".split())
 57 | 
 58 | 
 59 |     def test_args_show(self):
 60 | 
 61 |         parser = esbench.client.args_parser()
 62 |         args = parser.parse_args("show".split())
 63 |         self.assertEqual(args.__dict__,
 64 |             {
 65 |                 'command': 'show',
 66 |                 'fields': '(?!observation.segments.segments)((benchmark.meta.benchmark_start)|(observation.meta.benchmark_id)|(observation.meta.observation_id)|(observation.meta.observation_sequence_no)|(observation.segments.num_committed_segments)|(observation.segments.num_search_segments)|(observation.segments.t_optimize_in_millis)|(observation.stats.docs.count)|(observation.stats.store.size_in_bytes)|(observation.stats.fielddata.memory_size_in_bytes)|(observation.stats.search.groups.*query_time_in_millis_per_query$))',
 67 |                 'host': 'localhost',
 68 |                 'port': 9200,
 69 |                 'format': 'csv',
 70 |                 'verbose': False,
 71 |                 'ids': ['all'],
 72 |             }
 73 |         )
 74 | 
 75 |         # running with -h flag will catch some errors
 76 |         self.assertRaises(SystemExit, parser.parse_args, "show -h".split())
 77 | 
 78 | 
 79 |     def test_parse_maxsize(self):
 80 | 
 81 |         self.assertRaises(AttributeError, esbench.client.parse_maxsize, (10,))
 82 |         self.assertEqual((10, 0), esbench.client.parse_maxsize('10'))
 83 |         self.assertEqual((0, 1<<10), esbench.client.parse_maxsize('1kb'))
 84 |         self.assertEqual((0, 1<<20), esbench.client.parse_maxsize('1mb'))
 85 | 
 86 | 
 87 | class TestConfig(unittest.TestCase):
 88 | 
 89 |     def setUp(self):
 90 |         self.tf = tempfile.NamedTemporaryFile()
 91 |         self.base_config = {
 92 |             "index": {
 93 |                 "settings": {
 94 |                     "index": {
 95 |                         "number_of_replicas": 0,
 96 |                         "number_of_shards": 1,
 97 |                     }
 98 |                 }
 99 |             },
100 |             "config": {
101 |                 "observations": 10,
102 |                 "segments": None,
103 |                 "reps": 100
104 |             }
105 |         }
106 |         self.tf.write(json.dumps(self.base_config))
107 |         self.tf.flush()
108 | #         self.argv = esbench.client.args_parser().parse_args("run")
109 |         self.maxDiff = None
110 | 
111 |     def tearDown(self):
112 |         self.tf.close()
113 | 
114 |     def test_load_config(self):
115 |         c = esbench.client.load_config(self.tf.name)
116 |         self.assertEqual(c, self.base_config)
117 | 
118 |     def test_merge_config(self):
119 | 
120 |         argv = esbench.client.args_parser().parse_args("run".split())
121 |         c = esbench.client.merge_config(argv=argv, config=copy.deepcopy(self.base_config))
122 |         c['config']['name'] = None # need to clear it as it is a timestamp, will fail tests, changes at every test run
123 |         c['config']['config_file_path'] = None # ditto
124 |         self.assertEqual(c, {
125 |                 'index': {
126 |                     'settings': {
127 |                         'index': {
128 |                             'number_of_replicas': 0,
129 |                             'number_of_shards': 1
130 |                         }
131 |                     }
132 |                 },
133 |                 'config': {
134 |                     'verbose': False,
135 |                     'segments': None,
136 |                     'reps': 100,
137 |                     'shards': None,
138 |                     'maxsize': '1mb',
139 |                     'no_load': False,
140 |                     'command': 'run',
141 |                     'observations': 10,
142 |                     'host': 'localhost',
143 |                     'config_file_path': None,
144 |                     'data': None,
145 |                     'port': 9200,
146 |                     'append': False,
147 |                     'name': None,
148 |                     'max_byte_size': 1048576,
149 |                     'max_n': 0
150 |                 }
151 |             }
152 |         )
153 | 
154 |         argv = esbench.client.args_parser().parse_args("run --observations 100 --shards 2 100mb".split())
155 |         c = esbench.client.merge_config(argv=argv, config=copy.deepcopy(self.base_config))
156 |         self.assertEqual(c['config']['observations'], 100)
157 |         self.assertEqual(c['config']['shards'], 2)
158 |         self.assertEqual(c['index']['settings']['index']['number_of_shards'], 2)
159 |         self.assertEqual(c['config']['maxsize'], '100mb')
160 |         self.assertEqual(c['config']['max_byte_size'], 104857600)
161 |         self.assertEqual(c['config']['max_n'], 0)
162 | 
163 |         argv = esbench.client.args_parser().parse_args("run --observations 100 --no-load --shards 2 100".split())
164 |         c = esbench.client.merge_config(argv=argv, config=copy.deepcopy(self.base_config))
165 |         self.assertEqual(c['config']['max_byte_size'], 0)
166 |         self.assertEqual(c['config']['max_n'], 100)
167 |         self.assertEqual(c['config']['no_load'], True)
168 | 
169 | 
170 | if __name__ == "__main__":
171 |     unittest.main()
172 | 
173 | 


--------------------------------------------------------------------------------
/esbench/test/test_data.py:
--------------------------------------------------------------------------------
 1 | # -*- coding: UTF-8 -*-
 2 | # (c)2013 Mik Kocikowski, MIT License (http://opensource.org/licenses/MIT)
 3 | # https://github.com/mkocikowski/esbench
 4 | 
 5 | import datetime
 6 | import os.path
 7 | import unittest
 8 | import json
 9 | import logging
10 | 
11 | import esbench.data
12 | 
13 | class DataTest(unittest.TestCase):
14 | 
15 |     def test_aa(self):
16 | 
17 |         l = list(esbench.data._aa())
18 |         self.assertEqual(len(l), 676)
19 |         self.assertEqual(l[0], 'aa')
20 |         self.assertEqual(l[-1], 'zz')
21 |         l = list(esbench.data._aa(10))
22 |         self.assertEqual(len(l), 10)
23 |         self.assertEqual(l[0], 'aa')
24 |         self.assertEqual(l[-1], 'aj')
25 | 
26 | 
27 |     def test_urls(self):
28 | 
29 |         urls = list(esbench.data.urls(url_template=esbench.data.URL_TEMPLATE))
30 |         self.assertEqual(urls[0], "https://s3-us-west-1.amazonaws.com/esbench/appl_2005_aa.gz")
31 |         self.assertEqual(urls[75], "https://s3-us-west-1.amazonaws.com/esbench/appl_2006_aa.gz")
32 |         self.assertEqual(len(urls), 8*75)
33 | 
34 | 
35 |     def test_download(self):
36 | 
37 |         resp = esbench.data.download("foo.com/bar")
38 |         self.assertIsNone(resp)
39 | #         self.assertRaises(ValueError, esbench.data.download, "foobar")
40 | 
41 | 
42 |     def test_get_data(self):
43 | 
44 |         line = esbench.data.get_data().next()
45 |         self.assertEqual(u'2005-01-06', json.loads(line)['dates']['date_published'])
46 | 
47 |         # make sure that we can skip over nonexistent urls
48 |         line = esbench.data.get_data(urls_f=lambda x: ["http://foo.bar/baz", "https://s3-us-west-1.amazonaws.com/esbench/appl_2005_aa.gz"]).next()
49 |         self.assertEqual(u'2005-01-06', json.loads(line)['dates']['date_published'])
50 | 
51 | 
52 |     def test_get_feed(self):
53 | 
54 |         with esbench.data.feed() as f:
55 |             line = f.next()
56 |             self.assertEqual(u'2005-01-06', json.loads(line)['dates']['date_published'])
57 | 
58 |         with esbench.data.feed(lines_i=iter([1,2,3])) as f:
59 |             line = f.next()
60 |             self.assertEqual(1, line)
61 | 
62 |         with open("/tmp/esbench_test_get_feed", "w") as f: f.write('foo\nbar\nbaz\n')
63 |         with esbench.data.feed(path="/tmp/esbench_test_get_feed") as f:
64 |             line = f.next().strip()
65 |             self.assertEqual('foo', line)
66 | 
67 | 
68 |     def test_batch_iterator(self):
69 | 
70 |         lines = ("line %i" % i for i in range(100))
71 | 
72 |         batch = esbench.data.batch_iterator(lines=lines, max_batch_n=10, max_batch_byte_size=0)
73 |         batch_l = list(batch)
74 |         self.assertEqual(10, len(batch_l))
75 | 
76 |         batch = esbench.data.batch_iterator(lines=lines, max_batch_n=0, max_batch_byte_size=40)
77 |         batch_l = list(batch)
78 |         self.assertEqual(6, len(batch_l))
79 |         self.assertEqual(42, sum([len(l) for l in batch_l]))
80 | 
81 | 
82 |     def test_batches_iterator(self):
83 | 
84 |         lines = ("line %i" % i for i in range(100))
85 | 
86 |         batches = list(esbench.data.batches_iterator(lines, batch_count=10, max_n=50, max_byte_size=0))
87 |         self.assertEqual(10, len(batches))
88 |         for batch in batches:
89 |             lines = list(batch)
90 |             self.assertEqual(5, len(lines))
91 | 
92 | 
93 | 
94 | if __name__ == "__main__":
95 |     logging.basicConfig(level=logging.DEBUG)
96 |     unittest.main()
97 | 
98 | 


--------------------------------------------------------------------------------
/esbench/test/units.py:
--------------------------------------------------------------------------------
 1 | # -*- coding: UTF-8 -*-
 2 | # (c)2013 Mik Kocikowski, MIT License (http://opensource.org/licenses/MIT)
 3 | # https://github.com/mkocikowski/esbench
 4 | 
 5 | 
 6 | import sys
 7 | import os.path
 8 | import unittest
 9 | import logging
10 | 
11 | logging.basicConfig(level=logging.CRITICAL)
12 | 
13 | 
14 | def suite():
15 | #     return unittest.defaultTestLoader.discover(os.path.dirname(__file__), top_level_dir=os.path.abspath("../../"))
16 |     return unittest.defaultTestLoader.discover(os.path.dirname(__file__))
17 | 
18 | if __name__ == "__main__":
19 | 
20 |     runner = unittest.TextTestRunner(verbosity=2)
21 |     result = runner.run(suite())
22 | 
23 |     # doing sys.exit(1) on test failure will signal test failure to other
24 |     # processes (this is for when the suite is run automatically, not by hand
25 |     # from the command line)
26 |     #
27 |     if not result.wasSuccessful():
28 |         sys.exit(1)
29 | 
30 | 


--------------------------------------------------------------------------------
/setup.py:
--------------------------------------------------------------------------------
 1 | # -*- coding: UTF-8 -*-
 2 | # (c)2013 Mik Kocikowski, MIT License (http://opensource.org/licenses/MIT)
 3 | # https://github.com/mkocikowski/esbench
 4 | 
 5 | 
 6 | # python setup.py -r http://testpypi.python.org/pypi register
 7 | # python setup.py sdist upload -r http://pypi.python.org/pypi
 8 | 
 9 | # import ez_setup
10 | # ez_setup.use_setuptools()
11 | #
12 | from setuptools import setup
13 | 
14 | ld = """Tool for benchmarking performance of Elasticsearch nodes.
15 | 
16 | The two primary uses are for capacity planning (guessing how much oomph you
17 | need to do what what you think you need to do), and for performance tuning
18 | (trying out various index, mapping, and query settings in a consistent and
19 | reproducible manner).
20 | 
21 | An Elasticsearch index is composed of a set of 1 or more Lucene indexes
22 | (designated as primary and replica 'shards' by ES). A single Lucene index is
23 | the basic unit on which indexing and search operations are executed, and so
24 | the performance of individual Lucene indexes largely determines the
25 | performance of a cluster.
26 | 
27 | The basic approach is to create an index with 1 primary and no replica shards
28 | (a single Lucene index), load it with data, and periodically run
29 | representative use patterns against it, recording observations, adding more
30 | data until the performance drops below acceptable levels.
31 | 
32 | This tool comes with 'batteries included' (ie large sample data set,
33 | downloaded on demand). See the README.md file, or even better, the project's
34 | github page.
35 | 
36 | """
37 | 
38 | setup(
39 |     name = 'esbench',
40 |     version = '0.2.3',
41 |     author = 'Mik Kocikowski',
42 |     author_email = 'mkocikowski@gmail.com',
43 |     url = 'https://github.com/mkocikowski/esbench',
44 |     description = 'Elasticsearch performance benchmark tool',
45 |     long_description = ld,
46 |     install_requires = ['tabulate >= 0.6', ],
47 |     packages = ['esbench', 'esbench.test'],
48 |     package_data = {
49 |         '': ['README.md'],
50 |         'esbench': ['config.json'],
51 |     },
52 |     entry_points = {
53 |         'console_scripts': [
54 |             'esbench = esbench.client:main',
55 | #             'elasticsearch_bench = esbench.client:main',
56 |         ]
57 |     },
58 |     classifiers = [
59 |         "Development Status :: 4 - Beta",
60 |         "Environment :: Console",
61 |         "Intended Audience :: Developers",
62 |         "License :: OSI Approved :: MIT License",
63 |         "Natural Language :: English",
64 |         "Operating System :: POSIX",
65 |         "Topic :: System :: Benchmark",
66 |         "Topic :: Internet :: WWW/HTTP :: Indexing/Search",
67 |         "Topic :: Utilities",
68 |     ],
69 |     license = 'MIT',
70 |     test_suite = "esbench.test.units.suite",
71 | )
72 | 
73 | 


--------------------------------------------------------------------------------