├── LICENSE
├── README.md
├── cactus.py
├── defaults.json
├── examples
├── cactus.pdf
├── cactus.png
├── cactus.svg
├── csv-data.csv
├── scatter.pdf
├── scatter.png
├── scatter.svg
├── solver1.json
└── solver2.json
├── load.py
├── mkplot.py
├── plot.py
├── scatter.py
└── statutil.py
/LICENSE:
--------------------------------------------------------------------------------
1 | MIT License
2 |
3 | Copyright (c) 2017 Alexey Ignatiev
4 |
5 | Permission is hereby granted, free of charge, to any person obtaining a copy
6 | of this software and associated documentation files (the "Software"), to deal
7 | in the Software without restriction, including without limitation the rights
8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9 | copies of the Software, and to permit persons to whom the Software is
10 | furnished to do so, subject to the following conditions:
11 |
12 | The above copyright notice and this permission notice shall be included in all
13 | copies or substantial portions of the Software.
14 |
15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21 | SOFTWARE.
22 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # mkplot
2 |
3 | A Python script to create cactus and scatter plots based on the [matplotlib](http://matplotlib.org/) plotting library. The script makes use of only a tiny subset of what matplotlib can do.
4 |
5 | mkplot was originally designed to make my life a lot easier by automating the plotting tasks, which a typical CS researcher has to deal with quite often. After a few years of using the script, I realized that it may be helpful for some other people (hopefully!) and so to make it publicly available. Enjoy! :)
6 |
7 | ## Getting Started
8 |
9 | First of all, make sure you have a Python interpreter installed. To run the script, you also need to install the [matplotlib](http://matplotlib.org/) library. Please, see the corresponding [installation instructions](http://matplotlib.org/users/installing.html). Once matplotlib is installed on your computer, you can start using mkplot.
10 |
11 | ## Usage
12 |
13 | The script has a number of parameters, which can be set from the command line. To see the list of options, run:
14 |
15 | ```
16 | mkplot.py -h
17 | ```
18 |
19 | ### Input Format
20 |
21 | The data to plot can be given in one of the two formats:
22 |
23 | * [JSON](https://en.wikipedia.org/wiki/JSON)
24 | * [CSV](https://en.wikipedia.org/wiki/Comma-separated_values)
25 |
26 | While the CSV format is a simple table of values aggregating all the data, the preferred format is a series of JSON files, which describe the data for an individual tool/solver following this example:
27 |
28 | ```json
29 | {
30 | "preamble": {
31 | "program": "program-name",
32 | "prog_args": "-a some -b program --arguments",
33 | "prog_alias": "some-short-alias-to-use-in-the-plot",
34 | "benchmark": "name-of-benchmark-set"
35 | },
36 | "stats": {
37 | "some_problem_instance": {
38 | "status": true,
39 | "rtime": 10.567,
40 | "mempeak": "171864 KiB",
41 | "some-other-key": "key-value1"
42 | },
43 | "another_problem_instance": {
44 | "status": false,
45 | "rtime": 1000.00,
46 | "mempeak": "245759 KiB",
47 | "some-other-key": "key-value2"
48 | },
49 | "instance3": {
50 | "status": true,
51 | "rtime": 256.32,
52 | "mempeak": "57261 KiB",
53 | "some-other-key": "key-value3"
54 | }
55 | }
56 | }
57 | ```
58 |
59 | Here, the data describes the result of running a tool referred to as *"program-name"* with the given list of command-line arguments on a benchmark set called *"name-of-benchmark-set"* containing three problem instances. The result for each instance **must** have the information on its status: *true* or *false* meaning that the instance is solved or unsolved, respectively. All the other fields are non-mandatory (you can use whatever key/value you want). However, note that the *rtime* key is used by default when working with JSON files (to change this use the `-k` option).
60 |
61 | For further details of the input format, please, see the [example files](examples).
62 |
63 | ### Using mkplot
64 |
65 | A few usage examples of mkplot follow.
66 |
67 | For instance, running
68 |
69 | ```
70 | mkplot.py -l --legend prog_alias -t 1000 -b png --save-to examples/cactus.png examples/solver?.json
71 | ```
72 |
73 | results in a simple cactus plot showing the performance of `very-nice-solver` and `another-good-tool`: 
74 |
75 | Here, mkplot is set to show program aliases in the legend instead of their complete names.
76 |
77 | If you need to create a scatter plot detailing the difference between the two solvers, just do
78 |
79 | ```
80 | mkplot.py -l -p scatter -b png --save-to examples/scatter.png --shape squared -t 1000 --ylog --ymax 10000 --ymin 0.1 --xlog examples/csv-data.csv
81 | ```
82 |
83 | The resulting scatter plot is the following:
84 |
85 | 
86 |
87 | Observe that here instead of JSON files, a CSV table is used.
88 |
89 | ## License
90 |
91 | This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
92 |
--------------------------------------------------------------------------------
/cactus.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python
2 | #-*- coding:utf-8 -*-
3 | ##
4 | ## cactus.py
5 | ##
6 | ## Created on: Jun 05, 2015
7 | ## Author: Alexey S. Ignatiev
8 | ## E-mail: aignatiev@ciencias.ulisboa.pt
9 | ##
10 |
11 | #
12 | #==============================================================================
13 | import json
14 | import matplotlib.pyplot as plt
15 | from matplotlib import __version__ as mpl_version
16 | import math
17 | import numpy as np
18 | import os
19 | from plot import Plot
20 | import six
21 |
22 |
23 | #
24 | #==============================================================================
25 | class Cactus(Plot, object):
26 | """
27 | Cactus plot class.
28 | """
29 |
30 | def __init__(self, options):
31 | """
32 | Cactus constructor.
33 | """
34 |
35 | super(Cactus, self).__init__(options)
36 |
37 | with open(self.def_path, 'r') as fp:
38 | self.linestyles = json.load(fp)['cactus_linestyle']
39 |
40 | def create(self, data):
41 | """
42 | Does the plotting.
43 | """
44 |
45 | # making lines
46 | coords = []
47 | for d in data:
48 | coords.append(np.arange(1, len(d[1]) + 1)) # xs (separate for each line)
49 | coords.append(np.array(sorted(d[1])))
50 | lines = plt.plot(*coords, zorder=3)
51 |
52 | # setting line styles
53 | if self.byname == False: # by default, assign fist line to best tool
54 | lmap = lambda i: i
55 | else: # assign line styles by tool name
56 | tnames = [(d[0], i) for i, d in enumerate(data)]
57 | tnames.sort(key=lambda pair: pair[0])
58 | tmap = {tn[1]: i for i, tn in enumerate(tnames)}
59 | lmap = lambda i: tmap[i]
60 |
61 | for i, l in enumerate(lines):
62 | plt.setp(l, **self.linestyles[lmap(i) % len(self.linestyles)])
63 |
64 | # turning the grid on
65 | if not self.no_grid:
66 | plt.grid(True, color=self.grid_color, ls=self.grid_style, lw=self.grid_width, zorder=1)
67 |
68 | # axes limits
69 | plt.xlim(self.x_min, self.x_max if self.x_max else math.ceil(max([d[2] for d in data]) / float(100)) * 100)
70 | plt.ylim(self.y_min, self.y_max if self.y_max else self.timeout)
71 |
72 | # axes labels
73 | if self.x_label:
74 | plt.xlabel(self.x_label)
75 | else:
76 | plt.xlabel('instances')
77 |
78 | if self.y_label:
79 | plt.ylabel(self.y_label)
80 | else:
81 | plt.ylabel('CPU time (s)')
82 |
83 | # choosing logarithmic scales if needed
84 | ax = plt.gca()
85 | if self.x_log:
86 | ax.set_xscale('log')
87 | if self.y_log:
88 | ax.set_yscale('log')
89 |
90 | # setting ticks
91 | # plt.xticks(np.arange(self.x_min, self.x_max + 1, 2))
92 | # if not self.y_log:
93 | # # plt.yticks(list(plt.yticks()[0]) + [self.timeout])
94 | # ax.set_yticks(range(0, 2 * (int(self.y_max) if self.y_max else int(self.timeout)), 200))
95 |
96 | # setting ticks font properties
97 | # set_*ticklables() seems to be not needed in matplotlib 1.5.0
98 | if float(mpl_version[:3]) < 1.5:
99 | ax.set_xticklabels(ax.get_xticks(), self.f_props)
100 | ax.set_yticklabels(ax.get_yticks(), self.f_props)
101 |
102 | strFormatter = plt.FormatStrFormatter('%d')
103 | logFormatter = plt.LogFormatterMathtext(base=10)
104 | ax.xaxis.set_major_formatter(strFormatter if not self.x_log else logFormatter)
105 | ax.yaxis.set_major_formatter(strFormatter if not self.y_log else logFormatter)
106 |
107 | # making the legend
108 | if self.lgd_loc != 'off':
109 | lgtext = [d[0] for d in data]
110 | lg = ax.legend(lines, lgtext, ncol=self.lgd_ncol, loc=self.lgd_loc, fancybox=self.lgd_fancy, shadow=self.lgd_shadow if self.lgd_alpha == 1.0 else False)
111 | fr = lg.get_frame()
112 | fr.set_lw(1)
113 | fr.set_alpha(self.lgd_alpha)
114 | fr.set_edgecolor('black')
115 |
116 | # setting frame thickness
117 | for i in six.itervalues(ax.spines):
118 | i.set_linewidth(1)
119 |
120 | plt.savefig(self.save_to, bbox_inches='tight', transparent=self.transparent)
121 |
--------------------------------------------------------------------------------
/defaults.json:
--------------------------------------------------------------------------------
1 | {
2 | "cactus_linestyle":
3 | [
4 | {"c": "green", "marker": "H", "ms": 5, "lw": 1, "alpha": 0.7, "mfc": "white", "mec": "green", "mew": 0.75},
5 | {"c": "red", "marker": "^", "ms": 5, "lw": 1, "alpha": 0.7, "mfc": "white", "mec": "red", "mew": 0.75},
6 | {"c": "blue", "marker": "x", "ms": 5, "lw": 1, "alpha": 0.7, "mfc": "white", "mec": "blue", "mew": 0.75},
7 | {"c": "brown", "marker": "+", "ms": 5, "lw": 1, "alpha": 0.7, "mfc": "white", "mec": "brown", "mew": 0.75},
8 | {"c": "orange", "marker": "D", "ms": 5, "lw": 1, "alpha": 0.7, "mfc": "white", "mec": "orange", "mew": 0.75},
9 | {"c": "magenta", "marker": "*", "ms": 5, "lw": 1, "alpha": 0.7, "mfc": "white", "mec": "magenta", "mew": 0.75},
10 | {"c": "cyan", "marker": "o", "ms": 5, "lw": 1, "alpha": 0.7, "mfc": "white", "mec": "cyan", "mew": 0.75},
11 | {"c": "black", "marker": "d", "ms": 5, "lw": 1, "alpha": 0.7, "mfc": "white", "mec": "black", "mew": 0.75},
12 | {"c": "#666aee", "marker": "v", "ms": 5, "lw": 1, "alpha": 0.7, "mfc": "white", "mec": "#666aee", "mew": 0.75},
13 | {"c": "grey", "marker": ">", "ms": 5, "lw": 1, "alpha": 0.7, "mfc": "white", "mec": "grey", "mew": 0.75},
14 | {"c": "green", "marker": "^", "ms": 5, "ls": "--", "lw": 1, "alpha": 0.7, "mfc": "white", "mec": "green", "mew": 0.75},
15 | {"c": "red", "marker": "H", "ms": 5, "ls": "--", "lw": 1, "alpha": 0.7, "mfc": "white", "mec": "red", "mew": 0.75},
16 | {"c": "blue", "marker": "+", "ms": 5, "ls": "--", "lw": 1, "alpha": 0.7, "mfc": "white", "mec": "blue", "mew": 0.75},
17 | {"c": "brown", "marker": "x", "ms": 5, "ls": "--", "lw": 1, "alpha": 0.7, "mfc": "white", "mec": "brown", "mew": 0.75},
18 | {"c": "orange", "marker": "*", "ms": 5, "ls": "--", "lw": 1, "alpha": 0.7, "mfc": "white", "mec": "orange", "mew": 0.75},
19 | {"c": "magenta", "marker": "D", "ms": 5, "ls": "--", "lw": 1, "alpha": 0.7, "mfc": "white", "mec": "magenta", "mew": 0.75},
20 | {"c": "cyan", "marker": "d", "ms": 5, "ls": "--", "lw": 1, "alpha": 0.7, "mfc": "white", "mec": "cyan", "mew": 0.75},
21 | {"c": "black", "marker": "o", "ms": 5, "ls": "--", "lw": 1, "alpha": 0.7, "mfc": "white", "mec": "black", "mew": 0.75},
22 | {"c": "#666aee", "marker": ">", "ms": 5, "ls": "--", "lw": 1, "alpha": 0.7, "mfc": "white", "mec": "#666aee", "mew": 0.75},
23 | {"c": "grey", "marker": "v", "ms": 5, "ls": "--", "lw": 1, "alpha": 0.7, "mfc": "white", "mec": "grey", "mew": 0.75},
24 | {"c": "green", "marker": "v", "ms": 5, "ls": ":", "lw": 1, "alpha": 0.7, "mfc": "white", "mec": "green", "mew": 0.75},
25 | {"c": "red", "marker": ">", "ms": 5, "ls": ":", "lw": 1, "alpha": 0.7, "mfc": "white", "mec": "red", "mew": 0.75},
26 | {"c": "blue", "marker": "o", "ms": 5, "ls": ":", "lw": 1, "alpha": 0.7, "mfc": "white", "mec": "blue", "mew": 0.75},
27 | {"c": "brown", "marker": "d", "ms": 5, "ls": ":", "lw": 1, "alpha": 0.7, "mfc": "white", "mec": "brown", "mew": 0.75},
28 | {"c": "orange", "marker": "D", "ms": 5, "ls": ":", "lw": 1, "alpha": 0.7, "mfc": "white", "mec": "orange", "mew": 0.75},
29 | {"c": "magenta", "marker": "*", "ms": 5, "ls": ":", "lw": 1, "alpha": 0.7, "mfc": "white", "mec": "magenta", "mew": 0.75},
30 | {"c": "cyan", "marker": "x", "ms": 5, "ls": ":", "lw": 1, "alpha": 0.7, "mfc": "white", "mec": "cyan", "mew": 0.75},
31 | {"c": "black", "marker": "+", "ms": 5, "ls": ":", "lw": 1, "alpha": 0.7, "mfc": "white", "mec": "black", "mew": 0.75},
32 | {"c": "#666aee", "marker": "H", "ms": 5, "ls": ":", "lw": 1, "alpha": 0.7, "mfc": "white", "mec": "#666aee", "mew": 0.75},
33 | {"c": "grey", "marker": "^", "ms": 5, "ls": ":", "lw": 1, "alpha": 0.7, "mfc": "white", "mec": "grey", "mew": 0.75}
34 | ],
35 | "scatter_style":
36 | {
37 | "color": "r",
38 | "edgecolor": "black",
39 | "marker": "o",
40 | "size": 25
41 | },
42 | "settings":
43 | {
44 | "alpha": 0.3,
45 | "backend": "pdf",
46 | "by_name": false,
47 | "dry_run": false,
48 | "font": "times",
49 | "font_sz": 12.0,
50 | "no_grid": false,
51 | "grid_color": "black",
52 | "grid_style": ":",
53 | "grid_width": "1",
54 | "join_key": null,
55 | "key": "rtime",
56 | "legend": "program",
57 | "lgd_alpha": 1.0,
58 | "lgd_fancy": true,
59 | "lgd_shadow": true,
60 | "lgd_loc": "upper left",
61 | "lgd_ncol": 1,
62 | "only": null,
63 | "plot_type": "cactus",
64 | "repls": null,
65 | "reverse": false,
66 | "save_to": "plot",
67 | "shape": "standard",
68 | "timeout": 3600.0,
69 | "t_label": null,
70 | "tlb_loc": "after",
71 | "transparent": false,
72 | "usetex": false,
73 | "vbs": null,
74 | "xkcd": false,
75 | "x_label": null,
76 | "x_log": false,
77 | "x_min": 0,
78 | "x_max": null,
79 | "y_label": null,
80 | "y_log": false,
81 | "y_min": 0.000000001,
82 | "y_max": null
83 | }
84 | }
85 |
--------------------------------------------------------------------------------
/examples/cactus.pdf:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/alexeyignatiev/mkplot/5ce38fcc3ece381cea6d82c273f1b55664b315c9/examples/cactus.pdf
--------------------------------------------------------------------------------
/examples/cactus.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/alexeyignatiev/mkplot/5ce38fcc3ece381cea6d82c273f1b55664b315c9/examples/cactus.png
--------------------------------------------------------------------------------
/examples/cactus.svg:
--------------------------------------------------------------------------------
1 |
2 |
4 |
5 |
1278 |
--------------------------------------------------------------------------------
/examples/csv-data.csv:
--------------------------------------------------------------------------------
1 | instance very-nice-solver another-good-tool
2 | instance1 4.6136 4.6136
3 | instance2 17.197 27.197
4 | instance3 80.8731 125.8731
5 | instance4 66.8706 58.8706
6 | instance5 199.701 287.701
7 | instance6 1.3634 1.3634
8 | instance7 102.321 101.321
9 | instance8 45.8964 71.8964
10 | instance9 58.8135 77.8135
11 | instance10 1.9281 1.9281
12 | instance11 123.5724 165.5724
13 | instance12 128.645 212.645
14 | instance13 117.9125 107.9125
15 | instance14 54.3457 78.3457
16 | instance15 133.7901 213.7901
17 | instance16 143.2799 127.2799
18 | instance17 44.1856 71.1856
19 | instance18 112.592 157.592
20 | instance19 134.6126 120.6126
21 | instance20 2.4147 2.4147
22 | instance21 164.851 252.851
23 | instance22 69.6312 68.6312
24 | instance23 74.5844 109.5844
25 | instance24 196.5466 277.5466
26 | instance25 184.3031 183.3031
27 | instance26 36.4459 51.4459
28 | instance27 5.4189 7.4189
29 | instance28 182.6557 177.6557
30 | instance29 163.3557 266.3557
31 | instance30 126.5003 191.5003
32 | instance31 119.1347 112.1347
33 | instance32 152.3262 238.3262
34 | instance33 141.9863 195.9863
35 | instance34 194.9343 179.9343
36 | instance35 38.2602 56.2602
37 | instance36 176.7886 256.7886
38 | instance37 41.7866 41.7866
39 | instance38 47.516 76.516
40 | instance39 83.8463 127.8463
41 | instance40 112.9365 101.9365
42 | instance41 19.5951 31.5951
43 | instance42 485.5096 664.5096
44 | instance43 231.7043 207.7043
45 | instance44 21.1671 31.1671
46 | instance45 321.2542 515.2542
47 | instance46 398.7485 373.7485
48 | instance47 472.182 764.182
49 | instance48 551.279 825.279
50 | instance49 191.2439 175.2439
51 | instance50 65.6477 90.6477
52 | instance51 54.5806 89.5806
53 | instance52 306.7591 296.7591
54 | instance53 274.3517 426.3517
55 | instance54 160.419 251.419
56 | instance55 387.1159 359.1159
57 | instance56 303.1698 438.1698
58 | instance57 582.7249 888.7249
59 | instance58 306.2356 299.2356
60 | instance59 406.7282 633.7282
61 | instance60 221.8447 314.8447
62 | instance61 445.1688 433.1688
63 | instance62 294.5508 446.5508
64 | instance63 544.9736 864.9736
65 | instance64 84.1849 80.1849
66 | instance65 492.8131 662.8131
67 | instance66 397.1919 605.1919
68 | instance67 487.3995 448.3995
69 | instance68 293.9778 453.9778
70 | instance69 221.7435 354.7435
71 | instance70 590.4992 525.4992
72 | instance71 552.754 783.754
73 | instance72 1000.0 1000.0
74 | instance73 74.5493 70.5493
75 | instance74 543.2072 796.2072
76 | instance75 484.566 786.566
77 | instance76 895.2961 804.2961
78 | instance77 614.2801 994.2801
79 | instance78 590.3233 852.3233
80 | instance79 261.3116 260.3116
81 | instance80 1000.0 1000.0
82 | instance81 673.9072 943.9072
83 | instance82 560.4319 509.4319
84 | instance83 894.45 1000.0
85 | instance84 237.2308 368.2308
86 | instance85 749.3951 724.3951
87 | instance86 242.3616 354.3616
88 | instance87 712.2565 1000.0
89 | instance88 1000.0 875.0
90 | instance89 90.243 137.243
91 | instance90 557.5511 789.5511
92 | instance91 432.9616 428.9616
93 | instance92 510.4372 687.4372
94 | instance93 142.9602 216.9602
95 | instance94 392.9865 365.9865
96 | instance95 428.84 652.84
97 | instance96 772.1114 1000.0
98 | instance97 346.7964 307.7964
99 | instance98 379.4744 610.4744
100 | instance99 788.865 1000.0
101 | instance100 410.851 371.851
102 |
--------------------------------------------------------------------------------
/examples/scatter.pdf:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/alexeyignatiev/mkplot/5ce38fcc3ece381cea6d82c273f1b55664b315c9/examples/scatter.pdf
--------------------------------------------------------------------------------
/examples/scatter.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/alexeyignatiev/mkplot/5ce38fcc3ece381cea6d82c273f1b55664b315c9/examples/scatter.png
--------------------------------------------------------------------------------
/examples/solver1.json:
--------------------------------------------------------------------------------
1 | {
2 | "stats": {
3 | "instance32": {
4 | "status": true,
5 | "rtime": 152.3262,
6 | "mempeak": "214016 KiB"
7 | },
8 | "instance33": {
9 | "status": true,
10 | "rtime": 141.9863,
11 | "mempeak": "389120 KiB"
12 | },
13 | "instance30": {
14 | "status": true,
15 | "rtime": 126.5003,
16 | "mempeak": "245760 KiB"
17 | },
18 | "instance31": {
19 | "status": true,
20 | "rtime": 119.1347,
21 | "mempeak": "25600 KiB"
22 | },
23 | "instance36": {
24 | "status": true,
25 | "rtime": 176.7886,
26 | "mempeak": "432128 KiB"
27 | },
28 | "instance37": {
29 | "status": true,
30 | "rtime": 41.7866,
31 | "mempeak": "162816 KiB"
32 | },
33 | "instance34": {
34 | "status": true,
35 | "rtime": 194.9343,
36 | "mempeak": "9216 KiB"
37 | },
38 | "instance35": {
39 | "status": true,
40 | "rtime": 38.2602,
41 | "mempeak": "481280 KiB"
42 | },
43 | "instance38": {
44 | "status": true,
45 | "rtime": 47.516,
46 | "mempeak": "346112 KiB"
47 | },
48 | "instance39": {
49 | "status": true,
50 | "rtime": 83.8463,
51 | "mempeak": "256000 KiB"
52 | },
53 | "instance98": {
54 | "status": true,
55 | "rtime": 379.4744,
56 | "mempeak": "499712 KiB"
57 | },
58 | "instance99": {
59 | "status": true,
60 | "rtime": 788.865,
61 | "mempeak": "420864 KiB"
62 | },
63 | "instance94": {
64 | "status": true,
65 | "rtime": 392.9865,
66 | "mempeak": "452608 KiB"
67 | },
68 | "instance95": {
69 | "status": true,
70 | "rtime": 428.84,
71 | "mempeak": "314368 KiB"
72 | },
73 | "instance96": {
74 | "status": true,
75 | "rtime": 772.1114,
76 | "mempeak": "4096 KiB"
77 | },
78 | "instance97": {
79 | "status": true,
80 | "rtime": 346.7964,
81 | "mempeak": "281600 KiB"
82 | },
83 | "instance90": {
84 | "status": true,
85 | "rtime": 557.5511,
86 | "mempeak": "431104 KiB"
87 | },
88 | "instance91": {
89 | "status": true,
90 | "rtime": 432.9616,
91 | "mempeak": "450560 KiB"
92 | },
93 | "instance92": {
94 | "status": true,
95 | "rtime": 510.4372,
96 | "mempeak": "369664 KiB"
97 | },
98 | "instance93": {
99 | "status": true,
100 | "rtime": 142.9602,
101 | "mempeak": "389120 KiB"
102 | },
103 | "instance21": {
104 | "status": true,
105 | "rtime": 164.851,
106 | "mempeak": "380928 KiB"
107 | },
108 | "instance20": {
109 | "status": true,
110 | "rtime": 2.4147,
111 | "mempeak": "260096 KiB"
112 | },
113 | "instance23": {
114 | "status": true,
115 | "rtime": 74.5844,
116 | "mempeak": "145408 KiB"
117 | },
118 | "instance22": {
119 | "status": true,
120 | "rtime": 69.6312,
121 | "mempeak": "330752 KiB"
122 | },
123 | "instance25": {
124 | "status": true,
125 | "rtime": 184.3031,
126 | "mempeak": "502784 KiB"
127 | },
128 | "instance24": {
129 | "status": true,
130 | "rtime": 196.5466,
131 | "mempeak": "360448 KiB"
132 | },
133 | "instance27": {
134 | "status": true,
135 | "rtime": 5.4189,
136 | "mempeak": "65536 KiB"
137 | },
138 | "instance26": {
139 | "status": true,
140 | "rtime": 36.4459,
141 | "mempeak": "457728 KiB"
142 | },
143 | "instance29": {
144 | "status": true,
145 | "rtime": 163.3557,
146 | "mempeak": "263168 KiB"
147 | },
148 | "instance28": {
149 | "status": true,
150 | "rtime": 182.6557,
151 | "mempeak": "115712 KiB"
152 | },
153 | "instance83": {
154 | "status": true,
155 | "rtime": 894.45,
156 | "mempeak": "11264 KiB"
157 | },
158 | "instance82": {
159 | "status": true,
160 | "rtime": 560.4319,
161 | "mempeak": "289792 KiB"
162 | },
163 | "instance81": {
164 | "status": true,
165 | "rtime": 673.9072,
166 | "mempeak": "518144 KiB"
167 | },
168 | "instance80": {
169 | "status": false,
170 | "rtime": 1000.0,
171 | "mempeak": "330752 KiB"
172 | },
173 | "instance87": {
174 | "status": true,
175 | "rtime": 712.2565,
176 | "mempeak": "271360 KiB"
177 | },
178 | "instance86": {
179 | "status": true,
180 | "rtime": 242.3616,
181 | "mempeak": "13312 KiB"
182 | },
183 | "instance85": {
184 | "status": true,
185 | "rtime": 749.3951,
186 | "mempeak": "150528 KiB"
187 | },
188 | "instance84": {
189 | "status": true,
190 | "rtime": 237.2308,
191 | "mempeak": "111616 KiB"
192 | },
193 | "instance89": {
194 | "status": true,
195 | "rtime": 90.243,
196 | "mempeak": "431104 KiB"
197 | },
198 | "instance88": {
199 | "status": false,
200 | "rtime": 1000.0,
201 | "mempeak": "53248 KiB"
202 | },
203 | "instance14": {
204 | "status": true,
205 | "rtime": 54.3457,
206 | "mempeak": "199680 KiB"
207 | },
208 | "instance15": {
209 | "status": true,
210 | "rtime": 133.7901,
211 | "mempeak": "198656 KiB"
212 | },
213 | "instance16": {
214 | "status": true,
215 | "rtime": 143.2799,
216 | "mempeak": "76800 KiB"
217 | },
218 | "instance17": {
219 | "status": true,
220 | "rtime": 44.1856,
221 | "mempeak": "261120 KiB"
222 | },
223 | "instance10": {
224 | "status": true,
225 | "rtime": 1.9281,
226 | "mempeak": "330752 KiB"
227 | },
228 | "instance11": {
229 | "status": true,
230 | "rtime": 123.5724,
231 | "mempeak": "362496 KiB"
232 | },
233 | "instance12": {
234 | "status": true,
235 | "rtime": 128.645,
236 | "mempeak": "22528 KiB"
237 | },
238 | "instance13": {
239 | "status": true,
240 | "rtime": 117.9125,
241 | "mempeak": "367616 KiB"
242 | },
243 | "instance18": {
244 | "status": true,
245 | "rtime": 112.592,
246 | "mempeak": "429056 KiB"
247 | },
248 | "instance19": {
249 | "status": true,
250 | "rtime": 134.6126,
251 | "mempeak": "271360 KiB"
252 | },
253 | "instance78": {
254 | "status": true,
255 | "rtime": 590.3233,
256 | "mempeak": "280576 KiB"
257 | },
258 | "instance79": {
259 | "status": true,
260 | "rtime": 261.3116,
261 | "mempeak": "498688 KiB"
262 | },
263 | "instance76": {
264 | "status": true,
265 | "rtime": 895.2961,
266 | "mempeak": "245760 KiB"
267 | },
268 | "instance77": {
269 | "status": true,
270 | "rtime": 614.2801,
271 | "mempeak": "165888 KiB"
272 | },
273 | "instance74": {
274 | "status": true,
275 | "rtime": 543.2072,
276 | "mempeak": "373760 KiB"
277 | },
278 | "instance75": {
279 | "status": true,
280 | "rtime": 484.566,
281 | "mempeak": "218112 KiB"
282 | },
283 | "instance72": {
284 | "status": false,
285 | "rtime": 1000.0,
286 | "mempeak": "475136 KiB"
287 | },
288 | "instance73": {
289 | "status": true,
290 | "rtime": 74.5493,
291 | "mempeak": "231424 KiB"
292 | },
293 | "instance70": {
294 | "status": true,
295 | "rtime": 590.4992,
296 | "mempeak": "173056 KiB"
297 | },
298 | "instance71": {
299 | "status": true,
300 | "rtime": 552.754,
301 | "mempeak": "12288 KiB"
302 | },
303 | "instance2": {
304 | "status": true,
305 | "rtime": 17.197,
306 | "mempeak": "230400 KiB"
307 | },
308 | "instance3": {
309 | "status": true,
310 | "rtime": 80.8731,
311 | "mempeak": "241664 KiB"
312 | },
313 | "instance1": {
314 | "status": true,
315 | "rtime": 4.6136,
316 | "mempeak": "358400 KiB"
317 | },
318 | "instance6": {
319 | "status": true,
320 | "rtime": 1.3634,
321 | "mempeak": "175104 KiB"
322 | },
323 | "instance7": {
324 | "status": true,
325 | "rtime": 102.321,
326 | "mempeak": "478208 KiB"
327 | },
328 | "instance4": {
329 | "status": true,
330 | "rtime": 66.8706,
331 | "mempeak": "411648 KiB"
332 | },
333 | "instance5": {
334 | "status": true,
335 | "rtime": 199.701,
336 | "mempeak": "442368 KiB"
337 | },
338 | "instance8": {
339 | "status": true,
340 | "rtime": 45.8964,
341 | "mempeak": "430080 KiB"
342 | },
343 | "instance9": {
344 | "status": true,
345 | "rtime": 58.8135,
346 | "mempeak": "260096 KiB"
347 | },
348 | "instance69": {
349 | "status": true,
350 | "rtime": 221.7435,
351 | "mempeak": "142336 KiB"
352 | },
353 | "instance68": {
354 | "status": true,
355 | "rtime": 293.9778,
356 | "mempeak": "141312 KiB"
357 | },
358 | "instance65": {
359 | "status": true,
360 | "rtime": 492.8131,
361 | "mempeak": "65536 KiB"
362 | },
363 | "instance64": {
364 | "status": true,
365 | "rtime": 84.1849,
366 | "mempeak": "124928 KiB"
367 | },
368 | "instance67": {
369 | "status": true,
370 | "rtime": 487.3995,
371 | "mempeak": "419840 KiB"
372 | },
373 | "instance66": {
374 | "status": true,
375 | "rtime": 397.1919,
376 | "mempeak": "62464 KiB"
377 | },
378 | "instance61": {
379 | "status": true,
380 | "rtime": 445.1688,
381 | "mempeak": "134144 KiB"
382 | },
383 | "instance60": {
384 | "status": true,
385 | "rtime": 221.8447,
386 | "mempeak": "507904 KiB"
387 | },
388 | "instance63": {
389 | "status": true,
390 | "rtime": 544.9736,
391 | "mempeak": "290816 KiB"
392 | },
393 | "instance62": {
394 | "status": true,
395 | "rtime": 294.5508,
396 | "mempeak": "248832 KiB"
397 | },
398 | "instance100": {
399 | "status": true,
400 | "rtime": 410.851,
401 | "mempeak": "322560 KiB"
402 | },
403 | "instance50": {
404 | "status": true,
405 | "rtime": 65.6477,
406 | "mempeak": "232448 KiB"
407 | },
408 | "instance51": {
409 | "status": true,
410 | "rtime": 54.5806,
411 | "mempeak": "317440 KiB"
412 | },
413 | "instance52": {
414 | "status": true,
415 | "rtime": 306.7591,
416 | "mempeak": "5120 KiB"
417 | },
418 | "instance53": {
419 | "status": true,
420 | "rtime": 274.3517,
421 | "mempeak": "429056 KiB"
422 | },
423 | "instance54": {
424 | "status": true,
425 | "rtime": 160.419,
426 | "mempeak": "181248 KiB"
427 | },
428 | "instance55": {
429 | "status": true,
430 | "rtime": 387.1159,
431 | "mempeak": "233472 KiB"
432 | },
433 | "instance56": {
434 | "status": true,
435 | "rtime": 303.1698,
436 | "mempeak": "487424 KiB"
437 | },
438 | "instance57": {
439 | "status": true,
440 | "rtime": 582.7249,
441 | "mempeak": "411648 KiB"
442 | },
443 | "instance58": {
444 | "status": true,
445 | "rtime": 306.2356,
446 | "mempeak": "489472 KiB"
447 | },
448 | "instance59": {
449 | "status": true,
450 | "rtime": 406.7282,
451 | "mempeak": "112640 KiB"
452 | },
453 | "instance47": {
454 | "status": true,
455 | "rtime": 472.182,
456 | "mempeak": "442368 KiB"
457 | },
458 | "instance46": {
459 | "status": true,
460 | "rtime": 398.7485,
461 | "mempeak": "340992 KiB"
462 | },
463 | "instance45": {
464 | "status": true,
465 | "rtime": 321.2542,
466 | "mempeak": "359424 KiB"
467 | },
468 | "instance44": {
469 | "status": true,
470 | "rtime": 21.1671,
471 | "mempeak": "505856 KiB"
472 | },
473 | "instance43": {
474 | "status": true,
475 | "rtime": 231.7043,
476 | "mempeak": "489472 KiB"
477 | },
478 | "instance42": {
479 | "status": true,
480 | "rtime": 485.5096,
481 | "mempeak": "488448 KiB"
482 | },
483 | "instance41": {
484 | "status": true,
485 | "rtime": 19.5951,
486 | "mempeak": "308224 KiB"
487 | },
488 | "instance40": {
489 | "status": true,
490 | "rtime": 112.9365,
491 | "mempeak": "45056 KiB"
492 | },
493 | "instance49": {
494 | "status": true,
495 | "rtime": 191.2439,
496 | "mempeak": "473088 KiB"
497 | },
498 | "instance48": {
499 | "status": true,
500 | "rtime": 551.279,
501 | "mempeak": "262144 KiB"
502 | }
503 | },
504 | "preamble": {
505 | "benchmark": "my-benchmark-set",
506 | "prog_args": "-a hello --arg2 world -v",
507 | "program": "very_nice_solver",
508 | "prog_alias": "vns"
509 | }
510 | }
--------------------------------------------------------------------------------
/examples/solver2.json:
--------------------------------------------------------------------------------
1 | {
2 | "stats": {
3 | "instance32": {
4 | "status": true,
5 | "rtime": 238.3262,
6 | "mempeak": "288768 KiB"
7 | },
8 | "instance33": {
9 | "status": true,
10 | "rtime": 195.9863,
11 | "mempeak": "492544 KiB"
12 | },
13 | "instance30": {
14 | "status": true,
15 | "rtime": 191.50029999999998,
16 | "mempeak": "43008 KiB"
17 | },
18 | "instance31": {
19 | "status": true,
20 | "rtime": 112.1347,
21 | "mempeak": "387072 KiB"
22 | },
23 | "instance36": {
24 | "status": true,
25 | "rtime": 256.7886,
26 | "mempeak": "409600 KiB"
27 | },
28 | "instance37": {
29 | "status": true,
30 | "rtime": 41.7866,
31 | "mempeak": "423936 KiB"
32 | },
33 | "instance34": {
34 | "status": true,
35 | "rtime": 179.9343,
36 | "mempeak": "244736 KiB"
37 | },
38 | "instance35": {
39 | "status": true,
40 | "rtime": 56.2602,
41 | "mempeak": "256000 KiB"
42 | },
43 | "instance38": {
44 | "status": true,
45 | "rtime": 76.51599999999999,
46 | "mempeak": "453632 KiB"
47 | },
48 | "instance39": {
49 | "status": true,
50 | "rtime": 127.8463,
51 | "mempeak": "273408 KiB"
52 | },
53 | "instance98": {
54 | "status": true,
55 | "rtime": 610.4744000000001,
56 | "mempeak": "281600 KiB"
57 | },
58 | "instance99": {
59 | "status": false,
60 | "rtime": 1000.0,
61 | "mempeak": "458752 KiB"
62 | },
63 | "instance94": {
64 | "status": true,
65 | "rtime": 365.9865,
66 | "mempeak": "299008 KiB"
67 | },
68 | "instance95": {
69 | "status": true,
70 | "rtime": 652.8399999999999,
71 | "mempeak": "475136 KiB"
72 | },
73 | "instance96": {
74 | "status": false,
75 | "rtime": 1000.0,
76 | "mempeak": "307200 KiB"
77 | },
78 | "instance97": {
79 | "status": true,
80 | "rtime": 307.7964,
81 | "mempeak": "223232 KiB"
82 | },
83 | "instance90": {
84 | "status": true,
85 | "rtime": 789.5511,
86 | "mempeak": "39936 KiB"
87 | },
88 | "instance91": {
89 | "status": true,
90 | "rtime": 428.9616,
91 | "mempeak": "491520 KiB"
92 | },
93 | "instance92": {
94 | "status": true,
95 | "rtime": 687.4372000000001,
96 | "mempeak": "83968 KiB"
97 | },
98 | "instance93": {
99 | "status": true,
100 | "rtime": 216.9602,
101 | "mempeak": "205824 KiB"
102 | },
103 | "instance21": {
104 | "status": true,
105 | "rtime": 252.851,
106 | "mempeak": "350208 KiB"
107 | },
108 | "instance20": {
109 | "status": true,
110 | "rtime": 2.4147,
111 | "mempeak": "80896 KiB"
112 | },
113 | "instance23": {
114 | "status": true,
115 | "rtime": 109.5844,
116 | "mempeak": "254976 KiB"
117 | },
118 | "instance22": {
119 | "status": true,
120 | "rtime": 68.6312,
121 | "mempeak": "250880 KiB"
122 | },
123 | "instance25": {
124 | "status": true,
125 | "rtime": 183.3031,
126 | "mempeak": "497664 KiB"
127 | },
128 | "instance24": {
129 | "status": true,
130 | "rtime": 277.5466,
131 | "mempeak": "34816 KiB"
132 | },
133 | "instance27": {
134 | "status": true,
135 | "rtime": 7.4189,
136 | "mempeak": "7168 KiB"
137 | },
138 | "instance26": {
139 | "status": true,
140 | "rtime": 51.4459,
141 | "mempeak": "277504 KiB"
142 | },
143 | "instance29": {
144 | "status": true,
145 | "rtime": 266.3557,
146 | "mempeak": "510976 KiB"
147 | },
148 | "instance28": {
149 | "status": true,
150 | "rtime": 177.6557,
151 | "mempeak": "279552 KiB"
152 | },
153 | "instance83": {
154 | "status": false,
155 | "rtime": 1000.0,
156 | "mempeak": "191488 KiB"
157 | },
158 | "instance82": {
159 | "status": true,
160 | "rtime": 509.43190000000004,
161 | "mempeak": "304128 KiB"
162 | },
163 | "instance81": {
164 | "status": true,
165 | "rtime": 943.9072,
166 | "mempeak": "503808 KiB"
167 | },
168 | "instance80": {
169 | "status": false,
170 | "rtime": 1000.0,
171 | "mempeak": "302080 KiB"
172 | },
173 | "instance87": {
174 | "status": false,
175 | "rtime": 1000.0,
176 | "mempeak": "197632 KiB"
177 | },
178 | "instance86": {
179 | "status": true,
180 | "rtime": 354.3616,
181 | "mempeak": "177152 KiB"
182 | },
183 | "instance85": {
184 | "status": true,
185 | "rtime": 724.3951,
186 | "mempeak": "208896 KiB"
187 | },
188 | "instance84": {
189 | "status": true,
190 | "rtime": 368.2308,
191 | "mempeak": "476160 KiB"
192 | },
193 | "instance89": {
194 | "status": true,
195 | "rtime": 137.243,
196 | "mempeak": "516096 KiB"
197 | },
198 | "instance88": {
199 | "status": true,
200 | "rtime": 875.0,
201 | "mempeak": "354304 KiB"
202 | },
203 | "instance14": {
204 | "status": true,
205 | "rtime": 78.3457,
206 | "mempeak": "101376 KiB"
207 | },
208 | "instance15": {
209 | "status": true,
210 | "rtime": 213.7901,
211 | "mempeak": "378880 KiB"
212 | },
213 | "instance16": {
214 | "status": true,
215 | "rtime": 127.2799,
216 | "mempeak": "176128 KiB"
217 | },
218 | "instance17": {
219 | "status": true,
220 | "rtime": 71.1856,
221 | "mempeak": "74752 KiB"
222 | },
223 | "instance10": {
224 | "status": true,
225 | "rtime": 1.9281,
226 | "mempeak": "471040 KiB"
227 | },
228 | "instance11": {
229 | "status": true,
230 | "rtime": 165.57240000000002,
231 | "mempeak": "204800 KiB"
232 | },
233 | "instance12": {
234 | "status": true,
235 | "rtime": 212.645,
236 | "mempeak": "27648 KiB"
237 | },
238 | "instance13": {
239 | "status": true,
240 | "rtime": 107.9125,
241 | "mempeak": "159744 KiB"
242 | },
243 | "instance18": {
244 | "status": true,
245 | "rtime": 157.59199999999998,
246 | "mempeak": "290816 KiB"
247 | },
248 | "instance19": {
249 | "status": true,
250 | "rtime": 120.61259999999999,
251 | "mempeak": "488448 KiB"
252 | },
253 | "instance78": {
254 | "status": true,
255 | "rtime": 852.3233,
256 | "mempeak": "126976 KiB"
257 | },
258 | "instance79": {
259 | "status": true,
260 | "rtime": 260.3116,
261 | "mempeak": "328704 KiB"
262 | },
263 | "instance76": {
264 | "status": true,
265 | "rtime": 804.2961,
266 | "mempeak": "16384 KiB"
267 | },
268 | "instance77": {
269 | "status": true,
270 | "rtime": 994.2801,
271 | "mempeak": "282624 KiB"
272 | },
273 | "instance74": {
274 | "status": true,
275 | "rtime": 796.2072,
276 | "mempeak": "326656 KiB"
277 | },
278 | "instance75": {
279 | "status": true,
280 | "rtime": 786.566,
281 | "mempeak": "407552 KiB"
282 | },
283 | "instance72": {
284 | "status": false,
285 | "rtime": 1000.0,
286 | "mempeak": "254976 KiB"
287 | },
288 | "instance73": {
289 | "status": true,
290 | "rtime": 70.5493,
291 | "mempeak": "207872 KiB"
292 | },
293 | "instance70": {
294 | "status": true,
295 | "rtime": 525.4992,
296 | "mempeak": "471040 KiB"
297 | },
298 | "instance71": {
299 | "status": true,
300 | "rtime": 783.754,
301 | "mempeak": "41984 KiB"
302 | },
303 | "instance2": {
304 | "status": true,
305 | "rtime": 27.197,
306 | "mempeak": "323584 KiB"
307 | },
308 | "instance3": {
309 | "status": true,
310 | "rtime": 125.8731,
311 | "mempeak": "344064 KiB"
312 | },
313 | "instance1": {
314 | "status": true,
315 | "rtime": 4.6136,
316 | "mempeak": "77824 KiB"
317 | },
318 | "instance6": {
319 | "status": true,
320 | "rtime": 1.3634,
321 | "mempeak": "202752 KiB"
322 | },
323 | "instance7": {
324 | "status": true,
325 | "rtime": 101.321,
326 | "mempeak": "367616 KiB"
327 | },
328 | "instance4": {
329 | "status": true,
330 | "rtime": 58.870599999999996,
331 | "mempeak": "343040 KiB"
332 | },
333 | "instance5": {
334 | "status": true,
335 | "rtime": 287.701,
336 | "mempeak": "119808 KiB"
337 | },
338 | "instance8": {
339 | "status": true,
340 | "rtime": 71.8964,
341 | "mempeak": "11264 KiB"
342 | },
343 | "instance9": {
344 | "status": true,
345 | "rtime": 77.8135,
346 | "mempeak": "484352 KiB"
347 | },
348 | "instance69": {
349 | "status": true,
350 | "rtime": 354.74350000000004,
351 | "mempeak": "211968 KiB"
352 | },
353 | "instance68": {
354 | "status": true,
355 | "rtime": 453.9778,
356 | "mempeak": "180224 KiB"
357 | },
358 | "instance65": {
359 | "status": true,
360 | "rtime": 662.8131000000001,
361 | "mempeak": "178176 KiB"
362 | },
363 | "instance64": {
364 | "status": true,
365 | "rtime": 80.1849,
366 | "mempeak": "378880 KiB"
367 | },
368 | "instance67": {
369 | "status": true,
370 | "rtime": 448.3995,
371 | "mempeak": "445440 KiB"
372 | },
373 | "instance66": {
374 | "status": true,
375 | "rtime": 605.1919,
376 | "mempeak": "276480 KiB"
377 | },
378 | "instance61": {
379 | "status": true,
380 | "rtime": 433.1688,
381 | "mempeak": "68608 KiB"
382 | },
383 | "instance60": {
384 | "status": true,
385 | "rtime": 314.8447,
386 | "mempeak": "120832 KiB"
387 | },
388 | "instance63": {
389 | "status": true,
390 | "rtime": 864.9736,
391 | "mempeak": "440320 KiB"
392 | },
393 | "instance62": {
394 | "status": true,
395 | "rtime": 446.5508,
396 | "mempeak": "72704 KiB"
397 | },
398 | "instance100": {
399 | "status": true,
400 | "rtime": 371.851,
401 | "mempeak": "145408 KiB"
402 | },
403 | "instance50": {
404 | "status": true,
405 | "rtime": 90.6477,
406 | "mempeak": "264192 KiB"
407 | },
408 | "instance51": {
409 | "status": true,
410 | "rtime": 89.5806,
411 | "mempeak": "392192 KiB"
412 | },
413 | "instance52": {
414 | "status": true,
415 | "rtime": 296.7591,
416 | "mempeak": "106496 KiB"
417 | },
418 | "instance53": {
419 | "status": true,
420 | "rtime": 426.3517,
421 | "mempeak": "129024 KiB"
422 | },
423 | "instance54": {
424 | "status": true,
425 | "rtime": 251.419,
426 | "mempeak": "1024 KiB"
427 | },
428 | "instance55": {
429 | "status": true,
430 | "rtime": 359.1159,
431 | "mempeak": "365568 KiB"
432 | },
433 | "instance56": {
434 | "status": true,
435 | "rtime": 438.1698,
436 | "mempeak": "338944 KiB"
437 | },
438 | "instance57": {
439 | "status": true,
440 | "rtime": 888.7249,
441 | "mempeak": "281600 KiB"
442 | },
443 | "instance58": {
444 | "status": true,
445 | "rtime": 299.2356,
446 | "mempeak": "193536 KiB"
447 | },
448 | "instance59": {
449 | "status": true,
450 | "rtime": 633.7282,
451 | "mempeak": "197632 KiB"
452 | },
453 | "instance47": {
454 | "status": true,
455 | "rtime": 764.182,
456 | "mempeak": "166912 KiB"
457 | },
458 | "instance46": {
459 | "status": true,
460 | "rtime": 373.7485,
461 | "mempeak": "208896 KiB"
462 | },
463 | "instance45": {
464 | "status": true,
465 | "rtime": 515.2542000000001,
466 | "mempeak": "330752 KiB"
467 | },
468 | "instance44": {
469 | "status": true,
470 | "rtime": 31.1671,
471 | "mempeak": "72704 KiB"
472 | },
473 | "instance43": {
474 | "status": true,
475 | "rtime": 207.7043,
476 | "mempeak": "190464 KiB"
477 | },
478 | "instance42": {
479 | "status": true,
480 | "rtime": 664.5096,
481 | "mempeak": "343040 KiB"
482 | },
483 | "instance41": {
484 | "status": true,
485 | "rtime": 31.5951,
486 | "mempeak": "256000 KiB"
487 | },
488 | "instance40": {
489 | "status": true,
490 | "rtime": 101.9365,
491 | "mempeak": "517120 KiB"
492 | },
493 | "instance49": {
494 | "status": true,
495 | "rtime": 175.2439,
496 | "mempeak": "44032 KiB"
497 | },
498 | "instance48": {
499 | "status": true,
500 | "rtime": 825.279,
501 | "mempeak": "415744 KiB"
502 | }
503 | },
504 | "preamble": {
505 | "benchmark": "my-benchmark-set",
506 | "prog_args": null,
507 | "program": "another-good-tool",
508 | "prog_alias": "agt"
509 | }
510 | }
--------------------------------------------------------------------------------
/load.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python
2 | #-*- coding:utf-8 -*-
3 | ##
4 | ## load.py
5 | ##
6 | ## Created on: Jun 05, 2015
7 | ## Author: Alexey S. Ignatiev
8 | ## E-mail: aignatiev@ciencias.ulisboa.pt
9 | ##
10 |
11 | #
12 | #==============================================================================
13 | import csv
14 | import json
15 | import statutil
16 | import six
17 | import sys
18 |
19 |
20 | #
21 | #==============================================================================
22 | def load_data(files, options):
23 | """
24 | Loads data from the input files.
25 | """
26 |
27 | try: # if JSON data
28 | return load_json(statutil.StatArray(files), options)
29 | except statutil.JSONException as e:
30 | sys.stderr.write('\033[33;1mWarning:\033[m ' + str(e) + '\033[m\n')
31 | sys.stderr.write('Probably not a JSON format. Trying to read as CSV.\n')
32 |
33 | # reading CSV
34 | # expecting exactly one input file
35 | with open(files[0], 'r') as fp:
36 | # try:
37 | rows = csv.reader(fp, delimiter=' ', quotechar='|')
38 | rows = [row for row in rows]
39 |
40 | stats = []
41 | names = [n.strip() for n in rows[0][1:] if n.strip()]
42 | for row in rows[1:]:
43 | stats.append([val.strip() for val in row[1:] if val.strip()])
44 |
45 | return load_csv(names, stats, options)
46 | # except:
47 | # sys.stderr.write('\033[31;1mError:\033[m Unable to read input file(s).\n')
48 |
49 |
50 | #
51 | #==============================================================================
52 | def load_json(stat_arr, options):
53 | """
54 | Loads runtime data from STAT objects.
55 | """
56 |
57 | # preparing data
58 | if options['join_key']:
59 | stat_arr.cluster(use_key=options['join_key'])
60 |
61 | data = []
62 |
63 | # choosing the minimal value
64 | min_val = 0.000000001
65 | if options['plot_type'] == 'scatter':
66 | if options['x_min']:
67 | min_val = max(options['x_min'], options['y_min'])
68 | else:
69 | min_val = options['y_min'] # options['y_min'] is always defined
70 |
71 | # processing (normal) separate data
72 | for stat_obj in stat_arr:
73 | vals = []
74 | num_solved = 0
75 |
76 | last_val = -1
77 | for inst in stat_obj.insts_own: # insts_own are sorted
78 | if options['key'] in stat_obj.data[inst]:
79 | val = stat_obj.data[inst][options['key']]
80 | else:
81 | val = float(options['timeout'])
82 | if stat_obj.data[inst]['status'] == True:
83 | if val > last_val:
84 | last_val = val
85 |
86 | if val >= float(options['timeout']):
87 | val = float(options['timeout'])
88 | elif val <= min_val:
89 | val = min_val
90 |
91 | num_solved += 1
92 | else:
93 | val = float(options['timeout'])
94 | if options['plot_type'] == 'cactus':
95 | val *= 10
96 |
97 | vals.append(val)
98 |
99 | if type(options['legend']) is list:
100 | label = ' '.join([stat_obj.preamble[k] for k in options['legend']])
101 | else:
102 | label = stat_obj.preamble[options['legend']]
103 |
104 | label = label.strip()
105 | data.append((label, vals, num_solved, last_val))
106 |
107 | # processing VBSes
108 | if options['vbs']:
109 | for vbs_name, tools in options['vbs'].items():
110 | max_value = float(options['timeout']) if options['plot_type'] == 'scatter' else 10 * float(options['timeout'])
111 | vals = { i: max_value for i in stat_arr.inst_full}
112 | num_solved = 0
113 |
114 | if tools != 'all':
115 | for stat_obj in stat_arr:
116 | if type(options['legend']) is list:
117 | p = ' '.join([stat_obj.preamble[k] for k in options['legend']])
118 | else:
119 | p = stat_obj.preamble[options['legend']]
120 |
121 | p = p.strip()
122 |
123 | if p in tools:
124 | for inst, d in six.iteritems(stat_obj.data):
125 | if d['status'] == True:
126 | if d[options['key']] >= float(options['timeout']):
127 | d[options['key']] = max_value
128 | elif vals[inst] >= max_value:
129 | num_solved += 1
130 |
131 | vals[inst] = max([min_val, min([d[options['key']], vals[inst]])])
132 | else: # VBS among all the tools
133 | for stat_obj in stat_arr:
134 | for inst, d in six.iteritems(stat_obj.data):
135 | if d['status'] == True:
136 | if d[options['key']] >= float(options['timeout']):
137 | d[options['key']] = max_value
138 | elif vals[inst] >= max_value:
139 | num_solved += 1
140 |
141 | vals[inst] = max([min_val, min([d[options['key']], vals[inst]])])
142 |
143 | last_val = -1
144 | for v in six.itervalues(vals):
145 | if v > last_val and v < max_value:
146 | last_val = v
147 |
148 | data.append((vbs_name, [vals[i] for i in stat_arr.inst_full], num_solved, last_val))
149 |
150 | if options['only']:
151 | data = [d for d in data if d[0] in options['only']]
152 |
153 | if options['repls']:
154 | data = [(options['repls'][n], v, s, l) if n in options['repls'] else (n, v, s, l) for n, v, s, l in data]
155 |
156 | return sorted(data, key=lambda x: x[2] + len(x[1]) / sum(x[1]), reverse=not options['reverse'])
157 |
158 |
159 | #
160 | #==============================================================================
161 | def load_csv(names, stats, options):
162 | """
163 | Loads runtime CSV data.
164 | """
165 |
166 | # choosing the minimal value
167 | min_val = 0.000000001
168 | if options['plot_type'] == 'scatter':
169 | if options['x_min']:
170 | min_val = max(options['x_min'], options['y_min'])
171 | else:
172 | min_val = options['y_min'] # options['y_min'] is always defined
173 |
174 | names_orig = names[:]
175 |
176 | if options['repls']:
177 | names = [options['repls'][n] if n in options['repls'] else n for n in names]
178 |
179 | # processing (normal) separate data
180 | lens = [0 for n in names]
181 | vals_all = [[] for n in names]
182 | last_vals = [-1 for n in names]
183 |
184 | for vlist in stats:
185 | vlist = [float(val) for val in vlist]
186 |
187 | for i, val in enumerate(vlist):
188 | if val < float(options['timeout']):
189 | if val > last_vals[i]:
190 | last_vals[i] = val
191 |
192 | if val < min_val:
193 | val = min_val
194 |
195 | lens[i] += 1
196 | else:
197 | val = float(options['timeout'])
198 | if options['plot_type'] == 'cactus':
199 | val *= 10
200 |
201 | vals_all[i].append(val)
202 |
203 | # processing VBSes
204 | if options['vbs']:
205 | for vbs_name, tools in options['vbs'].items():
206 | vals = []
207 | len_ = 0
208 | last_val = -1
209 |
210 | if tools != 'all':
211 | tools = [n if n in tools else '' for n in names_orig]
212 |
213 | for vlist in stats:
214 | vlist = [float(val) for i, val in enumerate(vlist) if tools[i]]
215 | val = min(vlist)
216 |
217 | if val < float(options['timeout']):
218 | if val > last_val:
219 | last_val = val
220 |
221 | if val < min_val:
222 | val = min_val
223 |
224 | len_ += 1
225 | else:
226 | val = options['timeout']
227 | if options['plot_type'] == 'cactus':
228 | val *= 10
229 |
230 | vals.append(val)
231 | else: # VBS among all the tools
232 | for vlist in stats:
233 | val = min([float(val) for val in vlist])
234 |
235 | if val < float(options['timeout']):
236 | if val > last_val:
237 | last_val = val
238 |
239 | if val < min_val:
240 | val = min_val
241 |
242 | len_ += 1
243 | else:
244 | val = options['timeout']
245 | if options['plot_type'] == 'cactus':
246 | val *= 10
247 |
248 | vals.append(val)
249 |
250 | names.append(vbs_name)
251 | names_orig.append(vbs_name)
252 | vals_all.append(vals)
253 | lens.append(len_)
254 | last_vals.append(last_val)
255 |
256 | data = [[n, t, s, l] for n, t, s, l in zip(names, vals_all, lens, last_vals)]
257 |
258 | if options['only']:
259 | data = [d for i, d in enumerate(data) if names_orig[i] in options['only']]
260 |
261 | return sorted(data, key=lambda x: x[2] + len(x[1]) / sum(x[1]), reverse=not options['reverse'])
262 |
--------------------------------------------------------------------------------
/mkplot.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python
2 | #-*- coding:utf-8 -*-
3 | ##
4 | ## mkplot.py
5 | ##
6 | ## Created on: Jan 31, 2014
7 | ## Author: Alexey S. Ignatiev
8 | ## E-mail: aignatiev@ciencias.ulisboa.pt
9 | ##
10 |
11 | #
12 | #==============================================================================
13 | from __future__ import print_function
14 | import matplotlib
15 | matplotlib.use('pdf') # for not loading GUI modules
16 |
17 | from cactus import Cactus
18 | import getopt
19 | import json
20 | from load import load_data
21 | import os
22 | from scatter import Scatter
23 | import sys
24 |
25 |
26 | #
27 | #==============================================================================
28 | def parse_options():
29 | """
30 | Parses command-line options:
31 | """
32 |
33 | try:
34 | opts, args = getopt.getopt(sys.argv[1:],
35 | 'a:b:c:df:hj:k:lnp:r:t:',
36 | ['alpha=',
37 | 'backend=',
38 | 'by-name',
39 | 'config=',
40 | 'dry-run',
41 | 'font=',
42 | 'font-sz=',
43 | 'no-grid',
44 | 'help',
45 | 'join-key=',
46 | 'key=',
47 | 'latex',
48 | 'lalpha=',
49 | 'legend=',
50 | 'lloc=',
51 | 'lncol=',
52 | 'only=',
53 | 'plot-type=',
54 | 'replace=',
55 | 'reverse',
56 | 'save-to=',
57 | 'shape=',
58 | 'timeout=',
59 | 'tlabel=',
60 | 'tol-loc=',
61 | 'transparent',
62 | 'vbs=',
63 | 'xkcd',
64 | 'xlabel=',
65 | 'xlog',
66 | 'xmin=',
67 | 'xmax=',
68 | 'ylabel=',
69 | 'ylog',
70 | 'ymin=',
71 | 'ymax='])
72 | except getopt.GetoptError as err:
73 | sys.stderr.write(str(err).capitalize() + '\n')
74 | usage()
75 | sys.exit(1)
76 |
77 | # loading default options
78 | for opt, arg in opts:
79 | if opt in ('-c', '--config'):
80 | def_path = str(arg)
81 | break
82 | else:
83 | def_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'defaults.json')
84 |
85 | with open(def_path, 'r') as fp:
86 | options = json.load(fp)['settings']
87 | options['def_path'] = def_path
88 |
89 | # parsing command-line options
90 | for opt, arg in opts:
91 | if opt in ('-a', '--alpha'):
92 | options['alpha'] = float(arg)
93 | elif opt in ('-b', '--backend'):
94 | options['backend'] = str(arg)
95 | elif opt in ('-c', '--config'):
96 | pass # already processed
97 | elif opt in ('-d', '--dry-run'):
98 | options['dry_run'] = True
99 | elif opt in ('-f', '--font'):
100 | options['font'] = str(arg)
101 | elif opt == '--font-sz':
102 | options['font_sz'] = float(arg)
103 | elif opt in ('-h', '--help'):
104 | usage()
105 | sys.exit(0)
106 | elif opt == '--no-grid':
107 | options['no_grid'] = True
108 | elif opt in ('-j', '--join-key'):
109 | options['join_key'] = [k.strip() for k in str(arg).split(',')]
110 | elif opt in ('-k', '--key'):
111 | options['key'] = str(arg)
112 | elif opt in ('-l', '--latex'):
113 | options['usetex'] = True
114 | elif opt == '--lalpha':
115 | options['lgd_alpha'] = float(arg)
116 | elif opt == '--legend':
117 | options['legend'] = [k.strip() for k in str(arg).split(',')]
118 | elif opt == '--lloc':
119 | options['lgd_loc'] = str(arg)
120 | elif opt == '--lncol':
121 | options['lgd_ncol'] = int(arg)
122 | elif opt in ('-n', '--by-name'):
123 | options['by_name'] = True
124 | elif opt == '--only':
125 | options['only'] = [t.strip() for t in str(arg).split(',')]
126 | elif opt in ('-p', '--plot-type'):
127 | options['plot_type'] = str(arg)
128 | elif opt in ('-r', '--replace'):
129 | options['repls'] = json.loads(str(arg))
130 | elif opt == '--reverse':
131 | options['reverse'] = True
132 | elif opt == '--save-to':
133 | options['save_to'] = str(arg)
134 | elif opt == '--shape':
135 | options['shape'] = str(arg)
136 | elif opt in ('-t', '--timeout'):
137 | options['timeout'] = float(arg)
138 | elif opt == '--tlabel':
139 | options['t_label'] = str(arg)
140 | elif opt == '--tol-loc':
141 | options['tlb_loc'] = str(arg)
142 | elif opt == '--transparent':
143 | options['transparent'] = True
144 | elif opt == '--vbs':
145 | options['vbs'] = json.loads(str(arg))
146 | elif opt == '--xkcd':
147 | options['xkcd'] = True
148 | elif opt == '--xlabel':
149 | options['x_label'] = str(arg)
150 | elif opt == '--xlog':
151 | options['x_log'] = True
152 | elif opt == '--xmin':
153 | options['x_min'] = float(arg)
154 | elif opt == '--xmax':
155 | options['x_max'] = float(arg)
156 | elif opt == '--ylabel':
157 | options['y_label'] = str(arg)
158 | elif opt == '--ylog':
159 | options['y_log'] = True
160 | elif opt == '--ymin':
161 | options['y_min'] = float(arg)
162 | elif opt == '--ymax':
163 | options['y_max'] = float(arg)
164 | else:
165 | assert False, 'Unhandled option: {0} {1}'.format(opt, arg)
166 |
167 | return options, args
168 |
169 |
170 | #
171 | #==============================================================================
172 | def usage():
173 | """
174 | Prints usage message.
175 | """
176 |
177 | print('Usage:', os.path.basename(sys.argv[0]), ' [options] stat-files')
178 | print('Options:')
179 | print(' -a, --alpha= Alpha value (only for scatter plots)')
180 | print(' Available values: [0 .. 1] (default = 0.3)')
181 | print(' -b, --backend= Backend to use')
182 | print(' Available values: pdf, pgf, png, ps, svg (default = pdf)')
183 | print(' -c, --config= Path to the default configuration file (default = $MKPLOT/defaults.json)')
184 | print(' -d, --dry-run Do not create a plot but instead show the tools sorted in the terminal')
185 | print(' -f, --font= Font to use')
186 | print(' Available values: cmr, helvetica, palatino, times (default = times)')
187 | print(' --font-sz= Font size to use')
188 | print(' Available values: [0 .. INT_MAX] (default = 12)')
189 | print(' -h, --help Show this message')
190 | print(' --no-grid Do not show the grid')
191 | print(' -j, --join-key= Comma-separated list of keys to join all benchmarks per each tool')
192 | print(' -k, --key= Key to measure')
193 | print(' Available values: \'rtime\', for others look at the STAT file (default = \'rtime\')')
194 | print(' -l, --latex Use latex')
195 | print(' --lalpha= Legend transparency level')
196 | print(' Available values: [0 .. 1] (default = 1.0)')
197 | print(' --legend= Comma-separated list of keys to use in the legend of a plot')
198 | print(' Format: "program,prog_args" (default = program)')
199 | print(' --lloc= Legend location')
200 | print(' Available values: upper/center/lower left/right, center, best, off (default = upper left)')
201 | print(' --lncol= Number of columns in the legend')
202 | print(' Available values: [1 .. INT_MAX] (default = 1)')
203 | print(' -n, --by-name Assign line style to tools by their name')
204 | print(' --only= Comma-separated list of names')
205 | print(' Format: "tool1,tool2" (default = none)')
206 | print(' -p, --plot-type= Plot type to produce')
207 | print(' Available values: cactus or scatter (default = cactus)')
208 | print(' -r, --replace= List of name replacements')
209 | print(' Format: {"name1": "$nice_name1$", "name2": "$nice_name2$"} (default = none)')
210 | print(' --reverse Use reversed sorting')
211 | print(' --save-to= Where result figure should be saved')
212 | print(' Default value: plot')
213 | print(' --shape= Shape of the plot')
214 | print(' Available values: long, squared, standard (default = standard)')
215 | print(' -t, --timeout= Timeout value')
216 | print(' Available values: [0 .. INT_MAX] (default = 3600)')
217 | print(' --tlabel= Timeout label (for scatter plots only)')
218 | print(' --tol-loc= Where to put the timeout label')
219 | print(' Available values: before, after (default = after)')
220 | print(' --transparent Save the file in the transparent mode')
221 | print(' --vbs= List of VBSes')
222 | print(' Format: {"vbs1": ["tool1", "tool2"], "vbs2": "all"} (default = none)')
223 | print(' --xkcd Use xkcd-style sketch plotting')
224 | print(' --xlabel= X label')
225 | print(' --xlog Use logarithmic scale for X axis')
226 | print(' --xmax= X axis ends at this value')
227 | print(' Available values: [0 .. INT_MAX] (default = none)')
228 | print(' --xmin= X axis starts from this value')
229 | print(' Available values: [0 .. INT_MAX] (default = 0)')
230 | print(' --ylabel= Y label')
231 | print(' --ylog Use logarithmic scale for Y axis')
232 | print(' --ymax= Y axis ends at this value')
233 | print(' Available values: [0 .. INT_MAX] (default = none)')
234 | print(' --ymin= Y axis starts from this value')
235 | print(' Available values: [0 .. INT_MAX] (default = 0)')
236 |
237 |
238 | #
239 | #==============================================================================
240 | if __name__ == '__main__':
241 | options, fns = parse_options()
242 |
243 | if not fns:
244 | pass # error handling
245 |
246 | data = load_data(fns, options)
247 |
248 | if options['dry_run']:
249 | for d in data:
250 | d1 = list(map(lambda x: min(x, options['timeout']), d[1]))
251 |
252 | print('{0}:'.format(d[0]))
253 | print(' # solved: {0}'.format(d[2]))
254 | print(' min. val: {0:.1f}'.format(float(min(d1))))
255 | print(' max. val: {0:.1f}'.format(float(max(d1))))
256 | print(' avg. val: {0:.1f}'.format(float(sum(d1)) / len(d1)))
257 | else:
258 | if options['plot_type'] == 'cactus':
259 | plotter = Cactus(options)
260 | else:
261 | plotter = Scatter(options)
262 |
263 | plotter.create(data)
264 |
--------------------------------------------------------------------------------
/plot.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python
2 | #-*- coding:utf-8 -*-
3 | ##
4 | ## plot.py
5 | ##
6 | ## Created on: Jun 05, 2015
7 | ## Author: Alexey S. Ignatiev
8 | ## E-mail: aignatiev@ciencias.ulisboa.pt
9 | ##
10 |
11 | #
12 | #==============================================================================
13 | import matplotlib.pyplot as plt
14 | import numpy as np
15 | import os
16 |
17 |
18 | #
19 | #==============================================================================
20 | class Plot():
21 | """
22 | Basic plotting class.
23 | """
24 |
25 | def __init__(self, options):
26 | """
27 | Constructor.
28 | """
29 |
30 | self.alpha = options['alpha']
31 | self.backend = options['backend']
32 | self.save_to = options['save_to']
33 | self.def_path = options['def_path']
34 | self.transparent = options['transparent']
35 |
36 | self.timeout = options['timeout']
37 | self.t_label = options['t_label']
38 | self.tlb_loc = options['tlb_loc']
39 |
40 | self.x_label = options['x_label']
41 | self.x_log = options['x_log']
42 | self.x_max = options['x_max']
43 | self.x_min = options['x_min']
44 | self.y_label = options['y_label']
45 | self.y_log = options['y_log']
46 | self.y_max = options['y_max']
47 | self.y_min = options['y_min']
48 |
49 | self.lgd_loc = options['lgd_loc']
50 | self.lgd_ncol = options['lgd_ncol']
51 | self.lgd_alpha = options['lgd_alpha']
52 | self.lgd_fancy = options['lgd_fancy']
53 | self.lgd_shadow = options['lgd_shadow']
54 |
55 | self.no_grid = options['no_grid']
56 | self.grid_color = options['grid_color']
57 | self.grid_style = options['grid_style']
58 | self.grid_width = options['grid_width']
59 | self.byname = options['by_name']
60 |
61 | # where to save
62 | self.save_to = '{0}.{1}'.format(os.path.splitext(self.save_to)[0], self.backend)
63 |
64 | # font properties
65 | self.f_props = {'serif': ['Times'], 'sans-serif': ['Helvetica'],
66 | 'weight': 'normal', 'size': options['font_sz']}
67 |
68 | if options['font'].lower() in ('sans', 'sans-serif', 'helvetica'): # Helvetica
69 | self.f_props['family'] = 'sans-serif'
70 | elif options['font'].lower() in ('serif', 'times'): # Times
71 | self.f_props['family'] = 'serif'
72 | elif options['font'].lower() == 'cmr': # Computer Modern Roman
73 | self.f_props['family'] = 'serif'
74 | self.f_props['serif'] = 'Computer Modern Roman'
75 | elif options['font'].lower() == 'palatino': # Palatino
76 | self.f_props['family'] = 'serif'
77 | self.f_props['serif'] = 'Palatino'
78 |
79 | plt.rc('text', usetex=options['usetex'])
80 | plt.rc('font', **self.f_props)
81 |
82 | # figure properties
83 | nof_subplots = 1
84 | fig_width_pt = 252.0 # Get this from LaTeX using \showthe\columnwidth
85 | inches_per_pt = 1.0 / 72.27 # Convert pt to inch
86 | golden_mean = (np.sqrt(5) + 1.0) / 2.0 # Aesthetic ratio
87 | fig_width = fig_width_pt * inches_per_pt + 0.2 # width in inches
88 | fig_height = fig_width / golden_mean * nof_subplots + 0.395 * (nof_subplots - 1) # height in inches
89 | if options['shape'] == 'squared':
90 | fig_width = fig_height
91 | elif len(options['shape']) >= 4 and options['shape'][:4] == 'long':
92 | coeff = options['shape'][4:]
93 | fig_width *= 1.2 if not coeff else float(coeff) # default coefficient is 1.2
94 |
95 | fig_size = [fig_width * 2.5, fig_height * 2.5]
96 |
97 | params = {'backend': 'pdf', 'text.usetex': options['usetex'], 'figure.figsize': fig_size}
98 |
99 | plt.rcParams.update(params)
100 |
101 | # choosing backend
102 | if self.backend in ('pdf', 'ps', 'svg'): # default is pdf
103 | plt.switch_backend(self.backend)
104 | elif self.backend == 'pgf': # PGF/TikZ
105 | pgf_params = {'pgf.texsystem': 'pdflatex',
106 | 'pgf.preamble': [r'\usepackage[utf8x]{inputenc}', r'\usepackage[T1]{fontenc}']}
107 | params.update(pgf_params)
108 | plt.rcParams.update(params)
109 | plt.switch_backend(self.backend)
110 | elif self.backend == 'png':
111 | plt.switch_backend('agg')
112 |
113 | # funny mode
114 | if options['xkcd']:
115 | plt.xkcd()
116 |
--------------------------------------------------------------------------------
/scatter.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python
2 | #-*- coding:utf-8 -*-
3 | ##
4 | ## scatter.py
5 | ##
6 | ## Created on: Jun 05, 2015
7 | ## Author: Alexey S. Ignatiev
8 | ## E-mail: aignatiev@ciencias.ulisboa.pt
9 | ##
10 |
11 | #
12 | #==============================================================================
13 | import json
14 | import math
15 | import matplotlib.pyplot as plt
16 | from matplotlib import __version__ as mpl_version
17 | import numpy as np
18 | from plot import Plot
19 | import six
20 | from six.moves import range
21 |
22 |
23 | #
24 | #==============================================================================
25 | class ScatterException(Exception):
26 | pass
27 |
28 |
29 | #
30 | #==============================================================================
31 | class Scatter(Plot, object):
32 | """
33 | Scatter plot class.
34 | """
35 |
36 | def __init__(self, options):
37 | """
38 | Scatter constructor.
39 | """
40 |
41 | super(Scatter, self).__init__(options)
42 |
43 | # setting up axes limits
44 | if not self.x_min:
45 | self.x_min = self.y_min # self.y_min is supposed to have a default value
46 | else:
47 | self.y_min = self.x_min
48 |
49 | if not self.x_max:
50 | self.x_max = 0
51 | if not self.y_max:
52 | self.y_max = 0
53 | if self.x_max and self.y_max and self.x_max != self.y_max:
54 | assert 0, 'right-most positions must be the same for X and Y axes'
55 | elif self.x_max == 0 and self.y_max == 0:
56 | self.x_max = 10
57 | while self.x_max < self.timeout:
58 | self.x_max *= 10
59 | self.y_max = self.x_max
60 | else:
61 | self.x_max = self.y_max = max(self.x_max, self.y_max)
62 |
63 | # setting timeout-line label
64 | if not self.t_label:
65 | self.t_label = '{0} sec. timeout'.format(int(self.timeout))
66 |
67 | with open(self.def_path, 'r') as fp:
68 | self.marker_style = json.load(fp)['scatter_style']
69 |
70 | def create(self, data):
71 | """
72 | Does the plotting.
73 | """
74 |
75 | if len(data[0][1]) != len(data[1][1]):
76 | raise ScatterException('Number of instances for each competitor must be the same')
77 |
78 | step = math.ceil((self.x_max - self.x_min) / 10)
79 | x = np.arange(self.x_min, self.x_max + self.x_min + step, step)
80 |
81 | # "good" area
82 | plt.plot(x, x, color='black', ls=':', lw=1.5, zorder=3)
83 | plt.plot(x, 0.1 * x, 'g:', lw=1.5, zorder=3)
84 | plt.plot(x, 10 * x, 'g:', lw=1.5, zorder=3)
85 | plt.fill_between(x, 0.1 * x, 10 * x, facecolor='green', alpha=0.15,
86 | zorder=3)
87 |
88 | plt.xlim([self.x_min, self.x_max])
89 | plt.ylim([self.y_min, self.y_max])
90 |
91 | # timeout lines
92 | if self.tlb_loc != 'none':
93 | plt.axvline(self.timeout, linewidth=1, color='red', ls=':',
94 | label=str(self.timeout), zorder=3)
95 | plt.axhline(self.timeout, linewidth=1, color='red', ls=':',
96 | label=str(self.timeout), zorder=3)
97 |
98 | if self.tlb_loc == 'after':
99 | plt.text(2 * self.x_min, self.timeout + self.x_max / 40,
100 | self.t_label, horizontalalignment='left',
101 | verticalalignment='bottom', fontsize=self.f_props['size'] * 0.8)
102 | plt.text(self.timeout + self.x_max / 40, 2 * self.x_min,
103 | self.t_label, horizontalalignment='left',
104 | verticalalignment='bottom', fontsize=self.f_props['size'] * 0.8,
105 | rotation=90)
106 | else:
107 | plt.text(2 * self.x_min, self.timeout - self.x_max / 3.5,
108 | self.t_label, horizontalalignment='left',
109 | verticalalignment='bottom', fontsize=self.f_props['size'] * 0.8)
110 | plt.text(self.timeout - self.x_max / 3.5, 2 * self.x_min,
111 | self.t_label, horizontalalignment='left',
112 | verticalalignment='bottom', fontsize=self.f_props['size'] * 0.8,
113 | rotation=90)
114 |
115 | # scatter
116 | plt.scatter(data[0][1], data[1][1], c=self.marker_style['color'],
117 | marker=self.marker_style['marker'],
118 | edgecolors=self.marker_style['edgecolor'],
119 | s=self.marker_style['size'],
120 | alpha=self.alpha, zorder=5)
121 |
122 | # axes' labels
123 | if self.x_label:
124 | plt.xlabel(self.x_label)
125 | else:
126 | plt.xlabel(data[0][0])
127 |
128 | if self.y_label:
129 | plt.ylabel(self.y_label)
130 | else:
131 | plt.ylabel(data[1][0])
132 |
133 | # turning the grid on
134 | if not self.no_grid:
135 | plt.grid(True, color='black', ls=':', lw=1, zorder=1)
136 |
137 | # choosing logarithmic scales
138 | ax = plt.gca()
139 | ax.set_xscale('log')
140 | ax.set_yscale('log')
141 |
142 | # setting ticks font properties
143 | # set_*ticklables() seems to be not needed in matplotlib 1.5.0
144 | if float(mpl_version[:3]) < 1.5:
145 | ax.set_xticklabels(ax.get_xticks(), self.f_props)
146 | ax.set_yticklabels(ax.get_yticks(), self.f_props)
147 |
148 | # formatter
149 | majorFormatter = plt.LogFormatterMathtext(base=10)
150 | ax.xaxis.set_major_formatter(majorFormatter)
151 | ax.yaxis.set_major_formatter(majorFormatter)
152 |
153 | # setting frame thickness
154 | for i in six.itervalues(ax.spines):
155 | i.set_linewidth(1)
156 |
157 | plt.savefig(self.save_to, bbox_inches='tight', transparent=self.transparent)
158 |
159 | # def create(self, data):
160 | # """
161 | # Does the plotting.
162 | # """
163 |
164 | # if len(data[0][1]) != len(data[1][1]):
165 | # raise ScatterException('Number of instances for each competitor must be the same')
166 |
167 | # # trying to remove duplicated points with the same coordinates
168 | # xs, ys = self.remove_dups(zip(data[0][1], data[1][1]))
169 |
170 | # step = int((self.x_max - self.x_min) / 10)
171 | # x = np.arange(self.x_min, self.x_max + self.x_min + step, step)
172 |
173 | # # "good" area
174 | # plt.plot(x, x, color='black', ls=':', lw=1.5, zorder=3)
175 | # plt.plot(x, 0.1 * x, 'g:', lw=1.5, zorder=3)
176 | # plt.plot(x, 10 * x, 'g:', lw=1.5, zorder=3)
177 | # plt.fill_between(x, 0.1 * x, 10 * x, facecolor='green', alpha=0.15,
178 | # zorder=3)
179 |
180 | # plt.xlim([self.x_min, self.x_max])
181 | # plt.ylim([self.y_min, self.y_max])
182 |
183 | # # timeout lines
184 | # plt.axvline(self.timeout, linewidth=1, color='red', ls=':',
185 | # label=str(self.timeout), zorder=3)
186 | # plt.axhline(self.timeout, linewidth=1, color='red', ls=':',
187 | # label=str(self.timeout), zorder=3)
188 |
189 | # if self.tlb_loc == 'after':
190 | # plt.text(2 * self.x_min, self.timeout + self.x_max / 40,
191 | # self.t_label, horizontalalignment='left',
192 | # verticalalignment='bottom', fontsize=self.f_props['size'] * 0.8)
193 | # plt.text(self.timeout + self.x_max / 40, 2 * self.x_min,
194 | # self.t_label, horizontalalignment='left',
195 | # verticalalignment='bottom', fontsize=self.f_props['size'] * 0.8,
196 | # rotation=90)
197 | # else:
198 | # plt.text(2 * self.x_min, self.timeout - self.x_max / 3.5,
199 | # self.t_label, horizontalalignment='left',
200 | # verticalalignment='bottom', fontsize=self.f_props['size'] * 0.8)
201 | # plt.text(self.timeout - self.x_max / 3.5, 2 * self.x_min,
202 | # self.t_label, horizontalalignment='left',
203 | # verticalalignment='bottom', fontsize=self.f_props['size'] * 0.8,
204 | # rotation=90)
205 |
206 | # # making the scatter plot step by step for each level
207 | # print sum([len(x) for x in xs]), len(data[0][1])
208 | # xx = []
209 | # yy = []
210 | # rgba_c =[]
211 | # rgba_e =[]
212 | # for l in range(len(xs)):
213 | # xx.extend(xs[l])
214 | # yy.extend(ys[l])
215 |
216 | # # ctuple = (1., 0., 0., 1 - (1.0 - self.alpha) ** (l + 1))
217 | # # rgba_c.extend([ctuple for i in xs[l]])
218 |
219 | # # etuple = (0., 0., 0., 1 - (1.0 - self.alpha) ** (l + 1))
220 | # # rgba_e.extend([etuple for i in xs[l]])
221 |
222 | # # plt.scatter(xx, yy, c=rgba_c, marker='o', edgecolor=rgba_e, s=25, zorder=5)
223 | # plt.scatter(xx, yy, c='r', marker='o', alpha=self.alpha, s=25, zorder=5)
224 |
225 | # # axes' labels
226 | # if self.x_label:
227 | # plt.xlabel(self.x_label)
228 | # else:
229 | # plt.xlabel(data[0][0])
230 |
231 | # if self.y_label:
232 | # plt.ylabel(self.y_label)
233 | # else:
234 | # plt.ylabel(data[1][0])
235 |
236 | # # turning the grid on
237 | # if not self.no_grid:
238 | # plt.grid(True, color='black', ls=':', lw=1, zorder=1)
239 |
240 | # # choosing logarithmic scales
241 | # ax = plt.gca()
242 | # ax.set_xscale('log')
243 | # ax.set_yscale('log')
244 |
245 | # # setting ticks font properties
246 | # ax.set_xticklabels(ax.get_xticks(), self.f_props)
247 | # ax.set_yticklabels(ax.get_yticks(), self.f_props)
248 |
249 | # # formatter
250 | # majorFormatter = plt.LogFormatterMathtext(base=10)
251 | # ax.xaxis.set_major_formatter(majorFormatter)
252 | # ax.yaxis.set_major_formatter(majorFormatter)
253 |
254 | # # setting frame thickness
255 | # for i in six.itervalues(ax.spines):
256 | # i.set_linewidth(1)
257 |
258 | # plt.savefig(self.save_to, bbox_inches='tight', transparent=self.transparent)
259 |
260 | # def remove_dups(self, data):
261 | # """
262 | # Removes duplicated points.
263 | # """
264 |
265 | # pset = set([])
266 | # pdup = {}
267 | # distance = lambda p,q: ((p[0] - q[0]) ** 2 + (p[1] - q[1]) ** 2) ** 0.5
268 |
269 | # for p in data:
270 | # for q in pset:
271 | # if distance(p, q) == 0:
272 | # pdup[q] += 1
273 | # break
274 | # else:
275 | # pdup[p] = 1
276 | # pset.add(p)
277 |
278 | # maxdups = max(six.itervalues(pdup))
279 | # xs = [[] for i in range(maxdups)]
280 | # ys = [[] for i in range(maxdups)]
281 |
282 | # for p, l in six.iteritems(pdup):
283 | # xs[l - 1].append(p[0])
284 | # ys[l - 1].append(p[1])
285 |
286 | # return xs, ys
287 |
--------------------------------------------------------------------------------
/statutil.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python
2 | #-*- coding:utf-8 -*-
3 | ##
4 | ## statutil.py
5 | ##
6 | ## Created on: May 22, 2013
7 | ## Author: Alexey S. Ignatiev
8 | ## E-mail: aignatiev@ciencias.ulisboa.pt
9 | ##
10 |
11 | #
12 | #==============================================================================
13 | from __future__ import print_function
14 | import json
15 | import sys
16 |
17 |
18 | #
19 | #==============================================================================
20 | class JSONException(Exception):
21 | pass
22 |
23 |
24 | #
25 | #==============================================================================
26 | class Stat:
27 | """
28 | Simple statistical data class.
29 | """
30 |
31 | def __init__(self, filename=None):
32 | """
33 | Constructor.
34 | """
35 |
36 | if filename is None:
37 | self.insts_own = []
38 | self.preamble = {}
39 | self.data = {}
40 | elif type(filename) is list:
41 | print( 'in case of several files use "StatArray" class', file=sys.stderr)
42 | else:
43 | self.read(filename)
44 |
45 | def read(self, filename=None):
46 | """
47 | Reads a file into a Stat object.
48 | """
49 |
50 | if filename is None:
51 | print( 'no filename was specified', file=sys.stderr)
52 | return
53 |
54 | with open(filename, 'r') as fp:
55 | print('reading {0}'.format(filename), file=sys.stderr)
56 | try:
57 | data_full = json.load(fp)
58 | except:
59 | raise JSONException('Unable to parse \'{0}\'.'.format(filename))
60 |
61 | self.data = data_full['stats']
62 | self.preamble = data_full['preamble']
63 | self.preamble['origin'] = filename
64 |
65 | self.insts_own = sorted(list(set(self.data.keys())))
66 |
67 | def write(self, to=None):
68 | """
69 | Writes a Stat object to a file.
70 | """
71 |
72 | to_write = {'preamble': self.preamble, 'stats': self.data}
73 |
74 | if to is None:
75 | to = self.preamble['origin']
76 |
77 | # 'origin' field is not needed anymore
78 | del(self.preamble['origin'])
79 |
80 | if type(to) is str:
81 | with open(to, 'w') as fp:
82 | json.dump(to_write, fp, indent=4, separators=(',', ': '))
83 | elif type(to) is file:
84 | json.dump(to_write, to, indent=4, separators=(',', ': '))
85 | else:
86 | print('don\'t know how to write to {0}'.format(type(to)), file=sys.stderr)
87 |
88 | def update(self, success=None, failure=None):
89 | """
90 | Updates stats using additional success and failure signs.
91 | """
92 |
93 | if success:
94 | pass
95 |
96 | if failure:
97 | sign = lambda x: x
98 | key = failure
99 | if failure[:3] == 'no-':
100 | sign = lambda x: not x
101 | key = failure[3:]
102 |
103 | for inst in self.insts_own:
104 | if inst in self.data and self.data[inst]['status'] == True:
105 | if sign(key in self.data[inst]):
106 | print('updating', inst, file=sys.stderr)
107 | self.data[inst]['status'] = False
108 |
109 | self.write()
110 |
111 |
112 | def list(self, crit=None):
113 | """
114 | Lists instances satisfying the criterion.
115 | """
116 |
117 | if crit:
118 | pred = lambda x: x == crit['val']
119 | if crit['pred'] == '<':
120 | pred = lambda x: x < crit['val']
121 | elif crit['pred'] == '<=':
122 | pred = lambda x: x <= crit['val']
123 | elif crit['pred'] == '>':
124 | pred = lambda x: x > crit['val']
125 | elif crit['pred'] == '>=':
126 | pred = lambda x: x >= crit['val']
127 |
128 | for inst in self.insts_own:
129 | if inst in self.data and self.data[inst]['status'] == True:
130 | if pred(self.data[inst][crit['key']]):
131 | print('{0}: {1} = {2}'.format(inst, crit['key'], self.data[inst][crit['key']]))
132 |
133 |
134 | #
135 | #==============================================================================
136 | class StatArray:
137 | """
138 | Contains statistical data for several files.
139 | """
140 |
141 | def __init__(self, files=None):
142 | """
143 | Constructor.
144 | """
145 |
146 | if files is None:
147 | self.inst_full = []
148 | self.stat_objs = []
149 | elif type(files) is list:
150 | self.read(files)
151 | else:
152 | print('in case of just one file use "Stat" class', file=sys.stderr)
153 | self.read([files])
154 |
155 | def __getitem__(self, key):
156 | if key < len(self.stat_objs):
157 | return self.stat_objs[key]
158 |
159 | def __len__(self):
160 | return len(self.stat_objs)
161 |
162 | def __iter__(self):
163 | for stat_obj in self.stat_objs:
164 | yield stat_obj
165 |
166 | def read(self, files=None):
167 | """
168 | Reads several files into a StatArray object.
169 | """
170 |
171 | if files is None:
172 | print('no files was specified', file=sys.stderr)
173 | return
174 |
175 | self.stat_objs = []
176 | for f in files:
177 | self.stat_objs.append(Stat(f))
178 |
179 | inst_set = set()
180 | for stat_obj in self.stat_objs:
181 | inst_set = inst_set.union(set(stat_obj.insts_own))
182 | self.inst_full = sorted(list(inst_set))
183 |
184 | def write(self, files=None):
185 | """
186 | Writes a StatArray object to given files.
187 | """
188 |
189 | if files is None:
190 | files = [stat_obj.preamble['origin'] for stat_obj in self.stat_objs]
191 |
192 | assert len(files) == len(self.stat_objs), 'wrong number of filenames'
193 |
194 | for f, stat_obj in zip(files, self.stat_objs):
195 | stat_obj.write(f)
196 |
197 | def cluster(self, use_key=['program', 'prog_args']):
198 | """
199 | Clasters Stat objects according to their preamble values.
200 | """
201 |
202 | # the key should be a list
203 | if type(use_key) is not list:
204 | use_key = [use_key]
205 |
206 | clusters = {}
207 |
208 | for stat_obj in self.stat_objs:
209 | # updating the Stat object
210 | for i, i_old in enumerate(stat_obj.insts_own):
211 | i_new = '{0}@{1}'.format(i_old, stat_obj.preamble['benchmark'])
212 | stat_obj.insts_own[i] = i_new
213 | stat_obj.data[i_new] = stat_obj.data.pop(i_old)
214 |
215 | key = ' '.join([stat_obj.preamble[one_key] for one_key in use_key])
216 | if key in clusters:
217 | # update the cluster
218 | clusters[key].insts_own.extend(stat_obj.insts_own)
219 | clusters[key].data.update(stat_obj.data)
220 |
221 | clusters[key].preamble['benchmark'].append(stat_obj.preamble['benchmark'])
222 | clusters[key].preamble['runsolver_args'].append(stat_obj.preamble['runsolver_args'])
223 | else:
224 | # add new cluster
225 | clusters[key] = stat_obj
226 | clusters[key].preamble['benchmark'] = [clusters[key].preamble['benchmark']]
227 | clusters[key].preamble['runsolver_args'] = [clusters[key].preamble['runsolver_args']]
228 |
229 | self.stat_objs = [cl for cl in clusters.values()]
230 |
231 | inst_set = set()
232 | for stat_obj in self.stat_objs:
233 | inst_set = inst_set.union(set(stat_obj.insts_own))
234 | self.inst_full = sorted(list(inst_set))
235 |
236 | def unclaster(self):
237 | """
238 | Unclasters previously clastered Stat objects.
239 | """
240 |
241 | print('unclaster() method is not implemented yet', file=sys.stderr)
242 |
243 | def make_vbs(self, addit_key=None):
244 | """
245 | Makes vbs using the status, rtime and additional key as the measurement.
246 | NOTE: the use of addit_key is not implemented yet.
247 | """
248 |
249 | vbs = Stat()
250 | vbs.insts_own = self.inst_full
251 |
252 | vbs.preamble = self.stat_objs[0].preamble
253 | vbs.preamble['program'] = 'vbs'
254 | vbs.preamble['prog_args'] = ''
255 | vbs.preamble['origin'] = [obj.preamble['origin'] for obj in self.stat_objs]
256 |
257 | for inst in self.inst_full:
258 | alts = []
259 | for stat_obj in self.stat_objs:
260 | if inst in stat_obj.data and stat_obj.data[inst]['status'] == True:
261 | alts.append(stat_obj.data[inst])
262 |
263 | if alts:
264 | vbs.data[inst] = min(alts, key=lambda x: x['rtime'])
265 | else:
266 | # all fail; choose any:
267 | vbs.data[inst] = self.stat_objs[0].data[inst]
268 |
269 | self.stat_objs.append(vbs)
270 |
271 | def compare(self, cmp_key=None):
272 | """
273 | Compares values for a specific key. Do nothing if cmp_key is None.
274 | """
275 |
276 | if cmp_key:
277 | for inst in self.inst_full:
278 | vals = {}
279 |
280 | for stat_obj in self.stat_objs:
281 | if inst in stat_obj.data and stat_obj.data[inst]['status'] == True and cmp_key in stat_obj.data[inst]:
282 | if stat_obj.data[inst][cmp_key] in vals:
283 | vals[stat_obj.data[inst][cmp_key]].append(stat_obj.preamble['origin'])
284 | else:
285 | vals[stat_obj.data[inst][cmp_key]] = [stat_obj.preamble['origin']]
286 |
287 | if len(vals.keys()) > 1:
288 | print('different values found', file=sys.stderr)
289 | print('instance:', inst, file=sys.stderr)
290 | print('values:', vals, file=sys.stderr)
291 |
292 | def list_simple(self, to_list='all'):
293 | """
294 | Shows instances required by user.
295 | """
296 |
297 | if to_list:
298 | print('showing {0}:'.format(to_list))
299 | if to_list == 'all':
300 | for inst in self.inst_full:
301 | print(inst)
302 |
303 | else:
304 | status = False if to_list == 'failed' else True
305 |
306 | for inst in self.inst_full:
307 | objs = []
308 |
309 | for stat_obj in self.stat_objs:
310 | if inst in stat_obj.data and stat_obj.data[inst]['status'] == status:
311 | p = stat_obj.preamble
312 | if 'prog_alias' in p:
313 | objs.append(p['prog_alias'])
314 | else:
315 | objs.append(p['program'] + ' ' + p['prog_args'])
316 |
317 | if objs:
318 | if len(self.stat_objs) > 1:
319 | objs = '[{0}]'.format(', '.join(obj for obj in objs))
320 | print('{0}: {1}'.format(inst, objs))
321 | else:
322 | print(inst)
323 |
324 | def list(self, crit=None):
325 | """
326 | Shows instances required by user.
327 | """
328 |
329 | if crit:
330 | for stat_obj in self.stat_objs:
331 | stat_obj.list(crit)
332 |
333 | def update(self, success=None, failure=None):
334 | """
335 | Update stats using additional success and failure signs.
336 | """
337 |
338 | if success or failure:
339 | for stat_obj in self.stat_objs:
340 | stat_obj.update(success, failure)
341 |
--------------------------------------------------------------------------------