├── .github └── workflows │ └── python-app.yml ├── .gitignore ├── MANIFEST.in ├── README.md ├── config ├── cmr10.ttf └── default.json ├── deprecated └── database_to_runtime_table.py ├── examples ├── example.db ├── example.png └── simple.db ├── ompl_benchmark_plotter.py ├── pytest.ini ├── requirements.txt ├── setup.py ├── src ├── __init__.py ├── database_info.py ├── database_to_graph.py ├── get_diverse_color.py └── get_plot_style.py └── tests ├── __init__.py ├── data ├── example.db ├── example.json ├── simple.db └── simple.json └── test_ompl_benchmark_plotter.py /.github/workflows/python-app.yml: -------------------------------------------------------------------------------- 1 | name: OmplBenchmarkPlotterCI 2 | on: 3 | push: 4 | branches: [ "master" ] 5 | pull_request: 6 | branches: [ "master" ] 7 | permissions: 8 | contents: read 9 | jobs: 10 | test: 11 | runs-on: ubuntu-latest 12 | steps: 13 | - uses: actions/checkout@v3 14 | - name: Set up Python 3.10 15 | uses: actions/setup-python@v4 16 | with: 17 | python-version: '3.10' 18 | - name: Install dependencies 19 | run: | 20 | python -m pip install --upgrade pip 21 | pip install pytest 22 | if [ -f requirements.txt ]; then pip install -r requirements.txt; fi 23 | - name: Add src to PYTHONPATH 24 | run: echo "PYTHONPATH=$PYTHONPATH:$(pwd)/src" >> $GITHUB_ENV 25 | - name: Test with pytest 26 | run: | 27 | pytest tests/ 28 | 29 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | #Ignore 2 | *.json 3 | *.pyc 4 | *.pdf 5 | *.png 6 | *.aux 7 | *.log 8 | *.tex 9 | 10 | #Allow 11 | !tests/data/simple.json 12 | !tests/data/example.json 13 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include README.md 2 | include config/default.json * 3 | include config/cmr10.ttf * 4 | recursive-include tests/data/ 5 | 6 | 7 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # ompl_benchmark_plotter 2 | 3 | This is an easy-to-use script to quickly generate cost-success graphs (as pdf) from benchmark database file(s) generated in the [Open Motion Planning Library (OMPL)](https://github.com/ompl/ompl). 4 | 5 | ## Quick start 6 | 7 | To produce a cost-success graph from a database file, simply run the following command: 8 | ``` 9 | ./ompl_benchmark_plotter.py examples/example.db -s 10 | ``` 11 | 12 | ![Cost-success Graph](examples/example.png) 13 | 14 | ### Input 15 | 16 | There are two options to use the script: 17 | 18 | * Single database file: As input, you can specify one database (.db) files. 19 | * Multiple database files: If more than one database file is specified, the script will generate a single graph by merging the database files. This allows you to compare different planners for the same experiment, which are stored in different database files. If you want to compare different experiments, you can simply run the script multiple times with different database files. 20 | 21 | ### General Options 22 | 23 | * **-o**, **--output-file** _PDF-FILENAME_ 24 | Specify output pdf filename 25 | * **-s**, **--show** 26 | Show output as pdf (requires xdg-open). 27 | * **-v {0,1,2,3}**, **--verbose {0,1,2,3}** 28 | Select verbosity level. Default: 1. 29 | * **--quiet** 30 | Do not show any output. Invalidates any verbose values. 31 | 32 | ### Graph generation Options 33 | 34 | NOTE: default values can be found in ```config/default.json``` 35 | 36 | * **--max-cost** _MAX-COST_ 37 | Specify upper bound on cost axis. 38 | * **--min-cost** _MIN-COST_ 39 | Specify lower bound on cost axis. 40 | * **--max-time** _MAX-TIME_ 41 | Specify upper bound on time axis. 42 | * **--min-time** _MIN-TIME_ 43 | Specify lower bound on time axis. 44 | * **--fontsize** _FONTSIZE_ 45 | Fontsize of title and descriptions. 46 | * **--label-fontsize** _LABEL-FONTSIZE_ 47 | Fontsize of tick labels. 48 | * **--ignore-non-optimal-planner** 49 | Do not plot non-optimal planner. 50 | * **--title-name** _TITLE-NAME_ 51 | Set title name 52 | * **--legend-separate-file** 53 | Print legend as separate file. 54 | * **--legend-below-figure** 55 | Print legend below graph. 56 | * **--legend-none** 57 | Do not print legend. 58 | 59 | ### Run unit tests 60 | ``` 61 | pytest 62 | ``` 63 | 64 | # Acknowledgements 65 | 66 | [@gammell](https://github.com/gammell): This repository has been created as a tool to easily generate graphs similar to the 67 | ones which [Jonathan Gammell](https://robotic-esp.com/people/gammell/) has used throughout his work (Check out his 68 | work on [asymptotically-optimal motion planning](https://robotic-esp.com/code/bitstar/).) 69 | 70 | [@frangrothe](https://github.com/frangrothe): This library is also based upon earlier work by [Francesco Grothe](https://github.com/frangrothe), 71 | who used cost-success graphs extensively for his work on time-based motion 72 | planning (Check it out here: https://github.com/frangrothe/bt-motion-planning). 73 | 74 | [@servetb](https://github.com/servetb) and [@JayKamat99](https://github.com/JayKamat99): For feedback on earlier versions, and advice on improvements. 75 | -------------------------------------------------------------------------------- /config/cmr10.ttf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aorthey/ompl_benchmark_plotter/9e49aff3ac233eb16826dc9f55f18c06c90a5c0a/config/cmr10.ttf -------------------------------------------------------------------------------- /config/default.json: -------------------------------------------------------------------------------- 1 | { 2 | "alpha_percentile": 0.4, 3 | "resolution": 200, 4 | "resolution_linear": 10, 5 | "min_time": { 6 | "success": 0.001, 7 | "optimization": 0.001 8 | }, 9 | "max_time": { 10 | "success": 30, 11 | "optimization": 30 12 | }, 13 | "min_cost": 0, 14 | "max_cost": 100, 15 | "ci_left": 25, 16 | "ci_left_default": 39, 17 | "ci_right": 75, 18 | "ci_right_default": 59, 19 | "fontsize": 30, 20 | "label_fontsize": 18, 21 | "ylabel_success": "success [%]", 22 | "ylabel_optimization": "solution cost", 23 | "xlabel": "run time [s]", 24 | "linestyle": "-", 25 | "linewidth": "3", 26 | "legend_linewidth": "5", 27 | "markerstyle": "x" 28 | } 29 | 30 | 31 | -------------------------------------------------------------------------------- /deprecated/database_to_runtime_table.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | from collections import defaultdict 3 | import subprocess 4 | import glob 5 | import re 6 | import sys 7 | import argparse 8 | import os 9 | import sqlite3 10 | import numpy as np 11 | import json 12 | from src.plot_style import * 13 | from src.database_info import * 14 | 15 | def get_cell_entry(data, experiment, planner, config): 16 | hide_variance = config['hide_variance'] 17 | decimals = int(config['decimals']) 18 | time = data['experiments'][experiment][planner]['time_mean'] 19 | timelimit = data['experiments'][experiment][planner]['time_limit'] 20 | if time > timelimit: 21 | time = timelimit 22 | 23 | var = data['experiments'][experiment][planner]['time_variance'] 24 | best = data['experiments'][experiment][planner]['best_planner'] 25 | number_runs = data['experiments'][experiment][planner]['number_runs'] 26 | 27 | cell_entry = "$" 28 | if best: 29 | cell_entry += "\\textbf{%.*f}"%(decimals, time) 30 | else: 31 | cell_entry += "%.*f"%(decimals, time) 32 | if not hide_variance: 33 | cell_entry += "\\color{gray}{\\pm %.*f}"%(decimals, var) 34 | cell_entry += "$" 35 | return cell_entry 36 | 37 | def tex_table_from_json_data(database_filepaths, data, config): 38 | 39 | if config['output_file']: 40 | name = config['output_file'] 41 | tex_name = change_filename_extension(name, ".tex") 42 | pdf_name = change_filename_extension(name, ".pdf") 43 | tex_filepath = get_filename_from_database_filepaths_and_name(database_filepaths, tex_name) 44 | pdf_filepath = get_filename_from_database_filepaths_and_name(database_filepaths, pdf_name) 45 | else: 46 | tex_filepath = get_tex_from_database_filepaths(database_filepaths) 47 | pdf_filepath = get_pdf_from_database_filepaths(database_filepaths) 48 | 49 | ############################################################ 50 | ## Map planner to experiments 51 | ############################################################ 52 | planner_map = {} 53 | for (ctr, experiment) in enumerate(data['experiments']): 54 | for (ctr, planner) in enumerate(data['experiments'][experiment]): 55 | if not planner in planner_map: 56 | planner_map[planner] = [] 57 | planner_map[planner].append(experiment) 58 | 59 | if config['reverse']: 60 | longest_name=get_longest_name_from_planners(planner_map) 61 | else: 62 | longest_name=get_longest_name_from_experiments(data['experiments']) 63 | 64 | f = open(tex_filepath, 'w') 65 | 66 | f.write("\\documentclass{article}\n") 67 | f.write("\\usepackage{tabularx}\n") 68 | f.write("\\usepackage{rotating}\n") 69 | f.write("\\usepackage{makecell}\n") 70 | f.write("\\usepackage{xcolor}\n") 71 | f.write("\\usepackage[text={174mm,258mm}, papersize={210mm,297mm}, columnsep=12pt, headsep=21pt, centering]{geometry}") 72 | f.write("\\begin{document}\n\n") 73 | 74 | f.write("\\newcolumntype{V}{>{\\centering\\arraybackslash}m{.033\\linewidth}}\n") 75 | f.write("\\newcolumntype{Z}{>{\\raggedleft\\arraybackslash}m{.01\\linewidth}}\n") 76 | f.write("\\newcolumntype{+}{!{\\vrule width 1.2pt}}\n") 77 | 78 | f.write("\\begin{table*}[t]\n") 79 | f.write("\\centering\n") 80 | f.write("\\renewcommand{\\cellrotangle}{90}\n") 81 | f.write("\\renewcommand\\theadfont{\\bfseries}\n") 82 | 83 | f.write("\\settowidth{\\rotheadsize}{\\theadfont ") 84 | f.write(str(longest_name)) 85 | f.write("}\n") 86 | 87 | f.write("\\footnotesize\\centering\n") 88 | f.write("\\renewcommand{\\arraystretch}{1.2}\n") 89 | f.write("\\setlength\\tabcolsep{3pt}\n") 90 | 91 | if config['reverse']: 92 | n_columns = len(planner_map) 93 | else: 94 | n_columns = len(data['experiments']) 95 | 96 | format_str = "|ZX+" 97 | for i in range(0,n_columns): 98 | format_str += "X|" 99 | 100 | s = "\\begin{tabularx}{\\linewidth}{"+format_str+"}" 101 | 102 | s += "\\hline\n" 103 | s += "& & \\multicolumn{"+str(n_columns) 104 | s += "}{>{\\centering}p{.8\\textwidth}|}{List of Scenarios}\\\\" 105 | s += "\\cline{3-"+str(2+n_columns)+"}\n" 106 | 107 | ############################################################ 108 | ## Create X-axis 109 | ############################################################ 110 | if config['reverse']: 111 | s += "\\multicolumn{2}{|>{\\centering}p{2.5cm}+}{\\rothead{Scenario}}\n" 112 | else: 113 | s += "\\multicolumn{2}{|>{\\centering}p{2.5cm}+}{\\rothead{Motion Planner}}\n" 114 | 115 | if config['reverse']: 116 | for (ctr, planner) in enumerate(planner_map): 117 | name = get_label(planner).replace("#","\#") 118 | s += " & \\rothead{%s} \n"%(name) 119 | else: 120 | for (ctr, experiment) in enumerate(data['experiments']): 121 | name = get_experiment_label(experiment) 122 | s += " & \\rothead{%s} \n"%(name) 123 | s += " \\\\ \\hline\n" 124 | f.write(s) 125 | 126 | if config['reverse']: 127 | for (ctr, experiment) in enumerate(data['experiments']): 128 | label = get_experiment_label(experiment).replace("_"," ") 129 | pstr = str(ctr+1) + " & \\mbox{" + str(label) + "} & " 130 | for (pctr, planner) in enumerate(planner_map): 131 | if experiment in planner_map[planner]: 132 | pstr += get_cell_entry(data, experiment, planner, config) 133 | else: 134 | pstr += "$-$" 135 | if pctr < len(planner_map) - 1: 136 | pstr += " & " 137 | else: 138 | pstr += " \\\\ \n" 139 | f.write(pstr) 140 | else: 141 | for (ctr, planner) in enumerate(planner_map): 142 | label = get_label(planner).replace("#","\#") 143 | pstr = str(ctr+1) + " & \\mbox{" + str(label) + "} & " 144 | for (ctr, experiment) in enumerate(data['experiments']): 145 | if experiment in planner_map[planner]: 146 | pstr += get_cell_entry(data, experiment, planner, config) 147 | else: 148 | pstr += "$-$" 149 | 150 | if ctr < len(data['experiments']) - 1: 151 | pstr += " & " 152 | else: 153 | pstr += " \\\\ \n" 154 | f.write(pstr) 155 | 156 | f.write("\\hline\n") 157 | f.write("\\end{tabularx}\n") 158 | 159 | f.write("\\caption{Runtime (s) of %d runs with %s planning algorithms on %s scenarios \ 160 | with cut-off time limit of %.2fs. \ 161 | Entry '$-$' means that planner does not support this planning scenario.}\n" 162 | %(data['info']['run_count'], len(planner_map), len(data['experiments']), data['info']['timelimit'])) 163 | 164 | f.write("\\end{table*}\n") 165 | f.write("\\end{document}\n") 166 | f.close() 167 | 168 | if config['verbosity'] > 1: 169 | os.system("pdflatex -output-directory %s %s" % (os.path.dirname(tex_filepath), tex_filepath)) 170 | print(80*"#") 171 | print("Wrote tex file to %s" % tex_filepath) 172 | print(80*"#") 173 | else: 174 | os.system("pdflatex -output-directory %s %s > /dev/null 2>&1" % (os.path.dirname(tex_filepath), tex_filepath)) 175 | 176 | ## Remove temporary/auxiliary files 177 | aux_file = os.path.splitext(tex_filepath)[0] + '.aux' 178 | log_file = os.path.splitext(tex_filepath)[0] + '.log' 179 | os.system("rm -rf %s" % (aux_file)) 180 | os.system("rm -rf %s" % (log_file)) 181 | 182 | if config['verbosity'] > 0: 183 | print("Created pdf file {}".format(pdf_filepath)) 184 | if config['show']: 185 | os.system("xdg-open %s" % pdf_filepath) 186 | 187 | def create_runtime_table_from_databases(database_filepaths, config): 188 | data = {} 189 | data["info"] = load_config() 190 | data["info"]["timelimit"] = 0 191 | data["info"]["run_count"] = 0 192 | data["experiments"] = {} 193 | 194 | for database_filepath in database_filepaths: 195 | con = sqlite3.connect(database_filepath) 196 | cursor = con.cursor() 197 | if config['verbosity'] > 1: 198 | print_metadata_from_database(cursor) 199 | if config['verbosity'] > 2: 200 | get_run_results_from_database(cursor) 201 | 202 | experiments = cursor.execute("SELECT id, name FROM {}".format('experiments')).fetchall() 203 | for experiment in experiments: 204 | experiment_name = experiment[1] 205 | experiment_id = experiment[0] 206 | timelimit = get_time_limit_for_experiment(cursor, experiment_id) 207 | data["info"]["timelimit"] = timelimit 208 | timespace = create_time_space_linear(data) 209 | planners = cursor.execute("SELECT id, name FROM {}".format('plannerConfigs')).fetchall() 210 | planner_data = {} 211 | best_time = float("inf") 212 | best_planner = "" 213 | 214 | for planner in planners: 215 | planner_id = planner[0] 216 | planner_name = planner[1] 217 | 218 | run_count = cursor.execute("SELECT COUNT(*) FROM {} WHERE plannerid={} AND experimentid={}".format('runs', planner_id, experiment_id)).fetchall()[0][0] 219 | 220 | times_per_run = cursor.execute("SELECT time FROM {} WHERE plannerid={} AND experimentid={}".format('runs', planner_id, experiment_id)).fetchall() 221 | times = np.array(times_per_run) 222 | if is_planner_optimal(planner_name): 223 | getids = cursor.execute("SELECT id FROM {} WHERE plannerid={}".format('runs',planner_id)).fetchall() 224 | runs = np.array(getids).flatten() 225 | runids = ','.join(str(run) for run in runs) 226 | success = get_count_success(cursor, len(runs), runids, timespace) 227 | times = get_times_from_success_over_time(success, timespace) 228 | 229 | solved_per_run = cursor.execute("SELECT solved FROM {} WHERE plannerid={} AND experimentid={}".format('runs', planner_id, experiment_id)).fetchall() 230 | 231 | success_percentage = np.sum(np.array(solved_per_run).flatten())/run_count 232 | time_mean = np.mean(times) 233 | planner_data[planner_name] = { 'time_mean' : time_mean, 234 | 'time_variance' : np.std(times), 'success' : success_percentage, 235 | 'time_limit' : timelimit, 'best_planner' : False, 'number_runs' : run_count} 236 | if timelimit > data['info']['timelimit']: 237 | data['info']['timelimit'] = timelimit 238 | if time_mean < best_time: 239 | best_time = time_mean 240 | best_planner = planner_name 241 | 242 | if best_time < float("inf"): 243 | planner_data[best_planner]['best_planner'] = True 244 | 245 | if config['ignore_ending_name']: 246 | experiment_name = experiment_name.rsplit('_', 1)[0] 247 | 248 | if not experiment_name in data['experiments']: 249 | data['experiments'][experiment_name] = planner_data 250 | else: 251 | planner_data = combine_planner_data(data['experiments'][experiment_name], planner_data) 252 | data['experiments'][experiment_name] = planner_data 253 | 254 | if len(database_filepaths) > 0: 255 | tex_table_from_json_data(database_filepaths, data, config) 256 | 257 | -------------------------------------------------------------------------------- /examples/example.db: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aorthey/ompl_benchmark_plotter/9e49aff3ac233eb16826dc9f55f18c06c90a5c0a/examples/example.db -------------------------------------------------------------------------------- /examples/example.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aorthey/ompl_benchmark_plotter/9e49aff3ac233eb16826dc9f55f18c06c90a5c0a/examples/example.png -------------------------------------------------------------------------------- /examples/simple.db: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aorthey/ompl_benchmark_plotter/9e49aff3ac233eb16826dc9f55f18c06c90a5c0a/examples/simple.db -------------------------------------------------------------------------------- /ompl_benchmark_plotter.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | import argparse 3 | import os 4 | from src.database_to_graph import * 5 | 6 | ############################################################ 7 | ## Setup argument parser 8 | ############################################################ 9 | 10 | def run_benchmark_plotter(input_arguments): 11 | parser = argparse.ArgumentParser(description='Plotting of OMPL Benchmark Files.') 12 | 13 | parser.add_argument('database_files', type=str, nargs='+', help='Database (.db) file(s)') 14 | parser.add_argument('-v','--verbose', type=int, choices=[0,1,2,3], default=1, help='Select verbosity level for stdout.') 15 | parser.add_argument('-q', '--quiet', action='store_const', const=True, help='Do not show any output. Invalidates any verbose values.') 16 | parser.add_argument('-s','--show', action='store_const', const=True, help='Show output as pdf (requires xdg-open).') 17 | parser.add_argument('-o','--output-file', type=str, help='Save as filename.') 18 | 19 | #### Options for optimality graph 20 | graph_group = parser.add_argument_group("Cost-success graph options") 21 | graph_group.add_argument('--max-cost', type=float, help='Specify upper bound on cost display.') 22 | graph_group.add_argument('--min-cost', type=float, help='Specify lower bound on cost display.') 23 | graph_group.add_argument('--max-time', type=float, help='Specify upper bound on time display.') 24 | graph_group.add_argument('--min-time', type=float, help='Specify lower bound on time display.') 25 | graph_group.add_argument('--fontsize', type=float, help='Fontsize of title and descriptions.') 26 | graph_group.add_argument('--label-fontsize', type=float, help='Fontsize of tick labels.') 27 | graph_group.add_argument('--only-success-graph', action='store_const', const=True, help='Plot only the success graph.') 28 | graph_group.add_argument('--ignore-non-optimal-planner', action='store_const', const=True, help='Do not plot non-optimal planner.') 29 | graph_group.add_argument('--legend-separate-file', action='store_const', const=True, help='Print legend as separate file.') 30 | graph_group.add_argument('--legend-below-figure', action='store_const', 31 | const=True, help='Print legend below graph.') 32 | graph_group.add_argument('--legend-none', action='store_const', 33 | const=True, help='Do not print legend.') 34 | graph_group.add_argument('--title-name', action='store', type=str, help='Set title name.') 35 | 36 | args = parser.parse_args(input_arguments) 37 | if args.quiet: 38 | args.verbose = 0 39 | 40 | ############################################################ 41 | ## Sanity checks 42 | ############################################################ 43 | if args.show: 44 | from shutil import which 45 | if which("xdg-open") is None: 46 | print("Error: Cannot run with --show option if xdg-open is not installed.") 47 | return 1 48 | 49 | for fname in args.database_files: 50 | if fname is None or not os.path.isfile(fname): 51 | if args.verbose > 0: 52 | print("Error: {} is not a file.".format(fname)) 53 | return 1 54 | 55 | ############################################################ 56 | ## Run the chosen program 57 | ############################################################ 58 | if args.verbose > 0: 59 | print("Create optimality graphs for {} files.".format(len(args.database_files))) 60 | 61 | plot_config = make_config(args) 62 | plot_graph_from_databases(args.database_files, plot_config) 63 | 64 | return 0 65 | 66 | if __name__ == '__main__': 67 | sys.exit(run_benchmark_plotter(sys.argv[1:])) 68 | -------------------------------------------------------------------------------- /pytest.ini: -------------------------------------------------------------------------------- 1 | [pytest] 2 | log_cli=true 3 | log_level=DEBUG 4 | log_format = %(asctime)s %(levelname)s %(message)s 5 | log_date_format = %Y-%m-%d %H:%M:%S 6 | addopts=-s --capture=no 7 | pythonpath = src 8 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | numpy==1.24.3 2 | matplotlib==3.7.1 3 | argparse==1.4.0 4 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from setuptools import setup, find_packages 2 | 3 | with open('README.md', 'r') as fh: 4 | long_description = fh.read() 5 | 6 | setup( 7 | name='ompl_benchmark_plotter', 8 | version='0.1.0', 9 | author='Andreas Orthey', 10 | author_email='aorthey@rtr.ai', 11 | description='A simple tool to plot OMPL benchmark results', 12 | long_description=long_description, 13 | long_description_content_type='text/markdown', 14 | url='https://github.com/aorthey/ompl_benchmark_plotter', 15 | packages=find_packages(where='src'), 16 | package_dir={'': 'src'}, 17 | install_requires=[ 18 | 'numpy==1.24.3', 19 | 'matplotlib==3.7.1', 20 | 'argparse==1.4.0' 21 | ], 22 | classifiers=[ 23 | 'Development Status :: 3 - Alpha', 24 | 'Intended Audience :: Developers', 25 | 'License :: OSI Approved :: MIT License', 26 | 'Programming Language :: Python :: 3', 27 | 'Programming Language :: Python :: 3.6', 28 | 'Programming Language :: Python :: 3.7', 29 | 'Programming Language :: Python :: 3.8', 30 | 'Programming Language :: Python :: 3.9', 31 | 'Programming Language :: Python :: 3.10', 32 | ], 33 | python_requires='>=3.6', 34 | ) 35 | 36 | -------------------------------------------------------------------------------- /src/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aorthey/ompl_benchmark_plotter/9e49aff3ac233eb16826dc9f55f18c06c90a5c0a/src/__init__.py -------------------------------------------------------------------------------- /src/database_info.py: -------------------------------------------------------------------------------- 1 | import json 2 | import os 3 | import sys 4 | import numpy as np 5 | import sqlite3 6 | from itertools import repeat 7 | from pathlib import Path 8 | from src.get_plot_style import * 9 | 10 | def get_cost_results(cursor, runids, times, max_cost, ci_left, ci_right): 11 | medians = np.zeros(len(times)) 12 | quantile5 = np.zeros(len(times)) 13 | quantile95 = np.zeros(len(times)) 14 | 15 | improvement = False 16 | for i in range(len(times)): 17 | data = np.array(cursor.execute("SELECT a.best_cost FROM (SELECT MAX(time), \ 18 | best_cost FROM {0} WHERE time<={1} AND runid in ({2}) GROUP BY runid) a".format('progress', times[i], runids)).fetchall()).flatten() 19 | if data.size == 0: 20 | medians[i] = max_cost 21 | quantile5[i] = max_cost 22 | quantile95[i] = max_cost 23 | else: 24 | data = np.where(data == None, max_cost, data) 25 | medians[i] = np.median(data) 26 | quantile5[i] = np.percentile(data, ci_left, interpolation='nearest') 27 | quantile95[i] = np.percentile(data, ci_right, interpolation='nearest') 28 | improvement = True 29 | 30 | return [improvement, medians, quantile5, quantile95] 31 | 32 | def is_planner_optimal(cursor, planner_id): 33 | getids = cursor.execute("SELECT id FROM {} WHERE plannerid={}".format('runs', planner_id)).fetchone() 34 | runid = getids[0] 35 | data = cursor.execute("SELECT time, best_cost FROM {} WHERE runid in ({})".format('progress', runid)).fetchall() 36 | if data is None: 37 | return False 38 | return len(data) > 10 39 | 40 | def remove_non_optimal_planner(cursor, planners): 41 | if not has_best_cost(cursor): 42 | return [] 43 | return [x for x in planners if is_planner_optimal(cursor, x[0])] 44 | 45 | def get_tables_from_database(cursor): 46 | tables = cursor.execute("SELECT name FROM sqlite_master WHERE type='table' ORDER BY name").fetchall() 47 | for table in tables: 48 | cursor.execute("SELECT * FROM {}".format(table[0])).fetchall() 49 | names = list(map(lambda x: x[0], cursor.description)) 50 | print("\nTable \'{}\': {}".format(table[0], names)) 51 | 52 | def combine_planner_data(planner_data1, planner_data2): 53 | if planner_data1 is None: 54 | return planner_data2 55 | if planner_data2 is None: 56 | return planner_data1 57 | 58 | planner_data = {} 59 | best_time = float("inf") 60 | best_planner = "" 61 | for name1 in planner_data1: 62 | found = False 63 | for name2 in planner_data2: 64 | if name1 == name2: 65 | t1 = planner_data1[name1]['time_mean'] 66 | t2 = planner_data2[name2]['time_mean'] 67 | time_limit = planner_data1[name1]['time_limit'] 68 | s1 = planner_data1[name1]['time_variance'] 69 | s2 = planner_data2[name2]['time_variance'] 70 | time_variance = np.std([s1, s2]) 71 | time_mean = np.mean([t1, t2]) 72 | n1 = planner_data1[name1]['number_runs'] 73 | n2 = planner_data2[name2]['number_runs'] 74 | number_runs = n1 + n2 75 | su1 = planner_data1[name1]['success'] 76 | su2 = planner_data2[name2]['success'] 77 | success = 0.5*(su1 + su2) 78 | planner_data[name1] = { 'time_mean' : time_mean, 'time_limit': 79 | time_limit, 'time_variance' : time_variance, 'success' : success, 80 | 'best_planner' : False, 81 | 'number_runs' : number_runs } 82 | found = True 83 | if time_mean < best_time: 84 | best_time = time_mean 85 | best_planner = name1 86 | break 87 | if not found: 88 | planner_data[name1] = planner_data1[name1] 89 | if best_time < float("inf"): 90 | planner_data[best_planner]['best_planner'] = True 91 | return planner_data 92 | 93 | def get_experiment_names_from_database(cursor): 94 | experiments = cursor.execute("SELECT id, name FROM {}".format('experiments')).fetchall() 95 | experiment_names = [] 96 | for experiment in experiments: 97 | experiment_name = experiment[1] 98 | experiment_names.append("Experiment {} [ID {}]".format(experiment[1], experiment[0])) 99 | return experiment_names 100 | 101 | def get_maxtime_from_database(cursor): 102 | times = cursor.execute("SELECT timelimit FROM {}".format('experiments')).fetchall() 103 | times = np.array(times).flatten() 104 | return times.max() 105 | 106 | def get_maxtime_from_database_or_config(cursor, config, data): 107 | if config['max_time'] > 0: 108 | time = config['max_time'] 109 | else: 110 | time = get_maxtime_from_database(cursor) 111 | 112 | data["info"]['max_time']['success'] = time 113 | data["info"]['max_time']['optimization'] = time 114 | return time 115 | 116 | def get_mintime_from_database_or_config(cursor, config, data): 117 | kDefaultTimeDifferenceOrderOfMagnitude = 3 118 | if config['min_time'] > 0: 119 | time = config['min_time'] 120 | else: 121 | maxtime = float(get_maxtime_from_database_or_config(cursor, config, data)) 122 | oom = int(np.floor(np.log10(maxtime))) 123 | time = 10**(oom-kDefaultTimeDifferenceOrderOfMagnitude) 124 | 125 | data["info"]['min_time']['success'] = time 126 | data["info"]['min_time']['optimization'] = time 127 | return time 128 | 129 | def get_start_index(medians, times, max_cost): 130 | for i in range(len(times)): 131 | if medians[i] < max_cost: 132 | return i 133 | return len(times) 134 | 135 | def get_maxcost_from_json_or_config(data, config, times): 136 | kPercentagePaddingAboveMax = 0.1 137 | max_cost_in_data_acquisition = data["info"]["max_cost"] 138 | max_cost = 0 139 | if config['max_cost'] > 0: 140 | max_cost = config['max_cost'] 141 | else: 142 | planner_data = data["planners"] 143 | for planner in planner_data: 144 | ycost = max_cost_in_data_acquisition 145 | planner_optimization_success = planner_data[planner]["optimization_success"] 146 | if planner_optimization_success: 147 | planner_median = planner_data[planner]["median"] 148 | start = get_start_index(planner_median, times, max_cost_in_data_acquisition) 149 | cost_below_max = planner_median[start:] 150 | if len(cost_below_max) > 0: 151 | ycost = np.array(cost_below_max).max() 152 | else: 153 | planner_point = planner_data[planner]["point"] 154 | ycost = planner_point["cost"][0] 155 | if ycost > max_cost: 156 | if ycost < max_cost_in_data_acquisition: 157 | max_cost = ycost 158 | max_cost = max_cost + kPercentagePaddingAboveMax*max_cost 159 | 160 | data["info"]["max_cost"] = max_cost 161 | return max_cost 162 | 163 | def get_mincost_from_json_or_config(data, config, times, max_cost): 164 | kPercentagePaddingBelowMin = 0.1 165 | min_cost = max_cost 166 | if config['min_cost'] > 0: 167 | min_cost = config['min_cost'] 168 | else: 169 | planner_data = data["planners"] 170 | for planner in planner_data: 171 | planner_optimization_success = planner_data[planner]["optimization_success"] 172 | if planner_optimization_success: 173 | planner_median = planner_data[planner]["median"] 174 | ycost = np.array(planner_median).min() 175 | else: 176 | planner_point = planner_data[planner]["point"] 177 | ycost = planner_point["cost"][0] 178 | if ycost < min_cost: 179 | min_cost = ycost 180 | 181 | dc = max_cost - min_cost 182 | min_cost = np.maximum(0.0, min_cost - kPercentagePaddingBelowMin*dc) 183 | data["info"]["min_cost"] = min_cost 184 | return min_cost 185 | 186 | def get_experiment_name_from_array(experiment_names): 187 | if len(experiment_names) < 1: 188 | return "unknown" 189 | else: 190 | return experiment_names[0] 191 | 192 | def assert_equivalent_experiment_names(cursor): 193 | experiments = cursor.execute("SELECT id, name FROM {}".format('experiments')).fetchall() 194 | if len(experiments) > 0: 195 | name = experiments[0][1] 196 | for experiment in experiments: 197 | next_name = experiment[1] 198 | if not (name == next_name): 199 | return False 200 | 201 | return True 202 | 203 | def get_experiment_names_from_database(cursor): 204 | experiments = cursor.execute("SELECT id, name FROM {}".format('experiments')).fetchall() 205 | experiment_names = [] 206 | for experiment in experiments: 207 | experiment_name = experiment[1] 208 | # experiment_names.append("Experiment {} [ID {}]".format(experiment[1], experiment[0])) 209 | experiment_names.append(experiment_name) 210 | return experiment_names 211 | 212 | def get_longest_name_from_experiments(names): 213 | longest_name = "" 214 | for name in names: 215 | name = get_experiment_label(name) 216 | if len(name) > len(longest_name) : 217 | longest_name = name 218 | return longest_name 219 | 220 | def get_longest_name_from_planners(names): 221 | longest_name = "" 222 | for name in names: 223 | name = get_label(name) 224 | if len(name) > len(longest_name) : 225 | longest_name = name 226 | return longest_name 227 | 228 | def get_time_limit_for_experiment(cursor, experiment_id): 229 | return cursor.execute("SELECT timelimit FROM {} WHERE id={}".format('experiments',experiment_id)).fetchall()[0][0] 230 | 231 | def get_planner_names_from_database(cursor): 232 | planners = cursor.execute("SELECT id, name FROM {}".format('plannerConfigs')).fetchall() 233 | planner_names = [] 234 | for planner in planners: 235 | planner_names.append(planner[1]) 236 | return planner_names 237 | 238 | def has_table_column(cursor, table_name, column_name): 239 | cursor.execute("SELECT * FROM {}".format(table_name)).fetchall() 240 | names = list(map(lambda x: x[0], cursor.description)) 241 | return True if column_name in names else False 242 | 243 | def has_solution_length(cursor): 244 | return has_table_column(cursor, 'runs', 'solution_length') 245 | 246 | def has_best_cost(cursor): 247 | return has_table_column(cursor, 'progress', 'best_cost') 248 | 249 | def get_filename_from_database_filepaths_and_name(filepaths, name): 250 | if len(filepaths) <= 0: 251 | return "unknown" 252 | filepath = filepaths[0] 253 | directory = os.path.dirname(filepath) 254 | if len(directory) > 0: 255 | directory += "/" 256 | filename = directory + name 257 | return filename 258 | 259 | def get_filename_from_database_filepaths(filepaths): 260 | if len(filepaths) <= 0: 261 | return "unknown" 262 | filepath = filepaths[0] 263 | directory = os.path.dirname(filepath) 264 | filename = os.path.splitext(os.path.basename(filepath))[0] 265 | if len(directory) > 0: 266 | directory += "/" 267 | 268 | filename_without_extension = directory + filename 269 | return filename_without_extension 270 | 271 | def create_filename_with_extension(filename_without_extension, extension): 272 | i = 0 273 | filename = filename_without_extension + ("%s"%(extension)) 274 | path_exist = os.path.exists(filename) 275 | if path_exist: 276 | print("Warning: overwriting file %s."%(filename)) 277 | return filename 278 | 279 | def get_pdf_from_database_filepaths(filepaths): 280 | filename_without_extension = get_filename_from_database_filepaths(filepaths) 281 | return create_filename_with_extension(filename_without_extension, ".pdf") 282 | 283 | def get_json_filepath_from_databases(filepaths): 284 | filename_without_extension = get_filename_from_database_filepaths(filepaths) 285 | return create_filename_with_extension(filename_without_extension, ".json") 286 | 287 | def change_filename_extension(filepath, extension): 288 | filepath_without_extension = os.path.splitext(filepath)[0] 289 | return filepath_without_extension + extension 290 | 291 | def create_time_space(data): 292 | return np.logspace(np.log10(data["info"]["min_time"]["success"]), \ 293 | np.log10(data["info"]["max_time"]["success"]), \ 294 | data["info"]["resolution"]) 295 | 296 | def create_time_space_linear(data): 297 | return np.linspace(0, data["info"]["timelimit"], \ 298 | data["info"]["resolution_linear"]) 299 | 300 | def get_times_from_success_over_time(success, timespace): 301 | # input: number of successes for each time element 302 | all_count = 0 303 | times = [] 304 | for (count, time) in zip(success, timespace): 305 | Nitems = int(count - all_count) 306 | if Nitems > 0: 307 | times.extend(repeat(time, Nitems)) 308 | all_count = count 309 | ## add last time element to fill it up for unsuccessful runs 310 | if len(times) < 100: 311 | times.extend(repeat(time, 100-len(times))) 312 | times = np.array(times) 313 | return times 314 | 315 | def get_count_success(cursor, run_count, runids, times): 316 | success = np.zeros(len(times)) 317 | for i in range(len(times)): 318 | ### Select from one time slice all best costs 319 | data = np.array(cursor.execute("SELECT a.best_cost FROM (SELECT MAX(time), \ 320 | best_cost FROM {0} WHERE time<={1} AND runid in ({2}) GROUP BY runid) a".format('progress', times[i], runids)).fetchall()).flatten() 321 | if data.size == 0: 322 | success[i] = 0 323 | else: 324 | sum_not_none = sum(x is not None for x in data) 325 | success[i] = (sum_not_none / run_count ) * 100.0 326 | 327 | return success 328 | 329 | def load_config(): 330 | cwd = Path(__file__).parent.absolute() 331 | json_config = "{}/../config/default.json".format(cwd) 332 | with open(json_config, 'r') as jsonfile: 333 | info = json.load(jsonfile) 334 | return info 335 | 336 | def get_average_runtime_from_database(cursor, data): 337 | raise Exception("NYI") 338 | 339 | ############################################################ 340 | ### Print All Runs 341 | ############################################################ 342 | # print(80*"-") 343 | # print("-- Runs in database") 344 | # print(80*"-") 345 | 346 | # runs = cursor.execute("SELECT id, experimentid, plannerid, correct_solution, time, best_cost FROM {}".format('runs')).fetchall() 347 | # for run in runs: 348 | # print("Run {} on environment {} with planner {}. Correct solution:{}. Time {}".format(run[0], run[1], run[2], run[3], run[4])) 349 | 350 | def print_run_results_from_database(cursor): 351 | planners = cursor.execute("SELECT id, name FROM {}".format('plannerConfigs')).fetchall() 352 | for planner in planners: 353 | planner_id = planner[0] 354 | getids = cursor.execute("SELECT id FROM {} WHERE plannerid={}".format('runs',planner_id)).fetchall() 355 | runs = np.array(getids).flatten() 356 | runids = ','.join(str(run) for run in runs) 357 | 358 | exp_getids = cursor.execute("SELECT id FROM {}".format('experiments')).fetchall() 359 | exps = np.array(exp_getids).flatten() 360 | expids = ','.join(str(exp) for exp in exps) 361 | 362 | print("Run ids {}".format(runids)) 363 | print("Exp ids {}".format(expids)) 364 | 365 | if has_best_cost(cursor): 366 | data = np.array(cursor.execute("SELECT time, best_cost FROM {} WHERE runid in ({})".format('progress', runids)).fetchall()).flatten() 367 | if len(data) > 0: 368 | print("Planner {} with time {} and cost {}".format(planner[1], data[0], data[1])) 369 | else: 370 | data = np.array(cursor.execute("SELECT time FROM {} WHERE id in ({}) and \ 371 | experimentid in ({})".format('runs', runids, expids)).fetchall()).flatten() 372 | print("Planner {} with time {}".format(planner[1], np.around(data))) 373 | 374 | for planner in planners: 375 | planner_id = planner[0] 376 | getids = cursor.execute("SELECT id FROM {} WHERE plannerid={}".format('runs',planner_id)).fetchall() 377 | runs = np.array(getids).flatten() 378 | runids = ','.join(str(run) for run in runs) 379 | if has_best_cost(cursor): 380 | data = np.array(cursor.execute("SELECT time, best_cost FROM {} WHERE runid in ({})".format('progress', runids)).fetchall()).flatten() 381 | if len(data) > 0: 382 | print("Planner {} with time {} and cost {}".format(planner[1], data[0], data[1])) 383 | 384 | def print_metadata_from_database(cursor): 385 | print(80*"-") 386 | print("-- Tables in database file") 387 | print(80*"-") 388 | print(get_tables_from_database(cursor)) 389 | print(80*"-") 390 | print("-- Planner in database") 391 | print(80*"-") 392 | print(get_planner_names_from_database(cursor)) 393 | print(80*"-") 394 | print("-- Experiments in database") 395 | print(80*"-") 396 | print(get_experiment_names_from_database(cursor)) 397 | 398 | def make_config(args): 399 | max_cost = args.max_cost if args.max_cost else -1 400 | min_cost = args.min_cost if args.min_cost else -1 401 | max_time = args.max_time if args.max_time else -1 402 | min_time = args.min_time if args.min_time else -1 403 | fontsize = args.fontsize if args.fontsize else -1 404 | only_success_graph = args.only_success_graph if args.only_success_graph else False 405 | label_fontsize = args.label_fontsize if args.label_fontsize else -1 406 | plot_config = { 407 | 'show': args.show, 408 | 'only_success_graph': args.only_success_graph, 409 | 'output_file': args.output_file, 410 | 'verbosity': args.verbose, 411 | 'max_cost': max_cost, 412 | 'min_cost': min_cost, 413 | 'max_time': max_time, 414 | 'min_time': min_time, 415 | 'fontsize': fontsize, 416 | 'label_fontsize': label_fontsize, 417 | 'ignore_non_optimal_planner': args.ignore_non_optimal_planner, 418 | 'legend_below_figure': args.legend_below_figure, 419 | 'legend_separate_file': args.legend_separate_file, 420 | 'legend_none': args.legend_none 421 | } 422 | if args.title_name: 423 | plot_config['title_name'] = args.title_name 424 | return plot_config 425 | -------------------------------------------------------------------------------- /src/database_to_graph.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import json 3 | import os 4 | import sqlite3 5 | import numpy as np 6 | import matplotlib 7 | import matplotlib.pyplot as plt 8 | from matplotlib import font_manager, rcParams 9 | import argparse 10 | 11 | from src.database_info import * 12 | from src.get_diverse_color import * 13 | from src.get_plot_style import * 14 | 15 | font_path = "config/cmr10.ttf" 16 | 17 | fe = font_manager.FontEntry( 18 | fname=font_path, 19 | name='cmr10') 20 | font_manager.fontManager.ttflist.insert(0, fe) 21 | plt.rcParams['font.family'] = fe.name 22 | plt.rcParams['axes.formatter.use_mathtext'] = True 23 | plt.rcParams['mathtext.fontset']='cm' 24 | matplotlib.rcParams['pdf.fonttype'] = 42 25 | matplotlib.rcParams['ps.fonttype'] = 42 26 | 27 | def get_json_from_database(cursor, data, config): 28 | verbosity = config["verbosity"] 29 | ignore_non_optimal_planner = config["ignore_non_optimal_planner"] 30 | 31 | if verbosity > 1: 32 | print_metadata_from_database(cursor) 33 | 34 | if verbosity > 2: 35 | print_run_results_from_database(cursor) 36 | 37 | ############################################################ 38 | ### Print Average Success per Planner over Time 39 | ############################################################ 40 | times = create_time_space(data) 41 | 42 | planners = cursor.execute("SELECT id, name FROM {}".format('plannerConfigs')).fetchall() 43 | if ignore_non_optimal_planner: 44 | planners = remove_non_optimal_planner(planners) 45 | 46 | for planner in planners: 47 | planner_id = planner[0] 48 | planner_name = planner[1] 49 | number_runs = cursor.execute("SELECT COUNT(*) FROM {} WHERE plannerid={}".format('runs', planner_id)).fetchall()[0][0] 50 | 51 | percentages = np.empty(len(times)) 52 | for i in range(len(times)): 53 | percentage = cursor.execute( 54 | "SELECT COUNT(*) FROM {0} WHERE plannerid={2} AND time < {1}".format('runs', times[i], planner_id)).fetchall() 55 | percentages[i] = (percentage[0][0] / number_runs) * 100 56 | data["planners"][planner_name] = { 57 | "success": percentages.tolist() 58 | } 59 | if verbosity > 1: 60 | print("Planner {} (id {}) has {} runs.".format(planner_name, planner_id, number_runs)) 61 | 62 | ############################################################ 63 | ### Get Cost per Planner over Time 64 | ############################################################ 65 | max_time = data["info"]["max_time"]["optimization"] 66 | min_time = data["info"]["min_time"]["optimization"] 67 | max_cost = data["info"]["max_cost"] 68 | ci_left = data["info"]["ci_left"] 69 | ci_right = data["info"]["ci_right"] 70 | if has_best_cost(cursor): 71 | times = np.logspace(np.log10(min_time), np.log10(max_time), data["info"]["resolution"]) 72 | 73 | for planner in planners: 74 | planner_id = planner[0] 75 | planner_name = planner[1] 76 | getids = cursor.execute("SELECT id FROM {} WHERE plannerid={}".format('runs',planner_id)).fetchall() 77 | runs = np.array(getids).flatten() 78 | runids = ','.join(str(run) for run in runs) 79 | 80 | results = get_cost_results(cursor, runids, times, max_cost, ci_left, ci_right) 81 | data["planners"][planner_name]["optimization_success"] = results[0] 82 | if results[0]: 83 | data["planners"][planner_name]["median"] = results[1].tolist() 84 | data["planners"][planner_name]["quantile5"] = results[2].tolist() 85 | data["planners"][planner_name]["quantile95"] = results[3].tolist() 86 | success = get_count_success(cursor, len(runs), runids, times) 87 | if verbosity > 0: 88 | print("Planner {} success {} (runs {})".format(planner_name, success.tolist(), len(runs))) 89 | print("Planner {} median {} (runs {})".format(planner_name, results[1].tolist(), len(runs))) 90 | data["planners"][planner_name]["success"] = success.tolist() 91 | else: 92 | point_data = get_best_cost_from_runs(cursor, planner_id, ci_left, ci_right) 93 | if point_data is None: 94 | point_data = max_point(max_time, max_cost) 95 | data["planners"][planner_name]["point"] = point_data 96 | else: 97 | if verbosity > 0: 98 | print("WARNING: No best_cost entry in database file. Using solution_length instead.") 99 | for planner in planners: 100 | planner_id = planner[0] 101 | planner_name = planner[1] 102 | data["planners"][planner_name]["optimization_success"] = False 103 | 104 | point_data = get_best_cost_from_runs(cursor, planner_id, ci_left, ci_right) 105 | if point_data is None: 106 | point_data = max_point(max_time, max_cost) 107 | data["planners"][planner_name]["point"] = point_data 108 | 109 | def get_best_cost_from_runs(cursor, planner_id, ci_left, ci_right): 110 | if not has_solution_length(cursor): 111 | return None 112 | pair = np.array(cursor.execute("SELECT time, solution_length FROM {0} WHERE plannerid={1} AND status=6".format('runs', planner_id)).fetchall()) 113 | if pair.size == 0: 114 | return None 115 | split = np.split(pair, 2, axis=1) 116 | times = split[0] 117 | costs = split[1] 118 | return calculate_points(times, costs, ci_left, ci_right) 119 | 120 | def calculate_points(times, costs, ci_left, ci_right): 121 | if None in costs: 122 | return None 123 | 124 | data = {"time": [np.median(times), np.percentile(times, ci_left, interpolation='nearest'), 125 | np.percentile(times, ci_right, interpolation='nearest')], 126 | "cost": [np.median(costs), np.percentile(costs, ci_left, interpolation='nearest'), 127 | np.percentile(costs, ci_right, interpolation='nearest')]} 128 | return data 129 | 130 | def max_point(max_time, max_cost): 131 | data = {"time": [max_time, max_time, 132 | max_time], 133 | "cost": [max_cost, max_cost, max_cost]} 134 | return data 135 | 136 | def get_errors(point): 137 | time_errors = np.array([[point["time"][0] - point["time"][1]], 138 | [point["time"][2] - point["time"][0]]]) 139 | cost_errors = np.array([[point["cost"][0] - point["cost"][1]], 140 | [point["cost"][2] - point["cost"][0]]]) 141 | return time_errors, cost_errors 142 | 143 | def init_planner_colors(data): 144 | planner_data = data["planners"] 145 | planners = list(planner_data.keys()) 146 | planners.sort() 147 | for planner in planners: 148 | color = get_diverse_color(planner) 149 | 150 | def plot_success(ax, data): 151 | 152 | min_time = data["info"]["min_time"]["success"] 153 | max_time = data["info"]["max_time"]["success"] 154 | resolution = data["info"]["resolution"] 155 | fontsize = data["info"]["fontsize"] 156 | 157 | times = np.logspace(np.log10(min_time), np.log10(max_time), resolution) 158 | ax.set_xscale('log') 159 | ax.set_yscale('linear') 160 | ax.set_xlim(min_time, max_time) 161 | ax.set_ylim(0.0, 100.0) 162 | 163 | init_planner_colors(data) 164 | 165 | planner_data = data["planners"] 166 | for planner in planner_data: 167 | color = get_diverse_color(planner) 168 | success_over_time = planner_data[planner]["success"] 169 | ax.plot(times, success_over_time, color=color, 170 | linestyle=get_line_style(data, planner), 171 | linewidth=data["info"]["linewidth"], label=get_label(planner)) 172 | 173 | ax.grid(True, which="both", ls='--') 174 | ylabel = data["info"]["ylabel_success"] 175 | ax.set_ylabel(ylabel, fontsize=fontsize) 176 | 177 | def plot_optimization(ax, data, config): 178 | 179 | min_time = data["info"]["min_time"]["optimization"] 180 | max_time = data["info"]["max_time"]["optimization"] 181 | resolution = data["info"]["resolution"] 182 | times = np.logspace(np.log10(min_time), np.log10(max_time), resolution) 183 | 184 | max_cost = get_maxcost_from_json_or_config(data, config, times) 185 | min_cost = get_mincost_from_json_or_config(data, config, times, max_cost) 186 | 187 | fontsize = data["info"]["fontsize"] 188 | 189 | ax.set_xscale('log') 190 | ax.set_yscale('linear') 191 | ax.set_xlim(min_time, max_time) 192 | ax.set_ylim(min_cost, max_cost) 193 | 194 | planner_data = data["planners"] 195 | for planner in planner_data: 196 | planner_optimization_success = planner_data[planner]["optimization_success"] 197 | #color = get_color(data, planner) 198 | color = get_diverse_color(planner) 199 | if planner_optimization_success: 200 | planner_median = planner_data[planner]["median"] 201 | planner_q5 = planner_data[planner]["quantile5"] 202 | planner_q95 = planner_data[planner]["quantile95"] 203 | 204 | start = get_start_index(planner_median, times, max_cost) 205 | ax.plot(times[start:], planner_median[start:], color=color, linewidth=data["info"]["linewidth"], label=get_label(planner)) 206 | ax.fill_between(times[start:], planner_q5[start:], planner_q95[start:], color=color, alpha=data["info"]["alpha_percentile"]) 207 | else: 208 | planner_point = planner_data[planner]["point"] 209 | time_errors, cost_errors = get_errors(planner_point) 210 | plt.errorbar(planner_point["time"][0], planner_point["cost"][0], cost_errors, time_errors, c=color, marker=get_marker_style(data, planner), ms=10, lw=0.5) 211 | 212 | ax.grid(True, which="both", ls='--') 213 | ylabel = data["info"]["ylabel_optimization"] 214 | xlabel = data["info"]["xlabel"] 215 | ax.set_xlabel(xlabel, fontsize=fontsize) 216 | ax.set_ylabel(ylabel, fontsize=fontsize) 217 | 218 | def json_to_graph(json_filepath, pdf_filepath, config): 219 | with open(json_filepath, 'r') as jsonfile: 220 | data = json.load(jsonfile) 221 | 222 | if config["only_success_graph"]: 223 | fig, axs = plt.subplots(1, 1, figsize=(16,10)) 224 | plot_success(axs, data) 225 | ax_success = axs 226 | else: 227 | fig, axs = plt.subplots(2, 1, sharex='col', figsize=(16,10)) 228 | ax_success = axs[0] 229 | ax_cost = axs[1] 230 | plot_success(ax_success, data) 231 | plot_optimization(ax_cost, data, config) 232 | 233 | fontsize = data["info"]["fontsize"] 234 | label_fontsize = data["info"]["label_fontsize"] 235 | experiment_name = get_experiment_label(data["info"]["experiment"]) 236 | 237 | if 'title_name' in config: 238 | ax_success.set_title(config['title_name'], fontsize=fontsize) 239 | else: 240 | ax_success.set_title(experiment_name, fontsize=fontsize) 241 | 242 | legend_title_name = 'Planner' 243 | if not config["legend_none"]: 244 | if config["legend_separate_file"]: 245 | figl, axl = plt.subplots() 246 | label_params = ax_success.get_legend_handles_labels() 247 | legend = axl.legend(*label_params, loc="center", frameon=True, ncol=4, fontsize=fontsize) 248 | for obj in legend.legendHandles: 249 | obj.set_linewidth(data["info"]["legend_linewidth"]) 250 | axl.axis('off') 251 | legend_filepath = change_filename_extension(json_filepath, '_legend.pdf') 252 | figl.savefig(legend_filepath, bbox_extra_artists=(legend,), bbox_inches='tight') 253 | plt.close(figl) 254 | else: 255 | legend = ax_success.legend(loc='upper left', title=legend_title_name, fontsize=label_fontsize) 256 | for obj in legend.legendHandles: 257 | obj.set_linewidth(data["info"]["legend_linewidth"]) 258 | plt.setp(legend.get_title(),fontsize=label_fontsize) 259 | 260 | ax_success.tick_params(labelsize=label_fontsize) 261 | if not config["only_success_graph"]: 262 | ax_cost.tick_params(labelsize=label_fontsize) 263 | 264 | fig = plt.gcf() 265 | if config['legend_separate_file'] or config['legend_none']: 266 | plt.savefig(pdf_filepath, format='pdf', dpi=300, bbox_inches='tight') 267 | else: 268 | plt.savefig(pdf_filepath, format='pdf', dpi=300, bbox_inches='tight', bbox_extra_artists=(legend,)) 269 | 270 | if config['verbosity'] > 0: 271 | print("Wrote pdf with dpi %d to file %s" %(fig.dpi,pdf_filepath)) 272 | if config['show']: 273 | os.system('xdg-open %s' % pdf_filepath) 274 | 275 | 276 | def plot_graph_from_databases(database_filepaths, config): 277 | ############################################################ 278 | ### Create data structure from database file 279 | ############################################################ 280 | data = {} 281 | data["info"] = load_config() 282 | data["planners"] = {} 283 | if config['max_cost'] > 0: 284 | data["info"]['max_cost'] = config['max_cost'] 285 | if config['min_cost'] > 0: 286 | data["info"]['min_cost'] = config['min_cost'] 287 | if config['fontsize'] > 0: 288 | data["info"]['fontsize'] = config['fontsize'] 289 | if config['label_fontsize'] > 0: 290 | data["info"]['label_fontsize'] = config['label_fontsize'] 291 | if config['verbosity'] > 0: 292 | data["info"]['verbosity'] = config['verbosity'] 293 | 294 | experiment_names = [] 295 | experiment_times = [] 296 | for database_filepath in database_filepaths: 297 | if not os.path.isfile(database_filepath): 298 | raise Exception("{} is not an existing file.".format(database_filepath)) 299 | 300 | extension = os.path.splitext(database_filepath)[1] 301 | if not (extension == '.db'): 302 | raise Exception("{} is not a .db file.".format(database_filepath)) 303 | 304 | con = sqlite3.connect(database_filepath) 305 | cursor = con.cursor() 306 | 307 | get_maxtime_from_database_or_config(cursor, config, data) 308 | get_mintime_from_database_or_config(cursor, config, data) 309 | 310 | get_json_from_database(cursor, data, config) 311 | experiment_names.append(get_experiment_names_from_database(cursor)) 312 | 313 | experiment_names = [item for sublist in experiment_names for item in sublist] 314 | experiment_names = list(set(experiment_names)) 315 | 316 | data["info"]["experiment"] = get_experiment_name_from_array(experiment_names) 317 | 318 | ############################################################ 319 | ### Verify that all experiment names match 320 | ############################################################ 321 | 322 | if len(experiment_names) < 1 : 323 | raise Exception("Could not load experiments.") 324 | 325 | if len(experiment_names) > 1: 326 | raise Exception("Mismatching experiment names in database: {}".format(experiment_names)) 327 | 328 | if len(experiment_names) > 0 : 329 | data["info"]["experiment"] = get_experiment_name_from_array(experiment_names) 330 | 331 | ############################################################ 332 | ### Create json file from data structure 333 | ############################################################ 334 | json_filepath = get_json_filepath_from_databases(database_filepaths) 335 | with open(json_filepath, 'w') as jsonfile: 336 | json.dump(data, jsonfile, indent=4) 337 | 338 | ############################################################ 339 | ### Plot json file to pdf 340 | ############################################################ 341 | 342 | if config['output_file']: 343 | name = config['output_file'] 344 | pdf_filepath = get_filename_from_database_filepaths_and_name(database_filepaths, name) 345 | else: 346 | pdf_filepath = change_filename_extension(json_filepath, '.pdf') 347 | 348 | json_to_graph(json_filepath, pdf_filepath, config) 349 | -------------------------------------------------------------------------------- /src/get_diverse_color.py: -------------------------------------------------------------------------------- 1 | import colorsys 2 | import itertools 3 | 4 | # Define a good color palette for graphs 5 | # These colors are chosen to be distinct and visually appealing for data representation 6 | graph_colors = [ 7 | '#d62728', # brick red 8 | '#1f77b4', # muted blue 9 | '#2ca02c', # cooked asparagus green 10 | '#9467bd', # muted purple 11 | '#8c564b', # chestnut brown 12 | '#ff7f0e', # safety orange 13 | '#e377c2', # raspberry yogurt pink 14 | '#7f7f7f', # middle gray 15 | '#bcbd22', # curry yellow-green 16 | '#17becf' # blue-teal 17 | ] 18 | 19 | # Create a global color map to store the color for each planner 20 | global_color_map = {} 21 | 22 | # Use itertools.cycle to create an infinite loop through these colors 23 | color_cycler = itertools.cycle(graph_colors) 24 | 25 | def get_diverse_color(planner_name): 26 | global global_color_map 27 | 28 | if planner_name in global_color_map: 29 | return global_color_map[planner_name] 30 | 31 | hex_color = next(color_cycler) 32 | global_color_map[planner_name] = hex_color 33 | return hex_color 34 | -------------------------------------------------------------------------------- /src/get_plot_style.py: -------------------------------------------------------------------------------- 1 | def get_marker_style(data, planner_name): 2 | return data["info"]["markerstyle"] 3 | 4 | def get_line_style(data, planner_name): 5 | return data["info"]["linestyle"] 6 | 7 | def get_label(planner_name): 8 | name = planner_name.lstrip("geometric_") 9 | name = name.replace("Star","*") 10 | name = name.replace("star","*") 11 | name = name.replace("kBIT","BIT") 12 | return name 13 | 14 | def get_experiment_label(experiment_name): 15 | name = experiment_name.replace("_"," ") 16 | name = name.title() 17 | return name 18 | 19 | # def get_color(data, planner_name): 20 | # global colormap 21 | # name = get_label(planner_name) 22 | # colors = data["info"]["colors"] 23 | # if name in colors: 24 | # color = data["info"]["colors"][name] 25 | # else: 26 | # color = data["info"]["colors"]["default"] 27 | # return color 28 | 29 | -------------------------------------------------------------------------------- /tests/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aorthey/ompl_benchmark_plotter/9e49aff3ac233eb16826dc9f55f18c06c90a5c0a/tests/__init__.py -------------------------------------------------------------------------------- /tests/data/example.db: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aorthey/ompl_benchmark_plotter/9e49aff3ac233eb16826dc9f55f18c06c90a5c0a/tests/data/example.db -------------------------------------------------------------------------------- /tests/data/simple.db: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aorthey/ompl_benchmark_plotter/9e49aff3ac233eb16826dc9f55f18c06c90a5c0a/tests/data/simple.db -------------------------------------------------------------------------------- /tests/data/simple.json: -------------------------------------------------------------------------------- 1 | { 2 | "info": { 3 | "alpha_percentile": 0.4, 4 | "resolution": 200, 5 | "resolution_linear": 10, 6 | "min_time": { 7 | "success": 0.01, 8 | "optimization": 0.01 9 | }, 10 | "max_time": { 11 | "success": 10.0, 12 | "optimization": 10.0 13 | }, 14 | "min_cost": 0, 15 | "max_cost": 100, 16 | "ci_left": 25, 17 | "ci_left_default": 39, 18 | "ci_right": 75, 19 | "ci_right_default": 59, 20 | "fontsize": 30, 21 | "label_fontsize": 18, 22 | "ylabel_success": "success [%]", 23 | "ylabel_optimization": "solution cost", 24 | "xlabel": "run time [s]", 25 | "linestyle": "-", 26 | "linewidth": "3", 27 | "legend_linewidth": "5", 28 | "markerstyle": "x", 29 | "experiment": "chain" 30 | }, 31 | "planners": { 32 | "geometric_RRTConnect": { 33 | "success": [ 34 | 0.0, 35 | 0.0, 36 | 0.0, 37 | 0.0, 38 | 0.0, 39 | 0.0, 40 | 0.0, 41 | 0.0, 42 | 0.0, 43 | 0.0, 44 | 0.0, 45 | 0.0, 46 | 0.0, 47 | 0.0, 48 | 0.0, 49 | 0.0, 50 | 0.0, 51 | 0.0, 52 | 0.0, 53 | 0.0, 54 | 0.0, 55 | 0.0, 56 | 0.0, 57 | 0.0, 58 | 0.0, 59 | 0.0, 60 | 0.0, 61 | 0.0, 62 | 0.0, 63 | 0.0, 64 | 0.0, 65 | 0.0, 66 | 0.0, 67 | 0.0, 68 | 0.0, 69 | 0.0, 70 | 0.0, 71 | 0.0, 72 | 0.0, 73 | 0.0, 74 | 0.0, 75 | 0.0, 76 | 0.0, 77 | 0.0, 78 | 0.0, 79 | 0.0, 80 | 0.0, 81 | 0.0, 82 | 0.0, 83 | 0.0, 84 | 0.0, 85 | 0.0, 86 | 0.0, 87 | 0.0, 88 | 0.0, 89 | 0.0, 90 | 0.0, 91 | 0.0, 92 | 0.0, 93 | 0.0, 94 | 0.0, 95 | 0.0, 96 | 0.0, 97 | 0.0, 98 | 0.0, 99 | 0.0, 100 | 0.0, 101 | 0.0, 102 | 1.0, 103 | 2.0, 104 | 2.0, 105 | 2.0, 106 | 2.0, 107 | 2.0, 108 | 2.0, 109 | 2.0, 110 | 2.0, 111 | 2.0, 112 | 2.0, 113 | 2.0, 114 | 2.0, 115 | 2.0, 116 | 2.0, 117 | 3.0, 118 | 3.0, 119 | 3.0, 120 | 4.0, 121 | 4.0, 122 | 5.0, 123 | 6.0, 124 | 7.000000000000001, 125 | 7.000000000000001, 126 | 7.000000000000001, 127 | 7.000000000000001, 128 | 7.000000000000001, 129 | 7.000000000000001, 130 | 7.000000000000001, 131 | 7.000000000000001, 132 | 7.000000000000001, 133 | 7.000000000000001, 134 | 7.000000000000001, 135 | 7.000000000000001, 136 | 7.000000000000001, 137 | 9.0, 138 | 10.0, 139 | 10.0, 140 | 10.0, 141 | 12.0, 142 | 12.0, 143 | 12.0, 144 | 13.0, 145 | 14.000000000000002, 146 | 16.0, 147 | 19.0, 148 | 19.0, 149 | 21.0, 150 | 22.0, 151 | 22.0, 152 | 23.0, 153 | 23.0, 154 | 23.0, 155 | 26.0, 156 | 28.000000000000004, 157 | 28.999999999999996, 158 | 31.0, 159 | 31.0, 160 | 33.0, 161 | 34.0, 162 | 34.0, 163 | 34.0, 164 | 34.0, 165 | 36.0, 166 | 36.0, 167 | 37.0, 168 | 38.0, 169 | 41.0, 170 | 42.0, 171 | 44.0, 172 | 45.0, 173 | 46.0, 174 | 46.0, 175 | 47.0, 176 | 48.0, 177 | 51.0, 178 | 53.0, 179 | 55.00000000000001, 180 | 55.00000000000001, 181 | 56.99999999999999, 182 | 57.99999999999999, 183 | 61.0, 184 | 64.0, 185 | 67.0, 186 | 71.0, 187 | 71.0, 188 | 71.0, 189 | 74.0, 190 | 76.0, 191 | 79.0, 192 | 80.0, 193 | 81.0, 194 | 83.0, 195 | 86.0, 196 | 88.0, 197 | 89.0, 198 | 90.0, 199 | 90.0, 200 | 90.0, 201 | 92.0, 202 | 94.0, 203 | 95.0, 204 | 96.0, 205 | 96.0, 206 | 97.0, 207 | 97.0, 208 | 98.0, 209 | 98.0, 210 | 98.0, 211 | 99.0, 212 | 99.0, 213 | 99.0, 214 | 99.0, 215 | 100.0, 216 | 100.0, 217 | 100.0, 218 | 100.0, 219 | 100.0, 220 | 100.0, 221 | 100.0, 222 | 100.0, 223 | 100.0, 224 | 100.0, 225 | 100.0, 226 | 100.0, 227 | 100.0, 228 | 100.0, 229 | 100.0, 230 | 100.0, 231 | 100.0, 232 | 100.0, 233 | 100.0 234 | ], 235 | "optimization_success": false, 236 | "point": { 237 | "time": [ 238 | 1.422815, 239 | 0.662068, 240 | 2.17676 241 | ], 242 | "cost": [ 243 | 29.1651, 244 | 26.242, 245 | 33.9237 246 | ] 247 | } 248 | }, 249 | "geometric_PRM": { 250 | "success": [ 251 | 0.0, 252 | 0.0, 253 | 0.0, 254 | 0.0, 255 | 0.0, 256 | 0.0, 257 | 0.0, 258 | 0.0, 259 | 0.0, 260 | 0.0, 261 | 0.0, 262 | 0.0, 263 | 0.0, 264 | 0.0, 265 | 0.0, 266 | 0.0, 267 | 0.0, 268 | 0.0, 269 | 0.0, 270 | 0.0, 271 | 0.0, 272 | 0.0, 273 | 0.0, 274 | 0.0, 275 | 0.0, 276 | 0.0, 277 | 0.0, 278 | 0.0, 279 | 0.0, 280 | 0.0, 281 | 0.0, 282 | 0.0, 283 | 0.0, 284 | 0.0, 285 | 0.0, 286 | 0.0, 287 | 0.0, 288 | 0.0, 289 | 0.0, 290 | 0.0, 291 | 0.0, 292 | 0.0, 293 | 0.0, 294 | 0.0, 295 | 0.0, 296 | 0.0, 297 | 0.0, 298 | 0.0, 299 | 0.0, 300 | 0.0, 301 | 0.0, 302 | 0.0, 303 | 0.0, 304 | 0.0, 305 | 0.0, 306 | 0.0, 307 | 0.0, 308 | 0.0, 309 | 0.0, 310 | 0.0, 311 | 0.0, 312 | 0.0, 313 | 0.0, 314 | 0.0, 315 | 0.0, 316 | 0.0, 317 | 0.0, 318 | 1.0, 319 | 1.0, 320 | 1.0, 321 | 1.0, 322 | 1.0, 323 | 1.0, 324 | 1.0, 325 | 1.0, 326 | 1.0, 327 | 1.0, 328 | 1.0, 329 | 1.0, 330 | 1.0, 331 | 1.0, 332 | 1.0, 333 | 1.0, 334 | 1.0, 335 | 1.0, 336 | 1.0, 337 | 1.0, 338 | 1.0, 339 | 1.0, 340 | 1.0, 341 | 1.0, 342 | 1.0, 343 | 1.0, 344 | 1.0, 345 | 1.0, 346 | 1.0, 347 | 1.0, 348 | 1.0, 349 | 1.0, 350 | 1.0, 351 | 1.0, 352 | 1.0, 353 | 1.0, 354 | 1.0, 355 | 1.0, 356 | 1.0, 357 | 1.0, 358 | 3.0, 359 | 3.0, 360 | 3.0, 361 | 3.0, 362 | 3.0, 363 | 3.0, 364 | 5.0, 365 | 5.0, 366 | 5.0, 367 | 5.0, 368 | 5.0, 369 | 5.0, 370 | 5.0, 371 | 5.0, 372 | 5.0, 373 | 5.0, 374 | 6.0, 375 | 6.0, 376 | 6.0, 377 | 6.0, 378 | 6.0, 379 | 6.0, 380 | 6.0, 381 | 7.000000000000001, 382 | 7.000000000000001, 383 | 7.000000000000001, 384 | 7.000000000000001, 385 | 7.000000000000001, 386 | 7.000000000000001, 387 | 7.000000000000001, 388 | 7.000000000000001, 389 | 8.0, 390 | 8.0, 391 | 8.0, 392 | 8.0, 393 | 8.0, 394 | 8.0, 395 | 8.0, 396 | 8.0, 397 | 8.0, 398 | 8.0, 399 | 8.0, 400 | 8.0, 401 | 8.0, 402 | 8.0, 403 | 8.0, 404 | 8.0, 405 | 8.0, 406 | 8.0, 407 | 8.0, 408 | 8.0, 409 | 9.0, 410 | 9.0, 411 | 9.0, 412 | 9.0, 413 | 9.0, 414 | 9.0, 415 | 9.0, 416 | 9.0, 417 | 9.0, 418 | 9.0, 419 | 9.0, 420 | 9.0, 421 | 9.0, 422 | 9.0, 423 | 9.0, 424 | 9.0, 425 | 9.0, 426 | 9.0, 427 | 9.0, 428 | 9.0, 429 | 9.0, 430 | 9.0, 431 | 9.0, 432 | 9.0, 433 | 9.0, 434 | 9.0, 435 | 9.0, 436 | 9.0, 437 | 9.0, 438 | 9.0, 439 | 9.0, 440 | 9.0, 441 | 9.0, 442 | 9.0, 443 | 9.0, 444 | 9.0, 445 | 9.0, 446 | 9.0, 447 | 9.0, 448 | 9.0, 449 | 9.0, 450 | 9.0 451 | ], 452 | "optimization_success": true, 453 | "median": [ 454 | 100.0, 455 | 100.0, 456 | 100.0, 457 | 100.0, 458 | 100.0, 459 | 100.0, 460 | 100.0, 461 | 100.0, 462 | 100.0, 463 | 100.0, 464 | 100.0, 465 | 100.0, 466 | 100.0, 467 | 100.0, 468 | 100.0, 469 | 100.0, 470 | 100.0, 471 | 100.0, 472 | 100.0, 473 | 100.0, 474 | 100.0, 475 | 100.0, 476 | 100.0, 477 | 100.0, 478 | 100.0, 479 | 100.0, 480 | 100.0, 481 | 100.0, 482 | 100.0, 483 | 100.0, 484 | 100.0, 485 | 100.0, 486 | 100.0, 487 | 100.0, 488 | 100.0, 489 | 100.0, 490 | 100.0, 491 | 100.0, 492 | 100.0, 493 | 100.0, 494 | 100.0, 495 | 100.0, 496 | 100.0, 497 | 100.0, 498 | 100.0, 499 | 100.0, 500 | 100.0, 501 | 100.0, 502 | 100.0, 503 | 100.0, 504 | 100.0, 505 | 100.0, 506 | 100.0, 507 | 100.0, 508 | 100.0, 509 | 100.0, 510 | 100.0, 511 | 100.0, 512 | 100.0, 513 | 100.0, 514 | 100.0, 515 | 100.0, 516 | 100.0, 517 | 100.0, 518 | 100.0, 519 | 100.0, 520 | 100.0, 521 | 100.0, 522 | 100.0, 523 | 100.0, 524 | 100.0, 525 | 100.0, 526 | 100.0, 527 | 100.0, 528 | 100.0, 529 | 100.0, 530 | 100.0, 531 | 100.0, 532 | 100.0, 533 | 100.0, 534 | 100.0, 535 | 100.0, 536 | 100.0, 537 | 100.0, 538 | 100.0, 539 | 100.0, 540 | 100.0, 541 | 100.0, 542 | 100.0, 543 | 100.0, 544 | 100.0, 545 | 100.0, 546 | 100.0, 547 | 100.0, 548 | 100.0, 549 | 100.0, 550 | 100.0, 551 | 100.0, 552 | 100.0, 553 | 100.0, 554 | 100.0, 555 | 100.0, 556 | 100.0, 557 | 100.0, 558 | 100.0, 559 | 100.0, 560 | 100.0, 561 | 100.0, 562 | 100.0, 563 | 100.0, 564 | 100.0, 565 | 100.0, 566 | 100.0, 567 | 100.0, 568 | 100.0, 569 | 100.0, 570 | 100.0, 571 | 100.0, 572 | 100.0, 573 | 100.0, 574 | 100.0, 575 | 100.0, 576 | 100.0, 577 | 100.0, 578 | 100.0, 579 | 100.0, 580 | 100.0, 581 | 100.0, 582 | 100.0, 583 | 100.0, 584 | 100.0, 585 | 100.0, 586 | 100.0, 587 | 100.0, 588 | 100.0, 589 | 100.0, 590 | 100.0, 591 | 100.0, 592 | 100.0, 593 | 100.0, 594 | 100.0, 595 | 100.0, 596 | 100.0, 597 | 100.0, 598 | 100.0, 599 | 100.0, 600 | 100.0, 601 | 100.0, 602 | 100.0, 603 | 100.0, 604 | 100.0, 605 | 100.0, 606 | 100.0, 607 | 100.0, 608 | 100.0, 609 | 100.0, 610 | 100.0, 611 | 100.0, 612 | 100.0, 613 | 100.0, 614 | 100.0, 615 | 100.0, 616 | 100.0, 617 | 100.0, 618 | 100.0, 619 | 100.0, 620 | 100.0, 621 | 100.0, 622 | 100.0, 623 | 100.0, 624 | 100.0, 625 | 100.0, 626 | 100.0, 627 | 100.0, 628 | 100.0, 629 | 100.0, 630 | 100.0, 631 | 100.0, 632 | 100.0, 633 | 100.0, 634 | 100.0, 635 | 100.0, 636 | 100.0, 637 | 100.0, 638 | 100.0, 639 | 100.0, 640 | 100.0, 641 | 100.0, 642 | 100.0, 643 | 100.0, 644 | 100.0, 645 | 100.0, 646 | 100.0, 647 | 100.0, 648 | 100.0, 649 | 100.0, 650 | 100.0, 651 | 100.0, 652 | 100.0, 653 | 100.0 654 | ], 655 | "quantile5": [ 656 | 100.0, 657 | 100.0, 658 | 100.0, 659 | 100.0, 660 | 100.0, 661 | 100.0, 662 | 100.0, 663 | 100.0, 664 | 100.0, 665 | 100.0, 666 | 100.0, 667 | 100.0, 668 | 100.0, 669 | 100.0, 670 | 100.0, 671 | 100.0, 672 | 100.0, 673 | 100.0, 674 | 100.0, 675 | 100.0, 676 | 100.0, 677 | 100.0, 678 | 100.0, 679 | 100.0, 680 | 100.0, 681 | 100.0, 682 | 100.0, 683 | 100.0, 684 | 100.0, 685 | 100.0, 686 | 100.0, 687 | 100.0, 688 | 100.0, 689 | 100.0, 690 | 100.0, 691 | 100.0, 692 | 100.0, 693 | 100.0, 694 | 100.0, 695 | 100.0, 696 | 100.0, 697 | 100.0, 698 | 100.0, 699 | 100.0, 700 | 100.0, 701 | 100.0, 702 | 100.0, 703 | 100.0, 704 | 100.0, 705 | 100.0, 706 | 100.0, 707 | 100.0, 708 | 100.0, 709 | 100.0, 710 | 100.0, 711 | 100.0, 712 | 100.0, 713 | 100.0, 714 | 100.0, 715 | 100.0, 716 | 100.0, 717 | 100.0, 718 | 100.0, 719 | 100.0, 720 | 100.0, 721 | 100.0, 722 | 100.0, 723 | 100.0, 724 | 100.0, 725 | 100.0, 726 | 100.0, 727 | 100.0, 728 | 100.0, 729 | 100.0, 730 | 100.0, 731 | 100.0, 732 | 100.0, 733 | 100.0, 734 | 100.0, 735 | 100.0, 736 | 100.0, 737 | 100.0, 738 | 100.0, 739 | 100.0, 740 | 100.0, 741 | 100.0, 742 | 100.0, 743 | 100.0, 744 | 100.0, 745 | 100.0, 746 | 100.0, 747 | 100.0, 748 | 100.0, 749 | 100.0, 750 | 100.0, 751 | 100.0, 752 | 100.0, 753 | 100.0, 754 | 100.0, 755 | 100.0, 756 | 100.0, 757 | 100.0, 758 | 100.0, 759 | 100.0, 760 | 100.0, 761 | 100.0, 762 | 100.0, 763 | 100.0, 764 | 100.0, 765 | 100.0, 766 | 100.0, 767 | 100.0, 768 | 100.0, 769 | 100.0, 770 | 100.0, 771 | 100.0, 772 | 100.0, 773 | 100.0, 774 | 100.0, 775 | 100.0, 776 | 100.0, 777 | 100.0, 778 | 100.0, 779 | 100.0, 780 | 100.0, 781 | 100.0, 782 | 100.0, 783 | 100.0, 784 | 100.0, 785 | 100.0, 786 | 100.0, 787 | 100.0, 788 | 100.0, 789 | 100.0, 790 | 100.0, 791 | 100.0, 792 | 100.0, 793 | 100.0, 794 | 100.0, 795 | 100.0, 796 | 100.0, 797 | 100.0, 798 | 100.0, 799 | 100.0, 800 | 100.0, 801 | 100.0, 802 | 100.0, 803 | 100.0, 804 | 100.0, 805 | 100.0, 806 | 100.0, 807 | 100.0, 808 | 100.0, 809 | 100.0, 810 | 100.0, 811 | 100.0, 812 | 100.0, 813 | 100.0, 814 | 100.0, 815 | 100.0, 816 | 100.0, 817 | 100.0, 818 | 100.0, 819 | 100.0, 820 | 100.0, 821 | 100.0, 822 | 100.0, 823 | 100.0, 824 | 100.0, 825 | 100.0, 826 | 100.0, 827 | 100.0, 828 | 100.0, 829 | 100.0, 830 | 100.0, 831 | 100.0, 832 | 100.0, 833 | 100.0, 834 | 100.0, 835 | 100.0, 836 | 100.0, 837 | 100.0, 838 | 100.0, 839 | 100.0, 840 | 100.0, 841 | 100.0, 842 | 100.0, 843 | 100.0, 844 | 100.0, 845 | 100.0, 846 | 100.0, 847 | 100.0, 848 | 100.0, 849 | 100.0, 850 | 100.0, 851 | 100.0, 852 | 100.0, 853 | 100.0, 854 | 100.0, 855 | 100.0 856 | ], 857 | "quantile95": [ 858 | 100.0, 859 | 100.0, 860 | 100.0, 861 | 100.0, 862 | 100.0, 863 | 100.0, 864 | 100.0, 865 | 100.0, 866 | 100.0, 867 | 100.0, 868 | 100.0, 869 | 100.0, 870 | 100.0, 871 | 100.0, 872 | 100.0, 873 | 100.0, 874 | 100.0, 875 | 100.0, 876 | 100.0, 877 | 100.0, 878 | 100.0, 879 | 100.0, 880 | 100.0, 881 | 100.0, 882 | 100.0, 883 | 100.0, 884 | 100.0, 885 | 100.0, 886 | 100.0, 887 | 100.0, 888 | 100.0, 889 | 100.0, 890 | 100.0, 891 | 100.0, 892 | 100.0, 893 | 100.0, 894 | 100.0, 895 | 100.0, 896 | 100.0, 897 | 100.0, 898 | 100.0, 899 | 100.0, 900 | 100.0, 901 | 100.0, 902 | 100.0, 903 | 100.0, 904 | 100.0, 905 | 100.0, 906 | 100.0, 907 | 100.0, 908 | 100.0, 909 | 100.0, 910 | 100.0, 911 | 100.0, 912 | 100.0, 913 | 100.0, 914 | 100.0, 915 | 100.0, 916 | 100.0, 917 | 100.0, 918 | 100.0, 919 | 100.0, 920 | 100.0, 921 | 100.0, 922 | 100.0, 923 | 100.0, 924 | 100.0, 925 | 100.0, 926 | 100.0, 927 | 100.0, 928 | 100.0, 929 | 100.0, 930 | 100.0, 931 | 100.0, 932 | 100.0, 933 | 100.0, 934 | 100.0, 935 | 100.0, 936 | 100.0, 937 | 100.0, 938 | 100.0, 939 | 100.0, 940 | 100.0, 941 | 100.0, 942 | 100.0, 943 | 100.0, 944 | 100.0, 945 | 100.0, 946 | 100.0, 947 | 100.0, 948 | 100.0, 949 | 100.0, 950 | 100.0, 951 | 100.0, 952 | 100.0, 953 | 100.0, 954 | 100.0, 955 | 100.0, 956 | 100.0, 957 | 100.0, 958 | 100.0, 959 | 100.0, 960 | 100.0, 961 | 100.0, 962 | 100.0, 963 | 100.0, 964 | 100.0, 965 | 100.0, 966 | 100.0, 967 | 100.0, 968 | 100.0, 969 | 100.0, 970 | 100.0, 971 | 100.0, 972 | 100.0, 973 | 100.0, 974 | 100.0, 975 | 100.0, 976 | 100.0, 977 | 100.0, 978 | 100.0, 979 | 100.0, 980 | 100.0, 981 | 100.0, 982 | 100.0, 983 | 100.0, 984 | 100.0, 985 | 100.0, 986 | 100.0, 987 | 100.0, 988 | 100.0, 989 | 100.0, 990 | 100.0, 991 | 100.0, 992 | 100.0, 993 | 100.0, 994 | 100.0, 995 | 100.0, 996 | 100.0, 997 | 100.0, 998 | 100.0, 999 | 100.0, 1000 | 100.0, 1001 | 100.0, 1002 | 100.0, 1003 | 100.0, 1004 | 100.0, 1005 | 100.0, 1006 | 100.0, 1007 | 100.0, 1008 | 100.0, 1009 | 100.0, 1010 | 100.0, 1011 | 100.0, 1012 | 100.0, 1013 | 100.0, 1014 | 100.0, 1015 | 100.0, 1016 | 100.0, 1017 | 100.0, 1018 | 100.0, 1019 | 100.0, 1020 | 100.0, 1021 | 100.0, 1022 | 100.0, 1023 | 100.0, 1024 | 100.0, 1025 | 100.0, 1026 | 100.0, 1027 | 100.0, 1028 | 100.0, 1029 | 100.0, 1030 | 100.0, 1031 | 100.0, 1032 | 100.0, 1033 | 100.0, 1034 | 100.0, 1035 | 100.0, 1036 | 100.0, 1037 | 100.0, 1038 | 100.0, 1039 | 100.0, 1040 | 100.0, 1041 | 100.0, 1042 | 100.0, 1043 | 100.0, 1044 | 100.0, 1045 | 100.0, 1046 | 100.0, 1047 | 100.0, 1048 | 100.0, 1049 | 100.0, 1050 | 100.0, 1051 | 100.0, 1052 | 100.0, 1053 | 100.0, 1054 | 100.0, 1055 | 100.0, 1056 | 100.0, 1057 | 100.0 1058 | ] 1059 | }, 1060 | "geometric_EST": { 1061 | "success": [ 1062 | 0.0, 1063 | 0.0, 1064 | 0.0, 1065 | 0.0, 1066 | 0.0, 1067 | 0.0, 1068 | 0.0, 1069 | 0.0, 1070 | 0.0, 1071 | 0.0, 1072 | 0.0, 1073 | 0.0, 1074 | 0.0, 1075 | 0.0, 1076 | 0.0, 1077 | 0.0, 1078 | 0.0, 1079 | 0.0, 1080 | 0.0, 1081 | 0.0, 1082 | 0.0, 1083 | 0.0, 1084 | 0.0, 1085 | 0.0, 1086 | 0.0, 1087 | 0.0, 1088 | 0.0, 1089 | 0.0, 1090 | 0.0, 1091 | 0.0, 1092 | 0.0, 1093 | 0.0, 1094 | 0.0, 1095 | 0.0, 1096 | 0.0, 1097 | 0.0, 1098 | 0.0, 1099 | 1.0, 1100 | 1.0, 1101 | 1.0, 1102 | 1.0, 1103 | 1.0, 1104 | 2.0, 1105 | 2.0, 1106 | 2.0, 1107 | 2.0, 1108 | 2.0, 1109 | 2.0, 1110 | 2.0, 1111 | 2.0, 1112 | 2.0, 1113 | 2.0, 1114 | 2.0, 1115 | 2.0, 1116 | 2.0, 1117 | 2.0, 1118 | 2.0, 1119 | 2.0, 1120 | 2.0, 1121 | 2.0, 1122 | 2.0, 1123 | 2.0, 1124 | 2.0, 1125 | 2.0, 1126 | 2.0, 1127 | 2.0, 1128 | 2.0, 1129 | 2.0, 1130 | 2.0, 1131 | 3.0, 1132 | 3.0, 1133 | 3.0, 1134 | 3.0, 1135 | 3.0, 1136 | 3.0, 1137 | 4.0, 1138 | 4.0, 1139 | 5.0, 1140 | 5.0, 1141 | 5.0, 1142 | 5.0, 1143 | 5.0, 1144 | 5.0, 1145 | 5.0, 1146 | 5.0, 1147 | 5.0, 1148 | 5.0, 1149 | 5.0, 1150 | 5.0, 1151 | 5.0, 1152 | 6.0, 1153 | 6.0, 1154 | 6.0, 1155 | 6.0, 1156 | 6.0, 1157 | 6.0, 1158 | 6.0, 1159 | 6.0, 1160 | 6.0, 1161 | 6.0, 1162 | 6.0, 1163 | 6.0, 1164 | 7.000000000000001, 1165 | 7.000000000000001, 1166 | 8.0, 1167 | 8.0, 1168 | 8.0, 1169 | 8.0, 1170 | 9.0, 1171 | 10.0, 1172 | 11.0, 1173 | 11.0, 1174 | 11.0, 1175 | 11.0, 1176 | 12.0, 1177 | 13.0, 1178 | 14.000000000000002, 1179 | 15.0, 1180 | 15.0, 1181 | 15.0, 1182 | 16.0, 1183 | 17.0, 1184 | 19.0, 1185 | 19.0, 1186 | 19.0, 1187 | 21.0, 1188 | 22.0, 1189 | 23.0, 1190 | 23.0, 1191 | 24.0, 1192 | 25.0, 1193 | 28.000000000000004, 1194 | 28.000000000000004, 1195 | 31.0, 1196 | 31.0, 1197 | 31.0, 1198 | 32.0, 1199 | 37.0, 1200 | 42.0, 1201 | 44.0, 1202 | 45.0, 1203 | 48.0, 1204 | 50.0, 1205 | 51.0, 1206 | 54.0, 1207 | 54.0, 1208 | 56.99999999999999, 1209 | 59.0, 1210 | 63.0, 1211 | 64.0, 1212 | 65.0, 1213 | 67.0, 1214 | 68.0, 1215 | 69.0, 1216 | 70.0, 1217 | 73.0, 1218 | 75.0, 1219 | 77.0, 1220 | 79.0, 1221 | 79.0, 1222 | 83.0, 1223 | 85.0, 1224 | 85.0, 1225 | 85.0, 1226 | 85.0, 1227 | 85.0, 1228 | 86.0, 1229 | 91.0, 1230 | 92.0, 1231 | 93.0, 1232 | 94.0, 1233 | 95.0, 1234 | 97.0, 1235 | 98.0, 1236 | 98.0, 1237 | 98.0, 1238 | 99.0, 1239 | 99.0, 1240 | 99.0, 1241 | 99.0, 1242 | 99.0, 1243 | 99.0, 1244 | 99.0, 1245 | 99.0, 1246 | 99.0, 1247 | 100.0, 1248 | 100.0, 1249 | 100.0, 1250 | 100.0, 1251 | 100.0, 1252 | 100.0, 1253 | 100.0, 1254 | 100.0, 1255 | 100.0, 1256 | 100.0, 1257 | 100.0, 1258 | 100.0, 1259 | 100.0, 1260 | 100.0, 1261 | 100.0 1262 | ], 1263 | "optimization_success": false, 1264 | "point": { 1265 | "time": [ 1266 | 1.38809, 1267 | 0.921583, 1268 | 2.24759 1269 | ], 1270 | "cost": [ 1271 | 32.286249999999995, 1272 | 26.844, 1273 | 38.662 1274 | ] 1275 | } 1276 | }, 1277 | "geometric_RRTstar": { 1278 | "success": [ 1279 | 0.0, 1280 | 0.0, 1281 | 0.0, 1282 | 0.0, 1283 | 0.0, 1284 | 0.0, 1285 | 0.0, 1286 | 0.0, 1287 | 0.0, 1288 | 0.0, 1289 | 0.0, 1290 | 0.0, 1291 | 0.0, 1292 | 0.0, 1293 | 0.0, 1294 | 0.0, 1295 | 0.0, 1296 | 0.0, 1297 | 0.0, 1298 | 0.0, 1299 | 0.0, 1300 | 0.0, 1301 | 0.0, 1302 | 0.0, 1303 | 0.0, 1304 | 0.0, 1305 | 0.0, 1306 | 0.0, 1307 | 0.0, 1308 | 0.0, 1309 | 0.0, 1310 | 0.0, 1311 | 0.0, 1312 | 0.0, 1313 | 0.0, 1314 | 0.0, 1315 | 0.0, 1316 | 0.0, 1317 | 0.0, 1318 | 0.0, 1319 | 0.0, 1320 | 0.0, 1321 | 0.0, 1322 | 0.0, 1323 | 0.0, 1324 | 0.0, 1325 | 0.0, 1326 | 0.0, 1327 | 0.0, 1328 | 0.0, 1329 | 0.0, 1330 | 0.0, 1331 | 0.0, 1332 | 0.0, 1333 | 0.0, 1334 | 0.0, 1335 | 0.0, 1336 | 0.0, 1337 | 0.0, 1338 | 0.0, 1339 | 0.0, 1340 | 0.0, 1341 | 0.0, 1342 | 0.0, 1343 | 0.0, 1344 | 0.0, 1345 | 0.0, 1346 | 0.0, 1347 | 0.0, 1348 | 0.0, 1349 | 0.0, 1350 | 0.0, 1351 | 0.0, 1352 | 0.0, 1353 | 0.0, 1354 | 0.0, 1355 | 0.0, 1356 | 0.0, 1357 | 0.0, 1358 | 0.0, 1359 | 0.0, 1360 | 0.0, 1361 | 0.0, 1362 | 0.0, 1363 | 0.0, 1364 | 0.0, 1365 | 0.0, 1366 | 0.0, 1367 | 0.0, 1368 | 0.0, 1369 | 0.0, 1370 | 0.0, 1371 | 0.0, 1372 | 0.0, 1373 | 0.0, 1374 | 0.0, 1375 | 0.0, 1376 | 0.0, 1377 | 0.0, 1378 | 1.0, 1379 | 1.0, 1380 | 1.0, 1381 | 1.0, 1382 | 1.0, 1383 | 1.0, 1384 | 1.0, 1385 | 1.0, 1386 | 3.0, 1387 | 3.0, 1388 | 3.0, 1389 | 3.0, 1390 | 3.0, 1391 | 3.0, 1392 | 6.0, 1393 | 6.0, 1394 | 6.0, 1395 | 6.0, 1396 | 6.0, 1397 | 7.000000000000001, 1398 | 7.000000000000001, 1399 | 7.000000000000001, 1400 | 7.000000000000001, 1401 | 7.000000000000001, 1402 | 9.0, 1403 | 9.0, 1404 | 9.0, 1405 | 9.0, 1406 | 9.0, 1407 | 9.0, 1408 | 9.0, 1409 | 10.0, 1410 | 10.0, 1411 | 10.0, 1412 | 12.0, 1413 | 12.0, 1414 | 12.0, 1415 | 14.000000000000002, 1416 | 14.000000000000002, 1417 | 15.0, 1418 | 15.0, 1419 | 15.0, 1420 | 17.0, 1421 | 17.0, 1422 | 20.0, 1423 | 20.0, 1424 | 22.0, 1425 | 22.0, 1426 | 24.0, 1427 | 27.0, 1428 | 27.0, 1429 | 28.999999999999996, 1430 | 28.999999999999996, 1431 | 30.0, 1432 | 34.0, 1433 | 34.0, 1434 | 36.0, 1435 | 38.0, 1436 | 40.0, 1437 | 43.0, 1438 | 43.0, 1439 | 45.0, 1440 | 47.0, 1441 | 47.0, 1442 | 49.0, 1443 | 49.0, 1444 | 52.0, 1445 | 56.99999999999999, 1446 | 60.0, 1447 | 66.0, 1448 | 66.0, 1449 | 67.0, 1450 | 69.0, 1451 | 70.0, 1452 | 71.0, 1453 | 72.0, 1454 | 75.0, 1455 | 78.0, 1456 | 78.0, 1457 | 82.0, 1458 | 82.0, 1459 | 85.0, 1460 | 86.0, 1461 | 89.0, 1462 | 93.0, 1463 | 93.0, 1464 | 93.0, 1465 | 93.0, 1466 | 95.0, 1467 | 95.0, 1468 | 95.0, 1469 | 97.0, 1470 | 98.0, 1471 | 98.0, 1472 | 98.0, 1473 | 98.0, 1474 | 98.0, 1475 | 100.0, 1476 | 100.0, 1477 | 100.0, 1478 | 100.0 1479 | ], 1480 | "optimization_success": true, 1481 | "median": [ 1482 | 100.0, 1483 | 100.0, 1484 | 100.0, 1485 | 100.0, 1486 | 100.0, 1487 | 100.0, 1488 | 100.0, 1489 | 100.0, 1490 | 100.0, 1491 | 100.0, 1492 | 100.0, 1493 | 100.0, 1494 | 100.0, 1495 | 100.0, 1496 | 100.0, 1497 | 100.0, 1498 | 100.0, 1499 | 100.0, 1500 | 100.0, 1501 | 100.0, 1502 | 100.0, 1503 | 100.0, 1504 | 100.0, 1505 | 100.0, 1506 | 100.0, 1507 | 100.0, 1508 | 100.0, 1509 | 100.0, 1510 | 100.0, 1511 | 100.0, 1512 | 100.0, 1513 | 100.0, 1514 | 100.0, 1515 | 100.0, 1516 | 100.0, 1517 | 100.0, 1518 | 100.0, 1519 | 100.0, 1520 | 100.0, 1521 | 100.0, 1522 | 100.0, 1523 | 100.0, 1524 | 100.0, 1525 | 100.0, 1526 | 100.0, 1527 | 100.0, 1528 | 100.0, 1529 | 100.0, 1530 | 100.0, 1531 | 100.0, 1532 | 100.0, 1533 | 100.0, 1534 | 100.0, 1535 | 100.0, 1536 | 100.0, 1537 | 100.0, 1538 | 100.0, 1539 | 100.0, 1540 | 100.0, 1541 | 100.0, 1542 | 100.0, 1543 | 100.0, 1544 | 100.0, 1545 | 100.0, 1546 | 100.0, 1547 | 100.0, 1548 | 100.0, 1549 | 100.0, 1550 | 100.0, 1551 | 100.0, 1552 | 100.0, 1553 | 100.0, 1554 | 100.0, 1555 | 100.0, 1556 | 100.0, 1557 | 100.0, 1558 | 100.0, 1559 | 100.0, 1560 | 100.0, 1561 | 100.0, 1562 | 100.0, 1563 | 100.0, 1564 | 100.0, 1565 | 100.0, 1566 | 100.0, 1567 | 100.0, 1568 | 100.0, 1569 | 100.0, 1570 | 100.0, 1571 | 100.0, 1572 | 100.0, 1573 | 100.0, 1574 | 100.0, 1575 | 100.0, 1576 | 100.0, 1577 | 100.0, 1578 | 100.0, 1579 | 100.0, 1580 | 100.0, 1581 | 100.0, 1582 | 100.0, 1583 | 100.0, 1584 | 100.0, 1585 | 100.0, 1586 | 100.0, 1587 | 100.0, 1588 | 100.0, 1589 | 100.0, 1590 | 100.0, 1591 | 100.0, 1592 | 100.0, 1593 | 100.0, 1594 | 100.0, 1595 | 100.0, 1596 | 100.0, 1597 | 100.0, 1598 | 100.0, 1599 | 100.0, 1600 | 100.0, 1601 | 100.0, 1602 | 100.0, 1603 | 100.0, 1604 | 100.0, 1605 | 100.0, 1606 | 100.0, 1607 | 100.0, 1608 | 100.0, 1609 | 100.0, 1610 | 100.0, 1611 | 100.0, 1612 | 100.0, 1613 | 100.0, 1614 | 100.0, 1615 | 100.0, 1616 | 100.0, 1617 | 100.0, 1618 | 100.0, 1619 | 100.0, 1620 | 100.0, 1621 | 100.0, 1622 | 100.0, 1623 | 100.0, 1624 | 100.0, 1625 | 100.0, 1626 | 100.0, 1627 | 100.0, 1628 | 100.0, 1629 | 100.0, 1630 | 100.0, 1631 | 100.0, 1632 | 100.0, 1633 | 100.0, 1634 | 100.0, 1635 | 100.0, 1636 | 100.0, 1637 | 100.0, 1638 | 100.0, 1639 | 100.0, 1640 | 100.0, 1641 | 100.0, 1642 | 100.0, 1643 | 100.0, 1644 | 100.0, 1645 | 100.0, 1646 | 100.0, 1647 | 19.196721, 1648 | 17.293014999999997, 1649 | 17.100723000000002, 1650 | 16.932947499999997, 1651 | 16.932947499999997, 1652 | 16.7679845, 1653 | 16.696946500000003, 1654 | 16.4475295, 1655 | 16.3959275, 1656 | 16.36564, 1657 | 16.209378, 1658 | 16.096, 1659 | 16.046536, 1660 | 15.9041185, 1661 | 15.8641015, 1662 | 15.831208499999999, 1663 | 15.827977, 1664 | 15.793631000000001, 1665 | 15.701402, 1666 | 15.6300295, 1667 | 15.6300295, 1668 | 15.5408085, 1669 | 15.5408085, 1670 | 15.452085, 1671 | 15.433697500000001, 1672 | 15.425607, 1673 | 15.356546, 1674 | 15.356546, 1675 | 15.3315245, 1676 | 15.3315245, 1677 | 15.320057, 1678 | 15.2761555, 1679 | 15.272939000000001, 1680 | 15.2693035, 1681 | 15.216835 1682 | ], 1683 | "quantile5": [ 1684 | 100.0, 1685 | 100.0, 1686 | 100.0, 1687 | 100.0, 1688 | 100.0, 1689 | 100.0, 1690 | 100.0, 1691 | 100.0, 1692 | 100.0, 1693 | 100.0, 1694 | 100.0, 1695 | 100.0, 1696 | 100.0, 1697 | 100.0, 1698 | 100.0, 1699 | 100.0, 1700 | 100.0, 1701 | 100.0, 1702 | 100.0, 1703 | 100.0, 1704 | 100.0, 1705 | 100.0, 1706 | 100.0, 1707 | 100.0, 1708 | 100.0, 1709 | 100.0, 1710 | 100.0, 1711 | 100.0, 1712 | 100.0, 1713 | 100.0, 1714 | 100.0, 1715 | 100.0, 1716 | 100.0, 1717 | 100.0, 1718 | 100.0, 1719 | 100.0, 1720 | 100.0, 1721 | 100.0, 1722 | 100.0, 1723 | 100.0, 1724 | 100.0, 1725 | 100.0, 1726 | 100.0, 1727 | 100.0, 1728 | 100.0, 1729 | 100.0, 1730 | 100.0, 1731 | 100.0, 1732 | 100.0, 1733 | 100.0, 1734 | 100.0, 1735 | 100.0, 1736 | 100.0, 1737 | 100.0, 1738 | 100.0, 1739 | 100.0, 1740 | 100.0, 1741 | 100.0, 1742 | 100.0, 1743 | 100.0, 1744 | 100.0, 1745 | 100.0, 1746 | 100.0, 1747 | 100.0, 1748 | 100.0, 1749 | 100.0, 1750 | 100.0, 1751 | 100.0, 1752 | 100.0, 1753 | 100.0, 1754 | 100.0, 1755 | 100.0, 1756 | 100.0, 1757 | 100.0, 1758 | 100.0, 1759 | 100.0, 1760 | 100.0, 1761 | 100.0, 1762 | 100.0, 1763 | 100.0, 1764 | 100.0, 1765 | 100.0, 1766 | 100.0, 1767 | 100.0, 1768 | 100.0, 1769 | 100.0, 1770 | 100.0, 1771 | 100.0, 1772 | 100.0, 1773 | 100.0, 1774 | 100.0, 1775 | 100.0, 1776 | 100.0, 1777 | 100.0, 1778 | 100.0, 1779 | 100.0, 1780 | 100.0, 1781 | 100.0, 1782 | 100.0, 1783 | 100.0, 1784 | 100.0, 1785 | 100.0, 1786 | 100.0, 1787 | 100.0, 1788 | 100.0, 1789 | 100.0, 1790 | 100.0, 1791 | 100.0, 1792 | 100.0, 1793 | 100.0, 1794 | 100.0, 1795 | 100.0, 1796 | 100.0, 1797 | 100.0, 1798 | 100.0, 1799 | 100.0, 1800 | 100.0, 1801 | 100.0, 1802 | 100.0, 1803 | 100.0, 1804 | 100.0, 1805 | 100.0, 1806 | 100.0, 1807 | 100.0, 1808 | 100.0, 1809 | 100.0, 1810 | 100.0, 1811 | 100.0, 1812 | 100.0, 1813 | 100.0, 1814 | 100.0, 1815 | 100.0, 1816 | 100.0, 1817 | 100.0, 1818 | 100.0, 1819 | 100.0, 1820 | 100.0, 1821 | 100.0, 1822 | 100.0, 1823 | 100.0, 1824 | 100.0, 1825 | 100.0, 1826 | 100.0, 1827 | 100.0, 1828 | 100.0, 1829 | 100.0, 1830 | 100.0, 1831 | 100.0, 1832 | 18.06849, 1833 | 18.06849, 1834 | 17.99651, 1835 | 17.99651, 1836 | 17.99651, 1837 | 17.323143, 1838 | 17.323143, 1839 | 17.132865, 1840 | 17.076028, 1841 | 16.944019, 1842 | 16.6279, 1843 | 16.6279, 1844 | 16.512208, 1845 | 16.415507, 1846 | 16.200774, 1847 | 16.200774, 1848 | 16.200774, 1849 | 16.103435, 1850 | 15.898722, 1851 | 15.891732, 1852 | 15.833488, 1853 | 15.833488, 1854 | 15.708504, 1855 | 15.573626, 1856 | 15.573626, 1857 | 15.569407, 1858 | 15.503879, 1859 | 15.453833, 1860 | 15.423684, 1861 | 15.376104, 1862 | 15.376104, 1863 | 15.36568, 1864 | 15.336399, 1865 | 15.326924, 1866 | 15.279683, 1867 | 15.211442, 1868 | 15.171892, 1869 | 15.147336, 1870 | 15.139071, 1871 | 15.136435, 1872 | 15.109974, 1873 | 15.071633, 1874 | 15.050263, 1875 | 14.980595, 1876 | 14.95171, 1877 | 14.944048, 1878 | 14.944048, 1879 | 14.944048, 1880 | 14.938075, 1881 | 14.938075, 1882 | 14.926381, 1883 | 14.911667 1884 | ], 1885 | "quantile95": [ 1886 | 100.0, 1887 | 100.0, 1888 | 100.0, 1889 | 100.0, 1890 | 100.0, 1891 | 100.0, 1892 | 100.0, 1893 | 100.0, 1894 | 100.0, 1895 | 100.0, 1896 | 100.0, 1897 | 100.0, 1898 | 100.0, 1899 | 100.0, 1900 | 100.0, 1901 | 100.0, 1902 | 100.0, 1903 | 100.0, 1904 | 100.0, 1905 | 100.0, 1906 | 100.0, 1907 | 100.0, 1908 | 100.0, 1909 | 100.0, 1910 | 100.0, 1911 | 100.0, 1912 | 100.0, 1913 | 100.0, 1914 | 100.0, 1915 | 100.0, 1916 | 100.0, 1917 | 100.0, 1918 | 100.0, 1919 | 100.0, 1920 | 100.0, 1921 | 100.0, 1922 | 100.0, 1923 | 100.0, 1924 | 100.0, 1925 | 100.0, 1926 | 100.0, 1927 | 100.0, 1928 | 100.0, 1929 | 100.0, 1930 | 100.0, 1931 | 100.0, 1932 | 100.0, 1933 | 100.0, 1934 | 100.0, 1935 | 100.0, 1936 | 100.0, 1937 | 100.0, 1938 | 100.0, 1939 | 100.0, 1940 | 100.0, 1941 | 100.0, 1942 | 100.0, 1943 | 100.0, 1944 | 100.0, 1945 | 100.0, 1946 | 100.0, 1947 | 100.0, 1948 | 100.0, 1949 | 100.0, 1950 | 100.0, 1951 | 100.0, 1952 | 100.0, 1953 | 100.0, 1954 | 100.0, 1955 | 100.0, 1956 | 100.0, 1957 | 100.0, 1958 | 100.0, 1959 | 100.0, 1960 | 100.0, 1961 | 100.0, 1962 | 100.0, 1963 | 100.0, 1964 | 100.0, 1965 | 100.0, 1966 | 100.0, 1967 | 100.0, 1968 | 100.0, 1969 | 100.0, 1970 | 100.0, 1971 | 100.0, 1972 | 100.0, 1973 | 100.0, 1974 | 100.0, 1975 | 100.0, 1976 | 100.0, 1977 | 100.0, 1978 | 100.0, 1979 | 100.0, 1980 | 100.0, 1981 | 100.0, 1982 | 100.0, 1983 | 100.0, 1984 | 100.0, 1985 | 100.0, 1986 | 100.0, 1987 | 100.0, 1988 | 100.0, 1989 | 100.0, 1990 | 100.0, 1991 | 100.0, 1992 | 100.0, 1993 | 100.0, 1994 | 100.0, 1995 | 100.0, 1996 | 100.0, 1997 | 100.0, 1998 | 100.0, 1999 | 100.0, 2000 | 100.0, 2001 | 100.0, 2002 | 100.0, 2003 | 100.0, 2004 | 100.0, 2005 | 100.0, 2006 | 100.0, 2007 | 100.0, 2008 | 100.0, 2009 | 100.0, 2010 | 100.0, 2011 | 100.0, 2012 | 100.0, 2013 | 100.0, 2014 | 100.0, 2015 | 100.0, 2016 | 100.0, 2017 | 100.0, 2018 | 100.0, 2019 | 100.0, 2020 | 100.0, 2021 | 100.0, 2022 | 100.0, 2023 | 100.0, 2024 | 100.0, 2025 | 100.0, 2026 | 100.0, 2027 | 100.0, 2028 | 100.0, 2029 | 100.0, 2030 | 100.0, 2031 | 100.0, 2032 | 100.0, 2033 | 100.0, 2034 | 100.0, 2035 | 100.0, 2036 | 100.0, 2037 | 100.0, 2038 | 100.0, 2039 | 100.0, 2040 | 100.0, 2041 | 100.0, 2042 | 100.0, 2043 | 100.0, 2044 | 100.0, 2045 | 100.0, 2046 | 100.0, 2047 | 100.0, 2048 | 100.0, 2049 | 100.0, 2050 | 100.0, 2051 | 100.0, 2052 | 100.0, 2053 | 100.0, 2054 | 100.0, 2055 | 100.0, 2056 | 100.0, 2057 | 100.0, 2058 | 100.0, 2059 | 100.0, 2060 | 100.0, 2061 | 18.902343, 2062 | 17.989602, 2063 | 17.929638, 2064 | 17.164118, 2065 | 16.979416, 2066 | 16.644896, 2067 | 16.630586, 2068 | 16.416397, 2069 | 16.342882, 2070 | 16.311183, 2071 | 16.220401, 2072 | 16.210095, 2073 | 16.161494, 2074 | 16.030165, 2075 | 15.997861, 2076 | 15.910659, 2077 | 15.803959, 2078 | 15.765157, 2079 | 15.755835, 2080 | 15.755835, 2081 | 15.706756, 2082 | 15.590362, 2083 | 15.477124, 2084 | 15.475306, 2085 | 15.453464 2086 | ] 2087 | }, 2088 | "geometric_kBITstar": { 2089 | "success": [ 2090 | 0.0, 2091 | 0.0, 2092 | 0.0, 2093 | 0.0, 2094 | 0.0, 2095 | 0.0, 2096 | 0.0, 2097 | 0.0, 2098 | 0.0, 2099 | 0.0, 2100 | 0.0, 2101 | 0.0, 2102 | 0.0, 2103 | 0.0, 2104 | 0.0, 2105 | 0.0, 2106 | 0.0, 2107 | 0.0, 2108 | 0.0, 2109 | 0.0, 2110 | 0.0, 2111 | 0.0, 2112 | 0.0, 2113 | 0.0, 2114 | 0.0, 2115 | 0.0, 2116 | 0.0, 2117 | 0.0, 2118 | 0.0, 2119 | 0.0, 2120 | 0.0, 2121 | 0.0, 2122 | 0.0, 2123 | 0.0, 2124 | 0.0, 2125 | 0.0, 2126 | 0.0, 2127 | 0.0, 2128 | 0.0, 2129 | 0.0, 2130 | 0.0, 2131 | 0.0, 2132 | 0.0, 2133 | 0.0, 2134 | 0.0, 2135 | 0.0, 2136 | 0.0, 2137 | 0.0, 2138 | 0.0, 2139 | 0.0, 2140 | 0.0, 2141 | 0.0, 2142 | 0.0, 2143 | 0.0, 2144 | 0.0, 2145 | 0.0, 2146 | 0.0, 2147 | 0.0, 2148 | 0.0, 2149 | 0.0, 2150 | 0.0, 2151 | 0.0, 2152 | 0.0, 2153 | 0.0, 2154 | 0.0, 2155 | 0.0, 2156 | 0.0, 2157 | 19.0, 2158 | 19.0, 2159 | 19.0, 2160 | 19.0, 2161 | 19.0, 2162 | 19.0, 2163 | 19.0, 2164 | 19.0, 2165 | 19.0, 2166 | 19.0, 2167 | 19.0, 2168 | 19.0, 2169 | 19.0, 2170 | 19.0, 2171 | 19.0, 2172 | 19.0, 2173 | 19.0, 2174 | 19.0, 2175 | 19.0, 2176 | 19.0, 2177 | 32.0, 2178 | 32.0, 2179 | 32.0, 2180 | 32.0, 2181 | 32.0, 2182 | 32.0, 2183 | 32.0, 2184 | 32.0, 2185 | 32.0, 2186 | 32.0, 2187 | 32.0, 2188 | 32.0, 2189 | 37.0, 2190 | 37.0, 2191 | 37.0, 2192 | 37.0, 2193 | 37.0, 2194 | 37.0, 2195 | 37.0, 2196 | 37.0, 2197 | 41.0, 2198 | 41.0, 2199 | 41.0, 2200 | 41.0, 2201 | 41.0, 2202 | 41.0, 2203 | 45.0, 2204 | 45.0, 2205 | 45.0, 2206 | 45.0, 2207 | 45.0, 2208 | 47.0, 2209 | 47.0, 2210 | 47.0, 2211 | 47.0, 2212 | 47.0, 2213 | 48.0, 2214 | 48.0, 2215 | 48.0, 2216 | 48.0, 2217 | 50.0, 2218 | 50.0, 2219 | 50.0, 2220 | 52.0, 2221 | 52.0, 2222 | 52.0, 2223 | 55.00000000000001, 2224 | 55.00000000000001, 2225 | 55.00000000000001, 2226 | 56.00000000000001, 2227 | 56.00000000000001, 2228 | 56.99999999999999, 2229 | 56.99999999999999, 2230 | 56.99999999999999, 2231 | 59.0, 2232 | 59.0, 2233 | 61.0, 2234 | 61.0, 2235 | 62.0, 2236 | 62.0, 2237 | 64.0, 2238 | 64.0, 2239 | 64.0, 2240 | 65.0, 2241 | 65.0, 2242 | 65.0, 2243 | 65.0, 2244 | 65.0, 2245 | 66.0, 2246 | 66.0, 2247 | 67.0, 2248 | 67.0, 2249 | 67.0, 2250 | 68.0, 2251 | 68.0, 2252 | 68.0, 2253 | 69.0, 2254 | 69.0, 2255 | 69.0, 2256 | 70.0, 2257 | 71.0, 2258 | 71.0, 2259 | 71.0, 2260 | 71.0, 2261 | 71.0, 2262 | 71.0, 2263 | 71.0, 2264 | 71.0, 2265 | 72.0, 2266 | 72.0, 2267 | 72.0, 2268 | 72.0, 2269 | 72.0, 2270 | 72.0, 2271 | 73.0, 2272 | 73.0, 2273 | 74.0, 2274 | 74.0, 2275 | 74.0, 2276 | 75.0, 2277 | 75.0, 2278 | 75.0, 2279 | 75.0, 2280 | 75.0, 2281 | 76.0, 2282 | 76.0, 2283 | 76.0, 2284 | 76.0, 2285 | 76.0, 2286 | 77.0, 2287 | 79.0, 2288 | 80.0, 2289 | 80.0 2290 | ], 2291 | "optimization_success": true, 2292 | "median": [ 2293 | 100.0, 2294 | 100.0, 2295 | 100.0, 2296 | 100.0, 2297 | 100.0, 2298 | 100.0, 2299 | 100.0, 2300 | 100.0, 2301 | 100.0, 2302 | 100.0, 2303 | 100.0, 2304 | 100.0, 2305 | 100.0, 2306 | 100.0, 2307 | 100.0, 2308 | 100.0, 2309 | 100.0, 2310 | 100.0, 2311 | 100.0, 2312 | 100.0, 2313 | 100.0, 2314 | 100.0, 2315 | 100.0, 2316 | 100.0, 2317 | 100.0, 2318 | 100.0, 2319 | 100.0, 2320 | 100.0, 2321 | 100.0, 2322 | 100.0, 2323 | 100.0, 2324 | 100.0, 2325 | 100.0, 2326 | 100.0, 2327 | 100.0, 2328 | 100.0, 2329 | 100.0, 2330 | 100.0, 2331 | 100.0, 2332 | 100.0, 2333 | 100.0, 2334 | 100.0, 2335 | 100.0, 2336 | 100.0, 2337 | 100.0, 2338 | 100.0, 2339 | 100.0, 2340 | 100.0, 2341 | 100.0, 2342 | 100.0, 2343 | 100.0, 2344 | 100.0, 2345 | 100.0, 2346 | 100.0, 2347 | 100.0, 2348 | 100.0, 2349 | 100.0, 2350 | 100.0, 2351 | 100.0, 2352 | 100.0, 2353 | 100.0, 2354 | 100.0, 2355 | 100.0, 2356 | 100.0, 2357 | 100.0, 2358 | 100.0, 2359 | 100.0, 2360 | 100.0, 2361 | 100.0, 2362 | 100.0, 2363 | 100.0, 2364 | 100.0, 2365 | 100.0, 2366 | 100.0, 2367 | 100.0, 2368 | 100.0, 2369 | 100.0, 2370 | 100.0, 2371 | 100.0, 2372 | 100.0, 2373 | 100.0, 2374 | 100.0, 2375 | 100.0, 2376 | 100.0, 2377 | 100.0, 2378 | 100.0, 2379 | 100.0, 2380 | 100.0, 2381 | 100.0, 2382 | 100.0, 2383 | 100.0, 2384 | 100.0, 2385 | 100.0, 2386 | 100.0, 2387 | 100.0, 2388 | 100.0, 2389 | 100.0, 2390 | 100.0, 2391 | 100.0, 2392 | 100.0, 2393 | 100.0, 2394 | 100.0, 2395 | 100.0, 2396 | 100.0, 2397 | 100.0, 2398 | 100.0, 2399 | 100.0, 2400 | 100.0, 2401 | 100.0, 2402 | 100.0, 2403 | 100.0, 2404 | 100.0, 2405 | 100.0, 2406 | 100.0, 2407 | 100.0, 2408 | 100.0, 2409 | 100.0, 2410 | 100.0, 2411 | 100.0, 2412 | 100.0, 2413 | 100.0, 2414 | 100.0, 2415 | 100.0, 2416 | 100.0, 2417 | 100.0, 2418 | 100.0, 2419 | 100.0, 2420 | 61.801500000000004, 2421 | 61.801500000000004, 2422 | 61.801500000000004, 2423 | 23.1456, 2424 | 23.1456, 2425 | 23.1456, 2426 | 21.6988, 2427 | 21.6988, 2428 | 21.6988, 2429 | 21.61725, 2430 | 21.61725, 2431 | 21.15805, 2432 | 21.15805, 2433 | 21.15805, 2434 | 20.6768, 2435 | 20.6768, 2436 | 20.38325, 2437 | 20.38325, 2438 | 20.15335, 2439 | 20.15335, 2440 | 20.1201, 2441 | 20.1201, 2442 | 20.1201, 2443 | 19.93895, 2444 | 19.93895, 2445 | 19.93895, 2446 | 19.93895, 2447 | 19.93895, 2448 | 19.93895, 2449 | 19.93895, 2450 | 19.8933, 2451 | 19.8933, 2452 | 19.8933, 2453 | 19.8933, 2454 | 19.8933, 2455 | 19.8933, 2456 | 19.8933, 2457 | 19.8933, 2458 | 19.8933, 2459 | 19.68775, 2460 | 19.4292, 2461 | 19.4292, 2462 | 19.4292, 2463 | 19.4292, 2464 | 19.4292, 2465 | 19.4292, 2466 | 19.4292, 2467 | 19.4292, 2468 | 19.4292, 2469 | 19.4292, 2470 | 19.4292, 2471 | 19.4292, 2472 | 19.4292, 2473 | 19.4292, 2474 | 19.228, 2475 | 19.228, 2476 | 19.1685, 2477 | 19.08815, 2478 | 19.08815, 2479 | 19.08815, 2480 | 19.08815, 2481 | 19.08815, 2482 | 19.08815, 2483 | 19.08815, 2484 | 19.0303, 2485 | 19.0303, 2486 | 19.0303, 2487 | 19.0303, 2488 | 19.0303, 2489 | 18.98545, 2490 | 18.92605, 2491 | 18.92605, 2492 | 18.92605 2493 | ], 2494 | "quantile5": [ 2495 | 100.0, 2496 | 100.0, 2497 | 100.0, 2498 | 100.0, 2499 | 100.0, 2500 | 100.0, 2501 | 100.0, 2502 | 100.0, 2503 | 100.0, 2504 | 100.0, 2505 | 100.0, 2506 | 100.0, 2507 | 100.0, 2508 | 100.0, 2509 | 100.0, 2510 | 100.0, 2511 | 100.0, 2512 | 100.0, 2513 | 100.0, 2514 | 100.0, 2515 | 100.0, 2516 | 100.0, 2517 | 100.0, 2518 | 100.0, 2519 | 100.0, 2520 | 100.0, 2521 | 100.0, 2522 | 100.0, 2523 | 100.0, 2524 | 100.0, 2525 | 100.0, 2526 | 100.0, 2527 | 100.0, 2528 | 100.0, 2529 | 100.0, 2530 | 100.0, 2531 | 100.0, 2532 | 100.0, 2533 | 100.0, 2534 | 100.0, 2535 | 100.0, 2536 | 100.0, 2537 | 100.0, 2538 | 100.0, 2539 | 100.0, 2540 | 100.0, 2541 | 100.0, 2542 | 100.0, 2543 | 100.0, 2544 | 100.0, 2545 | 100.0, 2546 | 100.0, 2547 | 100.0, 2548 | 100.0, 2549 | 100.0, 2550 | 100.0, 2551 | 100.0, 2552 | 100.0, 2553 | 100.0, 2554 | 100.0, 2555 | 100.0, 2556 | 100.0, 2557 | 100.0, 2558 | 100.0, 2559 | 100.0, 2560 | 100.0, 2561 | 100.0, 2562 | 100.0, 2563 | 100.0, 2564 | 100.0, 2565 | 100.0, 2566 | 100.0, 2567 | 100.0, 2568 | 100.0, 2569 | 100.0, 2570 | 100.0, 2571 | 100.0, 2572 | 100.0, 2573 | 100.0, 2574 | 100.0, 2575 | 100.0, 2576 | 100.0, 2577 | 100.0, 2578 | 100.0, 2579 | 100.0, 2580 | 100.0, 2581 | 100.0, 2582 | 20.3373, 2583 | 20.3373, 2584 | 20.3373, 2585 | 20.3373, 2586 | 20.3373, 2587 | 20.3373, 2588 | 20.3373, 2589 | 20.3373, 2590 | 20.3373, 2591 | 20.3373, 2592 | 20.3373, 2593 | 20.3373, 2594 | 19.4015, 2595 | 19.4015, 2596 | 19.4015, 2597 | 19.4015, 2598 | 19.4015, 2599 | 19.4015, 2600 | 19.4015, 2601 | 19.4015, 2602 | 19.3212, 2603 | 19.3212, 2604 | 19.3212, 2605 | 19.3212, 2606 | 19.3212, 2607 | 19.3212, 2608 | 18.5959, 2609 | 18.5959, 2610 | 18.5959, 2611 | 18.5959, 2612 | 18.5959, 2613 | 18.4048, 2614 | 18.4048, 2615 | 18.4048, 2616 | 18.4048, 2617 | 18.4048, 2618 | 18.3533, 2619 | 18.3533, 2620 | 18.3533, 2621 | 18.3533, 2622 | 18.3533, 2623 | 18.3533, 2624 | 18.3533, 2625 | 18.0442, 2626 | 18.0442, 2627 | 18.0442, 2628 | 18.0442, 2629 | 18.0442, 2630 | 18.0442, 2631 | 17.9611, 2632 | 17.9611, 2633 | 17.9611, 2634 | 17.9611, 2635 | 17.9611, 2636 | 17.9611, 2637 | 17.9611, 2638 | 17.9611, 2639 | 17.9611, 2640 | 17.9611, 2641 | 17.9611, 2642 | 17.9611, 2643 | 17.9611, 2644 | 17.9611, 2645 | 17.9611, 2646 | 17.9611, 2647 | 17.9185, 2648 | 17.9185, 2649 | 17.9185, 2650 | 17.9185, 2651 | 17.9185, 2652 | 17.9185, 2653 | 17.9185, 2654 | 17.9185, 2655 | 17.9185, 2656 | 17.9185, 2657 | 17.9185, 2658 | 17.9185, 2659 | 17.9185, 2660 | 17.9185, 2661 | 17.9059, 2662 | 17.9059, 2663 | 17.9049, 2664 | 17.9049, 2665 | 17.9049, 2666 | 17.9049, 2667 | 17.9049, 2668 | 17.9049, 2669 | 17.9049, 2670 | 17.9049, 2671 | 17.9049, 2672 | 17.9049, 2673 | 17.9049, 2674 | 17.9049, 2675 | 17.9049, 2676 | 17.9049, 2677 | 17.8599, 2678 | 17.8599, 2679 | 17.8599, 2680 | 17.8599, 2681 | 17.8599, 2682 | 17.8599, 2683 | 17.8599, 2684 | 17.8599, 2685 | 17.8599, 2686 | 17.8599, 2687 | 17.8599, 2688 | 17.8599, 2689 | 17.8599, 2690 | 17.8599, 2691 | 17.8599, 2692 | 17.8599, 2693 | 17.7446, 2694 | 17.7446 2695 | ], 2696 | "quantile95": [ 2697 | 100.0, 2698 | 100.0, 2699 | 100.0, 2700 | 100.0, 2701 | 100.0, 2702 | 100.0, 2703 | 100.0, 2704 | 100.0, 2705 | 100.0, 2706 | 100.0, 2707 | 100.0, 2708 | 100.0, 2709 | 100.0, 2710 | 100.0, 2711 | 100.0, 2712 | 100.0, 2713 | 100.0, 2714 | 100.0, 2715 | 100.0, 2716 | 100.0, 2717 | 100.0, 2718 | 100.0, 2719 | 100.0, 2720 | 100.0, 2721 | 100.0, 2722 | 100.0, 2723 | 100.0, 2724 | 100.0, 2725 | 100.0, 2726 | 100.0, 2727 | 100.0, 2728 | 100.0, 2729 | 100.0, 2730 | 100.0, 2731 | 100.0, 2732 | 100.0, 2733 | 100.0, 2734 | 100.0, 2735 | 100.0, 2736 | 100.0, 2737 | 100.0, 2738 | 100.0, 2739 | 100.0, 2740 | 100.0, 2741 | 100.0, 2742 | 100.0, 2743 | 100.0, 2744 | 100.0, 2745 | 100.0, 2746 | 100.0, 2747 | 100.0, 2748 | 100.0, 2749 | 100.0, 2750 | 100.0, 2751 | 100.0, 2752 | 100.0, 2753 | 100.0, 2754 | 100.0, 2755 | 100.0, 2756 | 100.0, 2757 | 100.0, 2758 | 100.0, 2759 | 100.0, 2760 | 100.0, 2761 | 100.0, 2762 | 100.0, 2763 | 100.0, 2764 | 100.0, 2765 | 100.0, 2766 | 100.0, 2767 | 100.0, 2768 | 100.0, 2769 | 100.0, 2770 | 100.0, 2771 | 100.0, 2772 | 100.0, 2773 | 100.0, 2774 | 100.0, 2775 | 100.0, 2776 | 100.0, 2777 | 100.0, 2778 | 100.0, 2779 | 100.0, 2780 | 100.0, 2781 | 100.0, 2782 | 100.0, 2783 | 100.0, 2784 | 100.0, 2785 | 100.0, 2786 | 100.0, 2787 | 100.0, 2788 | 100.0, 2789 | 100.0, 2790 | 100.0, 2791 | 100.0, 2792 | 100.0, 2793 | 100.0, 2794 | 100.0, 2795 | 100.0, 2796 | 100.0, 2797 | 100.0, 2798 | 100.0, 2799 | 100.0, 2800 | 100.0, 2801 | 100.0, 2802 | 100.0, 2803 | 100.0, 2804 | 100.0, 2805 | 100.0, 2806 | 100.0, 2807 | 100.0, 2808 | 100.0, 2809 | 100.0, 2810 | 100.0, 2811 | 100.0, 2812 | 100.0, 2813 | 100.0, 2814 | 100.0, 2815 | 100.0, 2816 | 100.0, 2817 | 100.0, 2818 | 100.0, 2819 | 100.0, 2820 | 100.0, 2821 | 100.0, 2822 | 100.0, 2823 | 100.0, 2824 | 100.0, 2825 | 100.0, 2826 | 100.0, 2827 | 100.0, 2828 | 100.0, 2829 | 100.0, 2830 | 100.0, 2831 | 100.0, 2832 | 100.0, 2833 | 100.0, 2834 | 100.0, 2835 | 100.0, 2836 | 100.0, 2837 | 100.0, 2838 | 100.0, 2839 | 100.0, 2840 | 100.0, 2841 | 100.0, 2842 | 100.0, 2843 | 100.0, 2844 | 100.0, 2845 | 100.0, 2846 | 100.0, 2847 | 100.0, 2848 | 100.0, 2849 | 100.0, 2850 | 100.0, 2851 | 100.0, 2852 | 100.0, 2853 | 100.0, 2854 | 100.0, 2855 | 100.0, 2856 | 100.0, 2857 | 100.0, 2858 | 100.0, 2859 | 100.0, 2860 | 100.0, 2861 | 100.0, 2862 | 100.0, 2863 | 100.0, 2864 | 100.0, 2865 | 100.0, 2866 | 100.0, 2867 | 100.0, 2868 | 100.0, 2869 | 100.0, 2870 | 100.0, 2871 | 100.0, 2872 | 100.0, 2873 | 100.0, 2874 | 100.0, 2875 | 100.0, 2876 | 100.0, 2877 | 100.0, 2878 | 100.0, 2879 | 100.0, 2880 | 100.0, 2881 | 100.0, 2882 | 100.0, 2883 | 23.5719, 2884 | 23.5719, 2885 | 23.5719, 2886 | 23.5719, 2887 | 23.5719, 2888 | 22.4995, 2889 | 22.4995, 2890 | 22.4995, 2891 | 22.4995, 2892 | 22.4995, 2893 | 22.459, 2894 | 22.0838, 2895 | 21.6827, 2896 | 21.6827 2897 | ] 2898 | } 2899 | } 2900 | } -------------------------------------------------------------------------------- /tests/test_ompl_benchmark_plotter.py: -------------------------------------------------------------------------------- 1 | import pytest 2 | from src.database_info import * 3 | from src.database_to_graph import * 4 | from src.get_diverse_color import * 5 | from ompl_benchmark_plotter import * 6 | import os.path 7 | 8 | def test_load_simple_database(): 9 | database_filepath = "tests/data/simple.db" 10 | assert os.path.isfile(database_filepath) 11 | 12 | connection = sqlite3.connect(database_filepath) 13 | assert connection is not None 14 | cursor = connection.cursor() 15 | assert cursor is not None 16 | 17 | ## Check if the database has the correct entries 18 | assert has_solution_length(cursor) 19 | assert has_best_cost(cursor) 20 | 21 | ## Check if we loaded the correct number of planners and experiments 22 | experiment_names = get_experiment_names_from_database(cursor) 23 | assert len(experiment_names) == 1 24 | planner_names = get_planner_names_from_database(cursor) 25 | assert len(planner_names) == 5 26 | 27 | ## Check specific entries are loaded correctly 28 | assert get_maxtime_from_database(cursor) == pytest.approx(10.0) 29 | assert get_longest_name_from_planners(planner_names) == "RRTConnect" 30 | 31 | def test_load_default_config(): 32 | data = load_config() 33 | assert data is not None 34 | assert data["max_cost"] > 0.0 35 | assert data["min_cost"] >= 0.0 36 | assert data["min_cost"] <= data["max_cost"] 37 | assert data.get("final_cost") is None 38 | 39 | def test_get_diverse_colors(): 40 | c1 = get_diverse_color("first") 41 | c2 = get_diverse_color("second") 42 | c3 = get_diverse_color("third") 43 | c4 = get_diverse_color("first") 44 | c5 = get_diverse_color("second") 45 | assert c1 == c4 46 | assert c1 != c2 47 | assert c1 != c3 48 | assert c2 == c5 49 | assert c2 != c4 50 | 51 | def test_colors_are_consistent_for_same_planner_set(): 52 | json_filepath = "tests/data/simple.json" 53 | with open(json_filepath, 'r') as jsonfile: 54 | data = json.load(jsonfile) 55 | 56 | init_planner_colors(data) 57 | 58 | planner_data = data["planners"] 59 | planner_names = list(planner_data.keys()) 60 | 61 | planner_to_color_unsorted = {} 62 | for planner in planner_names: 63 | planner_to_color_unsorted[planner] = get_diverse_color(planner) 64 | 65 | planner_names.sort() 66 | 67 | planner_to_color_sorted = {} 68 | for planner in planner_names: 69 | planner_to_color_sorted[planner] = get_diverse_color(planner) 70 | 71 | for planner in planner_names: 72 | print(planner, planner_to_color_sorted[planner], planner_to_color_unsorted[planner]) 73 | assert planner_to_color_sorted[planner] == planner_to_color_unsorted[planner] 74 | 75 | def test_remove_nonoptimal_planners(): 76 | database_filepath = "tests/data/example.db" 77 | connection = sqlite3.connect(database_filepath) 78 | assert connection is not None 79 | cursor = connection.cursor() 80 | assert cursor is not None 81 | 82 | planners = cursor.execute("SELECT id, name, settings FROM {}".format('plannerConfigs')).fetchall() 83 | assert len(planners) == 8 84 | planners_optimal = remove_non_optimal_planner(cursor, planners) 85 | assert len(planners_optimal) == 3 86 | 87 | def test_remove_nonoptimal_planners_simple(): 88 | database_filepath = "tests/data/simple.db" 89 | connection = sqlite3.connect(database_filepath) 90 | assert connection is not None 91 | cursor = connection.cursor() 92 | assert cursor is not None 93 | 94 | planners = cursor.execute("SELECT id, name FROM {}".format('plannerConfigs')).fetchall() 95 | assert len(planners) == 5 96 | planners_optimal = remove_non_optimal_planner(cursor, planners) 97 | assert len(planners_optimal) == 2 98 | 99 | def test_raise_exception_on_non_existing_file(): 100 | with pytest.raises(Exception): 101 | run_benchmark_plotter(["tests/data/UnKnOwN.db", "-q"]) 102 | 103 | def test_raise_exception_on_non_database_file(): 104 | non_db_file = "tests/data/UnKnOwN.txt" 105 | assert not os.path.isfile(non_db_file) 106 | file = open(non_db_file, "w+") 107 | assert os.path.isfile(non_db_file) 108 | with pytest.raises(Exception): 109 | run_benchmark_plotter([non_db_file, "-q"]) 110 | os.remove(non_db_file) 111 | assert not os.path.isfile(non_db_file) 112 | 113 | def test_raise_exception_on_empty_database_file(): 114 | non_db_file = "tests/data/UnKnOwN.db" 115 | assert not os.path.isfile(non_db_file) 116 | file = open(non_db_file, "w+") 117 | assert os.path.isfile(non_db_file) 118 | with pytest.raises(Exception): 119 | run_benchmark_plotter([non_db_file, "-q"]) 120 | os.remove(non_db_file) 121 | assert not os.path.isfile(non_db_file) 122 | 123 | def test_verify_graph_plotting_from_simple_database(): 124 | pdffile = "tests/data/simple.pdf" 125 | assert not os.path.isfile(pdffile) 126 | assert run_benchmark_plotter(["tests/data/simple.db", "-q"]) == 0 127 | assert os.path.isfile(pdffile) 128 | os.remove(pdffile) 129 | assert not os.path.isfile(pdffile) 130 | 131 | def test_verify_graph_plotting_from_example_database(): 132 | pdffile = "tests/data/example.pdf" 133 | assert not os.path.isfile(pdffile) 134 | assert run_benchmark_plotter(["tests/data/example.db", "-q"]) == 0 135 | assert os.path.isfile(pdffile) 136 | os.remove(pdffile) 137 | assert not os.path.isfile(pdffile) 138 | 139 | --------------------------------------------------------------------------------