4 |
5 |
2 |
3 |
4 |
5 |
Experiment design, deployment, and optimization
7 | 8 | 9 | [](http://www.apache.org/licenses/LICENSE-2.0.html) 10 | 11 | 12 | EXP is a python experiment management toolset created to simplify two simple use cases: design and deploy experiments in the form of python modules/files. 13 | 14 | An experiment is a series of runs of a given configurable module for a specified set of parameters. This tool covers one of the most prevalent experiment deployment scenarios: testing a set of parameters in parallel in a local machine or homogeneous cluster. EXP also supports [global optimization](https://www.cs.ox.ac.uk/people/nando.defreitas/publications/BayesOptLoop.pdf) using **gaussian processes** or other surrogate models such as **random forests**. This can be used for instance as a tool for **hyperoparameter tuning** for machine learning models. 15 | 16 | ## Features 17 | * **parameter space design** based on configuration files ([TOML](https://github.com/toml-lang/toml) format); 18 | * **parallel experiment deployment** using ``multiprocessing`` processes; 19 | * **CUDA gpu workers** one parallel process per available GPUs: uses the variable [CUDA_VISIBLE_DEVICES](https://devblogs.nvidia.com/cuda-pro-tip-control-gpu-visibility-cuda_visible_devices); 20 | * **global optimization** from parameter spaces (e.g. for hyperparameter tunning) using [scikit-optimize](https://scikit-optimize.github.io/). 21 | 22 | ## 23 |
24 |
25 | ## Installation
26 | ``pip install exp``
27 |
28 | ``pipenv install exp`` with [pipenv](https://pipenv.readthedocs.io/en/latest/install/#pragmatic-installation-of-pipenv)
29 |
30 | ## Available CLI tools
31 | EXP provides two CLI modules:
32 | * exp.run: ``python -m exp.run -p basic.conf -m runnable.py --workers 10``
33 | * exp.gopt:``python -m exp.gopt -p basic.conf -m runnable.py --workers 4 -n 100 --plot``
34 |
35 | for more information check each commands help:
36 |
37 | ``python -m exp.run -h``
38 |
39 | ## Getting Started: Optimization
40 |
41 | ### 1. Runnable Module
42 | The first step is to create a module to use in our experiments. A basic configurable module ``runnable.py`` looks like this:
43 |
44 | ```python
45 | def run(x=1, **kwargs):
46 | return x ** 2
47 | ```
48 |
49 | This module computes the square of a parameter ``x``. Note that ``kwargs`` is included to capture other parameters that the experiment runner might use (even if they are not used by your module). Since run receives a dictionary, you could also define it as follows.
50 |
51 | ```python
52 | def run(**kwargs):
53 | x = kwargs.get('x',1)
54 | return x ** 2
55 | ```
56 |
57 | ### 2. Parameter Space Definition
58 | Next, we need a configuration file ``basic.conf`` were the parameters are specified:
59 | ```markdown
60 | [x]
61 | type = "range"
62 | bounds = [-10,10]
63 | ```
64 | This defines a parameter space with a single parameter ``x`` with values in the range ``[-10,10]``. For how to specify parameter spaces, see the [Parameter Space Specification](#parameter-space-specification).
65 |
66 | ### 3. Module Optimization
67 | Our simple module returns the ``x**2``, the optimizer tries to find the minimum value of this function based on the parameter space given by the configuration file. In this case, the optimizer will look at values of ``x`` between ``[-10,10]`` and try to find the minimum value.
68 |
69 | ```bash
70 | python -m exp.gopt --params basic.conf --module runnable.py --n 20 --workers 4
71 | ```
72 |
73 |
74 | 
75 |
88 | 
89 |
214 |
215 | ## Licence
216 |
217 | [Apache License 2.0](LICENSE)
218 |
--------------------------------------------------------------------------------
/exp/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/davidenunes/exp/091dba3b7ef60b463c9be7ad2e9c1458f06a1e38/exp/__init__.py
--------------------------------------------------------------------------------
/exp/__version__.py:
--------------------------------------------------------------------------------
1 | __version__ = "0.5.1"
2 |
--------------------------------------------------------------------------------
/exp/args.py:
--------------------------------------------------------------------------------
1 | """ args is a simple module to allow for quick
2 | specification of function parameters with default arguments,
3 | option restriction and type conversion based on type classes
4 |
5 | Param: namedtuple for specification parameters from tuples
6 | Namespace: simple object that maps param["param"] parameter
7 | dictionaries to namespace.param access.
8 | ParamDict: stores Param instances with default values
9 | """
10 | from collections import namedtuple
11 |
12 |
13 | class Param(namedtuple('Param', ["typefn", "value", "options"])):
14 | """ Simple class representing parameters
15 |
16 | with values, validated/restricted by options
17 | and be converted from other values using a type function
18 | used to read parameter tuples that serve as entries to :obj:`ParamDict`
19 |
20 | """
21 |
22 | def __new__(cls, typefn, value=None, options=None):
23 | if options is not None:
24 | options = set(options)
25 | typefn = convert_type(typefn)
26 |
27 | if value is not None:
28 | value = typefn(value)
29 |
30 | if options is not None and value is not None:
31 | if value not in options:
32 | raise ValueError("Invalid Param Value: {} not in options {}".format(value, options))
33 |
34 | return super().__new__(cls, typefn, value, options)
35 |
36 |
37 | class Namespace(object):
38 | def __init__(self, dict_attr):
39 | if not isinstance(dict_attr, dict):
40 | raise TypeError("Namespace requires a dict, {} found".format(dict_attr))
41 |
42 | for k, v in dict_attr.items():
43 | self.__setattr__(k, v)
44 |
45 | def __str__(self):
46 | attr = ["{}={}".format(k, v) for k, v in self.__dict__.items()]
47 |
48 | return "Namespace({s})".format(s=','.join(attr))
49 |
50 | def __setattr__(self, key, value):
51 | super().__setattr__(key, value)
52 |
53 |
54 | class ParamDict(dict):
55 | """ Dictionary of parameters with default values
56 |
57 | Attributes:
58 | defaults = a dictionary :py:`str` -> :obj:`Param`
59 |
60 | """
61 |
62 | def __init__(self, defaults):
63 | self.defaults = dict()
64 | self.add_params(defaults)
65 |
66 | for arg in self.defaults:
67 | param = self.defaults[arg]
68 | dict.__setitem__(self, arg, param.value)
69 |
70 | # self.__dict__ .update(self)
71 |
72 | def __setitem__(self, key, val):
73 | if key in self.defaults:
74 | default = self.defaults[key]
75 | if val is None:
76 | val = default.value
77 | else:
78 | val = default.typefn(val)
79 | if default.options is not None:
80 | if val not in default.options:
81 | raise ValueError("Invalid Param Value: {} not in options {}".format(val, default.options))
82 |
83 | dict.__setitem__(self, key, val)
84 |
85 | def add_params(self, param_dict):
86 | """ Adds a set of parameter values from a given dictionary to the current values
87 | overwrites the default values for the parameters that already exist in the defaults
88 |
89 | Args:
90 | param_dict: a dictionary with param_name -> (type,vale,options) :obj:`Param`
91 | """
92 | for arg in param_dict:
93 | param = Param(*param_dict[arg])
94 | self.defaults[arg] = param
95 |
96 | def from_dict(self, args):
97 | for arg in args:
98 | self.__setitem__(arg, args[arg])
99 |
100 | def to_namespace(self):
101 | """ Converts the ParamDict to a :obj:`Namespace` object
102 | which allows you to access ``namespace.param1``
103 |
104 | Returns:
105 | a :obj:`Namespace` object with the current values of this parameter dictionary
106 | """
107 | return Namespace(self)
108 |
109 |
110 | def as_bool(v):
111 | """ Converts a given value to a boolean
112 |
113 | Args:
114 | v (int,str,bool): and integer, string, or boolean to be converted to boolean value.
115 | Returns:
116 | (bool): if the value is an int any value <= 0 returns False, else True
117 | if the value is a boolean simply forwards this value
118 | if the value is a string, ignores case and converts any (yes,true,t,y,1) to True
119 | and ('no', 'false', 'f', 'n', '0') to False
120 | """
121 | if v is None:
122 | return False
123 | elif isinstance(v, bool):
124 | return v
125 | elif isinstance(v, str):
126 | if v.lower() in ('yes', 'true', 't', 'y', '1'):
127 | return True
128 | elif v.lower() in ('no', 'false', 'f', 'n', '0'):
129 | return False
130 | else:
131 | raise TypeError('Boolean value expected.')
132 | elif isinstance(v, int) or isinstance(v, float):
133 | if v <= 0:
134 | return False
135 | else:
136 | return True
137 |
138 |
139 | def as_int(v):
140 | """ Converts a value to int by converting it first to float
141 |
142 | because calling ``int("2.3")`` raises a ValueError since 2.3 is
143 | not an integer. ``as_int("2.3")`` returns the same as ``int(2.3)``
144 |
145 | Args:
146 | v: a value convertible to numerical
147 |
148 | Returns:
149 | (int) the integer value of the given value
150 |
151 | """
152 | return int(float(v))
153 |
154 |
155 | def convert_type(type_class):
156 | """ Maps classes to convert functions present in this module
157 | Args:
158 | type_class: some class that can also be called to convert values into its types
159 |
160 |
161 | Returns:
162 | a type conversion function for the given class capable of converting more than literal values, for instance,
163 | requesting a type conversion class for boolean, returns a function capable of converting strings, or integers
164 | to a boolean value (see :obj:`as_bool`)
165 | """
166 | if type_class == bool:
167 | return as_bool
168 | elif type_class == int:
169 | return as_int
170 | else:
171 | return type_class
172 |
--------------------------------------------------------------------------------
/exp/gopt.py:
--------------------------------------------------------------------------------
1 | """ CLI for Hyper parameter optimisation
2 |
3 | prompt_toolkit run sequential model algorithmic optimisation
4 | based on gaussian processes, random forests etc.
5 | """
6 | import sys
7 |
8 | import GPUtil
9 | import click
10 | import csv
11 | import importlib
12 | import logging
13 | import multiprocessing as mp
14 | import os
15 | from matplotlib import pyplot as plt
16 | from multiprocessing import Event
17 | from multiprocessing import Queue, Process
18 | from multiprocessing.queues import Empty
19 | from skopt import Optimizer, plots
20 | from skopt.space import Integer, Real, Categorical
21 | from tqdm import tqdm
22 | from tqdm._utils import _term_move_up
23 | import traceback
24 | from toml import TomlDecodeError
25 | from exp.params import ParamSpace, DTypes, ParamDecodeError
26 |
27 |
28 | def params_to_skopt(param_space: ParamSpace):
29 | """ Converts a parameter space to a list of Dimention objects that can be used with
30 | a skopt Optimizer.
31 |
32 | A skopt Optimizer only receives 3 types of Dimensions: Categorical, Real, or Integer
33 | we convert parameters from our parameter space into one of those 3 types. Note that we only
34 | convert parameters that have either bounds or with a categorical domain with more than 1 value.
35 | If we have constant values in our parameter space, these don't need to be optimized anyway.
36 |
37 | Another function is provided to convert skopt output values back into a dictionary with
38 | a full configuration according to the parameter space (@see values_to_params).
39 |
40 | Args:
41 | param_space: a ParameterSpace where we can get the domain of each parameter
42 |
43 | Returns:
44 | a list of Dimension that can be passed to a skopt Optimizer
45 |
46 | """
47 | dimensions = []
48 | for param_name in param_space.param_names():
49 | domain_param = param_space.domain(param_name)
50 | domain = domain_param["domain"]
51 | dtype = DTypes.from_type(domain_param["dtype"])
52 | if len(domain) > 1:
53 | if dtype == DTypes.INT:
54 | low = min(domain)
55 | high = max(domain)
56 | dimensions.append(Integer(low, high, name=param_name))
57 | elif dtype == DTypes.FLOAT:
58 | low = min(domain)
59 | high = max(domain)
60 | prior = domain_param.get("prior", None)
61 | dimensions.append(
62 | Real(low, high, prior=prior, name=param_name))
63 | elif dtype == DTypes.CATEGORICAL:
64 | prior = domain_param.get("prior", None)
65 | dimensions.append(Categorical(
66 | domain, prior, transform="onehot", name=param_name))
67 | return dimensions
68 |
69 |
70 | def values_to_params(param_values, param_space):
71 | """
72 | We don't need to optimize parameters with a single value in their domain so we filter
73 | them out with params to skopt and convert back to configuration using this method
74 |
75 | Args:
76 | param_values: dict {param_name: value}
77 | param_space: rest of the parameter space used to complete this configuration
78 |
79 | """
80 | cfg = {}
81 | for param_name in param_space.param_names():
82 | if param_name in param_values:
83 | cfg[param_name] = param_values[param_name]
84 | else:
85 | # complete with value from parameter space
86 | value = param_space.get_param(param_name)
87 |
88 | if isinstance(value, (tuple, list)) and len(value) > 1:
89 | raise ValueError(
90 | "don't know how to complete a configuration that might have multiple values")
91 | else:
92 | if isinstance(value, (tuple, list)):
93 | value = value[0]
94 | cfg[param_name] = value
95 |
96 | return cfg
97 |
98 |
99 | def load_module(runnable_path):
100 | """ Loads a python file with the module to be evaluated.
101 |
102 | The module is a python file with a run function member. Each worker then
103 | passes each configuration it receives from a Queue to run as keyword arguments
104 |
105 | Args:
106 | runnable_path:
107 |
108 | Raises:
109 | TypeError: if the loaded module doesn't have a run function
110 |
111 | Returns:
112 | a reference to the newly loaded module so
113 |
114 | """
115 | runnable_path = os.path.abspath(runnable_path)
116 | spec = importlib.util.spec_from_file_location(
117 | "runnable", location=runnable_path)
118 | runnable = importlib.util.module_from_spec(spec)
119 | spec.loader.exec_module(runnable)
120 |
121 | try:
122 | getattr(runnable, "run")
123 | except AttributeError:
124 | raise TypeError(
125 | "module in {} does not contain a \"run\" method".format(runnable_path))
126 |
127 | return runnable
128 |
129 |
130 | def submit(n, optimizer: Optimizer, opt_param_names, current_configs, param_space: ParamSpace, queue: Queue):
131 | """ Generate and submit n new configurations to a queue.
132 |
133 | Asks the optimizer for n new values to explore, creates configurations for those points and puts them
134 | in the given queue.
135 |
136 | Args:
137 | n: the number of configurations to be generated
138 | optimizer: the optimiser object from skopt with the model used for the suggested points to explore
139 | opt_param_names: the names for the parameters using the same order of the dimensions in the optimizer
140 | current_configs: current list of configurations (updated with the newly generated ones)
141 | param_space: parameter space which we can use to convert optimizer points to fully specified configurations
142 | queue: que multiprocessing queue in which we put the new configurations
143 | """
144 | dims = opt_param_names
145 | xs = optimizer.ask(n_points=n)
146 | cfgs = [values_to_params(dict(zip(dims, x)), param_space) for x in xs]
147 | for i, c in enumerate(cfgs):
148 | c["id"] = i + len(current_configs)
149 | queue.put(c)
150 | current_configs += cfgs
151 |
152 |
153 | def worker(pid: int,
154 | module_path: str,
155 | config_queue: Queue,
156 | result_queue: Queue,
157 | error_queue: Queue,
158 | terminated: Event):
159 | """ Worker to be executed in its own process
160 |
161 | Args:
162 | module_path: path to model runnable that must be imported and runned on a given configuration
163 | terminated: each worker should have its own flag
164 | pid: (int) with worker id
165 | config_queue: configuration queue used to receive the parameters for this worker, each configuration is a task
166 | result_queue: queue where the worker deposits the results
167 |
168 | Returns:
169 | each time a new result is returned from calling the run function on the module, the worker puts this in to its
170 | result multiprocessing Queue in the form (worker_id, configuration_id, result)
171 |
172 | If an exception occurs during run(...) the worker puts that exception as the result into the queue instead
173 |
174 | """
175 | # ids should be 0, 1, 2, n where n is the maximum number of gpus available
176 | os.environ["CUDA_VISIBLE_DEVICES"] = str(pid)
177 | module = load_module(module_path)
178 |
179 | while not terminated.is_set():
180 | try:
181 | kwargs = config_queue.get(timeout=0.5)
182 | cfg_id = kwargs["id"]
183 | kwargs["pid"] = pid
184 | # e.g. model score
185 | result = module.run(**kwargs)
186 | result_queue.put((pid, cfg_id, result))
187 | except Empty:
188 | # nothing to do, check if it's time to terminate
189 | pass
190 | except Exception as e:
191 | terminated.set()
192 | error_queue.put((pid, cfg_id, traceback.format_exc()))
193 | result_queue.put((pid, cfg_id, e))
194 |
195 |
196 | def update_progress_kuma(progress):
197 | tqdm.write(_term_move_up() * 10) # move cursor up
198 | offset = " " * int(progress.n / progress.total * (progress.ncols - 40))
199 |
200 | tqdm.write(offset + ' _______________')
201 | tqdm.write(offset + ' | |')
202 | tqdm.write(offset + ' | KUMA-SAN IS |')
203 | tqdm.write(offset + ' | OPTIMIZING! |')
204 | tqdm.write(
205 | offset + ' | {:>3}/{:<3} |'.format(progress.n, progress.total))
206 | tqdm.write(offset + ' |＿＿＿_＿＿＿＿|')
207 | tqdm.write(offset + ' ( ) ( )||')
208 | tqdm.write(offset + ' ( •(ｴ)•)|| ')
209 | tqdm.write(offset + ' / づ')
210 |
211 |
212 | @click.command(help='optimizes the hyperparameters for a given function',
213 | context_settings=dict(help_option_names=['-h', '--help'])
214 | )
215 | @click.option('-p', '--params', required=True, type=click.Path(exists=True), help='path to parameter space file')
216 | @click.option('-m', '--module', required=True, type=click.Path(exists=True), help='path to python module file')
217 | @click.option('-w', '--workers', default=1, type=int, help="number of workers: limited to CPU core count or GPU "
218 | "count, cannot be <=0.")
219 | @click.option('-g', '--gpu', is_flag=True,
220 | help="bounds the number of workers to the number of available GPUs (not under load)."
221 | "Each process only sees a single GPU.")
222 | @click.option('-n', '--n', default=1, type=int, help='number of configuration runs')
223 | @click.option('--name', default="opt", type=str,
224 | help='optimization experiment name: used as prefix for some output files')
225 | @click.option('-s', '--surrogate', default="GP",
226 | type=click.Choice(["GP", "RF", "ET"], case_sensitive=False),
227 | help='surrogate model for global optimisation: '
228 | '(GP) gaussian process, '
229 | '(RF) random forest, or'
230 | '(ET) extra trees.')
231 | @click.option('-a', '--acquisition', default="EI",
232 | type=click.Choice(["LCB", "EI", "PI"], case_sensitive=False),
233 | help='acquisition function: '
234 | '(LCB) Lower Confidence Bound, '
235 | '(EI) Expected Improvement, or '
236 | '(PI) Probability of Improvement.')
237 | @click.option('--plot', is_flag=True, help='shows a convergence plot during the optimization process and saves it at'
238 | 'the current working dir')
239 | @click.option('-o', '--out', type=click.Path(), help="output directory for the results file. If plotting "
240 | "convergence, the plot is also saved in the first directory in the"
241 | "path.")
242 | @click.option('--sync/--async', default=False, help="--async (default) submission, means that we submit a new "
243 | "configuration to a worker for each new result the optimizer gets."
244 | "--sync mode makes the optimizer wait for all workers before "
245 | "submitting new configurations to all of them")
246 | @click.option('--kappa', type=float, default=1.96, help="(default=1.96) Used when the acquisition is LCB."
247 | "Controls how much of the variance in the "
248 | "predicted values should be taken into account. High values "
249 | "favour exploration vs exploitation")
250 | @click.option('--xi', type=float, default=0.01, help="(default=0.01) Used with EI acquisition: controls how much "
251 | "improvement we want over previous best.")
252 | @click.option('--kuma', is_flag=True, help='kuma-san will display the progress on your global optimization procedure.')
253 | def run(params, module, workers, gpu, n, surrogate, acquisition, name, plot, out, sync, kappa, xi, kuma):
254 | logger = logging.getLogger(__name__)
255 | handler = logging.FileHandler('{name}.log'.format(name=name), delay=True)
256 | handler.setLevel(logging.ERROR)
257 | formatter = logging.Formatter(
258 | '%(asctime)s - %(name)s - %(levelname)s - %(message)s')
259 | handler.setFormatter(formatter)
260 | logger.addHandler(handler)
261 |
262 | opt_results = None
263 | out_file = None
264 | try:
265 | if gpu:
266 | # detecting available gpus with load < 0.1
267 | gpu_ids = [g.id for g in GPUtil.getGPUs() if g.load < 0.2]
268 | num_workers = min(workers, len(gpu_ids))
269 |
270 | if num_workers <= 0:
271 | sys.exit(1)
272 | else:
273 | num_workers = min(workers, mp.cpu_count())
274 |
275 | logger.log(logging.DEBUG, "Spawning {} workers".format(num_workers))
276 | if num_workers <= 0:
277 | logger.log(logging.ERROR, "--workers cannot be 0")
278 | sys.exit(1)
279 |
280 | # prepare output file
281 | out_file_name = '{}_configurations.csv'.format(name)
282 | out = out_file_name if out is None else out
283 | if out is not None and os.path.isdir(out):
284 | out_file_path = os.path.join(out, out_file_name)
285 | else:
286 | out_file_path = out
287 |
288 | out_dir = os.path.abspath(os.path.join(out_file_path, os.pardir))
289 | out_file_path = os.path.join(out_dir, out_file_name)
290 |
291 | param_space = ParamSpace(params)
292 |
293 | dimensions = params_to_skopt(param_space)
294 | optimizer_dims = [d.name for d in dimensions]
295 | acquisition_kwargs = None
296 | if acquisition == "LCB":
297 | acquisition_kwargs = {'kappa': kappa}
298 | elif acquisition == "EI":
299 | acquisition_kwargs = {'xi': xi}
300 |
301 | optimizer = Optimizer(dimensions=dimensions,
302 | acq_func_kwargs=acquisition_kwargs,
303 | base_estimator=surrogate,
304 | acq_func=acquisition)
305 |
306 | out_file = open(out_file_path, 'w')
307 | out_writer = csv.DictWriter(
308 | out_file, fieldnames=param_space.param_names() + ["id", "evaluation"])
309 | out_writer.writeheader()
310 |
311 | # setup process pool and queues
312 | # manager = mp.Manager()
313 | config_queue = Queue()
314 | result_queue = Queue()
315 | error_queue = Queue()
316 |
317 | terminate_flags = [Event() for _ in range(num_workers)]
318 | processes = [
319 | Process(target=worker, args=(i, module, config_queue,
320 | result_queue, error_queue, terminate_flags[i]))
321 | for i in range(num_workers)]
322 |
323 | configs = []
324 | scores = {}
325 | # get initial points at random and submit one job per worker
326 | submit(num_workers, optimizer, optimizer_dims,
327 | configs, param_space, config_queue)
328 | # cfg_if: score
329 |
330 | num_completed = 0
331 | pending = len(configs)
332 | cancel = False
333 |
334 | for p in processes:
335 | p.daemon = True
336 | p.start()
337 |
338 | if plot:
339 | fig = plt.gcf()
340 | fig.show()
341 | fig.canvas.draw()
342 |
343 | progress_bar = tqdm(total=n, leave=True)
344 |
345 | if kuma:
346 | update_progress_kuma(progress_bar)
347 |
348 | while num_completed < n and not cancel:
349 | try:
350 | res = result_queue.get(timeout=1)
351 | pid, cfg_id, result = res
352 | if not isinstance(result, Exception):
353 | cfg = configs[cfg_id]
354 | # convert dictionary to x vector that optimizer takes
355 | x = [cfg[param] for param in optimizer_dims]
356 | # store scores for each config
357 | scores[cfg_id] = result
358 |
359 | out_row = dict(cfg)
360 | out_row["evaluation"] = result
361 | out_writer.writerow(out_row)
362 | # make sure we can see the results in the file as we run the optimizer
363 | out_file.flush()
364 | opt_results = optimizer.tell(x, result)
365 |
366 | num_completed += 1
367 | pending -= 1
368 |
369 | if plot:
370 | plots.plot_convergence(opt_results)
371 | fig.canvas.draw()
372 |
373 | # sync submission of jobs means we wait for all workers to finish
374 | if sync and pending == 0:
375 | if num_completed != n:
376 | num_submit = min(num_workers, n - num_completed)
377 | submit(num_submit, optimizer, optimizer_dims,
378 | configs, param_space, config_queue)
379 | pending = num_submit
380 | else:
381 | terminate_flags[pid].set()
382 |
383 | # async submission of jobs: as soon as we receive one result we submit the next
384 | if not sync:
385 | if (num_completed + pending) != n:
386 | submit(1, optimizer, optimizer_dims,
387 | configs, param_space, config_queue)
388 | pending += 1
389 | else:
390 | # signal the current worker for termination
391 | terminate_flags[pid].set()
392 |
393 | progress_bar.update()
394 | progress_bar.set_postfix(
395 | {"best solution ": opt_results["fun"]})
396 |
397 | if kuma:
398 | update_progress_kuma(progress_bar)
399 |
400 | else:
401 | _, cfg_id_err, err = error_queue.get()
402 | logger.error("configuration {} failed".format(cfg_id_err))
403 | logger.error(err)
404 |
405 | cancel = True
406 | except Empty:
407 | pass
408 |
409 | # try to wait for process termination
410 | for process in processes:
411 | process.join(timeout=0.5)
412 |
413 | if process.is_alive():
414 | process.terminate()
415 |
416 | progress_bar.close()
417 |
418 | except TomlDecodeError as e:
419 | logger.error(traceback.format_exc())
420 | print("\n\n[Invalid parameter file] TOML decode error:\n {}".format(
421 | e), file=sys.stderr)
422 | except ParamDecodeError as e:
423 | logger.error(traceback.format_exc())
424 | print("\n\n[Invalid parameter file]\n {}".format(e), file=sys.stderr)
425 | except Exception as e:
426 | logger.error(traceback.format_exc())
427 | raise e
428 | except KeyboardInterrupt:
429 | pass
430 | finally:
431 | # debugging
432 | if opt_results is not None and plot:
433 | plt_file = '{}_convergence.pdf'.format(name)
434 | out_path = os.path.join(out_dir, plt_file)
435 | plt.savefig(out_path, bbox_inches='tight')
436 |
437 | if out_file is not None:
438 | out_file.close()
439 |
440 |
441 | if __name__ == '__main__':
442 | run()
443 |
--------------------------------------------------------------------------------
/exp/params.py:
--------------------------------------------------------------------------------
1 | """ Parameter Space definition writing and loading
2 |
3 | :obj:`ParamSpace` uses TOML as the underlying format to write and load
4 | configurations to and from
5 | """
6 | import csv
7 | import itertools
8 | import math
9 | import os
10 | from enum import Enum
11 |
12 | import numpy as np
13 |
14 | import toml
15 |
16 |
17 | def _repeat_it(iterable, n):
18 | return itertools.chain.from_iterable(itertools.repeat(x, n) for x in iterable)
19 |
20 |
21 | class ParamError(Exception):
22 | pass
23 |
24 |
25 | class ParamDecodeError(ParamError):
26 | pass
27 |
28 |
29 | class Types(Enum):
30 | """ Enum with valid parameter types supported by :obj:`ParamSpace`
31 |
32 | """
33 | VALUE = "value"
34 | LIST = "list"
35 | RANGE = "range"
36 | RANDOM = "random"
37 |
38 | @staticmethod
39 | def from_str(value, case_sensitive=False):
40 | if not case_sensitive:
41 | value.lower()
42 | if value == Types.VALUE.value:
43 | return Types.VALUE
44 | elif value == Types.LIST.value:
45 | return Types.LIST
46 | elif value == Types.RANGE.value:
47 | return Types.RANGE
48 | elif value == Types.RANDOM.value:
49 | return Types.RANDOM
50 | else:
51 | raise ValueError("invalid parameter type: {}\n"
52 | "supported values: {}".format(value,
53 | ",".join([t.name for t in Types])))
54 |
55 |
56 | class DTypes(Enum):
57 | """ Enum with valid parameter dtypes supported by :obj:`ParamSpace`
58 |
59 | Useful to convert parameter spaces into scikit-optimization Dimensions
60 | """
61 | INT = "int"
62 | FLOAT = "float"
63 | CATEGORICAL = "categorical"
64 |
65 | @staticmethod
66 | def from_type(dtype, case_sensitive=False):
67 | if not case_sensitive and isinstance(dtype, str):
68 | dtype.lower()
69 |
70 | if dtype in (DTypes.FLOAT.value, float):
71 | return DTypes.FLOAT
72 | elif dtype in (DTypes.INT.value, int):
73 | return DTypes.INT
74 | elif dtype == DTypes.CATEGORICAL.value:
75 | return DTypes.CATEGORICAL
76 | else:
77 | raise ValueError("invalid parameter dtype: {}\n"
78 | "supported values: {}".format(dtype,
79 | ",".join([t.name for t in DTypes])))
80 |
81 |
82 | class ParamSpace:
83 | """ ParamSpace
84 |
85 | Create parameter spaces from configuration files.
86 |
87 | ParamSpace creates and read from parameter configuration files
88 | to create parameter spaces used for hyperparameter grid search
89 | (:py:mod:`exp.run`) or Global optimization procedures (:py:mod:`exp.gopt`)
90 |
91 | Args:
92 | filename (str): [optional] path to a configuration file. If not specified, creates an empty ParamSpace.
93 | """
94 |
95 | def __init__(self, filename=None):
96 | if filename is not None:
97 | self.params = toml.load(filename)
98 | else:
99 | self.params = {}
100 | self.size = self._compute_size()
101 |
102 | def _compute_size(self):
103 | """ Returns the size of the parameter space in terms of number of
104 | unique configurations
105 |
106 | Returns:
107 | (int) number of unique configurations
108 |
109 | """
110 | params = self.params.keys()
111 | if len(params) > 0:
112 | size = 1
113 | else:
114 | return 0
115 |
116 | for param in params:
117 | param_type = Types.from_str(self.params[param]["type"])
118 | param_value = self.get_param(param, param_type)
119 |
120 | if param_type == Types.VALUE:
121 | param_value = [param_value]
122 | size *= len(param_value)
123 | return size
124 |
125 | def param_names(self):
126 | """ param_names.
127 |
128 | Returns:
129 | (list) list of strings with the names of the parameters in this space
130 |
131 | """
132 | return list(self.params.keys())
133 |
134 | def _update_space_size(self, n):
135 | """ Updates the static grid size each instance maintains so it's cheap to return it
136 |
137 | Args:
138 | n: number of values a new parameter being added has
139 | """
140 | if self.size == 0:
141 | self.size = n
142 | else:
143 | self.size *= n
144 |
145 | def _get_param(self, name, param_type):
146 | """ get parameter
147 |
148 | gets a parameter and checks if a parameter is of a given type
149 |
150 | Args:
151 | name: name of the parameter to be returned
152 | param_type: (Type) parameter type
153 |
154 | Raises:
155 | LookupError: if parameter is not found in parameter space
156 | TypeError: if parameter found is not of the type specified
157 |
158 | Returns:
159 | (dict) dictionary with the parameter value and configurations
160 |
161 | """
162 | if name not in self.params:
163 | raise LookupError("Parameter not found: {}".format(name))
164 |
165 | param = self.params[name]
166 | actual_type = Types.from_str(param['type'])
167 | if actual_type != param_type:
168 | raise TypeError("expected {param} to be a {expected} but got {actual}".format(param=name,
169 | expected=param_type,
170 | actual=actual_type))
171 |
172 | return param
173 |
174 | def add_random(self, name, low=0., high=1., prior="uniform", dtype=float, n=None, persist=False):
175 | """ Specify random params within bounds and a given prior distribution
176 |
177 | Args:
178 | name: name for the parameter
179 | low: lower bound for the distribution (optional)
180 | high: higher bound for the distribution (optional)
181 | prior: distribution to be used for the random sampling, one of the following values:
182 | -uniform
183 | -log-uniform
184 | n: number of random values to be sampled if persist
185 | persist:
186 | Returns:
187 |
188 | """
189 | if dtype not in (float, int):
190 | raise TypeError(
191 | """\n Unknown dtype "{}", valid dtypes are:\n \t-float, \n \t-int """.format(dtype))
192 |
193 | param = self.params[name] = {}
194 | param["type"] = Types.RANDOM.value
195 | param["dtype"] = dtype.__name__
196 | if n:
197 | param["n"] = n
198 | self._update_space_size(n)
199 | else:
200 | self._update_space_size(1)
201 |
202 | param["bounds"] = [low, high]
203 |
204 | if prior not in ("uniform", "log-uniform"):
205 | raise TypeError(
206 | """\n Unknown prior "{}", valid priors are:\n \t-"uniform", \n \t-"log-uniform" """.format(prior))
207 | param["prior"] = prior
208 |
209 | if persist:
210 | value = self.get_random(name)
211 | del self.params[name]
212 | self.add_list(name, value)
213 |
214 | def get_random(self, name):
215 | """ get random parameter
216 |
217 | Args:
218 | name: parameter name
219 |
220 | Returns:
221 | array with one or more random parameter values according to the prior distribution
222 | and the given bounds, if no bounds are found defaults to [0,1), if no priors
223 | are found uniform is used
224 |
225 | """
226 | param: dict = self._get_param(name, Types.RANDOM)
227 | n = param.get("n", 1)
228 | bounds = param.get("bounds", [0, 1])
229 | prior = param.get("prior", "uniform")
230 | dtype = param.get("dtype", "float")
231 |
232 | if prior == "uniform":
233 | if dtype == "float":
234 | rvs = np.random.uniform(low=bounds[0], high=bounds[1], size=n)
235 | else:
236 | rvs = np.random.randint(low=bounds[0], high=bounds[1], size=n)
237 | elif prior == "log-uniform":
238 | if dtype == "float":
239 | if bounds[0] == 0:
240 | raise ParamDecodeError(
241 | "Invalid bounds for parameter [{}] lower bound on a log space cannot be 0".format(name))
242 | low = np.log10(bounds[0])
243 | high = np.log10(bounds[1])
244 | rvs = np.power(10, np.random.uniform(low, high, size=n))
245 | else:
246 | raise ParamDecodeError(
247 | """\n Invalid prior value "{}" for parameter [{}] with dtype="float":
248 | random integer only supports "uniform" prior """.format(prior, name))
249 | else:
250 | raise ParamDecodeError(
251 | """\n Invalid prior value "{}" for parameter [{}], valid priors are:\n
252 | \t"uniform" \n
253 | \t"log-uniform" """.format(prior, name))
254 |
255 | return rvs
256 |
257 | def add_value(self, name, value):
258 | """ Create a single value parameter
259 |
260 | Args:
261 | name: parameter name
262 | value: value to be attributed to param
263 | """
264 | param = self.params[name] = {}
265 | param["type"] = Types.VALUE.value
266 | param["value"] = value
267 | self._update_space_size(1)
268 |
269 | def get_value(self, name):
270 | """ get single value parameter
271 |
272 | Args:
273 | name: parameter name
274 |
275 | Returns:
276 | obj some value for the single value parameter
277 |
278 | """
279 | param = self._get_param(name, Types.VALUE)
280 | return param["value"]
281 |
282 | def add_list(self, name, values):
283 | """ Create list parameter
284 |
285 | Args:
286 | name: parameter name
287 | values: list of values
288 | """
289 | values = list(values)
290 | param = self.params[name] = {}
291 | param["type"] = Types.LIST.value
292 | param["value"] = values
293 | self._update_space_size(len(values))
294 |
295 | def add_range(self, name, low=0, high=1, step=1, dtype=float):
296 | """ Creates range parameter
297 |
298 | Args:
299 | name: name for the parameter
300 | low: where the range starts
301 | high: where the range stops (not included in the values)
302 | step: distance between each point in the range
303 | dtype: float or int if int the points in the range are rounded
304 | """
305 | param = self.params[name] = {}
306 | param["type"] = Types.RANGE.value
307 | param["bounds"] = [low, high]
308 | param["step"] = step
309 | param["dtype"] = dtype.__name__
310 | self._update_space_size(math.ceil((high - low) / step))
311 |
312 | def get_range(self, name):
313 | """ get range value for given range parameter
314 |
315 | works like numpy.arange
316 |
317 | Args:
318 | name: parameter name associated with the range
319 |
320 | Returns:
321 | an array with n numbers according to the range specification
322 | """
323 | param = self._get_param(name, Types.RANGE)
324 | if "bounds" not in param:
325 | raise ParamDecodeError(""" "bounds" not found for parameter {}""".format(name))
326 | low = float(param["bounds"][0])
327 | # high = float(param["bounds"]["high"])
328 | high = float(param["bounds"][1])
329 |
330 | step = float(param.get("step", 1.0))
331 | if "dtype" not in param:
332 | dtype = float
333 | else:
334 | dtype = param["dtype"]
335 | if dtype not in ("int", "float"):
336 | raise ParamDecodeError("""invalid "dtype" for parameter "{}":
337 | expected int or float, got {}""".format(name, dtype))
338 |
339 | dtype = int if dtype == "int" else float
340 |
341 | return np.arange(low, high, step, dtype=dtype)
342 |
343 | def get_list(self, name, unique=False):
344 | """ get list parameter
345 |
346 | Args:
347 | name: parameter name
348 |
349 | Returns:
350 | a list with the parameter values
351 |
352 | """
353 | param = self._get_param(name, Types.LIST)
354 |
355 | # return list of unique items
356 | value = param["value"]
357 | if not unique:
358 | return value
359 | else:
360 | _, idx = np.unique(value, return_index=True)
361 | return np.array(value)[np.sort(idx)].tolist()
362 |
363 | def get_param(self, param, type=None):
364 | if param not in self.params:
365 | raise KeyError("Parameter {} not found".format(param))
366 |
367 | if "type" not in self.params[param]:
368 | raise ValueError("Parameter found but not specified properly: missing \"type\" property")
369 | actual_type = Types.from_str(self.params[param]["type"])
370 |
371 | if type is not None and actual_type != type:
372 | raise ValueError("Parameter {p} has type {ta}, you requested {t}".format(p=param, ta=actual_type, t=type))
373 |
374 | if actual_type == Types.RANGE:
375 | return self.get_range(param)
376 | elif actual_type == Types.VALUE:
377 | return self.get_value(param)
378 | elif actual_type == Types.LIST:
379 | return self.get_list(param)
380 | elif actual_type == Types.RANDOM:
381 | return self.get_random(param)
382 | else:
383 | raise TypeError("Unknown Parameter Type")
384 |
385 | def domain(self, param_name):
386 |
387 | param = self.params[param_name]
388 | param_type = Types.from_str(param["type"])
389 | prior = param.get("prior", None)
390 |
391 | if param_type == Types.LIST:
392 | return {"domain": self.get_list(param_name, unique=True),
393 | "dtype": DTypes.CATEGORICAL.value}
394 |
395 | elif param_type == Types.RANDOM:
396 | bounds = param.get("bounds", [0., 1.])
397 | dtype = param.get("dtype", DTypes.FLOAT.value)
398 |
399 | if prior is None:
400 | prior = "uniform"
401 | return {"domain": bounds, "dtype": dtype, "prior": prior}
402 |
403 | elif param_type == Types.RANGE:
404 | dtype = param.get("dtype", DTypes.FLOAT.value)
405 | bounds = param.get("bounds", [0., 1.])
406 | if prior is None:
407 | prior = "uniform"
408 | return {"domain": bounds, "dtype": dtype, "prior": prior}
409 |
410 | else:
411 | dtype = param.get("dtype", DTypes.CATEGORICAL.value)
412 | bounds = [self.get_param(param_name)]
413 | return {"domain": bounds, "dtype": dtype}
414 |
415 | def sample_param(self, name):
416 | """ draws a sample from a single parameter
417 |
418 | If this is a value it returns the value itself
419 | for a list or range draws one element uniformly at random
420 | for a random parameter, respects the distribution specified
421 |
422 | Args:
423 | name: parameter name to be sampled
424 |
425 | Returns:
426 | returns the sampled value for the given parameter name
427 |
428 | """
429 | if name in self.params:
430 | param = self.params[name]
431 | param_type = Types.from_str(param["type"])
432 | value = self.get_param(name, param_type)
433 | if param_type == Types.VALUE:
434 | return value
435 | elif param_type in (Types.LIST, Types.RANGE):
436 | randi = np.random.randint(0, len(value))
437 | return value[randi]
438 | elif param_type == Types.RANDOM:
439 | return value[0]
440 | else:
441 | raise KeyError("{} not in parameter space".format(name))
442 |
443 | def sample_space(self):
444 | """ Samples a configuration from the parameter space
445 |
446 | Returns:
447 | a dictionary with the sampled configuration
448 | """
449 | return {param: self.sample_param(param) for param in self.params.keys()}
450 |
451 | def param_grid(self, runs=1):
452 | """ Returns a generator of dictionaries with all the possible parameter combinations
453 | the keys are the parameter names the values are the current value for each parameter.
454 |
455 | Warnings:
456 | you shouldn't use id and run as parameter names, param grid automatically uses those names to identify each
457 | parameter combination.
458 |
459 | Args:
460 | runs(int): number of repeats for each unique configuration
461 | """
462 | if runs < 1:
463 | raise ValueError("runs must be >0: runs set to {}".format(runs))
464 |
465 | param_values = []
466 | params = list(self.params.keys())
467 |
468 | for param in self.params.keys():
469 | param_type = self.params[param]["type"]
470 | param_type = Types.from_str(param_type)
471 | param_value = self.get_param(param, param_type)
472 | if param_type == Types.VALUE:
473 | param_value = [param_value]
474 | param_values.append(param_value)
475 |
476 | if runs < 1:
477 | raise ValueError("runs must be >0: runs set to {}".format(runs))
478 | # add run to parameter names and run number to parameters
479 | params.append("run")
480 | run_ids = np.linspace(1, runs, runs, dtype=np.int32)
481 | param_values.append(run_ids)
482 |
483 | param_product = itertools.product(*param_values)
484 | param_product = (dict(zip(params, values)) for values in param_product)
485 |
486 | # create ids for each unique configuration
487 | ids = np.linspace(0, self.size - 1, self.size, dtype=np.int32)
488 | ids = list(_repeat_it(ids, runs))
489 | id_param_names = ["id"] * (self.size * runs)
490 |
491 | id_params = [{k: v} for k, v in zip(id_param_names, ids)]
492 |
493 | param_product = ({**p, **i} for i, p in zip(id_params, param_product))
494 | return param_product
495 |
496 | def write(self, filename):
497 | with open(filename, "w") as f:
498 | toml.dump(self.params, f)
499 |
500 | def write_configs(self, output_path="params.csv"):
501 | """ Writes a csv file with each line containing a configuration value with a unique id for each configuration
502 |
503 | Args:
504 | output_path: the output path for the summary file
505 | """
506 | summary_header = ["id", "run"]
507 | summary_header += self.param_names()
508 |
509 | with open(output_path, mode="w", newline='') as outfile:
510 | writer = csv.DictWriter(outfile, fieldnames=summary_header)
511 | writer.writeheader()
512 |
513 | param_grid = self.param_grid()
514 | for param_row in param_grid:
515 | writer.writerow(param_row)
516 |
517 | def write_config_files(self, output_path="params", file_prefix="params"):
518 | """ Writes one configuration file per unique configuration in the grid space
519 | Args:
520 | output_path:
521 | file_prefix:
522 | """
523 | param_grid = self.param_grid()
524 | conf_id = 0
525 | for current_config in param_grid:
526 | conf_file = "{prefix}_{id}.conf".format(prefix=file_prefix, id=conf_id)
527 | conf_file = os.path.join(output_path, conf_file)
528 |
529 | with open(conf_file, "w") as f:
530 | toml.dump(current_config, f)
531 |
532 | conf_id += 1
533 |
--------------------------------------------------------------------------------
/exp/run.py:
--------------------------------------------------------------------------------
1 | import sys
2 |
3 | import GPUtil
4 |
5 | import importlib
6 | import logging
7 | import multiprocessing as mp
8 | import os
9 | import traceback
10 | import click
11 | from toml import TomlDecodeError
12 | from tqdm import tqdm
13 | from multiprocessing import Queue, Event, Process
14 | from multiprocessing.queues import Empty as QueueEmpty
15 | from exp.params import ParamSpace, ParamDecodeError
16 |
17 |
18 | def load_module(runnable_path):
19 | """ Loads a python file with the module to be evaluated.
20 |
21 | The module is a python file with a run function member. Each worker then
22 | passes each configuration it receives from a Queue to run as keyword arguments
23 |
24 | Args:
25 | runnable_path:
26 |
27 | Raises:
28 | TypeError: if the loaded module doesn't have a run function
29 |
30 | Returns:
31 | a reference to the newly loaded module so
32 |
33 | """
34 | runnable_path = os.path.abspath(runnable_path)
35 | spec = importlib.util.spec_from_file_location("runnable", location=runnable_path)
36 | runnable = importlib.util.module_from_spec(spec)
37 | spec.loader.exec_module(runnable)
38 | try:
39 | getattr(runnable, "run")
40 | except AttributeError:
41 | raise TypeError("module in {} does not contain a \"run\" method".format(runnable_path))
42 |
43 | return runnable
44 |
45 |
46 | def worker(pid: int,
47 | module_path: str,
48 | config_queue: Queue,
49 | result_queue: Queue,
50 | error_queue: Queue,
51 | terminated: QueueEmpty,
52 | cancel):
53 | """ Worker to be executed in its own process
54 |
55 | Args:
56 | cancel: if true terminates when an error is encountered, otherwise keeps running
57 | error_queue: used to pass formatted stack traces to the main process
58 | module_path: path to model runnable that is imported. It's method run is called on a given configuration
59 | terminated: each worker should have its own flag
60 | pid: (int) with worker id
61 | config_queue: configuration queue used to receive the parameters for this worker, each configuration is a task
62 | result_queue: queue where the worker deposits the results
63 |
64 | Returns:
65 | each time a new result is returned from calling the run function on the module, the worker puts this in to its
66 | result multiprocessing Queue in the form (worker_id, configuration_id, result)
67 |
68 | If an exception occurs during run(...) the worker puts that exception as the result into the queue instead
69 |
70 | """
71 |
72 | os.environ["CUDA_VISIBLE_DEVICES"] = str(pid)
73 | module = load_module(module_path)
74 |
75 | while not terminated.is_set():
76 | try:
77 | kwargs = config_queue.get(timeout=0.5)
78 | cfg_id = kwargs["id"]
79 | kwargs["pid"] = pid
80 | result = module.run(**kwargs)
81 | result_queue.put((pid, cfg_id, result))
82 | except QueueEmpty:
83 | pass
84 | except Exception as e:
85 | if cancel:
86 | terminated.set()
87 | error_queue.put((pid, cfg_id, traceback.format_exc()))
88 | result_queue.put((pid, cfg_id, e))
89 |
90 |
91 | @click.command(help='runs all the configurations in a defined space')
92 | @click.option('-p', '--params', required=True, type=click.Path(exists=True), help='path to parameter space file')
93 | @click.option('-m', '--module', required=True, type=click.Path(exists=True), help='path to python module file')
94 | @click.option('-r', '--runs', default=1, type=int, help='number of configuration runs')
95 | @click.option('--name', default="exp", type=str, help='experiment name: used as prefix for some output files')
96 | @click.option('-w', '--workers', default=1, type=int, help="number of workers: limited to CPU core count or GPU "
97 | "count, cannot be <=0.")
98 | @click.option('-g', '--gpu', is_flag=True,
99 | help="bounds the number of workers to the number of available GPUs (not under load)."
100 | "Each process only sees a single GPU.")
101 | @click.option('-c', '--config-ids', type=int, multiple=True)
102 | @click.option('--cancel', is_flag=True, help="cancel all tasks if one fails")
103 | def main(params, module, runs, name, workers, gpu, config_ids, cancel):
104 | logger = logging.getLogger(__name__)
105 | handler = logging.FileHandler('{name}.log'.format(name=name), delay=True)
106 | handler.setLevel(logging.ERROR)
107 | formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
108 | handler.setFormatter(formatter)
109 | logger.addHandler(handler)
110 |
111 | try:
112 | if gpu:
113 | # detecting available gpus with load < 0.1
114 | worker_ids = [g.id for g in GPUtil.getGPUs() if g.load < 0.2]
115 | num_workers = min(workers, len(worker_ids))
116 |
117 | if num_workers <= 0:
118 | logger.log(logging.ERROR, "no gpus available")
119 | sys.exit(1)
120 | else:
121 | num_workers = min(workers, mp.cpu_count())
122 | if num_workers <= 0:
123 | logger.log(logging.ERROR, "--workers cannot be 0")
124 | sys.exit(1)
125 |
126 | ps = ParamSpace(filename=params)
127 | ps.write_configs('{}_params.csv'.format(name))
128 |
129 | param_grid = ps.param_grid(runs=runs)
130 | n_tasks = ps.size * runs
131 |
132 | if len(config_ids) > 0:
133 | n_tasks = len(config_ids) * runs
134 | param_grid = [p for p in param_grid if p["id"] in config_ids]
135 | param_grid = iter(param_grid)
136 |
137 | num_workers = min(n_tasks, num_workers)
138 |
139 | print("----------Parameter Space Runner------------")
140 | print(":: tasks: {}".format(n_tasks))
141 | print(":: workers: {}".format(num_workers))
142 | print("--------------------------------------------")
143 |
144 | config_queue = Queue()
145 | result_queue = Queue()
146 | error_queue = Queue()
147 | progress_bar = tqdm(total=n_tasks, leave=True)
148 |
149 | terminate_flags = [Event() for _ in range(num_workers)]
150 | processes = [
151 | Process(target=worker,
152 | args=(i, module, config_queue, result_queue, error_queue, terminate_flags[i], cancel))
153 | for i in range(num_workers)]
154 |
155 | scores = {}
156 | configs = {}
157 |
158 | # submit num worker jobs
159 | for _ in range(num_workers):
160 | next_cfg = next(param_grid)
161 | configs[next_cfg["id"]] = next_cfg
162 | config_queue.put(next_cfg)
163 |
164 | for p in processes:
165 | p.daemon = True
166 | p.start()
167 |
168 | num_completed = 0
169 | pending = num_workers
170 | done = False
171 | successful = set()
172 |
173 | while num_completed < n_tasks and not done:
174 | try:
175 | res = result_queue.get(timeout=1)
176 | pid, cfg_id, result = res
177 | if not isinstance(result, Exception):
178 | successful.add(cfg_id)
179 | # cfg = configs[cfg_id]
180 | scores[cfg_id] = result
181 | num_completed += 1
182 | pending -= 1
183 |
184 | if (num_completed + pending) != n_tasks:
185 | next_cfg = next(param_grid)
186 | configs[next_cfg["id"]] = next_cfg
187 | config_queue.put(next_cfg)
188 |
189 | pending += 1
190 | else:
191 | # signal the current worker for termination no more work to be done
192 | terminate_flags[pid].set()
193 |
194 | progress_bar.update()
195 | else:
196 | # retrieve one error from queue, might not be exactly the one that failed
197 | # since other worker can write to the queue, but we will have at least one error to retrieve
198 | _, cfg_id_err, err = error_queue.get()
199 | logger.error("configuration {} failed".format(cfg_id_err))
200 | logger.error(err)
201 |
202 | if cancel:
203 | done = True
204 | else:
205 | num_completed += 1
206 | pending -= 1
207 |
208 | if (num_completed + pending) != n_tasks:
209 | next_cfg = next(param_grid)
210 | configs[next_cfg["id"]] = next_cfg
211 | config_queue.put(next_cfg)
212 | pending += 1
213 | else:
214 | # signal the current worker for termination no more work to be done
215 | terminate_flags[pid].set()
216 | progress_bar.update()
217 |
218 | except QueueEmpty:
219 | pass
220 | # try to wait for process termination
221 | for process in processes:
222 | process.join(timeout=0.5)
223 |
224 | if process.is_alive():
225 | process.terminate()
226 |
227 | if len(config_ids) > 0:
228 | all_ids = set(config_ids)
229 | else:
230 | all_ids = set(range(ps.size))
231 | failed_tasks = all_ids.difference(successful)
232 | if len(failed_tasks) > 0:
233 | ids = " ".join(map(str, failed_tasks))
234 | fail_runs = "failed runs: {}".format(ids)
235 | print(fail_runs, file=sys.stderr)
236 | logger.warn(fail_runs)
237 |
238 | progress_bar.close()
239 |
240 | except TomlDecodeError as e:
241 | logger.error(traceback.format_exc())
242 | print("\n\n[Invalid parameter file] TOML decode error:\n {}".format(e), file=sys.stderr)
243 | except ParamDecodeError as e:
244 | logger.error(traceback.format_exc())
245 | print("\n\n[Invalid parameter file]\n {}".format(e), file=sys.stderr)
246 |
247 |
248 | if __name__ == '__main__':
249 | main()
250 |
--------------------------------------------------------------------------------
/extras/convergence.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/davidenunes/exp/091dba3b7ef60b463c9be7ad2e9c1458f06a1e38/extras/convergence.png
--------------------------------------------------------------------------------
/extras/exp.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/davidenunes/exp/091dba3b7ef60b463c9be7ad2e9c1458f06a1e38/extras/exp.png
--------------------------------------------------------------------------------
/extras/getting_started.gif:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/davidenunes/exp/091dba3b7ef60b463c9be7ad2e9c1458f06a1e38/extras/getting_started.gif
--------------------------------------------------------------------------------
/setup.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python
2 | import os
3 | from setuptools import find_packages, setup, Command
4 | import codecs
5 | import sys
6 | from shutil import rmtree
7 |
8 | here = os.path.abspath(os.path.dirname(__file__))
9 |
10 | about = {}
11 |
12 | with open(os.path.join(here, "exp", "__version__.py")) as f:
13 | exec(f.read(), about)
14 |
15 | with codecs.open(os.path.join(here, "README.md"), encoding="utf-8") as f:
16 | long_description = "\n" + f.read()
17 |
18 | if sys.argv[-1] == "publish":
19 | os.system("python setup.py sdist bdist_wheel upload")
20 | sys.exit()
21 |
22 | required = [
23 | 'toml',
24 | 'click',
25 | 'numpy',
26 | 'matplotlib',
27 | 'GPUtil',
28 | 'scikit-optimize'
29 | ]
30 |
31 |
32 | class UploadCommand(Command):
33 | """Support setup.py publish."""
34 |
35 | description = "Build and publish the package."
36 | user_options = []
37 |
38 | @staticmethod
39 | def status(s):
40 | """Prints things in bold."""
41 | print("\033[1m{0}\033[0m".format(s))
42 |
43 | def initialize_options(self):
44 | pass
45 |
46 | def finalize_options(self):
47 | pass
48 |
49 | def run(self):
50 | try:
51 | self.status("Removing previous builds…")
52 | rmtree(os.path.join(here, "dist"))
53 | except FileNotFoundError:
54 | pass
55 | self.status("Building Source distribution…")
56 | os.system("{0} setup.py sdist bdist_wheel".format(sys.executable))
57 | self.status("Uploading the package to PyPI via Twine…")
58 | os.system("twine upload dist/*")
59 | self.status("Pushing git tags…")
60 | os.system("git tag v{0}".format(about["__version__"]))
61 | os.system("git push --tags")
62 | sys.exit()
63 |
64 | setup(name='exp',
65 | version=about["__version__"],
66 | description='Python tool do design and run experiments with global optimisation and grid search',
67 | long_description=long_description,
68 | long_description_content_type="text/markdown",
69 | author='Davide Nunes',
70 | author_email='mail@davidenunes.com',
71 | url='https://github.com/davidenunes/exp',
72 | packages=find_packages(exclude=["tests", "tests.*"]),
73 | install_requires=required,
74 | python_requires=">=3.6",
75 | license="Apache 2.0",
76 | package_data={
77 | "": ["LICENSE"],
78 | },
79 | include_package_data=True,
80 | classifiers=[
81 | 'Programming Language :: Python',
82 | 'Programming Language :: Python :: 3.6',
83 | 'Programming Language :: Python :: 3.7',
84 | ],
85 | cmdclass={"upload": UploadCommand},
86 | )
87 |
--------------------------------------------------------------------------------
/test/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/davidenunes/exp/091dba3b7ef60b463c9be7ad2e9c1458f06a1e38/test/__init__.py
--------------------------------------------------------------------------------
/test/assets/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/davidenunes/exp/091dba3b7ef60b463c9be7ad2e9c1458f06a1e38/test/assets/__init__.py
--------------------------------------------------------------------------------
/test/assets/dummy.conf:
--------------------------------------------------------------------------------
1 | [x]
2 | type="range"
3 | bounds=[-10,10]
4 |
5 | [param2]
6 | type = "random"
7 | bounds = [2, 1, 3]
8 |
9 |
--------------------------------------------------------------------------------
/test/assets/dummy_runnable.py:
--------------------------------------------------------------------------------
1 | import time
2 | import random
3 |
4 |
5 | def run(x=1, **kwargs):
6 | time.sleep(random.uniform(0, 0.5))
7 | if x == 3:
8 | raise ValueError("oops!!")
9 | return x ** 2
10 |
--------------------------------------------------------------------------------
/test/assets/multiple_params.conf:
--------------------------------------------------------------------------------
1 | [x]
2 | type = "range"
3 | bounds = [0,3]
4 |
5 | [param1]
6 | type = "value"
7 | value = "param 1 value"
8 | [param2]
9 | type = "list"
10 | value = ["param 2 value 1", 1]
--------------------------------------------------------------------------------
/test/assets/runnable_tensorflow.py:
--------------------------------------------------------------------------------
1 | import sys
2 | import os
3 | import tensorflow as tf
4 | from exp.args import ParamDict,Namespace
5 |
6 | defaults = {
7 | 'x': (float, None),
8 | 'id': (int, 0),
9 | 'run': (int, 1)
10 | }
11 |
12 | os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
13 |
14 |
15 | def run(**kargs):
16 | args = ParamDict(defaults)
17 | args.from_dict(kargs)
18 | ns = args.to_namespace()
19 | #args = Namespace(args)
20 | # print(dargs)
21 |
22 | gpu = os.environ["CUDA_VISIBLE_DEVICES"]
23 | # test exceptions
24 | # if random.random() < 0.3:
25 | # raise Exception("failled with params: \n{}".format(kargs))
26 |
27 | a = tf.random_uniform([100000, 3])
28 | b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
29 | c = tf.matmul(a, b)
30 | d = tf.multiply(c, ns.x)
31 | #d = tf.matmul(c, ns.x)
32 |
33 | # cfg = tf.ConfigProto(log_device_placement=True)
34 | # sess = tf.Session(config=cfg)
35 | sess = tf.Session()
36 | res = sess.run(d)
37 | sess.close()
38 |
39 | debug = "INSIDE GPU WORKER ---------------\n" \
40 | "params: {params}\n" \
41 | "using GPU: {env}\n " \
42 | "result: \n {res}" \
43 | "-----------------------------------".format(params=args, env=gpu, res=res)
44 |
45 | tf.reset_default_graph()
46 | return debug
47 |
48 |
49 | if __name__ == "__main__":
50 | # note I can use argparse in the scripts to run directly from main
51 | run()
52 |
--------------------------------------------------------------------------------
/test/test_params.py:
--------------------------------------------------------------------------------
1 | import unittest
2 | from exp.params import ParamSpace, Types, DTypes
3 |
4 | import csv
5 | import os
6 |
7 |
8 | class MyTestCase(unittest.TestCase):
9 | def test_param_grid(self):
10 | ps = ParamSpace()
11 | ps.add_value("p1", True)
12 | ps.add_list("p2", ["A", "B"])
13 | ps.add_random("p3", low=0, high=4, prior="uniform", n=3)
14 | # print("param space size ", ps.grid_size)
15 |
16 | grid = ps.param_grid()
17 |
18 | # for params in grid:
19 | # print(params)
20 |
21 | grid = ps.param_grid()
22 | grid = list(grid)
23 | self.assertEqual(len(grid), 1 * 2 * 3)
24 | self.assertEqual(len(grid), ps.size)
25 |
26 | def test_write_recover(self):
27 | """ There is one issue with writing the param assets which is the fact that these do not preserve the
28 | value types, this is expected, the only issue was that we need to ensure that we can use np.random.uniform
29 | so regardless of the add_random and add_range arg types, they will be converted to float parameters
30 | """
31 | ps = ParamSpace()
32 | ps.add_value("p1", True)
33 | ps.add_list("p2", ["A", "B"])
34 | ps.add_random("p3", low=0, high=4, prior="uniform", n=3)
35 |
36 | param_filename = "test.conf"
37 | ps.write(param_filename)
38 | self.assertTrue(os.path.exists(param_filename))
39 | ParamSpace(param_filename)
40 | os.remove(param_filename)
41 |
42 | def test_write_summary(self):
43 | summary_file = "params.csv"
44 |
45 | ps = ParamSpace()
46 | ps.add_value("p1", True)
47 | ps.add_list("p2", ["A", "B"])
48 | ps.add_random("p3", low=0, high=4, prior="uniform", n=3)
49 | # print("param space size ", ps.grid_size)
50 |
51 | ps.write_configs(summary_file)
52 |
53 | written_summary = open(summary_file)
54 | reader = csv.DictReader(written_summary)
55 |
56 | params = [dict(config) for config in reader]
57 | # print("read parameters")
58 | # for config in params:
59 | # print(config)
60 |
61 | written_summary.close()
62 | os.remove(summary_file)
63 |
64 | self.assertEqual(len(params), ps.size)
65 |
66 | def test_add_random(self):
67 | """ If persist is not set to True for add_random
68 | each time we call param_grid, it samples new random values
69 | this is because persist = True saves the parameter as a list
70 | or randomly generated parameters
71 | """
72 | ps = ParamSpace()
73 | name = "param1"
74 | ps.add_random(name, low=2, high=4, persist=False, n=10, prior="uniform")
75 |
76 | params1 = ps.param_grid()
77 | self.assertTrue(ps.size, 1)
78 | r1 = next(params1)[name]
79 |
80 | params2 = ps.param_grid()
81 | r2 = next(params2)[name]
82 |
83 | ps.write("test.cfg")
84 |
85 | self.assertNotEqual(r1, r2)
86 |
87 | def test_add_range(self):
88 | filename = "test.cfg"
89 | ps = ParamSpace()
90 | ps.add_range("range_param", 0, 10, 1, dtype=int)
91 |
92 | ps.write(filename)
93 |
94 | ps = ParamSpace(filename)
95 | # print(ps.params["range_param"])
96 | # print(ps.get_range("range_param"))
97 |
98 | os.remove(filename)
99 |
100 | def test_domain(self):
101 | ps = ParamSpace()
102 | ps.add_value("value", True)
103 | domain = ps.domain("value")
104 | self.assertIn("domain", domain)
105 | self.assertIn("dtype", domain)
106 | self.assertEqual(DTypes.CATEGORICAL.value, domain["dtype"])
107 |
108 | ps.add_list("bool", [True, False, True])
109 | domain = ps.domain("bool")
110 | self.assertIn("domain", domain)
111 | self.assertIn("dtype", domain)
112 | self.assertEqual(DTypes.CATEGORICAL.value, domain["dtype"])
113 | self.assertListEqual([True, False], domain["domain"])
114 |
115 | ps.add_range("bounds", 0, 10, dtype=float)
116 | domain = ps.domain("bounds")
117 | self.assertIn("domain", domain)
118 | self.assertIn("dtype", domain)
119 | self.assertIn("prior", domain)
120 | self.assertEqual("float", domain["dtype"])
121 | self.assertEqual("uniform", domain["prior"])
122 |
123 | ps.add_random("random", 0, 10, prior="log-uniform", dtype=float)
124 | domain = ps.domain("bounds")
125 | self.assertIn("domain", domain)
126 | self.assertIn("dtype", domain)
127 | self.assertIn("prior", domain)
128 | self.assertEqual("float", domain["dtype"])
129 | self.assertEqual("uniform", domain["prior"])
130 |
131 | def test_param_grid_with_id(self):
132 | ps = ParamSpace()
133 | ps.add_value("p1", True)
134 | ps.add_list("p2", ["A", "B"])
135 |
136 | params1 = ps.param_grid(runs=5)
137 |
138 | self.assertEqual(len(list(params1)), 1 * 2 * 5)
139 |
140 | def test_write_grid_files(self):
141 | ps = ParamSpace()
142 | ps.add_value("p1", True)
143 | ps.add_list("p2", ["A", "B"])
144 | ps.add_random("p3", n=2, prior="uniform", low=1, high=3)
145 | # print("param space size ", ps.grid_size)
146 |
147 | out_path = "/tmp/test_params/"
148 | if not os.path.exists(out_path) or not os.path.isdir(out_path):
149 | os.makedirs(out_path)
150 | ps.write_config_files(out_path)
151 |
152 | def test_sample_params(self):
153 | ps = ParamSpace()
154 | ps.add_value("p1", True)
155 | ps.add_list("p2", ["A", "B"])
156 | ps.add_random("p3", n=1, prior="uniform", low=1, high=3)
157 |
158 | x = ps.sample_space()
159 | self.assertIsInstance(x, dict)
160 |
161 |
162 | if __name__ == '__main__':
163 | unittest.main()
164 |
--------------------------------------------------------------------------------