├── README.md ├── mylib ├── __init__.py ├── change_compare.py ├── online_compare.py ├── parameters_plot.py ├── ridgecv_op_online.py └── ridgecv_op_rbcu_hmm.py ├── optforwardtest.py ├── optunity ├── __init__.py ├── api.py ├── communication.py ├── constraints.py ├── cross_validation.py ├── functions.py ├── metrics.py ├── parallel.py ├── search_spaces.py ├── solvers │ ├── BayesOpt.py │ ├── CMAES.py │ ├── GridSearch.py │ ├── NelderMead.py │ ├── ParticleSwarm.py │ ├── ParticleSwarm_New.py │ ├── RandomSearch.py │ ├── Sobol.py │ ├── TPE.py │ ├── __init__.py │ ├── solver_registry.py │ └── util.py ├── standalone.py ├── tests │ ├── __init__.py │ ├── test_all.py │ └── test_solvers.py └── util.py ├── ridgecv_op.py └── ridgecv_op_rbcu.py /README.md: -------------------------------------------------------------------------------- 1 | optforwardtest system 2 | ===================== 3 | 4 | 5 | command: 6 | 7 | python optforwardtest.py -i ./data/IF.csv -of ./results/results.csv -op ./results/results.png -nj 3 -lp 10 -dn 1 -cl knn 8 | help: 9 | 10 | -i the input file 11 | -of the output file which records time index, predict label, and close 12 | -op the output file which describes sigsum and accuracy 13 | -nj number of jobs to run in parallel 14 | -lp length of periods to predict, not number of points to predict 15 | -dn length of periods to shift 16 | -cl classifier 17 | 18 | **'''1. DATA PREPARING'''** 19 | key point: 20 | 21 | rs_num and nrows: 22 | you can change rs_num and nrows to determine the beginning and last time for the csv file reading 23 | resample_time: 24 | you can change resample time, by time-resampling way or point-resampling way 25 | zero_propotion: 26 | you can change the variable, which describes the propotion of zero, to calculate the train data label 27 | 28 | **'''2. FEATURE EXTRACTION'''** 29 | key point: 30 | 31 | features: 32 | window_size: 33 | length of data default, you can change it if you like 34 | test_size: 35 | diff_n default, or greater than diff_n 36 | 37 | **'''3. CLASSIFICATION'''** 38 | * in this stage, you can achivement classification by multi_feature way or multi_classification way 39 | * in multi_feature way, different kinds of features may be gernerated together for one classifier 40 | * in multi_classification way, each classifier may output results, and you can combine them by vote or other 41 | * a simple classifier named mean classifier is added, based on the assumption that next close is the mean close of current window including current point 42 | 43 | > 44 | 窗口长度是window_size，这里的diff_n即timeshift 45 | diff_n的取值可以是1,2,3,4,5等等，也就是说如果diff_n等于2的时候，在训练数据定义label的时候，是指当前close与其后第2个点close的对比做出的一个方向。 46 | 换句话说，当前的close的label利用了未来第2个点的close信息。 47 | 所以我们在划分trainset和testset的时候，test_size必须大于等于diff_n才行，才能保证trainset不会偷看testset之后的close信息。（由此得出test_size=diff_n） 48 | 而我们要预测的是testset的label，特别是test[-1]。这就意味着，test[-1]得到的label，我们可以知道其后第2个点的close的升降。 49 | 所以窗口滑动的时候，要保证每次都取到当前预测点其后第2个点。（由此得出step=diff_n） 50 | 这样写入文件的，是每隔2分钟的预测点的close，index和预测的label。我们可以根据这些信息画出sigsum曲线来。 51 | > 52 | 而resampletime是我们每次读取窗口内数据的采样，对于最后一个点test[-1]来说，意味着前面数据的稀释。我们采样的目的是，数据可能冗余太多，所以要按照resampletime采样。 53 | 但是对于采样后的窗口来说，里面数据的label定义还是基于diff_n来做的，也就是一个点的label是当前close与其后第2个点的close的对比做出的一个方向。 54 | 55 | 56 | **前推的思路：** 57 | 58 | 先截取原始数据，窗口步长为diff_n，因为预测的是后diff_n的方向。 59 | 再对截取的片断采样。 60 | 注意计算label时候采用periods=int(diff_n/resample_time)，并生成特征。 61 | 为了保证训练集相邻点之间的特征计算，resample_time应该与diff_n一致。 62 | 测试集为int(diff_n/resample_time)。 63 | 64 | **不前推的思路：** 65 | 66 | 先采样原始数据。 67 | 注意计算label时候采用periods=int(diff_n/resample_time)，并生成特征。 68 | 为了保证训练集相邻点之间的特征计算，resample_time应该与diff_n一致。 69 | 再截取生成特征，窗口步长为int(diff_n/resample_time)，因为预测的是后diff_n的方向。 70 | 测试集为int(diff_n/resample_time)。 71 | 72 | 73 | ## The Default Project for LiNing 74 | 75 | ##### Please Do Not make any change without permission~ 76 | 77 | ### 一般修改的地方： 78 | 79 | file_path 80 | size1 81 | size2 82 | resample_num_list 83 | timeshift_num_list 84 | results_dir 85 | default_nrows 86 | whether timeshift equals resample or not 87 | num 88 | dynamic 89 | 90 | step 91 | target 92 | maxlag = [30, 90] 93 | windowsize = [50000, 900000] 94 | threshold = [95, 100] 95 | search = { 96 | 'algorithm':{'ridgecv':None}, 97 | # 'algorithm':{'ridgecv':None,'elasticnetcv':None,'knnreg':None,'linearsvr':None}, 98 | 'maxlag':maxlag, 99 | 'windowsize':windowsize, 100 | 'threshold':threshold, 101 | } 102 | num_evals = 100 103 | optunity.maximize or optunity.maximize_structured 104 | 105 | ### Optunity Revision Note 106 | 107 | 一直遇到的一个问题就是： 108 | op开启了比如30个进程，其实真正在计算的并没有这么多，很多同名进程都在挂载等待。 109 | 110 | 真正的原因还在于optunity内部： 111 | 在PS0源码中suggest_from_box定义，有 112 | d = dict(kwargs) 113 | if num_evals > 1000: 114 | d['num_particles'] = 100 115 | elif num_evals >= 200: 116 | d['num_particles'] = 20 117 | elif num_evals >= 10: 118 | d['num_particles'] = 10 119 | else: 120 | d['num_particles'] = num_evals 121 | d['num_generations'] = int(math.ceil(float(num_evals) / d['num_particles'])) 122 | return d 123 | 124 | 在PS0源码中optimize定义，有 125 | pop = [self.generate() for _ in range(self.num_particles)] 126 | best = None 127 | 128 | for g in range(self.num_generations): 129 | fitnesses = pmap(evaluate, list(map(self.particle2dict, pop))) 130 | 131 | 由此，可以看到，其实真正运行的进程数，不与num_evals相关，也不与你开启的进程数相关，而是与num_particles相关：它是对每个粒子的分支进行map操作！！！！ 132 | num_evals为3，同一时间调优就是3个，也就占3个进程，所以开30个进程的话，会出现27个等待进程 133 | num_evals为30，同一时间调优就是10个，也就占10个进程，所以开30个进程的话，会出现20个等待进程 134 | num_evals为300，同一时间调优就是20个，也就占20个进程，所以开30个进程的话，会出现10个等待进程 135 | 但是总共寻优的次数是与num_evals相关的，可能会略小于num_evals。 136 | -------------------------------------------------------------------------------- /mylib/__init__.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- -------------------------------------------------------------------------------- /mylib/change_compare.py: -------------------------------------------------------------------------------- 1 | #coding: utf-8 2 | import numpy as np 3 | import pandas as pd 4 | import matplotlib.pyplot as plt 5 | plt.style.use('ggplot') 6 | 7 | file_path1 = 'results_pred_change.txt' 8 | data1 = pd.read_csv(file_path1, header=None, index_col=0, names=['index','label_change'], sep='\t') 9 | # print data1 10 | file_path2 = 'results_pred_nochange.txt' 11 | data2 = pd.read_csv(file_path2, header=None, index_col=0, names=['index','label_nochange'], sep='\t') 12 | # print data2 13 | data = pd.concat([data1, data2], axis=1) 14 | date_list = [] 15 | time_list = [] 16 | time_stage_list = [] 17 | for idx in data.index: 18 | date, time = idx.split(' ') 19 | date_list.append(date) 20 | time_list.append(time) 21 | # if '09:01:00'<=time<='11:30:00': 22 | # time_stage_list.append('morning') 23 | # elif '13:01:00'<=time<='15:00:00': 24 | # time_stage_list.append('afternoon') 25 | # elif '21:01:00'<=time<='23:59:59' or '00:00:00'<=time<='01:00:00': 26 | # time_stage_list.append('evening') 27 | # else: 28 | # time_stage_list.append('') 29 | if '09:01:00'<=time<='09:30:00': 30 | time_stage_list.append('09:01:00-09:30:00') 31 | elif '09:31:00'<=time<='10:00:00': 32 | time_stage_list.append('09:31:00-10:00:00') 33 | elif '10:01:00'<=time<='10:30:00': 34 | time_stage_list.append('10:01:00-10:30:00') 35 | elif '10:31:00'<=time<='11:00:00': 36 | time_stage_list.append('10:31:00-11:00:00') 37 | elif '11:01:00'<=time<='11:30:00': 38 | time_stage_list.append('11:01:00-11:30:00') 39 | elif '13:31:00'<=time<='14:00:00': 40 | time_stage_list.append('13:31:00-14:00:00') 41 | elif '14:01:00'<=time<='14:30:00': 42 | time_stage_list.append('14:01:00-14:30:00') 43 | elif '14:31:00'<=time<='15:00:00': 44 | time_stage_list.append('14:31:00-15:00:00') 45 | elif '21:01:00'<=time<='21:30:00': 46 | time_stage_list.append('21:01:00-21:30:00') 47 | elif '21:31:00'<=time<='22:00:00': 48 | time_stage_list.append('21:31:00-22:00:00') 49 | elif '22:01:00'<=time<='22:30:00': 50 | time_stage_list.append('22:01:00-22:30:00') 51 | elif '22:31:00'<=time<='23:00:00': 52 | time_stage_list.append('22:31:00-23:00:00') 53 | elif '23:01:00'<=time<='23:30:00': 54 | time_stage_list.append('23:01:00-23:30:00') 55 | elif '23:31:00'<=time<='23:59:59' or time=='00:00:00': 56 | time_stage_list.append('23:31:00-00:00:00') 57 | elif '00:01:00'<=time<='00:30:00': 58 | time_stage_list.append('00:01:00-00:30:00') 59 | elif '00:31:00'<=time<='01:00:00': 60 | time_stage_list.append('00:31:00-01:00:00') 61 | else: 62 | time_stage_list.append('') 63 | 64 | data['date'] = date_list 65 | data['time'] = time_list 66 | data['time_stage'] = time_stage_list 67 | data['bool_different'] = pd.Series(data.label_change != data.label_nochange).astype(int) 68 | print data.head() 69 | dif_rate = data['bool_different'].sum()*1.0/data.shape[0] 70 | print 'different rate:%.2f%%' % (100*dif_rate) 71 | data_dif = data.ix[data.label_change != data.label_nochange] 72 | # print data_dif.head() 73 | # data_dif_groupsize = data_dif.groupby(data_dif.time).size().sort_values(ascending=True) 74 | data_dif_groupsize = data_dif.groupby(data_dif.time_stage).size().sort_values(ascending=True) 75 | print data_dif_groupsize 76 | plt.figure(figsize=(15,10)) 77 | data_dif_groupsize.plot.barh() 78 | plt.title('different rate:%.2f%%' % (100*dif_rate)) 79 | plt.savefig('data_different_groupsize.png') 80 | -------------------------------------------------------------------------------- /mylib/online_compare.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python2 2 | #coding: utf-8 3 | from __future__ import division 4 | import os 5 | import shutil 6 | import csv 7 | import copy 8 | import numpy as np 9 | import pandas as pd 10 | import matplotlib.pyplot as plt 11 | from mpl_toolkits.mplot3d import Axes3D 12 | plt.style.use('ggplot') 13 | import seaborn as sns 14 | from pandas.tseries.offsets import Milli 15 | import datetime 16 | from sklearn.linear_model import Ridge, RidgeCV, ElasticNet, ElasticNetCV 17 | from sklearn.svm import SVR, LinearSVR 18 | from sklearn.neighbors import KNeighborsRegressor 19 | import optunity 20 | import optunity.metrics 21 | from optunity.constraints import wrap_constraints 22 | from optunity.solvers.GridSearch import GridSearch 23 | from optunity.solvers.ParticleSwarm import ParticleSwarm 24 | import multiprocessing 25 | import gc 26 | 27 | try: 28 | import cPickle as pickle 29 | except ImportError: 30 | import pickle 31 | 32 | 33 | def pp(pic_path, accuracy_list, sigsum, real_sigsum, adjusted_sigsum): 34 | plt.figure() 35 | plt.subplot(211) 36 | plt.plot(accuracy_list, label='$accuracy$') 37 | plt.legend() 38 | plt.ylabel('accuracy') 39 | # plt.figtext(0.39, 0.95, 'accuracy:{:.4f}'.format(accuracy_list[-1]), color='green') 40 | # plt.figtext(0.13, 0.91, 'sigsum:{:6.1f}'.format(sigsum[-1]), color='green') 41 | # plt.figtext(0.39, 0.91, 'real_sigsum:{:6.1f}'.format(real_sigsum[-1]), color='green') 42 | # plt.figtext(0.65, 0.91, 'adjusted_sigsum:{:6.1f}'.format(adjusted_sigsum[-1]), color='green') 43 | plt.title('accuracy:{:.4f}, sigsum:{:6.1f}, real_sigsum:{:6.1f}, adjusted_sigsum:{:6.1f}'.format( 44 | accuracy_list[-1], sigsum[-1], real_sigsum[-1], adjusted_sigsum[-1])) 45 | plt.subplot(212) 46 | plt.plot(sigsum, 'r-', label='$sigsum$') 47 | plt.plot(real_sigsum, 'g-', label='$realsigsum$') 48 | plt.plot(adjusted_sigsum, 'b-', label='$adjustedsigsum$') 49 | plt.legend() 50 | plt.ylabel('sigsum') 51 | plt.savefig(pic_path) 52 | plt.close() 53 | plt.figure() 54 | plt.plot(adjusted_sigsum, 'b-', label='$adjustedsigsum$') 55 | plt.legend() 56 | plt.ylabel('adjusted_sigsum') 57 | # plt.figtext(0.39, 0.95, 'adjusted_sigsum:{:6.1f}'.format(adjusted_sigsum[-1]), color='green') 58 | plt.title('adjusted_sigsum:{:6.1f}'.format(adjusted_sigsum[-1])) 59 | plt.savefig(os.path.splitext(pic_path)[0]+'_'+os.path.splitext(pic_path)[1]) 60 | plt.close() 61 | 62 | if __name__ == '__main__': 63 | 64 | ########################################################################################## 65 | ''' 66 | show the results 67 | ''' 68 | plt.figure() 69 | 70 | results_path = './results_dir_online/results_resample1timeshift1/results.pkl' 71 | with open(results_path, 'rb') as fp: 72 | (accuracy_list, sigsum, real_sigsum, adjusted_sigsum) = pickle.load(fp) 73 | print 'accuracy: %.4f, sigsum: %6.1f, real_sigsum: %6.1f, adjusted_sigsum: %6.1f' % \ 74 | (accuracy_list[-1], sigsum[-1], real_sigsum[-1], adjusted_sigsum[-1]) 75 | # pp('./accuracy_sigsum.png', accuracy_list, sigsum, real_sigsum, adjusted_sigsum) 76 | plt.plot(adjusted_sigsum, label='$online$') 77 | 78 | results_path = './results_dir_offline/results_resample1timeshift1/results.pkl' 79 | with open(results_path, 'rb') as fp: 80 | (accuracy_list, sigsum, real_sigsum, adjusted_sigsum) = pickle.load(fp) 81 | print 'accuracy: %.4f, sigsum: %6.1f, real_sigsum: %6.1f, adjusted_sigsum: %6.1f' % \ 82 | (accuracy_list[-1], sigsum[-1], real_sigsum[-1], adjusted_sigsum[-1]) 83 | # pp('./accuracy_sigsum.png', accuracy_list, sigsum, real_sigsum, adjusted_sigsum) 84 | plt.plot(adjusted_sigsum, label='$offline$') 85 | 86 | plt.legend() 87 | plt.ylabel('adjusted_sigsum') 88 | # plt.figtext(0.39, 0.95, 'adjusted_sigsum:{:6.1f}'.format(adjusted_sigsum[-1]), color='green') 89 | # plt.title('adjusted_sigsum:{:6.1f}'.format(adjusted_sigsum[-1])) 90 | plt.savefig('./adjusted_sigsum.png') 91 | -------------------------------------------------------------------------------- /mylib/parameters_plot.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python2 2 | #coding: utf-8 3 | from __future__ import division 4 | import os 5 | import shutil 6 | import csv 7 | import copy 8 | import numpy as np 9 | import pandas as pd 10 | import matplotlib.pyplot as plt 11 | from mpl_toolkits.mplot3d import Axes3D 12 | plt.style.use('ggplot') 13 | import seaborn as sns 14 | from pandas.tseries.offsets import Milli 15 | import datetime 16 | from sklearn.linear_model import Ridge, RidgeCV, ElasticNet, ElasticNetCV 17 | from sklearn.svm import SVR, LinearSVR 18 | from sklearn.neighbors import KNeighborsRegressor 19 | import optunity 20 | import optunity.metrics 21 | from optunity.constraints import wrap_constraints 22 | from optunity.solvers.GridSearch import GridSearch 23 | from optunity.solvers.ParticleSwarm import ParticleSwarm 24 | import multiprocessing 25 | import gc 26 | 27 | try: 28 | import cPickle as pickle 29 | except ImportError: 30 | import pickle 31 | 32 | 33 | def pp(pic_path, accuracy_list, sigsum, real_sigsum, adjusted_sigsum): 34 | plt.figure() 35 | plt.subplot(211) 36 | plt.plot(accuracy_list, label='$accuracy$') 37 | plt.legend() 38 | plt.ylabel('accuracy') 39 | # plt.figtext(0.39, 0.95, 'accuracy:{:.4f}'.format(accuracy_list[-1]), color='green') 40 | # plt.figtext(0.13, 0.91, 'sigsum:{:6.1f}'.format(sigsum[-1]), color='green') 41 | # plt.figtext(0.39, 0.91, 'real_sigsum:{:6.1f}'.format(real_sigsum[-1]), color='green') 42 | # plt.figtext(0.65, 0.91, 'adjusted_sigsum:{:6.1f}'.format(adjusted_sigsum[-1]), color='green') 43 | plt.title('accuracy:{:.4f}, sigsum:{:6.1f}, real_sigsum:{:6.1f}, adjusted_sigsum:{:6.1f}'.format( 44 | accuracy_list[-1], sigsum[-1], real_sigsum[-1], adjusted_sigsum[-1])) 45 | plt.subplot(212) 46 | plt.plot(sigsum, 'r-', label='$sigsum$') 47 | plt.plot(real_sigsum, 'g-', label='$realsigsum$') 48 | plt.plot(adjusted_sigsum, 'b-', label='$adjustedsigsum$') 49 | plt.legend() 50 | plt.ylabel('sigsum') 51 | plt.savefig(pic_path) 52 | plt.close() 53 | plt.figure() 54 | plt.plot(adjusted_sigsum, 'b-', label='$adjustedsigsum$') 55 | plt.legend() 56 | plt.ylabel('adjusted_sigsum') 57 | # plt.figtext(0.39, 0.95, 'adjusted_sigsum:{:6.1f}'.format(adjusted_sigsum[-1]), color='green') 58 | plt.title('adjusted_sigsum:{:6.1f}'.format(adjusted_sigsum[-1])) 59 | plt.savefig(os.path.splitext(pic_path)[0]+'_'+os.path.splitext(pic_path)[1]) 60 | plt.close() 61 | 62 | if __name__ == '__main__': 63 | 64 | ########################################################################################## 65 | ''' 66 | show parameters distribution 67 | ''' 68 | # best_parameters_path = './best_parameters' 69 | # with open(best_parameters_path, 'rb') as fp: 70 | # (best_params, optimum, df_sort) = pickle.load(fp) 71 | # print 'best parameters:', best_params 72 | # print 'best score:', optimum 73 | # print 'sorted best parameters:' 74 | # print df_sort 75 | # 76 | # # df_sort.plot.scatter(x='maxlag', y='threshold') 77 | # # plt.show() 78 | # # df_sort.plot.scatter(x='threshold', y='windowsize') 79 | # # plt.show() 80 | # # df_sort.plot.scatter(x='windowsize', y='maxlag') 81 | # # plt.show() 82 | # # df_sort.plot.scatter(x='maxlag', y='threshold', c='windowsize', s=50) 83 | # # plt.show() 84 | # 85 | # #### 86 | # fig = plt.figure() 87 | # ax = fig.add_subplot(111, projection='3d') 88 | # # for xs, ys, zs in zip(x, y, z): 89 | # # ax.scatter(xs, ys, zs) 90 | # ax.scatter(df_sort['maxlag'], df_sort['windowsize'], df_sort['threshold']) 91 | # ax.set_xlabel('maxlag') 92 | # ax.set_ylabel('windowsize') 93 | # ax.set_zlabel('threshold') 94 | # # plt.show() 95 | # # #### 96 | # # sns.set() 97 | # # sns.pairplot(df_sort.drop('value', axis=1)) 98 | # # # plt.show() 99 | # plt.savefig('./best_parameters.png') 100 | 101 | ########################################################################################## 102 | ''' 103 | show the results 104 | ''' 105 | results_path = './results.pkl' 106 | with open(results_path, 'rb') as fp: 107 | (accuracy_list, sigsum, real_sigsum, adjusted_sigsum) = pickle.load(fp) 108 | print 'accuracy: %.4f, sigsum: %6.1f, real_sigsum: %6.1f, adjusted_sigsum: %6.1f' % \ 109 | (accuracy_list[-1], sigsum[-1], real_sigsum[-1], adjusted_sigsum[-1]) 110 | pp('./accuracy_sigsum.png', accuracy_list, sigsum, real_sigsum, adjusted_sigsum) 111 | 112 | #### 113 | plt.figure() 114 | plt.plot(adjusted_sigsum, label='$adjustedsigsum$') 115 | plt.legend() 116 | plt.ylabel('adjusted_sigsum') 117 | # plt.figtext(0.39, 0.95, 'adjusted_sigsum:{:6.1f}'.format(adjusted_sigsum[-1]), color='green') 118 | plt.title('adjusted_sigsum:{:6.1f}'.format(adjusted_sigsum[-1])) 119 | plt.savefig('./adjusted_sigsum.png') 120 | 121 | ########################################################################################## 122 | ''' 123 | show the results for different resample and timeshift 124 | ''' 125 | # # resample_num_list = [1] 126 | # # resample_num_list = [1, 2, 4, 8] 127 | # resample_num_list = [1, 2, 3, 4, 5, 6, 7, 8] 128 | # # timeshift_num_list = [1] 129 | # # timeshift_num_list = [1, 2, 4, 8, 16] 130 | # timeshift_num_list = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20] 131 | # 132 | # # results_dir = './' 133 | # results_dir = './results_dir/' 134 | # results_list = [] 135 | # 136 | # for resample_num in resample_num_list: 137 | # 138 | # '''you can change the condition here: whether timeshift equals resample or not''' 139 | # # timeshift_num_list = [resample_num] ## timeshift == resample 140 | # timeshift_num_list = filter(lambda x: x>=resample_num, timeshift_num_list) ## timeshift >= resample 141 | # 142 | # plt.figure(0) 143 | # plt.figure(1) 144 | # plt.figure(2) 145 | # plt.figure(3) 146 | # 147 | # for timeshift_num in timeshift_num_list: 148 | # results_path = results_dir+'results_'+'resample'+str(resample_num)+'timeshift'+str(timeshift_num) 149 | # with open(os.path.join(results_path, 'results.pkl'), 'rb') as fp: 150 | # (accuracy_list, sigsum, real_sigsum, adjusted_sigsum) = pickle.load(fp) 151 | # print 'accuracy: %.4f, sigsum: %6.1f, real_sigsum: %6.1f, adjusted_sigsum: %6.1f' % \ 152 | # (accuracy_list[-1], sigsum[-1], real_sigsum[-1], adjusted_sigsum[-1]) 153 | # # pp(os.path.join(results_path, './accuracy_sigsum.png'), accuracy_list, sigsum, real_sigsum, adjusted_sigsum) 154 | # filter_results = filter(lambda x:x.startswith('accuracy_sigsum_tradecount'), os.listdir(results_path)) 155 | # tradecount = int(filter_results[0].replace('accuracy_sigsum_tradecount', '').replace('.png', '')) 156 | # 157 | # results_list.append( 158 | # { 159 | # 'resample':resample_num, 160 | # 'timeshift':timeshift_num, 161 | # 'accuracy':accuracy_list[-1], 162 | # 'sigsum':sigsum[-1], 163 | # 'realsigsum':real_sigsum[-1], 164 | # 'adjustedsigsum':adjusted_sigsum[-1], 165 | # 'tradecount':tradecount, 166 | # 'sigpertrade':adjusted_sigsum[-1]/tradecount, 167 | # } 168 | # ) 169 | # 170 | # plt.figure(0) 171 | # plt.title('accuracy') 172 | # plt.plot(accuracy_list, label='$timeshift%d$' % timeshift_num) 173 | # plt.figure(1) 174 | # plt.title('sigsum') 175 | # plt.plot(sigsum, label='$timeshift%d$' % timeshift_num) 176 | # plt.figure(2) 177 | # plt.title('realsigsum') 178 | # plt.plot(real_sigsum, label='$timeshift%d$' % timeshift_num) 179 | # plt.figure(3) 180 | # plt.title('adjustedsigsum') 181 | # plt.plot(adjusted_sigsum, label='$timeshift%d$' % timeshift_num) 182 | # 183 | # plt.figure(0) 184 | # plt.legend() 185 | # plt.savefig(results_dir+'accuracy_resample%d' % resample_num) 186 | # plt.close() 187 | # plt.figure(1) 188 | # plt.legend() 189 | # plt.savefig(results_dir+'sigsum_resample%d' % resample_num) 190 | # plt.close() 191 | # plt.figure(2) 192 | # plt.legend() 193 | # plt.savefig(results_dir+'realsigsum_resample%d' % resample_num) 194 | # plt.close() 195 | # plt.figure(3) 196 | # plt.legend() 197 | # plt.savefig(results_dir+'adjustedsigsum_resample%d' % resample_num) 198 | # plt.close() 199 | # 200 | # df_results = pd.DataFrame(results_list) 201 | # print df_results 202 | # 203 | # with open(results_dir+'df_results.pkl', 'wb') as fp: 204 | # pickle.dump(df_results, fp) 205 | # 206 | # with open(results_dir+'df_results.pkl', 'rb') as fp: 207 | # df_results = pickle.load(fp) 208 | # 209 | # print df_results.sort_values(by='adjustedsigsum', axis=0, ascending=False).head(20) 210 | # plt.figure() 211 | # df_results.plot.scatter(x='resample', y='timeshift', c='accuracy', s=50) 212 | # plt.savefig(results_dir+'accuracylist_resample_timeshift.png') 213 | # plt.close() 214 | # plt.figure() 215 | # df_results.plot.scatter(x='resample', y='timeshift', c='sigsum', s=50) 216 | # plt.savefig(results_dir+'sigsum_resample_timeshift.png') 217 | # plt.close() 218 | # plt.figure() 219 | # df_results.plot.scatter(x='resample', y='timeshift', c='realsigsum', s=50) 220 | # plt.savefig(results_dir+'realsigsum_resample_timeshift.png') 221 | # plt.close() 222 | # plt.figure() 223 | # df_results.plot.scatter(x='resample', y='timeshift', c='adjustedsigsum', s=50) 224 | # plt.savefig(results_dir+'adjustedsigsum_resample_timeshift.png') 225 | # df_results.plot.scatter(x='resample', y='timeshift', s=df_results['adjustedsigsum']*0.1) 226 | # plt.savefig(results_dir+'adjustedsigsum_resample_timeshift_.png') 227 | # 228 | # df_results.sort_values(by='adjustedsigsum', axis=0, ascending=False, inplace=True) 229 | # df_results.to_excel(results_dir+'output.xlsx', 'df_results') 230 | -------------------------------------------------------------------------------- /optunity/__init__.py: -------------------------------------------------------------------------------- 1 | """ 2 | Optunity 3 | ========== 4 | 5 | Provides 6 | 1. Routines to efficiently optimize hyperparameters 7 | 2. Function decorators to implicitly log evaluations, constrain the domain and more. 8 | 3. Facilities for k-fold cross-validation to estimate generalization performance. 9 | 10 | Available modules 11 | --------------------- 12 | solvers 13 | contains all officially supported solvers 14 | functions 15 | a variety of useful function decorators for hyperparameter tuning 16 | cross_validation 17 | k-fold cross-validation 18 | 19 | Available subpackages 20 | --------------------- 21 | tests 22 | regression test suite 23 | solvers 24 | solver implementations and auxillary functions 25 | 26 | Utilities 27 | --------- 28 | 29 | __version__ 30 | Optunity version string 31 | __revision__ 32 | Optunity revision string 33 | __author__ 34 | Main authors of the package 35 | 36 | """ 37 | 38 | from .api import manual, maximize, minimize, optimize, available_solvers, maximize_structured, minimize_structured 39 | from .api import wrap_call_log, wrap_constraints, make_solver, suggest_solver 40 | from .cross_validation import cross_validated, generate_folds 41 | from .parallel import pmap 42 | from .functions import call_log2dataframe 43 | 44 | __author__ = "Marc Claesen, Jaak Simm and Dusan Popovic" 45 | __version__ = "1.0.0" 46 | __revision__ = "1.0.1" 47 | 48 | __all__ = ['manual', 'maximize', 'minimize', 'optimize', 49 | 'wrap_call_log', 'wrap_constraints', 'make_solver', 50 | 'suggest_solver', 'cross_validated', 'generate_folds', 51 | 'pmap', 'available_solvers', 'call_log2dataframe', 52 | 'maximize_structured', 'minimize_structured'] 53 | -------------------------------------------------------------------------------- /optunity/api.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | 3 | # Copyright (c) 2014 KU Leuven, ESAT-STADIUS 4 | # All rights reserved. 5 | # 6 | # Redistribution and use in source and binary forms, with or without 7 | # modification, are permitted provided that the following conditions 8 | # are met: 9 | # 10 | # 1. Redistributions of source code must retain the above copyright 11 | # notice, this list of conditions and the following disclaimer. 12 | # 13 | # 2. Redistributions in binary form must reproduce the above copyright 14 | # notice, this list of conditions and the following disclaimer in the 15 | # documentation and/or other materials provided with the distribution. 16 | # 17 | # 3. Neither name of copyright holders nor the names of its contributors 18 | # may be used to endorse or promote products derived from this software 19 | # without specific prior written permission. 20 | # 21 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 22 | # ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 23 | # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 24 | # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR 25 | # CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, 26 | # EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, 27 | # PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR 28 | # PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF 29 | # LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 30 | # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 31 | # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 32 | 33 | """A collection of top-level API functions for Optunity. 34 | 35 | Main functions in this module: 36 | 37 | * :func:`make_solver` 38 | * :func:`suggest_solver` 39 | * :func:`manual` 40 | * :func:`maximize` 41 | * :func:`maximize_structured` 42 | * :func:`minimize` 43 | * :func:`minimize_structured` 44 | * :func:`optimize` 45 | 46 | We recommend using these functions rather than equivalents found in other places, 47 | e.g. :mod:`optunity.solvers`. 48 | 49 | .. moduleauthor:: Marc Claesen 50 | 51 | """ 52 | 53 | import timeit 54 | import sys 55 | import operator 56 | 57 | # optunity imports 58 | from . import functions as fun 59 | from . import solvers 60 | from . import search_spaces 61 | from .solvers import solver_registry 62 | from .util import DocumentedNamedTuple as DocTup 63 | from .constraints import wrap_constraints 64 | 65 | 66 | def _manual_lines(solver_name=None): 67 | """Brief solver manual. 68 | 69 | :param solver_name: (optional) name of the solver to request a manual from. 70 | If none is specified, a general manual and list of all registered solvers is returned. 71 | 72 | :result: 73 | * list of strings that contain the requested manual 74 | * solver name(s): name of the solver that was specified or list of all registered solvers. 75 | 76 | Raises ``KeyError`` if ``solver_name`` is not registered.""" 77 | if solver_name: 78 | return solver_registry.get(solver_name).desc_full, [solver_name] 79 | else: 80 | return solver_registry.manual(), solver_registry.solver_names() 81 | 82 | 83 | def available_solvers(): 84 | """Returns a list of all available solvers. 85 | 86 | These can be used in :func:`optunity.make_solver`. 87 | """ 88 | return solver_registry.solver_names() 89 | 90 | 91 | def manual(solver_name=None): 92 | """Prints the manual of requested solver. 93 | 94 | :param solver_name: (optional) name of the solver to request a manual from. 95 | If none is specified, a general manual is printed. 96 | 97 | Raises ``KeyError`` if ``solver_name`` is not registered.""" 98 | if solver_name: 99 | man = solver_registry.get(solver_name).desc_full 100 | else: 101 | man = solver_registry.manual() 102 | print('\n'.join(man)) 103 | 104 | 105 | optimize_results = DocTup(""" 106 | **Result details includes the following**: 107 | 108 | optimum 109 | optimal function value f(solution) 110 | 111 | stats 112 | statistics about the solving process 113 | 114 | call_log 115 | the call log 116 | 117 | report 118 | solver report, can be None 119 | """, 120 | 'optimize_results', ['optimum', 121 | 'stats', 122 | 'call_log', 123 | 'report'] 124 | ) 125 | optimize_stats = DocTup(""" 126 | **Statistics gathered while solving a problem**: 127 | 128 | num_evals 129 | number of function evaluations 130 | time 131 | wall clock time needed to solve 132 | """, 133 | 'optimize_stats', ['num_evals', 'time']) 134 | 135 | 136 | def suggest_solver(num_evals=50, solver_name=None, **kwargs): 137 | if solver_name: 138 | solvercls = solver_registry.get(solver_name) 139 | else: 140 | solver_name = 'particle swarm' 141 | solvercls = solvers.ParticleSwarm 142 | if hasattr(solvercls, 'suggest_from_box'): 143 | suggestion = solvercls.suggest_from_box(num_evals, **kwargs) 144 | elif hasattr(solvercls, 'suggest_from_seed'): 145 | # the seed will be the center of the box that is provided to us 146 | seed = dict([(k, float(v[0] + v[1]) / 2) for k, v in kwargs.items()]) 147 | suggestion = solvercls.suggest_from_seed(num_evals, **seed) 148 | else: 149 | raise ValueError('Unable to instantiate ' + solvercls.name + '.') 150 | suggestion['solver_name'] = solver_name 151 | return suggestion 152 | 153 | 154 | def maximize(f, num_evals=50, solver_name=None, pmap=map, **kwargs): 155 | """Basic function maximization routine. Maximizes ``f`` within 156 | the given box constraints. 157 | 158 | :param f: the function to be maximized 159 | :param num_evals: number of permitted function evaluations 160 | :param solver_name: name of the solver to use (optional) 161 | :type solver_name: string 162 | :param pmap: the map function to use 163 | :type pmap: callable 164 | :param kwargs: box constraints, a dict of the following form 165 | ``{'parameter_name': [lower_bound, upper_bound], ...}`` 166 | :returns: retrieved maximum, extra information and solver info 167 | 168 | This function will implicitly choose an appropriate solver and 169 | its initialization based on ``num_evals`` and the box constraints. 170 | 171 | """ 172 | # sanity check on box constraints 173 | assert all([len(v) == 2 and v[0] < v[1] 174 | for v in kwargs.values()]), 'Box constraints improperly specified: should be [lb, ub] pairs' 175 | 176 | f = _wrap_hard_box_constraints(f, kwargs, -sys.float_info.max) 177 | 178 | suggestion = suggest_solver(num_evals, solver_name, **kwargs) 179 | solver = make_solver(**suggestion) 180 | solution, details = optimize(solver, f, maximize=True, max_evals=num_evals, 181 | pmap=pmap) 182 | return solution, details, suggestion 183 | 184 | 185 | def minimize(f, num_evals=50, solver_name=None, pmap=map, **kwargs): 186 | """Basic function minimization routine. Minimizes ``f`` within 187 | the given box constraints. 188 | 189 | :param f: the function to be minimized 190 | :param num_evals: number of permitted function evaluations 191 | :param solver_name: name of the solver to use (optional) 192 | :type solver_name: string 193 | :param pmap: the map function to use 194 | :type pmap: callable 195 | :param kwargs: box constraints, a dict of the following form 196 | ``{'parameter_name': [lower_bound, upper_bound], ...}`` 197 | :returns: retrieved minimum, extra information and solver info 198 | 199 | This function will implicitly choose an appropriate solver and 200 | its initialization based on ``num_evals`` and the box constraints. 201 | 202 | """ 203 | # sanity check on box constraints 204 | assert all([len(v) == 2 and v[0] < v[1] 205 | for v in kwargs.values()]), 'Box constraints improperly specified: should be [lb, ub] pairs' 206 | 207 | func = _wrap_hard_box_constraints(f, kwargs, sys.float_info.max) 208 | 209 | suggestion = suggest_solver(num_evals, solver_name, **kwargs) 210 | solver = make_solver(**suggestion) 211 | solution, details = optimize(solver, func, maximize=False, max_evals=num_evals, 212 | pmap=pmap) 213 | return solution, details, suggestion 214 | 215 | 216 | def optimize(solver, func, maximize=True, max_evals=0, pmap=map, decoder=None): 217 | """Optimizes func with given solver. 218 | 219 | :param solver: the solver to be used, for instance a result from :func:`optunity.make_solver` 220 | :param func: the objective function 221 | :type func: callable 222 | :param maximize: maximize or minimize? 223 | :type maximize: bool 224 | :param max_evals: maximum number of permitted function evaluations 225 | :type max_evals: int 226 | :param pmap: the map() function to use, to vectorize use :func:`optunity.parallel.pmap` 227 | :type pmap: function 228 | 229 | Returns the solution and a namedtuple with further details. 230 | Please refer to docs of optunity.maximize_results 231 | and optunity.maximize_stats. 232 | 233 | """ 234 | 235 | if max_evals > 0: 236 | f = fun.max_evals(max_evals)(func) 237 | else: 238 | f = func 239 | 240 | f = fun.logged(f) 241 | num_evals = -len(f.call_log) 242 | 243 | time = timeit.default_timer() 244 | try: 245 | solution, report = solver.optimize(f, maximize, pmap=pmap) 246 | except fun.MaximumEvaluationsException: 247 | # early stopping because maximum number of evaluations is reached 248 | # retrieve solution from the call log 249 | report = None 250 | if maximize: 251 | index, _ = max(enumerate(f.call_log.values()), key=operator.itemgetter(1)) 252 | else: 253 | index, _ = min(enumerate(f.call_log.values()), key=operator.itemgetter(1)) 254 | solution = list(f.call_log.keys())[index]._asdict() 255 | time = timeit.default_timer() - time 256 | 257 | # TODO why is this necessary? 258 | if decoder: solution = decoder(solution) 259 | 260 | optimum = f.call_log.get(**solution) 261 | num_evals += len(f.call_log) 262 | 263 | # use namedtuple to enforce uniformity in case of changes 264 | stats = optimize_stats(num_evals, time) 265 | 266 | call_dict = f.call_log.to_dict() 267 | return solution, optimize_results(optimum, stats._asdict(), 268 | call_dict, report) 269 | 270 | 271 | optimize.__doc__ = ''' 272 | Optimizes func with given solver. 273 | 274 | :param solver: the solver to be used, for instance a result from :func:`optunity.make_solver` 275 | :param func: the objective function 276 | :type func: callable 277 | :param maximize: maximize or minimize? 278 | :type maximize: bool 279 | :param max_evals: maximum number of permitted function evaluations 280 | :type max_evals: int 281 | :param pmap: the map() function to use, to vectorize use :func:`optunity.pmap` 282 | :type pmap: function 283 | 284 | Returns the solution and a ``namedtuple`` with further details. 285 | ''' + optimize_results.__doc__ + optimize_stats.__doc__ 286 | 287 | 288 | def make_solver(solver_name, *args, **kwargs): 289 | """Creates a Solver from given parameters. 290 | 291 | :param solver_name: the solver to instantiate 292 | :type solver_name: string 293 | :param args: positional arguments to solver constructor. 294 | :param kwargs: keyword arguments to solver constructor. 295 | 296 | Use :func:`optunity.manual` to get a list of registered solvers. 297 | For constructor arguments per solver, please refer to :doc:`/user/solvers`. 298 | 299 | Raises ``KeyError`` if 300 | 301 | - ``solver_name`` is not registered 302 | - ``*args`` and ``**kwargs`` are invalid to instantiate the solver. 303 | 304 | """ 305 | solvercls = solver_registry.get(solver_name) 306 | return solvercls(*args, **kwargs) 307 | 308 | 309 | def wrap_call_log(f, call_dict): 310 | """Wraps an existing call log (as dictionary) around f. 311 | 312 | This allows you to communicate known function values to solvers. 313 | (currently available solvers do not use this info) 314 | 315 | """ 316 | f = fun.logged(f) 317 | call_log = fun.CallLog.from_dict(call_dict) 318 | if f.call_log: 319 | f.call_log.update(call_log) 320 | else: 321 | f.call_log = call_log 322 | return f 323 | 324 | 325 | def _wrap_hard_box_constraints(f, box, default): 326 | """Places hard box constraints on the domain of ``f`` 327 | and defaults function values if constraints are violated. 328 | 329 | :param f: the function to be wrapped with constraints 330 | :type f: callable 331 | :param box: the box, as a dict: ``{'param_name': [lb, ub], ...}`` 332 | :type box: dict 333 | :param default: function value to default to when constraints 334 | are violated 335 | :type default: number 336 | 337 | """ 338 | return wrap_constraints(f, default, range_oo=box) 339 | 340 | 341 | def maximize_structured(f, search_space, num_evals=50, pmap=map): 342 | """Basic function maximization routine. Maximizes ``f`` within 343 | the given box constraints. 344 | 345 | :param f: the function to be maximized 346 | :param search_space: the search space (see :doc:`/user/structured_search_spaces` for details) 347 | :param num_evals: number of permitted function evaluations 348 | :param pmap: the map function to use 349 | :type pmap: callable 350 | :returns: retrieved maximum, extra information and solver info 351 | 352 | This function will implicitly choose an appropriate solver and 353 | its initialization based on ``num_evals`` and the box constraints. 354 | 355 | """ 356 | tree = search_spaces.SearchTree(search_space) 357 | box = tree.to_box() 358 | 359 | # we need to position the call log here 360 | # because the function signature used later on is internal logic 361 | f = fun.logged(f) 362 | 363 | # wrap the decoder and constraints for the internal search space representation 364 | f = tree.wrap_decoder(f) 365 | f = _wrap_hard_box_constraints(f, box, -sys.float_info.max) 366 | 367 | suggestion = suggest_solver(num_evals, "particle swarm", **box) 368 | solver = make_solver(**suggestion) 369 | solution, details = optimize(solver, f, maximize=True, max_evals=num_evals, 370 | pmap=pmap, decoder=tree.decode) 371 | return solution, details, suggestion 372 | 373 | def minimize_structured(f, search_space, num_evals=50, pmap=map): 374 | """Basic function minimization routine. Minimizes ``f`` within 375 | the given box constraints. 376 | 377 | :param f: the function to be maximized 378 | :param search_space: the search space (see :doc:`/user/structured_search_spaces` for details) 379 | :param num_evals: number of permitted function evaluations 380 | :param pmap: the map function to use 381 | :type pmap: callable 382 | :returns: retrieved maximum, extra information and solver info 383 | 384 | This function will implicitly choose an appropriate solver and 385 | its initialization based on ``num_evals`` and the box constraints. 386 | 387 | """ 388 | tree = search_spaces.SearchTree(search_space) 389 | box = tree.to_box() 390 | 391 | # we need to position the call log here 392 | # because the function signature used later on is internal logic 393 | f = fun.logged(f) 394 | 395 | # wrap the decoder and constraints for the internal search space representation 396 | f = tree.wrap_decoder(f) 397 | f = _wrap_hard_box_constraints(f, box, sys.float_info.max) 398 | 399 | suggestion = suggest_solver(num_evals, "particle swarm", **box) 400 | solver = make_solver(**suggestion) 401 | solution, details = optimize(solver, f, maximize=False, max_evals=num_evals, 402 | pmap=pmap, decoder=tree.decode) 403 | return solution, details, suggestion 404 | 405 | -------------------------------------------------------------------------------- /optunity/communication.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | 3 | # Author: Marc Claesen 4 | # 5 | # Copyright (c) 2014 KU Leuven, ESAT-STADIUS 6 | # All rights reserved. 7 | # 8 | # Redistribution and use in source and binary forms, with or without 9 | # modification, are permitted provided that the following conditions 10 | # are met: 11 | # 12 | # 1. Redistributions of source code must retain the above copyright 13 | # notice, this list of conditions and the following disclaimer. 14 | # 15 | # 2. Redistributions in binary form must reproduce the above copyright 16 | # notice, this list of conditions and the following disclaimer in the 17 | # documentation and/or other materials provided with the distribution. 18 | # 19 | # 3. Neither name of copyright holders nor the names of its contributors 20 | # may be used to endorse or promote products derived from this software 21 | # without specific prior written permission. 22 | # 23 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 24 | # ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 25 | # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 26 | # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR 27 | # CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, 28 | # EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, 29 | # PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR 30 | # PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF 31 | # LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 32 | # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 33 | # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 34 | 35 | from __future__ import print_function 36 | import json 37 | import sys 38 | import socket 39 | import itertools 40 | from . import functions 41 | from . import parallel 42 | import math 43 | 44 | 45 | import multiprocessing 46 | import threading 47 | 48 | __DEBUG = False 49 | 50 | 51 | __channel_in = sys.stdin 52 | __channel_out = sys.stdout 53 | 54 | 55 | def _find_replacement(key, kwargs): 56 | """Finds a replacement for key that doesn't collide with anything in kwargs.""" 57 | key += '_' 58 | while key in kwargs: 59 | key += '_' 60 | return key 61 | 62 | 63 | def _find_replacements(illegal_keys, kwargs): 64 | """Finds replacements for all illegal keys listed in kwargs.""" 65 | replacements = {} 66 | for key in illegal_keys: 67 | if key in kwargs: 68 | replacements[key] = _find_replacement(key, kwargs) 69 | return replacements 70 | 71 | 72 | def _replace_keys(kwargs, replacements): 73 | """Replace illegal keys in kwargs with another value.""" 74 | 75 | for key, replacement in replacements.items(): 76 | if key in kwargs: 77 | kwargs[replacement] = kwargs[key] 78 | del kwargs[key] 79 | return kwargs 80 | 81 | 82 | def json_encode(data): 83 | """Encodes given data in a JSON string.""" 84 | return json.dumps(data) 85 | 86 | 87 | def json_decode(data): 88 | """Decodes given JSON string and returns its data. 89 | 90 | >>> orig = {1: 2, 'a': [1,2]} 91 | >>> data = json_decode(json_encode(orig)) 92 | >>> data[str(1)] 93 | 2 94 | >>> data['a'] 95 | [1, 2] 96 | """ 97 | return json.loads(data) 98 | 99 | 100 | def send(data): 101 | """Writes data to channel and flushes.""" 102 | print(data, file=__channel_out) 103 | __channel_out.flush() 104 | 105 | 106 | def receive(): 107 | """Reads data from channel.""" 108 | line = __channel_in.readline()[:-1] 109 | if not line: 110 | raise EOFError("Unexpected end of communication.") 111 | return line 112 | 113 | 114 | def open_socket(port, host='localhost'): 115 | """Opens a socket to host:port and reconfigures internal channels.""" 116 | global __channel_in 117 | global __channel_out 118 | try: 119 | sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 120 | sock.bind(('', 0)) 121 | sock.connect((host, port)) 122 | __channel_in = sock.makefile('r') 123 | __channel_out = sock.makefile('w') 124 | except (socket.error, OverflowError, ValueError) as e: 125 | print('Error making socket: ' + str(e), file=sys.stderr) 126 | sys.exit(1) 127 | 128 | 129 | def open_server_socket(): 130 | serv_sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 131 | try: 132 | serv_sock.bind(('', 0)) 133 | except socket.err as msg: 134 | print('Bind failed. Error Code : ' + str(msg[0]) + ' Message ' + msg[1]) 135 | sys.exit() 136 | serv_sock.listen(0) 137 | port = serv_sock.getsockname()[1] 138 | return port, serv_sock 139 | 140 | 141 | def accept_server_connection(server_socket): 142 | global __channel_in 143 | global __channel_out 144 | try: 145 | sock, _ = server_socket.accept() 146 | __channel_in = sock.makefile('r') 147 | __channel_out = sock.makefile('w') 148 | except (socket.error, OverflowError, ValueError) as e: 149 | print('Error making socket: ' + str(e), file=sys.stderr) 150 | sys.exit(1) 151 | 152 | 153 | class EvalManager(object): 154 | 155 | def __init__(self, max_vectorized=100, replacements={}): 156 | """Constructs an EvalManager object. 157 | 158 | :param max_vectorized: the maximum size of a vector evaluation 159 | larger vectorizations will be chunked 160 | :type max_vectorized: int 161 | :param replacements: a mapping of `original:replacement` keyword names 162 | :type replacements: dict 163 | 164 | """ 165 | # are we doing a parallel function evaluation? 166 | self._vectorized = False 167 | self._max_vectorized = max_vectorized 168 | 169 | # the queue used for parallel evaluations 170 | self._queue = None 171 | 172 | # lock to use to add evaluations to the queue 173 | self._queue_lock = multiprocessing.Lock() 174 | 175 | # lock to get results 176 | self._result_lock = multiprocessing.Lock() 177 | 178 | # used to check whether Future started running 179 | self._semaphore = None 180 | 181 | # used to signal Future's to get their result 182 | self._processed_semaphore = None 183 | 184 | # keys that must be replaced 185 | self._replacements = dict((v, k) for k, v in replacements.items()) 186 | 187 | 188 | @property 189 | def replacements(self): 190 | """Keys that must be replaced: keys are current values, values are 191 | what must be sent through the pipe.""" 192 | return self._replacements 193 | 194 | @property 195 | def max_vectorized(self): 196 | return self._max_vectorized 197 | 198 | @property 199 | def cv(self): 200 | return self._cv 201 | 202 | @property 203 | def semaphore(self): 204 | return self._semaphore 205 | 206 | @property 207 | def processed_semaphore(self): 208 | return self._processed_semaphore 209 | 210 | def pmap(self, f, *args): 211 | """Performs vector evaluations through pipes. 212 | 213 | :param f: the objective function (piped_function_eval) 214 | :type f: callable 215 | :param args: function arguments 216 | :type args: iterables 217 | 218 | The vector evaluation is sent in chunks of size self.max_vectorized. 219 | 220 | """ 221 | argslist = zip(*args) 222 | results = [] 223 | 224 | # partition the vector evaluation in chunks <= self.max_vectorized 225 | for idx in range(int(math.ceil(1.0 * len(argslist) / self.max_vectorized))): 226 | chunk = argslist[idx * self.max_vectorized:(idx + 1) * self.max_vectorized] 227 | results.extend(self._vector_eval(f, *zip(*chunk))) 228 | 229 | return results 230 | 231 | def _vector_eval(self, f, *args): 232 | self._queue = [] 233 | self._vectorized = True 234 | self._semaphore = multiprocessing.Semaphore(0) 235 | self._processed_semaphore = multiprocessing.Semaphore(0) 236 | 237 | # make sure correct evaluations lead to counter increments 238 | def wrap(*args): 239 | result = f(*args) 240 | # signal mgr to add next future 241 | self.semaphore.release() 242 | return result 243 | 244 | # create list of futures 245 | futures = [] 246 | for ar in zip(*args): 247 | futures.append(parallel.Future(wrap, *ar)) 248 | try: 249 | self.semaphore.acquire() 250 | except functions.MaximumEvaluationsException: 251 | if not len(self.queue): 252 | # FIXME: some threads may be running atm 253 | raise 254 | break 255 | 256 | # process queue 257 | self.flush_queue() 258 | 259 | # notify all waiting futures 260 | for _ in range(len(self.queue)): 261 | self.processed_semaphore.release() 262 | 263 | # gather results 264 | results = [f() for f in futures] 265 | for f in futures: 266 | f.join() 267 | return results 268 | 269 | def pipe_eval(self, **kwargs): 270 | # fix python keywords back to original 271 | kwargs = _replace_keys(kwargs, self.replacements) 272 | 273 | json_data = json_encode(kwargs) 274 | send(json_data) 275 | 276 | json_reply = receive() 277 | decoded = json_decode(json_reply) 278 | if decoded.get('error', False): # TODO: allow error handling higher up? 279 | sys.stderr('ERROR: ' + decoded['error']) 280 | sys.exit(1) 281 | return decoded['value'] 282 | 283 | def add_to_queue(self, **kwargs): 284 | kwargs = _replace_keys(kwargs, self.replacements) 285 | try: 286 | self.queue_lock.acquire() 287 | self.queue.append(kwargs) 288 | idx = len(self.queue) - 1 289 | finally: 290 | self.queue_lock.release() 291 | return idx 292 | 293 | @property 294 | def vectorized(self): 295 | return self._vectorized 296 | 297 | @property 298 | def queue(self): 299 | return self._queue 300 | 301 | @property 302 | def queue_lock(self): 303 | return self._queue_lock 304 | 305 | def get(self, number): 306 | try: 307 | self._result_lock.acquire() 308 | value = self._results[number] 309 | finally: 310 | self._result_lock.release() 311 | return value 312 | 313 | def flush_queue(self): 314 | self._results = [] 315 | if self._queue: 316 | json_data = json_encode(self.queue) 317 | send(json_data) 318 | 319 | json_reply = receive() 320 | decoded = json_decode(json_reply) 321 | if decoded.get('error', False): 322 | sys.stderr('ERROR: ' + decoded['error']) 323 | sys.exit(1) 324 | self._results = decoded['values'] 325 | 326 | 327 | def make_piped_function(mgr): 328 | def piped_function_eval(**kwargs): 329 | """Returns a function evaluated through a pipe with arguments args. 330 | 331 | args must be a namedtuple.""" 332 | if mgr.vectorized: 333 | # add to mgr queue 334 | number = mgr.add_to_queue(**kwargs) 335 | # signal mgr to continue 336 | mgr.semaphore.release() 337 | 338 | # wait for results 339 | mgr.processed_semaphore.acquire() 340 | return mgr.get(number) 341 | else: 342 | return mgr.pipe_eval(**kwargs) 343 | return piped_function_eval 344 | 345 | 346 | if __name__ == '__main__': 347 | import doctest 348 | doctest.testmod() 349 | -------------------------------------------------------------------------------- /optunity/constraints.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | 3 | # Copyright (c) 2014 KU Leuven, ESAT-STADIUS 4 | # All rights reserved. 5 | # 6 | # Redistribution and use in source and binary forms, with or without 7 | # modification, are permitted provided that the following conditions 8 | # are met: 9 | # 10 | # 1. Redistributions of source code must retain the above copyright 11 | # notice, this list of conditions and the following disclaimer. 12 | # 13 | # 2. Redistributions in binary form must reproduce the above copyright 14 | # notice, this list of conditions and the following disclaimer in the 15 | # documentation and/or other materials provided with the distribution. 16 | # 17 | # 3. Neither name of copyright holders nor the names of its contributors 18 | # may be used to endorse or promote products derived from this software 19 | # without specific prior written permission. 20 | # 21 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 22 | # ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 23 | # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 24 | # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR 25 | # CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, 26 | # EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, 27 | # PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR 28 | # PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF 29 | # LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 30 | # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 31 | # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 32 | 33 | """ 34 | All functionality related to domain constraints on objective function. 35 | 36 | Main features in this module: 37 | 38 | * :func:`constrained` 39 | * :func:`wrap_constraints` 40 | 41 | .. moduleauthor:: Marc Claesen 42 | """ 43 | 44 | import functools 45 | from . import functions 46 | 47 | def constr_ub_o(field, bounds, *args, **kwargs): 48 | """Models ``args.field < bounds``.""" 49 | return kwargs[field] < bounds 50 | 51 | 52 | def constr_ub_c(field, bounds, *args, **kwargs): 53 | """Models ``args.field <= bounds``.""" 54 | return kwargs[field] <= bounds 55 | 56 | 57 | def constr_lb_o(field, bounds, *args, **kwargs): 58 | """Models ``args.field > bounds``.""" 59 | return kwargs[field] > bounds 60 | 61 | 62 | def constr_lb_c(field, bounds, *args, **kwargs): 63 | """Models ``args.field >= bounds``.""" 64 | return kwargs[field] >= bounds 65 | 66 | 67 | def constr_range_oo(field, bounds, *args, **kwargs): 68 | """Models ``args.field in (bounds[0], bounds[1])``.""" 69 | return kwargs[field] > bounds[0] and kwargs[field] < bounds[1] 70 | 71 | 72 | def constr_range_cc(field, bounds, *args, **kwargs): 73 | """Models ``args.field in [bounds[0], bounds[1]]``.""" 74 | return kwargs[field] >= bounds[0] and kwargs[field] <= bounds[1] 75 | 76 | 77 | def constr_range_oc(field, bounds, *args, **kwargs): 78 | """Models ``args.field in (bounds[0], bounds[1]]``.""" 79 | return kwargs[field] > bounds[0] and kwargs[field] <= bounds[1] 80 | 81 | 82 | def constr_range_co(field, bounds, **kwargs): 83 | """Models ``args.field in [bounds[0], bounds[1])``.""" 84 | return kwargs[field] >= bounds[0] and kwargs[field] < bounds[1] 85 | 86 | 87 | class ConstraintViolation(Exception): 88 | """Thrown when constraints are not met.""" 89 | def __init__(self, constraint, *args, **kwargs): 90 | self.__constraint = constraint 91 | self.__args = args 92 | self.__kwargs = kwargs 93 | 94 | @property 95 | def args(self): 96 | return self.__args 97 | 98 | @property 99 | def constraint(self): 100 | return self.__constraint 101 | 102 | @property 103 | def kwargs(self): 104 | return self.__kwargs 105 | 106 | 107 | def constrained(constraints): 108 | """Decorator that puts constraints on the domain of f. 109 | 110 | >>> @constrained([lambda x: x > 0]) 111 | ... def f(x): return x+1 112 | >>> f(1) 113 | 2 114 | >>> f(0) #doctest:+SKIP 115 | Traceback (most recent call last): 116 | ... 117 | ConstraintViolation 118 | >>> len(f.constraints) 119 | 1 120 | 121 | """ 122 | def wrapper(f): 123 | @functions.wraps(f) 124 | def wrapped_f(*args, **kwargs): 125 | violations = [c for c in wrapped_f.constraints 126 | if not c(*args, **kwargs)] 127 | if violations: 128 | raise ConstraintViolation(violations, *args, **kwargs) 129 | return f(*args, **kwargs) 130 | wrapped_f.constraints = constraints 131 | return wrapped_f 132 | return wrapper 133 | 134 | 135 | def violations_defaulted(default): 136 | """Decorator to default function value when a :class:`ConstraintViolation` occurs. 137 | 138 | >>> @violations_defaulted("foobar") 139 | ... @constrained([lambda x: x > 0]) 140 | ... def f(x): return x+1 141 | >>> f(1) 142 | 2 143 | >>> f(0) 144 | 'foobar' 145 | 146 | """ 147 | def wrapper(f): 148 | @functions.wraps(f) 149 | def wrapped_f(*args, **kwargs): 150 | try: 151 | return f(*args, **kwargs) 152 | except ConstraintViolation: 153 | return default 154 | return wrapped_f 155 | return wrapper 156 | 157 | 158 | def wrap_constraints(f, default=None, ub_o=None, ub_c=None, 159 | lb_o=None, lb_c=None, range_oo=None, 160 | range_co=None, range_oc=None, range_cc=None, 161 | custom=None): 162 | """Decorates f with given input domain constraints. 163 | 164 | :param f: the function that will be constrained 165 | :type f: callable 166 | :param default: function value to default to in case of constraint violations 167 | :type default: number 168 | :param ub_o: open upper bound constraints, e.g. :math:`x < c` 169 | :type ub_o: dict 170 | :param ub_c: closed upper bound constraints, e.g. :math:`x \leq c` 171 | :type ub_c: dict 172 | :param lb_o: open lower bound constraints, e.g. :math:`x > c` 173 | :type lb_o: dict 174 | :param lb_c: closed lower bound constraints, e.g. :math:`x \geq c` 175 | :type lb_c: dict 176 | :param range_oo: range constraints (open lb and open ub) 177 | :math:`lb < x < ub` 178 | :type range_oo: dict with 2-element lists as values ([lb, ub]) 179 | :param range_co: range constraints (closed lb and open ub) 180 | :math:`lb \leq x < ub` 181 | :type range_co: dict with 2-element lists as values ([lb, ub]) 182 | :param range_oc: range constraints (open lb and closed ub) 183 | :math:`lb < x \leq ub` 184 | :type range_oc: dict with 2-element lists as values ([lb, ub]) 185 | :param range_cc: range constraints (closed lb and closed ub) 186 | :math:`lb \leq x \leq ub` 187 | :type range_cc: dict with 2-element lists as values ([lb, ub]) 188 | :param custom: custom, user-defined constraints 189 | :type custom: list of constraints 190 | 191 | *custom constraints are binary functions that yield False in case of violations. 192 | 193 | >>> def f(x): 194 | ... return x 195 | >>> fc = wrap_constraints(f, default=-1, range_oc={'x': [0, 1]}) 196 | >>> fc(x=0.5) 197 | 0.5 198 | >>> fc(x=1) 199 | 1 200 | >>> fc(x=5) 201 | -1 202 | >>> fc(x=0) 203 | -1 204 | 205 | We can define any custom constraint that we want. For instance, 206 | assume we have a binary function with arguments `x` and `y`, and we want 207 | to make sure that the provided values remain within the unit circle. 208 | 209 | >>> def f(x, y): 210 | ... return x + y 211 | >>> circle_constraint = lambda x, y: (x ** 2 + y ** 2) <= 1 212 | >>> fc = wrap_constraints(f, default=1234, custom=[circle_constraint]) 213 | >>> fc(0.0, 0.0) 214 | 0.0 215 | >>> fc(1.0, 0.0) 216 | 1.0 217 | >>> fc(0.5, 0.5) 218 | 1.0 219 | >>> fc(1, 0.5) 220 | 1234 221 | 222 | """ 223 | kwargs = locals() 224 | del kwargs['f'] 225 | del kwargs['default'] 226 | del kwargs['custom'] 227 | for k, v in list(kwargs.items()): 228 | if v is None: 229 | del kwargs[k] 230 | 231 | if not kwargs and not custom: 232 | return f 233 | 234 | # jump table to get the right constraint function 235 | jt = {'ub_o': constr_ub_o, 236 | 'ub_c': constr_ub_c, 237 | 'lb_o': constr_lb_o, 238 | 'lb_c': constr_lb_c, 239 | 'range_oo': constr_range_oo, 240 | 'range_oc': constr_range_oc, 241 | 'range_co': constr_range_co, 242 | 'range_cc': constr_range_cc} 243 | 244 | # construct constraint list 245 | constraints = [] 246 | for constr_name, pars in kwargs.items(): 247 | constr_fun = jt[constr_name] 248 | for field, bounds in pars.items(): 249 | constraints.append(functools.partial(constr_fun, 250 | field=field, 251 | bounds=bounds)) 252 | if custom: 253 | constraints.extend(custom) 254 | 255 | # wrap function 256 | if default is None: 257 | @constrained(constraints) 258 | @functions.wraps(f) 259 | def func(*args, **kwargs): 260 | return f(*args, **kwargs) 261 | else: 262 | @violations_defaulted(default) 263 | @constrained(constraints) 264 | @functions.wraps(f) 265 | def func(*args, **kwargs): 266 | return f(*args, **kwargs) 267 | return func 268 | 269 | -------------------------------------------------------------------------------- /optunity/functions.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | 3 | # Copyright (c) 2014 KU Leuven, ESAT-STADIUS 4 | # All rights reserved. 5 | # 6 | # Redistribution and use in source and binary forms, with or without 7 | # modification, are permitted provided that the following conditions 8 | # are met: 9 | # 10 | # 1. Redistributions of source code must retain the above copyright 11 | # notice, this list of conditions and the following disclaimer. 12 | # 13 | # 2. Redistributions in binary form must reproduce the above copyright 14 | # notice, this list of conditions and the following disclaimer in the 15 | # documentation and/or other materials provided with the distribution. 16 | # 17 | # 3. Neither name of copyright holders nor the names of its contributors 18 | # may be used to endorse or promote products derived from this software 19 | # without specific prior written permission. 20 | # 21 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 22 | # ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 23 | # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 24 | # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR 25 | # CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, 26 | # EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, 27 | # PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR 28 | # PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF 29 | # LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 30 | # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 31 | # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 32 | 33 | 34 | """A variety of useful function decorators for logging and more. 35 | 36 | Main features in this module: 37 | 38 | * :func:`logged` 39 | * :func:`max_evals` 40 | 41 | .. moduleauthor:: Marc Claesen 42 | """ 43 | 44 | import collections 45 | import functools 46 | import threading 47 | import operator as op 48 | 49 | 50 | try: 51 | import pandas 52 | _pandas_available = True 53 | except: 54 | _pandas_available = False 55 | 56 | # http://stackoverflow.com/a/28752007 57 | def wraps(obj, attr_names=functools.WRAPPER_ASSIGNMENTS): 58 | """Safe version of , that can deal with missing attributes 59 | such as missing __module__ and __name__. 60 | """ 61 | return functools.wraps(obj, assigned=(name for name in attr_names 62 | if hasattr(obj, name))) 63 | 64 | 65 | class Args(object): 66 | """Class to model arguments to a function evaluation. 67 | Objects of this class are hashable and can be used as dict keys. 68 | 69 | Arguments and keyword arguments are stored in a frozenset. 70 | """ 71 | 72 | def __init__(self, *args, **kwargs): 73 | d = kwargs.copy() 74 | d.update(dict([('pos_' + str(i), item) 75 | for i, item in enumerate(args)])) 76 | self._parameters = frozenset(sorted(d.items())) 77 | 78 | @property 79 | def parameters(self): 80 | """Returns the internal representation.""" 81 | return self._parameters 82 | 83 | def __hash__(self): 84 | return hash(self.parameters) 85 | 86 | def __eq__(self, other): 87 | return (self.parameters) == (other.parameters) 88 | 89 | def __iter__(self): 90 | for x in self.parameters: 91 | yield x 92 | 93 | def __str__(self): 94 | return "{" + ", ".join(['\'' + str(k) + '\'' + ': ' + str(v) 95 | for k, v in sorted(self.parameters)]) + "}" 96 | 97 | def _asdict(self): 98 | return dict([(k, v) for k, v in self.parameters]) 99 | 100 | def keys(self): 101 | """Returns a list of argument names.""" 102 | return map(op.itemgetter(0), self.parameters) 103 | 104 | def values(self): 105 | """Returns a list of argument values.""" 106 | return map(op.itemgetter(1), self.parameters) 107 | 108 | 109 | class CallLog(object): 110 | """Thread-safe call log. 111 | 112 | The call log is an ordered dictionary containing all previous function calls. 113 | Its keys are dictionaries representing the arguments and its values are the 114 | function values. As dictionaries can't be used as keys in dictionaries, 115 | a custom internal representation is used. 116 | """ 117 | 118 | def __init__(self): 119 | """Initialize an empty CallLog.""" 120 | self._data =collections.OrderedDict() 121 | self._lock = threading.Lock() 122 | 123 | @property 124 | def lock(self): 125 | return self._lock 126 | 127 | @property 128 | def data(self): 129 | """Access internal data after obtaining lock.""" 130 | with self.lock: 131 | return self._data 132 | 133 | def delete(self, *args, **kwargs): 134 | del self.data[Args(*args, **kwargs)] 135 | 136 | def get(self, *args, **kwargs): 137 | """Returns the result of given evaluation or None if not previously done.""" 138 | return self.data.get(Args(*args, **kwargs), None) 139 | 140 | def __setitem__(self, key, value): 141 | """Sets key=value in internal dictionary. 142 | 143 | :param key: key in the internal dictionary 144 | :type key: Args 145 | :param value: value in the internal dictionary 146 | :type value: float 147 | """ 148 | assert(type(key) is Args) 149 | self.data[key] = value 150 | 151 | def __getitem__(self, key): 152 | """Returns the value corresponding to key. Can throw KeyError. 153 | 154 | :param key: arguments to retrieve function value for 155 | :type key: Args 156 | """ 157 | assert(type(key) is Args) 158 | return self.data[key] 159 | 160 | def insert(self, value, *args, **kwargs): 161 | self.data[Args(*args, **kwargs)] = value 162 | 163 | def __iter__(self): 164 | for k, v in self.data: 165 | yield (dict([(key, val) for key, val in k]), v) 166 | 167 | def __len__(self): 168 | return len(self.data) 169 | 170 | def __nonzero__(self): 171 | return bool(self.data) 172 | 173 | def __str__(self): 174 | return "\n".join([str(k) + ' --> ' + str(v) 175 | for k, v in self.data.items()]) 176 | 177 | def keys(self): 178 | return self.data.keys() 179 | 180 | def values(self): 181 | return self.data.values() 182 | 183 | def items(self): 184 | return self.data.items() 185 | 186 | def update(self, other): 187 | assert(type(other) is CallLog) 188 | self.data.update(other.data) 189 | 190 | @staticmethod 191 | def from_dict(d): 192 | """Converts given dict to a valid call log used by logged functions. 193 | 194 | Given dictionary must have the following structure: 195 | ``{'args': {'argname': []}, 'values': []}`` 196 | 197 | >>> log = CallLog.from_dict({'args': {'x': [1, 2]}, 'values': [2, 3]}) 198 | >>> print(log) 199 | {'x': 1} --> 2 200 | {'x': 2} --> 3 201 | 202 | """ 203 | log = CallLog() 204 | keys = d['args'].keys() 205 | for k, v in zip(zip(*d['args'].values()), d['values']): 206 | args = dict([(key, val) for key, val in zip(keys, k)]) 207 | log.insert(v, **args) 208 | return log 209 | 210 | def to_dict(self): 211 | """Returns given call_log into a dictionary. 212 | 213 | The result is a dict with the following structure: 214 | ``{'args': {'argname': []}, 'values': []}`` 215 | 216 | >>> call_log = CallLog() 217 | >>> call_log.insert(3, x=1, y=2) 218 | >>> d = call_log.to_dict() 219 | >>> d['args']['x'] 220 | [1] 221 | >>> d['args']['y'] 222 | [2] 223 | >>> d['values'] 224 | [3] 225 | 226 | """ 227 | if self.data: 228 | args = dict([(k, []) for k in list(self.keys())[0].keys()]) 229 | values = [] 230 | for k, v in self.data.items(): 231 | for key, value in k: 232 | args[key].append(value) 233 | values.append(v) 234 | return {'args': args, 'values': values} 235 | else: 236 | return {'args': {}, 'values': []} 237 | 238 | 239 | def logged(f): 240 | """Decorator that logs unique calls to ``f``. 241 | 242 | The call log can always be retrieved using ``f.call_log``. 243 | Decorating a function that is already being logged has 244 | no effect. 245 | 246 | The call log is an instance of CallLog. 247 | 248 | >>> @logged 249 | ... def f(x): return x+1 250 | >>> a, b, c = f(1), f(1), f(2) 251 | >>> print(f.call_log) 252 | {'pos_0': 1} --> 2 253 | {'pos_0': 2} --> 3 254 | 255 | logged as inner decorator: 256 | 257 | >>> from .constraints import constrained 258 | >>> @logged 259 | ... @constrained([lambda x: x > 1]) 260 | ... def f2(x): return x+1 261 | >>> len(f2.call_log) 262 | 0 263 | >>> f2(2) 264 | 3 265 | >>> print(f2.call_log) 266 | {'pos_0': 2} --> 3 267 | 268 | logged as outer decorator: 269 | 270 | >>> from .constraints import constrained 271 | >>> @constrained([lambda x: x > 1]) 272 | ... @logged 273 | ... def f3(x): return x+1 274 | >>> len(f3.call_log) 275 | 0 276 | >>> f3(2) 277 | 3 278 | >>> print(f3.call_log) 279 | {'pos_0': 2} --> 3 280 | 281 | >>> @logged 282 | ... def f(x): return 1 283 | >>> f(1) 284 | 1 285 | >>> print(f.call_log) 286 | {'pos_0': 1} --> 1 287 | >>> @logged 288 | ... @wraps(f) 289 | ... def f2(x): return f(x) 290 | >>> print(f2.call_log) 291 | {'pos_0': 1} --> 1 292 | 293 | """ 294 | if hasattr(f, 'call_log'): 295 | return f 296 | 297 | @wraps(f) 298 | def wrapped_f(*args, **kwargs): 299 | value = wrapped_f.call_log.get(*args, **kwargs) 300 | if value is None: 301 | value = f(*args, **kwargs) 302 | wrapped_f.call_log.insert(value, *args, **kwargs) 303 | return value 304 | wrapped_f.call_log = CallLog() 305 | return wrapped_f 306 | 307 | 308 | def negated(f): 309 | """Decorator to negate f such that f'(x) = -f(x).""" 310 | @wraps(f) 311 | def wrapped_f(*args, **kwargs): 312 | return -f(*args, **kwargs) 313 | return wrapped_f 314 | 315 | 316 | class MaximumEvaluationsException(Exception): 317 | """Raised when the maximum number of function evaluations are used.""" 318 | def __init__(self, max_evals): 319 | self._max_evals = max_evals 320 | 321 | @property 322 | def max_evals(self): 323 | """Returns the maximum number of evaluations that was permitted.""" 324 | return self._max_evals 325 | 326 | 327 | def max_evals(max_evals): 328 | """Decorator to enforce a maximum number of function evaluations. 329 | 330 | Throws a MaximumEvaluationsException during evaluations after 331 | the maximum is reached. Adds a field ``f.num_evals`` which tracks 332 | the number of evaluations that have been performed. 333 | 334 | >>> @max_evals(1) 335 | ... def f(x): return 2 336 | >>> f(2) 337 | 2 338 | >>> f(1) #doctest:+SKIP 339 | Traceback (most recent call last): 340 | ... 341 | MaximumEvaluationsException 342 | >>> try: 343 | ... f(1) 344 | ... except MaximumEvaluationsException as e: 345 | ... e.max_evals 346 | 1 347 | 348 | """ 349 | def wrapper(f): 350 | @wraps(f) 351 | def wrapped_f(*args, **kwargs): 352 | if wrapped_f.num_evals >= max_evals: 353 | raise MaximumEvaluationsException(max_evals) 354 | else: 355 | wrapped_f.num_evals += 1 356 | return f(*args, **kwargs) 357 | wrapped_f.num_evals = 0 358 | return wrapped_f 359 | return wrapper 360 | 361 | 362 | def static_key_order(keys): 363 | """Decorator to fix the key order for use in function evaluations. 364 | 365 | A fixed key order allows the function to be evaluated with a list of 366 | unnamed arguments rather than kwargs. 367 | 368 | >>> @static_key_order(['foo', 'bar']) 369 | ... def f(bar, foo): return bar + 2 * foo 370 | >>> f(3, 5) 371 | 11 372 | 373 | """ 374 | def wrapper(f): 375 | @wraps(f) 376 | def wrapped_f(*args): 377 | return f(**dict([(k, v) for k, v in zip(keys, args)])) 378 | return wrapped_f 379 | return wrapper 380 | 381 | 382 | def call_log2dataframe(log): 383 | """Converts a call log into a pandas data frame. 384 | This function errors if you don't have pandas available. 385 | 386 | :param log: call log to be converted, as returned by e.g. `optunity.minimize` 387 | :returns: a pandas data frame capturing the same information as the call log 388 | 389 | """ 390 | if not _pandas_available: 391 | raise NotImplementedError('This function requires pandas') 392 | 393 | args = log['args'] 394 | values = log['values'] 395 | hpar_names = args.keys() 396 | 397 | # construct a list of dictionaries 398 | zipped= zip(zip(*args.values()), values) 399 | dictlist = [dict([(k, v) for k, v in zip(hpar_names, args)] + [('value', value)]) 400 | for args, value in zipped] 401 | df = pandas.DataFrame(dictlist) 402 | return df 403 | 404 | 405 | if __name__ == '__main__': 406 | pass 407 | -------------------------------------------------------------------------------- /optunity/metrics.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | 3 | # Author: Marc Claesen 4 | # 5 | # Copyright (c) 2014 KU Leuven, ESAT-STADIUS 6 | # All rights reserved. 7 | # 8 | # Redistribution and use in source and binary forms, with or without 9 | # modification, are permitted provided that the following conditions 10 | # are met: 11 | # 12 | # 1. Redistributions of source code must retain the above copyright 13 | # notice, this list of conditions and the following disclaimer. 14 | # 15 | # 2. Redistributions in binary form must reproduce the above copyright 16 | # notice, this list of conditions and the following disclaimer in the 17 | # documentation and/or other materials provided with the distribution. 18 | # 19 | # 3. Neither name of copyright holders nor the names of its contributors 20 | # may be used to endorse or promote products derived from this software 21 | # without specific prior written permission. 22 | # 23 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 24 | # ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 25 | # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 26 | # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR 27 | # CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, 28 | # EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, 29 | # PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR 30 | # PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF 31 | # LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 32 | # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 33 | # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 34 | 35 | import math 36 | import operator as op 37 | 38 | def contingency_tables(ys, decision_values, positive=True, presorted=False): 39 | """Computes contingency tables for every unique decision value. 40 | 41 | :param ys: true labels 42 | :type ys: iterable 43 | :param decision_values: decision values (higher = stronger positive) 44 | :type decision_values: iterable 45 | :param positive: the positive label 46 | :param presorted: whether or not ys and yhat are already sorted 47 | :type presorted: bool 48 | 49 | :returns: a list of contingency tables `(TP, FP, TN, FN)` and the corresponding thresholds. 50 | Contingency tables are built based on decision :math:`decision\_value \geq threshold`. 51 | 52 | The first contingency table corresponds with a (potentially unseen) threshold that yields all negatives. 53 | 54 | >>> y = [0, 0, 0, 0, 1, 1, 1, 1] 55 | >>> d = [2, 2, 1, 1, 1, 2, 3, 3] 56 | >>> tables, thresholds = contingency_tables(y, d, 1) 57 | >>> print(tables) 58 | [(0, 0, 4, 4), (2, 0, 4, 2), (3, 2, 2, 1), (4, 4, 0, 0)] 59 | >>> print(thresholds) 60 | [None, 3, 2, 1] 61 | 62 | """ 63 | if presorted: 64 | if decision_values[0] > decision_values[-1]: 65 | ind = range(len(decision_values)) 66 | srt = decision_values 67 | else: 68 | ind = reversed(range(len(decision_values))) 69 | srt = reversed(decision_values) 70 | else: 71 | # sort decision values 72 | ind, srt = zip(*sorted(enumerate(decision_values), reverse=True, 73 | key=op.itemgetter(1))) 74 | 75 | # resort y 76 | y = list(map(lambda x: ys[x] == positive, ind)) 77 | 78 | num_instances = len(ind) 79 | total_num_pos = sum(y) 80 | 81 | thresholds = [None] 82 | tables = [(0, 0, num_instances - total_num_pos, total_num_pos)] 83 | 84 | current_idx = 0 85 | while current_idx < num_instances: 86 | # determine number of identical decision values 87 | num_ties = 1 88 | while current_idx + num_ties < num_instances and srt[current_idx + num_ties] == srt[current_idx]: 89 | num_ties += 1 90 | 91 | if current_idx == 0: 92 | previous_table = (0, 0, num_instances - total_num_pos, total_num_pos) 93 | 94 | # find number of new true positives at this threshold 95 | num_pos = 0 96 | for i in range(current_idx, current_idx + num_ties): 97 | num_pos += y[i] 98 | 99 | # difference compared to previous contingency_table 100 | diff = (num_pos, num_ties - num_pos, - num_ties + num_pos, -num_pos) 101 | 102 | new_table = tuple(map(op.add, previous_table, diff)) 103 | tables.append(new_table[:]) 104 | thresholds.append(srt[current_idx]) 105 | 106 | # prepare for next iteration 107 | previous_table = new_table 108 | current_idx += num_ties 109 | 110 | return tables, thresholds 111 | 112 | 113 | def compute_curve(ys, decision_values, xfun, yfun, positive=True, presorted=False): 114 | """Computes a curve based on contingency tables at different decision values. 115 | 116 | :param ys: true labels 117 | :type ys: iterable 118 | :param decision_values: decision values 119 | :type decision_values: iterable 120 | :param positive: positive label 121 | :param xfun: function to compute x values, based on contingency tables 122 | :type xfun: callable 123 | :param yfun: function to compute y values, based on contingency tables 124 | :type yfun: callable 125 | :param presorted: whether or not ys and yhat are already sorted 126 | :type presorted: bool 127 | 128 | :returns: the resulting curve, as a list of (x, y)-tuples 129 | 130 | """ 131 | curve = [] 132 | tables, _ = contingency_tables(ys, decision_values, positive, presorted) 133 | curve = list(map(lambda t: (xfun(t), yfun(t)), tables)) 134 | return curve 135 | 136 | 137 | def auc(curve): 138 | """Computes the area under the specified curve. 139 | 140 | :param curve: a curve, specified as a list of (x, y) tuples 141 | :type curve: [(x, y), ...] 142 | 143 | .. seealso:: :func:`optunity.score_functions.compute_curve` 144 | 145 | """ 146 | area = 0.0 147 | for i in range(len(curve) - 1): 148 | x1, y1 = curve[i] 149 | x2, y2 = curve[i + 1] 150 | if y1 is None: 151 | y1 = 0.0 152 | area += float(min(y1, y2)) * float(x2 - x1) + math.fabs(float(y2 - y1)) * float(x2 - x1) / 2 153 | 154 | return area 155 | 156 | 157 | def contingency_table(ys, yhats, positive=True): 158 | """Computes a contingency table for given predictions. 159 | 160 | :param ys: true labels 161 | :type ys: iterable 162 | :param yhats: predicted labels 163 | :type yhats: iterable 164 | :param positive: the positive label 165 | 166 | :return: TP, FP, TN, FN 167 | 168 | >>> ys = [True, True, True, True, True, False] 169 | >>> yhats = [True, True, False, False, False, True] 170 | >>> tab = contingency_table(ys, yhats, 1) 171 | >>> print(tab) 172 | (2, 1, 0, 3) 173 | 174 | """ 175 | TP = 0 176 | TN = 0 177 | FP = 0 178 | FN = 0 179 | for y, yhat in zip(ys, yhats): 180 | if y == positive: 181 | if y == yhat: 182 | TP += 1 183 | else: 184 | FN += 1 185 | else: 186 | if y == yhat: 187 | TN += 1 188 | else: 189 | FP += 1 190 | return TP, FP, TN, FN 191 | 192 | def _precision(table): 193 | TP = table[0] 194 | FP = table[1] 195 | try: 196 | return float(TP) / (TP + FP) 197 | except ZeroDivisionError: 198 | return None 199 | 200 | def _recall(table): 201 | TP = table[0] 202 | FN = table[3] 203 | try: 204 | return float(TP) / (TP + FN) 205 | except ZeroDivisionError: 206 | return None 207 | 208 | def _fpr(table): 209 | FP = table[1] 210 | TN = table[2] 211 | return float(FP) / (FP + TN) 212 | 213 | 214 | def mse(y, yhat): 215 | """Returns the mean squared error between y and yhat. 216 | 217 | :param y: true function values 218 | :param yhat: predicted function values 219 | :returns: 220 | .. math:: \\frac{1}{n} \sum_{i=1}^n \\big[(\hat{y}-y)^2\\big] 221 | 222 | Lower is better. 223 | 224 | >>> mse([0, 0], [2, 3]) 225 | 6.5 226 | 227 | """ 228 | return float(sum([(l - p) ** 2 229 | for l, p in zip(y, yhat)])) / len(y) 230 | 231 | 232 | def absolute_error(y, yhat): 233 | """Returns the maximal absolute error between y and yhat. 234 | 235 | :param y: true function values 236 | :param yhat: predicted function values 237 | 238 | Lower is better. 239 | 240 | >>> absolute_error([0,1,2,3], [0,0,1,1]) 241 | 2.0 242 | 243 | """ 244 | return float(max(map(lambda x, y: math.fabs(x-y), y, yhat))) 245 | 246 | 247 | def accuracy(y, yhat): 248 | """Returns the accuracy. Higher is better. 249 | 250 | :param y: true function values 251 | :param yhat: predicted function values 252 | 253 | """ 254 | return float(sum(map(lambda x: x[0] == x[1], 255 | zip(y, yhat)))) / len(y) 256 | 257 | 258 | def logloss(y, yhat): 259 | """Returns the log loss between labels and predictions. 260 | 261 | :param y: true function values 262 | :param yhat: predicted function values 263 | :returns: 264 | .. math:: -\\frac{1}{n}\sum_{i=1}^n\\big[y \\times \log \hat{y}+(1-y) \\times \log (1-\hat{y})\\big] 265 | 266 | y must be a binary vector, e.g. elements in {True, False} 267 | yhat must be a vector of probabilities, e.g. elements in [0, 1] 268 | 269 | Lower is better. 270 | 271 | .. note:: This loss function should only be used for probabilistic models. 272 | 273 | """ 274 | loss = sum([math.log(pred) for _, pred in 275 | filter(lambda i: i[0], zip(y, yhat))]) 276 | loss += sum([math.log(1 - pred) for _, pred in 277 | filter(lambda i: not i[0], zip(y, yhat))]) 278 | return - float(loss) / len(y) 279 | 280 | 281 | def brier(y, yhat, positive=True): 282 | """Returns the Brier score between y and yhat. 283 | 284 | :param y: true function values 285 | :param yhat: predicted function values 286 | :returns: 287 | .. math:: \\frac{1}{n} \sum_{i=1}^n \\big[(\hat{y}-y)^2\\big] 288 | 289 | yhat must be a vector of probabilities, e.g. elements in [0, 1] 290 | 291 | Lower is better. 292 | 293 | .. note:: This loss function should only be used for probabilistic models. 294 | 295 | """ 296 | y = map(lambda x: x == positive, y) 297 | return sum([(yp - float(yt)) ** 2 for yt, yp in zip(y, yhat)]) / len(y) 298 | 299 | 300 | def pu_score(y, yhat): 301 | """ 302 | Returns a score used for PU learning as introduced in [LEE2003]_. 303 | 304 | :param y: true function values 305 | :param yhat: predicted function values 306 | :returns: 307 | .. math:: \\frac{P(\hat{y}=1 | y=1)^2}{P(\hat{y}=1)} 308 | 309 | y and yhat must be boolean vectors. 310 | 311 | Higher is better. 312 | 313 | .. [LEE2003] Wee Sun Lee and Bing Liu. Learning with positive and unlabeled examples 314 | using weighted logistic regression. In Proceedings of the Twentieth 315 | International Conference on Machine Learning (ICML), 2003. 316 | """ 317 | num_pos = sum(y) 318 | p_pred_pos = float(sum(yhat)) / len(y) 319 | if p_pred_pos == 0: 320 | return 0.0 321 | tp = sum([all(x) for x in zip(y, yhat)]) 322 | return tp * tp / (num_pos * num_pos * p_pred_pos) 323 | 324 | def fbeta(y, yhat, beta, positive=True): 325 | """Returns the :math:`F_\\beta`-score. 326 | 327 | :param y: true function values 328 | :param yhat: predicted function values 329 | :param beta: the value for beta to be used 330 | :type beta: float (positive) 331 | :param positive: the positive label 332 | 333 | :returns: 334 | .. math:: (1 + \\beta^2)\\frac{cdot precision\\cdot recall}{(\\beta^2 * precision)+recall} 335 | 336 | """ 337 | bsq = beta ** 2 338 | TP, FP, _, FN = contingency_table(y, yhat, positive) 339 | return float(1 + bsq) * TP / ((1 + bsq) * TP + bsq * FN + FP) 340 | 341 | def precision(y, yhat, positive=True): 342 | """Returns the precision (higher is better). 343 | 344 | :param y: true function values 345 | :param yhat: predicted function values 346 | :param positive: the positive label 347 | 348 | :returns: number of true positive predictions / number of positive predictions 349 | 350 | """ 351 | TP, FP, _, _ = contingency_table(y, yhat, positive) 352 | return _precision(TP, FP) 353 | 354 | def recall(y, yhat, positive=True): 355 | """Returns the recall (higher is better). 356 | 357 | :param y: true function values 358 | :param yhat: predicted function values 359 | :param positive: the positive label 360 | 361 | :returns: number of true positive predictions / number of true positives 362 | 363 | """ 364 | TP, _, _, FN = contingency_table(y, yhat, positive) 365 | return _recall(TP, FN) 366 | 367 | def npv(y, yhat, positive=True): 368 | """Returns the negative predictive value (higher is better). 369 | 370 | :param y: true function values 371 | :param yhat: predicted function values 372 | :param positive: the positive label 373 | 374 | :returns: number of true negative predictions / number of negative predictions 375 | 376 | """ 377 | _, _, TN, FN = contingency_table(y, yhat, positive) 378 | return float(TN) / (TN + FN) 379 | 380 | def error_rate(y, yhat): 381 | """Returns the error rate (lower is better). 382 | 383 | :param y: true function values 384 | :param yhat: predicted function values 385 | 386 | >>> error_rate([0,0,1,1], [0,0,0,1]) 387 | 0.25 388 | 389 | """ 390 | return 1.0 - accuracy(y, yhat) 391 | 392 | def roc_auc(ys, yhat, positive=True, presorted=False, return_curve=False): 393 | """Computes the area under the receiver operating characteristic curve (higher is better). 394 | 395 | :param y: true function values 396 | :param yhat: predicted function values 397 | :param positive: the positive label 398 | :param presorted: whether or not ys and yhat are already sorted 399 | :type presorted: bool 400 | :param return_curve: whether or not the curve should be returned 401 | :type return_curve: bool 402 | 403 | >>> roc_auc([0, 0, 1, 1], [0, 0, 1, 1], 1) 404 | 1.0 405 | 406 | >>> roc_auc([0,0,1,1], [0,1,1,2], 1) 407 | 0.875 408 | 409 | """ 410 | curve = compute_curve(ys, yhat, _fpr, _recall, positive) 411 | if return_curve: 412 | return auc(curve), curve 413 | else: 414 | return auc(curve) 415 | 416 | 417 | def pr_auc(ys, yhat, positive=True, presorted=False, return_curve=False): 418 | """Computes the area under the precision-recall curve (higher is better). 419 | 420 | :param y: true function values 421 | :param yhat: predicted function values 422 | :param positive: the positive label 423 | :param presorted: whether or not ys and yhat are already sorted 424 | :type presorted: bool 425 | :param return_curve: whether or not the curve should be returned 426 | :type return_curve: bool 427 | 428 | >>> pr_auc([0, 0, 1, 1], [0, 0, 1, 1], 1) 429 | 1.0 430 | 431 | >>> round(pr_auc([0,0,1,1], [0,1,1,2], 1), 2) 432 | 0.92 433 | 434 | .. note:: Precision is undefined at recall = 0. 435 | In this case, we set precision equal to the precision that was obtained at the lowest non-zero recall. 436 | 437 | """ 438 | curve = compute_curve(ys, yhat, _recall, _precision, positive, presorted) 439 | # precision is undefined when no positives are predicted 440 | # we approximate by using the precision at the lowest recall 441 | curve[0] = (0.0, curve[1][1]) 442 | if return_curve: 443 | return auc(curve), curve 444 | else: 445 | return auc(curve) 446 | 447 | def r_squared(y, yhat): 448 | """Returns the R squared statistic, also known as coefficient of determination (higher is better). 449 | 450 | :param y: true function values 451 | :param yhat: predicted function values 452 | :param positive: the positive label 453 | 454 | :returns: 455 | .. math:: R^2 = 1-\\frac{SS_{res}}{SS_{tot}} = 1-\\frac{\sum_i (y_i - yhat_i)^2}{\sum_i (y_i - mean(y))^2} 456 | 457 | """ 458 | ymean = float(sum(y)) / len(y) 459 | SStot = sum(map(lambda yi: (yi-ymean) ** 2, y)) 460 | SSres = sum(map(lambda yi, fi: (yi-fi) ** 2, y, yhat)) 461 | return 1.0 - SSres / SStot 462 | -------------------------------------------------------------------------------- /optunity/parallel.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | 3 | # Copyright (c) 2014 KU Leuven, ESAT-STADIUS 4 | # All rights reserved. 5 | # 6 | # Redistribution and use in source and binary forms, with or without 7 | # modification, are permitted provided that the following conditions 8 | # are met: 9 | # 10 | # 1. Redistributions of source code must retain the above copyright 11 | # notice, this list of conditions and the following disclaimer. 12 | # 13 | # 2. Redistributions in binary form must reproduce the above copyright 14 | # notice, this list of conditions and the following disclaimer in the 15 | # documentation and/or other materials provided with the distribution. 16 | # 17 | # 3. Neither name of copyright holders nor the names of its contributors 18 | # may be used to endorse or promote products derived from this software 19 | # without specific prior written permission. 20 | # 21 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 22 | # ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 23 | # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 24 | # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR 25 | # CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, 26 | # EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, 27 | # PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR 28 | # PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF 29 | # LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 30 | # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 31 | # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 32 | 33 | import threading 34 | import copy 35 | import functools 36 | 37 | __all__ = ['pmap', 'Future', 'create_pmap'] 38 | 39 | def _fun(f, q_in, q_out): 40 | while True: 41 | i, x = q_in.get() 42 | if i is None: 43 | break 44 | value = f(*x) 45 | if hasattr(f, 'call_log'): 46 | k = list(f.call_log.keys())[-1] 47 | q_out.put((i, value, k)) 48 | else: 49 | q_out.put((i, value)) 50 | 51 | try: 52 | import multiprocessing 53 | 54 | # http://stackoverflow.com/a/16071616 55 | def pmap(f, *args, **kwargs): 56 | """Parallel map using multiprocessing. 57 | 58 | :param f: the callable 59 | :param args: arguments to f, as iterables 60 | :returns: a list containing the results 61 | 62 | .. warning:: 63 | This function will not work in IPython: https://github.com/claesenm/optunity/issues/8. 64 | 65 | .. warning:: 66 | Python's multiprocessing library is incompatible with Jython. 67 | 68 | """ 69 | nprocs = kwargs.get('number_of_processes', multiprocessing.cpu_count()) 70 | # nprocs = multiprocessing.cpu_count() 71 | q_in = multiprocessing.Queue(1) 72 | q_out = multiprocessing.Queue() 73 | 74 | proc = [multiprocessing.Process(target=_fun, args=(f, q_in, q_out)) 75 | for _ in range(nprocs)] 76 | for p in proc: 77 | p.daemon = True 78 | p.start() 79 | 80 | sent = [q_in.put((i, x)) for i, x in enumerate(zip(*args))] 81 | [q_in.put((None, None)) for _ in range(nprocs)] 82 | res = [q_out.get() for _ in range(len(sent))] 83 | [p.join() for p in proc] 84 | 85 | # FIXME: strong coupling between pmap and functions.logged 86 | if hasattr(f, 'call_log'): 87 | for _, value, k in sorted(res): 88 | f.call_log[k] = value 89 | return [x for i, x, _ in sorted(res)] 90 | else: 91 | return [x for i, x in sorted(res)] 92 | 93 | def create_pmap(number_of_processes): 94 | def pmap_bound(f, *args): 95 | return pmap(f, *args, number_of_processes=number_of_processes) 96 | return pmap_bound 97 | 98 | # http://code.activestate.com/recipes/84317-easy-threading-with-futures/ 99 | class Future: 100 | def __init__(self,func,*param): 101 | # Constructor 102 | self.__done=0 103 | self.__result=None 104 | self.__status='working' 105 | 106 | self.__S=threading.Semaphore(0) 107 | 108 | # Run the actual function in a separate thread 109 | self.__T=threading.Thread(target=self.Wrapper, args=(func, param)) 110 | self.__T.setName("FutureThread") 111 | self.__T.daemon=True 112 | self.__T.start() 113 | 114 | def __repr__(self): 115 | return '' 116 | 117 | def __call__(self): 118 | try: 119 | self.__S.acquire() 120 | # We deepcopy __result to prevent accidental tampering with it. 121 | a=copy.deepcopy(self.__result) 122 | finally: 123 | self.__S.release() 124 | return a 125 | 126 | def join(self): 127 | self.__T.join() 128 | 129 | def Wrapper(self, func, param): 130 | # Run the actual function, and let us housekeep around it 131 | # try: 132 | self.__result=func(*param) 133 | # except: 134 | # self.__result="Exception raised within Future" 135 | self.__status=str(self.__result) 136 | self.__S.release() 137 | 138 | except ImportError: 139 | pmap = map 140 | Future = None 141 | 142 | if __name__ == '__main__': 143 | pass 144 | -------------------------------------------------------------------------------- /optunity/search_spaces.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | 3 | # Copyright (c) 2014 KU Leuven, ESAT-STADIUS 4 | # All rights reserved. 5 | # 6 | # Redistribution and use in source and binary forms, with or without 7 | # modification, are permitted provided that the following conditions 8 | # are met: 9 | # 10 | # 1. Redistributions of source code must retain the above copyright 11 | # notice, this list of conditions and the following disclaimer. 12 | # 13 | # 2. Redistributions in binary form must reproduce the above copyright 14 | # notice, this list of conditions and the following disclaimer in the 15 | # documentation and/or other materials provided with the distribution. 16 | # 17 | # 3. Neither name of copyright holders nor the names of its contributors 18 | # may be used to endorse or promote products derived from this software 19 | # without specific prior written permission. 20 | # 21 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 22 | # ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 23 | # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 24 | # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR 25 | # CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, 26 | # EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, 27 | # PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR 28 | # PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF 29 | # LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 30 | # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 31 | # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 32 | 33 | """ 34 | 35 | Functionality to deal with structured search spaces, in which the existence of some hyperparameters is contingent on some discrete choice(s). 36 | 37 | For more information on the syntax and modalities of defining structured search spaces, please refer to :doc:`/user/structured_search_spaces`. 38 | 39 | A search space is defined as a dictionary mapping strings to nodes. Each node is one of the following: 40 | 41 | 1. A new sub search space, that is a new dictionary with the same structure. 42 | 2. 2-element list or tuple, containing (lb, ub) for the associated hyperparameter. 43 | 3. None, to indicate a terminal node that has no numeric value associated to it. 44 | 45 | Internally, structured search spaces are encoded as a vector, based on the following rules: 46 | 47 | * a standard hyperparameter with box constraints is a single dimension 48 | * when a choice must be maded, this is encoded as a single dimension (regardless of the amount of options), with values in the range [0, num_choices] 49 | * choice options without additional hyperparameters (i.e., None nodes) do not require further coding 50 | 51 | For example, consider an SVM kernel family, choosing between linear and RBF. This is specified as follows: 52 | 53 | .. code:: 54 | 55 | search = {'kernel': {'linear': None, 56 | 'rbf': {'gamma': [0, 3]} 57 | } 58 | } 59 | 60 | In vector format, this is encoded as a vector with 2 entries :math:`[kernel \in [0, 2],\ gamma \in [0, 3]]`. 61 | 62 | The main API of this module is the SearchTree class. 63 | 64 | .. moduleauthor:: Marc Claesen 65 | """ 66 | 67 | import functools 68 | import itertools 69 | import collections 70 | import math 71 | 72 | DELIM = '|' 73 | 74 | class Options(object): 75 | 76 | def __init__(self, cases): 77 | self._cases = cases 78 | 79 | @property 80 | def cases(self): return self._cases 81 | 82 | def __iter__(self): 83 | for case in self.cases: 84 | yield case 85 | 86 | def __repr__(self): return "{%s}" % ", ".join(self.cases) 87 | 88 | def __len__(self): return len(self.cases) 89 | 90 | def __getitem__(self, idx): return self.cases[idx] 91 | 92 | 93 | class Node(object): 94 | """Models a node within a search space. 95 | 96 | Nodes can be internal or terminal, and may or may not be choices. 97 | A choice is a node that models a discrete choice out of k > 1 options. 98 | 99 | """ 100 | 101 | def __init__(self, key, value): 102 | self._key = key 103 | if type(value) is dict: 104 | self._value = [Node(k, v) for k, v in sorted(value.items())] 105 | else: self._value = value 106 | 107 | @property 108 | def key(self): return self._key 109 | 110 | @property 111 | def value(self): return self._value 112 | 113 | @property 114 | def terminal(self): 115 | """Returns whether or not this Node is terminal. 116 | 117 | A terminal node has a non-dictionary value (numeric, list or None). 118 | """ 119 | if self.value: 120 | return not type(self.value[0]) == type(self) 121 | return True 122 | 123 | @property 124 | def choice(self): 125 | """Determines whether this node is a choice. 126 | 127 | A choice is a node that models a discrete choice out of k > 1 options. 128 | """ 129 | return self.value is None 130 | 131 | def __iter__(self): 132 | """Iterates over this node. 133 | 134 | If the node is terminal, yields the key and value. 135 | Otherwise, first yields all values and then iterates over the values. 136 | 137 | """ 138 | if self.terminal: 139 | yield self.key, self.value 140 | else: 141 | value = list(itertools.chain(*self.value)) 142 | 143 | if any([not x.terminal or x.choice for x in self.value]): 144 | value.insert(0, ([], Options([node.key for node in self.value]))) 145 | 146 | for k, v in value: 147 | if type(k) is list: key = [self.key] + k 148 | else: key = [self.key, k] 149 | yield key, v 150 | 151 | 152 | class SearchTree(object): 153 | """Tree structure to model a search space. 154 | 155 | Fairly elaborate unit test. 156 | 157 | >>> space = {'a': {'b0': {'c0': {'d0': {'e0': [0, 10], 'e1': [-2, -1]}, 158 | ... 'd1': {'e2': [-3, -1]}, 159 | ... 'd2': None 160 | ... }, 161 | ... 'c1': [0.0, 1.0], 162 | ... }, 163 | ... 'b1': {'c2': [-2.0, -1.0]}, 164 | ... 'b2': None 165 | ... } 166 | ... } 167 | >>> tree = SearchTree(space) 168 | >>> b = tree.to_box() 169 | >>> print(b['a'] == [0.0, 3.0] and 170 | ... b['a|b0|c0'] == [0.0, 3.0] and 171 | ... b['a|b0|c1'] == [0.0, 1.0] and 172 | ... b['a|b1|c2'] == [-2.0, -1.0] and 173 | ... b['a|b0|c0|d0|e0'] == [0, 10] and 174 | ... b['a|b0|c0|d0|e1'] == [-2, -1] and 175 | ... b['a|b0|c0|d1|e2'] == [-3, -1]) 176 | True 177 | 178 | >>> d = tree.decode({'a': 2.5}) 179 | >>> d['a'] == 'b2' 180 | True 181 | 182 | >>> d = tree.decode({'a': 1.5, 'a|b1|c2': -1.5}) 183 | >>> print(d['a'] == 'b1' and 184 | ... d['c2'] == -1.5) 185 | True 186 | 187 | >>> d = tree.decode({'a': 0.5, 'a|b0|c0': 1.7, 'a|b0|c0|d1|e2': -1.2}) 188 | >>> print(d['a'] == 'b0' and 189 | ... d['c0'] == 'd1' and 190 | ... d['e2'] == -1.2) 191 | True 192 | 193 | >>> d = tree.decode({'a': 0.5, 'a|b0|c0': 2.7}) 194 | >>> print(d['a'] == 'b0' and 195 | ... d['c0'] == 'd2') 196 | True 197 | 198 | >>> d = tree.decode({'a': 0.5, 'a|b0|c0': 0.7, 'a|b0|c0|d0|e0': 2.3, 'a|b0|c0|d0|e1': -1.5}) 199 | >>> print(d['a'] == 'b0' and 200 | ... d['c0'] == 'd0' and 201 | ... d['e0'] == 2.3 and 202 | ... d['e1'] == -1.5) 203 | True 204 | 205 | """ 206 | 207 | def __init__(self, d): 208 | self._content = [Node(k, v) for k, v in sorted(d.items())] 209 | self._vectordict = collections.OrderedDict() 210 | self._vectorcontent = collections.OrderedDict() 211 | 212 | @property 213 | def vectordict(self): return self._vectordict 214 | 215 | @property 216 | def vectorcontent(self): return self._vectorcontent 217 | 218 | @property 219 | def content(self): return self._content 220 | 221 | def __iter__(self): 222 | for i in self.content: 223 | for k, v in i: 224 | yield k, v 225 | 226 | def to_box(self): 227 | """ 228 | Creates a set of box constraints to define the given search space. 229 | 230 | :returns: a set of box constraints (e.g. dictionary, with box constraint values) 231 | 232 | Use these box constraints to initialize a solver, which will then implicitly work in the 233 | vector representation of the search space. 234 | 235 | To evaluate such vector representations, decorate the objective function with :func:`optunity.search_spaces.SearchTree.decode` 236 | of the same object. 237 | 238 | """ 239 | if not self.vectordict: 240 | for k, v in self: 241 | if type(k) is list: key = DELIM.join(k) 242 | else: key = k 243 | if type(v) is Options: 244 | if len(v) > 1: # options of length one aren't really options 245 | self.vectordict[key] = [0.0, float(len(v))] 246 | self.vectorcontent[key] = v 247 | elif v is None: 248 | pass 249 | else: 250 | self.vectordict[key] = v 251 | self.vectorcontent[key] = v 252 | 253 | return dict([(k, v) for k, v in self.vectordict.items()]) 254 | 255 | def decode(self, vd): 256 | """ 257 | Decodes a vector representation (as a dictionary) into a result dictionary. 258 | 259 | :param vd: vector representation 260 | :type vd: dict 261 | :returns: dict containing the decoded representation. 262 | 263 | * Choices are given as key:value pairs. 264 | * Active hyperparameters have numeric values. 265 | * Inactive hyperparameters have value None. 266 | 267 | """ 268 | result = {} 269 | currently_decoding_nested = [] 270 | items = sorted(vd.items()) 271 | idx = 0 272 | 273 | while idx < len(items): 274 | 275 | k, v = items[idx] 276 | keylist = k.split(DELIM) 277 | 278 | if currently_decoding_nested and len(keylist) >= len(currently_decoding_nested): 279 | if not all(map(lambda t: t[0] == t[1], zip(currently_decoding_nested, keylist))): 280 | # keylist doesnt match what we are currently decoding 281 | # and it is longer than what we are decoding 282 | # so this must be the wrong key, skip it 283 | idx += 1 284 | 285 | # add a None value for this particular function argument 286 | # this is required to keep call logs functional 287 | key = keylist[-1] 288 | if not key in result: result[key] = None 289 | continue 290 | 291 | elif currently_decoding_nested: 292 | # keylist is shorter than decoding list -> move up one nesting level 293 | currently_decoding_nested = currently_decoding_nested[:-2] 294 | continue 295 | 296 | content = self.vectorcontent[k] 297 | if type(content) is Options: 298 | # determine which option to use 299 | # this is done by checking in which partition the current value 300 | # of the choice parameter (v) falls 301 | 302 | option_idx = int(math.floor(v)) 303 | option = content[option_idx] 304 | result[DELIM.join(keylist[len(currently_decoding_nested):])] = option 305 | currently_decoding_nested.extend([keylist[-1], option]) 306 | 307 | idx += 1 308 | 309 | else: 310 | result[keylist[-1]] = v 311 | idx += 1 312 | 313 | return result 314 | 315 | def wrap_decoder(self, f): 316 | """ 317 | 318 | Wraps a function to automatically decode arguments based on given SearchTree. 319 | 320 | Use in conjunction with :func:`optunity.search_spaces.SearchTree.to_box`. 321 | 322 | """ 323 | @functools.wraps(f) 324 | def wrapped(**kwargs): 325 | decoded = self.decode(kwargs) 326 | return f(**decoded) 327 | return wrapped 328 | 329 | 330 | 331 | #hpars = {'kernel': {'linear': {'c': [0, 1]}, 332 | # 'rbf': {'gamma': [0, 1], 'c': [0, 10]}, 333 | # 'poly': {'degree': [2, 4], 'c': [0, 2]} 334 | # } 335 | # } 336 | 337 | 338 | #hpars = {'algorithm': {'k-nn': {'k': [1, 10]}, 339 | # 'SVM': {'kernel': {'linear': {'C': [0, 2]}, 340 | # 'rbf': {'gamma': [0, 1], 'C': [0, 10]}, 341 | # 'poly': {'degree': [2, 5], 'C': [0, 50], 'coef0': [0, 1]} 342 | # } 343 | # }, 344 | # 'naive-bayes': None, 345 | # 'random-forest': {'n_estimators': [100, 300], 'max_features': [5, 100]} 346 | # } 347 | # } 348 | 349 | #hpars = {'kernel': {'linear': None, 350 | # 'rbf': {'gamma': [0, 1]}, 351 | # 'poly': {'degree': [2, 4]} 352 | # }, 353 | # 'c': [0, 1] 354 | # } 355 | 356 | #hpars = {'kernel': {'linear': {'c': [0, 1]}, 357 | # 'rbf': {'gamma': [0, 1], 'c': [0, 10]}, 358 | # 'poly': {'degree': [2, 4], 'c': [0, 2]}, 359 | # 'choice': {'choice1': None, 'choice2': None} 360 | # } 361 | # } 362 | 363 | #tree.decode(v2) 364 | 365 | #tree = SearchTree(hpars) 366 | #l = list(tree) 367 | #print('============================') 368 | #print('list') 369 | #print("\n".join(map(str, l))) 370 | #v = tree.to_box() 371 | #print('============================') 372 | #print('box') 373 | #print("\n".join(map(str, v.items()))) 374 | #v2 = v.copy() 375 | #v2['kernel'] = 3.5 376 | #v2['kernel-choice'] = 0.2 377 | 378 | -------------------------------------------------------------------------------- /optunity/solvers/BayesOpt.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | 3 | # Copyright (c) 2014 KU Leuven, ESAT-STADIUS 4 | # All rights reserved. 5 | # 6 | # Redistribution and use in source and binary forms, with or without 7 | # modification, are permitted provided that the following conditions 8 | # are met: 9 | # 10 | # 1. Redistributions of source code must retain the above copyright 11 | # notice, this list of conditions and the following disclaimer. 12 | # 13 | # 2. Redistributions in binary form must reproduce the above copyright 14 | # notice, this list of conditions and the following disclaimer in the 15 | # documentation and/or other materials provided with the distribution. 16 | # 17 | # 3. Neither name of copyright holders nor the names of its contributors 18 | # may be used to endorse or promote products derived from this software 19 | # without specific prior written permission. 20 | # 21 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 22 | # ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 23 | # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 24 | # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR 25 | # CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, 26 | # EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, 27 | # PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR 28 | # PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF 29 | # LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 30 | # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 31 | # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 32 | 33 | from .solver_registry import register_solver 34 | from .util import Solver, _copydoc 35 | import functools 36 | 37 | import random 38 | 39 | _numpy_available = True 40 | try: 41 | import numpy as np 42 | except ImportError: 43 | _numpy_available = False 44 | 45 | _bayesopt_available = True 46 | try: 47 | import bayesopt 48 | except ImportError: 49 | _bayesopt_available = False 50 | 51 | class BayesOpt(Solver): 52 | """ 53 | .. include:: /global.rst 54 | 55 | This solver provides an interface to BayesOpt, as described in [BO2014]_. 56 | This solver uses BayesOpt in the back-end and exposes its solver with uniform priors. 57 | 58 | Please refer to |bayesopt| for details about this algorithm. 59 | 60 | .. note:: 61 | 62 | This solver will always output some text upon running. 63 | This is caused internally by BayesOpt, which provides no way to disable all output. 64 | 65 | .. [BO2014] Martinez-Cantin, Ruben. "BayesOpt: A Bayesian optimization library for nonlinear optimization, experimental design and bandits." The Journal of Machine Learning Research 15.1 (2014): 3735-3739. 66 | 67 | """ 68 | 69 | def __init__(self, num_evals=100, seed=None, **kwargs): 70 | """ 71 | 72 | Initialize the BayesOpt solver. 73 | 74 | :param num_evals: number of permitted function evaluations 75 | :type num_evals: int 76 | :param seed: the random seed to be used 77 | :type seed: double 78 | :param kwargs: box constraints for each hyperparameter 79 | :type kwargs: {'name': [lb, ub], ...} 80 | 81 | 82 | """ 83 | if not _bayesopt_available: 84 | raise ImportError('This solver requires bayesopt but it is missing.') 85 | if not _numpy_available: 86 | raise ImportError('This solver requires NumPy but it is missing.') 87 | 88 | self._seed = seed 89 | self._bounds = kwargs 90 | self._num_evals = num_evals 91 | # bayesopt does not support open intervals, 92 | # while optunity uses open intervals 93 | # so we manually shrink the bounding box slightly (yes, this is a dirty fix) 94 | delta = 0.001 95 | self._lb = np.array(map(lambda x: float(x[1][0] + delta * (x[1][1] - x[1][0])), 96 | sorted(kwargs.items()))) 97 | self._ub = np.array(map(lambda x: float(x[1][1] - delta * (x[1][1] - x[1][0])), 98 | sorted(kwargs.items()))) 99 | 100 | @staticmethod 101 | def suggest_from_box(num_evals, **kwargs): 102 | """ 103 | Verify that we can effectively make a solver from box. 104 | 105 | >>> s = BayesOpt.suggest_from_box(30, x=[0, 1], y=[-1, 0], z=[-1, 1]) 106 | >>> solver = BayesOpt(**s) #doctest:+SKIP 107 | 108 | """ 109 | d = dict(kwargs) 110 | d['num_evals'] = num_evals 111 | return d 112 | 113 | @property 114 | def seed(self): 115 | return self._seed 116 | 117 | @property 118 | def bounds(self): 119 | return self._bounds 120 | 121 | @property 122 | def lb(self): 123 | return self._lb 124 | 125 | @property 126 | def ub(self): 127 | return self._ub 128 | 129 | @property 130 | def num_evals(self): 131 | return self._num_evals 132 | 133 | @_copydoc(Solver.optimize) 134 | def optimize(self, f, maximize=True, pmap=map): 135 | 136 | seed = self.seed if self.seed else random.randint(0, 9999) 137 | params = {'n_iterations': self.num_evals, 138 | 'random_seed': seed, 139 | 'n_iter_relearn': 3, 140 | 'verbose_level': 0} 141 | n_dimensions = len(self.lb) 142 | 143 | print('lb %s' % str(self.lb)) 144 | print('ub %s' % str(self.ub)) 145 | 146 | if maximize: 147 | def obj(args): 148 | kwargs = dict([(k, v) for k, v in zip(sorted(self.bounds.keys()), args)]) 149 | return -f(**kwargs) 150 | 151 | else: 152 | def obj(args): 153 | kwargs = dict([(k, v) for k, v in zip(sorted(self.bounds.keys()), args)]) 154 | return f(**kwargs) 155 | 156 | mvalue, x_out, error = bayesopt.optimize(obj, n_dimensions, 157 | self.lb, self.ub, params) 158 | best = dict([(k, v) for k, v in zip(sorted(self.bounds.keys()), x_out)]) 159 | return best, None 160 | 161 | 162 | # BayesOpt is a simple wrapper around bayesopt's BayesOpt solver 163 | if _bayesopt_available and _numpy_available: 164 | BayesOpt = register_solver('BayesOpt', 'Tree of Parzen estimators', 165 | ['BayesOpt: Tree of Parzen Estimators'] 166 | )(BayesOpt) 167 | -------------------------------------------------------------------------------- /optunity/solvers/CMAES.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | 3 | # Copyright (c) 2014 KU Leuven, ESAT-STADIUS 4 | # All rights reserved. 5 | # 6 | # Redistribution and use in source and binary forms, with or without 7 | # modification, are permitted provided that the following conditions 8 | # are met: 9 | # 10 | # 1. Redistributions of source code must retain the above copyright 11 | # notice, this list of conditions and the following disclaimer. 12 | # 13 | # 2. Redistributions in binary form must reproduce the above copyright 14 | # notice, this list of conditions and the following disclaimer in the 15 | # documentation and/or other materials provided with the distribution. 16 | # 17 | # 3. Neither name of copyright holders nor the names of its contributors 18 | # may be used to endorse or promote products derived from this software 19 | # without specific prior written permission. 20 | # 21 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 22 | # ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 23 | # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 24 | # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR 25 | # CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, 26 | # EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, 27 | # PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR 28 | # PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF 29 | # LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 30 | # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 31 | # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 32 | 33 | import math 34 | import functools 35 | 36 | from .solver_registry import register_solver 37 | from .util import Solver, _copydoc 38 | from . import util 39 | 40 | _numpy_available = True 41 | try: 42 | import numpy as np 43 | except ImportError: 44 | _numpy_available = False 45 | 46 | _deap_available = True 47 | try: 48 | import deap 49 | import deap.creator 50 | import deap.base 51 | import deap.tools 52 | import deap.cma 53 | import deap.algorithms 54 | except ImportError: 55 | _deap_available = False 56 | except TypeError: 57 | # this can happen because DEAP is in Python 2 58 | # install needs to take proper care of converting 59 | # 2 to 3 when necessary 60 | _deap_available = False 61 | 62 | 63 | class CMA_ES(Solver): 64 | """ 65 | .. include:: /global.rst 66 | 67 | Please refer to |cmaes| for details about this algorithm. 68 | 69 | This solver uses an implementation available in the DEAP library [DEAP2012]_. 70 | 71 | .. warning:: This solver has dependencies on DEAP_ and NumPy_ 72 | and will be unavailable if these are not met. 73 | 74 | .. _DEAP: https://code.google.com/p/deap/ 75 | .. _NumPy: http://www.numpy.org 76 | 77 | """ 78 | 79 | def __init__(self, num_generations, sigma=1.0, Lambda=None, **kwargs): 80 | """blah 81 | 82 | .. warning:: |warning-unconstrained| 83 | 84 | """ 85 | if not _deap_available: 86 | raise ImportError('This solver requires DEAP but it is missing.') 87 | if not _numpy_available: 88 | raise ImportError('This solver requires NumPy but it is missing.') 89 | 90 | self._num_generations = num_generations 91 | self._start = kwargs 92 | self._sigma = sigma 93 | self._lambda = Lambda 94 | 95 | @staticmethod 96 | def suggest_from_seed(num_evals, **kwargs): 97 | """Verify that we can effectively make a solver. 98 | The doctest has to be skipped from automated builds, because DEAP may not be available 99 | and yet we want documentation to be generated. 100 | 101 | >>> s = CMA_ES.suggest_from_seed(30, x=1.0, y=-1.0, z=2.0) 102 | >>> solver = CMA_ES(**s) #doctest:+SKIP 103 | 104 | """ 105 | fertility = 4 + 3 * math.log(len(kwargs)) 106 | d = dict(kwargs) 107 | d['num_generations'] = int(math.ceil(float(num_evals) / fertility)) 108 | # num_gen is overestimated 109 | # this will require slightly more function evaluations than permitted by num_evals 110 | return d 111 | 112 | @property 113 | def num_generations(self): 114 | return self._num_generations 115 | 116 | @property 117 | def start(self): 118 | """Returns the starting point for CMA-ES.""" 119 | return self._start 120 | 121 | @property 122 | def lambda_(self): 123 | return self._lambda 124 | 125 | @property 126 | def sigma(self): 127 | return self._sigma 128 | 129 | @_copydoc(Solver.optimize) 130 | def optimize(self, f, maximize=True, pmap=map): 131 | toolbox = deap.base.Toolbox() 132 | if maximize: 133 | fit = 1.0 134 | else: 135 | fit = -1.0 136 | deap.creator.create("FitnessMax", deap.base.Fitness, 137 | weights=(fit,)) 138 | Fit = deap.creator.FitnessMax 139 | deap.creator.create("Individual", list, 140 | fitness=Fit) 141 | Individual = deap.creator.Individual 142 | 143 | if self.lambda_: 144 | strategy = deap.cma.Strategy(centroid=list(self.start.values()), 145 | sigma=self.sigma, lambda_=self.lambda_) 146 | else: 147 | strategy = deap.cma.Strategy(centroid=list(self.start.values()), 148 | sigma=self.sigma) 149 | toolbox.register("generate", strategy.generate, Individual) 150 | toolbox.register("update", strategy.update) 151 | 152 | @functools.wraps(f) 153 | def evaluate(individual): 154 | return (util.score(f(**dict([(k, v) 155 | for k, v in zip(self.start.keys(), 156 | individual)]))),) 157 | toolbox.register("evaluate", evaluate) 158 | toolbox.register("map", pmap) 159 | 160 | hof = deap.tools.HallOfFame(1) 161 | deap.algorithms.eaGenerateUpdate(toolbox=toolbox, 162 | ngen=self._num_generations, 163 | halloffame=hof, verbose=False) 164 | 165 | return dict([(k, v) 166 | for k, v in zip(self.start.keys(), hof[0])]), None 167 | 168 | # CMA_ES solver requires deap > 1.0.1 169 | # http://deap.readthedocs.org/en/latest/examples/cmaes.html 170 | if _deap_available and _numpy_available: 171 | CMA_ES = register_solver('cma-es', 'covariance matrix adaptation evolutionary strategy', 172 | ['CMA-ES: covariance matrix adaptation evolutionary strategy', 173 | ' ', 174 | 'This method requires the following parameters:', 175 | '- num_generations :: number of generations to use', 176 | '- sigma :: (optional) initial covariance, default 1', 177 | '- Lambda :: (optional) measure of reproducibility', 178 | '- starting point: through kwargs' 179 | ' ', 180 | 'This method is described in detail in:', 181 | 'Hansen and Ostermeier, 2001. Completely Derandomized Self-Adaptation in Evolution Strategies. Evolutionary Computation' 182 | ])(CMA_ES) 183 | -------------------------------------------------------------------------------- /optunity/solvers/GridSearch.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | 3 | # Copyright (c) 2014 KU Leuven, ESAT-STADIUS 4 | # All rights reserved. 5 | # 6 | # Redistribution and use in source and binary forms, with or without 7 | # modification, are permitted provided that the following conditions 8 | # are met: 9 | # 10 | # 1. Redistributions of source code must retain the above copyright 11 | # notice, this list of conditions and the following disclaimer. 12 | # 13 | # 2. Redistributions in binary form must reproduce the above copyright 14 | # notice, this list of conditions and the following disclaimer in the 15 | # documentation and/or other materials provided with the distribution. 16 | # 17 | # 3. Neither name of copyright holders nor the names of its contributors 18 | # may be used to endorse or promote products derived from this software 19 | # without specific prior written permission. 20 | # 21 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 22 | # ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 23 | # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 24 | # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR 25 | # CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, 26 | # EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, 27 | # PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR 28 | # PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF 29 | # LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 30 | # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 31 | # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 32 | 33 | import operator as op 34 | import itertools 35 | 36 | from ..functions import static_key_order 37 | from .solver_registry import register_solver 38 | from .util import Solver, _copydoc, shrink_bounds 39 | from . import util 40 | 41 | # http://stackoverflow.com/a/15978862 42 | def nth_root(val, n): 43 | ret = int(val**(1./n)) 44 | return ret + 1 if (ret + 1) ** n == val else ret 45 | 46 | @register_solver('grid search', 47 | 'finds optimal parameter values on a predefined grid', 48 | ['Retrieves the best parameter tuple on a predefined grid.', 49 | ' ', 50 | 'This function requires the grid to be specified via named arguments:', 51 | '- names :: argument names', 52 | '- values :: list of grid coordinates to test', 53 | ' ', 54 | 'The solver performs evaluation on the Cartesian product of grid values.', 55 | 'The number of evaluations is the product of the length of all value vectors.' 56 | ]) 57 | class GridSearch(Solver): 58 | """ 59 | .. include:: /global.rst 60 | 61 | Please refer to |gridsearch| for more information about this algorithm. 62 | 63 | Exhaustive search over the Cartesian product of parameter tuples. 64 | Returns x (the tuple which maximizes f) and its score f(x). 65 | 66 | >>> s = GridSearch(x=[1,2,3], y=[-1,0,1]) 67 | >>> best_pars, _ = s.optimize(lambda x, y: x*y) 68 | >>> best_pars['x'] 69 | 3 70 | >>> best_pars['y'] 71 | 1 72 | 73 | """ 74 | 75 | def __init__(self, **kwargs): 76 | """Initializes the solver with a tuple indicating parameter values. 77 | 78 | >>> s = GridSearch(x=[1,2], y=[3,4]) 79 | >>> s.parameter_tuples['x'] 80 | [1, 2] 81 | >>> s.parameter_tuples['y'] 82 | [3, 4] 83 | 84 | """ 85 | self._parameter_tuples = kwargs 86 | 87 | @staticmethod 88 | def assign_grid_points(lb, ub, density): 89 | """Assigns equally spaced grid points with given density in [ub, lb]. 90 | The bounds are always used. ``density`` must be at least 2. 91 | 92 | :param lb: lower bound of resulting grid 93 | :param ub: upper bound of resulting grid 94 | :param density: number of points to use 95 | :type lb: float 96 | :type ub: float 97 | :type density: int 98 | 99 | >>> s = GridSearch.assign_grid_points(1.0, 2.0, 3) 100 | >>> s #doctest:+SKIP 101 | [1.0, 1.5, 2.0] 102 | 103 | """ 104 | density = int(density) 105 | assert(density >= 2) 106 | step = float(ub-lb)/(density-1) 107 | return [lb+i*step for i in range(density)] 108 | 109 | @staticmethod 110 | def suggest_from_box(num_evals, **kwargs): 111 | """Creates a GridSearch solver that uses less than num_evals evaluations 112 | within given bounds (lb, ub). The bounds are first tightened, resulting in 113 | new bounds covering 99% of the area. 114 | 115 | The resulting solver will use an equally spaced grid with the same number 116 | of points in every dimension. The amount of points that is used is per 117 | dimension is the nth root of num_evals, rounded down, where n is the number 118 | of hyperparameters. 119 | 120 | >>> s = GridSearch.suggest_from_box(30, x=[0, 1], y=[-1, 0], z=[-1, 1]) 121 | >>> s['x'] #doctest:+SKIP 122 | [0.005, 0.5, 0.995] 123 | >>> s['y'] #doctest:+SKIP 124 | [-0.995, -0.5, -0.005] 125 | >>> s['z'] #doctest:+SKIP 126 | [-0.99, 0.0, 0.99] 127 | 128 | 129 | Verify that we can effectively make a solver from box. 130 | 131 | >>> s = GridSearch.suggest_from_box(30, x=[0, 1], y=[-1, 0], z=[-1, 1]) 132 | >>> solver = GridSearch(**s) 133 | 134 | """ 135 | bounds = shrink_bounds(kwargs) 136 | num_pars = len(bounds) 137 | 138 | # number of grid points in each dimension 139 | # so we get density^num_par grid points in total 140 | density = nth_root(num_evals, num_pars) 141 | grid = dict([(k, GridSearch.assign_grid_points(b[0], b[1], density)) 142 | for k, b in bounds.items()]) 143 | return grid 144 | 145 | @property 146 | def parameter_tuples(self): 147 | """Returns the possible values of every parameter.""" 148 | return self._parameter_tuples 149 | 150 | @_copydoc(Solver.optimize) 151 | def optimize(self, f, maximize=True, pmap=map): 152 | 153 | best_pars = None 154 | f = static_key_order(self.parameter_tuples.keys())(f) 155 | 156 | if maximize: 157 | comp = lambda score, best: score > best 158 | else: 159 | comp = lambda score, best: score < best 160 | 161 | tuples = list(zip(*itertools.product(*list(zip(*self.parameter_tuples.items()))[1]))) 162 | scores = pmap(f, *tuples) 163 | scores = map(util.score, scores) 164 | 165 | if maximize: 166 | comp = max 167 | else: 168 | comp = min 169 | best_idx, _ = comp(enumerate(scores), key=op.itemgetter(1)) 170 | best_pars = op.itemgetter(best_idx)(list(zip(*tuples))) 171 | return dict([(k, v) for k, v in zip(self.parameter_tuples.keys(), best_pars)]), None 172 | -------------------------------------------------------------------------------- /optunity/solvers/NelderMead.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | 3 | # Copyright (c) 2014 KU Leuven, ESAT-STADIUS 4 | # All rights reserved. 5 | # 6 | # Redistribution and use in source and binary forms, with or without 7 | # modification, are permitted provided that the following conditions 8 | # are met: 9 | # 10 | # 1. Redistributions of source code must retain the above copyright 11 | # notice, this list of conditions and the following disclaimer. 12 | # 13 | # 2. Redistributions in binary form must reproduce the above copyright 14 | # notice, this list of conditions and the following disclaimer in the 15 | # documentation and/or other materials provided with the distribution. 16 | # 17 | # 3. Neither name of copyright holders nor the names of its contributors 18 | # may be used to endorse or promote products derived from this software 19 | # without specific prior written permission. 20 | # 21 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 22 | # ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 23 | # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 24 | # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR 25 | # CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, 26 | # EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, 27 | # PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR 28 | # PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF 29 | # LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 30 | # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 31 | # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 32 | 33 | import array 34 | import operator as op 35 | 36 | from .. import functions as fun 37 | from .solver_registry import register_solver 38 | from .util import Solver, _copydoc 39 | from . import util 40 | 41 | @register_solver('nelder-mead', 42 | 'simplex method for unconstrained optimization', 43 | ['Simplex method for unconstrained optimization', 44 | ' ', 45 | 'The simplex algorithm is a simple way to optimize a fairly well-behaved function.', 46 | 'The function is assumed to be convex. If not, this solver may yield poor solutions.', 47 | ' ', 48 | 'This solver requires the following arguments:', 49 | '- start :: starting point for the solver (through kwargs)', 50 | '- ftol :: accuracy up to which to optimize the function (default 1e-4)' 51 | ]) 52 | class NelderMead(Solver): 53 | """ 54 | .. include:: /global.rst 55 | 56 | Please refer to |nelder-mead| for details about this algorithm. 57 | 58 | >>> s = NelderMead(x=1, y=1, xtol=1e-8) #doctest:+SKIP 59 | >>> best_pars, _ = s.optimize(lambda x, y: -x**2 - y**2) #doctest:+SKIP 60 | >>> [math.fabs(best_pars['x']) < 1e-8, math.fabs(best_pars['y']) < 1e-8] #doctest:+SKIP 61 | [True, True] 62 | 63 | """ 64 | 65 | def __init__(self, ftol=1e-4, max_iter=None, **kwargs): 66 | """Initializes the solver with a tuple indicating parameter values. 67 | 68 | >>> s = NelderMead(x=1, ftol=2) #doctest:+SKIP 69 | >>> s.start #doctest:+SKIP 70 | {'x': 1} 71 | >>> s.ftol #doctest:+SKIP 72 | 2 73 | 74 | .. warning:: |warning-unconstrained| 75 | 76 | """ 77 | 78 | self._start = kwargs 79 | self._ftol = ftol 80 | self._max_iter = max_iter 81 | if max_iter is None: 82 | self._max_iter = len(kwargs) * 200 83 | 84 | @staticmethod 85 | def suggest_from_seed(num_evals, **kwargs): 86 | """Verify that we can effectively make a solver. 87 | 88 | >>> s = NelderMead.suggest_from_seed(30, x=1.0, y=-1.0, z=2.0) 89 | >>> solver = NelderMead(**s) 90 | 91 | """ 92 | return kwargs 93 | 94 | @property 95 | def ftol(self): 96 | """Returns the tolerance.""" 97 | return self._ftol 98 | 99 | @property 100 | def max_iter(self): 101 | """Returns the maximum number of iterations.""" 102 | return self._max_iter 103 | 104 | @property 105 | def start(self): 106 | """Returns the starting point.""" 107 | return self._start 108 | 109 | @_copydoc(Solver.optimize) 110 | def optimize(self, f, maximize=True, pmap=map): 111 | if maximize: 112 | f = fun.negated(f) 113 | 114 | sortedkeys = sorted(self.start.keys()) 115 | x0 = [float(self.start[k]) for k in sortedkeys] 116 | 117 | f = fun.static_key_order(sortedkeys)(f) 118 | 119 | def func(x): 120 | return util.score(f(*x)) 121 | 122 | xopt = self._solve(func, x0) 123 | return dict([(k, v) for k, v in zip(sortedkeys, xopt)]), None 124 | 125 | def _solve(self, func, x0): 126 | def f(x): 127 | return func(list(x)) 128 | 129 | x0 = array.array('f', x0) 130 | N = len(x0) 131 | 132 | vertices = [x0] 133 | values = [f(x0)] 134 | 135 | # defaults taken from Wikipedia and SciPy 136 | alpha = 1.; gamma = 2.; rho = -0.5; sigma = 0.5; 137 | nonzdelt = 0.05 138 | zdelt = 0.00025 139 | 140 | # generate vertices 141 | for k in range(N): 142 | vert = vertices[0][:] 143 | if vert[k] != 0: 144 | vert[k] = (1 + nonzdelt) * vert[k] 145 | else: 146 | vert[k] = zdelt 147 | 148 | vertices.append(vert) 149 | values.append(f(vert)) 150 | 151 | niter = 1 152 | while niter < self.max_iter: 153 | 154 | # sort vertices by ftion value 155 | vertices, values = NelderMead.sort_vertices(vertices, values) 156 | 157 | # check for convergence 158 | if abs(values[0] - values[-1]) <= self.ftol: 159 | break 160 | 161 | niter += 1 162 | 163 | # compute center of gravity 164 | x0 = NelderMead.simplex_center(vertices[:-1]) 165 | 166 | # reflect 167 | xr = NelderMead.reflect(x0, vertices[-1], alpha) 168 | fxr = f(xr) 169 | if values[0] < fxr < values[-2]: 170 | vertices[-1] = xr 171 | values[-1] = fxr 172 | continue 173 | 174 | # expand 175 | if fxr < values[0]: 176 | xe = NelderMead.reflect(x0, vertices[-1], gamma) 177 | fxe = f(xe) 178 | if fxe < fxr: 179 | vertices[-1] = xe 180 | values[-1] = fxe 181 | else: 182 | vertices[-1] = xr 183 | values[-1] = fxr 184 | continue 185 | 186 | # contract 187 | xc = NelderMead.reflect(x0, vertices[-1], rho) 188 | fxc = f(xc) 189 | if fxc < values[-1]: 190 | vertices[-1] = xc 191 | values[-1] = fxc 192 | continue 193 | 194 | # reduce 195 | for idx in range(1, len(vertices)): 196 | vertices[idx] = NelderMead.reflect(vertices[0], vertices[idx], 197 | sigma) 198 | values[idx] = f(vertices[idx]) 199 | 200 | return list(vertices[min(enumerate(values), key=op.itemgetter(1))[0]]) 201 | 202 | @staticmethod 203 | def simplex_center(vertices): 204 | vector_sum = map(sum, zip(*vertices)) 205 | return array.array('f', map(lambda x: x / len(vertices), vector_sum)) 206 | 207 | @staticmethod 208 | def sort_vertices(vertices, values): 209 | sort_idx, values = zip(*sorted(enumerate(values), key=op.itemgetter(1))) 210 | # doing the same with a map bugs out, for some reason 211 | vertices = [vertices[x] for x in sort_idx] 212 | return vertices, list(values) 213 | 214 | @staticmethod 215 | def scale(vertex, coeff): 216 | return array.array('f', map(lambda x: coeff * x, vertex)) 217 | 218 | @staticmethod 219 | def reflect(x0, xn1, alpha): 220 | diff = map(op.sub, x0, xn1) 221 | xr = array.array('f', map(op.add, x0, NelderMead.scale(diff, alpha))) 222 | return xr 223 | -------------------------------------------------------------------------------- /optunity/solvers/ParticleSwarm.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | 3 | # Copyright (c) 2014 KU Leuven, ESAT-STADIUS 4 | # All rights reserved. 5 | # 6 | # Redistribution and use in source and binary forms, with or without 7 | # modification, are permitted provided that the following conditions 8 | # are met: 9 | # 10 | # 1. Redistributions of source code must retain the above copyright 11 | # notice, this list of conditions and the following disclaimer. 12 | # 13 | # 2. Redistributions in binary form must reproduce the above copyright 14 | # notice, this list of conditions and the following disclaimer in the 15 | # documentation and/or other materials provided with the distribution. 16 | # 17 | # 3. Neither name of copyright holders nor the names of its contributors 18 | # may be used to endorse or promote products derived from this software 19 | # without specific prior written permission. 20 | # 21 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 22 | # ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 23 | # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 24 | # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR 25 | # CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, 26 | # EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, 27 | # PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR 28 | # PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF 29 | # LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 30 | # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 31 | # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 32 | 33 | import math 34 | import operator as op 35 | import random 36 | import array 37 | import functools 38 | 39 | from .solver_registry import register_solver 40 | from .util import Solver, _copydoc, uniform_in_bounds 41 | from . import util 42 | from .Sobol import Sobol 43 | 44 | @register_solver('particle swarm', 45 | 'particle swarm optimization', 46 | ['Maximizes the function using particle swarm optimization.', 47 | ' ', 48 | 'This is a two-phase approach:', 49 | '1. Initialization: randomly initializes num_particles particles.', 50 | ' Particles are randomized uniformly within the box constraints.', 51 | '2. Iteration: particles move during num_generations iterations.', 52 | ' Movement is based on their velocities and mutual attractions.', 53 | ' ', 54 | 'This function requires the following arguments:', 55 | '- num_particles: number of particles to use in the swarm', 56 | '- num_generations: number of iterations used by the swarm', 57 | '- max_speed: maximum speed of the particles in each direction (in (0, 1])', 58 | '- box constraints via key words: constraints are lists [lb, ub]', ' ', 59 | 'This solver performs num_particles*num_generations function evaluations.' 60 | ]) 61 | class ParticleSwarm(Solver): 62 | """ 63 | .. include:: /global.rst 64 | 65 | Please refer to |pso| for details on this algorithm. 66 | """ 67 | 68 | class Particle: 69 | def __init__(self, position, speed, best, fitness, best_fitness): 70 | """Constructs a Particle.""" 71 | self.position = position 72 | self.speed = speed 73 | self.best = best 74 | self.fitness = fitness 75 | self.best_fitness = best_fitness 76 | 77 | def clone(self): 78 | """Clones this Particle.""" 79 | return ParticleSwarm.Particle(position=self.position[:], speed=self.speed[:], 80 | best=self.best[:], fitness=self.fitness, 81 | best_fitness=self.best_fitness) 82 | 83 | def __str__(self): 84 | string = 'Particle{position=' + str(self.position) 85 | string += ', speed=' + str(self.speed) 86 | string += ', best=' + str(self.best) 87 | string += ', fitness=' + str(self.fitness) 88 | string += ', best_fitness=' + str(self.best_fitness) 89 | string += '}' 90 | return string 91 | 92 | def __init__(self, num_particles, num_generations, max_speed=None, phi1=1.5, phi2=2.0, **kwargs): 93 | """ 94 | Initializes a PSO solver. 95 | 96 | :param num_particles: number of particles to use 97 | :type num_particles: int 98 | :param num_generations: number of generations to use 99 | :type num_generations: int 100 | :param max_speed: maximum velocity of each particle 101 | :type max_speed: float or None 102 | :param phi1: parameter used in updating position based on local best 103 | :type phi1: float 104 | :param phi2: parameter used in updating position based on global best 105 | :type phi2: float 106 | :param kwargs: box constraints for each hyperparameter 107 | :type kwargs: {'name': [lb, ub], ...} 108 | 109 | The number of function evaluations it will perform is `num_particles`*`num_generations`. 110 | The search space is rescaled to the unit hypercube before the solving process begins. 111 | 112 | >>> solver = ParticleSwarm(num_particles=10, num_generations=5, x=[-1, 1], y=[0, 2]) 113 | >>> solver.bounds['x'] 114 | [-1, 1] 115 | >>> solver.bounds['y'] 116 | [0, 2] 117 | >>> solver.num_particles 118 | 10 119 | >>> solver.num_generations 120 | 5 121 | 122 | .. warning:: |warning-unconstrained| 123 | 124 | """ 125 | 126 | assert all([len(v) == 2 and v[0] <= v[1] 127 | for v in kwargs.values()]), 'kwargs.values() are not [lb, ub] pairs' 128 | self._bounds = kwargs 129 | self._num_particles = num_particles 130 | self._num_generations = num_generations 131 | 132 | self._sobolseed = random.randint(100,2000) 133 | 134 | if max_speed is None: 135 | max_speed = 0.7 / num_generations 136 | # max_speed = 0.2 / math.sqrt(num_generations) 137 | self._max_speed = max_speed 138 | self._smax = [self.max_speed * (b[1] - b[0]) 139 | for _, b in self.bounds.items()] 140 | self._smin = list(map(op.neg, self.smax)) 141 | 142 | self._phi1 = phi1 143 | self._phi2 = phi2 144 | 145 | @property 146 | def phi1(self): 147 | return self._phi1 148 | 149 | @property 150 | def phi2(self): 151 | return self._phi2 152 | 153 | @property 154 | def sobolseed(self): return self._sobolseed 155 | 156 | @sobolseed.setter 157 | def sobolseed(self, value): self._sobolseed = value 158 | 159 | @staticmethod 160 | def suggest_from_box(num_evals, **kwargs): 161 | """Create a configuration for a ParticleSwarm solver. 162 | 163 | :param num_evals: number of permitted function evaluations 164 | :type num_evals: int 165 | :param kwargs: box constraints 166 | :type kwargs: {'param': [lb, ub], ...} 167 | 168 | >>> config = ParticleSwarm.suggest_from_box(200, x=[-1, 1], y=[0, 1]) 169 | >>> config['x'] 170 | [-1, 1] 171 | >>> config['y'] 172 | [0, 1] 173 | >>> config['num_particles'] > 0 174 | True 175 | >>> config['num_generations'] > 0 176 | True 177 | >>> solver = ParticleSwarm(**config) 178 | >>> solver.bounds['x'] 179 | [-1, 1] 180 | >>> solver.bounds['y'] 181 | [0, 1] 182 | 183 | """ 184 | d = dict(kwargs) 185 | if num_evals > 1000: 186 | d['num_particles'] = 100 187 | elif num_evals >= 200: 188 | d['num_particles'] = 20 189 | elif num_evals >= 10: 190 | d['num_particles'] = 10 191 | else: 192 | d['num_particles'] = num_evals 193 | d['num_generations'] = int(math.ceil(float(num_evals) / d['num_particles'])) 194 | return d 195 | 196 | @property 197 | def num_particles(self): 198 | return self._num_particles 199 | 200 | @property 201 | def num_generations(self): 202 | return self._num_generations 203 | 204 | @property 205 | def max_speed(self): 206 | return self._max_speed 207 | 208 | @property 209 | def smax(self): 210 | return self._smax 211 | 212 | @property 213 | def smin(self): 214 | return self._smin 215 | 216 | @property 217 | def bounds(self): 218 | return self._bounds 219 | 220 | def generate(self): 221 | """Generate a new Particle.""" 222 | if len(self.bounds) < Sobol.maxdim(): 223 | sobol_vector, self.sobolseed = Sobol.i4_sobol(len(self.bounds), self.sobolseed) 224 | vector = util.scale_unit_to_bounds(sobol_vector, self.bounds.values()) 225 | else: vector = uniform_in_bounds(self.bounds) 226 | 227 | part = ParticleSwarm.Particle(position=array.array('d', vector), 228 | speed=array.array('d', map(random.uniform, 229 | self.smin, self.smax)), 230 | best=None, fitness=None, best_fitness=None) 231 | return part 232 | 233 | def updateParticle(self, part, best, phi1, phi2): 234 | """Update the particle.""" 235 | u1 = (random.uniform(0, phi1) for _ in range(len(part.position))) 236 | u2 = (random.uniform(0, phi2) for _ in range(len(part.position))) 237 | v_u1 = map(op.mul, u1, 238 | map(op.sub, part.best, part.position)) 239 | v_u2 = map(op.mul, u2, 240 | map(op.sub, best.position, part.position)) 241 | part.speed = array.array('d', map(op.add, part.speed, 242 | map(op.add, v_u1, v_u2))) 243 | for i, speed in enumerate(part.speed): 244 | if speed < self.smin[i]: 245 | part.speed[i] = self.smin[i] 246 | elif speed > self.smax[i]: 247 | part.speed[i] = self.smax[i] 248 | part.position[:] = array.array('d', map(op.add, part.position, part.speed)) 249 | 250 | def particle2dict(self, particle): 251 | return dict([(k, v) for k, v in zip(self.bounds.keys(), 252 | particle.position)]) 253 | 254 | @_copydoc(Solver.optimize) 255 | def optimize(self, f, maximize=True, pmap=map): 256 | 257 | @functools.wraps(f) 258 | def evaluate(d): 259 | return f(**d) 260 | 261 | if maximize: 262 | fit = 1.0 263 | else: 264 | fit = -1.0 265 | 266 | pop = [self.generate() for _ in range(self.num_particles)] 267 | best = None 268 | 269 | for g in range(self.num_generations): 270 | fitnesses = pmap(evaluate, list(map(self.particle2dict, pop))) 271 | for part, fitness in zip(pop, fitnesses): 272 | part.fitness = fit * util.score(fitness) 273 | if not part.best or part.best_fitness < part.fitness: 274 | part.best = part.position 275 | part.best_fitness = part.fitness 276 | if not best or best.fitness < part.fitness: 277 | best = part.clone() 278 | for part in pop: 279 | self.updateParticle(part, best, self.phi1, self.phi2) 280 | 281 | return dict([(k, v) 282 | for k, v in zip(self.bounds.keys(), best.position)]), None 283 | -------------------------------------------------------------------------------- /optunity/solvers/ParticleSwarm_New.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | 3 | # Copyright (c) 2014 KU Leuven, ESAT-STADIUS 4 | # All rights reserved. 5 | # 6 | # Redistribution and use in source and binary forms, with or without 7 | # modification, are permitted provided that the following conditions 8 | # are met: 9 | # 10 | # 1. Redistributions of source code must retain the above copyright 11 | # notice, this list of conditions and the following disclaimer. 12 | # 13 | # 2. Redistributions in binary form must reproduce the above copyright 14 | # notice, this list of conditions and the following disclaimer in the 15 | # documentation and/or other materials provided with the distribution. 16 | # 17 | # 3. Neither name of copyright holders nor the names of its contributors 18 | # may be used to endorse or promote products derived from this software 19 | # without specific prior written permission. 20 | # 21 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 22 | # ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 23 | # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 24 | # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR 25 | # CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, 26 | # EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, 27 | # PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR 28 | # PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF 29 | # LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 30 | # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 31 | # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 32 | 33 | import math 34 | import operator as op 35 | import random 36 | import array 37 | import functools 38 | 39 | from .solver_registry import register_solver 40 | from .util import Solver, _copydoc, uniform_in_bounds 41 | from . import util 42 | from .Sobol import Sobol 43 | 44 | @register_solver('particle swarm new', 45 | 'particle swarm optimization', 46 | ['Maximizes the function using particle swarm optimization.', 47 | ' ', 48 | 'This is a two-phase approach:', 49 | '1. Initialization: randomly initializes num_particles particles.', 50 | ' Particles are randomized uniformly within the box constraints.', 51 | '2. Iteration: particles move during num_generations iterations.', 52 | ' Movement is based on their velocities and mutual attractions.', 53 | ' ', 54 | 'This function requires the following arguments:', 55 | '- num_particles: number of particles to use in the swarm', 56 | '- num_generations: number of iterations used by the swarm', 57 | '- max_speed: maximum speed of the particles in each direction (in (0, 1])', 58 | '- box constraints via key words: constraints are lists [lb, ub]', ' ', 59 | 'This solver performs num_particles*num_generations function evaluations.' 60 | ]) 61 | class ParticleSwarm_New(Solver): 62 | """ 63 | .. include:: /global.rst 64 | 65 | Please refer to |pso| for details on this algorithm. 66 | """ 67 | 68 | class Particle: 69 | def __init__(self, position, speed, best, fitness, best_fitness): 70 | """Constructs a Particle.""" 71 | self.position = position 72 | self.speed = speed 73 | self.best = best 74 | self.fitness = fitness 75 | self.best_fitness = best_fitness 76 | 77 | def clone(self): 78 | """Clones this Particle.""" 79 | return ParticleSwarm_New.Particle(position=self.position[:], speed=self.speed[:], 80 | best=self.best[:], fitness=self.fitness, 81 | best_fitness=self.best_fitness) 82 | 83 | def __str__(self): 84 | string = 'Particle{position=' + str(self.position) 85 | string += ', speed=' + str(self.speed) 86 | string += ', best=' + str(self.best) 87 | string += ', fitness=' + str(self.fitness) 88 | string += ', best_fitness=' + str(self.best_fitness) 89 | string += '}' 90 | return string 91 | 92 | def __init__(self, num_particles, num_generations, max_speed=None, phi1=1.5, phi2=2.0, **kwargs): 93 | """ 94 | Initializes a PSO solver. 95 | 96 | :param num_particles: number of particles to use 97 | :type num_particles: int 98 | :param num_generations: number of generations to use 99 | :type num_generations: int 100 | :param max_speed: maximum velocity of each particle 101 | :type max_speed: float or None 102 | :param phi1: parameter used in updating position based on local best 103 | :type phi1: float 104 | :param phi2: parameter used in updating position based on global best 105 | :type phi2: float 106 | :param kwargs: box constraints for each hyperparameter 107 | :type kwargs: {'name': [lb, ub], ...} 108 | 109 | The number of function evaluations it will perform is `num_particles`*`num_generations`. 110 | The search space is rescaled to the unit hypercube before the solving process begins. 111 | 112 | >>> solver = ParticleSwarm_New(num_particles=10, num_generations=5, x=[-1, 1], y=[0, 2]) 113 | >>> solver.bounds['x'] 114 | [-1, 1] 115 | >>> solver.bounds['y'] 116 | [0, 2] 117 | >>> solver.num_particles 118 | 10 119 | >>> solver.num_generations 120 | 5 121 | 122 | .. warning:: |warning-unconstrained| 123 | 124 | """ 125 | 126 | assert all([len(v) == 2 and v[0] <= v[1] 127 | for v in kwargs.values()]), 'kwargs.values() are not [lb, ub] pairs' 128 | self._bounds = kwargs 129 | self._num_particles = num_particles 130 | self._num_generations = num_generations 131 | 132 | self._sobolseed = random.randint(100,2000) 133 | 134 | if max_speed is None: 135 | max_speed = 0.7 / num_generations 136 | # max_speed = 0.2 / math.sqrt(num_generations) 137 | self._max_speed = max_speed 138 | self._smax = [self.max_speed * (b[1] - b[0]) 139 | for _, b in self.bounds.items()] 140 | self._smin = list(map(op.neg, self.smax)) 141 | 142 | self._phi1 = phi1 143 | self._phi2 = phi2 144 | 145 | @property 146 | def phi1(self): 147 | return self._phi1 148 | 149 | @property 150 | def phi2(self): 151 | return self._phi2 152 | 153 | @property 154 | def sobolseed(self): return self._sobolseed 155 | 156 | @sobolseed.setter 157 | def sobolseed(self, value): self._sobolseed = value 158 | 159 | @staticmethod 160 | def suggest_from_box(num_evals, **kwargs): 161 | """Create a configuration for a ParticleSwarm_New solver. 162 | 163 | :param num_evals: number of permitted function evaluations 164 | :type num_evals: int 165 | :param kwargs: box constraints 166 | :type kwargs: {'param': [lb, ub], ...} 167 | 168 | >>> config = ParticleSwarm_New.suggest_from_box(200, x=[-1, 1], y=[0, 1]) 169 | >>> config['x'] 170 | [-1, 1] 171 | >>> config['y'] 172 | [0, 1] 173 | >>> config['num_particles'] > 0 174 | True 175 | >>> config['num_generations'] > 0 176 | True 177 | >>> solver = ParticleSwarm_New(**config) 178 | >>> solver.bounds['x'] 179 | [-1, 1] 180 | >>> solver.bounds['y'] 181 | [0, 1] 182 | 183 | """ 184 | # d = dict(kwargs) 185 | # if num_evals > 1000: 186 | # d['num_particles'] = 100 187 | # elif num_evals >= 200: 188 | # d['num_particles'] = 20 189 | # elif num_evals >= 10: 190 | # d['num_particles'] = 10 191 | # else: 192 | # d['num_particles'] = num_evals 193 | # d['num_generations'] = int(math.ceil(float(num_evals) / d['num_particles'])) 194 | # return d 195 | #### change num_particles 196 | d = dict(kwargs) 197 | if num_evals > 1000: 198 | d['num_particles'] = 100 199 | elif num_evals >= 500: 200 | d['num_particles'] = 50 201 | elif num_evals >= 300: 202 | d['num_particles'] = 30 203 | elif num_evals >= 100: 204 | d['num_particles'] = 20 205 | elif num_evals >= 30: 206 | d['num_particles'] = 10 207 | elif num_evals >= 10: 208 | d['num_particles'] = 5 209 | else: 210 | d['num_particles'] = num_evals 211 | d['num_generations'] = int(math.ceil(float(num_evals) / d['num_particles'])) 212 | return d 213 | 214 | @property 215 | def num_particles(self): 216 | return self._num_particles 217 | 218 | @property 219 | def num_generations(self): 220 | return self._num_generations 221 | 222 | @property 223 | def max_speed(self): 224 | return self._max_speed 225 | 226 | @property 227 | def smax(self): 228 | return self._smax 229 | 230 | @property 231 | def smin(self): 232 | return self._smin 233 | 234 | @property 235 | def bounds(self): 236 | return self._bounds 237 | 238 | def generate(self): 239 | """Generate a new Particle.""" 240 | if len(self.bounds) < Sobol.maxdim(): 241 | sobol_vector, self.sobolseed = Sobol.i4_sobol(len(self.bounds), self.sobolseed) 242 | vector = util.scale_unit_to_bounds(sobol_vector, self.bounds.values()) 243 | else: vector = uniform_in_bounds(self.bounds) 244 | 245 | #### int position 246 | vector = [int(i) for i in vector] 247 | part = ParticleSwarm_New.Particle(position=array.array('d', vector), 248 | speed=array.array('d', map(random.uniform, 249 | self.smin, self.smax)), 250 | best=None, fitness=None, best_fitness=None) 251 | 252 | return part 253 | 254 | def updateParticle(self, part, best, phi1, phi2): 255 | """Update the particle.""" 256 | u1 = (random.uniform(0, phi1) for _ in range(len(part.position))) 257 | u2 = (random.uniform(0, phi2) for _ in range(len(part.position))) 258 | v_u1 = map(op.mul, u1, 259 | map(op.sub, part.best, part.position)) 260 | v_u2 = map(op.mul, u2, 261 | map(op.sub, best.position, part.position)) 262 | #### avoid too small speed 263 | # part.speed = array.array('d', map(op.add, part.speed, 264 | # map(op.add, v_u1, v_u2))) 265 | def to_int(speed): 266 | import numpy as np 267 | if np.sum([int(i) for i in speed]) == 0: 268 | return [np.sign(i) for i in speed] 269 | else: 270 | return [int(i) for i in speed] 271 | part.speed = array.array('d', to_int(map(op.add, part.speed, 272 | map(op.add, v_u1, v_u2)))) 273 | for i, speed in enumerate(part.speed): 274 | if speed < self.smin[i]: 275 | part.speed[i] = self.smin[i] 276 | elif speed > self.smax[i]: 277 | part.speed[i] = self.smax[i] 278 | #### int position 279 | position = map(op.add, part.position, part.speed) 280 | position = [int(i) for i in position] 281 | part.position[:] = array.array('d', position) 282 | #### position revision 283 | for i, j in enumerate(self._bounds.values()): 284 | if part.position[i] < j[0]: 285 | part.position[i] = j[0] 286 | elif part.position[i] > j[1]: 287 | part.position[i] = j[1] 288 | 289 | def particle2dict(self, particle): 290 | return dict([(k, v) for k, v in zip(self.bounds.keys(), 291 | particle.position)]) 292 | 293 | @_copydoc(Solver.optimize) 294 | def optimize(self, f, maximize=True, pmap=map): 295 | 296 | @functools.wraps(f) 297 | def evaluate(d): 298 | return f(**d) 299 | 300 | if maximize: 301 | fit = 1.0 302 | else: 303 | fit = -1.0 304 | 305 | #### different particles 306 | # pop = [self.generate() for _ in range(self.num_particles)] 307 | pop = [] 308 | pop0 = [self.generate() for _ in range(self.num_particles*2)] 309 | for i in pop0: 310 | if len(pop) == self.num_particles: 311 | break 312 | if i not in pop: 313 | pop.append(i) 314 | best = None 315 | 316 | #### add termination condition, if best get no-update then break the loop 317 | termination_count = 5 318 | for g in range(self.num_generations): 319 | found_new_best = False 320 | fitnesses = pmap(evaluate, list(map(self.particle2dict, pop))) 321 | for part, fitness in zip(pop, fitnesses): 322 | part.fitness = fit * util.score(fitness) 323 | if not part.best or part.best_fitness < part.fitness: 324 | part.best = part.position 325 | part.best_fitness = part.fitness 326 | if not best or best.fitness < part.fitness: 327 | best = part.clone() 328 | found_new_best = True 329 | for part in pop: 330 | self.updateParticle(part, best, self.phi1, self.phi2) 331 | if found_new_best is False: 332 | termination_count -= 1 333 | if termination_count == 0: 334 | break 335 | 336 | return dict([(k, v) 337 | for k, v in zip(self.bounds.keys(), best.position)]), None 338 | -------------------------------------------------------------------------------- /optunity/solvers/RandomSearch.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | 3 | # Copyright (c) 2014 KU Leuven, ESAT-STADIUS 4 | # All rights reserved. 5 | # 6 | # Redistribution and use in source and binary forms, with or without 7 | # modification, are permitted provided that the following conditions 8 | # are met: 9 | # 10 | # 1. Redistributions of source code must retain the above copyright 11 | # notice, this list of conditions and the following disclaimer. 12 | # 13 | # 2. Redistributions in binary form must reproduce the above copyright 14 | # notice, this list of conditions and the following disclaimer in the 15 | # documentation and/or other materials provided with the distribution. 16 | # 17 | # 3. Neither name of copyright holders nor the names of its contributors 18 | # may be used to endorse or promote products derived from this software 19 | # without specific prior written permission. 20 | # 21 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 22 | # ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 23 | # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 24 | # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR 25 | # CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, 26 | # EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, 27 | # PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR 28 | # PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF 29 | # LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 30 | # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 31 | # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 32 | 33 | import operator as op 34 | import random 35 | 36 | from ..functions import static_key_order 37 | from .solver_registry import register_solver 38 | from .util import Solver, _copydoc, shrink_bounds, uniform_in_bounds 39 | from . import util 40 | 41 | @register_solver('random search', 42 | 'random parameter tuples sampled uniformly within box constraints', 43 | ['Tests random parameter tuples sampled uniformly within the box constraints.', 44 | ' ', 45 | 'This function requires the following arguments:', 46 | '- num_evals :: number of tuples to test', 47 | '- box constraints via keywords: constraints are lists [lb, ub]', 48 | ' ', 49 | 'This solver performs num_evals function evaluations.', 50 | ' ', 51 | 'This solver implements the technique described here:', 52 | 'Bergstra, James, and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research 13 (2012): 281-305.'] 53 | ) 54 | class RandomSearch(Solver): 55 | """ 56 | .. include:: /global.rst 57 | 58 | Please refer to |randomsearch| for more details about this algorithm. 59 | 60 | """ 61 | 62 | 63 | def __init__(self, num_evals, **kwargs): 64 | """Initializes the solver with bounds and a number of allowed evaluations. 65 | kwargs must be a dictionary of parameter-bound pairs representing the box constraints. 66 | Bounds are a 2-element list: [lower_bound, upper_bound]. 67 | 68 | >>> s = RandomSearch(x=[0, 1], y=[-1, 2], num_evals=50) 69 | >>> s.bounds['x'] 70 | [0, 1] 71 | >>> s.bounds['y'] 72 | [-1, 2] 73 | >>> s.num_evals 74 | 50 75 | 76 | """ 77 | assert all([len(v) == 2 and v[0] <= v[1] 78 | for v in kwargs.values()]), 'kwargs.values() are not [lb, ub] pairs' 79 | self._bounds = kwargs 80 | self._num_evals = num_evals 81 | 82 | @staticmethod 83 | def suggest_from_box(num_evals, **kwargs): 84 | """Creates a RandomSearch solver that uses ``num_evals`` evaluations 85 | within given bounds (lb, ub). The bounds are first tightened, resulting in 86 | new bounds covering 99% of the area. 87 | 88 | >>> s = RandomSearch.suggest_from_box(30, x=[0, 1], y=[-1, 0], z=[-1, 1]) 89 | >>> s['x'] #doctest:+SKIP 90 | [0.005, 0.995] 91 | >>> s['y'] #doctest:+SKIP 92 | [-0.995, -0.005] 93 | >>> s['z'] #doctest:+SKIP 94 | [-0.99, 0.99] 95 | >>> s['num_evals'] 96 | 30 97 | 98 | Verify that we can effectively make a solver from box. 99 | 100 | >>> s = RandomSearch.suggest_from_box(30, x=[0, 1], y=[-1, 0], z=[-1, 1]) 101 | >>> solver = RandomSearch(**s) 102 | 103 | """ 104 | d = shrink_bounds(kwargs) 105 | d['num_evals'] = num_evals 106 | return d 107 | 108 | @property 109 | def upper(self, par): 110 | """Returns the upper bound of par.""" 111 | return self._bounds[par][1] 112 | 113 | @property 114 | def lower(self, par): 115 | """Returns the lower bound of par.""" 116 | return self._bounds[par][0] 117 | 118 | @property 119 | def bounds(self): 120 | """Returns a dictionary containing the box constraints.""" 121 | return self._bounds 122 | 123 | @property 124 | def num_evals(self): 125 | """Returns the number of evaluations this solver may do.""" 126 | return self._num_evals 127 | 128 | @_copydoc(Solver.optimize) 129 | def optimize(self, f, maximize=True, pmap=map): 130 | 131 | def generate_rand_args(len=1): 132 | # return [uniform_in_bounds(self.bounds)] 133 | return [[random.uniform(bounds[0], bounds[1]) for _ in range(len)] 134 | for _, bounds in self.bounds.items()] 135 | 136 | best_pars = None 137 | f = static_key_order(self.bounds.keys())(f) 138 | 139 | if maximize: 140 | comp = lambda score, best: score > best 141 | else: 142 | comp = lambda score, best: score < best 143 | 144 | tuples = generate_rand_args(self.num_evals) 145 | scores = pmap(f, *tuples) 146 | scores = map(util.score, scores) 147 | 148 | if maximize: 149 | comp = max 150 | else: 151 | comp = min 152 | best_idx, _ = comp(enumerate(scores), key=op.itemgetter(1)) 153 | best_pars = op.itemgetter(best_idx)(list(zip(*tuples))) 154 | return dict([(k, v) for k, v in zip(self.bounds.keys(), best_pars)]), None 155 | -------------------------------------------------------------------------------- /optunity/solvers/Sobol.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | 3 | # Copyright (c) 2014 KU Leuven, ESAT-STADIUS 4 | # All rights reserved. 5 | # 6 | # Redistribution and use in source and binary forms, with or without 7 | # modification, are permitted provided that the following conditions 8 | # are met: 9 | # 10 | # 1. Redistributions of source code must retain the above copyright 11 | # notice, this list of conditions and the following disclaimer. 12 | # 13 | # 2. Redistributions in binary form must reproduce the above copyright 14 | # notice, this list of conditions and the following disclaimer in the 15 | # documentation and/or other materials provided with the distribution. 16 | # 17 | # 3. Neither name of copyright holders nor the names of its contributors 18 | # may be used to endorse or promote products derived from this software 19 | # without specific prior written permission. 20 | # 21 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 22 | # ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 23 | # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 24 | # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR 25 | # CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, 26 | # EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, 27 | # PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR 28 | # PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF 29 | # LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 30 | # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 31 | # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 32 | 33 | import math 34 | import operator as op 35 | import random 36 | import array 37 | import functools 38 | try: 39 | # Python 2 40 | from itertools import izip_longest 41 | except ImportError: 42 | # Python 3 43 | from itertools import zip_longest as izip_longest 44 | 45 | from ..functions import static_key_order 46 | from .solver_registry import register_solver 47 | from .util import Solver, _copydoc, uniform_in_bounds 48 | from . import util 49 | 50 | try: 51 | # Python 2 52 | irange = irange 53 | except NameError: 54 | #Python 3 55 | irange = range 56 | 57 | # Parts of this implementation were obtained from here: 58 | # obtained from http://people.sc.fsu.edu/~jburkardt/py_src/sobol/sobol.html 59 | # we have removed all dependencies on numpy and replaced with standard 60 | # library functions 61 | # All functions we reused are annotated. 62 | 63 | # The MIT License (MIT) 64 | # 65 | # Copyright (c) 2014 John Burkardt 66 | # 67 | # Permission is hereby granted, free of charge, to any person obtaining a copy 68 | # of this software and associated documentation files (the "Software"), to deal 69 | # in the Software without restriction, including without limitation the rights 70 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 71 | # copies of the Software, and to permit persons to whom the Software is 72 | # furnished to do so, subject to the following conditions: 73 | # 74 | # The above copyright notice and this permission notice shall be included in 75 | # all copies or substantial portions of the Software. 76 | # 77 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 78 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 79 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 80 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 81 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 82 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 83 | # THE SOFTWARE. 84 | 85 | 86 | 87 | @register_solver('sobol', 88 | 'sample the search space using a Sobol sequence', 89 | ['Generates a Sobol sequence of points to sample in the search space.', 90 | '', 91 | 'A Sobol sequence is a low discrepancy quasirandom sequence,', 92 | 'specifically designed to cover the search space evenly.', 93 | '', 94 | 'Details are available at: http://en.wikipedia.org/wiki/Sobol_sequence', 95 | 'This sampling method should be preferred over sampling uniformly at random.']) 96 | class Sobol(Solver): 97 | """ 98 | .. include:: /global.rst 99 | 100 | Please refer to |sobol| for details on this algorithm. 101 | """ 102 | 103 | 104 | def __init__(self, num_evals, seed=None, skip=None, **kwargs): 105 | """ 106 | Initializes a Sobol sequence solver. 107 | 108 | :param num_evals: number of evaluations to use 109 | :type num_evals: int 110 | :param skip: the number of initial elements of the sequence to skip, if None a random skip is generated 111 | :type skip: int or None 112 | :param kwargs: box constraints for each hyperparameter 113 | :type kwargs: {'name': [lb, ub], ...} 114 | 115 | The search space is rescaled to the unit hypercube before the solving process begins. 116 | 117 | """ 118 | 119 | assert all([len(v) == 2 and v[0] <= v[1] 120 | for v in kwargs.values()]), 'kwargs.values() are not [lb, ub] pairs' 121 | self._bounds = kwargs 122 | self._num_evals = num_evals 123 | self._skip = skip if skip else random.randint(200, 1000) 124 | 125 | 126 | @_copydoc(Solver.optimize) 127 | def optimize(self, f, maximize=True, pmap=map): 128 | 129 | sequence = Sobol.i4_sobol_generate(len(self.bounds), self.num_evals, self.skip) 130 | scaled = list(map(lambda x: util.scale_unit_to_bounds(x, self.bounds.values()), 131 | sequence)) 132 | 133 | best_pars = None 134 | @functools.wraps(f) 135 | def fwrap(args): 136 | kwargs = dict([(k, v) for k, v in zip(self.bounds.keys(), args)]) 137 | return f(**kwargs) 138 | 139 | if maximize: 140 | comp = lambda score, best: score > best 141 | else: 142 | comp = lambda score, best: score < best 143 | 144 | scores = pmap(fwrap, scaled) 145 | scores = map(util.score, scores) 146 | 147 | if maximize: 148 | comp = max 149 | else: 150 | comp = min 151 | best_idx, _ = comp(enumerate(scores), key=op.itemgetter(1)) 152 | best_pars = scaled[best_idx] #op.itemgetter(best_idx)(scaled) 153 | return dict([(k, v) for k, v in zip(self.bounds.keys(), best_pars)]), None 154 | 155 | @property 156 | def bounds(self): return self._bounds 157 | 158 | @property 159 | def num_evals(self): return self._num_evals 160 | 161 | @property 162 | def skip(self): return self._skip 163 | 164 | @staticmethod 165 | def suggest_from_box(num_evals, **kwargs): 166 | """Create a configuration for a Sobol solver. 167 | 168 | :param num_evals: number of permitted function evaluations 169 | :type num_evals: int 170 | :param kwargs: box constraints 171 | :type kwargs: {'param': [lb, ub], ...} 172 | 173 | 174 | Verify that we can effectively make a solver from box. 175 | 176 | >>> s = Sobol.suggest_from_box(30, x=[0, 1], y=[-1, 0], z=[-1, 1]) 177 | >>> solver = Sobol(**s) 178 | 179 | """ 180 | d = util.shrink_bounds(kwargs) 181 | d['num_evals'] = num_evals 182 | return d 183 | 184 | @staticmethod 185 | def bitwise_xor(a, b): 186 | """ 187 | Returns the bitwise_xor of a and b as a bitstring. 188 | 189 | :param a: first number 190 | :type a: int 191 | :param b: second number 192 | :type b: int 193 | 194 | >>> Sobol.bitwise_xor(13, 17) 195 | 28 196 | >>> Sobol.bitwise_xor(31, 5) 197 | 26 198 | 199 | """ 200 | to_binary = lambda x: bin(x)[2:] 201 | abin = to_binary(a)[::-1] 202 | bbin = to_binary(b)[::-1] 203 | xor = lambda x, y: '0' if (x == y) else '1' 204 | lst = [xor(x, y) for x, y in izip_longest(abin, bbin, fillvalue='0')] 205 | lst.reverse() 206 | return int("".join(lst), 2) 207 | 208 | @staticmethod 209 | def i4_bit_hi1 ( n ): 210 | """ 211 | Returns the position of the high 1 bit base 2 in an integer. 212 | 213 | :param n: the integer to be measured 214 | :type n: int 215 | :returns: (int) the number of bits base 2 216 | 217 | This was taken from http://people.sc.fsu.edu/~jburkardt/py_src/sobol/sobol.html 218 | Licensing: 219 | This code is distributed under the MIT license. 220 | 221 | Modified: 222 | 22 February 2011 223 | 224 | Author: 225 | Original MATLAB version by John Burkardt. 226 | PYTHON version by Corrado Chisari 227 | 228 | """ 229 | 230 | i = math.floor ( n ) 231 | bit = 0 232 | while ( 1 ): 233 | if ( i <= 0 ): 234 | break 235 | bit += 1 236 | i = math.floor ( i / 2. ) 237 | return bit 238 | 239 | @staticmethod 240 | def i4_bit_lo0 ( n ): 241 | """ 242 | Returns the position of the low 0 bit base 2 in an integer. 243 | 244 | :param n: the integer to be measured 245 | :type n: int 246 | :returns: (int) the number of bits base 2 247 | 248 | This was taken from http://people.sc.fsu.edu/~jburkardt/py_src/sobol/sobol.html 249 | Licensing: 250 | This code is distributed under the MIT license. 251 | 252 | Modified: 253 | 22 February 2011 254 | 255 | Author: 256 | Original MATLAB version by John Burkardt. 257 | PYTHON version by Corrado Chisari 258 | 259 | """ 260 | bit = 0 261 | i = math.floor ( n ) 262 | while ( 1 ): 263 | bit = bit + 1 264 | i2 = math.floor ( i / 2. ) 265 | if ( i == 2 * i2 ): 266 | break 267 | 268 | i = i2 269 | return bit 270 | 271 | @staticmethod 272 | def i4_sobol_generate ( m, n, skip ): 273 | """Generates a Sobol sequence. 274 | 275 | :param m: the number of dimensions (our implementation supports up to 40) 276 | :type m: int 277 | :param n: the length of the sequence to generate 278 | :type n: int 279 | :param skip: the number of initial elements in the sequence to skip 280 | :type skip: int 281 | :returns: a list of length n containing m-dimensional points of the Sobol sequence 282 | 283 | """ 284 | 285 | r = [Sobol.i4_sobol(m, seed)[0] for seed in irange(skip, skip + n)] 286 | return r 287 | 288 | @staticmethod 289 | def i4_sobol ( dim_num, seed ): 290 | """ 291 | Generates a new quasi-random Sobol vector with each call. 292 | 293 | :param dim_num: number of dimensions of the Sobol vector 294 | :type dim_num: int 295 | :param seed: the seed to use to generate the Sobol vector 296 | :type seed: int 297 | :returns: the next quasirandom vector and the next seed to use 298 | 299 | This was taken from http://people.sc.fsu.edu/~jburkardt/py_src/sobol/sobol.html 300 | Licensing: 301 | This code is distributed under the MIT license. 302 | 303 | Modified: 304 | 22 February 2011 305 | 306 | Author: 307 | Original MATLAB version by John Burkardt. 308 | PYTHON version by Corrado Chisari 309 | 310 | """ 311 | global atmost 312 | global dim_max 313 | global dim_num_save 314 | global initialized 315 | global lastq 316 | global log_max 317 | global maxcol 318 | global poly 319 | global recipd 320 | global seed_save 321 | global v 322 | 323 | if ( not 'initialized' in globals().keys() ): 324 | initialized = 0 325 | dim_num_save = -1 326 | 327 | if ( not initialized or dim_num != dim_num_save ): 328 | initialized = 1 329 | dim_max = 40 330 | dim_num_save = -1 331 | log_max = 30 332 | seed_save = -1 333 | # 334 | # Initialize (part of) V. 335 | # 336 | v = [[0] * dim_max for _ in irange(log_max)] 337 | v[0][0:40] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, \ 338 | 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, \ 339 | 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, \ 340 | 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 ] 341 | 342 | v[1][2:40] = [1, 3, 1, 3, 1, 3, 3, 1, \ 343 | 3, 1, 3, 1, 3, 1, 1, 3, 1, 3, \ 344 | 1, 3, 1, 3, 3, 1, 3, 1, 3, 1, \ 345 | 3, 1, 1, 3, 1, 3, 1, 3, 1, 3 ] 346 | 347 | v[2][3:40] = [7, 5, 1, 3, 3, 7, 5, \ 348 | 5, 7, 7, 1, 3, 3, 7, 5, 1, 1, \ 349 | 5, 3, 3, 1, 7, 5, 1, 3, 3, 7, \ 350 | 5, 1, 1, 5, 7, 7, 5, 1, 3, 3 ] 351 | 352 | v[3][5:40] = [1, 7, 9, 13, 11, \ 353 | 1, 3, 7, 9, 5, 13, 13, 11, 3, 15, \ 354 | 5, 3, 15, 7, 9, 13, 9, 1, 11, 7, \ 355 | 5, 15, 1, 15, 11, 5, 3, 1, 7, 9 ] 356 | 357 | v[4][7:40] = [9, 3,27, \ 358 | 15,29,21,23,19,11,25, 7,13,17, \ 359 | 1,25,29, 3,31,11, 5,23,27,19, \ 360 | 21, 5, 1,17,13, 7,15, 9,31, 9 ] 361 | 362 | v[5][13:40] = [37, 33, 7, 5,11, 39, 63, \ 363 | 27, 17, 15, 23, 29, 3, 21, 13, 31, 25, \ 364 | 9, 49, 33, 19, 29, 11, 19, 27, 15, 25 ] 365 | 366 | v[6][19:40] = [13, \ 367 | 33, 115, 41, 79, 17, 29, 119, 75, 73, 105, \ 368 | 7, 59, 65, 21, 3, 113, 61, 89, 45, 107 ] 369 | 370 | v[7][37:40] = [7, 23, 39 ] 371 | # 372 | # Set POLY. 373 | # 374 | poly= [ \ 375 | 1, 3, 7, 11, 13, 19, 25, 37, 59, 47, \ 376 | 61, 55, 41, 67, 97, 91, 109, 103, 115, 131, \ 377 | 193, 137, 145, 143, 241, 157, 185, 167, 229, 171, \ 378 | 213, 191, 253, 203, 211, 239, 247, 285, 369, 299 ] 379 | 380 | atmost = 2**log_max - 1 381 | # 382 | # Find the number of bits in ATMOST. 383 | # 384 | maxcol = Sobol.i4_bit_hi1 ( atmost ) 385 | # 386 | # Initialize row 1 of V. 387 | # 388 | # v[0,0:maxcol] = 1 389 | for i in irange(maxcol): 390 | v[i][0] = 1 391 | 392 | # 393 | # Things to do only if the dimension changed. 394 | # 395 | if ( dim_num != dim_num_save ): 396 | # 397 | # Check parameters. 398 | # 399 | if ( dim_num < 1 or dim_max < dim_num ): 400 | raise ValueError('I4_SOBOL - Fatal error! The spatial dimension DIM_NUM should satisfy: 1 <= DIM_NUM <= %d, But this input value is DIM_NUM = %d' % (dim_max, dim_num)) 401 | 402 | dim_num_save = dim_num 403 | # 404 | # Initialize the remaining rows of V. 405 | # 406 | for i in irange(2 , dim_num+1): 407 | # 408 | # The bits of the integer POLY(I) gives the form of polynomial I. 409 | # 410 | # Find the degree of polynomial I from binary encoding. 411 | # 412 | j = poly[i-1] 413 | m = 0 414 | while ( 1 ): 415 | j = math.floor ( j / 2. ) 416 | if ( j <= 0 ): 417 | break 418 | m = m + 1 419 | # 420 | # Expand this bit pattern to separate components of the logical array INCLUD. 421 | # 422 | j = poly[i-1] 423 | includ = [0 for _ in irange(m)] 424 | for k in irange(m, 0, -1): 425 | j2 = math.floor ( j / 2. ) 426 | includ[k-1] = (j != 2 * j2 ) 427 | j = j2 428 | # 429 | # Calculate the remaining elements of row I as explained 430 | # in Bratley and Fox, section 2. 431 | # 432 | for j in irange( m+1, maxcol+1 ): 433 | newv = v[j-m-1][i-1] 434 | l = 1 435 | for k in irange(1, m+1): 436 | l = 2 * l 437 | if ( includ[k-1] ): 438 | newv = Sobol.bitwise_xor ( int(newv), int(l * v[j-k-1][i-1]) ) 439 | v[j-1][i-1] = newv 440 | # 441 | # Multiply columns of V by appropriate power of 2. 442 | # 443 | l = 1 444 | for j in irange( maxcol-1, 0, -1): 445 | l = 2 * l 446 | v[j-1][0:dim_num] = map(lambda x: x * l, v[j-1][0:dim_num]) 447 | # 448 | # RECIPD is 1/(common denominator of the elements in V). 449 | # 450 | recipd = 1.0 / ( 2 * l ) 451 | lastq = [0 for _ in irange(dim_num)] 452 | 453 | seed = int(math.floor ( seed )) 454 | 455 | if ( seed < 0 ): 456 | seed = 0 457 | 458 | if ( seed == 0 ): 459 | l = 1 460 | lastq = [0 for _ in irange(dim_num)] 461 | 462 | elif ( seed == seed_save + 1 ): 463 | # 464 | # Find the position of the right-hand zero in SEED. 465 | # 466 | l = Sobol.i4_bit_lo0 ( seed ) 467 | 468 | elif ( seed <= seed_save ): 469 | 470 | seed_save = 0 471 | l = 1 472 | lastq = [0 for _ in irange(dim_num)] 473 | 474 | for seed_temp in irange( int(seed_save), int(seed)): 475 | l = Sobol.i4_bit_lo0 ( seed_temp ) 476 | for i in irange(1 , dim_num+1): 477 | lastq[i-1] = Sobol.bitwise_xor ( int(lastq[i-1]), int(v[l-1][i-1]) ) 478 | 479 | l = Sobol.i4_bit_lo0 ( seed ) 480 | 481 | elif ( seed_save + 1 < seed ): 482 | 483 | for seed_temp in irange( int(seed_save + 1), int(seed) ): 484 | l = Sobol.i4_bit_lo0 ( seed_temp ) 485 | for i in irange(1, dim_num+1): 486 | lastq[i-1] = Sobol.bitwise_xor ( int(lastq[i-1]), int(v[l-1][i-1]) ) 487 | 488 | l = Sobol.i4_bit_lo0 ( seed ) 489 | # 490 | # Check that the user is not calling too many times! 491 | # 492 | if ( maxcol < l ): 493 | raise ValueError('I4_SOBOL - Fatal error! Too many calls: MAXCOL = %d, L = %d' % (maxcol, l)) 494 | # 495 | # Calculate the new components of QUASI. 496 | # 497 | quasi = [0 for _ in irange(dim_num)] 498 | for i in irange( 1, dim_num+1): 499 | quasi[i-1] = lastq[i-1] * recipd 500 | lastq[i-1] = Sobol.bitwise_xor ( int(lastq[i-1]), int(v[l-1][i-1]) ) 501 | 502 | seed_save = seed 503 | seed = seed + 1 504 | 505 | return [ quasi, seed ] 506 | 507 | @staticmethod 508 | def maxdim(): 509 | """The maximum dimensionality that we currently support.""" 510 | return 40 511 | 512 | -------------------------------------------------------------------------------- /optunity/solvers/TPE.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | 3 | # Copyright (c) 2014 KU Leuven, ESAT-STADIUS 4 | # All rights reserved. 5 | # 6 | # Redistribution and use in source and binary forms, with or without 7 | # modification, are permitted provided that the following conditions 8 | # are met: 9 | # 10 | # 1. Redistributions of source code must retain the above copyright 11 | # notice, this list of conditions and the following disclaimer. 12 | # 13 | # 2. Redistributions in binary form must reproduce the above copyright 14 | # notice, this list of conditions and the following disclaimer in the 15 | # documentation and/or other materials provided with the distribution. 16 | # 17 | # 3. Neither name of copyright holders nor the names of its contributors 18 | # may be used to endorse or promote products derived from this software 19 | # without specific prior written permission. 20 | # 21 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 22 | # ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 23 | # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 24 | # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR 25 | # CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, 26 | # EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, 27 | # PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR 28 | # PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF 29 | # LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 30 | # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 31 | # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 32 | 33 | from .solver_registry import register_solver 34 | from .util import Solver, _copydoc 35 | import functools 36 | 37 | import random 38 | 39 | _numpy_available = True 40 | try: 41 | import numpy 42 | except ImportError: 43 | _numpy_available = False 44 | 45 | _hyperopt_available = True 46 | try: 47 | import hyperopt 48 | except ImportError: 49 | _hyperopt_available = False 50 | 51 | class TPE(Solver): 52 | """ 53 | .. include:: /global.rst 54 | 55 | This solver implements the Tree-structured Parzen Estimator, as described in [TPE2011]_. 56 | This solver uses Hyperopt in the back-end and exposes the TPE estimator with uniform priors. 57 | 58 | Please refer to |tpe| for details about this algorithm. 59 | 60 | .. [TPE2011] Bergstra, James S., et al. "Algorithms for hyper-parameter optimization." Advances in Neural Information Processing Systems. 2011 61 | 62 | """ 63 | 64 | def __init__(self, num_evals=100, seed=None, **kwargs): 65 | """ 66 | 67 | Initialize the TPE solver. 68 | 69 | :param num_evals: number of permitted function evaluations 70 | :type num_evals: int 71 | :param seed: the random seed to be used 72 | :type seed: double 73 | :param kwargs: box constraints for each hyperparameter 74 | :type kwargs: {'name': [lb, ub], ...} 75 | 76 | 77 | """ 78 | if not _hyperopt_available: 79 | raise ImportError('This solver requires Hyperopt but it is missing.') 80 | if not _numpy_available: 81 | raise ImportError('This solver requires NumPy but it is missing.') 82 | 83 | self._seed = seed 84 | self._bounds = kwargs 85 | self._num_evals = num_evals 86 | 87 | @staticmethod 88 | def suggest_from_box(num_evals, **kwargs): 89 | """ 90 | Verify that we can effectively make a solver from box. 91 | 92 | >>> s = TPE.suggest_from_box(30, x=[0, 1], y=[-1, 0], z=[-1, 1]) 93 | >>> solver = TPE(**s) #doctest:+SKIP 94 | 95 | """ 96 | d = dict(kwargs) 97 | d['num_evals'] = num_evals 98 | return d 99 | 100 | @property 101 | def seed(self): 102 | return self._seed 103 | 104 | @property 105 | def bounds(self): 106 | return self._bounds 107 | 108 | @property 109 | def num_evals(self): 110 | return self._num_evals 111 | 112 | @_copydoc(Solver.optimize) 113 | def optimize(self, f, maximize=True, pmap=map): 114 | 115 | if maximize: 116 | def obj(args): 117 | kwargs = dict([(k, v) for k, v in zip(self.bounds.keys(), args)]) 118 | return -f(**kwargs) 119 | 120 | else: 121 | def obj(args): 122 | kwargs = dict([(k, v) for k, v in zip(self.bounds.keys(), args)]) 123 | return f(**kwargs) 124 | 125 | seed = self.seed if self.seed else random.randint(0, 9999999999) 126 | algo = functools.partial(hyperopt.tpe.suggest, seed=seed) 127 | 128 | space = [hyperopt.hp.uniform(k, v[0], v[1]) for k, v in self.bounds.items()] 129 | best = hyperopt.fmin(obj, space=space, algo=algo, max_evals=self.num_evals) 130 | return best, None 131 | 132 | 133 | # TPE is a simple wrapper around Hyperopt's TPE solver 134 | if _hyperopt_available and _numpy_available: 135 | TPE = register_solver('TPE', 'Tree of Parzen estimators', 136 | ['TPE: Tree of Parzen Estimators'] 137 | )(TPE) 138 | -------------------------------------------------------------------------------- /optunity/solvers/__init__.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | 3 | # Copyright (c) 2014 KU Leuven, ESAT-STADIUS 4 | # All rights reserved. 5 | # 6 | # Redistribution and use in source and binary forms, with or without 7 | # modification, are permitted provided that the following conditions 8 | # are met: 9 | # 10 | # 1. Redistributions of source code must retain the above copyright 11 | # notice, this list of conditions and the following disclaimer. 12 | # 13 | # 2. Redistributions in binary form must reproduce the above copyright 14 | # notice, this list of conditions and the following disclaimer in the 15 | # documentation and/or other materials provided with the distribution. 16 | # 17 | # 3. Neither name of copyright holders nor the names of its contributors 18 | # may be used to endorse or promote products derived from this software 19 | # without specific prior written permission. 20 | # 21 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 22 | # ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 23 | # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 24 | # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR 25 | # CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, 26 | # EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, 27 | # PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR 28 | # PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF 29 | # LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 30 | # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 31 | # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 32 | 33 | """Module to take care of registering solvers for use in the main Optunity API. 34 | 35 | Main classes in this module: 36 | 37 | * :class:`GridSearch` 38 | * :class:`RandomSearch` 39 | * :class:`NelderMead` 40 | * :class:`ParticleSwarm` 41 | * :class:`CMA_ES` 42 | * :class:`TPE` 43 | * :class:`Sobol` 44 | * :class:`BayesOpt` 45 | 46 | .. warning:: 47 | :class:`CMA_ES` require DEAP_ and NumPy_. 48 | 49 | .. _DEAP: https://code.google.com/p/deap/ 50 | .. _NumPy: http://www.numpy.org 51 | 52 | .. warning:: 53 | :class:`TPE` require Hyperopt_ and NumPy_. 54 | 55 | .. _Hyperopt: http://jaberg.github.io/hyperopt/ 56 | .. _NumPy: http://www.numpy.org 57 | 58 | .. warning:: 59 | :class:`BayesOpt` require BayesOpt_ and NumPy_. 60 | 61 | .. _BayesOpt: http://rmcantin.bitbucket.org/html/ 62 | .. _NumPy: http://www.numpy.org 63 | 64 | .. moduleauthor:: Marc Claesen 65 | 66 | """ 67 | 68 | from .GridSearch import GridSearch 69 | from .RandomSearch import RandomSearch 70 | from .NelderMead import NelderMead 71 | from .ParticleSwarm import ParticleSwarm 72 | from .CMAES import CMA_ES 73 | from .TPE import TPE 74 | from .Sobol import Sobol 75 | from .BayesOpt import BayesOpt 76 | -------------------------------------------------------------------------------- /optunity/solvers/solver_registry.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | 3 | # Copyright (c) 2014 KU Leuven, ESAT-STADIUS 4 | # All rights reserved. 5 | # 6 | # Redistribution and use in source and binary forms, with or without 7 | # modification, are permitted provided that the following conditions 8 | # are met: 9 | # 10 | # 1. Redistributions of source code must retain the above copyright 11 | # notice, this list of conditions and the following disclaimer. 12 | # 13 | # 2. Redistributions in binary form must reproduce the above copyright 14 | # notice, this list of conditions and the following disclaimer in the 15 | # documentation and/or other materials provided with the distribution. 16 | # 17 | # 3. Neither name of copyright holders nor the names of its contributors 18 | # may be used to endorse or promote products derived from this software 19 | # without specific prior written permission. 20 | # 21 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 22 | # ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 23 | # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 24 | # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR 25 | # CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, 26 | # EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, 27 | # PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR 28 | # PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF 29 | # LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 30 | # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 31 | # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 32 | 33 | """Module to take care of registering solvers for use in the main Optunity API. 34 | 35 | Main functions in this module: 36 | 37 | * :func:`register_solver` 38 | * :func:`manual` 39 | * :func:`get` 40 | 41 | .. moduleauthor:: Marc Claesen 42 | 43 | """ 44 | 45 | __all__ = ['get', 'manual', 'register_solver', 'solver_names'] 46 | 47 | __registered_solvers = {} 48 | 49 | 50 | def __register(cls): 51 | global __registered_solvers 52 | __registered_solvers[cls.name.lower()] = cls 53 | 54 | 55 | def get(solver_name): 56 | """Returns the class of the solver registered under given name. 57 | 58 | :param solver_name: name of the solver 59 | :returns: the solver class""" 60 | global __registered_solvers 61 | return __registered_solvers[solver_name] 62 | 63 | 64 | def manual(): 65 | """ 66 | Returns the general manual of Optunity, with a brief introduction of all registered solvers. 67 | 68 | :returns: the manual as a list of strings (lines) 69 | """ 70 | global __registered_solvers 71 | manual = ['Optunity: optimization algorithms for hyperparameter tuning', ' ', 72 | 'The following solvers are available:'] 73 | for name, cls in __registered_solvers.items(): 74 | manual.append(name + ' :: ' + cls.desc_brief) 75 | manual.append(' ') 76 | manual.append("For a solver-specific manual, include its name in the request.") 77 | manual.append("For more detailed info, please consult the Optunity documentation at:") 78 | manual.append("http://docs.optunity.net") 79 | return manual 80 | 81 | 82 | def solver_names(): 83 | """Returns a list of all registered solvers.""" 84 | global __registered_solvers 85 | return list(__registered_solvers.keys()) 86 | 87 | 88 | def register_solver(name, desc_brief, desc_full): 89 | """Class decorator to register a :class:`optunity.solvers.Solver` subclass in the registry. 90 | Registered solvers will be available through Optunity's main API functions, 91 | e.g. :func:`optunity.make_solver` and :func:`optunity.manual`. 92 | 93 | :param name: name to register the solver with 94 | :param desc_brief: one-line description of the solver 95 | :param desc_full: 96 | extensive description and manual of the solver 97 | returns a list of strings representing manual lines 98 | :returns: a class decorator to register solvers in the solver registry 99 | 100 | The resulting class decorator attaches attributes to the class before registering: 101 | 102 | :_name: the name using which the solver is registered 103 | :_desc_full: extensive description and manual as list of strings (lines) 104 | :_desc_brief: attribute with one-line description 105 | 106 | These attributes will be available as class properties. 107 | """ 108 | def class_wrapper(cls): 109 | cls.name = name 110 | cls.desc_brief = desc_brief 111 | cls.desc_full = desc_full 112 | __register(cls) 113 | return cls 114 | return class_wrapper 115 | -------------------------------------------------------------------------------- /optunity/solvers/util.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | 3 | # Copyright (c) 2014 KU Leuven, ESAT-STADIUS 4 | # All rights reserved. 5 | # 6 | # Redistribution and use in source and binary forms, with or without 7 | # modification, are permitted provided that the following conditions 8 | # are met: 9 | # 10 | # 1. Redistributions of source code must retain the above copyright 11 | # notice, this list of conditions and the following disclaimer. 12 | # 13 | # 2. Redistributions in binary form must reproduce the above copyright 14 | # notice, this list of conditions and the following disclaimer in the 15 | # documentation and/or other materials provided with the distribution. 16 | # 17 | # 3. Neither name of copyright holders nor the names of its contributors 18 | # may be used to endorse or promote products derived from this software 19 | # without specific prior written permission. 20 | # 21 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 22 | # ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 23 | # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 24 | # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR 25 | # CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, 26 | # EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, 27 | # PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR 28 | # PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF 29 | # LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 30 | # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 31 | # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 32 | 33 | 34 | import abc 35 | import random 36 | import threading 37 | 38 | def uniform_in_bounds(bounds): 39 | """Generates a random uniform sample between ``bounds``. 40 | 41 | :param bounds: the bounds we must adhere to 42 | :type bounds: dict {"name": [lb ub], ...} 43 | """ 44 | return map(random.uniform, *zip(*bounds.values())) 45 | 46 | def scale_unit_to_bounds(seq, bounds): 47 | """ 48 | Scales all elements in seq (unit hypercube) to the box constraints in bounds. 49 | 50 | :param seq: the sequence in the unit hypercube to scale 51 | :type seq: iterable 52 | :param bounds: bounds to scale to 53 | :type seq: iterable of [lb, ub] pairs 54 | :returns: a list of scaled elements of `seq` 55 | 56 | >>> scale_unit_to_bounds([0.0, 0.5, 0.5, 1.0], [[-1.0, 2.0], [-2.0, 0.0], [0.0, 3.0], [0.0, 2.0]]) 57 | [-1.0, -1.0, 1.5, 2.0] 58 | 59 | """ 60 | assert(len(seq) == len(bounds)) 61 | return [float(x) * float(b[1] - b[0]) + b[0] 62 | for x, b in zip(seq, bounds)] 63 | 64 | 65 | # python version-independent metaclass usage 66 | SolverBase = abc.ABCMeta('SolverBase', (object, ), {}) 67 | 68 | class Solver(SolverBase): 69 | """Base class of all Optunity solvers. 70 | """ 71 | 72 | @abc.abstractmethod 73 | def optimize(self, f, maximize=True, pmap=map): 74 | """Optimizes ``f``. 75 | 76 | :param f: the objective function 77 | :type f: callable 78 | :param maximize: do we want to maximizes? 79 | :type maximize: boolean 80 | :param pmap: the map() function to use 81 | :type pmap: callable 82 | :returns: 83 | - the arguments which optimize ``f`` 84 | - an optional solver report, can be None 85 | 86 | """ 87 | pass 88 | 89 | def maximize(self, f, pmap=map): 90 | """Maximizes f. 91 | 92 | :param f: the objective function 93 | :type f: callable 94 | :param pmap: the map() function to use 95 | :type pmap: callable 96 | :returns: 97 | - the arguments which optimize ``f`` 98 | - an optional solver report, can be None 99 | 100 | """ 101 | return self.optimize(f, True, pmap=pmap) 102 | 103 | def minimize(self, f, pmap=map): 104 | """Minimizes ``f``. 105 | 106 | :param f: the objective function 107 | :type f: callable 108 | :param pmap: the map() function to use 109 | :type pmap: callable 110 | :returns: 111 | - the arguments which optimize ``f`` 112 | - an optional solver report, can be None 113 | 114 | """ 115 | return self.optimize(f, False, pmap=pmap) 116 | 117 | 118 | # http://stackoverflow.com/a/13743316 119 | def _copydoc(fromfunc, sep="\n"): 120 | """ 121 | Decorator: Copy the docstring of `fromfunc` 122 | """ 123 | def _decorator(func): 124 | sourcedoc = fromfunc.__doc__ 125 | if func.__doc__ == None: 126 | func.__doc__ = sourcedoc 127 | else: 128 | func.__doc__ = sep.join([sourcedoc, func.__doc__]) 129 | return func 130 | return _decorator 131 | 132 | def shrink_bounds(bounds, coverage=0.99): 133 | """Shrinks the bounds. The new bounds will cover the fraction ``coverage``. 134 | 135 | >>> [round(x, 3) for x in shrink_bounds([0, 1], coverage=0.99)] 136 | [0.005, 0.995] 137 | 138 | """ 139 | def shrink(lb, ub, coverage): 140 | new_range = float(ub - lb) * coverage / 2 141 | middle = float(ub + lb) / 2 142 | return [middle-new_range, middle+new_range] 143 | 144 | return dict([(k, shrink(v[0], v[1], coverage)) 145 | for k, v in bounds.items()]) 146 | 147 | 148 | def score(value): 149 | """General wrapper around objective function evaluations to get the score. 150 | 151 | :param value: output of the objective function 152 | :returns: the score 153 | 154 | If value is a scalar, it is returned immediately. If value is iterable, its first element is returned. 155 | """ 156 | try: 157 | return value[0] 158 | except (TypeError, IndexError): 159 | return value 160 | 161 | 162 | class ThreadSafeQueue(object): 163 | def __init__(self, lst=None): 164 | """ 165 | Initializes a new object. 166 | 167 | :param lst: initial content 168 | :type lst: list or None 169 | """ 170 | if lst: self._content = lst 171 | else: self._content = [] 172 | self._lock = threading.Lock() 173 | 174 | @property 175 | def lock(self): return self._lock 176 | 177 | @property 178 | def content(self): return self._content 179 | 180 | def append(self, value): 181 | """ 182 | Acquires lock and appends value to the content. 183 | 184 | >>> q1 = ThreadSafeQueue() 185 | >>> q1 186 | [] 187 | >>> q1.append(1) 188 | [1] 189 | 190 | """ 191 | with self.lock: self.content.append(value) 192 | 193 | def __iter__(self): 194 | for i in self.content: 195 | yield i 196 | 197 | def __len__(self): return len(self.content) 198 | def __getitem__(self, idx): return self.content[idx] 199 | def __repr__(self): return str(self.content) 200 | 201 | def copy(self): 202 | """ 203 | Makes a deep copy of this ThreadSafeQueue. 204 | 205 | >>> q1 = ThreadSafeQueue([1,2,3]) 206 | >>> q2 = q1.copy() 207 | >>> q2.append(4) 208 | >>> q1 209 | [1, 2, 3] 210 | >>> q2 211 | [1, 2, 3, 4] 212 | 213 | """ 214 | return ThreadSafeQueue(self.content[:]) 215 | 216 | -------------------------------------------------------------------------------- /optunity/tests/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lining0806/ridgecvtest/26bf940eda7c149ba1deff803ff71b4014cfdbba/optunity/tests/__init__.py -------------------------------------------------------------------------------- /optunity/tests/test_all.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import unittest 4 | import doctest 5 | 6 | modules = ['cross_validation', 'functions', 'solvers', 'communication', 7 | 'solvers.GridSearch', 'solvers.RandomSearch', 'solvers.ParticleSwarm', 8 | 'solvers.CMAES', 'solvers.NelderMead'] 9 | 10 | def load_tests(loader, tests, ignore): 11 | for mod in modules: 12 | tests.addTests(doctest.DocTestSuite("optunity." + mod)) 13 | return tests 14 | 15 | if __name__ == '__main__': 16 | unittest.main() 17 | -------------------------------------------------------------------------------- /optunity/tests/test_solvers.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | # A simple smoke test for all available solvers. 4 | 5 | import optunity 6 | 7 | def f(x, y): 8 | return x + y 9 | 10 | solvers = optunity.available_solvers() 11 | 12 | for solver in solvers: 13 | # simple API 14 | opt, _, _ = optunity.maximize(f, 100, 15 | x=[0, 5], y=[-5, 5], 16 | solver_name=solver) 17 | 18 | # expert API 19 | suggestion = optunity.suggest_solver(num_evals=100, x=[0, 5], y=[-5, 5], 20 | solver_name=solver) 21 | s = optunity.make_solver(**suggestion) 22 | # without parallel evaluations 23 | opt, _ = optunity.optimize(s, f) 24 | # with parallel evaluations 25 | opt, _ = optunity.optimize(s, f, pmap=optunity.pmap) 26 | -------------------------------------------------------------------------------- /optunity/util.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | 3 | # Author: Marc Claesen 4 | # 5 | # Copyright (c) 2014 KU Leuven, ESAT-STADIUS 6 | # All rights reserved. 7 | # 8 | # Redistribution and use in source and binary forms, with or without 9 | # modification, are permitted provided that the following conditions 10 | # are met: 11 | # 12 | # 1. Redistributions of source code must retain the above copyright 13 | # notice, this list of conditions and the following disclaimer. 14 | # 15 | # 2. Redistributions in binary form must reproduce the above copyright 16 | # notice, this list of conditions and the following disclaimer in the 17 | # documentation and/or other materials provided with the distribution. 18 | # 19 | # 3. Neither name of copyright holders nor the names of its contributors 20 | # may be used to endorse or promote products derived from this software 21 | # without specific prior written permission. 22 | # 23 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 24 | # ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 25 | # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 26 | # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR 27 | # CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, 28 | # EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, 29 | # PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR 30 | # PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF 31 | # LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 32 | # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 33 | # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 34 | 35 | import collections 36 | import itertools 37 | import inspect 38 | 39 | def nth(iterable, n): 40 | """Returns the nth item from iterable.""" 41 | try: 42 | return iterable[n] 43 | except TypeError: 44 | try: 45 | return next(itertools.islice(iterable, n, None)) 46 | except StopIteration: 47 | raise IndexError('index out of range') 48 | 49 | 50 | def DocumentedNamedTuple(docstring, *ntargs): 51 | """Factory function to construct collections.namedtuple with 52 | a docstring. Useful to attach meta-information to data structures. 53 | 54 | Inspired by http://stackoverflow.com/a/1606478""" 55 | nt = collections.namedtuple(*ntargs) 56 | 57 | class NT(nt): 58 | __doc__ = docstring 59 | return NT 60 | 61 | 62 | def get_default_args(func): 63 | """ 64 | returns a dictionary of arg_name:default_values for the input function 65 | """ 66 | if not callable(func): 67 | raise TypeError("%s is not callable" % type(func)) 68 | if inspect.isfunction(func): 69 | print('a') 70 | spec = inspect.getargspec(func) 71 | elif hasattr(func, 'im_func'): 72 | print('b') 73 | spec = inspect.getargspec(func.im_func) 74 | print(spec) 75 | elif inspect.isclass(func): 76 | print('c') 77 | return get_default_args(func.__init__) 78 | elif isinstance(func, object): 79 | print('d') 80 | # We already know the instance is callable, 81 | # so it must have a __call__ method defined. 82 | # Return the arguments it expects. 83 | return get_default_args(func.__call__) 84 | 85 | try: 86 | return dict(zip(reversed(spec.args), reversed(spec.defaults))) 87 | except TypeError: 88 | # list of defaults and/or args are empty 89 | return {} 90 | 91 | 92 | # taken from http://kbyanc.blogspot.be/2007/07/python-more-generic-getargspec.html 93 | def getargspec(obj): 94 | """Get the names and default values of a callable's 95 | arguments 96 | 97 | A tuple of four things is returned: (args, varargs, 98 | varkw, defaults). 99 | - args is a list of the argument names (it may 100 | contain nested lists). 101 | - varargs and varkw are the names of the * and 102 | ** arguments or None. 103 | - defaults is a tuple of default argument values 104 | or None if there are no default arguments; if 105 | this tuple has n elements, they correspond to 106 | the last n elements listed in args. 107 | 108 | Unlike inspect.getargspec(), can return argument 109 | specification for functions, methods, callable 110 | objects, and classes. Does not support builtin 111 | functions or methods. 112 | """ 113 | if not callable(obj): 114 | raise TypeError("%s is not callable" % type(obj)) 115 | try: 116 | if inspect.isfunction(obj): 117 | return inspect.getargspec(obj) 118 | elif hasattr(obj, 'im_func'): 119 | # For methods or classmethods drop the first 120 | # argument from the returned list because 121 | # python supplies that automatically for us. 122 | # Note that this differs from what 123 | # inspect.getargspec() returns for methods. 124 | # NB: We use im_func so we work with 125 | # instancemethod objects also. 126 | spec = list(inspect.getargspec(obj.im_func)) 127 | spec[0] = spec[0][1:] 128 | return spec 129 | elif inspect.isclass(obj): 130 | return getargspec(obj.__init__) 131 | #elif isinstance(obj, object) and \ 132 | # not isinstance(obj, type(arglist.__get__)): # TODO: what does this clause do? 133 | elif isinstance(obj, object): 134 | # We already know the instance is callable, 135 | # so it must have a __call__ method defined. 136 | # Return the arguments it expects. 137 | return getargspec(obj.__call__) 138 | except NotImplementedError: 139 | # If a nested call to our own getargspec() 140 | # raises NotImplementedError, re-raise the 141 | # exception with the real object type to make 142 | # the error message more meaningful (the caller 143 | # only knows what they passed us; they shouldn't 144 | # care what aspect(s) of that object we actually 145 | # examined). 146 | pass 147 | raise NotImplementedError("do not know how to get argument list for %s" % type(obj)) 148 | --------------------------------------------------------------------------------