├── .travis.yml ├── DESCRIPTION ├── NAMESPACE ├── R ├── BayesianOptimization.R ├── KFold.R ├── Utility.R ├── Utility_Max.R ├── Utils.R └── rBayesianOptimization.R ├── README.md ├── appveyor.yml └── man ├── BayesianOptimization.Rd ├── KFold.Rd ├── Matrix_runif.Rd ├── Min_Max_Inverse_Scale_Vec.Rd ├── Min_Max_Scale_Mat.Rd ├── Utility.Rd ├── Utility_Max.Rd └── rBayesianOptimization.Rd /.travis.yml: -------------------------------------------------------------------------------- 1 | # Sample .travis.yml for R projects from https://github.com/craigcitro/r-travis 2 | 3 | language: c 4 | sudo: required 5 | 6 | before_install: 7 | - curl -OL http://raw.github.com/craigcitro/r-travis/master/scripts/travis-tool.sh 8 | - chmod 755 ./travis-tool.sh 9 | - ./travis-tool.sh bootstrap 10 | 11 | install: 12 | - ./travis-tool.sh install_deps 13 | 14 | script: ./travis-tool.sh run_tests 15 | 16 | after_failure: 17 | - ./travis-tool.sh dump_logs 18 | 19 | env: 20 | - WARNINGS_ARE_ERRORS=1 21 | 22 | notifications: 23 | email: 24 | on_success: change 25 | on_failure: change 26 | -------------------------------------------------------------------------------- /DESCRIPTION: -------------------------------------------------------------------------------- 1 | Package: rBayesianOptimization 2 | Type: Package 3 | Title: Bayesian Optimization of Hyperparameters 4 | Version: 1.2.1 5 | Authors@R: person("Yachen", "Yan", email = "yanyachen21@gmail.com", role = c("aut", "cre")) 6 | Description: A Pure R implementation of Bayesian Global Optimization with Gaussian Processes. 7 | URL: https://github.com/yanyachen/rBayesianOptimization 8 | BugReports: https://github.com/yanyachen/rBayesianOptimization/issues 9 | Depends: R (>= 2.10) 10 | Imports: stats, utils, data.table (>= 1.9.6), magrittr, foreach, GPfit 11 | Suggests: xgboost 12 | License: GPL-2 13 | RoxygenNote: 7.3.1 14 | NeedsCompilation: no 15 | Packaged: 2024-04-01 02:06:00 UTC; Administrator 16 | Author: Yachen Yan [aut, cre] 17 | Maintainer: Yachen Yan 18 | -------------------------------------------------------------------------------- /NAMESPACE: -------------------------------------------------------------------------------- 1 | # Generated by roxygen2: do not edit by hand 2 | 3 | export(BayesianOptimization) 4 | export(KFold) 5 | export(Matrix_runif) 6 | export(Min_Max_Inverse_Scale_Vec) 7 | export(Min_Max_Scale_Mat) 8 | export(Utility) 9 | export(Utility_Max) 10 | importFrom(data.table,":=") 11 | importFrom(data.table,data.table) 12 | importFrom(data.table,set) 13 | importFrom(data.table,setDT) 14 | importFrom(data.table,setnames) 15 | importFrom(foreach,"%do%") 16 | importFrom(foreach,foreach) 17 | importFrom(magrittr,"%>%") 18 | importFrom(magrittr,"%T>%") 19 | importFrom(magrittr,extract) 20 | importFrom(magrittr,extract2) 21 | importFrom(magrittr,inset) 22 | importFrom(stats,dnorm) 23 | importFrom(stats,na.omit) 24 | importFrom(stats,optim) 25 | importFrom(stats,pnorm) 26 | importFrom(stats,runif) 27 | -------------------------------------------------------------------------------- /R/BayesianOptimization.R: -------------------------------------------------------------------------------- 1 | #' @title Bayesian Optimization 2 | #' 3 | #' @description 4 | #' Bayesian Optimization of Hyperparameters. 5 | #' 6 | #' @param FUN The function to be maximized. This Function should return a named list with 2 components. 7 | #' The first component "Score" should be the metrics to be maximized, and the second component "Pred" should be 8 | #' the validation/cross-validation prediction for ensembling/stacking. 9 | #' @param bounds A named list of lower and upper bounds for each hyperparameter. 10 | #' The names of the list should be identical to the arguments of FUN. 11 | #' All the sample points in init_grid_dt should be in the range of bounds. 12 | #' Please use "L" suffix to indicate integer hyperparameter. 13 | #' @param init_grid_dt User specified points to sample the target function, should 14 | #' be a \code{data.frame} or \code{data.table} with identical column names as bounds. 15 | #' User can add one "Value" column at the end, if target function is pre-sampled. 16 | #' @param init_points Number of randomly chosen points to sample the 17 | #' target function before Bayesian Optimization fitting the Gaussian Process. 18 | #' @param n_iter Total number of times the Bayesian Optimization is to repeated. 19 | #' @param acq Acquisition function type to be used. Can be "ucb", "ei" or "poi". 20 | #' \itemize{ 21 | #' \item \code{ucb} GP Upper Confidence Bound 22 | #' \item \code{ei} Expected Improvement 23 | #' \item \code{poi} Probability of Improvement 24 | #' } 25 | #' @param kappa tunable parameter kappa of GP Upper Confidence Bound, to balance exploitation against exploration, 26 | #' increasing kappa will make the optimized hyperparameters pursuing exploration. 27 | #' @param eps tunable parameter epsilon of Expected Improvement and Probability of Improvement, to balance exploitation against exploration, 28 | #' increasing epsilon will make the optimized hyperparameters are more spread out across the whole range. 29 | #' @param kernel Kernel (aka correlation function) for the underlying Gaussian Process. This parameter should be a list 30 | #' that specifies the type of correlation function along with the smoothness parameter. Popular choices are square exponential (default) or matern 5/2 31 | #' @param verbose Whether or not to print progress. 32 | #' @param ... Other arguments passed on to \code{\link{GP_fit}}. 33 | #' @return a list of Bayesian Optimization result is returned: 34 | #' \itemize{ 35 | #' \item \code{Best_Par} a named vector of the best hyperparameter set found 36 | #' \item \code{Best_Value} the value of metrics achieved by the best hyperparameter set 37 | #' \item \code{History} a \code{data.table} of the bayesian optimization history 38 | #' \item \code{Pred} a \code{data.table} with validation/cross-validation prediction for each round of bayesian optimization history 39 | #' } 40 | #' @references Jasper Snoek, Hugo Larochelle, Ryan P. Adams (2012) \emph{Practical Bayesian Optimization of Machine Learning Algorithms} 41 | #' @examples 42 | #' # Example 1: Optimization 43 | #' ## Set Pred = 0, as placeholder 44 | #' Test_Fun <- function(x) { 45 | #' list(Score = exp(-(x - 2)^2) + exp(-(x - 6)^2/10) + 1/ (x^2 + 1), 46 | #' Pred = 0) 47 | #' } 48 | #' ## Set larger init_points and n_iter for better optimization result 49 | #' OPT_Res <- BayesianOptimization(Test_Fun, 50 | #' bounds = list(x = c(1, 3)), 51 | #' init_points = 2, n_iter = 1, 52 | #' acq = "ucb", kappa = 2.576, eps = 0.0, 53 | #' verbose = TRUE) 54 | #' \dontrun{ 55 | #' # Example 2: Parameter Tuning 56 | #' library(xgboost) 57 | #' data(agaricus.train, package = "xgboost") 58 | #' dtrain <- xgb.DMatrix(agaricus.train$data, 59 | #' label = agaricus.train$label) 60 | #' cv_folds <- KFold(agaricus.train$label, nfolds = 5, 61 | #' stratified = TRUE, seed = 0) 62 | #' xgb_cv_bayes <- function(max_depth, min_child_weight, subsample) { 63 | #' cv <- xgb.cv(params = list(booster = "gbtree", eta = 0.01, 64 | #' max_depth = max_depth, 65 | #' min_child_weight = min_child_weight, 66 | #' subsample = subsample, colsample_bytree = 0.3, 67 | #' lambda = 1, alpha = 0, 68 | #' objective = "binary:logistic", 69 | #' eval_metric = "auc"), 70 | #' data = dtrain, nround = 100, 71 | #' folds = cv_folds, prediction = TRUE, showsd = TRUE, 72 | #' early_stopping_rounds = 5, maximize = TRUE, verbose = 0) 73 | #' list(Score = cv$evaluation_log$test_auc_mean[cv$best_iteration], 74 | #' Pred = cv$pred) 75 | #' } 76 | #' OPT_Res <- BayesianOptimization(xgb_cv_bayes, 77 | #' bounds = list(max_depth = c(2L, 6L), 78 | #' min_child_weight = c(1L, 10L), 79 | #' subsample = c(0.5, 0.8)), 80 | #' init_grid_dt = NULL, init_points = 10, n_iter = 20, 81 | #' acq = "ucb", kappa = 2.576, eps = 0.0, 82 | #' verbose = TRUE) 83 | #' } 84 | #' @importFrom magrittr %>% %T>% extract extract2 inset 85 | #' @importFrom data.table data.table setnames set setDT := 86 | #' @export 87 | 88 | BayesianOptimization <- function(FUN, bounds, init_grid_dt = NULL, init_points = 0, n_iter, acq = "ucb", kappa = 2.576, eps = 0.0, kernel = list(type = "exponential", power = 2), verbose = TRUE, ...) { 89 | # Preparation 90 | ## DT_bounds 91 | DT_bounds <- data.table(Parameter = names(bounds), 92 | Lower = sapply(bounds, extract2, 1), 93 | Upper = sapply(bounds, extract2, 2), 94 | Type = sapply(bounds, class)) 95 | ## init_grid_dt 96 | setDT(init_grid_dt) 97 | if (nrow(init_grid_dt) != 0) { 98 | if (identical(names(init_grid_dt), DT_bounds[, Parameter]) == TRUE) { 99 | init_grid_dt[, Value := -Inf] 100 | } else if (identical(names(init_grid_dt), c(DT_bounds[, Parameter], "Value")) == TRUE) { 101 | paste(nrow(init_grid_dt), "points in hyperparameter space were pre-sampled\n", sep = " ") %>% 102 | cat(.) 103 | } else { 104 | stop("bounds and init_grid_dt should be compatible") 105 | } 106 | } 107 | ## init_points_dt 108 | init_points_dt <- Matrix_runif(n = init_points, lower = DT_bounds[, Lower], upper = DT_bounds[, Upper]) %>% 109 | data.table(.) %T>% 110 | setnames(., old = names(.), new = DT_bounds[, Parameter]) %T>% { 111 | if (any(DT_bounds[, Type] == "integer")) { 112 | set(., 113 | j = DT_bounds[Type == "integer", Parameter], 114 | value = round(extract(., j = DT_bounds[Type == "integer", Parameter], with = FALSE))) 115 | } else { 116 | . 117 | } 118 | } %T>% 119 | extract(., j = Value:=-Inf) 120 | ## iter_points_dt 121 | iter_points_dt <- data.table(matrix(-Inf, nrow = n_iter, ncol = nrow(DT_bounds) + 1)) %>% 122 | setnames(., old = names(.), new = c(DT_bounds[, Parameter], "Value")) 123 | ## DT_history 124 | DT_history <- rbind(init_grid_dt, init_points_dt, iter_points_dt) %>% 125 | cbind(data.table(Round = 1:nrow(.)), .) 126 | ## Pred_list 127 | Pred_list <- vector(mode = "list", length = nrow(DT_history)) 128 | # Initialization 129 | for (i in 1:(nrow(init_grid_dt) + nrow(init_points_dt))) { 130 | if (is.infinite(DT_history[i, Value]) == TRUE) { 131 | This_Par <- DT_history[i, DT_bounds[, Parameter], with = FALSE] 132 | } else { 133 | next 134 | } 135 | # Function Evaluation 136 | This_Log <- utils::capture.output({ 137 | This_Time <- system.time({ 138 | This_Score_Pred <- do.call(what = FUN, args = as.list(This_Par)) 139 | }) 140 | }) 141 | # Saving History and Prediction 142 | data.table::set(DT_history, 143 | i = as.integer(i), 144 | j = "Value", 145 | value = as.list(c(This_Score_Pred$Score))) 146 | Pred_list[[i]] <- This_Score_Pred$Pred 147 | # Printing History 148 | if (verbose == TRUE) { 149 | paste(c("elapsed", names(DT_history)), 150 | c(format(This_Time["elapsed"], trim = FALSE, digits = NULL, nsmall = 2), 151 | format(DT_history[i, "Round", with = FALSE], trim = FALSE, digits = NULL, nsmall = 0), 152 | format(DT_history[i, -"Round", with = FALSE], trim = FALSE, digits = NULL, nsmall = 4)), 153 | sep = " = ", collapse = "\t") %>% 154 | cat(., "\n") 155 | } 156 | } 157 | # Optimization 158 | for (j in (nrow(init_grid_dt) + nrow(init_points_dt) + 1):nrow(DT_history)) { 159 | if (nrow(iter_points_dt) == 0) { 160 | next 161 | } 162 | # Fitting Gaussian Process 163 | Par_Mat <- Min_Max_Scale_Mat(as.matrix(DT_history[1:(j - 1), DT_bounds[, Parameter], with = FALSE]), 164 | lower = DT_bounds[, Lower], 165 | upper = DT_bounds[, Upper]) 166 | Rounds_Unique <- setdiff(1:(j - 1), which(duplicated(Par_Mat) == TRUE)) 167 | Value_Vec <- DT_history[1:(j - 1), Value] 168 | GP_Log <- utils::capture.output({ 169 | GP <- GPfit::GP_fit(X = Par_Mat[Rounds_Unique, ], 170 | Y = Value_Vec[Rounds_Unique], 171 | corr = kernel, ...) 172 | }) 173 | # Minimizing Negative Utility Function 174 | Next_Par <- Utility_Max(DT_bounds, GP, acq = acq, y_max = max(DT_history[, Value]), kappa = kappa, eps = eps) %>% 175 | Min_Max_Inverse_Scale_Vec(., lower = DT_bounds[, Lower], upper = DT_bounds[, Upper]) %>% 176 | magrittr::set_names(., DT_bounds[, Parameter]) %>% 177 | inset(., DT_bounds[Type == "integer", Parameter], round(extract(., DT_bounds[Type == "integer", Parameter]))) 178 | # Function Evaluation 179 | Next_Log <- utils::capture.output({ 180 | Next_Time <- system.time({ 181 | Next_Score_Pred <- do.call(what = FUN, args = as.list(Next_Par)) 182 | }) 183 | }) 184 | # Saving History and Prediction 185 | data.table::set(DT_history, 186 | i = as.integer(j), 187 | j = c(DT_bounds[, Parameter], "Value"), 188 | value = as.list(c(Next_Par, Value = Next_Score_Pred$Score))) 189 | Pred_list[[j]] <- Next_Score_Pred$Pred 190 | # Printing History 191 | if (verbose == TRUE) { 192 | paste(c("elapsed", names(DT_history)), 193 | c(format(Next_Time["elapsed"], trim = FALSE, digits = NULL, nsmall = 2), 194 | format(DT_history[j, "Round", with = FALSE], trim = FALSE, digits = NULL, nsmall = 0), 195 | format(DT_history[j, -"Round", with = FALSE], trim = FALSE, digits = NULL, nsmall = 4)), sep = " = ", collapse = "\t") %>% 196 | cat(., "\n") 197 | } 198 | } 199 | # Computing Result 200 | Best_Par <- as.numeric(DT_history[which.max(Value), DT_bounds[, Parameter], with = FALSE]) %>% 201 | magrittr::set_names(., DT_bounds[, Parameter]) 202 | Best_Value <- max(DT_history[, Value], na.rm = TRUE) 203 | Pred_DT <- data.table::as.data.table(Pred_list) 204 | Result <- list(Best_Par = Best_Par, 205 | Best_Value = Best_Value, 206 | History = DT_history, 207 | Pred = Pred_DT) 208 | # Printing Best 209 | cat("\n Best Parameters Found: \n") 210 | paste(names(DT_history), 211 | c(format(DT_history[which.max(Value), "Round", with = FALSE], trim = FALSE, digits = NULL, nsmall = 0), 212 | format(DT_history[which.max(Value), -"Round", with = FALSE], trim = FALSE, digits = NULL, nsmall = 4)), 213 | sep = " = ", collapse = "\t") %>% 214 | cat(., "\n") 215 | # Return 216 | return(Result) 217 | } 218 | utils::globalVariables(c("Type", "Value")) 219 | -------------------------------------------------------------------------------- /R/KFold.R: -------------------------------------------------------------------------------- 1 | #' @title K-Folds cross validation index generator 2 | #' 3 | #' @description 4 | #' Generates a list of indices for K-Folds Cross-Validation. 5 | #' 6 | #' @param target Samples to split in K folds. 7 | #' @param nfolds Number of folds. 8 | #' @param stratified whether to apply Stratified KFold. 9 | #' @param seed random seed to be used. 10 | #' @return a list of indices for K-Folds Cross-Validation 11 | #' @importFrom stats na.omit 12 | #' @export 13 | 14 | KFold <- function(target, nfolds = 10, stratified = FALSE, seed = 0) { 15 | # Function for generating index for each fold 16 | nfold_index <- function(index, nfolds, seed) { 17 | NA_how_many <- ceiling(length(index) / nfolds) * nfolds - length(index) 18 | set.seed(seed) 19 | index_mat <- matrix(c(sample(index), rep(NA, NA_how_many)), 20 | ncol = nfolds) 21 | index_list <- list() 22 | for(i in 1:nfolds) index_list[[i]] <- as.integer(na.omit(index_mat[,i])) 23 | return(index_list) 24 | } 25 | # Function for concatenate corresponding vector in the list 26 | list_c <- function(list1, list2) { 27 | stopifnot(length(list1) == length(list2)) 28 | list12 <- list() 29 | for(i in seq_along(list1)) list12[[i]] <- c(list1[[i]], list2[[i]]) 30 | return(list12) 31 | } 32 | # CV-Folds Generating 33 | if (stratified == FALSE) { 34 | index_nfold_list <- nfold_index(index = 1:length(target), nfolds = nfolds, seed = seed) 35 | } else { 36 | target_table <- table(target) 37 | target_list <- list() 38 | for(i in seq_along(names(target_table))) target_list[[i]] <- which(as.character(target) == names(target_table)[i]) 39 | index_list_of_each_target <- lapply(target_list, nfold_index, nfolds = nfolds, seed = seed) 40 | index_nfold_list <- Reduce(list_c, index_list_of_each_target) 41 | for (i in seq_along(index_nfold_list)) index_nfold_list[[i]] <- sort(index_nfold_list[[i]]) 42 | } 43 | return(index_nfold_list) 44 | } 45 | -------------------------------------------------------------------------------- /R/Utility.R: -------------------------------------------------------------------------------- 1 | #' @title Utility Computing Function 2 | #' 3 | #' @description 4 | #' Computing Utility. 5 | #' 6 | #' @param x_vec a vector of scaled hyperparameters 7 | #' @param GP an object of class GP 8 | #' @param acq Acquisition function type to be used 9 | #' @param y_max The current maximum known value of the target utility function 10 | #' @param kappa tunable parameter kappa to balance exploitation against exploration 11 | #' @param eps tunable parameter epsilon to balance exploitation against exploration 12 | #' @return negative utility to be minimized 13 | #' @importFrom stats pnorm dnorm 14 | #' @importFrom magrittr %>% 15 | #' @keywords internal 16 | #' @export 17 | 18 | Utility <- function(x_vec, GP, acq = "ucb", y_max, kappa, eps) { 19 | # Gaussian Process Prediction 20 | GP_Pred <- GPfit::predict.GP(object = GP, xnew = matrix(x_vec, nrow = 1)) 21 | GP_Mean <- GP_Pred$Y_hat 22 | GP_MSE <- GP_Pred$MSE %>% pmax(., 1e-9) 23 | # Utility Function Type 24 | if (acq == "ucb") { 25 | Utility <- GP_Mean + kappa * sqrt(GP_MSE) 26 | } else if (acq == "ei") { 27 | z <- (GP_Mean - y_max - eps) / sqrt(GP_MSE) 28 | Utility <- (GP_Mean - y_max - eps) * pnorm(z) + sqrt(GP_MSE) * dnorm(z) 29 | } else if (acq == "poi") { 30 | z <- (GP_Mean - y_max - eps) / sqrt(GP_MSE) 31 | Utility <- pnorm(z) 32 | } 33 | return(-Utility) 34 | } 35 | -------------------------------------------------------------------------------- /R/Utility_Max.R: -------------------------------------------------------------------------------- 1 | #' @title Utility Maximization Function 2 | #' 3 | #' @description 4 | #' Utility Maximization. 5 | #' 6 | #' @param DT_bounds hyperparameters lower and upper bounds to limit the search of the acq max 7 | #' @param GP an object of class GP 8 | #' @param acq Acquisition function type to be used 9 | #' @param y_max The current maximum known value of the target utility function 10 | #' @param kappa tunable parameter kappa to balance exploitation against exploration 11 | #' @param eps tunable parameter epsilon to balance exploitation against exploration 12 | #' @return The arg max of the acquisition function 13 | #' @importFrom stats optim 14 | #' @importFrom data.table data.table setnames 15 | #' @importFrom foreach foreach %do% 16 | #' @importFrom magrittr %>% 17 | #' @keywords internal 18 | #' @export 19 | 20 | Utility_Max <- function(DT_bounds, GP, acq = "ucb", y_max, kappa, eps) { 21 | # Try Different Initial Values 22 | Mat_tries <- Matrix_runif(100, 23 | lower = rep(0, length(DT_bounds[, Lower])), 24 | upper = rep(1, length(DT_bounds[, Upper]))) 25 | # Negative Utility Function Minimization 26 | Mat_optim <- foreach(i = 1:nrow(Mat_tries), .combine = "rbind") %do% { 27 | optim_result <- optim(par = Mat_tries[i,], 28 | fn = Utility, 29 | GP = GP, acq = acq, y_max = y_max, kappa = kappa, eps = eps, 30 | method = "L-BFGS-B", 31 | lower = rep(0, length(DT_bounds[, Lower])), 32 | upper = rep(1, length(DT_bounds[, Upper])), 33 | control = list(maxit = 100, 34 | factr = 5e11)) 35 | c(optim_result$par, optim_result$value) 36 | } %>% 37 | data.table(.) %>% 38 | setnames(., old = names(.), new = c(DT_bounds[, Parameter], "Negetive_Utility")) 39 | # Return Best set of Parameters 40 | argmax <- as.numeric(Mat_optim[which.min(Negetive_Utility), DT_bounds[, Parameter], with = FALSE]) 41 | return(argmax) 42 | } 43 | utils::globalVariables(c("Lower", "Upper", "Parameter", "Negetive_Utility")) 44 | -------------------------------------------------------------------------------- /R/Utils.R: -------------------------------------------------------------------------------- 1 | #' @title Matrix runif 2 | #' 3 | #' @description 4 | #' Generates random deviates for multiple hyperparameters in matrix format. 5 | #' 6 | #' @param n number of observations 7 | #' @param lower lower bounds 8 | #' @param upper upper bounds 9 | #' @return a matrix of original hyperparameters 10 | #' @importFrom stats runif 11 | #' @importFrom foreach foreach %do% 12 | #' @importFrom magrittr %>% 13 | #' @keywords internal 14 | #' @export 15 | 16 | Matrix_runif <- function(n, lower, upper) { 17 | foreach(i = seq_along(lower), .combine = "cbind") %do% { 18 | runif(n, min = lower[i], max = upper[i]) %>% 19 | pmin(., upper[i] - sqrt(.Machine$double.eps)) %>% 20 | pmax(., lower[i] + sqrt(.Machine$double.eps)) 21 | } %>% 22 | matrix(., nrow = n, ncol = length(lower)) 23 | } 24 | utils::globalVariables(c("i", ".")) 25 | 26 | 27 | #' @title Matrix MinMax Scaling 28 | #' 29 | #' @description 30 | #' Transforms hyperparameters by scaling each hyperparameter to a given range. 31 | #' 32 | #' @param mat a matrix of original hyperparameters 33 | #' @param lower lower bounds 34 | #' @param upper upper bounds 35 | #' @return a matrix of scaled hyperparameters 36 | #' @importFrom magrittr %>% 37 | #' @keywords internal 38 | #' @export 39 | 40 | Min_Max_Scale_Mat <- function(mat, lower, upper) { 41 | t((t(mat) - lower) / (upper - lower)) %>% 42 | pmin(., 1 - .Machine$double.eps) %>% 43 | pmax(., 0 + .Machine$double.eps) 44 | } 45 | utils::globalVariables(".") 46 | 47 | 48 | #' @title MinMax Inverse Scaling 49 | #' 50 | #' @description 51 | #' Transforms scaled hyperparameters to original range. 52 | #' 53 | #' @param vec a vector of scaled hyperparameters 54 | #' @param lower lower bounds 55 | #' @param upper upper bounds 56 | #' @return a vector of original hyperparameters 57 | #' @importFrom magrittr %>% 58 | #' @keywords internal 59 | #' @export 60 | 61 | Min_Max_Inverse_Scale_Vec <- function(vec, lower, upper) { 62 | (vec * (upper - lower) + lower) %>% 63 | pmin(., upper - .Machine$double.eps) %>% 64 | pmax(., lower + .Machine$double.eps) 65 | } 66 | -------------------------------------------------------------------------------- /R/rBayesianOptimization.R: -------------------------------------------------------------------------------- 1 | #' rBayesianOptimization: Bayesian Optimization of Hyperparameters 2 | #' 3 | #' A Pure R implementation of bayesian global optimization with gaussian processes. 4 | #' 5 | #' @docType package 6 | #' @name rBayesianOptimization 7 | NULL 8 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # rBayesianOptimization 2 | 3 | [![CRAN_Status_Badge](https://www.r-pkg.org/badges/version/rBayesianOptimization)](https://CRAN.R-project.org/package=rBayesianOptimization) 4 | [![CRAN Downloads](https://cranlogs.r-pkg.org/badges/rBayesianOptimization)](https://CRAN.R-project.org/package=rBayesianOptimization) 5 | [![CRAN Downloads Total](https://cranlogs.r-pkg.org/badges/grand-total/rBayesianOptimization?color=brightgreen)](https://CRAN.R-project.org/package=rBayesianOptimization) 6 | 7 | Bayesian Optimization of Hyper-parameters 8 | 9 | A Pure R implementation of Bayesian Global Optimization with Gaussian Processes. 10 | 11 | To install: 12 | * the stable version from [CRAN](https://CRAN.R-project.org/package=rBayesianOptimization): 13 | ```r 14 | install.packages("rBayesianOptimization") 15 | ``` 16 | 17 | * the latest development version: 18 | ```r 19 | devtools::install_github("yanyachen/rBayesianOptimization") 20 | ``` 21 | -------------------------------------------------------------------------------- /appveyor.yml: -------------------------------------------------------------------------------- 1 | # DO NOT CHANGE the "init" and "install" sections below 2 | 3 | # Download script file from GitHub 4 | init: 5 | ps: | 6 | $ErrorActionPreference = "Stop" 7 | Invoke-WebRequest http://raw.github.com/krlmlr/r-appveyor/master/scripts/appveyor-tool.ps1 -OutFile "..\appveyor-tool.ps1" 8 | Import-Module '..\appveyor-tool.ps1' 9 | 10 | install: 11 | ps: Bootstrap 12 | 13 | # Adapt as necessary starting from here 14 | 15 | build_script: 16 | - travis-tool.sh install_deps 17 | 18 | test_script: 19 | - travis-tool.sh run_tests 20 | 21 | on_failure: 22 | - 7z a failure.zip *.Rcheck\* 23 | - appveyor PushArtifact failure.zip 24 | 25 | artifacts: 26 | - path: '*.Rcheck\**\*.log' 27 | name: Logs 28 | 29 | - path: '*.Rcheck\**\*.out' 30 | name: Logs 31 | 32 | - path: '*.Rcheck\**\*.fail' 33 | name: Logs 34 | 35 | - path: '*.Rcheck\**\*.Rout' 36 | name: Logs 37 | 38 | - path: '\*_*.tar.gz' 39 | name: Bits 40 | 41 | - path: '\*_*.zip' 42 | name: Bits 43 | -------------------------------------------------------------------------------- /man/BayesianOptimization.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/BayesianOptimization.R 3 | \name{BayesianOptimization} 4 | \alias{BayesianOptimization} 5 | \title{Bayesian Optimization} 6 | \usage{ 7 | BayesianOptimization( 8 | FUN, 9 | bounds, 10 | init_grid_dt = NULL, 11 | init_points = 0, 12 | n_iter, 13 | acq = "ucb", 14 | kappa = 2.576, 15 | eps = 0, 16 | kernel = list(type = "exponential", power = 2), 17 | verbose = TRUE, 18 | ... 19 | ) 20 | } 21 | \arguments{ 22 | \item{FUN}{The function to be maximized. This Function should return a named list with 2 components. 23 | The first component "Score" should be the metrics to be maximized, and the second component "Pred" should be 24 | the validation/cross-validation prediction for ensembling/stacking.} 25 | 26 | \item{bounds}{A named list of lower and upper bounds for each hyperparameter. 27 | The names of the list should be identical to the arguments of FUN. 28 | All the sample points in init_grid_dt should be in the range of bounds. 29 | Please use "L" suffix to indicate integer hyperparameter.} 30 | 31 | \item{init_grid_dt}{User specified points to sample the target function, should 32 | be a \code{data.frame} or \code{data.table} with identical column names as bounds. 33 | User can add one "Value" column at the end, if target function is pre-sampled.} 34 | 35 | \item{init_points}{Number of randomly chosen points to sample the 36 | target function before Bayesian Optimization fitting the Gaussian Process.} 37 | 38 | \item{n_iter}{Total number of times the Bayesian Optimization is to repeated.} 39 | 40 | \item{acq}{Acquisition function type to be used. Can be "ucb", "ei" or "poi". 41 | \itemize{ 42 | \item \code{ucb} GP Upper Confidence Bound 43 | \item \code{ei} Expected Improvement 44 | \item \code{poi} Probability of Improvement 45 | }} 46 | 47 | \item{kappa}{tunable parameter kappa of GP Upper Confidence Bound, to balance exploitation against exploration, 48 | increasing kappa will make the optimized hyperparameters pursuing exploration.} 49 | 50 | \item{eps}{tunable parameter epsilon of Expected Improvement and Probability of Improvement, to balance exploitation against exploration, 51 | increasing epsilon will make the optimized hyperparameters are more spread out across the whole range.} 52 | 53 | \item{kernel}{Kernel (aka correlation function) for the underlying Gaussian Process. This parameter should be a list 54 | that specifies the type of correlation function along with the smoothness parameter. Popular choices are square exponential (default) or matern 5/2} 55 | 56 | \item{verbose}{Whether or not to print progress.} 57 | 58 | \item{...}{Other arguments passed on to \code{\link{GP_fit}}.} 59 | } 60 | \value{ 61 | a list of Bayesian Optimization result is returned: 62 | \itemize{ 63 | \item \code{Best_Par} a named vector of the best hyperparameter set found 64 | \item \code{Best_Value} the value of metrics achieved by the best hyperparameter set 65 | \item \code{History} a \code{data.table} of the bayesian optimization history 66 | \item \code{Pred} a \code{data.table} with validation/cross-validation prediction for each round of bayesian optimization history 67 | } 68 | } 69 | \description{ 70 | Bayesian Optimization of Hyperparameters. 71 | } 72 | \examples{ 73 | # Example 1: Optimization 74 | ## Set Pred = 0, as placeholder 75 | Test_Fun <- function(x) { 76 | list(Score = exp(-(x - 2)^2) + exp(-(x - 6)^2/10) + 1/ (x^2 + 1), 77 | Pred = 0) 78 | } 79 | ## Set larger init_points and n_iter for better optimization result 80 | OPT_Res <- BayesianOptimization(Test_Fun, 81 | bounds = list(x = c(1, 3)), 82 | init_points = 2, n_iter = 1, 83 | acq = "ucb", kappa = 2.576, eps = 0.0, 84 | verbose = TRUE) 85 | \dontrun{ 86 | # Example 2: Parameter Tuning 87 | library(xgboost) 88 | data(agaricus.train, package = "xgboost") 89 | dtrain <- xgb.DMatrix(agaricus.train$data, 90 | label = agaricus.train$label) 91 | cv_folds <- KFold(agaricus.train$label, nfolds = 5, 92 | stratified = TRUE, seed = 0) 93 | xgb_cv_bayes <- function(max_depth, min_child_weight, subsample) { 94 | cv <- xgb.cv(params = list(booster = "gbtree", eta = 0.01, 95 | max_depth = max_depth, 96 | min_child_weight = min_child_weight, 97 | subsample = subsample, colsample_bytree = 0.3, 98 | lambda = 1, alpha = 0, 99 | objective = "binary:logistic", 100 | eval_metric = "auc"), 101 | data = dtrain, nround = 100, 102 | folds = cv_folds, prediction = TRUE, showsd = TRUE, 103 | early_stopping_rounds = 5, maximize = TRUE, verbose = 0) 104 | list(Score = cv$evaluation_log$test_auc_mean[cv$best_iteration], 105 | Pred = cv$pred) 106 | } 107 | OPT_Res <- BayesianOptimization(xgb_cv_bayes, 108 | bounds = list(max_depth = c(2L, 6L), 109 | min_child_weight = c(1L, 10L), 110 | subsample = c(0.5, 0.8)), 111 | init_grid_dt = NULL, init_points = 10, n_iter = 20, 112 | acq = "ucb", kappa = 2.576, eps = 0.0, 113 | verbose = TRUE) 114 | } 115 | } 116 | \references{ 117 | Jasper Snoek, Hugo Larochelle, Ryan P. Adams (2012) \emph{Practical Bayesian Optimization of Machine Learning Algorithms} 118 | } 119 | -------------------------------------------------------------------------------- /man/KFold.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/KFold.R 3 | \name{KFold} 4 | \alias{KFold} 5 | \title{K-Folds cross validation index generator} 6 | \usage{ 7 | KFold(target, nfolds = 10, stratified = FALSE, seed = 0) 8 | } 9 | \arguments{ 10 | \item{target}{Samples to split in K folds.} 11 | 12 | \item{nfolds}{Number of folds.} 13 | 14 | \item{stratified}{whether to apply Stratified KFold.} 15 | 16 | \item{seed}{random seed to be used.} 17 | } 18 | \value{ 19 | a list of indices for K-Folds Cross-Validation 20 | } 21 | \description{ 22 | Generates a list of indices for K-Folds Cross-Validation. 23 | } 24 | -------------------------------------------------------------------------------- /man/Matrix_runif.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/Utils.R 3 | \name{Matrix_runif} 4 | \alias{Matrix_runif} 5 | \title{Matrix runif} 6 | \usage{ 7 | Matrix_runif(n, lower, upper) 8 | } 9 | \arguments{ 10 | \item{n}{number of observations} 11 | 12 | \item{lower}{lower bounds} 13 | 14 | \item{upper}{upper bounds} 15 | } 16 | \value{ 17 | a matrix of original hyperparameters 18 | } 19 | \description{ 20 | Generates random deviates for multiple hyperparameters in matrix format. 21 | } 22 | \keyword{internal} 23 | -------------------------------------------------------------------------------- /man/Min_Max_Inverse_Scale_Vec.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/Utils.R 3 | \name{Min_Max_Inverse_Scale_Vec} 4 | \alias{Min_Max_Inverse_Scale_Vec} 5 | \title{MinMax Inverse Scaling} 6 | \usage{ 7 | Min_Max_Inverse_Scale_Vec(vec, lower, upper) 8 | } 9 | \arguments{ 10 | \item{vec}{a vector of scaled hyperparameters} 11 | 12 | \item{lower}{lower bounds} 13 | 14 | \item{upper}{upper bounds} 15 | } 16 | \value{ 17 | a vector of original hyperparameters 18 | } 19 | \description{ 20 | Transforms scaled hyperparameters to original range. 21 | } 22 | \keyword{internal} 23 | -------------------------------------------------------------------------------- /man/Min_Max_Scale_Mat.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/Utils.R 3 | \name{Min_Max_Scale_Mat} 4 | \alias{Min_Max_Scale_Mat} 5 | \title{Matrix MinMax Scaling} 6 | \usage{ 7 | Min_Max_Scale_Mat(mat, lower, upper) 8 | } 9 | \arguments{ 10 | \item{mat}{a matrix of original hyperparameters} 11 | 12 | \item{lower}{lower bounds} 13 | 14 | \item{upper}{upper bounds} 15 | } 16 | \value{ 17 | a matrix of scaled hyperparameters 18 | } 19 | \description{ 20 | Transforms hyperparameters by scaling each hyperparameter to a given range. 21 | } 22 | \keyword{internal} 23 | -------------------------------------------------------------------------------- /man/Utility.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/Utility.R 3 | \name{Utility} 4 | \alias{Utility} 5 | \title{Utility Computing Function} 6 | \usage{ 7 | Utility(x_vec, GP, acq = "ucb", y_max, kappa, eps) 8 | } 9 | \arguments{ 10 | \item{x_vec}{a vector of scaled hyperparameters} 11 | 12 | \item{GP}{an object of class GP} 13 | 14 | \item{acq}{Acquisition function type to be used} 15 | 16 | \item{y_max}{The current maximum known value of the target utility function} 17 | 18 | \item{kappa}{tunable parameter kappa to balance exploitation against exploration} 19 | 20 | \item{eps}{tunable parameter epsilon to balance exploitation against exploration} 21 | } 22 | \value{ 23 | negative utility to be minimized 24 | } 25 | \description{ 26 | Computing Utility. 27 | } 28 | \keyword{internal} 29 | -------------------------------------------------------------------------------- /man/Utility_Max.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/Utility_Max.R 3 | \name{Utility_Max} 4 | \alias{Utility_Max} 5 | \title{Utility Maximization Function} 6 | \usage{ 7 | Utility_Max(DT_bounds, GP, acq = "ucb", y_max, kappa, eps) 8 | } 9 | \arguments{ 10 | \item{DT_bounds}{hyperparameters lower and upper bounds to limit the search of the acq max} 11 | 12 | \item{GP}{an object of class GP} 13 | 14 | \item{acq}{Acquisition function type to be used} 15 | 16 | \item{y_max}{The current maximum known value of the target utility function} 17 | 18 | \item{kappa}{tunable parameter kappa to balance exploitation against exploration} 19 | 20 | \item{eps}{tunable parameter epsilon to balance exploitation against exploration} 21 | } 22 | \value{ 23 | The arg max of the acquisition function 24 | } 25 | \description{ 26 | Utility Maximization. 27 | } 28 | \keyword{internal} 29 | -------------------------------------------------------------------------------- /man/rBayesianOptimization.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/rBayesianOptimization.R 3 | \docType{package} 4 | \name{rBayesianOptimization} 5 | \alias{rBayesianOptimization-package} 6 | \alias{rBayesianOptimization} 7 | \title{rBayesianOptimization: Bayesian Optimization of Hyperparameters} 8 | \description{ 9 | A Pure R implementation of bayesian global optimization with gaussian processes. 10 | } 11 | \seealso{ 12 | Useful links: 13 | \itemize{ 14 | \item \url{https://github.com/yanyachen/rBayesianOptimization} 15 | \item Report bugs at \url{https://github.com/yanyachen/rBayesianOptimization/issues} 16 | } 17 | 18 | } 19 | \author{ 20 | \strong{Maintainer}: Yachen Yan \email{yanyachen21@gmail.com} 21 | 22 | } 23 | --------------------------------------------------------------------------------