├── .Rbuildignore ├── .gitignore ├── NAMESPACE ├── vignettes ├── imgperc001.png ├── imgperc002.png ├── imgperc003.png ├── imgperc004.png ├── howto_automl.pdf ├── automl.Rmd └── howto_automl.Rnw ├── docs ├── articles │ ├── imgperc001.png │ ├── imgperc002.png │ ├── imgperc003.png │ ├── imgperc004.png │ └── index.html ├── pkgdown.yml ├── link.svg ├── docsearch.js ├── pkgdown.js ├── 404.html ├── authors.html ├── pkgdown.css ├── index.html ├── reference │ ├── index.html │ ├── pso.html │ ├── automl_train.html │ ├── hpar.html │ ├── automl_predict.html │ ├── autopar.html │ └── automl_train_manual.html └── docsearch.css ├── automl.Rproj ├── README.md ├── DESCRIPTION ├── man ├── pso.Rd ├── automl_predict.Rd ├── automl_train.Rd ├── automl_train_manual.Rd ├── hpar.Rd └── autopar.Rd └── NEWS /.Rbuildignore: -------------------------------------------------------------------------------- 1 | ^.*\.Rproj$ 2 | ^\.Rproj\.user$ 3 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .Rproj.user 2 | .Rhistory 3 | .RData 4 | .Ruserdata 5 | -------------------------------------------------------------------------------- /NAMESPACE: -------------------------------------------------------------------------------- 1 | export("automl_train_manual") 2 | export("automl_train") 3 | export("automl_predict") 4 | -------------------------------------------------------------------------------- /vignettes/imgperc001.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aboulaboul/automl/HEAD/vignettes/imgperc001.png -------------------------------------------------------------------------------- /vignettes/imgperc002.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aboulaboul/automl/HEAD/vignettes/imgperc002.png -------------------------------------------------------------------------------- /vignettes/imgperc003.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aboulaboul/automl/HEAD/vignettes/imgperc003.png -------------------------------------------------------------------------------- /vignettes/imgperc004.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aboulaboul/automl/HEAD/vignettes/imgperc004.png -------------------------------------------------------------------------------- /vignettes/howto_automl.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aboulaboul/automl/HEAD/vignettes/howto_automl.pdf -------------------------------------------------------------------------------- /docs/articles/imgperc001.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aboulaboul/automl/HEAD/docs/articles/imgperc001.png -------------------------------------------------------------------------------- /docs/articles/imgperc002.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aboulaboul/automl/HEAD/docs/articles/imgperc002.png -------------------------------------------------------------------------------- /docs/articles/imgperc003.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aboulaboul/automl/HEAD/docs/articles/imgperc003.png -------------------------------------------------------------------------------- /docs/articles/imgperc004.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aboulaboul/automl/HEAD/docs/articles/imgperc004.png -------------------------------------------------------------------------------- /docs/pkgdown.yml: -------------------------------------------------------------------------------- 1 | pandoc: 2.3.1 2 | pkgdown: 1.4.0 3 | pkgdown_sha: ~ 4 | articles: 5 | automl: automl.html 6 | 7 | -------------------------------------------------------------------------------- /automl.Rproj: -------------------------------------------------------------------------------- 1 | Version: 1.0 2 | 3 | RestoreWorkspace: Default 4 | SaveWorkspace: Default 5 | AlwaysSaveHistory: Yes 6 | 7 | EnableCodeIndexing: Yes 8 | UseSpacesForTab: Yes 9 | NumSpacesForTab: 1 10 | Encoding: UTF-8 11 | 12 | RnwWeave: Sweave 13 | LaTeX: pdfLaTeX 14 | 15 | AutoAppendNewline: Yes 16 | StripTrailingWhitespace: Yes 17 | 18 | BuildType: Package 19 | PackageUseDevtools: Yes 20 | PackageInstallArgs: --no-multiarch --with-keep.source 21 | PackageCheckArgs: R CMD check --as-cran 22 | PackageRoxygenize: vignette 23 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | automl package fits from simple regression to highly customizable deep neural networks 2 | either with gradient descent or metaheuristic, using automatic hyper parameters 3 | tuning and custom cost function. 4 | A mix inspired by the common tricks on Deep Learning and Particle Swarm Optimization. 5 | 6 | (Key words: autoML, Deep Learning, Particle Swarm Optimization, learning rate, minibatch, 7 | batch normalization, lambda, RMSprop, momentum, adam optimization, learning rate decay, 8 | inverted dropout, particles number, kappa, regression, logistic regression) 9 | -------------------------------------------------------------------------------- /DESCRIPTION: -------------------------------------------------------------------------------- 1 | Package: automl 2 | Type: Package 3 | Title: Deep Learning with Metaheuristic 4 | Version: 1.3.2 5 | Authors@R: person("Alex", "Boulangé", role = c("aut", "cre"), email = "aboul@free.fr") 6 | BugReports: https://github.com/aboulaboul/automl/issues 7 | Description: Fits from simple regression to highly customizable deep neural networks 8 | either with gradient descent or metaheuristic, using automatic hyper parameters 9 | tuning and custom cost function. 10 | A mix inspired by the common tricks on Deep Learning and Particle Swarm Optimization. 11 | URL: https://aboulaboul.github.io/automl 12 | https://github.com/aboulaboul/automl 13 | License: GNU General Public License 14 | Encoding: UTF-8 15 | LazyData: TRUE 16 | Imports: stats, utils, parallel 17 | Suggests: datasets 18 | RoxygenNote: 6.1.1 19 | -------------------------------------------------------------------------------- /docs/link.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 5 | 8 | 12 | 13 | -------------------------------------------------------------------------------- /man/pso.Rd: -------------------------------------------------------------------------------- 1 | \name{pso} 2 | \alias{pso} 3 | \title{PSO parameters and hyperparameters} 4 | \description{ 5 | List of parameters and hyperparameters for Particle Swarm Optimization 6 | } 7 | \arguments{ 8 | All PSO parameters and hyperparameters are preset with default value\cr 9 | 10 | \item{psopartpopsize}{ number of particles in swarm (discrete value)\cr 11 | (\sQuote{autopar} context: default value 8, which means that 8 different neural net hyperparameters sets will be tested\cr 12 | (\sQuote{hpar} context: default value 50, which means that 50 neural net weights sets will be tested\cr 13 | #tuning priority 1} (impact on memory consumption)\cr 14 | 15 | \emph{CAUTION: you should only change the values below if you know what you are doing} 16 | 17 | \item{psovarvalmin}{ Minimum value for particles positions (default value -10)} 18 | 19 | \item{psovarvalmax}{ maximum value for particles positions (default value 10)} 20 | 21 | \item{psovelocitymaxratio}{ ratio applied to limit velocities (continuous value between 0 and 1, default value 0.2)} 22 | 23 | \item{psoinertiadampratio}{ inertia damp ratio (continuous value between 0 and 1, default value 1 equivalent to OFF)} 24 | 25 | \item{psokappa}{ kappa (default value 1)} 26 | 27 | \item{psophi1}{ Phi 1 (default value 2.05)} 28 | 29 | \item{psophi2}{ Phi 2 (default value 2.05)}\cr 30 | 31 | \emph{back to \link{autopar}, \link{hpar}} 32 | } 33 | 34 | \seealso{ 35 | PSO video tutorial from Yarpiz 36 | } 37 | -------------------------------------------------------------------------------- /man/automl_predict.Rd: -------------------------------------------------------------------------------- 1 | \name{automl_predict} 2 | \alias{automl_predict} 3 | \title{automl_predict} 4 | \description{ 5 | Predictions function, to apply a trained model on datas 6 | } 7 | \usage{ 8 | automl_predict(model, X, layoutputnum) 9 | } 10 | \arguments{ 11 | \item{model}{ model trained previously with \link{automl_train} or \link{automl_train_manual}} 12 | 13 | \item{X}{ inputs matrix or data.frame (containing numerical values only)} 14 | 15 | \item{layoutputnum}{ which layer number to output especially for auto encoding (default 0: no particular layer, the last one)} 16 | } 17 | \examples{ 18 | ##REGRESSION (predict Sepal.Length given other parameters) 19 | data(iris) 20 | xmat <- as.matrix(cbind(iris[,2:4], as.numeric(iris$Species))) 21 | ymat <- iris[,1] 22 | amlmodel <- automl_train_manual(Xref = xmat, Yref = ymat, 23 | hpar = list(modexec = 'trainwpso', verbose = FALSE)) 24 | res <- cbind(ymat, automl_predict(model = amlmodel, X = xmat)) 25 | colnames(res) <- c('actual', 'predict') 26 | head(res) 27 | # 28 | \dontrun{ 29 | ##CLASSIFICATION (predict Species given other Iris parameters) 30 | data(iris) 31 | xmat = iris[,1:4] 32 | lab2pred <- levels(iris$Species) 33 | lghlab <- length(lab2pred) 34 | iris$Species <- as.numeric(iris$Species) 35 | ymat <- matrix(seq(from = 1, to = lghlab, by = 1), nrow(xmat), 36 | lghlab, byrow = TRUE) 37 | ymat <- (ymat == as.numeric(iris$Species)) + 0 38 | amlmodel <- automl_train_manual(Xref = xmat, Yref = ymat, 39 | hpar = list(modexec = 'trainwpso', verbose = FALSE)) 40 | res <- cbind(ymat, round(automl_predict(model = amlmodel, X = xmat))) 41 | colnames(res) <- c(paste('act',lab2pred, sep = '_'), 42 | paste('pred',lab2pred, sep = '_')) 43 | head(res) 44 | } 45 | } 46 | -------------------------------------------------------------------------------- /docs/docsearch.js: -------------------------------------------------------------------------------- 1 | $(function() { 2 | 3 | // register a handler to move the focus to the search bar 4 | // upon pressing shift + "/" (i.e. "?") 5 | $(document).on('keydown', function(e) { 6 | if (e.shiftKey && e.keyCode == 191) { 7 | e.preventDefault(); 8 | $("#search-input").focus(); 9 | } 10 | }); 11 | 12 | $(document).ready(function() { 13 | // do keyword highlighting 14 | /* modified from https://jsfiddle.net/julmot/bL6bb5oo/ */ 15 | var mark = function() { 16 | 17 | var referrer = document.URL ; 18 | var paramKey = "q" ; 19 | 20 | if (referrer.indexOf("?") !== -1) { 21 | var qs = referrer.substr(referrer.indexOf('?') + 1); 22 | var qs_noanchor = qs.split('#')[0]; 23 | var qsa = qs_noanchor.split('&'); 24 | var keyword = ""; 25 | 26 | for (var i = 0; i < qsa.length; i++) { 27 | var currentParam = qsa[i].split('='); 28 | 29 | if (currentParam.length !== 2) { 30 | continue; 31 | } 32 | 33 | if (currentParam[0] == paramKey) { 34 | keyword = decodeURIComponent(currentParam[1].replace(/\+/g, "%20")); 35 | } 36 | } 37 | 38 | if (keyword !== "") { 39 | $(".contents").unmark({ 40 | done: function() { 41 | $(".contents").mark(keyword); 42 | } 43 | }); 44 | } 45 | } 46 | }; 47 | 48 | mark(); 49 | }); 50 | }); 51 | 52 | /* Search term highlighting ------------------------------*/ 53 | 54 | function matchedWords(hit) { 55 | var words = []; 56 | 57 | var hierarchy = hit._highlightResult.hierarchy; 58 | // loop to fetch from lvl0, lvl1, etc. 59 | for (var idx in hierarchy) { 60 | words = words.concat(hierarchy[idx].matchedWords); 61 | } 62 | 63 | var content = hit._highlightResult.content; 64 | if (content) { 65 | words = words.concat(content.matchedWords); 66 | } 67 | 68 | // return unique words 69 | var words_uniq = [...new Set(words)]; 70 | return words_uniq; 71 | } 72 | 73 | function updateHitURL(hit) { 74 | 75 | var words = matchedWords(hit); 76 | var url = ""; 77 | 78 | if (hit.anchor) { 79 | url = hit.url_without_anchor + '?q=' + escape(words.join(" ")) + '#' + hit.anchor; 80 | } else { 81 | url = hit.url + '?q=' + escape(words.join(" ")); 82 | } 83 | 84 | return url; 85 | } 86 | -------------------------------------------------------------------------------- /man/automl_train.Rd: -------------------------------------------------------------------------------- 1 | \name{automl_train} 2 | \alias{automl_train} 3 | \title{automl_train} 4 | \description{ 5 | The multi deep neural network automatic train function (several deep neural networks are trained with automatic hyperparameters tuning, best model is kept)\cr 6 | This function launches the \link{automl_train_manual} function by passing it parameters 7 | for each particle at each converging step 8 | } 9 | \usage{ 10 | automl_train(Xref, Yref, autopar = list(), hpar = list(), mdlref = NULL) 11 | } 12 | \arguments{ 13 | \item{Xref}{ inputs matrix or data.frame (containing numerical values only) 14 | } 15 | 16 | \item{Yref}{ target matrix or data.frame (containing numerical values only) 17 | } 18 | 19 | \item{autopar}{ list of parameters for hyperparameters optimization, see \link{autopar} section\cr 20 | Not mandatory (the list is preset and all arguments are initialized with default value) but it is advisable to adjust some important arguments for performance reasons (including processing time) 21 | } 22 | 23 | \item{hpar}{ list of parameters and hyperparameters for Deep Neural Network, see \link{hpar} section\cr 24 | Not mandatory (the list is preset and all arguments are initialized with default value) but it is advisable to adjust some important arguments for performance reasons (including processing time) 25 | } 26 | 27 | \item{mdlref}{ model trained with \link{automl_train} to start training with saved \link{hpar} and \link{autopar} 28 | (not the model)\cr 29 | nb: manually entered parameters above override loaded ones} 30 | } 31 | 32 | \examples{ 33 | \dontrun{ 34 | ##REGRESSION (predict Sepal.Length given other Iris parameters) 35 | data(iris) 36 | xmat <- cbind(iris[,2:4], as.numeric(iris$Species)) 37 | ymat <- iris[,1] 38 | amlmodel <- automl_train(Xref = xmat, Yref = ymat) 39 | } 40 | ##CLASSIFICATION (predict Species given other Iris parameters) 41 | data(iris) 42 | xmat = iris[,1:4] 43 | lab2pred <- levels(iris$Species) 44 | lghlab <- length(lab2pred) 45 | iris$Species <- as.numeric(iris$Species) 46 | ymat <- matrix(seq(from = 1, to = lghlab, by = 1), nrow(xmat), lghlab, byrow = TRUE) 47 | ymat <- (ymat == as.numeric(iris$Species)) + 0 48 | #with gradient descent and random hyperparameters sets 49 | amlmodel <- automl_train(Xref = xmat, Yref = ymat, 50 | autopar = list(numiterations = 1, psopartpopsize = 1, seed = 11), 51 | hpar = list(numiterations = 10)) 52 | } 53 | -------------------------------------------------------------------------------- /docs/pkgdown.js: -------------------------------------------------------------------------------- 1 | /* http://gregfranko.com/blog/jquery-best-practices/ */ 2 | (function($) { 3 | $(function() { 4 | 5 | $('.navbar-fixed-top').headroom(); 6 | 7 | $('body').css('padding-top', $('.navbar').height() + 10); 8 | $(window).resize(function(){ 9 | $('body').css('padding-top', $('.navbar').height() + 10); 10 | }); 11 | 12 | $('body').scrollspy({ 13 | target: '#sidebar', 14 | offset: 60 15 | }); 16 | 17 | $('[data-toggle="tooltip"]').tooltip(); 18 | 19 | var cur_path = paths(location.pathname); 20 | var links = $("#navbar ul li a"); 21 | var max_length = -1; 22 | var pos = -1; 23 | for (var i = 0; i < links.length; i++) { 24 | if (links[i].getAttribute("href") === "#") 25 | continue; 26 | // Ignore external links 27 | if (links[i].host !== location.host) 28 | continue; 29 | 30 | var nav_path = paths(links[i].pathname); 31 | 32 | var length = prefix_length(nav_path, cur_path); 33 | if (length > max_length) { 34 | max_length = length; 35 | pos = i; 36 | } 37 | } 38 | 39 | // Add class to parent
  • , and enclosing
  • if in dropdown 40 | if (pos >= 0) { 41 | var menu_anchor = $(links[pos]); 42 | menu_anchor.parent().addClass("active"); 43 | menu_anchor.closest("li.dropdown").addClass("active"); 44 | } 45 | }); 46 | 47 | function paths(pathname) { 48 | var pieces = pathname.split("/"); 49 | pieces.shift(); // always starts with / 50 | 51 | var end = pieces[pieces.length - 1]; 52 | if (end === "index.html" || end === "") 53 | pieces.pop(); 54 | return(pieces); 55 | } 56 | 57 | // Returns -1 if not found 58 | function prefix_length(needle, haystack) { 59 | if (needle.length > haystack.length) 60 | return(-1); 61 | 62 | // Special case for length-0 haystack, since for loop won't run 63 | if (haystack.length === 0) { 64 | return(needle.length === 0 ? 0 : -1); 65 | } 66 | 67 | for (var i = 0; i < haystack.length; i++) { 68 | if (needle[i] != haystack[i]) 69 | return(i); 70 | } 71 | 72 | return(haystack.length); 73 | } 74 | 75 | /* Clipboard --------------------------*/ 76 | 77 | function changeTooltipMessage(element, msg) { 78 | var tooltipOriginalTitle=element.getAttribute('data-original-title'); 79 | element.setAttribute('data-original-title', msg); 80 | $(element).tooltip('show'); 81 | element.setAttribute('data-original-title', tooltipOriginalTitle); 82 | } 83 | 84 | if(ClipboardJS.isSupported()) { 85 | $(document).ready(function() { 86 | var copyButton = ""; 87 | 88 | $(".examples, div.sourceCode").addClass("hasCopyButton"); 89 | 90 | // Insert copy buttons: 91 | $(copyButton).prependTo(".hasCopyButton"); 92 | 93 | // Initialize tooltips: 94 | $('.btn-copy-ex').tooltip({container: 'body'}); 95 | 96 | // Initialize clipboard: 97 | var clipboardBtnCopies = new ClipboardJS('[data-clipboard-copy]', { 98 | text: function(trigger) { 99 | return trigger.parentNode.textContent; 100 | } 101 | }); 102 | 103 | clipboardBtnCopies.on('success', function(e) { 104 | changeTooltipMessage(e.trigger, 'Copied!'); 105 | e.clearSelection(); 106 | }); 107 | 108 | clipboardBtnCopies.on('error', function() { 109 | changeTooltipMessage(e.trigger,'Press Ctrl+C or Command+C to copy'); 110 | }); 111 | }); 112 | } 113 | })(window.jQuery || window.$) 114 | -------------------------------------------------------------------------------- /man/automl_train_manual.Rd: -------------------------------------------------------------------------------- 1 | \name{automl_train_manual} 2 | \alias{automl_train_manual} 3 | \title{automl_train_manual} 4 | \description{ 5 | The base deep neural network train function (one deep neural network trained without automatic hyperparameters tuning) 6 | } 7 | \usage{ 8 | automl_train_manual(Xref, Yref, hpar = list(), mdlref = NULL) 9 | } 10 | \arguments{ 11 | \item{Xref}{ inputs matrix or data.frame (containing numerical values only)} 12 | 13 | \item{Yref}{ target matrix or data.frame (containing numerical values only)} 14 | 15 | \item{hpar}{ list of parameters and hyperparameters for Deep Neural Network, see \link{hpar} section\cr 16 | Not mandatory (the list is preset and all arguments are initialized with default value) but it is advisable to adjust some important arguments for performance reasons (including processing time)} 17 | 18 | \item{mdlref}{ model trained with \link{automl_train} or \link{automl_train_manual} to start training from a saved model (shape, 19 | weights...) for fine tuning\cr 20 | nb: manually entered parameters above override loaded ones} 21 | } 22 | 23 | \examples{ 24 | ##REGRESSION (predict Sepal.Length given other Iris parameters) 25 | data(iris) 26 | xmat <- cbind(iris[,2:4], as.numeric(iris$Species)) 27 | ymat <- iris[,1] 28 | #with gradient descent 29 | amlmodel <- automl_train_manual(Xref = xmat, Yref = ymat, 30 | hpar = list(learningrate = 0.01, 31 | numiterations = 30, 32 | minibatchsize = 2^2)) 33 | \dontrun{ 34 | #with PSO 35 | amlmodel <- automl_train_manual(Xref = xmat, Yref = ymat, 36 | hpar = list(modexec = 'trainwpso', 37 | numiterations = 30, 38 | psopartpopsize = 50)) 39 | #with PSO and custom cost function 40 | f <- 'J=abs((y-yhat)/y)' 41 | f <- c(f, 'J=sum(J[!is.infinite(J)],na.rm=TRUE)') 42 | f <- c(f, 'J=(J/length(y))') 43 | f <- paste(f, collapse = ';') 44 | amlmodel <- automl_train_manual(Xref = xmat, Yref = ymat, 45 | hpar = list(modexec = 'trainwpso', 46 | numiterations = 30, 47 | psopartpopsize = 50, 48 | costcustformul = f)) 49 | 50 | ##CLASSIFICATION (predict Species given other Iris parameters) 51 | data(iris) 52 | xmat = iris[,1:4] 53 | lab2pred <- levels(iris$Species) 54 | lghlab <- length(lab2pred) 55 | iris$Species <- as.numeric(iris$Species) 56 | ymat <- matrix(seq(from = 1, to = lghlab, by = 1), nrow(xmat), lghlab, byrow = TRUE) 57 | ymat <- (ymat == as.numeric(iris$Species)) + 0 58 | #with gradient descent and 2 hidden layers 59 | amlmodel <- automl_train_manual(Xref = xmat, Yref = ymat, 60 | hpar = list(layersshape = c(10, 10, 0), 61 | layersacttype = c('tanh', 'relu', 'sigmoid'), 62 | layersdropoprob = c(0, 0, 0))) 63 | #with gradient descent and no hidden layer (logistic regression) 64 | amlmodel <- automl_train_manual(Xref = xmat, Yref = ymat, 65 | hpar = list(layersshape = c(0), 66 | layersacttype = c('sigmoid'), 67 | layersdropoprob = c(0))) 68 | #with PSO and softmax 69 | amlmodel <- automl_train_manual(Xref = xmat, Yref = ymat, 70 | hpar = list(modexec = 'trainwpso', 71 | layersshape = c(10, 0), 72 | layersacttype = c('relu', 'softmax'), 73 | layersdropoprob = c(0, 0), 74 | numiterations = 50, 75 | psopartpopsize = 50)) 76 | } 77 | } 78 | -------------------------------------------------------------------------------- /NEWS: -------------------------------------------------------------------------------- 1 | automl 1.3.2 2 | ============================================ 3 | * one step forward, one step back! 4 | MAPE cost function is back and validated 5 | RMSE cost function removed (derivative 6 | challenge + no added value vs MSE imo) 7 | * bug fix: 'chkgradevery' is now available 8 | 9 | automl 1.3.1 10 | ============================================ 11 | * direct ‘overfitting’ & ‘regularization’ 12 | modes for auto_runtype 13 | * batch normalization set to 0 by default 14 | (not good for regression targets) 15 | * bug fix: now exits if no gain when no cros- 16 | s validation sample 17 | * bug fix: testgainunder now kept as is 18 | * MAPE cost function removed (probably an er- 19 | ror to be fixed in derivative: todo list) 20 | 21 | automl 1.3.0 22 | ============================================ 23 | * rmse & mape cost function added for regres- 24 | sion 25 | 26 | automl 1.2.9 27 | ============================================ 28 | * documentation update (typo correction and 29 | more detailed explanations concerning 30 | modexec and autopar) 31 | thanks to Alexey 32 | 33 | automl 1.2.8 34 | ============================================ 35 | * time limit for sub modelizations in automl_ 36 | train function, to avoid waiting too long 37 | for a specific particle to finish its mode- 38 | lization 39 | 40 | automl 1.2.7 41 | ============================================ 42 | * lightening models produced by removing 43 | unnecessary data (Z, A layers, etc...) 44 | 45 | automl 1.2.6 46 | ============================================ 47 | * just a proper Authors@R field 4 CRAN :-) 48 | 49 | automl 1.2.5 50 | ============================================ 51 | * automl_train bug fix when particle produces 52 | no exploitable model (continued work) 53 | 54 | automl 1.2.4 55 | ============================================ 56 | * automl_train bug fix when particle produces 57 | no exploitable model 58 | 59 | automl 1.2.3 60 | ============================================ 61 | * dropout bug fix (now random at each mini 62 | batch and reproductible) 63 | * bug fix: continue training on auto trained model 64 | with manual function 65 | 66 | automl 1.2.2 67 | ============================================ 68 | * pkgdown site with vignettes (Rmd file added) 69 | 70 | automl 1.2.1 71 | ============================================ 72 | * pkgdown site (nb: Rnw vignettes are not included) 73 | 74 | automl 1.2.0 75 | ============================================ 76 | * New param 'mdlref' for automl_train_manual: 77 | to start training from saved model (shape, 78 | weights...) for fine tuning 79 | * New param 'mdlref' for automl_train: 80 | to start training with loaded hpar and autopar 81 | (not the model) 82 | * New 'auto_runtype' autopar for automl_train 83 | to run automatically the 2 steps below; 84 | 1: overfitting, goal: performance 85 | 2: regularization, goal: generalization 86 | * dropout bug fix 87 | * lambda bug fix 88 | 89 | automl 1.1.0 90 | ============================================ 91 | * same train/test sampling for each particle with automl_train 92 | * stick to a format in variables naming 93 | 94 | automl 1.0.9 95 | ============================================ 96 | * testcvsize = 0 bug fix 97 | 98 | automl 1.0.8 99 | ============================================ 100 | * New automatic hyperparameters adjustments below: 101 | 'auto_psovelocitymaxratio' autopar PSO velocity max ratio 102 | 'auto_layer' autopar layer shape (layers number, nodes 103 | number per layer, activation types and dropout ratios) 104 | * 'auto_lambda' bug fix 105 | 106 | automl 1.0.7 107 | ============================================ 108 | * vignette completion 109 | 110 | automl 1.0.6 111 | ============================================ 112 | * vignette howto_automl: why, how and basic howto 113 | * 'auto_lambda' autopar regularization hyperparameter 114 | * seed bug fix for reproductibility 115 | * display enhancement in NN structure display 116 | 117 | automl 1.0.5 118 | ============================================ 119 | * first public release on CRAN 120 | * There's so much to do; transfert learning, CNN, RNN ... 121 | feel free to join 122 | -------------------------------------------------------------------------------- /docs/404.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | Page not found (404) • automl 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 51 | 52 | 53 | 54 | 55 | 56 | 57 |
    58 |
    59 | 101 | 102 | 103 | 104 |
    105 | 106 |
    107 |
    108 |
    109 |

    Page not found (404)

    110 |
    111 | 112 | Content not found. Please use links in the navbar. 113 | 114 |
    115 | 116 |
    117 | 118 | 119 | 120 |
    121 |
    122 |

    Developed by Alex Boulangé.

    123 |
    124 | 125 |
    126 |

    Site built with pkgdown 1.4.0.

    127 |
    128 | 129 |
    130 |
    131 | 132 | 133 | 134 | 135 | 136 | 137 | 138 | 139 | -------------------------------------------------------------------------------- /docs/authors.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | Authors • automl 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 51 | 52 | 53 | 54 | 55 | 56 | 57 |
    58 |
    59 | 101 | 102 | 103 | 104 |
    105 | 106 |
    107 |
    108 |
    109 |

    Authors

    110 |
    111 | 112 |
      113 |
    • 114 |

      Alex Boulangé. Author, maintainer. 115 |

      116 |
      • 117 |
    118 | 119 |
    120 | 121 |
    122 | 123 | 124 | 125 |
    126 |
    127 |

    Developed by Alex Boulangé.

    128 |
    129 | 130 |
    131 |

    Site built with pkgdown 1.4.0.

    132 |
    133 | 134 |
    135 |
    136 | 137 | 138 | 139 | 140 | 141 | 142 | 143 | 144 | -------------------------------------------------------------------------------- /docs/articles/index.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | Articles • automl 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 51 | 52 | 53 | 54 | 55 | 56 | 57 |
    58 |
    59 | 101 | 102 | 103 | 104 |
    105 | 106 |
    107 |
    108 |
    109 |

    Articles

    110 |
    111 | 112 |
    113 |

    All vignettes

    114 |

    115 | 116 | 119 |
    120 |
    121 |
    122 | 123 | 124 |
    125 |
    126 |

    Developed by Alex Boulangé.

    127 |
    128 | 129 |
    130 |

    Site built with pkgdown 1.4.0.

    131 |
    132 | 133 |
    134 |
    135 | 136 | 137 | 138 | 139 | 140 | 141 | 142 | 143 | -------------------------------------------------------------------------------- /man/hpar.Rd: -------------------------------------------------------------------------------- 1 | \name{hpar} 2 | \alias{hpar} 3 | \title{Deep Neural Net parameters and hyperparameters} 4 | \description{ 5 | List of Neural Network parameters and hyperparameters to train with gradient descent or particle swarm optimization\cr 6 | Not mandatory (the list is preset and all arguments are initialized with default value) but it is advisable to adjust some important arguments for performance reasons (including processing time) 7 | } 8 | \arguments{ 9 | 10 | \item{modexec}{ \sQuote{trainwgrad} (the default value) to train with gradient descent (suitable for all volume of data)\cr 11 | \sQuote{trainwpso} to train using Particle Swarm Optimization, each particle represents a set of neural network weights (CAUTION: suitable for low volume of data, time consuming for medium to large volume of data)}\cr 12 | 13 | \emph{Below specific arguments to \sQuote{trainwgrad} execution mode} 14 | 15 | \item{learningrate}{ learningrate alpha (default value 0.001)\cr 16 | #tuning priority 1} 17 | 18 | \item{beta1}{ see below} 19 | \item{beta2}{ \sQuote{Momentum} if beta1 different from 0 and beta2 equal 0 )\cr 20 | \sQuote{RMSprop} if beta1 equal 0 and beta2 different from 0\cr 21 | \sQuote{adam optimization} if beta1 different from 0 and beta2 different from 0 (default)\cr 22 | (default value beta1 equal 0.9 and beta2 equal 0.999)\cr 23 | #tuning priority 2} 24 | 25 | \item{lrdecayrate}{ learning rate decay value (default value 0, no learning rate decay, 1e-6 should be a good value to start with)\cr 26 | #tuning priority 4} 27 | 28 | \item{chkgradevery}{ epoch interval to run gradient check function (default value 0, for debug only)} 29 | 30 | \item{chkgradepsilon}{ epsilon value for derivative calculations and threshold test in gradient check function (default 0.0000001)}\cr 31 | 32 | \emph{Below specific arguments to \sQuote{trainwpso} execution mode} 33 | 34 | \item{psoxxx}{ see \link{pso} for PSO specific arguments details} 35 | 36 | \item{costcustformul}{ custom cost formula (default \sQuote{}, no custom cost function)\cr 37 | standard input variables: yhat (prediction), y (target actual value)\cr 38 | custom input variables: any variable declared in hpar may be used via alias mydl (ie: hpar(list = (foo = 1.5)) will be used in custom cost formula as mydl$foo))\cr 39 | result: J\cr 40 | see \sQuote{automl_train_manual} example using Mean Average Percentage Error cost function\cr 41 | nb: X and Y matrices used as input into automl_train_manual or automl_train_manual functions are transposed (features in rows and cases in columns)}\cr 42 | 43 | \emph{Below arguments for both execution modes} 44 | 45 | \item{numiterations}{ number of training epochs (default value 50))\cr 46 | #tuning priority 1} 47 | 48 | \item{seed}{ seed for reproductibility (default 4)} 49 | 50 | \item{minibatchsize}{ mini batch size, 2 to the power 0 for stochastic gradient descent (default 2 to the power 5) 51 | #tuning priority 3} 52 | 53 | \item{layersshape}{ number of nodes per layer, each nodes number initialize a hidden layer\cr 54 | output layer nodes number, may be left to 0 it will be automatically set by Y matrix shape\cr 55 | default value one hidden layer with 10 nodes: c(10, 0)\cr 56 | #tuning priority 4} 57 | 58 | \item{layersacttype}{ activation function for each layer; \sQuote{linear} for no activation or \sQuote{sigmoid}, \sQuote{relu} or \sQuote{reluleaky} or \sQuote{tanh} or \sQuote{softmax} (softmax for output layer only supported in trainwpso exec mode)\cr 59 | output layer activation function may be left to \sQuote{}, default value \sQuote{linear} for regression, \sQuote{sigmoid} for classification\cr 60 | nb: layersacttype parameter vector must have same length as layersshape parameter vector\cr 61 | default value c(\sQuote{relu}, \sQuote{})\cr 62 | #tuning priority 4} 63 | 64 | \item{layersdropoprob}{ drop out probability for each layer, continuous value from 0 to less than 1 (give the percentage of matrix weight values to drop out randomly)\cr 65 | nb: layersdropoprob parameter vector must have same length as layersshape parameter vector\cr 66 | default value no drop out: c(0, 0)\cr 67 | #tuning priority for regularization} 68 | 69 | \item{printcostevery}{ epoch interval to test and print costs (train and cross validation cost: default value 10, for 1 test every 10 epochs)} 70 | 71 | \item{testcvsize}{ size of cross validation sample, 0 for no cross validation sample (default 10, for 10 percent)} 72 | 73 | \item{testgainunder}{ threshold to stop the training if the gain between last train or cross validation cost is smaller than the threshold, 0 for no stop test (default 0.000001)} 74 | 75 | \item{costtype}{ cost type function name \sQuote{mse} or \sQuote{crossentropy} or \sQuote{custom}\cr 76 | \sQuote{mse} for Mean Squared Error, set automatically for continuous target type (\sQuote{mape} Mean Absolute Percentage Error may be specified)\cr 77 | \sQuote{crossentropy} set automatically for binary target type\cr 78 | \sQuote{custom} set automatically if \sQuote{costcustformul} different from \sQuote{} 79 | } 80 | 81 | \item{lambda}{ regularization term added to cost function (default value 0, no regularization) 82 | } 83 | 84 | \item{batchnor_mom}{ batch normalization momentum for j and B (default 0, no batch normalization, may be set to 0.9 for deep neural net) 85 | } 86 | 87 | \item{epsil}{ epsilon the low value to avoid dividing by 0 or log(0) in cost function, etc ... (default value 1e-12)\cr 88 | } 89 | 90 | \item{verbose}{ to display or not the costs and the shapes (default TRUE)}\cr 91 | 92 | \emph{back to \link{automl_train}, \link{automl_train_manual}} 93 | } 94 | 95 | \seealso{ 96 | Deep Learning specialization from Andrew NG on Coursera 97 | } 98 | -------------------------------------------------------------------------------- /docs/pkgdown.css: -------------------------------------------------------------------------------- 1 | /* Sticky footer */ 2 | 3 | /** 4 | * Basic idea: https://philipwalton.github.io/solved-by-flexbox/demos/sticky-footer/ 5 | * Details: https://github.com/philipwalton/solved-by-flexbox/blob/master/assets/css/components/site.css 6 | * 7 | * .Site -> body > .container 8 | * .Site-content -> body > .container .row 9 | * .footer -> footer 10 | * 11 | * Key idea seems to be to ensure that .container and __all its parents__ 12 | * have height set to 100% 13 | * 14 | */ 15 | 16 | html, body { 17 | height: 100%; 18 | } 19 | 20 | body > .container { 21 | display: flex; 22 | height: 100%; 23 | flex-direction: column; 24 | } 25 | 26 | body > .container .row { 27 | flex: 1 0 auto; 28 | } 29 | 30 | footer { 31 | margin-top: 45px; 32 | padding: 35px 0 36px; 33 | border-top: 1px solid #e5e5e5; 34 | color: #666; 35 | display: flex; 36 | flex-shrink: 0; 37 | } 38 | footer p { 39 | margin-bottom: 0; 40 | } 41 | footer div { 42 | flex: 1; 43 | } 44 | footer .pkgdown { 45 | text-align: right; 46 | } 47 | footer p { 48 | margin-bottom: 0; 49 | } 50 | 51 | img.icon { 52 | float: right; 53 | } 54 | 55 | img { 56 | max-width: 100%; 57 | } 58 | 59 | /* Fix bug in bootstrap (only seen in firefox) */ 60 | summary { 61 | display: list-item; 62 | } 63 | 64 | /* Typographic tweaking ---------------------------------*/ 65 | 66 | .contents .page-header { 67 | margin-top: calc(-60px + 1em); 68 | } 69 | 70 | /* Section anchors ---------------------------------*/ 71 | 72 | a.anchor { 73 | margin-left: -30px; 74 | display:inline-block; 75 | width: 30px; 76 | height: 30px; 77 | visibility: hidden; 78 | 79 | background-image: url(./link.svg); 80 | background-repeat: no-repeat; 81 | background-size: 20px 20px; 82 | background-position: center center; 83 | } 84 | 85 | .hasAnchor:hover a.anchor { 86 | visibility: visible; 87 | } 88 | 89 | @media (max-width: 767px) { 90 | .hasAnchor:hover a.anchor { 91 | visibility: hidden; 92 | } 93 | } 94 | 95 | 96 | /* Fixes for fixed navbar --------------------------*/ 97 | 98 | .contents h1, .contents h2, .contents h3, .contents h4 { 99 | padding-top: 60px; 100 | margin-top: -40px; 101 | } 102 | 103 | /* Sidebar --------------------------*/ 104 | 105 | #sidebar { 106 | margin-top: 30px; 107 | position: -webkit-sticky; 108 | position: sticky; 109 | top: 70px; 110 | } 111 | #sidebar h2 { 112 | font-size: 1.5em; 113 | margin-top: 1em; 114 | } 115 | 116 | #sidebar h2:first-child { 117 | margin-top: 0; 118 | } 119 | 120 | #sidebar .list-unstyled li { 121 | margin-bottom: 0.5em; 122 | } 123 | 124 | .orcid { 125 | height: 16px; 126 | /* margins are required by official ORCID trademark and display guidelines */ 127 | margin-left:4px; 128 | margin-right:4px; 129 | vertical-align: middle; 130 | } 131 | 132 | /* Reference index & topics ----------------------------------------------- */ 133 | 134 | .ref-index th {font-weight: normal;} 135 | 136 | .ref-index td {vertical-align: top;} 137 | .ref-index .icon {width: 40px;} 138 | .ref-index .alias {width: 40%;} 139 | .ref-index-icons .alias {width: calc(40% - 40px);} 140 | .ref-index .title {width: 60%;} 141 | 142 | .ref-arguments th {text-align: right; padding-right: 10px;} 143 | .ref-arguments th, .ref-arguments td {vertical-align: top;} 144 | .ref-arguments .name {width: 20%;} 145 | .ref-arguments .desc {width: 80%;} 146 | 147 | /* Nice scrolling for wide elements --------------------------------------- */ 148 | 149 | table { 150 | display: block; 151 | overflow: auto; 152 | } 153 | 154 | /* Syntax highlighting ---------------------------------------------------- */ 155 | 156 | pre { 157 | word-wrap: normal; 158 | word-break: normal; 159 | border: 1px solid #eee; 160 | } 161 | 162 | pre, code { 163 | background-color: #f8f8f8; 164 | color: #333; 165 | } 166 | 167 | pre code { 168 | overflow: auto; 169 | word-wrap: normal; 170 | white-space: pre; 171 | } 172 | 173 | pre .img { 174 | margin: 5px 0; 175 | } 176 | 177 | pre .img img { 178 | background-color: #fff; 179 | display: block; 180 | height: auto; 181 | } 182 | 183 | code a, pre a { 184 | color: #375f84; 185 | } 186 | 187 | a.sourceLine:hover { 188 | text-decoration: none; 189 | } 190 | 191 | .fl {color: #1514b5;} 192 | .fu {color: #000000;} /* function */ 193 | .ch,.st {color: #036a07;} /* string */ 194 | .kw {color: #264D66;} /* keyword */ 195 | .co {color: #888888;} /* comment */ 196 | 197 | .message { color: black; font-weight: bolder;} 198 | .error { color: orange; font-weight: bolder;} 199 | .warning { color: #6A0366; font-weight: bolder;} 200 | 201 | /* Clipboard --------------------------*/ 202 | 203 | .hasCopyButton { 204 | position: relative; 205 | } 206 | 207 | .btn-copy-ex { 208 | position: absolute; 209 | right: 0; 210 | top: 0; 211 | visibility: hidden; 212 | } 213 | 214 | .hasCopyButton:hover button.btn-copy-ex { 215 | visibility: visible; 216 | } 217 | 218 | /* headroom.js ------------------------ */ 219 | 220 | .headroom { 221 | will-change: transform; 222 | transition: transform 200ms linear; 223 | } 224 | .headroom--pinned { 225 | transform: translateY(0%); 226 | } 227 | .headroom--unpinned { 228 | transform: translateY(-100%); 229 | } 230 | 231 | /* mark.js ----------------------------*/ 232 | 233 | mark { 234 | background-color: rgba(255, 255, 51, 0.5); 235 | border-bottom: 2px solid rgba(255, 153, 51, 0.3); 236 | padding: 1px; 237 | } 238 | 239 | /* vertical spacing after htmlwidgets */ 240 | .html-widget { 241 | margin-bottom: 10px; 242 | } 243 | 244 | /* fontawesome ------------------------ */ 245 | 246 | .fab { 247 | font-family: "Font Awesome 5 Brands" !important; 248 | } 249 | 250 | /* don't display links in code chunks when printing */ 251 | /* source: https://stackoverflow.com/a/10781533 */ 252 | @media print { 253 | code a:link:after, code a:visited:after { 254 | content: ""; 255 | } 256 | } 257 | -------------------------------------------------------------------------------- /man/autopar.Rd: -------------------------------------------------------------------------------- 1 | \name{autopar} 2 | \alias{autopar} 3 | \title{parameters for automatic hyperparameters optimization} 4 | \description{ 5 | List of parameters to allow multi deep neural network automatic hyperparameters tuning with Particle Swarm Optimization\cr 6 | Not mandatory (the list is preset and all arguments are initialized with default value) but it is advisable to adjust some important arguments for performance reasons (including processing time) 7 | } 8 | \arguments{ 9 | \item{psopartpopsize}{ number of particles in swarm, the main argument that should be tuned (default value 8, which is quite low)\cr 10 | #tuning priority 1} 11 | 12 | \item{psoxxx}{ see \link{pso} for other PSO specific arguments details} 13 | 14 | \item{numiterations}{ number of convergence steps between particles (hyperparameters), default value 3)\cr 15 | #tuning priority 1} 16 | 17 | \item{auto_modexec}{ if \sQuote{TRUE} the type of Neural Net optimization will be randomly choosen between \sQuote{trainwgrad} and \sQuote{trainwpso} for each particle\cr 18 | default value is \sQuote{FALSE} (so default value of argument \sQuote{modexec} in \link{automl_train_manual} function, actually \sQuote{trainwgrad} as default is more suited to large data volume)\cr 19 | the value can be forced if defined in \link{hpar} list} 20 | 21 | \item{auto_runtype}{ if \sQuote{2steps} the 2 following steps will be run automatically (default value is \sQuote{normal}):\cr 22 | 1st overfitting, the goal is performance\cr 23 | 2nd regularization, the goal is generalization \cr 24 | nb: \sQuote{overfitting} or \sQuote{regularization} may be directly specified to avoid the 2 steps\cr} 25 | 26 | \item{auto_minibatchsize}{ see below} 27 | \item{auto_minibatchsize_min}{ see below} 28 | \item{auto_minibatchsize_max}{ \sQuote{auto_minibatch} default value \sQuote{TRUE} for automatic adjustment of \sQuote{minibatchsize} argument in \link{automl_train_manual} function\cr 29 | the minimum and maximum value for \sQuote{minibatchsize} corespond to 2 to the power value (default 0 for \sQuote{auto_minibatchsize_min} and 9 for \sQuote{auto_minibatchsize_max})} 30 | 31 | \item{auto_learningrate}{ see below} 32 | \item{auto_learningrate_min}{ see below} 33 | \item{auto_learningrate_max}{ \sQuote{auto_learningrate} default value \sQuote{TRUE} for automatic adjustment of \sQuote{learningrate} argument in \link{automl_train_manual} function\cr 34 | the minimum and maximum value for \sQuote{learningrate} correspond to 10 to the power negative value (default -5 for \sQuote{auto_learningrate_min} and -2 for \sQuote{auto_learningrate_max})} 35 | 36 | \item{auto_beta1}{ see below} 37 | \item{auto_beta2}{ \sQuote{auto_beta1} and \sQuote{auto_beta2} default value \sQuote{TRUE} for automatic adjustment of \sQuote{beta1} and \sQuote{beta2} argument in \link{automl_train_manual} function} 38 | 39 | \item{auto_psopartpopsize}{ see below} 40 | \item{auto_psopartpopsize_min}{ see below} 41 | \item{auto_psopartpopsize_max}{ \sQuote{auto_psopartpopsize} default value \sQuote{TRUE} for automatic adjustment of \sQuote{psopartpopsize} argument in \link{automl_train_manual} function (concern only \sQuote{modexec} set to \sQuote{trainwpso})\cr 42 | the minimum and maximum value for \sQuote{psopartpopsize} ; default 2 for \sQuote{auto_psopartpopsize_min} and 50 for \sQuote{auto_psopartpopsize_max})} 43 | 44 | \item{auto_lambda}{ see below} 45 | \item{auto_lambda_min}{ see below} 46 | \item{auto_lambda_max}{ \sQuote{auto_lambda} default value \sQuote{FALSE} for automatic adjustment of \sQuote{lambda} regularization argument in \link{automl_train_manual} function\cr 47 | the minimum and maximum value for \sQuote{lambda} correspond to 10 to the power value (default -2) for \sQuote{auto_lambda_min} and (default 4) for \sQuote{auto_lambda_max})} 48 | 49 | \item{auto_psovelocitymaxratio}{ see below} 50 | \item{auto_psovelocitymaxratio_min}{ see below} 51 | \item{auto_psovelocitymaxratio_max}{ \sQuote{auto_psovelocitymaxratio} default value \sQuote{TRUE} for automatic adjustment of \sQuote{psovelocitymaxratio} PSO velocity max ratio argument in \link{automl_train_manual} function\cr 52 | the minimum and maximum value for \sQuote{psovelocitymaxratio}; default 0.01 for \sQuote{auto_psovelocitymaxratio_min} and 0.5 for \sQuote{auto_psovelocitymaxratio_max}} 53 | 54 | \item{auto_layers}{ see below (\sQuote{auto_layers} default value \sQuote{TRUE} for automatic adjustment of layers shape in \link{automl_train_manual} function)} 55 | \item{auto_layers_min}{ (linked to \sQuote{auto_layers} above, set \link{hpar} \sQuote{layersshape} and \sQuote{layersacttype}) the minimum number of hidden layers (default 1 no hidden layer)} 56 | \item{auto_layers_max}{ (linked to \sQuote{auto_layers} above, set \link{hpar} \sQuote{layersshape} and \sQuote{layersacttype}) the maximum number of hidden layers (default 2)} 57 | \item{auto_layersnodes_min}{ (linked to \sQuote{auto_layers} above, set \link{hpar} \sQuote{layersshape} and \sQuote{layersacttype}) the minimum number of nodes per layer (default 3)} 58 | \item{auto_layersnodes_max}{ (linked to \sQuote{auto_layers} above, set \link{hpar} \sQuote{layersshape} and \sQuote{layersacttype}) the maximum number of nodes per layer (default 33)} 59 | 60 | \item{auto_layersdropo}{ see below} 61 | \item{auto_layersdropoprob_min}{ see below} 62 | \item{auto_layersdropoprob_max}{ \sQuote{auto_layersdropo} default value \sQuote{FALSE} for automatic adjustment of \link{hpar} \sQuote{layersdropoprob} in \link{automl_train_manual} function)\cr 63 | the minimum and maximum value for \sQuote{layersdropoprob}; default 0.05 for \sQuote{auto_layersdropoprob_min} and 0.75 for \sQuote{auto_layersdropoprob_max}} 64 | 65 | \item{seed}{ seed for reproductibility (default 4)} 66 | 67 | \item{nbcores}{ number of cores used to parallelize particles optimization, not available on Windows (default 1, automatically reduced if not enough cores)} 68 | 69 | \item{verbose}{ to display or not the costs at each iteration for each particle (default TRUE)}\cr 70 | 71 | \item{subtimelimit}{ time limit in seconds for sub modelizations to avoid waiting too long for a specific particle to finish its modelization (default 3600)}\cr 72 | 73 | \emph{back to \link{automl_train}} 74 | } 75 | -------------------------------------------------------------------------------- /docs/index.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | Deep Learning with Metaheuristic • automl 9 | 10 | 11 | 12 | 13 | 14 | 18 | 19 | 23 | 24 | 25 |
    26 |
    69 | 70 | 71 | 72 | 73 |
    74 |
    75 | 76 |

    automl package fits from simple regression to highly customizable deep neural networks either with gradient descent or metaheuristic, using automatic hyper parameters tuning and custom cost function. A mix inspired by the common tricks on Deep Learning and Particle Swarm Optimization.

    77 |

    (Key words: autoML, Deep Learning, Particle Swarm Optimization, learning rate, minibatch, batch normalization, lambda, RMSprop, momentum, adam optimization, learning rate decay, inverted dropout, particles number, kappa, regression, logistic regression)

    78 | 79 |
    80 | 81 | 107 |
    108 | 109 | 110 |
    111 |

    Developed by Alex Boulangé.

    112 |
    113 | 114 |
    115 |

    Site built with pkgdown 1.4.0.

    116 |
    117 | 118 |
    119 |
    120 | 121 | 122 | 123 | 124 | 125 | 126 | -------------------------------------------------------------------------------- /docs/reference/index.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | Function reference • automl 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 51 | 52 | 53 | 54 | 55 | 56 | 57 |
    58 |
    59 | 101 | 102 | 103 | 104 |
    105 | 106 |
    107 |
    108 |
    109 |

    Reference

    110 |
    111 | 112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | 120 | 121 | 122 | 126 | 127 | 128 | 129 | 132 | 133 | 134 | 135 | 138 | 139 | 140 | 141 | 144 | 145 | 146 | 147 | 150 | 151 | 152 | 153 | 156 | 157 | 158 | 159 | 162 | 163 | 164 | 165 |
    123 |

    All functions

    124 |

    125 |
    130 |

    automl_predict()

    131 |

    automl_predict

    136 |

    automl_train()

    137 |

    automl_train

    142 |

    automl_train_manual()

    143 |

    automl_train_manual

    148 |

    autopar

    149 |

    parameters for automatic hyperparameters optimization

    154 |

    hpar

    155 |

    Deep Neural Net parameters and hyperparameters

    160 |

    pso

    161 |

    PSO parameters and hyperparameters

    166 |
    167 | 168 | 174 |
    175 | 176 | 177 |
    178 |
    179 |

    Developed by Alex Boulangé.

    180 |
    181 | 182 |
    183 |

    Site built with pkgdown 1.4.0.

    184 |
    185 | 186 |
    187 |
    188 | 189 | 190 | 191 | 192 | 193 | 194 | 195 | 196 | -------------------------------------------------------------------------------- /docs/reference/pso.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | PSO parameters and hyperparameters — pso • automl 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 53 | 54 | 55 | 56 | 57 | 58 | 59 |
    60 |
    61 | 103 | 104 | 105 | 106 |
    107 | 108 |
    109 |
    110 |
    111 |

    PSO parameters and hyperparameters

    112 | 113 | 114 |
    115 | 116 |
    117 |

    List of parameters and hyperparameters for Particle Swarm Optimization

    118 |
    119 | 120 | 121 |

    Arguments

    122 | 123 | 124 | 125 | 126 | 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 138 | 139 | 140 | 141 | 142 | 143 | 144 | 145 | 146 | 147 | 148 | 149 | 150 | 151 | 152 | 153 | 154 | 155 | 156 | 157 | 158 | 159 |
    psopartpopsize

    number of particles in swarm (discrete value)
    127 | (‘autopar’ context: default value 8, which means that 8 different neural net hyperparameters sets will be tested
    128 | (‘hpar’ context: default value 50, which means that 50 neural net weights sets will be tested
    129 | #tuning priority 1

    psovarvalmin

    Minimum value for particles positions (default value -10)

    psovarvalmax

    maximum value for particles positions (default value 10)

    psovelocitymaxratio

    ratio applied to limit velocities (continuous value between 0 and 1, default value 0.2)

    psoinertiadampratio

    inertia damp ratio (continuous value between 0 and 1, default value 1 equivalent to OFF)

    psokappa

    kappa (default value 1)

    psophi1

    Phi 1 (default value 2.05)

    psophi2

    Phi 2 (default value 2.05)

    160 | 161 |

    See also

    162 | 163 |

    PSO video tutorial from Yarpiz

    164 | 165 |
    166 | 174 |
    175 | 176 | 177 |
    178 |
    179 |

    Developed by Alex Boulangé.

    180 |
    181 | 182 |
    183 |

    Site built with pkgdown 1.4.0.

    184 |
    185 | 186 |
    187 |
    188 | 189 | 190 | 191 | 192 | 193 | 194 | 195 | 196 | -------------------------------------------------------------------------------- /docs/docsearch.css: -------------------------------------------------------------------------------- 1 | /* Docsearch -------------------------------------------------------------- */ 2 | /* 3 | Source: https://github.com/algolia/docsearch/ 4 | License: MIT 5 | */ 6 | 7 | .algolia-autocomplete { 8 | display: block; 9 | -webkit-box-flex: 1; 10 | -ms-flex: 1; 11 | flex: 1 12 | } 13 | 14 | .algolia-autocomplete .ds-dropdown-menu { 15 | width: 100%; 16 | min-width: none; 17 | max-width: none; 18 | padding: .75rem 0; 19 | background-color: #fff; 20 | background-clip: padding-box; 21 | border: 1px solid rgba(0, 0, 0, .1); 22 | box-shadow: 0 .5rem 1rem rgba(0, 0, 0, .175); 23 | } 24 | 25 | @media (min-width:768px) { 26 | .algolia-autocomplete .ds-dropdown-menu { 27 | width: 175% 28 | } 29 | } 30 | 31 | .algolia-autocomplete .ds-dropdown-menu::before { 32 | display: none 33 | } 34 | 35 | .algolia-autocomplete .ds-dropdown-menu [class^=ds-dataset-] { 36 | padding: 0; 37 | background-color: rgb(255,255,255); 38 | border: 0; 39 | max-height: 80vh; 40 | } 41 | 42 | .algolia-autocomplete .ds-dropdown-menu .ds-suggestions { 43 | margin-top: 0 44 | } 45 | 46 | .algolia-autocomplete .algolia-docsearch-suggestion { 47 | padding: 0; 48 | overflow: visible 49 | } 50 | 51 | .algolia-autocomplete .algolia-docsearch-suggestion--category-header { 52 | padding: .125rem 1rem; 53 | margin-top: 0; 54 | font-size: 1.3em; 55 | font-weight: 500; 56 | color: #00008B; 57 | border-bottom: 0 58 | } 59 | 60 | .algolia-autocomplete .algolia-docsearch-suggestion--wrapper { 61 | float: none; 62 | padding-top: 0 63 | } 64 | 65 | .algolia-autocomplete .algolia-docsearch-suggestion--subcategory-column { 66 | float: none; 67 | width: auto; 68 | padding: 0; 69 | text-align: left 70 | } 71 | 72 | .algolia-autocomplete .algolia-docsearch-suggestion--content { 73 | float: none; 74 | width: auto; 75 | padding: 0 76 | } 77 | 78 | .algolia-autocomplete .algolia-docsearch-suggestion--content::before { 79 | display: none 80 | } 81 | 82 | .algolia-autocomplete .ds-suggestion:not(:first-child) .algolia-docsearch-suggestion--category-header { 83 | padding-top: .75rem; 84 | margin-top: .75rem; 85 | border-top: 1px solid rgba(0, 0, 0, .1) 86 | } 87 | 88 | .algolia-autocomplete .ds-suggestion .algolia-docsearch-suggestion--subcategory-column { 89 | display: block; 90 | padding: .1rem 1rem; 91 | margin-bottom: 0.1; 92 | font-size: 1.0em; 93 | font-weight: 400 94 | /* display: none */ 95 | } 96 | 97 | .algolia-autocomplete .algolia-docsearch-suggestion--title { 98 | display: block; 99 | padding: .25rem 1rem; 100 | margin-bottom: 0; 101 | font-size: 0.9em; 102 | font-weight: 400 103 | } 104 | 105 | .algolia-autocomplete .algolia-docsearch-suggestion--text { 106 | padding: 0 1rem .5rem; 107 | margin-top: -.25rem; 108 | font-size: 0.8em; 109 | font-weight: 400; 110 | line-height: 1.25 111 | } 112 | 113 | .algolia-autocomplete .algolia-docsearch-footer { 114 | width: 110px; 115 | height: 20px; 116 | z-index: 3; 117 | margin-top: 10.66667px; 118 | float: right; 119 | font-size: 0; 120 | line-height: 0; 121 | } 122 | 123 | .algolia-autocomplete .algolia-docsearch-footer--logo { 124 | background-image: url("data:image/svg+xml;utf8,"); 125 | background-repeat: no-repeat; 126 | background-position: 50%; 127 | background-size: 100%; 128 | overflow: hidden; 129 | text-indent: -9000px; 130 | width: 100%; 131 | height: 100%; 132 | display: block; 133 | transform: translate(-8px); 134 | } 135 | 136 | .algolia-autocomplete .algolia-docsearch-suggestion--highlight { 137 | color: #FF8C00; 138 | background: rgba(232, 189, 54, 0.1) 139 | } 140 | 141 | 142 | .algolia-autocomplete .algolia-docsearch-suggestion--text .algolia-docsearch-suggestion--highlight { 143 | box-shadow: inset 0 -2px 0 0 rgba(105, 105, 105, .5) 144 | } 145 | 146 | .algolia-autocomplete .ds-suggestion.ds-cursor .algolia-docsearch-suggestion--content { 147 | background-color: rgba(192, 192, 192, .15) 148 | } 149 | -------------------------------------------------------------------------------- /docs/reference/automl_train.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | automl_train — automl_train • automl 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 55 | 56 | 57 | 58 | 59 | 60 | 61 |
    62 |
    63 | 105 | 106 | 107 | 108 |
    109 | 110 |
    111 |
    112 |
    113 |

    automl_train

    114 | 115 | 116 |
    117 | 118 |
    119 |

    The multi deep neural network automatic train function (several deep neural networks are trained with automatic hyperparameters tuning, best model is kept)
    120 | This function launches the automl_train_manual function by passing it parameters 121 | for each particle at each converging step

    122 |
    123 | 124 | 
    automl_train(Xref, Yref, autopar = list(), hpar = list(), mdlref = NULL)
    125 | 126 |

    Arguments

    127 | 128 | 129 | 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 138 | 139 | 141 | 142 | 143 | 144 | 146 | 147 | 148 | 149 | 152 | 153 |
    Xref

    inputs matrix or data.frame (containing numerical values only)

    Yref

    target matrix or data.frame (containing numerical values only)

    autopar

    list of parameters for hyperparameters optimization, see autopar section
    140 | Not mandatory (the list is preset and all arguments are initialized with default value) but it is advisable to adjust some important arguments for performance reasons (including processing time)

    hpar

    list of parameters and hyperparameters for Deep Neural Network, see hpar section
    145 | Not mandatory (the list is preset and all arguments are initialized with default value) but it is advisable to adjust some important arguments for performance reasons (including processing time)

    mdlref

    model trained with automl_train to start training with saved hpar and autopar 150 | (not the model)
    151 | nb: manually entered parameters above override loaded ones

    154 | 155 | 156 |

    Examples

    157 | 
    if (FALSE) {
158 | ##REGRESSION (predict Sepal.Length given other Iris parameters)
159 | data(iris)
160 | xmat <- cbind(iris[,2:4], as.numeric(iris$Species))
161 | ymat <- iris[,1]
162 | amlmodel <- automl_train(Xref = xmat, Yref = ymat)
163 | }
164 | ##CLASSIFICATION (predict Species given other Iris parameters)
165 | data(iris)
166 | xmat = iris[,1:4]
167 | lab2pred <- levels(iris$Species)
168 | lghlab <- length(lab2pred)
169 | iris$Species <- as.numeric(iris$Species)
170 | ymat <- matrix(seq(from = 1, to = lghlab, by = 1), nrow(xmat), lghlab, byrow = TRUE)
171 | ymat <- (ymat == as.numeric(iris$Species)) + 0
172 | #with gradient descent and random hyperparameters sets
173 | amlmodel <- automl_train(Xref = xmat, Yref = ymat,
174 |                           autopar = list(numiterations = 1, psopartpopsize = 1, seed = 11),
175 |                           hpar = list(numiterations = 10))
    #> (cost: crossentropy)
176 | #> iteration 1 particle 1 weighted err: 2.08682 (train: 2.06276 cvalid: 2.00259 ) BEST MODEL KEPT
    177 |
    178 | 186 |
    187 | 188 | 189 |
    190 |
    191 |

    Developed by Alex Boulangé.

    192 |
    193 | 194 |
    195 |

    Site built with pkgdown 1.4.0.

    196 |
    197 | 198 |
    199 |
    200 | 201 | 202 | 203 | 204 | 205 | 206 | 207 | 208 | -------------------------------------------------------------------------------- /docs/reference/hpar.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | Deep Neural Net parameters and hyperparameters — hpar • automl 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 54 | 55 | 56 | 57 | 58 | 59 | 60 |
    61 |
    62 | 104 | 105 | 106 | 107 |
    108 | 109 |
    110 |
    111 |
    112 |

    Deep Neural Net parameters and hyperparameters

    113 | 114 | 115 |
    116 | 117 |
    118 |

    List of Neural Network parameters and hyperparameters to train with gradient descent or particle swarm optimization
    119 | Not mandatory (the list is preset and all arguments are initialized with default value) but it is advisable to adjust some important arguments for performance reasons (including processing time)

    120 |
    121 | 122 | 123 |

    Arguments

    124 | 125 | 126 | 127 | 128 | 130 | 131 | 132 | 133 | 135 | 136 | 137 | 138 | 139 | 140 | 141 | 142 | 147 | 148 | 149 | 150 | 152 | 153 | 154 | 155 | 156 | 157 | 158 | 159 | 160 | 161 | 162 | 163 | 164 | 165 | 166 | 167 | 173 | 174 | 175 | 176 | 178 | 179 | 180 | 181 | 182 | 183 | 184 | 185 | 187 | 188 | 189 | 190 | 194 | 195 | 196 | 197 | 202 | 203 | 204 | 205 | 209 | 210 | 211 | 212 | 213 | 214 | 215 | 216 | 217 | 218 | 219 | 220 | 221 | 222 | 223 | 224 | 228 | 229 | 230 | 231 | 232 | 233 | 234 | 235 | 236 | 237 | 238 | 239 | 240 | 241 | 242 | 243 | 244 | 245 |
    modexec

    ‘trainwgrad’ (the default value) to train with gradient descent (suitable for all volume of data)
    129 | ‘trainwpso’ to train using Particle Swarm Optimization, each particle represents a set of neural network weights (CAUTION: suitable for low volume of data, time consuming for medium to large volume of data)

    learningrate

    learningrate alpha (default value 0.001)
    134 | #tuning priority 1

    beta1

    see below

    beta2

    ‘Momentum’ if beta1 different from 0 and beta2 equal 0 )
    143 | ‘RMSprop’ if beta1 equal 0 and beta2 different from 0
    144 | ‘adam optimization’ if beta1 different from 0 and beta2 different from 0 (default)
    145 | (default value beta1 equal 0.9 and beta2 equal 0.999)
    146 | #tuning priority 2

    lrdecayrate

    learning rate decay value (default value 0, no learning rate decay, 1e-6 should be a good value to start with)
    151 | #tuning priority 4

    chkgradevery

    epoch interval to run gradient check function (default value 0, for debug only)

    chkgradepsilon

    epsilon value for derivative calculations and threshold test in gradient check function (default 0.0000001)

    psoxxx

    see pso for PSO specific arguments details

    costcustformul

    custom cost formula (default ‘’, no custom cost function)
    168 | standard input variables: yhat (prediction), y (target actual value)
    169 | custom input variables: any variable declared in hpar may be used via alias mydl (ie: hpar(list = (foo = 1.5)) will be used in custom cost formula as mydl$foo))
    170 | result: J
    171 | see ‘automl_train_manual’ example using Mean Average Percentage Error cost function
    172 | nb: X and Y matrices used as input into automl_train_manual or automl_train_manual functions are transposed (features in rows and cases in columns)

    numiterations

    number of training epochs (default value 50))
    177 | #tuning priority 1

    seed

    seed for reproductibility (default 4)

    minibatchsize

    mini batch size, 2 to the power 0 for stochastic gradient descent (default 2 to the power 5) 186 | #tuning priority 3

    layersshape

    number of nodes per layer, each nodes number initialize a hidden layer
    191 | output layer nodes number, may be left to 0 it will be automatically set by Y matrix shape
    192 | default value one hidden layer with 10 nodes: c(10, 0)
    193 | #tuning priority 4

    layersacttype

    activation function for each layer; ‘linear’ for no activation or ‘sigmoid’, ‘relu’ or ‘reluleaky’ or ‘tanh’ or ‘softmax’ (softmax for output layer only supported in trainwpso exec mode)
    198 | output layer activation function may be left to ‘’, default value ‘linear’ for regression, ‘sigmoid’ for classification
    199 | nb: layersacttype parameter vector must have same length as layersshape parameter vector
    200 | default value c(‘relu’, ‘’)
    201 | #tuning priority 4

    layersdropoprob

    drop out probability for each layer, continuous value from 0 to less than 1 (give the percentage of matrix weight values to drop out randomly)
    206 | nb: layersdropoprob parameter vector must have same length as layersshape parameter vector
    207 | default value no drop out: c(0, 0)
    208 | #tuning priority for regularization

    printcostevery

    epoch interval to test and print costs (train and cross validation cost: default value 10, for 1 test every 10 epochs)

    testcvsize

    size of cross validation sample, 0 for no cross validation sample (default 10, for 10 percent)

    testgainunder

    threshold to stop the training if the gain between last train or cross validation cost is smaller than the threshold, 0 for no stop test (default 0.000001)

    costtype

    cost type function name ‘mse’ or ‘crossentropy’ or ‘custom’
    225 | ‘mse’ for Mean Squared Error, set automatically for continuous target type (‘mape’ Mean Absolute Percentage Error may be specified)
    226 | ‘crossentropy’ set automatically for binary target type
    227 | ‘custom’ set automatically if ‘costcustformul’ different from ‘’

    lambda

    regularization term added to cost function (default value 0, no regularization)

    batchnor_mom

    batch normalization momentum for j and B (default 0, no batch normalization, may be set to 0.9 for deep neural net)

    epsil

    epsilon the low value to avoid dividing by 0 or log(0) in cost function, etc ... (default value 1e-12)

    verbose

    to display or not the costs and the shapes (default TRUE)

    246 | 247 |

    See also

    248 | 249 |

    Deep Learning specialization from Andrew NG on Coursera

    250 | 251 |
    252 | 260 |
    261 | 262 | 263 |
    264 |
    265 |

    Developed by Alex Boulangé.

    266 |
    267 | 268 |
    269 |

    Site built with pkgdown 1.4.0.

    270 |
    271 | 272 |
    273 |
    274 | 275 | 276 | 277 | 278 | 279 | 280 | 281 | 282 | -------------------------------------------------------------------------------- /docs/reference/automl_predict.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | automl_predict — automl_predict • automl 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 53 | 54 | 55 | 56 | 57 | 58 | 59 |
    60 |
    61 | 103 | 104 | 105 | 106 |
    107 | 108 |
    109 |
    110 |
    111 |

    automl_predict

    112 | 113 | 114 |
    115 | 116 |
    117 |

    Predictions function, to apply a trained model on datas

    118 |
    119 | 120 | 
    automl_predict(model, X, layoutputnum)
    121 | 122 |

    Arguments

    123 | 124 | 125 | 126 | 127 | 128 | 129 | 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 |
    model

    model trained previously with automl_train or automl_train_manual

    X

    inputs matrix or data.frame (containing numerical values only)

    layoutputnum

    which layer number to output especially for auto encoding (default 0: no particular layer, the last one)

    138 | 139 | 140 |

    Examples

    141 | 
    ##REGRESSION (predict Sepal.Length given other parameters)
142 | data(iris)
143 | xmat <- as.matrix(cbind(iris[,2:4], as.numeric(iris$Species)))
144 | ymat <- iris[,1]
145 | amlmodel <- automl_train_manual(Xref = xmat, Yref = ymat,
146 |  hpar = list(modexec = 'trainwpso', verbose = FALSE))
147 | res <- cbind(ymat, automl_predict(model = amlmodel, X = xmat))
148 | colnames(res) <- c('actual', 'predict')
149 | head(res)
    #>      actual  predict
150 | #> [1,]    5.1 5.113573
151 | #> [2,]    4.9 4.562844
152 | #> [3,]    4.7 4.715187
153 | #> [4,]    4.6 4.740938
154 | #> [5,]    5.0 5.223718
155 | #> [6,]    5.4 5.629219
    #
156 | if (FALSE) {
157 | ##CLASSIFICATION (predict Species given other Iris parameters)
158 | data(iris)
159 | xmat = iris[,1:4]
160 | lab2pred <- levels(iris$Species)
161 | lghlab <- length(lab2pred)
162 | iris$Species <- as.numeric(iris$Species)
163 | ymat <- matrix(seq(from = 1, to = lghlab, by = 1), nrow(xmat),
164 |  lghlab, byrow = TRUE)
165 | ymat <- (ymat == as.numeric(iris$Species)) + 0
166 | amlmodel <- automl_train_manual(Xref = xmat, Yref = ymat,
167 |  hpar = list(modexec = 'trainwpso', verbose = FALSE))
168 | res <- cbind(ymat, round(automl_predict(model = amlmodel, X = xmat)))
169 | colnames(res) <- c(paste('act',lab2pred, sep = '_'),
170 |  paste('pred',lab2pred, sep = '_'))
171 | head(res)
172 | }
    173 |
    174 | 182 |
    183 | 184 | 185 |
    186 |
    187 |

    Developed by Alex Boulangé.

    188 |
    189 | 190 |
    191 |

    Site built with pkgdown 1.4.0.

    192 |
    193 | 194 |
    195 |
    196 | 197 | 198 | 199 | 200 | 201 | 202 | 203 | 204 | -------------------------------------------------------------------------------- /docs/reference/autopar.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | parameters for automatic hyperparameters optimization — autopar • automl 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 54 | 55 | 56 | 57 | 58 | 59 | 60 |
    61 |
    62 | 104 | 105 | 106 | 107 |
    108 | 109 |
    110 |
    111 |
    112 |

    parameters for automatic hyperparameters optimization

    113 | 114 | 115 |
    116 | 117 |
    118 |

    List of parameters to allow multi deep neural network automatic hyperparameters tuning with Particle Swarm Optimization
    119 | Not mandatory (the list is preset and all arguments are initialized with default value) but it is advisable to adjust some important arguments for performance reasons (including processing time)

    120 |
    121 | 122 | 123 |

    Arguments

    124 | 125 | 126 | 127 | 128 | 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 139 | 140 | 141 | 142 | 145 | 146 | 147 | 148 | 152 | 153 | 154 | 155 | 156 | 157 | 158 | 159 | 160 | 161 | 162 | 163 | 165 | 166 | 167 | 168 | 169 | 170 | 171 | 172 | 173 | 174 | 175 | 176 | 178 | 179 | 180 | 181 | 182 | 183 | 184 | 185 | 186 | 187 | 188 | 189 | 190 | 191 | 192 | 193 | 194 | 195 | 196 | 197 | 199 | 200 | 201 | 202 | 203 | 204 | 205 | 206 | 207 | 208 | 209 | 210 | 212 | 213 | 214 | 215 | 216 | 217 | 218 | 219 | 220 | 221 | 222 | 223 | 225 | 226 | 227 | 228 | 229 | 230 | 231 | 232 | 233 | 234 | 235 | 236 | 237 | 238 | 239 | 240 | 241 | 242 | 243 | 244 | 245 | 246 | 247 | 248 | 249 | 250 | 251 | 252 | 253 | 254 | 255 | 256 | 258 | 259 | 260 | 261 | 262 | 263 | 264 | 265 | 266 | 267 | 268 | 269 | 270 | 271 | 272 | 273 | 274 | 275 |
    psopartpopsize

    number of particles in swarm, the main argument that should be tuned (default value 8, which is quite low)
    129 | #tuning priority 1

    psoxxx

    see pso for other PSO specific arguments details

    numiterations

    number of convergence steps between particles (hyperparameters), default value 3)
    138 | #tuning priority 1

    auto_modexec

    if ‘TRUE’ the type of Neural Net optimization will be randomly choosen between ‘trainwgrad’ and ‘trainwpso’ for each particle
    143 | default value is ‘FALSE’ (so default value of argument ‘modexec’ in automl_train_manual function, actually ‘trainwgrad’ as default is more suited to large data volume)
    144 | the value can be forced if defined in hpar list

    auto_runtype

    if ‘2steps’ the 2 following steps will be run automatically (default value is ‘normal’):
    149 | 1st overfitting, the goal is performance
    150 | 2nd regularization, the goal is generalization
    151 | nb: ‘overfitting’ or ‘regularization’ may be directly specified to avoid the 2 steps

    auto_minibatchsize

    see below

    auto_minibatchsize_min

    see below

    auto_minibatchsize_max

    ‘auto_minibatch’ default value ‘TRUE’ for automatic adjustment of ‘minibatchsize’ argument in automl_train_manual function
    164 | the minimum and maximum value for ‘minibatchsize’ corespond to 2 to the power value (default 0 for ‘auto_minibatchsize_min’ and 9 for ‘auto_minibatchsize_max’)

    auto_learningrate

    see below

    auto_learningrate_min

    see below

    auto_learningrate_max

    ‘auto_learningrate’ default value ‘TRUE’ for automatic adjustment of ‘learningrate’ argument in automl_train_manual function
    177 | the minimum and maximum value for ‘learningrate’ correspond to 10 to the power negative value (default -5 for ‘auto_learningrate_min’ and -2 for ‘auto_learningrate_max’)

    auto_beta1

    see below

    auto_beta2

    ‘auto_beta1’ and ‘auto_beta2’ default value ‘TRUE’ for automatic adjustment of ‘beta1’ and ‘beta2’ argument in automl_train_manual function

    auto_psopartpopsize

    see below

    auto_psopartpopsize_min

    see below

    auto_psopartpopsize_max

    ‘auto_psopartpopsize’ default value ‘TRUE’ for automatic adjustment of ‘psopartpopsize’ argument in automl_train_manual function (concern only ‘modexec’ set to ‘trainwpso’)
    198 | the minimum and maximum value for ‘psopartpopsize’ ; default 2 for ‘auto_psopartpopsize_min’ and 50 for ‘auto_psopartpopsize_max’)

    auto_lambda

    see below

    auto_lambda_min

    see below

    auto_lambda_max

    ‘auto_lambda’ default value ‘FALSE’ for automatic adjustment of ‘lambda’ regularization argument in automl_train_manual function
    211 | the minimum and maximum value for ‘lambda’ correspond to 10 to the power value (default -2) for ‘auto_lambda_min’ and (default 4) for ‘auto_lambda_max’)

    auto_psovelocitymaxratio

    see below

    auto_psovelocitymaxratio_min

    see below

    auto_psovelocitymaxratio_max

    ‘auto_psovelocitymaxratio’ default value ‘TRUE’ for automatic adjustment of ‘psovelocitymaxratio’ PSO velocity max ratio argument in automl_train_manual function
    224 | the minimum and maximum value for ‘psovelocitymaxratio’; default 0.01 for ‘auto_psovelocitymaxratio_min’ and 0.5 for ‘auto_psovelocitymaxratio_max’

    auto_layers

    see below (‘auto_layers’ default value ‘TRUE’ for automatic adjustment of layers shape in automl_train_manual function)

    auto_layers_min

    (linked to ‘auto_layers’ above, set hpar ‘layersshape’ and ‘layersacttype’) the minimum number of hidden layers (default 1 no hidden layer)

    auto_layers_max

    (linked to ‘auto_layers’ above, set hpar ‘layersshape’ and ‘layersacttype’) the maximum number of hidden layers (default 2)

    auto_layersnodes_min

    (linked to ‘auto_layers’ above, set hpar ‘layersshape’ and ‘layersacttype’) the minimum number of nodes per layer (default 3)

    auto_layersnodes_max

    (linked to ‘auto_layers’ above, set hpar ‘layersshape’ and ‘layersacttype’) the maximum number of nodes per layer (default 33)

    auto_layersdropo

    see below

    auto_layersdropoprob_min

    see below

    auto_layersdropoprob_max

    ‘auto_layersdropo’ default value ‘FALSE’ for automatic adjustment of hpar ‘layersdropoprob’ in automl_train_manual function)
    257 | the minimum and maximum value for ‘layersdropoprob’; default 0.05 for ‘auto_layersdropoprob_min’ and 0.75 for ‘auto_layersdropoprob_max’

    seed

    seed for reproductibility (default 4)

    nbcores

    number of cores used to parallelize particles optimization, not available on Windows (default 1, automatically reduced if not enough cores)

    verbose

    to display or not the costs at each iteration for each particle (default TRUE)

    subtimelimit

    time limit in seconds for sub modelizations to avoid waiting too long for a specific particle to finish its modelization (default 3600)

    276 | 277 | 278 |
    279 | 286 |
    287 | 288 | 289 |
    290 |
    291 |

    Developed by Alex Boulangé.

    292 |
    293 | 294 |
    295 |

    Site built with pkgdown 1.4.0.

    296 |
    297 | 298 |
    299 |
    300 | 301 | 302 | 303 | 304 | 305 | 306 | 307 | 308 | -------------------------------------------------------------------------------- /vignettes/automl.Rmd: -------------------------------------------------------------------------------- 1 | --- 2 | title: "automl" 3 | author: "Alex Boulangé" 4 | date: "january 16, 2020" 5 | output: 6 | html_document: default 7 | pdf_document: default 8 | --- 9 | 10 | ```{r setup, include=FALSE} 11 | knitr::opts_chunk$set(echo = TRUE) 12 | require(automl) 13 | ``` 14 | 15 |

    Contents

    16 |
    17 |  0.1 Introduction to automl package
    18 |  0.2 Why & how automl
    19 |   0.2.1 Deep Learning existing frameworks, disadvantages
    20 |   0.2.2 Neural Network - Deep Learning, disadvantages
    21 |   0.2.3 Metaheuristic - PSO, benefits
    22 |   0.2.4 Birth of automl package
    23 |   0.2.5 Mix 1: hyperparameters tuning with PSO
    24 |   0.2.6 Mix 2: PSO instead of gradient descent
    25 |  0.3 First steps: How to
    26 |   0.3.1 fit a regression model manually (hard way)
    27 |   0.3.2 fit a regression model automatically (easy way, Mix 1)
    28 |   0.3.3 fit a regression model experimentally (experimental way, Mix 2)
    29 |   0.3.4 fit a regression model with custom cost (experimental way, Mix 2)
    30 |   0.3.5 fit a classification model with softmax (Mix 2)
    31 |   0.3.6 change the model parameters (shape ...)
    32 |   0.3.7 continue training on saved model (fine tuning ...)
    33 |   0.3.8 use the 2 steps automatic approach
    34 |  0.4 ToDo List 35 |
    36 |

    0.1 Introduction to automl package

    37 |

    This document is intended to answer the following questions; why & how automl and how to use it
    38 |
    39 | automl package provides:
    40 | -Deep Learning last tricks (those who have taken Andrew NG’s MOOC on Coursera will be in familiar territory)
    41 | -hyperparameters autotune with metaheuristic (PSO)
    42 | -experimental stuff and more to come (you’re welcome as coauthor!)
    43 |

    44 |

    0.2 Why & how automl

    45 |

    0.2.1 Deep Learning existing frameworks, disadvantages

    46 |

    Deploying and maintaining most Deep Learning frameworks means: Python...
    47 | R language is so simple to install and maintain in production environments that it is obvious to use a pure R based package for deep learning !
    48 |

    49 |

    0.2.2 Neural Network - Deep Learning, disadvantages

    50 |

    Disadvantages :
    51 | 1st disadvantage: you have to test manually different combinations of parameters (number of layers, nodes, activation function, etc ...) and then also tune manually hyper parameters for training (learning rate, momentum, mini batch size, etc ...)
    52 | 2nd disadvantage: only for those who are not mathematicians, calculating derivative in case of new cost or activation function, may by an issue.
    53 |
    54 | PIC
    55 |

    56 |

    0.2.3 Metaheuristic - PSO, benefits

    57 |

    The Particle Swarm Optimization algorithm is a great and simple one.
    58 | In a few words, the first step consists in throwing randomly a set of particles in a space and the next steps consist in discovering the best solution while converging.
    59 | PIC
    60 | video tutorial from Yarpiz is a great ressource
    61 |

    62 |

    0.2.4 Birth of automl package

    63 |

    automl package was born from the idea to use metaheuristic PSO to address the identified disadvantages above.
    64 | And last but not the least reason: use R and R only :-)
    65 | 3 functions are available:
    66 | - automl_train_manual: the manual mode to train a model
    67 | - automl_train: the automatic mode to train model
    68 | - automl_predict: the prediction function to apply a trained model on datas
    69 |

    70 |

    0.2.5 Mix 1: hyperparameters tuning with PSO

    71 |

    Mix 1 consists in using PSO algorithm to optimize the hyperparameters: each particle corresponds to a set of hyperparameters.
    72 | The automl_train function was made to do that.
    73 |
    74 | PIC
    75 |

    76 |

    0.2.6 Mix 2: PSO instead of gradient descent

    77 |

    Mix 2 is experimental, it consists in using PSO algorithm to optimize the weights of Neural Network in place of gradient descent: each particle corresponds to a set of neural network weights matrices.
    78 | The automl_train_manual function do that too.
    79 |
    80 | PIC
    81 |

    82 |

    0.3 First steps: How to

    83 |

    For those who will laugh at seeing deep learning with one hidden layer and the Iris data set of 150 records, I will say: you’re perfectly right :-)
    84 | The goal at this stage is simply to take the first steps

    85 | 86 | 87 |

    0.3.1 fit a regression model manually (hard way)

    88 |

    Subject: predict Sepal.Length given other Iris parameters
    89 | 1st with gradient descent and default hyperparameters value for learning rate (0.001) and mini batch size (32)
    90 |

    91 | 92 | ```{r} 93 | data(iris) 94 | xmat <- cbind(iris[,2:4], as.numeric(iris$Species)) 95 | ymat <- iris[,1] 96 | amlmodel <- automl_train_manual(Xref = xmat, Yref = ymat) 97 | ``` 98 | ```{r} 99 | res <- cbind(ymat, automl_predict(model = amlmodel, X = xmat)) 100 | colnames(res) <- c('actual', 'predict') 101 | head(res) 102 | ``` 103 | :-[] no pain, no gain ...
    104 | After some manual fine tuning on learning rate, mini batch size and iterations number (epochs):
    105 | ```{r} 106 | data(iris) 107 | xmat <- cbind(iris[,2:4], as.numeric(iris$Species)) 108 | ymat <- iris[,1] 109 | amlmodel <- automl_train_manual(Xref = xmat, Yref = ymat, 110 | hpar = list(learningrate = 0.01, 111 | minibatchsize = 2^2, 112 | numiterations = 30)) 113 | ``` 114 | ```{r} 115 | res <- cbind(ymat, automl_predict(model = amlmodel, X = xmat)) 116 | colnames(res) <- c('actual', 'predict') 117 | head(res) 118 | ``` 119 | Better result, but with human efforts!
    120 | 121 | 122 |

    0.3.2 fit a regression model automatically (easy way, Mix 1)

    123 | Same subject: predict Sepal.Length given other Iris parameters
    124 | ```{r} 125 | data(iris) 126 | xmat <- as.matrix(cbind(iris[,2:4], as.numeric(iris$Species))) 127 | ymat <- iris[,1] 128 | start.time <- Sys.time() 129 | amlmodel <- automl_train(Xref = xmat, Yref = ymat, 130 | autopar = list(psopartpopsize = 15, 131 | numiterations = 5, 132 | auto_layers_max = 1, 133 | nbcores = 4)) 134 | end.time <- Sys.time() 135 | cat(paste('time ellapsed:', end.time - start.time, '\n')) 136 | ``` 137 | ```{r} 138 | res <- cbind(ymat, automl_predict(model = amlmodel, X = xmat)) 139 | colnames(res) <- c('actual', 'predict') 140 | head(res) 141 | ``` 142 | It’s even better, with no human efforts but machine time
    143 | Windows users won’t benefit from parallelization, the function uses parallel package included with R base...
    144 | 145 | 146 |

    0.3.3 fit a regression model experimentally (experimental way, Mix 2)

    147 | Same subject: predict Sepal.Length given other Iris parameters
    148 | ```{r} 149 | data(iris) 150 | xmat <- as.matrix(cbind(iris[,2:4], as.numeric(iris$Species))) 151 | ymat <- iris[,1] 152 | amlmodel <- automl_train_manual(Xref = xmat, Yref = ymat, 153 | hpar = list(modexec = 'trainwpso', 154 | numiterations = 30, 155 | psopartpopsize = 50)) 156 | ``` 157 | ```{r} 158 | res <- cbind(ymat, automl_predict(model = amlmodel, X = xmat)) 159 | colnames(res) <- c('actual', 'predict') 160 | head(res) 161 | ``` 162 | Pretty good too, even better!
    163 | 164 |

    0.3.4 fit a regression model with custom cost (experimental way, Mix 2)

    165 | Same subject: predict Sepal.Length given other Iris parameters
    166 | Let’s try with Mean Absolute Percentage Error instead of Mean Square Error
    167 | ```{r} 168 | data(iris) 169 | xmat <- as.matrix(cbind(iris[,2:4], as.numeric(iris$Species))) 170 | ymat <- iris[,1] 171 | f <- 'J=abs((y-yhat)/y)' 172 | f <- c(f, 'J=sum(J[!is.infinite(J)],na.rm=TRUE)') 173 | f <- c(f, 'J=(J/length(y))') 174 | f <- paste(f, collapse = ';') 175 | amlmodel <- automl_train_manual(Xref = xmat, Yref = ymat, 176 | hpar = list(modexec = 'trainwpso', 177 | numiterations = 30, 178 | psopartpopsize = 50, 179 | costcustformul = f)) 180 | ``` 181 | ```{r} 182 | res <- cbind(ymat, automl_predict(model = amlmodel, X = xmat)) 183 | colnames(res) <- c('actual', 'predict') 184 | head(res) 185 | ``` 186 | 187 |

    0.3.5 fit a classification model with softmax (Mix 2)

    188 | Subject: predict Species given other Iris parameters
    189 | Softmax is available with PSO, no derivative needed ;-)
    190 | ```{r} 191 | data(iris) 192 | xmat = iris[,1:4] 193 | lab2pred <- levels(iris$Species) 194 | lghlab <- length(lab2pred) 195 | iris$Species <- as.numeric(iris$Species) 196 | ymat <- matrix(seq(from = 1, to = lghlab, by = 1), nrow(xmat), lghlab, byrow = TRUE) 197 | ymat <- (ymat == as.numeric(iris$Species)) + 0 198 | amlmodel <- automl_train_manual(Xref = xmat, Yref = ymat, 199 | hpar = list(modexec = 'trainwpso', 200 | layersshape = c(10, 0), 201 | layersacttype = c('relu', 'softmax'), 202 | layersdropoprob = c(0, 0), 203 | numiterations = 50, 204 | psopartpopsize = 50)) 205 | ``` 206 | ```{r} 207 | res <- cbind(ymat, automl_predict(model = amlmodel, X = xmat)) 208 | colnames(res) <- c(paste('act',lab2pred, sep = '_'), 209 | paste('pred',lab2pred, sep = '_')) 210 | head(res) 211 | tail(res) 212 | ``` 213 | 214 |

    0.3.6 change the model parameters (shape ...)

    215 | Same subject: predict Species given other Iris parameters
    216 | 1st example: with gradient descent and 2 hidden layers containing 10 nodes, with various activation functions for hidden layers
    217 | ```{r} 218 | data(iris) 219 | xmat = iris[,1:4] 220 | lab2pred <- levels(iris$Species) 221 | lghlab <- length(lab2pred) 222 | iris$Species <- as.numeric(iris$Species) 223 | ymat <- matrix(seq(from = 1, to = lghlab, by = 1), nrow(xmat), lghlab, byrow = TRUE) 224 | ymat <- (ymat == as.numeric(iris$Species)) + 0 225 | amlmodel <- automl_train_manual( 226 | Xref = xmat, Yref = ymat, 227 | hpar = list( 228 | layersshape = c(10, 10, 0), 229 | layersacttype = c('tanh', 'relu', ''), 230 | layersdropoprob = c(0, 0, 0))) 231 | ``` 232 | nb: last activation type may be left to blank (it will be set automatically)
    233 |
    234 | 2nd example: with gradient descent and no hidden layer (logistic regression)
    235 | ```{r} 236 | data(iris) 237 | xmat = iris[,1:4] 238 | lab2pred <- levels(iris$Species) 239 | lghlab <- length(lab2pred) 240 | iris$Species <- as.numeric(iris$Species) 241 | ymat <- matrix(seq(from = 1, to = lghlab, by = 1), nrow(xmat), lghlab, byrow = TRUE) 242 | ymat <- (ymat == as.numeric(iris$Species)) + 0 243 | amlmodel <- automl_train_manual(Xref = xmat, Yref = ymat, 244 | hpar = list(layersshape = c(0), 245 | layersacttype = c('sigmoid'), 246 | layersdropoprob = c(0))) 247 | ``` 248 | We saved the model to continue training later (see below in next section) 249 | ```{r} 250 | amlmodelsaved <- amlmodel 251 | ``` 252 | 253 | 254 |

    0.3.7 continue training on saved model (fine tuning ...)

    255 | Subject: continue training on saved model (model saved above in last section)
    256 | ```{r} 257 | amlmodel <- automl_train_manual(Xref = xmat, Yref = ymat, 258 | hpar = list(numiterations = 100, 259 | psopartpopsize = 50), 260 | mdlref = amlmodelsaved) 261 | ``` 262 | We can see the error continuing to decrease from last training
    263 | The training continued with the same parameters, but notice that we were able to change the number of iterations
    264 | 265 |

    0.3.8 use the 2 steps automatic approach

    266 | Same subject: predict Species given other Iris parameters
    267 | Let’s try the automatic approach in 2 steps with the same Logistic Regression architecture;
    268 | 1st step goal is performance, overfitting
    269 | 2nd step is robustness, regularization
    270 | ```{r} 271 | data(iris) 272 | xmat = iris[,1:4] 273 | lab2pred <- levels(iris$Species) 274 | lghlab <- length(lab2pred) 275 | iris$Species <- as.numeric(iris$Species) 276 | ymat <- matrix(seq(from = 1, to = lghlab, by = 1), nrow(xmat), lghlab, byrow = TRUE) 277 | ymat <- (ymat == as.numeric(iris$Species)) + 0 278 | amlmodel <- automl_train(Xref = xmat, Yref = ymat, 279 | hpar = list(layersshape = c(0), 280 | layersacttype = c('sigmoid'), 281 | layersdropoprob = c(0)), 282 | autopar = list(auto_runtype = '2steps')) 283 | ``` 284 | Compared to the last runs (in previous sections above), difference between train and cross validation errors is much more tenuous
    285 | Automatically :-)
    286 | 287 |

    0.4 ToDo List idea

    288 |

    - review the code to object oriented
    289 | - manage transfert learning from existing frameworks
    290 | - implement CNN
    291 | - implement RNN
    292 | - ...
    293 |
    294 | -> I won't do it alone, let's create a team !
    295 | https://aboulaboul.github.io/automl
    296 | https://github.com/aboulaboul/automl

    297 | 298 | 299 | -------------------------------------------------------------------------------- /docs/reference/automl_train_manual.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | automl_train_manual — automl_train_manual • automl 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 53 | 54 | 55 | 56 | 57 | 58 | 59 |
    60 |
    61 | 103 | 104 | 105 | 106 |
    107 | 108 |
    109 |
    110 |
    111 |

    automl_train_manual

    112 | 113 | 114 |
    115 | 116 |
    117 |

    The base deep neural network train function (one deep neural network trained without automatic hyperparameters tuning)

    118 |
    119 | 120 | 
    automl_train_manual(Xref, Yref, hpar = list(), mdlref = NULL)
    121 | 122 |

    Arguments

    123 | 124 | 125 | 126 | 127 | 128 | 129 | 130 | 131 | 132 | 133 | 134 | 135 | 137 | 138 | 139 | 140 | 143 | 144 |
    Xref

    inputs matrix or data.frame (containing numerical values only)

    Yref

    target matrix or data.frame (containing numerical values only)

    hpar

    list of parameters and hyperparameters for Deep Neural Network, see hpar section
    136 | Not mandatory (the list is preset and all arguments are initialized with default value) but it is advisable to adjust some important arguments for performance reasons (including processing time)

    mdlref

    model trained with automl_train or automl_train_manual to start training from a saved model (shape, 141 | weights...) for fine tuning
    142 | nb: manually entered parameters above override loaded ones

    145 | 146 | 147 |

    Examples

    148 | 
    ##REGRESSION (predict Sepal.Length given other Iris parameters)
149 | data(iris)
150 | xmat <- cbind(iris[,2:4], as.numeric(iris$Species))
151 | ymat <- iris[,1]
152 | #with gradient descent
153 | amlmodel <- automl_train_manual(Xref = xmat, Yref = ymat,
154 |                                 hpar = list(learningrate = 0.01,
155 |                                             numiterations = 30,
156 |                                             minibatchsize = 2^2))
    #> (cost: mse)
157 | #> cost epoch10: 0.07483378655409 (cv cost: 0.466478285718589) (LR:  0.01 ) 
158 | #> cost epoch20: 0.275546796158403 (cv cost: 0.294895895020273) (LR:  0.01 ) 
159 | #> cost epoch30: 0.231827727788502 (cv cost: 0.233179461368555) (LR:  0.01 ) 
160 | #>    dim X: [4,135]
161 | #>    dim W1: [10,4] (min|max: -1.54429770594677, 2.73130744979701)
162 | #>    dim bB1: [10,1] (min|max: -0.317658298585351, 1.99139261324871)
163 | #>    dim W2: [1,10] (min|max: -0.313978049365135, 0.286454921099375)
164 | #>    dim bB2: [1,1] (min|max: 0.337210920487028, 0.337210920487028)
165 | #>    dim Y: [1,135]
    if (FALSE) {
166 | #with PSO
167 | amlmodel <- automl_train_manual(Xref = xmat, Yref = ymat,
168 |                                 hpar = list(modexec = 'trainwpso',
169 |                                             numiterations = 30,
170 |                                             psopartpopsize = 50))
171 | #with PSO and custom cost function
172 | f <- 'J=abs((y-yhat)/y)'
173 | f <- c(f, 'J=sum(J[!is.infinite(J)],na.rm=TRUE)')
174 | f <- c(f, 'J=(J/length(y))')
175 | f <- paste(f, collapse = ';')
176 | amlmodel <- automl_train_manual(Xref = xmat, Yref = ymat,
177 |                                 hpar = list(modexec = 'trainwpso',
178 |                                             numiterations = 30,
179 |                                             psopartpopsize = 50,
180 |                                             costcustformul = f))
181 | 
182 | ##CLASSIFICATION (predict Species given other Iris parameters)
183 | data(iris)
184 | xmat = iris[,1:4]
185 | lab2pred <- levels(iris$Species)
186 | lghlab <- length(lab2pred)
187 | iris$Species <- as.numeric(iris$Species)
188 | ymat <- matrix(seq(from = 1, to = lghlab, by = 1), nrow(xmat), lghlab, byrow = TRUE)
189 | ymat <- (ymat == as.numeric(iris$Species)) + 0
190 | #with gradient descent and 2 hidden layers
191 | amlmodel <- automl_train_manual(Xref = xmat, Yref = ymat,
192 |                                 hpar = list(layersshape = c(10, 10, 0),
193 |                                             layersacttype = c('tanh', 'relu', 'sigmoid'),
194 |                                             layersdropoprob = c(0, 0, 0)))
195 | #with gradient descent and no hidden layer (logistic regression)
196 | amlmodel <- automl_train_manual(Xref = xmat, Yref = ymat,
197 |                                 hpar = list(layersshape = c(0),
198 |                                             layersacttype = c('sigmoid'),
199 |                                             layersdropoprob = c(0)))
200 | #with PSO and softmax
201 | amlmodel <- automl_train_manual(Xref = xmat, Yref = ymat,
202 |                                 hpar = list(modexec = 'trainwpso',
203 |                                             layersshape = c(10, 0),
204 |                                             layersacttype = c('relu', 'softmax'),
205 |                                             layersdropoprob = c(0, 0),
206 |                                             numiterations = 50,
207 |                                             psopartpopsize = 50))
208 | }
    209 |
    210 | 218 |
    219 | 220 | 221 | 231 |
    232 | 233 | 234 | 235 | 236 | 237 | 238 | 239 | 240 | -------------------------------------------------------------------------------- /vignettes/howto_automl.Rnw: -------------------------------------------------------------------------------- 1 | %\VignetteIndexEntry{howto_automl.pdf} 2 | \documentclass[10pt,a4paper,oneside]{report} 3 | \usepackage[a4paper]{geometry} 4 | \geometry{hscale=0.90,vscale=0.90,centering} 5 | \begin{document} 6 | \SweaveOpts{concordance=TRUE} 7 | \title{automl R package vignette} 8 | \author[1]{Alex BOULANGE - aboul@free.fr} 9 | \maketitle 10 | \tableofcontents 11 | \newpage 12 | \section{Introduction to automl package} 13 | This document is intended to answer the following questions; why \& how automl and how to use it\\\\ 14 | automl package provides: \\ 15 | -Deep Learning last tricks (those who have taken Andrew NG’s MOOC on Coursera will be in familiar territory)\\ 16 | -hyperparameters autotune with metaheuristic (PSO)\\ 17 | -experimental stuff and more to come (you’re welcome as coauthor!)\\ 18 | 19 | 20 | 21 | \section{Why \& how automl} 22 | 23 | \subsection{Deep Learning existing frameworks, disadvantages} 24 | Deploying and maintaining most Deep Learning frameworks means: Python...\\ 25 | R language is so simple to install and maintain in production environments that it is obvious to use a pure R based package for deep learning !\\ 26 | 27 | \subsection{Neural Network - Deep Learning, disadvantages} 28 | Disadvantages :\\ 29 | 1st disadvantage: you have to test manually different combinations of parameters (number of layers, nodes, activation function, etc ...) and then also tune manually hyper parameters for training (learning rate, momentum, mini batch size, etc ...)\\ 30 | 2nd disadvantage: only for those who are not mathematicians, calculating derivative in case of new cost or activation function, may by an issue.\\\\ 31 | \includegraphics[width=0.9\textwidth]{imgperc001.png}\\ 32 | 33 | \subsection{Metaheuristic - PSO, benefits} 34 | The Particle Swarm Optimization algorithm is a great and simple one.\\ 35 | In a few words, the first step consists in throwing randomly a set of particles in a space and the next steps consist in discovering the best solution while converging.\\ 36 | \includegraphics[width=0.9\textwidth]{imgperc002.png}\\ 37 | video tutorial from Yarpiz is a great ressource\\ 38 | 39 | \subsection{Birth of automl package} 40 | automl package was born from the idea to use metaheuristic PSO to address the identified disadvantages above.\\ 41 | And last but not the least reason: use R and R only :-)\\ 42 | 3 functions are available:\\ 43 | - automl\_train\_manual: the manual mode to train a model\\ 44 | - automl\_train: the automatic mode to train model\\ 45 | - automl\_predict: the prediction function to apply a trained model on datas\\ 46 | 47 | \subsection{Mix 1: hyperparameters tuning with PSO} 48 | Mix 1 consists in using PSO algorithm to optimize the hyperparameters: each particle corresponds to a set of hyperparameters.\\ 49 | The automl\_train function was made to do that.\\\\ 50 | \includegraphics[width=0.9\textwidth]{imgperc003.png}\\ 51 | 52 | \subsection{Mix 2: PSO instead of gradient descent} 53 | Mix 2 is experimental, it consists in using PSO algorithm to optimize the weights of Neural Network in place of gradient descent: each particle corresponds to a set of neural network weights matrices.\\ 54 | The automl\_train\_manual function do that too.\\\\ 55 | \includegraphics[width=0.9\textwidth]{imgperc004.png}\\ 56 | 57 | 58 | \section{First steps: How to} 59 | For those who will laugh at seeing deep learning with one hidden layer and the Iris data set of 150 records, I will say: you're perfectly right :-)\\ 60 | The goal at this stage is simply to take the first steps 61 | 62 | \subsection{fit a regression model manually (hard way)} 63 | Subject: predict Sepal.Length given other Iris parameters\\ 64 | 1st with gradient descent and default hyperparameters value for learning rate (0.001) and mini batch size (32)\\ 65 | \begin{Schunk} 66 | \begin{Sinput} 67 | data(iris) 68 | xmat <- cbind(iris[,2:4], as.numeric(iris$Species)) 69 | ymat <- iris[,1] 70 | amlmodel <- automl_train_manual(Xref = xmat, Yref = ymat) 71 | \end{Sinput} 72 | \begin{Soutput} 73 | (cost: mse) 74 | cost epoch10: 20.9340400047156 (cv cost: 25.205632342013) (LR: 0.001 ) 75 | cost epoch20: 20.6280923387762 (cv cost: 23.8214521197268) (LR: 0.001 ) 76 | cost epoch30: 20.3222407903838 (cv cost: 22.1899741289456) (LR: 0.001 ) 77 | cost epoch40: 20.0217966054298 (cv cost: 21.3908446693146) (LR: 0.001 ) 78 | cost epoch50: 19.7584058034009 (cv cost: 20.7170232035934) (LR: 0.001 ) 79 | dim X: ... 80 | \end{Soutput} 81 | \begin{Sinput} 82 | res <- cbind(ymat, automl_predict(model = amlmodel, X = xmat)) 83 | colnames(res) <- c('actual', 'predict') 84 | head(res) 85 | \end{Sinput} 86 | \begin{Soutput} 87 | actual predict 88 | [1,] 5.1 -2.063614 89 | [2,] 4.9 -2.487673 90 | [3,] 4.7 -2.471912 91 | [4,] 4.6 -2.281035 92 | [5,] 5.0 -1.956937 93 | [6,] 5.4 -1.729314 94 | \end{Soutput} 95 | \end{Schunk} 96 | :-[] no pain, no gain ...\\\\ 97 | After some manual fine tuning on learning rate, mini batch size and iterations number (epochs):\\ 98 | \begin{Schunk} 99 | \begin{Sinput} 100 | data(iris) 101 | xmat <- cbind(iris[,2:4], as.numeric(iris$Species)) 102 | ymat <- iris[,1] 103 | amlmodel <- automl_train_manual(Xref = xmat, Yref = ymat, 104 | hpar = list(learningrate = 0.01, 105 | minibatchsize = 2^2, 106 | numiterations = 30)) 107 | \end{Sinput} 108 | \begin{Soutput} 109 | (cost: mse) 110 | cost epoch10: 5.55679482839698 (cv cost: 4.87492997304325) (LR: 0.01 ) 111 | cost epoch20: 1.64996951479802 (cv cost: 1.50339773126712) (LR: 0.01 ) 112 | cost epoch30: 0.647727077375946 (cv cost: 0.60142564484723) (LR: 0.01 ) 113 | dim X: ... 114 | \end{Soutput} 115 | \begin{Sinput} 116 | res <- cbind(ymat, automl_predict(model = amlmodel, X = xmat)) 117 | colnames(res) <- c('actual', 'predict') 118 | head(res) 119 | \end{Sinput} 120 | \begin{Soutput} 121 | actual predict 122 | [1,] 5.1 4.478478 123 | [2,] 4.9 4.215683 124 | [3,] 4.7 4.275902 125 | [4,] 4.6 4.313141 126 | [5,] 5.0 4.531038 127 | [6,] 5.4 4.742847 128 | \end{Soutput} 129 | \end{Schunk} 130 | Better result, but with human efforts!\\ 131 | 132 | \subsection{fit a regression model automatically (easy way, Mix 1)} 133 | Same subject: predict Sepal.Length given other Iris parameters\\ 134 | \begin{Schunk} 135 | \begin{Sinput} 136 | data(iris) 137 | xmat <- as.matrix(cbind(iris[,2:4], as.numeric(iris$Species))) 138 | ymat <- iris[,1] 139 | start.time <- Sys.time() 140 | amlmodel <- automl_train(Xref = xmat, Yref = ymat, 141 | autopar = list(psopartpopsize = 15, 142 | numiterations = 5, 143 | auto_layers_max = 1, 144 | nbcores = 4)) 145 | end.time <- Sys.time() 146 | cat(paste('time ellapsed:', end.time - start.time, '\n')) 147 | \end{Sinput} 148 | \begin{Soutput} 149 | (cost: mse) 150 | iteration 1 particle 1 weighted err: 22.20611 (train: 20.93004 cvalid: 17.73985 ) BEST MODEL KEPT 151 | iteration 1 particle 2 weighted err: 21.48398 (train: 20.87709 cvalid: 19.35987 ) BEST MODEL KEPT 152 | iteration 1 particle 3 weighted err: 22.98075 (train: 20.96822 cvalid: 15.9369 ) 153 | iteration 1 particle 4 weighted err: 22.64428 (train: 21.44958 cvalid: 22.19627 ) 154 | iteration 1 particle 5 weighted err: 22.03677 (train: 20.83123 cvalid: 17.81739 ) 155 | iteration 1 particle 6 weighted err: 22.30679 (train: 21.02051 cvalid: 17.80482 ) 156 | iteration 1 particle 7 weighted err: 22.15835 (train: 20.67513 cvalid: 16.96709 ) 157 | iteration 1 particle 8 weighted err: 21.04265 (train: 20.1281 cvalid: 17.84174 ) BEST MODEL KEPT 158 | iteration 1 particle 9 weighted err: 21.0425 (train: 20.72162 cvalid: 19.91941 ) BEST MODEL KEPT 159 | iteration 1 particle 10 weighted err: 13.78127 (train: 13.54355 cvalid: 12.94923 ) BEST MODEL KEPT 160 | iteration 1 particle 11 weighted err: 20.34595 (train: 19.54807 cvalid: 17.55337 ) 161 | iteration 1 particle 12 weighted err: 21.86865 (train: 20.99222 cvalid: 18.80114 ) 162 | iteration 1 particle 13 weighted err: 14.11782 (train: 13.54468 cvalid: 12.11184 ) 163 | iteration 1 particle 14 weighted err: 23.71203 (train: 20.88439 cvalid: 13.81528 ) 164 | iteration 1 particle 15 weighted err: 22.20701 (train: 20.90727 cvalid: 17.65792 ) 165 | ... 166 | iteration 4 particle 1 weighted err: 12.94662 (train: 4.40029 cvalid: 9.74174 ) 167 | iteration 4 particle 2 weighted err: 18.67702 (train: 17.28413 cvalid: 18.15469 ) 168 | iteration 4 particle 3 weighted err: 0.76426 (train: 0.01617 cvalid: 0.48373 ) 169 | iteration 4 particle 4 weighted err: 11.2243 (train: 11.00578 cvalid: 10.45947 ) 170 | iteration 4 particle 5 weighted err: 1.93528 (train: 0.00141 cvalid: 1.21008 ) 171 | iteration 4 particle 6 weighted err: 17.89662 (train: 17.65917 cvalid: 17.06555 ) 172 | iteration 4 particle 7 weighted err: 0.49851 (train: 0.42819 cvalid: 0.47214 ) BEST MODEL KEPT 173 | ... 174 | time ellapsed: 1.84548579454422 175 | \end{Soutput} 176 | \begin{Sinput} 177 | res <- cbind(ymat, automl_predict(model = amlmodel, X = xmat)) 178 | colnames(res) <- c('actual', 'predict') 179 | head(res) 180 | \end{Sinput} 181 | \begin{Soutput} 182 | actual predict 183 | [1,] 5.1 4.975551 184 | [2,] 4.9 4.667541 185 | [3,] 4.7 4.735524 186 | [4,] 4.6 4.784364 187 | [5,] 5.0 5.037152 188 | [6,] 5.4 5.325999 189 | \end{Soutput} 190 | \end{Schunk} 191 | It's even better, with no human efforts but machine time\\ 192 | Windows users won't benefit from parallelization, the function uses parallel package included with R base...\\ 193 | 194 | \subsection{fit a regression model experimentally (experimental way, Mix 2)} 195 | Same subject: predict Sepal.Length given other Iris parameters\\ 196 | \begin{Schunk} 197 | \begin{Sinput} 198 | data(iris) 199 | xmat <- as.matrix(cbind(iris[,2:4], as.numeric(iris$Species))) 200 | ymat <- iris[,1] 201 | amlmodel <- automl_train_manual(Xref = xmat, Yref = ymat, 202 | hpar = list(modexec = 'trainwpso', 203 | numiterations = 30, 204 | psopartpopsize = 50)) 205 | \end{Sinput} 206 | \begin{Soutput} 207 | (cost: mse) 208 | cost epoch10: 0.113576786377019 (cv cost: 0.0967069106128153) (LR: 0 ) 209 | cost epoch20: 0.0595472259640828 (cv cost: 0.0831404427407914) (LR: 0 ) 210 | cost epoch30: 0.0494578776185938 (cv cost: 0.0538888075333611) (LR: 0 ) 211 | dim X: ... 212 | \end{Soutput} 213 | \begin{Sinput} 214 | res <- cbind(ymat, automl_predict(model = amlmodel, X = xmat)) 215 | colnames(res) <- c('actual', 'predict') 216 | head(res) 217 | \end{Sinput} 218 | \begin{Soutput} 219 | actual predict 220 | [1,] 5.1 5.028114 221 | [2,] 4.9 4.673366 222 | [3,] 4.7 4.738188 223 | [4,] 4.6 4.821392 224 | [5,] 5.0 5.099064 225 | [6,] 5.4 5.277315 226 | \end{Soutput} 227 | \end{Schunk} 228 | Pretty good too, even better!\\ 229 | 230 | \subsection{fit a regression model with custom cost (experimental way, Mix 2)} 231 | Same subject: predict Sepal.Length given other Iris parameters\\ 232 | Let's try with Mean Absolute Percentage Error instead of Mean Square Error 233 | \begin{Schunk} 234 | \begin{Sinput} 235 | data(iris) 236 | xmat <- as.matrix(cbind(iris[,2:4], as.numeric(iris$Species))) 237 | ymat <- iris[,1] 238 | f <- 'J=abs((y-yhat)/y)' 239 | f <- c(f, 'J=sum(J[!is.infinite(J)],na.rm=TRUE)') 240 | f <- c(f, 'J=(J/length(y))') 241 | f <- paste(f, collapse = ';') 242 | amlmodel <- automl_train_manual(Xref = xmat, Yref = ymat, 243 | hpar = list(modexec = 'trainwpso', 244 | numiterations = 30, 245 | psopartpopsize = 50, 246 | costcustformul = f)) 247 | \end{Sinput} 248 | \begin{Soutput} 249 | (cost: custom) 250 | cost epoch10: 0.901580275333795 (cv cost: 1.15936129555304) (LR: 0 ) 251 | cost epoch20: 0.890142834441629 (cv cost: 1.24167078564786) (LR: 0 ) 252 | cost epoch30: 0.886088388448652 (cv cost: 1.22756121243449) (LR: 0 ) 253 | dim X: ... 254 | \end{Soutput} 255 | \begin{Sinput} 256 | res <- cbind(ymat, automl_predict(model = amlmodel, X = xmat)) 257 | colnames(res) <- c('actual', 'predict') 258 | head(res) 259 | \end{Sinput} 260 | \begin{Soutput} 261 | actual predict 262 | [1,] 5.1 4.693915 263 | [2,] 4.9 4.470968 264 | [3,] 4.7 4.482036 265 | [4,] 4.6 4.593667 266 | [5,] 5.0 4.738504 267 | [6,] 5.4 4.914144 268 | \end{Soutput} 269 | \end{Schunk} 270 | 271 | \subsection{fit a classification model with softmax (Mix 2)} 272 | Subject: predict Species given other Iris parameters\\ 273 | Softmax is available with PSO, no derivative needed ;-)\\ 274 | \begin{Schunk} 275 | \begin{Sinput} 276 | data(iris) 277 | xmat = iris[,1:4] 278 | lab2pred <- levels(iris$Species) 279 | lghlab <- length(lab2pred) 280 | iris$Species <- as.numeric(iris$Species) 281 | ymat <- matrix(seq(from = 1, to = lghlab, by = 1), nrow(xmat), lghlab, byrow = TRUE) 282 | ymat <- (ymat == as.numeric(iris$Species)) + 0 283 | amlmodel <- automl_train_manual(Xref = xmat, Yref = ymat, 284 | hpar = list(modexec = 'trainwpso', 285 | layersshape = c(10, 0), 286 | layersacttype = c('relu', 'softmax'), 287 | layersdropoprob = c(0, 0), 288 | numiterations = 50, 289 | psopartpopsize = 50)) 290 | \end{Sinput} 291 | \begin{Soutput} 292 | (cost: crossentropy) 293 | cost epoch10: 0.373706545886467 (cv cost: 0.36117608867856) (LR: 0 ) 294 | cost epoch20: 0.267034060152876 (cv cost: 0.163635821437066) (LR: 0 ) 295 | cost epoch30: 0.212054571476337 (cv cost: 0.112664100290429) (LR: 0 ) 296 | cost epoch40: 0.154158717402463 (cv cost: 0.102895917099299) (LR: 0 ) 297 | cost epoch50: 0.141037927317585 (cv cost: 0.0864623836595045) (LR: 0 ) 298 | dim X: ... 299 | \end{Soutput} 300 | \begin{Sinput} 301 | res <- cbind(ymat, automl_predict(model = amlmodel, X = xmat)) 302 | colnames(res) <- c(paste('act',lab2pred, sep = '_'), 303 | paste('pred',lab2pred, sep = '_')) 304 | head(res) 305 | tail(res) 306 | \end{Sinput} 307 | \begin{Soutput} 308 | act_setosa act_versicolor act_virginica pred_setosa pred_versicolor pred_virginica 309 | 1 1 0 0 0.9863481 0.003268881 0.010383018 310 | 2 1 0 0 0.9897295 0.003387193 0.006883349 311 | 3 1 0 0 0.9856347 0.002025946 0.012339349 312 | 4 1 0 0 0.9819881 0.004638452 0.013373451 313 | 5 1 0 0 0.9827623 0.003115452 0.014122277 314 | 6 1 0 0 0.9329747 0.031624836 0.035400439 315 | 316 | act_setosa act_versicolor act_virginica pred_setosa pred_versicolor pred_virginica 317 | 145 0 0 1 0.02549091 2.877957e-05 0.9744803 318 | 146 0 0 1 0.08146753 2.005664e-03 0.9165268 319 | 147 0 0 1 0.05465750 1.979652e-02 0.9255460 320 | 148 0 0 1 0.06040415 1.974869e-02 0.9198472 321 | 149 0 0 1 0.02318048 4.133826e-04 0.9764061 322 | 150 0 0 1 0.03696852 5.230936e-02 0.9107221 323 | \end{Soutput} 324 | \end{Schunk} 325 | 326 | 327 | \subsection{change the model parameters (shape ...)} 328 | Same subject: predict Species given other Iris parameters\\ 329 | 1st example: with gradient descent and 2 hidden layers containing 10 nodes, with various activation functions for hidden layers\\ 330 | \begin{Schunk} 331 | \begin{Sinput} 332 | data(iris) 333 | xmat = iris[,1:4] 334 | lab2pred <- levels(iris$Species) 335 | lghlab <- length(lab2pred) 336 | iris$Species <- as.numeric(iris$Species) 337 | ymat <- matrix(seq(from = 1, to = lghlab, by = 1), nrow(xmat), lghlab, byrow = TRUE) 338 | ymat <- (ymat == as.numeric(iris$Species)) + 0 339 | amlmodel <- automl_train_manual( 340 | Xref = xmat, Yref = ymat, 341 | hpar = list( 342 | layersshape = c(10, 10, 0), 343 | layersacttype = c('tanh', 'relu', ''), 344 | layersdropoprob = c(0, 0, 0) 345 | ) 346 | ) 347 | \end{Sinput} 348 | \end{Schunk} 349 | nb: last activation type may be left to blank (it will be set automatically)\\\\ 350 | 2nd example: with gradient descent and no hidden layer (logistic regression)\\ 351 | \begin{Schunk} 352 | \begin{Sinput} 353 | data(iris) 354 | xmat = iris[,1:4] 355 | lab2pred <- levels(iris$Species) 356 | lghlab <- length(lab2pred) 357 | iris$Species <- as.numeric(iris$Species) 358 | ymat <- matrix(seq(from = 1, to = lghlab, by = 1), nrow(xmat), lghlab, byrow = TRUE) 359 | ymat <- (ymat == as.numeric(iris$Species)) + 0 360 | amlmodel <- automl_train_manual(Xref = xmat, Yref = ymat, 361 | hpar = list(layersshape = c(0), 362 | layersacttype = c('sigmoid'), 363 | layersdropoprob = c(0))) 364 | \end{Sinput} 365 | \begin{Soutput} 366 | (cost: crossentropy) 367 | cost epoch10: 2.41256625698174 (cv cost: 1.52787772773656) (LR: 0.001 ) 368 | cost epoch20: 2.39487960709668 (cv cost: 1.30539484912328) (LR: 0.001 ) 369 | cost epoch30: 2.37790838174604 (cv cost: 0.914874371089546) (LR: 0.001 ) 370 | cost epoch40: 2.36237291354489 (cv cost: 1.23368286521283) (LR: 0.001 ) 371 | cost epoch50: 2.34978864866666 (cv cost: 1.26105975399258) (LR: 0.001 ) 372 | dim X: ... 373 | \end{Soutput} 374 | \begin{Sinput} 375 | amlmodelsaved <- amlmodel 376 | \end{Sinput} 377 | \end{Schunk} 378 | We saved the model to continue training later (see below in next section)\\ 379 | 380 | \subsection{continue training on saved model (fine tuning ...)} 381 | Subject: continue training on saved model (model saved above in last section)\\ 382 | \begin{Schunk} 383 | \begin{Sinput} 384 | amlmodel <- automl_train_manual(Xref = xmat, Yref = ymat, 385 | hpar = list(numiterations = 100, 386 | psopartpopsize = 50), 387 | mdlref = amlmodelsaved) 388 | \end{Sinput} 389 | \begin{Soutput} 390 | (cost: crossentropy) 391 | cost epoch10: 2.33948479711291 (cv cost: 1.3572198299724) (LR: 0.001 ) 392 | cost epoch20: 2.32669285753264 (cv cost: 1.30152708067868) (LR: 0.001 ) 393 | cost epoch30: 2.31504805288479 (cv cost: 1.29552449383989) (LR: 0.001 ) 394 | cost epoch40: 2.30430035748777 (cv cost: 1.29397405747142) (LR: 0.001 ) 395 | cost epoch50: 2.29430374216071 (cv cost: 1.29240693651625) (LR: 0.001 ) 396 | cost epoch60: 2.28495815493859 (cv cost: 1.29246202979512) (LR: 0.001 ) 397 | cost epoch70: 2.27608005019126 (cv cost: 1.29620090185108) (LR: 0.001 ) 398 | cost epoch80: 2.26753980962059 (cv cost: 1.3019296243155) (LR: 0.001 ) 399 | cost epoch90: 2.25934270485252 (cv cost: 1.30591289162664) (LR: 0.001 ) 400 | cost epoch100: 2.25229290115905 (cv cost: 1.30730635529277) (LR: 0.001 ) 401 | dim X: ... 402 | \end{Soutput} 403 | \end{Schunk} 404 | We can see the error continuing to decrease from last training\\ 405 | The training continued with the same parameters, but notice that we were able to change the number of iterations 406 | 407 | 408 | \subsection{use the 2 steps automatic approach} 409 | Same subject: predict Species given other Iris parameters\\ 410 | Let's try the automatic approach in 2 steps with the same Logistic Regression architecture;\\ 411 | 1st step goal is performance, overfitting\\ 412 | 2nd step is robustness, regularization\\ 413 | 414 | \begin{Schunk} 415 | \begin{Sinput} 416 | data(iris) 417 | xmat = iris[,1:4] 418 | lab2pred <- levels(iris$Species) 419 | lghlab <- length(lab2pred) 420 | iris$Species <- as.numeric(iris$Species) 421 | ymat <- matrix(seq(from = 1, to = lghlab, by = 1), nrow(xmat), lghlab, byrow = TRUE) 422 | ymat <- (ymat == as.numeric(iris$Species)) + 0 423 | amlmodel <- automl_train(Xref = xmat, Yref = ymat, 424 | hpar = list(layersshape = c(0), 425 | layersacttype = c('sigmoid'), 426 | layersdropoprob = c(0)), 427 | autopar = list(auto_runtype = '2steps')) 428 | \end{Sinput} 429 | \begin{Soutput} 430 | STEP: 1 (overfitting) 431 | (cost: crossentropy) 432 | iteration 1 particle 1 weighted err: 2.42056 BEST MODEL KEPT 433 | iteration 1 particle 2 weighted err: 2.40494 BEST MODEL KEPT 434 | iteration 1 particle 3 weighted err: 2.41726 435 | iteration 1 particle 4 weighted err: 2.42334 436 | iteration 1 particle 5 weighted err: 2.4091 437 | iteration 1 particle 6 weighted err: 2.41962 438 | iteration 1 particle 7 weighted err: 2.40137 BEST MODEL KEPT 439 | iteration 1 particle 8 weighted err: 2.34594 BEST MODEL KEPT 440 | ... 441 | iteration 3 particle 1 weighted err: 2.39013 442 | iteration 3 particle 2 weighted err: 2.36849 443 | iteration 3 particle 3 weighted err: 2.23385 BEST MODEL KEPT 444 | iteration 3 particle 4 weighted err: 2.40762 445 | iteration 3 particle 5 weighted err: 2.33986 446 | iteration 3 particle 6 weighted err: 2.39917 447 | iteration 3 particle 7 weighted err: 2.38197 448 | iteration 3 particle 8 weighted err: 2.34594 449 | STEP: 2 (regularization) 450 | (cost: crossentropy) 451 | iteration 1 particle 1 weighted err: 2.70508 (train: 2.24474 cvalid: 1.09389 ) BEST MODEL KEPT 452 | iteration 1 particle 2 weighted err: 2.71015 (train: 2.2452 cvalid: 1.08283 ) 453 | iteration 1 particle 3 weighted err: 2.2944 (train: 2.23677 cvalid: 2.0927 ) BEST MODEL KEPT 454 | iteration 1 particle 4 weighted err: 2.6911 (train: 2.23902 cvalid: 1.1088 ) 455 | iteration 1 particle 5 weighted err: 2.69633 (train: 2.24061 cvalid: 1.10131 ) 456 | iteration 1 particle 6 weighted err: 2.68252 (train: 2.24303 cvalid: 1.14432 ) 457 | iteration 1 particle 7 weighted err: 2.75194 (train: 2.24576 cvalid: 0.98029 ) 458 | iteration 1 particle 8 weighted err: 4.64173 (train: 2.23456 cvalid: 3.73904 ) 459 | ... 460 | iteration 2 particle 1 weighted err: 2.70508 (train: 2.24474 cvalid: 1.09389 ) 461 | iteration 2 particle 2 weighted err: 2.71015 (train: 2.2452 cvalid: 1.08283 ) 462 | iteration 2 particle 3 weighted err: 2.24647 (train: 2.23637 cvalid: 2.21111 ) BEST MODEL KEPT 463 | ... 464 | \end{Soutput} 465 | \end{Schunk} 466 | Compared to the last runs (in previous sections above), difference between train and cross validation errors is much more tenuous\\ 467 | Automatically :-)\\ 468 | 469 | 470 | \section{ToDo List idea} 471 | 472 | - review the code to object oriented\\ 473 | - manage transfert learning from existing frameworks\\ 474 | - implement CNN\\ 475 | - implement RNN\\ 476 | - ...\\\\ 477 | -> I won't do it alone, let's create a team !\\ 478 | https://aboulaboul.github.io/automl \\ 479 | https://github.com/aboulaboul/automl 480 | \end{document} 481 | --------------------------------------------------------------------------------