├── .gitignore ├── 01-getting-started.Rmd ├── 02-convex-problem-types.Rmd ├── 03-solving-convex-problems.Rmd ├── 04-disciplined-convex-programming.Rmd ├── 05-a-simple-regression-example.Rmd ├── 06-how-cvxr-works.Rmd ├── 07-logistic-regression.Rmd ├── 08-isotonic-regression.Rmd ├── 09-lasso-and-elastic-net.Rmd ├── 10-near-isotonic-fit.Rmd ├── 11-speed-considerations.Rmd ├── 12-pliable-lasso.Rmd ├── 13-survey-sampling.Rmd ├── 14-sparse-inverse-covariance-estimation.Rmd ├── 15-portfolio-estimation.Rmd ├── 16-using-other-solvers.Rmd ├── 97-the-structure-of-atoms.Rmd ├── 98-future.Rmd ├── 99-references.Rmd ├── LICENSE ├── README.md ├── _book ├── a-simple-regression-example.html ├── cvxr_tutorial_files │ └── figure-html │ │ ├── unnamed-chunk-118-1.png │ │ ├── unnamed-chunk-122-1.png │ │ ├── unnamed-chunk-123-1.png │ │ └── unnamed-chunk-66-1.png ├── disciplined-convex-programming.html ├── figures │ └── 02 │ │ └── convexchord.gif ├── future-enhancements.html ├── getting-started.html ├── how-cvxr-works.html ├── index.html ├── isotonic-regression.html ├── lasso-and-elastic-net.html ├── libs │ ├── anchor-sections-1.0.1 │ │ ├── anchor-sections.css │ │ └── anchor-sections.js │ ├── gitbook-2.6.7 │ │ ├── css │ │ │ ├── fontawesome │ │ │ │ └── fontawesome-webfont.ttf │ │ │ ├── plugin-bookdown.css │ │ │ ├── plugin-clipboard.css │ │ │ ├── plugin-fontsettings.css │ │ │ ├── plugin-highlight.css │ │ │ ├── plugin-search.css │ │ │ ├── plugin-table.css │ │ │ └── style.css │ │ └── js │ │ │ ├── app.min.js │ │ │ ├── clipboard.min.js │ │ │ ├── jquery.highlight.js │ │ │ ├── lunr.js │ │ │ ├── plugin-bookdown.js │ │ │ ├── plugin-clipboard.js │ │ │ ├── plugin-fontsettings.js │ │ │ ├── plugin-search.js │ │ │ └── plugin-sharing.js │ ├── header-attrs-2.9 │ │ └── header-attrs.js │ ├── jquery-2.2.3 │ │ └── jquery.min.js │ ├── kePrint-0.0.1 │ │ └── kePrint.js │ └── lightable-0.0.1 │ │ └── lightable.css ├── logistic-regression.html ├── nearly-isotonic-fits.html ├── pliable-lasso.html ├── portfolio-estimation.html ├── reference-keys.txt ├── references-8.html ├── references-9.html ├── search_index.json ├── solving-convex-problems.html ├── sparse-inverse-covariance-estimation.html ├── speed-considerations.html ├── style.css ├── survey-calibration.html ├── the-structure-of-atoms.html ├── types-of-convex-optimization-problems.html └── using-other-solvers.html ├── _bookdown.yml ├── _bookdown_files └── cvxr_tutorial_files │ └── figure-html │ ├── unnamed-chunk-118-1.png │ ├── unnamed-chunk-122-1.png │ ├── unnamed-chunk-123-1.png │ └── unnamed-chunk-66-1.png ├── _output.yml ├── book.bib ├── cvxr.bib ├── cvxr_tutorial.Rproj ├── extra ├── code │ └── cvxr_tutorial.R └── slides │ ├── book.bib │ ├── cvxr.bib │ ├── figures │ ├── 02 │ │ └── convexchord.gif │ ├── iso1.png │ └── iso2.png │ ├── packages.bib │ ├── preamble.tex │ ├── slides.Rmd │ └── slides.html ├── figures └── 02 │ └── convexchord.gif ├── header.html ├── index.Rmd ├── packages.bib ├── preamble.tex └── style.css /.gitignore: -------------------------------------------------------------------------------- 1 | # History files 2 | .Rhistory 3 | .Rapp.history 4 | 5 | # Session Data files 6 | .RData 7 | 8 | # Example code in package build process 9 | *-Ex.R 10 | 11 | # Output files from R CMD build 12 | /*.tar.gz 13 | 14 | # Output files from R CMD check 15 | /*.Rcheck/ 16 | 17 | # RStudio files 18 | .Rproj.user/ 19 | 20 | # produced vignettes 21 | vignettes/*.html 22 | vignettes/*.pdf 23 | 24 | # OAuth2 token, see https://github.com/hadley/httr/releases/tag/v0.3 25 | .httr-oauth 26 | 27 | # knitr and R markdown default cache directories 28 | /*_cache/ 29 | /cache/ 30 | 31 | # Temporary files created by R markdown 32 | *.utf8.md 33 | *.knit.md 34 | 35 | # Shiny token, see https://shiny.rstudio.com/articles/shinyapps.html 36 | rsconnect/ 37 | -------------------------------------------------------------------------------- /01-getting-started.Rmd: -------------------------------------------------------------------------------- 1 | # Getting Started 2 | 3 | The following are prerequisites. 4 | 5 | - A recent R installation (version 3.5.x and up) will do. However, we 6 | highly recommend 3.6.0+ so as to avoid any unforeseen problems with `CVXR`. 7 | 8 | - [`CVXR`](https://cran.r-project.org/package=CVXR) version 1.0-11, 9 | which can be installed like any other package from 10 | [CRAN](https://cran.r-project.org). 11 | 12 | - An IDE/editing environment such as [Rstudio](https://rstudio.com), 13 | [Emacs](https://www.gnu.org/software/emacs/), or equivalent. If you 14 | use Emacs, we recommend the version packaged and distributed by 15 | [Vincent Goulet](https://vgoulet.act.ulaval.ca/en/home/) as it 16 | includes many useful modes. (We find, however, that recent versions 17 | of these Emacs binaries do not include `poly-R` or the 18 | `poly-markdown` packages by default; you'll have to install them 19 | like any other Emacs package.) 20 | 21 | - All the additional libraries that are used in this tutorial. The 22 | following code snippet in your R session will install them for you. 23 | 24 | ```{r, eval = FALSE} 25 | required_packages <- c("tidyr", 26 | "ggplot2", 27 | "nnls", 28 | "ISLR", 29 | "glmnet", 30 | "isotone", 31 | "profvis", 32 | "dplyr", 33 | "survey", 34 | "expm", 35 | "RColorBrewer", 36 | "kableExtra") 37 | install.packages(required_packages) 38 | ``` 39 | 40 | ## Solver Prerequisites 41 | 42 | `CVXR` comes with open source solvers, and most of the tutorial will 43 | use these _built-in_ open source solvers. However, we will discuss use 44 | of commercial solvers, [MOSEK](https://www.mosek.com) and 45 | [GUROBI](https://www.gurobi.com) in particular. These are optional and 46 | not necessary for the tutorial. 47 | 48 | If, however, you wish to follow along, you will need to install the 49 | (binary) solver packages provided by the vendors. Luckily, academic 50 | and evaluation single user licenses are available for free. 51 | 52 | Ensure you have the R development tools installed: 53 | [Xcode](https://developer.apple.com/xcode/) on Macs, 54 | [Rtools](http://cran.r-project.org/bin/windows/Rtools/) on 55 | Windows. 56 | 57 | ### MOSEK 58 | 59 | 1. Follow the [general 60 | setup](https://docs.mosek.com/9.0/install/index.html) instructions. 61 | 62 | 2. Obtain an [evaluation or personal academic 63 | license](https://docs.mosek.com/9.0/licensing/quickstart.html#i-don-t-have-a-license-file-yet). 64 | 65 | 3. Follow the [instructions 66 | provided](https://docs.mosek.com/9.0/rmosek/install-interface.html#system-requirements) 67 | to install the `Rmosek` package. 68 | 69 | 4. Ensure that you can run any one of the [R examples](https://docs.mosek.com/9.0/rmosek/optimization-tutorials.html). 70 | 71 | ### GUROBI 72 | 73 | 1. Download [GUROBI optimizer version 74 | 8.1.1](http://www.gurobi.com/downloads/gurobi-optimizer). You will 75 | have to register for an account and accept the license agreement. 76 | 77 | 2. Obtain an [evaluation or academic licence](http://www.gurobi.com/downloads/licenses/license-center). 78 | 79 | 3. Install the GUROBI R package using [these 80 | instructions](http://www.gurobi.com/documentation/8.1/quickstart_mac/r_installing_the_r_package.html). 81 | 82 | 4. Ensure that you can run any one of the [R 83 | examples](http://www.gurobi.com/documentation/8.1/examples/r_examples.html). 84 | 85 | -------------------------------------------------------------------------------- /02-convex-problem-types.Rmd: -------------------------------------------------------------------------------- 1 | # Types of Convex Optimization Problems 2 | 3 | Recall that a convex optimization problem is of the form 4 | \[ 5 | \begin{array}{lll} \text{minimize} & f_0(x) & \\ 6 | \text{subject to} & f_i(x) \leq 0, & i=1, \ldots, M \\ 7 | & Ax=b & 8 | (\#eq:cvx-problem) 9 | \end{array} 10 | \] 11 | for $x \in {\mathbf R}^n$. 12 | 13 | 14 | ## Linear Programs 15 | 16 | When the functions $f_i$ for all $i$ are linear in $x$, the 17 | problem is called a linear program. These are much easier to solve, 18 | and the celebrated work of [George 19 | Dantzig](https://en.wikipedia.org/wiki/George_Dantzig) resulted in the 20 | [_Simplex Algorithm_](https://en.wikipedia.org/wiki/Simplex_algorithm) 21 | for linear programs. 22 | 23 | ## Quadratic Programs 24 | 25 | When the objective $f_0$ is quadratic in $x$, the problem is called a 26 | quadratic program. 27 | 28 | ## Cone Programs 29 | 30 | When $f_i(x) \leq 0$ for all $i$ constrain $x$ to lie in a convex cone $C$, the problem is called a cone program. 31 | Examples of $C$ are the positive orthant $\mathbf{R}^n$, the set of positive semidefinite matrices $\mathbf{S}_+^n$, and the second-order cone $\{(x,t) \in \mathbf{R}^n \times \mathbf{R}: \|x\| \leq t\}$. 32 | 33 | ## Integer and Mixed Integer Programs 34 | 35 | When all elements of $x$ are constrained to be integers, or even binary ($\{0,1\}$) values, the problem is called an integer program. If only a few elements of $x$ are constrained to be integers, then the problem is a mixed integer program. 36 | -------------------------------------------------------------------------------- /03-solving-convex-problems.Rmd: -------------------------------------------------------------------------------- 1 | # Solving Convex Problems 2 | 3 | Although convex problems can look very difficult (nonlinear, even 4 | nondifferentiable), they can be solved very efficiently like a linear 5 | program. So, how does one solve such problems in R? 6 | 7 | One possibility is to match the form of the problem to an existing 8 | solver routine. For example, the well-known 9 | [`optimx`](https://cran.r-project.org/package=optimx) package provides 10 | the following omnibus function: 11 | 12 | ```{r, eval = FALSE} 13 | optimx(par, fn, gr=NULL, hess=NULL, lower=-Inf, upper=Inf, 14 | method=c("Nelder-Mead","BFGS"), itnmax=NULL, hessian=FALSE, 15 | control=list(), 16 | ...) 17 | ``` 18 | 19 | Here one has to specify a vector of initial parameters (`par`), the 20 | objective function (`fn`), optional gradient (`gr`) and Hessian 21 | functions (`hess`) depending on the method used, and upper and lower 22 | bounds for the solution. Obviously, the objective and the constraints 23 | must be supported by the `optimx` routines. 24 | 25 | Another possibility is to use a package such as 26 | [`ROI`](https://cran.r-project.org/package=ROI), which provides 27 | interfaces to a number of solvers, including solvers for convex 28 | problems. It offers an object-oriented framework for defining 29 | optimization problems, but one still has to explicitly identify the 30 | type of every objective and constraint. If you can transform your 31 | problem into a cone program and use a standard cone program solver, 32 | then you can use `ROI` in a straightforward way. 33 | 34 | ## Verifying Convexity 35 | 36 | Even before attempting a solution, one has to verify a problem is 37 | convex. One can start with the basic definition 38 | \@ref{eq:convex-function} or use first or second order conditions 39 | $\nabla^2{f}\succeq 0.$ These often turn out to be tedious and hard 40 | to derive. 41 | 42 | Another possibility is to construct $f_i$ out of a library of basic 43 | examples or atoms that are convex. These could be combined using 44 | calculus rules and transformations that preserve convexity, yielding 45 | a problem that is automatically verified to be convex. 46 | 47 | ## Domain Specific Languages for Convex Optimization 48 | 49 | Domain Specific Languages (DSLs) are specialized languages for a 50 | particular application, implemented in a general purpose programming 51 | language. They have become useful for expressing, manipulating, and 52 | solving problems in specific contexts: circuit design (VHDL), graph 53 | layout (DOT), data (XML), etc. 54 | 55 | Over the last few years, specialized languages have become available for 56 | general convex optimization using the constructive approach discussed 57 | above. 58 | 59 | - [CVX](https://cvxr.com/cvx/) and [YALMIP](https://yalmip.github.io), both implemented in Matlab 60 | - [CVXPY](https//www.cvxpy.org) implemented in Python 61 | - [Convex.jl](https://github.com/JuliaOpt/Convex.jl) implemented in Julia 62 | - [CVXR](https://cvxr.rbind.io) implemented in R 63 | 64 | Such DSLs may result in code that is slightly slower, but they are 65 | extremely flexible and enable fast prototyping of novel methods. 66 | 67 | The last one, `CVXR`, is our focus and is described in a paper 68 | [@fu:naras:boyd:2019] that due to appear in the [Journal of Statistical 69 | Software](https://www.jstatsoft.org) any day now. 70 | 71 | ## References 72 | -------------------------------------------------------------------------------- /04-disciplined-convex-programming.Rmd: -------------------------------------------------------------------------------- 1 | # Disciplined Convex Programming 2 | 3 | > "All... new systems of notation are such that one can accomplish 4 | nothing by means of them which would not also be accomplished 5 | without them; but the advantage is that when such a system of 6 | notation corresponds to the innermost essence of frequently occuring 7 | needs, one can solve the problems belonging in that category, indeed 8 | can mechanically solve them in cases so complicated that without 9 | such an aid even the genius becomes powerless. Thus it is with the 10 | invention of calculating by letters in general; thus it was with the 11 | differential calculus..." 12 | > 13 | > ---C. F. Gauss to Schumacher, May 15, 1843 14 | 15 | > "That’s awesome! I was disappointed to not see a direct 16 | reference to DCP, but still, it’s pretty clear!" 17 | > 18 | > ---Stephen Boyd, on Gauss' letter to Schumacher, April 8, 2019 19 | 20 | 21 | ## Basic Convex Functions 22 | 23 | The following are some basic convex functions. 24 | 25 | - $x^p$ for $p \geq 1$ or $p \leq 0$; $-x^p$ for $0 \leq p \leq 1$ 26 | - $\exp(x)$, $-\log(x)$, $x\log(x)$ 27 | - $a^Tx + b$ 28 | - $x^Tx$; $x^Tx/y$ for $y>0$; $(x^Tx)^{1/2}$ 29 | - $||x||$ (any norm) 30 | - $\max(x_1, x_2, \ldots, x_n)$, $\log(e^x_{1}+ \ldots + e^x_{n})$ 31 | - $\log(\Phi(x))$, where $\Phi$ is Gaussian CDF 32 | - $\log(\text{det}X^{-1})$ for $X \succ 0$ 33 | 34 | 35 | ## Calculus Rules 36 | 37 | - _Nonnegative Scaling_: if $f$ is convex and $\alpha \geq 0$, then $\alpha f$ is convex 38 | - _Sum_: if $f$ and $g$ are convex, so is $f+g$ 39 | - _Affine Composition_: if $f$ is convex, so is $f(Ax+b)$ 40 | - _Pointwise Maximum_: if $f_1,f_2, \ldots, f_m$ are convex, so is $f(x) = \underset{i}{\text{max}}f_i(x)$ 41 | - _Partial Minimization_: if $f(x, y)$ is convex and $C$ is a convex set, then $g(x) = \underset{y\in C}{\text{inf}}f(x,y)$ is convex 42 | - _Composition_: if $h$ is convex and increasing and $f$ is convex, then $g(x) = h(f(x))$ is convex 43 | 44 | There are many other rules, but the above will get you far. 45 | 46 | 47 | ## Examples 48 | 49 | - Piecewise-linear function: $f(x) = \underset{i}{\text{max}}(a_i^Tx + b_i)$ 50 | - $l_1$-regularized least-squares cost: $||Ax-b||_2^2 + \lambda ||x||_1$ with $\lambda \geq 0$ 51 | - Sum of $k$ largest elements of $x$: $f(x) = \sum_{i=1}^mx_i - \sum_{i=1}^{m-k}x_{(i)}$ 52 | - Log-barrier: $-\sum_{i=1}^m\log(−f_i(x))$ (on $\{x | f_i(x) < 0\}$, $f_i$ convex) 53 | - Distance to convex set $C$: $f(x) = \text{dist}(x,C) =\underset{y\in C}{\text{inf}}||x-y||_2$ 54 | 55 | Except for log-barrier, these functions are nondifferentiable. 56 | 57 | -------------------------------------------------------------------------------- /05-a-simple-regression-example.Rmd: -------------------------------------------------------------------------------- 1 | # A Simple Regression Example 2 | 3 | ```{r, echo = FALSE} 4 | library(nnls) 5 | library(kableExtra) 6 | 7 | #' Print a matrix in a stylized way using row and column names if specified 8 | #' @param the matrix to be printed 9 | #' @param row_names optional row names to use can be math 10 | #' @param col_names optional col names to use can be math 11 | print_matrix <- function(m, row_names = NULL, col_names = NULL) { 12 | if (!is.null(row_names)) rownames(m) <- row_names 13 | if (!is.null(col_names)) colnames(m) <- col_names 14 | knitr::kable(m, format = "html") %>% 15 | kable_styling("striped") %>% 16 | column_spec(1:2, background = "#ececec") 17 | } 18 | ``` 19 | 20 | ## Goals 21 | 22 | - Basic introduction to `CVXR` 23 | - Exercise on formulating a different objective, demonstrating how 24 | `CVXR` works with native R functions 25 | - Exercises on formulating linear dependence constraints on estimates 26 | using linear algebra 27 | - Exercises on formulating monotonicity constraints using `CVXR` 28 | atoms 29 | 30 | ## Ordinary Least-Squares Regression 31 | 32 | Consider a simple linear regression problem, where it is desired to 33 | estimate a set of parameters using a least-squares criterion. 34 | 35 | We generate some synthetic data in which we know the model completely, 36 | i.e. 37 | 38 | $$ 39 | Y = X\beta + \epsilon, 40 | $$ 41 | 42 | where $Y$ is a $100\times 1$ vector, $X$ is a $100\times 10$ matrix, 43 | $\beta = [-4, -3, \ldots ,4, 5]$ is a $10\times 1$ vector, and 44 | $\epsilon \sim N(0, 1)$. 45 | ```{r} 46 | set.seed(123) 47 | n <- 50; p <- 10; 48 | beta <- -4:5 # beta is just -4 through 5. 49 | X <- matrix(rnorm(n * p), nrow=n) 50 | colnames(X) <- paste0("beta_", beta) 51 | Y <- X %*% beta + rnorm(n) 52 | ``` 53 | 54 | Given the data $X$ and $Y$, we can estimate the $\beta$ vector using the 55 | `lm` function in R, which fits a standard regression model. 56 | 57 | ```{r} 58 | ls.model <- lm(Y ~ 0 + X) # There is no intercept in our model above 59 | m <- matrix(coef(ls.model), ncol = 1) 60 | ``` 61 | 62 | ```{r, echo = FALSE} 63 | print_matrix(m, row_names = paste0("$\\beta_{", 1:p, "}$")) 64 | ``` 65 | These are the least-squares estimates and can be seen to be reasonably 66 | close to the original $\beta$ values -4 through 5. 67 | 68 | ## The `CVXR` Formulation 69 | 70 | The `CVXR` formulation states the above as an optimization problem: 71 | $$ 72 | \begin{array}{ll} 73 | \underset{\beta}{\mbox{minimize}} & \|y - X\beta\|_2^2, 74 | \end{array} 75 | $$ 76 | which directly translates into a problem that `CVXR` can solve as shown 77 | in the steps below. 78 | 79 | - Step 0. Load the `CVXR` library 80 | 81 | ```{r, message = FALSE} 82 | library(CVXR, warn.conflicts=FALSE) 83 | ``` 84 | 85 | - Step 1. Define the variable to be estimated 86 | 87 | ```{r} 88 | beta <- Variable(p) 89 | ``` 90 | 91 | - Step 2. Define the objective to be optimized 92 | 93 | ```{r} 94 | objective <- Minimize(sum((Y - X %*% beta)^2)) 95 | ``` 96 | 97 | Notice how the objective is specified using functions such as `sum`, 98 | `*%*`, and `^` that are familiar to R users despite the fact that 99 | `beta` is no ordinary R expression, but a `CVXR` expression. 100 | 101 | - Step 3. Create a problem to solve 102 | 103 | ```{r} 104 | problem <- Problem(objective) 105 | ``` 106 | 107 | - Step 4. Solve it! 108 | 109 | ```{r} 110 | result <- solve(problem) 111 | ``` 112 | 113 | - Step 5. Examine solution status and obtain objective value and estimate 114 | 115 | ```{r, echo = FALSE} 116 | solution_status <- result$status 117 | objective_value <- result$value 118 | solution <- result$getValue(beta) 119 | cat(sprintf("OLS Solution Status: %s, OLS Objective value: %f\n", solution_status, objective_value)) 120 | ``` 121 | 122 | We can indeed satisfy ourselves that the results we get match those 123 | from `lm`. 124 | 125 | ```{r} 126 | m <- cbind(result$getValue(beta), coef(ls.model)) 127 | ``` 128 | 129 | ```{r, echo = FALSE} 130 | print_matrix(m, row_names = paste0("$\\beta_{", 1:p, "}$"), col_names = c("CVXR est.", "lm est.")) 131 | ``` 132 | 133 | ### Exercise 134 | 135 | Modify the objective to perform least absolute deviation regression 136 | and solve the problem. Compare the results to OLS. Which objective has 137 | a lower value? 138 | 139 | _Hint_: In LAD regression, we minimize the sum of the absolute value 140 | of the residuals. Also, since the default solver `OSQP` has some 141 | issues at the time of this writing, do pass on the parameter `solver = 142 | "ECOS"` in your call to `CVXR::solve`. 143 | 144 | #### Solution 145 | 146 | ```{r} 147 | objective2 <- Minimize(sum(abs(Y - X %*% beta))) 148 | problem2 <- Problem(objective2) 149 | result2 <- solve(problem2, solver = "ECOS") 150 | cat(sprintf("LAD Solution Status: %s, LAD Objective value: %f\n", result2$status, result2$value)) 151 | m2 <- cbind(result2$getValue(beta), coef(ls.model)) 152 | ``` 153 | 154 | ```{r, echo = FALSE} 155 | print_matrix(m2, row_names = paste0("$\\beta_{", 1:p, "}$"), col_names = c("CVXR LAD est.", "lm est.")) 156 | ``` 157 | 158 | ```{r} 159 | cat("LAD objective value: %f, OLS objective value: %f\n", 160 | result2$value, result$value) 161 | ``` 162 | 163 | __N.B.__ Note the reuse of `beta` in `objective2`. The value of 164 | `beta` will change depending on the problem context, and the 165 | function `result$getValue()` or `result2$getValue()` will account for 166 | the context as shown below. 167 | 168 | ```{r} 169 | m3 <- cbind(result$getValue(beta), result2$getValue(beta)) 170 | ``` 171 | 172 | ```{r, echo = FALSE} 173 | print_matrix(m3, row_names = paste0("$\\beta_{", 1:p, "}$"), col_names = c("Problem 1 est.", "Problem 2 est.")) 174 | ``` 175 | 176 | ## Adding Constraints 177 | 178 | On the surface, it appears that we have replaced one call to `lm` with 179 | at least five or six lines of new R code. On top of that, the code 180 | actually runs slower, so it is not clear what we really achieved. 181 | 182 | However, suppose we knew that the $\beta$s were nonnegative and wished to 183 | take this fact into account in our model. This 184 | is [nonnegative least squares regression](https://en.wikipedia.org/wiki/Non-negative_least_squares), and 185 | `lm` would no longer do the job. 186 | 187 | In `CVXR`, the modified problem merely requires the addition of a constraint to the 188 | problem definition. 189 | 190 | ```{r} 191 | problem <- Problem(objective, constraints = list(beta >= 0)) 192 | result <- solve(problem) 193 | betaEstimate <- result$getValue(beta) 194 | ``` 195 | 196 | ```{r, echo = FALSE} 197 | m <- matrix(betaEstimate, ncol = 1) 198 | print_matrix(m, row_names = paste0("$\\beta_{", 1:p, "}$")) 199 | ``` 200 | 201 | We can verify once again that these values are comparable to those 202 | obtained from another R package, 203 | say [nnls]( https://CRAN.R-project.org/package=nnls). 204 | 205 | ```{r} 206 | nnls.fit <- nnls::nnls(X, Y)$x 207 | m <- cbind(betaEstimate, nnls.fit) 208 | ``` 209 | 210 | ```{r, echo = FALSE} 211 | print_matrix(m, row_names = paste0("$\\beta_{", 1:p, "}$"), col_names = c("CVXR NNLS est.", "nnls est.")) 212 | ``` 213 | 214 | ### Exercise 215 | 216 | Suppose it is known that $\sum_{i=1}^4\beta_i \leq 217 | 0$. Modify the original OLS problem to add this constraint. 218 | 219 | #### Solution 220 | 221 | The obvious solution is to add a constraint of the form 222 | 223 | ```{r, eval = FALSE} 224 | constraint1 <- beta[1] + beta[2] + beta[3] + beta[4] <= 0 225 | ``` 226 | 227 | but it is generally easier working with matrices in `CVXR`, and so 228 | we construct a row vector with zeros everywhere except in positions 1 229 | through 4. 230 | 231 | ```{r} 232 | A <- matrix(c(rep(1, 4), rep(0, 6)), nrow = 1) 233 | ``` 234 | 235 | ```{r, echo = FALSE} 236 | print_matrix(A, col_names = paste0("$\\beta_{", 1:p, "}$")) 237 | ``` 238 | 239 | The sum constraint on $\beta$ is therefore 240 | $$ 241 | A\beta \leq 0 242 | $$ 243 | which we express in R as 244 | 245 | ```{r} 246 | constraint1 <- A %*% beta <= 0 247 | ``` 248 | 249 | We are ready to solve the problem. 250 | 251 | ```{r} 252 | problem <- Problem(objective, constraints = list(constraint1)) 253 | ex1 <- solve(problem) 254 | ``` 255 | 256 | And we can get the estimates of $\beta$. 257 | ```{r} 258 | betaEstimate <- ex1$getValue(beta) 259 | ``` 260 | 261 | ```{r, echo = FALSE} 262 | m <- matrix(betaEstimate, ncol = 1) 263 | print_matrix(m, row_names = paste0("$\\beta_{", 1:p, "}$")) 264 | ``` 265 | 266 | ### Exercise 267 | 268 | Add an additional constraint to the previous exercise that 269 | $\beta_i \leq 4$ for $i=5,\ldots,10$. 270 | 271 | #### Solution 272 | 273 | We create a diagonal matrix with ones along the diagonal entries $i=5,\ldots,10$. 274 | 275 | ```{r} 276 | B <- diag(c(rep(0, 4), rep(1, 6))) 277 | ``` 278 | 279 | ```{r, echo = FALSE} 280 | print_matrix(B, row_names = paste0("$\\beta_{", 1:p, "}$"), col_names = paste0("$\\beta_{", 1:p, "}$")) 281 | ``` 282 | 283 | So this new constraint is nothing but 284 | 285 | ```{r} 286 | constraint2 <- B %*% beta <= 4 287 | problem2 <- Problem(objective, constraints = list(constraint1, constraint2)) 288 | ex2 <- solve(problem2) 289 | betaEstimate <- ex2$getValue(beta) 290 | ``` 291 | 292 | ```{r, echo = FALSE} 293 | m <- matrix(betaEstimate, ncol = 1) 294 | print_matrix(m, row_names = paste0("$\\beta_{", 1:p, "}$")) 295 | ``` 296 | 297 | ### Exercise 298 | 299 | Solve the OLS regression problem under the constraint that the 300 | $\beta_i$ are nonnegative and monotonically nondecreasing. 301 | 302 | _Hint_: What function in R computes lagged differences? 303 | 304 | #### Solution 305 | 306 | This requires some additional knowledge about R and `CVXR` 307 | functions. The `base::diff` generic function generates lagged 308 | differences of any order. `CVXR` provides a method for the generic. So 309 | the monotonicity constraint can be succintly expressed as `diff(beta) >= 0`. 310 | 311 | ```{r} 312 | problem3 <- Problem(objective, 313 | constraints = list(beta >= 0, diff(beta) >= 0)) 314 | ex3 <- solve(problem3) 315 | betaEstimate <- ex3$getValue(beta) 316 | ``` 317 | 318 | ```{r, echo = FALSE} 319 | m <- matrix(betaEstimate, ncol = 1) 320 | print_matrix(m, row_names = paste0("$\\beta_{", 1:p, "}$")) 321 | ``` 322 | 323 | ### Exercise 324 | 325 | Fit OLS with just the following order constraints on $\beta$: 326 | $\beta_{i} \leq \beta_{i+1}$ for $i=1,\ldots, 4$ and $\beta_i \geq 327 | \beta_{i+1}$ for $i=5,\ldots,p$. 328 | 329 | #### Solution 330 | 331 | We have to combine all that we have learned earlier. 332 | 333 | ```{r} 334 | D1 <- cbind(diag(5), diag(0, 5)) 335 | D2 <- cbind(matrix(0, 6, 4), diag(6)) 336 | constraints = list(diff(D1 %*% beta) >= 0, diff(D2 %*% beta) <= 0) 337 | problem4 <- Problem(objective, constraints) 338 | ex4 <- solve(problem4) 339 | betaEstimate <- ex4$getValue(beta) 340 | ``` 341 | 342 | ```{r, echo = FALSE} 343 | m <- matrix(betaEstimate, ncol = 1) 344 | print_matrix(m, row_names = paste0("$\\beta_{", 1:p, "}$")) 345 | ``` 346 | 347 | ## Summary 348 | 349 | This introduction demonstrates a chief advantage of `CVXR`: 350 | _flexibility_. Users can quickly modify and re-solve a problem, which 351 | is ideal for prototyping and experimenting with new statistical 352 | methods. The `CVXR` syntax is simple and mathematically 353 | intuitive. Furthermore, `CVXR` combines seamlessly with native R code 354 | as well as several popular packages, allowing it to be incorporated 355 | easily into a larger analytical framework. The user is free to 356 | construct statistical estimators that are solutions to a convex 357 | optimization problem where there may not be a closed form solution or 358 | even an implementation. Later, we will see how such solutions can be 359 | used with resampling techniques like the bootstrap to estimate 360 | variability. 361 | -------------------------------------------------------------------------------- /06-how-cvxr-works.Rmd: -------------------------------------------------------------------------------- 1 | # How `CVXR` Works 2 | 3 | Let us consider the nonnegative least squares regression example once 4 | again. 5 | 6 | ```{r, eval = FALSE} 7 | beta <- Variable(p) 8 | objective <- Minimize(sum((Y - X %*% beta)^2)) 9 | constraints <- list(beta >= 0) 10 | problem <- Problem(objective, constraints) 11 | result <- solve(problem) 12 | solution_status <- result$status 13 | objective_value <- result$value 14 | beta_hat <- result$getValue(beta) 15 | ``` 16 | 17 | ## Variables 18 | 19 | The `CVXR::Variable` function constructs an S4 class describing the 20 | argument of an optimization problem. 21 | 22 | - `Variable()` specifies a 1-vector, essentially a scalar 23 | - `Variable(m)` specifies an $m$-vector 24 | - `Variable(m, n)` specifies an $m\times n$ matrix 25 | 26 | There are also S4 classes representing certain special constructs such 27 | as semidefinite matrices as we shall see later. 28 | 29 | ## Objectives, Constraints, and Problems 30 | 31 | The objective function should yield a scalar value. `CVXR` provides 32 | the `Minimize` and `Maximize` functions that take an `Expression` 33 | (another S4 class) as an argument. Standard arithmetic and generics 34 | are overloaded so that one may use any R function in the construction. 35 | 36 | Constraints are specified as a list. One may construct this list 37 | directly or via some iterative computation, once again using various R 38 | and `CVXR` functions in the process. 39 | 40 | In the above problem, the objective is 41 | 42 | ```{r, eval = FALSE} 43 | objective <- Minimize(sum((Y - X %*% beta)^2)) 44 | ``` 45 | 46 | which uses the `CVXR::Minimize` function along with standard R 47 | functions such as `sum` and squaring. This allows one to seamlessly 48 | work with all standard R constructs. However, the same objective may 49 | also be specified as 50 | 51 | ```{r, eval = FALSE} 52 | objective <- Minimize(sum_squares(Y - X %*% beta)) 53 | ``` 54 | 55 | using the `CVXR::sum_squares` function. As you use `CVXR` 56 | more and more, you will need to refer to the [`CVXR` functions 57 | list](https://cvxr.rbind.io/cvxr_functions/) to learn about these 58 | built-in functions. 59 | 60 | A problem takes an objective and an optional constraint. It serves as 61 | the complete representation of the problem along with associated 62 | data, like the `Y` and `X` in the code snippet above. 63 | 64 | ## Solving the Problem 65 | 66 | Calling the `solve` function on a problem sets several things in motion. 67 | 68 | 1. The problem is verified for convexity. If it is not convex, the 69 | solve attempt fails with an error message and a non-optimal status. 70 | 71 | 2. The problem along with the data is converted into a canonical form. 72 | 73 | 3. The problem is analyzed and classified according to its type: LP, 74 | QP, SDP, etc. 75 | 76 | 4. Among the available solvers, a suitable one is chosen that can 77 | handle the problem. Two open source solvers are built-in: Embedded 78 | Conic Solver (ECOS) and Splitting Conic Solver (SCS), but there is 79 | also support for commercial solvers. 80 | 81 | 5. The canonicalized data structures (matrices, cone dimensions) along 82 | with solver options, if any, are dispatched to the solver in 83 | appropriate form. 84 | 85 | 6. Finally, the results from the solver along with some accessor 86 | functions for retrieving the solution and other quantities in the 87 | context of the solved problem are prepared and returned to the 88 | caller. 89 | 90 | There are several modes of failure that can occur. The problem may not 91 | be convex, and that is indicated via an error message. However, even 92 | when the problem is convex, the solver may not converge to a 93 | solution. The latter could be due to a number of reasons: tight 94 | tolerances, too few iterations, numerical issues, etc. Therefore, 95 | the solution status should always be examined. 96 | 97 | One option that can be very useful is verbosity, and this is specified 98 | by simply passing another parameter to `CVXR::solve`. 99 | 100 | ```{r} 101 | result <- solve(problem, verbose = TRUE) 102 | ``` 103 | 104 | ## Solver Options 105 | 106 | Solver options are unique to the chosen solver, so any arguments to 107 | `CVXR::solve` besides the three documented above are simply passed 108 | along to the solver. The reference for the specific solver must be 109 | consulted to set these options. 110 | -------------------------------------------------------------------------------- /07-logistic-regression.Rmd: -------------------------------------------------------------------------------- 1 | # Logistic Regression 2 | 3 | ## Goals 4 | 5 | - Formulating the logistic regression likelihood using `CVXR` atoms 6 | - Example comparing `CVXR` results with R results from `glm` 7 | - Exercise on extracting fitted values from `CVXR` logistic fit using 8 | lexically scoped `CVXR` facilities. 9 | 10 | ## Logistic Regression Problem 11 | 12 | In logistic regression [@james2013], the response $y_i$ is 13 | binary: 0 or 1. (In a classification setting, the values of the 14 | response would represent class membership in one of two classes.) The 15 | conditional response given covariates $x$ is modeled as 16 | 17 | \[ 18 | y|x \sim \mbox{Bernoulli}(g_{\beta}(x)), 19 | \] 20 | 21 | where $g_{\beta}(x) = \frac{1}{1 + 22 | e^{-x^T\beta}}$ is the logistic function. We want to maximize the 23 | log-likelihood function, yielding the optimization problem 24 | 25 | \[ 26 | \begin{array}{ll} 27 | \underset{\beta}{\mbox{maximize}} & \sum_{i=1}^m \{ 28 | y_i\log(g_{\beta}(x_i)) + (1-y_i)\log(1 - g_{\beta}(x_i)) \}. 29 | \end{array} 30 | \] 31 | 32 | One may be tempted to use `log(1 + exp(X %*% beta))` as in 33 | conventional `R` syntax. However, this representation of $f(z)$ 34 | violates the DCP composition rule, so the `CVXR` parser will reject 35 | the problem even though the objective is convex. Users who wish to 36 | employ a function that is convex, but not DCP compliant should check 37 | the documentation for a custom atom or consider a different 38 | formulation. 39 | 40 | `CVXR` provides the [`logistic` 41 | atom](https://cvxr.rbind.io/cvxr_functions/) as a shortcut for $f(z) = 42 | \log(1 + e^z)$. 43 | 44 | ## Example 45 | 46 | We use example 4.6.2 of [@james2013] with the `Smarket` data where a 47 | `glm` is fit as follows. 48 | 49 | ```{r} 50 | library(ISLR) 51 | data(Smarket) 52 | glmfit <- stats::glm(formula = Direction ~ Lag1 + Lag2 + Lag3 + Lag4 + Lag5 + Volume, 53 | data = Smarket, family = binomial) 54 | ``` 55 | 56 | The `CVXR` formulation merely has to specify the objective after 57 | setting up the data matrices appropriately. 58 | 59 | ```{r} 60 | y <- as.integer(Smarket$Direction) - 1L 61 | X <- cbind(1, as.matrix(Smarket[, c("Lag1", "Lag2", "Lag3", "Lag4", "Lag5", "Volume")])) 62 | p <- ncol(X) 63 | beta <- Variable(p) 64 | objective <- -sum(X[y <= 0, ] %*% beta) - sum(logistic(-X %*% beta)) 65 | problem <- Problem(Maximize(objective)) 66 | result <- solve(problem) 67 | beta_hat <- result$getValue(beta) 68 | ``` 69 | 70 | We can compare with the standard `stats::glm` estimate. 71 | 72 | ```{r, echo = FALSE} 73 | print_matrix(cbind(beta_hat, coef(glmfit)), row_names = paste0("$\\beta_{", seq.int(0, p-1L), "}$"), col_names = c("CVXR", "GLM")) 74 | ``` 75 | 76 | ## Exercise 77 | 78 | Standard `stats::glm` returns an object that has fitted values: 79 | `glmfit$fitted.values`. How would you compute the fitted values from 80 | `CVXR`? 81 | 82 | _Hint:_ The `result$getValue()` evalutes expressions in the 83 | problem context. 84 | 85 | ### Solution 86 | 87 | A key feature of `CVXR` is that it exploits lexical scoping built into 88 | R. So one can evaluate various functions of the variables that are 89 | solutions to the optimization problem. 90 | 91 | The fitted values can be computed using 92 | 93 | ```{r} 94 | fitted_values <- 1 / (1 + exp(result$getValue(-X %*% beta))) 95 | ``` 96 | 97 | We can also satisfy ourselves that the fitted values match the `glm` 98 | estimate, by computing the sum of squared differences. 99 | 100 | ```{r} 101 | sum((fitted_values - glmfit$fitted.values))^2 102 | ``` 103 | 104 | Similarly, the log-odds, $X\hat{\beta}$ , where $\hat{\beta}$ is the 105 | logistic regression estimate, can be computed as follows. 106 | 107 | ```{r} 108 | log_odds <- result$getValue(X %*% beta) 109 | ``` 110 | 111 | ## References 112 | -------------------------------------------------------------------------------- /08-isotonic-regression.Rmd: -------------------------------------------------------------------------------- 1 | # Isotonic Regression 2 | 3 | ```{r, echo = FALSE, message = FALSE} 4 | library(isotone) 5 | ``` 6 | 7 | ## Goals 8 | 9 | - Formulate the isotonic regression objective using some new `CVXR` 10 | atoms 11 | - Compare with results from `isotone` package 12 | - Exercise on handling ties, the secondary method of `isotone`, using 13 | `CVXR` atoms 14 | - Exercise on handling ties, the tertiary method of `isotone`, using 15 | `CVXR` atoms 16 | 17 | 18 | [Isotonic regression](https://en.wikipedia.org/wiki/Isotonic_regression) is 19 | regression with monotonicity constraints. There are several packages in R 20 | to fit isotonic regression models. In this example, we 21 | consider [`isotone`](https://cran.r-project.org/package=isotone), which 22 | uses a pooled-adjacent-violators algorithm (PAVA) and active set 23 | methods to perform the fit. 24 | 25 | ## Pituitary Data Example 26 | 27 | We will use data from the `isotone` package [@isotone] on the size of 28 | pituitary fissures for 11 subjects between 8 and 14 years of age. 29 | 30 | ```{r} 31 | data("pituitary") 32 | str(pituitary) 33 | ``` 34 | 35 | Since the size is expected to increase with age, an isotonic fit is 36 | suggested, so we fit using the `isotone` package. 37 | 38 | ```{r} 39 | res_p <- with(pituitary, gpava(age, size)) 40 | ``` 41 | 42 | The `CVXR` formulation expresses this pretty much in the mathematical 43 | form. We define a variable `x` of size `n`, the number of 44 | observations. The objective to be minimized is the least-squares error 45 | (`cvxr_norm`), yet another way of specifying least-squares loss. The 46 | monotonicity is specified using the `diff` function. 47 | 48 | ### Exercise 49 | 50 | Can you explain why `CVXR` provide functions such as `cvxr_norm` and `p_norm` 51 | rather than just plain `pnorm`? 52 | 53 | #### Solution 54 | 55 | In R, `pnorm` is already defined and refers to the density of the normal 56 | distribution. So we use a new generic `cvxr_norm` or `p_norm` (see 57 | [`CVXR` functions](https://cvxr.rbind.io/cvxr_functions/) to avoid 58 | confusion. The function `cvxr_norm` provides some specialized norms for matrices, 59 | whereas `p_norm` allows one to specify $p$. 60 | 61 | ```{r} 62 | x_p <- with(pituitary, { 63 | n <- length(size) 64 | x <- Variable(n) 65 | objective <- Minimize(cvxr_norm(size - x, 2)) 66 | constraint <- list(diff(x) >= 0) 67 | problem <- Problem(objective, constraint) 68 | result <- solve(problem) 69 | result$getValue(x) 70 | }) 71 | ``` 72 | As the output below shows, the results are very close. 73 | 74 | ```{r} 75 | print_matrix(cbind(res_p$x, x_p), col_names = c("isotone", "CVXR")) 76 | ``` 77 | 78 | ## Handling Ties 79 | 80 | Package `isotone` provides additional methods for handling tied data 81 | besides the default `ties = "primary"` method; `ties = "secondary"` 82 | enforces equality within ties, and `ties = "tertiary"` enforces 83 | monotonicity on the means. (The latter may cause individual fits to be 84 | non-monotonic.) 85 | 86 | ```{r} 87 | res_s <- with(pituitary, gpava(age, size, ties = "secondary")) 88 | res_t <- with(pituitary, gpava(age, size, ties = "tertiary")) 89 | ``` 90 | 91 | ### Exercise 92 | 93 | Implement the secondary method of ties using `CVXR` and compare the 94 | results with the `isotone` package. 95 | 96 | #### Solution 97 | 98 | The secondary method for ties just requires an additional constraint 99 | to enforce equality within tied values; no other modification is 100 | necessary. We do this below by figuring out the tied observation 101 | indices using `base::split` and forcing those `x` values to be equal 102 | (i.e. `diff` == 0). 103 | 104 | ```{r} 105 | x_s <- with(pituitary, { 106 | n <- length(size) 107 | x <- Variable(n) 108 | objective <- Minimize(p_norm(size - x, 2)) 109 | secondary_constraints <- lapply(base::split(x = seq_len(n), 110 | f = age), 111 | function(i) diff(x[i]) == 0) 112 | constraint <- c(diff(x) >= 0, 113 | secondary_constraints) 114 | problem <- Problem(objective, constraint) 115 | solve(problem)$getValue(x) 116 | }) 117 | ``` 118 | 119 | Here's the comparison table. 120 | 121 | ```{r, echo = FALSE} 122 | m <- cbind(res_s$x, x_s) 123 | print_matrix(m, col_names = c("Isotone (S)", "CVXR (S)")) 124 | ``` 125 | 126 | ### Exercise 127 | 128 | Implement the tertiary method for ties using `CVXR` and compare with 129 | the `isotone` package. 130 | 131 | #### Solution 132 | 133 | The tertiary method requires computing the block means 134 | for use in enforcing monotonicity. We call the 135 | [`CVXR::vstack`](https://cvxr.rbind.io/cvxr_functions/) function to 136 | create a single vector of the block means. 137 | 138 | Basically, `CVXR::hstack` is the equivalent of `base::cbind` and 139 | `CVXR::vstack` is the equivalent of `base::rbind`. 140 | 141 | ```{r} 142 | x_t <- with(pituitary, { 143 | n <- length(size) 144 | x <- Variable(n) 145 | objective <- Minimize(p_norm(size - x, 2)) 146 | blocks <- base::split(x = seq_len(n), 147 | f = pituitary$age) 148 | block_means <- lapply(blocks, function(i) { 149 | v <- numeric(n) 150 | v[i] <- 1.0 / length(i) 151 | matrix(v, nrow = 1) %*% x 152 | }) 153 | block_mean_vector <- do.call(vstack, block_means) 154 | constraint <- list(diff(block_mean_vector) >= 0) 155 | problem <- Problem(objective, constraint) 156 | solve(problem)$getValue(x) 157 | }) 158 | ``` 159 | 160 | Here's the comparison table. 161 | 162 | ```{r, echo = FALSE} 163 | m <- cbind(res_t$x, x_t) 164 | print_matrix(m, col_names = c("Isotone (T)", "CVXR (T)")) 165 | ``` 166 | -------------------------------------------------------------------------------- /09-lasso-and-elastic-net.Rmd: -------------------------------------------------------------------------------- 1 | # Lasso and Elastic Net 2 | 3 | ```{r, echo = FALSE, message = FALSE} 4 | library(glmnet) 5 | ``` 6 | 7 | ## Goals 8 | 9 | - Formulate lasso and elastic net regression models 10 | - Compare with results from `glmnet` package 11 | - Use loss functions besides squared loss with elastic net penalty 12 | 13 | ## Regularized Regression 14 | 15 | Often in applications, we encounter problems that require 16 | regularization to prevent overfitting, introduce sparsity, facilitate 17 | variable selection, or impose prior distributions on parameters. Two 18 | of the most common regularization functions are the $l_1$-norm and 19 | squared $l_2$-norm, combined in the elastic net regression model 20 | (@elasticnet and @glmnet). 21 | 22 | \[ 23 | \begin{array}{ll} 24 | \underset{\beta}{\mbox{minimize}} & \frac{1}{2m}\|y - X\beta\|_2^2 + 25 | \lambda(\frac{1-\alpha}{2}\|\beta\|_2^2 + \alpha\|\beta\|_1). 26 | \end{array} 27 | \] 28 | 29 | Here $\lambda \geq 0$ is the overall regularization weight and 30 | $\alpha \in [0,1]$ controls the relative $l_1$ versus squared $l_2$ 31 | penalty. Thus, this model encompasses both ridge ($\alpha = 0$) and 32 | lasso ($\alpha = 1$) regression. 33 | 34 | It is convenient to define a function that calculates just the 35 | regularization term given the variable and penalty parameters. This 36 | modular approach will allow us to easily incorporate elastic net 37 | regularization into other regression models as we will see below. 38 | 39 | ```{r} 40 | #' Define the elastic penalty 41 | #' @param beta the arg min variable 42 | #' @param lambda the penalization parameter 43 | #' @param alpha the elastic net parameter, 0 = ridge, 1 = lasso 44 | elastic_penalty <- function(beta, lambda = 0, alpha = 0) { 45 | ridge <- (1 - alpha) / 2 * sum_squares(beta) 46 | lasso <- alpha * cvxr_norm(beta, 1) 47 | lambda * (lasso + ridge) 48 | } 49 | ``` 50 | 51 | We generate some synthetic sparse data for this example. 52 | 53 | ```{r} 54 | ## Problem data 55 | set.seed(4321) 56 | p <- 10 57 | n <- 500 58 | DENSITY <- 0.25 # Fraction of non-zero beta 59 | beta_true <- matrix(rnorm(p), ncol = 1) 60 | idxs <- sample.int(p, size = floor((1 - DENSITY) * p), replace = FALSE) 61 | beta_true[idxs] <- 0 62 | sigma <- 45 63 | X <- matrix(rnorm(n * p, sd = 5), nrow = n, ncol = p) 64 | eps <- matrix(rnorm(n, sd = sigma), ncol = 1) 65 | Y <- X %*% beta_true + eps 66 | ``` 67 | 68 | We fit the elastic net model for several values of $\lambda$. 69 | 70 | ```{r} 71 | TRIALS <- 10 72 | beta_vals <- matrix(0, nrow = p, ncol = TRIALS) 73 | lambda_vals <- 10^seq(-2, log10(50), length.out = TRIALS) 74 | ``` 75 | 76 | ```{r} 77 | beta <- Variable(p) 78 | loss <- sum_squares(Y - X %*% beta) / (2 * n) 79 | ## Elastic-net regression LASSO 80 | alpha <- 1 81 | beta_vals <- sapply(lambda_vals, 82 | function (lambda) { 83 | obj <- loss + elastic_penalty(beta, lambda, alpha) 84 | prob <- Problem(Minimize(obj)) 85 | result <- solve(prob) 86 | result$getValue(beta) 87 | }) 88 | ``` 89 | 90 | We can now get a table of the coefficients. 91 | 92 | ```{r, echo = FALSE} 93 | print_matrix(round(beta_vals, 3), 94 | row_names = sprintf("$\\beta_{%d}$", seq_len(p)), 95 | col_names = sprintf("$\\lambda = %.3f$", lambda_vals)) 96 | ``` 97 | 98 | We plot the coefficients against the regularization. 99 | 100 | ```{r} 101 | plot(0, 0, type = "n", main = "CVXR Regularization Path for Lasso Regression", 102 | xlab = "Log Lambda", ylab = "Coefficients", 103 | ylim = c(-1, 2), xlim = c(-4, 4)) 104 | matlines(log(lambda_vals), t(beta_vals)) 105 | ``` 106 | 107 | We then compare with the `glmnet` results. 108 | 109 | ```{r} 110 | model_net <- glmnet(X, Y, family = "gaussian", alpha = alpha, 111 | lambda = lambda_vals, 112 | standardize = FALSE, 113 | intercept = FALSE, 114 | thresh = 1e-8) 115 | ## Reverse order to match beta_vals 116 | coef_net <- as.data.frame(as.matrix(coef(model_net)[-1, seq(TRIALS, 1, by = -1)])) 117 | ``` 118 | 119 | ```{r, echo = FALSE} 120 | print_matrix(round(coef_net, 3), 121 | row_names = sprintf("$\\beta_{%d}$", seq_len(p)), 122 | col_names = sprintf("$\\lambda = %.3f$", lambda_vals)) 123 | ``` 124 | 125 | ### Exercise 126 | 127 | [A Stack Overflow 128 | Question](https://stats.stackexchange.com/questions/334007/elastic-net-in-glmnet-vs-cvxr)! 129 | 130 | _I'm attempting to match some simple results in R using `glmnet` and 131 | `CVXR`. I have the following code._ 132 | ```{r} 133 | library(glmnet) 134 | data(QuickStartExample) 135 | x <- QuickStartExample$x 136 | y <- QuickStartExample$y 137 | catn <- function(...) cat(..., "\n") 138 | objective_value <- function(y, x, coefs, lambda, alpha) { 139 | n <- nrow(x) 140 | ridge <- sum(coefs^2) ; l1 <- sum(abs(coefs)) 141 | sum((y - (x %*% coefs))^2) / (2 * n) + lambda * ((1 - alpha) / 2 * ridge + alpha * l1) 142 | } 143 | alpha <- 0; lambda <- 1; 144 | fit <- glmnet(x, y, intercept=F, standardize=F, lambda=1, alpha=0) 145 | ``` 146 | _which gives me one set of coefficients and the objective function 147 | value `2.559086` via_ 148 | 149 | ```{r} 150 | objective_value(y, x, coef(fit)[-1, ], lambda, alpha) 151 | ``` 152 | _but this `CVXR` code _ 153 | ```{r} 154 | beta <- Variable(20) 155 | elastic_reg <- function(beta, lambda = 0, alpha = 0) { 156 | ridge <- (1 - alpha) * sum(beta^2) * .5 157 | lasso <- alpha * p_norm(beta, 1) 158 | lambda * (lasso + ridge) 159 | } 160 | loss <- sum((y - x %*% beta)^2)/(2*length(y)) 161 | obj <- loss + elastic_reg(beta, lambda = 1, 0) 162 | prob <- Problem(Minimize(obj)) 163 | result <- solve(prob) 164 | print(result$value) 165 | ``` 166 | _gives a different objective value `2.859259` and a somewhat different set of 167 | coefficients. Can you help?_ 168 | 169 | #### Solution 170 | 171 | To compare glmnet results to `CVXR` for the Gaussian case, it is 172 | advisable to standardize the response per the `glmnet` documentation. 173 | Note also that `glmnet` uses $n$ rather than $n-1$ in the denominator 174 | for $y$. This will ensure that the $\lambda$ is on the same scale as 175 | shown below. 176 | 177 | ```{r} 178 | catn <- function(...) cat(..., "\n") 179 | ## Standardize the y 180 | y_s <- local({ 181 | n <- length(y) 182 | m <- mean(y); s <- as.numeric(sqrt(var(y) * (n - 1) / n)); 183 | result <- (y - m) / s ## scale using 1/n 184 | attr(result, "scaled:center") <- m 185 | attr(result, "scaled:scale") <- s 186 | result 187 | }) 188 | ``` 189 | We can do a comparison on the standardized $y$. First, the `glmnet` answer: 190 | 191 | ```{r} 192 | ## STANDARDIZED COMPARISON 193 | fit_s <- glmnet(x, y_s, intercept=F, standardize=F, lambda = lambda, alpha=alpha) 194 | catn("Glmnet objective (scaled y)", 195 | objective_value(y_s, x, coef(fit_s)[-1], lambda, alpha)) 196 | ``` 197 | 198 | Next, the `CVXR` answer: 199 | 200 | ```{r} 201 | elastic_reg <- function(beta, lambda = 0, alpha = 0) { 202 | ridge <- (1 - alpha) / 2 * sum_squares(beta) 203 | lasso <- alpha * p_norm(beta, 1) 204 | lambda * (lasso + ridge) 205 | } 206 | loss <- sum_squares(y_s - x %*% beta) / (2 * nrow(x)) 207 | obj <- loss + elastic_reg(beta, lambda = lambda, alpha) 208 | prob <- Problem(Minimize(obj)) 209 | beta_est <- solve(prob)$getValue(beta) 210 | catn("CVXR objective (scaled y):", objective_value(y_s, x, beta_est, lambda, alpha)) 211 | ``` 212 | 213 | To work on the non-standardized scale, we need to match the `lamba` 214 | values as noted by the `glmnet` authors in Appendix 2 of the package 215 | vignette [@glmnet]. 216 | 217 | ```{r} 218 | ## NONSTANDARDIZED COMPARISON 219 | fit <- glmnet(x, y, intercept=F, standardize=F, lambda = lambda, alpha=alpha) 220 | catn("Glmnet objective (unscaled y)", objective_value(y, x, coef(fit)[-1], lambda, alpha)) 221 | ``` 222 | 223 | ```{r} 224 | loss <- sum_squares(y - x %*% beta) / (2 * nrow(x)) 225 | obj <- loss + elastic_reg(beta, lambda = lambda / attr(y_s, "scaled:scale"), alpha) 226 | prob <- Problem(Minimize(obj)) 227 | beta_est <- solve(prob)$getValue(beta) 228 | catn("CVXR objective (unscaled y)", objective_value(y, x, beta_est, lambda, alpha)) 229 | ``` 230 | 231 | Finally, we can check that the coefficients are close enough. 232 | 233 | ```{r} 234 | print_matrix(round(cbind(beta_est, coef(fit)[-1]), 3), 235 | row_names = sprintf("$\\beta_{%d}$", seq_len(20)), 236 | col_names = c("CVXR", "GLMNET")) 237 | ``` 238 | 239 | ### Exercise 240 | 241 | Using the data (`X`, `Y`) above, solve an elastic net problem with 242 | [Huber loss](https://cvxr.rbind.io/cvxr_functions/) using the Huber 243 | threshold $M = 0.5$. 244 | 245 | #### Solution 246 | 247 | Just set the loss as follows. 248 | 249 | ```{r} 250 | beta <- Variable(p) 251 | loss <- sum(huber(Y - X %*% beta, M = 0.5)) 252 | ## Elastic-net regression LASSO 253 | alpha <- 1 254 | beta_vals <- sapply(lambda_vals, 255 | function (lambda) { 256 | obj <- loss + elastic_penalty(beta, lambda, alpha) 257 | prob <- Problem(Minimize(obj)) 258 | result <- solve(prob) 259 | result$getValue(beta) 260 | }) 261 | ``` 262 | 263 | The estimates are below. 264 | 265 | ```{r, echo = FALSE} 266 | print_matrix(round(beta_vals, 3), 267 | row_names = sprintf("$\\beta_{%d}$", seq_len(p)), 268 | col_names = sprintf("$\\lambda = %.3f$", lambda_vals)) 269 | ``` 270 | -------------------------------------------------------------------------------- /10-near-isotonic-fit.Rmd: -------------------------------------------------------------------------------- 1 | # Nearly Isotonic Fits 2 | 3 | ## Goals 4 | 5 | - Formulate nearly-isotonic and nearly-convex fits using `CVXR` atoms 6 | - Use the bootstrap to estimate variance of estimates 7 | 8 | ```{r, message = FALSE, echo = FALSE} 9 | library(ggplot2) 10 | library(boot) 11 | ``` 12 | 13 | Given a set of data points $y \in {\mathbf R}^m$, 14 | @TibshiraniHoefling:2011 fit a nearly-isotonic approximation $\beta 15 | \in {\mathbf R}^m$ by solving 16 | 17 | $$ 18 | \begin{array}{ll} 19 | \underset{\beta}{\mbox{minimize}} & \frac{1}{2}\sum_{i=1}^m (y_i - \beta_i)^2 + \lambda \sum_{i=1}^{m-1}(\beta_i - \beta_{i+1})_+, 20 | \end{array} 21 | $$ 22 | 23 | where $\lambda \geq 0$ is a penalty parameter and $x_+ 24 | =\max(x,0)$. This can be directly formulated in `CVXR`. 25 | 26 | ## Global Warming Example 27 | 28 | As an 29 | example, we use global warming data from 30 | the 31 | [Carbon Dioxide Information Analysis Center (CDIAC)](http://cdiac.ess-dive.lbl.gov/ftp/trends/temp/jonescru/). The 32 | data points are the annual temperature anomalies relative to the 33 | 1961--1990 mean. 34 | 35 | ```{r} 36 | data(cdiac) 37 | str(cdiac) 38 | ``` 39 | 40 | Since we plan to fit the regression and also get some idea of the 41 | standard errors, we write a function that computes the fit for use in 42 | bootstrapping. 43 | 44 | ```{r} 45 | neariso_fit <- function(y, lambda) { 46 | m <- length(y) 47 | beta <- Variable(m) 48 | obj <- 0.5 * sum_squares(y - beta) + lambda * sum(pos(diff(beta))) 49 | prob <- Problem(Minimize(obj)) 50 | solve(prob)$getValue(beta) 51 | } 52 | ``` 53 | 54 | The `CVXR::pos` atom evaluates $x_+ = \max(x,0)$ elementwise on the input 55 | expression. 56 | 57 | The `boot` library provides all the tools for bootstrapping, but 58 | requires a statistic function that takes particular arguments: a data 59 | frame, followed by the bootstrap indices and any other arguments 60 | ($\lambda$ for instance). This is defined below. 61 | 62 | _NOTE_ In what follows, we use a very small number of bootstrap 63 | samples as the fits are time consuming. 64 | 65 | ```{r} 66 | neariso_fit_stat <- function(data, index, lambda) { 67 | sample <- data[index,] # Bootstrap sample of rows 68 | sample <- sample[order(sample$year),] # Order ascending by year 69 | neariso_fit(sample$annual, lambda) 70 | } 71 | ``` 72 | 73 | ```{r, eval = FALSE} 74 | set.seed(123) 75 | boot.neariso <- boot(data = cdiac, 76 | statistic = neariso_fit_stat, 77 | R = 10, lambda = 0.44) 78 | ci.neariso <- t(sapply(seq_len(nrow(cdiac)), 79 | function(i) boot.ci(boot.out = boot.neariso, conf = 0.95, 80 | type = "norm", index = i)$normal[-1])) 81 | data.neariso <- data.frame(year = cdiac$year, 82 | annual = cdiac$annual, 83 | est = boot.neariso$t0, 84 | lower = ci.neariso[, 1], 85 | upper = ci.neariso[, 2]) 86 | ``` 87 | 88 | We can now plot the fit and confidence bands for the nearly-isotonic 89 | fit. 90 | 91 | ```{r, eval = FALSE} 92 | (plot.neariso <- ggplot(data = data.neariso) + 93 | geom_point(mapping = aes(year, annual), color = "red") + 94 | geom_line(mapping = aes(year, est), color = "blue") + 95 | geom_ribbon(mapping = aes(x = year, ymin = lower,ymax = upper),alpha=0.3) + 96 | labs(x = "Year", y = "Temperature Anomalies") 97 | ) 98 | ``` 99 | The curve follows the data well, but exhibits some choppiness in 100 | regions with a steep trend. 101 | 102 | ### Exercise 103 | 104 | Fit a smoother curve using a nearly-convex fit described in the same 105 | paper: 106 | 107 | $$ 108 | \begin{array}{ll} 109 | \underset{\beta}{\mbox{minimize}} & \frac{1}{2}\sum_{i=1}^m (y_i - 110 | \beta_i)^2 + \lambda \sum_{i=1}^{m-2}(\beta_i - 2\beta_{i+1} + \beta_{i+2})_+ \end{array} 111 | $$ 112 | 113 | #### Solution 114 | 115 | This replaces the first difference term with an approximation to the 116 | second derivative at $\beta_{i+1}$. In `CVXR`, the only change 117 | necessary is the penalty line: replace `diff(x)` by 118 | `diff(x, differences = 2)`. 119 | 120 | ```{r, eval = FALSE} 121 | nearconvex_fit <- function(y, lambda) { 122 | m <- length(y) 123 | beta <- Variable(m) 124 | obj <- 0.5 * sum_squares(y - beta) + lambda * sum(pos(diff(beta, differences = 2))) 125 | prob <- Problem(Minimize(obj)) 126 | solve(prob)$getValue(beta) 127 | } 128 | 129 | nearconvex_fit_stat <- function(data, index, lambda) { 130 | sample <- data[index,] # Bootstrap sample of rows 131 | sample <- sample[order(sample$year),] # Order ascending by year 132 | nearconvex_fit(sample$annual, lambda) 133 | } 134 | 135 | set.seed(987) 136 | boot.nearconvex <- boot(data = cdiac, 137 | statistic = nearconvex_fit_stat, 138 | R = 5, 139 | lambda = 0.44) 140 | 141 | ci.nearconvex <- t(sapply(seq_len(nrow(cdiac)), 142 | function(i) boot.ci(boot.out = boot.nearconvex, conf = 0.95, 143 | type = "norm", index = i)$normal[-1])) 144 | data.nearconvex <- data.frame(year = cdiac$year, 145 | annual = cdiac$annual, 146 | est = boot.nearconvex$t0, 147 | lower = ci.nearconvex[, 1], 148 | upper = ci.nearconvex[, 2]) 149 | 150 | ``` 151 | 152 | The resulting curve for the nearly-convex fit is depicted below with 153 | 95\% confidence bands generated from $R = 5$ samples. Note the jagged 154 | staircase pattern has been smoothed out. 155 | 156 | 157 | ```{r, eval = FALSE} 158 | (plot.nearconvex <- ggplot(data = data.nearconvex) + 159 | geom_point(mapping = aes(year, annual), color = "red") + 160 | geom_line(mapping = aes(year, est), color = "blue") + 161 | geom_ribbon(mapping = aes(x = year, ymin = lower,ymax = upper),alpha=0.3) + 162 | labs(x = "Year", y = "Temperature Anomalies") 163 | ) 164 | ``` 165 | 166 | ## References 167 | -------------------------------------------------------------------------------- /11-speed-considerations.Rmd: -------------------------------------------------------------------------------- 1 | # Speed Considerations 2 | 3 | 4 | __NOTE__: The material in this section was written for `CVXR` pre 5 | 1.0. It no longer applies to versions 1.0 and above. Indeed `CVXR` is 6 | actually much faster now by default. We will update this section for 7 | 1.0 as appropriate later. 8 | 9 | -------------------------------------------------------------------------------- /12-pliable-lasso.Rmd: -------------------------------------------------------------------------------- 1 | # Pliable Lasso 2 | 3 | ## Goals 4 | 5 | - Demonstrate how to fit a complex model 6 | - Show how to apply an atom along a row/column axis 7 | 8 | ## Pliable Lasso Problem 9 | 10 | @tibsjhf:2017 propose a generalization of the lasso that allows the 11 | model coefficients to vary as a function of a general set of modifying 12 | variables, such as gender, age, and time. The pliable lasso model has 13 | the form 14 | 15 | \[ 16 | \begin{equation} 17 | \hat{y} = \beta_0{\mathbf 1} + Z\theta_0 + \sum_{j=1}^p(X_j\beta_j + 18 | W_j\theta_j), 19 | \end{equation} 20 | \] 21 | 22 | where $\hat{y}$ is the predicted $N\times1$ vector, $\beta_0$ is a 23 | scalar, $\theta_0$ is a $K$-vector, $X$ and $Z$ are $N\times p$ and 24 | $N\times K$ matrices containing values of the predictor and modifying 25 | variables, respectively, and $W_j=X_j \circ Z$ denotes the elementwise 26 | multiplication of $Z$ by column $X_j$ of $X$. 27 | 28 | The objective function used for pliable lasso is 29 | 30 | \[ 31 | J(\beta_0, \theta_0, \beta, \Theta) = 32 | \frac{1}{2N}\sum_{i=1}^N (y_i-\hat{y}_i)^2 + 33 | (1-\alpha)\lambda\sum_{j=1}^p\biggl(||(\beta_j,\theta_j)||_2 + 34 | ||\theta_j||_2\biggr) + \alpha\lambda\sum_{j,k}|\theta_{j,k}|_1. 35 | \] 36 | 37 | In the above, $\Theta$ is a $p\times K$ matrix of parameters with 38 | $j$-th row $\theta_j$ and individual entries $\theta_{j,k}$, and $\alpha$ and $\lambda$ 39 | are tuning parameters. As $\alpha \rightarrow 1$ (but $<1$), the 40 | solution approaches the lasso solution. The default value used in the paper is 41 | $\alpha = 0.5.$ 42 | 43 | An R package for the pliable lasso is forthcoming from the 44 | authors. Nevertheless, the pliable lasso is an excellent example to 45 | highlight the prototyping capabilities of `CVXR` for research. Along 46 | the way, we also introduce some new atoms that are needed in this example. 47 | 48 | ## Example 49 | 50 | We will use a simulated example from Section 3 of @tibsjhf:2017 with 51 | $n=100$, $p=50$ and $K=4$. The response is generated as 52 | 53 | \[ 54 | \begin{eqnarray*} 55 | y &=& \mu(x) + 0.5\epsilon;\ \ \epsilon \sim N(0, 1)\\ 56 | \mu(x) &=& x_1\beta_1 + x_2\beta_2 + x_3(\beta_3 e + 2z_1) + 57 | x_4\beta_4(e - 2z_2);\ \ \beta = (2, -2, 2, 2, 0, 0, \ldots), 58 | \end{eqnarray*} 59 | \] 60 | where $e=(1,1,\ldots ,1)^T$. 61 | 62 | ```{r} 63 | ## Simulation data. 64 | set.seed(123) 65 | N <- 100 66 | K <- 4 67 | p <- 50 68 | X <- matrix(rnorm(n = N * p, mean = 0, sd = 1), nrow = N, ncol = p) 69 | Z <- matrix(rbinom(n = N * K, size = 1, prob = 0.5), nrow = N, ncol = K) 70 | 71 | ## Response model. 72 | beta <- rep(x = 0, times = p) 73 | beta[1:4] <- c(2, -2, 2, 2) 74 | coeffs <- cbind(beta[1], beta[2], beta[3] + 2 * Z[, 1], beta[4] * (1 - 2 * Z[, 2])) 75 | mu <- diag(X[, 1:4] %*% t(coeffs)) 76 | y <- mu + 0.5 * rnorm(N, mean = 0, sd = 1) 77 | ``` 78 | 79 | It seems worthwhile to write a function that will fit the model for us 80 | so that we can customize a few things such as the intercept term, 81 | verbosity, etc. The function has the following structure with 82 | comments as placeholders for code we shall construct later. 83 | 84 | ```{r, eval = FALSE} 85 | plasso_fit <- function(y, X, Z, lambda, alpha = 0.5, intercept = TRUE, 86 | ZERO_THRESHOLD= 1e-6, verbose = FALSE) { 87 | N <- length(y) 88 | p <- ncol(X) 89 | K <- ncol(Z) 90 | 91 | beta0 <- 0 92 | if (intercept) { 93 | beta0 <- Variable(1) * matrix(1, nrow = N, ncol = 1) 94 | } 95 | ## Define_Parameters 96 | ## Build_Penalty_Terms 97 | ## Compute_Fitted_Value 98 | ## Build_Objective 99 | ## Define_and_Solve_Problem 100 | ## Return_Values 101 | } 102 | 103 | ## Fit pliable lasso using CVXR. 104 | # pliable <- pliable_lasso(y, X, Z, alpha = 0.5, lambda = lambda) 105 | ``` 106 | 107 | ### Parameters 108 | 109 | The parameters are easy: we just have $\beta$, $\theta_0$ and 110 | $\Theta$. 111 | 112 | ```{r Define_Parameters, eval = FALSE} 113 | beta <- Variable(p) 114 | theta0 <- Variable(K) 115 | theta <- Variable(p, K); theta_transpose <- t(theta) 116 | ``` 117 | Note that we also define the transpose of $\Theta$ for use later. 118 | 119 | ### Penalty Terms 120 | 121 | There are three of them. The first term in the parentheses, 122 | $\sum_{j=1}^p\biggl(||(\beta_j,\theta_j)||_2\biggr)$, involves components of 123 | $\beta$ and rows of $\Theta$. `CVXR` provides two functions to express 124 | this norm: 125 | 126 | - `hstack` to bind columns of $\beta$ and the matrix $\Theta$, the 127 | equivalent of `rbind` in R, 128 | - `cvxr_norm`, which accepts a matrix variable and an `axis` denoting the axis along which 129 | the norm is to be taken. The penalty requires us to use the row axis, 130 | so `axis = 1` per the usual R convention. 131 | 132 | The second term in the parentheses, $\sum_{j}||\theta_j||_2$, is also a norm 133 | along rows as the $\theta_j$ are rows of $\Theta$. The last term is simply a $l_1$-norm. 134 | 135 | ```{r Build_Penalty_Terms, eval = FALSE} 136 | penalty_term1 <- sum(cvxr_norm(hstack(beta, theta), 2, axis = 1)) 137 | penalty_term2 <- sum(cvxr_norm(theta, 2, axis = 1)) 138 | penalty_term3 <- sum(cvxr_norm(theta, 1)) 139 | ``` 140 | 141 | ### Fitted Value 142 | 143 | Equation 1 above for $\hat{y}$ contains a sum: 144 | $\sum_{j=1}^p(X_j\beta_j + W_j\theta_j)$. This requires multiplication 145 | of $Z$ by the columns of $X$ component-wise and is thus a natural candidate 146 | for a map-reduce combination: map the column multiplication function 147 | appropriately and reduce using `+` to obtain the `XZ_term` below. 148 | 149 | ```{r Compute_Fitted_Value, eval = FALSE} 150 | xz_theta <- lapply(seq_len(p), 151 | function(j) (matrix(X[, j], nrow = N, ncol = K) * Z) %*% theta_transpose[, j]) 152 | XZ_term <- Reduce(f = '+', x = xz_theta) 153 | y_hat <- beta0 + X %*% beta + Z %*% theta0 + XZ_term 154 | ``` 155 | 156 | ### Objective Function 157 | 158 | Building the objective is now straightforward. 159 | 160 | ```{r Build_Objective, eval = FALSE} 161 | objective <- sum_squares(y - y_hat) / (2 * N) + 162 | (1 - alpha) * lambda * (penalty_term1 + penalty_term2) + 163 | alpha * lambda * penalty_term3 164 | ``` 165 | 166 | ### Solving the Problem 167 | 168 | ```{r Define_and_Solve_Problem, eval = FALSE} 169 | prob <- Problem(Minimize(objective)) 170 | result <- solve(prob, verbose = verbose) 171 | beta_hat <- result$getValue(beta) 172 | ``` 173 | 174 | ### Return Values 175 | 176 | We create a list with values of interest to us. However, since 177 | sparsity is desired, we set values below `ZERO_THRESHOLD` to 178 | zero. 179 | 180 | ```{r Return_Results, eval = FALSE} 181 | theta0_hat <- result$getValue(theta0) 182 | theta_hat <- result$getValue(theta) 183 | 184 | ## Zero out stuff before returning 185 | beta_hat[abs(beta_hat) < ZERO_THRESHOLD] <- 0.0 186 | theta0_hat[abs(theta0_hat) < ZERO_THRESHOLD] <- 0.0 187 | theta_hat[abs(theta_hat) < ZERO_THRESHOLD] <- 0.0 188 | list(beta0_hat = if (intercept) result$getValue(beta0)[1] else 0.0, 189 | beta_hat = beta_hat, 190 | theta0_hat = theta0_hat, 191 | theta_hat = theta_hat, 192 | criterion = result$value) 193 | ``` 194 | 195 | ## Full Function 196 | 197 | We now put it all together. 198 | 199 | ```{r} 200 | plasso_fit <- function(y, X, Z, lambda, alpha = 0.5, intercept = TRUE, 201 | ZERO_THRESHOLD= 1e-6, verbose = FALSE) { 202 | N <- length(y) 203 | p <- ncol(X) 204 | K <- ncol(Z) 205 | 206 | beta0 <- 0 207 | if (intercept) { 208 | beta0 <- Variable(1) * matrix(1, nrow = N, ncol = 1) 209 | } 210 | <> 211 | <> 212 | <> 213 | <> 214 | <> 215 | <> 216 | } 217 | ``` 218 | 219 | ## Results 220 | 221 | Using $\lambda = 0.6$, we fit the pliable lasso without an intercept. 222 | 223 | ```{r} 224 | result <- plasso_fit(y, X, Z, lambda = 0.6, alpha = 0.5, intercept = FALSE) 225 | ``` 226 | 227 | We can print the various estimates. 228 | 229 | ```{r} 230 | cat(sprintf("Objective value: %f\n", result$criterion)) 231 | ``` 232 | 233 | We only print the nonzero $\beta$ values. 234 | 235 | ```{r} 236 | index <- which(result$beta_hat != 0) 237 | est.table <- data.frame(matrix(result$beta_hat[index], nrow = 1)) 238 | names(est.table) <- paste0("$\\beta_{", index, "}$") 239 | knitr::kable(est.table, format = "html", digits = 3) %>% 240 | kable_styling("striped") 241 | ``` 242 | 243 | For this value of $\lambda$, the nonzero $(\beta_1, \beta_2, \beta_3,\beta_4)$ are picked up along 244 | with a few others like $(\beta_{20}, \beta_{34},\beta_{39})$. 245 | 246 | The values for $\theta_0$ are given below. 247 | 248 | ```{r} 249 | est.table <- data.frame(matrix(result$theta0_hat, nrow = 1)) 250 | names(est.table) <- paste0("$\\theta_{0,", 1:K, "}$") 251 | knitr::kable(est.table, format = "html", digits = 3) %>% 252 | kable_styling("striped") 253 | ``` 254 | 255 | Finally, we display just the first five rows of $\Theta$, which happen to contain all 256 | the nonzero values for this result. 257 | ```{r} 258 | est.table <- data.frame(result$theta_hat[1:5, ]) 259 | names(est.table) <- paste0("$\\theta_{,", 1:K, "}$") 260 | knitr::kable(est.table, format = "html", digits = 3) %>% 261 | kable_styling("striped") 262 | ``` 263 | 264 | ## Final Comments 265 | 266 | Typically, one would run the fits for various values of $\lambda$, 267 | choose one based on cross-validation, and assess the prediction against 268 | a test set. Here, even a single fit takes a while, but techniques 269 | discussed previously can be used to speed up the 270 | computations. However, the potential of `CVXR` in prototyping novel 271 | methods is clear. 272 | 273 | ## References 274 | -------------------------------------------------------------------------------- /13-survey-sampling.Rmd: -------------------------------------------------------------------------------- 1 | # Survey Calibration 2 | 3 | ```{r, message = FALSE, echo = FALSE} 4 | library(dplyr) 5 | library(survey) 6 | build_df <- function(api, method, wt_value) { 7 | d <- data.frame(apisrs$stype, apisrs$sch.wide, wt_value ) 8 | names(d) <- c("stype", "sch.wide", "weight") 9 | rownames(d) <- NULL 10 | d %<>% 11 | group_by(stype, sch.wide) %>% 12 | summarize(value = first(weight), frequency = n()) 13 | names(d) <- c("stype", "sch.wide", paste(method, "wts."), "Frequency") 14 | d 15 | } 16 | build_table <- function(d1, d2, title) { 17 | d <- inner_join(d1, d2, by = c("stype", "sch.wide")) 18 | names(d) <- gsub("Frequency.x|Frequency.y", "Frequency", names(d)) 19 | d %>% 20 | knitr::kable(format = "html", digits = 3, caption = title) %>% 21 | kable_styling("striped") %>% 22 | column_spec(1:6, background = "#ececec") 23 | } 24 | ``` 25 | 26 | ## Goals 27 | 28 | - Demonstrate different methods for calibrating data 29 | 30 | ## Survey Calibration Problem 31 | 32 | Calibration is a widely used technique in survey sampling. Suppose 33 | $m$ sampling units in a survey have been assigned initial weights 34 | $d_i$ for $i = 1,\ldots,m$, and furthermore, there are $n$ auxiliary 35 | variables whose values in the sample are known. Calibration seeks to 36 | improve the initial weights $d_i$ by finding new weights $w_i$ that 37 | incorporate this auxiliary information while perturbing the initial 38 | weights as little as possible, \ie, the ratio $g_i = w_i/d_i$ must 39 | be close to one. Such reweighting improves precision of estimates 40 | (Chapter 7, @Lumley:2010). 41 | 42 | Let $X \in {\mathbf R}^{m \times n}$ be the matrix of survey samples, with 43 | each column corresponding to an auxiliary variable. Reweighting can be 44 | expressed as the optimization problem (see @Davies:2016): 45 | 46 | \[ 47 | \begin{array}{ll} 48 | \mbox{minimize} & \sum_{i=1}^m d_i\phi(g_i) \\ 49 | \mbox{subject to} & A^Tg = r 50 | \end{array} 51 | \] 52 | 53 | with respect to $g \in {\mathbf R}^m$, where $\phi:{\mathbf R} \rightarrow 54 | {\mathbf R}$ is a strictly convex function with $\phi(1) = 0$, $r \in 55 | {\mathbf R}^n$ are the known population totals of the auxiliary variables, 56 | and $A \in {\mathbf R}^{m \times n}$ is related to $X$ by $A_{ij} = 57 | d_iX_{ij}$ for $i = 1,\ldots,m$ and $j = 1,\ldots,n$. 58 | 59 | ## Raking 60 | 61 | A common calibration technique is _raking_, which uses the penalty 62 | function $\phi(g_i) = g_i\log(g_i) - g_i + 1$ as the calibration 63 | metric. 64 | 65 | We illustrate this with the California Academic Performance Index data in 66 | the `survey` package (@lumley:2018), which also supplies facilities for 67 | calibration via the function `calibrate`. Both the population dataset 68 | (`apipop`) and a simple random sample of $m = 200$ (`apisrs`) are 69 | provided. Suppose that we wish to reweight the observations in the 70 | sample using known totals for two variables from the population: 71 | `stype`, the school type (elementary, middle, or high) and `sch.wide`, 72 | whether the school met the yearly target or not. This reweighting 73 | would make the sample more representative of the general population. 74 | 75 | The code below estimates the weights using `survey::calibrate`. 76 | 77 | ```{r} 78 | data(api) 79 | design_api <- svydesign(id = ~dnum, weights = ~pw, data = apisrs) 80 | formula <- ~stype + sch.wide 81 | T <- apply(model.matrix(object = formula, data = apipop), 82 | 2, 83 | sum) 84 | 85 | cal_api <- calibrate(design_api, formula, population = T, calfun = cal.raking) 86 | w_survey <- weights(cal_api) 87 | ``` 88 | 89 | The `CVXR` formulation follows. 90 | 91 | ```{r} 92 | di <- apisrs$pw 93 | X <- model.matrix(object = formula, data = apisrs) 94 | A <- di * X 95 | n <- nrow(apisrs) 96 | g <- Variable(n) 97 | constraints <- list(t(A) %*% g == T) 98 | 99 | ## Raking 100 | Phi_R <- Minimize(sum(di * (-entr(g) - g + 1))) 101 | p <- Problem(Phi_R, constraints) 102 | res <- solve(p) 103 | w_cvxr <- di * res$getValue(g) 104 | ``` 105 | 106 | The results are identical as shown in the table below. 107 | 108 | ```{r, echo = FALSE} 109 | ## Using functions in the *un echoed* preamble of this document... 110 | build_table(d1 = build_df(apisrs, "Survey", w_survey), 111 | d2 = build_df(apisrs, "CVXR", w_cvxr), 112 | title = "Calibration weights from Raking") 113 | ``` 114 | 115 | ### Exercise 116 | 117 | The quadratic penalty function 118 | 119 | \[ 120 | \phi^{Q}(g) = \frac{1}{2}(g-1)^2 121 | \] 122 | 123 | is also sometimes used. Calibrate using `CVXR` and the `SCS` solver and compare with the 124 | `survey::cal.linear` results, which can be obtained via 125 | 126 | ```{r} 127 | w_survey_q <- weights(calibrate(design_api, formula, population = T, calfun = cal.linear)) 128 | ``` 129 | 130 | #### Solution 131 | 132 | ```{r} 133 | ## Quadratic 134 | Phi_Q <- Minimize(sum_squares(g - 1) / 2) 135 | p <- Problem(Phi_Q, constraints) 136 | res <- solve(p, solver = "SCS") 137 | w_cvxr_q <- di * res$getValue(g) 138 | ``` 139 | 140 | The default `ECOS` solver produces a different number of unique 141 | weights. Such differences are not unheard of among solvers. 142 | 143 | ```{r, echo = FALSE} 144 | build_table(d1 = build_df(apisrs, "Survey", w_survey_q), 145 | d2 = build_df(apisrs, "CVXR", w_cvxr_q), 146 | title = "Calibration weights from Quadratic metric") 147 | ``` 148 | 149 | ### Exercise 150 | 151 | Repeat the above exercise with the logistic function 152 | 153 | \[ 154 | \phi^{L}(g; l, u) = \frac{1}{C}\biggl[ (g-l)\log\left(\frac{g-l}{1-l}\right) + 155 | (u-g)\log\left(\frac{u-g}{u-1}\right) \biggr] \mbox{ for } C = \frac{u-l}{(1-l)(u-1)}, 156 | \] 157 | 158 | which requires bounds $l$ and $u$ on the coefficients. Use $l=0.9$ and $u=1.1$. The results from `survey::cal.linear` can be obtained with the code below. 159 | 160 | ```{r} 161 | u <- 1.10; l <- 0.90 162 | w_survey_l <- weights(calibrate(design_api, formula, population = T, calfun = cal.linear, 163 | bounds = c(l, u))) 164 | ``` 165 | 166 | ### Solution 167 | 168 | ```{r} 169 | Phi_L <- Minimize(sum(-entr((g - l) / (u - l)) - 170 | entr((u - g) / (u - l)))) 171 | p <- Problem(Phi_L, c(constraints, list(l <= g, g <= u))) 172 | res <- solve(p) 173 | w_cvxr_l <- di * res$getValue(g) 174 | ``` 175 | 176 | ```{r, echo = FALSE} 177 | build_table(d1 = build_df(apisrs, "Survey", w_survey_l), 178 | d2 = build_df(apisrs, "CVXR", w_cvxr_l), 179 | title = "Calibration weights from Logit metric") 180 | ``` 181 | 182 | ### Exercise 183 | 184 | Repeat with the Hellinger distance 185 | 186 | \[ 187 | \Phi^{H}(g) = \frac{1}{(1 - g/2)^2} 188 | \] 189 | 190 | and the following results. 191 | 192 | ```{r} 193 | hellinger <- make.calfun(Fm1 = function(u, bounds) ((1 - u / 2)^-2) - 1, 194 | dF= function(u, bounds) (1 -u / 2)^-3 , 195 | name = "Hellinger distance") 196 | w_survey_h <- weights(calibrate(design_api, formula, population = T, calfun = hellinger)) 197 | ``` 198 | 199 | #### Solution 200 | 201 | ```{r} 202 | Phi_h <- Minimize(sum((1 - g / 2)^(-2))) 203 | p <- Problem(Phi_h, constraints) 204 | res <- solve(p) 205 | w_cvxr_h <- di * res$getValue(g) 206 | ``` 207 | 208 | ```{r, echo = FALSE} 209 | build_table(d1 = build_df(apisrs, "Survey", w_survey_h), 210 | d2 = build_df(apisrs, "CVXR", w_cvxr_h), 211 | title = "Calibration weights from Hellinger distance metric") 212 | ``` 213 | 214 | ### Exercise 215 | 216 | Lastly, use the derivative of the inverse hyperbolic sine 217 | 218 | \[ 219 | \Phi^{S}(g) = \frac{1}{2}(e^g + e^{-g}) 220 | \] 221 | 222 | along with the results produced below. 223 | 224 | ```{r} 225 | w_survey_s <- weights(calibrate(design_api, formula, population = T, calfun = cal.sinh, 226 | bounds = c(l, u))) 227 | ``` 228 | 229 | #### Solution 230 | 231 | ```{r} 232 | Phi_s <- Minimize(sum( 0.5 * (exp(g) + exp(-g)))) 233 | p <- Problem(Phi_s, c(constraints, list(l <= g, g <= u))) 234 | res <- solve(p) 235 | w_cvxr_s <- di * res$getValue(g) 236 | ``` 237 | 238 | ```{r, echo = FALSE} 239 | build_table(d1 = build_df(apisrs, "Survey", w_survey_s), 240 | d2 = build_df(apisrs, "CVXR", w_cvxr_s), 241 | title = "Calibration weights from derivative of sinh metric") 242 | ``` 243 | 244 | ## References 245 | 246 | -------------------------------------------------------------------------------- /14-sparse-inverse-covariance-estimation.Rmd: -------------------------------------------------------------------------------- 1 | # Sparse Inverse Covariance Estimation 2 | 3 | ```{r, message = FALSE, echo = FALSE} 4 | library(CVXR) 5 | library(ggplot2) 6 | library(grid) 7 | library(Matrix) 8 | library(expm) 9 | ## 10 | ## Reference: http://www.cookbook-r.com/Graphs/Multiple_graphs_on_one_page_(ggplot2)/ 11 | # Multiple plot function 12 | # 13 | # ggplot objects can be passed in ..., or to plotlist (as a list of ggplot objects) 14 | # - cols: Number of columns in layout 15 | # - layout: A matrix specifying the layout. If present, 'cols' is ignored. 16 | # 17 | # If the layout is something like matrix(c(1,2,3,3), nrow=2, byrow=TRUE), 18 | # then plot 1 will go in the upper left, 2 will go in the upper right, and 19 | # 3 will go all the way across the bottom. 20 | # 21 | multiplot <- function(..., plotlist=NULL, file, cols=1, layout=NULL) { 22 | 23 | # Make a list from the ... arguments and plotlist 24 | plots <- c(list(...), plotlist) 25 | 26 | numPlots = length(plots) 27 | 28 | # If layout is NULL, then use 'cols' to determine layout 29 | if (is.null(layout)) { 30 | # Make the panel 31 | # ncol: Number of columns of plots 32 | # nrow: Number of rows needed, calculated from # of cols 33 | layout <- matrix(seq(1, cols * ceiling(numPlots/cols)), 34 | ncol = cols, nrow = ceiling(numPlots/cols)) 35 | } 36 | 37 | if (numPlots==1) { 38 | print(plots[[1]]) 39 | 40 | } else { 41 | # Set up the page 42 | grid.newpage() 43 | pushViewport(viewport(layout = grid.layout(nrow(layout), ncol(layout)))) 44 | 45 | # Make each plot, in the correct location 46 | for (i in 1:numPlots) { 47 | # Get the i,j matrix positions of the regions that contain this subplot 48 | matchidx <- as.data.frame(which(layout == i, arr.ind = TRUE)) 49 | 50 | print(plots[[i]], vp = viewport(layout.pos.row = matchidx$row, 51 | layout.pos.col = matchidx$col)) 52 | } 53 | } 54 | } 55 | 56 | theme_bare <- theme( 57 | axis.line = element_blank(), 58 | axis.text.x = element_blank(), 59 | axis.text.y = element_blank(), 60 | axis.ticks = element_blank(), 61 | axis.title.y = element_blank(), 62 | legend.position = "none", 63 | panel.background = element_rect(fill = "white"), 64 | panel.border = element_blank(), 65 | panel.grid.major = element_blank(), 66 | panel.grid.minor = element_blank(), 67 | ) 68 | ``` 69 | 70 | ## Goals 71 | 72 | - Introduce positive semidefinite matrices and `CVXR::log_det` 73 | - Demonstrate how to build functions for complex tasks 74 | 75 | ## Sparse Inverse Covariance Problem 76 | 77 | Assume we are given i.i.d. observations $x_i \sim N(0,\Sigma)$ for $i 78 | = 1,\ldots,m$, where the covariance matrix $\Sigma \in {\mathbf S}_+^n$, the 79 | set of symmetric positive semidefinite matrices, has a sparse inverse 80 | $S = \Sigma^{-1}$. Let $Q = \frac{1}{m-1}\sum_{i=1}^m (x_i - \bar 81 | x)(x_i - \bar x)^T$ be our sample covariance. One way to estimate 82 | $\Sigma$ is to maximize the log-likelihood with the prior knowledge 83 | that $S$ is sparse [@spinvcov], which amounts to the optimization 84 | problem: 85 | 86 | $$ 87 | \begin{array}{ll} \underset{S}{\mbox{maximize}} & \log\det(S) - \mbox{tr}(SQ) \\ 88 | \mbox{subject to} & \sum_{i=1}^n \sum_{j=1}^n |S_{ij}| \leq \alpha 89 | \end{array} 90 | $$ 91 | 92 | with respect to $S \in {\mathbf S}_+^n$, where the parameter $\alpha \geq 0$ controls the degree of sparsity. The problem 93 | is convex, so we can solve it using `CVXR`. 94 | 95 | ## Example 96 | 97 | We'll create a sparse positive semidefinite matrix $S$ using 98 | synthetic data. 99 | 100 | ```{r} 101 | set.seed(1) 102 | n <- 10 ## Dimension of matrix 103 | m <- 1000 ## Number of samples 104 | 105 | ## Create sparse, symmetric PSD matrix S 106 | A <- rsparsematrix(n, n, 0.15, rand.x = stats::rnorm) 107 | Strue <- A %*% t(A) + 0.05 * diag(rep(1, n)) ## Force matrix to be strictly positive definite 108 | ``` 109 | 110 | We then invert $S$ to get $R = S^{-1}$. 111 | 112 | ```{r} 113 | R <- base::solve(Strue) 114 | ``` 115 | 116 | As test data, we sample from a multivariate normal distribution using the fact that 117 | if $Y \sim N(0, I)$, then $R^{1/2}Y \sim N(0, R)$ since $R$ is 118 | symmetric. 119 | 120 | ```{r} 121 | x_sample <- matrix(stats::rnorm(n * m), nrow = m, ncol = n) %*% t(expm::sqrtm(R)) 122 | Q <- cov(x_sample) ## Sample covariance matrix 123 | ``` 124 | 125 | Finally, we solve our convex program for a range of $\alpha$ values. 126 | 127 | ```{r} 128 | alphas <- c(10, 8, 6, 4, 1) 129 | S <- Variable(c(n, n), PSD = TRUE) ## Variable constrained to positive semidefinite cone 130 | obj <- Maximize(log_det(S) - matrix_trace(S %*% Q)) 131 | 132 | S.est <- lapply(alphas, 133 | function(alpha) { 134 | constraints <- list(sum(abs(S)) <= alpha) 135 | ## Form and solve optimization problem 136 | prob <- Problem(obj, constraints) 137 | result <- solve(prob) 138 | 139 | ## Create covariance matrix 140 | R_hat <- base::solve(result$getValue(S)) 141 | Sres <- result$getValue(S) 142 | Sres[abs(Sres) <= 1e-4] <- 0 143 | Sres 144 | }) 145 | ``` 146 | 147 | In the code above, the `Variable` constructor restricts `S` to the 148 | positive semidefinite cone. We use `CVXR` functions 149 | for the log-determinant and trace in the objective. The expression `matrix_trace(S %*% 150 | Q)` is equivalent to `sum(diag(S %*% Q))}, but the former is preferred 151 | because it is more efficient than making nested function calls. 152 | 153 | However, a standalone atom does not exist for the determinant, so we 154 | cannot replace `log_det(S)` with `log(det(S))` since `det` is 155 | undefined for a positive semidefinite object in `CVXR`. 156 | 157 | ## Results 158 | 159 | The figures below depict the solutions for the dataset with $m = 160 | 1000, n = 10$, and $S$ containing 26% non-zero entries, represented by 161 | the dark squares in the images below. The sparsity of our inverse 162 | covariance estimate decreases for higher $\alpha$, so that when 163 | $\alpha = 1$, most of the off-diagonal entries are zero, while if 164 | $\alpha = 10$, over half the matrix is dense. At $\alpha = 4$, we 165 | achieve the true percentage of non-zeros. 166 | 167 | ```{r, echo = FALSE, fig.width=6, fig.height=6} 168 | ## Plotting function 169 | plotSpMat <- function(S, alpha) { 170 | n <- nrow(S) 171 | df <- expand.grid(j = seq_len(n), i = seq_len(n)) 172 | df$z = as.character(as.numeric(S) != 0) 173 | p <- ggplot(data = df, mapping = aes(x = i, y = j, fill = z)) + 174 | geom_tile(color = "black") + 175 | scale_fill_brewer(type = "qual", palette = "Paired") + 176 | scale_y_reverse() 177 | if (missing(alpha)) { 178 | p <- p + xlab("Truth") 179 | } else { 180 | p <- p + xlab(parse(text=(paste0("alpha == ", alpha)))) 181 | } 182 | p + theme_bare 183 | } 184 | ``` 185 | 186 | ```{r, fig.width=6, fig.height=4} 187 | do.call(multiplot, args = c(list(plotSpMat(Strue)), 188 | mapply(plotSpMat, S.est, alphas, SIMPLIFY = FALSE), 189 | list(layout = matrix(1:6, nrow = 2, byrow = TRUE)))) 190 | ``` 191 | 192 | 193 | ## References 194 | 195 | -------------------------------------------------------------------------------- /15-portfolio-estimation.Rmd: -------------------------------------------------------------------------------- 1 | # Portfolio Estimation 2 | 3 | ```{r, message = FALSE, echo = FALSE} 4 | library(CVXR) 5 | library(ggplot2) 6 | library(RColorBrewer) 7 | library(tidyr) 8 | ``` 9 | 10 | ## Goals 11 | 12 | - Demonstrate `CVXR` on a financial example 13 | 14 | ## Markowitz Portfolio Problem 15 | 16 | The Markowitz portfolio problem [@Markowitz:1952; @Roy:1952; 17 | @LoboFazelBoyd:2007] is well known in finance. We will solve this 18 | problem under various constraints. 19 | 20 | We have $n$ assets or stocks in our portfolio and must determine the 21 | amount of money to invest in each. Let $w_i$ denote the fraction of 22 | our budget invested in asset $i = 1,\ldots,m$, and let $r_i$ be the 23 | returns (\ie, fractional change in price) over the period of 24 | interest. We model returns as a random vector $r \in {\mathbf R}^n$ with 25 | known mean $\mathbf{E}[r] = \mu$ and covariance $\mathbf{Var}(r) = \Sigma$. Thus, 26 | given a portfolio $w \in {\mathbf R}^n$, the overall return is $R = r^Tw$. 27 | 28 | Portfolio optimization involves a trade-off between the expected 29 | return $\mathbf{E}[R] = \mu^Tw$ and the associated risk, which we take to be the 30 | return variance $\mathbf{Var}(R) = w^T\Sigma w$. Initially, we consider only 31 | long portfolios, so our problem is 32 | $$ 33 | \begin{array}{ll} 34 | \underset{w}{\mbox{maximize}} & \mu^Tw - \gamma w^T\Sigma w \\ 35 | \mbox{subject to} & w \geq 0, \quad \sum_{i=1}^n w = 1, 36 | \end{array} 37 | $$ 38 | where the objective is the risk-adjusted return and $\gamma > 0$ is a 39 | risk aversion parameter. 40 | 41 | ## Example 42 | 43 | We construct the risk-return trade-off curve for $n = 10$ assets and 44 | $\mu$ and $\Sigma^{1/2}$ drawn from a standard normal 45 | distribution. 46 | 47 | ```{r} 48 | ## Problem data 49 | set.seed(10) 50 | n <- 10 51 | SAMPLES <- 100 52 | mu <- matrix(abs(rnorm(n)), nrow = n) 53 | Sigma <- matrix(rnorm(n^2), nrow = n, ncol = n) 54 | Sigma <- t(Sigma) %*% Sigma 55 | 56 | ## Form problem 57 | w <- Variable(n) 58 | ret <- t(mu) %*% w 59 | risk <- quad_form(w, Sigma) 60 | constraints <- list(w >= 0, sum(w) == 1) 61 | 62 | ## Risk aversion parameters 63 | gammas <- 10^seq(-2, 3, length.out = SAMPLES) 64 | ret_data <- rep(0, SAMPLES) 65 | risk_data <- rep(0, SAMPLES) 66 | w_data <- matrix(0, nrow = SAMPLES, ncol = n) 67 | 68 | ## Compute trade-off curve 69 | for(i in seq_along(gammas)) { 70 | gamma <- gammas[i] 71 | objective <- ret - gamma * risk 72 | prob <- Problem(Maximize(objective), constraints) 73 | result <- solve(prob) 74 | 75 | ## Evaluate risk/return for current solution 76 | risk_data[i] <- result$getValue(sqrt(risk)) 77 | ret_data[i] <- result$getValue(ret) 78 | w_data[i,] <- result$getValue(w) 79 | } 80 | ``` 81 | 82 | Note how we obtain the risk and return by _directly evaluating_ 83 | the value of the separate expressions: 84 | 85 | ```{r, eval = FALSE} 86 | result$getValue(risk) 87 | result$getValue(ret) 88 | ``` 89 | 90 | The trade-off curve is shown below. The $x$-axis represents the standard 91 | deviation of the return. Red points indicate the result from investing 92 | the entire budget in a single asset. As $\gamma$ increases, our 93 | portfolio becomes more diverse, reducing risk but also yielding a 94 | lower return. 95 | 96 | ```{r} 97 | cbPalette <- brewer.pal(n = 10, name = "Paired") 98 | p1 <- ggplot() + 99 | geom_line(mapping = aes(x = risk_data, y = ret_data), color = "blue") + 100 | geom_point(mapping = aes(x = sqrt(diag(Sigma)), y = mu), color = "red") 101 | 102 | markers_on <- c(10, 20, 30, 40) 103 | nstr <- sprintf("gamma == %.2f", gammas[markers_on]) 104 | df <- data.frame(markers = markers_on, x = risk_data[markers_on], 105 | y = ret_data[markers_on], labels = nstr) 106 | 107 | p1 + geom_point(data = df, mapping = aes(x = x, y = y), color = "black") + 108 | annotate("text", x = df$x + 0.2, y = df$y - 0.05, label = df$labels, parse = TRUE) + 109 | labs(x = "Risk (Standard Deviation)", y = "Return") 110 | ``` 111 | 112 | We can also plot the fraction of budget invested in each asset. 113 | 114 | ```{r} 115 | w_df <- data.frame(paste0("grp", seq_len(ncol(w_data))), 116 | t(w_data[markers_on,])) 117 | names(w_df) <- c("grp", sprintf("gamma == %.2f", gammas[markers_on])) 118 | tidyW <- gather(w_df, key = "gamma", value = "fraction", names(w_df)[-1], factor_key = TRUE) 119 | ggplot(data = tidyW, mapping = aes(x = gamma, y = fraction)) + 120 | geom_bar(mapping = aes(fill = grp), stat = "identity") + 121 | scale_x_discrete(labels = parse(text = levels(tidyW$gamma))) + 122 | scale_fill_manual(values = cbPalette) + 123 | guides(fill = FALSE) + 124 | labs(x = "Risk Aversion", y = "Fraction of Budget") 125 | ``` 126 | 127 | ## Discussion 128 | 129 | Many variations on the classical portfolio problem exist. For 130 | instance, we could allow long and short positions, but impose a 131 | leverage limit $\|w\|_1 \leq L^{max}$ by changing 132 | ```{r, eval = FALSE} 133 | constr <- list(p_norm(w,1) <= Lmax, sum(w) == 1) 134 | ``` 135 | 136 | An alternative is to set a lower bound on the return and minimize just 137 | the risk. To account for transaction costs, we could add a term to the 138 | objective that penalizes deviations of $w$ from the previous 139 | portfolio. These extensions and more are described in 140 | @BoydBusseti:2017. The key takeaway is that all of these convex 141 | problems can be easily solved in `CVXR` with just a few alterations 142 | to the code above. 143 | 144 | ## References 145 | -------------------------------------------------------------------------------- /16-using-other-solvers.Rmd: -------------------------------------------------------------------------------- 1 | # Using Other Solvers 2 | 3 | ## Goals 4 | 5 | - Show how to use non-default solvers like `MOSEK` and `GUROBI` 6 | - Discuss solver peculiarities 7 | 8 | ## Default Solvers 9 | 10 | The default installation of `CVXR` comes with two (imported) open 11 | source solvers: 12 | 13 | - [ECOS](https://github.com/embotech/ecos) and its mixed integer 14 | cousin `ECOS_BB` via the CRAN package 15 | [ECOSolveR](https://cloud.r-project.org/package=ECOSolveR) 16 | - [SCS](https://github.com/cvxgrp/scs) via the CRAN package 17 | [scs](https://cloud.r-project.org/package=scs). 18 | 19 | `CVXR` can also make use of several other open source solvers 20 | implemented in R packages. 21 | 22 | - The linear and mixed integer programming package 23 | [`lpSolve`](http://lpsolve.sourceforge.net/5.5/) via the 24 | [`lpSolveAPI`](https://cloud.r-project.org/package=lpSolveAPI) package 25 | - The linear and mixed integer programming package [`GLPK`](https://www.gnu.org/software/glpk/) via the 26 | [`Rglpk`](https://cloud.r-project.org/package=Rglpk) package. 27 | 28 | Since these are optional, you must install the packages yourself. 29 | 30 | ```{r} 31 | lapply(list(LPSOLVE = "lpSolveAPI", 32 | GLPK = "Rglpk"), 33 | function(x) x %in% installed.packages()[, 1]) 34 | ``` 35 | 36 | Once the packages are installed, a call to `installed_solvers` will 37 | display the solvers that `CVXR` has detected. 38 | 39 | ## Commercial Solvers 40 | 41 | A few commercial solvers are also currently supported: [MOSEK](https://www.mosek.com) and 42 | [GUROBI](https://www.gurobi.com). 43 | 44 | At the moment, `CVXR` calls vendor Python solver packages via 45 | [`reticulate`](https://cran.r-project.org/package=reticulate), _not R 46 | packages_. Future versions will provide support directly using 47 | [_problem 48 | reductions_](https://web.stanford.edu/~boyd/papers/cvxpy_rewriting.html), 49 | implemented in [`CVXPY` version 1.0](https://www.cvxpy.org/). 50 | 51 | Thus, one needs two prerequisites to use these commercial solvers: 52 | 53 | - A Python installation in addition to R 54 | - The 55 | [`reticulate`](https://cran.r-project.org/package=reticulate) R 56 | package. 57 | 58 | We also recommend that you ensure `reticulate` is installed 59 | correctly and working. For example `reticulate::py_eval('1+1')` should return 60 | `2`. 61 | 62 | ### Installing `MOSEK` 63 | 64 | [MOSEK](https://www.mosek.com) provides an academic version that is 65 | free of charge. As noted in the downloads page, Anaconda users can 66 | install merely via: 67 | 68 | ```{bash, eval=FALSE} 69 | conda install -c mosek mosek 70 | ``` 71 | 72 | Others can use the `pip` command: 73 | 74 | ```{bash, eval = FALSE} 75 | pip install -f https://download.mosek.com/stable/wheel/index.html Mosek 76 | ``` 77 | 78 | In addition, the license for the product has to be activated per 79 | instructions on the `Sales` section of the MOSEK web page. 80 | 81 | Once activated, you can check that `CVXR` recognizes the solver; 82 | `installed_solvers()` should list `MOSEK`. Otherwise, rinse and repeat 83 | until success. 84 | 85 | ### Installing `GUROBI` 86 | 87 | [GUROBI](https://www.gurobi.com) also provides an academic version 88 | that is free of charge. You must register to receive the license. 89 | 90 | Once registered, install the _Gurobi Optimizer_ software and activate 91 | your license as necessary. 92 | 93 | After activation, you can check that `CVXR::installed_solvers()` lists 94 | `GUROBI`. Otherwise, rinse and repeat until success. 95 | 96 | ### Gotchas 97 | 98 | If you have an Anaconda installation in your path, you have 99 | to account for the fact that there may be interactions when using 100 | RStudio and rendering documents. In particular, Anaconda may include 101 | its own version of pandoc and other tools that may conflict with what 102 | Rstudio needs to work properly. 103 | 104 | To be concrete, one problem we found was that the `MOSEK` solver was 105 | not recognized as available in this rendered document, even 106 | though the command line interface showed it to be present. Ensuring an 107 | appropriate `PATH` variable solves the problem. 108 | 109 | ### Example Session 110 | 111 | ```{r} 112 | installed_solvers() 113 | ``` 114 | 115 | ## Solver Peculiarities 116 | 117 | The default solver in `CVXR` is `ECOS`. However, it is not always the 118 | best solver to use. As an example, let us consider again the [catenary 119 | problem](/cvxr_examples/cvxr_catenary/). 120 | 121 | We will change the problem slightly to use a finer discretization from 122 | 101 to 501 points. 123 | 124 | ```{r} 125 | ## Problem data 126 | m <- 501 127 | L <- 2 128 | h <- L / (m - 1) 129 | 130 | ## Form objective 131 | x <- Variable(m) 132 | y <- Variable(m) 133 | objective <- Minimize(sum(y)) 134 | 135 | ## Form constraints 136 | constraints <- list(x[1] == 0, y[1] == 1, 137 | x[m] == 1, y[m] == 1, 138 | diff(x)^2 + diff(y)^2 <= h^2) 139 | 140 | ## Solve the catenary problem 141 | prob <- Problem(objective, constraints) 142 | result <- solve(prob) 143 | ``` 144 | 145 | The solution status is no longer optimal. 146 | 147 | ```{r} 148 | cat("Solution status is", result$status) 149 | ``` 150 | 151 | In such cases, using a different solver may give more accurate 152 | results. Let us try `MOSEK`. 153 | 154 | ```{r} 155 | if ("MOSEK" %in% installed_solvers()) { 156 | result <- solve(prob, solver = "MOSEK") 157 | cat("Solution status is", result$status) 158 | } else { 159 | cat("Solution not available as MOSEK is not installed!") 160 | } 161 | ``` 162 | 163 | This returns an optimal solution. 164 | 165 | Here again, even commercial solvers differ; `GUROBI`, for example, 166 | does not completely solve the problem and in fact throws an error. 167 | 168 | 169 | ## References 170 | -------------------------------------------------------------------------------- /97-the-structure-of-atoms.Rmd: -------------------------------------------------------------------------------- 1 | # The Structure of Atoms 2 | 3 | `CVXR` comes with a rich library of atoms that represent common 4 | functions in convex analysis. When combined using the DCP rules, these 5 | atoms are sufficient to model and solve most convex optimization 6 | problems, and we encourage most users to work with the `CVXR` 7 | library. However, it is possible for a sophisticated user to add a new 8 | atom. To do this requires an understanding of mathematical programming 9 | and the S4 class system. In this guide, we go over the basics of 10 | implementing an atom, the quadratic-over-linear function, which should 11 | provide a starting point for potential developers. 12 | 13 | ## Definition 14 | 15 | For any $x \in \mathbf{R}^n$ and $y \in \mathbf{R}$, the 16 | quadratic-over-linear (QoL) function is 17 | 18 | $$ 19 | f(x,y) := \frac{\|x\|_2^2}{y} \quad \mbox{with} \quad \mathbf{dom}\;f = \{(x,y) \in \mathbf{R}^n \times \mathbf{R}: y > 0\}. 20 | $$ 21 | 22 | In `CVXR`, atoms like this (along with variables, constraints, etc) are 23 | represented by S4 class objects. S4 allows us to overload standard 24 | mathematical operations so `CVXR` combines seamlessly with native R 25 | script and other packages. The class for the QoL function is 26 | `QuadOverLin`, defined as 27 | 28 | ```{r, eval=FALSE} 29 | setClass("QuadOverLin", representation(x = "ConstValORExpr", y = "ConstValORExpr"), contains = "Atom") 30 | ``` 31 | We also provide a constructor 32 | ```{r, eval=FALSE} 33 | quad_over_lin <- function(x, y) { new("QuadOverLin", x = x, y = y) } 34 | ``` 35 | 36 | The `QuadOverLin` class inherits from the `Atom` superclass. It takes 37 | as input two arguments, `x` and `y`, which must be R numeric constants 38 | or `CVXR` `Expression` objects. These input types are encapsulated in 39 | the "ConstaValORExpr" class union. Since `Expression`s themselves may 40 | contain other atoms, this allows us to use the QoL function in nested 41 | compositions. 42 | 43 | During initialization, `x` and `y` must be passed to the `Atom` 44 | superclass for processing and validation. Consequently, we override 45 | the default `QuadOverLin` initialization method with 46 | 47 | ```{r, eval=FALSE} 48 | setMethod("initialize", "QuadOverLin", function(.Object, ..., x, y) { 49 | .Object@x <- x 50 | .Object@y <- y 51 | callNextMethod(.Object, ..., args = list(.Object@x, .Object@y)) 52 | }) 53 | ``` 54 | 55 | The first two lines save `x` and `y` to their respective slots in 56 | `QuadOverLin`, while the third calls `Atom`'s `initialize` with 57 | `args` equal to the list of arguments. For explanatory purposes, 58 | we reproduce this method below. 59 | 60 | ```{r, eval=FALSE} 61 | setMethod("initialize", "Atom", function(.Object, ..., args = list(), .size = NA_real_) { 62 | .Object@args <- lapply(args, as.Constant) 63 | validate_args(.Object) 64 | .Object@.size <- size_from_args(.Object) 65 | callNextMethod(.Object, ...) 66 | }) 67 | ``` 68 | 69 | It is not important to understand the details of the above code. Only 70 | recognize that the atom arguments are converted into `CVXR` objects 71 | and saved in a list in the `args` slot. We will be accessing this slot 72 | later. 73 | 74 | ## Atom Properties 75 | 76 | Now that we have created the atom's S4 class, it is time to define 77 | methods that characterize its mathematical properties. 78 | 79 | ### Mathematical Basics 80 | 81 | First, we provide a method for validating the atom's inputs. This 82 | method is called in `initialize` to check whether the arguments make 83 | sense for the atom. For our function $f$, the second argument must be 84 | a scalar, so our validation method looks like 85 | 86 | ```{r, eval=FALSE} 87 | setMethod("validate_args", "QuadOverLin", function(object) { 88 | if(!is_scalar(object@args[[2]])) 89 | stop("[QuadOverLin: validation] y must be a scalar") 90 | }) 91 | ``` 92 | 93 | The input `object` is a `QuadOverLin` atom. We access its arguments 94 | via its `args` slot inherited from `Atom`. Given our ordering in 95 | `args`, we know `object@args[[1]]` contains the argument for $x$ 96 | and `object@args[[2]]` the one for $y$. Hence, we invoke `is_scalar`, 97 | an `Expression` class method, on the latter to check if $y$ is indeed 98 | scalar. If this check passes, then a call is made up the stack to 99 | `validate_args` in `Atom`. 100 | 101 | Notice that we did not check whether $y > 0$. This is because we 102 | generally do not know the value of an argument at the time of 103 | construction. For instance, `y` may contain a `Variable` that we are 104 | solving for in a problem, and we cannot define the problem if 105 | validation fails. It is the user's responsibility to include domain 106 | constraints during problem construction. 107 | 108 | To facilitate this, we define a method for that returns a list of 109 | `Constraint` objects delineating an atom's domain. For the QoL 110 | function, we need only constrain its second argument to be 111 | positive. Strict inequalities are not supported in `CVXR`, so we 112 | impose a weak inequality $y \geq 0$ as shown below. 113 | 114 | ```{r, eval=FALSE} 115 | setMethod(".domain", "QuadOverLin", function(object) { list(object@args[[2]] >= 0) }) 116 | ``` 117 | 118 | Both validation and domain methods may be dropped if an atom's 119 | arguments span the reals. 120 | 121 | The `to_numeric` method is always required. It takes as input an atom and a list `values` containing the numeric values of the its arguments (given in the same order as in `args`), then evaluates the atom at these values. For the QoL function, this method is simply 122 | 123 | ```{r, eval=FALSE} 124 | setMethod("to_numeric", "QuadOverLin", function(object, values) { sum(values[[1]]^2) / values[[2]] }) 125 | ``` 126 | 127 | Note the difference between `object@args` and `values`. The former is 128 | a list of `Expression`s, which represent a composition of constants, 129 | variables, and atoms. The latter is a list of R numeric constants like 130 | `numeric` and `matrix`, denoting the values of the corresponding 131 | expressions. 132 | 133 | ### Dimension, Sign, and Curvature 134 | 135 | For DCP analysis to work, we must explicitly define each atom's dimension, sign, and curvature. `CVXR` uses this information when applying the composition rules to an expression. It is easy to see that $f$ is scalar-valued, nonnegative, and convex over its domain. These properties are encoded in 136 | 137 | ```{r, eval=FALSE} 138 | setMethod("size_from_args", "QuadOverLin", function(object) { c(1,1) }) 139 | setMethod("sign_from_args", "QuadOverLin", function(object) { c(TRUE, FALSE) }) 140 | setMethod("is_atom_convex", "QuadOverLin", function(object) { TRUE }) 141 | setMethod("is_atom_concave", "QuadOverLin", function(object) { FALSE }) 142 | ``` 143 | 144 | Here, the `sign_from_args` function returns a vector of two logical 145 | values - the first indicates if the atom's value is positive, and the 146 | second if it is negative. We also define whether the atom is weakly 147 | increasing or decreasing in each of its arguments. By taking 148 | derivatives, we find that $f$ is weakly decreasing in $y$, weakly 149 | increasing in $x$ on $\mathbf{R}_+^n$, and weakly decreasing in $x$ on 150 | $\mathbf{R}_-^n$. These properties are spelled out in 151 | 152 | ```{r, eval=FALSE} 153 | setMethod("is_incr", "QuadOverLin", function(object, idx) { (idx == 1) && is_positive(object@args[[idx]]) }) 154 | setMethod("is_decr", "QuadOverLin", function(object, idx) { ((idx == 1) && is_negative(object@args[[idx]])) || (idx == 2) }) 155 | ``` 156 | 157 | where `idx` is the index of the argument of interest. We call the 158 | `Expression` class method `is_positive` (resp. `is_negative`) to determine 159 | if an argument is nonnegative (resp. nonpositive). The user may be 160 | tempted to write `object@args[[1]] >= 0`, but this is **incorrect** 161 | because it returns a `Constraint` rather than a logical value. 162 | 163 | ### Additional Characteristics 164 | 165 | The methods we have just described are the bare minimum required to 166 | define an atom. Additional methods may be implemented that provide further 167 | functional characterization. For instance, we furnish a method for 168 | computing the gradient of the QoL function, 169 | 170 | $$ 171 | \nabla_x f(x,y) = \frac{2x}{y}, \quad \nabla_y f(x,y) = -\frac{\|x\|_2^2}{y^2}, 172 | $$ 173 | 174 | at a point $(x,y)$. 175 | 176 | ```{r, eval=FALSE} 177 | setMethod(".grad", "QuadOverLin", function(object, values) { 178 | X <- values[[1]] 179 | y <- as.numeric(values[[2]]) 180 | if(y <= 0) 181 | return(list(NA_real_, NA_real_)) 182 | else { 183 | # DX = 2X/y, Dy = -||X||^2_2/y^2 184 | Dy <- -sum(X^2)/y^2 185 | Dy <- Matrix(Dy, sparse = TRUE) 186 | DX <- 2.0*X/y 187 | DX <- Matrix(as.numeric(t(DX)), sparse = TRUE) 188 | return(list(DX, Dy)) 189 | } 190 | }) 191 | ``` 192 | 193 | This method calculates the vectors $\nabla_x f(x,y)$ and $\nabla_y 194 | f(x,y))$, wraps each in a sparse `Matrix`, and returns them in a list 195 | ordered exactly like in the input values. If $(x,y) \notin 196 | \mathbf{dom}\;f$, all gradients are set to `NA`. 197 | 198 | ## Canonicalization 199 | 200 | Once `CVXR` verifies a problem is DCP, it converts that problem into a 201 | solver-compatible form. This *canonicalization* process is carried out 202 | through a series of calls to individual atom canonicalizers. 203 | To implement an atom, we must explicitly derive its canonicalizer for a conic program. 204 | Below we provide a derivation based on the graph implementation. 205 | 206 | A function $g: \mathbf{R}^n \rightarrow \mathbf{R}$ is convex if and 207 | only if its epigraph 208 | 209 | $$ 210 | \mathbf{epi}\;g = \{(x,t) \in \mathbf{R}^n \times \mathbf{R}: g(x) \leq t\} 211 | $$ 212 | is a convex set. Then, it can be written as 213 | $$ 214 | g(x) = \inf \{t \in \mathbf{R}: (x,t) \in \mathbf{epi}\;g \}. 215 | $$ 216 | 217 | A similar relationship holds between concave functions and their 218 | hypographs. The *graph implementation* of a function is a 219 | representation of its epigraph or hypograph as a disciplined convex 220 | feasibility problem. This offers an elegant means of defining a 221 | nondifferentiable function in terms of a canonical optimization 222 | problem, which can be directly evaluated by a solver. 223 | 224 | For instance, the QoL function can be written as a second-order cone 225 | program (SOCP). Given $(x,y) \in \mathbf{dom}\;f$, the inequality 226 | $f(x,y) \leq t$ is equivalent to 227 | 228 | $$ 229 | \begin{align*} 230 | 4\|x\|_2^2 &\leq 4ty = (y+t)^2 - (y-t)^2 \\ 231 | (y-t)^2 + \|2x\|_2^2 &\leq (y+t)^2 \\ 232 | \left\|\begin{pmatrix} y-t \\ 2x \end{pmatrix}\right\|_2 &\leq y+t 233 | \end{align*} 234 | $$ 235 | so that 236 | $$ 237 | f(x,y) = \inf \left\{t \in \mathbf{R}: \left(\begin{pmatrix} y-t \\ 2x \end{pmatrix}, y+t\right) \in \mathcal{K} \right\}, 238 | $$ 239 | where $\mathcal{K} := \{(u,v) \in \mathbf{R}^{n+1} \times \mathbf{R}: 240 | \|u\|_2 \leq v\}$ is a second-order cone. Thus, $f(x,y)$ is the 241 | solution to an SOCP, which may be evaluated using any conic 242 | solver. The canonicalizer for the QoL function takes as input $(x,y)$ 243 | and outputs the above SOCP. 244 | 245 | In `CVXR`, this canonicalizer function is defined as 246 | 247 | ```{r, eval=FALSE} 248 | QuadOverLin.graph_implementation <- function(arg_objs, size, data = NA_real_) { 249 | x <- arg_objs[[1]] 250 | y <- arg_objs[[2]] # Known to be a scalar. 251 | t <- create_var(c(1,1)) 252 | two <- create_const(2, c(1,1)) 253 | constraints <- list(SOC(lo.sum_expr(list(y, t)), 254 | list(lo.sub_expr(y, t), 255 | lo.mul_expr(two, x, x$size))), 256 | create_geq(y)) 257 | list(t, constraints) 258 | } 259 | ``` 260 | 261 | It takes as input a list of arguments `arg_objs`, which specify the input values $(x,y)$, the `size` of the resulting expression, and a list of additional `data` required by the atom. 262 | The first two lines of the function extract $x$ and $y$ from `args`, the third line constructs the `Variable` $t \in \mathbf{R}$, and the fourth line defines the constant $2$. 263 | The fifth line forms the constraint $\left(\begin{pmatrix} y-t \\ 2x \end{pmatrix}, y+t\right) \in \mathcal{K}$ with a call to the constructor `SOC`. 264 | If $u \in \mathbf{R}^n$, then `SOC(t,u)` enforces $\|u\|_2 \leq t$. Finally, the canonicalizer returns a list containing 265 | $t$, the epigraph variable, and a list of `Constraint`s from the graph implementation - in this case, the second-order cone constraint and $y \geq 0$. 266 | 267 | ## Putting It All Together 268 | 269 | With this machinery in place, we are ready to use the `QuadOverLin` 270 | atom in a problem. Suppose we are given $A \in \mathbf{R}^{m \times 271 | n}$ and $b \in \mathbf{R}^m$, and we would like to solve 272 | 273 | $$ 274 | \begin{array}{ll} 275 | \mbox{minimize} & f(x,y) \\ 276 | \mbox{subject to} & Ax = b 277 | \end{array} 278 | $$ 279 | 280 | with respect to $x \in \mathbf{R}^n$ and $y \in \mathbf{R}_{++}$. In 281 | `CVXR`, we write 282 | 283 | ```{r, eval=FALSE} 284 | x <- Variable(n) 285 | y <- Variable() 286 | obj <- quad_over_lin(x,y) 287 | constr <- c(domain(obj), A*x == b) 288 | prob <- Problem(Minimize(obj), constr) 289 | solve(prob) 290 | ``` 291 | 292 | Once `solve` is invoked, `CVXR` checks if the problem is DCP with calls 293 | to `is_atom_convex`, `is_incr`, and `is_decr` for each atom in the 294 | expression tree. It then converts the problem as modeled to a form compatible with the desired 295 | solver. 296 | 297 | In our example, this requires a transformation into a 298 | cone program. Substituting in the graph implementation of $f$, our 299 | original problem can be written as 300 | 301 | $$ 302 | \begin{array}{ll} 303 | \mbox{minimize} & t \\ 304 | \mbox{subject to} & Ax = b, \quad (x,y) \in \mathbf{dom}\;f, \quad \left(\begin{pmatrix} y-t \\ 2x \end{pmatrix}, y+t\right) \in \mathcal{K}, 305 | \end{array} 306 | $$ 307 | 308 | where $t \in \mathbf{R}$ is the additional epigraph variable. If $b 309 | \in \mathbf{null}\;A$, a solution is trivially $x^* = \vec{0}$ with 310 | any $y^* > 0$. Otherwise, the point $y = 0$ is infeasible, so we can 311 | relax the domain constraint to get 312 | 313 | $$ 314 | \begin{array}{ll} 315 | \mbox{minimize} & t \\ 316 | \mbox{subject to} & Ax = b, \quad y \geq 0, \quad \left(\begin{pmatrix} y-t \\ 2x \end{pmatrix}, y+t\right) \in \mathcal{K}. 317 | \end{array} 318 | $$ 319 | 320 | In `CVXR`, this SOCP canonicalization is performed automatically via a 321 | call to `QuadOverLin.graph_implementation`. Then, the relevant matrices are formed and passed to the selected conic 322 | solver. After the solver returns $(x^*,y^*,t^*)$, the transformation is reversed to obtain the optimal point $(x^*,y^*)$ and 323 | objective value $f(x^*,y^*)$ for the original problem. 324 | 325 | -------------------------------------------------------------------------------- /98-future.Rmd: -------------------------------------------------------------------------------- 1 | # Future Enhancements 2 | 3 | ## Version 1.0 4 | - Modular system of reductions [@reductions:2018] 5 | * New standard forms and solvers 6 | * Advanced user inputs, e.g. complex variables 7 | - Disciplined geometric programming [@DGP] 8 | - Variable and parameter attributes 9 | - Transformations: linearize, indicator, partial optimization 10 | 11 | ## Internal Changes 12 | - Uniform interface for solver settings 13 | - Speed improvements by moving operations to C++ 14 | - Possibly connect to `ROI` library 15 | 16 | # References -------------------------------------------------------------------------------- /99-references.Rmd: -------------------------------------------------------------------------------- 1 | `r if (knitr::is_html_output()) ' 2 | # References {-} 3 | '` 4 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright [yyyy] [name of copyright owner] 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ======= 2 | # `CVXR` Tutorial 3 | 4 | This is the bookdown for the [`CVXR` 5 | tutorial](http://www.user2019.fr/tutorials/) at [useR! 6 | 2019](http://www.user2019.fr). 7 | 8 | ## Preparatory Steps 9 | 10 | 1. Clone this repo or download this repo as one master zip file. 11 | 12 | 2. Open the file `_book/index.html` in your browser. 13 | 14 | 3. Ensure that you follow the steps outlined in _Chapter 2: Getting 15 | Started_ to be ready for the tutorial. 16 | 17 | If you want a single markdown file with all the code, you will find it 18 | in the `extra/code` folder. 19 | 20 | We are happy to see you at _useR! 2019_, Toulouse! 21 | -------------------------------------------------------------------------------- /_book/cvxr_tutorial_files/figure-html/unnamed-chunk-118-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bnaras/cvxr_tutorial/7b8dde276eb749a91c17511d8829bd3eb7971d5e/_book/cvxr_tutorial_files/figure-html/unnamed-chunk-118-1.png -------------------------------------------------------------------------------- /_book/cvxr_tutorial_files/figure-html/unnamed-chunk-122-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bnaras/cvxr_tutorial/7b8dde276eb749a91c17511d8829bd3eb7971d5e/_book/cvxr_tutorial_files/figure-html/unnamed-chunk-122-1.png -------------------------------------------------------------------------------- /_book/cvxr_tutorial_files/figure-html/unnamed-chunk-123-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bnaras/cvxr_tutorial/7b8dde276eb749a91c17511d8829bd3eb7971d5e/_book/cvxr_tutorial_files/figure-html/unnamed-chunk-123-1.png -------------------------------------------------------------------------------- /_book/cvxr_tutorial_files/figure-html/unnamed-chunk-66-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bnaras/cvxr_tutorial/7b8dde276eb749a91c17511d8829bd3eb7971d5e/_book/cvxr_tutorial_files/figure-html/unnamed-chunk-66-1.png -------------------------------------------------------------------------------- /_book/figures/02/convexchord.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bnaras/cvxr_tutorial/7b8dde276eb749a91c17511d8829bd3eb7971d5e/_book/figures/02/convexchord.gif -------------------------------------------------------------------------------- /_book/libs/anchor-sections-1.0.1/anchor-sections.css: -------------------------------------------------------------------------------- 1 | /* Styles for section anchors */ 2 | a.anchor-section {margin-left: 10px; visibility: hidden; color: inherit;} 3 | a.anchor-section::before {content: '#';} 4 | .hasAnchor:hover a.anchor-section {visibility: visible;} 5 | ul > li > .anchor-section {display: none;} 6 | -------------------------------------------------------------------------------- /_book/libs/anchor-sections-1.0.1/anchor-sections.js: -------------------------------------------------------------------------------- 1 | // Anchor sections v1.0 written by Atsushi Yasumoto on Oct 3rd, 2020. 2 | document.addEventListener('DOMContentLoaded', function() { 3 | // Do nothing if AnchorJS is used 4 | if (typeof window.anchors === 'object' && anchors.hasOwnProperty('hasAnchorJSLink')) { 5 | return; 6 | } 7 | 8 | const h = document.querySelectorAll('h1, h2, h3, h4, h5, h6'); 9 | 10 | // Do nothing if sections are already anchored 11 | if (Array.from(h).some(x => x.classList.contains('hasAnchor'))) { 12 | return null; 13 | } 14 | 15 | // Use section id when pandoc runs with --section-divs 16 | const section_id = function(x) { 17 | return ((x.classList.contains('section') || (x.tagName === 'SECTION')) 18 | ? x.id : ''); 19 | }; 20 | 21 | // Add anchors 22 | h.forEach(function(x) { 23 | const id = x.id || section_id(x.parentElement); 24 | if (id === '' || x.matches(':empty')) { 25 | return null; 26 | } 27 | let anchor = document.createElement('a'); 28 | anchor.href = '#' + id; 29 | anchor.classList = ['anchor-section']; 30 | x.classList.add('hasAnchor'); 31 | x.appendChild(anchor); 32 | }); 33 | }); 34 | -------------------------------------------------------------------------------- /_book/libs/gitbook-2.6.7/css/fontawesome/fontawesome-webfont.ttf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bnaras/cvxr_tutorial/7b8dde276eb749a91c17511d8829bd3eb7971d5e/_book/libs/gitbook-2.6.7/css/fontawesome/fontawesome-webfont.ttf -------------------------------------------------------------------------------- /_book/libs/gitbook-2.6.7/css/plugin-bookdown.css: -------------------------------------------------------------------------------- 1 | .book .book-header h1 { 2 | padding-left: 20px; 3 | padding-right: 20px; 4 | } 5 | .book .book-header.fixed { 6 | position: fixed; 7 | right: 0; 8 | top: 0; 9 | left: 0; 10 | border-bottom: 1px solid rgba(0,0,0,.07); 11 | } 12 | span.search-highlight { 13 | background-color: #ffff88; 14 | } 15 | @media (min-width: 600px) { 16 | .book.with-summary .book-header.fixed { 17 | left: 300px; 18 | } 19 | } 20 | @media (max-width: 1240px) { 21 | .book .book-body.fixed { 22 | top: 50px; 23 | } 24 | .book .book-body.fixed .body-inner { 25 | top: auto; 26 | } 27 | } 28 | @media (max-width: 600px) { 29 | .book.with-summary .book-header.fixed { 30 | left: calc(100% - 60px); 31 | min-width: 300px; 32 | } 33 | .book.with-summary .book-body { 34 | transform: none; 35 | left: calc(100% - 60px); 36 | min-width: 300px; 37 | } 38 | .book .book-body.fixed { 39 | top: 0; 40 | } 41 | } 42 | 43 | .book .book-body.fixed .body-inner { 44 | top: 50px; 45 | } 46 | .book .book-body .page-wrapper .page-inner section.normal sub, .book .book-body .page-wrapper .page-inner section.normal sup { 47 | font-size: 85%; 48 | } 49 | 50 | @media print { 51 | .book .book-summary, .book .book-body .book-header, .fa { 52 | display: none !important; 53 | } 54 | .book .book-body.fixed { 55 | left: 0px; 56 | } 57 | .book .book-body,.book .book-body .body-inner, .book.with-summary { 58 | overflow: visible !important; 59 | } 60 | } 61 | .kable_wrapper { 62 | border-spacing: 20px 0; 63 | border-collapse: separate; 64 | border: none; 65 | margin: auto; 66 | } 67 | .kable_wrapper > tbody > tr > td { 68 | vertical-align: top; 69 | } 70 | .book .book-body .page-wrapper .page-inner section.normal table tr.header { 71 | border-top-width: 2px; 72 | } 73 | .book .book-body .page-wrapper .page-inner section.normal table tr:last-child td { 74 | border-bottom-width: 2px; 75 | } 76 | .book .book-body .page-wrapper .page-inner section.normal table td, .book .book-body .page-wrapper .page-inner section.normal table th { 77 | border-left: none; 78 | border-right: none; 79 | } 80 | .book .book-body .page-wrapper .page-inner section.normal table.kable_wrapper > tbody > tr, .book .book-body .page-wrapper .page-inner section.normal table.kable_wrapper > tbody > tr > td { 81 | border-top: none; 82 | } 83 | .book .book-body .page-wrapper .page-inner section.normal table.kable_wrapper > tbody > tr:last-child > td { 84 | border-bottom: none; 85 | } 86 | 87 | div.theorem, div.lemma, div.corollary, div.proposition, div.conjecture { 88 | font-style: italic; 89 | } 90 | span.theorem, span.lemma, span.corollary, span.proposition, span.conjecture { 91 | font-style: normal; 92 | } 93 | div.proof>*:last-child:after { 94 | content: "\25a2"; 95 | float: right; 96 | } 97 | .header-section-number { 98 | padding-right: .5em; 99 | } 100 | -------------------------------------------------------------------------------- /_book/libs/gitbook-2.6.7/css/plugin-clipboard.css: -------------------------------------------------------------------------------- 1 | div.sourceCode { 2 | position: relative; 3 | } 4 | 5 | .copy-to-clipboard-button { 6 | position: absolute; 7 | right: 0; 8 | top: 0; 9 | visibility: hidden; 10 | } 11 | 12 | .copy-to-clipboard-button:focus { 13 | outline: 0; 14 | } 15 | 16 | div.sourceCode:hover > .copy-to-clipboard-button { 17 | visibility: visible; 18 | } 19 | -------------------------------------------------------------------------------- /_book/libs/gitbook-2.6.7/css/plugin-fontsettings.css: -------------------------------------------------------------------------------- 1 | /* 2 | * Theme 1 3 | */ 4 | .color-theme-1 .dropdown-menu { 5 | background-color: #111111; 6 | border-color: #7e888b; 7 | } 8 | .color-theme-1 .dropdown-menu .dropdown-caret .caret-inner { 9 | border-bottom: 9px solid #111111; 10 | } 11 | .color-theme-1 .dropdown-menu .buttons { 12 | border-color: #7e888b; 13 | } 14 | .color-theme-1 .dropdown-menu .button { 15 | color: #afa790; 16 | } 17 | .color-theme-1 .dropdown-menu .button:hover { 18 | color: #73553c; 19 | } 20 | /* 21 | * Theme 2 22 | */ 23 | .color-theme-2 .dropdown-menu { 24 | background-color: #2d3143; 25 | border-color: #272a3a; 26 | } 27 | .color-theme-2 .dropdown-menu .dropdown-caret .caret-inner { 28 | border-bottom: 9px solid #2d3143; 29 | } 30 | .color-theme-2 .dropdown-menu .buttons { 31 | border-color: #272a3a; 32 | } 33 | .color-theme-2 .dropdown-menu .button { 34 | color: #62677f; 35 | } 36 | .color-theme-2 .dropdown-menu .button:hover { 37 | color: #f4f4f5; 38 | } 39 | .book .book-header .font-settings .font-enlarge { 40 | line-height: 30px; 41 | font-size: 1.4em; 42 | } 43 | .book .book-header .font-settings .font-reduce { 44 | line-height: 30px; 45 | font-size: 1em; 46 | } 47 | 48 | /* sidebar transition background */ 49 | div.book.color-theme-1 { 50 | background: #f3eacb; 51 | } 52 | .book.color-theme-1 .book-body { 53 | color: #704214; 54 | background: #f3eacb; 55 | } 56 | .book.color-theme-1 .book-body .page-wrapper .page-inner section { 57 | background: #f3eacb; 58 | } 59 | 60 | /* sidebar transition background */ 61 | div.book.color-theme-2 { 62 | background: #1c1f2b; 63 | } 64 | 65 | .book.color-theme-2 .book-body { 66 | color: #bdcadb; 67 | background: #1c1f2b; 68 | } 69 | .book.color-theme-2 .book-body .page-wrapper .page-inner section { 70 | background: #1c1f2b; 71 | } 72 | .book.font-size-0 .book-body .page-inner section { 73 | font-size: 1.2rem; 74 | } 75 | .book.font-size-1 .book-body .page-inner section { 76 | font-size: 1.4rem; 77 | } 78 | .book.font-size-2 .book-body .page-inner section { 79 | font-size: 1.6rem; 80 | } 81 | .book.font-size-3 .book-body .page-inner section { 82 | font-size: 2.2rem; 83 | } 84 | .book.font-size-4 .book-body .page-inner section { 85 | font-size: 4rem; 86 | } 87 | .book.font-family-0 { 88 | font-family: Georgia, serif; 89 | } 90 | .book.font-family-1 { 91 | font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; 92 | } 93 | .book.color-theme-1 .book-body .page-wrapper .page-inner section.normal { 94 | color: #704214; 95 | } 96 | .book.color-theme-1 .book-body .page-wrapper .page-inner section.normal a { 97 | color: inherit; 98 | } 99 | .book.color-theme-1 .book-body .page-wrapper .page-inner section.normal h1, 100 | .book.color-theme-1 .book-body .page-wrapper .page-inner section.normal h2, 101 | .book.color-theme-1 .book-body .page-wrapper .page-inner section.normal h3, 102 | .book.color-theme-1 .book-body .page-wrapper .page-inner section.normal h4, 103 | .book.color-theme-1 .book-body .page-wrapper .page-inner section.normal h5, 104 | .book.color-theme-1 .book-body .page-wrapper .page-inner section.normal h6 { 105 | color: inherit; 106 | } 107 | .book.color-theme-1 .book-body .page-wrapper .page-inner section.normal h1, 108 | .book.color-theme-1 .book-body .page-wrapper .page-inner section.normal h2 { 109 | border-color: inherit; 110 | } 111 | .book.color-theme-1 .book-body .page-wrapper .page-inner section.normal h6 { 112 | color: inherit; 113 | } 114 | .book.color-theme-1 .book-body .page-wrapper .page-inner section.normal hr { 115 | background-color: inherit; 116 | } 117 | .book.color-theme-1 .book-body .page-wrapper .page-inner section.normal blockquote { 118 | border-color: #c4b29f; 119 | opacity: 0.9; 120 | } 121 | .book.color-theme-1 .book-body .page-wrapper .page-inner section.normal pre, 122 | .book.color-theme-1 .book-body .page-wrapper .page-inner section.normal code { 123 | background: #fdf6e3; 124 | color: #657b83; 125 | border-color: #f8df9c; 126 | } 127 | .book.color-theme-1 .book-body .page-wrapper .page-inner section.normal .highlight { 128 | background-color: inherit; 129 | } 130 | .book.color-theme-1 .book-body .page-wrapper .page-inner section.normal table th, 131 | .book.color-theme-1 .book-body .page-wrapper .page-inner section.normal table td { 132 | border-color: #f5d06c; 133 | } 134 | .book.color-theme-1 .book-body .page-wrapper .page-inner section.normal table tr { 135 | color: inherit; 136 | background-color: #fdf6e3; 137 | border-color: #444444; 138 | } 139 | .book.color-theme-1 .book-body .page-wrapper .page-inner section.normal table tr:nth-child(2n) { 140 | background-color: #fbeecb; 141 | } 142 | .book.color-theme-2 .book-body .page-wrapper .page-inner section.normal { 143 | color: #bdcadb; 144 | } 145 | .book.color-theme-2 .book-body .page-wrapper .page-inner section.normal a { 146 | color: #3eb1d0; 147 | } 148 | .book.color-theme-2 .book-body .page-wrapper .page-inner section.normal h1, 149 | .book.color-theme-2 .book-body .page-wrapper .page-inner section.normal h2, 150 | .book.color-theme-2 .book-body .page-wrapper .page-inner section.normal h3, 151 | .book.color-theme-2 .book-body .page-wrapper .page-inner section.normal h4, 152 | .book.color-theme-2 .book-body .page-wrapper .page-inner section.normal h5, 153 | .book.color-theme-2 .book-body .page-wrapper .page-inner section.normal h6 { 154 | color: #fffffa; 155 | } 156 | .book.color-theme-2 .book-body .page-wrapper .page-inner section.normal h1, 157 | .book.color-theme-2 .book-body .page-wrapper .page-inner section.normal h2 { 158 | border-color: #373b4e; 159 | } 160 | .book.color-theme-2 .book-body .page-wrapper .page-inner section.normal h6 { 161 | color: #373b4e; 162 | } 163 | .book.color-theme-2 .book-body .page-wrapper .page-inner section.normal hr { 164 | background-color: #373b4e; 165 | } 166 | .book.color-theme-2 .book-body .page-wrapper .page-inner section.normal blockquote { 167 | border-color: #373b4e; 168 | } 169 | .book.color-theme-2 .book-body .page-wrapper .page-inner section.normal pre, 170 | .book.color-theme-2 .book-body .page-wrapper .page-inner section.normal code { 171 | color: #9dbed8; 172 | background: #2d3143; 173 | border-color: #2d3143; 174 | } 175 | .book.color-theme-2 .book-body .page-wrapper .page-inner section.normal .highlight { 176 | background-color: #282a39; 177 | } 178 | .book.color-theme-2 .book-body .page-wrapper .page-inner section.normal table th, 179 | .book.color-theme-2 .book-body .page-wrapper .page-inner section.normal table td { 180 | border-color: #3b3f54; 181 | } 182 | .book.color-theme-2 .book-body .page-wrapper .page-inner section.normal table tr { 183 | color: #b6c2d2; 184 | background-color: #2d3143; 185 | border-color: #3b3f54; 186 | } 187 | .book.color-theme-2 .book-body .page-wrapper .page-inner section.normal table tr:nth-child(2n) { 188 | background-color: #35394b; 189 | } 190 | .book.color-theme-1 .book-header { 191 | color: #afa790; 192 | background: transparent; 193 | } 194 | .book.color-theme-1 .book-header .btn { 195 | color: #afa790; 196 | } 197 | .book.color-theme-1 .book-header .btn:hover { 198 | color: #73553c; 199 | background: none; 200 | } 201 | .book.color-theme-1 .book-header h1 { 202 | color: #704214; 203 | } 204 | .book.color-theme-2 .book-header { 205 | color: #7e888b; 206 | background: transparent; 207 | } 208 | .book.color-theme-2 .book-header .btn { 209 | color: #3b3f54; 210 | } 211 | .book.color-theme-2 .book-header .btn:hover { 212 | color: #fffff5; 213 | background: none; 214 | } 215 | .book.color-theme-2 .book-header h1 { 216 | color: #bdcadb; 217 | } 218 | .book.color-theme-1 .book-body .navigation { 219 | color: #afa790; 220 | } 221 | .book.color-theme-1 .book-body .navigation:hover { 222 | color: #73553c; 223 | } 224 | .book.color-theme-2 .book-body .navigation { 225 | color: #383f52; 226 | } 227 | .book.color-theme-2 .book-body .navigation:hover { 228 | color: #fffff5; 229 | } 230 | /* 231 | * Theme 1 232 | */ 233 | .book.color-theme-1 .book-summary { 234 | color: #afa790; 235 | background: #111111; 236 | border-right: 1px solid rgba(0, 0, 0, 0.07); 237 | } 238 | .book.color-theme-1 .book-summary .book-search { 239 | background: transparent; 240 | } 241 | .book.color-theme-1 .book-summary .book-search input, 242 | .book.color-theme-1 .book-summary .book-search input:focus { 243 | border: 1px solid transparent; 244 | } 245 | .book.color-theme-1 .book-summary ul.summary li.divider { 246 | background: #7e888b; 247 | box-shadow: none; 248 | } 249 | .book.color-theme-1 .book-summary ul.summary li i.fa-check { 250 | color: #33cc33; 251 | } 252 | .book.color-theme-1 .book-summary ul.summary li.done > a { 253 | color: #877f6a; 254 | } 255 | .book.color-theme-1 .book-summary ul.summary li a, 256 | .book.color-theme-1 .book-summary ul.summary li span { 257 | color: #877f6a; 258 | background: transparent; 259 | font-weight: normal; 260 | } 261 | .book.color-theme-1 .book-summary ul.summary li.active > a, 262 | .book.color-theme-1 .book-summary ul.summary li a:hover { 263 | color: #704214; 264 | background: transparent; 265 | font-weight: normal; 266 | } 267 | /* 268 | * Theme 2 269 | */ 270 | .book.color-theme-2 .book-summary { 271 | color: #bcc1d2; 272 | background: #2d3143; 273 | border-right: none; 274 | } 275 | .book.color-theme-2 .book-summary .book-search { 276 | background: transparent; 277 | } 278 | .book.color-theme-2 .book-summary .book-search input, 279 | .book.color-theme-2 .book-summary .book-search input:focus { 280 | border: 1px solid transparent; 281 | } 282 | .book.color-theme-2 .book-summary ul.summary li.divider { 283 | background: #272a3a; 284 | box-shadow: none; 285 | } 286 | .book.color-theme-2 .book-summary ul.summary li i.fa-check { 287 | color: #33cc33; 288 | } 289 | .book.color-theme-2 .book-summary ul.summary li.done > a { 290 | color: #62687f; 291 | } 292 | .book.color-theme-2 .book-summary ul.summary li a, 293 | .book.color-theme-2 .book-summary ul.summary li span { 294 | color: #c1c6d7; 295 | background: transparent; 296 | font-weight: 600; 297 | } 298 | .book.color-theme-2 .book-summary ul.summary li.active > a, 299 | .book.color-theme-2 .book-summary ul.summary li a:hover { 300 | color: #f4f4f5; 301 | background: #252737; 302 | font-weight: 600; 303 | } 304 | -------------------------------------------------------------------------------- /_book/libs/gitbook-2.6.7/css/plugin-search.css: -------------------------------------------------------------------------------- 1 | .book .book-summary .book-search { 2 | padding: 6px; 3 | background: transparent; 4 | position: absolute; 5 | top: -50px; 6 | left: 0px; 7 | right: 0px; 8 | transition: top 0.5s ease; 9 | } 10 | .book .book-summary .book-search input, 11 | .book .book-summary .book-search input:focus, 12 | .book .book-summary .book-search input:hover { 13 | width: 100%; 14 | background: transparent; 15 | border: 1px solid #ccc; 16 | box-shadow: none; 17 | outline: none; 18 | line-height: 22px; 19 | padding: 7px 4px; 20 | color: inherit; 21 | box-sizing: border-box; 22 | } 23 | .book.with-search .book-summary .book-search { 24 | top: 0px; 25 | } 26 | .book.with-search .book-summary ul.summary { 27 | top: 50px; 28 | } 29 | .with-search .summary li[data-level] a[href*=".html#"] { 30 | display: none; 31 | } 32 | -------------------------------------------------------------------------------- /_book/libs/gitbook-2.6.7/css/plugin-table.css: -------------------------------------------------------------------------------- 1 | .book .book-body .page-wrapper .page-inner section.normal table{display:table;width:100%;border-collapse:collapse;border-spacing:0;overflow:auto}.book .book-body .page-wrapper .page-inner section.normal table td,.book .book-body .page-wrapper .page-inner section.normal table th{padding:6px 13px;border:1px solid #ddd}.book .book-body .page-wrapper .page-inner section.normal table tr{background-color:#fff;border-top:1px solid #ccc}.book .book-body .page-wrapper .page-inner section.normal table tr:nth-child(2n){background-color:#f8f8f8}.book .book-body .page-wrapper .page-inner section.normal table th{font-weight:700} 2 | -------------------------------------------------------------------------------- /_book/libs/gitbook-2.6.7/js/clipboard.min.js: -------------------------------------------------------------------------------- 1 | /*! 2 | * clipboard.js v2.0.4 3 | * https://zenorocha.github.io/clipboard.js 4 | * 5 | * Licensed MIT © Zeno Rocha 6 | */ 7 | !function(t,e){"object"==typeof exports&&"object"==typeof module?module.exports=e():"function"==typeof define&&define.amd?define([],e):"object"==typeof exports?exports.ClipboardJS=e():t.ClipboardJS=e()}(this,function(){return function(n){var o={};function r(t){if(o[t])return o[t].exports;var e=o[t]={i:t,l:!1,exports:{}};return n[t].call(e.exports,e,e.exports,r),e.l=!0,e.exports}return r.m=n,r.c=o,r.d=function(t,e,n){r.o(t,e)||Object.defineProperty(t,e,{enumerable:!0,get:n})},r.r=function(t){"undefined"!=typeof Symbol&&Symbol.toStringTag&&Object.defineProperty(t,Symbol.toStringTag,{value:"Module"}),Object.defineProperty(t,"__esModule",{value:!0})},r.t=function(e,t){if(1&t&&(e=r(e)),8&t)return e;if(4&t&&"object"==typeof e&&e&&e.__esModule)return e;var n=Object.create(null);if(r.r(n),Object.defineProperty(n,"default",{enumerable:!0,value:e}),2&t&&"string"!=typeof e)for(var o in e)r.d(n,o,function(t){return e[t]}.bind(null,o));return n},r.n=function(t){var e=t&&t.__esModule?function(){return t.default}:function(){return t};return r.d(e,"a",e),e},r.o=function(t,e){return Object.prototype.hasOwnProperty.call(t,e)},r.p="",r(r.s=0)}([function(t,e,n){"use strict";var r="function"==typeof Symbol&&"symbol"==typeof Symbol.iterator?function(t){return typeof t}:function(t){return t&&"function"==typeof Symbol&&t.constructor===Symbol&&t!==Symbol.prototype?"symbol":typeof t},i=function(){function o(t,e){for(var n=0;no;o++){for(var r=t[o],s=0;i>s&&(r=this._stack[s](r,o,t),void 0!==r);s++);void 0!==r&&e.push(r)}return e},t.Pipeline.prototype.reset=function(){this._stack=[]},t.Pipeline.prototype.toJSON=function(){return this._stack.map(function(e){return t.Pipeline.warnIfFunctionNotRegistered(e),e.label})},t.Vector=function(){this._magnitude=null,this.list=void 0,this.length=0},t.Vector.Node=function(t,e,n){this.idx=t,this.val=e,this.next=n},t.Vector.prototype.insert=function(e,n){this._magnitude=void 0;var i=this.list;if(!i)return this.list=new t.Vector.Node(e,n,i),this.length++;if(en.idx?n=n.next:(i+=e.val*n.val,e=e.next,n=n.next);return i},t.Vector.prototype.similarity=function(t){return this.dot(t)/(this.magnitude()*t.magnitude())},t.SortedSet=function(){this.length=0,this.elements=[]},t.SortedSet.load=function(t){var e=new this;return e.elements=t,e.length=t.length,e},t.SortedSet.prototype.add=function(){var t,e;for(t=0;t1;){if(r===t)return o;t>r&&(e=o),r>t&&(n=o),i=n-e,o=e+Math.floor(i/2),r=this.elements[o]}return r===t?o:-1},t.SortedSet.prototype.locationFor=function(t){for(var e=0,n=this.elements.length,i=n-e,o=e+Math.floor(i/2),r=this.elements[o];i>1;)t>r&&(e=o),r>t&&(n=o),i=n-e,o=e+Math.floor(i/2),r=this.elements[o];return r>t?o:t>r?o+1:void 0},t.SortedSet.prototype.intersect=function(e){for(var n=new t.SortedSet,i=0,o=0,r=this.length,s=e.length,a=this.elements,h=e.elements;;){if(i>r-1||o>s-1)break;a[i]!==h[o]?a[i]h[o]&&o++:(n.add(a[i]),i++,o++)}return n},t.SortedSet.prototype.clone=function(){var e=new t.SortedSet;return e.elements=this.toArray(),e.length=e.elements.length,e},t.SortedSet.prototype.union=function(t){var e,n,i;return this.length>=t.length?(e=this,n=t):(e=t,n=this),i=e.clone(),i.add.apply(i,n.toArray()),i},t.SortedSet.prototype.toJSON=function(){return this.toArray()},t.Index=function(){this._fields=[],this._ref="id",this.pipeline=new t.Pipeline,this.documentStore=new t.Store,this.tokenStore=new t.TokenStore,this.corpusTokens=new t.SortedSet,this.eventEmitter=new t.EventEmitter,this._idfCache={},this.on("add","remove","update",function(){this._idfCache={}}.bind(this))},t.Index.prototype.on=function(){var t=Array.prototype.slice.call(arguments);return this.eventEmitter.addListener.apply(this.eventEmitter,t)},t.Index.prototype.off=function(t,e){return this.eventEmitter.removeListener(t,e)},t.Index.load=function(e){e.version!==t.version&&t.utils.warn("version mismatch: current "+t.version+" importing "+e.version);var n=new this;return n._fields=e.fields,n._ref=e.ref,n.documentStore=t.Store.load(e.documentStore),n.tokenStore=t.TokenStore.load(e.tokenStore),n.corpusTokens=t.SortedSet.load(e.corpusTokens),n.pipeline=t.Pipeline.load(e.pipeline),n},t.Index.prototype.field=function(t,e){var e=e||{},n={name:t,boost:e.boost||1};return this._fields.push(n),this},t.Index.prototype.ref=function(t){return this._ref=t,this},t.Index.prototype.add=function(e,n){var i={},o=new t.SortedSet,r=e[this._ref],n=void 0===n?!0:n;this._fields.forEach(function(n){var r=this.pipeline.run(t.tokenizer(e[n.name]));i[n.name]=r,t.SortedSet.prototype.add.apply(o,r)},this),this.documentStore.set(r,o),t.SortedSet.prototype.add.apply(this.corpusTokens,o.toArray());for(var s=0;s0&&(i=1+Math.log(this.documentStore.length/n)),this._idfCache[e]=i},t.Index.prototype.search=function(e){var n=this.pipeline.run(t.tokenizer(e)),i=new t.Vector,o=[],r=this._fields.reduce(function(t,e){return t+e.boost},0),s=n.some(function(t){return this.tokenStore.has(t)},this);if(!s)return[];n.forEach(function(e,n,s){var a=1/s.length*this._fields.length*r,h=this,l=this.tokenStore.expand(e).reduce(function(n,o){var r=h.corpusTokens.indexOf(o),s=h.idf(o),l=1,u=new t.SortedSet;if(o!==e){var c=Math.max(3,o.length-e.length);l=1/Math.log(c)}return r>-1&&i.insert(r,a*s*l),Object.keys(h.tokenStore.get(o)).forEach(function(t){u.add(t)}),n.union(u)},new t.SortedSet);o.push(l)},this);var a=o.reduce(function(t,e){return t.intersect(e)});return a.map(function(t){return{ref:t,score:i.similarity(this.documentVector(t))}},this).sort(function(t,e){return e.score-t.score})},t.Index.prototype.documentVector=function(e){for(var n=this.documentStore.get(e),i=n.length,o=new t.Vector,r=0;i>r;r++){var s=n.elements[r],a=this.tokenStore.get(s)[e].tf,h=this.idf(s);o.insert(this.corpusTokens.indexOf(s),a*h)}return o},t.Index.prototype.toJSON=function(){return{version:t.version,fields:this._fields,ref:this._ref,documentStore:this.documentStore.toJSON(),tokenStore:this.tokenStore.toJSON(),corpusTokens:this.corpusTokens.toJSON(),pipeline:this.pipeline.toJSON()}},t.Index.prototype.use=function(t){var e=Array.prototype.slice.call(arguments,1);e.unshift(this),t.apply(this,e)},t.Store=function(){this.store={},this.length=0},t.Store.load=function(e){var n=new this;return n.length=e.length,n.store=Object.keys(e.store).reduce(function(n,i){return n[i]=t.SortedSet.load(e.store[i]),n},{}),n},t.Store.prototype.set=function(t,e){this.has(t)||this.length++,this.store[t]=e},t.Store.prototype.get=function(t){return this.store[t]},t.Store.prototype.has=function(t){return t in this.store},t.Store.prototype.remove=function(t){this.has(t)&&(delete this.store[t],this.length--)},t.Store.prototype.toJSON=function(){return{store:this.store,length:this.length}},t.stemmer=function(){var t={ational:"ate",tional:"tion",enci:"ence",anci:"ance",izer:"ize",bli:"ble",alli:"al",entli:"ent",eli:"e",ousli:"ous",ization:"ize",ation:"ate",ator:"ate",alism:"al",iveness:"ive",fulness:"ful",ousness:"ous",aliti:"al",iviti:"ive",biliti:"ble",logi:"log"},e={icate:"ic",ative:"",alize:"al",iciti:"ic",ical:"ic",ful:"",ness:""},n="[^aeiou]",i="[aeiouy]",o=n+"[^aeiouy]*",r=i+"[aeiou]*",s="^("+o+")?"+r+o,a="^("+o+")?"+r+o+"("+r+")?$",h="^("+o+")?"+r+o+r+o,l="^("+o+")?"+i,u=new RegExp(s),c=new RegExp(h),f=new RegExp(a),d=new RegExp(l),p=/^(.+?)(ss|i)es$/,m=/^(.+?)([^s])s$/,v=/^(.+?)eed$/,y=/^(.+?)(ed|ing)$/,g=/.$/,S=/(at|bl|iz)$/,w=new RegExp("([^aeiouylsz])\\1$"),x=new RegExp("^"+o+i+"[^aeiouwxy]$"),k=/^(.+?[^aeiou])y$/,b=/^(.+?)(ational|tional|enci|anci|izer|bli|alli|entli|eli|ousli|ization|ation|ator|alism|iveness|fulness|ousness|aliti|iviti|biliti|logi)$/,E=/^(.+?)(icate|ative|alize|iciti|ical|ful|ness)$/,_=/^(.+?)(al|ance|ence|er|ic|able|ible|ant|ement|ment|ent|ou|ism|ate|iti|ous|ive|ize)$/,F=/^(.+?)(s|t)(ion)$/,O=/^(.+?)e$/,P=/ll$/,N=new RegExp("^"+o+i+"[^aeiouwxy]$"),T=function(n){var i,o,r,s,a,h,l;if(n.length<3)return n;if(r=n.substr(0,1),"y"==r&&(n=r.toUpperCase()+n.substr(1)),s=p,a=m,s.test(n)?n=n.replace(s,"$1$2"):a.test(n)&&(n=n.replace(a,"$1$2")),s=v,a=y,s.test(n)){var T=s.exec(n);s=u,s.test(T[1])&&(s=g,n=n.replace(s,""))}else if(a.test(n)){var T=a.exec(n);i=T[1],a=d,a.test(i)&&(n=i,a=S,h=w,l=x,a.test(n)?n+="e":h.test(n)?(s=g,n=n.replace(s,"")):l.test(n)&&(n+="e"))}if(s=k,s.test(n)){var T=s.exec(n);i=T[1],n=i+"i"}if(s=b,s.test(n)){var T=s.exec(n);i=T[1],o=T[2],s=u,s.test(i)&&(n=i+t[o])}if(s=E,s.test(n)){var T=s.exec(n);i=T[1],o=T[2],s=u,s.test(i)&&(n=i+e[o])}if(s=_,a=F,s.test(n)){var T=s.exec(n);i=T[1],s=c,s.test(i)&&(n=i)}else if(a.test(n)){var T=a.exec(n);i=T[1]+T[2],a=c,a.test(i)&&(n=i)}if(s=O,s.test(n)){var T=s.exec(n);i=T[1],s=c,a=f,h=N,(s.test(i)||a.test(i)&&!h.test(i))&&(n=i)}return s=P,a=c,s.test(n)&&a.test(n)&&(s=g,n=n.replace(s,"")),"y"==r&&(n=r.toLowerCase()+n.substr(1)),n};return T}(),t.Pipeline.registerFunction(t.stemmer,"stemmer"),t.stopWordFilter=function(e){return e&&t.stopWordFilter.stopWords[e]!==e?e:void 0},t.stopWordFilter.stopWords={a:"a",able:"able",about:"about",across:"across",after:"after",all:"all",almost:"almost",also:"also",am:"am",among:"among",an:"an",and:"and",any:"any",are:"are",as:"as",at:"at",be:"be",because:"because",been:"been",but:"but",by:"by",can:"can",cannot:"cannot",could:"could",dear:"dear",did:"did","do":"do",does:"does",either:"either","else":"else",ever:"ever",every:"every","for":"for",from:"from",get:"get",got:"got",had:"had",has:"has",have:"have",he:"he",her:"her",hers:"hers",him:"him",his:"his",how:"how",however:"however",i:"i","if":"if","in":"in",into:"into",is:"is",it:"it",its:"its",just:"just",least:"least",let:"let",like:"like",likely:"likely",may:"may",me:"me",might:"might",most:"most",must:"must",my:"my",neither:"neither",no:"no",nor:"nor",not:"not",of:"of",off:"off",often:"often",on:"on",only:"only",or:"or",other:"other",our:"our",own:"own",rather:"rather",said:"said",say:"say",says:"says",she:"she",should:"should",since:"since",so:"so",some:"some",than:"than",that:"that",the:"the",their:"their",them:"them",then:"then",there:"there",these:"these",they:"they","this":"this",tis:"tis",to:"to",too:"too",twas:"twas",us:"us",wants:"wants",was:"was",we:"we",were:"were",what:"what",when:"when",where:"where",which:"which","while":"while",who:"who",whom:"whom",why:"why",will:"will","with":"with",would:"would",yet:"yet",you:"you",your:"your"},t.Pipeline.registerFunction(t.stopWordFilter,"stopWordFilter"),t.trimmer=function(t){var e=t.replace(/^\W+/,"").replace(/\W+$/,"");return""===e?void 0:e},t.Pipeline.registerFunction(t.trimmer,"trimmer"),t.TokenStore=function(){this.root={docs:{}},this.length=0},t.TokenStore.load=function(t){var e=new this;return e.root=t.root,e.length=t.length,e},t.TokenStore.prototype.add=function(t,e,n){var n=n||this.root,i=t[0],o=t.slice(1);return i in n||(n[i]={docs:{}}),0===o.length?(n[i].docs[e.ref]=e,void(this.length+=1)):this.add(o,e,n[i])},t.TokenStore.prototype.has=function(t){if(!t)return!1;for(var e=this.root,n=0;n indicates arrow keys):', 82 | '/: navigate to previous/next page', 83 | 's: Toggle sidebar']; 84 | if (config.search !== false) info.push('f: Toggle search input ' + 85 | '(use //Enter in the search input to navigate through search matches; ' + 86 | 'press Esc to cancel search)'); 87 | if (config.info !== false) gitbook.toolbar.createButton({ 88 | icon: 'fa fa-info', 89 | label: 'Information about the toolbar', 90 | position: 'left', 91 | onClick: function(e) { 92 | e.preventDefault(); 93 | window.alert(info.join('\n\n')); 94 | } 95 | }); 96 | 97 | // highlight the current section in TOC 98 | var href = window.location.pathname; 99 | href = href.substr(href.lastIndexOf('/') + 1); 100 | // accentuated characters need to be decoded (#819) 101 | href = decodeURIComponent(href); 102 | if (href === '') href = 'index.html'; 103 | var li = $('a[href^="' + href + location.hash + '"]').parent('li.chapter').first(); 104 | var summary = $('ul.summary'), chaps = summary.find('li.chapter'); 105 | if (li.length === 0) li = chaps.first(); 106 | li.addClass('active'); 107 | chaps.on('click', function(e) { 108 | chaps.removeClass('active'); 109 | $(this).addClass('active'); 110 | gs.set('tocScrollTop', summary.scrollTop()); 111 | }); 112 | 113 | var toc = config.toc; 114 | // collapse TOC items that are not for the current chapter 115 | if (toc && toc.collapse) (function() { 116 | var type = toc.collapse; 117 | if (type === 'none') return; 118 | if (type !== 'section' && type !== 'subsection') return; 119 | // sections under chapters 120 | var toc_sub = summary.children('li[data-level]').children('ul'); 121 | if (type === 'section') { 122 | toc_sub.hide() 123 | .parent().has(li).children('ul').show(); 124 | } else { 125 | toc_sub.children('li').children('ul').hide() 126 | .parent().has(li).children('ul').show(); 127 | } 128 | li.children('ul').show(); 129 | var toc_sub2 = toc_sub.children('li'); 130 | if (type === 'section') toc_sub2.children('ul').hide(); 131 | summary.children('li[data-level]').find('a') 132 | .on('click.bookdown', function(e) { 133 | if (href === $(this).attr('href').replace(/#.*/, '')) 134 | $(this).parent('li').children('ul').toggle(); 135 | }); 136 | })(); 137 | 138 | // add tooltips to the 's that are truncated 139 | $('a').each(function(i, el) { 140 | if (el.offsetWidth >= el.scrollWidth) return; 141 | if (typeof el.title === 'undefined') return; 142 | el.title = el.text; 143 | }); 144 | 145 | // restore TOC scroll position 146 | var pos = gs.get('tocScrollTop'); 147 | if (typeof pos !== 'undefined') summary.scrollTop(pos); 148 | 149 | // highlight the TOC item that has same text as the heading in view as scrolling 150 | if (toc && toc.scroll_highlight !== false && li.length > 0) (function() { 151 | // scroll the current TOC item into viewport 152 | var ht = $(window).height(), rect = li[0].getBoundingClientRect(); 153 | if (rect.top >= ht || rect.top <= 0 || rect.bottom <= 0) { 154 | summary.scrollTop(li[0].offsetTop); 155 | } 156 | // current chapter TOC items 157 | var items = $('a[href^="' + href + '"]').parent('li.chapter'), 158 | m = items.length; 159 | if (m === 0) { 160 | items = summary.find('li.chapter'); 161 | m = items.length; 162 | } 163 | if (m === 0) return; 164 | // all section titles on current page 165 | var hs = bookInner.find('.page-inner').find('h1,h2,h3'), n = hs.length, 166 | ts = hs.map(function(i, el) { return $(el).text(); }); 167 | if (n === 0) return; 168 | var scrollHandler = function(e) { 169 | var ht = $(window).height(); 170 | clearTimeout($.data(this, 'scrollTimer')); 171 | $.data(this, 'scrollTimer', setTimeout(function() { 172 | // find the first visible title in the viewport 173 | for (var i = 0; i < n; i++) { 174 | var rect = hs[i].getBoundingClientRect(); 175 | if (rect.top >= 0 && rect.bottom <= ht) break; 176 | } 177 | if (i === n) return; 178 | items.removeClass('active'); 179 | for (var j = 0; j < m; j++) { 180 | if (items.eq(j).children('a').first().text() === ts[i]) break; 181 | } 182 | if (j === m) j = 0; // highlight the chapter title 183 | // search bottom-up for a visible TOC item to highlight; if an item is 184 | // hidden, we check if its parent is visible, and so on 185 | while (j > 0 && items.eq(j).is(':hidden')) j--; 186 | items.eq(j).addClass('active'); 187 | }, 250)); 188 | }; 189 | bookInner.on('scroll.bookdown', scrollHandler); 190 | bookBody.on('scroll.bookdown', scrollHandler); 191 | })(); 192 | 193 | // do not refresh the page if the TOC item points to the current page 194 | $('a[href="' + href + '"]').parent('li.chapter').children('a') 195 | .on('click', function(e) { 196 | bookInner.scrollTop(0); 197 | bookBody.scrollTop(0); 198 | return false; 199 | }); 200 | 201 | var toolbar = config.toolbar; 202 | if (!toolbar || toolbar.position !== 'static') { 203 | var bookHeader = $('.book-header'); 204 | bookBody.addClass('fixed'); 205 | bookHeader.addClass('fixed') 206 | .css('background-color', bookBody.css('background-color')) 207 | .on('click.bookdown', function(e) { 208 | // the theme may have changed after user clicks the theme button 209 | bookHeader.css('background-color', bookBody.css('background-color')); 210 | }); 211 | } 212 | 213 | }); 214 | 215 | gitbook.events.bind("page.change", function(e) { 216 | // store TOC scroll position 217 | var summary = $('ul.summary'); 218 | gs.set('tocScrollTop', summary.scrollTop()); 219 | }); 220 | 221 | var bookBody = $('.book-body'), bookInner = bookBody.find('.body-inner'); 222 | var chapterTitle = function() { 223 | return bookInner.find('.page-inner').find('h1,h2').first().text(); 224 | }; 225 | var saveScrollPos = function(e) { 226 | // save scroll position before page is reloaded 227 | gs.set('bodyScrollTop', { 228 | body: bookBody.scrollTop(), 229 | inner: bookInner.scrollTop(), 230 | focused: document.hasFocus(), 231 | title: chapterTitle() 232 | }); 233 | }; 234 | $(document).on('servr:reload', saveScrollPos); 235 | 236 | // check if the page is loaded in an iframe (e.g. the RStudio preview window) 237 | var inIFrame = function() { 238 | var inIframe = true; 239 | try { inIframe = window.self !== window.top; } catch (e) {} 240 | return inIframe; 241 | }; 242 | if (inIFrame()) { 243 | $(window).on('blur unload', saveScrollPos); 244 | } 245 | 246 | $(function(e) { 247 | var pos = gs.get('bodyScrollTop'); 248 | if (pos) { 249 | if (pos.title === chapterTitle()) { 250 | if (pos.body !== 0) bookBody.scrollTop(pos.body); 251 | if (pos.inner !== 0) bookInner.scrollTop(pos.inner); 252 | } 253 | } 254 | if ((pos && pos.focused) || !inIFrame()) bookInner.find('.page-wrapper').focus(); 255 | // clear book body scroll position 256 | gs.remove('bodyScrollTop'); 257 | }); 258 | 259 | }); 260 | -------------------------------------------------------------------------------- /_book/libs/gitbook-2.6.7/js/plugin-clipboard.js: -------------------------------------------------------------------------------- 1 | gitbook.require(["gitbook", "jQuery"], function(gitbook, $) { 2 | 3 | var copyButton = ''; 4 | var clipboard; 5 | 6 | gitbook.events.bind("page.change", function() { 7 | 8 | if (!ClipboardJS.isSupported()) return; 9 | 10 | // the page.change event is thrown twice: before and after the page changes 11 | if (clipboard) { 12 | // clipboard is already defined 13 | // we can deduct that we are before page changes 14 | clipboard.destroy(); // destroy the previous events listeners 15 | clipboard = undefined; // reset the clipboard object 16 | return; 17 | } 18 | 19 | $(copyButton).prependTo("div.sourceCode"); 20 | 21 | clipboard = new ClipboardJS(".copy-to-clipboard-button", { 22 | text: function(trigger) { 23 | return trigger.parentNode.textContent; 24 | } 25 | }); 26 | 27 | }); 28 | 29 | }); 30 | -------------------------------------------------------------------------------- /_book/libs/gitbook-2.6.7/js/plugin-fontsettings.js: -------------------------------------------------------------------------------- 1 | gitbook.require(["gitbook", "lodash", "jQuery"], function(gitbook, _, $) { 2 | var fontState; 3 | 4 | var THEMES = { 5 | "white": 0, 6 | "sepia": 1, 7 | "night": 2 8 | }; 9 | 10 | var FAMILY = { 11 | "serif": 0, 12 | "sans": 1 13 | }; 14 | 15 | // Save current font settings 16 | function saveFontSettings() { 17 | gitbook.storage.set("fontState", fontState); 18 | update(); 19 | } 20 | 21 | // Increase font size 22 | function enlargeFontSize(e) { 23 | e.preventDefault(); 24 | if (fontState.size >= 4) return; 25 | 26 | fontState.size++; 27 | saveFontSettings(); 28 | }; 29 | 30 | // Decrease font size 31 | function reduceFontSize(e) { 32 | e.preventDefault(); 33 | if (fontState.size <= 0) return; 34 | 35 | fontState.size--; 36 | saveFontSettings(); 37 | }; 38 | 39 | // Change font family 40 | function changeFontFamily(index, e) { 41 | e.preventDefault(); 42 | 43 | fontState.family = index; 44 | saveFontSettings(); 45 | }; 46 | 47 | // Change type of color 48 | function changeColorTheme(index, e) { 49 | e.preventDefault(); 50 | 51 | var $book = $(".book"); 52 | 53 | if (fontState.theme !== 0) 54 | $book.removeClass("color-theme-"+fontState.theme); 55 | 56 | fontState.theme = index; 57 | if (fontState.theme !== 0) 58 | $book.addClass("color-theme-"+fontState.theme); 59 | 60 | saveFontSettings(); 61 | }; 62 | 63 | function update() { 64 | var $book = gitbook.state.$book; 65 | 66 | $(".font-settings .font-family-list li").removeClass("active"); 67 | $(".font-settings .font-family-list li:nth-child("+(fontState.family+1)+")").addClass("active"); 68 | 69 | $book[0].className = $book[0].className.replace(/\bfont-\S+/g, ''); 70 | $book.addClass("font-size-"+fontState.size); 71 | $book.addClass("font-family-"+fontState.family); 72 | 73 | if(fontState.theme !== 0) { 74 | $book[0].className = $book[0].className.replace(/\bcolor-theme-\S+/g, ''); 75 | $book.addClass("color-theme-"+fontState.theme); 76 | } 77 | }; 78 | 79 | function init(config) { 80 | var $bookBody, $book; 81 | 82 | //Find DOM elements. 83 | $book = gitbook.state.$book; 84 | $bookBody = $book.find(".book-body"); 85 | 86 | // Instantiate font state object 87 | fontState = gitbook.storage.get("fontState", { 88 | size: config.size || 2, 89 | family: FAMILY[config.family || "sans"], 90 | theme: THEMES[config.theme || "white"] 91 | }); 92 | 93 | update(); 94 | }; 95 | 96 | 97 | gitbook.events.bind("start", function(e, config) { 98 | var opts = config.fontsettings; 99 | if (!opts) return; 100 | 101 | // Create buttons in toolbar 102 | gitbook.toolbar.createButton({ 103 | icon: 'fa fa-font', 104 | label: 'Font Settings', 105 | className: 'font-settings', 106 | dropdown: [ 107 | [ 108 | { 109 | text: 'A', 110 | className: 'font-reduce', 111 | onClick: reduceFontSize 112 | }, 113 | { 114 | text: 'A', 115 | className: 'font-enlarge', 116 | onClick: enlargeFontSize 117 | } 118 | ], 119 | [ 120 | { 121 | text: 'Serif', 122 | onClick: _.partial(changeFontFamily, 0) 123 | }, 124 | { 125 | text: 'Sans', 126 | onClick: _.partial(changeFontFamily, 1) 127 | } 128 | ], 129 | [ 130 | { 131 | text: 'White', 132 | onClick: _.partial(changeColorTheme, 0) 133 | }, 134 | { 135 | text: 'Sepia', 136 | onClick: _.partial(changeColorTheme, 1) 137 | }, 138 | { 139 | text: 'Night', 140 | onClick: _.partial(changeColorTheme, 2) 141 | } 142 | ] 143 | ] 144 | }); 145 | 146 | 147 | // Init current settings 148 | init(opts); 149 | }); 150 | }); 151 | 152 | 153 | -------------------------------------------------------------------------------- /_book/libs/gitbook-2.6.7/js/plugin-search.js: -------------------------------------------------------------------------------- 1 | gitbook.require(["gitbook", "lodash", "jQuery"], function(gitbook, _, $) { 2 | var index = null; 3 | var $searchInput, $searchLabel, $searchForm; 4 | var $highlighted = [], hi, hiOpts = { className: 'search-highlight' }; 5 | var collapse = false, toc_visible = []; 6 | 7 | // Use a specific index 8 | function loadIndex(data) { 9 | // [Yihui] In bookdown, I use a character matrix to store the chapter 10 | // content, and the index is dynamically built on the client side. 11 | // Gitbook prebuilds the index data instead: https://github.com/GitbookIO/plugin-search 12 | // We can certainly do that via R packages V8 and jsonlite, but let's 13 | // see how slow it really is before improving it. On the other hand, 14 | // lunr cannot handle non-English text very well, e.g. the default 15 | // tokenizer cannot deal with Chinese text, so we may want to replace 16 | // lunr with a dumb simple text matching approach. 17 | index = lunr(function () { 18 | this.ref('url'); 19 | this.field('title', { boost: 10 }); 20 | this.field('body'); 21 | }); 22 | data.map(function(item) { 23 | index.add({ 24 | url: item[0], 25 | title: item[1], 26 | body: item[2] 27 | }); 28 | }); 29 | } 30 | 31 | // Fetch the search index 32 | function fetchIndex() { 33 | return $.getJSON(gitbook.state.basePath+"/search_index.json") 34 | .then(loadIndex); // [Yihui] we need to use this object later 35 | } 36 | 37 | // Search for a term and return results 38 | function search(q) { 39 | if (!index) return; 40 | 41 | var results = _.chain(index.search(q)) 42 | .map(function(result) { 43 | var parts = result.ref.split("#"); 44 | return { 45 | path: parts[0], 46 | hash: parts[1] 47 | }; 48 | }) 49 | .value(); 50 | 51 | // [Yihui] Highlight the search keyword on current page 52 | $highlighted = results.length === 0 ? [] : $('.page-inner') 53 | .unhighlight(hiOpts).highlight(q, hiOpts).find('span.search-highlight'); 54 | scrollToHighlighted(0); 55 | 56 | return results; 57 | } 58 | 59 | // [Yihui] Scroll the chapter body to the i-th highlighted string 60 | function scrollToHighlighted(d) { 61 | var n = $highlighted.length; 62 | hi = hi === undefined ? 0 : hi + d; 63 | // navignate to the previous/next page in the search results if reached the top/bottom 64 | var b = hi < 0; 65 | if (d !== 0 && (b || hi >= n)) { 66 | var path = currentPath(), n2 = toc_visible.length; 67 | if (n2 === 0) return; 68 | for (var i = b ? 0 : n2; (b && i < n2) || (!b && i >= 0); i += b ? 1 : -1) { 69 | if (toc_visible.eq(i).data('path') === path) break; 70 | } 71 | i += b ? -1 : 1; 72 | if (i < 0) i = n2 - 1; 73 | if (i >= n2) i = 0; 74 | var lnk = toc_visible.eq(i).find('a[href$=".html"]'); 75 | if (lnk.length) lnk[0].click(); 76 | return; 77 | } 78 | if (n === 0) return; 79 | var $p = $highlighted.eq(hi); 80 | $p[0].scrollIntoView(); 81 | $highlighted.css('background-color', ''); 82 | // an orange background color on the current item and removed later 83 | $p.css('background-color', 'orange'); 84 | setTimeout(function() { 85 | $p.css('background-color', ''); 86 | }, 2000); 87 | } 88 | 89 | function currentPath() { 90 | var href = window.location.pathname; 91 | href = href.substr(href.lastIndexOf('/') + 1); 92 | return href === '' ? 'index.html' : href; 93 | } 94 | 95 | // Create search form 96 | function createForm(value) { 97 | if ($searchForm) $searchForm.remove(); 98 | if ($searchLabel) $searchLabel.remove(); 99 | if ($searchInput) $searchInput.remove(); 100 | 101 | $searchForm = $('
', { 102 | 'class': 'book-search', 103 | 'role': 'search' 104 | }); 105 | 106 | $searchLabel = $('
  • Disciplined Convex Optimization in R
    • 7 | after: | 8 |
  • Published with bookdown
    • 9 | download: ["pdf", "epub"] 10 | includes: 11 | in_header: header.html 12 | -------------------------------------------------------------------------------- /book.bib: -------------------------------------------------------------------------------- 1 | @Book{xie2015, 2 | title = {Dynamic Documents with {R} and knitr}, 3 | author = {Yihui Xie}, 4 | publisher = {Chapman and Hall/CRC}, 5 | address = {Boca Raton, Florida}, 6 | year = {2015}, 7 | edition = {2nd}, 8 | note = {ISBN 978-1498716963}, 9 | url = {http://yihui.name/knitr/}, 10 | } 11 | -------------------------------------------------------------------------------- /cvxr_tutorial.Rproj: -------------------------------------------------------------------------------- 1 | Version: 1.0 2 | 3 | RestoreWorkspace: Default 4 | SaveWorkspace: Default 5 | AlwaysSaveHistory: Default 6 | 7 | EnableCodeIndexing: Yes 8 | UseSpacesForTab: Yes 9 | NumSpacesForTab: 2 10 | Encoding: UTF-8 11 | 12 | RnwWeave: Sweave 13 | LaTeX: pdfLaTeX 14 | 15 | BuildType: Website 16 | -------------------------------------------------------------------------------- /extra/slides/book.bib: -------------------------------------------------------------------------------- 1 | @Book{xie2015, 2 | title = {Dynamic Documents with {R} and knitr}, 3 | author = {Yihui Xie}, 4 | publisher = {Chapman and Hall/CRC}, 5 | address = {Boca Raton, Florida}, 6 | year = {2015}, 7 | edition = {2nd}, 8 | note = {ISBN 978-1498716963}, 9 | url = {http://yihui.name/knitr/}, 10 | } 11 | -------------------------------------------------------------------------------- /extra/slides/figures/02/convexchord.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bnaras/cvxr_tutorial/7b8dde276eb749a91c17511d8829bd3eb7971d5e/extra/slides/figures/02/convexchord.gif -------------------------------------------------------------------------------- /extra/slides/figures/iso1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bnaras/cvxr_tutorial/7b8dde276eb749a91c17511d8829bd3eb7971d5e/extra/slides/figures/iso1.png -------------------------------------------------------------------------------- /extra/slides/figures/iso2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bnaras/cvxr_tutorial/7b8dde276eb749a91c17511d8829bd3eb7971d5e/extra/slides/figures/iso2.png -------------------------------------------------------------------------------- /extra/slides/packages.bib: -------------------------------------------------------------------------------- 1 | @Manual{R-base, 2 | title = {R: A Language and Environment for Statistical Computing}, 3 | author = {{R Core Team}}, 4 | organization = {R Foundation for Statistical Computing}, 5 | address = {Vienna, Austria}, 6 | year = {2019}, 7 | url = {https://www.R-project.org/}, 8 | } 9 | @Manual{R-bookdown, 10 | title = {bookdown: Authoring Books and Technical Documents with R Markdown}, 11 | author = {Yihui Xie}, 12 | year = {2019}, 13 | note = {R package version 0.11}, 14 | url = {https://CRAN.R-project.org/package=bookdown}, 15 | } 16 | @Manual{R-devtools, 17 | title = {devtools: Tools to Make Developing R Packages Easier}, 18 | author = {Hadley Wickham and Jim Hester and Winston Chang}, 19 | year = {2019}, 20 | note = {R package version 2.0.2}, 21 | url = {https://CRAN.R-project.org/package=devtools}, 22 | } 23 | @Manual{R-knitr, 24 | title = {knitr: A General-Purpose Package for Dynamic Report Generation in R}, 25 | author = {Yihui Xie}, 26 | year = {2019}, 27 | note = {R package version 1.23}, 28 | url = {https://CRAN.R-project.org/package=knitr}, 29 | } 30 | @Manual{R-pkgdown, 31 | title = {pkgdown: Make Static HTML Documentation for a Package}, 32 | author = {Hadley Wickham and Jay Hesselberth}, 33 | year = {2018}, 34 | note = {R package version 1.3.0}, 35 | url = {https://CRAN.R-project.org/package=pkgdown}, 36 | } 37 | @Manual{R-rmarkdown, 38 | title = {rmarkdown: Dynamic Documents for R}, 39 | author = {JJ Allaire and Yihui Xie and Jonathan McPherson and Javier Luraschi and Kevin Ushey and Aron Atkins and Hadley Wickham and Joe Cheng and Winston Chang and Richard Iannone}, 40 | year = {2019}, 41 | note = {R package version 1.13}, 42 | url = {https://CRAN.R-project.org/package=rmarkdown}, 43 | } 44 | @Manual{R-usethis, 45 | title = {usethis: Automate Package and Project Setup}, 46 | author = {Hadley Wickham and Jennifer Bryan}, 47 | year = {2019}, 48 | note = {R package version 1.5.0}, 49 | url = {https://CRAN.R-project.org/package=usethis}, 50 | } 51 | 52 | -------------------------------------------------------------------------------- /extra/slides/preamble.tex: -------------------------------------------------------------------------------- 1 | \usepackage{booktabs} 2 | 3 | % some traditional defintions that can be blamed on craig barratt 4 | \newcommand{\BEAS}{\begin{eqnarray*}} 5 | \newcommand{\EEAS}{\end{eqnarray*}} 6 | \newcommand{\BEA}{\begin{eqnarray}} 7 | \newcommand{\EEA}{\end{eqnarray}} 8 | \newcommand{\BEQ}{\begin{equation}} 9 | \newcommand{\EEQ}{\end{equation}} 10 | \newcommand{\BIT}{\begin{itemize}} 11 | \newcommand{\EIT}{\end{itemize}} 12 | \newcommand{\BNUM}{\begin{enumerate}} 13 | \newcommand{\ENUM}{\end{enumerate}} 14 | \newcommand{\BF}{\begin{frame}} 15 | \newcommand{\EF}{\end{frame}} 16 | 17 | \newcommand{\argmin}{\mathop{\rm argmin}} 18 | \newcommand{\sign}{\mathop{\rm sign}} 19 | 20 | % text abbrevs 21 | \newcommand{\cf}{{\it cf.}} 22 | \newcommand{\eg}{{\it e.g.}} 23 | \newcommand{\ie}{{\it i.e.}} 24 | \newcommand{\etc}{{\it etc.}} 25 | 26 | \newcommand{\ones}{\mathbf 1} 27 | 28 | % std math stuff 29 | \newcommand{\reals}{{\mbox{\bf R}}} 30 | \newcommand{\integers}{{\mbox{\bf Z}}} 31 | \newcommand{\complex}{{\mbox{\bf C}}} 32 | \newcommand{\symm}{{\mbox{\bf S}}} % symmetric matrices 33 | 34 | % lin alg stuff 35 | \newcommand{\Span}{\mbox{\textrm{span}}} 36 | \newcommand{\Range}{\mbox{\textrm{range}}} 37 | \newcommand{\nullspace}{{\mathcal N}} 38 | \newcommand{\range}{{\mathcal R}} 39 | \newcommand{\Nullspace}{\mbox{\textrm{nullspace}}} 40 | \newcommand{\Rank}{\mathop{\bf Rank}} 41 | \newcommand{\Card}{\mathop{\bf Card}} 42 | \newcommand{\Tr}{\mathop{\bf Tr}} 43 | \newcommand{\diag}{\mathop{\bf diag}} 44 | \newcommand{\lambdamax}{{\lambda_{\rm max}}} 45 | \newcommand{\lambdamin}{\lambda_{\rm min}} 46 | 47 | % probability stuff 48 | \newcommand{\Expect}{\mathop{\bf E{}}} 49 | \newcommand{\Prob}{\mathop{\bf Prob}} 50 | \newcommand{\var}{\mathop{\bf var}} 51 | \newcommand{\std}{\mathop{\bf std}} 52 | \newcommand{\cov}{\mathop{\bf cov}} 53 | \newcommand{\erf}{\mathop{\bf erf}} 54 | 55 | \usepackage{pgfplots} 56 | \begin{tikzpicture} 57 | %preamble 58 | \begin{axis}[xmin=-1.5, xmax=1.5, samples=101] 59 | \addplot[blue, ultra thick] (x, {x^2}); 60 | \end{axis} 61 | \end{tikzpicture} -------------------------------------------------------------------------------- /figures/02/convexchord.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bnaras/cvxr_tutorial/7b8dde276eb749a91c17511d8829bd3eb7971d5e/figures/02/convexchord.gif -------------------------------------------------------------------------------- /header.html: -------------------------------------------------------------------------------- 1 | 25 | 26 | -------------------------------------------------------------------------------- /index.Rmd: -------------------------------------------------------------------------------- 1 | --- 2 | title: 'Disciplined Convex Optimization in R' 3 | author: "Anqi Fu, Balasubramanian Narasimhan, Stephen Boyd" 4 | date: "`r Sys.Date()`" 5 | bibliography: ["packages.bib", "cvxr.bib"] 6 | biblio-style: "apalike" 7 | link-citations: true 8 | output: 9 | bookdown::html_book: 10 | includes: 11 | in_header: header.html 12 | --- 13 | 14 | # Introduction 15 | 16 | Optimization problems are at the heart of statistical inference and 17 | machine learning, where a scalar 18 | criterion is either maximized (likelihood, for instance) or minimized 19 | (loss) to estimate a parameter of interest. Constraints might be 20 | added to narrow the search space or to ensure the solution has certain 21 | properties. 22 | 23 | The kinds of optimization problems we address in this tutorial have 24 | the following form: 25 | \[ 26 | \begin{array}{lll} \text{minimize} & f_0(x) & \\ 27 | \text{subject to} & f_i(x) \leq 0, & i=1, \ldots, M\\ 28 | & Ax=b & 29 | \end{array} 30 | \] 31 | with variable $x \in {\mathbf R}^n$. 32 | 33 | Further, we ask that 34 | 35 | - the objective and inequality constraints $f_0, \ldots, f_M$ are 36 | convex: for all $x$, $y$, $\theta \in [0,1]$, 37 | \[ 38 | \begin{equation} 39 | f_i(\theta x + (1-\theta) y) \leq \theta f_i(x) + (1-\theta) f_i(y) 40 | (\#eq:convex-function) 41 | \end{equation} 42 | \] 43 | i.e., graphs of $f_i$ curve upward, 44 | 45 | - the equality constraints are linear, i.e., they can be expressed as a 46 | simple matrix equation $Ax = b$. 47 | 48 | Geometrically, a function is convex if the chord or line segment drawn 49 | from any point $(x, f(x))$ to another point $(y, f(y))$ lies above the 50 | graph of $f$, as shown below. 51 | 52 | ```{r, echo = FALSE, fig = TRUE, fig.align = "center", fig.cap="Source: https://www.solver.com"} 53 | knitr::include_graphics("figures/02/convexchord.gif") 54 | ``` 55 | 56 | A function is concave if $-f$ is convex. Every linear function is both 57 | convex and concave. 58 | 59 | Convex optimization problems have many nice properties: 60 | 61 | - The region containing solutions, if any, is convex as it is the 62 | intersection of convex constraint functions. 63 | 64 | - Since the objective is also convex, if we find a local optimal 65 | solution, it is automatically the global optimal solution. 66 | 67 | 68 | 69 | There are a host of applications of this problem in many fields, 70 | including machine learning and statistics. 71 | 72 | ## Convex Problems in Statistics, Machine Learning 73 | 74 | - Maximum likelihood estimation with constraints 75 | - OLS regression, nonnegative least squares, logistic regression 76 | - Ridge, lasso, elastic-net regression 77 | - Isotonic regression 78 | - Huber (robust) regression 79 | - Support vector machine 80 | - Sparse inverse covariance estimation 81 | - Maximum entropy and related problems 82 | 83 | New methods are being invented every year! 84 | 85 | ## Non-convex Problems 86 | 87 | A non-convex optimization problem is any problem where the objective 88 | or any of the constraints are non-convex. Not every problem is convex, 89 | and in fact, the non-convex set is much larger. Non-convex problems are 90 | harder to solve in general. 91 | 92 | However, even when a problem is non-convex, one may be able to find a 93 | convex _relaxation_, an approximation to the original problem, that 94 | can yield useful results. Thus, developing tools to solve convex problems 95 | can go a long way. 96 | 97 | 98 | -------------------------------------------------------------------------------- /packages.bib: -------------------------------------------------------------------------------- 1 | @Manual{R-base, 2 | title = {R: A Language and Environment for Statistical Computing}, 3 | author = {{R Core Team}}, 4 | organization = {R Foundation for Statistical Computing}, 5 | address = {Vienna, Austria}, 6 | year = {2019}, 7 | url = {https://www.R-project.org/}, 8 | } 9 | @Manual{R-bookdown, 10 | title = {bookdown: Authoring Books and Technical Documents with R Markdown}, 11 | author = {Yihui Xie}, 12 | year = {2019}, 13 | note = {R package version 0.11}, 14 | url = {https://CRAN.R-project.org/package=bookdown}, 15 | } 16 | @Manual{R-devtools, 17 | title = {devtools: Tools to Make Developing R Packages Easier}, 18 | author = {Hadley Wickham and Jim Hester and Winston Chang}, 19 | year = {2019}, 20 | note = {R package version 2.0.2}, 21 | url = {https://CRAN.R-project.org/package=devtools}, 22 | } 23 | @Manual{R-knitr, 24 | title = {knitr: A General-Purpose Package for Dynamic Report Generation in R}, 25 | author = {Yihui Xie}, 26 | year = {2019}, 27 | note = {R package version 1.23}, 28 | url = {https://CRAN.R-project.org/package=knitr}, 29 | } 30 | @Manual{R-pkgdown, 31 | title = {pkgdown: Make Static HTML Documentation for a Package}, 32 | author = {Hadley Wickham and Jay Hesselberth}, 33 | year = {2018}, 34 | note = {R package version 1.3.0}, 35 | url = {https://CRAN.R-project.org/package=pkgdown}, 36 | } 37 | @Manual{R-rmarkdown, 38 | title = {rmarkdown: Dynamic Documents for R}, 39 | author = {JJ Allaire and Yihui Xie and Jonathan McPherson and Javier Luraschi and Kevin Ushey and Aron Atkins and Hadley Wickham and Joe Cheng and Winston Chang and Richard Iannone}, 40 | year = {2019}, 41 | note = {R package version 1.13}, 42 | url = {https://CRAN.R-project.org/package=rmarkdown}, 43 | } 44 | @Manual{R-usethis, 45 | title = {usethis: Automate Package and Project Setup}, 46 | author = {Hadley Wickham and Jennifer Bryan}, 47 | year = {2019}, 48 | note = {R package version 1.5.0}, 49 | url = {https://CRAN.R-project.org/package=usethis}, 50 | } 51 | 52 | -------------------------------------------------------------------------------- /preamble.tex: -------------------------------------------------------------------------------- 1 | \usepackage{booktabs} 2 | 3 | % some traditional defintions that can be blamed on craig barratt 4 | \newcommand{\BEAS}{\begin{eqnarray*}} 5 | \newcommand{\EEAS}{\end{eqnarray*}} 6 | \newcommand{\BEA}{\begin{eqnarray}} 7 | \newcommand{\EEA}{\end{eqnarray}} 8 | \newcommand{\BEQ}{\begin{equation}} 9 | \newcommand{\EEQ}{\end{equation}} 10 | \newcommand{\BIT}{\begin{itemize}} 11 | \newcommand{\EIT}{\end{itemize}} 12 | \newcommand{\BNUM}{\begin{enumerate}} 13 | \newcommand{\ENUM}{\end{enumerate}} 14 | \newcommand{\BF}{\begin{frame}} 15 | \newcommand{\EF}{\end{frame}} 16 | 17 | \newcommand{\argmin}{\mathop{\rm argmin}} 18 | \newcommand{\sign}{\mathop{\rm sign}} 19 | 20 | % text abbrevs 21 | \newcommand{\cf}{{\it cf.}} 22 | \newcommand{\eg}{{\it e.g.}} 23 | \newcommand{\ie}{{\it i.e.}} 24 | \newcommand{\etc}{{\it etc.}} 25 | 26 | \newcommand{\ones}{\mathbf 1} 27 | 28 | % std math stuff 29 | \newcommand{\reals}{{\mbox{\bf R}}} 30 | \newcommand{\integers}{{\mbox{\bf Z}}} 31 | \newcommand{\complex}{{\mbox{\bf C}}} 32 | \newcommand{\symm}{{\mbox{\bf S}}} % symmetric matrices 33 | 34 | % lin alg stuff 35 | \newcommand{\Span}{\mbox{\textrm{span}}} 36 | \newcommand{\Range}{\mbox{\textrm{range}}} 37 | \newcommand{\nullspace}{{\mathcal N}} 38 | \newcommand{\range}{{\mathcal R}} 39 | \newcommand{\Nullspace}{\mbox{\textrm{nullspace}}} 40 | \newcommand{\Rank}{\mathop{\bf Rank}} 41 | \newcommand{\Card}{\mathop{\bf Card}} 42 | \newcommand{\Tr}{\mathop{\bf Tr}} 43 | \newcommand{\diag}{\mathop{\bf diag}} 44 | \newcommand{\lambdamax}{{\lambda_{\rm max}}} 45 | \newcommand{\lambdamin}{\lambda_{\rm min}} 46 | 47 | % probability stuff 48 | \newcommand{\Expect}{\mathop{\bf E{}}} 49 | \newcommand{\Prob}{\mathop{\bf Prob}} 50 | \newcommand{\var}{\mathop{\bf var}} 51 | \newcommand{\std}{\mathop{\bf std}} 52 | \newcommand{\cov}{\mathop{\bf cov}} 53 | \newcommand{\erf}{\mathop{\bf erf}} 54 | 55 | \usepackage{pgfplots} 56 | \begin{tikzpicture} 57 | %preamble 58 | \begin{axis}[xmin=-1.5, xmax=1.5, samples=101] 59 | \addplot[blue, ultra thick] (x, {x^2}); 60 | \end{axis} 61 | \end{tikzpicture} -------------------------------------------------------------------------------- /style.css: -------------------------------------------------------------------------------- 1 | p.caption { 2 | color: #777; 3 | margin-top: 10px; 4 | } 5 | p code { 6 | white-space: inherit; 7 | } 8 | pre { 9 | word-break: normal; 10 | word-wrap: normal; 11 | } 12 | pre code { 13 | white-space: inherit; 14 | } 15 | --------------------------------------------------------------------------------