├── .gitignore ├── README.md ├── images ├── calculating-training-error.png ├── regularized-linear-regression-cost-function.png └── regularized-linear-regression-gradient.png ├── linear-regression ├── costFunction.m ├── fmincg.m ├── linRegData.mat ├── main.m └── trainLinearRegression.m ├── logistic-regression ├── costFunction.m ├── costFunctionReg.m ├── ex2.m ├── ex2_reg.m ├── ex2data1.txt ├── ex2data2.txt ├── lib │ ├── jsonlab │ │ ├── AUTHORS.txt │ │ ├── ChangeLog.txt │ │ ├── LICENSE_BSD.txt │ │ ├── README.txt │ │ ├── jsonopt.m │ │ ├── loadjson.m │ │ ├── loadubjson.m │ │ ├── mergestruct.m │ │ ├── savejson.m │ │ ├── saveubjson.m │ │ └── varargin2struct.m │ └── makeValidFieldName.m ├── mapFeature.m ├── plotData.m ├── plotDecisionBoundary.m ├── predict.m └── sigmoid.m ├── neural-networks-learning ├── checkNNGradients.m ├── computeNumericalGradient.m ├── debugInitializeWeights.m ├── displayData.m ├── ex4.m ├── ex4data1.mat ├── ex4weights.mat ├── fmincg.m ├── lib │ ├── jsonlab │ │ ├── AUTHORS.txt │ │ ├── ChangeLog.txt │ │ ├── LICENSE_BSD.txt │ │ ├── README.txt │ │ ├── jsonopt.m │ │ ├── loadjson.m │ │ ├── loadubjson.m │ │ ├── mergestruct.m │ │ ├── savejson.m │ │ ├── saveubjson.m │ │ └── varargin2struct.m │ └── makeValidFieldName.m ├── nnCostFunction.m ├── predict.m ├── randInitializeWeights.m ├── sigmoid.m └── sigmoidGradient.m ├── neural-networks ├── displayData.m ├── ex3.m ├── ex3_nn.m ├── ex3data1.mat ├── ex3weights.mat ├── fmincg.m ├── lib │ ├── jsonlab │ │ ├── AUTHORS.txt │ │ ├── ChangeLog.txt │ │ ├── LICENSE_BSD.txt │ │ ├── README.txt │ │ ├── jsonopt.m │ │ ├── loadjson.m │ │ ├── loadubjson.m │ │ ├── mergestruct.m │ │ ├── savejson.m │ │ ├── saveubjson.m │ │ └── varargin2struct.m │ └── makeValidFieldName.m ├── lrCostFunction.m ├── oneVsAll.m ├── predict.m ├── predictOneVsAll.m └── sigmoid.m ├── regularized-linear-regression ├── ex5.m ├── ex5data1.mat ├── featureNormalize.m ├── fmincg.m ├── learningCurve.m ├── lib │ ├── jsonlab │ │ ├── AUTHORS.txt │ │ ├── ChangeLog.txt │ │ ├── LICENSE_BSD.txt │ │ ├── README.txt │ │ ├── jsonopt.m │ │ ├── loadjson.m │ │ ├── loadubjson.m │ │ ├── mergestruct.m │ │ ├── savejson.m │ │ ├── saveubjson.m │ │ └── varargin2struct.m │ └── makeValidFieldName.m ├── linearRegCostFunction.m ├── plotFit.m ├── polyFeatures.m ├── trainLinearReg.m └── validationCurve.m └── support-vector-machines ├── dataset3Params.m ├── emailFeatures.m ├── emailSample1.txt ├── emailSample2.txt ├── ex6.m ├── ex6_spam.m ├── ex6data1.mat ├── ex6data2.mat ├── ex6data3.mat ├── gaussianKernel.m ├── getVocabList.m ├── lib └── jsonlab │ ├── AUTHORS.txt │ ├── ChangeLog.txt │ ├── LICENSE_BSD.txt │ ├── README.txt │ ├── jsonopt.m │ ├── loadjson.m │ ├── loadubjson.m │ ├── mergestruct.m │ ├── savejson.m │ ├── saveubjson.m │ └── varargin2struct.m ├── linearKernel.m ├── plotData.m ├── porterStemmer.m ├── processEmail.m ├── readFile.m ├── spamSample1.txt ├── spamSample2.txt ├── spamTest.mat ├── spamTrain.mat ├── svmPredict.m ├── svmTrain.m ├── token.mat ├── visualizeBoundary.m ├── visualizeBoundaryLinear.m └── vocab.txt /.gitignore: -------------------------------------------------------------------------------- 1 | .idea 2 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Machine Learning Algorithms 2 | 3 | This is a collection of notes and code for machine learning algorithms. 4 | 5 | Most of these will be Matlab/Octave files. Would like to add some Python/Numpy implementations later. 6 | 7 | 8 | 9 | ## Linear regression 10 | 11 | Regularized linear regression has the following cost function: 12 | 13 | ![regularized linear regression cost function](images/regularized-linear-regression-cost-function.png) 14 | 15 | 16 | Correspondingly, the partial derivative of regularized linear regression's cost for θj is defined as: 17 | 18 | ![Regularized linear regression gradient](images/regularized-linear-regression-gradient.png) 19 | 20 | 21 | To plot the learning curve, we need a training and cross validation set 22 | error for different training set sizes. To obtain different training set sizes, 23 | use different subsets of the original training set X. Specifically, for 24 | a training set size of i, you should use the first i examples (i.e., X(1:i,:) 25 | and y(1:i)). 26 | 27 | You can use the trainLinearRegression() function to find the θ parameters. Note 28 | that the lambda is passed as a parameter to the learningCurve function. 29 | After learning the θ parameters, you should compute the error on the training 30 | and cross validation sets. Recall that the training error for a dataset is 31 | defined as: 32 | 33 | ![Calculating training error](images/calculating-training-error.png) 34 | 35 | In particular, note that the training error does not include the regularization 36 | term. One way to compute the training error is to use your existing 37 | cost function and set λ to 0 only when using it to compute the training error 38 | and cross validation error. When you are computing the training set error, 39 | make sure you compute it on the training subset (i.e., X(1:n,:) and y(1:n)) 40 | (instead of the entire training set). However, for the cross validation error, 41 | you should compute it over the entire cross validation set. 42 | 43 | 44 | ## Logistic Regression 45 | 46 | *coming soon* 47 | 48 | ## Multi-class Classification 49 | 50 | *coming soon* 51 | 52 | ## Neural Networks 53 | 54 | *coming soon* 55 | 56 | ## Neural Network Learning 57 | 58 | *coming soon* 59 | 60 | ## Regularized Linear Regression 61 | 62 | *coming soon* 63 | 64 | ## Support Vector Machines 65 | 66 | *coming soon* -------------------------------------------------------------------------------- /images/calculating-training-error.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jwasham/machine-learning/565cc5ad9837c0fb6f73a666ea517c3e70b1e6ff/images/calculating-training-error.png -------------------------------------------------------------------------------- /images/regularized-linear-regression-cost-function.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jwasham/machine-learning/565cc5ad9837c0fb6f73a666ea517c3e70b1e6ff/images/regularized-linear-regression-cost-function.png -------------------------------------------------------------------------------- /images/regularized-linear-regression-gradient.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jwasham/machine-learning/565cc5ad9837c0fb6f73a666ea517c3e70b1e6ff/images/regularized-linear-regression-gradient.png -------------------------------------------------------------------------------- /linear-regression/costFunction.m: -------------------------------------------------------------------------------- 1 | function [J, grad] = costFunction(X, y, theta, lambda) 2 | 3 | %Compute cost and gradient for regularized linear 4 | %regression with multiple variables 5 | 6 | % [J, grad] = costFunction(X, y, theta, lambda) computes the 7 | % cost of using theta as the parameter for linear regression to fit the 8 | % data points in X and y. Returns the cost in J and the gradient in grad 9 | 10 | % Initialize some useful values 11 | m = length(y); % number of training examples 12 | 13 | % Returns the following variables 14 | J = 0; 15 | grad = zeros(size(theta)); 16 | 17 | % ========================================================================= 18 | 19 | hyp = X * theta; % 12x1 matrix 20 | 21 | % --------- cost function --------- 22 | 23 | % y is 12x1 24 | 25 | sum_sq_error = sum((hyp - y).^2); 26 | 27 | reg = ((lambda/(2*m)) * sum([0; theta(2:end)] .^ 2)); 28 | 29 | J = (1/(2*m) * sum_sq_error) + reg; 30 | 31 | % ------------ gradient ------------ 32 | 33 | sum_error = X' * (hyp - y); 34 | 35 | grad = ((1/m) * sum_error) + ((lambda/m) * [0; theta(2:end)]); 36 | 37 | % ========================================================================= 38 | 39 | grad = grad(:); 40 | 41 | end -------------------------------------------------------------------------------- /linear-regression/fmincg.m: -------------------------------------------------------------------------------- 1 | function [X, fX, i] = fmincg(f, X, options, P1, P2, P3, P4, P5) 2 | 3 | % Minimize a continuous differentialble multivariate function. Starting point 4 | % is given by "X" (D by 1), and the function named in the string "f", must 5 | % return a function value and a vector of partial derivatives. The Polack- 6 | % Ribiere flavour of conjugate gradients is used to compute search directions, 7 | % and a line search using quadratic and cubic polynomial approximations and the 8 | % Wolfe-Powell stopping criteria is used together with the slope ratio method 9 | % for guessing initial step sizes. Additionally a bunch of checks are made to 10 | % make sure that exploration is taking place and that extrapolation will not 11 | % be unboundedly large. The "length" gives the length of the run: if it is 12 | % positive, it gives the maximum number of line searches, if negative its 13 | % absolute gives the maximum allowed number of function evaluations. You can 14 | % (optionally) give "length" a second component, which will indicate the 15 | % reduction in function value to be expected in the first line-search (defaults 16 | % to 1.0). The function returns when either its length is up, or if no further 17 | % progress can be made (ie, we are at a minimum, or so close that due to 18 | % numerical problems, we cannot get any closer). If the function terminates 19 | % within a few iterations, it could be an indication that the function value 20 | % and derivatives are not consistent (ie, there may be a bug in the 21 | % implementation of your "f" function). The function returns the found 22 | % solution "X", a vector of function values "fX" indicating the progress made 23 | % and "i" the number of iterations (line searches or function evaluations, 24 | % depending on the sign of "length") used. 25 | % 26 | % Usage: [X, fX, i] = fmincg(f, X, options, P1, P2, P3, P4, P5) 27 | % 28 | % See also: checkgrad 29 | % 30 | % Copyright (C) 2001 and 2002 by Carl Edward Rasmussen. Date 2002-02-13 31 | % 32 | % 33 | % (C) Copyright 1999, 2000 & 2001, Carl Edward Rasmussen 34 | % 35 | % Permission is granted for anyone to copy, use, or modify these 36 | % programs and accompanying documents for purposes of research or 37 | % education, provided this copyright notice is retained, and note is 38 | % made of any changes that have been made. 39 | % 40 | % These programs and documents are distributed without any warranty, 41 | % express or implied. As the programs were written for research 42 | % purposes only, they have not been tested to the degree that would be 43 | % advisable in any important application. All use of these programs is 44 | % entirely at the user's own risk. 45 | % 46 | % [ml-class] Changes Made: 47 | % 1) Function name and argument specifications 48 | % 2) Output display 49 | % 50 | 51 | % Read options 52 | if exist('options', 'var') && ~isempty(options) && isfield(options, 'MaxIter') 53 | length = options.MaxIter; 54 | else 55 | length = 100; 56 | end 57 | 58 | 59 | RHO = 0.01; % a bunch of constants for line searches 60 | SIG = 0.5; % RHO and SIG are the constants in the Wolfe-Powell conditions 61 | INT = 0.1; % don't reevaluate within 0.1 of the limit of the current bracket 62 | EXT = 3.0; % extrapolate maximum 3 times the current bracket 63 | MAX = 20; % max 20 function evaluations per line search 64 | RATIO = 100; % maximum allowed slope ratio 65 | 66 | argstr = ['feval(f, X']; % compose string used to call function 67 | for i = 1:(nargin - 3) 68 | argstr = [argstr, ',P', int2str(i)]; 69 | end 70 | argstr = [argstr, ')']; 71 | 72 | if max(size(length)) == 2, red=length(2); length=length(1); else red=1; end 73 | S=['Iteration ']; 74 | 75 | i = 0; % zero the run length counter 76 | ls_failed = 0; % no previous line search has failed 77 | fX = []; 78 | [f1 df1] = eval(argstr); % get function value and gradient 79 | i = i + (length<0); % count epochs?! 80 | s = -df1; % search direction is steepest 81 | d1 = -s'*s; % this is the slope 82 | z1 = red/(1-d1); % initial step is red/(|s|+1) 83 | 84 | while i < abs(length) % while not finished 85 | i = i + (length>0); % count iterations?! 86 | 87 | X0 = X; f0 = f1; df0 = df1; % make a copy of current values 88 | X = X + z1*s; % begin line search 89 | [f2 df2] = eval(argstr); 90 | i = i + (length<0); % count epochs?! 91 | d2 = df2'*s; 92 | f3 = f1; d3 = d1; z3 = -z1; % initialize point 3 equal to point 1 93 | if length>0, M = MAX; else M = min(MAX, -length-i); end 94 | success = 0; limit = -1; % initialize quanteties 95 | while 1 96 | while ((f2 > f1+z1*RHO*d1) || (d2 > -SIG*d1)) && (M > 0) 97 | limit = z1; % tighten the bracket 98 | if f2 > f1 99 | z2 = z3 - (0.5*d3*z3*z3)/(d3*z3+f2-f3); % quadratic fit 100 | else 101 | A = 6*(f2-f3)/z3+3*(d2+d3); % cubic fit 102 | B = 3*(f3-f2)-z3*(d3+2*d2); 103 | z2 = (sqrt(B*B-A*d2*z3*z3)-B)/A; % numerical error possible - ok! 104 | end 105 | if isnan(z2) || isinf(z2) 106 | z2 = z3/2; % if we had a numerical problem then bisect 107 | end 108 | z2 = max(min(z2, INT*z3),(1-INT)*z3); % don't accept too close to limits 109 | z1 = z1 + z2; % update the step 110 | X = X + z2*s; 111 | [f2 df2] = eval(argstr); 112 | M = M - 1; i = i + (length<0); % count epochs?! 113 | d2 = df2'*s; 114 | z3 = z3-z2; % z3 is now relative to the location of z2 115 | end 116 | if f2 > f1+z1*RHO*d1 || d2 > -SIG*d1 117 | break; % this is a failure 118 | elseif d2 > SIG*d1 119 | success = 1; break; % success 120 | elseif M == 0 121 | break; % failure 122 | end 123 | A = 6*(f2-f3)/z3+3*(d2+d3); % make cubic extrapolation 124 | B = 3*(f3-f2)-z3*(d3+2*d2); 125 | z2 = -d2*z3*z3/(B+sqrt(B*B-A*d2*z3*z3)); % num. error possible - ok! 126 | if ~isreal(z2) || isnan(z2) || isinf(z2) || z2 < 0 % num prob or wrong sign? 127 | if limit < -0.5 % if we have no upper limit 128 | z2 = z1 * (EXT-1); % the extrapolate the maximum amount 129 | else 130 | z2 = (limit-z1)/2; % otherwise bisect 131 | end 132 | elseif (limit > -0.5) && (z2+z1 > limit) % extraplation beyond max? 133 | z2 = (limit-z1)/2; % bisect 134 | elseif (limit < -0.5) && (z2+z1 > z1*EXT) % extrapolation beyond limit 135 | z2 = z1*(EXT-1.0); % set to extrapolation limit 136 | elseif z2 < -z3*INT 137 | z2 = -z3*INT; 138 | elseif (limit > -0.5) && (z2 < (limit-z1)*(1.0-INT)) % too close to limit? 139 | z2 = (limit-z1)*(1.0-INT); 140 | end 141 | f3 = f2; d3 = d2; z3 = -z2; % set point 3 equal to point 2 142 | z1 = z1 + z2; X = X + z2*s; % update current estimates 143 | [f2 df2] = eval(argstr); 144 | M = M - 1; i = i + (length<0); % count epochs?! 145 | d2 = df2'*s; 146 | end % end of line search 147 | 148 | if success % if line search succeeded 149 | f1 = f2; fX = [fX' f1]'; 150 | fprintf('%s %4i | Cost: %4.6e\r', S, i, f1); 151 | s = (df2'*df2-df1'*df2)/(df1'*df1)*s - df2; % Polack-Ribiere direction 152 | tmp = df1; df1 = df2; df2 = tmp; % swap derivatives 153 | d2 = df1'*s; 154 | if d2 > 0 % new slope must be negative 155 | s = -df1; % otherwise use steepest direction 156 | d2 = -s'*s; 157 | end 158 | z1 = z1 * min(RATIO, d1/(d2-realmin)); % slope ratio but max RATIO 159 | d1 = d2; 160 | ls_failed = 0; % this line search did not fail 161 | else 162 | X = X0; f1 = f0; df1 = df0; % restore point from before failed line search 163 | if ls_failed || i > abs(length) % line search failed twice in a row 164 | break; % or we ran out of time, so we give up 165 | end 166 | tmp = df1; df1 = df2; df2 = tmp; % swap derivatives 167 | s = -df1; % try steepest 168 | d1 = -s'*s; 169 | z1 = 1/(1-d1); 170 | ls_failed = 1; % this line search failed 171 | end 172 | if exist('OCTAVE_VERSION') 173 | fflush(stdout); 174 | end 175 | end 176 | 177 | fprintf('\n'); 178 | -------------------------------------------------------------------------------- /linear-regression/linRegData.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jwasham/machine-learning/565cc5ad9837c0fb6f73a666ea517c3e70b1e6ff/linear-regression/linRegData.mat -------------------------------------------------------------------------------- /linear-regression/main.m: -------------------------------------------------------------------------------- 1 | %% Runs our training data through linear regression to fit the data 2 | 3 | %% Initialization 4 | clear ; close all; clc 5 | 6 | % Load Training Data 7 | fprintf('Loading and Visualizing Data ...\n') 8 | 9 | % Load from linRegData: 10 | % You will have X, y, Xval, yval, Xtest, ytest in your environment 11 | load ('linRegData.mat'); 12 | 13 | % m = Number of examples 14 | m = size(X, 1); 15 | 16 | % Plot training data 17 | plot(X, y, 'rx', 'MarkerSize', 10, 'LineWidth', 1.5); 18 | xlabel('Change in water level (x)'); 19 | ylabel('Water flow (y)'); 20 | 21 | %% =========== Part 2: Regularized Linear Regression Cost ============= 22 | 23 | theta = [1 ; 1]; 24 | J = linearRegCostFunction([ones(m, 1) X], y, theta, 1); 25 | 26 | fprintf(['Cost at theta = [1 ; 1]: %f '... 27 | '\n(this value should be about 303.993192)\n'], J); 28 | 29 | fprintf('Program paused. Press enter to continue.\n'); 30 | pause; 31 | 32 | %% =========== Part 3: Regularized Linear Regression Gradient ============= 33 | 34 | theta = [1 ; 1]; 35 | [J, grad] = costFunction([ones(m, 1) X], y, theta, 1); 36 | 37 | fprintf(['Gradient at theta = [1 ; 1]: [%f; %f] '... 38 | '\n(this value should be about [-15.303016; 598.250744])\n'], ... 39 | grad(1), grad(2)); 40 | 41 | fprintf('Program paused. Press enter to continue.\n'); 42 | pause; 43 | 44 | %% =========== Part 4: Train Linear Regression ============= 45 | 46 | % Train linear regression with lambda = 0 47 | lambda = 0; 48 | [theta] = trainLinearRegression([ones(m, 1) X], y, lambda); 49 | 50 | % Plot fit over the data 51 | plot(X, y, 'rx', 'MarkerSize', 10, 'LineWidth', 1.5); 52 | xlabel('Change in water level (x)'); 53 | ylabel('Water flow (y)'); 54 | hold on; 55 | plot(X, [ones(m, 1) X]*theta, '--', 'LineWidth', 2) 56 | hold off; -------------------------------------------------------------------------------- /linear-regression/trainLinearRegression.m: -------------------------------------------------------------------------------- 1 | function [theta] = trainLinearReg(X, y, lambda) 2 | 3 | % Trains linear regression given a dataset (X, y) and a 4 | % regularization parameter lambda 5 | % [theta] = trainLinearReg(X, y, lambda) trains linear regression using 6 | % the dataset (X, y) and regularization parameter lambda. Returns the 7 | % trained parameters theta. 8 | 9 | % Initialize Theta 10 | initial_theta = zeros(size(X, 2), 1); 11 | 12 | % Create "short hand" for the cost function to be minimized 13 | costFunc = @(t) costFunction(X, y, t, lambda); 14 | 15 | % Now, costFunc is a function that takes in only one argument 16 | options = optimset('MaxIter', 200, 'GradObj', 'on'); 17 | 18 | % Minimize using fmincg 19 | theta = fmincg(costFunc, initial_theta, options); 20 | 21 | end -------------------------------------------------------------------------------- /logistic-regression/costFunction.m: -------------------------------------------------------------------------------- 1 | function [J, grad] = costFunction(theta, X, y) 2 | %COSTFUNCTION Compute cost and gradient for logistic regression 3 | % J = COSTFUNCTION(theta, X, y) computes the cost of using theta as the 4 | % parameter for logistic regression and the gradient of the cost 5 | % w.r.t. to the parameters. 6 | 7 | % Initialize some useful values 8 | m = length(y); % number of training examples 9 | 10 | % You need to return the following variables correctly 11 | J = 0; 12 | grad = zeros(size(theta)); 13 | 14 | % ====================== YOUR CODE HERE ====================== 15 | % Instructions: Compute the cost of a particular choice of theta. 16 | % You should set J to the cost. 17 | % Compute the partial derivatives and set grad to the partial 18 | % derivatives of the cost w.r.t. each parameter in theta 19 | % 20 | % Note: grad should have the same dimensions as theta 21 | % 22 | 23 | 24 | hyp = sigmoid(X * theta); 25 | 26 | J = -1 * 1/m * sum( y .* log(hyp) + ((1-y) .* log(1 - hyp)) ); 27 | 28 | %for j = 1:m 29 | % j_temp = sigmoid(X(j) * theta(j)); 30 | % grad(j) = grad(j) + 1/m * sum((j_temp - y(j)) * X(:, j)); 31 | %endfor 32 | 33 | grad = 1/m * X' * (sigmoid(X * theta) - y); 34 | 35 | %for i = 1:m, 36 | % grad = grad + (y(i) - sigmoid(theta'*X(:,i)))* X(:,i); 37 | %end; 38 | 39 | % ---- Inaccurate below ---- 40 | 41 | %sumError = 0; 42 | 43 | %for i = 1:m 44 | % % hyp = sigmoid(X(i) * theta); 45 | % hyp = sigmoid(theta' * X(i)); 46 | % sumError = sumError + ( (y(i) * log(hyp)) - ( (1 - y(i)) * log(1 - hyp)) ); 47 | %endfor 48 | 49 | %J = -1 * 1/m * sumError; 50 | 51 | % ============================================================= 52 | 53 | end 54 | -------------------------------------------------------------------------------- /logistic-regression/costFunctionReg.m: -------------------------------------------------------------------------------- 1 | function [J, grad] = costFunctionReg(theta, X, y, lambda) 2 | %COSTFUNCTIONREG Compute cost and gradient for logistic regression with regularization 3 | % J = COSTFUNCTIONREG(theta, X, y, lambda) computes the cost of using 4 | % theta as the parameter for regularized logistic regression and the 5 | % gradient of the cost w.r.t. to the parameters. 6 | 7 | % Initialize some useful values 8 | m = length(y); % number of training examples 9 | 10 | % You need to return the following variables correctly 11 | J = 0; 12 | grad = zeros(size(theta)); 13 | 14 | % ====================== YOUR CODE HERE ====================== 15 | % Instructions: Compute the cost of a particular choice of theta. 16 | % You should set J to the cost. 17 | % Compute the partial derivatives and set grad to the partial 18 | % derivatives of the cost w.r.t. each parameter in theta 19 | 20 | hyp = sigmoid(X * theta); 21 | 22 | theta_no_1 = theta(2:size(theta)); 23 | theta_empty_0 = [0; theta_no_1]; 24 | 25 | J = (-1 * 1/m * sum( y .* log(hyp) + ((1-y) .* log(1 - hyp)) )) + lambda/(2*m) * theta_empty_0' * theta_empty_0; 26 | 27 | grad = (1/m * X' * (sigmoid(X * theta) - y)) + ((lambda/m) * theta_empty_0); 28 | 29 | % ============================================================= 30 | 31 | end 32 | -------------------------------------------------------------------------------- /logistic-regression/ex2.m: -------------------------------------------------------------------------------- 1 | %% Machine Learning Online Class - Exercise 2: Logistic Regression 2 | % 3 | % Instructions 4 | % ------------ 5 | % 6 | % This file contains code that helps you get started on the logistic 7 | % regression exercise. You will need to complete the following functions 8 | % in this exericse: 9 | % 10 | % sigmoid.m 11 | % costFunction.m 12 | % predict.m 13 | % costFunctionReg.m 14 | % 15 | % For this exercise, you will not need to change any code in this file, 16 | % or any other files other than those mentioned above. 17 | % 18 | 19 | %% Initialization 20 | clear ; close all; clc 21 | 22 | %% Load Data 23 | % The first two columns contains the exam scores and the third column 24 | % contains the label. 25 | 26 | data = load('ex2data1.txt'); 27 | X = data(:, [1, 2]); y = data(:, 3); 28 | 29 | %% ==================== Part 1: Plotting ==================== 30 | % We start the exercise by first plotting the data to understand the 31 | % the problem we are working with. 32 | 33 | fprintf(['Plotting data with + indicating (y = 1) examples and o ' ... 34 | 'indicating (y = 0) examples.\n']); 35 | 36 | plotData(X, y); 37 | 38 | % Put some labels 39 | hold on; 40 | % Labels and Legend 41 | xlabel('Exam 1 score') 42 | ylabel('Exam 2 score') 43 | 44 | % Specified in plot order 45 | legend('Admitted', 'Not admitted') 46 | hold off; 47 | 48 | fprintf('\nProgram paused. Press enter to continue.\n'); 49 | pause; 50 | 51 | 52 | %% ============ Part 2: Compute Cost and Gradient ============ 53 | % In this part of the exercise, you will implement the cost and gradient 54 | % for logistic regression. You neeed to complete the code in 55 | % costFunction.m 56 | 57 | % Setup the data matrix appropriately, and add ones for the intercept term 58 | [m, n] = size(X); 59 | 60 | % Add intercept term to x and X_test 61 | X = [ones(m, 1) X]; 62 | 63 | % Initialize fitting parameters 64 | initial_theta = zeros(n + 1, 1); 65 | 66 | % Compute and display initial cost and gradient 67 | [cost, grad] = costFunction(initial_theta, X, y); 68 | 69 | fprintf('Cost at initial theta (zeros): %f\n', cost); 70 | fprintf('Gradient at initial theta (zeros): \n'); 71 | fprintf(' %f \n', grad); 72 | 73 | fprintf('\nProgram paused. Press enter to continue.\n'); 74 | pause; 75 | 76 | 77 | %% ============= Part 3: Optimizing using fminunc ============= 78 | % In this exercise, you will use a built-in function (fminunc) to find the 79 | % optimal parameters theta. 80 | 81 | % Set options for fminunc 82 | options = optimset('GradObj', 'on', 'MaxIter', 400); 83 | 84 | % Run fminunc to obtain the optimal theta 85 | % This function will return theta and the cost 86 | [theta, cost] = ... 87 | fminunc(@(t)(costFunction(t, X, y)), initial_theta, options); 88 | 89 | % Print theta to screen 90 | fprintf('Cost at theta found by fminunc: %f\n', cost); 91 | fprintf('theta: \n'); 92 | fprintf(' %f \n', theta); 93 | 94 | % Plot Boundary 95 | plotDecisionBoundary(theta, X, y); 96 | 97 | % Put some labels 98 | hold on; 99 | % Labels and Legend 100 | xlabel('Exam 1 score') 101 | ylabel('Exam 2 score') 102 | 103 | % Specified in plot order 104 | legend('Admitted', 'Not admitted') 105 | hold off; 106 | 107 | fprintf('\nProgram paused. Press enter to continue.\n'); 108 | pause; 109 | 110 | %% ============== Part 4: Predict and Accuracies ============== 111 | % After learning the parameters, you'll like to use it to predict the outcomes 112 | % on unseen data. In this part, you will use the logistic regression model 113 | % to predict the probability that a student with score 45 on exam 1 and 114 | % score 85 on exam 2 will be admitted. 115 | % 116 | % Furthermore, you will compute the training and test set accuracies of 117 | % our model. 118 | % 119 | % Your task is to complete the code in predict.m 120 | 121 | % Predict probability for a student with score 45 on exam 1 122 | % and score 85 on exam 2 123 | 124 | prob = sigmoid([1 45 85] * theta); 125 | fprintf(['For a student with scores 45 and 85, we predict an admission ' ... 126 | 'probability of %f\n\n'], prob); 127 | 128 | % Compute accuracy on our training set 129 | p = predict(theta, X); 130 | 131 | fprintf('Train Accuracy: %f\n', mean(double(p == y)) * 100); 132 | 133 | fprintf('\nProgram paused. Press enter to continue.\n'); 134 | pause; 135 | 136 | -------------------------------------------------------------------------------- /logistic-regression/ex2_reg.m: -------------------------------------------------------------------------------- 1 | %% Machine Learning Online Class - Exercise 2: Logistic Regression 2 | % 3 | % Instructions 4 | % ------------ 5 | % 6 | % This file contains code that helps you get started on the second part 7 | % of the exercise which covers regularization with logistic regression. 8 | % 9 | % You will need to complete the following functions in this exericse: 10 | % 11 | % sigmoid.m 12 | % costFunction.m 13 | % predict.m 14 | % costFunctionReg.m 15 | % 16 | % For this exercise, you will not need to change any code in this file, 17 | % or any other files other than those mentioned above. 18 | % 19 | 20 | %% Initialization 21 | clear ; close all; clc 22 | 23 | %% Load Data 24 | % The first two columns contains the X values and the third column 25 | % contains the label (y). 26 | 27 | data = load('ex2data2.txt'); 28 | X = data(:, [1, 2]); y = data(:, 3); 29 | 30 | plotData(X, y); 31 | 32 | % Put some labels 33 | hold on; 34 | 35 | % Labels and Legend 36 | xlabel('Microchip Test 1') 37 | ylabel('Microchip Test 2') 38 | 39 | % Specified in plot order 40 | legend('y = 1', 'y = 0') 41 | hold off; 42 | 43 | 44 | %% =========== Part 1: Regularized Logistic Regression ============ 45 | % In this part, you are given a dataset with data points that are not 46 | % linearly separable. However, you would still like to use logistic 47 | % regression to classify the data points. 48 | % 49 | % To do so, you introduce more features to use -- in particular, you add 50 | % polynomial features to our data matrix (similar to polynomial 51 | % regression). 52 | % 53 | 54 | % Add Polynomial Features 55 | 56 | % Note that mapFeature also adds a column of ones for us, so the intercept 57 | % term is handled 58 | X = mapFeature(X(:,1), X(:,2)); 59 | 60 | % Initialize fitting parameters 61 | initial_theta = zeros(size(X, 2), 1); 62 | 63 | % Set regularization parameter lambda to 1 64 | lambda = 1; 65 | 66 | % Compute and display initial cost and gradient for regularized logistic 67 | % regression 68 | [cost, grad] = costFunctionReg(initial_theta, X, y, lambda); 69 | 70 | fprintf('Cost at initial theta (zeros): %f\n', cost); 71 | 72 | fprintf('\nProgram paused. Press enter to continue.\n'); 73 | pause; 74 | 75 | %% ============= Part 2: Regularization and Accuracies ============= 76 | % Optional Exercise: 77 | % In this part, you will get to try different values of lambda and 78 | % see how regularization affects the decision coundart 79 | % 80 | % Try the following values of lambda (0, 1, 10, 100). 81 | % 82 | % How does the decision boundary change when you vary lambda? How does 83 | % the training set accuracy vary? 84 | % 85 | 86 | % Initialize fitting parameters 87 | initial_theta = zeros(size(X, 2), 1); 88 | 89 | % Set regularization parameter lambda to 1 (you should vary this) 90 | lambda = 1; 91 | 92 | % Set Options 93 | options = optimset('GradObj', 'on', 'MaxIter', 400); 94 | 95 | % Optimize 96 | [theta, J, exit_flag] = ... 97 | fminunc(@(t)(costFunctionReg(t, X, y, lambda)), initial_theta, options); 98 | 99 | % Plot Boundary 100 | plotDecisionBoundary(theta, X, y); 101 | hold on; 102 | title(sprintf('lambda = %g', lambda)) 103 | 104 | % Labels and Legend 105 | xlabel('Microchip Test 1') 106 | ylabel('Microchip Test 2') 107 | 108 | legend('y = 1', 'y = 0', 'Decision boundary') 109 | hold off; 110 | 111 | % Compute accuracy on our training set 112 | p = predict(theta, X); 113 | 114 | fprintf('Train Accuracy: %f\n', mean(double(p == y)) * 100); 115 | 116 | 117 | -------------------------------------------------------------------------------- /logistic-regression/ex2data1.txt: -------------------------------------------------------------------------------- 1 | 34.62365962451697,78.0246928153624,0 2 | 30.28671076822607,43.89499752400101,0 3 | 35.84740876993872,72.90219802708364,0 4 | 60.18259938620976,86.30855209546826,1 5 | 79.0327360507101,75.3443764369103,1 6 | 45.08327747668339,56.3163717815305,0 7 | 61.10666453684766,96.51142588489624,1 8 | 75.02474556738889,46.55401354116538,1 9 | 76.09878670226257,87.42056971926803,1 10 | 84.43281996120035,43.53339331072109,1 11 | 95.86155507093572,38.22527805795094,0 12 | 75.01365838958247,30.60326323428011,0 13 | 82.30705337399482,76.48196330235604,1 14 | 69.36458875970939,97.71869196188608,1 15 | 39.53833914367223,76.03681085115882,0 16 | 53.9710521485623,89.20735013750205,1 17 | 69.07014406283025,52.74046973016765,1 18 | 67.94685547711617,46.67857410673128,0 19 | 70.66150955499435,92.92713789364831,1 20 | 76.97878372747498,47.57596364975532,1 21 | 67.37202754570876,42.83843832029179,0 22 | 89.67677575072079,65.79936592745237,1 23 | 50.534788289883,48.85581152764205,0 24 | 34.21206097786789,44.20952859866288,0 25 | 77.9240914545704,68.9723599933059,1 26 | 62.27101367004632,69.95445795447587,1 27 | 80.1901807509566,44.82162893218353,1 28 | 93.114388797442,38.80067033713209,0 29 | 61.83020602312595,50.25610789244621,0 30 | 38.78580379679423,64.99568095539578,0 31 | 61.379289447425,72.80788731317097,1 32 | 85.40451939411645,57.05198397627122,1 33 | 52.10797973193984,63.12762376881715,0 34 | 52.04540476831827,69.43286012045222,1 35 | 40.23689373545111,71.16774802184875,0 36 | 54.63510555424817,52.21388588061123,0 37 | 33.91550010906887,98.86943574220611,0 38 | 64.17698887494485,80.90806058670817,1 39 | 74.78925295941542,41.57341522824434,0 40 | 34.1836400264419,75.2377203360134,0 41 | 83.90239366249155,56.30804621605327,1 42 | 51.54772026906181,46.85629026349976,0 43 | 94.44336776917852,65.56892160559052,1 44 | 82.36875375713919,40.61825515970618,0 45 | 51.04775177128865,45.82270145776001,0 46 | 62.22267576120188,52.06099194836679,0 47 | 77.19303492601364,70.45820000180959,1 48 | 97.77159928000232,86.7278223300282,1 49 | 62.07306379667647,96.76882412413983,1 50 | 91.56497449807442,88.69629254546599,1 51 | 79.94481794066932,74.16311935043758,1 52 | 99.2725269292572,60.99903099844988,1 53 | 90.54671411399852,43.39060180650027,1 54 | 34.52451385320009,60.39634245837173,0 55 | 50.2864961189907,49.80453881323059,0 56 | 49.58667721632031,59.80895099453265,0 57 | 97.64563396007767,68.86157272420604,1 58 | 32.57720016809309,95.59854761387875,0 59 | 74.24869136721598,69.82457122657193,1 60 | 71.79646205863379,78.45356224515052,1 61 | 75.3956114656803,85.75993667331619,1 62 | 35.28611281526193,47.02051394723416,0 63 | 56.25381749711624,39.26147251058019,0 64 | 30.05882244669796,49.59297386723685,0 65 | 44.66826172480893,66.45008614558913,0 66 | 66.56089447242954,41.09209807936973,0 67 | 40.45755098375164,97.53518548909936,1 68 | 49.07256321908844,51.88321182073966,0 69 | 80.27957401466998,92.11606081344084,1 70 | 66.74671856944039,60.99139402740988,1 71 | 32.72283304060323,43.30717306430063,0 72 | 64.0393204150601,78.03168802018232,1 73 | 72.34649422579923,96.22759296761404,1 74 | 60.45788573918959,73.09499809758037,1 75 | 58.84095621726802,75.85844831279042,1 76 | 99.82785779692128,72.36925193383885,1 77 | 47.26426910848174,88.47586499559782,1 78 | 50.45815980285988,75.80985952982456,1 79 | 60.45555629271532,42.50840943572217,0 80 | 82.22666157785568,42.71987853716458,0 81 | 88.9138964166533,69.80378889835472,1 82 | 94.83450672430196,45.69430680250754,1 83 | 67.31925746917527,66.58935317747915,1 84 | 57.23870631569862,59.51428198012956,1 85 | 80.36675600171273,90.96014789746954,1 86 | 68.46852178591112,85.59430710452014,1 87 | 42.0754545384731,78.84478600148043,0 88 | 75.47770200533905,90.42453899753964,1 89 | 78.63542434898018,96.64742716885644,1 90 | 52.34800398794107,60.76950525602592,0 91 | 94.09433112516793,77.15910509073893,1 92 | 90.44855097096364,87.50879176484702,1 93 | 55.48216114069585,35.57070347228866,0 94 | 74.49269241843041,84.84513684930135,1 95 | 89.84580670720979,45.35828361091658,1 96 | 83.48916274498238,48.38028579728175,1 97 | 42.2617008099817,87.10385094025457,1 98 | 99.31500880510394,68.77540947206617,1 99 | 55.34001756003703,64.9319380069486,1 100 | 74.77589300092767,89.52981289513276,1 101 | -------------------------------------------------------------------------------- /logistic-regression/ex2data2.txt: -------------------------------------------------------------------------------- 1 | 0.051267,0.69956,1 2 | -0.092742,0.68494,1 3 | -0.21371,0.69225,1 4 | -0.375,0.50219,1 5 | -0.51325,0.46564,1 6 | -0.52477,0.2098,1 7 | -0.39804,0.034357,1 8 | -0.30588,-0.19225,1 9 | 0.016705,-0.40424,1 10 | 0.13191,-0.51389,1 11 | 0.38537,-0.56506,1 12 | 0.52938,-0.5212,1 13 | 0.63882,-0.24342,1 14 | 0.73675,-0.18494,1 15 | 0.54666,0.48757,1 16 | 0.322,0.5826,1 17 | 0.16647,0.53874,1 18 | -0.046659,0.81652,1 19 | -0.17339,0.69956,1 20 | -0.47869,0.63377,1 21 | -0.60541,0.59722,1 22 | -0.62846,0.33406,1 23 | -0.59389,0.005117,1 24 | -0.42108,-0.27266,1 25 | -0.11578,-0.39693,1 26 | 0.20104,-0.60161,1 27 | 0.46601,-0.53582,1 28 | 0.67339,-0.53582,1 29 | -0.13882,0.54605,1 30 | -0.29435,0.77997,1 31 | -0.26555,0.96272,1 32 | -0.16187,0.8019,1 33 | -0.17339,0.64839,1 34 | -0.28283,0.47295,1 35 | -0.36348,0.31213,1 36 | -0.30012,0.027047,1 37 | -0.23675,-0.21418,1 38 | -0.06394,-0.18494,1 39 | 0.062788,-0.16301,1 40 | 0.22984,-0.41155,1 41 | 0.2932,-0.2288,1 42 | 0.48329,-0.18494,1 43 | 0.64459,-0.14108,1 44 | 0.46025,0.012427,1 45 | 0.6273,0.15863,1 46 | 0.57546,0.26827,1 47 | 0.72523,0.44371,1 48 | 0.22408,0.52412,1 49 | 0.44297,0.67032,1 50 | 0.322,0.69225,1 51 | 0.13767,0.57529,1 52 | -0.0063364,0.39985,1 53 | -0.092742,0.55336,1 54 | -0.20795,0.35599,1 55 | -0.20795,0.17325,1 56 | -0.43836,0.21711,1 57 | -0.21947,-0.016813,1 58 | -0.13882,-0.27266,1 59 | 0.18376,0.93348,0 60 | 0.22408,0.77997,0 61 | 0.29896,0.61915,0 62 | 0.50634,0.75804,0 63 | 0.61578,0.7288,0 64 | 0.60426,0.59722,0 65 | 0.76555,0.50219,0 66 | 0.92684,0.3633,0 67 | 0.82316,0.27558,0 68 | 0.96141,0.085526,0 69 | 0.93836,0.012427,0 70 | 0.86348,-0.082602,0 71 | 0.89804,-0.20687,0 72 | 0.85196,-0.36769,0 73 | 0.82892,-0.5212,0 74 | 0.79435,-0.55775,0 75 | 0.59274,-0.7405,0 76 | 0.51786,-0.5943,0 77 | 0.46601,-0.41886,0 78 | 0.35081,-0.57968,0 79 | 0.28744,-0.76974,0 80 | 0.085829,-0.75512,0 81 | 0.14919,-0.57968,0 82 | -0.13306,-0.4481,0 83 | -0.40956,-0.41155,0 84 | -0.39228,-0.25804,0 85 | -0.74366,-0.25804,0 86 | -0.69758,0.041667,0 87 | -0.75518,0.2902,0 88 | -0.69758,0.68494,0 89 | -0.4038,0.70687,0 90 | -0.38076,0.91886,0 91 | -0.50749,0.90424,0 92 | -0.54781,0.70687,0 93 | 0.10311,0.77997,0 94 | 0.057028,0.91886,0 95 | -0.10426,0.99196,0 96 | -0.081221,1.1089,0 97 | 0.28744,1.087,0 98 | 0.39689,0.82383,0 99 | 0.63882,0.88962,0 100 | 0.82316,0.66301,0 101 | 0.67339,0.64108,0 102 | 1.0709,0.10015,0 103 | -0.046659,-0.57968,0 104 | -0.23675,-0.63816,0 105 | -0.15035,-0.36769,0 106 | -0.49021,-0.3019,0 107 | -0.46717,-0.13377,0 108 | -0.28859,-0.060673,0 109 | -0.61118,-0.067982,0 110 | -0.66302,-0.21418,0 111 | -0.59965,-0.41886,0 112 | -0.72638,-0.082602,0 113 | -0.83007,0.31213,0 114 | -0.72062,0.53874,0 115 | -0.59389,0.49488,0 116 | -0.48445,0.99927,0 117 | -0.0063364,0.99927,0 118 | 0.63265,-0.030612,0 119 | -------------------------------------------------------------------------------- /logistic-regression/lib/jsonlab/AUTHORS.txt: -------------------------------------------------------------------------------- 1 | The author of "jsonlab" toolbox is Qianqian Fang. Qianqian 2 | is currently an Assistant Professor at Massachusetts General Hospital, 3 | Harvard Medical School. 4 | 5 | Address: Martinos Center for Biomedical Imaging, 6 | Massachusetts General Hospital, 7 | Harvard Medical School 8 | Bldg 149, 13th St, Charlestown, MA 02129, USA 9 | URL: http://nmr.mgh.harvard.edu/~fangq/ 10 | Email: or 11 | 12 | 13 | The script loadjson.m was built upon previous works by 14 | 15 | - Nedialko Krouchev: http://www.mathworks.com/matlabcentral/fileexchange/25713 16 | date: 2009/11/02 17 | - François Glineur: http://www.mathworks.com/matlabcentral/fileexchange/23393 18 | date: 2009/03/22 19 | - Joel Feenstra: http://www.mathworks.com/matlabcentral/fileexchange/20565 20 | date: 2008/07/03 21 | 22 | 23 | This toolbox contains patches submitted by the following contributors: 24 | 25 | - Blake Johnson 26 | part of revision 341 27 | 28 | - Niclas Borlin 29 | various fixes in revision 394, including 30 | - loadjson crashes for all-zero sparse matrix. 31 | - loadjson crashes for empty sparse matrix. 32 | - Non-zero size of 0-by-N and N-by-0 empty matrices is lost after savejson/loadjson. 33 | - loadjson crashes for sparse real column vector. 34 | - loadjson crashes for sparse complex column vector. 35 | - Data is corrupted by savejson for sparse real row vector. 36 | - savejson crashes for sparse complex row vector. 37 | 38 | - Yul Kang 39 | patches for svn revision 415. 40 | - savejson saves an empty cell array as [] instead of null 41 | - loadjson differentiates an empty struct from an empty array 42 | -------------------------------------------------------------------------------- /logistic-regression/lib/jsonlab/ChangeLog.txt: -------------------------------------------------------------------------------- 1 | ============================================================================ 2 | 3 | JSONlab - a toolbox to encode/decode JSON/UBJSON files in MATLAB/Octave 4 | 5 | ---------------------------------------------------------------------------- 6 | 7 | JSONlab ChangeLog (key features marked by *): 8 | 9 | == JSONlab 1.0 (codename: Optimus - Final), FangQ == 10 | 11 | 2015/01/02 polish help info for all major functions, update examples, finalize 1.0 12 | 2014/12/19 fix a bug to strictly respect NoRowBracket in savejson 13 | 14 | == JSONlab 1.0.0-RC2 (codename: Optimus - RC2), FangQ == 15 | 16 | 2014/11/22 show progress bar in loadjson ('ShowProgress') 17 | 2014/11/17 add Compact option in savejson to output compact JSON format ('Compact') 18 | 2014/11/17 add FastArrayParser in loadjson to specify fast parser applicable levels 19 | 2014/09/18 start official github mirror: https://github.com/fangq/jsonlab 20 | 21 | == JSONlab 1.0.0-RC1 (codename: Optimus - RC1), FangQ == 22 | 23 | 2014/09/17 fix several compatibility issues when running on octave versions 3.2-3.8 24 | 2014/09/17 support 2D cell and struct arrays in both savejson and saveubjson 25 | 2014/08/04 escape special characters in a JSON string 26 | 2014/02/16 fix a bug when saving ubjson files 27 | 28 | == JSONlab 0.9.9 (codename: Optimus - beta), FangQ == 29 | 30 | 2014/01/22 use binary read and write in saveubjson and loadubjson 31 | 32 | == JSONlab 0.9.8-1 (codename: Optimus - alpha update 1), FangQ == 33 | 34 | 2013/10/07 better round-trip conservation for empty arrays and structs (patch submitted by Yul Kang) 35 | 36 | == JSONlab 0.9.8 (codename: Optimus - alpha), FangQ == 37 | 2013/08/23 *universal Binary JSON (UBJSON) support, including both saveubjson and loadubjson 38 | 39 | == JSONlab 0.9.1 (codename: Rodimus, update 1), FangQ == 40 | 2012/12/18 *handling of various empty and sparse matrices (fixes submitted by Niclas Borlin) 41 | 42 | == JSONlab 0.9.0 (codename: Rodimus), FangQ == 43 | 44 | 2012/06/17 *new format for an invalid leading char, unpacking hex code in savejson 45 | 2012/06/01 support JSONP in savejson 46 | 2012/05/25 fix the empty cell bug (reported by Cyril Davin) 47 | 2012/04/05 savejson can save to a file (suggested by Patrick Rapin) 48 | 49 | == JSONlab 0.8.1 (codename: Sentiel, Update 1), FangQ == 50 | 51 | 2012/02/28 loadjson quotation mark escape bug, see http://bit.ly/yyk1nS 52 | 2012/01/25 patch to handle root-less objects, contributed by Blake Johnson 53 | 54 | == JSONlab 0.8.0 (codename: Sentiel), FangQ == 55 | 56 | 2012/01/13 *speed up loadjson by 20 fold when parsing large data arrays in matlab 57 | 2012/01/11 remove row bracket if an array has 1 element, suggested by Mykel Kochenderfer 58 | 2011/12/22 *accept sequence of 'param',value input in savejson and loadjson 59 | 2011/11/18 fix struct array bug reported by Mykel Kochenderfer 60 | 61 | == JSONlab 0.5.1 (codename: Nexus Update 1), FangQ == 62 | 63 | 2011/10/21 fix a bug in loadjson, previous code does not use any of the acceleration 64 | 2011/10/20 loadjson supports JSON collections - concatenated JSON objects 65 | 66 | == JSONlab 0.5.0 (codename: Nexus), FangQ == 67 | 68 | 2011/10/16 package and release jsonlab 0.5.0 69 | 2011/10/15 *add json demo and regression test, support cpx numbers, fix double quote bug 70 | 2011/10/11 *speed up readjson dramatically, interpret _Array* tags, show data in root level 71 | 2011/10/10 create jsonlab project, start jsonlab website, add online documentation 72 | 2011/10/07 *speed up savejson by 25x using sprintf instead of mat2str, add options support 73 | 2011/10/06 *savejson works for structs, cells and arrays 74 | 2011/09/09 derive loadjson from JSON parser from MATLAB Central, draft savejson.m 75 | -------------------------------------------------------------------------------- /logistic-regression/lib/jsonlab/LICENSE_BSD.txt: -------------------------------------------------------------------------------- 1 | Copyright 2011-2015 Qianqian Fang . All rights reserved. 2 | 3 | Redistribution and use in source and binary forms, with or without modification, are 4 | permitted provided that the following conditions are met: 5 | 6 | 1. Redistributions of source code must retain the above copyright notice, this list of 7 | conditions and the following disclaimer. 8 | 9 | 2. Redistributions in binary form must reproduce the above copyright notice, this list 10 | of conditions and the following disclaimer in the documentation and/or other materials 11 | provided with the distribution. 12 | 13 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ''AS IS'' AND ANY EXPRESS OR IMPLIED 14 | WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND 15 | FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS 16 | OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 17 | CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 18 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON 19 | ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 20 | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF 21 | ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 22 | 23 | The views and conclusions contained in the software and documentation are those of the 24 | authors and should not be interpreted as representing official policies, either expressed 25 | or implied, of the copyright holders. 26 | -------------------------------------------------------------------------------- /logistic-regression/lib/jsonlab/README.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jwasham/machine-learning/565cc5ad9837c0fb6f73a666ea517c3e70b1e6ff/logistic-regression/lib/jsonlab/README.txt -------------------------------------------------------------------------------- /logistic-regression/lib/jsonlab/jsonopt.m: -------------------------------------------------------------------------------- 1 | function val=jsonopt(key,default,varargin) 2 | % 3 | % val=jsonopt(key,default,optstruct) 4 | % 5 | % setting options based on a struct. The struct can be produced 6 | % by varargin2struct from a list of 'param','value' pairs 7 | % 8 | % authors:Qianqian Fang (fangq nmr.mgh.harvard.edu) 9 | % 10 | % $Id: loadjson.m 371 2012-06-20 12:43:06Z fangq $ 11 | % 12 | % input: 13 | % key: a string with which one look up a value from a struct 14 | % default: if the key does not exist, return default 15 | % optstruct: a struct where each sub-field is a key 16 | % 17 | % output: 18 | % val: if key exists, val=optstruct.key; otherwise val=default 19 | % 20 | % license: 21 | % BSD, see LICENSE_BSD.txt files for details 22 | % 23 | % -- this function is part of jsonlab toolbox (http://iso2mesh.sf.net/cgi-bin/index.cgi?jsonlab) 24 | % 25 | 26 | val=default; 27 | if(nargin<=2) return; end 28 | opt=varargin{1}; 29 | if(isstruct(opt) && isfield(opt,key)) 30 | val=getfield(opt,key); 31 | end 32 | 33 | -------------------------------------------------------------------------------- /logistic-regression/lib/jsonlab/mergestruct.m: -------------------------------------------------------------------------------- 1 | function s=mergestruct(s1,s2) 2 | % 3 | % s=mergestruct(s1,s2) 4 | % 5 | % merge two struct objects into one 6 | % 7 | % authors:Qianqian Fang (fangq nmr.mgh.harvard.edu) 8 | % date: 2012/12/22 9 | % 10 | % input: 11 | % s1,s2: a struct object, s1 and s2 can not be arrays 12 | % 13 | % output: 14 | % s: the merged struct object. fields in s1 and s2 will be combined in s. 15 | % 16 | % license: 17 | % BSD, see LICENSE_BSD.txt files for details 18 | % 19 | % -- this function is part of jsonlab toolbox (http://iso2mesh.sf.net/cgi-bin/index.cgi?jsonlab) 20 | % 21 | 22 | if(~isstruct(s1) || ~isstruct(s2)) 23 | error('input parameters contain non-struct'); 24 | end 25 | if(length(s1)>1 || length(s2)>1) 26 | error('can not merge struct arrays'); 27 | end 28 | fn=fieldnames(s2); 29 | s=s1; 30 | for i=1:length(fn) 31 | s=setfield(s,fn{i},getfield(s2,fn{i})); 32 | end 33 | 34 | -------------------------------------------------------------------------------- /logistic-regression/lib/jsonlab/varargin2struct.m: -------------------------------------------------------------------------------- 1 | function opt=varargin2struct(varargin) 2 | % 3 | % opt=varargin2struct('param1',value1,'param2',value2,...) 4 | % or 5 | % opt=varargin2struct(...,optstruct,...) 6 | % 7 | % convert a series of input parameters into a structure 8 | % 9 | % authors:Qianqian Fang (fangq nmr.mgh.harvard.edu) 10 | % date: 2012/12/22 11 | % 12 | % input: 13 | % 'param', value: the input parameters should be pairs of a string and a value 14 | % optstruct: if a parameter is a struct, the fields will be merged to the output struct 15 | % 16 | % output: 17 | % opt: a struct where opt.param1=value1, opt.param2=value2 ... 18 | % 19 | % license: 20 | % BSD, see LICENSE_BSD.txt files for details 21 | % 22 | % -- this function is part of jsonlab toolbox (http://iso2mesh.sf.net/cgi-bin/index.cgi?jsonlab) 23 | % 24 | 25 | len=length(varargin); 26 | opt=struct; 27 | if(len==0) return; end 28 | i=1; 29 | while(i<=len) 30 | if(isstruct(varargin{i})) 31 | opt=mergestruct(opt,varargin{i}); 32 | elseif(ischar(varargin{i}) && i3 matrix, where the first column is all-ones 10 | 11 | % Plot Data 12 | plotData(X(:,2:3), y); 13 | hold on 14 | 15 | if size(X, 2) <= 3 16 | % Only need 2 points to define a line, so choose two endpoints 17 | plot_x = [min(X(:,2))-2, max(X(:,2))+2]; 18 | 19 | % Calculate the decision boundary line 20 | plot_y = (-1./theta(3)).*(theta(2).*plot_x + theta(1)); 21 | 22 | % Plot, and adjust axes for better viewing 23 | plot(plot_x, plot_y) 24 | 25 | % Legend, specific for the exercise 26 | legend('Admitted', 'Not admitted', 'Decision Boundary') 27 | axis([30, 100, 30, 100]) 28 | else 29 | % Here is the grid range 30 | u = linspace(-1, 1.5, 50); 31 | v = linspace(-1, 1.5, 50); 32 | 33 | z = zeros(length(u), length(v)); 34 | % Evaluate z = theta*x over the grid 35 | for i = 1:length(u) 36 | for j = 1:length(v) 37 | z(i,j) = mapFeature(u(i), v(j))*theta; 38 | end 39 | end 40 | z = z'; % important to transpose z before calling contour 41 | 42 | % Plot z = 0 43 | % Notice you need to specify the range [0, 0] 44 | contour(u, v, z, [0, 0], 'LineWidth', 2) 45 | end 46 | hold off 47 | 48 | end 49 | -------------------------------------------------------------------------------- /logistic-regression/predict.m: -------------------------------------------------------------------------------- 1 | function p = predict(theta, X) 2 | %PREDICT Predict whether the label is 0 or 1 using learned logistic 3 | %regression parameters theta 4 | % p = PREDICT(theta, X) computes the predictions for X using a 5 | % threshold at 0.5 (i.e., if sigmoid(theta'*x) >= 0.5, predict 1) 6 | 7 | m = size(X, 1); % Number of training examples 8 | 9 | % You need to return the following variables correctly 10 | p = zeros(m, 1); 11 | 12 | % ====================== YOUR CODE HERE ====================== 13 | % Instructions: Complete the following code to make predictions using 14 | % your learned logistic regression parameters. 15 | % You should set p to a vector of 0's and 1's 16 | % 17 | 18 | t = size(theta, 1); 19 | 20 | for i = 1:m 21 | result = 0; 22 | for j = 1:t 23 | result = result + theta(j) * X(i,j); 24 | endfor 25 | 26 | result = sigmoid(result); 27 | 28 | if result >= 0.5 29 | p(i) = 1; 30 | else 31 | p(i) = 0; 32 | endif 33 | endfor 34 | 35 | % ---- OR ------ 36 | 37 | % p = sigmoid(X * theta) >= 0.5 38 | 39 | % ========================================================================= 40 | 41 | 42 | end 43 | -------------------------------------------------------------------------------- /logistic-regression/sigmoid.m: -------------------------------------------------------------------------------- 1 | function g = sigmoid(z) 2 | %SIGMOID Compute sigmoid function 3 | % J = SIGMOID(z) computes the sigmoid of z. 4 | 5 | % You need to return the following variables correctly 6 | g = zeros(size(z)); 7 | 8 | % ====================== YOUR CODE HERE ====================== 9 | % Instructions: Compute the sigmoid of each value of z (z can be a matrix, 10 | % vector or scalar). 11 | 12 | %g = ones(size(z))/(1 + e^-z) 13 | 14 | g = 1 ./ (1 + exp(-z)); 15 | 16 | % ============================================================= 17 | 18 | end 19 | -------------------------------------------------------------------------------- /neural-networks-learning/checkNNGradients.m: -------------------------------------------------------------------------------- 1 | function checkNNGradients(lambda) 2 | %CHECKNNGRADIENTS Creates a small neural network to check the 3 | %backpropagation gradients 4 | % CHECKNNGRADIENTS(lambda) Creates a small neural network to check the 5 | % backpropagation gradients, it will output the analytical gradients 6 | % produced by your backprop code and the numerical gradients (computed 7 | % using computeNumericalGradient). These two gradient computations should 8 | % result in very similar values. 9 | % 10 | 11 | if ~exist('lambda', 'var') || isempty(lambda) 12 | lambda = 0; 13 | end 14 | 15 | input_layer_size = 3; 16 | hidden_layer_size = 5; 17 | num_labels = 3; 18 | m = 5; 19 | 20 | % We generate some 'random' test data 21 | Theta1 = debugInitializeWeights(hidden_layer_size, input_layer_size); 22 | Theta2 = debugInitializeWeights(num_labels, hidden_layer_size); 23 | % Reusing debugInitializeWeights to generate X 24 | X = debugInitializeWeights(m, input_layer_size - 1); 25 | y = 1 + mod(1:m, num_labels)'; 26 | 27 | % Unroll parameters 28 | nn_params = [Theta1(:) ; Theta2(:)]; 29 | 30 | % Short hand for cost function 31 | costFunc = @(p) nnCostFunction(p, input_layer_size, hidden_layer_size, ... 32 | num_labels, X, y, lambda); 33 | 34 | [cost, grad] = costFunc(nn_params); 35 | numgrad = computeNumericalGradient(costFunc, nn_params); 36 | 37 | % Visually examine the two gradient computations. The two columns 38 | % you get should be very similar. 39 | disp([numgrad grad]); 40 | fprintf(['The above two columns you get should be very similar.\n' ... 41 | '(Left-Your Numerical Gradient, Right-Analytical Gradient)\n\n']); 42 | 43 | % Evaluate the norm of the difference between two solutions. 44 | % If you have a correct implementation, and assuming you used EPSILON = 0.0001 45 | % in computeNumericalGradient.m, then diff below should be less than 1e-9 46 | diff = norm(numgrad-grad)/norm(numgrad+grad); 47 | 48 | fprintf(['If your backpropagation implementation is correct, then \n' ... 49 | 'the relative difference will be small (less than 1e-9). \n' ... 50 | '\nRelative Difference: %g\n'], diff); 51 | 52 | end 53 | -------------------------------------------------------------------------------- /neural-networks-learning/computeNumericalGradient.m: -------------------------------------------------------------------------------- 1 | function numgrad = computeNumericalGradient(J, theta) 2 | %COMPUTENUMERICALGRADIENT Computes the gradient using "finite differences" 3 | %and gives us a numerical estimate of the gradient. 4 | % numgrad = COMPUTENUMERICALGRADIENT(J, theta) computes the numerical 5 | % gradient of the function J around theta. Calling y = J(theta) should 6 | % return the function value at theta. 7 | 8 | % Notes: The following code implements numerical gradient checking, and 9 | % returns the numerical gradient.It sets numgrad(i) to (a numerical 10 | % approximation of) the partial derivative of J with respect to the 11 | % i-th input argument, evaluated at theta. (i.e., numgrad(i) should 12 | % be the (approximately) the partial derivative of J with respect 13 | % to theta(i).) 14 | % 15 | 16 | numgrad = zeros(size(theta)); 17 | perturb = zeros(size(theta)); 18 | e = 1e-4; 19 | for p = 1:numel(theta) 20 | % Set perturbation vector 21 | perturb(p) = e; 22 | loss1 = J(theta - perturb); 23 | loss2 = J(theta + perturb); 24 | % Compute Numerical Gradient 25 | numgrad(p) = (loss2 - loss1) / (2*e); 26 | perturb(p) = 0; 27 | end 28 | 29 | end 30 | -------------------------------------------------------------------------------- /neural-networks-learning/debugInitializeWeights.m: -------------------------------------------------------------------------------- 1 | function W = debugInitializeWeights(fan_out, fan_in) 2 | %DEBUGINITIALIZEWEIGHTS Initialize the weights of a layer with fan_in 3 | %incoming connections and fan_out outgoing connections using a fixed 4 | %strategy, this will help you later in debugging 5 | % W = DEBUGINITIALIZEWEIGHTS(fan_in, fan_out) initializes the weights 6 | % of a layer with fan_in incoming connections and fan_out outgoing 7 | % connections using a fix set of values 8 | % 9 | % Note that W should be set to a matrix of size(1 + fan_in, fan_out) as 10 | % the first row of W handles the "bias" terms 11 | % 12 | 13 | % Set W to zeros 14 | W = zeros(fan_out, 1 + fan_in); 15 | 16 | % Initialize W using "sin", this ensures that W is always of the same 17 | % values and will be useful for debugging 18 | W = reshape(sin(1:numel(W)), size(W)) / 10; 19 | 20 | % ========================================================================= 21 | 22 | end 23 | -------------------------------------------------------------------------------- /neural-networks-learning/displayData.m: -------------------------------------------------------------------------------- 1 | function [h, display_array] = displayData(X, example_width) 2 | %DISPLAYDATA Display 2D data in a nice grid 3 | % [h, display_array] = DISPLAYDATA(X, example_width) displays 2D data 4 | % stored in X in a nice grid. It returns the figure handle h and the 5 | % displayed array if requested. 6 | 7 | % Set example_width automatically if not passed in 8 | if ~exist('example_width', 'var') || isempty(example_width) 9 | example_width = round(sqrt(size(X, 2))); 10 | end 11 | 12 | % Gray Image 13 | colormap(gray); 14 | 15 | % Compute rows, cols 16 | [m n] = size(X); 17 | example_height = (n / example_width); 18 | 19 | % Compute number of items to display 20 | display_rows = floor(sqrt(m)); 21 | display_cols = ceil(m / display_rows); 22 | 23 | % Between images padding 24 | pad = 1; 25 | 26 | % Setup blank display 27 | display_array = - ones(pad + display_rows * (example_height + pad), ... 28 | pad + display_cols * (example_width + pad)); 29 | 30 | % Copy each example into a patch on the display array 31 | curr_ex = 1; 32 | for j = 1:display_rows 33 | for i = 1:display_cols 34 | if curr_ex > m, 35 | break; 36 | end 37 | % Copy the patch 38 | 39 | % Get the max value of the patch 40 | max_val = max(abs(X(curr_ex, :))); 41 | display_array(pad + (j - 1) * (example_height + pad) + (1:example_height), ... 42 | pad + (i - 1) * (example_width + pad) + (1:example_width)) = ... 43 | reshape(X(curr_ex, :), example_height, example_width) / max_val; 44 | curr_ex = curr_ex + 1; 45 | end 46 | if curr_ex > m, 47 | break; 48 | end 49 | end 50 | 51 | % Display Image 52 | h = imagesc(display_array, [-1 1]); 53 | 54 | % Do not show axis 55 | axis image off 56 | 57 | drawnow; 58 | 59 | end 60 | -------------------------------------------------------------------------------- /neural-networks-learning/ex4.m: -------------------------------------------------------------------------------- 1 | %% Machine Learning Online Class - Exercise 4 Neural Network Learning 2 | 3 | % Instructions 4 | % ------------ 5 | % 6 | % This file contains code that helps you get started on the 7 | % linear exercise. You will need to complete the following functions 8 | % in this exericse: 9 | % 10 | % sigmoidGradient.m 11 | % randInitializeWeights.m 12 | % nnCostFunction.m 13 | % 14 | % For this exercise, you will not need to change any code in this file, 15 | % or any other files other than those mentioned above. 16 | % 17 | 18 | %% Initialization 19 | clear ; close all; clc 20 | 21 | %% Setup the parameters you will use for this exercise 22 | input_layer_size = 400; % 20x20 Input Images of Digits 23 | hidden_layer_size = 25; % 25 hidden units 24 | num_labels = 10; % 10 labels, from 1 to 10 25 | % (note that we have mapped "0" to label 10) 26 | 27 | %% =========== Part 1: Loading and Visualizing Data ============= 28 | % We start the exercise by first loading and visualizing the dataset. 29 | % You will be working with a dataset that contains handwritten digits. 30 | % 31 | 32 | % Load Training Data 33 | fprintf('Loading and Visualizing Data ...\n') 34 | 35 | load('ex4data1.mat'); 36 | m = size(X, 1); 37 | 38 | % Randomly select 100 data points to display 39 | sel = randperm(size(X, 1)); 40 | sel = sel(1:100); 41 | 42 | displayData(X(sel, :)); 43 | 44 | fprintf('Program paused. Press enter to continue.\n'); 45 | pause; 46 | 47 | 48 | %% ================ Part 2: Loading Parameters ================ 49 | % In this part of the exercise, we load some pre-initialized 50 | % neural network parameters. 51 | 52 | fprintf('\nLoading Saved Neural Network Parameters ...\n') 53 | 54 | % Load the weights into variables Theta1 and Theta2 55 | load('ex4weights.mat'); 56 | 57 | % Unroll parameters 58 | nn_params = [Theta1(:) ; Theta2(:)]; 59 | 60 | %% ================ Part 3: Compute Cost (Feedforward) ================ 61 | % To the neural network, you should first start by implementing the 62 | % feedforward part of the neural network that returns the cost only. You 63 | % should complete the code in nnCostFunction.m to return cost. After 64 | % implementing the feedforward to compute the cost, you can verify that 65 | % your implementation is correct by verifying that you get the same cost 66 | % as us for the fixed debugging parameters. 67 | % 68 | % We suggest implementing the feedforward cost *without* regularization 69 | % first so that it will be easier for you to debug. Later, in part 4, you 70 | % will get to implement the regularized cost. 71 | % 72 | fprintf('\nFeedforward Using Neural Network ...\n') 73 | 74 | % Weight regularization parameter (we set this to 0 here). 75 | lambda = 0; 76 | 77 | J = nnCostFunction(nn_params, input_layer_size, hidden_layer_size, ... 78 | num_labels, X, y, lambda); 79 | 80 | fprintf(['Cost at parameters (loaded from ex4weights): %f '... 81 | '\n(this value should be about 0.287629)\n'], J); 82 | 83 | fprintf('\nProgram paused. Press enter to continue.\n'); 84 | pause; 85 | 86 | %% =============== Part 4: Implement Regularization =============== 87 | % Once your cost function implementation is correct, you should now 88 | % continue to implement the regularization with the cost. 89 | % 90 | 91 | fprintf('\nChecking Cost Function (w/ Regularization) ... \n') 92 | 93 | % Weight regularization parameter (we set this to 1 here). 94 | lambda = 1; 95 | 96 | J = nnCostFunction(nn_params, input_layer_size, hidden_layer_size, ... 97 | num_labels, X, y, lambda); 98 | 99 | fprintf(['Cost at parameters (loaded from ex4weights): %f '... 100 | '\n(this value should be about 0.383770)\n'], J); 101 | 102 | fprintf('Program paused. Press enter to continue.\n'); 103 | pause; 104 | 105 | 106 | %% ================ Part 5: Sigmoid Gradient ================ 107 | % Before you start implementing the neural network, you will first 108 | % implement the gradient for the sigmoid function. You should complete the 109 | % code in the sigmoidGradient.m file. 110 | % 111 | 112 | fprintf('\nEvaluating sigmoid gradient...\n') 113 | 114 | g = sigmoidGradient([1 -0.5 0 0.5 1]); 115 | fprintf('Sigmoid gradient evaluated at [1 -0.5 0 0.5 1]:\n '); 116 | fprintf('%f ', g); 117 | fprintf('\n\n'); 118 | 119 | fprintf('Program paused. Press enter to continue.\n'); 120 | pause; 121 | 122 | 123 | %% ================ Part 6: Initializing Pameters ================ 124 | % In this part of the exercise, you will be starting to implment a two 125 | % layer neural network that classifies digits. You will start by 126 | % implementing a function to initialize the weights of the neural network 127 | % (randInitializeWeights.m) 128 | 129 | fprintf('\nInitializing Neural Network Parameters ...\n') 130 | 131 | initial_Theta1 = randInitializeWeights(input_layer_size, hidden_layer_size); 132 | initial_Theta2 = randInitializeWeights(hidden_layer_size, num_labels); 133 | 134 | % Unroll parameters 135 | initial_nn_params = [initial_Theta1(:) ; initial_Theta2(:)]; 136 | 137 | 138 | %% =============== Part 7: Implement Backpropagation =============== 139 | % Once your cost matches up with ours, you should proceed to implement the 140 | % backpropagation algorithm for the neural network. You should add to the 141 | % code you've written in nnCostFunction.m to return the partial 142 | % derivatives of the parameters. 143 | % 144 | fprintf('\nChecking Backpropagation... \n'); 145 | 146 | % Check gradients by running checkNNGradients 147 | checkNNGradients; 148 | 149 | fprintf('\nProgram paused. Press enter to continue.\n'); 150 | pause; 151 | 152 | 153 | %% =============== Part 8: Implement Regularization =============== 154 | % Once your backpropagation implementation is correct, you should now 155 | % continue to implement the regularization with the cost and gradient. 156 | % 157 | 158 | fprintf('\nChecking Backpropagation (w/ Regularization) ... \n') 159 | 160 | % Check gradients by running checkNNGradients 161 | lambda = 3; 162 | checkNNGradients(lambda); 163 | 164 | % Also output the costFunction debugging values 165 | debug_J = nnCostFunction(nn_params, input_layer_size, ... 166 | hidden_layer_size, num_labels, X, y, lambda); 167 | 168 | fprintf(['\n\nCost at (fixed) debugging parameters (w/ lambda = 10): %f ' ... 169 | '\n(this value should be about 0.576051)\n\n'], debug_J); 170 | 171 | fprintf('Program paused. Press enter to continue.\n'); 172 | pause; 173 | 174 | 175 | %% =================== Part 8: Training NN =================== 176 | % You have now implemented all the code necessary to train a neural 177 | % network. To train your neural network, we will now use "fmincg", which 178 | % is a function which works similarly to "fminunc". Recall that these 179 | % advanced optimizers are able to train our cost functions efficiently as 180 | % long as we provide them with the gradient computations. 181 | % 182 | fprintf('\nTraining Neural Network... \n') 183 | 184 | % After you have completed the assignment, change the MaxIter to a larger 185 | % value to see how more training helps. 186 | options = optimset('MaxIter', 50); 187 | 188 | % You should also try different values of lambda 189 | lambda = 1; 190 | 191 | % Create "short hand" for the cost function to be minimized 192 | costFunction = @(p) nnCostFunction(p, ... 193 | input_layer_size, ... 194 | hidden_layer_size, ... 195 | num_labels, X, y, lambda); 196 | 197 | % Now, costFunction is a function that takes in only one argument (the 198 | % neural network parameters) 199 | [nn_params, cost] = fmincg(costFunction, initial_nn_params, options); 200 | 201 | % Obtain Theta1 and Theta2 back from nn_params 202 | Theta1 = reshape(nn_params(1:hidden_layer_size * (input_layer_size + 1)), ... 203 | hidden_layer_size, (input_layer_size + 1)); 204 | 205 | Theta2 = reshape(nn_params((1 + (hidden_layer_size * (input_layer_size + 1))):end), ... 206 | num_labels, (hidden_layer_size + 1)); 207 | 208 | fprintf('Program paused. Press enter to continue.\n'); 209 | pause; 210 | 211 | 212 | %% ================= Part 9: Visualize Weights ================= 213 | % You can now "visualize" what the neural network is learning by 214 | % displaying the hidden units to see what features they are capturing in 215 | % the data. 216 | 217 | fprintf('\nVisualizing Neural Network... \n') 218 | 219 | displayData(Theta1(:, 2:end)); 220 | 221 | fprintf('\nProgram paused. Press enter to continue.\n'); 222 | pause; 223 | 224 | %% ================= Part 10: Implement Predict ================= 225 | % After training the neural network, we would like to use it to predict 226 | % the labels. You will now implement the "predict" function to use the 227 | % neural network to predict the labels of the training set. This lets 228 | % you compute the training set accuracy. 229 | 230 | pred = predict(Theta1, Theta2, X); 231 | 232 | fprintf('\nTraining Set Accuracy: %f\n', mean(double(pred == y)) * 100); 233 | 234 | 235 | -------------------------------------------------------------------------------- /neural-networks-learning/ex4data1.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jwasham/machine-learning/565cc5ad9837c0fb6f73a666ea517c3e70b1e6ff/neural-networks-learning/ex4data1.mat -------------------------------------------------------------------------------- /neural-networks-learning/ex4weights.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jwasham/machine-learning/565cc5ad9837c0fb6f73a666ea517c3e70b1e6ff/neural-networks-learning/ex4weights.mat -------------------------------------------------------------------------------- /neural-networks-learning/fmincg.m: -------------------------------------------------------------------------------- 1 | function [X, fX, i] = fmincg(f, X, options, P1, P2, P3, P4, P5) 2 | % Minimize a continuous differentialble multivariate function. Starting point 3 | % is given by "X" (D by 1), and the function named in the string "f", must 4 | % return a function value and a vector of partial derivatives. The Polack- 5 | % Ribiere flavour of conjugate gradients is used to compute search directions, 6 | % and a line search using quadratic and cubic polynomial approximations and the 7 | % Wolfe-Powell stopping criteria is used together with the slope ratio method 8 | % for guessing initial step sizes. Additionally a bunch of checks are made to 9 | % make sure that exploration is taking place and that extrapolation will not 10 | % be unboundedly large. The "length" gives the length of the run: if it is 11 | % positive, it gives the maximum number of line searches, if negative its 12 | % absolute gives the maximum allowed number of function evaluations. You can 13 | % (optionally) give "length" a second component, which will indicate the 14 | % reduction in function value to be expected in the first line-search (defaults 15 | % to 1.0). The function returns when either its length is up, or if no further 16 | % progress can be made (ie, we are at a minimum, or so close that due to 17 | % numerical problems, we cannot get any closer). If the function terminates 18 | % within a few iterations, it could be an indication that the function value 19 | % and derivatives are not consistent (ie, there may be a bug in the 20 | % implementation of your "f" function). The function returns the found 21 | % solution "X", a vector of function values "fX" indicating the progress made 22 | % and "i" the number of iterations (line searches or function evaluations, 23 | % depending on the sign of "length") used. 24 | % 25 | % Usage: [X, fX, i] = fmincg(f, X, options, P1, P2, P3, P4, P5) 26 | % 27 | % See also: checkgrad 28 | % 29 | % Copyright (C) 2001 and 2002 by Carl Edward Rasmussen. Date 2002-02-13 30 | % 31 | % 32 | % (C) Copyright 1999, 2000 & 2001, Carl Edward Rasmussen 33 | % 34 | % Permission is granted for anyone to copy, use, or modify these 35 | % programs and accompanying documents for purposes of research or 36 | % education, provided this copyright notice is retained, and note is 37 | % made of any changes that have been made. 38 | % 39 | % These programs and documents are distributed without any warranty, 40 | % express or implied. As the programs were written for research 41 | % purposes only, they have not been tested to the degree that would be 42 | % advisable in any important application. All use of these programs is 43 | % entirely at the user's own risk. 44 | % 45 | % [ml-class] Changes Made: 46 | % 1) Function name and argument specifications 47 | % 2) Output display 48 | % 49 | 50 | % Read options 51 | if exist('options', 'var') && ~isempty(options) && isfield(options, 'MaxIter') 52 | length = options.MaxIter; 53 | else 54 | length = 100; 55 | end 56 | 57 | 58 | RHO = 0.01; % a bunch of constants for line searches 59 | SIG = 0.5; % RHO and SIG are the constants in the Wolfe-Powell conditions 60 | INT = 0.1; % don't reevaluate within 0.1 of the limit of the current bracket 61 | EXT = 3.0; % extrapolate maximum 3 times the current bracket 62 | MAX = 20; % max 20 function evaluations per line search 63 | RATIO = 100; % maximum allowed slope ratio 64 | 65 | argstr = ['feval(f, X']; % compose string used to call function 66 | for i = 1:(nargin - 3) 67 | argstr = [argstr, ',P', int2str(i)]; 68 | end 69 | argstr = [argstr, ')']; 70 | 71 | if max(size(length)) == 2, red=length(2); length=length(1); else red=1; end 72 | S=['Iteration ']; 73 | 74 | i = 0; % zero the run length counter 75 | ls_failed = 0; % no previous line search has failed 76 | fX = []; 77 | [f1 df1] = eval(argstr); % get function value and gradient 78 | i = i + (length<0); % count epochs?! 79 | s = -df1; % search direction is steepest 80 | d1 = -s'*s; % this is the slope 81 | z1 = red/(1-d1); % initial step is red/(|s|+1) 82 | 83 | while i < abs(length) % while not finished 84 | i = i + (length>0); % count iterations?! 85 | 86 | X0 = X; f0 = f1; df0 = df1; % make a copy of current values 87 | X = X + z1*s; % begin line search 88 | [f2 df2] = eval(argstr); 89 | i = i + (length<0); % count epochs?! 90 | d2 = df2'*s; 91 | f3 = f1; d3 = d1; z3 = -z1; % initialize point 3 equal to point 1 92 | if length>0, M = MAX; else M = min(MAX, -length-i); end 93 | success = 0; limit = -1; % initialize quanteties 94 | while 1 95 | while ((f2 > f1+z1*RHO*d1) || (d2 > -SIG*d1)) && (M > 0) 96 | limit = z1; % tighten the bracket 97 | if f2 > f1 98 | z2 = z3 - (0.5*d3*z3*z3)/(d3*z3+f2-f3); % quadratic fit 99 | else 100 | A = 6*(f2-f3)/z3+3*(d2+d3); % cubic fit 101 | B = 3*(f3-f2)-z3*(d3+2*d2); 102 | z2 = (sqrt(B*B-A*d2*z3*z3)-B)/A; % numerical error possible - ok! 103 | end 104 | if isnan(z2) || isinf(z2) 105 | z2 = z3/2; % if we had a numerical problem then bisect 106 | end 107 | z2 = max(min(z2, INT*z3),(1-INT)*z3); % don't accept too close to limits 108 | z1 = z1 + z2; % update the step 109 | X = X + z2*s; 110 | [f2 df2] = eval(argstr); 111 | M = M - 1; i = i + (length<0); % count epochs?! 112 | d2 = df2'*s; 113 | z3 = z3-z2; % z3 is now relative to the location of z2 114 | end 115 | if f2 > f1+z1*RHO*d1 || d2 > -SIG*d1 116 | break; % this is a failure 117 | elseif d2 > SIG*d1 118 | success = 1; break; % success 119 | elseif M == 0 120 | break; % failure 121 | end 122 | A = 6*(f2-f3)/z3+3*(d2+d3); % make cubic extrapolation 123 | B = 3*(f3-f2)-z3*(d3+2*d2); 124 | z2 = -d2*z3*z3/(B+sqrt(B*B-A*d2*z3*z3)); % num. error possible - ok! 125 | if ~isreal(z2) || isnan(z2) || isinf(z2) || z2 < 0 % num prob or wrong sign? 126 | if limit < -0.5 % if we have no upper limit 127 | z2 = z1 * (EXT-1); % the extrapolate the maximum amount 128 | else 129 | z2 = (limit-z1)/2; % otherwise bisect 130 | end 131 | elseif (limit > -0.5) && (z2+z1 > limit) % extraplation beyond max? 132 | z2 = (limit-z1)/2; % bisect 133 | elseif (limit < -0.5) && (z2+z1 > z1*EXT) % extrapolation beyond limit 134 | z2 = z1*(EXT-1.0); % set to extrapolation limit 135 | elseif z2 < -z3*INT 136 | z2 = -z3*INT; 137 | elseif (limit > -0.5) && (z2 < (limit-z1)*(1.0-INT)) % too close to limit? 138 | z2 = (limit-z1)*(1.0-INT); 139 | end 140 | f3 = f2; d3 = d2; z3 = -z2; % set point 3 equal to point 2 141 | z1 = z1 + z2; X = X + z2*s; % update current estimates 142 | [f2 df2] = eval(argstr); 143 | M = M - 1; i = i + (length<0); % count epochs?! 144 | d2 = df2'*s; 145 | end % end of line search 146 | 147 | if success % if line search succeeded 148 | f1 = f2; fX = [fX' f1]'; 149 | fprintf('%s %4i | Cost: %4.6e\r', S, i, f1); 150 | s = (df2'*df2-df1'*df2)/(df1'*df1)*s - df2; % Polack-Ribiere direction 151 | tmp = df1; df1 = df2; df2 = tmp; % swap derivatives 152 | d2 = df1'*s; 153 | if d2 > 0 % new slope must be negative 154 | s = -df1; % otherwise use steepest direction 155 | d2 = -s'*s; 156 | end 157 | z1 = z1 * min(RATIO, d1/(d2-realmin)); % slope ratio but max RATIO 158 | d1 = d2; 159 | ls_failed = 0; % this line search did not fail 160 | else 161 | X = X0; f1 = f0; df1 = df0; % restore point from before failed line search 162 | if ls_failed || i > abs(length) % line search failed twice in a row 163 | break; % or we ran out of time, so we give up 164 | end 165 | tmp = df1; df1 = df2; df2 = tmp; % swap derivatives 166 | s = -df1; % try steepest 167 | d1 = -s'*s; 168 | z1 = 1/(1-d1); 169 | ls_failed = 1; % this line search failed 170 | end 171 | if exist('OCTAVE_VERSION') 172 | fflush(stdout); 173 | end 174 | end 175 | fprintf('\n'); 176 | -------------------------------------------------------------------------------- /neural-networks-learning/lib/jsonlab/AUTHORS.txt: -------------------------------------------------------------------------------- 1 | The author of "jsonlab" toolbox is Qianqian Fang. Qianqian 2 | is currently an Assistant Professor at Massachusetts General Hospital, 3 | Harvard Medical School. 4 | 5 | Address: Martinos Center for Biomedical Imaging, 6 | Massachusetts General Hospital, 7 | Harvard Medical School 8 | Bldg 149, 13th St, Charlestown, MA 02129, USA 9 | URL: http://nmr.mgh.harvard.edu/~fangq/ 10 | Email: or 11 | 12 | 13 | The script loadjson.m was built upon previous works by 14 | 15 | - Nedialko Krouchev: http://www.mathworks.com/matlabcentral/fileexchange/25713 16 | date: 2009/11/02 17 | - François Glineur: http://www.mathworks.com/matlabcentral/fileexchange/23393 18 | date: 2009/03/22 19 | - Joel Feenstra: http://www.mathworks.com/matlabcentral/fileexchange/20565 20 | date: 2008/07/03 21 | 22 | 23 | This toolbox contains patches submitted by the following contributors: 24 | 25 | - Blake Johnson 26 | part of revision 341 27 | 28 | - Niclas Borlin 29 | various fixes in revision 394, including 30 | - loadjson crashes for all-zero sparse matrix. 31 | - loadjson crashes for empty sparse matrix. 32 | - Non-zero size of 0-by-N and N-by-0 empty matrices is lost after savejson/loadjson. 33 | - loadjson crashes for sparse real column vector. 34 | - loadjson crashes for sparse complex column vector. 35 | - Data is corrupted by savejson for sparse real row vector. 36 | - savejson crashes for sparse complex row vector. 37 | 38 | - Yul Kang 39 | patches for svn revision 415. 40 | - savejson saves an empty cell array as [] instead of null 41 | - loadjson differentiates an empty struct from an empty array 42 | -------------------------------------------------------------------------------- /neural-networks-learning/lib/jsonlab/ChangeLog.txt: -------------------------------------------------------------------------------- 1 | ============================================================================ 2 | 3 | JSONlab - a toolbox to encode/decode JSON/UBJSON files in MATLAB/Octave 4 | 5 | ---------------------------------------------------------------------------- 6 | 7 | JSONlab ChangeLog (key features marked by *): 8 | 9 | == JSONlab 1.0 (codename: Optimus - Final), FangQ == 10 | 11 | 2015/01/02 polish help info for all major functions, update examples, finalize 1.0 12 | 2014/12/19 fix a bug to strictly respect NoRowBracket in savejson 13 | 14 | == JSONlab 1.0.0-RC2 (codename: Optimus - RC2), FangQ == 15 | 16 | 2014/11/22 show progress bar in loadjson ('ShowProgress') 17 | 2014/11/17 add Compact option in savejson to output compact JSON format ('Compact') 18 | 2014/11/17 add FastArrayParser in loadjson to specify fast parser applicable levels 19 | 2014/09/18 start official github mirror: https://github.com/fangq/jsonlab 20 | 21 | == JSONlab 1.0.0-RC1 (codename: Optimus - RC1), FangQ == 22 | 23 | 2014/09/17 fix several compatibility issues when running on octave versions 3.2-3.8 24 | 2014/09/17 support 2D cell and struct arrays in both savejson and saveubjson 25 | 2014/08/04 escape special characters in a JSON string 26 | 2014/02/16 fix a bug when saving ubjson files 27 | 28 | == JSONlab 0.9.9 (codename: Optimus - beta), FangQ == 29 | 30 | 2014/01/22 use binary read and write in saveubjson and loadubjson 31 | 32 | == JSONlab 0.9.8-1 (codename: Optimus - alpha update 1), FangQ == 33 | 34 | 2013/10/07 better round-trip conservation for empty arrays and structs (patch submitted by Yul Kang) 35 | 36 | == JSONlab 0.9.8 (codename: Optimus - alpha), FangQ == 37 | 2013/08/23 *universal Binary JSON (UBJSON) support, including both saveubjson and loadubjson 38 | 39 | == JSONlab 0.9.1 (codename: Rodimus, update 1), FangQ == 40 | 2012/12/18 *handling of various empty and sparse matrices (fixes submitted by Niclas Borlin) 41 | 42 | == JSONlab 0.9.0 (codename: Rodimus), FangQ == 43 | 44 | 2012/06/17 *new format for an invalid leading char, unpacking hex code in savejson 45 | 2012/06/01 support JSONP in savejson 46 | 2012/05/25 fix the empty cell bug (reported by Cyril Davin) 47 | 2012/04/05 savejson can save to a file (suggested by Patrick Rapin) 48 | 49 | == JSONlab 0.8.1 (codename: Sentiel, Update 1), FangQ == 50 | 51 | 2012/02/28 loadjson quotation mark escape bug, see http://bit.ly/yyk1nS 52 | 2012/01/25 patch to handle root-less objects, contributed by Blake Johnson 53 | 54 | == JSONlab 0.8.0 (codename: Sentiel), FangQ == 55 | 56 | 2012/01/13 *speed up loadjson by 20 fold when parsing large data arrays in matlab 57 | 2012/01/11 remove row bracket if an array has 1 element, suggested by Mykel Kochenderfer 58 | 2011/12/22 *accept sequence of 'param',value input in savejson and loadjson 59 | 2011/11/18 fix struct array bug reported by Mykel Kochenderfer 60 | 61 | == JSONlab 0.5.1 (codename: Nexus Update 1), FangQ == 62 | 63 | 2011/10/21 fix a bug in loadjson, previous code does not use any of the acceleration 64 | 2011/10/20 loadjson supports JSON collections - concatenated JSON objects 65 | 66 | == JSONlab 0.5.0 (codename: Nexus), FangQ == 67 | 68 | 2011/10/16 package and release jsonlab 0.5.0 69 | 2011/10/15 *add json demo and regression test, support cpx numbers, fix double quote bug 70 | 2011/10/11 *speed up readjson dramatically, interpret _Array* tags, show data in root level 71 | 2011/10/10 create jsonlab project, start jsonlab website, add online documentation 72 | 2011/10/07 *speed up savejson by 25x using sprintf instead of mat2str, add options support 73 | 2011/10/06 *savejson works for structs, cells and arrays 74 | 2011/09/09 derive loadjson from JSON parser from MATLAB Central, draft savejson.m 75 | -------------------------------------------------------------------------------- /neural-networks-learning/lib/jsonlab/LICENSE_BSD.txt: -------------------------------------------------------------------------------- 1 | Copyright 2011-2015 Qianqian Fang . All rights reserved. 2 | 3 | Redistribution and use in source and binary forms, with or without modification, are 4 | permitted provided that the following conditions are met: 5 | 6 | 1. Redistributions of source code must retain the above copyright notice, this list of 7 | conditions and the following disclaimer. 8 | 9 | 2. Redistributions in binary form must reproduce the above copyright notice, this list 10 | of conditions and the following disclaimer in the documentation and/or other materials 11 | provided with the distribution. 12 | 13 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ''AS IS'' AND ANY EXPRESS OR IMPLIED 14 | WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND 15 | FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS 16 | OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 17 | CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 18 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON 19 | ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 20 | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF 21 | ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 22 | 23 | The views and conclusions contained in the software and documentation are those of the 24 | authors and should not be interpreted as representing official policies, either expressed 25 | or implied, of the copyright holders. 26 | -------------------------------------------------------------------------------- /neural-networks-learning/lib/jsonlab/README.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jwasham/machine-learning/565cc5ad9837c0fb6f73a666ea517c3e70b1e6ff/neural-networks-learning/lib/jsonlab/README.txt -------------------------------------------------------------------------------- /neural-networks-learning/lib/jsonlab/jsonopt.m: -------------------------------------------------------------------------------- 1 | function val=jsonopt(key,default,varargin) 2 | % 3 | % val=jsonopt(key,default,optstruct) 4 | % 5 | % setting options based on a struct. The struct can be produced 6 | % by varargin2struct from a list of 'param','value' pairs 7 | % 8 | % authors:Qianqian Fang (fangq nmr.mgh.harvard.edu) 9 | % 10 | % $Id: loadjson.m 371 2012-06-20 12:43:06Z fangq $ 11 | % 12 | % input: 13 | % key: a string with which one look up a value from a struct 14 | % default: if the key does not exist, return default 15 | % optstruct: a struct where each sub-field is a key 16 | % 17 | % output: 18 | % val: if key exists, val=optstruct.key; otherwise val=default 19 | % 20 | % license: 21 | % BSD, see LICENSE_BSD.txt files for details 22 | % 23 | % -- this function is part of jsonlab toolbox (http://iso2mesh.sf.net/cgi-bin/index.cgi?jsonlab) 24 | % 25 | 26 | val=default; 27 | if(nargin<=2) return; end 28 | opt=varargin{1}; 29 | if(isstruct(opt) && isfield(opt,key)) 30 | val=getfield(opt,key); 31 | end 32 | 33 | -------------------------------------------------------------------------------- /neural-networks-learning/lib/jsonlab/mergestruct.m: -------------------------------------------------------------------------------- 1 | function s=mergestruct(s1,s2) 2 | % 3 | % s=mergestruct(s1,s2) 4 | % 5 | % merge two struct objects into one 6 | % 7 | % authors:Qianqian Fang (fangq nmr.mgh.harvard.edu) 8 | % date: 2012/12/22 9 | % 10 | % input: 11 | % s1,s2: a struct object, s1 and s2 can not be arrays 12 | % 13 | % output: 14 | % s: the merged struct object. fields in s1 and s2 will be combined in s. 15 | % 16 | % license: 17 | % BSD, see LICENSE_BSD.txt files for details 18 | % 19 | % -- this function is part of jsonlab toolbox (http://iso2mesh.sf.net/cgi-bin/index.cgi?jsonlab) 20 | % 21 | 22 | if(~isstruct(s1) || ~isstruct(s2)) 23 | error('input parameters contain non-struct'); 24 | end 25 | if(length(s1)>1 || length(s2)>1) 26 | error('can not merge struct arrays'); 27 | end 28 | fn=fieldnames(s2); 29 | s=s1; 30 | for i=1:length(fn) 31 | s=setfield(s,fn{i},getfield(s2,fn{i})); 32 | end 33 | 34 | -------------------------------------------------------------------------------- /neural-networks-learning/lib/jsonlab/varargin2struct.m: -------------------------------------------------------------------------------- 1 | function opt=varargin2struct(varargin) 2 | % 3 | % opt=varargin2struct('param1',value1,'param2',value2,...) 4 | % or 5 | % opt=varargin2struct(...,optstruct,...) 6 | % 7 | % convert a series of input parameters into a structure 8 | % 9 | % authors:Qianqian Fang (fangq nmr.mgh.harvard.edu) 10 | % date: 2012/12/22 11 | % 12 | % input: 13 | % 'param', value: the input parameters should be pairs of a string and a value 14 | % optstruct: if a parameter is a struct, the fields will be merged to the output struct 15 | % 16 | % output: 17 | % opt: a struct where opt.param1=value1, opt.param2=value2 ... 18 | % 19 | % license: 20 | % BSD, see LICENSE_BSD.txt files for details 21 | % 22 | % -- this function is part of jsonlab toolbox (http://iso2mesh.sf.net/cgi-bin/index.cgi?jsonlab) 23 | % 24 | 25 | len=length(varargin); 26 | opt=struct; 27 | if(len==0) return; end 28 | i=1; 29 | while(i<=len) 30 | if(isstruct(varargin{i})) 31 | opt=mergestruct(opt,varargin{i}); 32 | elseif(ischar(varargin{i}) && i m, 35 | break; 36 | end 37 | % Copy the patch 38 | 39 | % Get the max value of the patch 40 | max_val = max(abs(X(curr_ex, :))); 41 | display_array(pad + (j - 1) * (example_height + pad) + (1:example_height), ... 42 | pad + (i - 1) * (example_width + pad) + (1:example_width)) = ... 43 | reshape(X(curr_ex, :), example_height, example_width) / max_val; 44 | curr_ex = curr_ex + 1; 45 | end 46 | if curr_ex > m, 47 | break; 48 | end 49 | end 50 | 51 | % Display Image 52 | h = imagesc(display_array, [-1 1]); 53 | 54 | % Do not show axis 55 | axis image off 56 | 57 | drawnow; 58 | 59 | end 60 | -------------------------------------------------------------------------------- /neural-networks/ex3.m: -------------------------------------------------------------------------------- 1 | %% Machine Learning Online Class - Exercise 3 | Part 1: One-vs-all 2 | 3 | % Instructions 4 | % ------------ 5 | % 6 | % This file contains code that helps you get started on the 7 | % linear exercise. You will need to complete the following functions 8 | % in this exericse: 9 | % 10 | % lrCostFunction.m (logistic regression cost function) 11 | % oneVsAll.m 12 | % predictOneVsAll.m 13 | % predict.m 14 | % 15 | % For this exercise, you will not need to change any code in this file, 16 | % or any other files other than those mentioned above. 17 | % 18 | 19 | %% Initialization 20 | clear ; close all; clc 21 | 22 | %% Setup the parameters you will use for this part of the exercise 23 | input_layer_size = 400; % 20x20 Input Images of Digits 24 | num_labels = 10; % 10 labels, from 1 to 10 25 | % (note that we have mapped "0" to label 10) 26 | 27 | %% =========== Part 1: Loading and Visualizing Data ============= 28 | % We start the exercise by first loading and visualizing the dataset. 29 | % You will be working with a dataset that contains handwritten digits. 30 | % 31 | 32 | % Load Training Data 33 | fprintf('Loading and Visualizing Data ...\n') 34 | 35 | load('ex3data1.mat'); % training data stored in arrays X, y 36 | m = size(X, 1); 37 | 38 | % Randomly select 100 data points to display 39 | rand_indices = randperm(m); 40 | sel = X(rand_indices(1:100), :); 41 | 42 | displayData(sel); 43 | 44 | fprintf('Program paused. Press enter to continue.\n'); 45 | pause; 46 | 47 | %% ============ Part 2: Vectorize Logistic Regression ============ 48 | % In this part of the exercise, you will reuse your logistic regression 49 | % code from the last exercise. You task here is to make sure that your 50 | % regularized logistic regression implementation is vectorized. After 51 | % that, you will implement one-vs-all classification for the handwritten 52 | % digit dataset. 53 | % 54 | 55 | fprintf('\nTraining One-vs-All Logistic Regression...\n') 56 | 57 | lambda = 0.1; 58 | [all_theta] = oneVsAll(X, y, num_labels, lambda); 59 | 60 | fprintf('Program paused. Press enter to continue.\n'); 61 | pause; 62 | 63 | 64 | %% ================ Part 3: Predict for One-Vs-All ================ 65 | % After ... 66 | pred = predictOneVsAll(all_theta, X); 67 | 68 | fprintf('\nTraining Set Accuracy: %f\n', mean(double(pred == y)) * 100); 69 | 70 | -------------------------------------------------------------------------------- /neural-networks/ex3_nn.m: -------------------------------------------------------------------------------- 1 | %% Machine Learning Online Class - Exercise 3 | Part 2: Neural Networks 2 | 3 | % Instructions 4 | % ------------ 5 | % 6 | % This file contains code that helps you get started on the 7 | % linear exercise. You will need to complete the following functions 8 | % in this exericse: 9 | % 10 | % lrCostFunction.m (logistic regression cost function) 11 | % oneVsAll.m 12 | % predictOneVsAll.m 13 | % predict.m 14 | % 15 | % For this exercise, you will not need to change any code in this file, 16 | % or any other files other than those mentioned above. 17 | % 18 | 19 | %% Initialization 20 | clear ; close all; clc 21 | 22 | %% Setup the parameters you will use for this exercise 23 | input_layer_size = 400; % 20x20 Input Images of Digits 24 | hidden_layer_size = 25; % 25 hidden units 25 | num_labels = 10; % 10 labels, from 1 to 10 26 | % (note that we have mapped "0" to label 10) 27 | 28 | %% =========== Part 1: Loading and Visualizing Data ============= 29 | % We start the exercise by first loading and visualizing the dataset. 30 | % You will be working with a dataset that contains handwritten digits. 31 | % 32 | 33 | % Load Training Data 34 | fprintf('Loading and Visualizing Data ...\n') 35 | 36 | load('ex3data1.mat'); 37 | m = size(X, 1); 38 | 39 | % Randomly select 100 data points to display 40 | sel = randperm(size(X, 1)); 41 | sel = sel(1:100); 42 | 43 | displayData(X(sel, :)); 44 | 45 | fprintf('Program paused. Press enter to continue.\n'); 46 | pause; 47 | 48 | %% ================ Part 2: Loading Pameters ================ 49 | % In this part of the exercise, we load some pre-initialized 50 | % neural network parameters. 51 | 52 | fprintf('\nLoading Saved Neural Network Parameters ...\n') 53 | 54 | % Load the weights into variables Theta1 and Theta2 55 | load('ex3weights.mat'); 56 | 57 | %% ================= Part 3: Implement Predict ================= 58 | % After training the neural network, we would like to use it to predict 59 | % the labels. You will now implement the "predict" function to use the 60 | % neural network to predict the labels of the training set. This lets 61 | % you compute the training set accuracy. 62 | 63 | pred = predict(Theta1, Theta2, X); 64 | 65 | fprintf('\nTraining Set Accuracy: %f\n', mean(double(pred == y)) * 100); 66 | 67 | fprintf('Program paused. Press enter to continue.\n'); 68 | pause; 69 | 70 | % To give you an idea of the network's output, you can also run 71 | % through the examples one at the a time to see what it is predicting. 72 | 73 | % Randomly permute examples 74 | rp = randperm(m); 75 | 76 | for i = 1:m 77 | % Display 78 | fprintf('\nDisplaying Example Image\n'); 79 | displayData(X(rp(i), :)); 80 | 81 | pred = predict(Theta1, Theta2, X(rp(i),:)); 82 | fprintf('\nNeural Network Prediction: %d (digit %d)\n', pred, mod(pred, 10)); 83 | 84 | % Pause 85 | fprintf('Program paused. Press enter to continue.\n'); 86 | pause; 87 | end 88 | 89 | -------------------------------------------------------------------------------- /neural-networks/ex3data1.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jwasham/machine-learning/565cc5ad9837c0fb6f73a666ea517c3e70b1e6ff/neural-networks/ex3data1.mat -------------------------------------------------------------------------------- /neural-networks/ex3weights.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jwasham/machine-learning/565cc5ad9837c0fb6f73a666ea517c3e70b1e6ff/neural-networks/ex3weights.mat -------------------------------------------------------------------------------- /neural-networks/fmincg.m: -------------------------------------------------------------------------------- 1 | function [X, fX, i] = fmincg(f, X, options, P1, P2, P3, P4, P5) 2 | % Minimize a continuous differentialble multivariate function. Starting point 3 | % is given by "X" (D by 1), and the function named in the string "f", must 4 | % return a function value and a vector of partial derivatives. The Polack- 5 | % Ribiere flavour of conjugate gradients is used to compute search directions, 6 | % and a line search using quadratic and cubic polynomial approximations and the 7 | % Wolfe-Powell stopping criteria is used together with the slope ratio method 8 | % for guessing initial step sizes. Additionally a bunch of checks are made to 9 | % make sure that exploration is taking place and that extrapolation will not 10 | % be unboundedly large. The "length" gives the length of the run: if it is 11 | % positive, it gives the maximum number of line searches, if negative its 12 | % absolute gives the maximum allowed number of function evaluations. You can 13 | % (optionally) give "length" a second component, which will indicate the 14 | % reduction in function value to be expected in the first line-search (defaults 15 | % to 1.0). The function returns when either its length is up, or if no further 16 | % progress can be made (ie, we are at a minimum, or so close that due to 17 | % numerical problems, we cannot get any closer). If the function terminates 18 | % within a few iterations, it could be an indication that the function value 19 | % and derivatives are not consistent (ie, there may be a bug in the 20 | % implementation of your "f" function). The function returns the found 21 | % solution "X", a vector of function values "fX" indicating the progress made 22 | % and "i" the number of iterations (line searches or function evaluations, 23 | % depending on the sign of "length") used. 24 | % 25 | % Usage: [X, fX, i] = fmincg(f, X, options, P1, P2, P3, P4, P5) 26 | % 27 | % See also: checkgrad 28 | % 29 | % Copyright (C) 2001 and 2002 by Carl Edward Rasmussen. Date 2002-02-13 30 | % 31 | % 32 | % (C) Copyright 1999, 2000 & 2001, Carl Edward Rasmussen 33 | % 34 | % Permission is granted for anyone to copy, use, or modify these 35 | % programs and accompanying documents for purposes of research or 36 | % education, provided this copyright notice is retained, and note is 37 | % made of any changes that have been made. 38 | % 39 | % These programs and documents are distributed without any warranty, 40 | % express or implied. As the programs were written for research 41 | % purposes only, they have not been tested to the degree that would be 42 | % advisable in any important application. All use of these programs is 43 | % entirely at the user's own risk. 44 | % 45 | % [ml-class] Changes Made: 46 | % 1) Function name and argument specifications 47 | % 2) Output display 48 | % 49 | 50 | % Read options 51 | if exist('options', 'var') && ~isempty(options) && isfield(options, 'MaxIter') 52 | length = options.MaxIter; 53 | else 54 | length = 100; 55 | end 56 | 57 | 58 | RHO = 0.01; % a bunch of constants for line searches 59 | SIG = 0.5; % RHO and SIG are the constants in the Wolfe-Powell conditions 60 | INT = 0.1; % don't reevaluate within 0.1 of the limit of the current bracket 61 | EXT = 3.0; % extrapolate maximum 3 times the current bracket 62 | MAX = 20; % max 20 function evaluations per line search 63 | RATIO = 100; % maximum allowed slope ratio 64 | 65 | argstr = ['feval(f, X']; % compose string used to call function 66 | for i = 1:(nargin - 3) 67 | argstr = [argstr, ',P', int2str(i)]; 68 | end 69 | argstr = [argstr, ')']; 70 | 71 | if max(size(length)) == 2, red=length(2); length=length(1); else red=1; end 72 | S=['Iteration ']; 73 | 74 | i = 0; % zero the run length counter 75 | ls_failed = 0; % no previous line search has failed 76 | fX = []; 77 | [f1 df1] = eval(argstr); % get function value and gradient 78 | i = i + (length<0); % count epochs?! 79 | s = -df1; % search direction is steepest 80 | d1 = -s'*s; % this is the slope 81 | z1 = red/(1-d1); % initial step is red/(|s|+1) 82 | 83 | while i < abs(length) % while not finished 84 | i = i + (length>0); % count iterations?! 85 | 86 | X0 = X; f0 = f1; df0 = df1; % make a copy of current values 87 | X = X + z1*s; % begin line search 88 | [f2 df2] = eval(argstr); 89 | i = i + (length<0); % count epochs?! 90 | d2 = df2'*s; 91 | f3 = f1; d3 = d1; z3 = -z1; % initialize point 3 equal to point 1 92 | if length>0, M = MAX; else M = min(MAX, -length-i); end 93 | success = 0; limit = -1; % initialize quanteties 94 | while 1 95 | while ((f2 > f1+z1*RHO*d1) || (d2 > -SIG*d1)) && (M > 0) 96 | limit = z1; % tighten the bracket 97 | if f2 > f1 98 | z2 = z3 - (0.5*d3*z3*z3)/(d3*z3+f2-f3); % quadratic fit 99 | else 100 | A = 6*(f2-f3)/z3+3*(d2+d3); % cubic fit 101 | B = 3*(f3-f2)-z3*(d3+2*d2); 102 | z2 = (sqrt(B*B-A*d2*z3*z3)-B)/A; % numerical error possible - ok! 103 | end 104 | if isnan(z2) || isinf(z2) 105 | z2 = z3/2; % if we had a numerical problem then bisect 106 | end 107 | z2 = max(min(z2, INT*z3),(1-INT)*z3); % don't accept too close to limits 108 | z1 = z1 + z2; % update the step 109 | X = X + z2*s; 110 | [f2 df2] = eval(argstr); 111 | M = M - 1; i = i + (length<0); % count epochs?! 112 | d2 = df2'*s; 113 | z3 = z3-z2; % z3 is now relative to the location of z2 114 | end 115 | if f2 > f1+z1*RHO*d1 || d2 > -SIG*d1 116 | break; % this is a failure 117 | elseif d2 > SIG*d1 118 | success = 1; break; % success 119 | elseif M == 0 120 | break; % failure 121 | end 122 | A = 6*(f2-f3)/z3+3*(d2+d3); % make cubic extrapolation 123 | B = 3*(f3-f2)-z3*(d3+2*d2); 124 | z2 = -d2*z3*z3/(B+sqrt(B*B-A*d2*z3*z3)); % num. error possible - ok! 125 | if ~isreal(z2) || isnan(z2) || isinf(z2) || z2 < 0 % num prob or wrong sign? 126 | if limit < -0.5 % if we have no upper limit 127 | z2 = z1 * (EXT-1); % the extrapolate the maximum amount 128 | else 129 | z2 = (limit-z1)/2; % otherwise bisect 130 | end 131 | elseif (limit > -0.5) && (z2+z1 > limit) % extraplation beyond max? 132 | z2 = (limit-z1)/2; % bisect 133 | elseif (limit < -0.5) && (z2+z1 > z1*EXT) % extrapolation beyond limit 134 | z2 = z1*(EXT-1.0); % set to extrapolation limit 135 | elseif z2 < -z3*INT 136 | z2 = -z3*INT; 137 | elseif (limit > -0.5) && (z2 < (limit-z1)*(1.0-INT)) % too close to limit? 138 | z2 = (limit-z1)*(1.0-INT); 139 | end 140 | f3 = f2; d3 = d2; z3 = -z2; % set point 3 equal to point 2 141 | z1 = z1 + z2; X = X + z2*s; % update current estimates 142 | [f2 df2] = eval(argstr); 143 | M = M - 1; i = i + (length<0); % count epochs?! 144 | d2 = df2'*s; 145 | end % end of line search 146 | 147 | if success % if line search succeeded 148 | f1 = f2; fX = [fX' f1]'; 149 | fprintf('%s %4i | Cost: %4.6e\r', S, i, f1); 150 | s = (df2'*df2-df1'*df2)/(df1'*df1)*s - df2; % Polack-Ribiere direction 151 | tmp = df1; df1 = df2; df2 = tmp; % swap derivatives 152 | d2 = df1'*s; 153 | if d2 > 0 % new slope must be negative 154 | s = -df1; % otherwise use steepest direction 155 | d2 = -s'*s; 156 | end 157 | z1 = z1 * min(RATIO, d1/(d2-realmin)); % slope ratio but max RATIO 158 | d1 = d2; 159 | ls_failed = 0; % this line search did not fail 160 | else 161 | X = X0; f1 = f0; df1 = df0; % restore point from before failed line search 162 | if ls_failed || i > abs(length) % line search failed twice in a row 163 | break; % or we ran out of time, so we give up 164 | end 165 | tmp = df1; df1 = df2; df2 = tmp; % swap derivatives 166 | s = -df1; % try steepest 167 | d1 = -s'*s; 168 | z1 = 1/(1-d1); 169 | ls_failed = 1; % this line search failed 170 | end 171 | if exist('OCTAVE_VERSION') 172 | fflush(stdout); 173 | end 174 | end 175 | fprintf('\n'); 176 | -------------------------------------------------------------------------------- /neural-networks/lib/jsonlab/AUTHORS.txt: -------------------------------------------------------------------------------- 1 | The author of "jsonlab" toolbox is Qianqian Fang. Qianqian 2 | is currently an Assistant Professor at Massachusetts General Hospital, 3 | Harvard Medical School. 4 | 5 | Address: Martinos Center for Biomedical Imaging, 6 | Massachusetts General Hospital, 7 | Harvard Medical School 8 | Bldg 149, 13th St, Charlestown, MA 02129, USA 9 | URL: http://nmr.mgh.harvard.edu/~fangq/ 10 | Email: or 11 | 12 | 13 | The script loadjson.m was built upon previous works by 14 | 15 | - Nedialko Krouchev: http://www.mathworks.com/matlabcentral/fileexchange/25713 16 | date: 2009/11/02 17 | - François Glineur: http://www.mathworks.com/matlabcentral/fileexchange/23393 18 | date: 2009/03/22 19 | - Joel Feenstra: http://www.mathworks.com/matlabcentral/fileexchange/20565 20 | date: 2008/07/03 21 | 22 | 23 | This toolbox contains patches submitted by the following contributors: 24 | 25 | - Blake Johnson 26 | part of revision 341 27 | 28 | - Niclas Borlin 29 | various fixes in revision 394, including 30 | - loadjson crashes for all-zero sparse matrix. 31 | - loadjson crashes for empty sparse matrix. 32 | - Non-zero size of 0-by-N and N-by-0 empty matrices is lost after savejson/loadjson. 33 | - loadjson crashes for sparse real column vector. 34 | - loadjson crashes for sparse complex column vector. 35 | - Data is corrupted by savejson for sparse real row vector. 36 | - savejson crashes for sparse complex row vector. 37 | 38 | - Yul Kang 39 | patches for svn revision 415. 40 | - savejson saves an empty cell array as [] instead of null 41 | - loadjson differentiates an empty struct from an empty array 42 | -------------------------------------------------------------------------------- /neural-networks/lib/jsonlab/ChangeLog.txt: -------------------------------------------------------------------------------- 1 | ============================================================================ 2 | 3 | JSONlab - a toolbox to encode/decode JSON/UBJSON files in MATLAB/Octave 4 | 5 | ---------------------------------------------------------------------------- 6 | 7 | JSONlab ChangeLog (key features marked by *): 8 | 9 | == JSONlab 1.0 (codename: Optimus - Final), FangQ == 10 | 11 | 2015/01/02 polish help info for all major functions, update examples, finalize 1.0 12 | 2014/12/19 fix a bug to strictly respect NoRowBracket in savejson 13 | 14 | == JSONlab 1.0.0-RC2 (codename: Optimus - RC2), FangQ == 15 | 16 | 2014/11/22 show progress bar in loadjson ('ShowProgress') 17 | 2014/11/17 add Compact option in savejson to output compact JSON format ('Compact') 18 | 2014/11/17 add FastArrayParser in loadjson to specify fast parser applicable levels 19 | 2014/09/18 start official github mirror: https://github.com/fangq/jsonlab 20 | 21 | == JSONlab 1.0.0-RC1 (codename: Optimus - RC1), FangQ == 22 | 23 | 2014/09/17 fix several compatibility issues when running on octave versions 3.2-3.8 24 | 2014/09/17 support 2D cell and struct arrays in both savejson and saveubjson 25 | 2014/08/04 escape special characters in a JSON string 26 | 2014/02/16 fix a bug when saving ubjson files 27 | 28 | == JSONlab 0.9.9 (codename: Optimus - beta), FangQ == 29 | 30 | 2014/01/22 use binary read and write in saveubjson and loadubjson 31 | 32 | == JSONlab 0.9.8-1 (codename: Optimus - alpha update 1), FangQ == 33 | 34 | 2013/10/07 better round-trip conservation for empty arrays and structs (patch submitted by Yul Kang) 35 | 36 | == JSONlab 0.9.8 (codename: Optimus - alpha), FangQ == 37 | 2013/08/23 *universal Binary JSON (UBJSON) support, including both saveubjson and loadubjson 38 | 39 | == JSONlab 0.9.1 (codename: Rodimus, update 1), FangQ == 40 | 2012/12/18 *handling of various empty and sparse matrices (fixes submitted by Niclas Borlin) 41 | 42 | == JSONlab 0.9.0 (codename: Rodimus), FangQ == 43 | 44 | 2012/06/17 *new format for an invalid leading char, unpacking hex code in savejson 45 | 2012/06/01 support JSONP in savejson 46 | 2012/05/25 fix the empty cell bug (reported by Cyril Davin) 47 | 2012/04/05 savejson can save to a file (suggested by Patrick Rapin) 48 | 49 | == JSONlab 0.8.1 (codename: Sentiel, Update 1), FangQ == 50 | 51 | 2012/02/28 loadjson quotation mark escape bug, see http://bit.ly/yyk1nS 52 | 2012/01/25 patch to handle root-less objects, contributed by Blake Johnson 53 | 54 | == JSONlab 0.8.0 (codename: Sentiel), FangQ == 55 | 56 | 2012/01/13 *speed up loadjson by 20 fold when parsing large data arrays in matlab 57 | 2012/01/11 remove row bracket if an array has 1 element, suggested by Mykel Kochenderfer 58 | 2011/12/22 *accept sequence of 'param',value input in savejson and loadjson 59 | 2011/11/18 fix struct array bug reported by Mykel Kochenderfer 60 | 61 | == JSONlab 0.5.1 (codename: Nexus Update 1), FangQ == 62 | 63 | 2011/10/21 fix a bug in loadjson, previous code does not use any of the acceleration 64 | 2011/10/20 loadjson supports JSON collections - concatenated JSON objects 65 | 66 | == JSONlab 0.5.0 (codename: Nexus), FangQ == 67 | 68 | 2011/10/16 package and release jsonlab 0.5.0 69 | 2011/10/15 *add json demo and regression test, support cpx numbers, fix double quote bug 70 | 2011/10/11 *speed up readjson dramatically, interpret _Array* tags, show data in root level 71 | 2011/10/10 create jsonlab project, start jsonlab website, add online documentation 72 | 2011/10/07 *speed up savejson by 25x using sprintf instead of mat2str, add options support 73 | 2011/10/06 *savejson works for structs, cells and arrays 74 | 2011/09/09 derive loadjson from JSON parser from MATLAB Central, draft savejson.m 75 | -------------------------------------------------------------------------------- /neural-networks/lib/jsonlab/LICENSE_BSD.txt: -------------------------------------------------------------------------------- 1 | Copyright 2011-2015 Qianqian Fang . All rights reserved. 2 | 3 | Redistribution and use in source and binary forms, with or without modification, are 4 | permitted provided that the following conditions are met: 5 | 6 | 1. Redistributions of source code must retain the above copyright notice, this list of 7 | conditions and the following disclaimer. 8 | 9 | 2. Redistributions in binary form must reproduce the above copyright notice, this list 10 | of conditions and the following disclaimer in the documentation and/or other materials 11 | provided with the distribution. 12 | 13 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ''AS IS'' AND ANY EXPRESS OR IMPLIED 14 | WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND 15 | FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS 16 | OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 17 | CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 18 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON 19 | ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 20 | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF 21 | ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 22 | 23 | The views and conclusions contained in the software and documentation are those of the 24 | authors and should not be interpreted as representing official policies, either expressed 25 | or implied, of the copyright holders. 26 | -------------------------------------------------------------------------------- /neural-networks/lib/jsonlab/README.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jwasham/machine-learning/565cc5ad9837c0fb6f73a666ea517c3e70b1e6ff/neural-networks/lib/jsonlab/README.txt -------------------------------------------------------------------------------- /neural-networks/lib/jsonlab/jsonopt.m: -------------------------------------------------------------------------------- 1 | function val=jsonopt(key,default,varargin) 2 | % 3 | % val=jsonopt(key,default,optstruct) 4 | % 5 | % setting options based on a struct. The struct can be produced 6 | % by varargin2struct from a list of 'param','value' pairs 7 | % 8 | % authors:Qianqian Fang (fangq nmr.mgh.harvard.edu) 9 | % 10 | % $Id: loadjson.m 371 2012-06-20 12:43:06Z fangq $ 11 | % 12 | % input: 13 | % key: a string with which one look up a value from a struct 14 | % default: if the key does not exist, return default 15 | % optstruct: a struct where each sub-field is a key 16 | % 17 | % output: 18 | % val: if key exists, val=optstruct.key; otherwise val=default 19 | % 20 | % license: 21 | % BSD, see LICENSE_BSD.txt files for details 22 | % 23 | % -- this function is part of jsonlab toolbox (http://iso2mesh.sf.net/cgi-bin/index.cgi?jsonlab) 24 | % 25 | 26 | val=default; 27 | if(nargin<=2) return; end 28 | opt=varargin{1}; 29 | if(isstruct(opt) && isfield(opt,key)) 30 | val=getfield(opt,key); 31 | end 32 | 33 | -------------------------------------------------------------------------------- /neural-networks/lib/jsonlab/mergestruct.m: -------------------------------------------------------------------------------- 1 | function s=mergestruct(s1,s2) 2 | % 3 | % s=mergestruct(s1,s2) 4 | % 5 | % merge two struct objects into one 6 | % 7 | % authors:Qianqian Fang (fangq nmr.mgh.harvard.edu) 8 | % date: 2012/12/22 9 | % 10 | % input: 11 | % s1,s2: a struct object, s1 and s2 can not be arrays 12 | % 13 | % output: 14 | % s: the merged struct object. fields in s1 and s2 will be combined in s. 15 | % 16 | % license: 17 | % BSD, see LICENSE_BSD.txt files for details 18 | % 19 | % -- this function is part of jsonlab toolbox (http://iso2mesh.sf.net/cgi-bin/index.cgi?jsonlab) 20 | % 21 | 22 | if(~isstruct(s1) || ~isstruct(s2)) 23 | error('input parameters contain non-struct'); 24 | end 25 | if(length(s1)>1 || length(s2)>1) 26 | error('can not merge struct arrays'); 27 | end 28 | fn=fieldnames(s2); 29 | s=s1; 30 | for i=1:length(fn) 31 | s=setfield(s,fn{i},getfield(s2,fn{i})); 32 | end 33 | 34 | -------------------------------------------------------------------------------- /neural-networks/lib/jsonlab/varargin2struct.m: -------------------------------------------------------------------------------- 1 | function opt=varargin2struct(varargin) 2 | % 3 | % opt=varargin2struct('param1',value1,'param2',value2,...) 4 | % or 5 | % opt=varargin2struct(...,optstruct,...) 6 | % 7 | % convert a series of input parameters into a structure 8 | % 9 | % authors:Qianqian Fang (fangq nmr.mgh.harvard.edu) 10 | % date: 2012/12/22 11 | % 12 | % input: 13 | % 'param', value: the input parameters should be pairs of a string and a value 14 | % optstruct: if a parameter is a struct, the fields will be merged to the output struct 15 | % 16 | % output: 17 | % opt: a struct where opt.param1=value1, opt.param2=value2 ... 18 | % 19 | % license: 20 | % BSD, see LICENSE_BSD.txt files for details 21 | % 22 | % -- this function is part of jsonlab toolbox (http://iso2mesh.sf.net/cgi-bin/index.cgi?jsonlab) 23 | % 24 | 25 | len=length(varargin); 26 | opt=struct; 27 | if(len==0) return; end 28 | i=1; 29 | while(i<=len) 30 | if(isstruct(varargin{i})) 31 | opt=mergestruct(opt,varargin{i}); 32 | elseif(ischar(varargin{i}) && i or 11 | 12 | 13 | The script loadjson.m was built upon previous works by 14 | 15 | - Nedialko Krouchev: http://www.mathworks.com/matlabcentral/fileexchange/25713 16 | date: 2009/11/02 17 | - François Glineur: http://www.mathworks.com/matlabcentral/fileexchange/23393 18 | date: 2009/03/22 19 | - Joel Feenstra: http://www.mathworks.com/matlabcentral/fileexchange/20565 20 | date: 2008/07/03 21 | 22 | 23 | This toolbox contains patches submitted by the following contributors: 24 | 25 | - Blake Johnson 26 | part of revision 341 27 | 28 | - Niclas Borlin 29 | various fixes in revision 394, including 30 | - loadjson crashes for all-zero sparse matrix. 31 | - loadjson crashes for empty sparse matrix. 32 | - Non-zero size of 0-by-N and N-by-0 empty matrices is lost after savejson/loadjson. 33 | - loadjson crashes for sparse real column vector. 34 | - loadjson crashes for sparse complex column vector. 35 | - Data is corrupted by savejson for sparse real row vector. 36 | - savejson crashes for sparse complex row vector. 37 | 38 | - Yul Kang 39 | patches for svn revision 415. 40 | - savejson saves an empty cell array as [] instead of null 41 | - loadjson differentiates an empty struct from an empty array 42 | -------------------------------------------------------------------------------- /regularized-linear-regression/lib/jsonlab/ChangeLog.txt: -------------------------------------------------------------------------------- 1 | ============================================================================ 2 | 3 | JSONlab - a toolbox to encode/decode JSON/UBJSON files in MATLAB/Octave 4 | 5 | ---------------------------------------------------------------------------- 6 | 7 | JSONlab ChangeLog (key features marked by *): 8 | 9 | == JSONlab 1.0 (codename: Optimus - Final), FangQ == 10 | 11 | 2015/01/02 polish help info for all major functions, update examples, finalize 1.0 12 | 2014/12/19 fix a bug to strictly respect NoRowBracket in savejson 13 | 14 | == JSONlab 1.0.0-RC2 (codename: Optimus - RC2), FangQ == 15 | 16 | 2014/11/22 show progress bar in loadjson ('ShowProgress') 17 | 2014/11/17 add Compact option in savejson to output compact JSON format ('Compact') 18 | 2014/11/17 add FastArrayParser in loadjson to specify fast parser applicable levels 19 | 2014/09/18 start official github mirror: https://github.com/fangq/jsonlab 20 | 21 | == JSONlab 1.0.0-RC1 (codename: Optimus - RC1), FangQ == 22 | 23 | 2014/09/17 fix several compatibility issues when running on octave versions 3.2-3.8 24 | 2014/09/17 support 2D cell and struct arrays in both savejson and saveubjson 25 | 2014/08/04 escape special characters in a JSON string 26 | 2014/02/16 fix a bug when saving ubjson files 27 | 28 | == JSONlab 0.9.9 (codename: Optimus - beta), FangQ == 29 | 30 | 2014/01/22 use binary read and write in saveubjson and loadubjson 31 | 32 | == JSONlab 0.9.8-1 (codename: Optimus - alpha update 1), FangQ == 33 | 34 | 2013/10/07 better round-trip conservation for empty arrays and structs (patch submitted by Yul Kang) 35 | 36 | == JSONlab 0.9.8 (codename: Optimus - alpha), FangQ == 37 | 2013/08/23 *universal Binary JSON (UBJSON) support, including both saveubjson and loadubjson 38 | 39 | == JSONlab 0.9.1 (codename: Rodimus, update 1), FangQ == 40 | 2012/12/18 *handling of various empty and sparse matrices (fixes submitted by Niclas Borlin) 41 | 42 | == JSONlab 0.9.0 (codename: Rodimus), FangQ == 43 | 44 | 2012/06/17 *new format for an invalid leading char, unpacking hex code in savejson 45 | 2012/06/01 support JSONP in savejson 46 | 2012/05/25 fix the empty cell bug (reported by Cyril Davin) 47 | 2012/04/05 savejson can save to a file (suggested by Patrick Rapin) 48 | 49 | == JSONlab 0.8.1 (codename: Sentiel, Update 1), FangQ == 50 | 51 | 2012/02/28 loadjson quotation mark escape bug, see http://bit.ly/yyk1nS 52 | 2012/01/25 patch to handle root-less objects, contributed by Blake Johnson 53 | 54 | == JSONlab 0.8.0 (codename: Sentiel), FangQ == 55 | 56 | 2012/01/13 *speed up loadjson by 20 fold when parsing large data arrays in matlab 57 | 2012/01/11 remove row bracket if an array has 1 element, suggested by Mykel Kochenderfer 58 | 2011/12/22 *accept sequence of 'param',value input in savejson and loadjson 59 | 2011/11/18 fix struct array bug reported by Mykel Kochenderfer 60 | 61 | == JSONlab 0.5.1 (codename: Nexus Update 1), FangQ == 62 | 63 | 2011/10/21 fix a bug in loadjson, previous code does not use any of the acceleration 64 | 2011/10/20 loadjson supports JSON collections - concatenated JSON objects 65 | 66 | == JSONlab 0.5.0 (codename: Nexus), FangQ == 67 | 68 | 2011/10/16 package and release jsonlab 0.5.0 69 | 2011/10/15 *add json demo and regression test, support cpx numbers, fix double quote bug 70 | 2011/10/11 *speed up readjson dramatically, interpret _Array* tags, show data in root level 71 | 2011/10/10 create jsonlab project, start jsonlab website, add online documentation 72 | 2011/10/07 *speed up savejson by 25x using sprintf instead of mat2str, add options support 73 | 2011/10/06 *savejson works for structs, cells and arrays 74 | 2011/09/09 derive loadjson from JSON parser from MATLAB Central, draft savejson.m 75 | -------------------------------------------------------------------------------- /regularized-linear-regression/lib/jsonlab/LICENSE_BSD.txt: -------------------------------------------------------------------------------- 1 | Copyright 2011-2015 Qianqian Fang . All rights reserved. 2 | 3 | Redistribution and use in source and binary forms, with or without modification, are 4 | permitted provided that the following conditions are met: 5 | 6 | 1. Redistributions of source code must retain the above copyright notice, this list of 7 | conditions and the following disclaimer. 8 | 9 | 2. Redistributions in binary form must reproduce the above copyright notice, this list 10 | of conditions and the following disclaimer in the documentation and/or other materials 11 | provided with the distribution. 12 | 13 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ''AS IS'' AND ANY EXPRESS OR IMPLIED 14 | WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND 15 | FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS 16 | OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 17 | CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 18 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON 19 | ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 20 | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF 21 | ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 22 | 23 | The views and conclusions contained in the software and documentation are those of the 24 | authors and should not be interpreted as representing official policies, either expressed 25 | or implied, of the copyright holders. 26 | -------------------------------------------------------------------------------- /regularized-linear-regression/lib/jsonlab/README.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jwasham/machine-learning/565cc5ad9837c0fb6f73a666ea517c3e70b1e6ff/regularized-linear-regression/lib/jsonlab/README.txt -------------------------------------------------------------------------------- /regularized-linear-regression/lib/jsonlab/jsonopt.m: -------------------------------------------------------------------------------- 1 | function val=jsonopt(key,default,varargin) 2 | % 3 | % val=jsonopt(key,default,optstruct) 4 | % 5 | % setting options based on a struct. The struct can be produced 6 | % by varargin2struct from a list of 'param','value' pairs 7 | % 8 | % authors:Qianqian Fang (fangq nmr.mgh.harvard.edu) 9 | % 10 | % $Id: loadjson.m 371 2012-06-20 12:43:06Z fangq $ 11 | % 12 | % input: 13 | % key: a string with which one look up a value from a struct 14 | % default: if the key does not exist, return default 15 | % optstruct: a struct where each sub-field is a key 16 | % 17 | % output: 18 | % val: if key exists, val=optstruct.key; otherwise val=default 19 | % 20 | % license: 21 | % BSD, see LICENSE_BSD.txt files for details 22 | % 23 | % -- this function is part of jsonlab toolbox (http://iso2mesh.sf.net/cgi-bin/index.cgi?jsonlab) 24 | % 25 | 26 | val=default; 27 | if(nargin<=2) return; end 28 | opt=varargin{1}; 29 | if(isstruct(opt) && isfield(opt,key)) 30 | val=getfield(opt,key); 31 | end 32 | 33 | -------------------------------------------------------------------------------- /regularized-linear-regression/lib/jsonlab/mergestruct.m: -------------------------------------------------------------------------------- 1 | function s=mergestruct(s1,s2) 2 | % 3 | % s=mergestruct(s1,s2) 4 | % 5 | % merge two struct objects into one 6 | % 7 | % authors:Qianqian Fang (fangq nmr.mgh.harvard.edu) 8 | % date: 2012/12/22 9 | % 10 | % input: 11 | % s1,s2: a struct object, s1 and s2 can not be arrays 12 | % 13 | % output: 14 | % s: the merged struct object. fields in s1 and s2 will be combined in s. 15 | % 16 | % license: 17 | % BSD, see LICENSE_BSD.txt files for details 18 | % 19 | % -- this function is part of jsonlab toolbox (http://iso2mesh.sf.net/cgi-bin/index.cgi?jsonlab) 20 | % 21 | 22 | if(~isstruct(s1) || ~isstruct(s2)) 23 | error('input parameters contain non-struct'); 24 | end 25 | if(length(s1)>1 || length(s2)>1) 26 | error('can not merge struct arrays'); 27 | end 28 | fn=fieldnames(s2); 29 | s=s1; 30 | for i=1:length(fn) 31 | s=setfield(s,fn{i},getfield(s2,fn{i})); 32 | end 33 | 34 | -------------------------------------------------------------------------------- /regularized-linear-regression/lib/jsonlab/varargin2struct.m: -------------------------------------------------------------------------------- 1 | function opt=varargin2struct(varargin) 2 | % 3 | % opt=varargin2struct('param1',value1,'param2',value2,...) 4 | % or 5 | % opt=varargin2struct(...,optstruct,...) 6 | % 7 | % convert a series of input parameters into a structure 8 | % 9 | % authors:Qianqian Fang (fangq nmr.mgh.harvard.edu) 10 | % date: 2012/12/22 11 | % 12 | % input: 13 | % 'param', value: the input parameters should be pairs of a string and a value 14 | % optstruct: if a parameter is a struct, the fields will be merged to the output struct 15 | % 16 | % output: 17 | % opt: a struct where opt.param1=value1, opt.param2=value2 ... 18 | % 19 | % license: 20 | % BSD, see LICENSE_BSD.txt files for details 21 | % 22 | % -- this function is part of jsonlab toolbox (http://iso2mesh.sf.net/cgi-bin/index.cgi?jsonlab) 23 | % 24 | 25 | len=length(varargin); 26 | opt=struct; 27 | if(len==0) return; end 28 | i=1; 29 | while(i<=len) 30 | if(isstruct(varargin{i})) 31 | opt=mergestruct(opt,varargin{i}); 32 | elseif(ischar(varargin{i}) && i Anyone knows how much it costs to host a web portal ? 2 | > 3 | Well, it depends on how many visitors you're expecting. 4 | This can be anywhere from less than 10 bucks a month to a couple of $100. 5 | You should checkout http://www.rackspace.com/ or perhaps Amazon EC2 6 | if youre running something big.. 7 | 8 | To unsubscribe yourself from this mailing list, send an email to: 9 | groupname-unsubscribe@egroups.com 10 | 11 | -------------------------------------------------------------------------------- /support-vector-machines/emailSample2.txt: -------------------------------------------------------------------------------- 1 | Folks, 2 | 3 | my first time posting - have a bit of Unix experience, but am new to Linux. 4 | 5 | 6 | Just got a new PC at home - Dell box with Windows XP. Added a second hard disk 7 | for Linux. Partitioned the disk and have installed Suse 7.2 from CD, which went 8 | fine except it didn't pick up my monitor. 9 | 10 | I have a Dell branded E151FPp 15" LCD flat panel monitor and a nVidia GeForce4 11 | Ti4200 video card, both of which are probably too new to feature in Suse's default 12 | set. I downloaded a driver from the nVidia website and installed it using RPM. 13 | Then I ran Sax2 (as was recommended in some postings I found on the net), but 14 | it still doesn't feature my video card in the available list. What next? 15 | 16 | Another problem. I have a Dell branded keyboard and if I hit Caps-Lock twice, 17 | the whole machine crashes (in Linux, not Windows) - even the on/off switch is 18 | inactive, leaving me to reach for the power cable instead. 19 | 20 | If anyone can help me in any way with these probs., I'd be really grateful - 21 | I've searched the 'net but have run out of ideas. 22 | 23 | Or should I be going for a different version of Linux such as RedHat? Opinions 24 | welcome. 25 | 26 | Thanks a lot, 27 | Peter 28 | 29 | -- 30 | Irish Linux Users' Group: ilug@linux.ie 31 | http://www.linux.ie/mailman/listinfo/ilug for (un)subscription information. 32 | List maintainer: listmaster@linux.ie 33 | 34 | 35 | -------------------------------------------------------------------------------- /support-vector-machines/ex6.m: -------------------------------------------------------------------------------- 1 | %% Machine Learning Online Class 2 | % Exercise 6 | Support Vector Machines 3 | % 4 | % Instructions 5 | % ------------ 6 | % 7 | % This file contains code that helps you get started on the 8 | % exercise. You will need to complete the following functions: 9 | % 10 | % gaussianKernel.m 11 | % dataset3Params.m 12 | % processEmail.m 13 | % emailFeatures.m 14 | % 15 | % For this exercise, you will not need to change any code in this file, 16 | % or any other files other than those mentioned above. 17 | % 18 | 19 | %% Initialization 20 | clear ; close all; clc 21 | 22 | %% =============== Part 1: Loading and Visualizing Data ================ 23 | % We start the exercise by first loading and visualizing the dataset. 24 | % The following code will load the dataset into your environment and plot 25 | % the data. 26 | % 27 | 28 | fprintf('Loading and Visualizing Data ...\n') 29 | 30 | % Load from ex6data1: 31 | % You will have X, y in your environment 32 | load('ex6data1.mat'); 33 | 34 | % Plot training data 35 | plotData(X, y); 36 | 37 | fprintf('Program paused. Press enter to continue.\n'); 38 | pause; 39 | 40 | %% ==================== Part 2: Training Linear SVM ==================== 41 | % The following code will train a linear SVM on the dataset and plot the 42 | % decision boundary learned. 43 | % 44 | 45 | % Load from ex6data1: 46 | % You will have X, y in your environment 47 | load('ex6data1.mat'); 48 | 49 | fprintf('\nTraining Linear SVM ...\n') 50 | 51 | % You should try to change the C value below and see how the decision 52 | % boundary varies (e.g., try C = 1000) 53 | C = 1; 54 | model = svmTrain(X, y, C, @linearKernel, 1e-3, 20); 55 | visualizeBoundaryLinear(X, y, model); 56 | 57 | fprintf('Program paused. Press enter to continue.\n'); 58 | pause; 59 | 60 | %% =============== Part 3: Implementing Gaussian Kernel =============== 61 | % You will now implement the Gaussian kernel to use 62 | % with the SVM. You should complete the code in gaussianKernel.m 63 | % 64 | fprintf('\nEvaluating the Gaussian Kernel ...\n') 65 | 66 | x1 = [1 2 1]; x2 = [0 4 -1]; sigma = 2; 67 | sigma, sim = gaussianKernel(x1, x2, sigma); 68 | 69 | fprintf(['Gaussian Kernel between x1 = [1; 2; 1], x2 = [0; 4; -1], sigma = %0.5f :' ... 70 | '\n\t%f\n(this value should be about 0.324652)\n'], sim); 71 | 72 | fprintf('Program paused. Press enter to continue.\n'); 73 | pause; 74 | 75 | %% =============== Part 4: Visualizing Dataset 2 ================ 76 | % The following code will load the next dataset into your environment and 77 | % plot the data. 78 | % 79 | 80 | fprintf('Loading and Visualizing Data ...\n') 81 | 82 | % Load from ex6data2: 83 | % You will have X, y in your environment 84 | load('ex6data2.mat'); 85 | 86 | % Plot training data 87 | plotData(X, y); 88 | 89 | fprintf('Program paused. Press enter to continue.\n'); 90 | pause; 91 | 92 | %% ========== Part 5: Training SVM with RBF Kernel (Dataset 2) ========== 93 | % After you have implemented the kernel, we can now use it to train the 94 | % SVM classifier. 95 | % 96 | fprintf('\nTraining SVM with RBF Kernel (this may take 1 to 2 minutes) ...\n'); 97 | 98 | % Load from ex6data2: 99 | % You will have X, y in your environment 100 | load('ex6data2.mat'); 101 | 102 | % SVM Parameters 103 | C = 1; sigma = 0.1; 104 | 105 | % We set the tolerance and max_passes lower here so that the code will run 106 | % faster. However, in practice, you will want to run the training to 107 | % convergence. 108 | model= svmTrain(X, y, C, @(x1, x2) gaussianKernel(x1, x2, sigma)); 109 | visualizeBoundary(X, y, model); 110 | 111 | fprintf('Program paused. Press enter to continue.\n'); 112 | pause; 113 | 114 | %% =============== Part 6: Visualizing Dataset 3 ================ 115 | % The following code will load the next dataset into your environment and 116 | % plot the data. 117 | % 118 | 119 | fprintf('Loading and Visualizing Data ...\n') 120 | 121 | % Load from ex6data3: 122 | % You will have X, y in your environment 123 | load('ex6data3.mat'); 124 | 125 | % Plot training data 126 | plotData(X, y); 127 | 128 | fprintf('Program paused. Press enter to continue.\n'); 129 | pause; 130 | 131 | %% ========== Part 7: Training SVM with RBF Kernel (Dataset 3) ========== 132 | 133 | % This is a different dataset that you can use to experiment with. Try 134 | % different values of C and sigma here. 135 | % 136 | 137 | % Load from ex6data3: 138 | % You will have X, y in your environment 139 | load('ex6data3.mat'); 140 | 141 | % Try different SVM Parameters here 142 | [C, sigma] = dataset3Params(X, y, Xval, yval); 143 | 144 | % Train the SVM 145 | model= svmTrain(X, y, C, @(x1, x2) gaussianKernel(x1, x2, sigma)); 146 | visualizeBoundary(X, y, model); 147 | 148 | fprintf('Program paused. Press enter to continue.\n'); 149 | pause; 150 | 151 | -------------------------------------------------------------------------------- /support-vector-machines/ex6_spam.m: -------------------------------------------------------------------------------- 1 | %% Machine Learning Online Class 2 | % Exercise 6 | Spam Classification with SVMs 3 | % 4 | % Instructions 5 | % ------------ 6 | % 7 | % This file contains code that helps you get started on the 8 | % exercise. You will need to complete the following functions: 9 | % 10 | % gaussianKernel.m 11 | % dataset3Params.m 12 | % processEmail.m 13 | % emailFeatures.m 14 | % 15 | % For this exercise, you will not need to change any code in this file, 16 | % or any other files other than those mentioned above. 17 | % 18 | 19 | %% Initialization 20 | clear ; close all; clc 21 | 22 | %% ==================== Part 1: Email Preprocessing ==================== 23 | % To use an SVM to classify emails into Spam v.s. Non-Spam, you first need 24 | % to convert each email into a vector of features. In this part, you will 25 | % implement the preprocessing steps for each email. You should 26 | % complete the code in processEmail.m to produce a word indices vector 27 | % for a given email. 28 | 29 | fprintf('\nPreprocessing sample email (emailSample1.txt)\n'); 30 | 31 | % Extract Features 32 | file_contents = readFile('emailSample1.txt'); 33 | word_indices = processEmail(file_contents); 34 | 35 | % Print Stats 36 | fprintf('Word Indices: \n'); 37 | fprintf(' %d', word_indices); 38 | fprintf('\n\n'); 39 | 40 | fprintf('Program paused. Press enter to continue.\n'); 41 | pause; 42 | 43 | %% ==================== Part 2: Feature Extraction ==================== 44 | % Now, you will convert each email into a vector of features in R^n. 45 | % You should complete the code in emailFeatures.m to produce a feature 46 | % vector for a given email. 47 | 48 | fprintf('\nExtracting features from sample email (emailSample1.txt)\n'); 49 | 50 | % Extract Features 51 | file_contents = readFile('emailSample1.txt'); 52 | word_indices = processEmail(file_contents); 53 | features = emailFeatures(word_indices); 54 | 55 | % Print Stats 56 | fprintf('Length of feature vector: %d\n', length(features)); 57 | fprintf('Number of non-zero entries: %d\n', sum(features > 0)); 58 | 59 | fprintf('Program paused. Press enter to continue.\n'); 60 | pause; 61 | 62 | %% =========== Part 3: Train Linear SVM for Spam Classification ======== 63 | % In this section, you will train a linear classifier to determine if an 64 | % email is Spam or Not-Spam. 65 | 66 | % Load the Spam Email dataset 67 | % You will have X, y in your environment 68 | load('spamTrain.mat'); 69 | 70 | fprintf('\nTraining Linear SVM (Spam Classification)\n') 71 | fprintf('(this may take 1 to 2 minutes) ...\n') 72 | 73 | C = 0.1; 74 | model = svmTrain(X, y, C, @linearKernel); 75 | 76 | p = svmPredict(model, X); 77 | 78 | fprintf('Training Accuracy: %f\n', mean(double(p == y)) * 100); 79 | 80 | %% =================== Part 4: Test Spam Classification ================ 81 | % After training the classifier, we can evaluate it on a test set. We have 82 | % included a test set in spamTest.mat 83 | 84 | % Load the test dataset 85 | % You will have Xtest, ytest in your environment 86 | load('spamTest.mat'); 87 | 88 | fprintf('\nEvaluating the trained Linear SVM on a test set ...\n') 89 | 90 | p = svmPredict(model, Xtest); 91 | 92 | fprintf('Test Accuracy: %f\n', mean(double(p == ytest)) * 100); 93 | pause; 94 | 95 | 96 | %% ================= Part 5: Top Predictors of Spam ==================== 97 | % Since the model we are training is a linear SVM, we can inspect the 98 | % weights learned by the model to understand better how it is determining 99 | % whether an email is spam or not. The following code finds the words with 100 | % the highest weights in the classifier. Informally, the classifier 101 | % 'thinks' that these words are the most likely indicators of spam. 102 | % 103 | 104 | % Sort the weights and obtin the vocabulary list 105 | [weight, idx] = sort(model.w, 'descend'); 106 | vocabList = getVocabList(); 107 | 108 | fprintf('\nTop predictors of spam: \n'); 109 | for i = 1:15 110 | fprintf(' %-15s (%f) \n', vocabList{idx(i)}, weight(i)); 111 | end 112 | 113 | fprintf('\n\n'); 114 | fprintf('\nProgram paused. Press enter to continue.\n'); 115 | pause; 116 | 117 | %% =================== Part 6: Try Your Own Emails ===================== 118 | % Now that you've trained the spam classifier, you can use it on your own 119 | % emails! In the starter code, we have included spamSample1.txt, 120 | % spamSample2.txt, emailSample1.txt and emailSample2.txt as examples. 121 | % The following code reads in one of these emails and then uses your 122 | % learned SVM classifier to determine whether the email is Spam or 123 | % Not Spam 124 | 125 | % Set the file to be read in (change this to spamSample2.txt, 126 | % emailSample1.txt or emailSample2.txt to see different predictions on 127 | % different emails types). Try your own emails as well! 128 | filename = 'spamSample1.txt'; 129 | 130 | % Read and predict 131 | file_contents = readFile(filename); 132 | word_indices = processEmail(file_contents); 133 | x = emailFeatures(word_indices); 134 | p = svmPredict(model, x); 135 | 136 | fprintf('\nProcessed %s\n\nSpam Classification: %d\n', filename, p); 137 | fprintf('(1 indicates spam, 0 indicates not spam)\n\n'); 138 | 139 | -------------------------------------------------------------------------------- /support-vector-machines/ex6data1.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jwasham/machine-learning/565cc5ad9837c0fb6f73a666ea517c3e70b1e6ff/support-vector-machines/ex6data1.mat -------------------------------------------------------------------------------- /support-vector-machines/ex6data2.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jwasham/machine-learning/565cc5ad9837c0fb6f73a666ea517c3e70b1e6ff/support-vector-machines/ex6data2.mat -------------------------------------------------------------------------------- /support-vector-machines/ex6data3.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jwasham/machine-learning/565cc5ad9837c0fb6f73a666ea517c3e70b1e6ff/support-vector-machines/ex6data3.mat -------------------------------------------------------------------------------- /support-vector-machines/gaussianKernel.m: -------------------------------------------------------------------------------- 1 | function sim = gaussianKernel(x1, x2, sigma) 2 | %RBFKERNEL returns a radial basis function kernel between x1 and x2 3 | % sim = gaussianKernel(x1, x2) returns a gaussian kernel between x1 and x2 4 | % and returns the value in sim 5 | 6 | % Ensure that x1 and x2 are column vectors 7 | x1 = x1(:); x2 = x2(:); 8 | 9 | % You need to return the following variables correctly. 10 | sim = 0; 11 | 12 | % ====================== YOUR CODE HERE ====================== 13 | % Instructions: Fill in this function to return the similarity between x1 14 | % and x2 computed using a Gaussian kernel with bandwidth 15 | % sigma 16 | % 17 | % 18 | 19 | distance = sum((x1 - x2).^2); 20 | 21 | sim = e ^ (-1 * (distance / (2 * sigma^2) )); 22 | 23 | % ============================================================= 24 | 25 | end 26 | -------------------------------------------------------------------------------- /support-vector-machines/getVocabList.m: -------------------------------------------------------------------------------- 1 | function vocabList = getVocabList() 2 | %GETVOCABLIST reads the fixed vocabulary list in vocab.txt and returns a 3 | %cell array of the words 4 | % vocabList = GETVOCABLIST() reads the fixed vocabulary list in vocab.txt 5 | % and returns a cell array of the words in vocabList. 6 | 7 | 8 | %% Read the fixed vocabulary list 9 | fid = fopen('vocab.txt'); 10 | 11 | % Store all dictionary words in cell array vocab{} 12 | n = 1899; % Total number of words in the dictionary 13 | 14 | % For ease of implementation, we use a struct to map the strings => integers 15 | % In practice, you'll want to use some form of hashmap 16 | vocabList = cell(n, 1); 17 | for i = 1:n 18 | % Word Index (can ignore since it will be = i) 19 | fscanf(fid, '%d', 1); 20 | % Actual Word 21 | vocabList{i} = fscanf(fid, '%s', 1); 22 | end 23 | fclose(fid); 24 | 25 | end 26 | -------------------------------------------------------------------------------- /support-vector-machines/lib/jsonlab/AUTHORS.txt: -------------------------------------------------------------------------------- 1 | The author of "jsonlab" toolbox is Qianqian Fang. Qianqian 2 | is currently an Assistant Professor at Massachusetts General Hospital, 3 | Harvard Medical School. 4 | 5 | Address: Martinos Center for Biomedical Imaging, 6 | Massachusetts General Hospital, 7 | Harvard Medical School 8 | Bldg 149, 13th St, Charlestown, MA 02129, USA 9 | URL: http://nmr.mgh.harvard.edu/~fangq/ 10 | Email: or 11 | 12 | 13 | The script loadjson.m was built upon previous works by 14 | 15 | - Nedialko Krouchev: http://www.mathworks.com/matlabcentral/fileexchange/25713 16 | date: 2009/11/02 17 | - François Glineur: http://www.mathworks.com/matlabcentral/fileexchange/23393 18 | date: 2009/03/22 19 | - Joel Feenstra: http://www.mathworks.com/matlabcentral/fileexchange/20565 20 | date: 2008/07/03 21 | 22 | 23 | This toolbox contains patches submitted by the following contributors: 24 | 25 | - Blake Johnson 26 | part of revision 341 27 | 28 | - Niclas Borlin 29 | various fixes in revision 394, including 30 | - loadjson crashes for all-zero sparse matrix. 31 | - loadjson crashes for empty sparse matrix. 32 | - Non-zero size of 0-by-N and N-by-0 empty matrices is lost after savejson/loadjson. 33 | - loadjson crashes for sparse real column vector. 34 | - loadjson crashes for sparse complex column vector. 35 | - Data is corrupted by savejson for sparse real row vector. 36 | - savejson crashes for sparse complex row vector. 37 | 38 | - Yul Kang 39 | patches for svn revision 415. 40 | - savejson saves an empty cell array as [] instead of null 41 | - loadjson differentiates an empty struct from an empty array 42 | -------------------------------------------------------------------------------- /support-vector-machines/lib/jsonlab/ChangeLog.txt: -------------------------------------------------------------------------------- 1 | ============================================================================ 2 | 3 | JSONlab - a toolbox to encode/decode JSON/UBJSON files in MATLAB/Octave 4 | 5 | ---------------------------------------------------------------------------- 6 | 7 | JSONlab ChangeLog (key features marked by *): 8 | 9 | == JSONlab 1.0 (codename: Optimus - Final), FangQ == 10 | 11 | 2015/01/02 polish help info for all major functions, update examples, finalize 1.0 12 | 2014/12/19 fix a bug to strictly respect NoRowBracket in savejson 13 | 14 | == JSONlab 1.0.0-RC2 (codename: Optimus - RC2), FangQ == 15 | 16 | 2014/11/22 show progress bar in loadjson ('ShowProgress') 17 | 2014/11/17 add Compact option in savejson to output compact JSON format ('Compact') 18 | 2014/11/17 add FastArrayParser in loadjson to specify fast parser applicable levels 19 | 2014/09/18 start official github mirror: https://github.com/fangq/jsonlab 20 | 21 | == JSONlab 1.0.0-RC1 (codename: Optimus - RC1), FangQ == 22 | 23 | 2014/09/17 fix several compatibility issues when running on octave versions 3.2-3.8 24 | 2014/09/17 support 2D cell and struct arrays in both savejson and saveubjson 25 | 2014/08/04 escape special characters in a JSON string 26 | 2014/02/16 fix a bug when saving ubjson files 27 | 28 | == JSONlab 0.9.9 (codename: Optimus - beta), FangQ == 29 | 30 | 2014/01/22 use binary read and write in saveubjson and loadubjson 31 | 32 | == JSONlab 0.9.8-1 (codename: Optimus - alpha update 1), FangQ == 33 | 34 | 2013/10/07 better round-trip conservation for empty arrays and structs (patch submitted by Yul Kang) 35 | 36 | == JSONlab 0.9.8 (codename: Optimus - alpha), FangQ == 37 | 2013/08/23 *universal Binary JSON (UBJSON) support, including both saveubjson and loadubjson 38 | 39 | == JSONlab 0.9.1 (codename: Rodimus, update 1), FangQ == 40 | 2012/12/18 *handling of various empty and sparse matrices (fixes submitted by Niclas Borlin) 41 | 42 | == JSONlab 0.9.0 (codename: Rodimus), FangQ == 43 | 44 | 2012/06/17 *new format for an invalid leading char, unpacking hex code in savejson 45 | 2012/06/01 support JSONP in savejson 46 | 2012/05/25 fix the empty cell bug (reported by Cyril Davin) 47 | 2012/04/05 savejson can save to a file (suggested by Patrick Rapin) 48 | 49 | == JSONlab 0.8.1 (codename: Sentiel, Update 1), FangQ == 50 | 51 | 2012/02/28 loadjson quotation mark escape bug, see http://bit.ly/yyk1nS 52 | 2012/01/25 patch to handle root-less objects, contributed by Blake Johnson 53 | 54 | == JSONlab 0.8.0 (codename: Sentiel), FangQ == 55 | 56 | 2012/01/13 *speed up loadjson by 20 fold when parsing large data arrays in matlab 57 | 2012/01/11 remove row bracket if an array has 1 element, suggested by Mykel Kochenderfer 58 | 2011/12/22 *accept sequence of 'param',value input in savejson and loadjson 59 | 2011/11/18 fix struct array bug reported by Mykel Kochenderfer 60 | 61 | == JSONlab 0.5.1 (codename: Nexus Update 1), FangQ == 62 | 63 | 2011/10/21 fix a bug in loadjson, previous code does not use any of the acceleration 64 | 2011/10/20 loadjson supports JSON collections - concatenated JSON objects 65 | 66 | == JSONlab 0.5.0 (codename: Nexus), FangQ == 67 | 68 | 2011/10/16 package and release jsonlab 0.5.0 69 | 2011/10/15 *add json demo and regression test, support cpx numbers, fix double quote bug 70 | 2011/10/11 *speed up readjson dramatically, interpret _Array* tags, show data in root level 71 | 2011/10/10 create jsonlab project, start jsonlab website, add online documentation 72 | 2011/10/07 *speed up savejson by 25x using sprintf instead of mat2str, add options support 73 | 2011/10/06 *savejson works for structs, cells and arrays 74 | 2011/09/09 derive loadjson from JSON parser from MATLAB Central, draft savejson.m 75 | -------------------------------------------------------------------------------- /support-vector-machines/lib/jsonlab/LICENSE_BSD.txt: -------------------------------------------------------------------------------- 1 | Copyright 2011-2015 Qianqian Fang . All rights reserved. 2 | 3 | Redistribution and use in source and binary forms, with or without modification, are 4 | permitted provided that the following conditions are met: 5 | 6 | 1. Redistributions of source code must retain the above copyright notice, this list of 7 | conditions and the following disclaimer. 8 | 9 | 2. Redistributions in binary form must reproduce the above copyright notice, this list 10 | of conditions and the following disclaimer in the documentation and/or other materials 11 | provided with the distribution. 12 | 13 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ''AS IS'' AND ANY EXPRESS OR IMPLIED 14 | WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND 15 | FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS 16 | OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 17 | CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 18 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON 19 | ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 20 | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF 21 | ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 22 | 23 | The views and conclusions contained in the software and documentation are those of the 24 | authors and should not be interpreted as representing official policies, either expressed 25 | or implied, of the copyright holders. 26 | -------------------------------------------------------------------------------- /support-vector-machines/lib/jsonlab/README.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jwasham/machine-learning/565cc5ad9837c0fb6f73a666ea517c3e70b1e6ff/support-vector-machines/lib/jsonlab/README.txt -------------------------------------------------------------------------------- /support-vector-machines/lib/jsonlab/jsonopt.m: -------------------------------------------------------------------------------- 1 | function val=jsonopt(key,default,varargin) 2 | % 3 | % val=jsonopt(key,default,optstruct) 4 | % 5 | % setting options based on a struct. The struct can be produced 6 | % by varargin2struct from a list of 'param','value' pairs 7 | % 8 | % authors:Qianqian Fang (fangq nmr.mgh.harvard.edu) 9 | % 10 | % $Id: loadjson.m 371 2012-06-20 12:43:06Z fangq $ 11 | % 12 | % input: 13 | % key: a string with which one look up a value from a struct 14 | % default: if the key does not exist, return default 15 | % optstruct: a struct where each sub-field is a key 16 | % 17 | % output: 18 | % val: if key exists, val=optstruct.key; otherwise val=default 19 | % 20 | % license: 21 | % BSD, see LICENSE_BSD.txt files for details 22 | % 23 | % -- this function is part of jsonlab toolbox (http://iso2mesh.sf.net/cgi-bin/index.cgi?jsonlab) 24 | % 25 | 26 | val=default; 27 | if(nargin<=2) return; end 28 | opt=varargin{1}; 29 | if(isstruct(opt) && isfield(opt,key)) 30 | val=getfield(opt,key); 31 | end 32 | 33 | -------------------------------------------------------------------------------- /support-vector-machines/lib/jsonlab/mergestruct.m: -------------------------------------------------------------------------------- 1 | function s=mergestruct(s1,s2) 2 | % 3 | % s=mergestruct(s1,s2) 4 | % 5 | % merge two struct objects into one 6 | % 7 | % authors:Qianqian Fang (fangq nmr.mgh.harvard.edu) 8 | % date: 2012/12/22 9 | % 10 | % input: 11 | % s1,s2: a struct object, s1 and s2 can not be arrays 12 | % 13 | % output: 14 | % s: the merged struct object. fields in s1 and s2 will be combined in s. 15 | % 16 | % license: 17 | % BSD, see LICENSE_BSD.txt files for details 18 | % 19 | % -- this function is part of jsonlab toolbox (http://iso2mesh.sf.net/cgi-bin/index.cgi?jsonlab) 20 | % 21 | 22 | if(~isstruct(s1) || ~isstruct(s2)) 23 | error('input parameters contain non-struct'); 24 | end 25 | if(length(s1)>1 || length(s2)>1) 26 | error('can not merge struct arrays'); 27 | end 28 | fn=fieldnames(s2); 29 | s=s1; 30 | for i=1:length(fn) 31 | s=setfield(s,fn{i},getfield(s2,fn{i})); 32 | end 33 | 34 | -------------------------------------------------------------------------------- /support-vector-machines/lib/jsonlab/varargin2struct.m: -------------------------------------------------------------------------------- 1 | function opt=varargin2struct(varargin) 2 | % 3 | % opt=varargin2struct('param1',value1,'param2',value2,...) 4 | % or 5 | % opt=varargin2struct(...,optstruct,...) 6 | % 7 | % convert a series of input parameters into a structure 8 | % 9 | % authors:Qianqian Fang (fangq nmr.mgh.harvard.edu) 10 | % date: 2012/12/22 11 | % 12 | % input: 13 | % 'param', value: the input parameters should be pairs of a string and a value 14 | % optstruct: if a parameter is a struct, the fields will be merged to the output struct 15 | % 16 | % output: 17 | % opt: a struct where opt.param1=value1, opt.param2=value2 ... 18 | % 19 | % license: 20 | % BSD, see LICENSE_BSD.txt files for details 21 | % 22 | % -- this function is part of jsonlab toolbox (http://iso2mesh.sf.net/cgi-bin/index.cgi?jsonlab) 23 | % 24 | 25 | len=length(varargin); 26 | opt=struct; 27 | if(len==0) return; end 28 | i=1; 29 | while(i<=len) 30 | if(isstruct(varargin{i})) 31 | opt=mergestruct(opt,varargin{i}); 32 | elseif(ischar(varargin{i}) && i and replace 29 | % and does not have any < or > in the tag it with a space 30 | email_contents = regexprep(email_contents, '<[^<>]+>', ' '); 31 | 32 | % Handle Numbers 33 | % Look for one or more characters between 0-9 34 | email_contents = regexprep(email_contents, '[0-9]+', 'number'); 35 | 36 | % Handle URLS 37 | % Look for strings starting with http:// or https:// 38 | email_contents = regexprep(email_contents, ... 39 | '(http|https)://[^\s]*', 'httpaddr'); 40 | 41 | % Handle Email Addresses 42 | % Look for strings with @ in the middle 43 | email_contents = regexprep(email_contents, '[^\s]+@[^\s]+', 'emailaddr'); 44 | 45 | % Handle $ sign 46 | email_contents = regexprep(email_contents, '[$]+', 'dollar'); 47 | 48 | 49 | % ========================== Tokenize Email =========================== 50 | 51 | % Output the email to screen as well 52 | fprintf('\n==== Processed Email ====\n\n'); 53 | 54 | % Process file 55 | l = 0; 56 | 57 | while ~isempty(email_contents) 58 | 59 | % Tokenize and also get rid of any punctuation 60 | [str, email_contents] = ... 61 | strtok(email_contents, ... 62 | [' @$/#.-:&*+=[]?!(){},''">_<;%' char(10) char(13)]); 63 | 64 | % Remove any non alphanumeric characters 65 | str = regexprep(str, '[^a-zA-Z0-9]', ''); 66 | 67 | % Stem the word 68 | % (the porterStemmer sometimes has issues, so we use a try catch block) 69 | try str = porterStemmer(strtrim(str)); 70 | catch str = ''; continue; 71 | end; 72 | 73 | % Skip the word if it is too short 74 | if length(str) < 1 75 | continue; 76 | end 77 | 78 | % Look up the word in the dictionary and add to word_indices if 79 | % found 80 | % ====================== YOUR CODE HERE ====================== 81 | % Instructions: Fill in this function to add the index of str to 82 | % word_indices if it is in the vocabulary. At this point 83 | % of the code, you have a stemmed word from the email in 84 | % the variable str. You should look up str in the 85 | % vocabulary list (vocabList). If a match exists, you 86 | % should add the index of the word to the word_indices 87 | % vector. Concretely, if str = 'action', then you should 88 | % look up the vocabulary list to find where in vocabList 89 | % 'action' appears. For example, if vocabList{18} = 90 | % 'action', then, you should add 18 to the word_indices 91 | % vector (e.g., word_indices = [word_indices ; 18]; ). 92 | % 93 | % Note: vocabList{idx} returns a the word with index idx in the 94 | % vocabulary list. 95 | % 96 | % Note: You can use strcmp(str1, str2) to compare two strings (str1 and 97 | % str2). It will return 1 only if the two strings are equivalent. 98 | % 99 | 100 | for i = 1:length(vocabList) 101 | if(strcmp(str, vocabList{i})) 102 | word_indices = [word_indices ; i]; 103 | end 104 | end 105 | 106 | % ============================================================= 107 | 108 | 109 | % Print to screen, ensuring that the output lines are not too long 110 | if (l + length(str) + 1) > 78 111 | fprintf('\n'); 112 | l = 0; 113 | end 114 | fprintf('%s ', str); 115 | l = l + length(str) + 1; 116 | 117 | end 118 | 119 | % Print footer 120 | fprintf('\n\n=========================\n'); 121 | 122 | end 123 | -------------------------------------------------------------------------------- /support-vector-machines/readFile.m: -------------------------------------------------------------------------------- 1 | function file_contents = readFile(filename) 2 | %READFILE reads a file and returns its entire contents 3 | % file_contents = READFILE(filename) reads a file and returns its entire 4 | % contents in file_contents 5 | % 6 | 7 | % Load File 8 | fid = fopen(filename); 9 | if fid 10 | file_contents = fscanf(fid, '%c', inf); 11 | fclose(fid); 12 | else 13 | file_contents = ''; 14 | fprintf('Unable to open %s\n', filename); 15 | end 16 | 17 | end 18 | 19 | -------------------------------------------------------------------------------- /support-vector-machines/spamSample1.txt: -------------------------------------------------------------------------------- 1 | Do You Want To Make $1000 Or More Per Week? 2 | 3 | 4 | 5 | If you are a motivated and qualified individual - I 6 | will personally demonstrate to you a system that will 7 | make you $1,000 per week or more! This is NOT mlm. 8 | 9 | 10 | 11 | Call our 24 hour pre-recorded number to get the 12 | details. 13 | 14 | 15 | 16 | 000-456-789 17 | 18 | 19 | 20 | I need people who want to make serious money. Make 21 | the call and get the facts. 22 | 23 | Invest 2 minutes in yourself now! 24 | 25 | 26 | 27 | 000-456-789 28 | 29 | 30 | 31 | Looking forward to your call and I will introduce you 32 | to people like yourself who 33 | are currently making $10,000 plus per week! 34 | 35 | 36 | 37 | 000-456-789 38 | 39 | 40 | 41 | 3484lJGv6-241lEaN9080lRmS6-271WxHo7524qiyT5-438rjUv5615hQcf0-662eiDB9057dMtVl72 42 | 43 | -------------------------------------------------------------------------------- /support-vector-machines/spamSample2.txt: -------------------------------------------------------------------------------- 1 | Best Buy Viagra Generic Online 2 | 3 | Viagra 100mg x 60 Pills $125, Free Pills & Reorder Discount, Top Selling 100% Quality & Satisfaction guaranteed! 4 | 5 | We accept VISA, Master & E-Check Payments, 90000+ Satisfied Customers! 6 | http://medphysitcstech.ru 7 | 8 | 9 | -------------------------------------------------------------------------------- /support-vector-machines/spamTest.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jwasham/machine-learning/565cc5ad9837c0fb6f73a666ea517c3e70b1e6ff/support-vector-machines/spamTest.mat -------------------------------------------------------------------------------- /support-vector-machines/spamTrain.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jwasham/machine-learning/565cc5ad9837c0fb6f73a666ea517c3e70b1e6ff/support-vector-machines/spamTrain.mat -------------------------------------------------------------------------------- /support-vector-machines/svmPredict.m: -------------------------------------------------------------------------------- 1 | function pred = svmPredict(model, X) 2 | %SVMPREDICT returns a vector of predictions using a trained SVM model 3 | %(svmTrain). 4 | % pred = SVMPREDICT(model, X) returns a vector of predictions using a 5 | % trained SVM model (svmTrain). X is a mxn matrix where there each 6 | % example is a row. model is a svm model returned from svmTrain. 7 | % predictions pred is a m x 1 column of predictions of {0, 1} values. 8 | % 9 | 10 | % Check if we are getting a column vector, if so, then assume that we only 11 | % need to do prediction for a single example 12 | if (size(X, 2) == 1) 13 | % Examples should be in rows 14 | X = X'; 15 | end 16 | 17 | % Dataset 18 | m = size(X, 1); 19 | p = zeros(m, 1); 20 | pred = zeros(m, 1); 21 | 22 | if strcmp(func2str(model.kernelFunction), 'linearKernel') 23 | % We can use the weights and bias directly if working with the 24 | % linear kernel 25 | p = X * model.w + model.b; 26 | elseif strfind(func2str(model.kernelFunction), 'gaussianKernel') 27 | % Vectorized RBF Kernel 28 | % This is equivalent to computing the kernel on every pair of examples 29 | X1 = sum(X.^2, 2); 30 | X2 = sum(model.X.^2, 2)'; 31 | K = bsxfun(@plus, X1, bsxfun(@plus, X2, - 2 * X * model.X')); 32 | K = model.kernelFunction(1, 0) .^ K; 33 | K = bsxfun(@times, model.y', K); 34 | K = bsxfun(@times, model.alphas', K); 35 | p = sum(K, 2); 36 | else 37 | % Other Non-linear kernel 38 | for i = 1:m 39 | prediction = 0; 40 | for j = 1:size(model.X, 1) 41 | prediction = prediction + ... 42 | model.alphas(j) * model.y(j) * ... 43 | model.kernelFunction(X(i,:)', model.X(j,:)'); 44 | end 45 | p(i) = prediction + model.b; 46 | end 47 | end 48 | 49 | % Convert predictions into 0 / 1 50 | pred(p >= 0) = 1; 51 | pred(p < 0) = 0; 52 | 53 | end 54 | 55 | -------------------------------------------------------------------------------- /support-vector-machines/svmTrain.m: -------------------------------------------------------------------------------- 1 | function [model] = svmTrain(X, Y, C, kernelFunction, ... 2 | tol, max_passes) 3 | %SVMTRAIN Trains an SVM classifier using a simplified version of the SMO 4 | %algorithm. 5 | % [model] = SVMTRAIN(X, Y, C, kernelFunction, tol, max_passes) trains an 6 | % SVM classifier and returns trained model. X is the matrix of training 7 | % examples. Each row is a training example, and the jth column holds the 8 | % jth feature. Y is a column matrix containing 1 for positive examples 9 | % and 0 for negative examples. C is the standard SVM regularization 10 | % parameter. tol is a tolerance value used for determining equality of 11 | % floating point numbers. max_passes controls the number of iterations 12 | % over the dataset (without changes to alpha) before the algorithm quits. 13 | % 14 | % Note: This is a simplified version of the SMO algorithm for training 15 | % SVMs. In practice, if you want to train an SVM classifier, we 16 | % recommend using an optimized package such as: 17 | % 18 | % LIBSVM (http://www.csie.ntu.edu.tw/~cjlin/libsvm/) 19 | % SVMLight (http://svmlight.joachims.org/) 20 | % 21 | % 22 | 23 | if ~exist('tol', 'var') || isempty(tol) 24 | tol = 1e-3; 25 | end 26 | 27 | if ~exist('max_passes', 'var') || isempty(max_passes) 28 | max_passes = 5; 29 | end 30 | 31 | % Data parameters 32 | m = size(X, 1); 33 | n = size(X, 2); 34 | 35 | % Map 0 to -1 36 | Y(Y==0) = -1; 37 | 38 | % Variables 39 | alphas = zeros(m, 1); 40 | b = 0; 41 | E = zeros(m, 1); 42 | passes = 0; 43 | eta = 0; 44 | L = 0; 45 | H = 0; 46 | 47 | % Pre-compute the Kernel Matrix since our dataset is small 48 | % (in practice, optimized SVM packages that handle large datasets 49 | % gracefully will _not_ do this) 50 | % 51 | % We have implemented optimized vectorized version of the Kernels here so 52 | % that the svm training will run faster. 53 | if strcmp(func2str(kernelFunction), 'linearKernel') 54 | % Vectorized computation for the Linear Kernel 55 | % This is equivalent to computing the kernel on every pair of examples 56 | K = X*X'; 57 | elseif strfind(func2str(kernelFunction), 'gaussianKernel') 58 | % Vectorized RBF Kernel 59 | % This is equivalent to computing the kernel on every pair of examples 60 | X2 = sum(X.^2, 2); 61 | K = bsxfun(@plus, X2, bsxfun(@plus, X2', - 2 * (X * X'))); 62 | K = kernelFunction(1, 0) .^ K; 63 | else 64 | % Pre-compute the Kernel Matrix 65 | % The following can be slow due to the lack of vectorization 66 | K = zeros(m); 67 | for i = 1:m 68 | for j = i:m 69 | K(i,j) = kernelFunction(X(i,:)', X(j,:)'); 70 | K(j,i) = K(i,j); %the matrix is symmetric 71 | end 72 | end 73 | end 74 | 75 | % Train 76 | fprintf('\nTraining ...'); 77 | dots = 12; 78 | while passes < max_passes, 79 | 80 | num_changed_alphas = 0; 81 | for i = 1:m, 82 | 83 | % Calculate Ei = f(x(i)) - y(i) using (2). 84 | % E(i) = b + sum (X(i, :) * (repmat(alphas.*Y,1,n).*X)') - Y(i); 85 | E(i) = b + sum (alphas.*Y.*K(:,i)) - Y(i); 86 | 87 | if ((Y(i)*E(i) < -tol && alphas(i) < C) || (Y(i)*E(i) > tol && alphas(i) > 0)), 88 | 89 | % In practice, there are many heuristics one can use to select 90 | % the i and j. In this simplified code, we select them randomly. 91 | j = ceil(m * rand()); 92 | while j == i, % Make sure i \neq j 93 | j = ceil(m * rand()); 94 | end 95 | 96 | % Calculate Ej = f(x(j)) - y(j) using (2). 97 | E(j) = b + sum (alphas.*Y.*K(:,j)) - Y(j); 98 | 99 | % Save old alphas 100 | alpha_i_old = alphas(i); 101 | alpha_j_old = alphas(j); 102 | 103 | % Compute L and H by (10) or (11). 104 | if (Y(i) == Y(j)), 105 | L = max(0, alphas(j) + alphas(i) - C); 106 | H = min(C, alphas(j) + alphas(i)); 107 | else 108 | L = max(0, alphas(j) - alphas(i)); 109 | H = min(C, C + alphas(j) - alphas(i)); 110 | end 111 | 112 | if (L == H), 113 | % continue to next i. 114 | continue; 115 | end 116 | 117 | % Compute eta by (14). 118 | eta = 2 * K(i,j) - K(i,i) - K(j,j); 119 | if (eta >= 0), 120 | % continue to next i. 121 | continue; 122 | end 123 | 124 | % Compute and clip new value for alpha j using (12) and (15). 125 | alphas(j) = alphas(j) - (Y(j) * (E(i) - E(j))) / eta; 126 | 127 | % Clip 128 | alphas(j) = min (H, alphas(j)); 129 | alphas(j) = max (L, alphas(j)); 130 | 131 | % Check if change in alpha is significant 132 | if (abs(alphas(j) - alpha_j_old) < tol), 133 | % continue to next i. 134 | % replace anyway 135 | alphas(j) = alpha_j_old; 136 | continue; 137 | end 138 | 139 | % Determine value for alpha i using (16). 140 | alphas(i) = alphas(i) + Y(i)*Y(j)*(alpha_j_old - alphas(j)); 141 | 142 | % Compute b1 and b2 using (17) and (18) respectively. 143 | b1 = b - E(i) ... 144 | - Y(i) * (alphas(i) - alpha_i_old) * K(i,j)' ... 145 | - Y(j) * (alphas(j) - alpha_j_old) * K(i,j)'; 146 | b2 = b - E(j) ... 147 | - Y(i) * (alphas(i) - alpha_i_old) * K(i,j)' ... 148 | - Y(j) * (alphas(j) - alpha_j_old) * K(j,j)'; 149 | 150 | % Compute b by (19). 151 | if (0 < alphas(i) && alphas(i) < C), 152 | b = b1; 153 | elseif (0 < alphas(j) && alphas(j) < C), 154 | b = b2; 155 | else 156 | b = (b1+b2)/2; 157 | end 158 | 159 | num_changed_alphas = num_changed_alphas + 1; 160 | 161 | end 162 | 163 | end 164 | 165 | if (num_changed_alphas == 0), 166 | passes = passes + 1; 167 | else 168 | passes = 0; 169 | end 170 | 171 | fprintf('.'); 172 | dots = dots + 1; 173 | if dots > 78 174 | dots = 0; 175 | fprintf('\n'); 176 | end 177 | if exist('OCTAVE_VERSION') 178 | fflush(stdout); 179 | end 180 | end 181 | fprintf(' Done! \n\n'); 182 | 183 | % Save the model 184 | idx = alphas > 0; 185 | model.X= X(idx,:); 186 | model.y= Y(idx); 187 | model.kernelFunction = kernelFunction; 188 | model.b= b; 189 | model.alphas= alphas(idx); 190 | model.w = ((alphas.*Y)'*X)'; 191 | 192 | end 193 | -------------------------------------------------------------------------------- /support-vector-machines/token.mat: -------------------------------------------------------------------------------- 1 | # Created by Octave 3.8.1, Sun Mar 13 22:27:09 2016 PDT 2 | # name: email 3 | # type: sq_string 4 | # elements: 1 5 | # length: 20 6 | john@readyprompt.com 7 | 8 | 9 | # name: token 10 | # type: sq_string 11 | # elements: 1 12 | # length: 16 13 | BKN5XarcMk9UbFUd 14 | 15 | 16 | -------------------------------------------------------------------------------- /support-vector-machines/visualizeBoundary.m: -------------------------------------------------------------------------------- 1 | function visualizeBoundary(X, y, model, varargin) 2 | %VISUALIZEBOUNDARY plots a non-linear decision boundary learned by the SVM 3 | % VISUALIZEBOUNDARYLINEAR(X, y, model) plots a non-linear decision 4 | % boundary learned by the SVM and overlays the data on it 5 | 6 | % Plot the training data on top of the boundary 7 | plotData(X, y) 8 | 9 | % Make classification predictions over a grid of values 10 | x1plot = linspace(min(X(:,1)), max(X(:,1)), 100)'; 11 | x2plot = linspace(min(X(:,2)), max(X(:,2)), 100)'; 12 | [X1, X2] = meshgrid(x1plot, x2plot); 13 | vals = zeros(size(X1)); 14 | for i = 1:size(X1, 2) 15 | this_X = [X1(:, i), X2(:, i)]; 16 | vals(:, i) = svmPredict(model, this_X); 17 | end 18 | 19 | % Plot the SVM boundary 20 | hold on 21 | %contour(X1, X2, vals, [0 0], 'Color', 'b'); 22 | contour(X1, X2, vals, [1 1], 'b'); 23 | hold off; 24 | 25 | end 26 | -------------------------------------------------------------------------------- /support-vector-machines/visualizeBoundaryLinear.m: -------------------------------------------------------------------------------- 1 | function visualizeBoundaryLinear(X, y, model) 2 | %VISUALIZEBOUNDARYLINEAR plots a linear decision boundary learned by the 3 | %SVM 4 | % VISUALIZEBOUNDARYLINEAR(X, y, model) plots a linear decision boundary 5 | % learned by the SVM and overlays the data on it 6 | 7 | w = model.w; 8 | b = model.b; 9 | xp = linspace(min(X(:,1)), max(X(:,1)), 100); 10 | yp = - (w(1)*xp + b)/w(2); 11 | plotData(X, y); 12 | hold on; 13 | plot(xp, yp, '-b'); 14 | hold off 15 | 16 | end 17 | --------------------------------------------------------------------------------