├── AnomalyDetection ├── data1.mat ├── data2.mat ├── estimateGaussian.m ├── main.m ├── multivariateGaussian.m ├── selectThreshold.m └── visualizeFit.m ├── CollaborativeFiltering ├── checkCostFunction.m ├── cofiCostFunction.m ├── computeNumericalGradient.m ├── fmincg.m ├── loadMovieList.m ├── main.m ├── movieParams.mat ├── movie_ids.txt ├── movies.mat └── normalizeRatings.m ├── K-Means ├── bird.mat ├── bird.png ├── computeCentroids.m ├── data.mat ├── drawLine.m ├── findClosestCentroids.m ├── kMeansInitCentroids.m ├── main.m ├── plotDataPoints.m ├── plotProgresskMeans.m ├── readme.md └── runkMeans.m ├── LinearRegression ├── computerCost.m ├── data.txt ├── featureNormalize.m ├── gradientDescent.m ├── main.m ├── normalEquations.m └── readme.md ├── LogisticRegression ├── costFunctionReg.m ├── data1.txt ├── data2.txt ├── data_OneVsAll.mat ├── displayData.m ├── displayData_OneVsAll.m ├── fmincg.m ├── main.m ├── main_OneVsAll.m ├── mapFeature.m ├── oneVsAll.m ├── plotData.m ├── plotDecisionBoundary.m ├── predict.m ├── predict_OneVsAll.m ├── readme.md └── sigmoid.m ├── NeuralNetwork ├── checkNNGradients.m ├── computeNumericalGradient.m ├── data.mat ├── debugInitializeWeights.m ├── displayData.m ├── fmincg.m ├── main.m ├── nnCostFunction.m ├── predict.m ├── randInitializeWeights.m ├── readme.md ├── sigmoid.m ├── sigmoidGradient.m └── weights.mat ├── PCA ├── data1.mat ├── data_faces.mat ├── displayData.m ├── drawLine.m ├── featureNormalize.m ├── main.m ├── projectData.m └── recoverData.m ├── SVM ├── data.txt ├── data1.mat ├── data2.mat ├── data3.mat ├── dataset3Params.m ├── emailFeatures.m ├── emailSample1.txt ├── emailSample2.txt ├── gaussianKernel.m ├── getVocabList.m ├── linearKernel.m ├── main.m ├── main_spam.m ├── plotBoundary.m ├── plotData.m ├── porterStemmer.m ├── processEmail.m ├── readFile.m ├── readme.md ├── spamSample1.txt ├── spamSample2.txt ├── spamTest.mat ├── spamTrain.mat ├── svmPredict.m ├── svmTrain.m ├── test.m └── vocab.txt ├── images ├── AnomalyDetection_01.png ├── LinearRegression_01.png ├── LogisticRegression_01.png ├── LogisticRegression_02.png ├── PCA_01.png ├── PCA_02.png ├── svm_01.png ├── svm_02.png ├── svm_03.png └── svm_04.png └── readme.md /AnomalyDetection/data1.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/AnomalyDetection/data1.mat -------------------------------------------------------------------------------- /AnomalyDetection/data2.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/AnomalyDetection/data2.mat -------------------------------------------------------------------------------- /AnomalyDetection/estimateGaussian.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/AnomalyDetection/estimateGaussian.m -------------------------------------------------------------------------------- /AnomalyDetection/main.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/AnomalyDetection/main.m -------------------------------------------------------------------------------- /AnomalyDetection/multivariateGaussian.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/AnomalyDetection/multivariateGaussian.m -------------------------------------------------------------------------------- /AnomalyDetection/selectThreshold.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/AnomalyDetection/selectThreshold.m -------------------------------------------------------------------------------- /AnomalyDetection/visualizeFit.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/AnomalyDetection/visualizeFit.m -------------------------------------------------------------------------------- /CollaborativeFiltering/checkCostFunction.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/CollaborativeFiltering/checkCostFunction.m -------------------------------------------------------------------------------- /CollaborativeFiltering/cofiCostFunction.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/CollaborativeFiltering/cofiCostFunction.m -------------------------------------------------------------------------------- /CollaborativeFiltering/computeNumericalGradient.m: -------------------------------------------------------------------------------- 1 | function numgrad = computeNumericalGradient(J, theta) 2 | %COMPUTENUMERICALGRADIENT Computes the gradient using "finite differences" 3 | %and gives us a numerical estimate of the gradient. 4 | % numgrad = COMPUTENUMERICALGRADIENT(J, theta) computes the numerical 5 | % gradient of the function J around theta. Calling y = J(theta) should 6 | % return the function value at theta. 7 | 8 | % Notes: The following code implements numerical gradient checking, and 9 | % returns the numerical gradient.It sets numgrad(i) to (a numerical 10 | % approximation of) the partial derivative of J with respect to the 11 | % i-th input argument, evaluated at theta. (i.e., numgrad(i) should 12 | % be the (approximately) the partial derivative of J with respect 13 | % to theta(i).) 14 | % 15 | 16 | numgrad = zeros(size(theta)); 17 | perturb = zeros(size(theta)); 18 | e = 1e-4; 19 | for p = 1:numel(theta) 20 | % Set perturbation vector 21 | perturb(p) = e; 22 | loss1 = J(theta - perturb); 23 | loss2 = J(theta + perturb); 24 | % Compute Numerical Gradient 25 | numgrad(p) = (loss2 - loss1) / (2*e); 26 | perturb(p) = 0; 27 | end 28 | 29 | end 30 | -------------------------------------------------------------------------------- /CollaborativeFiltering/fmincg.m: -------------------------------------------------------------------------------- 1 | function [X, fX, i] = fmincg(f, X, options, P1, P2, P3, P4, P5) 2 | % Minimize a continuous differentialble multivariate function. Starting point 3 | % is given by "X" (D by 1), and the function named in the string "f", must 4 | % return a function value and a vector of partial derivatives. The Polack- 5 | % Ribiere flavour of conjugate gradients is used to compute search directions, 6 | % and a line search using quadratic and cubic polynomial approximations and the 7 | % Wolfe-Powell stopping criteria is used together with the slope ratio method 8 | % for guessing initial step sizes. Additionally a bunch of checks are made to 9 | % make sure that exploration is taking place and that extrapolation will not 10 | % be unboundedly large. The "length" gives the length of the run: if it is 11 | % positive, it gives the maximum number of line searches, if negative its 12 | % absolute gives the maximum allowed number of function evaluations. You can 13 | % (optionally) give "length" a second component, which will indicate the 14 | % reduction in function value to be expected in the first line-search (defaults 15 | % to 1.0). The function returns when either its length is up, or if no further 16 | % progress can be made (ie, we are at a minimum, or so close that due to 17 | % numerical problems, we cannot get any closer). If the function terminates 18 | % within a few iterations, it could be an indication that the function value 19 | % and derivatives are not consistent (ie, there may be a bug in the 20 | % implementation of your "f" function). The function returns the found 21 | % solution "X", a vector of function values "fX" indicating the progress made 22 | % and "i" the number of iterations (line searches or function evaluations, 23 | % depending on the sign of "length") used. 24 | % 25 | % Usage: [X, fX, i] = fmincg(f, X, options, P1, P2, P3, P4, P5) 26 | % 27 | % See also: checkgrad 28 | % 29 | % Copyright (C) 2001 and 2002 by Carl Edward Rasmussen. Date 2002-02-13 30 | % 31 | % 32 | % (C) Copyright 1999, 2000 & 2001, Carl Edward Rasmussen 33 | % 34 | % Permission is granted for anyone to copy, use, or modify these 35 | % programs and accompanying documents for purposes of research or 36 | % education, provided this copyright notice is retained, and note is 37 | % made of any changes that have been made. 38 | % 39 | % These programs and documents are distributed without any warranty, 40 | % express or implied. As the programs were written for research 41 | % purposes only, they have not been tested to the degree that would be 42 | % advisable in any important application. All use of these programs is 43 | % entirely at the user's own risk. 44 | % 45 | % [ml-class] Changes Made: 46 | % 1) Function name and argument specifications 47 | % 2) Output display 48 | % 49 | 50 | % Read options 51 | if exist('options', 'var') && ~isempty(options) && isfield(options, 'MaxIter') 52 | length = options.MaxIter; 53 | else 54 | length = 100; 55 | end 56 | 57 | 58 | RHO = 0.01; % a bunch of constants for line searches 59 | SIG = 0.5; % RHO and SIG are the constants in the Wolfe-Powell conditions 60 | INT = 0.1; % don't reevaluate within 0.1 of the limit of the current bracket 61 | EXT = 3.0; % extrapolate maximum 3 times the current bracket 62 | MAX = 20; % max 20 function evaluations per line search 63 | RATIO = 100; % maximum allowed slope ratio 64 | 65 | argstr = ['feval(f, X']; % compose string used to call function 66 | for i = 1:(nargin - 3) 67 | argstr = [argstr, ',P', int2str(i)]; 68 | end 69 | argstr = [argstr, ')']; 70 | 71 | if max(size(length)) == 2, red=length(2); length=length(1); else red=1; end 72 | S=['Iteration ']; 73 | 74 | i = 0; % zero the run length counter 75 | ls_failed = 0; % no previous line search has failed 76 | fX = []; 77 | [f1 df1] = eval(argstr); % get function value and gradient 78 | i = i + (length<0); % count epochs?! 79 | s = -df1; % search direction is steepest 80 | d1 = -s'*s; % this is the slope 81 | z1 = red/(1-d1); % initial step is red/(|s|+1) 82 | 83 | while i < abs(length) % while not finished 84 | i = i + (length>0); % count iterations?! 85 | 86 | X0 = X; f0 = f1; df0 = df1; % make a copy of current values 87 | X = X + z1*s; % begin line search 88 | [f2 df2] = eval(argstr); 89 | i = i + (length<0); % count epochs?! 90 | d2 = df2'*s; 91 | f3 = f1; d3 = d1; z3 = -z1; % initialize point 3 equal to point 1 92 | if length>0, M = MAX; else M = min(MAX, -length-i); end 93 | success = 0; limit = -1; % initialize quanteties 94 | while 1 95 | while ((f2 > f1+z1*RHO*d1) || (d2 > -SIG*d1)) && (M > 0) 96 | limit = z1; % tighten the bracket 97 | if f2 > f1 98 | z2 = z3 - (0.5*d3*z3*z3)/(d3*z3+f2-f3); % quadratic fit 99 | else 100 | A = 6*(f2-f3)/z3+3*(d2+d3); % cubic fit 101 | B = 3*(f3-f2)-z3*(d3+2*d2); 102 | z2 = (sqrt(B*B-A*d2*z3*z3)-B)/A; % numerical error possible - ok! 103 | end 104 | if isnan(z2) || isinf(z2) 105 | z2 = z3/2; % if we had a numerical problem then bisect 106 | end 107 | z2 = max(min(z2, INT*z3),(1-INT)*z3); % don't accept too close to limits 108 | z1 = z1 + z2; % update the step 109 | X = X + z2*s; 110 | [f2 df2] = eval(argstr); 111 | M = M - 1; i = i + (length<0); % count epochs?! 112 | d2 = df2'*s; 113 | z3 = z3-z2; % z3 is now relative to the location of z2 114 | end 115 | if f2 > f1+z1*RHO*d1 || d2 > -SIG*d1 116 | break; % this is a failure 117 | elseif d2 > SIG*d1 118 | success = 1; break; % success 119 | elseif M == 0 120 | break; % failure 121 | end 122 | A = 6*(f2-f3)/z3+3*(d2+d3); % make cubic extrapolation 123 | B = 3*(f3-f2)-z3*(d3+2*d2); 124 | z2 = -d2*z3*z3/(B+sqrt(B*B-A*d2*z3*z3)); % num. error possible - ok! 125 | if ~isreal(z2) || isnan(z2) || isinf(z2) || z2 < 0 % num prob or wrong sign? 126 | if limit < -0.5 % if we have no upper limit 127 | z2 = z1 * (EXT-1); % the extrapolate the maximum amount 128 | else 129 | z2 = (limit-z1)/2; % otherwise bisect 130 | end 131 | elseif (limit > -0.5) && (z2+z1 > limit) % extraplation beyond max? 132 | z2 = (limit-z1)/2; % bisect 133 | elseif (limit < -0.5) && (z2+z1 > z1*EXT) % extrapolation beyond limit 134 | z2 = z1*(EXT-1.0); % set to extrapolation limit 135 | elseif z2 < -z3*INT 136 | z2 = -z3*INT; 137 | elseif (limit > -0.5) && (z2 < (limit-z1)*(1.0-INT)) % too close to limit? 138 | z2 = (limit-z1)*(1.0-INT); 139 | end 140 | f3 = f2; d3 = d2; z3 = -z2; % set point 3 equal to point 2 141 | z1 = z1 + z2; X = X + z2*s; % update current estimates 142 | [f2 df2] = eval(argstr); 143 | M = M - 1; i = i + (length<0); % count epochs?! 144 | d2 = df2'*s; 145 | end % end of line search 146 | 147 | if success % if line search succeeded 148 | f1 = f2; fX = [fX' f1]'; 149 | fprintf('%s %4i | Cost: %4.6e\r', S, i, f1); 150 | s = (df2'*df2-df1'*df2)/(df1'*df1)*s - df2; % Polack-Ribiere direction 151 | tmp = df1; df1 = df2; df2 = tmp; % swap derivatives 152 | d2 = df1'*s; 153 | if d2 > 0 % new slope must be negative 154 | s = -df1; % otherwise use steepest direction 155 | d2 = -s'*s; 156 | end 157 | z1 = z1 * min(RATIO, d1/(d2-realmin)); % slope ratio but max RATIO 158 | d1 = d2; 159 | ls_failed = 0; % this line search did not fail 160 | else 161 | X = X0; f1 = f0; df1 = df0; % restore point from before failed line search 162 | if ls_failed || i > abs(length) % line search failed twice in a row 163 | break; % or we ran out of time, so we give up 164 | end 165 | tmp = df1; df1 = df2; df2 = tmp; % swap derivatives 166 | s = -df1; % try steepest 167 | d1 = -s'*s; 168 | z1 = 1/(1-d1); 169 | ls_failed = 1; % this line search failed 170 | end 171 | if exist('OCTAVE_VERSION') 172 | fflush(stdout); 173 | end 174 | end 175 | fprintf('\n'); 176 | -------------------------------------------------------------------------------- /CollaborativeFiltering/loadMovieList.m: -------------------------------------------------------------------------------- 1 | function movieList = loadMovieList() 2 | %GETMOVIELIST reads the fixed movie list in movie.txt and returns a 3 | %cell array of the words 4 | % movieList = GETMOVIELIST() reads the fixed movie list in movie.txt 5 | % and returns a cell array of the words in movieList. 6 | 7 | 8 | %% Read the fixed movieulary list 9 | fid = fopen('movie_ids.txt'); 10 | 11 | % Store all movies in cell array movie{} 12 | n = 1682; % Total number of movies 13 | 14 | movieList = cell(n, 1); 15 | for i = 1:n 16 | % Read line 17 | line = fgets(fid); 18 | % Word Index (can ignore since it will be = i) 19 | [idx, movieName] = strtok(line, ' '); 20 | % Actual Word 21 | movieList{i} = strtrim(movieName); 22 | end 23 | fclose(fid); 24 | 25 | end 26 | -------------------------------------------------------------------------------- /CollaborativeFiltering/main.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/CollaborativeFiltering/main.m -------------------------------------------------------------------------------- /CollaborativeFiltering/movieParams.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/CollaborativeFiltering/movieParams.mat -------------------------------------------------------------------------------- /CollaborativeFiltering/movie_ids.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/CollaborativeFiltering/movie_ids.txt -------------------------------------------------------------------------------- /CollaborativeFiltering/movies.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/CollaborativeFiltering/movies.mat -------------------------------------------------------------------------------- /CollaborativeFiltering/normalizeRatings.m: -------------------------------------------------------------------------------- 1 | function [Ynorm, Ymean] = normalizeRatings(Y, R) 2 | %NORMALIZERATINGS Preprocess data by subtracting mean rating for every 3 | %movie (every row) 4 | % [Ynorm, Ymean] = NORMALIZERATINGS(Y, R) normalized Y so that each movie 5 | % has a rating of 0 on average, and returns the mean rating in Ymean. 6 | % 7 | 8 | [m, n] = size(Y); 9 | Ymean = zeros(m, 1); 10 | Ynorm = zeros(size(Y)); 11 | for i = 1:m 12 | idx = find(R(i, :) == 1); 13 | Ymean(i) = mean(Y(i, idx)); 14 | Ynorm(i, idx) = Y(i, idx) - Ymean(i); 15 | end 16 | 17 | end 18 | -------------------------------------------------------------------------------- /K-Means/bird.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/K-Means/bird.mat -------------------------------------------------------------------------------- /K-Means/bird.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/K-Means/bird.png -------------------------------------------------------------------------------- /K-Means/computeCentroids.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/K-Means/computeCentroids.m -------------------------------------------------------------------------------- /K-Means/data.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/K-Means/data.mat -------------------------------------------------------------------------------- /K-Means/drawLine.m: -------------------------------------------------------------------------------- 1 | function drawLine(p1, p2, varargin) 2 | %DRAWLINE Draws a line from point p1 to point p2 3 | % DRAWLINE(p1, p2) Draws a line from point p1 to point p2 and holds the 4 | % current figure 5 | 6 | plot([p1(1) p2(1)], [p1(2) p2(2)], varargin{:}); 7 | 8 | end -------------------------------------------------------------------------------- /K-Means/findClosestCentroids.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/K-Means/findClosestCentroids.m -------------------------------------------------------------------------------- /K-Means/kMeansInitCentroids.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/K-Means/kMeansInitCentroids.m -------------------------------------------------------------------------------- /K-Means/main.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/K-Means/main.m -------------------------------------------------------------------------------- /K-Means/plotDataPoints.m: -------------------------------------------------------------------------------- 1 | function plotDataPoints(X, idx, K) 2 | %PLOTDATAPOINTS plots data points in X, coloring them so that those with the same 3 | %index assignments in idx have the same color 4 | % PLOTDATAPOINTS(X, idx, K) plots data points in X, coloring them so that those 5 | % with the same index assignments in idx have the same color 6 | 7 | % Create palette 8 | palette = hsv(K + 1); 9 | colors = palette(idx, :); 10 | 11 | % Plot the data 12 | scatter(X(:,1), X(:,2), 15, colors); 13 | 14 | end 15 | -------------------------------------------------------------------------------- /K-Means/plotProgresskMeans.m: -------------------------------------------------------------------------------- 1 | function plotProgresskMeans(X, centroids, previous, idx, K, i) 2 | %PLOTPROGRESSKMEANS is a helper function that displays the progress of 3 | %k-Means as it is running. It is intended for use only with 2D data. 4 | % PLOTPROGRESSKMEANS(X, centroids, previous, idx, K, i) plots the data 5 | % points with colors assigned to each centroid. With the previous 6 | % centroids, it also plots a line between the previous locations and 7 | % current locations of the centroids. 8 | % 9 | 10 | % Plot the examples 11 | plotDataPoints(X, idx, K); 12 | 13 | % Plot the centroids as black x's 14 | plot(centroids(:,1), centroids(:,2), 'x', ... 15 | 'MarkerEdgeColor','k', ... 16 | 'MarkerSize', 10, 'LineWidth', 3); 17 | 18 | % Plot the history of the centroids with lines 19 | for j=1:size(centroids,1) 20 | drawLine(centroids(j, :), previous(j, :)); 21 | end 22 | 23 | % Title 24 | title(sprintf('Iteration number %d', i)) 25 | 26 | end 27 | 28 | -------------------------------------------------------------------------------- /K-Means/readme.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/K-Means/readme.md -------------------------------------------------------------------------------- /K-Means/runkMeans.m: -------------------------------------------------------------------------------- 1 | function [centroids, idx] = runkMeans(X, initial_centroids, ... 2 | max_iters, plot_progress) 3 | %RUNKMEANS runs the K-Means algorithm on data matrix X, where each row of X 4 | %is a single example 5 | % [centroids, idx] = RUNKMEANS(X, initial_centroids, max_iters, ... 6 | % plot_progress) runs the K-Means algorithm on data matrix X, where each 7 | % row of X is a single example. It uses initial_centroids used as the 8 | % initial centroids. max_iters specifies the total number of interactions 9 | % of K-Means to execute. plot_progress is a true/false flag that 10 | % indicates if the function should also plot its progress as the 11 | % learning happens. This is set to false by default. runkMeans returns 12 | % centroids, a Kxn matrix of the computed centroids and idx, a m x 1 13 | % vector of centroid assignments (i.e. each entry in range [1..K]) 14 | % 15 | 16 | % Set default value for plot progress 17 | if ~exist('plot_progress', 'var') || isempty(plot_progress) 18 | plot_progress = false; 19 | end 20 | 21 | % Plot the data if we are plotting progress 22 | if plot_progress 23 | figure; 24 | hold on; 25 | end 26 | 27 | % Initialize values 28 | [m n] = size(X); 29 | K = size(initial_centroids, 1); 30 | centroids = initial_centroids; 31 | previous_centroids = centroids; 32 | idx = zeros(m, 1); 33 | 34 | % Run K-Means 35 | for i=1:max_iters 36 | 37 | % Output progress 38 | fprintf('K-Means iteration %d/%d...\n', i, max_iters); 39 | if exist('OCTAVE_VERSION') 40 | fflush(stdout); 41 | end 42 | 43 | % For each example in X, assign it to the closest centroid 44 | idx = findClosestCentroids(X, centroids); 45 | 46 | % Optionally, plot progress here 47 | if plot_progress 48 | plotProgresskMeans(X, centroids, previous_centroids, idx, K, i); 49 | previous_centroids = centroids; 50 | fprintf('Press enter to continue.\n'); 51 | pause; 52 | end 53 | 54 | % Given the memberships, compute new centroids 55 | centroids = computeCentroids(X, idx, K); 56 | end 57 | 58 | % Hold off if we are plotting progress 59 | if plot_progress 60 | hold off; 61 | end 62 | 63 | end 64 | 65 | -------------------------------------------------------------------------------- /LinearRegression/computerCost.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/LinearRegression/computerCost.m -------------------------------------------------------------------------------- /LinearRegression/data.txt: -------------------------------------------------------------------------------- 1 | 2104,3,399900 2 | 1600,3,329900 3 | 2400,3,369000 4 | 1416,2,232000 5 | 3000,4,539900 6 | 1985,4,299900 7 | 1534,3,314900 8 | 1427,3,198999 9 | 1380,3,212000 10 | 1494,3,242500 11 | 1940,4,239999 12 | 2000,3,347000 13 | 1890,3,329999 14 | 4478,5,699900 15 | 1268,3,259900 16 | 2300,4,449900 17 | 1320,2,299900 18 | 1236,3,199900 19 | 2609,4,499998 20 | 3031,4,599000 21 | 1767,3,252900 22 | 1888,2,255000 23 | 1604,3,242900 24 | 1962,4,259900 25 | 3890,3,573900 26 | 1100,3,249900 27 | 1458,3,464500 28 | 2526,3,469000 29 | 2200,3,475000 30 | 2637,3,299900 31 | 1839,2,349900 32 | 1000,1,169900 33 | 2040,4,314900 34 | 3137,3,579900 35 | 1811,4,285900 36 | 1437,3,249900 37 | 1239,3,229900 38 | 2132,4,345000 39 | 4215,4,549000 40 | 2162,4,287000 41 | 1664,2,368500 42 | 2238,3,329900 43 | 2567,4,314000 44 | 1200,3,299000 45 | 852,2,179900 46 | 1852,4,299900 47 | 1203,3,239500 48 | -------------------------------------------------------------------------------- /LinearRegression/featureNormalize.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/LinearRegression/featureNormalize.m -------------------------------------------------------------------------------- /LinearRegression/gradientDescent.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/LinearRegression/gradientDescent.m -------------------------------------------------------------------------------- /LinearRegression/main.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/LinearRegression/main.m -------------------------------------------------------------------------------- /LinearRegression/normalEquations.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/LinearRegression/normalEquations.m -------------------------------------------------------------------------------- /LinearRegression/readme.md: -------------------------------------------------------------------------------- 1 | 线性回归算法 2 | ======= 3 | ### 一、文件说明 4 | - [main.m][1.1] 5 | - 主运行程序 6 | - [featureNormalize.m][1.2] 7 | - 特征向量归一化函数 8 | - [gradientDescent.m][1.3] 9 | - 梯度下降求解函数 10 | - [computerCost.m][1.4] 11 | - 计算代价J函数 12 | - [normalEquations.m][1.5] 13 | - 正规方程求解函数 14 | 15 | ### 二、重要文件说明 16 | - main.m 17 | - 注意合理修改学习速率参数`alpha` 18 | - 归一化数据，预测时也需要归一化（因为是归一化之后求出的theta参数） 19 | 20 | ### 三、测试数据 21 | - 代价函数随迭代次数收敛 22 | ![线性回归][3.1] 23 | 24 | 25 | 26 | 27 | 28 | [1.1]:main.m 29 | [1.2]:featureNormalize.m 30 | [1.3]:gradientDescent.m 31 | [1.4]:computerCost.m 32 | [1.5]:normalEquations.m 33 | 34 | 35 | [3.1]: ../images/LinearRegression_01.png "LinearRegression_01.png" 36 | -------------------------------------------------------------------------------- /LogisticRegression/costFunctionReg.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/LogisticRegression/costFunctionReg.m -------------------------------------------------------------------------------- /LogisticRegression/data1.txt: -------------------------------------------------------------------------------- 1 | 34.62365962451697,78.0246928153624,0 2 | 30.28671076822607,43.89499752400101,0 3 | 35.84740876993872,72.90219802708364,0 4 | 60.18259938620976,86.30855209546826,1 5 | 79.0327360507101,75.3443764369103,1 6 | 45.08327747668339,56.3163717815305,0 7 | 61.10666453684766,96.51142588489624,1 8 | 75.02474556738889,46.55401354116538,1 9 | 76.09878670226257,87.42056971926803,1 10 | 84.43281996120035,43.53339331072109,1 11 | 95.86155507093572,38.22527805795094,0 12 | 75.01365838958247,30.60326323428011,0 13 | 82.30705337399482,76.48196330235604,1 14 | 69.36458875970939,97.71869196188608,1 15 | 39.53833914367223,76.03681085115882,0 16 | 53.9710521485623,89.20735013750205,1 17 | 69.07014406283025,52.74046973016765,1 18 | 67.94685547711617,46.67857410673128,0 19 | 70.66150955499435,92.92713789364831,1 20 | 76.97878372747498,47.57596364975532,1 21 | 67.37202754570876,42.83843832029179,0 22 | 89.67677575072079,65.79936592745237,1 23 | 50.534788289883,48.85581152764205,0 24 | 34.21206097786789,44.20952859866288,0 25 | 77.9240914545704,68.9723599933059,1 26 | 62.27101367004632,69.95445795447587,1 27 | 80.1901807509566,44.82162893218353,1 28 | 93.114388797442,38.80067033713209,0 29 | 61.83020602312595,50.25610789244621,0 30 | 38.78580379679423,64.99568095539578,0 31 | 61.379289447425,72.80788731317097,1 32 | 85.40451939411645,57.05198397627122,1 33 | 52.10797973193984,63.12762376881715,0 34 | 52.04540476831827,69.43286012045222,1 35 | 40.23689373545111,71.16774802184875,0 36 | 54.63510555424817,52.21388588061123,0 37 | 33.91550010906887,98.86943574220611,0 38 | 64.17698887494485,80.90806058670817,1 39 | 74.78925295941542,41.57341522824434,0 40 | 34.1836400264419,75.2377203360134,0 41 | 83.90239366249155,56.30804621605327,1 42 | 51.54772026906181,46.85629026349976,0 43 | 94.44336776917852,65.56892160559052,1 44 | 82.36875375713919,40.61825515970618,0 45 | 51.04775177128865,45.82270145776001,0 46 | 62.22267576120188,52.06099194836679,0 47 | 77.19303492601364,70.45820000180959,1 48 | 97.77159928000232,86.7278223300282,1 49 | 62.07306379667647,96.76882412413983,1 50 | 91.56497449807442,88.69629254546599,1 51 | 79.94481794066932,74.16311935043758,1 52 | 99.2725269292572,60.99903099844988,1 53 | 90.54671411399852,43.39060180650027,1 54 | 34.52451385320009,60.39634245837173,0 55 | 50.2864961189907,49.80453881323059,0 56 | 49.58667721632031,59.80895099453265,0 57 | 97.64563396007767,68.86157272420604,1 58 | 32.57720016809309,95.59854761387875,0 59 | 74.24869136721598,69.82457122657193,1 60 | 71.79646205863379,78.45356224515052,1 61 | 75.3956114656803,85.75993667331619,1 62 | 35.28611281526193,47.02051394723416,0 63 | 56.25381749711624,39.26147251058019,0 64 | 30.05882244669796,49.59297386723685,0 65 | 44.66826172480893,66.45008614558913,0 66 | 66.56089447242954,41.09209807936973,0 67 | 40.45755098375164,97.53518548909936,1 68 | 49.07256321908844,51.88321182073966,0 69 | 80.27957401466998,92.11606081344084,1 70 | 66.74671856944039,60.99139402740988,1 71 | 32.72283304060323,43.30717306430063,0 72 | 64.0393204150601,78.03168802018232,1 73 | 72.34649422579923,96.22759296761404,1 74 | 60.45788573918959,73.09499809758037,1 75 | 58.84095621726802,75.85844831279042,1 76 | 99.82785779692128,72.36925193383885,1 77 | 47.26426910848174,88.47586499559782,1 78 | 50.45815980285988,75.80985952982456,1 79 | 60.45555629271532,42.50840943572217,0 80 | 82.22666157785568,42.71987853716458,0 81 | 88.9138964166533,69.80378889835472,1 82 | 94.83450672430196,45.69430680250754,1 83 | 67.31925746917527,66.58935317747915,1 84 | 57.23870631569862,59.51428198012956,1 85 | 80.36675600171273,90.96014789746954,1 86 | 68.46852178591112,85.59430710452014,1 87 | 42.0754545384731,78.84478600148043,0 88 | 75.47770200533905,90.42453899753964,1 89 | 78.63542434898018,96.64742716885644,1 90 | 52.34800398794107,60.76950525602592,0 91 | 94.09433112516793,77.15910509073893,1 92 | 90.44855097096364,87.50879176484702,1 93 | 55.48216114069585,35.57070347228866,0 94 | 74.49269241843041,84.84513684930135,1 95 | 89.84580670720979,45.35828361091658,1 96 | 83.48916274498238,48.38028579728175,1 97 | 42.2617008099817,87.10385094025457,1 98 | 99.31500880510394,68.77540947206617,1 99 | 55.34001756003703,64.9319380069486,1 100 | 74.77589300092767,89.52981289513276,1 101 | -------------------------------------------------------------------------------- /LogisticRegression/data2.txt: -------------------------------------------------------------------------------- 1 | 0.051267,0.69956,1 2 | -0.092742,0.68494,1 3 | -0.21371,0.69225,1 4 | -0.375,0.50219,1 5 | -0.51325,0.46564,1 6 | -0.52477,0.2098,1 7 | -0.39804,0.034357,1 8 | -0.30588,-0.19225,1 9 | 0.016705,-0.40424,1 10 | 0.13191,-0.51389,1 11 | 0.38537,-0.56506,1 12 | 0.52938,-0.5212,1 13 | 0.63882,-0.24342,1 14 | 0.73675,-0.18494,1 15 | 0.54666,0.48757,1 16 | 0.322,0.5826,1 17 | 0.16647,0.53874,1 18 | -0.046659,0.81652,1 19 | -0.17339,0.69956,1 20 | -0.47869,0.63377,1 21 | -0.60541,0.59722,1 22 | -0.62846,0.33406,1 23 | -0.59389,0.005117,1 24 | -0.42108,-0.27266,1 25 | -0.11578,-0.39693,1 26 | 0.20104,-0.60161,1 27 | 0.46601,-0.53582,1 28 | 0.67339,-0.53582,1 29 | -0.13882,0.54605,1 30 | -0.29435,0.77997,1 31 | -0.26555,0.96272,1 32 | -0.16187,0.8019,1 33 | -0.17339,0.64839,1 34 | -0.28283,0.47295,1 35 | -0.36348,0.31213,1 36 | -0.30012,0.027047,1 37 | -0.23675,-0.21418,1 38 | -0.06394,-0.18494,1 39 | 0.062788,-0.16301,1 40 | 0.22984,-0.41155,1 41 | 0.2932,-0.2288,1 42 | 0.48329,-0.18494,1 43 | 0.64459,-0.14108,1 44 | 0.46025,0.012427,1 45 | 0.6273,0.15863,1 46 | 0.57546,0.26827,1 47 | 0.72523,0.44371,1 48 | 0.22408,0.52412,1 49 | 0.44297,0.67032,1 50 | 0.322,0.69225,1 51 | 0.13767,0.57529,1 52 | -0.0063364,0.39985,1 53 | -0.092742,0.55336,1 54 | -0.20795,0.35599,1 55 | -0.20795,0.17325,1 56 | -0.43836,0.21711,1 57 | -0.21947,-0.016813,1 58 | -0.13882,-0.27266,1 59 | 0.18376,0.93348,0 60 | 0.22408,0.77997,0 61 | 0.29896,0.61915,0 62 | 0.50634,0.75804,0 63 | 0.61578,0.7288,0 64 | 0.60426,0.59722,0 65 | 0.76555,0.50219,0 66 | 0.92684,0.3633,0 67 | 0.82316,0.27558,0 68 | 0.96141,0.085526,0 69 | 0.93836,0.012427,0 70 | 0.86348,-0.082602,0 71 | 0.89804,-0.20687,0 72 | 0.85196,-0.36769,0 73 | 0.82892,-0.5212,0 74 | 0.79435,-0.55775,0 75 | 0.59274,-0.7405,0 76 | 0.51786,-0.5943,0 77 | 0.46601,-0.41886,0 78 | 0.35081,-0.57968,0 79 | 0.28744,-0.76974,0 80 | 0.085829,-0.75512,0 81 | 0.14919,-0.57968,0 82 | -0.13306,-0.4481,0 83 | -0.40956,-0.41155,0 84 | -0.39228,-0.25804,0 85 | -0.74366,-0.25804,0 86 | -0.69758,0.041667,0 87 | -0.75518,0.2902,0 88 | -0.69758,0.68494,0 89 | -0.4038,0.70687,0 90 | -0.38076,0.91886,0 91 | -0.50749,0.90424,0 92 | -0.54781,0.70687,0 93 | 0.10311,0.77997,0 94 | 0.057028,0.91886,0 95 | -0.10426,0.99196,0 96 | -0.081221,1.1089,0 97 | 0.28744,1.087,0 98 | 0.39689,0.82383,0 99 | 0.63882,0.88962,0 100 | 0.82316,0.66301,0 101 | 0.67339,0.64108,0 102 | 1.0709,0.10015,0 103 | -0.046659,-0.57968,0 104 | -0.23675,-0.63816,0 105 | -0.15035,-0.36769,0 106 | -0.49021,-0.3019,0 107 | -0.46717,-0.13377,0 108 | -0.28859,-0.060673,0 109 | -0.61118,-0.067982,0 110 | -0.66302,-0.21418,0 111 | -0.59965,-0.41886,0 112 | -0.72638,-0.082602,0 113 | -0.83007,0.31213,0 114 | -0.72062,0.53874,0 115 | -0.59389,0.49488,0 116 | -0.48445,0.99927,0 117 | -0.0063364,0.99927,0 118 | 0.63265,-0.030612,0 119 | -------------------------------------------------------------------------------- /LogisticRegression/data_OneVsAll.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/LogisticRegression/data_OneVsAll.mat -------------------------------------------------------------------------------- /LogisticRegression/displayData.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/LogisticRegression/displayData.m -------------------------------------------------------------------------------- /LogisticRegression/displayData_OneVsAll.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/LogisticRegression/displayData_OneVsAll.m -------------------------------------------------------------------------------- /LogisticRegression/fmincg.m: -------------------------------------------------------------------------------- 1 | function [X, fX, i] = fmincg(f, X, options, P1, P2, P3, P4, P5) 2 | % Minimize a continuous differentialble multivariate function. Starting point 3 | % is given by "X" (D by 1), and the function named in the string "f", must 4 | % return a function value and a vector of partial derivatives. The Polack- 5 | % Ribiere flavour of conjugate gradients is used to compute search directions, 6 | % and a line search using quadratic and cubic polynomial approximations and the 7 | % Wolfe-Powell stopping criteria is used together with the slope ratio method 8 | % for guessing initial step sizes. Additionally a bunch of checks are made to 9 | % make sure that exploration is taking place and that extrapolation will not 10 | % be unboundedly large. The "length" gives the length of the run: if it is 11 | % positive, it gives the maximum number of line searches, if negative its 12 | % absolute gives the maximum allowed number of function evaluations. You can 13 | % (optionally) give "length" a second component, which will indicate the 14 | % reduction in function value to be expected in the first line-search (defaults 15 | % to 1.0). The function returns when either its length is up, or if no further 16 | % progress can be made (ie, we are at a minimum, or so close that due to 17 | % numerical problems, we cannot get any closer). If the function terminates 18 | % within a few iterations, it could be an indication that the function value 19 | % and derivatives are not consistent (ie, there may be a bug in the 20 | % implementation of your "f" function). The function returns the found 21 | % solution "X", a vector of function values "fX" indicating the progress made 22 | % and "i" the number of iterations (line searches or function evaluations, 23 | % depending on the sign of "length") used. 24 | % 25 | % Usage: [X, fX, i] = fmincg(f, X, options, P1, P2, P3, P4, P5) 26 | % 27 | % See also: checkgrad 28 | % 29 | % Copyright (C) 2001 and 2002 by Carl Edward Rasmussen. Date 2002-02-13 30 | % 31 | % 32 | % (C) Copyright 1999, 2000 & 2001, Carl Edward Rasmussen 33 | % 34 | % Permission is granted for anyone to copy, use, or modify these 35 | % programs and accompanying documents for purposes of research or 36 | % education, provided this copyright notice is retained, and note is 37 | % made of any changes that have been made. 38 | % 39 | % These programs and documents are distributed without any warranty, 40 | % express or implied. As the programs were written for research 41 | % purposes only, they have not been tested to the degree that would be 42 | % advisable in any important application. All use of these programs is 43 | % entirely at the user's own risk. 44 | % 45 | % [ml-class] Changes Made: 46 | % 1) Function name and argument specifications 47 | % 2) Output display 48 | % 49 | 50 | % Read options 51 | if exist('options', 'var') && ~isempty(options) && isfield(options, 'MaxIter') 52 | length = options.MaxIter; 53 | else 54 | length = 100; 55 | end 56 | 57 | 58 | RHO = 0.01; % a bunch of constants for line searches 59 | SIG = 0.5; % RHO and SIG are the constants in the Wolfe-Powell conditions 60 | INT = 0.1; % don't reevaluate within 0.1 of the limit of the current bracket 61 | EXT = 3.0; % extrapolate maximum 3 times the current bracket 62 | MAX = 20; % max 20 function evaluations per line search 63 | RATIO = 100; % maximum allowed slope ratio 64 | 65 | argstr = ['feval(f, X']; % compose string used to call function 66 | for i = 1:(nargin - 3) 67 | argstr = [argstr, ',P', int2str(i)]; 68 | end 69 | argstr = [argstr, ')']; 70 | 71 | if max(size(length)) == 2, red=length(2); length=length(1); else red=1; end 72 | S=['Iteration ']; 73 | 74 | i = 0; % zero the run length counter 75 | ls_failed = 0; % no previous line search has failed 76 | fX = []; 77 | [f1 df1] = eval(argstr); % get function value and gradient 78 | i = i + (length<0); % count epochs?! 79 | s = -df1; % search direction is steepest 80 | d1 = -s'*s; % this is the slope 81 | z1 = red/(1-d1); % initial step is red/(|s|+1) 82 | 83 | while i < abs(length) % while not finished 84 | i = i + (length>0); % count iterations?! 85 | 86 | X0 = X; f0 = f1; df0 = df1; % make a copy of current values 87 | X = X + z1*s; % begin line search 88 | [f2 df2] = eval(argstr); 89 | i = i + (length<0); % count epochs?! 90 | d2 = df2'*s; 91 | f3 = f1; d3 = d1; z3 = -z1; % initialize point 3 equal to point 1 92 | if length>0, M = MAX; else M = min(MAX, -length-i); end 93 | success = 0; limit = -1; % initialize quanteties 94 | while 1 95 | while ((f2 > f1+z1*RHO*d1) || (d2 > -SIG*d1)) && (M > 0) 96 | limit = z1; % tighten the bracket 97 | if f2 > f1 98 | z2 = z3 - (0.5*d3*z3*z3)/(d3*z3+f2-f3); % quadratic fit 99 | else 100 | A = 6*(f2-f3)/z3+3*(d2+d3); % cubic fit 101 | B = 3*(f3-f2)-z3*(d3+2*d2); 102 | z2 = (sqrt(B*B-A*d2*z3*z3)-B)/A; % numerical error possible - ok! 103 | end 104 | if isnan(z2) || isinf(z2) 105 | z2 = z3/2; % if we had a numerical problem then bisect 106 | end 107 | z2 = max(min(z2, INT*z3),(1-INT)*z3); % don't accept too close to limits 108 | z1 = z1 + z2; % update the step 109 | X = X + z2*s; 110 | [f2 df2] = eval(argstr); 111 | M = M - 1; i = i + (length<0); % count epochs?! 112 | d2 = df2'*s; 113 | z3 = z3-z2; % z3 is now relative to the location of z2 114 | end 115 | if f2 > f1+z1*RHO*d1 || d2 > -SIG*d1 116 | break; % this is a failure 117 | elseif d2 > SIG*d1 118 | success = 1; break; % success 119 | elseif M == 0 120 | break; % failure 121 | end 122 | A = 6*(f2-f3)/z3+3*(d2+d3); % make cubic extrapolation 123 | B = 3*(f3-f2)-z3*(d3+2*d2); 124 | z2 = -d2*z3*z3/(B+sqrt(B*B-A*d2*z3*z3)); % num. error possible - ok! 125 | if ~isreal(z2) || isnan(z2) || isinf(z2) || z2 < 0 % num prob or wrong sign? 126 | if limit < -0.5 % if we have no upper limit 127 | z2 = z1 * (EXT-1); % the extrapolate the maximum amount 128 | else 129 | z2 = (limit-z1)/2; % otherwise bisect 130 | end 131 | elseif (limit > -0.5) && (z2+z1 > limit) % extraplation beyond max? 132 | z2 = (limit-z1)/2; % bisect 133 | elseif (limit < -0.5) && (z2+z1 > z1*EXT) % extrapolation beyond limit 134 | z2 = z1*(EXT-1.0); % set to extrapolation limit 135 | elseif z2 < -z3*INT 136 | z2 = -z3*INT; 137 | elseif (limit > -0.5) && (z2 < (limit-z1)*(1.0-INT)) % too close to limit? 138 | z2 = (limit-z1)*(1.0-INT); 139 | end 140 | f3 = f2; d3 = d2; z3 = -z2; % set point 3 equal to point 2 141 | z1 = z1 + z2; X = X + z2*s; % update current estimates 142 | [f2 df2] = eval(argstr); 143 | M = M - 1; i = i + (length<0); % count epochs?! 144 | d2 = df2'*s; 145 | end % end of line search 146 | 147 | if success % if line search succeeded 148 | f1 = f2; fX = [fX' f1]'; 149 | fprintf('%s %4i | Cost: %4.6e\r', S, i, f1); 150 | s = (df2'*df2-df1'*df2)/(df1'*df1)*s - df2; % Polack-Ribiere direction 151 | tmp = df1; df1 = df2; df2 = tmp; % swap derivatives 152 | d2 = df1'*s; 153 | if d2 > 0 % new slope must be negative 154 | s = -df1; % otherwise use steepest direction 155 | d2 = -s'*s; 156 | end 157 | z1 = z1 * min(RATIO, d1/(d2-realmin)); % slope ratio but max RATIO 158 | d1 = d2; 159 | ls_failed = 0; % this line search did not fail 160 | else 161 | X = X0; f1 = f0; df1 = df0; % restore point from before failed line search 162 | if ls_failed || i > abs(length) % line search failed twice in a row 163 | break; % or we ran out of time, so we give up 164 | end 165 | tmp = df1; df1 = df2; df2 = tmp; % swap derivatives 166 | s = -df1; % try steepest 167 | d1 = -s'*s; 168 | z1 = 1/(1-d1); 169 | ls_failed = 1; % this line search failed 170 | end 171 | if exist('OCTAVE_VERSION') 172 | fflush(stdout); 173 | end 174 | end 175 | fprintf('\n'); 176 | -------------------------------------------------------------------------------- /LogisticRegression/main.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/LogisticRegression/main.m -------------------------------------------------------------------------------- /LogisticRegression/main_OneVsAll.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/LogisticRegression/main_OneVsAll.m -------------------------------------------------------------------------------- /LogisticRegression/mapFeature.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/LogisticRegression/mapFeature.m -------------------------------------------------------------------------------- /LogisticRegression/oneVsAll.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/LogisticRegression/oneVsAll.m -------------------------------------------------------------------------------- /LogisticRegression/plotData.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/LogisticRegression/plotData.m -------------------------------------------------------------------------------- /LogisticRegression/plotDecisionBoundary.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/LogisticRegression/plotDecisionBoundary.m -------------------------------------------------------------------------------- /LogisticRegression/predict.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/LogisticRegression/predict.m -------------------------------------------------------------------------------- /LogisticRegression/predict_OneVsAll.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/LogisticRegression/predict_OneVsAll.m -------------------------------------------------------------------------------- /LogisticRegression/readme.md: -------------------------------------------------------------------------------- 1 | 逻辑回归算法 2 | ============= 3 | ### 一、文件说明 4 | - [main.m][1.1] 5 | - 主运行程序 6 | - [sigmoid.m][1.2] 7 | - S形函数 8 | - [plotData.m][1.3] 9 | - 作图函数（主要是两个变量） 10 | - [costFunctionReg.m][1.4] 11 | - 计算代价函数J和梯度grad 12 | - [plotDecisionBoundary.m][1.5] 13 | - 画决策边界 14 | - [predict.m][1.6] 15 | - 根据得到的假设函数再次预测训练集，输出准确度 16 | 17 | ### 二、重要文件说明 18 | - plotDecisionBoundary.m 19 | 决策边界的u,v坐标的范围需要根据实际数据修改 20 | - mapFeature.m 21 | 映射多项式函数，需要根据实际情况degree，防止过拟合（尽管已经采用正则化防止过拟合了） 22 | 23 | ### 三、测试数据 24 | - data1.txt文件 25 | ![逻辑回归][3.1] 26 | - data2.txt文件 27 | ![逻辑回归][3.2] 28 | 29 | 30 | 31 | [1.1]:main.m 32 | [1.2]:sigmoid.m 33 | [1.3]:plotData.m 34 | [1.4]:costFunctionReg.m 35 | [1.5]:plotDecisionBoundary.m 36 | [1.6]:predict.m 37 | 38 | [3.1]: ../images/LogisticRegression_01.png "LogisticRegression_01.png" 39 | [3.2]: ../images/LogisticRegression_02.png "LogisticRegression_02.png" 40 | -------------------------------------------------------------------------------- /LogisticRegression/sigmoid.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/LogisticRegression/sigmoid.m -------------------------------------------------------------------------------- /NeuralNetwork/checkNNGradients.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/NeuralNetwork/checkNNGradients.m -------------------------------------------------------------------------------- /NeuralNetwork/computeNumericalGradient.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/NeuralNetwork/computeNumericalGradient.m -------------------------------------------------------------------------------- /NeuralNetwork/data.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/NeuralNetwork/data.mat -------------------------------------------------------------------------------- /NeuralNetwork/debugInitializeWeights.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/NeuralNetwork/debugInitializeWeights.m -------------------------------------------------------------------------------- /NeuralNetwork/displayData.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/NeuralNetwork/displayData.m -------------------------------------------------------------------------------- /NeuralNetwork/fmincg.m: -------------------------------------------------------------------------------- 1 | function [X, fX, i] = fmincg(f, X, options, P1, P2, P3, P4, P5) 2 | % Minimize a continuous differentialble multivariate function. Starting point 3 | % is given by "X" (D by 1), and the function named in the string "f", must 4 | % return a function value and a vector of partial derivatives. The Polack- 5 | % Ribiere flavour of conjugate gradients is used to compute search directions, 6 | % and a line search using quadratic and cubic polynomial approximations and the 7 | % Wolfe-Powell stopping criteria is used together with the slope ratio method 8 | % for guessing initial step sizes. Additionally a bunch of checks are made to 9 | % make sure that exploration is taking place and that extrapolation will not 10 | % be unboundedly large. The "length" gives the length of the run: if it is 11 | % positive, it gives the maximum number of line searches, if negative its 12 | % absolute gives the maximum allowed number of function evaluations. You can 13 | % (optionally) give "length" a second component, which will indicate the 14 | % reduction in function value to be expected in the first line-search (defaults 15 | % to 1.0). The function returns when either its length is up, or if no further 16 | % progress can be made (ie, we are at a minimum, or so close that due to 17 | % numerical problems, we cannot get any closer). If the function terminates 18 | % within a few iterations, it could be an indication that the function value 19 | % and derivatives are not consistent (ie, there may be a bug in the 20 | % implementation of your "f" function). The function returns the found 21 | % solution "X", a vector of function values "fX" indicating the progress made 22 | % and "i" the number of iterations (line searches or function evaluations, 23 | % depending on the sign of "length") used. 24 | % 25 | % Usage: [X, fX, i] = fmincg(f, X, options, P1, P2, P3, P4, P5) 26 | % 27 | % See also: checkgrad 28 | % 29 | % Copyright (C) 2001 and 2002 by Carl Edward Rasmussen. Date 2002-02-13 30 | % 31 | % 32 | % (C) Copyright 1999, 2000 & 2001, Carl Edward Rasmussen 33 | % 34 | % Permission is granted for anyone to copy, use, or modify these 35 | % programs and accompanying documents for purposes of research or 36 | % education, provided this copyright notice is retained, and note is 37 | % made of any changes that have been made. 38 | % 39 | % These programs and documents are distributed without any warranty, 40 | % express or implied. As the programs were written for research 41 | % purposes only, they have not been tested to the degree that would be 42 | % advisable in any important application. All use of these programs is 43 | % entirely at the user's own risk. 44 | % 45 | % [ml-class] Changes Made: 46 | % 1) Function name and argument specifications 47 | % 2) Output display 48 | % 49 | 50 | % Read options 51 | if exist('options', 'var') && ~isempty(options) && isfield(options, 'MaxIter') 52 | length = options.MaxIter; 53 | else 54 | length = 100; 55 | end 56 | 57 | 58 | RHO = 0.01; % a bunch of constants for line searches 59 | SIG = 0.5; % RHO and SIG are the constants in the Wolfe-Powell conditions 60 | INT = 0.1; % don't reevaluate within 0.1 of the limit of the current bracket 61 | EXT = 3.0; % extrapolate maximum 3 times the current bracket 62 | MAX = 20; % max 20 function evaluations per line search 63 | RATIO = 100; % maximum allowed slope ratio 64 | 65 | argstr = ['feval(f, X']; % compose string used to call function 66 | for i = 1:(nargin - 3) 67 | argstr = [argstr, ',P', int2str(i)]; 68 | end 69 | argstr = [argstr, ')']; 70 | 71 | if max(size(length)) == 2, red=length(2); length=length(1); else red=1; end 72 | S=['Iteration ']; 73 | 74 | i = 0; % zero the run length counter 75 | ls_failed = 0; % no previous line search has failed 76 | fX = []; 77 | [f1 df1] = eval(argstr); % get function value and gradient 78 | i = i + (length<0); % count epochs?! 79 | s = -df1; % search direction is steepest 80 | d1 = -s'*s; % this is the slope 81 | z1 = red/(1-d1); % initial step is red/(|s|+1) 82 | 83 | while i < abs(length) % while not finished 84 | i = i + (length>0); % count iterations?! 85 | 86 | X0 = X; f0 = f1; df0 = df1; % make a copy of current values 87 | X = X + z1*s; % begin line search 88 | [f2 df2] = eval(argstr); 89 | i = i + (length<0); % count epochs?! 90 | d2 = df2'*s; 91 | f3 = f1; d3 = d1; z3 = -z1; % initialize point 3 equal to point 1 92 | if length>0, M = MAX; else M = min(MAX, -length-i); end 93 | success = 0; limit = -1; % initialize quanteties 94 | while 1 95 | while ((f2 > f1+z1*RHO*d1) || (d2 > -SIG*d1)) && (M > 0) 96 | limit = z1; % tighten the bracket 97 | if f2 > f1 98 | z2 = z3 - (0.5*d3*z3*z3)/(d3*z3+f2-f3); % quadratic fit 99 | else 100 | A = 6*(f2-f3)/z3+3*(d2+d3); % cubic fit 101 | B = 3*(f3-f2)-z3*(d3+2*d2); 102 | z2 = (sqrt(B*B-A*d2*z3*z3)-B)/A; % numerical error possible - ok! 103 | end 104 | if isnan(z2) || isinf(z2) 105 | z2 = z3/2; % if we had a numerical problem then bisect 106 | end 107 | z2 = max(min(z2, INT*z3),(1-INT)*z3); % don't accept too close to limits 108 | z1 = z1 + z2; % update the step 109 | X = X + z2*s; 110 | [f2 df2] = eval(argstr); 111 | M = M - 1; i = i + (length<0); % count epochs?! 112 | d2 = df2'*s; 113 | z3 = z3-z2; % z3 is now relative to the location of z2 114 | end 115 | if f2 > f1+z1*RHO*d1 || d2 > -SIG*d1 116 | break; % this is a failure 117 | elseif d2 > SIG*d1 118 | success = 1; break; % success 119 | elseif M == 0 120 | break; % failure 121 | end 122 | A = 6*(f2-f3)/z3+3*(d2+d3); % make cubic extrapolation 123 | B = 3*(f3-f2)-z3*(d3+2*d2); 124 | z2 = -d2*z3*z3/(B+sqrt(B*B-A*d2*z3*z3)); % num. error possible - ok! 125 | if ~isreal(z2) || isnan(z2) || isinf(z2) || z2 < 0 % num prob or wrong sign? 126 | if limit < -0.5 % if we have no upper limit 127 | z2 = z1 * (EXT-1); % the extrapolate the maximum amount 128 | else 129 | z2 = (limit-z1)/2; % otherwise bisect 130 | end 131 | elseif (limit > -0.5) && (z2+z1 > limit) % extraplation beyond max? 132 | z2 = (limit-z1)/2; % bisect 133 | elseif (limit < -0.5) && (z2+z1 > z1*EXT) % extrapolation beyond limit 134 | z2 = z1*(EXT-1.0); % set to extrapolation limit 135 | elseif z2 < -z3*INT 136 | z2 = -z3*INT; 137 | elseif (limit > -0.5) && (z2 < (limit-z1)*(1.0-INT)) % too close to limit? 138 | z2 = (limit-z1)*(1.0-INT); 139 | end 140 | f3 = f2; d3 = d2; z3 = -z2; % set point 3 equal to point 2 141 | z1 = z1 + z2; X = X + z2*s; % update current estimates 142 | [f2 df2] = eval(argstr); 143 | M = M - 1; i = i + (length<0); % count epochs?! 144 | d2 = df2'*s; 145 | end % end of line search 146 | 147 | if success % if line search succeeded 148 | f1 = f2; fX = [fX' f1]'; 149 | fprintf('%s %4i | Cost: %4.6e\r', S, i, f1); 150 | s = (df2'*df2-df1'*df2)/(df1'*df1)*s - df2; % Polack-Ribiere direction 151 | tmp = df1; df1 = df2; df2 = tmp; % swap derivatives 152 | d2 = df1'*s; 153 | if d2 > 0 % new slope must be negative 154 | s = -df1; % otherwise use steepest direction 155 | d2 = -s'*s; 156 | end 157 | z1 = z1 * min(RATIO, d1/(d2-realmin)); % slope ratio but max RATIO 158 | d1 = d2; 159 | ls_failed = 0; % this line search did not fail 160 | else 161 | X = X0; f1 = f0; df1 = df0; % restore point from before failed line search 162 | if ls_failed || i > abs(length) % line search failed twice in a row 163 | break; % or we ran out of time, so we give up 164 | end 165 | tmp = df1; df1 = df2; df2 = tmp; % swap derivatives 166 | s = -df1; % try steepest 167 | d1 = -s'*s; 168 | z1 = 1/(1-d1); 169 | ls_failed = 1; % this line search failed 170 | end 171 | if exist('OCTAVE_VERSION') 172 | fflush(stdout); 173 | end 174 | end 175 | fprintf('\n'); 176 | -------------------------------------------------------------------------------- /NeuralNetwork/main.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/NeuralNetwork/main.m -------------------------------------------------------------------------------- /NeuralNetwork/nnCostFunction.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/NeuralNetwork/nnCostFunction.m -------------------------------------------------------------------------------- /NeuralNetwork/predict.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/NeuralNetwork/predict.m -------------------------------------------------------------------------------- /NeuralNetwork/randInitializeWeights.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/NeuralNetwork/randInitializeWeights.m -------------------------------------------------------------------------------- /NeuralNetwork/readme.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/NeuralNetwork/readme.md -------------------------------------------------------------------------------- /NeuralNetwork/sigmoid.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/NeuralNetwork/sigmoid.m -------------------------------------------------------------------------------- /NeuralNetwork/sigmoidGradient.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/NeuralNetwork/sigmoidGradient.m -------------------------------------------------------------------------------- /NeuralNetwork/weights.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/NeuralNetwork/weights.mat -------------------------------------------------------------------------------- /PCA/data1.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/PCA/data1.mat -------------------------------------------------------------------------------- /PCA/data_faces.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/PCA/data_faces.mat -------------------------------------------------------------------------------- /PCA/displayData.m: -------------------------------------------------------------------------------- 1 | function [h, display_array] = displayData(X, example_width) 2 | %DISPLAYDATA Display 2D data in a nice grid 3 | % [h, display_array] = DISPLAYDATA(X, example_width) displays 2D data 4 | % stored in X in a nice grid. It returns the figure handle h and the 5 | % displayed array if requested. 6 | 7 | % Set example_width automatically if not passed in 8 | if ~exist('example_width', 'var') || isempty(example_width) 9 | example_width = round(sqrt(size(X, 2))); 10 | end 11 | 12 | % Gray Image 13 | colormap(gray); 14 | 15 | % Compute rows, cols 16 | [m n] = size(X); 17 | example_height = (n / example_width); 18 | 19 | % Compute number of items to display 20 | display_rows = floor(sqrt(m)); 21 | display_cols = ceil(m / display_rows); 22 | 23 | % Between images padding 24 | pad = 1; 25 | 26 | % Setup blank display 27 | display_array = - ones(pad + display_rows * (example_height + pad), ... 28 | pad + display_cols * (example_width + pad)); 29 | 30 | % Copy each example into a patch on the display array 31 | curr_ex = 1; 32 | for j = 1:display_rows 33 | for i = 1:display_cols 34 | if curr_ex > m, 35 | break; 36 | end 37 | % Copy the patch 38 | 39 | % Get the max value of the patch 40 | max_val = max(abs(X(curr_ex, :))); 41 | display_array(pad + (j - 1) * (example_height + pad) + (1:example_height), ... 42 | pad + (i - 1) * (example_width + pad) + (1:example_width)) = ... 43 | reshape(X(curr_ex, :), example_height, example_width) / max_val; 44 | curr_ex = curr_ex + 1; 45 | end 46 | if curr_ex > m, 47 | break; 48 | end 49 | end 50 | 51 | % Display Image 52 | h = imagesc(display_array, [-1 1]); 53 | 54 | % Do not show axis 55 | axis image off 56 | 57 | drawnow; 58 | 59 | end 60 | -------------------------------------------------------------------------------- /PCA/drawLine.m: -------------------------------------------------------------------------------- 1 | function drawLine(p1, p2, varargin) 2 | %DRAWLINE Draws a line from point p1 to point p2 3 | % DRAWLINE(p1, p2) Draws a line from point p1 to point p2 and holds the 4 | % current figure 5 | 6 | plot([p1(1) p2(1)], [p1(2) p2(2)], varargin{:}); 7 | 8 | end -------------------------------------------------------------------------------- /PCA/featureNormalize.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/PCA/featureNormalize.m -------------------------------------------------------------------------------- /PCA/main.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/PCA/main.m -------------------------------------------------------------------------------- /PCA/projectData.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/PCA/projectData.m -------------------------------------------------------------------------------- /PCA/recoverData.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/PCA/recoverData.m -------------------------------------------------------------------------------- /SVM/data.txt: -------------------------------------------------------------------------------- 1 | 0.051267,0.69956,1 2 | -0.092742,0.68494,1 3 | -0.21371,0.69225,1 4 | -0.375,0.50219,1 5 | -0.51325,0.46564,1 6 | -0.52477,0.2098,1 7 | -0.39804,0.034357,1 8 | -0.30588,-0.19225,1 9 | 0.016705,-0.40424,1 10 | 0.13191,-0.51389,1 11 | 0.38537,-0.56506,1 12 | 0.52938,-0.5212,1 13 | 0.63882,-0.24342,1 14 | 0.73675,-0.18494,1 15 | 0.54666,0.48757,1 16 | 0.322,0.5826,1 17 | 0.16647,0.53874,1 18 | -0.046659,0.81652,1 19 | -0.17339,0.69956,1 20 | -0.47869,0.63377,1 21 | -0.60541,0.59722,1 22 | -0.62846,0.33406,1 23 | -0.59389,0.005117,1 24 | -0.42108,-0.27266,1 25 | -0.11578,-0.39693,1 26 | 0.20104,-0.60161,1 27 | 0.46601,-0.53582,1 28 | 0.67339,-0.53582,1 29 | -0.13882,0.54605,1 30 | -0.29435,0.77997,1 31 | -0.26555,0.96272,1 32 | -0.16187,0.8019,1 33 | -0.17339,0.64839,1 34 | -0.28283,0.47295,1 35 | -0.36348,0.31213,1 36 | -0.30012,0.027047,1 37 | -0.23675,-0.21418,1 38 | -0.06394,-0.18494,1 39 | 0.062788,-0.16301,1 40 | 0.22984,-0.41155,1 41 | 0.2932,-0.2288,1 42 | 0.48329,-0.18494,1 43 | 0.64459,-0.14108,1 44 | 0.46025,0.012427,1 45 | 0.6273,0.15863,1 46 | 0.57546,0.26827,1 47 | 0.72523,0.44371,1 48 | 0.22408,0.52412,1 49 | 0.44297,0.67032,1 50 | 0.322,0.69225,1 51 | 0.13767,0.57529,1 52 | -0.0063364,0.39985,1 53 | -0.092742,0.55336,1 54 | -0.20795,0.35599,1 55 | -0.20795,0.17325,1 56 | -0.43836,0.21711,1 57 | -0.21947,-0.016813,1 58 | -0.13882,-0.27266,1 59 | 0.18376,0.93348,0 60 | 0.22408,0.77997,0 61 | 0.29896,0.61915,0 62 | 0.50634,0.75804,0 63 | 0.61578,0.7288,0 64 | 0.60426,0.59722,0 65 | 0.76555,0.50219,0 66 | 0.92684,0.3633,0 67 | 0.82316,0.27558,0 68 | 0.96141,0.085526,0 69 | 0.93836,0.012427,0 70 | 0.86348,-0.082602,0 71 | 0.89804,-0.20687,0 72 | 0.85196,-0.36769,0 73 | 0.82892,-0.5212,0 74 | 0.79435,-0.55775,0 75 | 0.59274,-0.7405,0 76 | 0.51786,-0.5943,0 77 | 0.46601,-0.41886,0 78 | 0.35081,-0.57968,0 79 | 0.28744,-0.76974,0 80 | 0.085829,-0.75512,0 81 | 0.14919,-0.57968,0 82 | -0.13306,-0.4481,0 83 | -0.40956,-0.41155,0 84 | -0.39228,-0.25804,0 85 | -0.74366,-0.25804,0 86 | -0.69758,0.041667,0 87 | -0.75518,0.2902,0 88 | -0.69758,0.68494,0 89 | -0.4038,0.70687,0 90 | -0.38076,0.91886,0 91 | -0.50749,0.90424,0 92 | -0.54781,0.70687,0 93 | 0.10311,0.77997,0 94 | 0.057028,0.91886,0 95 | -0.10426,0.99196,0 96 | -0.081221,1.1089,0 97 | 0.28744,1.087,0 98 | 0.39689,0.82383,0 99 | 0.63882,0.88962,0 100 | 0.82316,0.66301,0 101 | 0.67339,0.64108,0 102 | 1.0709,0.10015,0 103 | -0.046659,-0.57968,0 104 | -0.23675,-0.63816,0 105 | -0.15035,-0.36769,0 106 | -0.49021,-0.3019,0 107 | -0.46717,-0.13377,0 108 | -0.28859,-0.060673,0 109 | -0.61118,-0.067982,0 110 | -0.66302,-0.21418,0 111 | -0.59965,-0.41886,0 112 | -0.72638,-0.082602,0 113 | -0.83007,0.31213,0 114 | -0.72062,0.53874,0 115 | -0.59389,0.49488,0 116 | -0.48445,0.99927,0 117 | -0.0063364,0.99927,0 118 | 0.63265,-0.030612,0 119 | -------------------------------------------------------------------------------- /SVM/data1.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/SVM/data1.mat -------------------------------------------------------------------------------- /SVM/data2.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/SVM/data2.mat -------------------------------------------------------------------------------- /SVM/data3.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/SVM/data3.mat -------------------------------------------------------------------------------- /SVM/dataset3Params.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/SVM/dataset3Params.m -------------------------------------------------------------------------------- /SVM/emailFeatures.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/SVM/emailFeatures.m -------------------------------------------------------------------------------- /SVM/emailSample1.txt: -------------------------------------------------------------------------------- 1 | > Anyone knows how much it costs to host a web portal ? 2 | > 3 | Well, it depends on how many visitors you're expecting. 4 | This can be anywhere from less than 10 bucks a month to a couple of $100. 5 | You should checkout http://www.rackspace.com/ or perhaps Amazon EC2 6 | if youre running something big.. 7 | 8 | To unsubscribe yourself from this mailing list, send an email to: 9 | groupname-unsubscribe@egroups.com 10 | 11 | -------------------------------------------------------------------------------- /SVM/emailSample2.txt: -------------------------------------------------------------------------------- 1 | Folks, 2 | 3 | my first time posting - have a bit of Unix experience, but am new to Linux. 4 | 5 | 6 | Just got a new PC at home - Dell box with Windows XP. Added a second hard disk 7 | for Linux. Partitioned the disk and have installed Suse 7.2 from CD, which went 8 | fine except it didn't pick up my monitor. 9 | 10 | I have a Dell branded E151FPp 15" LCD flat panel monitor and a nVidia GeForce4 11 | Ti4200 video card, both of which are probably too new to feature in Suse's default 12 | set. I downloaded a driver from the nVidia website and installed it using RPM. 13 | Then I ran Sax2 (as was recommended in some postings I found on the net), but 14 | it still doesn't feature my video card in the available list. What next? 15 | 16 | Another problem. I have a Dell branded keyboard and if I hit Caps-Lock twice, 17 | the whole machine crashes (in Linux, not Windows) - even the on/off switch is 18 | inactive, leaving me to reach for the power cable instead. 19 | 20 | If anyone can help me in any way with these probs., I'd be really grateful - 21 | I've searched the 'net but have run out of ideas. 22 | 23 | Or should I be going for a different version of Linux such as RedHat? Opinions 24 | welcome. 25 | 26 | Thanks a lot, 27 | Peter 28 | 29 | -- 30 | Irish Linux Users' Group: ilug@linux.ie 31 | http://www.linux.ie/mailman/listinfo/ilug for (un)subscription information. 32 | List maintainer: listmaster@linux.ie 33 | 34 | 35 | -------------------------------------------------------------------------------- /SVM/gaussianKernel.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/SVM/gaussianKernel.m -------------------------------------------------------------------------------- /SVM/getVocabList.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/SVM/getVocabList.m -------------------------------------------------------------------------------- /SVM/linearKernel.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/SVM/linearKernel.m -------------------------------------------------------------------------------- /SVM/main.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/SVM/main.m -------------------------------------------------------------------------------- /SVM/main_spam.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/SVM/main_spam.m -------------------------------------------------------------------------------- /SVM/plotBoundary.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/SVM/plotBoundary.m -------------------------------------------------------------------------------- /SVM/plotData.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/SVM/plotData.m -------------------------------------------------------------------------------- /SVM/porterStemmer.m: -------------------------------------------------------------------------------- 1 | function stem = porterStemmer(inString) 2 | % Applies the Porter Stemming algorithm as presented in the following 3 | % paper: 4 | % Porter, 1980, An algorithm for suffix stripping, Program, Vol. 14, 5 | % no. 3, pp 130-137 6 | 7 | % Original code modeled after the C version provided at: 8 | % http://www.tartarus.org/~martin/PorterStemmer/c.txt 9 | 10 | % The main part of the stemming algorithm starts here. b is an array of 11 | % characters, holding the word to be stemmed. The letters are in b[k0], 12 | % b[k0+1] ending at b[k]. In fact k0 = 1 in this demo program (since 13 | % matlab begins indexing by 1 instead of 0). k is readjusted downwards as 14 | % the stemming progresses. Zero termination is not in fact used in the 15 | % algorithm. 16 | 17 | % To call this function, use the string to be stemmed as the input 18 | % argument. This function returns the stemmed word as a string. 19 | 20 | % Lower-case string 21 | inString = lower(inString); 22 | 23 | global j; 24 | b = inString; 25 | k = length(b); 26 | k0 = 1; 27 | j = k; 28 | 29 | 30 | 31 | % With this if statement, strings of length 1 or 2 don't go through the 32 | % stemming process. Remove this conditional to match the published 33 | % algorithm. 34 | stem = b; 35 | if k > 2 36 | % Output displays per step are commented out. 37 | %disp(sprintf('Word to stem: %s', b)); 38 | x = step1ab(b, k, k0); 39 | %disp(sprintf('Steps 1A and B yield: %s', x{1})); 40 | x = step1c(x{1}, x{2}, k0); 41 | %disp(sprintf('Step 1C yields: %s', x{1})); 42 | x = step2(x{1}, x{2}, k0); 43 | %disp(sprintf('Step 2 yields: %s', x{1})); 44 | x = step3(x{1}, x{2}, k0); 45 | %disp(sprintf('Step 3 yields: %s', x{1})); 46 | x = step4(x{1}, x{2}, k0); 47 | %disp(sprintf('Step 4 yields: %s', x{1})); 48 | x = step5(x{1}, x{2}, k0); 49 | %disp(sprintf('Step 5 yields: %s', x{1})); 50 | stem = x{1}; 51 | end 52 | 53 | % cons(j) is TRUE <=> b[j] is a consonant. 54 | function c = cons(i, b, k0) 55 | c = true; 56 | switch(b(i)) 57 | case {'a', 'e', 'i', 'o', 'u'} 58 | c = false; 59 | case 'y' 60 | if i == k0 61 | c = true; 62 | else 63 | c = ~cons(i - 1, b, k0); 64 | end 65 | end 66 | 67 | % mseq() measures the number of consonant sequences between k0 and j. If 68 | % c is a consonant sequence and v a vowel sequence, and <..> indicates 69 | % arbitrary presence, 70 | 71 | % gives 0 72 | % vc gives 1 73 | % vcvc gives 2 74 | % vcvcvc gives 3 75 | % .... 76 | function n = measure(b, k0) 77 | global j; 78 | n = 0; 79 | i = k0; 80 | while true 81 | if i > j 82 | return 83 | end 84 | if ~cons(i, b, k0) 85 | break; 86 | end 87 | i = i + 1; 88 | end 89 | i = i + 1; 90 | while true 91 | while true 92 | if i > j 93 | return 94 | end 95 | if cons(i, b, k0) 96 | break; 97 | end 98 | i = i + 1; 99 | end 100 | i = i + 1; 101 | n = n + 1; 102 | while true 103 | if i > j 104 | return 105 | end 106 | if ~cons(i, b, k0) 107 | break; 108 | end 109 | i = i + 1; 110 | end 111 | i = i + 1; 112 | end 113 | 114 | 115 | % vowelinstem() is TRUE <=> k0,...j contains a vowel 116 | function vis = vowelinstem(b, k0) 117 | global j; 118 | for i = k0:j, 119 | if ~cons(i, b, k0) 120 | vis = true; 121 | return 122 | end 123 | end 124 | vis = false; 125 | 126 | %doublec(i) is TRUE <=> i,(i-1) contain a double consonant. 127 | function dc = doublec(i, b, k0) 128 | if i < k0+1 129 | dc = false; 130 | return 131 | end 132 | if b(i) ~= b(i-1) 133 | dc = false; 134 | return 135 | end 136 | dc = cons(i, b, k0); 137 | 138 | 139 | % cvc(j) is TRUE <=> j-2,j-1,j has the form consonant - vowel - consonant 140 | % and also if the second c is not w,x or y. this is used when trying to 141 | % restore an e at the end of a short word. e.g. 142 | % 143 | % cav(e), lov(e), hop(e), crim(e), but 144 | % snow, box, tray. 145 | 146 | function c1 = cvc(i, b, k0) 147 | if ((i < (k0+2)) || ~cons(i, b, k0) || cons(i-1, b, k0) || ~cons(i-2, b, k0)) 148 | c1 = false; 149 | else 150 | if (b(i) == 'w' || b(i) == 'x' || b(i) == 'y') 151 | c1 = false; 152 | return 153 | end 154 | c1 = true; 155 | end 156 | 157 | % ends(s) is TRUE <=> k0,...k ends with the string s. 158 | function s = ends(str, b, k) 159 | global j; 160 | if (str(length(str)) ~= b(k)) 161 | s = false; 162 | return 163 | end % tiny speed-up 164 | if (length(str) > k) 165 | s = false; 166 | return 167 | end 168 | if strcmp(b(k-length(str)+1:k), str) 169 | s = true; 170 | j = k - length(str); 171 | return 172 | else 173 | s = false; 174 | end 175 | 176 | % setto(s) sets (j+1),...k to the characters in the string s, readjusting 177 | % k accordingly. 178 | 179 | function so = setto(s, b, k) 180 | global j; 181 | for i = j+1:(j+length(s)) 182 | b(i) = s(i-j); 183 | end 184 | if k > j+length(s) 185 | b((j+length(s)+1):k) = ''; 186 | end 187 | k = length(b); 188 | so = {b, k}; 189 | 190 | % rs(s) is used further down. 191 | % [Note: possible null/value for r if rs is called] 192 | function r = rs(str, b, k, k0) 193 | r = {b, k}; 194 | if measure(b, k0) > 0 195 | r = setto(str, b, k); 196 | end 197 | 198 | % step1ab() gets rid of plurals and -ed or -ing. e.g. 199 | 200 | % caresses -> caress 201 | % ponies -> poni 202 | % ties -> ti 203 | % caress -> caress 204 | % cats -> cat 205 | 206 | % feed -> feed 207 | % agreed -> agree 208 | % disabled -> disable 209 | 210 | % matting -> mat 211 | % mating -> mate 212 | % meeting -> meet 213 | % milling -> mill 214 | % messing -> mess 215 | 216 | % meetings -> meet 217 | 218 | function s1ab = step1ab(b, k, k0) 219 | global j; 220 | if b(k) == 's' 221 | if ends('sses', b, k) 222 | k = k-2; 223 | elseif ends('ies', b, k) 224 | retVal = setto('i', b, k); 225 | b = retVal{1}; 226 | k = retVal{2}; 227 | elseif (b(k-1) ~= 's') 228 | k = k-1; 229 | end 230 | end 231 | if ends('eed', b, k) 232 | if measure(b, k0) > 0; 233 | k = k-1; 234 | end 235 | elseif (ends('ed', b, k) || ends('ing', b, k)) && vowelinstem(b, k0) 236 | k = j; 237 | retVal = {b, k}; 238 | if ends('at', b, k) 239 | retVal = setto('ate', b(k0:k), k); 240 | elseif ends('bl', b, k) 241 | retVal = setto('ble', b(k0:k), k); 242 | elseif ends('iz', b, k) 243 | retVal = setto('ize', b(k0:k), k); 244 | elseif doublec(k, b, k0) 245 | retVal = {b, k-1}; 246 | if b(retVal{2}) == 'l' || b(retVal{2}) == 's' || ... 247 | b(retVal{2}) == 'z' 248 | retVal = {retVal{1}, retVal{2}+1}; 249 | end 250 | elseif measure(b, k0) == 1 && cvc(k, b, k0) 251 | retVal = setto('e', b(k0:k), k); 252 | end 253 | k = retVal{2}; 254 | b = retVal{1}(k0:k); 255 | end 256 | j = k; 257 | s1ab = {b(k0:k), k}; 258 | 259 | % step1c() turns terminal y to i when there is another vowel in the stem. 260 | function s1c = step1c(b, k, k0) 261 | global j; 262 | if ends('y', b, k) && vowelinstem(b, k0) 263 | b(k) = 'i'; 264 | end 265 | j = k; 266 | s1c = {b, k}; 267 | 268 | % step2() maps double suffices to single ones. so -ization ( = -ize plus 269 | % -ation) maps to -ize etc. note that the string before the suffix must give 270 | % m() > 0. 271 | function s2 = step2(b, k, k0) 272 | global j; 273 | s2 = {b, k}; 274 | switch b(k-1) 275 | case {'a'} 276 | if ends('ational', b, k) s2 = rs('ate', b, k, k0); 277 | elseif ends('tional', b, k) s2 = rs('tion', b, k, k0); end; 278 | case {'c'} 279 | if ends('enci', b, k) s2 = rs('ence', b, k, k0); 280 | elseif ends('anci', b, k) s2 = rs('ance', b, k, k0); end; 281 | case {'e'} 282 | if ends('izer', b, k) s2 = rs('ize', b, k, k0); end; 283 | case {'l'} 284 | if ends('bli', b, k) s2 = rs('ble', b, k, k0); 285 | elseif ends('alli', b, k) s2 = rs('al', b, k, k0); 286 | elseif ends('entli', b, k) s2 = rs('ent', b, k, k0); 287 | elseif ends('eli', b, k) s2 = rs('e', b, k, k0); 288 | elseif ends('ousli', b, k) s2 = rs('ous', b, k, k0); end; 289 | case {'o'} 290 | if ends('ization', b, k) s2 = rs('ize', b, k, k0); 291 | elseif ends('ation', b, k) s2 = rs('ate', b, k, k0); 292 | elseif ends('ator', b, k) s2 = rs('ate', b, k, k0); end; 293 | case {'s'} 294 | if ends('alism', b, k) s2 = rs('al', b, k, k0); 295 | elseif ends('iveness', b, k) s2 = rs('ive', b, k, k0); 296 | elseif ends('fulness', b, k) s2 = rs('ful', b, k, k0); 297 | elseif ends('ousness', b, k) s2 = rs('ous', b, k, k0); end; 298 | case {'t'} 299 | if ends('aliti', b, k) s2 = rs('al', b, k, k0); 300 | elseif ends('iviti', b, k) s2 = rs('ive', b, k, k0); 301 | elseif ends('biliti', b, k) s2 = rs('ble', b, k, k0); end; 302 | case {'g'} 303 | if ends('logi', b, k) s2 = rs('log', b, k, k0); end; 304 | end 305 | j = s2{2}; 306 | 307 | % step3() deals with -ic-, -full, -ness etc. similar strategy to step2. 308 | function s3 = step3(b, k, k0) 309 | global j; 310 | s3 = {b, k}; 311 | switch b(k) 312 | case {'e'} 313 | if ends('icate', b, k) s3 = rs('ic', b, k, k0); 314 | elseif ends('ative', b, k) s3 = rs('', b, k, k0); 315 | elseif ends('alize', b, k) s3 = rs('al', b, k, k0); end; 316 | case {'i'} 317 | if ends('iciti', b, k) s3 = rs('ic', b, k, k0); end; 318 | case {'l'} 319 | if ends('ical', b, k) s3 = rs('ic', b, k, k0); 320 | elseif ends('ful', b, k) s3 = rs('', b, k, k0); end; 321 | case {'s'} 322 | if ends('ness', b, k) s3 = rs('', b, k, k0); end; 323 | end 324 | j = s3{2}; 325 | 326 | % step4() takes off -ant, -ence etc., in context vcvc. 327 | function s4 = step4(b, k, k0) 328 | global j; 329 | switch b(k-1) 330 | case {'a'} 331 | if ends('al', b, k) end; 332 | case {'c'} 333 | if ends('ance', b, k) 334 | elseif ends('ence', b, k) end; 335 | case {'e'} 336 | if ends('er', b, k) end; 337 | case {'i'} 338 | if ends('ic', b, k) end; 339 | case {'l'} 340 | if ends('able', b, k) 341 | elseif ends('ible', b, k) end; 342 | case {'n'} 343 | if ends('ant', b, k) 344 | elseif ends('ement', b, k) 345 | elseif ends('ment', b, k) 346 | elseif ends('ent', b, k) end; 347 | case {'o'} 348 | if ends('ion', b, k) 349 | if j == 0 350 | elseif ~(strcmp(b(j),'s') || strcmp(b(j),'t')) 351 | j = k; 352 | end 353 | elseif ends('ou', b, k) end; 354 | case {'s'} 355 | if ends('ism', b, k) end; 356 | case {'t'} 357 | if ends('ate', b, k) 358 | elseif ends('iti', b, k) end; 359 | case {'u'} 360 | if ends('ous', b, k) end; 361 | case {'v'} 362 | if ends('ive', b, k) end; 363 | case {'z'} 364 | if ends('ize', b, k) end; 365 | end 366 | if measure(b, k0) > 1 367 | s4 = {b(k0:j), j}; 368 | else 369 | s4 = {b(k0:k), k}; 370 | end 371 | 372 | % step5() removes a final -e if m() > 1, and changes -ll to -l if m() > 1. 373 | function s5 = step5(b, k, k0) 374 | global j; 375 | j = k; 376 | if b(k) == 'e' 377 | a = measure(b, k0); 378 | if (a > 1) || ((a == 1) && ~cvc(k-1, b, k0)) 379 | k = k-1; 380 | end 381 | end 382 | if (b(k) == 'l') && doublec(k, b, k0) && (measure(b, k0) > 1) 383 | k = k-1; 384 | end 385 | s5 = {b(k0:k), k}; 386 | -------------------------------------------------------------------------------- /SVM/processEmail.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/SVM/processEmail.m -------------------------------------------------------------------------------- /SVM/readFile.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/SVM/readFile.m -------------------------------------------------------------------------------- /SVM/readme.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/SVM/readme.md -------------------------------------------------------------------------------- /SVM/spamSample1.txt: -------------------------------------------------------------------------------- 1 | Do You Want To Make $1000 Or More Per Week? 2 | 3 | 4 | 5 | If you are a motivated and qualified individual - I 6 | will personally demonstrate to you a system that will 7 | make you $1,000 per week or more! This is NOT mlm. 8 | 9 | 10 | 11 | Call our 24 hour pre-recorded number to get the 12 | details. 13 | 14 | 15 | 16 | 000-456-789 17 | 18 | 19 | 20 | I need people who want to make serious money. Make 21 | the call and get the facts. 22 | 23 | Invest 2 minutes in yourself now! 24 | 25 | 26 | 27 | 000-456-789 28 | 29 | 30 | 31 | Looking forward to your call and I will introduce you 32 | to people like yourself who 33 | are currently making $10,000 plus per week! 34 | 35 | 36 | 37 | 000-456-789 38 | 39 | 40 | 41 | 3484lJGv6-241lEaN9080lRmS6-271WxHo7524qiyT5-438rjUv5615hQcf0-662eiDB9057dMtVl72 42 | 43 | -------------------------------------------------------------------------------- /SVM/spamSample2.txt: -------------------------------------------------------------------------------- 1 | Best Buy Viagra Generic Online 2 | 3 | Viagra 100mg x 60 Pills $125, Free Pills & Reorder Discount, Top Selling 100% Quality & Satisfaction guaranteed! 4 | 5 | We accept VISA, Master & E-Check Payments, 90000+ Satisfied Customers! 6 | http://medphysitcstech.ru 7 | 8 | 9 | -------------------------------------------------------------------------------- /SVM/spamTest.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/SVM/spamTest.mat -------------------------------------------------------------------------------- /SVM/spamTrain.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/SVM/spamTrain.mat -------------------------------------------------------------------------------- /SVM/svmPredict.m: -------------------------------------------------------------------------------- 1 | function pred = svmPredict(model, X) 2 | %SVMPREDICT returns a vector of predictions using a trained SVM model 3 | %(svmTrain). 4 | % pred = SVMPREDICT(model, X) returns a vector of predictions using a 5 | % trained SVM model (svmTrain). X is a mxn matrix where there each 6 | % example is a row. model is a svm model returned from svmTrain. 7 | % predictions pred is a m x 1 column of predictions of {0, 1} values. 8 | % 9 | 10 | % Check if we are getting a column vector, if so, then assume that we only 11 | % need to do prediction for a single example 12 | if (size(X, 2) == 1) 13 | % Examples should be in rows 14 | X = X'; 15 | end 16 | 17 | % Dataset 18 | m = size(X, 1); 19 | p = zeros(m, 1); 20 | pred = zeros(m, 1); 21 | 22 | if strcmp(func2str(model.kernelFunction), 'linearKernel') 23 | % We can use the weights and bias directly if working with the 24 | % linear kernel 25 | p = X * model.w + model.b; 26 | elseif strfind(func2str(model.kernelFunction), 'gaussianKernel') 27 | % Vectorized RBF Kernel 28 | % This is equivalent to computing the kernel on every pair of examples 29 | X1 = sum(X.^2, 2); 30 | X2 = sum(model.X.^2, 2)'; 31 | K = bsxfun(@plus, X1, bsxfun(@plus, X2, - 2 * X * model.X')); 32 | K = model.kernelFunction(1, 0) .^ K; 33 | K = bsxfun(@times, model.y', K); 34 | K = bsxfun(@times, model.alphas', K); 35 | p = sum(K, 2); 36 | else 37 | % Other Non-linear kernel 38 | for i = 1:m 39 | prediction = 0; 40 | for j = 1:size(model.X, 1) 41 | prediction = prediction + ... 42 | model.alphas(j) * model.y(j) * ... 43 | model.kernelFunction(X(i,:)', model.X(j,:)'); 44 | end 45 | p(i) = prediction + model.b; 46 | end 47 | end 48 | 49 | % Convert predictions into 0 / 1 50 | pred(p >= 0) = 1; 51 | pred(p < 0) = 0; 52 | 53 | end 54 | -------------------------------------------------------------------------------- /SVM/svmTrain.m: -------------------------------------------------------------------------------- 1 | function [model] = svmTrain(X, Y, C, kernelFunction, ... 2 | tol, max_passes) 3 | %SVMTRAIN Trains an SVM classifier using a simplified version of the SMO 4 | %algorithm. 5 | % [model] = SVMTRAIN(X, Y, C, kernelFunction, tol, max_passes) trains an 6 | % SVM classifier and returns trained model. X is the matrix of training 7 | % examples. Each row is a training example, and the jth column holds the 8 | % jth feature. Y is a column matrix containing 1 for positive examples 9 | % and 0 for negative examples. C is the standard SVM regularization 10 | % parameter. tol is a tolerance value used for determining equality of 11 | % floating point numbers. max_passes controls the number of iterations 12 | % over the dataset (without changes to alpha) before the algorithm quits. 13 | % 14 | % Note: This is a simplified version of the SMO algorithm for training 15 | % SVMs. In practice, if you want to train an SVM classifier, we 16 | % recommend using an optimized package such as: 17 | % 18 | % LIBSVM (http://www.csie.ntu.edu.tw/~cjlin/libsvm/) 19 | % SVMLight (http://svmlight.joachims.org/) 20 | % 21 | % 22 | 23 | if ~exist('tol', 'var') || isempty(tol) 24 | tol = 1e-3; 25 | end 26 | 27 | if ~exist('max_passes', 'var') || isempty(max_passes) 28 | max_passes = 5; 29 | end 30 | 31 | % Data parameters 32 | m = size(X, 1); 33 | n = size(X, 2); 34 | 35 | % Map 0 to -1 36 | Y(Y==0) = -1; 37 | 38 | % Variables 39 | alphas = zeros(m, 1); 40 | b = 0; 41 | E = zeros(m, 1); 42 | passes = 0; 43 | eta = 0; 44 | L = 0; 45 | H = 0; 46 | 47 | % Pre-compute the Kernel Matrix since our dataset is small 48 | % (in practice, optimized SVM packages that handle large datasets 49 | % gracefully will _not_ do this) 50 | % 51 | % We have implemented optimized vectorized version of the Kernels here so 52 | % that the svm training will run faster. 53 | if strcmp(func2str(kernelFunction), 'linearKernel') 54 | % Vectorized computation for the Linear Kernel 55 | % This is equivalent to computing the kernel on every pair of examples 56 | K = X*X'; 57 | elseif strfind(func2str(kernelFunction), 'gaussianKernel') 58 | % Vectorized RBF Kernel 59 | % This is equivalent to computing the kernel on every pair of examples 60 | X2 = sum(X.^2, 2); 61 | K = bsxfun(@plus, X2, bsxfun(@plus, X2', - 2 * (X * X'))); 62 | K = kernelFunction(1, 0) .^ K; 63 | else 64 | % Pre-compute the Kernel Matrix 65 | % The following can be slow due to the lack of vectorization 66 | K = zeros(m); 67 | for i = 1:m 68 | for j = i:m 69 | K(i,j) = kernelFunction(X(i,:)', X(j,:)'); 70 | K(j,i) = K(i,j); %the matrix is symmetric 71 | end 72 | end 73 | end 74 | 75 | % Train 76 | fprintf('\nTraining ...'); 77 | dots = 12; 78 | while passes < max_passes, 79 | 80 | num_changed_alphas = 0; 81 | for i = 1:m, 82 | 83 | % Calculate Ei = f(x(i)) - y(i) using (2). 84 | % E(i) = b + sum (X(i, :) * (repmat(alphas.*Y,1,n).*X)') - Y(i); 85 | E(i) = b + sum (alphas.*Y.*K(:,i)) - Y(i); 86 | 87 | if ((Y(i)*E(i) < -tol && alphas(i) < C) || (Y(i)*E(i) > tol && alphas(i) > 0)), 88 | 89 | % In practice, there are many heuristics one can use to select 90 | % the i and j. In this simplified code, we select them randomly. 91 | j = ceil(m * rand()); 92 | while j == i, % Make sure i \neq j 93 | j = ceil(m * rand()); 94 | end 95 | 96 | % Calculate Ej = f(x(j)) - y(j) using (2). 97 | E(j) = b + sum (alphas.*Y.*K(:,j)) - Y(j); 98 | 99 | % Save old alphas 100 | alpha_i_old = alphas(i); 101 | alpha_j_old = alphas(j); 102 | 103 | % Compute L and H by (10) or (11). 104 | if (Y(i) == Y(j)), 105 | L = max(0, alphas(j) + alphas(i) - C); 106 | H = min(C, alphas(j) + alphas(i)); 107 | else 108 | L = max(0, alphas(j) - alphas(i)); 109 | H = min(C, C + alphas(j) - alphas(i)); 110 | end 111 | 112 | if (L == H), 113 | % continue to next i. 114 | continue; 115 | end 116 | 117 | % Compute eta by (14). 118 | eta = 2 * K(i,j) - K(i,i) - K(j,j); 119 | if (eta >= 0), 120 | % continue to next i. 121 | continue; 122 | end 123 | 124 | % Compute and clip new value for alpha j using (12) and (15). 125 | alphas(j) = alphas(j) - (Y(j) * (E(i) - E(j))) / eta; 126 | 127 | % Clip 128 | alphas(j) = min (H, alphas(j)); 129 | alphas(j) = max (L, alphas(j)); 130 | 131 | % Check if change in alpha is significant 132 | if (abs(alphas(j) - alpha_j_old) < tol), 133 | % continue to next i. 134 | % replace anyway 135 | alphas(j) = alpha_j_old; 136 | continue; 137 | end 138 | 139 | % Determine value for alpha i using (16). 140 | alphas(i) = alphas(i) + Y(i)*Y(j)*(alpha_j_old - alphas(j)); 141 | 142 | % Compute b1 and b2 using (17) and (18) respectively. 143 | b1 = b - E(i) ... 144 | - Y(i) * (alphas(i) - alpha_i_old) * K(i,j)' ... 145 | - Y(j) * (alphas(j) - alpha_j_old) * K(i,j)'; 146 | b2 = b - E(j) ... 147 | - Y(i) * (alphas(i) - alpha_i_old) * K(i,j)' ... 148 | - Y(j) * (alphas(j) - alpha_j_old) * K(j,j)'; 149 | 150 | % Compute b by (19). 151 | if (0 < alphas(i) && alphas(i) < C), 152 | b = b1; 153 | elseif (0 < alphas(j) && alphas(j) < C), 154 | b = b2; 155 | else 156 | b = (b1+b2)/2; 157 | end 158 | 159 | num_changed_alphas = num_changed_alphas + 1; 160 | 161 | end 162 | 163 | end 164 | 165 | if (num_changed_alphas == 0), 166 | passes = passes + 1; 167 | else 168 | passes = 0; 169 | end 170 | 171 | fprintf('.'); 172 | dots = dots + 1; 173 | if dots > 78 174 | dots = 0; 175 | fprintf('\n'); 176 | end 177 | if exist('OCTAVE_VERSION') 178 | fflush(stdout); 179 | end 180 | end 181 | fprintf(' Done! \n\n'); 182 | 183 | % Save the model 184 | idx = alphas > 0; 185 | model.X= X(idx,:); 186 | model.y= Y(idx); 187 | model.kernelFunction = kernelFunction; 188 | model.b= b; 189 | model.alphas= alphas(idx); 190 | model.w = ((alphas.*Y)'*X)'; 191 | 192 | end 193 | -------------------------------------------------------------------------------- /SVM/test.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/SVM/test.m -------------------------------------------------------------------------------- /SVM/vocab.txt: -------------------------------------------------------------------------------- 1 | 1 aa 2 | 2 ab 3 | 3 abil 4 | 4 abl 5 | 5 about 6 | 6 abov 7 | 7 absolut 8 | 8 abus 9 | 9 ac 10 | 10 accept 11 | 11 access 12 | 12 accord 13 | 13 account 14 | 14 achiev 15 | 15 acquir 16 | 16 across 17 | 17 act 18 | 18 action 19 | 19 activ 20 | 20 actual 21 | 21 ad 22 | 22 adam 23 | 23 add 24 | 24 addit 25 | 25 address 26 | 26 administr 27 | 27 adult 28 | 28 advanc 29 | 29 advantag 30 | 30 advertis 31 | 31 advic 32 | 32 advis 33 | 33 ae 34 | 34 af 35 | 35 affect 36 | 36 affili 37 | 37 afford 38 | 38 africa 39 | 39 after 40 | 40 ag 41 | 41 again 42 | 42 against 43 | 43 agenc 44 | 44 agent 45 | 45 ago 46 | 46 agre 47 | 47 agreement 48 | 48 aid 49 | 49 air 50 | 50 al 51 | 51 alb 52 | 52 align 53 | 53 all 54 | 54 allow 55 | 55 almost 56 | 56 alon 57 | 57 along 58 | 58 alreadi 59 | 59 alsa 60 | 60 also 61 | 61 altern 62 | 62 although 63 | 63 alwai 64 | 64 am 65 | 65 amaz 66 | 66 america 67 | 67 american 68 | 68 among 69 | 69 amount 70 | 70 amp 71 | 71 an 72 | 72 analysi 73 | 73 analyst 74 | 74 and 75 | 75 ani 76 | 76 anim 77 | 77 announc 78 | 78 annual 79 | 79 annuiti 80 | 80 anoth 81 | 81 answer 82 | 82 anti 83 | 83 anumb 84 | 84 anybodi 85 | 85 anymor 86 | 86 anyon 87 | 87 anyth 88 | 88 anywai 89 | 89 anywher 90 | 90 aol 91 | 91 ap 92 | 92 apolog 93 | 93 app 94 | 94 appar 95 | 95 appear 96 | 96 appl 97 | 97 appli 98 | 98 applic 99 | 99 appreci 100 | 100 approach 101 | 101 approv 102 | 102 apt 103 | 103 ar 104 | 104 archiv 105 | 105 area 106 | 106 aren 107 | 107 argument 108 | 108 arial 109 | 109 arm 110 | 110 around 111 | 111 arrai 112 | 112 arriv 113 | 113 art 114 | 114 articl 115 | 115 artist 116 | 116 as 117 | 117 ascii 118 | 118 ask 119 | 119 asset 120 | 120 assist 121 | 121 associ 122 | 122 assum 123 | 123 assur 124 | 124 at 125 | 125 atol 126 | 126 attach 127 | 127 attack 128 | 128 attempt 129 | 129 attent 130 | 130 attornei 131 | 131 attract 132 | 132 audio 133 | 133 aug 134 | 134 august 135 | 135 author 136 | 136 auto 137 | 137 autom 138 | 138 automat 139 | 139 avail 140 | 140 averag 141 | 141 avoid 142 | 142 awai 143 | 143 awar 144 | 144 award 145 | 145 ba 146 | 146 babi 147 | 147 back 148 | 148 background 149 | 149 backup 150 | 150 bad 151 | 151 balanc 152 | 152 ban 153 | 153 bank 154 | 154 bar 155 | 155 base 156 | 156 basenumb 157 | 157 basi 158 | 158 basic 159 | 159 bb 160 | 160 bc 161 | 161 bd 162 | 162 be 163 | 163 beat 164 | 164 beberg 165 | 165 becaus 166 | 166 becom 167 | 167 been 168 | 168 befor 169 | 169 begin 170 | 170 behalf 171 | 171 behavior 172 | 172 behind 173 | 173 believ 174 | 174 below 175 | 175 benefit 176 | 176 best 177 | 177 beta 178 | 178 better 179 | 179 between 180 | 180 bf 181 | 181 big 182 | 182 bill 183 | 183 billion 184 | 184 bin 185 | 185 binari 186 | 186 bit 187 | 187 black 188 | 188 blank 189 | 189 block 190 | 190 blog 191 | 191 blood 192 | 192 blue 193 | 193 bnumber 194 | 194 board 195 | 195 bodi 196 | 196 boi 197 | 197 bonu 198 | 198 book 199 | 199 boot 200 | 200 border 201 | 201 boss 202 | 202 boston 203 | 203 botan 204 | 204 both 205 | 205 bottl 206 | 206 bottom 207 | 207 boundari 208 | 208 box 209 | 209 brain 210 | 210 brand 211 | 211 break 212 | 212 brian 213 | 213 bring 214 | 214 broadcast 215 | 215 broker 216 | 216 browser 217 | 217 bug 218 | 218 bui 219 | 219 build 220 | 220 built 221 | 221 bulk 222 | 222 burn 223 | 223 bush 224 | 224 busi 225 | 225 but 226 | 226 button 227 | 227 by 228 | 228 byte 229 | 229 ca 230 | 230 cabl 231 | 231 cach 232 | 232 calcul 233 | 233 california 234 | 234 call 235 | 235 came 236 | 236 camera 237 | 237 campaign 238 | 238 can 239 | 239 canada 240 | 240 cannot 241 | 241 canon 242 | 242 capabl 243 | 243 capillari 244 | 244 capit 245 | 245 car 246 | 246 card 247 | 247 care 248 | 248 career 249 | 249 carri 250 | 250 cartridg 251 | 251 case 252 | 252 cash 253 | 253 cat 254 | 254 catch 255 | 255 categori 256 | 256 caus 257 | 257 cb 258 | 258 cc 259 | 259 cd 260 | 260 ce 261 | 261 cell 262 | 262 cent 263 | 263 center 264 | 264 central 265 | 265 centuri 266 | 266 ceo 267 | 267 certain 268 | 268 certainli 269 | 269 cf 270 | 270 challeng 271 | 271 chanc 272 | 272 chang 273 | 273 channel 274 | 274 char 275 | 275 charact 276 | 276 charg 277 | 277 charset 278 | 278 chat 279 | 279 cheap 280 | 280 check 281 | 281 cheer 282 | 282 chief 283 | 283 children 284 | 284 china 285 | 285 chip 286 | 286 choic 287 | 287 choos 288 | 288 chri 289 | 289 citi 290 | 290 citizen 291 | 291 civil 292 | 292 claim 293 | 293 class 294 | 294 classifi 295 | 295 clean 296 | 296 clear 297 | 297 clearli 298 | 298 click 299 | 299 client 300 | 300 close 301 | 301 clue 302 | 302 cnet 303 | 303 cnumber 304 | 304 co 305 | 305 code 306 | 306 collect 307 | 307 colleg 308 | 308 color 309 | 309 com 310 | 310 combin 311 | 311 come 312 | 312 comfort 313 | 313 command 314 | 314 comment 315 | 315 commentari 316 | 316 commerci 317 | 317 commiss 318 | 318 commit 319 | 319 common 320 | 320 commun 321 | 321 compani 322 | 322 compar 323 | 323 comparison 324 | 324 compat 325 | 325 compet 326 | 326 competit 327 | 327 compil 328 | 328 complet 329 | 329 comprehens 330 | 330 comput 331 | 331 concentr 332 | 332 concept 333 | 333 concern 334 | 334 condit 335 | 335 conf 336 | 336 confer 337 | 337 confid 338 | 338 confidenti 339 | 339 config 340 | 340 configur 341 | 341 confirm 342 | 342 conflict 343 | 343 confus 344 | 344 congress 345 | 345 connect 346 | 346 consid 347 | 347 consolid 348 | 348 constitut 349 | 349 construct 350 | 350 consult 351 | 351 consum 352 | 352 contact 353 | 353 contain 354 | 354 content 355 | 355 continu 356 | 356 contract 357 | 357 contribut 358 | 358 control 359 | 359 conveni 360 | 360 convers 361 | 361 convert 362 | 362 cool 363 | 363 cooper 364 | 364 copi 365 | 365 copyright 366 | 366 core 367 | 367 corpor 368 | 368 correct 369 | 369 correspond 370 | 370 cost 371 | 371 could 372 | 372 couldn 373 | 373 count 374 | 374 countri 375 | 375 coupl 376 | 376 cours 377 | 377 court 378 | 378 cover 379 | 379 coverag 380 | 380 crash 381 | 381 creat 382 | 382 creativ 383 | 383 credit 384 | 384 critic 385 | 385 cross 386 | 386 cultur 387 | 387 current 388 | 388 custom 389 | 389 cut 390 | 390 cv 391 | 391 da 392 | 392 dagga 393 | 393 dai 394 | 394 daili 395 | 395 dan 396 | 396 danger 397 | 397 dark 398 | 398 data 399 | 399 databas 400 | 400 datapow 401 | 401 date 402 | 402 dave 403 | 403 david 404 | 404 dc 405 | 405 de 406 | 406 dead 407 | 407 deal 408 | 408 dear 409 | 409 death 410 | 410 debt 411 | 411 decad 412 | 412 decid 413 | 413 decis 414 | 414 declar 415 | 415 declin 416 | 416 decor 417 | 417 default 418 | 418 defend 419 | 419 defens 420 | 420 defin 421 | 421 definit 422 | 422 degre 423 | 423 delai 424 | 424 delet 425 | 425 deliv 426 | 426 deliveri 427 | 427 dell 428 | 428 demand 429 | 429 democrat 430 | 430 depart 431 | 431 depend 432 | 432 deposit 433 | 433 describ 434 | 434 descript 435 | 435 deserv 436 | 436 design 437 | 437 desir 438 | 438 desktop 439 | 439 despit 440 | 440 detail 441 | 441 detect 442 | 442 determin 443 | 443 dev 444 | 444 devel 445 | 445 develop 446 | 446 devic 447 | 447 di 448 | 448 dial 449 | 449 did 450 | 450 didn 451 | 451 diet 452 | 452 differ 453 | 453 difficult 454 | 454 digit 455 | 455 direct 456 | 456 directli 457 | 457 director 458 | 458 directori 459 | 459 disabl 460 | 460 discount 461 | 461 discov 462 | 462 discoveri 463 | 463 discuss 464 | 464 disk 465 | 465 displai 466 | 466 disposit 467 | 467 distanc 468 | 468 distribut 469 | 469 dn 470 | 470 dnumber 471 | 471 do 472 | 472 doc 473 | 473 document 474 | 474 doe 475 | 475 doer 476 | 476 doesn 477 | 477 dollar 478 | 478 dollarac 479 | 479 dollarnumb 480 | 480 domain 481 | 481 don 482 | 482 done 483 | 483 dont 484 | 484 doubl 485 | 485 doubt 486 | 486 down 487 | 487 download 488 | 488 dr 489 | 489 draw 490 | 490 dream 491 | 491 drive 492 | 492 driver 493 | 493 drop 494 | 494 drug 495 | 495 due 496 | 496 dure 497 | 497 dvd 498 | 498 dw 499 | 499 dynam 500 | 500 ea 501 | 501 each 502 | 502 earli 503 | 503 earlier 504 | 504 earn 505 | 505 earth 506 | 506 easi 507 | 507 easier 508 | 508 easili 509 | 509 eat 510 | 510 eb 511 | 511 ebai 512 | 512 ec 513 | 513 echo 514 | 514 econom 515 | 515 economi 516 | 516 ed 517 | 517 edg 518 | 518 edit 519 | 519 editor 520 | 520 educ 521 | 521 eff 522 | 522 effect 523 | 523 effici 524 | 524 effort 525 | 525 either 526 | 526 el 527 | 527 electron 528 | 528 elimin 529 | 529 els 530 | 530 email 531 | 531 emailaddr 532 | 532 emerg 533 | 533 empir 534 | 534 employ 535 | 535 employe 536 | 536 en 537 | 537 enabl 538 | 538 encod 539 | 539 encourag 540 | 540 end 541 | 541 enemi 542 | 542 enenkio 543 | 543 energi 544 | 544 engin 545 | 545 english 546 | 546 enhanc 547 | 547 enjoi 548 | 548 enough 549 | 549 ensur 550 | 550 enter 551 | 551 enterpris 552 | 552 entertain 553 | 553 entir 554 | 554 entri 555 | 555 enumb 556 | 556 environ 557 | 557 equal 558 | 558 equip 559 | 559 equival 560 | 560 error 561 | 561 especi 562 | 562 essenti 563 | 563 establish 564 | 564 estat 565 | 565 estim 566 | 566 et 567 | 567 etc 568 | 568 euro 569 | 569 europ 570 | 570 european 571 | 571 even 572 | 572 event 573 | 573 eventu 574 | 574 ever 575 | 575 everi 576 | 576 everyon 577 | 577 everyth 578 | 578 evid 579 | 579 evil 580 | 580 exactli 581 | 581 exampl 582 | 582 excel 583 | 583 except 584 | 584 exchang 585 | 585 excit 586 | 586 exclus 587 | 587 execut 588 | 588 exercis 589 | 589 exist 590 | 590 exmh 591 | 591 expand 592 | 592 expect 593 | 593 expens 594 | 594 experi 595 | 595 expert 596 | 596 expir 597 | 597 explain 598 | 598 explor 599 | 599 express 600 | 600 extend 601 | 601 extens 602 | 602 extra 603 | 603 extract 604 | 604 extrem 605 | 605 ey 606 | 606 fa 607 | 607 face 608 | 608 fact 609 | 609 factor 610 | 610 fail 611 | 611 fair 612 | 612 fall 613 | 613 fals 614 | 614 famili 615 | 615 faq 616 | 616 far 617 | 617 fast 618 | 618 faster 619 | 619 fastest 620 | 620 fat 621 | 621 father 622 | 622 favorit 623 | 623 fax 624 | 624 fb 625 | 625 fd 626 | 626 featur 627 | 627 feder 628 | 628 fee 629 | 629 feed 630 | 630 feedback 631 | 631 feel 632 | 632 femal 633 | 633 few 634 | 634 ffffff 635 | 635 ffnumber 636 | 636 field 637 | 637 fight 638 | 638 figur 639 | 639 file 640 | 640 fill 641 | 641 film 642 | 642 filter 643 | 643 final 644 | 644 financ 645 | 645 financi 646 | 646 find 647 | 647 fine 648 | 648 finish 649 | 649 fire 650 | 650 firewal 651 | 651 firm 652 | 652 first 653 | 653 fit 654 | 654 five 655 | 655 fix 656 | 656 flag 657 | 657 flash 658 | 658 flow 659 | 659 fnumber 660 | 660 focu 661 | 661 folder 662 | 662 folk 663 | 663 follow 664 | 664 font 665 | 665 food 666 | 666 for 667 | 667 forc 668 | 668 foreign 669 | 669 forev 670 | 670 forget 671 | 671 fork 672 | 672 form 673 | 673 format 674 | 674 former 675 | 675 fortun 676 | 676 forward 677 | 677 found 678 | 678 foundat 679 | 679 four 680 | 680 franc 681 | 681 free 682 | 682 freedom 683 | 683 french 684 | 684 freshrpm 685 | 685 fri 686 | 686 fridai 687 | 687 friend 688 | 688 from 689 | 689 front 690 | 690 ftoc 691 | 691 ftp 692 | 692 full 693 | 693 fulli 694 | 694 fun 695 | 695 function 696 | 696 fund 697 | 697 further 698 | 698 futur 699 | 699 ga 700 | 700 gain 701 | 701 game 702 | 702 gari 703 | 703 garrigu 704 | 704 gave 705 | 705 gcc 706 | 706 geek 707 | 707 gener 708 | 708 get 709 | 709 gif 710 | 710 gift 711 | 711 girl 712 | 712 give 713 | 713 given 714 | 714 global 715 | 715 gnome 716 | 716 gnu 717 | 717 gnupg 718 | 718 go 719 | 719 goal 720 | 720 god 721 | 721 goe 722 | 722 gold 723 | 723 gone 724 | 724 good 725 | 725 googl 726 | 726 got 727 | 727 govern 728 | 728 gpl 729 | 729 grand 730 | 730 grant 731 | 731 graphic 732 | 732 great 733 | 733 greater 734 | 734 ground 735 | 735 group 736 | 736 grow 737 | 737 growth 738 | 738 gt 739 | 739 guarante 740 | 740 guess 741 | 741 gui 742 | 742 guid 743 | 743 ha 744 | 744 hack 745 | 745 had 746 | 746 half 747 | 747 ham 748 | 748 hand 749 | 749 handl 750 | 750 happen 751 | 751 happi 752 | 752 hard 753 | 753 hardwar 754 | 754 hat 755 | 755 hate 756 | 756 have 757 | 757 haven 758 | 758 he 759 | 759 head 760 | 760 header 761 | 761 headlin 762 | 762 health 763 | 763 hear 764 | 764 heard 765 | 765 heart 766 | 766 heaven 767 | 767 hei 768 | 768 height 769 | 769 held 770 | 770 hello 771 | 771 help 772 | 772 helvetica 773 | 773 her 774 | 774 herba 775 | 775 here 776 | 776 hermio 777 | 777 hettinga 778 | 778 hi 779 | 779 high 780 | 780 higher 781 | 781 highli 782 | 782 highlight 783 | 783 him 784 | 784 histori 785 | 785 hit 786 | 786 hold 787 | 787 home 788 | 788 honor 789 | 789 hope 790 | 790 host 791 | 791 hot 792 | 792 hour 793 | 793 hous 794 | 794 how 795 | 795 howev 796 | 796 hp 797 | 797 html 798 | 798 http 799 | 799 httpaddr 800 | 800 huge 801 | 801 human 802 | 802 hundr 803 | 803 ibm 804 | 804 id 805 | 805 idea 806 | 806 ident 807 | 807 identifi 808 | 808 idnumb 809 | 809 ie 810 | 810 if 811 | 811 ignor 812 | 812 ii 813 | 813 iii 814 | 814 iiiiiiihnumberjnumberhnumberjnumberhnumb 815 | 815 illeg 816 | 816 im 817 | 817 imag 818 | 818 imagin 819 | 819 immedi 820 | 820 impact 821 | 821 implement 822 | 822 import 823 | 823 impress 824 | 824 improv 825 | 825 in 826 | 826 inc 827 | 827 includ 828 | 828 incom 829 | 829 increas 830 | 830 incred 831 | 831 inde 832 | 832 independ 833 | 833 index 834 | 834 india 835 | 835 indian 836 | 836 indic 837 | 837 individu 838 | 838 industri 839 | 839 info 840 | 840 inform 841 | 841 initi 842 | 842 inlin 843 | 843 innov 844 | 844 input 845 | 845 insert 846 | 846 insid 847 | 847 instal 848 | 848 instanc 849 | 849 instant 850 | 850 instead 851 | 851 institut 852 | 852 instruct 853 | 853 insur 854 | 854 int 855 | 855 integr 856 | 856 intel 857 | 857 intellig 858 | 858 intend 859 | 859 interact 860 | 860 interest 861 | 861 interfac 862 | 862 intern 863 | 863 internet 864 | 864 interview 865 | 865 into 866 | 866 intro 867 | 867 introduc 868 | 868 inumb 869 | 869 invest 870 | 870 investig 871 | 871 investor 872 | 872 invok 873 | 873 involv 874 | 874 ip 875 | 875 ireland 876 | 876 irish 877 | 877 is 878 | 878 island 879 | 879 isn 880 | 880 iso 881 | 881 isp 882 | 882 issu 883 | 883 it 884 | 884 item 885 | 885 itself 886 | 886 jabber 887 | 887 jame 888 | 888 java 889 | 889 jim 890 | 890 jnumberiiiiiiihepihepihf 891 | 891 job 892 | 892 joe 893 | 893 john 894 | 894 join 895 | 895 journal 896 | 896 judg 897 | 897 judgment 898 | 898 jul 899 | 899 juli 900 | 900 jump 901 | 901 june 902 | 902 just 903 | 903 justin 904 | 904 keep 905 | 905 kei 906 | 906 kept 907 | 907 kernel 908 | 908 kevin 909 | 909 keyboard 910 | 910 kid 911 | 911 kill 912 | 912 kind 913 | 913 king 914 | 914 kingdom 915 | 915 knew 916 | 916 know 917 | 917 knowledg 918 | 918 known 919 | 919 la 920 | 920 lack 921 | 921 land 922 | 922 languag 923 | 923 laptop 924 | 924 larg 925 | 925 larger 926 | 926 largest 927 | 927 laser 928 | 928 last 929 | 929 late 930 | 930 later 931 | 931 latest 932 | 932 launch 933 | 933 law 934 | 934 lawrenc 935 | 935 le 936 | 936 lead 937 | 937 leader 938 | 938 learn 939 | 939 least 940 | 940 leav 941 | 941 left 942 | 942 legal 943 | 943 lender 944 | 944 length 945 | 945 less 946 | 946 lesson 947 | 947 let 948 | 948 letter 949 | 949 level 950 | 950 lib 951 | 951 librari 952 | 952 licens 953 | 953 life 954 | 954 lifetim 955 | 955 light 956 | 956 like 957 | 957 limit 958 | 958 line 959 | 959 link 960 | 960 linux 961 | 961 list 962 | 962 listen 963 | 963 littl 964 | 964 live 965 | 965 ll 966 | 966 lo 967 | 967 load 968 | 968 loan 969 | 969 local 970 | 970 locat 971 | 971 lock 972 | 972 lockergnom 973 | 973 log 974 | 974 long 975 | 975 longer 976 | 976 look 977 | 977 lose 978 | 978 loss 979 | 979 lost 980 | 980 lot 981 | 981 love 982 | 982 low 983 | 983 lower 984 | 984 lowest 985 | 985 lt 986 | 986 ma 987 | 987 mac 988 | 988 machin 989 | 989 made 990 | 990 magazin 991 | 991 mai 992 | 992 mail 993 | 993 mailer 994 | 994 main 995 | 995 maintain 996 | 996 major 997 | 997 make 998 | 998 maker 999 | 999 male 1000 | 1000 man 1001 | 1001 manag 1002 | 1002 mani 1003 | 1003 manual 1004 | 1004 manufactur 1005 | 1005 map 1006 | 1006 march 1007 | 1007 margin 1008 | 1008 mark 1009 | 1009 market 1010 | 1010 marshal 1011 | 1011 mass 1012 | 1012 master 1013 | 1013 match 1014 | 1014 materi 1015 | 1015 matter 1016 | 1016 matthia 1017 | 1017 mayb 1018 | 1018 me 1019 | 1019 mean 1020 | 1020 measur 1021 | 1021 mechan 1022 | 1022 media 1023 | 1023 medic 1024 | 1024 meet 1025 | 1025 member 1026 | 1026 membership 1027 | 1027 memori 1028 | 1028 men 1029 | 1029 mention 1030 | 1030 menu 1031 | 1031 merchant 1032 | 1032 messag 1033 | 1033 method 1034 | 1034 mh 1035 | 1035 michael 1036 | 1036 microsoft 1037 | 1037 middl 1038 | 1038 might 1039 | 1039 mike 1040 | 1040 mile 1041 | 1041 militari 1042 | 1042 million 1043 | 1043 mime 1044 | 1044 mind 1045 | 1045 mine 1046 | 1046 mini 1047 | 1047 minimum 1048 | 1048 minut 1049 | 1049 miss 1050 | 1050 mistak 1051 | 1051 mobil 1052 | 1052 mode 1053 | 1053 model 1054 | 1054 modem 1055 | 1055 modifi 1056 | 1056 modul 1057 | 1057 moment 1058 | 1058 mon 1059 | 1059 mondai 1060 | 1060 monei 1061 | 1061 monitor 1062 | 1062 month 1063 | 1063 monthli 1064 | 1064 more 1065 | 1065 morn 1066 | 1066 mortgag 1067 | 1067 most 1068 | 1068 mostli 1069 | 1069 mother 1070 | 1070 motiv 1071 | 1071 move 1072 | 1072 movi 1073 | 1073 mpnumber 1074 | 1074 mr 1075 | 1075 ms 1076 | 1076 msg 1077 | 1077 much 1078 | 1078 multi 1079 | 1079 multipart 1080 | 1080 multipl 1081 | 1081 murphi 1082 | 1082 music 1083 | 1083 must 1084 | 1084 my 1085 | 1085 myself 1086 | 1086 name 1087 | 1087 nation 1088 | 1088 natur 1089 | 1089 nbsp 1090 | 1090 near 1091 | 1091 nearli 1092 | 1092 necessari 1093 | 1093 need 1094 | 1094 neg 1095 | 1095 net 1096 | 1096 netscap 1097 | 1097 network 1098 | 1098 never 1099 | 1099 new 1100 | 1100 newslett 1101 | 1101 next 1102 | 1102 nextpart 1103 | 1103 nice 1104 | 1104 nigeria 1105 | 1105 night 1106 | 1106 no 1107 | 1107 nobodi 1108 | 1108 non 1109 | 1109 none 1110 | 1110 nor 1111 | 1111 normal 1112 | 1112 north 1113 | 1113 not 1114 | 1114 note 1115 | 1115 noth 1116 | 1116 notic 1117 | 1117 now 1118 | 1118 nt 1119 | 1119 null 1120 | 1120 number 1121 | 1121 numbera 1122 | 1122 numberam 1123 | 1123 numberanumb 1124 | 1124 numberb 1125 | 1125 numberbit 1126 | 1126 numberc 1127 | 1127 numbercb 1128 | 1128 numbercbr 1129 | 1129 numbercfont 1130 | 1130 numbercli 1131 | 1131 numbercnumb 1132 | 1132 numbercp 1133 | 1133 numberctd 1134 | 1134 numberd 1135 | 1135 numberdari 1136 | 1136 numberdnumb 1137 | 1137 numberenumb 1138 | 1138 numberf 1139 | 1139 numberfb 1140 | 1140 numberff 1141 | 1141 numberffont 1142 | 1142 numberfp 1143 | 1143 numberftd 1144 | 1144 numberk 1145 | 1145 numberm 1146 | 1146 numbermb 1147 | 1147 numberp 1148 | 1148 numberpd 1149 | 1149 numberpm 1150 | 1150 numberpx 1151 | 1151 numberst 1152 | 1152 numberth 1153 | 1153 numbertnumb 1154 | 1154 numberx 1155 | 1155 object 1156 | 1156 oblig 1157 | 1157 obtain 1158 | 1158 obvious 1159 | 1159 occur 1160 | 1160 oct 1161 | 1161 octob 1162 | 1162 of 1163 | 1163 off 1164 | 1164 offer 1165 | 1165 offic 1166 | 1166 offici 1167 | 1167 often 1168 | 1168 oh 1169 | 1169 ok 1170 | 1170 old 1171 | 1171 on 1172 | 1172 onc 1173 | 1173 onli 1174 | 1174 onlin 1175 | 1175 open 1176 | 1176 oper 1177 | 1177 opinion 1178 | 1178 opportun 1179 | 1179 opt 1180 | 1180 optim 1181 | 1181 option 1182 | 1182 or 1183 | 1183 order 1184 | 1184 org 1185 | 1185 organ 1186 | 1186 origin 1187 | 1187 os 1188 | 1188 osdn 1189 | 1189 other 1190 | 1190 otherwis 1191 | 1191 our 1192 | 1192 out 1193 | 1193 outlook 1194 | 1194 output 1195 | 1195 outsid 1196 | 1196 over 1197 | 1197 own 1198 | 1198 owner 1199 | 1199 oz 1200 | 1200 pacif 1201 | 1201 pack 1202 | 1202 packag 1203 | 1203 page 1204 | 1204 pai 1205 | 1205 paid 1206 | 1206 pain 1207 | 1207 palm 1208 | 1208 panel 1209 | 1209 paper 1210 | 1210 paragraph 1211 | 1211 parent 1212 | 1212 part 1213 | 1213 parti 1214 | 1214 particip 1215 | 1215 particular 1216 | 1216 particularli 1217 | 1217 partit 1218 | 1218 partner 1219 | 1219 pass 1220 | 1220 password 1221 | 1221 past 1222 | 1222 patch 1223 | 1223 patent 1224 | 1224 path 1225 | 1225 pattern 1226 | 1226 paul 1227 | 1227 payment 1228 | 1228 pc 1229 | 1229 peac 1230 | 1230 peopl 1231 | 1231 per 1232 | 1232 percent 1233 | 1233 percentag 1234 | 1234 perfect 1235 | 1235 perfectli 1236 | 1236 perform 1237 | 1237 perhap 1238 | 1238 period 1239 | 1239 perl 1240 | 1240 perman 1241 | 1241 permiss 1242 | 1242 person 1243 | 1243 pgp 1244 | 1244 phone 1245 | 1245 photo 1246 | 1246 php 1247 | 1247 phrase 1248 | 1248 physic 1249 | 1249 pick 1250 | 1250 pictur 1251 | 1251 piec 1252 | 1252 piiiiiiii 1253 | 1253 pipe 1254 | 1254 pjnumber 1255 | 1255 place 1256 | 1256 plai 1257 | 1257 plain 1258 | 1258 plan 1259 | 1259 planet 1260 | 1260 plant 1261 | 1261 planta 1262 | 1262 platform 1263 | 1263 player 1264 | 1264 pleas 1265 | 1265 plu 1266 | 1266 plug 1267 | 1267 pm 1268 | 1268 pocket 1269 | 1269 point 1270 | 1270 polic 1271 | 1271 polici 1272 | 1272 polit 1273 | 1273 poor 1274 | 1274 pop 1275 | 1275 popul 1276 | 1276 popular 1277 | 1277 port 1278 | 1278 posit 1279 | 1279 possibl 1280 | 1280 post 1281 | 1281 potenti 1282 | 1282 pound 1283 | 1283 powel 1284 | 1284 power 1285 | 1285 powershot 1286 | 1286 practic 1287 | 1287 pre 1288 | 1288 predict 1289 | 1289 prefer 1290 | 1290 premium 1291 | 1291 prepar 1292 | 1292 present 1293 | 1293 presid 1294 | 1294 press 1295 | 1295 pretti 1296 | 1296 prevent 1297 | 1297 previou 1298 | 1298 previous 1299 | 1299 price 1300 | 1300 principl 1301 | 1301 print 1302 | 1302 printabl 1303 | 1303 printer 1304 | 1304 privaci 1305 | 1305 privat 1306 | 1306 prize 1307 | 1307 pro 1308 | 1308 probabl 1309 | 1309 problem 1310 | 1310 procedur 1311 | 1311 process 1312 | 1312 processor 1313 | 1313 procmail 1314 | 1314 produc 1315 | 1315 product 1316 | 1316 profession 1317 | 1317 profil 1318 | 1318 profit 1319 | 1319 program 1320 | 1320 programm 1321 | 1321 progress 1322 | 1322 project 1323 | 1323 promis 1324 | 1324 promot 1325 | 1325 prompt 1326 | 1326 properti 1327 | 1327 propos 1328 | 1328 proprietari 1329 | 1329 prospect 1330 | 1330 protect 1331 | 1331 protocol 1332 | 1332 prove 1333 | 1333 proven 1334 | 1334 provid 1335 | 1335 proxi 1336 | 1336 pub 1337 | 1337 public 1338 | 1338 publish 1339 | 1339 pudg 1340 | 1340 pull 1341 | 1341 purchas 1342 | 1342 purpos 1343 | 1343 put 1344 | 1344 python 1345 | 1345 qnumber 1346 | 1346 qualifi 1347 | 1347 qualiti 1348 | 1348 quarter 1349 | 1349 question 1350 | 1350 quick 1351 | 1351 quickli 1352 | 1352 quit 1353 | 1353 quot 1354 | 1354 radio 1355 | 1355 ragga 1356 | 1356 rais 1357 | 1357 random 1358 | 1358 rang 1359 | 1359 rate 1360 | 1360 rather 1361 | 1361 ratio 1362 | 1362 razor 1363 | 1363 razornumb 1364 | 1364 re 1365 | 1365 reach 1366 | 1366 read 1367 | 1367 reader 1368 | 1368 readi 1369 | 1369 real 1370 | 1370 realiz 1371 | 1371 realli 1372 | 1372 reason 1373 | 1373 receiv 1374 | 1374 recent 1375 | 1375 recipi 1376 | 1376 recommend 1377 | 1377 record 1378 | 1378 red 1379 | 1379 redhat 1380 | 1380 reduc 1381 | 1381 refer 1382 | 1382 refin 1383 | 1383 reg 1384 | 1384 regard 1385 | 1385 region 1386 | 1386 regist 1387 | 1387 regul 1388 | 1388 regular 1389 | 1389 rel 1390 | 1390 relat 1391 | 1391 relationship 1392 | 1392 releas 1393 | 1393 relev 1394 | 1394 reliabl 1395 | 1395 remain 1396 | 1396 rememb 1397 | 1397 remot 1398 | 1398 remov 1399 | 1399 replac 1400 | 1400 repli 1401 | 1401 report 1402 | 1402 repositori 1403 | 1403 repres 1404 | 1404 republ 1405 | 1405 request 1406 | 1406 requir 1407 | 1407 research 1408 | 1408 reserv 1409 | 1409 resid 1410 | 1410 resourc 1411 | 1411 respect 1412 | 1412 respond 1413 | 1413 respons 1414 | 1414 rest 1415 | 1415 result 1416 | 1416 retail 1417 | 1417 return 1418 | 1418 reveal 1419 | 1419 revenu 1420 | 1420 revers 1421 | 1421 review 1422 | 1422 revok 1423 | 1423 rh 1424 | 1424 rich 1425 | 1425 right 1426 | 1426 risk 1427 | 1427 road 1428 | 1428 robert 1429 | 1429 rock 1430 | 1430 role 1431 | 1431 roll 1432 | 1432 rom 1433 | 1433 roman 1434 | 1434 room 1435 | 1435 root 1436 | 1436 round 1437 | 1437 rpm 1438 | 1438 rss 1439 | 1439 rule 1440 | 1440 run 1441 | 1441 sa 1442 | 1442 safe 1443 | 1443 sai 1444 | 1444 said 1445 | 1445 sale 1446 | 1446 same 1447 | 1447 sampl 1448 | 1448 san 1449 | 1449 saou 1450 | 1450 sat 1451 | 1451 satellit 1452 | 1452 save 1453 | 1453 saw 1454 | 1454 scan 1455 | 1455 schedul 1456 | 1456 school 1457 | 1457 scienc 1458 | 1458 score 1459 | 1459 screen 1460 | 1460 script 1461 | 1461 se 1462 | 1462 search 1463 | 1463 season 1464 | 1464 second 1465 | 1465 secret 1466 | 1466 section 1467 | 1467 secur 1468 | 1468 see 1469 | 1469 seed 1470 | 1470 seek 1471 | 1471 seem 1472 | 1472 seen 1473 | 1473 select 1474 | 1474 self 1475 | 1475 sell 1476 | 1476 seminar 1477 | 1477 send 1478 | 1478 sender 1479 | 1479 sendmail 1480 | 1480 senior 1481 | 1481 sens 1482 | 1482 sensit 1483 | 1483 sent 1484 | 1484 sep 1485 | 1485 separ 1486 | 1486 septemb 1487 | 1487 sequenc 1488 | 1488 seri 1489 | 1489 serif 1490 | 1490 seriou 1491 | 1491 serv 1492 | 1492 server 1493 | 1493 servic 1494 | 1494 set 1495 | 1495 setup 1496 | 1496 seven 1497 | 1497 seventh 1498 | 1498 sever 1499 | 1499 sex 1500 | 1500 sexual 1501 | 1501 sf 1502 | 1502 shape 1503 | 1503 share 1504 | 1504 she 1505 | 1505 shell 1506 | 1506 ship 1507 | 1507 shop 1508 | 1508 short 1509 | 1509 shot 1510 | 1510 should 1511 | 1511 show 1512 | 1512 side 1513 | 1513 sign 1514 | 1514 signatur 1515 | 1515 signific 1516 | 1516 similar 1517 | 1517 simpl 1518 | 1518 simpli 1519 | 1519 sinc 1520 | 1520 sincer 1521 | 1521 singl 1522 | 1522 sit 1523 | 1523 site 1524 | 1524 situat 1525 | 1525 six 1526 | 1526 size 1527 | 1527 skeptic 1528 | 1528 skill 1529 | 1529 skin 1530 | 1530 skip 1531 | 1531 sleep 1532 | 1532 slow 1533 | 1533 small 1534 | 1534 smart 1535 | 1535 smoke 1536 | 1536 smtp 1537 | 1537 snumber 1538 | 1538 so 1539 | 1539 social 1540 | 1540 societi 1541 | 1541 softwar 1542 | 1542 sold 1543 | 1543 solut 1544 | 1544 solv 1545 | 1545 some 1546 | 1546 someon 1547 | 1547 someth 1548 | 1548 sometim 1549 | 1549 son 1550 | 1550 song 1551 | 1551 soni 1552 | 1552 soon 1553 | 1553 sorri 1554 | 1554 sort 1555 | 1555 sound 1556 | 1556 sourc 1557 | 1557 south 1558 | 1558 space 1559 | 1559 spain 1560 | 1560 spam 1561 | 1561 spamassassin 1562 | 1562 spamd 1563 | 1563 spammer 1564 | 1564 speak 1565 | 1565 spec 1566 | 1566 special 1567 | 1567 specif 1568 | 1568 specifi 1569 | 1569 speech 1570 | 1570 speed 1571 | 1571 spend 1572 | 1572 sponsor 1573 | 1573 sport 1574 | 1574 spot 1575 | 1575 src 1576 | 1576 ssh 1577 | 1577 st 1578 | 1578 stabl 1579 | 1579 staff 1580 | 1580 stai 1581 | 1581 stand 1582 | 1582 standard 1583 | 1583 star 1584 | 1584 start 1585 | 1585 state 1586 | 1586 statement 1587 | 1587 statu 1588 | 1588 step 1589 | 1589 steve 1590 | 1590 still 1591 | 1591 stock 1592 | 1592 stop 1593 | 1593 storag 1594 | 1594 store 1595 | 1595 stori 1596 | 1596 strategi 1597 | 1597 stream 1598 | 1598 street 1599 | 1599 string 1600 | 1600 strip 1601 | 1601 strong 1602 | 1602 structur 1603 | 1603 studi 1604 | 1604 stuff 1605 | 1605 stupid 1606 | 1606 style 1607 | 1607 subject 1608 | 1608 submit 1609 | 1609 subscrib 1610 | 1610 subscript 1611 | 1611 substanti 1612 | 1612 success 1613 | 1613 such 1614 | 1614 suffer 1615 | 1615 suggest 1616 | 1616 suit 1617 | 1617 sum 1618 | 1618 summari 1619 | 1619 summer 1620 | 1620 sun 1621 | 1621 super 1622 | 1622 suppli 1623 | 1623 support 1624 | 1624 suppos 1625 | 1625 sure 1626 | 1626 surpris 1627 | 1627 suse 1628 | 1628 suspect 1629 | 1629 sweet 1630 | 1630 switch 1631 | 1631 system 1632 | 1632 tab 1633 | 1633 tabl 1634 | 1634 tablet 1635 | 1635 tag 1636 | 1636 take 1637 | 1637 taken 1638 | 1638 talk 1639 | 1639 tape 1640 | 1640 target 1641 | 1641 task 1642 | 1642 tax 1643 | 1643 teach 1644 | 1644 team 1645 | 1645 tech 1646 | 1646 technic 1647 | 1647 techniqu 1648 | 1648 technolog 1649 | 1649 tel 1650 | 1650 telecom 1651 | 1651 telephon 1652 | 1652 tell 1653 | 1653 temperatur 1654 | 1654 templ 1655 | 1655 ten 1656 | 1656 term 1657 | 1657 termin 1658 | 1658 terror 1659 | 1659 terrorist 1660 | 1660 test 1661 | 1661 texa 1662 | 1662 text 1663 | 1663 than 1664 | 1664 thank 1665 | 1665 that 1666 | 1666 the 1667 | 1667 thei 1668 | 1668 their 1669 | 1669 them 1670 | 1670 themselv 1671 | 1671 then 1672 | 1672 theori 1673 | 1673 there 1674 | 1674 therefor 1675 | 1675 these 1676 | 1676 thi 1677 | 1677 thing 1678 | 1678 think 1679 | 1679 thinkgeek 1680 | 1680 third 1681 | 1681 those 1682 | 1682 though 1683 | 1683 thought 1684 | 1684 thousand 1685 | 1685 thread 1686 | 1686 threat 1687 | 1687 three 1688 | 1688 through 1689 | 1689 thu 1690 | 1690 thursdai 1691 | 1691 ti 1692 | 1692 ticket 1693 | 1693 tim 1694 | 1694 time 1695 | 1695 tip 1696 | 1696 tire 1697 | 1697 titl 1698 | 1698 tm 1699 | 1699 to 1700 | 1700 todai 1701 | 1701 togeth 1702 | 1702 token 1703 | 1703 told 1704 | 1704 toll 1705 | 1705 tom 1706 | 1706 toner 1707 | 1707 toni 1708 | 1708 too 1709 | 1709 took 1710 | 1710 tool 1711 | 1711 top 1712 | 1712 topic 1713 | 1713 total 1714 | 1714 touch 1715 | 1715 toward 1716 | 1716 track 1717 | 1717 trade 1718 | 1718 tradit 1719 | 1719 traffic 1720 | 1720 train 1721 | 1721 transact 1722 | 1722 transfer 1723 | 1723 travel 1724 | 1724 treat 1725 | 1725 tree 1726 | 1726 tri 1727 | 1727 trial 1728 | 1728 trick 1729 | 1729 trip 1730 | 1730 troubl 1731 | 1731 true 1732 | 1732 truli 1733 | 1733 trust 1734 | 1734 truth 1735 | 1735 try 1736 | 1736 tue 1737 | 1737 tuesdai 1738 | 1738 turn 1739 | 1739 tv 1740 | 1740 two 1741 | 1741 type 1742 | 1742 uk 1743 | 1743 ultim 1744 | 1744 un 1745 | 1745 under 1746 | 1746 understand 1747 | 1747 unfortun 1748 | 1748 uniqu 1749 | 1749 unison 1750 | 1750 unit 1751 | 1751 univers 1752 | 1752 unix 1753 | 1753 unless 1754 | 1754 unlik 1755 | 1755 unlimit 1756 | 1756 unseen 1757 | 1757 unsolicit 1758 | 1758 unsubscrib 1759 | 1759 until 1760 | 1760 up 1761 | 1761 updat 1762 | 1762 upgrad 1763 | 1763 upon 1764 | 1764 urgent 1765 | 1765 url 1766 | 1766 us 1767 | 1767 usa 1768 | 1768 usag 1769 | 1769 usb 1770 | 1770 usd 1771 | 1771 usdollarnumb 1772 | 1772 useless 1773 | 1773 user 1774 | 1774 usr 1775 | 1775 usual 1776 | 1776 util 1777 | 1777 vacat 1778 | 1778 valid 1779 | 1779 valu 1780 | 1780 valuabl 1781 | 1781 var 1782 | 1782 variabl 1783 | 1783 varieti 1784 | 1784 variou 1785 | 1785 ve 1786 | 1786 vendor 1787 | 1787 ventur 1788 | 1788 veri 1789 | 1789 verifi 1790 | 1790 version 1791 | 1791 via 1792 | 1792 video 1793 | 1793 view 1794 | 1794 virtual 1795 | 1795 visa 1796 | 1796 visit 1797 | 1797 visual 1798 | 1798 vnumber 1799 | 1799 voic 1800 | 1800 vote 1801 | 1801 vs 1802 | 1802 vulner 1803 | 1803 wa 1804 | 1804 wai 1805 | 1805 wait 1806 | 1806 wake 1807 | 1807 walk 1808 | 1808 wall 1809 | 1809 want 1810 | 1810 war 1811 | 1811 warm 1812 | 1812 warn 1813 | 1813 warranti 1814 | 1814 washington 1815 | 1815 wasn 1816 | 1816 wast 1817 | 1817 watch 1818 | 1818 water 1819 | 1819 we 1820 | 1820 wealth 1821 | 1821 weapon 1822 | 1822 web 1823 | 1823 weblog 1824 | 1824 websit 1825 | 1825 wed 1826 | 1826 wednesdai 1827 | 1827 week 1828 | 1828 weekli 1829 | 1829 weight 1830 | 1830 welcom 1831 | 1831 well 1832 | 1832 went 1833 | 1833 were 1834 | 1834 west 1835 | 1835 what 1836 | 1836 whatev 1837 | 1837 when 1838 | 1838 where 1839 | 1839 whether 1840 | 1840 which 1841 | 1841 while 1842 | 1842 white 1843 | 1843 whitelist 1844 | 1844 who 1845 | 1845 whole 1846 | 1846 whose 1847 | 1847 why 1848 | 1848 wi 1849 | 1849 wide 1850 | 1850 width 1851 | 1851 wife 1852 | 1852 will 1853 | 1853 william 1854 | 1854 win 1855 | 1855 window 1856 | 1856 wing 1857 | 1857 winner 1858 | 1858 wireless 1859 | 1859 wish 1860 | 1860 with 1861 | 1861 within 1862 | 1862 without 1863 | 1863 wnumberp 1864 | 1864 woman 1865 | 1865 women 1866 | 1866 won 1867 | 1867 wonder 1868 | 1868 word 1869 | 1869 work 1870 | 1870 worker 1871 | 1871 world 1872 | 1872 worldwid 1873 | 1873 worri 1874 | 1874 worst 1875 | 1875 worth 1876 | 1876 would 1877 | 1877 wouldn 1878 | 1878 write 1879 | 1879 written 1880 | 1880 wrong 1881 | 1881 wrote 1882 | 1882 www 1883 | 1883 ximian 1884 | 1884 xml 1885 | 1885 xp 1886 | 1886 yahoo 1887 | 1887 ye 1888 | 1888 yeah 1889 | 1889 year 1890 | 1890 yesterdai 1891 | 1891 yet 1892 | 1892 york 1893 | 1893 you 1894 | 1894 young 1895 | 1895 your 1896 | 1896 yourself 1897 | 1897 zdnet 1898 | 1898 zero 1899 | 1899 zip 1900 | -------------------------------------------------------------------------------- /images/AnomalyDetection_01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/images/AnomalyDetection_01.png -------------------------------------------------------------------------------- /images/LinearRegression_01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/images/LinearRegression_01.png -------------------------------------------------------------------------------- /images/LogisticRegression_01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/images/LogisticRegression_01.png -------------------------------------------------------------------------------- /images/LogisticRegression_02.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/images/LogisticRegression_02.png -------------------------------------------------------------------------------- /images/PCA_01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/images/PCA_01.png -------------------------------------------------------------------------------- /images/PCA_02.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/images/PCA_02.png -------------------------------------------------------------------------------- /images/svm_01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/images/svm_01.png -------------------------------------------------------------------------------- /images/svm_02.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/images/svm_02.png -------------------------------------------------------------------------------- /images/svm_03.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/images/svm_03.png -------------------------------------------------------------------------------- /images/svm_04.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lawlite19/MachineLearningAlgorithm/9b1787f1b1b78cb61dc717ae14084c4a05e60dad/images/svm_04.png -------------------------------------------------------------------------------- /readme.md: -------------------------------------------------------------------------------- 1 | 机器学习算法 2 | ============= 3 | 4 | ### [一、线性回归][1] 5 | - 梯度下降 6 | - 归一化 7 | - 正规方程 8 | 9 | ### [二、逻辑回归][2] 10 | - 梯度下降优化函数 11 | - 正则化防止过拟合 12 | 13 | ### [三、BP神经网络][3] 14 | - 反向传播 15 | 16 | ### [四、支持向量机SVM][4] 17 | - 线性核函数 18 | - 高斯核函数 19 | - 垃圾邮件预测 20 | 21 | ### [五、K-Means聚类][5] 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | [1]:./LinearRegression/readme.md 38 | [2]:./LogisticRegression/readme.md 39 | [3]:./NeuralNetwork/readme.md 40 | [4]:./SVM/readme.md 41 | [5]:./K-Means/readme.md 42 | --------------------------------------------------------------------------------