├── README.md ├── RvmModel ├── BaseRVM.m ├── Kernel.m ├── RvmOptimization.m ├── RvmOptimizationOption.m └── RvmOption.m ├── SB2_Release_200 ├── Readme.txt ├── SB2_ControlSettings.m ├── SB2_Diagnostic.m ├── SB2_FormatTime.m ├── SB2_FullStatistics.m ├── SB2_Initialisation.m ├── SB2_Likelihoods.m ├── SB2_Manual.pdf ├── SB2_ParameterSettings.m ├── SB2_PosteriorMode.m ├── SB2_PreProcessBasis.m ├── SB2_Sigmoid.m ├── SB2_UserOptions.m ├── SparseBayes.m ├── SparseBayesDemo.m └── licence.txt ├── custom_option.m ├── data ├── UCI_data.mat └── sinc_data.mat ├── demo_RVC.m ├── demo_RVR.m ├── demo_hybrid_kernel.m ├── demo_hybrid_kernel_optimization.m ├── demo_kernel_function.m └── demo_optimization.m /README.md: -------------------------------------------------------------------------------- 1 |

2 | 3 |

4 | 5 |

Relevance Vector Machine (RVM)

6 | 7 |

MATLAB code for Relevance Vector Machine

8 |

Version 2.1, 31-AUG-2021

9 |

Email: iqiukp@outlook.com

10 | 11 |

12 | 13 | [![View Relevance Vector Machine (RVM) on File Exchange](https://www.mathworks.com/matlabcentral/images/matlab-file-exchange.svg)](https://www.mathworks.com/matlabcentral/fileexchange/69407-relevance-vector-machine-rvm) 14 |

15 |

16 |

17 |

18 |

19 |

20 |

21 | 22 |

23 | 24 | ## Main features 25 | 26 | - RVM model for binary classification (RVC) or regression (RVR) 27 | - Multiple kinds of kernel functions (linear, gaussian, polynomial, sigmoid, laplacian) 28 | - Hybrid kernel functions (K =w1×K1+w2×K2+...+wn×Kn) 29 | - Parameter Optimization using Bayesian optimization, Genetic Algorithm, and Particle Swarm Optimization 30 | 31 | ## Notices 32 | 33 | - This version of the code is not compatible with the versions lower than R2016b. 34 | - Detailed applications please see the demonstrations. 35 | - This code is for reference only. 36 | 37 | ## Citation 38 | ``` 39 | @article{tipping2001sparse, 40 | title={Sparse Bayesian learning and the relevance vector machine}, 41 | author={Tipping, Michael E}, 42 | journal={Journal of machine learning research}, 43 | volume={1}, 44 | number={Jun}, 45 | pages={211--244}, 46 | year={2001} 47 | } 48 | ``` 49 | ``` 50 | @article{qiu2021soft, 51 | title={Soft sensor development based on kernel dynamic time warping and a relevant vector machine for unequal-length batch processes}, 52 | author={Qiu, Kepeng and Wang, Jianlin and Wang, Rutong and Guo, Yongqi and Zhao, Liqiang}, 53 | journal={Expert Systems with Applications}, 54 | volume={182}, 55 | pages={115223}, 56 | year={2021}, 57 | publisher={Elsevier} 58 | } 59 | ``` 60 | 61 | 62 | ## How to use 63 | 64 | ### 01. Classification using RVM (RVC) 65 | 66 | A demo for classification using RVM 67 | ```MATLAB 68 | clc 69 | clear all 70 | close all 71 | addpath(genpath(pwd)) 72 | 73 | % use fisheriris dataset 74 | load fisheriris 75 | inds = ~strcmp(species, 'setosa'); 76 | data_ = meas(inds, 3:4); 77 | label_ = species(inds); 78 | cvIndices = crossvalind('HoldOut', length(data_), 0.3); 79 | trainData = data_(cvIndices, :); 80 | trainLabel = label_(cvIndices, :); 81 | testData = data_(~cvIndices, :); 82 | testLabel = label_(~cvIndices, :); 83 | 84 | % kernel function 85 | kernel = Kernel('type', 'gaussian', 'gamma', 0.2); 86 | 87 | % parameter 88 | parameter = struct( 'display', 'on',... 89 | 'type', 'RVC',... 90 | 'kernelFunc', kernel); 91 | rvm = BaseRVM(parameter); 92 | 93 | % RVM model training, testing, and visualization 94 | rvm.train(trainData, trainLabel); 95 | results = rvm.test(testData, testLabel); 96 | rvm.draw(results) 97 | ``` 98 | 99 | results: 100 | ```MATLAB 101 | *** RVM model (classification) train finished *** 102 | running time = 0.1604 seconds 103 | iterations = 20 104 | number of samples = 70 105 | number of RVs = 2 106 | ratio of RVs = 2.8571% 107 | accuracy = 94.2857% 108 | 109 | 110 | *** RVM model (classification) test finished *** 111 | running time = 0.0197 seconds 112 | number of samples = 30 113 | accuracy = 96.6667% 114 | ``` 115 | 116 |

117 | 118 | 119 |

174 | 175 |

176 | 177 | 178 | ### 03. Kernel funcions 179 | 180 | A class named ***Kernel*** is defined to compute kernel function matrix. 181 | ```MATLAB 182 | %{ 183 | type - 184 | 185 | linear : k(x,y) = x'*y 186 | polynomial : k(x,y) = (γ*x'*y+c)^d 187 | gaussian : k(x,y) = exp(-γ*||x-y||^2) 188 | sigmoid : k(x,y) = tanh(γ*x'*y+c) 189 | laplacian : k(x,y) = exp(-γ*||x-y||) 190 | 191 | 192 | degree - d 193 | offset - c 194 | gamma - γ 195 | %} 196 | kernel = Kernel('type', 'gaussian', 'gamma', value); 197 | kernel = Kernel('type', 'polynomial', 'degree', value); 198 | kernel = Kernel('type', 'linear'); 199 | kernel = Kernel('type', 'sigmoid', 'gamma', value); 200 | kernel = Kernel('type', 'laplacian', 'gamma', value); 201 | ``` 202 | For example, compute the kernel matrix between **X** and **Y** 203 | ```MATLAB 204 | X = rand(5, 2); 205 | Y = rand(3, 2); 206 | kernel = Kernel('type', 'gaussian', 'gamma', 2); 207 | kernelMatrix = kernel.computeMatrix(X, Y); 208 | >> kernelMatrix 209 | 210 | kernelMatrix = 211 | 212 | 0.5684 0.5607 0.4007 213 | 0.4651 0.8383 0.5091 214 | 0.8392 0.7116 0.9834 215 | 0.4731 0.8816 0.8052 216 | 0.5034 0.9807 0.7274 217 | ``` 218 | 219 | ### 04. Hybrid kernel 220 | 221 | A demo for regression using RVM with hybrid_kernel (K =w1×K1+w2×K2+...+wn×Kn) 222 | ```MATLAB 223 | clc 224 | clear all 225 | close all 226 | addpath(genpath(pwd)) 227 | 228 | % sinc funciton 229 | load sinc_data 230 | trainData = x; 231 | trainLabel = y; 232 | testData = xt; 233 | testLabel = yt; 234 | 235 | % kernel function 236 | kernel_1 = Kernel('type', 'gaussian', 'gamma', 0.3); 237 | kernel_2 = Kernel('type', 'polynomial', 'degree', 2); 238 | kernelWeight = [0.5, 0.5]; 239 | % parameter 240 | parameter = struct( 'display', 'on',... 241 | 'type', 'RVR',... 242 | 'kernelFunc', [kernel_1, kernel_2],... 243 | 'kernelWeight', kernelWeight); 244 | rvm = BaseRVM(parameter); 245 | 246 | % RVM model training, testing, and visualization 247 | rvm.train(trainData, trainLabel); 248 | results = rvm.test(testData, testLabel); 249 | rvm.draw(results) 250 | ``` 251 | 252 | ### 05. Parameter Optimization for single-kernel-RVM 253 | 254 | A demo for RVM model with Parameter Optimization 255 | 256 | ```MATLAB 257 | clc 258 | clear all 259 | close all 260 | addpath(genpath(pwd)) 261 | 262 | % use fisheriris dataset 263 | load fisheriris 264 | inds = ~strcmp(species, 'setosa'); 265 | data_ = meas(inds, 3:4); 266 | label_ = species(inds); 267 | cvIndices = crossvalind('HoldOut', length(data_), 0.3); 268 | trainData = data_(cvIndices, :); 269 | trainLabel = label_(cvIndices, :); 270 | testData = data_(~cvIndices, :); 271 | testLabel = label_(~cvIndices, :); 272 | 273 | % kernel function 274 | kernel = Kernel('type', 'gaussian', 'gamma', 5); 275 | 276 | % parameter optimization 277 | opt.method = 'bayes'; % bayes, ga, pso 278 | opt.display = 'on'; 279 | opt.iteration = 20; 280 | 281 | % parameter 282 | parameter = struct( 'display', 'on',... 283 | 'type', 'RVC',... 284 | 'kernelFunc', kernel,... 285 | 'optimization', opt); 286 | rvm = BaseRVM(parameter); 287 | 288 | % RVM model training, testing, and visualization 289 | rvm.train(trainData, trainLabel); 290 | results = rvm.test(trainData, trainLabel); 291 | rvm.draw(results) 292 | 293 | ``` 294 | 295 | results: 296 | ```MATLAB 297 | *** RVM model (classification) train finished *** 298 | running time = 13.3356 seconds 299 | iterations = 88 300 | number of samples = 70 301 | number of RVs = 4 302 | ratio of RVs = 5.7143% 303 | accuracy = 97.1429% 304 | Optimized parameter table 305 | 306 | gaussian_gamma 307 | ______________ 308 | 309 | 7.8261 310 | 311 | *** RVM model (classification) test finished *** 312 | running time = 0.0195 seconds 313 | number of samples = 70 314 | accuracy = 97.1429% 315 | ``` 316 | 317 |

318 | 319 | 320 |

321 | 322 | 323 | ### 06. Parameter Optimization for hybrid-kernel-RVM 324 | 325 | A demo for RVM model with Parameter Optimization 326 | 327 | ```MATLAB 328 | %{ 329 | A demo for hybrid-kernel RVM model with Parameter Optimization 330 | %} 331 | 332 | 333 | clc 334 | clear all 335 | close all 336 | addpath(genpath(pwd)) 337 | 338 | % data 339 | load UCI_data 340 | trainData = x; 341 | trainLabel = y; 342 | testData = xt; 343 | testLabel = yt; 344 | 345 | % kernel function 346 | kernel_1 = Kernel('type', 'gaussian', 'gamma', 0.5); 347 | kernel_2 = Kernel('type', 'polynomial', 'degree', 2); 348 | 349 | % parameter optimization 350 | opt.method = 'bayes'; % bayes, ga, pso 351 | opt.display = 'on'; 352 | opt.iteration = 30; 353 | 354 | % parameter 355 | parameter = struct( 'display', 'on',... 356 | 'type', 'RVR',... 357 | 'kernelFunc', [kernel_1, kernel_2],... 358 | 'optimization', opt); 359 | rvm = BaseRVM(parameter); 360 | 361 | % RVM model training, testing, and visualization 362 | rvm.train(trainData, trainLabel); 363 | results = rvm.test(testData, testLabel); 364 | rvm.draw(results) 365 | 366 | ``` 367 | 368 | results: 369 | ```MATLAB 370 | *** RVM model (regression) train finished *** 371 | running time = 24.4042 seconds 372 | iterations = 377 373 | number of samples = 264 374 | number of RVs = 22 375 | ratio of RVs = 8.3333% 376 | RMSE = 0.4864 377 | R2 = 0.7719 378 | MAE = 0.3736 379 | Optimized parameter 1×6 table 380 | 381 | gaussian_gamma polynomial_gamma polynomial_offset polynomial_degree gaussian_weight polynomial_weight 382 | ______________ ________________ _________________ _________________ _______________ _________________ 383 | 384 | 22.315 13.595 44.83 6 0.042058 0.95794 385 | 386 | 387 | 388 | 389 | *** RVM model (regression) test finished *** 390 | running time = 0.0008 seconds 391 | number of samples = 112 392 | RMSE = 0.7400 393 | R2 = 0.6668 394 | MAE = 0.4867 395 | ``` 396 |

397 | 398 |

399 | 400 | 401 | ### 07. Cross Validation 402 | 403 | In this code, two cross-validation methods are supported: 'K-Folds' and 'Holdout'. 404 | For example, the cross-validation of 5-Folds is 405 | ```MATLAB 406 | parameter = struct( 'display', 'on',... 407 | 'type', 'RVC',... 408 | 'kernelFunc', kernel,... 409 | 'KFold', 5); 410 | ``` 411 | For example, the cross-validation of the Holdout method with a ratio of 0.3 is 412 | ```MATLAB 413 | parameter = struct( 'display', 'on',... 414 | 'type', 'RVC',... 415 | 'kernelFunc', kernel,... 416 | 'HoldOut', 0.3); 417 | ``` 418 | 419 | ### 08. Other option 420 | ```MATLAB 421 | %% custom optimization option 422 | %{ 423 | opt.method = 'bayes'; % bayes, ga, pso 424 | opt.display = 'on'; 425 | opt.iteration = 20; 426 | opt.point = 10; 427 | 428 | % gaussian kernel function 429 | opt.gaussian.parameterName = {'gamma'}; 430 | opt.gaussian.parameterType = {'real'}; 431 | opt.gaussian.lowerBound = 2^-6; 432 | opt.gaussian.upperBound = 2^6; 433 | 434 | % laplacian kernel function 435 | opt.laplacian.parameterName = {'gamma'}; 436 | opt.laplacian.parameterType = {'real'}; 437 | opt.laplacian.lowerBound = 2^-6; 438 | opt.laplacian.upperBound = 2^6; 439 | 440 | % polynomial kernel function 441 | opt.polynomial.parameterName = {'gamma'; 'offset'; 'degree'}; 442 | opt.polynomial.parameterType = {'real'; 'real'; 'integer'}; 443 | opt.polynomial.lowerBound = [2^-6; 2^-6; 1]; 444 | opt.polynomial.upperBound = [2^6; 2^6; 7]; 445 | 446 | % sigmoid kernel function 447 | opt.sigmoid.parameterName = {'gamma'; 'offset'}; 448 | opt.sigmoid.parameterType = {'real'; 'real'}; 449 | opt.sigmoid.lowerBound = [2^-6; 2^-6]; 450 | opt.sigmoid.upperBound = [2^6; 2^6]; 451 | %} 452 | 453 | %% RVM model parameter 454 | %{ 455 | 'display' : 'on', 'off' 456 | 'type' : 'RVR', 'RVC' 457 | 'kernelFunc' : kernel function 458 | 'KFolds' : cross validation, for example, 5 459 | 'HoldOut' : cross validation, for example, 0.3 460 | 'freeBasis' : 'on', 'off' 461 | 'maxIter' : max iteration, for example, 1000 462 | %} 463 | ``` 464 | 465 | 466 | 467 | 468 | -------------------------------------------------------------------------------- /RvmModel/BaseRVM.m: -------------------------------------------------------------------------------- 1 | classdef BaseRVM < handle & matlab.mixin.Copyable 2 | %{ 3 | Version 2.1, 31-AUG-2021 4 | Email: iqiukp@outlook.com 5 | %} 6 | 7 | properties 8 | data 9 | label 10 | label_ 11 | type = 'RVR' % classification regression 12 | kernelFunc = Kernel('type', 'gaussian', 'gamma', 0.5) 13 | kernelFuncName 14 | kernelType = 'single' 15 | kernelWeight 16 | relevanceVectors 17 | relevanceVectorIndices 18 | numRelevanceVectors 19 | numIterations 20 | numKernel 21 | rationRelevanceVectors 22 | alpha 23 | alphaTolerance = 1e-6 24 | relevanceVectorAlpha 25 | freeBasis = 'off' 26 | runningTime 27 | maxIter = 1000 28 | bias 29 | sigma 30 | beta 31 | PHI 32 | weight 33 | predictedLabel 34 | predictedLabel_ 35 | display = 'on' 36 | optimization 37 | crossValidation 38 | performance 39 | evaluationMode = 'test' 40 | classNames 41 | end 42 | 43 | properties (Dependent) 44 | numSamples 45 | numFeatures 46 | end 47 | 48 | methods 49 | % create an object of RVM 50 | function obj = BaseRVM(parameter) 51 | RvmOption.setParameter(obj, parameter); 52 | end 53 | 54 | % train RVM model 55 | function varargout = train(obj, data, label) % varargin 56 | tStart = tic; 57 | obj.label = label; 58 | obj.data = data; 59 | obj.classNames = unique(label); 60 | if strcmp(obj.type, 'RVC') 61 | obj.label_ = RvmOption.checkLabel(obj, label); 62 | else 63 | obj.label_ = label; 64 | end 65 | % parameter optimization 66 | if strcmp(obj.optimization.switch, 'on') 67 | RvmOptimization.getModel(obj); 68 | else 69 | getModel(obj); 70 | end 71 | % % cross validation 72 | % if strcmp(obj.crossValidation.switch, 'on') 73 | % rvm_ = copy(obj); 74 | % obj.crossValidation.accuracy = RvmOptimization.crossvalFunc(rvm_); 75 | % end 76 | 77 | obj.runningTime = toc(tStart); 78 | % display 79 | if strcmp(obj.display, 'on') 80 | RvmOption.displayTrain(obj); 81 | end 82 | % output 83 | if nargout == 1 84 | varargout{1} = obj; 85 | end 86 | end 87 | 88 | function getModel(obj) 89 | switch obj.type 90 | case 'RVC' 91 | likelihoodFunc = 'Bernoulli'; 92 | case 'RVR' 93 | likelihoodFunc = 'Gaussian'; 94 | end 95 | 96 | switch obj.kernelType 97 | case 'single' 98 | K = obj.kernelFunc.computeMatrix(obj.data, obj.data); 99 | case 'hybrid' 100 | K = 0; 101 | for i = 1:obj.numKernel 102 | K = K+obj.kernelWeight(i)*obj.kernelFunc(i).computeMatrix(obj.data, obj.data); 103 | end 104 | end 105 | 106 | switch obj.freeBasis 107 | case 'off' 108 | % no bias 109 | BASIS = K; 110 | options_ = SB2_UserOptions('iterations', obj.maxIter); 111 | case 'on' 112 | BASIS = [K, ones(obj.numSamples, 1)]; 113 | options_ = SB2_UserOptions('freeBasis', obj.numSamples+1, 'iterations', obj.maxIter); 114 | end 115 | 116 | % model 117 | [PARAMETER, HYPERPARAMETER, DIAGNOSTIC] =... 118 | SparseBayes(likelihoodFunc, BASIS, obj.label_, options_); 119 | 120 | % store the model 121 | obj.relevanceVectorIndices = PARAMETER.Relevant; % index of relevance vectors 122 | 123 | switch obj.freeBasis 124 | case 'off' 125 | obj.bias = 0; % value of bias 126 | case 'on' 127 | obj.bias = PARAMETER.Value(end); % value of bias 128 | obj.relevanceVectorIndices(obj.relevanceVectorIndices == obj.numSamples+1) = []; 129 | end 130 | 131 | obj.weight = PARAMETER.Value; % vector of weight values 132 | obj.beta = HYPERPARAMETER.beta; % noise precision 133 | obj.alpha = HYPERPARAMETER.Alpha; % vector of weight precision values 134 | obj.PHI = BASIS(:, PARAMETER.Relevant); % Used BASIS 135 | obj.numIterations = DIAGNOSTIC.iterations; % iteration 136 | obj.numRelevanceVectors = size(obj.relevanceVectorIndices, 1); % number of relevance vectors 137 | obj.rationRelevanceVectors = obj.numRelevanceVectors/obj.numSamples; 138 | obj.relevanceVectors = obj.data(obj.relevanceVectorIndices, :); % relevance vectors 139 | 140 | switch obj.type 141 | case 'RVC' 142 | obj.predictedLabel_ = zeros(obj.numSamples, 1); 143 | index_ = SB2_Sigmoid(obj.PHI*obj.weight)>0.5; 144 | obj.predictedLabel_(index_) = 1; 145 | obj.predictedLabel = RvmOption.restoreLabel(obj.classNames, obj.predictedLabel_); 146 | case 'RVR' 147 | obj.sigma = inv(diag(obj.alpha)+obj.beta*(obj.PHI'*obj.PHI)); 148 | obj.predictedLabel = obj.PHI*obj.weight; 149 | end 150 | 151 | % model evaluation 152 | display_ = obj.display; 153 | evaluationMode_ = obj.evaluationMode; 154 | obj.display = 'off'; 155 | obj.evaluationMode = 'train'; 156 | results_ = test(obj, obj.data, obj.label_); 157 | obj.display = display_; 158 | obj.evaluationMode = evaluationMode_; 159 | obj.performance = evaluateModel(obj, results_); 160 | end 161 | 162 | function results = test(obj, data, label) 163 | tStart = tic; 164 | results.type = obj.type; 165 | results.label = label; 166 | results.label_ = label; 167 | results.data = data; 168 | results.classNames = unique(label); 169 | if strcmp(obj.type, 'RVC') && strcmp(obj.evaluationMode,'test') 170 | results.label_ = RvmOption.checkLabel(results, label); 171 | end 172 | results.numSamples = size(results.data, 1); 173 | 174 | switch obj.kernelType 175 | case 'single' 176 | K = obj.kernelFunc.computeMatrix(results.data, obj.relevanceVectors); 177 | case 'hybrid' 178 | K = 0; 179 | for i = 1:obj.numKernel 180 | K = K+obj.kernelWeight(i)*... 181 | obj.kernelFunc(i).computeMatrix(results.data, obj.relevanceVectors); 182 | end 183 | end 184 | 185 | switch obj.freeBasis 186 | case 'off' 187 | BASIS = K; 188 | case 'on' 189 | BASIS = [K, ones(size(results.data, 1), 1)]; 190 | end 191 | 192 | switch obj.type 193 | case 'RVC' 194 | results.predictedLabel_ = zeros(size(results.data, 1), 1); 195 | index_ = SB2_Sigmoid(BASIS*obj.weight)>0.5; 196 | results.predictedLabel_(index_) = 1; 197 | results.predictedLabel = RvmOption.restoreLabel... 198 | (results.classNames, results.predictedLabel_); 199 | case 'RVR' 200 | results.predictedLabel = BASIS*obj.weight; 201 | results.predictedVariance = obj.beta^-1+diag(BASIS*obj.sigma*BASIS'); 202 | end 203 | results.performance = obj.evaluateModel(results); 204 | results.runningTime = toc(tStart); 205 | 206 | % display 207 | if strcmp(obj.display, 'on') 208 | RvmOption.displayTest(results); 209 | end 210 | end 211 | 212 | function performance = evaluateModel(obj, results) 213 | switch obj.type 214 | case 'RVC' 215 | performance.accuracy = sum(results.predictedLabel_ == results.label_)/results.numSamples; 216 | [performance.confusionMatrix, performance.classOrder] = confusionmat... 217 | (results.label, results.predictedLabel); 218 | case 'RVR' 219 | performance.RMSE= sqrt(sum((results.label-results.predictedLabel).^2)/results.numSamples); 220 | SSR = sum((results.predictedLabel-mean(results.predictedLabel)).^2); 221 | SSE = sum((results.label-results.predictedLabel).^2); 222 | SST = SSR+SSE; 223 | performance.R2 = 1-SSE./SST; 224 | performance.MAE = sum(abs(results.label-results.predictedLabel))/results.numSamples; 225 | end 226 | end 227 | 228 | function draw(obj, results) 229 | switch obj.type 230 | case 'RVR' 231 | RvmOption.drawRVR(results); 232 | case 'RVC' 233 | RvmOption.drawRVC(results); 234 | end 235 | end 236 | 237 | function numSamples = get.numSamples(obj) 238 | numSamples= size(obj.data, 1); 239 | end 240 | 241 | function numFeatures = get.numFeatures(obj) 242 | numFeatures= size(obj.data, 2); 243 | end 244 | end 245 | end -------------------------------------------------------------------------------- /RvmModel/Kernel.m: -------------------------------------------------------------------------------- 1 | classdef Kernel < handle 2 | %{ 3 | Kernel function 4 | 5 | INPUT 6 | X data (n*d) 7 | Y data (m*d) 8 | 9 | OUTPUT 10 | K kernel matrix (n*m) 11 | 12 | ---------------------------------------------------------------------- 13 | 14 | type - 15 | 16 | linear : k(x,y) = x'*y 17 | polynomial : k(x,y) = (γ*x'*y+c)^d 18 | gaussian : k(x,y) = exp(-γ*||x-y||^2) 19 | sigmoid : k(x,y) = tanh(γ*x'*y+c) 20 | laplacian : k(x,y) = exp(-γ*||x-y||) 21 | 22 | 23 | degree - d 24 | offset - c 25 | gamma - γ 26 | 27 | ---------------------------------------------------------------------- 28 | 29 | Version 1.1, 11-MAY-2021 30 | Email: iqiukp@outlook.com 31 | ---------------------------------------------------------------------- 32 | %} 33 | 34 | properties 35 | type = 'gaussian' 36 | offset = 0 37 | gamma = 0.1 38 | degree = 2 39 | end 40 | 41 | methods 42 | function obj = Kernel(varargin) 43 | inputValue = varargin; 44 | nParameter = size(inputValue, 2)/2; 45 | supportedKernelFunc = {'linear', 'gaussian', 'polynomial', 'sigmoid', 'laplacian'}; 46 | for n = 1:nParameter 47 | parameter = inputValue{(n-1)*2+1}; 48 | value = inputValue{(n-1)*2+2}; 49 | if strcmp(parameter, 'type') 50 | if ~any(strcmp(value, supportedKernelFunc)) 51 | errorText = sprintf([ 52 | 'Unsupported kernel function.\n',... 53 | 'Use one of these kernel functions:\n', ... 54 | 'linear, gaussian, polynomial, sigmoid, laplacian.']); 55 | error(errorText) 56 | end 57 | end 58 | obj.(parameter) = value; 59 | end 60 | end 61 | 62 | function K = computeMatrix(obj, x, y) 63 | K = zeros(size(x, 1), size(y, 1)); 64 | % compute the kernel matrix 65 | switch obj.type 66 | case 'linear' % linear kernel function 67 | K = x*y'; 68 | 69 | case 'gaussian' % gaussian kernel function 70 | try 71 | K = exp(-obj.gamma*pdist2(x, y, 'squaredeuclidean')); 72 | catch 73 | sx = sum(x.^2, 2); 74 | sy = sum(y.^2, 2); 75 | xy = 2*x*y'; 76 | K = exp((bsxfun(@minus, bsxfun(@minus, xy, sx), sy'))*obj.gamma); 77 | end 78 | 79 | case 'polynomial' % polynomial kernel function 80 | K = (obj.gamma*x*y'+obj.offset).^double(obj.degree); 81 | 82 | case 'sigmoid' % sigmoid kernel function 83 | K = tanh(obj.gamma*x*y'+obj.offset); 84 | 85 | case 'laplacian' % laplacian kernel function 86 | K = exp(-obj.gamma*pdist2(x, y, 'cityblock')); 87 | end 88 | end 89 | end 90 | end -------------------------------------------------------------------------------- /RvmModel/RvmOptimization.m: -------------------------------------------------------------------------------- 1 | classdef RvmOptimization < handle 2 | %{ 3 | CLASS DESCRIPTION 4 | 5 | Parameter optimization for RVM model. 6 | 7 | -------------------------------------------------------------- 8 | %} 9 | 10 | methods (Static) 11 | function getModel(rvm) 12 | RvmOptimizationOption.checkParameter(rvm); 13 | objFun = @(parameter) RvmOptimization.getObjValue(parameter, rvm); 14 | RvmOptimizationOption.constructParamTable(rvm); 15 | switch rvm.optimization.method 16 | case 'bayes' 17 | RvmOptimization.bayesopt(rvm, objFun); 18 | case 'ga' 19 | RvmOptimization.gaopt(rvm, objFun); 20 | case 'pso' 21 | RvmOptimization.psoopt(rvm, objFun); 22 | end 23 | RvmOptimizationOption.setParameter(rvm); 24 | rvm.getModel; 25 | end 26 | 27 | function bayesopt(rvm, objFun) 28 | %{ 29 | Optimize the parameters using Bayesian optimization. 30 | For detailed introduction of the algorithm and parameter 31 | setting, please enter 'help bayesopt' in the command line. 32 | %} 33 | parameter = RvmOptimizationOption.setParameterForBayes(rvm); 34 | switch rvm.optimization.display 35 | case 'on' 36 | plotFcn_ = {@plotObjectiveModel, @plotMinObjective}; 37 | case 'off' 38 | plotFcn_ = []; 39 | end 40 | results = bayesopt(objFun, parameter, 'Verbose', 1,... 41 | 'MaxObjectiveEvaluations', rvm.optimization.iteration,... 42 | 'NumSeedPoints', rvm.optimization.point, 'PlotFcn', plotFcn_); 43 | % optimization results 44 | [rvm.optimization.bestParam, ~, ~] = bestPoint(results); 45 | end 46 | 47 | function gaopt(rvm, objFun) 48 | %{ 49 | Optimize the parameters using Genetic Algorithm (GA) 50 | For detailed introduction of the algorithm and parameter 51 | setting, please enter 'help ga' in the command line. 52 | %} 53 | seedSize = 10*rvm.optimization.numParameter; 54 | [lowerBound_, upperBound_] = RvmOptimizationOption.setParameterForPsoAndGa(rvm); 55 | try 56 | switch rvm.optimization.display 57 | case 'on' 58 | plotFcn_ = 'gaplotbestf'; 59 | case 'off' 60 | plotFcn_ = []; 61 | end 62 | options = optimoptions('ga', 'PopulationSize', seedSize,... 63 | 'MaxGenerations', rvm.optimization.iteration,... 64 | 'Display', 'iter', 'PlotFcn', plotFcn_); 65 | bestParam_ = ga(objFun, rvm.optimization.numParameter, [], [], [], [],... 66 | lowerBound_, upperBound_, [], [], options); 67 | catch % older vision 68 | switch rvm.optimization.display 69 | case 'on' 70 | plotFcn_ = @gaplotbestf; 71 | case 'off' 72 | plotFcn_ = []; 73 | end 74 | options = optimoptions('ga', 'PopulationSize', seedSize,... 75 | 'MaxGenerations', rvm.optimization.iteration,... 76 | 'Display', 'iter', 'PlotFcn', plotFcn_); 77 | bestParam_ = ga(objFun, rvm.optimization.numParameter, [], [], [], [],... 78 | lowerBound_, upperBound_, [], [], options); 79 | end 80 | % optimization results 81 | rvm.optimization.bestParam.Variables = bestParam_; 82 | end 83 | 84 | function psoopt(rvm, objFun) 85 | %{ 86 | Optimize the parameters using Particle Swarm Optimization (PSO) 87 | For detailed introduction of the algorithm and parameter 88 | setting, please enter 'help particleswarm' in the command line. 89 | %} 90 | seedSize = 10*rvm.optimization.numParameter; 91 | switch rvm.optimization.display 92 | case 'on' 93 | plotFcn_ = 'pswplotbestf'; 94 | case 'off' 95 | plotFcn_ = []; 96 | end 97 | options = optimoptions('particleswarm', 'SwarmSize', seedSize,... 98 | 'MaxIterations', rvm.optimization.iteration,... 99 | 'Display', 'iter', 'PlotFcn', plotFcn_); 100 | [lowerBound_, upperBound_] = RvmOptimizationOption.setParameterForPsoAndGa(rvm); 101 | bestParam_ = particleswarm(objFun, rvm.optimization.numParameter,... 102 | lowerBound_, upperBound_, options); 103 | % optimization results 104 | rvm.optimization.bestParam.Variables = bestParam_; 105 | end 106 | 107 | function objValue = getObjValue(parameter, rvm) 108 | %{ 109 | Compute the value of objective function 110 | %} 111 | rvm_ = copy(rvm); 112 | rvm_.display = 'off'; 113 | switch class(parameter) 114 | case 'table' % bayes 115 | rvm_.optimization.bestParam = parameter; 116 | case 'double' % ga, pso 117 | rvm_.optimization.bestParam.Variables = parameter; 118 | end 119 | % parameter setting 120 | RvmOptimizationOption.setParameter(rvm_); 121 | % cross validation 122 | if strcmp(rvm_.crossValidation.switch, 'on') 123 | objValue = RvmOptimization.crossvalFunc(rvm_); 124 | else 125 | % train with all samples 126 | rvm_.getModel; 127 | evaluationMode_ = rvm_.evaluationMode; 128 | rvm_.evaluationMode = 'train'; 129 | results_ = test(rvm_, rvm_.data, rvm_.label_); 130 | rvm_.performance = rvm_.evaluateModel(results_); 131 | switch rvm_.type 132 | case 'RVR' 133 | objValue = rvm_.performance.RMSE; 134 | case 'RVC' 135 | objValue = 1-rvm_.performance.accuracy; 136 | end 137 | rvm_.evaluationMode = evaluationMode_; 138 | end 139 | end 140 | 141 | function accuracy = crossvalFunc(rvm) 142 | %{ 143 | Compute the cross validation accuracy 144 | %} 145 | rng('default') 146 | rvm_ = copy(rvm); 147 | data_ = rvm_.data; 148 | label_ = rvm_.label_; 149 | rvm_.display = 'off'; 150 | rvm_.evaluationMode = 'train'; 151 | cvIndices = crossvalind(rvm.crossValidation.method, ... 152 | rvm_.numSamples, rvm.crossValidation.param); 153 | switch rvm.crossValidation.method 154 | case 'KFold' 155 | accuracy_ = Inf(rvm.crossValidation.param, 1); 156 | for i = 1:rvm.crossValidation.param 157 | testIndices = (cvIndices == i); 158 | testData = data_(testIndices, :); 159 | testLabel = label_(testIndices, :); 160 | rvm_.data = data_(~testIndices, :); 161 | rvm_.label_ = label_(~testIndices, :); 162 | try 163 | rvm_.getModel; 164 | catch 165 | continue 166 | end 167 | results = rvm_.test(testData, testLabel); 168 | switch rvm_.type 169 | case 'RVR' 170 | accuracy_(i, 1) = results.performance.RMSE; 171 | case 'RVC' 172 | accuracy_(i, 1) = 1-results.performance.accuracy; 173 | end 174 | 175 | end 176 | accuracy_(accuracy_ == Inf) = []; 177 | accuracy = mean(accuracy_); 178 | case 'HoldOut' 179 | testIndices = (cvIndices == 0); 180 | testData = data_(testIndices, :); 181 | testLabel = label_(testIndices, :); 182 | rvm_.data = data_(~testIndices, :); 183 | rvm_.label_ = label_(~testIndices, :); 184 | rvm_.getModel; 185 | results = rvm_.test(testData, testLabel); 186 | switch rvm_.type 187 | case 'RVR' 188 | accuracy = results.performance.RMSE; 189 | case 'RVC' 190 | accuracy = 1-results.performance.accuracy; 191 | end 192 | end 193 | end 194 | end 195 | end -------------------------------------------------------------------------------- /RvmModel/RvmOptimizationOption.m: -------------------------------------------------------------------------------- 1 | classdef RvmOptimizationOption < handle 2 | %{ 3 | CLASS DESCRIPTION 4 | 5 | Parameter optimization option for RVM model. 6 | 7 | --------------------------------------------------------------- 8 | 9 | %} 10 | 11 | methods (Static) 12 | function checkParameter(rvm) 13 | backup_ = rvm.optimization; 14 | % gaussian 15 | gaussian.parameterName = {'gamma'}; 16 | gaussian.parameterType = {'real'}; 17 | gaussian.lowerBound = 2^-6; 18 | gaussian.upperBound = 2^6; 19 | 20 | % laplacian 21 | laplacian.parameterName = {'gamma'}; 22 | laplacian.parameterType = {'real'}; 23 | laplacian.lowerBound = 2^-6; 24 | laplacian.upperBound = 2^6; 25 | 26 | % polynomial 27 | polynomial.parameterName = {'gamma'; 'offset'; 'degree'}; 28 | polynomial.parameterType = {'real'; 'real'; 'integer'}; 29 | polynomial.lowerBound = [2^-6; 2^-6; 1]; 30 | polynomial.upperBound = [2^6; 2^6; 7]; 31 | 32 | % sigmoid 33 | sigmoid.parameterName = {'gamma'; 'offset'}; 34 | sigmoid.parameterType = {'real'; 'real'}; 35 | sigmoid.lowerBound = [2^-6; 2^-6]; 36 | sigmoid.upperBound = [2^6; 2^6]; 37 | 38 | % others 39 | rvm.optimization.method = 'bayes'; % bayes, ga, pso 40 | rvm.optimization.display = 'on'; 41 | rvm.optimization.iteration = 30; 42 | rvm.optimization.point = 10; 43 | 44 | % parameter initialization 45 | name_ = fieldnames(backup_); 46 | count_ = 0; 47 | % 48 | for i = 1:rvm.numKernel 49 | if ~isfield(backup_, rvm.kernelFuncName{i}) 50 | rvm.optimization.(rvm.kernelFuncName{i}) = eval(rvm.kernelFuncName{i}); 51 | else 52 | tmp_ = rvm.optimization.(rvm.kernelFuncName{i}); 53 | rvm.optimization.(name_{i}) = RvmOptimizationOption.setKernelFuncParam(rvm.kernelFuncName{i}, gaussian, tmp_); 54 | end 55 | count_ = count_+length(rvm.optimization.(rvm.kernelFuncName{i}).parameterName); 56 | end 57 | 58 | for i = 1:size(name_, 1) 59 | if isfield(rvm.optimization, name_{i}) 60 | rvm.optimization.(name_{i}) = backup_.(name_{i}); 61 | end 62 | end 63 | 64 | if strcmp(rvm.kernelType, 'hybrid') 65 | tmp_ = cell(rvm.numKernel, 1); 66 | tmp_(:) = {'real'}; 67 | rvm.optimization.weight.parameterType = tmp_(:); 68 | rvm.optimization.weight.lowerBound = zeros(rvm.numKernel, 1); 69 | rvm.optimization.weight.upperBound = ones(rvm.numKernel, 1); 70 | count_ = count_+rvm.numKernel; 71 | end 72 | 73 | % 74 | rvm.optimization.numParameter = count_; 75 | if isfield(rvm.optimization, 'polynomial') 76 | if ~strcmp(rvm.optimization.method, 'bayes') 77 | errorText = strcat(... 78 | 'The parameter of the polynomial kernel funcion should be optimized by Bayesian optimization.\n',... 79 | 'For examples, the optimization method should be set as follows:\n',... 80 | '--------------------------------------------\n',... 81 | 'opt.method = ''bayes''\n'); 82 | error(sprintf(errorText)) 83 | end 84 | end 85 | end 86 | 87 | function constructParamTable(rvm) 88 | name_ = cell(1, rvm.optimization.numParameter); 89 | count_ = 0; 90 | for i = 1:rvm.numKernel 91 | for j = 1:length(rvm.optimization.(rvm.kernelFuncName{i, 1}).parameterName) 92 | count_ = count_+1; 93 | name_{count_} = [rvm.kernelFuncName{i, 1},'_', rvm.optimization.(rvm.kernelFuncName{i, 1}).parameterName{j}]; 94 | end 95 | end 96 | 97 | if strcmp(rvm.kernelType, 'hybrid') 98 | for i = 1:rvm.numKernel 99 | count_ = count_+1; 100 | name_{count_} = [rvm.kernelFuncName{i, 1}, '_', 'weight']; 101 | end 102 | end 103 | rvm.optimization.bestParam = array2table(zeros(1, rvm.optimization.numParameter)); 104 | rvm.optimization.bestParam.Properties.VariableNames = name_; 105 | end 106 | 107 | function setParameter(rvm) 108 | for i = 1:rvm.numKernel 109 | switch rvm.kernelFuncName{i} 110 | case 'gaussian' 111 | rvm.kernelFunc(i).gamma = rvm.optimization.bestParam.gaussian_gamma; 112 | case 'laplacian' 113 | rvm.kernelFunc(i).gamma = rvm.optimization.bestParam.laplacian_gamma; 114 | case 'polynomial' 115 | rvm.kernelFunc(i).gamma = rvm.optimization.bestParam.polynomial_gamma; 116 | rvm.kernelFunc(i).offset = rvm.optimization.bestParam.polynomial_offset; 117 | rvm.kernelFunc(i).degree = rvm.optimization.bestParam.polynomial_degree; 118 | case 'sigmoid' 119 | rvm.kernelFunc(i).gamma = rvm.optimization.bestParam.sigmoid_gamma; 120 | rvm.kernelFunc(i).offset = rvm.optimization.bestParam.sigmoid_offset; 121 | end 122 | end 123 | 124 | if strcmp(rvm.kernelType, 'hybrid') 125 | for i = 1:rvm.numKernel 126 | rvm.kernelWeight(i) = rvm.optimization.bestParam.([rvm.kernelFuncName{i}, '_weight']); 127 | end 128 | rvm.kernelWeight = rvm.kernelWeight/sum(rvm.kernelWeight); 129 | for i = 1:rvm.numKernel 130 | rvm.optimization.bestParam.([rvm.kernelFuncName{i}, '_weight']) = rvm.kernelWeight(i); 131 | end 132 | end 133 | end 134 | 135 | function parameter = setParameterForBayes(rvm) 136 | parameter = []; 137 | for i = 1:rvm.numKernel 138 | for j = 1: length(rvm.optimization.(rvm.kernelFuncName{i, 1}).parameterName) 139 | name_ = [rvm.kernelFuncName{i, 1},'_', rvm.optimization.(rvm.kernelFuncName{i, 1}).parameterName{j}]; 140 | tmp_ = optimizableVariable(name_,... 141 | [rvm.optimization.(rvm.kernelFuncName{i, 1}).lowerBound(j)... 142 | rvm.optimization.(rvm.kernelFuncName{i, 1}).upperBound(j)],... 143 | 'Type', rvm.optimization.(rvm.kernelFuncName{i, 1}).parameterType{j}); 144 | parameter = [parameter; tmp_]; 145 | end 146 | end 147 | if strcmp(rvm.kernelType, 'hybrid') 148 | for i = 1:rvm.numKernel 149 | name_ = [rvm.kernelFuncName{i, 1}, '_', 'weight']; 150 | tmp_ = optimizableVariable(name_,... 151 | [rvm.optimization.weight.lowerBound(i)... 152 | rvm.optimization.weight.upperBound(i)],... 153 | 'Type', rvm.optimization.weight.parameterType{i}); 154 | parameter = [parameter; tmp_]; 155 | end 156 | end 157 | end 158 | 159 | function [lowerBound_, upperBound_] = setParameterForPsoAndGa(rvm) 160 | lowerBound_ = []; 161 | for i = 1:rvm.numKernel 162 | for j = 1: length(rvm.optimization.(rvm.kernelFuncName{i, 1}).parameterName) 163 | tmp_ = rvm.optimization.(rvm.kernelFuncName{i, 1}).lowerBound(j); 164 | lowerBound_ = [lowerBound_; tmp_]; 165 | end 166 | end 167 | if strcmp(rvm.kernelType, 'hybrid') 168 | for i = 1:rvm.numKernel 169 | tmp_ = rvm.optimization.weight.lowerBound(i); 170 | lowerBound_ = [lowerBound_; tmp_]; 171 | end 172 | end 173 | 174 | upperBound_ = []; 175 | for i = 1:rvm.numKernel 176 | for j = 1: length(rvm.optimization.(rvm.kernelFuncName{i, 1}).parameterName) 177 | tmp_ = rvm.optimization.(rvm.kernelFuncName{i, 1}).upperBound(j); 178 | upperBound_ = [upperBound_; tmp_]; 179 | end 180 | end 181 | if strcmp(rvm.kernelType, 'hybrid') 182 | for i = 1:rvm.numKernel 183 | tmp_ = rvm.optimization.weight.upperBound(i); 184 | upperBound_ = [upperBound_; tmp_]; 185 | end 186 | end 187 | end 188 | 189 | function defaultParam = setKernelFuncParam(paramType, defaultParam, inputParam) 190 | names = fieldnames(inputParam); 191 | lab = isfield(defaultParam, names); 192 | if ~all(lab(:) == 1) 193 | RvmOptimizationOption.showErrorText(paramType); 194 | end 195 | for i = 1:numel(names) 196 | defaultParam.(names{i}) = inputParam.(names{i}); 197 | end 198 | end 199 | 200 | function showErrorText(paramType) 201 | switch paramType 202 | case 'gaussian' 203 | errorText = strcat(... 204 | 'Incorrected optimization parameters of the gaussian kernel function.\n',... 205 | 'For examples, the parameters should be set as follows:\n',... 206 | '--------------------------------------------\n',... 207 | 'opt.gaussian.parameterName = {''gamma''}\n',... 208 | 'opt.gaussian.parameterType = {''real''}\n',... 209 | 'opt.gaussian.lowerBound = 2^-6\n',... 210 | 'opt.gaussian.upperBound = 2^6\n'); 211 | 212 | case 'laplacian' 213 | errorText = strcat(... 214 | 'Incorrected optimization parameters of the laplacian kernel function.\n',... 215 | 'For examples, the parameters should be set as follows:\n',... 216 | '--------------------------------------------\n',... 217 | 'opt.laplacian.parameterName = {''gamma''}\n',... 218 | 'opt.laplacian.parameterType = {''real''}\n',... 219 | 'opt.laplacian.lowerBound = 2^-6\n',... 220 | 'opt.laplacian.upperBound = 2^6\n'); 221 | 222 | case 'polynomial' 223 | errorText = strcat(... 224 | 'Incorrected optimization parameters of the polynomial kernel function.\n',... 225 | 'For examples, the parameters should be set as follows:\n',... 226 | '--------------------------------------------\n',... 227 | 'opt.polynomial.parameterName = {''gamma''; ''offset''; ''degree''}\n',... 228 | 'opt.polynomial.parameterType = {''real''; ''real''; ''integer''}\n',... 229 | 'opt.polynomial.lowerBound = [2^-6; 2^-6; 1]\n',... 230 | 'opt.polynomial.upperBound = [2^6; 2^6; 7]\n'); 231 | 232 | case 'sigmoid' 233 | errorText = strcat(... 234 | 'Incorrected optimization parameters of the sigmoid kernel function.\n',... 235 | 'For examples, the parameters should be set as follows:\n',... 236 | '--------------------------------------------\n',... 237 | 'opt.sigmoid.parameterName = {''gamma''; ''offset''}\n',... 238 | 'opt.sigmoid.parameterType = {''real''; ''real''}\n',... 239 | 'opt.sigmoid.lowerBound = [2^-6; 2^-6]\n',... 240 | 'opt.sigmoid.upperBound = [2^6; 2^6]\n'); 241 | end 242 | error(sprintf(errorText)) 243 | end 244 | end 245 | end -------------------------------------------------------------------------------- /RvmModel/RvmOption.m: -------------------------------------------------------------------------------- 1 | classdef RvmOption < handle 2 | %{ 3 | CLASS DESCRIPTION 4 | 5 | Option of RVM model 6 | 7 | ----------------------------------------------------------------- 8 | %} 9 | 10 | methods(Static) 11 | function label_ = checkLabel(obj, label) 12 | label_ = zeros(length(label), 1); 13 | if length(obj.classNames) ~= 2 14 | error('RVM is only supported for binary classification.') 15 | end 16 | switch class(obj.classNames) 17 | case 'double' 18 | label_(label==obj.classNames(1)) = 0; 19 | label_(label==obj.classNames(2)) = 1; 20 | case 'cell' 21 | label_(strcmp(label, obj.classNames{1})) = 0; 22 | label_(strcmp(label, obj.classNames{2})) = 1; 23 | end 24 | end 25 | 26 | function label_ = restoreLabel(names, label) 27 | numLabel = length(label); 28 | switch class(names) 29 | case 'double' 30 | label_ = zeros(numLabel, 1); 31 | label_(label==0) = names(1); 32 | label_(label==1) = names(2); 33 | case 'cell' 34 | label_ = cell(numLabel, 1); 35 | for i = 1:numLabel 36 | if label(i) == 0 37 | label_{i} = names{1}; 38 | else 39 | label_{i} = names{2}; 40 | end 41 | end 42 | end 43 | end 44 | 45 | function drawRVR(results) 46 | figure 47 | set(gcf, 'position', [300 150 600 400]) 48 | hold on 49 | grid on 50 | % boundary 51 | index = (1:size(results.label, 1))'; 52 | area_color = [229, 235, 245]/255; 53 | area = [results.predictedLabel(:,1)+2*sqrt(results.predictedVariance(:, 1));... 54 | flip(results.predictedLabel(:, 1)-2*sqrt(results.predictedVariance(:, 1)), 1)]; 55 | fill([index; flip(index, 1)], area, area_color, 'EdgeColor', area_color) 56 | plot(index, results.label,... 57 | '-', 'LineWidth', 2, 'color', [254, 67, 101]/255) 58 | plot(index, results.predictedLabel,... 59 | '-','LineWidth', 2,'color', [0, 114, 189]/255) 60 | legend('3�� boundary', 'Real value', 'Predicted value') 61 | xlabel('Observations'); 62 | ylabel('Predictions'); 63 | end 64 | 65 | function drawRVC(results) 66 | titleStr = ['Classification based on RVM (Accuracy: ',... 67 | sprintf('%.2f', 100*results.performance.accuracy), '%)']; 68 | figure 69 | set(gcf, 'position', [300 150 600 400]) 70 | hold on 71 | grid on 72 | index = (1:length(results.label))'; 73 | plot(index, results.label_,... 74 | 'LineWidth', 2, 'Color', [0, 114, 189]/255,... 75 | 'LineStyle', '-') 76 | plot(index, results.predictedLabel_,... 77 | 'LineWidth', 2, 'Color', [162, 20, 47]/255,... 78 | 'LineStyle', ':') 79 | legend('Real class', 'Predicted class') 80 | xlabel('Observations'); 81 | ylabel('Predictions'); 82 | ylim([-0.5, 1.5]) 83 | yticks([0 1]) 84 | yticklabels(results.classNames) 85 | title(titleStr) 86 | 87 | try 88 | figure 89 | set(gcf, 'position', [300 150 600 400]) 90 | confusionchart(results.performance.confusionMatrix, results.performance.classOrder,... 91 | 'Title', titleStr,... 92 | 'RowSummary','row-normalized', ... 93 | 'ColumnSummary','column-normalized'); 94 | catch 95 | fprintf('Confusion matrix chart is not effective with the versions lower than R2018b.') 96 | close 97 | end 98 | 99 | end 100 | 101 | function setParameter(obj, parameter) 102 | version_ = version('-release'); 103 | year_ = str2double(version_(1:4)); 104 | if year_ < 2016 || (year_ == 2016 && version_(5) == 'a') 105 | error('RVM V2.1 is not compatible with the versions lower than R2016b.') 106 | end 107 | % 108 | obj.crossValidation.switch = 'off'; 109 | obj.optimization.switch = 'off'; 110 | name_ = fieldnames(parameter); 111 | for i = 1:size(name_, 1) 112 | switch name_{i} 113 | case {'HoldOut', 'KFold'} 114 | obj.crossValidation.switch = 'on'; 115 | obj.crossValidation.method = name_{i, 1}; 116 | obj.crossValidation.param = parameter.(name_{i, 1}); 117 | case {'optimization'} 118 | obj.(name_{i}) = parameter.(name_{i, 1}); 119 | obj.(name_{i}).switch = 'on'; 120 | otherwise 121 | obj.(name_{i, 1}) = parameter.(name_{i, 1}); 122 | end 123 | end 124 | obj.numKernel = numel(obj.kernelFunc); 125 | for i = 1:obj.numKernel 126 | obj.kernelFuncName{i, 1} = obj.kernelFunc(i).type; 127 | end 128 | if obj.numKernel == 1 129 | obj.kernelType = 'single'; 130 | obj.kernelFuncName{1} = obj.kernelFunc.type; 131 | else 132 | obj.kernelType = 'hybrid'; 133 | for i = 1:obj.numKernel 134 | obj.kernelFuncName{i, 1} = obj.kernelFunc(i).type; 135 | end 136 | end 137 | if isempty(obj.kernelWeight) 138 | obj.kernelWeight = 1/obj.numKernel*ones(obj.numKernel, 1); 139 | end 140 | end 141 | 142 | function displayTrain(obj) 143 | fprintf('\n') 144 | switch obj.type 145 | case 'RVR' 146 | fprintf('*** RVM model (regression) train finished ***\n') 147 | case 'RVC' 148 | fprintf('*** RVM model (classification) train finished ***\n') 149 | end 150 | fprintf('running time = %.4f seconds\n', obj.runningTime) 151 | fprintf('iterations = %d \n', obj.numIterations) 152 | fprintf('number of samples = %d \n', obj.numSamples) 153 | fprintf('number of RVs = %d \n', obj.numRelevanceVectors) 154 | fprintf('ratio of RVs = %.4f%% \n', 100*obj.numRelevanceVectors/obj.numSamples) 155 | switch obj.type 156 | case 'RVR' 157 | fprintf('RMSE = %.4f\n', obj.performance.RMSE) 158 | fprintf('R2 = %.4f\n', obj.performance.R2) 159 | fprintf('MAE = %.4f\n', obj.performance.MAE) 160 | case 'RVC' 161 | fprintf('accuracy = %.4f%%\n', 100*obj.performance.accuracy) 162 | end 163 | if strcmp(obj.optimization.switch, 'on') 164 | fprintf('Optimized parameter') 165 | display(obj.optimization.bestParam) 166 | fprintf('\n') 167 | end 168 | if obj.numRelevanceVectors/obj.numSamples > 0.5 169 | warning('The trained RVM model may be overfitting.') 170 | end 171 | fprintf('\n') 172 | end 173 | 174 | function displayTest(obj) 175 | fprintf('\n') 176 | switch obj.type 177 | case 'RVR' 178 | fprintf('*** RVM model (regression) test finished ***\n') 179 | case 'RVC' 180 | fprintf('*** RVM model (classification) test finished ***\n') 181 | end 182 | fprintf('running time = %.4f seconds\n', obj.runningTime) 183 | fprintf('number of samples = %d \n', obj.numSamples) 184 | switch obj.type 185 | case 'RVR' 186 | fprintf('RMSE = %.4f\n', obj.performance.RMSE) 187 | fprintf('R2 = %.4f\n', obj.performance.R2) 188 | fprintf('MAE = %.4f\n', obj.performance.MAE) 189 | case 'RVC' 190 | fprintf('accuracy = %.4f%%\n', 100*obj.performance.accuracy) 191 | end 192 | fprintf('\n') 193 | end 194 | end 195 | end -------------------------------------------------------------------------------- /SB2_Release_200/Readme.txt: -------------------------------------------------------------------------------- 1 | SPARSEBAYES Matlab Toolbox 2 | 3 | Version 2.00 4 | 5 | -- INTRODUCTION -- 6 | 7 | "SparseBayes" is a package of Matlab functions designed to implement 8 | an efficient learning algorithm for "Sparse Bayesian" models. 9 | 10 | The "Version 2" package is an expanded implementation of the algorithm 11 | detailed in: 12 | 13 | Tipping, M. E. and A. C. Faul (2003). "Fast marginal likelihood 14 | maximisation for sparse Bayesian models." In C. M. Bishop and 15 | B. J. Frey (Eds.), Proceedings of the Ninth International Workshop 16 | on Artificial Intelligence and Statistics, Key West, FL, Jan 3-6. 17 | 18 | This paper, the accompanying code, and further information regarding 19 | Sparse Bayesian models and the related "relevance vector machine" may 20 | be obtained from: 21 | 22 | http://www.relevancevector.com 23 | 24 | 25 | -- CODE STATUS -- 26 | 27 | March 2009: The code is currently at Version 2.0, and has seen limited 28 | testing under Matlab Version 7.4. 29 | 30 | 31 | -- GETTING STARTED -- 32 | 33 | The SparseBayes distribution comes with a basic user manual: see 34 | SB2_Manual.pdf. 35 | 36 | In summary, there are a number of files supplied in the SB2 37 | distribution, but the user should only need to call a subset of them. 38 | 39 | To set up options and parameters: 40 | 41 | SB2_UserOptions.m 42 | SB2_ParameterSettings.m 43 | 44 | The main algorithm 45 | 46 | SparseBayes.m 47 | 48 | Type "help SparseBayes" etc. at the Matlab prompt for further details 49 | on how to use these functions. 50 | 51 | There is a also a simple demonstration program, showing how 52 | SparseBayes may be used, in: 53 | 54 | SparseBayesDemo.m 55 | 56 | 57 | -- LICENCE -- 58 | 59 | Note that the "SparseBayes" software is supplied subject to version 2 60 | of the "GNU General Public License" (detailed in the file "licence.txt"). 61 | 62 | SPARSEBAYES is free software; you can redistribute it and/or modify it 63 | under the terms of the GNU General Public License as published by the 64 | Free Software Foundation; either version 2 of the License, or (at your 65 | option) any later version. 66 | 67 | SPARSEBAYES is distributed in the hope that it will be useful, but 68 | WITHOUT ANY WARRANTY; without even the implied warranty of 69 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 70 | General Public License for more details. 71 | 72 | You should have received a copy of the GNU General Public License 73 | along with SPARSEBAYES in the accompanying file "licence.txt"; if not, 74 | write to the Free Software Foundation, Inc., 51 Franklin St, Fifth 75 | Floor, Boston, MA 02110-1301 USA 76 | 77 | 78 | -- ACKNOWLEDGEMENTS -- 79 | 80 | The author would like to thank Mark Hatton, Anita Faul, Ian Nabney, 81 | Arnulf Graf and Gavin Cawley for their assistance in producing this 82 | code. 83 | 84 | 85 | -- 86 | 87 | Mike Tipping 88 | www.relevancevector.com 89 | m a i l [at] m i k e t i p p i n g . c o m 90 | 91 | This README file (Readme.txt) was created on 13-Mar-2009 at 8:58 AM. 92 | -------------------------------------------------------------------------------- /SB2_Release_200/SB2_ControlSettings.m: -------------------------------------------------------------------------------- 1 | % SB2_CONTROLSETTINGS Set parameters to control the SPARSEBAYES algorithm 2 | % 3 | % CONTROLS = SB2_CONTROLSETTINGS 4 | % 5 | % OUTPUT ARGUMENTS: 6 | % 7 | % CONTROLS A structure whose fields control various aspects of the 8 | % running of the SPARSEBAYES algorithm. 9 | % 10 | % .ZeroFactor Small number equivalent to zero for Q^2-S 11 | % .MinDeltaLogAlpha Termination criterion for changes in log-alpha 12 | % .MinDeltaLogBeta Termination criterion for changes in log-beta 13 | % 14 | % .PriorityAddition Prefer "addition" operations 15 | % .PriorityDeletion Prefer "deletion" operations 16 | % 17 | % .BetaUpdateStart How many "fast start" beta updates 18 | % .BetaUpdateFrequency 19 | % How regularly to update beta after the above 20 | % .BetaMaxFactor Minimum value control for noise estimate 21 | % 22 | % .PosteriorModeFrequency 23 | % How regularly to re-find the posterior mode 24 | % 25 | % .BasisAlignmentTest Test for redundant basis vectors? 26 | % .AlignmentMax Basis redundancy criterion 27 | % 28 | % NOTES: 29 | % 30 | % The various definitions in the file are effectively "fixed" and not 31 | % modified elsewhere. 32 | % 33 | % The interested user may wish to experiment with the operation of the 34 | % SPARSEBAYES algorithm by modifying the values the file directly. See the 35 | % inline comments for hints on the various control settings. 36 | % 37 | 38 | % 39 | % Copyright 2009, Vector Anomaly Ltd 40 | % 41 | % This file is part of the SPARSEBAYES library for Matlab (V2.0). 42 | % 43 | % SPARSEBAYES is free software; you can redistribute it and/or modify it 44 | % under the terms of the GNU General Public License as published by the Free 45 | % Software Foundation; either version 2 of the License, or (at your option) 46 | % any later version. 47 | % 48 | % SPARSEBAYES is distributed in the hope that it will be useful, but WITHOUT 49 | % ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or 50 | % FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for 51 | % more details. 52 | % 53 | % You should have received a copy of the GNU General Public License along 54 | % with SPARSEBAYES in the accompanying file "licence.txt"; if not, write to 55 | % the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, 56 | % MA 02110-1301 USA 57 | % 58 | % Contact the author: m a i l [at] m i k e t i p p i n g . c o m 59 | % 60 | function CONTROLS = SB2_ControlSettings 61 | 62 | %% Define parameters which influence the underlying operation of the 63 | %% SparseBayes inference algorithm 64 | 65 | % TOLERANCES 66 | % 67 | % Any Q^2-S "relevance factor" less than this is considered to be zero 68 | % 69 | CONTROLS.ZeroFactor = 1e-12; 70 | % 71 | % If the change in log-alpha for the best re-estimation is less than this, 72 | % we consider termination 73 | % 74 | CONTROLS.MinDeltaLogAlpha = 1e-3; 75 | % 76 | % In the Gaussian case, we also require a beta update to change the value 77 | % of log-beta (inverse noise variance) less than this to terminate 78 | % 79 | CONTROLS.MinDeltaLogBeta = 1e-6; 80 | 81 | % ADD/DELETE 82 | % 83 | % - preferring addition where possible will probably make the algorithm a 84 | % little slower and perhaps less "greedy" 85 | % 86 | % - preferring deletion may make the model a little more sparse and the 87 | % algorithm may run slightly quicker 88 | % 89 | % Note: both these can be set to 'true' at the same time, in which case 90 | % both take equal priority over re-estimation. 91 | % 92 | CONTROLS.PriorityAddition = false; 93 | CONTROLS.PriorityDeletion = true; 94 | 95 | % (GAUSSIAN) NOISE 96 | % 97 | % When to update the noise estimate 98 | % 99 | % The number of iterations from the start for which we update it every 100 | % iteration (to get in the right ball-park to begin with) 101 | % 102 | CONTROLS.BetaUpdateStart = 10; 103 | % 104 | % After the above, we only regularly update it after 105 | % a given number of iterations 106 | % 107 | CONTROLS.BetaUpdateFrequency = 5; 108 | % 109 | % Prevent zero-noise estimate (perfect fit) problem 110 | % - effectively says the noise variance estimate is clamped to be no 111 | % lower than variance-of-targets / BetaMaxFactor. 112 | % 113 | CONTROLS.BetaMaxFactor = 1e6; 114 | 115 | % POSTERIORMODE 116 | % 117 | % How many alpha updates to do in between each full posterior mode 118 | % computation in the non-Gaussian case 119 | % 120 | % In principle, this should be set to one (to update the posterior every 121 | % iteration) but it may be more efficient to do several alpha updates before 122 | % re-finding the posterior mode. 123 | % 124 | CONTROLS.PosteriorModeFrequency = 1; 125 | 126 | % REDUNDANT BASIS 127 | % 128 | % Check for basis vector alignment/correlation redundancy 129 | % 130 | CONTROLS.BasisAlignmentTest = true; 131 | % 132 | ALIGNMENT_ZERO = 1e-3; 133 | % 134 | % If BasisAlignmentTest is true, any basis vector with inner product more 135 | % than MAX_ALIGNMENT with any existing model vector will not be added 136 | % 137 | CONTROLS.AlignmentMax = 1 - ALIGNMENT_ZERO; 138 | 139 | -------------------------------------------------------------------------------- /SB2_Release_200/SB2_Diagnostic.m: -------------------------------------------------------------------------------- 1 | % SB2_DIAGNOSTIC Helper function to output diagnostics in SPARSEBAYES 2 | % 3 | % USAGE (1): 4 | % 5 | % OPTIONS = SB2_DIAGNOSTIC(OPTIONS, ACTION) 6 | % 7 | % OUTPUT ARGUMENTS: 8 | % 9 | % OPTIONS Returned OPTIONS settings, possibly updated 10 | % (e.g. with file handle after 'open' action) 11 | % 12 | % INPUT ARGUMENTS: 13 | % 14 | % OPTIONS Options settings (SB2_USEROPTIONS) 15 | % 16 | % ACTION One of: 17 | % 'OPEN' or 'START' Begin diagnostics 18 | % 'CLOSE', 'END' or 'FINISH' End diagnostics 19 | % 20 | % 21 | % USAGE (2): 22 | % 23 | % SB2_DIAGNOSTIC(OPTIONS, LEVEL, MESSAGE, VARARGIN ...) 24 | % 25 | % OUTPUT ARGUMENTS: None 26 | % 27 | % INPUT ARGUMENTS: 28 | % 29 | % OPTIONS Options settings (SB2_USEROPTIONS) 30 | % 31 | % LEVEL Importance level of message (0 to 4) 32 | % NB: LOWER is MORE IMPORTANT 33 | % 34 | % MESSAGE Message string 35 | % 36 | % VARARGIN Optional arguments to fill in string placeholders 37 | % 38 | % EXAMPLE: 39 | % 40 | % OPTIONS = SB2_Diagnostic(OPTIONS, 'start'); 41 | % SB2_Diagnostic(OPTIONS,2,'Total of %d likelihood updates\n', 25); 42 | % OPTIONS = SB2_Diagnostic(OPTIONS, 'end'); 43 | % 44 | % NOTES: 45 | % 46 | % This function offers a convenient way to output diagnostic information, 47 | % either to screen or to a file. 48 | % 49 | % It will "filter" the incoming messages and only display/write them if 50 | % their LEVEL is less than or equal to the DIAGNOSTICLEVEL as set by 51 | % SB2_USEROPTIONS. 52 | % 53 | % e.g. after 54 | % 55 | % OPTIONS = SB2_UserOptions('diagnosticLevel',2) 56 | % 57 | % This message won't display: 58 | % 59 | % SB2_Diagnostic(OPTIONS, 3, 'Pointless message') 60 | % 61 | % But this will: 62 | % 63 | % SB2_Diagnostic(OPTIONS, 1, 'Really important message') 64 | % 65 | % Messages of level 0 are therefore most "important", and those of level 66 | % 4 least important. Level-0 messages are always displayed, since the 67 | % DIAGNOSTICLEVEL will always be at least zero. 68 | % 69 | 70 | % 71 | % Copyright 2009, Vector Anomaly Ltd 72 | % 73 | % This file is part of the SPARSEBAYES library for Matlab (V2.0). 74 | % 75 | % SPARSEBAYES is free software; you can redistribute it and/or modify it 76 | % under the terms of the GNU General Public License as published by the Free 77 | % Software Foundation; either version 2 of the License, or (at your option) 78 | % any later version. 79 | % 80 | % SPARSEBAYES is distributed in the hope that it will be useful, but WITHOUT 81 | % ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or 82 | % FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for 83 | % more details. 84 | % 85 | % You should have received a copy of the GNU General Public License along 86 | % with SPARSEBAYES in the accompanying file "licence.txt"; if not, write to 87 | % the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, 88 | % MA 02110-1301 USA 89 | % 90 | % Contact the author: m a i l [at] m i k e t i p p i n g . c o m 91 | % 92 | function OPTIONS = SB2_Diagnostic(OPTIONS, level, message_, varargin) 93 | 94 | % 95 | % Check if "level" is actually a string (this is a control call) 96 | % 97 | if ischar(level) 98 | switch upper(level) 99 | case {'OPEN','START'}, 100 | if OPTIONS.diagnosticFID~=1 101 | % 102 | % If diagnosticFID is not 1, we must have previously set a 103 | % diagnostics file via SB2_USEROPTIONS 104 | % 105 | OPTIONS.diagnosticFID = fopen(OPTIONS.diagnosticFile_,'w'); 106 | % Check for file open error 107 | if OPTIONS.diagnosticFID==-1 108 | error('Could not open diagnostic file ''%s''', value); 109 | end 110 | end 111 | % 112 | case {'CLOSE','END','FINISH'}, 113 | if OPTIONS.diagnosticFID~=1 114 | % diagnosticFID refers to a file opened above 115 | fclose(OPTIONS.diagnosticFID); 116 | end 117 | end 118 | else 119 | % 120 | % Its a message call. 121 | % 122 | % Only output diagnostic messages if the "diagnostic level", 123 | % set via SB2_USEROPTIONS, is equal to or greater than the level of 124 | % the message to be displayed. 125 | % 126 | if level<=OPTIONS.diagnosticLevel 127 | fprintf(OPTIONS.diagnosticFID, repmat(' ', [1 level])); 128 | fprintf(OPTIONS.diagnosticFID, message_, varargin{:}); 129 | end 130 | end 131 | 132 | -------------------------------------------------------------------------------- /SB2_Release_200/SB2_FormatTime.m: -------------------------------------------------------------------------------- 1 | % SB2_FORMATTIME Pretty output of diagnostic SPARSEBAYES time information 2 | % 3 | % STRING = SB2_FORMATTIME(ELAPSEDTIMEINSECONDS) 4 | % 5 | 6 | % 7 | % Copyright 2009, Vector Anomaly Ltd 8 | % 9 | % This file is part of the SPARSEBAYES library for Matlab (V2.0). 10 | % 11 | % SPARSEBAYES is free software; you can redistribute it and/or modify it 12 | % under the terms of the GNU General Public License as published by the Free 13 | % Software Foundation; either version 2 of the License, or (at your option) 14 | % any later version. 15 | % 16 | % SPARSEBAYES is distributed in the hope that it will be useful, but WITHOUT 17 | % ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or 18 | % FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for 19 | % more details. 20 | % 21 | % You should have received a copy of the GNU General Public License along 22 | % with SPARSEBAYES in the accompanying file "licence.txt"; if not, write to 23 | % the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, 24 | % MA 02110-1301 USA 25 | % 26 | % Contact the author: m a i l [at] m i k e t i p p i n g . c o m 27 | % 28 | function timeString_ = SB2_FormatTime(elapsedTime) 29 | 30 | if elapsedTime>=3600 31 | % More than an hour... 32 | % 33 | h = floor(elapsedTime/3600); 34 | m = floor(rem(elapsedTime,3600)/60); 35 | s = floor(rem(elapsedTime,60)); 36 | % 37 | timeString_ = sprintf('%dh %dm %ds', h,m,s); 38 | elseif elapsedTime>=60 39 | % More than one minute (but less than an hour) 40 | m = floor(elapsedTime/60); 41 | s = floor(rem(elapsedTime,60)); 42 | % 43 | timeString_ = sprintf('%dm %ds', m,s); 44 | else 45 | % Under a minute 46 | s = elapsedTime; 47 | % 48 | timeString_ = sprintf('%2.2f secs', s); 49 | end -------------------------------------------------------------------------------- /SB2_Release_200/SB2_FullStatistics.m: -------------------------------------------------------------------------------- 1 | % SB2_FULLSTATISTICS Compute all relevant statistics in full for SPARSEBAYES 2 | % 3 | % [SIGMA,MU,S_IN,Q_IN,S_OUT,Q_OUT,FACTOR,LOGML,GAMMA,... 4 | % BETABASIS_PHI,BETA] = ... 5 | % SB2_FULLSTATISTICS(LIKELIHOOD,BASIS,PHI,TARGETS,USED,ALPHA,BETA,... 6 | % MU,BASIS_PHI,BASIS_TARGETS,OPTIONS) 7 | % 8 | % OUTPUT ARGUMENTS: 9 | % 10 | % SIGMA Posterior covariance matrix for relevant bases 11 | % MU Posterior mean 12 | % S_IN S-factors for in-model (relevant) basis vectors 13 | % Q_IN Q-factors for in-model (relevant) basis vectors 14 | % S_OUT S-factors for all basis vectors 15 | % Q_OUT Q-factors for all basis vectors 16 | % FACTOR Q^2-S relevance factors 17 | % LOGML Log-marginal-likelihood 18 | % GAMMA "Well-determinedness" factors 19 | % BETABASIS_PHI Cached value of BASIS'*B*PHI matrix 20 | % BETA Inverse noise variance (vector of beta 21 | % approximations in non-Gaussian case) 22 | % 23 | % INPUT ARGUMENTS: 24 | % 25 | % LIKELIHOOD LIKELIHOOD structure 26 | % BASIS Full NxM basis matrix 27 | % PHI Current relevant basis matrix 28 | % TARGETS N-vector with target output values 29 | % USED Relevant basis vector indices 30 | % ALPHA Hyperparameters 31 | % BETA Inverse noise variance (ignored in non-Gauss case) 32 | % MU Current posterior mean (for non-Gauss) 33 | % BASIS_PHI Cached value of BASIS'*PHI (for Gauss only) 34 | % BASIS_TARGETS Cached value of BASIS'*TARGETS (for Gauss only) 35 | % OPTIONS Standard OPTIONS structure (see SB2_USEROPTIONS) 36 | % 37 | % NOTES: 38 | % 39 | % This function computes the posterior, and other, statistics for the 40 | % SPARSEBAYES algorithm in "long-hand" fashion. i.e. it does not use the 41 | % more effecient "iterative" updates. 42 | % 43 | % It is required on every iteration (unless approximating) in the 44 | % non-Gaussian case, and on iterations in the Gaussian case where the noise 45 | % estimate (BETA) is updated. 46 | % 47 | % This function is intended for internal use by SPARSEBAYES only. 48 | % 49 | 50 | 51 | % 52 | % Copyright 2009, Vector Anomaly Ltd 53 | % 54 | % This file is part of the SPARSEBAYES library for Matlab (V2.0). 55 | % 56 | % SPARSEBAYES is free software; you can redistribute it and/or modify it 57 | % under the terms of the GNU General Public License as published by the Free 58 | % Software Foundation; either version 2 of the License, or (at your option) 59 | % any later version. 60 | % 61 | % SPARSEBAYES is distributed in the hope that it will be useful, but WITHOUT 62 | % ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or 63 | % FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for 64 | % more details. 65 | % 66 | % You should have received a copy of the GNU General Public License along 67 | % with SPARSEBAYES in the accompanying file "licence.txt"; if not, write to 68 | % the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, 69 | % MA 02110-1301 USA 70 | % 71 | % Contact the author: m a i l [at] m i k e t i p p i n g . c o m 72 | % 73 | function [SIGMA,Mu,S_in,Q_in,S_out,Q_out,Factor,logML,Gamma,... 74 | betaBASIS_PHI,beta] = ... 75 | SB2_FullStatistics(LIKELIHOOD,BASIS,PHI,Targets,Used,Alpha,beta,... 76 | Mu,BASIS_PHI,BASIS_Targets,OPTIONS) 77 | 78 | % Mu is only required for non-Gauss 79 | % BASIS_PHI and BASIS_Targets are only required for Gauss 80 | 81 | % beta (a vector for non-Gauss) is returned only for non-Gauss 82 | 83 | MAX_POSTMODE_ITS = 25; % More than enough 84 | 85 | [N M_full] = size(BASIS); 86 | M = size(PHI,2); 87 | 88 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 89 | 90 | % COMPUTE POSTERIOR 91 | 92 | % 93 | % Compute full relevant statistics 94 | % 95 | if LIKELIHOOD.InUse==LIKELIHOOD.Gaussian 96 | % 97 | % Posterior is analytic: use linear algebra here 98 | % 99 | % Compute posterior covariance SIGMA (inverse Hessian) 100 | % via Cholesky decomposition 101 | % 102 | U = chol(PHI'*PHI*beta + diag(Alpha)); 103 | Ui = inv(U); 104 | SIGMA = Ui * Ui'; 105 | % Posterior mean Mu 106 | Mu = (SIGMA * (PHI'*Targets)) * beta; 107 | % Data error and likelihood 108 | y = PHI * Mu; 109 | e = (Targets - y); 110 | ED = e'*e; 111 | % 112 | dataLikely = (N*log(beta) - beta*ED)/2; 113 | % 114 | else 115 | % 116 | % Posterior must be approximated: find the posterior mode as the basis 117 | % for the Laplace approximation 118 | % 119 | [Mu U beta dataLikely] = ... 120 | SB2_PosteriorMode(LIKELIHOOD,PHI,Targets,Alpha,Mu, ... 121 | MAX_POSTMODE_ITS,OPTIONS); 122 | % Compute covariance approximation 123 | % 124 | Ui = inv(U); 125 | SIGMA = Ui * Ui'; 126 | % Compute posterior-mean-based outputs and errors 127 | % 128 | switch LIKELIHOOD.InUse 129 | case LIKELIHOOD.Bernoulli, 130 | y = SB2_Sigmoid(PHI * Mu); 131 | case LIKELIHOOD.Poisson 132 | y = exp(PHI*Mu); 133 | end 134 | e = (Targets-y); 135 | end 136 | 137 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 138 | 139 | % COMPUTE LOG MARGINAL LIKELIHOOD 140 | 141 | logdetHOver2 = sum(log(diag(U))); 142 | logML = dataLikely - (Mu.^2)'*Alpha/2 + ... 143 | sum(log(Alpha))/2 - logdetHOver2; 144 | % Well-determinedness factors 145 | DiagC = sum(Ui.^2,2); 146 | Gamma = 1 - Alpha.*DiagC; 147 | 148 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 149 | 150 | % COMPUTE THE Q & S VALUES 151 | 152 | % 153 | % Q: "quality" factor - related to how well the basis function contributes 154 | % to reducing the error 155 | % 156 | % S: "sparsity factor" - related to how orthogonal a given basis function 157 | % is to the currently used set of basis functions 158 | % 159 | if LIKELIHOOD.InUse==LIKELIHOOD.Gaussian 160 | % 161 | % Gaussian case simple: beta a scalar 162 | % 163 | betaBASIS_PHI = beta*BASIS_PHI; 164 | % 165 | % The S_in calculation exploits the fact that BASIS is normalised, 166 | % i.e. sum(BASIS.^2,1)==ones(1,M_full) 167 | % 168 | S_in = beta - sum((betaBASIS_PHI*Ui).^2,2); 169 | Q_in = beta*(BASIS_Targets - BASIS_PHI*Mu); 170 | else 171 | % 172 | % Non-Gaussian case: beta an N-vector 173 | % 174 | betaBASIS_PHI = BASIS'*(PHI.* (beta*ones(1,M))); 175 | S_in = (beta'*(BASIS.^2))' - sum((betaBASIS_PHI*Ui).^2,2); 176 | Q_in = BASIS'*e; 177 | end 178 | % 179 | S_out = S_in; 180 | Q_out = Q_in; 181 | % 182 | % S,Q with that basis excluded: equations (23) 183 | % 184 | S_out(Used) = (Alpha .* S_in(Used)) ./ (Alpha - S_in(Used)); 185 | Q_out(Used) = (Alpha .* Q_in(Used)) ./ (Alpha - S_in(Used)); 186 | % 187 | % Pre-compute the "relevance factor" for ongoing convenience 188 | % 189 | Factor = (Q_out.*Q_out - S_out); 190 | 191 | -------------------------------------------------------------------------------- /SB2_Release_200/SB2_Initialisation.m: -------------------------------------------------------------------------------- 1 | % SB2_INITIALISATION Initialise everything for the SPARSEBAYES algorithm 2 | % 3 | % [LIKELIHOOD, BASIS, BASISSCALES, ALPHA, BETA, MU, PHI, USED] = ... 4 | % SB2_INITIALISATION(TYPE, BASIS, TARGETS, SETTINGS, OPTIONS) 5 | % 6 | % OUTPUT ARGUMENTS: 7 | % 8 | % LIKELIHOOD Likelihood structure (from SB2_LIKELIHOODS) 9 | % BASIS Pre-processed full basis matrix 10 | % BASISSCALES Scaling factors from full basis pre-processing 11 | % ALPHA Initial hyperparameter alpha values 12 | % BETA Initial noise level (Gaussian) 13 | % MU Initial weight values 14 | % PHI Initial "relevant" basis 15 | % USED Indices of initial basis vectors (columns of BASIS) 16 | % 17 | % INPUT ARGUMENTS: 18 | % 19 | % TYPE Likelihood: one of 'Gaussian', 'Bernoulli', 'Poisson' 20 | % 21 | % BASIS NxM matrix of all possible basis vectors 22 | % (one column per basis function) 23 | % 24 | % TARGETS N-vector with target output values 25 | % 26 | % SETTINGS Initialisation structure for main parameter values via 27 | % SB2_PARAMETERSETTINGS 28 | % 29 | % OPTIONS User options structure from SB2_USEROPTIONS 30 | % 31 | % NOTES: 32 | % 33 | % This function initialises all necessary model parameters appropriately 34 | % before entering the main loop of the SPARSEBAYES inference algorithm. 35 | % 36 | % This function is intended for internal use by SPARSEBAYES only. 37 | % 38 | 39 | % 40 | % Copyright 2009, Vector Anomaly Ltd 41 | % 42 | % This file is part of the SPARSEBAYES library for Matlab (V2.0). 43 | % 44 | % SPARSEBAYES is free software; you can redistribute it and/or modify it 45 | % under the terms of the GNU General Public License as published by the Free 46 | % Software Foundation; either version 2 of the License, or (at your option) 47 | % any later version. 48 | % 49 | % SPARSEBAYES is distributed in the hope that it will be useful, but WITHOUT 50 | % ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or 51 | % FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for 52 | % more details. 53 | % 54 | % You should have received a copy of the GNU General Public License along 55 | % with SPARSEBAYES in the accompanying file "licence.txt"; if not, write to 56 | % the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, 57 | % MA 02110-1301 USA 58 | % 59 | % Contact the author: m a i l [at] m i k e t i p p i n g . c o m 60 | % 61 | function [LIKELIHOOD, BASIS, BasisScales, Alpha, beta, Mu, PHI, Used] = ... 62 | SB2_Initialisation(likelihood_, BASIS, Targets, SETTINGS, OPTIONS) 63 | 64 | % A "reasonable" initial value for the noise in the Gaussian case 65 | GAUSSIAN_SNR_INIT = 0.1; 66 | 67 | % "Reasonable" initial alpha bounds 68 | INIT_ALPHA_MAX = 1e3; 69 | INIT_ALPHA_MIN = 1e-3; 70 | 71 | % 72 | % BASIS PREPROCESSING: 73 | % 74 | % Scale basis vectors to unit norm. This eases some calculations and 75 | % will improve numerical robustness later. 76 | % 77 | [BASIS, BasisScales] = SB2_PreProcessBasis(BASIS); 78 | 79 | %% Noise model considerations 80 | 81 | % Validate the likelihood model specification 82 | % 83 | LIKELIHOOD = SB2_Likelihoods(likelihood_); 84 | 85 | % Default beta value for non-Gaussian likelihood 86 | beta = []; 87 | % 88 | % In the Gaussian case, initialise "sensibly" to a proportion of the 89 | % signal level (e.g. 10 percent) 90 | % 91 | if LIKELIHOOD.InUse == LIKELIHOOD.Gaussian 92 | % Noise initialisation 93 | if ~isempty(SETTINGS.beta) 94 | % The user set "beta" 95 | beta = SETTINGS.beta; 96 | elseif ~isempty(SETTINGS.noiseStdDev) && SETTINGS.noiseStdDev>0 97 | % The user set the "noise std dev" 98 | beta = 1/SETTINGS.noiseStdDev^2; 99 | else 100 | % catch the pathological case where all outputs are zero 101 | % (although we're probably doomed anyway if that's true) 102 | stdt = max([1e-6 std(Targets)]); 103 | % Initialise automatically approximately according to "SNR" 104 | beta = 1/(stdt*GAUSSIAN_SNR_INIT)^2; 105 | end 106 | end 107 | 108 | 109 | % Initialise basis (PHI), mu and alpha 110 | % 111 | % Either as specified by the SETTINGS structure, or heuristically 112 | % 113 | 114 | % First, compute 'linearised' output for use in heuristic initialisation 115 | % 116 | TargetsPseudoLinear = Targets; % standard linear case 117 | switch LIKELIHOOD.InUse 118 | case LIKELIHOOD.Bernoulli, 119 | TargetsPseudoLinear = (2*Targets-1); 120 | case LIKELIHOOD.Poisson, 121 | TargetsPseudoLinear = log(Targets+1e-3); 122 | end 123 | 124 | 125 | % 1) the starting basis, PHI 126 | 127 | % Take account of "free basis": it needs to be included from the outset 128 | % 129 | Extra = setdiff(OPTIONS.freeBasis, SETTINGS.Relevant); 130 | % 131 | % [TODO:] More elegant to put "Extra" first in Used list? 132 | % 133 | Used = [SETTINGS.Relevant Extra]; 134 | % 135 | % At this point Used will contain both pre-specified relevant basis 136 | % functions (SB2_PARAMETERSETTINGS) and any "free" ones (SB2_USEROPTIONS). 137 | % If neither are active, Used will be empty. 138 | % 139 | if isempty(Used) 140 | % Set initial basis to be the largest projection with the targets 141 | proj = (BASIS'*TargetsPseudoLinear); 142 | [foo Used] = max(abs(proj)); 143 | SB2_Diagnostic(OPTIONS, 2, ... 144 | ['Initialising with maximally aligned ' ... 145 | 'basis vector (%d)\n'], Used); 146 | else 147 | SB2_Diagnostic(OPTIONS, 2, ... 148 | 'Initialising with supplied basis of size M = %d\n', ... 149 | length(Used)); 150 | end 151 | PHI = BASIS(:,Used); 152 | M = length(Used); 153 | 154 | % 2) the most probable weights 155 | 156 | if isempty(SETTINGS.Mu) 157 | switch LIKELIHOOD.InUse 158 | % 159 | case LIKELIHOOD.Gaussian, 160 | % Mu will be analytically calculated later in the Gaussian case 161 | % 162 | Mu = []; 163 | case LIKELIHOOD.Bernoulli, 164 | % Heuristic initialisation based on log-odds 165 | LogOut = (TargetsPseudoLinear*0.9+1)/2; 166 | Mu = PHI \ (log(LogOut./(1-LogOut))); 167 | % 168 | case LIKELIHOOD.Poisson 169 | % Heuristic initialisation based on log 170 | Mu = PHI \ TargetsPseudoLinear; 171 | % 172 | end 173 | else 174 | if length(SETTINGS.Mu)~=length(SETTINGS.Relevant) 175 | error('Basis length (%d) should equal weight vector length (%d)', ... 176 | length(SETTINGS.Relevant), length(SETTINGS.Mu)) 177 | end 178 | SB2_Diagnostic(OPTIONS, 2,'Initialising with supplied weights\n'); 179 | % We may have extra "freebasis" functions without supplied weights 180 | % - set those to zero for now 181 | Mu = [Mu ; zeros(length(Extra),1)]; 182 | end 183 | 184 | % 3) the hyperparameters 185 | 186 | if isempty(SETTINGS.Alpha) 187 | switch LIKELIHOOD.InUse 188 | % 189 | case LIKELIHOOD.Gaussian, 190 | % Exact for single basis function case (diag irrelevant), 191 | % heuristic in the multiple case 192 | p = diag(PHI'*PHI)*beta; 193 | q = (PHI'*Targets)*beta; 194 | Alpha = p.^2./(q.^2-p); % 195 | % Its possible that one of the basis functions could already be 196 | % irrelevant (alpha<0), so trap that 197 | if all(Alpha<0) 198 | SB2_Diagnostic(OPTIONS, 1,... 199 | ['WARNING: no relevant basis function at' ... 200 | ' initialisation\n']); 201 | end 202 | % The main algorithm will handle these automatically shortly 203 | % (i.e. prune them) 204 | Alpha(Alpha<0) = INIT_ALPHA_MAX; 205 | % 206 | case {LIKELIHOOD.Bernoulli, LIKELIHOOD.Poisson} 207 | % Heuristic initialisation, trapping Mu==0 208 | Alpha = 1./(Mu+(Mu==0)).^2; 209 | % Limit max/min alpha 210 | Alpha(AlphaINIT_ALPHA_MAX) = INIT_ALPHA_MAX; 212 | % 213 | end 214 | if M==1 215 | SB2_Diagnostic(OPTIONS,2,'Initial alpha = %g\n', Alpha); 216 | end 217 | else 218 | if length(SETTINGS.Alpha)~=length(SETTINGS.Relevant) 219 | error('Basis length (%d) should equal alpha vector length (%d)', ... 220 | length(SETTINGS.Relevant), length(SETTINGS.Alpha)) 221 | end 222 | % We may have extra "freebasis" functions without supplied alpha 223 | % - set those to zero for now 224 | Alpha = [SETTINGS.Alpha ; zeros(length(Extra),1)]; 225 | end 226 | 227 | % Finally - set Alphas to zero for free basis 228 | ALPHA_ZERO = eps; 229 | Alpha(ismember(Used,OPTIONS.freeBasis)) = ALPHA_ZERO; -------------------------------------------------------------------------------- /SB2_Release_200/SB2_Likelihoods.m: -------------------------------------------------------------------------------- 1 | % SB2_LIKELIHOODS Convenience function to encapsulate likelihood types 2 | % 3 | % LIKELIHOOD = SB2_LIKELIHOODS(TYPE) 4 | % 5 | % OUTPUT ARGUMENTS: 6 | % 7 | % LIKELIHOOD A structure containing an enumeration of possible 8 | % likelihood types, and specification of the type in use. 9 | % 10 | % INPUT ARGUMENTS: 11 | % 12 | % TYPE A string. One of: 13 | % 'Gaussian' 14 | % 'Bernoulli' 15 | % 'Poisson' 16 | % 17 | % EXAMPLE: 18 | % 19 | % LIKELIHOOD = SB2_Likelihoods('Gaussian'); 20 | % 21 | % NOTES: 22 | % 23 | % This function is "for internal use only" and exists purely for convenience 24 | % of coding and to facilitate potential future expansion. 25 | % 26 | 27 | % 28 | % Copyright 2009, Vector Anomaly Ltd 29 | % 30 | % This file is part of the SPARSEBAYES library for Matlab (V2.0). 31 | % 32 | % SPARSEBAYES is free software; you can redistribute it and/or modify it 33 | % under the terms of the GNU General Public License as published by the Free 34 | % Software Foundation; either version 2 of the License, or (at your option) 35 | % any later version. 36 | % 37 | % SPARSEBAYES is distributed in the hope that it will be useful, but WITHOUT 38 | % ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or 39 | % FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for 40 | % more details. 41 | % 42 | % You should have received a copy of the GNU General Public License along 43 | % with SPARSEBAYES in the accompanying file "licence.txt"; if not, write to 44 | % the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, 45 | % MA 02110-1301 USA 46 | % 47 | % Contact the author: m a i l [at] m i k e t i p p i n g . c o m 48 | % 49 | function LIKELIHOOD = SB2_Likelihoods(likelihood_) 50 | 51 | % For convenience in code, and later expandability, 52 | % we "enum" the available likelihood models 53 | % 54 | % Continuous 55 | LIKELIHOOD.Gaussian = 1; 56 | % Discrete 57 | LIKELIHOOD.Bernoulli = 2; 58 | LIKELIHOOD.Poisson = 3; 59 | 60 | % Feel free to add your own ... and work out the maths :-) 61 | 62 | % Determine the likelihood to be used 63 | % 64 | LIKELIHOOD.InUse = 0; 65 | % 66 | switch upper(likelihood_(1:5)) 67 | case 'GAUSS', 68 | LIKELIHOOD.InUse = LIKELIHOOD.Gaussian; 69 | % 70 | case 'BERNO', 71 | LIKELIHOOD.InUse = LIKELIHOOD.Bernoulli; 72 | % 73 | case 'POISS', 74 | LIKELIHOOD.InUse = LIKELIHOOD.Poisson; 75 | % 76 | otherwise 77 | error('Unknown likelihood ''%s'' specified\n', likelihood_) 78 | end 79 | 80 | -------------------------------------------------------------------------------- /SB2_Release_200/SB2_Manual.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/iqiukp/RVM-MATLAB/fd70db51f1333525ad4603a35b0c97ccf6ba68ba/SB2_Release_200/SB2_Manual.pdf -------------------------------------------------------------------------------- /SB2_Release_200/SB2_ParameterSettings.m: -------------------------------------------------------------------------------- 1 | % SB2_PARAMETERSETTINGS User parameter initialisation for SPARSEBAYES 2 | % 3 | % SETTINGS = SB2_PARAMETERSETTINGS(parameter1, value1, parameter2, value2,...) 4 | % 5 | % OUTPUT ARGUMENTS: 6 | % 7 | % SETTINGS An initialisation structure to pass to SPARSEBAYES 8 | % 9 | % INPUT ARGUMENTS: 10 | % 11 | % Optional number of parameter-value pairs to specify some, all, or 12 | % none of the following: 13 | % 14 | % BETA (Gaussian) noise precision (inverse variance) 15 | % NOISESTD (Gaussian) noise standard deviation 16 | % RELEVANT Indices of columns of basis matrix to use at start-up 17 | % WEIGHTS Corresponding vector of weights to RELEVANT 18 | % ALPHA Corresponding vector of hyperparameter values to RELEVANT 19 | % 20 | % EXAMPLE: 21 | % 22 | % SETTINGS = SB2_ParameterSettings('NoiseStd',0.1); 23 | % 24 | % NOTES: 25 | % 26 | % 1. If no input arguments are supplied, defaults (effectively an 27 | % empty structure) will be returned. 28 | % 29 | % 2. If both BETA and NOISESTD are specified, BETA will take 30 | % precedence. 31 | % 32 | % 3. RELEVANT may be specified without WEIGHTS or ALPHA (these will be 33 | % sensibly initialised later). 34 | % 35 | % 4. If RELEVANT is specified, WEIGHTS may be specified also without ALPHA. 36 | % 37 | 38 | % 39 | % Copyright 2009, Vector Anomaly Ltd 40 | % 41 | % This file is part of the SPARSEBAYES library for Matlab (V2.0). 42 | % 43 | % SPARSEBAYES is free software; you can redistribute it and/or modify it 44 | % under the terms of the GNU General Public License as published by the Free 45 | % Software Foundation; either version 2 of the License, or (at your option) 46 | % any later version. 47 | % 48 | % SPARSEBAYES is distributed in the hope that it will be useful, but WITHOUT 49 | % ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or 50 | % FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for 51 | % more details. 52 | % 53 | % You should have received a copy of the GNU General Public License along 54 | % with SPARSEBAYES in the accompanying file "licence.txt"; if not, write to 55 | % the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, 56 | % MA 02110-1301 USA 57 | % 58 | % Contact the author: m a i l [at] m i k e t i p p i n g . c o m 59 | % 60 | function SETTINGS = SB2_ParameterSettings(varargin) 61 | 62 | % Ensure arguments are supplied in pairs 63 | % 64 | if rem(nargin,2) 65 | error('Arguments to SB2_ParameterSettings should be (property, value) pairs') 66 | end 67 | % Any settings specified? 68 | numSettings = nargin/2; 69 | 70 | %% Defaults - over-ridden later if requested 71 | 72 | % Two options for setting noise level (purely for convenience) 73 | % - if 'beta' set, 'noiseStdDev' will be over-ridden 74 | % 75 | SETTINGS.beta = []; 76 | SETTINGS.noiseStdDev = []; 77 | % 78 | SETTINGS.Relevant = []; 79 | SETTINGS.Mu = []; 80 | SETTINGS.Alpha = []; 81 | 82 | %% Requested overrides 83 | 84 | % Parse string/variable pairs 85 | for n=1:numSettings 86 | property_ = varargin{(n-1)*2+1}; 87 | value = varargin{(n-1)*2+2}; 88 | switch upper(property_) 89 | case 'BETA', 90 | SETTINGS.beta = value; 91 | case 'NOISESTD', 92 | SETTINGS.noiseStdDev = value; 93 | case 'ALPHA', 94 | SETTINGS.Alpha = value; 95 | case 'WEIGHTS', 96 | SETTINGS.Mu = value; 97 | case 'RELEVANT', 98 | SETTINGS.Relevant = value; 99 | otherwise, 100 | error('Unrecognised initialisation property: ''%s''\n', property_) 101 | end 102 | end 103 | -------------------------------------------------------------------------------- /SB2_Release_200/SB2_PosteriorMode.m: -------------------------------------------------------------------------------- 1 | % SB2_POSTERIORMODE Posterior mode-finder for the SPARSEBAYES algorithm 2 | % 3 | % [MU, U, BETA, LIKELIHOODMODE, BADHESS] = ... 4 | % SB2_POSTERIORMODE(LIKELIHOOD,BASIS,TARGETS,ALPHA,MU,ITSMAX,OPTIONS) 5 | % 6 | % OUTPUT ARGUMENTS: 7 | % 8 | % MU Parameter values at the mode 9 | % U Cholesky factor of the covariance at the mode 10 | % BETA Vector of pseudo-noise variances at the mode 11 | % LIKELIHOODMODE Data likelihood at the mode 12 | % BADHESS Returns true if Hessian is "bad" (becomes 13 | % non-positive-definite during maximisation) 14 | % 15 | % INPUT ARGUMENTS: 16 | % 17 | % LIKELIHOOD LIKELIHOOD structure 18 | % BASIS Current relevant basis matrix 19 | % TARGETS N-vector with target output values 20 | % ALPHA Current hyperparameters 21 | % MU Current weights 22 | % ITSMAX Maximum number of iterations to run 23 | % OPTIONS Standard OPTIONS structure (only affects diagnostics) 24 | % 25 | % NOTES: 26 | % 27 | % SB2_POSTERIORMODE finds the posterior mode (with respect to the 28 | % weights) of the likelihood function in the non-Gaussian case to 29 | % facilitate subsequent Laplace approximation. 30 | % 31 | % This function is intended for internal use by SPARSEBAYES only (within 32 | % SB2_FullStatistics). 33 | % 34 | 35 | % 36 | % Copyright 2009, Vector Anomaly Ltd 37 | % 38 | % This file is part of the SPARSEBAYES library for Matlab (V2.0). 39 | % 40 | % SPARSEBAYES is free software; you can redistribute it and/or modify it 41 | % under the terms of the GNU General Public License as published by the Free 42 | % Software Foundation; either version 2 of the License, or (at your option) 43 | % any later version. 44 | % 45 | % SPARSEBAYES is distributed in the hope that it will be useful, but WITHOUT 46 | % ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or 47 | % FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for 48 | % more details. 49 | % 50 | % You should have received a copy of the GNU General Public License along 51 | % with SPARSEBAYES in the accompanying file "licence.txt"; if not, write to 52 | % the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, 53 | % MA 02110-1301 USA 54 | % 55 | % Contact the author: m a i l [at] m i k e t i p p i n g . c o m 56 | % 57 | function [Mu, U, Beta, likelihoodMode, badHess] = ... 58 | SB2_PosteriorMode(LIKELIHOOD,BASIS,Targets,Alpha,Mu,itsMax,OPTIONS) 59 | 60 | % TOLERANCES 61 | % 62 | % Termination criterion for each gradient dimension 63 | % 64 | GRADIENT_MIN = 1e-6; 65 | % 66 | % Minimum fraction of the full Newton step considered 67 | % 68 | STEP_MIN = 1/(2^8); 69 | % 70 | 71 | [N, M] = size(BASIS); 72 | A = diag(Alpha); 73 | 74 | % NB: for historical reasons, we work in term of error here (negative 75 | % log-liklihood) and minimise 76 | 77 | % 78 | % Get current model output and data error 79 | % 80 | BASIS_Mu = BASIS*Mu; % Linear output 81 | [dataError, y] = SB2_DataError(LIKELIHOOD, BASIS_Mu, Targets); 82 | % 83 | % Add on the weight penalty 84 | % 85 | regulariser = (Alpha'*(Mu.^2))/2; 86 | newTotalError = dataError + regulariser; 87 | % 88 | badHess = false; 89 | errorLog = zeros(itsMax,1); 90 | % 91 | for iteration=1:itsMax 92 | % 93 | % Log the error value each iteration 94 | % 95 | errorLog(iteration) = newTotalError; 96 | SB2_Diagnostic(OPTIONS,4,'PM cycle: %2d\t error: %.6f\n', ... 97 | iteration, errorLog(iteration)); 98 | 99 | % Construct the gradient 100 | % 101 | e = (Targets-y); 102 | g = BASIS'*e - Alpha.*Mu; 103 | % 104 | % Compute the likelihood-dependent analogue of the noise precision. 105 | % NB: Beta now a vector. 106 | % 107 | switch LIKELIHOOD.InUse 108 | case LIKELIHOOD.Bernoulli 109 | Beta = y.*(1-y); 110 | case LIKELIHOOD.Poisson 111 | Beta = y; 112 | end 113 | % Compute the Hessian 114 | BASIS_B = BASIS .* (Beta * ones(1,M)); 115 | H = (BASIS_B'*BASIS + A); 116 | % Invert Hessian via Cholesky, watching out for ill-conditioning 117 | [U, pdErr] = chol(H); 118 | % Make sure its positive definite 119 | if pdErr 120 | % If you see this, it's *probably* the result of a bad choice of 121 | % basis. e.g. a kernel with too large a "width" 122 | % 123 | SB2_Diagnostic(OPTIONS, 1,... 124 | '** Warning ** Ill-conditioned Hessian (%g)\n', ... 125 | rcond(H)); 126 | badHess = true; 127 | U = []; 128 | Beta = []; 129 | likelihoodMode = []; 130 | return 131 | end 132 | % 133 | % Before progressing, check for termination based on the gradient norm 134 | % 135 | if all(abs(g)STEP_MIN 149 | % Follow gradient to get new value of parameters 150 | % 151 | Mu_new = Mu + step*DeltaMu; 152 | BASIS_Mu = BASIS*Mu_new; 153 | % 154 | % Compute outputs and error at new point 155 | % 156 | [dataError,y] = SB2_DataError(LIKELIHOOD,BASIS_Mu,Targets); 157 | regulariser = (Alpha'*(Mu_new.^2))/2; 158 | newTotalError = dataError + regulariser; 159 | % 160 | % Test that we haven't made things worse 161 | % 162 | if newTotalError>=errorLog(iteration) 163 | % If so, back off! 164 | % 165 | step = step/2; 166 | SB2_Diagnostic(OPTIONS, 4,['PM error increase! Backing off to l=' ... 167 | '%.3f\n'], step); 168 | else 169 | Mu = Mu_new; 170 | step = 0; % this will force exit from the "while" loop 171 | end 172 | end 173 | % 174 | % If we get here with non-zero "step", it means that the smallest 175 | % offset from the current point along the "downhill" direction did not 176 | % lead to a decrease in error. In other words, we must be 177 | % infinitesimally close to a minimum (which is OK). 178 | % 179 | if step 180 | SB2_Diagnostic(OPTIONS, 4, ... 181 | 'PM stopping due to back-off limit (|g| = %g)\n', ... 182 | max(abs(g))); 183 | break; 184 | end 185 | end 186 | % 187 | % Simple computation of return value of log likelihood at mode 188 | % 189 | likelihoodMode = -dataError; 190 | 191 | %%%%%%%%%%%%%%%%%%%%%% 192 | % 193 | % Support function 194 | % 195 | %%%%%%%%%%%%%%%%%%%%%% 196 | 197 | function [e,y] = SB2_DataError(LIKELIHOOD,BASIS_Mu,Targets) 198 | 199 | switch LIKELIHOOD.InUse 200 | case LIKELIHOOD.Bernoulli 201 | y = SB2_Sigmoid(BASIS_Mu); 202 | % Handle probability zero cases 203 | y0 = (y==0); 204 | y1 = (y==1); 205 | if any(y0(Targets>0)) || any(y1(Targets<1)) 206 | % Error infinite when model gives zero probability in 207 | % contradiction to data 208 | e = inf; 209 | else 210 | % Any y=0 or y=1 cases must now be accompanied by appropriate 211 | % output=0 or output=1 values, so can be excluded. 212 | e = -(Targets(~y0)'*log(y(~y0)) + (1-Targets(~y1))'*log(1-y(~y1))); 213 | end 214 | % 215 | case LIKELIHOOD.Poisson 216 | y = exp(BASIS_Mu); 217 | e = -sum(Targets.*BASIS_Mu - y); 218 | % 219 | end 220 | -------------------------------------------------------------------------------- /SB2_Release_200/SB2_PreProcessBasis.m: -------------------------------------------------------------------------------- 1 | % SB2_PREPROCESSBASIS Do basis matrix pre-processing for SPARSEBAYES 2 | % 3 | % [BASIS, SCALES] = SB2_PREPROCESSBASIS(BASIS) 4 | % 5 | % OUTPUT ARGUMENTS: 6 | % 7 | % BASIS NxM matrix of basis vectors appropriately pre-processed 8 | % (scaled to unit length per column) 9 | % 10 | % SCALES Vector of scaling factors applied 11 | % 12 | % 13 | % INPUT ARGUMENTS: 14 | % 15 | % BASIS NxM matrix of basis vectors (one column per basis function) 16 | % 17 | % NOTES: 18 | % 19 | % Simply normalises the basis vectors to unit length, returning the 20 | % original lengths so that the weights can be rescaled appropriately 21 | % before returning to the user. 22 | % 23 | 24 | % 25 | % Copyright 2009, Vector Anomaly Ltd 26 | % 27 | % This file is part of the SPARSEBAYES library for Matlab (V2.0). 28 | % 29 | % SPARSEBAYES is free software; you can redistribute it and/or modify it 30 | % under the terms of the GNU General Public License as published by the Free 31 | % Software Foundation; either version 2 of the License, or (at your option) 32 | % any later version. 33 | % 34 | % SPARSEBAYES is distributed in the hope that it will be useful, but WITHOUT 35 | % ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or 36 | % FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for 37 | % more details. 38 | % 39 | % You should have received a copy of the GNU General Public License along 40 | % with SPARSEBAYES in the accompanying file "licence.txt"; if not, write to 41 | % the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, 42 | % MA 02110-1301 USA 43 | % 44 | % Contact the author: m a i l [at] m i k e t i p p i n g . c o m 45 | % 46 | function [BASIS, Scales] = SB2_PreProcessBasis(BASIS) 47 | 48 | % 49 | [N,M] = size(BASIS); 50 | % 51 | % Compute "lengths" of basis vectors (columns of BASIS) 52 | % 53 | Scales = sqrt(sum(BASIS.^2)); 54 | % 55 | % Work-around divide-by-zero inconvenience 56 | % 57 | Scales(Scales==0) = 1; 58 | % 59 | % Normalise each basis vector to "unit length" 60 | % 61 | for m=1:M 62 | BASIS(:,m) = BASIS(:,m) / Scales(m); 63 | end 64 | -------------------------------------------------------------------------------- /SB2_Release_200/SB2_Sigmoid.m: -------------------------------------------------------------------------------- 1 | % SB2_SIGMOID Generic sigmoid function for SPARSEBAYES computations 2 | % 3 | % Y = SB2_SIGMOID(X) 4 | % 5 | % Implements the function f(x) = 1 / (1+exp(-x)) 6 | % 7 | 8 | % 9 | % Copyright 2009, Vector Anomaly Ltd 10 | % 11 | % This file is part of the SPARSEBAYES library for Matlab (V2.0). 12 | % 13 | % SPARSEBAYES is free software; you can redistribute it and/or modify it 14 | % under the terms of the GNU General Public License as published by the Free 15 | % Software Foundation; either version 2 of the License, or (at your option) 16 | % any later version. 17 | % 18 | % SPARSEBAYES is distributed in the hope that it will be useful, but WITHOUT 19 | % ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or 20 | % FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for 21 | % more details. 22 | % 23 | % You should have received a copy of the GNU General Public License along 24 | % with SPARSEBAYES in the accompanying file "licence.txt"; if not, write to 25 | % the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, 26 | % MA 02110-1301 USA 27 | % 28 | % Contact the author: m a i l [at] m i k e t i p p i n g . c o m 29 | % 30 | function y = SB2_Sigmoid(x) 31 | 32 | y = 1 ./ (1+exp(-x)); 33 | -------------------------------------------------------------------------------- /SB2_Release_200/SB2_UserOptions.m: -------------------------------------------------------------------------------- 1 | % SB2_USEROPTIONS User option specification for SPARSEBAYES 2 | % 3 | % OPTIONS = SB2_USEROPTIONS(parameter1, value1, parameter2, value2,...) 4 | % 5 | % OUTPUT ARGUMENTS: 6 | % 7 | % OPTIONS An options structure to pass to SPARSEBAYES 8 | % 9 | % INPUT ARGUMENTS: 10 | % 11 | % Optional number of parameter-value pairs to specify the following: 12 | % 13 | % ITERATIONS Number of interations to run for. 14 | % 15 | % TIME Time limit to run for, expressed as a space-separated 16 | % string. e.g. '1.5 hours', '30 minutes', '1 second'. 17 | % 18 | % DIAGNOSTICLEVEL Integer [0,4] or string to determine the verbosity of 19 | % diagnostic output. 20 | % 0 or 'ZERO' or 'NONE' No output 21 | % 1 or 'LOW' Low level of output 22 | % 2 or 'MEDIUM' etc... 23 | % 3 or 'HIGH' 24 | % 4 or 'ULTRA' 25 | % 26 | % DIAGNOSTICFILE Filename to write diagnostics to file instead of 27 | % the default stdout. 28 | % 29 | % MONITOR Integer number: diagnostic information is output 30 | % every MONITOR iterations. 31 | % 32 | % FIXEDNOISE True/false whether the Gaussian noise is to be fixed 33 | % (default: false. 34 | % 35 | % FREEBASIS Indices of basis vectors considered "free" and not 36 | % constrained by the Bayesian prior (e.g. the "bias"). 37 | % 38 | % CALLBACK External function to call each iteration of the algorithm 39 | % (string). Intended to facilitate graphical demos etc. 40 | % 41 | % CALLBACKDATA Arbitrary additional data to pass to the CALLBACK 42 | % function. 43 | % 44 | % EXAMPLE: 45 | % 46 | % OPTIONS = SB2_UserOptions('diagnosticLevel','medium',... 47 | % 'monitor',25,... 48 | % 'diagnosticFile', 'logfile.txt'); 49 | % 50 | % NOTES: 51 | % 52 | % Each option (field of OPTIONS) is given a default value in 53 | % SB2_USEROPTIONS. Any supplied property-value pairs over-ride those 54 | % defaults. 55 | % 56 | 57 | % 58 | % Copyright 2009, Vector Anomaly Ltd 59 | % 60 | % This file is part of the SPARSEBAYES library for Matlab (V2.0). 61 | % 62 | % SPARSEBAYES is free software; you can redistribute it and/or modify it 63 | % under the terms of the GNU General Public License as published by the Free 64 | % Software Foundation; either version 2 of the License, or (at your option) 65 | % any later version. 66 | % 67 | % SPARSEBAYES is distributed in the hope that it will be useful, but WITHOUT 68 | % ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or 69 | % FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for 70 | % more details. 71 | % 72 | % You should have received a copy of the GNU General Public License along 73 | % with SPARSEBAYES in the accompanying file "licence.txt"; if not, write to 74 | % the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, 75 | % MA 02110-1301 USA 76 | % 77 | % Contact the author: m a i l [at] m i k e t i p p i n g . c o m 78 | % 79 | function OPTIONS = SB2_UserOptions(varargin) 80 | 81 | % Ensure arguments are supplied in pairs 82 | % 83 | if rem(nargin,2) 84 | error('Arguments to SB2_UserOptions should be (property, value) pairs') 85 | end 86 | % Any options specified? 87 | numSettings = nargin/2; 88 | 89 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 90 | 91 | % Set defaults 92 | % 93 | % Assume we will infer the noise in the Gaussian case 94 | % 95 | OPTIONS.fixedNoise = false; 96 | % 97 | % Option to allow subset of the basis (e.g. bias) to be unregularised 98 | % 99 | OPTIONS.freeBasis = []; 100 | % 101 | % Option to set max iterations to run for 102 | % 103 | OPTIONS.iterations = 10000; 104 | % 105 | % Option to set max time to run for 106 | % 107 | OPTIONS.time = 10000; % seconds 108 | % 109 | % Set options for monitoring and recording the algorithm's progress 110 | % 111 | OPTIONS.monitor = 0; 112 | OPTIONS.diagnosticLevel = 0; 113 | OPTIONS.diagnosticFID = 1; % stdout 114 | OPTIONS.diagnosticFile_ = []; 115 | % 116 | % Option to call a function during each iteration (to create demos etc) 117 | % 118 | OPTIONS.callback = false; 119 | OPTIONS.callbackFunc = []; 120 | OPTIONS.callbackData = {}; 121 | 122 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 123 | 124 | % Parse string/variable pairs 125 | 126 | for n=1:numSettings 127 | property_ = varargin{(n-1)*2+1}; 128 | value = varargin{(n-1)*2+2}; 129 | switch upper(property_) 130 | % 131 | case 'FIXEDNOISE', 132 | OPTIONS.fixedNoise = value; 133 | % 134 | case 'FREEBASIS', 135 | OPTIONS.freeBasis = value; 136 | % 137 | case 'ITERATIONS', 138 | OPTIONS.iterations = value; 139 | % 140 | case 'TIME', 141 | OPTIONS.time = timeInSeconds(value); 142 | % 143 | case 'MONITOR', 144 | OPTIONS.monitor = value; 145 | % 146 | case 'DIAGNOSTICLEVEL', 147 | OPTIONS.diagnosticLevel = value; 148 | MAX_LEVEL = 4; 149 | if ischar(value) 150 | switch upper(value) 151 | case {'ZERO','NONE'}, 152 | OPTIONS.diagnosticLevel = 0; 153 | case 'LOW', 154 | OPTIONS.diagnosticLevel = 1; 155 | case 'MEDIUM', 156 | OPTIONS.diagnosticLevel = 2; 157 | case 'HIGH', 158 | OPTIONS.diagnosticLevel = 3; 159 | case 'ULTRA', 160 | OPTIONS.diagnosticLevel = 4; 161 | otherwise, 162 | error('Unrecognised textual diagnostic level: ''%s''\n', value) 163 | end 164 | elseif isnumeric(value) 165 | if value>=0 & value<=MAX_LEVEL 166 | OPTIONS.diagnosticLevel = value; 167 | else 168 | error(['Supplied level should be integer in [0,%d], '... 169 | 'or one of ZERO/LOW/MEDIUM/HIGH/ULTRA'], MAX_LEVEL); 170 | 171 | end 172 | end 173 | % 174 | case 'DIAGNOSTICFILE', 175 | OPTIONS.diagnosticFile_ = value; 176 | OPTIONS.diagnosticFID = -1; % It will be opened later 177 | % 178 | case 'CALLBACK', 179 | OPTIONS.callback = true; 180 | OPTIONS.callbackFunc = value; 181 | if exist(OPTIONS.callbackFunc)~=2 182 | error('Callback function ''%s'' does not appear to exist\n', value) 183 | end 184 | % 185 | case 'CALLBACKDATA', 186 | OPTIONS.callbackData = value; 187 | otherwise, 188 | error('Unrecognised user option: ''%s''\n', property_) 189 | % 190 | end 191 | end 192 | 193 | %% 194 | %% Support function: parse time specification 195 | %% 196 | function s = timeInSeconds(value_) 197 | % 198 | [v_ r_] = strtok(value_,' '); 199 | v = str2num(v_); 200 | r_(isspace(r_)) = []; 201 | switch upper(r_) 202 | case {'SECONDS', 'SECOND'} 203 | s = v; 204 | case {'MINUTES', 'MINUTE'} 205 | s = 60*v; 206 | case {'HOURS', 'HOUR'} 207 | s = 3600*v; 208 | otherwise, 209 | error('Badly formed time string: ''%s''', value_) 210 | end 211 | % Returns time in seconds 212 | % -------------------------------------------------------------------------------- /SB2_Release_200/SparseBayes.m: -------------------------------------------------------------------------------- 1 | % SPARSEBAYES Sparse Bayesian modelling: main estimation algorithm 2 | % 3 | % [PARAMETER, HYPERPARAMETER, DIAGNOSTIC] = ... 4 | % SPARSEBAYES(LIKELIHOOD, BASIS, TARGETS, OPTIONS, SETTINGS) 5 | % 6 | % OUTPUT ARGUMENTS: 7 | % 8 | % PARAMETER Structure specifying inferred primary parameters: 9 | % 10 | % .Value Vector of weight values 11 | % .Relevant Vector of corresponding indices of relevant 12 | % columns of BASIS matrix (sorted ascending) 13 | % 14 | % HYPERPARAMETER Structure specifying inferred hyperparameters: 15 | % 16 | % .Alpha Vector of weight precision values 17 | % .beta Noise precision (Gaussian likelihood case) 18 | % 19 | % DIAGNOSTIC Structure containing various diagnostics: 20 | % 21 | % .Gamma Vector of "well-determined" factors [0,1] for 22 | % relevant weights 23 | % .Likelihood Vector of evolving log-marginal-likelihood 24 | % .iterations Number of iterations run 25 | % .S_Factor Vector of S ("Sparsity") values for relevant weights 26 | % .Q_Factor Vector of Q ("Quality") values for relevant weights 27 | % 28 | % INPUT ARGUMENTS: 29 | % 30 | % LIKELIHOOD String comprising one of 'Gaussian', 'Bernoulli' or 'Poisson' 31 | % 32 | % BASIS NxM matrix of basis vectors (one column per basis function) 33 | % 34 | % TARGETS N-vector with target output values 35 | % 36 | % OPTIONS User-specified settings via SB2_USEROPTIONS [Optional] 37 | % 38 | % SETTINGS Initialisation of main parameter values via 39 | % SB2_PARAMETERSETTINGS [Optional] 40 | % 41 | % NOTES: 42 | % 43 | % SPARSEBAYES is the implementation of the main algorithm for parameter 44 | % inference in "sparse Bayesian" models. 45 | % 46 | % Given inputs (BASIS), desired outputs (TARGETS) and an appropriate 47 | % LIKELIHOOD function, SPARSEBAYES will optimise the log-marginal-likelihood 48 | % of the corresponding sparse Bayesian model and should return (given 49 | % reasonable choice of basis) a sparse vector of model parameters. 50 | % 51 | % OPTIONS and SETTINGS arguments may be omitted, and if so, will assume 52 | % sensible default values. 53 | % 54 | % SEE ALSO: 55 | % 56 | % SB2_USEROPTIONS: available user-definable options, such as the 57 | % number of iterations to run for, level of diagnostic output etc. 58 | % 59 | % SB2_PARAMETERSETTINGS: facility to change default initial values for 60 | % parameters and hyperparameters. 61 | % 62 | % SB2_CONTROLSETTINGS: hard-wired internal algorithm settings. 63 | % 64 | % The main algorithm is based upon that outlined in "Fast marginal 65 | % likelihood maximisation for sparse Bayesian models", by Tipping & Faul, in 66 | % Proceedings of AISTATS 2003. That paper may be downloaded from 67 | % www.relevancevector.com or via the conference online proceedings site at 68 | % http://research.microsoft.com/conferences/aistats2003/proceedings/. 69 | % 70 | 71 | % 72 | % Copyright 2009, Vector Anomaly Ltd 73 | % 74 | % This file is part of the SPARSEBAYES library for Matlab (V2.0). 75 | % 76 | % SPARSEBAYES is free software; you can redistribute it and/or modify it 77 | % under the terms of the GNU General Public License as published by the Free 78 | % Software Foundation; either version 2 of the License, or (at your option) 79 | % any later version. 80 | % 81 | % SPARSEBAYES is distributed in the hope that it will be useful, but WITHOUT 82 | % ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or 83 | % FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for 84 | % more details. 85 | % 86 | % You should have received a copy of the GNU General Public License along 87 | % with SPARSEBAYES in the accompanying file "licence.txt"; if not, write to 88 | % the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, 89 | % MA 02110-1301 USA 90 | % 91 | % Contact the author: m a i l [at] m i k e t i p p i n g . c o m 92 | % 93 | function [PARAMETER, HYPERPARAMETER, DIAGNOSTIC] = ... 94 | SparseBayes(likelihood_, BASIS, Targets, OPTIONS, SETTINGS) 95 | 96 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 97 | %% 98 | %% SET UP INITIAL PARAMETERS, USER OPTIONS AND ALGORITHM CONTROL SETTINGS 99 | %% 100 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 101 | 102 | %% If no initial parameter setting structure passed, import the defaults 103 | if ~exist('SETTINGS','var') 104 | SETTINGS = SB2_ParameterSettings; 105 | end 106 | 107 | %% If no user options passed, import defaults 108 | if ~exist('OPTIONS','var') 109 | OPTIONS = SB2_UserOptions; 110 | end 111 | 112 | %% Any sanity checks on options and initialisation here 113 | 114 | % Error if fixed noise specified but value not set 115 | % 116 | if OPTIONS.fixedNoise && isempty(SETTINGS.beta) && ... 117 | isempty(SETTINGS.noiseStdDev) 118 | error('Option to fix noise variance is set but value is not supplied.') 119 | end 120 | 121 | %% Get the default algorithm control settings 122 | 123 | CONTROLS = SB2_ControlSettings; 124 | 125 | % Start the clock now for diagnostic purposes (and to allow the algorithm to 126 | % run for a fixed time) 127 | % 128 | t0 = clock; 129 | 130 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 131 | %% 132 | %% INITIALISATION 133 | %% 134 | %% Pre-process basis, set up internal parameters according to initial 135 | %% settings and likelihod specification 136 | %% 137 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 138 | 139 | % Kick off diagnostics (primarily, open log file if specified) 140 | % 141 | OPTIONS = SB2_Diagnostic(OPTIONS, 'start'); 142 | 143 | % 144 | % Initialise everything, based on SETTINGS and OPTIONS 145 | % 146 | [LIKELIHOOD, BASIS, BasisScales, Alpha, beta, Mu, PHI, Used] = ... 147 | SB2_Initialisation(likelihood_, BASIS, Targets, SETTINGS, OPTIONS); 148 | % 149 | % Cache some values for later efficiency 150 | % 151 | if LIKELIHOOD.InUse==LIKELIHOOD.Gaussian 152 | % It will be computationally advantageous to "cache" this quantity 153 | % in the Gaussian case 154 | BASIS_PHI = BASIS'*PHI; 155 | else 156 | BASIS_PHI = []; 157 | end 158 | BASIS_Targets = BASIS'*Targets; 159 | 160 | % FULL COMPUTATION 161 | % 162 | % Initialise with a full explicit computation of the statistics 163 | % 164 | % NOTE: The AISTATS paper uses "S/Q" (upper case) to denote the key 165 | % "sparsity/quality" Factors for "included" basis functions, and "s/q" 166 | % (lower case) for the factors calculated when the relevant basis 167 | % functions are "excluded". 168 | % 169 | % Here, for greater clarity: 170 | % 171 | % All S/Q are denoted by vectors S_in, Q_in 172 | % All s/q are denoted by vectors S_out, Q_out 173 | % 174 | [SIGMA,Mu,S_in,Q_in,S_out,Q_out,Factor,logML,Gamma,BASIS_B_PHI,beta] = ... 175 | SB2_FullStatistics(LIKELIHOOD,BASIS,PHI,Targets,Used, ... 176 | Alpha,beta,Mu,BASIS_PHI,BASIS_Targets,OPTIONS); 177 | 178 | % 179 | % Avoid falling over in pathological case of zero iterations 180 | % 181 | if OPTIONS.iterations==0 182 | PARAMETER = []; 183 | HYPERPARAMETER = []; 184 | DIAGNOSTIC.Likelihood = logML; 185 | return 186 | end 187 | 188 | % 189 | [N M_full] = size(BASIS); 190 | M = size(PHI,2); 191 | % 192 | % Some diagnostics 193 | % 194 | addCount = 0; 195 | deleteCount = 0; 196 | updateCount = 0; 197 | % 198 | % Create storage to record evolution of log marginal likelihood 199 | % 200 | maxLogSize = OPTIONS.iterations + CONTROLS.BetaUpdateStart + ... 201 | ceil(OPTIONS.iterations/CONTROLS.BetaUpdateFrequency); 202 | logMarginalLog = zeros(maxLogSize,1); 203 | count = 0; 204 | % 205 | % If we're doing basis alignment testing, we'll need to maintain lists of 206 | % those functions that are near identical, both in and out of the current 207 | % model. 208 | % 209 | if CONTROLS.BasisAlignmentTest 210 | Aligned_out = []; 211 | Aligned_in = []; 212 | alignDeferCount = 0; 213 | end 214 | 215 | % ACTION CODES 216 | % 217 | % Assign an integer code to the basic action types 218 | % 219 | ACTION_REESTIMATE = 0; 220 | ACTION_ADD = 1; 221 | ACTION_DELETE = -1; 222 | % 223 | % Some extra types 224 | % 225 | ACTION_TERMINATE = 10; 226 | % 227 | ACTION_NOISE_ONLY = 11; 228 | % 229 | ACTION_ALIGNMENT_SKIP = 12; 230 | 231 | % 232 | % Before kicking off the main loop, call the specified "callback" function 233 | % with "ACTION_ADD" to take account of the initialisation of the model 234 | % 235 | if OPTIONS.callback 236 | feval(OPTIONS.callbackFunc, 0, ACTION_ADD,logML/N, ... 237 | Used, Mu./BasisScales(Used)', SIGMA,... 238 | Alpha, beta, Gamma, PHI,... 239 | OPTIONS.callbackData{:}) 240 | end 241 | 242 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 243 | %% 244 | %% MAIN LOOP 245 | %% 246 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 247 | % 248 | i = 0; % Iteration number 249 | LAST_ITERATION = false; 250 | % 251 | while (~LAST_ITERATION) 252 | % 253 | i = i+1; 254 | % 255 | % "UpdateIteration": set to true if this is an iteration where fast matrix 256 | % update rules can be used compute the appropriate quantities 257 | % 258 | % This can be done if: 259 | % 260 | % - we are using a Gaussian likelihood 261 | % - we are using other likelihoods and we have not specified a full 262 | % posterior mode computation this cycle 263 | % 264 | UpdateIteration = LIKELIHOOD.InUse==LIKELIHOOD.Gaussian || ... 265 | rem(i,CONTROLS.PosteriorModeFrequency); 266 | 267 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 268 | 269 | %% DECISION PHASE 270 | %% 271 | %% Assess all potential actions 272 | %% 273 | 274 | % 275 | % Compute change in likelihood for all possible updates 276 | % 277 | DeltaML = zeros(M_full,1); 278 | % 279 | Action = ACTION_REESTIMATE*ones(M_full,1); % Default 280 | % 281 | % 'Relevance Factor' (Q^S-S) values for basis functions in model 282 | % 283 | UsedFactor = Factor(Used); 284 | 285 | % 286 | % RE-ESTIMATION: must be a POSITIVE 'factor' and already IN the model 287 | % 288 | iu = UsedFactor>CONTROLS.ZeroFactor; 289 | index = Used(iu); 290 | NewAlpha = S_out(index).^2 ./ Factor(index); 291 | Delta = (1./NewAlpha - 1./Alpha(iu)); % Temp vector 292 | % 293 | % Quick computation of change in log-likelihood given all re-estimations 294 | % 295 | DeltaML(index) = (Delta.*(Q_in(index).^2) ./ ... 296 | (Delta.*S_in(index) + 1) - ... 297 | log(1 + S_in(index).*Delta))/2; 298 | 299 | % 300 | % DELETION: if NEGATIVE factor and IN model 301 | % 302 | % But don't delete: 303 | % - any "free" basis functions (e.g. the "bias") 304 | % - if there is only one basis function (M=1) 305 | % 306 | % (In practice, this latter event ought only to happen with the Gaussian 307 | % likelihood when initial noise is too high. In that case, a later beta 308 | % update should 'cure' this.) 309 | % 310 | iu = ~iu; % iu = UsedFactor <= CONTROLS.ZeroFactor 311 | index = Used(iu); 312 | anyToDelete = ~isempty(setdiff(index,OPTIONS.freeBasis)) && M>1; 313 | % 314 | if anyToDelete 315 | % 316 | % Quick computation of change in log-likelihood given all deletions 317 | % 318 | DeltaML(index) = -(Q_out(index).^2 ./ (S_out(index) + Alpha(iu)) - ... 319 | log(1 + S_out(index) ./ Alpha(iu)))/2; 320 | Action(index) = ACTION_DELETE; 321 | % Note: if M==1, DeltaML will be left as zero, which is fine 322 | end 323 | 324 | % 325 | % ADDITION: must be a POSITIVE factor and OUT of the model 326 | % 327 | % Find ALL good factors ... 328 | GoodFactor = Factor>CONTROLS.ZeroFactor; 329 | % ... then mask out those already in model 330 | GoodFactor(Used) = 0; 331 | % ... and then mask out any that are aligned with those in the model 332 | if CONTROLS.BasisAlignmentTest 333 | GoodFactor(Aligned_out) = 0; 334 | end 335 | % 336 | index = find(GoodFactor); 337 | anyToAdd = ~isempty(index); 338 | if anyToAdd 339 | % 340 | % Quick computation of change in log-likelihood given all additions 341 | % 342 | quot = Q_in(index).^2 ./ S_in(index); 343 | DeltaML(index) = (quot - 1 - log(quot))/2; 344 | Action(index) = ACTION_ADD; 345 | end 346 | 347 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 348 | 349 | % Post-process action results to take account of preferences 350 | 351 | % Ensure that nothing happens with "free basis" functions 352 | % 353 | DeltaML(OPTIONS.freeBasis) = 0; 354 | 355 | % If we prefer ADD or DELETE actions over RE-ESTIMATION 356 | % 357 | if (anyToAdd && CONTROLS.PriorityAddition) || ... 358 | (anyToDelete && CONTROLS.PriorityDeletion) 359 | % We won't perform re-estimation this iteration, which we achieve by 360 | % zero-ing out the delta 361 | DeltaML(Action==ACTION_REESTIMATE) = 0; 362 | % Furthermore, we should enforce ADD if preferred and DELETE is not 363 | % - and vice-versa 364 | if (anyToAdd && CONTROLS.PriorityAddition && ~CONTROLS.PriorityDeletion) 365 | DeltaML(Action==ACTION_DELETE) = 0; 366 | end 367 | if (anyToDelete && CONTROLS.PriorityDeletion && ~CONTROLS.PriorityAddition) 368 | DeltaML(Action==ACTION_ADD) = 0; 369 | end 370 | end 371 | 372 | % Finally...we choose the action that results 373 | % in the greatest change in likelihood 374 | % 375 | [deltaLogMarginal nu] = max(DeltaML); 376 | selectedAction = Action(nu); 377 | anyWorthwhileAction = deltaLogMarginal>0; 378 | % 379 | % We need to note if basis nu is already in the model, and if so, 380 | % find its interior index, denoted by "j" 381 | % 382 | if selectedAction==ACTION_REESTIMATE || selectedAction==ACTION_DELETE 383 | j = find(Used==nu); 384 | end 385 | % 386 | % Get the individual basis vector for update and compute its optimal alpha, 387 | % according to equation (20): alpha = S_out^2 / (Q_out^2 - S_out) 388 | % 389 | Phi = BASIS(:,nu); 390 | newAlpha = S_out(nu)^2 / Factor(nu); 391 | 392 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 393 | 394 | % TERMINATION CONDITIONS 395 | % 396 | % Propose to terminate if: 397 | % 398 | % 1. there is no worthwhile (likelihood-increasing) action, OR 399 | % 400 | % 2a. the best action is an ACTION_REESTIMATE but this would only lead to 401 | % an infinitesimal alpha change, AND 402 | % 2b. at the same time there are no potential awaiting deletions 403 | % 404 | if ~anyWorthwhileAction || ... 405 | (selectedAction==ACTION_REESTIMATE && ... 406 | abs(log(newAlpha) - log(Alpha(j)))CONTROLS.AlignmentMax); 432 | numAligned = length(findAligned); 433 | if numAligned>0 434 | % The added basis function is effectively indistinguishable from 435 | % one present already 436 | selectedAction = ACTION_ALIGNMENT_SKIP; 437 | act_ = 'alignment-deferred addition'; 438 | alignDeferCount = alignDeferCount+1; 439 | % Make a note so we don't try this next time 440 | % May be more than one in the model, which we need to note was 441 | % the cause of function 'nu' being rejected 442 | Aligned_out = [Aligned_out ; nu*ones(numAligned,1)]; 443 | Aligned_in = [Aligned_in ; Used(findAligned)]; 444 | end 445 | end 446 | % 447 | % Deletion: reinstate any previously deferred basis functions 448 | % resulting from this basis function 449 | % 450 | if selectedAction==ACTION_DELETE 451 | findAligned = find(Aligned_in==nu); 452 | numAligned = length(findAligned); 453 | if numAligned>0 454 | reinstated = Aligned_out(findAligned); 455 | Aligned_in(findAligned) = []; 456 | Aligned_out(findAligned) = []; 457 | % 458 | r_ = sprintf('%d ', reinstated); 459 | SB2_Diagnostic(OPTIONS,3,'Alignment reinstatement of %s\n', r_); 460 | end 461 | end 462 | end 463 | 464 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 465 | 466 | %% ACTION PHASE 467 | %% 468 | %% Implement above decision 469 | %% 470 | 471 | % We'll want to note if we've made a change which necessitates later 472 | % updating of the statistics 473 | % 474 | UPDATE_REQUIRED = false; 475 | % 476 | switch selectedAction 477 | % 478 | case ACTION_REESTIMATE, 479 | % 480 | % Basis function 'nu' is already in the model, 481 | % and we're re-estimating its corresponding alpha 482 | % 483 | % - should correspond to Appendix A.3 484 | % 485 | oldAlpha = Alpha(j); 486 | Alpha(j) = newAlpha; 487 | s_j = SIGMA(:,j); 488 | deltaInv = 1/(newAlpha - oldAlpha); 489 | kappa = 1/(SIGMA(j,j)+deltaInv); 490 | tmp = kappa*s_j; 491 | SIGMANEW = SIGMA - tmp * s_j'; 492 | deltaMu = -Mu(j)*tmp; 493 | Mu = Mu + deltaMu; 494 | % 495 | if UpdateIteration 496 | S_in = S_in + kappa*(BASIS_B_PHI * s_j).^2; 497 | Q_in = Q_in - BASIS_B_PHI*(deltaMu); 498 | end 499 | updateCount = updateCount+1; 500 | act_ = 're-estimation'; 501 | % 502 | UPDATE_REQUIRED = true; 503 | % 504 | 505 | case ACTION_ADD, 506 | % 507 | % Basis function nu is not in the model, and we're adding it in 508 | % 509 | % - should correspond to Appendix A.2 510 | % 511 | if LIKELIHOOD.InUse==LIKELIHOOD.Gaussian 512 | BASIS_Phi = BASIS'*Phi; 513 | BASIS_PHI = [BASIS_PHI BASIS_Phi]; 514 | B_Phi = beta*Phi; 515 | BASIS_B_Phi = beta*BASIS_Phi; 516 | else 517 | B_Phi = (Phi.*beta); 518 | BASIS_B_phi = BASIS'*B_Phi; 519 | end 520 | tmp = ((B_Phi'*PHI)*SIGMA)'; 521 | % 522 | Alpha = [Alpha ; newAlpha]; 523 | PHI = [PHI Phi]; 524 | % 525 | s_ii = 1/(newAlpha+S_in(nu)); 526 | s_i = -s_ii*tmp; 527 | TAU = -s_i*tmp'; 528 | SIGMANEW = [SIGMA+TAU s_i ; s_i' s_ii]; 529 | mu_i = s_ii*Q_in(nu); 530 | deltaMu = [-mu_i*tmp ; mu_i]; 531 | Mu = [Mu ; 0] + deltaMu; 532 | % 533 | if UpdateIteration 534 | mCi = BASIS_B_Phi - BASIS_B_PHI*tmp; 535 | S_in = S_in - s_ii * mCi.^2; 536 | Q_in = Q_in - mu_i * mCi; 537 | end 538 | Used = [Used; nu]; 539 | addCount = addCount+1; 540 | act_ = 'addition'; 541 | % 542 | UPDATE_REQUIRED = true; 543 | 544 | case ACTION_DELETE, 545 | % 546 | % Basis function nu is in the model, but we're removing it 547 | % 548 | % - should correspond to Appendix A.4 549 | % 550 | if LIKELIHOOD.InUse==LIKELIHOOD.Gaussian 551 | BASIS_PHI(:,j) = []; 552 | end 553 | PHI(:,j) = []; 554 | Alpha(j) = []; 555 | % 556 | s_jj = SIGMA(j,j); 557 | s_j = SIGMA(:,j); 558 | tmp = s_j/s_jj; 559 | SIGMANEW = SIGMA - tmp*s_j'; 560 | SIGMANEW(j,:) = []; 561 | SIGMANEW(:,j) = []; 562 | deltaMu = - Mu(j)*tmp; 563 | mu_j = Mu(j); 564 | Mu = Mu +deltaMu; 565 | Mu(j) = []; 566 | % 567 | if UpdateIteration 568 | jPm = (BASIS_B_PHI * s_j); 569 | S_in = S_in + jPm.^2 / s_jj; 570 | Q_in = Q_in + jPm * mu_j / s_jj; 571 | end 572 | Used(j) = []; 573 | deleteCount = deleteCount+1; 574 | act_ = 'deletion'; 575 | % 576 | UPDATE_REQUIRED = true; 577 | % 578 | end % of switch over actions 579 | M = length(Used); 580 | % 581 | SB2_Diagnostic(OPTIONS, 3, 'ACTION: %s of %d (%g)\n', ... 582 | act_, nu, deltaLogMarginal); 583 | % 584 | 585 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 586 | 587 | %% UPDATE STATISTICS 588 | 589 | % If we've performed a meaningful action, 590 | % update the relevant variables 591 | % 592 | if UPDATE_REQUIRED 593 | % 594 | % S_in & Q_in values were calculated earlier 595 | % 596 | % Here: update the S_out/Q_out values and relevance factors 597 | % 598 | if UpdateIteration 599 | % 600 | % Previous "update" statisics calculated earlier are valid 601 | % 602 | S_out = S_in; 603 | Q_out = Q_in; 604 | tmp = Alpha ./ (Alpha - S_in(Used)); 605 | S_out(Used) = tmp .* S_in(Used); 606 | Q_out(Used) = tmp .* Q_in(Used); 607 | Factor = (Q_out.*Q_out - S_out); 608 | SIGMA = SIGMANEW; 609 | Gamma = 1 - Alpha.*diag(SIGMA); 610 | if LIKELIHOOD.InUse==LIKELIHOOD.Gaussian 611 | BASIS_B_PHI = beta*BASIS_PHI; 612 | else 613 | BASIS_B_PHI = ((PHI.* (beta*ones(1,M)))'*BASIS)'; 614 | end 615 | else 616 | % 617 | % Compute all statistics in "full" form (non-Gaussian likelihoods) 618 | % 619 | [SIGMA,Mu,S_in,Q_in,S_out,Q_out,Factor,newLogM,Gamma,... 620 | BASIS_B_PHI,beta] = ... 621 | SB2_FullStatistics(LIKELIHOOD,BASIS,PHI,Targets,Used,... 622 | Alpha,beta,Mu,BASIS_PHI,BASIS_Targets,OPTIONS); 623 | deltaLogMarginal = newLogM - logML; 624 | end 625 | % 626 | if UpdateIteration && deltaLogMarginal<0 627 | SB2_Diagnostic(OPTIONS,1,... 628 | '** Alert ** DECREASE IN LIKELIHOOD !! (%g)\n',... 629 | deltaLogMarginal); 630 | end 631 | % 632 | logML = logML + deltaLogMarginal; 633 | count = count + 1; 634 | logMarginalLog(count) = logML; 635 | end 636 | 637 | % GAUSSIAN NOISE ESTIMATE 638 | % 639 | % For Gaussian likelihood, re-estimate noise variance if: 640 | % 641 | % - not fixed, AND 642 | % - an update is specified this cycle as normal, OR 643 | % - we're considering termination 644 | % 645 | if LIKELIHOOD.InUse==LIKELIHOOD.Gaussian && ... 646 | ~OPTIONS.fixedNoise && ... 647 | (selectedAction == ACTION_TERMINATE || ... 648 | i<=CONTROLS.BetaUpdateStart || ... 649 | rem(i,CONTROLS.BetaUpdateFrequency)==0) 650 | % 651 | betaZ1 = beta; 652 | y = PHI * Mu; 653 | e = (Targets-y); 654 | beta = (N - sum(Gamma))/(e'*e); 655 | % Work-around zero-noise issue 656 | beta = min([beta CONTROLS.BetaMaxFactor/var(Targets)]); 657 | % 658 | deltaLogBeta = log(beta)-log(betaZ1); 659 | % 660 | if abs(deltaLogBeta) > CONTROLS.MinDeltaLogBeta 661 | % 662 | % Full re-computation of statistics required after beta update 663 | % 664 | [SIGMA,Mu,S_in,Q_in,S_out,Q_out,Factor,logML,Gamma,BASIS_B_PHI] = ... 665 | SB2_FullStatistics(LIKELIHOOD,BASIS,PHI,Targets,Used,Alpha,beta,... 666 | Mu, BASIS_PHI,BASIS_Targets,OPTIONS); 667 | % 668 | count = count + 1; 669 | logMarginalLog(count) = logML; 670 | % 671 | if selectedAction==ACTION_TERMINATE 672 | % 673 | % We considered terminating above as no alpha update seemed 674 | % worthwhile. However, a beta update has made a non-trivial 675 | % increase in the likelihood, so we continue. 676 | % 677 | selectedAction = ACTION_NOISE_ONLY; 678 | SB2_Diagnostic(OPTIONS,3,'Noise update (termination deferred)\n'); 679 | end 680 | end 681 | end 682 | 683 | % CALLBACK 684 | % 685 | % Call callback function if specified 686 | % - this can be useful for demos etc where it is desired to display 687 | % graphical information at each iteration 688 | % 689 | if OPTIONS.callback 690 | feval(OPTIONS.callbackFunc, i, selectedAction,... 691 | logMarginalLog(1:count)/N, ... 692 | Used, Mu./BasisScales(Used)', SIGMA,... 693 | Alpha, beta, Gamma, PHI.*(ones(N,1)*BasisScales(Used)),... 694 | OPTIONS.callbackData{:}) 695 | end 696 | 697 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 698 | 699 | % END OF CYCLE PROCESSING 700 | % 701 | % Check if termination still specified, and output diagnostics 702 | % 703 | if (selectedAction==ACTION_TERMINATE) 704 | % 705 | % If we're here, then no update to the model was considered worthwhile 706 | % 707 | SB2_Diagnostic(OPTIONS,2,... 708 | '** Stopping at iteration %d (Max_delta_ml=%g) **\n', ... 709 | i, deltaLogMarginal); 710 | if LIKELIHOOD.InUse~=LIKELIHOOD.Gaussian 711 | SB2_Diagnostic(OPTIONS,2,'%4d> L = %.6f\t Gamma = %.2f (M = %d)\n',... 712 | i, logML/N, sum(Gamma), M); 713 | else 714 | SB2_Diagnostic(OPTIONS,2,... 715 | '%4d> L = %.6f\t Gamma = %.2f (M = %d)\t s=%.3f\n',... 716 | i, logML/N, sum(Gamma), M, sqrt(1/beta)); 717 | end 718 | % Exit the main loop 719 | break; 720 | end 721 | % 722 | % Check for "natural" termination 723 | % 724 | ITERATION_LIMIT = (i==OPTIONS.iterations); 725 | TIME_LIMIT = (etime(clock,t0)>OPTIONS.time); 726 | LAST_ITERATION = ITERATION_LIMIT || TIME_LIMIT; 727 | % 728 | if (OPTIONS.monitor && ~rem(i,OPTIONS.monitor)) || LAST_ITERATION 729 | % 730 | % Output diagnostic progress info if desired 731 | % 732 | if LIKELIHOOD.InUse~=LIKELIHOOD.Gaussian 733 | SB2_Diagnostic(OPTIONS,2,'%5d> L = %.6f\t Gamma = %.2f (M = %d)\n',... 734 | i, logML/N, sum(Gamma), M); 735 | else 736 | SB2_Diagnostic(OPTIONS,2,... 737 | '%5d> L = %.6f\t Gamma = %.2f (M = %d)\t s=%.3f \n',... 738 | i, logML/N, sum(Gamma), M, sqrt(1/beta)); 739 | end 740 | end 741 | 742 | end % of MAIN LOOP 743 | 744 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 745 | 746 | %% 747 | %% POST-PROCESSING 748 | %% 749 | 750 | % 751 | % Warn if we exited the main loop without terminating automatically 752 | % 753 | if (selectedAction~=ACTION_TERMINATE) 754 | % 755 | if ITERATION_LIMIT 756 | SB2_Diagnostic(OPTIONS,1,... 757 | '** Iteration limit: algorithm did not converge\n'); 758 | elseif TIME_LIMIT 759 | SB2_Diagnostic(OPTIONS,1,... 760 | '** Time limit: algorithm did not converge\n'); 761 | end 762 | end 763 | % 764 | % Output action summary if desired 765 | % 766 | if OPTIONS.diagnosticLevel>1 767 | % Stop timer 768 | t1 = etime(clock,t0); 769 | total = addCount + deleteCount + updateCount; 770 | if CONTROLS.BasisAlignmentTest 771 | total = total+alignDeferCount; 772 | end 773 | % 774 | SB2_Diagnostic(OPTIONS,2,'Action Summary\n'); 775 | SB2_Diagnostic(OPTIONS,2,'==============\n'); 776 | SB2_Diagnostic(OPTIONS,2,'Added\t\t%6d (%.0f%%)\n',... 777 | addCount, 100*addCount/total); 778 | SB2_Diagnostic(OPTIONS,2,'Deleted\t\t%6d (%.0f%%)\n',... 779 | deleteCount, 100*deleteCount/total); 780 | SB2_Diagnostic(OPTIONS,2,'Reestimated\t%6d (%.0f%%)\n',... 781 | updateCount, 100*updateCount/total); 782 | % 783 | if CONTROLS.BasisAlignmentTest && alignDeferCount 784 | SB2_Diagnostic(OPTIONS,2,'--------------\n'); 785 | SB2_Diagnostic(OPTIONS,2,'Deferred\t%6d (%.0f%%)\n',... 786 | alignDeferCount, 100*alignDeferCount/total); 787 | end 788 | % 789 | SB2_Diagnostic(OPTIONS,2,'==============\n'); 790 | SB2_Diagnostic(OPTIONS,2,'Total of %d likelihood updates\n', count); 791 | SB2_Diagnostic(OPTIONS,2,'Time to run: %s\n', SB2_FormatTime(t1)); 792 | end 793 | 794 | % Terminate diagnostics 795 | OPTIONS = SB2_Diagnostic(OPTIONS, 'end'); 796 | 797 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 798 | 799 | %% 800 | %% OUTPUT VARIABLES 801 | %% 802 | if nargout>=1 803 | % We also choose to sort here - it can't hurt and may help later 804 | [PARAMETER.Relevant, index] = sort(Used); 805 | % Not forgetting to correct for normalisation too 806 | PARAMETER.Value = Mu(index) ./ BasisScales(Used(index))'; 807 | if nargout>=2 808 | % 809 | HYPERPARAMETER.Alpha = Alpha(index)./(BasisScales(Used(index))'.^2); 810 | HYPERPARAMETER.beta = beta; 811 | if nargout>=3 812 | % 813 | DIAGNOSTIC.Gamma = Gamma(index); 814 | DIAGNOSTIC.Likelihood = logMarginalLog(1:count); 815 | DIAGNOSTIC.iterations = i; 816 | DIAGNOSTIC.S_Factor = S_out; 817 | DIAGNOSTIC.Q_Factor = Q_out; 818 | DIAGNOSTIC.M_full = M_full; 819 | end 820 | end 821 | end 822 | 823 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 824 | -------------------------------------------------------------------------------- /SB2_Release_200/SparseBayesDemo.m: -------------------------------------------------------------------------------- 1 | % SPARSEBAYESDEMO Simple demonstration of the SPARSEBAYES algorithm 2 | % 3 | % SPARSEBAYESDEMO(LIKELIHOOD, DIMENSION, NOISETOSIGNAL) 4 | % 5 | % OUTPUT ARGUMENTS: None 6 | % 7 | % INPUT ARGUMENTS: 8 | % 9 | % LIKELIHOOD Text string, one of 'Gaussian' or 'Bernoulli' 10 | % DIMENSION Integer, 1 or 2 11 | % NOISETOSIGNAL An optional positive number to specify the 12 | % noise-to-signal (standard deviation) fraction. 13 | % (Optional: default value is 0.2). 14 | % 15 | % EXAMPLES: 16 | % 17 | % SPARSEBAYESDEMO("Bernoulli",2) 18 | % SPARSEBAYESDEMO("Gaussian",1,0.5) 19 | % 20 | % NOTES: 21 | % 22 | % This program offers a simple demonstration of how to use the 23 | % SPARSEBAYES (V2) Matlab software. 24 | % 25 | % Synthetic data is generated from an underlying linear model based 26 | % on a set of "Gaussian" basis functions, with the generator being 27 | % "sparse" such that 10% of potential weights are non-zero. Data may be 28 | % generated in an input space of one or two dimensions. 29 | % 30 | % This generator is then used either as the basis for real-valued data with 31 | % additive Gaussian noise (whose level may be varied), or for binary 32 | % class-labelled data based on probabilities given by a sigmoid link 33 | % function. 34 | % 35 | % The SPARSEBAYES algorithm is then run on the data, and results and 36 | % diagnostic information are graphed. 37 | % 38 | 39 | % 40 | % Copyright 2009, Vector Anomaly Ltd 41 | % 42 | % This file is part of the SPARSEBAYES library for Matlab (V2.0). 43 | % 44 | % SPARSEBAYES is free software; you can redistribute it and/or modify it 45 | % under the terms of the GNU General Public License as published by the Free 46 | % Software Foundation; either version 2 of the License, or (at your option) 47 | % any later version. 48 | % 49 | % SPARSEBAYES is distributed in the hope that it will be useful, but WITHOUT 50 | % ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or 51 | % FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for 52 | % more details. 53 | % 54 | % You should have received a copy of the GNU General Public License along 55 | % with SPARSEBAYES in the accompanying file "licence.txt"; if not, write to 56 | % the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, 57 | % MA 02110-1301 USA 58 | % 59 | % Contact the author: m a i l [at] m i k e t i p p i n g . c o m 60 | % 61 | function SparseBayesDemo(likelihood_, dimension, noiseToSignal) 62 | 63 | % Fix the random seed for reproducibility of results 64 | % 65 | rseed = 1; 66 | rand('state',rseed); 67 | randn('state',rseed); 68 | 69 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 70 | % 71 | % --- VALIDATE INPUT ARGUMENTS --- 72 | % 73 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 74 | % 75 | % (1) likelihood 76 | % 77 | LIKELIHOOD = SB2_Likelihoods(likelihood_); 78 | % 79 | % (2) dimension 80 | % 81 | if dimension~=1 && dimension~=2 82 | error('Specified dimension should be 1 or 2') 83 | end 84 | % 85 | % Set up default for "noiseToSignal" variable. 86 | % For ease of use, we'll just ignore it in the case of a non-Gaussian 87 | % likelihood model. 88 | % 89 | if ~exist('noiseToSignal','var') 90 | noiseToSignal = 0.2; 91 | end 92 | 93 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 94 | % 95 | % --- SET UP DEMO PARAMETERS --- 96 | % 97 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 98 | % 99 | % Experiment with these values to vary the demo if you wish 100 | % 101 | if dimension==1 102 | N = 100; % Number of points 103 | else 104 | N = 900; % Gives a nice square grid of decent size 105 | end 106 | % 107 | basisWidth = 0.05; % NB: data is in [0,1] 108 | % 109 | % Define probability of a basis function NOT being used by the generative 110 | % model. i.e. if pSparse=0.90, only 10% of basis functions (on average) will 111 | % be used to synthesise the data. 112 | % 113 | pSparse = 0.90; 114 | iterations = 500; 115 | % 116 | % Heuristically adjust basis width to account for 117 | % distance scaling with dimension. 118 | % 119 | basisWidth = basisWidth^(1/dimension); 120 | % 121 | 122 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 123 | % 124 | % --- SYNTHETIC DATA GENERATION --- 125 | % 126 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 127 | % 128 | % First define the input data over a regular grid 129 | % 130 | if dimension==1 131 | X = [0:N-1]'/N; 132 | else 133 | % dimension is 2 134 | sqrtN = floor(sqrt(N)); 135 | N = sqrtN*sqrtN; 136 | x = [0:sqrtN-1]'/sqrtN; 137 | [gx gy] = meshgrid(x); 138 | X = [gx(:) gy(:)]; 139 | end 140 | % 141 | % Now define the basis 142 | % 143 | % Locate basis functions at data points 144 | % 145 | C = X; 146 | % 147 | % Compute ("Gaussian") basis (design) matrix 148 | % 149 | BASIS = exp(-distSquared(X,C)/(basisWidth^2)); 150 | % 151 | % 152 | % Randomise some weights, then make each weight sparse with probability 153 | % pSparse 154 | % 155 | M = size(BASIS,2); 156 | w = randn(M,1)*100 / (M*(1-pSparse)); 157 | sparse = rand(M,1)0.5); 261 | end 262 | % 263 | 264 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 265 | % 266 | % --- PLOT THE RESULTS --- 267 | % 268 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 269 | % 270 | % Likelihood trace (and Gaussian noise info) 271 | % 272 | subplot(fRows,fCols,SP_LIKELY) 273 | lsteps = length(DIAGNOSTIC.Likelihood); 274 | plot(1:lsteps, DIAGNOSTIC.Likelihood, 'g-') 275 | set(gca,'Xlim',[0 lsteps+1]) 276 | title('Log marginal likelihood trace','FontSize',TITLE_SIZE) 277 | if LIKELIHOOD.InUse==LIKELIHOOD.Gaussian 278 | ax = axis; 279 | dx = ax(2)-ax(1); 280 | dy = ax(4)-ax(3); 281 | t_ = sprintf('Actual noise: %.5f', noise); 282 | text(ax(1)+0.1*dx,ax(3)+0.6*dy,t_,'FontName','Courier') 283 | t_ = sprintf('Inferred noise: %.5f', 1/sqrt(HYPERPARAMETER.beta)); 284 | text(ax(1)+0.1*dx,ax(3)+0.5*dy,t_,'FontName','Courier') 285 | end 286 | % 287 | % Compare the generative and predictive linear models 288 | % 289 | subplot(fRows,fCols,SP_LINEAR) 290 | if dimension==1 291 | plot(X,z,'w-','linewidth',4); 292 | hold on 293 | plot(X,y,'r-','linewidth',3); 294 | hold off 295 | else 296 | mesh(gx,gy,reshape(z,size(gx)),'edgecolor','w','facecolor','w') 297 | hold on 298 | mesh(gx,gy,reshape(y,size(gx)),'edgecolor','r','facecolor','r') 299 | hold off 300 | light 301 | end 302 | title('Generative function and linear model','FontSize',TITLE_SIZE) 303 | legend('Actual','Model','Location','NorthWest') 304 | legend('boxoff') 305 | % 306 | % Compare the data and the predictive model (post link-function) 307 | % 308 | subplot(fRows,fCols,SP_COMPARE) 309 | if dimension==1 310 | plot(X,Outputs,'w.','linewidth',4); 311 | hold on 312 | plot(X,y_l,'r-','linewidth',3); 313 | plot(X(PARAMETER.Relevant),Outputs(PARAMETER.Relevant),'yo') 314 | hold off 315 | else 316 | plot3(X(:,1),X(:,2),Outputs,'w.') 317 | hold on 318 | mesh(gx,gy,reshape(y_l,size(gx)),'edgecolor','r','facecolor','r') 319 | hold off 320 | light 321 | end 322 | title('Data and predictor','FontSize',TITLE_SIZE) 323 | % 324 | % Show the inferred weights 325 | % 326 | subplot(fRows,fCols,SP_WEIGHTS) 327 | h = stem(w_infer,'filled'); 328 | set(h,'Markersize',3,'Color','r') 329 | set(gca,'Xlim',[0 N+1]) 330 | t_ = sprintf('Inferred weights (%d)', length(PARAMETER.Relevant)); 331 | title(t_,'FontSize',TITLE_SIZE) 332 | % 333 | % Show the "well-determinedness" factors 334 | % 335 | subplot(fRows,fCols,SP_GAMMA) 336 | bar(DIAGNOSTIC.Gamma,'g') 337 | axis([0 length(PARAMETER.Relevant)+1 0 1.1]) 338 | title('Well-determinedness (gamma)','FontSize',TITLE_SIZE) 339 | 340 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 341 | % 342 | % Support function to compute basis 343 | % 344 | function D2 = distSquared(X,Y) 345 | % 346 | nx = size(X,1); 347 | ny = size(Y,1); 348 | % 349 | D2 = (sum((X.^2), 2) * ones(1,ny)) + (ones(nx, 1) * sum((Y.^2),2)') - ... 350 | 2*X*Y'; 351 | 352 | -------------------------------------------------------------------------------- /SB2_Release_200/licence.txt: -------------------------------------------------------------------------------- 1 | GNU GENERAL PUBLIC LICENSE 2 | Version 2, June 1991 3 | 4 | Copyright (C) 1989, 1991 Free Software Foundation, Inc., 5 | 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA 6 | Everyone is permitted to copy and distribute verbatim copies 7 | of this license document, but changing it is not allowed. 8 | 9 | Preamble 10 | 11 | The licenses for most software are designed to take away your 12 | freedom to share and change it. By contrast, the GNU General Public 13 | License is intended to guarantee your freedom to share and change free 14 | software--to make sure the software is free for all its users. This 15 | General Public License applies to most of the Free Software 16 | Foundation's software and to any other program whose authors commit to 17 | using it. (Some other Free Software Foundation software is covered by 18 | the GNU Lesser General Public License instead.) You can apply it to 19 | your programs, too. 20 | 21 | When we speak of free software, we are referring to freedom, not 22 | price. Our General Public Licenses are designed to make sure that you 23 | have the freedom to distribute copies of free software (and charge for 24 | this service if you wish), that you receive source code or can get it 25 | if you want it, that you can change the software or use pieces of it 26 | in new free programs; and that you know you can do these things. 27 | 28 | To protect your rights, we need to make restrictions that forbid 29 | anyone to deny you these rights or to ask you to surrender the rights. 30 | These restrictions translate to certain responsibilities for you if you 31 | distribute copies of the software, or if you modify it. 32 | 33 | For example, if you distribute copies of such a program, whether 34 | gratis or for a fee, you must give the recipients all the rights that 35 | you have. You must make sure that they, too, receive or can get the 36 | source code. And you must show them these terms so they know their 37 | rights. 38 | 39 | We protect your rights with two steps: (1) copyright the software, and 40 | (2) offer you this license which gives you legal permission to copy, 41 | distribute and/or modify the software. 42 | 43 | Also, for each author's protection and ours, we want to make certain 44 | that everyone understands that there is no warranty for this free 45 | software. If the software is modified by someone else and passed on, we 46 | want its recipients to know that what they have is not the original, so 47 | that any problems introduced by others will not reflect on the original 48 | authors' reputations. 49 | 50 | Finally, any free program is threatened constantly by software 51 | patents. We wish to avoid the danger that redistributors of a free 52 | program will individually obtain patent licenses, in effect making the 53 | program proprietary. To prevent this, we have made it clear that any 54 | patent must be licensed for everyone's free use or not licensed at all. 55 | 56 | The precise terms and conditions for copying, distribution and 57 | modification follow. 58 | 59 | GNU GENERAL PUBLIC LICENSE 60 | TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 61 | 62 | 0. This License applies to any program or other work which contains 63 | a notice placed by the copyright holder saying it may be distributed 64 | under the terms of this General Public License. The "Program", below, 65 | refers to any such program or work, and a "work based on the Program" 66 | means either the Program or any derivative work under copyright law: 67 | that is to say, a work containing the Program or a portion of it, 68 | either verbatim or with modifications and/or translated into another 69 | language. (Hereinafter, translation is included without limitation in 70 | the term "modification".) Each licensee is addressed as "you". 71 | 72 | Activities other than copying, distribution and modification are not 73 | covered by this License; they are outside its scope. The act of 74 | running the Program is not restricted, and the output from the Program 75 | is covered only if its contents constitute a work based on the 76 | Program (independent of having been made by running the Program). 77 | Whether that is true depends on what the Program does. 78 | 79 | 1. You may copy and distribute verbatim copies of the Program's 80 | source code as you receive it, in any medium, provided that you 81 | conspicuously and appropriately publish on each copy an appropriate 82 | copyright notice and disclaimer of warranty; keep intact all the 83 | notices that refer to this License and to the absence of any warranty; 84 | and give any other recipients of the Program a copy of this License 85 | along with the Program. 86 | 87 | You may charge a fee for the physical act of transferring a copy, and 88 | you may at your option offer warranty protection in exchange for a fee. 89 | 90 | 2. You may modify your copy or copies of the Program or any portion 91 | of it, thus forming a work based on the Program, and copy and 92 | distribute such modifications or work under the terms of Section 1 93 | above, provided that you also meet all of these conditions: 94 | 95 | a) You must cause the modified files to carry prominent notices 96 | stating that you changed the files and the date of any change. 97 | 98 | b) You must cause any work that you distribute or publish, that in 99 | whole or in part contains or is derived from the Program or any 100 | part thereof, to be licensed as a whole at no charge to all third 101 | parties under the terms of this License. 102 | 103 | c) If the modified program normally reads commands interactively 104 | when run, you must cause it, when started running for such 105 | interactive use in the most ordinary way, to print or display an 106 | announcement including an appropriate copyright notice and a 107 | notice that there is no warranty (or else, saying that you provide 108 | a warranty) and that users may redistribute the program under 109 | these conditions, and telling the user how to view a copy of this 110 | License. (Exception: if the Program itself is interactive but 111 | does not normally print such an announcement, your work based on 112 | the Program is not required to print an announcement.) 113 | 114 | These requirements apply to the modified work as a whole. If 115 | identifiable sections of that work are not derived from the Program, 116 | and can be reasonably considered independent and separate works in 117 | themselves, then this License, and its terms, do not apply to those 118 | sections when you distribute them as separate works. But when you 119 | distribute the same sections as part of a whole which is a work based 120 | on the Program, the distribution of the whole must be on the terms of 121 | this License, whose permissions for other licensees extend to the 122 | entire whole, and thus to each and every part regardless of who wrote it. 123 | 124 | Thus, it is not the intent of this section to claim rights or contest 125 | your rights to work written entirely by you; rather, the intent is to 126 | exercise the right to control the distribution of derivative or 127 | collective works based on the Program. 128 | 129 | In addition, mere aggregation of another work not based on the Program 130 | with the Program (or with a work based on the Program) on a volume of 131 | a storage or distribution medium does not bring the other work under 132 | the scope of this License. 133 | 134 | 3. You may copy and distribute the Program (or a work based on it, 135 | under Section 2) in object code or executable form under the terms of 136 | Sections 1 and 2 above provided that you also do one of the following: 137 | 138 | a) Accompany it with the complete corresponding machine-readable 139 | source code, which must be distributed under the terms of Sections 140 | 1 and 2 above on a medium customarily used for software interchange; or, 141 | 142 | b) Accompany it with a written offer, valid for at least three 143 | years, to give any third party, for a charge no more than your 144 | cost of physically performing source distribution, a complete 145 | machine-readable copy of the corresponding source code, to be 146 | distributed under the terms of Sections 1 and 2 above on a medium 147 | customarily used for software interchange; or, 148 | 149 | c) Accompany it with the information you received as to the offer 150 | to distribute corresponding source code. (This alternative is 151 | allowed only for noncommercial distribution and only if you 152 | received the program in object code or executable form with such 153 | an offer, in accord with Subsection b above.) 154 | 155 | The source code for a work means the preferred form of the work for 156 | making modifications to it. For an executable work, complete source 157 | code means all the source code for all modules it contains, plus any 158 | associated interface definition files, plus the scripts used to 159 | control compilation and installation of the executable. However, as a 160 | special exception, the source code distributed need not include 161 | anything that is normally distributed (in either source or binary 162 | form) with the major components (compiler, kernel, and so on) of the 163 | operating system on which the executable runs, unless that component 164 | itself accompanies the executable. 165 | 166 | If distribution of executable or object code is made by offering 167 | access to copy from a designated place, then offering equivalent 168 | access to copy the source code from the same place counts as 169 | distribution of the source code, even though third parties are not 170 | compelled to copy the source along with the object code. 171 | 172 | 4. You may not copy, modify, sublicense, or distribute the Program 173 | except as expressly provided under this License. Any attempt 174 | otherwise to copy, modify, sublicense or distribute the Program is 175 | void, and will automatically terminate your rights under this License. 176 | However, parties who have received copies, or rights, from you under 177 | this License will not have their licenses terminated so long as such 178 | parties remain in full compliance. 179 | 180 | 5. You are not required to accept this License, since you have not 181 | signed it. However, nothing else grants you permission to modify or 182 | distribute the Program or its derivative works. These actions are 183 | prohibited by law if you do not accept this License. Therefore, by 184 | modifying or distributing the Program (or any work based on the 185 | Program), you indicate your acceptance of this License to do so, and 186 | all its terms and conditions for copying, distributing or modifying 187 | the Program or works based on it. 188 | 189 | 6. Each time you redistribute the Program (or any work based on the 190 | Program), the recipient automatically receives a license from the 191 | original licensor to copy, distribute or modify the Program subject to 192 | these terms and conditions. You may not impose any further 193 | restrictions on the recipients' exercise of the rights granted herein. 194 | You are not responsible for enforcing compliance by third parties to 195 | this License. 196 | 197 | 7. If, as a consequence of a court judgment or allegation of patent 198 | infringement or for any other reason (not limited to patent issues), 199 | conditions are imposed on you (whether by court order, agreement or 200 | otherwise) that contradict the conditions of this License, they do not 201 | excuse you from the conditions of this License. If you cannot 202 | distribute so as to satisfy simultaneously your obligations under this 203 | License and any other pertinent obligations, then as a consequence you 204 | may not distribute the Program at all. For example, if a patent 205 | license would not permit royalty-free redistribution of the Program by 206 | all those who receive copies directly or indirectly through you, then 207 | the only way you could satisfy both it and this License would be to 208 | refrain entirely from distribution of the Program. 209 | 210 | If any portion of this section is held invalid or unenforceable under 211 | any particular circumstance, the balance of the section is intended to 212 | apply and the section as a whole is intended to apply in other 213 | circumstances. 214 | 215 | It is not the purpose of this section to induce you to infringe any 216 | patents or other property right claims or to contest validity of any 217 | such claims; this section has the sole purpose of protecting the 218 | integrity of the free software distribution system, which is 219 | implemented by public license practices. Many people have made 220 | generous contributions to the wide range of software distributed 221 | through that system in reliance on consistent application of that 222 | system; it is up to the author/donor to decide if he or she is willing 223 | to distribute software through any other system and a licensee cannot 224 | impose that choice. 225 | 226 | This section is intended to make thoroughly clear what is believed to 227 | be a consequence of the rest of this License. 228 | 229 | 8. If the distribution and/or use of the Program is restricted in 230 | certain countries either by patents or by copyrighted interfaces, the 231 | original copyright holder who places the Program under this License 232 | may add an explicit geographical distribution limitation excluding 233 | those countries, so that distribution is permitted only in or among 234 | countries not thus excluded. In such case, this License incorporates 235 | the limitation as if written in the body of this License. 236 | 237 | 9. The Free Software Foundation may publish revised and/or new versions 238 | of the General Public License from time to time. Such new versions will 239 | be similar in spirit to the present version, but may differ in detail to 240 | address new problems or concerns. 241 | 242 | Each version is given a distinguishing version number. If the Program 243 | specifies a version number of this License which applies to it and "any 244 | later version", you have the option of following the terms and conditions 245 | either of that version or of any later version published by the Free 246 | Software Foundation. If the Program does not specify a version number of 247 | this License, you may choose any version ever published by the Free Software 248 | Foundation. 249 | 250 | 10. If you wish to incorporate parts of the Program into other free 251 | programs whose distribution conditions are different, write to the author 252 | to ask for permission. For software which is copyrighted by the Free 253 | Software Foundation, write to the Free Software Foundation; we sometimes 254 | make exceptions for this. Our decision will be guided by the two goals 255 | of preserving the free status of all derivatives of our free software and 256 | of promoting the sharing and reuse of software generally. 257 | 258 | NO WARRANTY 259 | 260 | 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY 261 | FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN 262 | OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES 263 | PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED 264 | OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF 265 | MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS 266 | TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE 267 | PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, 268 | REPAIR OR CORRECTION. 269 | 270 | 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING 271 | WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR 272 | REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, 273 | INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING 274 | OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED 275 | TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY 276 | YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER 277 | PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE 278 | POSSIBILITY OF SUCH DAMAGES. 279 | 280 | END OF TERMS AND CONDITIONS 281 | -------------------------------------------------------------------------------- /custom_option.m: -------------------------------------------------------------------------------- 1 | %% custom optimization option 2 | %{ 3 | opt.method = 'bayes'; % bayes, ga, pso 4 | opt.display = 'on'; 5 | opt.iteration = 20; 6 | opt.point = 10; 7 | 8 | % gaussian kernel function 9 | opt.gaussian.parameterName = {'gamma'}; 10 | opt.gaussian.parameterType = {'real'}; 11 | opt.gaussian.lowerBound = 2^-6; 12 | opt.gaussian.upperBound = 2^6; 13 | 14 | % laplacian kernel function 15 | opt.laplacian.parameterName = {'gamma'}; 16 | opt.laplacian.parameterType = {'real'}; 17 | opt.laplacian.lowerBound = 2^-6; 18 | opt.laplacian.upperBound = 2^6; 19 | 20 | % polynomial kernel function 21 | opt.polynomial.parameterName = {'gamma'; 'offset'; 'degree'}; 22 | opt.polynomial.parameterType = {'real'; 'real'; 'integer'}; 23 | opt.polynomial.lowerBound = [2^-6; 2^-6; 1]; 24 | opt.polynomial.upperBound = [2^6; 2^6; 7]; 25 | 26 | % sigmoid kernel function 27 | opt.sigmoid.parameterName = {'gamma'; 'offset'}; 28 | opt.sigmoid.parameterType = {'real'; 'real'}; 29 | opt.sigmoid.lowerBound = [2^-6; 2^-6]; 30 | opt.sigmoid.upperBound = [2^6; 2^6]; 31 | %} 32 | 33 | %% RVM model parameter 34 | %{ 35 | 'display' : 'on', 'off' 36 | 'type' : 'RVR', 'RVC' 37 | 'kernelFunc' : kernel function 38 | 'KFolds' : cross validation, for example, 5 39 | 'HoldOut' : cross validation, for example, 0.3 40 | 'freeBasis' : 'on', 'off' 41 | 'maxIter' : max iteration, for example, 1000 42 | %} 43 | -------------------------------------------------------------------------------- /data/UCI_data.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/iqiukp/RVM-MATLAB/fd70db51f1333525ad4603a35b0c97ccf6ba68ba/data/UCI_data.mat -------------------------------------------------------------------------------- /data/sinc_data.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/iqiukp/RVM-MATLAB/fd70db51f1333525ad4603a35b0c97ccf6ba68ba/data/sinc_data.mat -------------------------------------------------------------------------------- /demo_RVC.m: -------------------------------------------------------------------------------- 1 | %{ 2 | A demo for classification using RVM 3 | %} 4 | 5 | clc 6 | clear all 7 | close all 8 | addpath(genpath(pwd)) 9 | 10 | % use fisheriris dataset 11 | load fisheriris 12 | inds = ~strcmp(species, 'setosa'); 13 | data_ = meas(inds, 3:4); 14 | label_ = species(inds); 15 | cvIndices = crossvalind('HoldOut', length(data_), 0.3); 16 | trainData = data_(cvIndices, :); 17 | trainLabel = label_(cvIndices, :); 18 | testData = data_(~cvIndices, :); 19 | testLabel = label_(~cvIndices, :); 20 | 21 | % kernel function 22 | kernel = Kernel('type', 'gaussian', 'gamma', 0.2); 23 | 24 | % parameter 25 | parameter = struct( 'display', 'on',... 26 | 'type', 'RVC',... 27 | 'kernelFunc', kernel); 28 | rvm = BaseRVM(parameter); 29 | 30 | % RVM model training, testing, and visualization 31 | rvm.train(trainData, trainLabel); 32 | results = rvm.test(testData, testLabel); 33 | rvm.draw(results) -------------------------------------------------------------------------------- /demo_RVR.m: -------------------------------------------------------------------------------- 1 | %{ 2 | A demo for regression using RVM 3 | %} 4 | 5 | clc 6 | clear all 7 | close all 8 | addpath(genpath(pwd)) 9 | 10 | % sinc funciton 11 | load sinc_data 12 | trainData = x; 13 | trainLabel = y; 14 | testData = xt; 15 | testLabel = yt; 16 | 17 | % kernel function 18 | kernel = Kernel('type', 'gaussian', 'gamma', 0.1); 19 | 20 | % parameter 21 | parameter = struct( 'display', 'on',... 22 | 'type', 'RVR',... 23 | 'kernelFunc', kernel); 24 | rvm = BaseRVM(parameter); 25 | 26 | % RVM model training, testing, and visualization 27 | rvm.train(trainData, trainLabel); 28 | results = rvm.test(testData, testLabel); 29 | rvm.draw(results) 30 | 31 | 32 | -------------------------------------------------------------------------------- /demo_hybrid_kernel.m: -------------------------------------------------------------------------------- 1 | %{ 2 | A demo for regression using RVM with hybrid_kernel 3 | %} 4 | 5 | clc 6 | clear all 7 | close all 8 | addpath(genpath(pwd)) 9 | 10 | % sinc funciton 11 | load sinc_data 12 | trainData = x; 13 | trainLabel = y; 14 | testData = xt; 15 | testLabel = yt; 16 | 17 | % kernel function 18 | kernel_1 = Kernel('type', 'gaussian', 'gamma', 0.3); 19 | kernel_2 = Kernel('type', 'polynomial', 'degree', 2); 20 | kernelWeight = [0.5, 0.5]; 21 | % parameter 22 | parameter = struct( 'display', 'on',... 23 | 'type', 'RVR',... 24 | 'kernelFunc', [kernel_1, kernel_2],... 25 | 'kernelWeight', kernelWeight); 26 | rvm = BaseRVM(parameter); 27 | 28 | % RVM model training, testing, and visualization 29 | rvm.train(trainData, trainLabel); 30 | results = rvm.test(testData, testLabel); 31 | rvm.draw(results) 32 | 33 | 34 | -------------------------------------------------------------------------------- /demo_hybrid_kernel_optimization.m: -------------------------------------------------------------------------------- 1 | %{ 2 | A demo for hybrid-kernel RVM model with Parameter Optimization 3 | %} 4 | 5 | 6 | clc 7 | clear all 8 | close all 9 | addpath(genpath(pwd)) 10 | 11 | % data 12 | load UCI_data 13 | trainData = x; 14 | trainLabel = y; 15 | testData = xt; 16 | testLabel = yt; 17 | 18 | % kernel function 19 | kernel_1 = Kernel('type', 'gaussian', 'gamma', 0.5); 20 | kernel_2 = Kernel('type', 'polynomial', 'degree', 2); 21 | 22 | % parameter optimization 23 | opt.method = 'bayes'; % bayes, ga, pso 24 | opt.display = 'on'; 25 | opt.iteration = 30; 26 | 27 | % parameter 28 | parameter = struct( 'display', 'on',... 29 | 'type', 'RVR',... 30 | 'kernelFunc', [kernel_1, kernel_2],... 31 | 'optimization', opt); 32 | rvm = BaseRVM(parameter); 33 | 34 | % RVM model training, testing, and visualization 35 | rvm.train(trainData, trainLabel); 36 | results = rvm.test(testData, testLabel); 37 | rvm.draw(results) -------------------------------------------------------------------------------- /demo_kernel_function.m: -------------------------------------------------------------------------------- 1 | %{ 2 | Demonstration of computing the kernel function matrix. 3 | %} 4 | 5 | clc 6 | clear all 7 | close all 8 | addpath(genpath(pwd)) 9 | 10 | % data 11 | X = rand(33, 2); 12 | Y = rand(23, 2); 13 | 14 | % kernel setting 15 | kernel1 = Kernel('type', 'gaussian', 'gamma', 0.01); 16 | kernel2 = Kernel('type', 'polynomial', 'degree', 3); 17 | kernel3 = Kernel('type', 'linear'); 18 | kernel4 = Kernel('type', 'sigmoid', 'gamma', 0.01); 19 | kernel5 = Kernel('type', 'laplacian', 'gamma', 0.1); 20 | 21 | % compute the kernel function matrix 22 | matrix{1} = kernel1.computeMatrix(X, Y); 23 | matrix{2} = kernel2.computeMatrix(X, Y); 24 | matrix{3} = kernel3.computeMatrix(X, Y); 25 | matrix{4} = kernel4.computeMatrix(X, Y); 26 | matrix{5} = kernel5.computeMatrix(X, Y); -------------------------------------------------------------------------------- /demo_optimization.m: -------------------------------------------------------------------------------- 1 | %{ 2 | A demo for RVM model with Parameter Optimization 3 | %} 4 | 5 | clc 6 | clear all 7 | close all 8 | addpath(genpath(pwd)) 9 | 10 | % use fisheriris dataset 11 | load fisheriris 12 | inds = ~strcmp(species, 'setosa'); 13 | data_ = meas(inds, 3:4); 14 | label_ = species(inds); 15 | cvIndices = crossvalind('HoldOut', length(data_), 0.3); 16 | trainData = data_(cvIndices, :); 17 | trainLabel = label_(cvIndices, :); 18 | testData = data_(~cvIndices, :); 19 | testLabel = label_(~cvIndices, :); 20 | 21 | % kernel function 22 | kernel = Kernel('type', 'gaussian', 'gamma', 5); 23 | 24 | % parameter optimization 25 | opt.method = 'bayes'; % bayes, ga, pso 26 | opt.display = 'on'; 27 | opt.iteration = 20; 28 | 29 | % parameter 30 | parameter = struct( 'display', 'on',... 31 | 'type', 'RVC',... 32 | 'kernelFunc', kernel,... 33 | 'optimization', opt); 34 | rvm = BaseRVM(parameter); 35 | 36 | % RVM model training, testing, and visualization 37 | rvm.train(trainData, trainLabel); 38 | results = rvm.test(testData, testLabel); 39 | rvm.draw(results) 40 | 41 | 42 | --------------------------------------------------------------------------------